2024-03-06T07:09:50.080 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:09:50.081 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} --name leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi --verbose -- /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061/orig.config.yaml 2024-03-06T07:09:50.123 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:09:50.226 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} email: null first_in_suite: false job_id: '7583061' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL92MghNmvkK0dl0CgJMYwSUjvF1nywFkmBXqC2ED6NNW+83viPD25Wkyb6fgauJmStWWAHWHrm5awA7vedLBX4= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIiHbBK7ER2izv74TGbbmBGXlapy6lqgySg8fZnYGemXRtRo+HJIO8A9KRgScyXvENad4XZmvrFXKwKusDHYcjU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:04:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:09:50.226 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:09:50.226 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:09:50.227 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:09:50.228 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:09:50.252 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:09:50.252 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:09:50.253 INFO:teuthology.packaging:ref: None 2024-03-06T07:09:50.253 INFO:teuthology.packaging:tag: None 2024-03-06T07:09:50.253 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:09:50.253 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:09:50.253 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:09:50.595 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:09:50.596 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:09:50.602 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:09:50.602 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:09:50.629 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:09:50.644 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:09:50.649 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:09:50.673 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.055963', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL92MghNmvkK0dl0CgJMYwSUjvF1nywFkmBXqC2ED6NNW+83viPD25Wkyb6fgauJmStWWAHWHrm5awA7vedLBX4='} 2024-03-06T07:09:50.694 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.054351', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIiHbBK7ER2izv74TGbbmBGXlapy6lqgySg8fZnYGemXRtRo+HJIO8A9KRgScyXvENad4XZmvrFXKwKusDHYcjU='} 2024-03-06T07:09:50.694 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:09:50.700 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-06T07:09:50.700 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-06T07:09:50.700 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:09:50.766 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1ca7bbaf70>, signals=[15]) 2024-03-06T07:09:50.766 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:09:50.771 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:09:50.772 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:09:50.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:50.889 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:09:50.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:09:50.980 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:09:50.986 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2024-03-06T07:09:51.004 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2024-03-06T07:09:51.004 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2024-03-06T07:09:51.061 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2024-03-06T07:09:51.061 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="9" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="9" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:09:51.062 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:09:51.063 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-03-06T07:09:51.097 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-03-06T07:09:51.115 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-03-06T07:09:51.115 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-03-06T07:09:51.172 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2024-03-06T07:09:51.172 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="9" 2024-03-06T07:09:51.172 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2024-03-06T07:09:51.172 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:09:51.172 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="9" 2024-03-06T07:09:51.172 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:09:51.173 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:09:51.173 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2024-03-06T07:09:51.173 INFO:teuthology.orchestra.run.smithi164.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:09:51.173 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:09:51.173 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:09:51.173 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:09:51.173 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:09:51.173 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:09:51.173 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-03-06T07:09:51.197 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:09:51.206 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:09:51.211 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:09:51.211 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:09:51.214 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:09:51.229 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:09:51.234 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:09:51.235 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:09:51.270 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:09:51.430 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:09:51.486 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:09:51.501 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-06T07:09:51.501 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-06T07:09:51.501 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:09:51.501 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:09:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:09:51.518 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-03-06T07:09:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:09:51.519 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-03-06T07:09:51.575 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:09:51.575 INFO:teuthology.task.kernel:Running kernel on smithi164: 5.14.0-361.el9.x86_64 2024-03-06T07:09:51.575 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum install -y kernel 2024-03-06T07:09:51.577 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:09:51.577 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.14.0-361.el9.x86_64 2024-03-06T07:09:51.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2024-03-06T07:09:54.215 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - AppStream 6.9 MB/s | 6.7 MB 00:00 2024-03-06T07:09:54.216 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 6.5 MB/s | 6.7 MB 00:01 2024-03-06T07:09:55.832 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-06T07:09:55.858 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-06T07:09:56.956 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-06T07:09:57.648 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-03-06T07:09:59.665 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-06T07:10:04.744 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-03-06T07:10:06.250 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 15 kB 00:00 2024-03-06T07:10:07.576 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-06T07:10:12.776 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-06T07:10:15.033 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:10:15.033 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:10:15.139 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-06T07:10:15.143 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-06T07:10:15.143 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2024-03-06T07:10:15.143 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-06T07:10:15.143 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-06T07:10:15.143 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:10:15.143 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-06T07:10:15.144 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:10:15.144 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:10:15.144 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:10:15.144 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:10:15.144 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-06T07:10:15.144 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2024-03-06T07:10:15.144 INFO:teuthology.orchestra.run.smithi059.stdout:Install 4 Packages 2024-03-06T07:10:15.144 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:10:15.145 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 100 M 2024-03-06T07:10:15.145 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 123 M 2024-03-06T07:10:15.145 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-06T07:10:16.861 INFO:teuthology.orchestra.run.smithi059.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 3.9 MB/s | 6.0 MB 00:01 2024-03-06T07:10:21.261 INFO:teuthology.orchestra.run.smithi059.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 3.5 MB/s | 21 MB 00:05 2024-03-06T07:10:23.120 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 767 kB/s | 19 MB 00:25 2024-03-06T07:10:23.719 INFO:teuthology.orchestra.run.smithi059.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 4.7 MB/s | 39 MB 00:08 2024-03-06T07:10:24.544 INFO:teuthology.orchestra.run.smithi059.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 4.3 MB/s | 33 MB 00:07 2024-03-06T07:10:24.544 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:10:24.545 INFO:teuthology.orchestra.run.smithi059.stdout:Total 11 MB/s | 100 MB 00:09 2024-03-06T07:10:25.468 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-06T07:10:25.919 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-06T07:10:25.920 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-06T07:10:27.213 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.1 MB 00:00 2024-03-06T07:10:27.314 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-06T07:10:27.314 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-06T07:10:28.597 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 15 kB 00:00 2024-03-06T07:10:30.279 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-06T07:10:30.301 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-06T07:10:30.976 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:10:31.111 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:34.030 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:34.206 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:10:35.323 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2024-03-06T07:10:37.513 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:10:37.513 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:10:37.619 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-06T07:10:37.623 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-06T07:10:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-03-06T07:10:37.623 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-06T07:10:37.623 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-06T07:10:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:10:37.623 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-06T07:10:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:10:37.623 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:10:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:10:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:10:37.624 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-06T07:10:37.624 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-06T07:10:37.624 INFO:teuthology.orchestra.run.smithi164.stdout:Install 4 Packages 2024-03-06T07:10:37.624 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:10:37.624 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 100 M 2024-03-06T07:10:37.625 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 123 M 2024-03-06T07:10:37.625 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-06T07:10:39.058 INFO:teuthology.orchestra.run.smithi164.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.3 MB/s | 6.0 MB 00:01 2024-03-06T07:10:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:10:39.435 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:10:39.658 INFO:teuthology.orchestra.run.smithi164.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 10 MB/s | 21 MB 00:01 2024-03-06T07:10:44.291 INFO:teuthology.orchestra.run.smithi164.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 5.9 MB/s | 39 MB 00:06 2024-03-06T07:10:44.561 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:10:44.591 INFO:teuthology.orchestra.run.smithi164.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 6.0 MB/s | 33 MB 00:05 2024-03-06T07:10:44.592 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:10:44.592 INFO:teuthology.orchestra.run.smithi164.stdout:Total 14 MB/s | 100 MB 00:06 2024-03-06T07:10:45.482 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-06T07:10:45.859 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-06T07:10:45.859 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-06T07:10:47.211 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-06T07:10:47.211 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-06T07:10:50.172 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-06T07:10:50.824 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:10:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:53.771 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:53.960 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:10:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:10:58.937 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:03.944 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:15.477 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:15.477 INFO:teuthology.orchestra.run.smithi059.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:11:15.477 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:11:15.566 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:16.017 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:16.017 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:16.017 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:16.017 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:16.452 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:16.452 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:11:16.452 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-06T07:11:16.452 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:16.452 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:16.453 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:11:16.453 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:16.453 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:11:16.453 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-06T07:11:16.969 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:17.154 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:17.154 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:17.155 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:11:17.155 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:11:17.155 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:11:17.155 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-06T07:11:17.155 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:11:17.155 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:17.282 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:17.283 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:17.283 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2024-03-06T07:11:17.320 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.06 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 68.el9 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Group : Unspecified 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 8264254 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:11:17.321 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : builder@centos.org 2024-03-06T07:11:17.322 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2024-03-06T07:11:17.322 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:11:17.322 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2024-03-06T07:11:17.322 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2024-03-06T07:11:17.322 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:11:17.322 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:11:17.322 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:11:17.322 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:11:17.322 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2024-03-06T07:11:17.322 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:11:17.322 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:11:17.323 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:11:17.323 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:11:17.323 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:11:17.824 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2024-03-06T07:11:19.171 INFO:teuthology.orchestra.run.smithi059.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:11:19.184 INFO:teuthology.orchestra.run.smithi059.stderr:done 2024-03-06T07:11:19.187 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-03-06T07:11:19.203 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.XjtRyodQnc 2024-03-06T07:11:19.203 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XjtRyodQnc 2024-03-06T07:11:19.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.XjtRyodQnc 2024-03-06T07:11:19.420 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.XjtRyodQnc is 6KB 2024-03-06T07:11:19.469 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.XjtRyodQnc 2024-03-06T07:11:19.484 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:11:19.552 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:11:19.552 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:11:19.553 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:11:19.553 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:11:19.554 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:11:19.712 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2024-03-06T07:11:33.143 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:33.144 INFO:teuthology.orchestra.run.smithi164.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:11:33.144 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:11:33.231 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:33.721 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:33.755 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:33.756 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:33.756 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:34.134 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:34.134 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:11:34.134 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-06T07:11:34.134 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:34.134 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:34.134 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:11:34.135 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:34.135 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:11:34.135 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-06T07:11:34.646 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:34.785 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:34.785 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:34.785 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:11:34.785 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:11:34.786 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:11:34.786 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-06T07:11:34.786 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:11:34.786 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:34.907 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:34.908 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:34.908 DEBUG:teuthology.orchestra.run.smithi164:> sudo rpm -qi grub2-tools 2024-03-06T07:11:34.952 INFO:teuthology.orchestra.run.smithi164.stdout:Name : grub2-tools 2024-03-06T07:11:34.952 INFO:teuthology.orchestra.run.smithi164.stdout:Epoch : 1 2024-03-06T07:11:34.952 INFO:teuthology.orchestra.run.smithi164.stdout:Version : 2.06 2024-03-06T07:11:34.952 INFO:teuthology.orchestra.run.smithi164.stdout:Release : 68.el9 2024-03-06T07:11:34.952 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: x86_64 2024-03-06T07:11:34.952 INFO:teuthology.orchestra.run.smithi164.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:11:34.952 INFO:teuthology.orchestra.run.smithi164.stdout:Group : Unspecified 2024-03-06T07:11:34.952 INFO:teuthology.orchestra.run.smithi164.stdout:Size : 8264254 2024-03-06T07:11:34.952 INFO:teuthology.orchestra.run.smithi164.stdout:License : GPLv3+ 2024-03-06T07:11:34.952 INFO:teuthology.orchestra.run.smithi164.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:Packager : builder@centos.org 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:Vendor : CentOS 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:Summary : Support tools for GRUB. 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:Description : 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout:hardware devices. 2024-03-06T07:11:34.953 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:11:34.954 INFO:teuthology.orchestra.run.smithi164.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:11:34.954 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:11:34.954 INFO:teuthology.task.kernel:Updating grub on smithi164 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:11:34.955 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:11:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2024-03-06T07:11:36.928 INFO:teuthology.orchestra.run.smithi164.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:11:36.960 INFO:teuthology.orchestra.run.smithi164.stderr:done 2024-03-06T07:11:36.963 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-03-06T07:11:36.978 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.ULChfPJUGb 2024-03-06T07:11:36.978 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ULChfPJUGb 2024-03-06T07:11:37.046 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.ULChfPJUGb 2024-03-06T07:11:37.184 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.ULChfPJUGb is 6KB 2024-03-06T07:11:37.233 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.ULChfPJUGb 2024-03-06T07:11:37.247 DEBUG:teuthology.orchestra.run.smithi164:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:11:37.311 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:11:37.311 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:11:37.311 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:11:37.312 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:11:37.313 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:11:37.457 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2024-03-06T07:11:49.728 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-06T07:11:49.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-06T07:11:49.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:07.476 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-06T07:12:07.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-06T07:12:07.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:42.049 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-03-06T07:12:49.738 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:12:51.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-06T07:12:51.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:58.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-06T07:12:58.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:59.211 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-03-06T07:12:59.764 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-06T07:12:59.765 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:12:59.765 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2024-03-06T07:12:59.779 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:12:59.779 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:12:59.779 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:12:59.779 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:13:00.780 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-06T07:13:00.804 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:13:00.873 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2024-03-06T07:13:00.916 DEBUG:teuthology.parallel:result is None 2024-03-06T07:13:09.436 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-03-06T07:13:21.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-06T07:13:21.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:24.508 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-03-06T07:13:39.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-06T07:13:39.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:39.808 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-03-06T07:13:40.236 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-06T07:13:40.236 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:13:40.236 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-03-06T07:13:40.252 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:13:40.253 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:13:40.253 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:13:40.253 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:13:41.254 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-06T07:13:41.254 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:13:41.317 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2024-03-06T07:13:41.352 DEBUG:teuthology.parallel:result is None 2024-03-06T07:13:41.353 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:13:41.361 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:13:41.361 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:13:41.364 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:13:41.381 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:13:41.461 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:13:41.470 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:13:41.471 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:13:41.475 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:13:41.506 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:13:41.515 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:13:41.515 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:13:41.543 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:13:41.579 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:41.594 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:41.599 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:41.614 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:41.615 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:13:41.626 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:13:41.627 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:13:41.639 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:13:41.671 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:13:41.681 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:13:41.681 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:13:41.709 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:13:41.727 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:41.807 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:41.859 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:41.941 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:41.984 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:13:41.984 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:13:42.051 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:42.095 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:42.155 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:42.240 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:42.285 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:13:42.285 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:13:42.354 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2024-03-06T07:13:42.356 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-03-06T07:13:42.406 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:13:42.440 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:13:42.918 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:13:42.926 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:13:42.926 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:13:42.939 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:13:42.948 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2024-03-06T07:13:43.021 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2024-03-06T07:13:43.023 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2024-03-06T07:13:43.087 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2024-03-06T07:13:43.089 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:13:43.089 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2024-03-06T07:13:43.118 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2024-03-06T07:13:43.118 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2024-03-06T07:13:43.139 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2024-03-06T07:13:43.139 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:13:43.139 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:13:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:13:43.187 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:13:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:13:43.216 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:13:43.216 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:13:43.262 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:13:43.300 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:13:43.311 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:13:43.329 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:13:43.331 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:18:01.502 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2024-03-06T07:18:01.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-06T07:18:01.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:18:01.599 DEBUG:teuthology.orchestra.run.smithi059:> true 2024-03-06T07:18:01.679 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2024-03-06T07:18:01.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-06T07:18:01.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:18:01.798 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-03-06T07:18:01.891 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-06T07:18:01.892 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:18:01.903 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:18:01.903 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:18:01.904 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:18:01.906 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:18:01.907 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:18:01.935 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:18:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:18:01.964 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:18:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:18:01.981 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2024-03-06T07:18:01.995 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2024-03-06T07:18:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:18:02.013 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2024-03-06T07:18:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:18:02.028 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2024-03-06T07:18:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:18:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:18:02.085 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:18:02.088 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:18:02.088 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-03-06T07:18:02.088 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:02.088 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:02.088 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:02.088 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:02.116 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:18:02.119 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:18:02.119 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-03-06T07:18:02.119 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:02.119 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:02.119 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:02.119 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:18:02.120 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:18:02.129 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:18:02.129 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:18:02.129 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:18:02.129 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:18:02.136 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:18:02.136 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:18:02.137 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:18:02.137 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:18:02.140 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:18:02.140 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:18:02.294 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:18:02.295 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:18:02.430 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:18:02.430 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:18:02.766 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:18:02.766 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:18:02.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:18:02.768 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:18:02.769 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:18:02.769 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:18:02.797 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:18:02.797 DEBUG:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:18:02.798 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:18:02.799 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:18:02.865 DEBUG:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:18:02.868 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:18:02.945 DEBUG:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:18:02.950 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:18:03.012 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2024-03-06T07:18:03.013 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum clean all 2024-03-06T07:18:03.018 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2024-03-06T07:18:03.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2024-03-06T07:18:03.256 INFO:teuthology.orchestra.run.smithi164.stdout:74 files removed 2024-03-06T07:18:03.276 INFO:teuthology.orchestra.run.smithi059.stdout:74 files removed 2024-03-06T07:18:03.287 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:18:03.303 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:18:04.390 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 106 kB/s | 83 kB 00:00 2024-03-06T07:18:04.496 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 95 kB/s | 83 kB 00:00 2024-03-06T07:18:05.213 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 15 kB/s | 10 kB 00:00 2024-03-06T07:18:05.266 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-06T07:18:06.116 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-06T07:18:06.222 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 2.0 kB/s | 1.9 kB 00:00 2024-03-06T07:18:06.760 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-06T07:18:06.891 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-06T07:18:08.218 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-06T07:18:08.658 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-06T07:18:09.126 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-06T07:18:10.998 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.0 MB 00:02 2024-03-06T07:18:13.418 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 19 MB 00:03 2024-03-06T07:18:24.665 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 19 MB 00:12 2024-03-06T07:18:24.968 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 781 kB/s | 6.1 MB 00:08 2024-03-06T07:18:26.762 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 15 kB 00:00 2024-03-06T07:18:27.904 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-06T07:18:28.783 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-06T07:18:30.300 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2024-03-06T07:18:32.987 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-06T07:18:35.514 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:35.515 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:35.527 INFO:teuthology.orchestra.run.smithi164.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:18:35.528 INFO:teuthology.orchestra.run.smithi164.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:18:35.528 INFO:teuthology.orchestra.run.smithi164.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:18:35.529 INFO:teuthology.orchestra.run.smithi164.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:35.531 INFO:teuthology.orchestra.run.smithi164.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:35.534 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:18:35.535 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:18:35.620 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-06T07:18:35.642 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-06T07:18:35.642 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-03-06T07:18:35.642 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-06T07:18:35.642 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:18:35.643 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:18:35.644 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2024-03-06T07:18:35.645 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:18:35.646 INFO:teuthology.orchestra.run.smithi164.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:18:35.647 INFO:teuthology.orchestra.run.smithi164.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:18:35.648 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:18:35.649 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:18:35.650 INFO:teuthology.orchestra.run.smithi164.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:18:35.651 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:18:35.652 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:18:35.653 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:18:35.654 INFO:teuthology.orchestra.run.smithi164.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout:Installing weak dependencies: 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:18:35.655 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:18:35.656 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-06T07:18:35.656 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2024-03-06T07:18:35.656 INFO:teuthology.orchestra.run.smithi164.stdout:Install 150 Packages 2024-03-06T07:18:35.656 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 26 Packages 2024-03-06T07:18:35.656 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:18:35.656 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 224 M 2024-03-06T07:18:35.657 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-06T07:18:36.446 INFO:teuthology.orchestra.run.smithi164.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-03-06T07:18:37.355 INFO:teuthology.orchestra.run.smithi164.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 978 kB/s | 888 kB 00:00 2024-03-06T07:18:37.522 INFO:teuthology.orchestra.run.smithi164.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 871 kB/s | 145 kB 00:00 2024-03-06T07:18:38.966 INFO:teuthology.orchestra.run.smithi164.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 1.6 MB/s | 2.3 MB 00:01 2024-03-06T07:18:39.589 INFO:teuthology.orchestra.run.smithi164.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 2.4 MB/s | 1.5 MB 00:00 2024-03-06T07:18:41.006 INFO:teuthology.orchestra.run.smithi164.stdout:(6/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-03-06T07:18:44.297 INFO:teuthology.orchestra.run.smithi164.stdout:(7/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 670 kB/s | 5.3 MB 00:08 2024-03-06T07:18:46.339 INFO:teuthology.orchestra.run.smithi164.stdout:(8/176): ceph-common-19.0.0-1739.g1bc250c7.el9. 2.0 MB/s | 20 MB 00:10 2024-03-06T07:18:46.439 INFO:teuthology.orchestra.run.smithi164.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 252 kB/s | 25 kB 00:00 2024-03-06T07:18:50.648 INFO:teuthology.orchestra.run.smithi164.stdout:(10/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 1.7 MB/s | 17 MB 00:09 2024-03-06T07:18:50.964 INFO:teuthology.orchestra.run.smithi164.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 697 kB/s | 220 kB 00:00 2024-03-06T07:18:51.056 INFO:teuthology.orchestra.run.smithi164.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 296 kB/s | 27 kB 00:00 2024-03-06T07:18:51.165 INFO:teuthology.orchestra.run.smithi164.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 288 kB/s | 31 kB 00:00 2024-03-06T07:18:51.274 INFO:teuthology.orchestra.run.smithi164.stdout:(14/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 1.4 MB/s | 9.8 MB 00:06 2024-03-06T07:18:51.483 INFO:teuthology.orchestra.run.smithi164.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 785 kB/s | 163 kB 00:00 2024-03-06T07:18:51.682 INFO:teuthology.orchestra.run.smithi164.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 633 kB/s | 127 kB 00:00 2024-03-06T07:18:52.174 INFO:teuthology.orchestra.run.smithi164.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 1.0 MB/s | 501 kB 00:00 2024-03-06T07:18:52.219 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 965 kB/s | 21 MB 00:21 2024-03-06T07:18:52.267 INFO:teuthology.orchestra.run.smithi164.stdout:(18/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 678 kB/s | 747 kB 00:01 2024-03-06T07:18:52.375 INFO:teuthology.orchestra.run.smithi164.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 417 kB/s | 45 kB 00:00 2024-03-06T07:18:52.617 INFO:teuthology.orchestra.run.smithi164.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 544 kB/s | 131 kB 00:00 2024-03-06T07:18:52.859 INFO:teuthology.orchestra.run.smithi164.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 669 kB/s | 162 kB 00:00 2024-03-06T07:18:53.317 INFO:teuthology.orchestra.run.smithi164.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 700 kB/s | 321 kB 00:00 2024-03-06T07:18:53.695 INFO:teuthology.orchestra.run.smithi164.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 794 kB/s | 298 kB 00:00 2024-03-06T07:18:53.859 INFO:teuthology.orchestra.run.smithi164.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 596 kB/s | 99 kB 00:00 2024-03-06T07:18:54.034 INFO:teuthology.orchestra.run.smithi164.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 487 kB/s | 85 kB 00:00 2024-03-06T07:18:56.735 INFO:teuthology.orchestra.run.smithi164.stdout:(26/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 1.1 MB/s | 5.1 MB 00:04 2024-03-06T07:18:56.935 INFO:teuthology.orchestra.run.smithi164.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 853 kB/s | 171 kB 00:00 2024-03-06T07:18:57.043 INFO:teuthology.orchestra.run.smithi164.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 290 kB/s | 31 kB 00:00 2024-03-06T07:18:57.260 INFO:teuthology.orchestra.run.smithi164.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 637 kB/s | 138 kB 00:00 2024-03-06T07:18:57.391 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-06T07:18:58.827 INFO:teuthology.orchestra.run.smithi164.stdout:(30/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.1 MB/s | 1.8 MB 00:01 2024-03-06T07:18:58.886 INFO:teuthology.orchestra.run.smithi164.stdout:(31/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 652 kB/s | 3.1 MB 00:04 2024-03-06T07:18:59.286 INFO:teuthology.orchestra.run.smithi164.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 621 kB/s | 248 kB 00:00 2024-03-06T07:18:59.386 INFO:teuthology.orchestra.run.smithi164.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 494 kB/s | 49 kB 00:00 2024-03-06T07:18:59.486 INFO:teuthology.orchestra.run.smithi164.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 153 kB/s | 15 kB 00:00 2024-03-06T07:18:59.870 INFO:teuthology.orchestra.run.smithi164.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 688 kB/s | 264 kB 00:00 2024-03-06T07:19:00.009 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:19:00.010 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:19:00.022 INFO:teuthology.orchestra.run.smithi059.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:19:00.023 INFO:teuthology.orchestra.run.smithi059.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:19:00.024 INFO:teuthology.orchestra.run.smithi059.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:19:00.025 INFO:teuthology.orchestra.run.smithi059.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:19:00.026 INFO:teuthology.orchestra.run.smithi059.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:19:00.029 INFO:teuthology.orchestra.run.smithi059.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:19:00.030 INFO:teuthology.orchestra.run.smithi059.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:19:00.117 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2024-03-06T07:19:00.146 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-03-06T07:19:00.146 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2024-03-06T07:19:00.146 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-03-06T07:19:00.146 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2024-03-06T07:19:00.146 INFO:teuthology.orchestra.run.smithi059.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:19:00.146 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:19:00.146 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:19:00.146 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:19:00.146 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:19:00.146 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:19:00.147 INFO:teuthology.orchestra.run.smithi059.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:19:00.148 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:19:00.149 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:19:00.150 INFO:teuthology.orchestra.run.smithi059.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:19:00.151 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:19:00.152 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:19:00.153 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:19:00.154 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:19:00.155 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:19:00.156 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:19:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2024-03-06T07:19:00.159 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:19:00.160 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:19:00.160 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2024-03-06T07:19:00.160 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2024-03-06T07:19:00.160 INFO:teuthology.orchestra.run.smithi059.stdout:Install 150 Packages 2024-03-06T07:19:00.160 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 26 Packages 2024-03-06T07:19:00.160 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:19:00.160 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 224 M 2024-03-06T07:19:00.160 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2024-03-06T07:19:00.762 INFO:teuthology.orchestra.run.smithi164.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 839 kB/s | 748 kB 00:00 2024-03-06T07:19:00.862 INFO:teuthology.orchestra.run.smithi164.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 182 kB/s | 18 kB 00:00 2024-03-06T07:19:00.884 INFO:teuthology.orchestra.run.smithi059.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-03-06T07:19:01.013 INFO:teuthology.orchestra.run.smithi164.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-03-06T07:19:01.097 INFO:teuthology.orchestra.run.smithi164.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 11 MB/s | 920 kB 00:00 2024-03-06T07:19:01.138 INFO:teuthology.orchestra.run.smithi164.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-06T07:19:01.172 INFO:teuthology.orchestra.run.smithi164.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 617 kB/s | 20 kB 00:00 2024-03-06T07:19:01.222 INFO:teuthology.orchestra.run.smithi164.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-06T07:19:01.256 INFO:teuthology.orchestra.run.smithi164.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 958 kB/s | 32 kB 00:00 2024-03-06T07:19:01.365 INFO:teuthology.orchestra.run.smithi164.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-06T07:19:01.398 INFO:teuthology.orchestra.run.smithi164.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 515 kB/s | 17 kB 00:00 2024-03-06T07:19:01.432 INFO:teuthology.orchestra.run.smithi164.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 302 kB/s | 10 kB 00:00 2024-03-06T07:19:01.474 INFO:teuthology.orchestra.run.smithi164.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-06T07:19:01.508 INFO:teuthology.orchestra.run.smithi164.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-06T07:19:01.550 INFO:teuthology.orchestra.run.smithi164.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 16 MB/s | 661 kB 00:00 2024-03-06T07:19:01.600 INFO:teuthology.orchestra.run.smithi164.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-06T07:19:01.633 INFO:teuthology.orchestra.run.smithi164.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 734 kB/s | 24 kB 00:00 2024-03-06T07:19:01.675 INFO:teuthology.orchestra.run.smithi164.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-06T07:19:01.717 INFO:teuthology.orchestra.run.smithi164.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-06T07:19:01.759 INFO:teuthology.orchestra.run.smithi164.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-06T07:19:01.801 INFO:teuthology.orchestra.run.smithi164.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 823 kB/s | 34 kB 00:00 2024-03-06T07:19:01.968 INFO:teuthology.orchestra.run.smithi164.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-03-06T07:19:02.094 INFO:teuthology.orchestra.run.smithi164.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 24 MB/s | 3.0 MB 00:00 2024-03-06T07:19:02.169 INFO:teuthology.orchestra.run.smithi164.stdout:(58/176): postgresql-13.11-1.el9.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2024-03-06T07:19:02.211 INFO:teuthology.orchestra.run.smithi164.stdout:(59/176): postgresql-contrib-13.11-1.el9.x86_64 19 MB/s | 811 kB 00:00 2024-03-06T07:19:02.226 INFO:teuthology.orchestra.run.smithi059.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 662 kB/s | 888 kB 00:01 2024-03-06T07:19:02.244 INFO:teuthology.orchestra.run.smithi164.stdout:(60/176): postgresql-private-libs-13.11-1.el9.x 3.9 MB/s | 134 kB 00:00 2024-03-06T07:19:02.395 INFO:teuthology.orchestra.run.smithi164.stdout:(61/176): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-06T07:19:02.427 INFO:teuthology.orchestra.run.smithi059.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 726 kB/s | 145 kB 00:00 2024-03-06T07:19:02.453 INFO:teuthology.orchestra.run.smithi164.stdout:(62/176): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-06T07:19:02.487 INFO:teuthology.orchestra.run.smithi164.stdout:(63/176): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-06T07:19:02.646 INFO:teuthology.orchestra.run.smithi164.stdout:(64/176): python3-babel-2.9.1-2.el9.noarch.rpm 36 MB/s | 5.8 MB 00:00 2024-03-06T07:19:02.688 INFO:teuthology.orchestra.run.smithi164.stdout:(65/176): python3-jinja2-2.11.3-5.el9.noarch.rp 5.4 MB/s | 227 kB 00:00 2024-03-06T07:19:02.713 INFO:teuthology.orchestra.run.smithi164.stdout:(66/176): python3-jsonpatch-1.21-16.el9.noarch. 971 kB/s | 24 kB 00:00 2024-03-06T07:19:02.747 INFO:teuthology.orchestra.run.smithi164.stdout:(67/176): python3-jsonpointer-2.0-4.el9.noarch. 540 kB/s | 18 kB 00:00 2024-03-06T07:19:02.781 INFO:teuthology.orchestra.run.smithi164.stdout:(68/176): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-06T07:19:02.814 INFO:teuthology.orchestra.run.smithi164.stdout:(69/176): python3-markupsafe-1.1.1-12.el9.x86_6 947 kB/s | 32 kB 00:00 2024-03-06T07:19:02.948 INFO:teuthology.orchestra.run.smithi164.stdout:(70/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-06T07:19:02.998 INFO:teuthology.orchestra.run.smithi164.stdout:(71/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.3 MB/s | 217 kB 00:00 2024-03-06T07:19:03.040 INFO:teuthology.orchestra.run.smithi164.stdout:(72/176): python3-oauthlib-3.1.1-5.el9.noarch.r 4.1 MB/s | 174 kB 00:00 2024-03-06T07:19:03.073 INFO:teuthology.orchestra.run.smithi164.stdout:(73/176): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:03.107 INFO:teuthology.orchestra.run.smithi164.stdout:(74/176): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:03.157 INFO:teuthology.orchestra.run.smithi164.stdout:(75/176): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-06T07:19:03.199 INFO:teuthology.orchestra.run.smithi164.stdout:(76/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 132 kB 00:00 2024-03-06T07:19:03.233 INFO:teuthology.orchestra.run.smithi164.stdout:(77/176): python3-pyasn1-modules-0.4.8-6.el9.no 6.1 MB/s | 211 kB 00:00 2024-03-06T07:19:03.267 INFO:teuthology.orchestra.run.smithi164.stdout:(78/176): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-06T07:19:03.300 INFO:teuthology.orchestra.run.smithi164.stdout:(79/176): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-06T07:19:03.643 INFO:teuthology.orchestra.run.smithi059.stdout:(4/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.7 MB/s | 5.3 MB 00:03 2024-03-06T07:19:03.742 INFO:teuthology.orchestra.run.smithi164.stdout:(80/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-06T07:19:03.776 INFO:teuthology.orchestra.run.smithi164.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-06T07:19:03.810 INFO:teuthology.orchestra.run.smithi164.stdout:(82/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:03.851 INFO:teuthology.orchestra.run.smithi164.stdout:(83/176): qatlib-23.11.0-1.el9.x86_64.rpm 5.1 MB/s | 215 kB 00:00 2024-03-06T07:19:03.885 INFO:teuthology.orchestra.run.smithi164.stdout:(84/176): qatlib-service-23.11.0-1.el9.x86_64.r 1.0 MB/s | 34 kB 00:00 2024-03-06T07:19:03.919 INFO:teuthology.orchestra.run.smithi164.stdout:(85/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-06T07:19:03.960 INFO:teuthology.orchestra.run.smithi164.stdout:(86/176): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-06T07:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout:(87/176): uuid-1.6.2-55.el9.x86_64.rpm 1.6 MB/s | 56 kB 00:00 2024-03-06T07:19:04.028 INFO:teuthology.orchestra.run.smithi164.stdout:(88/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.6 MB/s | 87 kB 00:00 2024-03-06T07:19:04.062 INFO:teuthology.orchestra.run.smithi164.stdout:(89/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-06T07:19:04.095 INFO:teuthology.orchestra.run.smithi164.stdout:(90/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-06T07:19:04.129 INFO:teuthology.orchestra.run.smithi164.stdout:(91/176): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-06T07:19:04.179 INFO:teuthology.orchestra.run.smithi164.stdout:(92/176): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-03-06T07:19:04.238 INFO:teuthology.orchestra.run.smithi164.stdout:(93/176): libgfortran-11.4.1-3.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:19:04.280 INFO:teuthology.orchestra.run.smithi164.stdout:(94/176): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 194 kB 00:00 2024-03-06T07:19:04.330 INFO:teuthology.orchestra.run.smithi164.stdout:(95/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-06T07:19:04.361 INFO:teuthology.orchestra.run.smithi059.stdout:(5/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 1.2 MB/s | 2.3 MB 00:01 2024-03-06T07:19:04.372 INFO:teuthology.orchestra.run.smithi164.stdout:(96/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-06T07:19:04.439 INFO:teuthology.orchestra.run.smithi164.stdout:(97/176): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-06T07:19:04.473 INFO:teuthology.orchestra.run.smithi164.stdout:(98/176): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-06T07:19:04.506 INFO:teuthology.orchestra.run.smithi164.stdout:(99/176): python3-pycparser-2.20-6.el9.noarch.r 3.6 MB/s | 124 kB 00:00 2024-03-06T07:19:04.540 INFO:teuthology.orchestra.run.smithi164.stdout:(100/176): python3-requests-2.25.1-8.el9.noarch 3.3 MB/s | 113 kB 00:00 2024-03-06T07:19:04.582 INFO:teuthology.orchestra.run.smithi164.stdout:(101/176): python3-urllib3-1.26.5-5.el9.noarch. 4.4 MB/s | 187 kB 00:00 2024-03-06T07:19:05.174 INFO:teuthology.orchestra.run.smithi164.stdout:(102/176): python3-devel-3.9.17-2.el9.x86_64.rp 415 kB/s | 245 kB 00:00 2024-03-06T07:19:05.219 INFO:teuthology.orchestra.run.smithi059.stdout:(6/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 971 kB/s | 1.5 MB 00:01 2024-03-06T07:19:05.266 INFO:teuthology.orchestra.run.smithi164.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2024-03-06T07:19:05.408 INFO:teuthology.orchestra.run.smithi164.stdout:(104/176): protobuf-compiler-3.14.0-13.el9.x86_ 6.0 MB/s | 863 kB 00:00 2024-03-06T07:19:05.992 INFO:teuthology.orchestra.run.smithi164.stdout:(105/176): abseil-cpp-20211102.0-3.el9.x86_64.r 939 kB/s | 548 kB 00:00 2024-03-06T07:19:06.117 INFO:teuthology.orchestra.run.smithi164.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 2.4 MB/s | 309 kB 00:00 2024-03-06T07:19:06.225 INFO:teuthology.orchestra.run.smithi164.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-03-06T07:19:06.609 INFO:teuthology.orchestra.run.smithi164.stdout:(108/176): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-06T07:19:06.710 INFO:teuthology.orchestra.run.smithi164.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-06T07:19:06.801 INFO:teuthology.orchestra.run.smithi164.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-03-06T07:19:06.960 INFO:teuthology.orchestra.run.smithi164.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 426 kB/s | 67 kB 00:00 2024-03-06T07:19:07.060 INFO:teuthology.orchestra.run.smithi164.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-06T07:19:07.177 INFO:teuthology.orchestra.run.smithi164.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-06T07:19:07.278 INFO:teuthology.orchestra.run.smithi164.stdout:(114/176): python3-asyncssh-2.13.2-4.el9.noarch 5.4 MB/s | 548 kB 00:00 2024-03-06T07:19:07.380 INFO:teuthology.orchestra.run.smithi164.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 427 kB/s | 43 kB 00:00 2024-03-06T07:19:07.470 INFO:teuthology.orchestra.run.smithi164.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 358 kB/s | 32 kB 00:00 2024-03-06T07:19:07.562 INFO:teuthology.orchestra.run.smithi164.stdout:(117/176): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-03-06T07:19:07.654 INFO:teuthology.orchestra.run.smithi164.stdout:(118/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-06T07:19:07.771 INFO:teuthology.orchestra.run.smithi164.stdout:(119/176): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-06T07:19:07.879 INFO:teuthology.orchestra.run.smithi164.stdout:(120/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-06T07:19:07.988 INFO:teuthology.orchestra.run.smithi164.stdout:(121/176): python3-colorama-0.4.4-12.el9.noarch 325 kB/s | 35 kB 00:00 2024-03-06T07:19:08.097 INFO:teuthology.orchestra.run.smithi164.stdout:(122/176): python3-google-auth-2.28.0-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-06T07:19:08.264 INFO:teuthology.orchestra.run.smithi164.stdout:(123/176): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-06T07:19:08.364 INFO:teuthology.orchestra.run.smithi164.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-06T07:19:08.464 INFO:teuthology.orchestra.run.smithi164.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-03-06T07:19:08.573 INFO:teuthology.orchestra.run.smithi164.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-03-06T07:19:08.665 INFO:teuthology.orchestra.run.smithi164.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 253 kB/s | 23 kB 00:00 2024-03-06T07:19:08.719 INFO:teuthology.orchestra.run.smithi059.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 1.1 MB/s | 4.7 MB 00:04 2024-03-06T07:19:08.757 INFO:teuthology.orchestra.run.smithi164.stdout:(128/176): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-03-06T07:19:08.866 INFO:teuthology.orchestra.run.smithi164.stdout:(129/176): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2024-03-06T07:19:09.005 INFO:teuthology.orchestra.run.smithi164.stdout:(130/176): python3-kubernetes-26.1.0-2.el9.noar 7.3 MB/s | 1.0 MB 00:00 2024-03-06T07:19:09.100 INFO:teuthology.orchestra.run.smithi164.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 493 kB/s | 46 kB 00:00 2024-03-06T07:19:09.192 INFO:teuthology.orchestra.run.smithi164.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 860 kB/s | 79 kB 00:00 2024-03-06T07:19:09.284 INFO:teuthology.orchestra.run.smithi164.stdout:(133/176): python3-natsort-7.1.1-5.el9.noarch.r 629 kB/s | 58 kB 00:00 2024-03-06T07:19:09.392 INFO:teuthology.orchestra.run.smithi164.stdout:(134/176): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-06T07:19:09.484 INFO:teuthology.orchestra.run.smithi164.stdout:(135/176): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-03-06T07:19:09.576 INFO:teuthology.orchestra.run.smithi164.stdout:(136/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 981 kB/s | 90 kB 00:00 2024-03-06T07:19:09.668 INFO:teuthology.orchestra.run.smithi164.stdout:(137/176): python3-pyperclip-1.8.2-2.el9.noarch 308 kB/s | 28 kB 00:00 2024-03-06T07:19:09.760 INFO:teuthology.orchestra.run.smithi164.stdout:(138/176): python3-repoze-lru-0.7-16.el9.noarch 335 kB/s | 31 kB 00:00 2024-03-06T07:19:09.852 INFO:teuthology.orchestra.run.smithi164.stdout:(139/176): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-03-06T07:19:09.944 INFO:teuthology.orchestra.run.smithi164.stdout:(140/176): python3-rsa-4.9-2.el9.noarch.rpm 645 kB/s | 59 kB 00:00 2024-03-06T07:19:10.028 INFO:teuthology.orchestra.run.smithi164.stdout:(141/176): python3-tempora-5.0.0-2.el9.noarch.r 430 kB/s | 36 kB 00:00 2024-03-06T07:19:10.111 INFO:teuthology.orchestra.run.smithi164.stdout:(142/176): python3-typing-extensions-4.4.0-2.el 611 kB/s | 51 kB 00:00 2024-03-06T07:19:10.212 INFO:teuthology.orchestra.run.smithi164.stdout:(143/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-06T07:19:10.320 INFO:teuthology.orchestra.run.smithi164.stdout:(144/176): python3-websocket-client-1.2.3-2.el9 828 kB/s | 90 kB 00:00 2024-03-06T07:19:10.437 INFO:teuthology.orchestra.run.smithi164.stdout:(145/176): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-06T07:19:10.529 INFO:teuthology.orchestra.run.smithi164.stdout:(146/176): python3-zc-lockfile-2.0-10.el9.noarc 218 kB/s | 20 kB 00:00 2024-03-06T07:19:10.630 INFO:teuthology.orchestra.run.smithi164.stdout:(147/176): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-06T07:19:10.763 INFO:teuthology.orchestra.run.smithi164.stdout:(148/176): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-06T07:19:11.189 INFO:teuthology.orchestra.run.smithi164.stdout:(149/176): ceph-mgr-diskprediction-local-19.0.0 614 kB/s | 7.4 MB 00:12 2024-03-06T07:19:13.855 INFO:teuthology.orchestra.run.smithi164.stdout:(150/176): librados2-19.0.0-1739.g1bc250c7.el9. 1.1 MB/s | 3.4 MB 00:03 2024-03-06T07:19:13.889 INFO:teuthology.orchestra.run.smithi164.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-06T07:19:13.998 INFO:teuthology.orchestra.run.smithi164.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 21 MB/s | 2.3 MB 00:00 2024-03-06T07:19:14.231 INFO:teuthology.orchestra.run.smithi164.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 7.7 MB/s | 1.8 MB 00:00 2024-03-06T07:19:14.265 INFO:teuthology.orchestra.run.smithi164.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:14.306 INFO:teuthology.orchestra.run.smithi164.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-06T07:19:14.340 INFO:teuthology.orchestra.run.smithi164.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:14.374 INFO:teuthology.orchestra.run.smithi164.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 35 kB 00:00 2024-03-06T07:19:14.408 INFO:teuthology.orchestra.run.smithi164.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 5.6 MB/s | 191 kB 00:00 2024-03-06T07:19:14.458 INFO:teuthology.orchestra.run.smithi164.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 254 kB 00:00 2024-03-06T07:19:14.508 INFO:teuthology.orchestra.run.smithi164.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 16 MB/s | 802 kB 00:00 2024-03-06T07:19:14.550 INFO:teuthology.orchestra.run.smithi164.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 10 MB/s | 433 kB 00:00 2024-03-06T07:19:14.584 INFO:teuthology.orchestra.run.smithi164.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 663 kB/s | 22 kB 00:00 2024-03-06T07:19:14.620 INFO:teuthology.orchestra.run.smithi164.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.9 MB/s | 107 kB 00:00 2024-03-06T07:19:14.662 INFO:teuthology.orchestra.run.smithi164.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 154 kB 00:00 2024-03-06T07:19:14.704 INFO:teuthology.orchestra.run.smithi164.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 135 kB 00:00 2024-03-06T07:19:14.746 INFO:teuthology.orchestra.run.smithi164.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 63 kB 00:00 2024-03-06T07:19:14.780 INFO:teuthology.orchestra.run.smithi164.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 823 kB/s | 28 kB 00:00 2024-03-06T07:19:14.847 INFO:teuthology.orchestra.run.smithi164.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 17 MB/s | 1.2 MB 00:00 2024-03-06T07:19:14.955 INFO:teuthology.orchestra.run.smithi164.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 20 MB/s | 2.2 MB 00:00 2024-03-06T07:19:15.072 INFO:teuthology.orchestra.run.smithi164.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 19 MB/s | 2.2 MB 00:00 2024-03-06T07:19:15.114 INFO:teuthology.orchestra.run.smithi164.stdout:(171/176): util-linux-core-2.37.4-18.el9.x86_64 10 MB/s | 438 kB 00:00 2024-03-06T07:19:15.140 INFO:teuthology.orchestra.run.smithi164.stdout:(172/176): util-linux-user-2.37.4-18.el9.x86_64 1.2 MB/s | 30 kB 00:00 2024-03-06T07:19:15.182 INFO:teuthology.orchestra.run.smithi164.stdout:(173/176): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-06T07:19:15.248 INFO:teuthology.orchestra.run.smithi164.stdout:(174/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-03-06T07:19:15.490 INFO:teuthology.orchestra.run.smithi164.stdout:(175/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 746 kB/s | 3.1 MB 00:04 2024-03-06T07:19:18.743 INFO:teuthology.orchestra.run.smithi059.stdout:(8/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x86 1.2 MB/s | 17 MB 00:13 2024-03-06T07:19:18.851 INFO:teuthology.orchestra.run.smithi059.stdout:(9/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el9 988 kB/s | 9.8 MB 00:10 2024-03-06T07:19:18.877 INFO:teuthology.orchestra.run.smithi059.stdout:(10/176): ceph-selinux-19.0.0-1739.g1bc250c7.el 189 kB/s | 25 kB 00:00 2024-03-06T07:19:19.177 INFO:teuthology.orchestra.run.smithi059.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 735 kB/s | 220 kB 00:00 2024-03-06T07:19:19.294 INFO:teuthology.orchestra.run.smithi059.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 231 kB/s | 27 kB 00:00 2024-03-06T07:19:19.419 INFO:teuthology.orchestra.run.smithi059.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 250 kB/s | 31 kB 00:00 2024-03-06T07:19:20.877 INFO:teuthology.orchestra.run.smithi059.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 512 kB/s | 747 kB 00:01 2024-03-06T07:19:21.253 INFO:teuthology.orchestra.run.smithi059.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 435 kB/s | 163 kB 00:00 2024-03-06T07:19:21.828 INFO:teuthology.orchestra.run.smithi059.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 220 kB/s | 127 kB 00:00 2024-03-06T07:19:23.286 INFO:teuthology.orchestra.run.smithi059.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 344 kB/s | 501 kB 00:01 2024-03-06T07:19:27.578 INFO:teuthology.orchestra.run.smithi059.stdout:(18/176): ceph-common-19.0.0-1739.g1bc250c7.el9 770 kB/s | 20 MB 00:27 2024-03-06T07:19:27.761 INFO:teuthology.orchestra.run.smithi059.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 246 kB/s | 45 kB 00:00 2024-03-06T07:19:28.078 INFO:teuthology.orchestra.run.smithi059.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 415 kB/s | 131 kB 00:00 2024-03-06T07:19:28.379 INFO:teuthology.orchestra.run.smithi059.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 539 kB/s | 162 kB 00:00 2024-03-06T07:19:28.912 INFO:teuthology.orchestra.run.smithi059.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 602 kB/s | 321 kB 00:00 2024-03-06T07:19:29.396 INFO:teuthology.orchestra.run.smithi059.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 616 kB/s | 298 kB 00:00 2024-03-06T07:19:29.596 INFO:teuthology.orchestra.run.smithi059.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 496 kB/s | 99 kB 00:00 2024-03-06T07:19:29.813 INFO:teuthology.orchestra.run.smithi059.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 393 kB/s | 85 kB 00:00 2024-03-06T07:19:33.480 INFO:teuthology.orchestra.run.smithi059.stdout:(26/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 863 kB/s | 3.1 MB 00:03 2024-03-06T07:19:33.780 INFO:teuthology.orchestra.run.smithi059.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 569 kB/s | 171 kB 00:00 2024-03-06T07:19:33.913 INFO:teuthology.orchestra.run.smithi059.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 236 kB/s | 31 kB 00:00 2024-03-06T07:19:34.064 INFO:teuthology.orchestra.run.smithi059.stdout:(29/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 486 kB/s | 5.1 MB 00:10 2024-03-06T07:19:34.131 INFO:teuthology.orchestra.run.smithi059.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 636 kB/s | 138 kB 00:00 2024-03-06T07:19:37.014 INFO:teuthology.orchestra.run.smithi059.stdout:(31/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 2.6 MB/s | 47 MB 00:18 2024-03-06T07:19:37.056 INFO:teuthology.orchestra.run.smithi059.stdout:(32/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 609 kB/s | 1.8 MB 00:02 2024-03-06T07:19:37.156 INFO:teuthology.orchestra.run.smithi059.stdout:(33/176): ceph-mgr-modules-core-19.0.0-1739.g1b 1.7 MB/s | 248 kB 00:00 2024-03-06T07:19:37.240 INFO:teuthology.orchestra.run.smithi059.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 184 kB/s | 15 kB 00:00 2024-03-06T07:19:37.257 INFO:teuthology.orchestra.run.smithi059.stdout:(35/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 246 kB/s | 49 kB 00:00 2024-03-06T07:19:37.332 INFO:teuthology.orchestra.run.smithi059.stdout:(36/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 2.8 MB/s | 264 kB 00:00 2024-03-06T07:19:37.416 INFO:teuthology.orchestra.run.smithi059.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 219 kB/s | 18 kB 00:00 2024-03-06T07:19:37.599 INFO:teuthology.orchestra.run.smithi059.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 662 kB 00:00 2024-03-06T07:19:37.658 INFO:teuthology.orchestra.run.smithi059.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-06T07:19:37.683 INFO:teuthology.orchestra.run.smithi059.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 4.2 MB/s | 106 kB 00:00 2024-03-06T07:19:37.717 INFO:teuthology.orchestra.run.smithi059.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 611 kB/s | 20 kB 00:00 2024-03-06T07:19:37.759 INFO:teuthology.orchestra.run.smithi059.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-06T07:19:37.792 INFO:teuthology.orchestra.run.smithi059.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 952 kB/s | 32 kB 00:00 2024-03-06T07:19:37.901 INFO:teuthology.orchestra.run.smithi059.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-06T07:19:37.934 INFO:teuthology.orchestra.run.smithi059.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 516 kB/s | 17 kB 00:00 2024-03-06T07:19:37.968 INFO:teuthology.orchestra.run.smithi059.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 302 kB/s | 10 kB 00:00 2024-03-06T07:19:38.002 INFO:teuthology.orchestra.run.smithi059.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-06T07:19:38.035 INFO:teuthology.orchestra.run.smithi059.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-06T07:19:38.086 INFO:teuthology.orchestra.run.smithi059.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-06T07:19:38.127 INFO:teuthology.orchestra.run.smithi059.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-06T07:19:38.161 INFO:teuthology.orchestra.run.smithi059.stdout:(51/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 828 kB/s | 748 kB 00:00 2024-03-06T07:19:38.178 INFO:teuthology.orchestra.run.smithi059.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 486 kB/s | 24 kB 00:00 2024-03-06T07:19:38.195 INFO:teuthology.orchestra.run.smithi059.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-06T07:19:38.220 INFO:teuthology.orchestra.run.smithi059.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-06T07:19:38.246 INFO:teuthology.orchestra.run.smithi059.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-03-06T07:19:38.262 INFO:teuthology.orchestra.run.smithi059.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 817 kB/s | 34 kB 00:00 2024-03-06T07:19:38.363 INFO:teuthology.orchestra.run.smithi059.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 41 MB/s | 4.7 MB 00:00 2024-03-06T07:19:38.413 INFO:teuthology.orchestra.run.smithi059.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 20 MB/s | 3.0 MB 00:00 2024-03-06T07:19:38.455 INFO:teuthology.orchestra.run.smithi059.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 17 MB/s | 1.5 MB 00:00 2024-03-06T07:19:38.488 INFO:teuthology.orchestra.run.smithi059.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-06T07:19:38.513 INFO:teuthology.orchestra.run.smithi059.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-06T07:19:38.631 INFO:teuthology.orchestra.run.smithi059.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-06T07:19:38.664 INFO:teuthology.orchestra.run.smithi059.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 6.8 MB/s | 1.0 MB 00:00 2024-03-06T07:19:38.690 INFO:teuthology.orchestra.run.smithi059.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-06T07:19:38.840 INFO:teuthology.orchestra.run.smithi059.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-03-06T07:19:38.865 INFO:teuthology.orchestra.run.smithi059.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.3 MB/s | 227 kB 00:00 2024-03-06T07:19:38.882 INFO:teuthology.orchestra.run.smithi059.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 587 kB/s | 24 kB 00:00 2024-03-06T07:19:38.916 INFO:teuthology.orchestra.run.smithi059.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-06T07:19:38.941 INFO:teuthology.orchestra.run.smithi059.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-06T07:19:38.966 INFO:teuthology.orchestra.run.smithi059.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 627 kB/s | 32 kB 00:00 2024-03-06T07:19:39.150 INFO:teuthology.orchestra.run.smithi059.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-03-06T07:19:39.175 INFO:teuthology.orchestra.run.smithi059.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2024-03-06T07:19:39.211 INFO:teuthology.orchestra.run.smithi059.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 5.1 MB/s | 174 kB 00:00 2024-03-06T07:19:39.234 INFO:teuthology.orchestra.run.smithi059.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-03-06T07:19:39.259 INFO:teuthology.orchestra.run.smithi059.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 824 kB/s | 41 kB 00:00 2024-03-06T07:19:39.284 INFO:teuthology.orchestra.run.smithi059.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-06T07:19:39.310 INFO:teuthology.orchestra.run.smithi059.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-06T07:19:39.335 INFO:teuthology.orchestra.run.smithi059.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-06T07:19:39.360 INFO:teuthology.orchestra.run.smithi059.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2024-03-06T07:19:39.385 INFO:teuthology.orchestra.run.smithi059.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-03-06T07:19:39.538 INFO:teuthology.orchestra.run.smithi164.stdout:(176/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 914 kB/s | 47 MB 00:53 2024-03-06T07:19:39.542 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:19:39.542 INFO:teuthology.orchestra.run.smithi164.stdout:Total 3.5 MB/s | 224 MB 01:03 2024-03-06T07:19:39.587 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-06T07:19:39.628 INFO:teuthology.orchestra.run.smithi059.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 161 kB/s | 39 kB 00:00 2024-03-06T07:19:39.870 INFO:teuthology.orchestra.run.smithi059.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-06T07:19:39.894 INFO:teuthology.orchestra.run.smithi059.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 155 kB/s | 41 kB 00:00 2024-03-06T07:19:39.899 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-06T07:19:39.900 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-06T07:19:39.920 INFO:teuthology.orchestra.run.smithi059.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 215 kB 00:00 2024-03-06T07:19:39.945 INFO:teuthology.orchestra.run.smithi059.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 668 kB/s | 34 kB 00:00 2024-03-06T07:19:39.971 INFO:teuthology.orchestra.run.smithi059.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 862 kB/s | 43 kB 00:00 2024-03-06T07:19:39.996 INFO:teuthology.orchestra.run.smithi059.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-06T07:19:40.021 INFO:teuthology.orchestra.run.smithi059.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-06T07:19:40.047 INFO:teuthology.orchestra.run.smithi059.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-03-06T07:19:40.080 INFO:teuthology.orchestra.run.smithi059.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-06T07:19:40.105 INFO:teuthology.orchestra.run.smithi059.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 756 kB/s | 44 kB 00:00 2024-03-06T07:19:40.131 INFO:teuthology.orchestra.run.smithi059.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-06T07:19:40.156 INFO:teuthology.orchestra.run.smithi059.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-03-06T07:19:40.215 INFO:teuthology.orchestra.run.smithi059.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 802 kB 00:00 2024-03-06T07:19:40.240 INFO:teuthology.orchestra.run.smithi059.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 194 kB 00:00 2024-03-06T07:19:40.282 INFO:teuthology.orchestra.run.smithi059.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-06T07:19:40.307 INFO:teuthology.orchestra.run.smithi059.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-06T07:19:40.341 INFO:teuthology.orchestra.run.smithi059.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-06T07:19:40.366 INFO:teuthology.orchestra.run.smithi059.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-06T07:19:40.383 INFO:teuthology.orchestra.run.smithi059.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.9 MB/s | 124 kB 00:00 2024-03-06T07:19:40.408 INFO:teuthology.orchestra.run.smithi059.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.7 MB/s | 113 kB 00:00 2024-03-06T07:19:40.434 INFO:teuthology.orchestra.run.smithi059.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-06T07:19:40.492 INFO:teuthology.orchestra.run.smithi059.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 384 kB/s | 22 kB 00:00 2024-03-06T07:19:40.692 INFO:teuthology.orchestra.run.smithi059.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 866 kB/s | 245 kB 00:00 2024-03-06T07:19:40.751 INFO:teuthology.orchestra.run.smithi059.stdout:(105/176): protobuf-compiler-3.14.0-13.el9.x86_ 3.3 MB/s | 863 kB 00:00 2024-03-06T07:19:40.860 INFO:teuthology.orchestra.run.smithi059.stdout:(106/176): abseil-cpp-20211102.0-3.el9.x86_64.r 3.2 MB/s | 548 kB 00:00 2024-03-06T07:19:40.902 INFO:teuthology.orchestra.run.smithi059.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 468 kB/s | 19 kB 00:00 2024-03-06T07:19:40.927 INFO:teuthology.orchestra.run.smithi059.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.7 MB/s | 309 kB 00:00 2024-03-06T07:19:40.969 INFO:teuthology.orchestra.run.smithi059.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 627 kB/s | 26 kB 00:00 2024-03-06T07:19:41.011 INFO:teuthology.orchestra.run.smithi059.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-06T07:19:41.078 INFO:teuthology.orchestra.run.smithi059.stdout:(111/176): libarrow-9.0.0-10.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-03-06T07:19:41.103 INFO:teuthology.orchestra.run.smithi059.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-03-06T07:19:41.128 INFO:teuthology.orchestra.run.smithi059.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-03-06T07:19:41.179 INFO:teuthology.orchestra.run.smithi059.stdout:(114/176): python3-asyncssh-2.13.2-4.el9.noarch 11 MB/s | 548 kB 00:00 2024-03-06T07:19:41.204 INFO:teuthology.orchestra.run.smithi059.stdout:(115/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.1 MB/s | 838 kB 00:00 2024-03-06T07:19:41.229 INFO:teuthology.orchestra.run.smithi059.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 861 kB/s | 43 kB 00:00 2024-03-06T07:19:41.255 INFO:teuthology.orchestra.run.smithi059.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-03-06T07:19:41.280 INFO:teuthology.orchestra.run.smithi059.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 300 kB/s | 15 kB 00:00 2024-03-06T07:19:41.305 INFO:teuthology.orchestra.run.smithi059.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-03-06T07:19:41.330 INFO:teuthology.orchestra.run.smithi059.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-06T07:19:41.356 INFO:teuthology.orchestra.run.smithi059.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-03-06T07:19:41.373 INFO:teuthology.orchestra.run.smithi059.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 842 kB/s | 35 kB 00:00 2024-03-06T07:19:41.406 INFO:teuthology.orchestra.run.smithi059.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 3.9 MB/s | 203 kB 00:00 2024-03-06T07:19:41.456 INFO:teuthology.orchestra.run.smithi059.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-03-06T07:19:41.499 INFO:teuthology.orchestra.run.smithi059.stdout:(125/176): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-03-06T07:19:41.524 INFO:teuthology.orchestra.run.smithi059.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-06T07:19:41.549 INFO:teuthology.orchestra.run.smithi059.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-03-06T07:19:41.574 INFO:teuthology.orchestra.run.smithi059.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-03-06T07:19:41.600 INFO:teuthology.orchestra.run.smithi059.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 389 kB/s | 19 kB 00:00 2024-03-06T07:19:41.625 INFO:teuthology.orchestra.run.smithi059.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-03-06T07:19:41.658 INFO:teuthology.orchestra.run.smithi059.stdout:(131/176): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-03-06T07:19:41.684 INFO:teuthology.orchestra.run.smithi059.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 793 kB/s | 46 kB 00:00 2024-03-06T07:19:41.717 INFO:teuthology.orchestra.run.smithi059.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-06T07:19:41.743 INFO:teuthology.orchestra.run.smithi059.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 985 kB/s | 58 kB 00:00 2024-03-06T07:19:41.768 INFO:teuthology.orchestra.run.smithi059.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-06T07:19:41.793 INFO:teuthology.orchestra.run.smithi059.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-03-06T07:19:41.818 INFO:teuthology.orchestra.run.smithi059.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-06T07:19:41.844 INFO:teuthology.orchestra.run.smithi059.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 564 kB/s | 28 kB 00:00 2024-03-06T07:19:41.869 INFO:teuthology.orchestra.run.smithi059.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-03-06T07:19:41.885 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-06T07:19:41.885 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-06T07:19:41.894 INFO:teuthology.orchestra.run.smithi059.stdout:(140/176): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-03-06T07:19:41.919 INFO:teuthology.orchestra.run.smithi059.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-06T07:19:41.945 INFO:teuthology.orchestra.run.smithi059.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-03-06T07:19:41.970 INFO:teuthology.orchestra.run.smithi059.stdout:(143/176): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-06T07:19:41.995 INFO:teuthology.orchestra.run.smithi059.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-06T07:19:42.012 INFO:teuthology.orchestra.run.smithi059.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-03-06T07:19:42.046 INFO:teuthology.orchestra.run.smithi059.stdout:(146/176): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-03-06T07:19:42.071 INFO:teuthology.orchestra.run.smithi059.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-03-06T07:19:42.096 INFO:teuthology.orchestra.run.smithi059.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-06T07:19:42.138 INFO:teuthology.orchestra.run.smithi059.stdout:(149/176): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-03-06T07:19:42.405 INFO:teuthology.orchestra.run.smithi059.stdout:(150/176): ceph-mgr-diskprediction-local-19.0.0 917 kB/s | 7.4 MB 00:08 2024-03-06T07:19:42.431 INFO:teuthology.orchestra.run.smithi059.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.7 MB/s | 68 kB 00:00 2024-03-06T07:19:42.515 INFO:teuthology.orchestra.run.smithi059.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 27 MB/s | 2.3 MB 00:00 2024-03-06T07:19:42.581 INFO:teuthology.orchestra.run.smithi059.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 27 MB/s | 1.8 MB 00:00 2024-03-06T07:19:42.615 INFO:teuthology.orchestra.run.smithi059.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:42.657 INFO:teuthology.orchestra.run.smithi059.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-06T07:19:42.690 INFO:teuthology.orchestra.run.smithi059.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:42.725 INFO:teuthology.orchestra.run.smithi059.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 35 kB 00:00 2024-03-06T07:19:42.757 INFO:teuthology.orchestra.run.smithi059.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 5.6 MB/s | 191 kB 00:00 2024-03-06T07:19:42.799 INFO:teuthology.orchestra.run.smithi059.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-06T07:19:42.858 INFO:teuthology.orchestra.run.smithi059.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:19:42.900 INFO:teuthology.orchestra.run.smithi059.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 10 MB/s | 433 kB 00:00 2024-03-06T07:19:42.925 INFO:teuthology.orchestra.run.smithi059.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 874 kB/s | 22 kB 00:00 2024-03-06T07:19:42.959 INFO:teuthology.orchestra.run.smithi059.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 107 kB 00:00 2024-03-06T07:19:42.992 INFO:teuthology.orchestra.run.smithi059.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 4.5 MB/s | 154 kB 00:00 2024-03-06T07:19:43.026 INFO:teuthology.orchestra.run.smithi059.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 4.0 MB/s | 135 kB 00:00 2024-03-06T07:19:43.068 INFO:teuthology.orchestra.run.smithi059.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 63 kB 00:00 2024-03-06T07:19:43.119 INFO:teuthology.orchestra.run.smithi059.stdout:(167/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 3.2 MB/s | 3.1 MB 00:00 2024-03-06T07:19:43.143 INFO:teuthology.orchestra.run.smithi059.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 367 kB/s | 28 kB 00:00 2024-03-06T07:19:43.177 INFO:teuthology.orchestra.run.smithi059.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 20 MB/s | 1.2 MB 00:00 2024-03-06T07:19:43.227 INFO:teuthology.orchestra.run.smithi059.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 26 MB/s | 2.2 MB 00:00 2024-03-06T07:19:43.278 INFO:teuthology.orchestra.run.smithi059.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 23 MB/s | 2.2 MB 00:00 2024-03-06T07:19:43.303 INFO:teuthology.orchestra.run.smithi059.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 5.9 MB/s | 438 kB 00:00 2024-03-06T07:19:43.329 INFO:teuthology.orchestra.run.smithi059.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 607 kB/s | 30 kB 00:00 2024-03-06T07:19:43.362 INFO:teuthology.orchestra.run.smithi059.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-06T07:19:43.395 INFO:teuthology.orchestra.run.smithi059.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-03-06T07:19:45.954 INFO:teuthology.orchestra.run.smithi059.stdout:(176/176): librados2-19.0.0-1739.g1bc250c7.el9. 901 kB/s | 3.4 MB 00:03 2024-03-06T07:19:45.959 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:19:45.959 INFO:teuthology.orchestra.run.smithi059.stdout:Total 4.9 MB/s | 224 MB 00:45 2024-03-06T07:19:46.006 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2024-03-06T07:19:46.345 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2024-03-06T07:19:46.346 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2024-03-06T07:19:48.387 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2024-03-06T07:19:48.388 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2024-03-06T07:19:48.914 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-06T07:19:49.017 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:19:49.051 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:19:49.089 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:49.110 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:49.177 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:19:49.224 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:19:49.278 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:19:49.318 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:19:49.344 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:19:49.399 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:19:49.725 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:19:49.779 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:49.903 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:50.029 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:50.065 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:19:50.232 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:19:50.290 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:19:50.317 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:50.404 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:50.426 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:50.475 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:50.514 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:19:50.536 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:50.551 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:50.745 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:50.776 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:50.816 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:19:50.996 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:19:51.135 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:51.187 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:51.294 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:51.388 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:51.608 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:51.721 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:19:51.823 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:19:51.863 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:19:51.937 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:19:51.990 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:19:52.022 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:19:52.065 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:19:52.113 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:19:52.176 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:19:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:19:52.255 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:19:52.290 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:19:52.318 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:19:52.387 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:19:52.423 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:19:52.466 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:19:52.495 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:19:52.564 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:19:52.610 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:19:52.683 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:19:52.826 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:19:52.879 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:19:53.402 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:19:53.474 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:19:53.619 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:19:53.627 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:19:53.666 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:19:53.704 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:19:53.741 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:19:53.848 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:19:53.929 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:19:54.206 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:19:54.393 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:19:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:19:54.508 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:19:54.650 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:19:54.703 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:19:54.757 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:19:54.831 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:19:54.870 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:19:54.907 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:19:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:19:54.990 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:19:55.032 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:19:55.075 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:19:55.218 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:19:55.264 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:19:55.294 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:19:55.336 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:19:55.431 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:19:55.469 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:19:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:19:55.535 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:19:55.559 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:19:55.618 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2024-03-06T07:19:55.649 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:19:55.725 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:19:55.760 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:19:55.777 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:19:55.806 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:55.835 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:55.892 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:19:55.894 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:19:55.924 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:19:55.934 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:19:55.978 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:19:55.983 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:19:56.019 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:19:56.044 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:19:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:19:56.088 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:19:56.240 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:19:56.288 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:19:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:19:56.445 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:19:56.462 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:19:56.494 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:19:56.498 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:56.526 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:19:56.553 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:19:56.584 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:19:56.605 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:56.612 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:19:56.632 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:56.646 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:19:56.685 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:19:56.708 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:56.717 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:19:56.742 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:19:56.747 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:19:56.824 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:19:56.860 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:19:56.893 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:19:56.913 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:19:56.935 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:19:56.969 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:19:56.975 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:19:57.020 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:57.060 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:19:57.125 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:57.140 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:19:57.154 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:57.184 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:19:57.216 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:19:57.218 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:57.258 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:19:57.262 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:19:57.280 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:57.295 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:57.361 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:19:57.408 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:19:57.464 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:57.495 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:57.535 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:19:57.700 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:19:57.845 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:57.905 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:58.013 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:58.092 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:58.144 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:19:58.189 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:19:58.222 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:19:58.259 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:58.271 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:19:58.367 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:19:58.374 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:19:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:19:58.469 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:19:58.504 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:19:58.574 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:19:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:19:58.597 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:19:58.635 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:19:58.665 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:19:58.674 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:19:58.701 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:19:58.741 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:19:58.806 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:19:58.839 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:19:58.883 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:19:58.916 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:19:58.935 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:19:59.006 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:19:59.040 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:19:59.084 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:19:59.112 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:19:59.183 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:19:59.235 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:19:59.310 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:19:59.319 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:19:59.428 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:19:59.472 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:19:59.530 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:20:00.079 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:20:00.158 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:20:00.249 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:20:00.295 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:00.304 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:20:00.343 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:20:00.372 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:20:00.404 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:20:00.498 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:20:00.563 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:20:00.657 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:20:00.734 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:20:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:20:00.817 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:20:01.035 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:20:01.072 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:20:01.108 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:01.139 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:20:01.192 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:20:01.198 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:01.221 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:20:01.258 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:20:01.333 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:20:01.386 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:20:01.434 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:20:01.507 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:20:01.537 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:20:01.573 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:20:01.620 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:20:01.656 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:20:01.699 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:20:01.718 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:20:01.741 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:20:01.879 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:20:01.930 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:20:01.952 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:01.960 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:20:02.002 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:20:02.098 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:20:02.134 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:20:02.174 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:02.175 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:20:02.200 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:20:02.204 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:20:02.224 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:20:02.312 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:20:02.436 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:20:02.543 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:20:02.582 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:20:02.649 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:20:02.713 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:20:02.901 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:20:02.953 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:20:03.094 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:20:03.151 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:20:03.183 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:20:03.215 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:20:03.242 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:20:03.273 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:20:03.301 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:20:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:20:03.374 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:20:03.406 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:20:03.436 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:20:03.514 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:20:03.546 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:20:03.579 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:20:03.612 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:20:03.647 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:20:03.738 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:20:03.828 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:20:03.869 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:20:03.901 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:20:03.948 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:20:04.048 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:20:04.103 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:20:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:04.434 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:04.860 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:20:04.907 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:20:04.941 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:20:04.977 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:04.990 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:20:05.013 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:05.080 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:20:05.091 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:05.091 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:20:05.091 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:20:05.091 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:05.122 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:20:05.131 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:05.305 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:20:05.337 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:20:05.425 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:20:06.102 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:20:06.212 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:20:07.482 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:20:07.573 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:20:07.599 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:20:07.926 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:20:07.962 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:08.053 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:08.083 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:20:08.121 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:20:08.597 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:20:08.632 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:08.695 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:08.709 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:20:10.977 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:11.005 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:11.572 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:11.700 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:11.774 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:11.774 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:20:11.774 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:20:11.774 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:11.817 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:16.788 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:16.788 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2024-03-06T07:20:16.788 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2024-03-06T07:20:16.788 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2024-03-06T07:20:16.788 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2024-03-06T07:20:16.788 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2024-03-06T07:20:16.788 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /root 2024-03-06T07:20:16.789 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2024-03-06T07:20:16.789 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:16.962 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:16.977 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:16.977 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:16.977 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:20:16.977 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:18.473 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:18.498 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:18.498 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:18.498 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:20:18.499 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:18.499 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:18.499 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:18.716 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:18.738 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:18.738 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:18.738 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:20:18.738 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:18.738 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:18.738 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:19.184 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:19.205 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:19.205 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:19.205 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:20:19.205 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:19.205 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:19.206 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:19.293 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:20:19.702 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:19.725 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:19.748 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:20:19.809 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:20:19.831 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:20.060 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:20.099 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:21.062 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:21.091 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:21.192 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:21.361 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:20:21.385 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:21.408 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:21.408 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:21.408 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:20:21.408 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:21.408 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:21.408 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:21.453 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:21.463 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:21.504 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:20:22.357 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:20:22.388 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:22.411 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:22.411 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:20:22.411 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:22.411 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:22.456 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:22.477 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:22.478 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:22.478 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:20:22.478 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:22.523 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:22.545 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:22.545 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:22.545 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:20:22.545 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:22.545 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:22.545 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:22.865 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:22.887 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:22.888 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:22.888 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:20:22.888 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:22.888 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:22.888 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:23.799 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:23.799 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2024-03-06T07:20:23.799 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2024-03-06T07:20:23.799 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2024-03-06T07:20:23.799 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2024-03-06T07:20:23.799 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2024-03-06T07:20:23.799 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /root 2024-03-06T07:20:23.799 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2024-03-06T07:20:23.799 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:23.979 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:23.993 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:23.993 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:23.993 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:20:23.993 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:25.524 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:25.548 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:25.548 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:25.548 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:20:25.549 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:25.549 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:25.549 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:25.774 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:25.796 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:25.796 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:25.796 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:20:25.796 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:25.796 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:25.796 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:26.258 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:26.280 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:26.280 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:26.280 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:20:26.280 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:26.280 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:26.280 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:26.376 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:20:26.811 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:26.833 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:26.856 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:20:26.910 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:20:26.939 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:27.176 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:27.215 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:27.312 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:20:27.352 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:20:27.383 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:20:27.410 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:20:27.600 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:20:27.718 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:20:27.767 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:20:27.817 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:20:27.875 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:20:27.948 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:20:27.986 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:20:28.021 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:20:28.052 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:20:28.081 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:20:28.166 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:20:28.224 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:28.232 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:20:28.248 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:28.272 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:20:28.352 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:28.518 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:20:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:28.572 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:28.572 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:28.572 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:20:28.573 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:28.573 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:28.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:28.628 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:20:28.843 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:28.880 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:28.899 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:29.571 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:20:29.603 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:29.624 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:29.624 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:29.625 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:20:29.625 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:29.625 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:29.625 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:29.670 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:29.691 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:29.692 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:29.692 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:20:29.692 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:29.746 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:29.755 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:20:29.767 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:29.767 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:29.768 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:20:29.768 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:29.768 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:29.768 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:29.826 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:20:29.867 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:20:29.954 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:20:29.987 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:20:30.013 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:20:30.038 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:20:30.064 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:20:30.065 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:30.105 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:30.127 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:30.127 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:30.128 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:20:30.128 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:30.128 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:30.128 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:30.149 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:30.177 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:20:30.232 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:20:30.232 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:30.317 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:30.343 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:20:30.377 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:20:30.402 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:20:30.434 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:20:30.462 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:20:30.496 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:20:30.534 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:20:30.575 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:20:30.600 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:20:30.625 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:20:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:30.667 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:20:30.674 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:30.674 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:30.854 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:30.937 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:30.969 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:30.996 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:20:31.020 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:20:31.046 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:20:31.112 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:34.785 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:20:34.826 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:20:34.855 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:20:34.883 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:20:35.087 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:20:35.202 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:20:35.239 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:20:35.289 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:20:35.355 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:20:35.439 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:20:35.482 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:20:35.526 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:20:35.556 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:20:35.603 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:20:35.676 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:20:35.736 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:20:35.767 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:20:36.040 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:36.040 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:20:36.040 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:20:36.040 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:20:36.040 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:20:36.040 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:20:36.040 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:20:36.041 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:20:36.042 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:20:36.044 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:20:36.045 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:20:36.045 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:20:36.045 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:20:36.045 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:20:36.045 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:20:36.045 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:20:36.045 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:20:36.045 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:20:36.045 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:20:36.045 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:20:36.046 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:20:36.047 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:20:36.048 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:20:36.050 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:20:36.051 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:20:36.052 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:20:36.053 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:20:36.054 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:20:36.055 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:20:36.407 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:36.440 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:36.460 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:37.320 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:20:37.412 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:20:37.460 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:20:37.552 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:20:37.588 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:20:37.613 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:20:37.639 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:20:37.665 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:20:37.665 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:37.750 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:37.778 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:20:37.808 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:20:37.808 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:37.892 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:37.919 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:20:37.952 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:20:37.977 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:20:38.015 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:20:38.054 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:20:38.088 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:20:38.128 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:20:38.160 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:20:38.185 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:20:38.210 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:20:38.227 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:38.252 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:20:38.259 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:38.259 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:38.447 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:38.447 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:38.513 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:38.545 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:38.571 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:20:38.597 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:20:38.632 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:20:38.700 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:38.871 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:20:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:38.872 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2024-03-06T07:20:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:20:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:20:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:20:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:20:38.872 INFO:teuthology.orchestra.run.smithi164.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:20:38.873 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:38.874 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.875 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:20:38.876 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:20:38.877 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:20:38.878 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.879 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:20:38.880 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:38.881 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.882 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:20:38.883 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:20:38.884 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:20:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:20:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:20:38.885 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:38.885 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-06T07:20:39.532 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:20:43.359 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:20:43.360 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:20:43.361 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:20:43.378 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:20:43.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:20:43.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:20:43.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:20:43.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:20:43.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:20:43.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:20:43.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:20:43.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:20:43.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:20:43.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:20:43.379 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:20:43.380 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:20:43.380 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:20:43.380 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:20:43.380 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:20:43.380 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:20:43.380 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:20:43.380 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:20:43.380 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:20:43.380 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:20:43.380 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:20:43.380 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:20:43.381 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:20:43.382 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:20:43.383 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:20:43.384 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:20:43.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:20:43.386 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:20:43.387 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:20:43.388 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:20:43.389 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:20:43.390 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:20:43.391 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:20:43.392 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:20:43.393 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:20:46.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:20:46.504 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:46.504 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.505 INFO:teuthology.orchestra.run.smithi059.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:20:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:20:46.508 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:20:46.509 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:20:46.509 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:20:46.509 INFO:teuthology.orchestra.run.smithi059.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:20:46.509 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:20:46.509 INFO:teuthology.orchestra.run.smithi059.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:20:46.509 INFO:teuthology.orchestra.run.smithi059.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:20:46.509 INFO:teuthology.orchestra.run.smithi059.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:20:46.509 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.510 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:46.511 INFO:teuthology.orchestra.run.smithi059.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.512 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:20:46.513 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:20:46.514 INFO:teuthology.orchestra.run.smithi059.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:20:46.515 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:20:46.516 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:20:46.517 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:20:46.518 INFO:teuthology.orchestra.run.smithi059.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:20:46.518 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:46.518 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2024-03-06T07:20:47.119 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:47.120 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:20:47.337 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:20:47.360 INFO:teuthology.orchestra.run.smithi059.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:47.361 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:47.361 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:20:47.362 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:20:47.500 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:20:47.525 INFO:teuthology.orchestra.run.smithi164.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:47.526 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:47.526 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:20:47.528 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:20:47.528 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:20:47.528 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:20:47.560 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:20:47.560 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:20:47.599 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:20:47.599 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:20:47.599 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:20:47.628 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:20:47.696 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:20:47.697 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:20:47.727 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:20:47.795 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:20:47.796 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:20:47.796 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:20:47.823 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:20:47.890 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:20:47.890 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:20:47.919 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:20:47.988 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:20:47.988 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:20:47.988 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:20:48.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:20:48.083 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:20:48.083 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:20:48.111 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:20:48.180 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:20:48.268 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:20:48.268 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /var/log/ceph 2024-03-06T07:20:48.270 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2024-03-06T07:20:48.298 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:20:48.298 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:20:48.339 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:20:48.366 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:20:48.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:20:48.406 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:20:48.439 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:20:48.440 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:20:48.440 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:20:48.440 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:20:48.440 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:20:48.485 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:48.606 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:48.632 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:20:48.633 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:20:48.647 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:48.769 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:48.802 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:20:48.802 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:20:48.802 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7583061', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi059.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL92MghNmvkK0dl0CgJMYwSUjvF1nywFkmBXqC2ED6NNW+83viPD25Wkyb6fgauJmStWWAHWHrm5awA7vedLBX4=', 'smithi164.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIiHbBK7ER2izv74TGbbmBGXlapy6lqgySg8fZnYGemXRtRo+HJIO8A9KRgScyXvENad4XZmvrFXKwKusDHYcjU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:04:29', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:20:48.803 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:20:48.806 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:20:48.851 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:20:48.861 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:20:48.918 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:20:48.919 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:20:48.941 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:20:48.941 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2024-03-06T07:20:48.995 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:20:48.995 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:48.995 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-06T07:20:48.995 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:48.995 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:48.995 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-06 07:20:39.040370442 +0000 2024-03-06T07:20:48.996 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-06 07:17:37.662822717 +0000 2024-03-06T07:20:48.996 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-06 07:17:37.662822717 +0000 2024-03-06T07:20:48.996 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-06 07:17:37.662822717 +0000 2024-03-06T07:20:48.996 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:20:49.061 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-06T07:20:49.061 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-06T07:20:49.061 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000198373 s, 2.6 MB/s 2024-03-06T07:20:49.062 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:20:49.118 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2024-03-06T07:20:49.173 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:20:49.173 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:49.173 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-06T07:20:49.174 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:49.174 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:49.174 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-06 07:20:39.041370430 +0000 2024-03-06T07:20:49.174 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-06 07:17:38.014818472 +0000 2024-03-06T07:20:49.174 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-06 07:17:38.014818472 +0000 2024-03-06T07:20:49.174 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-06 07:17:38.014818472 +0000 2024-03-06T07:20:49.175 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:20:49.238 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-06T07:20:49.239 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-06T07:20:49.239 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.00015597 s, 3.3 MB/s 2024-03-06T07:20:49.239 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:20:49.295 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2024-03-06T07:20:49.350 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:20:49.351 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:49.351 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-06T07:20:49.351 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:49.351 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:49.351 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-06 07:20:39.041370430 +0000 2024-03-06T07:20:49.351 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-06 07:17:38.349814432 +0000 2024-03-06T07:20:49.352 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-06 07:17:38.349814432 +0000 2024-03-06T07:20:49.352 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-06 07:17:38.349814432 +0000 2024-03-06T07:20:49.352 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:20:49.415 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-06T07:20:49.415 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-06T07:20:49.415 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000210407 s, 2.4 MB/s 2024-03-06T07:20:49.416 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:20:49.472 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2024-03-06T07:20:49.527 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:20:49.527 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:49.528 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-06T07:20:49.528 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:49.528 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:49.528 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-06 07:20:39.042370417 +0000 2024-03-06T07:20:49.528 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-06 07:17:38.687810356 +0000 2024-03-06T07:20:49.528 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-06 07:17:38.687810356 +0000 2024-03-06T07:20:49.529 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2024-03-06 07:17:38.687810356 +0000 2024-03-06T07:20:49.529 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:20:49.593 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2024-03-06T07:20:49.593 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2024-03-06T07:20:49.593 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000178502 s, 2.9 MB/s 2024-03-06T07:20:49.595 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:20:49.650 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:49.650 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:20:49.650 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:20:49.667 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:20:49.667 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-03-06T07:20:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:20:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:49.723 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-06T07:20:49.724 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:49.724 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:49.724 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-06 07:20:31.519055771 +0000 2024-03-06T07:20:49.724 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-06 07:17:37.650426318 +0000 2024-03-06T07:20:49.724 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-06 07:17:37.650426318 +0000 2024-03-06T07:20:49.724 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-06 07:17:37.650426318 +0000 2024-03-06T07:20:49.724 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:20:49.790 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-06T07:20:49.790 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-06T07:20:49.790 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000227793 s, 2.2 MB/s 2024-03-06T07:20:49.791 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:20:49.848 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-03-06T07:20:49.905 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:20:49.906 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:49.906 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-06T07:20:49.906 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:49.906 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:49.906 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-06 07:20:31.520055755 +0000 2024-03-06T07:20:49.906 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-06 07:17:37.991420887 +0000 2024-03-06T07:20:49.907 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-06 07:17:37.991420887 +0000 2024-03-06T07:20:49.907 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-06 07:17:37.991420887 +0000 2024-03-06T07:20:49.907 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:20:49.971 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-06T07:20:49.971 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-06T07:20:49.971 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000166464 s, 3.1 MB/s 2024-03-06T07:20:49.973 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:20:50.028 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-03-06T07:20:50.083 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:20:50.083 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:50.083 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-06T07:20:50.083 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:50.083 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:50.084 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-06 07:20:31.520055755 +0000 2024-03-06T07:20:50.084 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-06 07:17:38.351415154 +0000 2024-03-06T07:20:50.084 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-06 07:17:38.351415154 +0000 2024-03-06T07:20:50.084 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-06 07:17:38.351415154 +0000 2024-03-06T07:20:50.084 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:20:50.149 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-06T07:20:50.149 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-06T07:20:50.149 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000183207 s, 2.8 MB/s 2024-03-06T07:20:50.150 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:20:50.206 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-03-06T07:20:50.260 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:20:50.260 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:50.260 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-06T07:20:50.260 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:50.260 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:50.260 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-06 07:20:31.520055755 +0000 2024-03-06T07:20:50.260 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-06 07:17:38.695409676 +0000 2024-03-06T07:20:50.261 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-06 07:17:38.695409676 +0000 2024-03-06T07:20:50.261 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-06 07:17:38.695409676 +0000 2024-03-06T07:20:50.261 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:20:50.330 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-06T07:20:50.330 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-06T07:20:50.330 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000218239 s, 2.3 MB/s 2024-03-06T07:20:50.331 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:20:50.387 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:50.387 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:20:50.387 INFO:tasks.ceph:Generating config... 2024-03-06T07:20:50.388 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:20:50.389 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:20:50.389 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:20:50.389 INFO:tasks.ceph:[client] pid file = 2024-03-06T07:20:50.389 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:20:50.389 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:20:50.389 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:20:50.389 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:20:50.389 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:20:50.389 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:20:50.389 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:20:50.389 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:20:50.390 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:20:50.390 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:20:50.390 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:20:50.390 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:20:50.390 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:20:50.390 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:20:50.390 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:20:50.391 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:20:50.391 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:20:50.391 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:20:50.391 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-06T07:20:50.391 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-06T07:20:50.391 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:20:50.392 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:20:50.393 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:20:50.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:20:50.475 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:20:50.478 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:20:50.525 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:20:50.553 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.59'), ('mon.b', '172.21.15.164'), ('mon.c', '[v2:172.21.15.164:3301,v1:172.21.15.164:6790]')] 2024-03-06T07:20:50.553 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:20:50.554 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:20:50.555 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:20:50.608 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.59 --add b 172.21.15.164 --addv c '[v2:172.21.15.164:3301,v1:172.21.15.164:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:50.693 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:50.693 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: generated fsid 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:20:50.693 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = quincy 2024-03-06T07:20:50.693 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 0 2024-03-06T07:20:50.693 INFO:teuthology.orchestra.run.smithi059.stdout:fsid 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:20:50.693 INFO:teuthology.orchestra.run.smithi059.stdout:last_changed 2024-03-06T07:20:50.692283+0000 2024-03-06T07:20:50.693 INFO:teuthology.orchestra.run.smithi059.stdout:created 2024-03-06T07:20:50.692283+0000 2024-03-06T07:20:50.693 INFO:teuthology.orchestra.run.smithi059.stdout:min_mon_release 17 (quincy) 2024-03-06T07:20:50.693 INFO:teuthology.orchestra.run.smithi059.stdout:election_strategy: 1 2024-03-06T07:20:50.693 INFO:teuthology.orchestra.run.smithi059.stdout:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2024-03-06T07:20:50.693 INFO:teuthology.orchestra.run.smithi059.stdout:1: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.b 2024-03-06T07:20:50.694 INFO:teuthology.orchestra.run.smithi059.stdout:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-03-06T07:20:50.694 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:20:50.695 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:20:50.709 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 43c1ee46-a208-44bd-a1f3-57a84ef7ed13... 2024-03-06T07:20:50.710 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:20:50.752 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:20:50.800 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-03-06T07:20:50.800 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-03-06T07:20:50.800 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:20:50.800 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-03-06T07:20:50.800 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.800 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-03-06T07:20:50.800 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.800 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-03-06T07:20:50.800 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.801 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-03-06T07:20:50.802 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:20:50.803 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-03-06T07:20:50.805 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.805 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-03-06T07:20:50.805 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.805 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:20:50.805 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-03-06T07:20:50.805 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.805 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-03-06T07:20:50.805 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-03-06T07:20:50.805 INFO:teuthology.orchestra.run.smithi059.stdout: bdev async discard = True 2024-03-06T07:20:50.805 INFO:teuthology.orchestra.run.smithi059.stdout: bdev enable discard = True 2024-03-06T07:20:50.805 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore allocator = bitmap 2024-03-06T07:20:50.806 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-03-06T07:20:50.806 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-03-06T07:20:50.806 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 1/20 2024-03-06T07:20:50.806 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 1/20 2024-03-06T07:20:50.806 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:20:50.806 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-03-06T07:20:50.806 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 4/10 2024-03-06T07:20:50.806 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:50.807 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-03-06T07:20:50.807 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:20:50.807 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:20:50.807 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-03-06T07:20:50.807 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:20:50.807 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.807 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-03-06T07:20:50.808 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:20:50.808 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-03-06T07:20:50.808 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:20:50.808 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:20:50.808 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:50.808 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:50.809 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-03-06T07:20:50.809 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.809 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-03-06T07:20:50.809 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:20:50.809 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:20:50.809 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-03-06T07:20:50.810 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:20:50.810 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-03-06T07:20:50.810 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-03-06T07:20:50.810 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:50.810 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-03-06T07:20:50.810 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-03-06T07:20:50.810 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:50.811 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.811 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:20:50.811 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:20:50.811 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:20:50.811 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.811 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:20:50.811 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:20:50.812 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:20:50.812 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.812 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-03-06T07:20:50.812 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:20:50.812 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.812 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:20:50.812 INFO:teuthology.orchestra.run.smithi059.stdout: mon op complaint time = 120 2024-03-06T07:20:50.812 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:20:50.813 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-03-06T07:20:50.813 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-03-06T07:20:50.813 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-03-06T07:20:50.813 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-03-06T07:20:50.813 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:20:50.813 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:20:50.813 INFO:teuthology.orchestra.run.smithi059.stdout: client mount timeout = 600 2024-03-06T07:20:50.814 INFO:teuthology.orchestra.run.smithi059.stdout: debug client = 20 2024-03-06T07:20:50.814 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:20:50.814 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = "" 2024-03-06T07:20:50.814 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:20:50.814 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:20:50.815 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-03-06T07:20:50.815 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-03-06T07:20:50.815 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-03-06T07:20:50.815 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-03-06T07:20:50.815 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-03-06T07:20:50.815 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-03-06T07:20:50.815 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-03-06T07:20:50.816 INFO:teuthology.orchestra.run.smithi059.stdout:[mds] 2024-03-06T07:20:50.816 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds = 20 2024-03-06T07:20:50.816 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds balancer = 20 2024-03-06T07:20:50.816 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:20:50.817 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug frag = True 2024-03-06T07:20:50.817 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug scatterstat = True 2024-03-06T07:20:50.817 INFO:teuthology.orchestra.run.smithi059.stdout: mds op complaint time = 180 2024-03-06T07:20:50.817 INFO:teuthology.orchestra.run.smithi059.stdout: mds verify scatter = True 2024-03-06T07:20:50.817 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:20:50.817 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:20:50.817 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:20:50.819 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-03-06T07:20:50.820 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-03-06T07:20:50.820 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:20:50.820 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-03-06T07:20:50.820 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.820 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-03-06T07:20:50.820 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.820 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-03-06T07:20:50.820 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.821 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-03-06T07:20:50.821 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-03-06T07:20:50.821 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.821 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-03-06T07:20:50.821 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-03-06T07:20:50.821 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.821 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:20:50.821 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-03-06T07:20:50.821 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.822 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:20:50.822 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:20:50.822 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.822 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-03-06T07:20:50.822 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.822 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-03-06T07:20:50.822 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-03-06T07:20:50.822 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:20:50.823 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:20:50.823 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-03-06T07:20:50.823 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:20:50.823 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-03-06T07:20:50.823 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:20:50.823 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:20:50.823 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-03-06T07:20:50.823 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.823 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:20:50.824 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.824 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-03-06T07:20:50.824 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.824 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-03-06T07:20:50.824 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.824 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-03-06T07:20:50.824 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-03-06T07:20:50.824 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-03-06T07:20:50.825 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:20:50.825 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:20:50.825 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.825 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-03-06T07:20:50.825 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-03-06T07:20:50.825 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.825 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-03-06T07:20:50.825 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-03-06T07:20:50.826 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:20:50.826 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.826 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-03-06T07:20:50.826 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-03-06T07:20:50.826 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.826 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-03-06T07:20:50.826 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-03-06T07:20:50.826 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-03-06T07:20:50.826 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-03-06T07:20:50.826 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.827 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-03-06T07:20:50.827 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-03-06T07:20:50.827 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.827 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:20:50.827 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.827 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-03-06T07:20:50.827 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.828 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-03-06T07:20:50.828 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.828 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:20:50.828 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-03-06T07:20:50.828 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.828 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-03-06T07:20:50.828 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-03-06T07:20:50.828 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2024-03-06T07:20:50.829 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2024-03-06T07:20:50.829 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = bitmap 2024-03-06T07:20:50.829 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-03-06T07:20:50.829 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-03-06T07:20:50.829 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2024-03-06T07:20:50.829 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2024-03-06T07:20:50.829 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:20:50.829 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-03-06T07:20:50.829 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2024-03-06T07:20:50.830 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:50.830 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-03-06T07:20:50.830 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:20:50.830 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:20:50.830 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-03-06T07:20:50.830 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:20:50.830 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.830 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-03-06T07:20:50.830 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:20:50.831 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-03-06T07:20:50.831 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:20:50.831 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:20:50.831 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:50.831 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:50.831 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-03-06T07:20:50.831 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.831 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:20:50.832 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: mon op complaint time = 120 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-03-06T07:20:50.833 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout: client mount timeout = 600 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout: debug client = 20 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = "" 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2024-03-06T07:20:50.834 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout:[mds] 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds = 20 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds balancer = 20 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug frag = True 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug scatterstat = True 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout: mds op complaint time = 180 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout: mds verify scatter = True 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:20:50.835 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:20:50.835 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:20:50.836 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:20:50.915 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:20:50.915 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:20:50.915 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:20:50.930 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:20:50.930 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:20:50.985 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:20:50.985 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:20:50.985 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:20:50.985 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:20:51.064 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:20:51.064 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:51.119 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:20:51.119 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:20:51.119 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:20:51.119 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:20:51.169 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:20:51.169 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:51.224 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:20:51.224 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:20:51.224 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:20:51.279 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:20:51.281 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:20:51.352 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:20:51.354 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:20:51.355 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:20:51.409 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:20:51.411 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:20:51.438 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:20:51.536 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:20:51.536 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:20:51.563 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:20:51.624 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:20:51.627 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:20:51.654 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:20:51.751 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:20:51.753 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:20:51.780 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:20:51.780 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:20:51.822 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:20:51.838 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-06T07:20:51.919 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-06T07:20:51.935 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:20:51.978 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:20:51.993 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-06T07:20:52.075 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-06T07:20:52.091 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:20:52.091 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:20:52.091 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:20:52.118 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:52.118 INFO:tasks.ceph:role: osd.0 2024-03-06T07:20:52.118 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:20:52.118 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:20:52.191 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:52.191 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:52.191 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:52.192 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:52.192 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:52.192 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:52.192 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:52.192 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:52.192 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:52.192 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:52.683 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-06T07:20:52.685 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-06T07:20:52.685 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:20:52.803 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:20:52.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:20:52.907 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:52.907 INFO:tasks.ceph:role: osd.1 2024-03-06T07:20:52.907 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:20:52.907 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:20:52.982 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:52.982 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:52.982 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:52.982 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:52.982 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:52.982 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:52.982 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:52.982 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:52.982 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:52.982 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:53.454 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-06T07:20:53.456 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-06T07:20:53.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:20:53.559 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:20:53.592 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:20:53.660 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:53.660 INFO:tasks.ceph:role: osd.2 2024-03-06T07:20:53.660 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:20:53.661 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:20:53.736 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:53.737 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:53.737 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:53.737 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:53.737 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:53.737 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:53.737 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:53.737 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:53.737 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:53.737 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:54.219 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-06T07:20:54.221 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-06T07:20:54.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:20:54.356 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:20:54.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:20:54.544 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:54.544 INFO:tasks.ceph:role: osd.3 2024-03-06T07:20:54.544 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:20:54.544 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:20:54.619 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:54.620 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:54.620 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:54.620 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:54.620 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:54.620 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:54.620 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:54.620 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:54.620 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:54.620 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:55.116 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2024-03-06T07:20:55.118 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2024-03-06T07:20:55.119 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:20:55.205 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:20:55.235 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:55.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:55.379+0000 7f24cc418640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:20:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:55.379+0000 7f24cc418640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:20:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:55.379+0000 7f24cc418640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:20:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:55.379+0000 7f24cc418640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:20:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:55.379+0000 7f24cc418640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:20:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:55.379+0000 7f24cc418640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:20:58.240 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:20:58.272 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:58.361+0000 7f7e7b0a2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:20:58.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:58.362+0000 7f7e7b0a2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:20:58.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:58.362+0000 7f7e7b0a2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:20:58.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:58.362+0000 7f7e7b0a2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:20:58.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:58.362+0000 7f7e7b0a2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:20:58.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:20:58.362+0000 7f7e7b0a2640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:21:01.219 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:01.250 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:01.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:01.340+0000 7f8d9b6c8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:21:01.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:01.340+0000 7f8d9b6c8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:21:01.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:01.340+0000 7f8d9b6c8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:01.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:01.340+0000 7f8d9b6c8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:01.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:01.340+0000 7f8d9b6c8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:01.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:01.340+0000 7f8d9b6c8640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:21:04.190 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:04.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:04.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:04.310+0000 7f35dd011640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:21:04.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:04.311+0000 7f35dd011640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:21:04.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:04.311+0000 7f35dd011640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:04.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:04.311+0000 7f35dd011640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:04.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:04.311+0000 7f35dd011640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:04.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:04.311+0000 7f35dd011640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:21:07.168 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:07.199 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:07.233 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:07.233 INFO:tasks.ceph:role: osd.4 2024-03-06T07:21:07.233 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:21:07.233 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:21:07.312 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:07.312 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:07.312 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:07.312 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:07.312 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:07.312 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:07.312 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:07.312 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:07.312 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:07.312 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:07.800 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-03-06T07:21:07.803 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-03-06T07:21:07.803 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:07.919 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:07.954 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:08.024 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:08.024 INFO:tasks.ceph:role: osd.5 2024-03-06T07:21:08.024 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:21:08.024 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:21:08.099 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:08.099 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:08.099 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:08.099 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:08.099 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:08.099 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:08.100 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:08.100 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:08.100 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:08.100 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:08.558 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-03-06T07:21:08.560 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-03-06T07:21:08.560 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:08.644 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:08.715 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:08.782 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:08.783 INFO:tasks.ceph:role: osd.6 2024-03-06T07:21:08.783 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:21:08.783 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:21:08.856 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:08.857 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:08.857 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:08.857 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:08.857 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:08.857 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:08.857 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:08.857 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:08.857 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:08.857 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:09.353 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-03-06T07:21:09.355 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-03-06T07:21:09.355 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:09.484 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:09.515 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:09.582 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:09.582 INFO:tasks.ceph:role: osd.7 2024-03-06T07:21:09.583 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:21:09.583 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:21:09.657 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:09.657 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:09.657 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:09.657 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:09.657 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:09.657 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:10.172 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-03-06T07:21:10.174 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-03-06T07:21:10.174 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:10.317 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:10.352 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:10.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:10.451+0000 7f4a99e0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:21:10.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:10.451+0000 7f4a99e0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:21:10.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:10.451+0000 7f4a99e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:10.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:10.451+0000 7f4a99e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:10.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:10.451+0000 7f4a99e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:10.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:10.452+0000 7f4a99e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:21:13.304 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:13.332 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:13.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:13.419+0000 7f6878e68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:21:13.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:13.419+0000 7f6878e68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:21:13.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:13.419+0000 7f6878e68640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:13.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:13.419+0000 7f6878e68640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:13.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:13.419+0000 7f6878e68640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:13.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:13.420+0000 7f6878e68640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:21:16.269 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:16.298 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:16.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:16.385+0000 7f3ec058a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:21:16.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:16.385+0000 7f3ec058a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:21:16.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:16.385+0000 7f3ec058a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:16.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:16.385+0000 7f3ec058a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:16.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:16.385+0000 7f3ec058a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:16.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:16.385+0000 7f3ec058a640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:21:18.804 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:18.807 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:19.233 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:19.262 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:19.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:19.350+0000 7fe6bd820640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:21:19.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:19.350+0000 7fe6bd820640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:21:19.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:19.350+0000 7fe6bd820640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:19.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:19.350+0000 7fe6bd820640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:19.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:19.350+0000 7fe6bd820640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:19.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:19.351+0000 7fe6bd820640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:21:22.211 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:22.242 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:21:22.242 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:22.243 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:21:22.275 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:22.275 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:21:22.345 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:22.345 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:21:22.416 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:22.416 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:21:22.485 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:22.486 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:21:22.554 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:22.554 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:21:22.622 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:22.622 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:21:22.690 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:22.690 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:21:22.722 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:22.722 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:21:22.791 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:22.792 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:21:22.863 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:22.864 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:21:22.932 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:22.932 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:21:23.000 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:23.001 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:21:23.071 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:23.071 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:21:23.134 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:23.135 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:21:23.153 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:23.153 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-06T07:21:23.210 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:23.211 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:21:23.226 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:23.226 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-06T07:21:23.281 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:21:23.282 DEBUG:teuthology.orchestra.run.smithi059:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:21:23.284 DEBUG:teuthology.orchestra.run.smithi164:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:21:23.325 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.y] 2024-03-06T07:21:23.325 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlAgqXEBAACCHZ9Lp2lEyF93xD4Ns9Nw== 2024-03-06T07:21:23.325 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-03-06T07:21:23.325 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlODBZGBAAqV+TH2t2ZKyV9l13QEn6ew== 2024-03-06T07:21:23.325 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlDXW2HxAAnKyuS1GW0wLO98/F9wdOAg== 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBXGehlzdinFhAA0ACDAs9dtoY6EhUIveeIvA== 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBaGehlH92eFRAANQceHGuQyrbI6gP1m/Gakw== 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBdGehlMWpVFBAAbOG9nScbFlvKHXZMQJoQug== 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.3] 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBgGehlLCyWEhAAyfoW/FmU37QD2IohknBwWw== 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.x] 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlLSTzFBAACeLdHsCX9KJGEsq44aqTyw== 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlR1MuJRAAp6OAQ9gZemWtwI1dh3H2SA== 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-03-06T07:21:23.326 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehl5MO4LBAAKmt2dvKk7BtcCM4o7cSwgg== 2024-03-06T07:21:23.327 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.4] 2024-03-06T07:21:23.327 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBmGehldqv5GhAAWEloqqKgdpKPHjt7VKFodw== 2024-03-06T07:21:23.327 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.5] 2024-03-06T07:21:23.327 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBpGehlc0ENGRAA5V9Az+vxAR3OUwWpDP7w1A== 2024-03-06T07:21:23.327 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.6] 2024-03-06T07:21:23.327 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBsGehl5SYEFxAAPLyrHQV9SGguCV9f5VIkYg== 2024-03-06T07:21:23.327 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.7] 2024-03-06T07:21:23.327 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBvGehle3v0FBAA4xUPwX5couppHgi9VwSi2g== 2024-03-06T07:21:23.327 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-06T07:21:23.328 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA== 2024-03-06T07:21:23.328 INFO:teuthology.orchestra.run.smithi059.stdout:[client.2] 2024-03-06T07:21:23.328 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehltL/GNhAAZqSQ+UtwjI/PA95O1RGBNg== 2024-03-06T07:21:23.328 INFO:teuthology.orchestra.run.smithi059.stdout:[client.1] 2024-03-06T07:21:23.328 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlqhlAOhAAQ4l3uI8MlcCH+NAIWo/UuA== 2024-03-06T07:21:23.328 INFO:teuthology.orchestra.run.smithi059.stdout:[client.3] 2024-03-06T07:21:23.328 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBUGehlltx5BBAAFa/+/w8Hd8lZegfIWtpJOA== 2024-03-06T07:21:23.347 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.y] 2024-03-06T07:21:23.347 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBTGehlAgqXEBAACCHZ9Lp2lEyF93xD4Ns9Nw== 2024-03-06T07:21:23.347 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2024-03-06T07:21:23.347 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBTGehlODBZGBAAqV+TH2t2ZKyV9l13QEn6ew== 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBTGehlDXW2HxAAnKyuS1GW0wLO98/F9wdOAg== 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBXGehlzdinFhAA0ACDAs9dtoY6EhUIveeIvA== 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBaGehlH92eFRAANQceHGuQyrbI6gP1m/Gakw== 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBdGehlMWpVFBAAbOG9nScbFlvKHXZMQJoQug== 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.3] 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBgGehlLCyWEhAAyfoW/FmU37QD2IohknBwWw== 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.x] 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBTGehlLSTzFBAACeLdHsCX9KJGEsq44aqTyw== 2024-03-06T07:21:23.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBTGehlR1MuJRAAp6OAQ9gZemWtwI1dh3H2SA== 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBTGehl5MO4LBAAKmt2dvKk7BtcCM4o7cSwgg== 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.4] 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBmGehldqv5GhAAWEloqqKgdpKPHjt7VKFodw== 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.5] 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBpGehlc0ENGRAA5V9Az+vxAR3OUwWpDP7w1A== 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.6] 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBsGehl5SYEFxAAPLyrHQV9SGguCV9f5VIkYg== 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.7] 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBvGehle3v0FBAA4xUPwX5couppHgi9VwSi2g== 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout:[client.0] 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA== 2024-03-06T07:21:23.349 INFO:teuthology.orchestra.run.smithi164.stdout:[client.2] 2024-03-06T07:21:23.350 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBTGehltL/GNhAAZqSQ+UtwjI/PA95O1RGBNg== 2024-03-06T07:21:23.350 INFO:teuthology.orchestra.run.smithi164.stdout:[client.1] 2024-03-06T07:21:23.350 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBTGehlqhlAOhAAQ4l3uI8MlcCH+NAIWo/UuA== 2024-03-06T07:21:23.350 INFO:teuthology.orchestra.run.smithi164.stdout:[client.3] 2024-03-06T07:21:23.350 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBUGehlltx5BBAAFa/+/w8Hd8lZegfIWtpJOA== 2024-03-06T07:21:23.351 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:23.372 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:23.444 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.446 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.502 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.505 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:23.564 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:23.615 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:23.618 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:23.669 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:23.672 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:23.724 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:23.726 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:23.779 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:23.781 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:23.833 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.835 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.889 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.892 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.946 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:23.948 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.000 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.002 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.054 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.056 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.108 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.110 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.163 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.218 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.221 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.275 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.278 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.333 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.335 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.390 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:21:24.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:21:24.416 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:24.512 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:21:24.539 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:21:24.567 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:24.663 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:21:24.690 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:21:24.757 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:24.853 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:21:24.880 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:24.883 DEBUG:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:24.935 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:21:24.935 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:21:24.936 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:21:24.938 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:21:24.938 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:21:24.938 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:21:24.978 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:21:24.978 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:21:24.978 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:21:24.980 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:21:24.980 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:21:24.980 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:21:24.980 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:21:24.983 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:21:24.983 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:21:24.983 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:21:24.985 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:21:24.985 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:21:24.985 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:21:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.084+0000 7ff20e84e640 1 Processor -- start 2024-03-06T07:21:25.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.084+0000 7ff20e84e640 1 -- start start 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff20e84e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff20e84e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff20e84e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff20e84e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff208071c50 con 0x7ff2080712d0 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff20e84e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20806caa0 con 0x7ff20815a870 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff20e84e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080b3340 con 0x7ff20815a490 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff2077fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff2077fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51166/0 (socket says 172.21.15.59:51166) 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff2077fe640 1 -- 172.21.15.59:0/2048910187 learned_addr learned my addr 172.21.15.59:0/2048910187 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff2077fe640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 msgr2=0x7ff208071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff2077fe640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 msgr2=0x7ff208071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.085+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.086+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.086+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 msgr2=0x7ff20815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:21:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.086+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 msgr2=0x7ff20815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.086+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.086+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.086+0000 7ff207fff640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 msgr2=0x7ff208072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:21:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.086+0000 7ff207fff640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 msgr2=0x7ff208072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.086+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.086+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.286+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.286+0000 7ff2077fe640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 msgr2=0x7ff208071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:21:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.286+0000 7ff2077fe640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 msgr2=0x7ff208071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.286+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.286+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.287+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.287+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.287+0000 7ff207fff640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 msgr2=0x7ff208072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.287+0000 7ff207fff640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 msgr2=0x7ff208072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.287+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.287+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.287+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 msgr2=0x7ff20815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.287+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 msgr2=0x7ff20815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.287+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.287+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:21:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.687+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.687+0000 7ff2077fe640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 msgr2=0x7ff208071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:21:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.687+0000 7ff2077fe640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 msgr2=0x7ff208071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.687+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.687+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-06T07:21:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.688+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.688+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.688+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 msgr2=0x7ff20815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:21:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.688+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 msgr2=0x7ff20815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.688+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.688+0000 7ff207fff640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 msgr2=0x7ff208072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:21:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.688+0000 7ff207fff640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 msgr2=0x7ff208072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.688+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.688+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-06T07:21:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:25.688+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-06T07:21:26.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.488+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.488+0000 7ff2077fe640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 msgr2=0x7ff208071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:21:26.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.488+0000 7ff2077fe640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 msgr2=0x7ff208071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:26.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.488+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:26.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.488+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-06T07:21:26.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.489+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.489+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 msgr2=0x7ff20815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:21:26.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.489+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 msgr2=0x7ff20815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:26.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.489+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:26.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.489+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-06T07:21:26.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.489+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.489+0000 7ff207fff640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 msgr2=0x7ff208072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:21:26.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.489+0000 7ff207fff640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 msgr2=0x7ff208072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:26.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.489+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:26.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:26.489+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-06T07:21:28.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:28.085+0000 7ff205ffb640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 msgr2=0x7ff208072530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:28.085+0000 7ff205ffb640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815a490 0x7ff208072530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:28.085+0000 7ff205ffb640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 msgr2=0x7ff20815ac80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:28.085+0000 7ff205ffb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.164:3300/0 conn(0x7ff20815a870 0x7ff20815ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:28.085+0000 7ff205ffb640 1 -- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 msgr2=0x7ff208071710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:28.085+0000 7ff205ffb640 1 --2- 172.21.15.59:0/2048910187 >> v2:172.21.15.59:3300/0 conn(0x7ff2080712d0 0x7ff208071710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:28.085+0000 7ff205ffb640 1 -- 172.21.15.59:0/2048910187 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2080b3620 con 0x7ff1f00058c0 2024-03-06T07:21:28.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:28.085+0000 7ff205ffb640 1 -- 172.21.15.59:0/2048910187 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f000e310 con 0x7ff1f00095c0 2024-03-06T07:21:30.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403559037 0 0) 0x7ff1f000e310 con 0x7ff1f00095c0 2024-03-06T07:21:30.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f4003520 con 0x7ff1f00095c0 2024-03-06T07:21:30.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1273137899 0 0) 0x7ff1f4003520 con 0x7ff1f00095c0 2024-03-06T07:21:30.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 >> v1:172.21.15.59:6789/0 conn(0x7ff1f00058c0 legacy=0x7ff1f0007d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f000f550 con 0x7ff1f00095c0 2024-03-06T07:21:30.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7ff1f800b040 con 0x7ff1f00095c0 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 >> v1:172.21.15.164:6789/0 conn(0x7ff1f00095c0 legacy=0x7ff1f000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 reap_dead start 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1f4003a50 0x7ff1f4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1f4004800 0x7ff1f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f40099b0 0x7ff1f400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f400c390 con 0x7ff1f40099b0 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f400c540 con 0x7ff1f4003a50 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f400c740 con 0x7ff1f4004800 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.302+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f40099b0 0x7ff1f400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.303+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1f4003a50 0x7ff1f4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.303+0000 7ff2077fe640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1f4004800 0x7ff1f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.303+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1f4004800 msgr2=0x7ff1f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.303+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1f4004800 0x7ff1f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.303+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1f4003a50 msgr2=0x7ff1f4003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.303+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1f4003a50 0x7ff1f4003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.303+0000 7ff20cdeb640 1 -- 172.21.15.59:0/2048910187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4003520 con 0x7ff1f40099b0 2024-03-06T07:21:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.303+0000 7ff207fff640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1f4003a50 0x7ff1f4003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.304+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f40099b0 0x7ff1f400bda0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff2080d0400 tx=0x7ff1f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77c6651d254a52b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.304+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20003a070 con 0x7ff1f40099b0 2024-03-06T07:21:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.304+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff200035410 con 0x7ff1f40099b0 2024-03-06T07:21:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.304+0000 7ff206ffd640 1 -- 172.21.15.59:0/2048910187 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200003dd0 con 0x7ff1f40099b0 2024-03-06T07:21:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.305+0000 7ff20e84e640 1 -- 172.21.15.59:0/2048910187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f40099b0 msgr2=0x7ff1f400bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.305+0000 7ff20e84e640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f40099b0 0x7ff1f400bda0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff2080d0400 tx=0x7ff1f4005830 comp rx=0 tx=0).stop 2024-03-06T07:21:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.305+0000 7ff20e84e640 1 -- 172.21.15.59:0/2048910187 shutdown_connections 2024-03-06T07:21:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.305+0000 7ff20e84e640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f40099b0 0x7ff1f400bda0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.305+0000 7ff20e84e640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1f4004800 0x7ff1f4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.305+0000 7ff20e84e640 1 --2- 172.21.15.59:0/2048910187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1f4003a50 0x7ff1f4003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.305+0000 7ff20e84e640 1 -- 172.21.15.59:0/2048910187 >> 172.21.15.59:0/2048910187 conn(0x7ff20806afb0 msgr2=0x7ff20806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.305+0000 7ff20e84e640 1 -- 172.21.15.59:0/2048910187 shutdown_connections 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.305+0000 7ff20e84e640 1 -- 172.21.15.59:0/2048910187 wait complete. 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.306+0000 7ff20e84e640 1 Processor -- start 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.307+0000 7ff20e84e640 1 -- start start 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.307+0000 7ff20e84e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f4003a50 0x7ff2081442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.307+0000 7ff20e84e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1f4004800 0x7ff2081563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.307+0000 7ff20e84e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1f40099b0 0x7ff2081568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.307+0000 7ff20e84e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080b3a80 con 0x7ff1f4003a50 2024-03-06T07:21:30.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.307+0000 7ff20e84e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080b3900 con 0x7ff1f4004800 2024-03-06T07:21:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.307+0000 7ff20e84e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff208071fb0 con 0x7ff1f40099b0 2024-03-06T07:21:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.307+0000 7ff207fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f4003a50 0x7ff2081442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff207fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f4003a50 0x7ff2081442c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51238/0 (socket says 172.21.15.59:51238) 2024-03-06T07:21:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff207fff640 1 -- 172.21.15.59:0/994690489 learned_addr learned my addr 172.21.15.59:0/994690489 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff20cdeb640 1 --2- 172.21.15.59:0/994690489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1f40099b0 0x7ff2081568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff2077fe640 1 --2- 172.21.15.59:0/994690489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1f4004800 0x7ff2081563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff207fff640 1 -- 172.21.15.59:0/994690489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1f40099b0 msgr2=0x7ff2081568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff207fff640 1 --2- 172.21.15.59:0/994690489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1f40099b0 0x7ff2081568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff207fff640 1 -- 172.21.15.59:0/994690489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1f4004800 msgr2=0x7ff2081563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff207fff640 1 --2- 172.21.15.59:0/994690489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1f4004800 0x7ff2081563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff207fff640 1 -- 172.21.15.59:0/994690489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208144800 con 0x7ff1f4003a50 2024-03-06T07:21:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff207fff640 1 --2- 172.21.15.59:0/994690489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f4003a50 0x7ff2081442c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc003690 tx=0x7ff1fc003b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff204ff9640 1 -- 172.21.15.59:0/994690489 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc003e60 con 0x7ff1f4003a50 2024-03-06T07:21:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff204ff9640 1 -- 172.21.15.59:0/994690489 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc0169b0 con 0x7ff1f4003a50 2024-03-06T07:21:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff204ff9640 1 -- 172.21.15.59:0/994690489 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc016de0 con 0x7ff1f4003a50 2024-03-06T07:21:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff20e84e640 1 -- 172.21.15.59:0/994690489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2080ce250 con 0x7ff1f4003a50 2024-03-06T07:21:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.308+0000 7ff20e84e640 1 -- 172.21.15.59:0/994690489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2080ce810 con 0x7ff1f4003a50 2024-03-06T07:21:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.309+0000 7ff204ff9640 1 -- 172.21.15.59:0/994690489 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff1fc006020 con 0x7ff1f4003a50 2024-03-06T07:21:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.309+0000 7ff204ff9640 1 -- 172.21.15.59:0/994690489 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff1fc0179d0 con 0x7ff1f4003a50 2024-03-06T07:21:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.309+0000 7ff20e84e640 1 -- 172.21.15.59:0/994690489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20804f330 con 0x7ff1f4003a50 2024-03-06T07:21:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.312+0000 7ff204ff9640 1 -- 172.21.15.59:0/994690489 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff1fc017c20 con 0x7ff1f4003a50 2024-03-06T07:21:30.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.380+0000 7ff20e84e640 1 -- 172.21.15.59:0/994690489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff20806e0c0 con 0x7ff1f4003a50 2024-03-06T07:21:30.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.381+0000 7ff204ff9640 1 -- 172.21.15.59:0/994690489 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff1fc017e00 con 0x7ff1f4003a50 2024-03-06T07:21:30.383 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2024-03-06T07:21:30.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.382+0000 7ff20e84e640 1 -- 172.21.15.59:0/994690489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f4003a50 msgr2=0x7ff2081442c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:30.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.382+0000 7ff20e84e640 1 --2- 172.21.15.59:0/994690489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f4003a50 0x7ff2081442c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc003690 tx=0x7ff1fc003b50 comp rx=0 tx=0).stop 2024-03-06T07:21:30.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.382+0000 7ff20e84e640 1 -- 172.21.15.59:0/994690489 shutdown_connections 2024-03-06T07:21:30.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.382+0000 7ff20e84e640 1 --2- 172.21.15.59:0/994690489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1f40099b0 0x7ff2081568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.382+0000 7ff20e84e640 1 --2- 172.21.15.59:0/994690489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1f4004800 0x7ff2081563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.382+0000 7ff20e84e640 1 --2- 172.21.15.59:0/994690489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1f4003a50 0x7ff2081442c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:30.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.382+0000 7ff20e84e640 1 -- 172.21.15.59:0/994690489 >> 172.21.15.59:0/994690489 conn(0x7ff20806afb0 msgr2=0x7ff2080cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:30.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.382+0000 7ff20e84e640 1 -- 172.21.15.59:0/994690489 shutdown_connections 2024-03-06T07:21:30.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:30.382+0000 7ff20e84e640 1 -- 172.21.15.59:0/994690489 wait complete. 2024-03-06T07:21:30.396 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:21:30.396 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:21:30.396 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:30.396 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:21:30.423 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:30.424 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:21:30.490 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:30.490 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:21:30.557 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:21:30.558 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:21:30.625 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:30.626 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:21:30.658 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:30.658 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:21:30.729 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:30.729 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:21:30.799 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:21:30.799 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:21:30.868 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 5677b231-bdb9-45ae-8058-3b283377d567 0 2024-03-06T07:21:31.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.011+0000 7f68bfe33640 1 Processor -- start 2024-03-06T07:21:31.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.011+0000 7f68bfe33640 1 -- start start 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bfe33640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b815a220 0x7f68b80cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bfe33640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f68b80712d0 0x7f68b8071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bfe33640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b80b3320 con 0x7f68b80cfa90 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bfe33640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b80cf610 con 0x7f68b80712d0 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bfe33640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8071c70 con 0x7f68b815a220 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bdbcf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b815a220 0x7f68b80cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bdbcf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b815a220 0x7f68b80cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59846/0 (socket says 172.21.15.164:59846) 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bdbcf640 1 -- 172.21.15.164:0/3584051932 learned_addr learned my addr 172.21.15.164:0/3584051932 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bd3ce640 1 --2- 172.21.15.164:0/3584051932 >> v2:172.21.15.59:3300/0 conn(0x7f68b80712d0 0x7f68b8071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bdbcf640 1 -- 172.21.15.164:0/3584051932 >> v2:172.21.15.59:3300/0 conn(0x7f68b80712d0 msgr2=0x7f68b8071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bdbcf640 1 --2- 172.21.15.164:0/3584051932 >> v2:172.21.15.59:3300/0 conn(0x7f68b80712d0 0x7f68b8071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bdbcf640 1 -- 172.21.15.164:0/3584051932 >> v1:172.21.15.59:6789/0 conn(0x7f68b80cfa90 legacy=0x7f68b815de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.012+0000 7f68bdbcf640 1 -- 172.21.15.164:0/3584051932 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b80728d0 con 0x7f68b815a220 2024-03-06T07:21:31.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.013+0000 7f68bdbcf640 1 --2- 172.21.15.164:0/3584051932 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b815a220 0x7f68b80cf0d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f68b4009870 tx=0x7f68b402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed59bd0007ab6799 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.013+0000 7f68bcbcd640 1 -- 172.21.15.164:0/3584051932 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4035020 con 0x7f68b815a220 2024-03-06T07:21:31.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.013+0000 7f68bcbcd640 1 -- 172.21.15.164:0/3584051932 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b4031640 con 0x7f68b815a220 2024-03-06T07:21:31.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.013+0000 7f68bcbcd640 1 -- 172.21.15.164:0/3584051932 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b402d040 con 0x7f68b815a220 2024-03-06T07:21:31.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.014+0000 7f68bfe33640 1 -- 172.21.15.164:0/3584051932 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b815a220 msgr2=0x7f68b80cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.014+0000 7f68bfe33640 1 --2- 172.21.15.164:0/3584051932 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b815a220 0x7f68b80cf0d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f68b4009870 tx=0x7f68b402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.014+0000 7f68bfe33640 1 -- 172.21.15.164:0/3584051932 shutdown_connections 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.014+0000 7f68bfe33640 1 --2- 172.21.15.164:0/3584051932 >> v2:172.21.15.59:3300/0 conn(0x7f68b80712d0 0x7f68b8071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.014+0000 7f68bfe33640 1 --2- 172.21.15.164:0/3584051932 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b815a220 0x7f68b80cf0d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.014+0000 7f68bfe33640 1 -- 172.21.15.164:0/3584051932 >> 172.21.15.164:0/3584051932 conn(0x7f68b806b190 msgr2=0x7f68b806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.014+0000 7f68bfe33640 1 -- 172.21.15.164:0/3584051932 shutdown_connections 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.014+0000 7f68bfe33640 1 -- 172.21.15.164:0/3584051932 wait complete. 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.015+0000 7f68bfe33640 1 Processor -- start 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.015+0000 7f68bfe33640 1 -- start start 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bfe33640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b80712d0 0x7f68b8156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bfe33640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68b80cfa90 0x7f68b81e5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bfe33640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68b815a220 0x7f68b814a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bfe33640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8072030 con 0x7f68b815a220 2024-03-06T07:21:31.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bfe33640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8072bb0 con 0x7f68b80cfa90 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bfe33640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8072530 con 0x7f68b80712d0 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bdbcf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b80712d0 0x7f68b8156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bdbcf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b80712d0 0x7f68b8156740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59848/0 (socket says 172.21.15.164:59848) 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bdbcf640 1 -- 172.21.15.164:0/3239503761 learned_addr learned my addr 172.21.15.164:0/3239503761 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bd3ce640 1 --2- 172.21.15.164:0/3239503761 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68b80cfa90 0x7f68b81e5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68be3d0640 1 --2- 172.21.15.164:0/3239503761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68b815a220 0x7f68b814a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bdbcf640 1 -- 172.21.15.164:0/3239503761 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68b80cfa90 msgr2=0x7f68b81e5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bdbcf640 1 --2- 172.21.15.164:0/3239503761 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68b80cfa90 0x7f68b81e5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bdbcf640 1 -- 172.21.15.164:0/3239503761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68b815a220 msgr2=0x7f68b814a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bdbcf640 1 --2- 172.21.15.164:0/3239503761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68b815a220 0x7f68b814a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bdbcf640 1 -- 172.21.15.164:0/3239503761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b81e5810 con 0x7f68b80712d0 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68be3d0640 1 --2- 172.21.15.164:0/3239503761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68b815a220 0x7f68b814a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:31.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bdbcf640 1 --2- 172.21.15.164:0/3239503761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b80712d0 0x7f68b8156740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68b4000c00 tx=0x7f68b40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68ae7fc640 1 -- 172.21.15.164:0/3239503761 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4040070 con 0x7f68b80712d0 2024-03-06T07:21:31.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68ae7fc640 1 -- 172.21.15.164:0/3239503761 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b40399f0 con 0x7f68b80712d0 2024-03-06T07:21:31.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bfe33640 1 -- 172.21.15.164:0/3239503761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b81e5af0 con 0x7f68b80712d0 2024-03-06T07:21:31.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68ae7fc640 1 -- 172.21.15.164:0/3239503761 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4035040 con 0x7f68b80712d0 2024-03-06T07:21:31.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68bfe33640 1 -- 172.21.15.164:0/3239503761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b81e60b0 con 0x7f68b80712d0 2024-03-06T07:21:31.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68ae7fc640 1 -- 172.21.15.164:0/3239503761 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f68b404b060 con 0x7f68b80712d0 2024-03-06T07:21:31.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.016+0000 7f68ae7fc640 1 -- 172.21.15.164:0/3239503761 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f68b403a420 con 0x7f68b80712d0 2024-03-06T07:21:31.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.017+0000 7f68bfe33640 1 -- 172.21.15.164:0/3239503761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b81e7f80 con 0x7f68b80712d0 2024-03-06T07:21:31.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.019+0000 7f68ae7fc640 1 -- 172.21.15.164:0/3239503761 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f68b81e7f80 con 0x7f68b80712d0 2024-03-06T07:21:31.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.092+0000 7f68bfe33640 1 -- 172.21.15.164:0/3239503761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5677b231-bdb9-45ae-8058-3b283377d567", "id": 0} v 0) v1 -- 0x7f68b80732b0 con 0x7f68b80712d0 2024-03-06T07:21:31.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.095+0000 7f68ae7fc640 1 -- 172.21.15.164:0/3239503761 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5677b231-bdb9-45ae-8058-3b283377d567", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f68b404d970 con 0x7f68b80712d0 2024-03-06T07:21:31.097 INFO:teuthology.orchestra.run.smithi164.stdout:0 2024-03-06T07:21:31.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.096+0000 7f68bfe33640 1 -- 172.21.15.164:0/3239503761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b80712d0 msgr2=0x7f68b8156740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.096+0000 7f68bfe33640 1 --2- 172.21.15.164:0/3239503761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b80712d0 0x7f68b8156740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68b4000c00 tx=0x7f68b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.096+0000 7f68bfe33640 1 -- 172.21.15.164:0/3239503761 shutdown_connections 2024-03-06T07:21:31.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.096+0000 7f68bfe33640 1 --2- 172.21.15.164:0/3239503761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68b815a220 0x7f68b814a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.096+0000 7f68bfe33640 1 --2- 172.21.15.164:0/3239503761 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68b80cfa90 0x7f68b81e5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.096+0000 7f68bfe33640 1 --2- 172.21.15.164:0/3239503761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68b80712d0 0x7f68b8156740 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.096+0000 7f68bfe33640 1 -- 172.21.15.164:0/3239503761 >> 172.21.15.164:0/3239503761 conn(0x7f68b806b190 msgr2=0x7f68b8159350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.096+0000 7f68bfe33640 1 -- 172.21.15.164:0/3239503761 shutdown_connections 2024-03-06T07:21:31.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.096+0000 7f68bfe33640 1 -- 172.21.15.164:0/3239503761 wait complete. 2024-03-06T07:21:31.108 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new e04c5ccb-765c-4542-b113-8960735735bd 1 2024-03-06T07:21:31.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.209+0000 7f4de200e640 1 Processor -- start 2024-03-06T07:21:31.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.209+0000 7f4de200e640 1 -- start start 2024-03-06T07:21:31.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4de200e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc1455d0 0x7f4ddc072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4de200e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4ddc06afb0 0x7f4ddc06b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4de200e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ddc1459b0 con 0x7f4ddc06ded0 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4de200e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ddc0cfdf0 con 0x7f4ddc06afb0 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4de200e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc06b930 con 0x7f4ddc1455d0 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4ddaffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4ddc06afb0 0x7f4ddc06b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4ddb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc1455d0 0x7f4ddc072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4ddaffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4ddc06afb0 0x7f4ddc06b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45396/0 (socket says 172.21.15.164:45396) 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4ddaffd640 1 -- 172.21.15.164:0/2271920485 learned_addr learned my addr 172.21.15.164:0/2271920485 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4ddb7fe640 1 -- 172.21.15.164:0/2271920485 >> v2:172.21.15.164:3300/0 conn(0x7f4ddc06afb0 msgr2=0x7f4ddc06b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4ddb7fe640 1 --2- 172.21.15.164:0/2271920485 >> v2:172.21.15.164:3300/0 conn(0x7f4ddc06afb0 0x7f4ddc06b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4ddb7fe640 1 -- 172.21.15.164:0/2271920485 >> v1:172.21.15.59:6789/0 conn(0x7f4ddc06ded0 legacy=0x7f4ddc06e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4ddb7fe640 1 -- 172.21.15.164:0/2271920485 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc0cf1e0 con 0x7f4ddc1455d0 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.210+0000 7f4ddaffd640 1 --2- 172.21.15.164:0/2271920485 >> v2:172.21.15.164:3300/0 conn(0x7f4ddc06afb0 0x7f4ddc06b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.211+0000 7f4ddb7fe640 1 --2- 172.21.15.164:0/2271920485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc1455d0 0x7f4ddc072480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0097a0 tx=0x7f4dcc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d0ca2215c9f683a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.211+0000 7f4dda7fc640 1 -- 172.21.15.164:0/2271920485 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc035020 con 0x7f4ddc1455d0 2024-03-06T07:21:31.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.211+0000 7f4dda7fc640 1 -- 172.21.15.164:0/2271920485 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc031810 con 0x7f4ddc1455d0 2024-03-06T07:21:31.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.211+0000 7f4dda7fc640 1 -- 172.21.15.164:0/2271920485 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc031be0 con 0x7f4ddc1455d0 2024-03-06T07:21:31.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.211+0000 7f4de200e640 1 -- 172.21.15.164:0/2271920485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc1455d0 msgr2=0x7f4ddc072480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.211+0000 7f4de200e640 1 --2- 172.21.15.164:0/2271920485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc1455d0 0x7f4ddc072480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0097a0 tx=0x7f4dcc02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.211+0000 7f4de200e640 1 -- 172.21.15.164:0/2271920485 shutdown_connections 2024-03-06T07:21:31.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.211+0000 7f4de200e640 1 --2- 172.21.15.164:0/2271920485 >> v2:172.21.15.164:3300/0 conn(0x7f4ddc06afb0 0x7f4ddc06b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.211+0000 7f4de200e640 1 --2- 172.21.15.164:0/2271920485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc1455d0 0x7f4ddc072480 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.212+0000 7f4de200e640 1 -- 172.21.15.164:0/2271920485 >> 172.21.15.164:0/2271920485 conn(0x7f4ddc0b2660 msgr2=0x7f4ddc0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.212+0000 7f4de200e640 1 -- 172.21.15.164:0/2271920485 shutdown_connections 2024-03-06T07:21:31.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.212+0000 7f4de200e640 1 -- 172.21.15.164:0/2271920485 wait complete. 2024-03-06T07:21:31.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.213+0000 7f4de200e640 1 Processor -- start 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.213+0000 7f4de200e640 1 -- start start 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4de200e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc06afb0 0x7f4ddc15b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4de200e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc06ded0 0x7f4ddc1391d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4de200e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ddc1455d0 0x7f4ddc13ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4de200e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0cfa80 con 0x7f4ddc06ded0 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4de200e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0cf7a0 con 0x7f4ddc1455d0 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4de200e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0729c0 con 0x7f4ddc06afb0 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4ddb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc06afb0 0x7f4ddc15b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4ddb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc06afb0 0x7f4ddc15b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59888/0 (socket says 172.21.15.164:59888) 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4ddbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ddc1455d0 0x7f4ddc13ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4ddb7fe640 1 -- 172.21.15.164:0/3727690857 learned_addr learned my addr 172.21.15.164:0/3727690857 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4ddaffd640 1 --2- 172.21.15.164:0/3727690857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc06ded0 0x7f4ddc1391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4ddb7fe640 1 -- 172.21.15.164:0/3727690857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ddc1455d0 msgr2=0x7f4ddc13ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4ddb7fe640 1 --2- 172.21.15.164:0/3727690857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ddc1455d0 0x7f4ddc13ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4ddb7fe640 1 -- 172.21.15.164:0/3727690857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc06ded0 msgr2=0x7f4ddc1391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4ddb7fe640 1 --2- 172.21.15.164:0/3727690857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc06ded0 0x7f4ddc1391d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4ddb7fe640 1 -- 172.21.15.164:0/3727690857 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc139b20 con 0x7f4ddc06afb0 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4ddb7fe640 1 --2- 172.21.15.164:0/3727690857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc06afb0 0x7f4ddc15b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc02bec0 tx=0x7f4dcc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4dc3fff640 1 -- 172.21.15.164:0/3727690857 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc041070 con 0x7f4ddc06afb0 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4dc3fff640 1 -- 172.21.15.164:0/3727690857 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc03ad00 con 0x7f4ddc06afb0 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4de200e640 1 -- 172.21.15.164:0/3727690857 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ddc139e00 con 0x7f4ddc06afb0 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4dc3fff640 1 -- 172.21.15.164:0/3727690857 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc035040 con 0x7f4ddc06afb0 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4de200e640 1 -- 172.21.15.164:0/3727690857 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ddc13a3c0 con 0x7f4ddc06afb0 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.214+0000 7f4dc3fff640 1 -- 172.21.15.164:0/3727690857 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4dcc04c060 con 0x7f4ddc06afb0 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.215+0000 7f4dc3fff640 1 -- 172.21.15.164:0/3727690857 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f4dcc04e800 con 0x7f4ddc06afb0 2024-03-06T07:21:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.215+0000 7f4de200e640 1 -- 172.21.15.164:0/3727690857 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ddc073390 con 0x7f4ddc06afb0 2024-03-06T07:21:31.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.217+0000 7f4dc3fff640 1 -- 172.21.15.164:0/3727690857 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4dcc053020 con 0x7f4ddc06afb0 2024-03-06T07:21:31.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.290+0000 7f4de200e640 1 -- 172.21.15.164:0/3727690857 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e04c5ccb-765c-4542-b113-8960735735bd", "id": 1} v 0) v1 -- 0x7f4ddc068ff0 con 0x7f4ddc06afb0 2024-03-06T07:21:31.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.294+0000 7f4dc3fff640 1 -- 172.21.15.164:0/3727690857 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e04c5ccb-765c-4542-b113-8960735735bd", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4dcc03bd70 con 0x7f4ddc06afb0 2024-03-06T07:21:31.296 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-03-06T07:21:31.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.295+0000 7f4de200e640 1 -- 172.21.15.164:0/3727690857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc06afb0 msgr2=0x7f4ddc15b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.295+0000 7f4de200e640 1 --2- 172.21.15.164:0/3727690857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc06afb0 0x7f4ddc15b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc02bec0 tx=0x7f4dcc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.295+0000 7f4de200e640 1 -- 172.21.15.164:0/3727690857 shutdown_connections 2024-03-06T07:21:31.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.295+0000 7f4de200e640 1 --2- 172.21.15.164:0/3727690857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ddc1455d0 0x7f4ddc13ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.295+0000 7f4de200e640 1 --2- 172.21.15.164:0/3727690857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ddc06ded0 0x7f4ddc1391d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.295+0000 7f4de200e640 1 --2- 172.21.15.164:0/3727690857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ddc06afb0 0x7f4ddc15b170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.295+0000 7f4de200e640 1 -- 172.21.15.164:0/3727690857 >> 172.21.15.164:0/3727690857 conn(0x7f4ddc0b2660 msgr2=0x7f4ddc0b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.295+0000 7f4de200e640 1 -- 172.21.15.164:0/3727690857 shutdown_connections 2024-03-06T07:21:31.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.295+0000 7f4de200e640 1 -- 172.21.15.164:0/3727690857 wait complete. 2024-03-06T07:21:31.308 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 9cc1541e-9425-46a4-b873-738ca8ce77ac 2 2024-03-06T07:21:31.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.407+0000 7f95d7c29640 1 Processor -- start 2024-03-06T07:21:31.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.408+0000 7f95d7c29640 1 -- start start 2024-03-06T07:21:31.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.408+0000 7f95d7c29640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f95d01614d0 0x7f95d00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.408+0000 7f95d7c29640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d00d24b0 0x7f95d00b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.408+0000 7f95d7c29640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d01618b0 con 0x7f95d00d1910 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.408+0000 7f95d7c29640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95d00cf400 con 0x7f95d01614d0 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.408+0000 7f95d7c29640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d00cf6a0 con 0x7f95d00d24b0 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.409+0000 7f95d59c5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f95d01614d0 0x7f95d00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.409+0000 7f95d61c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d00d24b0 0x7f95d00b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.409+0000 7f95d59c5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f95d01614d0 0x7f95d00d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45412/0 (socket says 172.21.15.164:45412) 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.409+0000 7f95d59c5640 1 -- 172.21.15.164:0/3929928245 learned_addr learned my addr 172.21.15.164:0/3929928245 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.409+0000 7f95d59c5640 1 -- 172.21.15.164:0/3929928245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d00d24b0 msgr2=0x7f95d00b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.409+0000 7f95d59c5640 1 --2- 172.21.15.164:0/3929928245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d00d24b0 0x7f95d00b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.409+0000 7f95d59c5640 1 -- 172.21.15.164:0/3929928245 >> v1:172.21.15.164:6789/0 conn(0x7f95d00d1910 legacy=0x7f95d00d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.409+0000 7f95d59c5640 1 -- 172.21.15.164:0/3929928245 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d00cf980 con 0x7f95d01614d0 2024-03-06T07:21:31.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.409+0000 7f95d61c6640 1 --2- 172.21.15.164:0/3929928245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d00d24b0 0x7f95d00b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:31.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.410+0000 7f95d59c5640 1 --2- 172.21.15.164:0/3929928245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d01614d0 0x7f95d00d4860 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009e50 tx=0x7f95c802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6980d026e9c1d4bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.410+0000 7f95d49c3640 1 -- 172.21.15.164:0/3929928245 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c802cb00 con 0x7f95d01614d0 2024-03-06T07:21:31.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.410+0000 7f95d49c3640 1 -- 172.21.15.164:0/3929928245 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c802cca0 con 0x7f95d01614d0 2024-03-06T07:21:31.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.410+0000 7f95d49c3640 1 -- 172.21.15.164:0/3929928245 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c8034610 con 0x7f95d01614d0 2024-03-06T07:21:31.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.410+0000 7f95d7c29640 1 -- 172.21.15.164:0/3929928245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d01614d0 msgr2=0x7f95d00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.410+0000 7f95d7c29640 1 --2- 172.21.15.164:0/3929928245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d01614d0 0x7f95d00d4860 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009e50 tx=0x7f95c802bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:31.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.410+0000 7f95d7c29640 1 -- 172.21.15.164:0/3929928245 shutdown_connections 2024-03-06T07:21:31.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.410+0000 7f95d7c29640 1 --2- 172.21.15.164:0/3929928245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d00d24b0 0x7f95d00b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.410+0000 7f95d7c29640 1 --2- 172.21.15.164:0/3929928245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d01614d0 0x7f95d00d4860 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.410+0000 7f95d7c29640 1 -- 172.21.15.164:0/3929928245 >> 172.21.15.164:0/3929928245 conn(0x7f95d00d3d40 msgr2=0x7f95d00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.411+0000 7f95d7c29640 1 -- 172.21.15.164:0/3929928245 shutdown_connections 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.411+0000 7f95d7c29640 1 -- 172.21.15.164:0/3929928245 wait complete. 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.412+0000 7f95d7c29640 1 Processor -- start 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.412+0000 7f95d7c29640 1 -- start start 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.412+0000 7f95d7c29640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d00d1910 0x7f95d00c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.412+0000 7f95d7c29640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d00d24b0 0x7f95d00b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.412+0000 7f95d7c29640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d01614d0 0x7f95d01560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.412+0000 7f95d7c29640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d00b1370 con 0x7f95d00d1910 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.412+0000 7f95d7c29640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d00b0910 con 0x7f95d00d24b0 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.412+0000 7f95d7c29640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d00b14f0 con 0x7f95d01614d0 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d51c4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d00d24b0 0x7f95d00b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d61c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d01614d0 0x7f95d01560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d61c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d01614d0 0x7f95d01560d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59898/0 (socket says 172.21.15.164:59898) 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d51c4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d00d24b0 0x7f95d00b8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45418/0 (socket says 172.21.15.164:45418) 2024-03-06T07:21:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d61c6640 1 -- 172.21.15.164:0/3697416874 learned_addr learned my addr 172.21.15.164:0/3697416874 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d59c5640 1 --2- 172.21.15.164:0/3697416874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d00d1910 0x7f95d00c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d51c4640 1 -- 172.21.15.164:0/3697416874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d01614d0 msgr2=0x7f95d01560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d51c4640 1 --2- 172.21.15.164:0/3697416874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d01614d0 0x7f95d01560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d51c4640 1 -- 172.21.15.164:0/3697416874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d00d1910 msgr2=0x7f95d00c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d51c4640 1 --2- 172.21.15.164:0/3697416874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d00d1910 0x7f95d00c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d51c4640 1 -- 172.21.15.164:0/3697416874 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d00c8960 con 0x7f95d00d24b0 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d61c6640 1 --2- 172.21.15.164:0/3697416874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d01614d0 0x7f95d01560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d51c4640 1 --2- 172.21.15.164:0/3697416874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d00d24b0 0x7f95d00b8a60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f95c000c340 tx=0x7f95c000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95c67fc640 1 -- 172.21.15.164:0/3697416874 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c00052d0 con 0x7f95d00d24b0 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95c67fc640 1 -- 172.21.15.164:0/3697416874 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c0005490 con 0x7f95d00d24b0 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95c67fc640 1 -- 172.21.15.164:0/3697416874 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c0007260 con 0x7f95d00d24b0 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d7c29640 1 -- 172.21.15.164:0/3697416874 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d00c8c40 con 0x7f95d00d24b0 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95d7c29640 1 -- 172.21.15.164:0/3697416874 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d00c9200 con 0x7f95d00d24b0 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95c67fc640 1 -- 172.21.15.164:0/3697416874 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f95c0007400 con 0x7f95d00d24b0 2024-03-06T07:21:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.413+0000 7f95c67fc640 1 -- 172.21.15.164:0/3697416874 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f95c0011590 con 0x7f95d00d24b0 2024-03-06T07:21:31.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.414+0000 7f95d7c29640 1 -- 172.21.15.164:0/3697416874 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d0068ff0 con 0x7f95d00d24b0 2024-03-06T07:21:31.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.417+0000 7f95c67fc640 1 -- 172.21.15.164:0/3697416874 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f95c0003a50 con 0x7f95d00d24b0 2024-03-06T07:21:31.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.490+0000 7f95d7c29640 1 -- 172.21.15.164:0/3697416874 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9cc1541e-9425-46a4-b873-738ca8ce77ac", "id": 2} v 0) v1 -- 0x7f95d0160aa0 con 0x7f95d00d24b0 2024-03-06T07:21:31.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.495+0000 7f95c67fc640 1 -- 172.21.15.164:0/3697416874 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9cc1541e-9425-46a4-b873-738ca8ce77ac", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f95c0003610 con 0x7f95d00d24b0 2024-03-06T07:21:31.497 INFO:teuthology.orchestra.run.smithi164.stdout:2 2024-03-06T07:21:31.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.496+0000 7f95d7c29640 1 -- 172.21.15.164:0/3697416874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d00d24b0 msgr2=0x7f95d00b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.496+0000 7f95d7c29640 1 --2- 172.21.15.164:0/3697416874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d00d24b0 0x7f95d00b8a60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f95c000c340 tx=0x7f95c000c800 comp rx=0 tx=0).stop 2024-03-06T07:21:31.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.496+0000 7f95d7c29640 1 -- 172.21.15.164:0/3697416874 shutdown_connections 2024-03-06T07:21:31.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.496+0000 7f95d7c29640 1 --2- 172.21.15.164:0/3697416874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d01614d0 0x7f95d01560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.496+0000 7f95d7c29640 1 --2- 172.21.15.164:0/3697416874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d00d24b0 0x7f95d00b8a60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.496+0000 7f95d7c29640 1 --2- 172.21.15.164:0/3697416874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d00d1910 0x7f95d00c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.496+0000 7f95d7c29640 1 -- 172.21.15.164:0/3697416874 >> 172.21.15.164:0/3697416874 conn(0x7f95d00d3d40 msgr2=0x7f95d00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.496+0000 7f95d7c29640 1 -- 172.21.15.164:0/3697416874 shutdown_connections 2024-03-06T07:21:31.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.496+0000 7f95d7c29640 1 -- 172.21.15.164:0/3697416874 wait complete. 2024-03-06T07:21:31.508 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 0a46fed5-501b-48ba-8269-929a8b3337a6 3 2024-03-06T07:21:31.532 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:31.530+0000 7fce9cbbb040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:21:31.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:31.547+0000 7f716d933040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:21:31.587 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:31.586+0000 7fce9cbbb040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:21:31.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:31.604+0000 7f716d933040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:21:31.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.607+0000 7f7ef3615640 1 Processor -- start 2024-03-06T07:21:31.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.607+0000 7f7ef3615640 1 -- start start 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef3615640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7eec15f570 0x7f7eec0d0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef3615640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7eec0b3e30 0x7f7eec0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef3615640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec0d0ec0 con 0x7f7eec0b3120 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef3615640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eec0d0c90 con 0x7f7eec15f570 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef3615640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eec0cc750 con 0x7f7eec0b3e30 2024-03-06T07:21:31.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef1bb2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7eec0b3e30 0x7f7eec0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef1bb2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7eec0b3e30 0x7f7eec0ccbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45430/0 (socket says 172.21.15.164:45430) 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef1bb2640 1 -- 172.21.15.164:0/2574994807 learned_addr learned my addr 172.21.15.164:0/2574994807 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef13b1640 1 --2- 172.21.15.164:0/2574994807 >> v2:172.21.15.59:3300/0 conn(0x7f7eec15f570 0x7f7eec0d0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef1bb2640 1 -- 172.21.15.164:0/2574994807 >> v2:172.21.15.59:3300/0 conn(0x7f7eec15f570 msgr2=0x7f7eec0d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef1bb2640 1 --2- 172.21.15.164:0/2574994807 >> v2:172.21.15.59:3300/0 conn(0x7f7eec15f570 0x7f7eec0d0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.608+0000 7f7ef1bb2640 1 -- 172.21.15.164:0/2574994807 >> v1:172.21.15.59:6789/0 conn(0x7f7eec0b3120 legacy=0x7f7eec0b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.609+0000 7f7ef1bb2640 1 -- 172.21.15.164:0/2574994807 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec0d11a0 con 0x7f7eec0b3e30 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.609+0000 7f7ef1bb2640 1 --2- 172.21.15.164:0/2574994807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eec0b3e30 0x7f7eec0ccbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0c8af0 tx=0x7f7ee402bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5baf4e1135da9d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.609+0000 7f7ee3fff640 1 -- 172.21.15.164:0/2574994807 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee402cb20 con 0x7f7eec0b3e30 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.609+0000 7f7ee3fff640 1 -- 172.21.15.164:0/2574994807 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee402ccc0 con 0x7f7eec0b3e30 2024-03-06T07:21:31.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.609+0000 7f7ee3fff640 1 -- 172.21.15.164:0/2574994807 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4034610 con 0x7f7eec0b3e30 2024-03-06T07:21:31.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.610+0000 7f7ef3615640 1 -- 172.21.15.164:0/2574994807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eec0b3e30 msgr2=0x7f7eec0ccbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.610+0000 7f7ef3615640 1 --2- 172.21.15.164:0/2574994807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eec0b3e30 0x7f7eec0ccbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0c8af0 tx=0x7f7ee402bf80 comp rx=0 tx=0).stop 2024-03-06T07:21:31.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.610+0000 7f7ef3615640 1 -- 172.21.15.164:0/2574994807 shutdown_connections 2024-03-06T07:21:31.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.610+0000 7f7ef3615640 1 --2- 172.21.15.164:0/2574994807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eec0b3e30 0x7f7eec0ccbe0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.610+0000 7f7ef3615640 1 --2- 172.21.15.164:0/2574994807 >> v2:172.21.15.59:3300/0 conn(0x7f7eec15f570 0x7f7eec0d0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.610+0000 7f7ef3615640 1 -- 172.21.15.164:0/2574994807 >> 172.21.15.164:0/2574994807 conn(0x7f7eec0d1b40 msgr2=0x7f7eec0d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.610+0000 7f7ef3615640 1 -- 172.21.15.164:0/2574994807 shutdown_connections 2024-03-06T07:21:31.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.610+0000 7f7ef3615640 1 -- 172.21.15.164:0/2574994807 wait complete. 2024-03-06T07:21:31.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.611+0000 7f7ef3615640 1 Processor -- start 2024-03-06T07:21:31.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.611+0000 7f7ef3615640 1 -- start start 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef3615640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eec0b3120 0x7f7eec15d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef3615640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7eec0b3e30 0x7f7eec1e56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef3615640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec15f570 0x7f7eec1e5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef3615640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec0cd120 con 0x7f7eec15f570 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef3615640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec0cc3e0 con 0x7f7eec0b3120 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef3615640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec0cd2a0 con 0x7f7eec0b3e30 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef13b1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eec0b3120 0x7f7eec15d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef0bb0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7eec0b3e30 0x7f7eec1e56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef13b1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eec0b3120 0x7f7eec15d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45438/0 (socket says 172.21.15.164:45438) 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef13b1640 1 -- 172.21.15.164:0/3617572147 learned_addr learned my addr 172.21.15.164:0/3617572147 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef1bb2640 1 --2- 172.21.15.164:0/3617572147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec15f570 0x7f7eec1e5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef13b1640 1 -- 172.21.15.164:0/3617572147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7eec0b3e30 msgr2=0x7f7eec1e56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef13b1640 1 --2- 172.21.15.164:0/3617572147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7eec0b3e30 0x7f7eec1e56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef13b1640 1 -- 172.21.15.164:0/3617572147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec15f570 msgr2=0x7f7eec1e5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef13b1640 1 --2- 172.21.15.164:0/3617572147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec15f570 0x7f7eec1e5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef13b1640 1 -- 172.21.15.164:0/3617572147 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec0ca0f0 con 0x7f7eec0b3120 2024-03-06T07:21:31.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.612+0000 7f7ef0bb0640 1 --2- 172.21.15.164:0/3617572147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7eec0b3e30 0x7f7eec1e56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:31.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.613+0000 7f7ef13b1640 1 --2- 172.21.15.164:0/3617572147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eec0b3120 0x7f7eec15d2a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00e870 tx=0x7f7edc00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.613+0000 7f7ee1ffb640 1 -- 172.21.15.164:0/3617572147 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc017070 con 0x7f7eec0b3120 2024-03-06T07:21:31.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.613+0000 7f7ee1ffb640 1 -- 172.21.15.164:0/3617572147 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7edc00c940 con 0x7f7eec0b3120 2024-03-06T07:21:31.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.613+0000 7f7ef3615640 1 -- 172.21.15.164:0/3617572147 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eec0ca3d0 con 0x7f7eec0b3120 2024-03-06T07:21:31.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.613+0000 7f7ee1ffb640 1 -- 172.21.15.164:0/3617572147 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00cd30 con 0x7f7eec0b3120 2024-03-06T07:21:31.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.613+0000 7f7ef3615640 1 -- 172.21.15.164:0/3617572147 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eec0ca990 con 0x7f7eec0b3120 2024-03-06T07:21:31.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.613+0000 7f7ee1ffb640 1 -- 172.21.15.164:0/3617572147 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7edc01a070 con 0x7f7eec0b3120 2024-03-06T07:21:31.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.613+0000 7f7ee1ffb640 1 -- 172.21.15.164:0/3617572147 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f7edc01c8e0 con 0x7f7eec0b3120 2024-03-06T07:21:31.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.614+0000 7f7ef3615640 1 -- 172.21.15.164:0/3617572147 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eec09fdf0 con 0x7f7eec0b3120 2024-03-06T07:21:31.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.616+0000 7f7ee1ffb640 1 -- 172.21.15.164:0/3617572147 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7edc01cb70 con 0x7f7eec0b3120 2024-03-06T07:21:31.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.689+0000 7f7ef3615640 1 -- 172.21.15.164:0/3617572147 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0a46fed5-501b-48ba-8269-929a8b3337a6", "id": 3} v 0) v1 -- 0x7f7eec06f7c0 con 0x7f7eec0b3120 2024-03-06T07:21:31.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.695+0000 7f7ee1ffb640 1 -- 172.21.15.164:0/3617572147 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0a46fed5-501b-48ba-8269-929a8b3337a6", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7edc01ce10 con 0x7f7eec0b3120 2024-03-06T07:21:31.697 INFO:teuthology.orchestra.run.smithi164.stdout:3 2024-03-06T07:21:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.696+0000 7f7ef3615640 1 -- 172.21.15.164:0/3617572147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eec0b3120 msgr2=0x7f7eec15d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.696+0000 7f7ef3615640 1 --2- 172.21.15.164:0/3617572147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eec0b3120 0x7f7eec15d2a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00e870 tx=0x7f7edc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:21:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.696+0000 7f7ef3615640 1 -- 172.21.15.164:0/3617572147 shutdown_connections 2024-03-06T07:21:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.696+0000 7f7ef3615640 1 --2- 172.21.15.164:0/3617572147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7eec15f570 0x7f7eec1e5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.696+0000 7f7ef3615640 1 --2- 172.21.15.164:0/3617572147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7eec0b3e30 0x7f7eec1e56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.696+0000 7f7ef3615640 1 --2- 172.21.15.164:0/3617572147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eec0b3120 0x7f7eec15d2a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.696+0000 7f7ef3615640 1 -- 172.21.15.164:0/3617572147 >> 172.21.15.164:0/3617572147 conn(0x7f7eec0d1b40 msgr2=0x7f7eec06f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.696+0000 7f7ef3615640 1 -- 172.21.15.164:0/3617572147 shutdown_connections 2024-03-06T07:21:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.696+0000 7f7ef3615640 1 -- 172.21.15.164:0/3617572147 wait complete. 2024-03-06T07:21:31.708 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new d1c273be-2527-465c-bef8-be9289c8c326 4 2024-03-06T07:21:31.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.848+0000 7f7e1997a640 1 Processor -- start 2024-03-06T07:21:31.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.848+0000 7f7e1997a640 1 -- start start 2024-03-06T07:21:31.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.848+0000 7f7e1997a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e141614d0 0x7f7e140d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e1997a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7e140d1910 0x7f7e140d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e1997a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7e140d22b0 0x7f7e140d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e1997a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e140d2740 con 0x7f7e140d22b0 2024-03-06T07:21:31.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e1997a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e141618b0 con 0x7f7e140d1910 2024-03-06T07:21:31.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e1997a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140cf400 con 0x7f7e141614d0 2024-03-06T07:21:31.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e0bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7e140d1910 0x7f7e140d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e0bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7e140d1910 0x7f7e140d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45446/0 (socket says 172.21.15.164:45446) 2024-03-06T07:21:31.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e0bfff640 1 -- 172.21.15.164:0/3021084724 learned_addr learned my addr 172.21.15.164:0/3021084724 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:31.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e12ffd640 1 --2- 172.21.15.164:0/3021084724 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e141614d0 0x7f7e140d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e137fe640 1 --2- 172.21.15.164:0/3021084724 >> v2:172.21.15.59:3300/0 conn(0x7f7e140d22b0 0x7f7e140d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e0bfff640 1 -- 172.21.15.164:0/3021084724 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e141614d0 msgr2=0x7f7e140d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e0bfff640 1 --2- 172.21.15.164:0/3021084724 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e141614d0 0x7f7e140d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e0bfff640 1 -- 172.21.15.164:0/3021084724 >> v2:172.21.15.59:3300/0 conn(0x7f7e140d22b0 msgr2=0x7f7e140d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e0bfff640 1 --2- 172.21.15.164:0/3021084724 >> v2:172.21.15.59:3300/0 conn(0x7f7e140d22b0 0x7f7e140d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.849+0000 7f7e0bfff640 1 -- 172.21.15.164:0/3021084724 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e140cf6a0 con 0x7f7e140d1910 2024-03-06T07:21:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.850+0000 7f7e0bfff640 1 --2- 172.21.15.164:0/3021084724 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e140d1910 0x7f7e140d1d70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7e1408c4f0 tx=0x7f7dfc02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63e48ef45539dd3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.850+0000 7f7e127fc640 1 -- 172.21.15.164:0/3021084724 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc035020 con 0x7f7e140d1910 2024-03-06T07:21:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.850+0000 7f7e127fc640 1 -- 172.21.15.164:0/3021084724 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc031870 con 0x7f7e140d1910 2024-03-06T07:21:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.850+0000 7f7e127fc640 1 -- 172.21.15.164:0/3021084724 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc031c40 con 0x7f7e140d1910 2024-03-06T07:21:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.851+0000 7f7e1997a640 1 -- 172.21.15.164:0/3021084724 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e140d1910 msgr2=0x7f7e140d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.851+0000 7f7e1997a640 1 --2- 172.21.15.164:0/3021084724 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e140d1910 0x7f7e140d1d70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7e1408c4f0 tx=0x7f7dfc02b910 comp rx=0 tx=0).stop 2024-03-06T07:21:31.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.851+0000 7f7e1997a640 1 -- 172.21.15.164:0/3021084724 shutdown_connections 2024-03-06T07:21:31.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.851+0000 7f7e1997a640 1 --2- 172.21.15.164:0/3021084724 >> v2:172.21.15.59:3300/0 conn(0x7f7e140d22b0 0x7f7e140d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.851+0000 7f7e1997a640 1 --2- 172.21.15.164:0/3021084724 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e140d1910 0x7f7e140d1d70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.851+0000 7f7e1997a640 1 --2- 172.21.15.164:0/3021084724 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e141614d0 0x7f7e140d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.851+0000 7f7e1997a640 1 -- 172.21.15.164:0/3021084724 >> 172.21.15.164:0/3021084724 conn(0x7f7e140d3d40 msgr2=0x7f7e140d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.851+0000 7f7e1997a640 1 -- 172.21.15.164:0/3021084724 shutdown_connections 2024-03-06T07:21:31.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.851+0000 7f7e1997a640 1 -- 172.21.15.164:0/3021084724 wait complete. 2024-03-06T07:21:31.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.852+0000 7f7e1997a640 1 Processor -- start 2024-03-06T07:21:31.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.852+0000 7f7e1997a640 1 -- start start 2024-03-06T07:21:31.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.852+0000 7f7e1997a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140d1910 0x7f7e141f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.852+0000 7f7e1997a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e140c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.852+0000 7f7e1997a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e141614d0 0x7f7e140cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:31.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.852+0000 7f7e1997a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140b0910 con 0x7f7e140d1910 2024-03-06T07:21:31.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.852+0000 7f7e1997a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140cfc60 con 0x7f7e141614d0 2024-03-06T07:21:31.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.852+0000 7f7e1997a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140d0be0 con 0x7f7e140d22b0 2024-03-06T07:21:31.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e0bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e140c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e137fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e141614d0 0x7f7e140cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e0bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e140c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59914/0 (socket says 172.21.15.164:59914) 2024-03-06T07:21:31.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e0bfff640 1 -- 172.21.15.164:0/4176543215 learned_addr learned my addr 172.21.15.164:0/4176543215 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:31.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e12ffd640 1 --2- 172.21.15.164:0/4176543215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140d1910 0x7f7e141f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:31.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e0bfff640 1 -- 172.21.15.164:0/4176543215 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e141614d0 msgr2=0x7f7e140cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e0bfff640 1 --2- 172.21.15.164:0/4176543215 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e141614d0 0x7f7e140cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e0bfff640 1 -- 172.21.15.164:0/4176543215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140d1910 msgr2=0x7f7e141f7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e0bfff640 1 --2- 172.21.15.164:0/4176543215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140d1910 0x7f7e141f7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e0bfff640 1 -- 172.21.15.164:0/4176543215 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e140c8310 con 0x7f7e140d22b0 2024-03-06T07:21:31.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e137fe640 1 --2- 172.21.15.164:0/4176543215 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e141614d0 0x7f7e140cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:31.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e0bfff640 1 --2- 172.21.15.164:0/4176543215 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e140c7870 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc02b950 tx=0x7f7dfc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:31.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e10ff9640 1 -- 172.21.15.164:0/4176543215 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc040070 con 0x7f7e140d22b0 2024-03-06T07:21:31.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e10ff9640 1 -- 172.21.15.164:0/4176543215 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc031e10 con 0x7f7e140d22b0 2024-03-06T07:21:31.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e10ff9640 1 -- 172.21.15.164:0/4176543215 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc035040 con 0x7f7e140d22b0 2024-03-06T07:21:31.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e1997a640 1 -- 172.21.15.164:0/4176543215 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e140c85f0 con 0x7f7e140d22b0 2024-03-06T07:21:31.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e1997a640 1 -- 172.21.15.164:0/4176543215 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e140c8e90 con 0x7f7e140d22b0 2024-03-06T07:21:31.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e10ff9640 1 -- 172.21.15.164:0/4176543215 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7dfc04b060 con 0x7f7e140d22b0 2024-03-06T07:21:31.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.853+0000 7f7e10ff9640 1 -- 172.21.15.164:0/4176543215 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f7dfc03a980 con 0x7f7e140d22b0 2024-03-06T07:21:31.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.855+0000 7f7e1997a640 1 -- 172.21.15.164:0/4176543215 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e140c89e0 con 0x7f7e140d22b0 2024-03-06T07:21:31.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.857+0000 7f7e10ff9640 1 -- 172.21.15.164:0/4176543215 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7dfc03ac20 con 0x7f7e140d22b0 2024-03-06T07:21:31.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.929+0000 7f7e1997a640 1 -- 172.21.15.164:0/4176543215 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d1c273be-2527-465c-bef8-be9289c8c326", "id": 4} v 0) v1 -- 0x7f7e140c9090 con 0x7f7e140d22b0 2024-03-06T07:21:31.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.935+0000 7f7e10ff9640 1 -- 172.21.15.164:0/4176543215 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d1c273be-2527-465c-bef8-be9289c8c326", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7e140c9090 con 0x7f7e140d22b0 2024-03-06T07:21:31.937 INFO:teuthology.orchestra.run.smithi164.stdout:4 2024-03-06T07:21:31.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.936+0000 7f7e1997a640 1 -- 172.21.15.164:0/4176543215 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 msgr2=0x7f7e140c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:31.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.936+0000 7f7e1997a640 1 --2- 172.21.15.164:0/4176543215 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e140c7870 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc02b950 tx=0x7f7dfc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.936+0000 7f7e1997a640 1 -- 172.21.15.164:0/4176543215 shutdown_connections 2024-03-06T07:21:31.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.936+0000 7f7e1997a640 1 --2- 172.21.15.164:0/4176543215 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e141614d0 0x7f7e140cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.936+0000 7f7e1997a640 1 --2- 172.21.15.164:0/4176543215 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e140c7870 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.936+0000 7f7e1997a640 1 --2- 172.21.15.164:0/4176543215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140d1910 0x7f7e141f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:31.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.936+0000 7f7e1997a640 1 -- 172.21.15.164:0/4176543215 >> 172.21.15.164:0/4176543215 conn(0x7f7e140d3d40 msgr2=0x7f7e1406c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:31.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.936+0000 7f7e1997a640 1 -- 172.21.15.164:0/4176543215 shutdown_connections 2024-03-06T07:21:31.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:31.937+0000 7f7e1997a640 1 -- 172.21.15.164:0/4176543215 wait complete. 2024-03-06T07:21:31.952 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new e0eb5922-8339-4154-92be-772acae72ccb 5 2024-03-06T07:21:32.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.055+0000 7fc3be288640 1 Processor -- start 2024-03-06T07:21:32.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.055+0000 7fc3be288640 1 -- start start 2024-03-06T07:21:32.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3be288640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc3b815ad90 0x7fc3b80b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3be288640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b815b260 con 0x7fc3b8071a70 2024-03-06T07:21:32.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3be288640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b815b540 con 0x7fc3b80b4110 2024-03-06T07:21:32.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3be288640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b815c020 con 0x7fc3b815ad90 2024-03-06T07:21:32.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3bc825640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc3b80b4110 0x7fc3b815c210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:47936/0 (socket says 172.21.15.164:47936) 2024-03-06T07:21:32.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3bc825640 1 -- 172.21.15.164:0/1722387953 learned_addr learned my addr 172.21.15.164:0/1722387953 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-03-06T07:21:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3b77fe640 1 --2- 172.21.15.164:0/1722387953 >> v2:172.21.15.59:3300/0 conn(0x7fc3b815ad90 0x7fc3b80b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3b6ffd640 1 -- 172.21.15.164:0/1722387953 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580127811 0 0) 0x7fc3b815b540 con 0x7fc3b80b4110 2024-03-06T07:21:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3b6ffd640 1 -- 172.21.15.164:0/1722387953 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a0003520 con 0x7fc3b80b4110 2024-03-06T07:21:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3b77fe640 1 -- 172.21.15.164:0/1722387953 >> v1:172.21.15.164:6789/0 conn(0x7fc3b80b4110 legacy=0x7fc3b815c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3b77fe640 1 -- 172.21.15.164:0/1722387953 >> v1:172.21.15.59:6789/0 conn(0x7fc3b8071a70 legacy=0x7fc3b8071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.056+0000 7fc3b77fe640 1 -- 172.21.15.164:0/1722387953 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b815b820 con 0x7fc3b815ad90 2024-03-06T07:21:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.057+0000 7fc3b77fe640 1 --2- 172.21.15.164:0/1722387953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b815ad90 0x7fc3b80b3bd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc3a80097a0 tx=0x7fc3a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c4284497500dd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.057+0000 7fc3b6ffd640 1 -- 172.21.15.164:0/1722387953 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8035020 con 0x7fc3b815ad90 2024-03-06T07:21:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.057+0000 7fc3b6ffd640 1 -- 172.21.15.164:0/1722387953 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a8031830 con 0x7fc3b815ad90 2024-03-06T07:21:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.057+0000 7fc3b6ffd640 1 -- 172.21.15.164:0/1722387953 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8031c00 con 0x7fc3b815ad90 2024-03-06T07:21:32.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.058+0000 7fc3be288640 1 -- 172.21.15.164:0/1722387953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b815ad90 msgr2=0x7fc3b80b3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.058+0000 7fc3be288640 1 --2- 172.21.15.164:0/1722387953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b815ad90 0x7fc3b80b3bd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc3a80097a0 tx=0x7fc3a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.058+0000 7fc3be288640 1 -- 172.21.15.164:0/1722387953 shutdown_connections 2024-03-06T07:21:32.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.058+0000 7fc3be288640 1 --2- 172.21.15.164:0/1722387953 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b815ad90 0x7fc3b80b3bd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.058+0000 7fc3be288640 1 -- 172.21.15.164:0/1722387953 >> 172.21.15.164:0/1722387953 conn(0x7fc3b80b2660 msgr2=0x7fc3b80b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.058+0000 7fc3be288640 1 -- 172.21.15.164:0/1722387953 shutdown_connections 2024-03-06T07:21:32.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.058+0000 7fc3be288640 1 -- 172.21.15.164:0/1722387953 wait complete. 2024-03-06T07:21:32.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.059+0000 7fc3be288640 1 Processor -- start 2024-03-06T07:21:32.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.059+0000 7fc3be288640 1 -- start start 2024-03-06T07:21:32.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.060+0000 7fc3be288640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b8071a70 0x7fc3b81e6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.060+0000 7fc3be288640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b80b4110 0x7fc3b8148590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.060+0000 7fc3be288640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b815ad90 0x7fc3b80ced20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.060+0000 7fc3be288640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80b4700 con 0x7fc3b80b4110 2024-03-06T07:21:32.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.060+0000 7fc3be288640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80b4580 con 0x7fc3b815ad90 2024-03-06T07:21:32.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.060+0000 7fc3be288640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b815c920 con 0x7fc3b8071a70 2024-03-06T07:21:32.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.060+0000 7fc3b7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b8071a70 0x7fc3b81e6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.060+0000 7fc3b7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b8071a70 0x7fc3b81e6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59924/0 (socket says 172.21.15.164:59924) 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.060+0000 7fc3bc825640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b815ad90 0x7fc3b80ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.060+0000 7fc3b7fff640 1 -- 172.21.15.164:0/4090081856 learned_addr learned my addr 172.21.15.164:0/4090081856 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.060+0000 7fc3b77fe640 1 --2- 172.21.15.164:0/4090081856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b80b4110 0x7fc3b8148590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3bc825640 1 -- 172.21.15.164:0/4090081856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b8071a70 msgr2=0x7fc3b81e6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3bc825640 1 --2- 172.21.15.164:0/4090081856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b8071a70 0x7fc3b81e6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3bc825640 1 -- 172.21.15.164:0/4090081856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b80b4110 msgr2=0x7fc3b8148590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3bc825640 1 --2- 172.21.15.164:0/4090081856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b80b4110 0x7fc3b8148590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3bc825640 1 -- 172.21.15.164:0/4090081856 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b81e5bb0 con 0x7fc3b815ad90 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3b7fff640 1 --2- 172.21.15.164:0/4090081856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b8071a70 0x7fc3b81e6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3b77fe640 1 --2- 172.21.15.164:0/4090081856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b80b4110 0x7fc3b8148590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3bc825640 1 --2- 172.21.15.164:0/4090081856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b815ad90 0x7fc3b80ced20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000ca80 tx=0x7fc3b000cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3b4ff9640 1 -- 172.21.15.164:0/4090081856 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0017070 con 0x7fc3b815ad90 2024-03-06T07:21:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3b4ff9640 1 -- 172.21.15.164:0/4090081856 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b000ac30 con 0x7fc3b815ad90 2024-03-06T07:21:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3be288640 1 -- 172.21.15.164:0/4090081856 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b81e4b10 con 0x7fc3b815ad90 2024-03-06T07:21:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3b4ff9640 1 -- 172.21.15.164:0/4090081856 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0012600 con 0x7fc3b815ad90 2024-03-06T07:21:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3be288640 1 -- 172.21.15.164:0/4090081856 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b81e50d0 con 0x7fc3b815ad90 2024-03-06T07:21:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3b4ff9640 1 -- 172.21.15.164:0/4090081856 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc3b001a070 con 0x7fc3b815ad90 2024-03-06T07:21:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.061+0000 7fc3b4ff9640 1 -- 172.21.15.164:0/4090081856 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fc3b001cd30 con 0x7fc3b815ad90 2024-03-06T07:21:32.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.062+0000 7fc3be288640 1 -- 172.21.15.164:0/4090081856 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b8073390 con 0x7fc3b815ad90 2024-03-06T07:21:32.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.064+0000 7fc3b4ff9640 1 -- 172.21.15.164:0/4090081856 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc3b001c7b0 con 0x7fc3b815ad90 2024-03-06T07:21:32.128 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:32.126+0000 7fce9cbbb040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:21:32.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.137+0000 7fc3be288640 1 -- 172.21.15.164:0/4090081856 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e0eb5922-8339-4154-92be-772acae72ccb", "id": 5} v 0) v1 -- 0x7fc3b8159ba0 con 0x7fc3b815ad90 2024-03-06T07:21:32.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.143+0000 7fc3b4ff9640 1 -- 172.21.15.164:0/4090081856 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e0eb5922-8339-4154-92be-772acae72ccb", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc3b001ca60 con 0x7fc3b815ad90 2024-03-06T07:21:32.146 INFO:teuthology.orchestra.run.smithi164.stdout:5 2024-03-06T07:21:32.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.144+0000 7fc3be288640 1 -- 172.21.15.164:0/4090081856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b815ad90 msgr2=0x7fc3b80ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.144+0000 7fc3be288640 1 --2- 172.21.15.164:0/4090081856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b815ad90 0x7fc3b80ced20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000ca80 tx=0x7fc3b000cf40 comp rx=0 tx=0).stop 2024-03-06T07:21:32.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.144+0000 7fc3be288640 1 -- 172.21.15.164:0/4090081856 shutdown_connections 2024-03-06T07:21:32.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.144+0000 7fc3be288640 1 --2- 172.21.15.164:0/4090081856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b815ad90 0x7fc3b80ced20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.144+0000 7fc3be288640 1 --2- 172.21.15.164:0/4090081856 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b80b4110 0x7fc3b8148590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.144+0000 7fc3be288640 1 --2- 172.21.15.164:0/4090081856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b8071a70 0x7fc3b81e6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.144+0000 7fc3be288640 1 -- 172.21.15.164:0/4090081856 >> 172.21.15.164:0/4090081856 conn(0x7fc3b80b2660 msgr2=0x7fc3b8072060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.144+0000 7fc3be288640 1 -- 172.21.15.164:0/4090081856 shutdown_connections 2024-03-06T07:21:32.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.144+0000 7fc3be288640 1 -- 172.21.15.164:0/4090081856 wait complete. 2024-03-06T07:21:32.158 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a 6 2024-03-06T07:21:32.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:32.179+0000 7f716d933040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:21:32.184 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:32.182+0000 7fce9cbbb040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:21:32.230 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:32.228+0000 7fce9cbbb040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:21:32.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:32.236+0000 7f716d933040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:21:32.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.258+0000 7f9fac7d6640 1 Processor -- start 2024-03-06T07:21:32.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.258+0000 7f9fac7d6640 1 -- start start 2024-03-06T07:21:32.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.259+0000 7f9fac7d6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9fa415bdc0 0x7f9fa4066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.259+0000 7f9fac7d6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa406bc10 con 0x7f9fa415c1a0 2024-03-06T07:21:32.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.259+0000 7f9fac7d6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa406bef0 con 0x7f9fa40b2590 2024-03-06T07:21:32.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.259+0000 7f9fac7d6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fa4067240 con 0x7f9fa415bdc0 2024-03-06T07:21:32.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.259+0000 7f9faa572640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9fa415bdc0 0x7f9fa4066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.259+0000 7f9faa572640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9fa415bdc0 0x7f9fa4066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45478/0 (socket says 172.21.15.164:45478) 2024-03-06T07:21:32.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.260+0000 7f9faa572640 1 -- 172.21.15.164:0/3277091688 learned_addr learned my addr 172.21.15.164:0/3277091688 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:32.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.260+0000 7f9faa572640 1 -- 172.21.15.164:0/3277091688 >> v1:172.21.15.164:6789/0 conn(0x7f9fa40b2590 legacy=0x7f9fa415cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.260+0000 7f9faa572640 1 -- 172.21.15.164:0/3277091688 >> v1:172.21.15.59:6789/0 conn(0x7f9fa415c1a0 legacy=0x7f9fa40b1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.260+0000 7f9faa572640 1 -- 172.21.15.164:0/3277091688 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa415c730 con 0x7f9fa415bdc0 2024-03-06T07:21:32.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.260+0000 7f9faa572640 1 --2- 172.21.15.164:0/3277091688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fa415bdc0 0x7f9fa4066d00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0009870 tx=0x7f9fa002bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba978a34fb89dee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.261+0000 7f9fa9570640 1 -- 172.21.15.164:0/3277091688 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa002d870 con 0x7f9fa415bdc0 2024-03-06T07:21:32.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.261+0000 7f9fa9570640 1 -- 172.21.15.164:0/3277091688 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa002da10 con 0x7f9fa415bdc0 2024-03-06T07:21:32.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.261+0000 7f9fa9570640 1 -- 172.21.15.164:0/3277091688 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa002dde0 con 0x7f9fa415bdc0 2024-03-06T07:21:32.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.261+0000 7f9fac7d6640 1 -- 172.21.15.164:0/3277091688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fa415bdc0 msgr2=0x7f9fa4066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.261+0000 7f9fac7d6640 1 --2- 172.21.15.164:0/3277091688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fa415bdc0 0x7f9fa4066d00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0009870 tx=0x7f9fa002bab0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.262+0000 7f9fac7d6640 1 -- 172.21.15.164:0/3277091688 shutdown_connections 2024-03-06T07:21:32.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.262+0000 7f9fac7d6640 1 --2- 172.21.15.164:0/3277091688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fa415bdc0 0x7f9fa4066d00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.262+0000 7f9fac7d6640 1 -- 172.21.15.164:0/3277091688 >> 172.21.15.164:0/3277091688 conn(0x7f9fa40b4fe0 msgr2=0x7f9fa40b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.262+0000 7f9fac7d6640 1 -- 172.21.15.164:0/3277091688 shutdown_connections 2024-03-06T07:21:32.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.262+0000 7f9fac7d6640 1 -- 172.21.15.164:0/3277091688 wait complete. 2024-03-06T07:21:32.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.263+0000 7f9fac7d6640 1 Processor -- start 2024-03-06T07:21:32.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.264+0000 7f9fac7d6640 1 -- start start 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9fac7d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fa40b2590 0x7f9fa4158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9fac7d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fa415bdc0 0x7f9fa40999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9fac7d6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fa415c1a0 0x7f9fa41563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9fac7d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa406b8a0 con 0x7f9fa40b2590 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9fac7d6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa406c1d0 con 0x7f9fa415c1a0 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9fac7d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa40b5f70 con 0x7f9fa415bdc0 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9fa9d71640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fa415bdc0 0x7f9fa40999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faad73640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fa415c1a0 0x7f9fa41563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faa572640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fa40b2590 0x7f9fa4158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faad73640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fa415c1a0 0x7f9fa41563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45482/0 (socket says 172.21.15.164:45482) 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faa572640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fa40b2590 0x7f9fa4158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.164:49134/0 (socket says 172.21.15.164:49134) 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faad73640 1 -- 172.21.15.164:0/3092001656 learned_addr learned my addr 172.21.15.164:0/3092001656 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:32.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9fa9d71640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fa415bdc0 0x7f9fa40999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59932/0 (socket says 172.21.15.164:59932) 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faad73640 1 -- 172.21.15.164:0/3092001656 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fa415bdc0 msgr2=0x7f9fa40999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faad73640 1 --2- 172.21.15.164:0/3092001656 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fa415bdc0 0x7f9fa40999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faad73640 1 -- 172.21.15.164:0/3092001656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fa40b2590 msgr2=0x7f9fa4158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faad73640 1 --2- 172.21.15.164:0/3092001656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fa40b2590 0x7f9fa4158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faad73640 1 -- 172.21.15.164:0/3092001656 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa41e4ae0 con 0x7f9fa415c1a0 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faa572640 1 --2- 172.21.15.164:0/3092001656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fa40b2590 0x7f9fa4158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9faad73640 1 --2- 172.21.15.164:0/3092001656 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fa415c1a0 0x7f9fa41563f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00c6e0 tx=0x7f9f9c00cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9f9affd640 1 -- 172.21.15.164:0/3092001656 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c003b70 con 0x7f9fa415c1a0 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9f9affd640 1 -- 172.21.15.164:0/3092001656 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9c003d30 con 0x7f9fa415c1a0 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9f9affd640 1 -- 172.21.15.164:0/3092001656 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c005310 con 0x7f9fa415c1a0 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.265+0000 7f9fac7d6640 1 -- 172.21.15.164:0/3092001656 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa41e4dc0 con 0x7f9fa415c1a0 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.266+0000 7f9fac7d6640 1 -- 172.21.15.164:0/3092001656 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa41e5380 con 0x7f9fa415c1a0 2024-03-06T07:21:32.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.266+0000 7f9f9affd640 1 -- 172.21.15.164:0/3092001656 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9f9c0054b0 con 0x7f9fa415c1a0 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.266+0000 7f9f9affd640 1 -- 172.21.15.164:0/3092001656 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f9f9c012ab0 con 0x7f9fa415c1a0 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.267+0000 7f9f98ff9640 1 -- 172.21.15.164:0/3092001656 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f80005290 con 0x7f9fa415c1a0 2024-03-06T07:21:32.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.269+0000 7f9f9affd640 1 -- 172.21.15.164:0/3092001656 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9f9c012d90 con 0x7f9fa415c1a0 2024-03-06T07:21:32.285 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:32.284+0000 7fce9cbbb040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:21:32.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:32.286+0000 7f716d933040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:21:32.330 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:32.328+0000 7fce9cbbb040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:21:32.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:32.342+0000 7f716d933040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:21:32.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.343+0000 7f9f98ff9640 1 -- 172.21.15.164:0/3092001656 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a", "id": 6} v 0) v1 -- 0x7f9f80005580 con 0x7f9fa415c1a0 2024-03-06T07:21:32.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.348+0000 7f9f9affd640 1 -- 172.21.15.164:0/3092001656 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9f9c0124c0 con 0x7f9fa415c1a0 2024-03-06T07:21:32.350 INFO:teuthology.orchestra.run.smithi164.stdout:6 2024-03-06T07:21:32.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.349+0000 7f9f98ff9640 1 -- 172.21.15.164:0/3092001656 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fa415c1a0 msgr2=0x7f9fa41563f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.349+0000 7f9f98ff9640 1 --2- 172.21.15.164:0/3092001656 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fa415c1a0 0x7f9fa41563f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00c6e0 tx=0x7f9f9c00cba0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.349+0000 7f9f98ff9640 1 -- 172.21.15.164:0/3092001656 shutdown_connections 2024-03-06T07:21:32.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.349+0000 7f9f98ff9640 1 --2- 172.21.15.164:0/3092001656 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fa415c1a0 0x7f9fa41563f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.349+0000 7f9f98ff9640 1 --2- 172.21.15.164:0/3092001656 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fa415bdc0 0x7f9fa40999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.349+0000 7f9f98ff9640 1 --2- 172.21.15.164:0/3092001656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fa40b2590 0x7f9fa4158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.349+0000 7f9f98ff9640 1 -- 172.21.15.164:0/3092001656 >> 172.21.15.164:0/3092001656 conn(0x7f9fa40b4fe0 msgr2=0x7f9fa4151b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.349+0000 7f9f98ff9640 1 -- 172.21.15.164:0/3092001656 shutdown_connections 2024-03-06T07:21:32.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.349+0000 7f9f98ff9640 1 -- 172.21.15.164:0/3092001656 wait complete. 2024-03-06T07:21:32.361 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 8f485029-f8fc-4491-a820-12cbbd3b8d44 7 2024-03-06T07:21:32.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:32.387+0000 7f716d933040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:21:32.449 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:32.447+0000 7fce9cbbb040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:21:32.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.487+0000 7fb25ff16640 1 Processor -- start 2024-03-06T07:21:32.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.488+0000 7fb25ff16640 1 -- start start 2024-03-06T07:21:32.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.488+0000 7fb25ff16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25815dd00 0x7fb25806e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.488+0000 7fb25ff16640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb258161bd0 con 0x7fb25815e0e0 2024-03-06T07:21:32.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.488+0000 7fb25ff16640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb258070220 con 0x7fb25806b6c0 2024-03-06T07:21:32.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.488+0000 7fb25ff16640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb258161e90 con 0x7fb25815dd00 2024-03-06T07:21:32.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.488+0000 7fb25dcb2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25815dd00 0x7fb25806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.489+0000 7fb25dcb2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25815dd00 0x7fb25806e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59936/0 (socket says 172.21.15.164:59936) 2024-03-06T07:21:32.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.489+0000 7fb25e4b3640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb25806b6c0 0x7fb2580b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:47946/0 (socket says 172.21.15.164:47946) 2024-03-06T07:21:32.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.489+0000 7fb25dcb2640 1 -- 172.21.15.164:0/1899644494 learned_addr learned my addr 172.21.15.164:0/1899644494 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:32.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.489+0000 7fb25dcb2640 1 -- 172.21.15.164:0/1899644494 >> v1:172.21.15.164:6789/0 conn(0x7fb25806b6c0 legacy=0x7fb2580b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.489+0000 7fb25dcb2640 1 -- 172.21.15.164:0/1899644494 >> v1:172.21.15.59:6789/0 conn(0x7fb25815e0e0 legacy=0x7fb25806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.489+0000 7fb25dcb2640 1 -- 172.21.15.164:0/1899644494 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb258162170 con 0x7fb25815dd00 2024-03-06T07:21:32.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.490+0000 7fb25dcb2640 1 --2- 172.21.15.164:0/1899644494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25815dd00 0x7fb25806e620 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb2540098a0 tx=0x7fb25402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6ff3fbf836b3b59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.490+0000 7fb25ccb0640 1 -- 172.21.15.164:0/1899644494 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25402f830 con 0x7fb25815dd00 2024-03-06T07:21:32.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.490+0000 7fb25ccb0640 1 -- 172.21.15.164:0/1899644494 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25402f9d0 con 0x7fb25815dd00 2024-03-06T07:21:32.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.490+0000 7fb25ccb0640 1 -- 172.21.15.164:0/1899644494 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25402fda0 con 0x7fb25815dd00 2024-03-06T07:21:32.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.490+0000 7fb25ff16640 1 -- 172.21.15.164:0/1899644494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25815dd00 msgr2=0x7fb25806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.490+0000 7fb25ff16640 1 --2- 172.21.15.164:0/1899644494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25815dd00 0x7fb25806e620 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb2540098a0 tx=0x7fb25402ba10 comp rx=0 tx=0).stop 2024-03-06T07:21:32.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.490+0000 7fb25ff16640 1 -- 172.21.15.164:0/1899644494 shutdown_connections 2024-03-06T07:21:32.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.490+0000 7fb25ff16640 1 --2- 172.21.15.164:0/1899644494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25815dd00 0x7fb25806e620 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.490+0000 7fb25ff16640 1 -- 172.21.15.164:0/1899644494 >> 172.21.15.164:0/1899644494 conn(0x7fb2580b2660 msgr2=0x7fb2580b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.490+0000 7fb25ff16640 1 -- 172.21.15.164:0/1899644494 shutdown_connections 2024-03-06T07:21:32.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.490+0000 7fb25ff16640 1 -- 172.21.15.164:0/1899644494 wait complete. 2024-03-06T07:21:32.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25ff16640 1 Processor -- start 2024-03-06T07:21:32.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25ff16640 1 -- start start 2024-03-06T07:21:32.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25ff16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25806b6c0 0x7fb25814cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25ff16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb25815dd00 0x7fb258153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25ff16640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2581e8520 0x7fb2581519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25ff16640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580b0180 con 0x7fb25815dd00 2024-03-06T07:21:32.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25ff16640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580b0000 con 0x7fb2581e8520 2024-03-06T07:21:32.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25ff16640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580b0300 con 0x7fb25806b6c0 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25dcb2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25806b6c0 0x7fb25814cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25dcb2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25806b6c0 0x7fb25814cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59952/0 (socket says 172.21.15.164:59952) 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25dcb2640 1 -- 172.21.15.164:0/3555489053 learned_addr learned my addr 172.21.15.164:0/3555489053 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25e4b3640 1 --2- 172.21.15.164:0/3555489053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2581e8520 0x7fb2581519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.492+0000 7fb25d4b1640 1 --2- 172.21.15.164:0/3555489053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb25815dd00 0x7fb258153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb25dcb2640 1 -- 172.21.15.164:0/3555489053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2581e8520 msgr2=0x7fb2581519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb25dcb2640 1 --2- 172.21.15.164:0/3555489053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2581e8520 0x7fb2581519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb25dcb2640 1 -- 172.21.15.164:0/3555489053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb25815dd00 msgr2=0x7fb258153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb25dcb2640 1 --2- 172.21.15.164:0/3555489053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb25815dd00 0x7fb258153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb25dcb2640 1 -- 172.21.15.164:0/3555489053 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2581542f0 con 0x7fb25806b6c0 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb25dcb2640 1 --2- 172.21.15.164:0/3555489053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25806b6c0 0x7fb25814cbf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb2540098a0 tx=0x7fb254004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb24e7fc640 1 -- 172.21.15.164:0/3555489053 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25402f7e0 con 0x7fb25806b6c0 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb24e7fc640 1 -- 172.21.15.164:0/3555489053 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb254037d40 con 0x7fb25806b6c0 2024-03-06T07:21:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb25ff16640 1 -- 172.21.15.164:0/3555489053 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25815f830 con 0x7fb25806b6c0 2024-03-06T07:21:32.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb24e7fc640 1 -- 172.21.15.164:0/3555489053 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254041640 con 0x7fb25806b6c0 2024-03-06T07:21:32.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb25ff16640 1 -- 172.21.15.164:0/3555489053 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25815fdf0 con 0x7fb25806b6c0 2024-03-06T07:21:32.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb24e7fc640 1 -- 172.21.15.164:0/3555489053 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb254048030 con 0x7fb25806b6c0 2024-03-06T07:21:32.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.493+0000 7fb24e7fc640 1 -- 172.21.15.164:0/3555489053 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fb25404ab10 con 0x7fb25806b6c0 2024-03-06T07:21:32.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.494+0000 7fb25ff16640 1 -- 172.21.15.164:0/3555489053 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb258073390 con 0x7fb25806b6c0 2024-03-06T07:21:32.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.496+0000 7fb24e7fc640 1 -- 172.21.15.164:0/3555489053 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb25404f060 con 0x7fb25806b6c0 2024-03-06T07:21:32.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:32.509+0000 7f716d933040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:21:32.521 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:32.520+0000 7fce9cbbb040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:32.569 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:32.567+0000 7fce9cbbb040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:21:32.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.567+0000 7fb25ff16640 1 -- 172.21.15.164:0/3555489053 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8f485029-f8fc-4491-a820-12cbbd3b8d44", "id": 7} v 0) v1 -- 0x7fb258068ff0 con 0x7fb25806b6c0 2024-03-06T07:21:32.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.575+0000 7fb24e7fc640 1 -- 172.21.15.164:0/3555489053 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f485029-f8fc-4491-a820-12cbbd3b8d44", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb254033030 con 0x7fb25806b6c0 2024-03-06T07:21:32.577 INFO:teuthology.orchestra.run.smithi164.stdout:7 2024-03-06T07:21:32.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.576+0000 7fb25ff16640 1 -- 172.21.15.164:0/3555489053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25806b6c0 msgr2=0x7fb25814cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.576+0000 7fb25ff16640 1 --2- 172.21.15.164:0/3555489053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25806b6c0 0x7fb25814cbf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb2540098a0 tx=0x7fb254004730 comp rx=0 tx=0).stop 2024-03-06T07:21:32.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.576+0000 7fb25ff16640 1 -- 172.21.15.164:0/3555489053 shutdown_connections 2024-03-06T07:21:32.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.576+0000 7fb25ff16640 1 --2- 172.21.15.164:0/3555489053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2581e8520 0x7fb2581519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.576+0000 7fb25ff16640 1 --2- 172.21.15.164:0/3555489053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb25815dd00 0x7fb258153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.576+0000 7fb25ff16640 1 --2- 172.21.15.164:0/3555489053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb25806b6c0 0x7fb25814cbf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.576+0000 7fb25ff16640 1 -- 172.21.15.164:0/3555489053 >> 172.21.15.164:0/3555489053 conn(0x7fb2580b2660 msgr2=0x7fb2580b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.576+0000 7fb25ff16640 1 -- 172.21.15.164:0/3555489053 shutdown_connections 2024-03-06T07:21:32.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:21:32.576+0000 7fb25ff16640 1 -- 172.21.15.164:0/3555489053 wait complete. 2024-03-06T07:21:32.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:32.580+0000 7f716d933040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:32.589 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:21:32.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:21:32.591 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:21:32.591 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:21:32.591 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:21:32.593 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:21:32.593 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:21:32.594 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:21:32.596 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:21:32.596 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:21:32.596 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:21:32.598 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:21:32.598 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:21:32.598 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:21:32.600 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:21:32.600 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:21:32.601 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:21:32.602 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:21:32.602 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:21:32.603 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:21:32.604 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:21:32.605 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:21:32.605 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:21:32.607 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:21:32.607 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:32.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:32.636+0000 7f716d933040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:21:32.639 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:32.638+0000 7fce9cbbb040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:21:32.708 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:32.706+0000 7fce9cbbb040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:21:32.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:32.706+0000 7f716d933040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:21:32.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.722+0000 7f1fe785c640 1 Processor -- start 2024-03-06T07:21:32.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.723+0000 7f1fe785c640 1 -- start start 2024-03-06T07:21:32.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.723+0000 7f1fe785c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1fe00b4170 0x7f1fe00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.723+0000 7f1fe785c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00d25d0 con 0x7f1fe00d0a80 2024-03-06T07:21:32.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.723+0000 7f1fe785c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00d2870 con 0x7f1fe00d4760 2024-03-06T07:21:32.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.723+0000 7f1fe785c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe00d4bf0 con 0x7f1fe00b4170 2024-03-06T07:21:32.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.724+0000 7f1fe5df9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1fe00d0a80 0x7f1fe00cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40646/0 (socket says 172.21.15.59:40646) 2024-03-06T07:21:32.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.724+0000 7f1fe5df9640 1 -- 172.21.15.59:0/3898299634 learned_addr learned my addr 172.21.15.59:0/3898299634 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:32.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.724+0000 7f1fe55f8640 1 --2- 172.21.15.59:0/3898299634 >> v2:172.21.15.59:3300/0 conn(0x7f1fe00b4170 0x7f1fe00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.724+0000 7f1fe55f8640 1 -- 172.21.15.59:0/3898299634 >> v1:172.21.15.164:6789/0 conn(0x7f1fe00d4760 legacy=0x7f1fe00d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.724+0000 7f1fe55f8640 1 -- 172.21.15.59:0/3898299634 >> v1:172.21.15.59:6789/0 conn(0x7f1fe00d0a80 legacy=0x7f1fe00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.724+0000 7f1fe55f8640 1 -- 172.21.15.59:0/3898299634 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00d2b50 con 0x7f1fe00b4170 2024-03-06T07:21:32.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.725+0000 7f1fe55f8640 1 --2- 172.21.15.59:0/3898299634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe00b4170 0x7f1fe00b6a90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0097a0 tx=0x7f1fdc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e16648127f277b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.725+0000 7f1fd3fff640 1 -- 172.21.15.59:0/3898299634 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc037070 con 0x7f1fe00b4170 2024-03-06T07:21:32.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.725+0000 7f1fd3fff640 1 -- 172.21.15.59:0/3898299634 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc030710 con 0x7f1fe00b4170 2024-03-06T07:21:32.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.725+0000 7f1fd3fff640 1 -- 172.21.15.59:0/3898299634 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc030ae0 con 0x7f1fe00b4170 2024-03-06T07:21:32.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.725+0000 7f1fe785c640 1 -- 172.21.15.59:0/3898299634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe00b4170 msgr2=0x7f1fe00b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.725+0000 7f1fe785c640 1 --2- 172.21.15.59:0/3898299634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe00b4170 0x7f1fe00b6a90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0097a0 tx=0x7f1fdc02b790 comp rx=0 tx=0).stop 2024-03-06T07:21:32.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.726+0000 7f1fe785c640 1 -- 172.21.15.59:0/3898299634 shutdown_connections 2024-03-06T07:21:32.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.726+0000 7f1fe785c640 1 --2- 172.21.15.59:0/3898299634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe00b4170 0x7f1fe00b6a90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.726+0000 7f1fe785c640 1 -- 172.21.15.59:0/3898299634 >> 172.21.15.59:0/3898299634 conn(0x7f1fe00d3c40 msgr2=0x7f1fe00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.726+0000 7f1fe785c640 1 -- 172.21.15.59:0/3898299634 shutdown_connections 2024-03-06T07:21:32.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.726+0000 7f1fe785c640 1 -- 172.21.15.59:0/3898299634 wait complete. 2024-03-06T07:21:32.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.727+0000 7f1fe785c640 1 Processor -- start 2024-03-06T07:21:32.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.727+0000 7f1fe785c640 1 -- start start 2024-03-06T07:21:32.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.727+0000 7f1fe785c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe00b4170 0x7f1fe015b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.727+0000 7f1fe785c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe00d0a80 0x7f1fe0157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.727+0000 7f1fe785c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe00cb790 0x7f1fe015ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.727+0000 7f1fe785c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00b4dd0 con 0x7f1fe00cb790 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.727+0000 7f1fe785c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00ce9e0 con 0x7f1fe00d0a80 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe785c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00b4f50 con 0x7f1fe00b4170 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe5df9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe00cb790 0x7f1fe015ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe5df9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe00cb790 0x7f1fe015ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51292/0 (socket says 172.21.15.59:51292) 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe5df9640 1 -- 172.21.15.59:0/1443351366 learned_addr learned my addr 172.21.15.59:0/1443351366 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe55f8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe00b4170 0x7f1fe015b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe4df7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe00d0a80 0x7f1fe0157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe5df9640 1 -- 172.21.15.59:0/1443351366 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe00b4170 msgr2=0x7f1fe015b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe5df9640 1 --2- 172.21.15.59:0/1443351366 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe00b4170 0x7f1fe015b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe5df9640 1 -- 172.21.15.59:0/1443351366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe00d0a80 msgr2=0x7f1fe0157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe5df9640 1 --2- 172.21.15.59:0/1443351366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe00d0a80 0x7f1fe0157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe5df9640 1 -- 172.21.15.59:0/1443351366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe015bc30 con 0x7f1fe00cb790 2024-03-06T07:21:32.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe5df9640 1 --2- 172.21.15.59:0/1443351366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe00cb790 0x7f1fe015ccd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800caf0 tx=0x7f1fd80020f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:32.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fd1ffb640 1 -- 172.21.15.59:0/1443351366 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8018070 con 0x7f1fe00cb790 2024-03-06T07:21:32.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fd1ffb640 1 -- 172.21.15.59:0/1443351366 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd800a760 con 0x7f1fe00cb790 2024-03-06T07:21:32.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fe785c640 1 -- 172.21.15.59:0/1443351366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe015bf10 con 0x7f1fe00cb790 2024-03-06T07:21:32.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.728+0000 7f1fd1ffb640 1 -- 172.21.15.59:0/1443351366 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd800ab90 con 0x7f1fe00cb790 2024-03-06T07:21:32.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.729+0000 7f1fe785c640 1 -- 172.21.15.59:0/1443351366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe015c4d0 con 0x7f1fe00cb790 2024-03-06T07:21:32.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.729+0000 7f1fd1ffb640 1 -- 172.21.15.59:0/1443351366 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1fd801b070 con 0x7f1fe00cb790 2024-03-06T07:21:32.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.729+0000 7f1fd1ffb640 1 -- 172.21.15.59:0/1443351366 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f1fd800ad30 con 0x7f1fe00cb790 2024-03-06T07:21:32.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.729+0000 7f1fe785c640 1 -- 172.21.15.59:0/1443351366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe004f330 con 0x7f1fe00cb790 2024-03-06T07:21:32.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.732+0000 7f1fd1ffb640 1 -- 172.21.15.59:0/1443351366 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1fd8033dd0 con 0x7f1fe00cb790 2024-03-06T07:21:32.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:32.755+0000 7f716d933040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:21:32.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.794+0000 7f1fe785c640 1 -- 172.21.15.59:0/1443351366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fe00733b0 con 0x7f1fe00cb790 2024-03-06T07:21:32.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.794+0000 7f1fd1ffb640 1 -- 172.21.15.59:0/1443351366 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f1fd8022030 con 0x7f1fe00cb790 2024-03-06T07:21:32.796 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:32.796 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":10,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:32.569300+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:32.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.795+0000 7f1fe785c640 1 -- 172.21.15.59:0/1443351366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe00cb790 msgr2=0x7f1fe015ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:32.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.795+0000 7f1fe785c640 1 --2- 172.21.15.59:0/1443351366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe00cb790 0x7f1fe015ccd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800caf0 tx=0x7f1fd80020f0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.795+0000 7f1fe785c640 1 -- 172.21.15.59:0/1443351366 shutdown_connections 2024-03-06T07:21:32.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.795+0000 7f1fe785c640 1 --2- 172.21.15.59:0/1443351366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe00cb790 0x7f1fe015ccd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.795+0000 7f1fe785c640 1 --2- 172.21.15.59:0/1443351366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe00d0a80 0x7f1fe0157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.795+0000 7f1fe785c640 1 --2- 172.21.15.59:0/1443351366 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe00b4170 0x7f1fe015b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:32.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.795+0000 7f1fe785c640 1 -- 172.21.15.59:0/1443351366 >> 172.21.15.59:0/1443351366 conn(0x7f1fe00d3c40 msgr2=0x7f1fe00cf650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:32.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.795+0000 7f1fe785c640 1 -- 172.21.15.59:0/1443351366 shutdown_connections 2024-03-06T07:21:32.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:32.795+0000 7f1fe785c640 1 -- 172.21.15.59:0/1443351366 wait complete. 2024-03-06T07:21:32.807 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:21:32.807 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:21:33.103 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:33.101+0000 7fce9cbbb040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:21:33.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:33.133+0000 7f716d933040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:21:33.199 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-03-06T07:21:33.197+0000 7f3df1e15640 -1 Falling back to public interface 2024-03-06T07:21:33.204 INFO:tasks.ceph.osd.5.smithi164.stderr:2024-03-06T07:21:33.202+0000 7f4538bff640 -1 Falling back to public interface 2024-03-06T07:21:33.212 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-03-06T07:21:33.211+0000 7fcd6ec1c640 -1 Falling back to public interface 2024-03-06T07:21:33.216 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-03-06T07:21:33.214+0000 7f4c26fbb640 -1 Falling back to public interface 2024-03-06T07:21:33.220 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-03-06T07:21:33.218+0000 7ff590216640 -1 Falling back to public interface 2024-03-06T07:21:33.220 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-03-06T07:21:33.218+0000 7f6476668640 -1 Falling back to public interface 2024-03-06T07:21:33.234 INFO:tasks.ceph.osd.6.smithi164.stderr:2024-03-06T07:21:33.232+0000 7f12708cc640 -1 Falling back to public interface 2024-03-06T07:21:33.234 INFO:tasks.ceph.osd.4.smithi164.stderr:2024-03-06T07:21:33.233+0000 7fdb28e9d640 -1 Falling back to public interface 2024-03-06T07:21:33.248 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:33.246+0000 7fce9cbbb040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:21:33.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:33.273+0000 7f716d933040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:21:33.360 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:33.358+0000 7fce9cbbb040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:21:33.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:33.390+0000 7f716d933040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:21:33.407 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:33.405+0000 7fce9cbbb040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:21:33.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:33.435+0000 7f716d933040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:21:33.524 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:33.522+0000 7fce9cbbb040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:33.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:33.551+0000 7f716d933040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:33.613 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:33.621 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:33.620+0000 7fce9cbbb040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:21:33.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:33.646+0000 7f716d933040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:21:33.711 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.701+0000 7fc9ddc23640 1 Processor -- start 2024-03-06T07:21:33.711 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.702+0000 7fc9ddc23640 1 -- start start 2024-03-06T07:21:33.711 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.702+0000 7fc9ddc23640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc9d815dfd0 0x7fc9d80ce360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.711 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.702+0000 7fc9ddc23640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc9d80b1050 0x7fc9d80cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.711 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.702+0000 7fc9ddc23640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80d1a00 0x7fc9d80d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.711 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.702+0000 7fc9ddc23640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d80d23d0 con 0x7fc9d815dfd0 2024-03-06T07:21:33.711 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.702+0000 7fc9ddc23640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d815e3b0 con 0x7fc9d80b1050 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.702+0000 7fc9ddc23640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d80cebf0 con 0x7fc9d80d1a00 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.702+0000 7fc9d77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc9d815dfd0 0x7fc9d80ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.702+0000 7fc9d77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc9d815dfd0 0x7fc9d80ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41186/0 (socket says 172.21.15.59:41186) 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.703+0000 7fc9d77fe640 1 -- 172.21.15.59:0/2115541023 learned_addr learned my addr 172.21.15.59:0/2115541023 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.703+0000 7fc9d7fff640 1 --2- 172.21.15.59:0/2115541023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80d1a00 0x7fc9d80d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.703+0000 7fc9d6ffd640 1 --2- 172.21.15.59:0/2115541023 >> v2:172.21.15.164:3300/0 conn(0x7fc9d80b1050 0x7fc9d80cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.703+0000 7fc9d77fe640 1 -- 172.21.15.59:0/2115541023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80d1a00 msgr2=0x7fc9d80d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.703+0000 7fc9d77fe640 1 --2- 172.21.15.59:0/2115541023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80d1a00 0x7fc9d80d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.703+0000 7fc9d77fe640 1 -- 172.21.15.59:0/2115541023 >> v2:172.21.15.164:3300/0 conn(0x7fc9d80b1050 msgr2=0x7fc9d80cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.703+0000 7fc9d77fe640 1 --2- 172.21.15.59:0/2115541023 >> v2:172.21.15.164:3300/0 conn(0x7fc9d80b1050 0x7fc9d80cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.703+0000 7fc9d77fe640 1 -- 172.21.15.59:0/2115541023 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d80ceed0 con 0x7fc9d815dfd0 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.703+0000 7fc9d7fff640 1 --2- 172.21.15.59:0/2115541023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80d1a00 0x7fc9d80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.703+0000 7fc9d77fe640 1 --2- 172.21.15.59:0/2115541023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d815dfd0 0x7fc9d80ce360 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc9d806bfd0 tx=0x7fc9c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a47a315d129791c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.712 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9d67fc640 1 -- 172.21.15.59:0/2115541023 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8035020 con 0x7fc9d815dfd0 2024-03-06T07:21:33.713 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9d67fc640 1 -- 172.21.15.59:0/2115541023 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c8031870 con 0x7fc9d815dfd0 2024-03-06T07:21:33.713 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9d67fc640 1 -- 172.21.15.59:0/2115541023 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8031c40 con 0x7fc9d815dfd0 2024-03-06T07:21:33.713 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9ddc23640 1 -- 172.21.15.59:0/2115541023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d815dfd0 msgr2=0x7fc9d80ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.713 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9ddc23640 1 --2- 172.21.15.59:0/2115541023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d815dfd0 0x7fc9d80ce360 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc9d806bfd0 tx=0x7fc9c802b910 comp rx=0 tx=0).stop 2024-03-06T07:21:33.713 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9ddc23640 1 -- 172.21.15.59:0/2115541023 shutdown_connections 2024-03-06T07:21:33.713 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9ddc23640 1 --2- 172.21.15.59:0/2115541023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80d1a00 0x7fc9d80d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.713 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9ddc23640 1 --2- 172.21.15.59:0/2115541023 >> v2:172.21.15.164:3300/0 conn(0x7fc9d80b1050 0x7fc9d80cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9ddc23640 1 --2- 172.21.15.59:0/2115541023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d815dfd0 0x7fc9d80ce360 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9ddc23640 1 -- 172.21.15.59:0/2115541023 >> 172.21.15.59:0/2115541023 conn(0x7fc9d809a9e0 msgr2=0x7fc9d809adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9ddc23640 1 -- 172.21.15.59:0/2115541023 shutdown_connections 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.704+0000 7fc9ddc23640 1 -- 172.21.15.59:0/2115541023 wait complete. 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.705+0000 7fc9ddc23640 1 Processor -- start 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.706+0000 7fc9ddc23640 1 -- start start 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.706+0000 7fc9ddc23640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80b1050 0x7fc9d815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.706+0000 7fc9ddc23640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9d80d1a00 0x7fc9d8152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.706+0000 7fc9ddc23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d815dfd0 0x7fc9d80b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.706+0000 7fc9ddc23640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d80ce8a0 con 0x7fc9d815dfd0 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.706+0000 7fc9ddc23640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d806c5d0 con 0x7fc9d80d1a00 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.706+0000 7fc9ddc23640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d80d01b0 con 0x7fc9d80b1050 2024-03-06T07:21:33.717 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.706+0000 7fc9d7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d815dfd0 0x7fc9d80b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.706+0000 7fc9d7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d815dfd0 0x7fc9d80b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41198/0 (socket says 172.21.15.59:41198) 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.706+0000 7fc9d7fff640 1 -- 172.21.15.59:0/3707531064 learned_addr learned my addr 172.21.15.59:0/3707531064 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.706+0000 7fc9d77fe640 1 --2- 172.21.15.59:0/3707531064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80b1050 0x7fc9d815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9d6ffd640 1 --2- 172.21.15.59:0/3707531064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9d80d1a00 0x7fc9d8152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9d7fff640 1 -- 172.21.15.59:0/3707531064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80b1050 msgr2=0x7fc9d815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9d7fff640 1 --2- 172.21.15.59:0/3707531064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80b1050 0x7fc9d815e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9d7fff640 1 -- 172.21.15.59:0/3707531064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9d80d1a00 msgr2=0x7fc9d8152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9d7fff640 1 --2- 172.21.15.59:0/3707531064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9d80d1a00 0x7fc9d8152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9d7fff640 1 -- 172.21.15.59:0/3707531064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d80cae30 con 0x7fc9d815dfd0 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9d77fe640 1 --2- 172.21.15.59:0/3707531064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80b1050 0x7fc9d815e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9d7fff640 1 --2- 172.21.15.59:0/3707531064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d815dfd0 0x7fc9d80b1ee0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0007ef0 tx=0x7fc9d000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9bffff640 1 -- 172.21.15.59:0/3707531064 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d001a070 con 0x7fc9d815dfd0 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9bffff640 1 -- 172.21.15.59:0/3707531064 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d0003b90 con 0x7fc9d815dfd0 2024-03-06T07:21:33.718 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9ddc23640 1 -- 172.21.15.59:0/3707531064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d80cb110 con 0x7fc9d815dfd0 2024-03-06T07:21:33.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9bffff640 1 -- 172.21.15.59:0/3707531064 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0004030 con 0x7fc9d815dfd0 2024-03-06T07:21:33.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9ddc23640 1 -- 172.21.15.59:0/3707531064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d80b19c0 con 0x7fc9d815dfd0 2024-03-06T07:21:33.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9bffff640 1 -- 172.21.15.59:0/3707531064 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc9d001d070 con 0x7fc9d815dfd0 2024-03-06T07:21:33.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.707+0000 7fc9bffff640 1 -- 172.21.15.59:0/3707531064 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc9d0015440 con 0x7fc9d815dfd0 2024-03-06T07:21:33.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.708+0000 7fc9ddc23640 1 -- 172.21.15.59:0/3707531064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d80b1bc0 con 0x7fc9d815dfd0 2024-03-06T07:21:33.719 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.710+0000 7fc9bffff640 1 -- 172.21.15.59:0/3707531064 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc9d80b1bc0 con 0x7fc9d815dfd0 2024-03-06T07:21:33.775 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.773+0000 7fc9ddc23640 1 -- 172.21.15.59:0/3707531064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9d804f330 con 0x7fc9d815dfd0 2024-03-06T07:21:33.775 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.773+0000 7fc9bffff640 1 -- 172.21.15.59:0/3707531064 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fc9d0010070 con 0x7fc9d815dfd0 2024-03-06T07:21:33.776 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-06T07:21:33.776 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:32.569300+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:33.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.774+0000 7fc9ddc23640 1 -- 172.21.15.59:0/3707531064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d815dfd0 msgr2=0x7fc9d80b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:33.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.774+0000 7fc9ddc23640 1 --2- 172.21.15.59:0/3707531064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d815dfd0 0x7fc9d80b1ee0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0007ef0 tx=0x7fc9d000d4a0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.774+0000 7fc9ddc23640 1 -- 172.21.15.59:0/3707531064 shutdown_connections 2024-03-06T07:21:33.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.774+0000 7fc9ddc23640 1 --2- 172.21.15.59:0/3707531064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d815dfd0 0x7fc9d80b1ee0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.774+0000 7fc9ddc23640 1 --2- 172.21.15.59:0/3707531064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9d80d1a00 0x7fc9d8152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.774+0000 7fc9ddc23640 1 --2- 172.21.15.59:0/3707531064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d80b1050 0x7fc9d815e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:33.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.774+0000 7fc9ddc23640 1 -- 172.21.15.59:0/3707531064 >> 172.21.15.59:0/3707531064 conn(0x7fc9d809a9e0 msgr2=0x7fc9d806c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:33.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.774+0000 7fc9ddc23640 1 -- 172.21.15.59:0/3707531064 shutdown_connections 2024-03-06T07:21:33.777 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:33.774+0000 7fc9ddc23640 1 -- 172.21.15.59:0/3707531064 wait complete. 2024-03-06T07:21:33.785 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:21:33.785 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:33.784+0000 7fce9cbbb040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:33.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:33.803+0000 7f716d933040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:33.829 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:33.828+0000 7fce9cbbb040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:33.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:33.846+0000 7f716d933040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:33.991 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:33.989+0000 7fce9cbbb040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:21:34.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:34.000+0000 7f716d933040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:21:34.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:34.828+0000 7f716d933040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:21:34.864 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:34.862+0000 7fce9cbbb040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:21:35.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:35.107+0000 7f716d933040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:21:35.151 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:35.149+0000 7fce9cbbb040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:21:35.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:35.160+0000 7f716d933040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:21:35.204 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:35.202+0000 7fce9cbbb040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:21:35.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:35.277+0000 7f716d933040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:21:35.323 INFO:tasks.ceph.mgr.y.smithi059.stderr:2024-03-06T07:21:35.322+0000 7fce9cbbb040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:21:36.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:36.317+0000 7f710eda2640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:21:36.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2024-03-06T07:21:36.317+0000 7f710eda2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:21:36.662 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-03-06T07:21:36.660+0000 7f3df1e15640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:21:36.691 INFO:tasks.ceph.osd.5.smithi164.stderr:2024-03-06T07:21:36.690+0000 7f4538bff640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:21:36.697 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-03-06T07:21:36.696+0000 7f4c26fbb640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:21:36.739 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-03-06T07:21:36.737+0000 7fcd6ec1c640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:21:36.740 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-03-06T07:21:36.738+0000 7f6476668640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:21:36.745 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-03-06T07:21:36.743+0000 7ff590216640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:21:36.772 INFO:tasks.ceph.osd.4.smithi164.stderr:2024-03-06T07:21:36.771+0000 7fdb28e9d640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:21:36.816 INFO:tasks.ceph.osd.6.smithi164.stderr:2024-03-06T07:21:36.814+0000 7f12708cc640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:21:40.199 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-03-06T07:21:40.197+0000 7f3decf62640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:21:40.213 INFO:tasks.ceph.osd.2.smithi059.stderr:2024-03-06T07:21:40.212+0000 7f3de5594640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:40.372 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-03-06T07:21:40.370+0000 7f4c2272e640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:21:40.387 INFO:tasks.ceph.osd.1.smithi059.stderr:2024-03-06T07:21:40.385+0000 7f4c1ad60640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:40.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2024-03-06T07:21:40.401+0000 7f126c03f640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:21:40.408 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-03-06T07:21:40.406+0000 7fcd69d3e640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:21:40.409 INFO:tasks.ceph.osd.5.smithi164.stderr:2024-03-06T07:21:40.406+0000 7f4534372640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:21:40.417 INFO:tasks.ceph.osd.6.smithi164.stderr:2024-03-06T07:21:40.415+0000 7f1264671640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:40.419 INFO:tasks.ceph.osd.3.smithi059.stderr:2024-03-06T07:21:40.418+0000 7fcd62370640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:40.422 INFO:tasks.ceph.osd.5.smithi164.stderr:2024-03-06T07:21:40.420+0000 7f452c9a4640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:40.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-03-06T07:21:40.426+0000 7f6471ddb640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:21:40.433 INFO:tasks.ceph.osd.4.smithi164.stderr:2024-03-06T07:21:40.430+0000 7fdb24610640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:21:40.433 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-03-06T07:21:40.431+0000 7ff58b3c9640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:21:40.440 INFO:tasks.ceph.osd.7.smithi164.stderr:2024-03-06T07:21:40.438+0000 7f646a40d640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:40.445 INFO:tasks.ceph.osd.4.smithi164.stderr:2024-03-06T07:21:40.444+0000 7fdb1cc42640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:40.447 INFO:tasks.ceph.osd.0.smithi059.stderr:2024-03-06T07:21:40.446+0000 7ff5839fb640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:40.593 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:40.684 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.682+0000 7f4d91422640 1 Processor -- start 2024-03-06T07:21:40.684 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.682+0000 7f4d91422640 1 -- start start 2024-03-06T07:21:40.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.683+0000 7f4d91422640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 0x7f4d8c0b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.683+0000 7f4d91422640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c0ccf70 con 0x7f4d8c0726f0 2024-03-06T07:21:40.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.683+0000 7f4d91422640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c0cd230 con 0x7f4d8c066d00 2024-03-06T07:21:40.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.683+0000 7f4d91422640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c0cd510 con 0x7f4d8c067160 2024-03-06T07:21:40.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.683+0000 7f4d8affd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4d8c0726f0 0x7f4d8c072ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:33206/0 (socket says 172.21.15.59:33206) 2024-03-06T07:21:40.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.683+0000 7f4d8affd640 1 -- 172.21.15.59:0/4273597412 learned_addr learned my addr 172.21.15.59:0/4273597412 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:40.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.683+0000 7f4d8b7fe640 1 --2- 172.21.15.59:0/4273597412 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 0x7f4d8c0b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.683+0000 7f4d89ffb640 1 -- 172.21.15.59:0/4273597412 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710125334 0 0) 0x7f4d8c0ccf70 con 0x7f4d8c0726f0 2024-03-06T07:21:40.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.683+0000 7f4d89ffb640 1 -- 172.21.15.59:0/4273597412 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d78003520 con 0x7f4d8c0726f0 2024-03-06T07:21:40.686 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.684+0000 7f4d89ffb640 1 -- 172.21.15.59:0/4273597412 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552818610 0 0) 0x7f4d8c0cd230 con 0x7f4d8c066d00 2024-03-06T07:21:40.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.684+0000 7f4d8b7fe640 1 -- 172.21.15.59:0/4273597412 >> v1:172.21.15.164:6789/0 conn(0x7f4d8c066d00 legacy=0x7f4d8c0b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.684+0000 7f4d8b7fe640 1 -- 172.21.15.59:0/4273597412 >> v1:172.21.15.59:6789/0 conn(0x7f4d8c0726f0 legacy=0x7f4d8c072ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.684+0000 7f4d8b7fe640 1 -- 172.21.15.59:0/4273597412 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c0cd7f0 con 0x7f4d8c067160 2024-03-06T07:21:40.687 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.684+0000 7f4d8b7fe640 1 --2- 172.21.15.59:0/4273597412 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 0x7f4d8c0b09f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009dc0 tx=0x7f4d7c02bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba45a641fc38a156 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.685+0000 7f4d89ffb640 1 -- 172.21.15.59:0/4273597412 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c039070 con 0x7f4d8c067160 2024-03-06T07:21:40.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.685+0000 7f4d89ffb640 1 -- 172.21.15.59:0/4273597412 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7c02c840 con 0x7f4d8c067160 2024-03-06T07:21:40.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.685+0000 7f4d89ffb640 1 -- 172.21.15.59:0/4273597412 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84001670 con 0x7f4d8c067160 2024-03-06T07:21:40.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.685+0000 7f4d91422640 1 -- 172.21.15.59:0/4273597412 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 msgr2=0x7f4d8c0b09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.688 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.685+0000 7f4d91422640 1 --2- 172.21.15.59:0/4273597412 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 0x7f4d8c0b09f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009dc0 tx=0x7f4d7c02bd10 comp rx=0 tx=0).stop 2024-03-06T07:21:40.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.685+0000 7f4d91422640 1 -- 172.21.15.59:0/4273597412 shutdown_connections 2024-03-06T07:21:40.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.685+0000 7f4d91422640 1 --2- 172.21.15.59:0/4273597412 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 0x7f4d8c0b09f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.685+0000 7f4d91422640 1 -- 172.21.15.59:0/4273597412 >> 172.21.15.59:0/4273597412 conn(0x7f4d8c0b3370 msgr2=0x7f4d8c0b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:40.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.685+0000 7f4d91422640 1 -- 172.21.15.59:0/4273597412 shutdown_connections 2024-03-06T07:21:40.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.685+0000 7f4d91422640 1 -- 172.21.15.59:0/4273597412 wait complete. 2024-03-06T07:21:40.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.686+0000 7f4d91422640 1 Processor -- start 2024-03-06T07:21:40.689 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.686+0000 7f4d91422640 1 -- start start 2024-03-06T07:21:40.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.687+0000 7f4d91422640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d8c066d00 0x7f4d8c099ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.691 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.687+0000 7f4d91422640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 0x7f4d8c06d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.692 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.687+0000 7f4d91422640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d8c0726f0 0x7f4d8c06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.692 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.687+0000 7f4d91422640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c15e230 con 0x7f4d8c066d00 2024-03-06T07:21:40.695 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.687+0000 7f4d91422640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c15e0b0 con 0x7f4d8c0726f0 2024-03-06T07:21:40.695 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.687+0000 7f4d91422640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c0b1ae0 con 0x7f4d8c067160 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.687+0000 7f4d8affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d8c066d00 0x7f4d8c099ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.687+0000 7f4d8affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d8c066d00 0x7f4d8c099ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41246/0 (socket says 172.21.15.59:41246) 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.687+0000 7f4d8affd640 1 -- 172.21.15.59:0/2393288101 learned_addr learned my addr 172.21.15.59:0/2393288101 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.687+0000 7f4d8b7fe640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d8c0726f0 0x7f4d8c06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.687+0000 7f4d8a7fc640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 0x7f4d8c06d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d8affd640 1 -- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 msgr2=0x7f4d8c06d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d8affd640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 0x7f4d8c06d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d8affd640 1 -- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d8c0726f0 msgr2=0x7f4d8c06d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d8affd640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d8c0726f0 0x7f4d8c06d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d8affd640 1 -- 172.21.15.59:0/2393288101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c1f4700 con 0x7f4d8c066d00 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d8a7fc640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 0x7f4d8c06d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d8affd640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d8c066d00 0x7f4d8c099ac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400c820 tx=0x7f4d8400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.696 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d837fe640 1 -- 172.21.15.59:0/2393288101 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84018070 con 0x7f4d8c066d00 2024-03-06T07:21:40.697 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d837fe640 1 -- 172.21.15.59:0/2393288101 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d84007500 con 0x7f4d8c066d00 2024-03-06T07:21:40.697 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d91422640 1 -- 172.21.15.59:0/2393288101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8c1f49e0 con 0x7f4d8c066d00 2024-03-06T07:21:40.697 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d837fe640 1 -- 172.21.15.59:0/2393288101 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84004a20 con 0x7f4d8c066d00 2024-03-06T07:21:40.697 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.688+0000 7f4d91422640 1 -- 172.21.15.59:0/2393288101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8c1f5370 con 0x7f4d8c066d00 2024-03-06T07:21:40.697 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.689+0000 7f4d91422640 1 -- 172.21.15.59:0/2393288101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8c04f330 con 0x7f4d8c066d00 2024-03-06T07:21:40.697 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.689+0000 7f4d837fe640 1 -- 172.21.15.59:0/2393288101 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f4d8401b070 con 0x7f4d8c066d00 2024-03-06T07:21:40.697 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.690+0000 7f4d837fe640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d6c06b440 0x7f4d6c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:40.697 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.690+0000 7f4d837fe640 1 -- 172.21.15.59:0/2393288101 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f4d84088970 con 0x7f4d8c066d00 2024-03-06T07:21:40.697 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.690+0000 7f4d8a7fc640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d6c06b440 0x7f4d6c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:40.698 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.690+0000 7f4d8a7fc640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d6c06b440 0x7f4d6c06d900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0ad7b0 tx=0x7f4d74009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:40.698 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.694+0000 7f4d837fe640 1 -- 172.21.15.59:0/2393288101 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d84058660 con 0x7f4d8c066d00 2024-03-06T07:21:40.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.876+0000 7f4d91422640 1 -- 172.21.15.59:0/2393288101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d8c1f4bb0 con 0x7f4d8c066d00 2024-03-06T07:21:40.878 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.876+0000 7f4d837fe640 1 -- 172.21.15.59:0/2393288101 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f4d8405be70 con 0x7f4d8c066d00 2024-03-06T07:21:40.878 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-06T07:21:40.879 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":12,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:38.317094+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:40.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 -- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d6c06b440 msgr2=0x7f4d6c06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d6c06b440 0x7f4d6c06d900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0ad7b0 tx=0x7f4d74009040 comp rx=0 tx=0).stop 2024-03-06T07:21:40.880 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 -- 172.21.15.59:0/2393288101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d8c066d00 msgr2=0x7f4d8c099ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:40.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d8c066d00 0x7f4d8c099ac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400c820 tx=0x7f4d8400cce0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 -- 172.21.15.59:0/2393288101 shutdown_connections 2024-03-06T07:21:40.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d6c06b440 0x7f4d6c06d900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d8c0726f0 0x7f4d8c06d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d8c067160 0x7f4d8c06d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 --2- 172.21.15.59:0/2393288101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d8c066d00 0x7f4d8c099ac0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:40.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 -- 172.21.15.59:0/2393288101 >> 172.21.15.59:0/2393288101 conn(0x7f4d8c0b3370 msgr2=0x7f4d8c0cbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:40.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 -- 172.21.15.59:0/2393288101 shutdown_connections 2024-03-06T07:21:40.881 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:40.878+0000 7f4d91422640 1 -- 172.21.15.59:0/2393288101 wait complete. 2024-03-06T07:21:40.890 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:21:47.697 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:47.799 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.797+0000 7fde34033640 1 Processor -- start 2024-03-06T07:21:47.799 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.797+0000 7fde34033640 1 -- start start 2024-03-06T07:21:47.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde34033640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde2c0d0d70 0x7fde2c0ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde34033640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fde2c0ced50 0x7fde2c0b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde34033640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c0b33f0 con 0x7fde2c15e210 2024-03-06T07:21:47.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde34033640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde2c0b38d0 con 0x7fde2c0ced50 2024-03-06T07:21:47.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde34033640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0cf4f0 con 0x7fde2c0d0d70 2024-03-06T07:21:47.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde325d0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fde2c0ced50 0x7fde2c0b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde31dcf640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fde2c15e210 0x7fde2c0d0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39450/0 (socket says 172.21.15.59:39450) 2024-03-06T07:21:47.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde31dcf640 1 -- 172.21.15.59:0/1615033777 learned_addr learned my addr 172.21.15.59:0/1615033777 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:47.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde325d0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fde2c0ced50 0x7fde2c0b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41638/0 (socket says 172.21.15.59:41638) 2024-03-06T07:21:47.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde315ce640 1 --2- 172.21.15.59:0/1615033777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde2c0d0d70 0x7fde2c0ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.801 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde325d0640 1 -- 172.21.15.59:0/1615033777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde2c0d0d70 msgr2=0x7fde2c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:47.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde325d0640 1 --2- 172.21.15.59:0/1615033777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde2c0d0d70 0x7fde2c0ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde325d0640 1 -- 172.21.15.59:0/1615033777 >> v1:172.21.15.59:6789/0 conn(0x7fde2c15e210 legacy=0x7fde2c0d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde325d0640 1 -- 172.21.15.59:0/1615033777 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c0cf7d0 con 0x7fde2c0ced50 2024-03-06T07:21:47.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.798+0000 7fde30dcd640 1 -- 172.21.15.59:0/1615033777 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976575818 0 0) 0x7fde2c0b33f0 con 0x7fde2c15e210 2024-03-06T07:21:47.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.799+0000 7fde325d0640 1 --2- 172.21.15.59:0/1615033777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde2c0ced50 0x7fde2c0b3d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde24009d90 tx=0x7fde2402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7756036b829fc8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.799+0000 7fde30dcd640 1 -- 172.21.15.59:0/1615033777 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde24039070 con 0x7fde2c0ced50 2024-03-06T07:21:47.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.799+0000 7fde30dcd640 1 -- 172.21.15.59:0/1615033777 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2402cc50 con 0x7fde2c0ced50 2024-03-06T07:21:47.802 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.799+0000 7fde30dcd640 1 -- 172.21.15.59:0/1615033777 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde24034440 con 0x7fde2c0ced50 2024-03-06T07:21:47.805 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.800+0000 7fde34033640 1 -- 172.21.15.59:0/1615033777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde2c0ced50 msgr2=0x7fde2c0b3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.800+0000 7fde34033640 1 --2- 172.21.15.59:0/1615033777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde2c0ced50 0x7fde2c0b3d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde24009d90 tx=0x7fde2402bea0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.800+0000 7fde34033640 1 -- 172.21.15.59:0/1615033777 shutdown_connections 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.800+0000 7fde34033640 1 --2- 172.21.15.59:0/1615033777 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde2c0ced50 0x7fde2c0b3d60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.800+0000 7fde34033640 1 --2- 172.21.15.59:0/1615033777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde2c0d0d70 0x7fde2c0ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.800+0000 7fde34033640 1 -- 172.21.15.59:0/1615033777 >> 172.21.15.59:0/1615033777 conn(0x7fde2c09d050 msgr2=0x7fde2c0d3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.800+0000 7fde34033640 1 -- 172.21.15.59:0/1615033777 shutdown_connections 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.800+0000 7fde34033640 1 -- 172.21.15.59:0/1615033777 wait complete. 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.801+0000 7fde34033640 1 Processor -- start 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.801+0000 7fde34033640 1 -- start start 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde34033640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde2c0ced50 0x7fde2c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde34033640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c0d0d70 0x7fde2c160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde34033640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde2c15e210 0x7fde2c0b9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.806 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde34033640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0b42a0 con 0x7fde2c15e210 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde34033640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0cf1e0 con 0x7fde2c0d0d70 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde34033640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0b4420 con 0x7fde2c0ced50 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde325d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde2c15e210 0x7fde2c0b9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde325d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde2c15e210 0x7fde2c0b9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41652/0 (socket says 172.21.15.59:41652) 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde325d0640 1 -- 172.21.15.59:0/293580759 learned_addr learned my addr 172.21.15.59:0/293580759 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde31dcf640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde2c0ced50 0x7fde2c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde315ce640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c0d0d70 0x7fde2c160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde325d0640 1 -- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde2c0ced50 msgr2=0x7fde2c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde325d0640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde2c0ced50 0x7fde2c0cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde325d0640 1 -- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c0d0d70 msgr2=0x7fde2c160350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde325d0640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c0d0d70 0x7fde2c160350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde325d0640 1 -- 172.21.15.59:0/293580759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c0cd010 con 0x7fde2c15e210 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde31dcf640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde2c0ced50 0x7fde2c0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.802+0000 7fde325d0640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde2c15e210 0x7fde2c0b9000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde24037040 tx=0x7fde24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.807 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.803+0000 7fde227fc640 1 -- 172.21.15.59:0/293580759 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde24044030 con 0x7fde2c15e210 2024-03-06T07:21:47.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.803+0000 7fde227fc640 1 -- 172.21.15.59:0/293580759 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2402ca50 con 0x7fde2c15e210 2024-03-06T07:21:47.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.803+0000 7fde34033640 1 -- 172.21.15.59:0/293580759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2c0cd2f0 con 0x7fde2c15e210 2024-03-06T07:21:47.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.803+0000 7fde227fc640 1 -- 172.21.15.59:0/293580759 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde24039040 con 0x7fde2c15e210 2024-03-06T07:21:47.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.803+0000 7fde34033640 1 -- 172.21.15.59:0/293580759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2c0cd8b0 con 0x7fde2c15e210 2024-03-06T07:21:47.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.804+0000 7fde34033640 1 -- 172.21.15.59:0/293580759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2c04f330 con 0x7fde2c15e210 2024-03-06T07:21:47.808 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.804+0000 7fde227fc640 1 -- 172.21.15.59:0/293580759 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde2403f470 con 0x7fde2c15e210 2024-03-06T07:21:47.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.804+0000 7fde227fc640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde1006b500 0x7fde1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.804+0000 7fde227fc640 1 -- 172.21.15.59:0/293580759 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fde240ac2c0 con 0x7fde2c15e210 2024-03-06T07:21:47.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.805+0000 7fde31dcf640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde1006b500 0x7fde1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.805+0000 7fde31dcf640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde1006b500 0x7fde1006d9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fde28000f60 tx=0x7fde28007ca0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.810 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.809+0000 7fde227fc640 1 -- 172.21.15.59:0/293580759 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde2404e020 con 0x7fde2c15e210 2024-03-06T07:21:47.992 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.991+0000 7fde34033640 1 -- 172.21.15.59:0/293580759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde2c06bb20 con 0x7fde2c15e210 2024-03-06T07:21:47.993 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.992+0000 7fde227fc640 1 -- 172.21.15.59:0/293580759 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12872 (secure 0 0 0) 0x7fde2402cc20 con 0x7fde2c15e210 2024-03-06T07:21:47.993 INFO:teuthology.misc.health.smithi059.stdout: 2024-03-06T07:21:47.994 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":16,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:44.206150+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:47.995 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.993+0000 7fde34033640 1 -- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde1006b500 msgr2=0x7fde1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.993+0000 7fde34033640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde1006b500 0x7fde1006d9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fde28000f60 tx=0x7fde28007ca0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.993+0000 7fde34033640 1 -- 172.21.15.59:0/293580759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde2c15e210 msgr2=0x7fde2c0b9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.993+0000 7fde34033640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde2c15e210 0x7fde2c0b9000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde24037040 tx=0x7fde24004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:47.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.994+0000 7fde34033640 1 -- 172.21.15.59:0/293580759 shutdown_connections 2024-03-06T07:21:47.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.994+0000 7fde34033640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde1006b500 0x7fde1006d9c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.994+0000 7fde34033640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde2c15e210 0x7fde2c0b9000 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.994+0000 7fde34033640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c0d0d70 0x7fde2c160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.994+0000 7fde34033640 1 --2- 172.21.15.59:0/293580759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde2c0ced50 0x7fde2c0cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.994+0000 7fde34033640 1 -- 172.21.15.59:0/293580759 >> 172.21.15.59:0/293580759 conn(0x7fde2c09d050 msgr2=0x7fde2c06c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.994+0000 7fde34033640 1 -- 172.21.15.59:0/293580759 shutdown_connections 2024-03-06T07:21:47.996 INFO:teuthology.misc.health.smithi059.stderr:2024-03-06T07:21:47.994+0000 7fde34033640 1 -- 172.21.15.59:0/293580759 wait complete. 2024-03-06T07:21:48.005 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:21:48.005 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:21:48.006 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:21:48.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:21:48.008 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:21:48.008 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:21:48.008 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:21:48.010 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:21:48.010 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:21:48.010 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:21:48.013 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:21:48.013 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:21:48.014 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:21:48.016 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:21:48.016 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:21:48.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:48.084 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:21:48.084 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:21:48.099 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:21:48.099 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:21:48.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.124+0000 7fa8f4ac9640 1 Processor -- start 2024-03-06T07:21:48.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.124+0000 7fa8f4ac9640 1 -- start start 2024-03-06T07:21:48.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.125+0000 7fa8f4ac9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f00b4170 0x7fa8f00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.125+0000 7fa8f4ac9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8f00d4760 0x7fa8f00d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.125+0000 7fa8f4ac9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f00d25d0 con 0x7fa8f00b4dd0 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.125+0000 7fa8f4ac9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f00d4bf0 con 0x7fa8f00d4760 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.125+0000 7fa8f4ac9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00ce9e0 con 0x7fa8f00b4170 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.125+0000 7fa8edd9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8f00d4760 0x7fa8f00d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.125+0000 7fa8edd9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8f00d4760 0x7fa8f00d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41708/0 (socket says 172.21.15.59:41708) 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.125+0000 7fa8edd9b640 1 -- 172.21.15.59:0/3221760911 learned_addr learned my addr 172.21.15.59:0/3221760911 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.125+0000 7fa8eed9d640 1 --1- 172.21.15.59:0/3221760911 >> v1:172.21.15.59:6789/0 conn(0x7fa8f00b4dd0 0x7fa8f00cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39474/0 (socket says 172.21.15.59:39474) 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.126+0000 7fa8ee59c640 1 --2- 172.21.15.59:0/3221760911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f00b4170 0x7fa8f00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.126+0000 7fa8edd9b640 1 -- 172.21.15.59:0/3221760911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f00b4170 msgr2=0x7fa8f00b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.126+0000 7fa8edd9b640 1 --2- 172.21.15.59:0/3221760911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f00b4170 0x7fa8f00b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.126+0000 7fa8edd9b640 1 -- 172.21.15.59:0/3221760911 >> v1:172.21.15.59:6789/0 conn(0x7fa8f00b4dd0 legacy=0x7fa8f00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.126+0000 7fa8edd9b640 1 -- 172.21.15.59:0/3221760911 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f00d2870 con 0x7fa8f00d4760 2024-03-06T07:21:48.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.126+0000 7fa8ed59a640 1 -- 172.21.15.59:0/3221760911 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621411134 0 0) 0x7fa8f00d25d0 con 0x7fa8f00b4dd0 2024-03-06T07:21:48.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.126+0000 7fa8edd9b640 1 --2- 172.21.15.59:0/3221760911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00d4760 0x7fa8f00d0370 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009870 tx=0x7fa8d802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64b0d7679959e9f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.127+0000 7fa8ed59a640 1 -- 172.21.15.59:0/3221760911 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8034070 con 0x7fa8f00d4760 2024-03-06T07:21:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.127+0000 7fa8ed59a640 1 -- 172.21.15.59:0/3221760911 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d8030840 con 0x7fa8f00d4760 2024-03-06T07:21:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.127+0000 7fa8ed59a640 1 -- 172.21.15.59:0/3221760911 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8030c10 con 0x7fa8f00d4760 2024-03-06T07:21:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.127+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/3221760911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00d4760 msgr2=0x7fa8f00d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.127+0000 7fa8f4ac9640 1 --2- 172.21.15.59:0/3221760911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00d4760 0x7fa8f00d0370 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009870 tx=0x7fa8d802b980 comp rx=0 tx=0).stop 2024-03-06T07:21:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.127+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/3221760911 shutdown_connections 2024-03-06T07:21:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.127+0000 7fa8f4ac9640 1 --2- 172.21.15.59:0/3221760911 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00d4760 0x7fa8f00d0370 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.127+0000 7fa8f4ac9640 1 --2- 172.21.15.59:0/3221760911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f00b4170 0x7fa8f00b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.127+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/3221760911 >> 172.21.15.59:0/3221760911 conn(0x7fa8f00d3c40 msgr2=0x7fa8f00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.127+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/3221760911 shutdown_connections 2024-03-06T07:21:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.127+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/3221760911 wait complete. 2024-03-06T07:21:48.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.128+0000 7fa8f4ac9640 1 Processor -- start 2024-03-06T07:21:48.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.128+0000 7fa8f4ac9640 1 -- start start 2024-03-06T07:21:48.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8f4ac9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00b4170 0x7fa8f0152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8f4ac9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f00b4dd0 0x7fa8f00c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8f4ac9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8f01f3b90 0x7fa8f01f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8ee59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00b4170 0x7fa8f0152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8f4ac9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00b47d0 con 0x7fa8f00b4170 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8f4ac9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00b4550 con 0x7fa8f01f3b90 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8f4ac9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00b4950 con 0x7fa8f00b4dd0 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8ee59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00b4170 0x7fa8f0152150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41716/0 (socket says 172.21.15.59:41716) 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8ee59c640 1 -- 172.21.15.59:0/1364651885 learned_addr learned my addr 172.21.15.59:0/1364651885 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8ee59c640 1 -- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f00b4dd0 msgr2=0x7fa8f00c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8ee59c640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f00b4dd0 0x7fa8f00c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8ee59c640 1 -- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8f01f3b90 msgr2=0x7fa8f01f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8ee59c640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8f01f3b90 0x7fa8f01f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.129+0000 7fa8ee59c640 1 -- 172.21.15.59:0/1364651885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f015bdc0 con 0x7fa8f00b4170 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.130+0000 7fa8ee59c640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00b4170 0x7fa8f0152150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800d800 tx=0x7fa8e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.130+0000 7fa8e6ffd640 1 -- 172.21.15.59:0/1364651885 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e800b6c0 con 0x7fa8f00b4170 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.130+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/1364651885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f015c0a0 con 0x7fa8f00b4170 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.130+0000 7fa8e6ffd640 1 -- 172.21.15.59:0/1364651885 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e8019070 con 0x7fa8f00b4170 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.130+0000 7fa8e6ffd640 1 -- 172.21.15.59:0/1364651885 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e800ba80 con 0x7fa8f00b4170 2024-03-06T07:21:48.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.130+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/1364651885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f015c660 con 0x7fa8f00b4170 2024-03-06T07:21:48.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.131+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/1364651885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f004f330 con 0x7fa8f00b4170 2024-03-06T07:21:48.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.131+0000 7fa8e6ffd640 1 -- 172.21.15.59:0/1364651885 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8e800bc20 con 0x7fa8f00b4170 2024-03-06T07:21:48.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.131+0000 7fa8e6ffd640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 0x7fa8d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.132+0000 7fa8e6ffd640 1 -- 172.21.15.59:0/1364651885 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa8e808a360 con 0x7fa8f00b4170 2024-03-06T07:21:48.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.132+0000 7fa8edd9b640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 0x7fa8d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.132+0000 7fa8edd9b640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 0x7fa8d006d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8036000 tx=0x7fa8d8030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.136+0000 7fa8e6ffd640 1 -- 172.21.15.59:0/1364651885 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8e8059530 con 0x7fa8f00b4170 2024-03-06T07:21:48.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.319+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/1364651885 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8f00733b0 con 0x7fa8f00b4170 2024-03-06T07:21:48.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.320+0000 7fa8e6ffd640 1 -- 172.21.15.59:0/1364651885 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12872 (secure 0 0 0) 0x7fa8e8058e30 con 0x7fa8f00b4170 2024-03-06T07:21:48.322 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:48.322 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":16,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:44.206150+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:48.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 msgr2=0x7fa8d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 0x7fa8d006d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8036000 tx=0x7fa8d8030000 comp rx=0 tx=0).stop 2024-03-06T07:21:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/1364651885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00b4170 msgr2=0x7fa8f0152150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00b4170 0x7fa8f0152150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800d800 tx=0x7fa8e800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/1364651885 shutdown_connections 2024-03-06T07:21:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 0x7fa8d006d990 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8f01f3b90 0x7fa8f01f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f00b4dd0 0x7fa8f00c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 --2- 172.21.15.59:0/1364651885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00b4170 0x7fa8f0152150 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/1364651885 >> 172.21.15.59:0/1364651885 conn(0x7fa8f00d3c40 msgr2=0x7fa8f00b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/1364651885 shutdown_connections 2024-03-06T07:21:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.322+0000 7fa8f4ac9640 1 -- 172.21.15.59:0/1364651885 wait complete. 2024-03-06T07:21:48.335 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:48.336 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:48.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.438+0000 7f094ed79640 1 Processor -- start 2024-03-06T07:21:48.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.438+0000 7f094ed79640 1 -- start start 2024-03-06T07:21:48.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f094ed79640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f09480d4860 0x7f09480d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f094ed79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09480b0e30 0x7f09480b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f094ed79640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09481618b0 con 0x7f09481614d0 2024-03-06T07:21:48.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f094ed79640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09480d4cf0 con 0x7f09480d4860 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f094ed79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09480cf400 con 0x7f09480b0e30 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f0947fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f09480d4860 0x7f09480d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f0947fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f09480d4860 0x7f09480d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41726/0 (socket says 172.21.15.59:41726) 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f094cb15640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f09481614d0 0x7f09480d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39476/0 (socket says 172.21.15.59:39476) 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f0947fff640 1 -- 172.21.15.59:0/2669835281 learned_addr learned my addr 172.21.15.59:0/2669835281 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f094d316640 1 --2- 172.21.15.59:0/2669835281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09480b0e30 0x7f09480b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f0947fff640 1 -- 172.21.15.59:0/2669835281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09480b0e30 msgr2=0x7f09480b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f0947fff640 1 --2- 172.21.15.59:0/2669835281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09480b0e30 0x7f09480b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f0947fff640 1 -- 172.21.15.59:0/2669835281 >> v1:172.21.15.59:6789/0 conn(0x7f09481614d0 legacy=0x7f09480d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f0947fff640 1 -- 172.21.15.59:0/2669835281 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09480cf6a0 con 0x7f09480d4860 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.439+0000 7f09477fe640 1 -- 172.21.15.59:0/2669835281 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450065214 0 0) 0x7f09481618b0 con 0x7f09481614d0 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.440+0000 7f0947fff640 1 --2- 172.21.15.59:0/2669835281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09480d4860 0x7f09480d2020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0934009870 tx=0x7f093402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e256ec1fe81034e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.440+0000 7f09477fe640 1 -- 172.21.15.59:0/2669835281 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934034070 con 0x7f09480d4860 2024-03-06T07:21:48.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.440+0000 7f09477fe640 1 -- 172.21.15.59:0/2669835281 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0934030840 con 0x7f09480d4860 2024-03-06T07:21:48.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.440+0000 7f09477fe640 1 -- 172.21.15.59:0/2669835281 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934030c10 con 0x7f09480d4860 2024-03-06T07:21:48.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.440+0000 7f094ed79640 1 -- 172.21.15.59:0/2669835281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09480d4860 msgr2=0x7f09480d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.440+0000 7f094ed79640 1 --2- 172.21.15.59:0/2669835281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09480d4860 0x7f09480d2020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0934009870 tx=0x7f093402b980 comp rx=0 tx=0).stop 2024-03-06T07:21:48.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.441+0000 7f094ed79640 1 -- 172.21.15.59:0/2669835281 shutdown_connections 2024-03-06T07:21:48.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.441+0000 7f094ed79640 1 --2- 172.21.15.59:0/2669835281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09480b0e30 0x7f09480b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.441+0000 7f094ed79640 1 --2- 172.21.15.59:0/2669835281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09480d4860 0x7f09480d2020 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.441+0000 7f094ed79640 1 -- 172.21.15.59:0/2669835281 >> 172.21.15.59:0/2669835281 conn(0x7f09480d3d40 msgr2=0x7f09480d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.441+0000 7f094ed79640 1 -- 172.21.15.59:0/2669835281 shutdown_connections 2024-03-06T07:21:48.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.441+0000 7f094ed79640 1 -- 172.21.15.59:0/2669835281 wait complete. 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.442+0000 7f094ed79640 1 Processor -- start 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.442+0000 7f094ed79640 1 -- start start 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f094ed79640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09480b0e30 0x7f09481f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f094ed79640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09480d4860 0x7f09480cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f094ed79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09481614d0 0x7f09481f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f094ed79640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09480b0910 con 0x7f09480d4860 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f094ed79640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09480cfc60 con 0x7f09480b0e30 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f094ed79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09480d2560 con 0x7f09481614d0 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f0947fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09480d4860 0x7f09480cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f0947fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09480d4860 0x7f09480cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41730/0 (socket says 172.21.15.59:41730) 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f0947fff640 1 -- 172.21.15.59:0/1423415509 learned_addr learned my addr 172.21.15.59:0/1423415509 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:48.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f094d316640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09481614d0 0x7f09481f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f094cb15640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09480b0e30 0x7f09481f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f0947fff640 1 -- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09481614d0 msgr2=0x7f09481f7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f0947fff640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09481614d0 0x7f09481f7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f0947fff640 1 -- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09480b0e30 msgr2=0x7f09481f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f0947fff640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09480b0e30 0x7f09481f3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f0947fff640 1 -- 172.21.15.59:0/1423415509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09480cafb0 con 0x7f09480d4860 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f0947fff640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09480d4860 0x7f09480cc050 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0934000c00 tx=0x7f09340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f09457fa640 1 -- 172.21.15.59:0/1423415509 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934034040 con 0x7f09480d4860 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f09457fa640 1 -- 172.21.15.59:0/1423415509 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0934030640 con 0x7f09480d4860 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f094ed79640 1 -- 172.21.15.59:0/1423415509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09480cb290 con 0x7f09480d4860 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f09457fa640 1 -- 172.21.15.59:0/1423415509 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934039d40 con 0x7f09480d4860 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.443+0000 7f094ed79640 1 -- 172.21.15.59:0/1423415509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09480cb850 con 0x7f09480d4860 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.444+0000 7f094ed79640 1 -- 172.21.15.59:0/1423415509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094806d700 con 0x7f09480d4860 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.445+0000 7f09457fa640 1 -- 172.21.15.59:0/1423415509 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f093404b060 con 0x7f09480d4860 2024-03-06T07:21:48.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.445+0000 7f09457fa640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f092c06b4d0 0x7f092c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.445+0000 7f09457fa640 1 -- 172.21.15.59:0/1423415509 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f09340aba40 con 0x7f09480d4860 2024-03-06T07:21:48.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.445+0000 7f094cb15640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f092c06b4d0 0x7f092c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.446+0000 7f094cb15640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f092c06b4d0 0x7f092c06d990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f093c000f60 tx=0x7f093c007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.449+0000 7f09457fa640 1 -- 172.21.15.59:0/1423415509 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f093407a6e0 con 0x7f09480d4860 2024-03-06T07:21:48.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.629+0000 7f094ed79640 1 -- 172.21.15.59:0/1423415509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0948068ff0 con 0x7f09480d4860 2024-03-06T07:21:48.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.629+0000 7f09457fa640 1 -- 172.21.15.59:0/1423415509 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f093407e780 con 0x7f09480d4860 2024-03-06T07:21:48.631 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.631+0000 7f094ed79640 1 -- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f092c06b4d0 msgr2=0x7f092c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.631+0000 7f094ed79640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f092c06b4d0 0x7f092c06d990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f093c000f60 tx=0x7f093c007d50 comp rx=0 tx=0).stop 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.631+0000 7f094ed79640 1 -- 172.21.15.59:0/1423415509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09480d4860 msgr2=0x7f09480cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.631+0000 7f094ed79640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09480d4860 0x7f09480cc050 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0934000c00 tx=0x7f09340046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.632+0000 7f094ed79640 1 -- 172.21.15.59:0/1423415509 shutdown_connections 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.632+0000 7f094ed79640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f092c06b4d0 0x7f092c06d990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.632+0000 7f094ed79640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09481614d0 0x7f09481f7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.632+0000 7f094ed79640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09480d4860 0x7f09480cc050 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.632+0000 7f094ed79640 1 --2- 172.21.15.59:0/1423415509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09480b0e30 0x7f09481f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.632+0000 7f094ed79640 1 -- 172.21.15.59:0/1423415509 >> 172.21.15.59:0/1423415509 conn(0x7f09480d3d40 msgr2=0x7f09480b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.632+0000 7f094ed79640 1 -- 172.21.15.59:0/1423415509 shutdown_connections 2024-03-06T07:21:48.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.632+0000 7f094ed79640 1 -- 172.21.15.59:0/1423415509 wait complete. 2024-03-06T07:21:48.644 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:48.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.743+0000 7f3bc2681640 1 Processor -- start 2024-03-06T07:21:48.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.743+0000 7f3bc2681640 1 -- start start 2024-03-06T07:21:48.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bc2681640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3bbc15e090 0x7f3bbc0d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bc2681640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d0650 0x7f3bbc0d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bc2681640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bbc15e470 con 0x7f3bbc0d2920 2024-03-06T07:21:48.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bc2681640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bbc0d18f0 con 0x7f3bbc15e090 2024-03-06T07:21:48.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bc2681640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0b3cf0 con 0x7f3bbc0d0650 2024-03-06T07:21:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bc0c1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d0650 0x7f3bbc0d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bc0c1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d0650 0x7f3bbc0d0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:58374/0 (socket says 172.21.15.59:58374) 2024-03-06T07:21:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bc0c1e640 1 -- 172.21.15.59:0/3613357781 learned_addr learned my addr 172.21.15.59:0/3613357781 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bbbfff640 1 --2- 172.21.15.59:0/3613357781 >> v2:172.21.15.164:3300/0 conn(0x7f3bbc15e090 0x7f3bbc0d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bbbfff640 1 -- 172.21.15.59:0/3613357781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d0650 msgr2=0x7f3bbc0d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bbbfff640 1 --2- 172.21.15.59:0/3613357781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d0650 0x7f3bbc0d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bbbfff640 1 -- 172.21.15.59:0/3613357781 >> v1:172.21.15.164:6789/0 conn(0x7f3bbc0d2920 legacy=0x7f3bbc0b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bbbfff640 1 -- 172.21.15.59:0/3613357781 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc0d1b90 con 0x7f3bbc15e090 2024-03-06T07:21:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.744+0000 7f3bc0c1e640 1 --2- 172.21.15.59:0/3613357781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d0650 0x7f3bbc0d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:48.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.745+0000 7f3bbbfff640 1 --2- 172.21.15.59:0/3613357781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bbc15e090 0x7f3bbc0d47a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40097a0 tx=0x7f3ba402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6b192a9cd647dcb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.745+0000 7f3bbaffd640 1 -- 172.21.15.59:0/3613357781 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4035020 con 0x7f3bbc15e090 2024-03-06T07:21:48.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.745+0000 7f3bbaffd640 1 -- 172.21.15.59:0/3613357781 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba4031830 con 0x7f3bbc15e090 2024-03-06T07:21:48.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.745+0000 7f3bbaffd640 1 -- 172.21.15.59:0/3613357781 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4031c00 con 0x7f3bbc15e090 2024-03-06T07:21:48.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.746+0000 7f3bc2681640 1 -- 172.21.15.59:0/3613357781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bbc15e090 msgr2=0x7f3bbc0d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.746+0000 7f3bc2681640 1 --2- 172.21.15.59:0/3613357781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bbc15e090 0x7f3bbc0d47a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40097a0 tx=0x7f3ba402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.746+0000 7f3bc2681640 1 -- 172.21.15.59:0/3613357781 shutdown_connections 2024-03-06T07:21:48.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.746+0000 7f3bc2681640 1 --2- 172.21.15.59:0/3613357781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d0650 0x7f3bbc0d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.746+0000 7f3bc2681640 1 --2- 172.21.15.59:0/3613357781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bbc15e090 0x7f3bbc0d47a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.746+0000 7f3bc2681640 1 -- 172.21.15.59:0/3613357781 >> 172.21.15.59:0/3613357781 conn(0x7f3bbc0d3c80 msgr2=0x7f3bbc0d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.746+0000 7f3bc2681640 1 -- 172.21.15.59:0/3613357781 shutdown_connections 2024-03-06T07:21:48.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.746+0000 7f3bc2681640 1 -- 172.21.15.59:0/3613357781 wait complete. 2024-03-06T07:21:48.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.747+0000 7f3bc2681640 1 Processor -- start 2024-03-06T07:21:48.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.747+0000 7f3bc2681640 1 -- start start 2024-03-06T07:21:48.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bc2681640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bbc0d0650 0x7f3bbc0cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bc2681640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d2920 0x7f3bbc0cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bc2681640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bbc15e090 0x7f3bbc14d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bc2681640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0b2df0 con 0x7f3bbc0d0650 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bc2681640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0cee30 con 0x7f3bbc15e090 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bc2681640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0ce800 con 0x7f3bbc0d2920 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bbbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bbc0d0650 0x7f3bbc0cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bbbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bbc0d0650 0x7f3bbc0cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41744/0 (socket says 172.21.15.59:41744) 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bbbfff640 1 -- 172.21.15.59:0/2113887294 learned_addr learned my addr 172.21.15.59:0/2113887294 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bc0c1e640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bbc15e090 0x7f3bbc14d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bbb7fe640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d2920 0x7f3bbc0cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bbbfff640 1 -- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d2920 msgr2=0x7f3bbc0cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bbbfff640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d2920 0x7f3bbc0cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bbbfff640 1 -- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bbc15e090 msgr2=0x7f3bbc14d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bbbfff640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bbc15e090 0x7f3bbc14d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.748+0000 7f3bbbfff640 1 -- 172.21.15.59:0/2113887294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc0cb290 con 0x7f3bbc0d0650 2024-03-06T07:21:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.749+0000 7f3bbbfff640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bbc0d0650 0x7f3bbc0cd190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40098c0 tx=0x7f3ba40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.749+0000 7f3bb8ff9640 1 -- 172.21.15.59:0/2113887294 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4040070 con 0x7f3bbc0d0650 2024-03-06T07:21:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.749+0000 7f3bb8ff9640 1 -- 172.21.15.59:0/2113887294 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba403e070 con 0x7f3bbc0d0650 2024-03-06T07:21:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.749+0000 7f3bb8ff9640 1 -- 172.21.15.59:0/2113887294 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4035040 con 0x7f3bbc0d0650 2024-03-06T07:21:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.749+0000 7f3bc2681640 1 -- 172.21.15.59:0/2113887294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bbc14d780 con 0x7f3bbc0d0650 2024-03-06T07:21:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.749+0000 7f3bc2681640 1 -- 172.21.15.59:0/2113887294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bbc0cb810 con 0x7f3bbc0d0650 2024-03-06T07:21:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.750+0000 7f3bc2681640 1 -- 172.21.15.59:0/2113887294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc04f330 con 0x7f3bbc0d0650 2024-03-06T07:21:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.750+0000 7f3bb8ff9640 1 -- 172.21.15.59:0/2113887294 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ba4038020 con 0x7f3bbc0d0650 2024-03-06T07:21:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.751+0000 7f3bb8ff9640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3b9c06b4d0 0x7f3b9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.751+0000 7f3bb8ff9640 1 -- 172.21.15.59:0/2113887294 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3ba40ab6b0 con 0x7f3bbc0d0650 2024-03-06T07:21:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.751+0000 7f3bbb7fe640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3b9c06b4d0 0x7f3b9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.751+0000 7f3bbb7fe640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3b9c06b4d0 0x7f3b9c06d990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0b98a0 tx=0x7f3bac009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.755+0000 7f3bb8ff9640 1 -- 172.21.15.59:0/2113887294 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ba407a1a0 con 0x7f3bbc0d0650 2024-03-06T07:21:48.891 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:48.894 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:48.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.976+0000 7f3bc2681640 1 -- 172.21.15.59:0/2113887294 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bbc1623e0 con 0x7f3bbc0d0650 2024-03-06T07:21:48.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.976+0000 7f3bb8ff9640 1 -- 172.21.15.59:0/2113887294 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f3ba407e240 con 0x7f3bbc0d0650 2024-03-06T07:21:48.978 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:48.978 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2132697832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2132697832},{"type":"v1","addr":"172.21.15.59:6835","nonce":2132697832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:21:48.978 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 2 2024-03-06T07:21:48.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 -- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3b9c06b4d0 msgr2=0x7f3b9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3b9c06b4d0 0x7f3b9c06d990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0b98a0 tx=0x7f3bac009040 comp rx=0 tx=0).stop 2024-03-06T07:21:48.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 -- 172.21.15.59:0/2113887294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bbc0d0650 msgr2=0x7f3bbc0cd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bbc0d0650 0x7f3bbc0cd190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40098c0 tx=0x7f3ba40046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 -- 172.21.15.59:0/2113887294 shutdown_connections 2024-03-06T07:21:48.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3b9c06b4d0 0x7f3b9c06d990 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bbc15e090 0x7f3bbc14d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bbc0d2920 0x7f3bbc0cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 --2- 172.21.15.59:0/2113887294 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3bbc0d0650 0x7f3bbc0cd190 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 -- 172.21.15.59:0/2113887294 >> 172.21.15.59:0/2113887294 conn(0x7f3bbc0d3c80 msgr2=0x7f3bbc1618a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 -- 172.21.15.59:0/2113887294 shutdown_connections 2024-03-06T07:21:48.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:48.978+0000 7f3bc2681640 1 -- 172.21.15.59:0/2113887294 wait complete. 2024-03-06T07:21:50.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:51.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.102+0000 7fc223488640 1 Processor -- start 2024-03-06T07:21:51.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.102+0000 7fc223488640 1 -- start start 2024-03-06T07:21:51.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc223488640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc21c0d22b0 0x7fc21c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc223488640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc21c0b41f0 0x7fc21c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc223488640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21c0b3430 con 0x7fc21c15dfe0 2024-03-06T07:21:51.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc223488640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21c0d2c50 con 0x7fc21c0b41f0 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc223488640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21c0b31e0 con 0x7fc21c0d22b0 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc221a25640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc21c0b41f0 0x7fc21c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc221a25640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc21c0b41f0 0x7fc21c0b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41754/0 (socket says 172.21.15.59:41754) 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc221a25640 1 -- 172.21.15.59:0/1390687046 learned_addr learned my addr 172.21.15.59:0/1390687046 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc220a23640 1 --2- 172.21.15.59:0/1390687046 >> v2:172.21.15.164:3300/0 conn(0x7fc21c0d22b0 0x7fc21c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc221a25640 1 -- 172.21.15.59:0/1390687046 >> v2:172.21.15.164:3300/0 conn(0x7fc21c0d22b0 msgr2=0x7fc21c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc221a25640 1 --2- 172.21.15.59:0/1390687046 >> v2:172.21.15.164:3300/0 conn(0x7fc21c0d22b0 0x7fc21c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc221a25640 1 -- 172.21.15.59:0/1390687046 >> v1:172.21.15.164:6789/0 conn(0x7fc21c15dfe0 legacy=0x7fc21c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc221a25640 1 -- 172.21.15.59:0/1390687046 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c0b3710 con 0x7fc21c0b41f0 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.103+0000 7fc220a23640 1 --2- 172.21.15.59:0/1390687046 >> v2:172.21.15.164:3300/0 conn(0x7fc21c0d22b0 0x7fc21c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.104+0000 7fc221a25640 1 --2- 172.21.15.59:0/1390687046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc21c0b41f0 0x7fc21c0b4680 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc214009870 tx=0x7fc21402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7101212a3f3c8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.104+0000 7fc21bfff640 1 -- 172.21.15.59:0/1390687046 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc214034070 con 0x7fc21c0b41f0 2024-03-06T07:21:51.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.104+0000 7fc21bfff640 1 -- 172.21.15.59:0/1390687046 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc214030860 con 0x7fc21c0b41f0 2024-03-06T07:21:51.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.104+0000 7fc21bfff640 1 -- 172.21.15.59:0/1390687046 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc214030c30 con 0x7fc21c0b41f0 2024-03-06T07:21:51.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.105+0000 7fc223488640 1 -- 172.21.15.59:0/1390687046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc21c0b41f0 msgr2=0x7fc21c0b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.105+0000 7fc223488640 1 --2- 172.21.15.59:0/1390687046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc21c0b41f0 0x7fc21c0b4680 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc214009870 tx=0x7fc21402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.105+0000 7fc223488640 1 -- 172.21.15.59:0/1390687046 shutdown_connections 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.105+0000 7fc223488640 1 --2- 172.21.15.59:0/1390687046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc21c0b41f0 0x7fc21c0b4680 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.105+0000 7fc223488640 1 --2- 172.21.15.59:0/1390687046 >> v2:172.21.15.164:3300/0 conn(0x7fc21c0d22b0 0x7fc21c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.105+0000 7fc223488640 1 -- 172.21.15.59:0/1390687046 >> 172.21.15.59:0/1390687046 conn(0x7fc21c0d3cd0 msgr2=0x7fc21c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.105+0000 7fc223488640 1 -- 172.21.15.59:0/1390687046 shutdown_connections 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.105+0000 7fc223488640 1 -- 172.21.15.59:0/1390687046 wait complete. 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.106+0000 7fc223488640 1 Processor -- start 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.106+0000 7fc223488640 1 -- start start 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc223488640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc21c0b41f0 0x7fc21c15c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc223488640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc21c0d22b0 0x7fc21c0b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc223488640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc21c15dfe0 0x7fc21c1603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc223488640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c0cfa00 con 0x7fc21c0b41f0 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc223488640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c0b3d60 con 0x7fc21c0d22b0 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc223488640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c0d1c70 con 0x7fc21c15dfe0 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc221224640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc21c0b41f0 0x7fc21c15c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc221224640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc21c0b41f0 0x7fc21c15c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41770/0 (socket says 172.21.15.59:41770) 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc221224640 1 -- 172.21.15.59:0/1452420964 learned_addr learned my addr 172.21.15.59:0/1452420964 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc221a25640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc21c15dfe0 0x7fc21c1603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc220a23640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc21c0d22b0 0x7fc21c0b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc221224640 1 -- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc21c15dfe0 msgr2=0x7fc21c1603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc221224640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc21c15dfe0 0x7fc21c1603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc221224640 1 -- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc21c0d22b0 msgr2=0x7fc21c0b85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc221224640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc21c0d22b0 0x7fc21c0b85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc221224640 1 -- 172.21.15.59:0/1452420964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c0cd170 con 0x7fc21c0b41f0 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc221a25640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc21c15dfe0 0x7fc21c1603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.107+0000 7fc220a23640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc21c0d22b0 0x7fc21c0b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.108+0000 7fc221224640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc21c0b41f0 0x7fc21c15c340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00c730 tx=0x7fc20c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.108+0000 7fc219ffb640 1 -- 172.21.15.59:0/1452420964 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c016020 con 0x7fc21c0b41f0 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.108+0000 7fc219ffb640 1 -- 172.21.15.59:0/1452420964 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20c006d50 con 0x7fc21c0b41f0 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.108+0000 7fc219ffb640 1 -- 172.21.15.59:0/1452420964 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c003bb0 con 0x7fc21c0b41f0 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.108+0000 7fc223488640 1 -- 172.21.15.59:0/1452420964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21c0cd450 con 0x7fc21c0b41f0 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.108+0000 7fc223488640 1 -- 172.21.15.59:0/1452420964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21c0cda10 con 0x7fc21c0b41f0 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.109+0000 7fc223488640 1 -- 172.21.15.59:0/1452420964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21c04f330 con 0x7fc21c0b41f0 2024-03-06T07:21:51.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.109+0000 7fc219ffb640 1 -- 172.21.15.59:0/1452420964 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc20c00a850 con 0x7fc21c0b41f0 2024-03-06T07:21:51.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.110+0000 7fc219ffb640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc20806b4d0 0x7fc20806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.110+0000 7fc219ffb640 1 -- 172.21.15.59:0/1452420964 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc20c089b00 con 0x7fc21c0b41f0 2024-03-06T07:21:51.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.110+0000 7fc220a23640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc20806b4d0 0x7fc20806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.110+0000 7fc220a23640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc20806b4d0 0x7fc20806d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc21c0b3150 tx=0x7fc210008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.114+0000 7fc219ffb640 1 -- 172.21.15.59:0/1452420964 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc20c020020 con 0x7fc21c0b41f0 2024-03-06T07:21:51.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.334+0000 7fc223488640 1 -- 172.21.15.59:0/1452420964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc21c15d2e0 con 0x7fc21c0b41f0 2024-03-06T07:21:51.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.334+0000 7fc219ffb640 1 -- 172.21.15.59:0/1452420964 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3697 (secure 0 0 0) 0x7fc20c019310 con 0x7fc21c0b41f0 2024-03-06T07:21:51.336 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:51.336 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1759048959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1759048959},{"type":"v1","addr":"172.21.15.164:6836","nonce":1759048959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/550023536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":550023536},{"type":"v1","addr":"172.21.15.59:6833","nonce":550023536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2132697832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2132697832},{"type":"v1","addr":"172.21.15.59:6835","nonce":2132697832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4302,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1957940318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1957940318},{"type":"v1","addr":"172.21.15.164:6837","nonce":1957940318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-06T07:21:51.337 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 3 2024-03-06T07:21:51.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 -- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc20806b4d0 msgr2=0x7fc20806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc20806b4d0 0x7fc20806d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc21c0b3150 tx=0x7fc210008040 comp rx=0 tx=0).stop 2024-03-06T07:21:51.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 -- 172.21.15.59:0/1452420964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc21c0b41f0 msgr2=0x7fc21c15c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc21c0b41f0 0x7fc21c15c340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00c730 tx=0x7fc20c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 -- 172.21.15.59:0/1452420964 shutdown_connections 2024-03-06T07:21:51.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc20806b4d0 0x7fc20806d990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc21c15dfe0 0x7fc21c1603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc21c0d22b0 0x7fc21c0b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 --2- 172.21.15.59:0/1452420964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc21c0b41f0 0x7fc21c15c340 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 -- 172.21.15.59:0/1452420964 >> 172.21.15.59:0/1452420964 conn(0x7fc21c0d3cd0 msgr2=0x7fc21c06ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 -- 172.21.15.59:0/1452420964 shutdown_connections 2024-03-06T07:21:51.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.336+0000 7fc223488640 1 -- 172.21.15.59:0/1452420964 wait complete. 2024-03-06T07:21:51.349 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:51.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.448+0000 7f0d45b3e640 1 Processor -- start 2024-03-06T07:21:51.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.448+0000 7f0d45b3e640 1 -- start start 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d45b3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d4015dfe0 0x7f0d400cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d45b3e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0d400cf830 0x7f0d400d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d45b3e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d400b3430 con 0x7f0d400d27f0 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d45b3e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d400d2c50 con 0x7f0d400cf830 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d45b3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d400b31e0 con 0x7f0d4015dfe0 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d3f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d4015dfe0 0x7f0d400cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d3effd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0d400cf830 0x7f0d400d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d3ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0d400d27f0 0x7f0d400d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:54698/0 (socket says 172.21.15.59:54698) 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d3ffff640 1 -- 172.21.15.59:0/654971322 learned_addr learned my addr 172.21.15.59:0/654971322 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d3effd640 1 --2- 172.21.15.59:0/654971322 >> v2:172.21.15.164:3300/0 conn(0x7f0d400cf830 0x7f0d400d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:48992/0 (socket says 172.21.15.59:48992) 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d3f7fe640 1 -- 172.21.15.59:0/654971322 >> v2:172.21.15.164:3300/0 conn(0x7f0d400cf830 msgr2=0x7f0d400d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d3f7fe640 1 --2- 172.21.15.59:0/654971322 >> v2:172.21.15.164:3300/0 conn(0x7f0d400cf830 0x7f0d400d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d3f7fe640 1 -- 172.21.15.59:0/654971322 >> v1:172.21.15.164:6789/0 conn(0x7f0d400d27f0 legacy=0x7f0d400d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d3f7fe640 1 -- 172.21.15.59:0/654971322 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d400b3710 con 0x7f0d4015dfe0 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d3e7fc640 1 -- 172.21.15.59:0/654971322 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023507735 0 0) 0x7f0d400b3430 con 0x7f0d400d27f0 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.449+0000 7f0d3effd640 1 --2- 172.21.15.59:0/654971322 >> v2:172.21.15.164:3300/0 conn(0x7f0d400cf830 0x7f0d400d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.450+0000 7f0d3f7fe640 1 --2- 172.21.15.59:0/654971322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d4015dfe0 0x7f0d400cf2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0d300097a0 tx=0x7f0d3002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e854bd1c8562aba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.450+0000 7f0d3e7fc640 1 -- 172.21.15.59:0/654971322 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30035020 con 0x7f0d4015dfe0 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.450+0000 7f0d3e7fc640 1 -- 172.21.15.59:0/654971322 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d30031830 con 0x7f0d4015dfe0 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.450+0000 7f0d3e7fc640 1 -- 172.21.15.59:0/654971322 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30031c00 con 0x7f0d4015dfe0 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.451+0000 7f0d45b3e640 1 -- 172.21.15.59:0/654971322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d4015dfe0 msgr2=0x7f0d400cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.451+0000 7f0d45b3e640 1 --2- 172.21.15.59:0/654971322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d4015dfe0 0x7f0d400cf2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0d300097a0 tx=0x7f0d3002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.451+0000 7f0d45b3e640 1 -- 172.21.15.59:0/654971322 shutdown_connections 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.451+0000 7f0d45b3e640 1 --2- 172.21.15.59:0/654971322 >> v2:172.21.15.164:3300/0 conn(0x7f0d400cf830 0x7f0d400d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.451+0000 7f0d45b3e640 1 --2- 172.21.15.59:0/654971322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d4015dfe0 0x7f0d400cf2f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.451+0000 7f0d45b3e640 1 -- 172.21.15.59:0/654971322 >> 172.21.15.59:0/654971322 conn(0x7f0d400d3cd0 msgr2=0x7f0d400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.451+0000 7f0d45b3e640 1 -- 172.21.15.59:0/654971322 shutdown_connections 2024-03-06T07:21:51.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.451+0000 7f0d45b3e640 1 -- 172.21.15.59:0/654971322 wait complete. 2024-03-06T07:21:51.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.452+0000 7f0d45b3e640 1 Processor -- start 2024-03-06T07:21:51.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.452+0000 7f0d45b3e640 1 -- start start 2024-03-06T07:21:51.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d45b3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d400cf830 0x7f0d400a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d45b3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d400d27f0 0x7f0d400b8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d45b3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d4015dfe0 0x7f0d401603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d45b3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d400d1c70 con 0x7f0d4015dfe0 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d45b3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d400b3d60 con 0x7f0d400d27f0 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d45b3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d400d1df0 con 0x7f0d400cf830 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d3ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d4015dfe0 0x7f0d401603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d3ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d4015dfe0 0x7f0d401603b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41772/0 (socket says 172.21.15.59:41772) 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d3ffff640 1 -- 172.21.15.59:0/2734539324 learned_addr learned my addr 172.21.15.59:0/2734539324 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d3effd640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d400d27f0 0x7f0d400b8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d3f7fe640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d400cf830 0x7f0d400a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d3ffff640 1 -- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d400cf830 msgr2=0x7f0d400a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d3ffff640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d400cf830 0x7f0d400a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d3ffff640 1 -- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d400d27f0 msgr2=0x7f0d400b8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d3ffff640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d400d27f0 0x7f0d400b8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.453+0000 7f0d3ffff640 1 -- 172.21.15.59:0/2734539324 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d400cbcb0 con 0x7f0d4015dfe0 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.454+0000 7f0d3ffff640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d4015dfe0 0x7f0d401603b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800c7a0 tx=0x7f0d3800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.454+0000 7f0d27fff640 1 -- 172.21.15.59:0/2734539324 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d38018070 con 0x7f0d4015dfe0 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.454+0000 7f0d27fff640 1 -- 172.21.15.59:0/2734539324 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3800a640 con 0x7f0d4015dfe0 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.454+0000 7f0d45b3e640 1 -- 172.21.15.59:0/2734539324 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d400cbf90 con 0x7f0d4015dfe0 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.454+0000 7f0d27fff640 1 -- 172.21.15.59:0/2734539324 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3800aab0 con 0x7f0d4015dfe0 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.454+0000 7f0d45b3e640 1 -- 172.21.15.59:0/2734539324 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d400cc550 con 0x7f0d4015dfe0 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.455+0000 7f0d45b3e640 1 -- 172.21.15.59:0/2734539324 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4008d3c0 con 0x7f0d4015dfe0 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.455+0000 7f0d27fff640 1 -- 172.21.15.59:0/2734539324 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d3801b070 con 0x7f0d4015dfe0 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.455+0000 7f0d27fff640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d2c06b4d0 0x7f0d2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.456+0000 7f0d27fff640 1 -- 172.21.15.59:0/2734539324 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0d38089460 con 0x7f0d4015dfe0 2024-03-06T07:21:51.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.456+0000 7f0d3f7fe640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d2c06b4d0 0x7f0d2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.456+0000 7f0d3f7fe640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d2c06b4d0 0x7f0d2c06d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0d30036000 tx=0x7f0d30008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.460+0000 7f0d27fff640 1 -- 172.21.15.59:0/2734539324 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d4008d3c0 con 0x7f0d4015dfe0 2024-03-06T07:21:51.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.640+0000 7f0d45b3e640 1 -- 172.21.15.59:0/2734539324 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d4004f330 con 0x7f0d4015dfe0 2024-03-06T07:21:51.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.641+0000 7f0d27fff640 1 -- 172.21.15.59:0/2734539324 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12872 (secure 0 0 0) 0x7f0d380586c0 con 0x7f0d4015dfe0 2024-03-06T07:21:51.642 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:51.643 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":16,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:44.206150+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:51.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.642+0000 7f0d45b3e640 1 -- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d2c06b4d0 msgr2=0x7f0d2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.642+0000 7f0d45b3e640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d2c06b4d0 0x7f0d2c06d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0d30036000 tx=0x7f0d30008030 comp rx=0 tx=0).stop 2024-03-06T07:21:51.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.642+0000 7f0d45b3e640 1 -- 172.21.15.59:0/2734539324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d4015dfe0 msgr2=0x7f0d401603b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.642+0000 7f0d45b3e640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d4015dfe0 0x7f0d401603b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800c7a0 tx=0x7f0d3800cc60 comp rx=0 tx=0).stop 2024-03-06T07:21:51.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.642+0000 7f0d45b3e640 1 -- 172.21.15.59:0/2734539324 shutdown_connections 2024-03-06T07:21:51.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.642+0000 7f0d45b3e640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d2c06b4d0 0x7f0d2c06d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.642+0000 7f0d45b3e640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d4015dfe0 0x7f0d401603b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.642+0000 7f0d45b3e640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d400d27f0 0x7f0d400b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.642+0000 7f0d45b3e640 1 --2- 172.21.15.59:0/2734539324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d400cf830 0x7f0d400a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.643+0000 7f0d45b3e640 1 -- 172.21.15.59:0/2734539324 >> 172.21.15.59:0/2734539324 conn(0x7f0d400d3cd0 msgr2=0x7f0d4006d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.643+0000 7f0d45b3e640 1 -- 172.21.15.59:0/2734539324 shutdown_connections 2024-03-06T07:21:51.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.643+0000 7f0d45b3e640 1 -- 172.21.15.59:0/2734539324 wait complete. 2024-03-06T07:21:51.655 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:51.655 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:51.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.757+0000 7fa5b9a4e640 1 Processor -- start 2024-03-06T07:21:51.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.757+0000 7fa5b9a4e640 1 -- start start 2024-03-06T07:21:51.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.758+0000 7fa5b9a4e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b40d06e0 0x7fa5b415fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.758+0000 7fa5b9a4e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa5b40cf200 0x7fa5b40cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.758+0000 7fa5b9a4e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b40d47f0 con 0x7fa5b40d0ac0 2024-03-06T07:21:51.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.758+0000 7fa5b9a4e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b40ced70 con 0x7fa5b40cf200 2024-03-06T07:21:51.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.758+0000 7fa5b9a4e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40d1ab0 con 0x7fa5b40d06e0 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.758+0000 7fa5b37fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa5b40cf200 0x7fa5b40cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.758+0000 7fa5b37fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa5b40cf200 0x7fa5b40cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41788/0 (socket says 172.21.15.59:41788) 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.758+0000 7fa5b37fe640 1 -- 172.21.15.59:0/1821184033 learned_addr learned my addr 172.21.15.59:0/1821184033 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.758+0000 7fa5b2ffd640 1 --2- 172.21.15.59:0/1821184033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b40d06e0 0x7fa5b415fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.759+0000 7fa5b37fe640 1 -- 172.21.15.59:0/1821184033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b40d06e0 msgr2=0x7fa5b415fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.759+0000 7fa5b37fe640 1 --2- 172.21.15.59:0/1821184033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b40d06e0 0x7fa5b415fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.759+0000 7fa5b37fe640 1 -- 172.21.15.59:0/1821184033 >> v1:172.21.15.164:6789/0 conn(0x7fa5b40d0ac0 legacy=0x7fa5b40b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.759+0000 7fa5b37fe640 1 -- 172.21.15.59:0/1821184033 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b40d1d90 con 0x7fa5b40cf200 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.759+0000 7fa5b37fe640 1 --2- 172.21.15.59:0/1821184033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b40cf200 0x7fa5b40cf690 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009870 tx=0x7fa5ac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efeebd7c8600579 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.760+0000 7fa5b1ffb640 1 -- 172.21.15.59:0/1821184033 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac034070 con 0x7fa5b40cf200 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.760+0000 7fa5b1ffb640 1 -- 172.21.15.59:0/1821184033 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac030840 con 0x7fa5b40cf200 2024-03-06T07:21:51.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.760+0000 7fa5b1ffb640 1 -- 172.21.15.59:0/1821184033 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac030c10 con 0x7fa5b40cf200 2024-03-06T07:21:51.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.760+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1821184033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b40cf200 msgr2=0x7fa5b40cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.760+0000 7fa5b9a4e640 1 --2- 172.21.15.59:0/1821184033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b40cf200 0x7fa5b40cf690 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009870 tx=0x7fa5ac02b980 comp rx=0 tx=0).stop 2024-03-06T07:21:51.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.760+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1821184033 shutdown_connections 2024-03-06T07:21:51.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.760+0000 7fa5b9a4e640 1 --2- 172.21.15.59:0/1821184033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b40cf200 0x7fa5b40cf690 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.760+0000 7fa5b9a4e640 1 --2- 172.21.15.59:0/1821184033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b40d06e0 0x7fa5b415fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.760+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1821184033 >> 172.21.15.59:0/1821184033 conn(0x7fa5b40d3cd0 msgr2=0x7fa5b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.760+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1821184033 shutdown_connections 2024-03-06T07:21:51.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.760+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1821184033 wait complete. 2024-03-06T07:21:51.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.761+0000 7fa5b9a4e640 1 Processor -- start 2024-03-06T07:21:51.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.761+0000 7fa5b9a4e640 1 -- start start 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b9a4e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b40cf200 0x7fa5b40b9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b9a4e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5b40d06e0 0x7fa5b4153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b9a4e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b40d0ac0 0x7fa5b41e8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b9a4e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40b44d0 con 0x7fa5b40cf200 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b9a4e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40ce990 con 0x7fa5b40d06e0 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b9a4e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40d0f00 con 0x7fa5b40d0ac0 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b40cf200 0x7fa5b40b9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b40cf200 0x7fa5b40b9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41792/0 (socket says 172.21.15.59:41792) 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b2ffd640 1 -- 172.21.15.59:0/1068981056 learned_addr learned my addr 172.21.15.59:0/1068981056 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b2ffd640 1 -- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b40d0ac0 msgr2=0x7fa5b41e8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b2ffd640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b40d0ac0 0x7fa5b41e8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b2ffd640 1 -- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5b40d06e0 msgr2=0x7fa5b4153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b2ffd640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5b40d06e0 0x7fa5b4153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.762+0000 7fa5b2ffd640 1 -- 172.21.15.59:0/1068981056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b40b7470 con 0x7fa5b40cf200 2024-03-06T07:21:51.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.763+0000 7fa5b2ffd640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b40cf200 0x7fa5b40b9280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009d10 tx=0x7fa5a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.763+0000 7fa5abfff640 1 -- 172.21.15.59:0/1068981056 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4018020 con 0x7fa5b40cf200 2024-03-06T07:21:51.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.763+0000 7fa5abfff640 1 -- 172.21.15.59:0/1068981056 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a4014480 con 0x7fa5b40cf200 2024-03-06T07:21:51.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.763+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1068981056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b40c9e70 con 0x7fa5b40cf200 2024-03-06T07:21:51.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.763+0000 7fa5abfff640 1 -- 172.21.15.59:0/1068981056 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4014830 con 0x7fa5b40cf200 2024-03-06T07:21:51.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.763+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1068981056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b40ca430 con 0x7fa5b40cf200 2024-03-06T07:21:51.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.764+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1068981056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b40d32e0 con 0x7fa5b40cf200 2024-03-06T07:21:51.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.764+0000 7fa5abfff640 1 -- 172.21.15.59:0/1068981056 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5a4005d10 con 0x7fa5b40cf200 2024-03-06T07:21:51.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.765+0000 7fa5abfff640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b500 0x7fa59406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:51.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.765+0000 7fa5abfff640 1 -- 172.21.15.59:0/1068981056 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa5a408a8c0 con 0x7fa5b40cf200 2024-03-06T07:21:51.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.765+0000 7fa5b27fc640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b500 0x7fa59406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:51.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.766+0000 7fa5b27fc640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b500 0x7fa59406d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa59c000f60 tx=0x7fa59c0079d0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:51.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.769+0000 7fa5abfff640 1 -- 172.21.15.59:0/1068981056 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5b40d32e0 con 0x7fa5b40cf200 2024-03-06T07:21:51.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.948+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1068981056 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5b40a1c80 con 0x7fa5b40cf200 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.948+0000 7fa5abfff640 1 -- 172.21.15.59:0/1068981056 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5a4022070 con 0x7fa5b40cf200 2024-03-06T07:21:51.950 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b500 msgr2=0x7fa59406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b500 0x7fa59406d9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa59c000f60 tx=0x7fa59c0079d0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1068981056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b40cf200 msgr2=0x7fa5b40b9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b40cf200 0x7fa5b40b9280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009d10 tx=0x7fa5a400c490 comp rx=0 tx=0).stop 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1068981056 shutdown_connections 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b500 0x7fa59406d9c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b40d0ac0 0x7fa5b41e8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5b40d06e0 0x7fa5b4153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 --2- 172.21.15.59:0/1068981056 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b40cf200 0x7fa5b40b9280 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1068981056 >> 172.21.15.59:0/1068981056 conn(0x7fa5b40d3cd0 msgr2=0x7fa5b406b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1068981056 shutdown_connections 2024-03-06T07:21:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:51.950+0000 7fa5b9a4e640 1 -- 172.21.15.59:0/1068981056 wait complete. 2024-03-06T07:21:51.963 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:21:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.062+0000 7f0804b3e640 1 Processor -- start 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.062+0000 7f0804b3e640 1 -- start start 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f0804b3e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f08000d0320 0x7f08000b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f0804b3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08000b54b0 0x7f08000cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f0804b3e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080015c270 con 0x7f08000d0700 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f0804b3e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08000b04e0 con 0x7f08000d0320 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f0804b3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08000d0e90 con 0x7f08000b54b0 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f07fe59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f08000d0320 0x7f08000b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f07fe59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f08000d0320 0x7f08000b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41798/0 (socket says 172.21.15.59:41798) 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f07fe59c640 1 -- 172.21.15.59:0/3086390587 learned_addr learned my addr 172.21.15.59:0/3086390587 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f07fed9d640 1 --2- 172.21.15.59:0/3086390587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08000b54b0 0x7f08000cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f07fe59c640 1 -- 172.21.15.59:0/3086390587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08000b54b0 msgr2=0x7f08000cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f07fe59c640 1 --2- 172.21.15.59:0/3086390587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08000b54b0 0x7f08000cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f07fe59c640 1 -- 172.21.15.59:0/3086390587 >> v1:172.21.15.164:6789/0 conn(0x7f08000d0700 legacy=0x7f08000ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f07fe59c640 1 -- 172.21.15.59:0/3086390587 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08000b5ae0 con 0x7f08000d0320 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.063+0000 7f07fed9d640 1 --2- 172.21.15.59:0/3086390587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08000b54b0 0x7f08000cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f07fe59c640 1 --2- 172.21.15.59:0/3086390587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08000d0320 0x7f08000b4f70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009870 tx=0x7f07f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37938986900650af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f07fd59a640 1 -- 172.21.15.59:0/3086390587 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f0035020 con 0x7f08000d0320 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f07fd59a640 1 -- 172.21.15.59:0/3086390587 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f0031640 con 0x7f08000d0320 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f07fd59a640 1 -- 172.21.15.59:0/3086390587 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f002d040 con 0x7f08000d0320 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f0804b3e640 1 -- 172.21.15.59:0/3086390587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08000d0320 msgr2=0x7f08000b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f0804b3e640 1 --2- 172.21.15.59:0/3086390587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08000d0320 0x7f08000b4f70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009870 tx=0x7f07f002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f0804b3e640 1 -- 172.21.15.59:0/3086390587 shutdown_connections 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f0804b3e640 1 --2- 172.21.15.59:0/3086390587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08000b54b0 0x7f08000cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f0804b3e640 1 --2- 172.21.15.59:0/3086390587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08000d0320 0x7f08000b4f70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f0804b3e640 1 -- 172.21.15.59:0/3086390587 >> 172.21.15.59:0/3086390587 conn(0x7f08000724d0 msgr2=0x7f08000728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f0804b3e640 1 -- 172.21.15.59:0/3086390587 shutdown_connections 2024-03-06T07:21:52.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.064+0000 7f0804b3e640 1 -- 172.21.15.59:0/3086390587 wait complete. 2024-03-06T07:21:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.066+0000 7f0804b3e640 1 Processor -- start 2024-03-06T07:21:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.066+0000 7f0804b3e640 1 -- start start 2024-03-06T07:21:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f0804b3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08000b54b0 0x7f08001464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f0804b3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08000d0320 0x7f0800158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f0804b3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08000d0700 0x7f08001592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f0804b3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08000b60a0 con 0x7f08000d0700 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f0804b3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08000b0170 con 0x7f08000d0320 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f0804b3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08000ca790 con 0x7f08000b54b0 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07fed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08000d0700 0x7f08001592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07fed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08000d0700 0x7f08001592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41804/0 (socket says 172.21.15.59:41804) 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07fed9d640 1 -- 172.21.15.59:0/3956929465 learned_addr learned my addr 172.21.15.59:0/3956929465 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07fe59c640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08000b54b0 0x7f08001464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07fdd9b640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08000d0320 0x7f0800158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07fed9d640 1 -- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08000b54b0 msgr2=0x7f08001464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07fed9d640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08000b54b0 0x7f08001464b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07fed9d640 1 -- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08000d0320 msgr2=0x7f0800158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07fed9d640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08000d0320 0x7f0800158db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07fed9d640 1 -- 172.21.15.59:0/3956929465 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08001570f0 con 0x7f08000d0700 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07fed9d640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08000d0700 0x7f08001592f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f07f800cd50 tx=0x7f07f800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07eeffd640 1 -- 172.21.15.59:0/3956929465 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f800f680 con 0x7f08000d0700 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07eeffd640 1 -- 172.21.15.59:0/3956929465 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f800f840 con 0x7f08000d0700 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f0804b3e640 1 -- 172.21.15.59:0/3956929465 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08001e58d0 con 0x7f08000d0700 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f07eeffd640 1 -- 172.21.15.59:0/3956929465 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f800fc30 con 0x7f08000d0700 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.067+0000 7f0804b3e640 1 -- 172.21.15.59:0/3956929465 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08001e5e90 con 0x7f08000d0700 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.068+0000 7f0804b3e640 1 -- 172.21.15.59:0/3956929465 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08001e56e0 con 0x7f08000d0700 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.069+0000 7f07eeffd640 1 -- 172.21.15.59:0/3956929465 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07f80213f0 con 0x7f08000d0700 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.069+0000 7f07eeffd640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07e006b500 0x7f07e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.069+0000 7f07eeffd640 1 -- 172.21.15.59:0/3956929465 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f07f808ac90 con 0x7f08000d0700 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.069+0000 7f07fe59c640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07e006b500 0x7f07e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.070+0000 7f07fe59c640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07e006b500 0x7f07e006d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07f0036000 tx=0x7f07f003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.074+0000 7f07eeffd640 1 -- 172.21.15.59:0/3956929465 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07f8014030 con 0x7f08000d0700 2024-03-06T07:21:52.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.255+0000 7f0804b3e640 1 -- 172.21.15.59:0/3956929465 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f08001e6060 con 0x7f08000d0700 2024-03-06T07:21:52.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.256+0000 7f07eeffd640 1 -- 172.21.15.59:0/3956929465 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f07f8059810 con 0x7f08000d0700 2024-03-06T07:21:52.258 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:52.258 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-03-06T07:21:52.258 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:21:52.258 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-03-06T07:21:52.258 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-03-06T07:21:52.258 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:21:52.258 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-03-06T07:21:52.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 -- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07e006b500 msgr2=0x7f07e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07e006b500 0x7f07e006d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07f0036000 tx=0x7f07f003c040 comp rx=0 tx=0).stop 2024-03-06T07:21:52.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 -- 172.21.15.59:0/3956929465 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08000d0700 msgr2=0x7f08001592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08000d0700 0x7f08001592f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f07f800cd50 tx=0x7f07f800a650 comp rx=0 tx=0).stop 2024-03-06T07:21:52.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 -- 172.21.15.59:0/3956929465 shutdown_connections 2024-03-06T07:21:52.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07e006b500 0x7f07e006d9c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08000d0700 0x7f08001592f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08000d0320 0x7f0800158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 --2- 172.21.15.59:0/3956929465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f08000b54b0 0x7f08001464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 -- 172.21.15.59:0/3956929465 >> 172.21.15.59:0/3956929465 conn(0x7f08000724d0 msgr2=0x7f08000b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 -- 172.21.15.59:0/3956929465 shutdown_connections 2024-03-06T07:21:52.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.258+0000 7f0804b3e640 1 -- 172.21.15.59:0/3956929465 wait complete. 2024-03-06T07:21:52.270 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:21:52.270 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:21:52.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.373+0000 7f387ba2a640 1 Processor -- start 2024-03-06T07:21:52.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.373+0000 7f387ba2a640 1 -- start start 2024-03-06T07:21:52.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.374+0000 7f387ba2a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f387415dfa0 0x7f38740d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.374+0000 7f387ba2a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38740d4630 0x7f38740d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.374+0000 7f387ba2a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387415e380 con 0x7f38740d2820 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.374+0000 7f387ba2a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38740d2100 con 0x7f38740d4630 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.374+0000 7f387ba2a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38740d1e30 con 0x7f387415dfa0 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.374+0000 7f3878fc5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38740d4630 0x7f38740d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.374+0000 7f3878fc5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38740d4630 0x7f38740d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41808/0 (socket says 172.21.15.59:41808) 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.374+0000 7f3878fc5640 1 -- 172.21.15.59:0/235693857 learned_addr learned my addr 172.21.15.59:0/235693857 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.374+0000 7f3879fc7640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f38740d2820 0x7f38740d05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39490/0 (socket says 172.21.15.59:39490) 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.374+0000 7f38797c6640 1 --2- 172.21.15.59:0/235693857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f387415dfa0 0x7f38740d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.375+0000 7f3878fc5640 1 -- 172.21.15.59:0/235693857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f387415dfa0 msgr2=0x7f38740d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.375+0000 7f3878fc5640 1 --2- 172.21.15.59:0/235693857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f387415dfa0 0x7f38740d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.375+0000 7f3878fc5640 1 -- 172.21.15.59:0/235693857 >> v1:172.21.15.59:6789/0 conn(0x7f38740d2820 legacy=0x7f38740d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.375+0000 7f3878fc5640 1 -- 172.21.15.59:0/235693857 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38740cf470 con 0x7f38740d4630 2024-03-06T07:21:52.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.375+0000 7f38797c6640 1 --2- 172.21.15.59:0/235693857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f387415dfa0 0x7f38740d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:52.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.375+0000 7f3878fc5640 1 --2- 172.21.15.59:0/235693857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38740d4630 0x7f38740d22e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3868009870 tx=0x7f386802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdebe7c71de90e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.375+0000 7f3873fff640 1 -- 172.21.15.59:0/235693857 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3868035020 con 0x7f38740d4630 2024-03-06T07:21:52.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.375+0000 7f3873fff640 1 -- 172.21.15.59:0/235693857 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3868031640 con 0x7f38740d4630 2024-03-06T07:21:52.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.375+0000 7f3873fff640 1 -- 172.21.15.59:0/235693857 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386802d040 con 0x7f38740d4630 2024-03-06T07:21:52.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.376+0000 7f387ba2a640 1 -- 172.21.15.59:0/235693857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38740d4630 msgr2=0x7f38740d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.376+0000 7f387ba2a640 1 --2- 172.21.15.59:0/235693857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38740d4630 0x7f38740d22e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3868009870 tx=0x7f386802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.376+0000 7f387ba2a640 1 -- 172.21.15.59:0/235693857 shutdown_connections 2024-03-06T07:21:52.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.376+0000 7f387ba2a640 1 --2- 172.21.15.59:0/235693857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38740d4630 0x7f38740d22e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.376+0000 7f387ba2a640 1 --2- 172.21.15.59:0/235693857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f387415dfa0 0x7f38740d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.376+0000 7f387ba2a640 1 -- 172.21.15.59:0/235693857 >> 172.21.15.59:0/235693857 conn(0x7f387409d000 msgr2=0x7f38740d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.376+0000 7f387ba2a640 1 -- 172.21.15.59:0/235693857 shutdown_connections 2024-03-06T07:21:52.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.376+0000 7f387ba2a640 1 -- 172.21.15.59:0/235693857 wait complete. 2024-03-06T07:21:52.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.377+0000 7f387ba2a640 1 Processor -- start 2024-03-06T07:21:52.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.377+0000 7f387ba2a640 1 -- start start 2024-03-06T07:21:52.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f387ba2a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38740d2820 0x7f38741f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f387ba2a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740d4630 0x7f387415c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f387ba2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387415dfa0 0x7f38740cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f387ba2a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38740cc1a0 con 0x7f387415dfa0 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f387ba2a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38740cfa30 con 0x7f38740d2820 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f387ba2a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38740cc320 con 0x7f38740d4630 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f3879fc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387415dfa0 0x7f38740cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f3879fc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387415dfa0 0x7f38740cc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41814/0 (socket says 172.21.15.59:41814) 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f3879fc7640 1 -- 172.21.15.59:0/433409165 learned_addr learned my addr 172.21.15.59:0/433409165 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f3878fc5640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740d4630 0x7f387415c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f38797c6640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38740d2820 0x7f38741f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f3879fc7640 1 -- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740d4630 msgr2=0x7f387415c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f3879fc7640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740d4630 0x7f387415c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f3879fc7640 1 -- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38740d2820 msgr2=0x7f38741f4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f3879fc7640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38740d2820 0x7f38741f4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f3879fc7640 1 -- 172.21.15.59:0/433409165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387415c880 con 0x7f387415dfa0 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f38797c6640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38740d2820 0x7f38741f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.378+0000 7f3879fc7640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387415dfa0 0x7f38740cc610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f386400c430 tx=0x7f386400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.379+0000 7f3871ffb640 1 -- 172.21.15.59:0/433409165 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38640074e0 con 0x7f387415dfa0 2024-03-06T07:21:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.379+0000 7f3871ffb640 1 -- 172.21.15.59:0/433409165 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3864007680 con 0x7f387415dfa0 2024-03-06T07:21:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.379+0000 7f387ba2a640 1 -- 172.21.15.59:0/433409165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38741f4ba0 con 0x7f387415dfa0 2024-03-06T07:21:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.379+0000 7f3871ffb640 1 -- 172.21.15.59:0/433409165 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38640049e0 con 0x7f387415dfa0 2024-03-06T07:21:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.379+0000 7f387ba2a640 1 -- 172.21.15.59:0/433409165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38741f5160 con 0x7f387415dfa0 2024-03-06T07:21:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.380+0000 7f387ba2a640 1 -- 172.21.15.59:0/433409165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387404f330 con 0x7f387415dfa0 2024-03-06T07:21:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.380+0000 7f3871ffb640 1 -- 172.21.15.59:0/433409165 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f386401a020 con 0x7f387415dfa0 2024-03-06T07:21:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.381+0000 7f3871ffb640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386006b4d0 0x7f386006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.381+0000 7f3871ffb640 1 -- 172.21.15.59:0/433409165 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3864089560 con 0x7f387415dfa0 2024-03-06T07:21:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.381+0000 7f38797c6640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386006b4d0 0x7f386006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.381+0000 7f38797c6640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386006b4d0 0x7f386006d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f387415cf40 tx=0x7f386c003a50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.385+0000 7f3871ffb640 1 -- 172.21.15.59:0/433409165 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f387415c880 con 0x7f387415dfa0 2024-03-06T07:21:52.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:52.568+0000 7f387ba2a640 1 -- 172.21.15.59:0/433409165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f38740d0510 con 0x7f387415dfa0 2024-03-06T07:21:53.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.394+0000 7f3871ffb640 1 -- 172.21.15.59:0/433409165 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f38640587c0 con 0x7f387415dfa0 2024-03-06T07:21:53.396 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-03-06T07:21:53.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.398+0000 7f387ba2a640 1 -- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386006b4d0 msgr2=0x7f386006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.398+0000 7f387ba2a640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386006b4d0 0x7f386006d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f387415cf40 tx=0x7f386c003a50 comp rx=0 tx=0).stop 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.398+0000 7f387ba2a640 1 -- 172.21.15.59:0/433409165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387415dfa0 msgr2=0x7f38740cc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.398+0000 7f387ba2a640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387415dfa0 0x7f38740cc610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f386400c430 tx=0x7f386400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.399+0000 7f387ba2a640 1 -- 172.21.15.59:0/433409165 shutdown_connections 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.399+0000 7f387ba2a640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386006b4d0 0x7f386006d990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.399+0000 7f387ba2a640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f387415dfa0 0x7f38740cc610 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.399+0000 7f387ba2a640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38740d4630 0x7f387415c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.399+0000 7f387ba2a640 1 --2- 172.21.15.59:0/433409165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38740d2820 0x7f38741f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.399+0000 7f387ba2a640 1 -- 172.21.15.59:0/433409165 >> 172.21.15.59:0/433409165 conn(0x7f387409d000 msgr2=0x7f38740ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.399+0000 7f387ba2a640 1 -- 172.21.15.59:0/433409165 shutdown_connections 2024-03-06T07:21:53.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.399+0000 7f387ba2a640 1 -- 172.21.15.59:0/433409165 wait complete. 2024-03-06T07:21:53.416 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:21:53.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.529+0000 7fc07c713640 1 Processor -- start 2024-03-06T07:21:53.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.529+0000 7fc07c713640 1 -- start start 2024-03-06T07:21:53.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.530+0000 7fc07c713640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0741614d0 0x7fc0740d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.530+0000 7fc07c713640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0741618b0 con 0x7fc0740d1910 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.530+0000 7fc07c713640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0740cf400 con 0x7fc0740d2480 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.530+0000 7fc07c713640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0740cf6a0 con 0x7fc0741614d0 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.530+0000 7fc079cae640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc0740d1910 0x7fc0740d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51064/0 (socket says 172.21.15.59:51064) 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.530+0000 7fc079cae640 1 -- 172.21.15.59:0/2076764666 learned_addr learned my addr 172.21.15.59:0/2076764666 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.530+0000 7fc07a4af640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0741614d0 0x7fc0740d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.530+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726454139 0 0) 0x7fc0741618b0 con 0x7fc0740d1910 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.530+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc05c003520 con 0x7fc0740d1910 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2791763433 0 0) 0x7fc05c003520 con 0x7fc0740d1910 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0741614d0 msgr2=0x7fc0740d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0741614d0 0x7fc0740d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 >> v1:172.21.15.164:6789/0 conn(0x7fc0740d2480 legacy=0x7fc0740b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0740cf980 con 0x7fc0740d1910 2024-03-06T07:21:53.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fc068001670 con 0x7fc0740d1910 2024-03-06T07:21:53.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 >> v1:172.21.15.59:6789/0 conn(0x7fc0740d1910 legacy=0x7fc0740d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc05c0039e0 0x7fc05c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0741614d0 0x7fc05c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 0x7fc05c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0740cf6a0 con 0x7fc05c004d40 2024-03-06T07:21:53.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c00b9c0 con 0x7fc05c0039e0 2024-03-06T07:21:53.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c00bbc0 con 0x7fc0741614d0 2024-03-06T07:21:53.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc07acb0640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 0x7fc05c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc07a4af640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc05c0039e0 0x7fc05c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc079cae640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0741614d0 0x7fc05c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc07acb0640 1 -- 172.21.15.59:0/2076764666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0741614d0 msgr2=0x7fc05c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc07acb0640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0741614d0 0x7fc05c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc07acb0640 1 -- 172.21.15.59:0/2076764666 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc05c0039e0 msgr2=0x7fc05c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc07acb0640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc05c0039e0 0x7fc05c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.531+0000 7fc07acb0640 1 -- 172.21.15.59:0/2076764666 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c003520 con 0x7fc05c004d40 2024-03-06T07:21:53.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.532+0000 7fc07acb0640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 0x7fc05c0051c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc064001f30 tx=0x7fc05c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c4ad30cd7e570 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.532+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc064001b30 con 0x7fc05c004d40 2024-03-06T07:21:53.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.532+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06402d6e0 con 0x7fc05c004d40 2024-03-06T07:21:53.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.532+0000 7fc0794ad640 1 -- 172.21.15.59:0/2076764666 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06402dad0 con 0x7fc05c004d40 2024-03-06T07:21:53.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.532+0000 7fc07c713640 1 -- 172.21.15.59:0/2076764666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 msgr2=0x7fc05c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.532+0000 7fc07c713640 1 --2- 172.21.15.59:0/2076764666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 0x7fc05c0051c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc064001f30 tx=0x7fc05c005830 comp rx=0 tx=0).stop 2024-03-06T07:21:53.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.532+0000 7fc07acb0640 1 -- 172.21.15.59:0/2076764666 reap_dead start 2024-03-06T07:21:53.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.533+0000 7fc07c713640 1 -- 172.21.15.59:0/2076764666 shutdown_connections 2024-03-06T07:21:53.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.533+0000 7fc07c713640 1 -- 172.21.15.59:0/2076764666 >> 172.21.15.59:0/2076764666 conn(0x7fc0740d3d40 msgr2=0x7fc0740d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.533+0000 7fc07c713640 1 -- 172.21.15.59:0/2076764666 shutdown_connections 2024-03-06T07:21:53.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.533+0000 7fc07c713640 1 -- 172.21.15.59:0/2076764666 wait complete. 2024-03-06T07:21:53.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.534+0000 7fc07c713640 1 Processor -- start 2024-03-06T07:21:53.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.534+0000 7fc07c713640 1 -- start start 2024-03-06T07:21:53.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.534+0000 7fc07c713640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 0x7fc07415a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07c713640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc074158440 0x7fc074159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07c713640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0741551f0 0x7fc0740c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07c713640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0740d4860 con 0x7fc05c004d40 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07c713640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0740b0910 con 0x7fc0741551f0 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07c713640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0740d49e0 con 0x7fc074158440 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07a4af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 0x7fc07415a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07a4af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 0x7fc07415a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57822/0 (socket says 172.21.15.59:57822) 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07a4af640 1 -- 172.21.15.59:0/2811860549 learned_addr learned my addr 172.21.15.59:0/2811860549 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07acb0640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0741551f0 0x7fc0740c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc079cae640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc074158440 0x7fc074159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07a4af640 1 -- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc074158440 msgr2=0x7fc074159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07a4af640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc074158440 0x7fc074159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07a4af640 1 -- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0741551f0 msgr2=0x7fc0740c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07a4af640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0741551f0 0x7fc0740c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07a4af640 1 -- 172.21.15.59:0/2811860549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc074159940 con 0x7fc05c004d40 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07acb0640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0741551f0 0x7fc0740c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07a4af640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 0x7fc07415a9e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc06c011a00 tx=0x7fc06c011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc072ffd640 1 -- 172.21.15.59:0/2811860549 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c00fb50 con 0x7fc05c004d40 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc072ffd640 1 -- 172.21.15.59:0/2811860549 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06c00fcf0 con 0x7fc05c004d40 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07c713640 1 -- 172.21.15.59:0/2811860549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc074159c20 con 0x7fc05c004d40 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc072ffd640 1 -- 172.21.15.59:0/2811860549 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c00fea0 con 0x7fc05c004d40 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.535+0000 7fc07c713640 1 -- 172.21.15.59:0/2811860549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07415a4c0 con 0x7fc05c004d40 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.536+0000 7fc07c713640 1 -- 172.21.15.59:0/2811860549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc074159df0 con 0x7fc05c004d40 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.537+0000 7fc072ffd640 1 -- 172.21.15.59:0/2811860549 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc06c012040 con 0x7fc05c004d40 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.537+0000 7fc072ffd640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc06006b4d0 0x7fc06006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.537+0000 7fc072ffd640 1 -- 172.21.15.59:0/2811860549 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc06c08a7a0 con 0x7fc05c004d40 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.537+0000 7fc079cae640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc06006b4d0 0x7fc06006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.538+0000 7fc079cae640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc06006b4d0 0x7fc06006d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc0740b29a0 tx=0x7fc068009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.541+0000 7fc072ffd640 1 -- 172.21.15.59:0/2811860549 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc06c019030 con 0x7fc05c004d40 2024-03-06T07:21:53.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:53.724+0000 7fc07c713640 1 -- 172.21.15.59:0/2811860549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc074068ff0 con 0x7fc05c004d40 2024-03-06T07:21:54.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.396+0000 7fc072ffd640 1 -- 172.21.15.59:0/2811860549 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc06c059800 con 0x7fc05c004d40 2024-03-06T07:21:54.398 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-03-06T07:21:54.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.400+0000 7fc07c713640 1 -- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc06006b4d0 msgr2=0x7fc06006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.400+0000 7fc07c713640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc06006b4d0 0x7fc06006d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc0740b29a0 tx=0x7fc068009040 comp rx=0 tx=0).stop 2024-03-06T07:21:54.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.400+0000 7fc07c713640 1 -- 172.21.15.59:0/2811860549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 msgr2=0x7fc07415a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.400+0000 7fc07c713640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 0x7fc07415a9e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc06c011a00 tx=0x7fc06c011ec0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.400+0000 7fc07c713640 1 -- 172.21.15.59:0/2811860549 shutdown_connections 2024-03-06T07:21:54.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.400+0000 7fc07c713640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc06006b4d0 0x7fc06006d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.400+0000 7fc07c713640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0741551f0 0x7fc0740c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.400+0000 7fc07c713640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc074158440 0x7fc074159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.400+0000 7fc07c713640 1 --2- 172.21.15.59:0/2811860549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc05c004d40 0x7fc07415a9e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.400+0000 7fc07c713640 1 -- 172.21.15.59:0/2811860549 >> 172.21.15.59:0/2811860549 conn(0x7fc0740d3d40 msgr2=0x7fc0740c9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.403+0000 7fc07c713640 1 -- 172.21.15.59:0/2811860549 shutdown_connections 2024-03-06T07:21:54.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.405+0000 7fc07c713640 1 -- 172.21.15.59:0/2811860549 wait complete. 2024-03-06T07:21:54.430 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:21:54.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.537+0000 7f0e59b77640 1 Processor -- start 2024-03-06T07:21:54.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.537+0000 7f0e59b77640 1 -- start start 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e59b77640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0e5415e0b0 0x7f0e540b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e59b77640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0e540b3df0 0x7f0e540d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e59b77640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e540b4290 con 0x7f0e540d28c0 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e59b77640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e540ce990 con 0x7f0e540b3df0 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e59b77640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e540b4570 con 0x7f0e5415e0b0 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e52ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0e540b3df0 0x7f0e540d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e52ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0e540b3df0 0x7f0e540d2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57838/0 (socket says 172.21.15.59:57838) 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e52ffd640 1 -- 172.21.15.59:0/1424398150 learned_addr learned my addr 172.21.15.59:0/1424398150 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e52ffd640 1 -- 172.21.15.59:0/1424398150 >> v2:172.21.15.164:3300/0 conn(0x7f0e5415e0b0 msgr2=0x7f0e540b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e537fe640 1 --2- 172.21.15.59:0/1424398150 >> v2:172.21.15.164:3300/0 conn(0x7f0e5415e0b0 0x7f0e540b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e52ffd640 1 --2- 172.21.15.59:0/1424398150 >> v2:172.21.15.164:3300/0 conn(0x7f0e5415e0b0 0x7f0e540b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e52ffd640 1 -- 172.21.15.59:0/1424398150 >> v1:172.21.15.164:6789/0 conn(0x7f0e540d28c0 legacy=0x7f0e540d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.538+0000 7f0e52ffd640 1 -- 172.21.15.59:0/1424398150 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e540b4850 con 0x7f0e540b3df0 2024-03-06T07:21:54.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.539+0000 7f0e52ffd640 1 --2- 172.21.15.59:0/1424398150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e540b3df0 0x7f0e540d2380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0e5408dbc0 tx=0x7f0e4802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669d942dcc61cf27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.539+0000 7f0e527fc640 1 -- 172.21.15.59:0/1424398150 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48035020 con 0x7f0e540b3df0 2024-03-06T07:21:54.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.539+0000 7f0e527fc640 1 -- 172.21.15.59:0/1424398150 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e48031640 con 0x7f0e540b3df0 2024-03-06T07:21:54.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.539+0000 7f0e527fc640 1 -- 172.21.15.59:0/1424398150 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4802d040 con 0x7f0e540b3df0 2024-03-06T07:21:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.539+0000 7f0e59b77640 1 -- 172.21.15.59:0/1424398150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e540b3df0 msgr2=0x7f0e540d2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.539+0000 7f0e59b77640 1 --2- 172.21.15.59:0/1424398150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e540b3df0 0x7f0e540d2380 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0e5408dbc0 tx=0x7f0e4802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.540+0000 7f0e59b77640 1 -- 172.21.15.59:0/1424398150 shutdown_connections 2024-03-06T07:21:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.540+0000 7f0e59b77640 1 --2- 172.21.15.59:0/1424398150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e540b3df0 0x7f0e540d2380 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.540+0000 7f0e59b77640 1 --2- 172.21.15.59:0/1424398150 >> v2:172.21.15.164:3300/0 conn(0x7f0e5415e0b0 0x7f0e540b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.540+0000 7f0e59b77640 1 -- 172.21.15.59:0/1424398150 >> 172.21.15.59:0/1424398150 conn(0x7f0e5408d6f0 msgr2=0x7f0e5409d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.540+0000 7f0e59b77640 1 -- 172.21.15.59:0/1424398150 shutdown_connections 2024-03-06T07:21:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.540+0000 7f0e59b77640 1 -- 172.21.15.59:0/1424398150 wait complete. 2024-03-06T07:21:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.541+0000 7f0e59b77640 1 Processor -- start 2024-03-06T07:21:54.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.541+0000 7f0e59b77640 1 -- start start 2024-03-06T07:21:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e59b77640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e540b3df0 0x7f0e540b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e59b77640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e540d28c0 0x7f0e540b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e59b77640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e5415e0b0 0x7f0e541e6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e59b77640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e540d1cd0 con 0x7f0e540d28c0 2024-03-06T07:21:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e59b77640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5415e490 con 0x7f0e5415e0b0 2024-03-06T07:21:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e59b77640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e540d1e50 con 0x7f0e540b3df0 2024-03-06T07:21:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e52ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e540d28c0 0x7f0e540b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e52ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e540d28c0 0x7f0e540b17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57842/0 (socket says 172.21.15.59:57842) 2024-03-06T07:21:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e52ffd640 1 -- 172.21.15.59:0/3716605788 learned_addr learned my addr 172.21.15.59:0/3716605788 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e537fe640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e540b3df0 0x7f0e540b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e53fff640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e5415e0b0 0x7f0e541e6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e52ffd640 1 -- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e540b3df0 msgr2=0x7f0e540b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e52ffd640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e540b3df0 0x7f0e540b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e52ffd640 1 -- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e5415e0b0 msgr2=0x7f0e541e6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e52ffd640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e5415e0b0 0x7f0e541e6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e52ffd640 1 -- 172.21.15.59:0/3716605788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e540c8cf0 con 0x7f0e540d28c0 2024-03-06T07:21:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e537fe640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e540b3df0 0x7f0e540b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e52ffd640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e540d28c0 0x7f0e540b17e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0e48000c00 tx=0x7f0e480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e3bfff640 1 -- 172.21.15.59:0/3716605788 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48040070 con 0x7f0e540d28c0 2024-03-06T07:21:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e3bfff640 1 -- 172.21.15.59:0/3716605788 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e48039c10 con 0x7f0e540d28c0 2024-03-06T07:21:54.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e3bfff640 1 -- 172.21.15.59:0/3716605788 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48035040 con 0x7f0e540d28c0 2024-03-06T07:21:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e59b77640 1 -- 172.21.15.59:0/3716605788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e540c8fd0 con 0x7f0e540d28c0 2024-03-06T07:21:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.542+0000 7f0e59b77640 1 -- 172.21.15.59:0/3716605788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e540c9590 con 0x7f0e540d28c0 2024-03-06T07:21:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.543+0000 7f0e59b77640 1 -- 172.21.15.59:0/3716605788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5404f330 con 0x7f0e540d28c0 2024-03-06T07:21:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.544+0000 7f0e3bfff640 1 -- 172.21.15.59:0/3716605788 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e4804b060 con 0x7f0e540d28c0 2024-03-06T07:21:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.544+0000 7f0e3bfff640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e4006b4d0 0x7f0e4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.544+0000 7f0e3bfff640 1 -- 172.21.15.59:0/3716605788 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f0e480abda0 con 0x7f0e540d28c0 2024-03-06T07:21:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.544+0000 7f0e537fe640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e4006b4d0 0x7f0e4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.545+0000 7f0e537fe640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e4006b4d0 0x7f0e4006d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e44006fd0 tx=0x7f0e44008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.549+0000 7f0e3bfff640 1 -- 172.21.15.59:0/3716605788 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e4802f300 con 0x7f0e540d28c0 2024-03-06T07:21:54.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:54.765+0000 7f0e59b77640 1 -- 172.21.15.59:0/3716605788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0e540d2160 con 0x7f0e540d28c0 2024-03-06T07:21:55.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.418+0000 7f0e3bfff640 1 -- 172.21.15.59:0/3716605788 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0e4807abe0 con 0x7f0e540d28c0 2024-03-06T07:21:55.421 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:21:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.423+0000 7f0e59b77640 1 -- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e4006b4d0 msgr2=0x7f0e4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.423+0000 7f0e59b77640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e4006b4d0 0x7f0e4006d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e44006fd0 tx=0x7f0e44008040 comp rx=0 tx=0).stop 2024-03-06T07:21:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.423+0000 7f0e59b77640 1 -- 172.21.15.59:0/3716605788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e540d28c0 msgr2=0x7f0e540b17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.423+0000 7f0e59b77640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e540d28c0 0x7f0e540b17e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0e48000c00 tx=0x7f0e480046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.423+0000 7f0e59b77640 1 -- 172.21.15.59:0/3716605788 shutdown_connections 2024-03-06T07:21:55.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.423+0000 7f0e59b77640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e4006b4d0 0x7f0e4006d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.423+0000 7f0e59b77640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e5415e0b0 0x7f0e541e6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.423+0000 7f0e59b77640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e540d28c0 0x7f0e540b17e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.423+0000 7f0e59b77640 1 --2- 172.21.15.59:0/3716605788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e540b3df0 0x7f0e540b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.423+0000 7f0e59b77640 1 -- 172.21.15.59:0/3716605788 >> 172.21.15.59:0/3716605788 conn(0x7f0e5408d6f0 msgr2=0x7f0e5406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.424+0000 7f0e59b77640 1 -- 172.21.15.59:0/3716605788 shutdown_connections 2024-03-06T07:21:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.424+0000 7f0e59b77640 1 -- 172.21.15.59:0/3716605788 wait complete. 2024-03-06T07:21:55.443 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:55.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.545+0000 7fa466da0640 1 Processor -- start 2024-03-06T07:21:55.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.546+0000 7fa466da0640 1 -- start start 2024-03-06T07:21:55.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.546+0000 7fa466da0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600d20c0 0x7fa4600d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.546+0000 7fa466da0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4600d0fc0 con 0x7fa4600b5030 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.546+0000 7fa466da0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4600d29e0 con 0x7fa4600cf190 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.546+0000 7fa466da0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4600b2fa0 con 0x7fa4600d20c0 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.546+0000 7fa46659f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa4600b5030 0x7fa4600d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51076/0 (socket says 172.21.15.59:51076) 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.546+0000 7fa46659f640 1 -- 172.21.15.59:0/3902461848 learned_addr learned my addr 172.21.15.59:0/3902461848 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa465d9e640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600d20c0 0x7fa4600d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676366183 0 0) 0x7fa4600d0fc0 con 0x7fa4600b5030 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa448003520 con 0x7fa4600b5030 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154463709 0 0) 0x7fa448003520 con 0x7fa4600b5030 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600d20c0 msgr2=0x7fa4600d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600d20c0 0x7fa4600d24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 >> v1:172.21.15.164:6789/0 conn(0x7fa4600cf190 legacy=0x7fa4600d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4600b3260 con 0x7fa4600b5030 2024-03-06T07:21:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fa458001670 con 0x7fa4600b5030 2024-03-06T07:21:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 >> v1:172.21.15.59:6789/0 conn(0x7fa4600b5030 legacy=0x7fa4600d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa448003a50 0x7fa448003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600d20c0 0x7fa448004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa448004d40 0x7fa4480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4600b2fa0 con 0x7fa448003a50 2024-03-06T07:21:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44800bb60 con 0x7fa448004d40 2024-03-06T07:21:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44800bd60 con 0x7fa4600d20c0 2024-03-06T07:21:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa465d9e640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa448003a50 0x7fa448003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa46559d640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600d20c0 0x7fa448004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa46659f640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa448004d40 0x7fa4480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa465d9e640 1 -- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600d20c0 msgr2=0x7fa448004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa465d9e640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600d20c0 0x7fa448004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa465d9e640 1 -- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa448004d40 msgr2=0x7fa4480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa465d9e640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa448004d40 0x7fa4480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.547+0000 7fa465d9e640 1 -- 172.21.15.59:0/3902461848 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa448003520 con 0x7fa448003a50 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.548+0000 7fa46659f640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa448004d40 0x7fa4480051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.548+0000 7fa465d9e640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa448003a50 0x7fa448003e70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa450009f90 tx=0x7fa448005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23143cb4feead29a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.548+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450039070 con 0x7fa448003a50 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.548+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45002cb30 con 0x7fa448003a50 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.548+0000 7fa464d9c640 1 -- 172.21.15.59:0/3902461848 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa450009c10 con 0x7fa448003a50 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.549+0000 7fa466da0640 1 -- 172.21.15.59:0/3902461848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa448003a50 msgr2=0x7fa448003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.549+0000 7fa466da0640 1 --2- 172.21.15.59:0/3902461848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa448003a50 0x7fa448003e70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa450009f90 tx=0x7fa448005830 comp rx=0 tx=0).stop 2024-03-06T07:21:55.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.549+0000 7fa46659f640 1 -- 172.21.15.59:0/3902461848 reap_dead start 2024-03-06T07:21:55.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.549+0000 7fa466da0640 1 -- 172.21.15.59:0/3902461848 shutdown_connections 2024-03-06T07:21:55.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.549+0000 7fa466da0640 1 -- 172.21.15.59:0/3902461848 >> 172.21.15.59:0/3902461848 conn(0x7fa46009ce20 msgr2=0x7fa4600d3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.549+0000 7fa466da0640 1 -- 172.21.15.59:0/3902461848 shutdown_connections 2024-03-06T07:21:55.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.549+0000 7fa466da0640 1 -- 172.21.15.59:0/3902461848 wait complete. 2024-03-06T07:21:55.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.550+0000 7fa466da0640 1 Processor -- start 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.550+0000 7fa466da0640 1 -- start start 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa466da0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa46014ff40 0x7fa4600b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa466da0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600b6700 0x7fa4600cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa466da0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4600cc240 0x7fa4600cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa466da0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4600b3b00 con 0x7fa4600cc240 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa466da0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4600b3820 con 0x7fa46014ff40 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa466da0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4600d1af0 con 0x7fa4600b6700 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa46659f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4600cc240 0x7fa4600cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa46659f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4600cc240 0x7fa4600cc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57852/0 (socket says 172.21.15.59:57852) 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa46659f640 1 -- 172.21.15.59:0/2076664390 learned_addr learned my addr 172.21.15.59:0/2076664390 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa46559d640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600b6700 0x7fa4600cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa465d9e640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa46014ff40 0x7fa4600b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa46659f640 1 -- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600b6700 msgr2=0x7fa4600cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa46659f640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600b6700 0x7fa4600cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa46659f640 1 -- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa46014ff40 msgr2=0x7fa4600b7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa46659f640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa46014ff40 0x7fa4600b7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa46659f640 1 -- 172.21.15.59:0/2076664390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa460150320 con 0x7fa4600cc240 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa46659f640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4600cc240 0x7fa4600cc6f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa4600a1a60 tx=0x7fa45800eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa45e7fc640 1 -- 172.21.15.59:0/2076664390 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa458019070 con 0x7fa4600cc240 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa45e7fc640 1 -- 172.21.15.59:0/2076664390 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45800c590 con 0x7fa4600cc240 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.551+0000 7fa466da0640 1 -- 172.21.15.59:0/2076664390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4600b6be0 con 0x7fa4600cc240 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.552+0000 7fa45e7fc640 1 -- 172.21.15.59:0/2076664390 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45800c960 con 0x7fa4600cc240 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.552+0000 7fa466da0640 1 -- 172.21.15.59:0/2076664390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4600b71a0 con 0x7fa4600cc240 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.552+0000 7fa466da0640 1 -- 172.21.15.59:0/2076664390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa460068f80 con 0x7fa4600cc240 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.553+0000 7fa45e7fc640 1 -- 172.21.15.59:0/2076664390 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa45801c070 con 0x7fa4600cc240 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.553+0000 7fa45e7fc640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa44c06b500 0x7fa44c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.553+0000 7fa45e7fc640 1 -- 172.21.15.59:0/2076664390 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa458089910 con 0x7fa4600cc240 2024-03-06T07:21:55.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.553+0000 7fa465d9e640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa44c06b500 0x7fa44c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.554+0000 7fa465d9e640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa44c06b500 0x7fa44c06d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa450036000 tx=0x7fa45002e000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.557+0000 7fa45e7fc640 1 -- 172.21.15.59:0/2076664390 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa458058660 con 0x7fa4600cc240 2024-03-06T07:21:55.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.739+0000 7fa466da0640 1 -- 172.21.15.59:0/2076664390 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4600d1ea0 con 0x7fa4600cc240 2024-03-06T07:21:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.740+0000 7fa45e7fc640 1 -- 172.21.15.59:0/2076664390 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16606 (secure 0 0 0) 0x7fa458057f60 con 0x7fa4600cc240 2024-03-06T07:21:55.742 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:55.742 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:54.766888+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 -- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa44c06b500 msgr2=0x7fa44c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa44c06b500 0x7fa44c06d9c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa450036000 tx=0x7fa45002e000 comp rx=0 tx=0).stop 2024-03-06T07:21:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 -- 172.21.15.59:0/2076664390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4600cc240 msgr2=0x7fa4600cc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4600cc240 0x7fa4600cc6f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa4600a1a60 tx=0x7fa45800eb70 comp rx=0 tx=0).stop 2024-03-06T07:21:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 -- 172.21.15.59:0/2076664390 shutdown_connections 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa44c06b500 0x7fa44c06d9c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4600cc240 0x7fa4600cc6f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4600b6700 0x7fa4600cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 --2- 172.21.15.59:0/2076664390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa46014ff40 0x7fa4600b7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 -- 172.21.15.59:0/2076664390 >> 172.21.15.59:0/2076664390 conn(0x7fa46009ce20 msgr2=0x7fa4600d09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 -- 172.21.15.59:0/2076664390 shutdown_connections 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.742+0000 7fa466da0640 1 -- 172.21.15.59:0/2076664390 wait complete. 2024-03-06T07:21:55.755 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:55.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.854+0000 7effa91c9640 1 Processor -- start 2024-03-06T07:21:55.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.854+0000 7effa91c9640 1 -- start start 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa91c9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7effa40d2920 0x7effa40d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa91c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa40d0650 0x7effa40d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa91c9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa415e470 con 0x7effa415e090 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa91c9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa40d18f0 con 0x7effa40d2920 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa91c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa40b3cf0 con 0x7effa40d0650 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa259c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7effa40d2920 0x7effa40d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa2d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7effa415e090 0x7effa40b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51092/0 (socket says 172.21.15.59:51092) 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa259c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7effa40d2920 0x7effa40d47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57868/0 (socket says 172.21.15.59:57868) 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa2d9d640 1 -- 172.21.15.59:0/3041447357 learned_addr learned my addr 172.21.15.59:0/3041447357 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa359e640 1 --2- 172.21.15.59:0/3041447357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa40d0650 0x7effa40d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa259c640 1 -- 172.21.15.59:0/3041447357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa40d0650 msgr2=0x7effa40d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa259c640 1 --2- 172.21.15.59:0/3041447357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa40d0650 0x7effa40d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa259c640 1 -- 172.21.15.59:0/3041447357 >> v1:172.21.15.59:6789/0 conn(0x7effa415e090 legacy=0x7effa40b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa259c640 1 -- 172.21.15.59:0/3041447357 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa40d1b90 con 0x7effa40d2920 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa1d9b640 1 -- 172.21.15.59:0/3041447357 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040230624 0 0) 0x7effa415e470 con 0x7effa415e090 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.855+0000 7effa359e640 1 --2- 172.21.15.59:0/3041447357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa40d0650 0x7effa40d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.856+0000 7effa259c640 1 --2- 172.21.15.59:0/3041447357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa40d2920 0x7effa40d47a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7eff980097a0 tx=0x7eff9802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c60c0af16bdeb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.856+0000 7effa1d9b640 1 -- 172.21.15.59:0/3041447357 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98035020 con 0x7effa40d2920 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.856+0000 7effa1d9b640 1 -- 172.21.15.59:0/3041447357 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff98031830 con 0x7effa40d2920 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.856+0000 7effa1d9b640 1 -- 172.21.15.59:0/3041447357 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98031c00 con 0x7effa40d2920 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.857+0000 7effa91c9640 1 -- 172.21.15.59:0/3041447357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa40d2920 msgr2=0x7effa40d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.857+0000 7effa91c9640 1 --2- 172.21.15.59:0/3041447357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa40d2920 0x7effa40d47a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7eff980097a0 tx=0x7eff9802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.857+0000 7effa91c9640 1 -- 172.21.15.59:0/3041447357 shutdown_connections 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.857+0000 7effa91c9640 1 --2- 172.21.15.59:0/3041447357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa40d0650 0x7effa40d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.857+0000 7effa91c9640 1 --2- 172.21.15.59:0/3041447357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa40d2920 0x7effa40d47a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.857+0000 7effa91c9640 1 -- 172.21.15.59:0/3041447357 >> 172.21.15.59:0/3041447357 conn(0x7effa40d3c80 msgr2=0x7effa40d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.857+0000 7effa91c9640 1 -- 172.21.15.59:0/3041447357 shutdown_connections 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.857+0000 7effa91c9640 1 -- 172.21.15.59:0/3041447357 wait complete. 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.858+0000 7effa91c9640 1 Processor -- start 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.858+0000 7effa91c9640 1 -- start start 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa91c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa40d0650 0x7effa40ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa91c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa40d2920 0x7effa40cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa91c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa415e090 0x7effa40cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa91c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa40b2df0 con 0x7effa415e090 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa91c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa40cee30 con 0x7effa40d0650 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa91c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa40ce800 con 0x7effa40d2920 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa415e090 0x7effa40cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa415e090 0x7effa40cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57884/0 (socket says 172.21.15.59:57884) 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa359e640 1 -- 172.21.15.59:0/124453952 learned_addr learned my addr 172.21.15.59:0/124453952 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa259c640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa40d2920 0x7effa40cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa2d9d640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa40d0650 0x7effa40ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa359e640 1 -- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa40d2920 msgr2=0x7effa40cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa359e640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa40d2920 0x7effa40cf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa359e640 1 -- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa40d0650 msgr2=0x7effa40ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa359e640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa40d0650 0x7effa40ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa359e640 1 -- 172.21.15.59:0/124453952 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa41f3e40 con 0x7effa415e090 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa2d9d640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa40d0650 0x7effa40ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.859+0000 7effa359e640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa415e090 0x7effa40cfbf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7eff9400cd50 tx=0x7eff9400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.860+0000 7eff8b7fe640 1 -- 172.21.15.59:0/124453952 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9400f680 con 0x7effa415e090 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.860+0000 7eff8b7fe640 1 -- 172.21.15.59:0/124453952 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9400f840 con 0x7effa415e090 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.860+0000 7effa91c9640 1 -- 172.21.15.59:0/124453952 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa41f4120 con 0x7effa415e090 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.860+0000 7eff8b7fe640 1 -- 172.21.15.59:0/124453952 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9400fc30 con 0x7effa415e090 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.860+0000 7effa91c9640 1 -- 172.21.15.59:0/124453952 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa41f46e0 con 0x7effa415e090 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.861+0000 7effa91c9640 1 -- 172.21.15.59:0/124453952 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa404f330 con 0x7effa415e090 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.861+0000 7eff8b7fe640 1 -- 172.21.15.59:0/124453952 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff940213f0 con 0x7effa415e090 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.861+0000 7eff8b7fe640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7eff9006b4d0 0x7eff9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.861+0000 7eff8b7fe640 1 -- 172.21.15.59:0/124453952 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff9408aed0 con 0x7effa415e090 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.862+0000 7effa2d9d640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7eff9006b4d0 0x7eff9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.862+0000 7effa2d9d640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7eff9006b4d0 0x7eff9006d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7eff9c000f60 tx=0x7eff9c007cc0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:55.866+0000 7eff8b7fe640 1 -- 172.21.15.59:0/124453952 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff94014030 con 0x7effa415e090 2024-03-06T07:21:56.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.047+0000 7effa91c9640 1 -- 172.21.15.59:0/124453952 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effa40cd1e0 con 0x7effa415e090 2024-03-06T07:21:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.048+0000 7eff8b7fe640 1 -- 172.21.15.59:0/124453952 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16606 (secure 0 0 0) 0x7eff94059de0 con 0x7effa415e090 2024-03-06T07:21:56.050 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:56.050 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:54.766888+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 -- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7eff9006b4d0 msgr2=0x7eff9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7eff9006b4d0 0x7eff9006d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7eff9c000f60 tx=0x7eff9c007cc0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 -- 172.21.15.59:0/124453952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa415e090 msgr2=0x7effa40cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa415e090 0x7effa40cfbf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7eff9400cd50 tx=0x7eff9400a650 comp rx=0 tx=0).stop 2024-03-06T07:21:56.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 -- 172.21.15.59:0/124453952 shutdown_connections 2024-03-06T07:21:56.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7eff9006b4d0 0x7eff9006d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa415e090 0x7effa40cfbf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effa40d2920 0x7effa40cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 --2- 172.21.15.59:0/124453952 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effa40d0650 0x7effa40ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 -- 172.21.15.59:0/124453952 >> 172.21.15.59:0/124453952 conn(0x7effa40d3c80 msgr2=0x7effa406b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 -- 172.21.15.59:0/124453952 shutdown_connections 2024-03-06T07:21:56.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.050+0000 7effa91c9640 1 -- 172.21.15.59:0/124453952 wait complete. 2024-03-06T07:21:56.063 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.165+0000 7fa0d7f2b640 1 Processor -- start 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.166+0000 7fa0d7f2b640 1 -- start start 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.166+0000 7fa0d7f2b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa0d00d10c0 0x7fa0d00d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.166+0000 7fa0d7f2b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0d015ecd0 0x7fa0d00cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.166+0000 7fa0d7f2b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d00cfa50 con 0x7fa0d00d14a0 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.166+0000 7fa0d7f2b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d015c050 con 0x7fa0d00d10c0 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.166+0000 7fa0d7f2b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d00d0060 con 0x7fa0d015ecd0 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.167+0000 7fa0d5cc7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa0d00d10c0 0x7fa0d00d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.167+0000 7fa0d64c8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0d015ecd0 0x7fa0d00cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.167+0000 7fa0d54c6640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa0d00d14a0 0x7fa0d015e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:47986/0 (socket says 172.21.15.59:47986) 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.167+0000 7fa0d54c6640 1 -- 172.21.15.59:0/3868858636 learned_addr learned my addr 172.21.15.59:0/3868858636 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.167+0000 7fa0d5cc7640 1 -- 172.21.15.59:0/3868858636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0d015ecd0 msgr2=0x7fa0d00cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.167+0000 7fa0d5cc7640 1 --2- 172.21.15.59:0/3868858636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0d015ecd0 0x7fa0d00cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.167+0000 7fa0d5cc7640 1 -- 172.21.15.59:0/3868858636 >> v1:172.21.15.164:6789/0 conn(0x7fa0d00d14a0 legacy=0x7fa0d015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.167+0000 7fa0d5cc7640 1 -- 172.21.15.59:0/3868858636 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d00d0a30 con 0x7fa0d00d10c0 2024-03-06T07:21:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.167+0000 7fa0d4cc5640 1 -- 172.21.15.59:0/3868858636 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990522155 0 0) 0x7fa0d00cfa50 con 0x7fa0d00d14a0 2024-03-06T07:21:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.167+0000 7fa0d64c8640 1 --2- 172.21.15.59:0/3868858636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0d015ecd0 0x7fa0d00cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.168+0000 7fa0d5cc7640 1 --2- 172.21.15.59:0/3868858636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0d00d10c0 0x7fa0d00d04f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc0097a0 tx=0x7fa0cc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e098bc9114bebce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.168+0000 7fa0d4cc5640 1 -- 172.21.15.59:0/3868858636 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc035020 con 0x7fa0d00d10c0 2024-03-06T07:21:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.168+0000 7fa0d4cc5640 1 -- 172.21.15.59:0/3868858636 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc031830 con 0x7fa0d00d10c0 2024-03-06T07:21:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.168+0000 7fa0d4cc5640 1 -- 172.21.15.59:0/3868858636 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc031c00 con 0x7fa0d00d10c0 2024-03-06T07:21:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.168+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/3868858636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0d00d10c0 msgr2=0x7fa0d00d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.168+0000 7fa0d7f2b640 1 --2- 172.21.15.59:0/3868858636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0d00d10c0 0x7fa0d00d04f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc0097a0 tx=0x7fa0cc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.168+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/3868858636 shutdown_connections 2024-03-06T07:21:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.168+0000 7fa0d7f2b640 1 --2- 172.21.15.59:0/3868858636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0d015ecd0 0x7fa0d00cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.168+0000 7fa0d7f2b640 1 --2- 172.21.15.59:0/3868858636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0d00d10c0 0x7fa0d00d04f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.168+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/3868858636 >> 172.21.15.59:0/3868858636 conn(0x7fa0d00b3800 msgr2=0x7fa0d00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.169+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/3868858636 shutdown_connections 2024-03-06T07:21:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.169+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/3868858636 wait complete. 2024-03-06T07:21:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.170+0000 7fa0d7f2b640 1 Processor -- start 2024-03-06T07:21:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.170+0000 7fa0d7f2b640 1 -- start start 2024-03-06T07:21:56.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.170+0000 7fa0d7f2b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0d00d10c0 0x7fa0d00b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.170+0000 7fa0d7f2b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d00d14a0 0x7fa0d0152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.170+0000 7fa0d7f2b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0d015b3c0 0x7fa0d01f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.170+0000 7fa0d7f2b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d015f440 con 0x7fa0d00d14a0 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.170+0000 7fa0d7f2b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d015f160 con 0x7fa0d00d10c0 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.170+0000 7fa0d7f2b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d00cfcf0 con 0x7fa0d015b3c0 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.170+0000 7fa0d54c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d00d14a0 0x7fa0d0152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.170+0000 7fa0d54c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d00d14a0 0x7fa0d0152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57894/0 (socket says 172.21.15.59:57894) 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0d54c6640 1 -- 172.21.15.59:0/4016134226 learned_addr learned my addr 172.21.15.59:0/4016134226 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0d64c8640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0d015b3c0 0x7fa0d01f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0d5cc7640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0d00d10c0 0x7fa0d00b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0d54c6640 1 -- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0d015b3c0 msgr2=0x7fa0d01f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0d54c6640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0d015b3c0 0x7fa0d01f3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0d54c6640 1 -- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0d00d10c0 msgr2=0x7fa0d00b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0d54c6640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0d00d10c0 0x7fa0d00b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0d54c6640 1 -- 172.21.15.59:0/4016134226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d00b4f80 con 0x7fa0d00d14a0 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0d54c6640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d00d14a0 0x7fa0d0152770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000c430 tx=0x7fa0c000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0c67fc640 1 -- 172.21.15.59:0/4016134226 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00074e0 con 0x7fa0d00d14a0 2024-03-06T07:21:56.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0c67fc640 1 -- 172.21.15.59:0/4016134226 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c00047e0 con 0x7fa0d00d14a0 2024-03-06T07:21:56.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/4016134226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d00b5260 con 0x7fa0d00d14a0 2024-03-06T07:21:56.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0c67fc640 1 -- 172.21.15.59:0/4016134226 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0004bd0 con 0x7fa0d00d14a0 2024-03-06T07:21:56.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.171+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/4016134226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d00b5820 con 0x7fa0d00d14a0 2024-03-06T07:21:56.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.172+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/4016134226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d009fe60 con 0x7fa0d00d14a0 2024-03-06T07:21:56.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.173+0000 7fa0c67fc640 1 -- 172.21.15.59:0/4016134226 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0c0011070 con 0x7fa0d00d14a0 2024-03-06T07:21:56.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.173+0000 7fa0c67fc640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0b406b4d0 0x7fa0b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.173+0000 7fa0c67fc640 1 -- 172.21.15.59:0/4016134226 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0c0089c60 con 0x7fa0d00d14a0 2024-03-06T07:21:56.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.173+0000 7fa0d5cc7640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0b406b4d0 0x7fa0b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.174+0000 7fa0d5cc7640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0b406b4d0 0x7fa0b406d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc036000 tx=0x7fa0cc008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.177+0000 7fa0c67fc640 1 -- 172.21.15.59:0/4016134226 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0c0058b70 con 0x7fa0d00d14a0 2024-03-06T07:21:56.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.401+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/4016134226 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa0d00d0c30 con 0x7fa0d00d14a0 2024-03-06T07:21:56.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.424+0000 7fa0c67fc640 1 -- 172.21.15.59:0/4016134226 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa0c0058470 con 0x7fa0d00d14a0 2024-03-06T07:21:56.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0b406b4d0 msgr2=0x7fa0b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0b406b4d0 0x7fa0b406d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc036000 tx=0x7fa0cc008030 comp rx=0 tx=0).stop 2024-03-06T07:21:56.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/4016134226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d00d14a0 msgr2=0x7fa0d0152770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d00d14a0 0x7fa0d0152770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000c430 tx=0x7fa0c000c830 comp rx=0 tx=0).stop 2024-03-06T07:21:56.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/4016134226 shutdown_connections 2024-03-06T07:21:56.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0b406b4d0 0x7fa0b406d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0d015b3c0 0x7fa0d01f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0d00d14a0 0x7fa0d0152770 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 --2- 172.21.15.59:0/4016134226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0d00d10c0 0x7fa0d00b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/4016134226 >> 172.21.15.59:0/4016134226 conn(0x7fa0d00b3800 msgr2=0x7fa0d00b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/4016134226 shutdown_connections 2024-03-06T07:21:56.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.426+0000 7fa0d7f2b640 1 -- 172.21.15.59:0/4016134226 wait complete. 2024-03-06T07:21:56.439 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:21:56.439 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:21:56.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.541+0000 7f60893d7640 1 Processor -- start 2024-03-06T07:21:56.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.541+0000 7f60893d7640 1 -- start start 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60893d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60840d4860 0x7f60840d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60893d7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60840b0e30 0x7f60840b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60893d7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60841618b0 con 0x7f60841614d0 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60893d7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60840d4cc0 con 0x7f60840b0e30 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60893d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60840cf400 con 0x7f60840d4860 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60837fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60840b0e30 0x7f60840b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60837fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60840b0e30 0x7f60840b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57900/0 (socket says 172.21.15.59:57900) 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60837fe640 1 -- 172.21.15.59:0/3070356349 learned_addr learned my addr 172.21.15.59:0/3070356349 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60827fc640 1 --2- 172.21.15.59:0/3070356349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60840d4860 0x7f60840d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60837fe640 1 -- 172.21.15.59:0/3070356349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60840d4860 msgr2=0x7f60840d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60837fe640 1 --2- 172.21.15.59:0/3070356349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60840d4860 0x7f60840d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60837fe640 1 -- 172.21.15.59:0/3070356349 >> v1:172.21.15.164:6789/0 conn(0x7f60841614d0 legacy=0x7f60840d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.542+0000 7f60837fe640 1 -- 172.21.15.59:0/3070356349 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60840cf6a0 con 0x7f60840b0e30 2024-03-06T07:21:56.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.543+0000 7f60837fe640 1 --2- 172.21.15.59:0/3070356349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60840b0e30 0x7f60840b12c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6074009e50 tx=0x7f607402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f43075e5662e6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.543+0000 7f6081ffb640 1 -- 172.21.15.59:0/3070356349 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607402cb00 con 0x7f60840b0e30 2024-03-06T07:21:56.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.543+0000 7f6081ffb640 1 -- 172.21.15.59:0/3070356349 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607402cca0 con 0x7f60840b0e30 2024-03-06T07:21:56.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.543+0000 7f6081ffb640 1 -- 172.21.15.59:0/3070356349 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074034610 con 0x7f60840b0e30 2024-03-06T07:21:56.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.544+0000 7f60893d7640 1 -- 172.21.15.59:0/3070356349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60840b0e30 msgr2=0x7f60840b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.544+0000 7f60893d7640 1 --2- 172.21.15.59:0/3070356349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60840b0e30 0x7f60840b12c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6074009e50 tx=0x7f607402bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:56.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.544+0000 7f60893d7640 1 -- 172.21.15.59:0/3070356349 shutdown_connections 2024-03-06T07:21:56.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.544+0000 7f60893d7640 1 --2- 172.21.15.59:0/3070356349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60840b0e30 0x7f60840b12c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.544+0000 7f60893d7640 1 --2- 172.21.15.59:0/3070356349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60840d4860 0x7f60840d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.544+0000 7f60893d7640 1 -- 172.21.15.59:0/3070356349 >> 172.21.15.59:0/3070356349 conn(0x7f60840d3d40 msgr2=0x7f60840d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.544+0000 7f60893d7640 1 -- 172.21.15.59:0/3070356349 shutdown_connections 2024-03-06T07:21:56.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.544+0000 7f60893d7640 1 -- 172.21.15.59:0/3070356349 wait complete. 2024-03-06T07:21:56.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.545+0000 7f60893d7640 1 Processor -- start 2024-03-06T07:21:56.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.545+0000 7f60893d7640 1 -- start start 2024-03-06T07:21:56.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60893d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60840b0e30 0x7f608415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60893d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60840d4860 0x7f60840c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60893d7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60841614d0 0x7f60840cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60893d7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60840b0910 con 0x7f60840d4860 2024-03-06T07:21:56.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60893d7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60840cfc60 con 0x7f60841614d0 2024-03-06T07:21:56.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60893d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60840d2560 con 0x7f60840b0e30 2024-03-06T07:21:56.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60827fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60840d4860 0x7f60840c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60827fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60840d4860 0x7f60840c89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57912/0 (socket says 172.21.15.59:57912) 2024-03-06T07:21:56.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60827fc640 1 -- 172.21.15.59:0/3576647336 learned_addr learned my addr 172.21.15.59:0/3576647336 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f6082ffd640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60840b0e30 0x7f608415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60837fe640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60841614d0 0x7f60840cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60827fc640 1 -- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60840b0e30 msgr2=0x7f608415ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60827fc640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60840b0e30 0x7f608415ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60827fc640 1 -- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60841614d0 msgr2=0x7f60840cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60827fc640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60841614d0 0x7f60840cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60827fc640 1 -- 172.21.15.59:0/3576647336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608415f380 con 0x7f60840d4860 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60827fc640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60840d4860 0x7f60840c89b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f606c00e9b0 tx=0x7f606c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f607b7fe640 1 -- 172.21.15.59:0/3576647336 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c017070 con 0x7f60840d4860 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f607b7fe640 1 -- 172.21.15.59:0/3576647336 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606c00c9a0 con 0x7f60840d4860 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60893d7640 1 -- 172.21.15.59:0/3576647336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60840c7910 con 0x7f60840d4860 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f607b7fe640 1 -- 172.21.15.59:0/3576647336 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c00cd90 con 0x7f60840d4860 2024-03-06T07:21:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.546+0000 7f60893d7640 1 -- 172.21.15.59:0/3576647336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60840c7ed0 con 0x7f60840d4860 2024-03-06T07:21:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.548+0000 7f60893d7640 1 -- 172.21.15.59:0/3576647336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60840b4d50 con 0x7f60840d4860 2024-03-06T07:21:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.548+0000 7f607b7fe640 1 -- 172.21.15.59:0/3576647336 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f606c01a070 con 0x7f60840d4860 2024-03-06T07:21:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.548+0000 7f607b7fe640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606406b500 0x7f606406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.548+0000 7f607b7fe640 1 -- 172.21.15.59:0/3576647336 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f606c08a140 con 0x7f60840d4860 2024-03-06T07:21:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.548+0000 7f6082ffd640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606406b500 0x7f606406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.549+0000 7f6082ffd640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606406b500 0x7f606406d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f607c0016e0 tx=0x7f607c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.552+0000 7f607b7fe640 1 -- 172.21.15.59:0/3576647336 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f606c058970 con 0x7f60840d4860 2024-03-06T07:21:56.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:56.775+0000 7f60893d7640 1 -- 172.21.15.59:0/3576647336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f60840d0040 con 0x7f60840d4860 2024-03-06T07:21:57.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.429+0000 7f607b7fe640 1 -- 172.21.15.59:0/3576647336 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f606c05ca10 con 0x7f60840d4860 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 -- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606406b500 msgr2=0x7f606406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606406b500 0x7f606406d9c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f607c0016e0 tx=0x7f607c009040 comp rx=0 tx=0).stop 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 -- 172.21.15.59:0/3576647336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60840d4860 msgr2=0x7f60840c89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60840d4860 0x7f60840c89b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f606c00e9b0 tx=0x7f606c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 -- 172.21.15.59:0/3576647336 shutdown_connections 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606406b500 0x7f606406d9c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60841614d0 0x7f60840cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60840d4860 0x7f60840c89b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 --2- 172.21.15.59:0/3576647336 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60840b0e30 0x7f608415ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 -- 172.21.15.59:0/3576647336 >> 172.21.15.59:0/3576647336 conn(0x7f60840d3d40 msgr2=0x7f60840d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 -- 172.21.15.59:0/3576647336 shutdown_connections 2024-03-06T07:21:57.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.432+0000 7f60893d7640 1 -- 172.21.15.59:0/3576647336 wait complete. 2024-03-06T07:21:57.447 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:21:57.447 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:57.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.554+0000 7fc821d0f640 1 Processor -- start 2024-03-06T07:21:57.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.554+0000 7fc821d0f640 1 -- start start 2024-03-06T07:21:57.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.555+0000 7fc821d0f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc81c0d2270 0x7fc81c0d26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.555+0000 7fc821d0f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc81c0b4600 0x7fc81c0b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.555+0000 7fc821d0f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c0b3840 con 0x7fc81c1617e0 2024-03-06T07:21:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.555+0000 7fc821d0f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc81c0d2c10 con 0x7fc81c0b4600 2024-03-06T07:21:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.555+0000 7fc821d0f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0b35f0 con 0x7fc81c0d2270 2024-03-06T07:21:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.555+0000 7fc81bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc81c0b4600 0x7fc81c0b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.555+0000 7fc81affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc81c0d2270 0x7fc81c0d26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.555+0000 7fc81b7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc81c1617e0 0x7fc81c0cf460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:48010/0 (socket says 172.21.15.59:48010) 2024-03-06T07:21:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.555+0000 7fc81b7fe640 1 -- 172.21.15.59:0/4059447295 learned_addr learned my addr 172.21.15.59:0/4059447295 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.556+0000 7fc81bfff640 1 -- 172.21.15.59:0/4059447295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc81c0d2270 msgr2=0x7fc81c0d26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.556+0000 7fc81bfff640 1 --2- 172.21.15.59:0/4059447295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc81c0d2270 0x7fc81c0d26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.556+0000 7fc81bfff640 1 -- 172.21.15.59:0/4059447295 >> v1:172.21.15.164:6789/0 conn(0x7fc81c1617e0 legacy=0x7fc81c0cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.556+0000 7fc81bfff640 1 -- 172.21.15.59:0/4059447295 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c0b3b20 con 0x7fc81c0b4600 2024-03-06T07:21:57.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.556+0000 7fc81a7fc640 1 -- 172.21.15.59:0/4059447295 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386373891 0 0) 0x7fc81c0b3840 con 0x7fc81c1617e0 2024-03-06T07:21:57.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.556+0000 7fc81bfff640 1 --2- 172.21.15.59:0/4059447295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc81c0b4600 0x7fc81c0b4a90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc814009d90 tx=0x7fc81402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16e143cbe2cf5617 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.557+0000 7fc81a7fc640 1 -- 172.21.15.59:0/4059447295 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc814039070 con 0x7fc81c0b4600 2024-03-06T07:21:57.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.557+0000 7fc81a7fc640 1 -- 172.21.15.59:0/4059447295 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81402cbc0 con 0x7fc81c0b4600 2024-03-06T07:21:57.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.557+0000 7fc81a7fc640 1 -- 172.21.15.59:0/4059447295 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc814034440 con 0x7fc81c0b4600 2024-03-06T07:21:57.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.557+0000 7fc821d0f640 1 -- 172.21.15.59:0/4059447295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc81c0b4600 msgr2=0x7fc81c0b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.557+0000 7fc821d0f640 1 --2- 172.21.15.59:0/4059447295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc81c0b4600 0x7fc81c0b4a90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc814009d90 tx=0x7fc81402be10 comp rx=0 tx=0).stop 2024-03-06T07:21:57.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.557+0000 7fc821d0f640 1 -- 172.21.15.59:0/4059447295 shutdown_connections 2024-03-06T07:21:57.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.557+0000 7fc821d0f640 1 --2- 172.21.15.59:0/4059447295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc81c0b4600 0x7fc81c0b4a90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.557+0000 7fc821d0f640 1 --2- 172.21.15.59:0/4059447295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc81c0d2270 0x7fc81c0d26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.557+0000 7fc821d0f640 1 -- 172.21.15.59:0/4059447295 >> 172.21.15.59:0/4059447295 conn(0x7fc81c0d3c90 msgr2=0x7fc81c0d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.557+0000 7fc821d0f640 1 -- 172.21.15.59:0/4059447295 shutdown_connections 2024-03-06T07:21:57.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.557+0000 7fc821d0f640 1 -- 172.21.15.59:0/4059447295 wait complete. 2024-03-06T07:21:57.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.558+0000 7fc821d0f640 1 Processor -- start 2024-03-06T07:21:57.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.558+0000 7fc821d0f640 1 -- start start 2024-03-06T07:21:57.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.559+0000 7fc821d0f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc81c0b4600 0x7fc81c15be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.559+0000 7fc821d0f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc81c0d2270 0x7fc81c0c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.559+0000 7fc821d0f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc81c1617e0 0x7fc81c0ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.559+0000 7fc821d0f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0cfb70 con 0x7fc81c1617e0 2024-03-06T07:21:57.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.559+0000 7fc821d0f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0b4170 con 0x7fc81c0d2270 2024-03-06T07:21:57.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.559+0000 7fc821d0f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81c0d4990 con 0x7fc81c0b4600 2024-03-06T07:21:57.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.559+0000 7fc81bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc81c1617e0 0x7fc81c0ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.559+0000 7fc81bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc81c1617e0 0x7fc81c0ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57922/0 (socket says 172.21.15.59:57922) 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.559+0000 7fc81bfff640 1 -- 172.21.15.59:0/2014664499 learned_addr learned my addr 172.21.15.59:0/2014664499 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.559+0000 7fc81affd640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc81c0d2270 0x7fc81c0c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.559+0000 7fc81b7fe640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc81c0b4600 0x7fc81c15be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc81bfff640 1 -- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc81c0b4600 msgr2=0x7fc81c15be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc81bfff640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc81c0b4600 0x7fc81c15be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc81bfff640 1 -- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc81c0d2270 msgr2=0x7fc81c0c9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc81bfff640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc81c0d2270 0x7fc81c0c9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc81bfff640 1 -- 172.21.15.59:0/2014664499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c0c8590 con 0x7fc81c1617e0 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc81affd640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc81c0d2270 0x7fc81c0c9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc81bfff640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc81c1617e0 0x7fc81c0ccde0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc814037040 tx=0x7fc814004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc803fff640 1 -- 172.21.15.59:0/2014664499 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc814044030 con 0x7fc81c1617e0 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc803fff640 1 -- 172.21.15.59:0/2014664499 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81402c9c0 con 0x7fc81c1617e0 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc821d0f640 1 -- 172.21.15.59:0/2014664499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81c0c8870 con 0x7fc81c1617e0 2024-03-06T07:21:57.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc803fff640 1 -- 172.21.15.59:0/2014664499 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc814039040 con 0x7fc81c1617e0 2024-03-06T07:21:57.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.560+0000 7fc821d0f640 1 -- 172.21.15.59:0/2014664499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81c0c8e30 con 0x7fc81c1617e0 2024-03-06T07:21:57.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.561+0000 7fc821d0f640 1 -- 172.21.15.59:0/2014664499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc81c08cce0 con 0x7fc81c1617e0 2024-03-06T07:21:57.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.561+0000 7fc803fff640 1 -- 172.21.15.59:0/2014664499 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc81403f470 con 0x7fc81c1617e0 2024-03-06T07:21:57.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.562+0000 7fc803fff640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc80806b4d0 0x7fc80806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.562+0000 7fc803fff640 1 -- 172.21.15.59:0/2014664499 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8140ac900 con 0x7fc81c1617e0 2024-03-06T07:21:57.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.562+0000 7fc81b7fe640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc80806b4d0 0x7fc80806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.562+0000 7fc81b7fe640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc80806b4d0 0x7fc80806d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc80c007fd0 tx=0x7fc80c007de0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.566+0000 7fc803fff640 1 -- 172.21.15.59:0/2014664499 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc81c08cce0 con 0x7fc81c1617e0 2024-03-06T07:21:57.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.787+0000 7fc821d0f640 1 -- 172.21.15.59:0/2014664499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc81c04f330 con 0x7fc81c1617e0 2024-03-06T07:21:57.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.788+0000 7fc803fff640 1 -- 172.21.15.59:0/2014664499 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7fc81402cb90 con 0x7fc81c1617e0 2024-03-06T07:21:57.790 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:57.790 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1759048959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1759048959},{"type":"v1","addr":"172.21.15.164:6836","nonce":1759048959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/550023536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":550023536},{"type":"v1","addr":"172.21.15.59:6833","nonce":550023536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2132697832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2132697832},{"type":"v1","addr":"172.21.15.59:6835","nonce":2132697832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:54.767210+0000","modified":"2024-03-06T07:21:56.776415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/1957940318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1957940318},{"type":"v1","addr":"172.21.15.164:6837","nonce":1957940318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-06T07:21:57.790 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 6 2024-03-06T07:21:57.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 -- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc80806b4d0 msgr2=0x7fc80806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc80806b4d0 0x7fc80806d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc80c007fd0 tx=0x7fc80c007de0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 -- 172.21.15.59:0/2014664499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc81c1617e0 msgr2=0x7fc81c0ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc81c1617e0 0x7fc81c0ccde0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc814037040 tx=0x7fc814004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:57.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 -- 172.21.15.59:0/2014664499 shutdown_connections 2024-03-06T07:21:57.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc80806b4d0 0x7fc80806d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc81c1617e0 0x7fc81c0ccde0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc81c0d2270 0x7fc81c0c9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 --2- 172.21.15.59:0/2014664499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc81c0b4600 0x7fc81c15be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 -- 172.21.15.59:0/2014664499 >> 172.21.15.59:0/2014664499 conn(0x7fc81c0d3c90 msgr2=0x7fc81c0d1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 -- 172.21.15.59:0/2014664499 shutdown_connections 2024-03-06T07:21:57.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.790+0000 7fc821d0f640 1 -- 172.21.15.59:0/2014664499 wait complete. 2024-03-06T07:21:57.803 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:57.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.901+0000 7fbc4157a640 1 Processor -- start 2024-03-06T07:21:57.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.902+0000 7fbc4157a640 1 -- start start 2024-03-06T07:21:57.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.902+0000 7fbc4157a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbc3c072140 0x7fbc3c06db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.902+0000 7fbc4157a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc3c0adf70 con 0x7fbc3c0cf790 2024-03-06T07:21:57.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.902+0000 7fbc4157a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc3c0ae240 con 0x7fbc3c071ce0 2024-03-06T07:21:57.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.902+0000 7fbc4157a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc3c06cf70 con 0x7fbc3c072140 2024-03-06T07:21:57.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.902+0000 7fbc3affd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbc3c0cf790 0x7fbc3c0ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51100/0 (socket says 172.21.15.59:51100) 2024-03-06T07:21:57.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.902+0000 7fbc3affd640 1 -- 172.21.15.59:0/1453270013 learned_addr learned my addr 172.21.15.59:0/1453270013 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:57.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.902+0000 7fbc3b7fe640 1 --2- 172.21.15.59:0/1453270013 >> v2:172.21.15.59:3300/0 conn(0x7fbc3c072140 0x7fbc3c06db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.903+0000 7fbc3b7fe640 1 -- 172.21.15.59:0/1453270013 >> v1:172.21.15.164:6789/0 conn(0x7fbc3c071ce0 legacy=0x7fbc3c06d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.903+0000 7fbc3b7fe640 1 -- 172.21.15.59:0/1453270013 >> v1:172.21.15.59:6789/0 conn(0x7fbc3c0cf790 legacy=0x7fbc3c0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.903+0000 7fbc39ffb640 1 -- 172.21.15.59:0/1453270013 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518341282 0 0) 0x7fbc3c0ae240 con 0x7fbc3c071ce0 2024-03-06T07:21:57.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.903+0000 7fbc3b7fe640 1 -- 172.21.15.59:0/1453270013 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c0ae520 con 0x7fbc3c072140 2024-03-06T07:21:57.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.903+0000 7fbc3b7fe640 1 --2- 172.21.15.59:0/1453270013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3c072140 0x7fbc3c06db10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009d90 tx=0x7fbc2c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f3b218589615d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.904+0000 7fbc39ffb640 1 -- 172.21.15.59:0/1453270013 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c039070 con 0x7fbc3c072140 2024-03-06T07:21:57.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.904+0000 7fbc39ffb640 1 -- 172.21.15.59:0/1453270013 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2c02cab0 con 0x7fbc3c072140 2024-03-06T07:21:57.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.904+0000 7fbc39ffb640 1 -- 172.21.15.59:0/1453270013 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c034440 con 0x7fbc3c072140 2024-03-06T07:21:57.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.904+0000 7fbc4157a640 1 -- 172.21.15.59:0/1453270013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3c072140 msgr2=0x7fbc3c06db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.904+0000 7fbc4157a640 1 --2- 172.21.15.59:0/1453270013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3c072140 0x7fbc3c06db10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009d90 tx=0x7fbc2c02bd80 comp rx=0 tx=0).stop 2024-03-06T07:21:57.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.904+0000 7fbc4157a640 1 -- 172.21.15.59:0/1453270013 shutdown_connections 2024-03-06T07:21:57.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.904+0000 7fbc4157a640 1 --2- 172.21.15.59:0/1453270013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3c072140 0x7fbc3c06db10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.904+0000 7fbc4157a640 1 -- 172.21.15.59:0/1453270013 >> 172.21.15.59:0/1453270013 conn(0x7fbc3c0b1a90 msgr2=0x7fbc3c0b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.904+0000 7fbc4157a640 1 -- 172.21.15.59:0/1453270013 shutdown_connections 2024-03-06T07:21:57.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.904+0000 7fbc4157a640 1 -- 172.21.15.59:0/1453270013 wait complete. 2024-03-06T07:21:57.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.905+0000 7fbc4157a640 1 Processor -- start 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc4157a640 1 -- start start 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc4157a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3c071ce0 0x7fbc3c1e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc4157a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc3c072140 0x7fbc3c13c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc4157a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc3c0cf790 0x7fbc3c1570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc4157a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c0725d0 con 0x7fbc3c071ce0 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc4157a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c0aeae0 con 0x7fbc3c0cf790 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc4157a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c15e580 con 0x7fbc3c072140 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc3affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3c071ce0 0x7fbc3c1e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc3affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3c071ce0 0x7fbc3c1e81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57942/0 (socket says 172.21.15.59:57942) 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc3affd640 1 -- 172.21.15.59:0/147674268 learned_addr learned my addr 172.21.15.59:0/147674268 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc3a7fc640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc3c072140 0x7fbc3c13c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.906+0000 7fbc3b7fe640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc3c0cf790 0x7fbc3c1570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc3affd640 1 -- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc3c072140 msgr2=0x7fbc3c13c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc3affd640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc3c072140 0x7fbc3c13c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc3affd640 1 -- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc3c0cf790 msgr2=0x7fbc3c1570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc3affd640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc3c0cf790 0x7fbc3c1570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc3affd640 1 -- 172.21.15.59:0/147674268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c1526f0 con 0x7fbc3c071ce0 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc3b7fe640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc3c0cf790 0x7fbc3c1570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc3a7fc640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc3c072140 0x7fbc3c13c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc3affd640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3c071ce0 0x7fbc3c1e81e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400c520 tx=0x7fbc3400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc237fe640 1 -- 172.21.15.59:0/147674268 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34017070 con 0x7fbc3c071ce0 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc237fe640 1 -- 172.21.15.59:0/147674268 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc340074e0 con 0x7fbc3c071ce0 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc4157a640 1 -- 172.21.15.59:0/147674268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc3c1529d0 con 0x7fbc3c071ce0 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc237fe640 1 -- 172.21.15.59:0/147674268 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34004a00 con 0x7fbc3c071ce0 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.907+0000 7fbc4157a640 1 -- 172.21.15.59:0/147674268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc3c152f90 con 0x7fbc3c071ce0 2024-03-06T07:21:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.908+0000 7fbc4157a640 1 -- 172.21.15.59:0/147674268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3c04f330 con 0x7fbc3c071ce0 2024-03-06T07:21:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.908+0000 7fbc237fe640 1 -- 172.21.15.59:0/147674268 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc3401a070 con 0x7fbc3c071ce0 2024-03-06T07:21:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.909+0000 7fbc237fe640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 0x7fbc2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.909+0000 7fbc237fe640 1 -- 172.21.15.59:0/147674268 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc34089670 con 0x7fbc3c071ce0 2024-03-06T07:21:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.909+0000 7fbc3a7fc640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 0x7fbc2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.909+0000 7fbc3a7fc640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 0x7fbc2806d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbc30007b20 tx=0x7fbc30009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:57.913+0000 7fbc237fe640 1 -- 172.21.15.59:0/147674268 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc340584f0 con 0x7fbc3c071ce0 2024-03-06T07:21:58.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.093+0000 7fbc4157a640 1 -- 172.21.15.59:0/147674268 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc3c0b1870 con 0x7fbc3c071ce0 2024-03-06T07:21:58.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.094+0000 7fbc237fe640 1 -- 172.21.15.59:0/147674268 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fbc34057df0 con 0x7fbc3c071ce0 2024-03-06T07:21:58.096 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:58.096 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:58.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 -- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 msgr2=0x7fbc2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 0x7fbc2806d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbc30007b20 tx=0x7fbc30009040 comp rx=0 tx=0).stop 2024-03-06T07:21:58.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 -- 172.21.15.59:0/147674268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3c071ce0 msgr2=0x7fbc3c1e81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3c071ce0 0x7fbc3c1e81e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400c520 tx=0x7fbc3400c9e0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 -- 172.21.15.59:0/147674268 shutdown_connections 2024-03-06T07:21:58.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 0x7fbc2806d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc3c0cf790 0x7fbc3c1570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc3c072140 0x7fbc3c13c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 --2- 172.21.15.59:0/147674268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3c071ce0 0x7fbc3c1e81e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 -- 172.21.15.59:0/147674268 >> 172.21.15.59:0/147674268 conn(0x7fbc3c0b1a90 msgr2=0x7fbc3c1602f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 -- 172.21.15.59:0/147674268 shutdown_connections 2024-03-06T07:21:58.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.096+0000 7fbc4157a640 1 -- 172.21.15.59:0/147674268 wait complete. 2024-03-06T07:21:58.109 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:21:58.109 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:58.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.208+0000 7fc9a67bc640 1 Processor -- start 2024-03-06T07:21:58.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.208+0000 7fc9a67bc640 1 -- start start 2024-03-06T07:21:58.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc9a67bc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc9a00d22b0 0x7fc9a00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc9a67bc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a00b3430 con 0x7fc9a015dfe0 2024-03-06T07:21:58.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc9a67bc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a00b3710 con 0x7fc9a00b41f0 2024-03-06T07:21:58.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc9a67bc640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a00d2c50 con 0x7fc9a00d22b0 2024-03-06T07:21:58.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc99f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc9a00d22b0 0x7fc9a00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc99f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc9a00d22b0 0x7fc9a00d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57944/0 (socket says 172.21.15.59:57944) 2024-03-06T07:21:58.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc99ffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc9a015dfe0 0x7fc9a00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51116/0 (socket says 172.21.15.59:51116) 2024-03-06T07:21:58.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc99f7fe640 1 -- 172.21.15.59:0/192864259 learned_addr learned my addr 172.21.15.59:0/192864259 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:58.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc99f7fe640 1 -- 172.21.15.59:0/192864259 >> v1:172.21.15.164:6789/0 conn(0x7fc9a00b41f0 legacy=0x7fc9a00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc99f7fe640 1 -- 172.21.15.59:0/192864259 >> v1:172.21.15.59:6789/0 conn(0x7fc9a015dfe0 legacy=0x7fc9a00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc99f7fe640 1 -- 172.21.15.59:0/192864259 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a00b39f0 con 0x7fc9a00d22b0 2024-03-06T07:21:58.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.209+0000 7fc99effd640 1 -- 172.21.15.59:0/192864259 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641640080 0 0) 0x7fc9a00b3430 con 0x7fc9a015dfe0 2024-03-06T07:21:58.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.210+0000 7fc99f7fe640 1 --2- 172.21.15.59:0/192864259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9a00d22b0 0x7fc9a00d2710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9900098a0 tx=0x7fc99002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2844e437e4b34ccb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.210+0000 7fc99effd640 1 -- 172.21.15.59:0/192864259 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99002f840 con 0x7fc9a00d22b0 2024-03-06T07:21:58.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.210+0000 7fc99effd640 1 -- 172.21.15.59:0/192864259 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99002f9e0 con 0x7fc9a00d22b0 2024-03-06T07:21:58.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.210+0000 7fc99effd640 1 -- 172.21.15.59:0/192864259 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99002fdb0 con 0x7fc9a00d22b0 2024-03-06T07:21:58.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.211+0000 7fc9a67bc640 1 -- 172.21.15.59:0/192864259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9a00d22b0 msgr2=0x7fc9a00d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.211+0000 7fc9a67bc640 1 --2- 172.21.15.59:0/192864259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9a00d22b0 0x7fc9a00d2710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9900098a0 tx=0x7fc99002ba20 comp rx=0 tx=0).stop 2024-03-06T07:21:58.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.211+0000 7fc9a67bc640 1 -- 172.21.15.59:0/192864259 shutdown_connections 2024-03-06T07:21:58.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.211+0000 7fc9a67bc640 1 --2- 172.21.15.59:0/192864259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9a00d22b0 0x7fc9a00d2710 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.211+0000 7fc9a67bc640 1 -- 172.21.15.59:0/192864259 >> 172.21.15.59:0/192864259 conn(0x7fc9a00d3cd0 msgr2=0x7fc9a00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.211+0000 7fc9a67bc640 1 -- 172.21.15.59:0/192864259 shutdown_connections 2024-03-06T07:21:58.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.211+0000 7fc9a67bc640 1 -- 172.21.15.59:0/192864259 wait complete. 2024-03-06T07:21:58.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.212+0000 7fc9a67bc640 1 Processor -- start 2024-03-06T07:21:58.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.212+0000 7fc9a67bc640 1 -- start start 2024-03-06T07:21:58.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc9a67bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a00b41f0 0x7fc9a0160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc9a67bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9a00d22b0 0x7fc9a00b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc9a67bc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a015dfe0 0x7fc9a015fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc9a67bc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a00b3d60 con 0x7fc9a00d22b0 2024-03-06T07:21:58.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc9a67bc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a00b31e0 con 0x7fc9a015dfe0 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc9a67bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a00d1c70 con 0x7fc9a00b41f0 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc99f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9a00d22b0 0x7fc9a00b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc99f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9a00d22b0 0x7fc9a00b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57950/0 (socket says 172.21.15.59:57950) 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc99f7fe640 1 -- 172.21.15.59:0/1682713917 learned_addr learned my addr 172.21.15.59:0/1682713917 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc99ffff640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a00b41f0 0x7fc9a0160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc9a4d59640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a015dfe0 0x7fc9a015fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc99f7fe640 1 -- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a00b41f0 msgr2=0x7fc9a0160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc99f7fe640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a00b41f0 0x7fc9a0160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc99f7fe640 1 -- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a015dfe0 msgr2=0x7fc9a015fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc99f7fe640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a015dfe0 0x7fc9a015fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc99f7fe640 1 -- 172.21.15.59:0/1682713917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a015e910 con 0x7fc9a00d22b0 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.213+0000 7fc99f7fe640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9a00d22b0 0x7fc9a00b68a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc9900090b0 tx=0x7fc990004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.214+0000 7fc99cff9640 1 -- 172.21.15.59:0/1682713917 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990047070 con 0x7fc9a00d22b0 2024-03-06T07:21:58.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.214+0000 7fc99cff9640 1 -- 172.21.15.59:0/1682713917 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc990037c40 con 0x7fc9a00d22b0 2024-03-06T07:21:58.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.214+0000 7fc9a67bc640 1 -- 172.21.15.59:0/1682713917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a015ebf0 con 0x7fc9a00d22b0 2024-03-06T07:21:58.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.214+0000 7fc99cff9640 1 -- 172.21.15.59:0/1682713917 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990004030 con 0x7fc9a00d22b0 2024-03-06T07:21:58.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.214+0000 7fc9a67bc640 1 -- 172.21.15.59:0/1682713917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a015f1b0 con 0x7fc9a00d22b0 2024-03-06T07:21:58.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.215+0000 7fc9a67bc640 1 -- 172.21.15.59:0/1682713917 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a004f330 con 0x7fc9a00d22b0 2024-03-06T07:21:58.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.215+0000 7fc99cff9640 1 -- 172.21.15.59:0/1682713917 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc99004a070 con 0x7fc9a00d22b0 2024-03-06T07:21:58.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.215+0000 7fc99cff9640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc98c06b4d0 0x7fc98c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.216+0000 7fc99cff9640 1 -- 172.21.15.59:0/1682713917 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9900ac290 con 0x7fc9a00d22b0 2024-03-06T07:21:58.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.216+0000 7fc99ffff640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc98c06b4d0 0x7fc98c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.216+0000 7fc99ffff640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc98c06b4d0 0x7fc98c06d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc994000f60 tx=0x7fc994007e40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.220+0000 7fc99cff9640 1 -- 172.21.15.59:0/1682713917 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc990033030 con 0x7fc9a00d22b0 2024-03-06T07:21:58.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.397+0000 7fc9a67bc640 1 -- 172.21.15.59:0/1682713917 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9a00ca210 con 0x7fc98c06b4d0 2024-03-06T07:21:58.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.404+0000 7fc99cff9640 1 -- 172.21.15.59:0/1682713917 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251878 (secure 0 0 0) 0x7fc9a00ca210 con 0x7fc98c06b4d0 2024-03-06T07:21:58.408 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:58.410 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:21:58.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.406+0000 7fc9a67bc640 1 -- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc98c06b4d0 msgr2=0x7fc98c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.406+0000 7fc9a67bc640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc98c06b4d0 0x7fc98c06d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc994000f60 tx=0x7fc994007e40 comp rx=0 tx=0).stop 2024-03-06T07:21:58.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.406+0000 7fc9a67bc640 1 -- 172.21.15.59:0/1682713917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9a00d22b0 msgr2=0x7fc9a00b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.406+0000 7fc9a67bc640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9a00d22b0 0x7fc9a00b68a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc9900090b0 tx=0x7fc990004730 comp rx=0 tx=0).stop 2024-03-06T07:21:58.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.406+0000 7fc9a67bc640 1 -- 172.21.15.59:0/1682713917 shutdown_connections 2024-03-06T07:21:58.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.406+0000 7fc9a67bc640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc98c06b4d0 0x7fc98c06d990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.406+0000 7fc9a67bc640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a015dfe0 0x7fc9a015fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.406+0000 7fc9a67bc640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9a00d22b0 0x7fc9a00b68a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.406+0000 7fc9a67bc640 1 --2- 172.21.15.59:0/1682713917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a00b41f0 0x7fc9a0160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.407+0000 7fc9a67bc640 1 -- 172.21.15.59:0/1682713917 >> 172.21.15.59:0/1682713917 conn(0x7fc9a00d3cd0 msgr2=0x7fc9a015d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.407+0000 7fc9a67bc640 1 -- 172.21.15.59:0/1682713917 shutdown_connections 2024-03-06T07:21:58.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.407+0000 7fc9a67bc640 1 -- 172.21.15.59:0/1682713917 wait complete. 2024-03-06T07:21:58.419 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-06T07:21:57.303056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":99,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218904,"kb_used_data":2728,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755816,"statfs":{"total":773094113280,"available":772869955584,"internally_reserved":0,"allocated":2793472,"data_stored":1702628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000792"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.431306+0000","last_change":"2024-03-06T07:21:55.429371+0000","last_active":"2024-03-06T07:21:55.431306+0000","last_peered":"2024-03-06T07:21:55.431306+0000","last_clean":"2024-03-06T07:21:55.431306+0000","last_became_active":"2024-03-06T07:21:55.429057+0000","last_became_peered":"2024-03-06T07:21:55.429057+0000","last_unstale":"2024-03-06T07:21:55.431306+0000","last_undegraded":"2024-03-06T07:21:55.431306+0000","last_fullsized":"2024-03-06T07:21:55.431306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.432159+0000","last_change":"2024-03-06T07:21:55.420380+0000","last_active":"2024-03-06T07:21:55.432159+0000","last_peered":"2024-03-06T07:21:55.432159+0000","last_clean":"2024-03-06T07:21:55.432159+0000","last_became_active":"2024-03-06T07:21:55.419640+0000","last_became_peered":"2024-03-06T07:21:55.419640+0000","last_unstale":"2024-03-06T07:21:55.432159+0000","last_undegraded":"2024-03-06T07:21:55.432159+0000","last_fullsized":"2024-03-06T07:21:55.432159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:29.441742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.429024+0000","last_change":"2024-03-06T07:21:55.418047+0000","last_active":"2024-03-06T07:21:55.429024+0000","last_peered":"2024-03-06T07:21:55.429024+0000","last_clean":"2024-03-06T07:21:55.429024+0000","last_became_active":"2024-03-06T07:21:55.417536+0000","last_became_peered":"2024-03-06T07:21:55.417536+0000","last_unstale":"2024-03-06T07:21:55.429024+0000","last_undegraded":"2024-03-06T07:21:55.429024+0000","last_fullsized":"2024-03-06T07:21:55.429024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:39.492542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435650+0000","last_change":"2024-03-06T07:21:55.422738+0000","last_active":"2024-03-06T07:21:55.435650+0000","last_peered":"2024-03-06T07:21:55.435650+0000","last_clean":"2024-03-06T07:21:55.435650+0000","last_became_active":"2024-03-06T07:21:55.422460+0000","last_became_peered":"2024-03-06T07:21:55.422460+0000","last_unstale":"2024-03-06T07:21:55.435650+0000","last_undegraded":"2024-03-06T07:21:55.435650+0000","last_fullsized":"2024-03-06T07:21:55.435650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.427805+0000","last_change":"2024-03-06T07:21:55.423628+0000","last_active":"2024-03-06T07:21:55.427805+0000","last_peered":"2024-03-06T07:21:55.427805+0000","last_clean":"2024-03-06T07:21:55.427805+0000","last_became_active":"2024-03-06T07:21:55.423360+0000","last_became_peered":"2024-03-06T07:21:55.423360+0000","last_unstale":"2024-03-06T07:21:55.427805+0000","last_undegraded":"2024-03-06T07:21:55.427805+0000","last_fullsized":"2024-03-06T07:21:55.427805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435728+0000","last_change":"2024-03-06T07:21:55.423187+0000","last_active":"2024-03-06T07:21:55.435728+0000","last_peered":"2024-03-06T07:21:55.435728+0000","last_clean":"2024-03-06T07:21:55.435728+0000","last_became_active":"2024-03-06T07:21:55.422907+0000","last_became_peered":"2024-03-06T07:21:55.422907+0000","last_unstale":"2024-03-06T07:21:55.435728+0000","last_undegraded":"2024-03-06T07:21:55.435728+0000","last_fullsized":"2024-03-06T07:21:55.435728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.422051+0000","last_change":"2024-03-06T07:21:55.419720+0000","last_active":"2024-03-06T07:21:55.422051+0000","last_peered":"2024-03-06T07:21:55.422051+0000","last_clean":"2024-03-06T07:21:55.422051+0000","last_became_active":"2024-03-06T07:21:55.419294+0000","last_became_peered":"2024-03-06T07:21:55.419294+0000","last_unstale":"2024-03-06T07:21:55.422051+0000","last_undegraded":"2024-03-06T07:21:55.422051+0000","last_fullsized":"2024-03-06T07:21:55.422051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.427044+0000","last_change":"2024-03-06T07:21:55.423177+0000","last_active":"2024-03-06T07:21:55.427044+0000","last_peered":"2024-03-06T07:21:55.427044+0000","last_clean":"2024-03-06T07:21:55.427044+0000","last_became_active":"2024-03-06T07:21:55.422875+0000","last_became_peered":"2024-03-06T07:21:55.422875+0000","last_unstale":"2024-03-06T07:21:55.427044+0000","last_undegraded":"2024-03-06T07:21:55.427044+0000","last_fullsized":"2024-03-06T07:21:55.427044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440175+0000","last_change":"2024-03-06T07:21:55.436966+0000","last_active":"2024-03-06T07:21:55.440175+0000","last_peered":"2024-03-06T07:21:55.440175+0000","last_clean":"2024-03-06T07:21:55.440175+0000","last_became_active":"2024-03-06T07:21:55.436375+0000","last_became_peered":"2024-03-06T07:21:55.436375+0000","last_unstale":"2024-03-06T07:21:55.440175+0000","last_undegraded":"2024-03-06T07:21:55.440175+0000","last_fullsized":"2024-03-06T07:21:55.440175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:41.154073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440431+0000","last_change":"2024-03-06T07:21:55.436554+0000","last_active":"2024-03-06T07:21:55.440431+0000","last_peered":"2024-03-06T07:21:55.440431+0000","last_clean":"2024-03-06T07:21:55.440431+0000","last_became_active":"2024-03-06T07:21:55.436292+0000","last_became_peered":"2024-03-06T07:21:55.436292+0000","last_unstale":"2024-03-06T07:21:55.440431+0000","last_undegraded":"2024-03-06T07:21:55.440431+0000","last_fullsized":"2024-03-06T07:21:55.440431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435647+0000","last_change":"2024-03-06T07:21:55.432397+0000","last_active":"2024-03-06T07:21:55.435647+0000","last_peered":"2024-03-06T07:21:55.435647+0000","last_clean":"2024-03-06T07:21:55.435647+0000","last_became_active":"2024-03-06T07:21:55.432090+0000","last_became_peered":"2024-03-06T07:21:55.432090+0000","last_unstale":"2024-03-06T07:21:55.435647+0000","last_undegraded":"2024-03-06T07:21:55.435647+0000","last_fullsized":"2024-03-06T07:21:55.435647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:41.897729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.426033+0000","last_change":"2024-03-06T07:21:55.423196+0000","last_active":"2024-03-06T07:21:55.426033+0000","last_peered":"2024-03-06T07:21:55.426033+0000","last_clean":"2024-03-06T07:21:55.426033+0000","last_became_active":"2024-03-06T07:21:55.422079+0000","last_became_peered":"2024-03-06T07:21:55.422079+0000","last_unstale":"2024-03-06T07:21:55.426033+0000","last_undegraded":"2024-03-06T07:21:55.426033+0000","last_fullsized":"2024-03-06T07:21:55.426033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:18.465323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424815+0000","last_change":"2024-03-06T07:21:55.421461+0000","last_active":"2024-03-06T07:21:55.424815+0000","last_peered":"2024-03-06T07:21:55.424815+0000","last_clean":"2024-03-06T07:21:55.424815+0000","last_became_active":"2024-03-06T07:21:55.420795+0000","last_became_peered":"2024-03-06T07:21:55.420795+0000","last_unstale":"2024-03-06T07:21:55.424815+0000","last_undegraded":"2024-03-06T07:21:55.424815+0000","last_fullsized":"2024-03-06T07:21:55.424815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435040+0000","last_change":"2024-03-06T07:21:55.420082+0000","last_active":"2024-03-06T07:21:55.435040+0000","last_peered":"2024-03-06T07:21:55.435040+0000","last_clean":"2024-03-06T07:21:55.435040+0000","last_became_active":"2024-03-06T07:21:55.419844+0000","last_became_peered":"2024-03-06T07:21:55.419844+0000","last_unstale":"2024-03-06T07:21:55.435040+0000","last_undegraded":"2024-03-06T07:21:55.435040+0000","last_fullsized":"2024-03-06T07:21:55.435040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:43.203590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.432820+0000","last_change":"2024-03-06T07:21:55.425776+0000","last_active":"2024-03-06T07:21:55.432820+0000","last_peered":"2024-03-06T07:21:55.432820+0000","last_clean":"2024-03-06T07:21:55.432820+0000","last_became_active":"2024-03-06T07:21:55.425479+0000","last_became_peered":"2024-03-06T07:21:55.425479+0000","last_unstale":"2024-03-06T07:21:55.432820+0000","last_undegraded":"2024-03-06T07:21:55.432820+0000","last_fullsized":"2024-03-06T07:21:55.432820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.430753+0000","last_change":"2024-03-06T07:21:55.416970+0000","last_active":"2024-03-06T07:21:55.430753+0000","last_peered":"2024-03-06T07:21:55.430753+0000","last_clean":"2024-03-06T07:21:55.430753+0000","last_became_active":"2024-03-06T07:21:55.416495+0000","last_became_peered":"2024-03-06T07:21:55.416495+0000","last_unstale":"2024-03-06T07:21:55.430753+0000","last_undegraded":"2024-03-06T07:21:55.430753+0000","last_fullsized":"2024-03-06T07:21:55.430753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.439989+0000","last_change":"2024-03-06T07:21:55.430721+0000","last_active":"2024-03-06T07:21:55.439989+0000","last_peered":"2024-03-06T07:21:55.439989+0000","last_clean":"2024-03-06T07:21:55.439989+0000","last_became_active":"2024-03-06T07:21:55.428901+0000","last_became_peered":"2024-03-06T07:21:55.428901+0000","last_unstale":"2024-03-06T07:21:55.439989+0000","last_undegraded":"2024-03-06T07:21:55.439989+0000","last_fullsized":"2024-03-06T07:21:55.439989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.432206+0000","last_change":"2024-03-06T07:21:55.428378+0000","last_active":"2024-03-06T07:21:55.432206+0000","last_peered":"2024-03-06T07:21:55.432206+0000","last_clean":"2024-03-06T07:21:55.432206+0000","last_became_active":"2024-03-06T07:21:55.428101+0000","last_became_peered":"2024-03-06T07:21:55.428101+0000","last_unstale":"2024-03-06T07:21:55.432206+0000","last_undegraded":"2024-03-06T07:21:55.432206+0000","last_fullsized":"2024-03-06T07:21:55.432206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:43.203590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.428760+0000","last_change":"2024-03-06T07:21:55.425219+0000","last_active":"2024-03-06T07:21:55.428760+0000","last_peered":"2024-03-06T07:21:55.428760+0000","last_clean":"2024-03-06T07:21:55.428760+0000","last_became_active":"2024-03-06T07:21:55.422831+0000","last_became_peered":"2024-03-06T07:21:55.422831+0000","last_unstale":"2024-03-06T07:21:55.428760+0000","last_undegraded":"2024-03-06T07:21:55.428760+0000","last_fullsized":"2024-03-06T07:21:55.428760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435751+0000","last_change":"2024-03-06T07:21:55.427953+0000","last_active":"2024-03-06T07:21:55.435751+0000","last_peered":"2024-03-06T07:21:55.435751+0000","last_clean":"2024-03-06T07:21:55.435751+0000","last_became_active":"2024-03-06T07:21:55.427652+0000","last_became_peered":"2024-03-06T07:21:55.427652+0000","last_unstale":"2024-03-06T07:21:55.435751+0000","last_undegraded":"2024-03-06T07:21:55.435751+0000","last_fullsized":"2024-03-06T07:21:55.435751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.434959+0000","last_change":"2024-03-06T07:21:55.421525+0000","last_active":"2024-03-06T07:21:55.434959+0000","last_peered":"2024-03-06T07:21:55.434959+0000","last_clean":"2024-03-06T07:21:55.434959+0000","last_became_active":"2024-03-06T07:21:55.421204+0000","last_became_peered":"2024-03-06T07:21:55.421204+0000","last_unstale":"2024-03-06T07:21:55.434959+0000","last_undegraded":"2024-03-06T07:21:55.434959+0000","last_fullsized":"2024-03-06T07:21:55.434959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:36.844180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435710+0000","last_change":"2024-03-06T07:21:55.424534+0000","last_active":"2024-03-06T07:21:55.435710+0000","last_peered":"2024-03-06T07:21:55.435710+0000","last_clean":"2024-03-06T07:21:55.435710+0000","last_became_active":"2024-03-06T07:21:55.424309+0000","last_became_peered":"2024-03-06T07:21:55.424309+0000","last_unstale":"2024-03-06T07:21:55.435710+0000","last_undegraded":"2024-03-06T07:21:55.435710+0000","last_fullsized":"2024-03-06T07:21:55.435710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.430301+0000","last_change":"2024-03-06T07:21:55.424084+0000","last_active":"2024-03-06T07:21:55.430301+0000","last_peered":"2024-03-06T07:21:55.430301+0000","last_clean":"2024-03-06T07:21:55.430301+0000","last_became_active":"2024-03-06T07:21:55.423245+0000","last_became_peered":"2024-03-06T07:21:55.423245+0000","last_unstale":"2024-03-06T07:21:55.430301+0000","last_undegraded":"2024-03-06T07:21:55.430301+0000","last_fullsized":"2024-03-06T07:21:55.430301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:29.441742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.420348+0000","last_change":"2024-03-06T07:21:55.415889+0000","last_active":"2024-03-06T07:21:55.420348+0000","last_peered":"2024-03-06T07:21:55.420348+0000","last_clean":"2024-03-06T07:21:55.420348+0000","last_became_active":"2024-03-06T07:21:55.415620+0000","last_became_peered":"2024-03-06T07:21:55.415620+0000","last_unstale":"2024-03-06T07:21:55.420348+0000","last_undegraded":"2024-03-06T07:21:55.420348+0000","last_fullsized":"2024-03-06T07:21:55.420348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424968+0000","last_change":"2024-03-06T07:21:55.422611+0000","last_active":"2024-03-06T07:21:55.424968+0000","last_peered":"2024-03-06T07:21:55.424968+0000","last_clean":"2024-03-06T07:21:55.424968+0000","last_became_active":"2024-03-06T07:21:55.422260+0000","last_became_peered":"2024-03-06T07:21:55.422260+0000","last_unstale":"2024-03-06T07:21:55.424968+0000","last_undegraded":"2024-03-06T07:21:55.424968+0000","last_fullsized":"2024-03-06T07:21:55.424968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.430230+0000","last_change":"2024-03-06T07:21:55.420808+0000","last_active":"2024-03-06T07:21:55.430230+0000","last_peered":"2024-03-06T07:21:55.430230+0000","last_clean":"2024-03-06T07:21:55.430230+0000","last_became_active":"2024-03-06T07:21:55.420529+0000","last_became_peered":"2024-03-06T07:21:55.420529+0000","last_unstale":"2024-03-06T07:21:55.430230+0000","last_undegraded":"2024-03-06T07:21:55.430230+0000","last_fullsized":"2024-03-06T07:21:55.430230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.433865+0000","last_change":"2024-03-06T07:21:55.420655+0000","last_active":"2024-03-06T07:21:55.433865+0000","last_peered":"2024-03-06T07:21:55.433865+0000","last_clean":"2024-03-06T07:21:55.433865+0000","last_became_active":"2024-03-06T07:21:55.420396+0000","last_became_peered":"2024-03-06T07:21:55.420396+0000","last_unstale":"2024-03-06T07:21:55.433865+0000","last_undegraded":"2024-03-06T07:21:55.433865+0000","last_fullsized":"2024-03-06T07:21:55.433865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435187+0000","last_change":"2024-03-06T07:21:55.431350+0000","last_active":"2024-03-06T07:21:55.435187+0000","last_peered":"2024-03-06T07:21:55.435187+0000","last_clean":"2024-03-06T07:21:55.435187+0000","last_became_active":"2024-03-06T07:21:55.431033+0000","last_became_peered":"2024-03-06T07:21:55.431033+0000","last_unstale":"2024-03-06T07:21:55.435187+0000","last_undegraded":"2024-03-06T07:21:55.435187+0000","last_fullsized":"2024-03-06T07:21:55.435187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.418173+0000","last_change":"2024-03-06T07:21:55.416463+0000","last_active":"2024-03-06T07:21:55.418173+0000","last_peered":"2024-03-06T07:21:55.418173+0000","last_clean":"2024-03-06T07:21:55.418173+0000","last_became_active":"2024-03-06T07:21:55.416174+0000","last_became_peered":"2024-03-06T07:21:55.416174+0000","last_unstale":"2024-03-06T07:21:55.418173+0000","last_undegraded":"2024-03-06T07:21:55.418173+0000","last_fullsized":"2024-03-06T07:21:55.418173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:06.592646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435688+0000","last_change":"2024-03-06T07:21:55.418595+0000","last_active":"2024-03-06T07:21:55.435688+0000","last_peered":"2024-03-06T07:21:55.435688+0000","last_clean":"2024-03-06T07:21:55.435688+0000","last_became_active":"2024-03-06T07:21:55.418073+0000","last_became_peered":"2024-03-06T07:21:55.418073+0000","last_unstale":"2024-03-06T07:21:55.435688+0000","last_undegraded":"2024-03-06T07:21:55.435688+0000","last_fullsized":"2024-03-06T07:21:55.435688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.433077+0000","last_change":"2024-03-06T07:21:55.429932+0000","last_active":"2024-03-06T07:21:55.433077+0000","last_peered":"2024-03-06T07:21:55.433077+0000","last_clean":"2024-03-06T07:21:55.433077+0000","last_became_active":"2024-03-06T07:21:55.429593+0000","last_became_peered":"2024-03-06T07:21:55.429593+0000","last_unstale":"2024-03-06T07:21:55.433077+0000","last_undegraded":"2024-03-06T07:21:55.433077+0000","last_fullsized":"2024-03-06T07:21:55.433077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:41.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440407+0000","last_change":"2024-03-06T07:21:55.436974+0000","last_active":"2024-03-06T07:21:55.440407+0000","last_peered":"2024-03-06T07:21:55.440407+0000","last_clean":"2024-03-06T07:21:55.440407+0000","last_became_active":"2024-03-06T07:21:55.436638+0000","last_became_peered":"2024-03-06T07:21:55.436638+0000","last_unstale":"2024-03-06T07:21:55.440407+0000","last_undegraded":"2024-03-06T07:21:55.440407+0000","last_fullsized":"2024-03-06T07:21:55.440407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440385+0000","last_change":"2024-03-06T07:21:55.435717+0000","last_active":"2024-03-06T07:21:55.440385+0000","last_peered":"2024-03-06T07:21:55.440385+0000","last_clean":"2024-03-06T07:21:55.440385+0000","last_became_active":"2024-03-06T07:21:55.435429+0000","last_became_peered":"2024-03-06T07:21:55.435429+0000","last_unstale":"2024-03-06T07:21:55.440385+0000","last_undegraded":"2024-03-06T07:21:55.440385+0000","last_fullsized":"2024-03-06T07:21:55.440385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:39.492542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.437829+0000","last_change":"2024-03-06T07:21:54.413886+0000","last_active":"2024-03-06T07:21:55.437829+0000","last_peered":"2024-03-06T07:21:55.437829+0000","last_clean":"2024-03-06T07:21:55.437829+0000","last_became_active":"2024-03-06T07:21:54.413573+0000","last_became_peered":"2024-03-06T07:21:54.413573+0000","last_unstale":"2024-03-06T07:21:55.437829+0000","last_undegraded":"2024-03-06T07:21:55.437829+0000","last_fullsized":"2024-03-06T07:21:55.437829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:17.458904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435069+0000","last_change":"2024-03-06T07:21:55.427074+0000","last_active":"2024-03-06T07:21:55.435069+0000","last_peered":"2024-03-06T07:21:55.435069+0000","last_clean":"2024-03-06T07:21:55.435069+0000","last_became_active":"2024-03-06T07:21:55.426775+0000","last_became_peered":"2024-03-06T07:21:55.426775+0000","last_unstale":"2024-03-06T07:21:55.435069+0000","last_undegraded":"2024-03-06T07:21:55.435069+0000","last_fullsized":"2024-03-06T07:21:55.435069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440666+0000","last_change":"2024-03-06T07:21:54.418543+0000","last_active":"2024-03-06T07:21:55.440666+0000","last_peered":"2024-03-06T07:21:55.440666+0000","last_clean":"2024-03-06T07:21:55.440666+0000","last_became_active":"2024-03-06T07:21:54.418148+0000","last_became_peered":"2024-03-06T07:21:54.418148+0000","last_unstale":"2024-03-06T07:21:55.440666+0000","last_undegraded":"2024-03-06T07:21:55.440666+0000","last_fullsized":"2024-03-06T07:21:55.440666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.410777+0000","last_change":"2024-03-06T07:21:54.417756+0000","last_active":"2024-03-06T07:21:55.410777+0000","last_peered":"2024-03-06T07:21:55.410777+0000","last_clean":"2024-03-06T07:21:55.410777+0000","last_became_active":"2024-03-06T07:21:54.417471+0000","last_became_peered":"2024-03-06T07:21:54.417471+0000","last_unstale":"2024-03-06T07:21:55.410777+0000","last_undegraded":"2024-03-06T07:21:55.410777+0000","last_fullsized":"2024-03-06T07:21:55.410777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424717+0000","last_change":"2024-03-06T07:21:55.418688+0000","last_active":"2024-03-06T07:21:55.424717+0000","last_peered":"2024-03-06T07:21:55.424717+0000","last_clean":"2024-03-06T07:21:55.424717+0000","last_became_active":"2024-03-06T07:21:55.418392+0000","last_became_peered":"2024-03-06T07:21:55.418392+0000","last_unstale":"2024-03-06T07:21:55.424717+0000","last_undegraded":"2024-03-06T07:21:55.424717+0000","last_fullsized":"2024-03-06T07:21:55.424717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:43.203590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.410374+0000","last_change":"2024-03-06T07:21:54.414567+0000","last_active":"2024-03-06T07:21:55.410374+0000","last_peered":"2024-03-06T07:21:55.410374+0000","last_clean":"2024-03-06T07:21:55.410374+0000","last_became_active":"2024-03-06T07:21:54.414292+0000","last_became_peered":"2024-03-06T07:21:54.414292+0000","last_unstale":"2024-03-06T07:21:55.410374+0000","last_undegraded":"2024-03-06T07:21:55.410374+0000","last_fullsized":"2024-03-06T07:21:55.410374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.432798+0000","last_change":"2024-03-06T07:21:55.420297+0000","last_active":"2024-03-06T07:21:55.432798+0000","last_peered":"2024-03-06T07:21:55.432798+0000","last_clean":"2024-03-06T07:21:55.432798+0000","last_became_active":"2024-03-06T07:21:55.419994+0000","last_became_peered":"2024-03-06T07:21:55.419994+0000","last_unstale":"2024-03-06T07:21:55.432798+0000","last_undegraded":"2024-03-06T07:21:55.432798+0000","last_fullsized":"2024-03-06T07:21:55.432798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:41.897729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.416725+0000","last_change":"2024-03-06T07:21:54.420034+0000","last_active":"2024-03-06T07:21:55.416725+0000","last_peered":"2024-03-06T07:21:55.416725+0000","last_clean":"2024-03-06T07:21:55.416725+0000","last_became_active":"2024-03-06T07:21:54.419745+0000","last_became_peered":"2024-03-06T07:21:54.419745+0000","last_unstale":"2024-03-06T07:21:55.416725+0000","last_undegraded":"2024-03-06T07:21:55.416725+0000","last_fullsized":"2024-03-06T07:21:55.416725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435588+0000","last_change":"2024-03-06T07:21:55.425571+0000","last_active":"2024-03-06T07:21:55.435588+0000","last_peered":"2024-03-06T07:21:55.435588+0000","last_clean":"2024-03-06T07:21:55.435588+0000","last_became_active":"2024-03-06T07:21:55.424876+0000","last_became_peered":"2024-03-06T07:21:55.424876+0000","last_unstale":"2024-03-06T07:21:55.435588+0000","last_undegraded":"2024-03-06T07:21:55.435588+0000","last_fullsized":"2024-03-06T07:21:55.435588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.410180+0000","last_change":"2024-03-06T07:21:54.416884+0000","last_active":"2024-03-06T07:21:55.410180+0000","last_peered":"2024-03-06T07:21:55.410180+0000","last_clean":"2024-03-06T07:21:55.410180+0000","last_became_active":"2024-03-06T07:21:54.416576+0000","last_became_peered":"2024-03-06T07:21:54.416576+0000","last_unstale":"2024-03-06T07:21:55.410180+0000","last_undegraded":"2024-03-06T07:21:55.410180+0000","last_fullsized":"2024-03-06T07:21:55.410180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.417059+0000","last_change":"2024-03-06T07:21:55.414257+0000","last_active":"2024-03-06T07:21:55.417059+0000","last_peered":"2024-03-06T07:21:55.417059+0000","last_clean":"2024-03-06T07:21:55.417059+0000","last_became_active":"2024-03-06T07:21:55.413950+0000","last_became_peered":"2024-03-06T07:21:55.413950+0000","last_unstale":"2024-03-06T07:21:55.417059+0000","last_undegraded":"2024-03-06T07:21:55.417059+0000","last_fullsized":"2024-03-06T07:21:55.417059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:18.465323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440811+0000","last_change":"2024-03-06T07:21:54.412699+0000","last_active":"2024-03-06T07:21:55.440811+0000","last_peered":"2024-03-06T07:21:55.440811+0000","last_clean":"2024-03-06T07:21:55.440811+0000","last_became_active":"2024-03-06T07:21:54.412392+0000","last_became_peered":"2024-03-06T07:21:54.412392+0000","last_unstale":"2024-03-06T07:21:55.440811+0000","last_undegraded":"2024-03-06T07:21:55.440811+0000","last_fullsized":"2024-03-06T07:21:55.440811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.426012+0000","last_change":"2024-03-06T07:21:55.421734+0000","last_active":"2024-03-06T07:21:55.426012+0000","last_peered":"2024-03-06T07:21:55.426012+0000","last_clean":"2024-03-06T07:21:55.426012+0000","last_became_active":"2024-03-06T07:21:55.421451+0000","last_became_peered":"2024-03-06T07:21:55.421451+0000","last_unstale":"2024-03-06T07:21:55.426012+0000","last_undegraded":"2024-03-06T07:21:55.426012+0000","last_fullsized":"2024-03-06T07:21:55.426012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:16.284196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.443283+0000","last_change":"2024-03-06T07:21:54.416848+0000","last_active":"2024-03-06T07:21:55.443283+0000","last_peered":"2024-03-06T07:21:55.443283+0000","last_clean":"2024-03-06T07:21:55.443283+0000","last_became_active":"2024-03-06T07:21:54.416561+0000","last_became_peered":"2024-03-06T07:21:54.416561+0000","last_unstale":"2024-03-06T07:21:55.443283+0000","last_undegraded":"2024-03-06T07:21:55.443283+0000","last_fullsized":"2024-03-06T07:21:55.443283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.429565+0000","last_change":"2024-03-06T07:21:55.425748+0000","last_active":"2024-03-06T07:21:55.429565+0000","last_peered":"2024-03-06T07:21:55.429565+0000","last_clean":"2024-03-06T07:21:55.429565+0000","last_became_active":"2024-03-06T07:21:55.425501+0000","last_became_peered":"2024-03-06T07:21:55.425501+0000","last_unstale":"2024-03-06T07:21:55.429565+0000","last_undegraded":"2024-03-06T07:21:55.429565+0000","last_fullsized":"2024-03-06T07:21:55.429565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.441049+0000","last_change":"2024-03-06T07:21:54.419364+0000","last_active":"2024-03-06T07:21:55.441049+0000","last_peered":"2024-03-06T07:21:55.441049+0000","last_clean":"2024-03-06T07:21:55.441049+0000","last_became_active":"2024-03-06T07:21:54.419094+0000","last_became_peered":"2024-03-06T07:21:54.419094+0000","last_unstale":"2024-03-06T07:21:55.441049+0000","last_undegraded":"2024-03-06T07:21:55.441049+0000","last_fullsized":"2024-03-06T07:21:55.441049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435063+0000","last_change":"2024-03-06T07:21:55.421461+0000","last_active":"2024-03-06T07:21:55.435063+0000","last_peered":"2024-03-06T07:21:55.435063+0000","last_clean":"2024-03-06T07:21:55.435063+0000","last_became_active":"2024-03-06T07:21:55.421153+0000","last_became_peered":"2024-03-06T07:21:55.421153+0000","last_unstale":"2024-03-06T07:21:55.435063+0000","last_undegraded":"2024-03-06T07:21:55.435063+0000","last_fullsized":"2024-03-06T07:21:55.435063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:41.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.411734+0000","last_change":"2024-03-06T07:21:54.416121+0000","last_active":"2024-03-06T07:21:55.411734+0000","last_peered":"2024-03-06T07:21:55.411734+0000","last_clean":"2024-03-06T07:21:55.411734+0000","last_became_active":"2024-03-06T07:21:54.415857+0000","last_became_peered":"2024-03-06T07:21:54.415857+0000","last_unstale":"2024-03-06T07:21:55.411734+0000","last_undegraded":"2024-03-06T07:21:55.411734+0000","last_fullsized":"2024-03-06T07:21:55.411734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.431425+0000","last_change":"2024-03-06T07:21:55.428224+0000","last_active":"2024-03-06T07:21:55.431425+0000","last_peered":"2024-03-06T07:21:55.431425+0000","last_clean":"2024-03-06T07:21:55.431425+0000","last_became_active":"2024-03-06T07:21:55.427932+0000","last_became_peered":"2024-03-06T07:21:55.427932+0000","last_unstale":"2024-03-06T07:21:55.431425+0000","last_undegraded":"2024-03-06T07:21:55.431425+0000","last_fullsized":"2024-03-06T07:21:55.431425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.416675+0000","last_change":"2024-03-06T07:21:54.418770+0000","last_active":"2024-03-06T07:21:55.416675+0000","last_peered":"2024-03-06T07:21:55.416675+0000","last_clean":"2024-03-06T07:21:55.416675+0000","last_became_active":"2024-03-06T07:21:54.418498+0000","last_became_peered":"2024-03-06T07:21:54.418498+0000","last_unstale":"2024-03-06T07:21:55.416675+0000","last_undegraded":"2024-03-06T07:21:55.416675+0000","last_fullsized":"2024-03-06T07:21:55.416675+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440338+0000","last_change":"2024-03-06T07:21:55.436175+0000","last_active":"2024-03-06T07:21:55.440338+0000","last_peered":"2024-03-06T07:21:55.440338+0000","last_clean":"2024-03-06T07:21:55.440338+0000","last_became_active":"2024-03-06T07:21:55.435873+0000","last_became_peered":"2024-03-06T07:21:55.435873+0000","last_unstale":"2024-03-06T07:21:55.440338+0000","last_undegraded":"2024-03-06T07:21:55.440338+0000","last_fullsized":"2024-03-06T07:21:55.440338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:29.441742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.441358+0000","last_change":"2024-03-06T07:21:54.410321+0000","last_active":"2024-03-06T07:21:55.441358+0000","last_peered":"2024-03-06T07:21:55.441358+0000","last_clean":"2024-03-06T07:21:55.441358+0000","last_became_active":"2024-03-06T07:21:54.410019+0000","last_became_peered":"2024-03-06T07:21:54.410019+0000","last_unstale":"2024-03-06T07:21:55.441358+0000","last_undegraded":"2024-03-06T07:21:55.441358+0000","last_fullsized":"2024-03-06T07:21:55.441358+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.430732+0000","last_change":"2024-03-06T07:21:55.427171+0000","last_active":"2024-03-06T07:21:55.430732+0000","last_peered":"2024-03-06T07:21:55.430732+0000","last_clean":"2024-03-06T07:21:55.430732+0000","last_became_active":"2024-03-06T07:21:55.426872+0000","last_became_peered":"2024-03-06T07:21:55.426872+0000","last_unstale":"2024-03-06T07:21:55.430732+0000","last_undegraded":"2024-03-06T07:21:55.430732+0000","last_fullsized":"2024-03-06T07:21:55.430732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.441069+0000","last_change":"2024-03-06T07:21:54.419357+0000","last_active":"2024-03-06T07:21:55.441069+0000","last_peered":"2024-03-06T07:21:55.441069+0000","last_clean":"2024-03-06T07:21:55.441069+0000","last_became_active":"2024-03-06T07:21:54.419053+0000","last_became_peered":"2024-03-06T07:21:54.419053+0000","last_unstale":"2024-03-06T07:21:55.441069+0000","last_undegraded":"2024-03-06T07:21:55.441069+0000","last_fullsized":"2024-03-06T07:21:55.441069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:42.184203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.429597+0000","last_change":"2024-03-06T07:21:55.427511+0000","last_active":"2024-03-06T07:21:55.429597+0000","last_peered":"2024-03-06T07:21:55.429597+0000","last_clean":"2024-03-06T07:21:55.429597+0000","last_became_active":"2024-03-06T07:21:55.427175+0000","last_became_peered":"2024-03-06T07:21:55.427175+0000","last_unstale":"2024-03-06T07:21:55.429597+0000","last_undegraded":"2024-03-06T07:21:55.429597+0000","last_fullsized":"2024-03-06T07:21:55.429597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440027+0000","last_change":"2024-03-06T07:21:54.416325+0000","last_active":"2024-03-06T07:21:55.440027+0000","last_peered":"2024-03-06T07:21:55.440027+0000","last_clean":"2024-03-06T07:21:55.440027+0000","last_became_active":"2024-03-06T07:21:54.416017+0000","last_became_peered":"2024-03-06T07:21:54.416017+0000","last_unstale":"2024-03-06T07:21:55.440027+0000","last_undegraded":"2024-03-06T07:21:55.440027+0000","last_fullsized":"2024-03-06T07:21:55.440027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.431284+0000","last_change":"2024-03-06T07:21:55.428378+0000","last_active":"2024-03-06T07:21:55.431284+0000","last_peered":"2024-03-06T07:21:55.431284+0000","last_clean":"2024-03-06T07:21:55.431284+0000","last_became_active":"2024-03-06T07:21:55.426715+0000","last_became_peered":"2024-03-06T07:21:55.426715+0000","last_unstale":"2024-03-06T07:21:55.431284+0000","last_undegraded":"2024-03-06T07:21:55.431284+0000","last_fullsized":"2024-03-06T07:21:55.431284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440720+0000","last_change":"2024-03-06T07:21:54.406786+0000","last_active":"2024-03-06T07:21:55.440720+0000","last_peered":"2024-03-06T07:21:55.440720+0000","last_clean":"2024-03-06T07:21:55.440720+0000","last_became_active":"2024-03-06T07:21:54.406454+0000","last_became_peered":"2024-03-06T07:21:54.406454+0000","last_unstale":"2024-03-06T07:21:55.440720+0000","last_undegraded":"2024-03-06T07:21:55.440720+0000","last_fullsized":"2024-03-06T07:21:55.440720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:50.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435628+0000","last_change":"2024-03-06T07:21:55.422470+0000","last_active":"2024-03-06T07:21:55.435628+0000","last_peered":"2024-03-06T07:21:55.435628+0000","last_clean":"2024-03-06T07:21:55.435628+0000","last_became_active":"2024-03-06T07:21:55.420656+0000","last_became_peered":"2024-03-06T07:21:55.420656+0000","last_unstale":"2024-03-06T07:21:55.435628+0000","last_undegraded":"2024-03-06T07:21:55.435628+0000","last_fullsized":"2024-03-06T07:21:55.435628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:29.441742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.413292+0000","last_change":"2024-03-06T07:21:54.412695+0000","last_active":"2024-03-06T07:21:55.413292+0000","last_peered":"2024-03-06T07:21:55.413292+0000","last_clean":"2024-03-06T07:21:55.413292+0000","last_became_active":"2024-03-06T07:21:54.412391+0000","last_became_peered":"2024-03-06T07:21:54.412391+0000","last_unstale":"2024-03-06T07:21:55.413292+0000","last_undegraded":"2024-03-06T07:21:55.413292+0000","last_fullsized":"2024-03-06T07:21:55.413292+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:17.458904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440146+0000","last_change":"2024-03-06T07:21:55.436949+0000","last_active":"2024-03-06T07:21:55.440146+0000","last_peered":"2024-03-06T07:21:55.440146+0000","last_clean":"2024-03-06T07:21:55.440146+0000","last_became_active":"2024-03-06T07:21:55.436502+0000","last_became_peered":"2024-03-06T07:21:55.436502+0000","last_unstale":"2024-03-06T07:21:55.440146+0000","last_undegraded":"2024-03-06T07:21:55.440146+0000","last_fullsized":"2024-03-06T07:21:55.440146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:49.817183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.417227+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:21:55.417227+0000","last_peered":"2024-03-06T07:21:55.417227+0000","last_clean":"2024-03-06T07:21:55.417227+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:21:55.417227+0000","last_undegraded":"2024-03-06T07:21:55.417227+0000","last_fullsized":"2024-03-06T07:21:55.417227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.409149+0000","last_change":"2024-03-06T07:21:54.411371+0000","last_active":"2024-03-06T07:21:55.409149+0000","last_peered":"2024-03-06T07:21:55.409149+0000","last_clean":"2024-03-06T07:21:55.409149+0000","last_became_active":"2024-03-06T07:21:54.411095+0000","last_became_peered":"2024-03-06T07:21:54.411095+0000","last_unstale":"2024-03-06T07:21:55.409149+0000","last_undegraded":"2024-03-06T07:21:55.409149+0000","last_fullsized":"2024-03-06T07:21:55.409149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.427078+0000","last_change":"2024-03-06T07:21:55.423928+0000","last_active":"2024-03-06T07:21:55.427078+0000","last_peered":"2024-03-06T07:21:55.427078+0000","last_clean":"2024-03-06T07:21:55.427078+0000","last_became_active":"2024-03-06T07:21:55.423681+0000","last_became_peered":"2024-03-06T07:21:55.423681+0000","last_unstale":"2024-03-06T07:21:55.427078+0000","last_undegraded":"2024-03-06T07:21:55.427078+0000","last_fullsized":"2024-03-06T07:21:55.427078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:41.897729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.416633+0000","last_change":"2024-03-06T07:21:54.415381+0000","last_active":"2024-03-06T07:21:55.416633+0000","last_peered":"2024-03-06T07:21:55.416633+0000","last_clean":"2024-03-06T07:21:55.416633+0000","last_became_active":"2024-03-06T07:21:54.415099+0000","last_became_peered":"2024-03-06T07:21:54.415099+0000","last_unstale":"2024-03-06T07:21:55.416633+0000","last_undegraded":"2024-03-06T07:21:55.416633+0000","last_fullsized":"2024-03-06T07:21:55.416633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.891311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.434971+0000","last_change":"2024-03-06T07:21:55.423013+0000","last_active":"2024-03-06T07:21:55.434971+0000","last_peered":"2024-03-06T07:21:55.434971+0000","last_clean":"2024-03-06T07:21:55.434971+0000","last_became_active":"2024-03-06T07:21:55.422717+0000","last_became_peered":"2024-03-06T07:21:55.422717+0000","last_unstale":"2024-03-06T07:21:55.434971+0000","last_undegraded":"2024-03-06T07:21:55.434971+0000","last_fullsized":"2024-03-06T07:21:55.434971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.414367+0000","last_change":"2024-03-06T07:21:54.416728+0000","last_active":"2024-03-06T07:21:55.414367+0000","last_peered":"2024-03-06T07:21:55.414367+0000","last_clean":"2024-03-06T07:21:55.414367+0000","last_became_active":"2024-03-06T07:21:54.416226+0000","last_became_peered":"2024-03-06T07:21:54.416226+0000","last_unstale":"2024-03-06T07:21:55.414367+0000","last_undegraded":"2024-03-06T07:21:55.414367+0000","last_fullsized":"2024-03-06T07:21:55.414367+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.437395+0000","last_change":"2024-03-06T07:21:55.429746+0000","last_active":"2024-03-06T07:21:55.437395+0000","last_peered":"2024-03-06T07:21:55.437395+0000","last_clean":"2024-03-06T07:21:55.437395+0000","last_became_active":"2024-03-06T07:21:55.429320+0000","last_became_peered":"2024-03-06T07:21:55.429320+0000","last_unstale":"2024-03-06T07:21:55.437395+0000","last_undegraded":"2024-03-06T07:21:55.437395+0000","last_fullsized":"2024-03-06T07:21:55.437395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.411567+0000","last_change":"2024-03-06T07:21:54.413058+0000","last_active":"2024-03-06T07:21:55.411567+0000","last_peered":"2024-03-06T07:21:55.411567+0000","last_clean":"2024-03-06T07:21:55.411567+0000","last_became_active":"2024-03-06T07:21:54.412469+0000","last_became_peered":"2024-03-06T07:21:54.412469+0000","last_unstale":"2024-03-06T07:21:55.411567+0000","last_undegraded":"2024-03-06T07:21:55.411567+0000","last_fullsized":"2024-03-06T07:21:55.411567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.431559+0000","last_change":"2024-03-06T07:21:55.425788+0000","last_active":"2024-03-06T07:21:55.431559+0000","last_peered":"2024-03-06T07:21:55.431559+0000","last_clean":"2024-03-06T07:21:55.431559+0000","last_became_active":"2024-03-06T07:21:55.425440+0000","last_became_peered":"2024-03-06T07:21:55.425440+0000","last_unstale":"2024-03-06T07:21:55.431559+0000","last_undegraded":"2024-03-06T07:21:55.431559+0000","last_fullsized":"2024-03-06T07:21:55.431559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:06.592646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.415776+0000","last_change":"2024-03-06T07:21:54.415739+0000","last_active":"2024-03-06T07:21:55.415776+0000","last_peered":"2024-03-06T07:21:55.415776+0000","last_clean":"2024-03-06T07:21:55.415776+0000","last_became_active":"2024-03-06T07:21:54.415452+0000","last_became_peered":"2024-03-06T07:21:54.415452+0000","last_unstale":"2024-03-06T07:21:55.415776+0000","last_undegraded":"2024-03-06T07:21:55.415776+0000","last_fullsized":"2024-03-06T07:21:55.415776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435784+0000","last_change":"2024-03-06T07:21:55.419140+0000","last_active":"2024-03-06T07:21:55.435784+0000","last_peered":"2024-03-06T07:21:55.435784+0000","last_clean":"2024-03-06T07:21:55.435784+0000","last_became_active":"2024-03-06T07:21:55.415992+0000","last_became_peered":"2024-03-06T07:21:55.415992+0000","last_unstale":"2024-03-06T07:21:55.435784+0000","last_undegraded":"2024-03-06T07:21:55.435784+0000","last_fullsized":"2024-03-06T07:21:55.435784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:41.897729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.416094+0000","last_change":"2024-03-06T07:21:54.417366+0000","last_active":"2024-03-06T07:21:55.416094+0000","last_peered":"2024-03-06T07:21:55.416094+0000","last_clean":"2024-03-06T07:21:55.416094+0000","last_became_active":"2024-03-06T07:21:54.416684+0000","last_became_peered":"2024-03-06T07:21:54.416684+0000","last_unstale":"2024-03-06T07:21:55.416094+0000","last_undegraded":"2024-03-06T07:21:55.416094+0000","last_fullsized":"2024-03-06T07:21:55.416094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:44.263686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.432854+0000","last_change":"2024-03-06T07:21:55.419798+0000","last_active":"2024-03-06T07:21:55.432854+0000","last_peered":"2024-03-06T07:21:55.432854+0000","last_clean":"2024-03-06T07:21:55.432854+0000","last_became_active":"2024-03-06T07:21:55.419490+0000","last_became_peered":"2024-03-06T07:21:55.419490+0000","last_unstale":"2024-03-06T07:21:55.432854+0000","last_undegraded":"2024-03-06T07:21:55.432854+0000","last_fullsized":"2024-03-06T07:21:55.432854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.416114+0000","last_change":"2024-03-06T07:21:54.418937+0000","last_active":"2024-03-06T07:21:55.416114+0000","last_peered":"2024-03-06T07:21:55.416114+0000","last_clean":"2024-03-06T07:21:55.416114+0000","last_became_active":"2024-03-06T07:21:54.418592+0000","last_became_peered":"2024-03-06T07:21:54.418592+0000","last_unstale":"2024-03-06T07:21:55.416114+0000","last_undegraded":"2024-03-06T07:21:55.416114+0000","last_fullsized":"2024-03-06T07:21:55.416114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:17.773963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424739+0000","last_change":"2024-03-06T07:21:55.420527+0000","last_active":"2024-03-06T07:21:55.424739+0000","last_peered":"2024-03-06T07:21:55.424739+0000","last_clean":"2024-03-06T07:21:55.424739+0000","last_became_active":"2024-03-06T07:21:55.420210+0000","last_became_peered":"2024-03-06T07:21:55.420210+0000","last_unstale":"2024-03-06T07:21:55.424739+0000","last_undegraded":"2024-03-06T07:21:55.424739+0000","last_fullsized":"2024-03-06T07:21:55.424739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:41.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.437818+0000","last_change":"2024-03-06T07:21:54.420765+0000","last_active":"2024-03-06T07:21:55.437818+0000","last_peered":"2024-03-06T07:21:55.437818+0000","last_clean":"2024-03-06T07:21:55.437818+0000","last_became_active":"2024-03-06T07:21:54.420541+0000","last_became_peered":"2024-03-06T07:21:54.420541+0000","last_unstale":"2024-03-06T07:21:55.437818+0000","last_undegraded":"2024-03-06T07:21:55.437818+0000","last_fullsized":"2024-03-06T07:21:55.437818+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.437992+0000","last_change":"2024-03-06T07:21:55.430019+0000","last_active":"2024-03-06T07:21:55.437992+0000","last_peered":"2024-03-06T07:21:55.437992+0000","last_clean":"2024-03-06T07:21:55.437992+0000","last_became_active":"2024-03-06T07:21:55.429671+0000","last_became_peered":"2024-03-06T07:21:55.429671+0000","last_unstale":"2024-03-06T07:21:55.437992+0000","last_undegraded":"2024-03-06T07:21:55.437992+0000","last_fullsized":"2024-03-06T07:21:55.437992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:41.897729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.417615+0000","last_change":"2024-03-06T07:21:54.416295+0000","last_active":"2024-03-06T07:21:55.417615+0000","last_peered":"2024-03-06T07:21:55.417615+0000","last_clean":"2024-03-06T07:21:55.417615+0000","last_became_active":"2024-03-06T07:21:54.415817+0000","last_became_peered":"2024-03-06T07:21:54.415817+0000","last_unstale":"2024-03-06T07:21:55.417615+0000","last_undegraded":"2024-03-06T07:21:55.417615+0000","last_fullsized":"2024-03-06T07:21:55.417615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:28.435323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.426155+0000","last_change":"2024-03-06T07:21:55.422908+0000","last_active":"2024-03-06T07:21:55.426155+0000","last_peered":"2024-03-06T07:21:55.426155+0000","last_clean":"2024-03-06T07:21:55.426155+0000","last_became_active":"2024-03-06T07:21:55.422624+0000","last_became_peered":"2024-03-06T07:21:55.422624+0000","last_unstale":"2024-03-06T07:21:55.426155+0000","last_undegraded":"2024-03-06T07:21:55.426155+0000","last_fullsized":"2024-03-06T07:21:55.426155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:37.109379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.415812+0000","last_change":"2024-03-06T07:21:54.411585+0000","last_active":"2024-03-06T07:21:55.415812+0000","last_peered":"2024-03-06T07:21:55.415812+0000","last_clean":"2024-03-06T07:21:55.415812+0000","last_became_active":"2024-03-06T07:21:54.411318+0000","last_became_peered":"2024-03-06T07:21:54.411318+0000","last_unstale":"2024-03-06T07:21:55.415812+0000","last_undegraded":"2024-03-06T07:21:55.415812+0000","last_fullsized":"2024-03-06T07:21:55.415812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424780+0000","last_change":"2024-03-06T07:21:55.421416+0000","last_active":"2024-03-06T07:21:55.424780+0000","last_peered":"2024-03-06T07:21:55.424780+0000","last_clean":"2024-03-06T07:21:55.424780+0000","last_became_active":"2024-03-06T07:21:55.421113+0000","last_became_peered":"2024-03-06T07:21:55.421113+0000","last_unstale":"2024-03-06T07:21:55.424780+0000","last_undegraded":"2024-03-06T07:21:55.424780+0000","last_fullsized":"2024-03-06T07:21:55.424780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:36.844180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.415626+0000","last_change":"2024-03-06T07:21:54.416417+0000","last_active":"2024-03-06T07:21:55.415626+0000","last_peered":"2024-03-06T07:21:55.415626+0000","last_clean":"2024-03-06T07:21:55.415626+0000","last_became_active":"2024-03-06T07:21:54.416108+0000","last_became_peered":"2024-03-06T07:21:54.416108+0000","last_unstale":"2024-03-06T07:21:55.415626+0000","last_undegraded":"2024-03-06T07:21:55.415626+0000","last_fullsized":"2024-03-06T07:21:55.415626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424681+0000","last_change":"2024-03-06T07:21:55.418459+0000","last_active":"2024-03-06T07:21:55.424681+0000","last_peered":"2024-03-06T07:21:55.424681+0000","last_clean":"2024-03-06T07:21:55.424681+0000","last_became_active":"2024-03-06T07:21:55.418179+0000","last_became_peered":"2024-03-06T07:21:55.418179+0000","last_unstale":"2024-03-06T07:21:55.424681+0000","last_undegraded":"2024-03-06T07:21:55.424681+0000","last_fullsized":"2024-03-06T07:21:55.424681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:39.492542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440606+0000","last_change":"2024-03-06T07:21:54.412875+0000","last_active":"2024-03-06T07:21:55.440606+0000","last_peered":"2024-03-06T07:21:55.440606+0000","last_clean":"2024-03-06T07:21:55.440606+0000","last_became_active":"2024-03-06T07:21:54.412617+0000","last_became_peered":"2024-03-06T07:21:54.412617+0000","last_unstale":"2024-03-06T07:21:55.440606+0000","last_undegraded":"2024-03-06T07:21:55.440606+0000","last_fullsized":"2024-03-06T07:21:55.440606+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:50.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.434473+0000","last_change":"2024-03-06T07:21:55.427872+0000","last_active":"2024-03-06T07:21:55.434473+0000","last_peered":"2024-03-06T07:21:55.434473+0000","last_clean":"2024-03-06T07:21:55.434473+0000","last_became_active":"2024-03-06T07:21:55.427504+0000","last_became_peered":"2024-03-06T07:21:55.427504+0000","last_unstale":"2024-03-06T07:21:55.434473+0000","last_undegraded":"2024-03-06T07:21:55.434473+0000","last_fullsized":"2024-03-06T07:21:55.434473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435812+0000","last_change":"2024-03-06T07:21:55.427757+0000","last_active":"2024-03-06T07:21:55.435812+0000","last_peered":"2024-03-06T07:21:55.435812+0000","last_clean":"2024-03-06T07:21:55.435812+0000","last_became_active":"2024-03-06T07:21:55.427471+0000","last_became_peered":"2024-03-06T07:21:55.427471+0000","last_unstale":"2024-03-06T07:21:55.435812+0000","last_undegraded":"2024-03-06T07:21:55.435812+0000","last_fullsized":"2024-03-06T07:21:55.435812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440945+0000","last_change":"2024-03-06T07:21:54.414035+0000","last_active":"2024-03-06T07:21:55.440945+0000","last_peered":"2024-03-06T07:21:55.440945+0000","last_clean":"2024-03-06T07:21:55.440945+0000","last_became_active":"2024-03-06T07:21:54.413618+0000","last_became_peered":"2024-03-06T07:21:54.413618+0000","last_unstale":"2024-03-06T07:21:55.440945+0000","last_undegraded":"2024-03-06T07:21:55.440945+0000","last_fullsized":"2024-03-06T07:21:55.440945+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435670+0000","last_change":"2024-03-06T07:21:55.423540+0000","last_active":"2024-03-06T07:21:55.435670+0000","last_peered":"2024-03-06T07:21:55.435670+0000","last_clean":"2024-03-06T07:21:55.435670+0000","last_became_active":"2024-03-06T07:21:55.423197+0000","last_became_peered":"2024-03-06T07:21:55.423197+0000","last_unstale":"2024-03-06T07:21:55.435670+0000","last_undegraded":"2024-03-06T07:21:55.435670+0000","last_fullsized":"2024-03-06T07:21:55.435670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440274+0000","last_change":"2024-03-06T07:21:54.417365+0000","last_active":"2024-03-06T07:21:55.440274+0000","last_peered":"2024-03-06T07:21:55.440274+0000","last_clean":"2024-03-06T07:21:55.440274+0000","last_became_active":"2024-03-06T07:21:54.417072+0000","last_became_peered":"2024-03-06T07:21:54.417072+0000","last_unstale":"2024-03-06T07:21:55.440274+0000","last_undegraded":"2024-03-06T07:21:55.440274+0000","last_fullsized":"2024-03-06T07:21:55.440274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:17.773963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435089+0000","last_change":"2024-03-06T07:21:55.424176+0000","last_active":"2024-03-06T07:21:55.435089+0000","last_peered":"2024-03-06T07:21:55.435089+0000","last_clean":"2024-03-06T07:21:55.435089+0000","last_became_active":"2024-03-06T07:21:55.423848+0000","last_became_peered":"2024-03-06T07:21:55.423848+0000","last_unstale":"2024-03-06T07:21:55.435089+0000","last_undegraded":"2024-03-06T07:21:55.435089+0000","last_fullsized":"2024-03-06T07:21:55.435089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.438215+0000","last_change":"2024-03-06T07:21:54.418869+0000","last_active":"2024-03-06T07:21:55.438215+0000","last_peered":"2024-03-06T07:21:55.438215+0000","last_clean":"2024-03-06T07:21:55.438215+0000","last_became_active":"2024-03-06T07:21:54.418582+0000","last_became_peered":"2024-03-06T07:21:54.418582+0000","last_unstale":"2024-03-06T07:21:55.438215+0000","last_undegraded":"2024-03-06T07:21:55.438215+0000","last_fullsized":"2024-03-06T07:21:55.438215+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.434556+0000","last_change":"2024-03-06T07:21:55.427658+0000","last_active":"2024-03-06T07:21:55.434556+0000","last_peered":"2024-03-06T07:21:55.434556+0000","last_clean":"2024-03-06T07:21:55.434556+0000","last_became_active":"2024-03-06T07:21:55.427370+0000","last_became_peered":"2024-03-06T07:21:55.427370+0000","last_unstale":"2024-03-06T07:21:55.434556+0000","last_undegraded":"2024-03-06T07:21:55.434556+0000","last_fullsized":"2024-03-06T07:21:55.434556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.438249+0000","last_change":"2024-03-06T07:21:54.417435+0000","last_active":"2024-03-06T07:21:55.438249+0000","last_peered":"2024-03-06T07:21:55.438249+0000","last_clean":"2024-03-06T07:21:55.438249+0000","last_became_active":"2024-03-06T07:21:54.417136+0000","last_became_peered":"2024-03-06T07:21:54.417136+0000","last_unstale":"2024-03-06T07:21:55.438249+0000","last_undegraded":"2024-03-06T07:21:55.438249+0000","last_fullsized":"2024-03-06T07:21:55.438249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:15.277777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":655584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":64760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":64794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":655640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:58.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:58.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.559+0000 7f2f041e9640 1 Processor -- start 2024-03-06T07:21:58.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.559+0000 7f2f041e9640 1 -- start start 2024-03-06T07:21:58.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f041e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2efc15dfe0 0x7f2efc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f041e9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2efc0cf830 0x7f2efc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f041e9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2efc0b3430 con 0x7f2efc0d27f0 2024-03-06T07:21:58.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f041e9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2efc0d2c50 con 0x7f2efc0cf830 2024-03-06T07:21:58.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f041e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0b31e0 con 0x7f2efc15dfe0 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f01784640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2efc0cf830 0x7f2efc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f01784640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2efc0cf830 0x7f2efc0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57964/0 (socket says 172.21.15.59:57964) 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f02786640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2efc0d27f0 0x7f2efc0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51126/0 (socket says 172.21.15.59:51126) 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f01784640 1 -- 172.21.15.59:0/1389200008 learned_addr learned my addr 172.21.15.59:0/1389200008 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f01f85640 1 --2- 172.21.15.59:0/1389200008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2efc15dfe0 0x7f2efc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f01784640 1 -- 172.21.15.59:0/1389200008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2efc15dfe0 msgr2=0x7f2efc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f01784640 1 --2- 172.21.15.59:0/1389200008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2efc15dfe0 0x7f2efc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f01784640 1 -- 172.21.15.59:0/1389200008 >> v1:172.21.15.59:6789/0 conn(0x7f2efc0d27f0 legacy=0x7f2efc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f00f83640 1 -- 172.21.15.59:0/1389200008 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958936198 0 0) 0x7f2efc0b3430 con 0x7f2efc0d27f0 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f01784640 1 -- 172.21.15.59:0/1389200008 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc0b3710 con 0x7f2efc0cf830 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.560+0000 7f2f01f85640 1 --2- 172.21.15.59:0/1389200008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2efc15dfe0 0x7f2efc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:58.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.561+0000 7f2f01784640 1 --2- 172.21.15.59:0/1389200008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2efc0cf830 0x7f2efc0d22b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009870 tx=0x7f2eec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a1aced2a3cfc0f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.561+0000 7f2f00f83640 1 -- 172.21.15.59:0/1389200008 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec034070 con 0x7f2efc0cf830 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.561+0000 7f2f00f83640 1 -- 172.21.15.59:0/1389200008 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec030840 con 0x7f2efc0cf830 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.561+0000 7f2f00f83640 1 -- 172.21.15.59:0/1389200008 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec030c10 con 0x7f2efc0cf830 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.561+0000 7f2f041e9640 1 -- 172.21.15.59:0/1389200008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2efc0cf830 msgr2=0x7f2efc0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.561+0000 7f2f041e9640 1 --2- 172.21.15.59:0/1389200008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2efc0cf830 0x7f2efc0d22b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009870 tx=0x7f2eec02b980 comp rx=0 tx=0).stop 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.561+0000 7f2f041e9640 1 -- 172.21.15.59:0/1389200008 shutdown_connections 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.561+0000 7f2f041e9640 1 --2- 172.21.15.59:0/1389200008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2efc0cf830 0x7f2efc0d22b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.561+0000 7f2f041e9640 1 --2- 172.21.15.59:0/1389200008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2efc15dfe0 0x7f2efc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.562+0000 7f2f041e9640 1 -- 172.21.15.59:0/1389200008 >> 172.21.15.59:0/1389200008 conn(0x7f2efc0d3cd0 msgr2=0x7f2efc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.562+0000 7f2f041e9640 1 -- 172.21.15.59:0/1389200008 shutdown_connections 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.562+0000 7f2f041e9640 1 -- 172.21.15.59:0/1389200008 wait complete. 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.563+0000 7f2f041e9640 1 Processor -- start 2024-03-06T07:21:58.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.563+0000 7f2f041e9640 1 -- start start 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.563+0000 7f2f041e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2efc0cf830 0x7f2efc161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.563+0000 7f2f041e9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2efc0d27f0 0x7f2efc15e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.563+0000 7f2f041e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2efc15dfe0 0x7f2efc15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.563+0000 7f2f041e9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0d1c70 con 0x7f2efc0cf830 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.563+0000 7f2f041e9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0b3d60 con 0x7f2efc0d27f0 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.563+0000 7f2f041e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0d1df0 con 0x7f2efc15dfe0 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f01f85640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2efc0cf830 0x7f2efc161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f01f85640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2efc0cf830 0x7f2efc161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57968/0 (socket says 172.21.15.59:57968) 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f01f85640 1 -- 172.21.15.59:0/797504077 learned_addr learned my addr 172.21.15.59:0/797504077 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f02786640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2efc15dfe0 0x7f2efc15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f01784640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2efc0d27f0 0x7f2efc15e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f01f85640 1 -- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2efc15dfe0 msgr2=0x7f2efc15ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f01f85640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2efc15dfe0 0x7f2efc15ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f01f85640 1 -- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2efc0d27f0 msgr2=0x7f2efc15e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f01f85640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2efc0d27f0 0x7f2efc15e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f01f85640 1 -- 172.21.15.59:0/797504077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc0b6c90 con 0x7f2efc0cf830 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f01f85640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2efc0cf830 0x7f2efc161180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800e960 tx=0x7f2ef800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2ef27fc640 1 -- 172.21.15.59:0/797504077 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8017070 con 0x7f2efc0cf830 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2ef27fc640 1 -- 172.21.15.59:0/797504077 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef800ca00 con 0x7f2efc0cf830 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f041e9640 1 -- 172.21.15.59:0/797504077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2efc0cc360 con 0x7f2efc0cf830 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2ef27fc640 1 -- 172.21.15.59:0/797504077 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef800cdd0 con 0x7f2efc0cf830 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.564+0000 7f2f041e9640 1 -- 172.21.15.59:0/797504077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2efc0cc920 con 0x7f2efc0cf830 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.565+0000 7f2f041e9640 1 -- 172.21.15.59:0/797504077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2efc04f330 con 0x7f2efc0cf830 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.565+0000 7f2ef27fc640 1 -- 172.21.15.59:0/797504077 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ef801a070 con 0x7f2efc0cf830 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.566+0000 7f2ef27fc640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ee006b4d0 0x7f2ee006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.566+0000 7f2ef27fc640 1 -- 172.21.15.59:0/797504077 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ef808a3c0 con 0x7f2efc0cf830 2024-03-06T07:21:58.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.566+0000 7f2f01784640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ee006b4d0 0x7f2ee006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.567+0000 7f2f01784640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ee006b4d0 0x7f2ee006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2eec036000 tx=0x7f2eec008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.570+0000 7f2ef27fc640 1 -- 172.21.15.59:0/797504077 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ef8058bf0 con 0x7f2efc0cf830 2024-03-06T07:21:58.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.747+0000 7f2f041e9640 1 -- 172.21.15.59:0/797504077 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2efc0cd020 con 0x7f2ee006b4d0 2024-03-06T07:21:58.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.754+0000 7f2ef27fc640 1 -- 172.21.15.59:0/797504077 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251878 (secure 0 0 0) 0x7f2efc0cd020 con 0x7f2ee006b4d0 2024-03-06T07:21:58.758 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:58.759 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:21:58.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 -- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ee006b4d0 msgr2=0x7f2ee006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ee006b4d0 0x7f2ee006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2eec036000 tx=0x7f2eec008030 comp rx=0 tx=0).stop 2024-03-06T07:21:58.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 -- 172.21.15.59:0/797504077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2efc0cf830 msgr2=0x7f2efc161180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2efc0cf830 0x7f2efc161180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800e960 tx=0x7f2ef800ee20 comp rx=0 tx=0).stop 2024-03-06T07:21:58.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 -- 172.21.15.59:0/797504077 shutdown_connections 2024-03-06T07:21:58.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ee006b4d0 0x7f2ee006d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2efc15dfe0 0x7f2efc15ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2efc0d27f0 0x7f2efc15e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 --2- 172.21.15.59:0/797504077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2efc0cf830 0x7f2efc161180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 -- 172.21.15.59:0/797504077 >> 172.21.15.59:0/797504077 conn(0x7f2efc0d3cd0 msgr2=0x7f2efc06bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 -- 172.21.15.59:0/797504077 shutdown_connections 2024-03-06T07:21:58.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.757+0000 7f2f041e9640 1 -- 172.21.15.59:0/797504077 wait complete. 2024-03-06T07:21:58.770 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-06T07:21:57.303056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":99,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218904,"kb_used_data":2728,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755816,"statfs":{"total":773094113280,"available":772869955584,"internally_reserved":0,"allocated":2793472,"data_stored":1702628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000792"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.431306+0000","last_change":"2024-03-06T07:21:55.429371+0000","last_active":"2024-03-06T07:21:55.431306+0000","last_peered":"2024-03-06T07:21:55.431306+0000","last_clean":"2024-03-06T07:21:55.431306+0000","last_became_active":"2024-03-06T07:21:55.429057+0000","last_became_peered":"2024-03-06T07:21:55.429057+0000","last_unstale":"2024-03-06T07:21:55.431306+0000","last_undegraded":"2024-03-06T07:21:55.431306+0000","last_fullsized":"2024-03-06T07:21:55.431306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.432159+0000","last_change":"2024-03-06T07:21:55.420380+0000","last_active":"2024-03-06T07:21:55.432159+0000","last_peered":"2024-03-06T07:21:55.432159+0000","last_clean":"2024-03-06T07:21:55.432159+0000","last_became_active":"2024-03-06T07:21:55.419640+0000","last_became_peered":"2024-03-06T07:21:55.419640+0000","last_unstale":"2024-03-06T07:21:55.432159+0000","last_undegraded":"2024-03-06T07:21:55.432159+0000","last_fullsized":"2024-03-06T07:21:55.432159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:29.441742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.429024+0000","last_change":"2024-03-06T07:21:55.418047+0000","last_active":"2024-03-06T07:21:55.429024+0000","last_peered":"2024-03-06T07:21:55.429024+0000","last_clean":"2024-03-06T07:21:55.429024+0000","last_became_active":"2024-03-06T07:21:55.417536+0000","last_became_peered":"2024-03-06T07:21:55.417536+0000","last_unstale":"2024-03-06T07:21:55.429024+0000","last_undegraded":"2024-03-06T07:21:55.429024+0000","last_fullsized":"2024-03-06T07:21:55.429024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:39.492542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435650+0000","last_change":"2024-03-06T07:21:55.422738+0000","last_active":"2024-03-06T07:21:55.435650+0000","last_peered":"2024-03-06T07:21:55.435650+0000","last_clean":"2024-03-06T07:21:55.435650+0000","last_became_active":"2024-03-06T07:21:55.422460+0000","last_became_peered":"2024-03-06T07:21:55.422460+0000","last_unstale":"2024-03-06T07:21:55.435650+0000","last_undegraded":"2024-03-06T07:21:55.435650+0000","last_fullsized":"2024-03-06T07:21:55.435650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.427805+0000","last_change":"2024-03-06T07:21:55.423628+0000","last_active":"2024-03-06T07:21:55.427805+0000","last_peered":"2024-03-06T07:21:55.427805+0000","last_clean":"2024-03-06T07:21:55.427805+0000","last_became_active":"2024-03-06T07:21:55.423360+0000","last_became_peered":"2024-03-06T07:21:55.423360+0000","last_unstale":"2024-03-06T07:21:55.427805+0000","last_undegraded":"2024-03-06T07:21:55.427805+0000","last_fullsized":"2024-03-06T07:21:55.427805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435728+0000","last_change":"2024-03-06T07:21:55.423187+0000","last_active":"2024-03-06T07:21:55.435728+0000","last_peered":"2024-03-06T07:21:55.435728+0000","last_clean":"2024-03-06T07:21:55.435728+0000","last_became_active":"2024-03-06T07:21:55.422907+0000","last_became_peered":"2024-03-06T07:21:55.422907+0000","last_unstale":"2024-03-06T07:21:55.435728+0000","last_undegraded":"2024-03-06T07:21:55.435728+0000","last_fullsized":"2024-03-06T07:21:55.435728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.422051+0000","last_change":"2024-03-06T07:21:55.419720+0000","last_active":"2024-03-06T07:21:55.422051+0000","last_peered":"2024-03-06T07:21:55.422051+0000","last_clean":"2024-03-06T07:21:55.422051+0000","last_became_active":"2024-03-06T07:21:55.419294+0000","last_became_peered":"2024-03-06T07:21:55.419294+0000","last_unstale":"2024-03-06T07:21:55.422051+0000","last_undegraded":"2024-03-06T07:21:55.422051+0000","last_fullsized":"2024-03-06T07:21:55.422051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.427044+0000","last_change":"2024-03-06T07:21:55.423177+0000","last_active":"2024-03-06T07:21:55.427044+0000","last_peered":"2024-03-06T07:21:55.427044+0000","last_clean":"2024-03-06T07:21:55.427044+0000","last_became_active":"2024-03-06T07:21:55.422875+0000","last_became_peered":"2024-03-06T07:21:55.422875+0000","last_unstale":"2024-03-06T07:21:55.427044+0000","last_undegraded":"2024-03-06T07:21:55.427044+0000","last_fullsized":"2024-03-06T07:21:55.427044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440175+0000","last_change":"2024-03-06T07:21:55.436966+0000","last_active":"2024-03-06T07:21:55.440175+0000","last_peered":"2024-03-06T07:21:55.440175+0000","last_clean":"2024-03-06T07:21:55.440175+0000","last_became_active":"2024-03-06T07:21:55.436375+0000","last_became_peered":"2024-03-06T07:21:55.436375+0000","last_unstale":"2024-03-06T07:21:55.440175+0000","last_undegraded":"2024-03-06T07:21:55.440175+0000","last_fullsized":"2024-03-06T07:21:55.440175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:41.154073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440431+0000","last_change":"2024-03-06T07:21:55.436554+0000","last_active":"2024-03-06T07:21:55.440431+0000","last_peered":"2024-03-06T07:21:55.440431+0000","last_clean":"2024-03-06T07:21:55.440431+0000","last_became_active":"2024-03-06T07:21:55.436292+0000","last_became_peered":"2024-03-06T07:21:55.436292+0000","last_unstale":"2024-03-06T07:21:55.440431+0000","last_undegraded":"2024-03-06T07:21:55.440431+0000","last_fullsized":"2024-03-06T07:21:55.440431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435647+0000","last_change":"2024-03-06T07:21:55.432397+0000","last_active":"2024-03-06T07:21:55.435647+0000","last_peered":"2024-03-06T07:21:55.435647+0000","last_clean":"2024-03-06T07:21:55.435647+0000","last_became_active":"2024-03-06T07:21:55.432090+0000","last_became_peered":"2024-03-06T07:21:55.432090+0000","last_unstale":"2024-03-06T07:21:55.435647+0000","last_undegraded":"2024-03-06T07:21:55.435647+0000","last_fullsized":"2024-03-06T07:21:55.435647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:41.897729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.426033+0000","last_change":"2024-03-06T07:21:55.423196+0000","last_active":"2024-03-06T07:21:55.426033+0000","last_peered":"2024-03-06T07:21:55.426033+0000","last_clean":"2024-03-06T07:21:55.426033+0000","last_became_active":"2024-03-06T07:21:55.422079+0000","last_became_peered":"2024-03-06T07:21:55.422079+0000","last_unstale":"2024-03-06T07:21:55.426033+0000","last_undegraded":"2024-03-06T07:21:55.426033+0000","last_fullsized":"2024-03-06T07:21:55.426033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:18.465323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424815+0000","last_change":"2024-03-06T07:21:55.421461+0000","last_active":"2024-03-06T07:21:55.424815+0000","last_peered":"2024-03-06T07:21:55.424815+0000","last_clean":"2024-03-06T07:21:55.424815+0000","last_became_active":"2024-03-06T07:21:55.420795+0000","last_became_peered":"2024-03-06T07:21:55.420795+0000","last_unstale":"2024-03-06T07:21:55.424815+0000","last_undegraded":"2024-03-06T07:21:55.424815+0000","last_fullsized":"2024-03-06T07:21:55.424815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435040+0000","last_change":"2024-03-06T07:21:55.420082+0000","last_active":"2024-03-06T07:21:55.435040+0000","last_peered":"2024-03-06T07:21:55.435040+0000","last_clean":"2024-03-06T07:21:55.435040+0000","last_became_active":"2024-03-06T07:21:55.419844+0000","last_became_peered":"2024-03-06T07:21:55.419844+0000","last_unstale":"2024-03-06T07:21:55.435040+0000","last_undegraded":"2024-03-06T07:21:55.435040+0000","last_fullsized":"2024-03-06T07:21:55.435040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:43.203590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.432820+0000","last_change":"2024-03-06T07:21:55.425776+0000","last_active":"2024-03-06T07:21:55.432820+0000","last_peered":"2024-03-06T07:21:55.432820+0000","last_clean":"2024-03-06T07:21:55.432820+0000","last_became_active":"2024-03-06T07:21:55.425479+0000","last_became_peered":"2024-03-06T07:21:55.425479+0000","last_unstale":"2024-03-06T07:21:55.432820+0000","last_undegraded":"2024-03-06T07:21:55.432820+0000","last_fullsized":"2024-03-06T07:21:55.432820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.430753+0000","last_change":"2024-03-06T07:21:55.416970+0000","last_active":"2024-03-06T07:21:55.430753+0000","last_peered":"2024-03-06T07:21:55.430753+0000","last_clean":"2024-03-06T07:21:55.430753+0000","last_became_active":"2024-03-06T07:21:55.416495+0000","last_became_peered":"2024-03-06T07:21:55.416495+0000","last_unstale":"2024-03-06T07:21:55.430753+0000","last_undegraded":"2024-03-06T07:21:55.430753+0000","last_fullsized":"2024-03-06T07:21:55.430753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.439989+0000","last_change":"2024-03-06T07:21:55.430721+0000","last_active":"2024-03-06T07:21:55.439989+0000","last_peered":"2024-03-06T07:21:55.439989+0000","last_clean":"2024-03-06T07:21:55.439989+0000","last_became_active":"2024-03-06T07:21:55.428901+0000","last_became_peered":"2024-03-06T07:21:55.428901+0000","last_unstale":"2024-03-06T07:21:55.439989+0000","last_undegraded":"2024-03-06T07:21:55.439989+0000","last_fullsized":"2024-03-06T07:21:55.439989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.432206+0000","last_change":"2024-03-06T07:21:55.428378+0000","last_active":"2024-03-06T07:21:55.432206+0000","last_peered":"2024-03-06T07:21:55.432206+0000","last_clean":"2024-03-06T07:21:55.432206+0000","last_became_active":"2024-03-06T07:21:55.428101+0000","last_became_peered":"2024-03-06T07:21:55.428101+0000","last_unstale":"2024-03-06T07:21:55.432206+0000","last_undegraded":"2024-03-06T07:21:55.432206+0000","last_fullsized":"2024-03-06T07:21:55.432206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:43.203590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.428760+0000","last_change":"2024-03-06T07:21:55.425219+0000","last_active":"2024-03-06T07:21:55.428760+0000","last_peered":"2024-03-06T07:21:55.428760+0000","last_clean":"2024-03-06T07:21:55.428760+0000","last_became_active":"2024-03-06T07:21:55.422831+0000","last_became_peered":"2024-03-06T07:21:55.422831+0000","last_unstale":"2024-03-06T07:21:55.428760+0000","last_undegraded":"2024-03-06T07:21:55.428760+0000","last_fullsized":"2024-03-06T07:21:55.428760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435751+0000","last_change":"2024-03-06T07:21:55.427953+0000","last_active":"2024-03-06T07:21:55.435751+0000","last_peered":"2024-03-06T07:21:55.435751+0000","last_clean":"2024-03-06T07:21:55.435751+0000","last_became_active":"2024-03-06T07:21:55.427652+0000","last_became_peered":"2024-03-06T07:21:55.427652+0000","last_unstale":"2024-03-06T07:21:55.435751+0000","last_undegraded":"2024-03-06T07:21:55.435751+0000","last_fullsized":"2024-03-06T07:21:55.435751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.434959+0000","last_change":"2024-03-06T07:21:55.421525+0000","last_active":"2024-03-06T07:21:55.434959+0000","last_peered":"2024-03-06T07:21:55.434959+0000","last_clean":"2024-03-06T07:21:55.434959+0000","last_became_active":"2024-03-06T07:21:55.421204+0000","last_became_peered":"2024-03-06T07:21:55.421204+0000","last_unstale":"2024-03-06T07:21:55.434959+0000","last_undegraded":"2024-03-06T07:21:55.434959+0000","last_fullsized":"2024-03-06T07:21:55.434959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:36.844180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435710+0000","last_change":"2024-03-06T07:21:55.424534+0000","last_active":"2024-03-06T07:21:55.435710+0000","last_peered":"2024-03-06T07:21:55.435710+0000","last_clean":"2024-03-06T07:21:55.435710+0000","last_became_active":"2024-03-06T07:21:55.424309+0000","last_became_peered":"2024-03-06T07:21:55.424309+0000","last_unstale":"2024-03-06T07:21:55.435710+0000","last_undegraded":"2024-03-06T07:21:55.435710+0000","last_fullsized":"2024-03-06T07:21:55.435710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.430301+0000","last_change":"2024-03-06T07:21:55.424084+0000","last_active":"2024-03-06T07:21:55.430301+0000","last_peered":"2024-03-06T07:21:55.430301+0000","last_clean":"2024-03-06T07:21:55.430301+0000","last_became_active":"2024-03-06T07:21:55.423245+0000","last_became_peered":"2024-03-06T07:21:55.423245+0000","last_unstale":"2024-03-06T07:21:55.430301+0000","last_undegraded":"2024-03-06T07:21:55.430301+0000","last_fullsized":"2024-03-06T07:21:55.430301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:29.441742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.420348+0000","last_change":"2024-03-06T07:21:55.415889+0000","last_active":"2024-03-06T07:21:55.420348+0000","last_peered":"2024-03-06T07:21:55.420348+0000","last_clean":"2024-03-06T07:21:55.420348+0000","last_became_active":"2024-03-06T07:21:55.415620+0000","last_became_peered":"2024-03-06T07:21:55.415620+0000","last_unstale":"2024-03-06T07:21:55.420348+0000","last_undegraded":"2024-03-06T07:21:55.420348+0000","last_fullsized":"2024-03-06T07:21:55.420348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424968+0000","last_change":"2024-03-06T07:21:55.422611+0000","last_active":"2024-03-06T07:21:55.424968+0000","last_peered":"2024-03-06T07:21:55.424968+0000","last_clean":"2024-03-06T07:21:55.424968+0000","last_became_active":"2024-03-06T07:21:55.422260+0000","last_became_peered":"2024-03-06T07:21:55.422260+0000","last_unstale":"2024-03-06T07:21:55.424968+0000","last_undegraded":"2024-03-06T07:21:55.424968+0000","last_fullsized":"2024-03-06T07:21:55.424968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.430230+0000","last_change":"2024-03-06T07:21:55.420808+0000","last_active":"2024-03-06T07:21:55.430230+0000","last_peered":"2024-03-06T07:21:55.430230+0000","last_clean":"2024-03-06T07:21:55.430230+0000","last_became_active":"2024-03-06T07:21:55.420529+0000","last_became_peered":"2024-03-06T07:21:55.420529+0000","last_unstale":"2024-03-06T07:21:55.430230+0000","last_undegraded":"2024-03-06T07:21:55.430230+0000","last_fullsized":"2024-03-06T07:21:55.430230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.433865+0000","last_change":"2024-03-06T07:21:55.420655+0000","last_active":"2024-03-06T07:21:55.433865+0000","last_peered":"2024-03-06T07:21:55.433865+0000","last_clean":"2024-03-06T07:21:55.433865+0000","last_became_active":"2024-03-06T07:21:55.420396+0000","last_became_peered":"2024-03-06T07:21:55.420396+0000","last_unstale":"2024-03-06T07:21:55.433865+0000","last_undegraded":"2024-03-06T07:21:55.433865+0000","last_fullsized":"2024-03-06T07:21:55.433865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435187+0000","last_change":"2024-03-06T07:21:55.431350+0000","last_active":"2024-03-06T07:21:55.435187+0000","last_peered":"2024-03-06T07:21:55.435187+0000","last_clean":"2024-03-06T07:21:55.435187+0000","last_became_active":"2024-03-06T07:21:55.431033+0000","last_became_peered":"2024-03-06T07:21:55.431033+0000","last_unstale":"2024-03-06T07:21:55.435187+0000","last_undegraded":"2024-03-06T07:21:55.435187+0000","last_fullsized":"2024-03-06T07:21:55.435187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.418173+0000","last_change":"2024-03-06T07:21:55.416463+0000","last_active":"2024-03-06T07:21:55.418173+0000","last_peered":"2024-03-06T07:21:55.418173+0000","last_clean":"2024-03-06T07:21:55.418173+0000","last_became_active":"2024-03-06T07:21:55.416174+0000","last_became_peered":"2024-03-06T07:21:55.416174+0000","last_unstale":"2024-03-06T07:21:55.418173+0000","last_undegraded":"2024-03-06T07:21:55.418173+0000","last_fullsized":"2024-03-06T07:21:55.418173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:06.592646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435688+0000","last_change":"2024-03-06T07:21:55.418595+0000","last_active":"2024-03-06T07:21:55.435688+0000","last_peered":"2024-03-06T07:21:55.435688+0000","last_clean":"2024-03-06T07:21:55.435688+0000","last_became_active":"2024-03-06T07:21:55.418073+0000","last_became_peered":"2024-03-06T07:21:55.418073+0000","last_unstale":"2024-03-06T07:21:55.435688+0000","last_undegraded":"2024-03-06T07:21:55.435688+0000","last_fullsized":"2024-03-06T07:21:55.435688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.433077+0000","last_change":"2024-03-06T07:21:55.429932+0000","last_active":"2024-03-06T07:21:55.433077+0000","last_peered":"2024-03-06T07:21:55.433077+0000","last_clean":"2024-03-06T07:21:55.433077+0000","last_became_active":"2024-03-06T07:21:55.429593+0000","last_became_peered":"2024-03-06T07:21:55.429593+0000","last_unstale":"2024-03-06T07:21:55.433077+0000","last_undegraded":"2024-03-06T07:21:55.433077+0000","last_fullsized":"2024-03-06T07:21:55.433077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:41.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440407+0000","last_change":"2024-03-06T07:21:55.436974+0000","last_active":"2024-03-06T07:21:55.440407+0000","last_peered":"2024-03-06T07:21:55.440407+0000","last_clean":"2024-03-06T07:21:55.440407+0000","last_became_active":"2024-03-06T07:21:55.436638+0000","last_became_peered":"2024-03-06T07:21:55.436638+0000","last_unstale":"2024-03-06T07:21:55.440407+0000","last_undegraded":"2024-03-06T07:21:55.440407+0000","last_fullsized":"2024-03-06T07:21:55.440407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440385+0000","last_change":"2024-03-06T07:21:55.435717+0000","last_active":"2024-03-06T07:21:55.440385+0000","last_peered":"2024-03-06T07:21:55.440385+0000","last_clean":"2024-03-06T07:21:55.440385+0000","last_became_active":"2024-03-06T07:21:55.435429+0000","last_became_peered":"2024-03-06T07:21:55.435429+0000","last_unstale":"2024-03-06T07:21:55.440385+0000","last_undegraded":"2024-03-06T07:21:55.440385+0000","last_fullsized":"2024-03-06T07:21:55.440385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:39.492542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.437829+0000","last_change":"2024-03-06T07:21:54.413886+0000","last_active":"2024-03-06T07:21:55.437829+0000","last_peered":"2024-03-06T07:21:55.437829+0000","last_clean":"2024-03-06T07:21:55.437829+0000","last_became_active":"2024-03-06T07:21:54.413573+0000","last_became_peered":"2024-03-06T07:21:54.413573+0000","last_unstale":"2024-03-06T07:21:55.437829+0000","last_undegraded":"2024-03-06T07:21:55.437829+0000","last_fullsized":"2024-03-06T07:21:55.437829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:17.458904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435069+0000","last_change":"2024-03-06T07:21:55.427074+0000","last_active":"2024-03-06T07:21:55.435069+0000","last_peered":"2024-03-06T07:21:55.435069+0000","last_clean":"2024-03-06T07:21:55.435069+0000","last_became_active":"2024-03-06T07:21:55.426775+0000","last_became_peered":"2024-03-06T07:21:55.426775+0000","last_unstale":"2024-03-06T07:21:55.435069+0000","last_undegraded":"2024-03-06T07:21:55.435069+0000","last_fullsized":"2024-03-06T07:21:55.435069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440666+0000","last_change":"2024-03-06T07:21:54.418543+0000","last_active":"2024-03-06T07:21:55.440666+0000","last_peered":"2024-03-06T07:21:55.440666+0000","last_clean":"2024-03-06T07:21:55.440666+0000","last_became_active":"2024-03-06T07:21:54.418148+0000","last_became_peered":"2024-03-06T07:21:54.418148+0000","last_unstale":"2024-03-06T07:21:55.440666+0000","last_undegraded":"2024-03-06T07:21:55.440666+0000","last_fullsized":"2024-03-06T07:21:55.440666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.410777+0000","last_change":"2024-03-06T07:21:54.417756+0000","last_active":"2024-03-06T07:21:55.410777+0000","last_peered":"2024-03-06T07:21:55.410777+0000","last_clean":"2024-03-06T07:21:55.410777+0000","last_became_active":"2024-03-06T07:21:54.417471+0000","last_became_peered":"2024-03-06T07:21:54.417471+0000","last_unstale":"2024-03-06T07:21:55.410777+0000","last_undegraded":"2024-03-06T07:21:55.410777+0000","last_fullsized":"2024-03-06T07:21:55.410777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424717+0000","last_change":"2024-03-06T07:21:55.418688+0000","last_active":"2024-03-06T07:21:55.424717+0000","last_peered":"2024-03-06T07:21:55.424717+0000","last_clean":"2024-03-06T07:21:55.424717+0000","last_became_active":"2024-03-06T07:21:55.418392+0000","last_became_peered":"2024-03-06T07:21:55.418392+0000","last_unstale":"2024-03-06T07:21:55.424717+0000","last_undegraded":"2024-03-06T07:21:55.424717+0000","last_fullsized":"2024-03-06T07:21:55.424717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:43.203590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.410374+0000","last_change":"2024-03-06T07:21:54.414567+0000","last_active":"2024-03-06T07:21:55.410374+0000","last_peered":"2024-03-06T07:21:55.410374+0000","last_clean":"2024-03-06T07:21:55.410374+0000","last_became_active":"2024-03-06T07:21:54.414292+0000","last_became_peered":"2024-03-06T07:21:54.414292+0000","last_unstale":"2024-03-06T07:21:55.410374+0000","last_undegraded":"2024-03-06T07:21:55.410374+0000","last_fullsized":"2024-03-06T07:21:55.410374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.432798+0000","last_change":"2024-03-06T07:21:55.420297+0000","last_active":"2024-03-06T07:21:55.432798+0000","last_peered":"2024-03-06T07:21:55.432798+0000","last_clean":"2024-03-06T07:21:55.432798+0000","last_became_active":"2024-03-06T07:21:55.419994+0000","last_became_peered":"2024-03-06T07:21:55.419994+0000","last_unstale":"2024-03-06T07:21:55.432798+0000","last_undegraded":"2024-03-06T07:21:55.432798+0000","last_fullsized":"2024-03-06T07:21:55.432798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:41.897729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.416725+0000","last_change":"2024-03-06T07:21:54.420034+0000","last_active":"2024-03-06T07:21:55.416725+0000","last_peered":"2024-03-06T07:21:55.416725+0000","last_clean":"2024-03-06T07:21:55.416725+0000","last_became_active":"2024-03-06T07:21:54.419745+0000","last_became_peered":"2024-03-06T07:21:54.419745+0000","last_unstale":"2024-03-06T07:21:55.416725+0000","last_undegraded":"2024-03-06T07:21:55.416725+0000","last_fullsized":"2024-03-06T07:21:55.416725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435588+0000","last_change":"2024-03-06T07:21:55.425571+0000","last_active":"2024-03-06T07:21:55.435588+0000","last_peered":"2024-03-06T07:21:55.435588+0000","last_clean":"2024-03-06T07:21:55.435588+0000","last_became_active":"2024-03-06T07:21:55.424876+0000","last_became_peered":"2024-03-06T07:21:55.424876+0000","last_unstale":"2024-03-06T07:21:55.435588+0000","last_undegraded":"2024-03-06T07:21:55.435588+0000","last_fullsized":"2024-03-06T07:21:55.435588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.410180+0000","last_change":"2024-03-06T07:21:54.416884+0000","last_active":"2024-03-06T07:21:55.410180+0000","last_peered":"2024-03-06T07:21:55.410180+0000","last_clean":"2024-03-06T07:21:55.410180+0000","last_became_active":"2024-03-06T07:21:54.416576+0000","last_became_peered":"2024-03-06T07:21:54.416576+0000","last_unstale":"2024-03-06T07:21:55.410180+0000","last_undegraded":"2024-03-06T07:21:55.410180+0000","last_fullsized":"2024-03-06T07:21:55.410180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.417059+0000","last_change":"2024-03-06T07:21:55.414257+0000","last_active":"2024-03-06T07:21:55.417059+0000","last_peered":"2024-03-06T07:21:55.417059+0000","last_clean":"2024-03-06T07:21:55.417059+0000","last_became_active":"2024-03-06T07:21:55.413950+0000","last_became_peered":"2024-03-06T07:21:55.413950+0000","last_unstale":"2024-03-06T07:21:55.417059+0000","last_undegraded":"2024-03-06T07:21:55.417059+0000","last_fullsized":"2024-03-06T07:21:55.417059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:18.465323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440811+0000","last_change":"2024-03-06T07:21:54.412699+0000","last_active":"2024-03-06T07:21:55.440811+0000","last_peered":"2024-03-06T07:21:55.440811+0000","last_clean":"2024-03-06T07:21:55.440811+0000","last_became_active":"2024-03-06T07:21:54.412392+0000","last_became_peered":"2024-03-06T07:21:54.412392+0000","last_unstale":"2024-03-06T07:21:55.440811+0000","last_undegraded":"2024-03-06T07:21:55.440811+0000","last_fullsized":"2024-03-06T07:21:55.440811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.426012+0000","last_change":"2024-03-06T07:21:55.421734+0000","last_active":"2024-03-06T07:21:55.426012+0000","last_peered":"2024-03-06T07:21:55.426012+0000","last_clean":"2024-03-06T07:21:55.426012+0000","last_became_active":"2024-03-06T07:21:55.421451+0000","last_became_peered":"2024-03-06T07:21:55.421451+0000","last_unstale":"2024-03-06T07:21:55.426012+0000","last_undegraded":"2024-03-06T07:21:55.426012+0000","last_fullsized":"2024-03-06T07:21:55.426012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:16.284196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.443283+0000","last_change":"2024-03-06T07:21:54.416848+0000","last_active":"2024-03-06T07:21:55.443283+0000","last_peered":"2024-03-06T07:21:55.443283+0000","last_clean":"2024-03-06T07:21:55.443283+0000","last_became_active":"2024-03-06T07:21:54.416561+0000","last_became_peered":"2024-03-06T07:21:54.416561+0000","last_unstale":"2024-03-06T07:21:55.443283+0000","last_undegraded":"2024-03-06T07:21:55.443283+0000","last_fullsized":"2024-03-06T07:21:55.443283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.429565+0000","last_change":"2024-03-06T07:21:55.425748+0000","last_active":"2024-03-06T07:21:55.429565+0000","last_peered":"2024-03-06T07:21:55.429565+0000","last_clean":"2024-03-06T07:21:55.429565+0000","last_became_active":"2024-03-06T07:21:55.425501+0000","last_became_peered":"2024-03-06T07:21:55.425501+0000","last_unstale":"2024-03-06T07:21:55.429565+0000","last_undegraded":"2024-03-06T07:21:55.429565+0000","last_fullsized":"2024-03-06T07:21:55.429565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.441049+0000","last_change":"2024-03-06T07:21:54.419364+0000","last_active":"2024-03-06T07:21:55.441049+0000","last_peered":"2024-03-06T07:21:55.441049+0000","last_clean":"2024-03-06T07:21:55.441049+0000","last_became_active":"2024-03-06T07:21:54.419094+0000","last_became_peered":"2024-03-06T07:21:54.419094+0000","last_unstale":"2024-03-06T07:21:55.441049+0000","last_undegraded":"2024-03-06T07:21:55.441049+0000","last_fullsized":"2024-03-06T07:21:55.441049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435063+0000","last_change":"2024-03-06T07:21:55.421461+0000","last_active":"2024-03-06T07:21:55.435063+0000","last_peered":"2024-03-06T07:21:55.435063+0000","last_clean":"2024-03-06T07:21:55.435063+0000","last_became_active":"2024-03-06T07:21:55.421153+0000","last_became_peered":"2024-03-06T07:21:55.421153+0000","last_unstale":"2024-03-06T07:21:55.435063+0000","last_undegraded":"2024-03-06T07:21:55.435063+0000","last_fullsized":"2024-03-06T07:21:55.435063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:41.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.411734+0000","last_change":"2024-03-06T07:21:54.416121+0000","last_active":"2024-03-06T07:21:55.411734+0000","last_peered":"2024-03-06T07:21:55.411734+0000","last_clean":"2024-03-06T07:21:55.411734+0000","last_became_active":"2024-03-06T07:21:54.415857+0000","last_became_peered":"2024-03-06T07:21:54.415857+0000","last_unstale":"2024-03-06T07:21:55.411734+0000","last_undegraded":"2024-03-06T07:21:55.411734+0000","last_fullsized":"2024-03-06T07:21:55.411734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.431425+0000","last_change":"2024-03-06T07:21:55.428224+0000","last_active":"2024-03-06T07:21:55.431425+0000","last_peered":"2024-03-06T07:21:55.431425+0000","last_clean":"2024-03-06T07:21:55.431425+0000","last_became_active":"2024-03-06T07:21:55.427932+0000","last_became_peered":"2024-03-06T07:21:55.427932+0000","last_unstale":"2024-03-06T07:21:55.431425+0000","last_undegraded":"2024-03-06T07:21:55.431425+0000","last_fullsized":"2024-03-06T07:21:55.431425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.416675+0000","last_change":"2024-03-06T07:21:54.418770+0000","last_active":"2024-03-06T07:21:55.416675+0000","last_peered":"2024-03-06T07:21:55.416675+0000","last_clean":"2024-03-06T07:21:55.416675+0000","last_became_active":"2024-03-06T07:21:54.418498+0000","last_became_peered":"2024-03-06T07:21:54.418498+0000","last_unstale":"2024-03-06T07:21:55.416675+0000","last_undegraded":"2024-03-06T07:21:55.416675+0000","last_fullsized":"2024-03-06T07:21:55.416675+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440338+0000","last_change":"2024-03-06T07:21:55.436175+0000","last_active":"2024-03-06T07:21:55.440338+0000","last_peered":"2024-03-06T07:21:55.440338+0000","last_clean":"2024-03-06T07:21:55.440338+0000","last_became_active":"2024-03-06T07:21:55.435873+0000","last_became_peered":"2024-03-06T07:21:55.435873+0000","last_unstale":"2024-03-06T07:21:55.440338+0000","last_undegraded":"2024-03-06T07:21:55.440338+0000","last_fullsized":"2024-03-06T07:21:55.440338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:29.441742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.441358+0000","last_change":"2024-03-06T07:21:54.410321+0000","last_active":"2024-03-06T07:21:55.441358+0000","last_peered":"2024-03-06T07:21:55.441358+0000","last_clean":"2024-03-06T07:21:55.441358+0000","last_became_active":"2024-03-06T07:21:54.410019+0000","last_became_peered":"2024-03-06T07:21:54.410019+0000","last_unstale":"2024-03-06T07:21:55.441358+0000","last_undegraded":"2024-03-06T07:21:55.441358+0000","last_fullsized":"2024-03-06T07:21:55.441358+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.430732+0000","last_change":"2024-03-06T07:21:55.427171+0000","last_active":"2024-03-06T07:21:55.430732+0000","last_peered":"2024-03-06T07:21:55.430732+0000","last_clean":"2024-03-06T07:21:55.430732+0000","last_became_active":"2024-03-06T07:21:55.426872+0000","last_became_peered":"2024-03-06T07:21:55.426872+0000","last_unstale":"2024-03-06T07:21:55.430732+0000","last_undegraded":"2024-03-06T07:21:55.430732+0000","last_fullsized":"2024-03-06T07:21:55.430732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.441069+0000","last_change":"2024-03-06T07:21:54.419357+0000","last_active":"2024-03-06T07:21:55.441069+0000","last_peered":"2024-03-06T07:21:55.441069+0000","last_clean":"2024-03-06T07:21:55.441069+0000","last_became_active":"2024-03-06T07:21:54.419053+0000","last_became_peered":"2024-03-06T07:21:54.419053+0000","last_unstale":"2024-03-06T07:21:55.441069+0000","last_undegraded":"2024-03-06T07:21:55.441069+0000","last_fullsized":"2024-03-06T07:21:55.441069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:42.184203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.429597+0000","last_change":"2024-03-06T07:21:55.427511+0000","last_active":"2024-03-06T07:21:55.429597+0000","last_peered":"2024-03-06T07:21:55.429597+0000","last_clean":"2024-03-06T07:21:55.429597+0000","last_became_active":"2024-03-06T07:21:55.427175+0000","last_became_peered":"2024-03-06T07:21:55.427175+0000","last_unstale":"2024-03-06T07:21:55.429597+0000","last_undegraded":"2024-03-06T07:21:55.429597+0000","last_fullsized":"2024-03-06T07:21:55.429597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440027+0000","last_change":"2024-03-06T07:21:54.416325+0000","last_active":"2024-03-06T07:21:55.440027+0000","last_peered":"2024-03-06T07:21:55.440027+0000","last_clean":"2024-03-06T07:21:55.440027+0000","last_became_active":"2024-03-06T07:21:54.416017+0000","last_became_peered":"2024-03-06T07:21:54.416017+0000","last_unstale":"2024-03-06T07:21:55.440027+0000","last_undegraded":"2024-03-06T07:21:55.440027+0000","last_fullsized":"2024-03-06T07:21:55.440027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.431284+0000","last_change":"2024-03-06T07:21:55.428378+0000","last_active":"2024-03-06T07:21:55.431284+0000","last_peered":"2024-03-06T07:21:55.431284+0000","last_clean":"2024-03-06T07:21:55.431284+0000","last_became_active":"2024-03-06T07:21:55.426715+0000","last_became_peered":"2024-03-06T07:21:55.426715+0000","last_unstale":"2024-03-06T07:21:55.431284+0000","last_undegraded":"2024-03-06T07:21:55.431284+0000","last_fullsized":"2024-03-06T07:21:55.431284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440720+0000","last_change":"2024-03-06T07:21:54.406786+0000","last_active":"2024-03-06T07:21:55.440720+0000","last_peered":"2024-03-06T07:21:55.440720+0000","last_clean":"2024-03-06T07:21:55.440720+0000","last_became_active":"2024-03-06T07:21:54.406454+0000","last_became_peered":"2024-03-06T07:21:54.406454+0000","last_unstale":"2024-03-06T07:21:55.440720+0000","last_undegraded":"2024-03-06T07:21:55.440720+0000","last_fullsized":"2024-03-06T07:21:55.440720+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:50.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435628+0000","last_change":"2024-03-06T07:21:55.422470+0000","last_active":"2024-03-06T07:21:55.435628+0000","last_peered":"2024-03-06T07:21:55.435628+0000","last_clean":"2024-03-06T07:21:55.435628+0000","last_became_active":"2024-03-06T07:21:55.420656+0000","last_became_peered":"2024-03-06T07:21:55.420656+0000","last_unstale":"2024-03-06T07:21:55.435628+0000","last_undegraded":"2024-03-06T07:21:55.435628+0000","last_fullsized":"2024-03-06T07:21:55.435628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:29.441742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.413292+0000","last_change":"2024-03-06T07:21:54.412695+0000","last_active":"2024-03-06T07:21:55.413292+0000","last_peered":"2024-03-06T07:21:55.413292+0000","last_clean":"2024-03-06T07:21:55.413292+0000","last_became_active":"2024-03-06T07:21:54.412391+0000","last_became_peered":"2024-03-06T07:21:54.412391+0000","last_unstale":"2024-03-06T07:21:55.413292+0000","last_undegraded":"2024-03-06T07:21:55.413292+0000","last_fullsized":"2024-03-06T07:21:55.413292+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:17.458904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440146+0000","last_change":"2024-03-06T07:21:55.436949+0000","last_active":"2024-03-06T07:21:55.440146+0000","last_peered":"2024-03-06T07:21:55.440146+0000","last_clean":"2024-03-06T07:21:55.440146+0000","last_became_active":"2024-03-06T07:21:55.436502+0000","last_became_peered":"2024-03-06T07:21:55.436502+0000","last_unstale":"2024-03-06T07:21:55.440146+0000","last_undegraded":"2024-03-06T07:21:55.440146+0000","last_fullsized":"2024-03-06T07:21:55.440146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:49.817183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.417227+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:21:55.417227+0000","last_peered":"2024-03-06T07:21:55.417227+0000","last_clean":"2024-03-06T07:21:55.417227+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:21:55.417227+0000","last_undegraded":"2024-03-06T07:21:55.417227+0000","last_fullsized":"2024-03-06T07:21:55.417227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.409149+0000","last_change":"2024-03-06T07:21:54.411371+0000","last_active":"2024-03-06T07:21:55.409149+0000","last_peered":"2024-03-06T07:21:55.409149+0000","last_clean":"2024-03-06T07:21:55.409149+0000","last_became_active":"2024-03-06T07:21:54.411095+0000","last_became_peered":"2024-03-06T07:21:54.411095+0000","last_unstale":"2024-03-06T07:21:55.409149+0000","last_undegraded":"2024-03-06T07:21:55.409149+0000","last_fullsized":"2024-03-06T07:21:55.409149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.427078+0000","last_change":"2024-03-06T07:21:55.423928+0000","last_active":"2024-03-06T07:21:55.427078+0000","last_peered":"2024-03-06T07:21:55.427078+0000","last_clean":"2024-03-06T07:21:55.427078+0000","last_became_active":"2024-03-06T07:21:55.423681+0000","last_became_peered":"2024-03-06T07:21:55.423681+0000","last_unstale":"2024-03-06T07:21:55.427078+0000","last_undegraded":"2024-03-06T07:21:55.427078+0000","last_fullsized":"2024-03-06T07:21:55.427078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:41.897729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.416633+0000","last_change":"2024-03-06T07:21:54.415381+0000","last_active":"2024-03-06T07:21:55.416633+0000","last_peered":"2024-03-06T07:21:55.416633+0000","last_clean":"2024-03-06T07:21:55.416633+0000","last_became_active":"2024-03-06T07:21:54.415099+0000","last_became_peered":"2024-03-06T07:21:54.415099+0000","last_unstale":"2024-03-06T07:21:55.416633+0000","last_undegraded":"2024-03-06T07:21:55.416633+0000","last_fullsized":"2024-03-06T07:21:55.416633+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.891311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.434971+0000","last_change":"2024-03-06T07:21:55.423013+0000","last_active":"2024-03-06T07:21:55.434971+0000","last_peered":"2024-03-06T07:21:55.434971+0000","last_clean":"2024-03-06T07:21:55.434971+0000","last_became_active":"2024-03-06T07:21:55.422717+0000","last_became_peered":"2024-03-06T07:21:55.422717+0000","last_unstale":"2024-03-06T07:21:55.434971+0000","last_undegraded":"2024-03-06T07:21:55.434971+0000","last_fullsized":"2024-03-06T07:21:55.434971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.414367+0000","last_change":"2024-03-06T07:21:54.416728+0000","last_active":"2024-03-06T07:21:55.414367+0000","last_peered":"2024-03-06T07:21:55.414367+0000","last_clean":"2024-03-06T07:21:55.414367+0000","last_became_active":"2024-03-06T07:21:54.416226+0000","last_became_peered":"2024-03-06T07:21:54.416226+0000","last_unstale":"2024-03-06T07:21:55.414367+0000","last_undegraded":"2024-03-06T07:21:55.414367+0000","last_fullsized":"2024-03-06T07:21:55.414367+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.437395+0000","last_change":"2024-03-06T07:21:55.429746+0000","last_active":"2024-03-06T07:21:55.437395+0000","last_peered":"2024-03-06T07:21:55.437395+0000","last_clean":"2024-03-06T07:21:55.437395+0000","last_became_active":"2024-03-06T07:21:55.429320+0000","last_became_peered":"2024-03-06T07:21:55.429320+0000","last_unstale":"2024-03-06T07:21:55.437395+0000","last_undegraded":"2024-03-06T07:21:55.437395+0000","last_fullsized":"2024-03-06T07:21:55.437395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.411567+0000","last_change":"2024-03-06T07:21:54.413058+0000","last_active":"2024-03-06T07:21:55.411567+0000","last_peered":"2024-03-06T07:21:55.411567+0000","last_clean":"2024-03-06T07:21:55.411567+0000","last_became_active":"2024-03-06T07:21:54.412469+0000","last_became_peered":"2024-03-06T07:21:54.412469+0000","last_unstale":"2024-03-06T07:21:55.411567+0000","last_undegraded":"2024-03-06T07:21:55.411567+0000","last_fullsized":"2024-03-06T07:21:55.411567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.431559+0000","last_change":"2024-03-06T07:21:55.425788+0000","last_active":"2024-03-06T07:21:55.431559+0000","last_peered":"2024-03-06T07:21:55.431559+0000","last_clean":"2024-03-06T07:21:55.431559+0000","last_became_active":"2024-03-06T07:21:55.425440+0000","last_became_peered":"2024-03-06T07:21:55.425440+0000","last_unstale":"2024-03-06T07:21:55.431559+0000","last_undegraded":"2024-03-06T07:21:55.431559+0000","last_fullsized":"2024-03-06T07:21:55.431559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:06.592646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.415776+0000","last_change":"2024-03-06T07:21:54.415739+0000","last_active":"2024-03-06T07:21:55.415776+0000","last_peered":"2024-03-06T07:21:55.415776+0000","last_clean":"2024-03-06T07:21:55.415776+0000","last_became_active":"2024-03-06T07:21:54.415452+0000","last_became_peered":"2024-03-06T07:21:54.415452+0000","last_unstale":"2024-03-06T07:21:55.415776+0000","last_undegraded":"2024-03-06T07:21:55.415776+0000","last_fullsized":"2024-03-06T07:21:55.415776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435784+0000","last_change":"2024-03-06T07:21:55.419140+0000","last_active":"2024-03-06T07:21:55.435784+0000","last_peered":"2024-03-06T07:21:55.435784+0000","last_clean":"2024-03-06T07:21:55.435784+0000","last_became_active":"2024-03-06T07:21:55.415992+0000","last_became_peered":"2024-03-06T07:21:55.415992+0000","last_unstale":"2024-03-06T07:21:55.435784+0000","last_undegraded":"2024-03-06T07:21:55.435784+0000","last_fullsized":"2024-03-06T07:21:55.435784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:41.897729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.416094+0000","last_change":"2024-03-06T07:21:54.417366+0000","last_active":"2024-03-06T07:21:55.416094+0000","last_peered":"2024-03-06T07:21:55.416094+0000","last_clean":"2024-03-06T07:21:55.416094+0000","last_became_active":"2024-03-06T07:21:54.416684+0000","last_became_peered":"2024-03-06T07:21:54.416684+0000","last_unstale":"2024-03-06T07:21:55.416094+0000","last_undegraded":"2024-03-06T07:21:55.416094+0000","last_fullsized":"2024-03-06T07:21:55.416094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:44.263686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.432854+0000","last_change":"2024-03-06T07:21:55.419798+0000","last_active":"2024-03-06T07:21:55.432854+0000","last_peered":"2024-03-06T07:21:55.432854+0000","last_clean":"2024-03-06T07:21:55.432854+0000","last_became_active":"2024-03-06T07:21:55.419490+0000","last_became_peered":"2024-03-06T07:21:55.419490+0000","last_unstale":"2024-03-06T07:21:55.432854+0000","last_undegraded":"2024-03-06T07:21:55.432854+0000","last_fullsized":"2024-03-06T07:21:55.432854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.416114+0000","last_change":"2024-03-06T07:21:54.418937+0000","last_active":"2024-03-06T07:21:55.416114+0000","last_peered":"2024-03-06T07:21:55.416114+0000","last_clean":"2024-03-06T07:21:55.416114+0000","last_became_active":"2024-03-06T07:21:54.418592+0000","last_became_peered":"2024-03-06T07:21:54.418592+0000","last_unstale":"2024-03-06T07:21:55.416114+0000","last_undegraded":"2024-03-06T07:21:55.416114+0000","last_fullsized":"2024-03-06T07:21:55.416114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:17.773963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424739+0000","last_change":"2024-03-06T07:21:55.420527+0000","last_active":"2024-03-06T07:21:55.424739+0000","last_peered":"2024-03-06T07:21:55.424739+0000","last_clean":"2024-03-06T07:21:55.424739+0000","last_became_active":"2024-03-06T07:21:55.420210+0000","last_became_peered":"2024-03-06T07:21:55.420210+0000","last_unstale":"2024-03-06T07:21:55.424739+0000","last_undegraded":"2024-03-06T07:21:55.424739+0000","last_fullsized":"2024-03-06T07:21:55.424739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:41.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.437818+0000","last_change":"2024-03-06T07:21:54.420765+0000","last_active":"2024-03-06T07:21:55.437818+0000","last_peered":"2024-03-06T07:21:55.437818+0000","last_clean":"2024-03-06T07:21:55.437818+0000","last_became_active":"2024-03-06T07:21:54.420541+0000","last_became_peered":"2024-03-06T07:21:54.420541+0000","last_unstale":"2024-03-06T07:21:55.437818+0000","last_undegraded":"2024-03-06T07:21:55.437818+0000","last_fullsized":"2024-03-06T07:21:55.437818+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.437992+0000","last_change":"2024-03-06T07:21:55.430019+0000","last_active":"2024-03-06T07:21:55.437992+0000","last_peered":"2024-03-06T07:21:55.437992+0000","last_clean":"2024-03-06T07:21:55.437992+0000","last_became_active":"2024-03-06T07:21:55.429671+0000","last_became_peered":"2024-03-06T07:21:55.429671+0000","last_unstale":"2024-03-06T07:21:55.437992+0000","last_undegraded":"2024-03-06T07:21:55.437992+0000","last_fullsized":"2024-03-06T07:21:55.437992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:41.897729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.417615+0000","last_change":"2024-03-06T07:21:54.416295+0000","last_active":"2024-03-06T07:21:55.417615+0000","last_peered":"2024-03-06T07:21:55.417615+0000","last_clean":"2024-03-06T07:21:55.417615+0000","last_became_active":"2024-03-06T07:21:54.415817+0000","last_became_peered":"2024-03-06T07:21:54.415817+0000","last_unstale":"2024-03-06T07:21:55.417615+0000","last_undegraded":"2024-03-06T07:21:55.417615+0000","last_fullsized":"2024-03-06T07:21:55.417615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:28.435323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.426155+0000","last_change":"2024-03-06T07:21:55.422908+0000","last_active":"2024-03-06T07:21:55.426155+0000","last_peered":"2024-03-06T07:21:55.426155+0000","last_clean":"2024-03-06T07:21:55.426155+0000","last_became_active":"2024-03-06T07:21:55.422624+0000","last_became_peered":"2024-03-06T07:21:55.422624+0000","last_unstale":"2024-03-06T07:21:55.426155+0000","last_undegraded":"2024-03-06T07:21:55.426155+0000","last_fullsized":"2024-03-06T07:21:55.426155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:37.109379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.415812+0000","last_change":"2024-03-06T07:21:54.411585+0000","last_active":"2024-03-06T07:21:55.415812+0000","last_peered":"2024-03-06T07:21:55.415812+0000","last_clean":"2024-03-06T07:21:55.415812+0000","last_became_active":"2024-03-06T07:21:54.411318+0000","last_became_peered":"2024-03-06T07:21:54.411318+0000","last_unstale":"2024-03-06T07:21:55.415812+0000","last_undegraded":"2024-03-06T07:21:55.415812+0000","last_fullsized":"2024-03-06T07:21:55.415812+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424780+0000","last_change":"2024-03-06T07:21:55.421416+0000","last_active":"2024-03-06T07:21:55.424780+0000","last_peered":"2024-03-06T07:21:55.424780+0000","last_clean":"2024-03-06T07:21:55.424780+0000","last_became_active":"2024-03-06T07:21:55.421113+0000","last_became_peered":"2024-03-06T07:21:55.421113+0000","last_unstale":"2024-03-06T07:21:55.424780+0000","last_undegraded":"2024-03-06T07:21:55.424780+0000","last_fullsized":"2024-03-06T07:21:55.424780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:36.844180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.415626+0000","last_change":"2024-03-06T07:21:54.416417+0000","last_active":"2024-03-06T07:21:55.415626+0000","last_peered":"2024-03-06T07:21:55.415626+0000","last_clean":"2024-03-06T07:21:55.415626+0000","last_became_active":"2024-03-06T07:21:54.416108+0000","last_became_peered":"2024-03-06T07:21:54.416108+0000","last_unstale":"2024-03-06T07:21:55.415626+0000","last_undegraded":"2024-03-06T07:21:55.415626+0000","last_fullsized":"2024-03-06T07:21:55.415626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.424681+0000","last_change":"2024-03-06T07:21:55.418459+0000","last_active":"2024-03-06T07:21:55.424681+0000","last_peered":"2024-03-06T07:21:55.424681+0000","last_clean":"2024-03-06T07:21:55.424681+0000","last_became_active":"2024-03-06T07:21:55.418179+0000","last_became_peered":"2024-03-06T07:21:55.418179+0000","last_unstale":"2024-03-06T07:21:55.424681+0000","last_undegraded":"2024-03-06T07:21:55.424681+0000","last_fullsized":"2024-03-06T07:21:55.424681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:39.492542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440606+0000","last_change":"2024-03-06T07:21:54.412875+0000","last_active":"2024-03-06T07:21:55.440606+0000","last_peered":"2024-03-06T07:21:55.440606+0000","last_clean":"2024-03-06T07:21:55.440606+0000","last_became_active":"2024-03-06T07:21:54.412617+0000","last_became_peered":"2024-03-06T07:21:54.412617+0000","last_unstale":"2024-03-06T07:21:55.440606+0000","last_undegraded":"2024-03-06T07:21:55.440606+0000","last_fullsized":"2024-03-06T07:21:55.440606+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:50.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.434473+0000","last_change":"2024-03-06T07:21:55.427872+0000","last_active":"2024-03-06T07:21:55.434473+0000","last_peered":"2024-03-06T07:21:55.434473+0000","last_clean":"2024-03-06T07:21:55.434473+0000","last_became_active":"2024-03-06T07:21:55.427504+0000","last_became_peered":"2024-03-06T07:21:55.427504+0000","last_unstale":"2024-03-06T07:21:55.434473+0000","last_undegraded":"2024-03-06T07:21:55.434473+0000","last_fullsized":"2024-03-06T07:21:55.434473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435812+0000","last_change":"2024-03-06T07:21:55.427757+0000","last_active":"2024-03-06T07:21:55.435812+0000","last_peered":"2024-03-06T07:21:55.435812+0000","last_clean":"2024-03-06T07:21:55.435812+0000","last_became_active":"2024-03-06T07:21:55.427471+0000","last_became_peered":"2024-03-06T07:21:55.427471+0000","last_unstale":"2024-03-06T07:21:55.435812+0000","last_undegraded":"2024-03-06T07:21:55.435812+0000","last_fullsized":"2024-03-06T07:21:55.435812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440945+0000","last_change":"2024-03-06T07:21:54.414035+0000","last_active":"2024-03-06T07:21:55.440945+0000","last_peered":"2024-03-06T07:21:55.440945+0000","last_clean":"2024-03-06T07:21:55.440945+0000","last_became_active":"2024-03-06T07:21:54.413618+0000","last_became_peered":"2024-03-06T07:21:54.413618+0000","last_unstale":"2024-03-06T07:21:55.440945+0000","last_undegraded":"2024-03-06T07:21:55.440945+0000","last_fullsized":"2024-03-06T07:21:55.440945+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435670+0000","last_change":"2024-03-06T07:21:55.423540+0000","last_active":"2024-03-06T07:21:55.435670+0000","last_peered":"2024-03-06T07:21:55.435670+0000","last_clean":"2024-03-06T07:21:55.435670+0000","last_became_active":"2024-03-06T07:21:55.423197+0000","last_became_peered":"2024-03-06T07:21:55.423197+0000","last_unstale":"2024-03-06T07:21:55.435670+0000","last_undegraded":"2024-03-06T07:21:55.435670+0000","last_fullsized":"2024-03-06T07:21:55.435670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.440274+0000","last_change":"2024-03-06T07:21:54.417365+0000","last_active":"2024-03-06T07:21:55.440274+0000","last_peered":"2024-03-06T07:21:55.440274+0000","last_clean":"2024-03-06T07:21:55.440274+0000","last_became_active":"2024-03-06T07:21:54.417072+0000","last_became_peered":"2024-03-06T07:21:54.417072+0000","last_unstale":"2024-03-06T07:21:55.440274+0000","last_undegraded":"2024-03-06T07:21:55.440274+0000","last_fullsized":"2024-03-06T07:21:55.440274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:17.773963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.435089+0000","last_change":"2024-03-06T07:21:55.424176+0000","last_active":"2024-03-06T07:21:55.435089+0000","last_peered":"2024-03-06T07:21:55.435089+0000","last_clean":"2024-03-06T07:21:55.435089+0000","last_became_active":"2024-03-06T07:21:55.423848+0000","last_became_peered":"2024-03-06T07:21:55.423848+0000","last_unstale":"2024-03-06T07:21:55.435089+0000","last_undegraded":"2024-03-06T07:21:55.435089+0000","last_fullsized":"2024-03-06T07:21:55.435089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.438215+0000","last_change":"2024-03-06T07:21:54.418869+0000","last_active":"2024-03-06T07:21:55.438215+0000","last_peered":"2024-03-06T07:21:55.438215+0000","last_clean":"2024-03-06T07:21:55.438215+0000","last_became_active":"2024-03-06T07:21:54.418582+0000","last_became_peered":"2024-03-06T07:21:54.418582+0000","last_unstale":"2024-03-06T07:21:55.438215+0000","last_undegraded":"2024-03-06T07:21:55.438215+0000","last_fullsized":"2024-03-06T07:21:55.438215+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.434556+0000","last_change":"2024-03-06T07:21:55.427658+0000","last_active":"2024-03-06T07:21:55.434556+0000","last_peered":"2024-03-06T07:21:55.434556+0000","last_clean":"2024-03-06T07:21:55.434556+0000","last_became_active":"2024-03-06T07:21:55.427370+0000","last_became_peered":"2024-03-06T07:21:55.427370+0000","last_unstale":"2024-03-06T07:21:55.434556+0000","last_undegraded":"2024-03-06T07:21:55.434556+0000","last_fullsized":"2024-03-06T07:21:55.434556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-06T07:21:55.438249+0000","last_change":"2024-03-06T07:21:54.417435+0000","last_active":"2024-03-06T07:21:55.438249+0000","last_peered":"2024-03-06T07:21:55.438249+0000","last_clean":"2024-03-06T07:21:55.438249+0000","last_became_active":"2024-03-06T07:21:54.417136+0000","last_became_peered":"2024-03-06T07:21:54.417136+0000","last_unstale":"2024-03-06T07:21:55.438249+0000","last_undegraded":"2024-03-06T07:21:55.438249+0000","last_fullsized":"2024-03-06T07:21:55.438249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:15.277777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":655584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":64760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":64794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":655640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:58.773 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:21:58.774 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:21:58.774 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:21:58.774 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:21:58.774 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:21:58.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.910+0000 7f135f5bf640 1 Processor -- start 2024-03-06T07:21:58.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.910+0000 7f135f5bf640 1 -- start start 2024-03-06T07:21:58.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135f5bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13580d0320 0x7f13580b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135f5bf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f13580d0700 0x7f13580b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135f5bf640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f13580ccae0 0x7f13580ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135f5bf640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13580b04e0 con 0x7f13580ccae0 2024-03-06T07:21:58.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135f5bf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135815c270 con 0x7f13580d0700 2024-03-06T07:21:58.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135f5bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13580d0e90 con 0x7f13580d0320 2024-03-06T07:21:58.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135db5c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f13580ccae0 0x7f13580ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135db5c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f13580ccae0 0x7f13580ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57972/0 (socket says 172.21.15.59:57972) 2024-03-06T07:21:58.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135db5c640 1 -- 172.21.15.59:0/2683758955 learned_addr learned my addr 172.21.15.59:0/2683758955 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:58.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135cb5a640 1 --2- 172.21.15.59:0/2683758955 >> v2:172.21.15.164:3300/0 conn(0x7f13580d0700 0x7f13580b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135d35b640 1 --2- 172.21.15.59:0/2683758955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13580d0320 0x7f13580b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135db5c640 1 -- 172.21.15.59:0/2683758955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13580d0320 msgr2=0x7f13580b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135db5c640 1 --2- 172.21.15.59:0/2683758955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13580d0320 0x7f13580b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135db5c640 1 -- 172.21.15.59:0/2683758955 >> v2:172.21.15.164:3300/0 conn(0x7f13580d0700 msgr2=0x7f13580b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135db5c640 1 --2- 172.21.15.59:0/2683758955 >> v2:172.21.15.164:3300/0 conn(0x7f13580d0700 0x7f13580b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.911+0000 7f135db5c640 1 -- 172.21.15.59:0/2683758955 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13580b5ae0 con 0x7f13580ccae0 2024-03-06T07:21:58.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.912+0000 7f135db5c640 1 --2- 172.21.15.59:0/2683758955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13580ccae0 0x7f13580ccf40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f13580cff90 tx=0x7f135002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74c3b145e480f2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.912+0000 7f134ffff640 1 -- 172.21.15.59:0/2683758955 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135002cb00 con 0x7f13580ccae0 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.912+0000 7f134ffff640 1 -- 172.21.15.59:0/2683758955 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135002cca0 con 0x7f13580ccae0 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.912+0000 7f134ffff640 1 -- 172.21.15.59:0/2683758955 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350034610 con 0x7f13580ccae0 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.912+0000 7f135f5bf640 1 -- 172.21.15.59:0/2683758955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13580ccae0 msgr2=0x7f13580ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.912+0000 7f135f5bf640 1 --2- 172.21.15.59:0/2683758955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13580ccae0 0x7f13580ccf40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f13580cff90 tx=0x7f135002bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.912+0000 7f135f5bf640 1 -- 172.21.15.59:0/2683758955 shutdown_connections 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.912+0000 7f135f5bf640 1 --2- 172.21.15.59:0/2683758955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13580ccae0 0x7f13580ccf40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.912+0000 7f135f5bf640 1 --2- 172.21.15.59:0/2683758955 >> v2:172.21.15.164:3300/0 conn(0x7f13580d0700 0x7f13580b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.912+0000 7f135f5bf640 1 --2- 172.21.15.59:0/2683758955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13580d0320 0x7f13580b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.912+0000 7f135f5bf640 1 -- 172.21.15.59:0/2683758955 >> 172.21.15.59:0/2683758955 conn(0x7f13580724d0 msgr2=0x7f13580728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.913+0000 7f135f5bf640 1 -- 172.21.15.59:0/2683758955 shutdown_connections 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.913+0000 7f135f5bf640 1 -- 172.21.15.59:0/2683758955 wait complete. 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.914+0000 7f135f5bf640 1 Processor -- start 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.914+0000 7f135f5bf640 1 -- start start 2024-03-06T07:21:58.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.914+0000 7f135f5bf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13580ccae0 0x7f1358156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.914+0000 7f135f5bf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13580d0320 0x7f135809dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.914+0000 7f135f5bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13580d0700 0x7f135814c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.914+0000 7f135f5bf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13580b60a0 con 0x7f13580ccae0 2024-03-06T07:21:58.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.914+0000 7f135f5bf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13580b0170 con 0x7f13580d0320 2024-03-06T07:21:58.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.914+0000 7f135f5bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13580ca790 con 0x7f13580d0700 2024-03-06T07:21:58.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.914+0000 7f135d35b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13580ccae0 0x7f1358156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.914+0000 7f135d35b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13580ccae0 0x7f1358156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57984/0 (socket says 172.21.15.59:57984) 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.914+0000 7f135d35b640 1 -- 172.21.15.59:0/3730648249 learned_addr learned my addr 172.21.15.59:0/3730648249 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f135cb5a640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13580d0320 0x7f135809dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f135db5c640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13580d0700 0x7f135814c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f135d35b640 1 -- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13580d0700 msgr2=0x7f135814c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f135d35b640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13580d0700 0x7f135814c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f135d35b640 1 -- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13580d0320 msgr2=0x7f135809dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f135d35b640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13580d0320 0x7f135809dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f135d35b640 1 -- 172.21.15.59:0/3730648249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135809e230 con 0x7f13580ccae0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f135d35b640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13580ccae0 0x7f1358156cb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f135400e890 tx=0x7f135400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f134dffb640 1 -- 172.21.15.59:0/3730648249 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354017070 con 0x7f13580ccae0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f134dffb640 1 -- 172.21.15.59:0/3730648249 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135400c8e0 con 0x7f13580ccae0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f135f5bf640 1 -- 172.21.15.59:0/3730648249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1358157300 con 0x7f13580ccae0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f134dffb640 1 -- 172.21.15.59:0/3730648249 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135400ccb0 con 0x7f13580ccae0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.915+0000 7f135f5bf640 1 -- 172.21.15.59:0/3730648249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13581578c0 con 0x7f13580ccae0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.916+0000 7f135f5bf640 1 -- 172.21.15.59:0/3730648249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1358157f80 con 0x7f13580ccae0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.917+0000 7f134dffb640 1 -- 172.21.15.59:0/3730648249 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f135401a070 con 0x7f13580ccae0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.917+0000 7f134dffb640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f133c06b4d0 0x7f133c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.917+0000 7f134dffb640 1 -- 172.21.15.59:0/3730648249 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f135408a020 con 0x7f13580ccae0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.917+0000 7f135cb5a640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f133c06b4d0 0x7f133c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.918+0000 7f135cb5a640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f133c06b4d0 0x7f133c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f135815b160 tx=0x7f1348009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:58.921+0000 7f134dffb640 1 -- 172.21.15.59:0/3730648249 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1358157f80 con 0x7f13580ccae0 2024-03-06T07:21:59.098 INFO:tasks.ceph.mds.d.smithi164.stderr:2024-03-06T07:21:59.096+0000 7f97ac61b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:59.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.152+0000 7f135f5bf640 1 -- 172.21.15.59:0/3730648249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f13580cbad0 con 0x7f13580ccae0 2024-03-06T07:21:59.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.154+0000 7f134dffb640 1 -- 172.21.15.59:0/3730648249 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171791 (secure 0 0 0) 0x7f13540588e0 con 0x7f13580ccae0 2024-03-06T07:21:59.158 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:59.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 -- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f133c06b4d0 msgr2=0x7f133c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f133c06b4d0 0x7f133c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f135815b160 tx=0x7f1348009190 comp rx=0 tx=0).stop 2024-03-06T07:21:59.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 -- 172.21.15.59:0/3730648249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13580ccae0 msgr2=0x7f1358156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13580ccae0 0x7f1358156cb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f135400e890 tx=0x7f135400ed50 comp rx=0 tx=0).stop 2024-03-06T07:21:59.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 -- 172.21.15.59:0/3730648249 shutdown_connections 2024-03-06T07:21:59.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f133c06b4d0 0x7f133c06d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13580d0700 0x7f135814c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13580d0320 0x7f135809dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 --2- 172.21.15.59:0/3730648249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13580ccae0 0x7f1358156cb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 -- 172.21.15.59:0/3730648249 >> 172.21.15.59:0/3730648249 conn(0x7f13580724d0 msgr2=0x7f135806b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 -- 172.21.15.59:0/3730648249 shutdown_connections 2024-03-06T07:21:59.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.157+0000 7f135f5bf640 1 -- 172.21.15.59:0/3730648249 wait complete. 2024-03-06T07:21:59.169 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":5,"flags":0,"active_gid":4115,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3140157334},{"type":"v1","addr":"172.21.15.164:6833","nonce":3140157334}]},"active_addr":"172.21.15.164:6833/3140157334","active_change":"2024-03-06T07:21:35.291516+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":882500644}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":2443724476}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":2836719257}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":2500429359}]}]} 2024-03-06T07:21:59.172 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:21:59.173 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:21:59.173 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:59.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.273+0000 7f6a64aa4640 1 Processor -- start 2024-03-06T07:21:59.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.273+0000 7f6a64aa4640 1 -- start start 2024-03-06T07:21:59.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.273+0000 7f6a64aa4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6a60158600 0x7f6a60072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.273+0000 7f6a64aa4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6015eb40 con 0x7f6a6015e3a0 2024-03-06T07:21:59.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.273+0000 7f6a64aa4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6015ee20 con 0x7f6a60162030 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.273+0000 7f6a64aa4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a60161ba0 con 0x7f6a60158600 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.273+0000 7f6a57fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6a6015e3a0 0x7f6a6006d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51130/0 (socket says 172.21.15.59:51130) 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.273+0000 7f6a57fff640 1 -- 172.21.15.59:0/4187901894 learned_addr learned my addr 172.21.15.59:0/4187901894 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.274+0000 7f6a5e59c640 1 --2- 172.21.15.59:0/4187901894 >> v2:172.21.15.164:3300/0 conn(0x7f6a60158600 0x7f6a60072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.274+0000 7f6a5dd9b640 1 -- 172.21.15.59:0/4187901894 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186062409 0 0) 0x7f6a6015eb40 con 0x7f6a6015e3a0 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.274+0000 7f6a5dd9b640 1 -- 172.21.15.59:0/4187901894 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a4c003520 con 0x7f6a6015e3a0 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.274+0000 7f6a5dd9b640 1 -- 172.21.15.59:0/4187901894 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502715692 0 0) 0x7f6a6015ee20 con 0x7f6a60162030 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.274+0000 7f6a5dd9b640 1 -- 172.21.15.59:0/4187901894 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a6015eb40 con 0x7f6a60162030 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.274+0000 7f6a5e59c640 1 -- 172.21.15.59:0/4187901894 >> v1:172.21.15.164:6789/0 conn(0x7f6a60162030 legacy=0x7f6a6006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.274+0000 7f6a5e59c640 1 -- 172.21.15.59:0/4187901894 >> v1:172.21.15.59:6789/0 conn(0x7f6a6015e3a0 legacy=0x7f6a6006d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.274+0000 7f6a5e59c640 1 -- 172.21.15.59:0/4187901894 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6015f100 con 0x7f6a60158600 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.275+0000 7f6a5e59c640 1 --2- 172.21.15.59:0/4187901894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a60158600 0x7f6a60072730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6a500097a0 tx=0x7f6a5002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d587262fe19f0564 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.275+0000 7f6a5dd9b640 1 -- 172.21.15.59:0/4187901894 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50035020 con 0x7f6a60158600 2024-03-06T07:21:59.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.275+0000 7f6a5dd9b640 1 -- 172.21.15.59:0/4187901894 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a50031830 con 0x7f6a60158600 2024-03-06T07:21:59.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.275+0000 7f6a5dd9b640 1 -- 172.21.15.59:0/4187901894 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50031c00 con 0x7f6a60158600 2024-03-06T07:21:59.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.275+0000 7f6a64aa4640 1 -- 172.21.15.59:0/4187901894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a60158600 msgr2=0x7f6a60072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.275+0000 7f6a64aa4640 1 --2- 172.21.15.59:0/4187901894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a60158600 0x7f6a60072730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6a500097a0 tx=0x7f6a5002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.275+0000 7f6a64aa4640 1 -- 172.21.15.59:0/4187901894 shutdown_connections 2024-03-06T07:21:59.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.275+0000 7f6a64aa4640 1 --2- 172.21.15.59:0/4187901894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a60158600 0x7f6a60072730 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.275+0000 7f6a64aa4640 1 -- 172.21.15.59:0/4187901894 >> 172.21.15.59:0/4187901894 conn(0x7f6a6006c0c0 msgr2=0x7f6a6006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.275+0000 7f6a64aa4640 1 -- 172.21.15.59:0/4187901894 shutdown_connections 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.275+0000 7f6a64aa4640 1 -- 172.21.15.59:0/4187901894 wait complete. 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a64aa4640 1 Processor -- start 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a64aa4640 1 -- start start 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a64aa4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a60158600 0x7f6a6014e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a64aa4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6015e3a0 0x7f6a601549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a64aa4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a60162030 0x7f6a60154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a64aa4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6006e080 con 0x7f6a6015e3a0 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a64aa4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6006b6c0 con 0x7f6a60162030 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a64aa4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a60155440 con 0x7f6a60158600 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a57fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6015e3a0 0x7f6a601549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a57fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6015e3a0 0x7f6a601549c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58000/0 (socket says 172.21.15.59:58000) 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a57fff640 1 -- 172.21.15.59:0/1439337875 learned_addr learned my addr 172.21.15.59:0/1439337875 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:59.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a5e59c640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a60158600 0x7f6a6014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.277+0000 7f6a5ed9d640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a60162030 0x7f6a60154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.278+0000 7f6a57fff640 1 -- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a60158600 msgr2=0x7f6a6014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.278+0000 7f6a57fff640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a60158600 0x7f6a6014e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.278+0000 7f6a57fff640 1 -- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a60162030 msgr2=0x7f6a60154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.278+0000 7f6a57fff640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a60162030 0x7f6a60154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.278+0000 7f6a57fff640 1 -- 172.21.15.59:0/1439337875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60157320 con 0x7f6a6015e3a0 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.278+0000 7f6a57fff640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6015e3a0 0x7f6a601549c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800c500 tx=0x7f6a4800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.278+0000 7f6a56ffd640 1 -- 172.21.15.59:0/1439337875 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a48017070 con 0x7f6a6015e3a0 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.278+0000 7f6a56ffd640 1 -- 172.21.15.59:0/1439337875 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a48005ce0 con 0x7f6a6015e3a0 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.278+0000 7f6a64aa4640 1 -- 172.21.15.59:0/1439337875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a60157040 con 0x7f6a6015e3a0 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.278+0000 7f6a56ffd640 1 -- 172.21.15.59:0/1439337875 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a480072f0 con 0x7f6a6015e3a0 2024-03-06T07:21:59.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.278+0000 7f6a64aa4640 1 -- 172.21.15.59:0/1439337875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6015f920 con 0x7f6a6015e3a0 2024-03-06T07:21:59.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.279+0000 7f6a64aa4640 1 -- 172.21.15.59:0/1439337875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6004f330 con 0x7f6a6015e3a0 2024-03-06T07:21:59.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.279+0000 7f6a56ffd640 1 -- 172.21.15.59:0/1439337875 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a4801a070 con 0x7f6a6015e3a0 2024-03-06T07:21:59.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.280+0000 7f6a56ffd640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a4006b4d0 0x7f6a4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.280+0000 7f6a56ffd640 1 -- 172.21.15.59:0/1439337875 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a48089490 con 0x7f6a6015e3a0 2024-03-06T07:21:59.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.280+0000 7f6a5e59c640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a4006b4d0 0x7f6a4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.280+0000 7f6a5e59c640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a4006b4d0 0x7f6a4006d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6a50036000 tx=0x7f6a50008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.284+0000 7f6a56ffd640 1 -- 172.21.15.59:0/1439337875 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a48057d50 con 0x7f6a6015e3a0 2024-03-06T07:21:59.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.467+0000 7f6a64aa4640 1 -- 172.21.15.59:0/1439337875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a600d06f0 con 0x7f6a6015e3a0 2024-03-06T07:21:59.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.468+0000 7f6a56ffd640 1 -- 172.21.15.59:0/1439337875 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f6a4805bdf0 con 0x7f6a6015e3a0 2024-03-06T07:21:59.470 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:59.470 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:59.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 -- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a4006b4d0 msgr2=0x7f6a4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a4006b4d0 0x7f6a4006d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6a50036000 tx=0x7f6a50008030 comp rx=0 tx=0).stop 2024-03-06T07:21:59.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 -- 172.21.15.59:0/1439337875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6015e3a0 msgr2=0x7f6a601549c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6015e3a0 0x7f6a601549c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800c500 tx=0x7f6a4800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 -- 172.21.15.59:0/1439337875 shutdown_connections 2024-03-06T07:21:59.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a4006b4d0 0x7f6a4006d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a60162030 0x7f6a60154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a6015e3a0 0x7f6a601549c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 --2- 172.21.15.59:0/1439337875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a60158600 0x7f6a6014e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 -- 172.21.15.59:0/1439337875 >> 172.21.15.59:0/1439337875 conn(0x7f6a6006c0c0 msgr2=0x7f6a600b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 -- 172.21.15.59:0/1439337875 shutdown_connections 2024-03-06T07:21:59.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.470+0000 7f6a64aa4640 1 -- 172.21.15.59:0/1439337875 wait complete. 2024-03-06T07:21:59.482 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:21:59.483 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:59.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.583+0000 7f7c855d0640 1 Processor -- start 2024-03-06T07:21:59.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.583+0000 7f7c855d0640 1 -- start start 2024-03-06T07:21:59.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.583+0000 7f7c855d0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7c800d0700 0x7f7c800b0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.583+0000 7f7c855d0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7c800b0eb0 0x7f7c800b1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.583+0000 7f7c855d0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8015c270 con 0x7f7c800d0320 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.583+0000 7f7c855d0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c800b04e0 con 0x7f7c800b0eb0 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.583+0000 7f7c855d0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c800d0e90 con 0x7f7c800d0700 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.584+0000 7f7c7f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7c800b0eb0 0x7f7c800b1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.584+0000 7f7c7effd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7c800d0320 0x7f7c800b4f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51140/0 (socket says 172.21.15.59:51140) 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.584+0000 7f7c7effd640 1 -- 172.21.15.59:0/338376596 learned_addr learned my addr 172.21.15.59:0/338376596 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.584+0000 7f7c7e7fc640 1 --2- 172.21.15.59:0/338376596 >> v2:172.21.15.164:3300/0 conn(0x7f7c800d0700 0x7f7c800b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.584+0000 7f7c7f7fe640 1 -- 172.21.15.59:0/338376596 >> v2:172.21.15.164:3300/0 conn(0x7f7c800d0700 msgr2=0x7f7c800b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.584+0000 7f7c7f7fe640 1 --2- 172.21.15.59:0/338376596 >> v2:172.21.15.164:3300/0 conn(0x7f7c800d0700 0x7f7c800b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.584+0000 7f7c7f7fe640 1 -- 172.21.15.59:0/338376596 >> v1:172.21.15.59:6789/0 conn(0x7f7c800d0320 legacy=0x7f7c800b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.584+0000 7f7c7f7fe640 1 -- 172.21.15.59:0/338376596 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c800b5ae0 con 0x7f7c800b0eb0 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.584+0000 7f7c7dffb640 1 -- 172.21.15.59:0/338376596 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916993985 0 0) 0x7f7c8015c270 con 0x7f7c800d0320 2024-03-06T07:21:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c7f7fe640 1 --2- 172.21.15.59:0/338376596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c800b0eb0 0x7f7c800b1360 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7c800cdba0 tx=0x7f7c7802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60c3c70227a346c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c7dffb640 1 -- 172.21.15.59:0/338376596 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78039070 con 0x7f7c800b0eb0 2024-03-06T07:21:59.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c7dffb640 1 -- 172.21.15.59:0/338376596 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7802cd00 con 0x7f7c800b0eb0 2024-03-06T07:21:59.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c7dffb640 1 -- 172.21.15.59:0/338376596 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78034660 con 0x7f7c800b0eb0 2024-03-06T07:21:59.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c855d0640 1 -- 172.21.15.59:0/338376596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c800b0eb0 msgr2=0x7f7c800b1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c855d0640 1 --2- 172.21.15.59:0/338376596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c800b0eb0 0x7f7c800b1360 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7c800cdba0 tx=0x7f7c7802bf50 comp rx=0 tx=0).stop 2024-03-06T07:21:59.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c855d0640 1 -- 172.21.15.59:0/338376596 shutdown_connections 2024-03-06T07:21:59.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c855d0640 1 --2- 172.21.15.59:0/338376596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c800b0eb0 0x7f7c800b1360 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c855d0640 1 --2- 172.21.15.59:0/338376596 >> v2:172.21.15.164:3300/0 conn(0x7f7c800d0700 0x7f7c800b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c855d0640 1 -- 172.21.15.59:0/338376596 >> 172.21.15.59:0/338376596 conn(0x7f7c800724d0 msgr2=0x7f7c800728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c855d0640 1 -- 172.21.15.59:0/338376596 shutdown_connections 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.585+0000 7f7c855d0640 1 -- 172.21.15.59:0/338376596 wait complete. 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.587+0000 7f7c855d0640 1 Processor -- start 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.587+0000 7f7c855d0640 1 -- start start 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.587+0000 7f7c855d0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c800b0eb0 0x7f7c80157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.587+0000 7f7c855d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c800d0320 0x7f7c801e5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.587+0000 7f7c855d0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c800d0700 0x7f7c80157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.587+0000 7f7c855d0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c800b60a0 con 0x7f7c800d0320 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.587+0000 7f7c855d0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c800b0170 con 0x7f7c800d0700 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.587+0000 7f7c855d0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c800ca790 con 0x7f7c800b0eb0 2024-03-06T07:21:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.587+0000 7f7c7e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c800d0320 0x7f7c801e5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.587+0000 7f7c7e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c800d0320 0x7f7c801e5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58012/0 (socket says 172.21.15.59:58012) 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.587+0000 7f7c7e7fc640 1 -- 172.21.15.59:0/3137363095 learned_addr learned my addr 172.21.15.59:0/3137363095 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c7f7fe640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c800d0700 0x7f7c80157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c7effd640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c800b0eb0 0x7f7c80157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c7e7fc640 1 -- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c800b0eb0 msgr2=0x7f7c80157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c7e7fc640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c800b0eb0 0x7f7c80157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c7e7fc640 1 -- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c800d0700 msgr2=0x7f7c80157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c7e7fc640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c800d0700 0x7f7c80157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c7e7fc640 1 -- 172.21.15.59:0/3137363095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c80158130 con 0x7f7c800d0320 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c7e7fc640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c800d0320 0x7f7c801e5eb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400e840 tx=0x7f7c7400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c677fe640 1 -- 172.21.15.59:0/3137363095 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74017070 con 0x7f7c800d0320 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c677fe640 1 -- 172.21.15.59:0/3137363095 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7400c940 con 0x7f7c800d0320 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c855d0640 1 -- 172.21.15.59:0/3137363095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c80156750 con 0x7f7c800d0320 2024-03-06T07:21:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c677fe640 1 -- 172.21.15.59:0/3137363095 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7400cd10 con 0x7f7c800d0320 2024-03-06T07:21:59.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.588+0000 7f7c855d0640 1 -- 172.21.15.59:0/3137363095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c80156d10 con 0x7f7c800d0320 2024-03-06T07:21:59.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.589+0000 7f7c855d0640 1 -- 172.21.15.59:0/3137363095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8004f330 con 0x7f7c800d0320 2024-03-06T07:21:59.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.589+0000 7f7c677fe640 1 -- 172.21.15.59:0/3137363095 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c7401a070 con 0x7f7c800d0320 2024-03-06T07:21:59.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.590+0000 7f7c677fe640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c6c06b4d0 0x7f7c6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.590+0000 7f7c677fe640 1 -- 172.21.15.59:0/3137363095 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c74089fc0 con 0x7f7c800d0320 2024-03-06T07:21:59.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.590+0000 7f7c7effd640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c6c06b4d0 0x7f7c6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.590+0000 7f7c7effd640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c6c06b4d0 0x7f7c6c06d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c70000f60 tx=0x7f7c70007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.594+0000 7f7c677fe640 1 -- 172.21.15.59:0/3137363095 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c740587f0 con 0x7f7c800d0320 2024-03-06T07:21:59.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.775+0000 7f7c855d0640 1 -- 172.21.15.59:0/3137363095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c80067480 con 0x7f7c800d0320 2024-03-06T07:21:59.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.776+0000 7f7c677fe640 1 -- 172.21.15.59:0/3137363095 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f7c7405c890 con 0x7f7c800d0320 2024-03-06T07:21:59.778 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:21:59.779 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:59.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 -- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c6c06b4d0 msgr2=0x7f7c6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c6c06b4d0 0x7f7c6c06d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c70000f60 tx=0x7f7c70007d50 comp rx=0 tx=0).stop 2024-03-06T07:21:59.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 -- 172.21.15.59:0/3137363095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c800d0320 msgr2=0x7f7c801e5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c800d0320 0x7f7c801e5eb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400e840 tx=0x7f7c7400ed00 comp rx=0 tx=0).stop 2024-03-06T07:21:59.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 -- 172.21.15.59:0/3137363095 shutdown_connections 2024-03-06T07:21:59.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c6c06b4d0 0x7f7c6c06d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c800d0700 0x7f7c80157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c800d0320 0x7f7c801e5eb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 --2- 172.21.15.59:0/3137363095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c800b0eb0 0x7f7c80157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 -- 172.21.15.59:0/3137363095 >> 172.21.15.59:0/3137363095 conn(0x7f7c800724d0 msgr2=0x7f7c800b1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 -- 172.21.15.59:0/3137363095 shutdown_connections 2024-03-06T07:21:59.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.778+0000 7f7c855d0640 1 -- 172.21.15.59:0/3137363095 wait complete. 2024-03-06T07:21:59.791 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:21:59.792 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:21:59.792 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:21:59.793 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:21:59.793 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:21:59.793 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:21:59.794 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:21:59.794 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:21:59.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.963+0000 7f5a94d0d640 1 Processor -- start 2024-03-06T07:21:59.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.963+0000 7fa06e16e640 1 Processor -- start 2024-03-06T07:21:59.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.963+0000 7f6dbcd35640 1 Processor -- start 2024-03-06T07:21:59.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.963+0000 7fa06e16e640 1 -- start start 2024-03-06T07:21:59.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.963+0000 7f67eb49a640 1 Processor -- start 2024-03-06T07:21:59.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.964+0000 7f67eb49a640 1 -- start start 2024-03-06T07:21:59.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.963+0000 7f6dbcd35640 1 -- start start 2024-03-06T07:21:59.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.963+0000 7f5a94d0d640 1 -- start start 2024-03-06T07:21:59.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06e16e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa06000aaa0 0x7fa0600df1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06e16e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa06004fa90 0x7fa06004fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06e16e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0600382c0 0x7fa060038750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06e16e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa060050430 con 0x7fa06000aaa0 2024-03-06T07:21:59.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06e16e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0600068a0 con 0x7fa06004fa90 2024-03-06T07:21:59.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06e16e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0600e5d20 con 0x7fa0600382c0 2024-03-06T07:21:59.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f6dbcd35640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b8b0 0x7f6db806d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06d96d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0600382c0 0x7fa060038750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06d96d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0600382c0 0x7fa060038750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43596/0 (socket says 172.21.15.59:43596) 2024-03-06T07:21:59.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06d96d640 1 -- 172.21.15.59:0/2651671285 learned_addr learned my addr 172.21.15.59:0/2651671285 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:59.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06c96b640 1 --2- 172.21.15.59:0/2651671285 >> v2:172.21.15.164:3300/0 conn(0x7fa06004fa90 0x7fa06004fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f6dbcd35640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db815e570 con 0x7f6db806b400 2024-03-06T07:21:59.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06d16c640 1 --2- 172.21.15.59:0/2651671285 >> v2:172.21.15.59:3300/0 conn(0x7fa06000aaa0 0x7fa0600df1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06d96d640 1 -- 172.21.15.59:0/2651671285 >> v2:172.21.15.164:3300/0 conn(0x7fa06004fa90 msgr2=0x7fa06004fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06d96d640 1 --2- 172.21.15.59:0/2651671285 >> v2:172.21.15.164:3300/0 conn(0x7fa06004fa90 0x7fa06004fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7fa06d96d640 1 -- 172.21.15.59:0/2651671285 >> v2:172.21.15.59:3300/0 conn(0x7fa06000aaa0 msgr2=0x7fa0600df1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7fa06d96d640 1 --2- 172.21.15.59:0/2651671285 >> v2:172.21.15.59:3300/0 conn(0x7fa06000aaa0 0x7fa0600df1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7fa06d96d640 1 -- 172.21.15.59:0/2651671285 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa060006dd0 con 0x7fa0600382c0 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f67eb49a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f67dc0546c0 0x7f67dc038170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f67eb49a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67dc04f0b0 0x7f67dc04fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f67eb49a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc005f60 con 0x7f67dc00c0d0 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f67eb49a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67dc0df4d0 con 0x7f67dc0546c0 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f67eb49a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc0e38e0 con 0x7f67dc04f0b0 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f67e9236640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f67dc0546c0 0x7f67dc038170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f67e9236640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f67dc0546c0 0x7f67dc038170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:46678/0 (socket says 172.21.15.59:46678) 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f67e9236640 1 -- 172.21.15.59:0/450506996 learned_addr learned my addr 172.21.15.59:0/450506996 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f67e9a37640 1 --2- 172.21.15.59:0/450506996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67dc04f0b0 0x7f67dc04fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f67e9236640 1 -- 172.21.15.59:0/450506996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67dc04f0b0 msgr2=0x7f67dc04fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f67e9236640 1 --2- 172.21.15.59:0/450506996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67dc04f0b0 0x7f67dc04fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f67e9236640 1 -- 172.21.15.59:0/450506996 >> v1:172.21.15.164:6789/0 conn(0x7f67dc00c0d0 legacy=0x7f67dc0dedc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:21:59.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f67e9236640 1 -- 172.21.15.59:0/450506996 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc006200 con 0x7f67dc0546c0 2024-03-06T07:21:59.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f6db659c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6db81402f0 0x7f6db815db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:48046/0 (socket says 172.21.15.59:48046) 2024-03-06T07:21:59.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f6db659c640 1 -- 172.21.15.59:0/3634615159 learned_addr learned my addr 172.21.15.59:0/3634615159 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:59.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f6db6d9d640 1 --2- 172.21.15.59:0/3634615159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b8b0 0x7f6db806d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f6dbcd35640 1 -- 172.21.15.59:0/3634615159 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db815e850 con 0x7f6db81402f0 2024-03-06T07:21:59.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f6dbcd35640 1 -- 172.21.15.59:0/3634615159 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db806c7d0 con 0x7f6db806b8b0 2024-03-06T07:21:59.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.964+0000 7f5a94d0d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a900b4fe0 0x7f5a900b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.964+0000 7f5a94d0d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 0x7f5a9015be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.964+0000 7f5a94d0d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a9015d510 con 0x7f5a900b4c00 2024-03-06T07:21:59.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.964+0000 7f5a94d0d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a9015e0b0 con 0x7f5a900b4fe0 2024-03-06T07:21:59.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.964+0000 7f5a94d0d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9015d780 con 0x7f5a9015ba50 2024-03-06T07:21:59.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f5a8e59c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5a900b4c00 0x7f5a900b28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:48044/0 (socket says 172.21.15.59:48044) 2024-03-06T07:21:59.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f5a8e59c640 1 -- 172.21.15.59:0/4245748751 learned_addr learned my addr 172.21.15.59:0/4245748751 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:59.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f5a8dd9b640 1 --2- 172.21.15.59:0/4245748751 >> v2:172.21.15.59:3300/0 conn(0x7f5a900b4fe0 0x7f5a900b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.965+0000 7f5a8ed9d640 1 --2- 172.21.15.59:0/4245748751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 0x7f5a9015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f5a8ed9d640 1 -- 172.21.15.59:0/4245748751 >> v2:172.21.15.59:3300/0 conn(0x7f5a900b4fe0 msgr2=0x7f5a900b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f5a8ed9d640 1 --2- 172.21.15.59:0/4245748751 >> v2:172.21.15.59:3300/0 conn(0x7f5a900b4fe0 0x7f5a900b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7fa06d96d640 1 --2- 172.21.15.59:0/2651671285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0600382c0 0x7fa060038750 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa060008560 tx=0x7fa05802b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58a06f47375322ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f3a1e6c9640 1 Processor -- start 2024-03-06T07:21:59.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f3a1e6c9640 1 -- start start 2024-03-06T07:21:59.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f5a8ed9d640 1 -- 172.21.15.59:0/4245748751 >> v1:172.21.15.164:6789/0 conn(0x7f5a900b4c00 legacy=0x7f5a900b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7feabdf16640 1 Processor -- start 2024-03-06T07:21:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7feabdf16640 1 -- start start 2024-03-06T07:21:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f5a8ed9d640 1 -- 172.21.15.59:0/4245748751 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9015da60 con 0x7f5a9015ba50 2024-03-06T07:21:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f5a8dd9b640 1 --2- 172.21.15.59:0/4245748751 >> v2:172.21.15.59:3300/0 conn(0x7f5a900b4fe0 0x7f5a900b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:59.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f6db5d9b640 1 -- 172.21.15.59:0/3634615159 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810261687 0 0) 0x7f6db815e850 con 0x7f6db81402f0 2024-03-06T07:21:59.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f67e9236640 1 --2- 172.21.15.59:0/450506996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67dc0546c0 0x7f67dc038170 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009870 tx=0x7f67e002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aad8bcb04b37574f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.966+0000 7f6db5d9b640 1 -- 172.21.15.59:0/3634615159 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da4003520 con 0x7f6db81402f0 2024-03-06T07:21:59.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f6db5d9b640 1 -- 172.21.15.59:0/3634615159 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 705895216 0 0) 0x7f6da4003520 con 0x7f6db81402f0 2024-03-06T07:21:59.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f6db6d9d640 1 -- 172.21.15.59:0/3634615159 >> v1:172.21.15.164:6789/0 conn(0x7f6db81402f0 legacy=0x7f6db815db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f6db6d9d640 1 -- 172.21.15.59:0/3634615159 >> v1:172.21.15.59:6789/0 conn(0x7f6db806b400 legacy=0x7f6db806cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f6db6d9d640 1 -- 172.21.15.59:0/3634615159 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db806cab0 con 0x7f6db806b8b0 2024-03-06T07:21:59.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7fa05ffff640 1 -- 172.21.15.59:0/2651671285 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa058034070 con 0x7fa0600382c0 2024-03-06T07:21:59.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f3a1e6c9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3a181638b0 0x7f3a180ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f3a1e6c9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3a1806b040 0x7f3a180ada60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f3a1e6c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a1806b4d0 0x7f3a180adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f3a1e6c9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a180b4060 con 0x7f3a181638b0 2024-03-06T07:21:59.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f3a1e6c9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a18165f10 con 0x7f3a1806b040 2024-03-06T07:21:59.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f3a1e6c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a180ae4e0 con 0x7f3a1806b4d0 2024-03-06T07:21:59.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f5a8ed9d640 1 --2- 172.21.15.59:0/4245748751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 0x7f5a9015be90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5a84009800 tx=0x7f5a84034170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50a40b93415a74c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7fa05ffff640 1 -- 172.21.15.59:0/2651671285 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa058030860 con 0x7fa0600382c0 2024-03-06T07:21:59.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f5a8d59a640 1 -- 172.21.15.59:0/4245748751 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84041070 con 0x7f5a9015ba50 2024-03-06T07:21:59.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f5a8d59a640 1 -- 172.21.15.59:0/4245748751 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8403c440 con 0x7f5a9015ba50 2024-03-06T07:21:59.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f5a8d59a640 1 -- 172.21.15.59:0/4245748751 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84041070 con 0x7f5a9015ba50 2024-03-06T07:21:59.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7feabdf16640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feab81621c0 0x7feab80b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7feabdf16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab8165cc0 0x7feab8166100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7feabdf16640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab80cf4e0 con 0x7feab80b29c0 2024-03-06T07:21:59.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7feabdf16640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab8070a10 con 0x7feab81621c0 2024-03-06T07:21:59.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7feabdf16640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab8166640 con 0x7feab8165cc0 2024-03-06T07:21:59.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7feab6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab8165cc0 0x7feab8166100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f67dbfff640 1 -- 172.21.15.59:0/450506996 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0034070 con 0x7f67dc0546c0 2024-03-06T07:21:59.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7feab6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab8165cc0 0x7feab8166100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43630/0 (socket says 172.21.15.59:43630) 2024-03-06T07:21:59.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7feab6ffd640 1 -- 172.21.15.59:0/1219561289 learned_addr learned my addr 172.21.15.59:0/1219561289 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:59.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7feab77fe640 1 --2- 172.21.15.59:0/1219561289 >> v2:172.21.15.59:3300/0 conn(0x7feab81621c0 0x7feab80b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f8b6759e640 1 Processor -- start 2024-03-06T07:21:59.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7fa9b813e640 1 Processor -- start 2024-03-06T07:21:59.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7feab6ffd640 1 -- 172.21.15.59:0/1219561289 >> v2:172.21.15.59:3300/0 conn(0x7feab81621c0 msgr2=0x7feab80b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f8b6759e640 1 -- start start 2024-03-06T07:21:59.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7fa05ffff640 1 -- 172.21.15.59:0/2651671285 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa058030c30 con 0x7fa0600382c0 2024-03-06T07:21:59.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7fa06e16e640 1 -- 172.21.15.59:0/2651671285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0600382c0 msgr2=0x7fa060038750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7fa06e16e640 1 --2- 172.21.15.59:0/2651671285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0600382c0 0x7fa060038750 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa060008560 tx=0x7fa05802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7feab6ffd640 1 --2- 172.21.15.59:0/1219561289 >> v2:172.21.15.59:3300/0 conn(0x7feab81621c0 0x7feab80b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7feab6ffd640 1 -- 172.21.15.59:0/1219561289 >> v1:172.21.15.59:6789/0 conn(0x7feab80b29c0 legacy=0x7feab80b2e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:59.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7fa06e16e640 1 -- 172.21.15.59:0/2651671285 shutdown_connections 2024-03-06T07:21:59.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7fa06e16e640 1 --2- 172.21.15.59:0/2651671285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0600382c0 0x7fa060038750 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7fa06e16e640 1 --2- 172.21.15.59:0/2651671285 >> v2:172.21.15.164:3300/0 conn(0x7fa06004fa90 0x7fa06004fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7fa06e16e640 1 --2- 172.21.15.59:0/2651671285 >> v2:172.21.15.59:3300/0 conn(0x7fa06000aaa0 0x7fa0600df1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7fa06e16e640 1 -- 172.21.15.59:0/2651671285 >> 172.21.15.59:0/2651671285 conn(0x7fa060009fb0 msgr2=0x7fa06000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7f5a94d0d640 1 -- 172.21.15.59:0/4245748751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 msgr2=0x7f5a9015be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7fa06e16e640 1 -- 172.21.15.59:0/2651671285 shutdown_connections 2024-03-06T07:21:59.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7f5a94d0d640 1 --2- 172.21.15.59:0/4245748751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 0x7f5a9015be90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5a84009800 tx=0x7f5a84034170 comp rx=0 tx=0).stop 2024-03-06T07:21:59.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7fa06e16e640 1 -- 172.21.15.59:0/2651671285 wait complete. 2024-03-06T07:21:59.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f67dbfff640 1 -- 172.21.15.59:0/450506996 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e0030840 con 0x7f67dc0546c0 2024-03-06T07:21:59.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f67dbfff640 1 -- 172.21.15.59:0/450506996 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0030c10 con 0x7f67dc0546c0 2024-03-06T07:21:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f67eb49a640 1 -- 172.21.15.59:0/450506996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67dc0546c0 msgr2=0x7f67dc038170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f67eb49a640 1 --2- 172.21.15.59:0/450506996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67dc0546c0 0x7f67dc038170 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009870 tx=0x7f67e002b980 comp rx=0 tx=0).stop 2024-03-06T07:21:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7f5a94d0d640 1 -- 172.21.15.59:0/4245748751 shutdown_connections 2024-03-06T07:21:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7f5a94d0d640 1 --2- 172.21.15.59:0/4245748751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 0x7f5a9015be90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7f5a94d0d640 1 --2- 172.21.15.59:0/4245748751 >> v2:172.21.15.59:3300/0 conn(0x7f5a900b4fe0 0x7f5a900b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7f5a94d0d640 1 -- 172.21.15.59:0/4245748751 >> 172.21.15.59:0/4245748751 conn(0x7f5a9009a9f0 msgr2=0x7f5a9009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7fa9b813e640 1 -- start start 2024-03-06T07:21:59.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7feab6ffd640 1 -- 172.21.15.59:0/1219561289 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab8070c60 con 0x7feab8165cc0 2024-03-06T07:21:59.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7feab77fe640 1 --2- 172.21.15.59:0/1219561289 >> v2:172.21.15.59:3300/0 conn(0x7feab81621c0 0x7feab80b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:59.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f3a17fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3a181638b0 0x7f3a180ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f3a17fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3a181638b0 0x7f3a180ccb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58042/0 (socket says 172.21.15.59:58042) 2024-03-06T07:21:59.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.967+0000 7f3a17fff640 1 -- 172.21.15.59:0/1151797690 learned_addr learned my addr 172.21.15.59:0/1151797690 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:59.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f3a177fe640 1 --2- 172.21.15.59:0/1151797690 >> v2:172.21.15.164:3300/0 conn(0x7f3a1806b040 0x7f3a180ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f3a1cc66640 1 --2- 172.21.15.59:0/1151797690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a1806b4d0 0x7f3a180adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f3a17fff640 1 -- 172.21.15.59:0/1151797690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a1806b4d0 msgr2=0x7f3a180adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f3a17fff640 1 --2- 172.21.15.59:0/1151797690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a1806b4d0 0x7f3a180adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f3a17fff640 1 -- 172.21.15.59:0/1151797690 >> v2:172.21.15.164:3300/0 conn(0x7f3a1806b040 msgr2=0x7f3a180ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f3a17fff640 1 --2- 172.21.15.59:0/1151797690 >> v2:172.21.15.164:3300/0 conn(0x7f3a1806b040 0x7f3a180ada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f3a17fff640 1 -- 172.21.15.59:0/1151797690 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a180cc5b0 con 0x7f3a181638b0 2024-03-06T07:21:59.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7f3a17fff640 1 --2- 172.21.15.59:0/1151797690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a181638b0 0x7f3a180ccb70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3a1806cac0 tx=0x7f3a1003cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b8447d7a50c348 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7f5a94d0d640 1 -- 172.21.15.59:0/4245748751 shutdown_connections 2024-03-06T07:21:59.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7f8b6759e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8b6815a1e0 0x7f8b680b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7f8b6759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b68071960 0x7f8b68071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7f8b6759e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b680cf460 con 0x7f8b6806cd50 2024-03-06T07:21:59.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7f8b6759e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b680722e0 con 0x7f8b6815a1e0 2024-03-06T07:21:59.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7f8b6759e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b680b2800 con 0x7f8b68071960 2024-03-06T07:21:59.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7fa9b813e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa9b015df20 0x7fa9b006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7f8b6659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8b6815a1e0 0x7f8b680b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7f8b6659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8b6815a1e0 0x7f8b680b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58054/0 (socket says 172.21.15.59:58054) 2024-03-06T07:21:59.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7f8b6659c640 1 -- 172.21.15.59:0/1049047652 learned_addr learned my addr 172.21.15.59:0/1049047652 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:21:59.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7fa9b813e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b006b4f0 0x7fa9b006b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7fa9b813e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b015e850 con 0x7fa9b015db40 2024-03-06T07:21:59.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7fa9b813e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b006c7d0 con 0x7fa9b015df20 2024-03-06T07:21:59.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7fa9b813e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b006cab0 con 0x7fa9b006b4f0 2024-03-06T07:21:59.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7fa9b56d9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa9b015df20 0x7fa9b006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7fa9b5eda640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa9b015db40 0x7fa9b0140080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51174/0 (socket says 172.21.15.59:51174) 2024-03-06T07:21:59.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7fa9b5eda640 1 -- 172.21.15.59:0/800357740 learned_addr learned my addr 172.21.15.59:0/800357740 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:21:59.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7fa06e16e640 1 Processor -- start 2024-03-06T07:21:59.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7fa9b66db640 1 --2- 172.21.15.59:0/800357740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b006b4f0 0x7fa9b006b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7f5a94d0d640 1 -- 172.21.15.59:0/4245748751 wait complete. 2024-03-06T07:21:59.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7fa9b66db640 1 -- 172.21.15.59:0/800357740 >> v2:172.21.15.59:3300/0 conn(0x7fa9b015df20 msgr2=0x7fa9b006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7fa9b66db640 1 --2- 172.21.15.59:0/800357740 >> v2:172.21.15.59:3300/0 conn(0x7fa9b015df20 0x7fa9b006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7fa9b66db640 1 -- 172.21.15.59:0/800357740 >> v1:172.21.15.59:6789/0 conn(0x7fa9b015db40 legacy=0x7fa9b0140080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7f3a16ffd640 1 -- 172.21.15.59:0/1151797690 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1003dab0 con 0x7f3a181638b0 2024-03-06T07:21:59.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.970+0000 7f3a16ffd640 1 -- 172.21.15.59:0/1151797690 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1003dc50 con 0x7f3a181638b0 2024-03-06T07:21:59.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/1151797690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a181638b0 msgr2=0x7f3a180ccb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f3a1e6c9640 1 --2- 172.21.15.59:0/1151797690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a181638b0 0x7f3a180ccb70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3a1806cac0 tx=0x7f3a1003cf10 comp rx=0 tx=0).stop 2024-03-06T07:21:59.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7fa9b66db640 1 -- 172.21.15.59:0/800357740 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b0140790 con 0x7fa9b006b4f0 2024-03-06T07:21:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f67eb49a640 1 -- 172.21.15.59:0/450506996 shutdown_connections 2024-03-06T07:21:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f67eb49a640 1 --2- 172.21.15.59:0/450506996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67dc04f0b0 0x7f67dc04fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f67eb49a640 1 --2- 172.21.15.59:0/450506996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67dc0546c0 0x7f67dc038170 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.968+0000 7f67eb49a640 1 -- 172.21.15.59:0/450506996 >> 172.21.15.59:0/450506996 conn(0x7f67dc009fe0 msgr2=0x7f67dc00a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7f67eb49a640 1 -- 172.21.15.59:0/450506996 shutdown_connections 2024-03-06T07:21:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.969+0000 7f67eb49a640 1 -- 172.21.15.59:0/450506996 wait complete. 2024-03-06T07:21:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f67eb49a640 1 Processor -- start 2024-03-06T07:21:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f67eb49a640 1 -- start start 2024-03-06T07:21:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/1151797690 shutdown_connections 2024-03-06T07:21:59.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f3a1e6c9640 1 --2- 172.21.15.59:0/1151797690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a1806b4d0 0x7f3a180adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f3a1e6c9640 1 --2- 172.21.15.59:0/1151797690 >> v2:172.21.15.164:3300/0 conn(0x7f3a1806b040 0x7f3a180ada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f3a1e6c9640 1 --2- 172.21.15.59:0/1151797690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a181638b0 0x7f3a180ccb70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/1151797690 >> 172.21.15.59:0/1151797690 conn(0x7f3a180b3170 msgr2=0x7f3a180b3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7fa06e16e640 1 -- start start 2024-03-06T07:21:59.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7fa9b56d9640 1 --2- 172.21.15.59:0/800357740 >> v2:172.21.15.59:3300/0 conn(0x7fa9b015df20 0x7fa9b006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:59.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f5a94d0d640 1 Processor -- start 2024-03-06T07:21:59.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f5a94d0d640 1 -- start start 2024-03-06T07:21:59.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f6db6d9d640 1 --2- 172.21.15.59:0/3634615159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b8b0 0x7f6db806d6e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6db000bfd0 tx=0x7f6db0030210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2d0d13d0b71773e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f6db5d9b640 1 -- 172.21.15.59:0/3634615159 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db0036070 con 0x7f6db806b8b0 2024-03-06T07:21:59.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f6db5d9b640 1 -- 172.21.15.59:0/3634615159 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db0030d40 con 0x7f6db806b8b0 2024-03-06T07:21:59.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f6db5d9b640 1 -- 172.21.15.59:0/3634615159 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8001670 con 0x7f6db806b8b0 2024-03-06T07:21:59.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7feab6ffd640 1 --2- 172.21.15.59:0/1219561289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab8165cc0 0x7feab8166100 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feaac009870 tx=0x7feaac02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbe29cea05d08bad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f8b6659c640 1 -- 172.21.15.59:0/1049047652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b68071960 msgr2=0x7f8b68071da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:21:59.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f8b6659c640 1 --2- 172.21.15.59:0/1049047652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b68071960 0x7f8b68071da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f8b6659c640 1 -- 172.21.15.59:0/1049047652 >> v1:172.21.15.59:6789/0 conn(0x7f8b6806cd50 legacy=0x7f8b680cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67eb49a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67dc00c0d0 0x7f67dc04a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67eb49a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67dc04f0b0 0x7f67dc0c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67eb49a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67dc0546c0 0x7f67dc0c5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7fa06e16e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa06000aaa0 0x7fa06004e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.971+0000 7f8b6659c640 1 -- 172.21.15.59:0/1049047652 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b68072500 con 0x7f8b6815a1e0 2024-03-06T07:21:59.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/1151797690 shutdown_connections 2024-03-06T07:21:59.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67e9a37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67dc0546c0 0x7f67dc0c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67e9a37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67dc0546c0 0x7f67dc0c5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58080/0 (socket says 172.21.15.59:58080) 2024-03-06T07:21:59.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/1151797690 wait complete. 2024-03-06T07:21:59.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67e9236640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67dc00c0d0 0x7f67dc04a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67e8a35640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67dc04f0b0 0x7f67dc0c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7feab67fc640 1 -- 172.21.15.59:0/1219561289 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac035020 con 0x7feab8165cc0 2024-03-06T07:22:00.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7feab67fc640 1 -- 172.21.15.59:0/1219561289 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac031640 con 0x7feab8165cc0 2024-03-06T07:22:00.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7feab67fc640 1 -- 172.21.15.59:0/1219561289 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac02d040 con 0x7feab8165cc0 2024-03-06T07:22:00.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67e9236640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67dc00c0d0 0x7f67dc04a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43662/0 (socket says 172.21.15.59:43662) 2024-03-06T07:22:00.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67e9a37640 1 -- 172.21.15.59:0/2856292018 learned_addr learned my addr 172.21.15.59:0/2856292018 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67eb49a640 1 -- 172.21.15.59:0/2856292018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc050640 con 0x7f67dc0546c0 2024-03-06T07:22:00.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67eb49a640 1 -- 172.21.15.59:0/2856292018 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc0e3bc0 con 0x7f67dc04f0b0 2024-03-06T07:22:00.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f67eb49a640 1 -- 172.21.15.59:0/2856292018 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc0507c0 con 0x7f67dc00c0d0 2024-03-06T07:22:00.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7fa06e16e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0600382c0 0x7fa0600e54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7fa06e16e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa06004fa90 0x7fa06004aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7fa06e16e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa060052b40 con 0x7fa06000aaa0 2024-03-06T07:22:00.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7fa06e16e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa060007390 con 0x7fa0600382c0 2024-03-06T07:22:00.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7fa06e16e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa060052cc0 con 0x7fa06004fa90 2024-03-06T07:22:00.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7fa06d96d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa06004fa90 0x7fa06004aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7fa06d96d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa06004fa90 0x7fa06004aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43674/0 (socket says 172.21.15.59:43674) 2024-03-06T07:22:00.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7fa06d96d640 1 -- 172.21.15.59:0/3841127584 learned_addr learned my addr 172.21.15.59:0/3841127584 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7f8b6659c640 1 --2- 172.21.15.59:0/1049047652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6815a1e0 0x7f8b680b3530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009870 tx=0x7f8b5802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d83f13d4c97fd45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7fa06d16c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa06000aaa0 0x7fa06004e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a94d0d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a900b4c00 0x7f5a9014c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f6dbcd35640 1 -- 172.21.15.59:0/3634615159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b8b0 msgr2=0x7f6db806d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f6dbcd35640 1 --2- 172.21.15.59:0/3634615159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b8b0 0x7f6db806d6e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6db000bfd0 tx=0x7f6db0030210 comp rx=0 tx=0).stop 2024-03-06T07:22:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7fa06c96b640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0600382c0 0x7fa0600e54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a94d0d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a900b4fe0 0x7f5a9014bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f8b6559a640 1 -- 172.21.15.59:0/1049047652 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b58034070 con 0x7f8b6815a1e0 2024-03-06T07:22:00.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f8b6559a640 1 -- 172.21.15.59:0/1049047652 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b58030860 con 0x7f8b6815a1e0 2024-03-06T07:22:00.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a8e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a900b4c00 0x7f5a9014c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f8b6559a640 1 -- 172.21.15.59:0/1049047652 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b58034070 con 0x7f8b6815a1e0 2024-03-06T07:22:00.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.972+0000 7fa9b66db640 1 --2- 172.21.15.59:0/800357740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b006b4f0 0x7fa9b006b920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0009d90 tx=0x7fa9a002bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13433f2dfac522b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7fa9b4ed8640 1 -- 172.21.15.59:0/800357740 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0039070 con 0x7fa9b006b4f0 2024-03-06T07:22:00.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7fa9b4ed8640 1 -- 172.21.15.59:0/800357740 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a002ccf0 con 0x7fa9b006b4f0 2024-03-06T07:22:00.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7fa9b4ed8640 1 -- 172.21.15.59:0/800357740 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0039070 con 0x7fa9b006b4f0 2024-03-06T07:22:00.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7fa06c96b640 1 -- 172.21.15.59:0/3841127584 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa06004fa90 msgr2=0x7fa06004aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7fa06c96b640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa06004fa90 0x7fa06004aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7fa06c96b640 1 -- 172.21.15.59:0/3841127584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa06000aaa0 msgr2=0x7fa06004e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7fa06c96b640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa06000aaa0 0x7fa06004e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7fa06c96b640 1 -- 172.21.15.59:0/3841127584 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06004d0c0 con 0x7fa0600382c0 2024-03-06T07:22:00.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f6dbcd35640 1 -- 172.21.15.59:0/3634615159 shutdown_connections 2024-03-06T07:22:00.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f6dbcd35640 1 --2- 172.21.15.59:0/3634615159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b8b0 0x7f6db806d6e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f6dbcd35640 1 -- 172.21.15.59:0/3634615159 >> 172.21.15.59:0/3634615159 conn(0x7f6db80b1830 msgr2=0x7f6db80b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a94d0d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 0x7f5a900b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a94d0d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9015f6d0 con 0x7f5a900b4fe0 2024-03-06T07:22:00.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a94d0d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9015f550 con 0x7f5a900b4c00 2024-03-06T07:22:00.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a94d0d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9015c3d0 con 0x7f5a9015ba50 2024-03-06T07:22:00.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a8ed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 0x7f5a900b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a8ed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 0x7f5a900b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43678/0 (socket says 172.21.15.59:43678) 2024-03-06T07:22:00.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a8ed9d640 1 -- 172.21.15.59:0/3536597107 learned_addr learned my addr 172.21.15.59:0/3536597107 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a8dd9b640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a900b4fe0 0x7f5a9014bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f6dbcd35640 1 -- 172.21.15.59:0/3634615159 shutdown_connections 2024-03-06T07:22:00.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7fa06c96b640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0600382c0 0x7fa0600e54e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa068050c30 tx=0x7fa0680683c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f6dbcd35640 1 -- 172.21.15.59:0/3634615159 wait complete. 2024-03-06T07:22:00.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f8b6759e640 1 -- 172.21.15.59:0/1049047652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6815a1e0 msgr2=0x7f8b680b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f8b6759e640 1 --2- 172.21.15.59:0/1049047652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6815a1e0 0x7f8b680b3530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009870 tx=0x7f8b5802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7feabdf16640 1 -- 172.21.15.59:0/1219561289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab8165cc0 msgr2=0x7feab8166100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7feabdf16640 1 --2- 172.21.15.59:0/1219561289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab8165cc0 0x7feab8166100 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feaac009870 tx=0x7feaac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7fa9b813e640 1 -- 172.21.15.59:0/800357740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b006b4f0 msgr2=0x7fa9b006b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7fa9b813e640 1 --2- 172.21.15.59:0/800357740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b006b4f0 0x7fa9b006b920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0009d90 tx=0x7fa9a002bf40 comp rx=0 tx=0).stop 2024-03-06T07:22:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a8ed9d640 1 -- 172.21.15.59:0/3536597107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a900b4c00 msgr2=0x7f5a9014c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7feabdf16640 1 -- 172.21.15.59:0/1219561289 shutdown_connections 2024-03-06T07:22:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7feabdf16640 1 --2- 172.21.15.59:0/1219561289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab8165cc0 0x7feab8166100 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7feabdf16640 1 --2- 172.21.15.59:0/1219561289 >> v2:172.21.15.59:3300/0 conn(0x7feab81621c0 0x7feab80b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f8b6759e640 1 -- 172.21.15.59:0/1049047652 shutdown_connections 2024-03-06T07:22:00.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f8b6759e640 1 --2- 172.21.15.59:0/1049047652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b68071960 0x7f8b68071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f8b6759e640 1 --2- 172.21.15.59:0/1049047652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6815a1e0 0x7f8b680b3530 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f8b6759e640 1 -- 172.21.15.59:0/1049047652 >> 172.21.15.59:0/1049047652 conn(0x7f8b680b16e0 msgr2=0x7f8b680b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7feabdf16640 1 -- 172.21.15.59:0/1219561289 >> 172.21.15.59:0/1219561289 conn(0x7feab80b16c0 msgr2=0x7feab80b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7feabdf16640 1 -- 172.21.15.59:0/1219561289 shutdown_connections 2024-03-06T07:22:00.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f8b6759e640 1 -- 172.21.15.59:0/1049047652 shutdown_connections 2024-03-06T07:22:00.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7feabdf16640 1 -- 172.21.15.59:0/1219561289 wait complete. 2024-03-06T07:22:00.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a8ed9d640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a900b4c00 0x7f5a9014c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a8ed9d640 1 -- 172.21.15.59:0/3536597107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a900b4fe0 msgr2=0x7f5a9014bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f3a1e6c9640 1 Processor -- start 2024-03-06T07:22:00.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f67e9a37640 1 -- 172.21.15.59:0/2856292018 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67dc00c0d0 msgr2=0x7f67dc04a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a8ed9d640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a900b4fe0 0x7f5a9014bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a8ed9d640 1 -- 172.21.15.59:0/3536597107 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9015af00 con 0x7f5a9015ba50 2024-03-06T07:22:00.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f5a8dd9b640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a900b4fe0 0x7f5a9014bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:00.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f5a8ed9d640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 0x7f5a900b0d00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5a840341b0 tx=0x7f5a840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f8b6759e640 1 -- 172.21.15.59:0/1049047652 wait complete. 2024-03-06T07:22:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7fa9b813e640 1 -- 172.21.15.59:0/800357740 shutdown_connections 2024-03-06T07:22:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7fa9b813e640 1 --2- 172.21.15.59:0/800357740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b006b4f0 0x7fa9b006b920 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7fa9b813e640 1 --2- 172.21.15.59:0/800357740 >> v2:172.21.15.59:3300/0 conn(0x7fa9b015df20 0x7fa9b006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7fa9b813e640 1 -- 172.21.15.59:0/800357740 >> 172.21.15.59:0/800357740 conn(0x7fa9b00b1830 msgr2=0x7fa9b00b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f67e9a37640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67dc00c0d0 0x7f67dc04a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f67e9a37640 1 -- 172.21.15.59:0/2856292018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67dc04f0b0 msgr2=0x7f67dc0c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7fa9b813e640 1 -- 172.21.15.59:0/800357740 shutdown_connections 2024-03-06T07:22:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f67e9a37640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67dc04f0b0 0x7f67dc0c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f67e9a37640 1 -- 172.21.15.59:0/2856292018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc04bd60 con 0x7f67dc0546c0 2024-03-06T07:22:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7fa05dffb640 1 -- 172.21.15.59:0/3841127584 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068073660 con 0x7fa0600382c0 2024-03-06T07:22:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7fa06e16e640 1 -- 172.21.15.59:0/3841127584 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06004d3a0 con 0x7fa0600382c0 2024-03-06T07:22:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7fa06e16e640 1 -- 172.21.15.59:0/3841127584 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06004d960 con 0x7fa0600382c0 2024-03-06T07:22:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.975+0000 7fa05dffb640 1 -- 172.21.15.59:0/3841127584 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa068094440 con 0x7fa0600382c0 2024-03-06T07:22:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.975+0000 7fa05dffb640 1 -- 172.21.15.59:0/3841127584 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068094830 con 0x7fa0600382c0 2024-03-06T07:22:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.973+0000 7f67e9236640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67dc00c0d0 0x7f67dc04a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f67e9a37640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67dc0546c0 0x7f67dc0c5260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f67e40505a0 tx=0x7f67e40683c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f67e8a35640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67dc04f0b0 0x7f67dc0c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f5a7effd640 1 -- 172.21.15.59:0/3536597107 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8404b070 con 0x7f5a9015ba50 2024-03-06T07:22:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f5a7effd640 1 -- 172.21.15.59:0/3536597107 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8403a710 con 0x7f5a9015ba50 2024-03-06T07:22:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f5a7effd640 1 -- 172.21.15.59:0/3536597107 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84041070 con 0x7f5a9015ba50 2024-03-06T07:22:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f5a94d0d640 1 -- 172.21.15.59:0/3536597107 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a901594a0 con 0x7f5a9015ba50 2024-03-06T07:22:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f3a1e6c9640 1 -- start start 2024-03-06T07:22:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.975+0000 7f3a1e6c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a1806b040 0x7f3a181f0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7f5a94d0d640 1 -- 172.21.15.59:0/3536597107 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a90159a60 con 0x7f5a9015ba50 2024-03-06T07:22:00.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.975+0000 7f5a94d0d640 1 -- 172.21.15.59:0/3536597107 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5a74001030 con 0x7f5a9015ba50 2024-03-06T07:22:00.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7fa06e16e640 1 -- 172.21.15.59:0/3841127584 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa060005390 con 0x7fa0600382c0 2024-03-06T07:22:00.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f67d9ffb640 1 -- 172.21.15.59:0/2856292018 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e4073660 con 0x7f67dc0546c0 2024-03-06T07:22:00.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f67d9ffb640 1 -- 172.21.15.59:0/2856292018 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e4083480 con 0x7f67dc0546c0 2024-03-06T07:22:00.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a1e6c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a1806b4d0 0x7f3a180cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a1e6c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a181638b0 0x7f3a180ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a1e6c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a180cd910 con 0x7f3a1806b040 2024-03-06T07:22:00.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a1e6c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a180cd630 con 0x7f3a181638b0 2024-03-06T07:22:00.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a1e6c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a18163c90 con 0x7f3a1806b4d0 2024-03-06T07:22:00.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f6dbcd35640 1 Processor -- start 2024-03-06T07:22:00.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a17fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a1806b040 0x7f3a181f0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a17fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a1806b040 0x7f3a181f0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58098/0 (socket says 172.21.15.59:58098) 2024-03-06T07:22:00.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a17fff640 1 -- 172.21.15.59:0/2886833197 learned_addr learned my addr 172.21.15.59:0/2886833197 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7feabdf16640 1 Processor -- start 2024-03-06T07:22:00.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7feabdf16640 1 -- start start 2024-03-06T07:22:00.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f6dbcd35640 1 -- start start 2024-03-06T07:22:00.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.974+0000 7fa9b813e640 1 -- 172.21.15.59:0/800357740 wait complete. 2024-03-06T07:22:00.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7fa9b813e640 1 Processor -- start 2024-03-06T07:22:00.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a17fff640 1 -- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a1806b4d0 msgr2=0x7f3a180cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a17fff640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a1806b4d0 0x7f3a180cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a17fff640 1 -- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a181638b0 msgr2=0x7f3a180ce4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:00.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a17fff640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a181638b0 0x7f3a180ce4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f3a17fff640 1 -- 172.21.15.59:0/2886833197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a180d0e30 con 0x7f3a1806b040 2024-03-06T07:22:00.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f3a17fff640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a1806b040 0x7f3a181f0020 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009f40 tx=0x7f3a10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f6dbcd35640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b400 0x7f6db813ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f5a7effd640 1 -- 172.21.15.59:0/3536597107 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a8403ab00 con 0x7f5a9015ba50 2024-03-06T07:22:00.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7feabdf16640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab80b29c0 0x7feab814d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7feabdf16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab81621c0 0x7feab815e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7feabdf16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab8165cc0 0x7feab81fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7feabdf16640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab81669a0 con 0x7feab8165cc0 2024-03-06T07:22:00.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7feabdf16640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab81625a0 con 0x7feab80b29c0 2024-03-06T07:22:00.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7feabdf16640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab8166b20 con 0x7feab81621c0 2024-03-06T07:22:00.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7feab6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab81621c0 0x7feab815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f6dbcd35640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6db806b8b0 0x7f6db81ec5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f8b6759e640 1 Processor -- start 2024-03-06T07:22:00.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f8b6759e640 1 -- start start 2024-03-06T07:22:00.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b6759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6806cd50 0x7f8b680cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b6759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b68071960 0x7f8b680c4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b6659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6806cd50 0x7f8b680cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b6759e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b6815a1e0 0x7f8b680ad570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b6659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6806cd50 0x7f8b680cb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58104/0 (socket says 172.21.15.59:58104) 2024-03-06T07:22:00.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b6759e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b680b3010 con 0x7f8b6806cd50 2024-03-06T07:22:00.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b6759e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b680b2d30 con 0x7f8b6815a1e0 2024-03-06T07:22:00.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b6759e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6806d1e0 con 0x7f8b68071960 2024-03-06T07:22:00.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b65d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b68071960 0x7f8b680c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b65d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b68071960 0x7f8b680c4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43686/0 (socket says 172.21.15.59:43686) 2024-03-06T07:22:00.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b65d9b640 1 -- 172.21.15.59:0/3015557111 learned_addr learned my addr 172.21.15.59:0/3015557111 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f8b66d9d640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b6815a1e0 0x7f8b680ad570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7feab6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab81621c0 0x7feab815e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43684/0 (socket says 172.21.15.59:43684) 2024-03-06T07:22:00.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7feab6ffd640 1 -- 172.21.15.59:0/3102295821 learned_addr learned my addr 172.21.15.59:0/3102295821 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7feab7fff640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab8165cc0 0x7feab81fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7feab77fe640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab80b29c0 0x7feab814d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7f6db659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b400 0x7f6db813ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7fa05dffb640 1 -- 172.21.15.59:0/3841127584 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0680949d0 con 0x7fa0600382c0 2024-03-06T07:22:00.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.977+0000 7fa9b813e640 1 -- start start 2024-03-06T07:22:00.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7fa9b813e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9b006b4f0 0x7fa9b0154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7fa9b813e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9b015db40 0x7fa9b015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7fa9b813e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b015df20 0x7fa9b015bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7fa9b813e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b006d150 con 0x7fa9b015db40 2024-03-06T07:22:00.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7fa9b813e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b006cfd0 con 0x7fa9b006b4f0 2024-03-06T07:22:00.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7fa9b813e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b006d2d0 con 0x7fa9b015df20 2024-03-06T07:22:00.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7feab77fe640 1 -- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab81621c0 msgr2=0x7feab815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7feab77fe640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab81621c0 0x7feab815e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7feab77fe640 1 -- 172.21.15.59:0/3102295821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab8165cc0 msgr2=0x7feab81fc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7feab77fe640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feab8165cc0 0x7feab81fc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7feab77fe640 1 -- 172.21.15.59:0/3102295821 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab815edb0 con 0x7feab80b29c0 2024-03-06T07:22:00.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f8b66d9d640 1 -- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b68071960 msgr2=0x7f8b680c4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f8b66d9d640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b68071960 0x7f8b680c4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f8b66d9d640 1 -- 172.21.15.59:0/3015557111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6806cd50 msgr2=0x7f8b680cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f8b66d9d640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b6806cd50 0x7f8b680cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f8b66d9d640 1 -- 172.21.15.59:0/3015557111 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b682016c0 con 0x7f8b6815a1e0 2024-03-06T07:22:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f6db659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b400 0x7f6db813ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43682/0 (socket says 172.21.15.59:43682) 2024-03-06T07:22:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f6db659c640 1 -- 172.21.15.59:0/2060485611 learned_addr learned my addr 172.21.15.59:0/2060485611 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f5a7effd640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7006b5a0 0x7f5a7006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f6daffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6db806b8b0 0x7f6db81ec5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f5a7effd640 1 -- 172.21.15.59:0/3536597107 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a840b5270 con 0x7f5a9015ba50 2024-03-06T07:22:00.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f5a7effd640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f5a70073820 0x7f5a70075c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f5a7effd640 1 -- 172.21.15.59:0/3536597107 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a84001d30 con 0x7f5a70073820 2024-03-06T07:22:00.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f5a7effd640 1 -- 172.21.15.59:0/3536597107 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a840b5660 con 0x7f5a9015ba50 2024-03-06T07:22:00.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f3a14ff9640 1 -- 172.21.15.59:0/2886833197 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10055020 con 0x7f3a1806b040 2024-03-06T07:22:00.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f3a14ff9640 1 -- 172.21.15.59:0/2886833197 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a10051690 con 0x7f3a1806b040 2024-03-06T07:22:00.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f3a14ff9640 1 -- 172.21.15.59:0/2886833197 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10051a80 con 0x7f3a1806b040 2024-03-06T07:22:00.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f6dbcd35640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6db81402f0 0x7f6db81ecb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f8b66d9d640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b6815a1e0 0x7f8b680ad570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000c730 tx=0x7f8b6000cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db80ce590 con 0x7f6db806b8b0 2024-03-06T07:22:00.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7fa9b56d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9b015db40 0x7fa9b015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db815e250 con 0x7f6db81402f0 2024-03-06T07:22:00.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db80ce710 con 0x7f6db806b400 2024-03-06T07:22:00.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7fa9b56d9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9b015db40 0x7fa9b015cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58126/0 (socket says 172.21.15.59:58126) 2024-03-06T07:22:00.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.978+0000 7f5a8e59c640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7006b5a0 0x7f5a7006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fa9b5eda640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9b006b4f0 0x7fa9b0154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fa9b66db640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b015df20 0x7fa9b015bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fa9b66db640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b015df20 0x7fa9b015bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43696/0 (socket says 172.21.15.59:43696) 2024-03-06T07:22:00.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fa9b66db640 1 -- 172.21.15.59:0/961933980 learned_addr learned my addr 172.21.15.59:0/961933980 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f6db6d9d640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6db81402f0 0x7f6db81ecb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f67d9ffb640 1 -- 172.21.15.59:0/2856292018 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e4083830 con 0x7f67dc0546c0 2024-03-06T07:22:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.976+0000 7f67eb49a640 1 -- 172.21.15.59:0/2856292018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67dc04c040 con 0x7f67dc0546c0 2024-03-06T07:22:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/2886833197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a180d0b50 con 0x7f3a1806b040 2024-03-06T07:22:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/2886833197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a180ceea0 con 0x7f3a1806b040 2024-03-06T07:22:00.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f8b56ffd640 1 -- 172.21.15.59:0/3015557111 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b60016020 con 0x7f8b6815a1e0 2024-03-06T07:22:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fa9b66db640 1 -- 172.21.15.59:0/961933980 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9b006b4f0 msgr2=0x7fa9b0154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fa9b66db640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9b006b4f0 0x7fa9b0154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fa9b66db640 1 -- 172.21.15.59:0/961933980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9b015db40 msgr2=0x7fa9b015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7feab77fe640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab80b29c0 0x7feab814d870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feab000efc0 tx=0x7feab000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f6db659c640 1 -- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6db81402f0 msgr2=0x7f6db81ecb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f6db659c640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6db81402f0 0x7f6db81ecb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f6db659c640 1 -- 172.21.15.59:0/2060485611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6db806b8b0 msgr2=0x7f6db81ec5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f6db659c640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6db806b8b0 0x7f6db81ec5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f6db659c640 1 -- 172.21.15.59:0/2060485611 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db81ed060 con 0x7f6db806b400 2024-03-06T07:22:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f6daffff640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6db806b8b0 0x7f6db81ec5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f5a8e59c640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7006b5a0 0x7f5a7006da60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5a88005c00 tx=0x7f5a88009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fa9b66db640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9b015db40 0x7fa9b015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fa9b66db640 1 -- 172.21.15.59:0/961933980 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b013d7f0 con 0x7fa9b015df20 2024-03-06T07:22:00.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f5a8dd9b640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f5a70073820 0x7f5a70075c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7f67eb49a640 1 -- 172.21.15.59:0/2856292018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67dc04c600 con 0x7f67dc0546c0 2024-03-06T07:22:00.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7fa05dffb640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa04c06b5d0 0x7fa04c06da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7fa05dffb640 1 -- 172.21.15.59:0/3841127584 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa068105050 con 0x7fa0600382c0 2024-03-06T07:22:00.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7fa05dffb640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7fa04c073850 0x7fa04c075c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fea9ffff640 1 -- 172.21.15.59:0/3102295821 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0018020 con 0x7feab80b29c0 2024-03-06T07:22:00.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fea9ffff640 1 -- 172.21.15.59:0/3102295821 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab0014680 con 0x7feab80b29c0 2024-03-06T07:22:00.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fea9ffff640 1 -- 172.21.15.59:0/3102295821 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0014a70 con 0x7feab80b29c0 2024-03-06T07:22:00.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7feabdf16640 1 -- 172.21.15.59:0/3102295821 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab8164350 con 0x7feab80b29c0 2024-03-06T07:22:00.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7feabdf16640 1 -- 172.21.15.59:0/3102295821 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab8164910 con 0x7feab80b29c0 2024-03-06T07:22:00.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7fa05dffb640 1 -- 172.21.15.59:0/3841127584 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa04c076320 con 0x7fa04c073850 2024-03-06T07:22:00.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7fa05dffb640 1 -- 172.21.15.59:0/3841127584 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa06810c050 con 0x7fa0600382c0 2024-03-06T07:22:00.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f8b6759e640 1 -- 172.21.15.59:0/3015557111 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b680aabc0 con 0x7f8b6815a1e0 2024-03-06T07:22:00.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7fa06d96d640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7fa04c073850 0x7fa04c075c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7fa06d16c640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa04c06b5d0 0x7fa04c06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f8b6759e640 1 -- 172.21.15.59:0/3015557111 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b680ab180 con 0x7f8b6815a1e0 2024-03-06T07:22:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7f8b56ffd640 1 -- 172.21.15.59:0/3015557111 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b60006d50 con 0x7f8b6815a1e0 2024-03-06T07:22:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7f8b56ffd640 1 -- 172.21.15.59:0/3015557111 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b60003bb0 con 0x7f8b6815a1e0 2024-03-06T07:22:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7f8b54ff9640 1 -- 172.21.15.59:0/3015557111 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8b40001030 con 0x7f8b6815a1e0 2024-03-06T07:22:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.979+0000 7fa9b66db640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b015df20 0x7fa9b015bf70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0009eb0 tx=0x7fa9a0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7fa9ae7fc640 1 -- 172.21.15.59:0/961933980 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0039070 con 0x7fa9b015df20 2024-03-06T07:22:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7fa9b813e640 1 -- 172.21.15.59:0/961933980 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b013dad0 con 0x7fa9b015df20 2024-03-06T07:22:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.980+0000 7fa9b813e640 1 -- 172.21.15.59:0/961933980 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b013e090 con 0x7fa9b015df20 2024-03-06T07:22:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.981+0000 7f67d9ffb640 1 -- 172.21.15.59:0/2856292018 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67e40839d0 con 0x7f67dc0546c0 2024-03-06T07:22:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.981+0000 7fa9ae7fc640 1 -- 172.21.15.59:0/961933980 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a0045070 con 0x7fa9b015df20 2024-03-06T07:22:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.981+0000 7fa9ae7fc640 1 -- 172.21.15.59:0/961933980 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a003e870 con 0x7fa9b015df20 2024-03-06T07:22:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.981+0000 7f67c77fe640 1 -- 172.21.15.59:0/2856292018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f67dc007ff0 con 0x7f67dc0546c0 2024-03-06T07:22:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.981+0000 7fea9dffb640 1 -- 172.21.15.59:0/3102295821 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feab80b5c90 con 0x7feab80b29c0 2024-03-06T07:22:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.981+0000 7fa9b813e640 1 -- 172.21.15.59:0/961933980 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa990001030 con 0x7fa9b015df20 2024-03-06T07:22:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.981+0000 7fa06d96d640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7fa04c073850 0x7fa04c075c70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.981+0000 7f6db659c640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b400 0x7f6db813ec90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6da800c750 tx=0x7f6da800cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f3a14ff9640 1 -- 172.21.15.59:0/2886833197 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a10058020 con 0x7f3a1806b040 2024-03-06T07:22:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f67d9ffb640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67c806b520 0x7f67c806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f67d9ffb640 1 -- 172.21.15.59:0/2856292018 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67e40f5d90 con 0x7f67dc0546c0 2024-03-06T07:22:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/2886833197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a180b6360 con 0x7f3a1806b040 2024-03-06T07:22:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f3a14ff9640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a0406b5a0 0x7f3a0406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f3a14ff9640 1 -- 172.21.15.59:0/2886833197 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a100bd770 con 0x7f3a1806b040 2024-03-06T07:22:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f3a14ff9640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f3a04073820 0x7f3a04075c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f3a14ff9640 1 -- 172.21.15.59:0/2886833197 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a040762f0 con 0x7f3a04073820 2024-03-06T07:22:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f3a177fe640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a0406b5a0 0x7f3a0406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f3a14ff9640 1 -- 172.21.15.59:0/2886833197 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a100c4050 con 0x7f3a1806b040 2024-03-06T07:22:00.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7fea9ffff640 1 -- 172.21.15.59:0/3102295821 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feab0005d10 con 0x7feab80b29c0 2024-03-06T07:22:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f6daeffd640 1 -- 172.21.15.59:0/2060485611 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8018070 con 0x7f6db806b400 2024-03-06T07:22:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f6daeffd640 1 -- 172.21.15.59:0/2060485611 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da80072d0 con 0x7f6db806b400 2024-03-06T07:22:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f6daeffd640 1 -- 172.21.15.59:0/2060485611 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da80047e0 con 0x7f6db806b400 2024-03-06T07:22:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f67d9ffb640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f67c80737a0 0x7f67c8075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f67d9ffb640 1 -- 172.21.15.59:0/2856292018 --> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f67c8076270 con 0x7f67c80737a0 2024-03-06T07:22:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f8b56ffd640 1 -- 172.21.15.59:0/3015557111 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b600052e0 con 0x7f8b6815a1e0 2024-03-06T07:22:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f8b56ffd640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b4806b5a0 0x7f8b4806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f8b6659c640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b4806b5a0 0x7f8b4806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.982+0000 7f8b56ffd640 1 -- 172.21.15.59:0/3015557111 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b60089e70 con 0x7f8b6815a1e0 2024-03-06T07:22:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f8b56ffd640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x7f8b480737c0 0x7f8b48075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f67d9ffb640 1 -- 172.21.15.59:0/2856292018 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67e40f9060 con 0x7f67dc0546c0 2024-03-06T07:22:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f67e9236640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67c806b520 0x7f67c806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f8b6659c640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b4806b5a0 0x7f8b4806da60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8b58036000 tx=0x7f8b5803b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f8b56ffd640 1 -- 172.21.15.59:0/3015557111 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b60006d50 con 0x7f8b480737c0 2024-03-06T07:22:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f8b56ffd640 1 -- 172.21.15.59:0/3015557111 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b6008c4d0 con 0x7f8b6815a1e0 2024-03-06T07:22:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f67e8a35640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f67c80737a0 0x7f67c8075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7fa9ae7fc640 1 -- 172.21.15.59:0/961933980 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9a003ea10 con 0x7fa9b015df20 2024-03-06T07:22:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7fa05dffb640 1 -- 172.21.15.59:0/3841127584 <== osd.1 v2:172.21.15.59:6816/1773892168 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa04c076320 con 0x7fa04c073850 2024-03-06T07:22:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f8b65d9b640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x7f8b480737c0 0x7f8b48075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db81ee3b0 con 0x7f6db806b400 2024-03-06T07:22:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7fea9ffff640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feaa406b5a0 0x7feaa406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f3a1cc66640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f3a04073820 0x7f3a04075c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db81ed5f0 con 0x7f6db806b400 2024-03-06T07:22:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7fea9ffff640 1 -- 172.21.15.59:0/3102295821 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feab008ba10 con 0x7feab80b29c0 2024-03-06T07:22:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7fa9ae7fc640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa99c06b4d0 0x7fa99c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7fea9ffff640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7feaa40737c0 0x7feaa4075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7feab6ffd640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feaa406b5a0 0x7feaa406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7fa06d16c640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa04c06b5d0 0x7fa04c06da90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa06400a9b0 tx=0x7fa064008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f67e8a35640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f67c80737a0 0x7f67c8075bc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f5a8dd9b640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f5a70073820 0x7f5a70075c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7f3a177fe640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a0406b5a0 0x7f3a0406da60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a08006a90 tx=0x7f3a08006a20 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7f3a1cc66640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f3a04073820 0x7f3a04075c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7f8b65d9b640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x7f8b480737c0 0x7f8b48075be0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7fa9ae7fc640 1 -- 172.21.15.59:0/961933980 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9a00ad850 con 0x7fa9b015df20 2024-03-06T07:22:00.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7fa9ae7fc640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7fa99c073670 0x7fa99c075a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7fa9ae7fc640 1 -- 172.21.15.59:0/961933980 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9a000aea0 con 0x7fa99c073670 2024-03-06T07:22:00.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7fa9ae7fc640 1 -- 172.21.15.59:0/961933980 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9a00adc00 con 0x7fa9b015df20 2024-03-06T07:22:00.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7fa9b56d9640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7fa99c073670 0x7fa99c075a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7fea9ffff640 1 -- 172.21.15.59:0/3102295821 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feaa4076290 con 0x7feaa40737c0 2024-03-06T07:22:00.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.983+0000 7fea9ffff640 1 -- 172.21.15.59:0/3102295821 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7feab008be00 con 0x7feab80b29c0 2024-03-06T07:22:00.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7feab7fff640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7feaa40737c0 0x7feaa4075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7f67e9236640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67c806b520 0x7f67c806d9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67e0036000 tx=0x7f67e003b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7fa9b5eda640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa99c06b4d0 0x7fa99c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7fa9b56d9640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7fa99c073670 0x7fa99c075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6db806e6f0 con 0x7f6db806b400 2024-03-06T07:22:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.984+0000 7f67d9ffb640 1 -- 172.21.15.59:0/2856292018 <== osd.2 v2:172.21.15.59:6800/1020996386 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f67c8076270 con 0x7f67c80737a0 2024-03-06T07:22:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.985+0000 7f3a14ff9640 1 -- 172.21.15.59:0/2886833197 <== osd.4 v2:172.21.15.164:6818/102497754 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3a040762f0 con 0x7f3a04073820 2024-03-06T07:22:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.985+0000 7feab6ffd640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feaa406b5a0 0x7feaa406da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feaac02b930 tx=0x7feaac03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.985+0000 7feab7fff640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7feaa40737c0 0x7feaa4075be0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.985+0000 7f8b56ffd640 1 -- 172.21.15.59:0/3015557111 <== osd.7 v2:172.21.15.164:6808/3612340295 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8b60006d50 con 0x7f8b480737c0 2024-03-06T07:22:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.985+0000 7f6daeffd640 1 -- 172.21.15.59:0/2060485611 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6da801b070 con 0x7f6db806b400 2024-03-06T07:22:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.985+0000 7f5a7effd640 1 -- 172.21.15.59:0/3536597107 <== osd.3 v2:172.21.15.59:6808/3602142643 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5a84001d30 con 0x7f5a70073820 2024-03-06T07:22:00.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.985+0000 7fa9b5eda640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa99c06b4d0 0x7fa99c06d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8007fd0 tx=0x7fa9a8007e80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.985+0000 7fa9ae7fc640 1 -- 172.21.15.59:0/961933980 <== osd.0 v2:172.21.15.59:6824/841983189 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa9a000aea0 con 0x7fa99c073670 2024-03-06T07:22:00.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.986+0000 7fea9ffff640 1 -- 172.21.15.59:0/3102295821 <== osd.5 v2:172.21.15.164:6800/2889679248 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7feaa4076290 con 0x7feaa40737c0 2024-03-06T07:22:00.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.986+0000 7f6daeffd640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d9806b520 0x7f6d9806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.986+0000 7f6daeffd640 1 -- 172.21.15.59:0/2060485611 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6da80897d0 con 0x7f6db806b400 2024-03-06T07:22:00.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.986+0000 7f6daffff640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d9806b520 0x7f6d9806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.986+0000 7f6daeffd640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f6d980737a0 0x7f6d98075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.986+0000 7f6daeffd640 1 -- 172.21.15.59:0/2060485611 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d98076270 con 0x7f6d980737a0 2024-03-06T07:22:00.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.986+0000 7f6daeffd640 1 -- 172.21.15.59:0/2060485611 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6da8090050 con 0x7f6db806b400 2024-03-06T07:22:00.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.986+0000 7f6db6d9d640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f6d980737a0 0x7f6d98075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.987+0000 7f6daffff640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d9806b520 0x7f6d9806d9e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6da0001ad0 tx=0x7f6da0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.987+0000 7f6db6d9d640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f6d980737a0 0x7f6d98075bc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:21:59.988+0000 7f6daeffd640 1 -- 172.21.15.59:0/2060485611 <== osd.6 v2:172.21.15.164:6816/1594998822 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6d98076270 con 0x7f6d980737a0 2024-03-06T07:22:00.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.018+0000 7fa06e16e640 1 -- 172.21.15.59:0/3841127584 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa060054190 con 0x7fa04c073850 2024-03-06T07:22:00.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.018+0000 7fa05dffb640 1 -- 172.21.15.59:0/3841127584 <== osd.1 v2:172.21.15.59:6816/1773892168 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa060054190 con 0x7fa04c073850 2024-03-06T07:22:00.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.018+0000 7f67c77fe640 1 -- 172.21.15.59:0/2856292018 --> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f67dc0053c0 con 0x7f67c80737a0 2024-03-06T07:22:00.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.018+0000 7f5a94d0d640 1 -- 172.21.15.59:0/3536597107 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5a74002dd0 con 0x7f5a70073820 2024-03-06T07:22:00.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.018+0000 7f5a7effd640 1 -- 172.21.15.59:0/3536597107 <== osd.3 v2:172.21.15.59:6808/3602142643 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5a74002dd0 con 0x7f5a70073820 2024-03-06T07:22:00.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/2886833197 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a180b6360 con 0x7f3a04073820 2024-03-06T07:22:00.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f5a94d0d640 1 -- 172.21.15.59:0/3536597107 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f5a70073820 msgr2=0x7f5a70075c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f5a94d0d640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f5a70073820 0x7f5a70075c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f67d9ffb640 1 -- 172.21.15.59:0/2856292018 <== osd.2 v2:172.21.15.59:6800/1020996386 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f67dc0053c0 con 0x7f67c80737a0 2024-03-06T07:22:00.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f5a94d0d640 1 -- 172.21.15.59:0/3536597107 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7006b5a0 msgr2=0x7f5a7006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7fa06e16e640 1 -- 172.21.15.59:0/3841127584 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7fa04c073850 msgr2=0x7fa04c075c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7fa06e16e640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7fa04c073850 0x7fa04c075c70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f5a94d0d640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7006b5a0 0x7f5a7006da60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5a88005c00 tx=0x7f5a88009040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f5a94d0d640 1 -- 172.21.15.59:0/3536597107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 msgr2=0x7f5a900b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f5a94d0d640 1 --2- 172.21.15.59:0/3536597107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9015ba50 0x7f5a900b0d00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5a840341b0 tx=0x7f5a840046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f5a8ed9d640 1 -- 172.21.15.59:0/3536597107 reap_dead start 2024-03-06T07:22:00.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f3a14ff9640 1 -- 172.21.15.59:0/2886833197 <== osd.4 v2:172.21.15.164:6818/102497754 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3a180b6360 con 0x7f3a04073820 2024-03-06T07:22:00.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f5a94d0d640 1 -- 172.21.15.59:0/3536597107 shutdown_connections 2024-03-06T07:22:00.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f5a94d0d640 1 -- 172.21.15.59:0/3536597107 >> 172.21.15.59:0/3536597107 conn(0x7f5a9009a9f0 msgr2=0x7f5a90157d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f8b54ff9640 1 -- 172.21.15.59:0/3015557111 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8b40002dd0 con 0x7f8b480737c0 2024-03-06T07:22:00.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7fa9b813e640 1 -- 172.21.15.59:0/961933980 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa990002dd0 con 0x7fa99c073670 2024-03-06T07:22:00.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7fa06e16e640 1 -- 172.21.15.59:0/3841127584 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa04c06b5d0 msgr2=0x7fa04c06da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7fa06e16e640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa04c06b5d0 0x7fa04c06da90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa06400a9b0 tx=0x7fa064008040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7fa06e16e640 1 -- 172.21.15.59:0/3841127584 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0600382c0 msgr2=0x7fa0600e54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7fa06e16e640 1 --2- 172.21.15.59:0/3841127584 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0600382c0 0x7fa0600e54e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa068050c30 tx=0x7fa0680683c0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7fa06d96d640 1 -- 172.21.15.59:0/3841127584 reap_dead start 2024-03-06T07:22:00.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f67c77fe640 1 -- 172.21.15.59:0/2856292018 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f67c80737a0 msgr2=0x7f67c8075bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f67c77fe640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f67c80737a0 0x7f67c8075bc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f5a94d0d640 1 -- 172.21.15.59:0/3536597107 shutdown_connections 2024-03-06T07:22:00.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.019+0000 7f5a94d0d640 1 -- 172.21.15.59:0/3536597107 wait complete. 2024-03-06T07:22:00.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f67c77fe640 1 -- 172.21.15.59:0/2856292018 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67c806b520 msgr2=0x7f67c806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f67c77fe640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67c806b520 0x7f67c806d9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67e0036000 tx=0x7f67e003b040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f8b56ffd640 1 -- 172.21.15.59:0/3015557111 <== osd.7 v2:172.21.15.164:6808/3612340295 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8b40002dd0 con 0x7f8b480737c0 2024-03-06T07:22:00.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f3a04073820 msgr2=0x7f3a04075c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f3a1e6c9640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f3a04073820 0x7f3a04075c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7fa06e16e640 1 -- 172.21.15.59:0/3841127584 shutdown_connections 2024-03-06T07:22:00.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7fa06e16e640 1 -- 172.21.15.59:0/3841127584 >> 172.21.15.59:0/3841127584 conn(0x7fa060009fb0 msgr2=0x7fa0600372a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f67c77fe640 1 -- 172.21.15.59:0/2856292018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67dc0546c0 msgr2=0x7f67dc0c5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f67c77fe640 1 --2- 172.21.15.59:0/2856292018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67dc0546c0 0x7f67dc0c5260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f67e40505a0 tx=0x7f67e40683c0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f67e9a37640 1 -- 172.21.15.59:0/2856292018 reap_dead start 2024-03-06T07:22:00.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7fa9ae7fc640 1 -- 172.21.15.59:0/961933980 <== osd.0 v2:172.21.15.59:6824/841983189 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa990002dd0 con 0x7fa99c073670 2024-03-06T07:22:00.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7fa06e16e640 1 -- 172.21.15.59:0/3841127584 shutdown_connections 2024-03-06T07:22:00.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7fea9dffb640 1 -- 172.21.15.59:0/3102295821 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feab80b1fa0 con 0x7feaa40737c0 2024-03-06T07:22:00.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7fa06e16e640 1 -- 172.21.15.59:0/3841127584 wait complete. 2024-03-06T07:22:00.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f67c77fe640 1 -- 172.21.15.59:0/2856292018 shutdown_connections 2024-03-06T07:22:00.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f67c77fe640 1 -- 172.21.15.59:0/2856292018 >> 172.21.15.59:0/2856292018 conn(0x7f67dc009fe0 msgr2=0x7f67dc004b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a0406b5a0 msgr2=0x7f3a0406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f3a1e6c9640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a0406b5a0 0x7f3a0406da60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a08006a90 tx=0x7f3a08006a20 comp rx=0 tx=0).stop 2024-03-06T07:22:00.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/2886833197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a1806b040 msgr2=0x7f3a181f0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f3a1e6c9640 1 --2- 172.21.15.59:0/2886833197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a1806b040 0x7f3a181f0020 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009f40 tx=0x7f3a10004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:00.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f3a1cc66640 1 -- 172.21.15.59:0/2886833197 reap_dead start 2024-03-06T07:22:00.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/2886833197 shutdown_connections 2024-03-06T07:22:00.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/2886833197 >> 172.21.15.59:0/2886833197 conn(0x7f3a180b3170 msgr2=0x7f3a180b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f8b54ff9640 1 -- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x7f8b480737c0 msgr2=0x7f8b48075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f8b54ff9640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x7f8b480737c0 0x7f8b48075be0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7fa9b813e640 1 -- 172.21.15.59:0/961933980 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7fa99c073670 msgr2=0x7fa99c075a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7fa9b813e640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7fa99c073670 0x7fa99c075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fea9ffff640 1 -- 172.21.15.59:0/3102295821 <== osd.5 v2:172.21.15.164:6800/2889679248 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feab80b1fa0 con 0x7feaa40737c0 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f8b54ff9640 1 -- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b4806b5a0 msgr2=0x7f8b4806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f8b54ff9640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b4806b5a0 0x7f8b4806da60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8b58036000 tx=0x7f8b5803b040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f8b54ff9640 1 -- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b6815a1e0 msgr2=0x7f8b680ad570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.020+0000 7f8b54ff9640 1 --2- 172.21.15.59:0/3015557111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b6815a1e0 0x7f8b680ad570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000c730 tx=0x7f8b6000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7f8b66d9d640 1 -- 172.21.15.59:0/3015557111 reap_dead start 2024-03-06T07:22:00.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fa9b813e640 1 -- 172.21.15.59:0/961933980 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa99c06b4d0 msgr2=0x7fa99c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fa9b813e640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa99c06b4d0 0x7fa99c06d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8007fd0 tx=0x7fa9a8007e80 comp rx=0 tx=0).stop 2024-03-06T07:22:00.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7f67c77fe640 1 -- 172.21.15.59:0/2856292018 shutdown_connections 2024-03-06T07:22:00.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7f67c77fe640 1 -- 172.21.15.59:0/2856292018 wait complete. 2024-03-06T07:22:00.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7f8b54ff9640 1 -- 172.21.15.59:0/3015557111 shutdown_connections 2024-03-06T07:22:00.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7f8b54ff9640 1 -- 172.21.15.59:0/3015557111 >> 172.21.15.59:0/3015557111 conn(0x7f8b680b16e0 msgr2=0x7f8b680b0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fa9b813e640 1 -- 172.21.15.59:0/961933980 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b015df20 msgr2=0x7fa9b015bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fa9b813e640 1 --2- 172.21.15.59:0/961933980 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9b015df20 0x7fa9b015bf70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0009eb0 tx=0x7fa9a0004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:00.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fa9b66db640 1 -- 172.21.15.59:0/961933980 reap_dead start 2024-03-06T07:22:00.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fa9b813e640 1 -- 172.21.15.59:0/961933980 shutdown_connections 2024-03-06T07:22:00.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fa9b813e640 1 -- 172.21.15.59:0/961933980 >> 172.21.15.59:0/961933980 conn(0x7fa9b00b1830 msgr2=0x7fa9b016ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fa9b813e640 1 -- 172.21.15.59:0/961933980 shutdown_connections 2024-03-06T07:22:00.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/2886833197 shutdown_connections 2024-03-06T07:22:00.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7f8b54ff9640 1 -- 172.21.15.59:0/3015557111 shutdown_connections 2024-03-06T07:22:00.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fa9b813e640 1 -- 172.21.15.59:0/961933980 wait complete. 2024-03-06T07:22:00.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7f3a1e6c9640 1 -- 172.21.15.59:0/2886833197 wait complete. 2024-03-06T07:22:00.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7f8b54ff9640 1 -- 172.21.15.59:0/3015557111 wait complete. 2024-03-06T07:22:00.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fea9dffb640 1 -- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7feaa40737c0 msgr2=0x7feaa4075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fea9dffb640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7feaa40737c0 0x7feaa4075be0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6db80b4e80 con 0x7f6d980737a0 2024-03-06T07:22:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fea9dffb640 1 -- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feaa406b5a0 msgr2=0x7feaa406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fea9dffb640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feaa406b5a0 0x7feaa406da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feaac02b930 tx=0x7feaac03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fea9dffb640 1 -- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab80b29c0 msgr2=0x7feab814d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.021+0000 7fea9dffb640 1 --2- 172.21.15.59:0/3102295821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab80b29c0 0x7feab814d870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feab000efc0 tx=0x7feab000c490 comp rx=0 tx=0).stop 2024-03-06T07:22:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.022+0000 7feab7fff640 1 -- 172.21.15.59:0/3102295821 reap_dead start 2024-03-06T07:22:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.022+0000 7fea9dffb640 1 -- 172.21.15.59:0/3102295821 shutdown_connections 2024-03-06T07:22:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.022+0000 7fea9dffb640 1 -- 172.21.15.59:0/3102295821 >> 172.21.15.59:0/3102295821 conn(0x7feab80b16c0 msgr2=0x7feab8154420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.022+0000 7fea9dffb640 1 -- 172.21.15.59:0/3102295821 shutdown_connections 2024-03-06T07:22:00.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.022+0000 7fea9dffb640 1 -- 172.21.15.59:0/3102295821 wait complete. 2024-03-06T07:22:00.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.022+0000 7f6daeffd640 1 -- 172.21.15.59:0/2060485611 <== osd.6 v2:172.21.15.164:6816/1594998822 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6db80b4e80 con 0x7f6d980737a0 2024-03-06T07:22:00.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.023+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f6d980737a0 msgr2=0x7f6d98075bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.023+0000 7f6dbcd35640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f6d980737a0 0x7f6d98075bc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.023+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d9806b520 msgr2=0x7f6d9806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.023+0000 7f6dbcd35640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d9806b520 0x7f6d9806d9e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6da0001ad0 tx=0x7f6da0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.023+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b400 msgr2=0x7f6db813ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.023+0000 7f6dbcd35640 1 --2- 172.21.15.59:0/2060485611 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6db806b400 0x7f6db813ec90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6da800c750 tx=0x7f6da800cc10 comp rx=0 tx=0).stop 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.023+0000 7f6db6d9d640 1 -- 172.21.15.59:0/2060485611 reap_dead start 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.024+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 shutdown_connections 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.024+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 >> 172.21.15.59:0/2060485611 conn(0x7f6db80b1830 msgr2=0x7f6db80aff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.025+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 shutdown_connections 2024-03-06T07:22:00.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.025+0000 7f6dbcd35640 1 -- 172.21.15.59:0/2060485611 wait complete. 2024-03-06T07:22:00.077 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:00.077 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:22:00.078 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:00.078 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:22:00.079 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:00.079 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:22:00.080 INFO:teuthology.orchestra.run.smithi059.stdout:55834574855 2024-03-06T07:22:00.080 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:22:00.081 INFO:teuthology.orchestra.run.smithi059.stdout:55834574853 2024-03-06T07:22:00.081 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:22:00.082 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:00.082 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:22:00.083 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:00.083 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:22:00.084 INFO:teuthology.orchestra.run.smithi059.stdout:55834574855 2024-03-06T07:22:00.084 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:22:00.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.287+0000 7fb09dde2640 1 Processor -- start 2024-03-06T07:22:00.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.287+0000 7fb09dde2640 1 -- start start 2024-03-06T07:22:00.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.288+0000 7fe79b0cf640 1 Processor -- start 2024-03-06T07:22:00.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.288+0000 7f2e6b59e640 1 Processor -- start 2024-03-06T07:22:00.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.288+0000 7fe79b0cf640 1 -- start start 2024-03-06T07:22:00.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7f928fd5c640 1 Processor -- start 2024-03-06T07:22:00.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb09dde2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb090005e80 0x7fb0900062e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb09dde2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb090006820 0x7fb090006c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb09dde2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb090048410 con 0x7fb090048030 2024-03-06T07:22:00.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb09dde2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb090050190 con 0x7fb090006820 2024-03-06T07:22:00.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb09dde2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb090054790 con 0x7fb090005e80 2024-03-06T07:22:00.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb096ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb090005e80 0x7fb0900062e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb0977fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb090048030 0x7fb09004fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:48064/0 (socket says 172.21.15.59:48064) 2024-03-06T07:22:00.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb0977fe640 1 -- 172.21.15.59:0/1306188931 learned_addr learned my addr 172.21.15.59:0/1306188931 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:00.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb097fff640 1 --2- 172.21.15.59:0/1306188931 >> v2:172.21.15.59:3300/0 conn(0x7fb090006820 0x7fb090006c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.288+0000 7f2e6b59e640 1 -- start start 2024-03-06T07:22:00.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7fe79b0cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe79415c5b0 0x7fe794162580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7fe79b0cf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe7940b3ba0 0x7fe79406cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7fe79b0cf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79406c850 con 0x7fe794165b70 2024-03-06T07:22:00.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7fe79b0cf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe79415c990 con 0x7fe7940b3ba0 2024-03-06T07:22:00.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7fe79b0cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe794162b70 con 0x7fe79415c5b0 2024-03-06T07:22:00.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7fe793fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe794165b70 0x7fe7940b3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51206/0 (socket says 172.21.15.59:51206) 2024-03-06T07:22:00.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7fe793fff640 1 -- 172.21.15.59:0/2649608507 learned_addr learned my addr 172.21.15.59:0/2649608507 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:00.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7fe79966c640 1 --2- 172.21.15.59:0/2649608507 >> v2:172.21.15.164:3300/0 conn(0x7fe7940b3ba0 0x7fe79406cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7f928fd5c640 1 -- start start 2024-03-06T07:22:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f928fd5c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f92881620a0 0x7f9288163020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f928fd5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92880b3e40 0x7f92880713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f928fd5c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9288163560 con 0x7f928806c4c0 2024-03-06T07:22:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f928fd5c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9288162480 con 0x7f92881620a0 2024-03-06T07:22:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f928fd5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92880ae6b0 con 0x7f92880b3e40 2024-03-06T07:22:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f928daf8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f92881620a0 0x7f9288163020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f928daf8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f92881620a0 0x7f9288163020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58142/0 (socket says 172.21.15.59:58142) 2024-03-06T07:22:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f928daf8640 1 -- 172.21.15.59:0/1120105297 learned_addr learned my addr 172.21.15.59:0/1120105297 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f928e2f9640 1 --2- 172.21.15.59:0/1120105297 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92880b3e40 0x7f92880713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb096ffd640 1 -- 172.21.15.59:0/1306188931 >> v2:172.21.15.59:3300/0 conn(0x7fb090006820 msgr2=0x7fb090006c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb096ffd640 1 --2- 172.21.15.59:0/1306188931 >> v2:172.21.15.59:3300/0 conn(0x7fb090006820 0x7fb090006c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb096ffd640 1 -- 172.21.15.59:0/1306188931 >> v1:172.21.15.164:6789/0 conn(0x7fb090048030 legacy=0x7fb09004fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7fb096ffd640 1 -- 172.21.15.59:0/1306188931 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0900536f0 con 0x7fb090005e80 2024-03-06T07:22:00.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f2e6b59e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2e64006570 0x7f2e6400de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f2e6b59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e640578e0 0x7f2e64051270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f2e6b59e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6400c2a0 con 0x7f2e640e11f0 2024-03-06T07:22:00.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f2e6b59e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e640395b0 con 0x7f2e64006570 2024-03-06T07:22:00.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f2e6b59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6400c540 con 0x7f2e640578e0 2024-03-06T07:22:00.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f2e6a59c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2e640e11f0 0x7f2e64005e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51190/0 (socket says 172.21.15.59:51190) 2024-03-06T07:22:00.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f2e6a59c640 1 -- 172.21.15.59:0/1216634171 learned_addr learned my addr 172.21.15.59:0/1216634171 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:00.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7f88f5125640 1 Processor -- start 2024-03-06T07:22:00.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.289+0000 7f88f5125640 1 -- start start 2024-03-06T07:22:00.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f937e684640 1 Processor -- start 2024-03-06T07:22:00.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f937e684640 1 -- start start 2024-03-06T07:22:00.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f88f5125640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 0x7f88f0072580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7f9d90b38640 1 Processor -- start 2024-03-06T07:22:00.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f9d90b38640 1 -- start start 2024-03-06T07:22:00.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f46bf531640 1 Processor -- start 2024-03-06T07:22:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f2e69d9b640 1 --2- 172.21.15.59:0/1216634171 >> v2:172.21.15.164:3300/0 conn(0x7f2e64006570 0x7f2e6400de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f2e6ad9d640 1 --2- 172.21.15.59:0/1216634171 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e640578e0 0x7f2e64051270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f928e2f9640 1 -- 172.21.15.59:0/1120105297 >> v2:172.21.15.59:3300/0 conn(0x7f92881620a0 msgr2=0x7f9288163020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f928e2f9640 1 --2- 172.21.15.59:0/1120105297 >> v2:172.21.15.59:3300/0 conn(0x7f92881620a0 0x7f9288163020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f937e684640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f937806dfe0 0x7f937806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f88eed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 0x7f88f0072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f88eed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 0x7f88f0072580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43728/0 (socket says 172.21.15.59:43728) 2024-03-06T07:22:00.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f88ee59c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f88f0072ac0 0x7f88f006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51214/0 (socket says 172.21.15.59:51214) 2024-03-06T07:22:00.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f937e684640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f937806b4f0 0x7f937806b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f88f5125640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f006caa0 con 0x7f88f0072ac0 2024-03-06T07:22:00.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f88f5125640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f006cd80 con 0x7f88f006b6c0 2024-03-06T07:22:00.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f88f5125640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f006d060 con 0x7f88f0161fb0 2024-03-06T07:22:00.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f928e2f9640 1 -- 172.21.15.59:0/1120105297 >> v1:172.21.15.59:6789/0 conn(0x7f928806c4c0 legacy=0x7f92880b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f88eed9d640 1 -- 172.21.15.59:0/1073655773 learned_addr learned my addr 172.21.15.59:0/1073655773 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f9377fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9378140260 0x7f937815db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:48076/0 (socket says 172.21.15.59:48076) 2024-03-06T07:22:00.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f937cc21640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f937806b4f0 0x7f937806b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f9377fff640 1 -- 172.21.15.59:0/2965310503 learned_addr learned my addr 172.21.15.59:0/2965310503 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:00.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f928e2f9640 1 -- 172.21.15.59:0/1120105297 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92880ae990 con 0x7f92880b3e40 2024-03-06T07:22:00.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f2e6959a640 1 -- 172.21.15.59:0/1216634171 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409961240 0 0) 0x7f2e6400c2a0 con 0x7f2e640e11f0 2024-03-06T07:22:00.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f2e6959a640 1 -- 172.21.15.59:0/1216634171 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e54003520 con 0x7f2e640e11f0 2024-03-06T07:22:00.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f937e684640 1 -- 172.21.15.59:0/2965310503 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937815e570 con 0x7f9378140260 2024-03-06T07:22:00.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f937e684640 1 -- 172.21.15.59:0/2965310503 --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f937815e850 con 0x7f937806b4f0 2024-03-06T07:22:00.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f2e69d9b640 1 -- 172.21.15.59:0/1216634171 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e640578e0 msgr2=0x7f2e64051270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f2e69d9b640 1 --2- 172.21.15.59:0/1216634171 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e640578e0 0x7f2e64051270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f937e684640 1 -- 172.21.15.59:0/2965310503 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f937806c7d0 con 0x7f937806dfe0 2024-03-06T07:22:00.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7fb096ffd640 1 --2- 172.21.15.59:0/1306188931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb090005e80 0x7fb0900062e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb09802d580 tx=0x7fb0980a6960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f2a7a9a3f05e860 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7fb0967fc640 1 -- 172.21.15.59:0/1306188931 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0980af070 con 0x7fb090005e80 2024-03-06T07:22:00.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7fb0967fc640 1 -- 172.21.15.59:0/1306188931 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0980ab8c0 con 0x7fb090005e80 2024-03-06T07:22:00.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215751142 0 0) 0x7f937815e570 con 0x7f9378140260 2024-03-06T07:22:00.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9360003520 con 0x7f9378140260 2024-03-06T07:22:00.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7fb0967fc640 1 -- 172.21.15.59:0/1306188931 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0980abcf0 con 0x7fb090005e80 2024-03-06T07:22:00.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9d90b38640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9d8c0b4f00 0x7f9d8c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9d90b38640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9d8c0c6ac0 0x7f9d8c0d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9d90b38640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0d4a00 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9d90b38640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c0cf8b0 con 0x7f9d8c0b4f00 2024-03-06T07:22:00.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9d90b38640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c0d4720 con 0x7f9d8c0c6ac0 2024-03-06T07:22:00.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f88eed9d640 1 -- 172.21.15.59:0/1073655773 >> v1:172.21.15.164:6789/0 conn(0x7f88f006b6c0 legacy=0x7f88f00acd20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:00.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f88eed9d640 1 -- 172.21.15.59:0/1073655773 >> v1:172.21.15.59:6789/0 conn(0x7f88f0072ac0 legacy=0x7f88f006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f88eed9d640 1 -- 172.21.15.59:0/1073655773 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f006d340 con 0x7f88f0161fb0 2024-03-06T07:22:00.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f2e69d9b640 1 -- 172.21.15.59:0/1216634171 >> v1:172.21.15.59:6789/0 conn(0x7f2e640e11f0 legacy=0x7f2e64005e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f2e69d9b640 1 -- 172.21.15.59:0/1216634171 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6400c820 con 0x7f2e64006570 2024-03-06T07:22:00.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9d8b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9d8c0b4f00 0x7f9d8c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9d8b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9d8c0b4f00 0x7f9d8c0d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58162/0 (socket says 172.21.15.59:58162) 2024-03-06T07:22:00.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9d8b7fe640 1 -- 172.21.15.59:0/2197115998 learned_addr learned my addr 172.21.15.59:0/2197115998 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7f46bf531640 1 -- start start 2024-03-06T07:22:00.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f46bf531640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 0x7f46b8071950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f46bf531640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f46b80b38f0 0x7f46b80cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46bf531640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b80cdfe0 con 0x7f46b80d07f0 2024-03-06T07:22:00.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f928e2f9640 1 --2- 172.21.15.59:0/1120105297 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92880b3e40 0x7f92880713f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f927c0097a0 tx=0x7f927c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f678efc321a9e652 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f9d8bfff640 1 --2- 172.21.15.59:0/2197115998 >> v2:172.21.15.164:3300/0 conn(0x7f9d8c0c6ac0 0x7f9d8c0d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46bf531640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b80cff00 con 0x7f46b80b38f0 2024-03-06T07:22:00.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46bf531640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b8071f30 con 0x7f46b80cfb20 2024-03-06T07:22:00.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.290+0000 7fe798e6b640 1 --2- 172.21.15.59:0/2649608507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe79415c5b0 0x7fe794162580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7fe7937fe640 1 -- 172.21.15.59:0/2649608507 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067011344 0 0) 0x7fe79406c850 con 0x7fe794165b70 2024-03-06T07:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.291+0000 7fe7937fe640 1 -- 172.21.15.59:0/2649608507 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe784003520 con 0x7fe794165b70 2024-03-06T07:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7fe79966c640 1 -- 172.21.15.59:0/2649608507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe79415c5b0 msgr2=0x7fe794162580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7fe79966c640 1 --2- 172.21.15.59:0/2649608507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe79415c5b0 0x7fe794162580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7fe79966c640 1 -- 172.21.15.59:0/2649608507 >> v1:172.21.15.59:6789/0 conn(0x7fe794165b70 legacy=0x7fe7940b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7fe79966c640 1 -- 172.21.15.59:0/2649608507 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe794162e50 con 0x7fe7940b3ba0 2024-03-06T07:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f928caf6640 1 -- 172.21.15.59:0/1120105297 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c035020 con 0x7f92880b3e40 2024-03-06T07:22:00.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f88eed9d640 1 --2- 172.21.15.59:0/1073655773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 0x7f88f0072580 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f88e40098a0 tx=0x7f88e402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34fc6e0e7d49f557 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46be52f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 0x7f46b8071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46bdd2e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f46b80b38f0 0x7f46b80cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46bdd2e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f46b80b38f0 0x7f46b80cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:46816/0 (socket says 172.21.15.59:46816) 2024-03-06T07:22:00.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46bdd2e640 1 -- 172.21.15.59:0/3126643048 learned_addr learned my addr 172.21.15.59:0/3126643048 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46be52f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 0x7f46b8071950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43732/0 (socket says 172.21.15.59:43732) 2024-03-06T07:22:00.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f928caf6640 1 -- 172.21.15.59:0/1120105297 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927c031810 con 0x7f92880b3e40 2024-03-06T07:22:00.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f928caf6640 1 -- 172.21.15.59:0/1120105297 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c031be0 con 0x7f92880b3e40 2024-03-06T07:22:00.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f2e69d9b640 1 --2- 172.21.15.59:0/1216634171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e64006570 0x7f2e6400de80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e600097a0 tx=0x7f2e6002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9429c229b27001aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 392066850 0 0) 0x7f9360003520 con 0x7f9378140260 2024-03-06T07:22:00.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 >> v2:172.21.15.164:3300/0 conn(0x7f937806dfe0 msgr2=0x7f937806afb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:00.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 --2- 172.21.15.59:0/2965310503 >> v2:172.21.15.164:3300/0 conn(0x7f937806dfe0 0x7f937806afb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 >> v2:172.21.15.59:3300/0 conn(0x7f937806b4f0 msgr2=0x7f937806b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f2e6959a640 1 -- 172.21.15.59:0/1216634171 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60035020 con 0x7f2e64006570 2024-03-06T07:22:00.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f2e6959a640 1 -- 172.21.15.59:0/1216634171 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e60031810 con 0x7f2e64006570 2024-03-06T07:22:00.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f2e6959a640 1 -- 172.21.15.59:0/1216634171 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60031be0 con 0x7f2e64006570 2024-03-06T07:22:00.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f88edd9b640 1 -- 172.21.15.59:0/1073655773 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e4035020 con 0x7f88f0161fb0 2024-03-06T07:22:00.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 --2- 172.21.15.59:0/2965310503 >> v2:172.21.15.59:3300/0 conn(0x7f937806b4f0 0x7f937806b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937806cab0 con 0x7f9378140260 2024-03-06T07:22:00.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f9368001670 con 0x7f9378140260 2024-03-06T07:22:00.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 >> v1:172.21.15.164:6789/0 conn(0x7f9378140260 legacy=0x7f937815db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 --2- 172.21.15.59:0/2965310503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9360003a00 0x7f9360003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 --2- 172.21.15.59:0/2965310503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9360004800 0x7f9360004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 --2- 172.21.15.59:0/2965310503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9360005210 0x7f936000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937815e850 con 0x7f9360005210 2024-03-06T07:22:00.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937806c7d0 con 0x7f9360003a00 2024-03-06T07:22:00.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.292+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936000c1d0 con 0x7f9360004800 2024-03-06T07:22:00.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f9377fff640 1 --2- 172.21.15.59:0/2965310503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9360003a00 0x7f9360003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f937cc21640 1 --2- 172.21.15.59:0/2965310503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9360005210 0x7f936000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f93777fe640 1 --2- 172.21.15.59:0/2965310503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9360004800 0x7f9360004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f88edd9b640 1 -- 172.21.15.59:0/1073655773 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e4031640 con 0x7f88f0161fb0 2024-03-06T07:22:00.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f88edd9b640 1 -- 172.21.15.59:0/1073655773 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e402d040 con 0x7f88f0161fb0 2024-03-06T07:22:00.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7fe79966c640 1 --2- 172.21.15.59:0/2649608507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7940b3ba0 0x7fe79406cd30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe7800097a0 tx=0x7fe78002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a7e01eb43698468 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7fe7937fe640 1 -- 172.21.15.59:0/2649608507 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe780035020 con 0x7fe7940b3ba0 2024-03-06T07:22:00.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7fe7937fe640 1 -- 172.21.15.59:0/2649608507 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe780031830 con 0x7fe7940b3ba0 2024-03-06T07:22:00.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7fe7937fe640 1 -- 172.21.15.59:0/2649608507 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe780031c40 con 0x7fe7940b3ba0 2024-03-06T07:22:00.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f928fd5c640 1 -- 172.21.15.59:0/1120105297 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92880b3e40 msgr2=0x7f92880713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f937cc21640 1 -- 172.21.15.59:0/2965310503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9360004800 msgr2=0x7f9360004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f928fd5c640 1 --2- 172.21.15.59:0/1120105297 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92880b3e40 0x7f92880713f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f927c0097a0 tx=0x7f927c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7fb09dde2640 1 -- 172.21.15.59:0/1306188931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb090005e80 msgr2=0x7fb0900062e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7fb09dde2640 1 --2- 172.21.15.59:0/1306188931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb090005e80 0x7fb0900062e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb09802d580 tx=0x7fb0980a6960 comp rx=0 tx=0).stop 2024-03-06T07:22:00.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7fb09dde2640 1 -- 172.21.15.59:0/1306188931 shutdown_connections 2024-03-06T07:22:00.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7fb09dde2640 1 --2- 172.21.15.59:0/1306188931 >> v2:172.21.15.59:3300/0 conn(0x7fb090006820 0x7fb090006c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7fb09dde2640 1 --2- 172.21.15.59:0/1306188931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb090005e80 0x7fb0900062e0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb09802d580 tx=0x7fb0980a6960 comp rx=0 tx=0).stop 2024-03-06T07:22:00.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7fb09dde2640 1 -- 172.21.15.59:0/1306188931 >> 172.21.15.59:0/1306188931 conn(0x7fb090009fa0 msgr2=0x7fb09000a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f937cc21640 1 --2- 172.21.15.59:0/2965310503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9360004800 0x7f9360004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f937cc21640 1 -- 172.21.15.59:0/2965310503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9360003a00 msgr2=0x7f9360003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f937cc21640 1 --2- 172.21.15.59:0/2965310503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9360003a00 0x7f9360003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f937cc21640 1 -- 172.21.15.59:0/2965310503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9360003520 con 0x7f9360005210 2024-03-06T07:22:00.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f928fd5c640 1 -- 172.21.15.59:0/1120105297 shutdown_connections 2024-03-06T07:22:00.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f928fd5c640 1 --2- 172.21.15.59:0/1120105297 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92880b3e40 0x7f92880713f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f928fd5c640 1 --2- 172.21.15.59:0/1120105297 >> v2:172.21.15.59:3300/0 conn(0x7f92881620a0 0x7f9288163020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f928fd5c640 1 -- 172.21.15.59:0/1120105297 >> 172.21.15.59:0/1120105297 conn(0x7f92880b16e0 msgr2=0x7f92880b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1216634171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e64006570 msgr2=0x7f2e6400de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f2e6b59e640 1 --2- 172.21.15.59:0/1216634171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e64006570 0x7f2e6400de80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e600097a0 tx=0x7f2e6002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7fb09dde2640 1 -- 172.21.15.59:0/1306188931 shutdown_connections 2024-03-06T07:22:00.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46be52f640 1 -- 172.21.15.59:0/3126643048 >> v2:172.21.15.164:3300/0 conn(0x7f46b80b38f0 msgr2=0x7f46b80cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46be52f640 1 --2- 172.21.15.59:0/3126643048 >> v2:172.21.15.164:3300/0 conn(0x7f46b80b38f0 0x7f46b80cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46be52f640 1 -- 172.21.15.59:0/3126643048 >> v1:172.21.15.164:6789/0 conn(0x7f46b80d07f0 legacy=0x7f46b806e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f46be52f640 1 -- 172.21.15.59:0/3126643048 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b8072210 con 0x7f46b80cfb20 2024-03-06T07:22:00.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f928fd5c640 1 -- 172.21.15.59:0/1120105297 shutdown_connections 2024-03-06T07:22:00.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f88f5125640 1 -- 172.21.15.59:0/1073655773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 msgr2=0x7f88f0072580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f88f5125640 1 --2- 172.21.15.59:0/1073655773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 0x7f88f0072580 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f88e40098a0 tx=0x7f88e402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1216634171 shutdown_connections 2024-03-06T07:22:00.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f2e6b59e640 1 --2- 172.21.15.59:0/1216634171 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e640578e0 0x7f2e64051270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f2e6b59e640 1 --2- 172.21.15.59:0/1216634171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e64006570 0x7f2e6400de80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1216634171 >> 172.21.15.59:0/1216634171 conn(0x7f2e64020040 msgr2=0x7f2e64056cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f9d8b7fe640 1 -- 172.21.15.59:0/2197115998 >> v2:172.21.15.164:3300/0 conn(0x7f9d8c0c6ac0 msgr2=0x7f9d8c0d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f9d8b7fe640 1 --2- 172.21.15.59:0/2197115998 >> v2:172.21.15.164:3300/0 conn(0x7f9d8c0c6ac0 0x7f9d8c0d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f9d8b7fe640 1 -- 172.21.15.59:0/2197115998 >> v1:172.21.15.164:6789/0 conn(0x7f9d8c0c70f0 legacy=0x7f9d8c0d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.293+0000 7f9d8b7fe640 1 -- 172.21.15.59:0/2197115998 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c0cf260 con 0x7f9d8c0b4f00 2024-03-06T07:22:00.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f88f5125640 1 -- 172.21.15.59:0/1073655773 shutdown_connections 2024-03-06T07:22:00.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f88f5125640 1 --2- 172.21.15.59:0/1073655773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 0x7f88f0072580 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f88f5125640 1 -- 172.21.15.59:0/1073655773 >> 172.21.15.59:0/1073655773 conn(0x7f88f00b18d0 msgr2=0x7f88f00b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.294+0000 7f928fd5c640 1 -- 172.21.15.59:0/1120105297 wait complete. 2024-03-06T07:22:00.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1216634171 shutdown_connections 2024-03-06T07:22:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7f46be52f640 1 --2- 172.21.15.59:0/3126643048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 0x7f46b8071950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009d90 tx=0x7f46b002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9226079734e2d81a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1216634171 wait complete. 2024-03-06T07:22:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7fb09dde2640 1 -- 172.21.15.59:0/1306188931 wait complete. 2024-03-06T07:22:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7f88f5125640 1 -- 172.21.15.59:0/1073655773 shutdown_connections 2024-03-06T07:22:00.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7f46bd52d640 1 -- 172.21.15.59:0/3126643048 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0039070 con 0x7f46b80cfb20 2024-03-06T07:22:00.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7f46bd52d640 1 -- 172.21.15.59:0/3126643048 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b002cbd0 con 0x7f46b80cfb20 2024-03-06T07:22:00.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7f46bd52d640 1 -- 172.21.15.59:0/3126643048 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0034440 con 0x7f46b80cfb20 2024-03-06T07:22:00.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7fe79b0cf640 1 -- 172.21.15.59:0/2649608507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7940b3ba0 msgr2=0x7fe79406cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7fe79b0cf640 1 --2- 172.21.15.59:0/2649608507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7940b3ba0 0x7fe79406cd30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe7800097a0 tx=0x7fe78002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7fe79b0cf640 1 -- 172.21.15.59:0/2649608507 shutdown_connections 2024-03-06T07:22:00.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7fe79b0cf640 1 --2- 172.21.15.59:0/2649608507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7940b3ba0 0x7fe79406cd30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7fe79b0cf640 1 --2- 172.21.15.59:0/2649608507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe79415c5b0 0x7fe794162580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7fe79b0cf640 1 -- 172.21.15.59:0/2649608507 >> 172.21.15.59:0/2649608507 conn(0x7fe7940b1990 msgr2=0x7fe7940b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7f9d8b7fe640 1 --2- 172.21.15.59:0/2197115998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d8c0b4f00 0x7f9d8c0d0470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c06b5b0 tx=0x7f9d8402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5015c49749e5b30a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7f88f5125640 1 -- 172.21.15.59:0/1073655773 wait complete. 2024-03-06T07:22:00.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7f937cc21640 1 --2- 172.21.15.59:0/2965310503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9360005210 0x7f936000b9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9370007360 tx=0x7f9360005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36214242aa862b31 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.295+0000 7f937cc21640 1 -- 172.21.15.59:0/2965310503 reap_dead start 2024-03-06T07:22:00.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9370039070 con 0x7f9360005210 2024-03-06T07:22:00.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937002cd10 con 0x7f9360005210 2024-03-06T07:22:00.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7f9376ffd640 1 -- 172.21.15.59:0/2965310503 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9370009be0 con 0x7f9360005210 2024-03-06T07:22:00.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7f46bf531640 1 -- 172.21.15.59:0/3126643048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 msgr2=0x7f46b8071950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7f46bf531640 1 --2- 172.21.15.59:0/3126643048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 0x7f46b8071950 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f46b0009d90 tx=0x7f46b002be20 comp rx=0 tx=0).stop 2024-03-06T07:22:00.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7f9d8a7fc640 1 -- 172.21.15.59:0/2197115998 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8402cb20 con 0x7f9d8c0b4f00 2024-03-06T07:22:00.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7f9d8a7fc640 1 -- 172.21.15.59:0/2197115998 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8402ccc0 con 0x7f9d8c0b4f00 2024-03-06T07:22:00.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7f9d8a7fc640 1 -- 172.21.15.59:0/2197115998 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d84034610 con 0x7f9d8c0b4f00 2024-03-06T07:22:00.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7fe79b0cf640 1 -- 172.21.15.59:0/2649608507 shutdown_connections 2024-03-06T07:22:00.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7fe79b0cf640 1 -- 172.21.15.59:0/2649608507 wait complete. 2024-03-06T07:22:00.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7f928fd5c640 1 Processor -- start 2024-03-06T07:22:00.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7f928fd5c640 1 -- start start 2024-03-06T07:22:00.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7fb09dde2640 1 Processor -- start 2024-03-06T07:22:00.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.296+0000 7fb09dde2640 1 -- start start 2024-03-06T07:22:00.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f2e6b59e640 1 Processor -- start 2024-03-06T07:22:00.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f46bf531640 1 -- 172.21.15.59:0/3126643048 shutdown_connections 2024-03-06T07:22:00.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f2e6b59e640 1 -- start start 2024-03-06T07:22:00.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f46bf531640 1 --2- 172.21.15.59:0/3126643048 >> v2:172.21.15.164:3300/0 conn(0x7f46b80b38f0 0x7f46b80cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f46bf531640 1 --2- 172.21.15.59:0/3126643048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 0x7f46b8071950 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f46bf531640 1 -- 172.21.15.59:0/3126643048 >> 172.21.15.59:0/3126643048 conn(0x7f46b806c0c0 msgr2=0x7f46b806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f9d90b38640 1 -- 172.21.15.59:0/2197115998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d8c0b4f00 msgr2=0x7f9d8c0d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f9d90b38640 1 --2- 172.21.15.59:0/2197115998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d8c0b4f00 0x7f9d8c0d0470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c06b5b0 tx=0x7f9d8402bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:00.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f9d90b38640 1 -- 172.21.15.59:0/2197115998 shutdown_connections 2024-03-06T07:22:00.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f9d90b38640 1 --2- 172.21.15.59:0/2197115998 >> v2:172.21.15.164:3300/0 conn(0x7f9d8c0c6ac0 0x7f9d8c0d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f9d90b38640 1 --2- 172.21.15.59:0/2197115998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d8c0b4f00 0x7f9d8c0d0470 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f9d90b38640 1 -- 172.21.15.59:0/2197115998 >> 172.21.15.59:0/2197115998 conn(0x7f9d8c09cee0 msgr2=0x7f9d8c0d3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f9d90b38640 1 -- 172.21.15.59:0/2197115998 shutdown_connections 2024-03-06T07:22:00.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f9d90b38640 1 -- 172.21.15.59:0/2197115998 wait complete. 2024-03-06T07:22:00.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f937e684640 1 -- 172.21.15.59:0/2965310503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9360005210 msgr2=0x7f936000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f937e684640 1 --2- 172.21.15.59:0/2965310503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9360005210 0x7f936000b9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9370007360 tx=0x7f9360005830 comp rx=0 tx=0).stop 2024-03-06T07:22:00.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f937e684640 1 -- 172.21.15.59:0/2965310503 shutdown_connections 2024-03-06T07:22:00.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f928fd5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f928806c4c0 0x7f92880afed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f937e684640 1 --2- 172.21.15.59:0/2965310503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9360005210 0x7f936000b9c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.297+0000 7f937e684640 1 -- 172.21.15.59:0/2965310503 >> 172.21.15.59:0/2965310503 conn(0x7f93780b1830 msgr2=0x7f93780b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7fe79b0cf640 1 Processor -- start 2024-03-06T07:22:00.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7fb09dde2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb090006820 0x7fb090048900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f928fd5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92880b3e40 0x7f92880b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7fe79b0cf640 1 -- start start 2024-03-06T07:22:00.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7fb09dde2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb090048030 0x7fb09004ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f928fd5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92881620a0 0x7f92880aaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f937e684640 1 -- 172.21.15.59:0/2965310503 shutdown_connections 2024-03-06T07:22:00.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f937e684640 1 -- 172.21.15.59:0/2965310503 wait complete. 2024-03-06T07:22:00.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f928daf8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f928806c4c0 0x7f92880afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f928daf8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f928806c4c0 0x7f92880afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43742/0 (socket says 172.21.15.59:43742) 2024-03-06T07:22:00.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7fb0977fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb090006820 0x7fb090048900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f88f5125640 1 Processor -- start 2024-03-06T07:22:00.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f928daf8640 1 -- 172.21.15.59:0/340490781 learned_addr learned my addr 172.21.15.59:0/340490781 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7fb0977fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb090006820 0x7fb090048900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:46818/0 (socket says 172.21.15.59:46818) 2024-03-06T07:22:00.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f88f5125640 1 -- start start 2024-03-06T07:22:00.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f928e2f9640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92881620a0 0x7f92880aaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f928fd5c640 1 -- 172.21.15.59:0/340490781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9288070ed0 con 0x7f92881620a0 2024-03-06T07:22:00.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f928fd5c640 1 -- 172.21.15.59:0/340490781 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92880aef50 con 0x7f92880b3e40 2024-03-06T07:22:00.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f928fd5c640 1 -- 172.21.15.59:0/340490781 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92880b4420 con 0x7f928806c4c0 2024-03-06T07:22:00.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f2e6b59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e64006570 0x7f2e6404b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f2e6b59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e640578e0 0x7f2e6404c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f2e6a59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e64006570 0x7f2e6404b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f2e6a59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e64006570 0x7f2e6404b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43748/0 (socket says 172.21.15.59:43748) 2024-03-06T07:22:00.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f2e6a59c640 1 -- 172.21.15.59:0/1345641125 learned_addr learned my addr 172.21.15.59:0/1345641125 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.298+0000 7f46bf531640 1 -- 172.21.15.59:0/3126643048 shutdown_connections 2024-03-06T07:22:00.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f2e6b59e640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e640e11f0 0x7f2e6404c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f2e69d9b640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e640578e0 0x7f2e6404c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f2e6ad9d640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e640e11f0 0x7f2e6404c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e640392c0 con 0x7f2e640578e0 2024-03-06T07:22:00.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64039140 con 0x7f2e640e11f0 2024-03-06T07:22:00.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e640517b0 con 0x7f2e64006570 2024-03-06T07:22:00.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f9d90b38640 1 Processor -- start 2024-03-06T07:22:00.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fe79b0cf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7940b3ba0 0x7fe79414d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fb096ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb090048030 0x7fb09004ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fb0977fe640 1 -- 172.21.15.59:0/3106946944 learned_addr learned my addr 172.21.15.59:0/3106946944 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fe79b0cf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe79415c5b0 0x7fe7940c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fe79b0cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe794165b70 0x7fe7940c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fe79b0cf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940710b0 con 0x7fe79415c5b0 2024-03-06T07:22:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fe79b0cf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe794163410 con 0x7fe7940b3ba0 2024-03-06T07:22:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fe79b0cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79406d270 con 0x7fe794165b70 2024-03-06T07:22:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f928e2f9640 1 -- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f928806c4c0 msgr2=0x7f92880afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f928e2f9640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f928806c4c0 0x7f92880afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f928e2f9640 1 -- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92880b3e40 msgr2=0x7f92880b0410 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f928e2f9640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92880b3e40 0x7f92880b0410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f928e2f9640 1 -- 172.21.15.59:0/340490781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92880a94f0 con 0x7f92881620a0 2024-03-06T07:22:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fe793fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe79415c5b0 0x7fe7940c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fe793fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe79415c5b0 0x7fe7940c7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58194/0 (socket says 172.21.15.59:58194) 2024-03-06T07:22:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fe793fff640 1 -- 172.21.15.59:0/2386338183 learned_addr learned my addr 172.21.15.59:0/2386338183 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f2e6a59c640 1 -- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e640e11f0 msgr2=0x7f2e6404c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fe798e6b640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7940b3ba0 0x7fe79414d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fe79966c640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe794165b70 0x7fe7940c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f2e6a59c640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e640e11f0 0x7f2e6404c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f928e2f9640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92881620a0 0x7f92880aaf10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f927c033000 tx=0x7f927c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f88f5125640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88f006b6c0 0x7f88f0146940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f2e6a59c640 1 -- 172.21.15.59:0/1345641125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e640578e0 msgr2=0x7f2e6404c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f88f5125640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f0072ac0 0x7f88f015b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f937e684640 1 Processor -- start 2024-03-06T07:22:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f2e6a59c640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e640578e0 0x7f2e6404c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f927a7fc640 1 -- 172.21.15.59:0/340490781 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c041070 con 0x7f92881620a0 2024-03-06T07:22:00.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f2e6a59c640 1 -- 172.21.15.59:0/1345641125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e640e2130 con 0x7f2e64006570 2024-03-06T07:22:00.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f88eed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88f006b6c0 0x7f88f0146940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f927a7fc640 1 -- 172.21.15.59:0/340490781 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927c03ad80 con 0x7f92881620a0 2024-03-06T07:22:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f927a7fc640 1 -- 172.21.15.59:0/340490781 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c035040 con 0x7f92881620a0 2024-03-06T07:22:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f928fd5c640 1 -- 172.21.15.59:0/340490781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92880a9830 con 0x7f92881620a0 2024-03-06T07:22:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f88eed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88f006b6c0 0x7f88f0146940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:46866/0 (socket says 172.21.15.59:46866) 2024-03-06T07:22:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f937e684640 1 -- start start 2024-03-06T07:22:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7fe79966c640 1 -- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7940b3ba0 msgr2=0x7fe79414d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7fe79966c640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7940b3ba0 0x7fe79414d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7fe79966c640 1 -- 172.21.15.59:0/2386338183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe79415c5b0 msgr2=0x7fe7940c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7fe79966c640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe79415c5b0 0x7fe7940c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7fe79966c640 1 -- 172.21.15.59:0/2386338183 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7940c5c60 con 0x7fe794165b70 2024-03-06T07:22:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f928fd5c640 1 -- 172.21.15.59:0/340490781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92880a9d90 con 0x7f92881620a0 2024-03-06T07:22:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7fb09dde2640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb09004d3b0 0x7fb0900d9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f9d90b38640 1 -- start start 2024-03-06T07:22:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f9d90b38640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d8c0b4f00 0x7f9d8c0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f9d90b38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d8c0c6ac0 0x7f9d8c15b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7fe79966c640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe794165b70 0x7fe7940c7aa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe780033000 tx=0x7fe7800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.299+0000 7f46bf531640 1 -- 172.21.15.59:0/3126643048 wait complete. 2024-03-06T07:22:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f46bf531640 1 Processor -- start 2024-03-06T07:22:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f46bf531640 1 -- start start 2024-03-06T07:22:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fe7917fa640 1 -- 172.21.15.59:0/2386338183 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe780035040 con 0x7fe794165b70 2024-03-06T07:22:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fe79b0cf640 1 -- 172.21.15.59:0/2386338183 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7940c5f40 con 0x7fe794165b70 2024-03-06T07:22:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fe7917fa640 1 -- 172.21.15.59:0/2386338183 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe78003b860 con 0x7fe794165b70 2024-03-06T07:22:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fe7917fa640 1 -- 172.21.15.59:0/2386338183 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78003bc50 con 0x7fe794165b70 2024-03-06T07:22:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fe79b0cf640 1 -- 172.21.15.59:0/2386338183 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7940c6500 con 0x7fe794165b70 2024-03-06T07:22:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb090053f90 con 0x7fb090048030 2024-03-06T07:22:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb090053cb0 con 0x7fb090006820 2024-03-06T07:22:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb090054270 con 0x7fb09004d3b0 2024-03-06T07:22:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f88ee59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f0072ac0 0x7f88f015b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f88ee59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f0072ac0 0x7f88f015b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58210/0 (socket says 172.21.15.59:58210) 2024-03-06T07:22:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f88f5125640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 0x7f88f0147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f88f5125640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f0162510 con 0x7f88f0072ac0 2024-03-06T07:22:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f88f5125640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f0162390 con 0x7f88f006b6c0 2024-03-06T07:22:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f88f5125640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f00ad430 con 0x7f88f0161fb0 2024-03-06T07:22:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f88eed9d640 1 -- 172.21.15.59:0/3807513883 learned_addr learned my addr 172.21.15.59:0/3807513883 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f88ef59e640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 0x7f88f0147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f88ef59e640 1 -- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88f006b6c0 msgr2=0x7f88f0146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f88ef59e640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88f006b6c0 0x7f88f0146940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f88ef59e640 1 -- 172.21.15.59:0/3807513883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f0072ac0 msgr2=0x7f88f015b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f88ef59e640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f0072ac0 0x7f88f015b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f88ef59e640 1 -- 172.21.15.59:0/3807513883 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f009d830 con 0x7f88f0161fb0 2024-03-06T07:22:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f9d90b38640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d8c0c70f0 0x7f9d8c1ebe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f9d90b38640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0b5460 con 0x7f9d8c0c6ac0 2024-03-06T07:22:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f9d90b38640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0b52e0 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.300+0000 7f9d90b38640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c0c8700 con 0x7f9d8c0b4f00 2024-03-06T07:22:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f9d8affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d8c0c6ac0 0x7f9d8c15b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f9d8affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d8c0c6ac0 0x7f9d8c15b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58220/0 (socket says 172.21.15.59:58220) 2024-03-06T07:22:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f9d8affd640 1 -- 172.21.15.59:0/2651223498 learned_addr learned my addr 172.21.15.59:0/2651223498 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f9d8bfff640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d8c0c70f0 0x7f9d8c1ebe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f9d8b7fe640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d8c0b4f00 0x7f9d8c0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f937e684640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9360005210 0x7f93781f7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f937e684640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93780ab150 0x7f937815ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fb096ffd640 1 -- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb09004d3b0 msgr2=0x7fb0900d9e30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:22:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fb096ffd640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb09004d3b0 0x7fb0900d9e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fb096ffd640 1 -- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb090006820 msgr2=0x7fb090048900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fb096ffd640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb090006820 0x7fb090048900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7fb096ffd640 1 -- 172.21.15.59:0/3106946944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09004d970 con 0x7fb090048030 2024-03-06T07:22:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7fb096ffd640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb090048030 0x7fb09004ce70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb09806a9f0 tx=0x7fb098069a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f88ee59c640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f0072ac0 0x7f88f015b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-06T07:22:00.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7fb083fff640 1 -- 172.21.15.59:0/3106946944 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0980c4070 con 0x7fb090048030 2024-03-06T07:22:00.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f88ef59e640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 0x7f88f0147080 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88e000b3e0 tx=0x7f88e000b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f46bf531640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b80b38f0 0x7f46b81fe8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09004dc50 con 0x7fb090048030 2024-03-06T07:22:00.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f46bf531640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 0x7f46b81fe160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7fb083fff640 1 -- 172.21.15.59:0/3106946944 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0980b4d00 con 0x7fb090048030 2024-03-06T07:22:00.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7fb083fff640 1 -- 172.21.15.59:0/3106946944 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0980af040 con 0x7fb090048030 2024-03-06T07:22:00.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f46bf531640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b80d07f0 0x7f46b815cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f9d8bfff640 1 -- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d8c0b4f00 msgr2=0x7f9d8c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f46bdd2e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 0x7f46b81fe160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f46bed30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b80d07f0 0x7f46b815cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f9377fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9360005210 0x7f93781f7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f9377fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9360005210 0x7f93781f7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43806/0 (socket says 172.21.15.59:43806) 2024-03-06T07:22:00.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f46bed30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b80d07f0 0x7f46b815cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58232/0 (socket says 172.21.15.59:58232) 2024-03-06T07:22:00.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f46bdd2e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 0x7f46b81fe160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43816/0 (socket says 172.21.15.59:43816) 2024-03-06T07:22:00.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f46bed30640 1 -- 172.21.15.59:0/3019544073 learned_addr learned my addr 172.21.15.59:0/3019544073 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09004e4f0 con 0x7fb090048030 2024-03-06T07:22:00.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b806dea0 con 0x7f46b80d07f0 2024-03-06T07:22:00.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80727d0 con 0x7f46b80b38f0 2024-03-06T07:22:00.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80b0250 con 0x7f46b80cfb20 2024-03-06T07:22:00.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f9d8bfff640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d8c0b4f00 0x7f9d8c0b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f9d8bfff640 1 -- 172.21.15.59:0/2651223498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d8c0c6ac0 msgr2=0x7f9d8c15b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f46be52f640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b80b38f0 0x7f46b81fe8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f2e6a59c640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e64006570 0x7f2e6404b960 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00c430 tx=0x7f2e5c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f2e5affd640 1 -- 172.21.15.59:0/1345641125 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c017070 con 0x7f2e64006570 2024-03-06T07:22:00.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f2e5affd640 1 -- 172.21.15.59:0/1345641125 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e5c0074e0 con 0x7f2e64006570 2024-03-06T07:22:00.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.301+0000 7f2e5affd640 1 -- 172.21.15.59:0/1345641125 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c0059c0 con 0x7f2e64006570 2024-03-06T07:22:00.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6404bea0 con 0x7f2e64006570 2024-03-06T07:22:00.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e640e23b0 con 0x7f2e64006570 2024-03-06T07:22:00.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f88d77fe640 1 -- 172.21.15.59:0/3807513883 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0015020 con 0x7f88f0161fb0 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f88d77fe640 1 -- 172.21.15.59:0/3807513883 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e0011830 con 0x7f88f0161fb0 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f88f5125640 1 -- 172.21.15.59:0/3807513883 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f0163710 con 0x7f88f0161fb0 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f88d77fe640 1 -- 172.21.15.59:0/3807513883 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0011c20 con 0x7f88f0161fb0 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f88f5125640 1 -- 172.21.15.59:0/3807513883 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f0163cd0 con 0x7f88f0161fb0 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f9d8bfff640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d8c0c6ac0 0x7f9d8c15b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f9d8bfff640 1 -- 172.21.15.59:0/2651223498 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c0b69a0 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f9d8affd640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d8c0c6ac0 0x7f9d8c15b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f46be52f640 1 -- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 msgr2=0x7f46b81fe160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f46be52f640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 0x7f46b81fe160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f46be52f640 1 -- 172.21.15.59:0/3019544073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b80d07f0 msgr2=0x7f46b815cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.302+0000 7f926bfff640 1 -- 172.21.15.59:0/340490781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9288068700 con 0x7f92881620a0 2024-03-06T07:22:00.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7fe79b0cf640 1 -- 172.21.15.59:0/2386338183 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe774005290 con 0x7fe794165b70 2024-03-06T07:22:00.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7fe7917fa640 1 -- 172.21.15.59:0/2386338183 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe780038020 con 0x7fe794165b70 2024-03-06T07:22:00.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7fe7917fa640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe77806b5d0 0x7fe77806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f9d8bfff640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d8c0c70f0 0x7f9d8c1ebe90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000e7f0 tx=0x7f9d8000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f46be52f640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b80d07f0 0x7f46b815cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f46be52f640 1 -- 172.21.15.59:0/3019544073 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b81fee00 con 0x7f46b80b38f0 2024-03-06T07:22:00.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f46bed30640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b80d07f0 0x7f46b815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:00.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7fe798e6b640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe77806b5d0 0x7fe77806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f937e684640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93780a95d0 0x7f937815d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f937e684640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93781407c0 con 0x7f93780ab150 2024-03-06T07:22:00.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f937e684640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9378140640 con 0x7f93780a95d0 2024-03-06T07:22:00.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f937e684640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937806afb0 con 0x7f9360005210 2024-03-06T07:22:00.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f93777fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93780ab150 0x7f937815ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f93777fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93780ab150 0x7f937815ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58236/0 (socket says 172.21.15.59:58236) 2024-03-06T07:22:00.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f93777fe640 1 -- 172.21.15.59:0/3781321593 learned_addr learned my addr 172.21.15.59:0/3781321593 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:00.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f937cc21640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93780a95d0 0x7f937815d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f937cc21640 1 -- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9360005210 msgr2=0x7f93781f7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f937cc21640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9360005210 0x7f93781f7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f937cc21640 1 -- 172.21.15.59:0/3781321593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93780ab150 msgr2=0x7f937815ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f937cc21640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93780ab150 0x7f937815ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.303+0000 7f937cc21640 1 -- 172.21.15.59:0/3781321593 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93780ab630 con 0x7f93780a95d0 2024-03-06T07:22:00.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f937cc21640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93780a95d0 0x7f937815d360 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9378070860 tx=0x7f9370004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7fe7917fa640 1 -- 172.21.15.59:0/2386338183 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7800ac250 con 0x7fe794165b70 2024-03-06T07:22:00.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f9d73fff640 1 -- 172.21.15.59:0/2651223498 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80017070 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f927a7fc640 1 -- 172.21.15.59:0/340490781 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f927c04c060 con 0x7f92881620a0 2024-03-06T07:22:00.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f927a7fc640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f926c06b4d0 0x7f926c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f927a7fc640 1 -- 172.21.15.59:0/340490781 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f927c0abe20 con 0x7f92881620a0 2024-03-06T07:22:00.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f9d90b38640 1 -- 172.21.15.59:0/2651223498 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c0b6c80 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f9d90b38640 1 -- 172.21.15.59:0/2651223498 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c0b7240 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.305+0000 7fe798e6b640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe77806b5d0 0x7fe77806da90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe78c0073f0 tx=0x7fe78c00b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.305+0000 7f928daf8640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f926c06b4d0 0x7f926c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.305+0000 7f9d73fff640 1 -- 172.21.15.59:0/2651223498 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8000c940 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f9374ff9640 1 -- 172.21.15.59:0/3781321593 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9370039040 con 0x7f93780a95d0 2024-03-06T07:22:00.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f9374ff9640 1 -- 172.21.15.59:0/3781321593 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9370043070 con 0x7f93780a95d0 2024-03-06T07:22:00.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.304+0000 7f9374ff9640 1 -- 172.21.15.59:0/3781321593 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9370009be0 con 0x7f93780a95d0 2024-03-06T07:22:00.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.305+0000 7f937e684640 1 -- 172.21.15.59:0/3781321593 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93780ab910 con 0x7f93780a95d0 2024-03-06T07:22:00.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.305+0000 7f937e684640 1 -- 172.21.15.59:0/3781321593 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93780abed0 con 0x7f93780a95d0 2024-03-06T07:22:00.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.306+0000 7f928daf8640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f926c06b4d0 0x7f926c06d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f92800093e0 tx=0x7f9280009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.306+0000 7f9d73fff640 1 -- 172.21.15.59:0/2651223498 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8000cd10 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.305+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09004de20 con 0x7fb090048030 2024-03-06T07:22:00.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.306+0000 7f88d77fe640 1 -- 172.21.15.59:0/3807513883 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88e0018020 con 0x7f88f0161fb0 2024-03-06T07:22:00.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.306+0000 7fb083fff640 1 -- 172.21.15.59:0/3106946944 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0980b2070 con 0x7fb090048030 2024-03-06T07:22:00.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.305+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e64005360 con 0x7f2e64006570 2024-03-06T07:22:00.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.306+0000 7f937e684640 1 -- 172.21.15.59:0/3781321593 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937804f330 con 0x7f93780a95d0 2024-03-06T07:22:00.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.306+0000 7f9d90b38640 1 -- 172.21.15.59:0/2651223498 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6c005290 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.306+0000 7f88f5125640 1 -- 172.21.15.59:0/3807513883 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f00b28e0 con 0x7f88f0161fb0 2024-03-06T07:22:00.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7f88d77fe640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88dc06b520 0x7f88dc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7f88eed9d640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88dc06b520 0x7f88dc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7fb083fff640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb07c06b4d0 0x7fb07c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7fb083fff640 1 -- 172.21.15.59:0/3106946944 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb098126b30 con 0x7fb090048030 2024-03-06T07:22:00.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.306+0000 7f46be52f640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b80b38f0 0x7f46b81fe8c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f46b0033000 tx=0x7f46b0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7f2e5affd640 1 -- 172.21.15.59:0/1345641125 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e5c01a070 con 0x7f2e64006570 2024-03-06T07:22:00.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7f9d73fff640 1 -- 172.21.15.59:0/2651223498 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d8001a070 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7f46aeffd640 1 -- 172.21.15.59:0/3019544073 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0044020 con 0x7f46b80b38f0 2024-03-06T07:22:00.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7f46aeffd640 1 -- 172.21.15.59:0/3019544073 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b003a040 con 0x7f46b80b38f0 2024-03-06T07:22:00.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7f46aeffd640 1 -- 172.21.15.59:0/3019544073 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0039070 con 0x7f46b80b38f0 2024-03-06T07:22:00.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7f2e5affd640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e4c06b520 0x7f2e4c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b81fe6a0 con 0x7f46b80b38f0 2024-03-06T07:22:00.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.307+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b82026e0 con 0x7f46b80b38f0 2024-03-06T07:22:00.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.308+0000 7f2e69d9b640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e4c06b520 0x7f2e4c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.308+0000 7f9d73fff640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b5a0 0x7f9d7806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.308+0000 7f9d73fff640 1 -- 172.21.15.59:0/2651223498 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d8008a330 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.308+0000 7f88eed9d640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88dc06b520 0x7f88dc06d9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f88e4034040 tx=0x7f88e403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.308+0000 7f2e69d9b640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e4c06b520 0x7f2e4c06d9e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e60036000 tx=0x7f2e60008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.308+0000 7f9374ff9640 1 -- 172.21.15.59:0/3781321593 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9370048460 con 0x7f93780a95d0 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.308+0000 7fb0977fe640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb07c06b4d0 0x7fb07c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.308+0000 7f9d8b7fe640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b5a0 0x7f9d7806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.308+0000 7f9374ff9640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f936406b540 0x7f936406da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.308+0000 7f9377fff640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f936406b540 0x7f936406da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.308+0000 7f9374ff9640 1 -- 172.21.15.59:0/3781321593 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93700abcf0 con 0x7f93780a95d0 2024-03-06T07:22:00.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.309+0000 7f9d8b7fe640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b5a0 0x7f9d7806da60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9d84035000 tx=0x7f9d84030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.309+0000 7fb0977fe640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb07c06b4d0 0x7fb07c06d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb084005880 tx=0x7fb084009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.309+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b8068f80 con 0x7f46b80b38f0 2024-03-06T07:22:00.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.309+0000 7f9377fff640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f936406b540 0x7f936406da00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9368000c00 tx=0x7f9368001e10 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.309+0000 7f2e5affd640 1 -- 172.21.15.59:0/1345641125 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e5c055e90 con 0x7f2e64006570 2024-03-06T07:22:00.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.309+0000 7f88d77fe640 1 -- 172.21.15.59:0/3807513883 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88e0056840 con 0x7f88f0161fb0 2024-03-06T07:22:00.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.310+0000 7fe7917fa640 1 -- 172.21.15.59:0/2386338183 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe78007b170 con 0x7fe794165b70 2024-03-06T07:22:00.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.311+0000 7f927a7fc640 1 -- 172.21.15.59:0/340490781 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92880a9830 con 0x7f92881620a0 2024-03-06T07:22:00.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.314+0000 7f2e5affd640 1 -- 172.21.15.59:0/1345641125 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e640e2130 con 0x7f2e64006570 2024-03-06T07:22:00.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.316+0000 7fb083fff640 1 -- 172.21.15.59:0/3106946944 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0980f5910 con 0x7fb090048030 2024-03-06T07:22:00.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.317+0000 7f9d73fff640 1 -- 172.21.15.59:0/2651223498 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d80058bd0 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.318+0000 7f46aeffd640 1 -- 172.21.15.59:0/3019544073 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46b0047020 con 0x7f46b80b38f0 2024-03-06T07:22:00.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.319+0000 7f88d77fe640 1 -- 172.21.15.59:0/3807513883 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88e001f020 con 0x7f88f0161fb0 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.319+0000 7f46aeffd640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469c06b520 0x7f469c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.319+0000 7f46bdd2e640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469c06b520 0x7f469c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.320+0000 7f46bdd2e640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469c06b520 0x7f469c06d9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f46b80af810 tx=0x7f46a8009240 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.326+0000 7f46aeffd640 1 -- 172.21.15.59:0/3019544073 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46b0079830 con 0x7f46b80b38f0 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.326+0000 7f9374ff9640 1 -- 172.21.15.59:0/3781321593 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f937007ab50 con 0x7f93780a95d0 2024-03-06T07:22:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.333+0000 7f46aeffd640 1 -- 172.21.15.59:0/3019544073 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46b004e030 con 0x7f46b80b38f0 2024-03-06T07:22:00.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.716+0000 7fe79b0cf640 1 -- 172.21.15.59:0/2386338183 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe774005580 con 0x7fe794165b70 2024-03-06T07:22:00.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.716+0000 7fe7917fa640 1 -- 172.21.15.59:0/2386338183 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe78007aa70 con 0x7fe794165b70 2024-03-06T07:22:00.718 INFO:teuthology.orchestra.run.smithi059.stdout:55834574853 2024-03-06T07:22:00.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.719+0000 7fe77effd640 1 -- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe77806b5d0 msgr2=0x7fe77806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.719+0000 7fe77effd640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe77806b5d0 0x7fe77806da90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe78c0073f0 tx=0x7fe78c00b040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.720+0000 7fe77effd640 1 -- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe794165b70 msgr2=0x7fe7940c7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.720+0000 7fe77effd640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe794165b70 0x7fe7940c7aa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe780033000 tx=0x7fe7800046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.720+0000 7fe77effd640 1 -- 172.21.15.59:0/2386338183 shutdown_connections 2024-03-06T07:22:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.720+0000 7fe77effd640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe77806b5d0 0x7fe77806da90 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe78c0073f0 tx=0x7fe78c00b040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.720+0000 7fe77effd640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe794165b70 0x7fe7940c7aa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.720+0000 7fe77effd640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe79415c5b0 0x7fe7940c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.720+0000 7fe77effd640 1 --2- 172.21.15.59:0/2386338183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7940b3ba0 0x7fe79414d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.720+0000 7fe77effd640 1 -- 172.21.15.59:0/2386338183 >> 172.21.15.59:0/2386338183 conn(0x7fe7940b1990 msgr2=0x7fe79415bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.721+0000 7fe77effd640 1 -- 172.21.15.59:0/2386338183 shutdown_connections 2024-03-06T07:22:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.721+0000 7fe77effd640 1 -- 172.21.15.59:0/2386338183 wait complete. 2024-03-06T07:22:00.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.730+0000 7f88f5125640 1 -- 172.21.15.59:0/3807513883 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f88f0068f10 con 0x7f88f0161fb0 2024-03-06T07:22:00.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.730+0000 7f88d77fe640 1 -- 172.21.15.59:0/3807513883 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f88e0058ea0 con 0x7f88f0161fb0 2024-03-06T07:22:00.732 INFO:teuthology.orchestra.run.smithi059.stdout:55834574853 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.733+0000 7f88d57fa640 1 -- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88dc06b520 msgr2=0x7f88dc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.734+0000 7f88d57fa640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88dc06b520 0x7f88dc06d9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f88e4034040 tx=0x7f88e403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.734+0000 7f88d57fa640 1 -- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 msgr2=0x7f88f0147080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.734+0000 7f88d57fa640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 0x7f88f0147080 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f88e000b3e0 tx=0x7f88e000b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.734+0000 7f88d57fa640 1 -- 172.21.15.59:0/3807513883 shutdown_connections 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.734+0000 7f88d57fa640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88dc06b520 0x7f88dc06d9e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.734+0000 7f88d57fa640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88f0161fb0 0x7f88f0147080 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.734+0000 7f88d57fa640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88f0072ac0 0x7f88f015b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.734+0000 7f88d57fa640 1 --2- 172.21.15.59:0/3807513883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88f006b6c0 0x7f88f0146940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.734+0000 7f88d57fa640 1 -- 172.21.15.59:0/3807513883 >> 172.21.15.59:0/3807513883 conn(0x7f88f00b18d0 msgr2=0x7f88f00b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.734+0000 7f88d57fa640 1 -- 172.21.15.59:0/3807513883 shutdown_connections 2024-03-06T07:22:00.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.735+0000 7f88d57fa640 1 -- 172.21.15.59:0/3807513883 wait complete. 2024-03-06T07:22:00.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.734+0000 7f926bfff640 1 -- 172.21.15.59:0/340490781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9288068f80 con 0x7f92881620a0 2024-03-06T07:22:00.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.735+0000 7f927a7fc640 1 -- 172.21.15.59:0/340490781 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f927c07ac00 con 0x7f92881620a0 2024-03-06T07:22:00.738 INFO:teuthology.orchestra.run.smithi059.stdout:55834574853 2024-03-06T07:22:00.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-03-06T07:22:00.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.738+0000 7f926bfff640 1 -- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f926c06b4d0 msgr2=0x7f926c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.738+0000 7f926bfff640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f926c06b4d0 0x7f926c06d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f92800093e0 tx=0x7f9280009040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.738+0000 7f926bfff640 1 -- 172.21.15.59:0/340490781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92881620a0 msgr2=0x7f92880aaf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.738+0000 7f926bfff640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92881620a0 0x7f92880aaf10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f927c033000 tx=0x7f927c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.739+0000 7f926bfff640 1 -- 172.21.15.59:0/340490781 shutdown_connections 2024-03-06T07:22:00.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.739+0000 7f926bfff640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f926c06b4d0 0x7f926c06d990 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.739+0000 7f926bfff640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92881620a0 0x7f92880aaf10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.739+0000 7f926bfff640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92880b3e40 0x7f92880b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.739+0000 7f926bfff640 1 --2- 172.21.15.59:0/340490781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f928806c4c0 0x7f92880afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.739+0000 7f926bfff640 1 -- 172.21.15.59:0/340490781 >> 172.21.15.59:0/340490781 conn(0x7f92880b16e0 msgr2=0x7f92880712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.739+0000 7f926bfff640 1 -- 172.21.15.59:0/340490781 shutdown_connections 2024-03-06T07:22:00.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.739+0000 7f926bfff640 1 -- 172.21.15.59:0/340490781 wait complete. 2024-03-06T07:22:00.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.741+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb0900089c0 con 0x7fb090048030 2024-03-06T07:22:00.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.741+0000 7fb083fff640 1 -- 172.21.15.59:0/3106946944 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb0980f5210 con 0x7fb090048030 2024-03-06T07:22:00.743 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.743+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2e640d9000 con 0x7f2e64006570 2024-03-06T07:22:00.745 INFO:teuthology.orchestra.run.smithi059.stdout:55834574852 2024-03-06T07:22:00.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.743+0000 7f2e5affd640 1 -- 172.21.15.59:0/1345641125 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e5c057df0 con 0x7f2e64006570 2024-03-06T07:22:00.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.744+0000 7f937e684640 1 -- 172.21.15.59:0/3781321593 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f937815dfa0 con 0x7f93780a95d0 2024-03-06T07:22:00.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.744+0000 7f9374ff9640 1 -- 172.21.15.59:0/3781321593 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f937007a450 con 0x7f93780a95d0 2024-03-06T07:22:00.747 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:00.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7f9d90b38640 1 -- 172.21.15.59:0/2651223498 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9d6c005580 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb07c06b4d0 msgr2=0x7fb07c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb07c06b4d0 0x7fb07c06d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb084005880 tx=0x7fb084009040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb090048030 msgr2=0x7fb09004ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb090048030 0x7fb09004ce70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb09806a9f0 tx=0x7fb098069a60 comp rx=0 tx=0).stop 2024-03-06T07:22:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 shutdown_connections 2024-03-06T07:22:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb07c06b4d0 0x7fb07c06d990 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb09004d3b0 0x7fb0900d9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb090048030 0x7fb09004ce70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 --2- 172.21.15.59:0/3106946944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb090006820 0x7fb090048900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 >> 172.21.15.59:0/3106946944 conn(0x7fb090009fa0 msgr2=0x7fb090037b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7f9d73fff640 1 -- 172.21.15.59:0/2651223498 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9d8005cc70 con 0x7f9d8c0c70f0 2024-03-06T07:22:00.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 shutdown_connections 2024-03-06T07:22:00.749 INFO:teuthology.orchestra.run.smithi059.stdout:55834574853 2024-03-06T07:22:00.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.745+0000 7fb09dde2640 1 -- 172.21.15.59:0/3106946944 wait complete. 2024-03-06T07:22:00.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.747+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e4c06b520 msgr2=0x7f2e4c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.747+0000 7f2e6b59e640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e4c06b520 0x7f2e4c06d9e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e60036000 tx=0x7f2e60008030 comp rx=0 tx=0).stop 2024-03-06T07:22:00.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.747+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e64006570 msgr2=0x7f2e6404b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.747+0000 7f2e6b59e640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e64006570 0x7f2e6404b960 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00c430 tx=0x7f2e5c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.747+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 shutdown_connections 2024-03-06T07:22:00.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.747+0000 7f2e6b59e640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e4c06b520 0x7f2e4c06d9e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.747+0000 7f2e6b59e640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e640e11f0 0x7f2e6404c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.747+0000 7f2e6b59e640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e640578e0 0x7f2e6404c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.747+0000 7f2e6b59e640 1 --2- 172.21.15.59:0/1345641125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e64006570 0x7f2e6404b960 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.747+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 >> 172.21.15.59:0/1345641125 conn(0x7f2e64020040 msgr2=0x7f2e64055900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 -- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f936406b540 msgr2=0x7f936406da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f936406b540 0x7f936406da00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9368000c00 tx=0x7f9368001e10 comp rx=0 tx=0).stop 2024-03-06T07:22:00.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 shutdown_connections 2024-03-06T07:22:00.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 -- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93780a95d0 msgr2=0x7f937815d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93780a95d0 0x7f937815d360 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9378070860 tx=0x7f9370004ab0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 -- 172.21.15.59:0/3781321593 shutdown_connections 2024-03-06T07:22:00.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f936406b540 0x7f936406da00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93780a95d0 0x7f937815d360 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93780ab150 0x7f937815ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 --2- 172.21.15.59:0/3781321593 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9360005210 0x7f93781f7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 -- 172.21.15.59:0/3781321593 >> 172.21.15.59:0/3781321593 conn(0x7f93780b1830 msgr2=0x7f93780b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f2e6b59e640 1 -- 172.21.15.59:0/1345641125 wait complete. 2024-03-06T07:22:00.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 -- 172.21.15.59:0/3781321593 shutdown_connections 2024-03-06T07:22:00.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f937e684640 1 -- 172.21.15.59:0/3781321593 wait complete. 2024-03-06T07:22:00.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.748+0000 7f9d90b38640 1 -- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b5a0 msgr2=0x7f9d7806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.749+0000 7f9d90b38640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b5a0 0x7f9d7806da60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9d84035000 tx=0x7f9d84030000 comp rx=0 tx=0).stop 2024-03-06T07:22:00.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.749+0000 7f9d90b38640 1 -- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d8c0c70f0 msgr2=0x7f9d8c1ebe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.749+0000 7f9d90b38640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d8c0c70f0 0x7f9d8c1ebe90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000e7f0 tx=0x7f9d8000ecb0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.749+0000 7f9d90b38640 1 -- 172.21.15.59:0/2651223498 shutdown_connections 2024-03-06T07:22:00.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.749+0000 7f9d90b38640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b5a0 0x7f9d7806da60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.749+0000 7f9d90b38640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d8c0c70f0 0x7f9d8c1ebe90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.749+0000 7f9d90b38640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d8c0c6ac0 0x7f9d8c15b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.749+0000 7f9d90b38640 1 --2- 172.21.15.59:0/2651223498 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d8c0b4f00 0x7f9d8c0b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.749+0000 7f9d90b38640 1 -- 172.21.15.59:0/2651223498 >> 172.21.15.59:0/2651223498 conn(0x7f9d8c09cee0 msgr2=0x7f9d8c0c7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.749+0000 7f9d90b38640 1 -- 172.21.15.59:0/2651223498 shutdown_connections 2024-03-06T07:22:00.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.749+0000 7f9d90b38640 1 -- 172.21.15.59:0/2651223498 wait complete. 2024-03-06T07:22:00.759 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-03-06T07:22:00.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.757+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f46b806dc80 con 0x7f46b80b38f0 2024-03-06T07:22:00.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.757+0000 7f46aeffd640 1 -- 172.21.15.59:0/3019544073 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f46b007b790 con 0x7f46b80b38f0 2024-03-06T07:22:00.759 INFO:teuthology.orchestra.run.smithi059.stdout:55834574853 2024-03-06T07:22:00.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469c06b520 msgr2=0x7f469c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469c06b520 0x7f469c06d9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f46b80af810 tx=0x7f46a8009240 comp rx=0 tx=0).stop 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b80b38f0 msgr2=0x7f46b81fe8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b80b38f0 0x7f46b81fe8c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f46b0033000 tx=0x7f46b0004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 shutdown_connections 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469c06b520 0x7f469c06d9e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b80d07f0 0x7f46b815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b80cfb20 0x7f46b81fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 --2- 172.21.15.59:0/3019544073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b80b38f0 0x7f46b81fe8c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 >> 172.21.15.59:0/3019544073 conn(0x7f46b806c0c0 msgr2=0x7f46b815eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 shutdown_connections 2024-03-06T07:22:00.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:00.761+0000 7f46bf531640 1 -- 172.21.15.59:0/3019544073 wait complete. 2024-03-06T07:22:00.767 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2024-03-06T07:22:00.768 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2024-03-06T07:22:00.770 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2024-03-06T07:22:00.771 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.2 2024-03-06T07:22:00.775 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2024-03-06T07:22:01.740 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:22:01.760 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:22:01.761 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:22:01.768 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:22:01.769 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:22:01.771 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:22:01.772 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:22:01.776 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:22:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.907+0000 7fc57b8b5640 1 Processor -- start 2024-03-06T07:22:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.907+0000 7fc57b8b5640 1 -- start start 2024-03-06T07:22:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.909+0000 7fc57b8b5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc57406fe70 0x7fc5741627a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.909+0000 7fc57b8b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc574163f90 0x7fc574164420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.909+0000 7fc57b8b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc574162e80 con 0x7fc57406de20 2024-03-06T07:22:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.909+0000 7fc57b8b5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc574163720 con 0x7fc57406fe70 2024-03-06T07:22:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.909+0000 7fc57b8b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc574163b00 con 0x7fc574163f90 2024-03-06T07:22:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.910+0000 7fc578e50640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc57406fe70 0x7fc5741627a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.910+0000 7fc578e50640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc57406fe70 0x7fc5741627a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58240/0 (socket says 172.21.15.59:58240) 2024-03-06T07:22:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.910+0000 7fc578e50640 1 -- 172.21.15.59:0/3844837573 learned_addr learned my addr 172.21.15.59:0/3844837573 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.910+0000 7fc578e50640 1 -- 172.21.15.59:0/3844837573 >> v2:172.21.15.164:3300/0 conn(0x7fc574163f90 msgr2=0x7fc574164420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.910+0000 7fc578e50640 1 --2- 172.21.15.59:0/3844837573 >> v2:172.21.15.164:3300/0 conn(0x7fc574163f90 0x7fc574164420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.910+0000 7fc578e50640 1 -- 172.21.15.59:0/3844837573 >> v1:172.21.15.164:6789/0 conn(0x7fc57406de20 legacy=0x7fc57406e200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:22:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.910+0000 7fc578e50640 1 -- 172.21.15.59:0/3844837573 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc574163160 con 0x7fc57406fe70 2024-03-06T07:22:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.912+0000 7fc578e50640 1 --2- 172.21.15.59:0/3844837573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc57406fe70 0x7fc5741627a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc574072e90 tx=0x7fc56c02c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47bcaa87c7cdf1d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.912+0000 7fc56bfff640 1 -- 172.21.15.59:0/3844837573 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56c02ca80 con 0x7fc57406fe70 2024-03-06T07:22:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.913+0000 7fc56bfff640 1 -- 172.21.15.59:0/3844837573 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56c02cc20 con 0x7fc57406fe70 2024-03-06T07:22:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.914+0000 7fc57b8b5640 1 -- 172.21.15.59:0/3844837573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc57406fe70 msgr2=0x7fc5741627a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.914+0000 7fc57b8b5640 1 --2- 172.21.15.59:0/3844837573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc57406fe70 0x7fc5741627a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc574072e90 tx=0x7fc56c02c0e0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.914+0000 7fc57b8b5640 1 -- 172.21.15.59:0/3844837573 shutdown_connections 2024-03-06T07:22:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.914+0000 7fc57b8b5640 1 --2- 172.21.15.59:0/3844837573 >> v2:172.21.15.164:3300/0 conn(0x7fc574163f90 0x7fc574164420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.914+0000 7fc57b8b5640 1 --2- 172.21.15.59:0/3844837573 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc57406fe70 0x7fc5741627a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.914+0000 7fc57b8b5640 1 -- 172.21.15.59:0/3844837573 >> 172.21.15.59:0/3844837573 conn(0x7fc5740d0a10 msgr2=0x7fc5740d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.914+0000 7fc57b8b5640 1 -- 172.21.15.59:0/3844837573 shutdown_connections 2024-03-06T07:22:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.915+0000 7fc57b8b5640 1 -- 172.21.15.59:0/3844837573 wait complete. 2024-03-06T07:22:01.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.916+0000 7fc57b8b5640 1 Processor -- start 2024-03-06T07:22:01.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.917+0000 7fc57b8b5640 1 -- start start 2024-03-06T07:22:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.918+0000 7fc57b8b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc57406de20 0x7fc574146ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.918+0000 7fc57b8b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc57406fe70 0x7fc5741511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.918+0000 7fc57b8b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc574163f90 0x7fc5741516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.919+0000 7fc578e50640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc57406fe70 0x7fc5741511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.919+0000 7fc578e50640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc57406fe70 0x7fc5741511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:46900/0 (socket says 172.21.15.59:46900) 2024-03-06T07:22:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.919+0000 7fc578e50640 1 -- 172.21.15.59:0/375845945 learned_addr learned my addr 172.21.15.59:0/375845945 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.919+0000 7fc579e52640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc574163f90 0x7fc5741516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.919+0000 7fc57b8b5640 1 -- 172.21.15.59:0/375845945 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57406b930 con 0x7fc574163f90 2024-03-06T07:22:01.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.919+0000 7fc57b8b5640 1 -- 172.21.15.59:0/375845945 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc57406d830 con 0x7fc57406fe70 2024-03-06T07:22:01.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.919+0000 7fc57b8b5640 1 -- 172.21.15.59:0/375845945 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57406bab0 con 0x7fc57406de20 2024-03-06T07:22:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.920+0000 7fc579e52640 1 -- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc57406de20 msgr2=0x7fc574146ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:22:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.920+0000 7fc579e52640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc57406de20 0x7fc574146ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.920+0000 7fc579e52640 1 -- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc57406fe70 msgr2=0x7fc5741511a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.920+0000 7fc579e52640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc57406fe70 0x7fc5741511a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.920+0000 7fc579e52640 1 -- 172.21.15.59:0/375845945 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5741478a0 con 0x7fc574163f90 2024-03-06T07:22:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.920+0000 7fc579e52640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc574163f90 0x7fc5741516e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc57000b500 tx=0x7fc57000b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.921+0000 7fc569ffb640 1 -- 172.21.15.59:0/375845945 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570014070 con 0x7fc574163f90 2024-03-06T07:22:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.921+0000 7fc57b8b5640 1 -- 172.21.15.59:0/375845945 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc574149650 con 0x7fc574163f90 2024-03-06T07:22:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.921+0000 7fc57b8b5640 1 -- 172.21.15.59:0/375845945 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc574149c10 con 0x7fc574163f90 2024-03-06T07:22:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.924+0000 7fc569ffb640 1 -- 172.21.15.59:0/375845945 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc570010680 con 0x7fc574163f90 2024-03-06T07:22:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.924+0000 7fc569ffb640 1 -- 172.21.15.59:0/375845945 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570010a70 con 0x7fc574163f90 2024-03-06T07:22:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.924+0000 7fc569ffb640 1 -- 172.21.15.59:0/375845945 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc570017070 con 0x7fc574163f90 2024-03-06T07:22:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.925+0000 7fc569ffb640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc55806b5a0 0x7fc55806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.925+0000 7fc579651640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc55806b5a0 0x7fc55806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.926+0000 7fc569ffb640 1 -- 172.21.15.59:0/375845945 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc570089e10 con 0x7fc574163f90 2024-03-06T07:22:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.926+0000 7fc579651640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc55806b5a0 0x7fc55806da60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc57414c750 tx=0x7fc564006bd0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.926+0000 7fc57b8b5640 1 -- 172.21.15.59:0/375845945 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc554005290 con 0x7fc574163f90 2024-03-06T07:22:01.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.936+0000 7fc569ffb640 1 -- 172.21.15.59:0/375845945 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc570058d90 con 0x7fc574163f90 2024-03-06T07:22:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.958+0000 7fab3759e640 1 Processor -- start 2024-03-06T07:22:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.958+0000 7fab3759e640 1 -- start start 2024-03-06T07:22:01.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7f059b59e640 1 Processor -- start 2024-03-06T07:22:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7fab3759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab300081c0 0x7fab30051760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7fab3759e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fab300548a0 0x7fab300f1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7fab3759e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab30048510 con 0x7fab3004fc00 2024-03-06T07:22:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7fab3759e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab300487b0 con 0x7fab300548a0 2024-03-06T07:22:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7fab3759e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab30048a70 con 0x7fab300081c0 2024-03-06T07:22:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7f72fd335640 1 Processor -- start 2024-03-06T07:22:01.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7f72fd335640 1 -- start start 2024-03-06T07:22:01.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7fab35d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab300081c0 0x7fab30051760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7fab35d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab300081c0 0x7fab30051760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43830/0 (socket says 172.21.15.59:43830) 2024-03-06T07:22:01.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7fab36d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fab300548a0 0x7fab300f1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7fab35d9b640 1 -- 172.21.15.59:0/440625478 learned_addr learned my addr 172.21.15.59:0/440625478 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:01.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.960+0000 7f059b59e640 1 -- start start 2024-03-06T07:22:01.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.961+0000 7fab36d9d640 1 -- 172.21.15.59:0/440625478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab300081c0 msgr2=0x7fab30051760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.961+0000 7fab36d9d640 1 --2- 172.21.15.59:0/440625478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab300081c0 0x7fab30051760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.961+0000 7fab36d9d640 1 -- 172.21.15.59:0/440625478 >> v1:172.21.15.164:6789/0 conn(0x7fab3004fc00 legacy=0x7fab3004ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.961+0000 7fab36d9d640 1 -- 172.21.15.59:0/440625478 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab30053090 con 0x7fab300548a0 2024-03-06T07:22:01.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.961+0000 7f72fd335640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f806b6c0 0x7f72f8165fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.961+0000 7f72fd335640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f806e3c0 con 0x7f72f814bb10 2024-03-06T07:22:01.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.961+0000 7f72fd335640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f81666d0 con 0x7f72f80ce8d0 2024-03-06T07:22:01.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.961+0000 7f72fd335640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f81669b0 con 0x7f72f806b6c0 2024-03-06T07:22:01.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.961+0000 7f72f6ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f72f814bb10 0x7f72f80cbae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51230/0 (socket says 172.21.15.59:51230) 2024-03-06T07:22:01.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.961+0000 7f72f6ffd640 1 -- 172.21.15.59:0/2331361706 learned_addr learned my addr 172.21.15.59:0/2331361706 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:01.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.961+0000 7f58cbf0f640 1 Processor -- start 2024-03-06T07:22:01.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.962+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210684058 0 0) 0x7f72f806e3c0 con 0x7f72f814bb10 2024-03-06T07:22:01.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.962+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e4003520 con 0x7f72f814bb10 2024-03-06T07:22:01.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.962+0000 7f059b59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05940e7680 0x7f0594052c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.962+0000 7f059b59e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0594051bb0 con 0x7f059404a3d0 2024-03-06T07:22:01.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.962+0000 7f059b59e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0594051e90 con 0x7f0594038e50 2024-03-06T07:22:01.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.962+0000 7f059b59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05940527c0 con 0x7f05940e7680 2024-03-06T07:22:01.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.962+0000 7fab36d9d640 1 --2- 172.21.15.59:0/440625478 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab300548a0 0x7fab300f1160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0097a0 tx=0x7fab2c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bbc2f9110bdb37c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.962+0000 7fab3559a640 1 -- 172.21.15.59:0/440625478 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c035020 con 0x7fab300548a0 2024-03-06T07:22:01.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7fab3559a640 1 -- 172.21.15.59:0/440625478 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2c031810 con 0x7fab300548a0 2024-03-06T07:22:01.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2978502673 0 0) 0x7f72e4003520 con 0x7f72f814bb10 2024-03-06T07:22:01.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f806b6c0 msgr2=0x7f72f8165fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:01.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 --2- 172.21.15.59:0/2331361706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f806b6c0 0x7f72f8165fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 >> v1:172.21.15.164:6789/0 conn(0x7f72f80ce8d0 legacy=0x7f72f806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7fab3559a640 1 -- 172.21.15.59:0/440625478 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c031be0 con 0x7fab300548a0 2024-03-06T07:22:01.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f8166c90 con 0x7f72f814bb10 2024-03-06T07:22:01.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f72e80019f0 con 0x7f72f814bb10 2024-03-06T07:22:01.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 >> v1:172.21.15.59:6789/0 conn(0x7f72f814bb10 legacy=0x7f72f80cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.962+0000 7f58cbf0f640 1 -- start start 2024-03-06T07:22:01.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 --2- 172.21.15.59:0/2331361706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72e40039e0 0x7f72e4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 --2- 172.21.15.59:0/2331361706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72e4004800 0x7f72e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 --2- 172.21.15.59:0/2331361706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e40052e0 0x7f72e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f81669b0 con 0x7f72e40052e0 2024-03-06T07:22:01.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e400c200 con 0x7f72e4004800 2024-03-06T07:22:01.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e400c400 con 0x7f72e40039e0 2024-03-06T07:22:01.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f0599d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f059404a3d0 0x7f0594038740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51246/0 (socket says 172.21.15.59:51246) 2024-03-06T07:22:01.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f0599d9b640 1 -- 172.21.15.59:0/2017784347 learned_addr learned my addr 172.21.15.59:0/2017784347 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:01.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f6ffd640 1 --2- 172.21.15.59:0/2331361706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72e40039e0 0x7f72e4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f72f77fe640 1 --2- 172.21.15.59:0/2331361706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e40052e0 0x7f72e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257734896 0 0) 0x7f0594051bb0 con 0x7f059404a3d0 2024-03-06T07:22:01.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0584003520 con 0x7f059404a3d0 2024-03-06T07:22:01.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7fab3759e640 1 -- 172.21.15.59:0/440625478 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab300548a0 msgr2=0x7fab300f1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7fab3759e640 1 --2- 172.21.15.59:0/440625478 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab300548a0 0x7fab300f1160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0097a0 tx=0x7fab2c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7fab3759e640 1 -- 172.21.15.59:0/440625478 shutdown_connections 2024-03-06T07:22:01.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059a59c640 1 --2- 172.21.15.59:0/2017784347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05940e7680 0x7f0594052c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7fab3759e640 1 --2- 172.21.15.59:0/440625478 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab300548a0 0x7fab300f1160 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7fab3759e640 1 --2- 172.21.15.59:0/440625478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab300081c0 0x7fab30051760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7fab3759e640 1 -- 172.21.15.59:0/440625478 >> 172.21.15.59:0/440625478 conn(0x7fab30055340 msgr2=0x7fab30055750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7fab3759e640 1 -- 172.21.15.59:0/440625478 shutdown_connections 2024-03-06T07:22:01.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7fab3759e640 1 -- 172.21.15.59:0/440625478 wait complete. 2024-03-06T07:22:01.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49650296 0 0) 0x7f0594051e90 con 0x7f0594038e50 2024-03-06T07:22:01.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f72f77fe640 1 -- 172.21.15.59:0/2331361706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72e40039e0 msgr2=0x7f72e4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0594051bb0 con 0x7f0594038e50 2024-03-06T07:22:01.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1956542031 0 0) 0x7f0584003520 con 0x7f059404a3d0 2024-03-06T07:22:01.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05940e7680 msgr2=0x7f0594052c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 --2- 172.21.15.59:0/2017784347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05940e7680 0x7f0594052c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 >> v1:172.21.15.164:6789/0 conn(0x7f0594038e50 legacy=0x7f0594007bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0594052170 con 0x7f059404a3d0 2024-03-06T07:22:01.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f72f77fe640 1 --2- 172.21.15.59:0/2331361706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72e40039e0 0x7f72e4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f059c062c00 con 0x7f059404a3d0 2024-03-06T07:22:01.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 >> v1:172.21.15.59:6789/0 conn(0x7f059404a3d0 legacy=0x7f0594038740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 --2- 172.21.15.59:0/2017784347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0584003b70 0x7f0584003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 --2- 172.21.15.59:0/2017784347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0584004800 0x7f0584004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 --2- 172.21.15.59:0/2017784347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05840051c0 0x7f058400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05940527c0 con 0x7f0584004800 2024-03-06T07:22:01.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058400bf00 con 0x7f05840051c0 2024-03-06T07:22:01.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058400c0b0 con 0x7f0584003b70 2024-03-06T07:22:01.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f72f77fe640 1 -- 172.21.15.59:0/2331361706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72e4004800 msgr2=0x7f72e4004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:01.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f72f77fe640 1 --2- 172.21.15.59:0/2331361706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72e4004800 0x7f72e4004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f72f77fe640 1 -- 172.21.15.59:0/2331361706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f8166c90 con 0x7f72e40052e0 2024-03-06T07:22:01.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f0599d9b640 1 --2- 172.21.15.59:0/2017784347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0584004800 0x7f0584004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f46bb415640 1 Processor -- start 2024-03-06T07:22:01.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f46bb415640 1 -- start start 2024-03-06T07:22:01.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f46bb415640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f46b406b6c0 0x7f46b4071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f46b89b0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f46b406b6c0 0x7f46b4071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f46b89b0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f46b406b6c0 0x7f46b4071770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58286/0 (socket says 172.21.15.59:58286) 2024-03-06T07:22:01.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f46b89b0640 1 -- 172.21.15.59:0/1244519387 learned_addr learned my addr 172.21.15.59:0/1244519387 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:01.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f46bb415640 1 --2- 172.21.15.59:0/1244519387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b40cece0 0x7f46b40cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f46bb415640 1 -- 172.21.15.59:0/1244519387 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b416b3a0 con 0x7f46b4165190 2024-03-06T07:22:01.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f46bb415640 1 -- 172.21.15.59:0/1244519387 --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b416bfb0 con 0x7f46b406b6c0 2024-03-06T07:22:01.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f46bb415640 1 -- 172.21.15.59:0/1244519387 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b416b680 con 0x7f46b40cece0 2024-03-06T07:22:01.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f58cbf0f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58c406de20 0x7f58c406fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f58cbf0f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c4162e80 con 0x7f58c4163f90 2024-03-06T07:22:01.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f58cbf0f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c4163160 con 0x7f58c41627a0 2024-03-06T07:22:01.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f58cbf0f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c4163720 con 0x7f58c406de20 2024-03-06T07:22:01.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f58c9cab640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58c406de20 0x7f58c406fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f58c9cab640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58c406de20 0x7f58c406fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58260/0 (socket says 172.21.15.59:58260) 2024-03-06T07:22:01.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f58c9cab640 1 -- 172.21.15.59:0/3479826253 learned_addr learned my addr 172.21.15.59:0/3479826253 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:01.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f58c9cab640 1 -- 172.21.15.59:0/3479826253 >> v1:172.21.15.164:6789/0 conn(0x7f58c41627a0 legacy=0x7f58c406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f58c9cab640 1 -- 172.21.15.59:0/3479826253 >> v1:172.21.15.59:6789/0 conn(0x7f58c4163f90 legacy=0x7f58c4164440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f58c9cab640 1 -- 172.21.15.59:0/3479826253 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c4163440 con 0x7f58c406de20 2024-03-06T07:22:01.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0599d9b640 1 -- 172.21.15.59:0/2017784347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0584003b70 msgr2=0x7f0584003f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:22:01.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.963+0000 7f0c6010d640 1 Processor -- start 2024-03-06T07:22:01.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.964+0000 7f0c6010d640 1 -- start start 2024-03-06T07:22:01.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7fab3759e640 1 Processor -- start 2024-03-06T07:22:01.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0c6010d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0c5814bb10 0x7f0c58165fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0c6010d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5806afb0 0x7f0c5806b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0c6010d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5806e3c0 con 0x7f0c580ce8d0 2024-03-06T07:22:01.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0c5d6a8640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0c580ce8d0 0x7f0c580cbae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:48118/0 (socket says 172.21.15.59:48118) 2024-03-06T07:22:01.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0c5d6a8640 1 -- 172.21.15.59:0/1337851937 learned_addr learned my addr 172.21.15.59:0/1337851937 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:01.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f58c9cab640 1 --2- 172.21.15.59:0/3479826253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58c406de20 0x7f58c406fe70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009e50 tx=0x7f58bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81be8329fe34011 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0c6010d640 1 -- 172.21.15.59:0/1337851937 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c5806b940 con 0x7f0c5814bb10 2024-03-06T07:22:01.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0c6010d640 1 -- 172.21.15.59:0/1337851937 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c581666d0 con 0x7f0c5806afb0 2024-03-06T07:22:01.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f72f77fe640 1 --2- 172.21.15.59:0/2331361706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e40052e0 0x7f72e400b9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f72f80cdcd0 tx=0x7f72e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b9c038490f885b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7fab3759e640 1 -- start start 2024-03-06T07:22:01.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0a1b433640 1 Processor -- start 2024-03-06T07:22:01.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0a1b433640 1 -- start start 2024-03-06T07:22:01.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0599d9b640 1 --2- 172.21.15.59:0/2017784347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0584003b70 0x7f0584003f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0599d9b640 1 -- 172.21.15.59:0/2017784347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05840051c0 msgr2=0x7f058400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0599d9b640 1 --2- 172.21.15.59:0/2017784347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05840051c0 0x7f058400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605778226 0 0) 0x7f0c5806e3c0 con 0x7f0c580ce8d0 2024-03-06T07:22:01.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c44003520 con 0x7f0c580ce8d0 2024-03-06T07:22:01.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5e6aa640 1 --2- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5806afb0 0x7f0c5806b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5dea9640 1 --2- 172.21.15.59:0/1337851937 >> v2:172.21.15.164:3300/0 conn(0x7f0c5814bb10 0x7f0c58165fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0037070 con 0x7f72e40052e0 2024-03-06T07:22:01.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f0599d9b640 1 -- 172.21.15.59:0/2017784347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0594052170 con 0x7f0584004800 2024-03-06T07:22:01.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f0032600 con 0x7f72e40052e0 2024-03-06T07:22:01.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f72f5ffb640 1 -- 172.21.15.59:0/2331361706 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f00093b0 con 0x7f72e40052e0 2024-03-06T07:22:01.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3101528495 0 0) 0x7f0c44003520 con 0x7f0c580ce8d0 2024-03-06T07:22:01.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5806afb0 msgr2=0x7f0c5806b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 --2- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5806afb0 0x7f0c5806b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.965+0000 7f46b99b2640 1 --2- 172.21.15.59:0/1244519387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b40cece0 0x7f46b40cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f46b89b0640 1 -- 172.21.15.59:0/1244519387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b40cece0 msgr2=0x7f46b40cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f46b89b0640 1 --2- 172.21.15.59:0/1244519387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b40cece0 0x7f46b40cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f46b89b0640 1 -- 172.21.15.59:0/1244519387 >> v1:172.21.15.59:6789/0 conn(0x7f46b4165190 legacy=0x7f46b406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f46b89b0640 1 -- 172.21.15.59:0/1244519387 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b416b960 con 0x7f46b406b6c0 2024-03-06T07:22:01.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 >> v2:172.21.15.164:3300/0 conn(0x7f0c5814bb10 msgr2=0x7f0c58165fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 --2- 172.21.15.59:0/1337851937 >> v2:172.21.15.164:3300/0 conn(0x7f0c5814bb10 0x7f0c58165fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c581669b0 con 0x7f0c580ce8d0 2024-03-06T07:22:01.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f0c4c001670 con 0x7f0c580ce8d0 2024-03-06T07:22:01.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 >> v1:172.21.15.164:6789/0 conn(0x7f0c580ce8d0 legacy=0x7f0c580cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 --2- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c440039b0 0x7f0c44003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f58c8ca9640 1 -- 172.21.15.59:0/3479826253 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc02ca80 con 0x7f58c406de20 2024-03-06T07:22:01.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.966+0000 7f0c5cea7640 1 --2- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5806afb0 0x7f0c44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0c5dea9640 1 --2- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c440039b0 0x7f0c44003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0a1b433640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c0df210 0x7f0a0c038070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f58c8ca9640 1 -- 172.21.15.59:0/3479826253 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58bc02cc20 con 0x7f58c406de20 2024-03-06T07:22:01.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0c5cea7640 1 --2- 172.21.15.59:0/1337851937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c44004d40 0x7f0c440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0a1a431640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c0df210 0x7f0a0c038070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0a1a431640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c0df210 0x7f0a0c038070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43900/0 (socket says 172.21.15.59:43900) 2024-03-06T07:22:01.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0a1a431640 1 -- 172.21.15.59:0/2491325469 learned_addr learned my addr 172.21.15.59:0/2491325469 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:01.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0c5d6a8640 1 --2- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5806afb0 0x7f0c44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0a1b433640 1 -- 172.21.15.59:0/2491325469 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0c0544b0 con 0x7f0a0c050190 2024-03-06T07:22:01.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f58c8ca9640 1 -- 172.21.15.59:0/3479826253 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc0343f0 con 0x7f58c406de20 2024-03-06T07:22:01.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0a1b433640 1 -- 172.21.15.59:0/2491325469 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0c0541d0 con 0x7f0a0c053c60 2024-03-06T07:22:01.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0a1b433640 1 -- 172.21.15.59:0/2491325469 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c005f20 con 0x7f0a0c0df210 2024-03-06T07:22:01.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0599d9b640 1 --2- 172.21.15.59:0/2017784347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0584004800 0x7f0584004c80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f059c050c30 tx=0x7f0584005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a37ff915c1bea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7fab3759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab300081c0 0x7fab300d7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7fab3759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab3004fc00 0x7fab300d8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7fab3759e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab300548a0 0x7fab3004ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7fab3759e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab300f16a0 con 0x7fab300081c0 2024-03-06T07:22:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7fab3759e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab300506f0 con 0x7fab300548a0 2024-03-06T07:22:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7fab3759e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab300f1820 con 0x7fab3004fc00 2024-03-06T07:22:01.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5806b940 con 0x7f0c44004d40 2024-03-06T07:22:01.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c581666d0 con 0x7f0c440039b0 2024-03-06T07:22:01.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4400bb50 con 0x7f0c5806afb0 2024-03-06T07:22:01.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7fab35d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab3004fc00 0x7fab300d8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7fab35d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab3004fc00 0x7fab300d8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43908/0 (socket says 172.21.15.59:43908) 2024-03-06T07:22:01.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7fab35d9b640 1 -- 172.21.15.59:0/2061309397 learned_addr learned my addr 172.21.15.59:0/2061309397 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:01.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f46b89b0640 1 --2- 172.21.15.59:0/1244519387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b406b6c0 0x7f46b4071770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f46b40cdb90 tx=0x7f46a402fe80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766dc4b767668544 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7fab36d9d640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab300548a0 0x7fab3004ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0c5d6a8640 1 -- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c440039b0 msgr2=0x7f0c44003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0c5d6a8640 1 --2- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c440039b0 0x7f0c44003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0c5d6a8640 1 -- 172.21.15.59:0/1337851937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c44004d40 msgr2=0x7f0c440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0c5d6a8640 1 --2- 172.21.15.59:0/1337851937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c44004d40 0x7f0c440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f72fd335640 1 -- 172.21.15.59:0/2331361706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e40052e0 msgr2=0x7f72e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f0c5d6a8640 1 -- 172.21.15.59:0/1337851937 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c44003520 con 0x7f0c5806afb0 2024-03-06T07:22:01.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f72fd335640 1 --2- 172.21.15.59:0/2331361706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e40052e0 0x7f72e400b9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f72f80cdcd0 tx=0x7f72e4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:01.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0a1a431640 1 -- 172.21.15.59:0/2491325469 >> v1:172.21.15.164:6789/0 conn(0x7f0a0c053c60 legacy=0x7f0a0c04fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0a1a431640 1 -- 172.21.15.59:0/2491325469 >> v1:172.21.15.59:6789/0 conn(0x7f0a0c050190 legacy=0x7f0a0c006a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.967+0000 7f0a1a431640 1 -- 172.21.15.59:0/2491325469 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0c006200 con 0x7f0a0c0df210 2024-03-06T07:22:01.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f72f77fe640 1 -- 172.21.15.59:0/2331361706 reap_dead start 2024-03-06T07:22:01.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f72fd335640 1 -- 172.21.15.59:0/2331361706 shutdown_connections 2024-03-06T07:22:01.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c0b4070 con 0x7f0584004800 2024-03-06T07:22:01.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3479826253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58c406de20 msgr2=0x7f58c406fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f58cbf0f640 1 --2- 172.21.15.59:0/3479826253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58c406de20 0x7f58c406fe70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009e50 tx=0x7f58bc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059c0af440 con 0x7f0584004800 2024-03-06T07:22:01.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f46abfff640 1 -- 172.21.15.59:0/1244519387 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a4030c30 con 0x7f46b406b6c0 2024-03-06T07:22:01.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f46abfff640 1 -- 172.21.15.59:0/1244519387 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a400ae50 con 0x7f46b406b6c0 2024-03-06T07:22:01.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f46abfff640 1 -- 172.21.15.59:0/1244519387 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a40383f0 con 0x7f46b406b6c0 2024-03-06T07:22:01.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3479826253 shutdown_connections 2024-03-06T07:22:01.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f58cbf0f640 1 --2- 172.21.15.59:0/3479826253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58c406de20 0x7f58c406fe70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3479826253 >> 172.21.15.59:0/3479826253 conn(0x7f58c40d0a10 msgr2=0x7f58c40d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3479826253 shutdown_connections 2024-03-06T07:22:01.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3479826253 wait complete. 2024-03-06T07:22:01.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7fab3659c640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab300081c0 0x7fab300d7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7fab35d9b640 1 -- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab300548a0 msgr2=0x7fab3004ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7fab35d9b640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab300548a0 0x7fab3004ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7fab35d9b640 1 -- 172.21.15.59:0/2061309397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab300081c0 msgr2=0x7fab300d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7fab35d9b640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab300081c0 0x7fab300d7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7fab35d9b640 1 -- 172.21.15.59:0/2061309397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab300d8f30 con 0x7fab3004fc00 2024-03-06T07:22:01.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f059959a640 1 -- 172.21.15.59:0/2017784347 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c0b4070 con 0x7f0584004800 2024-03-06T07:22:01.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f0a1a431640 1 --2- 172.21.15.59:0/2491325469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c0df210 0x7f0a0c038070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0a04007390 tx=0x7f0a0402da50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e30733c84faac33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.968+0000 7f72fd335640 1 -- 172.21.15.59:0/2331361706 >> 172.21.15.59:0/2331361706 conn(0x7f72f80b1830 msgr2=0x7f72f80b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f72fd335640 1 -- 172.21.15.59:0/2331361706 shutdown_connections 2024-03-06T07:22:01.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f0a1942f640 1 -- 172.21.15.59:0/2491325469 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a04036020 con 0x7f0a0c0df210 2024-03-06T07:22:01.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f72fd335640 1 -- 172.21.15.59:0/2331361706 wait complete. 2024-03-06T07:22:01.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7fab35d9b640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab3004fc00 0x7fab300d8470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fab38050c30 tx=0x7fab380683c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f46bb415640 1 -- 172.21.15.59:0/1244519387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b406b6c0 msgr2=0x7f46b4071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f46bb415640 1 --2- 172.21.15.59:0/1244519387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b406b6c0 0x7f46b4071770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f46b40cdb90 tx=0x7f46a402fe80 comp rx=0 tx=0).stop 2024-03-06T07:22:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f46bb415640 1 -- 172.21.15.59:0/1244519387 shutdown_connections 2024-03-06T07:22:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f46bb415640 1 --2- 172.21.15.59:0/1244519387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b40cece0 0x7f46b40cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f46bb415640 1 --2- 172.21.15.59:0/1244519387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b406b6c0 0x7f46b4071770 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f46bb415640 1 -- 172.21.15.59:0/1244519387 >> 172.21.15.59:0/1244519387 conn(0x7f46b40b1830 msgr2=0x7f46b40b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f46bb415640 1 -- 172.21.15.59:0/1244519387 shutdown_connections 2024-03-06T07:22:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f46bb415640 1 -- 172.21.15.59:0/1244519387 wait complete. 2024-03-06T07:22:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f0a1942f640 1 -- 172.21.15.59:0/2491325469 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a04008df0 con 0x7f0a0c0df210 2024-03-06T07:22:02.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f0a1942f640 1 -- 172.21.15.59:0/2491325469 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0400a2f0 con 0x7f0a0c0df210 2024-03-06T07:22:02.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f0a1b433640 1 -- 172.21.15.59:0/2491325469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c0df210 msgr2=0x7f0a0c038070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f0a1b433640 1 --2- 172.21.15.59:0/2491325469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c0df210 0x7f0a0c038070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0a04007390 tx=0x7f0a0402da50 comp rx=0 tx=0).stop 2024-03-06T07:22:02.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f58cbf0f640 1 Processor -- start 2024-03-06T07:22:02.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f58cbf0f640 1 -- start start 2024-03-06T07:22:02.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7fab26ffd640 1 -- 172.21.15.59:0/2061309397 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38073660 con 0x7fab3004fc00 2024-03-06T07:22:02.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7fab3759e640 1 -- 172.21.15.59:0/2061309397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab300d9210 con 0x7fab3004fc00 2024-03-06T07:22:02.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7fab3759e640 1 -- 172.21.15.59:0/2061309397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab300d97d0 con 0x7fab3004fc00 2024-03-06T07:22:02.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f059b59e640 1 -- 172.21.15.59:0/2017784347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0584004800 msgr2=0x7f0584004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f059b59e640 1 --2- 172.21.15.59:0/2017784347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0584004800 0x7f0584004c80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f059c050c30 tx=0x7f0584005830 comp rx=0 tx=0).stop 2024-03-06T07:22:02.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f059ad9d640 1 -- 172.21.15.59:0/2017784347 reap_dead start 2024-03-06T07:22:02.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f059b59e640 1 -- 172.21.15.59:0/2017784347 shutdown_connections 2024-03-06T07:22:02.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f059b59e640 1 -- 172.21.15.59:0/2017784347 >> 172.21.15.59:0/2017784347 conn(0x7f0594053740 msgr2=0x7f0594053b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f059b59e640 1 -- 172.21.15.59:0/2017784347 shutdown_connections 2024-03-06T07:22:02.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f059b59e640 1 -- 172.21.15.59:0/2017784347 wait complete. 2024-03-06T07:22:02.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f0c5d6a8640 1 --2- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5806afb0 0x7f0c44004800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c00c160 tx=0x7f0c44005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440ed58b43912796 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c007c80 con 0x7f0c5806afb0 2024-03-06T07:22:02.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c030070 con 0x7f0c5806afb0 2024-03-06T07:22:02.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.969+0000 7f0c5cea7640 1 -- 172.21.15.59:0/1337851937 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c005dd0 con 0x7f0c5806afb0 2024-03-06T07:22:02.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f0c6010d640 1 -- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5806afb0 msgr2=0x7f0c44004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f0c6010d640 1 --2- 172.21.15.59:0/1337851937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5806afb0 0x7f0c44004800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c00c160 tx=0x7f0c44005830 comp rx=0 tx=0).stop 2024-03-06T07:22:02.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f0c5e6aa640 1 -- 172.21.15.59:0/1337851937 reap_dead start 2024-03-06T07:22:02.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f0c6010d640 1 -- 172.21.15.59:0/1337851937 shutdown_connections 2024-03-06T07:22:02.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f0c6010d640 1 -- 172.21.15.59:0/1337851937 >> 172.21.15.59:0/1337851937 conn(0x7f0c580b1830 msgr2=0x7f0c580b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7fab26ffd640 1 -- 172.21.15.59:0/2061309397 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab38094440 con 0x7fab3004fc00 2024-03-06T07:22:02.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.971+0000 7fab26ffd640 1 -- 172.21.15.59:0/2061309397 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab380947f0 con 0x7fab3004fc00 2024-03-06T07:22:02.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.971+0000 7f72fd335640 1 Processor -- start 2024-03-06T07:22:02.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.971+0000 7f0a1b433640 1 -- 172.21.15.59:0/2491325469 shutdown_connections 2024-03-06T07:22:02.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.971+0000 7f72fd335640 1 -- start start 2024-03-06T07:22:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.971+0000 7f0a1b433640 1 --2- 172.21.15.59:0/2491325469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c0df210 0x7f0a0c038070 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.971+0000 7f0a1b433640 1 -- 172.21.15.59:0/2491325469 >> 172.21.15.59:0/2491325469 conn(0x7f0a0c009fa0 msgr2=0x7f0a0c00a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.971+0000 7fab24ff9640 1 -- 172.21.15.59:0/2061309397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab300057d0 con 0x7fab3004fc00 2024-03-06T07:22:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f0c6010d640 1 -- 172.21.15.59:0/1337851937 shutdown_connections 2024-03-06T07:22:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.970+0000 7f0c6010d640 1 -- 172.21.15.59:0/1337851937 wait complete. 2024-03-06T07:22:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f58cbf0f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58c41627a0 0x7f58c414ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.971+0000 7f46bb415640 1 Processor -- start 2024-03-06T07:22:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.971+0000 7f46bb415640 1 -- start start 2024-03-06T07:22:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f059b59e640 1 Processor -- start 2024-03-06T07:22:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f059b59e640 1 -- start start 2024-03-06T07:22:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f58cbf0f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58c4163f90 0x7f58c40a9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f58c9cab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58c41627a0 0x7f58c414ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f58c9cab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58c41627a0 0x7f58c414ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:46980/0 (socket says 172.21.15.59:46980) 2024-03-06T07:22:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f0c6010d640 1 Processor -- start 2024-03-06T07:22:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f58c9cab640 1 -- 172.21.15.59:0/3751053450 learned_addr learned my addr 172.21.15.59:0/3751053450 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:02.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f0c6010d640 1 -- start start 2024-03-06T07:22:02.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f58cbf0f640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58c40b4850 0x7f58c40b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c4162c20 con 0x7f58c4163f90 2024-03-06T07:22:02.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f58c94aa640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58c4163f90 0x7f58c40a9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.972+0000 7f58ca4ac640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58c40b4850 0x7f58c40b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f72fd335640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72f81478a0 0x7f72f81fba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f72fd335640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8161740 0x7f72f81649f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c4163b00 con 0x7f58c41627a0 2024-03-06T07:22:02.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7fab26ffd640 1 -- 172.21.15.59:0/2061309397 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab3809d440 con 0x7fab3004fc00 2024-03-06T07:22:02.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f46bb415640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b406b6c0 0x7f46b415be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f46bb415640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b40cece0 0x7f46b415e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c406b930 con 0x7f58c40b4850 2024-03-06T07:22:02.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f46bb415640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b4165190 0x7f46b41680d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f46b89b0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b40cece0 0x7f46b415e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f46b89b0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b40cece0 0x7f46b415e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:46994/0 (socket says 172.21.15.59:46994) 2024-03-06T07:22:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f46b89b0640 1 -- 172.21.15.59:0/1318713104 learned_addr learned my addr 172.21.15.59:0/1318713104 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f0a1b433640 1 -- 172.21.15.59:0/2491325469 shutdown_connections 2024-03-06T07:22:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f0a1b433640 1 -- 172.21.15.59:0/2491325469 wait complete. 2024-03-06T07:22:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f58c9cab640 1 -- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58c40b4850 msgr2=0x7f58c40b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f58c9cab640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58c40b4850 0x7f58c40b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f58c9cab640 1 -- 172.21.15.59:0/3751053450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58c4163f90 msgr2=0x7f58c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f72fd335640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72f8147c80 0x7f72f8164f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f72fd335640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f814bef0 con 0x7f72f8147c80 2024-03-06T07:22:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f72fd335640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8071770 con 0x7f72f81478a0 2024-03-06T07:22:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f72fd335640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f814c070 con 0x7f72f8161740 2024-03-06T07:22:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7fab26ffd640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fab1806b520 0x7fab1806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7fab26ffd640 1 -- 172.21.15.59:0/2061309397 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab38105070 con 0x7fab3004fc00 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059b59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0584004800 0x7f059404f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059b59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f059403c2d0 0x7f059403a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059b59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f059417d560 0x7f0594054ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059b59e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05940517d0 con 0x7f059417d560 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059b59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0594055be0 con 0x7f059403c2d0 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059b59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0594051950 con 0x7f0584004800 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f059417d560 0x7f0594054ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059a59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0584004800 0x7f059404f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f059417d560 0x7f0594054ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58312/0 (socket says 172.21.15.59:58312) 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059a59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0584004800 0x7f059404f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43938/0 (socket says 172.21.15.59:43938) 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059ad9d640 1 -- 172.21.15.59:0/2841917982 learned_addr learned my addr 172.21.15.59:0/2841917982 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f0599d9b640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f059403c2d0 0x7f059403a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059a59c640 1 -- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f059403c2d0 msgr2=0x7f059403a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059a59c640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f059403c2d0 0x7f059403a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059a59c640 1 -- 172.21.15.59:0/2841917982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f059417d560 msgr2=0x7f0594054ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059a59c640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f059417d560 0x7f0594054ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.973+0000 7f059a59c640 1 -- 172.21.15.59:0/2841917982 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059417df30 con 0x7f0584004800 2024-03-06T07:22:02.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f58c9cab640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58c4163f90 0x7f58c40a9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f0c6010d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5806afb0 0x7f0c58148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f0c6010d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c5814a7e0 0x7f0c58160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f58c9cab640 1 -- 172.21.15.59:0/3751053450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c41f9430 con 0x7f58c41627a0 2024-03-06T07:22:02.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f58c94aa640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58c4163f90 0x7f58c40a9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:02.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f46b99b2640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b4165190 0x7f46b41680d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b4165570 con 0x7f46b406b6c0 2024-03-06T07:22:02.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40cf690 con 0x7f46b40cece0 2024-03-06T07:22:02.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f059a59c640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0584004800 0x7f059404f470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f059400fe20 tx=0x7f059000d8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b416e880 con 0x7f46b4165190 2024-03-06T07:22:02.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7fab3659c640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fab1806b520 0x7fab1806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f72f67fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8161740 0x7f72f81649f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f72f6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72f81478a0 0x7f72f81fba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f46b91b1640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b406b6c0 0x7f46b415be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f72f6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72f81478a0 0x7f72f81fba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:47016/0 (socket says 172.21.15.59:47016) 2024-03-06T07:22:02.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f72f6ffd640 1 -- 172.21.15.59:0/3754824410 learned_addr learned my addr 172.21.15.59:0/3754824410 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:02.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f72f67fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8161740 0x7f72f81649f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:43958/0 (socket says 172.21.15.59:43958) 2024-03-06T07:22:02.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f58c9cab640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58c41627a0 0x7f58c414ec10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f58bc02c470 tx=0x7f58bc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f0c6010d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c58131520 0x7f0c58135170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f0c6010d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c58071770 con 0x7f0c5814a7e0 2024-03-06T07:22:02.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f0c5dea9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5806afb0 0x7f0c58148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f0c5dea9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5806afb0 0x7f0c58148be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:47010/0 (socket says 172.21.15.59:47010) 2024-03-06T07:22:02.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.974+0000 7f0c5dea9640 1 -- 172.21.15.59:0/178810455 learned_addr learned my addr 172.21.15.59:0/178810455 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:02.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72f77fe640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72f8147c80 0x7f72f8164f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72f6ffd640 1 -- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8161740 msgr2=0x7f72f81649f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72f6ffd640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8161740 0x7f72f81649f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72f6ffd640 1 -- 172.21.15.59:0/3754824410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72f8147c80 msgr2=0x7f72f8164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72f6ffd640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72f8147c80 0x7f72f8164f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72f6ffd640 1 -- 172.21.15.59:0/3754824410 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f81fc950 con 0x7f72f81478a0 2024-03-06T07:22:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7fab3659c640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fab1806b520 0x7fab1806d9e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fab28000f60 tx=0x7fab28009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0c5d6a8640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c5814a7e0 0x7f0c58160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c58071ac0 con 0x7f0c5806afb0 2024-03-06T07:22:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5814bef0 con 0x7f0c58131520 2024-03-06T07:22:02.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f46b91b1640 1 -- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b4165190 msgr2=0x7f46b41680d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0c5e6aa640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c58131520 0x7f0c58135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f46b91b1640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b4165190 0x7f46b41680d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f58ba7fc640 1 -- 172.21.15.59:0/3751053450 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc043020 con 0x7f58c41627a0 2024-03-06T07:22:02.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f58ba7fc640 1 -- 172.21.15.59:0/3751053450 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58bc03e450 con 0x7f58c41627a0 2024-03-06T07:22:02.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c41f9710 con 0x7f58c41627a0 2024-03-06T07:22:02.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f46b91b1640 1 -- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b40cece0 msgr2=0x7f46b415e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f58ba7fc640 1 -- 172.21.15.59:0/3751053450 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc03e840 con 0x7f58c41627a0 2024-03-06T07:22:02.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c41f9cd0 con 0x7f58c41627a0 2024-03-06T07:22:02.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f46b91b1640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b40cece0 0x7f46b415e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72f6ffd640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72f81478a0 0x7f72f81fba40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f72e800e480 tx=0x7f72e800e570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f46b91b1640 1 -- 172.21.15.59:0/1318713104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b416c4a0 con 0x7f46b406b6c0 2024-03-06T07:22:02.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0a1b433640 1 Processor -- start 2024-03-06T07:22:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0c5dea9640 1 -- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c58131520 msgr2=0x7f0c58135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0c5dea9640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c58131520 0x7f0c58135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0c5dea9640 1 -- 172.21.15.59:0/178810455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c5814a7e0 msgr2=0x7f0c58160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0c5dea9640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c5814a7e0 0x7f0c58160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0a1b433640 1 -- start start 2024-03-06T07:22:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0c5dea9640 1 -- 172.21.15.59:0/178810455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c581491c0 con 0x7f0c5806afb0 2024-03-06T07:22:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f0c5d6a8640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c5814a7e0 0x7f0c58160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72ef7fe640 1 -- 172.21.15.59:0/3754824410 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8013420 con 0x7f72f81478a0 2024-03-06T07:22:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72ef7fe640 1 -- 172.21.15.59:0/3754824410 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e8018070 con 0x7f72f81478a0 2024-03-06T07:22:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72ef7fe640 1 -- 172.21.15.59:0/3754824410 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8013820 con 0x7f72f81478a0 2024-03-06T07:22:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72fd335640 1 -- 172.21.15.59:0/3754824410 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f81fc390 con 0x7f72f81478a0 2024-03-06T07:22:02.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.975+0000 7f72fd335640 1 -- 172.21.15.59:0/3754824410 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f8145510 con 0x7f72f81478a0 2024-03-06T07:22:02.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.976+0000 7f46b91b1640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b406b6c0 0x7f46b415be90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f46b000b750 tx=0x7f46b000bc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.976+0000 7f0c5dea9640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5806afb0 0x7f0c58148be0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000b850 tx=0x7f0c5000bd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.976+0000 7f0c4a7fc640 1 -- 172.21.15.59:0/178810455 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50018070 con 0x7f0c5806afb0 2024-03-06T07:22:02.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.976+0000 7f0c4a7fc640 1 -- 172.21.15.59:0/178810455 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5000be40 con 0x7f0c5806afb0 2024-03-06T07:22:02.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.976+0000 7f46a9ffb640 1 -- 172.21.15.59:0/1318713104 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0004220 con 0x7f46b406b6c0 2024-03-06T07:22:02.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.976+0000 7f46a9ffb640 1 -- 172.21.15.59:0/1318713104 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b00043c0 con 0x7f46b406b6c0 2024-03-06T07:22:02.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.976+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b416c780 con 0x7f46b406b6c0 2024-03-06T07:22:02.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.976+0000 7f46a9ffb640 1 -- 172.21.15.59:0/1318713104 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0003830 con 0x7f46b406b6c0 2024-03-06T07:22:02.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.976+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c581494a0 con 0x7f0c5806afb0 2024-03-06T07:22:02.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0c4a7fc640 1 -- 172.21.15.59:0/178810455 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50018070 con 0x7f0c5806afb0 2024-03-06T07:22:02.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c58149a60 con 0x7f0c5806afb0 2024-03-06T07:22:02.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.978+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b416cd40 con 0x7f46b406b6c0 2024-03-06T07:22:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0a1b433640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c050190 0x7f0a0c04c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0a1b433640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c053c60 0x7f0a0c0d5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0a1b433640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a0c0df210 0x7f0a0c04ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0a1a431640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c050190 0x7f0a0c04c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0a1a431640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c050190 0x7f0a0c04c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58340/0 (socket says 172.21.15.59:58340) 2024-03-06T07:22:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0a1a431640 1 -- 172.21.15.59:0/3070088364 learned_addr learned my addr 172.21.15.59:0/3070088364 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0a19c30640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c053c60 0x7f0a0c0d5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c037a80 con 0x7f0a0c050190 2024-03-06T07:22:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0a1ac32640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a0c0df210 0x7f0a0c04ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c037900 con 0x7f0a0c0df210 2024-03-06T07:22:02.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c0df5f0 con 0x7f0a0c053c60 2024-03-06T07:22:02.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c404f330 con 0x7f58c41627a0 2024-03-06T07:22:02.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.977+0000 7f72fd335640 1 -- 172.21.15.59:0/3754824410 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f804f330 con 0x7f72f81478a0 2024-03-06T07:22:02.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.978+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5804f330 con 0x7f0c5806afb0 2024-03-06T07:22:02.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.978+0000 7f72ef7fe640 1 -- 172.21.15.59:0/3754824410 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72e80139c0 con 0x7f72f81478a0 2024-03-06T07:22:02.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.978+0000 7f72ef7fe640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b520 0x7f72d806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.978+0000 7f0a1a431640 1 -- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c053c60 msgr2=0x7f0a0c0d5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.978+0000 7f0a1a431640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c053c60 0x7f0a0c0d5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.978+0000 7f0a1a431640 1 -- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a0c0df210 msgr2=0x7f0a0c04ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.978+0000 7f0a1a431640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a0c0df210 0x7f0a0c04ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.978+0000 7f0a1a431640 1 -- 172.21.15.59:0/3070088364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0c0d2130 con 0x7f0a0c050190 2024-03-06T07:22:02.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.979+0000 7f58ba7fc640 1 -- 172.21.15.59:0/3751053450 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58bc046020 con 0x7f58c41627a0 2024-03-06T07:22:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.979+0000 7f0a1a431640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c050190 0x7f0a0c04c590 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009fd0 tx=0x7f0a04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.979+0000 7f72ef7fe640 1 -- 172.21.15.59:0/3754824410 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72e8057310 con 0x7f72f81478a0 2024-03-06T07:22:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.979+0000 7f72f67fc640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b520 0x7f72d806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.979+0000 7f58ba7fc640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58a806b4d0 0x7f58a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.979+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b404f330 con 0x7f46b406b6c0 2024-03-06T07:22:02.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.979+0000 7f46a9ffb640 1 -- 172.21.15.59:0/1318713104 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46b00039d0 con 0x7f46b406b6c0 2024-03-06T07:22:02.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.979+0000 7f58ba7fc640 1 -- 172.21.15.59:0/3751053450 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58bc0acc00 con 0x7f58c41627a0 2024-03-06T07:22:02.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.979+0000 7f0a0affd640 1 -- 172.21.15.59:0/3070088364 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a04048070 con 0x7f0a0c050190 2024-03-06T07:22:02.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.979+0000 7f58c94aa640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58a806b4d0 0x7f58a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.979+0000 7f0a0affd640 1 -- 172.21.15.59:0/3070088364 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a04032ae0 con 0x7f0a0c050190 2024-03-06T07:22:02.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.980+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0c04d2b0 con 0x7f0a0c050190 2024-03-06T07:22:02.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.980+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0c04d870 con 0x7f0a0c050190 2024-03-06T07:22:02.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.980+0000 7f0a0affd640 1 -- 172.21.15.59:0/3070088364 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a040414c0 con 0x7f0a0c050190 2024-03-06T07:22:02.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.980+0000 7f46a9ffb640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469806b4d0 0x7f469806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.980+0000 7f58c94aa640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58a806b4d0 0x7f58a806d990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f58c41ecc70 tx=0x7f58b4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.980+0000 7f46b89b0640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469806b4d0 0x7f469806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.980+0000 7f72f67fc640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b520 0x7f72d806d9e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f72f8144b70 tx=0x7f72e0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7fab26ffd640 1 -- 172.21.15.59:0/2061309397 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab38073070 con 0x7fab3004fc00 2024-03-06T07:22:02.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.980+0000 7f46a9ffb640 1 -- 172.21.15.59:0/1318713104 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46b008e060 con 0x7f46b406b6c0 2024-03-06T07:22:02.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7f46b89b0640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469806b4d0 0x7f469806d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f46b408b700 tx=0x7f46a4034000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7f058affd640 1 -- 172.21.15.59:0/2841917982 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0590004020 con 0x7f0584004800 2024-03-06T07:22:02.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7f058affd640 1 -- 172.21.15.59:0/2841917982 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05900041e0 con 0x7f0584004800 2024-03-06T07:22:02.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7f058affd640 1 -- 172.21.15.59:0/2841917982 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05900043d0 con 0x7f0584004800 2024-03-06T07:22:02.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc005290 con 0x7f0a0c050190 2024-03-06T07:22:02.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7f059b59e640 1 -- 172.21.15.59:0/2841917982 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059417e210 con 0x7f0584004800 2024-03-06T07:22:02.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7f059b59e640 1 -- 172.21.15.59:0/2841917982 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059403a5b0 con 0x7f0584004800 2024-03-06T07:22:02.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.980+0000 7f0c4a7fc640 1 -- 172.21.15.59:0/178810455 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c5001b070 con 0x7f0c5806afb0 2024-03-06T07:22:02.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7f0c4a7fc640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c3c06b700 0x7f0c3c06dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7f0c4a7fc640 1 -- 172.21.15.59:0/178810455 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c5008a690 con 0x7f0c5806afb0 2024-03-06T07:22:02.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7f0c5d6a8640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c3c06b700 0x7f0c3c06dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.981+0000 7f0c5d6a8640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c3c06b700 0x7f0c3c06dbc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c034000 tx=0x7f0c4c007000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.982+0000 7f0a0affd640 1 -- 172.21.15.59:0/3070088364 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a0404b070 con 0x7f0a0c050190 2024-03-06T07:22:02.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.982+0000 7f0a0affd640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09f806b500 0x7f09f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.982+0000 7f059b59e640 1 -- 172.21.15.59:0/2841917982 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05940057d0 con 0x7f0584004800 2024-03-06T07:22:02.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.983+0000 7f0a0affd640 1 -- 172.21.15.59:0/3070088364 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a040b02f0 con 0x7f0a0c050190 2024-03-06T07:22:02.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.983+0000 7f0a19c30640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09f806b500 0x7f09f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.983+0000 7f058affd640 1 -- 172.21.15.59:0/2841917982 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f059001a020 con 0x7f0584004800 2024-03-06T07:22:02.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.983+0000 7f0a19c30640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09f806b500 0x7f09f806d9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a14064ac0 tx=0x7f0a1406d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.983+0000 7f058affd640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057c06b520 0x7f057c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.983+0000 7f058affd640 1 -- 172.21.15.59:0/2841917982 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0590089ab0 con 0x7f0584004800 2024-03-06T07:22:02.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.983+0000 7f0599d9b640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057c06b520 0x7f057c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.984+0000 7f0599d9b640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057c06b520 0x7f057c06d9e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0584005770 tx=0x7f059c0ab000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.986+0000 7f46a9ffb640 1 -- 172.21.15.59:0/1318713104 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46b0089e20 con 0x7f46b406b6c0 2024-03-06T07:22:02.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.986+0000 7f0c4a7fc640 1 -- 172.21.15.59:0/178810455 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c50058eb0 con 0x7f0c5806afb0 2024-03-06T07:22:02.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.988+0000 7f058affd640 1 -- 172.21.15.59:0/2841917982 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0590058910 con 0x7f0584004800 2024-03-06T07:22:02.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.994+0000 7f0a0affd640 1 -- 172.21.15.59:0/3070088364 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a0c0d2130 con 0x7f0a0c050190 2024-03-06T07:22:02.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:01.995+0000 7f58ba7fc640 1 -- 172.21.15.59:0/3751053450 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58bc04d030 con 0x7f58c41627a0 2024-03-06T07:22:02.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.002+0000 7f72ef7fe640 1 -- 172.21.15.59:0/3754824410 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72e8059970 con 0x7f72f81478a0 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.349+0000 7fc57b8b5640 1 -- 172.21.15.59:0/375845945 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc554005580 con 0x7fc574163f90 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.349+0000 7fc569ffb640 1 -- 172.21.15.59:0/375845945 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc570058690 con 0x7fc574163f90 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.353+0000 7fc55f7fe640 1 -- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc55806b5a0 msgr2=0x7fc55806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.353+0000 7fc55f7fe640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc55806b5a0 0x7fc55806da60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc57414c750 tx=0x7fc564006bd0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.353+0000 7fc55f7fe640 1 -- 172.21.15.59:0/375845945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc574163f90 msgr2=0x7fc5741516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.353+0000 7fc55f7fe640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc574163f90 0x7fc5741516e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc57000b500 tx=0x7fc57000b9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.353+0000 7fc55f7fe640 1 -- 172.21.15.59:0/375845945 shutdown_connections 2024-03-06T07:22:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.353+0000 7fc55f7fe640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc55806b5a0 0x7fc55806da60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.353+0000 7fc55f7fe640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc574163f90 0x7fc5741516e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.353+0000 7fc55f7fe640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc57406fe70 0x7fc5741511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.354+0000 7fc55f7fe640 1 --2- 172.21.15.59:0/375845945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc57406de20 0x7fc574146ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.354+0000 7fc55f7fe640 1 -- 172.21.15.59:0/375845945 >> 172.21.15.59:0/375845945 conn(0x7fc5740d0a10 msgr2=0x7fc5740b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.354+0000 7fc55f7fe640 1 -- 172.21.15.59:0/375845945 shutdown_connections 2024-03-06T07:22:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.354+0000 7fc55f7fe640 1 -- 172.21.15.59:0/375845945 wait complete. 2024-03-06T07:22:02.372 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-03-06T07:22:02.372 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:02.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.382+0000 7fab24ff9640 1 -- 172.21.15.59:0/2061309397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fab300482f0 con 0x7fab3004fc00 2024-03-06T07:22:02.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.383+0000 7fab26ffd640 1 -- 172.21.15.59:0/2061309397 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fab380684f0 con 0x7fab3004fc00 2024-03-06T07:22:02.385 INFO:teuthology.orchestra.run.smithi059.stdout:55834574855 2024-03-06T07:22:02.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 -- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fab1806b520 msgr2=0x7fab1806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fab1806b520 0x7fab1806d9e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fab28000f60 tx=0x7fab28009040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 -- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab3004fc00 msgr2=0x7fab300d8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab3004fc00 0x7fab300d8470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fab38050c30 tx=0x7fab380683c0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 -- 172.21.15.59:0/2061309397 shutdown_connections 2024-03-06T07:22:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fab1806b520 0x7fab1806d9e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab300548a0 0x7fab3004ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab3004fc00 0x7fab300d8470 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 --2- 172.21.15.59:0/2061309397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fab300081c0 0x7fab300d7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 -- 172.21.15.59:0/2061309397 >> 172.21.15.59:0/2061309397 conn(0x7fab30055340 msgr2=0x7fab300e6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 -- 172.21.15.59:0/2061309397 shutdown_connections 2024-03-06T07:22:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.385+0000 7fab24ff9640 1 -- 172.21.15.59:0/2061309397 wait complete. 2024-03-06T07:22:02.398 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2024-03-06T07:22:02.399 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:02.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.405+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0c5806d090 con 0x7f0c5806afb0 2024-03-06T07:22:02.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.405+0000 7f0c4a7fc640 1 -- 172.21.15.59:0/178810455 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0c5005cf50 con 0x7f0c5806afb0 2024-03-06T07:22:02.408 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.408+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c3c06b700 msgr2=0x7f0c3c06dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.408+0000 7f0c6010d640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c3c06b700 0x7f0c3c06dbc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c034000 tx=0x7f0c4c007000 comp rx=0 tx=0).stop 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.408+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5806afb0 msgr2=0x7f0c58148be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.408+0000 7f0c6010d640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5806afb0 0x7f0c58148be0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000b850 tx=0x7f0c5000bd10 comp rx=0 tx=0).stop 2024-03-06T07:22:02.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.408+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 shutdown_connections 2024-03-06T07:22:02.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.408+0000 7f0c6010d640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c3c06b700 0x7f0c3c06dbc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.408+0000 7f0c6010d640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c58131520 0x7f0c58135170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.408+0000 7f0c6010d640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c5814a7e0 0x7f0c58160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.408+0000 7f0c6010d640 1 --2- 172.21.15.59:0/178810455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5806afb0 0x7f0c58148be0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.409+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 >> 172.21.15.59:0/178810455 conn(0x7f0c580b1830 msgr2=0x7f0c58165e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.409+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 shutdown_connections 2024-03-06T07:22:02.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.409+0000 7f0c6010d640 1 -- 172.21.15.59:0/178810455 wait complete. 2024-03-06T07:22:02.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.411+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f09fc005580 con 0x7f0a0c050190 2024-03-06T07:22:02.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.411+0000 7f0a0affd640 1 -- 172.21.15.59:0/3070088364 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0a0407ec30 con 0x7f0a0c050190 2024-03-06T07:22:02.414 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:02.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.412+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f58c40703e0 con 0x7f58c41627a0 2024-03-06T07:22:02.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.413+0000 7f58ba7fc640 1 -- 172.21.15.59:0/3751053450 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f58bc07b9e0 con 0x7f58c41627a0 2024-03-06T07:22:02.416 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:02.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.413+0000 7f059b59e640 1 -- 172.21.15.59:0/2841917982 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f059403cc40 con 0x7f0584004800 2024-03-06T07:22:02.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.413+0000 7f058affd640 1 -- 172.21.15.59:0/2841917982 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0590006c30 con 0x7f0584004800 2024-03-06T07:22:02.417 INFO:teuthology.orchestra.run.smithi059.stdout:55834574853 2024-03-06T07:22:02.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.414+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f46b40ce1a0 con 0x7f46b406b6c0 2024-03-06T07:22:02.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f46a9ffb640 1 -- 172.21.15.59:0/1318713104 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f46b0057e30 con 0x7f46b406b6c0 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09f806b500 msgr2=0x7f09f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09f806b500 0x7f09f806d9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0a14064ac0 tx=0x7f0a1406d040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c050190 msgr2=0x7f0a0c04c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c050190 0x7f0a0c04c590 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009fd0 tx=0x7f0a04004730 comp rx=0 tx=0).stop 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 shutdown_connections 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09f806b500 0x7f09f806d9c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a0c0df210 0x7f0a0c04ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.420 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2024-03-06T07:22:02.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0c053c60 0x7f0a0c0d5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 --2- 172.21.15.59:0/3070088364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a0c050190 0x7f0a0c04c590 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 >> 172.21.15.59:0/3070088364 conn(0x7f0a0c009fa0 msgr2=0x7f0a0c034190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 shutdown_connections 2024-03-06T07:22:02.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.415+0000 7f0a1b433640 1 -- 172.21.15.59:0/3070088364 wait complete. 2024-03-06T07:22:02.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.416+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58a806b4d0 msgr2=0x7f58a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.416+0000 7f58cbf0f640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58a806b4d0 0x7f58a806d990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f58c41ecc70 tx=0x7f58b4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.416+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58c41627a0 msgr2=0x7f58c414ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.416+0000 7f58cbf0f640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58c41627a0 0x7f58c414ec10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f58bc02c470 tx=0x7f58bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:02.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.416+0000 7f059b59e640 1 -- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057c06b520 msgr2=0x7f057c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 shutdown_connections 2024-03-06T07:22:02.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f58cbf0f640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58a806b4d0 0x7f58a806d990 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f58cbf0f640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58c40b4850 0x7f58c40b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f58cbf0f640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58c4163f90 0x7f58c40a9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f58cbf0f640 1 --2- 172.21.15.59:0/3751053450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58c41627a0 0x7f58c414ec10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.416+0000 7f059b59e640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057c06b520 0x7f057c06d9e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0584005770 tx=0x7f059c0ab000 comp rx=0 tx=0).stop 2024-03-06T07:22:02.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f059b59e640 1 -- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0584004800 msgr2=0x7f059404f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f059b59e640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0584004800 0x7f059404f470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f059400fe20 tx=0x7f059000d8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 >> 172.21.15.59:0/3751053450 conn(0x7f58c40d0a10 msgr2=0x7f58c4160060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f059b59e640 1 -- 172.21.15.59:0/2841917982 shutdown_connections 2024-03-06T07:22:02.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f059b59e640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057c06b520 0x7f057c06d9e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f059b59e640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f059417d560 0x7f0594054ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f059b59e640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f059403c2d0 0x7f059403a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f059b59e640 1 --2- 172.21.15.59:0/2841917982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0584004800 0x7f059404f470 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f059b59e640 1 -- 172.21.15.59:0/2841917982 >> 172.21.15.59:0/2841917982 conn(0x7f0594053740 msgr2=0x7f05940e1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 shutdown_connections 2024-03-06T07:22:02.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f58cbf0f640 1 -- 172.21.15.59:0/3751053450 wait complete. 2024-03-06T07:22:02.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f059b59e640 1 -- 172.21.15.59:0/2841917982 shutdown_connections 2024-03-06T07:22:02.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.417+0000 7f059b59e640 1 -- 172.21.15.59:0/2841917982 wait complete. 2024-03-06T07:22:02.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469806b4d0 msgr2=0x7f469806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469806b4d0 0x7f469806d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f46b408b700 tx=0x7f46a4034000 comp rx=0 tx=0).stop 2024-03-06T07:22:02.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b406b6c0 msgr2=0x7f46b415be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b406b6c0 0x7f46b415be90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f46b000b750 tx=0x7f46b000bc10 comp rx=0 tx=0).stop 2024-03-06T07:22:02.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 shutdown_connections 2024-03-06T07:22:02.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f469806b4d0 0x7f469806d990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46b4165190 0x7f46b41680d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46b40cece0 0x7f46b415e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 --2- 172.21.15.59:0/1318713104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46b406b6c0 0x7f46b415be90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 >> 172.21.15.59:0/1318713104 conn(0x7f46b40b1830 msgr2=0x7f46b40b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 shutdown_connections 2024-03-06T07:22:02.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.418+0000 7f46bb415640 1 -- 172.21.15.59:0/1318713104 wait complete. 2024-03-06T07:22:02.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.424+0000 7f72fd335640 1 -- 172.21.15.59:0/3754824410 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f72f80ce1a0 con 0x7f72f81478a0 2024-03-06T07:22:02.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.424+0000 7f72ef7fe640 1 -- 172.21.15.59:0/3754824410 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f72e8059270 con 0x7f72f81478a0 2024-03-06T07:22:02.428 INFO:teuthology.orchestra.run.smithi059.stdout:55834574855 2024-03-06T07:22:02.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 -- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b520 msgr2=0x7f72d806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b520 0x7f72d806d9e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f72f8144b70 tx=0x7f72e0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 -- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72f81478a0 msgr2=0x7f72f81fba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72f81478a0 0x7f72f81fba40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f72e800e480 tx=0x7f72e800e570 comp rx=0 tx=0).stop 2024-03-06T07:22:02.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 -- 172.21.15.59:0/3754824410 shutdown_connections 2024-03-06T07:22:02.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b520 0x7f72d806d9e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72f8147c80 0x7f72f8164f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8161740 0x7f72f81649f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 --2- 172.21.15.59:0/3754824410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72f81478a0 0x7f72f81fba40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 -- 172.21.15.59:0/3754824410 >> 172.21.15.59:0/3754824410 conn(0x7f72f80b1830 msgr2=0x7f72f8072580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 -- 172.21.15.59:0/3754824410 shutdown_connections 2024-03-06T07:22:02.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.428+0000 7f72fd335640 1 -- 172.21.15.59:0/3754824410 wait complete. 2024-03-06T07:22:02.432 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2024-03-06T07:22:02.432 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:02.434 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-03-06T07:22:02.434 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:02.437 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2024-03-06T07:22:02.438 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:02.438 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-03-06T07:22:02.438 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:02.439 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-03-06T07:22:02.439 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:02.444 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2024-03-06T07:22:02.444 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:02.444 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:22:02.445 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.583+0000 7f40cdceb640 1 Processor -- start 2024-03-06T07:22:02.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.583+0000 7f40cdceb640 1 -- start start 2024-03-06T07:22:02.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40cdceb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f40c80cf850 0x7f40c80cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40cdceb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f40c806cf20 0x7f40c806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40cdceb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c815e380 con 0x7f40c80d00c0 2024-03-06T07:22:02.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40cdceb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c806ca90 con 0x7f40c80cf850 2024-03-06T07:22:02.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40cdceb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c80cdc10 con 0x7f40c806cf20 2024-03-06T07:22:02.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40c6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f40c80cf850 0x7f40c80cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40c6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f40c80cf850 0x7f40c80cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58346/0 (socket says 172.21.15.59:58346) 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40c77fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f40c80d00c0 0x7f40c80d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51280/0 (socket says 172.21.15.59:51280) 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40c6ffd640 1 -- 172.21.15.59:0/1143948633 learned_addr learned my addr 172.21.15.59:0/1143948633 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40c7fff640 1 --2- 172.21.15.59:0/1143948633 >> v2:172.21.15.164:3300/0 conn(0x7f40c806cf20 0x7f40c806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40c6ffd640 1 -- 172.21.15.59:0/1143948633 >> v2:172.21.15.164:3300/0 conn(0x7f40c806cf20 msgr2=0x7f40c806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40c6ffd640 1 --2- 172.21.15.59:0/1143948633 >> v2:172.21.15.164:3300/0 conn(0x7f40c806cf20 0x7f40c806d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40c6ffd640 1 -- 172.21.15.59:0/1143948633 >> v1:172.21.15.59:6789/0 conn(0x7f40c80d00c0 legacy=0x7f40c80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40c67fc640 1 -- 172.21.15.59:0/1143948633 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124325552 0 0) 0x7f40c815e380 con 0x7f40c80d00c0 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.584+0000 7f40c6ffd640 1 -- 172.21.15.59:0/1143948633 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c806c160 con 0x7f40c80cf850 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.585+0000 7f40c6ffd640 1 --2- 172.21.15.59:0/1143948633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40c80cf850 0x7f40c80cd3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f40c8071f50 tx=0x7f40bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c91ebc4adfbf3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.585+0000 7f40c67fc640 1 -- 172.21.15.59:0/1143948633 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc034070 con 0x7f40c80cf850 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.585+0000 7f40c67fc640 1 -- 172.21.15.59:0/1143948633 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40bc030860 con 0x7f40c80cf850 2024-03-06T07:22:02.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.585+0000 7f40c67fc640 1 -- 172.21.15.59:0/1143948633 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40bc030c30 con 0x7f40c80cf850 2024-03-06T07:22:02.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.586+0000 7f40cdceb640 1 -- 172.21.15.59:0/1143948633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40c80cf850 msgr2=0x7f40c80cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.586+0000 7f40cdceb640 1 --2- 172.21.15.59:0/1143948633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40c80cf850 0x7f40c80cd3c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f40c8071f50 tx=0x7f40bc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.586+0000 7f40cdceb640 1 -- 172.21.15.59:0/1143948633 shutdown_connections 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.586+0000 7f40cdceb640 1 --2- 172.21.15.59:0/1143948633 >> v2:172.21.15.164:3300/0 conn(0x7f40c806cf20 0x7f40c806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.586+0000 7f40cdceb640 1 --2- 172.21.15.59:0/1143948633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40c80cf850 0x7f40c80cd3c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.586+0000 7f40cdceb640 1 -- 172.21.15.59:0/1143948633 >> 172.21.15.59:0/1143948633 conn(0x7f40c80b35a0 msgr2=0x7f40c80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.586+0000 7f40cdceb640 1 -- 172.21.15.59:0/1143948633 shutdown_connections 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.586+0000 7f40cdceb640 1 -- 172.21.15.59:0/1143948633 wait complete. 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.587+0000 7f40cdceb640 1 Processor -- start 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.587+0000 7f40cdceb640 1 -- start start 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40cdceb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40c806cf20 0x7f40c8154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40cdceb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40c80cf850 0x7f40c80b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40cdceb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40c80d00c0 0x7f40c80b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40cdceb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80cd900 con 0x7f40c80d00c0 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40cdceb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c806c720 con 0x7f40c806cf20 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40cdceb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80b1030 con 0x7f40c80cf850 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40c80d00c0 0x7f40c80b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40c80d00c0 0x7f40c80b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58356/0 (socket says 172.21.15.59:58356) 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c77fe640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40c806cf20 0x7f40c8154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c7fff640 1 -- 172.21.15.59:0/3558224708 learned_addr learned my addr 172.21.15.59:0/3558224708 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c6ffd640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40c80cf850 0x7f40c80b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c7fff640 1 -- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40c80cf850 msgr2=0x7f40c80b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c7fff640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40c80cf850 0x7f40c80b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c7fff640 1 -- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40c806cf20 msgr2=0x7f40c8154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c7fff640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40c806cf20 0x7f40c8154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c7fff640 1 -- 172.21.15.59:0/3558224708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c80b4e40 con 0x7f40c80d00c0 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c77fe640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40c806cf20 0x7f40c8154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:02.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.588+0000 7f40c7fff640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40c80d00c0 0x7f40c80b4810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f40c000efc0 tx=0x7f40c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.589+0000 7f40affff640 1 -- 172.21.15.59:0/3558224708 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c0018020 con 0x7f40c80d00c0 2024-03-06T07:22:02.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.589+0000 7f40affff640 1 -- 172.21.15.59:0/3558224708 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c0014480 con 0x7f40c80d00c0 2024-03-06T07:22:02.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.589+0000 7f40cdceb640 1 -- 172.21.15.59:0/3558224708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c80b5120 con 0x7f40c80d00c0 2024-03-06T07:22:02.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.589+0000 7f40affff640 1 -- 172.21.15.59:0/3558224708 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c0014870 con 0x7f40c80d00c0 2024-03-06T07:22:02.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.589+0000 7f40cdceb640 1 -- 172.21.15.59:0/3558224708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c80b56e0 con 0x7f40c80d00c0 2024-03-06T07:22:02.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.590+0000 7f40cdceb640 1 -- 172.21.15.59:0/3558224708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c804f330 con 0x7f40c80d00c0 2024-03-06T07:22:02.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.590+0000 7f40affff640 1 -- 172.21.15.59:0/3558224708 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40c0005d10 con 0x7f40c80d00c0 2024-03-06T07:22:02.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.590+0000 7f40affff640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40b406b4d0 0x7f40b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.590+0000 7f40affff640 1 -- 172.21.15.59:0/3558224708 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40c008abe0 con 0x7f40c80d00c0 2024-03-06T07:22:02.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.591+0000 7f40c77fe640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40b406b4d0 0x7f40b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.591+0000 7f40c77fe640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40b406b4d0 0x7f40b406d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f40b8007fd0 tx=0x7f40b8007de0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.595+0000 7f40affff640 1 -- 172.21.15.59:0/3558224708 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40c0022070 con 0x7f40c80d00c0 2024-03-06T07:22:02.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.772+0000 7f40cdceb640 1 -- 172.21.15.59:0/3558224708 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40c80cea80 con 0x7f40b406b4d0 2024-03-06T07:22:02.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.778+0000 7f40affff640 1 -- 172.21.15.59:0/3558224708 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251860 (secure 0 0 0) 0x7f40c80cea80 con 0x7f40b406b4d0 2024-03-06T07:22:02.783 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:02.785 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:22:02.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.781+0000 7f40cdceb640 1 -- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40b406b4d0 msgr2=0x7f40b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.781+0000 7f40cdceb640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40b406b4d0 0x7f40b406d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f40b8007fd0 tx=0x7f40b8007de0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.781+0000 7f40cdceb640 1 -- 172.21.15.59:0/3558224708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40c80d00c0 msgr2=0x7f40c80b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.781+0000 7f40cdceb640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40c80d00c0 0x7f40c80b4810 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f40c000efc0 tx=0x7f40c000c490 comp rx=0 tx=0).stop 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.781+0000 7f40cdceb640 1 -- 172.21.15.59:0/3558224708 shutdown_connections 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.781+0000 7f40cdceb640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40b406b4d0 0x7f40b406d990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.781+0000 7f40cdceb640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40c80d00c0 0x7f40c80b4810 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.781+0000 7f40cdceb640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40c80cf850 0x7f40c80b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.781+0000 7f40cdceb640 1 --2- 172.21.15.59:0/3558224708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40c806cf20 0x7f40c8154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.781+0000 7f40cdceb640 1 -- 172.21.15.59:0/3558224708 >> 172.21.15.59:0/3558224708 conn(0x7f40c80b35a0 msgr2=0x7f40c806d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.782+0000 7f40cdceb640 1 -- 172.21.15.59:0/3558224708 shutdown_connections 2024-03-06T07:22:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.782+0000 7f40cdceb640 1 -- 172.21.15.59:0/3558224708 wait complete. 2024-03-06T07:22:02.794 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-06T07:22:01.303672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219016,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755704,"statfs":{"total":773094113280,"available":772869840896,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.905649"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503744+0000","last_change":"2024-03-06T07:21:55.429371+0000","last_active":"2024-03-06T07:21:57.503744+0000","last_peered":"2024-03-06T07:21:57.503744+0000","last_clean":"2024-03-06T07:21:57.503744+0000","last_became_active":"2024-03-06T07:21:55.429057+0000","last_became_peered":"2024-03-06T07:21:55.429057+0000","last_unstale":"2024-03-06T07:21:57.503744+0000","last_undegraded":"2024-03-06T07:21:57.503744+0000","last_fullsized":"2024-03-06T07:21:57.503744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:44.793657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429512+0000","last_change":"2024-03-06T07:21:55.420380+0000","last_active":"2024-03-06T07:21:56.429512+0000","last_peered":"2024-03-06T07:21:56.429512+0000","last_clean":"2024-03-06T07:21:56.429512+0000","last_became_active":"2024-03-06T07:21:55.419640+0000","last_became_peered":"2024-03-06T07:21:55.419640+0000","last_unstale":"2024-03-06T07:21:56.429512+0000","last_undegraded":"2024-03-06T07:21:56.429512+0000","last_fullsized":"2024-03-06T07:21:56.429512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409011+0000","last_change":"2024-03-06T07:21:55.418047+0000","last_active":"2024-03-06T07:21:56.409011+0000","last_peered":"2024-03-06T07:21:56.409011+0000","last_clean":"2024-03-06T07:21:56.409011+0000","last_became_active":"2024-03-06T07:21:55.417536+0000","last_became_peered":"2024-03-06T07:21:55.417536+0000","last_unstale":"2024-03-06T07:21:56.409011+0000","last_undegraded":"2024-03-06T07:21:56.409011+0000","last_fullsized":"2024-03-06T07:21:56.409011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411527+0000","last_change":"2024-03-06T07:21:55.422738+0000","last_active":"2024-03-06T07:21:56.411527+0000","last_peered":"2024-03-06T07:21:56.411527+0000","last_clean":"2024-03-06T07:21:56.411527+0000","last_became_active":"2024-03-06T07:21:55.422460+0000","last_became_peered":"2024-03-06T07:21:55.422460+0000","last_unstale":"2024-03-06T07:21:56.411527+0000","last_undegraded":"2024-03-06T07:21:56.411527+0000","last_fullsized":"2024-03-06T07:21:56.411527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:44.132915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503278+0000","last_change":"2024-03-06T07:21:55.423628+0000","last_active":"2024-03-06T07:21:57.503278+0000","last_peered":"2024-03-06T07:21:57.503278+0000","last_clean":"2024-03-06T07:21:57.503278+0000","last_became_active":"2024-03-06T07:21:55.423360+0000","last_became_peered":"2024-03-06T07:21:55.423360+0000","last_unstale":"2024-03-06T07:21:57.503278+0000","last_undegraded":"2024-03-06T07:21:57.503278+0000","last_fullsized":"2024-03-06T07:21:57.503278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:16.890306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411324+0000","last_change":"2024-03-06T07:21:55.423187+0000","last_active":"2024-03-06T07:21:56.411324+0000","last_peered":"2024-03-06T07:21:56.411324+0000","last_clean":"2024-03-06T07:21:56.411324+0000","last_became_active":"2024-03-06T07:21:55.422907+0000","last_became_peered":"2024-03-06T07:21:55.422907+0000","last_unstale":"2024-03-06T07:21:56.411324+0000","last_undegraded":"2024-03-06T07:21:56.411324+0000","last_fullsized":"2024-03-06T07:21:56.411324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:47.365701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503159+0000","last_change":"2024-03-06T07:21:55.419720+0000","last_active":"2024-03-06T07:21:57.503159+0000","last_peered":"2024-03-06T07:21:57.503159+0000","last_clean":"2024-03-06T07:21:57.503159+0000","last_became_active":"2024-03-06T07:21:55.419294+0000","last_became_peered":"2024-03-06T07:21:55.419294+0000","last_unstale":"2024-03-06T07:21:57.503159+0000","last_undegraded":"2024-03-06T07:21:57.503159+0000","last_fullsized":"2024-03-06T07:21:57.503159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:47.365701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503213+0000","last_change":"2024-03-06T07:21:55.423177+0000","last_active":"2024-03-06T07:21:57.503213+0000","last_peered":"2024-03-06T07:21:57.503213+0000","last_clean":"2024-03-06T07:21:57.503213+0000","last_became_active":"2024-03-06T07:21:55.422875+0000","last_became_peered":"2024-03-06T07:21:55.422875+0000","last_unstale":"2024-03-06T07:21:57.503213+0000","last_undegraded":"2024-03-06T07:21:57.503213+0000","last_fullsized":"2024-03-06T07:21:57.503213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503645+0000","last_change":"2024-03-06T07:21:55.436966+0000","last_active":"2024-03-06T07:21:57.503645+0000","last_peered":"2024-03-06T07:21:57.503645+0000","last_clean":"2024-03-06T07:21:57.503645+0000","last_became_active":"2024-03-06T07:21:55.436375+0000","last_became_peered":"2024-03-06T07:21:55.436375+0000","last_unstale":"2024-03-06T07:21:57.503645+0000","last_undegraded":"2024-03-06T07:21:57.503645+0000","last_fullsized":"2024-03-06T07:21:57.503645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:26.847303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410503+0000","last_change":"2024-03-06T07:21:55.436554+0000","last_active":"2024-03-06T07:21:56.410503+0000","last_peered":"2024-03-06T07:21:56.410503+0000","last_clean":"2024-03-06T07:21:56.410503+0000","last_became_active":"2024-03-06T07:21:55.436292+0000","last_became_peered":"2024-03-06T07:21:55.436292+0000","last_unstale":"2024-03-06T07:21:56.410503+0000","last_undegraded":"2024-03-06T07:21:56.410503+0000","last_fullsized":"2024-03-06T07:21:56.410503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410321+0000","last_change":"2024-03-06T07:21:55.432397+0000","last_active":"2024-03-06T07:21:56.410321+0000","last_peered":"2024-03-06T07:21:56.410321+0000","last_clean":"2024-03-06T07:21:56.410321+0000","last_became_active":"2024-03-06T07:21:55.432090+0000","last_became_peered":"2024-03-06T07:21:55.432090+0000","last_unstale":"2024-03-06T07:21:56.410321+0000","last_undegraded":"2024-03-06T07:21:56.410321+0000","last_fullsized":"2024-03-06T07:21:56.410321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503158+0000","last_change":"2024-03-06T07:21:55.423196+0000","last_active":"2024-03-06T07:21:57.503158+0000","last_peered":"2024-03-06T07:21:57.503158+0000","last_clean":"2024-03-06T07:21:57.503158+0000","last_became_active":"2024-03-06T07:21:55.422079+0000","last_became_peered":"2024-03-06T07:21:55.422079+0000","last_unstale":"2024-03-06T07:21:57.503158+0000","last_undegraded":"2024-03-06T07:21:57.503158+0000","last_fullsized":"2024-03-06T07:21:57.503158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502844+0000","last_change":"2024-03-06T07:21:55.421461+0000","last_active":"2024-03-06T07:21:57.502844+0000","last_peered":"2024-03-06T07:21:57.502844+0000","last_clean":"2024-03-06T07:21:57.502844+0000","last_became_active":"2024-03-06T07:21:55.420795+0000","last_became_peered":"2024-03-06T07:21:55.420795+0000","last_unstale":"2024-03-06T07:21:57.502844+0000","last_undegraded":"2024-03-06T07:21:57.502844+0000","last_fullsized":"2024-03-06T07:21:57.502844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.408829+0000","last_change":"2024-03-06T07:21:55.420082+0000","last_active":"2024-03-06T07:21:56.408829+0000","last_peered":"2024-03-06T07:21:56.408829+0000","last_clean":"2024-03-06T07:21:56.408829+0000","last_became_active":"2024-03-06T07:21:55.419844+0000","last_became_peered":"2024-03-06T07:21:55.419844+0000","last_unstale":"2024-03-06T07:21:56.408829+0000","last_undegraded":"2024-03-06T07:21:56.408829+0000","last_fullsized":"2024-03-06T07:21:56.408829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429302+0000","last_change":"2024-03-06T07:21:55.425776+0000","last_active":"2024-03-06T07:21:56.429302+0000","last_peered":"2024-03-06T07:21:56.429302+0000","last_clean":"2024-03-06T07:21:56.429302+0000","last_became_active":"2024-03-06T07:21:55.425479+0000","last_became_peered":"2024-03-06T07:21:55.425479+0000","last_unstale":"2024-03-06T07:21:56.429302+0000","last_undegraded":"2024-03-06T07:21:56.429302+0000","last_fullsized":"2024-03-06T07:21:56.429302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410249+0000","last_change":"2024-03-06T07:21:55.416970+0000","last_active":"2024-03-06T07:21:56.410249+0000","last_peered":"2024-03-06T07:21:56.410249+0000","last_clean":"2024-03-06T07:21:56.410249+0000","last_became_active":"2024-03-06T07:21:55.416495+0000","last_became_peered":"2024-03-06T07:21:55.416495+0000","last_unstale":"2024-03-06T07:21:56.410249+0000","last_undegraded":"2024-03-06T07:21:56.410249+0000","last_fullsized":"2024-03-06T07:21:56.410249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411126+0000","last_change":"2024-03-06T07:21:55.430721+0000","last_active":"2024-03-06T07:21:56.411126+0000","last_peered":"2024-03-06T07:21:56.411126+0000","last_clean":"2024-03-06T07:21:56.411126+0000","last_became_active":"2024-03-06T07:21:55.428901+0000","last_became_peered":"2024-03-06T07:21:55.428901+0000","last_unstale":"2024-03-06T07:21:56.411126+0000","last_undegraded":"2024-03-06T07:21:56.411126+0000","last_fullsized":"2024-03-06T07:21:56.411126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:49.817183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502966+0000","last_change":"2024-03-06T07:21:55.428378+0000","last_active":"2024-03-06T07:21:57.502966+0000","last_peered":"2024-03-06T07:21:57.502966+0000","last_clean":"2024-03-06T07:21:57.502966+0000","last_became_active":"2024-03-06T07:21:55.428101+0000","last_became_peered":"2024-03-06T07:21:55.428101+0000","last_unstale":"2024-03-06T07:21:57.502966+0000","last_undegraded":"2024-03-06T07:21:57.502966+0000","last_fullsized":"2024-03-06T07:21:57.502966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503619+0000","last_change":"2024-03-06T07:21:55.425219+0000","last_active":"2024-03-06T07:21:57.503619+0000","last_peered":"2024-03-06T07:21:57.503619+0000","last_clean":"2024-03-06T07:21:57.503619+0000","last_became_active":"2024-03-06T07:21:55.422831+0000","last_became_peered":"2024-03-06T07:21:55.422831+0000","last_unstale":"2024-03-06T07:21:57.503619+0000","last_undegraded":"2024-03-06T07:21:57.503619+0000","last_fullsized":"2024-03-06T07:21:57.503619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:32.596022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410649+0000","last_change":"2024-03-06T07:21:55.427953+0000","last_active":"2024-03-06T07:21:56.410649+0000","last_peered":"2024-03-06T07:21:56.410649+0000","last_clean":"2024-03-06T07:21:56.410649+0000","last_became_active":"2024-03-06T07:21:55.427652+0000","last_became_peered":"2024-03-06T07:21:55.427652+0000","last_unstale":"2024-03-06T07:21:56.410649+0000","last_undegraded":"2024-03-06T07:21:56.410649+0000","last_fullsized":"2024-03-06T07:21:56.410649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409090+0000","last_change":"2024-03-06T07:21:55.421525+0000","last_active":"2024-03-06T07:21:56.409090+0000","last_peered":"2024-03-06T07:21:56.409090+0000","last_clean":"2024-03-06T07:21:56.409090+0000","last_became_active":"2024-03-06T07:21:55.421204+0000","last_became_peered":"2024-03-06T07:21:55.421204+0000","last_unstale":"2024-03-06T07:21:56.409090+0000","last_undegraded":"2024-03-06T07:21:56.409090+0000","last_fullsized":"2024-03-06T07:21:56.409090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411437+0000","last_change":"2024-03-06T07:21:55.424534+0000","last_active":"2024-03-06T07:21:56.411437+0000","last_peered":"2024-03-06T07:21:56.411437+0000","last_clean":"2024-03-06T07:21:56.411437+0000","last_became_active":"2024-03-06T07:21:55.424309+0000","last_became_peered":"2024-03-06T07:21:55.424309+0000","last_unstale":"2024-03-06T07:21:56.411437+0000","last_undegraded":"2024-03-06T07:21:56.411437+0000","last_fullsized":"2024-03-06T07:21:56.411437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:16.890306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503336+0000","last_change":"2024-03-06T07:21:55.424084+0000","last_active":"2024-03-06T07:21:57.503336+0000","last_peered":"2024-03-06T07:21:57.503336+0000","last_clean":"2024-03-06T07:21:57.503336+0000","last_became_active":"2024-03-06T07:21:55.423245+0000","last_became_peered":"2024-03-06T07:21:55.423245+0000","last_unstale":"2024-03-06T07:21:57.503336+0000","last_undegraded":"2024-03-06T07:21:57.503336+0000","last_fullsized":"2024-03-06T07:21:57.503336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:41.154073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410397+0000","last_change":"2024-03-06T07:21:55.415889+0000","last_active":"2024-03-06T07:21:56.410397+0000","last_peered":"2024-03-06T07:21:56.410397+0000","last_clean":"2024-03-06T07:21:56.410397+0000","last_became_active":"2024-03-06T07:21:55.415620+0000","last_became_peered":"2024-03-06T07:21:55.415620+0000","last_unstale":"2024-03-06T07:21:56.410397+0000","last_undegraded":"2024-03-06T07:21:56.410397+0000","last_fullsized":"2024-03-06T07:21:56.410397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503106+0000","last_change":"2024-03-06T07:21:55.422611+0000","last_active":"2024-03-06T07:21:57.503106+0000","last_peered":"2024-03-06T07:21:57.503106+0000","last_clean":"2024-03-06T07:21:57.503106+0000","last_became_active":"2024-03-06T07:21:55.422260+0000","last_became_peered":"2024-03-06T07:21:55.422260+0000","last_unstale":"2024-03-06T07:21:57.503106+0000","last_undegraded":"2024-03-06T07:21:57.503106+0000","last_fullsized":"2024-03-06T07:21:57.503106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411077+0000","last_change":"2024-03-06T07:21:55.420808+0000","last_active":"2024-03-06T07:21:56.411077+0000","last_peered":"2024-03-06T07:21:56.411077+0000","last_clean":"2024-03-06T07:21:56.411077+0000","last_became_active":"2024-03-06T07:21:55.420529+0000","last_became_peered":"2024-03-06T07:21:55.420529+0000","last_unstale":"2024-03-06T07:21:56.411077+0000","last_undegraded":"2024-03-06T07:21:56.411077+0000","last_fullsized":"2024-03-06T07:21:56.411077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429492+0000","last_change":"2024-03-06T07:21:55.420655+0000","last_active":"2024-03-06T07:21:56.429492+0000","last_peered":"2024-03-06T07:21:56.429492+0000","last_clean":"2024-03-06T07:21:56.429492+0000","last_became_active":"2024-03-06T07:21:55.420396+0000","last_became_peered":"2024-03-06T07:21:55.420396+0000","last_unstale":"2024-03-06T07:21:56.429492+0000","last_undegraded":"2024-03-06T07:21:56.429492+0000","last_fullsized":"2024-03-06T07:21:56.429492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503486+0000","last_change":"2024-03-06T07:21:55.431350+0000","last_active":"2024-03-06T07:21:57.503486+0000","last_peered":"2024-03-06T07:21:57.503486+0000","last_clean":"2024-03-06T07:21:57.503486+0000","last_became_active":"2024-03-06T07:21:55.431033+0000","last_became_peered":"2024-03-06T07:21:55.431033+0000","last_unstale":"2024-03-06T07:21:57.503486+0000","last_undegraded":"2024-03-06T07:21:57.503486+0000","last_fullsized":"2024-03-06T07:21:57.503486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:47.365701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409140+0000","last_change":"2024-03-06T07:21:55.416463+0000","last_active":"2024-03-06T07:21:56.409140+0000","last_peered":"2024-03-06T07:21:56.409140+0000","last_clean":"2024-03-06T07:21:56.409140+0000","last_became_active":"2024-03-06T07:21:55.416174+0000","last_became_peered":"2024-03-06T07:21:55.416174+0000","last_unstale":"2024-03-06T07:21:56.409140+0000","last_undegraded":"2024-03-06T07:21:56.409140+0000","last_fullsized":"2024-03-06T07:21:56.409140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411495+0000","last_change":"2024-03-06T07:21:55.418595+0000","last_active":"2024-03-06T07:21:56.411495+0000","last_peered":"2024-03-06T07:21:56.411495+0000","last_clean":"2024-03-06T07:21:56.411495+0000","last_became_active":"2024-03-06T07:21:55.418073+0000","last_became_peered":"2024-03-06T07:21:55.418073+0000","last_unstale":"2024-03-06T07:21:56.411495+0000","last_undegraded":"2024-03-06T07:21:56.411495+0000","last_fullsized":"2024-03-06T07:21:56.411495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:10.706857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502657+0000","last_change":"2024-03-06T07:21:55.429932+0000","last_active":"2024-03-06T07:21:57.502657+0000","last_peered":"2024-03-06T07:21:57.502657+0000","last_clean":"2024-03-06T07:21:57.502657+0000","last_became_active":"2024-03-06T07:21:55.429593+0000","last_became_peered":"2024-03-06T07:21:55.429593+0000","last_unstale":"2024-03-06T07:21:57.502657+0000","last_undegraded":"2024-03-06T07:21:57.502657+0000","last_fullsized":"2024-03-06T07:21:57.502657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410150+0000","last_change":"2024-03-06T07:21:55.436974+0000","last_active":"2024-03-06T07:21:56.410150+0000","last_peered":"2024-03-06T07:21:56.410150+0000","last_clean":"2024-03-06T07:21:56.410150+0000","last_became_active":"2024-03-06T07:21:55.436638+0000","last_became_peered":"2024-03-06T07:21:55.436638+0000","last_unstale":"2024-03-06T07:21:56.410150+0000","last_undegraded":"2024-03-06T07:21:56.410150+0000","last_fullsized":"2024-03-06T07:21:56.410150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502927+0000","last_change":"2024-03-06T07:21:55.435717+0000","last_active":"2024-03-06T07:21:57.502927+0000","last_peered":"2024-03-06T07:21:57.502927+0000","last_clean":"2024-03-06T07:21:57.502927+0000","last_became_active":"2024-03-06T07:21:55.435429+0000","last_became_peered":"2024-03-06T07:21:55.435429+0000","last_unstale":"2024-03-06T07:21:57.502927+0000","last_undegraded":"2024-03-06T07:21:57.502927+0000","last_fullsized":"2024-03-06T07:21:57.502927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:41.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503089+0000","last_change":"2024-03-06T07:21:54.413886+0000","last_active":"2024-03-06T07:21:57.503089+0000","last_peered":"2024-03-06T07:21:57.503089+0000","last_clean":"2024-03-06T07:21:57.503089+0000","last_became_active":"2024-03-06T07:21:54.413573+0000","last_became_peered":"2024-03-06T07:21:54.413573+0000","last_unstale":"2024-03-06T07:21:57.503089+0000","last_undegraded":"2024-03-06T07:21:57.503089+0000","last_fullsized":"2024-03-06T07:21:57.503089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:17.458904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409253+0000","last_change":"2024-03-06T07:21:55.427074+0000","last_active":"2024-03-06T07:21:56.409253+0000","last_peered":"2024-03-06T07:21:56.409253+0000","last_clean":"2024-03-06T07:21:56.409253+0000","last_became_active":"2024-03-06T07:21:55.426775+0000","last_became_peered":"2024-03-06T07:21:55.426775+0000","last_unstale":"2024-03-06T07:21:56.409253+0000","last_undegraded":"2024-03-06T07:21:56.409253+0000","last_fullsized":"2024-03-06T07:21:56.409253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:41.154073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410426+0000","last_change":"2024-03-06T07:21:54.418543+0000","last_active":"2024-03-06T07:21:56.410426+0000","last_peered":"2024-03-06T07:21:56.410426+0000","last_clean":"2024-03-06T07:21:56.410426+0000","last_became_active":"2024-03-06T07:21:54.418148+0000","last_became_peered":"2024-03-06T07:21:54.418148+0000","last_unstale":"2024-03-06T07:21:56.410426+0000","last_undegraded":"2024-03-06T07:21:56.410426+0000","last_fullsized":"2024-03-06T07:21:56.410426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503384+0000","last_change":"2024-03-06T07:21:54.417756+0000","last_active":"2024-03-06T07:21:57.503384+0000","last_peered":"2024-03-06T07:21:57.503384+0000","last_clean":"2024-03-06T07:21:57.503384+0000","last_became_active":"2024-03-06T07:21:54.417471+0000","last_became_peered":"2024-03-06T07:21:54.417471+0000","last_unstale":"2024-03-06T07:21:57.503384+0000","last_undegraded":"2024-03-06T07:21:57.503384+0000","last_fullsized":"2024-03-06T07:21:57.503384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503246+0000","last_change":"2024-03-06T07:21:55.418688+0000","last_active":"2024-03-06T07:21:57.503246+0000","last_peered":"2024-03-06T07:21:57.503246+0000","last_clean":"2024-03-06T07:21:57.503246+0000","last_became_active":"2024-03-06T07:21:55.418392+0000","last_became_peered":"2024-03-06T07:21:55.418392+0000","last_unstale":"2024-03-06T07:21:57.503246+0000","last_undegraded":"2024-03-06T07:21:57.503246+0000","last_fullsized":"2024-03-06T07:21:57.503246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410615+0000","last_change":"2024-03-06T07:21:54.414567+0000","last_active":"2024-03-06T07:21:56.410615+0000","last_peered":"2024-03-06T07:21:56.410615+0000","last_clean":"2024-03-06T07:21:56.410615+0000","last_became_active":"2024-03-06T07:21:54.414292+0000","last_became_peered":"2024-03-06T07:21:54.414292+0000","last_unstale":"2024-03-06T07:21:56.410615+0000","last_undegraded":"2024-03-06T07:21:56.410615+0000","last_fullsized":"2024-03-06T07:21:56.410615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429274+0000","last_change":"2024-03-06T07:21:55.420297+0000","last_active":"2024-03-06T07:21:56.429274+0000","last_peered":"2024-03-06T07:21:56.429274+0000","last_clean":"2024-03-06T07:21:56.429274+0000","last_became_active":"2024-03-06T07:21:55.419994+0000","last_became_peered":"2024-03-06T07:21:55.419994+0000","last_unstale":"2024-03-06T07:21:56.429274+0000","last_undegraded":"2024-03-06T07:21:56.429274+0000","last_fullsized":"2024-03-06T07:21:56.429274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410774+0000","last_change":"2024-03-06T07:21:54.420034+0000","last_active":"2024-03-06T07:21:56.410774+0000","last_peered":"2024-03-06T07:21:56.410774+0000","last_clean":"2024-03-06T07:21:56.410774+0000","last_became_active":"2024-03-06T07:21:54.419745+0000","last_became_peered":"2024-03-06T07:21:54.419745+0000","last_unstale":"2024-03-06T07:21:56.410774+0000","last_undegraded":"2024-03-06T07:21:56.410774+0000","last_fullsized":"2024-03-06T07:21:56.410774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410843+0000","last_change":"2024-03-06T07:21:55.425571+0000","last_active":"2024-03-06T07:21:56.410843+0000","last_peered":"2024-03-06T07:21:56.410843+0000","last_clean":"2024-03-06T07:21:56.410843+0000","last_became_active":"2024-03-06T07:21:55.424876+0000","last_became_peered":"2024-03-06T07:21:55.424876+0000","last_unstale":"2024-03-06T07:21:56.410843+0000","last_undegraded":"2024-03-06T07:21:56.410843+0000","last_fullsized":"2024-03-06T07:21:56.410843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:41.154073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410852+0000","last_change":"2024-03-06T07:21:54.416884+0000","last_active":"2024-03-06T07:21:56.410852+0000","last_peered":"2024-03-06T07:21:56.410852+0000","last_clean":"2024-03-06T07:21:56.410852+0000","last_became_active":"2024-03-06T07:21:54.416576+0000","last_became_peered":"2024-03-06T07:21:54.416576+0000","last_unstale":"2024-03-06T07:21:56.410852+0000","last_undegraded":"2024-03-06T07:21:56.410852+0000","last_fullsized":"2024-03-06T07:21:56.410852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409127+0000","last_change":"2024-03-06T07:21:55.414257+0000","last_active":"2024-03-06T07:21:56.409127+0000","last_peered":"2024-03-06T07:21:56.409127+0000","last_clean":"2024-03-06T07:21:56.409127+0000","last_became_active":"2024-03-06T07:21:55.413950+0000","last_became_peered":"2024-03-06T07:21:55.413950+0000","last_unstale":"2024-03-06T07:21:56.409127+0000","last_undegraded":"2024-03-06T07:21:56.409127+0000","last_fullsized":"2024-03-06T07:21:56.409127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429387+0000","last_change":"2024-03-06T07:21:54.412699+0000","last_active":"2024-03-06T07:21:56.429387+0000","last_peered":"2024-03-06T07:21:56.429387+0000","last_clean":"2024-03-06T07:21:56.429387+0000","last_became_active":"2024-03-06T07:21:54.412392+0000","last_became_peered":"2024-03-06T07:21:54.412392+0000","last_unstale":"2024-03-06T07:21:56.429387+0000","last_undegraded":"2024-03-06T07:21:56.429387+0000","last_fullsized":"2024-03-06T07:21:56.429387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503297+0000","last_change":"2024-03-06T07:21:55.421734+0000","last_active":"2024-03-06T07:21:57.503297+0000","last_peered":"2024-03-06T07:21:57.503297+0000","last_clean":"2024-03-06T07:21:57.503297+0000","last_became_active":"2024-03-06T07:21:55.421451+0000","last_became_peered":"2024-03-06T07:21:55.421451+0000","last_unstale":"2024-03-06T07:21:57.503297+0000","last_undegraded":"2024-03-06T07:21:57.503297+0000","last_fullsized":"2024-03-06T07:21:57.503297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410763+0000","last_change":"2024-03-06T07:21:54.416848+0000","last_active":"2024-03-06T07:21:56.410763+0000","last_peered":"2024-03-06T07:21:56.410763+0000","last_clean":"2024-03-06T07:21:56.410763+0000","last_became_active":"2024-03-06T07:21:54.416561+0000","last_became_peered":"2024-03-06T07:21:54.416561+0000","last_unstale":"2024-03-06T07:21:56.410763+0000","last_undegraded":"2024-03-06T07:21:56.410763+0000","last_fullsized":"2024-03-06T07:21:56.410763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503047+0000","last_change":"2024-03-06T07:21:55.425748+0000","last_active":"2024-03-06T07:21:57.503047+0000","last_peered":"2024-03-06T07:21:57.503047+0000","last_clean":"2024-03-06T07:21:57.503047+0000","last_became_active":"2024-03-06T07:21:55.425501+0000","last_became_peered":"2024-03-06T07:21:55.425501+0000","last_unstale":"2024-03-06T07:21:57.503047+0000","last_undegraded":"2024-03-06T07:21:57.503047+0000","last_fullsized":"2024-03-06T07:21:57.503047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:08.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409204+0000","last_change":"2024-03-06T07:21:54.419364+0000","last_active":"2024-03-06T07:21:56.409204+0000","last_peered":"2024-03-06T07:21:56.409204+0000","last_clean":"2024-03-06T07:21:56.409204+0000","last_became_active":"2024-03-06T07:21:54.419094+0000","last_became_peered":"2024-03-06T07:21:54.419094+0000","last_unstale":"2024-03-06T07:21:56.409204+0000","last_undegraded":"2024-03-06T07:21:56.409204+0000","last_fullsized":"2024-03-06T07:21:56.409204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.408850+0000","last_change":"2024-03-06T07:21:55.421461+0000","last_active":"2024-03-06T07:21:56.408850+0000","last_peered":"2024-03-06T07:21:56.408850+0000","last_clean":"2024-03-06T07:21:56.408850+0000","last_became_active":"2024-03-06T07:21:55.421153+0000","last_became_peered":"2024-03-06T07:21:55.421153+0000","last_unstale":"2024-03-06T07:21:56.408850+0000","last_undegraded":"2024-03-06T07:21:56.408850+0000","last_fullsized":"2024-03-06T07:21:56.408850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411554+0000","last_change":"2024-03-06T07:21:54.416121+0000","last_active":"2024-03-06T07:21:56.411554+0000","last_peered":"2024-03-06T07:21:56.411554+0000","last_clean":"2024-03-06T07:21:56.411554+0000","last_became_active":"2024-03-06T07:21:54.415857+0000","last_became_peered":"2024-03-06T07:21:54.415857+0000","last_unstale":"2024-03-06T07:21:56.411554+0000","last_undegraded":"2024-03-06T07:21:56.411554+0000","last_fullsized":"2024-03-06T07:21:56.411554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503422+0000","last_change":"2024-03-06T07:21:55.428224+0000","last_active":"2024-03-06T07:21:57.503422+0000","last_peered":"2024-03-06T07:21:57.503422+0000","last_clean":"2024-03-06T07:21:57.503422+0000","last_became_active":"2024-03-06T07:21:55.427932+0000","last_became_peered":"2024-03-06T07:21:55.427932+0000","last_unstale":"2024-03-06T07:21:57.503422+0000","last_undegraded":"2024-03-06T07:21:57.503422+0000","last_fullsized":"2024-03-06T07:21:57.503422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:10.706857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410172+0000","last_change":"2024-03-06T07:21:54.418770+0000","last_active":"2024-03-06T07:21:56.410172+0000","last_peered":"2024-03-06T07:21:56.410172+0000","last_clean":"2024-03-06T07:21:56.410172+0000","last_became_active":"2024-03-06T07:21:54.418498+0000","last_became_peered":"2024-03-06T07:21:54.418498+0000","last_unstale":"2024-03-06T07:21:56.410172+0000","last_undegraded":"2024-03-06T07:21:56.410172+0000","last_fullsized":"2024-03-06T07:21:56.410172+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410331+0000","last_change":"2024-03-06T07:21:55.436175+0000","last_active":"2024-03-06T07:21:56.410331+0000","last_peered":"2024-03-06T07:21:56.410331+0000","last_clean":"2024-03-06T07:21:56.410331+0000","last_became_active":"2024-03-06T07:21:55.435873+0000","last_became_peered":"2024-03-06T07:21:55.435873+0000","last_unstale":"2024-03-06T07:21:56.410331+0000","last_undegraded":"2024-03-06T07:21:56.410331+0000","last_fullsized":"2024-03-06T07:21:56.410331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502508+0000","last_change":"2024-03-06T07:21:54.410321+0000","last_active":"2024-03-06T07:21:57.502508+0000","last_peered":"2024-03-06T07:21:57.502508+0000","last_clean":"2024-03-06T07:21:57.502508+0000","last_became_active":"2024-03-06T07:21:54.410019+0000","last_became_peered":"2024-03-06T07:21:54.410019+0000","last_unstale":"2024-03-06T07:21:57.502508+0000","last_undegraded":"2024-03-06T07:21:57.502508+0000","last_fullsized":"2024-03-06T07:21:57.502508+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503336+0000","last_change":"2024-03-06T07:21:55.427171+0000","last_active":"2024-03-06T07:21:57.503336+0000","last_peered":"2024-03-06T07:21:57.503336+0000","last_clean":"2024-03-06T07:21:57.503336+0000","last_became_active":"2024-03-06T07:21:55.426872+0000","last_became_peered":"2024-03-06T07:21:55.426872+0000","last_unstale":"2024-03-06T07:21:57.503336+0000","last_undegraded":"2024-03-06T07:21:57.503336+0000","last_fullsized":"2024-03-06T07:21:57.503336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:44.132915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503361+0000","last_change":"2024-03-06T07:21:54.419357+0000","last_active":"2024-03-06T07:21:57.503361+0000","last_peered":"2024-03-06T07:21:57.503361+0000","last_clean":"2024-03-06T07:21:57.503361+0000","last_became_active":"2024-03-06T07:21:54.419053+0000","last_became_peered":"2024-03-06T07:21:54.419053+0000","last_unstale":"2024-03-06T07:21:57.503361+0000","last_undegraded":"2024-03-06T07:21:57.503361+0000","last_fullsized":"2024-03-06T07:21:57.503361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:42.184203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503583+0000","last_change":"2024-03-06T07:21:55.427511+0000","last_active":"2024-03-06T07:21:57.503583+0000","last_peered":"2024-03-06T07:21:57.503583+0000","last_clean":"2024-03-06T07:21:57.503583+0000","last_became_active":"2024-03-06T07:21:55.427175+0000","last_became_peered":"2024-03-06T07:21:55.427175+0000","last_unstale":"2024-03-06T07:21:57.503583+0000","last_undegraded":"2024-03-06T07:21:57.503583+0000","last_fullsized":"2024-03-06T07:21:57.503583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:05.685917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502735+0000","last_change":"2024-03-06T07:21:54.416325+0000","last_active":"2024-03-06T07:21:57.502735+0000","last_peered":"2024-03-06T07:21:57.502735+0000","last_clean":"2024-03-06T07:21:57.502735+0000","last_became_active":"2024-03-06T07:21:54.416017+0000","last_became_peered":"2024-03-06T07:21:54.416017+0000","last_unstale":"2024-03-06T07:21:57.502735+0000","last_undegraded":"2024-03-06T07:21:57.502735+0000","last_fullsized":"2024-03-06T07:21:57.502735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503730+0000","last_change":"2024-03-06T07:21:55.428378+0000","last_active":"2024-03-06T07:21:57.503730+0000","last_peered":"2024-03-06T07:21:57.503730+0000","last_clean":"2024-03-06T07:21:57.503730+0000","last_became_active":"2024-03-06T07:21:55.426715+0000","last_became_peered":"2024-03-06T07:21:55.426715+0000","last_unstale":"2024-03-06T07:21:57.503730+0000","last_undegraded":"2024-03-06T07:21:57.503730+0000","last_fullsized":"2024-03-06T07:21:57.503730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:29.756209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429581+0000","last_change":"2024-03-06T07:21:54.406786+0000","last_active":"2024-03-06T07:21:56.429581+0000","last_peered":"2024-03-06T07:21:56.429581+0000","last_clean":"2024-03-06T07:21:56.429581+0000","last_became_active":"2024-03-06T07:21:54.406454+0000","last_became_peered":"2024-03-06T07:21:54.406454+0000","last_unstale":"2024-03-06T07:21:56.429581+0000","last_undegraded":"2024-03-06T07:21:56.429581+0000","last_fullsized":"2024-03-06T07:21:56.429581+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:50.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411653+0000","last_change":"2024-03-06T07:21:55.422470+0000","last_active":"2024-03-06T07:21:56.411653+0000","last_peered":"2024-03-06T07:21:56.411653+0000","last_clean":"2024-03-06T07:21:56.411653+0000","last_became_active":"2024-03-06T07:21:55.420656+0000","last_became_peered":"2024-03-06T07:21:55.420656+0000","last_unstale":"2024-03-06T07:21:56.411653+0000","last_undegraded":"2024-03-06T07:21:56.411653+0000","last_fullsized":"2024-03-06T07:21:56.411653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:32.596022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502899+0000","last_change":"2024-03-06T07:21:54.412695+0000","last_active":"2024-03-06T07:21:57.502899+0000","last_peered":"2024-03-06T07:21:57.502899+0000","last_clean":"2024-03-06T07:21:57.502899+0000","last_became_active":"2024-03-06T07:21:54.412391+0000","last_became_peered":"2024-03-06T07:21:54.412391+0000","last_unstale":"2024-03-06T07:21:57.502899+0000","last_undegraded":"2024-03-06T07:21:57.502899+0000","last_fullsized":"2024-03-06T07:21:57.502899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:17.458904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503199+0000","last_change":"2024-03-06T07:21:55.436949+0000","last_active":"2024-03-06T07:21:57.503199+0000","last_peered":"2024-03-06T07:21:57.503199+0000","last_clean":"2024-03-06T07:21:57.503199+0000","last_became_active":"2024-03-06T07:21:55.436502+0000","last_became_peered":"2024-03-06T07:21:55.436502+0000","last_unstale":"2024-03-06T07:21:57.503199+0000","last_undegraded":"2024-03-06T07:21:57.503199+0000","last_fullsized":"2024-03-06T07:21:57.503199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:31.378193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503196+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:21:57.503196+0000","last_peered":"2024-03-06T07:21:57.503196+0000","last_clean":"2024-03-06T07:21:57.503196+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:21:57.503196+0000","last_undegraded":"2024-03-06T07:21:57.503196+0000","last_fullsized":"2024-03-06T07:21:57.503196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410901+0000","last_change":"2024-03-06T07:21:54.411371+0000","last_active":"2024-03-06T07:21:56.410901+0000","last_peered":"2024-03-06T07:21:56.410901+0000","last_clean":"2024-03-06T07:21:56.410901+0000","last_became_active":"2024-03-06T07:21:54.411095+0000","last_became_peered":"2024-03-06T07:21:54.411095+0000","last_unstale":"2024-03-06T07:21:56.410901+0000","last_undegraded":"2024-03-06T07:21:56.410901+0000","last_fullsized":"2024-03-06T07:21:56.410901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503359+0000","last_change":"2024-03-06T07:21:55.423928+0000","last_active":"2024-03-06T07:21:57.503359+0000","last_peered":"2024-03-06T07:21:57.503359+0000","last_clean":"2024-03-06T07:21:57.503359+0000","last_became_active":"2024-03-06T07:21:55.423681+0000","last_became_peered":"2024-03-06T07:21:55.423681+0000","last_unstale":"2024-03-06T07:21:57.503359+0000","last_undegraded":"2024-03-06T07:21:57.503359+0000","last_fullsized":"2024-03-06T07:21:57.503359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:49.817183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503508+0000","last_change":"2024-03-06T07:21:54.415381+0000","last_active":"2024-03-06T07:21:57.503508+0000","last_peered":"2024-03-06T07:21:57.503508+0000","last_clean":"2024-03-06T07:21:57.503508+0000","last_became_active":"2024-03-06T07:21:54.415099+0000","last_became_peered":"2024-03-06T07:21:54.415099+0000","last_unstale":"2024-03-06T07:21:57.503508+0000","last_undegraded":"2024-03-06T07:21:57.503508+0000","last_fullsized":"2024-03-06T07:21:57.503508+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.891311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.408864+0000","last_change":"2024-03-06T07:21:55.423013+0000","last_active":"2024-03-06T07:21:56.408864+0000","last_peered":"2024-03-06T07:21:56.408864+0000","last_clean":"2024-03-06T07:21:56.408864+0000","last_became_active":"2024-03-06T07:21:55.422717+0000","last_became_peered":"2024-03-06T07:21:55.422717+0000","last_unstale":"2024-03-06T07:21:56.408864+0000","last_undegraded":"2024-03-06T07:21:56.408864+0000","last_fullsized":"2024-03-06T07:21:56.408864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502628+0000","last_change":"2024-03-06T07:21:54.416728+0000","last_active":"2024-03-06T07:21:57.502628+0000","last_peered":"2024-03-06T07:21:57.502628+0000","last_clean":"2024-03-06T07:21:57.502628+0000","last_became_active":"2024-03-06T07:21:54.416226+0000","last_became_peered":"2024-03-06T07:21:54.416226+0000","last_unstale":"2024-03-06T07:21:57.502628+0000","last_undegraded":"2024-03-06T07:21:57.502628+0000","last_fullsized":"2024-03-06T07:21:57.502628+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429308+0000","last_change":"2024-03-06T07:21:55.429746+0000","last_active":"2024-03-06T07:21:56.429308+0000","last_peered":"2024-03-06T07:21:56.429308+0000","last_clean":"2024-03-06T07:21:56.429308+0000","last_became_active":"2024-03-06T07:21:55.429320+0000","last_became_peered":"2024-03-06T07:21:55.429320+0000","last_unstale":"2024-03-06T07:21:56.429308+0000","last_undegraded":"2024-03-06T07:21:56.429308+0000","last_fullsized":"2024-03-06T07:21:56.429308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411588+0000","last_change":"2024-03-06T07:21:54.413058+0000","last_active":"2024-03-06T07:21:56.411588+0000","last_peered":"2024-03-06T07:21:56.411588+0000","last_clean":"2024-03-06T07:21:56.411588+0000","last_became_active":"2024-03-06T07:21:54.412469+0000","last_became_peered":"2024-03-06T07:21:54.412469+0000","last_unstale":"2024-03-06T07:21:56.411588+0000","last_undegraded":"2024-03-06T07:21:56.411588+0000","last_fullsized":"2024-03-06T07:21:56.411588+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503091+0000","last_change":"2024-03-06T07:21:55.425788+0000","last_active":"2024-03-06T07:21:57.503091+0000","last_peered":"2024-03-06T07:21:57.503091+0000","last_clean":"2024-03-06T07:21:57.503091+0000","last_became_active":"2024-03-06T07:21:55.425440+0000","last_became_peered":"2024-03-06T07:21:55.425440+0000","last_unstale":"2024-03-06T07:21:57.503091+0000","last_undegraded":"2024-03-06T07:21:57.503091+0000","last_fullsized":"2024-03-06T07:21:57.503091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:36.844180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429371+0000","last_change":"2024-03-06T07:21:54.415739+0000","last_active":"2024-03-06T07:21:56.429371+0000","last_peered":"2024-03-06T07:21:56.429371+0000","last_clean":"2024-03-06T07:21:56.429371+0000","last_became_active":"2024-03-06T07:21:54.415452+0000","last_became_peered":"2024-03-06T07:21:54.415452+0000","last_unstale":"2024-03-06T07:21:56.429371+0000","last_undegraded":"2024-03-06T07:21:56.429371+0000","last_fullsized":"2024-03-06T07:21:56.429371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411402+0000","last_change":"2024-03-06T07:21:55.419140+0000","last_active":"2024-03-06T07:21:56.411402+0000","last_peered":"2024-03-06T07:21:56.411402+0000","last_clean":"2024-03-06T07:21:56.411402+0000","last_became_active":"2024-03-06T07:21:55.415992+0000","last_became_peered":"2024-03-06T07:21:55.415992+0000","last_unstale":"2024-03-06T07:21:56.411402+0000","last_undegraded":"2024-03-06T07:21:56.411402+0000","last_fullsized":"2024-03-06T07:21:56.411402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:31.378193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503514+0000","last_change":"2024-03-06T07:21:54.417366+0000","last_active":"2024-03-06T07:21:57.503514+0000","last_peered":"2024-03-06T07:21:57.503514+0000","last_clean":"2024-03-06T07:21:57.503514+0000","last_became_active":"2024-03-06T07:21:54.416684+0000","last_became_peered":"2024-03-06T07:21:54.416684+0000","last_unstale":"2024-03-06T07:21:57.503514+0000","last_undegraded":"2024-03-06T07:21:57.503514+0000","last_fullsized":"2024-03-06T07:21:57.503514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:44.263686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429397+0000","last_change":"2024-03-06T07:21:55.419798+0000","last_active":"2024-03-06T07:21:56.429397+0000","last_peered":"2024-03-06T07:21:56.429397+0000","last_clean":"2024-03-06T07:21:56.429397+0000","last_became_active":"2024-03-06T07:21:55.419490+0000","last_became_peered":"2024-03-06T07:21:55.419490+0000","last_unstale":"2024-03-06T07:21:56.429397+0000","last_undegraded":"2024-03-06T07:21:56.429397+0000","last_fullsized":"2024-03-06T07:21:56.429397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429552+0000","last_change":"2024-03-06T07:21:54.418937+0000","last_active":"2024-03-06T07:21:56.429552+0000","last_peered":"2024-03-06T07:21:56.429552+0000","last_clean":"2024-03-06T07:21:56.429552+0000","last_became_active":"2024-03-06T07:21:54.418592+0000","last_became_peered":"2024-03-06T07:21:54.418592+0000","last_unstale":"2024-03-06T07:21:56.429552+0000","last_undegraded":"2024-03-06T07:21:56.429552+0000","last_fullsized":"2024-03-06T07:21:56.429552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:17.773963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503269+0000","last_change":"2024-03-06T07:21:55.420527+0000","last_active":"2024-03-06T07:21:57.503269+0000","last_peered":"2024-03-06T07:21:57.503269+0000","last_clean":"2024-03-06T07:21:57.503269+0000","last_became_active":"2024-03-06T07:21:55.420210+0000","last_became_peered":"2024-03-06T07:21:55.420210+0000","last_unstale":"2024-03-06T07:21:57.503269+0000","last_undegraded":"2024-03-06T07:21:57.503269+0000","last_fullsized":"2024-03-06T07:21:57.503269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410686+0000","last_change":"2024-03-06T07:21:54.420765+0000","last_active":"2024-03-06T07:21:56.410686+0000","last_peered":"2024-03-06T07:21:56.410686+0000","last_clean":"2024-03-06T07:21:56.410686+0000","last_became_active":"2024-03-06T07:21:54.420541+0000","last_became_peered":"2024-03-06T07:21:54.420541+0000","last_unstale":"2024-03-06T07:21:56.410686+0000","last_undegraded":"2024-03-06T07:21:56.410686+0000","last_fullsized":"2024-03-06T07:21:56.410686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409085+0000","last_change":"2024-03-06T07:21:55.430019+0000","last_active":"2024-03-06T07:21:56.409085+0000","last_peered":"2024-03-06T07:21:56.409085+0000","last_clean":"2024-03-06T07:21:56.409085+0000","last_became_active":"2024-03-06T07:21:55.429671+0000","last_became_peered":"2024-03-06T07:21:55.429671+0000","last_unstale":"2024-03-06T07:21:56.409085+0000","last_undegraded":"2024-03-06T07:21:56.409085+0000","last_fullsized":"2024-03-06T07:21:56.409085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502709+0000","last_change":"2024-03-06T07:21:54.416295+0000","last_active":"2024-03-06T07:21:57.502709+0000","last_peered":"2024-03-06T07:21:57.502709+0000","last_clean":"2024-03-06T07:21:57.502709+0000","last_became_active":"2024-03-06T07:21:54.415817+0000","last_became_peered":"2024-03-06T07:21:54.415817+0000","last_unstale":"2024-03-06T07:21:57.502709+0000","last_undegraded":"2024-03-06T07:21:57.502709+0000","last_fullsized":"2024-03-06T07:21:57.502709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:28.435323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503166+0000","last_change":"2024-03-06T07:21:55.422908+0000","last_active":"2024-03-06T07:21:57.503166+0000","last_peered":"2024-03-06T07:21:57.503166+0000","last_clean":"2024-03-06T07:21:57.503166+0000","last_became_active":"2024-03-06T07:21:55.422624+0000","last_became_peered":"2024-03-06T07:21:55.422624+0000","last_unstale":"2024-03-06T07:21:57.503166+0000","last_undegraded":"2024-03-06T07:21:57.503166+0000","last_fullsized":"2024-03-06T07:21:57.503166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429396+0000","last_change":"2024-03-06T07:21:54.411585+0000","last_active":"2024-03-06T07:21:56.429396+0000","last_peered":"2024-03-06T07:21:56.429396+0000","last_clean":"2024-03-06T07:21:56.429396+0000","last_became_active":"2024-03-06T07:21:54.411318+0000","last_became_peered":"2024-03-06T07:21:54.411318+0000","last_unstale":"2024-03-06T07:21:56.429396+0000","last_undegraded":"2024-03-06T07:21:56.429396+0000","last_fullsized":"2024-03-06T07:21:56.429396+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503397+0000","last_change":"2024-03-06T07:21:55.421416+0000","last_active":"2024-03-06T07:21:57.503397+0000","last_peered":"2024-03-06T07:21:57.503397+0000","last_clean":"2024-03-06T07:21:57.503397+0000","last_became_active":"2024-03-06T07:21:55.421113+0000","last_became_peered":"2024-03-06T07:21:55.421113+0000","last_unstale":"2024-03-06T07:21:57.503397+0000","last_undegraded":"2024-03-06T07:21:57.503397+0000","last_fullsized":"2024-03-06T07:21:57.503397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:08.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429482+0000","last_change":"2024-03-06T07:21:54.416417+0000","last_active":"2024-03-06T07:21:56.429482+0000","last_peered":"2024-03-06T07:21:56.429482+0000","last_clean":"2024-03-06T07:21:56.429482+0000","last_became_active":"2024-03-06T07:21:54.416108+0000","last_became_peered":"2024-03-06T07:21:54.416108+0000","last_unstale":"2024-03-06T07:21:56.429482+0000","last_undegraded":"2024-03-06T07:21:56.429482+0000","last_fullsized":"2024-03-06T07:21:56.429482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502995+0000","last_change":"2024-03-06T07:21:55.418459+0000","last_active":"2024-03-06T07:21:57.502995+0000","last_peered":"2024-03-06T07:21:57.502995+0000","last_clean":"2024-03-06T07:21:57.502995+0000","last_became_active":"2024-03-06T07:21:55.418179+0000","last_became_peered":"2024-03-06T07:21:55.418179+0000","last_unstale":"2024-03-06T07:21:57.502995+0000","last_undegraded":"2024-03-06T07:21:57.502995+0000","last_fullsized":"2024-03-06T07:21:57.502995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503599+0000","last_change":"2024-03-06T07:21:54.412875+0000","last_active":"2024-03-06T07:21:57.503599+0000","last_peered":"2024-03-06T07:21:57.503599+0000","last_clean":"2024-03-06T07:21:57.503599+0000","last_became_active":"2024-03-06T07:21:54.412617+0000","last_became_peered":"2024-03-06T07:21:54.412617+0000","last_unstale":"2024-03-06T07:21:57.503599+0000","last_undegraded":"2024-03-06T07:21:57.503599+0000","last_fullsized":"2024-03-06T07:21:57.503599+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:50.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410564+0000","last_change":"2024-03-06T07:21:55.427872+0000","last_active":"2024-03-06T07:21:56.410564+0000","last_peered":"2024-03-06T07:21:56.410564+0000","last_clean":"2024-03-06T07:21:56.410564+0000","last_became_active":"2024-03-06T07:21:55.427504+0000","last_became_peered":"2024-03-06T07:21:55.427504+0000","last_unstale":"2024-03-06T07:21:56.410564+0000","last_undegraded":"2024-03-06T07:21:56.410564+0000","last_fullsized":"2024-03-06T07:21:56.410564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411242+0000","last_change":"2024-03-06T07:21:55.427757+0000","last_active":"2024-03-06T07:21:56.411242+0000","last_peered":"2024-03-06T07:21:56.411242+0000","last_clean":"2024-03-06T07:21:56.411242+0000","last_became_active":"2024-03-06T07:21:55.427471+0000","last_became_peered":"2024-03-06T07:21:55.427471+0000","last_unstale":"2024-03-06T07:21:56.411242+0000","last_undegraded":"2024-03-06T07:21:56.411242+0000","last_fullsized":"2024-03-06T07:21:56.411242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:08.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503412+0000","last_change":"2024-03-06T07:21:54.414035+0000","last_active":"2024-03-06T07:21:57.503412+0000","last_peered":"2024-03-06T07:21:57.503412+0000","last_clean":"2024-03-06T07:21:57.503412+0000","last_became_active":"2024-03-06T07:21:54.413618+0000","last_became_peered":"2024-03-06T07:21:54.413618+0000","last_unstale":"2024-03-06T07:21:57.503412+0000","last_undegraded":"2024-03-06T07:21:57.503412+0000","last_fullsized":"2024-03-06T07:21:57.503412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411590+0000","last_change":"2024-03-06T07:21:55.423540+0000","last_active":"2024-03-06T07:21:56.411590+0000","last_peered":"2024-03-06T07:21:56.411590+0000","last_clean":"2024-03-06T07:21:56.411590+0000","last_became_active":"2024-03-06T07:21:55.423197+0000","last_became_peered":"2024-03-06T07:21:55.423197+0000","last_unstale":"2024-03-06T07:21:56.411590+0000","last_undegraded":"2024-03-06T07:21:56.411590+0000","last_fullsized":"2024-03-06T07:21:56.411590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:05.685917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503403+0000","last_change":"2024-03-06T07:21:54.417365+0000","last_active":"2024-03-06T07:21:57.503403+0000","last_peered":"2024-03-06T07:21:57.503403+0000","last_clean":"2024-03-06T07:21:57.503403+0000","last_became_active":"2024-03-06T07:21:54.417072+0000","last_became_peered":"2024-03-06T07:21:54.417072+0000","last_unstale":"2024-03-06T07:21:57.503403+0000","last_undegraded":"2024-03-06T07:21:57.503403+0000","last_fullsized":"2024-03-06T07:21:57.503403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:17.773963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409094+0000","last_change":"2024-03-06T07:21:55.424176+0000","last_active":"2024-03-06T07:21:56.409094+0000","last_peered":"2024-03-06T07:21:56.409094+0000","last_clean":"2024-03-06T07:21:56.409094+0000","last_became_active":"2024-03-06T07:21:55.423848+0000","last_became_peered":"2024-03-06T07:21:55.423848+0000","last_unstale":"2024-03-06T07:21:56.409094+0000","last_undegraded":"2024-03-06T07:21:56.409094+0000","last_fullsized":"2024-03-06T07:21:56.409094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503028+0000","last_change":"2024-03-06T07:21:54.418869+0000","last_active":"2024-03-06T07:21:57.503028+0000","last_peered":"2024-03-06T07:21:57.503028+0000","last_clean":"2024-03-06T07:21:57.503028+0000","last_became_active":"2024-03-06T07:21:54.418582+0000","last_became_peered":"2024-03-06T07:21:54.418582+0000","last_unstale":"2024-03-06T07:21:57.503028+0000","last_undegraded":"2024-03-06T07:21:57.503028+0000","last_fullsized":"2024-03-06T07:21:57.503028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502919+0000","last_change":"2024-03-06T07:21:55.427658+0000","last_active":"2024-03-06T07:21:57.502919+0000","last_peered":"2024-03-06T07:21:57.502919+0000","last_clean":"2024-03-06T07:21:57.502919+0000","last_became_active":"2024-03-06T07:21:55.427370+0000","last_became_peered":"2024-03-06T07:21:55.427370+0000","last_unstale":"2024-03-06T07:21:57.502919+0000","last_undegraded":"2024-03-06T07:21:57.502919+0000","last_fullsized":"2024-03-06T07:21:57.502919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409199+0000","last_change":"2024-03-06T07:21:54.417435+0000","last_active":"2024-03-06T07:21:56.409199+0000","last_peered":"2024-03-06T07:21:56.409199+0000","last_clean":"2024-03-06T07:21:56.409199+0000","last_became_active":"2024-03-06T07:21:54.417136+0000","last_became_peered":"2024-03-06T07:21:54.417136+0000","last_unstale":"2024-03-06T07:21:56.409199+0000","last_undegraded":"2024-03-06T07:21:56.409199+0000","last_fullsized":"2024-03-06T07:21:56.409199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:15.277777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:02.799 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:02.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.934+0000 7f4999834640 1 Processor -- start 2024-03-06T07:22:02.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.934+0000 7f4999834640 1 -- start start 2024-03-06T07:22:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f4999834640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49940d00c0 0x7f49940d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f4999834640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f49940cf850 0x7f49940cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f4999834640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49940d09e0 con 0x7f49940ce1b0 2024-03-06T07:22:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f4999834640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f499406ca90 con 0x7f49940cf850 2024-03-06T07:22:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f4999834640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499406df30 con 0x7f49940d00c0 2024-03-06T07:22:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f498bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f49940cf850 0x7f49940cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f49937fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f49940ce1b0 0x7f49940ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51296/0 (socket says 172.21.15.59:51296) 2024-03-06T07:22:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f49937fe640 1 -- 172.21.15.59:0/990935143 learned_addr learned my addr 172.21.15.59:0/990935143 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f4992ffd640 1 --2- 172.21.15.59:0/990935143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49940d00c0 0x7f49940d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f498bfff640 1 -- 172.21.15.59:0/990935143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49940d00c0 msgr2=0x7f49940d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f498bfff640 1 --2- 172.21.15.59:0/990935143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49940d00c0 0x7f49940d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f498bfff640 1 -- 172.21.15.59:0/990935143 >> v1:172.21.15.59:6789/0 conn(0x7f49940ce1b0 legacy=0x7f49940ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f49927fc640 1 -- 172.21.15.59:0/990935143 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840350242 0 0) 0x7f49940d09e0 con 0x7f49940ce1b0 2024-03-06T07:22:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.935+0000 7f498bfff640 1 -- 172.21.15.59:0/990935143 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499415e380 con 0x7f49940cf850 2024-03-06T07:22:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.936+0000 7f498bfff640 1 --2- 172.21.15.59:0/990935143 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940cf850 0x7f49940cd3c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f497c0097a0 tx=0x7f497c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6bf689ae5deea29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.936+0000 7f49927fc640 1 -- 172.21.15.59:0/990935143 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c035020 con 0x7f49940cf850 2024-03-06T07:22:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.936+0000 7f49927fc640 1 -- 172.21.15.59:0/990935143 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497c031830 con 0x7f49940cf850 2024-03-06T07:22:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.936+0000 7f49927fc640 1 -- 172.21.15.59:0/990935143 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c031c00 con 0x7f49940cf850 2024-03-06T07:22:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.937+0000 7f4999834640 1 -- 172.21.15.59:0/990935143 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940cf850 msgr2=0x7f49940cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.937+0000 7f4999834640 1 --2- 172.21.15.59:0/990935143 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940cf850 0x7f49940cd3c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f497c0097a0 tx=0x7f497c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.937+0000 7f4999834640 1 -- 172.21.15.59:0/990935143 shutdown_connections 2024-03-06T07:22:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.937+0000 7f4999834640 1 --2- 172.21.15.59:0/990935143 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940cf850 0x7f49940cd3c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.937+0000 7f4999834640 1 --2- 172.21.15.59:0/990935143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49940d00c0 0x7f49940d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.937+0000 7f4999834640 1 -- 172.21.15.59:0/990935143 >> 172.21.15.59:0/990935143 conn(0x7f49940b35a0 msgr2=0x7f49940b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.937+0000 7f4999834640 1 -- 172.21.15.59:0/990935143 shutdown_connections 2024-03-06T07:22:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.937+0000 7f4999834640 1 -- 172.21.15.59:0/990935143 wait complete. 2024-03-06T07:22:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.938+0000 7f4999834640 1 Processor -- start 2024-03-06T07:22:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.938+0000 7f4999834640 1 -- start start 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.938+0000 7f4999834640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49940ce1b0 0x7f4994156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.938+0000 7f4999834640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49940cf850 0x7f49940c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.938+0000 7f4999834640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940d00c0 0x7f49940cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f4999834640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499406c440 con 0x7f49940d00c0 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f4999834640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499406c160 con 0x7f49940cf850 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f4999834640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499406c720 con 0x7f49940ce1b0 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f49937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940d00c0 0x7f49940cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f49937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940d00c0 0x7f49940cb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58370/0 (socket says 172.21.15.59:58370) 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f49937fe640 1 -- 172.21.15.59:0/3102776947 learned_addr learned my addr 172.21.15.59:0/3102776947 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f4992ffd640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49940ce1b0 0x7f4994156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f498bfff640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49940cf850 0x7f49940c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f49937fe640 1 -- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49940ce1b0 msgr2=0x7f4994156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f49937fe640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49940ce1b0 0x7f4994156fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f49937fe640 1 -- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49940cf850 msgr2=0x7f49940c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f49937fe640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49940cf850 0x7f49940c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f49937fe640 1 -- 172.21.15.59:0/3102776947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49940c84b0 con 0x7f49940d00c0 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f4992ffd640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49940ce1b0 0x7f4994156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f49937fe640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940d00c0 0x7f49940cb460 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f498c00c7a0 tx=0x7f498c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f4990ff9640 1 -- 172.21.15.59:0/3102776947 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c018070 con 0x7f49940d00c0 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f4990ff9640 1 -- 172.21.15.59:0/3102776947 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c00a640 con 0x7f49940d00c0 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f4990ff9640 1 -- 172.21.15.59:0/3102776947 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c00aab0 con 0x7f49940d00c0 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f4999834640 1 -- 172.21.15.59:0/3102776947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49940c6ed0 con 0x7f49940d00c0 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.939+0000 7f4999834640 1 -- 172.21.15.59:0/3102776947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49940c7770 con 0x7f49940d00c0 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.940+0000 7f4999834640 1 -- 172.21.15.59:0/3102776947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499404f330 con 0x7f49940d00c0 2024-03-06T07:22:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.941+0000 7f4990ff9640 1 -- 172.21.15.59:0/3102776947 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f498c01b070 con 0x7f49940d00c0 2024-03-06T07:22:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.941+0000 7f4990ff9640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f497406b500 0x7f497406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.941+0000 7f4990ff9640 1 -- 172.21.15.59:0/3102776947 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f498c089700 con 0x7f49940d00c0 2024-03-06T07:22:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.941+0000 7f4992ffd640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f497406b500 0x7f497406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.942+0000 7f4992ffd640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f497406b500 0x7f497406d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4994068aa0 tx=0x7f4984009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:02.945+0000 7f4990ff9640 1 -- 172.21.15.59:0/3102776947 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f498c057f30 con 0x7f49940d00c0 2024-03-06T07:22:03.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.123+0000 7f4999834640 1 -- 172.21.15.59:0/3102776947 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49940b44e0 con 0x7f497406b500 2024-03-06T07:22:03.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.135+0000 7f4990ff9640 1 -- 172.21.15.59:0/3102776947 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251860 (secure 0 0 0) 0x7f49940b44e0 con 0x7f497406b500 2024-03-06T07:22:03.141 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:03.143 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:22:03.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 -- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f497406b500 msgr2=0x7f497406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f497406b500 0x7f497406d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4994068aa0 tx=0x7f4984009190 comp rx=0 tx=0).stop 2024-03-06T07:22:03.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 -- 172.21.15.59:0/3102776947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940d00c0 msgr2=0x7f49940cb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940d00c0 0x7f49940cb460 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f498c00c7a0 tx=0x7f498c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 -- 172.21.15.59:0/3102776947 shutdown_connections 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f497406b500 0x7f497406d9c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49940d00c0 0x7f49940cb460 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49940cf850 0x7f49940c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 --2- 172.21.15.59:0/3102776947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49940ce1b0 0x7f4994156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 -- 172.21.15.59:0/3102776947 >> 172.21.15.59:0/3102776947 conn(0x7f49940b35a0 msgr2=0x7f499406fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 -- 172.21.15.59:0/3102776947 shutdown_connections 2024-03-06T07:22:03.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.138+0000 7f4999834640 1 -- 172.21.15.59:0/3102776947 wait complete. 2024-03-06T07:22:03.151 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-06T07:22:01.303672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219016,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755704,"statfs":{"total":773094113280,"available":772869840896,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.905649"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503744+0000","last_change":"2024-03-06T07:21:55.429371+0000","last_active":"2024-03-06T07:21:57.503744+0000","last_peered":"2024-03-06T07:21:57.503744+0000","last_clean":"2024-03-06T07:21:57.503744+0000","last_became_active":"2024-03-06T07:21:55.429057+0000","last_became_peered":"2024-03-06T07:21:55.429057+0000","last_unstale":"2024-03-06T07:21:57.503744+0000","last_undegraded":"2024-03-06T07:21:57.503744+0000","last_fullsized":"2024-03-06T07:21:57.503744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:44.793657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429512+0000","last_change":"2024-03-06T07:21:55.420380+0000","last_active":"2024-03-06T07:21:56.429512+0000","last_peered":"2024-03-06T07:21:56.429512+0000","last_clean":"2024-03-06T07:21:56.429512+0000","last_became_active":"2024-03-06T07:21:55.419640+0000","last_became_peered":"2024-03-06T07:21:55.419640+0000","last_unstale":"2024-03-06T07:21:56.429512+0000","last_undegraded":"2024-03-06T07:21:56.429512+0000","last_fullsized":"2024-03-06T07:21:56.429512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409011+0000","last_change":"2024-03-06T07:21:55.418047+0000","last_active":"2024-03-06T07:21:56.409011+0000","last_peered":"2024-03-06T07:21:56.409011+0000","last_clean":"2024-03-06T07:21:56.409011+0000","last_became_active":"2024-03-06T07:21:55.417536+0000","last_became_peered":"2024-03-06T07:21:55.417536+0000","last_unstale":"2024-03-06T07:21:56.409011+0000","last_undegraded":"2024-03-06T07:21:56.409011+0000","last_fullsized":"2024-03-06T07:21:56.409011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411527+0000","last_change":"2024-03-06T07:21:55.422738+0000","last_active":"2024-03-06T07:21:56.411527+0000","last_peered":"2024-03-06T07:21:56.411527+0000","last_clean":"2024-03-06T07:21:56.411527+0000","last_became_active":"2024-03-06T07:21:55.422460+0000","last_became_peered":"2024-03-06T07:21:55.422460+0000","last_unstale":"2024-03-06T07:21:56.411527+0000","last_undegraded":"2024-03-06T07:21:56.411527+0000","last_fullsized":"2024-03-06T07:21:56.411527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:44.132915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503278+0000","last_change":"2024-03-06T07:21:55.423628+0000","last_active":"2024-03-06T07:21:57.503278+0000","last_peered":"2024-03-06T07:21:57.503278+0000","last_clean":"2024-03-06T07:21:57.503278+0000","last_became_active":"2024-03-06T07:21:55.423360+0000","last_became_peered":"2024-03-06T07:21:55.423360+0000","last_unstale":"2024-03-06T07:21:57.503278+0000","last_undegraded":"2024-03-06T07:21:57.503278+0000","last_fullsized":"2024-03-06T07:21:57.503278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:16.890306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411324+0000","last_change":"2024-03-06T07:21:55.423187+0000","last_active":"2024-03-06T07:21:56.411324+0000","last_peered":"2024-03-06T07:21:56.411324+0000","last_clean":"2024-03-06T07:21:56.411324+0000","last_became_active":"2024-03-06T07:21:55.422907+0000","last_became_peered":"2024-03-06T07:21:55.422907+0000","last_unstale":"2024-03-06T07:21:56.411324+0000","last_undegraded":"2024-03-06T07:21:56.411324+0000","last_fullsized":"2024-03-06T07:21:56.411324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:47.365701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503159+0000","last_change":"2024-03-06T07:21:55.419720+0000","last_active":"2024-03-06T07:21:57.503159+0000","last_peered":"2024-03-06T07:21:57.503159+0000","last_clean":"2024-03-06T07:21:57.503159+0000","last_became_active":"2024-03-06T07:21:55.419294+0000","last_became_peered":"2024-03-06T07:21:55.419294+0000","last_unstale":"2024-03-06T07:21:57.503159+0000","last_undegraded":"2024-03-06T07:21:57.503159+0000","last_fullsized":"2024-03-06T07:21:57.503159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:47.365701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503213+0000","last_change":"2024-03-06T07:21:55.423177+0000","last_active":"2024-03-06T07:21:57.503213+0000","last_peered":"2024-03-06T07:21:57.503213+0000","last_clean":"2024-03-06T07:21:57.503213+0000","last_became_active":"2024-03-06T07:21:55.422875+0000","last_became_peered":"2024-03-06T07:21:55.422875+0000","last_unstale":"2024-03-06T07:21:57.503213+0000","last_undegraded":"2024-03-06T07:21:57.503213+0000","last_fullsized":"2024-03-06T07:21:57.503213+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503645+0000","last_change":"2024-03-06T07:21:55.436966+0000","last_active":"2024-03-06T07:21:57.503645+0000","last_peered":"2024-03-06T07:21:57.503645+0000","last_clean":"2024-03-06T07:21:57.503645+0000","last_became_active":"2024-03-06T07:21:55.436375+0000","last_became_peered":"2024-03-06T07:21:55.436375+0000","last_unstale":"2024-03-06T07:21:57.503645+0000","last_undegraded":"2024-03-06T07:21:57.503645+0000","last_fullsized":"2024-03-06T07:21:57.503645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:26.847303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410503+0000","last_change":"2024-03-06T07:21:55.436554+0000","last_active":"2024-03-06T07:21:56.410503+0000","last_peered":"2024-03-06T07:21:56.410503+0000","last_clean":"2024-03-06T07:21:56.410503+0000","last_became_active":"2024-03-06T07:21:55.436292+0000","last_became_peered":"2024-03-06T07:21:55.436292+0000","last_unstale":"2024-03-06T07:21:56.410503+0000","last_undegraded":"2024-03-06T07:21:56.410503+0000","last_fullsized":"2024-03-06T07:21:56.410503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410321+0000","last_change":"2024-03-06T07:21:55.432397+0000","last_active":"2024-03-06T07:21:56.410321+0000","last_peered":"2024-03-06T07:21:56.410321+0000","last_clean":"2024-03-06T07:21:56.410321+0000","last_became_active":"2024-03-06T07:21:55.432090+0000","last_became_peered":"2024-03-06T07:21:55.432090+0000","last_unstale":"2024-03-06T07:21:56.410321+0000","last_undegraded":"2024-03-06T07:21:56.410321+0000","last_fullsized":"2024-03-06T07:21:56.410321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503158+0000","last_change":"2024-03-06T07:21:55.423196+0000","last_active":"2024-03-06T07:21:57.503158+0000","last_peered":"2024-03-06T07:21:57.503158+0000","last_clean":"2024-03-06T07:21:57.503158+0000","last_became_active":"2024-03-06T07:21:55.422079+0000","last_became_peered":"2024-03-06T07:21:55.422079+0000","last_unstale":"2024-03-06T07:21:57.503158+0000","last_undegraded":"2024-03-06T07:21:57.503158+0000","last_fullsized":"2024-03-06T07:21:57.503158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502844+0000","last_change":"2024-03-06T07:21:55.421461+0000","last_active":"2024-03-06T07:21:57.502844+0000","last_peered":"2024-03-06T07:21:57.502844+0000","last_clean":"2024-03-06T07:21:57.502844+0000","last_became_active":"2024-03-06T07:21:55.420795+0000","last_became_peered":"2024-03-06T07:21:55.420795+0000","last_unstale":"2024-03-06T07:21:57.502844+0000","last_undegraded":"2024-03-06T07:21:57.502844+0000","last_fullsized":"2024-03-06T07:21:57.502844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.408829+0000","last_change":"2024-03-06T07:21:55.420082+0000","last_active":"2024-03-06T07:21:56.408829+0000","last_peered":"2024-03-06T07:21:56.408829+0000","last_clean":"2024-03-06T07:21:56.408829+0000","last_became_active":"2024-03-06T07:21:55.419844+0000","last_became_peered":"2024-03-06T07:21:55.419844+0000","last_unstale":"2024-03-06T07:21:56.408829+0000","last_undegraded":"2024-03-06T07:21:56.408829+0000","last_fullsized":"2024-03-06T07:21:56.408829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:08.056399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429302+0000","last_change":"2024-03-06T07:21:55.425776+0000","last_active":"2024-03-06T07:21:56.429302+0000","last_peered":"2024-03-06T07:21:56.429302+0000","last_clean":"2024-03-06T07:21:56.429302+0000","last_became_active":"2024-03-06T07:21:55.425479+0000","last_became_peered":"2024-03-06T07:21:55.425479+0000","last_unstale":"2024-03-06T07:21:56.429302+0000","last_undegraded":"2024-03-06T07:21:56.429302+0000","last_fullsized":"2024-03-06T07:21:56.429302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410249+0000","last_change":"2024-03-06T07:21:55.416970+0000","last_active":"2024-03-06T07:21:56.410249+0000","last_peered":"2024-03-06T07:21:56.410249+0000","last_clean":"2024-03-06T07:21:56.410249+0000","last_became_active":"2024-03-06T07:21:55.416495+0000","last_became_peered":"2024-03-06T07:21:55.416495+0000","last_unstale":"2024-03-06T07:21:56.410249+0000","last_undegraded":"2024-03-06T07:21:56.410249+0000","last_fullsized":"2024-03-06T07:21:56.410249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411126+0000","last_change":"2024-03-06T07:21:55.430721+0000","last_active":"2024-03-06T07:21:56.411126+0000","last_peered":"2024-03-06T07:21:56.411126+0000","last_clean":"2024-03-06T07:21:56.411126+0000","last_became_active":"2024-03-06T07:21:55.428901+0000","last_became_peered":"2024-03-06T07:21:55.428901+0000","last_unstale":"2024-03-06T07:21:56.411126+0000","last_undegraded":"2024-03-06T07:21:56.411126+0000","last_fullsized":"2024-03-06T07:21:56.411126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:49.817183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502966+0000","last_change":"2024-03-06T07:21:55.428378+0000","last_active":"2024-03-06T07:21:57.502966+0000","last_peered":"2024-03-06T07:21:57.502966+0000","last_clean":"2024-03-06T07:21:57.502966+0000","last_became_active":"2024-03-06T07:21:55.428101+0000","last_became_peered":"2024-03-06T07:21:55.428101+0000","last_unstale":"2024-03-06T07:21:57.502966+0000","last_undegraded":"2024-03-06T07:21:57.502966+0000","last_fullsized":"2024-03-06T07:21:57.502966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503619+0000","last_change":"2024-03-06T07:21:55.425219+0000","last_active":"2024-03-06T07:21:57.503619+0000","last_peered":"2024-03-06T07:21:57.503619+0000","last_clean":"2024-03-06T07:21:57.503619+0000","last_became_active":"2024-03-06T07:21:55.422831+0000","last_became_peered":"2024-03-06T07:21:55.422831+0000","last_unstale":"2024-03-06T07:21:57.503619+0000","last_undegraded":"2024-03-06T07:21:57.503619+0000","last_fullsized":"2024-03-06T07:21:57.503619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:32.596022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410649+0000","last_change":"2024-03-06T07:21:55.427953+0000","last_active":"2024-03-06T07:21:56.410649+0000","last_peered":"2024-03-06T07:21:56.410649+0000","last_clean":"2024-03-06T07:21:56.410649+0000","last_became_active":"2024-03-06T07:21:55.427652+0000","last_became_peered":"2024-03-06T07:21:55.427652+0000","last_unstale":"2024-03-06T07:21:56.410649+0000","last_undegraded":"2024-03-06T07:21:56.410649+0000","last_fullsized":"2024-03-06T07:21:56.410649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409090+0000","last_change":"2024-03-06T07:21:55.421525+0000","last_active":"2024-03-06T07:21:56.409090+0000","last_peered":"2024-03-06T07:21:56.409090+0000","last_clean":"2024-03-06T07:21:56.409090+0000","last_became_active":"2024-03-06T07:21:55.421204+0000","last_became_peered":"2024-03-06T07:21:55.421204+0000","last_unstale":"2024-03-06T07:21:56.409090+0000","last_undegraded":"2024-03-06T07:21:56.409090+0000","last_fullsized":"2024-03-06T07:21:56.409090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411437+0000","last_change":"2024-03-06T07:21:55.424534+0000","last_active":"2024-03-06T07:21:56.411437+0000","last_peered":"2024-03-06T07:21:56.411437+0000","last_clean":"2024-03-06T07:21:56.411437+0000","last_became_active":"2024-03-06T07:21:55.424309+0000","last_became_peered":"2024-03-06T07:21:55.424309+0000","last_unstale":"2024-03-06T07:21:56.411437+0000","last_undegraded":"2024-03-06T07:21:56.411437+0000","last_fullsized":"2024-03-06T07:21:56.411437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:16.890306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503336+0000","last_change":"2024-03-06T07:21:55.424084+0000","last_active":"2024-03-06T07:21:57.503336+0000","last_peered":"2024-03-06T07:21:57.503336+0000","last_clean":"2024-03-06T07:21:57.503336+0000","last_became_active":"2024-03-06T07:21:55.423245+0000","last_became_peered":"2024-03-06T07:21:55.423245+0000","last_unstale":"2024-03-06T07:21:57.503336+0000","last_undegraded":"2024-03-06T07:21:57.503336+0000","last_fullsized":"2024-03-06T07:21:57.503336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:41.154073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410397+0000","last_change":"2024-03-06T07:21:55.415889+0000","last_active":"2024-03-06T07:21:56.410397+0000","last_peered":"2024-03-06T07:21:56.410397+0000","last_clean":"2024-03-06T07:21:56.410397+0000","last_became_active":"2024-03-06T07:21:55.415620+0000","last_became_peered":"2024-03-06T07:21:55.415620+0000","last_unstale":"2024-03-06T07:21:56.410397+0000","last_undegraded":"2024-03-06T07:21:56.410397+0000","last_fullsized":"2024-03-06T07:21:56.410397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503106+0000","last_change":"2024-03-06T07:21:55.422611+0000","last_active":"2024-03-06T07:21:57.503106+0000","last_peered":"2024-03-06T07:21:57.503106+0000","last_clean":"2024-03-06T07:21:57.503106+0000","last_became_active":"2024-03-06T07:21:55.422260+0000","last_became_peered":"2024-03-06T07:21:55.422260+0000","last_unstale":"2024-03-06T07:21:57.503106+0000","last_undegraded":"2024-03-06T07:21:57.503106+0000","last_fullsized":"2024-03-06T07:21:57.503106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411077+0000","last_change":"2024-03-06T07:21:55.420808+0000","last_active":"2024-03-06T07:21:56.411077+0000","last_peered":"2024-03-06T07:21:56.411077+0000","last_clean":"2024-03-06T07:21:56.411077+0000","last_became_active":"2024-03-06T07:21:55.420529+0000","last_became_peered":"2024-03-06T07:21:55.420529+0000","last_unstale":"2024-03-06T07:21:56.411077+0000","last_undegraded":"2024-03-06T07:21:56.411077+0000","last_fullsized":"2024-03-06T07:21:56.411077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429492+0000","last_change":"2024-03-06T07:21:55.420655+0000","last_active":"2024-03-06T07:21:56.429492+0000","last_peered":"2024-03-06T07:21:56.429492+0000","last_clean":"2024-03-06T07:21:56.429492+0000","last_became_active":"2024-03-06T07:21:55.420396+0000","last_became_peered":"2024-03-06T07:21:55.420396+0000","last_unstale":"2024-03-06T07:21:56.429492+0000","last_undegraded":"2024-03-06T07:21:56.429492+0000","last_fullsized":"2024-03-06T07:21:56.429492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503486+0000","last_change":"2024-03-06T07:21:55.431350+0000","last_active":"2024-03-06T07:21:57.503486+0000","last_peered":"2024-03-06T07:21:57.503486+0000","last_clean":"2024-03-06T07:21:57.503486+0000","last_became_active":"2024-03-06T07:21:55.431033+0000","last_became_peered":"2024-03-06T07:21:55.431033+0000","last_unstale":"2024-03-06T07:21:57.503486+0000","last_undegraded":"2024-03-06T07:21:57.503486+0000","last_fullsized":"2024-03-06T07:21:57.503486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:47.365701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409140+0000","last_change":"2024-03-06T07:21:55.416463+0000","last_active":"2024-03-06T07:21:56.409140+0000","last_peered":"2024-03-06T07:21:56.409140+0000","last_clean":"2024-03-06T07:21:56.409140+0000","last_became_active":"2024-03-06T07:21:55.416174+0000","last_became_peered":"2024-03-06T07:21:55.416174+0000","last_unstale":"2024-03-06T07:21:56.409140+0000","last_undegraded":"2024-03-06T07:21:56.409140+0000","last_fullsized":"2024-03-06T07:21:56.409140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411495+0000","last_change":"2024-03-06T07:21:55.418595+0000","last_active":"2024-03-06T07:21:56.411495+0000","last_peered":"2024-03-06T07:21:56.411495+0000","last_clean":"2024-03-06T07:21:56.411495+0000","last_became_active":"2024-03-06T07:21:55.418073+0000","last_became_peered":"2024-03-06T07:21:55.418073+0000","last_unstale":"2024-03-06T07:21:56.411495+0000","last_undegraded":"2024-03-06T07:21:56.411495+0000","last_fullsized":"2024-03-06T07:21:56.411495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:10.706857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502657+0000","last_change":"2024-03-06T07:21:55.429932+0000","last_active":"2024-03-06T07:21:57.502657+0000","last_peered":"2024-03-06T07:21:57.502657+0000","last_clean":"2024-03-06T07:21:57.502657+0000","last_became_active":"2024-03-06T07:21:55.429593+0000","last_became_peered":"2024-03-06T07:21:55.429593+0000","last_unstale":"2024-03-06T07:21:57.502657+0000","last_undegraded":"2024-03-06T07:21:57.502657+0000","last_fullsized":"2024-03-06T07:21:57.502657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:51.071328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410150+0000","last_change":"2024-03-06T07:21:55.436974+0000","last_active":"2024-03-06T07:21:56.410150+0000","last_peered":"2024-03-06T07:21:56.410150+0000","last_clean":"2024-03-06T07:21:56.410150+0000","last_became_active":"2024-03-06T07:21:55.436638+0000","last_became_peered":"2024-03-06T07:21:55.436638+0000","last_unstale":"2024-03-06T07:21:56.410150+0000","last_undegraded":"2024-03-06T07:21:56.410150+0000","last_fullsized":"2024-03-06T07:21:56.410150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502927+0000","last_change":"2024-03-06T07:21:55.435717+0000","last_active":"2024-03-06T07:21:57.502927+0000","last_peered":"2024-03-06T07:21:57.502927+0000","last_clean":"2024-03-06T07:21:57.502927+0000","last_became_active":"2024-03-06T07:21:55.435429+0000","last_became_peered":"2024-03-06T07:21:55.435429+0000","last_unstale":"2024-03-06T07:21:57.502927+0000","last_undegraded":"2024-03-06T07:21:57.502927+0000","last_fullsized":"2024-03-06T07:21:57.502927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:41.628480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503089+0000","last_change":"2024-03-06T07:21:54.413886+0000","last_active":"2024-03-06T07:21:57.503089+0000","last_peered":"2024-03-06T07:21:57.503089+0000","last_clean":"2024-03-06T07:21:57.503089+0000","last_became_active":"2024-03-06T07:21:54.413573+0000","last_became_peered":"2024-03-06T07:21:54.413573+0000","last_unstale":"2024-03-06T07:21:57.503089+0000","last_undegraded":"2024-03-06T07:21:57.503089+0000","last_fullsized":"2024-03-06T07:21:57.503089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:17.458904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409253+0000","last_change":"2024-03-06T07:21:55.427074+0000","last_active":"2024-03-06T07:21:56.409253+0000","last_peered":"2024-03-06T07:21:56.409253+0000","last_clean":"2024-03-06T07:21:56.409253+0000","last_became_active":"2024-03-06T07:21:55.426775+0000","last_became_peered":"2024-03-06T07:21:55.426775+0000","last_unstale":"2024-03-06T07:21:56.409253+0000","last_undegraded":"2024-03-06T07:21:56.409253+0000","last_fullsized":"2024-03-06T07:21:56.409253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:41.154073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410426+0000","last_change":"2024-03-06T07:21:54.418543+0000","last_active":"2024-03-06T07:21:56.410426+0000","last_peered":"2024-03-06T07:21:56.410426+0000","last_clean":"2024-03-06T07:21:56.410426+0000","last_became_active":"2024-03-06T07:21:54.418148+0000","last_became_peered":"2024-03-06T07:21:54.418148+0000","last_unstale":"2024-03-06T07:21:56.410426+0000","last_undegraded":"2024-03-06T07:21:56.410426+0000","last_fullsized":"2024-03-06T07:21:56.410426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503384+0000","last_change":"2024-03-06T07:21:54.417756+0000","last_active":"2024-03-06T07:21:57.503384+0000","last_peered":"2024-03-06T07:21:57.503384+0000","last_clean":"2024-03-06T07:21:57.503384+0000","last_became_active":"2024-03-06T07:21:54.417471+0000","last_became_peered":"2024-03-06T07:21:54.417471+0000","last_unstale":"2024-03-06T07:21:57.503384+0000","last_undegraded":"2024-03-06T07:21:57.503384+0000","last_fullsized":"2024-03-06T07:21:57.503384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503246+0000","last_change":"2024-03-06T07:21:55.418688+0000","last_active":"2024-03-06T07:21:57.503246+0000","last_peered":"2024-03-06T07:21:57.503246+0000","last_clean":"2024-03-06T07:21:57.503246+0000","last_became_active":"2024-03-06T07:21:55.418392+0000","last_became_peered":"2024-03-06T07:21:55.418392+0000","last_unstale":"2024-03-06T07:21:57.503246+0000","last_undegraded":"2024-03-06T07:21:57.503246+0000","last_fullsized":"2024-03-06T07:21:57.503246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410615+0000","last_change":"2024-03-06T07:21:54.414567+0000","last_active":"2024-03-06T07:21:56.410615+0000","last_peered":"2024-03-06T07:21:56.410615+0000","last_clean":"2024-03-06T07:21:56.410615+0000","last_became_active":"2024-03-06T07:21:54.414292+0000","last_became_peered":"2024-03-06T07:21:54.414292+0000","last_unstale":"2024-03-06T07:21:56.410615+0000","last_undegraded":"2024-03-06T07:21:56.410615+0000","last_fullsized":"2024-03-06T07:21:56.410615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429274+0000","last_change":"2024-03-06T07:21:55.420297+0000","last_active":"2024-03-06T07:21:56.429274+0000","last_peered":"2024-03-06T07:21:56.429274+0000","last_clean":"2024-03-06T07:21:56.429274+0000","last_became_active":"2024-03-06T07:21:55.419994+0000","last_became_peered":"2024-03-06T07:21:55.419994+0000","last_unstale":"2024-03-06T07:21:56.429274+0000","last_undegraded":"2024-03-06T07:21:56.429274+0000","last_fullsized":"2024-03-06T07:21:56.429274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410774+0000","last_change":"2024-03-06T07:21:54.420034+0000","last_active":"2024-03-06T07:21:56.410774+0000","last_peered":"2024-03-06T07:21:56.410774+0000","last_clean":"2024-03-06T07:21:56.410774+0000","last_became_active":"2024-03-06T07:21:54.419745+0000","last_became_peered":"2024-03-06T07:21:54.419745+0000","last_unstale":"2024-03-06T07:21:56.410774+0000","last_undegraded":"2024-03-06T07:21:56.410774+0000","last_fullsized":"2024-03-06T07:21:56.410774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410843+0000","last_change":"2024-03-06T07:21:55.425571+0000","last_active":"2024-03-06T07:21:56.410843+0000","last_peered":"2024-03-06T07:21:56.410843+0000","last_clean":"2024-03-06T07:21:56.410843+0000","last_became_active":"2024-03-06T07:21:55.424876+0000","last_became_peered":"2024-03-06T07:21:55.424876+0000","last_unstale":"2024-03-06T07:21:56.410843+0000","last_undegraded":"2024-03-06T07:21:56.410843+0000","last_fullsized":"2024-03-06T07:21:56.410843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:40:41.154073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410852+0000","last_change":"2024-03-06T07:21:54.416884+0000","last_active":"2024-03-06T07:21:56.410852+0000","last_peered":"2024-03-06T07:21:56.410852+0000","last_clean":"2024-03-06T07:21:56.410852+0000","last_became_active":"2024-03-06T07:21:54.416576+0000","last_became_peered":"2024-03-06T07:21:54.416576+0000","last_unstale":"2024-03-06T07:21:56.410852+0000","last_undegraded":"2024-03-06T07:21:56.410852+0000","last_fullsized":"2024-03-06T07:21:56.410852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409127+0000","last_change":"2024-03-06T07:21:55.414257+0000","last_active":"2024-03-06T07:21:56.409127+0000","last_peered":"2024-03-06T07:21:56.409127+0000","last_clean":"2024-03-06T07:21:56.409127+0000","last_became_active":"2024-03-06T07:21:55.413950+0000","last_became_peered":"2024-03-06T07:21:55.413950+0000","last_unstale":"2024-03-06T07:21:56.409127+0000","last_undegraded":"2024-03-06T07:21:56.409127+0000","last_fullsized":"2024-03-06T07:21:56.409127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:44.697071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429387+0000","last_change":"2024-03-06T07:21:54.412699+0000","last_active":"2024-03-06T07:21:56.429387+0000","last_peered":"2024-03-06T07:21:56.429387+0000","last_clean":"2024-03-06T07:21:56.429387+0000","last_became_active":"2024-03-06T07:21:54.412392+0000","last_became_peered":"2024-03-06T07:21:54.412392+0000","last_unstale":"2024-03-06T07:21:56.429387+0000","last_undegraded":"2024-03-06T07:21:56.429387+0000","last_fullsized":"2024-03-06T07:21:56.429387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503297+0000","last_change":"2024-03-06T07:21:55.421734+0000","last_active":"2024-03-06T07:21:57.503297+0000","last_peered":"2024-03-06T07:21:57.503297+0000","last_clean":"2024-03-06T07:21:57.503297+0000","last_became_active":"2024-03-06T07:21:55.421451+0000","last_became_peered":"2024-03-06T07:21:55.421451+0000","last_unstale":"2024-03-06T07:21:57.503297+0000","last_undegraded":"2024-03-06T07:21:57.503297+0000","last_fullsized":"2024-03-06T07:21:57.503297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:45.270105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410763+0000","last_change":"2024-03-06T07:21:54.416848+0000","last_active":"2024-03-06T07:21:56.410763+0000","last_peered":"2024-03-06T07:21:56.410763+0000","last_clean":"2024-03-06T07:21:56.410763+0000","last_became_active":"2024-03-06T07:21:54.416561+0000","last_became_peered":"2024-03-06T07:21:54.416561+0000","last_unstale":"2024-03-06T07:21:56.410763+0000","last_undegraded":"2024-03-06T07:21:56.410763+0000","last_fullsized":"2024-03-06T07:21:56.410763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503047+0000","last_change":"2024-03-06T07:21:55.425748+0000","last_active":"2024-03-06T07:21:57.503047+0000","last_peered":"2024-03-06T07:21:57.503047+0000","last_clean":"2024-03-06T07:21:57.503047+0000","last_became_active":"2024-03-06T07:21:55.425501+0000","last_became_peered":"2024-03-06T07:21:55.425501+0000","last_unstale":"2024-03-06T07:21:57.503047+0000","last_undegraded":"2024-03-06T07:21:57.503047+0000","last_fullsized":"2024-03-06T07:21:57.503047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:08.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409204+0000","last_change":"2024-03-06T07:21:54.419364+0000","last_active":"2024-03-06T07:21:56.409204+0000","last_peered":"2024-03-06T07:21:56.409204+0000","last_clean":"2024-03-06T07:21:56.409204+0000","last_became_active":"2024-03-06T07:21:54.419094+0000","last_became_peered":"2024-03-06T07:21:54.419094+0000","last_unstale":"2024-03-06T07:21:56.409204+0000","last_undegraded":"2024-03-06T07:21:56.409204+0000","last_fullsized":"2024-03-06T07:21:56.409204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.408850+0000","last_change":"2024-03-06T07:21:55.421461+0000","last_active":"2024-03-06T07:21:56.408850+0000","last_peered":"2024-03-06T07:21:56.408850+0000","last_clean":"2024-03-06T07:21:56.408850+0000","last_became_active":"2024-03-06T07:21:55.421153+0000","last_became_peered":"2024-03-06T07:21:55.421153+0000","last_unstale":"2024-03-06T07:21:56.408850+0000","last_undegraded":"2024-03-06T07:21:56.408850+0000","last_fullsized":"2024-03-06T07:21:56.408850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:40.096287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411554+0000","last_change":"2024-03-06T07:21:54.416121+0000","last_active":"2024-03-06T07:21:56.411554+0000","last_peered":"2024-03-06T07:21:56.411554+0000","last_clean":"2024-03-06T07:21:56.411554+0000","last_became_active":"2024-03-06T07:21:54.415857+0000","last_became_peered":"2024-03-06T07:21:54.415857+0000","last_unstale":"2024-03-06T07:21:56.411554+0000","last_undegraded":"2024-03-06T07:21:56.411554+0000","last_fullsized":"2024-03-06T07:21:56.411554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503422+0000","last_change":"2024-03-06T07:21:55.428224+0000","last_active":"2024-03-06T07:21:57.503422+0000","last_peered":"2024-03-06T07:21:57.503422+0000","last_clean":"2024-03-06T07:21:57.503422+0000","last_became_active":"2024-03-06T07:21:55.427932+0000","last_became_peered":"2024-03-06T07:21:55.427932+0000","last_unstale":"2024-03-06T07:21:57.503422+0000","last_undegraded":"2024-03-06T07:21:57.503422+0000","last_fullsized":"2024-03-06T07:21:57.503422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:10.706857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410172+0000","last_change":"2024-03-06T07:21:54.418770+0000","last_active":"2024-03-06T07:21:56.410172+0000","last_peered":"2024-03-06T07:21:56.410172+0000","last_clean":"2024-03-06T07:21:56.410172+0000","last_became_active":"2024-03-06T07:21:54.418498+0000","last_became_peered":"2024-03-06T07:21:54.418498+0000","last_unstale":"2024-03-06T07:21:56.410172+0000","last_undegraded":"2024-03-06T07:21:56.410172+0000","last_fullsized":"2024-03-06T07:21:56.410172+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410331+0000","last_change":"2024-03-06T07:21:55.436175+0000","last_active":"2024-03-06T07:21:56.410331+0000","last_peered":"2024-03-06T07:21:56.410331+0000","last_clean":"2024-03-06T07:21:56.410331+0000","last_became_active":"2024-03-06T07:21:55.435873+0000","last_became_peered":"2024-03-06T07:21:55.435873+0000","last_unstale":"2024-03-06T07:21:56.410331+0000","last_undegraded":"2024-03-06T07:21:56.410331+0000","last_fullsized":"2024-03-06T07:21:56.410331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502508+0000","last_change":"2024-03-06T07:21:54.410321+0000","last_active":"2024-03-06T07:21:57.502508+0000","last_peered":"2024-03-06T07:21:57.502508+0000","last_clean":"2024-03-06T07:21:57.502508+0000","last_became_active":"2024-03-06T07:21:54.410019+0000","last_became_peered":"2024-03-06T07:21:54.410019+0000","last_unstale":"2024-03-06T07:21:57.502508+0000","last_undegraded":"2024-03-06T07:21:57.502508+0000","last_fullsized":"2024-03-06T07:21:57.502508+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:36.102960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503336+0000","last_change":"2024-03-06T07:21:55.427171+0000","last_active":"2024-03-06T07:21:57.503336+0000","last_peered":"2024-03-06T07:21:57.503336+0000","last_clean":"2024-03-06T07:21:57.503336+0000","last_became_active":"2024-03-06T07:21:55.426872+0000","last_became_peered":"2024-03-06T07:21:55.426872+0000","last_unstale":"2024-03-06T07:21:57.503336+0000","last_undegraded":"2024-03-06T07:21:57.503336+0000","last_fullsized":"2024-03-06T07:21:57.503336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:44.132915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503361+0000","last_change":"2024-03-06T07:21:54.419357+0000","last_active":"2024-03-06T07:21:57.503361+0000","last_peered":"2024-03-06T07:21:57.503361+0000","last_clean":"2024-03-06T07:21:57.503361+0000","last_became_active":"2024-03-06T07:21:54.419053+0000","last_became_peered":"2024-03-06T07:21:54.419053+0000","last_unstale":"2024-03-06T07:21:57.503361+0000","last_undegraded":"2024-03-06T07:21:57.503361+0000","last_fullsized":"2024-03-06T07:21:57.503361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:42.184203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503583+0000","last_change":"2024-03-06T07:21:55.427511+0000","last_active":"2024-03-06T07:21:57.503583+0000","last_peered":"2024-03-06T07:21:57.503583+0000","last_clean":"2024-03-06T07:21:57.503583+0000","last_became_active":"2024-03-06T07:21:55.427175+0000","last_became_peered":"2024-03-06T07:21:55.427175+0000","last_unstale":"2024-03-06T07:21:57.503583+0000","last_undegraded":"2024-03-06T07:21:57.503583+0000","last_fullsized":"2024-03-06T07:21:57.503583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:05.685917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502735+0000","last_change":"2024-03-06T07:21:54.416325+0000","last_active":"2024-03-06T07:21:57.502735+0000","last_peered":"2024-03-06T07:21:57.502735+0000","last_clean":"2024-03-06T07:21:57.502735+0000","last_became_active":"2024-03-06T07:21:54.416017+0000","last_became_peered":"2024-03-06T07:21:54.416017+0000","last_unstale":"2024-03-06T07:21:57.502735+0000","last_undegraded":"2024-03-06T07:21:57.502735+0000","last_fullsized":"2024-03-06T07:21:57.502735+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503730+0000","last_change":"2024-03-06T07:21:55.428378+0000","last_active":"2024-03-06T07:21:57.503730+0000","last_peered":"2024-03-06T07:21:57.503730+0000","last_clean":"2024-03-06T07:21:57.503730+0000","last_became_active":"2024-03-06T07:21:55.426715+0000","last_became_peered":"2024-03-06T07:21:55.426715+0000","last_unstale":"2024-03-06T07:21:57.503730+0000","last_undegraded":"2024-03-06T07:21:57.503730+0000","last_fullsized":"2024-03-06T07:21:57.503730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:14:29.756209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429581+0000","last_change":"2024-03-06T07:21:54.406786+0000","last_active":"2024-03-06T07:21:56.429581+0000","last_peered":"2024-03-06T07:21:56.429581+0000","last_clean":"2024-03-06T07:21:56.429581+0000","last_became_active":"2024-03-06T07:21:54.406454+0000","last_became_peered":"2024-03-06T07:21:54.406454+0000","last_unstale":"2024-03-06T07:21:56.429581+0000","last_undegraded":"2024-03-06T07:21:56.429581+0000","last_fullsized":"2024-03-06T07:21:56.429581+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:50.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411653+0000","last_change":"2024-03-06T07:21:55.422470+0000","last_active":"2024-03-06T07:21:56.411653+0000","last_peered":"2024-03-06T07:21:56.411653+0000","last_clean":"2024-03-06T07:21:56.411653+0000","last_became_active":"2024-03-06T07:21:55.420656+0000","last_became_peered":"2024-03-06T07:21:55.420656+0000","last_unstale":"2024-03-06T07:21:56.411653+0000","last_undegraded":"2024-03-06T07:21:56.411653+0000","last_fullsized":"2024-03-06T07:21:56.411653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:32.596022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502899+0000","last_change":"2024-03-06T07:21:54.412695+0000","last_active":"2024-03-06T07:21:57.502899+0000","last_peered":"2024-03-06T07:21:57.502899+0000","last_clean":"2024-03-06T07:21:57.502899+0000","last_became_active":"2024-03-06T07:21:54.412391+0000","last_became_peered":"2024-03-06T07:21:54.412391+0000","last_unstale":"2024-03-06T07:21:57.502899+0000","last_undegraded":"2024-03-06T07:21:57.502899+0000","last_fullsized":"2024-03-06T07:21:57.502899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:17.458904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503199+0000","last_change":"2024-03-06T07:21:55.436949+0000","last_active":"2024-03-06T07:21:57.503199+0000","last_peered":"2024-03-06T07:21:57.503199+0000","last_clean":"2024-03-06T07:21:57.503199+0000","last_became_active":"2024-03-06T07:21:55.436502+0000","last_became_peered":"2024-03-06T07:21:55.436502+0000","last_unstale":"2024-03-06T07:21:57.503199+0000","last_undegraded":"2024-03-06T07:21:57.503199+0000","last_fullsized":"2024-03-06T07:21:57.503199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:31.378193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503196+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:21:57.503196+0000","last_peered":"2024-03-06T07:21:57.503196+0000","last_clean":"2024-03-06T07:21:57.503196+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:21:57.503196+0000","last_undegraded":"2024-03-06T07:21:57.503196+0000","last_fullsized":"2024-03-06T07:21:57.503196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410901+0000","last_change":"2024-03-06T07:21:54.411371+0000","last_active":"2024-03-06T07:21:56.410901+0000","last_peered":"2024-03-06T07:21:56.410901+0000","last_clean":"2024-03-06T07:21:56.410901+0000","last_became_active":"2024-03-06T07:21:54.411095+0000","last_became_peered":"2024-03-06T07:21:54.411095+0000","last_unstale":"2024-03-06T07:21:56.410901+0000","last_undegraded":"2024-03-06T07:21:56.410901+0000","last_fullsized":"2024-03-06T07:21:56.410901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503359+0000","last_change":"2024-03-06T07:21:55.423928+0000","last_active":"2024-03-06T07:21:57.503359+0000","last_peered":"2024-03-06T07:21:57.503359+0000","last_clean":"2024-03-06T07:21:57.503359+0000","last_became_active":"2024-03-06T07:21:55.423681+0000","last_became_peered":"2024-03-06T07:21:55.423681+0000","last_unstale":"2024-03-06T07:21:57.503359+0000","last_undegraded":"2024-03-06T07:21:57.503359+0000","last_fullsized":"2024-03-06T07:21:57.503359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:49.817183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503508+0000","last_change":"2024-03-06T07:21:54.415381+0000","last_active":"2024-03-06T07:21:57.503508+0000","last_peered":"2024-03-06T07:21:57.503508+0000","last_clean":"2024-03-06T07:21:57.503508+0000","last_became_active":"2024-03-06T07:21:54.415099+0000","last_became_peered":"2024-03-06T07:21:54.415099+0000","last_unstale":"2024-03-06T07:21:57.503508+0000","last_undegraded":"2024-03-06T07:21:57.503508+0000","last_fullsized":"2024-03-06T07:21:57.503508+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:40.891311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.408864+0000","last_change":"2024-03-06T07:21:55.423013+0000","last_active":"2024-03-06T07:21:56.408864+0000","last_peered":"2024-03-06T07:21:56.408864+0000","last_clean":"2024-03-06T07:21:56.408864+0000","last_became_active":"2024-03-06T07:21:55.422717+0000","last_became_peered":"2024-03-06T07:21:55.422717+0000","last_unstale":"2024-03-06T07:21:56.408864+0000","last_undegraded":"2024-03-06T07:21:56.408864+0000","last_fullsized":"2024-03-06T07:21:56.408864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:45:11.682259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502628+0000","last_change":"2024-03-06T07:21:54.416728+0000","last_active":"2024-03-06T07:21:57.502628+0000","last_peered":"2024-03-06T07:21:57.502628+0000","last_clean":"2024-03-06T07:21:57.502628+0000","last_became_active":"2024-03-06T07:21:54.416226+0000","last_became_peered":"2024-03-06T07:21:54.416226+0000","last_unstale":"2024-03-06T07:21:57.502628+0000","last_undegraded":"2024-03-06T07:21:57.502628+0000","last_fullsized":"2024-03-06T07:21:57.502628+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:38.486123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429308+0000","last_change":"2024-03-06T07:21:55.429746+0000","last_active":"2024-03-06T07:21:56.429308+0000","last_peered":"2024-03-06T07:21:56.429308+0000","last_clean":"2024-03-06T07:21:56.429308+0000","last_became_active":"2024-03-06T07:21:55.429320+0000","last_became_peered":"2024-03-06T07:21:55.429320+0000","last_unstale":"2024-03-06T07:21:56.429308+0000","last_undegraded":"2024-03-06T07:21:56.429308+0000","last_fullsized":"2024-03-06T07:21:56.429308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411588+0000","last_change":"2024-03-06T07:21:54.413058+0000","last_active":"2024-03-06T07:21:56.411588+0000","last_peered":"2024-03-06T07:21:56.411588+0000","last_clean":"2024-03-06T07:21:56.411588+0000","last_became_active":"2024-03-06T07:21:54.412469+0000","last_became_peered":"2024-03-06T07:21:54.412469+0000","last_unstale":"2024-03-06T07:21:56.411588+0000","last_undegraded":"2024-03-06T07:21:56.411588+0000","last_fullsized":"2024-03-06T07:21:56.411588+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503091+0000","last_change":"2024-03-06T07:21:55.425788+0000","last_active":"2024-03-06T07:21:57.503091+0000","last_peered":"2024-03-06T07:21:57.503091+0000","last_clean":"2024-03-06T07:21:57.503091+0000","last_became_active":"2024-03-06T07:21:55.425440+0000","last_became_peered":"2024-03-06T07:21:55.425440+0000","last_unstale":"2024-03-06T07:21:57.503091+0000","last_undegraded":"2024-03-06T07:21:57.503091+0000","last_fullsized":"2024-03-06T07:21:57.503091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:36.844180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429371+0000","last_change":"2024-03-06T07:21:54.415739+0000","last_active":"2024-03-06T07:21:56.429371+0000","last_peered":"2024-03-06T07:21:56.429371+0000","last_clean":"2024-03-06T07:21:56.429371+0000","last_became_active":"2024-03-06T07:21:54.415452+0000","last_became_peered":"2024-03-06T07:21:54.415452+0000","last_unstale":"2024-03-06T07:21:56.429371+0000","last_undegraded":"2024-03-06T07:21:56.429371+0000","last_fullsized":"2024-03-06T07:21:56.429371+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:40.622061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411402+0000","last_change":"2024-03-06T07:21:55.419140+0000","last_active":"2024-03-06T07:21:56.411402+0000","last_peered":"2024-03-06T07:21:56.411402+0000","last_clean":"2024-03-06T07:21:56.411402+0000","last_became_active":"2024-03-06T07:21:55.415992+0000","last_became_peered":"2024-03-06T07:21:55.415992+0000","last_unstale":"2024-03-06T07:21:56.411402+0000","last_undegraded":"2024-03-06T07:21:56.411402+0000","last_fullsized":"2024-03-06T07:21:56.411402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:31.378193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503514+0000","last_change":"2024-03-06T07:21:54.417366+0000","last_active":"2024-03-06T07:21:57.503514+0000","last_peered":"2024-03-06T07:21:57.503514+0000","last_clean":"2024-03-06T07:21:57.503514+0000","last_became_active":"2024-03-06T07:21:54.416684+0000","last_became_peered":"2024-03-06T07:21:54.416684+0000","last_unstale":"2024-03-06T07:21:57.503514+0000","last_undegraded":"2024-03-06T07:21:57.503514+0000","last_fullsized":"2024-03-06T07:21:57.503514+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:41:44.263686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429397+0000","last_change":"2024-03-06T07:21:55.419798+0000","last_active":"2024-03-06T07:21:56.429397+0000","last_peered":"2024-03-06T07:21:56.429397+0000","last_clean":"2024-03-06T07:21:56.429397+0000","last_became_active":"2024-03-06T07:21:55.419490+0000","last_became_peered":"2024-03-06T07:21:55.419490+0000","last_unstale":"2024-03-06T07:21:56.429397+0000","last_undegraded":"2024-03-06T07:21:56.429397+0000","last_fullsized":"2024-03-06T07:21:56.429397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429552+0000","last_change":"2024-03-06T07:21:54.418937+0000","last_active":"2024-03-06T07:21:56.429552+0000","last_peered":"2024-03-06T07:21:56.429552+0000","last_clean":"2024-03-06T07:21:56.429552+0000","last_became_active":"2024-03-06T07:21:54.418592+0000","last_became_peered":"2024-03-06T07:21:54.418592+0000","last_unstale":"2024-03-06T07:21:56.429552+0000","last_undegraded":"2024-03-06T07:21:56.429552+0000","last_fullsized":"2024-03-06T07:21:56.429552+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:17.773963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503269+0000","last_change":"2024-03-06T07:21:55.420527+0000","last_active":"2024-03-06T07:21:57.503269+0000","last_peered":"2024-03-06T07:21:57.503269+0000","last_clean":"2024-03-06T07:21:57.503269+0000","last_became_active":"2024-03-06T07:21:55.420210+0000","last_became_peered":"2024-03-06T07:21:55.420210+0000","last_unstale":"2024-03-06T07:21:57.503269+0000","last_undegraded":"2024-03-06T07:21:57.503269+0000","last_fullsized":"2024-03-06T07:21:57.503269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:18.317708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410686+0000","last_change":"2024-03-06T07:21:54.420765+0000","last_active":"2024-03-06T07:21:56.410686+0000","last_peered":"2024-03-06T07:21:56.410686+0000","last_clean":"2024-03-06T07:21:56.410686+0000","last_became_active":"2024-03-06T07:21:54.420541+0000","last_became_peered":"2024-03-06T07:21:54.420541+0000","last_unstale":"2024-03-06T07:21:56.410686+0000","last_undegraded":"2024-03-06T07:21:56.410686+0000","last_fullsized":"2024-03-06T07:21:56.410686+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409085+0000","last_change":"2024-03-06T07:21:55.430019+0000","last_active":"2024-03-06T07:21:56.409085+0000","last_peered":"2024-03-06T07:21:56.409085+0000","last_clean":"2024-03-06T07:21:56.409085+0000","last_became_active":"2024-03-06T07:21:55.429671+0000","last_became_peered":"2024-03-06T07:21:55.429671+0000","last_unstale":"2024-03-06T07:21:56.409085+0000","last_undegraded":"2024-03-06T07:21:56.409085+0000","last_fullsized":"2024-03-06T07:21:56.409085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502709+0000","last_change":"2024-03-06T07:21:54.416295+0000","last_active":"2024-03-06T07:21:57.502709+0000","last_peered":"2024-03-06T07:21:57.502709+0000","last_clean":"2024-03-06T07:21:57.502709+0000","last_became_active":"2024-03-06T07:21:54.415817+0000","last_became_peered":"2024-03-06T07:21:54.415817+0000","last_unstale":"2024-03-06T07:21:57.502709+0000","last_undegraded":"2024-03-06T07:21:57.502709+0000","last_fullsized":"2024-03-06T07:21:57.502709+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:28.435323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503166+0000","last_change":"2024-03-06T07:21:55.422908+0000","last_active":"2024-03-06T07:21:57.503166+0000","last_peered":"2024-03-06T07:21:57.503166+0000","last_clean":"2024-03-06T07:21:57.503166+0000","last_became_active":"2024-03-06T07:21:55.422624+0000","last_became_peered":"2024-03-06T07:21:55.422624+0000","last_unstale":"2024-03-06T07:21:57.503166+0000","last_undegraded":"2024-03-06T07:21:57.503166+0000","last_fullsized":"2024-03-06T07:21:57.503166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:18.780382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429396+0000","last_change":"2024-03-06T07:21:54.411585+0000","last_active":"2024-03-06T07:21:56.429396+0000","last_peered":"2024-03-06T07:21:56.429396+0000","last_clean":"2024-03-06T07:21:56.429396+0000","last_became_active":"2024-03-06T07:21:54.411318+0000","last_became_peered":"2024-03-06T07:21:54.411318+0000","last_unstale":"2024-03-06T07:21:56.429396+0000","last_undegraded":"2024-03-06T07:21:56.429396+0000","last_fullsized":"2024-03-06T07:21:56.429396+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503397+0000","last_change":"2024-03-06T07:21:55.421416+0000","last_active":"2024-03-06T07:21:57.503397+0000","last_peered":"2024-03-06T07:21:57.503397+0000","last_clean":"2024-03-06T07:21:57.503397+0000","last_became_active":"2024-03-06T07:21:55.421113+0000","last_became_peered":"2024-03-06T07:21:55.421113+0000","last_unstale":"2024-03-06T07:21:57.503397+0000","last_undegraded":"2024-03-06T07:21:57.503397+0000","last_fullsized":"2024-03-06T07:21:57.503397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:08.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.429482+0000","last_change":"2024-03-06T07:21:54.416417+0000","last_active":"2024-03-06T07:21:56.429482+0000","last_peered":"2024-03-06T07:21:56.429482+0000","last_clean":"2024-03-06T07:21:56.429482+0000","last_became_active":"2024-03-06T07:21:54.416108+0000","last_became_peered":"2024-03-06T07:21:54.416108+0000","last_unstale":"2024-03-06T07:21:56.429482+0000","last_undegraded":"2024-03-06T07:21:56.429482+0000","last_fullsized":"2024-03-06T07:21:56.429482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:42.197171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502995+0000","last_change":"2024-03-06T07:21:55.418459+0000","last_active":"2024-03-06T07:21:57.502995+0000","last_peered":"2024-03-06T07:21:57.502995+0000","last_clean":"2024-03-06T07:21:57.502995+0000","last_became_active":"2024-03-06T07:21:55.418179+0000","last_became_peered":"2024-03-06T07:21:55.418179+0000","last_unstale":"2024-03-06T07:21:57.502995+0000","last_undegraded":"2024-03-06T07:21:57.502995+0000","last_fullsized":"2024-03-06T07:21:57.502995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:59.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503599+0000","last_change":"2024-03-06T07:21:54.412875+0000","last_active":"2024-03-06T07:21:57.503599+0000","last_peered":"2024-03-06T07:21:57.503599+0000","last_clean":"2024-03-06T07:21:57.503599+0000","last_became_active":"2024-03-06T07:21:54.412617+0000","last_became_peered":"2024-03-06T07:21:54.412617+0000","last_unstale":"2024-03-06T07:21:57.503599+0000","last_undegraded":"2024-03-06T07:21:57.503599+0000","last_fullsized":"2024-03-06T07:21:57.503599+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:50.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.410564+0000","last_change":"2024-03-06T07:21:55.427872+0000","last_active":"2024-03-06T07:21:56.410564+0000","last_peered":"2024-03-06T07:21:56.410564+0000","last_clean":"2024-03-06T07:21:56.410564+0000","last_became_active":"2024-03-06T07:21:55.427504+0000","last_became_peered":"2024-03-06T07:21:55.427504+0000","last_unstale":"2024-03-06T07:21:56.410564+0000","last_undegraded":"2024-03-06T07:21:56.410564+0000","last_fullsized":"2024-03-06T07:21:56.410564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:16.827851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411242+0000","last_change":"2024-03-06T07:21:55.427757+0000","last_active":"2024-03-06T07:21:56.411242+0000","last_peered":"2024-03-06T07:21:56.411242+0000","last_clean":"2024-03-06T07:21:56.411242+0000","last_became_active":"2024-03-06T07:21:55.427471+0000","last_became_peered":"2024-03-06T07:21:55.427471+0000","last_unstale":"2024-03-06T07:21:56.411242+0000","last_undegraded":"2024-03-06T07:21:56.411242+0000","last_fullsized":"2024-03-06T07:21:56.411242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:08.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503412+0000","last_change":"2024-03-06T07:21:54.414035+0000","last_active":"2024-03-06T07:21:57.503412+0000","last_peered":"2024-03-06T07:21:57.503412+0000","last_clean":"2024-03-06T07:21:57.503412+0000","last_became_active":"2024-03-06T07:21:54.413618+0000","last_became_peered":"2024-03-06T07:21:54.413618+0000","last_unstale":"2024-03-06T07:21:57.503412+0000","last_undegraded":"2024-03-06T07:21:57.503412+0000","last_fullsized":"2024-03-06T07:21:57.503412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:35.837761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.411590+0000","last_change":"2024-03-06T07:21:55.423540+0000","last_active":"2024-03-06T07:21:56.411590+0000","last_peered":"2024-03-06T07:21:56.411590+0000","last_clean":"2024-03-06T07:21:56.411590+0000","last_became_active":"2024-03-06T07:21:55.423197+0000","last_became_peered":"2024-03-06T07:21:55.423197+0000","last_unstale":"2024-03-06T07:21:56.411590+0000","last_undegraded":"2024-03-06T07:21:56.411590+0000","last_fullsized":"2024-03-06T07:21:56.411590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:59:05.685917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503403+0000","last_change":"2024-03-06T07:21:54.417365+0000","last_active":"2024-03-06T07:21:57.503403+0000","last_peered":"2024-03-06T07:21:57.503403+0000","last_clean":"2024-03-06T07:21:57.503403+0000","last_became_active":"2024-03-06T07:21:54.417072+0000","last_became_peered":"2024-03-06T07:21:54.417072+0000","last_unstale":"2024-03-06T07:21:57.503403+0000","last_undegraded":"2024-03-06T07:21:57.503403+0000","last_fullsized":"2024-03-06T07:21:57.503403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:17.773963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409094+0000","last_change":"2024-03-06T07:21:55.424176+0000","last_active":"2024-03-06T07:21:56.409094+0000","last_peered":"2024-03-06T07:21:56.409094+0000","last_clean":"2024-03-06T07:21:56.409094+0000","last_became_active":"2024-03-06T07:21:55.423848+0000","last_became_peered":"2024-03-06T07:21:55.423848+0000","last_unstale":"2024-03-06T07:21:56.409094+0000","last_undegraded":"2024-03-06T07:21:56.409094+0000","last_fullsized":"2024-03-06T07:21:56.409094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:39.983463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.503028+0000","last_change":"2024-03-06T07:21:54.418869+0000","last_active":"2024-03-06T07:21:57.503028+0000","last_peered":"2024-03-06T07:21:57.503028+0000","last_clean":"2024-03-06T07:21:57.503028+0000","last_became_active":"2024-03-06T07:21:54.418582+0000","last_became_peered":"2024-03-06T07:21:54.418582+0000","last_unstale":"2024-03-06T07:21:57.503028+0000","last_undegraded":"2024-03-06T07:21:57.503028+0000","last_fullsized":"2024-03-06T07:21:57.503028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:05.586227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:57.502919+0000","last_change":"2024-03-06T07:21:55.427658+0000","last_active":"2024-03-06T07:21:57.502919+0000","last_peered":"2024-03-06T07:21:57.502919+0000","last_clean":"2024-03-06T07:21:57.502919+0000","last_became_active":"2024-03-06T07:21:55.427370+0000","last_became_peered":"2024-03-06T07:21:55.427370+0000","last_unstale":"2024-03-06T07:21:57.502919+0000","last_undegraded":"2024-03-06T07:21:57.502919+0000","last_fullsized":"2024-03-06T07:21:57.502919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:54.395011+0000","last_clean_scrub_stamp":"2024-03-06T07:21:54.395011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.538449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-06T07:21:56.409199+0000","last_change":"2024-03-06T07:21:54.417435+0000","last_active":"2024-03-06T07:21:56.409199+0000","last_peered":"2024-03-06T07:21:56.409199+0000","last_clean":"2024-03-06T07:21:56.409199+0000","last_became_active":"2024-03-06T07:21:54.417136+0000","last_became_peered":"2024-03-06T07:21:54.417136+0000","last_unstale":"2024-03-06T07:21:56.409199+0000","last_undegraded":"2024-03-06T07:21:56.409199+0000","last_fullsized":"2024-03-06T07:21:56.409199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:53.388592+0000","last_clean_scrub_stamp":"2024-03-06T07:21:53.388592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:15.277777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:03.155 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:22:03.155 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:22:03.155 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:22:03.155 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:22:03.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.289+0000 7f92221d7640 1 Processor -- start 2024-03-06T07:22:03.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.289+0000 7f92221d7640 1 -- start start 2024-03-06T07:22:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f92221d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f921c1444c0 0x7f921c06ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f92221d7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f921c15b2d0 0x7f921c0708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f92221d7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921c06d370 con 0x7f921c070e10 2024-03-06T07:22:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f92221d7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f921c1448a0 con 0x7f921c15b2d0 2024-03-06T07:22:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f92221d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921c071260 con 0x7f921c1444c0 2024-03-06T07:22:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f921affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f921c15b2d0 0x7f921c0708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f921affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f921c15b2d0 0x7f921c0708d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58376/0 (socket says 172.21.15.59:58376) 2024-03-06T07:22:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f921affd640 1 -- 172.21.15.59:0/2837018613 learned_addr learned my addr 172.21.15.59:0/2837018613 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f921b7fe640 1 --2- 172.21.15.59:0/2837018613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f921c1444c0 0x7f921c06ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f921a7fc640 1 -- 172.21.15.59:0/2837018613 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353044592 0 0) 0x7f921c06d370 con 0x7f921c070e10 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f921affd640 1 -- 172.21.15.59:0/2837018613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f921c1444c0 msgr2=0x7f921c06ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f921affd640 1 --2- 172.21.15.59:0/2837018613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f921c1444c0 0x7f921c06ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.290+0000 7f921affd640 1 -- 172.21.15.59:0/2837018613 >> v1:172.21.15.164:6789/0 conn(0x7f921c070e10 legacy=0x7f921c15e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.291+0000 7f921affd640 1 -- 172.21.15.59:0/2837018613 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c06d650 con 0x7f921c15b2d0 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.291+0000 7f921b7fe640 1 --2- 172.21.15.59:0/2837018613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f921c1444c0 0x7f921c06ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.291+0000 7f921affd640 1 --2- 172.21.15.59:0/2837018613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f921c15b2d0 0x7f921c0708d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f921000bb10 tx=0x7f921002e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67eed0d73fe81600 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.291+0000 7f921a7fc640 1 -- 172.21.15.59:0/2837018613 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921003b070 con 0x7f921c15b2d0 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.291+0000 7f921a7fc640 1 -- 172.21.15.59:0/2837018613 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9210036440 con 0x7f921c15b2d0 2024-03-06T07:22:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.291+0000 7f921a7fc640 1 -- 172.21.15.59:0/2837018613 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210036810 con 0x7f921c15b2d0 2024-03-06T07:22:03.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.292+0000 7f92221d7640 1 -- 172.21.15.59:0/2837018613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f921c15b2d0 msgr2=0x7f921c0708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.292+0000 7f92221d7640 1 --2- 172.21.15.59:0/2837018613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f921c15b2d0 0x7f921c0708d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f921000bb10 tx=0x7f921002e150 comp rx=0 tx=0).stop 2024-03-06T07:22:03.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.292+0000 7f92221d7640 1 -- 172.21.15.59:0/2837018613 shutdown_connections 2024-03-06T07:22:03.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.292+0000 7f92221d7640 1 --2- 172.21.15.59:0/2837018613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f921c15b2d0 0x7f921c0708d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.292+0000 7f92221d7640 1 --2- 172.21.15.59:0/2837018613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f921c1444c0 0x7f921c06ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.292+0000 7f92221d7640 1 -- 172.21.15.59:0/2837018613 >> 172.21.15.59:0/2837018613 conn(0x7f921c0b1c80 msgr2=0x7f921c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.292+0000 7f92221d7640 1 -- 172.21.15.59:0/2837018613 shutdown_connections 2024-03-06T07:22:03.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.292+0000 7f92221d7640 1 -- 172.21.15.59:0/2837018613 wait complete. 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.293+0000 7f92221d7640 1 Processor -- start 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.293+0000 7f92221d7640 1 -- start start 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f92221d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f921c070e10 0x7f921c13e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f92221d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f921c1444c0 0x7f921c0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f92221d7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f921c15b2d0 0x7f921c0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f92221d7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c06def0 con 0x7f921c070e10 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f92221d7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c06dc10 con 0x7f921c15b2d0 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f92221d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f921c06cf40 con 0x7f921c1444c0 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f921b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f921c070e10 0x7f921c13e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f921b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f921c070e10 0x7f921c13e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58388/0 (socket says 172.21.15.59:58388) 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f921b7fe640 1 -- 172.21.15.59:0/2275229146 learned_addr learned my addr 172.21.15.59:0/2275229146 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f921affd640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f921c1444c0 0x7f921c0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f921bfff640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f921c15b2d0 0x7f921c0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f921b7fe640 1 -- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f921c1444c0 msgr2=0x7f921c0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f921b7fe640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f921c1444c0 0x7f921c0cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f921b7fe640 1 -- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f921c15b2d0 msgr2=0x7f921c0cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f921b7fe640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f921c15b2d0 0x7f921c0cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.294+0000 7f921b7fe640 1 -- 172.21.15.59:0/2275229146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c13e8a0 con 0x7f921c070e10 2024-03-06T07:22:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.295+0000 7f921b7fe640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f921c070e10 0x7f921c13e360 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f920c00ebe0 tx=0x7f920c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.295+0000 7f9203fff640 1 -- 172.21.15.59:0/2275229146 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c017070 con 0x7f921c070e10 2024-03-06T07:22:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.295+0000 7f9203fff640 1 -- 172.21.15.59:0/2275229146 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920c00cc30 con 0x7f921c070e10 2024-03-06T07:22:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.295+0000 7f92221d7640 1 -- 172.21.15.59:0/2275229146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921c0b3f30 con 0x7f921c070e10 2024-03-06T07:22:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.295+0000 7f9203fff640 1 -- 172.21.15.59:0/2275229146 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c013470 con 0x7f921c070e10 2024-03-06T07:22:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.295+0000 7f92221d7640 1 -- 172.21.15.59:0/2275229146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921c0b44f0 con 0x7f921c070e10 2024-03-06T07:22:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.296+0000 7f92221d7640 1 -- 172.21.15.59:0/2275229146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921c04f330 con 0x7f921c070e10 2024-03-06T07:22:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.296+0000 7f9203fff640 1 -- 172.21.15.59:0/2275229146 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f920c01a070 con 0x7f921c070e10 2024-03-06T07:22:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.296+0000 7f9203fff640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f920806b4d0 0x7f920806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.297+0000 7f9203fff640 1 -- 172.21.15.59:0/2275229146 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f920c08a470 con 0x7f921c070e10 2024-03-06T07:22:03.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.297+0000 7f921affd640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f920806b4d0 0x7f920806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.297+0000 7f921affd640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f920806b4d0 0x7f920806d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9210038000 tx=0x7f9210032000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.301+0000 7f9203fff640 1 -- 172.21.15.59:0/2275229146 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f921c13e8a0 con 0x7f921c070e10 2024-03-06T07:22:03.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.521+0000 7f92221d7640 1 -- 172.21.15.59:0/2275229146 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f921c15af40 con 0x7f921c070e10 2024-03-06T07:22:03.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.521+0000 7f9203fff640 1 -- 172.21.15.59:0/2275229146 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f920c058ca0 con 0x7f921c070e10 2024-03-06T07:22:03.523 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:03.524 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:22:03.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 -- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f920806b4d0 msgr2=0x7f920806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f920806b4d0 0x7f920806d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9210038000 tx=0x7f9210032000 comp rx=0 tx=0).stop 2024-03-06T07:22:03.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 -- 172.21.15.59:0/2275229146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f921c070e10 msgr2=0x7f921c13e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f921c070e10 0x7f921c13e360 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f920c00ebe0 tx=0x7f920c00c520 comp rx=0 tx=0).stop 2024-03-06T07:22:03.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 -- 172.21.15.59:0/2275229146 shutdown_connections 2024-03-06T07:22:03.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f920806b4d0 0x7f920806d990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f921c15b2d0 0x7f921c0cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f921c1444c0 0x7f921c0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 --2- 172.21.15.59:0/2275229146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f921c070e10 0x7f921c13e360 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 -- 172.21.15.59:0/2275229146 >> 172.21.15.59:0/2275229146 conn(0x7f921c0b1c80 msgr2=0x7f921c13eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 -- 172.21.15.59:0/2275229146 shutdown_connections 2024-03-06T07:22:03.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.523+0000 7f92221d7640 1 -- 172.21.15.59:0/2275229146 wait complete. 2024-03-06T07:22:03.536 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:22:03.536 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.635+0000 7f45b75ac640 1 Processor -- start 2024-03-06T07:22:03.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.635+0000 7f45b75ac640 1 -- start start 2024-03-06T07:22:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b75ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45b01621c0 0x7f45b00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b75ac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f45b01625a0 0x7f45b015dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b75ac640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b00ceac0 con 0x7f45b00d4920 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b75ac640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b00cf690 con 0x7f45b01625a0 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b75ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b00d2dc0 con 0x7f45b01621c0 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b5b49640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f45b01625a0 0x7f45b015dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b5b49640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f45b01625a0 0x7f45b015dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42678/0 (socket says 172.21.15.59:42678) 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b5b49640 1 -- 172.21.15.59:0/368409974 learned_addr learned my addr 172.21.15.59:0/368409974 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b5348640 1 --2- 172.21.15.59:0/368409974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45b01621c0 0x7f45b00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b5b49640 1 -- 172.21.15.59:0/368409974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45b01621c0 msgr2=0x7f45b00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b5b49640 1 --2- 172.21.15.59:0/368409974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45b01621c0 0x7f45b00cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b5b49640 1 -- 172.21.15.59:0/368409974 >> v1:172.21.15.164:6789/0 conn(0x7f45b00d4920 legacy=0x7f45b00d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.636+0000 7f45b5b49640 1 -- 172.21.15.59:0/368409974 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b00ced60 con 0x7f45b01625a0 2024-03-06T07:22:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.637+0000 7f45b5b49640 1 --2- 172.21.15.59:0/368409974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45b01625a0 0x7f45b015dd40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009e50 tx=0x7f45a802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856f6c27168f8cb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.637+0000 7f45a7fff640 1 -- 172.21.15.59:0/368409974 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a802cb00 con 0x7f45b01625a0 2024-03-06T07:22:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.637+0000 7f45a7fff640 1 -- 172.21.15.59:0/368409974 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a802cca0 con 0x7f45b01625a0 2024-03-06T07:22:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.637+0000 7f45a7fff640 1 -- 172.21.15.59:0/368409974 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8034610 con 0x7f45b01625a0 2024-03-06T07:22:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.638+0000 7f45b75ac640 1 -- 172.21.15.59:0/368409974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45b01625a0 msgr2=0x7f45b015dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.638+0000 7f45b75ac640 1 --2- 172.21.15.59:0/368409974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45b01625a0 0x7f45b015dd40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009e50 tx=0x7f45a802bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.638+0000 7f45b75ac640 1 -- 172.21.15.59:0/368409974 shutdown_connections 2024-03-06T07:22:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.638+0000 7f45b75ac640 1 --2- 172.21.15.59:0/368409974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45b01625a0 0x7f45b015dd40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.638+0000 7f45b75ac640 1 --2- 172.21.15.59:0/368409974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45b01621c0 0x7f45b00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.638+0000 7f45b75ac640 1 -- 172.21.15.59:0/368409974 >> 172.21.15.59:0/368409974 conn(0x7f45b00d3d10 msgr2=0x7f45b00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.638+0000 7f45b75ac640 1 -- 172.21.15.59:0/368409974 shutdown_connections 2024-03-06T07:22:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.638+0000 7f45b75ac640 1 -- 172.21.15.59:0/368409974 wait complete. 2024-03-06T07:22:03.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.639+0000 7f45b75ac640 1 Processor -- start 2024-03-06T07:22:03.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.639+0000 7f45b75ac640 1 -- start start 2024-03-06T07:22:03.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b75ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45b00d4920 0x7f45b00b9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b75ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45b01621c0 0x7f45b01f7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b75ac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45b01625a0 0x7f45b01f7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b75ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b015e280 con 0x7f45b01621c0 2024-03-06T07:22:03.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b75ac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b00cf320 con 0x7f45b01625a0 2024-03-06T07:22:03.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b75ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b015e400 con 0x7f45b00d4920 2024-03-06T07:22:03.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b4b47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45b01621c0 0x7f45b01f7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b4b47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45b01621c0 0x7f45b01f7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42686/0 (socket says 172.21.15.59:42686) 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b5348640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45b00d4920 0x7f45b00b9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b4b47640 1 -- 172.21.15.59:0/2847793539 learned_addr learned my addr 172.21.15.59:0/2847793539 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b5b49640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45b01625a0 0x7f45b01f7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b4b47640 1 -- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45b00d4920 msgr2=0x7f45b00b9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b4b47640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45b00d4920 0x7f45b00b9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b4b47640 1 -- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45b01625a0 msgr2=0x7f45b01f7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b4b47640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45b01625a0 0x7f45b01f7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b4b47640 1 -- 172.21.15.59:0/2847793539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b00a1ab0 con 0x7f45b01621c0 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b4b47640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45b01621c0 0x7f45b01f7300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f45a000c430 tx=0x7f45a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45a5ffb640 1 -- 172.21.15.59:0/2847793539 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a00052d0 con 0x7f45b01621c0 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45a5ffb640 1 -- 172.21.15.59:0/2847793539 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a0005490 con 0x7f45b01621c0 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45b75ac640 1 -- 172.21.15.59:0/2847793539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b00b6bc0 con 0x7f45b01621c0 2024-03-06T07:22:03.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.640+0000 7f45a5ffb640 1 -- 172.21.15.59:0/2847793539 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0007260 con 0x7f45b01621c0 2024-03-06T07:22:03.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.641+0000 7f45b75ac640 1 -- 172.21.15.59:0/2847793539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b00b7460 con 0x7f45b01621c0 2024-03-06T07:22:03.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.642+0000 7f45b75ac640 1 -- 172.21.15.59:0/2847793539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b004f330 con 0x7f45b01621c0 2024-03-06T07:22:03.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.642+0000 7f45a5ffb640 1 -- 172.21.15.59:0/2847793539 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45a0007400 con 0x7f45b01621c0 2024-03-06T07:22:03.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.642+0000 7f45a5ffb640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f459406b4d0 0x7f459406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.642+0000 7f45a5ffb640 1 -- 172.21.15.59:0/2847793539 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45a00895b0 con 0x7f45b01621c0 2024-03-06T07:22:03.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.642+0000 7f45b5348640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f459406b4d0 0x7f459406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.643+0000 7f45b5348640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f459406b4d0 0x7f459406d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009b40 tx=0x7f45ac009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.646+0000 7f45a5ffb640 1 -- 172.21.15.59:0/2847793539 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45a0057e70 con 0x7f45b01621c0 2024-03-06T07:22:03.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.829+0000 7f45b75ac640 1 -- 172.21.15.59:0/2847793539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45b0073380 con 0x7f45b01621c0 2024-03-06T07:22:03.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.830+0000 7f45a5ffb640 1 -- 172.21.15.59:0/2847793539 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f45a005bf10 con 0x7f45b01621c0 2024-03-06T07:22:03.832 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:03.832 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:03.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 -- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f459406b4d0 msgr2=0x7f459406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f459406b4d0 0x7f459406d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009b40 tx=0x7f45ac009190 comp rx=0 tx=0).stop 2024-03-06T07:22:03.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 -- 172.21.15.59:0/2847793539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45b01621c0 msgr2=0x7f45b01f7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45b01621c0 0x7f45b01f7300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f45a000c430 tx=0x7f45a000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 -- 172.21.15.59:0/2847793539 shutdown_connections 2024-03-06T07:22:03.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f459406b4d0 0x7f459406d990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45b01625a0 0x7f45b01f7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45b01621c0 0x7f45b01f7300 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 --2- 172.21.15.59:0/2847793539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45b00d4920 0x7f45b00b9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 -- 172.21.15.59:0/2847793539 >> 172.21.15.59:0/2847793539 conn(0x7f45b00d3d10 msgr2=0x7f45b015e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 -- 172.21.15.59:0/2847793539 shutdown_connections 2024-03-06T07:22:03.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.832+0000 7f45b75ac640 1 -- 172.21.15.59:0/2847793539 wait complete. 2024-03-06T07:22:03.845 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:03.845 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:03.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.947+0000 7f4aab981640 1 Processor -- start 2024-03-06T07:22:03.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.947+0000 7f4aab981640 1 -- start start 2024-03-06T07:22:03.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aab981640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4aa41614d0 0x7f4aa40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aab981640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4aa40d24b0 0x7f4aa40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aab981640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa41618b0 con 0x7f4aa40d1910 2024-03-06T07:22:03.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aab981640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa40cf400 con 0x7f4aa41614d0 2024-03-06T07:22:03.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aab981640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa40cf6a0 con 0x7f4aa40d24b0 2024-03-06T07:22:03.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aa971d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4aa41614d0 0x7f4aa40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aa971d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4aa41614d0 0x7f4aa40d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42690/0 (socket says 172.21.15.59:42690) 2024-03-06T07:22:03.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aa971d640 1 -- 172.21.15.59:0/2226790721 learned_addr learned my addr 172.21.15.59:0/2226790721 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:03.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aa9f1e640 1 --2- 172.21.15.59:0/2226790721 >> v2:172.21.15.164:3300/0 conn(0x7f4aa40d24b0 0x7f4aa40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aa971d640 1 -- 172.21.15.59:0/2226790721 >> v2:172.21.15.164:3300/0 conn(0x7f4aa40d24b0 msgr2=0x7f4aa40b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:03.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aa971d640 1 --2- 172.21.15.59:0/2226790721 >> v2:172.21.15.164:3300/0 conn(0x7f4aa40d24b0 0x7f4aa40b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aa971d640 1 -- 172.21.15.59:0/2226790721 >> v1:172.21.15.164:6789/0 conn(0x7f4aa40d1910 legacy=0x7f4aa40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.948+0000 7f4aa971d640 1 -- 172.21.15.59:0/2226790721 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa40cf980 con 0x7f4aa41614d0 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.949+0000 7f4aa971d640 1 --2- 172.21.15.59:0/2226790721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa41614d0 0x7f4aa40d4860 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40b2a70 tx=0x7f4a9c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56daacbd019c0552 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.949+0000 7f4aa3fff640 1 -- 172.21.15.59:0/2226790721 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c034070 con 0x7f4aa41614d0 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.949+0000 7f4aa3fff640 1 -- 172.21.15.59:0/2226790721 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9c030860 con 0x7f4aa41614d0 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.949+0000 7f4aa3fff640 1 -- 172.21.15.59:0/2226790721 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c030c30 con 0x7f4aa41614d0 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.950+0000 7f4aab981640 1 -- 172.21.15.59:0/2226790721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa41614d0 msgr2=0x7f4aa40d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.950+0000 7f4aab981640 1 --2- 172.21.15.59:0/2226790721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa41614d0 0x7f4aa40d4860 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40b2a70 tx=0x7f4a9c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.950+0000 7f4aab981640 1 -- 172.21.15.59:0/2226790721 shutdown_connections 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.950+0000 7f4aab981640 1 --2- 172.21.15.59:0/2226790721 >> v2:172.21.15.164:3300/0 conn(0x7f4aa40d24b0 0x7f4aa40b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.950+0000 7f4aab981640 1 --2- 172.21.15.59:0/2226790721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa41614d0 0x7f4aa40d4860 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.950+0000 7f4aab981640 1 -- 172.21.15.59:0/2226790721 >> 172.21.15.59:0/2226790721 conn(0x7f4aa40d3d40 msgr2=0x7f4aa40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.950+0000 7f4aab981640 1 -- 172.21.15.59:0/2226790721 shutdown_connections 2024-03-06T07:22:03.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.950+0000 7f4aab981640 1 -- 172.21.15.59:0/2226790721 wait complete. 2024-03-06T07:22:03.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.951+0000 7f4aab981640 1 Processor -- start 2024-03-06T07:22:03.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.951+0000 7f4aab981640 1 -- start start 2024-03-06T07:22:03.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aab981640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40d1910 0x7f4aa41edaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aab981640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa40d24b0 0x7f4aa40b7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aab981640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa41614d0 0x7f4aa40b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aab981640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40b1370 con 0x7f4aa40d1910 2024-03-06T07:22:03.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aab981640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40b0910 con 0x7f4aa41614d0 2024-03-06T07:22:03.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aab981640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40b14f0 con 0x7f4aa40d24b0 2024-03-06T07:22:03.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa971d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40d1910 0x7f4aa41edaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa971d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40d1910 0x7f4aa41edaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42692/0 (socket says 172.21.15.59:42692) 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa971d640 1 -- 172.21.15.59:0/1483706555 learned_addr learned my addr 172.21.15.59:0/1483706555 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa9f1e640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa41614d0 0x7f4aa40b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa8f1c640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa40d24b0 0x7f4aa40b7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa971d640 1 -- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa40d24b0 msgr2=0x7f4aa40b7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa971d640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa40d24b0 0x7f4aa40b7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa971d640 1 -- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa41614d0 msgr2=0x7f4aa40b7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa971d640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa41614d0 0x7f4aa40b7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa971d640 1 -- 172.21.15.59:0/1483706555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa40b6c40 con 0x7f4aa40d1910 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa8f1c640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa40d24b0 0x7f4aa40b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.952+0000 7f4aa971d640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40d1910 0x7f4aa41edaa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c000c00 tx=0x7f4a9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.953+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1483706555 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c034040 con 0x7f4aa40d1910 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.953+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1483706555 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9c039d90 con 0x7f4aa40d1910 2024-03-06T07:22:03.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.953+0000 7f4aab981640 1 -- 172.21.15.59:0/1483706555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa40b83c0 con 0x7f4aa40d1910 2024-03-06T07:22:03.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.953+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1483706555 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c043610 con 0x7f4aa40d1910 2024-03-06T07:22:03.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.953+0000 7f4aab981640 1 -- 172.21.15.59:0/1483706555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa40cb340 con 0x7f4aa40d1910 2024-03-06T07:22:03.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.954+0000 7f4aab981640 1 -- 172.21.15.59:0/1483706555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa4068ff0 con 0x7f4aa40d1910 2024-03-06T07:22:03.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.954+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1483706555 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a9c037070 con 0x7f4aa40d1910 2024-03-06T07:22:03.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.954+0000 7f4aa1ffb640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a9006b4d0 0x7f4a9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:03.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.954+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1483706555 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a9c0ac190 con 0x7f4aa40d1910 2024-03-06T07:22:03.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.955+0000 7f4aa8f1c640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a9006b4d0 0x7f4a9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:03.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.955+0000 7f4aa8f1c640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a9006b4d0 0x7f4a9006d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40b5d70 tx=0x7f4a98009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:03.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:03.959+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1483706555 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a9c03e070 con 0x7f4aa40d1910 2024-03-06T07:22:04.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.137+0000 7f4aab981640 1 -- 172.21.15.59:0/1483706555 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4aa40d0d10 con 0x7f4aa40d1910 2024-03-06T07:22:04.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.138+0000 7f4aa1ffb640 1 -- 172.21.15.59:0/1483706555 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a9c07a9c0 con 0x7f4aa40d1910 2024-03-06T07:22:04.140 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:04.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 -- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a9006b4d0 msgr2=0x7f4a9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a9006b4d0 0x7f4a9006d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40b5d70 tx=0x7f4a98009040 comp rx=0 tx=0).stop 2024-03-06T07:22:04.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 -- 172.21.15.59:0/1483706555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40d1910 msgr2=0x7f4aa41edaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40d1910 0x7f4aa41edaa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c000c00 tx=0x7f4a9c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 -- 172.21.15.59:0/1483706555 shutdown_connections 2024-03-06T07:22:04.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a9006b4d0 0x7f4a9006d990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa41614d0 0x7f4aa40b7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa40d24b0 0x7f4aa40b7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 --2- 172.21.15.59:0/1483706555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40d1910 0x7f4aa41edaa0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 -- 172.21.15.59:0/1483706555 >> 172.21.15.59:0/1483706555 conn(0x7f4aa40d3d40 msgr2=0x7f4aa40cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 -- 172.21.15.59:0/1483706555 shutdown_connections 2024-03-06T07:22:04.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.140+0000 7f4aab981640 1 -- 172.21.15.59:0/1483706555 wait complete. 2024-03-06T07:22:04.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:04.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.254+0000 7fe936e55640 1 Processor -- start 2024-03-06T07:22:04.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.254+0000 7fe936e55640 1 -- start start 2024-03-06T07:22:04.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe936e55640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe9300b41b0 0x7fe9300cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe936e55640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe9300b15c0 0x7fe9300b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe936e55640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9300d0240 con 0x7fe93015bdd0 2024-03-06T07:22:04.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe936e55640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9300cf4f0 con 0x7fe9300b41b0 2024-03-06T07:22:04.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe936e55640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9300d0850 con 0x7fe9300b15c0 2024-03-06T07:22:04.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe92ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe9300b41b0 0x7fe9300cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe92ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe9300b41b0 0x7fe9300cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42696/0 (socket says 172.21.15.59:42696) 2024-03-06T07:22:04.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe92ffff640 1 -- 172.21.15.59:0/1073658162 learned_addr learned my addr 172.21.15.59:0/1073658162 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:04.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe9353f2640 1 --2- 172.21.15.59:0/1073658162 >> v2:172.21.15.164:3300/0 conn(0x7fe9300b15c0 0x7fe9300b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe92ffff640 1 -- 172.21.15.59:0/1073658162 >> v2:172.21.15.164:3300/0 conn(0x7fe9300b15c0 msgr2=0x7fe9300b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe92ffff640 1 --2- 172.21.15.59:0/1073658162 >> v2:172.21.15.164:3300/0 conn(0x7fe9300b15c0 0x7fe9300b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe92ffff640 1 -- 172.21.15.59:0/1073658162 >> v1:172.21.15.164:6789/0 conn(0x7fe93015bdd0 legacy=0x7fe9300ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.255+0000 7fe92ffff640 1 -- 172.21.15.59:0/1073658162 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9300cef50 con 0x7fe9300b41b0 2024-03-06T07:22:04.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.256+0000 7fe92ffff640 1 --2- 172.21.15.59:0/1073658162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9300b41b0 0x7fe9300cea10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe930072b70 tx=0x7fe92402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89cd953808cb2315 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.256+0000 7fe92f7fe640 1 -- 172.21.15.59:0/1073658162 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe924034070 con 0x7fe9300b41b0 2024-03-06T07:22:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.256+0000 7fe92f7fe640 1 -- 172.21.15.59:0/1073658162 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe924030860 con 0x7fe9300b41b0 2024-03-06T07:22:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.256+0000 7fe92f7fe640 1 -- 172.21.15.59:0/1073658162 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe924030c30 con 0x7fe9300b41b0 2024-03-06T07:22:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.257+0000 7fe936e55640 1 -- 172.21.15.59:0/1073658162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9300b41b0 msgr2=0x7fe9300cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.257+0000 7fe936e55640 1 --2- 172.21.15.59:0/1073658162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9300b41b0 0x7fe9300cea10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe930072b70 tx=0x7fe92402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.257+0000 7fe936e55640 1 -- 172.21.15.59:0/1073658162 shutdown_connections 2024-03-06T07:22:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.257+0000 7fe936e55640 1 --2- 172.21.15.59:0/1073658162 >> v2:172.21.15.164:3300/0 conn(0x7fe9300b15c0 0x7fe9300b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.257+0000 7fe936e55640 1 --2- 172.21.15.59:0/1073658162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9300b41b0 0x7fe9300cea10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.257+0000 7fe936e55640 1 -- 172.21.15.59:0/1073658162 >> 172.21.15.59:0/1073658162 conn(0x7fe9300b3690 msgr2=0x7fe9300b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.257+0000 7fe936e55640 1 -- 172.21.15.59:0/1073658162 shutdown_connections 2024-03-06T07:22:04.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.257+0000 7fe936e55640 1 -- 172.21.15.59:0/1073658162 wait complete. 2024-03-06T07:22:04.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.258+0000 7fe936e55640 1 Processor -- start 2024-03-06T07:22:04.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.258+0000 7fe936e55640 1 -- start start 2024-03-06T07:22:04.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe936e55640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9300b15c0 0x7fe930151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe936e55640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9300b41b0 0x7fe9300b6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe936e55640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe93015bdd0 0x7fe9300b6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe936e55640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93015c1b0 con 0x7fe9300b41b0 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe936e55640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9300d04e0 con 0x7fe9300b15c0 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe936e55640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9300b10c0 con 0x7fe93015bdd0 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe92ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9300b41b0 0x7fe9300b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe92ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9300b41b0 0x7fe9300b6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42698/0 (socket says 172.21.15.59:42698) 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe92ffff640 1 -- 172.21.15.59:0/4014256736 learned_addr learned my addr 172.21.15.59:0/4014256736 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe934bf1640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9300b15c0 0x7fe930151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe9353f2640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe93015bdd0 0x7fe9300b6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe92ffff640 1 -- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe93015bdd0 msgr2=0x7fe9300b6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe92ffff640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe93015bdd0 0x7fe9300b6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe92ffff640 1 -- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9300b15c0 msgr2=0x7fe930151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe92ffff640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9300b15c0 0x7fe930151050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe92ffff640 1 -- 172.21.15.59:0/4014256736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9300b5790 con 0x7fe9300b41b0 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.259+0000 7fe92ffff640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9300b41b0 0x7fe9300b6200 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe92402b9e0 tx=0x7fe924004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.260+0000 7fe92d7fa640 1 -- 172.21.15.59:0/4014256736 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe924048070 con 0x7fe9300b41b0 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.260+0000 7fe92d7fa640 1 -- 172.21.15.59:0/4014256736 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe924030660 con 0x7fe9300b41b0 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.260+0000 7fe92d7fa640 1 -- 172.21.15.59:0/4014256736 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe924034040 con 0x7fe9300b41b0 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.260+0000 7fe936e55640 1 -- 172.21.15.59:0/4014256736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9300c54c0 con 0x7fe9300b41b0 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.260+0000 7fe936e55640 1 -- 172.21.15.59:0/4014256736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9300c5a80 con 0x7fe9300b41b0 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.261+0000 7fe936e55640 1 -- 172.21.15.59:0/4014256736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93009fcf0 con 0x7fe9300b41b0 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.261+0000 7fe92d7fa640 1 -- 172.21.15.59:0/4014256736 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe92404b070 con 0x7fe9300b41b0 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.261+0000 7fe92d7fa640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91c06b4d0 0x7fe91c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.261+0000 7fe92d7fa640 1 -- 172.21.15.59:0/4014256736 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9240ac3d0 con 0x7fe9300b41b0 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.262+0000 7fe934bf1640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91c06b4d0 0x7fe91c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.262+0000 7fe934bf1640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91c06b4d0 0x7fe91c06d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe9200016e0 tx=0x7fe920009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.266+0000 7fe92d7fa640 1 -- 172.21.15.59:0/4014256736 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe92407ad20 con 0x7fe9300b41b0 2024-03-06T07:22:04.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.445+0000 7fe936e55640 1 -- 172.21.15.59:0/4014256736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe930068ff0 con 0x7fe9300b41b0 2024-03-06T07:22:04.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.447+0000 7fe92d7fa640 1 -- 172.21.15.59:0/4014256736 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe92407edc0 con 0x7fe9300b41b0 2024-03-06T07:22:04.448 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:04.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 -- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91c06b4d0 msgr2=0x7fe91c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91c06b4d0 0x7fe91c06d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe9200016e0 tx=0x7fe920009040 comp rx=0 tx=0).stop 2024-03-06T07:22:04.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 -- 172.21.15.59:0/4014256736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9300b41b0 msgr2=0x7fe9300b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9300b41b0 0x7fe9300b6200 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe92402b9e0 tx=0x7fe924004730 comp rx=0 tx=0).stop 2024-03-06T07:22:04.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 -- 172.21.15.59:0/4014256736 shutdown_connections 2024-03-06T07:22:04.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91c06b4d0 0x7fe91c06d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe93015bdd0 0x7fe9300b6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9300b41b0 0x7fe9300b6200 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 --2- 172.21.15.59:0/4014256736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9300b15c0 0x7fe930151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 -- 172.21.15.59:0/4014256736 >> 172.21.15.59:0/4014256736 conn(0x7fe9300b3690 msgr2=0x7fe93006fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 -- 172.21.15.59:0/4014256736 shutdown_connections 2024-03-06T07:22:04.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.449+0000 7fe936e55640 1 -- 172.21.15.59:0/4014256736 wait complete. 2024-03-06T07:22:04.461 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:04.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.563+0000 7f2bf074b640 1 Processor -- start 2024-03-06T07:22:04.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.564+0000 7f2bf074b640 1 -- start start 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.564+0000 7f2bf074b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2be80d4860 0x7f2be80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.564+0000 7f2bf074b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b0e30 0x7f2be80b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.564+0000 7f2bf074b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be81618b0 con 0x7f2be81614d0 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.564+0000 7f2bf074b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be80d4cf0 con 0x7f2be80d4860 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.564+0000 7f2bf074b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80cf400 con 0x7f2be80b0e30 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.565+0000 7f2bedce6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2be80d4860 0x7f2be80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.565+0000 7f2bedce6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2be80d4860 0x7f2be80d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42714/0 (socket says 172.21.15.59:42714) 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.565+0000 7f2bee4e7640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2be81614d0 0x7f2be80d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:41168/0 (socket says 172.21.15.59:41168) 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.565+0000 7f2bedce6640 1 -- 172.21.15.59:0/3413495499 learned_addr learned my addr 172.21.15.59:0/3413495499 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.565+0000 7f2beece8640 1 --2- 172.21.15.59:0/3413495499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b0e30 0x7f2be80b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.565+0000 7f2bedce6640 1 -- 172.21.15.59:0/3413495499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b0e30 msgr2=0x7f2be80b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.565+0000 7f2bedce6640 1 --2- 172.21.15.59:0/3413495499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b0e30 0x7f2be80b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.565+0000 7f2bedce6640 1 -- 172.21.15.59:0/3413495499 >> v1:172.21.15.164:6789/0 conn(0x7f2be81614d0 legacy=0x7f2be80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.565+0000 7f2bedce6640 1 -- 172.21.15.59:0/3413495499 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80cf6a0 con 0x7f2be80d4860 2024-03-06T07:22:04.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.565+0000 7f2bed4e5640 1 -- 172.21.15.59:0/3413495499 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224515726 0 0) 0x7f2be81618b0 con 0x7f2be81614d0 2024-03-06T07:22:04.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.565+0000 7f2beece8640 1 --2- 172.21.15.59:0/3413495499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b0e30 0x7f2be80b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:04.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bedce6640 1 --2- 172.21.15.59:0/3413495499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be80d4860 0x7f2be80d2020 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009870 tx=0x7f2bd802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94fdc1526b497db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bed4e5640 1 -- 172.21.15.59:0/3413495499 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8034070 con 0x7f2be80d4860 2024-03-06T07:22:04.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bed4e5640 1 -- 172.21.15.59:0/3413495499 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd8030840 con 0x7f2be80d4860 2024-03-06T07:22:04.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bed4e5640 1 -- 172.21.15.59:0/3413495499 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8030c10 con 0x7f2be80d4860 2024-03-06T07:22:04.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bf074b640 1 -- 172.21.15.59:0/3413495499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be80d4860 msgr2=0x7f2be80d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bf074b640 1 --2- 172.21.15.59:0/3413495499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be80d4860 0x7f2be80d2020 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009870 tx=0x7f2bd802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:04.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bf074b640 1 -- 172.21.15.59:0/3413495499 shutdown_connections 2024-03-06T07:22:04.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bf074b640 1 --2- 172.21.15.59:0/3413495499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b0e30 0x7f2be80b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bf074b640 1 --2- 172.21.15.59:0/3413495499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be80d4860 0x7f2be80d2020 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bf074b640 1 -- 172.21.15.59:0/3413495499 >> 172.21.15.59:0/3413495499 conn(0x7f2be80d3d40 msgr2=0x7f2be80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bf074b640 1 -- 172.21.15.59:0/3413495499 shutdown_connections 2024-03-06T07:22:04.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.566+0000 7f2bf074b640 1 -- 172.21.15.59:0/3413495499 wait complete. 2024-03-06T07:22:04.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.567+0000 7f2bf074b640 1 Processor -- start 2024-03-06T07:22:04.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2bf074b640 1 -- start start 2024-03-06T07:22:04.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2bf074b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2be80b0e30 0x7f2be80cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2bf074b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80d4860 0x7f2be80c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2bf074b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be81614d0 0x7f2be80c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2bf074b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80b0910 con 0x7f2be81614d0 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2bf074b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80cfc60 con 0x7f2be80b0e30 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2bf074b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80d2560 con 0x7f2be80d4860 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2beece8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be81614d0 0x7f2be80c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2beece8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be81614d0 0x7f2be80c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42722/0 (socket says 172.21.15.59:42722) 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2beece8640 1 -- 172.21.15.59:0/2840696378 learned_addr learned my addr 172.21.15.59:0/2840696378 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2bee4e7640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2be80b0e30 0x7f2be80cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.568+0000 7f2bedce6640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80d4860 0x7f2be80c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.569+0000 7f2beece8640 1 -- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80d4860 msgr2=0x7f2be80c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.569+0000 7f2beece8640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80d4860 0x7f2be80c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.569+0000 7f2beece8640 1 -- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2be80b0e30 msgr2=0x7f2be80cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.569+0000 7f2beece8640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2be80b0e30 0x7f2be80cb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.569+0000 7f2beece8640 1 -- 172.21.15.59:0/2840696378 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80c9690 con 0x7f2be81614d0 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.569+0000 7f2beece8640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be81614d0 0x7f2be80c8ff0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2be000efc0 tx=0x7f2be0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.569+0000 7f2bdeffd640 1 -- 172.21.15.59:0/2840696378 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0018030 con 0x7f2be81614d0 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.569+0000 7f2bdeffd640 1 -- 172.21.15.59:0/2840696378 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be000cde0 con 0x7f2be81614d0 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.569+0000 7f2bf074b640 1 -- 172.21.15.59:0/2840696378 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be80c9970 con 0x7f2be81614d0 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.569+0000 7f2bdeffd640 1 -- 172.21.15.59:0/2840696378 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0003bb0 con 0x7f2be81614d0 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.569+0000 7f2bf074b640 1 -- 172.21.15.59:0/2840696378 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be80c9f30 con 0x7f2be81614d0 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.570+0000 7f2bf074b640 1 -- 172.21.15.59:0/2840696378 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be806b9b0 con 0x7f2be81614d0 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.570+0000 7f2bdeffd640 1 -- 172.21.15.59:0/2840696378 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2be001b070 con 0x7f2be81614d0 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.571+0000 7f2bdeffd640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bcc06b4d0 0x7f2bcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.571+0000 7f2bdeffd640 1 -- 172.21.15.59:0/2840696378 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2be008aa70 con 0x7f2be81614d0 2024-03-06T07:22:04.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.571+0000 7f2bee4e7640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bcc06b4d0 0x7f2bcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.571+0000 7f2bee4e7640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bcc06b4d0 0x7f2bcc06d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2be4007fd0 tx=0x7f2be4007de0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.575+0000 7f2bdeffd640 1 -- 172.21.15.59:0/2840696378 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2be806b9b0 con 0x7f2be81614d0 2024-03-06T07:22:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.756+0000 7f2bf074b640 1 -- 172.21.15.59:0/2840696378 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2be80ca630 con 0x7f2be81614d0 2024-03-06T07:22:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.756+0000 7f2bdeffd640 1 -- 172.21.15.59:0/2840696378 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2be0010040 con 0x7f2be81614d0 2024-03-06T07:22:04.759 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:04.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 -- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bcc06b4d0 msgr2=0x7f2bcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bcc06b4d0 0x7f2bcc06d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2be4007fd0 tx=0x7f2be4007de0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 -- 172.21.15.59:0/2840696378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be81614d0 msgr2=0x7f2be80c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be81614d0 0x7f2be80c8ff0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2be000efc0 tx=0x7f2be0009f90 comp rx=0 tx=0).stop 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 -- 172.21.15.59:0/2840696378 shutdown_connections 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bcc06b4d0 0x7f2bcc06d990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be81614d0 0x7f2be80c8ff0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80d4860 0x7f2be80c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 --2- 172.21.15.59:0/2840696378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2be80b0e30 0x7f2be80cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 -- 172.21.15.59:0/2840696378 >> 172.21.15.59:0/2840696378 conn(0x7f2be80d3d40 msgr2=0x7f2be806bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 -- 172.21.15.59:0/2840696378 shutdown_connections 2024-03-06T07:22:04.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.758+0000 7f2bf074b640 1 -- 172.21.15.59:0/2840696378 wait complete. 2024-03-06T07:22:04.771 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.870+0000 7f63cd6f4640 1 Processor -- start 2024-03-06T07:22:04.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.870+0000 7f63cd6f4640 1 -- start start 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63cd6f4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63c806c6d0 0x7f63c815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63cd6f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c8066d90 0x7f63c80ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63cd6f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c80cd450 con 0x7f63c80b4f70 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63cd6f4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c815e9f0 con 0x7f63c806c6d0 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63cd6f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80cda10 con 0x7f63c8066d90 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63c6ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f63c80b4f70 0x7f63c80b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42226/0 (socket says 172.21.15.59:42226) 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63c6ffd640 1 -- 172.21.15.59:0/3869763706 learned_addr learned my addr 172.21.15.59:0/3869763706 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63bffff640 1 --2- 172.21.15.59:0/3869763706 >> v2:172.21.15.164:3300/0 conn(0x7f63c806c6d0 0x7f63c815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63c77fe640 1 --2- 172.21.15.59:0/3869763706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c8066d90 0x7f63c80ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63c67fc640 1 -- 172.21.15.59:0/3869763706 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317488868 0 0) 0x7f63c80cd450 con 0x7f63c80b4f70 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63c67fc640 1 -- 172.21.15.59:0/3869763706 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b4003520 con 0x7f63c80b4f70 2024-03-06T07:22:04.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63c77fe640 1 -- 172.21.15.59:0/3869763706 >> v2:172.21.15.164:3300/0 conn(0x7f63c806c6d0 msgr2=0x7f63c815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63c77fe640 1 --2- 172.21.15.59:0/3869763706 >> v2:172.21.15.164:3300/0 conn(0x7f63c806c6d0 0x7f63c815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63c77fe640 1 -- 172.21.15.59:0/3869763706 >> v1:172.21.15.59:6789/0 conn(0x7f63c80b4f70 legacy=0x7f63c80b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63c77fe640 1 -- 172.21.15.59:0/3869763706 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c80cd730 con 0x7f63c8066d90 2024-03-06T07:22:04.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.871+0000 7f63bffff640 1 --2- 172.21.15.59:0/3869763706 >> v2:172.21.15.164:3300/0 conn(0x7f63c806c6d0 0x7f63c815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:04.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.872+0000 7f63c77fe640 1 --2- 172.21.15.59:0/3869763706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c8066d90 0x7f63c80ccb50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009d90 tx=0x7f63c002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7260c0561265586d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.872+0000 7f63c67fc640 1 -- 172.21.15.59:0/3869763706 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0039070 con 0x7f63c8066d90 2024-03-06T07:22:04.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.872+0000 7f63c67fc640 1 -- 172.21.15.59:0/3869763706 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c002cc50 con 0x7f63c8066d90 2024-03-06T07:22:04.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.872+0000 7f63c67fc640 1 -- 172.21.15.59:0/3869763706 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0034440 con 0x7f63c8066d90 2024-03-06T07:22:04.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.873+0000 7f63cd6f4640 1 -- 172.21.15.59:0/3869763706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c8066d90 msgr2=0x7f63c80ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.873+0000 7f63cd6f4640 1 --2- 172.21.15.59:0/3869763706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c8066d90 0x7f63c80ccb50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009d90 tx=0x7f63c002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.873+0000 7f63cd6f4640 1 -- 172.21.15.59:0/3869763706 shutdown_connections 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.873+0000 7f63cd6f4640 1 --2- 172.21.15.59:0/3869763706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c8066d90 0x7f63c80ccb50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.873+0000 7f63cd6f4640 1 --2- 172.21.15.59:0/3869763706 >> v2:172.21.15.164:3300/0 conn(0x7f63c806c6d0 0x7f63c815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.873+0000 7f63cd6f4640 1 -- 172.21.15.59:0/3869763706 >> 172.21.15.59:0/3869763706 conn(0x7f63c80724d0 msgr2=0x7f63c80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.873+0000 7f63cd6f4640 1 -- 172.21.15.59:0/3869763706 shutdown_connections 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.873+0000 7f63cd6f4640 1 -- 172.21.15.59:0/3869763706 wait complete. 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.874+0000 7f63cd6f4640 1 Processor -- start 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.874+0000 7f63cd6f4640 1 -- start start 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.874+0000 7f63cd6f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c8066d90 0x7f63c80adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.874+0000 7f63cd6f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c80b4f70 0x7f63c815fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.874+0000 7f63cd6f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c8160150 0x7f63c8159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.874+0000 7f63cd6f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80cd090 con 0x7f63c8160150 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.874+0000 7f63cd6f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c815f3c0 con 0x7f63c8066d90 2024-03-06T07:22:04.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.874+0000 7f63cd6f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80cd210 con 0x7f63c80b4f70 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63c77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c8160150 0x7f63c8159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63c77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c8160150 0x7f63c8159240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42730/0 (socket says 172.21.15.59:42730) 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63c77fe640 1 -- 172.21.15.59:0/4012260852 learned_addr learned my addr 172.21.15.59:0/4012260852 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63c6ffd640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c8066d90 0x7f63c80adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63bffff640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c80b4f70 0x7f63c815fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63c77fe640 1 -- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c80b4f70 msgr2=0x7f63c815fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63c77fe640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c80b4f70 0x7f63c815fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63c77fe640 1 -- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c8066d90 msgr2=0x7f63c80adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63c77fe640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c8066d90 0x7f63c80adea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63c77fe640 1 -- 172.21.15.59:0/4012260852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c81606d0 con 0x7f63c8160150 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63c77fe640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c8160150 0x7f63c8159240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009d90 tx=0x7f63c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63bf7fe640 1 -- 172.21.15.59:0/4012260852 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0039050 con 0x7f63c8160150 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63bf7fe640 1 -- 172.21.15.59:0/4012260852 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c0045070 con 0x7f63c8160150 2024-03-06T07:22:04.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63cd6f4640 1 -- 172.21.15.59:0/4012260852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c81609b0 con 0x7f63c8160150 2024-03-06T07:22:04.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63bf7fe640 1 -- 172.21.15.59:0/4012260852 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0009450 con 0x7f63c8160150 2024-03-06T07:22:04.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.875+0000 7f63cd6f4640 1 -- 172.21.15.59:0/4012260852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c8161250 con 0x7f63c8160150 2024-03-06T07:22:04.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.876+0000 7f63cd6f4640 1 -- 172.21.15.59:0/4012260852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c8160b80 con 0x7f63c8160150 2024-03-06T07:22:04.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.877+0000 7f63bf7fe640 1 -- 172.21.15.59:0/4012260852 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63c003e610 con 0x7f63c8160150 2024-03-06T07:22:04.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.877+0000 7f63bf7fe640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f63a806b4d0 0x7f63a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.877+0000 7f63bf7fe640 1 -- 172.21.15.59:0/4012260852 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63c00acb00 con 0x7f63c8160150 2024-03-06T07:22:04.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.877+0000 7f63c6ffd640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f63a806b4d0 0x7f63a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.878+0000 7f63c6ffd640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f63a806b4d0 0x7f63a806d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f63b8005880 tx=0x7f63b8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:04.881+0000 7f63bf7fe640 1 -- 172.21.15.59:0/4012260852 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63c007b490 con 0x7f63c8160150 2024-03-06T07:22:05.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.099+0000 7f63cd6f4640 1 -- 172.21.15.59:0/4012260852 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63c80b1860 con 0x7f63c8160150 2024-03-06T07:22:05.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.100+0000 7f63bf7fe640 1 -- 172.21.15.59:0/4012260852 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f63c007f530 con 0x7f63c8160150 2024-03-06T07:22:05.102 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:05.103 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1759048959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1759048959},{"type":"v1","addr":"172.21.15.164:6836","nonce":1759048959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/550023536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":550023536},{"type":"v1","addr":"172.21.15.59:6833","nonce":550023536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2132697832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2132697832},{"type":"v1","addr":"172.21.15.59:6835","nonce":2132697832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:54.767210+0000","modified":"2024-03-06T07:21:56.776415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/1957940318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1957940318},{"type":"v1","addr":"172.21.15.164:6837","nonce":1957940318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-06T07:22:05.104 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 6 2024-03-06T07:22:05.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 -- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f63a806b4d0 msgr2=0x7f63a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f63a806b4d0 0x7f63a806d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f63b8005880 tx=0x7f63b8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:05.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 -- 172.21.15.59:0/4012260852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c8160150 msgr2=0x7f63c8159240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c8160150 0x7f63c8159240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009d90 tx=0x7f63c0004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:05.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 -- 172.21.15.59:0/4012260852 shutdown_connections 2024-03-06T07:22:05.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f63a806b4d0 0x7f63a806d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63c8160150 0x7f63c8159240 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c80b4f70 0x7f63c815fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 --2- 172.21.15.59:0/4012260852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c8066d90 0x7f63c80adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 -- 172.21.15.59:0/4012260852 >> 172.21.15.59:0/4012260852 conn(0x7f63c80724d0 msgr2=0x7f63c806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 -- 172.21.15.59:0/4012260852 shutdown_connections 2024-03-06T07:22:05.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.102+0000 7f63cd6f4640 1 -- 172.21.15.59:0/4012260852 wait complete. 2024-03-06T07:22:05.115 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:05.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.214+0000 7f422122a640 1 Processor -- start 2024-03-06T07:22:05.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.214+0000 7f422122a640 1 -- start start 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f422122a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f421c0d00c0 0x7f421c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f422122a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f421c0cf850 0x7f421c0cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f422122a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421c0d09e0 con 0x7f421c0ce1b0 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f422122a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f421c06ca90 con 0x7f421c0cf850 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f422122a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f421c06df30 con 0x7f421c0d00c0 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f421a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f421c0cf850 0x7f421c0cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f421a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f421c0cf850 0x7f421c0cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42732/0 (socket says 172.21.15.59:42732) 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f421a59c640 1 -- 172.21.15.59:0/889504042 learned_addr learned my addr 172.21.15.59:0/889504042 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f421ad9d640 1 --2- 172.21.15.59:0/889504042 >> v2:172.21.15.164:3300/0 conn(0x7f421c0d00c0 0x7f421c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f421a59c640 1 -- 172.21.15.59:0/889504042 >> v2:172.21.15.164:3300/0 conn(0x7f421c0d00c0 msgr2=0x7f421c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f421a59c640 1 --2- 172.21.15.59:0/889504042 >> v2:172.21.15.164:3300/0 conn(0x7f421c0d00c0 0x7f421c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f421a59c640 1 -- 172.21.15.59:0/889504042 >> v1:172.21.15.164:6789/0 conn(0x7f421c0ce1b0 legacy=0x7f421c0ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f421a59c640 1 -- 172.21.15.59:0/889504042 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c15e380 con 0x7f421c0cf850 2024-03-06T07:22:05.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.215+0000 7f421ad9d640 1 --2- 172.21.15.59:0/889504042 >> v2:172.21.15.164:3300/0 conn(0x7f421c0d00c0 0x7f421c0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:05.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.216+0000 7f421a59c640 1 --2- 172.21.15.59:0/889504042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421c0cf850 0x7f421c0cd3c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f421c0d1a40 tx=0x7f420402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7881bce939674d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.216+0000 7f4219d9b640 1 -- 172.21.15.59:0/889504042 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204034070 con 0x7f421c0cf850 2024-03-06T07:22:05.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.216+0000 7f4219d9b640 1 -- 172.21.15.59:0/889504042 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4204030860 con 0x7f421c0cf850 2024-03-06T07:22:05.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.216+0000 7f4219d9b640 1 -- 172.21.15.59:0/889504042 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204030c30 con 0x7f421c0cf850 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.217+0000 7f422122a640 1 -- 172.21.15.59:0/889504042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421c0cf850 msgr2=0x7f421c0cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.217+0000 7f422122a640 1 --2- 172.21.15.59:0/889504042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421c0cf850 0x7f421c0cd3c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f421c0d1a40 tx=0x7f420402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.217+0000 7f422122a640 1 -- 172.21.15.59:0/889504042 shutdown_connections 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.217+0000 7f422122a640 1 --2- 172.21.15.59:0/889504042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421c0cf850 0x7f421c0cd3c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.217+0000 7f422122a640 1 --2- 172.21.15.59:0/889504042 >> v2:172.21.15.164:3300/0 conn(0x7f421c0d00c0 0x7f421c0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.217+0000 7f422122a640 1 -- 172.21.15.59:0/889504042 >> 172.21.15.59:0/889504042 conn(0x7f421c0b35a0 msgr2=0x7f421c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.217+0000 7f422122a640 1 -- 172.21.15.59:0/889504042 shutdown_connections 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.217+0000 7f422122a640 1 -- 172.21.15.59:0/889504042 wait complete. 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.218+0000 7f422122a640 1 Processor -- start 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.218+0000 7f422122a640 1 -- start start 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f422122a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421c0ce1b0 0x7f421c14eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f422122a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f421c0cf850 0x7f421c0c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f422122a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f421c0d00c0 0x7f421c1f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f422122a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c06c440 con 0x7f421c0ce1b0 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f422122a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c06c160 con 0x7f421c0d00c0 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f422122a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421c06c720 con 0x7f421c0cf850 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f421ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421c0ce1b0 0x7f421c14eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f421ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421c0ce1b0 0x7f421c14eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42744/0 (socket says 172.21.15.59:42744) 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f421ad9d640 1 -- 172.21.15.59:0/3587843583 learned_addr learned my addr 172.21.15.59:0/3587843583 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f421a59c640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f421c0cf850 0x7f421c0c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f421b59e640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f421c0d00c0 0x7f421c1f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f421ad9d640 1 -- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f421c0cf850 msgr2=0x7f421c0c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f421ad9d640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f421c0cf850 0x7f421c0c6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f421ad9d640 1 -- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f421c0d00c0 msgr2=0x7f421c1f4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f421ad9d640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f421c0d00c0 0x7f421c1f4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.219+0000 7f421ad9d640 1 -- 172.21.15.59:0/3587843583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c0c59a0 con 0x7f421c0ce1b0 2024-03-06T07:22:05.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.220+0000 7f421ad9d640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421c0ce1b0 0x7f421c14eee0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f421400e960 tx=0x7f421400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.220+0000 7f42137fe640 1 -- 172.21.15.59:0/3587843583 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4214017070 con 0x7f421c0ce1b0 2024-03-06T07:22:05.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.220+0000 7f42137fe640 1 -- 172.21.15.59:0/3587843583 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421400c9a0 con 0x7f421c0ce1b0 2024-03-06T07:22:05.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.220+0000 7f422122a640 1 -- 172.21.15.59:0/3587843583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421c0c5c80 con 0x7f421c0ce1b0 2024-03-06T07:22:05.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.220+0000 7f42137fe640 1 -- 172.21.15.59:0/3587843583 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421400cd70 con 0x7f421c0ce1b0 2024-03-06T07:22:05.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.220+0000 7f422122a640 1 -- 172.21.15.59:0/3587843583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421c0c6240 con 0x7f421c0ce1b0 2024-03-06T07:22:05.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.221+0000 7f422122a640 1 -- 172.21.15.59:0/3587843583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421c04f330 con 0x7f421c0ce1b0 2024-03-06T07:22:05.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.221+0000 7f42137fe640 1 -- 172.21.15.59:0/3587843583 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f421401a070 con 0x7f421c0ce1b0 2024-03-06T07:22:05.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.222+0000 7f42137fe640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41fc06b4d0 0x7f41fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.222+0000 7f42137fe640 1 -- 172.21.15.59:0/3587843583 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f421408a380 con 0x7f421c0ce1b0 2024-03-06T07:22:05.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.222+0000 7f421a59c640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41fc06b4d0 0x7f41fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.222+0000 7f421a59c640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41fc06b4d0 0x7f41fc06d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4204036000 tx=0x7f4204008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.226+0000 7f42137fe640 1 -- 172.21.15.59:0/3587843583 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4214058bb0 con 0x7f421c0ce1b0 2024-03-06T07:22:05.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.407+0000 7f422122a640 1 -- 172.21.15.59:0/3587843583 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f421c14f420 con 0x7f421c0ce1b0 2024-03-06T07:22:05.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.408+0000 7f42137fe640 1 -- 172.21.15.59:0/3587843583 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f421405cc50 con 0x7f421c0ce1b0 2024-03-06T07:22:05.410 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:05.411 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:05.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.410+0000 7f422122a640 1 -- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41fc06b4d0 msgr2=0x7f41fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.410+0000 7f422122a640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41fc06b4d0 0x7f41fc06d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4204036000 tx=0x7f4204008030 comp rx=0 tx=0).stop 2024-03-06T07:22:05.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.410+0000 7f422122a640 1 -- 172.21.15.59:0/3587843583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421c0ce1b0 msgr2=0x7f421c14eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.410+0000 7f422122a640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421c0ce1b0 0x7f421c14eee0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f421400e960 tx=0x7f421400ee20 comp rx=0 tx=0).stop 2024-03-06T07:22:05.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.411+0000 7f422122a640 1 -- 172.21.15.59:0/3587843583 shutdown_connections 2024-03-06T07:22:05.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.411+0000 7f422122a640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41fc06b4d0 0x7f41fc06d990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.411+0000 7f422122a640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f421c0d00c0 0x7f421c1f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.411+0000 7f422122a640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f421c0cf850 0x7f421c0c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.411+0000 7f422122a640 1 --2- 172.21.15.59:0/3587843583 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f421c0ce1b0 0x7f421c14eee0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.411+0000 7f422122a640 1 -- 172.21.15.59:0/3587843583 >> 172.21.15.59:0/3587843583 conn(0x7f421c0b35a0 msgr2=0x7f421c0700b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.411+0000 7f422122a640 1 -- 172.21.15.59:0/3587843583 shutdown_connections 2024-03-06T07:22:05.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.411+0000 7f422122a640 1 -- 172.21.15.59:0/3587843583 wait complete. 2024-03-06T07:22:05.423 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:05.424 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:05.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.527+0000 7fa92049a640 1 Processor -- start 2024-03-06T07:22:05.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.528+0000 7fa92049a640 1 -- start start 2024-03-06T07:22:05.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.528+0000 7fa92049a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa9181614d0 0x7fa9180d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.528+0000 7fa92049a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9180d1910 0x7fa9180d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.528+0000 7fa92049a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa9180d22b0 0x7fa9180d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.528+0000 7fa92049a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9180d2740 con 0x7fa9181614d0 2024-03-06T07:22:05.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.528+0000 7fa92049a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9181618b0 con 0x7fa9180d22b0 2024-03-06T07:22:05.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.528+0000 7fa92049a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9180cf400 con 0x7fa9180d1910 2024-03-06T07:22:05.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.529+0000 7fa91e236640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa9181614d0 0x7fa9180d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.529+0000 7fa91e236640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa9181614d0 0x7fa9180d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42758/0 (socket says 172.21.15.59:42758) 2024-03-06T07:22:05.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.529+0000 7fa91e236640 1 -- 172.21.15.59:0/465463578 learned_addr learned my addr 172.21.15.59:0/465463578 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:05.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.529+0000 7fa91da35640 1 --2- 172.21.15.59:0/465463578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9180d1910 0x7fa9180d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.529+0000 7fa91ea37640 1 --2- 172.21.15.59:0/465463578 >> v2:172.21.15.164:3300/0 conn(0x7fa9180d22b0 0x7fa9180d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.529+0000 7fa91e236640 1 -- 172.21.15.59:0/465463578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9180d1910 msgr2=0x7fa9180d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:05.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.529+0000 7fa91e236640 1 --2- 172.21.15.59:0/465463578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9180d1910 0x7fa9180d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.529+0000 7fa91e236640 1 -- 172.21.15.59:0/465463578 >> v2:172.21.15.164:3300/0 conn(0x7fa9180d22b0 msgr2=0x7fa9180d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.529+0000 7fa91e236640 1 --2- 172.21.15.59:0/465463578 >> v2:172.21.15.164:3300/0 conn(0x7fa9180d22b0 0x7fa9180d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.529+0000 7fa91e236640 1 -- 172.21.15.59:0/465463578 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9180cf6a0 con 0x7fa9181614d0 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa91e236640 1 --2- 172.21.15.59:0/465463578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9181614d0 0x7fa9180d4860 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa9180b57d0 tx=0x7fa91402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5b4a1dbf544894 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa91d234640 1 -- 172.21.15.59:0/465463578 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914035020 con 0x7fa9181614d0 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa91d234640 1 -- 172.21.15.59:0/465463578 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa914031870 con 0x7fa9181614d0 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa91d234640 1 -- 172.21.15.59:0/465463578 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914031c40 con 0x7fa9181614d0 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa92049a640 1 -- 172.21.15.59:0/465463578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9181614d0 msgr2=0x7fa9180d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa92049a640 1 --2- 172.21.15.59:0/465463578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9181614d0 0x7fa9180d4860 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa9180b57d0 tx=0x7fa91402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa92049a640 1 -- 172.21.15.59:0/465463578 shutdown_connections 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa92049a640 1 --2- 172.21.15.59:0/465463578 >> v2:172.21.15.164:3300/0 conn(0x7fa9180d22b0 0x7fa9180d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa92049a640 1 --2- 172.21.15.59:0/465463578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9180d1910 0x7fa9180d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa92049a640 1 --2- 172.21.15.59:0/465463578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9181614d0 0x7fa9180d4860 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa92049a640 1 -- 172.21.15.59:0/465463578 >> 172.21.15.59:0/465463578 conn(0x7fa9180d3d40 msgr2=0x7fa9180d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa92049a640 1 -- 172.21.15.59:0/465463578 shutdown_connections 2024-03-06T07:22:05.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.530+0000 7fa92049a640 1 -- 172.21.15.59:0/465463578 wait complete. 2024-03-06T07:22:05.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.531+0000 7fa92049a640 1 Processor -- start 2024-03-06T07:22:05.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.531+0000 7fa92049a640 1 -- start start 2024-03-06T07:22:05.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.532+0000 7fa92049a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9180d1910 0x7fa91806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.532+0000 7fa92049a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9180d22b0 0x7fa9180b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.532+0000 7fa92049a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9181614d0 0x7fa9180b90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.532+0000 7fa92049a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9180b0910 con 0x7fa9181614d0 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.532+0000 7fa92049a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9180cfc60 con 0x7fa9180d22b0 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.532+0000 7fa92049a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9180d0be0 con 0x7fa9180d1910 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.532+0000 7fa91ea37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9181614d0 0x7fa9180b90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.532+0000 7fa91ea37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9181614d0 0x7fa9180b90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42760/0 (socket says 172.21.15.59:42760) 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.532+0000 7fa91ea37640 1 -- 172.21.15.59:0/2247806287 learned_addr learned my addr 172.21.15.59:0/2247806287 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.532+0000 7fa91da35640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9180d22b0 0x7fa9180b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.532+0000 7fa91e236640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9180d1910 0x7fa91806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa91ea37640 1 -- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9180d1910 msgr2=0x7fa91806cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa91ea37640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9180d1910 0x7fa91806cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa91ea37640 1 -- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9180d22b0 msgr2=0x7fa9180b8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa91ea37640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9180d22b0 0x7fa9180b8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa91ea37640 1 -- 172.21.15.59:0/2247806287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9180b9620 con 0x7fa9181614d0 2024-03-06T07:22:05.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa91da35640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9180d22b0 0x7fa9180b8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa91ea37640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9181614d0 0x7fa9180b90e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa91000efc0 tx=0x7fa91000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa90effd640 1 -- 172.21.15.59:0/2247806287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910018020 con 0x7fa9181614d0 2024-03-06T07:22:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa90effd640 1 -- 172.21.15.59:0/2247806287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa910014480 con 0x7fa9181614d0 2024-03-06T07:22:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa92049a640 1 -- 172.21.15.59:0/2247806287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9180c8fe0 con 0x7fa9181614d0 2024-03-06T07:22:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa90effd640 1 -- 172.21.15.59:0/2247806287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910014870 con 0x7fa9181614d0 2024-03-06T07:22:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.533+0000 7fa92049a640 1 -- 172.21.15.59:0/2247806287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9180c95a0 con 0x7fa9181614d0 2024-03-06T07:22:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.534+0000 7fa92049a640 1 -- 172.21.15.59:0/2247806287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9180cdfd0 con 0x7fa9181614d0 2024-03-06T07:22:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.534+0000 7fa90effd640 1 -- 172.21.15.59:0/2247806287 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa910005d10 con 0x7fa9181614d0 2024-03-06T07:22:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.535+0000 7fa90effd640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8fc06b4d0 0x7fa8fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.535+0000 7fa90effd640 1 -- 172.21.15.59:0/2247806287 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa91008adc0 con 0x7fa9181614d0 2024-03-06T07:22:05.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.535+0000 7fa91e236640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8fc06b4d0 0x7fa8fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.535+0000 7fa91e236640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8fc06b4d0 0x7fa8fc06d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa914036000 tx=0x7fa91403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.539+0000 7fa90effd640 1 -- 172.21.15.59:0/2247806287 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa910022030 con 0x7fa9181614d0 2024-03-06T07:22:05.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.720+0000 7fa92049a640 1 -- 172.21.15.59:0/2247806287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa918068ff0 con 0x7fa9181614d0 2024-03-06T07:22:05.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.721+0000 7fa90effd640 1 -- 172.21.15.59:0/2247806287 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa910010040 con 0x7fa9181614d0 2024-03-06T07:22:05.723 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:05.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 -- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8fc06b4d0 msgr2=0x7fa8fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8fc06b4d0 0x7fa8fc06d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa914036000 tx=0x7fa91403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:05.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 -- 172.21.15.59:0/2247806287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9181614d0 msgr2=0x7fa9180b90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9181614d0 0x7fa9180b90e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa91000efc0 tx=0x7fa91000c490 comp rx=0 tx=0).stop 2024-03-06T07:22:05.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 -- 172.21.15.59:0/2247806287 shutdown_connections 2024-03-06T07:22:05.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8fc06b4d0 0x7fa8fc06d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9181614d0 0x7fa9180b90e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9180d22b0 0x7fa9180b8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 --2- 172.21.15.59:0/2247806287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9180d1910 0x7fa91806cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 -- 172.21.15.59:0/2247806287 >> 172.21.15.59:0/2247806287 conn(0x7fa9180d3d40 msgr2=0x7fa9180cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 -- 172.21.15.59:0/2247806287 shutdown_connections 2024-03-06T07:22:05.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.723+0000 7fa92049a640 1 -- 172.21.15.59:0/2247806287 wait complete. 2024-03-06T07:22:05.735 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:05.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.838+0000 7f4c6d5a0640 1 Processor -- start 2024-03-06T07:22:05.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.838+0000 7f4c6d5a0640 1 -- start start 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c6d5a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4c680b4ef0 0x7f4c680d45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c6d5a0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c680cec90 con 0x7f4c680d0bb0 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c6d5a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c680cef70 con 0x7f4c680ce640 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c6d5a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c680cf8a0 con 0x7f4c680b4ef0 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c6cd9f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4c680d0bb0 0x7f4c6815d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42230/0 (socket says 172.21.15.59:42230) 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c6cd9f640 1 -- 172.21.15.59:0/2021523295 learned_addr learned my addr 172.21.15.59:0/2021523295 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c67fff640 1 --2- 172.21.15.59:0/2021523295 >> v2:172.21.15.164:3300/0 conn(0x7f4c680b4ef0 0x7f4c680d45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c66ffd640 1 -- 172.21.15.59:0/2021523295 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967330536 0 0) 0x7f4c680cec90 con 0x7f4c680d0bb0 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c66ffd640 1 -- 172.21.15.59:0/2021523295 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c54003520 con 0x7f4c680d0bb0 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c67fff640 1 -- 172.21.15.59:0/2021523295 >> v1:172.21.15.164:6789/0 conn(0x7f4c680ce640 legacy=0x7f4c680d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c67fff640 1 -- 172.21.15.59:0/2021523295 >> v1:172.21.15.59:6789/0 conn(0x7f4c680d0bb0 legacy=0x7f4c6815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c66ffd640 1 -- 172.21.15.59:0/2021523295 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56660265 0 0) 0x7f4c680cef70 con 0x7f4c680ce640 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.839+0000 7f4c67fff640 1 -- 172.21.15.59:0/2021523295 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c680cf250 con 0x7f4c680b4ef0 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.840+0000 7f4c67fff640 1 --2- 172.21.15.59:0/2021523295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c680b4ef0 0x7f4c680d45e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4c500097a0 tx=0x7f4c5002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1be13b1d505b8d47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.840+0000 7f4c66ffd640 1 -- 172.21.15.59:0/2021523295 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50034070 con 0x7f4c680b4ef0 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.840+0000 7f4c66ffd640 1 -- 172.21.15.59:0/2021523295 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c50030610 con 0x7f4c680b4ef0 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.840+0000 7f4c66ffd640 1 -- 172.21.15.59:0/2021523295 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c500309c0 con 0x7f4c680b4ef0 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.841+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/2021523295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c680b4ef0 msgr2=0x7f4c680d45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.841+0000 7f4c6d5a0640 1 --2- 172.21.15.59:0/2021523295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c680b4ef0 0x7f4c680d45e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4c500097a0 tx=0x7f4c5002b960 comp rx=0 tx=0).stop 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.841+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/2021523295 shutdown_connections 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.841+0000 7f4c6d5a0640 1 --2- 172.21.15.59:0/2021523295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c680b4ef0 0x7f4c680d45e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.841+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/2021523295 >> 172.21.15.59:0/2021523295 conn(0x7f4c680d3ac0 msgr2=0x7f4c680d3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.841+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/2021523295 shutdown_connections 2024-03-06T07:22:05.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.841+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/2021523295 wait complete. 2024-03-06T07:22:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.842+0000 7f4c6d5a0640 1 Processor -- start 2024-03-06T07:22:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.842+0000 7f4c6d5a0640 1 -- start start 2024-03-06T07:22:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c6d5a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c680b4ef0 0x7f4c6815e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c6d5a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c680ce640 0x7f4c6815eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c6d5a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c680d0bb0 0x7f4c6809f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c6d5a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680b52d0 con 0x7f4c680ce640 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c6d5a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680d1b60 con 0x7f4c680d0bb0 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c6d5a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680b7a40 con 0x7f4c680b4ef0 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c680ce640 0x7f4c6815eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c67fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c680b4ef0 0x7f4c6815e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c680ce640 0x7f4c6815eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42764/0 (socket says 172.21.15.59:42764) 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c677fe640 1 -- 172.21.15.59:0/283531326 learned_addr learned my addr 172.21.15.59:0/283531326 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c6cd9f640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c680d0bb0 0x7f4c6809f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c677fe640 1 -- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c680b4ef0 msgr2=0x7f4c6815e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c677fe640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c680b4ef0 0x7f4c6815e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c677fe640 1 -- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c680d0bb0 msgr2=0x7f4c6809f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c677fe640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c680d0bb0 0x7f4c6809f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c677fe640 1 -- 172.21.15.59:0/283531326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c680cac30 con 0x7f4c680ce640 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c67fff640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c680b4ef0 0x7f4c6815e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:05.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c6cd9f640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c680d0bb0 0x7f4c6809f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.843+0000 7f4c677fe640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c680ce640 0x7f4c6815eeb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4c5800c4b0 tx=0x7f4c5800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.844+0000 7f4c64ff9640 1 -- 172.21.15.59:0/283531326 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58017070 con 0x7f4c680ce640 2024-03-06T07:22:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.844+0000 7f4c64ff9640 1 -- 172.21.15.59:0/283531326 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c58005480 con 0x7f4c680ce640 2024-03-06T07:22:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.844+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/283531326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c680cd580 con 0x7f4c680ce640 2024-03-06T07:22:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.844+0000 7f4c64ff9640 1 -- 172.21.15.59:0/283531326 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58003830 con 0x7f4c680ce640 2024-03-06T07:22:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.844+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/283531326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c680caed0 con 0x7f4c680ce640 2024-03-06T07:22:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.845+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/283531326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c68068f10 con 0x7f4c680ce640 2024-03-06T07:22:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.845+0000 7f4c64ff9640 1 -- 172.21.15.59:0/283531326 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c5801a070 con 0x7f4c680ce640 2024-03-06T07:22:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.845+0000 7f4c64ff9640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4c4806b4d0 0x7f4c4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.845+0000 7f4c64ff9640 1 -- 172.21.15.59:0/283531326 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c58089500 con 0x7f4c680ce640 2024-03-06T07:22:05.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.846+0000 7f4c67fff640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4c4806b4d0 0x7f4c4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:05.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.846+0000 7f4c67fff640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4c4806b4d0 0x7f4c4806d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4c50036000 tx=0x7f4c5003b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:05.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:05.850+0000 7f4c64ff9640 1 -- 172.21.15.59:0/283531326 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c58057d30 con 0x7f4c680ce640 2024-03-06T07:22:06.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.029+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/283531326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4c680d1070 con 0x7f4c680ce640 2024-03-06T07:22:06.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.029+0000 7f4c64ff9640 1 -- 172.21.15.59:0/283531326 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4c5805bdd0 con 0x7f4c680ce640 2024-03-06T07:22:06.031 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:06.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4c4806b4d0 msgr2=0x7f4c4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4c4806b4d0 0x7f4c4806d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4c50036000 tx=0x7f4c5003b040 comp rx=0 tx=0).stop 2024-03-06T07:22:06.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/283531326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c680ce640 msgr2=0x7f4c6815eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c680ce640 0x7f4c6815eeb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4c5800c4b0 tx=0x7f4c5800c970 comp rx=0 tx=0).stop 2024-03-06T07:22:06.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/283531326 shutdown_connections 2024-03-06T07:22:06.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4c4806b4d0 0x7f4c4806d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c680d0bb0 0x7f4c6809f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c680ce640 0x7f4c6815eeb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 --2- 172.21.15.59:0/283531326 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c680b4ef0 0x7f4c6815e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/283531326 >> 172.21.15.59:0/283531326 conn(0x7f4c680d3ac0 msgr2=0x7f4c680b9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/283531326 shutdown_connections 2024-03-06T07:22:06.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.031+0000 7f4c6d5a0640 1 -- 172.21.15.59:0/283531326 wait complete. 2024-03-06T07:22:06.044 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:06.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.146+0000 7f63815a0640 1 Processor -- start 2024-03-06T07:22:06.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.146+0000 7f63815a0640 1 -- start start 2024-03-06T07:22:06.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f63815a0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f637c0b4ef0 0x7f637c0d45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f63815a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f637c0ce640 0x7f637c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f63815a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c0cec90 con 0x7f637c0d09e0 2024-03-06T07:22:06.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f63815a0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637c0cf8a0 con 0x7f637c0b4ef0 2024-03-06T07:22:06.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f63815a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637c0cef70 con 0x7f637c0ce640 2024-03-06T07:22:06.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f637bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f637c0b4ef0 0x7f637c0d45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f637bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f637c0b4ef0 0x7f637c0d45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42776/0 (socket says 172.21.15.59:42776) 2024-03-06T07:22:06.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f6380d9f640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f637c0d09e0 0x7f637c15d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:41200/0 (socket says 172.21.15.59:41200) 2024-03-06T07:22:06.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f637bfff640 1 -- 172.21.15.59:0/3444696019 learned_addr learned my addr 172.21.15.59:0/3444696019 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:06.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f637b7fe640 1 --2- 172.21.15.59:0/3444696019 >> v2:172.21.15.164:3300/0 conn(0x7f637c0ce640 0x7f637c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f637affd640 1 -- 172.21.15.59:0/3444696019 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523009363 0 0) 0x7f637c0cec90 con 0x7f637c0d09e0 2024-03-06T07:22:06.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f637bfff640 1 -- 172.21.15.59:0/3444696019 >> v2:172.21.15.164:3300/0 conn(0x7f637c0ce640 msgr2=0x7f637c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f637bfff640 1 --2- 172.21.15.59:0/3444696019 >> v2:172.21.15.164:3300/0 conn(0x7f637c0ce640 0x7f637c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f637bfff640 1 -- 172.21.15.59:0/3444696019 >> v1:172.21.15.164:6789/0 conn(0x7f637c0d09e0 legacy=0x7f637c15d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.147+0000 7f637bfff640 1 -- 172.21.15.59:0/3444696019 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c0cf250 con 0x7f637c0b4ef0 2024-03-06T07:22:06.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.148+0000 7f637bfff640 1 --2- 172.21.15.59:0/3444696019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f637c0b4ef0 0x7f637c0d45e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f637c0d15c0 tx=0x7f636402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce8bfc5556bdbac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.148+0000 7f637affd640 1 -- 172.21.15.59:0/3444696019 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364034070 con 0x7f637c0b4ef0 2024-03-06T07:22:06.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.148+0000 7f637affd640 1 -- 172.21.15.59:0/3444696019 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6364030860 con 0x7f637c0b4ef0 2024-03-06T07:22:06.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.148+0000 7f637affd640 1 -- 172.21.15.59:0/3444696019 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364030c30 con 0x7f637c0b4ef0 2024-03-06T07:22:06.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.149+0000 7f63815a0640 1 -- 172.21.15.59:0/3444696019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f637c0b4ef0 msgr2=0x7f637c0d45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.149+0000 7f63815a0640 1 --2- 172.21.15.59:0/3444696019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f637c0b4ef0 0x7f637c0d45e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f637c0d15c0 tx=0x7f636402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.149+0000 7f63815a0640 1 -- 172.21.15.59:0/3444696019 shutdown_connections 2024-03-06T07:22:06.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.149+0000 7f63815a0640 1 --2- 172.21.15.59:0/3444696019 >> v2:172.21.15.164:3300/0 conn(0x7f637c0ce640 0x7f637c0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.149+0000 7f63815a0640 1 --2- 172.21.15.59:0/3444696019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f637c0b4ef0 0x7f637c0d45e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.149+0000 7f63815a0640 1 -- 172.21.15.59:0/3444696019 >> 172.21.15.59:0/3444696019 conn(0x7f637c0d3ac0 msgr2=0x7f637c0d3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.149+0000 7f63815a0640 1 -- 172.21.15.59:0/3444696019 shutdown_connections 2024-03-06T07:22:06.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.149+0000 7f63815a0640 1 -- 172.21.15.59:0/3444696019 wait complete. 2024-03-06T07:22:06.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.150+0000 7f63815a0640 1 Processor -- start 2024-03-06T07:22:06.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.150+0000 7f63815a0640 1 -- start start 2024-03-06T07:22:06.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f63815a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f637c0b4ef0 0x7f637c0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f63815a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f637c0ce640 0x7f637c159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f63815a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f637c0d09e0 0x7f637c0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f63815a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c0b52d0 con 0x7f637c0d09e0 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f63815a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c0d1b60 con 0x7f637c0ce640 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f63815a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637c15de20 con 0x7f637c0b4ef0 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f6380d9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f637c0d09e0 0x7f637c0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f6380d9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f637c0d09e0 0x7f637c0ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42778/0 (socket says 172.21.15.59:42778) 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f6380d9f640 1 -- 172.21.15.59:0/1646714876 learned_addr learned my addr 172.21.15.59:0/1646714876 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f637bfff640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f637c0b4ef0 0x7f637c0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f637b7fe640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f637c0ce640 0x7f637c159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f6380d9f640 1 -- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f637c0b4ef0 msgr2=0x7f637c0cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f6380d9f640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f637c0b4ef0 0x7f637c0cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f6380d9f640 1 -- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f637c0ce640 msgr2=0x7f637c159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f6380d9f640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f637c0ce640 0x7f637c159020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f6380d9f640 1 -- 172.21.15.59:0/1646714876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c159560 con 0x7f637c0d09e0 2024-03-06T07:22:06.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f6380d9f640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f637c0d09e0 0x7f637c0ccf10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f637400c730 tx=0x7f637400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f6378ff9640 1 -- 172.21.15.59:0/1646714876 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6374018070 con 0x7f637c0d09e0 2024-03-06T07:22:06.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.151+0000 7f6378ff9640 1 -- 172.21.15.59:0/1646714876 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637400ce40 con 0x7f637c0d09e0 2024-03-06T07:22:06.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.152+0000 7f63815a0640 1 -- 172.21.15.59:0/1646714876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637c0cccd0 con 0x7f637c0d09e0 2024-03-06T07:22:06.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.152+0000 7f6378ff9640 1 -- 172.21.15.59:0/1646714876 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637400a860 con 0x7f637c0d09e0 2024-03-06T07:22:06.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.152+0000 7f63815a0640 1 -- 172.21.15.59:0/1646714876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637c1605a0 con 0x7f637c0d09e0 2024-03-06T07:22:06.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.153+0000 7f63815a0640 1 -- 172.21.15.59:0/1646714876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637c15dc40 con 0x7f637c0d09e0 2024-03-06T07:22:06.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.153+0000 7f6378ff9640 1 -- 172.21.15.59:0/1646714876 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f637401b070 con 0x7f637c0d09e0 2024-03-06T07:22:06.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.153+0000 7f6378ff9640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f635c06b4d0 0x7f635c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.153+0000 7f6378ff9640 1 -- 172.21.15.59:0/1646714876 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6374089750 con 0x7f637c0d09e0 2024-03-06T07:22:06.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.153+0000 7f637bfff640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f635c06b4d0 0x7f635c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.154+0000 7f637bfff640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f635c06b4d0 0x7f635c06d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6364036000 tx=0x7f6364008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.158+0000 7f6378ff9640 1 -- 172.21.15.59:0/1646714876 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f637c15dc40 con 0x7f637c0d09e0 2024-03-06T07:22:06.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.341+0000 7f63815a0640 1 -- 172.21.15.59:0/1646714876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f637c0d1e80 con 0x7f637c0d09e0 2024-03-06T07:22:06.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.341+0000 7f6378ff9640 1 -- 172.21.15.59:0/1646714876 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f637c159560 con 0x7f637c0d09e0 2024-03-06T07:22:06.343 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:06.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 -- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f635c06b4d0 msgr2=0x7f635c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f635c06b4d0 0x7f635c06d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6364036000 tx=0x7f6364008030 comp rx=0 tx=0).stop 2024-03-06T07:22:06.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 -- 172.21.15.59:0/1646714876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f637c0d09e0 msgr2=0x7f637c0ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f637c0d09e0 0x7f637c0ccf10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f637400c730 tx=0x7f637400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 -- 172.21.15.59:0/1646714876 shutdown_connections 2024-03-06T07:22:06.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f635c06b4d0 0x7f635c06d990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f637c0d09e0 0x7f637c0ccf10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f637c0ce640 0x7f637c159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 --2- 172.21.15.59:0/1646714876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f637c0b4ef0 0x7f637c0cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 -- 172.21.15.59:0/1646714876 >> 172.21.15.59:0/1646714876 conn(0x7f637c0d3ac0 msgr2=0x7f637c0b6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 -- 172.21.15.59:0/1646714876 shutdown_connections 2024-03-06T07:22:06.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.344+0000 7f63815a0640 1 -- 172.21.15.59:0/1646714876 wait complete. 2024-03-06T07:22:06.356 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.456+0000 7f5300d8a640 1 Processor -- start 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.456+0000 7f5300d8a640 1 -- start start 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.457+0000 7f5300d8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52fc0b4170 0x7f52fc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.457+0000 7f5300d8a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f52fc0d0a80 0x7f52fc0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.457+0000 7f5300d8a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52fc0d25d0 con 0x7f52fc0d4760 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.457+0000 7f5300d8a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52fc0d4bf0 con 0x7f52fc0d0a80 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.457+0000 7f5300d8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc0ce9e0 con 0x7f52fc0b4170 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.457+0000 7f52fad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f52fc0d0a80 0x7f52fc0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.457+0000 7f52f3fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f52fc0d4760 0x7f52fc0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42240/0 (socket says 172.21.15.59:42240) 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.457+0000 7f52f3fff640 1 -- 172.21.15.59:0/1468125156 learned_addr learned my addr 172.21.15.59:0/1468125156 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.457+0000 7f52fa59c640 1 --2- 172.21.15.59:0/1468125156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52fc0b4170 0x7f52fc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.458+0000 7f52fad9d640 1 -- 172.21.15.59:0/1468125156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52fc0b4170 msgr2=0x7f52fc0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.458+0000 7f52fad9d640 1 --2- 172.21.15.59:0/1468125156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52fc0b4170 0x7f52fc0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.458+0000 7f52fad9d640 1 -- 172.21.15.59:0/1468125156 >> v1:172.21.15.59:6789/0 conn(0x7f52fc0d4760 legacy=0x7f52fc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.458+0000 7f52fad9d640 1 -- 172.21.15.59:0/1468125156 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc0d2870 con 0x7f52fc0d0a80 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.458+0000 7f52f9d9b640 1 -- 172.21.15.59:0/1468125156 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603088288 0 0) 0x7f52fc0d25d0 con 0x7f52fc0d4760 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.458+0000 7f52fad9d640 1 --2- 172.21.15.59:0/1468125156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52fc0d0a80 0x7f52fc0cee70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009d90 tx=0x7f52f402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34779210da4fa508 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.458+0000 7f52f9d9b640 1 -- 172.21.15.59:0/1468125156 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4039070 con 0x7f52fc0d0a80 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.459+0000 7f52f9d9b640 1 -- 172.21.15.59:0/1468125156 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f402cc50 con 0x7f52fc0d0a80 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.459+0000 7f52f9d9b640 1 -- 172.21.15.59:0/1468125156 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4034440 con 0x7f52fc0d0a80 2024-03-06T07:22:06.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.459+0000 7f5300d8a640 1 -- 172.21.15.59:0/1468125156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52fc0d0a80 msgr2=0x7f52fc0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.459+0000 7f5300d8a640 1 --2- 172.21.15.59:0/1468125156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52fc0d0a80 0x7f52fc0cee70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009d90 tx=0x7f52f402bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.459+0000 7f5300d8a640 1 -- 172.21.15.59:0/1468125156 shutdown_connections 2024-03-06T07:22:06.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.459+0000 7f5300d8a640 1 --2- 172.21.15.59:0/1468125156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52fc0d0a80 0x7f52fc0cee70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.459+0000 7f5300d8a640 1 --2- 172.21.15.59:0/1468125156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52fc0b4170 0x7f52fc0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.459+0000 7f5300d8a640 1 -- 172.21.15.59:0/1468125156 >> 172.21.15.59:0/1468125156 conn(0x7f52fc0d3c40 msgr2=0x7f52fc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.459+0000 7f5300d8a640 1 -- 172.21.15.59:0/1468125156 shutdown_connections 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.459+0000 7f5300d8a640 1 -- 172.21.15.59:0/1468125156 wait complete. 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.460+0000 7f5300d8a640 1 Processor -- start 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.460+0000 7f5300d8a640 1 -- start start 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f5300d8a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52fc0b4170 0x7f52fc06c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f5300d8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52fc0d0a80 0x7f52fc0cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f5300d8a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52fc0cd580 0x7f52fc0b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f5300d8a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc0b4dd0 con 0x7f52fc0cd580 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f5300d8a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc0b4550 con 0x7f52fc0b4170 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f5300d8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc0b4f50 con 0x7f52fc0d0a80 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f52fad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52fc0cd580 0x7f52fc0b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f52fad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52fc0cd580 0x7f52fc0b7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42796/0 (socket says 172.21.15.59:42796) 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f52fad9d640 1 -- 172.21.15.59:0/4221867378 learned_addr learned my addr 172.21.15.59:0/4221867378 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f52f3fff640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52fc0d0a80 0x7f52fc0cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f52fa59c640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52fc0b4170 0x7f52fc06c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f52fad9d640 1 -- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52fc0d0a80 msgr2=0x7f52fc0cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f52fad9d640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52fc0d0a80 0x7f52fc0cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f52fad9d640 1 -- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52fc0b4170 msgr2=0x7f52fc06c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f52fad9d640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52fc0b4170 0x7f52fc06c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.461+0000 7f52fad9d640 1 -- 172.21.15.59:0/4221867378 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc0b6bb0 con 0x7f52fc0cd580 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.462+0000 7f52fad9d640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52fc0cd580 0x7f52fc0b7c50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009d90 tx=0x7f52f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.462+0000 7f52f2ffd640 1 -- 172.21.15.59:0/4221867378 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4039050 con 0x7f52fc0cd580 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.462+0000 7f52f2ffd640 1 -- 172.21.15.59:0/4221867378 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f4045070 con 0x7f52fc0cd580 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.462+0000 7f5300d8a640 1 -- 172.21.15.59:0/4221867378 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52fc0b6e90 con 0x7f52fc0cd580 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.462+0000 7f52f2ffd640 1 -- 172.21.15.59:0/4221867378 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4009450 con 0x7f52fc0cd580 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.462+0000 7f5300d8a640 1 -- 172.21.15.59:0/4221867378 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52fc0b7450 con 0x7f52fc0cd580 2024-03-06T07:22:06.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.463+0000 7f5300d8a640 1 -- 172.21.15.59:0/4221867378 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52fc04f330 con 0x7f52fc0cd580 2024-03-06T07:22:06.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.463+0000 7f52f2ffd640 1 -- 172.21.15.59:0/4221867378 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52f403e610 con 0x7f52fc0cd580 2024-03-06T07:22:06.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.464+0000 7f52f2ffd640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52dc06b500 0x7f52dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.464+0000 7f52f2ffd640 1 -- 172.21.15.59:0/4221867378 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52f40acab0 con 0x7f52fc0cd580 2024-03-06T07:22:06.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.464+0000 7f52fa59c640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52dc06b500 0x7f52dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.464+0000 7f52fa59c640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52dc06b500 0x7f52dc06d9c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009b40 tx=0x7f52ec009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.468+0000 7f52f2ffd640 1 -- 172.21.15.59:0/4221867378 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52f407b2d0 con 0x7f52fc0cd580 2024-03-06T07:22:06.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.687+0000 7f5300d8a640 1 -- 172.21.15.59:0/4221867378 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52fc0733b0 con 0x7f52fc0cd580 2024-03-06T07:22:06.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.687+0000 7f52f2ffd640 1 -- 172.21.15.59:0/4221867378 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f52f407f370 con 0x7f52fc0cd580 2024-03-06T07:22:06.689 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:06.690 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1759048959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1759048959},{"type":"v1","addr":"172.21.15.164:6836","nonce":1759048959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/550023536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":550023536},{"type":"v1","addr":"172.21.15.59:6833","nonce":550023536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2132697832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2132697832},{"type":"v1","addr":"172.21.15.59:6835","nonce":2132697832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:54.767210+0000","modified":"2024-03-06T07:21:56.776415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/1957940318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1957940318},{"type":"v1","addr":"172.21.15.164:6837","nonce":1957940318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-06T07:22:06.690 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 6 2024-03-06T07:22:06.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 -- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52dc06b500 msgr2=0x7f52dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52dc06b500 0x7f52dc06d9c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f52ec009b40 tx=0x7f52ec009190 comp rx=0 tx=0).stop 2024-03-06T07:22:06.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 -- 172.21.15.59:0/4221867378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52fc0cd580 msgr2=0x7f52fc0b7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52fc0cd580 0x7f52fc0b7c50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009d90 tx=0x7f52f4004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:06.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 -- 172.21.15.59:0/4221867378 shutdown_connections 2024-03-06T07:22:06.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52dc06b500 0x7f52dc06d9c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52fc0cd580 0x7f52fc0b7c50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52fc0d0a80 0x7f52fc0cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 --2- 172.21.15.59:0/4221867378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52fc0b4170 0x7f52fc06c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 -- 172.21.15.59:0/4221867378 >> 172.21.15.59:0/4221867378 conn(0x7f52fc0d3c40 msgr2=0x7f52fc06b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 -- 172.21.15.59:0/4221867378 shutdown_connections 2024-03-06T07:22:06.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.689+0000 7f5300d8a640 1 -- 172.21.15.59:0/4221867378 wait complete. 2024-03-06T07:22:06.702 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:06.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.801+0000 7f6083c47640 1 Processor -- start 2024-03-06T07:22:06.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.801+0000 7f6083c47640 1 -- start start 2024-03-06T07:22:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f6083c47640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f607c1625a0 0x7f607c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f6083c47640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c0ceac0 con 0x7f607c0d4920 2024-03-06T07:22:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f6083c47640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c0d2dc0 con 0x7f607c1621c0 2024-03-06T07:22:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f6083c47640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607c0cf690 con 0x7f607c1625a0 2024-03-06T07:22:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f60811e2640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f607c0d4920 0x7f607c15dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42254/0 (socket says 172.21.15.59:42254) 2024-03-06T07:22:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f60811e2640 1 -- 172.21.15.59:0/4146562579 learned_addr learned my addr 172.21.15.59:0/4146562579 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f60821e4640 1 --2- 172.21.15.59:0/4146562579 >> v2:172.21.15.164:3300/0 conn(0x7f607c1625a0 0x7f607c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f60809e1640 1 -- 172.21.15.59:0/4146562579 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945262484 0 0) 0x7f607c0ceac0 con 0x7f607c0d4920 2024-03-06T07:22:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f60809e1640 1 -- 172.21.15.59:0/4146562579 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6068003520 con 0x7f607c0d4920 2024-03-06T07:22:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f60821e4640 1 -- 172.21.15.59:0/4146562579 >> v1:172.21.15.164:6789/0 conn(0x7f607c1621c0 legacy=0x7f607c0d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f60809e1640 1 -- 172.21.15.59:0/4146562579 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979764355 0 0) 0x7f607c0d2dc0 con 0x7f607c1621c0 2024-03-06T07:22:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f60821e4640 1 -- 172.21.15.59:0/4146562579 >> v1:172.21.15.59:6789/0 conn(0x7f607c0d4920 legacy=0x7f607c15dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.802+0000 7f60821e4640 1 -- 172.21.15.59:0/4146562579 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c0ced60 con 0x7f607c1625a0 2024-03-06T07:22:06.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.803+0000 7f60821e4640 1 --2- 172.21.15.59:0/4146562579 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f607c1625a0 0x7f607c0cfb20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6074009d90 tx=0x7f607402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=205e22a906f3245a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.803+0000 7f60809e1640 1 -- 172.21.15.59:0/4146562579 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074039070 con 0x7f607c1625a0 2024-03-06T07:22:06.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.803+0000 7f60809e1640 1 -- 172.21.15.59:0/4146562579 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607402cc60 con 0x7f607c1625a0 2024-03-06T07:22:06.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.803+0000 7f60809e1640 1 -- 172.21.15.59:0/4146562579 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074034660 con 0x7f607c1625a0 2024-03-06T07:22:06.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.804+0000 7f6083c47640 1 -- 172.21.15.59:0/4146562579 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f607c1625a0 msgr2=0x7f607c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.804+0000 7f6083c47640 1 --2- 172.21.15.59:0/4146562579 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f607c1625a0 0x7f607c0cfb20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6074009d90 tx=0x7f607402beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.804+0000 7f6083c47640 1 -- 172.21.15.59:0/4146562579 shutdown_connections 2024-03-06T07:22:06.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.804+0000 7f6083c47640 1 --2- 172.21.15.59:0/4146562579 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f607c1625a0 0x7f607c0cfb20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.804+0000 7f6083c47640 1 -- 172.21.15.59:0/4146562579 >> 172.21.15.59:0/4146562579 conn(0x7f607c0d3d10 msgr2=0x7f607c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.804+0000 7f6083c47640 1 -- 172.21.15.59:0/4146562579 shutdown_connections 2024-03-06T07:22:06.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.804+0000 7f6083c47640 1 -- 172.21.15.59:0/4146562579 wait complete. 2024-03-06T07:22:06.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.805+0000 7f6083c47640 1 Processor -- start 2024-03-06T07:22:06.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.805+0000 7f6083c47640 1 -- start start 2024-03-06T07:22:06.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f6083c47640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f607c0d4920 0x7f607c156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f6083c47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f607c1621c0 0x7f607c1f75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f6083c47640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f607c1625a0 0x7f607c0d0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f6083c47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c15e450 con 0x7f607c1621c0 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f6083c47640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c0cf320 con 0x7f607c1625a0 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f6083c47640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607c15e5d0 con 0x7f607c0d4920 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60811e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f607c1621c0 0x7f607c1f75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60811e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f607c1621c0 0x7f607c1f75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42812/0 (socket says 172.21.15.59:42812) 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60811e2640 1 -- 172.21.15.59:0/3044010323 learned_addr learned my addr 172.21.15.59:0/3044010323 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60821e4640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f607c1625a0 0x7f607c0d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60819e3640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f607c0d4920 0x7f607c156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60811e2640 1 -- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f607c0d4920 msgr2=0x7f607c156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60811e2640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f607c0d4920 0x7f607c156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60811e2640 1 -- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f607c1625a0 msgr2=0x7f607c0d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60811e2640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f607c1625a0 0x7f607c0d0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60811e2640 1 -- 172.21.15.59:0/3044010323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c1f7b30 con 0x7f607c1621c0 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60811e2640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f607c1621c0 0x7f607c1f75f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f606c00c430 tx=0x7f606c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60727fc640 1 -- 172.21.15.59:0/3044010323 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c017070 con 0x7f607c1621c0 2024-03-06T07:22:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60727fc640 1 -- 172.21.15.59:0/3044010323 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606c005ce0 con 0x7f607c1621c0 2024-03-06T07:22:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f6083c47640 1 -- 172.21.15.59:0/3044010323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607c0cc2f0 con 0x7f607c1621c0 2024-03-06T07:22:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f60727fc640 1 -- 172.21.15.59:0/3044010323 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c0072f0 con 0x7f607c1621c0 2024-03-06T07:22:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.806+0000 7f6083c47640 1 -- 172.21.15.59:0/3044010323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607c0cc8b0 con 0x7f607c1621c0 2024-03-06T07:22:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.807+0000 7f6083c47640 1 -- 172.21.15.59:0/3044010323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607c04f330 con 0x7f607c1621c0 2024-03-06T07:22:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.808+0000 7f60727fc640 1 -- 172.21.15.59:0/3044010323 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f606c01a070 con 0x7f607c1621c0 2024-03-06T07:22:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.808+0000 7f60727fc640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.808+0000 7f60727fc640 1 -- 172.21.15.59:0/3044010323 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f606c089630 con 0x7f607c1621c0 2024-03-06T07:22:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.808+0000 7f60819e3640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.809+0000 7f60819e3640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606006b4d0 0x7f606006d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6078000f60 tx=0x7f6078009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.812+0000 7f60727fc640 1 -- 172.21.15.59:0/3044010323 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f607c1f7b30 con 0x7f607c1621c0 2024-03-06T07:22:06.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.994+0000 7f6083c47640 1 -- 172.21.15.59:0/3044010323 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f607c073380 con 0x7f607c1621c0 2024-03-06T07:22:06.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.995+0000 7f60727fc640 1 -- 172.21.15.59:0/3044010323 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f606c05ce20 con 0x7f607c1621c0 2024-03-06T07:22:06.997 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:06.997 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 -- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606006b4d0 msgr2=0x7f606006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606006b4d0 0x7f606006d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6078000f60 tx=0x7f6078009040 comp rx=0 tx=0).stop 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 -- 172.21.15.59:0/3044010323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f607c1621c0 msgr2=0x7f607c1f75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f607c1621c0 0x7f607c1f75f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f606c00c430 tx=0x7f606c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 -- 172.21.15.59:0/3044010323 shutdown_connections 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f607c1625a0 0x7f607c0d0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f607c1621c0 0x7f607c1f75f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 --2- 172.21.15.59:0/3044010323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f607c0d4920 0x7f607c156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 -- 172.21.15.59:0/3044010323 >> 172.21.15.59:0/3044010323 conn(0x7f607c0d3d10 msgr2=0x7f607c0b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 -- 172.21.15.59:0/3044010323 shutdown_connections 2024-03-06T07:22:07.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:06.997+0000 7f6083c47640 1 -- 172.21.15.59:0/3044010323 wait complete. 2024-03-06T07:22:07.010 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:07.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.108+0000 7f253af19640 1 Processor -- start 2024-03-06T07:22:07.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.108+0000 7f253af19640 1 -- start start 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.109+0000 7f253af19640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253415cc60 0x7f253406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.109+0000 7f253af19640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253415d0a0 con 0x7f25341597c0 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.109+0000 7f253af19640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25340cd440 con 0x7f253406b4f0 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.109+0000 7f253af19640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2534159ba0 con 0x7f253415cc60 2024-03-06T07:22:07.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.109+0000 7f2538cb5640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f25341597c0 0x7f253406f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42256/0 (socket says 172.21.15.59:42256) 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.109+0000 7f2538cb5640 1 -- 172.21.15.59:0/905738484 learned_addr learned my addr 172.21.15.59:0/905738484 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.109+0000 7f2533fff640 1 --2- 172.21.15.59:0/905738484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253415cc60 0x7f253406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.110+0000 7f25337fe640 1 -- 172.21.15.59:0/905738484 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052306210 0 0) 0x7f2524001110 con 0x7f25341597c0 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.110+0000 7f25337fe640 1 -- 172.21.15.59:0/905738484 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f251c003520 con 0x7f25341597c0 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.110+0000 7f25337fe640 1 -- 172.21.15.59:0/905738484 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310027846 0 0) 0x7f25340cd440 con 0x7f253406b4f0 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.110+0000 7f25337fe640 1 -- 172.21.15.59:0/905738484 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2524001110 con 0x7f253406b4f0 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.110+0000 7f2533fff640 1 -- 172.21.15.59:0/905738484 >> v1:172.21.15.164:6789/0 conn(0x7f253406b4f0 legacy=0x7f25340d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.110+0000 7f2533fff640 1 -- 172.21.15.59:0/905738484 >> v1:172.21.15.59:6789/0 conn(0x7f25341597c0 legacy=0x7f253406f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.110+0000 7f2533fff640 1 -- 172.21.15.59:0/905738484 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25340cd720 con 0x7f253415cc60 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.110+0000 7f2533fff640 1 --2- 172.21.15.59:0/905738484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253415cc60 0x7f253406afb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f25280097d0 tx=0x7f252802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=256a6f0ec7992f5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.111+0000 7f25337fe640 1 -- 172.21.15.59:0/905738484 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528035020 con 0x7f253415cc60 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.111+0000 7f25337fe640 1 -- 172.21.15.59:0/905738484 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2528031890 con 0x7f253415cc60 2024-03-06T07:22:07.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.111+0000 7f25337fe640 1 -- 172.21.15.59:0/905738484 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528031c40 con 0x7f253415cc60 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.111+0000 7f253af19640 1 -- 172.21.15.59:0/905738484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253415cc60 msgr2=0x7f253406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.111+0000 7f253af19640 1 --2- 172.21.15.59:0/905738484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253415cc60 0x7f253406afb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f25280097d0 tx=0x7f252802b930 comp rx=0 tx=0).stop 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.111+0000 7f253af19640 1 -- 172.21.15.59:0/905738484 shutdown_connections 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.111+0000 7f253af19640 1 --2- 172.21.15.59:0/905738484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253415cc60 0x7f253406afb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.111+0000 7f253af19640 1 -- 172.21.15.59:0/905738484 >> 172.21.15.59:0/905738484 conn(0x7f253406c0c0 msgr2=0x7f253406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.111+0000 7f253af19640 1 -- 172.21.15.59:0/905738484 shutdown_connections 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.111+0000 7f253af19640 1 -- 172.21.15.59:0/905738484 wait complete. 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.112+0000 7f253af19640 1 Processor -- start 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.112+0000 7f253af19640 1 -- start start 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.113+0000 7f253af19640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253406b4f0 0x7f2534142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.113+0000 7f253af19640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25341597c0 0x7f25340cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.113+0000 7f253af19640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f253415cc60 0x7f25340cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.113+0000 7f253af19640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253415c150 con 0x7f253415cc60 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.113+0000 7f253af19640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253415c7d0 con 0x7f253406b4f0 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.113+0000 7f253af19640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25340cd0c0 con 0x7f25341597c0 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.113+0000 7f25394b6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f253415cc60 0x7f25340cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.113+0000 7f25394b6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f253415cc60 0x7f25340cb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42822/0 (socket says 172.21.15.59:42822) 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.113+0000 7f25394b6640 1 -- 172.21.15.59:0/3255693245 learned_addr learned my addr 172.21.15.59:0/3255693245 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.113+0000 7f2538cb5640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253406b4f0 0x7f2534142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.113+0000 7f2533fff640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25341597c0 0x7f25340cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.114+0000 7f25394b6640 1 -- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25341597c0 msgr2=0x7f25340cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.114+0000 7f25394b6640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25341597c0 0x7f25340cac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.114+0000 7f25394b6640 1 -- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253406b4f0 msgr2=0x7f2534142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.114+0000 7f25394b6640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253406b4f0 0x7f2534142ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.114+0000 7f25394b6640 1 -- 172.21.15.59:0/3255693245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25340cb790 con 0x7f253415cc60 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.114+0000 7f25394b6640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f253415cc60 0x7f25340cb1a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f252c00c6b0 tx=0x7f252c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.114+0000 7f25317fa640 1 -- 172.21.15.59:0/3255693245 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c003b70 con 0x7f253415cc60 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.114+0000 7f25317fa640 1 -- 172.21.15.59:0/3255693245 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252c003d30 con 0x7f253415cc60 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.114+0000 7f253af19640 1 -- 172.21.15.59:0/3255693245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25340cba70 con 0x7f253415cc60 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.114+0000 7f25317fa640 1 -- 172.21.15.59:0/3255693245 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252c005310 con 0x7f253415cc60 2024-03-06T07:22:07.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.114+0000 7f253af19640 1 -- 172.21.15.59:0/3255693245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25340cc030 con 0x7f253415cc60 2024-03-06T07:22:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.115+0000 7f253af19640 1 -- 172.21.15.59:0/3255693245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253404f330 con 0x7f253415cc60 2024-03-06T07:22:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.115+0000 7f25317fa640 1 -- 172.21.15.59:0/3255693245 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f252c0054b0 con 0x7f253415cc60 2024-03-06T07:22:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.116+0000 7f25317fa640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f252006b500 0x7f252006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.116+0000 7f25317fa640 1 -- 172.21.15.59:0/3255693245 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f252c089f50 con 0x7f253415cc60 2024-03-06T07:22:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.116+0000 7f2538cb5640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f252006b500 0x7f252006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.116+0000 7f2538cb5640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f252006b500 0x7f252006d9c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2524000f60 tx=0x7f2524007d30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.120+0000 7f25317fa640 1 -- 172.21.15.59:0/3255693245 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f252c058780 con 0x7f253415cc60 2024-03-06T07:22:07.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.337+0000 7f253af19640 1 -- 172.21.15.59:0/3255693245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25340cdb30 con 0x7f253415cc60 2024-03-06T07:22:07.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.338+0000 7f25317fa640 1 -- 172.21.15.59:0/3255693245 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7f252c05c820 con 0x7f253415cc60 2024-03-06T07:22:07.340 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:07.340 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1759048959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1759048959},{"type":"v1","addr":"172.21.15.164:6836","nonce":1759048959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/550023536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":550023536},{"type":"v1","addr":"172.21.15.59:6833","nonce":550023536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2132697832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2132697832},{"type":"v1","addr":"172.21.15.59:6835","nonce":2132697832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:54.767210+0000","modified":"2024-03-06T07:21:56.776415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/1957940318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1957940318},{"type":"v1","addr":"172.21.15.164:6837","nonce":1957940318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-06T07:22:07.340 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 6 2024-03-06T07:22:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 -- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f252006b500 msgr2=0x7f252006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f252006b500 0x7f252006d9c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2524000f60 tx=0x7f2524007d30 comp rx=0 tx=0).stop 2024-03-06T07:22:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 -- 172.21.15.59:0/3255693245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f253415cc60 msgr2=0x7f25340cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f253415cc60 0x7f25340cb1a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f252c00c6b0 tx=0x7f252c00cb70 comp rx=0 tx=0).stop 2024-03-06T07:22:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 -- 172.21.15.59:0/3255693245 shutdown_connections 2024-03-06T07:22:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f252006b500 0x7f252006d9c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f253415cc60 0x7f25340cb1a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25341597c0 0x7f25340cac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 --2- 172.21.15.59:0/3255693245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253406b4f0 0x7f2534142ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 -- 172.21.15.59:0/3255693245 >> 172.21.15.59:0/3255693245 conn(0x7f253406c0c0 msgr2=0x7f253415d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 -- 172.21.15.59:0/3255693245 shutdown_connections 2024-03-06T07:22:07.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.340+0000 7f253af19640 1 -- 172.21.15.59:0/3255693245 wait complete. 2024-03-06T07:22:07.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:54.767210+0000', 'modified': '2024-03-06T07:21:56.776415+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4302}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4302': {'gid': 4302, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.164:6837/1957940318', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 1957940318}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 1957940318}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4302, 'qdb_cluster': [4302]} 2024-03-06T07:22:07.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:07.756 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:22:07.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.862+0000 7f7e5f70c640 1 Processor -- start 2024-03-06T07:22:07.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.863+0000 7f7e5f70c640 1 -- start start 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.863+0000 7f7e5f70c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7e580b4fe0 0x7f7e580b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.863+0000 7f7e5f70c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7e5815ba50 0x7f7e5815bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.863+0000 7f7e5f70c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5815d510 con 0x7f7e580b4c00 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.863+0000 7f7e5f70c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5815e0b0 con 0x7f7e5815ba50 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.863+0000 7f7e5f70c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5815d780 con 0x7f7e580b4fe0 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.864+0000 7f7e5dca9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7e5815ba50 0x7f7e5815bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.864+0000 7f7e5dca9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7e5815ba50 0x7f7e5815bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42830/0 (socket says 172.21.15.59:42830) 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.864+0000 7f7e5dca9640 1 -- 172.21.15.59:0/981785150 learned_addr learned my addr 172.21.15.59:0/981785150 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.864+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/981785150 >> v2:172.21.15.164:3300/0 conn(0x7f7e580b4fe0 0x7f7e580b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.864+0000 7f7e5dca9640 1 -- 172.21.15.59:0/981785150 >> v2:172.21.15.164:3300/0 conn(0x7f7e580b4fe0 msgr2=0x7f7e580b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.864+0000 7f7e5dca9640 1 --2- 172.21.15.59:0/981785150 >> v2:172.21.15.164:3300/0 conn(0x7f7e580b4fe0 0x7f7e580b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.864+0000 7f7e5dca9640 1 -- 172.21.15.59:0/981785150 >> v1:172.21.15.164:6789/0 conn(0x7f7e580b4c00 legacy=0x7f7e580b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.864+0000 7f7e4ffff640 1 -- 172.21.15.59:0/981785150 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256176168 0 0) 0x7f7e5815d510 con 0x7f7e580b4c00 2024-03-06T07:22:07.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.864+0000 7f7e5dca9640 1 -- 172.21.15.59:0/981785150 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5815da60 con 0x7f7e5815ba50 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.865+0000 7f7e5dca9640 1 --2- 172.21.15.59:0/981785150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e5815ba50 0x7f7e5815bec0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7e580730b0 tx=0x7f7e5002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e776f4b0ca3891 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.865+0000 7f7e4ffff640 1 -- 172.21.15.59:0/981785150 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5002cbb0 con 0x7f7e5815ba50 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.865+0000 7f7e4ffff640 1 -- 172.21.15.59:0/981785150 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5002cd50 con 0x7f7e5815ba50 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.865+0000 7f7e4ffff640 1 -- 172.21.15.59:0/981785150 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e50034610 con 0x7f7e5815ba50 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.865+0000 7f7e5f70c640 1 -- 172.21.15.59:0/981785150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e5815ba50 msgr2=0x7f7e5815bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.865+0000 7f7e5f70c640 1 --2- 172.21.15.59:0/981785150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e5815ba50 0x7f7e5815bec0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7e580730b0 tx=0x7f7e5002c010 comp rx=0 tx=0).stop 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.865+0000 7f7e5f70c640 1 -- 172.21.15.59:0/981785150 shutdown_connections 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.865+0000 7f7e5f70c640 1 --2- 172.21.15.59:0/981785150 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e5815ba50 0x7f7e5815bec0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.865+0000 7f7e5f70c640 1 --2- 172.21.15.59:0/981785150 >> v2:172.21.15.164:3300/0 conn(0x7f7e580b4fe0 0x7f7e580b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.865+0000 7f7e5f70c640 1 -- 172.21.15.59:0/981785150 >> 172.21.15.59:0/981785150 conn(0x7f7e5809a9f0 msgr2=0x7f7e5809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.865+0000 7f7e5f70c640 1 -- 172.21.15.59:0/981785150 shutdown_connections 2024-03-06T07:22:07.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.866+0000 7f7e5f70c640 1 -- 172.21.15.59:0/981785150 wait complete. 2024-03-06T07:22:07.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.867+0000 7f7e5f70c640 1 Processor -- start 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.867+0000 7f7e5f70c640 1 -- start start 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.867+0000 7f7e5f70c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e580b4c00 0x7f7e581543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.867+0000 7f7e5f70c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e580b4fe0 0x7f7e58159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.867+0000 7f7e5f70c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5815ba50 0x7f7e58159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.867+0000 7f7e5f70c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5815f6d0 con 0x7f7e580b4c00 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.867+0000 7f7e5f70c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5815f550 con 0x7f7e580b4fe0 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.867+0000 7f7e5f70c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5815c400 con 0x7f7e5815ba50 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.867+0000 7f7e5d4a8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e580b4c00 0x7f7e581543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.867+0000 7f7e5d4a8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e580b4c00 0x7f7e581543d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42844/0 (socket says 172.21.15.59:42844) 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5d4a8640 1 -- 172.21.15.59:0/3943732831 learned_addr learned my addr 172.21.15.59:0/3943732831 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e580b4fe0 0x7f7e58159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5dca9640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5815ba50 0x7f7e58159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5d4a8640 1 -- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5815ba50 msgr2=0x7f7e58159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5815ba50 0x7f7e58159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5d4a8640 1 -- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e580b4fe0 msgr2=0x7f7e58159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e580b4fe0 0x7f7e58159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5d4a8640 1 -- 172.21.15.59:0/3943732831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e58154d40 con 0x7f7e580b4c00 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e580b4fe0 0x7f7e58159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e580b4c00 0x7f7e581543d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7e5400c4e0 tx=0x7f7e5400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e4dffb640 1 -- 172.21.15.59:0/3943732831 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e54017070 con 0x7f7e580b4c00 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e4dffb640 1 -- 172.21.15.59:0/3943732831 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e540035f0 con 0x7f7e580b4c00 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5f70c640 1 -- 172.21.15.59:0/3943732831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5806c310 con 0x7f7e580b4c00 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e4dffb640 1 -- 172.21.15.59:0/3943732831 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e540039e0 con 0x7f7e580b4c00 2024-03-06T07:22:07.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.868+0000 7f7e5f70c640 1 -- 172.21.15.59:0/3943732831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5806c8d0 con 0x7f7e580b4c00 2024-03-06T07:22:07.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.870+0000 7f7e4dffb640 1 -- 172.21.15.59:0/3943732831 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e5401a070 con 0x7f7e580b4c00 2024-03-06T07:22:07.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.870+0000 7f7e4dffb640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e3c06b4d0 0x7f7e3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.871+0000 7f7e4dffb640 1 -- 172.21.15.59:0/3943732831 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e54089450 con 0x7f7e580b4c00 2024-03-06T07:22:07.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.871+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e3c06b4d0 0x7f7e3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.871+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e3c06b4d0 0x7f7e3c06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7e5809f4e0 tx=0x7f7e48009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.871+0000 7f7e623d2740 1 Processor -- start 2024-03-06T07:22:07.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e623d2740 1 -- start start 2024-03-06T07:22:07.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264837870 0x556264837c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55626483b8c0 0x55626483d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264867600 0x556264867a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e623d2740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562647a16b0 con 0x556264837870 2024-03-06T07:22:07.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e623d2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55626474aa60 con 0x55626483b8c0 2024-03-06T07:22:07.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e623d2740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55626479d750 con 0x556264867600 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e5d4a8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264837870 0x556264837c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e5d4a8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264837870 0x556264837c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42854/0 (socket says 172.21.15.59:42854) 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e5d4a8640 1 -- 172.21.15.59:0/875014919 learned_addr learned my addr 172.21.15.59:0/875014919 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/875014919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55626483b8c0 0x55626483d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.872+0000 7f7e5dca9640 1 --2- 172.21.15.59:0/875014919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264867600 0x556264867a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e5d4a8640 1 -- 172.21.15.59:0/875014919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264867600 msgr2=0x556264867a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/875014919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264867600 0x556264867a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e5d4a8640 1 -- 172.21.15.59:0/875014919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55626483b8c0 msgr2=0x55626483d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/875014919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55626483b8c0 0x55626483d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e5d4a8640 1 -- 172.21.15.59:0/875014919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556264768f30 con 0x556264837870 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/875014919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264837870 0x556264837c50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7e54087a30 tx=0x7f7e54057670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290064e18dbd3b0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e42ffd640 1 -- 172.21.15.59:0/875014919 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e54060370 con 0x556264837870 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e42ffd640 1 -- 172.21.15.59:0/875014919 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e54056010 con 0x556264837870 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e42ffd640 1 -- 172.21.15.59:0/875014919 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e54089070 con 0x556264837870 2024-03-06T07:22:07.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e623d2740 1 -- 172.21.15.59:0/875014919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264837870 msgr2=0x556264837c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e623d2740 1 --2- 172.21.15.59:0/875014919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264837870 0x556264837c50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7e54087a30 tx=0x7f7e54057670 comp rx=0 tx=0).stop 2024-03-06T07:22:07.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e623d2740 1 -- 172.21.15.59:0/875014919 shutdown_connections 2024-03-06T07:22:07.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e623d2740 1 --2- 172.21.15.59:0/875014919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264867600 0x556264867a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e623d2740 1 --2- 172.21.15.59:0/875014919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55626483b8c0 0x55626483d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e623d2740 1 --2- 172.21.15.59:0/875014919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264837870 0x556264837c50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.873+0000 7f7e623d2740 1 -- 172.21.15.59:0/875014919 >> 172.21.15.59:0/875014919 conn(0x5562648398f0 msgr2=0x55626483b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.874+0000 7f7e623d2740 1 -- 172.21.15.59:0/875014919 shutdown_connections 2024-03-06T07:22:07.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.874+0000 7f7e623d2740 1 -- 172.21.15.59:0/875014919 wait complete. 2024-03-06T07:22:07.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.874+0000 7f7e623d2740 1 -- start start 2024-03-06T07:22:07.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.874+0000 7f7e623d2740 1 Processor -- start 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264867600 0x556264861e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264862490 0x55626487dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264880700 0x55626487ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e623d2740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55626483bd50 con 0x556264862490 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e623d2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55626475dea0 con 0x556264880700 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e623d2740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55626483c030 con 0x556264867600 2024-03-06T07:22:07.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e623d2740 10 client.0 _ll_register_callbacks cb 0x55626475d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e5cca7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264862490 0x55626487dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e5cca7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264862490 0x55626487dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42870/0 (socket says 172.21.15.59:42870) 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e5cca7640 1 -- 172.21.15.59:0/2711893831 learned_addr learned my addr 172.21.15.59:0/2711893831 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e5dca9640 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264880700 0x55626487ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264867600 0x556264861e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e5cca7640 1 -- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264867600 msgr2=0x556264861e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264867600 0x556264861e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e5cca7640 1 -- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264880700 msgr2=0x55626487ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264880700 0x55626487ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.875+0000 7f7e5cca7640 1 -- 172.21.15.59:0/2711893831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55626475e270 con 0x556264862490 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264862490 0x55626487dfc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7e58157eb0 tx=0x7f7e480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5308c26d7c0797ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e37fff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e37fff640 1 -- 172.21.15.59:0/2711893831 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48027070 con 0x556264862490 2024-03-06T07:22:07.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e37fff640 1 -- 172.21.15.59:0/2711893831 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e48003e60 con 0x556264862490 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e623d2740 1 -- 172.21.15.59:0/2711893831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e54056010 con 0x556264862490 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e37fff640 1 -- 172.21.15.59:0/2711893831 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e480149c0 con 0x556264862490 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e623d2740 10 client.4680 fetch_fsmap learned FSMap version 6 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e623d2740 1 -- 172.21.15.59:0/2711893831 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x5562644b62d0 con 0x556264862490 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e37fff640 1 -- 172.21.15.59:0/2711893831 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e54056010 con 0x556264862490 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e37fff640 1 -- 172.21.15.59:0/2711893831 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 6) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7e48020050 con 0x556264862490 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e623d2740 10 client.4680 fetch_fsmap finished waiting for FSMap version 6 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e623d2740 10 client.4680 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e623d2740 20 client.4680 populate_metadata read hostname 'smithi059' 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e623d2740 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:6835/1957940318,v1:172.21.15.164:6837/1957940318] conn(0x55626489ef60 0x5562648a1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e623d2740 4 client.4680 mds_command: new command op to 4302 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.876+0000 7f7e623d2740 1 -- 172.21.15.59:0/2711893831 --> [v2:172.21.15.164:6835/1957940318,v1:172.21.15.164:6837/1957940318] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5562647aebf0 con 0x55626489ef60 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.877+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:6835/1957940318,v1:172.21.15.164:6837/1957940318] conn(0x55626489ef60 0x5562648a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.877+0000 7f7e37fff640 10 client.4680 ms_handle_connect on v2:172.21.15.164:6835/1957940318 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.877+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:6835/1957940318,v1:172.21.15.164:6837/1957940318] conn(0x55626489ef60 0x5562648a1350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca80fa18894634ca server_cookie=777619b9f74f608c in_seq=0 out_seq=0 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.877+0000 7f7e37fff640 1 -- 172.21.15.59:0/2711893831 <== mds.0 v2:172.21.15.164:6835/1957940318 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5562647aebf0 con 0x55626489ef60 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.877+0000 7f7e37fff640 10 client.4680 handle_command_reply: tid=0 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 1 client.4680 shutdown 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 1 -- 172.21.15.59:0/2711893831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264862490 msgr2=0x55626487dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264862490 0x55626487dfc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7e58157eb0 tx=0x7f7e480046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 1 -- 172.21.15.59:0/2711893831 shutdown_connections 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:6835/1957940318,v1:172.21.15.164:6837/1957940318] conn(0x55626489ef60 0x5562648a1350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264880700 0x55626487ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264862490 0x55626487dfc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 1 --2- 172.21.15.59:0/2711893831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264867600 0x556264861e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e37fff640 0 client.4680 ms_handle_reset on v2:172.21.15.164:6835/1957940318 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 1 -- 172.21.15.59:0/2711893831 >> 172.21.15.59:0/2711893831 conn(0x556264837870 msgr2=0x5562648655c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 1 -- 172.21.15.59:0/2711893831 shutdown_connections 2024-03-06T07:22:07.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 1 -- 172.21.15.59:0/2711893831 wait complete. 2024-03-06T07:22:07.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.878+0000 7f7e623d2740 20 client.4680 trim_cache size 0 max 16384 2024-03-06T07:22:07.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e623d2740 1 Processor -- start 2024-03-06T07:22:07.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e623d2740 1 -- start start 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264862490 0x5562648831e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264867600 0x556264865ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264880700 0x556264882790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e623d2740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e581fefe0 con 0x556264880700 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e623d2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e581fe3e0 con 0x556264867600 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e623d2740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e581fe9e0 con 0x556264862490 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e5dca9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264880700 0x556264882790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e5dca9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264880700 0x556264882790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42872/0 (socket says 172.21.15.59:42872) 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e5dca9640 1 -- 172.21.15.59:0/2953702204 learned_addr learned my addr 172.21.15.59:0/2953702204 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/2953702204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264867600 0x556264865ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.895+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/2953702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264862490 0x5562648831e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e5dca9640 1 -- 172.21.15.59:0/2953702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264862490 msgr2=0x5562648831e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e5dca9640 1 --2- 172.21.15.59:0/2953702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264862490 0x5562648831e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e5dca9640 1 -- 172.21.15.59:0/2953702204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264867600 msgr2=0x556264865ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e5dca9640 1 --2- 172.21.15.59:0/2953702204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264867600 0x556264865ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e5dca9640 1 -- 172.21.15.59:0/2953702204 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562645806b0 con 0x556264880700 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e5dca9640 1 --2- 172.21.15.59:0/2953702204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264880700 0x556264882790 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7e500093e0 tx=0x7f7e5002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0187c7c0efb7ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e427fc640 1 -- 172.21.15.59:0/2953702204 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5002caf0 con 0x556264880700 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e427fc640 1 -- 172.21.15.59:0/2953702204 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5002cc90 con 0x556264880700 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e427fc640 1 -- 172.21.15.59:0/2953702204 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e50049430 con 0x556264880700 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e623d2740 1 -- 172.21.15.59:0/2953702204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264880700 msgr2=0x556264882790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e623d2740 1 --2- 172.21.15.59:0/2953702204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264880700 0x556264882790 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7e500093e0 tx=0x7f7e5002e000 comp rx=0 tx=0).stop 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e623d2740 1 -- 172.21.15.59:0/2953702204 shutdown_connections 2024-03-06T07:22:07.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e623d2740 1 --2- 172.21.15.59:0/2953702204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264880700 0x556264882790 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e623d2740 1 --2- 172.21.15.59:0/2953702204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264867600 0x556264865ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e623d2740 1 --2- 172.21.15.59:0/2953702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264862490 0x5562648831e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e623d2740 1 -- 172.21.15.59:0/2953702204 >> 172.21.15.59:0/2953702204 conn(0x556264837870 msgr2=0x55626483b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e623d2740 1 -- 172.21.15.59:0/2953702204 shutdown_connections 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.896+0000 7f7e623d2740 1 -- 172.21.15.59:0/2953702204 wait complete. 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.897+0000 7f7e623d2740 1 -- start start 2024-03-06T07:22:07.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.897+0000 7f7e623d2740 1 Processor -- start 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264862490 0x5562648ec630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264867600 0x55626483bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e623d2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264880700 0x5562648f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e623d2740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55626486b820 con 0x556264867600 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e623d2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55626487fb70 con 0x556264880700 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e623d2740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562648688c0 con 0x556264862490 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e623d2740 10 client.0 _ll_register_callbacks cb 0x7f7e48003840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5cca7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264867600 0x55626483bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5cca7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264867600 0x55626483bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42878/0 (socket says 172.21.15.59:42878) 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5dca9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264880700 0x5562648f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5cca7640 1 -- 172.21.15.59:0/3123162554 learned_addr learned my addr 172.21.15.59:0/3123162554 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264862490 0x5562648ec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5cca7640 1 -- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264862490 msgr2=0x5562648ec630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264862490 0x5562648ec630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5cca7640 1 -- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264880700 msgr2=0x5562648f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264880700 0x5562648f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5cca7640 1 -- 172.21.15.59:0/3123162554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556264866ae0 con 0x556264867600 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5dca9640 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264880700 0x5562648f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e5cca7640 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264867600 0x55626483bf60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7e48004770 tx=0x7f7e48019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d92a91e38f0f54f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.898+0000 7f7e37fff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e37fff640 1 -- 172.21.15.59:0/3123162554 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4803a070 con 0x556264867600 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e37fff640 1 -- 172.21.15.59:0/3123162554 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e4803f460 con 0x556264867600 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e623d2740 1 -- 172.21.15.59:0/3123162554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e5002cc90 con 0x556264867600 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e37fff640 1 -- 172.21.15.59:0/3123162554 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e48047050 con 0x556264867600 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e623d2740 10 client.4686 fetch_fsmap learned FSMap version 6 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e623d2740 1 -- 172.21.15.59:0/3123162554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x55626483c4a0 con 0x556264867600 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e37fff640 1 -- 172.21.15.59:0/3123162554 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5002cc90 con 0x556264867600 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e37fff640 1 -- 172.21.15.59:0/3123162554 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 6) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7e4804a020 con 0x556264867600 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e623d2740 10 client.4686 fetch_fsmap finished waiting for FSMap version 6 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e623d2740 10 client.4686 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e623d2740 20 client.4686 populate_metadata read hostname 'smithi059' 2024-03-06T07:22:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e623d2740 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:6835/1957940318,v1:172.21.15.164:6837/1957940318] conn(0x55626487e480 0x55626487c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e623d2740 4 client.4686 mds_command: new command op to 4302 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e623d2740 1 -- 172.21.15.59:0/3123162554 --> [v2:172.21.15.164:6835/1957940318,v1:172.21.15.164:6837/1957940318] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55626483a050 con 0x55626487e480 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.899+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:6835/1957940318,v1:172.21.15.164:6837/1957940318] conn(0x55626487e480 0x55626487c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e37fff640 10 client.4686 ms_handle_connect on v2:172.21.15.164:6835/1957940318 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e5d4a8640 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:6835/1957940318,v1:172.21.15.164:6837/1957940318] conn(0x55626487e480 0x55626487c690 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a3cc3e10dcf1b61 server_cookie=4c69f63dcfa43ce7 in_seq=0 out_seq=0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e37fff640 1 -- 172.21.15.59:0/3123162554 <== mds.0 v2:172.21.15.164:6835/1957940318 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+323 (crc 0 0 0) 0x55626483a050 con 0x55626487e480 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e37fff640 10 client.4686 handle_command_reply: tid=0 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 1 client.4686 shutdown 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 1 -- 172.21.15.59:0/3123162554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264867600 msgr2=0x55626483bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264867600 0x55626483bf60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7e48004770 tx=0x7f7e48019000 comp rx=0 tx=0).stop 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 1 -- 172.21.15.59:0/3123162554 shutdown_connections 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:6835/1957940318,v1:172.21.15.164:6837/1957940318] conn(0x55626487e480 0x55626487c690 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556264880700 0x5562648f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e37fff640 0 client.4686 ms_handle_reset on v2:172.21.15.164:6835/1957940318 2024-03-06T07:22:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556264867600 0x55626483bf60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 1 --2- 172.21.15.59:0/3123162554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556264862490 0x5562648ec630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 1 -- 172.21.15.59:0/3123162554 >> 172.21.15.59:0/3123162554 conn(0x556264837870 msgr2=0x55626483b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 1 -- 172.21.15.59:0/3123162554 shutdown_connections 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 1 -- 172.21.15.59:0/3123162554 wait complete. 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.900+0000 7f7e623d2740 20 client.4686 trim_cache size 0 max 16384 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 4302, 2024-03-06T07:22:07.908 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 12.480866559000001, 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 6, 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 20, 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 20, 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 19.80063552 2024-03-06T07:22:07.909 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 -- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e3c06b4d0 msgr2=0x7f7e3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e3c06b4d0 0x7f7e3c06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7e5809f4e0 tx=0x7f7e48009190 comp rx=0 tx=0).stop 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 -- 172.21.15.59:0/3943732831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e580b4c00 msgr2=0x7f7e581543d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e580b4c00 0x7f7e581543d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7e5400c4e0 tx=0x7f7e5400c9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 -- 172.21.15.59:0/3943732831 shutdown_connections 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e3c06b4d0 0x7f7e3c06d990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5815ba50 0x7f7e58159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e580b4fe0 0x7f7e58159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 --2- 172.21.15.59:0/3943732831 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e580b4c00 0x7f7e581543d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 -- 172.21.15.59:0/3943732831 >> 172.21.15.59:0/3943732831 conn(0x7f7e5809a9f0 msgr2=0x7f7e58070110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:07.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 -- 172.21.15.59:0/3943732831 shutdown_connections 2024-03-06T07:22:07.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:07.901+0000 7f7e5f70c640 1 -- 172.21.15.59:0/3943732831 wait complete. 2024-03-06T07:22:07.913 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-06T07:22:07.938 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-06T07:22:07.938 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-06T07:22:07.938 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-06T07:22:07.938 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:07.938 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:07.938 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-06T07:22:07.938 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:07.938 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:07.938 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-06T07:22:07.939 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:07.939 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:07.939 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-06T07:22:07.939 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:07.939 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:07.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:07.939 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-03-06T07:22:07.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:07.955 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link delete ceph-brx 2024-03-06T07:22:08.026 INFO:teuthology.orchestra.run.smithi164.stderr:Cannot find device "ceph-brx" 2024-03-06T07:22:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:22:08.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.028 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:22:08.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.044 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link delete ceph-brx 2024-03-06T07:22:08.111 INFO:teuthology.orchestra.run.smithi059.stderr:Cannot find device "ceph-brx" 2024-03-06T07:22:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:22:08.116 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-06T07:22:08.117 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:08.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.256+0000 7efc49eb5640 1 Processor -- start 2024-03-06T07:22:08.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.256+0000 7efc49eb5640 1 -- start start 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc49eb5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efc440d00c0 0x7efc440d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc49eb5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc440d09e0 con 0x7efc4406cf20 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc49eb5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4415e380 con 0x7efc440cf850 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc49eb5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc4406ca90 con 0x7efc440d00c0 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc43fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7efc4406cf20 0x7efc4406d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42272/0 (socket says 172.21.15.59:42272) 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc43fff640 1 -- 172.21.15.59:0/4150077000 learned_addr learned my addr 172.21.15.59:0/4150077000 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc437fe640 1 --2- 172.21.15.59:0/4150077000 >> v2:172.21.15.164:3300/0 conn(0x7efc440d00c0 0x7efc440d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc427fc640 1 -- 172.21.15.59:0/4150077000 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811901185 0 0) 0x7efc440d09e0 con 0x7efc4406cf20 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc427fc640 1 -- 172.21.15.59:0/4150077000 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2c003520 con 0x7efc4406cf20 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc427fc640 1 -- 172.21.15.59:0/4150077000 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919670815 0 0) 0x7efc4415e380 con 0x7efc440cf850 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc427fc640 1 -- 172.21.15.59:0/4150077000 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc440d09e0 con 0x7efc440cf850 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc437fe640 1 -- 172.21.15.59:0/4150077000 >> v1:172.21.15.164:6789/0 conn(0x7efc440cf850 legacy=0x7efc440cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc437fe640 1 -- 172.21.15.59:0/4150077000 >> v1:172.21.15.59:6789/0 conn(0x7efc4406cf20 legacy=0x7efc4406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc427fc640 1 -- 172.21.15.59:0/4150077000 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580018585 0 0) 0x7efc2c003520 con 0x7efc4406cf20 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.257+0000 7efc437fe640 1 -- 172.21.15.59:0/4150077000 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc440cdc10 con 0x7efc440d00c0 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.258+0000 7efc437fe640 1 --2- 172.21.15.59:0/4150077000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc440d00c0 0x7efc440d04a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efc340097a0 tx=0x7efc3402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d618a87001556c79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.258+0000 7efc427fc640 1 -- 172.21.15.59:0/4150077000 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34038070 con 0x7efc440d00c0 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.258+0000 7efc427fc640 1 -- 172.21.15.59:0/4150077000 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc34031500 con 0x7efc440d00c0 2024-03-06T07:22:08.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.258+0000 7efc427fc640 1 -- 172.21.15.59:0/4150077000 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0019f0 con 0x7efc440d00c0 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.259+0000 7efc49eb5640 1 -- 172.21.15.59:0/4150077000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc440d00c0 msgr2=0x7efc440d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.259+0000 7efc49eb5640 1 --2- 172.21.15.59:0/4150077000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc440d00c0 0x7efc440d04a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efc340097a0 tx=0x7efc3402b820 comp rx=0 tx=0).stop 2024-03-06T07:22:08.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.259+0000 7efc49eb5640 1 -- 172.21.15.59:0/4150077000 shutdown_connections 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.259+0000 7efc49eb5640 1 --2- 172.21.15.59:0/4150077000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc440d00c0 0x7efc440d04a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.259+0000 7efc49eb5640 1 -- 172.21.15.59:0/4150077000 >> 172.21.15.59:0/4150077000 conn(0x7efc440b35a0 msgr2=0x7efc440b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.259+0000 7efc49eb5640 1 -- 172.21.15.59:0/4150077000 shutdown_connections 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.259+0000 7efc49eb5640 1 -- 172.21.15.59:0/4150077000 wait complete. 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.260+0000 7efc49eb5640 1 Processor -- start 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.260+0000 7efc49eb5640 1 -- start start 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc49eb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc4406cf20 0x7efc441f8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc49eb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc440cf850 0x7efc440b2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc49eb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc440d00c0 0x7efc440c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc49eb5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4406c720 con 0x7efc440d00c0 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc49eb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4406c440 con 0x7efc4406cf20 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc49eb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc440b1030 con 0x7efc440cf850 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc43fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc440d00c0 0x7efc440c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc43fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc440d00c0 0x7efc440c55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42892/0 (socket says 172.21.15.59:42892) 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc43fff640 1 -- 172.21.15.59:0/3022665631 learned_addr learned my addr 172.21.15.59:0/3022665631 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc42ffd640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc440cf850 0x7efc440b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc437fe640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc4406cf20 0x7efc441f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc43fff640 1 -- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc440cf850 msgr2=0x7efc440b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc43fff640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc440cf850 0x7efc440b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc43fff640 1 -- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc4406cf20 msgr2=0x7efc441f8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc43fff640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc4406cf20 0x7efc441f8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc43fff640 1 -- 172.21.15.59:0/3022665631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc440c5f10 con 0x7efc440d00c0 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.261+0000 7efc43fff640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc440d00c0 0x7efc440c55a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00cc50 tx=0x7efc3c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.262+0000 7efc2bfff640 1 -- 172.21.15.59:0/3022665631 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c00ce60 con 0x7efc440d00c0 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.262+0000 7efc2bfff640 1 -- 172.21.15.59:0/3022665631 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c012400 con 0x7efc440d00c0 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.262+0000 7efc49eb5640 1 -- 172.21.15.59:0/3022665631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc440c61f0 con 0x7efc440d00c0 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.262+0000 7efc2bfff640 1 -- 172.21.15.59:0/3022665631 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0127f0 con 0x7efc440d00c0 2024-03-06T07:22:08.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.262+0000 7efc49eb5640 1 -- 172.21.15.59:0/3022665631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc441f7b70 con 0x7efc440d00c0 2024-03-06T07:22:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.263+0000 7efc49eb5640 1 -- 172.21.15.59:0/3022665631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4406a090 con 0x7efc440d00c0 2024-03-06T07:22:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.263+0000 7efc2bfff640 1 -- 172.21.15.59:0/3022665631 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc3c012990 con 0x7efc440d00c0 2024-03-06T07:22:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.263+0000 7efc2bfff640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc3006b4d0 0x7efc3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.263+0000 7efc2bfff640 1 -- 172.21.15.59:0/3022665631 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc3c08a890 con 0x7efc440d00c0 2024-03-06T07:22:08.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.264+0000 7efc437fe640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc3006b4d0 0x7efc3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.264+0000 7efc437fe640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc3006b4d0 0x7efc3006d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc34035000 tx=0x7efc34008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.268+0000 7efc2bfff640 1 -- 172.21.15.59:0/3022665631 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc4406a090 con 0x7efc440d00c0 2024-03-06T07:22:08.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.488+0000 7efc49eb5640 1 -- 172.21.15.59:0/3022665631 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efc441f7d70 con 0x7efc440d00c0 2024-03-06T07:22:08.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.489+0000 7efc2bfff640 1 -- 172.21.15.59:0/3022665631 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7efc3c0590d0 con 0x7efc440d00c0 2024-03-06T07:22:08.491 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:08.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 -- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc3006b4d0 msgr2=0x7efc3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc3006b4d0 0x7efc3006d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc34035000 tx=0x7efc34008030 comp rx=0 tx=0).stop 2024-03-06T07:22:08.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 -- 172.21.15.59:0/3022665631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc440d00c0 msgr2=0x7efc440c55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc440d00c0 0x7efc440c55a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00cc50 tx=0x7efc3c00a430 comp rx=0 tx=0).stop 2024-03-06T07:22:08.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 -- 172.21.15.59:0/3022665631 shutdown_connections 2024-03-06T07:22:08.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc3006b4d0 0x7efc3006d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc440d00c0 0x7efc440c55a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc440cf850 0x7efc440b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 --2- 172.21.15.59:0/3022665631 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc4406cf20 0x7efc441f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 -- 172.21.15.59:0/3022665631 >> 172.21.15.59:0/3022665631 conn(0x7efc440b35a0 msgr2=0x7efc440b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 -- 172.21.15.59:0/3022665631 shutdown_connections 2024-03-06T07:22:08.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:08.491+0000 7efc49eb5640 1 -- 172.21.15.59:0/3022665631 wait complete. 2024-03-06T07:22:08.503 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:08.503 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:08.504 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:22:08.504 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:22:08.504 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:08.504 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:08.504 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:08.504 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:08.504 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:08.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.504 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:22:08.520 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:08.520 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:08.520 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:08.521 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:08.521 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:08.521 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:08.521 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:08.521 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:08.522 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:22:08.522 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:08.522 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:08.522 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:22:08.522 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:08.522 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:08.523 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:22:08.523 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:22:08.523 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42650sec preferred_lft 42650sec 2024-03-06T07:22:08.523 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:22:08.523 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:08.523 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:08.524 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:08.524 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:22:08.524 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:22:08.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.525 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:22:08.525 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:22:08.525 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:22:08.525 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr flush dev ceph-brx 2024-03-06T07:22:08.526 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set ceph-brx up 2024-03-06T07:22:08.526 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:22:08.526 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:22:08.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.685 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:08.711 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:22:08.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.713 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:22:08.784 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:22:08.785 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:08.785 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:22:08.785 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:22:08.785 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:08.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.785 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:22:08.786 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:22:08.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:22:08.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:22:08.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:22:08.786 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:22:09.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.077 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:22:09.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.092 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list-id 2024-03-06T07:22:09.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.148 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:22:09.148 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:22:09.148 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:09.148 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:22:09.148 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:22:09.231 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:22:09.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.232 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:22:09.232 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:22:09.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:22:09.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:09.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:22:09.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:22:09.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:22:09.232 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:22:09.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.371 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:22:09.371 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:22:09.371 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:22:09.372 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:22:09.372 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:22:09.413 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:22:09.413 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:09.414 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:09.468 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:09.469 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:09.469 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:09.523 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:22:09.591 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:09.647 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:09.648 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:09.649 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:09.650 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:09.650 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:09.650 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:22:09.650 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:09.650 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:09.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.650 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:22:09.701 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:09.702 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:22:09.744 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:22:09.769 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:09.824 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:22:09.825 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:09.826 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:09.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.827 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:22:09.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:22:09.827+0000 7f95d26bb480 -1 init, newargv = 0x55d1beccc510 newargc=15 2024-03-06T07:22:09.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[47082]: starting ceph client 2024-03-06T07:22:09.833 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[47082]: starting fuse 2024-03-06T07:22:09.842 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:22:09.843 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:22:09.843 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:09.843 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:22:09.843 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:22:09.843 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:22:09.843 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:22:09.843 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:09.843 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:22:09.843 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:22:09.843 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:22:09.843 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:22:09.843 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:09.844 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:22:09.845 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:22:09.963 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.47082.asok 2024-03-06T07:22:09.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:09.973 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47082.asok 2024-03-06T07:22:09.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.973 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47082.asok status 2024-03-06T07:22:10.111 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:22:10.111 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:22:10.111 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:10.111 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:10.111 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "47082", 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 4698, 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 4698 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:22:10.112 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 59152021 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 59152021 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.4698 192.168.0.1:0/59152021", 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/59152021", 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 6, 2024-03-06T07:22:10.113 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 20, 2024-03-06T07:22:10.114 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:10.114 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:22:10.114 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:22:10.114 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:22:10.120 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:10.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.219+0000 7f92e8eb5640 1 Processor -- start 2024-03-06T07:22:10.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.219+0000 7f92e8eb5640 1 -- start start 2024-03-06T07:22:10.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e8eb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e415bf30 0x7f92e415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e8eb5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e415f580 con 0x7f92e40b3060 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e8eb5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e415b450 con 0x7f92e413dc10 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e8eb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e40cf3f0 con 0x7f92e415bf30 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e2d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e415bf30 0x7f92e415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92d9d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f92e40b3060 0x7f92e415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42280/0 (socket says 172.21.15.59:42280) 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92d9d9b640 1 -- 172.21.15.59:0/525016907 learned_addr learned my addr 172.21.15.59:0/525016907 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e259c640 1 --1- 172.21.15.59:0/525016907 >> v1:172.21.15.164:6789/0 conn(0x7f92e413dc10 0x7f92e40b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:41262/0 (socket says 172.21.15.59:41262) 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e1d9b640 1 -- 172.21.15.59:0/525016907 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005584646 0 0) 0x7f92e415f580 con 0x7f92e40b3060 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e1d9b640 1 -- 172.21.15.59:0/525016907 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d0003520 con 0x7f92e40b3060 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e1d9b640 1 -- 172.21.15.59:0/525016907 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406085914 0 0) 0x7f92e415b450 con 0x7f92e413dc10 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e2d9d640 1 -- 172.21.15.59:0/525016907 >> v1:172.21.15.164:6789/0 conn(0x7f92e413dc10 legacy=0x7f92e40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e2d9d640 1 -- 172.21.15.59:0/525016907 >> v1:172.21.15.59:6789/0 conn(0x7f92e40b3060 legacy=0x7f92e415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.220+0000 7f92e2d9d640 1 -- 172.21.15.59:0/525016907 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e415b730 con 0x7f92e415bf30 2024-03-06T07:22:10.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.221+0000 7f92e2d9d640 1 --2- 172.21.15.59:0/525016907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e415bf30 0x7f92e415ef60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f92dc009dc0 tx=0x7f92dc02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8e11c2b82802816 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.221+0000 7f92e1d9b640 1 -- 172.21.15.59:0/525016907 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92dc039070 con 0x7f92e415bf30 2024-03-06T07:22:10.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.221+0000 7f92e1d9b640 1 -- 172.21.15.59:0/525016907 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92dc02cc40 con 0x7f92e415bf30 2024-03-06T07:22:10.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.221+0000 7f92e1d9b640 1 -- 172.21.15.59:0/525016907 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92dc034440 con 0x7f92e415bf30 2024-03-06T07:22:10.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.222+0000 7f92e8eb5640 1 -- 172.21.15.59:0/525016907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e415bf30 msgr2=0x7f92e415ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.222+0000 7f92e8eb5640 1 --2- 172.21.15.59:0/525016907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e415bf30 0x7f92e415ef60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f92dc009dc0 tx=0x7f92dc02be90 comp rx=0 tx=0).stop 2024-03-06T07:22:10.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.222+0000 7f92e8eb5640 1 -- 172.21.15.59:0/525016907 shutdown_connections 2024-03-06T07:22:10.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.222+0000 7f92e8eb5640 1 --2- 172.21.15.59:0/525016907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e415bf30 0x7f92e415ef60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.222+0000 7f92e8eb5640 1 -- 172.21.15.59:0/525016907 >> 172.21.15.59:0/525016907 conn(0x7f92e40cf8a0 msgr2=0x7f92e40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.222+0000 7f92e8eb5640 1 -- 172.21.15.59:0/525016907 shutdown_connections 2024-03-06T07:22:10.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.222+0000 7f92e8eb5640 1 -- 172.21.15.59:0/525016907 wait complete. 2024-03-06T07:22:10.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.223+0000 7f92e8eb5640 1 Processor -- start 2024-03-06T07:22:10.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.223+0000 7f92e8eb5640 1 -- start start 2024-03-06T07:22:10.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e8eb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e40b3060 0x7f92e41407a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e8eb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92e413dc10 0x7f92e413c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e8eb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92e415bf30 0x7f92e413ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e8eb5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e40718e0 con 0x7f92e415bf30 2024-03-06T07:22:10.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e8eb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e415b210 con 0x7f92e413dc10 2024-03-06T07:22:10.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e8eb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e4071a60 con 0x7f92e40b3060 2024-03-06T07:22:10.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92e415bf30 0x7f92e413ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e40b3060 0x7f92e41407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92d9d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92e413dc10 0x7f92e413c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e40b3060 0x7f92e41407a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:38374/0 (socket says 172.21.15.59:38374) 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92e415bf30 0x7f92e413ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42894/0 (socket says 172.21.15.59:42894) 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e259c640 1 -- 172.21.15.59:0/3565604869 learned_addr learned my addr 172.21.15.59:0/3565604869 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92d9d9b640 1 -- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e40b3060 msgr2=0x7f92e41407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92d9d9b640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e40b3060 0x7f92e41407a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92d9d9b640 1 -- 172.21.15.59:0/3565604869 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92e415bf30 msgr2=0x7f92e413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92d9d9b640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92e415bf30 0x7f92e413ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92d9d9b640 1 -- 172.21.15.59:0/3565604869 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e41f4a30 con 0x7f92e413dc10 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e2d9d640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92e415bf30 0x7f92e413ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.224+0000 7f92e259c640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e40b3060 0x7f92e41407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.225+0000 7f92d9d9b640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92e413dc10 0x7f92e413c990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00c400 tx=0x7f92cc00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.225+0000 7f92db7fe640 1 -- 172.21.15.59:0/3565604869 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc017070 con 0x7f92e413dc10 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.225+0000 7f92db7fe640 1 -- 172.21.15.59:0/3565604869 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92cc005ce0 con 0x7f92e413dc10 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.225+0000 7f92db7fe640 1 -- 172.21.15.59:0/3565604869 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc0072f0 con 0x7f92e413dc10 2024-03-06T07:22:10.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.225+0000 7f92e8eb5640 1 -- 172.21.15.59:0/3565604869 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e41f3c70 con 0x7f92e413dc10 2024-03-06T07:22:10.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.225+0000 7f92e8eb5640 1 -- 172.21.15.59:0/3565604869 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e41f4230 con 0x7f92e413dc10 2024-03-06T07:22:10.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.226+0000 7f92e8eb5640 1 -- 172.21.15.59:0/3565604869 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e4068fb0 con 0x7f92e413dc10 2024-03-06T07:22:10.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.226+0000 7f92db7fe640 1 -- 172.21.15.59:0/3565604869 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92cc01a070 con 0x7f92e413dc10 2024-03-06T07:22:10.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.227+0000 7f92db7fe640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92c406b4d0 0x7f92c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.227+0000 7f92db7fe640 1 -- 172.21.15.59:0/3565604869 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92cc089540 con 0x7f92e413dc10 2024-03-06T07:22:10.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.227+0000 7f92e259c640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92c406b4d0 0x7f92c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.227+0000 7f92e259c640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92c406b4d0 0x7f92c406d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92d4007fd0 tx=0x7f92d4007e60 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.231+0000 7f92db7fe640 1 -- 172.21.15.59:0/3565604869 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92e4068fb0 con 0x7f92e413dc10 2024-03-06T07:22:10.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.451+0000 7f92e8eb5640 1 -- 172.21.15.59:0/3565604869 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f92e404f330 con 0x7f92e413dc10 2024-03-06T07:22:10.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.451+0000 7f92db7fe640 1 -- 172.21.15.59:0/3565604869 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f92cc057cf0 con 0x7f92e413dc10 2024-03-06T07:22:10.453 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:10.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 -- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92c406b4d0 msgr2=0x7f92c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92c406b4d0 0x7f92c406d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f92d4007fd0 tx=0x7f92d4007e60 comp rx=0 tx=0).stop 2024-03-06T07:22:10.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 -- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92e413dc10 msgr2=0x7f92e413c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92e413dc10 0x7f92e413c990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00c400 tx=0x7f92cc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 -- 172.21.15.59:0/3565604869 shutdown_connections 2024-03-06T07:22:10.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92c406b4d0 0x7f92c406d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92e415bf30 0x7f92e413ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92e413dc10 0x7f92e413c990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 --2- 172.21.15.59:0/3565604869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92e40b3060 0x7f92e41407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 -- 172.21.15.59:0/3565604869 >> 172.21.15.59:0/3565604869 conn(0x7f92e40cf8a0 msgr2=0x7f92e415a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 -- 172.21.15.59:0/3565604869 shutdown_connections 2024-03-06T07:22:10.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:10.453+0000 7f92e8eb5640 1 -- 172.21.15.59:0/3565604869 wait complete. 2024-03-06T07:22:10.465 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:10.465 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:10.465 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:22:10.466 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-06T07:22:10.466 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:10.466 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:10.466 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:10.466 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:10.466 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:22:10.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.466 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:22:10.481 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.481 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:10.481 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:22:10.481 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42648sec preferred_lft 42648sec 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:10.482 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link tentative 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.483 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:10.547 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:22:10.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.548 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:22:10.603 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:22:10.604 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:10.604 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:22:10.604 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:22:10.604 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:10.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.604 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:22:10.604 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:22:10.604 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:22:10.605 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:22:10.605 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:22:10.605 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:22:10.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.699 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:22:10.713 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:22:10.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.714 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list-id 2024-03-06T07:22:10.768 INFO:teuthology.orchestra.run.smithi059.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-06T07:22:10.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.769 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:22:10.769 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:22:10.769 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:10.769 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-06T07:22:10.769 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:22:10.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.850 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:22:10.917 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:10.917 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:10.917 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:10.917 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.917 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:10.918 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.918 INFO:teuthology.orchestra.run.smithi059.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:10.918 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether ee:4f:9c:6b:f3:65 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:10.918 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:10.918 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.918 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4f:9cff:fe6b:f365/64 scope link 2024-03-06T07:22:10.918 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.921 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:22:10.987 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:10.988 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:10.988 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:10.988 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.988 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:10.988 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.989 INFO:teuthology.orchestra.run.smithi059.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:10.989 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether ee:4f:9c:6b:f3:65 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:10.989 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:10.989 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.989 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4f:9cff:fe6b:f365/64 scope link 2024-03-06T07:22:10.989 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:10.990 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-06T07:22:10.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.990 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:22:10.990 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:22:10.990 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-06T07:22:10.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:10.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-06T07:22:10.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-06T07:22:10.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-06T07:22:10.991 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:22:11.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.117 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:22:11.117 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:22:11.117 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.1 up 2024-03-06T07:22:11.117 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:22:11.118 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:22:11.159 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-06T07:22:11.159 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:11.159 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:11.213 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:22:11.213 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:11.213 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:11.268 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:22:11.334 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:11.390 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:22:11.390 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:22:11.390 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:22:11.390 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:22:11.390 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:22:11.390 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:22:11.390 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:22:11.390 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:22:11.390 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:22:11.390 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:11.391 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:11.392 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:11.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.393 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:22:11.446 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:22:11.446 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-03-06T07:22:11.446 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-06T07:22:11.489 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:22:11.515 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:11.571 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:22:11.571 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:22:11.571 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:22:11.571 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:22:11.571 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:22:11.571 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:22:11.571 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:22:11.571 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:11.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:11.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:11.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.574 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:22:11.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi059.stderr:2024-03-06T07:22:11.572+0000 7f941183a480 -1 init, newargv = 0x558d6524daa0 newargc=15 2024-03-06T07:22:11.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi059.stderr:ceph-fuse[47576]: starting ceph client 2024-03-06T07:22:11.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi059.stderr:ceph-fuse[47576]: starting fuse 2024-03-06T07:22:11.589 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:22:11.589 INFO:teuthology.orchestra.run.smithi059.stdout:44 2024-03-06T07:22:11.589 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:22:11.590 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:22:11.591 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:22:11.591 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:22:11.591 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:11.591 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:22:11.591 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:11.591 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:11.591 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:11.591 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:22:11.591 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:22:11.591 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:22:11.591 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:11.592 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:11.592 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.2")) 2024-03-06T07:22:11.592 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:22:11.710 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.2.47576.asok 2024-03-06T07:22:11.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:11.720 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.47576.asok 2024-03-06T07:22:11.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:11.721 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.47576.asok status 2024-03-06T07:22:11.858 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:22:11.858 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "2", 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "47576", 2024-03-06T07:22:11.860 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:22:11.860 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:11.860 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:22:11.860 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:11.860 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 4710, 2024-03-06T07:22:11.861 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:22:11.861 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:22:11.861 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:22:11.861 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 4710 2024-03-06T07:22:11.861 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:11.861 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:22:11.861 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:22:11.862 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:11.862 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1979630483 2024-03-06T07:22:11.862 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:11.862 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:11.862 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:22:11.862 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:22:11.863 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:11.863 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1979630483 2024-03-06T07:22:11.863 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:11.863 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.4710 192.168.0.2:0/1979630483", 2024-03-06T07:22:11.863 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.2:0/1979630483", 2024-03-06T07:22:11.863 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:22:11.864 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 6, 2024-03-06T07:22:11.864 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 20, 2024-03-06T07:22:11.864 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:11.864 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:22:11.864 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:22:11.864 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:22:11.867 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.966+0000 7fa2562cf640 1 Processor -- start 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.966+0000 7fa2562cf640 1 -- start start 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.967+0000 7fa2562cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2500d22b0 0x7fa2500d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.967+0000 7fa2562cf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2500b3430 con 0x7fa2500b41f0 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.967+0000 7fa2562cf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2500b3710 con 0x7fa25015dfe0 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.967+0000 7fa2562cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2500d2c50 con 0x7fa2500d22b0 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.967+0000 7fa25486c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa2500b41f0 0x7fa2500b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42296/0 (socket says 172.21.15.59:42296) 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.967+0000 7fa25486c640 1 -- 172.21.15.59:0/4205436229 learned_addr learned my addr 172.21.15.59:0/4205436229 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.967+0000 7fa24f7fe640 1 --2- 172.21.15.59:0/4205436229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2500d22b0 0x7fa2500d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.968+0000 7fa24effd640 1 -- 172.21.15.59:0/4205436229 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301178669 0 0) 0x7fa2500b3430 con 0x7fa2500b41f0 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.968+0000 7fa24effd640 1 -- 172.21.15.59:0/4205436229 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa238003520 con 0x7fa2500b41f0 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.968+0000 7fa24effd640 1 -- 172.21.15.59:0/4205436229 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396154949 0 0) 0x7fa2500b3710 con 0x7fa25015dfe0 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.968+0000 7fa24effd640 1 -- 172.21.15.59:0/4205436229 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2500b3430 con 0x7fa25015dfe0 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.968+0000 7fa24f7fe640 1 -- 172.21.15.59:0/4205436229 >> v1:172.21.15.164:6789/0 conn(0x7fa25015dfe0 legacy=0x7fa2500cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.968+0000 7fa24f7fe640 1 -- 172.21.15.59:0/4205436229 >> v1:172.21.15.59:6789/0 conn(0x7fa2500b41f0 legacy=0x7fa2500b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.968+0000 7fa24f7fe640 1 -- 172.21.15.59:0/4205436229 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2500b39f0 con 0x7fa2500d22b0 2024-03-06T07:22:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.969+0000 7fa24f7fe640 1 --2- 172.21.15.59:0/4205436229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2500d22b0 0x7fa2500d2710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa2400097d0 tx=0x7fa24002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a1416bb5172ee4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.969+0000 7fa24effd640 1 -- 172.21.15.59:0/4205436229 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240035020 con 0x7fa2500d22b0 2024-03-06T07:22:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.969+0000 7fa24effd640 1 -- 172.21.15.59:0/4205436229 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa240031890 con 0x7fa2500d22b0 2024-03-06T07:22:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.969+0000 7fa24effd640 1 -- 172.21.15.59:0/4205436229 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240031c40 con 0x7fa2500d22b0 2024-03-06T07:22:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.969+0000 7fa2562cf640 1 -- 172.21.15.59:0/4205436229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2500d22b0 msgr2=0x7fa2500d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.969+0000 7fa2562cf640 1 --2- 172.21.15.59:0/4205436229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2500d22b0 0x7fa2500d2710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa2400097d0 tx=0x7fa24002b930 comp rx=0 tx=0).stop 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.969+0000 7fa2562cf640 1 -- 172.21.15.59:0/4205436229 shutdown_connections 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.969+0000 7fa2562cf640 1 --2- 172.21.15.59:0/4205436229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2500d22b0 0x7fa2500d2710 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.969+0000 7fa2562cf640 1 -- 172.21.15.59:0/4205436229 >> 172.21.15.59:0/4205436229 conn(0x7fa2500d3cd0 msgr2=0x7fa2500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.969+0000 7fa2562cf640 1 -- 172.21.15.59:0/4205436229 shutdown_connections 2024-03-06T07:22:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.969+0000 7fa2562cf640 1 -- 172.21.15.59:0/4205436229 wait complete. 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.970+0000 7fa2562cf640 1 Processor -- start 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa2562cf640 1 -- start start 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa2562cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2500b41f0 0x7fa25015ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa2562cf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500d22b0 0x7fa25014c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa2562cf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa25015dfe0 0x7fa25015fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa2562cf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2500b3d60 con 0x7fa2500d22b0 2024-03-06T07:22:11.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa2562cf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2500b31e0 con 0x7fa25015dfe0 2024-03-06T07:22:11.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa2562cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2500d1c70 con 0x7fa2500b41f0 2024-03-06T07:22:11.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa24f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500d22b0 0x7fa25014c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa24f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500d22b0 0x7fa25014c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42904/0 (socket says 172.21.15.59:42904) 2024-03-06T07:22:11.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa24f7fe640 1 -- 172.21.15.59:0/3114131986 learned_addr learned my addr 172.21.15.59:0/3114131986 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:11.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa24ffff640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2500b41f0 0x7fa25015ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.971+0000 7fa25486c640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa25015dfe0 0x7fa25015fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.972+0000 7fa24f7fe640 1 -- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2500b41f0 msgr2=0x7fa25015ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.972+0000 7fa24f7fe640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2500b41f0 0x7fa25015ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.972+0000 7fa24f7fe640 1 -- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa25015dfe0 msgr2=0x7fa25015fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.972+0000 7fa24f7fe640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa25015dfe0 0x7fa25015fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.972+0000 7fa24f7fe640 1 -- 172.21.15.59:0/3114131986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2501607d0 con 0x7fa2500d22b0 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.972+0000 7fa24f7fe640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500d22b0 0x7fa25014c080 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa240000c00 tx=0x7fa2400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.972+0000 7fa24cff9640 1 -- 172.21.15.59:0/3114131986 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240035040 con 0x7fa2500d22b0 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.972+0000 7fa24cff9640 1 -- 172.21.15.59:0/3114131986 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa240030d20 con 0x7fa2500d22b0 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.972+0000 7fa2562cf640 1 -- 172.21.15.59:0/3114131986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa250160ab0 con 0x7fa2500d22b0 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.972+0000 7fa24cff9640 1 -- 172.21.15.59:0/3114131986 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa240006e10 con 0x7fa2500d22b0 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.972+0000 7fa2562cf640 1 -- 172.21.15.59:0/3114131986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25015e740 con 0x7fa2500d22b0 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.973+0000 7fa2562cf640 1 -- 172.21.15.59:0/3114131986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25004f330 con 0x7fa2500d22b0 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.973+0000 7fa24cff9640 1 -- 172.21.15.59:0/3114131986 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa240038020 con 0x7fa2500d22b0 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.974+0000 7fa24cff9640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b4d0 0x7fa23c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.974+0000 7fa24cff9640 1 -- 172.21.15.59:0/3114131986 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2400ac6b0 con 0x7fa2500d22b0 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.974+0000 7fa24ffff640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b4d0 0x7fa23c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.974+0000 7fa24ffff640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b4d0 0x7fa23c06d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa244007fd0 tx=0x7fa244009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:11.978+0000 7fa24cff9640 1 -- 172.21.15.59:0/3114131986 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa24002d040 con 0x7fa2500d22b0 2024-03-06T07:22:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.200+0000 7fa2562cf640 1 -- 172.21.15.59:0/3114131986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa2500ceae0 con 0x7fa2500d22b0 2024-03-06T07:22:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.201+0000 7fa24cff9640 1 -- 172.21.15.59:0/3114131986 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa24007aee0 con 0x7fa2500d22b0 2024-03-06T07:22:12.203 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 -- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b4d0 msgr2=0x7fa23c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b4d0 0x7fa23c06d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa244007fd0 tx=0x7fa244009040 comp rx=0 tx=0).stop 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 -- 172.21.15.59:0/3114131986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500d22b0 msgr2=0x7fa25014c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500d22b0 0x7fa25014c080 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa240000c00 tx=0x7fa2400046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 -- 172.21.15.59:0/3114131986 shutdown_connections 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b4d0 0x7fa23c06d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa25015dfe0 0x7fa25015fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500d22b0 0x7fa25014c080 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 --2- 172.21.15.59:0/3114131986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2500b41f0 0x7fa25015ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 -- 172.21.15.59:0/3114131986 >> 172.21.15.59:0/3114131986 conn(0x7fa2500d3cd0 msgr2=0x7fa25006ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 -- 172.21.15.59:0/3114131986 shutdown_connections 2024-03-06T07:22:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:12.203+0000 7fa2562cf640 1 -- 172.21.15.59:0/3114131986 wait complete. 2024-03-06T07:22:12.215 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:12.215 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:12.215 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi164.front.sepia.ceph.com 2024-03-06T07:22:12.215 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:22:12.215 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:12.215 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:12.215 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:12.215 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:12.215 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:22:12.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:12.216 DEBUG:teuthology.orchestra.run.smithi164:> ip addr 2024-03-06T07:22:12.231 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:12.231 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:12.231 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:12.231 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f0 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f1 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout: inet 172.21.15.164/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:12.232 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft 42674sec preferred_lft 42674sec 2024-03-06T07:22:12.233 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::ec4:7aff:fe88:73f3/64 scope link 2024-03-06T07:22:12.233 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:12.233 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:22:12.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:12.233 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec bash -c ' 2024-03-06T07:22:12.233 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-03-06T07:22:12.233 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:22:12.233 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip addr flush dev ceph-brx 2024-03-06T07:22:12.233 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set ceph-brx up 2024-03-06T07:22:12.233 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:22:12.233 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-03-06T07:22:12.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:12.486 DEBUG:teuthology.orchestra.run.smithi164:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:12.514 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-03-06T07:22:12.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:12.516 DEBUG:teuthology.orchestra.run.smithi164:> route 2024-03-06T07:22:12.588 INFO:teuthology.orchestra.run.smithi164.stdout:Kernel IP routing table 2024-03-06T07:22:12.588 INFO:teuthology.orchestra.run.smithi164.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:12.588 INFO:teuthology.orchestra.run.smithi164.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:12.588 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:12.588 INFO:teuthology.orchestra.run.smithi164.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:12.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:12.589 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec bash -c ' 2024-03-06T07:22:12.589 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-03-06T07:22:12.589 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:12.589 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:12.589 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:12.589 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-03-06T07:22:12.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:12.862 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-03-06T07:22:12.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:12.877 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list-id 2024-03-06T07:22:12.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:12.933 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec bash -c ' 2024-03-06T07:22:12.933 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-03-06T07:22:12.933 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:12.933 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:22:12.933 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-03-06T07:22:13.020 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:22:13.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.021 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec bash -c ' 2024-03-06T07:22:13.021 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-03-06T07:22:13.021 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:22:13.021 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:13.021 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:22:13.021 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:22:13.021 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:22:13.021 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-03-06T07:22:13.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.140 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec bash -c ' 2024-03-06T07:22:13.141 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-03-06T07:22:13.141 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set brx.0 up 2024-03-06T07:22:13.141 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:22:13.141 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-03-06T07:22:13.185 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-06T07:22:13.186 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:13.186 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:13.199 INFO:teuthology.orchestra.run.smithi164.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:22:13.200 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:13.200 DEBUG:teuthology.orchestra.run.smithi164:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:13.254 DEBUG:teuthology.orchestra.run.smithi164:> sudo modprobe fuse 2024-03-06T07:22:13.321 DEBUG:teuthology.orchestra.run.smithi164:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:13.377 INFO:teuthology.orchestra.run.smithi164.stdout:/proc 2024-03-06T07:22:13.377 INFO:teuthology.orchestra.run.smithi164.stdout:/sys 2024-03-06T07:22:13.377 INFO:teuthology.orchestra.run.smithi164.stdout:/dev 2024-03-06T07:22:13.377 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/security 2024-03-06T07:22:13.377 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/shm 2024-03-06T07:22:13.377 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/pts 2024-03-06T07:22:13.377 INFO:teuthology.orchestra.run.smithi164.stdout:/run 2024-03-06T07:22:13.377 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/cgroup 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/pstore 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/bpf 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/ 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/selinux 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/mqueue 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/hugepages 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/debug 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/tracing 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/config 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi164.stdout:/run/user/1000 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:13.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.379 DEBUG:teuthology.orchestra.run.smithi164:> ls /sys/fs/fuse/connections 2024-03-06T07:22:13.432 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:13.433 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-06T07:22:13.474 DEBUG:teuthology.orchestra.run.smithi164:> sudo modprobe fuse 2024-03-06T07:22:13.504 DEBUG:teuthology.orchestra.run.smithi164:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/proc 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/sys 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/dev 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/security 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/shm 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/pts 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/run 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/cgroup 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/pstore 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/bpf 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/ 2024-03-06T07:22:13.559 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/selinux 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/mqueue 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/hugepages 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/debug 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/tracing 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/config 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/run/user/1000 2024-03-06T07:22:13.560 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph 2024-03-06T07:22:13.561 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:13.561 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:13.561 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:13.561 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:13.561 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns 2024-03-06T07:22:13.561 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:13.561 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:13.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.561 DEBUG:teuthology.orchestra.run.smithi164:> ls /sys/fs/fuse/connections 2024-03-06T07:22:13.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi164.stderr:2024-03-06T07:22:13.568+0000 7f94de639480 -1 init, newargv = 0x55f840ddc780 newargc=15 2024-03-06T07:22:13.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi164.stderr:ceph-fuse[44696]: starting ceph client 2024-03-06T07:22:13.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi164.stderr:ceph-fuse[44696]: starting fuse 2024-03-06T07:22:13.586 INFO:teuthology.orchestra.run.smithi164.stdout:41 2024-03-06T07:22:13.587 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:22:13.587 DEBUG:teuthology.orchestra.run.smithi164:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:13.587 DEBUG:teuthology.orchestra.run.smithi164:> import glob 2024-03-06T07:22:13.587 DEBUG:teuthology.orchestra.run.smithi164:> import re 2024-03-06T07:22:13.587 DEBUG:teuthology.orchestra.run.smithi164:> import os 2024-03-06T07:22:13.587 DEBUG:teuthology.orchestra.run.smithi164:> import subprocess 2024-03-06T07:22:13.587 DEBUG:teuthology.orchestra.run.smithi164:> 2024-03-06T07:22:13.587 DEBUG:teuthology.orchestra.run.smithi164:> def _find_admin_socket(client_name): 2024-03-06T07:22:13.587 DEBUG:teuthology.orchestra.run.smithi164:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-06T07:22:13.587 DEBUG:teuthology.orchestra.run.smithi164:> files = glob.glob(asok_path) 2024-03-06T07:22:13.587 DEBUG:teuthology.orchestra.run.smithi164:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> # Given a non-glob path, it better be there 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> if "*" not in asok_path: 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> assert(len(files) == 1) 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> return files[0] 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> for f in files: 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> contents = proc_f.read() 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> if mountpoint in contents: 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> return f 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:13.588 DEBUG:teuthology.orchestra.run.smithi164:> 2024-03-06T07:22:13.589 DEBUG:teuthology.orchestra.run.smithi164:> print(_find_admin_socket("client.1")) 2024-03-06T07:22:13.589 DEBUG:teuthology.orchestra.run.smithi164:> ' 2024-03-06T07:22:13.708 INFO:teuthology.orchestra.run.smithi164.stdout:/var/run/ceph/ceph-client.1.44696.asok 2024-03-06T07:22:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:22:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:13.718 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44696.asok 2024-03-06T07:22:13.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:13.719 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44696.asok status 2024-03-06T07:22:13.857 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-06T07:22:13.857 INFO:teuthology.orchestra.run.smithi164.stdout: "metadata": { 2024-03-06T07:22:13.857 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:13.857 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:13.857 INFO:teuthology.orchestra.run.smithi164.stdout: "entity_id": "1", 2024-03-06T07:22:13.857 INFO:teuthology.orchestra.run.smithi164.stdout: "hostname": "smithi164", 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "pid": "44696", 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "root": "/" 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "dentry_count": 0, 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "id": 4690, 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "inst": { 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "name": { 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "client", 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4690 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": { 2024-03-06T07:22:13.858 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 2874078236 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": { 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 2874078236 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "inst_str": "client.4690 192.168.0.1:0/2874078236", 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "addr_str": "192.168.0.1:0/2874078236", 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "inode_count": 1, 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_epoch": 6, 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_epoch": 20, 2024-03-06T07:22:13.859 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:13.860 INFO:teuthology.orchestra.run.smithi164.stdout: "blocklisted": false, 2024-03-06T07:22:13.860 INFO:teuthology.orchestra.run.smithi164.stdout: "fs_name": "cephfs" 2024-03-06T07:22:13.860 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-06T07:22:13.866 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:13.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.978+0000 7f3426e88640 1 Processor -- start 2024-03-06T07:22:13.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.978+0000 7f3426e88640 1 -- start start 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3426e88640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f34200cf850 0x7f34200cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3426e88640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f342006cf20 0x7f342006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3426e88640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f342015e380 con 0x7f34200d00c0 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3426e88640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f342006ca90 con 0x7f342006cf20 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3426e88640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34200cdc10 con 0x7f34200cf850 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3425425640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f342006cf20 0x7f342006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3425425640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f342006cf20 0x7f342006d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42708/0 (socket says 172.21.15.59:42708) 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3425425640 1 -- 172.21.15.59:0/1284166303 learned_addr learned my addr 172.21.15.59:0/1284166303 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f341ffff640 1 --2- 172.21.15.59:0/1284166303 >> v2:172.21.15.164:3300/0 conn(0x7f34200cf850 0x7f34200cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3425425640 1 -- 172.21.15.59:0/1284166303 >> v2:172.21.15.164:3300/0 conn(0x7f34200cf850 msgr2=0x7f34200cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3425425640 1 --2- 172.21.15.59:0/1284166303 >> v2:172.21.15.164:3300/0 conn(0x7f34200cf850 0x7f34200cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3425425640 1 -- 172.21.15.59:0/1284166303 >> v1:172.21.15.164:6789/0 conn(0x7f34200d00c0 legacy=0x7f34200d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.979+0000 7f3425425640 1 -- 172.21.15.59:0/1284166303 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342006c160 con 0x7f342006cf20 2024-03-06T07:22:13.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.980+0000 7f3425425640 1 --2- 172.21.15.59:0/1284166303 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f342006cf20 0x7f342006d3b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3420071de0 tx=0x7f341802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebbd5a8e0a39bce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.980+0000 7f341f7fe640 1 -- 172.21.15.59:0/1284166303 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341802cb20 con 0x7f342006cf20 2024-03-06T07:22:13.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.980+0000 7f341f7fe640 1 -- 172.21.15.59:0/1284166303 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341802ccc0 con 0x7f342006cf20 2024-03-06T07:22:13.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.980+0000 7f341f7fe640 1 -- 172.21.15.59:0/1284166303 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418034610 con 0x7f342006cf20 2024-03-06T07:22:13.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.981+0000 7f3426e88640 1 -- 172.21.15.59:0/1284166303 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f342006cf20 msgr2=0x7f342006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.981+0000 7f3426e88640 1 --2- 172.21.15.59:0/1284166303 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f342006cf20 0x7f342006d3b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3420071de0 tx=0x7f341802bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:13.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.981+0000 7f3426e88640 1 -- 172.21.15.59:0/1284166303 shutdown_connections 2024-03-06T07:22:13.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.981+0000 7f3426e88640 1 --2- 172.21.15.59:0/1284166303 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f342006cf20 0x7f342006d3b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.981+0000 7f3426e88640 1 --2- 172.21.15.59:0/1284166303 >> v2:172.21.15.164:3300/0 conn(0x7f34200cf850 0x7f34200cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.981+0000 7f3426e88640 1 -- 172.21.15.59:0/1284166303 >> 172.21.15.59:0/1284166303 conn(0x7f34200b35a0 msgr2=0x7f34200b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.981+0000 7f3426e88640 1 -- 172.21.15.59:0/1284166303 shutdown_connections 2024-03-06T07:22:13.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.981+0000 7f3426e88640 1 -- 172.21.15.59:0/1284166303 wait complete. 2024-03-06T07:22:13.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.982+0000 7f3426e88640 1 Processor -- start 2024-03-06T07:22:13.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.982+0000 7f3426e88640 1 -- start start 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f3426e88640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f342006cf20 0x7f34200c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f3426e88640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f3426e88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200d00c0 0x7f3420159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f3426e88640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34200cd900 con 0x7f34200cf850 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f3426e88640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342006c720 con 0x7f342006cf20 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f3426e88640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34200b1030 con 0x7f34200d00c0 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f341ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f341ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42712/0 (socket says 172.21.15.59:42712) 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f341ffff640 1 -- 172.21.15.59:0/660244876 learned_addr learned my addr 172.21.15.59:0/660244876 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f3424c24640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f342006cf20 0x7f34200c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f3425425640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200d00c0 0x7f3420159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f341ffff640 1 -- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200d00c0 msgr2=0x7f3420159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f341ffff640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200d00c0 0x7f3420159bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f341ffff640 1 -- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f342006cf20 msgr2=0x7f34200c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f341ffff640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f342006cf20 0x7f34200c8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f341ffff640 1 -- 172.21.15.59:0/660244876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34200c9440 con 0x7f34200cf850 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.983+0000 7f341ffff640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200c6000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f340c00e9e0 tx=0x7f340c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.984+0000 7f341d7fa640 1 -- 172.21.15.59:0/660244876 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c017070 con 0x7f34200cf850 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.984+0000 7f341d7fa640 1 -- 172.21.15.59:0/660244876 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c00c9d0 con 0x7f34200cf850 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.984+0000 7f3426e88640 1 -- 172.21.15.59:0/660244876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34200c9720 con 0x7f34200cf850 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.984+0000 7f341d7fa640 1 -- 172.21.15.59:0/660244876 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c00cdc0 con 0x7f34200cf850 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.984+0000 7f3426e88640 1 -- 172.21.15.59:0/660244876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34200c9ce0 con 0x7f34200cf850 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.985+0000 7f3426e88640 1 -- 172.21.15.59:0/660244876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342004f330 con 0x7f34200cf850 2024-03-06T07:22:13.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.985+0000 7f341d7fa640 1 -- 172.21.15.59:0/660244876 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f340c01a070 con 0x7f34200cf850 2024-03-06T07:22:13.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.985+0000 7f341d7fa640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340406b4d0 0x7f340406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.986+0000 7f341d7fa640 1 -- 172.21.15.59:0/660244876 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f340c08a190 con 0x7f34200cf850 2024-03-06T07:22:13.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.986+0000 7f3424c24640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340406b4d0 0x7f340406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.986+0000 7f3424c24640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340406b4d0 0x7f340406d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f34140016e0 tx=0x7f3414009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:13.990+0000 7f341d7fa640 1 -- 172.21.15.59:0/660244876 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f340c0589c0 con 0x7f34200cf850 2024-03-06T07:22:14.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.209+0000 7f3426e88640 1 -- 172.21.15.59:0/660244876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f34200ca0e0 con 0x7f34200cf850 2024-03-06T07:22:14.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.210+0000 7f341d7fa640 1 -- 172.21.15.59:0/660244876 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f340c05ca60 con 0x7f34200cf850 2024-03-06T07:22:14.212 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:14.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 -- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340406b4d0 msgr2=0x7f340406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340406b4d0 0x7f340406d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f34140016e0 tx=0x7f3414009040 comp rx=0 tx=0).stop 2024-03-06T07:22:14.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 -- 172.21.15.59:0/660244876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 msgr2=0x7f34200c6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200c6000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f340c00e9e0 tx=0x7f340c00eea0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 -- 172.21.15.59:0/660244876 shutdown_connections 2024-03-06T07:22:14.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340406b4d0 0x7f340406d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200d00c0 0x7f3420159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200c6000 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 --2- 172.21.15.59:0/660244876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f342006cf20 0x7f34200c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 -- 172.21.15.59:0/660244876 >> 172.21.15.59:0/660244876 conn(0x7f34200b35a0 msgr2=0x7f342006d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 -- 172.21.15.59:0/660244876 shutdown_connections 2024-03-06T07:22:14.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:14.212+0000 7f3426e88640 1 -- 172.21.15.59:0/660244876 wait complete. 2024-03-06T07:22:14.224 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:14.224 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:14.224 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi164.front.sepia.ceph.com 2024-03-06T07:22:14.224 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-06T07:22:14.224 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:14.225 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:14.225 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:14.225 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:14.225 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:22:14.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.225 DEBUG:teuthology.orchestra.run.smithi164:> ip addr 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f0 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:d9:34:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:14.241 INFO:teuthology.orchestra.run.smithi164.stdout: altname enp7s0f1 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f2 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 0c:c4:7a:88:73:f3 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: inet 172.21.15.164/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft 42672sec preferred_lft 42672sec 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::ec4:7aff:fe88:73f3/64 scope link 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 2a:e2:22:1a:ec:5b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::608e:33ff:fe34:546a/64 scope link 2024-03-06T07:22:14.243 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.243 INFO:teuthology.orchestra.run.smithi164.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:14.243 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 2a:e2:22:1a:ec:5b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:14.243 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::28e2:22ff:fe1a:ec5b/64 scope link 2024-03-06T07:22:14.243 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.243 DEBUG:teuthology.orchestra.run.smithi164:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:14.306 INFO:teuthology.orchestra.run.smithi164.stdout:1 2024-03-06T07:22:14.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.308 DEBUG:teuthology.orchestra.run.smithi164:> route 2024-03-06T07:22:14.365 INFO:teuthology.orchestra.run.smithi164.stdout:Kernel IP routing table 2024-03-06T07:22:14.365 INFO:teuthology.orchestra.run.smithi164.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:14.365 INFO:teuthology.orchestra.run.smithi164.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:14.365 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:14.365 INFO:teuthology.orchestra.run.smithi164.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:14.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.366 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec bash -c ' 2024-03-06T07:22:14.366 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-03-06T07:22:14.366 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:14.366 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:14.366 DEBUG:teuthology.orchestra.run.smithi164:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:14.366 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-03-06T07:22:14.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.460 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-03-06T07:22:14.476 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:22:14.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.476 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list-id 2024-03-06T07:22:14.530 INFO:teuthology.orchestra.run.smithi164.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-06T07:22:14.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.531 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec bash -c ' 2024-03-06T07:22:14.531 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-03-06T07:22:14.531 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:14.531 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-06T07:22:14.531 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-03-06T07:22:14.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.618 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:22:14.644 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:14.644 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:14.644 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:14.644 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.644 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:14.644 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.645 INFO:teuthology.orchestra.run.smithi164.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:14.645 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5e:f7:d5:59:49:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:14.645 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:14.645 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.645 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::5cf7:d5ff:fe59:4937/64 scope link tentative 2024-03-06T07:22:14.645 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.646 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:22:14.713 INFO:teuthology.orchestra.run.smithi164.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:14.713 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.714 INFO:teuthology.orchestra.run.smithi164.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:14.714 INFO:teuthology.orchestra.run.smithi164.stdout: link/ether 5e:f7:d5:59:49:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:14.714 INFO:teuthology.orchestra.run.smithi164.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:14.714 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.714 INFO:teuthology.orchestra.run.smithi164.stdout: inet6 fe80::5cf7:d5ff:fe59:4937/64 scope link tentative 2024-03-06T07:22:14.714 INFO:teuthology.orchestra.run.smithi164.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:14.715 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.715 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec bash -c ' 2024-03-06T07:22:14.715 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-03-06T07:22:14.715 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-06T07:22:14.715 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:14.715 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-06T07:22:14.715 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-06T07:22:14.716 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-06T07:22:14.716 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-03-06T07:22:14.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:14.846 DEBUG:teuthology.orchestra.run.smithi164:> (cd / && exec bash -c ' 2024-03-06T07:22:14.846 DEBUG:teuthology.orchestra.run.smithi164:> set -e 2024-03-06T07:22:14.846 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set brx.1 up 2024-03-06T07:22:14.846 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:22:14.846 DEBUG:teuthology.orchestra.run.smithi164:> ') 2024-03-06T07:22:14.890 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-06T07:22:14.890 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:14.890 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:14.945 INFO:teuthology.orchestra.run.smithi164.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:22:14.945 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:14.945 DEBUG:teuthology.orchestra.run.smithi164:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:14.998 DEBUG:teuthology.orchestra.run.smithi164:> sudo modprobe fuse 2024-03-06T07:22:15.065 DEBUG:teuthology.orchestra.run.smithi164:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:15.121 INFO:teuthology.orchestra.run.smithi164.stdout:/proc 2024-03-06T07:22:15.121 INFO:teuthology.orchestra.run.smithi164.stdout:/sys 2024-03-06T07:22:15.121 INFO:teuthology.orchestra.run.smithi164.stdout:/dev 2024-03-06T07:22:15.121 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/security 2024-03-06T07:22:15.121 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/shm 2024-03-06T07:22:15.121 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/pts 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/run 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/cgroup 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/pstore 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/bpf 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/ 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/selinux 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/mqueue 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/hugepages 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/debug 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/tracing 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:15.122 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/config 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/run/user/1000 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:15.123 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:15.124 INFO:teuthology.orchestra.run.smithi164.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:15.124 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:15.124 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:15.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.124 DEBUG:teuthology.orchestra.run.smithi164:> ls /sys/fs/fuse/connections 2024-03-06T07:22:15.177 INFO:teuthology.orchestra.run.smithi164.stdout:41 2024-03-06T07:22:15.177 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-06T07:22:15.177 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-06T07:22:15.220 DEBUG:teuthology.orchestra.run.smithi164:> sudo modprobe fuse 2024-03-06T07:22:15.247 DEBUG:teuthology.orchestra.run.smithi164:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:15.304 INFO:teuthology.orchestra.run.smithi164.stdout:/proc 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/sys 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/dev 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/security 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/shm 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/pts 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/run 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/cgroup 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/pstore 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/bpf 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/ 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/selinux 2024-03-06T07:22:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/mqueue 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/dev/hugepages 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/debug 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/tracing 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/sys/kernel/config 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/run/user/1000 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns 2024-03-06T07:22:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:15.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.307 DEBUG:teuthology.orchestra.run.smithi164:> ls /sys/fs/fuse/connections 2024-03-06T07:22:15.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi164.stderr:2024-03-06T07:22:15.305+0000 7f004ce45480 -1 init, newargv = 0x55e13295fa40 newargc=15 2024-03-06T07:22:15.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi164.stderr:ceph-fuse[45149]: starting ceph client 2024-03-06T07:22:15.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi164.stderr:ceph-fuse[45149]: starting fuse 2024-03-06T07:22:15.323 INFO:teuthology.orchestra.run.smithi164.stdout:41 2024-03-06T07:22:15.323 INFO:teuthology.orchestra.run.smithi164.stdout:44 2024-03-06T07:22:15.323 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-06T07:22:15.323 DEBUG:teuthology.orchestra.run.smithi164:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> import glob 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> import re 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> import os 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> import subprocess 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> def _find_admin_socket(client_name): 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> files = glob.glob(asok_path) 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> # Given a non-glob path, it better be there 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> if "*" not in asok_path: 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> assert(len(files) == 1) 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> return files[0] 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> 2024-03-06T07:22:15.324 DEBUG:teuthology.orchestra.run.smithi164:> for f in files: 2024-03-06T07:22:15.325 DEBUG:teuthology.orchestra.run.smithi164:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:15.325 DEBUG:teuthology.orchestra.run.smithi164:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:15.325 DEBUG:teuthology.orchestra.run.smithi164:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:15.325 DEBUG:teuthology.orchestra.run.smithi164:> contents = proc_f.read() 2024-03-06T07:22:15.325 DEBUG:teuthology.orchestra.run.smithi164:> if mountpoint in contents: 2024-03-06T07:22:15.325 DEBUG:teuthology.orchestra.run.smithi164:> return f 2024-03-06T07:22:15.325 DEBUG:teuthology.orchestra.run.smithi164:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:15.325 DEBUG:teuthology.orchestra.run.smithi164:> 2024-03-06T07:22:15.325 DEBUG:teuthology.orchestra.run.smithi164:> print(_find_admin_socket("client.3")) 2024-03-06T07:22:15.325 DEBUG:teuthology.orchestra.run.smithi164:> ' 2024-03-06T07:22:15.444 INFO:teuthology.orchestra.run.smithi164.stdout:/var/run/ceph/ceph-client.3.45149.asok 2024-03-06T07:22:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-03-06T07:22:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:15.455 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.45149.asok 2024-03-06T07:22:15.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.455 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.45149.asok status 2024-03-06T07:22:15.593 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-06T07:22:15.593 INFO:teuthology.orchestra.run.smithi164.stdout: "metadata": { 2024-03-06T07:22:15.593 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "entity_id": "3", 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "hostname": "smithi164", 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "pid": "45149", 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "root": "/" 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "dentry_count": 0, 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "id": 4705, 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "inst": { 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "name": { 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "client", 2024-03-06T07:22:15.594 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4705 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": { 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 490679778 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": { 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 490679778 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: "inst_str": "client.4705 192.168.0.2:0/490679778", 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: "addr_str": "192.168.0.2:0/490679778", 2024-03-06T07:22:15.595 INFO:teuthology.orchestra.run.smithi164.stdout: "inode_count": 1, 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_epoch": 6, 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_epoch": 20, 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi164.stdout: "blocklisted": false, 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi164.stdout: "fs_name": "cephfs" 2024-03-06T07:22:15.596 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-06T07:22:15.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.602 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:22:15.602 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:15.621 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:22:15.621 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:15.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.621 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:15.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.695 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:22:15.695 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:15.754 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:22:15.754 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:15.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.754 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:15.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.827 DEBUG:teuthology.orchestra.run.smithi164:> stat --file-system '--printf=%T 2024-03-06T07:22:15.828 DEBUG:teuthology.orchestra.run.smithi164:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:15.843 INFO:teuthology.orchestra.run.smithi164.stdout:fuseblk 2024-03-06T07:22:15.843 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:15.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.843 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:15.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.916 DEBUG:teuthology.orchestra.run.smithi164:> stat --file-system '--printf=%T 2024-03-06T07:22:15.916 DEBUG:teuthology.orchestra.run.smithi164:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:15.971 INFO:teuthology.orchestra.run.smithi164.stdout:fuseblk 2024-03-06T07:22:15.972 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:15.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:15.972 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:16.039 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:22:16.054 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.166+0000 7f6a3f08f640 1 Processor -- start 2024-03-06T07:22:16.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.166+0000 7f6a3f08f640 1 -- start start 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.167+0000 7f6a3f08f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a380d22b0 0x7f6a380d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.167+0000 7f6a3f08f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a380b3430 con 0x7f6a3815dfe0 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.167+0000 7f6a3f08f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a380b3710 con 0x7f6a380b41f0 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.167+0000 7f6a3f08f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380d2c50 con 0x7f6a380d22b0 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.167+0000 7f6a3ce2b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6a3815dfe0 0x7f6a380cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41886/0 (socket says 172.21.15.59:41886) 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.167+0000 7f6a3ce2b640 1 -- 172.21.15.59:0/1485726723 learned_addr learned my addr 172.21.15.59:0/1485726723 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.167+0000 7f6a37fff640 1 --2- 172.21.15.59:0/1485726723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a380d22b0 0x7f6a380d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.168+0000 7f6a377fe640 1 -- 172.21.15.59:0/1485726723 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521563784 0 0) 0x7f6a380b3430 con 0x7f6a3815dfe0 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.168+0000 7f6a377fe640 1 -- 172.21.15.59:0/1485726723 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a20003520 con 0x7f6a3815dfe0 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.168+0000 7f6a377fe640 1 -- 172.21.15.59:0/1485726723 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578771658 0 0) 0x7f6a380b3710 con 0x7f6a380b41f0 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.168+0000 7f6a377fe640 1 -- 172.21.15.59:0/1485726723 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a380b3430 con 0x7f6a380b41f0 2024-03-06T07:22:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.168+0000 7f6a37fff640 1 -- 172.21.15.59:0/1485726723 >> v1:172.21.15.164:6789/0 conn(0x7f6a380b41f0 legacy=0x7f6a380b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.168+0000 7f6a37fff640 1 -- 172.21.15.59:0/1485726723 >> v1:172.21.15.59:6789/0 conn(0x7f6a3815dfe0 legacy=0x7f6a380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.168+0000 7f6a37fff640 1 -- 172.21.15.59:0/1485726723 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a380b39f0 con 0x7f6a380d22b0 2024-03-06T07:22:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.169+0000 7f6a37fff640 1 --2- 172.21.15.59:0/1485726723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a380d22b0 0x7f6a380d2710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0097d0 tx=0x7f6a2c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d565e23cd245436 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.169+0000 7f6a377fe640 1 -- 172.21.15.59:0/1485726723 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c035020 con 0x7f6a380d22b0 2024-03-06T07:22:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.169+0000 7f6a377fe640 1 -- 172.21.15.59:0/1485726723 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2c031890 con 0x7f6a380d22b0 2024-03-06T07:22:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.169+0000 7f6a377fe640 1 -- 172.21.15.59:0/1485726723 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c031c60 con 0x7f6a380d22b0 2024-03-06T07:22:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.169+0000 7f6a3f08f640 1 -- 172.21.15.59:0/1485726723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a380d22b0 msgr2=0x7f6a380d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.169+0000 7f6a3f08f640 1 --2- 172.21.15.59:0/1485726723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a380d22b0 0x7f6a380d2710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0097d0 tx=0x7f6a2c02b930 comp rx=0 tx=0).stop 2024-03-06T07:22:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.169+0000 7f6a3f08f640 1 -- 172.21.15.59:0/1485726723 shutdown_connections 2024-03-06T07:22:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.169+0000 7f6a3f08f640 1 --2- 172.21.15.59:0/1485726723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a380d22b0 0x7f6a380d2710 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.169+0000 7f6a3f08f640 1 -- 172.21.15.59:0/1485726723 >> 172.21.15.59:0/1485726723 conn(0x7f6a380d3cd0 msgr2=0x7f6a380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.170+0000 7f6a3f08f640 1 -- 172.21.15.59:0/1485726723 shutdown_connections 2024-03-06T07:22:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.170+0000 7f6a3f08f640 1 -- 172.21.15.59:0/1485726723 wait complete. 2024-03-06T07:22:16.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.171+0000 7f6a3f08f640 1 Processor -- start 2024-03-06T07:22:16.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.171+0000 7f6a3f08f640 1 -- start start 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a3f08f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a380b41f0 0x7f6a380cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a3f08f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a380d22b0 0x7f6a380b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a3f08f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a3815dfe0 0x7f6a380b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a3f08f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380b3d60 con 0x7f6a380d22b0 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a3f08f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380b31e0 con 0x7f6a3815dfe0 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a3f08f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a380d1c70 con 0x7f6a380b41f0 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a380d22b0 0x7f6a380b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a380d22b0 0x7f6a380b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42720/0 (socket says 172.21.15.59:42720) 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a3ce2b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a380b41f0 0x7f6a380cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a37fff640 1 -- 172.21.15.59:0/3634731940 learned_addr learned my addr 172.21.15.59:0/3634731940 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a3d62c640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a3815dfe0 0x7f6a380b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a37fff640 1 -- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a380b41f0 msgr2=0x7f6a380cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a37fff640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a380b41f0 0x7f6a380cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a37fff640 1 -- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a3815dfe0 msgr2=0x7f6a380b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a37fff640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a3815dfe0 0x7f6a380b6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a37fff640 1 -- 172.21.15.59:0/3634731940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a380b86a0 con 0x7f6a380d22b0 2024-03-06T07:22:16.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a37fff640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a380d22b0 0x7f6a380b68a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0098f0 tx=0x7f6a2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a357fa640 1 -- 172.21.15.59:0/3634731940 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c035040 con 0x7f6a380d22b0 2024-03-06T07:22:16.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a357fa640 1 -- 172.21.15.59:0/3634731940 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2c030d20 con 0x7f6a380d22b0 2024-03-06T07:22:16.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a3f08f640 1 -- 172.21.15.59:0/3634731940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a380b8940 con 0x7f6a380d22b0 2024-03-06T07:22:16.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a357fa640 1 -- 172.21.15.59:0/3634731940 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c006e10 con 0x7f6a380d22b0 2024-03-06T07:22:16.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.172+0000 7f6a3f08f640 1 -- 172.21.15.59:0/3634731940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a380b8f00 con 0x7f6a380d22b0 2024-03-06T07:22:16.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.174+0000 7f6a3f08f640 1 -- 172.21.15.59:0/3634731940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3804f330 con 0x7f6a380d22b0 2024-03-06T07:22:16.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.174+0000 7f6a357fa640 1 -- 172.21.15.59:0/3634731940 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a2c038020 con 0x7f6a380d22b0 2024-03-06T07:22:16.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.174+0000 7f6a357fa640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a2406b4d0 0x7f6a2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.175+0000 7f6a357fa640 1 -- 172.21.15.59:0/3634731940 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a2c0ac700 con 0x7f6a380d22b0 2024-03-06T07:22:16.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.175+0000 7f6a3ce2b640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a2406b4d0 0x7f6a2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.175+0000 7f6a3ce2b640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a2406b4d0 0x7f6a2406d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6a300012e0 tx=0x7f6a30001a30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.179+0000 7f6a357fa640 1 -- 172.21.15.59:0/3634731940 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a2c02d040 con 0x7f6a380d22b0 2024-03-06T07:22:16.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.363+0000 7f6a3f08f640 1 -- 172.21.15.59:0/3634731940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a3815f160 con 0x7f6a380d22b0 2024-03-06T07:22:16.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.364+0000 7f6a357fa640 1 -- 172.21.15.59:0/3634731940 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f6a2c07af30 con 0x7f6a380d22b0 2024-03-06T07:22:16.366 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:16.366 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 -- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a2406b4d0 msgr2=0x7f6a2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a2406b4d0 0x7f6a2406d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6a300012e0 tx=0x7f6a30001a30 comp rx=0 tx=0).stop 2024-03-06T07:22:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 -- 172.21.15.59:0/3634731940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a380d22b0 msgr2=0x7f6a380b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a380d22b0 0x7f6a380b68a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0098f0 tx=0x7f6a2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 -- 172.21.15.59:0/3634731940 shutdown_connections 2024-03-06T07:22:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a2406b4d0 0x7f6a2406d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a3815dfe0 0x7f6a380b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a380d22b0 0x7f6a380b68a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 --2- 172.21.15.59:0/3634731940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a380b41f0 0x7f6a380cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 -- 172.21.15.59:0/3634731940 >> 172.21.15.59:0/3634731940 conn(0x7f6a380d3cd0 msgr2=0x7f6a380cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 -- 172.21.15.59:0/3634731940 shutdown_connections 2024-03-06T07:22:16.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.366+0000 7f6a3f08f640 1 -- 172.21.15.59:0/3634731940 wait complete. 2024-03-06T07:22:16.379 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:16.379 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:16.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.482+0000 7ff2dede9640 1 Processor -- start 2024-03-06T07:22:16.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.482+0000 7ff2dede9640 1 -- start start 2024-03-06T07:22:16.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.483+0000 7ff2dede9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff2d80cb1b0 0x7ff2d8066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.483+0000 7ff2dede9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d806afb0 con 0x7ff2d81616d0 2024-03-06T07:22:16.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.483+0000 7ff2dede9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d80b5090 con 0x7ff2d8161ab0 2024-03-06T07:22:16.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.483+0000 7ff2dede9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d80b53c0 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.483+0000 7ff2dd386640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff2d80cb1b0 0x7ff2d8066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.483+0000 7ff2dcb85640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff2d81616d0 0x7ff2d80b5a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41888/0 (socket says 172.21.15.59:41888) 2024-03-06T07:22:16.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.483+0000 7ff2dcb85640 1 -- 172.21.15.59:0/2809923629 learned_addr learned my addr 172.21.15.59:0/2809923629 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:16.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.484+0000 7ff2dd386640 1 -- 172.21.15.59:0/2809923629 >> v1:172.21.15.164:6789/0 conn(0x7ff2d8161ab0 legacy=0x7ff2d80caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.484+0000 7ff2dd386640 1 -- 172.21.15.59:0/2809923629 >> v1:172.21.15.59:6789/0 conn(0x7ff2d81616d0 legacy=0x7ff2d80b5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.484+0000 7ff2dd386640 1 -- 172.21.15.59:0/2809923629 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d806b250 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.484+0000 7ff2d77fe640 1 -- 172.21.15.59:0/2809923629 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274140800 0 0) 0x7ff2d806afb0 con 0x7ff2d81616d0 2024-03-06T07:22:16.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.484+0000 7ff2dd386640 1 --2- 172.21.15.59:0/2809923629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d80cb1b0 0x7ff2d8066d00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00097a0 tx=0x7ff2c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2309c91c26f4c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.484+0000 7ff2d77fe640 1 -- 172.21.15.59:0/2809923629 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0035020 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.484+0000 7ff2d77fe640 1 -- 172.21.15.59:0/2809923629 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c0031830 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.484+0000 7ff2d77fe640 1 -- 172.21.15.59:0/2809923629 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0031c00 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.485+0000 7ff2dede9640 1 -- 172.21.15.59:0/2809923629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d80cb1b0 msgr2=0x7ff2d8066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.485+0000 7ff2dede9640 1 --2- 172.21.15.59:0/2809923629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d80cb1b0 0x7ff2d8066d00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00097a0 tx=0x7ff2c002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.485+0000 7ff2dede9640 1 -- 172.21.15.59:0/2809923629 shutdown_connections 2024-03-06T07:22:16.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.485+0000 7ff2dede9640 1 --2- 172.21.15.59:0/2809923629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d80cb1b0 0x7ff2d8066d00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.485+0000 7ff2dede9640 1 -- 172.21.15.59:0/2809923629 >> 172.21.15.59:0/2809923629 conn(0x7ff2d80d0120 msgr2=0x7ff2d80d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.485+0000 7ff2dede9640 1 -- 172.21.15.59:0/2809923629 shutdown_connections 2024-03-06T07:22:16.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.485+0000 7ff2dede9640 1 -- 172.21.15.59:0/2809923629 wait complete. 2024-03-06T07:22:16.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.486+0000 7ff2dede9640 1 Processor -- start 2024-03-06T07:22:16.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.486+0000 7ff2dede9640 1 -- start start 2024-03-06T07:22:16.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dede9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d80cb1b0 0x7ff2d815e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dede9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d81616d0 0x7ff2d80b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dede9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d8161ab0 0x7ff2d80c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dede9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d80aea70 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dede9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d806b530 con 0x7ff2d8161ab0 2024-03-06T07:22:16.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dede9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d80aebf0 con 0x7ff2d81616d0 2024-03-06T07:22:16.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dcb85640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d80cb1b0 0x7ff2d815e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dcb85640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d80cb1b0 0x7ff2d815e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42742/0 (socket says 172.21.15.59:42742) 2024-03-06T07:22:16.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dcb85640 1 -- 172.21.15.59:0/3794960194 learned_addr learned my addr 172.21.15.59:0/3794960194 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dd386640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d8161ab0 0x7ff2d80c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2d7fff640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d81616d0 0x7ff2d80b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dcb85640 1 -- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d81616d0 msgr2=0x7ff2d80b0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dcb85640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d81616d0 0x7ff2d80b0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dcb85640 1 -- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d8161ab0 msgr2=0x7ff2d80c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dcb85640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d8161ab0 0x7ff2d80c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.487+0000 7ff2dcb85640 1 -- 172.21.15.59:0/3794960194 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d80c8bf0 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.488+0000 7ff2dcb85640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d80cb1b0 0x7ff2d815e710 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000c780 tx=0x7ff2d000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.488+0000 7ff2d57fa640 1 -- 172.21.15.59:0/3794960194 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d0018070 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.488+0000 7ff2d57fa640 1 -- 172.21.15.59:0/3794960194 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d000a5e0 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.488+0000 7ff2dede9640 1 -- 172.21.15.59:0/3794960194 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d80c8ed0 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.488+0000 7ff2d57fa640 1 -- 172.21.15.59:0/3794960194 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d000aa50 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.488+0000 7ff2dede9640 1 -- 172.21.15.59:0/3794960194 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d80c9770 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.489+0000 7ff2dede9640 1 -- 172.21.15.59:0/3794960194 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d809e3d0 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.489+0000 7ff2d57fa640 1 -- 172.21.15.59:0/3794960194 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2d001b070 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.489+0000 7ff2d57fa640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b806b4d0 0x7ff2b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.490+0000 7ff2d57fa640 1 -- 172.21.15.59:0/3794960194 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2d0089730 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.490+0000 7ff2d7fff640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b806b4d0 0x7ff2b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.490+0000 7ff2d7fff640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b806b4d0 0x7ff2b806d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80d06e0 tx=0x7ff2c8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.494+0000 7ff2d57fa640 1 -- 172.21.15.59:0/3794960194 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2d0057f60 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.673+0000 7ff2dede9640 1 -- 172.21.15.59:0/3794960194 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2d8068ff0 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.674+0000 7ff2d57fa640 1 -- 172.21.15.59:0/3794960194 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff2d005c000 con 0x7ff2d80cb1b0 2024-03-06T07:22:16.676 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 -- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b806b4d0 msgr2=0x7ff2b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b806b4d0 0x7ff2b806d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80d06e0 tx=0x7ff2c8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 -- 172.21.15.59:0/3794960194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d80cb1b0 msgr2=0x7ff2d815e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d80cb1b0 0x7ff2d815e710 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000c780 tx=0x7ff2d000cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 -- 172.21.15.59:0/3794960194 shutdown_connections 2024-03-06T07:22:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b806b4d0 0x7ff2b806d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d8161ab0 0x7ff2d80c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d81616d0 0x7ff2d80b0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 --2- 172.21.15.59:0/3794960194 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d80cb1b0 0x7ff2d815e710 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 -- 172.21.15.59:0/3794960194 >> 172.21.15.59:0/3794960194 conn(0x7ff2d80d0120 msgr2=0x7ff2d80cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 -- 172.21.15.59:0/3794960194 shutdown_connections 2024-03-06T07:22:16.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.676+0000 7ff2dede9640 1 -- 172.21.15.59:0/3794960194 wait complete. 2024-03-06T07:22:16.689 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:16.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.791+0000 7fa2b393e640 1 Processor -- start 2024-03-06T07:22:16.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.791+0000 7fa2b393e640 1 -- start start 2024-03-06T07:22:16.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.792+0000 7fa2b393e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa2ac0d0af0 0x7fa2ac0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.792+0000 7fa2b393e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac0cd4b0 con 0x7fa2ac15c050 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.792+0000 7fa2b393e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac0cd790 con 0x7fa2ac06fd10 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.792+0000 7fa2b393e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2ac0cda70 con 0x7fa2ac0d0af0 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.792+0000 7fa2b16da640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa2ac15c050 0x7fa2ac06f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41902/0 (socket says 172.21.15.59:41902) 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.792+0000 7fa2b16da640 1 -- 172.21.15.59:0/3735667191 learned_addr learned my addr 172.21.15.59:0/3735667191 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.792+0000 7fa2b1edb640 1 --2- 172.21.15.59:0/3735667191 >> v2:172.21.15.164:3300/0 conn(0x7fa2ac0d0af0 0x7fa2ac0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.793+0000 7fa2abfff640 1 -- 172.21.15.59:0/3735667191 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917041869 0 0) 0x7fa2ac0cd4b0 con 0x7fa2ac15c050 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.793+0000 7fa2abfff640 1 -- 172.21.15.59:0/3735667191 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa294003520 con 0x7fa2ac15c050 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.793+0000 7fa2abfff640 1 -- 172.21.15.59:0/3735667191 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765648599 0 0) 0x7fa2ac0cd790 con 0x7fa2ac06fd10 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.793+0000 7fa2abfff640 1 -- 172.21.15.59:0/3735667191 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ac0cd4b0 con 0x7fa2ac06fd10 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.793+0000 7fa2b1edb640 1 -- 172.21.15.59:0/3735667191 >> v1:172.21.15.164:6789/0 conn(0x7fa2ac06fd10 legacy=0x7fa2ac0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.793+0000 7fa2b1edb640 1 -- 172.21.15.59:0/3735667191 >> v1:172.21.15.59:6789/0 conn(0x7fa2ac15c050 legacy=0x7fa2ac06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.793+0000 7fa2b1edb640 1 -- 172.21.15.59:0/3735667191 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac0cdd50 con 0x7fa2ac0d0af0 2024-03-06T07:22:16.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.794+0000 7fa2b1edb640 1 --2- 172.21.15.59:0/3735667191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2ac0d0af0 0x7fa2ac0cce70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009d90 tx=0x7fa29c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f0ae175ca855213 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.794+0000 7fa2abfff640 1 -- 172.21.15.59:0/3735667191 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c039070 con 0x7fa2ac0d0af0 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.794+0000 7fa2abfff640 1 -- 172.21.15.59:0/3735667191 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29c02cc60 con 0x7fa2ac0d0af0 2024-03-06T07:22:16.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.794+0000 7fa2abfff640 1 -- 172.21.15.59:0/3735667191 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c034660 con 0x7fa2ac0d0af0 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.794+0000 7fa2b393e640 1 -- 172.21.15.59:0/3735667191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2ac0d0af0 msgr2=0x7fa2ac0cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.794+0000 7fa2b393e640 1 --2- 172.21.15.59:0/3735667191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2ac0d0af0 0x7fa2ac0cce70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009d90 tx=0x7fa29c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.794+0000 7fa2b393e640 1 -- 172.21.15.59:0/3735667191 shutdown_connections 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.794+0000 7fa2b393e640 1 --2- 172.21.15.59:0/3735667191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2ac0d0af0 0x7fa2ac0cce70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.794+0000 7fa2b393e640 1 -- 172.21.15.59:0/3735667191 >> 172.21.15.59:0/3735667191 conn(0x7fa2ac0b3800 msgr2=0x7fa2ac0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.794+0000 7fa2b393e640 1 -- 172.21.15.59:0/3735667191 shutdown_connections 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.794+0000 7fa2b393e640 1 -- 172.21.15.59:0/3735667191 wait complete. 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.795+0000 7fa2b393e640 1 Processor -- start 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.795+0000 7fa2b393e640 1 -- start start 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.796+0000 7fa2b393e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2ac06fd10 0x7fa2ac15a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.796+0000 7fa2b393e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2ac0d0af0 0x7fa2ac14cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.796+0000 7fa2b393e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2ac15c050 0x7fa2ac150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.796+0000 7fa2b393e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac166e50 con 0x7fa2ac15c050 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.796+0000 7fa2b393e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac166cd0 con 0x7fa2ac0d0af0 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.796+0000 7fa2b393e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac166fd0 con 0x7fa2ac06fd10 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.796+0000 7fa2b1edb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2ac15c050 0x7fa2ac150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.796+0000 7fa2b1edb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2ac15c050 0x7fa2ac150930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42762/0 (socket says 172.21.15.59:42762) 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.796+0000 7fa2b1edb640 1 -- 172.21.15.59:0/814507424 learned_addr learned my addr 172.21.15.59:0/814507424 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.796+0000 7fa2b16da640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2ac06fd10 0x7fa2ac15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.796+0000 7fa2b0ed9640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2ac0d0af0 0x7fa2ac14cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.797+0000 7fa2b1edb640 1 -- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2ac06fd10 msgr2=0x7fa2ac15a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.797+0000 7fa2b1edb640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2ac06fd10 0x7fa2ac15a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.797+0000 7fa2b1edb640 1 -- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2ac0d0af0 msgr2=0x7fa2ac14cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.797+0000 7fa2b1edb640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2ac0d0af0 0x7fa2ac14cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.797+0000 7fa2b1edb640 1 -- 172.21.15.59:0/814507424 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac0b4b40 con 0x7fa2ac15c050 2024-03-06T07:22:16.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.797+0000 7fa2b1edb640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2ac15c050 0x7fa2ac150930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009d90 tx=0x7fa29c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.797+0000 7fa2a9ffb640 1 -- 172.21.15.59:0/814507424 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c039050 con 0x7fa2ac15c050 2024-03-06T07:22:16.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.797+0000 7fa2a9ffb640 1 -- 172.21.15.59:0/814507424 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29c045070 con 0x7fa2ac15c050 2024-03-06T07:22:16.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.797+0000 7fa2b393e640 1 -- 172.21.15.59:0/814507424 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac0b4e20 con 0x7fa2ac15c050 2024-03-06T07:22:16.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.797+0000 7fa2a9ffb640 1 -- 172.21.15.59:0/814507424 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c009450 con 0x7fa2ac15c050 2024-03-06T07:22:16.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.797+0000 7fa2b393e640 1 -- 172.21.15.59:0/814507424 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ac14d8f0 con 0x7fa2ac15c050 2024-03-06T07:22:16.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.798+0000 7fa2b393e640 1 -- 172.21.15.59:0/814507424 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ac09fe60 con 0x7fa2ac15c050 2024-03-06T07:22:16.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.799+0000 7fa2a9ffb640 1 -- 172.21.15.59:0/814507424 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa29c03e850 con 0x7fa2ac15c050 2024-03-06T07:22:16.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.799+0000 7fa2a9ffb640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa29806b4d0 0x7fa29806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.799+0000 7fa2a9ffb640 1 -- 172.21.15.59:0/814507424 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa29c0acf40 con 0x7fa2ac15c050 2024-03-06T07:22:16.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.799+0000 7fa2b16da640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa29806b4d0 0x7fa29806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.800+0000 7fa2b16da640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa29806b4d0 0x7fa29806d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4000f60 tx=0x7fa2a4007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.803+0000 7fa2a9ffb640 1 -- 172.21.15.59:0/814507424 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa29c07b8a0 con 0x7fa2ac15c050 2024-03-06T07:22:16.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.983+0000 7fa2b393e640 1 -- 172.21.15.59:0/814507424 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa2ac0b4870 con 0x7fa2ac15c050 2024-03-06T07:22:16.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.984+0000 7fa2a9ffb640 1 -- 172.21.15.59:0/814507424 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa29c07f940 con 0x7fa2ac15c050 2024-03-06T07:22:16.986 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:16.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 -- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa29806b4d0 msgr2=0x7fa29806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa29806b4d0 0x7fa29806d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4000f60 tx=0x7fa2a4007d50 comp rx=0 tx=0).stop 2024-03-06T07:22:16.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 -- 172.21.15.59:0/814507424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2ac15c050 msgr2=0x7fa2ac150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2ac15c050 0x7fa2ac150930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009d90 tx=0x7fa29c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:16.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 -- 172.21.15.59:0/814507424 shutdown_connections 2024-03-06T07:22:16.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa29806b4d0 0x7fa29806d990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2ac15c050 0x7fa2ac150930 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2ac0d0af0 0x7fa2ac14cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 --2- 172.21.15.59:0/814507424 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2ac06fd10 0x7fa2ac15a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 -- 172.21.15.59:0/814507424 >> 172.21.15.59:0/814507424 conn(0x7fa2ac0b3800 msgr2=0x7fa2ac06eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 -- 172.21.15.59:0/814507424 shutdown_connections 2024-03-06T07:22:16.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:16.986+0000 7fa2b393e640 1 -- 172.21.15.59:0/814507424 wait complete. 2024-03-06T07:22:16.998 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:17.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.101+0000 7f04bfad4640 1 Processor -- start 2024-03-06T07:22:17.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.101+0000 7f04bfad4640 1 -- start start 2024-03-06T07:22:17.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bfad4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f04b815def0 0x7f04b80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bfad4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04b80cea50 0x7f04b80ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bfad4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b806d030 con 0x7f04b806cab0 2024-03-06T07:22:17.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bfad4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b80cf1c0 con 0x7f04b815def0 2024-03-06T07:22:17.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bfad4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b80cc760 con 0x7f04b80cea50 2024-03-06T07:22:17.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bd870640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f04b815def0 0x7f04b80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bd06f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f04b806cab0 0x7f04b80ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41912/0 (socket says 172.21.15.59:41912) 2024-03-06T07:22:17.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bd06f640 1 -- 172.21.15.59:0/2898846039 learned_addr learned my addr 172.21.15.59:0/2898846039 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:17.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04be071640 1 --2- 172.21.15.59:0/2898846039 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04b80cea50 0x7f04b80ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bd870640 1 -- 172.21.15.59:0/2898846039 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04b80cea50 msgr2=0x7f04b80ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:17.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bd870640 1 --2- 172.21.15.59:0/2898846039 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04b80cea50 0x7f04b80ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bd870640 1 -- 172.21.15.59:0/2898846039 >> v1:172.21.15.59:6789/0 conn(0x7f04b806cab0 legacy=0x7f04b80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bd870640 1 -- 172.21.15.59:0/2898846039 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b806d310 con 0x7f04b815def0 2024-03-06T07:22:17.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.102+0000 7f04bc86e640 1 -- 172.21.15.59:0/2898846039 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238322874 0 0) 0x7f04b806d030 con 0x7f04b806cab0 2024-03-06T07:22:17.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.103+0000 7f04bd870640 1 --2- 172.21.15.59:0/2898846039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b815def0 0x7f04b80b41b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f04b00097a0 tx=0x7f04b002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1254506a71c0075 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.103+0000 7f04bc86e640 1 -- 172.21.15.59:0/2898846039 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b0034070 con 0x7f04b815def0 2024-03-06T07:22:17.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.103+0000 7f04bc86e640 1 -- 172.21.15.59:0/2898846039 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b0030610 con 0x7f04b815def0 2024-03-06T07:22:17.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.103+0000 7f04bc86e640 1 -- 172.21.15.59:0/2898846039 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b00309e0 con 0x7f04b815def0 2024-03-06T07:22:17.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.104+0000 7f04bfad4640 1 -- 172.21.15.59:0/2898846039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b815def0 msgr2=0x7f04b80b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.104+0000 7f04bfad4640 1 --2- 172.21.15.59:0/2898846039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b815def0 0x7f04b80b41b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f04b00097a0 tx=0x7f04b002b960 comp rx=0 tx=0).stop 2024-03-06T07:22:17.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.104+0000 7f04bfad4640 1 -- 172.21.15.59:0/2898846039 shutdown_connections 2024-03-06T07:22:17.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.104+0000 7f04bfad4640 1 --2- 172.21.15.59:0/2898846039 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04b80cea50 0x7f04b80ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.104+0000 7f04bfad4640 1 --2- 172.21.15.59:0/2898846039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b815def0 0x7f04b80b41b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.104+0000 7f04bfad4640 1 -- 172.21.15.59:0/2898846039 >> 172.21.15.59:0/2898846039 conn(0x7f04b80b3690 msgr2=0x7f04b80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.104+0000 7f04bfad4640 1 -- 172.21.15.59:0/2898846039 shutdown_connections 2024-03-06T07:22:17.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.104+0000 7f04bfad4640 1 -- 172.21.15.59:0/2898846039 wait complete. 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.105+0000 7f04bfad4640 1 Processor -- start 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.105+0000 7f04bfad4640 1 -- start start 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04bfad4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04b806cab0 0x7f04b80b6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04bfad4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04b80cea50 0x7f04b8153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04bfad4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b815def0 0x7f04b8154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04bfad4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b815e2d0 con 0x7f04b815def0 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04bfad4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b80cc3f0 con 0x7f04b806cab0 2024-03-06T07:22:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04bfad4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b815e450 con 0x7f04b80cea50 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04be071640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b815def0 0x7f04b8154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04be071640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b815def0 0x7f04b8154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42784/0 (socket says 172.21.15.59:42784) 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04be071640 1 -- 172.21.15.59:0/435307179 learned_addr learned my addr 172.21.15.59:0/435307179 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04bd06f640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04b80cea50 0x7f04b8153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04bd870640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04b806cab0 0x7f04b80b6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04be071640 1 -- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04b80cea50 msgr2=0x7f04b8153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04be071640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04b80cea50 0x7f04b8153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04be071640 1 -- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04b806cab0 msgr2=0x7f04b80b6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04be071640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04b806cab0 0x7f04b80b6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04be071640 1 -- 172.21.15.59:0/435307179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b80c6740 con 0x7f04b815def0 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.106+0000 7f04be071640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b815def0 0x7f04b8154250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f04a800efc0 tx=0x7f04a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.107+0000 7f04b67fc640 1 -- 172.21.15.59:0/435307179 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8018020 con 0x7f04b815def0 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.107+0000 7f04b67fc640 1 -- 172.21.15.59:0/435307179 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a8014480 con 0x7f04b815def0 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.107+0000 7f04bfad4640 1 -- 172.21.15.59:0/435307179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b80c56e0 con 0x7f04b815def0 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.107+0000 7f04b67fc640 1 -- 172.21.15.59:0/435307179 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8014870 con 0x7f04b815def0 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.107+0000 7f04bfad4640 1 -- 172.21.15.59:0/435307179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b80c5c60 con 0x7f04b815def0 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.107+0000 7f04bfad4640 1 -- 172.21.15.59:0/435307179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b80cbec0 con 0x7f04b815def0 2024-03-06T07:22:17.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.108+0000 7f04b67fc640 1 -- 172.21.15.59:0/435307179 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04a8005d10 con 0x7f04b815def0 2024-03-06T07:22:17.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.108+0000 7f04b67fc640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a406b4d0 0x7f04a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.109+0000 7f04b67fc640 1 -- 172.21.15.59:0/435307179 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04a808ae90 con 0x7f04b815def0 2024-03-06T07:22:17.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.109+0000 7f04bd870640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a406b4d0 0x7f04a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.109+0000 7f04bd870640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a406b4d0 0x7f04a406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f04b0036000 tx=0x7f04b003b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.113+0000 7f04b67fc640 1 -- 172.21.15.59:0/435307179 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04b80cbec0 con 0x7f04b815def0 2024-03-06T07:22:17.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.294+0000 7f04bfad4640 1 -- 172.21.15.59:0/435307179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f04b809fcf0 con 0x7f04b815def0 2024-03-06T07:22:17.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.294+0000 7f04b67fc640 1 -- 172.21.15.59:0/435307179 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f04a8022030 con 0x7f04b815def0 2024-03-06T07:22:17.296 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:17.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 -- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a406b4d0 msgr2=0x7f04a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a406b4d0 0x7f04a406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f04b0036000 tx=0x7f04b003b040 comp rx=0 tx=0).stop 2024-03-06T07:22:17.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 -- 172.21.15.59:0/435307179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b815def0 msgr2=0x7f04b8154250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b815def0 0x7f04b8154250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f04a800efc0 tx=0x7f04a800c490 comp rx=0 tx=0).stop 2024-03-06T07:22:17.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 -- 172.21.15.59:0/435307179 shutdown_connections 2024-03-06T07:22:17.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a406b4d0 0x7f04a406d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04b815def0 0x7f04b8154250 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04b80cea50 0x7f04b8153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 --2- 172.21.15.59:0/435307179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04b806cab0 0x7f04b80b6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 -- 172.21.15.59:0/435307179 >> 172.21.15.59:0/435307179 conn(0x7f04b80b3690 msgr2=0x7f04b80ccbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 -- 172.21.15.59:0/435307179 shutdown_connections 2024-03-06T07:22:17.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.296+0000 7f04bfad4640 1 -- 172.21.15.59:0/435307179 wait complete. 2024-03-06T07:22:17.309 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:17.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.408+0000 7fdf370ed640 1 Processor -- start 2024-03-06T07:22:17.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.408+0000 7fdf370ed640 1 -- start start 2024-03-06T07:22:17.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.408+0000 7fdf370ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf3013dc10 0x7fdf300b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.408+0000 7fdf370ed640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf3015bf30 0x7fdf300b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.408+0000 7fdf370ed640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3015f580 con 0x7fdf3015e850 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.408+0000 7fdf370ed640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf300cf3f0 con 0x7fdf3015bf30 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.408+0000 7fdf370ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3015b210 con 0x7fdf3013dc10 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.409+0000 7fdf2ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf3015bf30 0x7fdf300b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.409+0000 7fdf3568a640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdf3015e850 0x7fdf3015eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41928/0 (socket says 172.21.15.59:41928) 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.409+0000 7fdf3568a640 1 -- 172.21.15.59:0/2020878720 learned_addr learned my addr 172.21.15.59:0/2020878720 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.409+0000 7fdf34e89640 1 --2- 172.21.15.59:0/2020878720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf3013dc10 0x7fdf300b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.409+0000 7fdf2ffff640 1 -- 172.21.15.59:0/2020878720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf3013dc10 msgr2=0x7fdf300b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.409+0000 7fdf2ffff640 1 --2- 172.21.15.59:0/2020878720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf3013dc10 0x7fdf300b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.409+0000 7fdf2ffff640 1 -- 172.21.15.59:0/2020878720 >> v1:172.21.15.59:6789/0 conn(0x7fdf3015e850 legacy=0x7fdf3015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.409+0000 7fdf2ffff640 1 -- 172.21.15.59:0/2020878720 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3015b450 con 0x7fdf3015bf30 2024-03-06T07:22:17.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.409+0000 7fdf2f7fe640 1 -- 172.21.15.59:0/2020878720 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252261622 0 0) 0x7fdf3015f580 con 0x7fdf3015e850 2024-03-06T07:22:17.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf2ffff640 1 --2- 172.21.15.59:0/2020878720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf3015bf30 0x7fdf300b2e90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800ba40 tx=0x7fdf1802dff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14e1da719c4c084 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf2f7fe640 1 -- 172.21.15.59:0/2020878720 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1803b070 con 0x7fdf3015bf30 2024-03-06T07:22:17.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf2f7fe640 1 -- 172.21.15.59:0/2020878720 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1802eda0 con 0x7fdf3015bf30 2024-03-06T07:22:17.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf2f7fe640 1 -- 172.21.15.59:0/2020878720 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf18036660 con 0x7fdf3015bf30 2024-03-06T07:22:17.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf370ed640 1 -- 172.21.15.59:0/2020878720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf3015bf30 msgr2=0x7fdf300b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf370ed640 1 --2- 172.21.15.59:0/2020878720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf3015bf30 0x7fdf300b2e90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800ba40 tx=0x7fdf1802dff0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf370ed640 1 -- 172.21.15.59:0/2020878720 shutdown_connections 2024-03-06T07:22:17.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf370ed640 1 --2- 172.21.15.59:0/2020878720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf3015bf30 0x7fdf300b2e90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf370ed640 1 --2- 172.21.15.59:0/2020878720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf3013dc10 0x7fdf300b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf370ed640 1 -- 172.21.15.59:0/2020878720 >> 172.21.15.59:0/2020878720 conn(0x7fdf300cf8a0 msgr2=0x7fdf300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf370ed640 1 -- 172.21.15.59:0/2020878720 shutdown_connections 2024-03-06T07:22:17.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.410+0000 7fdf370ed640 1 -- 172.21.15.59:0/2020878720 wait complete. 2024-03-06T07:22:17.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.411+0000 7fdf370ed640 1 Processor -- start 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.411+0000 7fdf370ed640 1 -- start start 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.412+0000 7fdf370ed640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf3013dc10 0x7fdf30150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.412+0000 7fdf370ed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf3015bf30 0x7fdf30155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.412+0000 7fdf370ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf3015e850 0x7fdf30153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.412+0000 7fdf370ed640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf300cd7a0 con 0x7fdf3013dc10 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.412+0000 7fdf370ed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3013dff0 con 0x7fdf3015bf30 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.412+0000 7fdf370ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf300cd920 con 0x7fdf3015e850 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.412+0000 7fdf34e89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf3013dc10 0x7fdf30150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.412+0000 7fdf34e89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf3013dc10 0x7fdf30150d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42814/0 (socket says 172.21.15.59:42814) 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.412+0000 7fdf34e89640 1 -- 172.21.15.59:0/4163973026 learned_addr learned my addr 172.21.15.59:0/4163973026 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.412+0000 7fdf3568a640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf3015e850 0x7fdf30153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf2ffff640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf3015bf30 0x7fdf30155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf34e89640 1 -- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf3015e850 msgr2=0x7fdf30153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf34e89640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf3015e850 0x7fdf30153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf34e89640 1 -- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf3015bf30 msgr2=0x7fdf30155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf34e89640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf3015bf30 0x7fdf30155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf34e89640 1 -- 172.21.15.59:0/4163973026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf30155730 con 0x7fdf3013dc10 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf34e89640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf3013dc10 0x7fdf30150d60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800efc0 tx=0x7fdf2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf2d7fa640 1 -- 172.21.15.59:0/4163973026 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28018020 con 0x7fdf3013dc10 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf2d7fa640 1 -- 172.21.15.59:0/4163973026 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf28014480 con 0x7fdf3013dc10 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf370ed640 1 -- 172.21.15.59:0/4163973026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf30155450 con 0x7fdf3013dc10 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf2d7fa640 1 -- 172.21.15.59:0/4163973026 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28014850 con 0x7fdf3013dc10 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.413+0000 7fdf370ed640 1 -- 172.21.15.59:0/4163973026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf301535e0 con 0x7fdf3013dc10 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.414+0000 7fdf370ed640 1 -- 172.21.15.59:0/4163973026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3004f330 con 0x7fdf3013dc10 2024-03-06T07:22:17.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.414+0000 7fdf2d7fa640 1 -- 172.21.15.59:0/4163973026 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf28005d10 con 0x7fdf3013dc10 2024-03-06T07:22:17.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.415+0000 7fdf2d7fa640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf1006b4d0 0x7fdf1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.415+0000 7fdf2d7fa640 1 -- 172.21.15.59:0/4163973026 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf2808af70 con 0x7fdf3013dc10 2024-03-06T07:22:17.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.415+0000 7fdf2ffff640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf1006b4d0 0x7fdf1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.415+0000 7fdf2ffff640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf1006b4d0 0x7fdf1006d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdf18038000 tx=0x7fdf18032000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.419+0000 7fdf2d7fa640 1 -- 172.21.15.59:0/4163973026 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf28022030 con 0x7fdf3013dc10 2024-03-06T07:22:17.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.601+0000 7fdf370ed640 1 -- 172.21.15.59:0/4163973026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf3006eb00 con 0x7fdf3013dc10 2024-03-06T07:22:17.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.602+0000 7fdf2d7fa640 1 -- 172.21.15.59:0/4163973026 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fdf28010040 con 0x7fdf3013dc10 2024-03-06T07:22:17.604 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:17.605 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 -- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf1006b4d0 msgr2=0x7fdf1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf1006b4d0 0x7fdf1006d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdf18038000 tx=0x7fdf18032000 comp rx=0 tx=0).stop 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 -- 172.21.15.59:0/4163973026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf3013dc10 msgr2=0x7fdf30150d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf3013dc10 0x7fdf30150d60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800efc0 tx=0x7fdf2800c490 comp rx=0 tx=0).stop 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 -- 172.21.15.59:0/4163973026 shutdown_connections 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf1006b4d0 0x7fdf1006d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf3015e850 0x7fdf30153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf3015bf30 0x7fdf30155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 --2- 172.21.15.59:0/4163973026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf3013dc10 0x7fdf30150d60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 -- 172.21.15.59:0/4163973026 >> 172.21.15.59:0/4163973026 conn(0x7fdf300cf8a0 msgr2=0x7fdf3015d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 -- 172.21.15.59:0/4163973026 shutdown_connections 2024-03-06T07:22:17.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.605+0000 7fdf370ed640 1 -- 172.21.15.59:0/4163973026 wait complete. 2024-03-06T07:22:17.618 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:17.618 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:17.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.720+0000 7fa7efd7a640 1 Processor -- start 2024-03-06T07:22:17.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.721+0000 7fa7efd7a640 1 -- start start 2024-03-06T07:22:17.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.721+0000 7fa7efd7a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa7e81614d0 0x7fa7e80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.721+0000 7fa7efd7a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa7e80d1910 0x7fa7e80d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.721+0000 7fa7efd7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7e80d22e0 0x7fa7e80d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.721+0000 7fa7efd7a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e80d2750 con 0x7fa7e81614d0 2024-03-06T07:22:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.721+0000 7fa7efd7a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e81618b0 con 0x7fa7e80d1910 2024-03-06T07:22:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.721+0000 7fa7efd7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80cf400 con 0x7fa7e80d22e0 2024-03-06T07:22:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.722+0000 7fa7edb16640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa7e81614d0 0x7fa7e80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.722+0000 7fa7edb16640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa7e81614d0 0x7fa7e80d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42816/0 (socket says 172.21.15.59:42816) 2024-03-06T07:22:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.722+0000 7fa7edb16640 1 -- 172.21.15.59:0/1014914086 learned_addr learned my addr 172.21.15.59:0/1014914086 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:17.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.722+0000 7fa7ee317640 1 --2- 172.21.15.59:0/1014914086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7e80d22e0 0x7fa7e80d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.722+0000 7fa7ed315640 1 --2- 172.21.15.59:0/1014914086 >> v2:172.21.15.164:3300/0 conn(0x7fa7e80d1910 0x7fa7e80d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.722+0000 7fa7edb16640 1 -- 172.21.15.59:0/1014914086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7e80d22e0 msgr2=0x7fa7e80d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:17.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.722+0000 7fa7edb16640 1 --2- 172.21.15.59:0/1014914086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7e80d22e0 0x7fa7e80d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.722+0000 7fa7edb16640 1 -- 172.21.15.59:0/1014914086 >> v2:172.21.15.164:3300/0 conn(0x7fa7e80d1910 msgr2=0x7fa7e80d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:17.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.722+0000 7fa7edb16640 1 --2- 172.21.15.59:0/1014914086 >> v2:172.21.15.164:3300/0 conn(0x7fa7e80d1910 0x7fa7e80d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.722+0000 7fa7edb16640 1 -- 172.21.15.59:0/1014914086 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e80cf6a0 con 0x7fa7e81614d0 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7edb16640 1 --2- 172.21.15.59:0/1014914086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7e81614d0 0x7fa7e80d4860 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa7e80ce550 tx=0x7fa7e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cda3fbba5b41001 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7ecb14640 1 -- 172.21.15.59:0/1014914086 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4035020 con 0x7fa7e81614d0 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7ecb14640 1 -- 172.21.15.59:0/1014914086 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e4031870 con 0x7fa7e81614d0 2024-03-06T07:22:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7ecb14640 1 -- 172.21.15.59:0/1014914086 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4031c40 con 0x7fa7e81614d0 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7efd7a640 1 -- 172.21.15.59:0/1014914086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7e81614d0 msgr2=0x7fa7e80d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7efd7a640 1 --2- 172.21.15.59:0/1014914086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7e81614d0 0x7fa7e80d4860 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa7e80ce550 tx=0x7fa7e402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7efd7a640 1 -- 172.21.15.59:0/1014914086 shutdown_connections 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7efd7a640 1 --2- 172.21.15.59:0/1014914086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7e80d22e0 0x7fa7e80d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7efd7a640 1 --2- 172.21.15.59:0/1014914086 >> v2:172.21.15.164:3300/0 conn(0x7fa7e80d1910 0x7fa7e80d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7efd7a640 1 --2- 172.21.15.59:0/1014914086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7e81614d0 0x7fa7e80d4860 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7efd7a640 1 -- 172.21.15.59:0/1014914086 >> 172.21.15.59:0/1014914086 conn(0x7fa7e80d3d40 msgr2=0x7fa7e80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7efd7a640 1 -- 172.21.15.59:0/1014914086 shutdown_connections 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.723+0000 7fa7efd7a640 1 -- 172.21.15.59:0/1014914086 wait complete. 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.725+0000 7fa7efd7a640 1 Processor -- start 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.725+0000 7fa7efd7a640 1 -- start start 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.725+0000 7fa7efd7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7e80d1910 0x7fa7e814d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.725+0000 7fa7efd7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7e80d22e0 0x7fa7e814d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.725+0000 7fa7efd7a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7e81614d0 0x7fa7e80cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.725+0000 7fa7efd7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80b0910 con 0x7fa7e80d1910 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.725+0000 7fa7efd7a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80cfc60 con 0x7fa7e81614d0 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.725+0000 7fa7efd7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80d0be0 con 0x7fa7e80d22e0 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.725+0000 7fa7edb16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7e80d1910 0x7fa7e814d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.725+0000 7fa7edb16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7e80d1910 0x7fa7e814d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42824/0 (socket says 172.21.15.59:42824) 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.725+0000 7fa7edb16640 1 -- 172.21.15.59:0/326116361 learned_addr learned my addr 172.21.15.59:0/326116361 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7ee317640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7e81614d0 0x7fa7e80cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7ed315640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7e80d22e0 0x7fa7e814d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7edb16640 1 -- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7e80d22e0 msgr2=0x7fa7e814d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7edb16640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7e80d22e0 0x7fa7e814d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7edb16640 1 -- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7e81614d0 msgr2=0x7fa7e80cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7edb16640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7e81614d0 0x7fa7e80cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7edb16640 1 -- 172.21.15.59:0/326116361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e80c8b60 con 0x7fa7e80d1910 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7edb16640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7e80d1910 0x7fa7e814d300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa7e402b950 tx=0x7fa7e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7de7fc640 1 -- 172.21.15.59:0/326116361 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4040070 con 0x7fa7e80d1910 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7de7fc640 1 -- 172.21.15.59:0/326116361 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e4031e10 con 0x7fa7e80d1910 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7efd7a640 1 -- 172.21.15.59:0/326116361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e80c8e40 con 0x7fa7e80d1910 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7de7fc640 1 -- 172.21.15.59:0/326116361 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4035040 con 0x7fa7e80d1910 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.726+0000 7fa7efd7a640 1 -- 172.21.15.59:0/326116361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e80c9400 con 0x7fa7e80d1910 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.727+0000 7fa7efd7a640 1 -- 172.21.15.59:0/326116361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e8068ff0 con 0x7fa7e80d1910 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.727+0000 7fa7de7fc640 1 -- 172.21.15.59:0/326116361 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7e404b060 con 0x7fa7e80d1910 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.728+0000 7fa7de7fc640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7cc06b4d0 0x7fa7cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.728+0000 7fa7de7fc640 1 -- 172.21.15.59:0/326116361 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7e40abc50 con 0x7fa7e80d1910 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.728+0000 7fa7ed315640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7cc06b4d0 0x7fa7cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.728+0000 7fa7ed315640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7cc06b4d0 0x7fa7cc06d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa7e80c7a20 tx=0x7fa7d8009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.732+0000 7fa7de7fc640 1 -- 172.21.15.59:0/326116361 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7e40b1020 con 0x7fa7e80d1910 2024-03-06T07:22:17.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.911+0000 7fa7efd7a640 1 -- 172.21.15.59:0/326116361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa7e814ded0 con 0x7fa7e80d1910 2024-03-06T07:22:17.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.911+0000 7fa7de7fc640 1 -- 172.21.15.59:0/326116361 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa7e407a510 con 0x7fa7e80d1910 2024-03-06T07:22:17.913 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 -- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7cc06b4d0 msgr2=0x7fa7cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7cc06b4d0 0x7fa7cc06d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa7e80c7a20 tx=0x7fa7d8009190 comp rx=0 tx=0).stop 2024-03-06T07:22:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 -- 172.21.15.59:0/326116361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7e80d1910 msgr2=0x7fa7e814d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7e80d1910 0x7fa7e814d300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa7e402b950 tx=0x7fa7e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 -- 172.21.15.59:0/326116361 shutdown_connections 2024-03-06T07:22:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7cc06b4d0 0x7fa7cc06d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7e81614d0 0x7fa7e80cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7e80d22e0 0x7fa7e814d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 --2- 172.21.15.59:0/326116361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7e80d1910 0x7fa7e814d300 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 -- 172.21.15.59:0/326116361 >> 172.21.15.59:0/326116361 conn(0x7fa7e80d3d40 msgr2=0x7fa7e80ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 -- 172.21.15.59:0/326116361 shutdown_connections 2024-03-06T07:22:17.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:17.913+0000 7fa7efd7a640 1 -- 172.21.15.59:0/326116361 wait complete. 2024-03-06T07:22:17.926 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:18.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.028+0000 7f2b374d6640 1 Processor -- start 2024-03-06T07:22:18.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.028+0000 7f2b374d6640 1 -- start start 2024-03-06T07:22:18.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b374d6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2b3015def0 0x7f2b300b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b374d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b300cea50 0x7f2b300ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b374d6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3006d030 con 0x7f2b3006cab0 2024-03-06T07:22:18.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b374d6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b300cf1c0 con 0x7f2b3015def0 2024-03-06T07:22:18.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b374d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b300cc760 con 0x7f2b300cea50 2024-03-06T07:22:18.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b34a71640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2b3006cab0 0x7f2b300ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41940/0 (socket says 172.21.15.59:41940) 2024-03-06T07:22:18.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b34a71640 1 -- 172.21.15.59:0/756912003 learned_addr learned my addr 172.21.15.59:0/756912003 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:18.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b35a73640 1 --2- 172.21.15.59:0/756912003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b300cea50 0x7f2b300ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b35272640 1 --2- 172.21.15.59:0/756912003 >> v2:172.21.15.164:3300/0 conn(0x7f2b3015def0 0x7f2b300b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b27fff640 1 -- 172.21.15.59:0/756912003 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285406510 0 0) 0x7f2b3006d030 con 0x7f2b3006cab0 2024-03-06T07:22:18.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b27fff640 1 -- 172.21.15.59:0/756912003 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b1c003520 con 0x7f2b3006cab0 2024-03-06T07:22:18.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b35a73640 1 -- 172.21.15.59:0/756912003 >> v2:172.21.15.164:3300/0 conn(0x7f2b3015def0 msgr2=0x7f2b300b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b35a73640 1 --2- 172.21.15.59:0/756912003 >> v2:172.21.15.164:3300/0 conn(0x7f2b3015def0 0x7f2b300b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b35a73640 1 -- 172.21.15.59:0/756912003 >> v1:172.21.15.59:6789/0 conn(0x7f2b3006cab0 legacy=0x7f2b300ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b35a73640 1 -- 172.21.15.59:0/756912003 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3006d310 con 0x7f2b300cea50 2024-03-06T07:22:18.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.029+0000 7f2b35272640 1 --2- 172.21.15.59:0/756912003 >> v2:172.21.15.164:3300/0 conn(0x7f2b3015def0 0x7f2b300b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.030+0000 7f2b35a73640 1 --2- 172.21.15.59:0/756912003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b300cea50 0x7f2b300ccbf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009d90 tx=0x7f2b2802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68acea3dbddae1e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.030+0000 7f2b27fff640 1 -- 172.21.15.59:0/756912003 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28039070 con 0x7f2b300cea50 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.030+0000 7f2b27fff640 1 -- 172.21.15.59:0/756912003 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2802cce0 con 0x7f2b300cea50 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.030+0000 7f2b27fff640 1 -- 172.21.15.59:0/756912003 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28034660 con 0x7f2b300cea50 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.031+0000 7f2b374d6640 1 -- 172.21.15.59:0/756912003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b300cea50 msgr2=0x7f2b300ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.031+0000 7f2b374d6640 1 --2- 172.21.15.59:0/756912003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b300cea50 0x7f2b300ccbf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009d90 tx=0x7f2b2802bf30 comp rx=0 tx=0).stop 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.031+0000 7f2b374d6640 1 -- 172.21.15.59:0/756912003 shutdown_connections 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.031+0000 7f2b374d6640 1 --2- 172.21.15.59:0/756912003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b300cea50 0x7f2b300ccbf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.031+0000 7f2b374d6640 1 --2- 172.21.15.59:0/756912003 >> v2:172.21.15.164:3300/0 conn(0x7f2b3015def0 0x7f2b300b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.031+0000 7f2b374d6640 1 -- 172.21.15.59:0/756912003 >> 172.21.15.59:0/756912003 conn(0x7f2b300b3690 msgr2=0x7f2b300b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.031+0000 7f2b374d6640 1 -- 172.21.15.59:0/756912003 shutdown_connections 2024-03-06T07:22:18.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.031+0000 7f2b374d6640 1 -- 172.21.15.59:0/756912003 wait complete. 2024-03-06T07:22:18.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.032+0000 7f2b374d6640 1 Processor -- start 2024-03-06T07:22:18.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.032+0000 7f2b374d6640 1 -- start start 2024-03-06T07:22:18.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b374d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b3006cab0 0x7f2b30154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b374d6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b300cea50 0x7f2b300c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b374d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b3015def0 0x7f2b300ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b374d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3015e2d0 con 0x7f2b3006cab0 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b374d6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b300cc3f0 con 0x7f2b300cea50 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b374d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3015e450 con 0x7f2b3015def0 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b35272640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b3006cab0 0x7f2b30154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b35272640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b3006cab0 0x7f2b30154e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42826/0 (socket says 172.21.15.59:42826) 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b35272640 1 -- 172.21.15.59:0/874037181 learned_addr learned my addr 172.21.15.59:0/874037181 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b34a71640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b300cea50 0x7f2b300c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b35a73640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b3015def0 0x7f2b300ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b35272640 1 -- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b3015def0 msgr2=0x7f2b300ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b35272640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b3015def0 0x7f2b300ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b35272640 1 -- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b300cea50 msgr2=0x7f2b300c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b35272640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b300cea50 0x7f2b300c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b35272640 1 -- 172.21.15.59:0/874037181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b301553a0 con 0x7f2b3006cab0 2024-03-06T07:22:18.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b35272640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b3006cab0 0x7f2b30154e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00e990 tx=0x7f2b2c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b25ffb640 1 -- 172.21.15.59:0/874037181 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c017070 con 0x7f2b3006cab0 2024-03-06T07:22:18.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b25ffb640 1 -- 172.21.15.59:0/874037181 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2c00c980 con 0x7f2b3006cab0 2024-03-06T07:22:18.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b374d6640 1 -- 172.21.15.59:0/874037181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b300b4b40 con 0x7f2b3006cab0 2024-03-06T07:22:18.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.033+0000 7f2b25ffb640 1 -- 172.21.15.59:0/874037181 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c00cd50 con 0x7f2b3006cab0 2024-03-06T07:22:18.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.034+0000 7f2b374d6640 1 -- 172.21.15.59:0/874037181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b300b5100 con 0x7f2b3006cab0 2024-03-06T07:22:18.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.035+0000 7f2b374d6640 1 -- 172.21.15.59:0/874037181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3009fcf0 con 0x7f2b3006cab0 2024-03-06T07:22:18.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.035+0000 7f2b25ffb640 1 -- 172.21.15.59:0/874037181 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b2c01a070 con 0x7f2b3006cab0 2024-03-06T07:22:18.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.035+0000 7f2b25ffb640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b1406b4d0 0x7f2b1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.035+0000 7f2b25ffb640 1 -- 172.21.15.59:0/874037181 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b2c089f70 con 0x7f2b3006cab0 2024-03-06T07:22:18.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.036+0000 7f2b34a71640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b1406b4d0 0x7f2b1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.036+0000 7f2b34a71640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b1406b4d0 0x7f2b1406d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2b301f71e0 tx=0x7f2b20009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.040+0000 7f2b25ffb640 1 -- 172.21.15.59:0/874037181 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b2c058830 con 0x7f2b3006cab0 2024-03-06T07:22:18.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.222+0000 7f2b374d6640 1 -- 172.21.15.59:0/874037181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2b30068ff0 con 0x7f2b3006cab0 2024-03-06T07:22:18.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.222+0000 7f2b25ffb640 1 -- 172.21.15.59:0/874037181 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2b2c05c8d0 con 0x7f2b3006cab0 2024-03-06T07:22:18.224 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 -- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b1406b4d0 msgr2=0x7f2b1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b1406b4d0 0x7f2b1406d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2b301f71e0 tx=0x7f2b20009040 comp rx=0 tx=0).stop 2024-03-06T07:22:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 -- 172.21.15.59:0/874037181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b3006cab0 msgr2=0x7f2b30154e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b3006cab0 0x7f2b30154e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00e990 tx=0x7f2b2c00ee50 comp rx=0 tx=0).stop 2024-03-06T07:22:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 -- 172.21.15.59:0/874037181 shutdown_connections 2024-03-06T07:22:18.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b1406b4d0 0x7f2b1406d990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b3015def0 0x7f2b300ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b300cea50 0x7f2b300c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 --2- 172.21.15.59:0/874037181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b3006cab0 0x7f2b30154e60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 -- 172.21.15.59:0/874037181 >> 172.21.15.59:0/874037181 conn(0x7f2b300b3690 msgr2=0x7f2b3006fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 -- 172.21.15.59:0/874037181 shutdown_connections 2024-03-06T07:22:18.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.224+0000 7f2b374d6640 1 -- 172.21.15.59:0/874037181 wait complete. 2024-03-06T07:22:18.236 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:18.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.339+0000 7f5f456df640 1 Processor -- start 2024-03-06T07:22:18.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.339+0000 7f5f456df640 1 -- start start 2024-03-06T07:22:18.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f456df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f40145650 0x7f5f4006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f456df640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5f4006b4f0 0x7f5f4015c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f456df640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4015f840 con 0x7f5f4015c8a0 2024-03-06T07:22:18.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f456df640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4006b950 con 0x7f5f4006b4f0 2024-03-06T07:22:18.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f456df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4015d120 con 0x7f5f40145650 2024-03-06T07:22:18.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f3effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f40145650 0x7f5f4006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f37fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5f4006b4f0 0x7f5f4015c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f3f7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5f4015c8a0 0x7f5f400d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:53984/0 (socket says 172.21.15.59:53984) 2024-03-06T07:22:18.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f3f7fe640 1 -- 172.21.15.59:0/2290586994 learned_addr learned my addr 172.21.15.59:0/2290586994 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:18.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f3effd640 1 -- 172.21.15.59:0/2290586994 >> v2:172.21.15.164:3300/0 conn(0x7f5f4006b4f0 msgr2=0x7f5f4015c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f3effd640 1 --2- 172.21.15.59:0/2290586994 >> v2:172.21.15.164:3300/0 conn(0x7f5f4006b4f0 0x7f5f4015c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f3effd640 1 -- 172.21.15.59:0/2290586994 >> v1:172.21.15.164:6789/0 conn(0x7f5f4015c8a0 legacy=0x7f5f400d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f3effd640 1 -- 172.21.15.59:0/2290586994 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4015fb20 con 0x7f5f40145650 2024-03-06T07:22:18.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.340+0000 7f5f37fff640 1 --2- 172.21.15.59:0/2290586994 >> v2:172.21.15.164:3300/0 conn(0x7f5f4006b4f0 0x7f5f4015c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:18.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.341+0000 7f5f3effd640 1 --2- 172.21.15.59:0/2290586994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f40145650 0x7f5f4006afb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5f300097a0 tx=0x7f5f3002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48731567d5397754 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.341+0000 7f5f3e7fc640 1 -- 172.21.15.59:0/2290586994 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f30035020 con 0x7f5f40145650 2024-03-06T07:22:18.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.341+0000 7f5f3e7fc640 1 -- 172.21.15.59:0/2290586994 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f30031830 con 0x7f5f40145650 2024-03-06T07:22:18.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.341+0000 7f5f3e7fc640 1 -- 172.21.15.59:0/2290586994 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f30031c00 con 0x7f5f40145650 2024-03-06T07:22:18.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.342+0000 7f5f456df640 1 -- 172.21.15.59:0/2290586994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f40145650 msgr2=0x7f5f4006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.342+0000 7f5f456df640 1 --2- 172.21.15.59:0/2290586994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f40145650 0x7f5f4006afb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5f300097a0 tx=0x7f5f3002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.342+0000 7f5f456df640 1 -- 172.21.15.59:0/2290586994 shutdown_connections 2024-03-06T07:22:18.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.342+0000 7f5f456df640 1 --2- 172.21.15.59:0/2290586994 >> v2:172.21.15.164:3300/0 conn(0x7f5f4006b4f0 0x7f5f4015c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.342+0000 7f5f456df640 1 --2- 172.21.15.59:0/2290586994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f40145650 0x7f5f4006afb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.342+0000 7f5f456df640 1 -- 172.21.15.59:0/2290586994 >> 172.21.15.59:0/2290586994 conn(0x7f5f4006c0c0 msgr2=0x7f5f4006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.342+0000 7f5f456df640 1 -- 172.21.15.59:0/2290586994 shutdown_connections 2024-03-06T07:22:18.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.342+0000 7f5f456df640 1 -- 172.21.15.59:0/2290586994 wait complete. 2024-03-06T07:22:18.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.343+0000 7f5f456df640 1 Processor -- start 2024-03-06T07:22:18.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.343+0000 7f5f456df640 1 -- start start 2024-03-06T07:22:18.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f456df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f4006b4f0 0x7f5f4012e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f456df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f40145650 0x7f5f40157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f456df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f4015c8a0 0x7f5f40157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f456df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f40160600 con 0x7f5f40145650 2024-03-06T07:22:18.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f456df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f401600e0 con 0x7f5f4006b4f0 2024-03-06T07:22:18.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f456df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400cfd90 con 0x7f5f4015c8a0 2024-03-06T07:22:18.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f40145650 0x7f5f40157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f3effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f4006b4f0 0x7f5f4012e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f3f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f4015c8a0 0x7f5f40157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f3effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f4006b4f0 0x7f5f4012e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:58210/0 (socket says 172.21.15.59:58210) 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f3effd640 1 -- 172.21.15.59:0/704128516 learned_addr learned my addr 172.21.15.59:0/704128516 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f3effd640 1 -- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f4015c8a0 msgr2=0x7f5f40157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f3effd640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f4015c8a0 0x7f5f40157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f3effd640 1 -- 172.21.15.59:0/704128516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f40145650 msgr2=0x7f5f40157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f3effd640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f40145650 0x7f5f40157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f3effd640 1 -- 172.21.15.59:0/704128516 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f401582b0 con 0x7f5f4006b4f0 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f3f7fe640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f4015c8a0 0x7f5f40157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f37fff640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f40145650 0x7f5f40157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.344+0000 7f5f3effd640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f4006b4f0 0x7f5f4012e090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5f300098c0 tx=0x7f5f300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.345+0000 7f5f377fe640 1 -- 172.21.15.59:0/704128516 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3003b980 con 0x7f5f4006b4f0 2024-03-06T07:22:18.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.345+0000 7f5f377fe640 1 -- 172.21.15.59:0/704128516 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3003bb40 con 0x7f5f4006b4f0 2024-03-06T07:22:18.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.345+0000 7f5f377fe640 1 -- 172.21.15.59:0/704128516 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f300453f0 con 0x7f5f4006b4f0 2024-03-06T07:22:18.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.345+0000 7f5f456df640 1 -- 172.21.15.59:0/704128516 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f40158590 con 0x7f5f4006b4f0 2024-03-06T07:22:18.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.345+0000 7f5f456df640 1 -- 172.21.15.59:0/704128516 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f401e5890 con 0x7f5f4006b4f0 2024-03-06T07:22:18.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.346+0000 7f5f456df640 1 -- 172.21.15.59:0/704128516 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f400733b0 con 0x7f5f4006b4f0 2024-03-06T07:22:18.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.346+0000 7f5f377fe640 1 -- 172.21.15.59:0/704128516 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f30038020 con 0x7f5f4006b4f0 2024-03-06T07:22:18.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.347+0000 7f5f377fe640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f2006b500 0x7f5f2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.347+0000 7f5f377fe640 1 -- 172.21.15.59:0/704128516 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f300abe20 con 0x7f5f4006b4f0 2024-03-06T07:22:18.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.347+0000 7f5f37fff640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f2006b500 0x7f5f2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.347+0000 7f5f37fff640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f2006b500 0x7f5f2006d9c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5f28009560 tx=0x7f5f28009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.356+0000 7f5f377fe640 1 -- 172.21.15.59:0/704128516 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f30040070 con 0x7f5f4006b4f0 2024-03-06T07:22:18.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.536+0000 7f5f456df640 1 -- 172.21.15.59:0/704128516 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5f400720e0 con 0x7f5f4006b4f0 2024-03-06T07:22:18.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.537+0000 7f5f377fe640 1 -- 172.21.15.59:0/704128516 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5f30031800 con 0x7f5f4006b4f0 2024-03-06T07:22:18.539 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:18.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 -- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f2006b500 msgr2=0x7f5f2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f2006b500 0x7f5f2006d9c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5f28009560 tx=0x7f5f28009190 comp rx=0 tx=0).stop 2024-03-06T07:22:18.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 -- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f4006b4f0 msgr2=0x7f5f4012e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f4006b4f0 0x7f5f4012e090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5f300098c0 tx=0x7f5f300046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 -- 172.21.15.59:0/704128516 shutdown_connections 2024-03-06T07:22:18.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f2006b500 0x7f5f2006d9c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f4015c8a0 0x7f5f40157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f40145650 0x7f5f40157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 --2- 172.21.15.59:0/704128516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f4006b4f0 0x7f5f4012e090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 -- 172.21.15.59:0/704128516 >> 172.21.15.59:0/704128516 conn(0x7f5f4006c0c0 msgr2=0x7f5f4015eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 -- 172.21.15.59:0/704128516 shutdown_connections 2024-03-06T07:22:18.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.539+0000 7f5f456df640 1 -- 172.21.15.59:0/704128516 wait complete. 2024-03-06T07:22:18.552 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:18.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.651+0000 7fda330df640 1 Processor -- start 2024-03-06T07:22:18.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.651+0000 7fda330df640 1 -- start start 2024-03-06T07:22:18.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda330df640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fda2c1621c0 0x7fda2c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda330df640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fda2c0d4920 0x7fda2c0d26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda330df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda2c0d2bf0 0x7fda2c15dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda330df640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2c0cf690 con 0x7fda2c0d4920 2024-03-06T07:22:18.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda330df640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2c0ceac0 con 0x7fda2c1621c0 2024-03-06T07:22:18.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda330df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c0ced60 con 0x7fda2c0d2bf0 2024-03-06T07:22:18.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda2bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fda2c0d4920 0x7fda2c0d26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda2bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fda2c0d4920 0x7fda2c0d26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42858/0 (socket says 172.21.15.59:42858) 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda2bfff640 1 -- 172.21.15.59:0/737283878 learned_addr learned my addr 172.21.15.59:0/737283878 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda3167c640 1 --2- 172.21.15.59:0/737283878 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda2c0d2bf0 0x7fda2c15dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda30e7b640 1 --2- 172.21.15.59:0/737283878 >> v2:172.21.15.164:3300/0 conn(0x7fda2c1621c0 0x7fda2c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda2bfff640 1 -- 172.21.15.59:0/737283878 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda2c0d2bf0 msgr2=0x7fda2c15dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda2bfff640 1 --2- 172.21.15.59:0/737283878 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda2c0d2bf0 0x7fda2c15dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda2bfff640 1 -- 172.21.15.59:0/737283878 >> v2:172.21.15.164:3300/0 conn(0x7fda2c1621c0 msgr2=0x7fda2c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda2bfff640 1 --2- 172.21.15.59:0/737283878 >> v2:172.21.15.164:3300/0 conn(0x7fda2c1621c0 0x7fda2c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.652+0000 7fda2bfff640 1 -- 172.21.15.59:0/737283878 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c0cf040 con 0x7fda2c0d4920 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.653+0000 7fda2bfff640 1 --2- 172.21.15.59:0/737283878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda2c0d4920 0x7fda2c0d26b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fda2c0d3ce0 tx=0x7fda2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238406eed3b6b42b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.653+0000 7fda2b7fe640 1 -- 172.21.15.59:0/737283878 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20035020 con 0x7fda2c0d4920 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.653+0000 7fda2b7fe640 1 -- 172.21.15.59:0/737283878 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda20031870 con 0x7fda2c0d4920 2024-03-06T07:22:18.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.653+0000 7fda2b7fe640 1 -- 172.21.15.59:0/737283878 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20031c40 con 0x7fda2c0d4920 2024-03-06T07:22:18.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.654+0000 7fda330df640 1 -- 172.21.15.59:0/737283878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda2c0d4920 msgr2=0x7fda2c0d26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.654+0000 7fda330df640 1 --2- 172.21.15.59:0/737283878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda2c0d4920 0x7fda2c0d26b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fda2c0d3ce0 tx=0x7fda2002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.654+0000 7fda330df640 1 -- 172.21.15.59:0/737283878 shutdown_connections 2024-03-06T07:22:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.654+0000 7fda330df640 1 --2- 172.21.15.59:0/737283878 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda2c0d2bf0 0x7fda2c15dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.654+0000 7fda330df640 1 --2- 172.21.15.59:0/737283878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda2c0d4920 0x7fda2c0d26b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.654+0000 7fda330df640 1 --2- 172.21.15.59:0/737283878 >> v2:172.21.15.164:3300/0 conn(0x7fda2c1621c0 0x7fda2c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.654+0000 7fda330df640 1 -- 172.21.15.59:0/737283878 >> 172.21.15.59:0/737283878 conn(0x7fda2c0d3d10 msgr2=0x7fda2c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.654+0000 7fda330df640 1 -- 172.21.15.59:0/737283878 shutdown_connections 2024-03-06T07:22:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.654+0000 7fda330df640 1 -- 172.21.15.59:0/737283878 wait complete. 2024-03-06T07:22:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.655+0000 7fda330df640 1 Processor -- start 2024-03-06T07:22:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.655+0000 7fda330df640 1 -- start start 2024-03-06T07:22:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda330df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda2c0d2bf0 0x7fda2c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda330df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda2c0d4920 0x7fda2c1e8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda330df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda2c1621c0 0x7fda2c0cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda330df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c162720 con 0x7fda2c1621c0 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda330df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c1625a0 con 0x7fda2c0d2bf0 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda330df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c1628a0 con 0x7fda2c0d4920 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda3167c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda2c1621c0 0x7fda2c0cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda3167c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda2c1621c0 0x7fda2c0cbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42864/0 (socket says 172.21.15.59:42864) 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda3167c640 1 -- 172.21.15.59:0/3450832409 learned_addr learned my addr 172.21.15.59:0/3450832409 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda30e7b640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda2c0d2bf0 0x7fda2c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda2bfff640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda2c0d4920 0x7fda2c1e8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda3167c640 1 -- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda2c0d4920 msgr2=0x7fda2c1e8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda3167c640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda2c0d4920 0x7fda2c1e8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda3167c640 1 -- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda2c0d2bf0 msgr2=0x7fda2c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda3167c640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda2c0d2bf0 0x7fda2c0cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.656+0000 7fda3167c640 1 -- 172.21.15.59:0/3450832409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c0ccb00 con 0x7fda2c1621c0 2024-03-06T07:22:18.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.657+0000 7fda3167c640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda2c1621c0 0x7fda2c0cbf20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00e7a0 tx=0x7fda1c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.657+0000 7fda297fa640 1 -- 172.21.15.59:0/3450832409 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c017070 con 0x7fda2c1621c0 2024-03-06T07:22:18.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.657+0000 7fda297fa640 1 -- 172.21.15.59:0/3450832409 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c00c8e0 con 0x7fda2c1621c0 2024-03-06T07:22:18.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.657+0000 7fda330df640 1 -- 172.21.15.59:0/3450832409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2c0ccde0 con 0x7fda2c1621c0 2024-03-06T07:22:18.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.657+0000 7fda297fa640 1 -- 172.21.15.59:0/3450832409 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c00ccd0 con 0x7fda2c1621c0 2024-03-06T07:22:18.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.657+0000 7fda330df640 1 -- 172.21.15.59:0/3450832409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2c0cd3a0 con 0x7fda2c1621c0 2024-03-06T07:22:18.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.658+0000 7fda330df640 1 -- 172.21.15.59:0/3450832409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2c04f330 con 0x7fda2c1621c0 2024-03-06T07:22:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.658+0000 7fda297fa640 1 -- 172.21.15.59:0/3450832409 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda1c01a070 con 0x7fda2c1621c0 2024-03-06T07:22:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.659+0000 7fda297fa640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1806b4d0 0x7fda1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.659+0000 7fda297fa640 1 -- 172.21.15.59:0/3450832409 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda1c08a380 con 0x7fda2c1621c0 2024-03-06T07:22:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.659+0000 7fda30e7b640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1806b4d0 0x7fda1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.659+0000 7fda30e7b640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1806b4d0 0x7fda1806d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fda24011340 tx=0x7fda2400b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.663+0000 7fda297fa640 1 -- 172.21.15.59:0/3450832409 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda1c058c40 con 0x7fda2c1621c0 2024-03-06T07:22:18.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.847+0000 7fda330df640 1 -- 172.21.15.59:0/3450832409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda2c0b50b0 con 0x7fda2c1621c0 2024-03-06T07:22:18.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.848+0000 7fda297fa640 1 -- 172.21.15.59:0/3450832409 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fda1c05cce0 con 0x7fda2c1621c0 2024-03-06T07:22:18.850 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:18.850 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:18.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.850+0000 7fda330df640 1 -- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1806b4d0 msgr2=0x7fda1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.850+0000 7fda330df640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1806b4d0 0x7fda1806d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fda24011340 tx=0x7fda2400b040 comp rx=0 tx=0).stop 2024-03-06T07:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.850+0000 7fda330df640 1 -- 172.21.15.59:0/3450832409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda2c1621c0 msgr2=0x7fda2c0cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.850+0000 7fda330df640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda2c1621c0 0x7fda2c0cbf20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00e7a0 tx=0x7fda1c00ec60 comp rx=0 tx=0).stop 2024-03-06T07:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.850+0000 7fda330df640 1 -- 172.21.15.59:0/3450832409 shutdown_connections 2024-03-06T07:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.850+0000 7fda330df640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1806b4d0 0x7fda1806d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.850+0000 7fda330df640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda2c1621c0 0x7fda2c0cbf20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.850+0000 7fda330df640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda2c0d4920 0x7fda2c1e8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.850+0000 7fda330df640 1 --2- 172.21.15.59:0/3450832409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda2c0d2bf0 0x7fda2c0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.850+0000 7fda330df640 1 -- 172.21.15.59:0/3450832409 >> 172.21.15.59:0/3450832409 conn(0x7fda2c0d3d10 msgr2=0x7fda2c0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.851+0000 7fda330df640 1 -- 172.21.15.59:0/3450832409 shutdown_connections 2024-03-06T07:22:18.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.851+0000 7fda330df640 1 -- 172.21.15.59:0/3450832409 wait complete. 2024-03-06T07:22:18.864 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:18.864 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:18.962 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:18.964 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:18.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.967+0000 7f9cab513640 1 Processor -- start 2024-03-06T07:22:18.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.967+0000 7f9cab513640 1 -- start start 2024-03-06T07:22:18.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9cab513640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca41614f0 0x7f9ca40c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9cab513640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9ca406c860 0x7f9ca41619d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9cab513640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca406cd20 con 0x7f9ca4161f10 2024-03-06T07:22:18.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9cab513640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca406d000 con 0x7f9ca406c860 2024-03-06T07:22:18.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9cab513640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca406d2e0 con 0x7f9ca41614f0 2024-03-06T07:22:18.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9ca8aae640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9ca406c860 0x7f9ca41619d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9ca92af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca41614f0 0x7f9ca40c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9ca9ab0640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9ca4161f10 0x7f9ca415e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:54000/0 (socket says 172.21.15.59:54000) 2024-03-06T07:22:18.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9ca9ab0640 1 -- 172.21.15.59:0/3821316694 learned_addr learned my addr 172.21.15.59:0/3821316694 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:18.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9ca8aae640 1 --2- 172.21.15.59:0/3821316694 >> v2:172.21.15.164:3300/0 conn(0x7f9ca406c860 0x7f9ca41619d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:58254/0 (socket says 172.21.15.59:58254) 2024-03-06T07:22:18.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9ca8aae640 1 -- 172.21.15.59:0/3821316694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca41614f0 msgr2=0x7f9ca40c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9ca8aae640 1 --2- 172.21.15.59:0/3821316694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca41614f0 0x7f9ca40c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9ca8aae640 1 -- 172.21.15.59:0/3821316694 >> v1:172.21.15.164:6789/0 conn(0x7f9ca4161f10 legacy=0x7f9ca415e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9ca8aae640 1 -- 172.21.15.59:0/3821316694 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca406d5c0 con 0x7f9ca406c860 2024-03-06T07:22:18.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9c9bfff640 1 -- 172.21.15.59:0/3821316694 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372806094 0 0) 0x7f9ca406cd20 con 0x7f9ca4161f10 2024-03-06T07:22:18.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.968+0000 7f9ca92af640 1 --2- 172.21.15.59:0/3821316694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca41614f0 0x7f9ca40c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.969+0000 7f9ca8aae640 1 --2- 172.21.15.59:0/3821316694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca406c860 0x7f9ca41619d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c940097a0 tx=0x7f9c9402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95432c1d04fffc08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.969+0000 7f9c9bfff640 1 -- 172.21.15.59:0/3821316694 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c940315a0 con 0x7f9ca406c860 2024-03-06T07:22:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.969+0000 7f9c9bfff640 1 -- 172.21.15.59:0/3821316694 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c94037070 con 0x7f9ca406c860 2024-03-06T07:22:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.969+0000 7f9c9bfff640 1 -- 172.21.15.59:0/3821316694 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94031960 con 0x7f9ca406c860 2024-03-06T07:22:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.970+0000 7f9cab513640 1 -- 172.21.15.59:0/3821316694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca406c860 msgr2=0x7f9ca41619d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.970+0000 7f9cab513640 1 --2- 172.21.15.59:0/3821316694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca406c860 0x7f9ca41619d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c940097a0 tx=0x7f9c9402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.970+0000 7f9cab513640 1 -- 172.21.15.59:0/3821316694 shutdown_connections 2024-03-06T07:22:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.970+0000 7f9cab513640 1 --2- 172.21.15.59:0/3821316694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca406c860 0x7f9ca41619d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.970+0000 7f9cab513640 1 --2- 172.21.15.59:0/3821316694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca41614f0 0x7f9ca40c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.970+0000 7f9cab513640 1 -- 172.21.15.59:0/3821316694 >> 172.21.15.59:0/3821316694 conn(0x7f9ca40d1b40 msgr2=0x7f9ca40d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.970+0000 7f9cab513640 1 -- 172.21.15.59:0/3821316694 shutdown_connections 2024-03-06T07:22:18.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.970+0000 7f9cab513640 1 -- 172.21.15.59:0/3821316694 wait complete. 2024-03-06T07:22:18.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.971+0000 7f9cab513640 1 Processor -- start 2024-03-06T07:22:18.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.971+0000 7f9cab513640 1 -- start start 2024-03-06T07:22:18.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.972+0000 7f9cab513640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca406c860 0x7f9ca41598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.972+0000 7f9cab513640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca41614f0 0x7f9ca4140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.972+0000 7f9cab513640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca4161f10 0x7f9ca4158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.972+0000 7f9cab513640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40c7cf0 con 0x7f9ca406c860 2024-03-06T07:22:18.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.972+0000 7f9cab513640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40d28f0 con 0x7f9ca41614f0 2024-03-06T07:22:18.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.972+0000 7f9cab513640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca4140830 con 0x7f9ca4161f10 2024-03-06T07:22:18.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.972+0000 7f9ca92af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca406c860 0x7f9ca41598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.972+0000 7f9ca92af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca406c860 0x7f9ca41598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42876/0 (socket says 172.21.15.59:42876) 2024-03-06T07:22:18.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.972+0000 7f9ca92af640 1 -- 172.21.15.59:0/3151369235 learned_addr learned my addr 172.21.15.59:0/3151369235 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:18.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.972+0000 7f9ca9ab0640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca4161f10 0x7f9ca4158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.972+0000 7f9ca8aae640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca41614f0 0x7f9ca4140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.973+0000 7f9ca92af640 1 -- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca4161f10 msgr2=0x7f9ca4158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.973+0000 7f9ca92af640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca4161f10 0x7f9ca4158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.973+0000 7f9ca92af640 1 -- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca41614f0 msgr2=0x7f9ca4140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.973+0000 7f9ca92af640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca41614f0 0x7f9ca4140020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.973+0000 7f9ca92af640 1 -- 172.21.15.59:0/3151369235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca4140560 con 0x7f9ca406c860 2024-03-06T07:22:18.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.973+0000 7f9ca92af640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca406c860 0x7f9ca41598b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000e990 tx=0x7f9ca000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.973+0000 7f9c99ffb640 1 -- 172.21.15.59:0/3151369235 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0017070 con 0x7f9ca406c860 2024-03-06T07:22:18.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.973+0000 7f9c99ffb640 1 -- 172.21.15.59:0/3151369235 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca000c9e0 con 0x7f9ca406c860 2024-03-06T07:22:18.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.973+0000 7f9c99ffb640 1 -- 172.21.15.59:0/3151369235 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca000cdb0 con 0x7f9ca406c860 2024-03-06T07:22:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.973+0000 7f9cab513640 1 -- 172.21.15.59:0/3151369235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca415e390 con 0x7f9ca406c860 2024-03-06T07:22:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.973+0000 7f9cab513640 1 -- 172.21.15.59:0/3151369235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca415e630 con 0x7f9ca406c860 2024-03-06T07:22:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.974+0000 7f9cab513640 1 -- 172.21.15.59:0/3151369235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca409fdf0 con 0x7f9ca406c860 2024-03-06T07:22:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.975+0000 7f9c99ffb640 1 -- 172.21.15.59:0/3151369235 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ca001a070 con 0x7f9ca406c860 2024-03-06T07:22:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.975+0000 7f9c99ffb640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b500 0x7f9c8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.975+0000 7f9c99ffb640 1 -- 172.21.15.59:0/3151369235 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ca008a050 con 0x7f9ca406c860 2024-03-06T07:22:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.975+0000 7f9ca8aae640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b500 0x7f9c8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.976+0000 7f9ca8aae640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b500 0x7f9c8806d9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c94036000 tx=0x7f9c94008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:18.979+0000 7f9c99ffb640 1 -- 172.21.15.59:0/3151369235 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ca0058880 con 0x7f9ca406c860 2024-03-06T07:22:19.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.160+0000 7f9cab513640 1 -- 172.21.15.59:0/3151369235 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ca4068ff0 con 0x7f9ca406c860 2024-03-06T07:22:19.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.161+0000 7f9c99ffb640 1 -- 172.21.15.59:0/3151369235 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ca005c920 con 0x7f9ca406c860 2024-03-06T07:22:19.163 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:19.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 -- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b500 msgr2=0x7f9c8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b500 0x7f9c8806d9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c94036000 tx=0x7f9c94008030 comp rx=0 tx=0).stop 2024-03-06T07:22:19.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 -- 172.21.15.59:0/3151369235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca406c860 msgr2=0x7f9ca41598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca406c860 0x7f9ca41598b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000e990 tx=0x7f9ca000ee50 comp rx=0 tx=0).stop 2024-03-06T07:22:19.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 -- 172.21.15.59:0/3151369235 shutdown_connections 2024-03-06T07:22:19.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b500 0x7f9c8806d9c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca4161f10 0x7f9ca4158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca41614f0 0x7f9ca4140020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 --2- 172.21.15.59:0/3151369235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca406c860 0x7f9ca41598b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 -- 172.21.15.59:0/3151369235 >> 172.21.15.59:0/3151369235 conn(0x7f9ca40d1b40 msgr2=0x7f9ca40d1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 -- 172.21.15.59:0/3151369235 shutdown_connections 2024-03-06T07:22:19.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.163+0000 7f9cab513640 1 -- 172.21.15.59:0/3151369235 wait complete. 2024-03-06T07:22:19.175 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.278+0000 7f9776d81640 1 Processor -- start 2024-03-06T07:22:19.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.278+0000 7f9776d81640 1 -- start start 2024-03-06T07:22:19.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.278+0000 7f9776d81640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f977015a220 0x7f97700cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.278+0000 7f9776d81640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 0x7f9770071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.278+0000 7f9776d81640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97700b3320 con 0x7f97700712d0 2024-03-06T07:22:19.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.278+0000 7f9776d81640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97700cf610 con 0x7f977015a220 2024-03-06T07:22:19.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.278+0000 7f9776d81640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9770071c70 con 0x7f977006da00 2024-03-06T07:22:19.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f977531e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 0x7f9770071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f9774b1d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f977015a220 0x7f97700cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f976ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f97700712d0 0x7f977015de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:54006/0 (socket says 172.21.15.59:54006) 2024-03-06T07:22:19.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f977531e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 0x7f9770071730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:45916/0 (socket says 172.21.15.59:45916) 2024-03-06T07:22:19.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f976ffff640 1 -- 172.21.15.59:0/418452833 learned_addr learned my addr 172.21.15.59:0/418452833 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:19.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f9774b1d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f977015a220 0x7f97700cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:58270/0 (socket says 172.21.15.59:58270) 2024-03-06T07:22:19.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f9774b1d640 1 -- 172.21.15.59:0/418452833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 msgr2=0x7f9770071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f9774b1d640 1 --2- 172.21.15.59:0/418452833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 0x7f9770071730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f9774b1d640 1 -- 172.21.15.59:0/418452833 >> v1:172.21.15.164:6789/0 conn(0x7f97700712d0 legacy=0x7f977015de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f976f7fe640 1 -- 172.21.15.59:0/418452833 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126283172 0 0) 0x7f97700b3320 con 0x7f97700712d0 2024-03-06T07:22:19.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f9774b1d640 1 -- 172.21.15.59:0/418452833 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97700728d0 con 0x7f977015a220 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.279+0000 7f977531e640 1 --2- 172.21.15.59:0/418452833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 0x7f9770071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.280+0000 7f9774b1d640 1 --2- 172.21.15.59:0/418452833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f977015a220 0x7f97700cf0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9768009d90 tx=0x7f976802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19f69a5fb2481c09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.280+0000 7f976f7fe640 1 -- 172.21.15.59:0/418452833 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9768039070 con 0x7f977015a220 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.280+0000 7f976f7fe640 1 -- 172.21.15.59:0/418452833 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976802cbc0 con 0x7f977015a220 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.280+0000 7f976f7fe640 1 -- 172.21.15.59:0/418452833 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9768034440 con 0x7f977015a220 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.280+0000 7f9776d81640 1 -- 172.21.15.59:0/418452833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f977015a220 msgr2=0x7f97700cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.280+0000 7f9776d81640 1 --2- 172.21.15.59:0/418452833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f977015a220 0x7f97700cf0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9768009d90 tx=0x7f976802be10 comp rx=0 tx=0).stop 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.281+0000 7f9776d81640 1 -- 172.21.15.59:0/418452833 shutdown_connections 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.281+0000 7f9776d81640 1 --2- 172.21.15.59:0/418452833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 0x7f9770071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.281+0000 7f9776d81640 1 --2- 172.21.15.59:0/418452833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f977015a220 0x7f97700cf0d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.281+0000 7f9776d81640 1 -- 172.21.15.59:0/418452833 >> 172.21.15.59:0/418452833 conn(0x7f977006b190 msgr2=0x7f977006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.281+0000 7f9776d81640 1 -- 172.21.15.59:0/418452833 shutdown_connections 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.281+0000 7f9776d81640 1 -- 172.21.15.59:0/418452833 wait complete. 2024-03-06T07:22:19.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.282+0000 7f9776d81640 1 Processor -- start 2024-03-06T07:22:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.282+0000 7f9776d81640 1 -- start start 2024-03-06T07:22:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f9776d81640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 0x7f97700ad590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f9776d81640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97700712d0 0x7f97700cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f9776d81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f977015a220 0x7f977014f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f9776d81640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9770072030 con 0x7f977015a220 2024-03-06T07:22:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f9776d81640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9770072bb0 con 0x7f97700712d0 2024-03-06T07:22:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f9776d81640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9770072530 con 0x7f977006da00 2024-03-06T07:22:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f977531e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f977015a220 0x7f977014f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f977531e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f977015a220 0x7f977014f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42880/0 (socket says 172.21.15.59:42880) 2024-03-06T07:22:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f977531e640 1 -- 172.21.15.59:0/4036238996 learned_addr learned my addr 172.21.15.59:0/4036238996 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f9774b1d640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 0x7f97700ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f976ffff640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97700712d0 0x7f97700cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f977531e640 1 -- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 msgr2=0x7f97700ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f977531e640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 0x7f97700ad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f977531e640 1 -- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97700712d0 msgr2=0x7f97700cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f977531e640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97700712d0 0x7f97700cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f977531e640 1 -- 172.21.15.59:0/4036238996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97700cce20 con 0x7f977015a220 2024-03-06T07:22:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f977531e640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f977015a220 0x7f977014f0e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f975800e900 tx=0x7f975800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f976d7fa640 1 -- 172.21.15.59:0/4036238996 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9758017070 con 0x7f977015a220 2024-03-06T07:22:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f976d7fa640 1 -- 172.21.15.59:0/4036238996 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975800ca00 con 0x7f977015a220 2024-03-06T07:22:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f976d7fa640 1 -- 172.21.15.59:0/4036238996 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975800cdf0 con 0x7f977015a220 2024-03-06T07:22:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.283+0000 7f9776d81640 1 -- 172.21.15.59:0/4036238996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97700ac7d0 con 0x7f977015a220 2024-03-06T07:22:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.284+0000 7f9776d81640 1 -- 172.21.15.59:0/4036238996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97700acd90 con 0x7f977015a220 2024-03-06T07:22:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.284+0000 7f9776d81640 1 -- 172.21.15.59:0/4036238996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9770068ff0 con 0x7f977015a220 2024-03-06T07:22:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.285+0000 7f976d7fa640 1 -- 172.21.15.59:0/4036238996 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f975801a070 con 0x7f977015a220 2024-03-06T07:22:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.285+0000 7f976d7fa640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f975006b4d0 0x7f975006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.285+0000 7f976d7fa640 1 -- 172.21.15.59:0/4036238996 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f975808a0a0 con 0x7f977015a220 2024-03-06T07:22:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.285+0000 7f9774b1d640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f975006b4d0 0x7f975006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.286+0000 7f9774b1d640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f975006b4d0 0x7f975006d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9768036000 tx=0x7f9768030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.289+0000 7f976d7fa640 1 -- 172.21.15.59:0/4036238996 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97580588c0 con 0x7f977015a220 2024-03-06T07:22:19.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.471+0000 7f9776d81640 1 -- 172.21.15.59:0/4036238996 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9770070ba0 con 0x7f977015a220 2024-03-06T07:22:19.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.471+0000 7f976d7fa640 1 -- 172.21.15.59:0/4036238996 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f975805c960 con 0x7f977015a220 2024-03-06T07:22:19.473 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:19.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 -- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f975006b4d0 msgr2=0x7f975006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f975006b4d0 0x7f975006d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9768036000 tx=0x7f9768030000 comp rx=0 tx=0).stop 2024-03-06T07:22:19.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 -- 172.21.15.59:0/4036238996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f977015a220 msgr2=0x7f977014f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f977015a220 0x7f977014f0e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f975800e900 tx=0x7f975800edc0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 -- 172.21.15.59:0/4036238996 shutdown_connections 2024-03-06T07:22:19.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f975006b4d0 0x7f975006d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f977015a220 0x7f977014f0e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97700712d0 0x7f97700cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 --2- 172.21.15.59:0/4036238996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f977006da00 0x7f97700ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 -- 172.21.15.59:0/4036238996 >> 172.21.15.59:0/4036238996 conn(0x7f977006b190 msgr2=0x7f9770159460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 -- 172.21.15.59:0/4036238996 shutdown_connections 2024-03-06T07:22:19.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.474+0000 7f9776d81640 1 -- 172.21.15.59:0/4036238996 wait complete. 2024-03-06T07:22:19.487 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.591+0000 7f83b231e640 1 Processor -- start 2024-03-06T07:22:19.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.591+0000 7f83b231e640 1 -- start start 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83b231e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f83ac0cf9a0 0x7f83ac0b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83b231e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac066d90 0x7f83ac0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83b231e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac06b710 con 0x7f83ac0b5410 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83b231e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ac06b150 con 0x7f83ac0cf9a0 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83b231e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac06b430 con 0x7f83ac066d90 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83ab7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac066d90 0x7f83ac0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83abfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f83ac0cf9a0 0x7f83ac0b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83b08bb640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f83ac0b5410 0x7f83ac072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:54008/0 (socket says 172.21.15.59:54008) 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83abfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f83ac0cf9a0 0x7f83ac0b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:58294/0 (socket says 172.21.15.59:58294) 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83ab7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac066d90 0x7f83ac0b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:45930/0 (socket says 172.21.15.59:45930) 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83abfff640 1 -- 172.21.15.59:0/2901074506 learned_addr learned my addr 172.21.15.59:0/2901074506 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83abfff640 1 -- 172.21.15.59:0/2901074506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac066d90 msgr2=0x7f83ac0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83abfff640 1 --2- 172.21.15.59:0/2901074506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac066d90 0x7f83ac0b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83abfff640 1 -- 172.21.15.59:0/2901074506 >> v1:172.21.15.164:6789/0 conn(0x7f83ac0b5410 legacy=0x7f83ac072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83abfff640 1 -- 172.21.15.59:0/2901074506 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac0b0970 con 0x7f83ac0cf9a0 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83aaffd640 1 -- 172.21.15.59:0/2901074506 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393409450 0 0) 0x7f83ac06b710 con 0x7f83ac0b5410 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.592+0000 7f83ab7fe640 1 --2- 172.21.15.59:0/2901074506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac066d90 0x7f83ac0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.593+0000 7f83abfff640 1 --2- 172.21.15.59:0/2901074506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83ac0cf9a0 0x7f83ac0b5ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009870 tx=0x7f83a002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4c2aab81da4ebea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.593+0000 7f83aaffd640 1 -- 172.21.15.59:0/2901074506 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0034070 con 0x7f83ac0cf9a0 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.593+0000 7f83aaffd640 1 -- 172.21.15.59:0/2901074506 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a0030840 con 0x7f83ac0cf9a0 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.593+0000 7f83aaffd640 1 -- 172.21.15.59:0/2901074506 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0030c10 con 0x7f83ac0cf9a0 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.594+0000 7f83b231e640 1 -- 172.21.15.59:0/2901074506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83ac0cf9a0 msgr2=0x7f83ac0b5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.594+0000 7f83b231e640 1 --2- 172.21.15.59:0/2901074506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83ac0cf9a0 0x7f83ac0b5ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009870 tx=0x7f83a002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.594+0000 7f83b231e640 1 -- 172.21.15.59:0/2901074506 shutdown_connections 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.594+0000 7f83b231e640 1 --2- 172.21.15.59:0/2901074506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac066d90 0x7f83ac0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.594+0000 7f83b231e640 1 --2- 172.21.15.59:0/2901074506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83ac0cf9a0 0x7f83ac0b5ab0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.594+0000 7f83b231e640 1 -- 172.21.15.59:0/2901074506 >> 172.21.15.59:0/2901074506 conn(0x7f83ac0b1d70 msgr2=0x7f83ac0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.594+0000 7f83b231e640 1 -- 172.21.15.59:0/2901074506 shutdown_connections 2024-03-06T07:22:19.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.594+0000 7f83b231e640 1 -- 172.21.15.59:0/2901074506 wait complete. 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.595+0000 7f83b231e640 1 Processor -- start 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.595+0000 7f83b231e640 1 -- start start 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83b231e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83ac066d90 0x7f83ac1531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83b231e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac0b5410 0x7f83ac15cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83b231e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac0cf9a0 0x7f83ac15d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83b231e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0b6170 con 0x7f83ac0b5410 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83b231e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0b5ff0 con 0x7f83ac066d90 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83b231e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0cf050 con 0x7f83ac0cf9a0 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83ab7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac0b5410 0x7f83ac15cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83ab7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac0b5410 0x7f83ac15cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42894/0 (socket says 172.21.15.59:42894) 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83ab7fe640 1 -- 172.21.15.59:0/562369347 learned_addr learned my addr 172.21.15.59:0/562369347 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83b08bb640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac0cf9a0 0x7f83ac15d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83abfff640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83ac066d90 0x7f83ac1531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83ab7fe640 1 -- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac0cf9a0 msgr2=0x7f83ac15d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83ab7fe640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac0cf9a0 0x7f83ac15d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83ab7fe640 1 -- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83ac066d90 msgr2=0x7f83ac1531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83ab7fe640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83ac066d90 0x7f83ac1531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83ab7fe640 1 -- 172.21.15.59:0/562369347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac1f7410 con 0x7f83ac0b5410 2024-03-06T07:22:19.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.596+0000 7f83ab7fe640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac0b5410 0x7f83ac15cd60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f839c00e9e0 tx=0x7f839c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.597+0000 7f83a8ff9640 1 -- 172.21.15.59:0/562369347 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c017070 con 0x7f83ac0b5410 2024-03-06T07:22:19.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.597+0000 7f83a8ff9640 1 -- 172.21.15.59:0/562369347 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c00c9d0 con 0x7f83ac0b5410 2024-03-06T07:22:19.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.597+0000 7f83b231e640 1 -- 172.21.15.59:0/562369347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac1f4420 con 0x7f83ac0b5410 2024-03-06T07:22:19.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.597+0000 7f83a8ff9640 1 -- 172.21.15.59:0/562369347 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c00cdc0 con 0x7f83ac0b5410 2024-03-06T07:22:19.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.597+0000 7f83b231e640 1 -- 172.21.15.59:0/562369347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac1f49e0 con 0x7f83ac0b5410 2024-03-06T07:22:19.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.598+0000 7f83b231e640 1 -- 172.21.15.59:0/562369347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac09e3d0 con 0x7f83ac0b5410 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.598+0000 7f83a8ff9640 1 -- 172.21.15.59:0/562369347 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f839c01a070 con 0x7f83ac0b5410 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.598+0000 7f83a8ff9640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f839806b4d0 0x7f839806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.599+0000 7f83a8ff9640 1 -- 172.21.15.59:0/562369347 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f839c089cc0 con 0x7f83ac0b5410 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.599+0000 7f83abfff640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f839806b4d0 0x7f839806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.599+0000 7f83abfff640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f839806b4d0 0x7f839806d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f83a0036000 tx=0x7f83a003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.603+0000 7f83a8ff9640 1 -- 172.21.15.59:0/562369347 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f839c058470 con 0x7f83ac0b5410 2024-03-06T07:22:19.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.784+0000 7f83b231e640 1 -- 172.21.15.59:0/562369347 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f83ac0d0150 con 0x7f83ac0b5410 2024-03-06T07:22:19.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.784+0000 7f83a8ff9640 1 -- 172.21.15.59:0/562369347 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f839c05c510 con 0x7f83ac0b5410 2024-03-06T07:22:19.786 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:19.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 -- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f839806b4d0 msgr2=0x7f839806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f839806b4d0 0x7f839806d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f83a0036000 tx=0x7f83a003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:19.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 -- 172.21.15.59:0/562369347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac0b5410 msgr2=0x7f83ac15cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac0b5410 0x7f83ac15cd60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f839c00e9e0 tx=0x7f839c00eea0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 -- 172.21.15.59:0/562369347 shutdown_connections 2024-03-06T07:22:19.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f839806b4d0 0x7f839806d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83ac0cf9a0 0x7f83ac15d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83ac0b5410 0x7f83ac15cd60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 --2- 172.21.15.59:0/562369347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83ac066d90 0x7f83ac1531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 -- 172.21.15.59:0/562369347 >> 172.21.15.59:0/562369347 conn(0x7f83ac0b1d70 msgr2=0x7f83ac0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 -- 172.21.15.59:0/562369347 shutdown_connections 2024-03-06T07:22:19.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.786+0000 7f83b231e640 1 -- 172.21.15.59:0/562369347 wait complete. 2024-03-06T07:22:19.799 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:19.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.898+0000 7fc8f4dac640 1 Processor -- start 2024-03-06T07:22:19.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.898+0000 7fc8f4dac640 1 -- start start 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.899+0000 7fc8f4dac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc8f015dfe0 0x7fc8f00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.899+0000 7fc8f4dac640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc8f00d29c0 0x7fc8f00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.899+0000 7fc8f4dac640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f00b3430 con 0x7fc8f00cf830 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.899+0000 7fc8f4dac640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f00b31e0 con 0x7fc8f015dfe0 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.899+0000 7fc8f4dac640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f00b3d60 con 0x7fc8f00d29c0 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.899+0000 7fc8ee59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc8f015dfe0 0x7fc8f00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.899+0000 7fc8ee59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc8f015dfe0 0x7fc8f00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42906/0 (socket says 172.21.15.59:42906) 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.899+0000 7fc8ee59c640 1 -- 172.21.15.59:0/3260796196 learned_addr learned my addr 172.21.15.59:0/3260796196 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.899+0000 7fc8eed9d640 1 --2- 172.21.15.59:0/3260796196 >> v2:172.21.15.164:3300/0 conn(0x7fc8f00d29c0 0x7fc8f00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.900+0000 7fc8ee59c640 1 -- 172.21.15.59:0/3260796196 >> v2:172.21.15.164:3300/0 conn(0x7fc8f00d29c0 msgr2=0x7fc8f00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.900+0000 7fc8ee59c640 1 --2- 172.21.15.59:0/3260796196 >> v2:172.21.15.164:3300/0 conn(0x7fc8f00d29c0 0x7fc8f00b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.900+0000 7fc8ee59c640 1 -- 172.21.15.59:0/3260796196 >> v1:172.21.15.164:6789/0 conn(0x7fc8f00cf830 legacy=0x7fc8f00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.900+0000 7fc8ee59c640 1 -- 172.21.15.59:0/3260796196 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f00b3710 con 0x7fc8f015dfe0 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.900+0000 7fc8ee59c640 1 --2- 172.21.15.59:0/3260796196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f015dfe0 0x7fc8f00cf2f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc8f00b3cd0 tx=0x7fc8e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91261836739458a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.901+0000 7fc8ed59a640 1 -- 172.21.15.59:0/3260796196 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8034070 con 0x7fc8f015dfe0 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.901+0000 7fc8ed59a640 1 -- 172.21.15.59:0/3260796196 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e8030860 con 0x7fc8f015dfe0 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.901+0000 7fc8ed59a640 1 -- 172.21.15.59:0/3260796196 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8030c30 con 0x7fc8f015dfe0 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.901+0000 7fc8f4dac640 1 -- 172.21.15.59:0/3260796196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f015dfe0 msgr2=0x7fc8f00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.901+0000 7fc8f4dac640 1 --2- 172.21.15.59:0/3260796196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f015dfe0 0x7fc8f00cf2f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc8f00b3cd0 tx=0x7fc8e802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.901+0000 7fc8f4dac640 1 -- 172.21.15.59:0/3260796196 shutdown_connections 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.901+0000 7fc8f4dac640 1 --2- 172.21.15.59:0/3260796196 >> v2:172.21.15.164:3300/0 conn(0x7fc8f00d29c0 0x7fc8f00b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.901+0000 7fc8f4dac640 1 --2- 172.21.15.59:0/3260796196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f015dfe0 0x7fc8f00cf2f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.901+0000 7fc8f4dac640 1 -- 172.21.15.59:0/3260796196 >> 172.21.15.59:0/3260796196 conn(0x7fc8f00d3cd0 msgr2=0x7fc8f00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.901+0000 7fc8f4dac640 1 -- 172.21.15.59:0/3260796196 shutdown_connections 2024-03-06T07:22:19.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.901+0000 7fc8f4dac640 1 -- 172.21.15.59:0/3260796196 wait complete. 2024-03-06T07:22:19.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.902+0000 7fc8f4dac640 1 Processor -- start 2024-03-06T07:22:19.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8f4dac640 1 -- start start 2024-03-06T07:22:19.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8f4dac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f00cf830 0x7fc8f00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8f4dac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8f00d29c0 0x7fc8f00b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8f4dac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8f015dfe0 0x7fc8f00c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8f4dac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f00d1df0 con 0x7fc8f00cf830 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8f4dac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f00d1c70 con 0x7fc8f015dfe0 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8f4dac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f015e3c0 con 0x7fc8f00d29c0 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8ee59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f00cf830 0x7fc8f00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8ee59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f00cf830 0x7fc8f00cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42908/0 (socket says 172.21.15.59:42908) 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8ee59c640 1 -- 172.21.15.59:0/2151094562 learned_addr learned my addr 172.21.15.59:0/2151094562 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8edd9b640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8f00d29c0 0x7fc8f00b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.903+0000 7fc8eed9d640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8f015dfe0 0x7fc8f00c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.904+0000 7fc8ee59c640 1 -- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8f00d29c0 msgr2=0x7fc8f00b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.904+0000 7fc8ee59c640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8f00d29c0 0x7fc8f00b4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.904+0000 7fc8ee59c640 1 -- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8f015dfe0 msgr2=0x7fc8f00c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.904+0000 7fc8ee59c640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8f015dfe0 0x7fc8f00c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.904+0000 7fc8ee59c640 1 -- 172.21.15.59:0/2151094562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f015fef0 con 0x7fc8f00cf830 2024-03-06T07:22:19.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.904+0000 7fc8ee59c640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f00cf830 0x7fc8f00cada0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc8e802b9e0 tx=0x7fc8e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.904+0000 7fc8e6ffd640 1 -- 172.21.15.59:0/2151094562 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8034040 con 0x7fc8f00cf830 2024-03-06T07:22:19.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.904+0000 7fc8e6ffd640 1 -- 172.21.15.59:0/2151094562 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e8039d90 con 0x7fc8f00cf830 2024-03-06T07:22:19.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.904+0000 7fc8f4dac640 1 -- 172.21.15.59:0/2151094562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f00cdcf0 con 0x7fc8f00cf830 2024-03-06T07:22:19.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.904+0000 7fc8e6ffd640 1 -- 172.21.15.59:0/2151094562 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8043610 con 0x7fc8f00cf830 2024-03-06T07:22:19.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.904+0000 7fc8f4dac640 1 -- 172.21.15.59:0/2151094562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f0160190 con 0x7fc8f00cf830 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.905+0000 7fc8f4dac640 1 -- 172.21.15.59:0/2151094562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f015fcc0 con 0x7fc8f00cf830 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.905+0000 7fc8e6ffd640 1 -- 172.21.15.59:0/2151094562 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8e804a060 con 0x7fc8f00cf830 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.906+0000 7fc8e6ffd640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc8d006b4d0 0x7fc8d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.906+0000 7fc8e6ffd640 1 -- 172.21.15.59:0/2151094562 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8e80ac110 con 0x7fc8f00cf830 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.906+0000 7fc8edd9b640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc8d006b4d0 0x7fc8d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.906+0000 7fc8edd9b640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc8d006b4d0 0x7fc8d006d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc8f00b5a30 tx=0x7fc8d8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:19.910+0000 7fc8e6ffd640 1 -- 172.21.15.59:0/2151094562 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8e803e070 con 0x7fc8f00cf830 2024-03-06T07:22:20.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.093+0000 7fc8f4dac640 1 -- 172.21.15.59:0/2151094562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8f004f330 con 0x7fc8f00cf830 2024-03-06T07:22:20.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.094+0000 7fc8e6ffd640 1 -- 172.21.15.59:0/2151094562 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fc8e807a940 con 0x7fc8f00cf830 2024-03-06T07:22:20.096 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:20.096 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:20.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 -- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc8d006b4d0 msgr2=0x7fc8d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc8d006b4d0 0x7fc8d006d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc8f00b5a30 tx=0x7fc8d8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:20.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 -- 172.21.15.59:0/2151094562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f00cf830 msgr2=0x7fc8f00cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f00cf830 0x7fc8f00cada0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc8e802b9e0 tx=0x7fc8e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 -- 172.21.15.59:0/2151094562 shutdown_connections 2024-03-06T07:22:20.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc8d006b4d0 0x7fc8d006d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8f015dfe0 0x7fc8f00c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8f00d29c0 0x7fc8f00b4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 --2- 172.21.15.59:0/2151094562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8f00cf830 0x7fc8f00cada0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 -- 172.21.15.59:0/2151094562 >> 172.21.15.59:0/2151094562 conn(0x7fc8f00d3cd0 msgr2=0x7fc8f00cf030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 -- 172.21.15.59:0/2151094562 shutdown_connections 2024-03-06T07:22:20.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.096+0000 7fc8f4dac640 1 -- 172.21.15.59:0/2151094562 wait complete. 2024-03-06T07:22:20.109 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:20.109 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:20.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.211+0000 7f5c5060e640 1 Processor -- start 2024-03-06T07:22:20.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.212+0000 7f5c5060e640 1 -- start start 2024-03-06T07:22:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.212+0000 7f5c5060e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5c480b41b0 0x7f5c480cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.212+0000 7f5c5060e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c480b15c0 0x7f5c480b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.212+0000 7f5c5060e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c480d0240 con 0x7f5c4815bdd0 2024-03-06T07:22:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.212+0000 7f5c5060e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c480cf4f0 con 0x7f5c480b41b0 2024-03-06T07:22:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.212+0000 7f5c5060e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480d0850 con 0x7f5c480b15c0 2024-03-06T07:22:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.213+0000 7f5c4ebab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c480b15c0 0x7f5c480b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.213+0000 7f5c4dba9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5c480b41b0 0x7f5c480cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.213+0000 7f5c4e3aa640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5c4815bdd0 0x7f5c480ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:54028/0 (socket says 172.21.15.59:54028) 2024-03-06T07:22:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.213+0000 7f5c4e3aa640 1 -- 172.21.15.59:0/2635651323 learned_addr learned my addr 172.21.15.59:0/2635651323 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.213+0000 7f5c4ebab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c480b15c0 0x7f5c480b1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:45962/0 (socket says 172.21.15.59:45962) 2024-03-06T07:22:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.213+0000 7f5c4dba9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5c480b41b0 0x7f5c480cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:58324/0 (socket says 172.21.15.59:58324) 2024-03-06T07:22:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.213+0000 7f5c4ebab640 1 -- 172.21.15.59:0/2635651323 >> v2:172.21.15.164:3300/0 conn(0x7f5c480b41b0 msgr2=0x7f5c480cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.213+0000 7f5c4ebab640 1 --2- 172.21.15.59:0/2635651323 >> v2:172.21.15.164:3300/0 conn(0x7f5c480b41b0 0x7f5c480cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.213+0000 7f5c4d3a8640 1 -- 172.21.15.59:0/2635651323 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877188539 0 0) 0x7f5c480d0240 con 0x7f5c4815bdd0 2024-03-06T07:22:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.213+0000 7f5c4ebab640 1 -- 172.21.15.59:0/2635651323 >> v1:172.21.15.164:6789/0 conn(0x7f5c4815bdd0 legacy=0x7f5c480ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.213+0000 7f5c4ebab640 1 -- 172.21.15.59:0/2635651323 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c480cef50 con 0x7f5c480b15c0 2024-03-06T07:22:20.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.214+0000 7f5c4ebab640 1 --2- 172.21.15.59:0/2635651323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c480b15c0 0x7f5c480b1a10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5c38009d90 tx=0x7f5c3802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58efd4836f141a56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.214+0000 7f5c4d3a8640 1 -- 172.21.15.59:0/2635651323 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c38039070 con 0x7f5c480b15c0 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.214+0000 7f5c4d3a8640 1 -- 172.21.15.59:0/2635651323 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3802cc50 con 0x7f5c480b15c0 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.214+0000 7f5c4d3a8640 1 -- 172.21.15.59:0/2635651323 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c38034440 con 0x7f5c480b15c0 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.214+0000 7f5c5060e640 1 -- 172.21.15.59:0/2635651323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c480b15c0 msgr2=0x7f5c480b1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.214+0000 7f5c5060e640 1 --2- 172.21.15.59:0/2635651323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c480b15c0 0x7f5c480b1a10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5c38009d90 tx=0x7f5c3802bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.215+0000 7f5c5060e640 1 -- 172.21.15.59:0/2635651323 shutdown_connections 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.215+0000 7f5c5060e640 1 --2- 172.21.15.59:0/2635651323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c480b15c0 0x7f5c480b1a10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.215+0000 7f5c5060e640 1 --2- 172.21.15.59:0/2635651323 >> v2:172.21.15.164:3300/0 conn(0x7f5c480b41b0 0x7f5c480cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.215+0000 7f5c5060e640 1 -- 172.21.15.59:0/2635651323 >> 172.21.15.59:0/2635651323 conn(0x7f5c480b3690 msgr2=0x7f5c480b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.215+0000 7f5c5060e640 1 -- 172.21.15.59:0/2635651323 shutdown_connections 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.215+0000 7f5c5060e640 1 -- 172.21.15.59:0/2635651323 wait complete. 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.216+0000 7f5c5060e640 1 Processor -- start 2024-03-06T07:22:20.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.216+0000 7f5c5060e640 1 -- start start 2024-03-06T07:22:20.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.216+0000 7f5c5060e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c480b15c0 0x7f5c481f7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.216+0000 7f5c5060e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c480b41b0 0x7f5c48158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.216+0000 7f5c5060e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c4815bdd0 0x7f5c4815edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.216+0000 7f5c5060e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4815c1b0 con 0x7f5c480b15c0 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.216+0000 7f5c5060e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480d04e0 con 0x7f5c480b41b0 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.216+0000 7f5c5060e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480b10c0 con 0x7f5c4815bdd0 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4e3aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c480b15c0 0x7f5c481f7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4dba9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c480b41b0 0x7f5c48158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4ebab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c4815bdd0 0x7f5c4815edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4e3aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c480b15c0 0x7f5c481f7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42922/0 (socket says 172.21.15.59:42922) 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4e3aa640 1 -- 172.21.15.59:0/1366712926 learned_addr learned my addr 172.21.15.59:0/1366712926 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4ebab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c4815bdd0 0x7f5c4815edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:45972/0 (socket says 172.21.15.59:45972) 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4dba9640 1 -- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c4815bdd0 msgr2=0x7f5c4815edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4dba9640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c4815bdd0 0x7f5c4815edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4dba9640 1 -- 172.21.15.59:0/1366712926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c480b15c0 msgr2=0x7f5c481f7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4dba9640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c480b15c0 0x7f5c481f7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4dba9640 1 -- 172.21.15.59:0/1366712926 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c481f8480 con 0x7f5c480b41b0 2024-03-06T07:22:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4ebab640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c4815bdd0 0x7f5c4815edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4e3aa640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c480b15c0 0x7f5c481f7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c4dba9640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c480b41b0 0x7f5c48158fd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00e870 tx=0x7f5c3c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c46ffd640 1 -- 172.21.15.59:0/1366712926 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c017070 con 0x7f5c480b41b0 2024-03-06T07:22:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c46ffd640 1 -- 172.21.15.59:0/1366712926 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c00ee60 con 0x7f5c480b41b0 2024-03-06T07:22:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c5060e640 1 -- 172.21.15.59:0/1366712926 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4815dd80 con 0x7f5c480b41b0 2024-03-06T07:22:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c46ffd640 1 -- 172.21.15.59:0/1366712926 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00cb60 con 0x7f5c480b41b0 2024-03-06T07:22:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.217+0000 7f5c5060e640 1 -- 172.21.15.59:0/1366712926 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4815e5c0 con 0x7f5c480b41b0 2024-03-06T07:22:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.218+0000 7f5c5060e640 1 -- 172.21.15.59:0/1366712926 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c480b06b0 con 0x7f5c480b41b0 2024-03-06T07:22:20.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.219+0000 7f5c46ffd640 1 -- 172.21.15.59:0/1366712926 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c3c01a070 con 0x7f5c480b41b0 2024-03-06T07:22:20.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.219+0000 7f5c46ffd640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c3406b4d0 0x7f5c3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.219+0000 7f5c46ffd640 1 -- 172.21.15.59:0/1366712926 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c3c089ff0 con 0x7f5c480b41b0 2024-03-06T07:22:20.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.219+0000 7f5c4e3aa640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c3406b4d0 0x7f5c3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.220+0000 7f5c4e3aa640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c3406b4d0 0x7f5c3406d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c40000f60 tx=0x7f5c40007ea0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.229+0000 7f5c46ffd640 1 -- 172.21.15.59:0/1366712926 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c4815dd80 con 0x7f5c480b41b0 2024-03-06T07:22:20.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.412+0000 7f5c5060e640 1 -- 172.21.15.59:0/1366712926 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5c48068ff0 con 0x7f5c480b41b0 2024-03-06T07:22:20.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.412+0000 7f5c46ffd640 1 -- 172.21.15.59:0/1366712926 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5c481f8480 con 0x7f5c480b41b0 2024-03-06T07:22:20.414 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.414+0000 7f5c5060e640 1 -- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c3406b4d0 msgr2=0x7f5c3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.414+0000 7f5c5060e640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c3406b4d0 0x7f5c3406d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c40000f60 tx=0x7f5c40007ea0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.414+0000 7f5c5060e640 1 -- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c480b41b0 msgr2=0x7f5c48158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.414+0000 7f5c5060e640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c480b41b0 0x7f5c48158fd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00e870 tx=0x7f5c3c00ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.415+0000 7f5c5060e640 1 -- 172.21.15.59:0/1366712926 shutdown_connections 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.415+0000 7f5c5060e640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c3406b4d0 0x7f5c3406d990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.415+0000 7f5c5060e640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c4815bdd0 0x7f5c4815edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.415+0000 7f5c5060e640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c480b41b0 0x7f5c48158fd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.415+0000 7f5c5060e640 1 --2- 172.21.15.59:0/1366712926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c480b15c0 0x7f5c481f7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.415+0000 7f5c5060e640 1 -- 172.21.15.59:0/1366712926 >> 172.21.15.59:0/1366712926 conn(0x7f5c480b3690 msgr2=0x7f5c480b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.415+0000 7f5c5060e640 1 -- 172.21.15.59:0/1366712926 shutdown_connections 2024-03-06T07:22:20.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.415+0000 7f5c5060e640 1 -- 172.21.15.59:0/1366712926 wait complete. 2024-03-06T07:22:20.427 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:20.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.530+0000 7fb14db72640 1 Processor -- start 2024-03-06T07:22:20.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.530+0000 7fb14db72640 1 -- start start 2024-03-06T07:22:20.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.530+0000 7fb14db72640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb14806b6c0 0x7fb14815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.530+0000 7fb14db72640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb14815ec40 0x7fb148071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.530+0000 7fb14db72640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1480cea90 con 0x7fb14815bbb0 2024-03-06T07:22:20.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.530+0000 7fb14db72640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb148161970 con 0x7fb14806b6c0 2024-03-06T07:22:20.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.530+0000 7fb14db72640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1481624f0 con 0x7fb14815ec40 2024-03-06T07:22:20.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.531+0000 7fb1477fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb14815bbb0 0x7fb14806afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41950/0 (socket says 172.21.15.59:41950) 2024-03-06T07:22:20.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.531+0000 7fb1477fe640 1 -- 172.21.15.59:0/1511230642 learned_addr learned my addr 172.21.15.59:0/1511230642 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:20.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.531+0000 7fb147fff640 1 --2- 172.21.15.59:0/1511230642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb14815ec40 0x7fb148071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.531+0000 7fb146ffd640 1 --2- 172.21.15.59:0/1511230642 >> v2:172.21.15.164:3300/0 conn(0x7fb14806b6c0 0x7fb14815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.531+0000 7fb1467fc640 1 -- 172.21.15.59:0/1511230642 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109007713 0 0) 0x7fb138001110 con 0x7fb14815bbb0 2024-03-06T07:22:20.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.531+0000 7fb1467fc640 1 -- 172.21.15.59:0/1511230642 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb130003520 con 0x7fb14815bbb0 2024-03-06T07:22:20.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.531+0000 7fb147fff640 1 -- 172.21.15.59:0/1511230642 >> v2:172.21.15.164:3300/0 conn(0x7fb14806b6c0 msgr2=0x7fb14815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.531+0000 7fb147fff640 1 --2- 172.21.15.59:0/1511230642 >> v2:172.21.15.164:3300/0 conn(0x7fb14806b6c0 0x7fb14815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.531+0000 7fb147fff640 1 -- 172.21.15.59:0/1511230642 >> v1:172.21.15.59:6789/0 conn(0x7fb14815bbb0 legacy=0x7fb14806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.531+0000 7fb147fff640 1 -- 172.21.15.59:0/1511230642 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb148161bc0 con 0x7fb14815ec40 2024-03-06T07:22:20.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.531+0000 7fb146ffd640 1 --2- 172.21.15.59:0/1511230642 >> v2:172.21.15.164:3300/0 conn(0x7fb14806b6c0 0x7fb14815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:20.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.532+0000 7fb147fff640 1 --2- 172.21.15.59:0/1511230642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb14815ec40 0x7fb148071ca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb140009d90 tx=0x7fb14002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eae5e17f88ed23e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.532+0000 7fb1467fc640 1 -- 172.21.15.59:0/1511230642 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb140039070 con 0x7fb14815ec40 2024-03-06T07:22:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.532+0000 7fb1467fc640 1 -- 172.21.15.59:0/1511230642 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14002cc50 con 0x7fb14815ec40 2024-03-06T07:22:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.532+0000 7fb1467fc640 1 -- 172.21.15.59:0/1511230642 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb140034440 con 0x7fb14815ec40 2024-03-06T07:22:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.532+0000 7fb14db72640 1 -- 172.21.15.59:0/1511230642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb14815ec40 msgr2=0x7fb148071ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.532+0000 7fb14db72640 1 --2- 172.21.15.59:0/1511230642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb14815ec40 0x7fb148071ca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb140009d90 tx=0x7fb14002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.532+0000 7fb14db72640 1 -- 172.21.15.59:0/1511230642 shutdown_connections 2024-03-06T07:22:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.532+0000 7fb14db72640 1 --2- 172.21.15.59:0/1511230642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb14815ec40 0x7fb148071ca0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.532+0000 7fb14db72640 1 --2- 172.21.15.59:0/1511230642 >> v2:172.21.15.164:3300/0 conn(0x7fb14806b6c0 0x7fb14815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.532+0000 7fb14db72640 1 -- 172.21.15.59:0/1511230642 >> 172.21.15.59:0/1511230642 conn(0x7fb1480b2660 msgr2=0x7fb1480b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.533+0000 7fb14db72640 1 -- 172.21.15.59:0/1511230642 shutdown_connections 2024-03-06T07:22:20.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.533+0000 7fb14db72640 1 -- 172.21.15.59:0/1511230642 wait complete. 2024-03-06T07:22:20.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.534+0000 7fb14db72640 1 Processor -- start 2024-03-06T07:22:20.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.534+0000 7fb14db72640 1 -- start start 2024-03-06T07:22:20.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.534+0000 7fb14db72640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb14806b6c0 0x7fb148156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.534+0000 7fb14db72640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb14815ec40 0x7fb14815ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.534+0000 7fb14db72640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1480b51e0 0x7fb1480b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.534+0000 7fb14db72640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14815f0f0 con 0x7fb14806b6c0 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb14db72640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb148162180 con 0x7fb1480b51e0 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb14db72640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1480721e0 con 0x7fb14815ec40 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb1477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb14806b6c0 0x7fb148156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb1477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb14806b6c0 0x7fb148156910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42924/0 (socket says 172.21.15.59:42924) 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb1477fe640 1 -- 172.21.15.59:0/3395047971 learned_addr learned my addr 172.21.15.59:0/3395047971 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb147fff640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1480b51e0 0x7fb1480b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb146ffd640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb14815ec40 0x7fb14815ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb1477fe640 1 -- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb14815ec40 msgr2=0x7fb14815ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb1477fe640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb14815ec40 0x7fb14815ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb1477fe640 1 -- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1480b51e0 msgr2=0x7fb1480b1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb1477fe640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1480b51e0 0x7fb1480b1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb1477fe640 1 -- 172.21.15.59:0/3395047971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1480b6320 con 0x7fb14806b6c0 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb1477fe640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb14806b6c0 0x7fb148156910 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb13800c730 tx=0x7fb13800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb12ffff640 1 -- 172.21.15.59:0/3395047971 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138017070 con 0x7fb14806b6c0 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb12ffff640 1 -- 172.21.15.59:0/3395047971 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb138005ad0 con 0x7fb14806b6c0 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb14db72640 1 -- 172.21.15.59:0/3395047971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1480b57a0 con 0x7fb14806b6c0 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb12ffff640 1 -- 172.21.15.59:0/3395047971 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138007e20 con 0x7fb14806b6c0 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.535+0000 7fb14db72640 1 -- 172.21.15.59:0/3395047971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1480b6040 con 0x7fb14806b6c0 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.536+0000 7fb14db72640 1 -- 172.21.15.59:0/3395047971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1480733b0 con 0x7fb14806b6c0 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.537+0000 7fb12ffff640 1 -- 172.21.15.59:0/3395047971 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb13801a070 con 0x7fb14806b6c0 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.537+0000 7fb12ffff640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb13406b4d0 0x7fb13406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.537+0000 7fb12ffff640 1 -- 172.21.15.59:0/3395047971 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb138089470 con 0x7fb14806b6c0 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.537+0000 7fb146ffd640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb13406b4d0 0x7fb13406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.538+0000 7fb146ffd640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb13406b4d0 0x7fb13406d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb13c009560 tx=0x7fb13c009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.541+0000 7fb12ffff640 1 -- 172.21.15.59:0/3395047971 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1480cea90 con 0x7fb14806b6c0 2024-03-06T07:22:20.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.722+0000 7fb14db72640 1 -- 172.21.15.59:0/3395047971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb148068ff0 con 0x7fb14806b6c0 2024-03-06T07:22:20.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.723+0000 7fb12ffff640 1 -- 172.21.15.59:0/3395047971 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb138057d30 con 0x7fb14806b6c0 2024-03-06T07:22:20.725 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:20.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 -- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb13406b4d0 msgr2=0x7fb13406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb13406b4d0 0x7fb13406d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb13c009560 tx=0x7fb13c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:20.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 -- 172.21.15.59:0/3395047971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb14806b6c0 msgr2=0x7fb148156910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb14806b6c0 0x7fb148156910 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb13800c730 tx=0x7fb13800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 -- 172.21.15.59:0/3395047971 shutdown_connections 2024-03-06T07:22:20.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb13406b4d0 0x7fb13406d990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1480b51e0 0x7fb1480b1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb14815ec40 0x7fb14815ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 --2- 172.21.15.59:0/3395047971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb14806b6c0 0x7fb148156910 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 -- 172.21.15.59:0/3395047971 >> 172.21.15.59:0/3395047971 conn(0x7fb1480b2660 msgr2=0x7fb148158c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 -- 172.21.15.59:0/3395047971 shutdown_connections 2024-03-06T07:22:20.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.725+0000 7fb14db72640 1 -- 172.21.15.59:0/3395047971 wait complete. 2024-03-06T07:22:20.737 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:20.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.839+0000 7f7698b56640 1 Processor -- start 2024-03-06T07:22:20.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.840+0000 7f7698b56640 1 -- start start 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.840+0000 7f7698b56640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f769415e0e0 0x7f769406e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.840+0000 7f7698b56640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f769406b6c0 0x7f76940b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.840+0000 7f7698b56640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7694161bd0 con 0x7f769415dd00 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.840+0000 7f7698b56640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7694070220 con 0x7f769415e0e0 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.840+0000 7f7698b56640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7694161e90 con 0x7f769406b6c0 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.840+0000 7f768bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f769415e0e0 0x7f769406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.840+0000 7f769259c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f769415dd00 0x7f769406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41966/0 (socket says 172.21.15.59:41966) 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.840+0000 7f769259c640 1 -- 172.21.15.59:0/1176980169 learned_addr learned my addr 172.21.15.59:0/1176980169 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.841+0000 7f7692d9d640 1 --2- 172.21.15.59:0/1176980169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f769406b6c0 0x7f76940b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.841+0000 7f768bfff640 1 -- 172.21.15.59:0/1176980169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f769406b6c0 msgr2=0x7f76940b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.841+0000 7f768bfff640 1 --2- 172.21.15.59:0/1176980169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f769406b6c0 0x7f76940b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.841+0000 7f768bfff640 1 -- 172.21.15.59:0/1176980169 >> v1:172.21.15.59:6789/0 conn(0x7f769415dd00 legacy=0x7f769406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.841+0000 7f768bfff640 1 -- 172.21.15.59:0/1176980169 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7694162170 con 0x7f769415e0e0 2024-03-06T07:22:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.841+0000 7f768bfff640 1 --2- 172.21.15.59:0/1176980169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f769415e0e0 0x7f769406e620 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f767c0097a0 tx=0x7f767c02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0077f3d587777 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.842+0000 7f7691d9b640 1 -- 172.21.15.59:0/1176980169 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c037070 con 0x7f769415e0e0 2024-03-06T07:22:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.842+0000 7f7691d9b640 1 -- 172.21.15.59:0/1176980169 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c030710 con 0x7f769415e0e0 2024-03-06T07:22:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.842+0000 7f7691d9b640 1 -- 172.21.15.59:0/1176980169 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c030ae0 con 0x7f769415e0e0 2024-03-06T07:22:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.842+0000 7f7698b56640 1 -- 172.21.15.59:0/1176980169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f769415e0e0 msgr2=0x7f769406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.842+0000 7f7698b56640 1 --2- 172.21.15.59:0/1176980169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f769415e0e0 0x7f769406e620 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f767c0097a0 tx=0x7f767c02b710 comp rx=0 tx=0).stop 2024-03-06T07:22:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.842+0000 7f7698b56640 1 -- 172.21.15.59:0/1176980169 shutdown_connections 2024-03-06T07:22:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.842+0000 7f7698b56640 1 --2- 172.21.15.59:0/1176980169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f769406b6c0 0x7f76940b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.842+0000 7f7698b56640 1 --2- 172.21.15.59:0/1176980169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f769415e0e0 0x7f769406e620 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.842+0000 7f7698b56640 1 -- 172.21.15.59:0/1176980169 >> 172.21.15.59:0/1176980169 conn(0x7f76940b2660 msgr2=0x7f76940b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.842+0000 7f7698b56640 1 -- 172.21.15.59:0/1176980169 shutdown_connections 2024-03-06T07:22:20.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.842+0000 7f7698b56640 1 -- 172.21.15.59:0/1176980169 wait complete. 2024-03-06T07:22:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f7698b56640 1 Processor -- start 2024-03-06T07:22:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f7698b56640 1 -- start start 2024-03-06T07:22:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f7698b56640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f769406b6c0 0x7f7694146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f7698b56640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f769415dd00 0x7f7694153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f7698b56640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76941f41f0 0x7f76941f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f7698b56640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76940b0180 con 0x7f769415dd00 2024-03-06T07:22:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f7698b56640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76940b0000 con 0x7f76941f41f0 2024-03-06T07:22:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f7698b56640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76940b0300 con 0x7f769406b6c0 2024-03-06T07:22:20.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f768bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f769415dd00 0x7f7694153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f768bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f769415dd00 0x7f7694153980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42938/0 (socket says 172.21.15.59:42938) 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f768bfff640 1 -- 172.21.15.59:0/554816236 learned_addr learned my addr 172.21.15.59:0/554816236 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.844+0000 7f769259c640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f769406b6c0 0x7f7694146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f7692d9d640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76941f41f0 0x7f76941f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f768bfff640 1 -- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f769406b6c0 msgr2=0x7f7694146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f768bfff640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f769406b6c0 0x7f7694146400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f768bfff640 1 -- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76941f41f0 msgr2=0x7f76941f46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f768bfff640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76941f41f0 0x7f76941f46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f768bfff640 1 -- 172.21.15.59:0/554816236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76941f4be0 con 0x7f769415dd00 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f768bfff640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f769415dd00 0x7f7694153980 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f767c035040 tx=0x7f767c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f768affd640 1 -- 172.21.15.59:0/554816236 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c041030 con 0x7f769415dd00 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f768affd640 1 -- 172.21.15.59:0/554816236 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c03c670 con 0x7f769415dd00 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f7698b56640 1 -- 172.21.15.59:0/554816236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76941562f0 con 0x7f769415dd00 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f768affd640 1 -- 172.21.15.59:0/554816236 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c037040 con 0x7f769415dd00 2024-03-06T07:22:20.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.845+0000 7f7698b56640 1 -- 172.21.15.59:0/554816236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76941568b0 con 0x7f769415dd00 2024-03-06T07:22:20.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.846+0000 7f7698b56640 1 -- 172.21.15.59:0/554816236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7694073390 con 0x7f769415dd00 2024-03-06T07:22:20.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.846+0000 7f768affd640 1 -- 172.21.15.59:0/554816236 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f767c044070 con 0x7f769415dd00 2024-03-06T07:22:20.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.847+0000 7f768affd640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f767406b4d0 0x7f767406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.847+0000 7f768affd640 1 -- 172.21.15.59:0/554816236 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f767c0aba30 con 0x7f769415dd00 2024-03-06T07:22:20.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.847+0000 7f769259c640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f767406b4d0 0x7f767406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.847+0000 7f769259c640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f767406b4d0 0x7f767406d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7684000f60 tx=0x7f7684009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:20.851+0000 7f768affd640 1 -- 172.21.15.59:0/554816236 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76941f4be0 con 0x7f769415dd00 2024-03-06T07:22:21.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.030+0000 7f7698b56640 1 -- 172.21.15.59:0/554816236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7694068ff0 con 0x7f769415dd00 2024-03-06T07:22:21.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.030+0000 7f768affd640 1 -- 172.21.15.59:0/554816236 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f767c07a260 con 0x7f769415dd00 2024-03-06T07:22:21.032 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:21.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 -- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f767406b4d0 msgr2=0x7f767406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f767406b4d0 0x7f767406d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7684000f60 tx=0x7f7684009040 comp rx=0 tx=0).stop 2024-03-06T07:22:21.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 -- 172.21.15.59:0/554816236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f769415dd00 msgr2=0x7f7694153980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f769415dd00 0x7f7694153980 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f767c035040 tx=0x7f767c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 -- 172.21.15.59:0/554816236 shutdown_connections 2024-03-06T07:22:21.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f767406b4d0 0x7f767406d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76941f41f0 0x7f76941f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f769415dd00 0x7f7694153980 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 --2- 172.21.15.59:0/554816236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f769406b6c0 0x7f7694146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 -- 172.21.15.59:0/554816236 >> 172.21.15.59:0/554816236 conn(0x7f76940b2660 msgr2=0x7f769415bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 -- 172.21.15.59:0/554816236 shutdown_connections 2024-03-06T07:22:21.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.032+0000 7f7698b56640 1 -- 172.21.15.59:0/554816236 wait complete. 2024-03-06T07:22:21.055 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-03-06T07:22:21.055 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:22:21.056 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:21.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.155+0000 7f916f04e640 1 Processor -- start 2024-03-06T07:22:21.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.155+0000 7f916f04e640 1 -- start start 2024-03-06T07:22:21.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f916f04e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f916815ddf0 0x7f91680cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f916f04e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f916815e1d0 0x7f91680cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f916f04e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91680b7b60 con 0x7f91680d06d0 2024-03-06T07:22:21.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f916f04e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91680d27c0 con 0x7f916815e1d0 2024-03-06T07:22:21.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f916f04e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91680d18d0 con 0x7f916815ddf0 2024-03-06T07:22:21.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f9167fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f916815e1d0 0x7f91680cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f916d5eb640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f91680d06d0 0x7f9168161ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:54032/0 (socket says 172.21.15.59:54032) 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f916d5eb640 1 -- 172.21.15.59:0/1222683147 learned_addr learned my addr 172.21.15.59:0/1222683147 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f916cdea640 1 --2- 172.21.15.59:0/1222683147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f916815ddf0 0x7f91680cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f9167fff640 1 -- 172.21.15.59:0/1222683147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f916815ddf0 msgr2=0x7f91680cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f9167fff640 1 --2- 172.21.15.59:0/1222683147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f916815ddf0 0x7f91680cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f9167fff640 1 -- 172.21.15.59:0/1222683147 >> v1:172.21.15.164:6789/0 conn(0x7f91680d06d0 legacy=0x7f9168161ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f9167fff640 1 -- 172.21.15.59:0/1222683147 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91680d1bb0 con 0x7f916815e1d0 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f91677fe640 1 -- 172.21.15.59:0/1222683147 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858189986 0 0) 0x7f91680b7b60 con 0x7f91680d06d0 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.156+0000 7f916cdea640 1 --2- 172.21.15.59:0/1222683147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f916815ddf0 0x7f91680cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.157+0000 7f9167fff640 1 --2- 172.21.15.59:0/1222683147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f916815e1d0 0x7f91680cf980 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f91680d2940 tx=0x7f915002b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d75f60140fdfbef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.157+0000 7f91677fe640 1 -- 172.21.15.59:0/1222683147 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9150035020 con 0x7f916815e1d0 2024-03-06T07:22:21.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.157+0000 7f91677fe640 1 -- 172.21.15.59:0/1222683147 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9150031800 con 0x7f916815e1d0 2024-03-06T07:22:21.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.157+0000 7f91677fe640 1 -- 172.21.15.59:0/1222683147 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9150031bd0 con 0x7f916815e1d0 2024-03-06T07:22:21.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.158+0000 7f916f04e640 1 -- 172.21.15.59:0/1222683147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f916815e1d0 msgr2=0x7f91680cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.158+0000 7f916f04e640 1 --2- 172.21.15.59:0/1222683147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f916815e1d0 0x7f91680cf980 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f91680d2940 tx=0x7f915002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.158+0000 7f916f04e640 1 -- 172.21.15.59:0/1222683147 shutdown_connections 2024-03-06T07:22:21.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.158+0000 7f916f04e640 1 --2- 172.21.15.59:0/1222683147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f916815e1d0 0x7f91680cf980 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.158+0000 7f916f04e640 1 --2- 172.21.15.59:0/1222683147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f916815ddf0 0x7f91680cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.158+0000 7f916f04e640 1 -- 172.21.15.59:0/1222683147 >> 172.21.15.59:0/1222683147 conn(0x7f916809cff0 msgr2=0x7f91680d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.158+0000 7f916f04e640 1 -- 172.21.15.59:0/1222683147 shutdown_connections 2024-03-06T07:22:21.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.158+0000 7f916f04e640 1 -- 172.21.15.59:0/1222683147 wait complete. 2024-03-06T07:22:21.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.159+0000 7f916f04e640 1 Processor -- start 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.159+0000 7f916f04e640 1 -- start start 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916f04e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91680d06d0 0x7f91680c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916f04e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f916815ddf0 0x7f916814d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916f04e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f916815e1d0 0x7f916814de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916f04e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91680d2450 con 0x7f91680d06d0 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916f04e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91680d2170 con 0x7f916815e1d0 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916f04e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91680d2c50 con 0x7f916815ddf0 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916cdea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91680d06d0 0x7f91680c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916cdea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91680d06d0 0x7f91680c9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42942/0 (socket says 172.21.15.59:42942) 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916cdea640 1 -- 172.21.15.59:0/4218852337 learned_addr learned my addr 172.21.15.59:0/4218852337 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f9167fff640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f916815ddf0 0x7f916814d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916d5eb640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f916815e1d0 0x7f916814de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916cdea640 1 -- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f916815ddf0 msgr2=0x7f916814d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916cdea640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f916815ddf0 0x7f916814d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916cdea640 1 -- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f916815e1d0 msgr2=0x7f916814de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916cdea640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f916815e1d0 0x7f916814de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.160+0000 7f916cdea640 1 -- 172.21.15.59:0/4218852337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91681602c0 con 0x7f91680d06d0 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.161+0000 7f916cdea640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91680d06d0 0x7f91680c9610 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f916000efc0 tx=0x7f916000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.161+0000 7f91657fa640 1 -- 172.21.15.59:0/4218852337 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160018020 con 0x7f91680d06d0 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.161+0000 7f91657fa640 1 -- 172.21.15.59:0/4218852337 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9160014680 con 0x7f91680d06d0 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.161+0000 7f916f04e640 1 -- 172.21.15.59:0/4218852337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91681605a0 con 0x7f91680d06d0 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.161+0000 7f91657fa640 1 -- 172.21.15.59:0/4218852337 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160014a50 con 0x7f91680d06d0 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.161+0000 7f916f04e640 1 -- 172.21.15.59:0/4218852337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9168148720 con 0x7f91680d06d0 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.162+0000 7f916f04e640 1 -- 172.21.15.59:0/4218852337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916804f330 con 0x7f91680d06d0 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.162+0000 7f91657fa640 1 -- 172.21.15.59:0/4218852337 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9160005d10 con 0x7f91680d06d0 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.163+0000 7f91657fa640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f914806b500 0x7f914806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.163+0000 7f91657fa640 1 -- 172.21.15.59:0/4218852337 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f916008aed0 con 0x7f91680d06d0 2024-03-06T07:22:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.163+0000 7f9167fff640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f914806b500 0x7f914806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.163+0000 7f9167fff640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f914806b500 0x7f914806d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9150036000 tx=0x7f915003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.167+0000 7f91657fa640 1 -- 172.21.15.59:0/4218852337 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9160022030 con 0x7f91680d06d0 2024-03-06T07:22:21.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.348+0000 7f916f04e640 1 -- 172.21.15.59:0/4218852337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91680b6670 con 0x7f91680d06d0 2024-03-06T07:22:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.349+0000 7f91657fa640 1 -- 172.21.15.59:0/4218852337 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f9160010040 con 0x7f91680d06d0 2024-03-06T07:22:21.351 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:21.351 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 -- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f914806b500 msgr2=0x7f914806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f914806b500 0x7f914806d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9150036000 tx=0x7f915003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 -- 172.21.15.59:0/4218852337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91680d06d0 msgr2=0x7f91680c9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91680d06d0 0x7f91680c9610 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f916000efc0 tx=0x7f916000c490 comp rx=0 tx=0).stop 2024-03-06T07:22:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 -- 172.21.15.59:0/4218852337 shutdown_connections 2024-03-06T07:22:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f914806b500 0x7f914806d9c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f916815e1d0 0x7f916814de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f916815ddf0 0x7f916814d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 --2- 172.21.15.59:0/4218852337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91680d06d0 0x7f91680c9610 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 -- 172.21.15.59:0/4218852337 >> 172.21.15.59:0/4218852337 conn(0x7f916809cff0 msgr2=0x7f916815bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 -- 172.21.15.59:0/4218852337 shutdown_connections 2024-03-06T07:22:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.351+0000 7f916f04e640 1 -- 172.21.15.59:0/4218852337 wait complete. 2024-03-06T07:22:21.364 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:21.365 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:21.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.467+0000 7f457b356640 1 Processor -- start 2024-03-06T07:22:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.467+0000 7f457b356640 1 -- start start 2024-03-06T07:22:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f457b356640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f457406b6c0 0x7f457415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f457b356640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f457415ec40 0x7f4574071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f457b356640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45740cea90 con 0x7f457415bbb0 2024-03-06T07:22:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f457b356640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4574161970 con 0x7f457415ec40 2024-03-06T07:22:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f457b356640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45741624f0 con 0x7f457406b6c0 2024-03-06T07:22:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f45790f2640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f457415bbb0 0x7f457406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41972/0 (socket says 172.21.15.59:41972) 2024-03-06T07:22:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f45790f2640 1 -- 172.21.15.59:0/2697113834 learned_addr learned my addr 172.21.15.59:0/2697113834 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f45788f1640 1 --2- 172.21.15.59:0/2697113834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f457406b6c0 0x7f457415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f45798f3640 1 --2- 172.21.15.59:0/2697113834 >> v2:172.21.15.164:3300/0 conn(0x7f457415ec40 0x7f4574071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f456bfff640 1 -- 172.21.15.59:0/2697113834 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326102185 0 0) 0x7f4570001110 con 0x7f457415bbb0 2024-03-06T07:22:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f456bfff640 1 -- 172.21.15.59:0/2697113834 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4560003520 con 0x7f457415bbb0 2024-03-06T07:22:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f45788f1640 1 -- 172.21.15.59:0/2697113834 >> v2:172.21.15.164:3300/0 conn(0x7f457415ec40 msgr2=0x7f4574071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f45788f1640 1 --2- 172.21.15.59:0/2697113834 >> v2:172.21.15.164:3300/0 conn(0x7f457415ec40 0x7f4574071ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f45788f1640 1 -- 172.21.15.59:0/2697113834 >> v1:172.21.15.59:6789/0 conn(0x7f457415bbb0 legacy=0x7f457406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.468+0000 7f45788f1640 1 -- 172.21.15.59:0/2697113834 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4574161bc0 con 0x7f457406b6c0 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.469+0000 7f45788f1640 1 --2- 172.21.15.59:0/2697113834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f457406b6c0 0x7f457415e700 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45640097a0 tx=0x7f456402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23a2f0b8ae48a6af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.469+0000 7f456bfff640 1 -- 172.21.15.59:0/2697113834 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4564035020 con 0x7f457406b6c0 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.469+0000 7f456bfff640 1 -- 172.21.15.59:0/2697113834 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4564031830 con 0x7f457406b6c0 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.469+0000 7f456bfff640 1 -- 172.21.15.59:0/2697113834 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4564031be0 con 0x7f457406b6c0 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.470+0000 7f457b356640 1 -- 172.21.15.59:0/2697113834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f457406b6c0 msgr2=0x7f457415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.470+0000 7f457b356640 1 --2- 172.21.15.59:0/2697113834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f457406b6c0 0x7f457415e700 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45640097a0 tx=0x7f456402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.470+0000 7f457b356640 1 -- 172.21.15.59:0/2697113834 shutdown_connections 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.470+0000 7f457b356640 1 --2- 172.21.15.59:0/2697113834 >> v2:172.21.15.164:3300/0 conn(0x7f457415ec40 0x7f4574071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.470+0000 7f457b356640 1 --2- 172.21.15.59:0/2697113834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f457406b6c0 0x7f457415e700 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.470+0000 7f457b356640 1 -- 172.21.15.59:0/2697113834 >> 172.21.15.59:0/2697113834 conn(0x7f45740b2660 msgr2=0x7f45740b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.470+0000 7f457b356640 1 -- 172.21.15.59:0/2697113834 shutdown_connections 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.470+0000 7f457b356640 1 -- 172.21.15.59:0/2697113834 wait complete. 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.471+0000 7f457b356640 1 Processor -- start 2024-03-06T07:22:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.471+0000 7f457b356640 1 -- start start 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f457b356640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f457406b6c0 0x7f45740cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f457b356640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f457415ec40 0x7f4574150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f457b356640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45740b1cf0 0x7f45741594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f457b356640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457415f0f0 con 0x7f457406b6c0 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f457b356640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4574162180 con 0x7f45740b1cf0 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f457b356640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45740721e0 con 0x7f457415ec40 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f45790f2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f457406b6c0 0x7f45740cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f45790f2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f457406b6c0 0x7f45740cafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42948/0 (socket says 172.21.15.59:42948) 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f45790f2640 1 -- 172.21.15.59:0/2426056393 learned_addr learned my addr 172.21.15.59:0/2426056393 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f45788f1640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f457415ec40 0x7f4574150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f45798f3640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45740b1cf0 0x7f45741594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f45790f2640 1 -- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f457415ec40 msgr2=0x7f4574150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f45790f2640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f457415ec40 0x7f4574150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f45790f2640 1 -- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45740b1cf0 msgr2=0x7f45741594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f45790f2640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45740b1cf0 0x7f45741594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f45790f2640 1 -- 172.21.15.59:0/2426056393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45740cb500 con 0x7f457406b6c0 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f45790f2640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f457406b6c0 0x7f45740cafc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f457000c730 tx=0x7f457000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f4569ffb640 1 -- 172.21.15.59:0/2426056393 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4570017070 con 0x7f457406b6c0 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f4569ffb640 1 -- 172.21.15.59:0/2426056393 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4570005ad0 con 0x7f457406b6c0 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f457b356640 1 -- 172.21.15.59:0/2426056393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4574150b60 con 0x7f457406b6c0 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.472+0000 7f4569ffb640 1 -- 172.21.15.59:0/2426056393 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4570007e20 con 0x7f457406b6c0 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.473+0000 7f457b356640 1 -- 172.21.15.59:0/2426056393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457415c260 con 0x7f457406b6c0 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.474+0000 7f457b356640 1 -- 172.21.15.59:0/2426056393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45740733b0 con 0x7f457406b6c0 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.474+0000 7f4569ffb640 1 -- 172.21.15.59:0/2426056393 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f457001a070 con 0x7f457406b6c0 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.474+0000 7f4569ffb640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f455806b4d0 0x7f455806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.474+0000 7f4569ffb640 1 -- 172.21.15.59:0/2426056393 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4570089500 con 0x7f457406b6c0 2024-03-06T07:22:21.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.474+0000 7f45788f1640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f455806b4d0 0x7f455806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.475+0000 7f45788f1640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f455806b4d0 0x7f455806d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4564036000 tx=0x7f456403d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.478+0000 7f4569ffb640 1 -- 172.21.15.59:0/2426056393 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4574150b60 con 0x7f457406b6c0 2024-03-06T07:22:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.658+0000 7f457b356640 1 -- 172.21.15.59:0/2426056393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4574068ff0 con 0x7f457406b6c0 2024-03-06T07:22:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.659+0000 7f4569ffb640 1 -- 172.21.15.59:0/2426056393 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f45740cea90 con 0x7f457406b6c0 2024-03-06T07:22:21.661 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:21.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 -- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f455806b4d0 msgr2=0x7f455806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f455806b4d0 0x7f455806d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4564036000 tx=0x7f456403d040 comp rx=0 tx=0).stop 2024-03-06T07:22:21.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 -- 172.21.15.59:0/2426056393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f457406b6c0 msgr2=0x7f45740cafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f457406b6c0 0x7f45740cafc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f457000c730 tx=0x7f457000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 -- 172.21.15.59:0/2426056393 shutdown_connections 2024-03-06T07:22:21.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f455806b4d0 0x7f455806d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45740b1cf0 0x7f45741594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f457415ec40 0x7f4574150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 --2- 172.21.15.59:0/2426056393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f457406b6c0 0x7f45740cafc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 -- 172.21.15.59:0/2426056393 >> 172.21.15.59:0/2426056393 conn(0x7f45740b2660 msgr2=0x7f4574158830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 -- 172.21.15.59:0/2426056393 shutdown_connections 2024-03-06T07:22:21.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.661+0000 7f457b356640 1 -- 172.21.15.59:0/2426056393 wait complete. 2024-03-06T07:22:21.673 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:21.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.776+0000 7f37bd8fb640 1 Processor -- start 2024-03-06T07:22:21.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.776+0000 7f37bd8fb640 1 -- start start 2024-03-06T07:22:21.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37bd8fb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f37b815bbb0 0x7f37b806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37bd8fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37b815ee10 0x7f37b8071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37bd8fb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b80cea90 con 0x7f37b806b4f0 2024-03-06T07:22:21.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37bd8fb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b8161970 con 0x7f37b815bbb0 2024-03-06T07:22:21.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37bd8fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b81624f0 con 0x7f37b815ee10 2024-03-06T07:22:21.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37b6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f37b815bbb0 0x7f37b806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37ae7fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f37b806b4f0 0x7f37b815e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41980/0 (socket says 172.21.15.59:41980) 2024-03-06T07:22:21.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37ae7fc640 1 -- 172.21.15.59:0/3917602595 learned_addr learned my addr 172.21.15.59:0/3917602595 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:21.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37b77fe640 1 --2- 172.21.15.59:0/3917602595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37b815ee10 0x7f37b8071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37b6ffd640 1 -- 172.21.15.59:0/3917602595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37b815ee10 msgr2=0x7f37b8071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37b6ffd640 1 --2- 172.21.15.59:0/3917602595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37b815ee10 0x7f37b8071ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37b6ffd640 1 -- 172.21.15.59:0/3917602595 >> v1:172.21.15.59:6789/0 conn(0x7f37b806b4f0 legacy=0x7f37b815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37b6ffd640 1 -- 172.21.15.59:0/3917602595 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b8161bc0 con 0x7f37b815bbb0 2024-03-06T07:22:21.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.777+0000 7f37b67fc640 1 -- 172.21.15.59:0/3917602595 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254145169 0 0) 0x7f37a0001110 con 0x7f37b806b4f0 2024-03-06T07:22:21.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.778+0000 7f37b6ffd640 1 --2- 172.21.15.59:0/3917602595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37b815bbb0 0x7f37b806afb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f37a80097a0 tx=0x7f37a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e4f3218ac9bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.778+0000 7f37b67fc640 1 -- 172.21.15.59:0/3917602595 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a8035020 con 0x7f37b815bbb0 2024-03-06T07:22:21.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.778+0000 7f37b67fc640 1 -- 172.21.15.59:0/3917602595 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a8031830 con 0x7f37b815bbb0 2024-03-06T07:22:21.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.778+0000 7f37b67fc640 1 -- 172.21.15.59:0/3917602595 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a8031c00 con 0x7f37b815bbb0 2024-03-06T07:22:21.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.778+0000 7f37bd8fb640 1 -- 172.21.15.59:0/3917602595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37b815bbb0 msgr2=0x7f37b806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.778+0000 7f37bd8fb640 1 --2- 172.21.15.59:0/3917602595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37b815bbb0 0x7f37b806afb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f37a80097a0 tx=0x7f37a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.778+0000 7f37bd8fb640 1 -- 172.21.15.59:0/3917602595 shutdown_connections 2024-03-06T07:22:21.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.778+0000 7f37bd8fb640 1 --2- 172.21.15.59:0/3917602595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37b815ee10 0x7f37b8071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.778+0000 7f37bd8fb640 1 --2- 172.21.15.59:0/3917602595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37b815bbb0 0x7f37b806afb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.778+0000 7f37bd8fb640 1 -- 172.21.15.59:0/3917602595 >> 172.21.15.59:0/3917602595 conn(0x7f37b80b2660 msgr2=0x7f37b80b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.779+0000 7f37bd8fb640 1 -- 172.21.15.59:0/3917602595 shutdown_connections 2024-03-06T07:22:21.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.779+0000 7f37bd8fb640 1 -- 172.21.15.59:0/3917602595 wait complete. 2024-03-06T07:22:21.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.780+0000 7f37bd8fb640 1 Processor -- start 2024-03-06T07:22:21.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.780+0000 7f37bd8fb640 1 -- start start 2024-03-06T07:22:21.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.780+0000 7f37bd8fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37b806b4f0 0x7f37b81f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.780+0000 7f37bd8fb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37b815ee10 0x7f37b80cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.780+0000 7f37bd8fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37b8154ac0 0x7f37b806dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.780+0000 7f37bd8fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b80721e0 con 0x7f37b8154ac0 2024-03-06T07:22:21.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.780+0000 7f37bd8fb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b8162180 con 0x7f37b815ee10 2024-03-06T07:22:21.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.780+0000 7f37bd8fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b8072360 con 0x7f37b806b4f0 2024-03-06T07:22:21.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37b8154ac0 0x7f37b806dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37b8154ac0 0x7f37b806dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42968/0 (socket says 172.21.15.59:42968) 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b77fe640 1 -- 172.21.15.59:0/1584490776 learned_addr learned my addr 172.21.15.59:0/1584490776 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b6ffd640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37b806b4f0 0x7f37b81f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37ae7fc640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37b815ee10 0x7f37b80cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b77fe640 1 -- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37b806b4f0 msgr2=0x7f37b81f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b77fe640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37b806b4f0 0x7f37b81f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b77fe640 1 -- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37b815ee10 msgr2=0x7f37b80cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b77fe640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37b815ee10 0x7f37b80cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b77fe640 1 -- 172.21.15.59:0/1584490776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b815f6f0 con 0x7f37b8154ac0 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b77fe640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37b8154ac0 0x7f37b806dfb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f37b000efc0 tx=0x7f37b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b4ff9640 1 -- 172.21.15.59:0/1584490776 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b0018020 con 0x7f37b8154ac0 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b4ff9640 1 -- 172.21.15.59:0/1584490776 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b0014480 con 0x7f37b8154ac0 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37bd8fb640 1 -- 172.21.15.59:0/1584490776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b81f7c80 con 0x7f37b8154ac0 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37b4ff9640 1 -- 172.21.15.59:0/1584490776 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b0014870 con 0x7f37b8154ac0 2024-03-06T07:22:21.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.781+0000 7f37bd8fb640 1 -- 172.21.15.59:0/1584490776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b81f8240 con 0x7f37b8154ac0 2024-03-06T07:22:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.782+0000 7f37bd8fb640 1 -- 172.21.15.59:0/1584490776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b80733b0 con 0x7f37b8154ac0 2024-03-06T07:22:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.783+0000 7f37b4ff9640 1 -- 172.21.15.59:0/1584490776 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37b0005d10 con 0x7f37b8154ac0 2024-03-06T07:22:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.783+0000 7f37b4ff9640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f379806b4d0 0x7f379806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.783+0000 7f37b4ff9640 1 -- 172.21.15.59:0/1584490776 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37b008ab70 con 0x7f37b8154ac0 2024-03-06T07:22:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.783+0000 7f37b6ffd640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f379806b4d0 0x7f379806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.784+0000 7f37b6ffd640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f379806b4d0 0x7f379806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f37a8036000 tx=0x7f37a8008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.787+0000 7f37b4ff9640 1 -- 172.21.15.59:0/1584490776 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37b0022070 con 0x7f37b8154ac0 2024-03-06T07:22:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.966+0000 7f37bd8fb640 1 -- 172.21.15.59:0/1584490776 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f37b8068ff0 con 0x7f37b8154ac0 2024-03-06T07:22:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.967+0000 7f37b4ff9640 1 -- 172.21.15.59:0/1584490776 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f37b0010040 con 0x7f37b8154ac0 2024-03-06T07:22:21.969 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 -- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f379806b4d0 msgr2=0x7f379806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f379806b4d0 0x7f379806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f37a8036000 tx=0x7f37a8008030 comp rx=0 tx=0).stop 2024-03-06T07:22:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 -- 172.21.15.59:0/1584490776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37b8154ac0 msgr2=0x7f37b806dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37b8154ac0 0x7f37b806dfb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f37b000efc0 tx=0x7f37b000c490 comp rx=0 tx=0).stop 2024-03-06T07:22:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 -- 172.21.15.59:0/1584490776 shutdown_connections 2024-03-06T07:22:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f379806b4d0 0x7f379806d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f37b8154ac0 0x7f37b806dfb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37b815ee10 0x7f37b80cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 --2- 172.21.15.59:0/1584490776 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37b806b4f0 0x7f37b81f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 -- 172.21.15.59:0/1584490776 >> 172.21.15.59:0/1584490776 conn(0x7f37b80b2660 msgr2=0x7f37b81505f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 -- 172.21.15.59:0/1584490776 shutdown_connections 2024-03-06T07:22:21.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:21.969+0000 7f37bd8fb640 1 -- 172.21.15.59:0/1584490776 wait complete. 2024-03-06T07:22:21.981 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:22.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.084+0000 7f4d3c0fb640 1 Processor -- start 2024-03-06T07:22:22.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.084+0000 7f4d3c0fb640 1 -- start start 2024-03-06T07:22:22.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d3c0fb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4d3406b6c0 0x7f4d3415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d3c0fb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d340cea90 con 0x7f4d3415ec40 2024-03-06T07:22:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d3c0fb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d34161bc0 con 0x7f4d3415bbb0 2024-03-06T07:22:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d3c0fb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d34161970 con 0x7f4d3406b6c0 2024-03-06T07:22:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d3a698640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4d3415ec40 0x7f4d34071ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41992/0 (socket says 172.21.15.59:41992) 2024-03-06T07:22:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d3a698640 1 -- 172.21.15.59:0/3878560628 learned_addr learned my addr 172.21.15.59:0/3878560628 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:22.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d39696640 1 --2- 172.21.15.59:0/3878560628 >> v2:172.21.15.164:3300/0 conn(0x7f4d3406b6c0 0x7f4d3415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d38e95640 1 -- 172.21.15.59:0/3878560628 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484368351 0 0) 0x7f4d2c001490 con 0x7f4d3415ec40 2024-03-06T07:22:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d38e95640 1 -- 172.21.15.59:0/3878560628 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d20003520 con 0x7f4d3415ec40 2024-03-06T07:22:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d39696640 1 -- 172.21.15.59:0/3878560628 >> v1:172.21.15.164:6789/0 conn(0x7f4d3415bbb0 legacy=0x7f4d3406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d38e95640 1 -- 172.21.15.59:0/3878560628 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210283325 0 0) 0x7f4d34161bc0 con 0x7f4d3415bbb0 2024-03-06T07:22:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d39696640 1 -- 172.21.15.59:0/3878560628 >> v1:172.21.15.59:6789/0 conn(0x7f4d3415ec40 legacy=0x7f4d34071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.085+0000 7f4d39696640 1 -- 172.21.15.59:0/3878560628 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d34161ea0 con 0x7f4d3406b6c0 2024-03-06T07:22:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.086+0000 7f4d39696640 1 --2- 172.21.15.59:0/3878560628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3406b6c0 0x7f4d3415e700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d240097a0 tx=0x7f4d2402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=673d052591475f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.086+0000 7f4d38e95640 1 -- 172.21.15.59:0/3878560628 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24034070 con 0x7f4d3406b6c0 2024-03-06T07:22:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.086+0000 7f4d38e95640 1 -- 172.21.15.59:0/3878560628 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d24030610 con 0x7f4d3406b6c0 2024-03-06T07:22:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.086+0000 7f4d38e95640 1 -- 172.21.15.59:0/3878560628 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d240309c0 con 0x7f4d3406b6c0 2024-03-06T07:22:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.087+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/3878560628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3406b6c0 msgr2=0x7f4d3415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.087+0000 7f4d3c0fb640 1 --2- 172.21.15.59:0/3878560628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3406b6c0 0x7f4d3415e700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d240097a0 tx=0x7f4d2402b960 comp rx=0 tx=0).stop 2024-03-06T07:22:22.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.087+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/3878560628 shutdown_connections 2024-03-06T07:22:22.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.087+0000 7f4d3c0fb640 1 --2- 172.21.15.59:0/3878560628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3406b6c0 0x7f4d3415e700 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.087+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/3878560628 >> 172.21.15.59:0/3878560628 conn(0x7f4d340b2660 msgr2=0x7f4d340b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.087+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/3878560628 shutdown_connections 2024-03-06T07:22:22.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.087+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/3878560628 wait complete. 2024-03-06T07:22:22.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.088+0000 7f4d3c0fb640 1 Processor -- start 2024-03-06T07:22:22.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.088+0000 7f4d3c0fb640 1 -- start start 2024-03-06T07:22:22.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d3c0fb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3406b6c0 0x7f4d341462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d3c0fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3415ec40 0x7f4d34154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d3c0fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d341550a0 0x7f4d34159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d3c0fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3415f0d0 con 0x7f4d3415ec40 2024-03-06T07:22:22.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d3c0fb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d341624f0 con 0x7f4d3406b6c0 2024-03-06T07:22:22.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d3c0fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d340723b0 con 0x7f4d341550a0 2024-03-06T07:22:22.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d39696640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3415ec40 0x7f4d34154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d39696640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3415ec40 0x7f4d34154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42974/0 (socket says 172.21.15.59:42974) 2024-03-06T07:22:22.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d39696640 1 -- 172.21.15.59:0/1420027576 learned_addr learned my addr 172.21.15.59:0/1420027576 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:22.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d39e97640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3406b6c0 0x7f4d341462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d3a698640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d341550a0 0x7f4d34159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d39696640 1 -- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d341550a0 msgr2=0x7f4d34159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d39696640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d341550a0 0x7f4d34159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d39696640 1 -- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3406b6c0 msgr2=0x7f4d341462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d39696640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3406b6c0 0x7f4d341462c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d39696640 1 -- 172.21.15.59:0/1420027576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3415aa80 con 0x7f4d3415ec40 2024-03-06T07:22:22.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d39696640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3415ec40 0x7f4d34154b60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4d24000c00 tx=0x7f4d240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d2a7fc640 1 -- 172.21.15.59:0/1420027576 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24048070 con 0x7f4d3415ec40 2024-03-06T07:22:22.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d2a7fc640 1 -- 172.21.15.59:0/1420027576 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d24030410 con 0x7f4d3415ec40 2024-03-06T07:22:22.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/1420027576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d3415ad60 con 0x7f4d3415ec40 2024-03-06T07:22:22.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d2a7fc640 1 -- 172.21.15.59:0/1420027576 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24034050 con 0x7f4d3415ec40 2024-03-06T07:22:22.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.089+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/1420027576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d3415b320 con 0x7f4d3415ec40 2024-03-06T07:22:22.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.090+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/1420027576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d340733b0 con 0x7f4d3415ec40 2024-03-06T07:22:22.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.091+0000 7f4d2a7fc640 1 -- 172.21.15.59:0/1420027576 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d2404b070 con 0x7f4d3415ec40 2024-03-06T07:22:22.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.091+0000 7f4d2a7fc640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1806b4d0 0x7f4d1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.092+0000 7f4d2a7fc640 1 -- 172.21.15.59:0/1420027576 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d240abd30 con 0x7f4d3415ec40 2024-03-06T07:22:22.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.092+0000 7f4d39e97640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1806b4d0 0x7f4d1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.092+0000 7f4d39e97640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1806b4d0 0x7f4d1806d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4d30000f60 tx=0x7f4d30009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.096+0000 7f4d2a7fc640 1 -- 172.21.15.59:0/1420027576 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d2407a5f0 con 0x7f4d3415ec40 2024-03-06T07:22:22.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.276+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/1420027576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4d34068ff0 con 0x7f4d3415ec40 2024-03-06T07:22:22.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.277+0000 7f4d2a7fc640 1 -- 172.21.15.59:0/1420027576 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4d2407e690 con 0x7f4d3415ec40 2024-03-06T07:22:22.279 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1806b4d0 msgr2=0x7f4d1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1806b4d0 0x7f4d1806d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4d30000f60 tx=0x7f4d30009040 comp rx=0 tx=0).stop 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/1420027576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3415ec40 msgr2=0x7f4d34154b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3415ec40 0x7f4d34154b60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4d24000c00 tx=0x7f4d240046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/1420027576 shutdown_connections 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1806b4d0 0x7f4d1806d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d341550a0 0x7f4d34159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3415ec40 0x7f4d34154b60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 --2- 172.21.15.59:0/1420027576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3406b6c0 0x7f4d341462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/1420027576 >> 172.21.15.59:0/1420027576 conn(0x7f4d340b2660 msgr2=0x7f4d340cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/1420027576 shutdown_connections 2024-03-06T07:22:22.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.279+0000 7f4d3c0fb640 1 -- 172.21.15.59:0/1420027576 wait complete. 2024-03-06T07:22:22.291 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:22:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.391+0000 7f20c9f5c640 1 Processor -- start 2024-03-06T07:22:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.391+0000 7f20c9f5c640 1 -- start start 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.392+0000 7f20c9f5c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f20c406bf10 0x7f20c4066d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.392+0000 7f20c9f5c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c415f560 con 0x7f20c406d2a0 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.392+0000 7f20c9f5c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c415ea00 con 0x7f20c415e570 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.392+0000 7f20c9f5c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c415ecc0 con 0x7f20c406bf10 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.392+0000 7f20c37fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f20c406bf10 0x7f20c4066d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.392+0000 7f20c3fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f20c406d2a0 0x7f20c40d0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42006/0 (socket says 172.21.15.59:42006) 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.392+0000 7f20c3fff640 1 -- 172.21.15.59:0/1942622454 learned_addr learned my addr 172.21.15.59:0/1942622454 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.392+0000 7f20c37fe640 1 -- 172.21.15.59:0/1942622454 >> v1:172.21.15.164:6789/0 conn(0x7f20c415e570 legacy=0x7f20c406c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.392+0000 7f20c37fe640 1 -- 172.21.15.59:0/1942622454 >> v1:172.21.15.59:6789/0 conn(0x7f20c406d2a0 legacy=0x7f20c40d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.392+0000 7f20c2ffd640 1 -- 172.21.15.59:0/1942622454 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745260726 0 0) 0x7f20c415f560 con 0x7f20c406d2a0 2024-03-06T07:22:22.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.392+0000 7f20c37fe640 1 -- 172.21.15.59:0/1942622454 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c415efa0 con 0x7f20c406bf10 2024-03-06T07:22:22.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.393+0000 7f20c37fe640 1 --2- 172.21.15.59:0/1942622454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c406bf10 0x7f20c4066d70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f20b40097a0 tx=0x7f20b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13879d1d0a999711 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.393+0000 7f20c2ffd640 1 -- 172.21.15.59:0/1942622454 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b40315a0 con 0x7f20c406bf10 2024-03-06T07:22:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.393+0000 7f20c2ffd640 1 -- 172.21.15.59:0/1942622454 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b4037070 con 0x7f20c406bf10 2024-03-06T07:22:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.393+0000 7f20c2ffd640 1 -- 172.21.15.59:0/1942622454 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4031960 con 0x7f20c406bf10 2024-03-06T07:22:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.393+0000 7f20c9f5c640 1 -- 172.21.15.59:0/1942622454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c406bf10 msgr2=0x7f20c4066d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.393+0000 7f20c9f5c640 1 --2- 172.21.15.59:0/1942622454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c406bf10 0x7f20c4066d70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f20b40097a0 tx=0x7f20b402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.394+0000 7f20c9f5c640 1 -- 172.21.15.59:0/1942622454 shutdown_connections 2024-03-06T07:22:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.394+0000 7f20c9f5c640 1 --2- 172.21.15.59:0/1942622454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c406bf10 0x7f20c4066d70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.394+0000 7f20c9f5c640 1 -- 172.21.15.59:0/1942622454 >> 172.21.15.59:0/1942622454 conn(0x7f20c40725f0 msgr2=0x7f20c4072a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.394+0000 7f20c9f5c640 1 -- 172.21.15.59:0/1942622454 shutdown_connections 2024-03-06T07:22:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.394+0000 7f20c9f5c640 1 -- 172.21.15.59:0/1942622454 wait complete. 2024-03-06T07:22:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.395+0000 7f20c9f5c640 1 Processor -- start 2024-03-06T07:22:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.395+0000 7f20c9f5c640 1 -- start start 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20c9f5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c406bf10 0x7f20c40ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20c9f5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c406d2a0 0x7f20c4157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20c9f5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c415e570 0x7f20c41584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20c9f5c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c415be70 con 0x7f20c406d2a0 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20c9f5c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40d00e0 con 0x7f20c415e570 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20c9f5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c415bff0 con 0x7f20c406bf10 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20baffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c406d2a0 0x7f20c4157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20baffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c406d2a0 0x7f20c4157fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43004/0 (socket says 172.21.15.59:43004) 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20baffd640 1 -- 172.21.15.59:0/4131535818 learned_addr learned my addr 172.21.15.59:0/4131535818 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20c37fe640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c406bf10 0x7f20c40ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20c3fff640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c415e570 0x7f20c41584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20baffd640 1 -- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c406bf10 msgr2=0x7f20c40ae420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20baffd640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c406bf10 0x7f20c40ae420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20baffd640 1 -- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c415e570 msgr2=0x7f20c41584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20baffd640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c415e570 0x7f20c41584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20baffd640 1 -- 172.21.15.59:0/4131535818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c409cf70 con 0x7f20c406d2a0 2024-03-06T07:22:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20baffd640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c406d2a0 0x7f20c4157fb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f20ac00c400 tx=0x7f20ac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20c0ff9640 1 -- 172.21.15.59:0/4131535818 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac017070 con 0x7f20c406d2a0 2024-03-06T07:22:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20c0ff9640 1 -- 172.21.15.59:0/4131535818 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ac0052d0 con 0x7f20c406d2a0 2024-03-06T07:22:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.396+0000 7f20c9f5c640 1 -- 172.21.15.59:0/4131535818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c409d2b0 con 0x7f20c406d2a0 2024-03-06T07:22:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.397+0000 7f20c0ff9640 1 -- 172.21.15.59:0/4131535818 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ac003830 con 0x7f20c406d2a0 2024-03-06T07:22:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.397+0000 7f20c9f5c640 1 -- 172.21.15.59:0/4131535818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c409daf0 con 0x7f20c406d2a0 2024-03-06T07:22:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.397+0000 7f20c9f5c640 1 -- 172.21.15.59:0/4131535818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c404f330 con 0x7f20c406d2a0 2024-03-06T07:22:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.398+0000 7f20c0ff9640 1 -- 172.21.15.59:0/4131535818 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20ac01a070 con 0x7f20c406d2a0 2024-03-06T07:22:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.398+0000 7f20c0ff9640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20a406b4d0 0x7f20a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.398+0000 7f20c0ff9640 1 -- 172.21.15.59:0/4131535818 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20ac089460 con 0x7f20c406d2a0 2024-03-06T07:22:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.398+0000 7f20c37fe640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20a406b4d0 0x7f20a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.399+0000 7f20c37fe640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20a406b4d0 0x7f20a406d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f20b4036000 tx=0x7f20b403d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.402+0000 7f20c0ff9640 1 -- 172.21.15.59:0/4131535818 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20c409d2b0 con 0x7f20c406d2a0 2024-03-06T07:22:22.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:22.656+0000 7f20c9f5c640 1 -- 172.21.15.59:0/4131535818 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f20c4160d30 con 0x7f20c406d2a0 2024-03-06T07:22:23.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.501+0000 7f20c0ff9640 1 -- 172.21.15.59:0/4131535818 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v51) v1 ==== 135+0+0 (secure 0 0 0) 0x7f20ac057d20 con 0x7f20c406d2a0 2024-03-06T07:22:23.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 -- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20a406b4d0 msgr2=0x7f20a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20a406b4d0 0x7f20a406d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f20b4036000 tx=0x7f20b403d040 comp rx=0 tx=0).stop 2024-03-06T07:22:23.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 -- 172.21.15.59:0/4131535818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c406d2a0 msgr2=0x7f20c4157fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c406d2a0 0x7f20c4157fb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f20ac00c400 tx=0x7f20ac00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 -- 172.21.15.59:0/4131535818 shutdown_connections 2024-03-06T07:22:23.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20a406b4d0 0x7f20a406d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c415e570 0x7f20c41584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20c406d2a0 0x7f20c4157fb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 --2- 172.21.15.59:0/4131535818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c406bf10 0x7f20c40ae420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 -- 172.21.15.59:0/4131535818 >> 172.21.15.59:0/4131535818 conn(0x7f20c40725f0 msgr2=0x7f20c40b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 -- 172.21.15.59:0/4131535818 shutdown_connections 2024-03-06T07:22:23.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.503+0000 7f20c9f5c640 1 -- 172.21.15.59:0/4131535818 wait complete. 2024-03-06T07:22:23.519 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:22:23.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.628+0000 7f8510993640 1 Processor -- start 2024-03-06T07:22:23.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.628+0000 7f8510993640 1 -- start start 2024-03-06T07:22:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f8510993640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f850c0b1e40 0x7f850c168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f8510993640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c0d2730 con 0x7f850c0d4860 2024-03-06T07:22:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f8510993640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c1618b0 con 0x7f850c1614d0 2024-03-06T07:22:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f8510993640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0d4cf0 con 0x7f850c0b1e40 2024-03-06T07:22:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f850ad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f850c0b1e40 0x7f850c168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f8509d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f850c0d4860 0x7f850c0d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46944/0 (socket says 172.21.15.59:46944) 2024-03-06T07:22:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f8509d9b640 1 -- 172.21.15.59:0/620053620 learned_addr learned my addr 172.21.15.59:0/620053620 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:23.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f850ad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f850c0b1e40 0x7f850c168b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52650/0 (socket says 172.21.15.59:52650) 2024-03-06T07:22:23.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f850ad9d640 1 -- 172.21.15.59:0/620053620 >> v1:172.21.15.164:6789/0 conn(0x7f850c1614d0 legacy=0x7f850c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f850ad9d640 1 -- 172.21.15.59:0/620053620 >> v1:172.21.15.59:6789/0 conn(0x7f850c0d4860 legacy=0x7f850c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f850959a640 1 -- 172.21.15.59:0/620053620 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190018781 0 0) 0x7f850c0d2730 con 0x7f850c0d4860 2024-03-06T07:22:23.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.629+0000 7f850ad9d640 1 -- 172.21.15.59:0/620053620 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0cf400 con 0x7f850c0b1e40 2024-03-06T07:22:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.630+0000 7f850ad9d640 1 --2- 172.21.15.59:0/620053620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f850c0b1e40 0x7f850c168b60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f850c0cf5d0 tx=0x7f84fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8e5c747879ccfa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.630+0000 7f850959a640 1 -- 172.21.15.59:0/620053620 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc035020 con 0x7f850c0b1e40 2024-03-06T07:22:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.630+0000 7f850959a640 1 -- 172.21.15.59:0/620053620 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc031830 con 0x7f850c0b1e40 2024-03-06T07:22:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.630+0000 7f850959a640 1 -- 172.21.15.59:0/620053620 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc031c00 con 0x7f850c0b1e40 2024-03-06T07:22:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.631+0000 7f8510993640 1 -- 172.21.15.59:0/620053620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f850c0b1e40 msgr2=0x7f850c168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.631+0000 7f8510993640 1 --2- 172.21.15.59:0/620053620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f850c0b1e40 0x7f850c168b60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f850c0cf5d0 tx=0x7f84fc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.631+0000 7f8510993640 1 -- 172.21.15.59:0/620053620 shutdown_connections 2024-03-06T07:22:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.631+0000 7f8510993640 1 --2- 172.21.15.59:0/620053620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f850c0b1e40 0x7f850c168b60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.631+0000 7f8510993640 1 -- 172.21.15.59:0/620053620 >> 172.21.15.59:0/620053620 conn(0x7f850c0d3d40 msgr2=0x7f850c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.631+0000 7f8510993640 1 -- 172.21.15.59:0/620053620 shutdown_connections 2024-03-06T07:22:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.631+0000 7f8510993640 1 -- 172.21.15.59:0/620053620 wait complete. 2024-03-06T07:22:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.632+0000 7f8510993640 1 Processor -- start 2024-03-06T07:22:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.632+0000 7f8510993640 1 -- start start 2024-03-06T07:22:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f8510993640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f850c0b1e40 0x7f850c0b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f8510993640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f850c0d4860 0x7f850c0b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f8510993640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f850c1614d0 0x7f850c1f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f8510993640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0cfc60 con 0x7f850c1614d0 2024-03-06T07:22:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f8510993640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0cf980 con 0x7f850c0b1e40 2024-03-06T07:22:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f8510993640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0b0910 con 0x7f850c0d4860 2024-03-06T07:22:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f850ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f850c1614d0 0x7f850c1f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f850ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f850c1614d0 0x7f850c1f73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52664/0 (socket says 172.21.15.59:52664) 2024-03-06T07:22:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f850ad9d640 1 -- 172.21.15.59:0/1112071840 learned_addr learned my addr 172.21.15.59:0/1112071840 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f8509d9b640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f850c0d4860 0x7f850c0b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f850a59c640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f850c0b1e40 0x7f850c0b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f850ad9d640 1 -- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f850c0d4860 msgr2=0x7f850c0b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f850ad9d640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f850c0d4860 0x7f850c0b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f850ad9d640 1 -- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f850c0b1e40 msgr2=0x7f850c0b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f850ad9d640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f850c0b1e40 0x7f850c0b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.633+0000 7f850ad9d640 1 -- 172.21.15.59:0/1112071840 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c14a8b0 con 0x7f850c1614d0 2024-03-06T07:22:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.634+0000 7f850ad9d640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f850c1614d0 0x7f850c1f73b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f84fc034040 tx=0x7f84fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.634+0000 7f84faffd640 1 -- 172.21.15.59:0/1112071840 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc041070 con 0x7f850c1614d0 2024-03-06T07:22:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.634+0000 7f84faffd640 1 -- 172.21.15.59:0/1112071840 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc004250 con 0x7f850c1614d0 2024-03-06T07:22:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.634+0000 7f8510993640 1 -- 172.21.15.59:0/1112071840 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c14ab90 con 0x7f850c1614d0 2024-03-06T07:22:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.634+0000 7f84faffd640 1 -- 172.21.15.59:0/1112071840 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc035040 con 0x7f850c1614d0 2024-03-06T07:22:23.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.634+0000 7f8510993640 1 -- 172.21.15.59:0/1112071840 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c0b3750 con 0x7f850c1614d0 2024-03-06T07:22:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.635+0000 7f8510993640 1 -- 172.21.15.59:0/1112071840 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c0b3540 con 0x7f850c1614d0 2024-03-06T07:22:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.635+0000 7f84faffd640 1 -- 172.21.15.59:0/1112071840 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84fc038020 con 0x7f850c1614d0 2024-03-06T07:22:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.636+0000 7f84faffd640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84ec06b4d0 0x7f84ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.636+0000 7f84faffd640 1 -- 172.21.15.59:0/1112071840 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84fc0abb40 con 0x7f850c1614d0 2024-03-06T07:22:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.636+0000 7f850a59c640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84ec06b4d0 0x7f84ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.636+0000 7f850a59c640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84ec06b4d0 0x7f84ec06d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f85040012e0 tx=0x7f8504009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.640+0000 7f84faffd640 1 -- 172.21.15.59:0/1112071840 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84fc07a360 con 0x7f850c1614d0 2024-03-06T07:22:23.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.820+0000 7f8510993640 1 -- 172.21.15.59:0/1112071840 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f850c0b3920 con 0x7f850c1614d0 2024-03-06T07:22:23.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.825+0000 7f84faffd640 1 -- 172.21.15.59:0/1112071840 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f84fc07e400 con 0x7f850c1614d0 2024-03-06T07:22:23.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 -- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84ec06b4d0 msgr2=0x7f84ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84ec06b4d0 0x7f84ec06d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f85040012e0 tx=0x7f8504009040 comp rx=0 tx=0).stop 2024-03-06T07:22:23.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 -- 172.21.15.59:0/1112071840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f850c1614d0 msgr2=0x7f850c1f73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f850c1614d0 0x7f850c1f73b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f84fc034040 tx=0x7f84fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 -- 172.21.15.59:0/1112071840 shutdown_connections 2024-03-06T07:22:23.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84ec06b4d0 0x7f84ec06d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f850c1614d0 0x7f850c1f73b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f850c0d4860 0x7f850c0b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 --2- 172.21.15.59:0/1112071840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f850c0b1e40 0x7f850c0b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 -- 172.21.15.59:0/1112071840 >> 172.21.15.59:0/1112071840 conn(0x7f850c0d3d40 msgr2=0x7f850c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 -- 172.21.15.59:0/1112071840 shutdown_connections 2024-03-06T07:22:23.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:23.827+0000 7f8510993640 1 -- 172.21.15.59:0/1112071840 wait complete. 2024-03-06T07:22:23.840 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:22:23.840 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:23.882 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:23.918 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:22:23.918 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:22:23.918 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:23.937 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:22:23.937 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:23.993 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:24.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.133+0000 7f6be6ab5640 1 Processor -- start 2024-03-06T07:22:24.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.133+0000 7f6be6ab5640 1 -- start start 2024-03-06T07:22:24.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.133+0000 7f6be6ab5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6be015dfe0 0x7f6be00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.133+0000 7f6be6ab5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be00b3430 con 0x7f6be00d29c0 2024-03-06T07:22:24.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.133+0000 7f6be6ab5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be00b3710 con 0x7f6be00cf830 2024-03-06T07:22:24.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.133+0000 7f6be6ab5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be00b31e0 con 0x7f6be015dfe0 2024-03-06T07:22:24.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.134+0000 7f6be5052640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6be00d29c0 0x7f6be00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46952/0 (socket says 172.21.15.59:46952) 2024-03-06T07:22:24.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.134+0000 7f6be5052640 1 -- 172.21.15.59:0/1061123979 learned_addr learned my addr 172.21.15.59:0/1061123979 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:24.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.134+0000 7f6be4851640 1 --2- 172.21.15.59:0/1061123979 >> v2:172.21.15.164:3300/0 conn(0x7f6be015dfe0 0x7f6be00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.134+0000 7f6bdf7fe640 1 -- 172.21.15.59:0/1061123979 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951329905 0 0) 0x7f6be00b3430 con 0x7f6be00d29c0 2024-03-06T07:22:24.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.134+0000 7f6bdf7fe640 1 -- 172.21.15.59:0/1061123979 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd0003520 con 0x7f6be00d29c0 2024-03-06T07:22:24.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.134+0000 7f6bdf7fe640 1 -- 172.21.15.59:0/1061123979 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013494772 0 0) 0x7f6be00b3710 con 0x7f6be00cf830 2024-03-06T07:22:24.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.134+0000 7f6bdf7fe640 1 -- 172.21.15.59:0/1061123979 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6be00b3430 con 0x7f6be00cf830 2024-03-06T07:22:24.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.134+0000 7f6be4851640 1 -- 172.21.15.59:0/1061123979 >> v1:172.21.15.164:6789/0 conn(0x7f6be00cf830 legacy=0x7f6be00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.134+0000 7f6be4851640 1 -- 172.21.15.59:0/1061123979 >> v1:172.21.15.59:6789/0 conn(0x7f6be00d29c0 legacy=0x7f6be00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.134+0000 7f6be4851640 1 -- 172.21.15.59:0/1061123979 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be00b39f0 con 0x7f6be015dfe0 2024-03-06T07:22:24.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.135+0000 7f6be4851640 1 --2- 172.21.15.59:0/1061123979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6be015dfe0 0x7f6be00cf2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6be00b3360 tx=0x7f6bd402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a81817e4eb334d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.135+0000 7f6bdf7fe640 1 -- 172.21.15.59:0/1061123979 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4034070 con 0x7f6be015dfe0 2024-03-06T07:22:24.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.135+0000 7f6bdf7fe640 1 -- 172.21.15.59:0/1061123979 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd4030610 con 0x7f6be015dfe0 2024-03-06T07:22:24.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.135+0000 7f6bdf7fe640 1 -- 172.21.15.59:0/1061123979 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40309e0 con 0x7f6be015dfe0 2024-03-06T07:22:24.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.135+0000 7f6be6ab5640 1 -- 172.21.15.59:0/1061123979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6be015dfe0 msgr2=0x7f6be00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.135+0000 7f6be6ab5640 1 --2- 172.21.15.59:0/1061123979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6be015dfe0 0x7f6be00cf2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6be00b3360 tx=0x7f6bd402b960 comp rx=0 tx=0).stop 2024-03-06T07:22:24.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.135+0000 7f6be6ab5640 1 -- 172.21.15.59:0/1061123979 shutdown_connections 2024-03-06T07:22:24.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.135+0000 7f6be6ab5640 1 --2- 172.21.15.59:0/1061123979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6be015dfe0 0x7f6be00cf2f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.135+0000 7f6be6ab5640 1 -- 172.21.15.59:0/1061123979 >> 172.21.15.59:0/1061123979 conn(0x7f6be00d3cd0 msgr2=0x7f6be00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.136+0000 7f6be6ab5640 1 -- 172.21.15.59:0/1061123979 shutdown_connections 2024-03-06T07:22:24.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.136+0000 7f6be6ab5640 1 -- 172.21.15.59:0/1061123979 wait complete. 2024-03-06T07:22:24.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.137+0000 7f6be6ab5640 1 Processor -- start 2024-03-06T07:22:24.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.137+0000 7f6be6ab5640 1 -- start start 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be6ab5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6be00cf830 0x7f6be00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be6ab5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6be00d29c0 0x7f6be006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be6ab5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6be015dfe0 0x7f6be014ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be6ab5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be015e3c0 con 0x7f6be00d29c0 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be6ab5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be00b3d60 con 0x7f6be015dfe0 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be6ab5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be015e540 con 0x7f6be00cf830 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6be00d29c0 0x7f6be006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be4851640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6be00cf830 0x7f6be00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6be00d29c0 0x7f6be006bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52668/0 (socket says 172.21.15.59:52668) 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be5052640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6be015dfe0 0x7f6be014ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdffff640 1 -- 172.21.15.59:0/3141116095 learned_addr learned my addr 172.21.15.59:0/3141116095 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:24.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdffff640 1 -- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6be00cf830 msgr2=0x7f6be00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdffff640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6be00cf830 0x7f6be00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdffff640 1 -- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6be015dfe0 msgr2=0x7f6be014ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdffff640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6be015dfe0 0x7f6be014ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdffff640 1 -- 172.21.15.59:0/3141116095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be01f75a0 con 0x7f6be00d29c0 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be5052640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6be015dfe0 0x7f6be014ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be4851640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6be00cf830 0x7f6be00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdffff640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6be00d29c0 0x7f6be006bea0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c400 tx=0x7f6bcc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdd7fa640 1 -- 172.21.15.59:0/3141116095 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc017070 con 0x7f6be00d29c0 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdd7fa640 1 -- 172.21.15.59:0/3141116095 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc0052d0 con 0x7f6be00d29c0 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6bdd7fa640 1 -- 172.21.15.59:0/3141116095 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc003810 con 0x7f6be00d29c0 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be6ab5640 1 -- 172.21.15.59:0/3141116095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be01f7880 con 0x7f6be00d29c0 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.138+0000 7f6be6ab5640 1 -- 172.21.15.59:0/3141116095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be01f8120 con 0x7f6be00d29c0 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.139+0000 7f6be6ab5640 1 -- 172.21.15.59:0/3141116095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be01f8320 con 0x7f6be00d29c0 2024-03-06T07:22:24.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.140+0000 7f6bdd7fa640 1 -- 172.21.15.59:0/3141116095 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bcc01a070 con 0x7f6be00d29c0 2024-03-06T07:22:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.140+0000 7f6bdd7fa640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bc406b4d0 0x7f6bc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.140+0000 7f6bdd7fa640 1 -- 172.21.15.59:0/3141116095 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bcc089610 con 0x7f6be00d29c0 2024-03-06T07:22:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.140+0000 7f6be4851640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bc406b4d0 0x7f6bc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.141+0000 7f6be4851640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bc406b4d0 0x7f6bc406d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4036000 tx=0x7f6bd403b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.144+0000 7f6bdd7fa640 1 -- 172.21.15.59:0/3141116095 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6be01f8320 con 0x7f6be00d29c0 2024-03-06T07:22:24.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.328+0000 7f6be6ab5640 1 -- 172.21.15.59:0/3141116095 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6be004f330 con 0x7f6be00d29c0 2024-03-06T07:22:24.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.329+0000 7f6bdd7fa640 1 -- 172.21.15.59:0/3141116095 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f6bcc057dc0 con 0x7f6be00d29c0 2024-03-06T07:22:24.331 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:24.331 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:24.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 -- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bc406b4d0 msgr2=0x7f6bc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bc406b4d0 0x7f6bc406d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4036000 tx=0x7f6bd403b040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 -- 172.21.15.59:0/3141116095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6be00d29c0 msgr2=0x7f6be006bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6be00d29c0 0x7f6be006bea0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c400 tx=0x7f6bcc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 -- 172.21.15.59:0/3141116095 shutdown_connections 2024-03-06T07:22:24.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bc406b4d0 0x7f6bc406d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6be015dfe0 0x7f6be014ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6be00d29c0 0x7f6be006bea0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 --2- 172.21.15.59:0/3141116095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6be00cf830 0x7f6be00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 -- 172.21.15.59:0/3141116095 >> 172.21.15.59:0/3141116095 conn(0x7f6be00d3cd0 msgr2=0x7f6be00b5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 -- 172.21.15.59:0/3141116095 shutdown_connections 2024-03-06T07:22:24.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.331+0000 7f6be6ab5640 1 -- 172.21.15.59:0/3141116095 wait complete. 2024-03-06T07:22:24.344 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:24.344 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.447+0000 7fcc9f82a640 1 Processor -- start 2024-03-06T07:22:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.447+0000 7fcc9f82a640 1 -- start start 2024-03-06T07:22:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9f82a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcc9815def0 0x7fcc980b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9f82a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9806cab0 0x7fcc980ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9f82a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcc980ce880 0x7fcc980cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9f82a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc980cecf0 con 0x7fcc9815def0 2024-03-06T07:22:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9f82a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc980cf1c0 con 0x7fcc980ce880 2024-03-06T07:22:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9f82a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc980cc760 con 0x7fcc9806cab0 2024-03-06T07:22:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9d5c6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcc9815def0 0x7fcc980b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9d5c6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcc9815def0 0x7fcc980b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52680/0 (socket says 172.21.15.59:52680) 2024-03-06T07:22:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9d5c6640 1 -- 172.21.15.59:0/2842208278 learned_addr learned my addr 172.21.15.59:0/2842208278 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9ddc7640 1 --2- 172.21.15.59:0/2842208278 >> v2:172.21.15.164:3300/0 conn(0x7fcc980ce880 0x7fcc980cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9cdc5640 1 --2- 172.21.15.59:0/2842208278 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9806cab0 0x7fcc980ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9d5c6640 1 -- 172.21.15.59:0/2842208278 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9806cab0 msgr2=0x7fcc980ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9d5c6640 1 --2- 172.21.15.59:0/2842208278 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9806cab0 0x7fcc980ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9d5c6640 1 -- 172.21.15.59:0/2842208278 >> v2:172.21.15.164:3300/0 conn(0x7fcc980ce880 msgr2=0x7fcc980cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9d5c6640 1 --2- 172.21.15.59:0/2842208278 >> v2:172.21.15.164:3300/0 conn(0x7fcc980ce880 0x7fcc980cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.448+0000 7fcc9d5c6640 1 -- 172.21.15.59:0/2842208278 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9806d030 con 0x7fcc9815def0 2024-03-06T07:22:24.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.449+0000 7fcc9d5c6640 1 --2- 172.21.15.59:0/2842208278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9815def0 0x7fcc980b41b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcc98071e70 tx=0x7fcc9402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d693cd09a8b49b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.449+0000 7fcc8ffff640 1 -- 172.21.15.59:0/2842208278 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94034070 con 0x7fcc9815def0 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.449+0000 7fcc8ffff640 1 -- 172.21.15.59:0/2842208278 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc94030860 con 0x7fcc9815def0 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.449+0000 7fcc8ffff640 1 -- 172.21.15.59:0/2842208278 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94030c30 con 0x7fcc9815def0 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.449+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2842208278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9815def0 msgr2=0x7fcc980b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.449+0000 7fcc9f82a640 1 --2- 172.21.15.59:0/2842208278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9815def0 0x7fcc980b41b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcc98071e70 tx=0x7fcc9402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.449+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2842208278 shutdown_connections 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.450+0000 7fcc9f82a640 1 --2- 172.21.15.59:0/2842208278 >> v2:172.21.15.164:3300/0 conn(0x7fcc980ce880 0x7fcc980cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.450+0000 7fcc9f82a640 1 --2- 172.21.15.59:0/2842208278 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9806cab0 0x7fcc980ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.450+0000 7fcc9f82a640 1 --2- 172.21.15.59:0/2842208278 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9815def0 0x7fcc980b41b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.450+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2842208278 >> 172.21.15.59:0/2842208278 conn(0x7fcc980b3690 msgr2=0x7fcc980b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.450+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2842208278 shutdown_connections 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.450+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2842208278 wait complete. 2024-03-06T07:22:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.451+0000 7fcc9f82a640 1 Processor -- start 2024-03-06T07:22:24.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.451+0000 7fcc9f82a640 1 -- start start 2024-03-06T07:22:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.451+0000 7fcc9f82a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9806cab0 0x7fcc980b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.451+0000 7fcc9f82a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc980ce880 0x7fcc9814bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.451+0000 7fcc9f82a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9815def0 0x7fcc9814c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.451+0000 7fcc9f82a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc980cc3f0 con 0x7fcc9806cab0 2024-03-06T07:22:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.451+0000 7fcc9f82a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc980cc110 con 0x7fcc980ce880 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.451+0000 7fcc9f82a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9815e2d0 con 0x7fcc9815def0 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9d5c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9806cab0 0x7fcc980b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9d5c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9806cab0 0x7fcc980b3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52684/0 (socket says 172.21.15.59:52684) 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9d5c6640 1 -- 172.21.15.59:0/2956579476 learned_addr learned my addr 172.21.15.59:0/2956579476 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9cdc5640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc980ce880 0x7fcc9814bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9ddc7640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9815def0 0x7fcc9814c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9d5c6640 1 -- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9815def0 msgr2=0x7fcc9814c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9d5c6640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9815def0 0x7fcc9814c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9d5c6640 1 -- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc980ce880 msgr2=0x7fcc9814bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9d5c6640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc980ce880 0x7fcc9814bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9d5c6640 1 -- 172.21.15.59:0/2956579476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc980b7620 con 0x7fcc9806cab0 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9ddc7640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9815def0 0x7fcc9814c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9cdc5640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc980ce880 0x7fcc9814bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9d5c6640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9806cab0 0x7fcc980b3ce0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcc9402b9e0 tx=0x7fcc940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc8dffb640 1 -- 172.21.15.59:0/2956579476 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94034040 con 0x7fcc9806cab0 2024-03-06T07:22:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc8dffb640 1 -- 172.21.15.59:0/2956579476 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc94039d90 con 0x7fcc9806cab0 2024-03-06T07:22:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2956579476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc980ca780 con 0x7fcc9806cab0 2024-03-06T07:22:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc8dffb640 1 -- 172.21.15.59:0/2956579476 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc940433f0 con 0x7fcc9806cab0 2024-03-06T07:22:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.452+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2956579476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc980cad40 con 0x7fcc9806cab0 2024-03-06T07:22:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.453+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2956579476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc980d19c0 con 0x7fcc9806cab0 2024-03-06T07:22:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.454+0000 7fcc8dffb640 1 -- 172.21.15.59:0/2956579476 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc9404a060 con 0x7fcc9806cab0 2024-03-06T07:22:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.454+0000 7fcc8dffb640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 0x7fcc7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.454+0000 7fcc8dffb640 1 -- 172.21.15.59:0/2956579476 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc940ac100 con 0x7fcc9806cab0 2024-03-06T07:22:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.454+0000 7fcc9cdc5640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 0x7fcc7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.455+0000 7fcc9cdc5640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 0x7fcc7c06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcc980b33d0 tx=0x7fcc88008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.458+0000 7fcc8dffb640 1 -- 172.21.15.59:0/2956579476 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc980d19c0 con 0x7fcc9806cab0 2024-03-06T07:22:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.639+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2956579476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc9815d710 con 0x7fcc9806cab0 2024-03-06T07:22:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.639+0000 7fcc8dffb640 1 -- 172.21.15.59:0/2956579476 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcc980b7620 con 0x7fcc9806cab0 2024-03-06T07:22:24.641 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 msgr2=0x7fcc7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 0x7fcc7c06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcc980b33d0 tx=0x7fcc88008040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2956579476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9806cab0 msgr2=0x7fcc980b3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9806cab0 0x7fcc980b3ce0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcc9402b9e0 tx=0x7fcc940046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2956579476 shutdown_connections 2024-03-06T07:22:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 0x7fcc7c06d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9815def0 0x7fcc9814c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc980ce880 0x7fcc9814bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 --2- 172.21.15.59:0/2956579476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9806cab0 0x7fcc980b3ce0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2956579476 >> 172.21.15.59:0/2956579476 conn(0x7fcc980b3690 msgr2=0x7fcc980d12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2956579476 shutdown_connections 2024-03-06T07:22:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.641+0000 7fcc9f82a640 1 -- 172.21.15.59:0/2956579476 wait complete. 2024-03-06T07:22:24.653 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:24.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.757+0000 7f96815e9640 1 Processor -- start 2024-03-06T07:22:24.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.757+0000 7f96815e9640 1 -- start start 2024-03-06T07:22:24.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.757+0000 7f96815e9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f967c15def0 0x7f967c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.757+0000 7f96815e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f967c0cea50 0x7f967c0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.757+0000 7f96815e9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c06d030 con 0x7f967c06cab0 2024-03-06T07:22:24.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.757+0000 7f96815e9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c0cf1c0 con 0x7f967c15def0 2024-03-06T07:22:24.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.757+0000 7f96815e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0cc760 con 0x7f967c0cea50 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.758+0000 7f967a7fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f967c06cab0 0x7f967c0ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46960/0 (socket says 172.21.15.59:46960) 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.758+0000 7f967a7fc640 1 -- 172.21.15.59:0/1061204414 learned_addr learned my addr 172.21.15.59:0/1061204414 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.758+0000 7f967b7fe640 1 --2- 172.21.15.59:0/1061204414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f967c0cea50 0x7f967c0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.758+0000 7f967affd640 1 --2- 172.21.15.59:0/1061204414 >> v2:172.21.15.164:3300/0 conn(0x7f967c15def0 0x7f967c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.758+0000 7f9679ffb640 1 -- 172.21.15.59:0/1061204414 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095549858 0 0) 0x7f967c06d030 con 0x7f967c06cab0 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.758+0000 7f9679ffb640 1 -- 172.21.15.59:0/1061204414 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9664003520 con 0x7f967c06cab0 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.758+0000 7f967b7fe640 1 -- 172.21.15.59:0/1061204414 >> v2:172.21.15.164:3300/0 conn(0x7f967c15def0 msgr2=0x7f967c0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.758+0000 7f967b7fe640 1 --2- 172.21.15.59:0/1061204414 >> v2:172.21.15.164:3300/0 conn(0x7f967c15def0 0x7f967c0b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.758+0000 7f967b7fe640 1 -- 172.21.15.59:0/1061204414 >> v1:172.21.15.59:6789/0 conn(0x7f967c06cab0 legacy=0x7f967c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.758+0000 7f967b7fe640 1 -- 172.21.15.59:0/1061204414 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c06d310 con 0x7f967c0cea50 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.758+0000 7f967affd640 1 --2- 172.21.15.59:0/1061204414 >> v2:172.21.15.164:3300/0 conn(0x7f967c15def0 0x7f967c0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.759+0000 7f967b7fe640 1 --2- 172.21.15.59:0/1061204414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f967c0cea50 0x7f967c0ccbf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9674009d90 tx=0x7f967402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55804bb35dd41ab0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.759+0000 7f9679ffb640 1 -- 172.21.15.59:0/1061204414 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674039070 con 0x7f967c0cea50 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.759+0000 7f9679ffb640 1 -- 172.21.15.59:0/1061204414 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967402cce0 con 0x7f967c0cea50 2024-03-06T07:22:24.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.759+0000 7f9679ffb640 1 -- 172.21.15.59:0/1061204414 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674034660 con 0x7f967c0cea50 2024-03-06T07:22:24.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.759+0000 7f96815e9640 1 -- 172.21.15.59:0/1061204414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f967c0cea50 msgr2=0x7f967c0ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.759+0000 7f96815e9640 1 --2- 172.21.15.59:0/1061204414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f967c0cea50 0x7f967c0ccbf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9674009d90 tx=0x7f967402bf30 comp rx=0 tx=0).stop 2024-03-06T07:22:24.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.759+0000 7f96815e9640 1 -- 172.21.15.59:0/1061204414 shutdown_connections 2024-03-06T07:22:24.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.759+0000 7f96815e9640 1 --2- 172.21.15.59:0/1061204414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f967c0cea50 0x7f967c0ccbf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.759+0000 7f96815e9640 1 --2- 172.21.15.59:0/1061204414 >> v2:172.21.15.164:3300/0 conn(0x7f967c15def0 0x7f967c0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.759+0000 7f96815e9640 1 -- 172.21.15.59:0/1061204414 >> 172.21.15.59:0/1061204414 conn(0x7f967c0b3690 msgr2=0x7f967c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.760+0000 7f96815e9640 1 -- 172.21.15.59:0/1061204414 shutdown_connections 2024-03-06T07:22:24.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.760+0000 7f96815e9640 1 -- 172.21.15.59:0/1061204414 wait complete. 2024-03-06T07:22:24.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.761+0000 7f96815e9640 1 Processor -- start 2024-03-06T07:22:24.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.761+0000 7f96815e9640 1 -- start start 2024-03-06T07:22:24.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f96815e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f967c06cab0 0x7f967c15c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f96815e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f967c0cea50 0x7f967c0b6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f96815e9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f967c15def0 0x7f967c0cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f96815e9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c15e2d0 con 0x7f967c0cea50 2024-03-06T07:22:24.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f96815e9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0cc3f0 con 0x7f967c15def0 2024-03-06T07:22:24.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f96815e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c15e450 con 0x7f967c06cab0 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f967a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f967c0cea50 0x7f967c0b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f967affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f967c06cab0 0x7f967c15c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f967a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f967c0cea50 0x7f967c0b6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52692/0 (socket says 172.21.15.59:52692) 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f967a7fc640 1 -- 172.21.15.59:0/3482422836 learned_addr learned my addr 172.21.15.59:0/3482422836 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f967b7fe640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f967c15def0 0x7f967c0cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f967a7fc640 1 -- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f967c06cab0 msgr2=0x7f967c15c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f967a7fc640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f967c06cab0 0x7f967c15c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f967a7fc640 1 -- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f967c15def0 msgr2=0x7f967c0cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f967a7fc640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f967c15def0 0x7f967c0cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f967a7fc640 1 -- 172.21.15.59:0/3482422836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c0b6fd0 con 0x7f967c0cea50 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f967a7fc640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f967c0cea50 0x7f967c0b6230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f967000c460 tx=0x7f967000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f96637fe640 1 -- 172.21.15.59:0/3482422836 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670017070 con 0x7f967c0cea50 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f96637fe640 1 -- 172.21.15.59:0/3482422836 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9670005ce0 con 0x7f967c0cea50 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f96815e9640 1 -- 172.21.15.59:0/3482422836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c0b6cf0 con 0x7f967c0cea50 2024-03-06T07:22:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f96637fe640 1 -- 172.21.15.59:0/3482422836 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670007290 con 0x7f967c0cea50 2024-03-06T07:22:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.762+0000 7f96815e9640 1 -- 172.21.15.59:0/3482422836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c0c92f0 con 0x7f967c0cea50 2024-03-06T07:22:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.763+0000 7f96815e9640 1 -- 172.21.15.59:0/3482422836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c06c070 con 0x7f967c0cea50 2024-03-06T07:22:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.764+0000 7f96637fe640 1 -- 172.21.15.59:0/3482422836 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f967001a070 con 0x7f967c0cea50 2024-03-06T07:22:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.764+0000 7f96637fe640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f966806b500 0x7f966806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.764+0000 7f96637fe640 1 -- 172.21.15.59:0/3482422836 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9670089460 con 0x7f967c0cea50 2024-03-06T07:22:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.764+0000 7f967affd640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f966806b500 0x7f966806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.765+0000 7f967affd640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f966806b500 0x7f966806d9c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f966c009560 tx=0x7f966c009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.769+0000 7f96637fe640 1 -- 172.21.15.59:0/3482422836 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f967c06c070 con 0x7f967c0cea50 2024-03-06T07:22:24.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.950+0000 7f96815e9640 1 -- 172.21.15.59:0/3482422836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f967c0c94f0 con 0x7f967c0cea50 2024-03-06T07:22:24.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.950+0000 7f96637fe640 1 -- 172.21.15.59:0/3482422836 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f967c0c94f0 con 0x7f967c0cea50 2024-03-06T07:22:24.952 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:24.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 -- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f966806b500 msgr2=0x7f966806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f966806b500 0x7f966806d9c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f966c009560 tx=0x7f966c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:24.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 -- 172.21.15.59:0/3482422836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f967c0cea50 msgr2=0x7f967c0b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f967c0cea50 0x7f967c0b6230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f967000c460 tx=0x7f967000c920 comp rx=0 tx=0).stop 2024-03-06T07:22:24.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 -- 172.21.15.59:0/3482422836 shutdown_connections 2024-03-06T07:22:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f966806b500 0x7f966806d9c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f967c15def0 0x7f967c0cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f967c0cea50 0x7f967c0b6230 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 --2- 172.21.15.59:0/3482422836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f967c06cab0 0x7f967c15c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 -- 172.21.15.59:0/3482422836 >> 172.21.15.59:0/3482422836 conn(0x7f967c0b3690 msgr2=0x7f967c160a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 -- 172.21.15.59:0/3482422836 shutdown_connections 2024-03-06T07:22:24.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:24.952+0000 7f96815e9640 1 -- 172.21.15.59:0/3482422836 wait complete. 2024-03-06T07:22:24.965 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:25.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.067+0000 7f9d9ceeb640 1 Processor -- start 2024-03-06T07:22:25.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.067+0000 7f9d9ceeb640 1 -- start start 2024-03-06T07:22:25.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.068+0000 7f9d9ceeb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9d9815c050 0x7f9d9806f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.068+0000 7f9d9ceeb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d980d0af0 0x7f9d980cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.068+0000 7f9d9ceeb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d980cd4b0 con 0x7f9d9806fb40 2024-03-06T07:22:25.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.068+0000 7f9d9ceeb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d9806ffb0 con 0x7f9d9815c050 2024-03-06T07:22:25.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.068+0000 7f9d9ceeb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d980cd790 con 0x7f9d980d0af0 2024-03-06T07:22:25.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.068+0000 7f9d95d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9d9806fb40 0x7f9d980cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46966/0 (socket says 172.21.15.59:46966) 2024-03-06T07:22:25.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.068+0000 7f9d95d9b640 1 -- 172.21.15.59:0/1612349399 learned_addr learned my addr 172.21.15.59:0/1612349399 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:25.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.068+0000 7f9d9659c640 1 --2- 172.21.15.59:0/1612349399 >> v2:172.21.15.164:3300/0 conn(0x7f9d9815c050 0x7f9d9806f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.068+0000 7f9d96d9d640 1 --2- 172.21.15.59:0/1612349399 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d980d0af0 0x7f9d980cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.069+0000 7f9d9559a640 1 -- 172.21.15.59:0/1612349399 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287263429 0 0) 0x7f9d980cd4b0 con 0x7f9d9806fb40 2024-03-06T07:22:25.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.069+0000 7f9d9559a640 1 -- 172.21.15.59:0/1612349399 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d80003520 con 0x7f9d9806fb40 2024-03-06T07:22:25.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.069+0000 7f9d9659c640 1 -- 172.21.15.59:0/1612349399 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d980d0af0 msgr2=0x7f9d980cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.069+0000 7f9d9659c640 1 --2- 172.21.15.59:0/1612349399 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d980d0af0 0x7f9d980cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.069+0000 7f9d9659c640 1 -- 172.21.15.59:0/1612349399 >> v1:172.21.15.59:6789/0 conn(0x7f9d9806fb40 legacy=0x7f9d980cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.069+0000 7f9d9659c640 1 -- 172.21.15.59:0/1612349399 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d980cda70 con 0x7f9d9815c050 2024-03-06T07:22:25.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.069+0000 7f9d96d9d640 1 --2- 172.21.15.59:0/1612349399 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d980d0af0 0x7f9d980cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9659c640 1 --2- 172.21.15.59:0/1612349399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d9815c050 0x7f9d9806f600 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009d90 tx=0x7f9d8802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8ce8c239d6db92a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9559a640 1 -- 172.21.15.59:0/1612349399 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88039070 con 0x7f9d9815c050 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9559a640 1 -- 172.21.15.59:0/1612349399 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8802cbc0 con 0x7f9d9815c050 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9559a640 1 -- 172.21.15.59:0/1612349399 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88034440 con 0x7f9d9815c050 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/1612349399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d9815c050 msgr2=0x7f9d9806f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9ceeb640 1 --2- 172.21.15.59:0/1612349399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d9815c050 0x7f9d9806f600 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009d90 tx=0x7f9d8802be10 comp rx=0 tx=0).stop 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/1612349399 shutdown_connections 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9ceeb640 1 --2- 172.21.15.59:0/1612349399 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d980d0af0 0x7f9d980cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9ceeb640 1 --2- 172.21.15.59:0/1612349399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d9815c050 0x7f9d9806f600 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/1612349399 >> 172.21.15.59:0/1612349399 conn(0x7f9d980b3800 msgr2=0x7f9d980b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/1612349399 shutdown_connections 2024-03-06T07:22:25.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.070+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/1612349399 wait complete. 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.072+0000 7f9d9ceeb640 1 Processor -- start 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.072+0000 7f9d9ceeb640 1 -- start start 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.072+0000 7f9d9ceeb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d9806fb40 0x7f9d98152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.072+0000 7f9d9ceeb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d980d0af0 0x7f9d98153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.072+0000 7f9d9ceeb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d9815c050 0x7f9d980b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.072+0000 7f9d9ceeb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d98165c30 con 0x7f9d9806fb40 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.072+0000 7f9d9ceeb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d98165ab0 con 0x7f9d980d0af0 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.072+0000 7f9d9ceeb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d98165db0 con 0x7f9d9815c050 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d9659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d9806fb40 0x7f9d98152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d9659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d9806fb40 0x7f9d98152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52706/0 (socket says 172.21.15.59:52706) 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d9659c640 1 -- 172.21.15.59:0/3315475687 learned_addr learned my addr 172.21.15.59:0/3315475687 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d96d9d640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d9815c050 0x7f9d980b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d95d9b640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d980d0af0 0x7f9d98153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d9659c640 1 -- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d9815c050 msgr2=0x7f9d980b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d9659c640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d9815c050 0x7f9d980b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d9659c640 1 -- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d980d0af0 msgr2=0x7f9d98153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d9659c640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d980d0af0 0x7f9d98153220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d9659c640 1 -- 172.21.15.59:0/3315475687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d98153b20 con 0x7f9d9806fb40 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d9659c640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d9806fb40 0x7f9d98152a00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9d88037040 tx=0x7f9d88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d86ffd640 1 -- 172.21.15.59:0/3315475687 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88039050 con 0x7f9d9806fb40 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d86ffd640 1 -- 172.21.15.59:0/3315475687 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d88045070 con 0x7f9d9806fb40 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d86ffd640 1 -- 172.21.15.59:0/3315475687 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88009450 con 0x7f9d9806fb40 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/3315475687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d98153e00 con 0x7f9d9806fb40 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.073+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/3315475687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d981543c0 con 0x7f9d9806fb40 2024-03-06T07:22:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.074+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/3315475687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9809fe60 con 0x7f9d9806fb40 2024-03-06T07:22:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.075+0000 7f9d86ffd640 1 -- 172.21.15.59:0/3315475687 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d8803e850 con 0x7f9d9806fb40 2024-03-06T07:22:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.075+0000 7f9d86ffd640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b500 0x7f9d7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.075+0000 7f9d86ffd640 1 -- 172.21.15.59:0/3315475687 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d880ac5f0 con 0x7f9d9806fb40 2024-03-06T07:22:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.075+0000 7f9d95d9b640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b500 0x7f9d7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.076+0000 7f9d95d9b640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b500 0x7f9d7806d9c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9d980d21e0 tx=0x7f9d8c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.079+0000 7f9d86ffd640 1 -- 172.21.15.59:0/3315475687 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d8804e020 con 0x7f9d9806fb40 2024-03-06T07:22:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.258+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/3315475687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9d980b2320 con 0x7f9d9806fb40 2024-03-06T07:22:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.258+0000 7f9d86ffd640 1 -- 172.21.15.59:0/3315475687 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9d8807aef0 con 0x7f9d9806fb40 2024-03-06T07:22:25.260 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:25.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.260+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b500 msgr2=0x7f9d7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.260+0000 7f9d9ceeb640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b500 0x7f9d7806d9c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9d980d21e0 tx=0x7f9d8c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.260+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/3315475687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d9806fb40 msgr2=0x7f9d98152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.260+0000 7f9d9ceeb640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d9806fb40 0x7f9d98152a00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9d88037040 tx=0x7f9d88004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.261+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/3315475687 shutdown_connections 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.261+0000 7f9d9ceeb640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d7806b500 0x7f9d7806d9c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.261+0000 7f9d9ceeb640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d9815c050 0x7f9d980b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.261+0000 7f9d9ceeb640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d980d0af0 0x7f9d98153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.261+0000 7f9d9ceeb640 1 --2- 172.21.15.59:0/3315475687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d9806fb40 0x7f9d98152a00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.261+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/3315475687 >> 172.21.15.59:0/3315475687 conn(0x7f9d980b3800 msgr2=0x7f9d9815b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.261+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/3315475687 shutdown_connections 2024-03-06T07:22:25.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.261+0000 7f9d9ceeb640 1 -- 172.21.15.59:0/3315475687 wait complete. 2024-03-06T07:22:25.273 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:25.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.374+0000 7fcd20840640 1 Processor -- start 2024-03-06T07:22:25.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.374+0000 7fcd20840640 1 -- start start 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.375+0000 7fcd20840640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0d04e0 0x7fcd1c0d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.375+0000 7fcd20840640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c0b2d80 con 0x7fcd1c15bed0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.375+0000 7fcd20840640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c0ae9c0 con 0x7fcd1c0d1fe0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.375+0000 7fcd20840640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0d2470 con 0x7fcd1c0d04e0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.376+0000 7fcd1a59c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcd1c15bed0 0x7fcd1c0b2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46976/0 (socket says 172.21.15.59:46976) 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.376+0000 7fcd1a59c640 1 -- 172.21.15.59:0/3877922716 learned_addr learned my addr 172.21.15.59:0/3877922716 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.376+0000 7fcd19d9b640 1 --2- 172.21.15.59:0/3877922716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0d04e0 0x7fcd1c0d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.376+0000 7fcd1959a640 1 -- 172.21.15.59:0/3877922716 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978638375 0 0) 0x7fcd1c0b2d80 con 0x7fcd1c15bed0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.376+0000 7fcd1959a640 1 -- 172.21.15.59:0/3877922716 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd08003520 con 0x7fcd1c15bed0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.376+0000 7fcd1959a640 1 -- 172.21.15.59:0/3877922716 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92665173 0 0) 0x7fcd1c0ae9c0 con 0x7fcd1c0d1fe0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.376+0000 7fcd1959a640 1 -- 172.21.15.59:0/3877922716 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd1c0b2d80 con 0x7fcd1c0d1fe0 2024-03-06T07:22:25.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.376+0000 7fcd19d9b640 1 -- 172.21.15.59:0/3877922716 >> v1:172.21.15.164:6789/0 conn(0x7fcd1c0d1fe0 legacy=0x7fcd1c0af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.376+0000 7fcd19d9b640 1 -- 172.21.15.59:0/3877922716 >> v1:172.21.15.59:6789/0 conn(0x7fcd1c15bed0 legacy=0x7fcd1c0b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.376+0000 7fcd19d9b640 1 -- 172.21.15.59:0/3877922716 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c0aeca0 con 0x7fcd1c0d04e0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.377+0000 7fcd19d9b640 1 --2- 172.21.15.59:0/3877922716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0d04e0 0x7fcd1c0d1aa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcd040097d0 tx=0x7fcd0402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70a3da446810647 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.377+0000 7fcd1959a640 1 -- 172.21.15.59:0/3877922716 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd04035020 con 0x7fcd1c0d04e0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.377+0000 7fcd1959a640 1 -- 172.21.15.59:0/3877922716 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd04031800 con 0x7fcd1c0d04e0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.377+0000 7fcd1959a640 1 -- 172.21.15.59:0/3877922716 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd04031bd0 con 0x7fcd1c0d04e0 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.377+0000 7fcd20840640 1 -- 172.21.15.59:0/3877922716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0d04e0 msgr2=0x7fcd1c0d1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.377+0000 7fcd20840640 1 --2- 172.21.15.59:0/3877922716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0d04e0 0x7fcd1c0d1aa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcd040097d0 tx=0x7fcd0402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.378+0000 7fcd20840640 1 -- 172.21.15.59:0/3877922716 shutdown_connections 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.378+0000 7fcd20840640 1 --2- 172.21.15.59:0/3877922716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0d04e0 0x7fcd1c0d1aa0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.378+0000 7fcd20840640 1 -- 172.21.15.59:0/3877922716 >> 172.21.15.59:0/3877922716 conn(0x7fcd1c09a9f0 msgr2=0x7fcd1c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.378+0000 7fcd20840640 1 -- 172.21.15.59:0/3877922716 shutdown_connections 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.378+0000 7fcd20840640 1 -- 172.21.15.59:0/3877922716 wait complete. 2024-03-06T07:22:25.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.379+0000 7fcd20840640 1 Processor -- start 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.379+0000 7fcd20840640 1 -- start start 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd20840640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0d04e0 0x7fcd1c06f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd20840640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd1c0d1fe0 0x7fcd1c14bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd20840640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd1c15bed0 0x7fcd1c14c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd20840640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c15c2b0 con 0x7fcd1c15bed0 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd20840640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0af260 con 0x7fcd1c0d1fe0 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd20840640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c14ca30 con 0x7fcd1c0d04e0 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd1ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd1c15bed0 0x7fcd1c14c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd1a59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0d04e0 0x7fcd1c06f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd1ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd1c15bed0 0x7fcd1c14c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52722/0 (socket says 172.21.15.59:52722) 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd1ad9d640 1 -- 172.21.15.59:0/2011201122 learned_addr learned my addr 172.21.15.59:0/2011201122 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd19d9b640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd1c0d1fe0 0x7fcd1c14bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd1ad9d640 1 -- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0d04e0 msgr2=0x7fcd1c06f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd1ad9d640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0d04e0 0x7fcd1c06f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd1ad9d640 1 -- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd1c0d1fe0 msgr2=0x7fcd1c14bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd1ad9d640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd1c0d1fe0 0x7fcd1c14bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd1ad9d640 1 -- 172.21.15.59:0/2011201122 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c1f54d0 con 0x7fcd1c15bed0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd1ad9d640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd1c15bed0 0x7fcd1c14c4f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400c790 tx=0x7fcd1400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd12ffd640 1 -- 172.21.15.59:0/2011201122 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14003b70 con 0x7fcd1c15bed0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd12ffd640 1 -- 172.21.15.59:0/2011201122 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd14003d10 con 0x7fcd1c15bed0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd20840640 1 -- 172.21.15.59:0/2011201122 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c1f57b0 con 0x7fcd1c15bed0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd12ffd640 1 -- 172.21.15.59:0/2011201122 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1400aa50 con 0x7fcd1c15bed0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.380+0000 7fcd20840640 1 -- 172.21.15.59:0/2011201122 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c1f5d70 con 0x7fcd1c15bed0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.381+0000 7fcd20840640 1 -- 172.21.15.59:0/2011201122 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c04f330 con 0x7fcd1c15bed0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.382+0000 7fcd12ffd640 1 -- 172.21.15.59:0/2011201122 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd1400abf0 con 0x7fcd1c15bed0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.382+0000 7fcd12ffd640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fccfc06b4d0 0x7fccfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.382+0000 7fcd12ffd640 1 -- 172.21.15.59:0/2011201122 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd1408a070 con 0x7fcd1c15bed0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.382+0000 7fcd1a59c640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fccfc06b4d0 0x7fccfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.383+0000 7fcd1a59c640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fccfc06b4d0 0x7fccfc06d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c000f60 tx=0x7fcd0c007d30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.387+0000 7fcd12ffd640 1 -- 172.21.15.59:0/2011201122 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd140588a0 con 0x7fcd1c15bed0 2024-03-06T07:22:25.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.582+0000 7fcd20840640 1 -- 172.21.15.59:0/2011201122 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcd1c0d28b0 con 0x7fcd1c15bed0 2024-03-06T07:22:25.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.582+0000 7fcd12ffd640 1 -- 172.21.15.59:0/2011201122 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fcd1405c940 con 0x7fcd1c15bed0 2024-03-06T07:22:25.584 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:22:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 -- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fccfc06b4d0 msgr2=0x7fccfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fccfc06b4d0 0x7fccfc06d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c000f60 tx=0x7fcd0c007d30 comp rx=0 tx=0).stop 2024-03-06T07:22:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 -- 172.21.15.59:0/2011201122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd1c15bed0 msgr2=0x7fcd1c14c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd1c15bed0 0x7fcd1c14c4f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400c790 tx=0x7fcd1400cc50 comp rx=0 tx=0).stop 2024-03-06T07:22:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 -- 172.21.15.59:0/2011201122 shutdown_connections 2024-03-06T07:22:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fccfc06b4d0 0x7fccfc06d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd1c15bed0 0x7fcd1c14c4f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd1c0d1fe0 0x7fcd1c14bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 --2- 172.21.15.59:0/2011201122 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0d04e0 0x7fcd1c06f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 -- 172.21.15.59:0/2011201122 >> 172.21.15.59:0/2011201122 conn(0x7fcd1c09a9f0 msgr2=0x7fcd1c070460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 -- 172.21.15.59:0/2011201122 shutdown_connections 2024-03-06T07:22:25.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.584+0000 7fcd20840640 1 -- 172.21.15.59:0/2011201122 wait complete. 2024-03-06T07:22:25.596 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:22:25.597 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:25.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.751+0000 7f229fdd7640 1 Processor -- start 2024-03-06T07:22:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.751+0000 7f229fdd7640 1 -- start start 2024-03-06T07:22:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.752+0000 7f229fdd7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f22980b4f70 0x7f22980b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.752+0000 7f229fdd7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22980b5890 con 0x7f2298066d90 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.752+0000 7f229fdd7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22980cd450 con 0x7f229806c6d0 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.752+0000 7f229fdd7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229815e9f0 con 0x7f22980b4f70 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.752+0000 7f229e374640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2298066d90 0x7f22980ccb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46988/0 (socket says 172.21.15.59:46988) 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.752+0000 7f229e374640 1 -- 172.21.15.59:0/687736533 learned_addr learned my addr 172.21.15.59:0/687736533 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.752+0000 7f229db73640 1 --2- 172.21.15.59:0/687736533 >> v2:172.21.15.164:3300/0 conn(0x7f22980b4f70 0x7f22980b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.752+0000 7f229cb71640 1 -- 172.21.15.59:0/687736533 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151487538 0 0) 0x7f22980b5890 con 0x7f2298066d90 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.752+0000 7f229cb71640 1 -- 172.21.15.59:0/687736533 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2284003520 con 0x7f2298066d90 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.753+0000 7f229db73640 1 -- 172.21.15.59:0/687736533 >> v1:172.21.15.164:6789/0 conn(0x7f229806c6d0 legacy=0x7f229815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.753+0000 7f229cb71640 1 -- 172.21.15.59:0/687736533 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786818286 0 0) 0x7f22980cd450 con 0x7f229806c6d0 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.753+0000 7f229db73640 1 -- 172.21.15.59:0/687736533 >> v1:172.21.15.59:6789/0 conn(0x7f2298066d90 legacy=0x7f22980ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.753+0000 7f229db73640 1 -- 172.21.15.59:0/687736533 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22980cd730 con 0x7f22980b4f70 2024-03-06T07:22:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.753+0000 7f229db73640 1 --2- 172.21.15.59:0/687736533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22980b4f70 0x7f22980b5350 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f22900097a0 tx=0x7f229002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fda46f1b766706c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.754+0000 7f229cb71640 1 -- 172.21.15.59:0/687736533 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290034070 con 0x7f22980b4f70 2024-03-06T07:22:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.754+0000 7f229cb71640 1 -- 172.21.15.59:0/687736533 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2290030610 con 0x7f22980b4f70 2024-03-06T07:22:25.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.754+0000 7f229cb71640 1 -- 172.21.15.59:0/687736533 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22900309e0 con 0x7f22980b4f70 2024-03-06T07:22:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.754+0000 7f229fdd7640 1 -- 172.21.15.59:0/687736533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22980b4f70 msgr2=0x7f22980b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.754+0000 7f229fdd7640 1 --2- 172.21.15.59:0/687736533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22980b4f70 0x7f22980b5350 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f22900097a0 tx=0x7f229002b960 comp rx=0 tx=0).stop 2024-03-06T07:22:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.754+0000 7f229fdd7640 1 -- 172.21.15.59:0/687736533 shutdown_connections 2024-03-06T07:22:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.754+0000 7f229fdd7640 1 --2- 172.21.15.59:0/687736533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22980b4f70 0x7f22980b5350 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.754+0000 7f229fdd7640 1 -- 172.21.15.59:0/687736533 >> 172.21.15.59:0/687736533 conn(0x7f22980724d0 msgr2=0x7f22980728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.754+0000 7f229fdd7640 1 -- 172.21.15.59:0/687736533 shutdown_connections 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.754+0000 7f229fdd7640 1 -- 172.21.15.59:0/687736533 wait complete. 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.755+0000 7f229fdd7640 1 Processor -- start 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.755+0000 7f229fdd7640 1 -- start start 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229fdd7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2298066d90 0x7f22981f7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229fdd7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22980b4f70 0x7f22981f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229fdd7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22980ce670 0x7f22980b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229fdd7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2298161ad0 con 0x7f2298066d90 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229fdd7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22980cda10 con 0x7f22980b4f70 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229fdd7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2298161c50 con 0x7f22980ce670 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229db73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2298066d90 0x7f22981f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229db73640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2298066d90 0x7f22981f7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52728/0 (socket says 172.21.15.59:52728) 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229db73640 1 -- 172.21.15.59:0/2868918185 learned_addr learned my addr 172.21.15.59:0/2868918185 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229e374640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22980ce670 0x7f22980b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229d372640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22980b4f70 0x7f22981f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229db73640 1 -- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22980ce670 msgr2=0x7f22980b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229db73640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22980ce670 0x7f22980b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229db73640 1 -- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22980b4f70 msgr2=0x7f22981f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229db73640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22980b4f70 0x7f22981f2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.756+0000 7f229db73640 1 -- 172.21.15.59:0/2868918185 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22980d0300 con 0x7f2298066d90 2024-03-06T07:22:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.757+0000 7f229db73640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2298066d90 0x7f22981f7620 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f229002bf70 tx=0x7f22900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.757+0000 7f228e7fc640 1 -- 172.21.15.59:0/2868918185 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290049070 con 0x7f2298066d90 2024-03-06T07:22:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.757+0000 7f228e7fc640 1 -- 172.21.15.59:0/2868918185 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229003a890 con 0x7f2298066d90 2024-03-06T07:22:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.757+0000 7f228e7fc640 1 -- 172.21.15.59:0/2868918185 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290034040 con 0x7f2298066d90 2024-03-06T07:22:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.757+0000 7f229fdd7640 1 -- 172.21.15.59:0/2868918185 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22981f7b60 con 0x7f2298066d90 2024-03-06T07:22:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.757+0000 7f229fdd7640 1 -- 172.21.15.59:0/2868918185 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22980cebe0 con 0x7f2298066d90 2024-03-06T07:22:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.758+0000 7f229fdd7640 1 -- 172.21.15.59:0/2868918185 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229804f330 con 0x7f2298066d90 2024-03-06T07:22:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.758+0000 7f228e7fc640 1 -- 172.21.15.59:0/2868918185 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f229004c070 con 0x7f2298066d90 2024-03-06T07:22:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.759+0000 7f228e7fc640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f227c06b4d0 0x7f227c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.759+0000 7f228e7fc640 1 -- 172.21.15.59:0/2868918185 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22900ac040 con 0x7f2298066d90 2024-03-06T07:22:25.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.759+0000 7f229d372640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f227c06b4d0 0x7f227c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.759+0000 7f229d372640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f227c06b4d0 0x7f227c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f22980d0df0 tx=0x7f2288009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.763+0000 7f228e7fc640 1 -- 172.21.15.59:0/2868918185 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22981f7b60 con 0x7f2298066d90 2024-03-06T07:22:25.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.944+0000 7f229fdd7640 1 -- 172.21.15.59:0/2868918185 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22980b1860 con 0x7f2298066d90 2024-03-06T07:22:25.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.945+0000 7f228e7fc640 1 -- 172.21.15.59:0/2868918185 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f22980d0300 con 0x7f2298066d90 2024-03-06T07:22:25.947 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:25.948 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:25.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 -- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f227c06b4d0 msgr2=0x7f227c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f227c06b4d0 0x7f227c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f22980d0df0 tx=0x7f2288009040 comp rx=0 tx=0).stop 2024-03-06T07:22:25.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 -- 172.21.15.59:0/2868918185 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2298066d90 msgr2=0x7f22981f7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2298066d90 0x7f22981f7620 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f229002bf70 tx=0x7f22900046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 -- 172.21.15.59:0/2868918185 shutdown_connections 2024-03-06T07:22:25.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f227c06b4d0 0x7f227c06d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22980ce670 0x7f22980b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22980b4f70 0x7f22981f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 --2- 172.21.15.59:0/2868918185 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2298066d90 0x7f22981f7620 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 -- 172.21.15.59:0/2868918185 >> 172.21.15.59:0/2868918185 conn(0x7f22980724d0 msgr2=0x7f22980b5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 -- 172.21.15.59:0/2868918185 shutdown_connections 2024-03-06T07:22:25.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:25.947+0000 7f229fdd7640 1 -- 172.21.15.59:0/2868918185 wait complete. 2024-03-06T07:22:25.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:25.961 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:26.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.063+0000 7fe52add7640 1 Processor -- start 2024-03-06T07:22:26.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.063+0000 7fe52add7640 1 -- start start 2024-03-06T07:22:26.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.064+0000 7fe52add7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe5240b1e40 0x7fe524168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.064+0000 7fe52add7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5240d2730 con 0x7fe5240d4860 2024-03-06T07:22:26.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.064+0000 7fe52add7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5241618b0 con 0x7fe5241614d0 2024-03-06T07:22:26.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.064+0000 7fe52add7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5240d4cf0 con 0x7fe5240b1e40 2024-03-06T07:22:26.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.064+0000 7fe523fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe5240d4860 0x7fe5240d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46996/0 (socket says 172.21.15.59:46996) 2024-03-06T07:22:26.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.064+0000 7fe523fff640 1 -- 172.21.15.59:0/2060854080 learned_addr learned my addr 172.21.15.59:0/2060854080 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:26.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.064+0000 7fe529374640 1 --2- 172.21.15.59:0/2060854080 >> v2:172.21.15.164:3300/0 conn(0x7fe5240b1e40 0x7fe524168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.065+0000 7fe5237fe640 1 -- 172.21.15.59:0/2060854080 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636034620 0 0) 0x7fe5240d2730 con 0x7fe5240d4860 2024-03-06T07:22:26.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.065+0000 7fe5237fe640 1 -- 172.21.15.59:0/2060854080 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe50c003520 con 0x7fe5240d4860 2024-03-06T07:22:26.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.065+0000 7fe5237fe640 1 -- 172.21.15.59:0/2060854080 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122568930 0 0) 0x7fe5241618b0 con 0x7fe5241614d0 2024-03-06T07:22:26.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.065+0000 7fe529374640 1 -- 172.21.15.59:0/2060854080 >> v1:172.21.15.164:6789/0 conn(0x7fe5241614d0 legacy=0x7fe5240d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.065+0000 7fe529374640 1 -- 172.21.15.59:0/2060854080 >> v1:172.21.15.59:6789/0 conn(0x7fe5240d4860 legacy=0x7fe5240d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.065+0000 7fe529374640 1 -- 172.21.15.59:0/2060854080 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5240cf400 con 0x7fe5240b1e40 2024-03-06T07:22:26.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.065+0000 7fe529374640 1 --2- 172.21.15.59:0/2060854080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5240b1e40 0x7fe524168b60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe5240cf5d0 tx=0x7fe51c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f730e259f7f94bb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.066+0000 7fe5237fe640 1 -- 172.21.15.59:0/2060854080 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c039070 con 0x7fe5240b1e40 2024-03-06T07:22:26.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.066+0000 7fe5237fe640 1 -- 172.21.15.59:0/2060854080 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c02cc50 con 0x7fe5240b1e40 2024-03-06T07:22:26.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.066+0000 7fe5237fe640 1 -- 172.21.15.59:0/2060854080 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c034440 con 0x7fe5240b1e40 2024-03-06T07:22:26.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.066+0000 7fe52add7640 1 -- 172.21.15.59:0/2060854080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5240b1e40 msgr2=0x7fe524168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.066+0000 7fe52add7640 1 --2- 172.21.15.59:0/2060854080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5240b1e40 0x7fe524168b60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe5240cf5d0 tx=0x7fe51c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.066+0000 7fe52add7640 1 -- 172.21.15.59:0/2060854080 shutdown_connections 2024-03-06T07:22:26.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.066+0000 7fe52add7640 1 --2- 172.21.15.59:0/2060854080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5240b1e40 0x7fe524168b60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.066+0000 7fe52add7640 1 -- 172.21.15.59:0/2060854080 >> 172.21.15.59:0/2060854080 conn(0x7fe5240d3d40 msgr2=0x7fe5240d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.066+0000 7fe52add7640 1 -- 172.21.15.59:0/2060854080 shutdown_connections 2024-03-06T07:22:26.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.066+0000 7fe52add7640 1 -- 172.21.15.59:0/2060854080 wait complete. 2024-03-06T07:22:26.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.067+0000 7fe52add7640 1 Processor -- start 2024-03-06T07:22:26.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.067+0000 7fe52add7640 1 -- start start 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.068+0000 7fe52add7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5240b1e40 0x7fe5241551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.068+0000 7fe52add7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5240d4860 0x7fe52415cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.068+0000 7fe52add7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5241614d0 0x7fe52415d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.068+0000 7fe52add7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240cfc60 con 0x7fe5241614d0 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.068+0000 7fe52add7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240cf980 con 0x7fe5240d4860 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.068+0000 7fe52add7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240b0910 con 0x7fe5240b1e40 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.068+0000 7fe529374640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5241614d0 0x7fe52415d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.068+0000 7fe529374640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5241614d0 0x7fe52415d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52730/0 (socket says 172.21.15.59:52730) 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.068+0000 7fe529374640 1 -- 172.21.15.59:0/3396711071 learned_addr learned my addr 172.21.15.59:0/3396711071 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.068+0000 7fe528b73640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5240b1e40 0x7fe5241551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.068+0000 7fe523fff640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5240d4860 0x7fe52415cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.069+0000 7fe529374640 1 -- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5240b1e40 msgr2=0x7fe5241551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.069+0000 7fe529374640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5240b1e40 0x7fe5241551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.069+0000 7fe529374640 1 -- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5240d4860 msgr2=0x7fe52415cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.069+0000 7fe529374640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5240d4860 0x7fe52415cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.069+0000 7fe529374640 1 -- 172.21.15.59:0/3396711071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5241f3d30 con 0x7fe5241614d0 2024-03-06T07:22:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.069+0000 7fe529374640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5241614d0 0x7fe52415d170 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe51c02c4b0 tx=0x7fe51c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.069+0000 7fe5217fa640 1 -- 172.21.15.59:0/3396711071 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c039050 con 0x7fe5241614d0 2024-03-06T07:22:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.069+0000 7fe5217fa640 1 -- 172.21.15.59:0/3396711071 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c045070 con 0x7fe5241614d0 2024-03-06T07:22:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.069+0000 7fe5217fa640 1 -- 172.21.15.59:0/3396711071 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c009450 con 0x7fe5241614d0 2024-03-06T07:22:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.069+0000 7fe52add7640 1 -- 172.21.15.59:0/3396711071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5241f4010 con 0x7fe5241614d0 2024-03-06T07:22:26.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.069+0000 7fe52add7640 1 -- 172.21.15.59:0/3396711071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5241f45d0 con 0x7fe5241614d0 2024-03-06T07:22:26.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.070+0000 7fe52add7640 1 -- 172.21.15.59:0/3396711071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5241f7010 con 0x7fe5241614d0 2024-03-06T07:22:26.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.071+0000 7fe5217fa640 1 -- 172.21.15.59:0/3396711071 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe51c03e610 con 0x7fe5241614d0 2024-03-06T07:22:26.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.071+0000 7fe5217fa640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe51006b4d0 0x7fe51006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.071+0000 7fe5217fa640 1 -- 172.21.15.59:0/3396711071 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe51c0acb80 con 0x7fe5241614d0 2024-03-06T07:22:26.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.071+0000 7fe528b73640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe51006b4d0 0x7fe51006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.072+0000 7fe528b73640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe51006b4d0 0x7fe51006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe514000f60 tx=0x7fe514009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.075+0000 7fe5217fa640 1 -- 172.21.15.59:0/3396711071 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe51c07b3c0 con 0x7fe5241614d0 2024-03-06T07:22:26.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.255+0000 7fe52add7640 1 -- 172.21.15.59:0/3396711071 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe524068ff0 con 0x7fe5241614d0 2024-03-06T07:22:26.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.255+0000 7fe5217fa640 1 -- 172.21.15.59:0/3396711071 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe51c07f460 con 0x7fe5241614d0 2024-03-06T07:22:26.257 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:26.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 -- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe51006b4d0 msgr2=0x7fe51006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe51006b4d0 0x7fe51006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe514000f60 tx=0x7fe514009040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 -- 172.21.15.59:0/3396711071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5241614d0 msgr2=0x7fe52415d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5241614d0 0x7fe52415d170 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe51c02c4b0 tx=0x7fe51c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 -- 172.21.15.59:0/3396711071 shutdown_connections 2024-03-06T07:22:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe51006b4d0 0x7fe51006d990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5241614d0 0x7fe52415d170 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5240d4860 0x7fe52415cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 --2- 172.21.15.59:0/3396711071 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5240b1e40 0x7fe5241551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 -- 172.21.15.59:0/3396711071 >> 172.21.15.59:0/3396711071 conn(0x7fe5240d3d40 msgr2=0x7fe5240cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 -- 172.21.15.59:0/3396711071 shutdown_connections 2024-03-06T07:22:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.257+0000 7fe52add7640 1 -- 172.21.15.59:0/3396711071 wait complete. 2024-03-06T07:22:26.270 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:26.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.372+0000 7ff4a7b2c640 1 Processor -- start 2024-03-06T07:22:26.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.373+0000 7ff4a7b2c640 1 -- start start 2024-03-06T07:22:26.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.373+0000 7ff4a7b2c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4a01614d0 0x7ff4a00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.373+0000 7ff4a7b2c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a00d1910 0x7ff4a00d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.373+0000 7ff4a7b2c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a00d2740 con 0x7ff4a00d22b0 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.373+0000 7ff4a7b2c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a01618b0 con 0x7ff4a01614d0 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.373+0000 7ff4a7b2c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a00cf400 con 0x7ff4a00d1910 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.374+0000 7ff4a58c8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4a01614d0 0x7ff4a00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.374+0000 7ff4a58c8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4a01614d0 0x7ff4a00d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52746/0 (socket says 172.21.15.59:52746) 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.374+0000 7ff4a58c8640 1 -- 172.21.15.59:0/319530983 learned_addr learned my addr 172.21.15.59:0/319530983 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.374+0000 7ff4a50c7640 1 --2- 172.21.15.59:0/319530983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a00d1910 0x7ff4a00d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.374+0000 7ff4a58c8640 1 -- 172.21.15.59:0/319530983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a00d1910 msgr2=0x7ff4a00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.374+0000 7ff4a58c8640 1 --2- 172.21.15.59:0/319530983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a00d1910 0x7ff4a00d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.374+0000 7ff4a58c8640 1 -- 172.21.15.59:0/319530983 >> v1:172.21.15.59:6789/0 conn(0x7ff4a00d22b0 legacy=0x7ff4a00d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.374+0000 7ff4a58c8640 1 -- 172.21.15.59:0/319530983 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a00cf6a0 con 0x7ff4a01614d0 2024-03-06T07:22:26.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a58c8640 1 --2- 172.21.15.59:0/319530983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a01614d0 0x7ff4a00d4860 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff498009870 tx=0x7ff49802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc89238bed166d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a48c6640 1 -- 172.21.15.59:0/319530983 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498034070 con 0x7ff4a01614d0 2024-03-06T07:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a48c6640 1 -- 172.21.15.59:0/319530983 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff498030840 con 0x7ff4a01614d0 2024-03-06T07:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a48c6640 1 -- 172.21.15.59:0/319530983 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498030c10 con 0x7ff4a01614d0 2024-03-06T07:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/319530983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a01614d0 msgr2=0x7ff4a00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a7b2c640 1 --2- 172.21.15.59:0/319530983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a01614d0 0x7ff4a00d4860 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff498009870 tx=0x7ff49802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/319530983 shutdown_connections 2024-03-06T07:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a7b2c640 1 --2- 172.21.15.59:0/319530983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a00d1910 0x7ff4a00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a7b2c640 1 --2- 172.21.15.59:0/319530983 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a01614d0 0x7ff4a00d4860 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/319530983 >> 172.21.15.59:0/319530983 conn(0x7ff4a00d3d40 msgr2=0x7ff4a00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/319530983 shutdown_connections 2024-03-06T07:22:26.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.375+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/319530983 wait complete. 2024-03-06T07:22:26.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.376+0000 7ff4a7b2c640 1 Processor -- start 2024-03-06T07:22:26.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a7b2c640 1 -- start start 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a7b2c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4a00d1910 0x7ff4a0151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a7b2c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00d22b0 0x7ff4a00c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a7b2c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a01614d0 0x7ff4a00c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a7b2c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a00b0910 con 0x7ff4a00d22b0 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a7b2c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a00cfc60 con 0x7ff4a00d1910 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a7b2c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a00b1e40 con 0x7ff4a01614d0 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a50c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00d22b0 0x7ff4a00c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a50c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00d22b0 0x7ff4a00c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52760/0 (socket says 172.21.15.59:52760) 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a50c7640 1 -- 172.21.15.59:0/1776782350 learned_addr learned my addr 172.21.15.59:0/1776782350 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a58c8640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4a00d1910 0x7ff4a0151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.377+0000 7ff4a60c9640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a01614d0 0x7ff4a00c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.378+0000 7ff4a50c7640 1 -- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a01614d0 msgr2=0x7ff4a00c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.378+0000 7ff4a50c7640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a01614d0 0x7ff4a00c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.378+0000 7ff4a50c7640 1 -- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4a00d1910 msgr2=0x7ff4a0151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.378+0000 7ff4a50c7640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4a00d1910 0x7ff4a0151220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.378+0000 7ff4a50c7640 1 -- 172.21.15.59:0/1776782350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a00ca220 con 0x7ff4a00d22b0 2024-03-06T07:22:26.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.378+0000 7ff4a50c7640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00d22b0 0x7ff4a00c7870 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff49400e9b0 tx=0x7ff49400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.378+0000 7ff49e7fc640 1 -- 172.21.15.59:0/1776782350 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff494017070 con 0x7ff4a00d22b0 2024-03-06T07:22:26.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.378+0000 7ff49e7fc640 1 -- 172.21.15.59:0/1776782350 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49400c9a0 con 0x7ff4a00d22b0 2024-03-06T07:22:26.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.378+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/1776782350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a00ca4e0 con 0x7ff4a00d22b0 2024-03-06T07:22:26.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.378+0000 7ff49e7fc640 1 -- 172.21.15.59:0/1776782350 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49400cd90 con 0x7ff4a00d22b0 2024-03-06T07:22:26.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.378+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/1776782350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a00caaa0 con 0x7ff4a00d22b0 2024-03-06T07:22:26.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.379+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/1776782350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a0068ff0 con 0x7ff4a00d22b0 2024-03-06T07:22:26.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.379+0000 7ff49e7fc640 1 -- 172.21.15.59:0/1776782350 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff49401a070 con 0x7ff4a00d22b0 2024-03-06T07:22:26.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.380+0000 7ff49e7fc640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48c06b4d0 0x7ff48c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.380+0000 7ff49e7fc640 1 -- 172.21.15.59:0/1776782350 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff49408a190 con 0x7ff4a00d22b0 2024-03-06T07:22:26.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.380+0000 7ff4a58c8640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48c06b4d0 0x7ff48c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.380+0000 7ff4a58c8640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48c06b4d0 0x7ff48c06d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff498036000 tx=0x7ff49803b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.384+0000 7ff49e7fc640 1 -- 172.21.15.59:0/1776782350 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4940589c0 con 0x7ff4a00d22b0 2024-03-06T07:22:26.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.568+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/1776782350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff4a00d0e70 con 0x7ff4a00d22b0 2024-03-06T07:22:26.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.568+0000 7ff49e7fc640 1 -- 172.21.15.59:0/1776782350 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff49405ca60 con 0x7ff4a00d22b0 2024-03-06T07:22:26.570 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48c06b4d0 msgr2=0x7ff48c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48c06b4d0 0x7ff48c06d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff498036000 tx=0x7ff49803b040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/1776782350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00d22b0 msgr2=0x7ff4a00c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00d22b0 0x7ff4a00c7870 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff49400e9b0 tx=0x7ff49400ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/1776782350 shutdown_connections 2024-03-06T07:22:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48c06b4d0 0x7ff48c06d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a01614d0 0x7ff4a00c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00d22b0 0x7ff4a00c7870 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 --2- 172.21.15.59:0/1776782350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4a00d1910 0x7ff4a0151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/1776782350 >> 172.21.15.59:0/1776782350 conn(0x7ff4a00d3d40 msgr2=0x7ff4a00d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/1776782350 shutdown_connections 2024-03-06T07:22:26.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.571+0000 7ff4a7b2c640 1 -- 172.21.15.59:0/1776782350 wait complete. 2024-03-06T07:22:26.583 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:26.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.686+0000 7f77a9377640 1 Processor -- start 2024-03-06T07:22:26.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.686+0000 7f77a9377640 1 -- start start 2024-03-06T07:22:26.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a9377640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f77a41614d0 0x7f77a40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a9377640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f77a40d1910 0x7f77a40d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a9377640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a40d22e0 0x7f77a40d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a9377640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a40d2750 con 0x7f77a41614d0 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a9377640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a41618b0 con 0x7f77a40d1910 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a9377640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40cf400 con 0x7f77a40d22e0 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a2ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f77a41614d0 0x7f77a40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a2ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f77a41614d0 0x7f77a40d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52770/0 (socket says 172.21.15.59:52770) 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a2ffd640 1 -- 172.21.15.59:0/2198986830 learned_addr learned my addr 172.21.15.59:0/2198986830 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a37fe640 1 --2- 172.21.15.59:0/2198986830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a40d22e0 0x7f77a40d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f779bfff640 1 --2- 172.21.15.59:0/2198986830 >> v2:172.21.15.164:3300/0 conn(0x7f77a40d1910 0x7f77a40d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a2ffd640 1 -- 172.21.15.59:0/2198986830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a40d22e0 msgr2=0x7f77a40d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a2ffd640 1 --2- 172.21.15.59:0/2198986830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a40d22e0 0x7f77a40d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a2ffd640 1 -- 172.21.15.59:0/2198986830 >> v2:172.21.15.164:3300/0 conn(0x7f77a40d1910 msgr2=0x7f77a40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a2ffd640 1 --2- 172.21.15.59:0/2198986830 >> v2:172.21.15.164:3300/0 conn(0x7f77a40d1910 0x7f77a40d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.687+0000 7f77a2ffd640 1 -- 172.21.15.59:0/2198986830 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a40cf6a0 con 0x7f77a41614d0 2024-03-06T07:22:26.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.688+0000 7f77a2ffd640 1 --2- 172.21.15.59:0/2198986830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a41614d0 0x7f77a40d4860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f77a40ce550 tx=0x7f779402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a47e3b962501cbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.688+0000 7f77a27fc640 1 -- 172.21.15.59:0/2198986830 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794035020 con 0x7f77a41614d0 2024-03-06T07:22:26.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.688+0000 7f77a27fc640 1 -- 172.21.15.59:0/2198986830 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7794031870 con 0x7f77a41614d0 2024-03-06T07:22:26.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.688+0000 7f77a27fc640 1 -- 172.21.15.59:0/2198986830 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794031c40 con 0x7f77a41614d0 2024-03-06T07:22:26.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.689+0000 7f77a9377640 1 -- 172.21.15.59:0/2198986830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a41614d0 msgr2=0x7f77a40d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.689+0000 7f77a9377640 1 --2- 172.21.15.59:0/2198986830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a41614d0 0x7f77a40d4860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f77a40ce550 tx=0x7f779402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:26.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.689+0000 7f77a9377640 1 -- 172.21.15.59:0/2198986830 shutdown_connections 2024-03-06T07:22:26.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.689+0000 7f77a9377640 1 --2- 172.21.15.59:0/2198986830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a40d22e0 0x7f77a40d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.689+0000 7f77a9377640 1 --2- 172.21.15.59:0/2198986830 >> v2:172.21.15.164:3300/0 conn(0x7f77a40d1910 0x7f77a40d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.689+0000 7f77a9377640 1 --2- 172.21.15.59:0/2198986830 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a41614d0 0x7f77a40d4860 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.689+0000 7f77a9377640 1 -- 172.21.15.59:0/2198986830 >> 172.21.15.59:0/2198986830 conn(0x7f77a40d3d40 msgr2=0x7f77a40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.689+0000 7f77a9377640 1 -- 172.21.15.59:0/2198986830 shutdown_connections 2024-03-06T07:22:26.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.689+0000 7f77a9377640 1 -- 172.21.15.59:0/2198986830 wait complete. 2024-03-06T07:22:26.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.690+0000 7f77a9377640 1 Processor -- start 2024-03-06T07:22:26.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.690+0000 7f77a9377640 1 -- start start 2024-03-06T07:22:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f77a9377640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77a40d1910 0x7f77a414d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f77a9377640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40d22e0 0x7f77a414d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f77a9377640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a41614d0 0x7f77a40cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f77a9377640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40b0910 con 0x7f77a40d22e0 2024-03-06T07:22:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f77a9377640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40cfc60 con 0x7f77a40d1910 2024-03-06T07:22:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f77a9377640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40d0be0 con 0x7f77a41614d0 2024-03-06T07:22:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40d22e0 0x7f77a414d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40d22e0 0x7f77a414d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52776/0 (socket says 172.21.15.59:52776) 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779bfff640 1 -- 172.21.15.59:0/2448007195 learned_addr learned my addr 172.21.15.59:0/2448007195 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f77a2ffd640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77a40d1910 0x7f77a414d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f77a37fe640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a41614d0 0x7f77a40cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779bfff640 1 -- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a41614d0 msgr2=0x7f77a40cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779bfff640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a41614d0 0x7f77a40cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779bfff640 1 -- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77a40d1910 msgr2=0x7f77a414d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779bfff640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77a40d1910 0x7f77a414d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779bfff640 1 -- 172.21.15.59:0/2448007195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a40c8b60 con 0x7f77a40d22e0 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779bfff640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40d22e0 0x7f77a414d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f778c00e9b0 tx=0x7f778c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779b7fe640 1 -- 172.21.15.59:0/2448007195 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c017070 con 0x7f77a40d22e0 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779b7fe640 1 -- 172.21.15.59:0/2448007195 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778c00c9a0 con 0x7f77a40d22e0 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f77a9377640 1 -- 172.21.15.59:0/2448007195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a40c8e40 con 0x7f77a40d22e0 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.691+0000 7f779b7fe640 1 -- 172.21.15.59:0/2448007195 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c00cd90 con 0x7f77a40d22e0 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.692+0000 7f77a9377640 1 -- 172.21.15.59:0/2448007195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a40c9400 con 0x7f77a40d22e0 2024-03-06T07:22:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.693+0000 7f77a9377640 1 -- 172.21.15.59:0/2448007195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a4068ff0 con 0x7f77a40d22e0 2024-03-06T07:22:26.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.693+0000 7f779b7fe640 1 -- 172.21.15.59:0/2448007195 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f778c01a070 con 0x7f77a40d22e0 2024-03-06T07:22:26.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.693+0000 7f779b7fe640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f778406b4d0 0x7f778406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.693+0000 7f779b7fe640 1 -- 172.21.15.59:0/2448007195 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f778c08a490 con 0x7f77a40d22e0 2024-03-06T07:22:26.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.693+0000 7f77a2ffd640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f778406b4d0 0x7f778406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.694+0000 7f77a2ffd640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f778406b4d0 0x7f778406d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7794036000 tx=0x7f779403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.698+0000 7f779b7fe640 1 -- 172.21.15.59:0/2448007195 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f778c058d50 con 0x7f77a40d22e0 2024-03-06T07:22:26.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.876+0000 7f77a9377640 1 -- 172.21.15.59:0/2448007195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f77a414ded0 con 0x7f77a40d22e0 2024-03-06T07:22:26.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.877+0000 7f779b7fe640 1 -- 172.21.15.59:0/2448007195 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f778c05cdf0 con 0x7f77a40d22e0 2024-03-06T07:22:26.879 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:26.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.878+0000 7f77a9377640 1 -- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f778406b4d0 msgr2=0x7f778406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.878+0000 7f77a9377640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f778406b4d0 0x7f778406d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7794036000 tx=0x7f779403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.878+0000 7f77a9377640 1 -- 172.21.15.59:0/2448007195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40d22e0 msgr2=0x7f77a414d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.878+0000 7f77a9377640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40d22e0 0x7f77a414d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f778c00e9b0 tx=0x7f778c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:26.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.878+0000 7f77a9377640 1 -- 172.21.15.59:0/2448007195 shutdown_connections 2024-03-06T07:22:26.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.878+0000 7f77a9377640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f778406b4d0 0x7f778406d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.878+0000 7f77a9377640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a41614d0 0x7f77a40cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.879+0000 7f77a9377640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40d22e0 0x7f77a414d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.879+0000 7f77a9377640 1 --2- 172.21.15.59:0/2448007195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77a40d1910 0x7f77a414d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.879+0000 7f77a9377640 1 -- 172.21.15.59:0/2448007195 >> 172.21.15.59:0/2448007195 conn(0x7f77a40d3d40 msgr2=0x7f77a40ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.879+0000 7f77a9377640 1 -- 172.21.15.59:0/2448007195 shutdown_connections 2024-03-06T07:22:26.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.879+0000 7f77a9377640 1 -- 172.21.15.59:0/2448007195 wait complete. 2024-03-06T07:22:26.891 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:26.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.992+0000 7fecfb51e640 1 Processor -- start 2024-03-06T07:22:26.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.992+0000 7fecfb51e640 1 -- start start 2024-03-06T07:22:26.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.992+0000 7fecfb51e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf4162970 0x7fecf40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.992+0000 7fecfb51e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fecf4161cf0 0x7fecf4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.992+0000 7fecfb51e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf40d22c0 con 0x7fecf40b4460 2024-03-06T07:22:26.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.992+0000 7fecfb51e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf4161b00 con 0x7fecf4161cf0 2024-03-06T07:22:26.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.992+0000 7fecfb51e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40cecc0 con 0x7fecf4162970 2024-03-06T07:22:26.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.993+0000 7fecf9abb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fecf4161cf0 0x7fecf4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.993+0000 7fecf9abb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fecf4161cf0 0x7fecf4162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52792/0 (socket says 172.21.15.59:52792) 2024-03-06T07:22:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.993+0000 7fecf9abb640 1 -- 172.21.15.59:0/1761136212 learned_addr learned my addr 172.21.15.59:0/1761136212 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.993+0000 7fecf8ab9640 1 --2- 172.21.15.59:0/1761136212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf4162970 0x7fecf40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.993+0000 7fecf9abb640 1 -- 172.21.15.59:0/1761136212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf4162970 msgr2=0x7fecf40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.993+0000 7fecf9abb640 1 --2- 172.21.15.59:0/1761136212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf4162970 0x7fecf40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.993+0000 7fecf9abb640 1 -- 172.21.15.59:0/1761136212 >> v1:172.21.15.59:6789/0 conn(0x7fecf40b4460 legacy=0x7fecf40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.993+0000 7fecf9abb640 1 -- 172.21.15.59:0/1761136212 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40d25a0 con 0x7fecf4161cf0 2024-03-06T07:22:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecf9abb640 1 --2- 172.21.15.59:0/1761136212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf4161cf0 0x7fecf4162180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009e50 tx=0x7fecf002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3f452ceab26169 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecebfff640 1 -- 172.21.15.59:0/1761136212 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf002cb00 con 0x7fecf4161cf0 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecebfff640 1 -- 172.21.15.59:0/1761136212 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf002cca0 con 0x7fecf4161cf0 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecebfff640 1 -- 172.21.15.59:0/1761136212 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0034610 con 0x7fecf4161cf0 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecfb51e640 1 -- 172.21.15.59:0/1761136212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf4161cf0 msgr2=0x7fecf4162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecfb51e640 1 --2- 172.21.15.59:0/1761136212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf4161cf0 0x7fecf4162180 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009e50 tx=0x7fecf002bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecfb51e640 1 -- 172.21.15.59:0/1761136212 shutdown_connections 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecfb51e640 1 --2- 172.21.15.59:0/1761136212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf4161cf0 0x7fecf4162180 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecfb51e640 1 --2- 172.21.15.59:0/1761136212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf4162970 0x7fecf40d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecfb51e640 1 -- 172.21.15.59:0/1761136212 >> 172.21.15.59:0/1761136212 conn(0x7fecf40d3ce0 msgr2=0x7fecf40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecfb51e640 1 -- 172.21.15.59:0/1761136212 shutdown_connections 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.994+0000 7fecfb51e640 1 -- 172.21.15.59:0/1761136212 wait complete. 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecfb51e640 1 Processor -- start 2024-03-06T07:22:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecfb51e640 1 -- start start 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecfb51e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf40b4460 0x7fecf4155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecfb51e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf4161cf0 0x7fecf40cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecfb51e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf4162970 0x7fecf40b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecfb51e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40ce950 con 0x7fecf4161cf0 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecfb51e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40d2b60 con 0x7fecf4162970 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecfb51e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40b4840 con 0x7fecf40b4460 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecf8ab9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf4161cf0 0x7fecf40cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecf8ab9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf4161cf0 0x7fecf40cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52804/0 (socket says 172.21.15.59:52804) 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecf8ab9640 1 -- 172.21.15.59:0/2929099862 learned_addr learned my addr 172.21.15.59:0/2929099862 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecf92ba640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf40b4460 0x7fecf4155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.996+0000 7fecf9abb640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf4162970 0x7fecf40b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.997+0000 7fecf8ab9640 1 -- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf40b4460 msgr2=0x7fecf4155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.997+0000 7fecf8ab9640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf40b4460 0x7fecf4155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.997+0000 7fecf8ab9640 1 -- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf4162970 msgr2=0x7fecf40b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.997+0000 7fecf8ab9640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf4162970 0x7fecf40b1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.997+0000 7fecf8ab9640 1 -- 172.21.15.59:0/2929099862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40cb6e0 con 0x7fecf4161cf0 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.997+0000 7fecf8ab9640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf4161cf0 0x7fecf40cadb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fece400e980 tx=0x7fece400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.997+0000 7fece9ffb640 1 -- 172.21.15.59:0/2929099862 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4017070 con 0x7fecf4161cf0 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.997+0000 7fece9ffb640 1 -- 172.21.15.59:0/2929099862 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece400ca00 con 0x7fecf4161cf0 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.997+0000 7fece9ffb640 1 -- 172.21.15.59:0/2929099862 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece400cdd0 con 0x7fecf4161cf0 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.997+0000 7fecfb51e640 1 -- 172.21.15.59:0/2929099862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf40cb9c0 con 0x7fecf4161cf0 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.997+0000 7fecfb51e640 1 -- 172.21.15.59:0/2929099862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf40cbf80 con 0x7fecf4161cf0 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.998+0000 7fecfb51e640 1 -- 172.21.15.59:0/2929099862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf404f330 con 0x7fecf4161cf0 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.999+0000 7fece9ffb640 1 -- 172.21.15.59:0/2929099862 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fece401a070 con 0x7fecf4161cf0 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.999+0000 7fece9ffb640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecd806b4d0 0x7fecd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.999+0000 7fece9ffb640 1 -- 172.21.15.59:0/2929099862 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fece408a540 con 0x7fecf4161cf0 2024-03-06T07:22:27.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:26.999+0000 7fecf92ba640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecd806b4d0 0x7fecd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.000+0000 7fecf92ba640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecd806b4d0 0x7fecd806d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fecec0016e0 tx=0x7fecec009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.003+0000 7fece9ffb640 1 -- 172.21.15.59:0/2929099862 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fece4058d70 con 0x7fecf4161cf0 2024-03-06T07:22:27.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.198+0000 7fecfb51e640 1 -- 172.21.15.59:0/2929099862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fecf406b180 con 0x7fecf4161cf0 2024-03-06T07:22:27.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.199+0000 7fece9ffb640 1 -- 172.21.15.59:0/2929099862 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fece405f0c0 con 0x7fecf4161cf0 2024-03-06T07:22:27.200 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:22:27.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.200+0000 7fecfb51e640 1 -- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecd806b4d0 msgr2=0x7fecd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.200+0000 7fecfb51e640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecd806b4d0 0x7fecd806d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fecec0016e0 tx=0x7fecec009040 comp rx=0 tx=0).stop 2024-03-06T07:22:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.200+0000 7fecfb51e640 1 -- 172.21.15.59:0/2929099862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf4161cf0 msgr2=0x7fecf40cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.200+0000 7fecfb51e640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf4161cf0 0x7fecf40cadb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fece400e980 tx=0x7fece400ee40 comp rx=0 tx=0).stop 2024-03-06T07:22:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.200+0000 7fecfb51e640 1 -- 172.21.15.59:0/2929099862 shutdown_connections 2024-03-06T07:22:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.200+0000 7fecfb51e640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecd806b4d0 0x7fecd806d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.200+0000 7fecfb51e640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf4162970 0x7fecf40b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.200+0000 7fecfb51e640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf4161cf0 0x7fecf40cadb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.200+0000 7fecfb51e640 1 --2- 172.21.15.59:0/2929099862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf40b4460 0x7fecf4155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.200+0000 7fecfb51e640 1 -- 172.21.15.59:0/2929099862 >> 172.21.15.59:0/2929099862 conn(0x7fecf40d3ce0 msgr2=0x7fecf4161250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.201+0000 7fecfb51e640 1 -- 172.21.15.59:0/2929099862 shutdown_connections 2024-03-06T07:22:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.201+0000 7fecfb51e640 1 -- 172.21.15.59:0/2929099862 wait complete. 2024-03-06T07:22:27.213 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:22:27.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:27.213 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:27.240 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:27.240 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:22:27.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:27.240 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:22:27.307 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:22:27.307 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:27.322 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:22:27.322 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:27.379 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:27.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.480+0000 7ff47772b640 1 Processor -- start 2024-03-06T07:22:27.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.480+0000 7ff47772b640 1 -- start start 2024-03-06T07:22:27.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.481+0000 7ff47772b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4701424f0 0x7ff4700cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.481+0000 7ff47772b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff47006da40 0x7ff47015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.481+0000 7ff47772b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4700cc980 con 0x7ff4700cd7a0 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.481+0000 7ff47772b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4700cd220 con 0x7ff4701424f0 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.481+0000 7ff47772b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700ccc60 con 0x7ff47006da40 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.481+0000 7ff4754c7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4701424f0 0x7ff4700cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.481+0000 7ff4754c7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4701424f0 0x7ff4700cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52814/0 (socket says 172.21.15.59:52814) 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.481+0000 7ff4754c7640 1 -- 172.21.15.59:0/1182036725 learned_addr learned my addr 172.21.15.59:0/1182036725 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.481+0000 7ff475cc8640 1 --2- 172.21.15.59:0/1182036725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff47006da40 0x7ff47015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.482+0000 7ff4754c7640 1 -- 172.21.15.59:0/1182036725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff47006da40 msgr2=0x7ff47015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.482+0000 7ff4754c7640 1 --2- 172.21.15.59:0/1182036725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff47006da40 0x7ff47015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.482+0000 7ff4754c7640 1 -- 172.21.15.59:0/1182036725 >> v1:172.21.15.164:6789/0 conn(0x7ff4700cd7a0 legacy=0x7ff47006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.482+0000 7ff4754c7640 1 -- 172.21.15.59:0/1182036725 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4700ccf40 con 0x7ff4701424f0 2024-03-06T07:22:27.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.482+0000 7ff4754c7640 1 --2- 172.21.15.59:0/1182036725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4701424f0 0x7ff4700cc2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009870 tx=0x7ff46c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d622d155b732f100 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.483+0000 7ff467fff640 1 -- 172.21.15.59:0/1182036725 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c034070 con 0x7ff4701424f0 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.483+0000 7ff467fff640 1 -- 172.21.15.59:0/1182036725 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46c030840 con 0x7ff4701424f0 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.483+0000 7ff467fff640 1 -- 172.21.15.59:0/1182036725 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c030c10 con 0x7ff4701424f0 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.483+0000 7ff47772b640 1 -- 172.21.15.59:0/1182036725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4701424f0 msgr2=0x7ff4700cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.483+0000 7ff47772b640 1 --2- 172.21.15.59:0/1182036725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4701424f0 0x7ff4700cc2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009870 tx=0x7ff46c02b980 comp rx=0 tx=0).stop 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.483+0000 7ff47772b640 1 -- 172.21.15.59:0/1182036725 shutdown_connections 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.483+0000 7ff47772b640 1 --2- 172.21.15.59:0/1182036725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff47006da40 0x7ff47015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.483+0000 7ff47772b640 1 --2- 172.21.15.59:0/1182036725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4701424f0 0x7ff4700cc2f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.483+0000 7ff47772b640 1 -- 172.21.15.59:0/1182036725 >> 172.21.15.59:0/1182036725 conn(0x7ff4700cf8a0 msgr2=0x7ff4700cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.483+0000 7ff47772b640 1 -- 172.21.15.59:0/1182036725 shutdown_connections 2024-03-06T07:22:27.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.483+0000 7ff47772b640 1 -- 172.21.15.59:0/1182036725 wait complete. 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.484+0000 7ff47772b640 1 Processor -- start 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.484+0000 7ff47772b640 1 -- start start 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff47772b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47006da40 0x7ff4700a9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff47772b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4700cd7a0 0x7ff4700b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff47772b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4701424f0 0x7ff4701f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff47772b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff470071ce0 con 0x7ff47006da40 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff47772b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff470071b60 con 0x7ff4700cd7a0 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff47772b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff470071e60 con 0x7ff4701424f0 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff4754c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47006da40 0x7ff4700a9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff4754c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47006da40 0x7ff4700a9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52816/0 (socket says 172.21.15.59:52816) 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff4754c7640 1 -- 172.21.15.59:0/2423066676 learned_addr learned my addr 172.21.15.59:0/2423066676 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff474cc6640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4700cd7a0 0x7ff4700b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff475cc8640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4701424f0 0x7ff4701f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff4754c7640 1 -- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4701424f0 msgr2=0x7ff4701f4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff4754c7640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4701424f0 0x7ff4701f4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff4754c7640 1 -- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4700cd7a0 msgr2=0x7ff4700b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff4754c7640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4700cd7a0 0x7ff4700b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.485+0000 7ff4754c7640 1 -- 172.21.15.59:0/2423066676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff470156720 con 0x7ff47006da40 2024-03-06T07:22:27.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.486+0000 7ff4754c7640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47006da40 0x7ff4700a9990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff46c000c00 tx=0x7ff46c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.486+0000 7ff465ffb640 1 -- 172.21.15.59:0/2423066676 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c034040 con 0x7ff47006da40 2024-03-06T07:22:27.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.486+0000 7ff465ffb640 1 -- 172.21.15.59:0/2423066676 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46c039d90 con 0x7ff47006da40 2024-03-06T07:22:27.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.486+0000 7ff465ffb640 1 -- 172.21.15.59:0/2423066676 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c043630 con 0x7ff47006da40 2024-03-06T07:22:27.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.486+0000 7ff47772b640 1 -- 172.21.15.59:0/2423066676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff470156a00 con 0x7ff47006da40 2024-03-06T07:22:27.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.486+0000 7ff47772b640 1 -- 172.21.15.59:0/2423066676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff470156fc0 con 0x7ff47006da40 2024-03-06T07:22:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.487+0000 7ff47772b640 1 -- 172.21.15.59:0/2423066676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47004f330 con 0x7ff47006da40 2024-03-06T07:22:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.487+0000 7ff465ffb640 1 -- 172.21.15.59:0/2423066676 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff46c04a060 con 0x7ff47006da40 2024-03-06T07:22:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.488+0000 7ff465ffb640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 0x7ff45406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.488+0000 7ff465ffb640 1 -- 172.21.15.59:0/2423066676 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff46c0abfb0 con 0x7ff47006da40 2024-03-06T07:22:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.488+0000 7ff474cc6640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 0x7ff45406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.489+0000 7ff474cc6640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 0x7ff45406d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff4701558e0 tx=0x7ff460009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.492+0000 7ff465ffb640 1 -- 172.21.15.59:0/2423066676 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff46c03e070 con 0x7ff47006da40 2024-03-06T07:22:27.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.674+0000 7ff47772b640 1 -- 172.21.15.59:0/2423066676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff47006cb80 con 0x7ff47006da40 2024-03-06T07:22:27.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.675+0000 7ff465ffb640 1 -- 172.21.15.59:0/2423066676 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7ff46c07a870 con 0x7ff47006da40 2024-03-06T07:22:27.677 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:27.677 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 -- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 msgr2=0x7ff45406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 0x7ff45406d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff4701558e0 tx=0x7ff460009040 comp rx=0 tx=0).stop 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 -- 172.21.15.59:0/2423066676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47006da40 msgr2=0x7ff4700a9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47006da40 0x7ff4700a9990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff46c000c00 tx=0x7ff46c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 -- 172.21.15.59:0/2423066676 shutdown_connections 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 0x7ff45406d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4701424f0 0x7ff4701f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4700cd7a0 0x7ff4700b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 --2- 172.21.15.59:0/2423066676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47006da40 0x7ff4700a9990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 -- 172.21.15.59:0/2423066676 >> 172.21.15.59:0/2423066676 conn(0x7ff4700cf8a0 msgr2=0x7ff47015cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 -- 172.21.15.59:0/2423066676 shutdown_connections 2024-03-06T07:22:27.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.677+0000 7ff47772b640 1 -- 172.21.15.59:0/2423066676 wait complete. 2024-03-06T07:22:27.690 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:27.690 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:27.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.793+0000 7fce70ac9640 1 Processor -- start 2024-03-06T07:22:27.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.793+0000 7fce70ac9640 1 -- start start 2024-03-06T07:22:27.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.793+0000 7fce70ac9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fce6c1614d0 0x7fce6c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.793+0000 7fce70ac9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fce6c0d1910 0x7fce6c0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.793+0000 7fce70ac9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6c1618b0 con 0x7fce6c0d22e0 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.794+0000 7fce70ac9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6c0cf400 con 0x7fce6c1614d0 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.794+0000 7fce70ac9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6c0d2710 con 0x7fce6c0d1910 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.794+0000 7fce6a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fce6c1614d0 0x7fce6c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.794+0000 7fce6a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fce6c1614d0 0x7fce6c0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52818/0 (socket says 172.21.15.59:52818) 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.794+0000 7fce6a59c640 1 -- 172.21.15.59:0/3622843751 learned_addr learned my addr 172.21.15.59:0/3622843751 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.794+0000 7fce63fff640 1 --2- 172.21.15.59:0/3622843751 >> v2:172.21.15.164:3300/0 conn(0x7fce6c0d1910 0x7fce6c0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.794+0000 7fce6a59c640 1 -- 172.21.15.59:0/3622843751 >> v2:172.21.15.164:3300/0 conn(0x7fce6c0d1910 msgr2=0x7fce6c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.794+0000 7fce6a59c640 1 --2- 172.21.15.59:0/3622843751 >> v2:172.21.15.164:3300/0 conn(0x7fce6c0d1910 0x7fce6c0d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.794+0000 7fce6a59c640 1 -- 172.21.15.59:0/3622843751 >> v1:172.21.15.59:6789/0 conn(0x7fce6c0d22e0 legacy=0x7fce6c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.794+0000 7fce6a59c640 1 -- 172.21.15.59:0/3622843751 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c0cf6a0 con 0x7fce6c1614d0 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce6a59c640 1 --2- 172.21.15.59:0/3622843751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c1614d0 0x7fce6c0d4860 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0ce550 tx=0x7fce5c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2e98d791b01da5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce69d9b640 1 -- 172.21.15.59:0/3622843751 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c034070 con 0x7fce6c1614d0 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce69d9b640 1 -- 172.21.15.59:0/3622843751 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c030860 con 0x7fce6c1614d0 2024-03-06T07:22:27.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce69d9b640 1 -- 172.21.15.59:0/3622843751 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c030c30 con 0x7fce6c1614d0 2024-03-06T07:22:27.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce70ac9640 1 -- 172.21.15.59:0/3622843751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c1614d0 msgr2=0x7fce6c0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce70ac9640 1 --2- 172.21.15.59:0/3622843751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c1614d0 0x7fce6c0d4860 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0ce550 tx=0x7fce5c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce70ac9640 1 -- 172.21.15.59:0/3622843751 shutdown_connections 2024-03-06T07:22:27.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce70ac9640 1 --2- 172.21.15.59:0/3622843751 >> v2:172.21.15.164:3300/0 conn(0x7fce6c0d1910 0x7fce6c0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce70ac9640 1 --2- 172.21.15.59:0/3622843751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c1614d0 0x7fce6c0d4860 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce70ac9640 1 -- 172.21.15.59:0/3622843751 >> 172.21.15.59:0/3622843751 conn(0x7fce6c0d3d40 msgr2=0x7fce6c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce70ac9640 1 -- 172.21.15.59:0/3622843751 shutdown_connections 2024-03-06T07:22:27.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.795+0000 7fce70ac9640 1 -- 172.21.15.59:0/3622843751 wait complete. 2024-03-06T07:22:27.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.797+0000 7fce70ac9640 1 Processor -- start 2024-03-06T07:22:27.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.797+0000 7fce70ac9640 1 -- start start 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.797+0000 7fce70ac9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce6c0d1910 0x7fce6c155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.797+0000 7fce70ac9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0d22e0 0x7fce6c1f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.797+0000 7fce70ac9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce6c1614d0 0x7fce6c0c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.797+0000 7fce70ac9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0b0910 con 0x7fce6c0d22e0 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.797+0000 7fce70ac9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0cfc60 con 0x7fce6c0d1910 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.797+0000 7fce70ac9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0b1e40 con 0x7fce6c1614d0 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce63fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0d22e0 0x7fce6c1f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce63fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0d22e0 0x7fce6c1f70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52826/0 (socket says 172.21.15.59:52826) 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce63fff640 1 -- 172.21.15.59:0/3875568287 learned_addr learned my addr 172.21.15.59:0/3875568287 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce6a59c640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce6c0d1910 0x7fce6c155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce6ad9d640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce6c1614d0 0x7fce6c0c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce63fff640 1 -- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce6c1614d0 msgr2=0x7fce6c0c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce63fff640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce6c1614d0 0x7fce6c0c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce63fff640 1 -- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce6c0d1910 msgr2=0x7fce6c155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce63fff640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce6c0d1910 0x7fce6c155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce63fff640 1 -- 172.21.15.59:0/3875568287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c1f79c0 con 0x7fce6c0d22e0 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce63fff640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0d22e0 0x7fce6c1f70c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fce5400e9b0 tx=0x7fce5400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce62ffd640 1 -- 172.21.15.59:0/3875568287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce54017070 con 0x7fce6c0d22e0 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce62ffd640 1 -- 172.21.15.59:0/3875568287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5400c9a0 con 0x7fce6c0d22e0 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce70ac9640 1 -- 172.21.15.59:0/3875568287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6c1f7ca0 con 0x7fce6c0d22e0 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce62ffd640 1 -- 172.21.15.59:0/3875568287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5400cd90 con 0x7fce6c0d22e0 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.798+0000 7fce70ac9640 1 -- 172.21.15.59:0/3875568287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6c1f7f80 con 0x7fce6c0d22e0 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.799+0000 7fce70ac9640 1 -- 172.21.15.59:0/3875568287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6c068ff0 con 0x7fce6c0d22e0 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.800+0000 7fce62ffd640 1 -- 172.21.15.59:0/3875568287 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce5401a070 con 0x7fce6c0d22e0 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.800+0000 7fce62ffd640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce4c06b500 0x7fce4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.800+0000 7fce62ffd640 1 -- 172.21.15.59:0/3875568287 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce5408a140 con 0x7fce6c0d22e0 2024-03-06T07:22:27.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.800+0000 7fce6a59c640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce4c06b500 0x7fce4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.801+0000 7fce6a59c640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce4c06b500 0x7fce4c06d9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fce5c036000 tx=0x7fce5c03b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.804+0000 7fce62ffd640 1 -- 172.21.15.59:0/3875568287 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce54058970 con 0x7fce6c0d22e0 2024-03-06T07:22:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.984+0000 7fce70ac9640 1 -- 172.21.15.59:0/3875568287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fce6c0ced20 con 0x7fce6c0d22e0 2024-03-06T07:22:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.984+0000 7fce62ffd640 1 -- 172.21.15.59:0/3875568287 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fce5405ca10 con 0x7fce6c0d22e0 2024-03-06T07:22:27.986 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 -- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce4c06b500 msgr2=0x7fce4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce4c06b500 0x7fce4c06d9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fce5c036000 tx=0x7fce5c03b040 comp rx=0 tx=0).stop 2024-03-06T07:22:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 -- 172.21.15.59:0/3875568287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0d22e0 msgr2=0x7fce6c1f70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0d22e0 0x7fce6c1f70c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fce5400e9b0 tx=0x7fce5400ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:27.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 -- 172.21.15.59:0/3875568287 shutdown_connections 2024-03-06T07:22:27.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce4c06b500 0x7fce4c06d9c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce6c1614d0 0x7fce6c0c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce6c0d22e0 0x7fce6c1f70c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 --2- 172.21.15.59:0/3875568287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce6c0d1910 0x7fce6c155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 -- 172.21.15.59:0/3875568287 >> 172.21.15.59:0/3875568287 conn(0x7fce6c0d3d40 msgr2=0x7fce6c06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 -- 172.21.15.59:0/3875568287 shutdown_connections 2024-03-06T07:22:27.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:27.986+0000 7fce70ac9640 1 -- 172.21.15.59:0/3875568287 wait complete. 2024-03-06T07:22:27.999 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:28.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.102+0000 7ff4a725c640 1 Processor -- start 2024-03-06T07:22:28.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.102+0000 7ff4a725c640 1 -- start start 2024-03-06T07:22:28.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.103+0000 7ff4a725c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4a00d4860 0x7ff4a00d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.103+0000 7ff4a725c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a01618b0 con 0x7ff4a00b0e30 2024-03-06T07:22:28.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.103+0000 7ff4a725c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a00cf400 con 0x7ff4a01614d0 2024-03-06T07:22:28.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.103+0000 7ff4a725c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a00d4cf0 con 0x7ff4a00d4860 2024-03-06T07:22:28.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.103+0000 7ff49ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4a00d4860 0x7ff4a00d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.103+0000 7ff49ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4a00d4860 0x7ff4a00d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52838/0 (socket says 172.21.15.59:52838) 2024-03-06T07:22:28.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.103+0000 7ff49ffff640 1 -- 172.21.15.59:0/2489263580 learned_addr learned my addr 172.21.15.59:0/2489263580 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:28.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.103+0000 7ff49ffff640 1 -- 172.21.15.59:0/2489263580 >> v1:172.21.15.164:6789/0 conn(0x7ff4a01614d0 legacy=0x7ff4a00d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.103+0000 7ff49f7fe640 1 -- 172.21.15.59:0/2489263580 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601282207 0 0) 0x7ff4a00cf400 con 0x7ff4a01614d0 2024-03-06T07:22:28.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.103+0000 7ff49ffff640 1 -- 172.21.15.59:0/2489263580 >> v1:172.21.15.59:6789/0 conn(0x7ff4a00b0e30 legacy=0x7ff4a00b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.103+0000 7ff49ffff640 1 -- 172.21.15.59:0/2489263580 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a00cf6a0 con 0x7ff4a00d4860 2024-03-06T07:22:28.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.104+0000 7ff49ffff640 1 --2- 172.21.15.59:0/2489263580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00d4860 0x7ff4a00d2020 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff488009870 tx=0x7ff48802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3f8e11fa14b40b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.104+0000 7ff49f7fe640 1 -- 172.21.15.59:0/2489263580 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48802f840 con 0x7ff4a00d4860 2024-03-06T07:22:28.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.104+0000 7ff49f7fe640 1 -- 172.21.15.59:0/2489263580 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48802f9e0 con 0x7ff4a00d4860 2024-03-06T07:22:28.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.104+0000 7ff49f7fe640 1 -- 172.21.15.59:0/2489263580 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48802fdb0 con 0x7ff4a00d4860 2024-03-06T07:22:28.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.105+0000 7ff4a725c640 1 -- 172.21.15.59:0/2489263580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00d4860 msgr2=0x7ff4a00d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.105+0000 7ff4a725c640 1 --2- 172.21.15.59:0/2489263580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00d4860 0x7ff4a00d2020 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff488009870 tx=0x7ff48802ba20 comp rx=0 tx=0).stop 2024-03-06T07:22:28.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.105+0000 7ff4a725c640 1 -- 172.21.15.59:0/2489263580 shutdown_connections 2024-03-06T07:22:28.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.105+0000 7ff4a725c640 1 --2- 172.21.15.59:0/2489263580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00d4860 0x7ff4a00d2020 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.105+0000 7ff4a725c640 1 -- 172.21.15.59:0/2489263580 >> 172.21.15.59:0/2489263580 conn(0x7ff4a00d3d40 msgr2=0x7ff4a00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.105+0000 7ff4a725c640 1 -- 172.21.15.59:0/2489263580 shutdown_connections 2024-03-06T07:22:28.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.105+0000 7ff4a725c640 1 -- 172.21.15.59:0/2489263580 wait complete. 2024-03-06T07:22:28.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.106+0000 7ff4a725c640 1 Processor -- start 2024-03-06T07:22:28.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.106+0000 7ff4a725c640 1 -- start start 2024-03-06T07:22:28.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a725c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00b0e30 0x7ff4a00cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a725c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a00d4860 0x7ff4a00b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a725c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4a01614d0 0x7ff4a00b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a725c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a00b0910 con 0x7ff4a00b0e30 2024-03-06T07:22:28.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a725c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a00cfc60 con 0x7ff4a01614d0 2024-03-06T07:22:28.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a725c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a00d2560 con 0x7ff4a00d4860 2024-03-06T07:22:28.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a4ff8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00b0e30 0x7ff4a00cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a4ff8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00b0e30 0x7ff4a00cd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52852/0 (socket says 172.21.15.59:52852) 2024-03-06T07:22:28.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a4ff8640 1 -- 172.21.15.59:0/2801557163 learned_addr learned my addr 172.21.15.59:0/2801557163 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:28.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a57f9640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4a01614d0 0x7ff4a00b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff49ffff640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a00d4860 0x7ff4a00b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a4ff8640 1 -- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a00d4860 msgr2=0x7ff4a00b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a4ff8640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a00d4860 0x7ff4a00b35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a4ff8640 1 -- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4a01614d0 msgr2=0x7ff4a00b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a4ff8640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4a01614d0 0x7ff4a00b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a4ff8640 1 -- 172.21.15.59:0/2801557163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a00b4060 con 0x7ff4a00b0e30 2024-03-06T07:22:28.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff4a4ff8640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00b0e30 0x7ff4a00cd250 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff49800c820 tx=0x7ff49800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.107+0000 7ff49d7fa640 1 -- 172.21.15.59:0/2801557163 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498017070 con 0x7ff4a00b0e30 2024-03-06T07:22:28.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.108+0000 7ff49d7fa640 1 -- 172.21.15.59:0/2801557163 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff498003da0 con 0x7ff4a00b0e30 2024-03-06T07:22:28.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.108+0000 7ff4a725c640 1 -- 172.21.15.59:0/2801557163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a00b45c0 con 0x7ff4a00b0e30 2024-03-06T07:22:28.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.108+0000 7ff49d7fa640 1 -- 172.21.15.59:0/2801557163 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498005290 con 0x7ff4a00b0e30 2024-03-06T07:22:28.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.108+0000 7ff4a725c640 1 -- 172.21.15.59:0/2801557163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a00b4b80 con 0x7ff4a00b0e30 2024-03-06T07:22:28.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.109+0000 7ff4a725c640 1 -- 172.21.15.59:0/2801557163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a00b4340 con 0x7ff4a00b0e30 2024-03-06T07:22:28.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.109+0000 7ff49d7fa640 1 -- 172.21.15.59:0/2801557163 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff49801a070 con 0x7ff4a00b0e30 2024-03-06T07:22:28.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.109+0000 7ff49d7fa640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48006b4d0 0x7ff48006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.109+0000 7ff49d7fa640 1 -- 172.21.15.59:0/2801557163 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff498089800 con 0x7ff4a00b0e30 2024-03-06T07:22:28.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.109+0000 7ff49ffff640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48006b4d0 0x7ff48006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.110+0000 7ff49ffff640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48006b4d0 0x7ff48006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff488036000 tx=0x7ff488008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.114+0000 7ff49d7fa640 1 -- 172.21.15.59:0/2801557163 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4a00b4340 con 0x7ff4a00b0e30 2024-03-06T07:22:28.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.295+0000 7ff4a725c640 1 -- 172.21.15.59:0/2801557163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff4a0068ff0 con 0x7ff4a00b0e30 2024-03-06T07:22:28.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.295+0000 7ff49d7fa640 1 -- 172.21.15.59:0/2801557163 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff4a00b4060 con 0x7ff4a00b0e30 2024-03-06T07:22:28.297 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:28.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 -- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48006b4d0 msgr2=0x7ff48006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48006b4d0 0x7ff48006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff488036000 tx=0x7ff488008030 comp rx=0 tx=0).stop 2024-03-06T07:22:28.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 -- 172.21.15.59:0/2801557163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00b0e30 msgr2=0x7ff4a00cd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00b0e30 0x7ff4a00cd250 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff49800c820 tx=0x7ff49800cce0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 -- 172.21.15.59:0/2801557163 shutdown_connections 2024-03-06T07:22:28.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff48006b4d0 0x7ff48006d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4a01614d0 0x7ff4a00b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4a00d4860 0x7ff4a00b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 --2- 172.21.15.59:0/2801557163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4a00b0e30 0x7ff4a00cd250 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 -- 172.21.15.59:0/2801557163 >> 172.21.15.59:0/2801557163 conn(0x7ff4a00d3d40 msgr2=0x7ff4a00ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 -- 172.21.15.59:0/2801557163 shutdown_connections 2024-03-06T07:22:28.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.297+0000 7ff4a725c640 1 -- 172.21.15.59:0/2801557163 wait complete. 2024-03-06T07:22:28.309 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.412+0000 7efd42177640 1 Processor -- start 2024-03-06T07:22:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.412+0000 7efd42177640 1 -- start start 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.412+0000 7efd42177640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd3c0cea50 0x7efd3c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.412+0000 7efd42177640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efd3c06cab0 0x7efd3c0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.412+0000 7efd42177640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c06d030 con 0x7efd3c15def0 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.412+0000 7efd42177640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd3c0cf1c0 con 0x7efd3c06cab0 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.413+0000 7efd42177640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0cc760 con 0x7efd3c0cea50 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.413+0000 7efd3bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efd3c06cab0 0x7efd3c0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.413+0000 7efd3bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efd3c06cab0 0x7efd3c0ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52862/0 (socket says 172.21.15.59:52862) 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.413+0000 7efd3bfff640 1 -- 172.21.15.59:0/1047375520 learned_addr learned my addr 172.21.15.59:0/1047375520 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.413+0000 7efd3affd640 1 --2- 172.21.15.59:0/1047375520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd3c0cea50 0x7efd3c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.413+0000 7efd3bfff640 1 -- 172.21.15.59:0/1047375520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd3c0cea50 msgr2=0x7efd3c0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.413+0000 7efd3bfff640 1 --2- 172.21.15.59:0/1047375520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd3c0cea50 0x7efd3c0b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.413+0000 7efd3bfff640 1 -- 172.21.15.59:0/1047375520 >> v1:172.21.15.59:6789/0 conn(0x7efd3c15def0 legacy=0x7efd3c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.413+0000 7efd3bfff640 1 -- 172.21.15.59:0/1047375520 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c06d310 con 0x7efd3c06cab0 2024-03-06T07:22:28.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd3bfff640 1 --2- 172.21.15.59:0/1047375520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd3c06cab0 0x7efd3c0ccbf0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efd34009e50 tx=0x7efd3402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=facf9f96beb23a0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd3a7fc640 1 -- 172.21.15.59:0/1047375520 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3402cb90 con 0x7efd3c06cab0 2024-03-06T07:22:28.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd3a7fc640 1 -- 172.21.15.59:0/1047375520 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3402cd30 con 0x7efd3c06cab0 2024-03-06T07:22:28.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd3a7fc640 1 -- 172.21.15.59:0/1047375520 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34034610 con 0x7efd3c06cab0 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd42177640 1 -- 172.21.15.59:0/1047375520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd3c06cab0 msgr2=0x7efd3c0ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd42177640 1 --2- 172.21.15.59:0/1047375520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd3c06cab0 0x7efd3c0ccbf0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7efd34009e50 tx=0x7efd3402bff0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd42177640 1 -- 172.21.15.59:0/1047375520 shutdown_connections 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd42177640 1 --2- 172.21.15.59:0/1047375520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd3c06cab0 0x7efd3c0ccbf0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd42177640 1 --2- 172.21.15.59:0/1047375520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd3c0cea50 0x7efd3c0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd42177640 1 -- 172.21.15.59:0/1047375520 >> 172.21.15.59:0/1047375520 conn(0x7efd3c0b3690 msgr2=0x7efd3c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd42177640 1 -- 172.21.15.59:0/1047375520 shutdown_connections 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.414+0000 7efd42177640 1 -- 172.21.15.59:0/1047375520 wait complete. 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.416+0000 7efd42177640 1 Processor -- start 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.416+0000 7efd42177640 1 -- start start 2024-03-06T07:22:28.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.416+0000 7efd42177640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd3c06cab0 0x7efd3c0cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.416+0000 7efd42177640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd3c0cea50 0x7efd3c154080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.416+0000 7efd42177640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd3c15def0 0x7efd3c0cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.416+0000 7efd42177640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c15e2d0 con 0x7efd3c15def0 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.416+0000 7efd42177640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0cc3f0 con 0x7efd3c0cea50 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.416+0000 7efd42177640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c15e450 con 0x7efd3c06cab0 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd3bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd3c15def0 0x7efd3c0cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd3bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd3c15def0 0x7efd3c0cb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52874/0 (socket says 172.21.15.59:52874) 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd3bfff640 1 -- 172.21.15.59:0/1010957697 learned_addr learned my addr 172.21.15.59:0/1010957697 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd3affd640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd3c0cea50 0x7efd3c154080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd3b7fe640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd3c06cab0 0x7efd3c0cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd3bfff640 1 -- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd3c06cab0 msgr2=0x7efd3c0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd3bfff640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd3c06cab0 0x7efd3c0cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd3bfff640 1 -- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd3c0cea50 msgr2=0x7efd3c154080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd3bfff640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd3c0cea50 0x7efd3c154080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd3bfff640 1 -- 172.21.15.59:0/1010957697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c0c96d0 con 0x7efd3c15def0 2024-03-06T07:22:28.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd3bfff640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd3c15def0 0x7efd3c0cb530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efd34009e50 tx=0x7efd34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd23fff640 1 -- 172.21.15.59:0/1010957697 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34043020 con 0x7efd3c15def0 2024-03-06T07:22:28.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd23fff640 1 -- 172.21.15.59:0/1010957697 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3403e450 con 0x7efd3c15def0 2024-03-06T07:22:28.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd23fff640 1 -- 172.21.15.59:0/1010957697 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34009510 con 0x7efd3c15def0 2024-03-06T07:22:28.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd42177640 1 -- 172.21.15.59:0/1010957697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3c0c99b0 con 0x7efd3c15def0 2024-03-06T07:22:28.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.417+0000 7efd42177640 1 -- 172.21.15.59:0/1010957697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3c0c9f70 con 0x7efd3c15def0 2024-03-06T07:22:28.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.418+0000 7efd42177640 1 -- 172.21.15.59:0/1010957697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3c09fcf0 con 0x7efd3c15def0 2024-03-06T07:22:28.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.419+0000 7efd23fff640 1 -- 172.21.15.59:0/1010957697 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd34046020 con 0x7efd3c15def0 2024-03-06T07:22:28.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.419+0000 7efd23fff640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd2806b4d0 0x7efd2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.419+0000 7efd23fff640 1 -- 172.21.15.59:0/1010957697 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd340ac6c0 con 0x7efd3c15def0 2024-03-06T07:22:28.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.419+0000 7efd3b7fe640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd2806b4d0 0x7efd2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.420+0000 7efd3b7fe640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd2806b4d0 0x7efd2806d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0016e0 tx=0x7efd2c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.423+0000 7efd23fff640 1 -- 172.21.15.59:0/1010957697 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd3404d030 con 0x7efd3c15def0 2024-03-06T07:22:28.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.606+0000 7efd42177640 1 -- 172.21.15.59:0/1010957697 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efd3c068ff0 con 0x7efd3c15def0 2024-03-06T07:22:28.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.607+0000 7efd23fff640 1 -- 172.21.15.59:0/1010957697 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7efd3407af80 con 0x7efd3c15def0 2024-03-06T07:22:28.609 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 -- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd2806b4d0 msgr2=0x7efd2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd2806b4d0 0x7efd2806d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0016e0 tx=0x7efd2c009040 comp rx=0 tx=0).stop 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 -- 172.21.15.59:0/1010957697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd3c15def0 msgr2=0x7efd3c0cb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd3c15def0 0x7efd3c0cb530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efd34009e50 tx=0x7efd34004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 -- 172.21.15.59:0/1010957697 shutdown_connections 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd2806b4d0 0x7efd2806d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd3c15def0 0x7efd3c0cb530 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd3c0cea50 0x7efd3c154080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 --2- 172.21.15.59:0/1010957697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd3c06cab0 0x7efd3c0cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 -- 172.21.15.59:0/1010957697 >> 172.21.15.59:0/1010957697 conn(0x7efd3c0b3690 msgr2=0x7efd3c06fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 -- 172.21.15.59:0/1010957697 shutdown_connections 2024-03-06T07:22:28.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.609+0000 7efd42177640 1 -- 172.21.15.59:0/1010957697 wait complete. 2024-03-06T07:22:28.621 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:28.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.721+0000 7f1c2f5f7640 1 Processor -- start 2024-03-06T07:22:28.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.722+0000 7f1c2f5f7640 1 -- start start 2024-03-06T07:22:28.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.722+0000 7f1c2f5f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c28162970 0x7f1c280d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.722+0000 7f1c2f5f7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c280d22c0 con 0x7f1c28161cf0 2024-03-06T07:22:28.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.722+0000 7f1c2f5f7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c280d25a0 con 0x7f1c280b4460 2024-03-06T07:22:28.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.722+0000 7f1c2f5f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c28161b00 con 0x7f1c28162970 2024-03-06T07:22:28.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.722+0000 7f1c2db94640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1c28161cf0 0x7f1c280cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47036/0 (socket says 172.21.15.59:47036) 2024-03-06T07:22:28.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.723+0000 7f1c2db94640 1 -- 172.21.15.59:0/2559177899 learned_addr learned my addr 172.21.15.59:0/2559177899 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:28.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.723+0000 7f1c2cb92640 1 --2- 172.21.15.59:0/2559177899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c28162970 0x7f1c280d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.723+0000 7f1c1ffff640 1 -- 172.21.15.59:0/2559177899 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340761592 0 0) 0x7f1c280d22c0 con 0x7f1c28161cf0 2024-03-06T07:22:28.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.723+0000 7f1c1ffff640 1 -- 172.21.15.59:0/2559177899 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c14003520 con 0x7f1c28161cf0 2024-03-06T07:22:28.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.723+0000 7f1c1ffff640 1 -- 172.21.15.59:0/2559177899 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610766939 0 0) 0x7f1c280d25a0 con 0x7f1c280b4460 2024-03-06T07:22:28.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.723+0000 7f1c1ffff640 1 -- 172.21.15.59:0/2559177899 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c280d22c0 con 0x7f1c280b4460 2024-03-06T07:22:28.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.723+0000 7f1c2cb92640 1 -- 172.21.15.59:0/2559177899 >> v1:172.21.15.164:6789/0 conn(0x7f1c280b4460 legacy=0x7f1c280b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.723+0000 7f1c2cb92640 1 -- 172.21.15.59:0/2559177899 >> v1:172.21.15.59:6789/0 conn(0x7f1c28161cf0 legacy=0x7f1c280cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.723+0000 7f1c2cb92640 1 -- 172.21.15.59:0/2559177899 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280d2880 con 0x7f1c28162970 2024-03-06T07:22:28.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.724+0000 7f1c2cb92640 1 --2- 172.21.15.59:0/2559177899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c28162970 0x7f1c280d4800 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c180097d0 tx=0x7f1c1802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dea38da90cc8245c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.724+0000 7f1c1ffff640 1 -- 172.21.15.59:0/2559177899 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c18035020 con 0x7f1c28162970 2024-03-06T07:22:28.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.724+0000 7f1c1ffff640 1 -- 172.21.15.59:0/2559177899 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c18031890 con 0x7f1c28162970 2024-03-06T07:22:28.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.724+0000 7f1c1ffff640 1 -- 172.21.15.59:0/2559177899 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c18031c60 con 0x7f1c28162970 2024-03-06T07:22:28.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.724+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/2559177899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c28162970 msgr2=0x7f1c280d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.724+0000 7f1c2f5f7640 1 --2- 172.21.15.59:0/2559177899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c28162970 0x7f1c280d4800 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c180097d0 tx=0x7f1c1802b930 comp rx=0 tx=0).stop 2024-03-06T07:22:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.724+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/2559177899 shutdown_connections 2024-03-06T07:22:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.724+0000 7f1c2f5f7640 1 --2- 172.21.15.59:0/2559177899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c28162970 0x7f1c280d4800 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.724+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/2559177899 >> 172.21.15.59:0/2559177899 conn(0x7f1c280d3ce0 msgr2=0x7f1c280d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.725+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/2559177899 shutdown_connections 2024-03-06T07:22:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.725+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/2559177899 wait complete. 2024-03-06T07:22:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.726+0000 7f1c2f5f7640 1 Processor -- start 2024-03-06T07:22:28.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.726+0000 7f1c2f5f7640 1 -- start start 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.726+0000 7f1c2f5f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c280b4460 0x7f1c280c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.726+0000 7f1c2f5f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c28161cf0 0x7f1c28154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.726+0000 7f1c2f5f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c28162970 0x7f1c280cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.726+0000 7f1c2f5f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280ce950 con 0x7f1c280b4460 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.726+0000 7f1c2f5f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280cecc0 con 0x7f1c28161cf0 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.726+0000 7f1c2f5f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280b4840 con 0x7f1c28162970 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.726+0000 7f1c2d393640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c280b4460 0x7f1c280c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2d393640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c280b4460 0x7f1c280c8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52878/0 (socket says 172.21.15.59:52878) 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2d393640 1 -- 172.21.15.59:0/3471717974 learned_addr learned my addr 172.21.15.59:0/3471717974 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2cb92640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c28161cf0 0x7f1c28154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2db94640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c28162970 0x7f1c280cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2d393640 1 -- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c28162970 msgr2=0x7f1c280cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2d393640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c28162970 0x7f1c280cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2d393640 1 -- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c28161cf0 msgr2=0x7f1c28154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2d393640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c28161cf0 0x7f1c28154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2d393640 1 -- 172.21.15.59:0/3471717974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280c7b60 con 0x7f1c280b4460 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2d393640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c280b4460 0x7f1c280c8c00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400c430 tx=0x7f1c2400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c1dffb640 1 -- 172.21.15.59:0/3471717974 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c24017070 con 0x7f1c280b4460 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c1dffb640 1 -- 172.21.15.59:0/3471717974 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c240052f0 con 0x7f1c280b4460 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/3471717974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c280c7e40 con 0x7f1c280b4460 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c1dffb640 1 -- 172.21.15.59:0/3471717974 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c24003830 con 0x7f1c280b4460 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.727+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/3471717974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c280c8400 con 0x7f1c280b4460 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.728+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/3471717974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2804f330 con 0x7f1c280b4460 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.729+0000 7f1c1dffb640 1 -- 172.21.15.59:0/3471717974 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c2401a070 con 0x7f1c280b4460 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.729+0000 7f1c1dffb640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c0c06b500 0x7f1c0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.729+0000 7f1c1dffb640 1 -- 172.21.15.59:0/3471717974 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c24089450 con 0x7f1c280b4460 2024-03-06T07:22:28.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.729+0000 7f1c2cb92640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c0c06b500 0x7f1c0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.730+0000 7f1c2cb92640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c0c06b500 0x7f1c0c06d9c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c18036000 tx=0x7f1c18008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.733+0000 7f1c1dffb640 1 -- 172.21.15.59:0/3471717974 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c24057c80 con 0x7f1c280b4460 2024-03-06T07:22:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.928+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/3471717974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1c280b9b70 con 0x7f1c280b4460 2024-03-06T07:22:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.928+0000 7f1c1dffb640 1 -- 172.21.15.59:0/3471717974 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1c2405bd20 con 0x7f1c280b4460 2024-03-06T07:22:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:22:28.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c0c06b500 msgr2=0x7f1c0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c0c06b500 0x7f1c0c06d9c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c18036000 tx=0x7f1c18008030 comp rx=0 tx=0).stop 2024-03-06T07:22:28.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/3471717974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c280b4460 msgr2=0x7f1c280c8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c280b4460 0x7f1c280c8c00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400c430 tx=0x7f1c2400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/3471717974 shutdown_connections 2024-03-06T07:22:28.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c0c06b500 0x7f1c0c06d9c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c28162970 0x7f1c280cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c28161cf0 0x7f1c28154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 --2- 172.21.15.59:0/3471717974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c280b4460 0x7f1c280c8c00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/3471717974 >> 172.21.15.59:0/3471717974 conn(0x7f1c280d3ce0 msgr2=0x7f1c280d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/3471717974 shutdown_connections 2024-03-06T07:22:28.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:28.930+0000 7f1c2f5f7640 1 -- 172.21.15.59:0/3471717974 wait complete. 2024-03-06T07:22:28.943 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:22:28.943 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:28.959 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:29.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.058+0000 7fa17eafa640 1 Processor -- start 2024-03-06T07:22:29.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.058+0000 7fa17eafa640 1 -- start start 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.059+0000 7fa17eafa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa17815dfe0 0x7fa1780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.059+0000 7fa17eafa640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa1780cf830 0x7fa1780d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.059+0000 7fa17eafa640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1780b3430 con 0x7fa1780d27f0 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.059+0000 7fa17eafa640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1780d2c50 con 0x7fa1780cf830 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.059+0000 7fa17eafa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780b31e0 con 0x7fa17815dfe0 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.059+0000 7fa17c896640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa17815dfe0 0x7fa1780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.059+0000 7fa177fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa1780cf830 0x7fa1780d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.059+0000 7fa17d097640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa1780d27f0 0x7fa1780d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:34684/0 (socket says 172.21.15.59:34684) 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.059+0000 7fa17c896640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa17815dfe0 0x7fa1780cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:35870/0 (socket says 172.21.15.59:35870) 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.059+0000 7fa17d097640 1 -- 172.21.15.59:0/3500050362 learned_addr learned my addr 172.21.15.59:0/3500050362 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.060+0000 7fa17c896640 1 -- 172.21.15.59:0/3500050362 >> v2:172.21.15.164:3300/0 conn(0x7fa1780cf830 msgr2=0x7fa1780d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.060+0000 7fa17c896640 1 --2- 172.21.15.59:0/3500050362 >> v2:172.21.15.164:3300/0 conn(0x7fa1780cf830 0x7fa1780d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.060+0000 7fa17c896640 1 -- 172.21.15.59:0/3500050362 >> v1:172.21.15.164:6789/0 conn(0x7fa1780d27f0 legacy=0x7fa1780d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.060+0000 7fa17c896640 1 -- 172.21.15.59:0/3500050362 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1780b3710 con 0x7fa17815dfe0 2024-03-06T07:22:29.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.060+0000 7fa1777fe640 1 -- 172.21.15.59:0/3500050362 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095657739 0 0) 0x7fa1780b3430 con 0x7fa1780d27f0 2024-03-06T07:22:29.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.060+0000 7fa177fff640 1 --2- 172.21.15.59:0/3500050362 >> v2:172.21.15.164:3300/0 conn(0x7fa1780cf830 0x7fa1780d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa17c896640 1 --2- 172.21.15.59:0/3500050362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa17815dfe0 0x7fa1780cf2f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa1680097a0 tx=0x7fa16802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=600d555816885e35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa1777fe640 1 -- 172.21.15.59:0/3500050362 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168035020 con 0x7fa17815dfe0 2024-03-06T07:22:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa1777fe640 1 -- 172.21.15.59:0/3500050362 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa168031830 con 0x7fa17815dfe0 2024-03-06T07:22:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa1777fe640 1 -- 172.21.15.59:0/3500050362 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168031c00 con 0x7fa17815dfe0 2024-03-06T07:22:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa17eafa640 1 -- 172.21.15.59:0/3500050362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa17815dfe0 msgr2=0x7fa1780cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa17eafa640 1 --2- 172.21.15.59:0/3500050362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa17815dfe0 0x7fa1780cf2f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa1680097a0 tx=0x7fa16802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa17eafa640 1 -- 172.21.15.59:0/3500050362 shutdown_connections 2024-03-06T07:22:29.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa17eafa640 1 --2- 172.21.15.59:0/3500050362 >> v2:172.21.15.164:3300/0 conn(0x7fa1780cf830 0x7fa1780d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa17eafa640 1 --2- 172.21.15.59:0/3500050362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa17815dfe0 0x7fa1780cf2f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa17eafa640 1 -- 172.21.15.59:0/3500050362 >> 172.21.15.59:0/3500050362 conn(0x7fa1780d3cd0 msgr2=0x7fa1780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa17eafa640 1 -- 172.21.15.59:0/3500050362 shutdown_connections 2024-03-06T07:22:29.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.061+0000 7fa17eafa640 1 -- 172.21.15.59:0/3500050362 wait complete. 2024-03-06T07:22:29.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.062+0000 7fa17eafa640 1 Processor -- start 2024-03-06T07:22:29.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.062+0000 7fa17eafa640 1 -- start start 2024-03-06T07:22:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.063+0000 7fa17eafa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1780cf830 0x7fa1780a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.063+0000 7fa17eafa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1780d27f0 0x7fa1780b8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.063+0000 7fa17eafa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa17815dfe0 0x7fa1781603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.063+0000 7fa17eafa640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780d1c70 con 0x7fa17815dfe0 2024-03-06T07:22:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.063+0000 7fa17eafa640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780b3d60 con 0x7fa1780cf830 2024-03-06T07:22:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.063+0000 7fa17eafa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1780d1df0 con 0x7fa1780d27f0 2024-03-06T07:22:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.063+0000 7fa17d097640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa17815dfe0 0x7fa1781603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17d097640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa17815dfe0 0x7fa1781603b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52888/0 (socket says 172.21.15.59:52888) 2024-03-06T07:22:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17d097640 1 -- 172.21.15.59:0/978031473 learned_addr learned my addr 172.21.15.59:0/978031473 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17c896640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1780cf830 0x7fa1780a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa177fff640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1780d27f0 0x7fa1780b8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17d097640 1 -- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1780d27f0 msgr2=0x7fa1780b8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17d097640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1780d27f0 0x7fa1780b8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17d097640 1 -- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1780cf830 msgr2=0x7fa1780a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17d097640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1780cf830 0x7fa1780a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17d097640 1 -- 172.21.15.59:0/978031473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1780cbcb0 con 0x7fa17815dfe0 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17c896640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1780cf830 0x7fa1780a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17d097640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa17815dfe0 0x7fa1781603b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa17000c7a0 tx=0x7fa17000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa1757fa640 1 -- 172.21.15.59:0/978031473 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa170018070 con 0x7fa17815dfe0 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa1757fa640 1 -- 172.21.15.59:0/978031473 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17000a640 con 0x7fa17815dfe0 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17eafa640 1 -- 172.21.15.59:0/978031473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1780cbf90 con 0x7fa17815dfe0 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa1757fa640 1 -- 172.21.15.59:0/978031473 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17000aab0 con 0x7fa17815dfe0 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.064+0000 7fa17eafa640 1 -- 172.21.15.59:0/978031473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1780cc550 con 0x7fa17815dfe0 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.065+0000 7fa17eafa640 1 -- 172.21.15.59:0/978031473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17808d3c0 con 0x7fa17815dfe0 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.066+0000 7fa1757fa640 1 -- 172.21.15.59:0/978031473 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa17001b070 con 0x7fa17815dfe0 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.066+0000 7fa1757fa640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa16406b4d0 0x7fa16406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.066+0000 7fa1757fa640 1 -- 172.21.15.59:0/978031473 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa170089700 con 0x7fa17815dfe0 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.066+0000 7fa17c896640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa16406b4d0 0x7fa16406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.067+0000 7fa17c896640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa16406b4d0 0x7fa16406d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa168036000 tx=0x7fa168008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.070+0000 7fa1757fa640 1 -- 172.21.15.59:0/978031473 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa17808d3c0 con 0x7fa17815dfe0 2024-03-06T07:22:29.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.250+0000 7fa17eafa640 1 -- 172.21.15.59:0/978031473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa17804f330 con 0x7fa17815dfe0 2024-03-06T07:22:29.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.251+0000 7fa1757fa640 1 -- 172.21.15.59:0/978031473 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fa170057fc0 con 0x7fa17815dfe0 2024-03-06T07:22:29.253 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:29.254 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:29.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 -- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa16406b4d0 msgr2=0x7fa16406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa16406b4d0 0x7fa16406d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa168036000 tx=0x7fa168008030 comp rx=0 tx=0).stop 2024-03-06T07:22:29.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 -- 172.21.15.59:0/978031473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa17815dfe0 msgr2=0x7fa1781603b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa17815dfe0 0x7fa1781603b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa17000c7a0 tx=0x7fa17000cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:29.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 -- 172.21.15.59:0/978031473 shutdown_connections 2024-03-06T07:22:29.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa16406b4d0 0x7fa16406d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa17815dfe0 0x7fa1781603b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1780d27f0 0x7fa1780b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 --2- 172.21.15.59:0/978031473 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1780cf830 0x7fa1780a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 -- 172.21.15.59:0/978031473 >> 172.21.15.59:0/978031473 conn(0x7fa1780d3cd0 msgr2=0x7fa17806d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 -- 172.21.15.59:0/978031473 shutdown_connections 2024-03-06T07:22:29.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.253+0000 7fa17eafa640 1 -- 172.21.15.59:0/978031473 wait complete. 2024-03-06T07:22:29.266 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:29.266 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.370+0000 7f12a2fac640 1 Processor -- start 2024-03-06T07:22:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.370+0000 7f12a2fac640 1 -- start start 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.370+0000 7f12a2fac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f129c15def0 0x7f129c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.370+0000 7f12a2fac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f129c0cea50 0x7f129c0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.370+0000 7f12a2fac640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c06d030 con 0x7f129c06cab0 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.370+0000 7f12a2fac640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f129c0cf1c0 con 0x7f129c0cea50 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.370+0000 7f12a2fac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c0cc760 con 0x7f129c15def0 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.371+0000 7f12a1549640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f129c0cea50 0x7f129c0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.371+0000 7f12a1549640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f129c0cea50 0x7f129c0ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52904/0 (socket says 172.21.15.59:52904) 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.371+0000 7f12a1549640 1 -- 172.21.15.59:0/494458780 learned_addr learned my addr 172.21.15.59:0/494458780 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.371+0000 7f12a0d48640 1 --2- 172.21.15.59:0/494458780 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f129c15def0 0x7f129c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.371+0000 7f12a1549640 1 -- 172.21.15.59:0/494458780 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f129c15def0 msgr2=0x7f129c0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.371+0000 7f12a1549640 1 --2- 172.21.15.59:0/494458780 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f129c15def0 0x7f129c0b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.371+0000 7f12a1549640 1 -- 172.21.15.59:0/494458780 >> v1:172.21.15.59:6789/0 conn(0x7f129c06cab0 legacy=0x7f129c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.371+0000 7f12a1549640 1 -- 172.21.15.59:0/494458780 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c06d310 con 0x7f129c0cea50 2024-03-06T07:22:29.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f12a1549640 1 --2- 172.21.15.59:0/494458780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c0cea50 0x7f129c0ccbf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1294009e50 tx=0x7f129402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93afdfbfb0580ee3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f129b7fe640 1 -- 172.21.15.59:0/494458780 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129402cb90 con 0x7f129c0cea50 2024-03-06T07:22:29.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f129b7fe640 1 -- 172.21.15.59:0/494458780 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129402cd30 con 0x7f129c0cea50 2024-03-06T07:22:29.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f129b7fe640 1 -- 172.21.15.59:0/494458780 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294034610 con 0x7f129c0cea50 2024-03-06T07:22:29.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f12a2fac640 1 -- 172.21.15.59:0/494458780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c0cea50 msgr2=0x7f129c0ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f12a2fac640 1 --2- 172.21.15.59:0/494458780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c0cea50 0x7f129c0ccbf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1294009e50 tx=0x7f129402bff0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f12a2fac640 1 -- 172.21.15.59:0/494458780 shutdown_connections 2024-03-06T07:22:29.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f12a2fac640 1 --2- 172.21.15.59:0/494458780 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c0cea50 0x7f129c0ccbf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f12a2fac640 1 --2- 172.21.15.59:0/494458780 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f129c15def0 0x7f129c0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f12a2fac640 1 -- 172.21.15.59:0/494458780 >> 172.21.15.59:0/494458780 conn(0x7f129c0b3690 msgr2=0x7f129c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f12a2fac640 1 -- 172.21.15.59:0/494458780 shutdown_connections 2024-03-06T07:22:29.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.372+0000 7f12a2fac640 1 -- 172.21.15.59:0/494458780 wait complete. 2024-03-06T07:22:29.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.373+0000 7f12a2fac640 1 Processor -- start 2024-03-06T07:22:29.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.373+0000 7f12a2fac640 1 -- start start 2024-03-06T07:22:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.374+0000 7f12a2fac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f129c06cab0 0x7f129c1f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.374+0000 7f12a2fac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f129c0cea50 0x7f129c0c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.374+0000 7f12a2fac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c15def0 0x7f129c0b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.374+0000 7f12a2fac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c15e2d0 con 0x7f129c15def0 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.374+0000 7f12a2fac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c0cc3f0 con 0x7f129c06cab0 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.374+0000 7f12a2fac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c15e450 con 0x7f129c0cea50 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.374+0000 7f12a1549640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c15def0 0x7f129c0b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.374+0000 7f12a1549640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c15def0 0x7f129c0b6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52916/0 (socket says 172.21.15.59:52916) 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.374+0000 7f12a1549640 1 -- 172.21.15.59:0/468172558 learned_addr learned my addr 172.21.15.59:0/468172558 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.374+0000 7f12a0d48640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f129c06cab0 0x7f129c1f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.374+0000 7f129bfff640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f129c0cea50 0x7f129c0c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.375+0000 7f12a1549640 1 -- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f129c0cea50 msgr2=0x7f129c0c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.375+0000 7f12a1549640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f129c0cea50 0x7f129c0c95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.375+0000 7f12a1549640 1 -- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f129c06cab0 msgr2=0x7f129c1f6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.375+0000 7f12a1549640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f129c06cab0 0x7f129c1f6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.375+0000 7f12a1549640 1 -- 172.21.15.59:0/468172558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c0c9f10 con 0x7f129c15def0 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.375+0000 7f12a1549640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c15def0 0x7f129c0b6950 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1294009e50 tx=0x7f1294004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.375+0000 7f12997fa640 1 -- 172.21.15.59:0/468172558 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294043020 con 0x7f129c15def0 2024-03-06T07:22:29.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.375+0000 7f12997fa640 1 -- 172.21.15.59:0/468172558 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129403e450 con 0x7f129c15def0 2024-03-06T07:22:29.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.375+0000 7f12997fa640 1 -- 172.21.15.59:0/468172558 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294009510 con 0x7f129c15def0 2024-03-06T07:22:29.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.375+0000 7f12a2fac640 1 -- 172.21.15.59:0/468172558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c0ca1f0 con 0x7f129c15def0 2024-03-06T07:22:29.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.375+0000 7f12a2fac640 1 -- 172.21.15.59:0/468172558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c0ca7b0 con 0x7f129c15def0 2024-03-06T07:22:29.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.376+0000 7f12a2fac640 1 -- 172.21.15.59:0/468172558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129c09fcf0 con 0x7f129c15def0 2024-03-06T07:22:29.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.376+0000 7f12997fa640 1 -- 172.21.15.59:0/468172558 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1294046020 con 0x7f129c15def0 2024-03-06T07:22:29.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.377+0000 7f12997fa640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f128006b500 0x7f128006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.377+0000 7f12997fa640 1 -- 172.21.15.59:0/468172558 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12940ac620 con 0x7f129c15def0 2024-03-06T07:22:29.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.377+0000 7f12a0d48640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f128006b500 0x7f128006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.377+0000 7f12a0d48640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f128006b500 0x7f128006d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f129c15d440 tx=0x7f1290009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.381+0000 7f12997fa640 1 -- 172.21.15.59:0/468172558 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f129404d030 con 0x7f129c15def0 2024-03-06T07:22:29.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.564+0000 7f12a2fac640 1 -- 172.21.15.59:0/468172558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f129c0cd9f0 con 0x7f129c15def0 2024-03-06T07:22:29.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.564+0000 7f12997fa640 1 -- 172.21.15.59:0/468172558 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f129407aee0 con 0x7f129c15def0 2024-03-06T07:22:29.566 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:29.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 -- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f128006b500 msgr2=0x7f128006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f128006b500 0x7f128006d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f129c15d440 tx=0x7f1290009190 comp rx=0 tx=0).stop 2024-03-06T07:22:29.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 -- 172.21.15.59:0/468172558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c15def0 msgr2=0x7f129c0b6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c15def0 0x7f129c0b6950 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1294009e50 tx=0x7f1294004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:29.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 -- 172.21.15.59:0/468172558 shutdown_connections 2024-03-06T07:22:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f128006b500 0x7f128006d9c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c15def0 0x7f129c0b6950 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f129c0cea50 0x7f129c0c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 --2- 172.21.15.59:0/468172558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f129c06cab0 0x7f129c1f6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 -- 172.21.15.59:0/468172558 >> 172.21.15.59:0/468172558 conn(0x7f129c0b3690 msgr2=0x7f129c0d2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 -- 172.21.15.59:0/468172558 shutdown_connections 2024-03-06T07:22:29.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.566+0000 7f12a2fac640 1 -- 172.21.15.59:0/468172558 wait complete. 2024-03-06T07:22:29.579 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:29.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.682+0000 7fbd63a81640 1 Processor -- start 2024-03-06T07:22:29.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.682+0000 7fbd63a81640 1 -- start start 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.682+0000 7fbd63a81640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbd5c066d90 0x7fbd5c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd63a81640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd5c0b5410 0x7fbd5c0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd63a81640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5c06b710 con 0x7fbd5c0cf9a0 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd63a81640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5c0b61c0 con 0x7fbd5c066d90 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd63a81640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c06b150 con 0x7fbd5c0b5410 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd6201e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd5c0b5410 0x7fbd5c0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd6101c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbd5c066d90 0x7fbd5c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd6181d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fbd5c0cf9a0 0x7fbd5c0b5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:34698/0 (socket says 172.21.15.59:34698) 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd6181d640 1 -- 172.21.15.59:0/3541709364 learned_addr learned my addr 172.21.15.59:0/3541709364 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd6101c640 1 -- 172.21.15.59:0/3541709364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd5c0b5410 msgr2=0x7fbd5c0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd6101c640 1 --2- 172.21.15.59:0/3541709364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd5c0b5410 0x7fbd5c0ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd6101c640 1 -- 172.21.15.59:0/3541709364 >> v1:172.21.15.164:6789/0 conn(0x7fbd5c0cf9a0 legacy=0x7fbd5c0b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd6101c640 1 -- 172.21.15.59:0/3541709364 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c06b430 con 0x7fbd5c066d90 2024-03-06T07:22:29.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.683+0000 7fbd6081b640 1 -- 172.21.15.59:0/3541709364 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119689747 0 0) 0x7fbd5c06b710 con 0x7fbd5c0cf9a0 2024-03-06T07:22:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.684+0000 7fbd6101c640 1 --2- 172.21.15.59:0/3541709364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd5c066d90 0x7fbd5c0b4ed0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0097a0 tx=0x7fbd4c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ded16d72938ef14a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.684+0000 7fbd6081b640 1 -- 172.21.15.59:0/3541709364 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c035020 con 0x7fbd5c066d90 2024-03-06T07:22:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.684+0000 7fbd6081b640 1 -- 172.21.15.59:0/3541709364 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c031830 con 0x7fbd5c066d90 2024-03-06T07:22:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.684+0000 7fbd6081b640 1 -- 172.21.15.59:0/3541709364 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c031c00 con 0x7fbd5c066d90 2024-03-06T07:22:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.684+0000 7fbd63a81640 1 -- 172.21.15.59:0/3541709364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd5c066d90 msgr2=0x7fbd5c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.684+0000 7fbd63a81640 1 --2- 172.21.15.59:0/3541709364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd5c066d90 0x7fbd5c0b4ed0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c0097a0 tx=0x7fbd4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.685+0000 7fbd63a81640 1 -- 172.21.15.59:0/3541709364 shutdown_connections 2024-03-06T07:22:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.685+0000 7fbd63a81640 1 --2- 172.21.15.59:0/3541709364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd5c0b5410 0x7fbd5c0ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.685+0000 7fbd63a81640 1 --2- 172.21.15.59:0/3541709364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd5c066d90 0x7fbd5c0b4ed0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.685+0000 7fbd63a81640 1 -- 172.21.15.59:0/3541709364 >> 172.21.15.59:0/3541709364 conn(0x7fbd5c0b1d70 msgr2=0x7fbd5c0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.685+0000 7fbd63a81640 1 -- 172.21.15.59:0/3541709364 shutdown_connections 2024-03-06T07:22:29.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.685+0000 7fbd63a81640 1 -- 172.21.15.59:0/3541709364 wait complete. 2024-03-06T07:22:29.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.686+0000 7fbd63a81640 1 Processor -- start 2024-03-06T07:22:29.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.686+0000 7fbd63a81640 1 -- start start 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd63a81640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd5c066d90 0x7fbd5c14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd63a81640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd5c0b5410 0x7fbd5c0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd63a81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd5c0cf9a0 0x7fbd5c0cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd63a81640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c0cfd80 con 0x7fbd5c0cf9a0 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd63a81640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c0b0c50 con 0x7fbd5c0b5410 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd63a81640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c0cff00 con 0x7fbd5c066d90 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6201e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd5c0cf9a0 0x7fbd5c0cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6181d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd5c066d90 0x7fbd5c14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6101c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd5c0b5410 0x7fbd5c0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6201e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd5c0cf9a0 0x7fbd5c0cd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52932/0 (socket says 172.21.15.59:52932) 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6201e640 1 -- 172.21.15.59:0/1706664680 learned_addr learned my addr 172.21.15.59:0/1706664680 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6181d640 1 -- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd5c0b5410 msgr2=0x7fbd5c0c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6181d640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd5c0b5410 0x7fbd5c0c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6181d640 1 -- 172.21.15.59:0/1706664680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd5c0cf9a0 msgr2=0x7fbd5c0cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6181d640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd5c0cf9a0 0x7fbd5c0cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6181d640 1 -- 172.21.15.59:0/1706664680 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c0c6e90 con 0x7fbd5c066d90 2024-03-06T07:22:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6201e640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd5c0cf9a0 0x7fbd5c0cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd6181d640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd5c066d90 0x7fbd5c14fd60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbd5400c410 tx=0x7fbd5400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd527fc640 1 -- 172.21.15.59:0/1706664680 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd540074e0 con 0x7fbd5c066d90 2024-03-06T07:22:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd527fc640 1 -- 172.21.15.59:0/1706664680 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd540047e0 con 0x7fbd5c066d90 2024-03-06T07:22:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd63a81640 1 -- 172.21.15.59:0/1706664680 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5c150c70 con 0x7fbd5c066d90 2024-03-06T07:22:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.687+0000 7fbd527fc640 1 -- 172.21.15.59:0/1706664680 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54004bd0 con 0x7fbd5c066d90 2024-03-06T07:22:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.688+0000 7fbd63a81640 1 -- 172.21.15.59:0/1706664680 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5c1506b0 con 0x7fbd5c066d90 2024-03-06T07:22:29.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.688+0000 7fbd63a81640 1 -- 172.21.15.59:0/1706664680 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5c09e3d0 con 0x7fbd5c066d90 2024-03-06T07:22:29.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.689+0000 7fbd527fc640 1 -- 172.21.15.59:0/1706664680 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd54011070 con 0x7fbd5c066d90 2024-03-06T07:22:29.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.689+0000 7fbd527fc640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbd4006b4d0 0x7fbd4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.689+0000 7fbd527fc640 1 -- 172.21.15.59:0/1706664680 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd54089c40 con 0x7fbd5c066d90 2024-03-06T07:22:29.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.689+0000 7fbd6101c640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbd4006b4d0 0x7fbd4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.690+0000 7fbd6101c640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbd4006b4d0 0x7fbd4006d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c036000 tx=0x7fbd4c03d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.695+0000 7fbd527fc640 1 -- 172.21.15.59:0/1706664680 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd54058520 con 0x7fbd5c066d90 2024-03-06T07:22:29.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.875+0000 7fbd63a81640 1 -- 172.21.15.59:0/1706664680 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbd5c06e7e0 con 0x7fbd5c066d90 2024-03-06T07:22:29.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.875+0000 7fbd527fc640 1 -- 172.21.15.59:0/1706664680 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbd5405c5c0 con 0x7fbd5c066d90 2024-03-06T07:22:29.877 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:29.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 -- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbd4006b4d0 msgr2=0x7fbd4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbd4006b4d0 0x7fbd4006d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c036000 tx=0x7fbd4c03d040 comp rx=0 tx=0).stop 2024-03-06T07:22:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 -- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd5c066d90 msgr2=0x7fbd5c14fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd5c066d90 0x7fbd5c14fd60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbd5400c410 tx=0x7fbd5400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 -- 172.21.15.59:0/1706664680 shutdown_connections 2024-03-06T07:22:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbd4006b4d0 0x7fbd4006d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd5c0cf9a0 0x7fbd5c0cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd5c0b5410 0x7fbd5c0c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 --2- 172.21.15.59:0/1706664680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd5c066d90 0x7fbd5c14fd60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 -- 172.21.15.59:0/1706664680 >> 172.21.15.59:0/1706664680 conn(0x7fbd5c0b1d70 msgr2=0x7fbd5c06d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 -- 172.21.15.59:0/1706664680 shutdown_connections 2024-03-06T07:22:29.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.877+0000 7fbd63a81640 1 -- 172.21.15.59:0/1706664680 wait complete. 2024-03-06T07:22:29.890 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:29.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.992+0000 7f6b26d4b640 1 Processor -- start 2024-03-06T07:22:29.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.992+0000 7f6b26d4b640 1 -- start start 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.993+0000 7f6b26d4b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6b201614d0 0x7f6b200d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.993+0000 7f6b26d4b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6b200d1910 0x7f6b200d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.993+0000 7f6b26d4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b200d22e0 0x7f6b200d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.993+0000 7f6b26d4b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b200d2750 con 0x7f6b200d1910 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.993+0000 7f6b26d4b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b201618b0 con 0x7f6b201614d0 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.993+0000 7f6b26d4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200cf400 con 0x7f6b200d22e0 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.993+0000 7f6b1ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6b200d1910 0x7f6b200d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.993+0000 7f6b1ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6b200d1910 0x7f6b200d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52934/0 (socket says 172.21.15.59:52934) 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.993+0000 7f6b1ffff640 1 -- 172.21.15.59:0/3359809099 learned_addr learned my addr 172.21.15.59:0/3359809099 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.993+0000 7f6b252e8640 1 --2- 172.21.15.59:0/3359809099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b200d22e0 0x7f6b200d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.993+0000 7f6b24ae7640 1 --2- 172.21.15.59:0/3359809099 >> v2:172.21.15.164:3300/0 conn(0x7f6b201614d0 0x7f6b200d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.994+0000 7f6b1ffff640 1 -- 172.21.15.59:0/3359809099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b200d22e0 msgr2=0x7f6b200d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.994+0000 7f6b1ffff640 1 --2- 172.21.15.59:0/3359809099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b200d22e0 0x7f6b200d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.994+0000 7f6b1ffff640 1 -- 172.21.15.59:0/3359809099 >> v2:172.21.15.164:3300/0 conn(0x7f6b201614d0 msgr2=0x7f6b200d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.994+0000 7f6b1ffff640 1 --2- 172.21.15.59:0/3359809099 >> v2:172.21.15.164:3300/0 conn(0x7f6b201614d0 0x7f6b200d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.994+0000 7f6b1ffff640 1 -- 172.21.15.59:0/3359809099 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b200cf6a0 con 0x7f6b200d1910 2024-03-06T07:22:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.994+0000 7f6b1ffff640 1 --2- 172.21.15.59:0/3359809099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b200d1910 0x7f6b200d1da0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6b2008c470 tx=0x7f6b1002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29198c40f4287b3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.994+0000 7f6b1f7fe640 1 -- 172.21.15.59:0/3359809099 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1002ca70 con 0x7f6b200d1910 2024-03-06T07:22:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.994+0000 7f6b1f7fe640 1 -- 172.21.15.59:0/3359809099 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1002cc10 con 0x7f6b200d1910 2024-03-06T07:22:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.994+0000 7f6b1f7fe640 1 -- 172.21.15.59:0/3359809099 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b100343f0 con 0x7f6b200d1910 2024-03-06T07:22:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.995+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3359809099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b200d1910 msgr2=0x7f6b200d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.995+0000 7f6b26d4b640 1 --2- 172.21.15.59:0/3359809099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b200d1910 0x7f6b200d1da0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6b2008c470 tx=0x7f6b1002bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.995+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3359809099 shutdown_connections 2024-03-06T07:22:29.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.995+0000 7f6b26d4b640 1 --2- 172.21.15.59:0/3359809099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b200d22e0 0x7f6b200d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.995+0000 7f6b26d4b640 1 --2- 172.21.15.59:0/3359809099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b200d1910 0x7f6b200d1da0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.995+0000 7f6b26d4b640 1 --2- 172.21.15.59:0/3359809099 >> v2:172.21.15.164:3300/0 conn(0x7f6b201614d0 0x7f6b200d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.995+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3359809099 >> 172.21.15.59:0/3359809099 conn(0x7f6b200d3d40 msgr2=0x7f6b200d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.995+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3359809099 shutdown_connections 2024-03-06T07:22:29.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.995+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3359809099 wait complete. 2024-03-06T07:22:29.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.996+0000 7f6b26d4b640 1 Processor -- start 2024-03-06T07:22:29.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.996+0000 7f6b26d4b640 1 -- start start 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b26d4b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b200d1910 0x7f6b20159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b26d4b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b200d22e0 0x7f6b200c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b26d4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b201614d0 0x7f6b20158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b26d4b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200b0910 con 0x7f6b200d22e0 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b26d4b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200cfc60 con 0x7f6b200d1910 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b26d4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200d0be0 con 0x7f6b201614d0 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b1ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b200d22e0 0x7f6b200c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b1ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b200d22e0 0x7f6b200c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52950/0 (socket says 172.21.15.59:52950) 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b1ffff640 1 -- 172.21.15.59:0/3166383110 learned_addr learned my addr 172.21.15.59:0/3166383110 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b24ae7640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b200d1910 0x7f6b20159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b252e8640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b201614d0 0x7f6b20158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b24ae7640 1 -- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b201614d0 msgr2=0x7f6b20158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b24ae7640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b201614d0 0x7f6b20158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b24ae7640 1 -- 172.21.15.59:0/3166383110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b200d22e0 msgr2=0x7f6b200c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b24ae7640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b200d22e0 0x7f6b200c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b24ae7640 1 -- 172.21.15.59:0/3166383110 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b200c8e40 con 0x7f6b200d1910 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.997+0000 7f6b1ffff640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b200d22e0 0x7f6b200c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.998+0000 7f6b24ae7640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b200d1910 0x7f6b20159570 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800e870 tx=0x7f6b1800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.998+0000 7f6b1d7fa640 1 -- 172.21.15.59:0/3166383110 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18017070 con 0x7f6b200d1910 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.998+0000 7f6b1d7fa640 1 -- 172.21.15.59:0/3166383110 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1800ee60 con 0x7f6b200d1910 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.998+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3166383110 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b200c9ff0 con 0x7f6b200d1910 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.998+0000 7f6b1d7fa640 1 -- 172.21.15.59:0/3166383110 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800cb60 con 0x7f6b200d1910 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.998+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3166383110 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b200ca540 con 0x7f6b200d1910 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.999+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3166383110 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b200b9c00 con 0x7f6b200d1910 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:29.999+0000 7f6b1d7fa640 1 -- 172.21.15.59:0/3166383110 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b1801a070 con 0x7f6b200d1910 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.000+0000 7f6b1d7fa640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b0006b4d0 0x7f6b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.000+0000 7f6b1d7fa640 1 -- 172.21.15.59:0/3166383110 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b18089e20 con 0x7f6b200d1910 2024-03-06T07:22:30.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.000+0000 7f6b1ffff640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b0006b4d0 0x7f6b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.000+0000 7f6b1ffff640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b0006b4d0 0x7f6b0006d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6b200ccde0 tx=0x7f6b10030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.005+0000 7f6b1d7fa640 1 -- 172.21.15.59:0/3166383110 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b180586c0 con 0x7f6b200d1910 2024-03-06T07:22:30.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.186+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3166383110 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6b20068ff0 con 0x7f6b200d1910 2024-03-06T07:22:30.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.186+0000 7f6b1d7fa640 1 -- 172.21.15.59:0/3166383110 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6b1805c760 con 0x7f6b200d1910 2024-03-06T07:22:30.188 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:30.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.188+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b0006b4d0 msgr2=0x7f6b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.188+0000 7f6b26d4b640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b0006b4d0 0x7f6b0006d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6b200ccde0 tx=0x7f6b10030000 comp rx=0 tx=0).stop 2024-03-06T07:22:30.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.188+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b200d1910 msgr2=0x7f6b20159570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.188+0000 7f6b26d4b640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b200d1910 0x7f6b20159570 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800e870 tx=0x7f6b1800ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:30.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.189+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3166383110 shutdown_connections 2024-03-06T07:22:30.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.189+0000 7f6b26d4b640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b0006b4d0 0x7f6b0006d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.189+0000 7f6b26d4b640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b201614d0 0x7f6b20158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.189+0000 7f6b26d4b640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b200d22e0 0x7f6b200c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.189+0000 7f6b26d4b640 1 --2- 172.21.15.59:0/3166383110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b200d1910 0x7f6b20159570 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.189+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3166383110 >> 172.21.15.59:0/3166383110 conn(0x7f6b200d3d40 msgr2=0x7f6b200b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.189+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3166383110 shutdown_connections 2024-03-06T07:22:30.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.189+0000 7f6b26d4b640 1 -- 172.21.15.59:0/3166383110 wait complete. 2024-03-06T07:22:30.201 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:30.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.303+0000 7f88a195e640 1 Processor -- start 2024-03-06T07:22:30.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.303+0000 7f88a195e640 1 -- start start 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f88a195e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f889c1444c0 0x7f889c06ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f88a195e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f889c15b2d0 0x7f889c0708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f88a195e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c06d370 con 0x7f889c070e10 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f88a195e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c1448a0 con 0x7f889c15b2d0 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f88a195e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c071260 con 0x7f889c1444c0 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f889a7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f889c15b2d0 0x7f889c0708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f889a7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f889c15b2d0 0x7f889c0708d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52964/0 (socket says 172.21.15.59:52964) 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f889b7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f889c070e10 0x7f889c15e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:34714/0 (socket says 172.21.15.59:34714) 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f889a7fc640 1 -- 172.21.15.59:0/3772214930 learned_addr learned my addr 172.21.15.59:0/3772214930 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f889affd640 1 --2- 172.21.15.59:0/3772214930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f889c1444c0 0x7f889c06ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f8899ffb640 1 -- 172.21.15.59:0/3772214930 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365487329 0 0) 0x7f889c06d370 con 0x7f889c070e10 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f889a7fc640 1 -- 172.21.15.59:0/3772214930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f889c1444c0 msgr2=0x7f889c06ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f889a7fc640 1 --2- 172.21.15.59:0/3772214930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f889c1444c0 0x7f889c06ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f889a7fc640 1 -- 172.21.15.59:0/3772214930 >> v1:172.21.15.164:6789/0 conn(0x7f889c070e10 legacy=0x7f889c15e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f889a7fc640 1 -- 172.21.15.59:0/3772214930 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c06d650 con 0x7f889c15b2d0 2024-03-06T07:22:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.304+0000 7f889affd640 1 --2- 172.21.15.59:0/3772214930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f889c1444c0 0x7f889c06ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.305+0000 7f889a7fc640 1 --2- 172.21.15.59:0/3772214930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f889c15b2d0 0x7f889c0708d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8890009870 tx=0x7f889002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f1d7d0575a2a23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.305+0000 7f8899ffb640 1 -- 172.21.15.59:0/3772214930 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890034070 con 0x7f889c15b2d0 2024-03-06T07:22:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.305+0000 7f8899ffb640 1 -- 172.21.15.59:0/3772214930 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8890030840 con 0x7f889c15b2d0 2024-03-06T07:22:30.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.305+0000 7f8899ffb640 1 -- 172.21.15.59:0/3772214930 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890030c10 con 0x7f889c15b2d0 2024-03-06T07:22:30.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.306+0000 7f88a195e640 1 -- 172.21.15.59:0/3772214930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f889c15b2d0 msgr2=0x7f889c0708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.306+0000 7f88a195e640 1 --2- 172.21.15.59:0/3772214930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f889c15b2d0 0x7f889c0708d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8890009870 tx=0x7f889002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:30.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.306+0000 7f88a195e640 1 -- 172.21.15.59:0/3772214930 shutdown_connections 2024-03-06T07:22:30.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.306+0000 7f88a195e640 1 --2- 172.21.15.59:0/3772214930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f889c15b2d0 0x7f889c0708d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.306+0000 7f88a195e640 1 --2- 172.21.15.59:0/3772214930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f889c1444c0 0x7f889c06ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.306+0000 7f88a195e640 1 -- 172.21.15.59:0/3772214930 >> 172.21.15.59:0/3772214930 conn(0x7f889c0b1c80 msgr2=0x7f889c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.306+0000 7f88a195e640 1 -- 172.21.15.59:0/3772214930 shutdown_connections 2024-03-06T07:22:30.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.306+0000 7f88a195e640 1 -- 172.21.15.59:0/3772214930 wait complete. 2024-03-06T07:22:30.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.307+0000 7f88a195e640 1 Processor -- start 2024-03-06T07:22:30.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.307+0000 7f88a195e640 1 -- start start 2024-03-06T07:22:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f88a195e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f889c070e10 0x7f889c133f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f88a195e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f889c1444c0 0x7f889c0cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f88a195e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f889c15b2d0 0x7f889c0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f88a195e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c06def0 con 0x7f889c1444c0 2024-03-06T07:22:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f88a195e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c06dc10 con 0x7f889c070e10 2024-03-06T07:22:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f88a195e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c06cf40 con 0x7f889c15b2d0 2024-03-06T07:22:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f889a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f889c1444c0 0x7f889c0cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f889a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f889c1444c0 0x7f889c0cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52966/0 (socket says 172.21.15.59:52966) 2024-03-06T07:22:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f889a7fc640 1 -- 172.21.15.59:0/3230461472 learned_addr learned my addr 172.21.15.59:0/3230461472 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f889affd640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f889c070e10 0x7f889c133f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f889b7fe640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f889c15b2d0 0x7f889c0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f889a7fc640 1 -- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f889c15b2d0 msgr2=0x7f889c0cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f889a7fc640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f889c15b2d0 0x7f889c0cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f889a7fc640 1 -- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f889c070e10 msgr2=0x7f889c133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f889a7fc640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f889c070e10 0x7f889c133f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.308+0000 7f889a7fc640 1 -- 172.21.15.59:0/3230461472 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c134770 con 0x7f889c1444c0 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.309+0000 7f889a7fc640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f889c1444c0 0x7f889c0cd8f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8890000c00 tx=0x7f88900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.309+0000 7f8883fff640 1 -- 172.21.15.59:0/3230461472 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890034040 con 0x7f889c1444c0 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.309+0000 7f8883fff640 1 -- 172.21.15.59:0/3230461472 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889003ab90 con 0x7f889c1444c0 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.309+0000 7f8883fff640 1 -- 172.21.15.59:0/3230461472 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890004030 con 0x7f889c1444c0 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.309+0000 7f88a195e640 1 -- 172.21.15.59:0/3230461472 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889c134a50 con 0x7f889c1444c0 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.309+0000 7f88a195e640 1 -- 172.21.15.59:0/3230461472 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c132110 con 0x7f889c1444c0 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.310+0000 7f88a195e640 1 -- 172.21.15.59:0/3230461472 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889c131f00 con 0x7f889c1444c0 2024-03-06T07:22:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.310+0000 7f8883fff640 1 -- 172.21.15.59:0/3230461472 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f889004b060 con 0x7f889c1444c0 2024-03-06T07:22:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.311+0000 7f8883fff640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 0x7f888806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.311+0000 7f8883fff640 1 -- 172.21.15.59:0/3230461472 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88900abf10 con 0x7f889c1444c0 2024-03-06T07:22:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.311+0000 7f889affd640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 0x7f888806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.311+0000 7f889affd640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 0x7f888806d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f888c009560 tx=0x7f888c009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.315+0000 7f8883fff640 1 -- 172.21.15.59:0/3230461472 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f889003f070 con 0x7f889c1444c0 2024-03-06T07:22:30.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.509+0000 7f88a195e640 1 -- 172.21.15.59:0/3230461472 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f889c04f330 con 0x7f889c1444c0 2024-03-06T07:22:30.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.510+0000 7f8883fff640 1 -- 172.21.15.59:0/3230461472 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f889007a620 con 0x7f889c1444c0 2024-03-06T07:22:30.512 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:22:30.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 -- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 msgr2=0x7f888806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 0x7f888806d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f888c009560 tx=0x7f888c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:30.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 -- 172.21.15.59:0/3230461472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f889c1444c0 msgr2=0x7f889c0cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f889c1444c0 0x7f889c0cd8f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8890000c00 tx=0x7f88900046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 -- 172.21.15.59:0/3230461472 shutdown_connections 2024-03-06T07:22:30.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 0x7f888806d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f889c15b2d0 0x7f889c0cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f889c1444c0 0x7f889c0cd8f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 --2- 172.21.15.59:0/3230461472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f889c070e10 0x7f889c133f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 -- 172.21.15.59:0/3230461472 >> 172.21.15.59:0/3230461472 conn(0x7f889c0b1c80 msgr2=0x7f889c142d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 -- 172.21.15.59:0/3230461472 shutdown_connections 2024-03-06T07:22:30.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.512+0000 7f88a195e640 1 -- 172.21.15.59:0/3230461472 wait complete. 2024-03-06T07:22:30.525 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi164.front.sepia.ceph.com... 2024-03-06T07:22:30.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:30.525 DEBUG:teuthology.orchestra.run.smithi164:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:30.552 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:30.552 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:22:30.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:30.552 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:22:30.611 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:22:30.611 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:30.627 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:22:30.627 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:30.682 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.822+0000 7fbe3dbc7640 1 Processor -- start 2024-03-06T07:22:30.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.822+0000 7fbe3dbc7640 1 -- start start 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.823+0000 7fbe3dbc7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbe3815dfe0 0x7fbe380cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.823+0000 7fbe3dbc7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe380d29c0 0x7fbe380b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.823+0000 7fbe3dbc7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380b3430 con 0x7fbe380cf830 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.823+0000 7fbe3dbc7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe380b31e0 con 0x7fbe3815dfe0 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.823+0000 7fbe3dbc7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380b3d60 con 0x7fbe380d29c0 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.823+0000 7fbe377fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbe3815dfe0 0x7fbe380cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.823+0000 7fbe377fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbe3815dfe0 0x7fbe380cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52982/0 (socket says 172.21.15.59:52982) 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.823+0000 7fbe377fe640 1 -- 172.21.15.59:0/1521305591 learned_addr learned my addr 172.21.15.59:0/1521305591 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.823+0000 7fbe37fff640 1 --2- 172.21.15.59:0/1521305591 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe380d29c0 0x7fbe380b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.824+0000 7fbe377fe640 1 -- 172.21.15.59:0/1521305591 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe380d29c0 msgr2=0x7fbe380b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.824+0000 7fbe377fe640 1 --2- 172.21.15.59:0/1521305591 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe380d29c0 0x7fbe380b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.824+0000 7fbe377fe640 1 -- 172.21.15.59:0/1521305591 >> v1:172.21.15.164:6789/0 conn(0x7fbe380cf830 legacy=0x7fbe380d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.824+0000 7fbe377fe640 1 -- 172.21.15.59:0/1521305591 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe380b3710 con 0x7fbe3815dfe0 2024-03-06T07:22:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.824+0000 7fbe377fe640 1 --2- 172.21.15.59:0/1521305591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe3815dfe0 0x7fbe380cf2f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009870 tx=0x7fbe2002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406a24b4ef784258 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.824+0000 7fbe367fc640 1 -- 172.21.15.59:0/1521305591 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20034070 con 0x7fbe3815dfe0 2024-03-06T07:22:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.825+0000 7fbe367fc640 1 -- 172.21.15.59:0/1521305591 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe20030840 con 0x7fbe3815dfe0 2024-03-06T07:22:30.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.825+0000 7fbe367fc640 1 -- 172.21.15.59:0/1521305591 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20030c10 con 0x7fbe3815dfe0 2024-03-06T07:22:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.825+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/1521305591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe3815dfe0 msgr2=0x7fbe380cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.825+0000 7fbe3dbc7640 1 --2- 172.21.15.59:0/1521305591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe3815dfe0 0x7fbe380cf2f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009870 tx=0x7fbe2002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.825+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/1521305591 shutdown_connections 2024-03-06T07:22:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.825+0000 7fbe3dbc7640 1 --2- 172.21.15.59:0/1521305591 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe380d29c0 0x7fbe380b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.825+0000 7fbe3dbc7640 1 --2- 172.21.15.59:0/1521305591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe3815dfe0 0x7fbe380cf2f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.825+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/1521305591 >> 172.21.15.59:0/1521305591 conn(0x7fbe380d3cd0 msgr2=0x7fbe380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.825+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/1521305591 shutdown_connections 2024-03-06T07:22:30.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.825+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/1521305591 wait complete. 2024-03-06T07:22:30.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.826+0000 7fbe3dbc7640 1 Processor -- start 2024-03-06T07:22:30.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe3dbc7640 1 -- start start 2024-03-06T07:22:30.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe3dbc7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe380cf830 0x7fbe3815ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe3dbc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe380d29c0 0x7fbe380ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe3dbc7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe3815dfe0 0x7fbe380d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe3dbc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380d1df0 con 0x7fbe380d29c0 2024-03-06T07:22:30.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe3dbc7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380d1c70 con 0x7fbe380cf830 2024-03-06T07:22:30.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe3dbc7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3815e3c0 con 0x7fbe3815dfe0 2024-03-06T07:22:30.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe36ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe380d29c0 0x7fbe380ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe36ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe380d29c0 0x7fbe380ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52994/0 (socket says 172.21.15.59:52994) 2024-03-06T07:22:30.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe36ffd640 1 -- 172.21.15.59:0/531258991 learned_addr learned my addr 172.21.15.59:0/531258991 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:30.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe37fff640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe3815dfe0 0x7fbe380d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.827+0000 7fbe377fe640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe380cf830 0x7fbe3815ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe37fff640 1 -- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe380cf830 msgr2=0x7fbe3815ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe37fff640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe380cf830 0x7fbe3815ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe37fff640 1 -- 172.21.15.59:0/531258991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe380d29c0 msgr2=0x7fbe380ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe37fff640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe380d29c0 0x7fbe380ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe37fff640 1 -- 172.21.15.59:0/531258991 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3815b3d0 con 0x7fbe3815dfe0 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe377fe640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe380cf830 0x7fbe3815ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe37fff640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe3815dfe0 0x7fbe380d02c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000efc0 tx=0x7fbe3000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe2ffff640 1 -- 172.21.15.59:0/531258991 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30018020 con 0x7fbe3815dfe0 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe2ffff640 1 -- 172.21.15.59:0/531258991 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe30014480 con 0x7fbe3815dfe0 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe2ffff640 1 -- 172.21.15.59:0/531258991 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30014870 con 0x7fbe3815dfe0 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/531258991 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe380caf80 con 0x7fbe3815dfe0 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.828+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/531258991 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe380cb540 con 0x7fbe3815dfe0 2024-03-06T07:22:30.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.829+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/531258991 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe38068090 con 0x7fbe3815dfe0 2024-03-06T07:22:30.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.830+0000 7fbe2ffff640 1 -- 172.21.15.59:0/531258991 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe30005d10 con 0x7fbe3815dfe0 2024-03-06T07:22:30.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.830+0000 7fbe2ffff640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbe1806b4d0 0x7fbe1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.830+0000 7fbe2ffff640 1 -- 172.21.15.59:0/531258991 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe3008ae60 con 0x7fbe3815dfe0 2024-03-06T07:22:30.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.830+0000 7fbe377fe640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbe1806b4d0 0x7fbe1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.831+0000 7fbe377fe640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbe1806b4d0 0x7fbe1806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe20036000 tx=0x7fbe2003b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:30.834+0000 7fbe2ffff640 1 -- 172.21.15.59:0/531258991 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe38068090 con 0x7fbe3815dfe0 2024-03-06T07:22:31.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.016+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/531258991 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe3804f330 con 0x7fbe3815dfe0 2024-03-06T07:22:31.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.018+0000 7fbe2ffff640 1 -- 172.21.15.59:0/531258991 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fbe30022030 con 0x7fbe3815dfe0 2024-03-06T07:22:31.020 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:31.020 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.019+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbe1806b4d0 msgr2=0x7fbe1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.019+0000 7fbe3dbc7640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbe1806b4d0 0x7fbe1806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe20036000 tx=0x7fbe2003b040 comp rx=0 tx=0).stop 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.019+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe3815dfe0 msgr2=0x7fbe380d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.019+0000 7fbe3dbc7640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe3815dfe0 0x7fbe380d02c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbe3000efc0 tx=0x7fbe3000c490 comp rx=0 tx=0).stop 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.020+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/531258991 shutdown_connections 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.020+0000 7fbe3dbc7640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbe1806b4d0 0x7fbe1806d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.020+0000 7fbe3dbc7640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe3815dfe0 0x7fbe380d02c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.020+0000 7fbe3dbc7640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe380d29c0 0x7fbe380ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.020+0000 7fbe3dbc7640 1 --2- 172.21.15.59:0/531258991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe380cf830 0x7fbe3815ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.020+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/531258991 >> 172.21.15.59:0/531258991 conn(0x7fbe380d3cd0 msgr2=0x7fbe3806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.020+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/531258991 shutdown_connections 2024-03-06T07:22:31.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.020+0000 7fbe3dbc7640 1 -- 172.21.15.59:0/531258991 wait complete. 2024-03-06T07:22:31.032 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:31.033 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:31.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.136+0000 7f3d7c556640 1 Processor -- start 2024-03-06T07:22:31.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.136+0000 7f3d7c556640 1 -- start start 2024-03-06T07:22:31.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.137+0000 7f3d7c556640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3d741586e0 0x7f3d740aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.137+0000 7f3d7c556640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3d740aef80 0x7f3d740cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.137+0000 7f3d7c556640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d74158ac0 con 0x7f3d740cffd0 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.137+0000 7f3d7c556640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7415b2a0 con 0x7f3d740aef80 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.137+0000 7f3d7c556640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7415b540 con 0x7f3d741586e0 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.137+0000 7f3d79af1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3d740aef80 0x7f3d740cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.137+0000 7f3d79af1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3d740aef80 0x7f3d740cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53000/0 (socket says 172.21.15.59:53000) 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.137+0000 7f3d79af1640 1 -- 172.21.15.59:0/2317365617 learned_addr learned my addr 172.21.15.59:0/2317365617 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.137+0000 7f3d7a2f2640 1 --2- 172.21.15.59:0/2317365617 >> v2:172.21.15.164:3300/0 conn(0x7f3d741586e0 0x7f3d740aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.138+0000 7f3d79af1640 1 -- 172.21.15.59:0/2317365617 >> v2:172.21.15.164:3300/0 conn(0x7f3d741586e0 msgr2=0x7f3d740aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.138+0000 7f3d79af1640 1 --2- 172.21.15.59:0/2317365617 >> v2:172.21.15.164:3300/0 conn(0x7f3d741586e0 0x7f3d740aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.138+0000 7f3d79af1640 1 -- 172.21.15.59:0/2317365617 >> v1:172.21.15.164:6789/0 conn(0x7f3d740cffd0 legacy=0x7f3d7406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.138+0000 7f3d79af1640 1 -- 172.21.15.59:0/2317365617 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7415b820 con 0x7f3d740aef80 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.138+0000 7f3d79af1640 1 --2- 172.21.15.59:0/2317365617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d740aef80 0x7f3d740cfa90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3d74070d10 tx=0x7f3d6802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b4faa8ca94dd16d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.139+0000 7f3d792f0640 1 -- 172.21.15.59:0/2317365617 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68034070 con 0x7f3d740aef80 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.139+0000 7f3d792f0640 1 -- 172.21.15.59:0/2317365617 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d68030860 con 0x7f3d740aef80 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.139+0000 7f3d792f0640 1 -- 172.21.15.59:0/2317365617 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68030c30 con 0x7f3d740aef80 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.139+0000 7f3d7c556640 1 -- 172.21.15.59:0/2317365617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d740aef80 msgr2=0x7f3d740cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.139+0000 7f3d7c556640 1 --2- 172.21.15.59:0/2317365617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d740aef80 0x7f3d740cfa90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3d74070d10 tx=0x7f3d6802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.139+0000 7f3d7c556640 1 -- 172.21.15.59:0/2317365617 shutdown_connections 2024-03-06T07:22:31.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.139+0000 7f3d7c556640 1 --2- 172.21.15.59:0/2317365617 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d740aef80 0x7f3d740cfa90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.139+0000 7f3d7c556640 1 --2- 172.21.15.59:0/2317365617 >> v2:172.21.15.164:3300/0 conn(0x7f3d741586e0 0x7f3d740aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.139+0000 7f3d7c556640 1 -- 172.21.15.59:0/2317365617 >> 172.21.15.59:0/2317365617 conn(0x7f3d7406b190 msgr2=0x7f3d7406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.139+0000 7f3d7c556640 1 -- 172.21.15.59:0/2317365617 shutdown_connections 2024-03-06T07:22:31.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.139+0000 7f3d7c556640 1 -- 172.21.15.59:0/2317365617 wait complete. 2024-03-06T07:22:31.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.140+0000 7f3d7c556640 1 Processor -- start 2024-03-06T07:22:31.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.140+0000 7f3d7c556640 1 -- start start 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.141+0000 7f3d7c556640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d740aef80 0x7f3d740a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.141+0000 7f3d7c556640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d740cffd0 0x7f3d74146950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.141+0000 7f3d7c556640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d741586e0 0x7f3d74159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.141+0000 7f3d7c556640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d74071350 con 0x7f3d741586e0 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.141+0000 7f3d7c556640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7415bde0 con 0x7f3d740aef80 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.141+0000 7f3d7c556640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740714d0 con 0x7f3d740cffd0 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.141+0000 7f3d7aaf3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d741586e0 0x7f3d74159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.141+0000 7f3d7aaf3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d741586e0 0x7f3d74159760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53010/0 (socket says 172.21.15.59:53010) 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.141+0000 7f3d7aaf3640 1 -- 172.21.15.59:0/3553131281 learned_addr learned my addr 172.21.15.59:0/3553131281 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:31.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.141+0000 7f3d7a2f2640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d740aef80 0x7f3d740a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.141+0000 7f3d79af1640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d740cffd0 0x7f3d74146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d7aaf3640 1 -- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d740cffd0 msgr2=0x7f3d74146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d7aaf3640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d740cffd0 0x7f3d74146950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d7aaf3640 1 -- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d740aef80 msgr2=0x7f3d740a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d7aaf3640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d740aef80 0x7f3d740a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d7aaf3640 1 -- 172.21.15.59:0/3553131281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d74146e90 con 0x7f3d741586e0 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d7a2f2640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d740aef80 0x7f3d740a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d7aaf3640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d741586e0 0x7f3d74159760 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00c7a0 tx=0x7f3d6c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d66ffd640 1 -- 172.21.15.59:0/3553131281 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c003b70 con 0x7f3d741586e0 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d66ffd640 1 -- 172.21.15.59:0/3553131281 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6c003d30 con 0x7f3d741586e0 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d7c556640 1 -- 172.21.15.59:0/3553131281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d740a9a70 con 0x7f3d741586e0 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d66ffd640 1 -- 172.21.15.59:0/3553131281 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c00aa30 con 0x7f3d741586e0 2024-03-06T07:22:31.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.142+0000 7f3d7c556640 1 -- 172.21.15.59:0/3553131281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d741441c0 con 0x7f3d741586e0 2024-03-06T07:22:31.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.143+0000 7f3d7c556640 1 -- 172.21.15.59:0/3553131281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d740681e0 con 0x7f3d741586e0 2024-03-06T07:22:31.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.144+0000 7f3d66ffd640 1 -- 172.21.15.59:0/3553131281 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d6c00abd0 con 0x7f3d741586e0 2024-03-06T07:22:31.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.144+0000 7f3d66ffd640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d5806b4d0 0x7f3d5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.144+0000 7f3d66ffd640 1 -- 172.21.15.59:0/3553131281 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d6c08a020 con 0x7f3d741586e0 2024-03-06T07:22:31.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.144+0000 7f3d7a2f2640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d5806b4d0 0x7f3d5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.145+0000 7f3d7a2f2640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d5806b4d0 0x7f3d5806d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009e90 tx=0x7f3d70009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.152+0000 7f3d66ffd640 1 -- 172.21.15.59:0/3553131281 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d740681e0 con 0x7f3d741586e0 2024-03-06T07:22:31.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.333+0000 7f3d7c556640 1 -- 172.21.15.59:0/3553131281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3d740732b0 con 0x7f3d741586e0 2024-03-06T07:22:31.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.334+0000 7f3d66ffd640 1 -- 172.21.15.59:0/3553131281 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3d6c0587c0 con 0x7f3d741586e0 2024-03-06T07:22:31.336 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:31.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 -- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d5806b4d0 msgr2=0x7f3d5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d5806b4d0 0x7f3d5806d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009e90 tx=0x7f3d70009190 comp rx=0 tx=0).stop 2024-03-06T07:22:31.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 -- 172.21.15.59:0/3553131281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d741586e0 msgr2=0x7f3d74159760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d741586e0 0x7f3d74159760 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00c7a0 tx=0x7f3d6c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:31.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 -- 172.21.15.59:0/3553131281 shutdown_connections 2024-03-06T07:22:31.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d5806b4d0 0x7f3d5806d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d741586e0 0x7f3d74159760 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d740cffd0 0x7f3d74146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 --2- 172.21.15.59:0/3553131281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d740aef80 0x7f3d740a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 -- 172.21.15.59:0/3553131281 >> 172.21.15.59:0/3553131281 conn(0x7f3d7406b190 msgr2=0x7f3d74156de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 -- 172.21.15.59:0/3553131281 shutdown_connections 2024-03-06T07:22:31.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.336+0000 7f3d7c556640 1 -- 172.21.15.59:0/3553131281 wait complete. 2024-03-06T07:22:31.349 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:31.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.450+0000 7fe55b8f4640 1 Processor -- start 2024-03-06T07:22:31.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.450+0000 7fe55b8f4640 1 -- start start 2024-03-06T07:22:31.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe55b8f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5540d4860 0x7fe5540d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe55b8f4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe5540b0e30 0x7fe5540b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe55b8f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5541618b0 con 0x7fe5541614d0 2024-03-06T07:22:31.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe55b8f4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5540d4cc0 con 0x7fe5540b0e30 2024-03-06T07:22:31.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe55b8f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540cf400 con 0x7fe5540d4860 2024-03-06T07:22:31.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe559e91640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe5540b0e30 0x7fe5540b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe559690640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe5541614d0 0x7fe5540d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47050/0 (socket says 172.21.15.59:47050) 2024-03-06T07:22:31.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe559690640 1 -- 172.21.15.59:0/2868738109 learned_addr learned my addr 172.21.15.59:0/2868738109 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:31.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe558e8f640 1 --2- 172.21.15.59:0/2868738109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5540d4860 0x7fe5540d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe559e91640 1 -- 172.21.15.59:0/2868738109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5540d4860 msgr2=0x7fe5540d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe559e91640 1 --2- 172.21.15.59:0/2868738109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5540d4860 0x7fe5540d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe559e91640 1 -- 172.21.15.59:0/2868738109 >> v1:172.21.15.59:6789/0 conn(0x7fe5541614d0 legacy=0x7fe5540d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe54bfff640 1 -- 172.21.15.59:0/2868738109 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264595481 0 0) 0x7fe5541618b0 con 0x7fe5541614d0 2024-03-06T07:22:31.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe559e91640 1 -- 172.21.15.59:0/2868738109 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5540cf6a0 con 0x7fe5540b0e30 2024-03-06T07:22:31.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.451+0000 7fe558e8f640 1 --2- 172.21.15.59:0/2868738109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5540d4860 0x7fe5540d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:31.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.452+0000 7fe559e91640 1 --2- 172.21.15.59:0/2868738109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5540b0e30 0x7fe5540b12c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009d90 tx=0x7fe54c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75ac9f57599cabb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.452+0000 7fe54bfff640 1 -- 172.21.15.59:0/2868738109 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c039070 con 0x7fe5540b0e30 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.452+0000 7fe54bfff640 1 -- 172.21.15.59:0/2868738109 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54c02cc50 con 0x7fe5540b0e30 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.452+0000 7fe54bfff640 1 -- 172.21.15.59:0/2868738109 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c034440 con 0x7fe5540b0e30 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.453+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2868738109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5540b0e30 msgr2=0x7fe5540b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.453+0000 7fe55b8f4640 1 --2- 172.21.15.59:0/2868738109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5540b0e30 0x7fe5540b12c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009d90 tx=0x7fe54c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.453+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2868738109 shutdown_connections 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.453+0000 7fe55b8f4640 1 --2- 172.21.15.59:0/2868738109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5540b0e30 0x7fe5540b12c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.453+0000 7fe55b8f4640 1 --2- 172.21.15.59:0/2868738109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5540d4860 0x7fe5540d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.453+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2868738109 >> 172.21.15.59:0/2868738109 conn(0x7fe5540d3d40 msgr2=0x7fe5540d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.453+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2868738109 shutdown_connections 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.453+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2868738109 wait complete. 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.454+0000 7fe55b8f4640 1 Processor -- start 2024-03-06T07:22:31.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.454+0000 7fe55b8f4640 1 -- start start 2024-03-06T07:22:31.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe55b8f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5540b0e30 0x7fe5540cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe55b8f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5540d4860 0x7fe5540a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe55b8f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5541614d0 0x7fe5540cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe55b8f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540b0910 con 0x7fe5540b0e30 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe55b8f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540cfc60 con 0x7fe5541614d0 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe55b8f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540d2560 con 0x7fe5540d4860 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe559690640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5540b0e30 0x7fe5540cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe559690640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5540b0e30 0x7fe5540cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53026/0 (socket says 172.21.15.59:53026) 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe559690640 1 -- 172.21.15.59:0/2408819654 learned_addr learned my addr 172.21.15.59:0/2408819654 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe558e8f640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5540d4860 0x7fe5540a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe559e91640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5541614d0 0x7fe5540cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe559690640 1 -- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5540d4860 msgr2=0x7fe5540a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe559690640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5540d4860 0x7fe5540a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe559690640 1 -- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5541614d0 msgr2=0x7fe5540cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe559690640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5541614d0 0x7fe5540cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.455+0000 7fe559690640 1 -- 172.21.15.59:0/2408819654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5540ccf00 con 0x7fe5540b0e30 2024-03-06T07:22:31.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.456+0000 7fe559690640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5540b0e30 0x7fe5540cae10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe55000c430 tx=0x7fe55000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.456+0000 7fe549ffb640 1 -- 172.21.15.59:0/2408819654 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5500074e0 con 0x7fe5540b0e30 2024-03-06T07:22:31.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.456+0000 7fe549ffb640 1 -- 172.21.15.59:0/2408819654 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5500047e0 con 0x7fe5540b0e30 2024-03-06T07:22:31.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.456+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2408819654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5540cbd70 con 0x7fe5540b0e30 2024-03-06T07:22:31.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.456+0000 7fe549ffb640 1 -- 172.21.15.59:0/2408819654 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe550004bd0 con 0x7fe5540b0e30 2024-03-06T07:22:31.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.456+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2408819654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5540cb7b0 con 0x7fe5540b0e30 2024-03-06T07:22:31.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.457+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2408819654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5540cbb50 con 0x7fe5540b0e30 2024-03-06T07:22:31.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.458+0000 7fe549ffb640 1 -- 172.21.15.59:0/2408819654 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe550011070 con 0x7fe5540b0e30 2024-03-06T07:22:31.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.458+0000 7fe549ffb640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe53806b4d0 0x7fe53806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.458+0000 7fe549ffb640 1 -- 172.21.15.59:0/2408819654 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe550089d90 con 0x7fe5540b0e30 2024-03-06T07:22:31.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.458+0000 7fe558e8f640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe53806b4d0 0x7fe53806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.459+0000 7fe558e8f640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe53806b4d0 0x7fe53806d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe544009560 tx=0x7fe544009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.462+0000 7fe549ffb640 1 -- 172.21.15.59:0/2408819654 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe550058530 con 0x7fe5540b0e30 2024-03-06T07:22:31.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.645+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2408819654 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe5540caad0 con 0x7fe5540b0e30 2024-03-06T07:22:31.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.645+0000 7fe549ffb640 1 -- 172.21.15.59:0/2408819654 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe55005c5d0 con 0x7fe5540b0e30 2024-03-06T07:22:31.649 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:31.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe53806b4d0 msgr2=0x7fe53806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe53806b4d0 0x7fe53806d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe544009560 tx=0x7fe544009190 comp rx=0 tx=0).stop 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2408819654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5540b0e30 msgr2=0x7fe5540cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5540b0e30 0x7fe5540cae10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe55000c430 tx=0x7fe55000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2408819654 shutdown_connections 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe53806b4d0 0x7fe53806d990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5541614d0 0x7fe5540cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5540d4860 0x7fe5540a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 --2- 172.21.15.59:0/2408819654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5540b0e30 0x7fe5540cae10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2408819654 >> 172.21.15.59:0/2408819654 conn(0x7fe5540d3d40 msgr2=0x7fe5540b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2408819654 shutdown_connections 2024-03-06T07:22:31.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.648+0000 7fe55b8f4640 1 -- 172.21.15.59:0/2408819654 wait complete. 2024-03-06T07:22:31.661 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:31.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.763+0000 7f016cf65640 1 Processor -- start 2024-03-06T07:22:31.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.763+0000 7f016cf65640 1 -- start start 2024-03-06T07:22:31.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f016cf65640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01680cfa90 0x7f01680cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f016cf65640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f01680b33a0 0x7f01680b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f016cf65640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0168158ac0 con 0x7f01681586e0 2024-03-06T07:22:31.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f016cf65640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016815b2a0 con 0x7f01680cfa90 2024-03-06T07:22:31.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f016cf65640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016815b540 con 0x7f01680b33a0 2024-03-06T07:22:31.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f0165d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01680cfa90 0x7f01680cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f0165d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01680cfa90 0x7f01680cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53030/0 (socket says 172.21.15.59:53030) 2024-03-06T07:22:31.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f0165d9b640 1 -- 172.21.15.59:0/3195067974 learned_addr learned my addr 172.21.15.59:0/3195067974 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:31.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f0166d9d640 1 --2- 172.21.15.59:0/3195067974 >> v2:172.21.15.164:3300/0 conn(0x7f01680b33a0 0x7f01680b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f0165d9b640 1 -- 172.21.15.59:0/3195067974 >> v2:172.21.15.164:3300/0 conn(0x7f01680b33a0 msgr2=0x7f01680b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f0165d9b640 1 --2- 172.21.15.59:0/3195067974 >> v2:172.21.15.164:3300/0 conn(0x7f01680b33a0 0x7f01680b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f0165d9b640 1 -- 172.21.15.59:0/3195067974 >> v1:172.21.15.164:6789/0 conn(0x7f01681586e0 legacy=0x7f01680aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f0165d9b640 1 -- 172.21.15.59:0/3195067974 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016815b820 con 0x7f01680cfa90 2024-03-06T07:22:31.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f0165d9b640 1 --2- 172.21.15.59:0/3195067974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01680cfa90 0x7f01680cff40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f01680b2ae0 tx=0x7f015002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d1b272d4f183e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f016559a640 1 -- 172.21.15.59:0/3195067974 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150034070 con 0x7f01680cfa90 2024-03-06T07:22:31.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f016559a640 1 -- 172.21.15.59:0/3195067974 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0150030860 con 0x7f01680cfa90 2024-03-06T07:22:31.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.765+0000 7f016559a640 1 -- 172.21.15.59:0/3195067974 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150030c30 con 0x7f01680cfa90 2024-03-06T07:22:31.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.766+0000 7f016cf65640 1 -- 172.21.15.59:0/3195067974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01680cfa90 msgr2=0x7f01680cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.766+0000 7f016cf65640 1 --2- 172.21.15.59:0/3195067974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01680cfa90 0x7f01680cff40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f01680b2ae0 tx=0x7f015002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.766+0000 7f016cf65640 1 -- 172.21.15.59:0/3195067974 shutdown_connections 2024-03-06T07:22:31.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.766+0000 7f016cf65640 1 --2- 172.21.15.59:0/3195067974 >> v2:172.21.15.164:3300/0 conn(0x7f01680b33a0 0x7f01680b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.766+0000 7f016cf65640 1 --2- 172.21.15.59:0/3195067974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01680cfa90 0x7f01680cff40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.766+0000 7f016cf65640 1 -- 172.21.15.59:0/3195067974 >> 172.21.15.59:0/3195067974 conn(0x7f016806b190 msgr2=0x7f016806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.766+0000 7f016cf65640 1 -- 172.21.15.59:0/3195067974 shutdown_connections 2024-03-06T07:22:31.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.766+0000 7f016cf65640 1 -- 172.21.15.59:0/3195067974 wait complete. 2024-03-06T07:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.767+0000 7f016cf65640 1 Processor -- start 2024-03-06T07:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.767+0000 7f016cf65640 1 -- start start 2024-03-06T07:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f016cf65640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01680b33a0 0x7f01680aa520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f016cf65640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01680cfa90 0x7f01680b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f016cf65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01681586e0 0x7f01680ce030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f016cf65640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01680af150 con 0x7f01681586e0 2024-03-06T07:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f016cf65640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016815bde0 con 0x7f01680cfa90 2024-03-06T07:22:31.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f016cf65640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0168071350 con 0x7f01680b33a0 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f0166d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01681586e0 0x7f01680ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f0166d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01681586e0 0x7f01680ce030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53038/0 (socket says 172.21.15.59:53038) 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f0166d9d640 1 -- 172.21.15.59:0/4094995255 learned_addr learned my addr 172.21.15.59:0/4094995255 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f0165d9b640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01680cfa90 0x7f01680b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f016659c640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01680b33a0 0x7f01680aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f0166d9d640 1 -- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01680b33a0 msgr2=0x7f01680aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f0166d9d640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01680b33a0 0x7f01680aa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f0166d9d640 1 -- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01680cfa90 msgr2=0x7f01680b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f0166d9d640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01680cfa90 0x7f01680b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.768+0000 7f0166d9d640 1 -- 172.21.15.59:0/4094995255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016815ab00 con 0x7f01681586e0 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.769+0000 7f0166d9d640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01681586e0 0x7f01680ce030 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f015800efc0 tx=0x7f015800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.769+0000 7f015effd640 1 -- 172.21.15.59:0/4094995255 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158018020 con 0x7f01681586e0 2024-03-06T07:22:31.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.769+0000 7f015effd640 1 -- 172.21.15.59:0/4094995255 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0158014480 con 0x7f01681586e0 2024-03-06T07:22:31.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.769+0000 7f016cf65640 1 -- 172.21.15.59:0/4094995255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0168159e30 con 0x7f01681586e0 2024-03-06T07:22:31.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.769+0000 7f015effd640 1 -- 172.21.15.59:0/4094995255 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158014870 con 0x7f01681586e0 2024-03-06T07:22:31.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.769+0000 7f016cf65640 1 -- 172.21.15.59:0/4094995255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01680aaa60 con 0x7f01681586e0 2024-03-06T07:22:31.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.770+0000 7f016cf65640 1 -- 172.21.15.59:0/4094995255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01680732b0 con 0x7f01681586e0 2024-03-06T07:22:31.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.770+0000 7f015effd640 1 -- 172.21.15.59:0/4094995255 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0158005d10 con 0x7f01681586e0 2024-03-06T07:22:31.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.770+0000 7f015effd640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f014806b4d0 0x7f014806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.771+0000 7f015effd640 1 -- 172.21.15.59:0/4094995255 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f015808ab70 con 0x7f01681586e0 2024-03-06T07:22:31.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.771+0000 7f016659c640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f014806b4d0 0x7f014806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.771+0000 7f016659c640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f014806b4d0 0x7f014806d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f016814fb10 tx=0x7f0160009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.775+0000 7f015effd640 1 -- 172.21.15.59:0/4094995255 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0158022070 con 0x7f01681586e0 2024-03-06T07:22:31.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.956+0000 7f016cf65640 1 -- 172.21.15.59:0/4094995255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0168068ff0 con 0x7f01681586e0 2024-03-06T07:22:31.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.957+0000 7f015effd640 1 -- 172.21.15.59:0/4094995255 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0158010040 con 0x7f01681586e0 2024-03-06T07:22:31.960 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:31.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 -- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f014806b4d0 msgr2=0x7f014806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f014806b4d0 0x7f014806d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f016814fb10 tx=0x7f0160009040 comp rx=0 tx=0).stop 2024-03-06T07:22:31.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 -- 172.21.15.59:0/4094995255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01681586e0 msgr2=0x7f01680ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01681586e0 0x7f01680ce030 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f015800efc0 tx=0x7f015800c490 comp rx=0 tx=0).stop 2024-03-06T07:22:31.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 -- 172.21.15.59:0/4094995255 shutdown_connections 2024-03-06T07:22:31.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f014806b4d0 0x7f014806d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01681586e0 0x7f01680ce030 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01680cfa90 0x7f01680b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 --2- 172.21.15.59:0/4094995255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01680b33a0 0x7f01680aa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 -- 172.21.15.59:0/4094995255 >> 172.21.15.59:0/4094995255 conn(0x7f016806b190 msgr2=0x7f016806e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 -- 172.21.15.59:0/4094995255 shutdown_connections 2024-03-06T07:22:31.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:31.959+0000 7f016cf65640 1 -- 172.21.15.59:0/4094995255 wait complete. 2024-03-06T07:22:31.973 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:32.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.072+0000 7f5075866640 1 Processor -- start 2024-03-06T07:22:32.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.072+0000 7f5075866640 1 -- start start 2024-03-06T07:22:32.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.072+0000 7f5075866640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f50700b4460 0x7f50700d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.072+0000 7f5075866640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5070162970 0x7f50700b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f5075866640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50700d22c0 con 0x7f5070161cf0 2024-03-06T07:22:32.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f5075866640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5070161b00 con 0x7f50700b4460 2024-03-06T07:22:32.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f5075866640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50700cecc0 con 0x7f5070162970 2024-03-06T07:22:32.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f506effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f50700b4460 0x7f50700d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f506f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f5070161cf0 0x7f50700cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47054/0 (socket says 172.21.15.59:47054) 2024-03-06T07:22:32.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f506f7fe640 1 -- 172.21.15.59:0/4210201767 learned_addr learned my addr 172.21.15.59:0/4210201767 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:32.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f5067fff640 1 --2- 172.21.15.59:0/4210201767 >> v2:172.21.15.164:3300/0 conn(0x7f5070162970 0x7f50700b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f506effd640 1 -- 172.21.15.59:0/4210201767 >> v2:172.21.15.164:3300/0 conn(0x7f5070162970 msgr2=0x7f50700b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f506effd640 1 --2- 172.21.15.59:0/4210201767 >> v2:172.21.15.164:3300/0 conn(0x7f5070162970 0x7f50700b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f506effd640 1 -- 172.21.15.59:0/4210201767 >> v1:172.21.15.59:6789/0 conn(0x7f5070161cf0 legacy=0x7f50700cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f506e7fc640 1 -- 172.21.15.59:0/4210201767 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943437928 0 0) 0x7f50700d22c0 con 0x7f5070161cf0 2024-03-06T07:22:32.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.073+0000 7f506effd640 1 -- 172.21.15.59:0/4210201767 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50700d25a0 con 0x7f50700b4460 2024-03-06T07:22:32.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.074+0000 7f506effd640 1 --2- 172.21.15.59:0/4210201767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50700b4460 0x7f50700d4800 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f50700d17e0 tx=0x7f506002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc625c4209bef80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.074+0000 7f506e7fc640 1 -- 172.21.15.59:0/4210201767 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060035020 con 0x7f50700b4460 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.074+0000 7f506e7fc640 1 -- 172.21.15.59:0/4210201767 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5060031830 con 0x7f50700b4460 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.074+0000 7f506e7fc640 1 -- 172.21.15.59:0/4210201767 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060031c00 con 0x7f50700b4460 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.074+0000 7f5075866640 1 -- 172.21.15.59:0/4210201767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50700b4460 msgr2=0x7f50700d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.074+0000 7f5075866640 1 --2- 172.21.15.59:0/4210201767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50700b4460 0x7f50700d4800 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f50700d17e0 tx=0x7f506002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.074+0000 7f5075866640 1 -- 172.21.15.59:0/4210201767 shutdown_connections 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.074+0000 7f5075866640 1 --2- 172.21.15.59:0/4210201767 >> v2:172.21.15.164:3300/0 conn(0x7f5070162970 0x7f50700b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.074+0000 7f5075866640 1 --2- 172.21.15.59:0/4210201767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50700b4460 0x7f50700d4800 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.074+0000 7f5075866640 1 -- 172.21.15.59:0/4210201767 >> 172.21.15.59:0/4210201767 conn(0x7f50700d3ce0 msgr2=0x7f50700d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.075+0000 7f5075866640 1 -- 172.21.15.59:0/4210201767 shutdown_connections 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.075+0000 7f5075866640 1 -- 172.21.15.59:0/4210201767 wait complete. 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.076+0000 7f5075866640 1 Processor -- start 2024-03-06T07:22:32.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.076+0000 7f5075866640 1 -- start start 2024-03-06T07:22:32.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.076+0000 7f5075866640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50700b4460 0x7f50700cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.076+0000 7f5075866640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5070161cf0 0x7f50700c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.076+0000 7f5075866640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5070162970 0x7f50700b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.076+0000 7f5075866640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50700ce950 con 0x7f5070161cf0 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.076+0000 7f5075866640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50700d2b60 con 0x7f5070162970 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.076+0000 7f5075866640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50700b3ac0 con 0x7f50700b4460 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.076+0000 7f5067fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5070161cf0 0x7f50700c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.076+0000 7f5067fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5070161cf0 0x7f50700c8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53060/0 (socket says 172.21.15.59:53060) 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.076+0000 7f5067fff640 1 -- 172.21.15.59:0/1369475937 learned_addr learned my addr 172.21.15.59:0/1369475937 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f506f7fe640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5070162970 0x7f50700b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f506effd640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50700b4460 0x7f50700cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f5067fff640 1 -- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50700b4460 msgr2=0x7f50700cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f5067fff640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50700b4460 0x7f50700cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f5067fff640 1 -- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5070162970 msgr2=0x7f50700b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f5067fff640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5070162970 0x7f50700b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f5067fff640 1 -- 172.21.15.59:0/1369475937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50700c9610 con 0x7f5070161cf0 2024-03-06T07:22:32.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f5067fff640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5070161cf0 0x7f50700c8ea0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f505800e980 tx=0x7f505800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f506cff9640 1 -- 172.21.15.59:0/1369475937 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058017070 con 0x7f5070161cf0 2024-03-06T07:22:32.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f506cff9640 1 -- 172.21.15.59:0/1369475937 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505800ca00 con 0x7f5070161cf0 2024-03-06T07:22:32.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f506cff9640 1 -- 172.21.15.59:0/1369475937 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505800cdd0 con 0x7f5070161cf0 2024-03-06T07:22:32.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f5075866640 1 -- 172.21.15.59:0/1369475937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50700c98f0 con 0x7f5070161cf0 2024-03-06T07:22:32.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.077+0000 7f5075866640 1 -- 172.21.15.59:0/1369475937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50700c9eb0 con 0x7f5070161cf0 2024-03-06T07:22:32.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.078+0000 7f5075866640 1 -- 172.21.15.59:0/1369475937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507004f330 con 0x7f5070161cf0 2024-03-06T07:22:32.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.079+0000 7f506cff9640 1 -- 172.21.15.59:0/1369475937 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f505801a070 con 0x7f5070161cf0 2024-03-06T07:22:32.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.079+0000 7f506cff9640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f505006b4d0 0x7f505006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.079+0000 7f506cff9640 1 -- 172.21.15.59:0/1369475937 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f505808a540 con 0x7f5070161cf0 2024-03-06T07:22:32.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.079+0000 7f506effd640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f505006b4d0 0x7f505006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.080+0000 7f506effd640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f505006b4d0 0x7f505006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5060036000 tx=0x7f506003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.084+0000 7f506cff9640 1 -- 172.21.15.59:0/1369475937 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5058058d70 con 0x7f5070161cf0 2024-03-06T07:22:32.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.277+0000 7f5075866640 1 -- 172.21.15.59:0/1369475937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f507006b180 con 0x7f5070161cf0 2024-03-06T07:22:32.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.277+0000 7f506cff9640 1 -- 172.21.15.59:0/1369475937 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f505805f0c0 con 0x7f5070161cf0 2024-03-06T07:22:32.280 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:22:32.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 -- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f505006b4d0 msgr2=0x7f505006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f505006b4d0 0x7f505006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5060036000 tx=0x7f506003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:32.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 -- 172.21.15.59:0/1369475937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5070161cf0 msgr2=0x7f50700c8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5070161cf0 0x7f50700c8ea0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f505800e980 tx=0x7f505800ee40 comp rx=0 tx=0).stop 2024-03-06T07:22:32.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 -- 172.21.15.59:0/1369475937 shutdown_connections 2024-03-06T07:22:32.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f505006b4d0 0x7f505006d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5070162970 0x7f50700b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5070161cf0 0x7f50700c8ea0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 --2- 172.21.15.59:0/1369475937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50700b4460 0x7f50700cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 -- 172.21.15.59:0/1369475937 >> 172.21.15.59:0/1369475937 conn(0x7f50700d3ce0 msgr2=0x7f50700d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 -- 172.21.15.59:0/1369475937 shutdown_connections 2024-03-06T07:22:32.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.279+0000 7f5075866640 1 -- 172.21.15.59:0/1369475937 wait complete. 2024-03-06T07:22:32.293 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:22:32.293 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:32.308 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:32.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f69208c3640 1 Processor -- start 2024-03-06T07:22:32.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f69208c3640 1 -- start start 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f69208c3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f691c06b220 0x7f691c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f69208c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c06b600 0x7f691c162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f69208c3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c15ea30 con 0x7f691c0ce920 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f69208c3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691c161b70 con 0x7f691c06b220 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f69208c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c15ed10 con 0x7f691c06b600 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f691a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f691c06b220 0x7f691c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f691a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f691c06b220 0x7f691c15e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53068/0 (socket says 172.21.15.59:53068) 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f691a59c640 1 -- 172.21.15.59:0/1604553208 learned_addr learned my addr 172.21.15.59:0/1604553208 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f691ad9d640 1 --2- 172.21.15.59:0/1604553208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c06b600 0x7f691c162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f691a59c640 1 -- 172.21.15.59:0/1604553208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c06b600 msgr2=0x7f691c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f691a59c640 1 --2- 172.21.15.59:0/1604553208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c06b600 0x7f691c162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f691a59c640 1 -- 172.21.15.59:0/1604553208 >> v1:172.21.15.164:6789/0 conn(0x7f691c0ce920 legacy=0x7f691c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.448+0000 7f691a59c640 1 -- 172.21.15.59:0/1604553208 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c15eff0 con 0x7f691c06b220 2024-03-06T07:22:32.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.449+0000 7f691a59c640 1 --2- 172.21.15.59:0/1604553208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c06b220 0x7f691c15e450 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6914009870 tx=0x7f691402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c64e9ccd8e2102 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.449+0000 7f691959a640 1 -- 172.21.15.59:0/1604553208 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6914035020 con 0x7f691c06b220 2024-03-06T07:22:32.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.449+0000 7f691959a640 1 -- 172.21.15.59:0/1604553208 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6914031640 con 0x7f691c06b220 2024-03-06T07:22:32.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.449+0000 7f691959a640 1 -- 172.21.15.59:0/1604553208 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691402d040 con 0x7f691c06b220 2024-03-06T07:22:32.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.450+0000 7f69208c3640 1 -- 172.21.15.59:0/1604553208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c06b220 msgr2=0x7f691c15e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.450+0000 7f69208c3640 1 --2- 172.21.15.59:0/1604553208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c06b220 0x7f691c15e450 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6914009870 tx=0x7f691402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.450+0000 7f69208c3640 1 -- 172.21.15.59:0/1604553208 shutdown_connections 2024-03-06T07:22:32.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.450+0000 7f69208c3640 1 --2- 172.21.15.59:0/1604553208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c06b600 0x7f691c162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.450+0000 7f69208c3640 1 --2- 172.21.15.59:0/1604553208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c06b220 0x7f691c15e450 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.450+0000 7f69208c3640 1 -- 172.21.15.59:0/1604553208 >> 172.21.15.59:0/1604553208 conn(0x7f691c06c0c0 msgr2=0x7f691c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.450+0000 7f69208c3640 1 -- 172.21.15.59:0/1604553208 shutdown_connections 2024-03-06T07:22:32.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.450+0000 7f69208c3640 1 -- 172.21.15.59:0/1604553208 wait complete. 2024-03-06T07:22:32.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.451+0000 7f69208c3640 1 Processor -- start 2024-03-06T07:22:32.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.451+0000 7f69208c3640 1 -- start start 2024-03-06T07:22:32.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f69208c3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c06b220 0x7f691c0ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f69208c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c06b600 0x7f691c1e8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f69208c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c0ce920 0x7f691c0b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f69208c3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c072090 con 0x7f691c06b600 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f69208c3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0ce200 con 0x7f691c06b220 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f69208c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c1e8a80 con 0x7f691c0ce920 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f6919d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c06b600 0x7f691c1e8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f6919d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c06b600 0x7f691c1e8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53082/0 (socket says 172.21.15.59:53082) 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f6919d9b640 1 -- 172.21.15.59:0/4148696944 learned_addr learned my addr 172.21.15.59:0/4148696944 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f691a59c640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c06b220 0x7f691c0ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f691ad9d640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c0ce920 0x7f691c0b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f6919d9b640 1 -- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c0ce920 msgr2=0x7f691c0b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f6919d9b640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c0ce920 0x7f691c0b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f6919d9b640 1 -- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c06b220 msgr2=0x7f691c0ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f6919d9b640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c06b220 0x7f691c0ccf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f6919d9b640 1 -- 172.21.15.59:0/4148696944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c1e8d60 con 0x7f691c06b600 2024-03-06T07:22:32.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f6919d9b640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c06b600 0x7f691c1e8220 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f691000c370 tx=0x7f691000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f690affd640 1 -- 172.21.15.59:0/4148696944 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910017070 con 0x7f691c06b600 2024-03-06T07:22:32.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f690affd640 1 -- 172.21.15.59:0/4148696944 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69100052d0 con 0x7f691c06b600 2024-03-06T07:22:32.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f69208c3640 1 -- 172.21.15.59:0/4148696944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c1e9040 con 0x7f691c06b600 2024-03-06T07:22:32.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.452+0000 7f690affd640 1 -- 172.21.15.59:0/4148696944 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6910003850 con 0x7f691c06b600 2024-03-06T07:22:32.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.453+0000 7f69208c3640 1 -- 172.21.15.59:0/4148696944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c1e8760 con 0x7f691c06b600 2024-03-06T07:22:32.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.454+0000 7f69208c3640 1 -- 172.21.15.59:0/4148696944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c04f330 con 0x7f691c06b600 2024-03-06T07:22:32.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.454+0000 7f690affd640 1 -- 172.21.15.59:0/4148696944 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f691001a070 con 0x7f691c06b600 2024-03-06T07:22:32.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.454+0000 7f690affd640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f68fc06b4d0 0x7f68fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.455+0000 7f690affd640 1 -- 172.21.15.59:0/4148696944 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69100894a0 con 0x7f691c06b600 2024-03-06T07:22:32.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.455+0000 7f691a59c640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f68fc06b4d0 0x7f68fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.455+0000 7f691a59c640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f68fc06b4d0 0x7f68fc06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6914036000 tx=0x7f691403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.459+0000 7f690affd640 1 -- 172.21.15.59:0/4148696944 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6910057c50 con 0x7f691c06b600 2024-03-06T07:22:32.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.640+0000 7f69208c3640 1 -- 172.21.15.59:0/4148696944 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f691c1e9210 con 0x7f691c06b600 2024-03-06T07:22:32.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.641+0000 7f690affd640 1 -- 172.21.15.59:0/4148696944 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f691005bcf0 con 0x7f691c06b600 2024-03-06T07:22:32.644 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:32.644 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:32.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.643+0000 7f69208c3640 1 -- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f68fc06b4d0 msgr2=0x7f68fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.643+0000 7f69208c3640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f68fc06b4d0 0x7f68fc06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6914036000 tx=0x7f691403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:32.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.643+0000 7f69208c3640 1 -- 172.21.15.59:0/4148696944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c06b600 msgr2=0x7f691c1e8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.643+0000 7f69208c3640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c06b600 0x7f691c1e8220 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f691000c370 tx=0x7f691000c830 comp rx=0 tx=0).stop 2024-03-06T07:22:32.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.643+0000 7f69208c3640 1 -- 172.21.15.59:0/4148696944 shutdown_connections 2024-03-06T07:22:32.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.643+0000 7f69208c3640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f68fc06b4d0 0x7f68fc06d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.643+0000 7f69208c3640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c0ce920 0x7f691c0b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.643+0000 7f69208c3640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c06b600 0x7f691c1e8220 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.643+0000 7f69208c3640 1 --2- 172.21.15.59:0/4148696944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c06b220 0x7f691c0ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.643+0000 7f69208c3640 1 -- 172.21.15.59:0/4148696944 >> 172.21.15.59:0/4148696944 conn(0x7f691c06c0c0 msgr2=0x7f691c15c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.644+0000 7f69208c3640 1 -- 172.21.15.59:0/4148696944 shutdown_connections 2024-03-06T07:22:32.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.644+0000 7f69208c3640 1 -- 172.21.15.59:0/4148696944 wait complete. 2024-03-06T07:22:32.657 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:32.657 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.758+0000 7f8e02a88640 1 Processor -- start 2024-03-06T07:22:32.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.758+0000 7f8e02a88640 1 -- start start 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8e02a88640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8dfc0d4860 0x7f8dfc0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8e02a88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dfc0b0e30 0x7f8dfc0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8e02a88640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc1618b0 con 0x7f8dfc1614d0 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8e02a88640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dfc0d4cf0 con 0x7f8dfc0d4860 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8e02a88640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0cf400 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8dfbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8dfc0d4860 0x7f8dfc0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8e00824640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8dfc1614d0 0x7f8dfc0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47068/0 (socket says 172.21.15.59:47068) 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8e00824640 1 -- 172.21.15.59:0/1382018032 learned_addr learned my addr 172.21.15.59:0/1382018032 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8e01025640 1 --2- 172.21.15.59:0/1382018032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dfc0b0e30 0x7f8dfc0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8dfbfff640 1 -- 172.21.15.59:0/1382018032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dfc0b0e30 msgr2=0x7f8dfc0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8dfbfff640 1 --2- 172.21.15.59:0/1382018032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dfc0b0e30 0x7f8dfc0b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8dfbfff640 1 -- 172.21.15.59:0/1382018032 >> v1:172.21.15.59:6789/0 conn(0x7f8dfc1614d0 legacy=0x7f8dfc0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8dfbfff640 1 -- 172.21.15.59:0/1382018032 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc0cf6a0 con 0x7f8dfc0d4860 2024-03-06T07:22:32.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8dfb7fe640 1 -- 172.21.15.59:0/1382018032 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459504817 0 0) 0x7f8dfc1618b0 con 0x7f8dfc1614d0 2024-03-06T07:22:32.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8e01025640 1 --2- 172.21.15.59:0/1382018032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dfc0b0e30 0x7f8dfc0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:32.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.760+0000 7f8dfbfff640 1 --2- 172.21.15.59:0/1382018032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dfc0d4860 0x7f8dfc0d2020 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8df00097a0 tx=0x7f8df002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b4b3ae0031b4fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.761+0000 7f8dfb7fe640 1 -- 172.21.15.59:0/1382018032 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df0035020 con 0x7f8dfc0d4860 2024-03-06T07:22:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.761+0000 7f8dfb7fe640 1 -- 172.21.15.59:0/1382018032 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df0031830 con 0x7f8dfc0d4860 2024-03-06T07:22:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.761+0000 7f8dfb7fe640 1 -- 172.21.15.59:0/1382018032 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df0031c00 con 0x7f8dfc0d4860 2024-03-06T07:22:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.761+0000 7f8e02a88640 1 -- 172.21.15.59:0/1382018032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dfc0d4860 msgr2=0x7f8dfc0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.761+0000 7f8e02a88640 1 --2- 172.21.15.59:0/1382018032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dfc0d4860 0x7f8dfc0d2020 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8df00097a0 tx=0x7f8df002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.761+0000 7f8e02a88640 1 -- 172.21.15.59:0/1382018032 shutdown_connections 2024-03-06T07:22:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.761+0000 7f8e02a88640 1 --2- 172.21.15.59:0/1382018032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dfc0b0e30 0x7f8dfc0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.761+0000 7f8e02a88640 1 --2- 172.21.15.59:0/1382018032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dfc0d4860 0x7f8dfc0d2020 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.761+0000 7f8e02a88640 1 -- 172.21.15.59:0/1382018032 >> 172.21.15.59:0/1382018032 conn(0x7f8dfc0d3d40 msgr2=0x7f8dfc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.761+0000 7f8e02a88640 1 -- 172.21.15.59:0/1382018032 shutdown_connections 2024-03-06T07:22:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.761+0000 7f8e02a88640 1 -- 172.21.15.59:0/1382018032 wait complete. 2024-03-06T07:22:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.762+0000 7f8e02a88640 1 Processor -- start 2024-03-06T07:22:32.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.762+0000 7f8e02a88640 1 -- start start 2024-03-06T07:22:32.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.763+0000 7f8e02a88640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dfc0b0e30 0x7f8dfc0c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.763+0000 7f8e02a88640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8dfc0d4860 0x7f8dfc1f7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.763+0000 7f8e02a88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dfc1614d0 0x7f8dfc0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.763+0000 7f8e02a88640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0b0910 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.763+0000 7f8e02a88640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0cfc60 con 0x7f8dfc0d4860 2024-03-06T07:22:32.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.763+0000 7f8e02a88640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc0d2560 con 0x7f8dfc1614d0 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.763+0000 7f8e00824640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dfc0b0e30 0x7f8dfc0c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.763+0000 7f8e00824640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dfc0b0e30 0x7f8dfc0c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53090/0 (socket says 172.21.15.59:53090) 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.763+0000 7f8e00824640 1 -- 172.21.15.59:0/4015568497 learned_addr learned my addr 172.21.15.59:0/4015568497 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.763+0000 7f8dfbfff640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8dfc0d4860 0x7f8dfc1f7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.763+0000 7f8e01025640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dfc1614d0 0x7f8dfc0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.764+0000 7f8e00824640 1 -- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dfc1614d0 msgr2=0x7f8dfc0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.764+0000 7f8e00824640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dfc1614d0 0x7f8dfc0cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.764+0000 7f8e00824640 1 -- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8dfc0d4860 msgr2=0x7f8dfc1f7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.764+0000 7f8e00824640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8dfc0d4860 0x7f8dfc1f7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.764+0000 7f8e00824640 1 -- 172.21.15.59:0/4015568497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc0c8470 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.764+0000 7f8e00824640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dfc0b0e30 0x7f8dfc0c96e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8df400c7c0 tx=0x7f8df400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.764+0000 7f8df97fa640 1 -- 172.21.15.59:0/4015568497 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4018070 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.764+0000 7f8df97fa640 1 -- 172.21.15.59:0/4015568497 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df400a640 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.764+0000 7f8e02a88640 1 -- 172.21.15.59:0/4015568497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dfc0c8750 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.764+0000 7f8df97fa640 1 -- 172.21.15.59:0/4015568497 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df400aa30 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.764+0000 7f8e02a88640 1 -- 172.21.15.59:0/4015568497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dfc1f7690 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.766+0000 7f8e02a88640 1 -- 172.21.15.59:0/4015568497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dfc0c8ab0 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.766+0000 7f8df97fa640 1 -- 172.21.15.59:0/4015568497 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8df401b070 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.766+0000 7f8df97fa640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8de806b4d0 0x7f8de806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.766+0000 7f8df97fa640 1 -- 172.21.15.59:0/4015568497 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8df40896d0 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.766+0000 7f8dfbfff640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8de806b4d0 0x7f8de806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.766+0000 7f8dfbfff640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8de806b4d0 0x7f8de806d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8df0036000 tx=0x7f8df0008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.771+0000 7f8df97fa640 1 -- 172.21.15.59:0/4015568497 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8df4057f90 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.949+0000 7f8e02a88640 1 -- 172.21.15.59:0/4015568497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8dfc068ff0 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.950+0000 7f8df97fa640 1 -- 172.21.15.59:0/4015568497 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8df405c030 con 0x7f8dfc0b0e30 2024-03-06T07:22:32.952 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 -- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8de806b4d0 msgr2=0x7f8de806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8de806b4d0 0x7f8de806d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8df0036000 tx=0x7f8df0008030 comp rx=0 tx=0).stop 2024-03-06T07:22:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 -- 172.21.15.59:0/4015568497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dfc0b0e30 msgr2=0x7f8dfc0c96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dfc0b0e30 0x7f8dfc0c96e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8df400c7c0 tx=0x7f8df400cc80 comp rx=0 tx=0).stop 2024-03-06T07:22:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 -- 172.21.15.59:0/4015568497 shutdown_connections 2024-03-06T07:22:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8de806b4d0 0x7f8de806d990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8dfc1614d0 0x7f8dfc0cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8dfc0d4860 0x7f8dfc1f7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 --2- 172.21.15.59:0/4015568497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8dfc0b0e30 0x7f8dfc0c96e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 -- 172.21.15.59:0/4015568497 >> 172.21.15.59:0/4015568497 conn(0x7f8dfc0d3d40 msgr2=0x7f8dfc0d1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 -- 172.21.15.59:0/4015568497 shutdown_connections 2024-03-06T07:22:32.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:32.952+0000 7f8e02a88640 1 -- 172.21.15.59:0/4015568497 wait complete. 2024-03-06T07:22:32.965 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:33.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.067+0000 7fdd56871640 1 Processor -- start 2024-03-06T07:22:33.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.067+0000 7fdd56871640 1 -- start start 2024-03-06T07:22:33.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.068+0000 7fdd56871640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdd501614d0 0x7fdd500d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.068+0000 7fdd56871640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdd500d24b0 0x7fdd500b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.068+0000 7fdd56871640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd501618b0 con 0x7fdd500d1910 2024-03-06T07:22:33.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.068+0000 7fdd56871640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd500cf400 con 0x7fdd500d24b0 2024-03-06T07:22:33.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.068+0000 7fdd56871640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd500cf6a0 con 0x7fdd501614d0 2024-03-06T07:22:33.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.068+0000 7fdd54e0e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdd500d24b0 0x7fdd500b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.068+0000 7fdd54e0e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdd500d24b0 0x7fdd500b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53106/0 (socket says 172.21.15.59:53106) 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.068+0000 7fdd54e0e640 1 -- 172.21.15.59:0/3359779863 learned_addr learned my addr 172.21.15.59:0/3359779863 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.068+0000 7fdd4ffff640 1 --2- 172.21.15.59:0/3359779863 >> v2:172.21.15.164:3300/0 conn(0x7fdd501614d0 0x7fdd500d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.069+0000 7fdd54e0e640 1 -- 172.21.15.59:0/3359779863 >> v2:172.21.15.164:3300/0 conn(0x7fdd501614d0 msgr2=0x7fdd500d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.069+0000 7fdd54e0e640 1 --2- 172.21.15.59:0/3359779863 >> v2:172.21.15.164:3300/0 conn(0x7fdd501614d0 0x7fdd500d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.069+0000 7fdd54e0e640 1 -- 172.21.15.59:0/3359779863 >> v1:172.21.15.164:6789/0 conn(0x7fdd500d1910 legacy=0x7fdd500d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.069+0000 7fdd54e0e640 1 -- 172.21.15.59:0/3359779863 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500cf980 con 0x7fdd500d24b0 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.069+0000 7fdd54e0e640 1 --2- 172.21.15.59:0/3359779863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd500d24b0 0x7fdd500b0e30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdd5008c470 tx=0x7fdd4802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622c075be87f9fe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.069+0000 7fdd4effd640 1 -- 172.21.15.59:0/3359779863 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4802cb20 con 0x7fdd500d24b0 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.069+0000 7fdd4effd640 1 -- 172.21.15.59:0/3359779863 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4802ccc0 con 0x7fdd500d24b0 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.069+0000 7fdd4effd640 1 -- 172.21.15.59:0/3359779863 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48034610 con 0x7fdd500d24b0 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.070+0000 7fdd56871640 1 -- 172.21.15.59:0/3359779863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd500d24b0 msgr2=0x7fdd500b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.070+0000 7fdd56871640 1 --2- 172.21.15.59:0/3359779863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd500d24b0 0x7fdd500b0e30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdd5008c470 tx=0x7fdd4802bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.070+0000 7fdd56871640 1 -- 172.21.15.59:0/3359779863 shutdown_connections 2024-03-06T07:22:33.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.070+0000 7fdd56871640 1 --2- 172.21.15.59:0/3359779863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd500d24b0 0x7fdd500b0e30 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.070+0000 7fdd56871640 1 --2- 172.21.15.59:0/3359779863 >> v2:172.21.15.164:3300/0 conn(0x7fdd501614d0 0x7fdd500d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.070+0000 7fdd56871640 1 -- 172.21.15.59:0/3359779863 >> 172.21.15.59:0/3359779863 conn(0x7fdd500d3d40 msgr2=0x7fdd500d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.070+0000 7fdd56871640 1 -- 172.21.15.59:0/3359779863 shutdown_connections 2024-03-06T07:22:33.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.070+0000 7fdd56871640 1 -- 172.21.15.59:0/3359779863 wait complete. 2024-03-06T07:22:33.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.071+0000 7fdd56871640 1 Processor -- start 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.071+0000 7fdd56871640 1 -- start start 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.072+0000 7fdd56871640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd500d1910 0x7fdd501551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.072+0000 7fdd56871640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd500d24b0 0x7fdd501f81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.072+0000 7fdd56871640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd501614d0 0x7fdd500b8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.072+0000 7fdd56871640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd500b1370 con 0x7fdd500d1910 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.072+0000 7fdd56871640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd500b0910 con 0x7fdd500d24b0 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.072+0000 7fdd56871640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd500b14f0 con 0x7fdd501614d0 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.072+0000 7fdd4ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd500d1910 0x7fdd501551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.072+0000 7fdd4ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd500d1910 0x7fdd501551f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53112/0 (socket says 172.21.15.59:53112) 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.072+0000 7fdd4ffff640 1 -- 172.21.15.59:0/4003077688 learned_addr learned my addr 172.21.15.59:0/4003077688 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.072+0000 7fdd54e0e640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd501614d0 0x7fdd500b8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.072+0000 7fdd4f7fe640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd500d24b0 0x7fdd501f81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.073+0000 7fdd4ffff640 1 -- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd501614d0 msgr2=0x7fdd500b8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.073+0000 7fdd4ffff640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd501614d0 0x7fdd500b8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.073+0000 7fdd4ffff640 1 -- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd500d24b0 msgr2=0x7fdd501f81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.073+0000 7fdd4ffff640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd500d24b0 0x7fdd501f81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.073+0000 7fdd4ffff640 1 -- 172.21.15.59:0/4003077688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500b4e70 con 0x7fdd500d1910 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.073+0000 7fdd4ffff640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd500d1910 0x7fdd501551f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000e870 tx=0x7fdd4000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.073+0000 7fdd4cff9640 1 -- 172.21.15.59:0/4003077688 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40017070 con 0x7fdd500d1910 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.073+0000 7fdd4cff9640 1 -- 172.21.15.59:0/4003077688 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4000c940 con 0x7fdd500d1910 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.073+0000 7fdd56871640 1 -- 172.21.15.59:0/4003077688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd500b5ed0 con 0x7fdd500d1910 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.073+0000 7fdd4cff9640 1 -- 172.21.15.59:0/4003077688 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4000cd30 con 0x7fdd500d1910 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.073+0000 7fdd56871640 1 -- 172.21.15.59:0/4003077688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd500b5110 con 0x7fdd500d1910 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.074+0000 7fdd56871640 1 -- 172.21.15.59:0/4003077688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd50155dd0 con 0x7fdd500d1910 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.074+0000 7fdd4cff9640 1 -- 172.21.15.59:0/4003077688 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd4001a070 con 0x7fdd500d1910 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.075+0000 7fdd4cff9640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd3c06b4d0 0x7fdd3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.075+0000 7fdd4cff9640 1 -- 172.21.15.59:0/4003077688 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd40089f80 con 0x7fdd500d1910 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.075+0000 7fdd4f7fe640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd3c06b4d0 0x7fdd3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.075+0000 7fdd4f7fe640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd3c06b4d0 0x7fdd3c06d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdd50159340 tx=0x7fdd44009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.079+0000 7fdd4cff9640 1 -- 172.21.15.59:0/4003077688 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd40058840 con 0x7fdd500d1910 2024-03-06T07:22:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.258+0000 7fdd56871640 1 -- 172.21.15.59:0/4003077688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdd50068ff0 con 0x7fdd500d1910 2024-03-06T07:22:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.258+0000 7fdd4cff9640 1 -- 172.21.15.59:0/4003077688 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdd4005c8e0 con 0x7fdd500d1910 2024-03-06T07:22:33.260 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 -- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd3c06b4d0 msgr2=0x7fdd3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd3c06b4d0 0x7fdd3c06d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdd50159340 tx=0x7fdd44009040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 -- 172.21.15.59:0/4003077688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd500d1910 msgr2=0x7fdd501551f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd500d1910 0x7fdd501551f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000e870 tx=0x7fdd4000ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 -- 172.21.15.59:0/4003077688 shutdown_connections 2024-03-06T07:22:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd3c06b4d0 0x7fdd3c06d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd501614d0 0x7fdd500b8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd500d24b0 0x7fdd501f81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 --2- 172.21.15.59:0/4003077688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd500d1910 0x7fdd501551f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 -- 172.21.15.59:0/4003077688 >> 172.21.15.59:0/4003077688 conn(0x7fdd500d3d40 msgr2=0x7fdd500d0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 -- 172.21.15.59:0/4003077688 shutdown_connections 2024-03-06T07:22:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.260+0000 7fdd56871640 1 -- 172.21.15.59:0/4003077688 wait complete. 2024-03-06T07:22:33.273 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:33.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.375+0000 7fdcdc229640 1 Processor -- start 2024-03-06T07:22:33.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.375+0000 7fdcdc229640 1 -- start start 2024-03-06T07:22:33.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.376+0000 7fdcdc229640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdcd40719e0 0x7fdcd40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.376+0000 7fdcdc229640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdcd406c9f0 0x7fdcd406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.376+0000 7fdcdc229640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd40b3320 con 0x7fdcd415a220 2024-03-06T07:22:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.376+0000 7fdcdc229640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd40cf610 con 0x7fdcd40719e0 2024-03-06T07:22:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.376+0000 7fdcdc229640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd40728d0 con 0x7fdcd406c9f0 2024-03-06T07:22:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.376+0000 7fdcd97c4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdcd40719e0 0x7fdcd40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.376+0000 7fdcd9fc5640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdcd415a220 0x7fdcd40712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47082/0 (socket says 172.21.15.59:47082) 2024-03-06T07:22:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.376+0000 7fdcd9fc5640 1 -- 172.21.15.59:0/444491909 learned_addr learned my addr 172.21.15.59:0/444491909 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.376+0000 7fdcda7c6640 1 --2- 172.21.15.59:0/444491909 >> v2:172.21.15.164:3300/0 conn(0x7fdcd406c9f0 0x7fdcd406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.377+0000 7fdcd97c4640 1 -- 172.21.15.59:0/444491909 >> v2:172.21.15.164:3300/0 conn(0x7fdcd406c9f0 msgr2=0x7fdcd406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.377+0000 7fdcd97c4640 1 --2- 172.21.15.59:0/444491909 >> v2:172.21.15.164:3300/0 conn(0x7fdcd406c9f0 0x7fdcd406ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.377+0000 7fdcd97c4640 1 -- 172.21.15.59:0/444491909 >> v1:172.21.15.59:6789/0 conn(0x7fdcd415a220 legacy=0x7fdcd40712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.377+0000 7fdcd97c4640 1 -- 172.21.15.59:0/444491909 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd40b3590 con 0x7fdcd40719e0 2024-03-06T07:22:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.377+0000 7fdcd8fc3640 1 -- 172.21.15.59:0/444491909 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158960846 0 0) 0x7fdcd40b3320 con 0x7fdcd415a220 2024-03-06T07:22:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.377+0000 7fdcd97c4640 1 --2- 172.21.15.59:0/444491909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd40719e0 0x7fdcd40cf0d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40aeed0 tx=0x7fdcc802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89339fbf13ad427d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.378+0000 7fdcd8fc3640 1 -- 172.21.15.59:0/444491909 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80315a0 con 0x7fdcd40719e0 2024-03-06T07:22:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.378+0000 7fdcd8fc3640 1 -- 172.21.15.59:0/444491909 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc8037070 con 0x7fdcd40719e0 2024-03-06T07:22:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.378+0000 7fdcd8fc3640 1 -- 172.21.15.59:0/444491909 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8031960 con 0x7fdcd40719e0 2024-03-06T07:22:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.378+0000 7fdcdc229640 1 -- 172.21.15.59:0/444491909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd40719e0 msgr2=0x7fdcd40cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.378+0000 7fdcdc229640 1 --2- 172.21.15.59:0/444491909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd40719e0 0x7fdcd40cf0d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40aeed0 tx=0x7fdcc802b840 comp rx=0 tx=0).stop 2024-03-06T07:22:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.378+0000 7fdcdc229640 1 -- 172.21.15.59:0/444491909 shutdown_connections 2024-03-06T07:22:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.378+0000 7fdcdc229640 1 --2- 172.21.15.59:0/444491909 >> v2:172.21.15.164:3300/0 conn(0x7fdcd406c9f0 0x7fdcd406ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.378+0000 7fdcdc229640 1 --2- 172.21.15.59:0/444491909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd40719e0 0x7fdcd40cf0d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.378+0000 7fdcdc229640 1 -- 172.21.15.59:0/444491909 >> 172.21.15.59:0/444491909 conn(0x7fdcd406b190 msgr2=0x7fdcd406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.378+0000 7fdcdc229640 1 -- 172.21.15.59:0/444491909 shutdown_connections 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.378+0000 7fdcdc229640 1 -- 172.21.15.59:0/444491909 wait complete. 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.379+0000 7fdcdc229640 1 Processor -- start 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.379+0000 7fdcdc229640 1 -- start start 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcdc229640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd406c9f0 0x7fdcd414a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcdc229640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdcd40719e0 0x7fdcd414aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcdc229640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcd415a220 0x7fdcd41f53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcdc229640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd4072530 con 0x7fdcd406c9f0 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcdc229640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd4072030 con 0x7fdcd415a220 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcdc229640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd40726b0 con 0x7fdcd40719e0 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcd9fc5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd406c9f0 0x7fdcd414a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcd9fc5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd406c9f0 0x7fdcd414a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53132/0 (socket says 172.21.15.59:53132) 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcd9fc5640 1 -- 172.21.15.59:0/3295433671 learned_addr learned my addr 172.21.15.59:0/3295433671 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcd97c4640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdcd40719e0 0x7fdcd414aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcda7c6640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcd415a220 0x7fdcd41f53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.380+0000 7fdcd9fc5640 1 -- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdcd40719e0 msgr2=0x7fdcd414aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.381+0000 7fdcd9fc5640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdcd40719e0 0x7fdcd414aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.381+0000 7fdcd9fc5640 1 -- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcd415a220 msgr2=0x7fdcd41f53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.381+0000 7fdcd9fc5640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcd415a220 0x7fdcd41f53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.381+0000 7fdcd9fc5640 1 -- 172.21.15.59:0/3295433671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd414ff60 con 0x7fdcd406c9f0 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.381+0000 7fdcd9fc5640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd406c9f0 0x7fdcd414a4e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00c430 tx=0x7fdccc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.381+0000 7fdcd27fc640 1 -- 172.21.15.59:0/3295433671 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc0074e0 con 0x7fdcd406c9f0 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.381+0000 7fdcd27fc640 1 -- 172.21.15.59:0/3295433671 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc0047e0 con 0x7fdcd406c9f0 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.381+0000 7fdcdc229640 1 -- 172.21.15.59:0/3295433671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd4150240 con 0x7fdcd406c9f0 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.381+0000 7fdcd27fc640 1 -- 172.21.15.59:0/3295433671 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc004bd0 con 0x7fdcd406c9f0 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.381+0000 7fdcdc229640 1 -- 172.21.15.59:0/3295433671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd4149780 con 0x7fdcd406c9f0 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.382+0000 7fdcdc229640 1 -- 172.21.15.59:0/3295433671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd40732b0 con 0x7fdcd406c9f0 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.382+0000 7fdcd27fc640 1 -- 172.21.15.59:0/3295433671 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdccc011070 con 0x7fdcd406c9f0 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.383+0000 7fdcd27fc640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcc006b4d0 0x7fdcc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.383+0000 7fdcd27fc640 1 -- 172.21.15.59:0/3295433671 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdccc089ca0 con 0x7fdcd406c9f0 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.383+0000 7fdcd97c4640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcc006b4d0 0x7fdcc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.383+0000 7fdcd97c4640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcc006b4d0 0x7fdcc006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8036000 tx=0x7fdcc803d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.387+0000 7fdcd27fc640 1 -- 172.21.15.59:0/3295433671 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdccc058560 con 0x7fdcd406c9f0 2024-03-06T07:22:33.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.569+0000 7fdcdc229640 1 -- 172.21.15.59:0/3295433671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdcd41670f0 con 0x7fdcd406c9f0 2024-03-06T07:22:33.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.570+0000 7fdcd27fc640 1 -- 172.21.15.59:0/3295433671 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdccc05c600 con 0x7fdcd406c9f0 2024-03-06T07:22:33.572 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:33.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 -- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcc006b4d0 msgr2=0x7fdcc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcc006b4d0 0x7fdcc006d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8036000 tx=0x7fdcc803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 -- 172.21.15.59:0/3295433671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd406c9f0 msgr2=0x7fdcd414a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd406c9f0 0x7fdcd414a4e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00c430 tx=0x7fdccc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 -- 172.21.15.59:0/3295433671 shutdown_connections 2024-03-06T07:22:33.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcc006b4d0 0x7fdcc006d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdcd415a220 0x7fdcd41f53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdcd40719e0 0x7fdcd414aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 --2- 172.21.15.59:0/3295433671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd406c9f0 0x7fdcd414a4e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 -- 172.21.15.59:0/3295433671 >> 172.21.15.59:0/3295433671 conn(0x7fdcd406b190 msgr2=0x7fdcd406db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 -- 172.21.15.59:0/3295433671 shutdown_connections 2024-03-06T07:22:33.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.572+0000 7fdcdc229640 1 -- 172.21.15.59:0/3295433671 wait complete. 2024-03-06T07:22:33.585 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:33.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.685+0000 7fe0eb17a640 1 Processor -- start 2024-03-06T07:22:33.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.685+0000 7fe0eb17a640 1 -- start start 2024-03-06T07:22:33.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.686+0000 7fe0eb17a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe0e40b4c00 0x7fe0e40b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.686+0000 7fe0eb17a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0e40b28a0 0x7fe0e40b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.686+0000 7fe0eb17a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e415d510 con 0x7fe0e415e540 2024-03-06T07:22:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.686+0000 7fe0eb17a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e415e0b0 con 0x7fe0e40b28a0 2024-03-06T07:22:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.686+0000 7fe0eb17a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e415d780 con 0x7fe0e40b4c00 2024-03-06T07:22:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.686+0000 7fe0e3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0e40b28a0 0x7fe0e40b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.686+0000 7fe0e3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0e40b28a0 0x7fe0e40b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58088/0 (socket says 172.21.15.59:58088) 2024-03-06T07:22:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.686+0000 7fe0e9717640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe0e415e540 0x7fe0e415e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51298/0 (socket says 172.21.15.59:51298) 2024-03-06T07:22:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.686+0000 7fe0e3fff640 1 -- 172.21.15.59:0/3211249798 learned_addr learned my addr 172.21.15.59:0/3211249798 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.686+0000 7fe0e8f16640 1 --2- 172.21.15.59:0/3211249798 >> v2:172.21.15.164:3300/0 conn(0x7fe0e40b4c00 0x7fe0e40b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.687+0000 7fe0e3fff640 1 -- 172.21.15.59:0/3211249798 >> v2:172.21.15.164:3300/0 conn(0x7fe0e40b4c00 msgr2=0x7fe0e40b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.687+0000 7fe0e3fff640 1 --2- 172.21.15.59:0/3211249798 >> v2:172.21.15.164:3300/0 conn(0x7fe0e40b4c00 0x7fe0e40b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.687+0000 7fe0e3fff640 1 -- 172.21.15.59:0/3211249798 >> v1:172.21.15.59:6789/0 conn(0x7fe0e415e540 legacy=0x7fe0e415e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.687+0000 7fe0e3fff640 1 -- 172.21.15.59:0/3211249798 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e415da60 con 0x7fe0e40b28a0 2024-03-06T07:22:33.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.687+0000 7fe0e37fe640 1 -- 172.21.15.59:0/3211249798 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018257055 0 0) 0x7fe0e415d510 con 0x7fe0e415e540 2024-03-06T07:22:33.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.687+0000 7fe0e3fff640 1 --2- 172.21.15.59:0/3211249798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e40b28a0 0x7fe0e40b2d10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40721d0 tx=0x7fe0d802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc4ec389fd149eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.688+0000 7fe0e37fe640 1 -- 172.21.15.59:0/3211249798 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d802f850 con 0x7fe0e40b28a0 2024-03-06T07:22:33.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.688+0000 7fe0e37fe640 1 -- 172.21.15.59:0/3211249798 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d802f9f0 con 0x7fe0e40b28a0 2024-03-06T07:22:33.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.688+0000 7fe0e37fe640 1 -- 172.21.15.59:0/3211249798 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d802fdc0 con 0x7fe0e40b28a0 2024-03-06T07:22:33.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.688+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3211249798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e40b28a0 msgr2=0x7fe0e40b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.688+0000 7fe0eb17a640 1 --2- 172.21.15.59:0/3211249798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e40b28a0 0x7fe0e40b2d10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40721d0 tx=0x7fe0d802ba30 comp rx=0 tx=0).stop 2024-03-06T07:22:33.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.688+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3211249798 shutdown_connections 2024-03-06T07:22:33.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.688+0000 7fe0eb17a640 1 --2- 172.21.15.59:0/3211249798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e40b28a0 0x7fe0e40b2d10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.688+0000 7fe0eb17a640 1 --2- 172.21.15.59:0/3211249798 >> v2:172.21.15.164:3300/0 conn(0x7fe0e40b4c00 0x7fe0e40b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.688+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3211249798 >> 172.21.15.59:0/3211249798 conn(0x7fe0e409a9f0 msgr2=0x7fe0e409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.688+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3211249798 shutdown_connections 2024-03-06T07:22:33.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.688+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3211249798 wait complete. 2024-03-06T07:22:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.689+0000 7fe0eb17a640 1 Processor -- start 2024-03-06T07:22:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.689+0000 7fe0eb17a640 1 -- start start 2024-03-06T07:22:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.690+0000 7fe0eb17a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e40b28a0 0x7fe0e4153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.690+0000 7fe0eb17a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0e40b4c00 0x7fe0e40afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.690+0000 7fe0eb17a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e415e540 0x7fe0e4141050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.690+0000 7fe0eb17a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e40b33d0 con 0x7fe0e415e540 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.690+0000 7fe0eb17a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e40b3250 con 0x7fe0e40b4c00 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.690+0000 7fe0eb17a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e415f0c0 con 0x7fe0e40b28a0 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.690+0000 7fe0e9717640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e415e540 0x7fe0e4141050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.690+0000 7fe0e9717640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e415e540 0x7fe0e4141050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58100/0 (socket says 172.21.15.59:58100) 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.690+0000 7fe0e9717640 1 -- 172.21.15.59:0/3625061622 learned_addr learned my addr 172.21.15.59:0/3625061622 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.690+0000 7fe0e8f16640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e40b28a0 0x7fe0e4153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.690+0000 7fe0e3fff640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0e40b4c00 0x7fe0e40afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.691+0000 7fe0e9717640 1 -- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e40b28a0 msgr2=0x7fe0e4153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.691+0000 7fe0e9717640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e40b28a0 0x7fe0e4153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.691+0000 7fe0e9717640 1 -- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0e40b4c00 msgr2=0x7fe0e40afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.691+0000 7fe0e9717640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0e40b4c00 0x7fe0e40afa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.691+0000 7fe0e9717640 1 -- 172.21.15.59:0/3625061622 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e40b0530 con 0x7fe0e415e540 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.691+0000 7fe0e9717640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e415e540 0x7fe0e4141050 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400c410 tx=0x7fe0d400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.691+0000 7fe0e17fa640 1 -- 172.21.15.59:0/3625061622 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40074e0 con 0x7fe0e415e540 2024-03-06T07:22:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.691+0000 7fe0e17fa640 1 -- 172.21.15.59:0/3625061622 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d40047e0 con 0x7fe0e415e540 2024-03-06T07:22:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.691+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3625061622 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e40b0810 con 0x7fe0e415e540 2024-03-06T07:22:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.691+0000 7fe0e17fa640 1 -- 172.21.15.59:0/3625061622 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4004bd0 con 0x7fe0e415e540 2024-03-06T07:22:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.691+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3625061622 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e41519d0 con 0x7fe0e415e540 2024-03-06T07:22:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.692+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3625061622 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e404f330 con 0x7fe0e415e540 2024-03-06T07:22:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.692+0000 7fe0e17fa640 1 -- 172.21.15.59:0/3625061622 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0d4011070 con 0x7fe0e415e540 2024-03-06T07:22:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.693+0000 7fe0e17fa640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0d006b4d0 0x7fe0d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.693+0000 7fe0e17fa640 1 -- 172.21.15.59:0/3625061622 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0d4089c90 con 0x7fe0e415e540 2024-03-06T07:22:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.693+0000 7fe0e8f16640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0d006b4d0 0x7fe0d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.693+0000 7fe0e8f16640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0d006b4d0 0x7fe0d006d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0113e0 tx=0x7fe0dc00b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.697+0000 7fe0e17fa640 1 -- 172.21.15.59:0/3625061622 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0d40584c0 con 0x7fe0e415e540 2024-03-06T07:22:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.893+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3625061622 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe0e40b2f30 con 0x7fe0e415e540 2024-03-06T07:22:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.893+0000 7fe0e17fa640 1 -- 172.21.15.59:0/3625061622 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe0d405c560 con 0x7fe0e415e540 2024-03-06T07:22:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:22:33.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.895+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0d006b4d0 msgr2=0x7fe0d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.895+0000 7fe0eb17a640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0d006b4d0 0x7fe0d006d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0113e0 tx=0x7fe0dc00b040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.895+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3625061622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e415e540 msgr2=0x7fe0e4141050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.895+0000 7fe0eb17a640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e415e540 0x7fe0e4141050 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe0d400c410 tx=0x7fe0d400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.896+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3625061622 shutdown_connections 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.896+0000 7fe0eb17a640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0d006b4d0 0x7fe0d006d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.896+0000 7fe0eb17a640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e415e540 0x7fe0e4141050 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.896+0000 7fe0eb17a640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0e40b4c00 0x7fe0e40afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.896+0000 7fe0eb17a640 1 --2- 172.21.15.59:0/3625061622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e40b28a0 0x7fe0e4153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.896+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3625061622 >> 172.21.15.59:0/3625061622 conn(0x7fe0e409a9f0 msgr2=0x7fe0e415cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.896+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3625061622 shutdown_connections 2024-03-06T07:22:33.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:33.896+0000 7fe0eb17a640 1 -- 172.21.15.59:0/3625061622 wait complete. 2024-03-06T07:22:33.908 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:22:33.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:33.908 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:33.935 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:33.935 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:22:33.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:33.935 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:22:33.990 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:22:33.991 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:34.007 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:22:34.007 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:34.062 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:34.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.161+0000 7f8488713640 1 Processor -- start 2024-03-06T07:22:34.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.161+0000 7f8488713640 1 -- start start 2024-03-06T07:22:34.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.162+0000 7f8488713640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f84800d00c0 0x7f84800d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.162+0000 7f8488713640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84800d09e0 con 0x7f84800cf850 2024-03-06T07:22:34.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.162+0000 7f8488713640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848015e380 con 0x7f848006cf20 2024-03-06T07:22:34.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.162+0000 7f8488713640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848006ca90 con 0x7f84800d00c0 2024-03-06T07:22:34.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.162+0000 7f84864af640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f84800d00c0 0x7f84800d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.162+0000 7f8485cae640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f84800cf850 0x7f84800cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51312/0 (socket says 172.21.15.59:51312) 2024-03-06T07:22:34.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.163+0000 7f84864af640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f84800d00c0 0x7f84800d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58104/0 (socket says 172.21.15.59:58104) 2024-03-06T07:22:34.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.163+0000 7f8485cae640 1 -- 172.21.15.59:0/2381327641 learned_addr learned my addr 172.21.15.59:0/2381327641 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:34.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.163+0000 7f84864af640 1 -- 172.21.15.59:0/2381327641 >> v1:172.21.15.164:6789/0 conn(0x7f848006cf20 legacy=0x7f848006d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.163+0000 7f84864af640 1 -- 172.21.15.59:0/2381327641 >> v1:172.21.15.59:6789/0 conn(0x7f84800cf850 legacy=0x7f84800cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.163+0000 7f84864af640 1 -- 172.21.15.59:0/2381327641 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84800cdc10 con 0x7f84800d00c0 2024-03-06T07:22:34.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.163+0000 7f84854ad640 1 -- 172.21.15.59:0/2381327641 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765711792 0 0) 0x7f84800d09e0 con 0x7f84800cf850 2024-03-06T07:22:34.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.164+0000 7f84864af640 1 --2- 172.21.15.59:0/2381327641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84800d00c0 0x7f84800d04a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f847c0097a0 tx=0x7f847c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88014ca5fac6c7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.164+0000 7f84854ad640 1 -- 172.21.15.59:0/2381327641 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c034070 con 0x7f84800d00c0 2024-03-06T07:22:34.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.164+0000 7f84854ad640 1 -- 172.21.15.59:0/2381327641 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c030610 con 0x7f84800d00c0 2024-03-06T07:22:34.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.164+0000 7f84854ad640 1 -- 172.21.15.59:0/2381327641 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c0309e0 con 0x7f84800d00c0 2024-03-06T07:22:34.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.164+0000 7f8488713640 1 -- 172.21.15.59:0/2381327641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84800d00c0 msgr2=0x7f84800d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.164+0000 7f8488713640 1 --2- 172.21.15.59:0/2381327641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84800d00c0 0x7f84800d04a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f847c0097a0 tx=0x7f847c02b960 comp rx=0 tx=0).stop 2024-03-06T07:22:34.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.164+0000 7f8488713640 1 -- 172.21.15.59:0/2381327641 shutdown_connections 2024-03-06T07:22:34.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.164+0000 7f8488713640 1 --2- 172.21.15.59:0/2381327641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84800d00c0 0x7f84800d04a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.164+0000 7f8488713640 1 -- 172.21.15.59:0/2381327641 >> 172.21.15.59:0/2381327641 conn(0x7f84800b35a0 msgr2=0x7f84800b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.164+0000 7f8488713640 1 -- 172.21.15.59:0/2381327641 shutdown_connections 2024-03-06T07:22:34.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.164+0000 7f8488713640 1 -- 172.21.15.59:0/2381327641 wait complete. 2024-03-06T07:22:34.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.165+0000 7f8488713640 1 Processor -- start 2024-03-06T07:22:34.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.165+0000 7f8488713640 1 -- start start 2024-03-06T07:22:34.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.166+0000 7f8488713640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f848006cf20 0x7f8480150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.166+0000 7f8488713640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84800cf850 0x7f84801f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.166+0000 7f8488713640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84800d00c0 0x7f84800ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.166+0000 7f8488713640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848006c720 con 0x7f84800cf850 2024-03-06T07:22:34.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.166+0000 7f8488713640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848006c440 con 0x7f84800d00c0 2024-03-06T07:22:34.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.166+0000 7f8488713640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84800b1030 con 0x7f848006cf20 2024-03-06T07:22:34.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.166+0000 7f8485cae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84800cf850 0x7f84801f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.166+0000 7f8485cae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84800cf850 0x7f84801f70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58112/0 (socket says 172.21.15.59:58112) 2024-03-06T07:22:34.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.166+0000 7f8485cae640 1 -- 172.21.15.59:0/730269562 learned_addr learned my addr 172.21.15.59:0/730269562 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:34.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.166+0000 7f84864af640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f848006cf20 0x7f8480150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.166+0000 7f8486cb0640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84800d00c0 0x7f84800ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.167+0000 7f8485cae640 1 -- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f848006cf20 msgr2=0x7f8480150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.167+0000 7f8485cae640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f848006cf20 0x7f8480150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.167+0000 7f8485cae640 1 -- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84800d00c0 msgr2=0x7f84800ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.167+0000 7f8485cae640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84800d00c0 0x7f84800ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.167+0000 7f8485cae640 1 -- 172.21.15.59:0/730269562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8480151440 con 0x7f84800cf850 2024-03-06T07:22:34.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.167+0000 7f8485cae640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84800cf850 0x7f84801f70b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f847000c4d0 tx=0x7f847000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.167+0000 7f8476ffd640 1 -- 172.21.15.59:0/730269562 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470018070 con 0x7f84800cf850 2024-03-06T07:22:34.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.167+0000 7f8476ffd640 1 -- 172.21.15.59:0/730269562 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84700074e0 con 0x7f84800cf850 2024-03-06T07:22:34.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.167+0000 7f8476ffd640 1 -- 172.21.15.59:0/730269562 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470004a20 con 0x7f84800cf850 2024-03-06T07:22:34.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.167+0000 7f8488713640 1 -- 172.21.15.59:0/730269562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84801f7fe0 con 0x7f84800cf850 2024-03-06T07:22:34.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.167+0000 7f8488713640 1 -- 172.21.15.59:0/730269562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84801f7d00 con 0x7f84800cf850 2024-03-06T07:22:34.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.168+0000 7f8488713640 1 -- 172.21.15.59:0/730269562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848004f330 con 0x7f84800cf850 2024-03-06T07:22:34.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.168+0000 7f8476ffd640 1 -- 172.21.15.59:0/730269562 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f847001b070 con 0x7f84800cf850 2024-03-06T07:22:34.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.169+0000 7f8476ffd640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f846406b4d0 0x7f846406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.169+0000 7f8476ffd640 1 -- 172.21.15.59:0/730269562 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84700893c0 con 0x7f84800cf850 2024-03-06T07:22:34.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.169+0000 7f84864af640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f846406b4d0 0x7f846406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.169+0000 7f84864af640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f846406b4d0 0x7f846406d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f847c036000 tx=0x7f847c03b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.173+0000 7f8476ffd640 1 -- 172.21.15.59:0/730269562 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8480151440 con 0x7f84800cf850 2024-03-06T07:22:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.355+0000 7f8488713640 1 -- 172.21.15.59:0/730269562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84800d04a0 con 0x7f84800cf850 2024-03-06T07:22:34.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.356+0000 7f8476ffd640 1 -- 172.21.15.59:0/730269562 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f8470057c80 con 0x7f84800cf850 2024-03-06T07:22:34.358 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:34.359 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:21:56.406051+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:34.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.358+0000 7f8488713640 1 -- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f846406b4d0 msgr2=0x7f846406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.358+0000 7f8488713640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f846406b4d0 0x7f846406d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f847c036000 tx=0x7f847c03b040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.358+0000 7f8488713640 1 -- 172.21.15.59:0/730269562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84800cf850 msgr2=0x7f84801f70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.358+0000 7f8488713640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84800cf850 0x7f84801f70b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f847000c4d0 tx=0x7f847000c990 comp rx=0 tx=0).stop 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.358+0000 7f8488713640 1 -- 172.21.15.59:0/730269562 shutdown_connections 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.358+0000 7f8488713640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f846406b4d0 0x7f846406d990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.358+0000 7f8488713640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84800d00c0 0x7f84800ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.358+0000 7f8488713640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84800cf850 0x7f84801f70b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.358+0000 7f8488713640 1 --2- 172.21.15.59:0/730269562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f848006cf20 0x7f8480150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.358+0000 7f8488713640 1 -- 172.21.15.59:0/730269562 >> 172.21.15.59:0/730269562 conn(0x7f84800b35a0 msgr2=0x7f848006fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.358+0000 7f8488713640 1 -- 172.21.15.59:0/730269562 shutdown_connections 2024-03-06T07:22:34.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.359+0000 7f8488713640 1 -- 172.21.15.59:0/730269562 wait complete. 2024-03-06T07:22:34.371 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:34.372 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:34.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.475+0000 7fa270cdf640 1 Processor -- start 2024-03-06T07:22:34.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.475+0000 7fa270cdf640 1 -- start start 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa270cdf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa26c15bbb0 0x7fa26c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa270cdf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa26c06b4f0 0x7fa26c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa270cdf640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa26c15ec40 0x7fa26c0cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa270cdf640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa26c161970 con 0x7fa26c15ec40 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa270cdf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa26c1624f0 con 0x7fa26c15bbb0 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa270cdf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c0cea90 con 0x7fa26c06b4f0 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa26ad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa26c15ec40 0x7fa26c0cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa26ad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa26c15ec40 0x7fa26c0cd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58120/0 (socket says 172.21.15.59:58120) 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa26a59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa26c15bbb0 0x7fa26c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa263fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa26c06b4f0 0x7fa26c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa26ad9d640 1 -- 172.21.15.59:0/3693658380 learned_addr learned my addr 172.21.15.59:0/3693658380 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa26ad9d640 1 -- 172.21.15.59:0/3693658380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa26c06b4f0 msgr2=0x7fa26c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa26ad9d640 1 --2- 172.21.15.59:0/3693658380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa26c06b4f0 0x7fa26c15e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa26ad9d640 1 -- 172.21.15.59:0/3693658380 >> v2:172.21.15.164:3300/0 conn(0x7fa26c15bbb0 msgr2=0x7fa26c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa26ad9d640 1 --2- 172.21.15.59:0/3693658380 >> v2:172.21.15.164:3300/0 conn(0x7fa26c15bbb0 0x7fa26c06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa26ad9d640 1 -- 172.21.15.59:0/3693658380 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c161bc0 con 0x7fa26c15ec40 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.476+0000 7fa26a59c640 1 --2- 172.21.15.59:0/3693658380 >> v2:172.21.15.164:3300/0 conn(0x7fa26c15bbb0 0x7fa26c06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.477+0000 7fa26ad9d640 1 --2- 172.21.15.59:0/3693658380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa26c15ec40 0x7fa26c0cd3a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0ce420 tx=0x7fa26402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db2466faf94c106 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.477+0000 7fa269d9b640 1 -- 172.21.15.59:0/3693658380 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26402ca70 con 0x7fa26c15ec40 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.477+0000 7fa269d9b640 1 -- 172.21.15.59:0/3693658380 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa26402cc10 con 0x7fa26c15ec40 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.477+0000 7fa269d9b640 1 -- 172.21.15.59:0/3693658380 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2640343f0 con 0x7fa26c15ec40 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.478+0000 7fa270cdf640 1 -- 172.21.15.59:0/3693658380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa26c15ec40 msgr2=0x7fa26c0cd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.478+0000 7fa270cdf640 1 --2- 172.21.15.59:0/3693658380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa26c15ec40 0x7fa26c0cd3a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0ce420 tx=0x7fa26402bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.478+0000 7fa270cdf640 1 -- 172.21.15.59:0/3693658380 shutdown_connections 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.478+0000 7fa270cdf640 1 --2- 172.21.15.59:0/3693658380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa26c15ec40 0x7fa26c0cd3a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.478+0000 7fa270cdf640 1 --2- 172.21.15.59:0/3693658380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa26c06b4f0 0x7fa26c15e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.478+0000 7fa270cdf640 1 --2- 172.21.15.59:0/3693658380 >> v2:172.21.15.164:3300/0 conn(0x7fa26c15bbb0 0x7fa26c06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.478+0000 7fa270cdf640 1 -- 172.21.15.59:0/3693658380 >> 172.21.15.59:0/3693658380 conn(0x7fa26c0b2660 msgr2=0x7fa26c0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.478+0000 7fa270cdf640 1 -- 172.21.15.59:0/3693658380 shutdown_connections 2024-03-06T07:22:34.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.478+0000 7fa270cdf640 1 -- 172.21.15.59:0/3693658380 wait complete. 2024-03-06T07:22:34.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.479+0000 7fa270cdf640 1 Processor -- start 2024-03-06T07:22:34.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.479+0000 7fa270cdf640 1 -- start start 2024-03-06T07:22:34.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa270cdf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa26c06b4f0 0x7fa26c160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa270cdf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa26c15ec40 0x7fa26c06ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa270cdf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa26c0b3750 0x7fa26c06e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa270cdf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c15f0d0 con 0x7fa26c0b3750 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa270cdf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c162180 con 0x7fa26c15ec40 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa270cdf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c0cd8e0 con 0x7fa26c06b4f0 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa26ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa26c0b3750 0x7fa26c06e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa26ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa26c0b3750 0x7fa26c06e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58128/0 (socket says 172.21.15.59:58128) 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa26ad9d640 1 -- 172.21.15.59:0/592411693 learned_addr learned my addr 172.21.15.59:0/592411693 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa26a59c640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa26c06b4f0 0x7fa26c160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa263fff640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa26c15ec40 0x7fa26c06ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa26ad9d640 1 -- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa26c06b4f0 msgr2=0x7fa26c160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa26ad9d640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa26c06b4f0 0x7fa26c160520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa26ad9d640 1 -- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa26c15ec40 msgr2=0x7fa26c06ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa26ad9d640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa26c15ec40 0x7fa26c06ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa26ad9d640 1 -- 172.21.15.59:0/592411693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c156f00 con 0x7fa26c0b3750 2024-03-06T07:22:34.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa26ad9d640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa26c0b3750 0x7fa26c06e300 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa264036040 tx=0x7fa264004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa262ffd640 1 -- 172.21.15.59:0/592411693 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa264044020 con 0x7fa26c0b3750 2024-03-06T07:22:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa262ffd640 1 -- 172.21.15.59:0/592411693 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa264040670 con 0x7fa26c0b3750 2024-03-06T07:22:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.480+0000 7fa270cdf640 1 -- 172.21.15.59:0/592411693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26c1571e0 con 0x7fa26c0b3750 2024-03-06T07:22:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.481+0000 7fa262ffd640 1 -- 172.21.15.59:0/592411693 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa264009510 con 0x7fa26c0b3750 2024-03-06T07:22:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.481+0000 7fa270cdf640 1 -- 172.21.15.59:0/592411693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26c1577a0 con 0x7fa26c0b3750 2024-03-06T07:22:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.481+0000 7fa270cdf640 1 -- 172.21.15.59:0/592411693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26c0733b0 con 0x7fa26c0b3750 2024-03-06T07:22:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.482+0000 7fa262ffd640 1 -- 172.21.15.59:0/592411693 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa264047020 con 0x7fa26c0b3750 2024-03-06T07:22:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.482+0000 7fa262ffd640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa24c06b500 0x7fa24c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.482+0000 7fa262ffd640 1 -- 172.21.15.59:0/592411693 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2640ac8e0 con 0x7fa26c0b3750 2024-03-06T07:22:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.482+0000 7fa26a59c640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa24c06b500 0x7fa24c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.483+0000 7fa26a59c640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa24c06b500 0x7fa24c06d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa25c009560 tx=0x7fa25c009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.487+0000 7fa262ffd640 1 -- 172.21.15.59:0/592411693 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa26404e070 con 0x7fa26c0b3750 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.666+0000 7fa270cdf640 1 -- 172.21.15.59:0/592411693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa26c068ff0 con 0x7fa26c0b3750 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.666+0000 7fa262ffd640 1 -- 172.21.15.59:0/592411693 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa26407b080 con 0x7fa26c0b3750 2024-03-06T07:22:34.668 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:34.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 -- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa24c06b500 msgr2=0x7fa24c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa24c06b500 0x7fa24c06d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa25c009560 tx=0x7fa25c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:34.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 -- 172.21.15.59:0/592411693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa26c0b3750 msgr2=0x7fa26c06e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa26c0b3750 0x7fa26c06e300 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa264036040 tx=0x7fa264004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:34.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 -- 172.21.15.59:0/592411693 shutdown_connections 2024-03-06T07:22:34.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa24c06b500 0x7fa24c06d9c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa26c0b3750 0x7fa26c06e300 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa26c15ec40 0x7fa26c06ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 --2- 172.21.15.59:0/592411693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa26c06b4f0 0x7fa26c160520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 -- 172.21.15.59:0/592411693 >> 172.21.15.59:0/592411693 conn(0x7fa26c0b2660 msgr2=0x7fa26c072740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 -- 172.21.15.59:0/592411693 shutdown_connections 2024-03-06T07:22:34.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.668+0000 7fa270cdf640 1 -- 172.21.15.59:0/592411693 wait complete. 2024-03-06T07:22:34.680 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:34.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.783+0000 7f55627ac640 1 Processor -- start 2024-03-06T07:22:34.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.783+0000 7f55627ac640 1 -- start start 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.783+0000 7f55627ac640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f555c145590 0x7f555c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.783+0000 7f55627ac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f555c06f6e0 0x7f555c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.783+0000 7f55627ac640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555c06b970 con 0x7f555c06b4f0 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.783+0000 7f55627ac640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555c145970 con 0x7f555c06f6e0 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.783+0000 7f55627ac640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555c160110 con 0x7f555c145590 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.784+0000 7f5560d49640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f555c06f6e0 0x7f555c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.784+0000 7f555b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f555c06b4f0 0x7f555c06efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51320/0 (socket says 172.21.15.59:51320) 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.784+0000 7f555b7fe640 1 -- 172.21.15.59:0/2315452365 learned_addr learned my addr 172.21.15.59:0/2315452365 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.784+0000 7f555bfff640 1 --2- 172.21.15.59:0/2315452365 >> v2:172.21.15.164:3300/0 conn(0x7f555c145590 0x7f555c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.784+0000 7f5560d49640 1 -- 172.21.15.59:0/2315452365 >> v2:172.21.15.164:3300/0 conn(0x7f555c145590 msgr2=0x7f555c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.784+0000 7f5560d49640 1 --2- 172.21.15.59:0/2315452365 >> v2:172.21.15.164:3300/0 conn(0x7f555c145590 0x7f555c06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.784+0000 7f5560d49640 1 -- 172.21.15.59:0/2315452365 >> v1:172.21.15.59:6789/0 conn(0x7f555c06b4f0 legacy=0x7f555c06efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.784+0000 7f5560d49640 1 -- 172.21.15.59:0/2315452365 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c15f7e0 con 0x7f555c06f6e0 2024-03-06T07:22:34.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.784+0000 7f555affd640 1 -- 172.21.15.59:0/2315452365 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809153657 0 0) 0x7f555c06b970 con 0x7f555c06b4f0 2024-03-06T07:22:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f5560d49640 1 --2- 172.21.15.59:0/2315452365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f555c06f6e0 0x7f555c0cc5c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f555c0d0830 tx=0x7f555402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aabb572637c25f4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f555affd640 1 -- 172.21.15.59:0/2315452365 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554039070 con 0x7f555c06f6e0 2024-03-06T07:22:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f555affd640 1 -- 172.21.15.59:0/2315452365 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555402cc80 con 0x7f555c06f6e0 2024-03-06T07:22:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f555affd640 1 -- 172.21.15.59:0/2315452365 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554034660 con 0x7f555c06f6e0 2024-03-06T07:22:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f55627ac640 1 -- 172.21.15.59:0/2315452365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f555c06f6e0 msgr2=0x7f555c0cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f55627ac640 1 --2- 172.21.15.59:0/2315452365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f555c06f6e0 0x7f555c0cc5c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f555c0d0830 tx=0x7f555402bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f55627ac640 1 -- 172.21.15.59:0/2315452365 shutdown_connections 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f55627ac640 1 --2- 172.21.15.59:0/2315452365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f555c06f6e0 0x7f555c0cc5c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f55627ac640 1 --2- 172.21.15.59:0/2315452365 >> v2:172.21.15.164:3300/0 conn(0x7f555c145590 0x7f555c06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f55627ac640 1 -- 172.21.15.59:0/2315452365 >> 172.21.15.59:0/2315452365 conn(0x7f555c0b2660 msgr2=0x7f555c0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f55627ac640 1 -- 172.21.15.59:0/2315452365 shutdown_connections 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.785+0000 7f55627ac640 1 -- 172.21.15.59:0/2315452365 wait complete. 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.786+0000 7f55627ac640 1 Processor -- start 2024-03-06T07:22:34.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.787+0000 7f55627ac640 1 -- start start 2024-03-06T07:22:34.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.787+0000 7f55627ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f555c06b4f0 0x7f555c1f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.787+0000 7f55627ac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f555c06f6e0 0x7f555c1f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.787+0000 7f55627ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f555c145590 0x7f555c0cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.787+0000 7f55627ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c1605a0 con 0x7f555c06b4f0 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.787+0000 7f55627ac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c15fda0 con 0x7f555c06f6e0 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.787+0000 7f55627ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555c1f2470 con 0x7f555c145590 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.787+0000 7f555bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f555c06b4f0 0x7f555c1f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.787+0000 7f555bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f555c06b4f0 0x7f555c1f18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58150/0 (socket says 172.21.15.59:58150) 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.787+0000 7f555bfff640 1 -- 172.21.15.59:0/3756903982 learned_addr learned my addr 172.21.15.59:0/3756903982 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f555b7fe640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f555c06f6e0 0x7f555c1f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f5560d49640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f555c145590 0x7f555c0cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f555bfff640 1 -- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f555c145590 msgr2=0x7f555c0cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f555bfff640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f555c145590 0x7f555c0cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f555bfff640 1 -- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f555c06f6e0 msgr2=0x7f555c1f1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f555bfff640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f555c06f6e0 0x7f555c1f1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f555bfff640 1 -- 172.21.15.59:0/3756903982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c1f2750 con 0x7f555c06b4f0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f555bfff640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f555c06b4f0 0x7f555c1f18f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f554c00e870 tx=0x7f554c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f5558ff9640 1 -- 172.21.15.59:0/3756903982 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c017070 con 0x7f555c06b4f0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f5558ff9640 1 -- 172.21.15.59:0/3756903982 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554c00c940 con 0x7f555c06b4f0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f55627ac640 1 -- 172.21.15.59:0/3756903982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555c0b48f0 con 0x7f555c06b4f0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f5558ff9640 1 -- 172.21.15.59:0/3756903982 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c00cd30 con 0x7f555c06b4f0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.788+0000 7f55627ac640 1 -- 172.21.15.59:0/3756903982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f555c1405f0 con 0x7f555c06b4f0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.789+0000 7f55627ac640 1 -- 172.21.15.59:0/3756903982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555c0b46e0 con 0x7f555c06b4f0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.789+0000 7f5558ff9640 1 -- 172.21.15.59:0/3756903982 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f554c01a070 con 0x7f555c06b4f0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.790+0000 7f5558ff9640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f554806b4d0 0x7f554806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.790+0000 7f5558ff9640 1 -- 172.21.15.59:0/3756903982 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f554c08a3d0 con 0x7f555c06b4f0 2024-03-06T07:22:34.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.790+0000 7f555b7fe640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f554806b4d0 0x7f554806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.790+0000 7f555b7fe640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f554806b4d0 0x7f554806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f555c13d4e0 tx=0x7f5550007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.794+0000 7f5558ff9640 1 -- 172.21.15.59:0/3756903982 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f554c058c00 con 0x7f555c06b4f0 2024-03-06T07:22:34.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.979+0000 7f55627ac640 1 -- 172.21.15.59:0/3756903982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f555c0733b0 con 0x7f555c06b4f0 2024-03-06T07:22:34.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.979+0000 7f5558ff9640 1 -- 172.21.15.59:0/3756903982 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f554c05cca0 con 0x7f555c06b4f0 2024-03-06T07:22:34.981 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:34.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 -- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f554806b4d0 msgr2=0x7f554806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f554806b4d0 0x7f554806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f555c13d4e0 tx=0x7f5550007d50 comp rx=0 tx=0).stop 2024-03-06T07:22:34.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 -- 172.21.15.59:0/3756903982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f555c06b4f0 msgr2=0x7f555c1f18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f555c06b4f0 0x7f555c1f18f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f554c00e870 tx=0x7f554c00ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:34.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 -- 172.21.15.59:0/3756903982 shutdown_connections 2024-03-06T07:22:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f554806b4d0 0x7f554806d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f555c145590 0x7f555c0cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f555c06f6e0 0x7f555c1f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 --2- 172.21.15.59:0/3756903982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f555c06b4f0 0x7f555c1f18f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 -- 172.21.15.59:0/3756903982 >> 172.21.15.59:0/3756903982 conn(0x7f555c0b2660 msgr2=0x7f555c06efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 -- 172.21.15.59:0/3756903982 shutdown_connections 2024-03-06T07:22:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:34.981+0000 7f55627ac640 1 -- 172.21.15.59:0/3756903982 wait complete. 2024-03-06T07:22:34.994 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:35.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.096+0000 7fbc4c488640 1 Processor -- start 2024-03-06T07:22:35.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.096+0000 7fbc4c488640 1 -- start start 2024-03-06T07:22:35.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4c488640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc4415bb50 0x7fbc441625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4c488640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbc4406e2c0 0x7fbc4406e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4c488640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc44072870 con 0x7fbc4415f140 2024-03-06T07:22:35.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4c488640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc44162160 con 0x7fbc4406e2c0 2024-03-06T07:22:35.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4c488640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc44072b50 con 0x7fbc4415bb50 2024-03-06T07:22:35.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc49a23640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbc4415f140 0x7fbc44071fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51336/0 (socket says 172.21.15.59:51336) 2024-03-06T07:22:35.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc49a23640 1 -- 172.21.15.59:0/1502068570 learned_addr learned my addr 172.21.15.59:0/1502068570 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:35.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4aa25640 1 --2- 172.21.15.59:0/1502068570 >> v2:172.21.15.164:3300/0 conn(0x7fbc4406e2c0 0x7fbc4406e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4a224640 1 --2- 172.21.15.59:0/1502068570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc4415bb50 0x7fbc441625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc49222640 1 -- 172.21.15.59:0/1502068570 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158672161 0 0) 0x7fbc44072870 con 0x7fbc4415f140 2024-03-06T07:22:35.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc49222640 1 -- 172.21.15.59:0/1502068570 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc30003520 con 0x7fbc4415f140 2024-03-06T07:22:35.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4aa25640 1 -- 172.21.15.59:0/1502068570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc4415bb50 msgr2=0x7fbc441625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4aa25640 1 --2- 172.21.15.59:0/1502068570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc4415bb50 0x7fbc441625f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4aa25640 1 -- 172.21.15.59:0/1502068570 >> v1:172.21.15.59:6789/0 conn(0x7fbc4415f140 legacy=0x7fbc44071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4aa25640 1 -- 172.21.15.59:0/1502068570 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc44161b10 con 0x7fbc4406e2c0 2024-03-06T07:22:35.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.097+0000 7fbc4a224640 1 --2- 172.21.15.59:0/1502068570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc4415bb50 0x7fbc441625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:35.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.098+0000 7fbc4aa25640 1 --2- 172.21.15.59:0/1502068570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc4406e2c0 0x7fbc4406e750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c009d90 tx=0x7fbc3c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9f6280b17a7efb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.098+0000 7fbc49222640 1 -- 172.21.15.59:0/1502068570 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c039070 con 0x7fbc4406e2c0 2024-03-06T07:22:35.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.098+0000 7fbc49222640 1 -- 172.21.15.59:0/1502068570 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c02cbc0 con 0x7fbc4406e2c0 2024-03-06T07:22:35.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.098+0000 7fbc49222640 1 -- 172.21.15.59:0/1502068570 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c034440 con 0x7fbc4406e2c0 2024-03-06T07:22:35.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.099+0000 7fbc4c488640 1 -- 172.21.15.59:0/1502068570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc4406e2c0 msgr2=0x7fbc4406e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.099+0000 7fbc4c488640 1 --2- 172.21.15.59:0/1502068570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc4406e2c0 0x7fbc4406e750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c009d90 tx=0x7fbc3c02be10 comp rx=0 tx=0).stop 2024-03-06T07:22:35.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.099+0000 7fbc4c488640 1 -- 172.21.15.59:0/1502068570 shutdown_connections 2024-03-06T07:22:35.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.099+0000 7fbc4c488640 1 --2- 172.21.15.59:0/1502068570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc4406e2c0 0x7fbc4406e750 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.099+0000 7fbc4c488640 1 --2- 172.21.15.59:0/1502068570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc4415bb50 0x7fbc441625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.099+0000 7fbc4c488640 1 -- 172.21.15.59:0/1502068570 >> 172.21.15.59:0/1502068570 conn(0x7fbc4406b830 msgr2=0x7fbc440b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.099+0000 7fbc4c488640 1 -- 172.21.15.59:0/1502068570 shutdown_connections 2024-03-06T07:22:35.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.099+0000 7fbc4c488640 1 -- 172.21.15.59:0/1502068570 wait complete. 2024-03-06T07:22:35.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.100+0000 7fbc4c488640 1 Processor -- start 2024-03-06T07:22:35.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.100+0000 7fbc4c488640 1 -- start start 2024-03-06T07:22:35.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc4c488640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc4406e2c0 0x7fbc441473d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc4c488640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc4415bb50 0x7fbc44147ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc4c488640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc4415f140 0x7fbc440c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc4c488640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc440ad9e0 con 0x7fbc4415bb50 2024-03-06T07:22:35.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc4c488640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc440ad860 con 0x7fbc4415f140 2024-03-06T07:22:35.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc4c488640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc440adb60 con 0x7fbc4406e2c0 2024-03-06T07:22:35.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc49a23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc4415bb50 0x7fbc44147ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc49a23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc4415bb50 0x7fbc44147ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58154/0 (socket says 172.21.15.59:58154) 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc49a23640 1 -- 172.21.15.59:0/607033890 learned_addr learned my addr 172.21.15.59:0/607033890 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc4aa25640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc4415f140 0x7fbc440c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc4a224640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc4406e2c0 0x7fbc441473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc49a23640 1 -- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc4406e2c0 msgr2=0x7fbc441473d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc49a23640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc4406e2c0 0x7fbc441473d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc49a23640 1 -- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc4415f140 msgr2=0x7fbc440c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc49a23640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc4415f140 0x7fbc440c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc49a23640 1 -- 172.21.15.59:0/607033890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc44148520 con 0x7fbc4415bb50 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc49a23640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc4415bb50 0x7fbc44147ad0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400c390 tx=0x7fbc3400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.101+0000 7fbc3affd640 1 -- 172.21.15.59:0/607033890 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34017070 con 0x7fbc4415bb50 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.102+0000 7fbc3affd640 1 -- 172.21.15.59:0/607033890 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc34005ce0 con 0x7fbc4415bb50 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.102+0000 7fbc3affd640 1 -- 172.21.15.59:0/607033890 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc340072f0 con 0x7fbc4415bb50 2024-03-06T07:22:35.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.102+0000 7fbc4c488640 1 -- 172.21.15.59:0/607033890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc44148800 con 0x7fbc4415bb50 2024-03-06T07:22:35.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.102+0000 7fbc4c488640 1 -- 172.21.15.59:0/607033890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc44148dc0 con 0x7fbc4415bb50 2024-03-06T07:22:35.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.103+0000 7fbc4c488640 1 -- 172.21.15.59:0/607033890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4406e0b0 con 0x7fbc4415bb50 2024-03-06T07:22:35.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.103+0000 7fbc3affd640 1 -- 172.21.15.59:0/607033890 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc3401a070 con 0x7fbc4415bb50 2024-03-06T07:22:35.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.103+0000 7fbc3affd640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 0x7fbc2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.103+0000 7fbc3affd640 1 -- 172.21.15.59:0/607033890 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc34089550 con 0x7fbc4415bb50 2024-03-06T07:22:35.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.104+0000 7fbc4a224640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 0x7fbc2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.104+0000 7fbc4a224640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 0x7fbc2806d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbc4415a280 tx=0x7fbc40009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.108+0000 7fbc3affd640 1 -- 172.21.15.59:0/607033890 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc34057d80 con 0x7fbc4415bb50 2024-03-06T07:22:35.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.290+0000 7fbc4c488640 1 -- 172.21.15.59:0/607033890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbc440b20a0 con 0x7fbc4415bb50 2024-03-06T07:22:35.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.290+0000 7fbc3affd640 1 -- 172.21.15.59:0/607033890 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbc3405be20 con 0x7fbc4415bb50 2024-03-06T07:22:35.292 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 -- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 msgr2=0x7fbc2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 0x7fbc2806d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbc4415a280 tx=0x7fbc40009190 comp rx=0 tx=0).stop 2024-03-06T07:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 -- 172.21.15.59:0/607033890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc4415bb50 msgr2=0x7fbc44147ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc4415bb50 0x7fbc44147ad0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400c390 tx=0x7fbc3400c850 comp rx=0 tx=0).stop 2024-03-06T07:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 -- 172.21.15.59:0/607033890 shutdown_connections 2024-03-06T07:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc2806b4d0 0x7fbc2806d990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc4415f140 0x7fbc440c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc4415bb50 0x7fbc44147ad0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 --2- 172.21.15.59:0/607033890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc4406e2c0 0x7fbc441473d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 -- 172.21.15.59:0/607033890 >> 172.21.15.59:0/607033890 conn(0x7fbc4406b830 msgr2=0x7fbc440adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 -- 172.21.15.59:0/607033890 shutdown_connections 2024-03-06T07:22:35.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.292+0000 7fbc4c488640 1 -- 172.21.15.59:0/607033890 wait complete. 2024-03-06T07:22:35.305 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:35.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.405+0000 7f4642c83640 1 Processor -- start 2024-03-06T07:22:35.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.405+0000 7f4642c83640 1 -- start start 2024-03-06T07:22:35.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f4642c83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c0d1ad0 0x7f463c0d1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f4642c83640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463c15e3f0 con 0x7f463c0b3240 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f4642c83640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463c0b1c10 con 0x7f463c15e010 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f4642c83640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463c0b1f40 con 0x7f463c0d1ad0 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f463bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f463c0b3240 0x7f463c0b23d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51348/0 (socket says 172.21.15.59:51348) 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f463bfff640 1 -- 172.21.15.59:0/2867525921 learned_addr learned my addr 172.21.15.59:0/2867525921 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f4641220640 1 --2- 172.21.15.59:0/2867525921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c0d1ad0 0x7f463c0d1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f463b7fe640 1 -- 172.21.15.59:0/2867525921 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789806065 0 0) 0x7f463c15e3f0 con 0x7f463c0b3240 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f463b7fe640 1 -- 172.21.15.59:0/2867525921 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4624003520 con 0x7f463c0b3240 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f463b7fe640 1 -- 172.21.15.59:0/2867525921 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961139491 0 0) 0x7f463c0b1c10 con 0x7f463c15e010 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f463b7fe640 1 -- 172.21.15.59:0/2867525921 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f463c15e3f0 con 0x7f463c15e010 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f4641220640 1 -- 172.21.15.59:0/2867525921 >> v1:172.21.15.164:6789/0 conn(0x7f463c15e010 legacy=0x7f463c0cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f4641220640 1 -- 172.21.15.59:0/2867525921 >> v1:172.21.15.59:6789/0 conn(0x7f463c0b3240 legacy=0x7f463c0b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.406+0000 7f4641220640 1 -- 172.21.15.59:0/2867525921 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c06d1b0 con 0x7f463c0d1ad0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.407+0000 7f4641220640 1 --2- 172.21.15.59:0/2867525921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c0d1ad0 0x7f463c0d1f80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f463c0b1de0 tx=0x7f462c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdafef88fc71bc13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.407+0000 7f463b7fe640 1 -- 172.21.15.59:0/2867525921 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c035020 con 0x7f463c0d1ad0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.407+0000 7f463b7fe640 1 -- 172.21.15.59:0/2867525921 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462c031800 con 0x7f463c0d1ad0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.407+0000 7f463b7fe640 1 -- 172.21.15.59:0/2867525921 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c031bb0 con 0x7f463c0d1ad0 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.408+0000 7f4642c83640 1 -- 172.21.15.59:0/2867525921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c0d1ad0 msgr2=0x7f463c0d1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.408+0000 7f4642c83640 1 --2- 172.21.15.59:0/2867525921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c0d1ad0 0x7f463c0d1f80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f463c0b1de0 tx=0x7f462c02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.408+0000 7f4642c83640 1 -- 172.21.15.59:0/2867525921 shutdown_connections 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.408+0000 7f4642c83640 1 --2- 172.21.15.59:0/2867525921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c0d1ad0 0x7f463c0d1f80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.408+0000 7f4642c83640 1 -- 172.21.15.59:0/2867525921 >> 172.21.15.59:0/2867525921 conn(0x7f463c09a9f0 msgr2=0x7f463c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.408+0000 7f4642c83640 1 -- 172.21.15.59:0/2867525921 shutdown_connections 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.408+0000 7f4642c83640 1 -- 172.21.15.59:0/2867525921 wait complete. 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.409+0000 7f4642c83640 1 Processor -- start 2024-03-06T07:22:35.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.409+0000 7f4642c83640 1 -- start start 2024-03-06T07:22:35.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f4642c83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c0d1ad0 0x7f463c14c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f4642c83640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f463c15e010 0x7f463c0c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f4642c83640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0c5670 0x7f463c0c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f4642c83640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c06ced0 con 0x7f463c15e010 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f4642c83640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c06cbf0 con 0x7f463c0c5670 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f4642c83640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463c0cfaa0 con 0x7f463c0d1ad0 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f463bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f463c15e010 0x7f463c0c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f463bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f463c15e010 0x7f463c0c71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58164/0 (socket says 172.21.15.59:58164) 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f463bfff640 1 -- 172.21.15.59:0/3049487537 learned_addr learned my addr 172.21.15.59:0/3049487537 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f4640a1f640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c0d1ad0 0x7f463c14c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f4641220640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0c5670 0x7f463c0c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f463bfff640 1 -- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c0d1ad0 msgr2=0x7f463c14c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f463bfff640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c0d1ad0 0x7f463c14c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f463bfff640 1 -- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0c5670 msgr2=0x7f463c0c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f463bfff640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0c5670 0x7f463c0c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.410+0000 7f463bfff640 1 -- 172.21.15.59:0/3049487537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c0c6130 con 0x7f463c15e010 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.411+0000 7f463bfff640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f463c15e010 0x7f463c0c71d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f463000c500 tx=0x7f463000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.411+0000 7f46397fa640 1 -- 172.21.15.59:0/3049487537 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4630017070 con 0x7f463c15e010 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.411+0000 7f46397fa640 1 -- 172.21.15.59:0/3049487537 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4630005ce0 con 0x7f463c15e010 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.411+0000 7f4642c83640 1 -- 172.21.15.59:0/3049487537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463c0c6410 con 0x7f463c15e010 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.411+0000 7f46397fa640 1 -- 172.21.15.59:0/3049487537 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46300072f0 con 0x7f463c15e010 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.411+0000 7f4642c83640 1 -- 172.21.15.59:0/3049487537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463c0c69d0 con 0x7f463c15e010 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.412+0000 7f4642c83640 1 -- 172.21.15.59:0/3049487537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463c04f330 con 0x7f463c15e010 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.412+0000 7f46397fa640 1 -- 172.21.15.59:0/3049487537 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f463001a070 con 0x7f463c15e010 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.413+0000 7f46397fa640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f462806b500 0x7f462806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.413+0000 7f46397fa640 1 -- 172.21.15.59:0/3049487537 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46300894c0 con 0x7f463c15e010 2024-03-06T07:22:35.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.413+0000 7f4640a1f640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f462806b500 0x7f462806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.413+0000 7f4640a1f640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f462806b500 0x7f462806d9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4634001e00 tx=0x7f4634009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.417+0000 7f46397fa640 1 -- 172.21.15.59:0/3049487537 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4630057d80 con 0x7f463c15e010 2024-03-06T07:22:35.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.618+0000 7f4642c83640 1 -- 172.21.15.59:0/3049487537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f463c0d2980 con 0x7f463c15e010 2024-03-06T07:22:35.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.618+0000 7f46397fa640 1 -- 172.21.15.59:0/3049487537 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f463005e0d0 con 0x7f463c15e010 2024-03-06T07:22:35.620 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:22:35.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 -- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f462806b500 msgr2=0x7f462806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f462806b500 0x7f462806d9c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4634001e00 tx=0x7f4634009040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 -- 172.21.15.59:0/3049487537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f463c15e010 msgr2=0x7f463c0c71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f463c15e010 0x7f463c0c71d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f463000c500 tx=0x7f463000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 -- 172.21.15.59:0/3049487537 shutdown_connections 2024-03-06T07:22:35.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f462806b500 0x7f462806d9c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0c5670 0x7f463c0c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f463c15e010 0x7f463c0c71d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 --2- 172.21.15.59:0/3049487537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c0d1ad0 0x7f463c14c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 -- 172.21.15.59:0/3049487537 >> 172.21.15.59:0/3049487537 conn(0x7f463c09a9f0 msgr2=0x7f463c0ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 -- 172.21.15.59:0/3049487537 shutdown_connections 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.620+0000 7f4642c83640 1 -- 172.21.15.59:0/3049487537 wait complete. 2024-03-06T07:22:35.632 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:22:35.633 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:35.649 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:35.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.748+0000 7fa72b881640 1 Processor -- start 2024-03-06T07:22:35.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.749+0000 7fa72b881640 1 -- start start 2024-03-06T07:22:35.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.749+0000 7fa72b881640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa7240d2710 0x7fa7240b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.749+0000 7fa72b881640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7240b3430 con 0x7fa72415dfe0 2024-03-06T07:22:35.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.749+0000 7fa72b881640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7240b3710 con 0x7fa7240d22b0 2024-03-06T07:22:35.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.749+0000 7fa72b881640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7240b31e0 con 0x7fa7240d2710 2024-03-06T07:22:35.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.749+0000 7fa729e1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa7240d2710 0x7fa7240b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.749+0000 7fa72961d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa72415dfe0 0x7fa7240cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51356/0 (socket says 172.21.15.59:51356) 2024-03-06T07:22:35.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.749+0000 7fa729e1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa7240d2710 0x7fa7240b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58166/0 (socket says 172.21.15.59:58166) 2024-03-06T07:22:35.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.749+0000 7fa72961d640 1 -- 172.21.15.59:0/971345543 learned_addr learned my addr 172.21.15.59:0/971345543 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:35.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.750+0000 7fa729e1e640 1 -- 172.21.15.59:0/971345543 >> v1:172.21.15.164:6789/0 conn(0x7fa7240d22b0 legacy=0x7fa7240b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.750+0000 7fa729e1e640 1 -- 172.21.15.59:0/971345543 >> v1:172.21.15.59:6789/0 conn(0x7fa72415dfe0 legacy=0x7fa7240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.750+0000 7fa729e1e640 1 -- 172.21.15.59:0/971345543 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7240b39f0 con 0x7fa7240d2710 2024-03-06T07:22:35.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.750+0000 7fa71bfff640 1 -- 172.21.15.59:0/971345543 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739562645 0 0) 0x7fa7240b3430 con 0x7fa72415dfe0 2024-03-06T07:22:35.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.750+0000 7fa729e1e640 1 --2- 172.21.15.59:0/971345543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7240d2710 0x7fa7240b4900 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa720009dc0 tx=0x7fa72002bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7955f81212cb42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.751+0000 7fa71bfff640 1 -- 172.21.15.59:0/971345543 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720039070 con 0x7fa7240d2710 2024-03-06T07:22:35.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.751+0000 7fa71bfff640 1 -- 172.21.15.59:0/971345543 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72002ccd0 con 0x7fa7240d2710 2024-03-06T07:22:35.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.751+0000 7fa71bfff640 1 -- 172.21.15.59:0/971345543 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720034660 con 0x7fa7240d2710 2024-03-06T07:22:35.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.751+0000 7fa72b881640 1 -- 172.21.15.59:0/971345543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7240d2710 msgr2=0x7fa7240b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.751+0000 7fa72b881640 1 --2- 172.21.15.59:0/971345543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7240d2710 0x7fa7240b4900 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa720009dc0 tx=0x7fa72002bf20 comp rx=0 tx=0).stop 2024-03-06T07:22:35.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.751+0000 7fa72b881640 1 -- 172.21.15.59:0/971345543 shutdown_connections 2024-03-06T07:22:35.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.751+0000 7fa72b881640 1 --2- 172.21.15.59:0/971345543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7240d2710 0x7fa7240b4900 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.751+0000 7fa72b881640 1 -- 172.21.15.59:0/971345543 >> 172.21.15.59:0/971345543 conn(0x7fa7240d3cd0 msgr2=0x7fa7240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.751+0000 7fa72b881640 1 -- 172.21.15.59:0/971345543 shutdown_connections 2024-03-06T07:22:35.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.751+0000 7fa72b881640 1 -- 172.21.15.59:0/971345543 wait complete. 2024-03-06T07:22:35.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.752+0000 7fa72b881640 1 Processor -- start 2024-03-06T07:22:35.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.752+0000 7fa72b881640 1 -- start start 2024-03-06T07:22:35.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa72b881640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7240d22b0 0x7fa724157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa72b881640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7240d2710 0x7fa7240b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa72b881640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa72415dfe0 0x7fa7240b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa72b881640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7240cfa00 con 0x7fa7240d2710 2024-03-06T07:22:35.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa72b881640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7240b3d60 con 0x7fa72415dfe0 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa72b881640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7240d1c70 con 0x7fa7240d22b0 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa728e1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7240d2710 0x7fa7240b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa728e1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7240d2710 0x7fa7240b1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58168/0 (socket says 172.21.15.59:58168) 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa728e1c640 1 -- 172.21.15.59:0/1437881956 learned_addr learned my addr 172.21.15.59:0/1437881956 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa72961d640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7240d22b0 0x7fa724157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa729e1e640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa72415dfe0 0x7fa7240b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa728e1c640 1 -- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7240d22b0 msgr2=0x7fa724157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa728e1c640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7240d22b0 0x7fa724157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa728e1c640 1 -- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa72415dfe0 msgr2=0x7fa7240b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa728e1c640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa72415dfe0 0x7fa7240b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.753+0000 7fa728e1c640 1 -- 172.21.15.59:0/1437881956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7240b8bb0 con 0x7fa7240d2710 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.754+0000 7fa728e1c640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7240d2710 0x7fa7240b1010 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa71400c480 tx=0x7fa71400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.754+0000 7fa719ffb640 1 -- 172.21.15.59:0/1437881956 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7140052d0 con 0x7fa7240d2710 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.754+0000 7fa719ffb640 1 -- 172.21.15.59:0/1437881956 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa714005470 con 0x7fa7240d2710 2024-03-06T07:22:35.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.754+0000 7fa72b881640 1 -- 172.21.15.59:0/1437881956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7240b8e90 con 0x7fa7240d2710 2024-03-06T07:22:35.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.754+0000 7fa719ffb640 1 -- 172.21.15.59:0/1437881956 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714012870 con 0x7fa7240d2710 2024-03-06T07:22:35.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.754+0000 7fa72b881640 1 -- 172.21.15.59:0/1437881956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7240b9450 con 0x7fa7240d2710 2024-03-06T07:22:35.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.755+0000 7fa72b881640 1 -- 172.21.15.59:0/1437881956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72404f330 con 0x7fa7240d2710 2024-03-06T07:22:35.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.755+0000 7fa719ffb640 1 -- 172.21.15.59:0/1437881956 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa714012a10 con 0x7fa7240d2710 2024-03-06T07:22:35.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.756+0000 7fa719ffb640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa70806b4d0 0x7fa70806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.756+0000 7fa719ffb640 1 -- 172.21.15.59:0/1437881956 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa714089d40 con 0x7fa7240d2710 2024-03-06T07:22:35.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.756+0000 7fa72961d640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa70806b4d0 0x7fa70806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.756+0000 7fa72961d640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa70806b4d0 0x7fa70806d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa71c000f60 tx=0x7fa71c007e40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.760+0000 7fa719ffb640 1 -- 172.21.15.59:0/1437881956 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa714058580 con 0x7fa7240d2710 2024-03-06T07:22:35.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.943+0000 7fa72b881640 1 -- 172.21.15.59:0/1437881956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7240ce960 con 0x7fa7240d2710 2024-03-06T07:22:35.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.944+0000 7fa719ffb640 1 -- 172.21.15.59:0/1437881956 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16842 (secure 0 0 0) 0x7fa71405c620 con 0x7fa7240d2710 2024-03-06T07:22:35.947 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:35.947 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":21,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:35.531606+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":1}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:35.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 -- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa70806b4d0 msgr2=0x7fa70806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa70806b4d0 0x7fa70806d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa71c000f60 tx=0x7fa71c007e40 comp rx=0 tx=0).stop 2024-03-06T07:22:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 -- 172.21.15.59:0/1437881956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7240d2710 msgr2=0x7fa7240b1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7240d2710 0x7fa7240b1010 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa71400c480 tx=0x7fa71400c940 comp rx=0 tx=0).stop 2024-03-06T07:22:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 -- 172.21.15.59:0/1437881956 shutdown_connections 2024-03-06T07:22:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa70806b4d0 0x7fa70806d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa72415dfe0 0x7fa7240b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7240d2710 0x7fa7240b1010 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 --2- 172.21.15.59:0/1437881956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7240d22b0 0x7fa724157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 -- 172.21.15.59:0/1437881956 >> 172.21.15.59:0/1437881956 conn(0x7fa7240d3cd0 msgr2=0x7fa7240ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 -- 172.21.15.59:0/1437881956 shutdown_connections 2024-03-06T07:22:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:35.946+0000 7fa72b881640 1 -- 172.21.15.59:0/1437881956 wait complete. 2024-03-06T07:22:35.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:35.960 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:36.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.062+0000 7f116e5e9640 1 Processor -- start 2024-03-06T07:22:36.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.062+0000 7f116e5e9640 1 -- start start 2024-03-06T07:22:36.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f116e5e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f116815bb40 0x7f11680b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f116e5e9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f116815f7d0 0x7f116806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f116e5e9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116815ff90 con 0x7f116806d5c0 2024-03-06T07:22:36.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f116e5e9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f116806c910 con 0x7f116815f7d0 2024-03-06T07:22:36.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f116e5e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1168160270 con 0x7f116815bb40 2024-03-06T07:22:36.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f116cb86640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f116806d5c0 0x7f11680cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51358/0 (socket says 172.21.15.59:51358) 2024-03-06T07:22:36.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f116cb86640 1 -- 172.21.15.59:0/3859068914 learned_addr learned my addr 172.21.15.59:0/3859068914 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:36.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f1167fff640 1 --2- 172.21.15.59:0/3859068914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f116815bb40 0x7f11680b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f11677fe640 1 --2- 172.21.15.59:0/3859068914 >> v2:172.21.15.164:3300/0 conn(0x7f116815f7d0 0x7f116806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f1166ffd640 1 -- 172.21.15.59:0/3859068914 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055333483 0 0) 0x7f116815ff90 con 0x7f116806d5c0 2024-03-06T07:22:36.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f1166ffd640 1 -- 172.21.15.59:0/3859068914 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1154003520 con 0x7f116806d5c0 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f1167fff640 1 -- 172.21.15.59:0/3859068914 >> v2:172.21.15.164:3300/0 conn(0x7f116815f7d0 msgr2=0x7f116806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f1167fff640 1 --2- 172.21.15.59:0/3859068914 >> v2:172.21.15.164:3300/0 conn(0x7f116815f7d0 0x7f116806d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f1167fff640 1 -- 172.21.15.59:0/3859068914 >> v1:172.21.15.59:6789/0 conn(0x7f116806d5c0 legacy=0x7f11680cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f1167fff640 1 -- 172.21.15.59:0/3859068914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1168160550 con 0x7f116815bb40 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.063+0000 7f11677fe640 1 --2- 172.21.15.59:0/3859068914 >> v2:172.21.15.164:3300/0 conn(0x7f116815f7d0 0x7f116806d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.064+0000 7f1167fff640 1 --2- 172.21.15.59:0/3859068914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f116815bb40 0x7f11680b4970 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f11580097a0 tx=0x7f115802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eefad0296d0ed41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.064+0000 7f1166ffd640 1 -- 172.21.15.59:0/3859068914 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158035020 con 0x7f116815bb40 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.064+0000 7f1166ffd640 1 -- 172.21.15.59:0/3859068914 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1158031830 con 0x7f116815bb40 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.064+0000 7f1166ffd640 1 -- 172.21.15.59:0/3859068914 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1158031c00 con 0x7f116815bb40 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.065+0000 7f116e5e9640 1 -- 172.21.15.59:0/3859068914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f116815bb40 msgr2=0x7f11680b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.065+0000 7f116e5e9640 1 --2- 172.21.15.59:0/3859068914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f116815bb40 0x7f11680b4970 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f11580097a0 tx=0x7f115802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.065+0000 7f116e5e9640 1 -- 172.21.15.59:0/3859068914 shutdown_connections 2024-03-06T07:22:36.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.065+0000 7f116e5e9640 1 --2- 172.21.15.59:0/3859068914 >> v2:172.21.15.164:3300/0 conn(0x7f116815f7d0 0x7f116806d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.065+0000 7f116e5e9640 1 --2- 172.21.15.59:0/3859068914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f116815bb40 0x7f11680b4970 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.065+0000 7f116e5e9640 1 -- 172.21.15.59:0/3859068914 >> 172.21.15.59:0/3859068914 conn(0x7f116806b190 msgr2=0x7f116806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.065+0000 7f116e5e9640 1 -- 172.21.15.59:0/3859068914 shutdown_connections 2024-03-06T07:22:36.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.065+0000 7f116e5e9640 1 -- 172.21.15.59:0/3859068914 wait complete. 2024-03-06T07:22:36.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.066+0000 7f116e5e9640 1 Processor -- start 2024-03-06T07:22:36.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.066+0000 7f116e5e9640 1 -- start start 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116e5e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f116806d5c0 0x7f11681f1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116e5e9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f116815bb40 0x7f11681f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116e5e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f116815f7d0 0x7f116815c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116e5e9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116815bf20 con 0x7f116815f7d0 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116e5e9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116815fc30 con 0x7f116815bb40 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116e5e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116815c0a0 con 0x7f116806d5c0 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116cb86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f116815f7d0 0x7f116815c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116cb86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f116815f7d0 0x7f116815c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58180/0 (socket says 172.21.15.59:58180) 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116cb86640 1 -- 172.21.15.59:0/1769845273 learned_addr learned my addr 172.21.15.59:0/1769845273 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f1167fff640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f116806d5c0 0x7f11681f1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f11677fe640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f116815bb40 0x7f11681f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116cb86640 1 -- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f116806d5c0 msgr2=0x7f11681f1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116cb86640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f116806d5c0 0x7f11681f1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116cb86640 1 -- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f116815bb40 msgr2=0x7f11681f2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116cb86640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f116815bb40 0x7f11681f2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.067+0000 7f116cb86640 1 -- 172.21.15.59:0/1769845273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11681f1970 con 0x7f116815f7d0 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.068+0000 7f116cb86640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f116815f7d0 0x7f116815c720 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f116000cb40 tx=0x7f116000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.068+0000 7f1164ff9640 1 -- 172.21.15.59:0/1769845273 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160017070 con 0x7f116815f7d0 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.068+0000 7f1164ff9640 1 -- 172.21.15.59:0/1769845273 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116000ac40 con 0x7f116815f7d0 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.068+0000 7f116e5e9640 1 -- 172.21.15.59:0/1769845273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116815db20 con 0x7f116815f7d0 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.068+0000 7f1164ff9640 1 -- 172.21.15.59:0/1769845273 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160012400 con 0x7f116815f7d0 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.068+0000 7f116e5e9640 1 -- 172.21.15.59:0/1769845273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f116815e0e0 con 0x7f116815f7d0 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.069+0000 7f116e5e9640 1 -- 172.21.15.59:0/1769845273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11680732b0 con 0x7f116815f7d0 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.069+0000 7f1164ff9640 1 -- 172.21.15.59:0/1769845273 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f116001a070 con 0x7f116815f7d0 2024-03-06T07:22:36.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.070+0000 7f1164ff9640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f114806b4d0 0x7f114806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.070+0000 7f1164ff9640 1 -- 172.21.15.59:0/1769845273 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1160089f50 con 0x7f116815f7d0 2024-03-06T07:22:36.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.070+0000 7f1167fff640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f114806b4d0 0x7f114806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.070+0000 7f1167fff640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f114806b4d0 0x7f114806d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1158036000 tx=0x7f1158008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.074+0000 7f1164ff9640 1 -- 172.21.15.59:0/1769845273 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1160058700 con 0x7f116815f7d0 2024-03-06T07:22:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.253+0000 7f116e5e9640 1 -- 172.21.15.59:0/1769845273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1168068ff0 con 0x7f116815f7d0 2024-03-06T07:22:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.253+0000 7f1164ff9640 1 -- 172.21.15.59:0/1769845273 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f116005c7a0 con 0x7f116815f7d0 2024-03-06T07:22:36.255 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:36.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 -- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f114806b4d0 msgr2=0x7f114806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f114806b4d0 0x7f114806d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1158036000 tx=0x7f1158008030 comp rx=0 tx=0).stop 2024-03-06T07:22:36.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 -- 172.21.15.59:0/1769845273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f116815f7d0 msgr2=0x7f116815c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f116815f7d0 0x7f116815c720 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f116000cb40 tx=0x7f116000a060 comp rx=0 tx=0).stop 2024-03-06T07:22:36.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 -- 172.21.15.59:0/1769845273 shutdown_connections 2024-03-06T07:22:36.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f114806b4d0 0x7f114806d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f116815f7d0 0x7f116815c720 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f116815bb40 0x7f11681f2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 --2- 172.21.15.59:0/1769845273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f116806d5c0 0x7f11681f1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 -- 172.21.15.59:0/1769845273 >> 172.21.15.59:0/1769845273 conn(0x7f116806b190 msgr2=0x7f11680ceb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 -- 172.21.15.59:0/1769845273 shutdown_connections 2024-03-06T07:22:36.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.255+0000 7f116e5e9640 1 -- 172.21.15.59:0/1769845273 wait complete. 2024-03-06T07:22:36.268 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:36.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.370+0000 7f7a3370c640 1 Processor -- start 2024-03-06T07:22:36.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.370+0000 7f7a3370c640 1 -- start start 2024-03-06T07:22:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a3370c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7a2c15def0 0x7f7a2c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a3370c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a2c0cea50 0x7f7a2c0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a3370c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c06d030 con 0x7f7a2c06cab0 2024-03-06T07:22:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a3370c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c0cf1c0 con 0x7f7a2c15def0 2024-03-06T07:22:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a3370c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c0cc760 con 0x7f7a2c0cea50 2024-03-06T07:22:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a31ca9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a2c0cea50 0x7f7a2c0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a30ca7640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7a2c06cab0 0x7f7a2c0ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:34478/0 (socket says 172.21.15.59:34478) 2024-03-06T07:22:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a314a8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7a2c15def0 0x7f7a2c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a30ca7640 1 -- 172.21.15.59:0/3375891347 learned_addr learned my addr 172.21.15.59:0/3375891347 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a314a8640 1 -- 172.21.15.59:0/3375891347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a2c0cea50 msgr2=0x7f7a2c0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a314a8640 1 --2- 172.21.15.59:0/3375891347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a2c0cea50 0x7f7a2c0ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a314a8640 1 -- 172.21.15.59:0/3375891347 >> v1:172.21.15.164:6789/0 conn(0x7f7a2c06cab0 legacy=0x7f7a2c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a314a8640 1 -- 172.21.15.59:0/3375891347 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c06d310 con 0x7f7a2c15def0 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a2bfff640 1 -- 172.21.15.59:0/3375891347 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457133975 0 0) 0x7f7a2c06d030 con 0x7f7a2c06cab0 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.371+0000 7f7a31ca9640 1 --2- 172.21.15.59:0/3375891347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a2c0cea50 0x7f7a2c0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.372+0000 7f7a314a8640 1 --2- 172.21.15.59:0/3375891347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c15def0 0x7f7a2c0b41b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009d90 tx=0x7f7a1c02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea97513cbe5333c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.372+0000 7f7a2bfff640 1 -- 172.21.15.59:0/3375891347 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c039070 con 0x7f7a2c15def0 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.372+0000 7f7a2bfff640 1 -- 172.21.15.59:0/3375891347 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c02cce0 con 0x7f7a2c15def0 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.372+0000 7f7a2bfff640 1 -- 172.21.15.59:0/3375891347 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c034660 con 0x7f7a2c15def0 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.373+0000 7f7a3370c640 1 -- 172.21.15.59:0/3375891347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c15def0 msgr2=0x7f7a2c0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.373+0000 7f7a3370c640 1 --2- 172.21.15.59:0/3375891347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c15def0 0x7f7a2c0b41b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009d90 tx=0x7f7a1c02bf30 comp rx=0 tx=0).stop 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.373+0000 7f7a3370c640 1 -- 172.21.15.59:0/3375891347 shutdown_connections 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.373+0000 7f7a3370c640 1 --2- 172.21.15.59:0/3375891347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a2c0cea50 0x7f7a2c0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.373+0000 7f7a3370c640 1 --2- 172.21.15.59:0/3375891347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c15def0 0x7f7a2c0b41b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.373+0000 7f7a3370c640 1 -- 172.21.15.59:0/3375891347 >> 172.21.15.59:0/3375891347 conn(0x7f7a2c0b3690 msgr2=0x7f7a2c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.373+0000 7f7a3370c640 1 -- 172.21.15.59:0/3375891347 shutdown_connections 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.373+0000 7f7a3370c640 1 -- 172.21.15.59:0/3375891347 wait complete. 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.374+0000 7f7a3370c640 1 Processor -- start 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.374+0000 7f7a3370c640 1 -- start start 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a3370c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c06cab0 0x7f7a2c1f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a3370c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a2c0cea50 0x7f7a2c1f6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a3370c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a2c15def0 0x7f7a2c0c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a3370c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c15e2d0 con 0x7f7a2c15def0 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a3370c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c0cc3f0 con 0x7f7a2c06cab0 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a3370c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c15e450 con 0x7f7a2c0cea50 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a31ca9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a2c15def0 0x7f7a2c0c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a314a8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c06cab0 0x7f7a2c1f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a30ca7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a2c0cea50 0x7f7a2c1f6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a31ca9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a2c15def0 0x7f7a2c0c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58192/0 (socket says 172.21.15.59:58192) 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a314a8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c06cab0 0x7f7a2c1f3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:45128/0 (socket says 172.21.15.59:45128) 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a31ca9640 1 -- 172.21.15.59:0/4033180140 learned_addr learned my addr 172.21.15.59:0/4033180140 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a31ca9640 1 -- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a2c0cea50 msgr2=0x7f7a2c1f6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a31ca9640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a2c0cea50 0x7f7a2c1f6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a31ca9640 1 -- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c06cab0 msgr2=0x7f7a2c1f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a31ca9640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c06cab0 0x7f7a2c1f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.375+0000 7f7a31ca9640 1 -- 172.21.15.59:0/4033180140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c1f4260 con 0x7f7a2c15def0 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.376+0000 7f7a31ca9640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a2c15def0 0x7f7a2c0c5f50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400e8f0 tx=0x7f7a2400edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.376+0000 7f7a29ffb640 1 -- 172.21.15.59:0/4033180140 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24017070 con 0x7f7a2c15def0 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.376+0000 7f7a29ffb640 1 -- 172.21.15.59:0/4033180140 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2400c940 con 0x7f7a2c15def0 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.376+0000 7f7a3370c640 1 -- 172.21.15.59:0/4033180140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c0c68c0 con 0x7f7a2c15def0 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.376+0000 7f7a29ffb640 1 -- 172.21.15.59:0/4033180140 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2400cd10 con 0x7f7a2c15def0 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.376+0000 7f7a3370c640 1 -- 172.21.15.59:0/4033180140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c0b4c00 con 0x7f7a2c15def0 2024-03-06T07:22:36.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.377+0000 7f7a3370c640 1 -- 172.21.15.59:0/4033180140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2c0cbec0 con 0x7f7a2c15def0 2024-03-06T07:22:36.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.377+0000 7f7a29ffb640 1 -- 172.21.15.59:0/4033180140 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a2401a070 con 0x7f7a2c15def0 2024-03-06T07:22:36.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.378+0000 7f7a29ffb640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a1806b4d0 0x7f7a1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.378+0000 7f7a29ffb640 1 -- 172.21.15.59:0/4033180140 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7a2408a100 con 0x7f7a2c15def0 2024-03-06T07:22:36.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.378+0000 7f7a314a8640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a1806b4d0 0x7f7a1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.378+0000 7f7a314a8640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a1806b4d0 0x7f7a1806d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c036000 tx=0x7f7a1c030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.382+0000 7f7a29ffb640 1 -- 172.21.15.59:0/4033180140 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a2c0cbec0 con 0x7f7a2c15def0 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.566+0000 7f7a3370c640 1 -- 172.21.15.59:0/4033180140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7a2c09fcf0 con 0x7f7a2c15def0 2024-03-06T07:22:36.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.567+0000 7f7a29ffb640 1 -- 172.21.15.59:0/4033180140 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7a240588c0 con 0x7f7a2c15def0 2024-03-06T07:22:36.569 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:36.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.568+0000 7f7a3370c640 1 -- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a1806b4d0 msgr2=0x7f7a1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.568+0000 7f7a3370c640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a1806b4d0 0x7f7a1806d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c036000 tx=0x7f7a1c030000 comp rx=0 tx=0).stop 2024-03-06T07:22:36.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.568+0000 7f7a3370c640 1 -- 172.21.15.59:0/4033180140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a2c15def0 msgr2=0x7f7a2c0c5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.568+0000 7f7a3370c640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a2c15def0 0x7f7a2c0c5f50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400e8f0 tx=0x7f7a2400edb0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.569+0000 7f7a3370c640 1 -- 172.21.15.59:0/4033180140 shutdown_connections 2024-03-06T07:22:36.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.569+0000 7f7a3370c640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a1806b4d0 0x7f7a1806d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.569+0000 7f7a3370c640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a2c15def0 0x7f7a2c0c5f50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.569+0000 7f7a3370c640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a2c0cea50 0x7f7a2c1f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.569+0000 7f7a3370c640 1 --2- 172.21.15.59:0/4033180140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a2c06cab0 0x7f7a2c1f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.569+0000 7f7a3370c640 1 -- 172.21.15.59:0/4033180140 >> 172.21.15.59:0/4033180140 conn(0x7f7a2c0b3690 msgr2=0x7f7a2c06f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.569+0000 7f7a3370c640 1 -- 172.21.15.59:0/4033180140 shutdown_connections 2024-03-06T07:22:36.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.569+0000 7f7a3370c640 1 -- 172.21.15.59:0/4033180140 wait complete. 2024-03-06T07:22:36.581 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:36.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.683+0000 7f5d5d13e640 1 Processor -- start 2024-03-06T07:22:36.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.683+0000 7f5d5d13e640 1 -- start start 2024-03-06T07:22:36.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.684+0000 7f5d5d13e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d581614d0 0x7f5d580d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.684+0000 7f5d5d13e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5d580d1910 0x7f5d580d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.684+0000 7f5d5d13e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5d580d22b0 0x7f5d580d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.684+0000 7f5d5d13e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d580d2740 con 0x7f5d580d22b0 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.684+0000 7f5d5d13e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d581618b0 con 0x7f5d580d1910 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.684+0000 7f5d5d13e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d580cf400 con 0x7f5d581614d0 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.684+0000 7f5d5759e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5d580d22b0 0x7f5d580d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.684+0000 7f5d5759e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5d580d22b0 0x7f5d580d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58194/0 (socket says 172.21.15.59:58194) 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.684+0000 7f5d5759e640 1 -- 172.21.15.59:0/1170787715 learned_addr learned my addr 172.21.15.59:0/1170787715 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.685+0000 7f5d56d9d640 1 --2- 172.21.15.59:0/1170787715 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d581614d0 0x7f5d580d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.685+0000 7f5d5659c640 1 --2- 172.21.15.59:0/1170787715 >> v2:172.21.15.164:3300/0 conn(0x7f5d580d1910 0x7f5d580d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.685+0000 7f5d5759e640 1 -- 172.21.15.59:0/1170787715 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d581614d0 msgr2=0x7f5d580d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.685+0000 7f5d5759e640 1 --2- 172.21.15.59:0/1170787715 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d581614d0 0x7f5d580d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.685+0000 7f5d5759e640 1 -- 172.21.15.59:0/1170787715 >> v2:172.21.15.164:3300/0 conn(0x7f5d580d1910 msgr2=0x7f5d580d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.685+0000 7f5d5759e640 1 --2- 172.21.15.59:0/1170787715 >> v2:172.21.15.164:3300/0 conn(0x7f5d580d1910 0x7f5d580d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.685+0000 7f5d5759e640 1 -- 172.21.15.59:0/1170787715 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d580cf6a0 con 0x7f5d580d22b0 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.685+0000 7f5d5759e640 1 --2- 172.21.15.59:0/1170787715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d580d22b0 0x7f5d580d06a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5d5808c4f0 tx=0x7f5d4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7349c6f01424e21 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d55d9b640 1 -- 172.21.15.59:0/1170787715 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4802ca70 con 0x7f5d580d22b0 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d55d9b640 1 -- 172.21.15.59:0/1170787715 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4802cc10 con 0x7f5d580d22b0 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d55d9b640 1 -- 172.21.15.59:0/1170787715 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d480343f0 con 0x7f5d580d22b0 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d5d13e640 1 -- 172.21.15.59:0/1170787715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d580d22b0 msgr2=0x7f5d580d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d5d13e640 1 --2- 172.21.15.59:0/1170787715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d580d22b0 0x7f5d580d06a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5d5808c4f0 tx=0x7f5d4802bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d5d13e640 1 -- 172.21.15.59:0/1170787715 shutdown_connections 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d5d13e640 1 --2- 172.21.15.59:0/1170787715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d580d22b0 0x7f5d580d06a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d5d13e640 1 --2- 172.21.15.59:0/1170787715 >> v2:172.21.15.164:3300/0 conn(0x7f5d580d1910 0x7f5d580d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d5d13e640 1 --2- 172.21.15.59:0/1170787715 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d581614d0 0x7f5d580d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d5d13e640 1 -- 172.21.15.59:0/1170787715 >> 172.21.15.59:0/1170787715 conn(0x7f5d580d3d40 msgr2=0x7f5d580d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d5d13e640 1 -- 172.21.15.59:0/1170787715 shutdown_connections 2024-03-06T07:22:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.686+0000 7f5d5d13e640 1 -- 172.21.15.59:0/1170787715 wait complete. 2024-03-06T07:22:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.688+0000 7f5d5d13e640 1 Processor -- start 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.688+0000 7f5d5d13e640 1 -- start start 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.688+0000 7f5d5d13e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d580d1910 0x7f5d58151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.688+0000 7f5d5d13e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d580d22b0 0x7f5d5814ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.688+0000 7f5d5d13e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d581614d0 0x7f5d580b90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.688+0000 7f5d5d13e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d580b0910 con 0x7f5d580d22b0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.688+0000 7f5d5d13e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d580cfc60 con 0x7f5d581614d0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.688+0000 7f5d5d13e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d580d0be0 con 0x7f5d580d1910 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d580d22b0 0x7f5d5814ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d580d22b0 0x7f5d5814ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58196/0 (socket says 172.21.15.59:58196) 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5659c640 1 -- 172.21.15.59:0/2224980377 learned_addr learned my addr 172.21.15.59:0/2224980377 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5759e640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d581614d0 0x7f5d580b90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d56d9d640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d580d1910 0x7f5d58151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5659c640 1 -- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d580d1910 msgr2=0x7f5d58151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5659c640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d580d1910 0x7f5d58151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5659c640 1 -- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d581614d0 msgr2=0x7f5d580b90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5659c640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d581614d0 0x7f5d580b90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5659c640 1 -- 172.21.15.59:0/2224980377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d580b95f0 con 0x7f5d580d22b0 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d56d9d640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d580d1910 0x7f5d58151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5659c640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d580d22b0 0x7f5d5814ddc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00e9b0 tx=0x7f5d4c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d3f7fe640 1 -- 172.21.15.59:0/2224980377 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c017070 con 0x7f5d580d22b0 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d3f7fe640 1 -- 172.21.15.59:0/2224980377 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c00c9a0 con 0x7f5d580d22b0 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5d13e640 1 -- 172.21.15.59:0/2224980377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d580c93b0 con 0x7f5d580d22b0 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d3f7fe640 1 -- 172.21.15.59:0/2224980377 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c00cd90 con 0x7f5d580d22b0 2024-03-06T07:22:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.689+0000 7f5d5d13e640 1 -- 172.21.15.59:0/2224980377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d580c9970 con 0x7f5d580d22b0 2024-03-06T07:22:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.690+0000 7f5d5d13e640 1 -- 172.21.15.59:0/2224980377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d580b6e00 con 0x7f5d580d22b0 2024-03-06T07:22:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.691+0000 7f5d3f7fe640 1 -- 172.21.15.59:0/2224980377 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d4c01a070 con 0x7f5d580d22b0 2024-03-06T07:22:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.691+0000 7f5d3f7fe640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d4406b4d0 0x7f5d4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.691+0000 7f5d3f7fe640 1 -- 172.21.15.59:0/2224980377 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5d4c08a1a0 con 0x7f5d580d22b0 2024-03-06T07:22:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.691+0000 7f5d56d9d640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d4406b4d0 0x7f5d4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.692+0000 7f5d56d9d640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d4406b4d0 0x7f5d4406d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5d50009e10 tx=0x7f5d50009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.695+0000 7f5d3f7fe640 1 -- 172.21.15.59:0/2224980377 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d4c058950 con 0x7f5d580d22b0 2024-03-06T07:22:36.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.878+0000 7f5d5d13e640 1 -- 172.21.15.59:0/2224980377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5d58068ff0 con 0x7f5d580d22b0 2024-03-06T07:22:36.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.878+0000 7f5d3f7fe640 1 -- 172.21.15.59:0/2224980377 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5d4c05c9f0 con 0x7f5d580d22b0 2024-03-06T07:22:36.880 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 -- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d4406b4d0 msgr2=0x7f5d4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d4406b4d0 0x7f5d4406d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5d50009e10 tx=0x7f5d50009190 comp rx=0 tx=0).stop 2024-03-06T07:22:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 -- 172.21.15.59:0/2224980377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d580d22b0 msgr2=0x7f5d5814ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d580d22b0 0x7f5d5814ddc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00e9b0 tx=0x7f5d4c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 -- 172.21.15.59:0/2224980377 shutdown_connections 2024-03-06T07:22:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d4406b4d0 0x7f5d4406d990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d581614d0 0x7f5d580b90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d580d22b0 0x7f5d5814ddc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 --2- 172.21.15.59:0/2224980377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d580d1910 0x7f5d58151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 -- 172.21.15.59:0/2224980377 >> 172.21.15.59:0/2224980377 conn(0x7f5d580d3d40 msgr2=0x7f5d580b1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 -- 172.21.15.59:0/2224980377 shutdown_connections 2024-03-06T07:22:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.880+0000 7f5d5d13e640 1 -- 172.21.15.59:0/2224980377 wait complete. 2024-03-06T07:22:36.893 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:36.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.993+0000 7ffa3afee640 1 Processor -- start 2024-03-06T07:22:36.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.993+0000 7ffa3afee640 1 -- start start 2024-03-06T07:22:36.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.993+0000 7ffa3afee640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffa3415e010 0x7ffa340cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.993+0000 7ffa3afee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa340d1ad0 0x7ffa340d1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.993+0000 7ffa3afee640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa3415e3f0 con 0x7ffa340b3240 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.993+0000 7ffa3afee640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa340b1f40 con 0x7ffa3415e010 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.993+0000 7ffa3afee640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa340b1c10 con 0x7ffa340d1ad0 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.994+0000 7ffa33fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ffa340b3240 0x7ffa340b23d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51360/0 (socket says 172.21.15.59:51360) 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.994+0000 7ffa33fff640 1 -- 172.21.15.59:0/1773759567 learned_addr learned my addr 172.21.15.59:0/1773759567 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.994+0000 7ffa3958b640 1 --2- 172.21.15.59:0/1773759567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa340d1ad0 0x7ffa340d1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.994+0000 7ffa38d8a640 1 --2- 172.21.15.59:0/1773759567 >> v2:172.21.15.164:3300/0 conn(0x7ffa3415e010 0x7ffa340cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.994+0000 7ffa337fe640 1 -- 172.21.15.59:0/1773759567 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727439778 0 0) 0x7ffa3415e3f0 con 0x7ffa340b3240 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.994+0000 7ffa337fe640 1 -- 172.21.15.59:0/1773759567 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa24003520 con 0x7ffa340b3240 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.994+0000 7ffa3958b640 1 -- 172.21.15.59:0/1773759567 >> v2:172.21.15.164:3300/0 conn(0x7ffa3415e010 msgr2=0x7ffa340cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.994+0000 7ffa3958b640 1 --2- 172.21.15.59:0/1773759567 >> v2:172.21.15.164:3300/0 conn(0x7ffa3415e010 0x7ffa340cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.994+0000 7ffa3958b640 1 -- 172.21.15.59:0/1773759567 >> v1:172.21.15.59:6789/0 conn(0x7ffa340b3240 legacy=0x7ffa340b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.994+0000 7ffa3958b640 1 -- 172.21.15.59:0/1773759567 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3406d1b0 con 0x7ffa340d1ad0 2024-03-06T07:22:36.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.995+0000 7ffa3958b640 1 --2- 172.21.15.59:0/1773759567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa340d1ad0 0x7ffa340d1f80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009d90 tx=0x7ffa2c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4827c6153615391 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.995+0000 7ffa337fe640 1 -- 172.21.15.59:0/1773759567 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c039070 con 0x7ffa340d1ad0 2024-03-06T07:22:36.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.995+0000 7ffa337fe640 1 -- 172.21.15.59:0/1773759567 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2c02cbc0 con 0x7ffa340d1ad0 2024-03-06T07:22:36.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.995+0000 7ffa337fe640 1 -- 172.21.15.59:0/1773759567 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c034440 con 0x7ffa340d1ad0 2024-03-06T07:22:37.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.995+0000 7ffa3afee640 1 -- 172.21.15.59:0/1773759567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa340d1ad0 msgr2=0x7ffa340d1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.995+0000 7ffa3afee640 1 --2- 172.21.15.59:0/1773759567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa340d1ad0 0x7ffa340d1f80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009d90 tx=0x7ffa2c02be10 comp rx=0 tx=0).stop 2024-03-06T07:22:37.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.995+0000 7ffa3afee640 1 -- 172.21.15.59:0/1773759567 shutdown_connections 2024-03-06T07:22:37.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.995+0000 7ffa3afee640 1 --2- 172.21.15.59:0/1773759567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa340d1ad0 0x7ffa340d1f80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.995+0000 7ffa3afee640 1 --2- 172.21.15.59:0/1773759567 >> v2:172.21.15.164:3300/0 conn(0x7ffa3415e010 0x7ffa340cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.995+0000 7ffa3afee640 1 -- 172.21.15.59:0/1773759567 >> 172.21.15.59:0/1773759567 conn(0x7ffa3409a9f0 msgr2=0x7ffa3409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.995+0000 7ffa3afee640 1 -- 172.21.15.59:0/1773759567 shutdown_connections 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.996+0000 7ffa3afee640 1 -- 172.21.15.59:0/1773759567 wait complete. 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.997+0000 7ffa3afee640 1 Processor -- start 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.997+0000 7ffa3afee640 1 -- start start 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.997+0000 7ffa3afee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa340d1ad0 0x7ffa34158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.997+0000 7ffa3afee640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa3415e010 0x7ffa340c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.997+0000 7ffa3afee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa34159500 0x7ffa340b5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3afee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3406ced0 con 0x7ffa34159500 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3afee640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3406cbf0 con 0x7ffa3415e010 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3afee640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa340cf310 con 0x7ffa340d1ad0 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3958b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa34159500 0x7ffa340b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3958b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa34159500 0x7ffa340b5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58204/0 (socket says 172.21.15.59:58204) 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3958b640 1 -- 172.21.15.59:0/1783099429 learned_addr learned my addr 172.21.15.59:0/1783099429 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa33fff640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa3415e010 0x7ffa340c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa38d8a640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa340d1ad0 0x7ffa34158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3958b640 1 -- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa340d1ad0 msgr2=0x7ffa34158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3958b640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa340d1ad0 0x7ffa34158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3958b640 1 -- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa3415e010 msgr2=0x7ffa340c6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3958b640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa3415e010 0x7ffa340c6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3958b640 1 -- 172.21.15.59:0/1783099429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa340c6b40 con 0x7ffa34159500 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3958b640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa34159500 0x7ffa340b5630 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009d90 tx=0x7ffa2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa317fa640 1 -- 172.21.15.59:0/1783099429 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c039050 con 0x7ffa34159500 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa317fa640 1 -- 172.21.15.59:0/1783099429 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2c045070 con 0x7ffa34159500 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa317fa640 1 -- 172.21.15.59:0/1783099429 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c009450 con 0x7ffa34159500 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3afee640 1 -- 172.21.15.59:0/1783099429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa340b4870 con 0x7ffa34159500 2024-03-06T07:22:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.998+0000 7ffa3afee640 1 -- 172.21.15.59:0/1783099429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa340b4e30 con 0x7ffa34159500 2024-03-06T07:22:37.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:36.999+0000 7ffa3afee640 1 -- 172.21.15.59:0/1783099429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa3404f330 con 0x7ffa34159500 2024-03-06T07:22:37.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.000+0000 7ffa317fa640 1 -- 172.21.15.59:0/1783099429 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa2c03e830 con 0x7ffa34159500 2024-03-06T07:22:37.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.000+0000 7ffa317fa640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffa1806b4d0 0x7ffa1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.000+0000 7ffa317fa640 1 -- 172.21.15.59:0/1783099429 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ffa2c0aced0 con 0x7ffa34159500 2024-03-06T07:22:37.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.000+0000 7ffa38d8a640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffa1806b4d0 0x7ffa1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.001+0000 7ffa38d8a640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffa1806b4d0 0x7ffa1806d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa340d1690 tx=0x7ffa28009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.005+0000 7ffa317fa640 1 -- 172.21.15.59:0/1783099429 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa2c07b8f0 con 0x7ffa34159500 2024-03-06T07:22:37.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.199+0000 7ffa3afee640 1 -- 172.21.15.59:0/1783099429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffa340cef60 con 0x7ffa34159500 2024-03-06T07:22:37.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.200+0000 7ffa317fa640 1 -- 172.21.15.59:0/1783099429 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffa2c07f990 con 0x7ffa34159500 2024-03-06T07:22:37.202 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.201+0000 7ffa3afee640 1 -- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffa1806b4d0 msgr2=0x7ffa1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.201+0000 7ffa3afee640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffa1806b4d0 0x7ffa1806d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffa340d1690 tx=0x7ffa28009190 comp rx=0 tx=0).stop 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.201+0000 7ffa3afee640 1 -- 172.21.15.59:0/1783099429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa34159500 msgr2=0x7ffa340b5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.201+0000 7ffa3afee640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa34159500 0x7ffa340b5630 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009d90 tx=0x7ffa2c004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.202+0000 7ffa3afee640 1 -- 172.21.15.59:0/1783099429 shutdown_connections 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.202+0000 7ffa3afee640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffa1806b4d0 0x7ffa1806d990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.202+0000 7ffa3afee640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa34159500 0x7ffa340b5630 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.202+0000 7ffa3afee640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa3415e010 0x7ffa340c6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.202+0000 7ffa3afee640 1 --2- 172.21.15.59:0/1783099429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa340d1ad0 0x7ffa34158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.202+0000 7ffa3afee640 1 -- 172.21.15.59:0/1783099429 >> 172.21.15.59:0/1783099429 conn(0x7ffa3409a9f0 msgr2=0x7ffa3406c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.202+0000 7ffa3afee640 1 -- 172.21.15.59:0/1783099429 shutdown_connections 2024-03-06T07:22:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.202+0000 7ffa3afee640 1 -- 172.21.15.59:0/1783099429 wait complete. 2024-03-06T07:22:37.214 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi164.front.sepia.ceph.com... 2024-03-06T07:22:37.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:37.214 DEBUG:teuthology.orchestra.run.smithi164:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:37.242 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:37.243 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:22:37.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:37.243 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:22:37.298 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:22:37.298 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:37.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.398+0000 7fd4502a4640 1 Processor -- start 2024-03-06T07:22:37.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.398+0000 7fd4502a4640 1 -- start start 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd4502a4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd44815bf30 0x7fd44815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd4502a4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44815f580 con 0x7fd44813dc10 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd4502a4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44815b450 con 0x7fd4480b3060 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd4502a4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4480cf3f0 con 0x7fd44815bf30 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd44e040640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd44813dc10 0x7fd4480b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51374/0 (socket says 172.21.15.59:51374) 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd44e040640 1 -- 172.21.15.59:0/2388866427 learned_addr learned my addr 172.21.15.59:0/2388866427 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd44e841640 1 --2- 172.21.15.59:0/2388866427 >> v2:172.21.15.164:3300/0 conn(0x7fd44815bf30 0x7fd44815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd44d03e640 1 -- 172.21.15.59:0/2388866427 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143728938 0 0) 0x7fd44815f580 con 0x7fd44813dc10 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd44d03e640 1 -- 172.21.15.59:0/2388866427 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd434003520 con 0x7fd44813dc10 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd44e841640 1 -- 172.21.15.59:0/2388866427 >> v1:172.21.15.164:6789/0 conn(0x7fd4480b3060 legacy=0x7fd44815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd44e841640 1 -- 172.21.15.59:0/2388866427 >> v1:172.21.15.59:6789/0 conn(0x7fd44813dc10 legacy=0x7fd4480b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.399+0000 7fd44e841640 1 -- 172.21.15.59:0/2388866427 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44815b730 con 0x7fd44815bf30 2024-03-06T07:22:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.400+0000 7fd44e841640 1 --2- 172.21.15.59:0/2388866427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44815bf30 0x7fd44815ef60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd444009d90 tx=0x7fd44402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=464c357351629af2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.400+0000 7fd44d03e640 1 -- 172.21.15.59:0/2388866427 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd444039070 con 0x7fd44815bf30 2024-03-06T07:22:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.400+0000 7fd44d03e640 1 -- 172.21.15.59:0/2388866427 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44402cc60 con 0x7fd44815bf30 2024-03-06T07:22:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.400+0000 7fd44d03e640 1 -- 172.21.15.59:0/2388866427 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd444034640 con 0x7fd44815bf30 2024-03-06T07:22:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.401+0000 7fd4502a4640 1 -- 172.21.15.59:0/2388866427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44815bf30 msgr2=0x7fd44815ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.401+0000 7fd4502a4640 1 --2- 172.21.15.59:0/2388866427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44815bf30 0x7fd44815ef60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd444009d90 tx=0x7fd44402beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.401+0000 7fd4502a4640 1 -- 172.21.15.59:0/2388866427 shutdown_connections 2024-03-06T07:22:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.401+0000 7fd4502a4640 1 --2- 172.21.15.59:0/2388866427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44815bf30 0x7fd44815ef60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.401+0000 7fd4502a4640 1 -- 172.21.15.59:0/2388866427 >> 172.21.15.59:0/2388866427 conn(0x7fd4480cf8a0 msgr2=0x7fd4480cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.401+0000 7fd4502a4640 1 -- 172.21.15.59:0/2388866427 shutdown_connections 2024-03-06T07:22:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.401+0000 7fd4502a4640 1 -- 172.21.15.59:0/2388866427 wait complete. 2024-03-06T07:22:37.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.402+0000 7fd4502a4640 1 Processor -- start 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.402+0000 7fd4502a4640 1 -- start start 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.402+0000 7fd4502a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4480b3060 0x7fd448150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.402+0000 7fd4502a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44813dc10 0x7fd4481420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.402+0000 7fd4502a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd44815bf30 0x7fd448136f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.402+0000 7fd4502a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4480718e0 con 0x7fd44813dc10 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.402+0000 7fd4502a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44815b210 con 0x7fd4480b3060 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.402+0000 7fd4502a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd448071a60 con 0x7fd44815bf30 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd44d83f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44813dc10 0x7fd4481420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd44e040640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4480b3060 0x7fd448150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd44d83f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44813dc10 0x7fd4481420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58206/0 (socket says 172.21.15.59:58206) 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd44d83f640 1 -- 172.21.15.59:0/2072312214 learned_addr learned my addr 172.21.15.59:0/2072312214 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd44e841640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd44815bf30 0x7fd448136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd44d83f640 1 -- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd44815bf30 msgr2=0x7fd448136f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd44d83f640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd44815bf30 0x7fd448136f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd44d83f640 1 -- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4480b3060 msgr2=0x7fd448150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd44d83f640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4480b3060 0x7fd448150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd44d83f640 1 -- 172.21.15.59:0/2072312214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd448150ca0 con 0x7fd44813dc10 2024-03-06T07:22:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd44d83f640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44813dc10 0x7fd4481420d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd43800c400 tx=0x7fd43800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd43effd640 1 -- 172.21.15.59:0/2072312214 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4380052d0 con 0x7fd44813dc10 2024-03-06T07:22:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd43effd640 1 -- 172.21.15.59:0/2072312214 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd438005490 con 0x7fd44813dc10 2024-03-06T07:22:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd4502a4640 1 -- 172.21.15.59:0/2072312214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd448150f20 con 0x7fd44813dc10 2024-03-06T07:22:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd43effd640 1 -- 172.21.15.59:0/2072312214 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd438007260 con 0x7fd44813dc10 2024-03-06T07:22:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.403+0000 7fd4502a4640 1 -- 172.21.15.59:0/2072312214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4481514e0 con 0x7fd44813dc10 2024-03-06T07:22:37.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.404+0000 7fd4502a4640 1 -- 172.21.15.59:0/2072312214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd44804f330 con 0x7fd44813dc10 2024-03-06T07:22:37.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.405+0000 7fd43effd640 1 -- 172.21.15.59:0/2072312214 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd438007400 con 0x7fd44813dc10 2024-03-06T07:22:37.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.405+0000 7fd43effd640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b4d0 0x7fd42c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.405+0000 7fd43effd640 1 -- 172.21.15.59:0/2072312214 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd438089670 con 0x7fd44813dc10 2024-03-06T07:22:37.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.405+0000 7fd44e040640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b4d0 0x7fd42c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.406+0000 7fd44e040640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b4d0 0x7fd42c06d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd4400011e0 tx=0x7fd440007d30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.409+0000 7fd43effd640 1 -- 172.21.15.59:0/2072312214 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd448150ca0 con 0x7fd44813dc10 2024-03-06T07:22:37.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.630+0000 7fd4502a4640 1 -- 172.21.15.59:0/2072312214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd44806eb00 con 0x7fd44813dc10 2024-03-06T07:22:37.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.630+0000 7fd43effd640 1 -- 172.21.15.59:0/2072312214 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7fd438057eb0 con 0x7fd44813dc10 2024-03-06T07:22:37.632 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:37.633 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1759048959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1759048959},{"type":"v1","addr":"172.21.15.164:6836","nonce":1759048959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/550023536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":550023536},{"type":"v1","addr":"172.21.15.59:6833","nonce":550023536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2132697832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2132697832},{"type":"v1","addr":"172.21.15.59:6835","nonce":2132697832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:54.767210+0000","modified":"2024-03-06T07:21:56.776415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/1957940318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1957940318},{"type":"v1","addr":"172.21.15.164:6837","nonce":1957940318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-06T07:22:37.633 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 6 2024-03-06T07:22:37.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.632+0000 7fd4502a4640 1 -- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b4d0 msgr2=0x7fd42c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.632+0000 7fd4502a4640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b4d0 0x7fd42c06d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd4400011e0 tx=0x7fd440007d30 comp rx=0 tx=0).stop 2024-03-06T07:22:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.632+0000 7fd4502a4640 1 -- 172.21.15.59:0/2072312214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44813dc10 msgr2=0x7fd4481420d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.632+0000 7fd4502a4640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44813dc10 0x7fd4481420d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd43800c400 tx=0x7fd43800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.632+0000 7fd4502a4640 1 -- 172.21.15.59:0/2072312214 shutdown_connections 2024-03-06T07:22:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.632+0000 7fd4502a4640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b4d0 0x7fd42c06d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.632+0000 7fd4502a4640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd44815bf30 0x7fd448136f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.632+0000 7fd4502a4640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44813dc10 0x7fd4481420d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.632+0000 7fd4502a4640 1 --2- 172.21.15.59:0/2072312214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4480b3060 0x7fd448150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.632+0000 7fd4502a4640 1 -- 172.21.15.59:0/2072312214 >> 172.21.15.59:0/2072312214 conn(0x7fd4480cf8a0 msgr2=0x7fd44815e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.633+0000 7fd4502a4640 1 -- 172.21.15.59:0/2072312214 shutdown_connections 2024-03-06T07:22:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.633+0000 7fd4502a4640 1 -- 172.21.15.59:0/2072312214 wait complete. 2024-03-06T07:22:37.645 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:37.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.744+0000 7f321b6df640 1 Processor -- start 2024-03-06T07:22:37.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.745+0000 7f321b6df640 1 -- start start 2024-03-06T07:22:37.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.745+0000 7f321b6df640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f32140d22b0 0x7f32140d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.745+0000 7f321b6df640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32140b3430 con 0x7f321415dfe0 2024-03-06T07:22:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.745+0000 7f321b6df640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32140b3710 con 0x7f32140b41f0 2024-03-06T07:22:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.745+0000 7f321b6df640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32140d2c50 con 0x7f32140d22b0 2024-03-06T07:22:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.745+0000 7f3218c7a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f32140d22b0 0x7f32140d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.745+0000 7f321947b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f321415dfe0 0x7f32140cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51380/0 (socket says 172.21.15.59:51380) 2024-03-06T07:22:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.745+0000 7f3218c7a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f32140d22b0 0x7f32140d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58208/0 (socket says 172.21.15.59:58208) 2024-03-06T07:22:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.745+0000 7f321947b640 1 -- 172.21.15.59:0/2107025235 learned_addr learned my addr 172.21.15.59:0/2107025235 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.746+0000 7f3218c7a640 1 -- 172.21.15.59:0/2107025235 >> v1:172.21.15.164:6789/0 conn(0x7f32140b41f0 legacy=0x7f32140b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.746+0000 7f3218c7a640 1 -- 172.21.15.59:0/2107025235 >> v1:172.21.15.59:6789/0 conn(0x7f321415dfe0 legacy=0x7f32140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.746+0000 7f3218c7a640 1 -- 172.21.15.59:0/2107025235 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32140b39f0 con 0x7f32140d22b0 2024-03-06T07:22:37.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.746+0000 7f3213fff640 1 -- 172.21.15.59:0/2107025235 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849320971 0 0) 0x7f32140b3430 con 0x7f321415dfe0 2024-03-06T07:22:37.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.746+0000 7f3218c7a640 1 --2- 172.21.15.59:0/2107025235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32140d22b0 0x7f32140d2710 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f32080097d0 tx=0x7f320802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998d98f7d839d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.747+0000 7f3213fff640 1 -- 172.21.15.59:0/2107025235 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208034070 con 0x7f32140d22b0 2024-03-06T07:22:37.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.747+0000 7f3213fff640 1 -- 172.21.15.59:0/2107025235 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3208030610 con 0x7f32140d22b0 2024-03-06T07:22:37.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.747+0000 7f3213fff640 1 -- 172.21.15.59:0/2107025235 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32080309e0 con 0x7f32140d22b0 2024-03-06T07:22:37.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.747+0000 7f321b6df640 1 -- 172.21.15.59:0/2107025235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32140d22b0 msgr2=0x7f32140d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.747+0000 7f321b6df640 1 --2- 172.21.15.59:0/2107025235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32140d22b0 0x7f32140d2710 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f32080097d0 tx=0x7f320802b960 comp rx=0 tx=0).stop 2024-03-06T07:22:37.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.747+0000 7f321b6df640 1 -- 172.21.15.59:0/2107025235 shutdown_connections 2024-03-06T07:22:37.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.747+0000 7f321b6df640 1 --2- 172.21.15.59:0/2107025235 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32140d22b0 0x7f32140d2710 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.747+0000 7f321b6df640 1 -- 172.21.15.59:0/2107025235 >> 172.21.15.59:0/2107025235 conn(0x7f32140d3cd0 msgr2=0x7f32140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.747+0000 7f321b6df640 1 -- 172.21.15.59:0/2107025235 shutdown_connections 2024-03-06T07:22:37.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.747+0000 7f321b6df640 1 -- 172.21.15.59:0/2107025235 wait complete. 2024-03-06T07:22:37.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.748+0000 7f321b6df640 1 Processor -- start 2024-03-06T07:22:37.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.748+0000 7f321b6df640 1 -- start start 2024-03-06T07:22:37.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.749+0000 7f321b6df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32140b41f0 0x7f3214160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.749+0000 7f321b6df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32140d22b0 0x7f32140b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.749+0000 7f321b6df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321415dfe0 0x7f321415fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.749+0000 7f321b6df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32140b3d60 con 0x7f321415dfe0 2024-03-06T07:22:37.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.749+0000 7f321b6df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32140b31e0 con 0x7f32140d22b0 2024-03-06T07:22:37.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.749+0000 7f321b6df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32140d1c70 con 0x7f32140b41f0 2024-03-06T07:22:37.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.749+0000 7f3219c7c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321415dfe0 0x7f321415fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.749+0000 7f3219c7c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321415dfe0 0x7f321415fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58222/0 (socket says 172.21.15.59:58222) 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.749+0000 7f3219c7c640 1 -- 172.21.15.59:0/2215362755 learned_addr learned my addr 172.21.15.59:0/2215362755 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.749+0000 7f3218c7a640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32140d22b0 0x7f32140b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.749+0000 7f321947b640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32140b41f0 0x7f3214160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.750+0000 7f3219c7c640 1 -- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32140b41f0 msgr2=0x7f3214160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.750+0000 7f3219c7c640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32140b41f0 0x7f3214160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.750+0000 7f3219c7c640 1 -- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32140d22b0 msgr2=0x7f32140b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.750+0000 7f3219c7c640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32140d22b0 0x7f32140b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.750+0000 7f3219c7c640 1 -- 172.21.15.59:0/2215362755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321415e910 con 0x7f321415dfe0 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.750+0000 7f3219c7c640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321415dfe0 0x7f321415fdb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f320400c7a0 tx=0x7f320400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.750+0000 7f3211ffb640 1 -- 172.21.15.59:0/2215362755 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204003b70 con 0x7f321415dfe0 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.750+0000 7f3211ffb640 1 -- 172.21.15.59:0/2215362755 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3204003d30 con 0x7f321415dfe0 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.750+0000 7f3211ffb640 1 -- 172.21.15.59:0/2215362755 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320400aa90 con 0x7f321415dfe0 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.750+0000 7f321b6df640 1 -- 172.21.15.59:0/2215362755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321415ebf0 con 0x7f321415dfe0 2024-03-06T07:22:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.750+0000 7f321b6df640 1 -- 172.21.15.59:0/2215362755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321415f1b0 con 0x7f321415dfe0 2024-03-06T07:22:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.751+0000 7f321b6df640 1 -- 172.21.15.59:0/2215362755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321404f330 con 0x7f321415dfe0 2024-03-06T07:22:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.751+0000 7f3211ffb640 1 -- 172.21.15.59:0/2215362755 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f320400ac30 con 0x7f321415dfe0 2024-03-06T07:22:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.752+0000 7f3211ffb640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f320006b4d0 0x7f320006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.752+0000 7f3211ffb640 1 -- 172.21.15.59:0/2215362755 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f320408a080 con 0x7f321415dfe0 2024-03-06T07:22:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.752+0000 7f321947b640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f320006b4d0 0x7f320006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.753+0000 7f321947b640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f320006b4d0 0x7f320006d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f320c000f60 tx=0x7f320c007e40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.756+0000 7f3211ffb640 1 -- 172.21.15.59:0/2215362755 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3204058830 con 0x7f321415dfe0 2024-03-06T07:22:37.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.942+0000 7f321b6df640 1 -- 172.21.15.59:0/2215362755 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32140cea90 con 0x7f321415dfe0 2024-03-06T07:22:37.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.944+0000 7f3211ffb640 1 -- 172.21.15.59:0/2215362755 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16842 (secure 0 0 0) 0x7f320405c8d0 con 0x7f321415dfe0 2024-03-06T07:22:37.946 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:37.946 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":22,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:36.536165+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":1}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:37.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 -- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f320006b4d0 msgr2=0x7f320006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f320006b4d0 0x7f320006d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f320c000f60 tx=0x7f320c007e40 comp rx=0 tx=0).stop 2024-03-06T07:22:37.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 -- 172.21.15.59:0/2215362755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321415dfe0 msgr2=0x7f321415fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321415dfe0 0x7f321415fdb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f320400c7a0 tx=0x7f320400cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:37.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 -- 172.21.15.59:0/2215362755 shutdown_connections 2024-03-06T07:22:37.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f320006b4d0 0x7f320006d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f321415dfe0 0x7f321415fdb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32140d22b0 0x7f32140b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 --2- 172.21.15.59:0/2215362755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32140b41f0 0x7f3214160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 -- 172.21.15.59:0/2215362755 >> 172.21.15.59:0/2215362755 conn(0x7f32140d3cd0 msgr2=0x7f321415d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 -- 172.21.15.59:0/2215362755 shutdown_connections 2024-03-06T07:22:37.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:37.946+0000 7f321b6df640 1 -- 172.21.15.59:0/2215362755 wait complete. 2024-03-06T07:22:37.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:52.587542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:53.743634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:37.959 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.063+0000 7fd0dd1b5640 1 Processor -- start 2024-03-06T07:22:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.063+0000 7fd0dd1b5640 1 -- start start 2024-03-06T07:22:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0dd1b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0d80d4860 0x7fd0d80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0dd1b5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd0d80b0e30 0x7fd0d80b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0dd1b5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d81618b0 con 0x7fd0d81614d0 2024-03-06T07:22:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0dd1b5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d80d4cc0 con 0x7fd0d80b0e30 2024-03-06T07:22:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0dd1b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80cf400 con 0x7fd0d80d4860 2024-03-06T07:22:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0d759e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd0d80b0e30 0x7fd0d80b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0d759e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd0d80b0e30 0x7fd0d80b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58234/0 (socket says 172.21.15.59:58234) 2024-03-06T07:22:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0d6d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd0d81614d0 0x7fd0d80d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51384/0 (socket says 172.21.15.59:51384) 2024-03-06T07:22:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0d759e640 1 -- 172.21.15.59:0/2388805713 learned_addr learned my addr 172.21.15.59:0/2388805713 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0cffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0d80d4860 0x7fd0d80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0d759e640 1 -- 172.21.15.59:0/2388805713 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0d80d4860 msgr2=0x7fd0d80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0d759e640 1 --2- 172.21.15.59:0/2388805713 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0d80d4860 0x7fd0d80d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0d759e640 1 -- 172.21.15.59:0/2388805713 >> v1:172.21.15.59:6789/0 conn(0x7fd0d81614d0 legacy=0x7fd0d80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0d659c640 1 -- 172.21.15.59:0/2388805713 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280088462 0 0) 0x7fd0d81618b0 con 0x7fd0d81614d0 2024-03-06T07:22:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.064+0000 7fd0d759e640 1 -- 172.21.15.59:0/2388805713 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d80cf6a0 con 0x7fd0d80b0e30 2024-03-06T07:22:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.065+0000 7fd0d759e640 1 --2- 172.21.15.59:0/2388805713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80b0e30 0x7fd0d80b12c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009e50 tx=0x7fd0d002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b0f0d5b2639582 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.065+0000 7fd0d659c640 1 -- 172.21.15.59:0/2388805713 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d002cb00 con 0x7fd0d80b0e30 2024-03-06T07:22:38.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.065+0000 7fd0d659c640 1 -- 172.21.15.59:0/2388805713 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d002cca0 con 0x7fd0d80b0e30 2024-03-06T07:22:38.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.065+0000 7fd0d659c640 1 -- 172.21.15.59:0/2388805713 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0034610 con 0x7fd0d80b0e30 2024-03-06T07:22:38.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.066+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/2388805713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80b0e30 msgr2=0x7fd0d80b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.066+0000 7fd0dd1b5640 1 --2- 172.21.15.59:0/2388805713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80b0e30 0x7fd0d80b12c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009e50 tx=0x7fd0d002bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:38.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.066+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/2388805713 shutdown_connections 2024-03-06T07:22:38.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.066+0000 7fd0dd1b5640 1 --2- 172.21.15.59:0/2388805713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80b0e30 0x7fd0d80b12c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.066+0000 7fd0dd1b5640 1 --2- 172.21.15.59:0/2388805713 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0d80d4860 0x7fd0d80d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.066+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/2388805713 >> 172.21.15.59:0/2388805713 conn(0x7fd0d80d3d40 msgr2=0x7fd0d80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.066+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/2388805713 shutdown_connections 2024-03-06T07:22:38.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.066+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/2388805713 wait complete. 2024-03-06T07:22:38.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.067+0000 7fd0dd1b5640 1 Processor -- start 2024-03-06T07:22:38.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.067+0000 7fd0dd1b5640 1 -- start start 2024-03-06T07:22:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0dd1b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0d80b0e30 0x7fd0d815e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0dd1b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80d4860 0x7fd0d80c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0dd1b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d81614d0 0x7fd0d80c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0dd1b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80b0910 con 0x7fd0d80d4860 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0dd1b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80cfc60 con 0x7fd0d81614d0 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0dd1b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80d2560 con 0x7fd0d80b0e30 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80d4860 0x7fd0d80c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80d4860 0x7fd0d80c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58248/0 (socket says 172.21.15.59:58248) 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cffff640 1 -- 172.21.15.59:0/356235568 learned_addr learned my addr 172.21.15.59:0/356235568 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0d6d9d640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0d80b0e30 0x7fd0d815e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0d759e640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d81614d0 0x7fd0d80c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cffff640 1 -- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0d80b0e30 msgr2=0x7fd0d815e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cffff640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0d80b0e30 0x7fd0d815e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cffff640 1 -- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d81614d0 msgr2=0x7fd0d80c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cffff640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d81614d0 0x7fd0d80c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cffff640 1 -- 172.21.15.59:0/356235568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d80b7780 con 0x7fd0d80d4860 2024-03-06T07:22:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cffff640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80d4860 0x7fd0d80c96c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000e9b0 tx=0x7fd0c000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cf7fe640 1 -- 172.21.15.59:0/356235568 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0017070 con 0x7fd0d80d4860 2024-03-06T07:22:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cf7fe640 1 -- 172.21.15.59:0/356235568 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c000c9a0 con 0x7fd0d80d4860 2024-03-06T07:22:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/356235568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d80b7a60 con 0x7fd0d80d4860 2024-03-06T07:22:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.068+0000 7fd0cf7fe640 1 -- 172.21.15.59:0/356235568 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c000cd90 con 0x7fd0d80d4860 2024-03-06T07:22:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.069+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/356235568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d80b8020 con 0x7fd0d80d4860 2024-03-06T07:22:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.070+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/356235568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d8068ff0 con 0x7fd0d80d4860 2024-03-06T07:22:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.070+0000 7fd0cf7fe640 1 -- 172.21.15.59:0/356235568 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0c001a070 con 0x7fd0d80d4860 2024-03-06T07:22:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.070+0000 7fd0cf7fe640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0b806b4d0 0x7fd0b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.070+0000 7fd0cf7fe640 1 -- 172.21.15.59:0/356235568 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd0c008a260 con 0x7fd0d80d4860 2024-03-06T07:22:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.071+0000 7fd0d6d9d640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0b806b4d0 0x7fd0b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.071+0000 7fd0d6d9d640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0b806b4d0 0x7fd0b806d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8000f60 tx=0x7fd0c8007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.075+0000 7fd0cf7fe640 1 -- 172.21.15.59:0/356235568 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0c0058980 con 0x7fd0d80d4860 2024-03-06T07:22:38.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.254+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/356235568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd0d80b8420 con 0x7fd0d80d4860 2024-03-06T07:22:38.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.254+0000 7fd0cf7fe640 1 -- 172.21.15.59:0/356235568 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd0c005ca20 con 0x7fd0d80d4860 2024-03-06T07:22:38.256 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:38.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0b806b4d0 msgr2=0x7fd0b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0b806b4d0 0x7fd0b806d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8000f60 tx=0x7fd0c8007d50 comp rx=0 tx=0).stop 2024-03-06T07:22:38.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/356235568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80d4860 msgr2=0x7fd0d80c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80d4860 0x7fd0d80c96c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000e9b0 tx=0x7fd0c000ee70 comp rx=0 tx=0).stop 2024-03-06T07:22:38.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/356235568 shutdown_connections 2024-03-06T07:22:38.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0b806b4d0 0x7fd0b806d990 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d81614d0 0x7fd0d80c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80d4860 0x7fd0d80c96c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 --2- 172.21.15.59:0/356235568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0d80b0e30 0x7fd0d815e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/356235568 >> 172.21.15.59:0/356235568 conn(0x7fd0d80d3d40 msgr2=0x7fd0d806b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/356235568 shutdown_connections 2024-03-06T07:22:38.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.256+0000 7fd0dd1b5640 1 -- 172.21.15.59:0/356235568 wait complete. 2024-03-06T07:22:38.269 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:38.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.371+0000 7f7dd3ac9640 1 Processor -- start 2024-03-06T07:22:38.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.371+0000 7f7dd3ac9640 1 -- start start 2024-03-06T07:22:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.372+0000 7f7dd3ac9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dcc1614d0 0x7f7dcc0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.372+0000 7f7dd3ac9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7dcc0d1910 0x7f7dcc0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.372+0000 7f7dd3ac9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7dcc0d22b0 0x7f7dcc0d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.372+0000 7f7dd3ac9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dcc0d2740 con 0x7f7dcc0d1910 2024-03-06T07:22:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.372+0000 7f7dd3ac9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dcc1618b0 con 0x7f7dcc0d22b0 2024-03-06T07:22:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.372+0000 7f7dd3ac9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0cf400 con 0x7f7dcc1614d0 2024-03-06T07:22:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.372+0000 7f7dd1064640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7dcc0d1910 0x7f7dcc0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.372+0000 7f7dd1064640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7dcc0d1910 0x7f7dcc0d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58252/0 (socket says 172.21.15.59:58252) 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.372+0000 7f7dd1064640 1 -- 172.21.15.59:0/994939666 learned_addr learned my addr 172.21.15.59:0/994939666 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.372+0000 7f7dd2066640 1 --2- 172.21.15.59:0/994939666 >> v2:172.21.15.164:3300/0 conn(0x7f7dcc0d22b0 0x7f7dcc0d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.372+0000 7f7dd1865640 1 --2- 172.21.15.59:0/994939666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dcc1614d0 0x7f7dcc0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.373+0000 7f7dd1064640 1 -- 172.21.15.59:0/994939666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dcc1614d0 msgr2=0x7f7dcc0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.373+0000 7f7dd1064640 1 --2- 172.21.15.59:0/994939666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dcc1614d0 0x7f7dcc0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.373+0000 7f7dd1064640 1 -- 172.21.15.59:0/994939666 >> v2:172.21.15.164:3300/0 conn(0x7f7dcc0d22b0 msgr2=0x7f7dcc0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.373+0000 7f7dd1064640 1 --2- 172.21.15.59:0/994939666 >> v2:172.21.15.164:3300/0 conn(0x7f7dcc0d22b0 0x7f7dcc0d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.373+0000 7f7dd1064640 1 -- 172.21.15.59:0/994939666 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc0cf6a0 con 0x7f7dcc0d1910 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.373+0000 7f7dd2066640 1 --2- 172.21.15.59:0/994939666 >> v2:172.21.15.164:3300/0 conn(0x7f7dcc0d22b0 0x7f7dcc0d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.373+0000 7f7dd1064640 1 --2- 172.21.15.59:0/994939666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dcc0d1910 0x7f7dcc0d1d70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0b57d0 tx=0x7f7dc002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8266ff676679da13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd0863640 1 -- 172.21.15.59:0/994939666 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0035020 con 0x7f7dcc0d1910 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd0863640 1 -- 172.21.15.59:0/994939666 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc0031870 con 0x7f7dcc0d1910 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd0863640 1 -- 172.21.15.59:0/994939666 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0031c40 con 0x7f7dcc0d1910 2024-03-06T07:22:38.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/994939666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dcc0d1910 msgr2=0x7f7dcc0d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd3ac9640 1 --2- 172.21.15.59:0/994939666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dcc0d1910 0x7f7dcc0d1d70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0b57d0 tx=0x7f7dc002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:38.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/994939666 shutdown_connections 2024-03-06T07:22:38.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd3ac9640 1 --2- 172.21.15.59:0/994939666 >> v2:172.21.15.164:3300/0 conn(0x7f7dcc0d22b0 0x7f7dcc0d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd3ac9640 1 --2- 172.21.15.59:0/994939666 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dcc0d1910 0x7f7dcc0d1d70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd3ac9640 1 --2- 172.21.15.59:0/994939666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dcc1614d0 0x7f7dcc0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/994939666 >> 172.21.15.59:0/994939666 conn(0x7f7dcc0d3d40 msgr2=0x7f7dcc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/994939666 shutdown_connections 2024-03-06T07:22:38.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.374+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/994939666 wait complete. 2024-03-06T07:22:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.375+0000 7f7dd3ac9640 1 Processor -- start 2024-03-06T07:22:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.375+0000 7f7dd3ac9640 1 -- start start 2024-03-06T07:22:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.376+0000 7f7dd3ac9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dcc0d1910 0x7f7dcc1f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.376+0000 7f7dd3ac9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dcc0d22b0 0x7f7dcc0b9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.376+0000 7f7dd3ac9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dcc1614d0 0x7f7dcc0b9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.376+0000 7f7dd3ac9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0b0910 con 0x7f7dcc0d1910 2024-03-06T07:22:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.376+0000 7f7dd3ac9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0cfc60 con 0x7f7dcc0d22b0 2024-03-06T07:22:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.376+0000 7f7dd3ac9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0d0be0 con 0x7f7dcc1614d0 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.376+0000 7f7dd1865640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dcc0d1910 0x7f7dcc1f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.376+0000 7f7dd1865640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dcc0d1910 0x7f7dcc1f38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58266/0 (socket says 172.21.15.59:58266) 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.376+0000 7f7dd1865640 1 -- 172.21.15.59:0/1343750921 learned_addr learned my addr 172.21.15.59:0/1343750921 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.376+0000 7f7dd1064640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dcc0d22b0 0x7f7dcc0b9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.376+0000 7f7dd2066640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dcc1614d0 0x7f7dcc0b9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.377+0000 7f7dd1865640 1 -- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dcc1614d0 msgr2=0x7f7dcc0b9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.377+0000 7f7dd1865640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dcc1614d0 0x7f7dcc0b9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.377+0000 7f7dd1865640 1 -- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dcc0d22b0 msgr2=0x7f7dcc0b9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.377+0000 7f7dd1865640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dcc0d22b0 0x7f7dcc0b9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.377+0000 7f7dd1865640 1 -- 172.21.15.59:0/1343750921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc1f3e20 con 0x7f7dcc0d1910 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.377+0000 7f7dd1865640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dcc0d1910 0x7f7dcc1f38e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400e870 tx=0x7f7dc400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.377+0000 7f7dca7fc640 1 -- 172.21.15.59:0/1343750921 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4017070 con 0x7f7dcc0d1910 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.377+0000 7f7dca7fc640 1 -- 172.21.15.59:0/1343750921 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc400ee60 con 0x7f7dcc0d1910 2024-03-06T07:22:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.377+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/1343750921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dcc0cbb40 con 0x7f7dcc0d1910 2024-03-06T07:22:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.377+0000 7f7dca7fc640 1 -- 172.21.15.59:0/1343750921 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc400cb80 con 0x7f7dcc0d1910 2024-03-06T07:22:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.377+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/1343750921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dcc0cc100 con 0x7f7dcc0d1910 2024-03-06T07:22:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.378+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/1343750921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dcc068ff0 con 0x7f7dcc0d1910 2024-03-06T07:22:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.378+0000 7f7dca7fc640 1 -- 172.21.15.59:0/1343750921 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7dc401a070 con 0x7f7dcc0d1910 2024-03-06T07:22:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.379+0000 7f7dca7fc640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7db806b4d0 0x7f7db806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.379+0000 7f7dca7fc640 1 -- 172.21.15.59:0/1343750921 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7dc4089f00 con 0x7f7dcc0d1910 2024-03-06T07:22:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.379+0000 7f7dd1064640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7db806b4d0 0x7f7db806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.379+0000 7f7dd1064640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7db806b4d0 0x7f7db806d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0036000 tx=0x7f7dc003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.383+0000 7f7dca7fc640 1 -- 172.21.15.59:0/1343750921 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7dc4058740 con 0x7f7dcc0d1910 2024-03-06T07:22:38.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.564+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/1343750921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7dcc0b62b0 con 0x7f7dcc0d1910 2024-03-06T07:22:38.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.564+0000 7f7dca7fc640 1 -- 172.21.15.59:0/1343750921 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7dc405c7e0 con 0x7f7dcc0d1910 2024-03-06T07:22:38.567 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7db806b4d0 msgr2=0x7f7db806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7db806b4d0 0x7f7db806d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0036000 tx=0x7f7dc003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/1343750921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dcc0d1910 msgr2=0x7f7dcc1f38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dcc0d1910 0x7f7dcc1f38e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400e870 tx=0x7f7dc400ec70 comp rx=0 tx=0).stop 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/1343750921 shutdown_connections 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7db806b4d0 0x7f7db806d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dcc1614d0 0x7f7dcc0b9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dcc0d22b0 0x7f7dcc0b9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 --2- 172.21.15.59:0/1343750921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dcc0d1910 0x7f7dcc1f38e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/1343750921 >> 172.21.15.59:0/1343750921 conn(0x7f7dcc0d3d40 msgr2=0x7f7dcc0b3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/1343750921 shutdown_connections 2024-03-06T07:22:38.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.567+0000 7f7dd3ac9640 1 -- 172.21.15.59:0/1343750921 wait complete. 2024-03-06T07:22:38.579 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:38.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.681+0000 7f990133a640 1 Processor -- start 2024-03-06T07:22:38.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.681+0000 7f990133a640 1 -- start start 2024-03-06T07:22:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.682+0000 7f990133a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f98fc0d0af0 0x7f98fc0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.682+0000 7f990133a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc0cd4b0 con 0x7f98fc15c050 2024-03-06T07:22:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.682+0000 7f990133a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc0cd790 con 0x7f98fc06fd10 2024-03-06T07:22:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.682+0000 7f990133a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98fc0cda70 con 0x7f98fc0d0af0 2024-03-06T07:22:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.682+0000 7f98fb7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f98fc0d0af0 0x7f98fc0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.682+0000 7f98faffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f98fc15c050 0x7f98fc06f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51394/0 (socket says 172.21.15.59:51394) 2024-03-06T07:22:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.682+0000 7f98faffd640 1 -- 172.21.15.59:0/3118895 learned_addr learned my addr 172.21.15.59:0/3118895 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:38.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.682+0000 7f98fb7fe640 1 -- 172.21.15.59:0/3118895 >> v1:172.21.15.164:6789/0 conn(0x7f98fc06fd10 legacy=0x7f98fc0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.682+0000 7f98fb7fe640 1 -- 172.21.15.59:0/3118895 >> v1:172.21.15.59:6789/0 conn(0x7f98fc15c050 legacy=0x7f98fc06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.682+0000 7f98fb7fe640 1 -- 172.21.15.59:0/3118895 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0cdd50 con 0x7f98fc0d0af0 2024-03-06T07:22:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.683+0000 7f98fb7fe640 1 --2- 172.21.15.59:0/3118895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98fc0d0af0 0x7f98fc0cce70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009d90 tx=0x7f98ec02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443a0fe36eeb0d32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.683+0000 7f98f9ffb640 1 -- 172.21.15.59:0/3118895 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec039070 con 0x7f98fc0d0af0 2024-03-06T07:22:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.683+0000 7f98f9ffb640 1 -- 172.21.15.59:0/3118895 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec02cc60 con 0x7f98fc0d0af0 2024-03-06T07:22:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.683+0000 7f98f9ffb640 1 -- 172.21.15.59:0/3118895 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec034660 con 0x7f98fc0d0af0 2024-03-06T07:22:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.684+0000 7f990133a640 1 -- 172.21.15.59:0/3118895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98fc0d0af0 msgr2=0x7f98fc0cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.684+0000 7f990133a640 1 --2- 172.21.15.59:0/3118895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98fc0d0af0 0x7f98fc0cce70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009d90 tx=0x7f98ec02beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.684+0000 7f990133a640 1 -- 172.21.15.59:0/3118895 shutdown_connections 2024-03-06T07:22:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.684+0000 7f990133a640 1 --2- 172.21.15.59:0/3118895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98fc0d0af0 0x7f98fc0cce70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.684+0000 7f990133a640 1 -- 172.21.15.59:0/3118895 >> 172.21.15.59:0/3118895 conn(0x7f98fc0b3800 msgr2=0x7f98fc0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.684+0000 7f990133a640 1 -- 172.21.15.59:0/3118895 shutdown_connections 2024-03-06T07:22:38.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.684+0000 7f990133a640 1 -- 172.21.15.59:0/3118895 wait complete. 2024-03-06T07:22:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.685+0000 7f990133a640 1 Processor -- start 2024-03-06T07:22:38.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.685+0000 7f990133a640 1 -- start start 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f990133a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98fc06fd10 0x7f98fc06cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f990133a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98fc0d0af0 0x7f98fc0cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f990133a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98fc15c050 0x7f98fc145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f990133a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc166e50 con 0x7f98fc0d0af0 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f990133a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc166cd0 con 0x7f98fc06fd10 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f990133a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc166fd0 con 0x7f98fc15c050 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98fa7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98fc0d0af0 0x7f98fc0cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98fa7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98fc0d0af0 0x7f98fc0cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58288/0 (socket says 172.21.15.59:58288) 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98faffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98fc06fd10 0x7f98fc06cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98fa7fc640 1 -- 172.21.15.59:0/4092593076 learned_addr learned my addr 172.21.15.59:0/4092593076 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98fb7fe640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98fc15c050 0x7f98fc145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98fa7fc640 1 -- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98fc15c050 msgr2=0x7f98fc145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98fa7fc640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98fc15c050 0x7f98fc145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98fa7fc640 1 -- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98fc06fd10 msgr2=0x7f98fc06cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98fa7fc640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98fc06fd10 0x7f98fc06cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98fa7fc640 1 -- 172.21.15.59:0/4092593076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc06d010 con 0x7f98fc0d0af0 2024-03-06T07:22:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98fa7fc640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98fc0d0af0 0x7f98fc0cb0f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f98e400c3c0 tx=0x7f98e400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98f37fe640 1 -- 172.21.15.59:0/4092593076 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e40052d0 con 0x7f98fc0d0af0 2024-03-06T07:22:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98f37fe640 1 -- 172.21.15.59:0/4092593076 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e4005490 con 0x7f98fc0d0af0 2024-03-06T07:22:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f990133a640 1 -- 172.21.15.59:0/4092593076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc0b48f0 con 0x7f98fc0d0af0 2024-03-06T07:22:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f98f37fe640 1 -- 172.21.15.59:0/4092593076 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e4007260 con 0x7f98fc0d0af0 2024-03-06T07:22:38.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.686+0000 7f990133a640 1 -- 172.21.15.59:0/4092593076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc0b5190 con 0x7f98fc0d0af0 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.687+0000 7f990133a640 1 -- 172.21.15.59:0/4092593076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc09fe60 con 0x7f98fc0d0af0 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.688+0000 7f98f37fe640 1 -- 172.21.15.59:0/4092593076 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98e4007400 con 0x7f98fc0d0af0 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.688+0000 7f98f37fe640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f98dc06b500 0x7f98dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.688+0000 7f98f37fe640 1 -- 172.21.15.59:0/4092593076 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f98e40896b0 con 0x7f98fc0d0af0 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.688+0000 7f98faffd640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f98dc06b500 0x7f98dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.689+0000 7f98faffd640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f98dc06b500 0x7f98dc06d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f98f4005880 tx=0x7f98f4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.693+0000 7f98f37fe640 1 -- 172.21.15.59:0/4092593076 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98e4057dd0 con 0x7f98fc0d0af0 2024-03-06T07:22:38.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.872+0000 7f990133a640 1 -- 172.21.15.59:0/4092593076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f98fc068ff0 con 0x7f98fc0d0af0 2024-03-06T07:22:38.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.873+0000 7f98f37fe640 1 -- 172.21.15.59:0/4092593076 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f98e405be70 con 0x7f98fc0d0af0 2024-03-06T07:22:38.875 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:22:38.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 -- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f98dc06b500 msgr2=0x7f98dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f98dc06b500 0x7f98dc06d9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f98f4005880 tx=0x7f98f4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:38.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 -- 172.21.15.59:0/4092593076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98fc0d0af0 msgr2=0x7f98fc0cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98fc0d0af0 0x7f98fc0cb0f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f98e400c3c0 tx=0x7f98e400c880 comp rx=0 tx=0).stop 2024-03-06T07:22:38.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 -- 172.21.15.59:0/4092593076 shutdown_connections 2024-03-06T07:22:38.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f98dc06b500 0x7f98dc06d9c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98fc15c050 0x7f98fc145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98fc0d0af0 0x7f98fc0cb0f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 --2- 172.21.15.59:0/4092593076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98fc06fd10 0x7f98fc06cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 -- 172.21.15.59:0/4092593076 >> 172.21.15.59:0/4092593076 conn(0x7f98fc0b3800 msgr2=0x7f98fc0b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 -- 172.21.15.59:0/4092593076 shutdown_connections 2024-03-06T07:22:38.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.875+0000 7f990133a640 1 -- 172.21.15.59:0/4092593076 wait complete. 2024-03-06T07:22:38.888 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:38.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.987+0000 7fcd3d3b5640 1 Processor -- start 2024-03-06T07:22:38.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.987+0000 7fcd3d3b5640 1 -- start start 2024-03-06T07:22:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.988+0000 7fcd3d3b5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcd380cd7a0 0x7fcd380cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.988+0000 7fcd3d3b5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd380cc980 con 0x7fcd3806da40 2024-03-06T07:22:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.988+0000 7fcd3d3b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd380ccc60 con 0x7fcd381424f0 2024-03-06T07:22:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.988+0000 7fcd3d3b5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd380cd220 con 0x7fcd380cd7a0 2024-03-06T07:22:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.988+0000 7fcd377fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcd380cd7a0 0x7fcd380cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.988+0000 7fcd367fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcd3806da40 0x7fcd38071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51398/0 (socket says 172.21.15.59:51398) 2024-03-06T07:22:38.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.988+0000 7fcd367fc640 1 -- 172.21.15.59:0/1111129329 learned_addr learned my addr 172.21.15.59:0/1111129329 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:38.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.988+0000 7fcd377fe640 1 -- 172.21.15.59:0/1111129329 >> v1:172.21.15.164:6789/0 conn(0x7fcd381424f0 legacy=0x7fcd3806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.988+0000 7fcd377fe640 1 -- 172.21.15.59:0/1111129329 >> v1:172.21.15.59:6789/0 conn(0x7fcd3806da40 legacy=0x7fcd38071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.988+0000 7fcd377fe640 1 -- 172.21.15.59:0/1111129329 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd380ccf40 con 0x7fcd380cd7a0 2024-03-06T07:22:38.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.988+0000 7fcd35ffb640 1 -- 172.21.15.59:0/1111129329 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958326676 0 0) 0x7fcd380cc980 con 0x7fcd3806da40 2024-03-06T07:22:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.989+0000 7fcd377fe640 1 --2- 172.21.15.59:0/1111129329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380cd7a0 0x7fcd380cc2f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009d90 tx=0x7fcd2802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1d822c58252a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.989+0000 7fcd35ffb640 1 -- 172.21.15.59:0/1111129329 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28039070 con 0x7fcd380cd7a0 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.989+0000 7fcd35ffb640 1 -- 172.21.15.59:0/1111129329 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2802cd00 con 0x7fcd380cd7a0 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.989+0000 7fcd35ffb640 1 -- 172.21.15.59:0/1111129329 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28034660 con 0x7fcd380cd7a0 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.989+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/1111129329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380cd7a0 msgr2=0x7fcd380cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.990+0000 7fcd3d3b5640 1 --2- 172.21.15.59:0/1111129329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380cd7a0 0x7fcd380cc2f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009d90 tx=0x7fcd2802bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.990+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/1111129329 shutdown_connections 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.990+0000 7fcd3d3b5640 1 --2- 172.21.15.59:0/1111129329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380cd7a0 0x7fcd380cc2f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.990+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/1111129329 >> 172.21.15.59:0/1111129329 conn(0x7fcd380cf8a0 msgr2=0x7fcd380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.990+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/1111129329 shutdown_connections 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.990+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/1111129329 wait complete. 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.991+0000 7fcd3d3b5640 1 Processor -- start 2024-03-06T07:22:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.991+0000 7fcd3d3b5640 1 -- start start 2024-03-06T07:22:38.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd3d3b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd3806da40 0x7fcd3813f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd3d3b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd380cd7a0 0x7fcd380aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd3d3b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd381424f0 0x7fcd3806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd3d3b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3815d070 con 0x7fcd3806da40 2024-03-06T07:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd3d3b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3815cef0 con 0x7fcd380cd7a0 2024-03-06T07:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd3d3b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3815d1f0 con 0x7fcd381424f0 2024-03-06T07:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd36ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd3806da40 0x7fcd3813f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd36ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd3806da40 0x7fcd3813f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58298/0 (socket says 172.21.15.59:58298) 2024-03-06T07:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd36ffd640 1 -- 172.21.15.59:0/275276121 learned_addr learned my addr 172.21.15.59:0/275276121 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd367fc640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd380cd7a0 0x7fcd380aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd377fe640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd381424f0 0x7fcd3806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd36ffd640 1 -- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd381424f0 msgr2=0x7fcd3806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd36ffd640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd381424f0 0x7fcd3806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd36ffd640 1 -- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd380cd7a0 msgr2=0x7fcd380aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd36ffd640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd380cd7a0 0x7fcd380aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd36ffd640 1 -- 172.21.15.59:0/275276121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd380a9fa0 con 0x7fcd3806da40 2024-03-06T07:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd377fe640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd381424f0 0x7fcd3806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd36ffd640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd3806da40 0x7fcd3813f210 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000c400 tx=0x7fcd3000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd1f7fe640 1 -- 172.21.15.59:0/275276121 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30017070 con 0x7fcd3806da40 2024-03-06T07:22:38.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd1f7fe640 1 -- 172.21.15.59:0/275276121 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd300052d0 con 0x7fcd3806da40 2024-03-06T07:22:38.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/275276121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd380aa280 con 0x7fcd3806da40 2024-03-06T07:22:38.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.992+0000 7fcd1f7fe640 1 -- 172.21.15.59:0/275276121 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30003830 con 0x7fcd3806da40 2024-03-06T07:22:38.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.993+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/275276121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd380aa840 con 0x7fcd3806da40 2024-03-06T07:22:38.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.993+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/275276121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3804f330 con 0x7fcd3806da40 2024-03-06T07:22:38.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.994+0000 7fcd1f7fe640 1 -- 172.21.15.59:0/275276121 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd3001a070 con 0x7fcd3806da40 2024-03-06T07:22:38.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.994+0000 7fcd1f7fe640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd2406b4d0 0x7fcd2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.994+0000 7fcd1f7fe640 1 -- 172.21.15.59:0/275276121 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcd30089510 con 0x7fcd3806da40 2024-03-06T07:22:38.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.994+0000 7fcd367fc640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd2406b4d0 0x7fcd2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.995+0000 7fcd367fc640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd2406b4d0 0x7fcd2406d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c007f70 tx=0x7fcd2c007f00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:38.999+0000 7fcd1f7fe640 1 -- 172.21.15.59:0/275276121 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd30057cc0 con 0x7fcd3806da40 2024-03-06T07:22:39.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.215+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/275276121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd380748f0 con 0x7fcd3806da40 2024-03-06T07:22:39.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.216+0000 7fcd1f7fe640 1 -- 172.21.15.59:0/275276121 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7fcd3005bd60 con 0x7fcd3806da40 2024-03-06T07:22:39.218 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:39.218 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1759048959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1759048959},{"type":"v1","addr":"172.21.15.164:6836","nonce":1759048959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/550023536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":550023536},{"type":"v1","addr":"172.21.15.59:6833","nonce":550023536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2132697832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2132697832},{"type":"v1","addr":"172.21.15.59:6835","nonce":2132697832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:54.767210+0000","modified":"2024-03-06T07:21:56.776415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/1957940318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1957940318},{"type":"v1","addr":"172.21.15.164:6837","nonce":1957940318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-06T07:22:39.218 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 6 2024-03-06T07:22:39.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd2406b4d0 msgr2=0x7fcd2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd2406b4d0 0x7fcd2406d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c007f70 tx=0x7fcd2c007f00 comp rx=0 tx=0).stop 2024-03-06T07:22:39.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/275276121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd3806da40 msgr2=0x7fcd3813f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd3806da40 0x7fcd3813f210 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000c400 tx=0x7fcd3000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/275276121 shutdown_connections 2024-03-06T07:22:39.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd2406b4d0 0x7fcd2406d990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd381424f0 0x7fcd3806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd380cd7a0 0x7fcd380aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 --2- 172.21.15.59:0/275276121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd3806da40 0x7fcd3813f210 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/275276121 >> 172.21.15.59:0/275276121 conn(0x7fcd380cf8a0 msgr2=0x7fcd3806c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/275276121 shutdown_connections 2024-03-06T07:22:39.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.218+0000 7fcd3d3b5640 1 -- 172.21.15.59:0/275276121 wait complete. 2024-03-06T07:22:39.230 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:39.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.330+0000 7f42c75e2640 1 Processor -- start 2024-03-06T07:22:39.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.330+0000 7f42c75e2640 1 -- start start 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42c75e2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f42c00cf850 0x7f42c00cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42c75e2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c015e380 con 0x7f42c006cf20 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42c75e2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c00cdc10 con 0x7f42c00d00c0 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42c75e2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c006ca90 con 0x7f42c00cf850 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42c4b7d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f42c00cf850 0x7f42c00cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42c5b7f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f42c006cf20 0x7f42c006d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51404/0 (socket says 172.21.15.59:51404) 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42c5b7f640 1 -- 172.21.15.59:0/4063757011 learned_addr learned my addr 172.21.15.59:0/4063757011 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42c4b7d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f42c00cf850 0x7f42c00cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58300/0 (socket says 172.21.15.59:58300) 2024-03-06T07:22:39.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42c4b7d640 1 -- 172.21.15.59:0/4063757011 >> v1:172.21.15.164:6789/0 conn(0x7f42c00d00c0 legacy=0x7f42c00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42c4b7d640 1 -- 172.21.15.59:0/4063757011 >> v1:172.21.15.59:6789/0 conn(0x7f42c006cf20 legacy=0x7f42c006d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42b7fff640 1 -- 172.21.15.59:0/4063757011 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581611485 0 0) 0x7f42c00cdc10 con 0x7f42c00d00c0 2024-03-06T07:22:39.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.331+0000 7f42c4b7d640 1 -- 172.21.15.59:0/4063757011 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c006c160 con 0x7f42c00cf850 2024-03-06T07:22:39.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.332+0000 7f42c4b7d640 1 --2- 172.21.15.59:0/4063757011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c00cf850 0x7f42c00cd3c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f42b00097a0 tx=0x7f42b002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ae5197a228c0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.332+0000 7f42b7fff640 1 -- 172.21.15.59:0/4063757011 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0037070 con 0x7f42c00cf850 2024-03-06T07:22:39.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.332+0000 7f42b7fff640 1 -- 172.21.15.59:0/4063757011 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b0030710 con 0x7f42c00cf850 2024-03-06T07:22:39.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.332+0000 7f42b7fff640 1 -- 172.21.15.59:0/4063757011 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0030ae0 con 0x7f42c00cf850 2024-03-06T07:22:39.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.332+0000 7f42c75e2640 1 -- 172.21.15.59:0/4063757011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c00cf850 msgr2=0x7f42c00cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.332+0000 7f42c75e2640 1 --2- 172.21.15.59:0/4063757011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c00cf850 0x7f42c00cd3c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f42b00097a0 tx=0x7f42b002b790 comp rx=0 tx=0).stop 2024-03-06T07:22:39.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.332+0000 7f42c75e2640 1 -- 172.21.15.59:0/4063757011 shutdown_connections 2024-03-06T07:22:39.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.333+0000 7f42c75e2640 1 --2- 172.21.15.59:0/4063757011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c00cf850 0x7f42c00cd3c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.333+0000 7f42c75e2640 1 -- 172.21.15.59:0/4063757011 >> 172.21.15.59:0/4063757011 conn(0x7f42c00b35a0 msgr2=0x7f42c00b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.333+0000 7f42c75e2640 1 -- 172.21.15.59:0/4063757011 shutdown_connections 2024-03-06T07:22:39.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.333+0000 7f42c75e2640 1 -- 172.21.15.59:0/4063757011 wait complete. 2024-03-06T07:22:39.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.334+0000 7f42c75e2640 1 Processor -- start 2024-03-06T07:22:39.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.334+0000 7f42c75e2640 1 -- start start 2024-03-06T07:22:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.334+0000 7f42c75e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c006cf20 0x7f42c0151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.334+0000 7f42c75e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42c00cf850 0x7f42c00b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c75e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42c00d00c0 0x7f42c00c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c75e2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c00cd900 con 0x7f42c006cf20 2024-03-06T07:22:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c75e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c006c720 con 0x7f42c00cf850 2024-03-06T07:22:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c75e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c00b1030 con 0x7f42c00d00c0 2024-03-06T07:22:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c537e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c006cf20 0x7f42c0151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c537e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c006cf20 0x7f42c0151890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58302/0 (socket says 172.21.15.59:58302) 2024-03-06T07:22:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c537e640 1 -- 172.21.15.59:0/1232951141 learned_addr learned my addr 172.21.15.59:0/1232951141 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c4b7d640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42c00cf850 0x7f42c00b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c5b7f640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42c00d00c0 0x7f42c00c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c537e640 1 -- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42c00d00c0 msgr2=0x7f42c00c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c537e640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42c00d00c0 0x7f42c00c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c537e640 1 -- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42c00cf850 msgr2=0x7f42c00b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c537e640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42c00cf850 0x7f42c00b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c537e640 1 -- 172.21.15.59:0/1232951141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c00b4c70 con 0x7f42c006cf20 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c537e640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c006cf20 0x7f42c0151890 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00c410 tx=0x7f42bc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42b5ffb640 1 -- 172.21.15.59:0/1232951141 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc017070 con 0x7f42c006cf20 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42b5ffb640 1 -- 172.21.15.59:0/1232951141 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc0035f0 con 0x7f42c006cf20 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42b5ffb640 1 -- 172.21.15.59:0/1232951141 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0039e0 con 0x7f42c006cf20 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.335+0000 7f42c75e2640 1 -- 172.21.15.59:0/1232951141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c00b4f50 con 0x7f42c006cf20 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.336+0000 7f42c75e2640 1 -- 172.21.15.59:0/1232951141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c0151dd0 con 0x7f42c006cf20 2024-03-06T07:22:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.336+0000 7f42c75e2640 1 -- 172.21.15.59:0/1232951141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c00b5120 con 0x7f42c006cf20 2024-03-06T07:22:39.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.337+0000 7f42b5ffb640 1 -- 172.21.15.59:0/1232951141 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42bc01a070 con 0x7f42c006cf20 2024-03-06T07:22:39.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.337+0000 7f42b5ffb640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a406b500 0x7f42a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.337+0000 7f42b5ffb640 1 -- 172.21.15.59:0/1232951141 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f42bc089670 con 0x7f42c006cf20 2024-03-06T07:22:39.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.337+0000 7f42c4b7d640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a406b500 0x7f42a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.338+0000 7f42c4b7d640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a406b500 0x7f42a406d9c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f42b0034000 tx=0x7f42b002f000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.341+0000 7f42b5ffb640 1 -- 172.21.15.59:0/1232951141 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42c00b5120 con 0x7f42c006cf20 2024-03-06T07:22:39.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.524+0000 7f42c75e2640 1 -- 172.21.15.59:0/1232951141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42c004f330 con 0x7f42c006cf20 2024-03-06T07:22:39.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.525+0000 7f42b5ffb640 1 -- 172.21.15.59:0/1232951141 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16842 (secure 0 0 0) 0x7f42bc057d90 con 0x7f42c006cf20 2024-03-06T07:22:39.527 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:39.527 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":22,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:36.536165+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":1}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 -- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a406b500 msgr2=0x7f42a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a406b500 0x7f42a406d9c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f42b0034000 tx=0x7f42b002f000 comp rx=0 tx=0).stop 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 -- 172.21.15.59:0/1232951141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c006cf20 msgr2=0x7f42c0151890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c006cf20 0x7f42c0151890 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00c410 tx=0x7f42bc00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 -- 172.21.15.59:0/1232951141 shutdown_connections 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a406b500 0x7f42a406d9c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42c00d00c0 0x7f42c00c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42c00cf850 0x7f42c00b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 --2- 172.21.15.59:0/1232951141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42c006cf20 0x7f42c0151890 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 -- 172.21.15.59:0/1232951141 >> 172.21.15.59:0/1232951141 conn(0x7f42c00b35a0 msgr2=0x7f42c00b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 -- 172.21.15.59:0/1232951141 shutdown_connections 2024-03-06T07:22:39.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.527+0000 7f42c75e2640 1 -- 172.21.15.59:0/1232951141 wait complete. 2024-03-06T07:22:39.540 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:22:39.540 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.639+0000 7fec0b7bc640 1 Processor -- start 2024-03-06T07:22:39.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.639+0000 7fec0b7bc640 1 -- start start 2024-03-06T07:22:39.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.640+0000 7fec0b7bc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fec0415bb20 0x7fec040d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.640+0000 7fec0b7bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cd8c0 0x7fec040d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.640+0000 7fec0b7bc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec04161980 con 0x7fec040cfff0 2024-03-06T07:22:39.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.640+0000 7fec0b7bc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec04161c60 con 0x7fec0415bb20 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.640+0000 7fec0b7bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec04161f40 con 0x7fec040cd8c0 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.640+0000 7fec09d59640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cd8c0 0x7fec040d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.640+0000 7fec09558640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fec0415bb20 0x7fec040d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.640+0000 7fec09d59640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cd8c0 0x7fec040d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:33752/0 (socket says 172.21.15.59:33752) 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.640+0000 7fec09d59640 1 -- 172.21.15.59:0/3218391471 learned_addr learned my addr 172.21.15.59:0/3218391471 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.640+0000 7fec08d57640 1 --1- 172.21.15.59:0/3218391471 >> v1:172.21.15.164:6789/0 conn(0x7fec040cfff0 0x7fec040ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:34536/0 (socket says 172.21.15.59:34536) 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.641+0000 7fec09558640 1 -- 172.21.15.59:0/3218391471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cd8c0 msgr2=0x7fec040d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.641+0000 7fec09558640 1 --2- 172.21.15.59:0/3218391471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cd8c0 0x7fec040d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.641+0000 7fec09558640 1 -- 172.21.15.59:0/3218391471 >> v1:172.21.15.164:6789/0 conn(0x7fec040cfff0 legacy=0x7fec040ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.641+0000 7fec03fff640 1 -- 172.21.15.59:0/3218391471 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525705463 0 0) 0x7fec04161980 con 0x7fec040cfff0 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.641+0000 7fec09558640 1 -- 172.21.15.59:0/3218391471 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec04162220 con 0x7fec0415bb20 2024-03-06T07:22:39.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.641+0000 7fec09d59640 1 --2- 172.21.15.59:0/3218391471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cd8c0 0x7fec040d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.641+0000 7fec09558640 1 --2- 172.21.15.59:0/3218391471 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec0415bb20 0x7fec040d1df0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7febf4009d90 tx=0x7febf402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=378c130ebc3eeb9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.642+0000 7fec03fff640 1 -- 172.21.15.59:0/3218391471 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf4039070 con 0x7fec0415bb20 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.642+0000 7fec03fff640 1 -- 172.21.15.59:0/3218391471 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf402cc50 con 0x7fec0415bb20 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.642+0000 7fec03fff640 1 -- 172.21.15.59:0/3218391471 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf4034440 con 0x7fec0415bb20 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.642+0000 7fec0b7bc640 1 -- 172.21.15.59:0/3218391471 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec0415bb20 msgr2=0x7fec040d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.642+0000 7fec0b7bc640 1 --2- 172.21.15.59:0/3218391471 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec0415bb20 0x7fec040d1df0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7febf4009d90 tx=0x7febf402bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.642+0000 7fec0b7bc640 1 -- 172.21.15.59:0/3218391471 shutdown_connections 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.642+0000 7fec0b7bc640 1 --2- 172.21.15.59:0/3218391471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cd8c0 0x7fec040d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.642+0000 7fec0b7bc640 1 --2- 172.21.15.59:0/3218391471 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec0415bb20 0x7fec040d1df0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.642+0000 7fec0b7bc640 1 -- 172.21.15.59:0/3218391471 >> 172.21.15.59:0/3218391471 conn(0x7fec0409a9f0 msgr2=0x7fec0409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.642+0000 7fec0b7bc640 1 -- 172.21.15.59:0/3218391471 shutdown_connections 2024-03-06T07:22:39.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.642+0000 7fec0b7bc640 1 -- 172.21.15.59:0/3218391471 wait complete. 2024-03-06T07:22:39.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.643+0000 7fec0b7bc640 1 Processor -- start 2024-03-06T07:22:39.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.643+0000 7fec0b7bc640 1 -- start start 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.644+0000 7fec0b7bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec040cd8c0 0x7fec040c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.644+0000 7fec0b7bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cfff0 0x7fec0406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.644+0000 7fec0b7bc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec0415bb20 0x7fec041e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.644+0000 7fec0b7bc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec040d24b0 con 0x7fec040cd8c0 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.644+0000 7fec0b7bc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec040d2330 con 0x7fec0415bb20 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.644+0000 7fec0b7bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec040d1980 con 0x7fec040cfff0 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.644+0000 7fec09558640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec040cd8c0 0x7fec040c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.644+0000 7fec09d59640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec0415bb20 0x7fec041e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.644+0000 7fec09558640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec040cd8c0 0x7fec040c8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58314/0 (socket says 172.21.15.59:58314) 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.644+0000 7fec09558640 1 -- 172.21.15.59:0/751317263 learned_addr learned my addr 172.21.15.59:0/751317263 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.644+0000 7fec08d57640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cfff0 0x7fec0406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec09d59640 1 -- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cfff0 msgr2=0x7fec0406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec09d59640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cfff0 0x7fec0406ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec09d59640 1 -- 172.21.15.59:0/751317263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec040cd8c0 msgr2=0x7fec040c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec09d59640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec040cd8c0 0x7fec040c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec09d59640 1 -- 172.21.15.59:0/751317263 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec040c6f80 con 0x7fec0415bb20 2024-03-06T07:22:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec09558640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec040cd8c0 0x7fec040c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec09d59640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec0415bb20 0x7fec041e8ae0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7febfc00e870 tx=0x7febfc00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec01ffb640 1 -- 172.21.15.59:0/751317263 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc017070 con 0x7fec0415bb20 2024-03-06T07:22:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec01ffb640 1 -- 172.21.15.59:0/751317263 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc004020 con 0x7fec0415bb20 2024-03-06T07:22:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec01ffb640 1 -- 172.21.15.59:0/751317263 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc0043f0 con 0x7fec0415bb20 2024-03-06T07:22:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec0b7bc640 1 -- 172.21.15.59:0/751317263 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec040c7260 con 0x7fec0415bb20 2024-03-06T07:22:39.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.645+0000 7fec0b7bc640 1 -- 172.21.15.59:0/751317263 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec040c7b00 con 0x7fec0415bb20 2024-03-06T07:22:39.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.646+0000 7fec0b7bc640 1 -- 172.21.15.59:0/751317263 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec040c7430 con 0x7fec0415bb20 2024-03-06T07:22:39.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.647+0000 7fec01ffb640 1 -- 172.21.15.59:0/751317263 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febfc00c940 con 0x7fec0415bb20 2024-03-06T07:22:39.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.647+0000 7fec01ffb640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7febf006b4d0 0x7febf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.647+0000 7fec01ffb640 1 -- 172.21.15.59:0/751317263 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7febfc08a2b0 con 0x7fec0415bb20 2024-03-06T07:22:39.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.647+0000 7fec09558640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7febf006b4d0 0x7febf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.648+0000 7fec09558640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7febf006b4d0 0x7febf006d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7febf4000f80 tx=0x7febf4030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.652+0000 7fec01ffb640 1 -- 172.21.15.59:0/751317263 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7febfc058bb0 con 0x7fec0415bb20 2024-03-06T07:22:39.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.869+0000 7fec0b7bc640 1 -- 172.21.15.59:0/751317263 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec040b1820 con 0x7fec0415bb20 2024-03-06T07:22:39.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.870+0000 7fec01ffb640 1 -- 172.21.15.59:0/751317263 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7febfc05cc50 con 0x7fec0415bb20 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1759048959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1759048959},{"type":"v1","addr":"172.21.15.164:6836","nonce":1759048959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/550023536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":550023536},{"type":"v1","addr":"172.21.15.59:6833","nonce":550023536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2132697832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2132697832},{"type":"v1","addr":"172.21.15.59:6835","nonce":2132697832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:54.767210+0000","modified":"2024-03-06T07:21:56.776415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/1957940318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1957940318},{"type":"v1","addr":"172.21.15.164:6837","nonce":1957940318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-06T07:22:39.872 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 6 2024-03-06T07:22:39.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 -- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7febf006b4d0 msgr2=0x7febf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7febf006b4d0 0x7febf006d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7febf4000f80 tx=0x7febf4030000 comp rx=0 tx=0).stop 2024-03-06T07:22:39.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 -- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec0415bb20 msgr2=0x7fec041e8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec0415bb20 0x7fec041e8ae0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7febfc00e870 tx=0x7febfc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:39.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 -- 172.21.15.59:0/751317263 shutdown_connections 2024-03-06T07:22:39.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7febf006b4d0 0x7febf006d990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec0415bb20 0x7fec041e8ae0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec040cfff0 0x7fec0406ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 --2- 172.21.15.59:0/751317263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec040cd8c0 0x7fec040c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 -- 172.21.15.59:0/751317263 >> 172.21.15.59:0/751317263 conn(0x7fec0409a9f0 msgr2=0x7fec0409add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 -- 172.21.15.59:0/751317263 shutdown_connections 2024-03-06T07:22:39.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.872+0000 7fec0b7bc640 1 -- 172.21.15.59:0/751317263 wait complete. 2024-03-06T07:22:39.884 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:39.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.984+0000 7fd5093d7640 1 Processor -- start 2024-03-06T07:22:39.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.984+0000 7fd5093d7640 1 -- start start 2024-03-06T07:22:39.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd5093d7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd5040d0320 0x7fd5040b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd5093d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0700 0x7fd5040b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd5093d7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50415c270 con 0x7fd5040ccae0 2024-03-06T07:22:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd5093d7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5040b04e0 con 0x7fd5040d0320 2024-03-06T07:22:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd5093d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5040d0e90 con 0x7fd5040d0700 2024-03-06T07:22:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd502ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd5040d0320 0x7fd5040b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd502ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd5040d0320 0x7fd5040b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58326/0 (socket says 172.21.15.59:58326) 2024-03-06T07:22:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd502ffd640 1 -- 172.21.15.59:0/3089418404 learned_addr learned my addr 172.21.15.59:0/3089418404 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd5027fc640 1 --2- 172.21.15.59:0/3089418404 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0700 0x7fd5040b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd502ffd640 1 -- 172.21.15.59:0/3089418404 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0700 msgr2=0x7fd5040b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd502ffd640 1 --2- 172.21.15.59:0/3089418404 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0700 0x7fd5040b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd502ffd640 1 -- 172.21.15.59:0/3089418404 >> v1:172.21.15.164:6789/0 conn(0x7fd5040ccae0 legacy=0x7fd5040ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd501ffb640 1 -- 172.21.15.59:0/3089418404 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141685115 0 0) 0x7fd50415c270 con 0x7fd5040ccae0 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.985+0000 7fd502ffd640 1 -- 172.21.15.59:0/3089418404 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5040b5ae0 con 0x7fd5040d0320 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.986+0000 7fd502ffd640 1 --2- 172.21.15.59:0/3089418404 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5040d0320 0x7fd5040b4f70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc009870 tx=0x7fd4fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8146fc9b31caa5f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.986+0000 7fd501ffb640 1 -- 172.21.15.59:0/3089418404 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc034070 con 0x7fd5040d0320 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.986+0000 7fd501ffb640 1 -- 172.21.15.59:0/3089418404 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4fc030840 con 0x7fd5040d0320 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.986+0000 7fd501ffb640 1 -- 172.21.15.59:0/3089418404 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc030c10 con 0x7fd5040d0320 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.987+0000 7fd5093d7640 1 -- 172.21.15.59:0/3089418404 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5040d0320 msgr2=0x7fd5040b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.987+0000 7fd5093d7640 1 --2- 172.21.15.59:0/3089418404 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5040d0320 0x7fd5040b4f70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc009870 tx=0x7fd4fc02b980 comp rx=0 tx=0).stop 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.987+0000 7fd5093d7640 1 -- 172.21.15.59:0/3089418404 shutdown_connections 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.987+0000 7fd5093d7640 1 --2- 172.21.15.59:0/3089418404 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0700 0x7fd5040b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.987+0000 7fd5093d7640 1 --2- 172.21.15.59:0/3089418404 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5040d0320 0x7fd5040b4f70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.987+0000 7fd5093d7640 1 -- 172.21.15.59:0/3089418404 >> 172.21.15.59:0/3089418404 conn(0x7fd5040724d0 msgr2=0x7fd5040728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.987+0000 7fd5093d7640 1 -- 172.21.15.59:0/3089418404 shutdown_connections 2024-03-06T07:22:39.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.987+0000 7fd5093d7640 1 -- 172.21.15.59:0/3089418404 wait complete. 2024-03-06T07:22:39.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.988+0000 7fd5093d7640 1 Processor -- start 2024-03-06T07:22:39.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.988+0000 7fd5093d7640 1 -- start start 2024-03-06T07:22:39.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5093d7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5040ccae0 0x7fd5041564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5093d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0320 0x7fd5041f7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5093d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5040d0700 0x7fd504154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5093d7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5040b60a0 con 0x7fd5040d0700 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5093d7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5040b0170 con 0x7fd5040ccae0 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5093d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5040ca790 con 0x7fd5040d0320 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5037fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5040d0700 0x7fd504154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5027fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0320 0x7fd5041f7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd502ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5040ccae0 0x7fd5041564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5027fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0320 0x7fd5041f7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:33786/0 (socket says 172.21.15.59:33786) 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5027fc640 1 -- 172.21.15.59:0/2085659354 learned_addr learned my addr 172.21.15.59:0/2085659354 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5037fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5040d0700 0x7fd504154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58328/0 (socket says 172.21.15.59:58328) 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5037fe640 1 -- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0320 msgr2=0x7fd5041f7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5037fe640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0320 0x7fd5041f7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5037fe640 1 -- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5040ccae0 msgr2=0x7fd5041564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5037fe640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5040ccae0 0x7fd5041564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5037fe640 1 -- 172.21.15.59:0/2085659354 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5040cfd00 con 0x7fd5040d0700 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5027fc640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0320 0x7fd5041f7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd502ffd640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5040ccae0 0x7fd5041564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.989+0000 7fd5037fe640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5040d0700 0x7fd504154d40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd4f400c6a0 tx=0x7fd4f400cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.990+0000 7fd4fb7fe640 1 -- 172.21.15.59:0/2085659354 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4003b70 con 0x7fd5040d0700 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.990+0000 7fd4fb7fe640 1 -- 172.21.15.59:0/2085659354 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f4003d30 con 0x7fd5040d0700 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.990+0000 7fd5093d7640 1 -- 172.21.15.59:0/2085659354 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd504156b00 con 0x7fd5040d0700 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.990+0000 7fd4fb7fe640 1 -- 172.21.15.59:0/2085659354 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f40052b0 con 0x7fd5040d0700 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.990+0000 7fd5093d7640 1 -- 172.21.15.59:0/2085659354 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5041f7590 con 0x7fd5040d0700 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.990+0000 7fd5093d7640 1 -- 172.21.15.59:0/2085659354 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5040ca2e0 con 0x7fd5040d0700 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.991+0000 7fd4fb7fe640 1 -- 172.21.15.59:0/2085659354 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4f4005450 con 0x7fd5040d0700 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.991+0000 7fd4fb7fe640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4e406b4d0 0x7fd4e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.992+0000 7fd4fb7fe640 1 -- 172.21.15.59:0/2085659354 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd4f4089f00 con 0x7fd5040d0700 2024-03-06T07:22:39.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.992+0000 7fd502ffd640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4e406b4d0 0x7fd4e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.992+0000 7fd502ffd640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4e406b4d0 0x7fd4e406d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc036000 tx=0x7fd4fc008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:39.996+0000 7fd4fb7fe640 1 -- 172.21.15.59:0/2085659354 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5040ca2e0 con 0x7fd5040d0700 2024-03-06T07:22:40.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.213+0000 7fd5093d7640 1 -- 172.21.15.59:0/2085659354 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd50404f330 con 0x7fd5040d0700 2024-03-06T07:22:40.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.214+0000 7fd4fb7fe640 1 -- 172.21.15.59:0/2085659354 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4974 (secure 0 0 0) 0x7fd4f4058740 con 0x7fd5040d0700 2024-03-06T07:22:40.216 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:40.216 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1759048959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1759048959},{"type":"v1","addr":"172.21.15.164:6836","nonce":1759048959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/550023536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":550023536},{"type":"v1","addr":"172.21.15.59:6833","nonce":550023536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2132697832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2132697832},{"type":"v1","addr":"172.21.15.59:6835","nonce":2132697832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:54.767210+0000","modified":"2024-03-06T07:21:56.776415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/1957940318","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1957940318},{"type":"v1","addr":"172.21.15.164:6837","nonce":1957940318}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302]},"id":1}]} 2024-03-06T07:22:40.216 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 6 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 -- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4e406b4d0 msgr2=0x7fd4e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4e406b4d0 0x7fd4e406d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc036000 tx=0x7fd4fc008030 comp rx=0 tx=0).stop 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 -- 172.21.15.59:0/2085659354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5040d0700 msgr2=0x7fd504154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5040d0700 0x7fd504154d40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd4f400c6a0 tx=0x7fd4f400cb60 comp rx=0 tx=0).stop 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 -- 172.21.15.59:0/2085659354 shutdown_connections 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4e406b4d0 0x7fd4e406d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5040d0700 0x7fd504154d40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5040d0320 0x7fd5041f7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 --2- 172.21.15.59:0/2085659354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5040ccae0 0x7fd5041564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 -- 172.21.15.59:0/2085659354 >> 172.21.15.59:0/2085659354 conn(0x7fd5040724d0 msgr2=0x7fd5040b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 -- 172.21.15.59:0/2085659354 shutdown_connections 2024-03-06T07:22:40.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.216+0000 7fd5093d7640 1 -- 172.21.15.59:0/2085659354 wait complete. 2024-03-06T07:22:40.229 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.329+0000 7fa936bf6640 1 Processor -- start 2024-03-06T07:22:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.329+0000 7fa936bf6640 1 -- start start 2024-03-06T07:22:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.329+0000 7fa936bf6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa93015fa10 0x7fa9300ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.329+0000 7fa936bf6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9300d16e0 0x7fa9300d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.329+0000 7fa936bf6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9300cebe0 con 0x7fa93015bdd0 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.329+0000 7fa936bf6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9300ceec0 con 0x7fa93015fa10 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.329+0000 7fa936bf6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9300cf1a0 con 0x7fa9300d16e0 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.329+0000 7fa92ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa93015fa10 0x7fa9300ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.330+0000 7fa934992640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa93015bdd0 0x7fa9300c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51412/0 (socket says 172.21.15.59:51412) 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.330+0000 7fa92ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa93015fa10 0x7fa9300ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58330/0 (socket says 172.21.15.59:58330) 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.330+0000 7fa934992640 1 -- 172.21.15.59:0/802305320 learned_addr learned my addr 172.21.15.59:0/802305320 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.330+0000 7fa935193640 1 --2- 172.21.15.59:0/802305320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9300d16e0 0x7fa9300d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.330+0000 7fa92ffff640 1 -- 172.21.15.59:0/802305320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9300d16e0 msgr2=0x7fa9300d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.330+0000 7fa92ffff640 1 --2- 172.21.15.59:0/802305320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9300d16e0 0x7fa9300d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.330+0000 7fa92ffff640 1 -- 172.21.15.59:0/802305320 >> v1:172.21.15.59:6789/0 conn(0x7fa93015bdd0 legacy=0x7fa9300c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.330+0000 7fa92ffff640 1 -- 172.21.15.59:0/802305320 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9300cc8e0 con 0x7fa93015fa10 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa92ffff640 1 --2- 172.21.15.59:0/802305320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa93015fa10 0x7fa9300ce270 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0097a0 tx=0x7fa91c02b660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af4db7a77197afd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa92f7fe640 1 -- 172.21.15.59:0/802305320 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c037070 con 0x7fa93015fa10 2024-03-06T07:22:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa92f7fe640 1 -- 172.21.15.59:0/802305320 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c030710 con 0x7fa93015fa10 2024-03-06T07:22:40.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa92f7fe640 1 -- 172.21.15.59:0/802305320 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c02be60 con 0x7fa93015fa10 2024-03-06T07:22:40.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa936bf6640 1 -- 172.21.15.59:0/802305320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa93015fa10 msgr2=0x7fa9300ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa936bf6640 1 --2- 172.21.15.59:0/802305320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa93015fa10 0x7fa9300ce270 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0097a0 tx=0x7fa91c02b660 comp rx=0 tx=0).stop 2024-03-06T07:22:40.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa936bf6640 1 -- 172.21.15.59:0/802305320 shutdown_connections 2024-03-06T07:22:40.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa936bf6640 1 --2- 172.21.15.59:0/802305320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9300d16e0 0x7fa9300d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa936bf6640 1 --2- 172.21.15.59:0/802305320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa93015fa10 0x7fa9300ce270 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa936bf6640 1 -- 172.21.15.59:0/802305320 >> 172.21.15.59:0/802305320 conn(0x7fa93009a9f0 msgr2=0x7fa93009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa936bf6640 1 -- 172.21.15.59:0/802305320 shutdown_connections 2024-03-06T07:22:40.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.331+0000 7fa936bf6640 1 -- 172.21.15.59:0/802305320 wait complete. 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.332+0000 7fa936bf6640 1 Processor -- start 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.332+0000 7fa936bf6640 1 -- start start 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.333+0000 7fa936bf6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9300d16e0 0x7fa930151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.333+0000 7fa936bf6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa93015bdd0 0x7fa93006c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.333+0000 7fa936bf6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa93015c3c0 0x7fa9300b6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.333+0000 7fa936bf6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93015c1b0 con 0x7fa93015c3c0 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.333+0000 7fa936bf6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9300cc5d0 con 0x7fa93015bdd0 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.333+0000 7fa936bf6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9300ce7b0 con 0x7fa9300d16e0 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.333+0000 7fa935193640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa93015c3c0 0x7fa9300b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.333+0000 7fa935193640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa93015c3c0 0x7fa9300b6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58342/0 (socket says 172.21.15.59:58342) 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.333+0000 7fa935193640 1 -- 172.21.15.59:0/4283199189 learned_addr learned my addr 172.21.15.59:0/4283199189 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.333+0000 7fa934992640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9300d16e0 0x7fa930151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.333+0000 7fa92ffff640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa93015bdd0 0x7fa93006c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.334+0000 7fa935193640 1 -- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9300d16e0 msgr2=0x7fa930151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.334+0000 7fa935193640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9300d16e0 0x7fa930151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.334+0000 7fa935193640 1 -- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa93015bdd0 msgr2=0x7fa93006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.334+0000 7fa935193640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa93015bdd0 0x7fa93006c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.334+0000 7fa935193640 1 -- 172.21.15.59:0/4283199189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa930152410 con 0x7fa93015c3c0 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.334+0000 7fa935193640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa93015c3c0 0x7fa9300b6010 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa92800efc0 tx=0x7fa92800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.334+0000 7fa92d7fa640 1 -- 172.21.15.59:0/4283199189 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928018020 con 0x7fa93015c3c0 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.334+0000 7fa92d7fa640 1 -- 172.21.15.59:0/4283199189 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa928014480 con 0x7fa93015c3c0 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.334+0000 7fa936bf6640 1 -- 172.21.15.59:0/4283199189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93006d120 con 0x7fa93015c3c0 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.334+0000 7fa92d7fa640 1 -- 172.21.15.59:0/4283199189 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928014870 con 0x7fa93015c3c0 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.334+0000 7fa936bf6640 1 -- 172.21.15.59:0/4283199189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93006d6e0 con 0x7fa93015c3c0 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.335+0000 7fa936bf6640 1 -- 172.21.15.59:0/4283199189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9300d1250 con 0x7fa93015c3c0 2024-03-06T07:22:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.335+0000 7fa92d7fa640 1 -- 172.21.15.59:0/4283199189 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa928005d10 con 0x7fa93015c3c0 2024-03-06T07:22:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.336+0000 7fa92d7fa640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa91406b4d0 0x7fa91406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.336+0000 7fa92d7fa640 1 -- 172.21.15.59:0/4283199189 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa92808ab60 con 0x7fa93015c3c0 2024-03-06T07:22:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.336+0000 7fa934992640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa91406b4d0 0x7fa91406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.336+0000 7fa934992640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa91406b4d0 0x7fa91406d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa924000f60 tx=0x7fa924007e20 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.340+0000 7fa92d7fa640 1 -- 172.21.15.59:0/4283199189 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9300d1250 con 0x7fa93015c3c0 2024-03-06T07:22:40.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.522+0000 7fa936bf6640 1 -- 172.21.15.59:0/4283199189 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9300b7a40 con 0x7fa93015c3c0 2024-03-06T07:22:40.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.523+0000 7fa92d7fa640 1 -- 172.21.15.59:0/4283199189 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16842 (secure 0 0 0) 0x7fa928022070 con 0x7fa93015c3c0 2024-03-06T07:22:40.525 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:40.526 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":22,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:36.536165+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:52.587542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:53.743634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":1}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:40.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.525+0000 7fa936bf6640 1 -- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa91406b4d0 msgr2=0x7fa91406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.525+0000 7fa936bf6640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa91406b4d0 0x7fa91406d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa924000f60 tx=0x7fa924007e20 comp rx=0 tx=0).stop 2024-03-06T07:22:40.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.525+0000 7fa936bf6640 1 -- 172.21.15.59:0/4283199189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa93015c3c0 msgr2=0x7fa9300b6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.525+0000 7fa936bf6640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa93015c3c0 0x7fa9300b6010 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa92800efc0 tx=0x7fa92800c490 comp rx=0 tx=0).stop 2024-03-06T07:22:40.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.526+0000 7fa936bf6640 1 -- 172.21.15.59:0/4283199189 shutdown_connections 2024-03-06T07:22:40.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.526+0000 7fa936bf6640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa91406b4d0 0x7fa91406d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.526+0000 7fa936bf6640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa93015c3c0 0x7fa9300b6010 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.526+0000 7fa936bf6640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa93015bdd0 0x7fa93006c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.526+0000 7fa936bf6640 1 --2- 172.21.15.59:0/4283199189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9300d16e0 0x7fa930151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.526+0000 7fa936bf6640 1 -- 172.21.15.59:0/4283199189 >> 172.21.15.59:0/4283199189 conn(0x7fa93009a9f0 msgr2=0x7fa93009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.526+0000 7fa936bf6640 1 -- 172.21.15.59:0/4283199189 shutdown_connections 2024-03-06T07:22:40.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.526+0000 7fa936bf6640 1 -- 172.21.15.59:0/4283199189 wait complete. 2024-03-06T07:22:40.539 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:22:40.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.639+0000 7f3a68657640 1 Processor -- start 2024-03-06T07:22:40.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.640+0000 7f3a68657640 1 -- start start 2024-03-06T07:22:40.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.640+0000 7f3a68657640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a60072520 0x7f3a60066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.640+0000 7f3a68657640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3a6015d040 0x7f3a6015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.640+0000 7f3a68657640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a6006c460 con 0x7f3a600b5c30 2024-03-06T07:22:40.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.640+0000 7f3a68657640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a60067240 con 0x7f3a6015d040 2024-03-06T07:22:40.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.640+0000 7f3a68657640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6006c740 con 0x7f3a60072520 2024-03-06T07:22:40.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.641+0000 7f3a65bf2640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3a600b5c30 0x7f3a60072900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51426/0 (socket says 172.21.15.59:51426) 2024-03-06T07:22:40.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.641+0000 7f3a65bf2640 1 -- 172.21.15.59:0/4129386599 learned_addr learned my addr 172.21.15.59:0/4129386599 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:40.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.641+0000 7f3a66bf4640 1 --2- 172.21.15.59:0/4129386599 >> v2:172.21.15.164:3300/0 conn(0x7f3a6015d040 0x7f3a6015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.641+0000 7f3a663f3640 1 --2- 172.21.15.59:0/4129386599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a60072520 0x7f3a60066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.641+0000 7f3a653f1640 1 -- 172.21.15.59:0/4129386599 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590269070 0 0) 0x7f3a6006c460 con 0x7f3a600b5c30 2024-03-06T07:22:40.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.641+0000 7f3a653f1640 1 -- 172.21.15.59:0/4129386599 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4c003520 con 0x7f3a600b5c30 2024-03-06T07:22:40.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.641+0000 7f3a663f3640 1 -- 172.21.15.59:0/4129386599 >> v2:172.21.15.164:3300/0 conn(0x7f3a6015d040 msgr2=0x7f3a6015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.641+0000 7f3a663f3640 1 --2- 172.21.15.59:0/4129386599 >> v2:172.21.15.164:3300/0 conn(0x7f3a6015d040 0x7f3a6015d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.641+0000 7f3a663f3640 1 -- 172.21.15.59:0/4129386599 >> v1:172.21.15.59:6789/0 conn(0x7f3a600b5c30 legacy=0x7f3a60072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.641+0000 7f3a663f3640 1 -- 172.21.15.59:0/4129386599 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6015fb50 con 0x7f3a60072520 2024-03-06T07:22:40.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.641+0000 7f3a66bf4640 1 --2- 172.21.15.59:0/4129386599 >> v2:172.21.15.164:3300/0 conn(0x7f3a6015d040 0x7f3a6015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:40.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.642+0000 7f3a663f3640 1 --2- 172.21.15.59:0/4129386599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a60072520 0x7f3a60066d00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0097a0 tx=0x7f3a5c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d598bf54e817539 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.642+0000 7f3a653f1640 1 -- 172.21.15.59:0/4129386599 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c0315a0 con 0x7f3a60072520 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.642+0000 7f3a653f1640 1 -- 172.21.15.59:0/4129386599 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5c037070 con 0x7f3a60072520 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.642+0000 7f3a653f1640 1 -- 172.21.15.59:0/4129386599 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c031940 con 0x7f3a60072520 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.643+0000 7f3a68657640 1 -- 172.21.15.59:0/4129386599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a60072520 msgr2=0x7f3a60066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.643+0000 7f3a68657640 1 --2- 172.21.15.59:0/4129386599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a60072520 0x7f3a60066d00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0097a0 tx=0x7f3a5c02b840 comp rx=0 tx=0).stop 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.643+0000 7f3a68657640 1 -- 172.21.15.59:0/4129386599 shutdown_connections 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.643+0000 7f3a68657640 1 --2- 172.21.15.59:0/4129386599 >> v2:172.21.15.164:3300/0 conn(0x7f3a6015d040 0x7f3a6015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.643+0000 7f3a68657640 1 --2- 172.21.15.59:0/4129386599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a60072520 0x7f3a60066d00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.643+0000 7f3a68657640 1 -- 172.21.15.59:0/4129386599 >> 172.21.15.59:0/4129386599 conn(0x7f3a600b4e60 msgr2=0x7f3a600b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.643+0000 7f3a68657640 1 -- 172.21.15.59:0/4129386599 shutdown_connections 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.643+0000 7f3a68657640 1 -- 172.21.15.59:0/4129386599 wait complete. 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.644+0000 7f3a68657640 1 Processor -- start 2024-03-06T07:22:40.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.644+0000 7f3a68657640 1 -- start start 2024-03-06T07:22:40.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a68657640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a60072520 0x7f3a60157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a68657640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a600b5c30 0x7f3a600c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a68657640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a6015d040 0x7f3a601f76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a68657640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6015dbb0 con 0x7f3a600b5c30 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a68657640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6015da30 con 0x7f3a6015d040 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a68657640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6015dd30 con 0x7f3a60072520 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a65bf2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a600b5c30 0x7f3a600c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a663f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a60072520 0x7f3a60157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a65bf2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a600b5c30 0x7f3a600c9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58348/0 (socket says 172.21.15.59:58348) 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a65bf2640 1 -- 172.21.15.59:0/726776126 learned_addr learned my addr 172.21.15.59:0/726776126 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a66bf4640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a6015d040 0x7f3a601f76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a65bf2640 1 -- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a60072520 msgr2=0x7f3a60157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a65bf2640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a60072520 0x7f3a60157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a65bf2640 1 -- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a6015d040 msgr2=0x7f3a601f76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a65bf2640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a6015d040 0x7f3a601f76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a65bf2640 1 -- 172.21.15.59:0/726776126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a601f8110 con 0x7f3a600b5c30 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a65bf2640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a600b5c30 0x7f3a600c9d20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000c430 tx=0x7f3a5000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a56ffd640 1 -- 172.21.15.59:0/726776126 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50017070 con 0x7f3a600b5c30 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a56ffd640 1 -- 172.21.15.59:0/726776126 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a50005ce0 con 0x7f3a600b5c30 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a68657640 1 -- 172.21.15.59:0/726776126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a601f83f0 con 0x7f3a600b5c30 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a56ffd640 1 -- 172.21.15.59:0/726776126 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a500072f0 con 0x7f3a600b5c30 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.645+0000 7f3a68657640 1 -- 172.21.15.59:0/726776126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a600c9800 con 0x7f3a600b5c30 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.646+0000 7f3a68657640 1 -- 172.21.15.59:0/726776126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6004f330 con 0x7f3a600b5c30 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.647+0000 7f3a56ffd640 1 -- 172.21.15.59:0/726776126 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a5001a070 con 0x7f3a600b5c30 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.647+0000 7f3a56ffd640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a4406b4d0 0x7f3a4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.647+0000 7f3a56ffd640 1 -- 172.21.15.59:0/726776126 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3a50089510 con 0x7f3a600b5c30 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.647+0000 7f3a663f3640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a4406b4d0 0x7f3a4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.648+0000 7f3a663f3640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a4406b4d0 0x7f3a4406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c036000 tx=0x7f3a5c03d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.652+0000 7f3a56ffd640 1 -- 172.21.15.59:0/726776126 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a50057d50 con 0x7f3a600b5c30 2024-03-06T07:22:40.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.875+0000 7f3a68657640 1 -- 172.21.15.59:0/726776126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3a600c99d0 con 0x7f3a600b5c30 2024-03-06T07:22:40.878 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:22:40.876+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:22:40.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.877+0000 7f3a56ffd640 1 -- 172.21.15.59:0/726776126 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3a5005bdf0 con 0x7f3a600b5c30 2024-03-06T07:22:40.879 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:22:40.884 INFO:tasks.ceph.mds.d.smithi164.stderr: -2965> 2024-03-06T07:21:59.096+0000 7f97ac61b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:40.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.879+0000 7f3a68657640 1 -- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a4406b4d0 msgr2=0x7f3a4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.879+0000 7f3a68657640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a4406b4d0 0x7f3a4406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c036000 tx=0x7f3a5c03d040 comp rx=0 tx=0).stop 2024-03-06T07:22:40.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.879+0000 7f3a68657640 1 -- 172.21.15.59:0/726776126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a600b5c30 msgr2=0x7f3a600c9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.879+0000 7f3a68657640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a600b5c30 0x7f3a600c9d20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000c430 tx=0x7f3a5000c830 comp rx=0 tx=0).stop 2024-03-06T07:22:40.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.880+0000 7f3a68657640 1 -- 172.21.15.59:0/726776126 shutdown_connections 2024-03-06T07:22:40.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.880+0000 7f3a68657640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a4406b4d0 0x7f3a4406d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.880+0000 7f3a68657640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a6015d040 0x7f3a601f76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.880+0000 7f3a68657640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a600b5c30 0x7f3a600c9d20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.880+0000 7f3a68657640 1 --2- 172.21.15.59:0/726776126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a60072520 0x7f3a60157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.880+0000 7f3a68657640 1 -- 172.21.15.59:0/726776126 >> 172.21.15.59:0/726776126 conn(0x7f3a600b4e60 msgr2=0x7f3a6015eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.880+0000 7f3a68657640 1 -- 172.21.15.59:0/726776126 shutdown_connections 2024-03-06T07:22:40.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.880+0000 7f3a68657640 1 -- 172.21.15.59:0/726776126 wait complete. 2024-03-06T07:22:40.893 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:22:40.910 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:22:40.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.997+0000 7fe77a3c7640 1 Processor -- start 2024-03-06T07:22:40.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.997+0000 7fe77a3c7640 1 -- start start 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.997+0000 7fe77a3c7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe7740d0af0 0x7fe7740cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.997+0000 7fe77a3c7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7740cd4b0 con 0x7fe77415c050 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.997+0000 7fe77a3c7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7740cd790 con 0x7fe77406fd10 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.997+0000 7fe77a3c7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7740cda70 con 0x7fe7740d0af0 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.997+0000 7fe773fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe77415c050 0x7fe77406f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51442/0 (socket says 172.21.15.59:51442) 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.997+0000 7fe773fff640 1 -- 172.21.15.59:0/2625371270 learned_addr learned my addr 172.21.15.59:0/2625371270 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.997+0000 7fe778964640 1 --2- 172.21.15.59:0/2625371270 >> v2:172.21.15.59:3300/0 conn(0x7fe7740d0af0 0x7fe7740cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.998+0000 7fe778964640 1 -- 172.21.15.59:0/2625371270 >> v1:172.21.15.164:6789/0 conn(0x7fe77406fd10 legacy=0x7fe7740cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.998+0000 7fe778964640 1 -- 172.21.15.59:0/2625371270 >> v1:172.21.15.59:6789/0 conn(0x7fe77415c050 legacy=0x7fe77406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.998+0000 7fe778964640 1 -- 172.21.15.59:0/2625371270 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7740cdd50 con 0x7fe7740d0af0 2024-03-06T07:22:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.998+0000 7fe772ffd640 1 -- 172.21.15.59:0/2625371270 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501010601 0 0) 0x7fe7740cd4b0 con 0x7fe77415c050 2024-03-06T07:22:41.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.999+0000 7fe778964640 1 --2- 172.21.15.59:0/2625371270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740d0af0 0x7fe7740cce70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009d90 tx=0x7fe76c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4b32cad2bcec28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.999+0000 7fe772ffd640 1 -- 172.21.15.59:0/2625371270 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c039070 con 0x7fe7740d0af0 2024-03-06T07:22:41.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.999+0000 7fe772ffd640 1 -- 172.21.15.59:0/2625371270 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76c02cc60 con 0x7fe7740d0af0 2024-03-06T07:22:41.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.999+0000 7fe772ffd640 1 -- 172.21.15.59:0/2625371270 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76c034660 con 0x7fe7740d0af0 2024-03-06T07:22:41.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.999+0000 7fe77a3c7640 1 -- 172.21.15.59:0/2625371270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740d0af0 msgr2=0x7fe7740cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.999+0000 7fe77a3c7640 1 --2- 172.21.15.59:0/2625371270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740d0af0 0x7fe7740cce70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009d90 tx=0x7fe76c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.999+0000 7fe77a3c7640 1 -- 172.21.15.59:0/2625371270 shutdown_connections 2024-03-06T07:22:41.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.999+0000 7fe77a3c7640 1 --2- 172.21.15.59:0/2625371270 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740d0af0 0x7fe7740cce70 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.999+0000 7fe77a3c7640 1 -- 172.21.15.59:0/2625371270 >> 172.21.15.59:0/2625371270 conn(0x7fe7740b3800 msgr2=0x7fe7740b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.999+0000 7fe77a3c7640 1 -- 172.21.15.59:0/2625371270 shutdown_connections 2024-03-06T07:22:41.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:40.999+0000 7fe77a3c7640 1 -- 172.21.15.59:0/2625371270 wait complete. 2024-03-06T07:22:41.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.000+0000 7fe77a3c7640 1 Processor -- start 2024-03-06T07:22:41.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.001+0000 7fe77a3c7640 1 -- start start 2024-03-06T07:22:41.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.001+0000 7fe77a3c7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe77406fd10 0x7fe77406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.001+0000 7fe77a3c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740d0af0 0x7fe774150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.001+0000 7fe77a3c7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe77415c050 0x7fe774153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.001+0000 7fe77a3c7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe774166e50 con 0x7fe7740d0af0 2024-03-06T07:22:41.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.001+0000 7fe77a3c7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe774166cd0 con 0x7fe77415c050 2024-03-06T07:22:41.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.001+0000 7fe77a3c7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe774166fd0 con 0x7fe77406fd10 2024-03-06T07:22:41.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.001+0000 7fe7737fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740d0af0 0x7fe774150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.001+0000 7fe7737fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740d0af0 0x7fe774150ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58378/0 (socket says 172.21.15.59:58378) 2024-03-06T07:22:41.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.001+0000 7fe7737fe640 1 -- 172.21.15.59:0/3948661097 learned_addr learned my addr 172.21.15.59:0/3948661097 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:41.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.001+0000 7fe773fff640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe77406fd10 0x7fe77406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe778964640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe77415c050 0x7fe774153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe7737fe640 1 -- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe77406fd10 msgr2=0x7fe77406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe7737fe640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe77406fd10 0x7fe77406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe7737fe640 1 -- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe77415c050 msgr2=0x7fe774153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe7737fe640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe77415c050 0x7fe774153370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe7737fe640 1 -- 172.21.15.59:0/3948661097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7741514b0 con 0x7fe7740d0af0 2024-03-06T07:22:41.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe7737fe640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740d0af0 0x7fe774150ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe76400c400 tx=0x7fe76400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe770ff9640 1 -- 172.21.15.59:0/3948661097 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764017070 con 0x7fe7740d0af0 2024-03-06T07:22:41.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe770ff9640 1 -- 172.21.15.59:0/3948661097 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7640052d0 con 0x7fe7740d0af0 2024-03-06T07:22:41.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe77a3c7640 1 -- 172.21.15.59:0/3948661097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe774151790 con 0x7fe7740d0af0 2024-03-06T07:22:41.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe770ff9640 1 -- 172.21.15.59:0/3948661097 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764003830 con 0x7fe7740d0af0 2024-03-06T07:22:41.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.002+0000 7fe77a3c7640 1 -- 172.21.15.59:0/3948661097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe774151d50 con 0x7fe7740d0af0 2024-03-06T07:22:41.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.003+0000 7fe77a3c7640 1 -- 172.21.15.59:0/3948661097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77409fe60 con 0x7fe7740d0af0 2024-03-06T07:22:41.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.004+0000 7fe770ff9640 1 -- 172.21.15.59:0/3948661097 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe76401a070 con 0x7fe7740d0af0 2024-03-06T07:22:41.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.004+0000 7fe770ff9640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe76006b4d0 0x7fe76006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.004+0000 7fe770ff9640 1 -- 172.21.15.59:0/3948661097 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fe764089520 con 0x7fe7740d0af0 2024-03-06T07:22:41.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.004+0000 7fe773fff640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe76006b4d0 0x7fe76006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.005+0000 7fe773fff640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe76006b4d0 0x7fe76006d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe768007f70 tx=0x7fe768007f00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.008+0000 7fe770ff9640 1 -- 172.21.15.59:0/3948661097 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe764057d10 con 0x7fe7740d0af0 2024-03-06T07:22:41.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.232+0000 7fe77a3c7640 1 -- 172.21.15.59:0/3948661097 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe7740cbb70 con 0x7fe7740d0af0 2024-03-06T07:22:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.885+0000 7fe770ff9640 1 -- 172.21.15.59:0/3948661097 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe76405bdb0 con 0x7fe7740d0af0 2024-03-06T07:22:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 -- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe76006b4d0 msgr2=0x7fe76006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe76006b4d0 0x7fe76006d990 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe768007f70 tx=0x7fe768007f00 comp rx=0 tx=0).stop 2024-03-06T07:22:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 -- 172.21.15.59:0/3948661097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740d0af0 msgr2=0x7fe774150ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740d0af0 0x7fe774150ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe76400c400 tx=0x7fe76400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 -- 172.21.15.59:0/3948661097 shutdown_connections 2024-03-06T07:22:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe76006b4d0 0x7fe76006d990 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe77415c050 0x7fe774153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7740d0af0 0x7fe774150ab0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 --2- 172.21.15.59:0/3948661097 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe77406fd10 0x7fe77406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 -- 172.21.15.59:0/3948661097 >> 172.21.15.59:0/3948661097 conn(0x7fe7740b3800 msgr2=0x7fe7740b1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 -- 172.21.15.59:0/3948661097 shutdown_connections 2024-03-06T07:22:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:41.888+0000 7fe77a3c7640 1 -- 172.21.15.59:0/3948661097 wait complete. 2024-03-06T07:22:41.901 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:22:42.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.005+0000 7f6fde3d6640 1 Processor -- start 2024-03-06T07:22:42.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.005+0000 7f6fde3d6640 1 -- start start 2024-03-06T07:22:42.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.006+0000 7f6fde3d6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6fd80708b0 0x7f6fd8070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.006+0000 7f6fde3d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd80ce0a0 0x7f6fd80ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.006+0000 7f6fde3d6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd815afe0 con 0x7f6fd815ac00 2024-03-06T07:22:42.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.006+0000 7f6fde3d6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd8071250 con 0x7f6fd80708b0 2024-03-06T07:22:42.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.006+0000 7f6fde3d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80cdc10 con 0x7f6fd80ce0a0 2024-03-06T07:22:42.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.006+0000 7f6fd7fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6fd815ac00 0x7f6fd806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51452/0 (socket says 172.21.15.59:51452) 2024-03-06T07:22:42.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.006+0000 7f6fd7fff640 1 -- 172.21.15.59:0/1848945088 learned_addr learned my addr 172.21.15.59:0/1848945088 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131294489 0 0) 0x7f6fd815afe0 con 0x7f6fd815ac00 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fdc973640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd80ce0a0 0x7f6fd80ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd77fe640 1 --2- 172.21.15.59:0/1848945088 >> v2:172.21.15.164:3300/0 conn(0x7f6fd80708b0 0x7f6fd8070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc0003520 con 0x7f6fd815ac00 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3811520045 0 0) 0x7f6fc0003520 con 0x7f6fd815ac00 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd80ce0a0 msgr2=0x7f6fd80ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd80ce0a0 0x7f6fd80ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 >> v2:172.21.15.164:3300/0 conn(0x7f6fd80708b0 msgr2=0x7f6fd8070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 --2- 172.21.15.59:0/1848945088 >> v2:172.21.15.164:3300/0 conn(0x7f6fd80708b0 0x7f6fd8070d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd80cd000 con 0x7f6fd815ac00 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f6fc8001670 con 0x7f6fd815ac00 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 >> v1:172.21.15.59:6789/0 conn(0x7f6fd815ac00 legacy=0x7f6fd806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc00039b0 0x7f6fc0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd80ce0a0 0x7f6fc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 0x7f6fc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80cdc10 con 0x7f6fc0004d40 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc000b9c0 con 0x7f6fc00039b0 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc000bb70 con 0x7f6fd80ce0a0 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fdc973640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 0x7f6fc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd7fff640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc00039b0 0x7f6fc0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fd77fe640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd80ce0a0 0x7f6fc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fdc973640 1 -- 172.21.15.59:0/1848945088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd80ce0a0 msgr2=0x7f6fc0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fdc973640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd80ce0a0 0x7f6fc0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fdc973640 1 -- 172.21.15.59:0/1848945088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc00039b0 msgr2=0x7f6fc0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fdc973640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc00039b0 0x7f6fc0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.007+0000 7f6fdc973640 1 -- 172.21.15.59:0/1848945088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc0003520 con 0x7f6fc0004d40 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.008+0000 7f6fdc973640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 0x7f6fc00051c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009fc0 tx=0x7f6fc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8885a4a5dbd9b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.008+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0039070 con 0x7f6fc0004d40 2024-03-06T07:22:42.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.008+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd002cc20 con 0x7f6fc0004d40 2024-03-06T07:22:42.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.008+0000 7f6fd6ffd640 1 -- 172.21.15.59:0/1848945088 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0009c10 con 0x7f6fc0004d40 2024-03-06T07:22:42.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.009+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1848945088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 msgr2=0x7f6fc00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.009+0000 7f6fde3d6640 1 --2- 172.21.15.59:0/1848945088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 0x7f6fc00051c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009fc0 tx=0x7f6fc0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:42.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.009+0000 7f6fdc973640 1 -- 172.21.15.59:0/1848945088 reap_dead start 2024-03-06T07:22:42.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.009+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1848945088 shutdown_connections 2024-03-06T07:22:42.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.009+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1848945088 >> 172.21.15.59:0/1848945088 conn(0x7f6fd80b1650 msgr2=0x7f6fd80b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.009+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1848945088 shutdown_connections 2024-03-06T07:22:42.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.009+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1848945088 wait complete. 2024-03-06T07:22:42.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.010+0000 7f6fde3d6640 1 Processor -- start 2024-03-06T07:22:42.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.010+0000 7f6fde3d6640 1 -- start start 2024-03-06T07:22:42.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fde3d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 0x7f6fd81f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fde3d6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fd814e2e0 0x7f6fd813fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fde3d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd813c9f0 0x7f6fd81555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fde3d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80cd8a0 con 0x7f6fc0004d40 2024-03-06T07:22:42.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fde3d6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80cd5c0 con 0x7f6fd814e2e0 2024-03-06T07:22:42.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fde3d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80cf5d0 con 0x7f6fd813c9f0 2024-03-06T07:22:42.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fd7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 0x7f6fd81f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fd7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 0x7f6fd81f7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58384/0 (socket says 172.21.15.59:58384) 2024-03-06T07:22:42.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fd7fff640 1 -- 172.21.15.59:0/1347557842 learned_addr learned my addr 172.21.15.59:0/1347557842 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:42.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fdc973640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd813c9f0 0x7f6fd81555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fd77fe640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fd814e2e0 0x7f6fd813fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fd7fff640 1 -- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd813c9f0 msgr2=0x7f6fd81555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fd7fff640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd813c9f0 0x7f6fd81555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fd7fff640 1 -- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fd814e2e0 msgr2=0x7f6fd813fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fd7fff640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fd814e2e0 0x7f6fd813fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fd7fff640 1 -- 172.21.15.59:0/1347557842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd81f7760 con 0x7f6fc0004d40 2024-03-06T07:22:42.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.011+0000 7f6fd77fe640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fd814e2e0 0x7f6fd813fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:42.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.012+0000 7f6fd7fff640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 0x7f6fd81f7070 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8007340 tx=0x7f6fc8007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.012+0000 7f6fd4ff9640 1 -- 172.21.15.59:0/1347557842 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc801a070 con 0x7f6fc0004d40 2024-03-06T07:22:42.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.012+0000 7f6fd4ff9640 1 -- 172.21.15.59:0/1347557842 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc800f920 con 0x7f6fc0004d40 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.012+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1347557842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd81f7a40 con 0x7f6fc0004d40 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.012+0000 7f6fd4ff9640 1 -- 172.21.15.59:0/1347557842 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc800fd10 con 0x7f6fc0004d40 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.012+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1347557842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd80adff0 con 0x7f6fc0004d40 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.013+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1347557842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd80687d0 con 0x7f6fc0004d40 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.013+0000 7f6fd4ff9640 1 -- 172.21.15.59:0/1347557842 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fc801d070 con 0x7f6fc0004d40 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.013+0000 7f6fd4ff9640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fc406b440 0x7f6fc406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.014+0000 7f6fd4ff9640 1 -- 172.21.15.59:0/1347557842 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f6fc8089cc0 con 0x7f6fc0004d40 2024-03-06T07:22:42.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.014+0000 7f6fd77fe640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fc406b440 0x7f6fc406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.014+0000 7f6fd77fe640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fc406b440 0x7f6fc406d900 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6fd806ecf0 tx=0x7f6fcc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.018+0000 7f6fd4ff9640 1 -- 172.21.15.59:0/1347557842 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fc8058420 con 0x7f6fc0004d40 2024-03-06T07:22:42.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.202+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1347557842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6fd81f7f60 con 0x7f6fc0004d40 2024-03-06T07:22:42.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.887+0000 7f6fd4ff9640 1 -- 172.21.15.59:0/1347557842 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6fd81f7f60 con 0x7f6fc0004d40 2024-03-06T07:22:42.890 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:22:42.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fc406b440 msgr2=0x7f6fc406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fc406b440 0x7f6fc406d900 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6fd806ecf0 tx=0x7f6fcc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:42.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1347557842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 msgr2=0x7f6fd81f7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 0x7f6fd81f7070 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8007340 tx=0x7f6fc8007800 comp rx=0 tx=0).stop 2024-03-06T07:22:42.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1347557842 shutdown_connections 2024-03-06T07:22:42.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fc406b440 0x7f6fc406d900 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd813c9f0 0x7f6fd81555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fd814e2e0 0x7f6fd813fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 --2- 172.21.15.59:0/1347557842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fc0004d40 0x7f6fd81f7070 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1347557842 >> 172.21.15.59:0/1347557842 conn(0x7f6fd80b1650 msgr2=0x7f6fd806c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1347557842 shutdown_connections 2024-03-06T07:22:42.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:42.890+0000 7f6fde3d6640 1 -- 172.21.15.59:0/1347557842 wait complete. 2024-03-06T07:22:42.908 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:22:43.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.017+0000 7fa54feeb640 1 Processor -- start 2024-03-06T07:22:43.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.018+0000 7fa54feeb640 1 -- start start 2024-03-06T07:22:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.018+0000 7fa54feeb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa5480d29f0 0x7fa548168ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.018+0000 7fa54feeb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54815e380 con 0x7fa54815dfa0 2024-03-06T07:22:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.018+0000 7fa54feeb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5480cf470 con 0x7fa5480d4630 2024-03-06T07:22:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.018+0000 7fa54feeb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5480d2100 con 0x7fa5480d29f0 2024-03-06T07:22:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.019+0000 7fa54dc87640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa54815dfa0 0x7fa5480d18f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51462/0 (socket says 172.21.15.59:51462) 2024-03-06T07:22:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.019+0000 7fa54dc87640 1 -- 172.21.15.59:0/3308919437 learned_addr learned my addr 172.21.15.59:0/3308919437 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.019+0000 7fa54e488640 1 --2- 172.21.15.59:0/3308919437 >> v2:172.21.15.59:3300/0 conn(0x7fa5480d29f0 0x7fa548168ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.019+0000 7fa54e488640 1 -- 172.21.15.59:0/3308919437 >> v1:172.21.15.164:6789/0 conn(0x7fa5480d4630 legacy=0x7fa5480d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.019+0000 7fa54e488640 1 -- 172.21.15.59:0/3308919437 >> v1:172.21.15.59:6789/0 conn(0x7fa54815dfa0 legacy=0x7fa5480d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.019+0000 7fa54cc85640 1 -- 172.21.15.59:0/3308919437 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258896284 0 0) 0x7fa54815e380 con 0x7fa54815dfa0 2024-03-06T07:22:43.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.019+0000 7fa54e488640 1 -- 172.21.15.59:0/3308919437 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5480cf750 con 0x7fa5480d29f0 2024-03-06T07:22:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.020+0000 7fa54e488640 1 --2- 172.21.15.59:0/3308919437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5480d29f0 0x7fa548168ad0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa5400097a0 tx=0x7fa54002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edfbf2fe382ffdab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.020+0000 7fa54cc85640 1 -- 172.21.15.59:0/3308919437 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540035020 con 0x7fa5480d29f0 2024-03-06T07:22:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.020+0000 7fa54cc85640 1 -- 172.21.15.59:0/3308919437 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa540031830 con 0x7fa5480d29f0 2024-03-06T07:22:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.020+0000 7fa54cc85640 1 -- 172.21.15.59:0/3308919437 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540031c00 con 0x7fa5480d29f0 2024-03-06T07:22:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.020+0000 7fa54feeb640 1 -- 172.21.15.59:0/3308919437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5480d29f0 msgr2=0x7fa548168ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.020+0000 7fa54feeb640 1 --2- 172.21.15.59:0/3308919437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5480d29f0 0x7fa548168ad0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa5400097a0 tx=0x7fa54002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.020+0000 7fa54feeb640 1 -- 172.21.15.59:0/3308919437 shutdown_connections 2024-03-06T07:22:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.020+0000 7fa54feeb640 1 --2- 172.21.15.59:0/3308919437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5480d29f0 0x7fa548168ad0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.020+0000 7fa54feeb640 1 -- 172.21.15.59:0/3308919437 >> 172.21.15.59:0/3308919437 conn(0x7fa54809d000 msgr2=0x7fa5480d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.021+0000 7fa54feeb640 1 -- 172.21.15.59:0/3308919437 shutdown_connections 2024-03-06T07:22:43.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.021+0000 7fa54feeb640 1 -- 172.21.15.59:0/3308919437 wait complete. 2024-03-06T07:22:43.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.022+0000 7fa54feeb640 1 Processor -- start 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.022+0000 7fa54feeb640 1 -- start start 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.022+0000 7fa54feeb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5480d29f0 0x7fa5481f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.022+0000 7fa54feeb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5480d4630 0x7fa54815c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.022+0000 7fa54feeb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa54815dfa0 0x7fa5480c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.022+0000 7fa54feeb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5480cc320 con 0x7fa54815dfa0 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.022+0000 7fa54feeb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5480cc1a0 con 0x7fa5480d4630 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.022+0000 7fa54feeb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5480cc4a0 con 0x7fa5480d29f0 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.022+0000 7fa54e488640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa54815dfa0 0x7fa5480c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54e488640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa54815dfa0 0x7fa5480c9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58400/0 (socket says 172.21.15.59:58400) 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54e488640 1 -- 172.21.15.59:0/96587351 learned_addr learned my addr 172.21.15.59:0/96587351 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54dc87640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5480d29f0 0x7fa5481f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54e488640 1 -- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5480d29f0 msgr2=0x7fa5481f3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54d486640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5480d4630 0x7fa54815c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54e488640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5480d29f0 0x7fa5481f3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54e488640 1 -- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5480d4630 msgr2=0x7fa54815c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54e488640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5480d4630 0x7fa54815c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54e488640 1 -- 172.21.15.59:0/96587351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54815c920 con 0x7fa54815dfa0 2024-03-06T07:22:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54d486640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5480d4630 0x7fa54815c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54dc87640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5480d29f0 0x7fa5481f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54e488640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa54815dfa0 0x7fa5480c9a20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa5400098c0 tx=0x7fa5400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa5467fc640 1 -- 172.21.15.59:0/96587351 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540041070 con 0x7fa54815dfa0 2024-03-06T07:22:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa5467fc640 1 -- 172.21.15.59:0/96587351 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54003a8d0 con 0x7fa54815dfa0 2024-03-06T07:22:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54feeb640 1 -- 172.21.15.59:0/96587351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5480b17e0 con 0x7fa54815dfa0 2024-03-06T07:22:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa5467fc640 1 -- 172.21.15.59:0/96587351 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540035040 con 0x7fa54815dfa0 2024-03-06T07:22:43.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.023+0000 7fa54feeb640 1 -- 172.21.15.59:0/96587351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5480b1da0 con 0x7fa54815dfa0 2024-03-06T07:22:43.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.024+0000 7fa54feeb640 1 -- 172.21.15.59:0/96587351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5480d4ad0 con 0x7fa54815dfa0 2024-03-06T07:22:43.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.025+0000 7fa5467fc640 1 -- 172.21.15.59:0/96587351 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa540038020 con 0x7fa54815dfa0 2024-03-06T07:22:43.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.025+0000 7fa5467fc640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa53406b4d0 0x7fa53406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.025+0000 7fa5467fc640 1 -- 172.21.15.59:0/96587351 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5473+0+0 (secure 0 0 0) 0x7fa5400abb00 con 0x7fa54815dfa0 2024-03-06T07:22:43.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.025+0000 7fa54dc87640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa53406b4d0 0x7fa53406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.026+0000 7fa54dc87640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa53406b4d0 0x7fa53406d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa5380012e0 tx=0x7fa538009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.029+0000 7fa5467fc640 1 -- 172.21.15.59:0/96587351 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5480d4ad0 con 0x7fa54815dfa0 2024-03-06T07:22:43.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.210+0000 7fa54feeb640 1 -- 172.21.15.59:0/96587351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa54804f330 con 0x7fa54815dfa0 2024-03-06T07:22:43.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.893+0000 7fa5467fc640 1 -- 172.21.15.59:0/96587351 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa54007aaf0 con 0x7fa54815dfa0 2024-03-06T07:22:43.895 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-03-06T07:22:43.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.895+0000 7fa54feeb640 1 -- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa53406b4d0 msgr2=0x7fa53406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.895+0000 7fa54feeb640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa53406b4d0 0x7fa53406d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa5380012e0 tx=0x7fa538009040 comp rx=0 tx=0).stop 2024-03-06T07:22:43.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.895+0000 7fa54feeb640 1 -- 172.21.15.59:0/96587351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa54815dfa0 msgr2=0x7fa5480c9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.895+0000 7fa54feeb640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa54815dfa0 0x7fa5480c9a20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa5400098c0 tx=0x7fa5400046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.895+0000 7fa54feeb640 1 -- 172.21.15.59:0/96587351 shutdown_connections 2024-03-06T07:22:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.895+0000 7fa54feeb640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa53406b4d0 0x7fa53406d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.895+0000 7fa54feeb640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa54815dfa0 0x7fa5480c9a20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.895+0000 7fa54feeb640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5480d4630 0x7fa54815c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.895+0000 7fa54feeb640 1 --2- 172.21.15.59:0/96587351 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5480d29f0 0x7fa5481f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.895+0000 7fa54feeb640 1 -- 172.21.15.59:0/96587351 >> 172.21.15.59:0/96587351 conn(0x7fa54809d000 msgr2=0x7fa5480b88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.896+0000 7fa54feeb640 1 -- 172.21.15.59:0/96587351 shutdown_connections 2024-03-06T07:22:43.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:43.896+0000 7fa54feeb640 1 -- 172.21.15.59:0/96587351 wait complete. 2024-03-06T07:22:43.919 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:22:43.920 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:22:43.920 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:22:43.920 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:43.920 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:22:43.920 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:22:43.920 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:22:43.921 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:43.921 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:22:43.921 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:22:43.921 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:22:43.921 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:43.921 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:22:43.921 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:22:43.922 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:22:43.922 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:43.932 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:22:43.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:22:43.933 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:22:43.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:22:43.934 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:22:43.934 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:22:43.935 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:22:43.935 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:22:43.974 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:22:43.974 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:43.974 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:22:43.975 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:43.978 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:22:43.978 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:43.978 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:22:43.979 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:43.979 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:22:44.054 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:22:44.054 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:22:44.061 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:22:44.062 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:22:44.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.087+0000 7f9a71c29640 1 Processor -- start 2024-03-06T07:22:44.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.087+0000 7f9a71c29640 1 -- start start 2024-03-06T07:22:44.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.087+0000 7f9a71c29640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9a6c0b4460 0x7f9a6c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.087+0000 7f9a71c29640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9a6c162970 0x7f9a6c0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.087+0000 7f9a71c29640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6c0d22c0 con 0x7f9a6c161cf0 2024-03-06T07:22:44.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.087+0000 7f9a71c29640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c161b00 con 0x7f9a6c162970 2024-03-06T07:22:44.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.087+0000 7f9a71c29640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c0cecc0 con 0x7f9a6c0b4460 2024-03-06T07:22:44.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.088+0000 7f9a6affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9a6c162970 0x7f9a6c0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.088+0000 7f9a6bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9a6c161cf0 0x7f9a6c0cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46890/0 (socket says 172.21.15.59:46890) 2024-03-06T07:22:44.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.088+0000 7f9a6bfff640 1 -- 172.21.15.59:0/791539323 learned_addr learned my addr 172.21.15.59:0/791539323 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:44.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.088+0000 7f9a6b7fe640 1 --2- 172.21.15.59:0/791539323 >> v2:172.21.15.164:3300/0 conn(0x7f9a6c0b4460 0x7f9a6c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.088+0000 7f9a6affd640 1 -- 172.21.15.59:0/791539323 >> v2:172.21.15.164:3300/0 conn(0x7f9a6c0b4460 msgr2=0x7f9a6c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.088+0000 7f9a6affd640 1 --2- 172.21.15.59:0/791539323 >> v2:172.21.15.164:3300/0 conn(0x7f9a6c0b4460 0x7f9a6c0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.088+0000 7f9a6affd640 1 -- 172.21.15.59:0/791539323 >> v1:172.21.15.59:6789/0 conn(0x7f9a6c161cf0 legacy=0x7f9a6c0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.088+0000 7f9a6affd640 1 -- 172.21.15.59:0/791539323 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c0d25a0 con 0x7f9a6c162970 2024-03-06T07:22:44.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.088+0000 7f9a6a7fc640 1 -- 172.21.15.59:0/791539323 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332022396 0 0) 0x7f9a6c0d22c0 con 0x7f9a6c161cf0 2024-03-06T07:22:44.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.088+0000 7f9a6b7fe640 1 --2- 172.21.15.59:0/791539323 >> v2:172.21.15.164:3300/0 conn(0x7f9a6c0b4460 0x7f9a6c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:44.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a6affd640 1 --2- 172.21.15.59:0/791539323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a6c162970 0x7f9a6c0b3580 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c08ddd0 tx=0x7f9a6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=279c8178898217c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a6a7fc640 1 -- 172.21.15.59:0/791539323 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60035020 con 0x7f9a6c162970 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a6a7fc640 1 -- 172.21.15.59:0/791539323 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a60031830 con 0x7f9a6c162970 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a6a7fc640 1 -- 172.21.15.59:0/791539323 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60031c00 con 0x7f9a6c162970 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a71c29640 1 -- 172.21.15.59:0/791539323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a6c162970 msgr2=0x7f9a6c0b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a71c29640 1 --2- 172.21.15.59:0/791539323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a6c162970 0x7f9a6c0b3580 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c08ddd0 tx=0x7f9a6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a71c29640 1 -- 172.21.15.59:0/791539323 shutdown_connections 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a71c29640 1 --2- 172.21.15.59:0/791539323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a6c162970 0x7f9a6c0b3580 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a71c29640 1 --2- 172.21.15.59:0/791539323 >> v2:172.21.15.164:3300/0 conn(0x7f9a6c0b4460 0x7f9a6c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a71c29640 1 -- 172.21.15.59:0/791539323 >> 172.21.15.59:0/791539323 conn(0x7f9a6c0d3ce0 msgr2=0x7f9a6c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a71c29640 1 -- 172.21.15.59:0/791539323 shutdown_connections 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.089+0000 7f9a71c29640 1 -- 172.21.15.59:0/791539323 wait complete. 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.091+0000 7f9a71c29640 1 Processor -- start 2024-03-06T07:22:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.091+0000 7f9a71c29640 1 -- start start 2024-03-06T07:22:44.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.091+0000 7f9a71c29640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c0b4460 0x7f9a6c156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.091+0000 7f9a71c29640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c161cf0 0x7f9a6c156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.091+0000 7f9a71c29640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a6c162970 0x7f9a6c0b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.091+0000 7f9a71c29640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c0ce950 con 0x7f9a6c162970 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.091+0000 7f9a71c29640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c0d2b60 con 0x7f9a6c161cf0 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.091+0000 7f9a71c29640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c0b3ac0 con 0x7f9a6c0b4460 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a6c162970 0x7f9a6c0b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a6c162970 0x7f9a6c0b1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35984/0 (socket says 172.21.15.59:35984) 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6bfff640 1 -- 172.21.15.59:0/923091891 learned_addr learned my addr 172.21.15.59:0/923091891 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6affd640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c161cf0 0x7f9a6c156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6b7fe640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c0b4460 0x7f9a6c156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6bfff640 1 -- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c0b4460 msgr2=0x7f9a6c156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6bfff640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c0b4460 0x7f9a6c156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6bfff640 1 -- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c161cf0 msgr2=0x7f9a6c156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6bfff640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c161cf0 0x7f9a6c156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6bfff640 1 -- 172.21.15.59:0/923091891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c0b6aa0 con 0x7f9a6c162970 2024-03-06T07:22:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6affd640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c161cf0 0x7f9a6c156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a6bfff640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a6c162970 0x7f9a6c0b1ec0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400c7a0 tx=0x7f9a6400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a53fff640 1 -- 172.21.15.59:0/923091891 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64018070 con 0x7f9a6c162970 2024-03-06T07:22:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a53fff640 1 -- 172.21.15.59:0/923091891 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6400a640 con 0x7f9a6c162970 2024-03-06T07:22:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a71c29640 1 -- 172.21.15.59:0/923091891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6c0b6d80 con 0x7f9a6c162970 2024-03-06T07:22:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a53fff640 1 -- 172.21.15.59:0/923091891 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6400aab0 con 0x7f9a6c162970 2024-03-06T07:22:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.092+0000 7f9a71c29640 1 -- 172.21.15.59:0/923091891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6c1f7f50 con 0x7f9a6c162970 2024-03-06T07:22:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.093+0000 7f9a71c29640 1 -- 172.21.15.59:0/923091891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6c04f330 con 0x7f9a6c162970 2024-03-06T07:22:44.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.094+0000 7f9a53fff640 1 -- 172.21.15.59:0/923091891 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a6401b070 con 0x7f9a6c162970 2024-03-06T07:22:44.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.094+0000 7f9a53fff640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5806b4d0 0x7f9a5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.094+0000 7f9a53fff640 1 -- 172.21.15.59:0/923091891 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9a64089540 con 0x7f9a6c162970 2024-03-06T07:22:44.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.094+0000 7f9a6b7fe640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5806b4d0 0x7f9a5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.095+0000 7f9a6b7fe640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5806b4d0 0x7f9a5806d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c006fd0 tx=0x7f9a5c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.099+0000 7f9a53fff640 1 -- 172.21.15.59:0/923091891 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a640586c0 con 0x7f9a6c162970 2024-03-06T07:22:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.295+0000 7f9a71c29640 1 -- 172.21.15.59:0/923091891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f9a6c06b180 con 0x7f9a6c162970 2024-03-06T07:22:44.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.902+0000 7f9a53fff640 1 -- 172.21.15.59:0/923091891 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9a64057fc0 con 0x7f9a6c162970 2024-03-06T07:22:44.904 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-03-06T07:22:44.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.905+0000 7f9a71c29640 1 -- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5806b4d0 msgr2=0x7f9a5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.905+0000 7f9a71c29640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5806b4d0 0x7f9a5806d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c006fd0 tx=0x7f9a5c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.905+0000 7f9a71c29640 1 -- 172.21.15.59:0/923091891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a6c162970 msgr2=0x7f9a6c0b1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.905+0000 7f9a71c29640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a6c162970 0x7f9a6c0b1ec0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400c7a0 tx=0x7f9a6400cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.906+0000 7f9a71c29640 1 -- 172.21.15.59:0/923091891 shutdown_connections 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.906+0000 7f9a71c29640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5806b4d0 0x7f9a5806d990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.906+0000 7f9a71c29640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a6c162970 0x7f9a6c0b1ec0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.906+0000 7f9a71c29640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c161cf0 0x7f9a6c156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.906+0000 7f9a71c29640 1 --2- 172.21.15.59:0/923091891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c0b4460 0x7f9a6c156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.906+0000 7f9a71c29640 1 -- 172.21.15.59:0/923091891 >> 172.21.15.59:0/923091891 conn(0x7f9a6c0d3ce0 msgr2=0x7f9a6c0d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.906+0000 7f9a71c29640 1 -- 172.21.15.59:0/923091891 shutdown_connections 2024-03-06T07:22:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:44.906+0000 7f9a71c29640 1 -- 172.21.15.59:0/923091891 wait complete. 2024-03-06T07:22:44.922 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:22:45.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.029+0000 7fd68e4fb640 1 Processor -- start 2024-03-06T07:22:45.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.029+0000 7fd68e4fb640 1 -- start start 2024-03-06T07:22:45.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd68e4fb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd6880d0660 0x7fd6880cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd68e4fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880cf850 0x7fd6880b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd68e4fb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68815e4d0 con 0x7fd68815e0f0 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd68e4fb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6880b1410 con 0x7fd6880d0660 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd68e4fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68806c8b0 con 0x7fd6880cf850 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd68ca98640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880cf850 0x7fd6880b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd6877fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd6880d0660 0x7fd6880cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd68ca98640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880cf850 0x7fd6880b25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:59874/0 (socket says 172.21.15.59:59874) 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd68ca98640 1 -- 172.21.15.59:0/3505454888 learned_addr learned my addr 172.21.15.59:0/3505454888 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd687fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd68815e0f0 0x7fd6880cff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:37890/0 (socket says 172.21.15.59:37890) 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd686ffd640 1 -- 172.21.15.59:0/3505454888 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795782073 0 0) 0x7fd68815e4d0 con 0x7fd68815e0f0 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd686ffd640 1 -- 172.21.15.59:0/3505454888 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd674003520 con 0x7fd68815e0f0 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd6877fe640 1 -- 172.21.15.59:0/3505454888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880cf850 msgr2=0x7fd6880b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd6877fe640 1 --2- 172.21.15.59:0/3505454888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880cf850 0x7fd6880b25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd6877fe640 1 -- 172.21.15.59:0/3505454888 >> v1:172.21.15.164:6789/0 conn(0x7fd68815e0f0 legacy=0x7fd6880cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd6877fe640 1 -- 172.21.15.59:0/3505454888 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6880cdac0 con 0x7fd6880d0660 2024-03-06T07:22:45.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.030+0000 7fd68ca98640 1 --2- 172.21.15.59:0/3505454888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880cf850 0x7fd6880b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.031+0000 7fd6877fe640 1 --2- 172.21.15.59:0/3505454888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6880d0660 0x7fd6880cd3c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd6780097a0 tx=0x7fd67802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=846c6ac5c1fc40f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.031+0000 7fd686ffd640 1 -- 172.21.15.59:0/3505454888 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678035020 con 0x7fd6880d0660 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.031+0000 7fd686ffd640 1 -- 172.21.15.59:0/3505454888 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd678031830 con 0x7fd6880d0660 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.031+0000 7fd686ffd640 1 -- 172.21.15.59:0/3505454888 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678031c00 con 0x7fd6880d0660 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.032+0000 7fd68e4fb640 1 -- 172.21.15.59:0/3505454888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6880d0660 msgr2=0x7fd6880cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.032+0000 7fd68e4fb640 1 --2- 172.21.15.59:0/3505454888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6880d0660 0x7fd6880cd3c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd6780097a0 tx=0x7fd67802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.032+0000 7fd68e4fb640 1 -- 172.21.15.59:0/3505454888 shutdown_connections 2024-03-06T07:22:45.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.032+0000 7fd68e4fb640 1 --2- 172.21.15.59:0/3505454888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880cf850 0x7fd6880b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.032+0000 7fd68e4fb640 1 --2- 172.21.15.59:0/3505454888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6880d0660 0x7fd6880cd3c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.032+0000 7fd68e4fb640 1 -- 172.21.15.59:0/3505454888 >> 172.21.15.59:0/3505454888 conn(0x7fd6880b35a0 msgr2=0x7fd6880b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.032+0000 7fd68e4fb640 1 -- 172.21.15.59:0/3505454888 shutdown_connections 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.032+0000 7fd68e4fb640 1 -- 172.21.15.59:0/3505454888 wait complete. 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.033+0000 7fd68e4fb640 1 Processor -- start 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.033+0000 7fd68e4fb640 1 -- start start 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd68e4fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6880cf850 0x7fd688161240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd68e4fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880d0660 0x7fd6881578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd68e4fb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd68815e0f0 0x7fd6881f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd68e4fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6880d0cc0 con 0x7fd6880cf850 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd68e4fb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68806c540 con 0x7fd68815e0f0 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd68e4fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6880b2af0 con 0x7fd6880d0660 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd687fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6880cf850 0x7fd688161240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd687fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6880cf850 0x7fd688161240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35986/0 (socket says 172.21.15.59:35986) 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd687fff640 1 -- 172.21.15.59:0/2788763184 learned_addr learned my addr 172.21.15.59:0/2788763184 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd6877fe640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880d0660 0x7fd6881578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd68ca98640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd68815e0f0 0x7fd6881f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd687fff640 1 -- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880d0660 msgr2=0x7fd6881578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd687fff640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880d0660 0x7fd6881578f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd687fff640 1 -- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd68815e0f0 msgr2=0x7fd6881f7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd687fff640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd68815e0f0 0x7fd6881f7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.034+0000 7fd687fff640 1 -- 172.21.15.59:0/2788763184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688157e30 con 0x7fd6880cf850 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.035+0000 7fd687fff640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6880cf850 0x7fd688161240 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd67000c390 tx=0x7fd67000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.035+0000 7fd684ff9640 1 -- 172.21.15.59:0/2788763184 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6700074e0 con 0x7fd6880cf850 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.035+0000 7fd684ff9640 1 -- 172.21.15.59:0/2788763184 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6700047e0 con 0x7fd6880cf850 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.035+0000 7fd68e4fb640 1 -- 172.21.15.59:0/2788763184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6880ca650 con 0x7fd6880cf850 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.035+0000 7fd684ff9640 1 -- 172.21.15.59:0/2788763184 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670004bd0 con 0x7fd6880cf850 2024-03-06T07:22:45.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.035+0000 7fd68e4fb640 1 -- 172.21.15.59:0/2788763184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6880cac10 con 0x7fd6880cf850 2024-03-06T07:22:45.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.036+0000 7fd68e4fb640 1 -- 172.21.15.59:0/2788763184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68804f330 con 0x7fd6880cf850 2024-03-06T07:22:45.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.036+0000 7fd684ff9640 1 -- 172.21.15.59:0/2788763184 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd67001a020 con 0x7fd6880cf850 2024-03-06T07:22:45.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.037+0000 7fd684ff9640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b500 0x7fd66806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.037+0000 7fd684ff9640 1 -- 172.21.15.59:0/2788763184 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd6700894b0 con 0x7fd6880cf850 2024-03-06T07:22:45.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.037+0000 7fd6877fe640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b500 0x7fd66806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.037+0000 7fd6877fe640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b500 0x7fd66806d9c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd678036000 tx=0x7fd678008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.041+0000 7fd684ff9640 1 -- 172.21.15.59:0/2788763184 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd670058100 con 0x7fd6880cf850 2024-03-06T07:22:45.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.291+0000 7fd68e4fb640 1 -- 172.21.15.59:0/2788763184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd6880d28a0 con 0x7fd6880cf850 2024-03-06T07:22:45.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.292+0000 7fd684ff9640 1 -- 172.21.15.59:0/2788763184 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fd67005c1a0 con 0x7fd6880cf850 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlODBZGBAAqV+TH2t2ZKyV9l13QEn6ew==", 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.294 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlR1MuJRAAp6OAQ9gZemWtwI1dh3H2SA==", 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlDXW2HxAAnKyuS1GW0wLO98/F9wdOAg==", 2024-03-06T07:22:45.295 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehl5MO4LBAAKmt2dvKk7BtcCM4o7cSwgg==", 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.296 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBXGehlzdinFhAA0ACDAs9dtoY6EhUIveeIvA==", 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGehlH92eFRAANQceHGuQyrbI6gP1m/Gakw==", 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGehlMWpVFBAAbOG9nScbFlvKHXZMQJoQug==", 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.298 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGehlLCyWEhAAyfoW/FmU37QD2IohknBwWw==", 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBmGehldqv5GhAAWEloqqKgdpKPHjt7VKFodw==", 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.299 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGehlc0ENGRAA5V9Az+vxAR3OUwWpDP7w1A==", 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBsGehl5SYEFxAAPLyrHQV9SGguCV9f5VIkYg==", 2024-03-06T07:22:45.300 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBvGehle3v0FBAA4xUPwX5couppHgi9VwSi2g==", 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.301 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA==", 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlqhlAOhAAQ4l3uI8MlcCH+NAIWo/UuA==", 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.302 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehltL/GNhAAZqSQ+UtwjI/PA95O1RGBNg==", 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:22:45.303 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGehlltx5BBAAFa/+/w8Hd8lZegfIWtpJOA==", 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-03-06T07:22:45.304 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBSGehlMExaNhAAQ2FD7afktMhrTbMMAK1epQ==", 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQB6GehlsDLVEBAAgumD7i/bPYU1i2+15Bf6+g==", 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.305 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQB6GehlyTrVEBAArjpQkr6aX9/vp7E1yfzQoA==", 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQB6GehlCkLVEBAAhRwIoVhKwZ9NhEbQ+Aw5UA==", 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQB6GehlVEnVEBAAGEv2gl+nGq/RifCwe6NEBg==", 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQB6GehlPlDVEBAADmQIwzuSaIVI7qEjUjcwjA==", 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:22:45.307 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQB6GehlalfVEBAAvWXpAHK9976KCZlRQ/3GBw==", 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlLSTzFBAACeLdHsCX9KJGEsq44aqTyw==", 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:45.308 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-03-06T07:22:45.309 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlAgqXEBAACCHZ9Lp2lEyF93xD4Ns9Nw==", 2024-03-06T07:22:45.309 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:22:45.309 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:22:45.309 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:45.309 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:22:45.309 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.309 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:45.309 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-06T07:22:45.309 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.293+0000 7fd68e4fb640 1 -- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b500 msgr2=0x7fd66806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.293+0000 7fd68e4fb640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b500 0x7fd66806d9c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd678036000 tx=0x7fd678008030 comp rx=0 tx=0).stop 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.293+0000 7fd68e4fb640 1 -- 172.21.15.59:0/2788763184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6880cf850 msgr2=0x7fd688161240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.293+0000 7fd68e4fb640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6880cf850 0x7fd688161240 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd67000c390 tx=0x7fd67000c850 comp rx=0 tx=0).stop 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.294+0000 7fd68e4fb640 1 -- 172.21.15.59:0/2788763184 shutdown_connections 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.294+0000 7fd68e4fb640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b500 0x7fd66806d9c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.294+0000 7fd68e4fb640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd68815e0f0 0x7fd6881f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.294+0000 7fd68e4fb640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880d0660 0x7fd6881578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.294+0000 7fd68e4fb640 1 --2- 172.21.15.59:0/2788763184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6880cf850 0x7fd688161240 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.294+0000 7fd68e4fb640 1 -- 172.21.15.59:0/2788763184 >> 172.21.15.59:0/2788763184 conn(0x7fd6880b35a0 msgr2=0x7fd6880b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.294+0000 7fd68e4fb640 1 -- 172.21.15.59:0/2788763184 shutdown_connections 2024-03-06T07:22:45.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.294+0000 7fd68e4fb640 1 -- 172.21.15.59:0/2788763184 wait complete. 2024-03-06T07:22:45.312 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:22:45.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.412+0000 7f099bc8d640 1 Processor -- start 2024-03-06T07:22:45.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.412+0000 7f099bc8d640 1 -- start start 2024-03-06T07:22:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f099bc8d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f099415e010 0x7f09940cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f099bc8d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09940d1ad0 0x7f09940d1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f099bc8d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099415e3f0 con 0x7f09940b3240 2024-03-06T07:22:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f099bc8d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09940b1f40 con 0x7f099415e010 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f099bc8d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09940b1c10 con 0x7f09940d1ad0 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f0999228640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f09940b3240 0x7f09940b23d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46898/0 (socket says 172.21.15.59:46898) 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f0999228640 1 -- 172.21.15.59:0/3911284265 learned_addr learned my addr 172.21.15.59:0/3911284265 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f0999a29640 1 --2- 172.21.15.59:0/3911284265 >> v2:172.21.15.164:3300/0 conn(0x7f099415e010 0x7f09940cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f099a22a640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09940d1ad0 0x7f09940d1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990293903 0 0) 0x7f099415e3f0 con 0x7f09940b3240 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0980003520 con 0x7f09940b3240 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183569228 0 0) 0x7f0980003520 con 0x7f09940b3240 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09940d1ad0 msgr2=0x7f09940d1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f0998a27640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09940d1ad0 0x7f09940d1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.413+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 >> v2:172.21.15.164:3300/0 conn(0x7f099415e010 msgr2=0x7f09940cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0998a27640 1 --2- 172.21.15.59:0/3911284265 >> v2:172.21.15.164:3300/0 conn(0x7f099415e010 0x7f09940cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099406d1b0 con 0x7f09940b3240 2024-03-06T07:22:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f0984001670 con 0x7f09940b3240 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 >> v1:172.21.15.59:6789/0 conn(0x7f09940b3240 legacy=0x7f09940b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0998a27640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 0x7f0980003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0998a27640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09940d1ad0 0x7f0980004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0998a27640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0980004d40 0x7f09800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09940b1f40 con 0x7f09800039b0 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09940b1c10 con 0x7f0980004d40 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098000b9c0 con 0x7f09940d1ad0 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0999a29640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 0x7f0980003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0999228640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09940d1ad0 0x7f0980004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f099a22a640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0980004d40 0x7f09800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0999a29640 1 -- 172.21.15.59:0/3911284265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09940d1ad0 msgr2=0x7f0980004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0999a29640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09940d1ad0 0x7f0980004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0999a29640 1 -- 172.21.15.59:0/3911284265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0980004d40 msgr2=0x7f09800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0999a29640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0980004d40 0x7f09800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.414+0000 7f0999a29640 1 -- 172.21.15.59:0/3911284265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0980003520 con 0x7f09800039b0 2024-03-06T07:22:45.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.415+0000 7f0999a29640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 0x7f0980003dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f098c009ac0 tx=0x7f0980005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e915096ea3ec06a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.415+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c039070 con 0x7f09800039b0 2024-03-06T07:22:45.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.415+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098c02c710 con 0x7f09800039b0 2024-03-06T07:22:45.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.415+0000 7f0998a27640 1 -- 172.21.15.59:0/3911284265 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c009690 con 0x7f09800039b0 2024-03-06T07:22:45.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.415+0000 7f099bc8d640 1 -- 172.21.15.59:0/3911284265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 msgr2=0x7f0980003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.415+0000 7f099bc8d640 1 --2- 172.21.15.59:0/3911284265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 0x7f0980003dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f098c009ac0 tx=0x7f0980005830 comp rx=0 tx=0).stop 2024-03-06T07:22:45.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.415+0000 7f099a22a640 1 -- 172.21.15.59:0/3911284265 reap_dead start 2024-03-06T07:22:45.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.415+0000 7f099bc8d640 1 -- 172.21.15.59:0/3911284265 shutdown_connections 2024-03-06T07:22:45.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.415+0000 7f099bc8d640 1 -- 172.21.15.59:0/3911284265 >> 172.21.15.59:0/3911284265 conn(0x7f099409a9f0 msgr2=0x7f099409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.416+0000 7f099bc8d640 1 -- 172.21.15.59:0/3911284265 shutdown_connections 2024-03-06T07:22:45.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.416+0000 7f099bc8d640 1 -- 172.21.15.59:0/3911284265 wait complete. 2024-03-06T07:22:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.417+0000 7f099bc8d640 1 Processor -- start 2024-03-06T07:22:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.417+0000 7f099bc8d640 1 -- start start 2024-03-06T07:22:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.417+0000 7f099bc8d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 0x7f099415bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.417+0000 7f099bc8d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f099409b9a0 0x7f099409dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.417+0000 7f099bc8d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09940c8140 0x7f099409e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.417+0000 7f099bc8d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099406ced0 con 0x7f09800039b0 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.417+0000 7f099bc8d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099406cbf0 con 0x7f09940c8140 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.417+0000 7f099bc8d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09940cf310 con 0x7f099409b9a0 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.417+0000 7f0999a29640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 0x7f099415bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.417+0000 7f0999a29640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 0x7f099415bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36006/0 (socket says 172.21.15.59:36006) 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f0999a29640 1 -- 172.21.15.59:0/4118033739 learned_addr learned my addr 172.21.15.59:0/4118033739 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f0999228640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f099409b9a0 0x7f099409dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f099a22a640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09940c8140 0x7f099409e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f0999a29640 1 -- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f099409b9a0 msgr2=0x7f099409dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f0999a29640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f099409b9a0 0x7f099409dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f0999a29640 1 -- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09940c8140 msgr2=0x7f099409e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f0999a29640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09940c8140 0x7f099409e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f0999a29640 1 -- 172.21.15.59:0/4118033739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09940c6080 con 0x7f09800039b0 2024-03-06T07:22:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f0999a29640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 0x7f099415bf60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f098c009be0 tx=0x7f098c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f098a7fc640 1 -- 172.21.15.59:0/4118033739 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c042020 con 0x7f09800039b0 2024-03-06T07:22:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f098a7fc640 1 -- 172.21.15.59:0/4118033739 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098c03a040 con 0x7f09800039b0 2024-03-06T07:22:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f099bc8d640 1 -- 172.21.15.59:0/4118033739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09940c6360 con 0x7f09800039b0 2024-03-06T07:22:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f098a7fc640 1 -- 172.21.15.59:0/4118033739 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c039070 con 0x7f09800039b0 2024-03-06T07:22:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.418+0000 7f099bc8d640 1 -- 172.21.15.59:0/4118033739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09940c6920 con 0x7f09800039b0 2024-03-06T07:22:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.419+0000 7f099bc8d640 1 -- 172.21.15.59:0/4118033739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f099404f330 con 0x7f09800039b0 2024-03-06T07:22:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.420+0000 7f098a7fc640 1 -- 172.21.15.59:0/4118033739 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f098c045020 con 0x7f09800039b0 2024-03-06T07:22:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.420+0000 7f098a7fc640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f097806b4d0 0x7f097806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.420+0000 7f098a7fc640 1 -- 172.21.15.59:0/4118033739 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f098c0ab230 con 0x7f09800039b0 2024-03-06T07:22:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.420+0000 7f0999228640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f097806b4d0 0x7f097806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.421+0000 7f0999228640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f097806b4d0 0x7f097806d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0984000c00 tx=0x7f0984007970 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.424+0000 7f098a7fc640 1 -- 172.21.15.59:0/4118033739 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f098c04c070 con 0x7f09800039b0 2024-03-06T07:22:45.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.674+0000 7f099bc8d640 1 -- 172.21.15.59:0/4118033739 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f09940d2980 con 0x7f09800039b0 2024-03-06T07:22:45.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.676+0000 7f098a7fc640 1 -- 172.21.15.59:0/4118033739 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f098c079d20 con 0x7f09800039b0 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 -- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f097806b4d0 msgr2=0x7f097806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f097806b4d0 0x7f097806d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0984000c00 tx=0x7f0984007970 comp rx=0 tx=0).stop 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 -- 172.21.15.59:0/4118033739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 msgr2=0x7f099415bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 0x7f099415bf60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f098c009be0 tx=0x7f098c004730 comp rx=0 tx=0).stop 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 -- 172.21.15.59:0/4118033739 shutdown_connections 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f097806b4d0 0x7f097806d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09940c8140 0x7f099409e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f099409b9a0 0x7f099409dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 --2- 172.21.15.59:0/4118033739 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09800039b0 0x7f099415bf60 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 -- 172.21.15.59:0/4118033739 >> 172.21.15.59:0/4118033739 conn(0x7f099409a9f0 msgr2=0x7f09940d1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 -- 172.21.15.59:0/4118033739 shutdown_connections 2024-03-06T07:22:45.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.678+0000 7f099bc8d640 1 -- 172.21.15.59:0/4118033739 wait complete. 2024-03-06T07:22:45.691 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:22:45.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.791+0000 7f7f2e6c9640 1 Processor -- start 2024-03-06T07:22:45.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.791+0000 7f7f2e6c9640 1 -- start start 2024-03-06T07:22:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.792+0000 7f7f2e6c9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f2815e000 0x7f7f280d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.792+0000 7f7f2e6c9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7f280d4810 0x7f7f280d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.792+0000 7f7f2e6c9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f280d25e0 con 0x7f7f280c80e0 2024-03-06T07:22:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.792+0000 7f7f2e6c9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f280c7c50 con 0x7f7f280d4810 2024-03-06T07:22:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.792+0000 7f7f2e6c9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f280d2880 con 0x7f7f2815e000 2024-03-06T07:22:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.792+0000 7f7f277fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7f280d4810 0x7f7f280d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.792+0000 7f7f277fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7f280d4810 0x7f7f280d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36022/0 (socket says 172.21.15.59:36022) 2024-03-06T07:22:45.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.792+0000 7f7f277fe640 1 -- 172.21.15.59:0/4243536064 learned_addr learned my addr 172.21.15.59:0/4243536064 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.792+0000 7f7f27fff640 1 --2- 172.21.15.59:0/4243536064 >> v2:172.21.15.164:3300/0 conn(0x7f7f2815e000 0x7f7f280d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.793+0000 7f7f277fe640 1 -- 172.21.15.59:0/4243536064 >> v2:172.21.15.164:3300/0 conn(0x7f7f2815e000 msgr2=0x7f7f280d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.793+0000 7f7f277fe640 1 --2- 172.21.15.59:0/4243536064 >> v2:172.21.15.164:3300/0 conn(0x7f7f2815e000 0x7f7f280d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.793+0000 7f7f277fe640 1 -- 172.21.15.59:0/4243536064 >> v1:172.21.15.164:6789/0 conn(0x7f7f280c80e0 legacy=0x7f7f280c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.793+0000 7f7f277fe640 1 -- 172.21.15.59:0/4243536064 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f280d2b40 con 0x7f7f280d4810 2024-03-06T07:22:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.793+0000 7f7f27fff640 1 --2- 172.21.15.59:0/4243536064 >> v2:172.21.15.164:3300/0 conn(0x7f7f2815e000 0x7f7f280d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.793+0000 7f7f277fe640 1 --2- 172.21.15.59:0/4243536064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f280d4810 0x7f7f280d0bc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7f2808c360 tx=0x7f7f1c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13e5dd4260e64df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.794+0000 7f7f26ffd640 1 -- 172.21.15.59:0/4243536064 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c02f850 con 0x7f7f280d4810 2024-03-06T07:22:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.794+0000 7f7f26ffd640 1 -- 172.21.15.59:0/4243536064 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1c02f9f0 con 0x7f7f280d4810 2024-03-06T07:22:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.794+0000 7f7f26ffd640 1 -- 172.21.15.59:0/4243536064 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c02fdc0 con 0x7f7f280d4810 2024-03-06T07:22:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.794+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/4243536064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f280d4810 msgr2=0x7f7f280d0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.794+0000 7f7f2e6c9640 1 --2- 172.21.15.59:0/4243536064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f280d4810 0x7f7f280d0bc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7f2808c360 tx=0x7f7f1c02ba30 comp rx=0 tx=0).stop 2024-03-06T07:22:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.794+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/4243536064 shutdown_connections 2024-03-06T07:22:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.794+0000 7f7f2e6c9640 1 --2- 172.21.15.59:0/4243536064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f280d4810 0x7f7f280d0bc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.794+0000 7f7f2e6c9640 1 --2- 172.21.15.59:0/4243536064 >> v2:172.21.15.164:3300/0 conn(0x7f7f2815e000 0x7f7f280d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.794+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/4243536064 >> 172.21.15.59:0/4243536064 conn(0x7f7f280d3cf0 msgr2=0x7f7f280d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.794+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/4243536064 shutdown_connections 2024-03-06T07:22:45.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.794+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/4243536064 wait complete. 2024-03-06T07:22:45.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.795+0000 7f7f2e6c9640 1 Processor -- start 2024-03-06T07:22:45.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.795+0000 7f7f2e6c9640 1 -- start start 2024-03-06T07:22:45.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.796+0000 7f7f2e6c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f280c80e0 0x7f7f280b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.796+0000 7f7f2e6c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f280d4810 0x7f7f280b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.796+0000 7f7f2e6c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2815e000 0x7f7f280b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.796+0000 7f7f2e6c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2815e3e0 con 0x7f7f280c80e0 2024-03-06T07:22:45.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.796+0000 7f7f2e6c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f280d22d0 con 0x7f7f280d4810 2024-03-06T07:22:45.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.796+0000 7f7f2e6c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f280cf7d0 con 0x7f7f2815e000 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.796+0000 7f7f27fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f280c80e0 0x7f7f280b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.796+0000 7f7f27fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f280c80e0 0x7f7f280b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36024/0 (socket says 172.21.15.59:36024) 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.796+0000 7f7f27fff640 1 -- 172.21.15.59:0/1755995836 learned_addr learned my addr 172.21.15.59:0/1755995836 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f277fe640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f280d4810 0x7f7f280b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f27fff640 1 -- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2815e000 msgr2=0x7f7f280b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f2cc66640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2815e000 0x7f7f280b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f27fff640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2815e000 0x7f7f280b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f27fff640 1 -- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f280d4810 msgr2=0x7f7f280b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f27fff640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f280d4810 0x7f7f280b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f27fff640 1 -- 172.21.15.59:0/1755995836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f280b1300 con 0x7f7f280c80e0 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f277fe640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f280d4810 0x7f7f280b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f2cc66640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2815e000 0x7f7f280b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f27fff640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f280c80e0 0x7f7f280b2110 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800d7d0 tx=0x7f7f1800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f24ff9640 1 -- 172.21.15.59:0/1755995836 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f18004030 con 0x7f7f280c80e0 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f24ff9640 1 -- 172.21.15.59:0/1755995836 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f180041d0 con 0x7f7f280c80e0 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/1755995836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f280b2720 con 0x7f7f280c80e0 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f24ff9640 1 -- 172.21.15.59:0/1755995836 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1800b6c0 con 0x7f7f280c80e0 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.797+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/1755995836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f280b2ce0 con 0x7f7f280c80e0 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.798+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/1755995836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2804f330 con 0x7f7f280c80e0 2024-03-06T07:22:45.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.798+0000 7f7f24ff9640 1 -- 172.21.15.59:0/1755995836 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f1800b860 con 0x7f7f280c80e0 2024-03-06T07:22:45.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.799+0000 7f7f24ff9640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f1406b4d0 0x7f7f1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.799+0000 7f7f24ff9640 1 -- 172.21.15.59:0/1755995836 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7f1808a0e0 con 0x7f7f280c80e0 2024-03-06T07:22:45.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.799+0000 7f7f277fe640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f1406b4d0 0x7f7f1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.800+0000 7f7f277fe640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f1406b4d0 0x7f7f1406d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c036000 tx=0x7f7f1c039040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:45.803+0000 7f7f24ff9640 1 -- 172.21.15.59:0/1755995836 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f18059340 con 0x7f7f280c80e0 2024-03-06T07:22:46.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.058+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/1755995836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f7f280b14d0 con 0x7f7f280c80e0 2024-03-06T07:22:46.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.062+0000 7f7f24ff9640 1 -- 172.21.15.59:0/1755995836 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7f18058c40 con 0x7f7f280c80e0 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.064+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f1406b4d0 msgr2=0x7f7f1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.064+0000 7f7f2e6c9640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f1406b4d0 0x7f7f1406d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c036000 tx=0x7f7f1c039040 comp rx=0 tx=0).stop 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.064+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/1755995836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f280c80e0 msgr2=0x7f7f280b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.064+0000 7f7f2e6c9640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f280c80e0 0x7f7f280b2110 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800d7d0 tx=0x7f7f1800dc90 comp rx=0 tx=0).stop 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.064+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/1755995836 shutdown_connections 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.064+0000 7f7f2e6c9640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f1406b4d0 0x7f7f1406d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.064+0000 7f7f2e6c9640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2815e000 0x7f7f280b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.064+0000 7f7f2e6c9640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f280d4810 0x7f7f280b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.064+0000 7f7f2e6c9640 1 --2- 172.21.15.59:0/1755995836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f280c80e0 0x7f7f280b2110 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.065+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/1755995836 >> 172.21.15.59:0/1755995836 conn(0x7f7f280d3cf0 msgr2=0x7f7f280b8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.065+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/1755995836 shutdown_connections 2024-03-06T07:22:46.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.065+0000 7f7f2e6c9640 1 -- 172.21.15.59:0/1755995836 wait complete. 2024-03-06T07:22:46.077 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:22:46.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.177+0000 7fac0f064640 1 Processor -- start 2024-03-06T07:22:46.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.177+0000 7fac0f064640 1 -- start start 2024-03-06T07:22:46.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.177+0000 7fac0f064640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fac0815e010 0x7fac080cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.177+0000 7fac0f064640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fac080b3240 0x7fac080b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.177+0000 7fac0f064640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0815e3f0 con 0x7fac080b2910 2024-03-06T07:22:46.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.177+0000 7fac0f064640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac080b1f40 con 0x7fac0815e010 2024-03-06T07:22:46.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.177+0000 7fac0f064640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac080b1c10 con 0x7fac080b3240 2024-03-06T07:22:46.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.178+0000 7fac0ce00640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fac0815e010 0x7fac080cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.178+0000 7fac0ce00640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fac0815e010 0x7fac080cedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36036/0 (socket says 172.21.15.59:36036) 2024-03-06T07:22:46.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.178+0000 7fac0ce00640 1 -- 172.21.15.59:0/2534141608 learned_addr learned my addr 172.21.15.59:0/2534141608 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:46.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.178+0000 7fac07fff640 1 --2- 172.21.15.59:0/2534141608 >> v2:172.21.15.164:3300/0 conn(0x7fac080b3240 0x7fac080b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.178+0000 7fac0ce00640 1 -- 172.21.15.59:0/2534141608 >> v2:172.21.15.164:3300/0 conn(0x7fac080b3240 msgr2=0x7fac080b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.178+0000 7fac0ce00640 1 --2- 172.21.15.59:0/2534141608 >> v2:172.21.15.164:3300/0 conn(0x7fac080b3240 0x7fac080b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.178+0000 7fac0ce00640 1 -- 172.21.15.59:0/2534141608 >> v1:172.21.15.164:6789/0 conn(0x7fac080b2910 legacy=0x7fac080d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.178+0000 7fac0ce00640 1 -- 172.21.15.59:0/2534141608 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0806d1b0 con 0x7fac0815e010 2024-03-06T07:22:46.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.178+0000 7fac07fff640 1 --2- 172.21.15.59:0/2534141608 >> v2:172.21.15.164:3300/0 conn(0x7fac080b3240 0x7fac080b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:46.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.179+0000 7fac0ce00640 1 --2- 172.21.15.59:0/2534141608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac0815e010 0x7fac080cedd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fac0806e2a0 tx=0x7fabf802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2310908f9cff1c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.179+0000 7fac077fe640 1 -- 172.21.15.59:0/2534141608 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8034070 con 0x7fac0815e010 2024-03-06T07:22:46.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.179+0000 7fac077fe640 1 -- 172.21.15.59:0/2534141608 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf8030860 con 0x7fac0815e010 2024-03-06T07:22:46.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.179+0000 7fac077fe640 1 -- 172.21.15.59:0/2534141608 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8030c30 con 0x7fac0815e010 2024-03-06T07:22:46.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.179+0000 7fac0f064640 1 -- 172.21.15.59:0/2534141608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac0815e010 msgr2=0x7fac080cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.179+0000 7fac0f064640 1 --2- 172.21.15.59:0/2534141608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac0815e010 0x7fac080cedd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fac0806e2a0 tx=0x7fabf802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.180+0000 7fac0f064640 1 -- 172.21.15.59:0/2534141608 shutdown_connections 2024-03-06T07:22:46.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.180+0000 7fac0f064640 1 --2- 172.21.15.59:0/2534141608 >> v2:172.21.15.164:3300/0 conn(0x7fac080b3240 0x7fac080b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.180+0000 7fac0f064640 1 --2- 172.21.15.59:0/2534141608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac0815e010 0x7fac080cedd0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.180+0000 7fac0f064640 1 -- 172.21.15.59:0/2534141608 >> 172.21.15.59:0/2534141608 conn(0x7fac0809a9f0 msgr2=0x7fac0809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.180+0000 7fac0f064640 1 -- 172.21.15.59:0/2534141608 shutdown_connections 2024-03-06T07:22:46.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.180+0000 7fac0f064640 1 -- 172.21.15.59:0/2534141608 wait complete. 2024-03-06T07:22:46.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.181+0000 7fac0f064640 1 Processor -- start 2024-03-06T07:22:46.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.181+0000 7fac0f064640 1 -- start start 2024-03-06T07:22:46.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0f064640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac0815e010 0x7fac081f7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0f064640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac080b6860 0x7fac081521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0f064640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac08156750 0x7fac080d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0f064640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac080cf310 con 0x7fac08156750 2024-03-06T07:22:46.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0f064640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0806ced0 con 0x7fac080b6860 2024-03-06T07:22:46.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0f064640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac080d21e0 con 0x7fac0815e010 2024-03-06T07:22:46.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0d601640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac08156750 0x7fac080d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0d601640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac08156750 0x7fac080d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36048/0 (socket says 172.21.15.59:36048) 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0d601640 1 -- 172.21.15.59:0/1861008825 learned_addr learned my addr 172.21.15.59:0/1861008825 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0ce00640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac0815e010 0x7fac081f7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac07fff640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac080b6860 0x7fac081521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0d601640 1 -- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac0815e010 msgr2=0x7fac081f7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0d601640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac0815e010 0x7fac081f7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0d601640 1 -- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac080b6860 msgr2=0x7fac081521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0d601640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac080b6860 0x7fac081521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0d601640 1 -- 172.21.15.59:0/1861008825 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac081f7cf0 con 0x7fac08156750 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0d601640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac08156750 0x7fac080d0ab0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fac0000c6b0 tx=0x7fac0000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac057fa640 1 -- 172.21.15.59:0/1861008825 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac00003b70 con 0x7fac08156750 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac057fa640 1 -- 172.21.15.59:0/1861008825 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac00003d30 con 0x7fac08156750 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0f064640 1 -- 172.21.15.59:0/1861008825 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac081f7fd0 con 0x7fac08156750 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac057fa640 1 -- 172.21.15.59:0/1861008825 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac00005310 con 0x7fac08156750 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.182+0000 7fac0f064640 1 -- 172.21.15.59:0/1861008825 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac081f8590 con 0x7fac08156750 2024-03-06T07:22:46.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.183+0000 7fac0f064640 1 -- 172.21.15.59:0/1861008825 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac080c5540 con 0x7fac08156750 2024-03-06T07:22:46.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.184+0000 7fac057fa640 1 -- 172.21.15.59:0/1861008825 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac000054b0 con 0x7fac08156750 2024-03-06T07:22:46.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.184+0000 7fac057fa640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabf406b4d0 0x7fabf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.184+0000 7fac057fa640 1 -- 172.21.15.59:0/1861008825 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fac00089bb0 con 0x7fac08156750 2024-03-06T07:22:46.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.184+0000 7fac0ce00640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabf406b4d0 0x7fabf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.185+0000 7fac0ce00640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabf406b4d0 0x7fabf406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fabf8036000 tx=0x7fabf8008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.188+0000 7fac057fa640 1 -- 172.21.15.59:0/1861008825 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac00058730 con 0x7fac08156750 2024-03-06T07:22:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.442+0000 7fac0f064640 1 -- 172.21.15.59:0/1861008825 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fac0804f330 con 0x7fac08156750 2024-03-06T07:22:46.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.444+0000 7fac057fa640 1 -- 172.21.15.59:0/1861008825 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fac0005c7d0 con 0x7fac08156750 2024-03-06T07:22:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 -- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabf406b4d0 msgr2=0x7fabf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabf406b4d0 0x7fabf406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fabf8036000 tx=0x7fabf8008030 comp rx=0 tx=0).stop 2024-03-06T07:22:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 -- 172.21.15.59:0/1861008825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac08156750 msgr2=0x7fac080d0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac08156750 0x7fac080d0ab0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fac0000c6b0 tx=0x7fac0000cb70 comp rx=0 tx=0).stop 2024-03-06T07:22:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 -- 172.21.15.59:0/1861008825 shutdown_connections 2024-03-06T07:22:46.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabf406b4d0 0x7fabf406d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac08156750 0x7fac080d0ab0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac080b6860 0x7fac081521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 --2- 172.21.15.59:0/1861008825 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac0815e010 0x7fac081f7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 -- 172.21.15.59:0/1861008825 >> 172.21.15.59:0/1861008825 conn(0x7fac0809a9f0 msgr2=0x7fac0815d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 -- 172.21.15.59:0/1861008825 shutdown_connections 2024-03-06T07:22:46.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.446+0000 7fac0f064640 1 -- 172.21.15.59:0/1861008825 wait complete. 2024-03-06T07:22:46.459 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:22:46.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.559+0000 7f44ed125640 1 Processor -- start 2024-03-06T07:22:46.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.559+0000 7f44ed125640 1 -- start start 2024-03-06T07:22:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44ed125640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f44e80700a0 0x7f44e81622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44ed125640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e806b600 0x7f44e806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44ed125640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e80b04c0 con 0x7f44e806b220 2024-03-06T07:22:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44ed125640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e81627e0 con 0x7f44e80700a0 2024-03-06T07:22:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44ed125640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e806d8d0 con 0x7f44e806b600 2024-03-06T07:22:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44e659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f44e80700a0 0x7f44e81622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44e659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f44e80700a0 0x7f44e81622a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36058/0 (socket says 172.21.15.59:36058) 2024-03-06T07:22:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44e659c640 1 -- 172.21.15.59:0/2959782887 learned_addr learned my addr 172.21.15.59:0/2959782887 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44e759e640 1 --2- 172.21.15.59:0/2959782887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e806b600 0x7f44e806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44e659c640 1 -- 172.21.15.59:0/2959782887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e806b600 msgr2=0x7f44e806ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44e659c640 1 --2- 172.21.15.59:0/2959782887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e806b600 0x7f44e806ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44e659c640 1 -- 172.21.15.59:0/2959782887 >> v1:172.21.15.164:6789/0 conn(0x7f44e806b220 legacy=0x7f44e806e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.560+0000 7f44e659c640 1 -- 172.21.15.59:0/2959782887 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e806db70 con 0x7f44e80700a0 2024-03-06T07:22:46.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.561+0000 7f44e659c640 1 --2- 172.21.15.59:0/2959782887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e80700a0 0x7f44e81622a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009870 tx=0x7f44d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde545f87717d76e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.561+0000 7f44e5d9b640 1 -- 172.21.15.59:0/2959782887 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0035020 con 0x7f44e80700a0 2024-03-06T07:22:46.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.561+0000 7f44e5d9b640 1 -- 172.21.15.59:0/2959782887 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d0031640 con 0x7f44e80700a0 2024-03-06T07:22:46.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.561+0000 7f44e5d9b640 1 -- 172.21.15.59:0/2959782887 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d002d040 con 0x7f44e80700a0 2024-03-06T07:22:46.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.562+0000 7f44ed125640 1 -- 172.21.15.59:0/2959782887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e80700a0 msgr2=0x7f44e81622a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.562+0000 7f44ed125640 1 --2- 172.21.15.59:0/2959782887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e80700a0 0x7f44e81622a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009870 tx=0x7f44d002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.562+0000 7f44ed125640 1 -- 172.21.15.59:0/2959782887 shutdown_connections 2024-03-06T07:22:46.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.562+0000 7f44ed125640 1 --2- 172.21.15.59:0/2959782887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e806b600 0x7f44e806ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.562+0000 7f44ed125640 1 --2- 172.21.15.59:0/2959782887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e80700a0 0x7f44e81622a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.562+0000 7f44ed125640 1 -- 172.21.15.59:0/2959782887 >> 172.21.15.59:0/2959782887 conn(0x7f44e80d0a10 msgr2=0x7f44e80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.562+0000 7f44ed125640 1 -- 172.21.15.59:0/2959782887 shutdown_connections 2024-03-06T07:22:46.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.562+0000 7f44ed125640 1 -- 172.21.15.59:0/2959782887 wait complete. 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.563+0000 7f44ed125640 1 Processor -- start 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.563+0000 7f44ed125640 1 -- start start 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.564+0000 7f44ed125640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e806b220 0x7f44e813dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.564+0000 7f44ed125640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e806b600 0x7f44e80b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.564+0000 7f44ed125640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80700a0 0x7f44e80b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.564+0000 7f44ed125640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e806fb10 con 0x7f44e806b220 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.564+0000 7f44ed125640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e806e130 con 0x7f44e80700a0 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.564+0000 7f44ed125640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e806fc90 con 0x7f44e806b600 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.564+0000 7f44e6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e806b220 0x7f44e813dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.564+0000 7f44e6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e806b220 0x7f44e813dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36060/0 (socket says 172.21.15.59:36060) 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.564+0000 7f44e6d9d640 1 -- 172.21.15.59:0/2964614305 learned_addr learned my addr 172.21.15.59:0/2964614305 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:46.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44e759e640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80700a0 0x7f44e80b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44e659c640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e806b600 0x7f44e80b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44e6d9d640 1 -- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e806b600 msgr2=0x7f44e80b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44e6d9d640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e806b600 0x7f44e80b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44e6d9d640 1 -- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80700a0 msgr2=0x7f44e80b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44e6d9d640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80700a0 0x7f44e80b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44e6d9d640 1 -- 172.21.15.59:0/2964614305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e8140760 con 0x7f44e806b220 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44e759e640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80700a0 0x7f44e80b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44e659c640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e806b600 0x7f44e80b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44e6d9d640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e806b220 0x7f44e813dd70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f44e000c3f0 tx=0x7f44e000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44df7fe640 1 -- 172.21.15.59:0/2964614305 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e00052d0 con 0x7f44e806b220 2024-03-06T07:22:46.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44df7fe640 1 -- 172.21.15.59:0/2964614305 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e0005490 con 0x7f44e806b220 2024-03-06T07:22:46.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44ed125640 1 -- 172.21.15.59:0/2964614305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e8140a40 con 0x7f44e806b220 2024-03-06T07:22:46.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44df7fe640 1 -- 172.21.15.59:0/2964614305 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0007260 con 0x7f44e806b220 2024-03-06T07:22:46.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.565+0000 7f44ed125640 1 -- 172.21.15.59:0/2964614305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e8156660 con 0x7f44e806b220 2024-03-06T07:22:46.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.566+0000 7f44ed125640 1 -- 172.21.15.59:0/2964614305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e804f330 con 0x7f44e806b220 2024-03-06T07:22:46.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.566+0000 7f44df7fe640 1 -- 172.21.15.59:0/2964614305 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44e0007400 con 0x7f44e806b220 2024-03-06T07:22:46.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.567+0000 7f44df7fe640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44c806b4d0 0x7f44c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.567+0000 7f44df7fe640 1 -- 172.21.15.59:0/2964614305 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f44e00893e0 con 0x7f44e806b220 2024-03-06T07:22:46.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.567+0000 7f44e659c640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44c806b4d0 0x7f44c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.567+0000 7f44e659c640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44c806b4d0 0x7f44c806d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f44d0036000 tx=0x7f44d0008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.571+0000 7f44df7fe640 1 -- 172.21.15.59:0/2964614305 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44e0058520 con 0x7f44e806b220 2024-03-06T07:22:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.822+0000 7f44ed125640 1 -- 172.21.15.59:0/2964614305 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f44e81609e0 con 0x7f44e806b220 2024-03-06T07:22:46.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.825+0000 7f44df7fe640 1 -- 172.21.15.59:0/2964614305 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f44e0057e20 con 0x7f44e806b220 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 -- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44c806b4d0 msgr2=0x7f44c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44c806b4d0 0x7f44c806d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f44d0036000 tx=0x7f44d0008030 comp rx=0 tx=0).stop 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 -- 172.21.15.59:0/2964614305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e806b220 msgr2=0x7f44e813dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e806b220 0x7f44e813dd70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f44e000c3f0 tx=0x7f44e000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 -- 172.21.15.59:0/2964614305 shutdown_connections 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44c806b4d0 0x7f44c806d990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80700a0 0x7f44e80b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e806b600 0x7f44e80b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 --2- 172.21.15.59:0/2964614305 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e806b220 0x7f44e813dd70 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 -- 172.21.15.59:0/2964614305 >> 172.21.15.59:0/2964614305 conn(0x7f44e80d0a10 msgr2=0x7f44e8161d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 -- 172.21.15.59:0/2964614305 shutdown_connections 2024-03-06T07:22:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.827+0000 7f44ed125640 1 -- 172.21.15.59:0/2964614305 wait complete. 2024-03-06T07:22:46.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:22:46.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.940+0000 7f0a74a5c640 1 Processor -- start 2024-03-06T07:22:46.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.940+0000 7f0a74a5c640 1 -- start start 2024-03-06T07:22:46.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.941+0000 7f0a74a5c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0a700b4460 0x7f0a700d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.941+0000 7f0a74a5c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a700d22c0 con 0x7f0a70162970 2024-03-06T07:22:46.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.941+0000 7f0a74a5c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a700d25a0 con 0x7f0a70161cf0 2024-03-06T07:22:46.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.941+0000 7f0a74a5c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a70161b00 con 0x7f0a700b4460 2024-03-06T07:22:46.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.941+0000 7f0a6dd9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0a70162970 0x7f0a700b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46910/0 (socket says 172.21.15.59:46910) 2024-03-06T07:22:46.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.941+0000 7f0a6dd9b640 1 -- 172.21.15.59:0/2464275841 learned_addr learned my addr 172.21.15.59:0/2464275841 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:46.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6e59c640 1 --2- 172.21.15.59:0/2464275841 >> v2:172.21.15.164:3300/0 conn(0x7f0a700b4460 0x7f0a700d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248876722 0 0) 0x7f0a700d22c0 con 0x7f0a70162970 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a5c003520 con 0x7f0a70162970 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1376066391 0 0) 0x7f0a5c003520 con 0x7f0a70162970 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 >> v2:172.21.15.164:3300/0 conn(0x7f0a700b4460 msgr2=0x7f0a700d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 --2- 172.21.15.59:0/2464275841 >> v2:172.21.15.164:3300/0 conn(0x7f0a700b4460 0x7f0a700d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 >> v1:172.21.15.164:6789/0 conn(0x7f0a70161cf0 legacy=0x7f0a700cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a700d2880 con 0x7f0a70162970 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f0a58001670 con 0x7f0a70162970 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 >> v1:172.21.15.59:6789/0 conn(0x7f0a70162970 legacy=0x7f0a700b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5c0039b0 0x7f0a5c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5c004800 0x7f0a5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 0x7f0a5c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a70161b00 con 0x7f0a5c0051c0 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c00c060 con 0x7f0a5c0039b0 2024-03-06T07:22:46.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c00c260 con 0x7f0a5c004800 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6ed9d640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 0x7f0a5c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6e59c640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5c0039b0 0x7f0a5c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.942+0000 7f0a6dd9b640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5c004800 0x7f0a5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.943+0000 7f0a6ed9d640 1 -- 172.21.15.59:0/2464275841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5c004800 msgr2=0x7f0a5c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.943+0000 7f0a6ed9d640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5c004800 0x7f0a5c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.943+0000 7f0a6ed9d640 1 -- 172.21.15.59:0/2464275841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5c0039b0 msgr2=0x7f0a5c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.943+0000 7f0a6ed9d640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5c0039b0 0x7f0a5c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.943+0000 7f0a6ed9d640 1 -- 172.21.15.59:0/2464275841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c003520 con 0x7f0a5c0051c0 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.943+0000 7f0a6dd9b640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5c004800 0x7f0a5c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.943+0000 7f0a6ed9d640 1 -- 172.21.15.59:0/2464275841 reap_dead start 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.943+0000 7f0a6ed9d640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 0x7f0a5c00b9c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0a60001e50 tx=0x7f0a5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e152043899e1db17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.943+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60001a70 con 0x7f0a5c0051c0 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.943+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a60003bb0 con 0x7f0a5c0051c0 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.943+0000 7f0a6d59a640 1 -- 172.21.15.59:0/2464275841 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a600050d0 con 0x7f0a5c0051c0 2024-03-06T07:22:46.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.944+0000 7f0a74a5c640 1 -- 172.21.15.59:0/2464275841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 msgr2=0x7f0a5c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.944+0000 7f0a74a5c640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 0x7f0a5c00b9c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0a60001e50 tx=0x7f0a5c005830 comp rx=0 tx=0).stop 2024-03-06T07:22:46.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.944+0000 7f0a74a5c640 1 -- 172.21.15.59:0/2464275841 shutdown_connections 2024-03-06T07:22:46.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.944+0000 7f0a74a5c640 1 --2- 172.21.15.59:0/2464275841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 0x7f0a5c00b9c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.944+0000 7f0a74a5c640 1 -- 172.21.15.59:0/2464275841 >> 172.21.15.59:0/2464275841 conn(0x7f0a700d3ce0 msgr2=0x7f0a700d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.944+0000 7f0a74a5c640 1 -- 172.21.15.59:0/2464275841 shutdown_connections 2024-03-06T07:22:46.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.944+0000 7f0a74a5c640 1 -- 172.21.15.59:0/2464275841 wait complete. 2024-03-06T07:22:46.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.945+0000 7f0a74a5c640 1 Processor -- start 2024-03-06T07:22:46.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.945+0000 7f0a74a5c640 1 -- start start 2024-03-06T07:22:46.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a74a5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 0x7f0a700c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a74a5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a7015c530 0x7f0a70152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a74a5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a700b8fb0 0x7f0a700b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a74a5c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a700ce950 con 0x7f0a5c0051c0 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a74a5c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a700cecc0 con 0x7f0a700b8fb0 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a74a5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a700b4840 con 0x7f0a7015c530 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a6e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 0x7f0a700c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a6e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 0x7f0a700c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36074/0 (socket says 172.21.15.59:36074) 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a6e59c640 1 -- 172.21.15.59:0/3707751467 learned_addr learned my addr 172.21.15.59:0/3707751467 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a6e59c640 1 -- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a7015c530 msgr2=0x7f0a70152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a6e59c640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a7015c530 0x7f0a70152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a6e59c640 1 -- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a700b8fb0 msgr2=0x7f0a700b1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.946+0000 7f0a6e59c640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a700b8fb0 0x7f0a700b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.947+0000 7f0a6e59c640 1 -- 172.21.15.59:0/3707751467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a700c7d50 con 0x7f0a5c0051c0 2024-03-06T07:22:46.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.947+0000 7f0a6e59c640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 0x7f0a700c7810 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0a68011910 tx=0x7f0a68011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.947+0000 7f0a66ffd640 1 -- 172.21.15.59:0/3707751467 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6800faf0 con 0x7f0a5c0051c0 2024-03-06T07:22:46.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.947+0000 7f0a66ffd640 1 -- 172.21.15.59:0/3707751467 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6800fc90 con 0x7f0a5c0051c0 2024-03-06T07:22:46.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.947+0000 7f0a74a5c640 1 -- 172.21.15.59:0/3707751467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a701528c0 con 0x7f0a5c0051c0 2024-03-06T07:22:46.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.947+0000 7f0a66ffd640 1 -- 172.21.15.59:0/3707751467 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68014440 con 0x7f0a5c0051c0 2024-03-06T07:22:46.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.947+0000 7f0a74a5c640 1 -- 172.21.15.59:0/3707751467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a700cc1d0 con 0x7f0a5c0051c0 2024-03-06T07:22:46.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.948+0000 7f0a74a5c640 1 -- 172.21.15.59:0/3707751467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7004f330 con 0x7f0a5c0051c0 2024-03-06T07:22:46.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.948+0000 7f0a66ffd640 1 -- 172.21.15.59:0/3707751467 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a68012040 con 0x7f0a5c0051c0 2024-03-06T07:22:46.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.948+0000 7f0a66ffd640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a5006b4d0 0x7f0a5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.949+0000 7f0a66ffd640 1 -- 172.21.15.59:0/3707751467 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0a6808a500 con 0x7f0a5c0051c0 2024-03-06T07:22:46.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.949+0000 7f0a6dd9b640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a5006b4d0 0x7f0a5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.949+0000 7f0a6dd9b640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a5006b4d0 0x7f0a5006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0a58001b80 tx=0x7f0a58001150 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:46.953+0000 7f0a66ffd640 1 -- 172.21.15.59:0/3707751467 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a701528c0 con 0x7f0a5c0051c0 2024-03-06T07:22:47.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.201+0000 7f0a74a5c640 1 -- 172.21.15.59:0/3707751467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f0a700ccbb0 con 0x7f0a5c0051c0 2024-03-06T07:22:47.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.203+0000 7f0a66ffd640 1 -- 172.21.15.59:0/3707751467 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0a68019030 con 0x7f0a5c0051c0 2024-03-06T07:22:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.205+0000 7f0a74a5c640 1 -- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a5006b4d0 msgr2=0x7f0a5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.205+0000 7f0a74a5c640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a5006b4d0 0x7f0a5006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0a58001b80 tx=0x7f0a58001150 comp rx=0 tx=0).stop 2024-03-06T07:22:47.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.205+0000 7f0a74a5c640 1 -- 172.21.15.59:0/3707751467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 msgr2=0x7f0a700c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.205+0000 7f0a74a5c640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 0x7f0a700c7810 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0a68011910 tx=0x7f0a68011dd0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.205+0000 7f0a74a5c640 1 -- 172.21.15.59:0/3707751467 shutdown_connections 2024-03-06T07:22:47.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.205+0000 7f0a74a5c640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a5006b4d0 0x7f0a5006d990 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.205+0000 7f0a74a5c640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a700b8fb0 0x7f0a700b1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.205+0000 7f0a74a5c640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a7015c530 0x7f0a70152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.205+0000 7f0a74a5c640 1 --2- 172.21.15.59:0/3707751467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a5c0051c0 0x7f0a700c7810 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.205+0000 7f0a74a5c640 1 -- 172.21.15.59:0/3707751467 >> 172.21.15.59:0/3707751467 conn(0x7f0a700d3ce0 msgr2=0x7f0a700d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.206+0000 7f0a74a5c640 1 -- 172.21.15.59:0/3707751467 shutdown_connections 2024-03-06T07:22:47.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.206+0000 7f0a74a5c640 1 -- 172.21.15.59:0/3707751467 wait complete. 2024-03-06T07:22:47.218 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:22:47.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.318+0000 7fe01b31e640 1 Processor -- start 2024-03-06T07:22:47.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.318+0000 7fe01b31e640 1 -- start start 2024-03-06T07:22:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.319+0000 7fe01b31e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe0140b4c00 0x7fe0140b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.319+0000 7fe01b31e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 0x7fe0140b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.319+0000 7fe01b31e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01415d510 con 0x7fe01415e540 2024-03-06T07:22:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.319+0000 7fe01b31e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01415e0b0 con 0x7fe0140b4c00 2024-03-06T07:22:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.319+0000 7fe01b31e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01415d780 con 0x7fe0140b28a0 2024-03-06T07:22:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.320+0000 7fe0188b9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 0x7fe0140b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.320+0000 7fe0190ba640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe0140b4c00 0x7fe0140b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.320+0000 7fe0188b9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 0x7fe0140b2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:59982/0 (socket says 172.21.15.59:59982) 2024-03-06T07:22:47.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.320+0000 7fe0190ba640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe0140b4c00 0x7fe0140b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:45188/0 (socket says 172.21.15.59:45188) 2024-03-06T07:22:47.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.320+0000 7fe0188b9640 1 -- 172.21.15.59:0/4091633702 learned_addr learned my addr 172.21.15.59:0/4091633702 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:47.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.320+0000 7fe0188b9640 1 -- 172.21.15.59:0/4091633702 >> v2:172.21.15.164:3300/0 conn(0x7fe0140b4c00 msgr2=0x7fe0140b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.320+0000 7fe0188b9640 1 --2- 172.21.15.59:0/4091633702 >> v2:172.21.15.164:3300/0 conn(0x7fe0140b4c00 0x7fe0140b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.320+0000 7fe0188b9640 1 -- 172.21.15.59:0/4091633702 >> v1:172.21.15.164:6789/0 conn(0x7fe01415e540 legacy=0x7fe01415e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.320+0000 7fe0188b9640 1 -- 172.21.15.59:0/4091633702 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01415da60 con 0x7fe0140b28a0 2024-03-06T07:22:47.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.321+0000 7fe0188b9640 1 --2- 172.21.15.59:0/4091633702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 0x7fe0140b2ce0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe008009870 tx=0x7fe00802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ccf738b1c183871 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.321+0000 7fe013fff640 1 -- 172.21.15.59:0/4091633702 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00802f840 con 0x7fe0140b28a0 2024-03-06T07:22:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.321+0000 7fe013fff640 1 -- 172.21.15.59:0/4091633702 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00802f9e0 con 0x7fe0140b28a0 2024-03-06T07:22:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.321+0000 7fe013fff640 1 -- 172.21.15.59:0/4091633702 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00802fdb0 con 0x7fe0140b28a0 2024-03-06T07:22:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.322+0000 7fe01b31e640 1 -- 172.21.15.59:0/4091633702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 msgr2=0x7fe0140b2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.322+0000 7fe01b31e640 1 --2- 172.21.15.59:0/4091633702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 0x7fe0140b2ce0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe008009870 tx=0x7fe00802ba20 comp rx=0 tx=0).stop 2024-03-06T07:22:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.322+0000 7fe01b31e640 1 -- 172.21.15.59:0/4091633702 shutdown_connections 2024-03-06T07:22:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.322+0000 7fe01b31e640 1 --2- 172.21.15.59:0/4091633702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 0x7fe0140b2ce0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.322+0000 7fe01b31e640 1 --2- 172.21.15.59:0/4091633702 >> v2:172.21.15.164:3300/0 conn(0x7fe0140b4c00 0x7fe0140b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.322+0000 7fe01b31e640 1 -- 172.21.15.59:0/4091633702 >> 172.21.15.59:0/4091633702 conn(0x7fe01409a9f0 msgr2=0x7fe01409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.322+0000 7fe01b31e640 1 -- 172.21.15.59:0/4091633702 shutdown_connections 2024-03-06T07:22:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.322+0000 7fe01b31e640 1 -- 172.21.15.59:0/4091633702 wait complete. 2024-03-06T07:22:47.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.323+0000 7fe01b31e640 1 Processor -- start 2024-03-06T07:22:47.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.323+0000 7fe01b31e640 1 -- start start 2024-03-06T07:22:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe01b31e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 0x7fe0140af160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe01b31e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0140b4c00 0x7fe014152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe01b31e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01415e540 0x7fe0140b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe01b31e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140b33a0 con 0x7fe01415e540 2024-03-06T07:22:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe01b31e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0140b3220 con 0x7fe0140b4c00 2024-03-06T07:22:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe01b31e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01415f090 con 0x7fe0140b28a0 2024-03-06T07:22:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe0198bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01415e540 0x7fe0140b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe0198bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01415e540 0x7fe0140b1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36078/0 (socket says 172.21.15.59:36078) 2024-03-06T07:22:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe0198bb640 1 -- 172.21.15.59:0/1463177850 learned_addr learned my addr 172.21.15.59:0/1463177850 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:47.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe0190ba640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 0x7fe0140af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe0188b9640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0140b4c00 0x7fe014152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe0198bb640 1 -- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 msgr2=0x7fe0140af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe0198bb640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 0x7fe0140af160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe0198bb640 1 -- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0140b4c00 msgr2=0x7fe014152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe0198bb640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0140b4c00 0x7fe014152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe0198bb640 1 -- 172.21.15.59:0/1463177850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe014152580 con 0x7fe01415e540 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.324+0000 7fe0198bb640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01415e540 0x7fe0140b1970 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00c390 tx=0x7fe00c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.325+0000 7fe011ffb640 1 -- 172.21.15.59:0/1463177850 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c017070 con 0x7fe01415e540 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.325+0000 7fe011ffb640 1 -- 172.21.15.59:0/1463177850 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00c0052d0 con 0x7fe01415e540 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.325+0000 7fe01b31e640 1 -- 172.21.15.59:0/1463177850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0140af820 con 0x7fe01415e540 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.325+0000 7fe011ffb640 1 -- 172.21.15.59:0/1463177850 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c003810 con 0x7fe01415e540 2024-03-06T07:22:47.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.325+0000 7fe01b31e640 1 -- 172.21.15.59:0/1463177850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0140afde0 con 0x7fe01415e540 2024-03-06T07:22:47.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.326+0000 7fe01b31e640 1 -- 172.21.15.59:0/1463177850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01404f330 con 0x7fe01415e540 2024-03-06T07:22:47.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.326+0000 7fe011ffb640 1 -- 172.21.15.59:0/1463177850 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe00c01a070 con 0x7fe01415e540 2024-03-06T07:22:47.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.326+0000 7fe011ffb640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe00006b4d0 0x7fe00006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.327+0000 7fe011ffb640 1 -- 172.21.15.59:0/1463177850 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe00c089280 con 0x7fe01415e540 2024-03-06T07:22:47.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.327+0000 7fe0190ba640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe00006b4d0 0x7fe00006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.327+0000 7fe0190ba640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe00006b4d0 0x7fe00006d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe0040113e0 tx=0x7fe00400b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.331+0000 7fe011ffb640 1 -- 172.21.15.59:0/1463177850 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe014152580 con 0x7fe01415e540 2024-03-06T07:22:47.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.583+0000 7fe01b31e640 1 -- 172.21.15.59:0/1463177850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fe014074850 con 0x7fe01415e540 2024-03-06T07:22:47.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.585+0000 7fe011ffb640 1 -- 172.21.15.59:0/1463177850 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe00c057cd0 con 0x7fe01415e540 2024-03-06T07:22:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 -- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe00006b4d0 msgr2=0x7fe00006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe00006b4d0 0x7fe00006d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe0040113e0 tx=0x7fe00400b040 comp rx=0 tx=0).stop 2024-03-06T07:22:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 -- 172.21.15.59:0/1463177850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01415e540 msgr2=0x7fe0140b1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01415e540 0x7fe0140b1970 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00c390 tx=0x7fe00c00c850 comp rx=0 tx=0).stop 2024-03-06T07:22:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 -- 172.21.15.59:0/1463177850 shutdown_connections 2024-03-06T07:22:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe00006b4d0 0x7fe00006d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe01415e540 0x7fe0140b1970 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0140b4c00 0x7fe014152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 --2- 172.21.15.59:0/1463177850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0140b28a0 0x7fe0140af160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 -- 172.21.15.59:0/1463177850 >> 172.21.15.59:0/1463177850 conn(0x7fe01409a9f0 msgr2=0x7fe01406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 -- 172.21.15.59:0/1463177850 shutdown_connections 2024-03-06T07:22:47.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.587+0000 7fe01b31e640 1 -- 172.21.15.59:0/1463177850 wait complete. 2024-03-06T07:22:47.600 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:47.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.698+0000 7f5d7c7f7640 1 Processor -- start 2024-03-06T07:22:47.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.698+0000 7f5d7c7f7640 1 -- start start 2024-03-06T07:22:47.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.699+0000 7f5d7c7f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d74166af0 0x7f5d74168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.699+0000 7f5d7c7f7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7415e380 con 0x7f5d740d00c0 2024-03-06T07:22:47.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.699+0000 7f5d7c7f7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d740cdc10 con 0x7f5d740cf850 2024-03-06T07:22:47.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.699+0000 7f5d7c7f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7406ca90 con 0x7f5d74166af0 2024-03-06T07:22:47.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d7a593640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f5d740d00c0 0x7f5d740d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46924/0 (socket says 172.21.15.59:46924) 2024-03-06T07:22:47.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d7a593640 1 -- 172.21.15.59:0/3271247459 learned_addr learned my addr 172.21.15.59:0/3271247459 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:47.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d7ad94640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d74166af0 0x7f5d74168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217919387 0 0) 0x7f5d7415e380 con 0x7f5d740d00c0 2024-03-06T07:22:47.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d60003520 con 0x7f5d740d00c0 2024-03-06T07:22:47.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 88566230 0 0) 0x7f5d60003520 con 0x7f5d740d00c0 2024-03-06T07:22:47.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d74166af0 msgr2=0x7f5d74168ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d74166af0 0x7f5d74168ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 >> v1:172.21.15.164:6789/0 conn(0x7f5d740cf850 legacy=0x7f5d740cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7406c160 con 0x7f5d740d00c0 2024-03-06T07:22:47.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f5d70001670 con 0x7f5d740d00c0 2024-03-06T07:22:47.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 >> v1:172.21.15.59:6789/0 conn(0x7f5d740d00c0 legacy=0x7f5d740d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d600039e0 0x7f5d60003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d60004800 0x7f5d60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d74166af0 0x7f5d600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7406ca90 con 0x7f5d60004800 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6000baa0 con 0x7f5d600039e0 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6000bca0 con 0x7f5d74166af0 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d79d92640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d60004800 0x7f5d60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d7a593640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d600039e0 0x7f5d60003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.700+0000 7f5d7ad94640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d74166af0 0x7f5d600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.701+0000 7f5d79d92640 1 -- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d74166af0 msgr2=0x7f5d600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.701+0000 7f5d79d92640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d74166af0 0x7f5d600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.701+0000 7f5d79d92640 1 -- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d600039e0 msgr2=0x7f5d60003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.701+0000 7f5d79d92640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d600039e0 0x7f5d60003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.701+0000 7f5d79d92640 1 -- 172.21.15.59:0/3271247459 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d60003520 con 0x7f5d60004800 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.701+0000 7f5d7a593640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d600039e0 0x7f5d60003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.701+0000 7f5d7ad94640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d74166af0 0x7f5d600051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.701+0000 7f5d79d92640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d60004800 0x7f5d60004c80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5d64001720 tx=0x7f5d60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971be2e0f32b1e95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.701+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64001370 con 0x7f5d60004800 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.701+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d640052d0 con 0x7f5d60004800 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.701+0000 7f5d79591640 1 -- 172.21.15.59:0/3271247459 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64003830 con 0x7f5d60004800 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.702+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/3271247459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d60004800 msgr2=0x7f5d60004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.702+0000 7f5d7c7f7640 1 --2- 172.21.15.59:0/3271247459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d60004800 0x7f5d60004c80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5d64001720 tx=0x7f5d60005830 comp rx=0 tx=0).stop 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.702+0000 7f5d7ad94640 1 -- 172.21.15.59:0/3271247459 reap_dead start 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.702+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/3271247459 shutdown_connections 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.702+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/3271247459 >> 172.21.15.59:0/3271247459 conn(0x7f5d740b35a0 msgr2=0x7f5d740b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.702+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/3271247459 shutdown_connections 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.702+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/3271247459 wait complete. 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.703+0000 7f5d7c7f7640 1 Processor -- start 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.703+0000 7f5d7c7f7640 1 -- start start 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d7c7f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d60004800 0x7f5d74152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d7c7f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d7415b8f0 0x7f5d7415bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d7c7f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d740cdde0 0x7f5d740ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d7c7f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7406c720 con 0x7f5d7415b8f0 2024-03-06T07:22:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d7c7f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7406df30 con 0x7f5d740cdde0 2024-03-06T07:22:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d7c7f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740b1030 con 0x7f5d60004800 2024-03-06T07:22:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d79d92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d7415b8f0 0x7f5d7415bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d79d92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d7415b8f0 0x7f5d7415bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36108/0 (socket says 172.21.15.59:36108) 2024-03-06T07:22:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d79d92640 1 -- 172.21.15.59:0/1463135430 learned_addr learned my addr 172.21.15.59:0/1463135430 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d7a593640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d60004800 0x7f5d74152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d7ad94640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d740cdde0 0x7f5d740ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d79d92640 1 -- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d60004800 msgr2=0x7f5d74152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d79d92640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d60004800 0x7f5d74152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d79d92640 1 -- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d740cdde0 msgr2=0x7f5d740ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d79d92640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d740cdde0 0x7f5d740ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d79d92640 1 -- 172.21.15.59:0/1463135430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d740c8b40 con 0x7f5d7415b8f0 2024-03-06T07:22:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.704+0000 7f5d7ad94640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d740cdde0 0x7f5d740ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.705+0000 7f5d79d92640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d7415b8f0 0x7f5d7415bd50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5d64005040 tx=0x7f5d64001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.705+0000 7f5d6affd640 1 -- 172.21.15.59:0/1463135430 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64041070 con 0x7f5d7415b8f0 2024-03-06T07:22:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.705+0000 7f5d6affd640 1 -- 172.21.15.59:0/1463135430 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d64005480 con 0x7f5d7415b8f0 2024-03-06T07:22:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.705+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/1463135430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7415a500 con 0x7f5d7415b8f0 2024-03-06T07:22:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.705+0000 7f5d6affd640 1 -- 172.21.15.59:0/1463135430 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64007a30 con 0x7f5d7415b8f0 2024-03-06T07:22:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.705+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/1463135430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7415aac0 con 0x7f5d7415b8f0 2024-03-06T07:22:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.706+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/1463135430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7404f330 con 0x7f5d7415b8f0 2024-03-06T07:22:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.706+0000 7f5d6affd640 1 -- 172.21.15.59:0/1463135430 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d64044070 con 0x7f5d7415b8f0 2024-03-06T07:22:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.706+0000 7f5d6affd640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d5806b500 0x7f5d5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.706+0000 7f5d6affd640 1 -- 172.21.15.59:0/1463135430 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5d640abb00 con 0x7f5d7415b8f0 2024-03-06T07:22:47.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.707+0000 7f5d7a593640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d5806b500 0x7f5d5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.707+0000 7f5d7a593640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d5806b500 0x7f5d5806d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5d70000fb0 tx=0x7f5d70009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.711+0000 7f5d6affd640 1 -- 172.21.15.59:0/1463135430 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d740c8b40 con 0x7f5d7415b8f0 2024-03-06T07:22:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.896+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/1463135430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d740b09f0 con 0x7f5d7415b8f0 2024-03-06T07:22:47.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.896+0000 7f5d6affd640 1 -- 172.21.15.59:0/1463135430 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f5d6407ac40 con 0x7f5d7415b8f0 2024-03-06T07:22:47.898 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:47.899 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":29,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:44.910266+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.898+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d5806b500 msgr2=0x7f5d5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.898+0000 7f5d7c7f7640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d5806b500 0x7f5d5806d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5d70000fb0 tx=0x7f5d70009040 comp rx=0 tx=0).stop 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.898+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/1463135430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d7415b8f0 msgr2=0x7f5d7415bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.898+0000 7f5d7c7f7640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d7415b8f0 0x7f5d7415bd50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5d64005040 tx=0x7f5d64001120 comp rx=0 tx=0).stop 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.899+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/1463135430 shutdown_connections 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.899+0000 7f5d7c7f7640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d5806b500 0x7f5d5806d9c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.899+0000 7f5d7c7f7640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d740cdde0 0x7f5d740ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.899+0000 7f5d7c7f7640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d7415b8f0 0x7f5d7415bd50 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.899+0000 7f5d7c7f7640 1 --2- 172.21.15.59:0/1463135430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d60004800 0x7f5d74152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.899+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/1463135430 >> 172.21.15.59:0/1463135430 conn(0x7f5d740b35a0 msgr2=0x7f5d7406f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.899+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/1463135430 shutdown_connections 2024-03-06T07:22:47.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:47.899+0000 7f5d7c7f7640 1 -- 172.21.15.59:0/1463135430 wait complete. 2024-03-06T07:22:47.911 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:47.912 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.014+0000 7fcfff7d0640 1 Processor -- start 2024-03-06T07:22:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.014+0000 7fcfff7d0640 1 -- start start 2024-03-06T07:22:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcfff7d0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 0x7fcff80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcfff7d0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcff806cab0 0x7fcff80ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcfff7d0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff806d030 con 0x7fcff815def0 2024-03-06T07:22:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcfff7d0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcff80cf1c0 con 0x7fcff806cab0 2024-03-06T07:22:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcfff7d0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80cc760 con 0x7fcff80cea50 2024-03-06T07:22:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcffdd6d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcff806cab0 0x7fcff80ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcffd56c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcff815def0 0x7fcff80ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46926/0 (socket says 172.21.15.59:46926) 2024-03-06T07:22:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcffd56c640 1 -- 172.21.15.59:0/1319427620 learned_addr learned my addr 172.21.15.59:0/1319427620 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcffcd6b640 1 --2- 172.21.15.59:0/1319427620 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 0x7fcff80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcffdd6d640 1 -- 172.21.15.59:0/1319427620 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 msgr2=0x7fcff80b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcffdd6d640 1 --2- 172.21.15.59:0/1319427620 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 0x7fcff80b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcffdd6d640 1 -- 172.21.15.59:0/1319427620 >> v1:172.21.15.59:6789/0 conn(0x7fcff815def0 legacy=0x7fcff80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcfeffff640 1 -- 172.21.15.59:0/1319427620 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063262092 0 0) 0x7fcff806d030 con 0x7fcff815def0 2024-03-06T07:22:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcffdd6d640 1 -- 172.21.15.59:0/1319427620 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff806d310 con 0x7fcff806cab0 2024-03-06T07:22:48.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.015+0000 7fcffcd6b640 1 --2- 172.21.15.59:0/1319427620 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 0x7fcff80b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:48.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.016+0000 7fcffdd6d640 1 --2- 172.21.15.59:0/1319427620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcff806cab0 0x7fcff80ccbf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcff0009d90 tx=0x7fcff002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98d25483ce46083 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.016+0000 7fcfeffff640 1 -- 172.21.15.59:0/1319427620 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0039070 con 0x7fcff806cab0 2024-03-06T07:22:48.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.016+0000 7fcfeffff640 1 -- 172.21.15.59:0/1319427620 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff002cce0 con 0x7fcff806cab0 2024-03-06T07:22:48.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.016+0000 7fcfeffff640 1 -- 172.21.15.59:0/1319427620 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0034660 con 0x7fcff806cab0 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.017+0000 7fcfff7d0640 1 -- 172.21.15.59:0/1319427620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcff806cab0 msgr2=0x7fcff80ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.017+0000 7fcfff7d0640 1 --2- 172.21.15.59:0/1319427620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcff806cab0 0x7fcff80ccbf0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcff0009d90 tx=0x7fcff002bf30 comp rx=0 tx=0).stop 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.017+0000 7fcfff7d0640 1 -- 172.21.15.59:0/1319427620 shutdown_connections 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.017+0000 7fcfff7d0640 1 --2- 172.21.15.59:0/1319427620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcff806cab0 0x7fcff80ccbf0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.017+0000 7fcfff7d0640 1 --2- 172.21.15.59:0/1319427620 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 0x7fcff80b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.017+0000 7fcfff7d0640 1 -- 172.21.15.59:0/1319427620 >> 172.21.15.59:0/1319427620 conn(0x7fcff80b3690 msgr2=0x7fcff80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.017+0000 7fcfff7d0640 1 -- 172.21.15.59:0/1319427620 shutdown_connections 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.017+0000 7fcfff7d0640 1 -- 172.21.15.59:0/1319427620 wait complete. 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.018+0000 7fcfff7d0640 1 Processor -- start 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.018+0000 7fcfff7d0640 1 -- start start 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcfff7d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcff806cab0 0x7fcff80c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcfff7d0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 0x7fcff80b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcfff7d0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcff815def0 0x7fcff80c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcfff7d0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff815e2d0 con 0x7fcff806cab0 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcfff7d0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80cc3f0 con 0x7fcff815def0 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcfff7d0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff815e450 con 0x7fcff80cea50 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffd56c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcff806cab0 0x7fcff80c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffd56c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcff806cab0 0x7fcff80c8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36120/0 (socket says 172.21.15.59:36120) 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffd56c640 1 -- 172.21.15.59:0/2983915512 learned_addr learned my addr 172.21.15.59:0/2983915512 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffcd6b640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 0x7fcff80b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffdd6d640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcff815def0 0x7fcff80c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffd56c640 1 -- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 msgr2=0x7fcff80b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffd56c640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 0x7fcff80b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffd56c640 1 -- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcff815def0 msgr2=0x7fcff80c60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffd56c640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcff815def0 0x7fcff80c60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffd56c640 1 -- 172.21.15.59:0/2983915512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff80b63b0 con 0x7fcff806cab0 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffcd6b640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 0x7fcff80b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:48.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcffd56c640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcff806cab0 0x7fcff80c8250 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcff400c430 tx=0x7fcff400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcfedffb640 1 -- 172.21.15.59:0/2983915512 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff40074e0 con 0x7fcff806cab0 2024-03-06T07:22:48.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcfedffb640 1 -- 172.21.15.59:0/2983915512 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff40047e0 con 0x7fcff806cab0 2024-03-06T07:22:48.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcfff7d0640 1 -- 172.21.15.59:0/2983915512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff80b6690 con 0x7fcff806cab0 2024-03-06T07:22:48.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.019+0000 7fcfedffb640 1 -- 172.21.15.59:0/2983915512 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4004bd0 con 0x7fcff806cab0 2024-03-06T07:22:48.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.020+0000 7fcfff7d0640 1 -- 172.21.15.59:0/2983915512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff80b6c50 con 0x7fcff806cab0 2024-03-06T07:22:48.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.021+0000 7fcfff7d0640 1 -- 172.21.15.59:0/2983915512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff809fcf0 con 0x7fcff806cab0 2024-03-06T07:22:48.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.021+0000 7fcfedffb640 1 -- 172.21.15.59:0/2983915512 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcff4011070 con 0x7fcff806cab0 2024-03-06T07:22:48.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.021+0000 7fcfedffb640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcfdc06b4d0 0x7fcfdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.021+0000 7fcfedffb640 1 -- 172.21.15.59:0/2983915512 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcff4089b80 con 0x7fcff806cab0 2024-03-06T07:22:48.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.022+0000 7fcffcd6b640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcfdc06b4d0 0x7fcfdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.022+0000 7fcffcd6b640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcfdc06b4d0 0x7fcfdc06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcff80ce660 tx=0x7fcfe8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.026+0000 7fcfedffb640 1 -- 172.21.15.59:0/2983915512 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcff4058d40 con 0x7fcff806cab0 2024-03-06T07:22:48.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.205+0000 7fcfff7d0640 1 -- 172.21.15.59:0/2983915512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcff815d710 con 0x7fcff806cab0 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.207+0000 7fcfedffb640 1 -- 172.21.15.59:0/2983915512 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcff4058640 con 0x7fcff806cab0 2024-03-06T07:22:48.208 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 -- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcfdc06b4d0 msgr2=0x7fcfdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcfdc06b4d0 0x7fcfdc06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcff80ce660 tx=0x7fcfe8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 -- 172.21.15.59:0/2983915512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcff806cab0 msgr2=0x7fcff80c8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcff806cab0 0x7fcff80c8250 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcff400c430 tx=0x7fcff400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 -- 172.21.15.59:0/2983915512 shutdown_connections 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcfdc06b4d0 0x7fcfdc06d990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcff815def0 0x7fcff80c60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcff80cea50 0x7fcff80b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 --2- 172.21.15.59:0/2983915512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcff806cab0 0x7fcff80c8250 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 -- 172.21.15.59:0/2983915512 >> 172.21.15.59:0/2983915512 conn(0x7fcff80b3690 msgr2=0x7fcff81605e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 -- 172.21.15.59:0/2983915512 shutdown_connections 2024-03-06T07:22:48.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.209+0000 7fcfff7d0640 1 -- 172.21.15.59:0/2983915512 wait complete. 2024-03-06T07:22:48.221 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.321+0000 7f8f0d85c640 1 Processor -- start 2024-03-06T07:22:48.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.321+0000 7f8f0d85c640 1 -- start start 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.322+0000 7f8f0d85c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8f0806c280 0x7f8f080cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.322+0000 7f8f0d85c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0815b160 con 0x7f8f080cf470 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.322+0000 7f8f0d85c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f080b16f0 con 0x7f8f080730d0 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.322+0000 7f8f0d85c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f080b4f70 con 0x7f8f0806c280 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.322+0000 7f8f077fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8f0806c280 0x7f8f080cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.322+0000 7f8f06ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8f080cf470 0x7f8f080b59c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46930/0 (socket says 172.21.15.59:46930) 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.322+0000 7f8f06ffd640 1 -- 172.21.15.59:0/3815652858 learned_addr learned my addr 172.21.15.59:0/3815652858 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:48.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.322+0000 7f8f077fe640 1 -- 172.21.15.59:0/3815652858 >> v1:172.21.15.164:6789/0 conn(0x7f8f080730d0 legacy=0x7f8f0806bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.322+0000 7f8f077fe640 1 -- 172.21.15.59:0/3815652858 >> v1:172.21.15.59:6789/0 conn(0x7f8f080cf470 legacy=0x7f8f080b59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.322+0000 7f8f077fe640 1 -- 172.21.15.59:0/3815652858 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f080b19d0 con 0x7f8f0806c280 2024-03-06T07:22:48.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.322+0000 7f8f067fc640 1 -- 172.21.15.59:0/3815652858 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298881187 0 0) 0x7f8f0815b160 con 0x7f8f080cf470 2024-03-06T07:22:48.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.323+0000 7f8f077fe640 1 --2- 172.21.15.59:0/3815652858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f0806c280 0x7f8f080cfb70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009d90 tx=0x7f8f0002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa0616344326e3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.323+0000 7f8f067fc640 1 -- 172.21.15.59:0/3815652858 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00039070 con 0x7f8f0806c280 2024-03-06T07:22:48.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.323+0000 7f8f067fc640 1 -- 172.21.15.59:0/3815652858 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0002cc60 con 0x7f8f0806c280 2024-03-06T07:22:48.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.323+0000 7f8f067fc640 1 -- 172.21.15.59:0/3815652858 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00034660 con 0x7f8f0806c280 2024-03-06T07:22:48.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.324+0000 7f8f0d85c640 1 -- 172.21.15.59:0/3815652858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f0806c280 msgr2=0x7f8f080cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.324+0000 7f8f0d85c640 1 --2- 172.21.15.59:0/3815652858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f0806c280 0x7f8f080cfb70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009d90 tx=0x7f8f0002beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.324+0000 7f8f0d85c640 1 -- 172.21.15.59:0/3815652858 shutdown_connections 2024-03-06T07:22:48.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.324+0000 7f8f0d85c640 1 --2- 172.21.15.59:0/3815652858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f0806c280 0x7f8f080cfb70 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.324+0000 7f8f0d85c640 1 -- 172.21.15.59:0/3815652858 >> 172.21.15.59:0/3815652858 conn(0x7f8f080724d0 msgr2=0x7f8f080728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.324+0000 7f8f0d85c640 1 -- 172.21.15.59:0/3815652858 shutdown_connections 2024-03-06T07:22:48.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.324+0000 7f8f0d85c640 1 -- 172.21.15.59:0/3815652858 wait complete. 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.325+0000 7f8f0d85c640 1 Processor -- start 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.325+0000 7f8f0d85c640 1 -- start start 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f0d85c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f0806c280 0x7f8f08152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f0d85c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f080730d0 0x7f8f081524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f0d85c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f080cf470 0x7f8f080c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f0d85c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080b1330 con 0x7f8f0806c280 2024-03-06T07:22:48.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f0d85c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080b54a0 con 0x7f8f080cf470 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f0d85c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080b14b0 con 0x7f8f080730d0 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f06ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f0806c280 0x7f8f08152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f06ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f0806c280 0x7f8f08152bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36136/0 (socket says 172.21.15.59:36136) 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f06ffd640 1 -- 172.21.15.59:0/1807817939 learned_addr learned my addr 172.21.15.59:0/1807817939 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f077fe640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f080cf470 0x7f8f080c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8efffff640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f080730d0 0x7f8f081524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f06ffd640 1 -- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f080730d0 msgr2=0x7f8f081524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f06ffd640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f080730d0 0x7f8f081524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f06ffd640 1 -- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f080cf470 msgr2=0x7f8f080c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f06ffd640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f080cf470 0x7f8f080c9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f06ffd640 1 -- 172.21.15.59:0/1807817939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f080ade70 con 0x7f8f0806c280 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f077fe640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f080cf470 0x7f8f080c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f06ffd640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f0806c280 0x7f8f08152bf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800c460 tx=0x7f8ef800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f04ff9640 1 -- 172.21.15.59:0/1807817939 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8018070 con 0x7f8f0806c280 2024-03-06T07:22:48.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f04ff9640 1 -- 172.21.15.59:0/1807817939 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef8007500 con 0x7f8f0806c280 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f0d85c640 1 -- 172.21.15.59:0/1807817939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0815cec0 con 0x7f8f0806c280 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f04ff9640 1 -- 172.21.15.59:0/1807817939 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8004a20 con 0x7f8f0806c280 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.326+0000 7f8f0d85c640 1 -- 172.21.15.59:0/1807817939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f080ada90 con 0x7f8f0806c280 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.327+0000 7f8f0d85c640 1 -- 172.21.15.59:0/1807817939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0804f330 con 0x7f8f0806c280 2024-03-06T07:22:48.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.328+0000 7f8f04ff9640 1 -- 172.21.15.59:0/1807817939 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ef801b070 con 0x7f8f0806c280 2024-03-06T07:22:48.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.328+0000 7f8f04ff9640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8ee806b4d0 0x7f8ee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.328+0000 7f8f04ff9640 1 -- 172.21.15.59:0/1807817939 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8ef8089060 con 0x7f8f0806c280 2024-03-06T07:22:48.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.328+0000 7f8efffff640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8ee806b4d0 0x7f8ee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.329+0000 7f8efffff640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8ee806b4d0 0x7f8ee806d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0007fd0 tx=0x7f8ef0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.332+0000 7f8f04ff9640 1 -- 172.21.15.59:0/1807817939 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ef80582c0 con 0x7f8f0806c280 2024-03-06T07:22:48.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.516+0000 7f8f0d85c640 1 -- 172.21.15.59:0/1807817939 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8f080713c0 con 0x7f8f0806c280 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.516+0000 7f8f04ff9640 1 -- 172.21.15.59:0/1807817939 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8ef8057bc0 con 0x7f8f0806c280 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:48.518 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-03-06T07:22:48.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 -- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8ee806b4d0 msgr2=0x7f8ee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8ee806b4d0 0x7f8ee806d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0007fd0 tx=0x7f8ef0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:48.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 -- 172.21.15.59:0/1807817939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f0806c280 msgr2=0x7f8f08152bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f0806c280 0x7f8f08152bf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800c460 tx=0x7f8ef800c920 comp rx=0 tx=0).stop 2024-03-06T07:22:48.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 -- 172.21.15.59:0/1807817939 shutdown_connections 2024-03-06T07:22:48.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8ee806b4d0 0x7f8ee806d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f080cf470 0x7f8f080c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f080730d0 0x7f8f081524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 --2- 172.21.15.59:0/1807817939 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f0806c280 0x7f8f08152bf0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 -- 172.21.15.59:0/1807817939 >> 172.21.15.59:0/1807817939 conn(0x7f8f080724d0 msgr2=0x7f8f080af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 -- 172.21.15.59:0/1807817939 shutdown_connections 2024-03-06T07:22:48.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.518+0000 7f8f0d85c640 1 -- 172.21.15.59:0/1807817939 wait complete. 2024-03-06T07:22:48.530 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:22:48.531 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.634+0000 7fedea1a0640 1 Processor -- start 2024-03-06T07:22:48.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.634+0000 7fedea1a0640 1 -- start start 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fedea1a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede415e0a0 0x7fede40ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fedea1a0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fede415efa0 0x7fede40d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fedea1a0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede40cf280 con 0x7fede40d0be0 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fedea1a0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede40cf560 con 0x7fede415efa0 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fedea1a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede40cf840 con 0x7fede415e0a0 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fede899d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fede415efa0 0x7fede40d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fede999f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fede40d0be0 0x7fede4161a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46946/0 (socket says 172.21.15.59:46946) 2024-03-06T07:22:48.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fede999f640 1 -- 172.21.15.59:0/203353078 learned_addr learned my addr 172.21.15.59:0/203353078 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:48.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fede919e640 1 --2- 172.21.15.59:0/203353078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede415e0a0 0x7fede40ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fede899d640 1 -- 172.21.15.59:0/203353078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede415e0a0 msgr2=0x7fede40ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fede899d640 1 --2- 172.21.15.59:0/203353078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede415e0a0 0x7fede40ceb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fede899d640 1 -- 172.21.15.59:0/203353078 >> v1:172.21.15.59:6789/0 conn(0x7fede40d0be0 legacy=0x7fede4161a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fede899d640 1 -- 172.21.15.59:0/203353078 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede40cfb20 con 0x7fede415efa0 2024-03-06T07:22:48.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.635+0000 7fede3fff640 1 -- 172.21.15.59:0/203353078 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311223880 0 0) 0x7fede40cf280 con 0x7fede40d0be0 2024-03-06T07:22:48.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.636+0000 7fede899d640 1 --2- 172.21.15.59:0/203353078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fede415efa0 0x7fede40d06a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fedd80097a0 tx=0x7fedd802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401fdf6b588b9e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.636+0000 7fede3fff640 1 -- 172.21.15.59:0/203353078 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd80315a0 con 0x7fede415efa0 2024-03-06T07:22:48.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.636+0000 7fede3fff640 1 -- 172.21.15.59:0/203353078 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd8037070 con 0x7fede415efa0 2024-03-06T07:22:48.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.636+0000 7fede3fff640 1 -- 172.21.15.59:0/203353078 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8031960 con 0x7fede415efa0 2024-03-06T07:22:48.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.637+0000 7fedea1a0640 1 -- 172.21.15.59:0/203353078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fede415efa0 msgr2=0x7fede40d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.637+0000 7fedea1a0640 1 --2- 172.21.15.59:0/203353078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fede415efa0 0x7fede40d06a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fedd80097a0 tx=0x7fedd802b840 comp rx=0 tx=0).stop 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.637+0000 7fedea1a0640 1 -- 172.21.15.59:0/203353078 shutdown_connections 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.637+0000 7fedea1a0640 1 --2- 172.21.15.59:0/203353078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fede415efa0 0x7fede40d06a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.637+0000 7fedea1a0640 1 --2- 172.21.15.59:0/203353078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede415e0a0 0x7fede40ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.637+0000 7fedea1a0640 1 -- 172.21.15.59:0/203353078 >> 172.21.15.59:0/203353078 conn(0x7fede40d3c90 msgr2=0x7fede40d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.637+0000 7fedea1a0640 1 -- 172.21.15.59:0/203353078 shutdown_connections 2024-03-06T07:22:48.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.637+0000 7fedea1a0640 1 -- 172.21.15.59:0/203353078 wait complete. 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.638+0000 7fedea1a0640 1 Processor -- start 2024-03-06T07:22:48.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.638+0000 7fedea1a0640 1 -- start start 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fedea1a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede40d0be0 0x7fede40c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fedea1a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fede415e0a0 0x7fede4150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fedea1a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fede415efa0 0x7fede41f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fedea1a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede40d29f0 con 0x7fede415e0a0 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fedea1a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede40d2870 con 0x7fede415efa0 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fedea1a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede40d2b70 con 0x7fede40d0be0 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede899d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fede415e0a0 0x7fede4150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede899d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fede415e0a0 0x7fede4150fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36152/0 (socket says 172.21.15.59:36152) 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede899d640 1 -- 172.21.15.59:0/2670428383 learned_addr learned my addr 172.21.15.59:0/2670428383 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede919e640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede40d0be0 0x7fede40c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede899d640 1 -- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede40d0be0 msgr2=0x7fede40c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede999f640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fede415efa0 0x7fede41f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede899d640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede40d0be0 0x7fede40c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede899d640 1 -- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fede415efa0 msgr2=0x7fede41f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede899d640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fede415efa0 0x7fede41f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede899d640 1 -- 172.21.15.59:0/2670428383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede40ca4d0 con 0x7fede415e0a0 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede919e640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede40d0be0 0x7fede40c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede999f640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fede415efa0 0x7fede41f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.639+0000 7fede899d640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fede415e0a0 0x7fede4150fd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fedd80097a0 tx=0x7fedd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.640+0000 7fede1ffb640 1 -- 172.21.15.59:0/2670428383 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8004230 con 0x7fede415e0a0 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.640+0000 7fede1ffb640 1 -- 172.21.15.59:0/2670428383 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd80043d0 con 0x7fede415e0a0 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.640+0000 7fede1ffb640 1 -- 172.21.15.59:0/2670428383 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8003850 con 0x7fede415e0a0 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.640+0000 7fedea1a0640 1 -- 172.21.15.59:0/2670428383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede40ca770 con 0x7fede415e0a0 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.640+0000 7fedea1a0640 1 -- 172.21.15.59:0/2670428383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede41612a0 con 0x7fede415e0a0 2024-03-06T07:22:48.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.641+0000 7fedea1a0640 1 -- 172.21.15.59:0/2670428383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede4068f10 con 0x7fede415e0a0 2024-03-06T07:22:48.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.641+0000 7fede1ffb640 1 -- 172.21.15.59:0/2670428383 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedd80039f0 con 0x7fede415e0a0 2024-03-06T07:22:48.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.642+0000 7fede1ffb640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fedd006b4d0 0x7fedd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.642+0000 7fede1ffb640 1 -- 172.21.15.59:0/2670428383 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fedd802f030 con 0x7fede415e0a0 2024-03-06T07:22:48.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.642+0000 7fede919e640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fedd006b4d0 0x7fedd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.642+0000 7fede919e640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fedd006b4d0 0x7fedd006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7feddc006fd0 tx=0x7feddc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.646+0000 7fede1ffb640 1 -- 172.21.15.59:0/2670428383 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedd80b1020 con 0x7fede415e0a0 2024-03-06T07:22:48.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:48.829+0000 7fedea1a0640 1 -- 172.21.15.59:0/2670428383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fede408dcc0 con 0x7fede415e0a0 2024-03-06T07:22:49.026 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:49.029 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:49.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.207+0000 7fede1ffb640 1 -- 172.21.15.59:0/2670428383 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7fedd8041070 con 0x7fede415e0a0 2024-03-06T07:22:49.209 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-03-06T07:22:49.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.210+0000 7fedea1a0640 1 -- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fedd006b4d0 msgr2=0x7fedd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.210+0000 7fedea1a0640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fedd006b4d0 0x7fedd006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7feddc006fd0 tx=0x7feddc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.210+0000 7fedea1a0640 1 -- 172.21.15.59:0/2670428383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fede415e0a0 msgr2=0x7fede4150fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.210+0000 7fedea1a0640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fede415e0a0 0x7fede4150fd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fedd80097a0 tx=0x7fedd80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.211+0000 7fedea1a0640 1 -- 172.21.15.59:0/2670428383 shutdown_connections 2024-03-06T07:22:49.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.211+0000 7fedea1a0640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fedd006b4d0 0x7fedd006d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.211+0000 7fedea1a0640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fede415efa0 0x7fede41f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.211+0000 7fedea1a0640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fede415e0a0 0x7fede4150fd0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.211+0000 7fedea1a0640 1 --2- 172.21.15.59:0/2670428383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fede40d0be0 0x7fede40c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.211+0000 7fedea1a0640 1 -- 172.21.15.59:0/2670428383 >> 172.21.15.59:0/2670428383 conn(0x7fede40d3c90 msgr2=0x7fede406be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.212+0000 7fedea1a0640 1 -- 172.21.15.59:0/2670428383 shutdown_connections 2024-03-06T07:22:49.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.212+0000 7fedea1a0640 1 -- 172.21.15.59:0/2670428383 wait complete. 2024-03-06T07:22:49.225 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:22:49.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.332+0000 7fd640439640 1 Processor -- start 2024-03-06T07:22:49.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.333+0000 7fd640439640 1 -- start start 2024-03-06T07:22:49.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.333+0000 7fd640439640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd63815bb40 0x7fd6380b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.333+0000 7fd640439640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6380cfa90 0x7fd6380cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.333+0000 7fd640439640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63815ff90 con 0x7fd63815f7d0 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.333+0000 7fd640439640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63806c910 con 0x7fd63815bb40 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.333+0000 7fd640439640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd638160270 con 0x7fd6380cfa90 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.334+0000 7fd63e1d5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd63815bb40 0x7fd6380b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.334+0000 7fd63d9d4640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd63815f7d0 0x7fd63806d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46954/0 (socket says 172.21.15.59:46954) 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.334+0000 7fd63e1d5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd63815bb40 0x7fd6380b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36160/0 (socket says 172.21.15.59:36160) 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.334+0000 7fd63d9d4640 1 -- 172.21.15.59:0/2938747903 learned_addr learned my addr 172.21.15.59:0/2938747903 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.334+0000 7fd63e9d6640 1 --2- 172.21.15.59:0/2938747903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6380cfa90 0x7fd6380cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.334+0000 7fd63e1d5640 1 -- 172.21.15.59:0/2938747903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6380cfa90 msgr2=0x7fd6380cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.334+0000 7fd63e1d5640 1 --2- 172.21.15.59:0/2938747903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6380cfa90 0x7fd6380cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.334+0000 7fd63e1d5640 1 -- 172.21.15.59:0/2938747903 >> v1:172.21.15.59:6789/0 conn(0x7fd63815f7d0 legacy=0x7fd63806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.334+0000 7fd63e1d5640 1 -- 172.21.15.59:0/2938747903 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd638160550 con 0x7fd63815bb40 2024-03-06T07:22:49.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.334+0000 7fd63d1d3640 1 -- 172.21.15.59:0/2938747903 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337131935 0 0) 0x7fd63815ff90 con 0x7fd63815f7d0 2024-03-06T07:22:49.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.334+0000 7fd63e9d6640 1 --2- 172.21.15.59:0/2938747903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6380cfa90 0x7fd6380cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:49.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd63e1d5640 1 --2- 172.21.15.59:0/2938747903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd63815bb40 0x7fd6380b4970 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0097a0 tx=0x7fd62c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86b83bdd3821fe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd63d1d3640 1 -- 172.21.15.59:0/2938747903 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c035020 con 0x7fd63815bb40 2024-03-06T07:22:49.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd63d1d3640 1 -- 172.21.15.59:0/2938747903 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c031830 con 0x7fd63815bb40 2024-03-06T07:22:49.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd63d1d3640 1 -- 172.21.15.59:0/2938747903 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c031c00 con 0x7fd63815bb40 2024-03-06T07:22:49.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd640439640 1 -- 172.21.15.59:0/2938747903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd63815bb40 msgr2=0x7fd6380b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd640439640 1 --2- 172.21.15.59:0/2938747903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd63815bb40 0x7fd6380b4970 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0097a0 tx=0x7fd62c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd640439640 1 -- 172.21.15.59:0/2938747903 shutdown_connections 2024-03-06T07:22:49.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd640439640 1 --2- 172.21.15.59:0/2938747903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6380cfa90 0x7fd6380cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd640439640 1 --2- 172.21.15.59:0/2938747903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd63815bb40 0x7fd6380b4970 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd640439640 1 -- 172.21.15.59:0/2938747903 >> 172.21.15.59:0/2938747903 conn(0x7fd63806b190 msgr2=0x7fd63806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd640439640 1 -- 172.21.15.59:0/2938747903 shutdown_connections 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.335+0000 7fd640439640 1 -- 172.21.15.59:0/2938747903 wait complete. 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.337+0000 7fd640439640 1 Processor -- start 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.337+0000 7fd640439640 1 -- start start 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.337+0000 7fd640439640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6380cfa90 0x7fd6381443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.337+0000 7fd640439640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd63815bb40 0x7fd638156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.337+0000 7fd640439640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd63815f7d0 0x7fd638157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.337+0000 7fd640439640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63806d790 con 0x7fd6380cfa90 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.337+0000 7fd640439640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63815fc30 con 0x7fd63815bb40 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.337+0000 7fd640439640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63815bf20 con 0x7fd63815f7d0 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.337+0000 7fd63e1d5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6380cfa90 0x7fd6381443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.337+0000 7fd63e1d5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6380cfa90 0x7fd6381443f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36170/0 (socket says 172.21.15.59:36170) 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.337+0000 7fd63e1d5640 1 -- 172.21.15.59:0/3928772563 learned_addr learned my addr 172.21.15.59:0/3928772563 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:49.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.338+0000 7fd63e1d5640 1 -- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd63815f7d0 msgr2=0x7fd638157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.338+0000 7fd63e1d5640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd63815f7d0 0x7fd638157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.338+0000 7fd63e1d5640 1 -- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd63815bb40 msgr2=0x7fd638156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.338+0000 7fd63e1d5640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd63815bb40 0x7fd638156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.338+0000 7fd63e1d5640 1 -- 172.21.15.59:0/3928772563 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd638144f70 con 0x7fd6380cfa90 2024-03-06T07:22:49.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.338+0000 7fd63e1d5640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6380cfa90 0x7fd6381443f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd62c02bee0 tx=0x7fd62c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.338+0000 7fd62affd640 1 -- 172.21.15.59:0/3928772563 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c041070 con 0x7fd6380cfa90 2024-03-06T07:22:49.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.338+0000 7fd62affd640 1 -- 172.21.15.59:0/3928772563 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c03a970 con 0x7fd6380cfa90 2024-03-06T07:22:49.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.338+0000 7fd640439640 1 -- 172.21.15.59:0/3928772563 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd638145250 con 0x7fd6380cfa90 2024-03-06T07:22:49.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.338+0000 7fd62affd640 1 -- 172.21.15.59:0/3928772563 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c035040 con 0x7fd6380cfa90 2024-03-06T07:22:49.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.338+0000 7fd640439640 1 -- 172.21.15.59:0/3928772563 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd638145810 con 0x7fd6380cfa90 2024-03-06T07:22:49.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.339+0000 7fd640439640 1 -- 172.21.15.59:0/3928772563 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6380732b0 con 0x7fd6380cfa90 2024-03-06T07:22:49.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.340+0000 7fd62affd640 1 -- 172.21.15.59:0/3928772563 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd62c04c060 con 0x7fd6380cfa90 2024-03-06T07:22:49.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.340+0000 7fd62affd640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd61c06b500 0x7fd61c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.340+0000 7fd62affd640 1 -- 172.21.15.59:0/3928772563 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd62c0abe20 con 0x7fd6380cfa90 2024-03-06T07:22:49.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.340+0000 7fd63d9d4640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd61c06b500 0x7fd61c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.341+0000 7fd63d9d4640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd61c06b500 0x7fd61c06d9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd6340011e0 tx=0x7fd634007940 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.344+0000 7fd62affd640 1 -- 172.21.15.59:0/3928772563 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd62c07af60 con 0x7fd6380cfa90 2024-03-06T07:22:49.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:49.526+0000 7fd640439640 1 -- 172.21.15.59:0/3928772563 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd638068ff0 con 0x7fd6380cfa90 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.212+0000 7fd62affd640 1 -- 172.21.15.59:0/3928772563 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd62c07a860 con 0x7fd6380cfa90 2024-03-06T07:22:50.214 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-03-06T07:22:50.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.216+0000 7fd640439640 1 -- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd61c06b500 msgr2=0x7fd61c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.216+0000 7fd640439640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd61c06b500 0x7fd61c06d9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd6340011e0 tx=0x7fd634007940 comp rx=0 tx=0).stop 2024-03-06T07:22:50.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.216+0000 7fd640439640 1 -- 172.21.15.59:0/3928772563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6380cfa90 msgr2=0x7fd6381443f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.216+0000 7fd640439640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6380cfa90 0x7fd6381443f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd62c02bee0 tx=0x7fd62c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.216+0000 7fd640439640 1 -- 172.21.15.59:0/3928772563 shutdown_connections 2024-03-06T07:22:50.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.216+0000 7fd640439640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd61c06b500 0x7fd61c06d9c0 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd6340011e0 tx=0x7fd634007940 comp rx=0 tx=0).stop 2024-03-06T07:22:50.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.216+0000 7fd640439640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd63815f7d0 0x7fd638157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.216+0000 7fd640439640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd63815bb40 0x7fd638156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.216+0000 7fd640439640 1 --2- 172.21.15.59:0/3928772563 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6380cfa90 0x7fd6381443f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.216+0000 7fd640439640 1 -- 172.21.15.59:0/3928772563 >> 172.21.15.59:0/3928772563 conn(0x7fd63806b190 msgr2=0x7fd6380b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.225+0000 7fd640439640 1 -- 172.21.15.59:0/3928772563 shutdown_connections 2024-03-06T07:22:50.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.226+0000 7fd640439640 1 -- 172.21.15.59:0/3928772563 wait complete. 2024-03-06T07:22:50.246 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:22:50.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.352+0000 7f6c13f01640 1 Processor -- start 2024-03-06T07:22:50.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.353+0000 7f6c13f01640 1 -- start start 2024-03-06T07:22:50.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.353+0000 7f6c13f01640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c0c0b2eb0 0x7f6c0c0d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.353+0000 7f6c13f01640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6c0c0d1cd0 0x7f6c0c0cce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.353+0000 7f6c13f01640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c15c2b0 con 0x7f6c0c15bed0 2024-03-06T07:22:50.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.353+0000 7f6c13f01640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0c0cca00 con 0x7f6c0c0d1cd0 2024-03-06T07:22:50.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.353+0000 7f6c13f01640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c0d00d0 con 0x7f6c0c0b2eb0 2024-03-06T07:22:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.354+0000 7f6c1249e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6c0c0d1cd0 0x7f6c0c0cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.354+0000 7f6c1149c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c0c0b2eb0 0x7f6c0c0d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.354+0000 7f6c11c9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6c0c15bed0 0x7f6c0c0cf9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:37954/0 (socket says 172.21.15.59:37954) 2024-03-06T07:22:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.354+0000 7f6c11c9d640 1 -- 172.21.15.59:0/3706605967 learned_addr learned my addr 172.21.15.59:0/3706605967 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.354+0000 7f6c10c9b640 1 -- 172.21.15.59:0/3706605967 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822475076 0 0) 0x7f6c0c15c2b0 con 0x7f6c0c15bed0 2024-03-06T07:22:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.354+0000 7f6c10c9b640 1 -- 172.21.15.59:0/3706605967 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf8003520 con 0x7f6c0c15bed0 2024-03-06T07:22:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.354+0000 7f6c1149c640 1 -- 172.21.15.59:0/3706605967 >> v2:172.21.15.164:3300/0 conn(0x7f6c0c0d1cd0 msgr2=0x7f6c0c0cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.354+0000 7f6c1149c640 1 --2- 172.21.15.59:0/3706605967 >> v2:172.21.15.164:3300/0 conn(0x7f6c0c0d1cd0 0x7f6c0c0cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.354+0000 7f6c1149c640 1 -- 172.21.15.59:0/3706605967 >> v1:172.21.15.164:6789/0 conn(0x7f6c0c15bed0 legacy=0x7f6c0c0cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.354+0000 7f6c1149c640 1 -- 172.21.15.59:0/3706605967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c0d2160 con 0x7f6c0c0b2eb0 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.354+0000 7f6c1249e640 1 --2- 172.21.15.59:0/3706605967 >> v2:172.21.15.164:3300/0 conn(0x7f6c0c0d1cd0 0x7f6c0c0cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.355+0000 7f6c1149c640 1 --2- 172.21.15.59:0/3706605967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c0c0b2eb0 0x7f6c0c0d1790 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0097d0 tx=0x7f6bfc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb039cb848413863 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.355+0000 7f6c10c9b640 1 -- 172.21.15.59:0/3706605967 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc035020 con 0x7f6c0c0b2eb0 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.355+0000 7f6c10c9b640 1 -- 172.21.15.59:0/3706605967 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc031800 con 0x7f6c0c0b2eb0 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.355+0000 7f6c10c9b640 1 -- 172.21.15.59:0/3706605967 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc031bb0 con 0x7f6c0c0b2eb0 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.356+0000 7f6c13f01640 1 -- 172.21.15.59:0/3706605967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c0c0b2eb0 msgr2=0x7f6c0c0d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.356+0000 7f6c13f01640 1 --2- 172.21.15.59:0/3706605967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c0c0b2eb0 0x7f6c0c0d1790 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0097d0 tx=0x7f6bfc02b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.356+0000 7f6c13f01640 1 -- 172.21.15.59:0/3706605967 shutdown_connections 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.356+0000 7f6c13f01640 1 --2- 172.21.15.59:0/3706605967 >> v2:172.21.15.164:3300/0 conn(0x7f6c0c0d1cd0 0x7f6c0c0cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.356+0000 7f6c13f01640 1 --2- 172.21.15.59:0/3706605967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c0c0b2eb0 0x7f6c0c0d1790 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.356+0000 7f6c13f01640 1 -- 172.21.15.59:0/3706605967 >> 172.21.15.59:0/3706605967 conn(0x7f6c0c09a9f0 msgr2=0x7f6c0c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.356+0000 7f6c13f01640 1 -- 172.21.15.59:0/3706605967 shutdown_connections 2024-03-06T07:22:50.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.356+0000 7f6c13f01640 1 -- 172.21.15.59:0/3706605967 wait complete. 2024-03-06T07:22:50.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.357+0000 7f6c13f01640 1 Processor -- start 2024-03-06T07:22:50.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.357+0000 7f6c13f01640 1 -- start start 2024-03-06T07:22:50.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c13f01640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c0c0d1cd0 0x7f6c0c149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c13f01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c0c15bed0 0x7f6c0c0ce830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c13f01640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c0c0ced70 0x7f6c0c1f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c13f01640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c0d08f0 con 0x7f6c0c15bed0 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c13f01640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c0d0610 con 0x7f6c0c0d1cd0 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c13f01640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c0cc5d0 con 0x7f6c0c0ced70 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c1149c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c0c15bed0 0x7f6c0c0ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c1149c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c0c15bed0 0x7f6c0c0ce830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36172/0 (socket says 172.21.15.59:36172) 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c1149c640 1 -- 172.21.15.59:0/3864358819 learned_addr learned my addr 172.21.15.59:0/3864358819 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c1249e640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c0c0ced70 0x7f6c0c1f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c11c9d640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c0c0d1cd0 0x7f6c0c149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c1149c640 1 -- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c0c0ced70 msgr2=0x7f6c0c1f73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c1149c640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c0c0ced70 0x7f6c0c1f73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c1149c640 1 -- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c0c0d1cd0 msgr2=0x7f6c0c149db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c1149c640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c0c0d1cd0 0x7f6c0c149db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c1149c640 1 -- 172.21.15.59:0/3864358819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c15d540 con 0x7f6c0c15bed0 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.358+0000 7f6c1149c640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c0c15bed0 0x7f6c0c0ce830 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0098f0 tx=0x7f6bfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.359+0000 7f6c027fc640 1 -- 172.21.15.59:0/3864358819 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc035040 con 0x7f6c0c15bed0 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.359+0000 7f6c027fc640 1 -- 172.21.15.59:0/3864358819 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc032e40 con 0x7f6c0c15bed0 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.359+0000 7f6c13f01640 1 -- 172.21.15.59:0/3864358819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0c15c4a0 con 0x7f6c0c15bed0 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.359+0000 7f6c027fc640 1 -- 172.21.15.59:0/3864358819 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc003a20 con 0x7f6c0c15bed0 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.359+0000 7f6c13f01640 1 -- 172.21.15.59:0/3864358819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c15ca60 con 0x7f6c0c15bed0 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.360+0000 7f6c13f01640 1 -- 172.21.15.59:0/3864358819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0c0cf180 con 0x7f6c0c15bed0 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.360+0000 7f6c027fc640 1 -- 172.21.15.59:0/3864358819 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bfc038020 con 0x7f6c0c15bed0 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.360+0000 7f6c027fc640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bf006b4d0 0x7f6bf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.361+0000 7f6c027fc640 1 -- 172.21.15.59:0/3864358819 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f6bfc0ac370 con 0x7f6c0c15bed0 2024-03-06T07:22:50.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.361+0000 7f6c11c9d640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bf006b4d0 0x7f6bf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.361+0000 7f6c11c9d640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bf006b4d0 0x7f6bf006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6c08005880 tx=0x7f6c08009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.365+0000 7f6c027fc640 1 -- 172.21.15.59:0/3864358819 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c0c0cf180 con 0x7f6c0c15bed0 2024-03-06T07:22:50.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:50.584+0000 7f6c13f01640 1 -- 172.21.15.59:0/3864358819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6c0c04f330 con 0x7f6c0c15bed0 2024-03-06T07:22:51.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.214+0000 7f6c027fc640 1 -- 172.21.15.59:0/3864358819 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v11) v1 ==== 172+0+0 (secure 0 0 0) 0x7f6bfc07b1c0 con 0x7f6c0c15bed0 2024-03-06T07:22:51.216 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 4 and data pool 5 2024-03-06T07:22:51.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.223+0000 7f6c13f01640 1 -- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bf006b4d0 msgr2=0x7f6bf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.223+0000 7f6c13f01640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bf006b4d0 0x7f6bf006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6c08005880 tx=0x7f6c08009040 comp rx=0 tx=0).stop 2024-03-06T07:22:51.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.223+0000 7f6c13f01640 1 -- 172.21.15.59:0/3864358819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c0c15bed0 msgr2=0x7f6c0c0ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.223+0000 7f6c13f01640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c0c15bed0 0x7f6c0c0ce830 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0098f0 tx=0x7f6bfc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.226+0000 7f6c13f01640 1 -- 172.21.15.59:0/3864358819 shutdown_connections 2024-03-06T07:22:51.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.226+0000 7f6c13f01640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6bf006b4d0 0x7f6bf006d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.226+0000 7f6c13f01640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c0c0ced70 0x7f6c0c1f73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.226+0000 7f6c13f01640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c0c15bed0 0x7f6c0c0ce830 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.226+0000 7f6c13f01640 1 --2- 172.21.15.59:0/3864358819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c0c0d1cd0 0x7f6c0c149db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.226+0000 7f6c13f01640 1 -- 172.21.15.59:0/3864358819 >> 172.21.15.59:0/3864358819 conn(0x7f6c0c09a9f0 msgr2=0x7f6c0c06d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.228+0000 7f6c13f01640 1 -- 172.21.15.59:0/3864358819 shutdown_connections 2024-03-06T07:22:51.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.232+0000 7f6c13f01640 1 -- 172.21.15.59:0/3864358819 wait complete. 2024-03-06T07:22:51.259 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:51.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.363+0000 7f218e283640 1 Processor -- start 2024-03-06T07:22:51.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.363+0000 7f218e283640 1 -- start start 2024-03-06T07:22:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.364+0000 7f218e283640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f218815e0f0 0x7f21880cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.364+0000 7f218e283640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218815e4d0 con 0x7f21880d0660 2024-03-06T07:22:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.364+0000 7f218e283640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21880cdac0 con 0x7f21880cf850 2024-03-06T07:22:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.364+0000 7f218e283640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21880b1410 con 0x7f218815e0f0 2024-03-06T07:22:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.364+0000 7f218c820640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f21880d0660 0x7f21880b25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46970/0 (socket says 172.21.15.59:46970) 2024-03-06T07:22:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.364+0000 7f218c820640 1 -- 172.21.15.59:0/2915051345 learned_addr learned my addr 172.21.15.59:0/2915051345 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.364+0000 7f2187fff640 1 --2- 172.21.15.59:0/2915051345 >> v2:172.21.15.164:3300/0 conn(0x7f218815e0f0 0x7f21880cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.364+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242915380 0 0) 0x7f218815e4d0 con 0x7f21880d0660 2024-03-06T07:22:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.364+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2170003520 con 0x7f21880d0660 2024-03-06T07:22:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 639224322 0 0) 0x7f2170003520 con 0x7f21880d0660 2024-03-06T07:22:51.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 >> v2:172.21.15.164:3300/0 conn(0x7f218815e0f0 msgr2=0x7f21880cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 --2- 172.21.15.59:0/2915051345 >> v2:172.21.15.164:3300/0 conn(0x7f218815e0f0 0x7f21880cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 >> v1:172.21.15.164:6789/0 conn(0x7f21880cf850 legacy=0x7f21880cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218806c260 con 0x7f21880d0660 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f21800019f0 con 0x7f21880d0660 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 >> v1:172.21.15.59:6789/0 conn(0x7f21880d0660 legacy=0x7f21880b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21700039b0 0x7f2170003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21700043b0 0x7f2170004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2170005080 0x7f217000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217000bfb0 con 0x7f2170005080 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217000c160 con 0x7f21700039b0 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217000c360 con 0x7f21700043b0 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f218c820640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2170005080 0x7f217000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2187fff640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21700039b0 0x7f2170003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f21877fe640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21700043b0 0x7f2170004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2187fff640 1 -- 172.21.15.59:0/2915051345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21700043b0 msgr2=0x7f2170004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2187fff640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21700043b0 0x7f2170004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2187fff640 1 -- 172.21.15.59:0/2915051345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2170005080 msgr2=0x7f217000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2187fff640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2170005080 0x7f217000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f2187fff640 1 -- 172.21.15.59:0/2915051345 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2170003520 con 0x7f21700039b0 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f218c820640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2170005080 0x7f217000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.365+0000 7f218c820640 1 -- 172.21.15.59:0/2915051345 reap_dead start 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.366+0000 7f2187fff640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21700039b0 0x7f2170003d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f217c009ac0 tx=0x7f2170005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93a4bd547a835aea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.366+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c039070 con 0x7f21700039b0 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.366+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217c02c6b0 con 0x7f21700039b0 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.366+0000 7f2186ffd640 1 -- 172.21.15.59:0/2915051345 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c009690 con 0x7f21700039b0 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.367+0000 7f218e283640 1 -- 172.21.15.59:0/2915051345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21700039b0 msgr2=0x7f2170003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.367+0000 7f218e283640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21700039b0 0x7f2170003d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f217c009ac0 tx=0x7f2170005830 comp rx=0 tx=0).stop 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.367+0000 7f218e283640 1 -- 172.21.15.59:0/2915051345 shutdown_connections 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.367+0000 7f218e283640 1 --2- 172.21.15.59:0/2915051345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21700039b0 0x7f2170003d90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.367+0000 7f218e283640 1 -- 172.21.15.59:0/2915051345 >> 172.21.15.59:0/2915051345 conn(0x7f21880b35a0 msgr2=0x7f21880b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.367+0000 7f218e283640 1 -- 172.21.15.59:0/2915051345 shutdown_connections 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.367+0000 7f218e283640 1 -- 172.21.15.59:0/2915051345 wait complete. 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.368+0000 7f218e283640 1 Processor -- start 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.368+0000 7f218e283640 1 -- start start 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218e283640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21700039b0 0x7f21880cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218e283640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21880ca140 0x7f21880ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218e283640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21881f6f00 0x7f21881f7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218e283640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21880d0cc0 con 0x7f21881f6f00 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218e283640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218806c8b0 con 0x7f21880ca140 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218e283640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21880cd3c0 con 0x7f21700039b0 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218c820640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21881f6f00 0x7f21881f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218c820640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21881f6f00 0x7f21881f7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36194/0 (socket says 172.21.15.59:36194) 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218c820640 1 -- 172.21.15.59:0/3193252339 learned_addr learned my addr 172.21.15.59:0/3193252339 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f21877fe640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21880ca140 0x7f21880ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f2187fff640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21700039b0 0x7f21880cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218c820640 1 -- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21700039b0 msgr2=0x7f21880cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218c820640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21700039b0 0x7f21880cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218c820640 1 -- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21880ca140 msgr2=0x7f21880ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218c820640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21880ca140 0x7f21880ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218c820640 1 -- 172.21.15.59:0/3193252339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2188161110 con 0x7f21881f6f00 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f21877fe640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21880ca140 0x7f21880ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218c820640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21881f6f00 0x7f21881f7380 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2188074cb0 tx=0x7f2180007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f2184ff9640 1 -- 172.21.15.59:0/3193252339 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218000a4c0 con 0x7f21881f6f00 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f2184ff9640 1 -- 172.21.15.59:0/3193252339 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2180019070 con 0x7f21881f6f00 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.369+0000 7f218e283640 1 -- 172.21.15.59:0/3193252339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2188160500 con 0x7f21881f6f00 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.370+0000 7f2184ff9640 1 -- 172.21.15.59:0/3193252339 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21800133f0 con 0x7f21881f6f00 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.370+0000 7f218e283640 1 -- 172.21.15.59:0/3193252339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2188160ac0 con 0x7f21881f6f00 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.370+0000 7f218e283640 1 -- 172.21.15.59:0/3193252339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21880b0f30 con 0x7f21881f6f00 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.371+0000 7f2184ff9640 1 -- 172.21.15.59:0/3193252339 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21800137a0 con 0x7f21881f6f00 2024-03-06T07:22:51.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.371+0000 7f2184ff9640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f217406b4d0 0x7f217406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.371+0000 7f2184ff9640 1 -- 172.21.15.59:0/3193252339 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f218008a9a0 con 0x7f21881f6f00 2024-03-06T07:22:51.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.372+0000 7f2187fff640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f217406b4d0 0x7f217406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.372+0000 7f2187fff640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f217406b4d0 0x7f217406d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f217c036000 tx=0x7f217c030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.376+0000 7f2184ff9640 1 -- 172.21.15.59:0/3193252339 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21880b0f30 con 0x7f21881f6f00 2024-03-06T07:22:51.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.560+0000 7f218e283640 1 -- 172.21.15.59:0/3193252339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f218804f330 con 0x7f21881f6f00 2024-03-06T07:22:51.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.561+0000 7f2184ff9640 1 -- 172.21.15.59:0/3193252339 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16670 (secure 0 0 0) 0x7f2188161110 con 0x7f21881f6f00 2024-03-06T07:22:51.563 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:51.564 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":32,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:50.585565+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:48.848650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:49.545360+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.563+0000 7f218e283640 1 -- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f217406b4d0 msgr2=0x7f217406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.563+0000 7f218e283640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f217406b4d0 0x7f217406d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f217c036000 tx=0x7f217c030000 comp rx=0 tx=0).stop 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.563+0000 7f218e283640 1 -- 172.21.15.59:0/3193252339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21881f6f00 msgr2=0x7f21881f7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.563+0000 7f218e283640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21881f6f00 0x7f21881f7380 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2188074cb0 tx=0x7f2180007430 comp rx=0 tx=0).stop 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.564+0000 7f218e283640 1 -- 172.21.15.59:0/3193252339 shutdown_connections 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.564+0000 7f218e283640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f217406b4d0 0x7f217406d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.564+0000 7f218e283640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21881f6f00 0x7f21881f7380 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.564+0000 7f218e283640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21880ca140 0x7f21880ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.564+0000 7f218e283640 1 --2- 172.21.15.59:0/3193252339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21700039b0 0x7f21880cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.564+0000 7f218e283640 1 -- 172.21.15.59:0/3193252339 >> 172.21.15.59:0/3193252339 conn(0x7f21880b35a0 msgr2=0x7f21880b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.564+0000 7f218e283640 1 -- 172.21.15.59:0/3193252339 shutdown_connections 2024-03-06T07:22:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.564+0000 7f218e283640 1 -- 172.21.15.59:0/3193252339 wait complete. 2024-03-06T07:22:51.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:51.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.677+0000 7fb842084640 1 Processor -- start 2024-03-06T07:22:51.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.677+0000 7fb842084640 1 -- start start 2024-03-06T07:22:51.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb842084640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb83c0cf850 0x7fb83c0cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb842084640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb83c06cf20 0x7fb83c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb842084640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83c15e380 con 0x7fb83c0d00c0 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb842084640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83c06ca90 con 0x7fb83c06cf20 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb842084640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83c0cdc10 con 0x7fb83c0cf850 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb83bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb83c06cf20 0x7fb83c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb83bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb83c06cf20 0x7fb83c06d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36198/0 (socket says 172.21.15.59:36198) 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb83bfff640 1 -- 172.21.15.59:0/3311893925 learned_addr learned my addr 172.21.15.59:0/3311893925 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb83affd640 1 --2- 172.21.15.59:0/3311893925 >> v2:172.21.15.164:3300/0 conn(0x7fb83c0cf850 0x7fb83c0cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb83bfff640 1 -- 172.21.15.59:0/3311893925 >> v2:172.21.15.164:3300/0 conn(0x7fb83c0cf850 msgr2=0x7fb83c0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb83bfff640 1 --2- 172.21.15.59:0/3311893925 >> v2:172.21.15.164:3300/0 conn(0x7fb83c0cf850 0x7fb83c0cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb83bfff640 1 -- 172.21.15.59:0/3311893925 >> v1:172.21.15.164:6789/0 conn(0x7fb83c0d00c0 legacy=0x7fb83c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.678+0000 7fb83bfff640 1 -- 172.21.15.59:0/3311893925 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c06c160 con 0x7fb83c06cf20 2024-03-06T07:22:51.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.679+0000 7fb83bfff640 1 --2- 172.21.15.59:0/3311893925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb83c06cf20 0x7fb83c06d3b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb83c071de0 tx=0x7fb83402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b5dc26543bcf3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.679+0000 7fb83a7fc640 1 -- 172.21.15.59:0/3311893925 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83402cb20 con 0x7fb83c06cf20 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.679+0000 7fb83a7fc640 1 -- 172.21.15.59:0/3311893925 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83402ccc0 con 0x7fb83c06cf20 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.679+0000 7fb83a7fc640 1 -- 172.21.15.59:0/3311893925 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb834034610 con 0x7fb83c06cf20 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.680+0000 7fb842084640 1 -- 172.21.15.59:0/3311893925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb83c06cf20 msgr2=0x7fb83c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.680+0000 7fb842084640 1 --2- 172.21.15.59:0/3311893925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb83c06cf20 0x7fb83c06d3b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb83c071de0 tx=0x7fb83402bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.680+0000 7fb842084640 1 -- 172.21.15.59:0/3311893925 shutdown_connections 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.680+0000 7fb842084640 1 --2- 172.21.15.59:0/3311893925 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb83c06cf20 0x7fb83c06d3b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.680+0000 7fb842084640 1 --2- 172.21.15.59:0/3311893925 >> v2:172.21.15.164:3300/0 conn(0x7fb83c0cf850 0x7fb83c0cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.680+0000 7fb842084640 1 -- 172.21.15.59:0/3311893925 >> 172.21.15.59:0/3311893925 conn(0x7fb83c0b35a0 msgr2=0x7fb83c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.680+0000 7fb842084640 1 -- 172.21.15.59:0/3311893925 shutdown_connections 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.680+0000 7fb842084640 1 -- 172.21.15.59:0/3311893925 wait complete. 2024-03-06T07:22:51.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.681+0000 7fb842084640 1 Processor -- start 2024-03-06T07:22:51.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.681+0000 7fb842084640 1 -- start start 2024-03-06T07:22:51.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb842084640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb83c06cf20 0x7fb83c0c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb842084640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb83c0cf850 0x7fb83c0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb842084640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb83c0d00c0 0x7fb83c159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb842084640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c0cd900 con 0x7fb83c06cf20 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb842084640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c06c720 con 0x7fb83c0cf850 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb842084640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c0b1030 con 0x7fb83c0d00c0 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb83b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb83c06cf20 0x7fb83c0c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb83b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb83c06cf20 0x7fb83c0c8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36206/0 (socket says 172.21.15.59:36206) 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb83b7fe640 1 -- 172.21.15.59:0/4217098835 learned_addr learned my addr 172.21.15.59:0/4217098835 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb83b7fe640 1 -- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb83c0d00c0 msgr2=0x7fb83c159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb83affd640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb83c0cf850 0x7fb83c0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb83b7fe640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb83c0d00c0 0x7fb83c159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb83b7fe640 1 -- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb83c0cf850 msgr2=0x7fb83c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb83b7fe640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb83c0cf850 0x7fb83c0c6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb83b7fe640 1 -- 172.21.15.59:0/4217098835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c0c9440 con 0x7fb83c06cf20 2024-03-06T07:22:51.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.682+0000 7fb83b7fe640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb83c06cf20 0x7fb83c0c8b40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00c3f0 tx=0x7fb82c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.683+0000 7fb823fff640 1 -- 172.21.15.59:0/4217098835 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c0052d0 con 0x7fb83c06cf20 2024-03-06T07:22:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.683+0000 7fb823fff640 1 -- 172.21.15.59:0/4217098835 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c005490 con 0x7fb83c06cf20 2024-03-06T07:22:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.683+0000 7fb842084640 1 -- 172.21.15.59:0/4217098835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83c0c9720 con 0x7fb83c06cf20 2024-03-06T07:22:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.683+0000 7fb823fff640 1 -- 172.21.15.59:0/4217098835 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c007260 con 0x7fb83c06cf20 2024-03-06T07:22:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.683+0000 7fb842084640 1 -- 172.21.15.59:0/4217098835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83c0c9ce0 con 0x7fb83c06cf20 2024-03-06T07:22:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.684+0000 7fb842084640 1 -- 172.21.15.59:0/4217098835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83c04f330 con 0x7fb83c06cf20 2024-03-06T07:22:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.684+0000 7fb823fff640 1 -- 172.21.15.59:0/4217098835 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb82c007400 con 0x7fb83c06cf20 2024-03-06T07:22:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.684+0000 7fb823fff640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb82806b4d0 0x7fb82806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.685+0000 7fb823fff640 1 -- 172.21.15.59:0/4217098835 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb82c0896e0 con 0x7fb83c06cf20 2024-03-06T07:22:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.685+0000 7fb83affd640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb82806b4d0 0x7fb82806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.685+0000 7fb83affd640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb82806b4d0 0x7fb82806d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb83c09fb60 tx=0x7fb830008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.689+0000 7fb823fff640 1 -- 172.21.15.59:0/4217098835 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb82c058560 con 0x7fb83c06cf20 2024-03-06T07:22:51.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.870+0000 7fb842084640 1 -- 172.21.15.59:0/4217098835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb83c0747a0 con 0x7fb83c06cf20 2024-03-06T07:22:51.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.871+0000 7fb823fff640 1 -- 172.21.15.59:0/4217098835 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16670 (secure 0 0 0) 0x7fb82c057e60 con 0x7fb83c06cf20 2024-03-06T07:22:51.873 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:51.873 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":32,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:50.585565+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:48.848650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:49.545360+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:51.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 -- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb82806b4d0 msgr2=0x7fb82806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb82806b4d0 0x7fb82806d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb83c09fb60 tx=0x7fb830008040 comp rx=0 tx=0).stop 2024-03-06T07:22:51.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 -- 172.21.15.59:0/4217098835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb83c06cf20 msgr2=0x7fb83c0c8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb83c06cf20 0x7fb83c0c8b40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00c3f0 tx=0x7fb82c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 -- 172.21.15.59:0/4217098835 shutdown_connections 2024-03-06T07:22:51.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb82806b4d0 0x7fb82806d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb83c0d00c0 0x7fb83c159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb83c0cf850 0x7fb83c0c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 --2- 172.21.15.59:0/4217098835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb83c06cf20 0x7fb83c0c8b40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 -- 172.21.15.59:0/4217098835 >> 172.21.15.59:0/4217098835 conn(0x7fb83c0b35a0 msgr2=0x7fb83c06d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 -- 172.21.15.59:0/4217098835 shutdown_connections 2024-03-06T07:22:51.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.873+0000 7fb842084640 1 -- 172.21.15.59:0/4217098835 wait complete. 2024-03-06T07:22:51.886 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:22:51.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.988+0000 7f23b83ac640 1 Processor -- start 2024-03-06T07:22:51.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.988+0000 7f23b83ac640 1 -- start start 2024-03-06T07:22:51.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.989+0000 7f23b83ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b00cff40 0x7f23b00b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.989+0000 7f23b83ac640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b0158ac0 con 0x7f23b01586e0 2024-03-06T07:22:51.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.989+0000 7f23b83ac640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b015b2a0 con 0x7f23b00cfa90 2024-03-06T07:22:51.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.989+0000 7f23b83ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b015b540 con 0x7f23b00cff40 2024-03-06T07:22:51.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.989+0000 7f23b6148640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f23b01586e0 0x7f23b00aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46978/0 (socket says 172.21.15.59:46978) 2024-03-06T07:22:51.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.989+0000 7f23b6148640 1 -- 172.21.15.59:0/4079637256 learned_addr learned my addr 172.21.15.59:0/4079637256 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:51.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.989+0000 7f23b6949640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b00cff40 0x7f23b00b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.989+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096472080 0 0) 0x7f23b0158ac0 con 0x7f23b01586e0 2024-03-06T07:22:51.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2398003520 con 0x7f23b01586e0 2024-03-06T07:22:51.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43155815 0 0) 0x7f23b015b2a0 con 0x7f23b00cfa90 2024-03-06T07:22:51.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b0158ac0 con 0x7f23b00cfa90 2024-03-06T07:22:51.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2260714724 0 0) 0x7f2398003520 con 0x7f23b01586e0 2024-03-06T07:22:51.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b00cff40 msgr2=0x7f23b00b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b00cff40 0x7f23b00b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 >> v1:172.21.15.164:6789/0 conn(0x7f23b00cfa90 legacy=0x7f23b00b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b015b820 con 0x7f23b01586e0 2024-03-06T07:22:51.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f23a80019f0 con 0x7f23b01586e0 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 >> v1:172.21.15.59:6789/0 conn(0x7f23b01586e0 legacy=0x7f23b00aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2398003b70 0x7f2398003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b00cff40 0x7f2398004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2398004d40 0x7f23980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b015b540 con 0x7f2398004d40 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239800ba70 con 0x7f2398003b70 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239800bc70 con 0x7f23b00cff40 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b6949640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2398004d40 0x7f23980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b5947640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b00cff40 0x7f2398004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b6148640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2398003b70 0x7f2398003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b6949640 1 -- 172.21.15.59:0/4079637256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b00cff40 msgr2=0x7f2398004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b6949640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b00cff40 0x7f2398004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b6949640 1 -- 172.21.15.59:0/4079637256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2398003b70 msgr2=0x7f2398003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b6949640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2398003b70 0x7f2398003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.990+0000 7f23b6949640 1 -- 172.21.15.59:0/4079637256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2398003520 con 0x7f2398004d40 2024-03-06T07:22:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.991+0000 7f23b6949640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2398004d40 0x7f23980051c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009ac0 tx=0x7f2398005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87556d7655b75c78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.991+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a0039070 con 0x7f2398004d40 2024-03-06T07:22:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.991+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a002c6b0 con 0x7f2398004d40 2024-03-06T07:22:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.991+0000 7f23b5146640 1 -- 172.21.15.59:0/4079637256 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a0009690 con 0x7f2398004d40 2024-03-06T07:22:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.992+0000 7f23b83ac640 1 -- 172.21.15.59:0/4079637256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2398004d40 msgr2=0x7f23980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.992+0000 7f23b83ac640 1 --2- 172.21.15.59:0/4079637256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2398004d40 0x7f23980051c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009ac0 tx=0x7f2398005830 comp rx=0 tx=0).stop 2024-03-06T07:22:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.992+0000 7f23b6949640 1 -- 172.21.15.59:0/4079637256 reap_dead start 2024-03-06T07:22:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.992+0000 7f23b83ac640 1 -- 172.21.15.59:0/4079637256 shutdown_connections 2024-03-06T07:22:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.992+0000 7f23b83ac640 1 -- 172.21.15.59:0/4079637256 >> 172.21.15.59:0/4079637256 conn(0x7f23b006b190 msgr2=0x7f23b006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.992+0000 7f23b83ac640 1 -- 172.21.15.59:0/4079637256 shutdown_connections 2024-03-06T07:22:51.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.992+0000 7f23b83ac640 1 -- 172.21.15.59:0/4079637256 wait complete. 2024-03-06T07:22:52.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.993+0000 7f23b83ac640 1 Processor -- start 2024-03-06T07:22:52.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.993+0000 7f23b83ac640 1 -- start start 2024-03-06T07:22:52.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b83ac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2398004d40 0x7f23b014f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b83ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23b01528a0 0x7f23b0159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b83ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b014c9e0 0x7f23b014ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b83ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00d0360 con 0x7f23b01528a0 2024-03-06T07:22:52.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b83ac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b015bde0 con 0x7f2398004d40 2024-03-06T07:22:52.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b83ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00af150 con 0x7f23b014c9e0 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b5947640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23b01528a0 0x7f23b0159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b5947640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23b01528a0 0x7f23b0159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36232/0 (socket says 172.21.15.59:36232) 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b5947640 1 -- 172.21.15.59:0/2130143332 learned_addr learned my addr 172.21.15.59:0/2130143332 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b6949640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b014c9e0 0x7f23b014ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b6148640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2398004d40 0x7f23b014f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b5947640 1 -- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b014c9e0 msgr2=0x7f23b014ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b5947640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b014c9e0 0x7f23b014ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b5947640 1 -- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2398004d40 msgr2=0x7f23b014f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b5947640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2398004d40 0x7f23b014f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b5947640 1 -- 172.21.15.59:0/2130143332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b0153060 con 0x7f23b01528a0 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b5947640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23b01528a0 0x7f23b0159f70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f23a400f840 tx=0x7f23a400fd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23aeffd640 1 -- 172.21.15.59:0/2130143332 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a40052e0 con 0x7f23b01528a0 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23aeffd640 1 -- 172.21.15.59:0/2130143332 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a4018070 con 0x7f23b01528a0 2024-03-06T07:22:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b83ac640 1 -- 172.21.15.59:0/2130143332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b0150100 con 0x7f23b01528a0 2024-03-06T07:22:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23aeffd640 1 -- 172.21.15.59:0/2130143332 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a400d970 con 0x7f23b01528a0 2024-03-06T07:22:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.994+0000 7f23b83ac640 1 -- 172.21.15.59:0/2130143332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b014fd40 con 0x7f23b01528a0 2024-03-06T07:22:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.995+0000 7f23b83ac640 1 -- 172.21.15.59:0/2130143332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b00732b0 con 0x7f23b01528a0 2024-03-06T07:22:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.996+0000 7f23aeffd640 1 -- 172.21.15.59:0/2130143332 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23a400db10 con 0x7f23b01528a0 2024-03-06T07:22:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.996+0000 7f23aeffd640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f239c06b4d0 0x7f239c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.996+0000 7f23aeffd640 1 -- 172.21.15.59:0/2130143332 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23a4089ff0 con 0x7f23b01528a0 2024-03-06T07:22:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.996+0000 7f23b6148640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f239c06b4d0 0x7f239c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:51.997+0000 7f23b6148640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f239c06b4d0 0x7f239c06d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f23a8004d40 tx=0x7f23a8006040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.000+0000 7f23aeffd640 1 -- 172.21.15.59:0/2130143332 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23a40588b0 con 0x7f23b01528a0 2024-03-06T07:22:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.219+0000 7f23b83ac640 1 -- 172.21.15.59:0/2130143332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f23b0071dd0 con 0x7f23b01528a0 2024-03-06T07:22:52.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.231+0000 7f23aeffd640 1 -- 172.21.15.59:0/2130143332 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v12) v1 ==== 118+0+0 (secure 0 0 0) 0x7f23a405c950 con 0x7f23b01528a0 2024-03-06T07:22:52.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 -- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f239c06b4d0 msgr2=0x7f239c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f239c06b4d0 0x7f239c06d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f23a8004d40 tx=0x7f23a8006040 comp rx=0 tx=0).stop 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 -- 172.21.15.59:0/2130143332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23b01528a0 msgr2=0x7f23b0159f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23b01528a0 0x7f23b0159f70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f23a400f840 tx=0x7f23a400fd00 comp rx=0 tx=0).stop 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 -- 172.21.15.59:0/2130143332 shutdown_connections 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f239c06b4d0 0x7f239c06d990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b014c9e0 0x7f23b014ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23b01528a0 0x7f23b0159f70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 --2- 172.21.15.59:0/2130143332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2398004d40 0x7f23b014f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 -- 172.21.15.59:0/2130143332 >> 172.21.15.59:0/2130143332 conn(0x7f23b006b190 msgr2=0x7f23b014ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 -- 172.21.15.59:0/2130143332 shutdown_connections 2024-03-06T07:22:52.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.233+0000 7f23b83ac640 1 -- 172.21.15.59:0/2130143332 wait complete. 2024-03-06T07:22:52.246 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:22:52.246 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:22:52.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.348+0000 7fddcb039640 1 Processor -- start 2024-03-06T07:22:52.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.348+0000 7fddcb039640 1 -- start start 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.349+0000 7fddcb039640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddc41614d0 0x7fddc40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.349+0000 7fddcb039640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fddc40d1910 0x7fddc40d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.349+0000 7fddcb039640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc40d2740 con 0x7fddc40d22b0 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.349+0000 7fddcb039640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc41618b0 con 0x7fddc40d1910 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.349+0000 7fddcb039640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40cf400 con 0x7fddc41614d0 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.349+0000 7fddc95d6640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fddc40d22b0 0x7fddc40d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46992/0 (socket says 172.21.15.59:46992) 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.349+0000 7fddc95d6640 1 -- 172.21.15.59:0/3405221168 learned_addr learned my addr 172.21.15.59:0/3405221168 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.349+0000 7fddc3fff640 1 --2- 172.21.15.59:0/3405221168 >> v2:172.21.15.164:3300/0 conn(0x7fddc40d1910 0x7fddc40d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.349+0000 7fddc8dd5640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddc41614d0 0x7fddc40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.349+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652219259 0 0) 0x7fddb4001110 con 0x7fddc40d22b0 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.349+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddac003520 con 0x7fddc40d22b0 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3512872948 0 0) 0x7fddac003520 con 0x7fddc40d22b0 2024-03-06T07:22:52.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddc41614d0 msgr2=0x7fddc40d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddc41614d0 0x7fddc40d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 >> v2:172.21.15.164:3300/0 conn(0x7fddc40d1910 msgr2=0x7fddc40d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 --2- 172.21.15.59:0/3405221168 >> v2:172.21.15.164:3300/0 conn(0x7fddc40d1910 0x7fddc40d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc40cf6a0 con 0x7fddc40d22b0 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fddb4001840 con 0x7fddc40d22b0 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 >> v1:172.21.15.59:6789/0 conn(0x7fddc40d22b0 legacy=0x7fddc40d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddc41614d0 0x7fddac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddac003f30 0x7fddac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddac004d40 0x7fddac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc41618b0 con 0x7fddac003f30 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40cf400 con 0x7fddac004d40 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddac00b9c0 con 0x7fddc41614d0 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc3fff640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddac003f30 0x7fddac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc8dd5640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddc41614d0 0x7fddac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc95d6640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddac004d40 0x7fddac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc3fff640 1 -- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddc41614d0 msgr2=0x7fddac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc3fff640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddc41614d0 0x7fddac0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc3fff640 1 -- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddac004d40 msgr2=0x7fddac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc3fff640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddac004d40 0x7fddac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc3fff640 1 -- 172.21.15.59:0/3405221168 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac003520 con 0x7fddac003f30 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.350+0000 7fddc95d6640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddac004d40 0x7fddac0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.351+0000 7fddc3fff640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddac003f30 0x7fddac004800 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009ac0 tx=0x7fddac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92065f1478815ac1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.352+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8039070 con 0x7fddac003f30 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.352+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb802c6d0 con 0x7fddac003f30 2024-03-06T07:22:52.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.352+0000 7fddc37fe640 1 -- 172.21.15.59:0/3405221168 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8009690 con 0x7fddac003f30 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.352+0000 7fddcb039640 1 -- 172.21.15.59:0/3405221168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddac003f30 msgr2=0x7fddac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.352+0000 7fddcb039640 1 --2- 172.21.15.59:0/3405221168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddac003f30 0x7fddac004800 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009ac0 tx=0x7fddac005830 comp rx=0 tx=0).stop 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.352+0000 7fddc95d6640 1 -- 172.21.15.59:0/3405221168 reap_dead start 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.352+0000 7fddcb039640 1 -- 172.21.15.59:0/3405221168 shutdown_connections 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.352+0000 7fddcb039640 1 -- 172.21.15.59:0/3405221168 >> 172.21.15.59:0/3405221168 conn(0x7fddc40d3d40 msgr2=0x7fddc40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.352+0000 7fddcb039640 1 -- 172.21.15.59:0/3405221168 shutdown_connections 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.352+0000 7fddcb039640 1 -- 172.21.15.59:0/3405221168 wait complete. 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.353+0000 7fddcb039640 1 Processor -- start 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.353+0000 7fddcb039640 1 -- start start 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.354+0000 7fddcb039640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddac003f30 0x7fddc41577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.354+0000 7fddcb039640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddc41491a0 0x7fddc40c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.354+0000 7fddcb039640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddc415d120 0x7fddc415d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.354+0000 7fddcb039640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40b0910 con 0x7fddc41491a0 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.354+0000 7fddcb039640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40cfc60 con 0x7fddc415d120 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.354+0000 7fddcb039640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40b1e40 con 0x7fddac003f30 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.354+0000 7fddc3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddc41491a0 0x7fddc40c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.354+0000 7fddc3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddc41491a0 0x7fddc40c9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36252/0 (socket says 172.21.15.59:36252) 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.354+0000 7fddc3fff640 1 -- 172.21.15.59:0/3800114236 learned_addr learned my addr 172.21.15.59:0/3800114236 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.354+0000 7fddc95d6640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddc415d120 0x7fddc415d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.354+0000 7fddc8dd5640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddac003f30 0x7fddc41577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddc3fff640 1 -- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddac003f30 msgr2=0x7fddc41577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddc3fff640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddac003f30 0x7fddc41577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddc3fff640 1 -- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddc415d120 msgr2=0x7fddc415d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddc3fff640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddc415d120 0x7fddc415d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddc3fff640 1 -- 172.21.15.59:0/3800114236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc4157e10 con 0x7fddc41491a0 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddc95d6640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddc415d120 0x7fddc415d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:52.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddc3fff640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddc41491a0 0x7fddc40c9d60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fddb8037040 tx=0x7fddb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddc17fa640 1 -- 172.21.15.59:0/3800114236 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8042020 con 0x7fddc41491a0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddc17fa640 1 -- 172.21.15.59:0/3800114236 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb803a040 con 0x7fddc41491a0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddcb039640 1 -- 172.21.15.59:0/3800114236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc41580f0 con 0x7fddc41491a0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddc17fa640 1 -- 172.21.15.59:0/3800114236 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8039070 con 0x7fddc41491a0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.355+0000 7fddcb039640 1 -- 172.21.15.59:0/3800114236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc41586b0 con 0x7fddc41491a0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.356+0000 7fddcb039640 1 -- 172.21.15.59:0/3800114236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc4068ff0 con 0x7fddc41491a0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.357+0000 7fddc17fa640 1 -- 172.21.15.59:0/3800114236 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddb8045020 con 0x7fddc41491a0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.357+0000 7fddc17fa640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddb006b500 0x7fddb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.357+0000 7fddc17fa640 1 -- 172.21.15.59:0/3800114236 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddb80ab600 con 0x7fddc41491a0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.357+0000 7fddc8dd5640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddb006b500 0x7fddb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:52.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.358+0000 7fddc8dd5640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddb006b500 0x7fddb006d9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0133c0 tx=0x7fddbc00b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:52.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.361+0000 7fddc17fa640 1 -- 172.21.15.59:0/3800114236 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddb804c070 con 0x7fddc41491a0 2024-03-06T07:22:52.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:52.582+0000 7fddcb039640 1 -- 172.21.15.59:0/3800114236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fddc40d2c00 con 0x7fddc41491a0 2024-03-06T07:22:53.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.232+0000 7fddc17fa640 1 -- 172.21.15.59:0/3800114236 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v13) v1 ==== 122+0+0 (secure 0 0 0) 0x7fddb807a510 con 0x7fddc41491a0 2024-03-06T07:22:53.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.234+0000 7fddcb039640 1 -- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddb006b500 msgr2=0x7fddb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.234+0000 7fddcb039640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddb006b500 0x7fddb006d9c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0133c0 tx=0x7fddbc00b040 comp rx=0 tx=0).stop 2024-03-06T07:22:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.234+0000 7fddcb039640 1 -- 172.21.15.59:0/3800114236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddc41491a0 msgr2=0x7fddc40c9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.234+0000 7fddcb039640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddc41491a0 0x7fddc40c9d60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fddb8037040 tx=0x7fddb8004730 comp rx=0 tx=0).stop 2024-03-06T07:22:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.235+0000 7fddcb039640 1 -- 172.21.15.59:0/3800114236 shutdown_connections 2024-03-06T07:22:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.235+0000 7fddcb039640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddb006b500 0x7fddb006d9c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.235+0000 7fddcb039640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddc415d120 0x7fddc415d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.235+0000 7fddcb039640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fddc41491a0 0x7fddc40c9d60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.235+0000 7fddcb039640 1 --2- 172.21.15.59:0/3800114236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddac003f30 0x7fddc41577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.235+0000 7fddcb039640 1 -- 172.21.15.59:0/3800114236 >> 172.21.15.59:0/3800114236 conn(0x7fddc40d3d40 msgr2=0x7fddc40b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.235+0000 7fddcb039640 1 -- 172.21.15.59:0/3800114236 shutdown_connections 2024-03-06T07:22:53.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.235+0000 7fddcb039640 1 -- 172.21.15.59:0/3800114236 wait complete. 2024-03-06T07:22:53.247 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:22:53.248 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:53.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.347+0000 7fb2a285c640 1 Processor -- start 2024-03-06T07:22:53.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.347+0000 7fb2a285c640 1 -- start start 2024-03-06T07:22:53.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.348+0000 7fb2a285c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c15dfe0 0x7fb29c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.348+0000 7fb2a285c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c0b3430 con 0x7fb29c0d29c0 2024-03-06T07:22:53.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.348+0000 7fb2a285c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c0b3710 con 0x7fb29c0cf830 2024-03-06T07:22:53.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.348+0000 7fb2a285c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c0b31e0 con 0x7fb29c15dfe0 2024-03-06T07:22:53.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.348+0000 7fb2a0df9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb29c0d29c0 0x7fb29c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47006/0 (socket says 172.21.15.59:47006) 2024-03-06T07:22:53.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.348+0000 7fb2a0df9640 1 -- 172.21.15.59:0/4213675380 learned_addr learned my addr 172.21.15.59:0/4213675380 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:53.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.348+0000 7fb29bfff640 1 --2- 172.21.15.59:0/4213675380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c15dfe0 0x7fb29c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.349+0000 7fb29affd640 1 -- 172.21.15.59:0/4213675380 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984375856 0 0) 0x7fb29c0b3430 con 0x7fb29c0d29c0 2024-03-06T07:22:53.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.349+0000 7fb29affd640 1 -- 172.21.15.59:0/4213675380 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb284003520 con 0x7fb29c0d29c0 2024-03-06T07:22:53.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.349+0000 7fb29affd640 1 -- 172.21.15.59:0/4213675380 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386705174 0 0) 0x7fb29c0b3710 con 0x7fb29c0cf830 2024-03-06T07:22:53.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.349+0000 7fb29affd640 1 -- 172.21.15.59:0/4213675380 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb29c0b3430 con 0x7fb29c0cf830 2024-03-06T07:22:53.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.349+0000 7fb29affd640 1 -- 172.21.15.59:0/4213675380 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3538167550 0 0) 0x7fb284003520 con 0x7fb29c0d29c0 2024-03-06T07:22:53.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.349+0000 7fb29bfff640 1 -- 172.21.15.59:0/4213675380 >> v1:172.21.15.164:6789/0 conn(0x7fb29c0cf830 legacy=0x7fb29c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.349+0000 7fb29bfff640 1 -- 172.21.15.59:0/4213675380 >> v1:172.21.15.59:6789/0 conn(0x7fb29c0d29c0 legacy=0x7fb29c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.349+0000 7fb29bfff640 1 -- 172.21.15.59:0/4213675380 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c0b39f0 con 0x7fb29c15dfe0 2024-03-06T07:22:53.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.350+0000 7fb29bfff640 1 --2- 172.21.15.59:0/4213675380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c15dfe0 0x7fb29c0cf2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0097d0 tx=0x7fb28c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1666ed112510928b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.350+0000 7fb29affd640 1 -- 172.21.15.59:0/4213675380 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c037070 con 0x7fb29c15dfe0 2024-03-06T07:22:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.350+0000 7fb29affd640 1 -- 172.21.15.59:0/4213675380 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28c030510 con 0x7fb29c15dfe0 2024-03-06T07:22:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.350+0000 7fb29affd640 1 -- 172.21.15.59:0/4213675380 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2940019f0 con 0x7fb29c15dfe0 2024-03-06T07:22:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.351+0000 7fb2a285c640 1 -- 172.21.15.59:0/4213675380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c15dfe0 msgr2=0x7fb29c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.351+0000 7fb2a285c640 1 --2- 172.21.15.59:0/4213675380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c15dfe0 0x7fb29c0cf2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0097d0 tx=0x7fb28c02b800 comp rx=0 tx=0).stop 2024-03-06T07:22:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.351+0000 7fb2a285c640 1 -- 172.21.15.59:0/4213675380 shutdown_connections 2024-03-06T07:22:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.351+0000 7fb2a285c640 1 --2- 172.21.15.59:0/4213675380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c15dfe0 0x7fb29c0cf2f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.351+0000 7fb2a285c640 1 -- 172.21.15.59:0/4213675380 >> 172.21.15.59:0/4213675380 conn(0x7fb29c0d3cd0 msgr2=0x7fb29c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.351+0000 7fb2a285c640 1 -- 172.21.15.59:0/4213675380 shutdown_connections 2024-03-06T07:22:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.351+0000 7fb2a285c640 1 -- 172.21.15.59:0/4213675380 wait complete. 2024-03-06T07:22:53.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.352+0000 7fb2a285c640 1 Processor -- start 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.352+0000 7fb2a285c640 1 -- start start 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb2a285c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c0cf830 0x7fb29c153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb2a285c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb29c0d29c0 0x7fb29c0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb2a285c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb29c15dfe0 0x7fb29c09f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb2a285c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c15e3c0 con 0x7fb29c0d29c0 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb2a285c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c0b3d60 con 0x7fb29c15dfe0 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb2a285c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c15e540 con 0x7fb29c0cf830 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb29b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb29c0d29c0 0x7fb29c0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb29b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb29c0d29c0 0x7fb29c0cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36264/0 (socket says 172.21.15.59:36264) 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb29b7fe640 1 -- 172.21.15.59:0/4001541796 learned_addr learned my addr 172.21.15.59:0/4001541796 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb2a0df9640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb29c15dfe0 0x7fb29c09f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb29bfff640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c0cf830 0x7fb29c153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb29b7fe640 1 -- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c0cf830 msgr2=0x7fb29c153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb29b7fe640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c0cf830 0x7fb29c153f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb29b7fe640 1 -- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb29c15dfe0 msgr2=0x7fb29c09f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.353+0000 7fb29b7fe640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb29c15dfe0 0x7fb29c09f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.354+0000 7fb29b7fe640 1 -- 172.21.15.59:0/4001541796 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c0cbba0 con 0x7fb29c0d29c0 2024-03-06T07:22:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.354+0000 7fb2a0df9640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb29c15dfe0 0x7fb29c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.354+0000 7fb29bfff640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c0cf830 0x7fb29c153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.354+0000 7fb29b7fe640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb29c0d29c0 0x7fb29c0cd8b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb29000c450 tx=0x7fb29000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.354+0000 7fb298ff9640 1 -- 172.21.15.59:0/4001541796 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb290017070 con 0x7fb29c0d29c0 2024-03-06T07:22:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.354+0000 7fb298ff9640 1 -- 172.21.15.59:0/4001541796 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2900052d0 con 0x7fb29c0d29c0 2024-03-06T07:22:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.354+0000 7fb2a285c640 1 -- 172.21.15.59:0/4001541796 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29c0cbe70 con 0x7fb29c0d29c0 2024-03-06T07:22:53.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.354+0000 7fb298ff9640 1 -- 172.21.15.59:0/4001541796 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb290003830 con 0x7fb29c0d29c0 2024-03-06T07:22:53.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.354+0000 7fb2a285c640 1 -- 172.21.15.59:0/4001541796 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29c0cc430 con 0x7fb29c0d29c0 2024-03-06T07:22:53.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.355+0000 7fb2a285c640 1 -- 172.21.15.59:0/4001541796 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c0b3710 con 0x7fb29c0d29c0 2024-03-06T07:22:53.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.355+0000 7fb298ff9640 1 -- 172.21.15.59:0/4001541796 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb29001a070 con 0x7fb29c0d29c0 2024-03-06T07:22:53.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.356+0000 7fb298ff9640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb28806b4d0 0x7fb28806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.356+0000 7fb298ff9640 1 -- 172.21.15.59:0/4001541796 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb290089600 con 0x7fb29c0d29c0 2024-03-06T07:22:53.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.356+0000 7fb29bfff640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb28806b4d0 0x7fb28806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.357+0000 7fb29bfff640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb28806b4d0 0x7fb28806d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb28c034000 tx=0x7fb28c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.360+0000 7fb298ff9640 1 -- 172.21.15.59:0/4001541796 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb29c0b3710 con 0x7fb29c0d29c0 2024-03-06T07:22:53.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.582+0000 7fb2a285c640 1 -- 172.21.15.59:0/4001541796 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb29c0cf0b0 con 0x7fb29c0d29c0 2024-03-06T07:22:53.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.583+0000 7fb298ff9640 1 -- 172.21.15.59:0/4001541796 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fb290058540 con 0x7fb29c0d29c0 2024-03-06T07:22:53.585 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:53.585 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:22:53.585 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:22:53.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.584+0000 7fb2a285c640 1 -- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb28806b4d0 msgr2=0x7fb28806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.584+0000 7fb2a285c640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb28806b4d0 0x7fb28806d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb28c034000 tx=0x7fb28c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:53.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.584+0000 7fb2a285c640 1 -- 172.21.15.59:0/4001541796 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb29c0d29c0 msgr2=0x7fb29c0cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.584+0000 7fb2a285c640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb29c0d29c0 0x7fb29c0cd8b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb29000c450 tx=0x7fb29000c910 comp rx=0 tx=0).stop 2024-03-06T07:22:53.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.585+0000 7fb2a285c640 1 -- 172.21.15.59:0/4001541796 shutdown_connections 2024-03-06T07:22:53.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.585+0000 7fb2a285c640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb28806b4d0 0x7fb28806d990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.585+0000 7fb2a285c640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb29c15dfe0 0x7fb29c09f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.585+0000 7fb2a285c640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb29c0d29c0 0x7fb29c0cd8b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.585+0000 7fb2a285c640 1 --2- 172.21.15.59:0/4001541796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb29c0cf830 0x7fb29c153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.585+0000 7fb2a285c640 1 -- 172.21.15.59:0/4001541796 >> 172.21.15.59:0/4001541796 conn(0x7fb29c0d3cd0 msgr2=0x7fb29c06cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.585+0000 7fb2a285c640 1 -- 172.21.15.59:0/4001541796 shutdown_connections 2024-03-06T07:22:53.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.585+0000 7fb2a285c640 1 -- 172.21.15.59:0/4001541796 wait complete. 2024-03-06T07:22:53.597 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.697+0000 7ffb6f4af640 1 Processor -- start 2024-03-06T07:22:53.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.697+0000 7ffb6f4af640 1 -- start start 2024-03-06T07:22:53.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.698+0000 7ffb6f4af640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffb6815dfe0 0x7ffb680cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.698+0000 7ffb6f4af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb680d29c0 0x7ffb680b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.698+0000 7ffb6f4af640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb680b3430 con 0x7ffb680cf830 2024-03-06T07:22:53.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.698+0000 7ffb6f4af640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb680b31e0 con 0x7ffb6815dfe0 2024-03-06T07:22:53.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.698+0000 7ffb6f4af640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680b3d60 con 0x7ffb680d29c0 2024-03-06T07:22:53.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.698+0000 7ffb6da4c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb680d29c0 0x7ffb680b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.698+0000 7ffb6d24b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffb6815dfe0 0x7ffb680cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.698+0000 7ffb6ca4a640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ffb680cf830 0x7ffb680d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:35752/0 (socket says 172.21.15.59:35752) 2024-03-06T07:22:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.698+0000 7ffb6ca4a640 1 -- 172.21.15.59:0/4088417025 learned_addr learned my addr 172.21.15.59:0/4088417025 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.698+0000 7ffb6d24b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffb6815dfe0 0x7ffb680cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:44086/0 (socket says 172.21.15.59:44086) 2024-03-06T07:22:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.699+0000 7ffb5ffff640 1 -- 172.21.15.59:0/4088417025 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622757306 0 0) 0x7ffb680b3430 con 0x7ffb680cf830 2024-03-06T07:22:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.699+0000 7ffb6da4c640 1 -- 172.21.15.59:0/4088417025 >> v2:172.21.15.164:3300/0 conn(0x7ffb6815dfe0 msgr2=0x7ffb680cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.699+0000 7ffb6da4c640 1 --2- 172.21.15.59:0/4088417025 >> v2:172.21.15.164:3300/0 conn(0x7ffb6815dfe0 0x7ffb680cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.699+0000 7ffb6da4c640 1 -- 172.21.15.59:0/4088417025 >> v1:172.21.15.164:6789/0 conn(0x7ffb680cf830 legacy=0x7ffb680d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.699+0000 7ffb6da4c640 1 -- 172.21.15.59:0/4088417025 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680b3710 con 0x7ffb680d29c0 2024-03-06T07:22:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.699+0000 7ffb6da4c640 1 --2- 172.21.15.59:0/4088417025 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb680d29c0 0x7ffb680b41f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009d90 tx=0x7ffb6002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=947757917c512864 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.700+0000 7ffb5ffff640 1 -- 172.21.15.59:0/4088417025 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60039070 con 0x7ffb680d29c0 2024-03-06T07:22:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.700+0000 7ffb5ffff640 1 -- 172.21.15.59:0/4088417025 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6002cc50 con 0x7ffb680d29c0 2024-03-06T07:22:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.700+0000 7ffb5ffff640 1 -- 172.21.15.59:0/4088417025 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60034440 con 0x7ffb680d29c0 2024-03-06T07:22:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.700+0000 7ffb6f4af640 1 -- 172.21.15.59:0/4088417025 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb680d29c0 msgr2=0x7ffb680b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.700+0000 7ffb6f4af640 1 --2- 172.21.15.59:0/4088417025 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb680d29c0 0x7ffb680b41f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009d90 tx=0x7ffb6002bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.700+0000 7ffb6f4af640 1 -- 172.21.15.59:0/4088417025 shutdown_connections 2024-03-06T07:22:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.700+0000 7ffb6f4af640 1 --2- 172.21.15.59:0/4088417025 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb680d29c0 0x7ffb680b41f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.700+0000 7ffb6f4af640 1 --2- 172.21.15.59:0/4088417025 >> v2:172.21.15.164:3300/0 conn(0x7ffb6815dfe0 0x7ffb680cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.700+0000 7ffb6f4af640 1 -- 172.21.15.59:0/4088417025 >> 172.21.15.59:0/4088417025 conn(0x7ffb680d3cd0 msgr2=0x7ffb680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.700+0000 7ffb6f4af640 1 -- 172.21.15.59:0/4088417025 shutdown_connections 2024-03-06T07:22:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.700+0000 7ffb6f4af640 1 -- 172.21.15.59:0/4088417025 wait complete. 2024-03-06T07:22:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.701+0000 7ffb6f4af640 1 Processor -- start 2024-03-06T07:22:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.702+0000 7ffb6f4af640 1 -- start start 2024-03-06T07:22:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.702+0000 7ffb6f4af640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb680cf830 0x7ffb681f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.702+0000 7ffb6f4af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb680d29c0 0x7ffb680cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.702+0000 7ffb6f4af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb6815dfe0 0x7ffb680cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.702+0000 7ffb6f4af640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680d1df0 con 0x7ffb6815dfe0 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.702+0000 7ffb6f4af640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680d1c70 con 0x7ffb680cf830 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.702+0000 7ffb6f4af640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6815e3c0 con 0x7ffb680d29c0 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.702+0000 7ffb6da4c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb6815dfe0 0x7ffb680cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.702+0000 7ffb6da4c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb6815dfe0 0x7ffb680cd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57756/0 (socket says 172.21.15.59:57756) 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.702+0000 7ffb6da4c640 1 -- 172.21.15.59:0/2525372588 learned_addr learned my addr 172.21.15.59:0/2525372588 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb6d24b640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb680cf830 0x7ffb681f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb6ca4a640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb680d29c0 0x7ffb680cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb6da4c640 1 -- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb680d29c0 msgr2=0x7ffb680cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb6da4c640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb680d29c0 0x7ffb680cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb6da4c640 1 -- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb680cf830 msgr2=0x7ffb681f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb6da4c640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb680cf830 0x7ffb681f4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb6da4c640 1 -- 172.21.15.59:0/2525372588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680b88a0 con 0x7ffb6815dfe0 2024-03-06T07:22:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb6d24b640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb680cf830 0x7ffb681f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb6da4c640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb6815dfe0 0x7ffb680cd9d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009d90 tx=0x7ffb60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb5dffb640 1 -- 172.21.15.59:0/2525372588 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60044020 con 0x7ffb6815dfe0 2024-03-06T07:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb5dffb640 1 -- 172.21.15.59:0/2525372588 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6003a040 con 0x7ffb6815dfe0 2024-03-06T07:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb6f4af640 1 -- 172.21.15.59:0/2525372588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb680caf20 con 0x7ffb6815dfe0 2024-03-06T07:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb5dffb640 1 -- 172.21.15.59:0/2525372588 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60039050 con 0x7ffb6815dfe0 2024-03-06T07:22:53.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.703+0000 7ffb6f4af640 1 -- 172.21.15.59:0/2525372588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb680cb4e0 con 0x7ffb6815dfe0 2024-03-06T07:22:53.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.704+0000 7ffb6f4af640 1 -- 172.21.15.59:0/2525372588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6804f330 con 0x7ffb6815dfe0 2024-03-06T07:22:53.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.705+0000 7ffb5dffb640 1 -- 172.21.15.59:0/2525372588 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb60003990 con 0x7ffb6815dfe0 2024-03-06T07:22:53.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.705+0000 7ffb5dffb640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb4c06b4d0 0x7ffb4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:53.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.705+0000 7ffb5dffb640 1 -- 172.21.15.59:0/2525372588 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb600ac580 con 0x7ffb6815dfe0 2024-03-06T07:22:53.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.705+0000 7ffb6d24b640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb4c06b4d0 0x7ffb4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:53.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.706+0000 7ffb6d24b640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb4c06b4d0 0x7ffb4c06d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffb680a1cd0 tx=0x7ffb64008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.711+0000 7ffb5dffb640 1 -- 172.21.15.59:0/2525372588 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb6004e060 con 0x7ffb6815dfe0 2024-03-06T07:22:53.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.893+0000 7ffb6f4af640 1 -- 172.21.15.59:0/2525372588 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb680d0b40 con 0x7ffb6815dfe0 2024-03-06T07:22:53.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.894+0000 7ffb5dffb640 1 -- 172.21.15.59:0/2525372588 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16657 (secure 0 0 0) 0x7ffb6003d360 con 0x7ffb6815dfe0 2024-03-06T07:22:53.896 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:53.896 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":33,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:52.226312+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:48.848650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:49.545360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:53.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 -- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb4c06b4d0 msgr2=0x7ffb4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb4c06b4d0 0x7ffb4c06d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffb680a1cd0 tx=0x7ffb64008040 comp rx=0 tx=0).stop 2024-03-06T07:22:53.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 -- 172.21.15.59:0/2525372588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb6815dfe0 msgr2=0x7ffb680cd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb6815dfe0 0x7ffb680cd9d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009d90 tx=0x7ffb60004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 -- 172.21.15.59:0/2525372588 shutdown_connections 2024-03-06T07:22:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb4c06b4d0 0x7ffb4c06d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb6815dfe0 0x7ffb680cd9d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb680d29c0 0x7ffb680cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 --2- 172.21.15.59:0/2525372588 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb680cf830 0x7ffb681f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 -- 172.21.15.59:0/2525372588 >> 172.21.15.59:0/2525372588 conn(0x7ffb680d3cd0 msgr2=0x7ffb680b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 -- 172.21.15.59:0/2525372588 shutdown_connections 2024-03-06T07:22:53.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:53.896+0000 7ffb6f4af640 1 -- 172.21.15.59:0/2525372588 wait complete. 2024-03-06T07:22:53.909 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:22:53.909 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:54.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.012+0000 7fad70565640 1 Processor -- start 2024-03-06T07:22:54.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.012+0000 7fad70565640 1 -- start start 2024-03-06T07:22:54.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.013+0000 7fad70565640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fad6815dfe0 0x7fad680cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.013+0000 7fad70565640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad680d29c0 0x7fad680b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.013+0000 7fad70565640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad680b3430 con 0x7fad680cf830 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.013+0000 7fad70565640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad680b31e0 con 0x7fad6815dfe0 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.013+0000 7fad70565640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad680b3d60 con 0x7fad680d29c0 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.013+0000 7fad6e301640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fad6815dfe0 0x7fad680cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.013+0000 7fad6e301640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fad6815dfe0 0x7fad680cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57764/0 (socket says 172.21.15.59:57764) 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.013+0000 7fad6e301640 1 -- 172.21.15.59:0/290969625 learned_addr learned my addr 172.21.15.59:0/290969625 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.013+0000 7fad6eb02640 1 --2- 172.21.15.59:0/290969625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad680d29c0 0x7fad680b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.013+0000 7fad6e301640 1 -- 172.21.15.59:0/290969625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad680d29c0 msgr2=0x7fad680b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.013+0000 7fad6e301640 1 --2- 172.21.15.59:0/290969625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad680d29c0 0x7fad680b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.014+0000 7fad6e301640 1 -- 172.21.15.59:0/290969625 >> v1:172.21.15.164:6789/0 conn(0x7fad680cf830 legacy=0x7fad680d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.014+0000 7fad6e301640 1 -- 172.21.15.59:0/290969625 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad680b3710 con 0x7fad6815dfe0 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.014+0000 7fad6eb02640 1 --2- 172.21.15.59:0/290969625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad680d29c0 0x7fad680b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.014+0000 7fad6e301640 1 --2- 172.21.15.59:0/290969625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6815dfe0 0x7fad680cf2f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fad64009870 tx=0x7fad6402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f262068639d3616 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.014+0000 7fad6d2ff640 1 -- 172.21.15.59:0/290969625 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64035020 con 0x7fad6815dfe0 2024-03-06T07:22:54.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.014+0000 7fad6d2ff640 1 -- 172.21.15.59:0/290969625 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad64031640 con 0x7fad6815dfe0 2024-03-06T07:22:54.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.014+0000 7fad6d2ff640 1 -- 172.21.15.59:0/290969625 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6402d040 con 0x7fad6815dfe0 2024-03-06T07:22:54.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.015+0000 7fad70565640 1 -- 172.21.15.59:0/290969625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6815dfe0 msgr2=0x7fad680cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.015+0000 7fad70565640 1 --2- 172.21.15.59:0/290969625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6815dfe0 0x7fad680cf2f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fad64009870 tx=0x7fad6402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.015+0000 7fad70565640 1 -- 172.21.15.59:0/290969625 shutdown_connections 2024-03-06T07:22:54.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.015+0000 7fad70565640 1 --2- 172.21.15.59:0/290969625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad680d29c0 0x7fad680b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.015+0000 7fad70565640 1 --2- 172.21.15.59:0/290969625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad6815dfe0 0x7fad680cf2f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.015+0000 7fad70565640 1 -- 172.21.15.59:0/290969625 >> 172.21.15.59:0/290969625 conn(0x7fad680d3cd0 msgr2=0x7fad680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.015+0000 7fad70565640 1 -- 172.21.15.59:0/290969625 shutdown_connections 2024-03-06T07:22:54.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.015+0000 7fad70565640 1 -- 172.21.15.59:0/290969625 wait complete. 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.016+0000 7fad70565640 1 Processor -- start 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.016+0000 7fad70565640 1 -- start start 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad70565640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad680cf830 0x7fad681547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad70565640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad680d29c0 0x7fad6815fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad70565640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad6815dfe0 0x7fad680b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad70565640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad680d1df0 con 0x7fad680cf830 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad70565640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad680d1c70 con 0x7fad680d29c0 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad70565640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6815e3c0 con 0x7fad6815dfe0 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad6e301640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad680cf830 0x7fad681547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad6e301640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad680cf830 0x7fad681547c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57780/0 (socket says 172.21.15.59:57780) 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad6e301640 1 -- 172.21.15.59:0/3193737065 learned_addr learned my addr 172.21.15.59:0/3193737065 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad6e301640 1 -- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad6815dfe0 msgr2=0x7fad680b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad6e301640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad6815dfe0 0x7fad680b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad6e301640 1 -- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad680d29c0 msgr2=0x7fad6815fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad6e301640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad680d29c0 0x7fad6815fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.017+0000 7fad6e301640 1 -- 172.21.15.59:0/3193737065 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad680b9b60 con 0x7fad680cf830 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.018+0000 7fad6e301640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad680cf830 0x7fad681547c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fad6402b930 tx=0x7fad640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.018+0000 7fad5effd640 1 -- 172.21.15.59:0/3193737065 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64040070 con 0x7fad680cf830 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.018+0000 7fad5effd640 1 -- 172.21.15.59:0/3193737065 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad64039bf0 con 0x7fad680cf830 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.018+0000 7fad70565640 1 -- 172.21.15.59:0/3193737065 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad680b92c0 con 0x7fad680cf830 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.018+0000 7fad5effd640 1 -- 172.21.15.59:0/3193737065 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64035040 con 0x7fad680cf830 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.018+0000 7fad70565640 1 -- 172.21.15.59:0/3193737065 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad680b9880 con 0x7fad680cf830 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.019+0000 7fad70565640 1 -- 172.21.15.59:0/3193737065 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6804f330 con 0x7fad680cf830 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.019+0000 7fad5effd640 1 -- 172.21.15.59:0/3193737065 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad6404b060 con 0x7fad680cf830 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.020+0000 7fad5effd640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad4c06b4d0 0x7fad4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.020+0000 7fad5effd640 1 -- 172.21.15.59:0/3193737065 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad640abd50 con 0x7fad680cf830 2024-03-06T07:22:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.020+0000 7fad6db00640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad4c06b4d0 0x7fad4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.020+0000 7fad6db00640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad4c06b4d0 0x7fad4c06d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fad680ca100 tx=0x7fad580079b0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.028+0000 7fad5effd640 1 -- 172.21.15.59:0/3193737065 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad6407ab30 con 0x7fad680cf830 2024-03-06T07:22:54.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.206+0000 7fad70565640 1 -- 172.21.15.59:0/3193737065 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad68154d00 con 0x7fad4c06b4d0 2024-03-06T07:22:54.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.213+0000 7fad5effd640 1 -- 172.21.15.59:0/3193737065 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251879 (secure 0 0 0) 0x7fad68154d00 con 0x7fad4c06b4d0 2024-03-06T07:22:54.217 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:54.219 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 -- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad4c06b4d0 msgr2=0x7fad4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad4c06b4d0 0x7fad4c06d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fad680ca100 tx=0x7fad580079b0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 -- 172.21.15.59:0/3193737065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad680cf830 msgr2=0x7fad681547c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad680cf830 0x7fad681547c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fad6402b930 tx=0x7fad640046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 -- 172.21.15.59:0/3193737065 shutdown_connections 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad4c06b4d0 0x7fad4c06d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad6815dfe0 0x7fad680b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad680d29c0 0x7fad6815fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 --2- 172.21.15.59:0/3193737065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad680cf830 0x7fad681547c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 -- 172.21.15.59:0/3193737065 >> 172.21.15.59:0/3193737065 conn(0x7fad680d3cd0 msgr2=0x7fad6806c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 -- 172.21.15.59:0/3193737065 shutdown_connections 2024-03-06T07:22:54.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.217+0000 7fad70565640 1 -- 172.21.15.59:0/3193737065 wait complete. 2024-03-06T07:22:54.229 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-03-06T07:22:53.313082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220072,"kb_used_data":3880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754648,"statfs":{"total":773094113280,"available":772868759552,"internally_reserved":0,"allocated":3973120,"data_stored":2249000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.103599"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.235416+0000","last_change":"2024-03-06T07:22:51.231309+0000","last_active":"2024-03-06T07:22:51.235416+0000","last_peered":"2024-03-06T07:22:51.235416+0000","last_clean":"2024-03-06T07:22:51.235416+0000","last_became_active":"2024-03-06T07:22:51.230973+0000","last_became_peered":"2024-03-06T07:22:51.230973+0000","last_unstale":"2024-03-06T07:22:51.235416+0000","last_undegraded":"2024-03-06T07:22:51.235416+0000","last_fullsized":"2024-03-06T07:22:51.235416+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:44.560090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231468+0000","last_change":"2024-03-06T07:22:51.227122+0000","last_active":"2024-03-06T07:22:51.231468+0000","last_peered":"2024-03-06T07:22:51.231468+0000","last_clean":"2024-03-06T07:22:51.231468+0000","last_became_active":"2024-03-06T07:22:51.226740+0000","last_became_peered":"2024-03-06T07:22:51.226740+0000","last_unstale":"2024-03-06T07:22:51.231468+0000","last_undegraded":"2024-03-06T07:22:51.231468+0000","last_fullsized":"2024-03-06T07:22:51.231468+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:18.562926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245869+0000","last_change":"2024-03-06T07:22:51.244796+0000","last_active":"2024-03-06T07:22:51.245869+0000","last_peered":"2024-03-06T07:22:51.245869+0000","last_clean":"2024-03-06T07:22:51.245869+0000","last_became_active":"2024-03-06T07:22:51.244511+0000","last_became_peered":"2024-03-06T07:22:51.244511+0000","last_unstale":"2024-03-06T07:22:51.245869+0000","last_undegraded":"2024-03-06T07:22:51.245869+0000","last_fullsized":"2024-03-06T07:22:51.245869+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:17.546892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249723+0000","last_change":"2024-03-06T07:22:51.246076+0000","last_active":"2024-03-06T07:22:51.249723+0000","last_peered":"2024-03-06T07:22:51.249723+0000","last_clean":"2024-03-06T07:22:51.249723+0000","last_became_active":"2024-03-06T07:22:51.245578+0000","last_became_peered":"2024-03-06T07:22:51.245578+0000","last_unstale":"2024-03-06T07:22:51.249723+0000","last_undegraded":"2024-03-06T07:22:51.249723+0000","last_fullsized":"2024-03-06T07:22:51.249723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:30.894809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231723+0000","last_change":"2024-03-06T07:22:51.225408+0000","last_active":"2024-03-06T07:22:51.231723+0000","last_peered":"2024-03-06T07:22:51.231723+0000","last_clean":"2024-03-06T07:22:51.231723+0000","last_became_active":"2024-03-06T07:22:51.225064+0000","last_became_peered":"2024-03-06T07:22:51.225064+0000","last_unstale":"2024-03-06T07:22:51.231723+0000","last_undegraded":"2024-03-06T07:22:51.231723+0000","last_fullsized":"2024-03-06T07:22:51.231723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:28.409637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.230930+0000","last_change":"2024-03-06T07:22:51.225396+0000","last_active":"2024-03-06T07:22:51.230930+0000","last_peered":"2024-03-06T07:22:51.230930+0000","last_clean":"2024-03-06T07:22:51.230930+0000","last_became_active":"2024-03-06T07:22:51.225106+0000","last_became_peered":"2024-03-06T07:22:51.225106+0000","last_unstale":"2024-03-06T07:22:51.230930+0000","last_undegraded":"2024-03-06T07:22:51.230930+0000","last_fullsized":"2024-03-06T07:22:51.230930+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:42.185715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.241887+0000","last_change":"2024-03-06T07:22:51.239743+0000","last_active":"2024-03-06T07:22:51.241887+0000","last_peered":"2024-03-06T07:22:51.241887+0000","last_clean":"2024-03-06T07:22:51.241887+0000","last_became_active":"2024-03-06T07:22:51.239424+0000","last_became_peered":"2024-03-06T07:22:51.239424+0000","last_unstale":"2024-03-06T07:22:51.241887+0000","last_undegraded":"2024-03-06T07:22:51.241887+0000","last_fullsized":"2024-03-06T07:22:51.241887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:40.361915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238577+0000","last_change":"2024-03-06T07:22:51.233614+0000","last_active":"2024-03-06T07:22:51.238577+0000","last_peered":"2024-03-06T07:22:51.238577+0000","last_clean":"2024-03-06T07:22:51.238577+0000","last_became_active":"2024-03-06T07:22:51.233350+0000","last_became_peered":"2024-03-06T07:22:51.233350+0000","last_unstale":"2024-03-06T07:22:51.238577+0000","last_undegraded":"2024-03-06T07:22:51.238577+0000","last_fullsized":"2024-03-06T07:22:51.238577+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:08.399514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248660+0000","last_change":"2024-03-06T07:22:51.248089+0000","last_active":"2024-03-06T07:22:51.248660+0000","last_peered":"2024-03-06T07:22:51.248660+0000","last_clean":"2024-03-06T07:22:51.248660+0000","last_became_active":"2024-03-06T07:22:51.247739+0000","last_became_peered":"2024-03-06T07:22:51.247739+0000","last_unstale":"2024-03-06T07:22:51.248660+0000","last_undegraded":"2024-03-06T07:22:51.248660+0000","last_fullsized":"2024-03-06T07:22:51.248660+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:30.148542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248430+0000","last_change":"2024-03-06T07:22:51.241023+0000","last_active":"2024-03-06T07:22:51.248430+0000","last_peered":"2024-03-06T07:22:51.248430+0000","last_clean":"2024-03-06T07:22:51.248430+0000","last_became_active":"2024-03-06T07:22:51.240743+0000","last_became_peered":"2024-03-06T07:22:51.240743+0000","last_unstale":"2024-03-06T07:22:51.248430+0000","last_undegraded":"2024-03-06T07:22:51.248430+0000","last_fullsized":"2024-03-06T07:22:51.248430+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:34.881905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247894+0000","last_change":"2024-03-06T07:22:51.243419+0000","last_active":"2024-03-06T07:22:51.247894+0000","last_peered":"2024-03-06T07:22:51.247894+0000","last_clean":"2024-03-06T07:22:51.247894+0000","last_became_active":"2024-03-06T07:22:51.243125+0000","last_became_peered":"2024-03-06T07:22:51.243125+0000","last_unstale":"2024-03-06T07:22:51.247894+0000","last_undegraded":"2024-03-06T07:22:51.247894+0000","last_fullsized":"2024-03-06T07:22:51.247894+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:34.055263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228046+0000","last_change":"2024-03-06T07:22:51.225041+0000","last_active":"2024-03-06T07:22:51.228046+0000","last_peered":"2024-03-06T07:22:51.228046+0000","last_clean":"2024-03-06T07:22:51.228046+0000","last_became_active":"2024-03-06T07:22:51.224729+0000","last_became_peered":"2024-03-06T07:22:51.224729+0000","last_unstale":"2024-03-06T07:22:51.228046+0000","last_undegraded":"2024-03-06T07:22:51.228046+0000","last_fullsized":"2024-03-06T07:22:51.228046+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:24.050069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.241246+0000","last_change":"2024-03-06T07:22:51.240331+0000","last_active":"2024-03-06T07:22:51.241246+0000","last_peered":"2024-03-06T07:22:51.241246+0000","last_clean":"2024-03-06T07:22:51.241246+0000","last_became_active":"2024-03-06T07:22:51.240044+0000","last_became_peered":"2024-03-06T07:22:51.240044+0000","last_unstale":"2024-03-06T07:22:51.241246+0000","last_undegraded":"2024-03-06T07:22:51.241246+0000","last_fullsized":"2024-03-06T07:22:51.241246+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:50.207597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.232059+0000","last_change":"2024-03-06T07:22:51.231053+0000","last_active":"2024-03-06T07:22:51.232059+0000","last_peered":"2024-03-06T07:22:51.232059+0000","last_clean":"2024-03-06T07:22:51.232059+0000","last_became_active":"2024-03-06T07:22:51.230704+0000","last_became_peered":"2024-03-06T07:22:51.230704+0000","last_unstale":"2024-03-06T07:22:51.232059+0000","last_undegraded":"2024-03-06T07:22:51.232059+0000","last_fullsized":"2024-03-06T07:22:51.232059+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:08.961572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238251+0000","last_change":"2024-03-06T07:22:51.231595+0000","last_active":"2024-03-06T07:22:51.238251+0000","last_peered":"2024-03-06T07:22:51.238251+0000","last_clean":"2024-03-06T07:22:51.238251+0000","last_became_active":"2024-03-06T07:22:51.231339+0000","last_became_peered":"2024-03-06T07:22:51.231339+0000","last_unstale":"2024-03-06T07:22:51.238251+0000","last_undegraded":"2024-03-06T07:22:51.238251+0000","last_fullsized":"2024-03-06T07:22:51.238251+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:01.602991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248006+0000","last_change":"2024-03-06T07:22:51.244736+0000","last_active":"2024-03-06T07:22:51.248006+0000","last_peered":"2024-03-06T07:22:51.248006+0000","last_clean":"2024-03-06T07:22:51.248006+0000","last_became_active":"2024-03-06T07:22:51.244482+0000","last_became_peered":"2024-03-06T07:22:51.244482+0000","last_unstale":"2024-03-06T07:22:51.248006+0000","last_undegraded":"2024-03-06T07:22:51.248006+0000","last_fullsized":"2024-03-06T07:22:51.248006+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:17.546892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238082+0000","last_change":"2024-03-06T07:22:51.233354+0000","last_active":"2024-03-06T07:22:51.238082+0000","last_peered":"2024-03-06T07:22:51.238082+0000","last_clean":"2024-03-06T07:22:51.238082+0000","last_became_active":"2024-03-06T07:22:51.233097+0000","last_became_peered":"2024-03-06T07:22:51.233097+0000","last_unstale":"2024-03-06T07:22:51.238082+0000","last_undegraded":"2024-03-06T07:22:51.238082+0000","last_fullsized":"2024-03-06T07:22:51.238082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:34.881905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249678+0000","last_change":"2024-03-06T07:22:51.244255+0000","last_active":"2024-03-06T07:22:51.249678+0000","last_peered":"2024-03-06T07:22:51.249678+0000","last_clean":"2024-03-06T07:22:51.249678+0000","last_became_active":"2024-03-06T07:22:51.243950+0000","last_became_peered":"2024-03-06T07:22:51.243950+0000","last_unstale":"2024-03-06T07:22:51.249678+0000","last_undegraded":"2024-03-06T07:22:51.249678+0000","last_fullsized":"2024-03-06T07:22:51.249678+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:51.604037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249653+0000","last_change":"2024-03-06T07:22:51.245682+0000","last_active":"2024-03-06T07:22:51.249653+0000","last_peered":"2024-03-06T07:22:51.249653+0000","last_clean":"2024-03-06T07:22:51.249653+0000","last_became_active":"2024-03-06T07:22:51.245357+0000","last_became_peered":"2024-03-06T07:22:51.245357+0000","last_unstale":"2024-03-06T07:22:51.249653+0000","last_undegraded":"2024-03-06T07:22:51.249653+0000","last_fullsized":"2024-03-06T07:22:51.249653+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:02.562762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231482+0000","last_change":"2024-03-06T07:22:51.226122+0000","last_active":"2024-03-06T07:22:51.231482+0000","last_peered":"2024-03-06T07:22:51.231482+0000","last_clean":"2024-03-06T07:22:51.231482+0000","last_became_active":"2024-03-06T07:22:51.225834+0000","last_became_peered":"2024-03-06T07:22:51.225834+0000","last_unstale":"2024-03-06T07:22:51.231482+0000","last_undegraded":"2024-03-06T07:22:51.231482+0000","last_fullsized":"2024-03-06T07:22:51.231482+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:22.660918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238913+0000","last_change":"2024-03-06T07:22:51.237756+0000","last_active":"2024-03-06T07:22:51.238913+0000","last_peered":"2024-03-06T07:22:51.238913+0000","last_clean":"2024-03-06T07:22:51.238913+0000","last_became_active":"2024-03-06T07:22:51.237478+0000","last_became_peered":"2024-03-06T07:22:51.237478+0000","last_unstale":"2024-03-06T07:22:51.238913+0000","last_undegraded":"2024-03-06T07:22:51.238913+0000","last_fullsized":"2024-03-06T07:22:51.238913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:44.258713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238345+0000","last_change":"2024-03-06T07:22:51.234722+0000","last_active":"2024-03-06T07:22:51.238345+0000","last_peered":"2024-03-06T07:22:51.238345+0000","last_clean":"2024-03-06T07:22:51.238345+0000","last_became_active":"2024-03-06T07:22:51.233880+0000","last_became_peered":"2024-03-06T07:22:51.233880+0000","last_unstale":"2024-03-06T07:22:51.238345+0000","last_undegraded":"2024-03-06T07:22:51.238345+0000","last_fullsized":"2024-03-06T07:22:51.238345+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:58.557417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.232273+0000","last_change":"2024-03-06T07:22:51.230552+0000","last_active":"2024-03-06T07:22:51.232273+0000","last_peered":"2024-03-06T07:22:51.232273+0000","last_clean":"2024-03-06T07:22:51.232273+0000","last_became_active":"2024-03-06T07:22:51.230276+0000","last_became_peered":"2024-03-06T07:22:51.230276+0000","last_unstale":"2024-03-06T07:22:51.232273+0000","last_undegraded":"2024-03-06T07:22:51.232273+0000","last_fullsized":"2024-03-06T07:22:51.232273+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:44.320357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.227197+0000","last_change":"2024-03-06T07:22:51.225024+0000","last_active":"2024-03-06T07:22:51.227197+0000","last_peered":"2024-03-06T07:22:51.227197+0000","last_clean":"2024-03-06T07:22:51.227197+0000","last_became_active":"2024-03-06T07:22:51.224697+0000","last_became_peered":"2024-03-06T07:22:51.224697+0000","last_unstale":"2024-03-06T07:22:51.227197+0000","last_undegraded":"2024-03-06T07:22:51.227197+0000","last_fullsized":"2024-03-06T07:22:51.227197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:08.399514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248777+0000","last_change":"2024-03-06T07:22:51.243340+0000","last_active":"2024-03-06T07:22:51.248777+0000","last_peered":"2024-03-06T07:22:51.248777+0000","last_clean":"2024-03-06T07:22:51.248777+0000","last_became_active":"2024-03-06T07:22:51.243020+0000","last_became_peered":"2024-03-06T07:22:51.243020+0000","last_unstale":"2024-03-06T07:22:51.248777+0000","last_undegraded":"2024-03-06T07:22:51.248777+0000","last_fullsized":"2024-03-06T07:22:51.248777+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:24.050069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.242355+0000","last_change":"2024-03-06T07:22:51.234440+0000","last_active":"2024-03-06T07:22:51.242355+0000","last_peered":"2024-03-06T07:22:51.242355+0000","last_clean":"2024-03-06T07:22:51.242355+0000","last_became_active":"2024-03-06T07:22:51.234075+0000","last_became_peered":"2024-03-06T07:22:51.234075+0000","last_unstale":"2024-03-06T07:22:51.242355+0000","last_undegraded":"2024-03-06T07:22:51.242355+0000","last_fullsized":"2024-03-06T07:22:51.242355+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:51.605065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249871+0000","last_change":"2024-03-06T07:22:51.247175+0000","last_active":"2024-03-06T07:22:51.249871+0000","last_peered":"2024-03-06T07:22:51.249871+0000","last_clean":"2024-03-06T07:22:51.249871+0000","last_became_active":"2024-03-06T07:22:51.246606+0000","last_became_peered":"2024-03-06T07:22:51.246606+0000","last_unstale":"2024-03-06T07:22:51.249871+0000","last_undegraded":"2024-03-06T07:22:51.249871+0000","last_fullsized":"2024-03-06T07:22:51.249871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:01.602991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228070+0000","last_change":"2024-03-06T07:22:51.227049+0000","last_active":"2024-03-06T07:22:51.228070+0000","last_peered":"2024-03-06T07:22:51.228070+0000","last_clean":"2024-03-06T07:22:51.228070+0000","last_became_active":"2024-03-06T07:22:51.226723+0000","last_became_peered":"2024-03-06T07:22:51.226723+0000","last_unstale":"2024-03-06T07:22:51.228070+0000","last_undegraded":"2024-03-06T07:22:51.228070+0000","last_fullsized":"2024-03-06T07:22:51.228070+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:30.894809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238365+0000","last_change":"2024-03-06T07:22:51.236037+0000","last_active":"2024-03-06T07:22:51.238365+0000","last_peered":"2024-03-06T07:22:51.238365+0000","last_clean":"2024-03-06T07:22:51.238365+0000","last_became_active":"2024-03-06T07:22:51.235720+0000","last_became_peered":"2024-03-06T07:22:51.235720+0000","last_unstale":"2024-03-06T07:22:51.238365+0000","last_undegraded":"2024-03-06T07:22:51.238365+0000","last_fullsized":"2024-03-06T07:22:51.238365+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:54.744059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.246827+0000","last_change":"2024-03-06T07:22:51.241408+0000","last_active":"2024-03-06T07:22:51.246827+0000","last_peered":"2024-03-06T07:22:51.246827+0000","last_clean":"2024-03-06T07:22:51.246827+0000","last_became_active":"2024-03-06T07:22:51.239979+0000","last_became_peered":"2024-03-06T07:22:51.239979+0000","last_unstale":"2024-03-06T07:22:51.246827+0000","last_undegraded":"2024-03-06T07:22:51.246827+0000","last_fullsized":"2024-03-06T07:22:51.246827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:18.562926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.239055+0000","last_change":"2024-03-06T07:22:51.235307+0000","last_active":"2024-03-06T07:22:51.239055+0000","last_peered":"2024-03-06T07:22:51.239055+0000","last_clean":"2024-03-06T07:22:51.239055+0000","last_became_active":"2024-03-06T07:22:51.234365+0000","last_became_peered":"2024-03-06T07:22:51.234365+0000","last_unstale":"2024-03-06T07:22:51.239055+0000","last_undegraded":"2024-03-06T07:22:51.239055+0000","last_fullsized":"2024-03-06T07:22:51.239055+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:49.157693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.239262+0000","last_change":"2024-03-06T07:22:51.236795+0000","last_active":"2024-03-06T07:22:51.239262+0000","last_peered":"2024-03-06T07:22:51.239262+0000","last_clean":"2024-03-06T07:22:51.239262+0000","last_became_active":"2024-03-06T07:22:51.236368+0000","last_became_peered":"2024-03-06T07:22:51.236368+0000","last_unstale":"2024-03-06T07:22:51.239262+0000","last_undegraded":"2024-03-06T07:22:51.239262+0000","last_fullsized":"2024-03-06T07:22:51.239262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:11.865087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.234856+0000","last_change":"2024-03-06T07:22:51.233257+0000","last_active":"2024-03-06T07:22:51.234856+0000","last_peered":"2024-03-06T07:22:51.234856+0000","last_clean":"2024-03-06T07:22:51.234856+0000","last_became_active":"2024-03-06T07:22:51.232888+0000","last_became_peered":"2024-03-06T07:22:51.232888+0000","last_unstale":"2024-03-06T07:22:51.234856+0000","last_undegraded":"2024-03-06T07:22:51.234856+0000","last_fullsized":"2024-03-06T07:22:51.234856+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:39.459733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.244467+0000","last_change":"2024-03-06T07:22:50.229694+0000","last_active":"2024-03-06T07:22:51.244467+0000","last_peered":"2024-03-06T07:22:51.244467+0000","last_clean":"2024-03-06T07:22:51.244467+0000","last_became_active":"2024-03-06T07:22:50.229438+0000","last_became_peered":"2024-03-06T07:22:50.229438+0000","last_unstale":"2024-03-06T07:22:51.244467+0000","last_undegraded":"2024-03-06T07:22:51.244467+0000","last_fullsized":"2024-03-06T07:22:51.244467+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:53.741735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.239155+0000","last_change":"2024-03-06T07:22:51.236821+0000","last_active":"2024-03-06T07:22:51.239155+0000","last_peered":"2024-03-06T07:22:51.239155+0000","last_clean":"2024-03-06T07:22:51.239155+0000","last_became_active":"2024-03-06T07:22:51.236521+0000","last_became_peered":"2024-03-06T07:22:51.236521+0000","last_unstale":"2024-03-06T07:22:51.239155+0000","last_undegraded":"2024-03-06T07:22:51.239155+0000","last_fullsized":"2024-03-06T07:22:51.239155+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:30.148542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247159+0000","last_change":"2024-03-06T07:22:50.229519+0000","last_active":"2024-03-06T07:22:51.247159+0000","last_peered":"2024-03-06T07:22:51.247159+0000","last_clean":"2024-03-06T07:22:51.247159+0000","last_became_active":"2024-03-06T07:22:50.229198+0000","last_became_peered":"2024-03-06T07:22:50.229198+0000","last_unstale":"2024-03-06T07:22:51.247159+0000","last_undegraded":"2024-03-06T07:22:51.247159+0000","last_fullsized":"2024-03-06T07:22:51.247159+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:43.256389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248356+0000","last_change":"2024-03-06T07:22:51.246439+0000","last_active":"2024-03-06T07:22:51.248356+0000","last_peered":"2024-03-06T07:22:51.248356+0000","last_clean":"2024-03-06T07:22:51.248356+0000","last_became_active":"2024-03-06T07:22:51.246117+0000","last_became_peered":"2024-03-06T07:22:51.246117+0000","last_unstale":"2024-03-06T07:22:51.248356+0000","last_undegraded":"2024-03-06T07:22:51.248356+0000","last_fullsized":"2024-03-06T07:22:51.248356+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:49.157693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247321+0000","last_change":"2024-03-06T07:22:50.227158+0000","last_active":"2024-03-06T07:22:51.247321+0000","last_peered":"2024-03-06T07:22:51.247321+0000","last_clean":"2024-03-06T07:22:51.247321+0000","last_became_active":"2024-03-06T07:22:50.226872+0000","last_became_peered":"2024-03-06T07:22:50.226872+0000","last_unstale":"2024-03-06T07:22:51.247321+0000","last_undegraded":"2024-03-06T07:22:51.247321+0000","last_fullsized":"2024-03-06T07:22:51.247321+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:33.879582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249767+0000","last_change":"2024-03-06T07:22:51.246600+0000","last_active":"2024-03-06T07:22:51.249767+0000","last_peered":"2024-03-06T07:22:51.249767+0000","last_clean":"2024-03-06T07:22:51.249767+0000","last_became_active":"2024-03-06T07:22:51.246328+0000","last_became_peered":"2024-03-06T07:22:51.246328+0000","last_unstale":"2024-03-06T07:22:51.249767+0000","last_undegraded":"2024-03-06T07:22:51.249767+0000","last_fullsized":"2024-03-06T07:22:51.249767+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:40.361915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.242632+0000","last_change":"2024-03-06T07:22:50.234493+0000","last_active":"2024-03-06T07:22:51.242632+0000","last_peered":"2024-03-06T07:22:51.242632+0000","last_clean":"2024-03-06T07:22:51.242632+0000","last_became_active":"2024-03-06T07:22:50.234216+0000","last_became_peered":"2024-03-06T07:22:50.234216+0000","last_unstale":"2024-03-06T07:22:51.242632+0000","last_undegraded":"2024-03-06T07:22:51.242632+0000","last_fullsized":"2024-03-06T07:22:51.242632+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:00.497208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.240165+0000","last_change":"2024-03-06T07:22:51.238298+0000","last_active":"2024-03-06T07:22:51.240165+0000","last_peered":"2024-03-06T07:22:51.240165+0000","last_clean":"2024-03-06T07:22:51.240165+0000","last_became_active":"2024-03-06T07:22:51.237989+0000","last_became_peered":"2024-03-06T07:22:51.237989+0000","last_unstale":"2024-03-06T07:22:51.240165+0000","last_undegraded":"2024-03-06T07:22:51.240165+0000","last_fullsized":"2024-03-06T07:22:51.240165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:30.894809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249825+0000","last_change":"2024-03-06T07:22:50.231916+0000","last_active":"2024-03-06T07:22:51.249825+0000","last_peered":"2024-03-06T07:22:51.249825+0000","last_clean":"2024-03-06T07:22:51.249825+0000","last_became_active":"2024-03-06T07:22:50.231660+0000","last_became_peered":"2024-03-06T07:22:50.231660+0000","last_unstale":"2024-03-06T07:22:51.249825+0000","last_undegraded":"2024-03-06T07:22:51.249825+0000","last_fullsized":"2024-03-06T07:22:51.249825+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:06.756105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228972+0000","last_change":"2024-03-06T07:22:51.228164+0000","last_active":"2024-03-06T07:22:51.228972+0000","last_peered":"2024-03-06T07:22:51.228972+0000","last_clean":"2024-03-06T07:22:51.228972+0000","last_became_active":"2024-03-06T07:22:51.227851+0000","last_became_peered":"2024-03-06T07:22:51.227851+0000","last_unstale":"2024-03-06T07:22:51.228972+0000","last_undegraded":"2024-03-06T07:22:51.228972+0000","last_fullsized":"2024-03-06T07:22:51.228972+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:40.361915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.252429+0000","last_change":"2024-03-06T07:22:50.231245+0000","last_active":"2024-03-06T07:22:51.252429+0000","last_peered":"2024-03-06T07:22:51.252429+0000","last_clean":"2024-03-06T07:22:51.252429+0000","last_became_active":"2024-03-06T07:22:50.230985+0000","last_became_peered":"2024-03-06T07:22:50.230985+0000","last_unstale":"2024-03-06T07:22:51.252429+0000","last_undegraded":"2024-03-06T07:22:51.252429+0000","last_fullsized":"2024-03-06T07:22:51.252429+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:40.080368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238379+0000","last_change":"2024-03-06T07:22:51.235842+0000","last_active":"2024-03-06T07:22:51.238379+0000","last_peered":"2024-03-06T07:22:51.238379+0000","last_clean":"2024-03-06T07:22:51.238379+0000","last_became_active":"2024-03-06T07:22:51.235518+0000","last_became_peered":"2024-03-06T07:22:51.235518+0000","last_unstale":"2024-03-06T07:22:51.238379+0000","last_undegraded":"2024-03-06T07:22:51.238379+0000","last_fullsized":"2024-03-06T07:22:51.238379+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:27.272822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.220166+0000","last_change":"2024-03-06T07:22:50.225708+0000","last_active":"2024-03-06T07:22:51.220166+0000","last_peered":"2024-03-06T07:22:51.220166+0000","last_clean":"2024-03-06T07:22:51.220166+0000","last_became_active":"2024-03-06T07:22:50.225428+0000","last_became_peered":"2024-03-06T07:22:50.225428+0000","last_unstale":"2024-03-06T07:22:51.220166+0000","last_undegraded":"2024-03-06T07:22:51.220166+0000","last_fullsized":"2024-03-06T07:22:51.220166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:10.862763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245033+0000","last_change":"2024-03-06T07:22:51.240183+0000","last_active":"2024-03-06T07:22:51.245033+0000","last_peered":"2024-03-06T07:22:51.245033+0000","last_clean":"2024-03-06T07:22:51.245033+0000","last_became_active":"2024-03-06T07:22:51.239883+0000","last_became_peered":"2024-03-06T07:22:51.239883+0000","last_unstale":"2024-03-06T07:22:51.245033+0000","last_undegraded":"2024-03-06T07:22:51.245033+0000","last_fullsized":"2024-03-06T07:22:51.245033+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:31.079398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228210+0000","last_change":"2024-03-06T07:22:50.230105+0000","last_active":"2024-03-06T07:22:51.228210+0000","last_peered":"2024-03-06T07:22:51.228210+0000","last_clean":"2024-03-06T07:22:51.228210+0000","last_became_active":"2024-03-06T07:22:50.229822+0000","last_became_peered":"2024-03-06T07:22:50.229822+0000","last_unstale":"2024-03-06T07:22:51.228210+0000","last_undegraded":"2024-03-06T07:22:51.228210+0000","last_fullsized":"2024-03-06T07:22:51.228210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:39.604948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228021+0000","last_change":"2024-03-06T07:22:51.225757+0000","last_active":"2024-03-06T07:22:51.228021+0000","last_peered":"2024-03-06T07:22:51.228021+0000","last_clean":"2024-03-06T07:22:51.228021+0000","last_became_active":"2024-03-06T07:22:51.225444+0000","last_became_peered":"2024-03-06T07:22:51.225444+0000","last_unstale":"2024-03-06T07:22:51.228021+0000","last_undegraded":"2024-03-06T07:22:51.228021+0000","last_fullsized":"2024-03-06T07:22:51.228021+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:11.865087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.219619+0000","last_change":"2024-03-06T07:22:50.230222+0000","last_active":"2024-03-06T07:22:51.219619+0000","last_peered":"2024-03-06T07:22:51.219619+0000","last_clean":"2024-03-06T07:22:51.219619+0000","last_became_active":"2024-03-06T07:22:50.229916+0000","last_became_peered":"2024-03-06T07:22:50.229916+0000","last_unstale":"2024-03-06T07:22:51.219619+0000","last_undegraded":"2024-03-06T07:22:51.219619+0000","last_fullsized":"2024-03-06T07:22:51.219619+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:07.397191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245042+0000","last_change":"2024-03-06T07:22:51.241804+0000","last_active":"2024-03-06T07:22:51.245042+0000","last_peered":"2024-03-06T07:22:51.245042+0000","last_clean":"2024-03-06T07:22:51.245042+0000","last_became_active":"2024-03-06T07:22:51.241468+0000","last_became_peered":"2024-03-06T07:22:51.241468+0000","last_unstale":"2024-03-06T07:22:51.245042+0000","last_undegraded":"2024-03-06T07:22:51.245042+0000","last_fullsized":"2024-03-06T07:22:51.245042+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:46.593686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.224405+0000","last_change":"2024-03-06T07:22:50.231068+0000","last_active":"2024-03-06T07:22:51.224405+0000","last_peered":"2024-03-06T07:22:51.224405+0000","last_clean":"2024-03-06T07:22:51.224405+0000","last_became_active":"2024-03-06T07:22:50.230704+0000","last_became_peered":"2024-03-06T07:22:50.230704+0000","last_unstale":"2024-03-06T07:22:51.224405+0000","last_undegraded":"2024-03-06T07:22:51.224405+0000","last_fullsized":"2024-03-06T07:22:51.224405+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:43.557767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231600+0000","last_change":"2024-03-06T07:22:51.228854+0000","last_active":"2024-03-06T07:22:51.231600+0000","last_peered":"2024-03-06T07:22:51.231600+0000","last_clean":"2024-03-06T07:22:51.231600+0000","last_became_active":"2024-03-06T07:22:51.228572+0000","last_became_peered":"2024-03-06T07:22:51.228572+0000","last_unstale":"2024-03-06T07:22:51.231600+0000","last_undegraded":"2024-03-06T07:22:51.231600+0000","last_fullsized":"2024-03-06T07:22:51.231600+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:17.546892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248474+0000","last_change":"2024-03-06T07:22:50.231134+0000","last_active":"2024-03-06T07:22:51.248474+0000","last_peered":"2024-03-06T07:22:51.248474+0000","last_clean":"2024-03-06T07:22:51.248474+0000","last_became_active":"2024-03-06T07:22:50.230815+0000","last_became_peered":"2024-03-06T07:22:50.230815+0000","last_unstale":"2024-03-06T07:22:51.248474+0000","last_undegraded":"2024-03-06T07:22:51.248474+0000","last_fullsized":"2024-03-06T07:22:51.248474+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:30.077075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245849+0000","last_change":"2024-03-06T07:22:51.244433+0000","last_active":"2024-03-06T07:22:51.245849+0000","last_peered":"2024-03-06T07:22:51.245849+0000","last_clean":"2024-03-06T07:22:51.245849+0000","last_became_active":"2024-03-06T07:22:51.244155+0000","last_became_peered":"2024-03-06T07:22:51.244155+0000","last_unstale":"2024-03-06T07:22:51.245849+0000","last_undegraded":"2024-03-06T07:22:51.245849+0000","last_fullsized":"2024-03-06T07:22:51.245849+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:34.055263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.252692+0000","last_change":"2024-03-06T07:22:50.228203+0000","last_active":"2024-03-06T07:22:51.252692+0000","last_peered":"2024-03-06T07:22:51.252692+0000","last_clean":"2024-03-06T07:22:51.252692+0000","last_became_active":"2024-03-06T07:22:50.227909+0000","last_became_peered":"2024-03-06T07:22:50.227909+0000","last_unstale":"2024-03-06T07:22:51.252692+0000","last_undegraded":"2024-03-06T07:22:51.252692+0000","last_fullsized":"2024-03-06T07:22:51.252692+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.047745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248653+0000","last_change":"2024-03-06T07:22:51.241258+0000","last_active":"2024-03-06T07:22:51.248653+0000","last_peered":"2024-03-06T07:22:51.248653+0000","last_clean":"2024-03-06T07:22:51.248653+0000","last_became_active":"2024-03-06T07:22:51.240607+0000","last_became_peered":"2024-03-06T07:22:51.240607+0000","last_unstale":"2024-03-06T07:22:51.248653+0000","last_undegraded":"2024-03-06T07:22:51.248653+0000","last_fullsized":"2024-03-06T07:22:51.248653+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:08.399514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247345+0000","last_change":"2024-03-06T07:22:50.232757+0000","last_active":"2024-03-06T07:22:51.247345+0000","last_peered":"2024-03-06T07:22:51.247345+0000","last_clean":"2024-03-06T07:22:51.247345+0000","last_became_active":"2024-03-06T07:22:50.232486+0000","last_became_peered":"2024-03-06T07:22:50.232486+0000","last_unstale":"2024-03-06T07:22:51.247345+0000","last_undegraded":"2024-03-06T07:22:51.247345+0000","last_fullsized":"2024-03-06T07:22:51.247345+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:13.376099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231445+0000","last_change":"2024-03-06T07:22:51.227945+0000","last_active":"2024-03-06T07:22:51.231445+0000","last_peered":"2024-03-06T07:22:51.231445+0000","last_clean":"2024-03-06T07:22:51.231445+0000","last_became_active":"2024-03-06T07:22:51.227662+0000","last_became_peered":"2024-03-06T07:22:51.227662+0000","last_unstale":"2024-03-06T07:22:51.231445+0000","last_undegraded":"2024-03-06T07:22:51.231445+0000","last_fullsized":"2024-03-06T07:22:51.231445+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:34.055263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238170+0000","last_change":"2024-03-06T07:22:50.230923+0000","last_active":"2024-03-06T07:22:51.238170+0000","last_peered":"2024-03-06T07:22:51.238170+0000","last_clean":"2024-03-06T07:22:51.238170+0000","last_became_active":"2024-03-06T07:22:50.229807+0000","last_became_peered":"2024-03-06T07:22:50.229807+0000","last_unstale":"2024-03-06T07:22:51.238170+0000","last_undegraded":"2024-03-06T07:22:51.238170+0000","last_fullsized":"2024-03-06T07:22:51.238170+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:29.146219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.244867+0000","last_change":"2024-03-06T07:22:51.240818+0000","last_active":"2024-03-06T07:22:51.244867+0000","last_peered":"2024-03-06T07:22:51.244867+0000","last_clean":"2024-03-06T07:22:51.244867+0000","last_became_active":"2024-03-06T07:22:51.240047+0000","last_became_peered":"2024-03-06T07:22:51.240047+0000","last_unstale":"2024-03-06T07:22:51.244867+0000","last_undegraded":"2024-03-06T07:22:51.244867+0000","last_fullsized":"2024-03-06T07:22:51.244867+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:42.185715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.220320+0000","last_change":"2024-03-06T07:22:50.229203+0000","last_active":"2024-03-06T07:22:51.220320+0000","last_peered":"2024-03-06T07:22:51.220320+0000","last_clean":"2024-03-06T07:22:51.220320+0000","last_became_active":"2024-03-06T07:22:50.228891+0000","last_became_peered":"2024-03-06T07:22:50.228891+0000","last_unstale":"2024-03-06T07:22:51.220320+0000","last_undegraded":"2024-03-06T07:22:51.220320+0000","last_fullsized":"2024-03-06T07:22:51.220320+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:29.892486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.239074+0000","last_change":"2024-03-06T07:22:51.237539+0000","last_active":"2024-03-06T07:22:51.239074+0000","last_peered":"2024-03-06T07:22:51.239074+0000","last_clean":"2024-03-06T07:22:51.239074+0000","last_became_active":"2024-03-06T07:22:51.237244+0000","last_became_peered":"2024-03-06T07:22:51.237244+0000","last_unstale":"2024-03-06T07:22:51.239074+0000","last_undegraded":"2024-03-06T07:22:51.239074+0000","last_fullsized":"2024-03-06T07:22:51.239074+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:44.320357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249642+0000","last_change":"2024-03-06T07:22:50.226418+0000","last_active":"2024-03-06T07:22:51.249642+0000","last_peered":"2024-03-06T07:22:51.249642+0000","last_clean":"2024-03-06T07:22:51.249642+0000","last_became_active":"2024-03-06T07:22:50.226146+0000","last_became_peered":"2024-03-06T07:22:50.226146+0000","last_unstale":"2024-03-06T07:22:51.249642+0000","last_undegraded":"2024-03-06T07:22:51.249642+0000","last_fullsized":"2024-03-06T07:22:51.249642+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:39.359591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245159+0000","last_change":"2024-03-06T07:22:51.242081+0000","last_active":"2024-03-06T07:22:51.245159+0000","last_peered":"2024-03-06T07:22:51.245159+0000","last_clean":"2024-03-06T07:22:51.245159+0000","last_became_active":"2024-03-06T07:22:51.241813+0000","last_became_peered":"2024-03-06T07:22:51.241813+0000","last_unstale":"2024-03-06T07:22:51.245159+0000","last_undegraded":"2024-03-06T07:22:51.245159+0000","last_fullsized":"2024-03-06T07:22:51.245159+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:44.560090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248341+0000","last_change":"2024-03-06T07:22:50.225669+0000","last_active":"2024-03-06T07:22:51.248341+0000","last_peered":"2024-03-06T07:22:51.248341+0000","last_clean":"2024-03-06T07:22:51.248341+0000","last_became_active":"2024-03-06T07:22:50.225221+0000","last_became_peered":"2024-03-06T07:22:50.225221+0000","last_unstale":"2024-03-06T07:22:51.248341+0000","last_undegraded":"2024-03-06T07:22:51.248341+0000","last_fullsized":"2024-03-06T07:22:51.248341+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:43.318033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238287+0000","last_change":"2024-03-06T07:22:51.236133+0000","last_active":"2024-03-06T07:22:51.238287+0000","last_peered":"2024-03-06T07:22:51.238287+0000","last_clean":"2024-03-06T07:22:51.238287+0000","last_became_active":"2024-03-06T07:22:51.235827+0000","last_became_peered":"2024-03-06T07:22:51.235827+0000","last_unstale":"2024-03-06T07:22:51.238287+0000","last_undegraded":"2024-03-06T07:22:51.238287+0000","last_fullsized":"2024-03-06T07:22:51.238287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:42.185715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.246563+0000","last_change":"2024-03-06T07:22:50.230566+0000","last_active":"2024-03-06T07:22:51.246563+0000","last_peered":"2024-03-06T07:22:51.246563+0000","last_clean":"2024-03-06T07:22:51.246563+0000","last_became_active":"2024-03-06T07:22:50.230310+0000","last_became_peered":"2024-03-06T07:22:50.230310+0000","last_unstale":"2024-03-06T07:22:51.246563+0000","last_undegraded":"2024-03-06T07:22:51.246563+0000","last_fullsized":"2024-03-06T07:22:51.246563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:51.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249700+0000","last_change":"2024-03-06T07:22:51.245706+0000","last_active":"2024-03-06T07:22:51.249700+0000","last_peered":"2024-03-06T07:22:51.249700+0000","last_clean":"2024-03-06T07:22:51.249700+0000","last_became_active":"2024-03-06T07:22:51.244432+0000","last_became_peered":"2024-03-06T07:22:51.244432+0000","last_unstale":"2024-03-06T07:22:51.249700+0000","last_undegraded":"2024-03-06T07:22:51.249700+0000","last_fullsized":"2024-03-06T07:22:51.249700+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:11.865087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247483+0000","last_change":"2024-03-06T07:22:50.232948+0000","last_active":"2024-03-06T07:22:51.247483+0000","last_peered":"2024-03-06T07:22:51.247483+0000","last_clean":"2024-03-06T07:22:51.247483+0000","last_became_active":"2024-03-06T07:22:50.232618+0000","last_became_peered":"2024-03-06T07:22:50.232618+0000","last_unstale":"2024-03-06T07:22:51.247483+0000","last_undegraded":"2024-03-06T07:22:51.247483+0000","last_fullsized":"2024-03-06T07:22:51.247483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:24.567500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.240860+0000","last_change":"2024-03-06T07:22:51.235491+0000","last_active":"2024-03-06T07:22:51.240860+0000","last_peered":"2024-03-06T07:22:51.240860+0000","last_clean":"2024-03-06T07:22:51.240860+0000","last_became_active":"2024-03-06T07:22:51.235220+0000","last_became_peered":"2024-03-06T07:22:51.235220+0000","last_unstale":"2024-03-06T07:22:51.240860+0000","last_undegraded":"2024-03-06T07:22:51.240860+0000","last_fullsized":"2024-03-06T07:22:51.240860+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:14.378422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231165+0000","last_change":"2024-03-06T07:22:50.232635+0000","last_active":"2024-03-06T07:22:51.231165+0000","last_peered":"2024-03-06T07:22:51.231165+0000","last_clean":"2024-03-06T07:22:51.231165+0000","last_became_active":"2024-03-06T07:22:50.232378+0000","last_became_peered":"2024-03-06T07:22:50.232378+0000","last_unstale":"2024-03-06T07:22:51.231165+0000","last_undegraded":"2024-03-06T07:22:51.231165+0000","last_fullsized":"2024-03-06T07:22:51.231165+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:10.052396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.241129+0000","last_change":"2024-03-06T07:22:51.236866+0000","last_active":"2024-03-06T07:22:51.241129+0000","last_peered":"2024-03-06T07:22:51.241129+0000","last_clean":"2024-03-06T07:22:51.241129+0000","last_became_active":"2024-03-06T07:22:51.236557+0000","last_became_peered":"2024-03-06T07:22:51.236557+0000","last_unstale":"2024-03-06T07:22:51.241129+0000","last_undegraded":"2024-03-06T07:22:51.241129+0000","last_fullsized":"2024-03-06T07:22:51.241129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:27.840741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249104+0000","last_change":"2024-03-06T07:22:50.234521+0000","last_active":"2024-03-06T07:22:51.249104+0000","last_peered":"2024-03-06T07:22:51.249104+0000","last_clean":"2024-03-06T07:22:51.249104+0000","last_became_active":"2024-03-06T07:22:50.234240+0000","last_became_peered":"2024-03-06T07:22:50.234240+0000","last_unstale":"2024-03-06T07:22:51.249104+0000","last_undegraded":"2024-03-06T07:22:51.249104+0000","last_fullsized":"2024-03-06T07:22:51.249104+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:48.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.232773+0000","last_change":"2024-03-06T07:22:51.231575+0000","last_active":"2024-03-06T07:22:51.232773+0000","last_peered":"2024-03-06T07:22:51.232773+0000","last_clean":"2024-03-06T07:22:51.232773+0000","last_became_active":"2024-03-06T07:22:51.231294+0000","last_became_peered":"2024-03-06T07:22:51.231294+0000","last_unstale":"2024-03-06T07:22:51.232773+0000","last_undegraded":"2024-03-06T07:22:51.232773+0000","last_fullsized":"2024-03-06T07:22:51.232773+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:07.758429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.229157+0000","last_change":"2024-03-06T07:22:50.234466+0000","last_active":"2024-03-06T07:22:51.229157+0000","last_peered":"2024-03-06T07:22:51.229157+0000","last_clean":"2024-03-06T07:22:51.229157+0000","last_became_active":"2024-03-06T07:22:50.234225+0000","last_became_peered":"2024-03-06T07:22:50.234225+0000","last_unstale":"2024-03-06T07:22:51.229157+0000","last_undegraded":"2024-03-06T07:22:51.229157+0000","last_fullsized":"2024-03-06T07:22:51.229157+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:43.256389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.232037+0000","last_change":"2024-03-06T07:22:51.226866+0000","last_active":"2024-03-06T07:22:51.232037+0000","last_peered":"2024-03-06T07:22:51.232037+0000","last_clean":"2024-03-06T07:22:51.232037+0000","last_became_active":"2024-03-06T07:22:51.226584+0000","last_became_peered":"2024-03-06T07:22:51.226584+0000","last_unstale":"2024-03-06T07:22:51.232037+0000","last_undegraded":"2024-03-06T07:22:51.232037+0000","last_fullsized":"2024-03-06T07:22:51.232037+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:25.569823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228765+0000","last_change":"2024-03-06T07:22:50.234450+0000","last_active":"2024-03-06T07:22:51.228765+0000","last_peered":"2024-03-06T07:22:51.228765+0000","last_clean":"2024-03-06T07:22:51.228765+0000","last_became_active":"2024-03-06T07:22:50.234165+0000","last_became_peered":"2024-03-06T07:22:50.234165+0000","last_unstale":"2024-03-06T07:22:51.228765+0000","last_undegraded":"2024-03-06T07:22:51.228765+0000","last_fullsized":"2024-03-06T07:22:51.228765+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:26.838417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.243195+0000","last_change":"2024-03-06T07:22:51.240030+0000","last_active":"2024-03-06T07:22:51.243195+0000","last_peered":"2024-03-06T07:22:51.243195+0000","last_clean":"2024-03-06T07:22:51.243195+0000","last_became_active":"2024-03-06T07:22:51.239706+0000","last_became_peered":"2024-03-06T07:22:51.239706+0000","last_unstale":"2024-03-06T07:22:51.243195+0000","last_undegraded":"2024-03-06T07:22:51.243195+0000","last_fullsized":"2024-03-06T07:22:51.243195+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:01.602991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.220278+0000","last_change":"2024-03-06T07:22:50.229025+0000","last_active":"2024-03-06T07:22:51.220278+0000","last_peered":"2024-03-06T07:22:51.220278+0000","last_clean":"2024-03-06T07:22:51.220278+0000","last_became_active":"2024-03-06T07:22:50.228724+0000","last_became_peered":"2024-03-06T07:22:50.228724+0000","last_unstale":"2024-03-06T07:22:51.220278+0000","last_undegraded":"2024-03-06T07:22:51.220278+0000","last_fullsized":"2024-03-06T07:22:51.220278+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:33.879582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.232702+0000","last_change":"2024-03-06T07:22:51.227853+0000","last_active":"2024-03-06T07:22:51.232702+0000","last_peered":"2024-03-06T07:22:51.232702+0000","last_clean":"2024-03-06T07:22:51.232702+0000","last_became_active":"2024-03-06T07:22:51.227548+0000","last_became_peered":"2024-03-06T07:22:51.227548+0000","last_unstale":"2024-03-06T07:22:51.232702+0000","last_undegraded":"2024-03-06T07:22:51.232702+0000","last_fullsized":"2024-03-06T07:22:51.232702+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:40.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245909+0000","last_change":"2024-03-06T07:22:50.229476+0000","last_active":"2024-03-06T07:22:51.245909+0000","last_peered":"2024-03-06T07:22:51.245909+0000","last_clean":"2024-03-06T07:22:51.245909+0000","last_became_active":"2024-03-06T07:22:50.229193+0000","last_became_peered":"2024-03-06T07:22:50.229193+0000","last_unstale":"2024-03-06T07:22:51.245909+0000","last_undegraded":"2024-03-06T07:22:51.245909+0000","last_fullsized":"2024-03-06T07:22:51.245909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:53.068916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.242679+0000","last_change":"2024-03-06T07:22:51.241810+0000","last_active":"2024-03-06T07:22:51.242679+0000","last_peered":"2024-03-06T07:22:51.242679+0000","last_clean":"2024-03-06T07:22:51.242679+0000","last_became_active":"2024-03-06T07:22:51.241258+0000","last_became_peered":"2024-03-06T07:22:51.241258+0000","last_unstale":"2024-03-06T07:22:51.242679+0000","last_undegraded":"2024-03-06T07:22:51.242679+0000","last_fullsized":"2024-03-06T07:22:51.242679+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:18.562926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.224555+0000","last_change":"2024-03-06T07:22:50.234590+0000","last_active":"2024-03-06T07:22:51.224555+0000","last_peered":"2024-03-06T07:22:51.224555+0000","last_clean":"2024-03-06T07:22:51.224555+0000","last_became_active":"2024-03-06T07:22:50.234278+0000","last_became_peered":"2024-03-06T07:22:50.234278+0000","last_unstale":"2024-03-06T07:22:51.224555+0000","last_undegraded":"2024-03-06T07:22:51.224555+0000","last_fullsized":"2024-03-06T07:22:51.224555+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:20.885193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245083+0000","last_change":"2024-03-06T07:22:51.243433+0000","last_active":"2024-03-06T07:22:51.245083+0000","last_peered":"2024-03-06T07:22:51.245083+0000","last_clean":"2024-03-06T07:22:51.245083+0000","last_became_active":"2024-03-06T07:22:51.243114+0000","last_became_peered":"2024-03-06T07:22:51.243114+0000","last_unstale":"2024-03-06T07:22:51.245083+0000","last_undegraded":"2024-03-06T07:22:51.245083+0000","last_fullsized":"2024-03-06T07:22:51.245083+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:31.079398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.237952+0000","last_change":"2024-03-06T07:22:50.228263+0000","last_active":"2024-03-06T07:22:51.237952+0000","last_peered":"2024-03-06T07:22:51.237952+0000","last_clean":"2024-03-06T07:22:51.237952+0000","last_became_active":"2024-03-06T07:22:50.227951+0000","last_became_peered":"2024-03-06T07:22:50.227951+0000","last_unstale":"2024-03-06T07:22:51.237952+0000","last_undegraded":"2024-03-06T07:22:51.237952+0000","last_fullsized":"2024-03-06T07:22:51.237952+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:48.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.239134+0000","last_change":"2024-03-06T07:22:51.234372+0000","last_active":"2024-03-06T07:22:51.239134+0000","last_peered":"2024-03-06T07:22:51.239134+0000","last_clean":"2024-03-06T07:22:51.239134+0000","last_became_active":"2024-03-06T07:22:51.234077+0000","last_became_peered":"2024-03-06T07:22:51.234077+0000","last_unstale":"2024-03-06T07:22:51.239134+0000","last_undegraded":"2024-03-06T07:22:51.239134+0000","last_fullsized":"2024-03-06T07:22:51.239134+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:21.887516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.225248+0000","last_change":"2024-03-06T07:22:50.228279+0000","last_active":"2024-03-06T07:22:51.225248+0000","last_peered":"2024-03-06T07:22:51.225248+0000","last_clean":"2024-03-06T07:22:51.225248+0000","last_became_active":"2024-03-06T07:22:50.227969+0000","last_became_peered":"2024-03-06T07:22:50.227969+0000","last_unstale":"2024-03-06T07:22:51.225248+0000","last_undegraded":"2024-03-06T07:22:51.225248+0000","last_fullsized":"2024-03-06T07:22:51.225248+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:29.146219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.234892+0000","last_change":"2024-03-06T07:22:51.233723+0000","last_active":"2024-03-06T07:22:51.234892+0000","last_peered":"2024-03-06T07:22:51.234892+0000","last_clean":"2024-03-06T07:22:51.234892+0000","last_became_active":"2024-03-06T07:22:51.233416+0000","last_became_peered":"2024-03-06T07:22:51.233416+0000","last_unstale":"2024-03-06T07:22:51.234892+0000","last_undegraded":"2024-03-06T07:22:51.234892+0000","last_fullsized":"2024-03-06T07:22:51.234892+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:54.071239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238336+0000","last_change":"2024-03-06T07:22:50.229301+0000","last_active":"2024-03-06T07:22:51.238336+0000","last_peered":"2024-03-06T07:22:51.238336+0000","last_clean":"2024-03-06T07:22:51.238336+0000","last_became_active":"2024-03-06T07:22:50.228221+0000","last_became_peered":"2024-03-06T07:22:50.228221+0000","last_unstale":"2024-03-06T07:22:51.238336+0000","last_undegraded":"2024-03-06T07:22:51.238336+0000","last_fullsized":"2024-03-06T07:22:51.238336+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:38.457410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.244146+0000","last_change":"2024-03-06T07:22:51.241655+0000","last_active":"2024-03-06T07:22:51.244146+0000","last_peered":"2024-03-06T07:22:51.244146+0000","last_clean":"2024-03-06T07:22:51.244146+0000","last_became_active":"2024-03-06T07:22:51.241363+0000","last_became_peered":"2024-03-06T07:22:51.241363+0000","last_unstale":"2024-03-06T07:22:51.244146+0000","last_undegraded":"2024-03-06T07:22:51.244146+0000","last_fullsized":"2024-03-06T07:22:51.244146+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:41.716689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228931+0000","last_change":"2024-03-06T07:22:50.229687+0000","last_active":"2024-03-06T07:22:51.228931+0000","last_peered":"2024-03-06T07:22:51.228931+0000","last_clean":"2024-03-06T07:22:51.228931+0000","last_became_active":"2024-03-06T07:22:50.229403+0000","last_became_peered":"2024-03-06T07:22:50.229403+0000","last_unstale":"2024-03-06T07:22:51.228931+0000","last_undegraded":"2024-03-06T07:22:51.228931+0000","last_fullsized":"2024-03-06T07:22:51.228931+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:49.205274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247863+0000","last_change":"2024-03-06T07:22:51.246300+0000","last_active":"2024-03-06T07:22:51.247863+0000","last_peered":"2024-03-06T07:22:51.247863+0000","last_clean":"2024-03-06T07:22:51.247863+0000","last_became_active":"2024-03-06T07:22:51.246023+0000","last_became_peered":"2024-03-06T07:22:51.246023+0000","last_unstale":"2024-03-06T07:22:51.247863+0000","last_undegraded":"2024-03-06T07:22:51.247863+0000","last_fullsized":"2024-03-06T07:22:51.247863+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:21.887516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.221119+0000","last_change":"2024-03-06T07:22:50.230404+0000","last_active":"2024-03-06T07:22:51.221119+0000","last_peered":"2024-03-06T07:22:51.221119+0000","last_clean":"2024-03-06T07:22:51.221119+0000","last_became_active":"2024-03-06T07:22:50.229797+0000","last_became_peered":"2024-03-06T07:22:50.229797+0000","last_unstale":"2024-03-06T07:22:51.221119+0000","last_undegraded":"2024-03-06T07:22:51.221119+0000","last_fullsized":"2024-03-06T07:22:51.221119+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:00.600668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238568+0000","last_change":"2024-03-06T07:22:51.236568+0000","last_active":"2024-03-06T07:22:51.238568+0000","last_peered":"2024-03-06T07:22:51.238568+0000","last_clean":"2024-03-06T07:22:51.238568+0000","last_became_active":"2024-03-06T07:22:51.236317+0000","last_became_peered":"2024-03-06T07:22:51.236317+0000","last_unstale":"2024-03-06T07:22:51.238568+0000","last_undegraded":"2024-03-06T07:22:51.238568+0000","last_fullsized":"2024-03-06T07:22:51.238568+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:24.050069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.227444+0000","last_change":"2024-03-06T07:22:50.231019+0000","last_active":"2024-03-06T07:22:51.227444+0000","last_peered":"2024-03-06T07:22:51.227444+0000","last_clean":"2024-03-06T07:22:51.227444+0000","last_became_active":"2024-03-06T07:22:50.230713+0000","last_became_peered":"2024-03-06T07:22:50.230713+0000","last_unstale":"2024-03-06T07:22:51.227444+0000","last_undegraded":"2024-03-06T07:22:51.227444+0000","last_fullsized":"2024-03-06T07:22:51.227444+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:26.270499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":192,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.229340+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:22:51.229340+0000","last_peered":"2024-03-06T07:22:51.229340+0000","last_clean":"2024-03-06T07:22:51.229340+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:22:51.229340+0000","last_undegraded":"2024-03-06T07:22:51.229340+0000","last_fullsized":"2024-03-06T07:22:51.229340+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:54.234 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:54.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.369+0000 7f6a1b20e640 1 Processor -- start 2024-03-06T07:22:54.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.369+0000 7f6a1b20e640 1 -- start start 2024-03-06T07:22:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a1b20e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6a14158600 0x7f6a14072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a1b20e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6a1415e3a0 0x7f6a1406d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a1b20e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a1415eb40 con 0x7f6a1406deb0 2024-03-06T07:22:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a1b20e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a14161ba0 con 0x7f6a14158600 2024-03-06T07:22:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a1b20e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1415ee20 con 0x7f6a1415e3a0 2024-03-06T07:22:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a18faa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6a14158600 0x7f6a14072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a18faa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6a14158600 0x7f6a14072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57794/0 (socket says 172.21.15.59:57794) 2024-03-06T07:22:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a18faa640 1 -- 172.21.15.59:0/1597579769 learned_addr learned my addr 172.21.15.59:0/1597579769 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a18faa640 1 -- 172.21.15.59:0/1597579769 >> v2:172.21.15.164:3300/0 conn(0x7f6a1415e3a0 msgr2=0x7f6a1406d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a18faa640 1 --2- 172.21.15.59:0/1597579769 >> v2:172.21.15.164:3300/0 conn(0x7f6a1415e3a0 0x7f6a1406d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a18faa640 1 -- 172.21.15.59:0/1597579769 >> v1:172.21.15.164:6789/0 conn(0x7f6a1406deb0 legacy=0x7f6a14162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.370+0000 7f6a18faa640 1 -- 172.21.15.59:0/1597579769 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1415f100 con 0x7f6a14158600 2024-03-06T07:22:54.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.371+0000 7f6a18faa640 1 --2- 172.21.15.59:0/1597579769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a14158600 0x7f6a14072730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6a14161d20 tx=0x7f6a0802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb8365b1112f540 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.371+0000 7f6a137fe640 1 -- 172.21.15.59:0/1597579769 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08038070 con 0x7f6a14158600 2024-03-06T07:22:54.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.371+0000 7f6a137fe640 1 -- 172.21.15.59:0/1597579769 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0802f050 con 0x7f6a14158600 2024-03-06T07:22:54.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.371+0000 7f6a137fe640 1 -- 172.21.15.59:0/1597579769 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a08031970 con 0x7f6a14158600 2024-03-06T07:22:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.372+0000 7f6a1b20e640 1 -- 172.21.15.59:0/1597579769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a14158600 msgr2=0x7f6a14072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.372+0000 7f6a1b20e640 1 --2- 172.21.15.59:0/1597579769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a14158600 0x7f6a14072730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6a14161d20 tx=0x7f6a0802b7f0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.372+0000 7f6a1b20e640 1 -- 172.21.15.59:0/1597579769 shutdown_connections 2024-03-06T07:22:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.372+0000 7f6a1b20e640 1 --2- 172.21.15.59:0/1597579769 >> v2:172.21.15.164:3300/0 conn(0x7f6a1415e3a0 0x7f6a1406d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.372+0000 7f6a1b20e640 1 --2- 172.21.15.59:0/1597579769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a14158600 0x7f6a14072730 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.372+0000 7f6a1b20e640 1 -- 172.21.15.59:0/1597579769 >> 172.21.15.59:0/1597579769 conn(0x7f6a1406c0c0 msgr2=0x7f6a1406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.372+0000 7f6a1b20e640 1 -- 172.21.15.59:0/1597579769 shutdown_connections 2024-03-06T07:22:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.372+0000 7f6a1b20e640 1 -- 172.21.15.59:0/1597579769 wait complete. 2024-03-06T07:22:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.373+0000 7f6a1b20e640 1 Processor -- start 2024-03-06T07:22:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.373+0000 7f6a1b20e640 1 -- start start 2024-03-06T07:22:54.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a1b20e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a1406deb0 0x7f6a141f80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a1b20e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a14158600 0x7f6a140ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a1b20e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a1415e3a0 0x7f6a14152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a1b20e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a141628c0 con 0x7f6a1415e3a0 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a1b20e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a14162740 con 0x7f6a1406deb0 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a1b20e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1406afb0 con 0x7f6a14158600 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a197ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a1415e3a0 0x7f6a14152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a197ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a1415e3a0 0x7f6a14152da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57810/0 (socket says 172.21.15.59:57810) 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a197ab640 1 -- 172.21.15.59:0/3861765094 learned_addr learned my addr 172.21.15.59:0/3861765094 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a18faa640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a1406deb0 0x7f6a141f80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a13fff640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a14158600 0x7f6a140ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a197ab640 1 -- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a14158600 msgr2=0x7f6a140ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a197ab640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a14158600 0x7f6a140ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a197ab640 1 -- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a1406deb0 msgr2=0x7f6a141f80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a197ab640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a1406deb0 0x7f6a141f80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.374+0000 7f6a197ab640 1 -- 172.21.15.59:0/3861765094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a141f7600 con 0x7f6a1415e3a0 2024-03-06T07:22:54.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.375+0000 7f6a197ab640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a1415e3a0 0x7f6a14152da0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c00c780 tx=0x7f6a0c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.375+0000 7f6a117fa640 1 -- 172.21.15.59:0/3861765094 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c017070 con 0x7f6a1415e3a0 2024-03-06T07:22:54.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.375+0000 7f6a117fa640 1 -- 172.21.15.59:0/3861765094 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0c003da0 con 0x7f6a1415e3a0 2024-03-06T07:22:54.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.375+0000 7f6a1b20e640 1 -- 172.21.15.59:0/3861765094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a14153510 con 0x7f6a1415e3a0 2024-03-06T07:22:54.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.375+0000 7f6a117fa640 1 -- 172.21.15.59:0/3861765094 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c0052b0 con 0x7f6a1415e3a0 2024-03-06T07:22:54.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.375+0000 7f6a1b20e640 1 -- 172.21.15.59:0/3861765094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a141f7b80 con 0x7f6a1415e3a0 2024-03-06T07:22:54.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.376+0000 7f6a1b20e640 1 -- 172.21.15.59:0/3861765094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a141f7d80 con 0x7f6a1415e3a0 2024-03-06T07:22:54.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.376+0000 7f6a117fa640 1 -- 172.21.15.59:0/3861765094 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a0c01a070 con 0x7f6a1415e3a0 2024-03-06T07:22:54.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.377+0000 7f6a117fa640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f69f806b4d0 0x7f69f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.377+0000 7f6a117fa640 1 -- 172.21.15.59:0/3861765094 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a0c089b10 con 0x7f6a1415e3a0 2024-03-06T07:22:54.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.377+0000 7f6a18faa640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f69f806b4d0 0x7f69f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.378+0000 7f6a18faa640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f69f806b4d0 0x7f69f806d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6a08035000 tx=0x7f6a08008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.384+0000 7f6a117fa640 1 -- 172.21.15.59:0/3861765094 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a141f7d80 con 0x7f6a1415e3a0 2024-03-06T07:22:54.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.561+0000 7f6a1b20e640 1 -- 172.21.15.59:0/3861765094 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a140d06f0 con 0x7f69f806b4d0 2024-03-06T07:22:54.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.568+0000 7f6a117fa640 1 -- 172.21.15.59:0/3861765094 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251879 (secure 0 0 0) 0x7f6a140d06f0 con 0x7f69f806b4d0 2024-03-06T07:22:54.572 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:54.573 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:22:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 -- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f69f806b4d0 msgr2=0x7f69f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f69f806b4d0 0x7f69f806d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6a08035000 tx=0x7f6a08008030 comp rx=0 tx=0).stop 2024-03-06T07:22:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 -- 172.21.15.59:0/3861765094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a1415e3a0 msgr2=0x7f6a14152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a1415e3a0 0x7f6a14152da0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c00c780 tx=0x7f6a0c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 -- 172.21.15.59:0/3861765094 shutdown_connections 2024-03-06T07:22:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f69f806b4d0 0x7f69f806d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a1415e3a0 0x7f6a14152da0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a14158600 0x7f6a140ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 --2- 172.21.15.59:0/3861765094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a1406deb0 0x7f6a141f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 -- 172.21.15.59:0/3861765094 >> 172.21.15.59:0/3861765094 conn(0x7f6a1406c0c0 msgr2=0x7f6a14160810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 -- 172.21.15.59:0/3861765094 shutdown_connections 2024-03-06T07:22:54.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.571+0000 7f6a1b20e640 1 -- 172.21.15.59:0/3861765094 wait complete. 2024-03-06T07:22:54.583 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-03-06T07:22:53.313082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220072,"kb_used_data":3880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754648,"statfs":{"total":773094113280,"available":772868759552,"internally_reserved":0,"allocated":3973120,"data_stored":2249000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.103599"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.235416+0000","last_change":"2024-03-06T07:22:51.231309+0000","last_active":"2024-03-06T07:22:51.235416+0000","last_peered":"2024-03-06T07:22:51.235416+0000","last_clean":"2024-03-06T07:22:51.235416+0000","last_became_active":"2024-03-06T07:22:51.230973+0000","last_became_peered":"2024-03-06T07:22:51.230973+0000","last_unstale":"2024-03-06T07:22:51.235416+0000","last_undegraded":"2024-03-06T07:22:51.235416+0000","last_fullsized":"2024-03-06T07:22:51.235416+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:44.560090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231468+0000","last_change":"2024-03-06T07:22:51.227122+0000","last_active":"2024-03-06T07:22:51.231468+0000","last_peered":"2024-03-06T07:22:51.231468+0000","last_clean":"2024-03-06T07:22:51.231468+0000","last_became_active":"2024-03-06T07:22:51.226740+0000","last_became_peered":"2024-03-06T07:22:51.226740+0000","last_unstale":"2024-03-06T07:22:51.231468+0000","last_undegraded":"2024-03-06T07:22:51.231468+0000","last_fullsized":"2024-03-06T07:22:51.231468+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:18.562926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245869+0000","last_change":"2024-03-06T07:22:51.244796+0000","last_active":"2024-03-06T07:22:51.245869+0000","last_peered":"2024-03-06T07:22:51.245869+0000","last_clean":"2024-03-06T07:22:51.245869+0000","last_became_active":"2024-03-06T07:22:51.244511+0000","last_became_peered":"2024-03-06T07:22:51.244511+0000","last_unstale":"2024-03-06T07:22:51.245869+0000","last_undegraded":"2024-03-06T07:22:51.245869+0000","last_fullsized":"2024-03-06T07:22:51.245869+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:17.546892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249723+0000","last_change":"2024-03-06T07:22:51.246076+0000","last_active":"2024-03-06T07:22:51.249723+0000","last_peered":"2024-03-06T07:22:51.249723+0000","last_clean":"2024-03-06T07:22:51.249723+0000","last_became_active":"2024-03-06T07:22:51.245578+0000","last_became_peered":"2024-03-06T07:22:51.245578+0000","last_unstale":"2024-03-06T07:22:51.249723+0000","last_undegraded":"2024-03-06T07:22:51.249723+0000","last_fullsized":"2024-03-06T07:22:51.249723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:30.894809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231723+0000","last_change":"2024-03-06T07:22:51.225408+0000","last_active":"2024-03-06T07:22:51.231723+0000","last_peered":"2024-03-06T07:22:51.231723+0000","last_clean":"2024-03-06T07:22:51.231723+0000","last_became_active":"2024-03-06T07:22:51.225064+0000","last_became_peered":"2024-03-06T07:22:51.225064+0000","last_unstale":"2024-03-06T07:22:51.231723+0000","last_undegraded":"2024-03-06T07:22:51.231723+0000","last_fullsized":"2024-03-06T07:22:51.231723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:28.409637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.230930+0000","last_change":"2024-03-06T07:22:51.225396+0000","last_active":"2024-03-06T07:22:51.230930+0000","last_peered":"2024-03-06T07:22:51.230930+0000","last_clean":"2024-03-06T07:22:51.230930+0000","last_became_active":"2024-03-06T07:22:51.225106+0000","last_became_peered":"2024-03-06T07:22:51.225106+0000","last_unstale":"2024-03-06T07:22:51.230930+0000","last_undegraded":"2024-03-06T07:22:51.230930+0000","last_fullsized":"2024-03-06T07:22:51.230930+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:42.185715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.241887+0000","last_change":"2024-03-06T07:22:51.239743+0000","last_active":"2024-03-06T07:22:51.241887+0000","last_peered":"2024-03-06T07:22:51.241887+0000","last_clean":"2024-03-06T07:22:51.241887+0000","last_became_active":"2024-03-06T07:22:51.239424+0000","last_became_peered":"2024-03-06T07:22:51.239424+0000","last_unstale":"2024-03-06T07:22:51.241887+0000","last_undegraded":"2024-03-06T07:22:51.241887+0000","last_fullsized":"2024-03-06T07:22:51.241887+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:40.361915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238577+0000","last_change":"2024-03-06T07:22:51.233614+0000","last_active":"2024-03-06T07:22:51.238577+0000","last_peered":"2024-03-06T07:22:51.238577+0000","last_clean":"2024-03-06T07:22:51.238577+0000","last_became_active":"2024-03-06T07:22:51.233350+0000","last_became_peered":"2024-03-06T07:22:51.233350+0000","last_unstale":"2024-03-06T07:22:51.238577+0000","last_undegraded":"2024-03-06T07:22:51.238577+0000","last_fullsized":"2024-03-06T07:22:51.238577+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:08.399514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248660+0000","last_change":"2024-03-06T07:22:51.248089+0000","last_active":"2024-03-06T07:22:51.248660+0000","last_peered":"2024-03-06T07:22:51.248660+0000","last_clean":"2024-03-06T07:22:51.248660+0000","last_became_active":"2024-03-06T07:22:51.247739+0000","last_became_peered":"2024-03-06T07:22:51.247739+0000","last_unstale":"2024-03-06T07:22:51.248660+0000","last_undegraded":"2024-03-06T07:22:51.248660+0000","last_fullsized":"2024-03-06T07:22:51.248660+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:30.148542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248430+0000","last_change":"2024-03-06T07:22:51.241023+0000","last_active":"2024-03-06T07:22:51.248430+0000","last_peered":"2024-03-06T07:22:51.248430+0000","last_clean":"2024-03-06T07:22:51.248430+0000","last_became_active":"2024-03-06T07:22:51.240743+0000","last_became_peered":"2024-03-06T07:22:51.240743+0000","last_unstale":"2024-03-06T07:22:51.248430+0000","last_undegraded":"2024-03-06T07:22:51.248430+0000","last_fullsized":"2024-03-06T07:22:51.248430+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:34.881905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247894+0000","last_change":"2024-03-06T07:22:51.243419+0000","last_active":"2024-03-06T07:22:51.247894+0000","last_peered":"2024-03-06T07:22:51.247894+0000","last_clean":"2024-03-06T07:22:51.247894+0000","last_became_active":"2024-03-06T07:22:51.243125+0000","last_became_peered":"2024-03-06T07:22:51.243125+0000","last_unstale":"2024-03-06T07:22:51.247894+0000","last_undegraded":"2024-03-06T07:22:51.247894+0000","last_fullsized":"2024-03-06T07:22:51.247894+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:34.055263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228046+0000","last_change":"2024-03-06T07:22:51.225041+0000","last_active":"2024-03-06T07:22:51.228046+0000","last_peered":"2024-03-06T07:22:51.228046+0000","last_clean":"2024-03-06T07:22:51.228046+0000","last_became_active":"2024-03-06T07:22:51.224729+0000","last_became_peered":"2024-03-06T07:22:51.224729+0000","last_unstale":"2024-03-06T07:22:51.228046+0000","last_undegraded":"2024-03-06T07:22:51.228046+0000","last_fullsized":"2024-03-06T07:22:51.228046+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:24.050069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.241246+0000","last_change":"2024-03-06T07:22:51.240331+0000","last_active":"2024-03-06T07:22:51.241246+0000","last_peered":"2024-03-06T07:22:51.241246+0000","last_clean":"2024-03-06T07:22:51.241246+0000","last_became_active":"2024-03-06T07:22:51.240044+0000","last_became_peered":"2024-03-06T07:22:51.240044+0000","last_unstale":"2024-03-06T07:22:51.241246+0000","last_undegraded":"2024-03-06T07:22:51.241246+0000","last_fullsized":"2024-03-06T07:22:51.241246+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:50.207597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.232059+0000","last_change":"2024-03-06T07:22:51.231053+0000","last_active":"2024-03-06T07:22:51.232059+0000","last_peered":"2024-03-06T07:22:51.232059+0000","last_clean":"2024-03-06T07:22:51.232059+0000","last_became_active":"2024-03-06T07:22:51.230704+0000","last_became_peered":"2024-03-06T07:22:51.230704+0000","last_unstale":"2024-03-06T07:22:51.232059+0000","last_undegraded":"2024-03-06T07:22:51.232059+0000","last_fullsized":"2024-03-06T07:22:51.232059+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:08.961572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238251+0000","last_change":"2024-03-06T07:22:51.231595+0000","last_active":"2024-03-06T07:22:51.238251+0000","last_peered":"2024-03-06T07:22:51.238251+0000","last_clean":"2024-03-06T07:22:51.238251+0000","last_became_active":"2024-03-06T07:22:51.231339+0000","last_became_peered":"2024-03-06T07:22:51.231339+0000","last_unstale":"2024-03-06T07:22:51.238251+0000","last_undegraded":"2024-03-06T07:22:51.238251+0000","last_fullsized":"2024-03-06T07:22:51.238251+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:01.602991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248006+0000","last_change":"2024-03-06T07:22:51.244736+0000","last_active":"2024-03-06T07:22:51.248006+0000","last_peered":"2024-03-06T07:22:51.248006+0000","last_clean":"2024-03-06T07:22:51.248006+0000","last_became_active":"2024-03-06T07:22:51.244482+0000","last_became_peered":"2024-03-06T07:22:51.244482+0000","last_unstale":"2024-03-06T07:22:51.248006+0000","last_undegraded":"2024-03-06T07:22:51.248006+0000","last_fullsized":"2024-03-06T07:22:51.248006+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:17.546892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238082+0000","last_change":"2024-03-06T07:22:51.233354+0000","last_active":"2024-03-06T07:22:51.238082+0000","last_peered":"2024-03-06T07:22:51.238082+0000","last_clean":"2024-03-06T07:22:51.238082+0000","last_became_active":"2024-03-06T07:22:51.233097+0000","last_became_peered":"2024-03-06T07:22:51.233097+0000","last_unstale":"2024-03-06T07:22:51.238082+0000","last_undegraded":"2024-03-06T07:22:51.238082+0000","last_fullsized":"2024-03-06T07:22:51.238082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:34.881905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249678+0000","last_change":"2024-03-06T07:22:51.244255+0000","last_active":"2024-03-06T07:22:51.249678+0000","last_peered":"2024-03-06T07:22:51.249678+0000","last_clean":"2024-03-06T07:22:51.249678+0000","last_became_active":"2024-03-06T07:22:51.243950+0000","last_became_peered":"2024-03-06T07:22:51.243950+0000","last_unstale":"2024-03-06T07:22:51.249678+0000","last_undegraded":"2024-03-06T07:22:51.249678+0000","last_fullsized":"2024-03-06T07:22:51.249678+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:51.604037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249653+0000","last_change":"2024-03-06T07:22:51.245682+0000","last_active":"2024-03-06T07:22:51.249653+0000","last_peered":"2024-03-06T07:22:51.249653+0000","last_clean":"2024-03-06T07:22:51.249653+0000","last_became_active":"2024-03-06T07:22:51.245357+0000","last_became_peered":"2024-03-06T07:22:51.245357+0000","last_unstale":"2024-03-06T07:22:51.249653+0000","last_undegraded":"2024-03-06T07:22:51.249653+0000","last_fullsized":"2024-03-06T07:22:51.249653+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:02.562762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231482+0000","last_change":"2024-03-06T07:22:51.226122+0000","last_active":"2024-03-06T07:22:51.231482+0000","last_peered":"2024-03-06T07:22:51.231482+0000","last_clean":"2024-03-06T07:22:51.231482+0000","last_became_active":"2024-03-06T07:22:51.225834+0000","last_became_peered":"2024-03-06T07:22:51.225834+0000","last_unstale":"2024-03-06T07:22:51.231482+0000","last_undegraded":"2024-03-06T07:22:51.231482+0000","last_fullsized":"2024-03-06T07:22:51.231482+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:22.660918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238913+0000","last_change":"2024-03-06T07:22:51.237756+0000","last_active":"2024-03-06T07:22:51.238913+0000","last_peered":"2024-03-06T07:22:51.238913+0000","last_clean":"2024-03-06T07:22:51.238913+0000","last_became_active":"2024-03-06T07:22:51.237478+0000","last_became_peered":"2024-03-06T07:22:51.237478+0000","last_unstale":"2024-03-06T07:22:51.238913+0000","last_undegraded":"2024-03-06T07:22:51.238913+0000","last_fullsized":"2024-03-06T07:22:51.238913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:44.258713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238345+0000","last_change":"2024-03-06T07:22:51.234722+0000","last_active":"2024-03-06T07:22:51.238345+0000","last_peered":"2024-03-06T07:22:51.238345+0000","last_clean":"2024-03-06T07:22:51.238345+0000","last_became_active":"2024-03-06T07:22:51.233880+0000","last_became_peered":"2024-03-06T07:22:51.233880+0000","last_unstale":"2024-03-06T07:22:51.238345+0000","last_undegraded":"2024-03-06T07:22:51.238345+0000","last_fullsized":"2024-03-06T07:22:51.238345+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:22:58.557417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.232273+0000","last_change":"2024-03-06T07:22:51.230552+0000","last_active":"2024-03-06T07:22:51.232273+0000","last_peered":"2024-03-06T07:22:51.232273+0000","last_clean":"2024-03-06T07:22:51.232273+0000","last_became_active":"2024-03-06T07:22:51.230276+0000","last_became_peered":"2024-03-06T07:22:51.230276+0000","last_unstale":"2024-03-06T07:22:51.232273+0000","last_undegraded":"2024-03-06T07:22:51.232273+0000","last_fullsized":"2024-03-06T07:22:51.232273+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:44.320357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.227197+0000","last_change":"2024-03-06T07:22:51.225024+0000","last_active":"2024-03-06T07:22:51.227197+0000","last_peered":"2024-03-06T07:22:51.227197+0000","last_clean":"2024-03-06T07:22:51.227197+0000","last_became_active":"2024-03-06T07:22:51.224697+0000","last_became_peered":"2024-03-06T07:22:51.224697+0000","last_unstale":"2024-03-06T07:22:51.227197+0000","last_undegraded":"2024-03-06T07:22:51.227197+0000","last_fullsized":"2024-03-06T07:22:51.227197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:08.399514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248777+0000","last_change":"2024-03-06T07:22:51.243340+0000","last_active":"2024-03-06T07:22:51.248777+0000","last_peered":"2024-03-06T07:22:51.248777+0000","last_clean":"2024-03-06T07:22:51.248777+0000","last_became_active":"2024-03-06T07:22:51.243020+0000","last_became_peered":"2024-03-06T07:22:51.243020+0000","last_unstale":"2024-03-06T07:22:51.248777+0000","last_undegraded":"2024-03-06T07:22:51.248777+0000","last_fullsized":"2024-03-06T07:22:51.248777+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:24.050069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.242355+0000","last_change":"2024-03-06T07:22:51.234440+0000","last_active":"2024-03-06T07:22:51.242355+0000","last_peered":"2024-03-06T07:22:51.242355+0000","last_clean":"2024-03-06T07:22:51.242355+0000","last_became_active":"2024-03-06T07:22:51.234075+0000","last_became_peered":"2024-03-06T07:22:51.234075+0000","last_unstale":"2024-03-06T07:22:51.242355+0000","last_undegraded":"2024-03-06T07:22:51.242355+0000","last_fullsized":"2024-03-06T07:22:51.242355+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:51.605065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249871+0000","last_change":"2024-03-06T07:22:51.247175+0000","last_active":"2024-03-06T07:22:51.249871+0000","last_peered":"2024-03-06T07:22:51.249871+0000","last_clean":"2024-03-06T07:22:51.249871+0000","last_became_active":"2024-03-06T07:22:51.246606+0000","last_became_peered":"2024-03-06T07:22:51.246606+0000","last_unstale":"2024-03-06T07:22:51.249871+0000","last_undegraded":"2024-03-06T07:22:51.249871+0000","last_fullsized":"2024-03-06T07:22:51.249871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:01.602991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228070+0000","last_change":"2024-03-06T07:22:51.227049+0000","last_active":"2024-03-06T07:22:51.228070+0000","last_peered":"2024-03-06T07:22:51.228070+0000","last_clean":"2024-03-06T07:22:51.228070+0000","last_became_active":"2024-03-06T07:22:51.226723+0000","last_became_peered":"2024-03-06T07:22:51.226723+0000","last_unstale":"2024-03-06T07:22:51.228070+0000","last_undegraded":"2024-03-06T07:22:51.228070+0000","last_fullsized":"2024-03-06T07:22:51.228070+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:30.894809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238365+0000","last_change":"2024-03-06T07:22:51.236037+0000","last_active":"2024-03-06T07:22:51.238365+0000","last_peered":"2024-03-06T07:22:51.238365+0000","last_clean":"2024-03-06T07:22:51.238365+0000","last_became_active":"2024-03-06T07:22:51.235720+0000","last_became_peered":"2024-03-06T07:22:51.235720+0000","last_unstale":"2024-03-06T07:22:51.238365+0000","last_undegraded":"2024-03-06T07:22:51.238365+0000","last_fullsized":"2024-03-06T07:22:51.238365+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:54.744059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.246827+0000","last_change":"2024-03-06T07:22:51.241408+0000","last_active":"2024-03-06T07:22:51.246827+0000","last_peered":"2024-03-06T07:22:51.246827+0000","last_clean":"2024-03-06T07:22:51.246827+0000","last_became_active":"2024-03-06T07:22:51.239979+0000","last_became_peered":"2024-03-06T07:22:51.239979+0000","last_unstale":"2024-03-06T07:22:51.246827+0000","last_undegraded":"2024-03-06T07:22:51.246827+0000","last_fullsized":"2024-03-06T07:22:51.246827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:18.562926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.239055+0000","last_change":"2024-03-06T07:22:51.235307+0000","last_active":"2024-03-06T07:22:51.239055+0000","last_peered":"2024-03-06T07:22:51.239055+0000","last_clean":"2024-03-06T07:22:51.239055+0000","last_became_active":"2024-03-06T07:22:51.234365+0000","last_became_peered":"2024-03-06T07:22:51.234365+0000","last_unstale":"2024-03-06T07:22:51.239055+0000","last_undegraded":"2024-03-06T07:22:51.239055+0000","last_fullsized":"2024-03-06T07:22:51.239055+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:49.157693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.239262+0000","last_change":"2024-03-06T07:22:51.236795+0000","last_active":"2024-03-06T07:22:51.239262+0000","last_peered":"2024-03-06T07:22:51.239262+0000","last_clean":"2024-03-06T07:22:51.239262+0000","last_became_active":"2024-03-06T07:22:51.236368+0000","last_became_peered":"2024-03-06T07:22:51.236368+0000","last_unstale":"2024-03-06T07:22:51.239262+0000","last_undegraded":"2024-03-06T07:22:51.239262+0000","last_fullsized":"2024-03-06T07:22:51.239262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:11.865087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.234856+0000","last_change":"2024-03-06T07:22:51.233257+0000","last_active":"2024-03-06T07:22:51.234856+0000","last_peered":"2024-03-06T07:22:51.234856+0000","last_clean":"2024-03-06T07:22:51.234856+0000","last_became_active":"2024-03-06T07:22:51.232888+0000","last_became_peered":"2024-03-06T07:22:51.232888+0000","last_unstale":"2024-03-06T07:22:51.234856+0000","last_undegraded":"2024-03-06T07:22:51.234856+0000","last_fullsized":"2024-03-06T07:22:51.234856+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:39.459733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.244467+0000","last_change":"2024-03-06T07:22:50.229694+0000","last_active":"2024-03-06T07:22:51.244467+0000","last_peered":"2024-03-06T07:22:51.244467+0000","last_clean":"2024-03-06T07:22:51.244467+0000","last_became_active":"2024-03-06T07:22:50.229438+0000","last_became_peered":"2024-03-06T07:22:50.229438+0000","last_unstale":"2024-03-06T07:22:51.244467+0000","last_undegraded":"2024-03-06T07:22:51.244467+0000","last_fullsized":"2024-03-06T07:22:51.244467+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:53.741735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.239155+0000","last_change":"2024-03-06T07:22:51.236821+0000","last_active":"2024-03-06T07:22:51.239155+0000","last_peered":"2024-03-06T07:22:51.239155+0000","last_clean":"2024-03-06T07:22:51.239155+0000","last_became_active":"2024-03-06T07:22:51.236521+0000","last_became_peered":"2024-03-06T07:22:51.236521+0000","last_unstale":"2024-03-06T07:22:51.239155+0000","last_undegraded":"2024-03-06T07:22:51.239155+0000","last_fullsized":"2024-03-06T07:22:51.239155+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:30.148542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247159+0000","last_change":"2024-03-06T07:22:50.229519+0000","last_active":"2024-03-06T07:22:51.247159+0000","last_peered":"2024-03-06T07:22:51.247159+0000","last_clean":"2024-03-06T07:22:51.247159+0000","last_became_active":"2024-03-06T07:22:50.229198+0000","last_became_peered":"2024-03-06T07:22:50.229198+0000","last_unstale":"2024-03-06T07:22:51.247159+0000","last_undegraded":"2024-03-06T07:22:51.247159+0000","last_fullsized":"2024-03-06T07:22:51.247159+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:43.256389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248356+0000","last_change":"2024-03-06T07:22:51.246439+0000","last_active":"2024-03-06T07:22:51.248356+0000","last_peered":"2024-03-06T07:22:51.248356+0000","last_clean":"2024-03-06T07:22:51.248356+0000","last_became_active":"2024-03-06T07:22:51.246117+0000","last_became_peered":"2024-03-06T07:22:51.246117+0000","last_unstale":"2024-03-06T07:22:51.248356+0000","last_undegraded":"2024-03-06T07:22:51.248356+0000","last_fullsized":"2024-03-06T07:22:51.248356+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:49.157693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247321+0000","last_change":"2024-03-06T07:22:50.227158+0000","last_active":"2024-03-06T07:22:51.247321+0000","last_peered":"2024-03-06T07:22:51.247321+0000","last_clean":"2024-03-06T07:22:51.247321+0000","last_became_active":"2024-03-06T07:22:50.226872+0000","last_became_peered":"2024-03-06T07:22:50.226872+0000","last_unstale":"2024-03-06T07:22:51.247321+0000","last_undegraded":"2024-03-06T07:22:51.247321+0000","last_fullsized":"2024-03-06T07:22:51.247321+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:33.879582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249767+0000","last_change":"2024-03-06T07:22:51.246600+0000","last_active":"2024-03-06T07:22:51.249767+0000","last_peered":"2024-03-06T07:22:51.249767+0000","last_clean":"2024-03-06T07:22:51.249767+0000","last_became_active":"2024-03-06T07:22:51.246328+0000","last_became_peered":"2024-03-06T07:22:51.246328+0000","last_unstale":"2024-03-06T07:22:51.249767+0000","last_undegraded":"2024-03-06T07:22:51.249767+0000","last_fullsized":"2024-03-06T07:22:51.249767+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:40.361915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.242632+0000","last_change":"2024-03-06T07:22:50.234493+0000","last_active":"2024-03-06T07:22:51.242632+0000","last_peered":"2024-03-06T07:22:51.242632+0000","last_clean":"2024-03-06T07:22:51.242632+0000","last_became_active":"2024-03-06T07:22:50.234216+0000","last_became_peered":"2024-03-06T07:22:50.234216+0000","last_unstale":"2024-03-06T07:22:51.242632+0000","last_undegraded":"2024-03-06T07:22:51.242632+0000","last_fullsized":"2024-03-06T07:22:51.242632+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:00.497208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.240165+0000","last_change":"2024-03-06T07:22:51.238298+0000","last_active":"2024-03-06T07:22:51.240165+0000","last_peered":"2024-03-06T07:22:51.240165+0000","last_clean":"2024-03-06T07:22:51.240165+0000","last_became_active":"2024-03-06T07:22:51.237989+0000","last_became_peered":"2024-03-06T07:22:51.237989+0000","last_unstale":"2024-03-06T07:22:51.240165+0000","last_undegraded":"2024-03-06T07:22:51.240165+0000","last_fullsized":"2024-03-06T07:22:51.240165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:30.894809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249825+0000","last_change":"2024-03-06T07:22:50.231916+0000","last_active":"2024-03-06T07:22:51.249825+0000","last_peered":"2024-03-06T07:22:51.249825+0000","last_clean":"2024-03-06T07:22:51.249825+0000","last_became_active":"2024-03-06T07:22:50.231660+0000","last_became_peered":"2024-03-06T07:22:50.231660+0000","last_unstale":"2024-03-06T07:22:51.249825+0000","last_undegraded":"2024-03-06T07:22:51.249825+0000","last_fullsized":"2024-03-06T07:22:51.249825+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:06.756105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228972+0000","last_change":"2024-03-06T07:22:51.228164+0000","last_active":"2024-03-06T07:22:51.228972+0000","last_peered":"2024-03-06T07:22:51.228972+0000","last_clean":"2024-03-06T07:22:51.228972+0000","last_became_active":"2024-03-06T07:22:51.227851+0000","last_became_peered":"2024-03-06T07:22:51.227851+0000","last_unstale":"2024-03-06T07:22:51.228972+0000","last_undegraded":"2024-03-06T07:22:51.228972+0000","last_fullsized":"2024-03-06T07:22:51.228972+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:40.361915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.252429+0000","last_change":"2024-03-06T07:22:50.231245+0000","last_active":"2024-03-06T07:22:51.252429+0000","last_peered":"2024-03-06T07:22:51.252429+0000","last_clean":"2024-03-06T07:22:51.252429+0000","last_became_active":"2024-03-06T07:22:50.230985+0000","last_became_peered":"2024-03-06T07:22:50.230985+0000","last_unstale":"2024-03-06T07:22:51.252429+0000","last_undegraded":"2024-03-06T07:22:51.252429+0000","last_fullsized":"2024-03-06T07:22:51.252429+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:40.080368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238379+0000","last_change":"2024-03-06T07:22:51.235842+0000","last_active":"2024-03-06T07:22:51.238379+0000","last_peered":"2024-03-06T07:22:51.238379+0000","last_clean":"2024-03-06T07:22:51.238379+0000","last_became_active":"2024-03-06T07:22:51.235518+0000","last_became_peered":"2024-03-06T07:22:51.235518+0000","last_unstale":"2024-03-06T07:22:51.238379+0000","last_undegraded":"2024-03-06T07:22:51.238379+0000","last_fullsized":"2024-03-06T07:22:51.238379+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:27.272822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.220166+0000","last_change":"2024-03-06T07:22:50.225708+0000","last_active":"2024-03-06T07:22:51.220166+0000","last_peered":"2024-03-06T07:22:51.220166+0000","last_clean":"2024-03-06T07:22:51.220166+0000","last_became_active":"2024-03-06T07:22:50.225428+0000","last_became_peered":"2024-03-06T07:22:50.225428+0000","last_unstale":"2024-03-06T07:22:51.220166+0000","last_undegraded":"2024-03-06T07:22:51.220166+0000","last_fullsized":"2024-03-06T07:22:51.220166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:10.862763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245033+0000","last_change":"2024-03-06T07:22:51.240183+0000","last_active":"2024-03-06T07:22:51.245033+0000","last_peered":"2024-03-06T07:22:51.245033+0000","last_clean":"2024-03-06T07:22:51.245033+0000","last_became_active":"2024-03-06T07:22:51.239883+0000","last_became_peered":"2024-03-06T07:22:51.239883+0000","last_unstale":"2024-03-06T07:22:51.245033+0000","last_undegraded":"2024-03-06T07:22:51.245033+0000","last_fullsized":"2024-03-06T07:22:51.245033+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:31.079398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228210+0000","last_change":"2024-03-06T07:22:50.230105+0000","last_active":"2024-03-06T07:22:51.228210+0000","last_peered":"2024-03-06T07:22:51.228210+0000","last_clean":"2024-03-06T07:22:51.228210+0000","last_became_active":"2024-03-06T07:22:50.229822+0000","last_became_peered":"2024-03-06T07:22:50.229822+0000","last_unstale":"2024-03-06T07:22:51.228210+0000","last_undegraded":"2024-03-06T07:22:51.228210+0000","last_fullsized":"2024-03-06T07:22:51.228210+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:39.604948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228021+0000","last_change":"2024-03-06T07:22:51.225757+0000","last_active":"2024-03-06T07:22:51.228021+0000","last_peered":"2024-03-06T07:22:51.228021+0000","last_clean":"2024-03-06T07:22:51.228021+0000","last_became_active":"2024-03-06T07:22:51.225444+0000","last_became_peered":"2024-03-06T07:22:51.225444+0000","last_unstale":"2024-03-06T07:22:51.228021+0000","last_undegraded":"2024-03-06T07:22:51.228021+0000","last_fullsized":"2024-03-06T07:22:51.228021+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:11.865087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.219619+0000","last_change":"2024-03-06T07:22:50.230222+0000","last_active":"2024-03-06T07:22:51.219619+0000","last_peered":"2024-03-06T07:22:51.219619+0000","last_clean":"2024-03-06T07:22:51.219619+0000","last_became_active":"2024-03-06T07:22:50.229916+0000","last_became_peered":"2024-03-06T07:22:50.229916+0000","last_unstale":"2024-03-06T07:22:51.219619+0000","last_undegraded":"2024-03-06T07:22:51.219619+0000","last_fullsized":"2024-03-06T07:22:51.219619+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:07.397191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245042+0000","last_change":"2024-03-06T07:22:51.241804+0000","last_active":"2024-03-06T07:22:51.245042+0000","last_peered":"2024-03-06T07:22:51.245042+0000","last_clean":"2024-03-06T07:22:51.245042+0000","last_became_active":"2024-03-06T07:22:51.241468+0000","last_became_peered":"2024-03-06T07:22:51.241468+0000","last_unstale":"2024-03-06T07:22:51.245042+0000","last_undegraded":"2024-03-06T07:22:51.245042+0000","last_fullsized":"2024-03-06T07:22:51.245042+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:46.593686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.224405+0000","last_change":"2024-03-06T07:22:50.231068+0000","last_active":"2024-03-06T07:22:51.224405+0000","last_peered":"2024-03-06T07:22:51.224405+0000","last_clean":"2024-03-06T07:22:51.224405+0000","last_became_active":"2024-03-06T07:22:50.230704+0000","last_became_peered":"2024-03-06T07:22:50.230704+0000","last_unstale":"2024-03-06T07:22:51.224405+0000","last_undegraded":"2024-03-06T07:22:51.224405+0000","last_fullsized":"2024-03-06T07:22:51.224405+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:43.557767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231600+0000","last_change":"2024-03-06T07:22:51.228854+0000","last_active":"2024-03-06T07:22:51.231600+0000","last_peered":"2024-03-06T07:22:51.231600+0000","last_clean":"2024-03-06T07:22:51.231600+0000","last_became_active":"2024-03-06T07:22:51.228572+0000","last_became_peered":"2024-03-06T07:22:51.228572+0000","last_unstale":"2024-03-06T07:22:51.231600+0000","last_undegraded":"2024-03-06T07:22:51.231600+0000","last_fullsized":"2024-03-06T07:22:51.231600+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:17.546892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248474+0000","last_change":"2024-03-06T07:22:50.231134+0000","last_active":"2024-03-06T07:22:51.248474+0000","last_peered":"2024-03-06T07:22:51.248474+0000","last_clean":"2024-03-06T07:22:51.248474+0000","last_became_active":"2024-03-06T07:22:50.230815+0000","last_became_peered":"2024-03-06T07:22:50.230815+0000","last_unstale":"2024-03-06T07:22:51.248474+0000","last_undegraded":"2024-03-06T07:22:51.248474+0000","last_fullsized":"2024-03-06T07:22:51.248474+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:30.077075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245849+0000","last_change":"2024-03-06T07:22:51.244433+0000","last_active":"2024-03-06T07:22:51.245849+0000","last_peered":"2024-03-06T07:22:51.245849+0000","last_clean":"2024-03-06T07:22:51.245849+0000","last_became_active":"2024-03-06T07:22:51.244155+0000","last_became_peered":"2024-03-06T07:22:51.244155+0000","last_unstale":"2024-03-06T07:22:51.245849+0000","last_undegraded":"2024-03-06T07:22:51.245849+0000","last_fullsized":"2024-03-06T07:22:51.245849+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:34.055263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.252692+0000","last_change":"2024-03-06T07:22:50.228203+0000","last_active":"2024-03-06T07:22:51.252692+0000","last_peered":"2024-03-06T07:22:51.252692+0000","last_clean":"2024-03-06T07:22:51.252692+0000","last_became_active":"2024-03-06T07:22:50.227909+0000","last_became_peered":"2024-03-06T07:22:50.227909+0000","last_unstale":"2024-03-06T07:22:51.252692+0000","last_undegraded":"2024-03-06T07:22:51.252692+0000","last_fullsized":"2024-03-06T07:22:51.252692+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:23.047745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248653+0000","last_change":"2024-03-06T07:22:51.241258+0000","last_active":"2024-03-06T07:22:51.248653+0000","last_peered":"2024-03-06T07:22:51.248653+0000","last_clean":"2024-03-06T07:22:51.248653+0000","last_became_active":"2024-03-06T07:22:51.240607+0000","last_became_peered":"2024-03-06T07:22:51.240607+0000","last_unstale":"2024-03-06T07:22:51.248653+0000","last_undegraded":"2024-03-06T07:22:51.248653+0000","last_fullsized":"2024-03-06T07:22:51.248653+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:18:08.399514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247345+0000","last_change":"2024-03-06T07:22:50.232757+0000","last_active":"2024-03-06T07:22:51.247345+0000","last_peered":"2024-03-06T07:22:51.247345+0000","last_clean":"2024-03-06T07:22:51.247345+0000","last_became_active":"2024-03-06T07:22:50.232486+0000","last_became_peered":"2024-03-06T07:22:50.232486+0000","last_unstale":"2024-03-06T07:22:51.247345+0000","last_undegraded":"2024-03-06T07:22:51.247345+0000","last_fullsized":"2024-03-06T07:22:51.247345+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:13.376099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231445+0000","last_change":"2024-03-06T07:22:51.227945+0000","last_active":"2024-03-06T07:22:51.231445+0000","last_peered":"2024-03-06T07:22:51.231445+0000","last_clean":"2024-03-06T07:22:51.231445+0000","last_became_active":"2024-03-06T07:22:51.227662+0000","last_became_peered":"2024-03-06T07:22:51.227662+0000","last_unstale":"2024-03-06T07:22:51.231445+0000","last_undegraded":"2024-03-06T07:22:51.231445+0000","last_fullsized":"2024-03-06T07:22:51.231445+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:34.055263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238170+0000","last_change":"2024-03-06T07:22:50.230923+0000","last_active":"2024-03-06T07:22:51.238170+0000","last_peered":"2024-03-06T07:22:51.238170+0000","last_clean":"2024-03-06T07:22:51.238170+0000","last_became_active":"2024-03-06T07:22:50.229807+0000","last_became_peered":"2024-03-06T07:22:50.229807+0000","last_unstale":"2024-03-06T07:22:51.238170+0000","last_undegraded":"2024-03-06T07:22:51.238170+0000","last_fullsized":"2024-03-06T07:22:51.238170+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:29.146219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.244867+0000","last_change":"2024-03-06T07:22:51.240818+0000","last_active":"2024-03-06T07:22:51.244867+0000","last_peered":"2024-03-06T07:22:51.244867+0000","last_clean":"2024-03-06T07:22:51.244867+0000","last_became_active":"2024-03-06T07:22:51.240047+0000","last_became_peered":"2024-03-06T07:22:51.240047+0000","last_unstale":"2024-03-06T07:22:51.244867+0000","last_undegraded":"2024-03-06T07:22:51.244867+0000","last_fullsized":"2024-03-06T07:22:51.244867+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:42.185715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.220320+0000","last_change":"2024-03-06T07:22:50.229203+0000","last_active":"2024-03-06T07:22:51.220320+0000","last_peered":"2024-03-06T07:22:51.220320+0000","last_clean":"2024-03-06T07:22:51.220320+0000","last_became_active":"2024-03-06T07:22:50.228891+0000","last_became_peered":"2024-03-06T07:22:50.228891+0000","last_unstale":"2024-03-06T07:22:51.220320+0000","last_undegraded":"2024-03-06T07:22:51.220320+0000","last_fullsized":"2024-03-06T07:22:51.220320+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:29.892486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.239074+0000","last_change":"2024-03-06T07:22:51.237539+0000","last_active":"2024-03-06T07:22:51.239074+0000","last_peered":"2024-03-06T07:22:51.239074+0000","last_clean":"2024-03-06T07:22:51.239074+0000","last_became_active":"2024-03-06T07:22:51.237244+0000","last_became_peered":"2024-03-06T07:22:51.237244+0000","last_unstale":"2024-03-06T07:22:51.239074+0000","last_undegraded":"2024-03-06T07:22:51.239074+0000","last_fullsized":"2024-03-06T07:22:51.239074+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:44.320357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249642+0000","last_change":"2024-03-06T07:22:50.226418+0000","last_active":"2024-03-06T07:22:51.249642+0000","last_peered":"2024-03-06T07:22:51.249642+0000","last_clean":"2024-03-06T07:22:51.249642+0000","last_became_active":"2024-03-06T07:22:50.226146+0000","last_became_peered":"2024-03-06T07:22:50.226146+0000","last_unstale":"2024-03-06T07:22:51.249642+0000","last_undegraded":"2024-03-06T07:22:51.249642+0000","last_fullsized":"2024-03-06T07:22:51.249642+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:39.359591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245159+0000","last_change":"2024-03-06T07:22:51.242081+0000","last_active":"2024-03-06T07:22:51.245159+0000","last_peered":"2024-03-06T07:22:51.245159+0000","last_clean":"2024-03-06T07:22:51.245159+0000","last_became_active":"2024-03-06T07:22:51.241813+0000","last_became_peered":"2024-03-06T07:22:51.241813+0000","last_unstale":"2024-03-06T07:22:51.245159+0000","last_undegraded":"2024-03-06T07:22:51.245159+0000","last_fullsized":"2024-03-06T07:22:51.245159+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:44.560090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.248341+0000","last_change":"2024-03-06T07:22:50.225669+0000","last_active":"2024-03-06T07:22:51.248341+0000","last_peered":"2024-03-06T07:22:51.248341+0000","last_clean":"2024-03-06T07:22:51.248341+0000","last_became_active":"2024-03-06T07:22:50.225221+0000","last_became_peered":"2024-03-06T07:22:50.225221+0000","last_unstale":"2024-03-06T07:22:51.248341+0000","last_undegraded":"2024-03-06T07:22:51.248341+0000","last_fullsized":"2024-03-06T07:22:51.248341+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:43.318033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238287+0000","last_change":"2024-03-06T07:22:51.236133+0000","last_active":"2024-03-06T07:22:51.238287+0000","last_peered":"2024-03-06T07:22:51.238287+0000","last_clean":"2024-03-06T07:22:51.238287+0000","last_became_active":"2024-03-06T07:22:51.235827+0000","last_became_peered":"2024-03-06T07:22:51.235827+0000","last_unstale":"2024-03-06T07:22:51.238287+0000","last_undegraded":"2024-03-06T07:22:51.238287+0000","last_fullsized":"2024-03-06T07:22:51.238287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:42.185715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.246563+0000","last_change":"2024-03-06T07:22:50.230566+0000","last_active":"2024-03-06T07:22:51.246563+0000","last_peered":"2024-03-06T07:22:51.246563+0000","last_clean":"2024-03-06T07:22:51.246563+0000","last_became_active":"2024-03-06T07:22:50.230310+0000","last_became_peered":"2024-03-06T07:22:50.230310+0000","last_unstale":"2024-03-06T07:22:51.246563+0000","last_undegraded":"2024-03-06T07:22:51.246563+0000","last_fullsized":"2024-03-06T07:22:51.246563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:51.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249700+0000","last_change":"2024-03-06T07:22:51.245706+0000","last_active":"2024-03-06T07:22:51.249700+0000","last_peered":"2024-03-06T07:22:51.249700+0000","last_clean":"2024-03-06T07:22:51.249700+0000","last_became_active":"2024-03-06T07:22:51.244432+0000","last_became_peered":"2024-03-06T07:22:51.244432+0000","last_unstale":"2024-03-06T07:22:51.249700+0000","last_undegraded":"2024-03-06T07:22:51.249700+0000","last_fullsized":"2024-03-06T07:22:51.249700+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:11.865087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247483+0000","last_change":"2024-03-06T07:22:50.232948+0000","last_active":"2024-03-06T07:22:51.247483+0000","last_peered":"2024-03-06T07:22:51.247483+0000","last_clean":"2024-03-06T07:22:51.247483+0000","last_became_active":"2024-03-06T07:22:50.232618+0000","last_became_peered":"2024-03-06T07:22:50.232618+0000","last_unstale":"2024-03-06T07:22:51.247483+0000","last_undegraded":"2024-03-06T07:22:51.247483+0000","last_fullsized":"2024-03-06T07:22:51.247483+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:24.567500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.240860+0000","last_change":"2024-03-06T07:22:51.235491+0000","last_active":"2024-03-06T07:22:51.240860+0000","last_peered":"2024-03-06T07:22:51.240860+0000","last_clean":"2024-03-06T07:22:51.240860+0000","last_became_active":"2024-03-06T07:22:51.235220+0000","last_became_peered":"2024-03-06T07:22:51.235220+0000","last_unstale":"2024-03-06T07:22:51.240860+0000","last_undegraded":"2024-03-06T07:22:51.240860+0000","last_fullsized":"2024-03-06T07:22:51.240860+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:14.378422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.231165+0000","last_change":"2024-03-06T07:22:50.232635+0000","last_active":"2024-03-06T07:22:51.231165+0000","last_peered":"2024-03-06T07:22:51.231165+0000","last_clean":"2024-03-06T07:22:51.231165+0000","last_became_active":"2024-03-06T07:22:50.232378+0000","last_became_peered":"2024-03-06T07:22:50.232378+0000","last_unstale":"2024-03-06T07:22:51.231165+0000","last_undegraded":"2024-03-06T07:22:51.231165+0000","last_fullsized":"2024-03-06T07:22:51.231165+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:10.052396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.241129+0000","last_change":"2024-03-06T07:22:51.236866+0000","last_active":"2024-03-06T07:22:51.241129+0000","last_peered":"2024-03-06T07:22:51.241129+0000","last_clean":"2024-03-06T07:22:51.241129+0000","last_became_active":"2024-03-06T07:22:51.236557+0000","last_became_peered":"2024-03-06T07:22:51.236557+0000","last_unstale":"2024-03-06T07:22:51.241129+0000","last_undegraded":"2024-03-06T07:22:51.241129+0000","last_fullsized":"2024-03-06T07:22:51.241129+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:27.840741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.249104+0000","last_change":"2024-03-06T07:22:50.234521+0000","last_active":"2024-03-06T07:22:51.249104+0000","last_peered":"2024-03-06T07:22:51.249104+0000","last_clean":"2024-03-06T07:22:51.249104+0000","last_became_active":"2024-03-06T07:22:50.234240+0000","last_became_peered":"2024-03-06T07:22:50.234240+0000","last_unstale":"2024-03-06T07:22:51.249104+0000","last_undegraded":"2024-03-06T07:22:51.249104+0000","last_fullsized":"2024-03-06T07:22:51.249104+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:48.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.232773+0000","last_change":"2024-03-06T07:22:51.231575+0000","last_active":"2024-03-06T07:22:51.232773+0000","last_peered":"2024-03-06T07:22:51.232773+0000","last_clean":"2024-03-06T07:22:51.232773+0000","last_became_active":"2024-03-06T07:22:51.231294+0000","last_became_peered":"2024-03-06T07:22:51.231294+0000","last_unstale":"2024-03-06T07:22:51.232773+0000","last_undegraded":"2024-03-06T07:22:51.232773+0000","last_fullsized":"2024-03-06T07:22:51.232773+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:07.758429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.229157+0000","last_change":"2024-03-06T07:22:50.234466+0000","last_active":"2024-03-06T07:22:51.229157+0000","last_peered":"2024-03-06T07:22:51.229157+0000","last_clean":"2024-03-06T07:22:51.229157+0000","last_became_active":"2024-03-06T07:22:50.234225+0000","last_became_peered":"2024-03-06T07:22:50.234225+0000","last_unstale":"2024-03-06T07:22:51.229157+0000","last_undegraded":"2024-03-06T07:22:51.229157+0000","last_fullsized":"2024-03-06T07:22:51.229157+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:43.256389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.232037+0000","last_change":"2024-03-06T07:22:51.226866+0000","last_active":"2024-03-06T07:22:51.232037+0000","last_peered":"2024-03-06T07:22:51.232037+0000","last_clean":"2024-03-06T07:22:51.232037+0000","last_became_active":"2024-03-06T07:22:51.226584+0000","last_became_peered":"2024-03-06T07:22:51.226584+0000","last_unstale":"2024-03-06T07:22:51.232037+0000","last_undegraded":"2024-03-06T07:22:51.232037+0000","last_fullsized":"2024-03-06T07:22:51.232037+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:25.569823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228765+0000","last_change":"2024-03-06T07:22:50.234450+0000","last_active":"2024-03-06T07:22:51.228765+0000","last_peered":"2024-03-06T07:22:51.228765+0000","last_clean":"2024-03-06T07:22:51.228765+0000","last_became_active":"2024-03-06T07:22:50.234165+0000","last_became_peered":"2024-03-06T07:22:50.234165+0000","last_unstale":"2024-03-06T07:22:51.228765+0000","last_undegraded":"2024-03-06T07:22:51.228765+0000","last_fullsized":"2024-03-06T07:22:51.228765+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:26.838417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.243195+0000","last_change":"2024-03-06T07:22:51.240030+0000","last_active":"2024-03-06T07:22:51.243195+0000","last_peered":"2024-03-06T07:22:51.243195+0000","last_clean":"2024-03-06T07:22:51.243195+0000","last_became_active":"2024-03-06T07:22:51.239706+0000","last_became_peered":"2024-03-06T07:22:51.239706+0000","last_unstale":"2024-03-06T07:22:51.243195+0000","last_undegraded":"2024-03-06T07:22:51.243195+0000","last_fullsized":"2024-03-06T07:22:51.243195+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:01.602991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.220278+0000","last_change":"2024-03-06T07:22:50.229025+0000","last_active":"2024-03-06T07:22:51.220278+0000","last_peered":"2024-03-06T07:22:51.220278+0000","last_clean":"2024-03-06T07:22:51.220278+0000","last_became_active":"2024-03-06T07:22:50.228724+0000","last_became_peered":"2024-03-06T07:22:50.228724+0000","last_unstale":"2024-03-06T07:22:51.220278+0000","last_undegraded":"2024-03-06T07:22:51.220278+0000","last_fullsized":"2024-03-06T07:22:51.220278+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:33.879582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.232702+0000","last_change":"2024-03-06T07:22:51.227853+0000","last_active":"2024-03-06T07:22:51.232702+0000","last_peered":"2024-03-06T07:22:51.232702+0000","last_clean":"2024-03-06T07:22:51.232702+0000","last_became_active":"2024-03-06T07:22:51.227548+0000","last_became_peered":"2024-03-06T07:22:51.227548+0000","last_unstale":"2024-03-06T07:22:51.232702+0000","last_undegraded":"2024-03-06T07:22:51.232702+0000","last_fullsized":"2024-03-06T07:22:51.232702+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:46:40.607272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245909+0000","last_change":"2024-03-06T07:22:50.229476+0000","last_active":"2024-03-06T07:22:51.245909+0000","last_peered":"2024-03-06T07:22:51.245909+0000","last_clean":"2024-03-06T07:22:51.245909+0000","last_became_active":"2024-03-06T07:22:50.229193+0000","last_became_peered":"2024-03-06T07:22:50.229193+0000","last_unstale":"2024-03-06T07:22:51.245909+0000","last_undegraded":"2024-03-06T07:22:51.245909+0000","last_fullsized":"2024-03-06T07:22:51.245909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:53.068916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.242679+0000","last_change":"2024-03-06T07:22:51.241810+0000","last_active":"2024-03-06T07:22:51.242679+0000","last_peered":"2024-03-06T07:22:51.242679+0000","last_clean":"2024-03-06T07:22:51.242679+0000","last_became_active":"2024-03-06T07:22:51.241258+0000","last_became_peered":"2024-03-06T07:22:51.241258+0000","last_unstale":"2024-03-06T07:22:51.242679+0000","last_undegraded":"2024-03-06T07:22:51.242679+0000","last_fullsized":"2024-03-06T07:22:51.242679+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:18.562926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.224555+0000","last_change":"2024-03-06T07:22:50.234590+0000","last_active":"2024-03-06T07:22:51.224555+0000","last_peered":"2024-03-06T07:22:51.224555+0000","last_clean":"2024-03-06T07:22:51.224555+0000","last_became_active":"2024-03-06T07:22:50.234278+0000","last_became_peered":"2024-03-06T07:22:50.234278+0000","last_unstale":"2024-03-06T07:22:51.224555+0000","last_undegraded":"2024-03-06T07:22:51.224555+0000","last_fullsized":"2024-03-06T07:22:51.224555+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:20.885193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.245083+0000","last_change":"2024-03-06T07:22:51.243433+0000","last_active":"2024-03-06T07:22:51.245083+0000","last_peered":"2024-03-06T07:22:51.245083+0000","last_clean":"2024-03-06T07:22:51.245083+0000","last_became_active":"2024-03-06T07:22:51.243114+0000","last_became_peered":"2024-03-06T07:22:51.243114+0000","last_unstale":"2024-03-06T07:22:51.245083+0000","last_undegraded":"2024-03-06T07:22:51.245083+0000","last_fullsized":"2024-03-06T07:22:51.245083+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:31.079398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.237952+0000","last_change":"2024-03-06T07:22:50.228263+0000","last_active":"2024-03-06T07:22:51.237952+0000","last_peered":"2024-03-06T07:22:51.237952+0000","last_clean":"2024-03-06T07:22:51.237952+0000","last_became_active":"2024-03-06T07:22:50.227951+0000","last_became_peered":"2024-03-06T07:22:50.227951+0000","last_unstale":"2024-03-06T07:22:51.237952+0000","last_undegraded":"2024-03-06T07:22:51.237952+0000","last_fullsized":"2024-03-06T07:22:51.237952+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:48.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.239134+0000","last_change":"2024-03-06T07:22:51.234372+0000","last_active":"2024-03-06T07:22:51.239134+0000","last_peered":"2024-03-06T07:22:51.239134+0000","last_clean":"2024-03-06T07:22:51.239134+0000","last_became_active":"2024-03-06T07:22:51.234077+0000","last_became_peered":"2024-03-06T07:22:51.234077+0000","last_unstale":"2024-03-06T07:22:51.239134+0000","last_undegraded":"2024-03-06T07:22:51.239134+0000","last_fullsized":"2024-03-06T07:22:51.239134+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:21.887516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.225248+0000","last_change":"2024-03-06T07:22:50.228279+0000","last_active":"2024-03-06T07:22:51.225248+0000","last_peered":"2024-03-06T07:22:51.225248+0000","last_clean":"2024-03-06T07:22:51.225248+0000","last_became_active":"2024-03-06T07:22:50.227969+0000","last_became_peered":"2024-03-06T07:22:50.227969+0000","last_unstale":"2024-03-06T07:22:51.225248+0000","last_undegraded":"2024-03-06T07:22:51.225248+0000","last_fullsized":"2024-03-06T07:22:51.225248+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:29.146219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.234892+0000","last_change":"2024-03-06T07:22:51.233723+0000","last_active":"2024-03-06T07:22:51.234892+0000","last_peered":"2024-03-06T07:22:51.234892+0000","last_clean":"2024-03-06T07:22:51.234892+0000","last_became_active":"2024-03-06T07:22:51.233416+0000","last_became_peered":"2024-03-06T07:22:51.233416+0000","last_unstale":"2024-03-06T07:22:51.234892+0000","last_undegraded":"2024-03-06T07:22:51.234892+0000","last_fullsized":"2024-03-06T07:22:51.234892+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:54.071239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238336+0000","last_change":"2024-03-06T07:22:50.229301+0000","last_active":"2024-03-06T07:22:51.238336+0000","last_peered":"2024-03-06T07:22:51.238336+0000","last_clean":"2024-03-06T07:22:51.238336+0000","last_became_active":"2024-03-06T07:22:50.228221+0000","last_became_peered":"2024-03-06T07:22:50.228221+0000","last_unstale":"2024-03-06T07:22:51.238336+0000","last_undegraded":"2024-03-06T07:22:51.238336+0000","last_fullsized":"2024-03-06T07:22:51.238336+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:38.457410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.244146+0000","last_change":"2024-03-06T07:22:51.241655+0000","last_active":"2024-03-06T07:22:51.244146+0000","last_peered":"2024-03-06T07:22:51.244146+0000","last_clean":"2024-03-06T07:22:51.244146+0000","last_became_active":"2024-03-06T07:22:51.241363+0000","last_became_peered":"2024-03-06T07:22:51.241363+0000","last_unstale":"2024-03-06T07:22:51.244146+0000","last_undegraded":"2024-03-06T07:22:51.244146+0000","last_fullsized":"2024-03-06T07:22:51.244146+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:03:41.716689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.228931+0000","last_change":"2024-03-06T07:22:50.229687+0000","last_active":"2024-03-06T07:22:51.228931+0000","last_peered":"2024-03-06T07:22:51.228931+0000","last_clean":"2024-03-06T07:22:51.228931+0000","last_became_active":"2024-03-06T07:22:50.229403+0000","last_became_peered":"2024-03-06T07:22:50.229403+0000","last_unstale":"2024-03-06T07:22:51.228931+0000","last_undegraded":"2024-03-06T07:22:51.228931+0000","last_fullsized":"2024-03-06T07:22:51.228931+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:49.205274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.247863+0000","last_change":"2024-03-06T07:22:51.246300+0000","last_active":"2024-03-06T07:22:51.247863+0000","last_peered":"2024-03-06T07:22:51.247863+0000","last_clean":"2024-03-06T07:22:51.247863+0000","last_became_active":"2024-03-06T07:22:51.246023+0000","last_became_peered":"2024-03-06T07:22:51.246023+0000","last_unstale":"2024-03-06T07:22:51.247863+0000","last_undegraded":"2024-03-06T07:22:51.247863+0000","last_fullsized":"2024-03-06T07:22:51.247863+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:21.887516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.221119+0000","last_change":"2024-03-06T07:22:50.230404+0000","last_active":"2024-03-06T07:22:51.221119+0000","last_peered":"2024-03-06T07:22:51.221119+0000","last_clean":"2024-03-06T07:22:51.221119+0000","last_became_active":"2024-03-06T07:22:50.229797+0000","last_became_peered":"2024-03-06T07:22:50.229797+0000","last_unstale":"2024-03-06T07:22:51.221119+0000","last_undegraded":"2024-03-06T07:22:51.221119+0000","last_fullsized":"2024-03-06T07:22:51.221119+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:00.600668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.238568+0000","last_change":"2024-03-06T07:22:51.236568+0000","last_active":"2024-03-06T07:22:51.238568+0000","last_peered":"2024-03-06T07:22:51.238568+0000","last_clean":"2024-03-06T07:22:51.238568+0000","last_became_active":"2024-03-06T07:22:51.236317+0000","last_became_peered":"2024-03-06T07:22:51.236317+0000","last_unstale":"2024-03-06T07:22:51.238568+0000","last_undegraded":"2024-03-06T07:22:51.238568+0000","last_fullsized":"2024-03-06T07:22:51.238568+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:50.208626+0000","last_clean_scrub_stamp":"2024-03-06T07:22:50.208626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:24.050069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.227444+0000","last_change":"2024-03-06T07:22:50.231019+0000","last_active":"2024-03-06T07:22:51.227444+0000","last_peered":"2024-03-06T07:22:51.227444+0000","last_clean":"2024-03-06T07:22:51.227444+0000","last_became_active":"2024-03-06T07:22:50.230713+0000","last_became_peered":"2024-03-06T07:22:50.230713+0000","last_unstale":"2024-03-06T07:22:51.227444+0000","last_undegraded":"2024-03-06T07:22:51.227444+0000","last_fullsized":"2024-03-06T07:22:51.227444+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:49.206302+0000","last_clean_scrub_stamp":"2024-03-06T07:22:49.206302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:26.270499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":192,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-06T07:22:51.229340+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:22:51.229340+0000","last_peered":"2024-03-06T07:22:51.229340+0000","last_clean":"2024-03-06T07:22:51.229340+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:22:51.229340+0000","last_undegraded":"2024-03-06T07:22:51.229340+0000","last_fullsized":"2024-03-06T07:22:51.229340+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:54.587 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:22:54.587 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:22:54.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.734+0000 7fc429913640 1 Processor -- start 2024-03-06T07:22:54.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.734+0000 7fc429913640 1 -- start start 2024-03-06T07:22:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.734+0000 7fc429913640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4240cf200 0x7fc4240cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.734+0000 7fc429913640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4241600a0 con 0x7fc42415fc30 2024-03-06T07:22:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.734+0000 7fc429913640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4240d47f0 con 0x7fc4240d06e0 2024-03-06T07:22:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.734+0000 7fc429913640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4240ced70 con 0x7fc4240cf200 2024-03-06T07:22:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.735+0000 7fc4237fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4240cf200 0x7fc4240cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.735+0000 7fc4227fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc42415fc30 0x7fc4240b47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:48098/0 (socket says 172.21.15.59:48098) 2024-03-06T07:22:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.735+0000 7fc4227fc640 1 -- 172.21.15.59:0/3523451776 learned_addr learned my addr 172.21.15.59:0/3523451776 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.735+0000 7fc4237fe640 1 -- 172.21.15.59:0/3523451776 >> v1:172.21.15.164:6789/0 conn(0x7fc4240d06e0 legacy=0x7fc4240d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.735+0000 7fc4237fe640 1 -- 172.21.15.59:0/3523451776 >> v1:172.21.15.59:6789/0 conn(0x7fc42415fc30 legacy=0x7fc4240b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.735+0000 7fc4237fe640 1 -- 172.21.15.59:0/3523451776 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4240d1ab0 con 0x7fc4240cf200 2024-03-06T07:22:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.735+0000 7fc421ffb640 1 -- 172.21.15.59:0/3523451776 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639490830 0 0) 0x7fc4241600a0 con 0x7fc42415fc30 2024-03-06T07:22:54.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.736+0000 7fc4237fe640 1 --2- 172.21.15.59:0/3523451776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4240cf200 0x7fc4240cf670 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc41c009d90 tx=0x7fc41c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab04e829f23b56c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.736+0000 7fc421ffb640 1 -- 172.21.15.59:0/3523451776 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c039070 con 0x7fc4240cf200 2024-03-06T07:22:54.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.736+0000 7fc421ffb640 1 -- 172.21.15.59:0/3523451776 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41c02cc60 con 0x7fc4240cf200 2024-03-06T07:22:54.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.736+0000 7fc421ffb640 1 -- 172.21.15.59:0/3523451776 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c034660 con 0x7fc4240cf200 2024-03-06T07:22:54.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.736+0000 7fc429913640 1 -- 172.21.15.59:0/3523451776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4240cf200 msgr2=0x7fc4240cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.736+0000 7fc429913640 1 --2- 172.21.15.59:0/3523451776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4240cf200 0x7fc4240cf670 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc41c009d90 tx=0x7fc41c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.736+0000 7fc429913640 1 -- 172.21.15.59:0/3523451776 shutdown_connections 2024-03-06T07:22:54.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.736+0000 7fc429913640 1 --2- 172.21.15.59:0/3523451776 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4240cf200 0x7fc4240cf670 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.736+0000 7fc429913640 1 -- 172.21.15.59:0/3523451776 >> 172.21.15.59:0/3523451776 conn(0x7fc4240d3cd0 msgr2=0x7fc4240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.736+0000 7fc429913640 1 -- 172.21.15.59:0/3523451776 shutdown_connections 2024-03-06T07:22:54.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.736+0000 7fc429913640 1 -- 172.21.15.59:0/3523451776 wait complete. 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.737+0000 7fc429913640 1 Processor -- start 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.738+0000 7fc429913640 1 -- start start 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.738+0000 7fc429913640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4240cf200 0x7fc4241e8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.738+0000 7fc429913640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4240d06e0 0x7fc42415d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.738+0000 7fc429913640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc42415fc30 0x7fc4240b9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.738+0000 7fc429913640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240ce990 con 0x7fc4240d06e0 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.738+0000 7fc429913640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240d4ab0 con 0x7fc42415fc30 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.738+0000 7fc429913640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4240b44d0 con 0x7fc4240cf200 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.738+0000 7fc4227fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4240d06e0 0x7fc42415d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.738+0000 7fc4227fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4240d06e0 0x7fc42415d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57820/0 (socket says 172.21.15.59:57820) 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.738+0000 7fc4227fc640 1 -- 172.21.15.59:0/1853305720 learned_addr learned my addr 172.21.15.59:0/1853305720 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc4237fe640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc42415fc30 0x7fc4240b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc422ffd640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4240cf200 0x7fc4241e8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc4227fc640 1 -- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4240cf200 msgr2=0x7fc4241e8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc4227fc640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4240cf200 0x7fc4241e8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc4227fc640 1 -- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc42415fc30 msgr2=0x7fc4240b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc4227fc640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc42415fc30 0x7fc4240b9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc4227fc640 1 -- 172.21.15.59:0/1853305720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4240b80c0 con 0x7fc4240d06e0 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc422ffd640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4240cf200 0x7fc4241e8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc4227fc640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4240d06e0 0x7fc42415d390 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc41800c430 tx=0x7fc41800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc40bfff640 1 -- 172.21.15.59:0/1853305720 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4180074e0 con 0x7fc4240d06e0 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc40bfff640 1 -- 172.21.15.59:0/1853305720 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4180047e0 con 0x7fc4240d06e0 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc429913640 1 -- 172.21.15.59:0/1853305720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4240b83a0 con 0x7fc4240d06e0 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc40bfff640 1 -- 172.21.15.59:0/1853305720 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418004bd0 con 0x7fc4240d06e0 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.739+0000 7fc429913640 1 -- 172.21.15.59:0/1853305720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4240b8960 con 0x7fc4240d06e0 2024-03-06T07:22:54.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.740+0000 7fc429913640 1 -- 172.21.15.59:0/1853305720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4240a1c80 con 0x7fc4240d06e0 2024-03-06T07:22:54.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.741+0000 7fc40bfff640 1 -- 172.21.15.59:0/1853305720 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc418011070 con 0x7fc4240d06e0 2024-03-06T07:22:54.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.741+0000 7fc40bfff640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc41006b500 0x7fc41006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.741+0000 7fc40bfff640 1 -- 172.21.15.59:0/1853305720 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc418089de0 con 0x7fc4240d06e0 2024-03-06T07:22:54.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.741+0000 7fc422ffd640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc41006b500 0x7fc41006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.742+0000 7fc422ffd640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc41006b500 0x7fc41006d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc41400a810 tx=0x7fc414008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.745+0000 7fc40bfff640 1 -- 172.21.15.59:0/1853305720 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc418058cf0 con 0x7fc4240d06e0 2024-03-06T07:22:54.904 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-03-06T07:22:54.902+0000 7fc2198f1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:54.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.995+0000 7fc429913640 1 -- 172.21.15.59:0/1853305720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc424162870 con 0x7fc4240d06e0 2024-03-06T07:22:54.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.997+0000 7fc40bfff640 1 -- 172.21.15.59:0/1853305720 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc4180585f0 con 0x7fc4240d06e0 2024-03-06T07:22:54.999 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-06T07:22:54.999 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA== 2024-03-06T07:22:54.999 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-03-06T07:22:54.999 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-03-06T07:22:54.999 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:22:55.000 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-03-06T07:22:55.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.999+0000 7fc429913640 1 -- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc41006b500 msgr2=0x7fc41006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.999+0000 7fc429913640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc41006b500 0x7fc41006d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc41400a810 tx=0x7fc414008040 comp rx=0 tx=0).stop 2024-03-06T07:22:55.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.999+0000 7fc429913640 1 -- 172.21.15.59:0/1853305720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4240d06e0 msgr2=0x7fc42415d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.999+0000 7fc429913640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4240d06e0 0x7fc42415d390 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc41800c430 tx=0x7fc41800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.999+0000 7fc429913640 1 -- 172.21.15.59:0/1853305720 shutdown_connections 2024-03-06T07:22:55.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.999+0000 7fc429913640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc41006b500 0x7fc41006d9c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.999+0000 7fc429913640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc42415fc30 0x7fc4240b9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.999+0000 7fc429913640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4240d06e0 0x7fc42415d390 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.999+0000 7fc429913640 1 --2- 172.21.15.59:0/1853305720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4240cf200 0x7fc4241e8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:54.999+0000 7fc429913640 1 -- 172.21.15.59:0/1853305720 >> 172.21.15.59:0/1853305720 conn(0x7fc4240d3cd0 msgr2=0x7fc424161090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.000+0000 7fc429913640 1 -- 172.21.15.59:0/1853305720 shutdown_connections 2024-03-06T07:22:55.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.000+0000 7fc429913640 1 -- 172.21.15.59:0/1853305720 wait complete. 2024-03-06T07:22:55.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:55.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.115+0000 7f13638f4640 1 Processor -- start 2024-03-06T07:22:55.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.115+0000 7f13638f4640 1 -- start start 2024-03-06T07:22:55.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f13638f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135c0b4170 0x7f135c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f13638f4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f135c0d4760 0x7f135c0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f13638f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f135c0d25d0 con 0x7f135c0b4dd0 2024-03-06T07:22:55.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f13638f4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135c0d4bf0 con 0x7f135c0d4760 2024-03-06T07:22:55.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f13638f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0ce9e0 con 0x7f135c0b4170 2024-03-06T07:22:55.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f1360e8f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f135c0d4760 0x7f135c0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f1361e91640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f135c0b4dd0 0x7f135c0cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:48104/0 (socket says 172.21.15.59:48104) 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f1361e91640 1 -- 172.21.15.59:0/1932147832 learned_addr learned my addr 172.21.15.59:0/1932147832 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f1360e8f640 1 -- 172.21.15.59:0/1932147832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135c0b4170 msgr2=0x7f135c0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f1361690640 1 --2- 172.21.15.59:0/1932147832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135c0b4170 0x7f135c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f1360e8f640 1 --2- 172.21.15.59:0/1932147832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135c0b4170 0x7f135c0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f1360e8f640 1 -- 172.21.15.59:0/1932147832 >> v1:172.21.15.59:6789/0 conn(0x7f135c0b4dd0 legacy=0x7f135c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f1353fff640 1 -- 172.21.15.59:0/1932147832 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908006350 0 0) 0x7f135c0d25d0 con 0x7f135c0b4dd0 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f1360e8f640 1 -- 172.21.15.59:0/1932147832 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c0d2870 con 0x7f135c0d4760 2024-03-06T07:22:55.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.116+0000 7f1361690640 1 --2- 172.21.15.59:0/1932147832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135c0b4170 0x7f135c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:55.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.117+0000 7f1360e8f640 1 --2- 172.21.15.59:0/1932147832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f135c0d4760 0x7f135c0d0370 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f134c0097a0 tx=0x7f134c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dae60c78ff01afa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.117+0000 7f1353fff640 1 -- 172.21.15.59:0/1932147832 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c035020 con 0x7f135c0d4760 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.117+0000 7f1353fff640 1 -- 172.21.15.59:0/1932147832 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c031830 con 0x7f135c0d4760 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.117+0000 7f1353fff640 1 -- 172.21.15.59:0/1932147832 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c031c00 con 0x7f135c0d4760 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.118+0000 7f13638f4640 1 -- 172.21.15.59:0/1932147832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f135c0d4760 msgr2=0x7f135c0d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.118+0000 7f13638f4640 1 --2- 172.21.15.59:0/1932147832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f135c0d4760 0x7f135c0d0370 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f134c0097a0 tx=0x7f134c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.118+0000 7f13638f4640 1 -- 172.21.15.59:0/1932147832 shutdown_connections 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.118+0000 7f13638f4640 1 --2- 172.21.15.59:0/1932147832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f135c0d4760 0x7f135c0d0370 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.118+0000 7f13638f4640 1 --2- 172.21.15.59:0/1932147832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135c0b4170 0x7f135c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.118+0000 7f13638f4640 1 -- 172.21.15.59:0/1932147832 >> 172.21.15.59:0/1932147832 conn(0x7f135c0d3c40 msgr2=0x7f135c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.118+0000 7f13638f4640 1 -- 172.21.15.59:0/1932147832 shutdown_connections 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.118+0000 7f13638f4640 1 -- 172.21.15.59:0/1932147832 wait complete. 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.119+0000 7f13638f4640 1 Processor -- start 2024-03-06T07:22:55.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.119+0000 7f13638f4640 1 -- start start 2024-03-06T07:22:55.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f13638f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135c0b4170 0x7f135c15ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f13638f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f135c0b4dd0 0x7f135c15b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f13638f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f135c1f3a10 0x7f135c0c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f13638f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0b47d0 con 0x7f135c1f3a10 2024-03-06T07:22:55.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f13638f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0b4550 con 0x7f135c0b4dd0 2024-03-06T07:22:55.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f13638f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f135c0b4950 con 0x7f135c0b4170 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f1361e91640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f135c1f3a10 0x7f135c0c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f1361e91640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f135c1f3a10 0x7f135c0c7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57830/0 (socket says 172.21.15.59:57830) 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f1361e91640 1 -- 172.21.15.59:0/3324992798 learned_addr learned my addr 172.21.15.59:0/3324992798 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f1361690640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135c0b4170 0x7f135c15ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f1360e8f640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f135c0b4dd0 0x7f135c15b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f1361e91640 1 -- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135c0b4170 msgr2=0x7f135c15ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f1361e91640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135c0b4170 0x7f135c15ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f1361e91640 1 -- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f135c0b4dd0 msgr2=0x7f135c15b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f1361e91640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f135c0b4dd0 0x7f135c15b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f1361e91640 1 -- 172.21.15.59:0/3324992798 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c1f42c0 con 0x7f135c1f3a10 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.120+0000 7f1361e91640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f135c1f3a10 0x7f135c0c7f40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f135400c7a0 tx=0x7f135400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.121+0000 7f1351ffb640 1 -- 172.21.15.59:0/3324992798 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354018070 con 0x7f135c1f3a10 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.121+0000 7f1351ffb640 1 -- 172.21.15.59:0/3324992798 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135400a640 con 0x7f135c1f3a10 2024-03-06T07:22:55.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.121+0000 7f1351ffb640 1 -- 172.21.15.59:0/3324992798 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135400aab0 con 0x7f135c1f3a10 2024-03-06T07:22:55.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.121+0000 7f13638f4640 1 -- 172.21.15.59:0/3324992798 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135c1f45a0 con 0x7f135c1f3a10 2024-03-06T07:22:55.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.121+0000 7f13638f4640 1 -- 172.21.15.59:0/3324992798 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c0c7740 con 0x7f135c1f3a10 2024-03-06T07:22:55.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.122+0000 7f13638f4640 1 -- 172.21.15.59:0/3324992798 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c04f330 con 0x7f135c1f3a10 2024-03-06T07:22:55.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.122+0000 7f1351ffb640 1 -- 172.21.15.59:0/3324992798 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f135400ac50 con 0x7f135c1f3a10 2024-03-06T07:22:55.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.122+0000 7f1351ffb640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134006b4d0 0x7f134006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.123+0000 7f1351ffb640 1 -- 172.21.15.59:0/3324992798 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1354089970 con 0x7f135c1f3a10 2024-03-06T07:22:55.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.123+0000 7f1361690640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134006b4d0 0x7f134006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.123+0000 7f1361690640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134006b4d0 0x7f134006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1358008850 tx=0x7f1358008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.127+0000 7f1351ffb640 1 -- 172.21.15.59:0/3324992798 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1354058770 con 0x7f135c1f3a10 2024-03-06T07:22:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.347+0000 7f13638f4640 1 -- 172.21.15.59:0/3324992798 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f135c0733b0 con 0x7f135c1f3a10 2024-03-06T07:22:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.347+0000 7f1351ffb640 1 -- 172.21.15.59:0/3324992798 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f1354058070 con 0x7f135c1f3a10 2024-03-06T07:22:55.349 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:55.349 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:22:55.350 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:22:55.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 -- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134006b4d0 msgr2=0x7f134006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134006b4d0 0x7f134006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1358008850 tx=0x7f1358008040 comp rx=0 tx=0).stop 2024-03-06T07:22:55.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 -- 172.21.15.59:0/3324992798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f135c1f3a10 msgr2=0x7f135c0c7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f135c1f3a10 0x7f135c0c7f40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f135400c7a0 tx=0x7f135400cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:55.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 -- 172.21.15.59:0/3324992798 shutdown_connections 2024-03-06T07:22:55.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134006b4d0 0x7f134006d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f135c1f3a10 0x7f135c0c7f40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f135c0b4dd0 0x7f135c15b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 --2- 172.21.15.59:0/3324992798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135c0b4170 0x7f135c15ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 -- 172.21.15.59:0/3324992798 >> 172.21.15.59:0/3324992798 conn(0x7f135c0d3c40 msgr2=0x7f135c0b6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 -- 172.21.15.59:0/3324992798 shutdown_connections 2024-03-06T07:22:55.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.349+0000 7f13638f4640 1 -- 172.21.15.59:0/3324992798 wait complete. 2024-03-06T07:22:55.768 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:50.585871+0000', 'modified': '2024-03-06T07:22:52.584255+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5250}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5250': {'gid': 5250, 'name': 'a', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1359034023', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6833', 'nonce': 1359034023}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1359034023}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5250, 'qdb_cluster': [5250]} 2024-03-06T07:22:55.768 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:55.768 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:22:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.872+0000 7f1fe82e1640 1 Processor -- start 2024-03-06T07:22:55.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.872+0000 7f1fe82e1640 1 -- start start 2024-03-06T07:22:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.873+0000 7f1fe82e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe01614a0 0x7f1fe006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.873+0000 7f1fe82e1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1fe00b1a20 0x7f1fe00b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.873+0000 7f1fe82e1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00af4b0 con 0x7f1fe00b0270 2024-03-06T07:22:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.873+0000 7f1fe82e1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe00af790 con 0x7f1fe00b1a20 2024-03-06T07:22:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.873+0000 7f1fe82e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00afa70 con 0x7f1fe01614a0 2024-03-06T07:22:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe587c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1fe00b0270 0x7f1fe00b1310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:48108/0 (socket says 172.21.15.59:48108) 2024-03-06T07:22:55.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe587c640 1 -- 172.21.15.59:0/1006647309 learned_addr learned my addr 172.21.15.59:0/1006647309 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe687e640 1 --2- 172.21.15.59:0/1006647309 >> v2:172.21.15.164:3300/0 conn(0x7f1fe00b1a20 0x7f1fe00b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe607d640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe01614a0 0x7f1fe006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179676180 0 0) 0x7f1fe00af4b0 con 0x7f1fe00b0270 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc003520 con 0x7f1fe00b0270 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1650612796 0 0) 0x7f1fcc003520 con 0x7f1fe00b0270 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe01614a0 msgr2=0x7f1fe006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe01614a0 0x7f1fe006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 >> v2:172.21.15.164:3300/0 conn(0x7f1fe00b1a20 msgr2=0x7f1fe00b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 --2- 172.21.15.59:0/1006647309 >> v2:172.21.15.164:3300/0 conn(0x7f1fe00b1a20 0x7f1fe00b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00afd50 con 0x7f1fe00b0270 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f1fd0001670 con 0x7f1fe00b0270 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 >> v1:172.21.15.59:6789/0 conn(0x7f1fe00b0270 legacy=0x7f1fe00b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fcc0039b0 0x7f1fcc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 0x7f1fcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe01614a0 0x7f1fcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00af790 con 0x7f1fcc004800 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00afa70 con 0x7f1fcc0039b0 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc00b9c0 con 0x7f1fe01614a0 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.874+0000 7f1fe587c640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 0x7f1fcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.875+0000 7f1fe607d640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fcc0039b0 0x7f1fcc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.875+0000 7f1fe687e640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe01614a0 0x7f1fcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.875+0000 7f1fe587c640 1 -- 172.21.15.59:0/1006647309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe01614a0 msgr2=0x7f1fcc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.875+0000 7f1fe587c640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe01614a0 0x7f1fcc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.875+0000 7f1fe587c640 1 -- 172.21.15.59:0/1006647309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fcc0039b0 msgr2=0x7f1fcc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.875+0000 7f1fe587c640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fcc0039b0 0x7f1fcc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.875+0000 7f1fe587c640 1 -- 172.21.15.59:0/1006647309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc003520 con 0x7f1fcc004800 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.875+0000 7f1fe587c640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 0x7f1fcc004c80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c210 tx=0x7f1fcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfdfe1af4737815c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.876+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0007d70 con 0x7f1fcc004800 2024-03-06T07:22:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.876+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd0030070 con 0x7f1fcc004800 2024-03-06T07:22:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.876+0000 7f1fe507b640 1 -- 172.21.15.59:0/1006647309 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd002cd60 con 0x7f1fcc004800 2024-03-06T07:22:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.876+0000 7f1fe82e1640 1 -- 172.21.15.59:0/1006647309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 msgr2=0x7f1fcc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.876+0000 7f1fe82e1640 1 --2- 172.21.15.59:0/1006647309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 0x7f1fcc004c80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c210 tx=0x7f1fcc005830 comp rx=0 tx=0).stop 2024-03-06T07:22:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.876+0000 7f1fe687e640 1 -- 172.21.15.59:0/1006647309 reap_dead start 2024-03-06T07:22:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.876+0000 7f1fe82e1640 1 -- 172.21.15.59:0/1006647309 shutdown_connections 2024-03-06T07:22:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.876+0000 7f1fe82e1640 1 -- 172.21.15.59:0/1006647309 >> 172.21.15.59:0/1006647309 conn(0x7f1fe009a9f0 msgr2=0x7f1fe009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.876+0000 7f1fe82e1640 1 -- 172.21.15.59:0/1006647309 shutdown_connections 2024-03-06T07:22:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.876+0000 7f1fe82e1640 1 -- 172.21.15.59:0/1006647309 wait complete. 2024-03-06T07:22:55.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.878+0000 7f1fe82e1640 1 Processor -- start 2024-03-06T07:22:55.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.878+0000 7f1fe82e1640 1 -- start start 2024-03-06T07:22:55.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.878+0000 7f1fe82e1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 0x7f1fe01e93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.878+0000 7f1fe82e1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe0158540 0x7f1fe01e98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.878+0000 7f1fe82e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe015f900 0x7f1fe01f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.878+0000 7f1fe82e1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00b06b0 con 0x7f1fcc004800 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.878+0000 7f1fe82e1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00b1e50 con 0x7f1fe0158540 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.878+0000 7f1fe82e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe006d3c0 con 0x7f1fe015f900 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.878+0000 7f1fe607d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 0x7f1fe01e93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.878+0000 7f1fe607d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 0x7f1fe01e93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57848/0 (socket says 172.21.15.59:57848) 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.878+0000 7f1fe607d640 1 -- 172.21.15.59:0/3693835600 learned_addr learned my addr 172.21.15.59:0/3693835600 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fe587c640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe0158540 0x7f1fe01e98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fe687e640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe015f900 0x7f1fe01f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fe607d640 1 -- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe015f900 msgr2=0x7f1fe01f3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fe607d640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe015f900 0x7f1fe01f3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fe607d640 1 -- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe0158540 msgr2=0x7f1fe01e98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fe607d640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe0158540 0x7f1fe01e98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fe607d640 1 -- 172.21.15.59:0/3693835600 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe01e6d30 con 0x7f1fcc004800 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fe687e640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe015f900 0x7f1fe01f3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:55.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fe607d640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 0x7f1fe01e93b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc011940 tx=0x7f1fdc011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fd6ffd640 1 -- 172.21.15.59:0/3693835600 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc00faf0 con 0x7f1fcc004800 2024-03-06T07:22:55.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fd6ffd640 1 -- 172.21.15.59:0/3693835600 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc00fc90 con 0x7f1fcc004800 2024-03-06T07:22:55.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fe82e1640 1 -- 172.21.15.59:0/3693835600 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe015ee20 con 0x7f1fcc004800 2024-03-06T07:22:55.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fd6ffd640 1 -- 172.21.15.59:0/3693835600 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc014440 con 0x7f1fcc004800 2024-03-06T07:22:55.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.879+0000 7f1fe82e1640 1 -- 172.21.15.59:0/3693835600 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe015f3e0 con 0x7f1fcc004800 2024-03-06T07:22:55.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.881+0000 7f1fd6ffd640 1 -- 172.21.15.59:0/3693835600 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fdc012040 con 0x7f1fcc004800 2024-03-06T07:22:55.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.881+0000 7f1fd6ffd640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fc406b500 0x7f1fc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.881+0000 7f1fd6ffd640 1 -- 172.21.15.59:0/3693835600 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fdc08a750 con 0x7f1fcc004800 2024-03-06T07:22:55.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.882+0000 7f1fe587c640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fc406b500 0x7f1fc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.882+0000 7f1feaf9c740 1 Processor -- start 2024-03-06T07:22:55.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.882+0000 7f1fe587c640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fc406b500 0x7f1fc406d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0034000 tx=0x7f1fd0007000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.882+0000 7f1feaf9c740 1 -- start start 2024-03-06T07:22:55.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.883+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae690870 0x55d6ae690c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.883+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6948c0 0x55d6ae696360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.883+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6c0600 0x55d6ae6c0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.883+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ae5fa6b0 con 0x55d6ae690870 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.883+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ae5a3a60 con 0x55d6ae6c0600 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.883+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d6ae5f6750 con 0x55d6ae6948c0 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.883+0000 7f1fe607d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae690870 0x55d6ae690c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.883+0000 7f1fe607d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae690870 0x55d6ae690c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57864/0 (socket says 172.21.15.59:57864) 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.883+0000 7f1fe607d640 1 -- 172.21.15.59:0/2743615803 learned_addr learned my addr 172.21.15.59:0/2743615803 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.883+0000 7f1fe687e640 1 --2- 172.21.15.59:0/2743615803 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6c0600 0x55d6ae6c0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.883+0000 7f1fe587c640 1 --2- 172.21.15.59:0/2743615803 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6948c0 0x55d6ae696360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.884+0000 7f1fe607d640 1 -- 172.21.15.59:0/2743615803 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6948c0 msgr2=0x55d6ae696360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.884+0000 7f1fe607d640 1 --2- 172.21.15.59:0/2743615803 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6948c0 0x55d6ae696360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.884+0000 7f1fe607d640 1 -- 172.21.15.59:0/2743615803 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6c0600 msgr2=0x55d6ae6c0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.884+0000 7f1fe607d640 1 --2- 172.21.15.59:0/2743615803 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6c0600 0x55d6ae6c0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.884+0000 7f1fe607d640 1 -- 172.21.15.59:0/2743615803 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6ae5c1f30 con 0x55d6ae690870 2024-03-06T07:22:55.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.884+0000 7f1fe607d640 1 --2- 172.21.15.59:0/2743615803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae690870 0x55d6ae690c50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc088bf0 tx=0x7f1fdc058930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356c7207efb3c869 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.884+0000 7f1fc3fff640 1 -- 172.21.15.59:0/2743615803 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0615e0 con 0x55d6ae690870 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.884+0000 7f1fc3fff640 1 -- 172.21.15.59:0/2743615803 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc0571d0 con 0x55d6ae690870 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.884+0000 7f1fc3fff640 1 -- 172.21.15.59:0/2743615803 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc059130 con 0x55d6ae690870 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.885+0000 7f1feaf9c740 1 -- 172.21.15.59:0/2743615803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae690870 msgr2=0x55d6ae690c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.885+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/2743615803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae690870 0x55d6ae690c50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc088bf0 tx=0x7f1fdc058930 comp rx=0 tx=0).stop 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.885+0000 7f1feaf9c740 1 -- 172.21.15.59:0/2743615803 shutdown_connections 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.885+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/2743615803 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6c0600 0x55d6ae6c0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.885+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/2743615803 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6948c0 0x55d6ae696360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.885+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/2743615803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae690870 0x55d6ae690c50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.885+0000 7f1feaf9c740 1 -- 172.21.15.59:0/2743615803 >> 172.21.15.59:0/2743615803 conn(0x55d6ae6928f0 msgr2=0x55d6ae6941b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.885+0000 7f1feaf9c740 1 -- 172.21.15.59:0/2743615803 shutdown_connections 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.885+0000 7f1feaf9c740 1 -- 172.21.15.59:0/2743615803 wait complete. 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.885+0000 7f1feaf9c740 1 -- start start 2024-03-06T07:22:55.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.885+0000 7f1feaf9c740 1 Processor -- start 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.886+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6c0600 0x55d6ae6bb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.886+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6b5410 0x55d6ae6b2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.886+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae6b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.886+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ae694d50 con 0x55d6ae6b3850 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.886+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ae5b6ea0 con 0x55d6ae6c0600 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.886+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d6ae695030 con 0x55d6ae6b5410 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.886+0000 7f1feaf9c740 10 client.0 _ll_register_callbacks cb 0x55d6ae5b6870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.886+0000 7f1fe687e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae6b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.886+0000 7f1fe687e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae6b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57870/0 (socket says 172.21.15.59:57870) 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.886+0000 7f1fe687e640 1 -- 172.21.15.59:0/3159206109 learned_addr learned my addr 172.21.15.59:0/3159206109 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fe607d640 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6c0600 0x55d6ae6bb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fe587c640 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6b5410 0x55d6ae6b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fe687e640 1 -- 172.21.15.59:0/3159206109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6b5410 msgr2=0x55d6ae6b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fe687e640 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6b5410 0x55d6ae6b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fe687e640 1 -- 172.21.15.59:0/3159206109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6c0600 msgr2=0x55d6ae6bb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fe687e640 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6c0600 0x55d6ae6bb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fe687e640 1 -- 172.21.15.59:0/3159206109 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d6ae5b7270 con 0x55d6ae6b3850 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fe587c640 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6b5410 0x55d6ae6b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fe687e640 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae6b2c90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0071d30 tx=0x7f1fd8004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed364a4f59de4be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fc17fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fc17fa640 1 -- 172.21.15.59:0/3159206109 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd801b070 con 0x55d6ae6b3850 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fc17fa640 1 -- 172.21.15.59:0/3159206109 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fd8007c40 con 0x55d6ae6b3850 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1feaf9c740 1 -- 172.21.15.59:0/3159206109 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1fdc0571d0 con 0x55d6ae6b3850 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fc17fa640 1 -- 172.21.15.59:0/3159206109 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fd8026050 con 0x55d6ae6b3850 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1feaf9c740 10 client.5427 fetch_fsmap learned FSMap version 13 2024-03-06T07:22:55.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1feaf9c740 1 -- 172.21.15.59:0/3159206109 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x55d6ae30f2d0 con 0x55d6ae6b3850 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fc17fa640 1 -- 172.21.15.59:0/3159206109 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc0571d0 con 0x55d6ae6b3850 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.887+0000 7f1fc17fa640 1 -- 172.21.15.59:0/3159206109 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 13) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f1fd8029020 con 0x55d6ae6b3850 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.888+0000 7f1feaf9c740 10 client.5427 fetch_fsmap finished waiting for FSMap version 13 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.888+0000 7f1feaf9c740 10 client.5427 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.888+0000 7f1feaf9c740 20 client.5427 populate_metadata read hostname 'smithi059' 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.888+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.59:6833/1359034023,v1:172.21.15.59:6835/1359034023] conn(0x55d6ae6bca60 0x55d6ae6fa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.888+0000 7f1feaf9c740 4 client.5427 mds_command: new command op to 5250 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.888+0000 7f1feaf9c740 1 -- 172.21.15.59:0/3159206109 --> [v2:172.21.15.59:6833/1359034023,v1:172.21.15.59:6835/1359034023] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d6ae607bf0 con 0x55d6ae6bca60 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.888+0000 7f1fe607d640 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.59:6833/1359034023,v1:172.21.15.59:6835/1359034023] conn(0x55d6ae6bca60 0x55d6ae6fa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.888+0000 7f1fc17fa640 10 client.5427 ms_handle_connect on v2:172.21.15.59:6833/1359034023 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.888+0000 7f1fe607d640 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.59:6833/1359034023,v1:172.21.15.59:6835/1359034023] conn(0x55d6ae6bca60 0x55d6ae6fa810 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdeceab4229a0c47 server_cookie=2ac3ceab8f5c2f3d in_seq=0 out_seq=0 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1fc17fa640 1 -- 172.21.15.59:0/3159206109 <== mds.0 v2:172.21.15.59:6833/1359034023 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d6ae607bf0 con 0x55d6ae6bca60 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1fc17fa640 10 client.5427 handle_command_reply: tid=0 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 1 client.5427 shutdown 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 1 -- 172.21.15.59:0/3159206109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 msgr2=0x55d6ae6b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae6b2c90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0071d30 tx=0x7f1fd8004700 comp rx=0 tx=0).stop 2024-03-06T07:22:55.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 1 -- 172.21.15.59:0/3159206109 shutdown_connections 2024-03-06T07:22:55.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.59:6833/1359034023,v1:172.21.15.59:6835/1359034023] conn(0x55d6ae6bca60 0x55d6ae6fa810 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae6b2c90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6b5410 0x55d6ae6b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/3159206109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6c0600 0x55d6ae6bb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1fc17fa640 0 client.5427 ms_handle_reset on v2:172.21.15.59:6833/1359034023 2024-03-06T07:22:55.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 1 -- 172.21.15.59:0/3159206109 >> 172.21.15.59:0/3159206109 conn(0x55d6ae690870 msgr2=0x55d6ae6be5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 1 -- 172.21.15.59:0/3159206109 shutdown_connections 2024-03-06T07:22:55.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 1 -- 172.21.15.59:0/3159206109 wait complete. 2024-03-06T07:22:55.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.889+0000 7f1feaf9c740 20 client.5427 trim_cache size 0 max 16384 2024-03-06T07:22:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.922+0000 7f1feaf9c740 1 Processor -- start 2024-03-06T07:22:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.922+0000 7f1feaf9c740 1 -- start start 2024-03-06T07:22:55.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.923+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae68bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.923+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6b5410 0x55d6ae6ad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.923+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6c0600 0x55d6ae6ee070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.923+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe02008b0 con 0x55d6ae6b3850 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.923+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe01ffa40 con 0x55d6ae6b5410 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.923+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe01e8710 con 0x55d6ae6c0600 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.923+0000 7f1fe607d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae68bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.923+0000 7f1fe607d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae68bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57872/0 (socket says 172.21.15.59:57872) 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.923+0000 7f1fe607d640 1 -- 172.21.15.59:0/1154019982 learned_addr learned my addr 172.21.15.59:0/1154019982 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.924+0000 7f1fe607d640 1 -- 172.21.15.59:0/1154019982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6c0600 msgr2=0x55d6ae6ee070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.924+0000 7f1fe607d640 1 --2- 172.21.15.59:0/1154019982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6c0600 0x55d6ae6ee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.924+0000 7f1fe607d640 1 -- 172.21.15.59:0/1154019982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6b5410 msgr2=0x55d6ae6ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.924+0000 7f1fe607d640 1 --2- 172.21.15.59:0/1154019982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6b5410 0x55d6ae6ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.924+0000 7f1fe607d640 1 -- 172.21.15.59:0/1154019982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6ae3d96b0 con 0x55d6ae6b3850 2024-03-06T07:22:55.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.924+0000 7f1fe607d640 1 --2- 172.21.15.59:0/1154019982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae68bea0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00f610 tx=0x7f1fdc06d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ef31c10b9dee10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.924+0000 7f1fc37fe640 1 -- 172.21.15.59:0/1154019982 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc07c070 con 0x55d6ae6b3850 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.924+0000 7f1fc37fe640 1 -- 172.21.15.59:0/1154019982 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc055540 con 0x55d6ae6b3850 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.924+0000 7f1fc37fe640 1 -- 172.21.15.59:0/1154019982 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc059330 con 0x55d6ae6b3850 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.925+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1154019982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 msgr2=0x55d6ae68bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.925+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/1154019982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae68bea0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00f610 tx=0x7f1fdc06d000 comp rx=0 tx=0).stop 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.925+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1154019982 shutdown_connections 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.925+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/1154019982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6c0600 0x55d6ae6ee070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.925+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/1154019982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6b5410 0x55d6ae6ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.925+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/1154019982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6b3850 0x55d6ae68bea0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.925+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1154019982 >> 172.21.15.59:0/1154019982 conn(0x55d6ae690870 msgr2=0x55d6ae694080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.925+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1154019982 shutdown_connections 2024-03-06T07:22:55.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.925+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1154019982 wait complete. 2024-03-06T07:22:55.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.925+0000 7f1feaf9c740 1 -- start start 2024-03-06T07:22:55.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.926+0000 7f1feaf9c740 1 Processor -- start 2024-03-06T07:22:55.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6b3850 0x55d6ae6bd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6b5410 0x55d6ae6ba7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1feaf9c740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6c0600 0x55d6ae74a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ae6c2080 con 0x55d6ae6c0600 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe01ff440 con 0x55d6ae6b5410 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1feaf9c740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d6ae6b8640 con 0x55d6ae6b3850 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1feaf9c740 10 client.0 _ll_register_callbacks cb 0x7f1fd8009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1fe687e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6c0600 0x55d6ae74a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1fe687e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6c0600 0x55d6ae74a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57888/0 (socket says 172.21.15.59:57888) 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1fe687e640 1 -- 172.21.15.59:0/1685741417 learned_addr learned my addr 172.21.15.59:0/1685741417 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1fe687e640 1 -- 172.21.15.59:0/1685741417 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6b3850 msgr2=0x55d6ae6bd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1fe687e640 1 --2- 172.21.15.59:0/1685741417 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6b3850 0x55d6ae6bd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1fe687e640 1 -- 172.21.15.59:0/1685741417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6b5410 msgr2=0x55d6ae6ba7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1fe687e640 1 --2- 172.21.15.59:0/1685741417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6b5410 0x55d6ae6ba7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.927+0000 7f1fe687e640 1 -- 172.21.15.59:0/1685741417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d6ae6f6290 con 0x55d6ae6c0600 2024-03-06T07:22:55.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1fe687e640 1 --2- 172.21.15.59:0/1685741417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6c0600 0x55d6ae74a4b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8021820 tx=0x7f1fd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49589edb8d275d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1fc17fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1fc17fa640 1 -- 172.21.15.59:0/1685741417 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd801b050 con 0x55d6ae6c0600 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1fc17fa640 1 -- 172.21.15.59:0/1685741417 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fd8017440 con 0x55d6ae6c0600 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1685741417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1fdc055540 con 0x55d6ae6c0600 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1fc17fa640 1 -- 172.21.15.59:0/1685741417 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fd803b050 con 0x55d6ae6c0600 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1feaf9c740 10 client.5433 fetch_fsmap learned FSMap version 13 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1685741417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x55d6ae6b7060 con 0x55d6ae6c0600 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1fc17fa640 1 -- 172.21.15.59:0/1685741417 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc055540 con 0x55d6ae6c0600 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1fc17fa640 1 -- 172.21.15.59:0/1685741417 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 13) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f1fd803e020 con 0x55d6ae6c0600 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1feaf9c740 10 client.5433 fetch_fsmap finished waiting for FSMap version 13 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1feaf9c740 10 client.5433 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1feaf9c740 20 client.5433 populate_metadata read hostname 'smithi059' 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/1685741417 >> [v2:172.21.15.59:6833/1359034023,v1:172.21.15.59:6835/1359034023] conn(0x55d6ae6b91d0 0x55d6ae6ac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1feaf9c740 4 client.5433 mds_command: new command op to 5250 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1685741417 --> [v2:172.21.15.59:6833/1359034023,v1:172.21.15.59:6835/1359034023] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d6ae601f40 con 0x55d6ae6b91d0 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.928+0000 7f1fe607d640 1 --2- 172.21.15.59:0/1685741417 >> [v2:172.21.15.59:6833/1359034023,v1:172.21.15.59:6835/1359034023] conn(0x55d6ae6b91d0 0x55d6ae6ac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1fc17fa640 10 client.5433 ms_handle_connect on v2:172.21.15.59:6833/1359034023 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1fe607d640 1 --2- 172.21.15.59:0/1685741417 >> [v2:172.21.15.59:6833/1359034023,v1:172.21.15.59:6835/1359034023] conn(0x55d6ae6b91d0 0x55d6ae6ac4e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab252fe6804588df server_cookie=dc75cee482f02e2d in_seq=0 out_seq=0 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1fc17fa640 1 -- 172.21.15.59:0/1685741417 <== mds.0 v2:172.21.15.59:6833/1359034023 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55d6ae601f40 con 0x55d6ae6b91d0 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1fc17fa640 10 client.5433 handle_command_reply: tid=0 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 1 client.5433 shutdown 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1685741417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6c0600 msgr2=0x55d6ae74a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/1685741417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6c0600 0x55d6ae74a4b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8021820 tx=0x7f1fd8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1685741417 shutdown_connections 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/1685741417 >> [v2:172.21.15.59:6833/1359034023,v1:172.21.15.59:6835/1359034023] conn(0x55d6ae6b91d0 0x55d6ae6ac4e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/1685741417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55d6ae6c0600 0x55d6ae74a4b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/1685741417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d6ae6b5410 0x55d6ae6ba7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1fc17fa640 0 client.5433 ms_handle_reset on v2:172.21.15.59:6833/1359034023 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 1 --2- 172.21.15.59:0/1685741417 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55d6ae6b3850 0x55d6ae6bd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1685741417 >> 172.21.15.59:0/1685741417 conn(0x55d6ae690870 msgr2=0x55d6ae694080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1685741417 shutdown_connections 2024-03-06T07:22:55.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 1 -- 172.21.15.59:0/1685741417 wait complete. 2024-03-06T07:22:55.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.929+0000 7f1feaf9c740 20 client.5433 trim_cache size 0 max 16384 2024-03-06T07:22:55.936 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:22:55.936 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:22:55.936 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:22:55.936 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 5250, 2024-03-06T07:22:55.936 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:22:55.936 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:22:55.936 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:22:55.936 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 4.7145333640000002, 2024-03-06T07:22:55.936 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 13, 2024-03-06T07:22:55.937 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 33, 2024-03-06T07:22:55.937 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 33, 2024-03-06T07:22:55.937 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 11.875276432 2024-03-06T07:22:55.937 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:22:55.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 -- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fc406b500 msgr2=0x7f1fc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fc406b500 0x7f1fc406d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0034000 tx=0x7f1fd0007000 comp rx=0 tx=0).stop 2024-03-06T07:22:55.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 -- 172.21.15.59:0/3693835600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 msgr2=0x7f1fe01e93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 0x7f1fe01e93b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc011940 tx=0x7f1fdc011e00 comp rx=0 tx=0).stop 2024-03-06T07:22:55.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 -- 172.21.15.59:0/3693835600 shutdown_connections 2024-03-06T07:22:55.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fc406b500 0x7f1fc406d9c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe015f900 0x7f1fe01f3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe0158540 0x7f1fe01e98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 --2- 172.21.15.59:0/3693835600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc004800 0x7f1fe01e93b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 -- 172.21.15.59:0/3693835600 >> 172.21.15.59:0/3693835600 conn(0x7f1fe009a9f0 msgr2=0x7f1fe00b5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 -- 172.21.15.59:0/3693835600 shutdown_connections 2024-03-06T07:22:55.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:55.930+0000 7f1fe82e1640 1 -- 172.21.15.59:0/3693835600 wait complete. 2024-03-06T07:22:55.942 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.041+0000 7f0c67864640 1 Processor -- start 2024-03-06T07:22:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.041+0000 7f0c67864640 1 -- start start 2024-03-06T07:22:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.042+0000 7f0c67864640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0c6015dfe0 0x7f0c600cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.042+0000 7f0c67864640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c600b3430 con 0x7f0c600d29c0 2024-03-06T07:22:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.042+0000 7f0c67864640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c600b3710 con 0x7f0c600cf830 2024-03-06T07:22:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.042+0000 7f0c67864640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c600b31e0 con 0x7f0c6015dfe0 2024-03-06T07:22:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.042+0000 7f0c65600640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0c6015dfe0 0x7f0c600cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.042+0000 7f0c65600640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0c6015dfe0 0x7f0c600cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57896/0 (socket says 172.21.15.59:57896) 2024-03-06T07:22:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.042+0000 7f0c65600640 1 -- 172.21.15.59:0/3652225426 learned_addr learned my addr 172.21.15.59:0/3652225426 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.043+0000 7f0c65600640 1 -- 172.21.15.59:0/3652225426 >> v1:172.21.15.164:6789/0 conn(0x7f0c600cf830 legacy=0x7f0c600d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.043+0000 7f0c5ffff640 1 -- 172.21.15.59:0/3652225426 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522854350 0 0) 0x7f0c600b3430 con 0x7f0c600d29c0 2024-03-06T07:22:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.043+0000 7f0c65600640 1 -- 172.21.15.59:0/3652225426 >> v1:172.21.15.59:6789/0 conn(0x7f0c600d29c0 legacy=0x7f0c600b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.043+0000 7f0c65600640 1 -- 172.21.15.59:0/3652225426 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c600b39f0 con 0x7f0c6015dfe0 2024-03-06T07:22:56.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.044+0000 7f0c65600640 1 --2- 172.21.15.59:0/3652225426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c6015dfe0 0x7f0c600cf2f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0c600b3360 tx=0x7f0c5002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7ac5d2fab884ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.044+0000 7f0c5ffff640 1 -- 172.21.15.59:0/3652225426 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50034070 con 0x7f0c6015dfe0 2024-03-06T07:22:56.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.044+0000 7f0c5ffff640 1 -- 172.21.15.59:0/3652225426 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c50030840 con 0x7f0c6015dfe0 2024-03-06T07:22:56.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.044+0000 7f0c5ffff640 1 -- 172.21.15.59:0/3652225426 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50030c10 con 0x7f0c6015dfe0 2024-03-06T07:22:56.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.044+0000 7f0c67864640 1 -- 172.21.15.59:0/3652225426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c6015dfe0 msgr2=0x7f0c600cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.044+0000 7f0c67864640 1 --2- 172.21.15.59:0/3652225426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c6015dfe0 0x7f0c600cf2f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0c600b3360 tx=0x7f0c5002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:56.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.044+0000 7f0c67864640 1 -- 172.21.15.59:0/3652225426 shutdown_connections 2024-03-06T07:22:56.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.044+0000 7f0c67864640 1 --2- 172.21.15.59:0/3652225426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c6015dfe0 0x7f0c600cf2f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.044+0000 7f0c67864640 1 -- 172.21.15.59:0/3652225426 >> 172.21.15.59:0/3652225426 conn(0x7f0c600d3cd0 msgr2=0x7f0c600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.044+0000 7f0c67864640 1 -- 172.21.15.59:0/3652225426 shutdown_connections 2024-03-06T07:22:56.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.045+0000 7f0c67864640 1 -- 172.21.15.59:0/3652225426 wait complete. 2024-03-06T07:22:56.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.046+0000 7f0c67864640 1 Processor -- start 2024-03-06T07:22:56.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.046+0000 7f0c67864640 1 -- start start 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.046+0000 7f0c67864640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c600cf830 0x7f0c60154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.046+0000 7f0c67864640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c600d29c0 0x7f0c601f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.046+0000 7f0c67864640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c6015dfe0 0x7f0c601f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.046+0000 7f0c67864640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6015e3c0 con 0x7f0c6015dfe0 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.046+0000 7f0c67864640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c600b3d60 con 0x7f0c600cf830 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.046+0000 7f0c67864640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6015e540 con 0x7f0c600d29c0 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.046+0000 7f0c65e01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c6015dfe0 0x7f0c601f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c65e01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c6015dfe0 0x7f0c601f69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57904/0 (socket says 172.21.15.59:57904) 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c65e01640 1 -- 172.21.15.59:0/3118973537 learned_addr learned my addr 172.21.15.59:0/3118973537 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c64dff640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c600d29c0 0x7f0c601f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c65600640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c600cf830 0x7f0c60154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c65e01640 1 -- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c600d29c0 msgr2=0x7f0c601f4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c65e01640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c600d29c0 0x7f0c601f4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c65e01640 1 -- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c600cf830 msgr2=0x7f0c60154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c65e01640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c600cf830 0x7f0c60154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c65e01640 1 -- 172.21.15.59:0/3118973537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c601f5950 con 0x7f0c6015dfe0 2024-03-06T07:22:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c64dff640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c600d29c0 0x7f0c601f4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c65600640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c600cf830 0x7f0c60154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c65e01640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c6015dfe0 0x7f0c601f69f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800c760 tx=0x7f0c5800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c5dffb640 1 -- 172.21.15.59:0/3118973537 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58003da0 con 0x7f0c6015dfe0 2024-03-06T07:22:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c5dffb640 1 -- 172.21.15.59:0/3118973537 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c58005070 con 0x7f0c6015dfe0 2024-03-06T07:22:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c67864640 1 -- 172.21.15.59:0/3118973537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c601f5c30 con 0x7f0c6015dfe0 2024-03-06T07:22:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c5dffb640 1 -- 172.21.15.59:0/3118973537 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58005460 con 0x7f0c6015dfe0 2024-03-06T07:22:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.047+0000 7f0c67864640 1 -- 172.21.15.59:0/3118973537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c601f61f0 con 0x7f0c6015dfe0 2024-03-06T07:22:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.048+0000 7f0c67864640 1 -- 172.21.15.59:0/3118973537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6004f330 con 0x7f0c6015dfe0 2024-03-06T07:22:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.049+0000 7f0c5dffb640 1 -- 172.21.15.59:0/3118973537 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c58005600 con 0x7f0c6015dfe0 2024-03-06T07:22:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.049+0000 7f0c5dffb640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4c06b500 0x7f0c4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.049+0000 7f0c5dffb640 1 -- 172.21.15.59:0/3118973537 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c5808a190 con 0x7f0c6015dfe0 2024-03-06T07:22:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.050+0000 7f0c65600640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4c06b500 0x7f0c4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.050+0000 7f0c65600640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4c06b500 0x7f0c4c06d9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0c50036000 tx=0x7f0c5003b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:56.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.054+0000 7f0c5dffb640 1 -- 172.21.15.59:0/3118973537 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c58058f70 con 0x7f0c6015dfe0 2024-03-06T07:22:56.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.337+0000 7f0c67864640 1 -- 172.21.15.59:0/3118973537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0c600cf5b0 con 0x7f0c6015dfe0 2024-03-06T07:22:56.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.337+0000 7f0c5dffb640 1 -- 172.21.15.59:0/3118973537 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0c58058870 con 0x7f0c6015dfe0 2024-03-06T07:22:56.339 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:56.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 -- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4c06b500 msgr2=0x7f0c4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4c06b500 0x7f0c4c06d9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0c50036000 tx=0x7f0c5003b040 comp rx=0 tx=0).stop 2024-03-06T07:22:56.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 -- 172.21.15.59:0/3118973537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c6015dfe0 msgr2=0x7f0c601f69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:56.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c6015dfe0 0x7f0c601f69f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800c760 tx=0x7f0c5800cc20 comp rx=0 tx=0).stop 2024-03-06T07:22:56.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 -- 172.21.15.59:0/3118973537 shutdown_connections 2024-03-06T07:22:56.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4c06b500 0x7f0c4c06d9c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c6015dfe0 0x7f0c601f69f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c600d29c0 0x7f0c601f4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 --2- 172.21.15.59:0/3118973537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c600cf830 0x7f0c60154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:56.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 -- 172.21.15.59:0/3118973537 >> 172.21.15.59:0/3118973537 conn(0x7f0c600d3cd0 msgr2=0x7f0c600cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:56.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 -- 172.21.15.59:0/3118973537 shutdown_connections 2024-03-06T07:22:56.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:56.339+0000 7f0c67864640 1 -- 172.21.15.59:0/3118973537 wait complete. 2024-03-06T07:22:56.352 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:56.352 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:56.352 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:22:56.352 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:22:56.352 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:56.352 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:56.352 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:56.352 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:56.352 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:56.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:56.353 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:22:56.368 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:56.368 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:56.368 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:56.368 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:56.368 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:56.368 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:56.368 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:56.368 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:56.368 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:22:56.368 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:56.368 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:56.369 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:22:56.369 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:56.369 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:56.369 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:22:56.369 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:22:56.369 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42602sec preferred_lft 42602sec 2024-03-06T07:22:56.369 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:22:56.369 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:56.369 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:56.369 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:56.369 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:22:56.370 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:56.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:56.371 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:56.433 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:22:56.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:56.434 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:22:56.490 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:22:56.490 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:56.490 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:22:56.490 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:22:56.490 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:56.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:56.490 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:22:56.490 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:22:56.490 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:22:56.491 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:22:56.491 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:22:56.491 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:22:56.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:56.586 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:22:56.601 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:22:56.601 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:22:56.601 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:56.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:56.601 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:22:56.670 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:22:56.670 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:56.670 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:56.724 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:56.724 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:56.778 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:22:56.845 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:22:56.901 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:22:56.902 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:56.903 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:56.903 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:56.903 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:56.903 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:22:56.903 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:56.903 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:56.903 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:56.903 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:56.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:56.904 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:22:56.959 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:56.959 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:22:57.002 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:22:57.029 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:22:57.086 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:57.087 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:57.088 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:57.088 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:22:57.088 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:57.088 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:57.088 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:57.088 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:57.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:57.088 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:22:57.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:22:57.092+0000 7f0a1c43c480 -1 init, newargv = 0x557be3f0b660 newargc=15 2024-03-06T07:22:57.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[52707]: starting ceph client 2024-03-06T07:22:57.099 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[52707]: starting fuse 2024-03-06T07:22:57.109 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:22:57.109 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:22:57.109 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:57.109 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:22:57.109 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:22:57.109 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:22:57.109 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:22:57.109 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:57.109 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:22:57.110 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:22:57.111 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:57.111 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:57.111 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:22:57.111 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:22:57.230 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.52707.asok 2024-03-06T07:22:57.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:57.312 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52707.asok 2024-03-06T07:22:57.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:57.312 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52707.asok status 2024-03-06T07:22:57.449 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:22:57.449 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:22:57.449 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "52707", 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 5448, 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:22:57.450 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 5448 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 582935141 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 582935141 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.5448 192.168.0.1:0/582935141", 2024-03-06T07:22:57.451 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/582935141", 2024-03-06T07:22:57.452 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:22:57.452 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 13, 2024-03-06T07:22:57.452 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 33, 2024-03-06T07:22:57.452 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:57.452 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:22:57.452 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:22:57.452 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:22:57.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:57.458 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:22:57.458 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:57.473 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:22:57.474 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:57.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:57.474 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:57.542 DEBUG:teuthology.orchestra.run.smithi059:> stdin-killer -- python3 -c ' 2024-03-06T07:22:57.542 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:22:57.542 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:22:57.542 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:22:57.542 DEBUG:teuthology.orchestra.run.smithi059:> subdir = "delete_me" 2024-03-06T07:22:57.542 DEBUG:teuthology.orchestra.run.smithi059:> size = 1024 2024-03-06T07:22:57.543 DEBUG:teuthology.orchestra.run.smithi059:> file_count = 1000 2024-03-06T07:22:57.543 DEBUG:teuthology.orchestra.run.smithi059:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:22:57.543 DEBUG:teuthology.orchestra.run.smithi059:> for i in range(0, file_count): 2024-03-06T07:22:57.543 DEBUG:teuthology.orchestra.run.smithi059:> filename = "{0}_{1}.bin".format(i, size) 2024-03-06T07:22:57.543 DEBUG:teuthology.orchestra.run.smithi059:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:22:57.543 DEBUG:teuthology.orchestra.run.smithi059:> f.write(size * '"'"'x'"'"') 2024-03-06T07:22:57.543 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:22:59.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:59.480 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:59.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.621+0000 7f0856e5c640 1 Processor -- start 2024-03-06T07:22:59.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.621+0000 7f0856e5c640 1 -- start start 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.621+0000 7f0856e5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f085015dfe0 0x7f08500cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.621+0000 7f0856e5c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500b3430 con 0x7f08500d29c0 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.621+0000 7f0856e5c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500b3710 con 0x7f08500cf830 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.621+0000 7f0856e5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08500b31e0 con 0x7f085015dfe0 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.622+0000 7f08553f9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f08500d29c0 0x7f08500b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:48128/0 (socket says 172.21.15.59:48128) 2024-03-06T07:22:59.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.622+0000 7f08553f9640 1 -- 172.21.15.59:0/3742967003 learned_addr learned my addr 172.21.15.59:0/3742967003 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:22:59.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.622+0000 7f0854bf8640 1 --2- 172.21.15.59:0/3742967003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f085015dfe0 0x7f08500cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.622+0000 7f084f7fe640 1 -- 172.21.15.59:0/3742967003 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305504578 0 0) 0x7f08500b3430 con 0x7f08500d29c0 2024-03-06T07:22:59.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.622+0000 7f084f7fe640 1 -- 172.21.15.59:0/3742967003 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0838003520 con 0x7f08500d29c0 2024-03-06T07:22:59.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.622+0000 7f084f7fe640 1 -- 172.21.15.59:0/3742967003 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482611011 0 0) 0x7f08500b3710 con 0x7f08500cf830 2024-03-06T07:22:59.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.622+0000 7f084f7fe640 1 -- 172.21.15.59:0/3742967003 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08500b3430 con 0x7f08500cf830 2024-03-06T07:22:59.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.622+0000 7f084f7fe640 1 -- 172.21.15.59:0/3742967003 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2698747377 0 0) 0x7f0838003520 con 0x7f08500d29c0 2024-03-06T07:22:59.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.622+0000 7f0854bf8640 1 -- 172.21.15.59:0/3742967003 >> v1:172.21.15.164:6789/0 conn(0x7f08500cf830 legacy=0x7f08500d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.622+0000 7f0854bf8640 1 -- 172.21.15.59:0/3742967003 >> v1:172.21.15.59:6789/0 conn(0x7f08500d29c0 legacy=0x7f08500b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.622+0000 7f0854bf8640 1 -- 172.21.15.59:0/3742967003 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500b39f0 con 0x7f085015dfe0 2024-03-06T07:22:59.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.623+0000 7f0854bf8640 1 --2- 172.21.15.59:0/3742967003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f085015dfe0 0x7f08500cf2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f08400097d0 tx=0x7f084002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb6af579688bf5a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.623+0000 7f084f7fe640 1 -- 172.21.15.59:0/3742967003 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840037070 con 0x7f085015dfe0 2024-03-06T07:22:59.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.623+0000 7f084f7fe640 1 -- 172.21.15.59:0/3742967003 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0840030510 con 0x7f085015dfe0 2024-03-06T07:22:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.623+0000 7f084f7fe640 1 -- 172.21.15.59:0/3742967003 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08480019f0 con 0x7f085015dfe0 2024-03-06T07:22:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.624+0000 7f0856e5c640 1 -- 172.21.15.59:0/3742967003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f085015dfe0 msgr2=0x7f08500cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.624+0000 7f0856e5c640 1 --2- 172.21.15.59:0/3742967003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f085015dfe0 0x7f08500cf2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f08400097d0 tx=0x7f084002b800 comp rx=0 tx=0).stop 2024-03-06T07:22:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.624+0000 7f0856e5c640 1 -- 172.21.15.59:0/3742967003 shutdown_connections 2024-03-06T07:22:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.624+0000 7f0856e5c640 1 --2- 172.21.15.59:0/3742967003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f085015dfe0 0x7f08500cf2f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.624+0000 7f0856e5c640 1 -- 172.21.15.59:0/3742967003 >> 172.21.15.59:0/3742967003 conn(0x7f08500d3cd0 msgr2=0x7f08500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.624+0000 7f0856e5c640 1 -- 172.21.15.59:0/3742967003 shutdown_connections 2024-03-06T07:22:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.624+0000 7f0856e5c640 1 -- 172.21.15.59:0/3742967003 wait complete. 2024-03-06T07:22:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.625+0000 7f0856e5c640 1 Processor -- start 2024-03-06T07:22:59.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.625+0000 7f0856e5c640 1 -- start start 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0856e5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08500cf830 0x7f08500cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0856e5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08500d29c0 0x7f085015a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0856e5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f085015dfe0 0x7f08500b7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0856e5c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085015e3c0 con 0x7f08500cf830 2024-03-06T07:22:59.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0856e5c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500b3d60 con 0x7f08500d29c0 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0856e5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085015e540 con 0x7f085015dfe0 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0854bf8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08500cf830 0x7f08500cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0854bf8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08500cf830 0x7f08500cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57914/0 (socket says 172.21.15.59:57914) 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0854bf8640 1 -- 172.21.15.59:0/2051994410 learned_addr learned my addr 172.21.15.59:0/2051994410 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f08553f9640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f085015dfe0 0x7f08500b7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f084ffff640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08500d29c0 0x7f085015a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0854bf8640 1 -- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f085015dfe0 msgr2=0x7f08500b7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0854bf8640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f085015dfe0 0x7f08500b7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0854bf8640 1 -- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08500d29c0 msgr2=0x7f085015a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0854bf8640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08500d29c0 0x7f085015a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0854bf8640 1 -- 172.21.15.59:0/2051994410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500cb2e0 con 0x7f08500cf830 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.626+0000 7f0854bf8640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08500cf830 0x7f08500cada0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0840000c00 tx=0x7f08400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.627+0000 7f084d7fa640 1 -- 172.21.15.59:0/2051994410 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840042070 con 0x7f08500cf830 2024-03-06T07:22:59.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.627+0000 7f084d7fa640 1 -- 172.21.15.59:0/2051994410 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084003e870 con 0x7f08500cf830 2024-03-06T07:22:59.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.627+0000 7f084d7fa640 1 -- 172.21.15.59:0/2051994410 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840037040 con 0x7f08500cf830 2024-03-06T07:22:59.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.627+0000 7f0856e5c640 1 -- 172.21.15.59:0/2051994410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085015bf70 con 0x7f08500cf830 2024-03-06T07:22:59.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.627+0000 7f0856e5c640 1 -- 172.21.15.59:0/2051994410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08500b6a00 con 0x7f08500cf830 2024-03-06T07:22:59.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.628+0000 7f0856e5c640 1 -- 172.21.15.59:0/2051994410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08500d1a90 con 0x7f08500cf830 2024-03-06T07:22:59.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.628+0000 7f084d7fa640 1 -- 172.21.15.59:0/2051994410 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0840045070 con 0x7f08500cf830 2024-03-06T07:22:59.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.629+0000 7f084d7fa640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f083c06b4d0 0x7f083c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.629+0000 7f084d7fa640 1 -- 172.21.15.59:0/2051994410 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08400abf40 con 0x7f08500cf830 2024-03-06T07:22:59.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.629+0000 7f084ffff640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f083c06b4d0 0x7f083c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.630+0000 7f084ffff640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f083c06b4d0 0x7f083c06d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f08500cdec0 tx=0x7f0844009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.637+0000 7f084d7fa640 1 -- 172.21.15.59:0/2051994410 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08500d1a90 con 0x7f08500cf830 2024-03-06T07:22:59.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.855+0000 7f0856e5c640 1 -- 172.21.15.59:0/2051994410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f085004f330 con 0x7f08500cf830 2024-03-06T07:22:59.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.855+0000 7f084d7fa640 1 -- 172.21.15.59:0/2051994410 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f084007a8a0 con 0x7f08500cf830 2024-03-06T07:22:59.857 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:22:59.858 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:22:59.858 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:22:59.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 -- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f083c06b4d0 msgr2=0x7f083c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f083c06b4d0 0x7f083c06d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f08500cdec0 tx=0x7f0844009040 comp rx=0 tx=0).stop 2024-03-06T07:22:59.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 -- 172.21.15.59:0/2051994410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08500cf830 msgr2=0x7f08500cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08500cf830 0x7f08500cada0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0840000c00 tx=0x7f08400046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 -- 172.21.15.59:0/2051994410 shutdown_connections 2024-03-06T07:22:59.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f083c06b4d0 0x7f083c06d990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f085015dfe0 0x7f08500b7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08500d29c0 0x7f085015a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 --2- 172.21.15.59:0/2051994410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08500cf830 0x7f08500cada0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 -- 172.21.15.59:0/2051994410 >> 172.21.15.59:0/2051994410 conn(0x7f08500d3cd0 msgr2=0x7f08500b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 -- 172.21.15.59:0/2051994410 shutdown_connections 2024-03-06T07:22:59.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:22:59.857+0000 7f0856e5c640 1 -- 172.21.15.59:0/2051994410 wait complete. 2024-03-06T07:22:59.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:23:00.237 INFO:teuthology.orchestra.run.smithi059.stdout:{"message":"","return_code":0} 2024-03-06T07:23:00.237 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:23:00.238 DEBUG:teuthology.orchestra.run.smithi059:> stdin-killer -- python3 -c ' 2024-03-06T07:23:00.238 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:23:00.238 DEBUG:teuthology.orchestra.run.smithi059:> import stat 2024-03-06T07:23:00.238 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:23:00.238 DEBUG:teuthology.orchestra.run.smithi059:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-03-06T07:23:00.238 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:23:00.303 INFO:teuthology.orchestra.run.smithi059.stdout:1099511627776 2024-03-06T07:23:00.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:00.312 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:23:00.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.392+0000 7efd0ea0cf40 1 Processor -- start 2024-03-06T07:23:00.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.392+0000 7efd0ea0cf40 1 -- start start 2024-03-06T07:23:00.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0ea0cf40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56025ee83010 0x56025eecb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0ea0cf40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56025eeca9e0 0x56025eec98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0ea0cf40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 0x56025eeafeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0ea0cf40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56025ee28b70 con 0x56025ef56390 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0ea0cf40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56025ee2cad0 con 0x56025ee83010 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0ea0cf40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x56025eeab210 con 0x56025eeca9e0 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0c23b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 0x56025eeafeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0ba3a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56025ee83010 0x56025eecb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0c23b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 0x56025eeafeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57936/0 (socket says 172.21.15.59:57936) 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0c23b640 1 -- 172.21.15.59:0/3820344710 learned_addr learned my addr 172.21.15.59:0/3820344710 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd03fff640 1 --2- 172.21.15.59:0/3820344710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56025eeca9e0 0x56025eec98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0c23b640 1 -- 172.21.15.59:0/3820344710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56025eeca9e0 msgr2=0x56025eec98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0c23b640 1 --2- 172.21.15.59:0/3820344710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56025eeca9e0 0x56025eec98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0c23b640 1 -- 172.21.15.59:0/3820344710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56025ee83010 msgr2=0x56025eecb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:00.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0c23b640 1 --2- 172.21.15.59:0/3820344710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56025ee83010 0x56025eecb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0c23b640 1 -- 172.21.15.59:0/3820344710 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56025eeab440 con 0x56025ef56390 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.393+0000 7efd0c23b640 1 --2- 172.21.15.59:0/3820344710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 0x56025eeafeb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efd04057040 tx=0x7efd0404f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d00f9eee233d17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0aa38640 1 -- 172.21.15.59:0/3820344710 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04066070 con 0x56025ef56390 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0aa38640 1 -- 172.21.15.59:0/3820344710 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0404dd40 con 0x56025ef56390 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0aa38640 1 -- 172.21.15.59:0/3820344710 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04061690 con 0x56025ef56390 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/3820344710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 msgr2=0x56025eeafeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0ea0cf40 1 --2- 172.21.15.59:0/3820344710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 0x56025eeafeb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efd04057040 tx=0x7efd0404f000 comp rx=0 tx=0).stop 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/3820344710 shutdown_connections 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0ea0cf40 1 --2- 172.21.15.59:0/3820344710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 0x56025eeafeb0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0ea0cf40 1 --2- 172.21.15.59:0/3820344710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56025eeca9e0 0x56025eec98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0ea0cf40 1 --2- 172.21.15.59:0/3820344710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56025ee83010 0x56025eecb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/3820344710 >> 172.21.15.59:0/3820344710 conn(0x56025eead350 msgr2=0x56025ee6c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/3820344710 shutdown_connections 2024-03-06T07:23:00.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.394+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/3820344710 wait complete. 2024-03-06T07:23:00.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0ea0cf40 1 Processor -- start 2024-03-06T07:23:00.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0ea0cf40 1 -- start start 2024-03-06T07:23:00.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0ea0cf40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56025ee83010 0x56025ef56d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0ea0cf40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56025eeca9e0 0x56025eec9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0ea0cf40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 0x56025ef55330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0ea0cf40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56025ef59200 con 0x56025ef56390 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0ea0cf40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56025ef38f20 con 0x56025ee83010 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0ea0cf40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x56025ef332e0 con 0x56025eeca9e0 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0c23b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 0x56025ef55330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0ba3a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56025ee83010 0x56025ef56d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0c23b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 0x56025ef55330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57942/0 (socket says 172.21.15.59:57942) 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd0c23b640 1 -- 172.21.15.59:0/2542582499 learned_addr learned my addr 172.21.15.59:0/2542582499 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.395+0000 7efd03fff640 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56025eeca9e0 0x56025eec9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.396+0000 7efd0c23b640 1 -- 172.21.15.59:0/2542582499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56025eeca9e0 msgr2=0x56025eec9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.396+0000 7efd0c23b640 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56025eeca9e0 0x56025eec9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.396+0000 7efd0c23b640 1 -- 172.21.15.59:0/2542582499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56025ee83010 msgr2=0x56025ef56d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.396+0000 7efd0c23b640 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56025ee83010 0x56025ef56d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.396+0000 7efd0c23b640 1 -- 172.21.15.59:0/2542582499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56025ef35610 con 0x56025ef56390 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.396+0000 7efd0c23b640 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 0x56025ef55330 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7efd0401e2d0 tx=0x7efd04003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.396+0000 7efd09a36640 1 -- 172.21.15.59:0/2542582499 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04066040 con 0x56025ef56390 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.396+0000 7efd09a36640 1 -- 172.21.15.59:0/2542582499 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd04061a40 con 0x56025ef56390 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.396+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/2542582499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56025eeb0100 con 0x56025ef56390 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.396+0000 7efd09a36640 1 -- 172.21.15.59:0/2542582499 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04061df0 con 0x56025ef56390 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.396+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/2542582499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56025eeb06c0 con 0x56025ef56390 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.398+0000 7efd09a36640 1 -- 172.21.15.59:0/2542582499 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd0406c440 con 0x56025ef56390 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.398+0000 7efd09a36640 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efcec069eb0 0x7efcec06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.398+0000 7efd09a36640 1 -- 172.21.15.59:0/2542582499 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd040ce3b0 con 0x56025ef56390 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.398+0000 7efd0ea0cf40 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x56025eeca220 0x56025ef33680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.398+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/2542582499 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x56025ef33bc0 con 0x56025eeca220 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.398+0000 7efd0ba3a640 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efcec069eb0 0x7efcec06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.398+0000 7efd03fff640 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x56025eeca220 0x56025ef33680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd03fff640 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x56025eeca220 0x56025ef33680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0ba3a640 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efcec069eb0 0x7efcec06c370 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efcfc001ed0 tx=0x7efcfc001b40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd03fff640 1 -- 172.21.15.59:0/2542582499 <== osd.3 v2:172.21.15.59:6808/3602142643 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7efcf4001a30 con 0x56025eeca220 2024-03-06T07:23:00.404 INFO:teuthology.orchestra.run.smithi059.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-06T07:23:00.000000+0000, size 0 2024-03-06T07:23:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/2542582499 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x56025eeca220 msgr2=0x56025ef33680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0ea0cf40 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x56025eeca220 0x56025ef33680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/2542582499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efcec069eb0 msgr2=0x7efcec06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0ea0cf40 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efcec069eb0 0x7efcec06c370 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efcfc001ed0 tx=0x7efcfc001b40 comp rx=0 tx=0).stop 2024-03-06T07:23:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/2542582499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 msgr2=0x56025ef55330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0ea0cf40 1 --2- 172.21.15.59:0/2542582499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56025ef56390 0x56025ef55330 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7efd0401e2d0 tx=0x7efd04003330 comp rx=0 tx=0).stop 2024-03-06T07:23:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0c23b640 1 -- 172.21.15.59:0/2542582499 reap_dead start 2024-03-06T07:23:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/2542582499 shutdown_connections 2024-03-06T07:23:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/2542582499 >> 172.21.15.59:0/2542582499 conn(0x56025eead350 msgr2=0x56025ee6c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/2542582499 shutdown_connections 2024-03-06T07:23:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:00.399+0000 7efd0ea0cf40 1 -- 172.21.15.59:0/2542582499 wait complete. 2024-03-06T07:23:00.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:00.407 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:23:02.726 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:02.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.826+0000 7f957271e640 1 Processor -- start 2024-03-06T07:23:02.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.826+0000 7f957271e640 1 -- start start 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.826+0000 7f957271e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f956c15dfe0 0x7f956c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.826+0000 7f957271e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0d29c0 0x7f956c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.826+0000 7f957271e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956c0b3430 con 0x7f956c0cf830 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.826+0000 7f957271e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956c0b31e0 con 0x7f956c15dfe0 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.826+0000 7f957271e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956c0b3d60 con 0x7f956c0d29c0 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.827+0000 7f956bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f956c15dfe0 0x7f956c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.827+0000 7f9570cbb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0d29c0 0x7f956c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.827+0000 7f956b7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f956c0cf830 0x7f956c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:35822/0 (socket says 172.21.15.59:35822) 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.827+0000 7f956b7fe640 1 -- 172.21.15.59:0/2435663794 learned_addr learned my addr 172.21.15.59:0/2435663794 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.827+0000 7f956affd640 1 -- 172.21.15.59:0/2435663794 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118933212 0 0) 0x7f956c0b3430 con 0x7f956c0cf830 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.827+0000 7f956affd640 1 -- 172.21.15.59:0/2435663794 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9554003520 con 0x7f956c0cf830 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.827+0000 7f956bfff640 1 -- 172.21.15.59:0/2435663794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0d29c0 msgr2=0x7f956c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:02.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.827+0000 7f956bfff640 1 --2- 172.21.15.59:0/2435663794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0d29c0 0x7f956c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.827+0000 7f956bfff640 1 -- 172.21.15.59:0/2435663794 >> v1:172.21.15.164:6789/0 conn(0x7f956c0cf830 legacy=0x7f956c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.827+0000 7f956bfff640 1 -- 172.21.15.59:0/2435663794 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c0b3710 con 0x7f956c15dfe0 2024-03-06T07:23:02.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.827+0000 7f9570cbb640 1 --2- 172.21.15.59:0/2435663794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0d29c0 0x7f956c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:02.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f956bfff640 1 --2- 172.21.15.59:0/2435663794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956c15dfe0 0x7f956c0cf2f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f955c0097a0 tx=0x7f955c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32b30d74c7a57a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f956affd640 1 -- 172.21.15.59:0/2435663794 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c035020 con 0x7f956c15dfe0 2024-03-06T07:23:02.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f956affd640 1 -- 172.21.15.59:0/2435663794 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955c031830 con 0x7f956c15dfe0 2024-03-06T07:23:02.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f956affd640 1 -- 172.21.15.59:0/2435663794 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c031be0 con 0x7f956c15dfe0 2024-03-06T07:23:02.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f957271e640 1 -- 172.21.15.59:0/2435663794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956c15dfe0 msgr2=0x7f956c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f957271e640 1 --2- 172.21.15.59:0/2435663794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956c15dfe0 0x7f956c0cf2f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f955c0097a0 tx=0x7f955c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f957271e640 1 -- 172.21.15.59:0/2435663794 shutdown_connections 2024-03-06T07:23:02.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f957271e640 1 --2- 172.21.15.59:0/2435663794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0d29c0 0x7f956c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f957271e640 1 --2- 172.21.15.59:0/2435663794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956c15dfe0 0x7f956c0cf2f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f957271e640 1 -- 172.21.15.59:0/2435663794 >> 172.21.15.59:0/2435663794 conn(0x7f956c0d3cd0 msgr2=0x7f956c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f957271e640 1 -- 172.21.15.59:0/2435663794 shutdown_connections 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.829+0000 7f957271e640 1 -- 172.21.15.59:0/2435663794 wait complete. 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.830+0000 7f957271e640 1 Processor -- start 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.831+0000 7f957271e640 1 -- start start 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.831+0000 7f957271e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0cf830 0x7f956c0b7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.831+0000 7f957271e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956c0d29c0 0x7f956c160300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.831+0000 7f957271e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c15dfe0 0x7f956c160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.831+0000 7f957271e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c0d1df0 con 0x7f956c15dfe0 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.831+0000 7f957271e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c0d1c70 con 0x7f956c0d29c0 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.831+0000 7f957271e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956c15e3c0 con 0x7f956c0cf830 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.831+0000 7f9570cbb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c15dfe0 0x7f956c160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.831+0000 7f9570cbb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c15dfe0 0x7f956c160840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57954/0 (socket says 172.21.15.59:57954) 2024-03-06T07:23:02.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.831+0000 7f9570cbb640 1 -- 172.21.15.59:0/575727100 learned_addr learned my addr 172.21.15.59:0/575727100 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f9570cbb640 1 -- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0cf830 msgr2=0x7f956c0b7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f956bfff640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0cf830 0x7f956c0b7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f956b7fe640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956c0d29c0 0x7f956c160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f9570cbb640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0cf830 0x7f956c0b7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f9570cbb640 1 -- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956c0d29c0 msgr2=0x7f956c160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f9570cbb640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956c0d29c0 0x7f956c160300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f9570cbb640 1 -- 172.21.15.59:0/575727100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c1f4920 con 0x7f956c15dfe0 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f956b7fe640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956c0d29c0 0x7f956c160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f956bfff640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0cf830 0x7f956c0b7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f9570cbb640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c15dfe0 0x7f956c160840 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f956400efc0 tx=0x7f9564009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f9568ff9640 1 -- 172.21.15.59:0/575727100 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9564018030 con 0x7f956c15dfe0 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f9568ff9640 1 -- 172.21.15.59:0/575727100 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956400cd80 con 0x7f956c15dfe0 2024-03-06T07:23:02.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f957271e640 1 -- 172.21.15.59:0/575727100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956c0cae40 con 0x7f956c15dfe0 2024-03-06T07:23:02.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f9568ff9640 1 -- 172.21.15.59:0/575727100 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9564003bb0 con 0x7f956c15dfe0 2024-03-06T07:23:02.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.832+0000 7f957271e640 1 -- 172.21.15.59:0/575727100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956c0cb400 con 0x7f956c15dfe0 2024-03-06T07:23:02.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.833+0000 7f957271e640 1 -- 172.21.15.59:0/575727100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956c04f330 con 0x7f956c15dfe0 2024-03-06T07:23:02.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.834+0000 7f9568ff9640 1 -- 172.21.15.59:0/575727100 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f956401b070 con 0x7f956c15dfe0 2024-03-06T07:23:02.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.835+0000 7f9568ff9640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f955806b4d0 0x7f955806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:02.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.835+0000 7f9568ff9640 1 -- 172.21.15.59:0/575727100 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f956408ab70 con 0x7f956c15dfe0 2024-03-06T07:23:02.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.835+0000 7f956bfff640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f955806b4d0 0x7f955806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:02.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.835+0000 7f956bfff640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f955806b4d0 0x7f955806d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f955c036000 tx=0x7f955c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:02.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:02.843+0000 7f9568ff9640 1 -- 172.21.15.59:0/575727100 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9564010040 con 0x7f956c15dfe0 2024-03-06T07:23:03.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.062+0000 7f957271e640 1 -- 172.21.15.59:0/575727100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f956c15d2a0 con 0x7f956c15dfe0 2024-03-06T07:23:03.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.063+0000 7f9568ff9640 1 -- 172.21.15.59:0/575727100 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f9564012340 con 0x7f956c15dfe0 2024-03-06T07:23:03.065 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:03.065 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:23:03.065 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:23:03.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 -- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f955806b4d0 msgr2=0x7f955806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f955806b4d0 0x7f955806d990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f955c036000 tx=0x7f955c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:03.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 -- 172.21.15.59:0/575727100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c15dfe0 msgr2=0x7f956c160840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c15dfe0 0x7f956c160840 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f956400efc0 tx=0x7f9564009f90 comp rx=0 tx=0).stop 2024-03-06T07:23:03.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 -- 172.21.15.59:0/575727100 shutdown_connections 2024-03-06T07:23:03.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f955806b4d0 0x7f955806d990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c15dfe0 0x7f956c160840 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f956c0d29c0 0x7f956c160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 --2- 172.21.15.59:0/575727100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f956c0cf830 0x7f956c0b7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 -- 172.21.15.59:0/575727100 >> 172.21.15.59:0/575727100 conn(0x7f956c0d3cd0 msgr2=0x7f956c06cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 -- 172.21.15.59:0/575727100 shutdown_connections 2024-03-06T07:23:03.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.065+0000 7f957271e640 1 -- 172.21.15.59:0/575727100 wait complete. 2024-03-06T07:23:03.078 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:23:03.353 INFO:teuthology.orchestra.run.smithi059.stdout:{"message":"","return_code":0} 2024-03-06T07:23:03.353 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:23:03.354 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:03.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.454+0000 7fad52dd6640 1 Processor -- start 2024-03-06T07:23:03.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.454+0000 7fad52dd6640 1 -- start start 2024-03-06T07:23:03.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad52dd6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fad4c15bb20 0x7fad4c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad52dd6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c0cfff0 0x7fad4c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad52dd6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4c161980 con 0x7fad4c0ce4b0 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad52dd6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4c161c60 con 0x7fad4c15bb20 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad52dd6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c161f40 con 0x7fad4c0cfff0 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad51373640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fad4c0ce4b0 0x7fad4c0ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:48138/0 (socket says 172.21.15.59:48138) 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad51373640 1 -- 172.21.15.59:0/2764871616 learned_addr learned my addr 172.21.15.59:0/2764871616 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad50b72640 1 --2- 172.21.15.59:0/2764871616 >> v2:172.21.15.164:3300/0 conn(0x7fad4c15bb20 0x7fad4c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573800733 0 0) 0x7fad4c161980 con 0x7fad4c0ce4b0 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4bfff640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c0cfff0 0x7fad4c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad3c003520 con 0x7fad4c0ce4b0 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2440378086 0 0) 0x7fad3c003520 con 0x7fad4c0ce4b0 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c0cfff0 msgr2=0x7fad4c0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4b7fe640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c0cfff0 0x7fad4c0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 >> v2:172.21.15.164:3300/0 conn(0x7fad4c15bb20 msgr2=0x7fad4c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4b7fe640 1 --2- 172.21.15.59:0/2764871616 >> v2:172.21.15.164:3300/0 conn(0x7fad4c15bb20 0x7fad4c0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c162220 con 0x7fad4c0ce4b0 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fad440019f0 con 0x7fad4c0ce4b0 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4bfff640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c0cfff0 0x7fad4c0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.455+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 >> v1:172.21.15.59:6789/0 conn(0x7fad4c0ce4b0 legacy=0x7fad4c0ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad4b7fe640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad3c0039b0 0x7fad3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad4b7fe640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c0cfff0 0x7fad3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad4b7fe640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c004d40 0x7fad3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c161c60 con 0x7fad3c004d40 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c161f40 con 0x7fad3c0039b0 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c00b9c0 con 0x7fad4c0cfff0 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad51373640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c004d40 0x7fad3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad50b72640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad3c0039b0 0x7fad3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad4bfff640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c0cfff0 0x7fad3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad51373640 1 -- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c0cfff0 msgr2=0x7fad3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad51373640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c0cfff0 0x7fad3c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad51373640 1 -- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad3c0039b0 msgr2=0x7fad3c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad51373640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad3c0039b0 0x7fad3c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.456+0000 7fad51373640 1 -- 172.21.15.59:0/2764871616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c003520 con 0x7fad3c004d40 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.457+0000 7fad51373640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c004d40 0x7fad3c0051c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fad4400a8a0 tx=0x7fad3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec790a7e4b92d28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.457+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4400a4c0 con 0x7fad3c004d40 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.457+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad44034630 con 0x7fad3c004d40 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.457+0000 7fad4b7fe640 1 -- 172.21.15.59:0/2764871616 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44034a60 con 0x7fad3c004d40 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.457+0000 7fad52dd6640 1 -- 172.21.15.59:0/2764871616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c004d40 msgr2=0x7fad3c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.457+0000 7fad52dd6640 1 --2- 172.21.15.59:0/2764871616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad3c004d40 0x7fad3c0051c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fad4400a8a0 tx=0x7fad3c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.457+0000 7fad51373640 1 -- 172.21.15.59:0/2764871616 reap_dead start 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.457+0000 7fad52dd6640 1 -- 172.21.15.59:0/2764871616 shutdown_connections 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.457+0000 7fad52dd6640 1 -- 172.21.15.59:0/2764871616 >> 172.21.15.59:0/2764871616 conn(0x7fad4c09a9f0 msgr2=0x7fad4c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.457+0000 7fad52dd6640 1 -- 172.21.15.59:0/2764871616 shutdown_connections 2024-03-06T07:23:03.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.457+0000 7fad52dd6640 1 -- 172.21.15.59:0/2764871616 wait complete. 2024-03-06T07:23:03.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.459+0000 7fad52dd6640 1 Processor -- start 2024-03-06T07:23:03.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.459+0000 7fad52dd6640 1 -- start start 2024-03-06T07:23:03.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.459+0000 7fad52dd6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c14f030 0x7fad4c160b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad52dd6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad4c0c55f0 0x7fad4c0c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad52dd6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad4c0c79a0 0x7fad4c159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad52dd6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c0d24b0 con 0x7fad4c0c79a0 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad52dd6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c0d2330 con 0x7fad4c0c55f0 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad52dd6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c0d1980 con 0x7fad4c14f030 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad51373640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad4c0c79a0 0x7fad4c159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad51373640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad4c0c79a0 0x7fad4c159970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57966/0 (socket says 172.21.15.59:57966) 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad51373640 1 -- 172.21.15.59:0/4083445220 learned_addr learned my addr 172.21.15.59:0/4083445220 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad50b72640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c14f030 0x7fad4c160b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad4bfff640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad4c0c55f0 0x7fad4c0c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad51373640 1 -- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c14f030 msgr2=0x7fad4c160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad51373640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c14f030 0x7fad4c160b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad51373640 1 -- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad4c0c55f0 msgr2=0x7fad4c0c7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad51373640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad4c0c55f0 0x7fad4c0c7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad51373640 1 -- 172.21.15.59:0/4083445220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c14f410 con 0x7fad4c0c79a0 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad51373640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad4c0c79a0 0x7fad4c159970 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fad4400a8a0 tx=0x7fad44001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad497fa640 1 -- 172.21.15.59:0/4083445220 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4400a4c0 con 0x7fad4c0c79a0 2024-03-06T07:23:03.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad497fa640 1 -- 172.21.15.59:0/4083445220 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad44045070 con 0x7fad4c0c79a0 2024-03-06T07:23:03.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad52dd6640 1 -- 172.21.15.59:0/4083445220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4c0c5e40 con 0x7fad4c0c79a0 2024-03-06T07:23:03.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad497fa640 1 -- 172.21.15.59:0/4083445220 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44002010 con 0x7fad4c0c79a0 2024-03-06T07:23:03.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.460+0000 7fad52dd6640 1 -- 172.21.15.59:0/4083445220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c0c6400 con 0x7fad4c0c79a0 2024-03-06T07:23:03.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.462+0000 7fad52dd6640 1 -- 172.21.15.59:0/4083445220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4c04f330 con 0x7fad4c0c79a0 2024-03-06T07:23:03.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.463+0000 7fad497fa640 1 -- 172.21.15.59:0/4083445220 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad44034bf0 con 0x7fad4c0c79a0 2024-03-06T07:23:03.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.463+0000 7fad497fa640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad3006b4d0 0x7fad3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.463+0000 7fad497fa640 1 -- 172.21.15.59:0/4083445220 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad440ac700 con 0x7fad4c0c79a0 2024-03-06T07:23:03.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.463+0000 7fad50b72640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad3006b4d0 0x7fad3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.464+0000 7fad50b72640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad3006b4d0 0x7fad3006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fad40009bc0 tx=0x7fad40009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.468+0000 7fad497fa640 1 -- 172.21.15.59:0/4083445220 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad4407b6d0 con 0x7fad4c0c79a0 2024-03-06T07:23:03.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.687+0000 7fad52dd6640 1 -- 172.21.15.59:0/4083445220 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad4c06fd90 con 0x7fad4c0c79a0 2024-03-06T07:23:03.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.687+0000 7fad497fa640 1 -- 172.21.15.59:0/4083445220 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fad4407afd0 con 0x7fad4c0c79a0 2024-03-06T07:23:03.690 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:03.690 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:23:03.691 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:23:03.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 -- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad3006b4d0 msgr2=0x7fad3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad3006b4d0 0x7fad3006d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fad40009bc0 tx=0x7fad40009190 comp rx=0 tx=0).stop 2024-03-06T07:23:03.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 -- 172.21.15.59:0/4083445220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad4c0c79a0 msgr2=0x7fad4c159970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad4c0c79a0 0x7fad4c159970 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fad4400a8a0 tx=0x7fad44001dc0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 -- 172.21.15.59:0/4083445220 shutdown_connections 2024-03-06T07:23:03.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad3006b4d0 0x7fad3006d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad4c0c79a0 0x7fad4c159970 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad4c0c55f0 0x7fad4c0c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 --2- 172.21.15.59:0/4083445220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad4c14f030 0x7fad4c160b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 -- 172.21.15.59:0/4083445220 >> 172.21.15.59:0/4083445220 conn(0x7fad4c09a9f0 msgr2=0x7fad4c06f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 -- 172.21.15.59:0/4083445220 shutdown_connections 2024-03-06T07:23:03.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.689+0000 7fad52dd6640 1 -- 172.21.15.59:0/4083445220 wait complete. 2024-03-06T07:23:03.703 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:23:03.822 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_created":1001}} 2024-03-06T07:23:03.822 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-03-06T07:23:03.822 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:03.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.922+0000 7f657c665640 1 Processor -- start 2024-03-06T07:23:03.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.922+0000 7f657c665640 1 -- start start 2024-03-06T07:23:03.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.923+0000 7f657c665640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f657415fa10 0x7f65740ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.923+0000 7f657c665640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65740d16e0 0x7f65740d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.923+0000 7f657c665640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65740cebe0 con 0x7f657415bdd0 2024-03-06T07:23:03.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.923+0000 7f657c665640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65740ceec0 con 0x7f657415fa10 2024-03-06T07:23:03.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.923+0000 7f657c665640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65740cf1a0 con 0x7f65740d16e0 2024-03-06T07:23:03.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.923+0000 7f6579c00640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f657415fa10 0x7f65740ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.923+0000 7f6579c00640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f657415fa10 0x7f65740ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:59470/0 (socket says 172.21.15.59:59470) 2024-03-06T07:23:03.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.923+0000 7f6579c00640 1 -- 172.21.15.59:0/1678207050 learned_addr learned my addr 172.21.15.59:0/1678207050 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:03.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.923+0000 7f657ac02640 1 --2- 172.21.15.59:0/1678207050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65740d16e0 0x7f65740d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.924+0000 7f6579c00640 1 -- 172.21.15.59:0/1678207050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65740d16e0 msgr2=0x7f65740d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:03.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.924+0000 7f6579c00640 1 --2- 172.21.15.59:0/1678207050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65740d16e0 0x7f65740d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.924+0000 7f6579c00640 1 -- 172.21.15.59:0/1678207050 >> v1:172.21.15.164:6789/0 conn(0x7f657415bdd0 legacy=0x7f65740c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.924+0000 7f6579c00640 1 -- 172.21.15.59:0/1678207050 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65740cc8e0 con 0x7f657415fa10 2024-03-06T07:23:03.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.924+0000 7f6579c00640 1 --2- 172.21.15.59:0/1678207050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f657415fa10 0x7f65740ce270 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6564009870 tx=0x7f656402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=722fc347f2be082f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.925+0000 7f65793ff640 1 -- 172.21.15.59:0/1678207050 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6564035020 con 0x7f657415fa10 2024-03-06T07:23:03.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.925+0000 7f65793ff640 1 -- 172.21.15.59:0/1678207050 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6564031640 con 0x7f657415fa10 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.925+0000 7f65793ff640 1 -- 172.21.15.59:0/1678207050 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656402d040 con 0x7f657415fa10 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.925+0000 7f657c665640 1 -- 172.21.15.59:0/1678207050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f657415fa10 msgr2=0x7f65740ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.925+0000 7f657c665640 1 --2- 172.21.15.59:0/1678207050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f657415fa10 0x7f65740ce270 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6564009870 tx=0x7f656402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.926+0000 7f657c665640 1 -- 172.21.15.59:0/1678207050 shutdown_connections 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.926+0000 7f657c665640 1 --2- 172.21.15.59:0/1678207050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65740d16e0 0x7f65740d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.926+0000 7f657c665640 1 --2- 172.21.15.59:0/1678207050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f657415fa10 0x7f65740ce270 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.926+0000 7f657c665640 1 -- 172.21.15.59:0/1678207050 >> 172.21.15.59:0/1678207050 conn(0x7f657409a9f0 msgr2=0x7f657409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.926+0000 7f657c665640 1 -- 172.21.15.59:0/1678207050 shutdown_connections 2024-03-06T07:23:03.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.926+0000 7f657c665640 1 -- 172.21.15.59:0/1678207050 wait complete. 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.927+0000 7f657c665640 1 Processor -- start 2024-03-06T07:23:03.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.928+0000 7f657c665640 1 -- start start 2024-03-06T07:23:03.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657c665640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65740d16e0 0x7f65740c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657c665640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f657415bdd0 0x7f6574145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657c665640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f657415fa10 0x7f65740c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657c665640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657415c1b0 con 0x7f65740d16e0 2024-03-06T07:23:03.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657c665640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65740cc5d0 con 0x7f657415fa10 2024-03-06T07:23:03.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657c665640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65740ce7b0 con 0x7f657415bdd0 2024-03-06T07:23:03.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657a401640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65740d16e0 0x7f65740c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657a401640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65740d16e0 0x7f65740c48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34846/0 (socket says 172.21.15.59:34846) 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657a401640 1 -- 172.21.15.59:0/2558929295 learned_addr learned my addr 172.21.15.59:0/2558929295 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f6579c00640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f657415bdd0 0x7f6574145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657ac02640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f657415fa10 0x7f65740c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657a401640 1 -- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f657415bdd0 msgr2=0x7f6574145a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657a401640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f657415bdd0 0x7f6574145a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657a401640 1 -- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f657415fa10 msgr2=0x7f65740c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657a401640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f657415fa10 0x7f65740c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657a401640 1 -- 172.21.15.59:0/2558929295 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65741593a0 con 0x7f65740d16e0 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f6579c00640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f657415bdd0 0x7f6574145a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657ac02640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f657415fa10 0x7f65740c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:03.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.929+0000 7f657a401640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65740d16e0 0x7f65740c48c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f657000c3f0 tx=0x7f657000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.930+0000 7f656affd640 1 -- 172.21.15.59:0/2558929295 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65700052d0 con 0x7f65740d16e0 2024-03-06T07:23:03.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.930+0000 7f656affd640 1 -- 172.21.15.59:0/2558929295 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6570005490 con 0x7f65740d16e0 2024-03-06T07:23:03.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.930+0000 7f657c665640 1 -- 172.21.15.59:0/2558929295 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65740c9770 con 0x7f65740d16e0 2024-03-06T07:23:03.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.930+0000 7f656affd640 1 -- 172.21.15.59:0/2558929295 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570007260 con 0x7f65740d16e0 2024-03-06T07:23:03.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.930+0000 7f657c665640 1 -- 172.21.15.59:0/2558929295 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65740ca960 con 0x7f65740d16e0 2024-03-06T07:23:03.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.931+0000 7f657c665640 1 -- 172.21.15.59:0/2558929295 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65740ca780 con 0x7f65740d16e0 2024-03-06T07:23:03.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.931+0000 7f656affd640 1 -- 172.21.15.59:0/2558929295 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6570007400 con 0x7f65740d16e0 2024-03-06T07:23:03.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.931+0000 7f656affd640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f655806b4d0 0x7f655806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:03.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.932+0000 7f656affd640 1 -- 172.21.15.59:0/2558929295 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6570089760 con 0x7f65740d16e0 2024-03-06T07:23:03.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.932+0000 7f6579c00640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f655806b4d0 0x7f655806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:03.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.932+0000 7f6579c00640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f655806b4d0 0x7f655806d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6564036000 tx=0x7f6564008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:03.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:03.936+0000 7f656affd640 1 -- 172.21.15.59:0/2558929295 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65740ca780 con 0x7f65740d16e0 2024-03-06T07:23:04.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.155+0000 7f657c665640 1 -- 172.21.15.59:0/2558929295 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f657404f330 con 0x7f65740d16e0 2024-03-06T07:23:04.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.156+0000 7f656affd640 1 -- 172.21.15.59:0/2558929295 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f65741593a0 con 0x7f65740d16e0 2024-03-06T07:23:04.158 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:04.158 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:23:04.158 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:23:04.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 -- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f655806b4d0 msgr2=0x7f655806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f655806b4d0 0x7f655806d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6564036000 tx=0x7f6564008030 comp rx=0 tx=0).stop 2024-03-06T07:23:04.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 -- 172.21.15.59:0/2558929295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65740d16e0 msgr2=0x7f65740c48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65740d16e0 0x7f65740c48c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f657000c3f0 tx=0x7f657000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 -- 172.21.15.59:0/2558929295 shutdown_connections 2024-03-06T07:23:04.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f655806b4d0 0x7f655806d990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f657415fa10 0x7f65740c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f657415bdd0 0x7f6574145a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 --2- 172.21.15.59:0/2558929295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65740d16e0 0x7f65740c48c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 -- 172.21.15.59:0/2558929295 >> 172.21.15.59:0/2558929295 conn(0x7f657409a9f0 msgr2=0x7f657415e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 -- 172.21.15.59:0/2558929295 shutdown_connections 2024-03-06T07:23:04.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:04.158+0000 7f657c665640 1 -- 172.21.15.59:0/2558929295 wait complete. 2024-03-06T07:23:04.171 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:23:04.293 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_enqueued":159}} 2024-03-06T07:23:04.293 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 159 } } 2024-03-06T07:23:04.294 DEBUG:tasks.ceph_test_case:wait_until_equal: 159 != 1001, waiting (timeout=600)... 2024-03-06T07:23:09.295 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:09.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.395+0000 7f2e8eb02640 1 Processor -- start 2024-03-06T07:23:09.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.395+0000 7f2e8eb02640 1 -- start start 2024-03-06T07:23:09.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.396+0000 7f2e8eb02640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2e8815ddf0 0x7f2e880cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.396+0000 7f2e8eb02640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e880d18d0 con 0x7f2e880d0de0 2024-03-06T07:23:09.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.396+0000 7f2e8eb02640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e880d1bb0 con 0x7f2e8815e1d0 2024-03-06T07:23:09.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.396+0000 7f2e8eb02640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e880d27c0 con 0x7f2e8815ddf0 2024-03-06T07:23:09.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.396+0000 7f2e8d09f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2e880d0de0 0x7f2e880b6b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36816/0 (socket says 172.21.15.59:36816) 2024-03-06T07:23:09.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.396+0000 7f2e8d09f640 1 -- 172.21.15.59:0/795533729 learned_addr learned my addr 172.21.15.59:0/795533729 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:09.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e8c89e640 1 --2- 172.21.15.59:0/795533729 >> v2:172.21.15.164:3300/0 conn(0x7f2e8815ddf0 0x7f2e880cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564937680 0 0) 0x7f2e880d18d0 con 0x7f2e880d0de0 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e78003520 con 0x7f2e880d0de0 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 246940921 0 0) 0x7f2e78003520 con 0x7f2e880d0de0 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 >> v2:172.21.15.164:3300/0 conn(0x7f2e8815ddf0 msgr2=0x7f2e880cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 --2- 172.21.15.59:0/795533729 >> v2:172.21.15.164:3300/0 conn(0x7f2e8815ddf0 0x7f2e880cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 >> v1:172.21.15.164:6789/0 conn(0x7f2e8815e1d0 legacy=0x7f2e880d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e880d1e90 con 0x7f2e880d0de0 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f2e800019f0 con 0x7f2e880d0de0 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 >> v1:172.21.15.59:6789/0 conn(0x7f2e880d0de0 legacy=0x7f2e880b6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 --2- 172.21.15.59:0/795533729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e780039b0 0x7f2e78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 --2- 172.21.15.59:0/795533729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e78004800 0x7f2e78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 --2- 172.21.15.59:0/795533729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e780051c0 0x7f2e7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e880d27c0 con 0x7f2e780051c0 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7800c060 con 0x7f2e780039b0 2024-03-06T07:23:09.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7800c260 con 0x7f2e78004800 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e8d09f640 1 --2- 172.21.15.59:0/795533729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e780051c0 0x7f2e7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e8c89e640 1 --2- 172.21.15.59:0/795533729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e780039b0 0x7f2e78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e87fff640 1 --2- 172.21.15.59:0/795533729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e78004800 0x7f2e78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e8d09f640 1 -- 172.21.15.59:0/795533729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e78004800 msgr2=0x7f2e78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e8d09f640 1 --2- 172.21.15.59:0/795533729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e78004800 0x7f2e78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e8d09f640 1 -- 172.21.15.59:0/795533729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e780039b0 msgr2=0x7f2e78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e8d09f640 1 --2- 172.21.15.59:0/795533729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e780039b0 0x7f2e78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.397+0000 7f2e8d09f640 1 -- 172.21.15.59:0/795533729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e78003520 con 0x7f2e780051c0 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.398+0000 7f2e8d09f640 1 -- 172.21.15.59:0/795533729 reap_dead start 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.398+0000 7f2e8d09f640 1 --2- 172.21.15.59:0/795533729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e780051c0 0x7f2e7800b9c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000a840 tx=0x7f2e78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d785d989cb05ed74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.398+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8000a460 con 0x7f2e780051c0 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.398+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e80034650 con 0x7f2e780051c0 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.398+0000 7f2e877fe640 1 -- 172.21.15.59:0/795533729 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80034a40 con 0x7f2e780051c0 2024-03-06T07:23:09.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.399+0000 7f2e8eb02640 1 -- 172.21.15.59:0/795533729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e780051c0 msgr2=0x7f2e7800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.399+0000 7f2e8eb02640 1 --2- 172.21.15.59:0/795533729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e780051c0 0x7f2e7800b9c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000a840 tx=0x7f2e78005830 comp rx=0 tx=0).stop 2024-03-06T07:23:09.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.399+0000 7f2e8eb02640 1 -- 172.21.15.59:0/795533729 shutdown_connections 2024-03-06T07:23:09.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.399+0000 7f2e8eb02640 1 --2- 172.21.15.59:0/795533729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e780051c0 0x7f2e7800b9c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.399+0000 7f2e8eb02640 1 -- 172.21.15.59:0/795533729 >> 172.21.15.59:0/795533729 conn(0x7f2e8809cff0 msgr2=0x7f2e880d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.399+0000 7f2e8eb02640 1 -- 172.21.15.59:0/795533729 shutdown_connections 2024-03-06T07:23:09.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.399+0000 7f2e8eb02640 1 -- 172.21.15.59:0/795533729 wait complete. 2024-03-06T07:23:09.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.400+0000 7f2e8eb02640 1 Processor -- start 2024-03-06T07:23:09.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.400+0000 7f2e8eb02640 1 -- start start 2024-03-06T07:23:09.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e8eb02640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e780051c0 0x7f2e88155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e8eb02640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e880c9670 0x7f2e880c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e8eb02640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e880c86f0 0x7f2e880c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e8eb02640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e880d2c50 con 0x7f2e880c9670 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e8eb02640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e880d2450 con 0x7f2e880c86f0 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e8eb02640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e880d06d0 con 0x7f2e780051c0 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e87fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e880c9670 0x7f2e880c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e87fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e880c9670 0x7f2e880c9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34866/0 (socket says 172.21.15.59:34866) 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e87fff640 1 -- 172.21.15.59:0/1148397065 learned_addr learned my addr 172.21.15.59:0/1148397065 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e87fff640 1 -- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e780051c0 msgr2=0x7f2e88155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e8c89e640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e780051c0 0x7f2e88155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e8d09f640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e880c86f0 0x7f2e880c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e87fff640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e780051c0 0x7f2e88155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e87fff640 1 -- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e880c86f0 msgr2=0x7f2e880c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e87fff640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e880c86f0 0x7f2e880c8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.401+0000 7f2e87fff640 1 -- 172.21.15.59:0/1148397065 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e880cb020 con 0x7f2e880c9670 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.402+0000 7f2e87fff640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e880c9670 0x7f2e880c9ad0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400f620 tx=0x7f2e7400fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.402+0000 7f2e857fa640 1 -- 172.21.15.59:0/1148397065 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e740052e0 con 0x7f2e880c9670 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.402+0000 7f2e857fa640 1 -- 172.21.15.59:0/1148397065 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e74018070 con 0x7f2e880c9670 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.402+0000 7f2e8eb02640 1 -- 172.21.15.59:0/1148397065 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e880cb300 con 0x7f2e880c9670 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.402+0000 7f2e857fa640 1 -- 172.21.15.59:0/1148397065 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e74003810 con 0x7f2e880c9670 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.402+0000 7f2e8eb02640 1 -- 172.21.15.59:0/1148397065 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e880cb8c0 con 0x7f2e880c9670 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.403+0000 7f2e8eb02640 1 -- 172.21.15.59:0/1148397065 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e880cadb0 con 0x7f2e880c9670 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.403+0000 7f2e857fa640 1 -- 172.21.15.59:0/1148397065 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e740039b0 con 0x7f2e880c9670 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.404+0000 7f2e857fa640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.404+0000 7f2e857fa640 1 -- 172.21.15.59:0/1148397065 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e74089fb0 con 0x7f2e880c9670 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.404+0000 7f2e8c89e640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.405+0000 7f2e8c89e640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009980 tx=0x7f2e7c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.409+0000 7f2e857fa640 1 -- 172.21.15.59:0/1148397065 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e74058e30 con 0x7f2e880c9670 2024-03-06T07:23:09.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.647+0000 7f2e8eb02640 1 -- 172.21.15.59:0/1148397065 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e8804f330 con 0x7f2e880c9670 2024-03-06T07:23:09.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.647+0000 7f2e857fa640 1 -- 172.21.15.59:0/1148397065 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f2e74058730 con 0x7f2e880c9670 2024-03-06T07:23:09.649 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:09.650 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:23:09.650 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 -- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e6c06b4d0 msgr2=0x7f2e6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009980 tx=0x7f2e7c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 -- 172.21.15.59:0/1148397065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e880c9670 msgr2=0x7f2e880c9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e880c9670 0x7f2e880c9ad0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400f620 tx=0x7f2e7400fae0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 -- 172.21.15.59:0/1148397065 shutdown_connections 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e880c86f0 0x7f2e880c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e880c9670 0x7f2e880c9ad0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 --2- 172.21.15.59:0/1148397065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e780051c0 0x7f2e88155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 -- 172.21.15.59:0/1148397065 >> 172.21.15.59:0/1148397065 conn(0x7f2e8809cff0 msgr2=0x7f2e8815c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 -- 172.21.15.59:0/1148397065 shutdown_connections 2024-03-06T07:23:09.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.650+0000 7f2e8eb02640 1 -- 172.21.15.59:0/1148397065 wait complete. 2024-03-06T07:23:09.663 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:23:09.779 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-03-06T07:23:09.780 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-03-06T07:23:09.780 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.882+0000 7f56da44e640 1 Processor -- start 2024-03-06T07:23:09.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.882+0000 7f56da44e640 1 -- start start 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.883+0000 7f56da44e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f56d40d04a0 0x7f56d40ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.883+0000 7f56da44e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d4161980 con 0x7f56d415bb20 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.883+0000 7f56da44e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d4161c60 con 0x7f56d40cfff0 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.883+0000 7f56da44e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d40d2500 con 0x7f56d40d04a0 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.883+0000 7f56d3fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f56d415bb20 0x7f56d40d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36822/0 (socket says 172.21.15.59:36822) 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.883+0000 7f56d3fff640 1 -- 172.21.15.59:0/1883102641 learned_addr learned my addr 172.21.15.59:0/1883102641 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d89eb640 1 --2- 172.21.15.59:0/1883102641 >> v2:172.21.15.164:3300/0 conn(0x7f56d40d04a0 0x7f56d40ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121380133 0 0) 0x7f56d4161980 con 0x7f56d415bb20 2024-03-06T07:23:09.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56c0003520 con 0x7f56d415bb20 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3216465447 0 0) 0x7f56c0003520 con 0x7f56d415bb20 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 >> v2:172.21.15.164:3300/0 conn(0x7f56d40d04a0 msgr2=0x7f56d40ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 --2- 172.21.15.59:0/1883102641 >> v2:172.21.15.164:3300/0 conn(0x7f56d40d04a0 0x7f56d40ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 >> v1:172.21.15.164:6789/0 conn(0x7f56d40cfff0 legacy=0x7f56d40cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d4161f40 con 0x7f56d415bb20 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f56c4001670 con 0x7f56d415bb20 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 >> v1:172.21.15.59:6789/0 conn(0x7f56d415bb20 legacy=0x7f56d40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56c0003a00 0x7f56c0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56c0004800 0x7f56c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56c00051c0 0x7f56c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d40d2500 con 0x7f56c0004800 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c000c0c0 con 0x7f56c00051c0 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c000c2c0 con 0x7f56c0003a00 2024-03-06T07:23:09.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d37fe640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56c0004800 0x7f56c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d89eb640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56c00051c0 0x7f56c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d3fff640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56c0003a00 0x7f56c0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d37fe640 1 -- 172.21.15.59:0/1883102641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56c0003a00 msgr2=0x7f56c0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d37fe640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56c0003a00 0x7f56c0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d37fe640 1 -- 172.21.15.59:0/1883102641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56c00051c0 msgr2=0x7f56c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d37fe640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56c00051c0 0x7f56c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d37fe640 1 -- 172.21.15.59:0/1883102641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c0003520 con 0x7f56c0004800 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d89eb640 1 -- 172.21.15.59:0/1883102641 reap_dead start 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.884+0000 7f56d3fff640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56c0003a00 0x7f56c0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.885+0000 7f56d37fe640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56c0004800 0x7f56c0004c80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f56bc001750 tx=0x7f56c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e658f36c1a6f8eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.885+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc001370 con 0x7f56c0004800 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.885+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc0052d0 con 0x7f56c0004800 2024-03-06T07:23:09.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.885+0000 7f56d2ffd640 1 -- 172.21.15.59:0/1883102641 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc003830 con 0x7f56c0004800 2024-03-06T07:23:09.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.886+0000 7f56da44e640 1 -- 172.21.15.59:0/1883102641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56c0004800 msgr2=0x7f56c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.886+0000 7f56da44e640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56c0004800 0x7f56c0004c80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f56bc001750 tx=0x7f56c0005830 comp rx=0 tx=0).stop 2024-03-06T07:23:09.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.886+0000 7f56da44e640 1 -- 172.21.15.59:0/1883102641 shutdown_connections 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.886+0000 7f56da44e640 1 --2- 172.21.15.59:0/1883102641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56c0004800 0x7f56c0004c80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.886+0000 7f56da44e640 1 -- 172.21.15.59:0/1883102641 >> 172.21.15.59:0/1883102641 conn(0x7f56d409a9f0 msgr2=0x7f56d409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.886+0000 7f56da44e640 1 -- 172.21.15.59:0/1883102641 shutdown_connections 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.886+0000 7f56da44e640 1 -- 172.21.15.59:0/1883102641 wait complete. 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.887+0000 7f56da44e640 1 Processor -- start 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.887+0000 7f56da44e640 1 -- start start 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56da44e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56c0004800 0x7f56d4156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56da44e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56d406dee0 0x7f56d40b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56da44e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d40b2780 0x7f56d40c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56da44e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d40d08d0 con 0x7f56d40b2780 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56da44e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d4162500 con 0x7f56d406dee0 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56da44e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d40d1980 con 0x7f56c0004800 2024-03-06T07:23:09.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d89eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d40b2780 0x7f56d40c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d89eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d40b2780 0x7f56d40c78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34876/0 (socket says 172.21.15.59:34876) 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d89eb640 1 -- 172.21.15.59:0/1052021608 learned_addr learned my addr 172.21.15.59:0/1052021608 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d3fff640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56c0004800 0x7f56d4156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d37fe640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56d406dee0 0x7f56d40b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d89eb640 1 -- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56c0004800 msgr2=0x7f56d4156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d89eb640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56c0004800 0x7f56d4156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d89eb640 1 -- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56d406dee0 msgr2=0x7f56d40b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d89eb640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56d406dee0 0x7f56d40b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d89eb640 1 -- 172.21.15.59:0/1052021608 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d406cea0 con 0x7f56d40b2780 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d3fff640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56c0004800 0x7f56d4156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.888+0000 7f56d37fe640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56d406dee0 0x7f56d40b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.889+0000 7f56d89eb640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d40b2780 0x7f56d40c78c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f56d40713f0 tx=0x7f56cc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.889+0000 7f56d0ff9640 1 -- 172.21.15.59:0/1052021608 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc00fd80 con 0x7f56d40b2780 2024-03-06T07:23:09.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.889+0000 7f56d0ff9640 1 -- 172.21.15.59:0/1052021608 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc015430 con 0x7f56d40b2780 2024-03-06T07:23:09.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.889+0000 7f56da44e640 1 -- 172.21.15.59:0/1052021608 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d406d120 con 0x7f56d40b2780 2024-03-06T07:23:09.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.889+0000 7f56d0ff9640 1 -- 172.21.15.59:0/1052021608 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc015820 con 0x7f56d40b2780 2024-03-06T07:23:09.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.889+0000 7f56da44e640 1 -- 172.21.15.59:0/1052021608 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d406d6e0 con 0x7f56d40b2780 2024-03-06T07:23:09.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.890+0000 7f56da44e640 1 -- 172.21.15.59:0/1052021608 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d404f330 con 0x7f56d40b2780 2024-03-06T07:23:09.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.891+0000 7f56d0ff9640 1 -- 172.21.15.59:0/1052021608 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56cc0159c0 con 0x7f56d40b2780 2024-03-06T07:23:09.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.891+0000 7f56d0ff9640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56b406b500 0x7f56b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.891+0000 7f56d0ff9640 1 -- 172.21.15.59:0/1052021608 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56cc08b140 con 0x7f56d40b2780 2024-03-06T07:23:09.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.892+0000 7f56d3fff640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56b406b500 0x7f56b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.892+0000 7f56d3fff640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56b406b500 0x7f56b406d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f56c4001eb0 tx=0x7f56c4001a90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:09.896+0000 7f56d0ff9640 1 -- 172.21.15.59:0/1052021608 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56d406cea0 con 0x7f56d40b2780 2024-03-06T07:23:10.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.157+0000 7f56da44e640 1 -- 172.21.15.59:0/1052021608 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56d415f150 con 0x7f56d40b2780 2024-03-06T07:23:10.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.157+0000 7f56d0ff9640 1 -- 172.21.15.59:0/1052021608 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f56cc019030 con 0x7f56d40b2780 2024-03-06T07:23:10.159 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:10.160 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:23:10.160 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 -- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56b406b500 msgr2=0x7f56b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56b406b500 0x7f56b406d9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f56c4001eb0 tx=0x7f56c4001a90 comp rx=0 tx=0).stop 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 -- 172.21.15.59:0/1052021608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d40b2780 msgr2=0x7f56d40c78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d40b2780 0x7f56d40c78c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f56d40713f0 tx=0x7f56cc00f580 comp rx=0 tx=0).stop 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 -- 172.21.15.59:0/1052021608 shutdown_connections 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56b406b500 0x7f56b406d9c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d40b2780 0x7f56d40c78c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56d406dee0 0x7f56d40b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 --2- 172.21.15.59:0/1052021608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56c0004800 0x7f56d4156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 -- 172.21.15.59:0/1052021608 >> 172.21.15.59:0/1052021608 conn(0x7f56d409a9f0 msgr2=0x7f56d40cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 -- 172.21.15.59:0/1052021608 shutdown_connections 2024-03-06T07:23:10.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.159+0000 7f56da44e640 1 -- 172.21.15.59:0/1052021608 wait complete. 2024-03-06T07:23:10.173 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:23:10.293 INFO:teuthology.orchestra.run.smithi059.stdout:{"purge_queue":{"pq_executed":1001}} 2024-03-06T07:23:10.294 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-03-06T07:23:10.294 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:23:10.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.333+0000 7f1f01dd4f40 1 Processor -- start 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.333+0000 7f1f01dd4f40 1 -- start start 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.333+0000 7f1f01dd4f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56325937a870 0x563259379ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f01dd4f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5632593f06a0 0x563259372580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f01dd4f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 0x5632593e9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f01dd4f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632592d8b70 con 0x563259407db0 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f01dd4f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632592dcad0 con 0x5632593f06a0 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f01dd4f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x563259361b70 con 0x56325937a870 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f000d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 0x5632593e9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f000d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 0x5632593e9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34888/0 (socket says 172.21.15.59:34888) 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f000d6640 1 -- 172.21.15.59:0/3607508795 learned_addr learned my addr 172.21.15.59:0/3607508795 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1eff0d4640 1 --2- 172.21.15.59:0/3607508795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5632593f06a0 0x563259372580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1eff8d5640 1 --2- 172.21.15.59:0/3607508795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56325937a870 0x563259379ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f000d6640 1 -- 172.21.15.59:0/3607508795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56325937a870 msgr2=0x563259379ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f000d6640 1 --2- 172.21.15.59:0/3607508795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56325937a870 0x563259379ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f000d6640 1 -- 172.21.15.59:0/3607508795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5632593f06a0 msgr2=0x563259372580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f000d6640 1 --2- 172.21.15.59:0/3607508795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5632593f06a0 0x563259372580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f000d6640 1 -- 172.21.15.59:0/3607508795 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56325935b550 con 0x563259407db0 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1f000d6640 1 --2- 172.21.15.59:0/3607508795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 0x5632593e9b00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8001620 tx=0x7f1ef8001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989049126f1bfee2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1efe0d2640 1 -- 172.21.15.59:0/3607508795 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8003650 con 0x563259407db0 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1efe0d2640 1 -- 172.21.15.59:0/3607508795 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef8003810 con 0x563259407db0 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.334+0000 7f1efe0d2640 1 -- 172.21.15.59:0/3607508795 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8003c00 con 0x563259407db0 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.335+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3607508795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 msgr2=0x5632593e9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.335+0000 7f1f01dd4f40 1 --2- 172.21.15.59:0/3607508795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 0x5632593e9b00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8001620 tx=0x7f1ef8001ae0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.335+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3607508795 shutdown_connections 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.335+0000 7f1f01dd4f40 1 --2- 172.21.15.59:0/3607508795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 0x5632593e9b00 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.335+0000 7f1f01dd4f40 1 --2- 172.21.15.59:0/3607508795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5632593f06a0 0x563259372580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.335+0000 7f1f01dd4f40 1 --2- 172.21.15.59:0/3607508795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56325937a870 0x563259379ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.335+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3607508795 >> 172.21.15.59:0/3607508795 conn(0x56325935d350 msgr2=0x56325931c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.335+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3607508795 shutdown_connections 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.335+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3607508795 wait complete. 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.335+0000 7f1f01dd4f40 1 Processor -- start 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.335+0000 7f1f01dd4f40 1 -- start start 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f01dd4f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56325937a870 0x5632594056c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f01dd4f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5632593f06a0 0x5632593ed080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f01dd4f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 0x5632593ed5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f01dd4f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632593ee390 con 0x563259407db0 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f01dd4f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632593ee130 con 0x56325937a870 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f01dd4f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x56325935b2b0 con 0x5632593f06a0 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f000d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 0x5632593ed5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f000d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 0x5632593ed5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34898/0 (socket says 172.21.15.59:34898) 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f000d6640 1 -- 172.21.15.59:0/3937967766 learned_addr learned my addr 172.21.15.59:0/3937967766 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1eff8d5640 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56325937a870 0x5632594056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1eff0d4640 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5632593f06a0 0x5632593ed080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f000d6640 1 -- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5632593f06a0 msgr2=0x5632593ed080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f000d6640 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5632593f06a0 0x5632593ed080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f000d6640 1 -- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56325937a870 msgr2=0x5632594056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f000d6640 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56325937a870 0x5632594056c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1f000d6640 1 -- 172.21.15.59:0/3937967766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563259405c00 con 0x563259407db0 2024-03-06T07:23:10.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.336+0000 7f1eff8d5640 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56325937a870 0x5632594056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:10.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.337+0000 7f1f000d6640 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 0x5632593ed5c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8000f80 tx=0x7f1ef8001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.337+0000 7f1efd0d0640 1 -- 172.21.15.59:0/3937967766 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8024070 con 0x563259407db0 2024-03-06T07:23:10.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.337+0000 7f1efd0d0640 1 -- 172.21.15.59:0/3937967766 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef8019040 con 0x563259407db0 2024-03-06T07:23:10.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.337+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3937967766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56325937a680 con 0x563259407db0 2024-03-06T07:23:10.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.337+0000 7f1efd0d0640 1 -- 172.21.15.59:0/3937967766 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8013a30 con 0x563259407db0 2024-03-06T07:23:10.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.337+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3937967766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56325937a0c0 con 0x563259407db0 2024-03-06T07:23:10.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.338+0000 7f1efd0d0640 1 -- 172.21.15.59:0/3937967766 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ef8010070 con 0x563259407db0 2024-03-06T07:23:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.339+0000 7f1efd0d0640 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec069eb0 0x7f1eec06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.339+0000 7f1efd0d0640 1 -- 172.21.15.59:0/3937967766 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ef8089df0 con 0x563259407db0 2024-03-06T07:23:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.339+0000 7f1f01dd4f40 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x56325935adf0 0x5632593e8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.339+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3937967766 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v9 -- 0x5632593e8a60 con 0x56325935adf0 2024-03-06T07:23:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.339+0000 7f1eff8d5640 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec069eb0 0x7f1eec06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.339+0000 7f1eff0d4640 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x56325935adf0 0x5632593e8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.339+0000 7f1eff0d4640 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x56325935adf0 0x5632593e8520 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.339+0000 7f1eff8d5640 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec069eb0 0x7f1eec06c370 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0000c00 tx=0x7f1ef0003050 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1eff0d4640 1 -- 172.21.15.59:0/3937967766 <== osd.3 v2:172.21.15.59:6808/3602142643 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f1ef404da10 con 0x56325935adf0 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3937967766 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x56325935adf0 msgr2=0x5632593e8520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1f01dd4f40 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x56325935adf0 0x5632593e8520 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec069eb0 msgr2=0x7f1eec06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1f01dd4f40 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec069eb0 0x7f1eec06c370 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0000c00 tx=0x7f1ef0003050 comp rx=0 tx=0).stop 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3937967766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 msgr2=0x5632593ed5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1f01dd4f40 1 --2- 172.21.15.59:0/3937967766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563259407db0 0x5632593ed5c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8000f80 tx=0x7f1ef8001250 comp rx=0 tx=0).stop 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1f000d6640 1 -- 172.21.15.59:0/3937967766 reap_dead start 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3937967766 shutdown_connections 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3937967766 >> 172.21.15.59:0/3937967766 conn(0x56325935d350 msgr2=0x56325931c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3937967766 shutdown_connections 2024-03-06T07:23:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.340+0000 7f1f01dd4f40 1 -- 172.21.15.59:0/3937967766 wait complete. 2024-03-06T07:23:10.348 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:23:10.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.485+0000 7fd68cdbf640 1 Processor -- start 2024-03-06T07:23:10.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.485+0000 7fd68cdbf640 1 -- start start 2024-03-06T07:23:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd68cdbf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd6880b4460 0x7fd6880d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd68cdbf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd688162970 0x7fd6880b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd68cdbf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6880d22c0 con 0x7fd688161cf0 2024-03-06T07:23:10.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd68cdbf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd688161b00 con 0x7fd6880b4460 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd68cdbf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6880cecc0 con 0x7fd688162970 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd686d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd688161cf0 0x7fd6880cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36832/0 (socket says 172.21.15.59:36832) 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd686d9d640 1 -- 172.21.15.59:0/4042319682 learned_addr learned my addr 172.21.15.59:0/4042319682 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd67ffff640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd688162970 0x7fd6880b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd68659c640 1 --2- 172.21.15.59:0/4042319682 >> v2:172.21.15.164:3300/0 conn(0x7fd6880b4460 0x7fd6880d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66696071 0 0) 0x7fd6880d22c0 con 0x7fd688161cf0 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd674003520 con 0x7fd688161cf0 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2929230850 0 0) 0x7fd674003520 con 0x7fd688161cf0 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd688162970 msgr2=0x7fd6880b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd685d9b640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd688162970 0x7fd6880b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 >> v2:172.21.15.164:3300/0 conn(0x7fd6880b4460 msgr2=0x7fd6880d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd685d9b640 1 --2- 172.21.15.59:0/4042319682 >> v2:172.21.15.164:3300/0 conn(0x7fd6880b4460 0x7fd6880d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6880d25a0 con 0x7fd688161cf0 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fd6800019f0 con 0x7fd688161cf0 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 >> v1:172.21.15.59:6789/0 conn(0x7fd688161cf0 legacy=0x7fd6880cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.487+0000 7fd685d9b640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6740039b0 0x7fd674003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd685d9b640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd688162970 0x7fd674004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd685d9b640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd674004d40 0x7fd6740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd688161b00 con 0x7fd674004d40 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6880cecc0 con 0x7fd6740039b0 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67400b9c0 con 0x7fd688162970 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd686d9d640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd674004d40 0x7fd6740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd68659c640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6740039b0 0x7fd674003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd67ffff640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd688162970 0x7fd674004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd686d9d640 1 -- 172.21.15.59:0/4042319682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd688162970 msgr2=0x7fd674004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd686d9d640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd688162970 0x7fd674004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd686d9d640 1 -- 172.21.15.59:0/4042319682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6740039b0 msgr2=0x7fd674003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd686d9d640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6740039b0 0x7fd674003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.488+0000 7fd686d9d640 1 -- 172.21.15.59:0/4042319682 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd674003520 con 0x7fd674004d40 2024-03-06T07:23:10.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.489+0000 7fd686d9d640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd674004d40 0x7fd6740051c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd6800013d0 tx=0x7fd674005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e72bfe7c55ff46b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.489+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68000a4e0 con 0x7fd674004d40 2024-03-06T07:23:10.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.489+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd680034650 con 0x7fd674004d40 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.489+0000 7fd685d9b640 1 -- 172.21.15.59:0/4042319682 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680034a40 con 0x7fd674004d40 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.490+0000 7fd68cdbf640 1 -- 172.21.15.59:0/4042319682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd674004d40 msgr2=0x7fd6740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.490+0000 7fd68cdbf640 1 --2- 172.21.15.59:0/4042319682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd674004d40 0x7fd6740051c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd6800013d0 tx=0x7fd674005830 comp rx=0 tx=0).stop 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.490+0000 7fd686d9d640 1 -- 172.21.15.59:0/4042319682 reap_dead start 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.490+0000 7fd68cdbf640 1 -- 172.21.15.59:0/4042319682 shutdown_connections 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.490+0000 7fd68cdbf640 1 -- 172.21.15.59:0/4042319682 >> 172.21.15.59:0/4042319682 conn(0x7fd6880d3ce0 msgr2=0x7fd6880d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.490+0000 7fd68cdbf640 1 -- 172.21.15.59:0/4042319682 shutdown_connections 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.490+0000 7fd68cdbf640 1 -- 172.21.15.59:0/4042319682 wait complete. 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.491+0000 7fd68cdbf640 1 Processor -- start 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.491+0000 7fd68cdbf640 1 -- start start 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68cdbf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd688155d90 0x7fd6880b8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68cdbf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd688156170 0x7fd6880c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68cdbf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880b9510 0x7fd6880c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68cdbf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6880ce950 con 0x7fd688155d90 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68cdbf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6880d2b60 con 0x7fd688156170 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68cdbf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6880b4840 con 0x7fd6880b9510 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd688155d90 0x7fd6880b8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd688155d90 0x7fd6880b8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34924/0 (socket says 172.21.15.59:34924) 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68659c640 1 -- 172.21.15.59:0/1837255531 learned_addr learned my addr 172.21.15.59:0/1837255531 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd686d9d640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880b9510 0x7fd6880c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd67ffff640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd688156170 0x7fd6880c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68659c640 1 -- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880b9510 msgr2=0x7fd6880c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68659c640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880b9510 0x7fd6880c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68659c640 1 -- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd688156170 msgr2=0x7fd6880c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68659c640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd688156170 0x7fd6880c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.492+0000 7fd68659c640 1 -- 172.21.15.59:0/1837255531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6880b7560 con 0x7fd688155d90 2024-03-06T07:23:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.493+0000 7fd68659c640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd688155d90 0x7fd6880b8fd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd678011ad0 tx=0x7fd678011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.493+0000 7fd67effd640 1 -- 172.21.15.59:0/1837255531 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67800fbb0 con 0x7fd688155d90 2024-03-06T07:23:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.493+0000 7fd67effd640 1 -- 172.21.15.59:0/1837255531 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67800fd50 con 0x7fd688155d90 2024-03-06T07:23:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.493+0000 7fd68cdbf640 1 -- 172.21.15.59:0/1837255531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6880b7840 con 0x7fd688155d90 2024-03-06T07:23:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.493+0000 7fd67effd640 1 -- 172.21.15.59:0/1837255531 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678014660 con 0x7fd688155d90 2024-03-06T07:23:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.493+0000 7fd68cdbf640 1 -- 172.21.15.59:0/1837255531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6880b7e00 con 0x7fd688155d90 2024-03-06T07:23:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.494+0000 7fd68cdbf640 1 -- 172.21.15.59:0/1837255531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd688067190 con 0x7fd688155d90 2024-03-06T07:23:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.495+0000 7fd67effd640 1 -- 172.21.15.59:0/1837255531 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd678012040 con 0x7fd688155d90 2024-03-06T07:23:10.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.495+0000 7fd67effd640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b4d0 0x7fd66806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.495+0000 7fd67effd640 1 -- 172.21.15.59:0/1837255531 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd67808a9d0 con 0x7fd688155d90 2024-03-06T07:23:10.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.495+0000 7fd67ffff640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b4d0 0x7fd66806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.496+0000 7fd67ffff640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b4d0 0x7fd66806d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd6880c8a40 tx=0x7fd670008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.500+0000 7fd67effd640 1 -- 172.21.15.59:0/1837255531 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd688067190 con 0x7fd688155d90 2024-03-06T07:23:10.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.752+0000 7fd68cdbf640 1 -- 172.21.15.59:0/1837255531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fd68804f330 con 0x7fd688155d90 2024-03-06T07:23:10.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.752+0000 7fd67effd640 1 -- 172.21.15.59:0/1837255531 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1657 (secure 0 0 0) 0x7fd678019030 con 0x7fd688155d90 2024-03-06T07:23:10.754 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:10.754 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "stats": { 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "total_avail_bytes": 772828397568, 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_bytes": 265715712, 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_raw_bytes": 265715712, 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_raw_ratio": 0.00034370421781204641, 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "num_osds": 8, 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "num_per_pool_osds": 8, 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "stats_by_class": { 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "ssd": { 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "total_avail_bytes": 772828397568, 2024-03-06T07:23:10.755 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_bytes": 265715712, 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_raw_bytes": 265715712, 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_raw_ratio": 0.00034370421781204641 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: "pools": [ 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: "name": ".mgr", 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 1, 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: "stats": { 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: "stored": 590368, 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: "objects": 2, 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: "kb_used": 1160, 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: "bytes_used": 1187840, 2024-03-06T07:23:10.756 INFO:teuthology.orchestra.run.smithi059.stdout: "percent_used": 1.6180357533812639e-06, 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: "max_avail": 367061762048 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs_metadata", 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 4, 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: "stats": { 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: "stored": 3077573, 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: "objects": 23, 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: "kb_used": 6072, 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: "bytes_used": 6217728, 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: "percent_used": 8.469522072118707e-06, 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: "max_avail": 367061762048 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs_data", 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 5, 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: "stats": { 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: "stored": 0, 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: "objects": 0, 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: "kb_used": 0, 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: "bytes_used": 0, 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: "percent_used": 0, 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: "max_avail": 367061762048 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-06T07:23:10.758 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:23:10.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.754+0000 7fd68cdbf640 1 -- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b4d0 msgr2=0x7fd66806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.754+0000 7fd68cdbf640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b4d0 0x7fd66806d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd6880c8a40 tx=0x7fd670008040 comp rx=0 tx=0).stop 2024-03-06T07:23:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.754+0000 7fd68cdbf640 1 -- 172.21.15.59:0/1837255531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd688155d90 msgr2=0x7fd6880b8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.754+0000 7fd68cdbf640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd688155d90 0x7fd6880b8fd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd678011ad0 tx=0x7fd678011f90 comp rx=0 tx=0).stop 2024-03-06T07:23:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.754+0000 7fd68cdbf640 1 -- 172.21.15.59:0/1837255531 shutdown_connections 2024-03-06T07:23:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.754+0000 7fd68cdbf640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd66806b4d0 0x7fd66806d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.754+0000 7fd68cdbf640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6880b9510 0x7fd6880c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.754+0000 7fd68cdbf640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd688156170 0x7fd6880c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.754+0000 7fd68cdbf640 1 --2- 172.21.15.59:0/1837255531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd688155d90 0x7fd6880b8fd0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.754+0000 7fd68cdbf640 1 -- 172.21.15.59:0/1837255531 >> 172.21.15.59:0/1837255531 conn(0x7fd6880d3ce0 msgr2=0x7fd6880d0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.755+0000 7fd68cdbf640 1 -- 172.21.15.59:0/1837255531 shutdown_connections 2024-03-06T07:23:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:10.755+0000 7fd68cdbf640 1 -- 172.21.15.59:0/1837255531 wait complete. 2024-03-06T07:23:10.769 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:23:10.770 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:10.770 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:10.811 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:10.857 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:10.858 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:23:10.858 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:10.872 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.012+0000 7fdc8a4ed640 1 Processor -- start 2024-03-06T07:23:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.012+0000 7fdc8a4ed640 1 -- start start 2024-03-06T07:23:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc8a4ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 0x7fdc840cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc8a4ed640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdc840cd7a0 0x7fdc840cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc8a4ed640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc84071f40 con 0x7fdc8415b430 2024-03-06T07:23:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc8a4ed640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc8415b8b0 con 0x7fdc840cd7a0 2024-03-06T07:23:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc8a4ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc84070a70 con 0x7fdc841515e0 2024-03-06T07:23:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc88a8a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdc840cd7a0 0x7fdc840cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc837fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdc8415b430 0x7fdc84071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36842/0 (socket says 172.21.15.59:36842) 2024-03-06T07:23:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc837fe640 1 -- 172.21.15.59:0/400531241 learned_addr learned my addr 172.21.15.59:0/400531241 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc83fff640 1 --2- 172.21.15.59:0/400531241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 0x7fdc840cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc88a8a640 1 -- 172.21.15.59:0/400531241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 msgr2=0x7fdc840cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc88a8a640 1 --2- 172.21.15.59:0/400531241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 0x7fdc840cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc88a8a640 1 -- 172.21.15.59:0/400531241 >> v1:172.21.15.59:6789/0 conn(0x7fdc8415b430 legacy=0x7fdc84071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc88a8a640 1 -- 172.21.15.59:0/400531241 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc84070d50 con 0x7fdc840cd7a0 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc82ffd640 1 -- 172.21.15.59:0/400531241 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617607376 0 0) 0x7fdc84071f40 con 0x7fdc8415b430 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.013+0000 7fdc83fff640 1 --2- 172.21.15.59:0/400531241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 0x7fdc840cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.014+0000 7fdc88a8a640 1 --2- 172.21.15.59:0/400531241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc840cd7a0 0x7fdc840cdbd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009d90 tx=0x7fdc7c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4415c916d6d6a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.014+0000 7fdc82ffd640 1 -- 172.21.15.59:0/400531241 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c039070 con 0x7fdc840cd7a0 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.014+0000 7fdc82ffd640 1 -- 172.21.15.59:0/400531241 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7c02cc50 con 0x7fdc840cd7a0 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.014+0000 7fdc82ffd640 1 -- 172.21.15.59:0/400531241 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c034440 con 0x7fdc840cd7a0 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.015+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/400531241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc840cd7a0 msgr2=0x7fdc840cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.015+0000 7fdc8a4ed640 1 --2- 172.21.15.59:0/400531241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc840cd7a0 0x7fdc840cdbd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009d90 tx=0x7fdc7c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.015+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/400531241 shutdown_connections 2024-03-06T07:23:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.015+0000 7fdc8a4ed640 1 --2- 172.21.15.59:0/400531241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc840cd7a0 0x7fdc840cdbd0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.015+0000 7fdc8a4ed640 1 --2- 172.21.15.59:0/400531241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 0x7fdc840cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.015+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/400531241 >> 172.21.15.59:0/400531241 conn(0x7fdc840cf8a0 msgr2=0x7fdc840cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.015+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/400531241 shutdown_connections 2024-03-06T07:23:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.015+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/400531241 wait complete. 2024-03-06T07:23:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.016+0000 7fdc8a4ed640 1 Processor -- start 2024-03-06T07:23:11.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.016+0000 7fdc8a4ed640 1 -- start start 2024-03-06T07:23:11.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc8a4ed640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc840cd7a0 0x7fdc840c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc8a4ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 0x7fdc841f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc8a4ed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc8415b430 0x7fdc8414ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc8a4ed640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc840ce110 con 0x7fdc840cd7a0 2024-03-06T07:23:11.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc8a4ed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc84071310 con 0x7fdc8415b430 2024-03-06T07:23:11.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc8a4ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc840cc3a0 con 0x7fdc841515e0 2024-03-06T07:23:11.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc83fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc840cd7a0 0x7fdc840c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc83fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc840cd7a0 0x7fdc840c6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34938/0 (socket says 172.21.15.59:34938) 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc83fff640 1 -- 172.21.15.59:0/1250076681 learned_addr learned my addr 172.21.15.59:0/1250076681 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc837fe640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 0x7fdc841f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc83fff640 1 -- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 msgr2=0x7fdc841f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc88a8a640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc8415b430 0x7fdc8414ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc83fff640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 0x7fdc841f38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc83fff640 1 -- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc8415b430 msgr2=0x7fdc8414ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc83fff640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc8415b430 0x7fdc8414ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc83fff640 1 -- 172.21.15.59:0/1250076681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc840c75c0 con 0x7fdc840cd7a0 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc837fe640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 0x7fdc841f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc88a8a640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc8415b430 0x7fdc8414ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:11.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc83fff640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc840cd7a0 0x7fdc840c6e90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400e7e0 tx=0x7fdc7400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc80ff9640 1 -- 172.21.15.59:0/1250076681 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7400f040 con 0x7fdc840cd7a0 2024-03-06T07:23:11.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc80ff9640 1 -- 172.21.15.59:0/1250076681 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc74004030 con 0x7fdc840cd7a0 2024-03-06T07:23:11.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/1250076681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc840c78a0 con 0x7fdc840cd7a0 2024-03-06T07:23:11.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc80ff9640 1 -- 172.21.15.59:0/1250076681 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc74004400 con 0x7fdc840cd7a0 2024-03-06T07:23:11.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.018+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/1250076681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc840c7e60 con 0x7fdc840cd7a0 2024-03-06T07:23:11.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.019+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/1250076681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc84068840 con 0x7fdc840cd7a0 2024-03-06T07:23:11.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.020+0000 7fdc80ff9640 1 -- 172.21.15.59:0/1250076681 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc7400c670 con 0x7fdc840cd7a0 2024-03-06T07:23:11.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.020+0000 7fdc80ff9640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdc6406b4d0 0x7fdc6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.021+0000 7fdc80ff9640 1 -- 172.21.15.59:0/1250076681 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc74089ed0 con 0x7fdc840cd7a0 2024-03-06T07:23:11.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.021+0000 7fdc837fe640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdc6406b4d0 0x7fdc6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.021+0000 7fdc837fe640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdc6406b4d0 0x7fdc6406d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdc84150fb0 tx=0x7fdc6c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.025+0000 7fdc80ff9640 1 -- 172.21.15.59:0/1250076681 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc74017020 con 0x7fdc840cd7a0 2024-03-06T07:23:11.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.207+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/1250076681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc8404f330 con 0x7fdc840cd7a0 2024-03-06T07:23:11.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.208+0000 7fdc80ff9640 1 -- 172.21.15.59:0/1250076681 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16657 (secure 0 0 0) 0x7fdc74058de0 con 0x7fdc840cd7a0 2024-03-06T07:23:11.210 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:11.210 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":33,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:52.226312+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:48.848650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:49.545360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:11.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdc6406b4d0 msgr2=0x7fdc6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdc6406b4d0 0x7fdc6406d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdc84150fb0 tx=0x7fdc6c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:11.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/1250076681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc840cd7a0 msgr2=0x7fdc840c6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc840cd7a0 0x7fdc840c6e90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fdc7400e7e0 tx=0x7fdc7400eca0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/1250076681 shutdown_connections 2024-03-06T07:23:11.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdc6406b4d0 0x7fdc6406d990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc8415b430 0x7fdc8414ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc841515e0 0x7fdc841f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 --2- 172.21.15.59:0/1250076681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc840cd7a0 0x7fdc840c6e90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/1250076681 >> 172.21.15.59:0/1250076681 conn(0x7fdc840cf8a0 msgr2=0x7fdc8414d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/1250076681 shutdown_connections 2024-03-06T07:23:11.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.210+0000 7fdc8a4ed640 1 -- 172.21.15.59:0/1250076681 wait complete. 2024-03-06T07:23:11.223 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:48.848650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:49.545360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:11.224 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:11.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.326+0000 7f72f1c9a640 1 Processor -- start 2024-03-06T07:23:11.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.326+0000 7f72f1c9a640 1 -- start start 2024-03-06T07:23:11.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.327+0000 7f72f1c9a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b1e40 0x7f72ec168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.327+0000 7f72f1c9a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec0d2730 con 0x7f72ec0d4860 2024-03-06T07:23:11.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.327+0000 7f72f1c9a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec1618b0 con 0x7f72ec1614d0 2024-03-06T07:23:11.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.327+0000 7f72f1c9a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0d4cf0 con 0x7f72ec0b1e40 2024-03-06T07:23:11.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.327+0000 7f72eaffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f72ec0d4860 0x7f72ec0d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36848/0 (socket says 172.21.15.59:36848) 2024-03-06T07:23:11.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.327+0000 7f72eaffd640 1 -- 172.21.15.59:0/112424926 learned_addr learned my addr 172.21.15.59:0/112424926 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:11.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.327+0000 7f72ebfff640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b1e40 0x7f72ec168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607762230 0 0) 0x7f72ec0d2730 con 0x7f72ec0d4860 2024-03-06T07:23:11.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d4003520 con 0x7f72ec0d4860 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476305655 0 0) 0x7f72ec1618b0 con 0x7f72ec1614d0 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ec0d2730 con 0x7f72ec1614d0 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1423059103 0 0) 0x7f72d4003520 con 0x7f72ec0d4860 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b1e40 msgr2=0x7f72ec168b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b1e40 0x7f72ec168b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 >> v1:172.21.15.164:6789/0 conn(0x7f72ec1614d0 legacy=0x7f72ec0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0cf400 con 0x7f72ec0d4860 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f72e0001670 con 0x7f72ec0d4860 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ebfff640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b1e40 0x7f72ec168b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 >> v1:172.21.15.59:6789/0 conn(0x7f72ec0d4860 legacy=0x7f72ec0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72d4003b70 0x7f72d4003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 0x7f72d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b1e40 0x7f72d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0d4cf0 con 0x7f72d4004800 2024-03-06T07:23:11.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d400ba70 con 0x7f72d4003b70 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d400bc70 con 0x7f72ec0b1e40 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72eaffd640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 0x7f72d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72eb7fe640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72d4003b70 0x7f72d4003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72ebfff640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b1e40 0x7f72d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72eaffd640 1 -- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b1e40 msgr2=0x7f72d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72eaffd640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b1e40 0x7f72d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72eaffd640 1 -- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72d4003b70 msgr2=0x7f72d4003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72eaffd640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72d4003b70 0x7f72d4003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.328+0000 7f72eaffd640 1 -- 172.21.15.59:0/112424926 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d4003520 con 0x7f72d4004800 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.329+0000 7f72eaffd640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 0x7f72d4004c80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f72e0001eb0 tx=0x7f72d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7c2c4b5292f156 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.329+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0007c00 con 0x7f72d4004800 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.329+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e0030070 con 0x7f72d4004800 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.329+0000 7f72ea7fc640 1 -- 172.21.15.59:0/112424926 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0005d30 con 0x7f72d4004800 2024-03-06T07:23:11.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.330+0000 7f72f1c9a640 1 -- 172.21.15.59:0/112424926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 msgr2=0x7f72d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.330+0000 7f72f1c9a640 1 --2- 172.21.15.59:0/112424926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 0x7f72d4004c80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f72e0001eb0 tx=0x7f72d4005830 comp rx=0 tx=0).stop 2024-03-06T07:23:11.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.330+0000 7f72ebfff640 1 -- 172.21.15.59:0/112424926 reap_dead start 2024-03-06T07:23:11.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.330+0000 7f72f1c9a640 1 -- 172.21.15.59:0/112424926 shutdown_connections 2024-03-06T07:23:11.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.330+0000 7f72f1c9a640 1 -- 172.21.15.59:0/112424926 >> 172.21.15.59:0/112424926 conn(0x7f72ec0d3d40 msgr2=0x7f72ec0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.330+0000 7f72f1c9a640 1 -- 172.21.15.59:0/112424926 shutdown_connections 2024-03-06T07:23:11.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.330+0000 7f72f1c9a640 1 -- 172.21.15.59:0/112424926 wait complete. 2024-03-06T07:23:11.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.332+0000 7f72f1c9a640 1 Processor -- start 2024-03-06T07:23:11.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.332+0000 7f72f1c9a640 1 -- start start 2024-03-06T07:23:11.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72f1c9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 0x7f72ec14aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72f1c9a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b87a0 0x7f72ec0b8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72f1c9a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72ec0b6930 0x7f72ec0b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72f1c9a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0cfc60 con 0x7f72d4004800 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72f1c9a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0cf980 con 0x7f72ec0b6930 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72f1c9a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0b0910 con 0x7f72ec0b87a0 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72eb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 0x7f72ec14aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72eb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 0x7f72ec14aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34950/0 (socket says 172.21.15.59:34950) 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72eb7fe640 1 -- 172.21.15.59:0/4078824874 learned_addr learned my addr 172.21.15.59:0/4078824874 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72eaffd640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b87a0 0x7f72ec0b8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72ebfff640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72ec0b6930 0x7f72ec0b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72eb7fe640 1 -- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b87a0 msgr2=0x7f72ec0b8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72eb7fe640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b87a0 0x7f72ec0b8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72eb7fe640 1 -- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72ec0b6930 msgr2=0x7f72ec0b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.333+0000 7f72eb7fe640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72ec0b6930 0x7f72ec0b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.334+0000 7f72eb7fe640 1 -- 172.21.15.59:0/4078824874 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec14b010 con 0x7f72d4004800 2024-03-06T07:23:11.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.334+0000 7f72eb7fe640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 0x7f72ec14aad0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00f7a0 tx=0x7f72dc00fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.334+0000 7f72d3fff640 1 -- 172.21.15.59:0/4078824874 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc01a070 con 0x7f72d4004800 2024-03-06T07:23:11.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.334+0000 7f72d3fff640 1 -- 172.21.15.59:0/4078824874 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc00fe50 con 0x7f72d4004800 2024-03-06T07:23:11.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.334+0000 7f72f1c9a640 1 -- 172.21.15.59:0/4078824874 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec0b92f0 con 0x7f72d4004800 2024-03-06T07:23:11.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.334+0000 7f72d3fff640 1 -- 172.21.15.59:0/4078824874 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc007050 con 0x7f72d4004800 2024-03-06T07:23:11.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.334+0000 7f72f1c9a640 1 -- 172.21.15.59:0/4078824874 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec0b98b0 con 0x7f72d4004800 2024-03-06T07:23:11.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.335+0000 7f72f1c9a640 1 -- 172.21.15.59:0/4078824874 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec068ff0 con 0x7f72d4004800 2024-03-06T07:23:11.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.335+0000 7f72d3fff640 1 -- 172.21.15.59:0/4078824874 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72dc0071f0 con 0x7f72d4004800 2024-03-06T07:23:11.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.336+0000 7f72d3fff640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 0x7f72d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.336+0000 7f72d3fff640 1 -- 172.21.15.59:0/4078824874 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72dc0899c0 con 0x7f72d4004800 2024-03-06T07:23:11.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.336+0000 7f72eaffd640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 0x7f72d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.337+0000 7f72eaffd640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 0x7f72d806d9c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f72e0001eb0 tx=0x7f72e0009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.340+0000 7f72d3fff640 1 -- 172.21.15.59:0/4078824874 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72dc08be20 con 0x7f72d4004800 2024-03-06T07:23:11.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.542+0000 7f72f1c9a640 1 -- 172.21.15.59:0/4078824874 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72ec06d660 con 0x7f72d4004800 2024-03-06T07:23:11.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.542+0000 7f72d3fff640 1 -- 172.21.15.59:0/4078824874 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f72dc0587c0 con 0x7f72d4004800 2024-03-06T07:23:11.544 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:23:11.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 -- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 msgr2=0x7f72d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 0x7f72d806d9c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f72e0001eb0 tx=0x7f72e0009030 comp rx=0 tx=0).stop 2024-03-06T07:23:11.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 -- 172.21.15.59:0/4078824874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 msgr2=0x7f72ec14aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 0x7f72ec14aad0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00f7a0 tx=0x7f72dc00fc60 comp rx=0 tx=0).stop 2024-03-06T07:23:11.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 -- 172.21.15.59:0/4078824874 shutdown_connections 2024-03-06T07:23:11.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 0x7f72d806d9c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72ec0b6930 0x7f72ec0b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec0b87a0 0x7f72ec0b8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 --2- 172.21.15.59:0/4078824874 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72d4004800 0x7f72ec14aad0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 -- 172.21.15.59:0/4078824874 >> 172.21.15.59:0/4078824874 conn(0x7f72ec0d3d40 msgr2=0x7f72ec0d0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 -- 172.21.15.59:0/4078824874 shutdown_connections 2024-03-06T07:23:11.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.544+0000 7f72f1c9a640 1 -- 172.21.15.59:0/4078824874 wait complete. 2024-03-06T07:23:11.557 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:11.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.659+0000 7f0b41088640 1 Processor -- start 2024-03-06T07:23:11.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.659+0000 7f0b41088640 1 -- start start 2024-03-06T07:23:11.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b41088640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b3c06fd10 0x7f0b3c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b41088640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0b3c0d0af0 0x7f0b3c0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b41088640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c0cd1e0 con 0x7f0b3c15c050 2024-03-06T07:23:11.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b41088640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3c0cd4b0 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b41088640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c0cd790 con 0x7f0b3c06fd10 2024-03-06T07:23:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b3b59e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0b3c0d0af0 0x7f0b3c0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b3b59e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0b3c0d0af0 0x7f0b3c0ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34960/0 (socket says 172.21.15.59:34960) 2024-03-06T07:23:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b3b59e640 1 -- 172.21.15.59:0/2611019318 learned_addr learned my addr 172.21.15.59:0/2611019318 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b3a59c640 1 --2- 172.21.15.59:0/2611019318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b3c06fd10 0x7f0b3c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b3b59e640 1 -- 172.21.15.59:0/2611019318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b3c06fd10 msgr2=0x7f0b3c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b3b59e640 1 --2- 172.21.15.59:0/2611019318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b3c06fd10 0x7f0b3c0cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b3b59e640 1 -- 172.21.15.59:0/2611019318 >> v1:172.21.15.164:6789/0 conn(0x7f0b3c15c050 legacy=0x7f0b3c06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.660+0000 7f0b3b59e640 1 -- 172.21.15.59:0/2611019318 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c0cda70 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.661+0000 7f0b3b59e640 1 --2- 172.21.15.59:0/2611019318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b3c0d0af0 0x7f0b3c0ccca0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009e50 tx=0x7f0b3402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3f875a22a23654 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.661+0000 7f0b39d9b640 1 -- 172.21.15.59:0/2611019318 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3402c9e0 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.661+0000 7f0b39d9b640 1 -- 172.21.15.59:0/2611019318 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3402cb80 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.661+0000 7f0b39d9b640 1 -- 172.21.15.59:0/2611019318 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b340343f0 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.662+0000 7f0b41088640 1 -- 172.21.15.59:0/2611019318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b3c0d0af0 msgr2=0x7f0b3c0ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.662+0000 7f0b41088640 1 --2- 172.21.15.59:0/2611019318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b3c0d0af0 0x7f0b3c0ccca0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009e50 tx=0x7f0b3402be40 comp rx=0 tx=0).stop 2024-03-06T07:23:11.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.662+0000 7f0b41088640 1 -- 172.21.15.59:0/2611019318 shutdown_connections 2024-03-06T07:23:11.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.662+0000 7f0b41088640 1 --2- 172.21.15.59:0/2611019318 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b3c0d0af0 0x7f0b3c0ccca0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.662+0000 7f0b41088640 1 --2- 172.21.15.59:0/2611019318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b3c06fd10 0x7f0b3c0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.662+0000 7f0b41088640 1 -- 172.21.15.59:0/2611019318 >> 172.21.15.59:0/2611019318 conn(0x7f0b3c0b3800 msgr2=0x7f0b3c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.662+0000 7f0b41088640 1 -- 172.21.15.59:0/2611019318 shutdown_connections 2024-03-06T07:23:11.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.662+0000 7f0b41088640 1 -- 172.21.15.59:0/2611019318 wait complete. 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.663+0000 7f0b41088640 1 Processor -- start 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.663+0000 7f0b41088640 1 -- start start 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.664+0000 7f0b41088640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b3c06fd10 0x7f0b3c145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.664+0000 7f0b41088640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b3c0d0af0 0x7f0b3c06c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.664+0000 7f0b41088640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b3c15c050 0x7f0b3c0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.664+0000 7f0b41088640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c165c30 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.664+0000 7f0b41088640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c165ab0 con 0x7f0b3c15c050 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.664+0000 7f0b41088640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c165db0 con 0x7f0b3c06fd10 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.664+0000 7f0b3a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b3c0d0af0 0x7f0b3c06c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.664+0000 7f0b3a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b3c0d0af0 0x7f0b3c06c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34972/0 (socket says 172.21.15.59:34972) 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.664+0000 7f0b3a59c640 1 -- 172.21.15.59:0/3223832490 learned_addr learned my addr 172.21.15.59:0/3223832490 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b3a59c640 1 -- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b3c06fd10 msgr2=0x7f0b3c145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b3b59e640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b3c15c050 0x7f0b3c0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b3a59c640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b3c06fd10 0x7f0b3c145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b3a59c640 1 -- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b3c15c050 msgr2=0x7f0b3c0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b3a59c640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b3c15c050 0x7f0b3c0cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b3a59c640 1 -- 172.21.15.59:0/3223832490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c06cdb0 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b3a59c640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b3c0d0af0 0x7f0b3c06c560 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000d5f0 tx=0x7f0b3000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b237fe640 1 -- 172.21.15.59:0/3223832490 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b300125f0 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b237fe640 1 -- 172.21.15.59:0/3223832490 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b30019070 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b41088640 1 -- 172.21.15.59:0/3223832490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3c06d090 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b237fe640 1 -- 172.21.15.59:0/3223832490 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b300129d0 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.665+0000 7f0b41088640 1 -- 172.21.15.59:0/3223832490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c06d650 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.666+0000 7f0b41088640 1 -- 172.21.15.59:0/3223832490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c09fe60 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.666+0000 7f0b237fe640 1 -- 172.21.15.59:0/3223832490 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b30012b70 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.667+0000 7f0b237fe640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b2806b4d0 0x7f0b2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.667+0000 7f0b237fe640 1 -- 172.21.15.59:0/3223832490 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b3008a5c0 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.667+0000 7f0b3ad9d640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b2806b4d0 0x7f0b2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.668+0000 7f0b3ad9d640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b2806b4d0 0x7f0b2806d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c006fd0 tx=0x7f0b2c006bc0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.671+0000 7f0b237fe640 1 -- 172.21.15.59:0/3223832490 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b30059440 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.851+0000 7f0b41088640 1 -- 172.21.15.59:0/3223832490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0b3c068ff0 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.851+0000 7f0b237fe640 1 -- 172.21.15.59:0/3223832490 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0b30058d40 con 0x7f0b3c0d0af0 2024-03-06T07:23:11.853 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:23:11.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 -- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b2806b4d0 msgr2=0x7f0b2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b2806b4d0 0x7f0b2806d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c006fd0 tx=0x7f0b2c006bc0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 -- 172.21.15.59:0/3223832490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b3c0d0af0 msgr2=0x7f0b3c06c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b3c0d0af0 0x7f0b3c06c560 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000d5f0 tx=0x7f0b3000dab0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 -- 172.21.15.59:0/3223832490 shutdown_connections 2024-03-06T07:23:11.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b2806b4d0 0x7f0b2806d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b3c15c050 0x7f0b3c0cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b3c0d0af0 0x7f0b3c06c560 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 --2- 172.21.15.59:0/3223832490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b3c06fd10 0x7f0b3c145610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 -- 172.21.15.59:0/3223832490 >> 172.21.15.59:0/3223832490 conn(0x7f0b3c0b3800 msgr2=0x7f0b3c0b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 -- 172.21.15.59:0/3223832490 shutdown_connections 2024-03-06T07:23:11.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.853+0000 7f0b41088640 1 -- 172.21.15.59:0/3223832490 wait complete. 2024-03-06T07:23:11.868 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:11.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.970+0000 7fbc3f9b5640 1 Processor -- start 2024-03-06T07:23:11.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.970+0000 7fbc3f9b5640 1 -- start start 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc3f9b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbc380b1e40 0x7fbc38168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc3f9b5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc380d2730 con 0x7fbc381614d0 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc3f9b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc381618b0 con 0x7fbc380d4860 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc3f9b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc380d4cf0 con 0x7fbc380b1e40 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc3d751640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbc381614d0 0x7fbc380d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36856/0 (socket says 172.21.15.59:36856) 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc3d751640 1 -- 172.21.15.59:0/3750401324 learned_addr learned my addr 172.21.15.59:0/3750401324 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc3df52640 1 --2- 172.21.15.59:0/3750401324 >> v2:172.21.15.164:3300/0 conn(0x7fbc380b1e40 0x7fbc38168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273655847 0 0) 0x7fbc380d2730 con 0x7fbc381614d0 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc24003520 con 0x7fbc381614d0 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3064065192 0 0) 0x7fbc24003520 con 0x7fbc381614d0 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 >> v2:172.21.15.164:3300/0 conn(0x7fbc380b1e40 msgr2=0x7fbc38168b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc2ffff640 1 --2- 172.21.15.59:0/3750401324 >> v2:172.21.15.164:3300/0 conn(0x7fbc380b1e40 0x7fbc38168b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 >> v1:172.21.15.164:6789/0 conn(0x7fbc380d4860 legacy=0x7fbc380d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc380cf400 con 0x7fbc381614d0 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fbc34001670 con 0x7fbc381614d0 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 >> v1:172.21.15.59:6789/0 conn(0x7fbc381614d0 legacy=0x7fbc380d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.971+0000 7fbc2ffff640 1 --2- 172.21.15.59:0/3750401324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc24003a00 0x7fbc24003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc2ffff640 1 --2- 172.21.15.59:0/3750401324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc24004800 0x7fbc24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc2ffff640 1 --2- 172.21.15.59:0/3750401324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc24005230 0x7fbc2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380d4cf0 con 0x7fbc24005230 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2400c190 con 0x7fbc24003a00 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2400c390 con 0x7fbc24004800 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc3df52640 1 --2- 172.21.15.59:0/3750401324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc24005230 0x7fbc2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc3d751640 1 --2- 172.21.15.59:0/3750401324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc24003a00 0x7fbc24003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc3cf50640 1 --2- 172.21.15.59:0/3750401324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc24004800 0x7fbc24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc3df52640 1 -- 172.21.15.59:0/3750401324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc24004800 msgr2=0x7fbc24004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc3df52640 1 --2- 172.21.15.59:0/3750401324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc24004800 0x7fbc24004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc3df52640 1 -- 172.21.15.59:0/3750401324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc24003a00 msgr2=0x7fbc24003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc3df52640 1 --2- 172.21.15.59:0/3750401324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc24003a00 0x7fbc24003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.972+0000 7fbc3df52640 1 -- 172.21.15.59:0/3750401324 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc24003520 con 0x7fbc24005230 2024-03-06T07:23:11.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.973+0000 7fbc3df52640 1 -- 172.21.15.59:0/3750401324 reap_dead start 2024-03-06T07:23:11.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.973+0000 7fbc3df52640 1 --2- 172.21.15.59:0/3750401324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc24005230 0x7fbc2400b9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009fc0 tx=0x7fbc24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42071c20b99dd9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.973+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30009c10 con 0x7fbc24005230 2024-03-06T07:23:11.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.973+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc30003bd0 con 0x7fbc24005230 2024-03-06T07:23:11.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.973+0000 7fbc2ffff640 1 -- 172.21.15.59:0/3750401324 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3002c990 con 0x7fbc24005230 2024-03-06T07:23:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.974+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3750401324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc24005230 msgr2=0x7fbc2400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.974+0000 7fbc3f9b5640 1 --2- 172.21.15.59:0/3750401324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc24005230 0x7fbc2400b9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009fc0 tx=0x7fbc24005830 comp rx=0 tx=0).stop 2024-03-06T07:23:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.974+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3750401324 shutdown_connections 2024-03-06T07:23:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.974+0000 7fbc3f9b5640 1 --2- 172.21.15.59:0/3750401324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc24005230 0x7fbc2400b9c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.974+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3750401324 >> 172.21.15.59:0/3750401324 conn(0x7fbc380d3d40 msgr2=0x7fbc380d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.974+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3750401324 shutdown_connections 2024-03-06T07:23:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.974+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3750401324 wait complete. 2024-03-06T07:23:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.975+0000 7fbc3f9b5640 1 Processor -- start 2024-03-06T07:23:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.975+0000 7fbc3f9b5640 1 -- start start 2024-03-06T07:23:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3f9b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc24005230 0x7fbc3814d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3f9b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc380d1910 0x7fbc380d1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3f9b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3814fda0 0x7fbc38150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3f9b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380cfc60 con 0x7fbc3814fda0 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3f9b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380cf980 con 0x7fbc380d1910 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3f9b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc380b0910 con 0x7fbc24005230 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3df52640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3814fda0 0x7fbc38150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3df52640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3814fda0 0x7fbc38150220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34992/0 (socket says 172.21.15.59:34992) 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3df52640 1 -- 172.21.15.59:0/3633572499 learned_addr learned my addr 172.21.15.59:0/3633572499 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3d751640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc24005230 0x7fbc3814d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3cf50640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc380d1910 0x7fbc380d1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3df52640 1 -- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc24005230 msgr2=0x7fbc3814d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3df52640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc24005230 0x7fbc3814d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3df52640 1 -- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc380d1910 msgr2=0x7fbc380d1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3df52640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc380d1910 0x7fbc380d1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3df52640 1 -- 172.21.15.59:0/3633572499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3814d6b0 con 0x7fbc3814fda0 2024-03-06T07:23:11.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc3df52640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3814fda0 0x7fbc38150220 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbc3002c600 tx=0x7fbc30009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.976+0000 7fbc2dffb640 1 -- 172.21.15.59:0/3633572499 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3002ce10 con 0x7fbc3814fda0 2024-03-06T07:23:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.977+0000 7fbc2dffb640 1 -- 172.21.15.59:0/3633572499 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc30046070 con 0x7fbc3814fda0 2024-03-06T07:23:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.977+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3633572499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc381f3a60 con 0x7fbc3814fda0 2024-03-06T07:23:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.977+0000 7fbc2dffb640 1 -- 172.21.15.59:0/3633572499 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30035ae0 con 0x7fbc3814fda0 2024-03-06T07:23:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.977+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3633572499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc381f4000 con 0x7fbc3814fda0 2024-03-06T07:23:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.978+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3633572499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc380d22d0 con 0x7fbc3814fda0 2024-03-06T07:23:11.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.978+0000 7fbc2dffb640 1 -- 172.21.15.59:0/3633572499 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc3003d040 con 0x7fbc3814fda0 2024-03-06T07:23:11.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.979+0000 7fbc2dffb640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc1c06b4d0 0x7fbc1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.979+0000 7fbc2dffb640 1 -- 172.21.15.59:0/3633572499 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc300ac590 con 0x7fbc3814fda0 2024-03-06T07:23:11.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.979+0000 7fbc3d751640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc1c06b4d0 0x7fbc1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.979+0000 7fbc3d751640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc1c06b4d0 0x7fbc1c06d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbc34001c00 tx=0x7fbc34001b90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:11.987+0000 7fbc2dffb640 1 -- 172.21.15.59:0/3633572499 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc3814d6b0 con 0x7fbc3814fda0 2024-03-06T07:23:12.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.168+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3633572499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbc38068ff0 con 0x7fbc3814fda0 2024-03-06T07:23:12.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.168+0000 7fbc2dffb640 1 -- 172.21.15.59:0/3633572499 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbc3007aec0 con 0x7fbc3814fda0 2024-03-06T07:23:12.170 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:23:12.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc1c06b4d0 msgr2=0x7fbc1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc1c06b4d0 0x7fbc1c06d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbc34001c00 tx=0x7fbc34001b90 comp rx=0 tx=0).stop 2024-03-06T07:23:12.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3633572499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3814fda0 msgr2=0x7fbc38150220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3814fda0 0x7fbc38150220 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbc3002c600 tx=0x7fbc30009750 comp rx=0 tx=0).stop 2024-03-06T07:23:12.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3633572499 shutdown_connections 2024-03-06T07:23:12.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbc1c06b4d0 0x7fbc1c06d990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbc3814fda0 0x7fbc38150220 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc380d1910 0x7fbc380d1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 --2- 172.21.15.59:0/3633572499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc24005230 0x7fbc3814d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3633572499 >> 172.21.15.59:0/3633572499 conn(0x7fbc380d3d40 msgr2=0x7fbc380d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3633572499 shutdown_connections 2024-03-06T07:23:12.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.170+0000 7fbc3f9b5640 1 -- 172.21.15.59:0/3633572499 wait complete. 2024-03-06T07:23:12.183 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:12.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.283+0000 7ff86c634640 1 Processor -- start 2024-03-06T07:23:12.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.283+0000 7ff86c634640 1 -- start start 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.284+0000 7ff86c634640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff8640b4fe0 0x7ff8640b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.284+0000 7ff86c634640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86415d510 con 0x7ff86415ba50 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.284+0000 7ff86c634640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86415d780 con 0x7ff8640b4c00 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.284+0000 7ff86c634640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86415e0b0 con 0x7ff8640b4fe0 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.284+0000 7ff86abd1640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff86415ba50 0x7ff86415bec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36868/0 (socket says 172.21.15.59:36868) 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.284+0000 7ff86abd1640 1 -- 172.21.15.59:0/3815897313 learned_addr learned my addr 172.21.15.59:0/3815897313 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.284+0000 7ff869bcf640 1 --2- 172.21.15.59:0/3815897313 >> v2:172.21.15.164:3300/0 conn(0x7ff8640b4fe0 0x7ff8640b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.285+0000 7ff8693ce640 1 -- 172.21.15.59:0/3815897313 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198754017 0 0) 0x7ff86415d510 con 0x7ff86415ba50 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.285+0000 7ff8693ce640 1 -- 172.21.15.59:0/3815897313 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff850003520 con 0x7ff86415ba50 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.285+0000 7ff869bcf640 1 -- 172.21.15.59:0/3815897313 >> v1:172.21.15.164:6789/0 conn(0x7ff8640b4c00 legacy=0x7ff8640b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.285+0000 7ff8693ce640 1 -- 172.21.15.59:0/3815897313 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2868224222 0 0) 0x7ff850003520 con 0x7ff86415ba50 2024-03-06T07:23:12.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.285+0000 7ff869bcf640 1 -- 172.21.15.59:0/3815897313 >> v1:172.21.15.59:6789/0 conn(0x7ff86415ba50 legacy=0x7ff86415bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.285+0000 7ff869bcf640 1 -- 172.21.15.59:0/3815897313 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86415da60 con 0x7ff8640b4fe0 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.286+0000 7ff869bcf640 1 --2- 172.21.15.59:0/3815897313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8640b4fe0 0x7ff8640b2fb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff8540097a0 tx=0x7ff85402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=276132dd9f20b838 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.286+0000 7ff8693ce640 1 -- 172.21.15.59:0/3815897313 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854037070 con 0x7ff8640b4fe0 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.286+0000 7ff8693ce640 1 -- 172.21.15.59:0/3815897313 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff854030510 con 0x7ff8640b4fe0 2024-03-06T07:23:12.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.286+0000 7ff8693ce640 1 -- 172.21.15.59:0/3815897313 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c0019f0 con 0x7ff8640b4fe0 2024-03-06T07:23:12.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.286+0000 7ff86c634640 1 -- 172.21.15.59:0/3815897313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8640b4fe0 msgr2=0x7ff8640b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.286+0000 7ff86c634640 1 --2- 172.21.15.59:0/3815897313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8640b4fe0 0x7ff8640b2fb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff8540097a0 tx=0x7ff85402b710 comp rx=0 tx=0).stop 2024-03-06T07:23:12.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.286+0000 7ff86c634640 1 -- 172.21.15.59:0/3815897313 shutdown_connections 2024-03-06T07:23:12.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.286+0000 7ff86c634640 1 --2- 172.21.15.59:0/3815897313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8640b4fe0 0x7ff8640b2fb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.286+0000 7ff86c634640 1 -- 172.21.15.59:0/3815897313 >> 172.21.15.59:0/3815897313 conn(0x7ff86409a9f0 msgr2=0x7ff86409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.286+0000 7ff86c634640 1 -- 172.21.15.59:0/3815897313 shutdown_connections 2024-03-06T07:23:12.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.287+0000 7ff86c634640 1 -- 172.21.15.59:0/3815897313 wait complete. 2024-03-06T07:23:12.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.288+0000 7ff86c634640 1 Processor -- start 2024-03-06T07:23:12.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.288+0000 7ff86c634640 1 -- start start 2024-03-06T07:23:12.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.288+0000 7ff86c634640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8640b4c00 0x7ff864154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.288+0000 7ff86c634640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8640b4fe0 0x7ff864156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.288+0000 7ff86c634640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff86415ba50 0x7ff8641534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.288+0000 7ff86c634640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86415f6d0 con 0x7ff8640b4c00 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.288+0000 7ff86c634640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86415f550 con 0x7ff86415ba50 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.288+0000 7ff86c634640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86415c5d0 con 0x7ff8640b4fe0 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.288+0000 7ff86a3d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8640b4c00 0x7ff864154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.288+0000 7ff86a3d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8640b4c00 0x7ff864154c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35004/0 (socket says 172.21.15.59:35004) 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.288+0000 7ff86a3d0640 1 -- 172.21.15.59:0/732008487 learned_addr learned my addr 172.21.15.59:0/732008487 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff86a3d0640 1 -- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8640b4fe0 msgr2=0x7ff864156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff869bcf640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8640b4fe0 0x7ff864156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff86abd1640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff86415ba50 0x7ff8641534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff86a3d0640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8640b4fe0 0x7ff864156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff86a3d0640 1 -- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff86415ba50 msgr2=0x7ff8641534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff86a3d0640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff86415ba50 0x7ff8641534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff86a3d0640 1 -- 172.21.15.59:0/732008487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8641539e0 con 0x7ff8640b4c00 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff869bcf640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8640b4fe0 0x7ff864156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff86abd1640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff86415ba50 0x7ff8641534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff86a3d0640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8640b4c00 0x7ff864154c20 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff86000c780 tx=0x7ff86000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff85affd640 1 -- 172.21.15.59:0/732008487 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff860017070 con 0x7ff8640b4c00 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff85affd640 1 -- 172.21.15.59:0/732008487 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff860005290 con 0x7ff8640b4c00 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff86c634640 1 -- 172.21.15.59:0/732008487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff864156ca0 con 0x7ff8640b4c00 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff85affd640 1 -- 172.21.15.59:0/732008487 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86000a880 con 0x7ff8640b4c00 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.289+0000 7ff86c634640 1 -- 172.21.15.59:0/732008487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff864157260 con 0x7ff8640b4c00 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.290+0000 7ff86c634640 1 -- 172.21.15.59:0/732008487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86404f330 con 0x7ff8640b4c00 2024-03-06T07:23:12.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.291+0000 7ff85affd640 1 -- 172.21.15.59:0/732008487 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff86001a070 con 0x7ff8640b4c00 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.291+0000 7ff85affd640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff84806b500 0x7ff84806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.291+0000 7ff85affd640 1 -- 172.21.15.59:0/732008487 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff860089710 con 0x7ff8640b4c00 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.292+0000 7ff869bcf640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff84806b500 0x7ff84806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.292+0000 7ff869bcf640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff84806b500 0x7ff84806d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff854034000 tx=0x7ff854008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.296+0000 7ff85affd640 1 -- 172.21.15.59:0/732008487 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff860058590 con 0x7ff8640b4c00 2024-03-06T07:23:12.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.492+0000 7ff86c634640 1 -- 172.21.15.59:0/732008487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff86406fcf0 con 0x7ff8640b4c00 2024-03-06T07:23:12.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.492+0000 7ff85affd640 1 -- 172.21.15.59:0/732008487 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff860057e90 con 0x7ff8640b4c00 2024-03-06T07:23:12.494 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:23:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 -- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff84806b500 msgr2=0x7ff84806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff84806b500 0x7ff84806d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff854034000 tx=0x7ff854008030 comp rx=0 tx=0).stop 2024-03-06T07:23:12.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 -- 172.21.15.59:0/732008487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8640b4c00 msgr2=0x7ff864154c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8640b4c00 0x7ff864154c20 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff86000c780 tx=0x7ff86000cc40 comp rx=0 tx=0).stop 2024-03-06T07:23:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 -- 172.21.15.59:0/732008487 shutdown_connections 2024-03-06T07:23:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff84806b500 0x7ff84806d9c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff86415ba50 0x7ff8641534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8640b4fe0 0x7ff864156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 --2- 172.21.15.59:0/732008487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8640b4c00 0x7ff864154c20 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 -- 172.21.15.59:0/732008487 >> 172.21.15.59:0/732008487 conn(0x7ff86409a9f0 msgr2=0x7ff86406ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 -- 172.21.15.59:0/732008487 shutdown_connections 2024-03-06T07:23:12.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:12.494+0000 7ff86c634640 1 -- 172.21.15.59:0/732008487 wait complete. 2024-03-06T07:23:12.506 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:23:12.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:12.507 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:12.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[52707]: fuse finished with error 0 and tester_r 0 2024-03-06T07:23:12.534 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:23:12.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:12.534 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:23:12.694 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:23:12.694 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:12.712 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:12.712 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:23:12.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:12.712 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:23:12.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:23:12.769 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:23:12.769 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:12.785 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:12.785 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:23:12.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:12.785 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:23:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:12.840 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:23:12.840 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:23:12.840 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:12.857 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:12.857 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:23:12.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:12.857 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:23:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:12.914 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:23:12.914 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.013+0000 7fad7f0e1640 1 Processor -- start 2024-03-06T07:23:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.013+0000 7fad7f0e1640 1 -- start start 2024-03-06T07:23:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.014+0000 7fad7f0e1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fad7815dfe0 0x7fad780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.014+0000 7fad7f0e1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fad780cf830 0x7fad780d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.014+0000 7fad7f0e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad780d27f0 0x7fad780d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.014+0000 7fad7f0e1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad780b31e0 con 0x7fad7815dfe0 2024-03-06T07:23:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.014+0000 7fad7f0e1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad780b3d60 con 0x7fad780cf830 2024-03-06T07:23:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.014+0000 7fad7f0e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad780b3430 con 0x7fad780d27f0 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.014+0000 7fad7ce7d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fad7815dfe0 0x7fad780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.014+0000 7fad7ce7d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fad7815dfe0 0x7fad780cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35008/0 (socket says 172.21.15.59:35008) 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.014+0000 7fad7ce7d640 1 -- 172.21.15.59:0/334946268 learned_addr learned my addr 172.21.15.59:0/334946268 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.014+0000 7fad7d67e640 1 --2- 172.21.15.59:0/334946268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad780d27f0 0x7fad780d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.014+0000 7fad6ffff640 1 --2- 172.21.15.59:0/334946268 >> v2:172.21.15.164:3300/0 conn(0x7fad780cf830 0x7fad780d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.015+0000 7fad7ce7d640 1 -- 172.21.15.59:0/334946268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad780d27f0 msgr2=0x7fad780d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.015+0000 7fad7ce7d640 1 --2- 172.21.15.59:0/334946268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad780d27f0 0x7fad780d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.015+0000 7fad7ce7d640 1 -- 172.21.15.59:0/334946268 >> v2:172.21.15.164:3300/0 conn(0x7fad780cf830 msgr2=0x7fad780d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.015+0000 7fad7ce7d640 1 --2- 172.21.15.59:0/334946268 >> v2:172.21.15.164:3300/0 conn(0x7fad780cf830 0x7fad780d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.015+0000 7fad7ce7d640 1 -- 172.21.15.59:0/334946268 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad780b3710 con 0x7fad7815dfe0 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.015+0000 7fad7ce7d640 1 --2- 172.21.15.59:0/334946268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad7815dfe0 0x7fad780cf2f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fad780d16b0 tx=0x7fad7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400068ba35b6bc03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.015+0000 7fad6f7fe640 1 -- 172.21.15.59:0/334946268 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74035020 con 0x7fad7815dfe0 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.016+0000 7fad6f7fe640 1 -- 172.21.15.59:0/334946268 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad74031870 con 0x7fad7815dfe0 2024-03-06T07:23:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.016+0000 7fad6f7fe640 1 -- 172.21.15.59:0/334946268 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74031c40 con 0x7fad7815dfe0 2024-03-06T07:23:13.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.016+0000 7fad7f0e1640 1 -- 172.21.15.59:0/334946268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad7815dfe0 msgr2=0x7fad780cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.016+0000 7fad7f0e1640 1 --2- 172.21.15.59:0/334946268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad7815dfe0 0x7fad780cf2f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fad780d16b0 tx=0x7fad7402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:13.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.016+0000 7fad7f0e1640 1 -- 172.21.15.59:0/334946268 shutdown_connections 2024-03-06T07:23:13.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.016+0000 7fad7f0e1640 1 --2- 172.21.15.59:0/334946268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad780d27f0 0x7fad780d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.016+0000 7fad7f0e1640 1 --2- 172.21.15.59:0/334946268 >> v2:172.21.15.164:3300/0 conn(0x7fad780cf830 0x7fad780d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.016+0000 7fad7f0e1640 1 --2- 172.21.15.59:0/334946268 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad7815dfe0 0x7fad780cf2f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.016+0000 7fad7f0e1640 1 -- 172.21.15.59:0/334946268 >> 172.21.15.59:0/334946268 conn(0x7fad780d3cd0 msgr2=0x7fad780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.016+0000 7fad7f0e1640 1 -- 172.21.15.59:0/334946268 shutdown_connections 2024-03-06T07:23:13.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.016+0000 7fad7f0e1640 1 -- 172.21.15.59:0/334946268 wait complete. 2024-03-06T07:23:13.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.018+0000 7fad7f0e1640 1 Processor -- start 2024-03-06T07:23:13.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.018+0000 7fad7f0e1640 1 -- start start 2024-03-06T07:23:13.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.019+0000 7fad7f0e1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad780cf830 0x7fad7814af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.019+0000 7fad7f0e1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad780d27f0 0x7fad780cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.019+0000 7fad7f0e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad7815dfe0 0x7fad781f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.019+0000 7fad7f0e1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad780d1df0 con 0x7fad780cf830 2024-03-06T07:23:13.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.019+0000 7fad7f0e1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad780d1c70 con 0x7fad780d27f0 2024-03-06T07:23:13.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.019+0000 7fad7f0e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7815e3c0 con 0x7fad7815dfe0 2024-03-06T07:23:13.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7ce7d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad780cf830 0x7fad7814af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7ce7d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad780cf830 0x7fad7814af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35012/0 (socket says 172.21.15.59:35012) 2024-03-06T07:23:13.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7ce7d640 1 -- 172.21.15.59:0/2612482621 learned_addr learned my addr 172.21.15.59:0/2612482621 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:13.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7d67e640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad7815dfe0 0x7fad781f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad6ffff640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad780d27f0 0x7fad780cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7ce7d640 1 -- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad7815dfe0 msgr2=0x7fad781f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7ce7d640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad7815dfe0 0x7fad781f3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7ce7d640 1 -- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad780d27f0 msgr2=0x7fad780cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7ce7d640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad780d27f0 0x7fad780cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7ce7d640 1 -- 172.21.15.59:0/2612482621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad781f44c0 con 0x7fad780cf830 2024-03-06T07:23:13.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7d67e640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad7815dfe0 0x7fad781f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:13.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad6ffff640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad780d27f0 0x7fad780cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:13.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7ce7d640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad780cf830 0x7fad7814af70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fad7402b950 tx=0x7fad740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad6d7fa640 1 -- 172.21.15.59:0/2612482621 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74040070 con 0x7fad780cf830 2024-03-06T07:23:13.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad6d7fa640 1 -- 172.21.15.59:0/2612482621 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad74030d00 con 0x7fad780cf830 2024-03-06T07:23:13.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad6d7fa640 1 -- 172.21.15.59:0/2612482621 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74035040 con 0x7fad780cf830 2024-03-06T07:23:13.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7f0e1640 1 -- 172.21.15.59:0/2612482621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad781f47a0 con 0x7fad780cf830 2024-03-06T07:23:13.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.020+0000 7fad7f0e1640 1 -- 172.21.15.59:0/2612482621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad780b8a10 con 0x7fad780cf830 2024-03-06T07:23:13.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.021+0000 7fad7f0e1640 1 -- 172.21.15.59:0/2612482621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7804f330 con 0x7fad780cf830 2024-03-06T07:23:13.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.022+0000 7fad6d7fa640 1 -- 172.21.15.59:0/2612482621 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad7404b060 con 0x7fad780cf830 2024-03-06T07:23:13.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.022+0000 7fad6d7fa640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 0x7fad6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.022+0000 7fad6d7fa640 1 -- 172.21.15.59:0/2612482621 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad740abe50 con 0x7fad780cf830 2024-03-06T07:23:13.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.022+0000 7fad6ffff640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 0x7fad6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.023+0000 7fad6ffff640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 0x7fad6406d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fad781f1810 tx=0x7fad68008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.031+0000 7fad6d7fa640 1 -- 172.21.15.59:0/2612482621 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad7407ad60 con 0x7fad780cf830 2024-03-06T07:23:13.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.249+0000 7fad7f0e1640 1 -- 172.21.15.59:0/2612482621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad7806b7f0 con 0x7fad780cf830 2024-03-06T07:23:13.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.249+0000 7fad6d7fa640 1 -- 172.21.15.59:0/2612482621 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fad7407a660 con 0x7fad780cf830 2024-03-06T07:23:13.251 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:13.251 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:23:13.252 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:23:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 -- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 msgr2=0x7fad6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 0x7fad6406d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fad781f1810 tx=0x7fad68008040 comp rx=0 tx=0).stop 2024-03-06T07:23:13.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 -- 172.21.15.59:0/2612482621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad780cf830 msgr2=0x7fad7814af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad780cf830 0x7fad7814af70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fad7402b950 tx=0x7fad740046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 -- 172.21.15.59:0/2612482621 shutdown_connections 2024-03-06T07:23:13.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 0x7fad6406d990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad7815dfe0 0x7fad781f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad780d27f0 0x7fad780cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 --2- 172.21.15.59:0/2612482621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad780cf830 0x7fad7814af70 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 -- 172.21.15.59:0/2612482621 >> 172.21.15.59:0/2612482621 conn(0x7fad780d3cd0 msgr2=0x7fad7806bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 -- 172.21.15.59:0/2612482621 shutdown_connections 2024-03-06T07:23:13.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.251+0000 7fad7f0e1640 1 -- 172.21.15.59:0/2612482621 wait complete. 2024-03-06T07:23:13.264 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.374+0000 7f9b3f1bd640 1 Processor -- start 2024-03-06T07:23:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.374+0000 7f9b3f1bd640 1 -- start start 2024-03-06T07:23:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b3f1bd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9b381424f0 0x7f9b380cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b3f1bd640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b380cc980 con 0x7f9b3806da40 2024-03-06T07:23:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b3f1bd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b380ccc60 con 0x7f9b380cd7a0 2024-03-06T07:23:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b3f1bd640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b380cd220 con 0x7f9b381424f0 2024-03-06T07:23:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b3cf59640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9b381424f0 0x7f9b380cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b3cf59640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9b381424f0 0x7f9b380cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35028/0 (socket says 172.21.15.59:35028) 2024-03-06T07:23:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b3cf59640 1 -- 172.21.15.59:0/464547341 learned_addr learned my addr 172.21.15.59:0/464547341 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b3d75a640 1 --1- 172.21.15.59:0/464547341 >> v1:172.21.15.59:6789/0 conn(0x7f9b3806da40 0x7f9b38071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36880/0 (socket says 172.21.15.59:36880) 2024-03-06T07:23:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b3cf59640 1 -- 172.21.15.59:0/464547341 >> v1:172.21.15.164:6789/0 conn(0x7f9b380cd7a0 legacy=0x7f9b3806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b3cf59640 1 -- 172.21.15.59:0/464547341 >> v1:172.21.15.59:6789/0 conn(0x7f9b3806da40 legacy=0x7f9b38071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b3cf59640 1 -- 172.21.15.59:0/464547341 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b380ccf40 con 0x7f9b381424f0 2024-03-06T07:23:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.375+0000 7f9b377fe640 1 -- 172.21.15.59:0/464547341 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992639899 0 0) 0x7f9b380cc980 con 0x7f9b3806da40 2024-03-06T07:23:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.376+0000 7f9b3cf59640 1 --2- 172.21.15.59:0/464547341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b381424f0 0x7f9b380cc2f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009e50 tx=0x7f9b3002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1109be146915c28e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.376+0000 7f9b377fe640 1 -- 172.21.15.59:0/464547341 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3002cb20 con 0x7f9b381424f0 2024-03-06T07:23:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.376+0000 7f9b377fe640 1 -- 172.21.15.59:0/464547341 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3002ccc0 con 0x7f9b381424f0 2024-03-06T07:23:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.376+0000 7f9b377fe640 1 -- 172.21.15.59:0/464547341 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30034610 con 0x7f9b381424f0 2024-03-06T07:23:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.377+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/464547341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b381424f0 msgr2=0x7f9b380cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.377+0000 7f9b3f1bd640 1 --2- 172.21.15.59:0/464547341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b381424f0 0x7f9b380cc2f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009e50 tx=0x7f9b3002bf80 comp rx=0 tx=0).stop 2024-03-06T07:23:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.377+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/464547341 shutdown_connections 2024-03-06T07:23:13.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.377+0000 7f9b3f1bd640 1 --2- 172.21.15.59:0/464547341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b381424f0 0x7f9b380cc2f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.377+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/464547341 >> 172.21.15.59:0/464547341 conn(0x7f9b380cf8a0 msgr2=0x7f9b380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.377+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/464547341 shutdown_connections 2024-03-06T07:23:13.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.377+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/464547341 wait complete. 2024-03-06T07:23:13.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.378+0000 7f9b3f1bd640 1 Processor -- start 2024-03-06T07:23:13.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.378+0000 7f9b3f1bd640 1 -- start start 2024-03-06T07:23:13.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b3f1bd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3806da40 0x7f9b38157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b3f1bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b380cd7a0 0x7f9b38158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b3f1bd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b381424f0 0x7f9b38159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b3f1bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3815d070 con 0x7f9b380cd7a0 2024-03-06T07:23:13.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b3f1bd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3815cef0 con 0x7f9b3806da40 2024-03-06T07:23:13.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b3f1bd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3815d1f0 con 0x7f9b381424f0 2024-03-06T07:23:13.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b380cd7a0 0x7f9b38158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b37fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b380cd7a0 0x7f9b38158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35042/0 (socket says 172.21.15.59:35042) 2024-03-06T07:23:13.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b37fff640 1 -- 172.21.15.59:0/3206460145 learned_addr learned my addr 172.21.15.59:0/3206460145 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:13.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b3cf59640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3806da40 0x7f9b38157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b37fff640 1 -- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b381424f0 msgr2=0x7f9b38159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b3d75a640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b381424f0 0x7f9b38159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b37fff640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b381424f0 0x7f9b38159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b37fff640 1 -- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3806da40 msgr2=0x7f9b38157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b37fff640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3806da40 0x7f9b38157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b37fff640 1 -- 172.21.15.59:0/3206460145 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b38140660 con 0x7f9b380cd7a0 2024-03-06T07:23:13.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.379+0000 7f9b3cf59640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3806da40 0x7f9b38157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:13.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.380+0000 7f9b37fff640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b380cd7a0 0x7f9b38158d50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00c430 tx=0x7f9b2c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.380+0000 7f9b357fa640 1 -- 172.21.15.59:0/3206460145 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c011790 con 0x7f9b380cd7a0 2024-03-06T07:23:13.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.380+0000 7f9b357fa640 1 -- 172.21.15.59:0/3206460145 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c017070 con 0x7f9b380cd7a0 2024-03-06T07:23:13.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.380+0000 7f9b357fa640 1 -- 172.21.15.59:0/3206460145 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c011bb0 con 0x7f9b380cd7a0 2024-03-06T07:23:13.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.380+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/3206460145 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b38140940 con 0x7f9b380cd7a0 2024-03-06T07:23:13.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.380+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/3206460145 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b380b34d0 con 0x7f9b380cd7a0 2024-03-06T07:23:13.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.381+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/3206460145 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b38140e60 con 0x7f9b380cd7a0 2024-03-06T07:23:13.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.382+0000 7f9b357fa640 1 -- 172.21.15.59:0/3206460145 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b2c01c460 con 0x7f9b380cd7a0 2024-03-06T07:23:13.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.382+0000 7f9b357fa640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b2406b4d0 0x7f9b2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.382+0000 7f9b357fa640 1 -- 172.21.15.59:0/3206460145 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b2c08a190 con 0x7f9b380cd7a0 2024-03-06T07:23:13.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.382+0000 7f9b3cf59640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b2406b4d0 0x7f9b2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.383+0000 7f9b3cf59640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b2406b4d0 0x7f9b2406d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9b30035000 tx=0x7f9b30030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.386+0000 7f9b357fa640 1 -- 172.21.15.59:0/3206460145 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b38140e60 con 0x7f9b380cd7a0 2024-03-06T07:23:13.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.569+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/3206460145 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b3804f330 con 0x7f9b380cd7a0 2024-03-06T07:23:13.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.570+0000 7f9b357fa640 1 -- 172.21.15.59:0/3206460145 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f9b2c058f70 con 0x7f9b380cd7a0 2024-03-06T07:23:13.572 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:13.572 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":33,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:52.226312+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:48.848650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:49.545360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:13.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b2406b4d0 msgr2=0x7f9b2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b2406b4d0 0x7f9b2406d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9b30035000 tx=0x7f9b30030000 comp rx=0 tx=0).stop 2024-03-06T07:23:13.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/3206460145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b380cd7a0 msgr2=0x7f9b38158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b380cd7a0 0x7f9b38158d50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00c430 tx=0x7f9b2c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/3206460145 shutdown_connections 2024-03-06T07:23:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b2406b4d0 0x7f9b2406d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b381424f0 0x7f9b38159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b380cd7a0 0x7f9b38158d50 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 --2- 172.21.15.59:0/3206460145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3806da40 0x7f9b38157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/3206460145 >> 172.21.15.59:0/3206460145 conn(0x7f9b380cf8a0 msgr2=0x7f9b3815a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/3206460145 shutdown_connections 2024-03-06T07:23:13.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.572+0000 7f9b3f1bd640 1 -- 172.21.15.59:0/3206460145 wait complete. 2024-03-06T07:23:13.585 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:48.848650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:49.545360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:13.585 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:13.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.688+0000 7f57ea5a0640 1 Processor -- start 2024-03-06T07:23:13.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.688+0000 7f57ea5a0640 1 -- start start 2024-03-06T07:23:13.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57ea5a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f57e406afb0 0x7f57e406b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57ea5a0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e406b8d0 con 0x7f57e406de40 2024-03-06T07:23:13.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57ea5a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e40afb50 con 0x7f57e40af6c0 2024-03-06T07:23:13.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57ea5a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e40afe30 con 0x7f57e406afb0 2024-03-06T07:23:13.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57e9d9f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f57e406de40 0x7f57e406e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54338/0 (socket says 172.21.15.59:54338) 2024-03-06T07:23:13.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57e9d9f640 1 -- 172.21.15.59:0/541563645 learned_addr learned my addr 172.21.15.59:0/541563645 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:13.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57e959e640 1 --2- 172.21.15.59:0/541563645 >> v2:172.21.15.164:3300/0 conn(0x7f57e406afb0 0x7f57e406b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57dbfff640 1 -- 172.21.15.59:0/541563645 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197325903 0 0) 0x7f57e406b8d0 con 0x7f57e406de40 2024-03-06T07:23:13.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57dbfff640 1 -- 172.21.15.59:0/541563645 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d0003520 con 0x7f57e406de40 2024-03-06T07:23:13.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57dbfff640 1 -- 172.21.15.59:0/541563645 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 711120397 0 0) 0x7f57d0003520 con 0x7f57e406de40 2024-03-06T07:23:13.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57e959e640 1 -- 172.21.15.59:0/541563645 >> v1:172.21.15.164:6789/0 conn(0x7f57e40af6c0 legacy=0x7f57e40cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57e959e640 1 -- 172.21.15.59:0/541563645 >> v1:172.21.15.59:6789/0 conn(0x7f57e406de40 legacy=0x7f57e406e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.689+0000 7f57e959e640 1 -- 172.21.15.59:0/541563645 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40b0110 con 0x7f57e406afb0 2024-03-06T07:23:13.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.690+0000 7f57e959e640 1 --2- 172.21.15.59:0/541563645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57e406afb0 0x7f57e406b390 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f57e00097a0 tx=0x7f57e002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c31f0f1a4bb80426 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.690+0000 7f57dbfff640 1 -- 172.21.15.59:0/541563645 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0038070 con 0x7f57e406afb0 2024-03-06T07:23:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.691+0000 7f57dbfff640 1 -- 172.21.15.59:0/541563645 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e0031500 con 0x7f57e406afb0 2024-03-06T07:23:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.691+0000 7f57dbfff640 1 -- 172.21.15.59:0/541563645 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc0019f0 con 0x7f57e406afb0 2024-03-06T07:23:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.691+0000 7f57ea5a0640 1 -- 172.21.15.59:0/541563645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57e406afb0 msgr2=0x7f57e406b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.691+0000 7f57ea5a0640 1 --2- 172.21.15.59:0/541563645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57e406afb0 0x7f57e406b390 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f57e00097a0 tx=0x7f57e002b820 comp rx=0 tx=0).stop 2024-03-06T07:23:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.691+0000 7f57ea5a0640 1 -- 172.21.15.59:0/541563645 shutdown_connections 2024-03-06T07:23:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.691+0000 7f57ea5a0640 1 --2- 172.21.15.59:0/541563645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57e406afb0 0x7f57e406b390 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.691+0000 7f57ea5a0640 1 -- 172.21.15.59:0/541563645 >> 172.21.15.59:0/541563645 conn(0x7f57e406c0c0 msgr2=0x7f57e406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.691+0000 7f57ea5a0640 1 -- 172.21.15.59:0/541563645 shutdown_connections 2024-03-06T07:23:13.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.691+0000 7f57ea5a0640 1 -- 172.21.15.59:0/541563645 wait complete. 2024-03-06T07:23:13.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.693+0000 7f57ea5a0640 1 Processor -- start 2024-03-06T07:23:13.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.693+0000 7f57ea5a0640 1 -- start start 2024-03-06T07:23:13.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.694+0000 7f57ea5a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57e406afb0 0x7f57e41f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.694+0000 7f57ea5a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57e406de40 0x7f57e4159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.694+0000 7f57ea5a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57e41f2570 0x7f57e4159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.694+0000 7f57ea5a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e406e9e0 con 0x7f57e41f2570 2024-03-06T07:23:13.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.694+0000 7f57ea5a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e40cd7d0 con 0x7f57e406afb0 2024-03-06T07:23:13.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.694+0000 7f57ea5a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e406eb60 con 0x7f57e406de40 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.694+0000 7f57e9d9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57e41f2570 0x7f57e4159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.694+0000 7f57e9d9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57e41f2570 0x7f57e4159860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39994/0 (socket says 172.21.15.59:39994) 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.694+0000 7f57e9d9f640 1 -- 172.21.15.59:0/3232888234 learned_addr learned my addr 172.21.15.59:0/3232888234 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.694+0000 7f57e8d9d640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57e406de40 0x7f57e4159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.694+0000 7f57e959e640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57e406afb0 0x7f57e41f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.695+0000 7f57e9d9f640 1 -- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57e406de40 msgr2=0x7f57e4159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.695+0000 7f57e9d9f640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57e406de40 0x7f57e4159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.695+0000 7f57e9d9f640 1 -- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57e406afb0 msgr2=0x7f57e41f4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.695+0000 7f57e9d9f640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57e406afb0 0x7f57e41f4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.695+0000 7f57e9d9f640 1 -- 172.21.15.59:0/3232888234 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40aa690 con 0x7f57e41f2570 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.695+0000 7f57e9d9f640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57e41f2570 0x7f57e4159860 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0018c0 tx=0x7f57dc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.695+0000 7f57d9ffb640 1 -- 172.21.15.59:0/3232888234 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc012400 con 0x7f57e41f2570 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.695+0000 7f57d9ffb640 1 -- 172.21.15.59:0/3232888234 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc0125c0 con 0x7f57e41f2570 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.695+0000 7f57d9ffb640 1 -- 172.21.15.59:0/3232888234 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc0129b0 con 0x7f57e41f2570 2024-03-06T07:23:13.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.695+0000 7f57ea5a0640 1 -- 172.21.15.59:0/3232888234 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e41f2230 con 0x7f57e41f2570 2024-03-06T07:23:13.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.695+0000 7f57ea5a0640 1 -- 172.21.15.59:0/3232888234 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e40aac10 con 0x7f57e41f2570 2024-03-06T07:23:13.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.696+0000 7f57ea5a0640 1 -- 172.21.15.59:0/3232888234 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e4068f10 con 0x7f57e41f2570 2024-03-06T07:23:13.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.696+0000 7f57d9ffb640 1 -- 172.21.15.59:0/3232888234 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57dc012b50 con 0x7f57e41f2570 2024-03-06T07:23:13.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.697+0000 7f57d9ffb640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57c806b4d0 0x7f57c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.697+0000 7f57d9ffb640 1 -- 172.21.15.59:0/3232888234 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57dc08aa60 con 0x7f57e41f2570 2024-03-06T07:23:13.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.697+0000 7f57e959e640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57c806b4d0 0x7f57c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.698+0000 7f57e959e640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57c806b4d0 0x7f57c806d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f57e0035000 tx=0x7f57e0008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.701+0000 7f57d9ffb640 1 -- 172.21.15.59:0/3232888234 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57dc0182b0 con 0x7f57e41f2570 2024-03-06T07:23:13.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.880+0000 7f57ea5a0640 1 -- 172.21.15.59:0/3232888234 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f57e41f46c0 con 0x7f57e41f2570 2024-03-06T07:23:13.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.881+0000 7f57d9ffb640 1 -- 172.21.15.59:0/3232888234 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7f57dc059a30 con 0x7f57e41f2570 2024-03-06T07:23:13.883 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:23:13.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.883+0000 7f57ea5a0640 1 -- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57c806b4d0 msgr2=0x7f57c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.883+0000 7f57ea5a0640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57c806b4d0 0x7f57c806d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f57e0035000 tx=0x7f57e0008030 comp rx=0 tx=0).stop 2024-03-06T07:23:13.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.883+0000 7f57ea5a0640 1 -- 172.21.15.59:0/3232888234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57e41f2570 msgr2=0x7f57e4159860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.883+0000 7f57ea5a0640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57e41f2570 0x7f57e4159860 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0018c0 tx=0x7f57dc00a430 comp rx=0 tx=0).stop 2024-03-06T07:23:13.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.883+0000 7f57ea5a0640 1 -- 172.21.15.59:0/3232888234 shutdown_connections 2024-03-06T07:23:13.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.883+0000 7f57ea5a0640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57c806b4d0 0x7f57c806d990 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.883+0000 7f57ea5a0640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57e41f2570 0x7f57e4159860 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.883+0000 7f57ea5a0640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57e406de40 0x7f57e4159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.883+0000 7f57ea5a0640 1 --2- 172.21.15.59:0/3232888234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57e406afb0 0x7f57e41f4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.883+0000 7f57ea5a0640 1 -- 172.21.15.59:0/3232888234 >> 172.21.15.59:0/3232888234 conn(0x7f57e406c0c0 msgr2=0x7f57e40b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.884+0000 7f57ea5a0640 1 -- 172.21.15.59:0/3232888234 shutdown_connections 2024-03-06T07:23:13.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:13.884+0000 7f57ea5a0640 1 -- 172.21.15.59:0/3232888234 wait complete. 2024-03-06T07:23:13.897 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:14.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.001+0000 7fea1d7eb640 1 Processor -- start 2024-03-06T07:23:14.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.001+0000 7fea1d7eb640 1 -- start start 2024-03-06T07:23:14.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.001+0000 7fea1d7eb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea1815def0 0x7fea180b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.001+0000 7fea1d7eb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1806d030 con 0x7fea1806cab0 2024-03-06T07:23:14.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.001+0000 7fea1d7eb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1806d310 con 0x7fea180cea50 2024-03-06T07:23:14.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.001+0000 7fea1d7eb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea180cf1c0 con 0x7fea1815def0 2024-03-06T07:23:14.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.002+0000 7fea167fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fea1806cab0 0x7fea180ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54344/0 (socket says 172.21.15.59:54344) 2024-03-06T07:23:14.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.002+0000 7fea167fc640 1 -- 172.21.15.59:0/1115938850 learned_addr learned my addr 172.21.15.59:0/1115938850 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:14.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.002+0000 7fea16ffd640 1 --2- 172.21.15.59:0/1115938850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea1815def0 0x7fea180b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.002+0000 7fea15ffb640 1 -- 172.21.15.59:0/1115938850 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313580769 0 0) 0x7fea1806d030 con 0x7fea1806cab0 2024-03-06T07:23:14.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.002+0000 7fea15ffb640 1 -- 172.21.15.59:0/1115938850 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea00003520 con 0x7fea1806cab0 2024-03-06T07:23:14.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.002+0000 7fea15ffb640 1 -- 172.21.15.59:0/1115938850 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365569090 0 0) 0x7fea1806d310 con 0x7fea180cea50 2024-03-06T07:23:14.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.002+0000 7fea15ffb640 1 -- 172.21.15.59:0/1115938850 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea1806d030 con 0x7fea180cea50 2024-03-06T07:23:14.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.002+0000 7fea16ffd640 1 -- 172.21.15.59:0/1115938850 >> v1:172.21.15.164:6789/0 conn(0x7fea180cea50 legacy=0x7fea180ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.002+0000 7fea16ffd640 1 -- 172.21.15.59:0/1115938850 >> v1:172.21.15.59:6789/0 conn(0x7fea1806cab0 legacy=0x7fea180ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.002+0000 7fea15ffb640 1 -- 172.21.15.59:0/1115938850 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1239841077 0 0) 0x7fea00003520 con 0x7fea1806cab0 2024-03-06T07:23:14.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.002+0000 7fea16ffd640 1 -- 172.21.15.59:0/1115938850 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea180cc110 con 0x7fea1815def0 2024-03-06T07:23:14.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.003+0000 7fea16ffd640 1 --2- 172.21.15.59:0/1115938850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea1815def0 0x7fea180b41b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0097d0 tx=0x7fea0c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cee8d512f464fdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.003+0000 7fea15ffb640 1 -- 172.21.15.59:0/1115938850 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c035020 con 0x7fea1815def0 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.003+0000 7fea15ffb640 1 -- 172.21.15.59:0/1115938850 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c031890 con 0x7fea1815def0 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.003+0000 7fea15ffb640 1 -- 172.21.15.59:0/1115938850 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c031c60 con 0x7fea1815def0 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.004+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1115938850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea1815def0 msgr2=0x7fea180b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.004+0000 7fea1d7eb640 1 --2- 172.21.15.59:0/1115938850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea1815def0 0x7fea180b41b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0097d0 tx=0x7fea0c02b930 comp rx=0 tx=0).stop 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.004+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1115938850 shutdown_connections 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.004+0000 7fea1d7eb640 1 --2- 172.21.15.59:0/1115938850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea1815def0 0x7fea180b41b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.004+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1115938850 >> 172.21.15.59:0/1115938850 conn(0x7fea180b3690 msgr2=0x7fea180b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.004+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1115938850 shutdown_connections 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.004+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1115938850 wait complete. 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.005+0000 7fea1d7eb640 1 Processor -- start 2024-03-06T07:23:14.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.005+0000 7fea1d7eb640 1 -- start start 2024-03-06T07:23:14.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea1d7eb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea1806cab0 0x7fea180c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea1d7eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea180cea50 0x7fea180c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea1d7eb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea1815def0 0x7fea180ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea1d7eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1815e2d0 con 0x7fea180cea50 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea1d7eb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea180cc760 con 0x7fea1806cab0 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea1d7eb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1815e450 con 0x7fea1815def0 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea167fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea180cea50 0x7fea180c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea167fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea180cea50 0x7fea180c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40006/0 (socket says 172.21.15.59:40006) 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea167fc640 1 -- 172.21.15.59:0/1086717855 learned_addr learned my addr 172.21.15.59:0/1086717855 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea16ffd640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea1806cab0 0x7fea180c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea177fe640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea1815def0 0x7fea180ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea167fc640 1 -- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea1815def0 msgr2=0x7fea180ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea167fc640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea1815def0 0x7fea180ca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea167fc640 1 -- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea1806cab0 msgr2=0x7fea180c78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea167fc640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea1806cab0 0x7fea180c78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea167fc640 1 -- 172.21.15.59:0/1086717855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea180c90d0 con 0x7fea180cea50 2024-03-06T07:23:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fea167fc640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea180cea50 0x7fea180c8b90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fea0800c450 tx=0x7fea0800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.006+0000 7fe9ff7fe640 1 -- 172.21.15.59:0/1086717855 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08017070 con 0x7fea180cea50 2024-03-06T07:23:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.007+0000 7fe9ff7fe640 1 -- 172.21.15.59:0/1086717855 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea08005d00 con 0x7fea180cea50 2024-03-06T07:23:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.007+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1086717855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea180c9370 con 0x7fea180cea50 2024-03-06T07:23:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.007+0000 7fe9ff7fe640 1 -- 172.21.15.59:0/1086717855 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea080072b0 con 0x7fea180cea50 2024-03-06T07:23:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.007+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1086717855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea180c9930 con 0x7fea180cea50 2024-03-06T07:23:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.007+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1086717855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1809fcf0 con 0x7fea180cea50 2024-03-06T07:23:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.008+0000 7fe9ff7fe640 1 -- 172.21.15.59:0/1086717855 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea0801a070 con 0x7fea180cea50 2024-03-06T07:23:14.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.008+0000 7fe9ff7fe640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fea0406b4d0 0x7fea0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.008+0000 7fe9ff7fe640 1 -- 172.21.15.59:0/1086717855 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea080894f0 con 0x7fea180cea50 2024-03-06T07:23:14.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.009+0000 7fea16ffd640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fea0406b4d0 0x7fea0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.009+0000 7fea16ffd640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fea0406b4d0 0x7fea0406d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fea0c036000 tx=0x7fea0c03d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.013+0000 7fe9ff7fe640 1 -- 172.21.15.59:0/1086717855 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea08057d20 con 0x7fea180cea50 2024-03-06T07:23:14.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.194+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1086717855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fea18068ff0 con 0x7fea180cea50 2024-03-06T07:23:14.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.194+0000 7fe9ff7fe640 1 -- 172.21.15.59:0/1086717855 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7fea0805bdc0 con 0x7fea180cea50 2024-03-06T07:23:14.196 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:23:14.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fea0406b4d0 msgr2=0x7fea0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fea0406b4d0 0x7fea0406d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fea0c036000 tx=0x7fea0c03d040 comp rx=0 tx=0).stop 2024-03-06T07:23:14.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1086717855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea180cea50 msgr2=0x7fea180c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea180cea50 0x7fea180c8b90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fea0800c450 tx=0x7fea0800c910 comp rx=0 tx=0).stop 2024-03-06T07:23:14.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1086717855 shutdown_connections 2024-03-06T07:23:14.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fea0406b4d0 0x7fea0406d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea1815def0 0x7fea180ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea180cea50 0x7fea180c8b90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 --2- 172.21.15.59:0/1086717855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea1806cab0 0x7fea180c78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1086717855 >> 172.21.15.59:0/1086717855 conn(0x7fea180b3690 msgr2=0x7fea1806fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1086717855 shutdown_connections 2024-03-06T07:23:14.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.196+0000 7fea1d7eb640 1 -- 172.21.15.59:0/1086717855 wait complete. 2024-03-06T07:23:14.209 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.313+0000 7ffbce9bc640 1 Processor -- start 2024-03-06T07:23:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.313+0000 7ffbce9bc640 1 -- start start 2024-03-06T07:23:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbce9bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffbc81614d0 0x7ffbc80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbce9bc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffbc80d1910 0x7ffbc80d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbce9bc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc80d2740 con 0x7ffbc80d22b0 2024-03-06T07:23:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbce9bc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc81618b0 con 0x7ffbc80d1910 2024-03-06T07:23:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbce9bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc80cf400 con 0x7ffbc81614d0 2024-03-06T07:23:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbc77fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffbc80d1910 0x7ffbc80d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbc7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffbc81614d0 0x7ffbc80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbc77fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffbc80d1910 0x7ffbc80d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:57678/0 (socket says 172.21.15.59:57678) 2024-03-06T07:23:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbccf59640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ffbc80d22b0 0x7ffbc80d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:55320/0 (socket says 172.21.15.59:55320) 2024-03-06T07:23:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbc77fe640 1 -- 172.21.15.59:0/1528664277 learned_addr learned my addr 172.21.15.59:0/1528664277 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:14.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbc77fe640 1 -- 172.21.15.59:0/1528664277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffbc81614d0 msgr2=0x7ffbc80d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbc77fe640 1 --2- 172.21.15.59:0/1528664277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffbc81614d0 0x7ffbc80d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbc77fe640 1 -- 172.21.15.59:0/1528664277 >> v1:172.21.15.164:6789/0 conn(0x7ffbc80d22b0 legacy=0x7ffbc80d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.314+0000 7ffbc77fe640 1 -- 172.21.15.59:0/1528664277 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc80cf6a0 con 0x7ffbc80d1910 2024-03-06T07:23:14.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.315+0000 7ffbc77fe640 1 --2- 172.21.15.59:0/1528664277 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffbc80d1910 0x7ffbc80d1d70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc009870 tx=0x7ffbbc02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64a6f75488d56b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.315+0000 7ffbc6ffd640 1 -- 172.21.15.59:0/1528664277 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc034070 con 0x7ffbc80d1910 2024-03-06T07:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.315+0000 7ffbc6ffd640 1 -- 172.21.15.59:0/1528664277 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc030840 con 0x7ffbc80d1910 2024-03-06T07:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.315+0000 7ffbc6ffd640 1 -- 172.21.15.59:0/1528664277 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc030c10 con 0x7ffbc80d1910 2024-03-06T07:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.316+0000 7ffbce9bc640 1 -- 172.21.15.59:0/1528664277 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffbc80d1910 msgr2=0x7ffbc80d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.316+0000 7ffbce9bc640 1 --2- 172.21.15.59:0/1528664277 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffbc80d1910 0x7ffbc80d1d70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc009870 tx=0x7ffbbc02b980 comp rx=0 tx=0).stop 2024-03-06T07:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.316+0000 7ffbce9bc640 1 -- 172.21.15.59:0/1528664277 shutdown_connections 2024-03-06T07:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.316+0000 7ffbce9bc640 1 --2- 172.21.15.59:0/1528664277 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffbc80d1910 0x7ffbc80d1d70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.316+0000 7ffbce9bc640 1 --2- 172.21.15.59:0/1528664277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffbc81614d0 0x7ffbc80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.316+0000 7ffbce9bc640 1 -- 172.21.15.59:0/1528664277 >> 172.21.15.59:0/1528664277 conn(0x7ffbc80d3d40 msgr2=0x7ffbc80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.316+0000 7ffbce9bc640 1 -- 172.21.15.59:0/1528664277 shutdown_connections 2024-03-06T07:23:14.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.316+0000 7ffbce9bc640 1 -- 172.21.15.59:0/1528664277 wait complete. 2024-03-06T07:23:14.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.317+0000 7ffbce9bc640 1 Processor -- start 2024-03-06T07:23:14.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.317+0000 7ffbce9bc640 1 -- start start 2024-03-06T07:23:14.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbce9bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffbc80d1910 0x7ffbc815c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbce9bc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffbc80d22b0 0x7ffbc80cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbce9bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbc81614d0 0x7ffbc81602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbce9bc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc80b0910 con 0x7ffbc81614d0 2024-03-06T07:23:14.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbce9bc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc80cfc60 con 0x7ffbc80d22b0 2024-03-06T07:23:14.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbce9bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc80b1e40 con 0x7ffbc80d1910 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbccf59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbc81614d0 0x7ffbc81602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbccf59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbc81614d0 0x7ffbc81602b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40012/0 (socket says 172.21.15.59:40012) 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbccf59640 1 -- 172.21.15.59:0/2232120019 learned_addr learned my addr 172.21.15.59:0/2232120019 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbc7fff640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffbc80d1910 0x7ffbc815c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbc77fe640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffbc80d22b0 0x7ffbc80cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbccf59640 1 -- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffbc80d1910 msgr2=0x7ffbc815c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbccf59640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffbc80d1910 0x7ffbc815c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbccf59640 1 -- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffbc80d22b0 msgr2=0x7ffbc80cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbccf59640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffbc80d22b0 0x7ffbc80cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbccf59640 1 -- 172.21.15.59:0/2232120019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc80b7b30 con 0x7ffbc81614d0 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.318+0000 7ffbc7fff640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffbc80d1910 0x7ffbc815c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.319+0000 7ffbccf59640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbc81614d0 0x7ffbc81602b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000c7b0 tx=0x7ffbc000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.319+0000 7ffbc4ff9640 1 -- 172.21.15.59:0/2232120019 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0017070 con 0x7ffbc81614d0 2024-03-06T07:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.319+0000 7ffbc4ff9640 1 -- 172.21.15.59:0/2232120019 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc000a640 con 0x7ffbc81614d0 2024-03-06T07:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.319+0000 7ffbce9bc640 1 -- 172.21.15.59:0/2232120019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc815c880 con 0x7ffbc81614d0 2024-03-06T07:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.319+0000 7ffbc4ff9640 1 -- 172.21.15.59:0/2232120019 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc000a9f0 con 0x7ffbc81614d0 2024-03-06T07:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.319+0000 7ffbce9bc640 1 -- 172.21.15.59:0/2232120019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc80b80b0 con 0x7ffbc81614d0 2024-03-06T07:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.320+0000 7ffbce9bc640 1 -- 172.21.15.59:0/2232120019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc8068ff0 con 0x7ffbc81614d0 2024-03-06T07:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.320+0000 7ffbc4ff9640 1 -- 172.21.15.59:0/2232120019 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffbc001a070 con 0x7ffbc81614d0 2024-03-06T07:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.321+0000 7ffbc4ff9640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffbb406b4d0 0x7ffbb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.321+0000 7ffbc4ff9640 1 -- 172.21.15.59:0/2232120019 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbc00896c0 con 0x7ffbc81614d0 2024-03-06T07:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.321+0000 7ffbc7fff640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffbb406b4d0 0x7ffbb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.321+0000 7ffbc7fff640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffbb406b4d0 0x7ffbb406d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80ca4a0 tx=0x7ffbb8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.325+0000 7ffbc4ff9640 1 -- 172.21.15.59:0/2232120019 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffbc80d2740 con 0x7ffbc81614d0 2024-03-06T07:23:14.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.505+0000 7ffbce9bc640 1 -- 172.21.15.59:0/2232120019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffbc80a1c10 con 0x7ffbc81614d0 2024-03-06T07:23:14.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.505+0000 7ffbc4ff9640 1 -- 172.21.15.59:0/2232120019 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffbc0057f80 con 0x7ffbc81614d0 2024-03-06T07:23:14.507 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:23:14.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 -- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffbb406b4d0 msgr2=0x7ffbb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffbb406b4d0 0x7ffbb406d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80ca4a0 tx=0x7ffbb8008040 comp rx=0 tx=0).stop 2024-03-06T07:23:14.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 -- 172.21.15.59:0/2232120019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbc81614d0 msgr2=0x7ffbc81602b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbc81614d0 0x7ffbc81602b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000c7b0 tx=0x7ffbc000cc70 comp rx=0 tx=0).stop 2024-03-06T07:23:14.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 -- 172.21.15.59:0/2232120019 shutdown_connections 2024-03-06T07:23:14.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffbb406b4d0 0x7ffbb406d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffbc81614d0 0x7ffbc81602b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffbc80d22b0 0x7ffbc80cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 --2- 172.21.15.59:0/2232120019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffbc80d1910 0x7ffbc815c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 -- 172.21.15.59:0/2232120019 >> 172.21.15.59:0/2232120019 conn(0x7ffbc80d3d40 msgr2=0x7ffbc815fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 -- 172.21.15.59:0/2232120019 shutdown_connections 2024-03-06T07:23:14.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.507+0000 7ffbce9bc640 1 -- 172.21.15.59:0/2232120019 wait complete. 2024-03-06T07:23:14.520 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:14.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.620+0000 7fc7938d4640 1 Processor -- start 2024-03-06T07:23:14.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.620+0000 7fc7938d4640 1 -- start start 2024-03-06T07:23:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc7938d4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 0x7fc78c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc7938d4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c0d25d0 con 0x7fc78c0d4760 2024-03-06T07:23:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc7938d4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c0d2870 con 0x7fc78c0d0a80 2024-03-06T07:23:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc7938d4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0d4bf0 con 0x7fc78c0b4170 2024-03-06T07:23:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc790e6f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc78c0d4760 0x7fc78c0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54352/0 (socket says 172.21.15.59:54352) 2024-03-06T07:23:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc790e6f640 1 -- 172.21.15.59:0/599907410 learned_addr learned my addr 172.21.15.59:0/599907410 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc791670640 1 --2- 172.21.15.59:0/599907410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 0x7fc78c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc783fff640 1 -- 172.21.15.59:0/599907410 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227151950 0 0) 0x7fc78c0d25d0 con 0x7fc78c0d4760 2024-03-06T07:23:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc783fff640 1 -- 172.21.15.59:0/599907410 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc778003520 con 0x7fc78c0d4760 2024-03-06T07:23:14.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc783fff640 1 -- 172.21.15.59:0/599907410 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205347616 0 0) 0x7fc78c0d2870 con 0x7fc78c0d0a80 2024-03-06T07:23:14.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc783fff640 1 -- 172.21.15.59:0/599907410 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc78c0d25d0 con 0x7fc78c0d0a80 2024-03-06T07:23:14.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc783fff640 1 -- 172.21.15.59:0/599907410 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2432554376 0 0) 0x7fc778003520 con 0x7fc78c0d4760 2024-03-06T07:23:14.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc791670640 1 -- 172.21.15.59:0/599907410 >> v1:172.21.15.164:6789/0 conn(0x7fc78c0d0a80 legacy=0x7fc78c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc791670640 1 -- 172.21.15.59:0/599907410 >> v1:172.21.15.59:6789/0 conn(0x7fc78c0d4760 legacy=0x7fc78c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.621+0000 7fc791670640 1 -- 172.21.15.59:0/599907410 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c0d2b50 con 0x7fc78c0b4170 2024-03-06T07:23:14.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.622+0000 7fc791670640 1 --2- 172.21.15.59:0/599907410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 0x7fc78c0b6a90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc788009dc0 tx=0x7fc78802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76fea5f3b48996d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.622+0000 7fc783fff640 1 -- 172.21.15.59:0/599907410 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788039070 con 0x7fc78c0b4170 2024-03-06T07:23:14.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.622+0000 7fc783fff640 1 -- 172.21.15.59:0/599907410 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78802c910 con 0x7fc78c0b4170 2024-03-06T07:23:14.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.622+0000 7fc783fff640 1 -- 172.21.15.59:0/599907410 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c001670 con 0x7fc78c0b4170 2024-03-06T07:23:14.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.623+0000 7fc7938d4640 1 -- 172.21.15.59:0/599907410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 msgr2=0x7fc78c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.623+0000 7fc7938d4640 1 --2- 172.21.15.59:0/599907410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 0x7fc78c0b6a90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc788009dc0 tx=0x7fc78802bde0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.623+0000 7fc7938d4640 1 -- 172.21.15.59:0/599907410 shutdown_connections 2024-03-06T07:23:14.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.623+0000 7fc7938d4640 1 --2- 172.21.15.59:0/599907410 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 0x7fc78c0b6a90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.623+0000 7fc7938d4640 1 -- 172.21.15.59:0/599907410 >> 172.21.15.59:0/599907410 conn(0x7fc78c0d3c40 msgr2=0x7fc78c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.623+0000 7fc7938d4640 1 -- 172.21.15.59:0/599907410 shutdown_connections 2024-03-06T07:23:14.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.623+0000 7fc7938d4640 1 -- 172.21.15.59:0/599907410 wait complete. 2024-03-06T07:23:14.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.625+0000 7fc7938d4640 1 Processor -- start 2024-03-06T07:23:14.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.625+0000 7fc7938d4640 1 -- start start 2024-03-06T07:23:14.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc7938d4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 0x7fc78c1499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc7938d4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc78c0d0a80 0x7fc78c0ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc7938d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc78c0cd430 0x7fc78c157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc7938d4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0b4dd0 con 0x7fc78c0cd430 2024-03-06T07:23:14.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc7938d4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0ce9e0 con 0x7fc78c0d0a80 2024-03-06T07:23:14.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc7938d4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0b4f50 con 0x7fc78c0b4170 2024-03-06T07:23:14.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc791e71640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc78c0cd430 0x7fc78c157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc791e71640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc78c0cd430 0x7fc78c157f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40020/0 (socket says 172.21.15.59:40020) 2024-03-06T07:23:14.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc791e71640 1 -- 172.21.15.59:0/2811114637 learned_addr learned my addr 172.21.15.59:0/2811114637 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:14.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc791670640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 0x7fc78c1499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc791e71640 1 -- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 msgr2=0x7fc78c1499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc790e6f640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc78c0d0a80 0x7fc78c0ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.626+0000 7fc791e71640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 0x7fc78c1499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.627+0000 7fc791e71640 1 -- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc78c0d0a80 msgr2=0x7fc78c0ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.627+0000 7fc791e71640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc78c0d0a80 0x7fc78c0ccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.627+0000 7fc791e71640 1 -- 172.21.15.59:0/2811114637 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c0cd970 con 0x7fc78c0cd430 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.627+0000 7fc791670640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 0x7fc78c1499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.627+0000 7fc790e6f640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc78c0d0a80 0x7fc78c0ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.627+0000 7fc791e71640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc78c0cd430 0x7fc78c157f00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc78400c430 tx=0x7fc78400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.627+0000 7fc781ffb640 1 -- 172.21.15.59:0/2811114637 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc784017070 con 0x7fc78c0cd430 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.627+0000 7fc781ffb640 1 -- 172.21.15.59:0/2811114637 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7840052d0 con 0x7fc78c0cd430 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.627+0000 7fc781ffb640 1 -- 172.21.15.59:0/2811114637 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc784003830 con 0x7fc78c0cd430 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.627+0000 7fc7938d4640 1 -- 172.21.15.59:0/2811114637 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78c0b7880 con 0x7fc78c0cd430 2024-03-06T07:23:14.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.627+0000 7fc7938d4640 1 -- 172.21.15.59:0/2811114637 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78c0b7e40 con 0x7fc78c0cd430 2024-03-06T07:23:14.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.628+0000 7fc7938d4640 1 -- 172.21.15.59:0/2811114637 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc78c04f330 con 0x7fc78c0cd430 2024-03-06T07:23:14.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.629+0000 7fc781ffb640 1 -- 172.21.15.59:0/2811114637 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc78401a070 con 0x7fc78c0cd430 2024-03-06T07:23:14.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.629+0000 7fc781ffb640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc77006b4d0 0x7fc77006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.629+0000 7fc781ffb640 1 -- 172.21.15.59:0/2811114637 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7840895f0 con 0x7fc78c0cd430 2024-03-06T07:23:14.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.629+0000 7fc791670640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc77006b4d0 0x7fc77006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.630+0000 7fc791670640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc77006b4d0 0x7fc77006d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc788036000 tx=0x7fc788030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.633+0000 7fc781ffb640 1 -- 172.21.15.59:0/2811114637 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc784058470 con 0x7fc78c0cd430 2024-03-06T07:23:14.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.852+0000 7fc7938d4640 1 -- 172.21.15.59:0/2811114637 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc78c0733b0 con 0x7fc78c0cd430 2024-03-06T07:23:14.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.852+0000 7fc781ffb640 1 -- 172.21.15.59:0/2811114637 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fc784057d70 con 0x7fc78c0cd430 2024-03-06T07:23:14.854 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:14.855 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:23:14.855 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:23:14.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 -- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc77006b4d0 msgr2=0x7fc77006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc77006b4d0 0x7fc77006d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc788036000 tx=0x7fc788030000 comp rx=0 tx=0).stop 2024-03-06T07:23:14.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 -- 172.21.15.59:0/2811114637 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc78c0cd430 msgr2=0x7fc78c157f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc78c0cd430 0x7fc78c157f00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc78400c430 tx=0x7fc78400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 -- 172.21.15.59:0/2811114637 shutdown_connections 2024-03-06T07:23:14.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc77006b4d0 0x7fc77006d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc78c0cd430 0x7fc78c157f00 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc78c0d0a80 0x7fc78c0ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 --2- 172.21.15.59:0/2811114637 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc78c0b4170 0x7fc78c1499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 -- 172.21.15.59:0/2811114637 >> 172.21.15.59:0/2811114637 conn(0x7fc78c0d3c40 msgr2=0x7fc78c0b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 -- 172.21.15.59:0/2811114637 shutdown_connections 2024-03-06T07:23:14.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.854+0000 7fc7938d4640 1 -- 172.21.15.59:0/2811114637 wait complete. 2024-03-06T07:23:14.867 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.967+0000 7f0e1d664640 1 Processor -- start 2024-03-06T07:23:14.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.967+0000 7f0e1d664640 1 -- start start 2024-03-06T07:23:14.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.967+0000 7f0e1d664640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0e180b4170 0x7f0e180b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.967+0000 7f0e1d664640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e180d4760 0x7f0e180d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.967+0000 7f0e1d664640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0e180b4dd0 0x7f0e180d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.967+0000 7f0e1d664640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e180d4bf0 con 0x7f0e180b4dd0 2024-03-06T07:23:14.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.967+0000 7f0e1d664640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e180ce9e0 con 0x7f0e180b4170 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.967+0000 7f0e1d664640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180d25d0 con 0x7f0e180d4760 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e177fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0e180b4dd0 0x7f0e180d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e177fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0e180b4dd0 0x7f0e180d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40022/0 (socket says 172.21.15.59:40022) 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e177fe640 1 -- 172.21.15.59:0/880590065 learned_addr learned my addr 172.21.15.59:0/880590065 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e16ffd640 1 --2- 172.21.15.59:0/880590065 >> v2:172.21.15.164:3300/0 conn(0x7f0e180b4170 0x7f0e180b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e167fc640 1 --2- 172.21.15.59:0/880590065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e180d4760 0x7f0e180d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e177fe640 1 -- 172.21.15.59:0/880590065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e180d4760 msgr2=0x7f0e180d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e177fe640 1 --2- 172.21.15.59:0/880590065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e180d4760 0x7f0e180d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e177fe640 1 -- 172.21.15.59:0/880590065 >> v2:172.21.15.164:3300/0 conn(0x7f0e180b4170 msgr2=0x7f0e180b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e177fe640 1 --2- 172.21.15.59:0/880590065 >> v2:172.21.15.164:3300/0 conn(0x7f0e180b4170 0x7f0e180b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e177fe640 1 -- 172.21.15.59:0/880590065 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e180d2870 con 0x7f0e180b4dd0 2024-03-06T07:23:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e16ffd640 1 --2- 172.21.15.59:0/880590065 >> v2:172.21.15.164:3300/0 conn(0x7f0e180b4170 0x7f0e180b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.968+0000 7f0e167fc640 1 --2- 172.21.15.59:0/880590065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e180d4760 0x7f0e180d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e177fe640 1 --2- 172.21.15.59:0/880590065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e180b4dd0 0x7f0e180d08b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0e1806e030 tx=0x7f0e0802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f84e9709fe2bcd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e15ffb640 1 -- 172.21.15.59:0/880590065 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08035020 con 0x7f0e180b4dd0 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e15ffb640 1 -- 172.21.15.59:0/880590065 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e08031870 con 0x7f0e180b4dd0 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e15ffb640 1 -- 172.21.15.59:0/880590065 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08031c40 con 0x7f0e180b4dd0 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e1d664640 1 -- 172.21.15.59:0/880590065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e180b4dd0 msgr2=0x7f0e180d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e1d664640 1 --2- 172.21.15.59:0/880590065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e180b4dd0 0x7f0e180d08b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0e1806e030 tx=0x7f0e0802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e1d664640 1 -- 172.21.15.59:0/880590065 shutdown_connections 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e1d664640 1 --2- 172.21.15.59:0/880590065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e180b4dd0 0x7f0e180d08b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e1d664640 1 --2- 172.21.15.59:0/880590065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e180d4760 0x7f0e180d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e1d664640 1 --2- 172.21.15.59:0/880590065 >> v2:172.21.15.164:3300/0 conn(0x7f0e180b4170 0x7f0e180b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e1d664640 1 -- 172.21.15.59:0/880590065 >> 172.21.15.59:0/880590065 conn(0x7f0e180d3c40 msgr2=0x7f0e180d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e1d664640 1 -- 172.21.15.59:0/880590065 shutdown_connections 2024-03-06T07:23:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.969+0000 7f0e1d664640 1 -- 172.21.15.59:0/880590065 wait complete. 2024-03-06T07:23:14.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.971+0000 7f0e1d664640 1 Processor -- start 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.971+0000 7f0e1d664640 1 -- start start 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e1d664640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e180b4170 0x7f0e181603a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e1d664640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e180b4dd0 0x7f0e180cbc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e1d664640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e180cc150 0x7f0e180c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e1d664640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180b47d0 con 0x7f0e180cc150 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e1d664640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180b4550 con 0x7f0e180b4170 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e1d664640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180b4950 con 0x7f0e180b4dd0 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e180cc150 0x7f0e180c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e177fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e180cc150 0x7f0e180c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40026/0 (socket says 172.21.15.59:40026) 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e177fe640 1 -- 172.21.15.59:0/4069438891 learned_addr learned my addr 172.21.15.59:0/4069438891 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e16ffd640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e180b4170 0x7f0e181603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e167fc640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e180b4dd0 0x7f0e180cbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e177fe640 1 -- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e180b4dd0 msgr2=0x7f0e180cbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e177fe640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e180b4dd0 0x7f0e180cbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e177fe640 1 -- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e180b4170 msgr2=0x7f0e181603a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e177fe640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e180b4170 0x7f0e181603a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e177fe640 1 -- 172.21.15.59:0/4069438891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e180cd070 con 0x7f0e180cc150 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e177fe640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e180cc150 0x7f0e180c96b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0e0802b950 tx=0x7f0e080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0dff7fe640 1 -- 172.21.15.59:0/4069438891 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08040070 con 0x7f0e180cc150 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0dff7fe640 1 -- 172.21.15.59:0/4069438891 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0803a420 con 0x7f0e180cc150 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0e1d664640 1 -- 172.21.15.59:0/4069438891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e180cd350 con 0x7f0e180cc150 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.973+0000 7f0dff7fe640 1 -- 172.21.15.59:0/4069438891 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08035040 con 0x7f0e180cc150 2024-03-06T07:23:14.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.974+0000 7f0e1d664640 1 -- 172.21.15.59:0/4069438891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e180cd910 con 0x7f0e180cc150 2024-03-06T07:23:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.975+0000 7f0e1d664640 1 -- 172.21.15.59:0/4069438891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1804f330 con 0x7f0e180cc150 2024-03-06T07:23:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.975+0000 7f0dff7fe640 1 -- 172.21.15.59:0/4069438891 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e08038020 con 0x7f0e180cc150 2024-03-06T07:23:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.975+0000 7f0dff7fe640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e0406b500 0x7f0e0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.975+0000 7f0dff7fe640 1 -- 172.21.15.59:0/4069438891 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e080ac0c0 con 0x7f0e180cc150 2024-03-06T07:23:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.975+0000 7f0e16ffd640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e0406b500 0x7f0e0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.976+0000 7f0e16ffd640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e0406b500 0x7f0e0406d9c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0e10006b10 tx=0x7f0e10008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:14.980+0000 7f0dff7fe640 1 -- 172.21.15.59:0/4069438891 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e0807a8f0 con 0x7f0e180cc150 2024-03-06T07:23:15.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.162+0000 7f0e1d664640 1 -- 172.21.15.59:0/4069438891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e180733b0 con 0x7f0e180cc150 2024-03-06T07:23:15.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.164+0000 7f0dff7fe640 1 -- 172.21.15.59:0/4069438891 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f0e0807e990 con 0x7f0e180cc150 2024-03-06T07:23:15.166 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:15.166 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":33,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:52.226312+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:48.848650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:49.545360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:15.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.165+0000 7f0e1d664640 1 -- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e0406b500 msgr2=0x7f0e0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.165+0000 7f0e1d664640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e0406b500 0x7f0e0406d9c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0e10006b10 tx=0x7f0e10008040 comp rx=0 tx=0).stop 2024-03-06T07:23:15.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.165+0000 7f0e1d664640 1 -- 172.21.15.59:0/4069438891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e180cc150 msgr2=0x7f0e180c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.165+0000 7f0e1d664640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e180cc150 0x7f0e180c96b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0e0802b950 tx=0x7f0e080046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.165+0000 7f0e1d664640 1 -- 172.21.15.59:0/4069438891 shutdown_connections 2024-03-06T07:23:15.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.165+0000 7f0e1d664640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e0406b500 0x7f0e0406d9c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.165+0000 7f0e1d664640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e180cc150 0x7f0e180c96b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.165+0000 7f0e1d664640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e180b4dd0 0x7f0e180cbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.165+0000 7f0e1d664640 1 --2- 172.21.15.59:0/4069438891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e180b4170 0x7f0e181603a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.165+0000 7f0e1d664640 1 -- 172.21.15.59:0/4069438891 >> 172.21.15.59:0/4069438891 conn(0x7f0e180d3c40 msgr2=0x7f0e180b7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.165+0000 7f0e1d664640 1 -- 172.21.15.59:0/4069438891 shutdown_connections 2024-03-06T07:23:15.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.166+0000 7f0e1d664640 1 -- 172.21.15.59:0/4069438891 wait complete. 2024-03-06T07:23:15.179 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:15.179 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:15.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.298+0000 7f2fee42a640 1 Processor -- start 2024-03-06T07:23:15.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.298+0000 7f2fee42a640 1 -- start start 2024-03-06T07:23:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fee42a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2fe80cce40 0x7f2fe815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fee42a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2fe80cd7a0 0x7f2fe80cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fee42a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe8071f40 con 0x7f2fe81515e0 2024-03-06T07:23:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fee42a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe80cd2f0 con 0x7f2fe80cce40 2024-03-06T07:23:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fee42a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe8070a70 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fe77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2fe80cce40 0x7f2fe815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fe77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2fe80cce40 0x7f2fe815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40042/0 (socket says 172.21.15.59:40042) 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fe77fe640 1 -- 172.21.15.59:0/1373095485 learned_addr learned my addr 172.21.15.59:0/1373095485 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fec9c7640 1 --2- 172.21.15.59:0/1373095485 >> v2:172.21.15.164:3300/0 conn(0x7f2fe80cd7a0 0x7f2fe80cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fe77fe640 1 -- 172.21.15.59:0/1373095485 >> v2:172.21.15.164:3300/0 conn(0x7f2fe80cd7a0 msgr2=0x7f2fe80cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fe77fe640 1 --2- 172.21.15.59:0/1373095485 >> v2:172.21.15.164:3300/0 conn(0x7f2fe80cd7a0 0x7f2fe80cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.299+0000 7f2fe77fe640 1 -- 172.21.15.59:0/1373095485 >> v1:172.21.15.164:6789/0 conn(0x7f2fe81515e0 legacy=0x7f2fe8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.300+0000 7f2fe77fe640 1 -- 172.21.15.59:0/1373095485 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe8070d50 con 0x7f2fe80cce40 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.300+0000 7f2fe77fe640 1 --2- 172.21.15.59:0/1373095485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fe80cce40 0x7f2fe815b430 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80cc340 tx=0x7f2fd802e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b1f69b8aa38777 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.300+0000 7f2fe6ffd640 1 -- 172.21.15.59:0/1373095485 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd803b070 con 0x7f2fe80cce40 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.300+0000 7f2fe6ffd640 1 -- 172.21.15.59:0/1373095485 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd8036440 con 0x7f2fe80cce40 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.300+0000 7f2fe6ffd640 1 -- 172.21.15.59:0/1373095485 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8036810 con 0x7f2fe80cce40 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.301+0000 7f2fee42a640 1 -- 172.21.15.59:0/1373095485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fe80cce40 msgr2=0x7f2fe815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.301+0000 7f2fee42a640 1 --2- 172.21.15.59:0/1373095485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fe80cce40 0x7f2fe815b430 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80cc340 tx=0x7f2fd802e170 comp rx=0 tx=0).stop 2024-03-06T07:23:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.301+0000 7f2fee42a640 1 -- 172.21.15.59:0/1373095485 shutdown_connections 2024-03-06T07:23:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.301+0000 7f2fee42a640 1 --2- 172.21.15.59:0/1373095485 >> v2:172.21.15.164:3300/0 conn(0x7f2fe80cd7a0 0x7f2fe80cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.301+0000 7f2fee42a640 1 --2- 172.21.15.59:0/1373095485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fe80cce40 0x7f2fe815b430 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.301+0000 7f2fee42a640 1 -- 172.21.15.59:0/1373095485 >> 172.21.15.59:0/1373095485 conn(0x7f2fe80cf8a0 msgr2=0x7f2fe80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.301+0000 7f2fee42a640 1 -- 172.21.15.59:0/1373095485 shutdown_connections 2024-03-06T07:23:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.301+0000 7f2fee42a640 1 -- 172.21.15.59:0/1373095485 wait complete. 2024-03-06T07:23:15.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.302+0000 7f2fee42a640 1 Processor -- start 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.302+0000 7f2fee42a640 1 -- start start 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fee42a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fe80cce40 0x7f2fe81ef7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fee42a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fe80cd7a0 0x7f2fe8153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fee42a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fe81515e0 0x7f2fe813e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fee42a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe80ce0c0 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fee42a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe8071310 con 0x7f2fe80cce40 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fee42a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe80cc3a0 con 0x7f2fe81515e0 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fe77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fe80cd7a0 0x7f2fe8153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fe77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fe80cd7a0 0x7f2fe8153480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40054/0 (socket says 172.21.15.59:40054) 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fe77fe640 1 -- 172.21.15.59:0/3848112873 learned_addr learned my addr 172.21.15.59:0/3848112873 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fe7fff640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fe80cce40 0x7f2fe81ef7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fec9c7640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fe81515e0 0x7f2fe813e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fe77fe640 1 -- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fe81515e0 msgr2=0x7f2fe813e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fe77fe640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fe81515e0 0x7f2fe813e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fe77fe640 1 -- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fe80cce40 msgr2=0x7f2fe81ef7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fe77fe640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fe80cce40 0x7f2fe81ef7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fe77fe640 1 -- 172.21.15.59:0/3848112873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe81539c0 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.303+0000 7f2fe77fe640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fe80cd7a0 0x7f2fe8153480 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2fd802e1b0 tx=0x7f2fd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.304+0000 7f2fe4ff9640 1 -- 172.21.15.59:0/3848112873 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8045020 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.304+0000 7f2fe4ff9640 1 -- 172.21.15.59:0/3848112873 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd80346c0 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.304+0000 7f2fee42a640 1 -- 172.21.15.59:0/3848112873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe813b0c0 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.304+0000 7f2fe4ff9640 1 -- 172.21.15.59:0/3848112873 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd803b040 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.304+0000 7f2fee42a640 1 -- 172.21.15.59:0/3848112873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe813b680 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.305+0000 7f2fee42a640 1 -- 172.21.15.59:0/3848112873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe8068840 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.305+0000 7f2fe4ff9640 1 -- 172.21.15.59:0/3848112873 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fd8048020 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.306+0000 7f2fe4ff9640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2fd406b500 0x7f2fd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.306+0000 7f2fe4ff9640 1 -- 172.21.15.59:0/3848112873 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fd80ae900 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.306+0000 7f2fe7fff640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2fd406b500 0x7f2fd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.307+0000 7f2fe7fff640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2fd406b500 0x7f2fd406d9c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc000f60 tx=0x7f2fdc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.310+0000 7f2fe4ff9640 1 -- 172.21.15.59:0/3848112873 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fd804f070 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.529+0000 7f2fee42a640 1 -- 172.21.15.59:0/3848112873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fe804f330 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.530+0000 7f2fe4ff9640 1 -- 172.21.15.59:0/3848112873 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f2fe81539c0 con 0x7f2fe80cd7a0 2024-03-06T07:23:15.532 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:15.532 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:23:15.532 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 -- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2fd406b500 msgr2=0x7f2fd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2fd406b500 0x7f2fd406d9c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc000f60 tx=0x7f2fdc009040 comp rx=0 tx=0).stop 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 -- 172.21.15.59:0/3848112873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fe80cd7a0 msgr2=0x7f2fe8153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fe80cd7a0 0x7f2fe8153480 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2fd802e1b0 tx=0x7f2fd80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 -- 172.21.15.59:0/3848112873 shutdown_connections 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2fd406b500 0x7f2fd406d9c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fe81515e0 0x7f2fe813e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fe80cd7a0 0x7f2fe8153480 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 --2- 172.21.15.59:0/3848112873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fe80cce40 0x7f2fe81ef7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 -- 172.21.15.59:0/3848112873 >> 172.21.15.59:0/3848112873 conn(0x7f2fe80cf8a0 msgr2=0x7f2fe8072110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 -- 172.21.15.59:0/3848112873 shutdown_connections 2024-03-06T07:23:15.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.532+0000 7f2fee42a640 1 -- 172.21.15.59:0/3848112873 wait complete. 2024-03-06T07:23:15.545 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:15.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.644+0000 7fa2fd7eb640 1 Processor -- start 2024-03-06T07:23:15.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.644+0000 7fa2fd7eb640 1 -- start start 2024-03-06T07:23:15.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.645+0000 7fa2fd7eb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa2f80d2700 0x7fa2f80b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.645+0000 7fa2fd7eb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f80d2b70 con 0x7fa2f80d2270 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.645+0000 7fa2fd7eb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f80b3840 con 0x7fa2f81617e0 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.645+0000 7fa2fd7eb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f80b35f0 con 0x7fa2f80d2700 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.645+0000 7fa2effff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa2f80d2270 0x7fa2f80b4600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54368/0 (socket says 172.21.15.59:54368) 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.645+0000 7fa2effff640 1 -- 172.21.15.59:0/2938976365 learned_addr learned my addr 172.21.15.59:0/2938976365 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.645+0000 7fa2f77fe640 1 --2- 172.21.15.59:0/2938976365 >> v2:172.21.15.164:3300/0 conn(0x7fa2f80d2700 0x7fa2f80b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.645+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366580626 0 0) 0x7fa2f80d2b70 con 0x7fa2f80d2270 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.645+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e4003520 con 0x7fa2f80d2270 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.645+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 312162346 0 0) 0x7fa2e4003520 con 0x7fa2f80d2270 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.645+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 >> v2:172.21.15.164:3300/0 conn(0x7fa2f80d2700 msgr2=0x7fa2f80b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f67fc640 1 --2- 172.21.15.59:0/2938976365 >> v2:172.21.15.164:3300/0 conn(0x7fa2f80d2700 0x7fa2f80b4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 >> v1:172.21.15.164:6789/0 conn(0x7fa2f81617e0 legacy=0x7fa2f80cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f80b3b20 con 0x7fa2f80d2270 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fa2e0001670 con 0x7fa2f80d2270 2024-03-06T07:23:15.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 >> v1:172.21.15.59:6789/0 conn(0x7fa2f80d2270 legacy=0x7fa2f80b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f67fc640 1 --2- 172.21.15.59:0/2938976365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2e40039b0 0x7fa2e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f67fc640 1 --2- 172.21.15.59:0/2938976365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4004800 0x7fa2e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f67fc640 1 --2- 172.21.15.59:0/2938976365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2e40051c0 0x7fa2e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80b35f0 con 0x7fa2e4004800 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e400c160 con 0x7fa2e40039b0 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e400c360 con 0x7fa2e40051c0 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2effff640 1 --2- 172.21.15.59:0/2938976365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4004800 0x7fa2e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f6ffd640 1 --2- 172.21.15.59:0/2938976365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2e40039b0 0x7fa2e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f77fe640 1 --2- 172.21.15.59:0/2938976365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2e40051c0 0x7fa2e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2effff640 1 -- 172.21.15.59:0/2938976365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2e40051c0 msgr2=0x7fa2e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2effff640 1 --2- 172.21.15.59:0/2938976365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2e40051c0 0x7fa2e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2effff640 1 -- 172.21.15.59:0/2938976365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2e40039b0 msgr2=0x7fa2e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2effff640 1 --2- 172.21.15.59:0/2938976365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2e40039b0 0x7fa2e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2f77fe640 1 -- 172.21.15.59:0/2938976365 reap_dead start 2024-03-06T07:23:15.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.646+0000 7fa2effff640 1 -- 172.21.15.59:0/2938976365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e4003520 con 0x7fa2e4004800 2024-03-06T07:23:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.647+0000 7fa2effff640 1 --2- 172.21.15.59:0/2938976365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4004800 0x7fa2e4004c80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c210 tx=0x7fa2e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d36ee5cae12d55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.647+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0007d30 con 0x7fa2e4004800 2024-03-06T07:23:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.647+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e0030070 con 0x7fa2e4004800 2024-03-06T07:23:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.647+0000 7fa2f67fc640 1 -- 172.21.15.59:0/2938976365 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0005d70 con 0x7fa2e4004800 2024-03-06T07:23:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.648+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/2938976365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4004800 msgr2=0x7fa2e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.648+0000 7fa2fd7eb640 1 --2- 172.21.15.59:0/2938976365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4004800 0x7fa2e4004c80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c210 tx=0x7fa2e4005830 comp rx=0 tx=0).stop 2024-03-06T07:23:15.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.648+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/2938976365 shutdown_connections 2024-03-06T07:23:15.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.648+0000 7fa2fd7eb640 1 --2- 172.21.15.59:0/2938976365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e4004800 0x7fa2e4004c80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.648+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/2938976365 >> 172.21.15.59:0/2938976365 conn(0x7fa2f80d3c90 msgr2=0x7fa2f80d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.648+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/2938976365 shutdown_connections 2024-03-06T07:23:15.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.648+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/2938976365 wait complete. 2024-03-06T07:23:15.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.650+0000 7fa2fd7eb640 1 Processor -- start 2024-03-06T07:23:15.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.650+0000 7fa2fd7eb640 1 -- start start 2024-03-06T07:23:15.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.651+0000 7fa2fd7eb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2e4004800 0x7fa2f80c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.651+0000 7fa2fd7eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f815c250 0x7fa2f80c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.651+0000 7fa2fd7eb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2f8152af0 0x7fa2f8152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.651+0000 7fa2fd7eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80d4990 con 0x7fa2f815c250 2024-03-06T07:23:15.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.651+0000 7fa2fd7eb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80b4170 con 0x7fa2e4004800 2024-03-06T07:23:15.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.651+0000 7fa2fd7eb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f80b0860 con 0x7fa2f8152af0 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.651+0000 7fa2effff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f815c250 0x7fa2f80c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.651+0000 7fa2effff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f815c250 0x7fa2f80c9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40072/0 (socket says 172.21.15.59:40072) 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.651+0000 7fa2effff640 1 -- 172.21.15.59:0/1333454345 learned_addr learned my addr 172.21.15.59:0/1333454345 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.651+0000 7fa2f6ffd640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2e4004800 0x7fa2f80c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2f77fe640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2f8152af0 0x7fa2f8152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2effff640 1 -- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2f8152af0 msgr2=0x7fa2f8152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2effff640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2f8152af0 0x7fa2f8152f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2effff640 1 -- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2e4004800 msgr2=0x7fa2f80c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2effff640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2e4004800 0x7fa2f80c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2effff640 1 -- 172.21.15.59:0/1333454345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f80c7b40 con 0x7fa2f815c250 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2f6ffd640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2e4004800 0x7fa2f80c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2effff640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f815c250 0x7fa2f80c9d20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c330 tx=0x7fa2e002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2ef7fe640 1 -- 172.21.15.59:0/1333454345 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0046070 con 0x7fa2f815c250 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2ef7fe640 1 -- 172.21.15.59:0/1333454345 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e0004bf0 con 0x7fa2f815c250 2024-03-06T07:23:15.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/1333454345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f80c7e20 con 0x7fa2f815c250 2024-03-06T07:23:15.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2ef7fe640 1 -- 172.21.15.59:0/1333454345 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0007d10 con 0x7fa2f815c250 2024-03-06T07:23:15.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.652+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/1333454345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f80c83e0 con 0x7fa2f815c250 2024-03-06T07:23:15.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.653+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/1333454345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f808cce0 con 0x7fa2f815c250 2024-03-06T07:23:15.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.653+0000 7fa2ef7fe640 1 -- 172.21.15.59:0/1333454345 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2e0007070 con 0x7fa2f815c250 2024-03-06T07:23:15.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.654+0000 7fa2ef7fe640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa2d806b4d0 0x7fa2d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.654+0000 7fa2ef7fe640 1 -- 172.21.15.59:0/1333454345 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2e00ab6b0 con 0x7fa2f815c250 2024-03-06T07:23:15.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.654+0000 7fa2f6ffd640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa2d806b4d0 0x7fa2d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.655+0000 7fa2f6ffd640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa2d806b4d0 0x7fa2d806d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80686a0 tx=0x7fa2e8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.658+0000 7fa2ef7fe640 1 -- 172.21.15.59:0/1333454345 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2f808cce0 con 0x7fa2f815c250 2024-03-06T07:23:15.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.879+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/1333454345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2f815ec20 con 0x7fa2f815c250 2024-03-06T07:23:15.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.879+0000 7fa2ef7fe640 1 -- 172.21.15.59:0/1333454345 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fa2e007a400 con 0x7fa2f815c250 2024-03-06T07:23:15.882 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:15.882 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:50.585871+0000","modified":"2024-03-06T07:22:52.584255+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5250},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5250":{"gid":5250,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1359034023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1359034023},{"type":"v1","addr":"172.21.15.59:6835","nonce":1359034023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5250,"qdb_cluster":[5250]},"id":2}]} 2024-03-06T07:23:15.882 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 13 2024-03-06T07:23:15.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa2d806b4d0 msgr2=0x7fa2d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa2d806b4d0 0x7fa2d806d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa2f80686a0 tx=0x7fa2e8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:15.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/1333454345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f815c250 msgr2=0x7fa2f80c9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f815c250 0x7fa2f80c9d20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c330 tx=0x7fa2e002e070 comp rx=0 tx=0).stop 2024-03-06T07:23:15.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/1333454345 shutdown_connections 2024-03-06T07:23:15.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa2d806b4d0 0x7fa2d806d990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2f8152af0 0x7fa2f8152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f815c250 0x7fa2f80c9d20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 --2- 172.21.15.59:0/1333454345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2e4004800 0x7fa2f80c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/1333454345 >> 172.21.15.59:0/1333454345 conn(0x7fa2f80d3c90 msgr2=0x7fa2f80d1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/1333454345 shutdown_connections 2024-03-06T07:23:15.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.882+0000 7fa2fd7eb640 1 -- 172.21.15.59:0/1333454345 wait complete. 2024-03-06T07:23:15.895 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:15.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.994+0000 7f163b913640 1 Processor -- start 2024-03-06T07:23:15.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.994+0000 7f163b913640 1 -- start start 2024-03-06T07:23:16.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.995+0000 7f163b913640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16340b4170 0x7f16340b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.995+0000 7f163b913640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f16340d0a80 0x7f16340cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.995+0000 7f163b913640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16340d25d0 con 0x7f16340d4760 2024-03-06T07:23:16.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.995+0000 7f163b913640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16340d4bf0 con 0x7f16340d0a80 2024-03-06T07:23:16.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.995+0000 7f163b913640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16340ce9e0 con 0x7f16340b4170 2024-03-06T07:23:16.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.995+0000 7f1638eae640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f16340d4760 0x7f16340d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54378/0 (socket says 172.21.15.59:54378) 2024-03-06T07:23:16.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.995+0000 7f1638eae640 1 -- 172.21.15.59:0/2058062087 learned_addr learned my addr 172.21.15.59:0/2058062087 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:16.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.995+0000 7f1639eb0640 1 --2- 172.21.15.59:0/2058062087 >> v2:172.21.15.164:3300/0 conn(0x7f16340d0a80 0x7f16340cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.995+0000 7f16396af640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16340b4170 0x7f16340b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.995+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724542073 0 0) 0x7f16340d25d0 con 0x7f16340d4760 2024-03-06T07:23:16.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.995+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f161c003520 con 0x7f16340d4760 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 194017858 0 0) 0x7f161c003520 con 0x7f16340d4760 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16340b4170 msgr2=0x7f16340b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16340b4170 0x7f16340b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 >> v2:172.21.15.164:3300/0 conn(0x7f16340d0a80 msgr2=0x7f16340cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 --2- 172.21.15.59:0/2058062087 >> v2:172.21.15.164:3300/0 conn(0x7f16340d0a80 0x7f16340cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16340d2870 con 0x7f16340d4760 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f16280019f0 con 0x7f16340d4760 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1639eb0640 1 --2- 172.21.15.59:0/2058062087 >> v2:172.21.15.164:3300/0 conn(0x7f16340d0a80 0x7f16340cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 >> v1:172.21.15.59:6789/0 conn(0x7f16340d4760 legacy=0x7f16340d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f161c0039b0 0x7f161c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16340b4170 0x7f161c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f161c004d40 0x7f161c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16340d4bf0 con 0x7f161c0039b0 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16340ce9e0 con 0x7f161c004d40 2024-03-06T07:23:16.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f161c00b9c0 con 0x7f16340b4170 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f16396af640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f161c0039b0 0x7f161c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1638eae640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16340b4170 0x7f161c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f1639eb0640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f161c004d40 0x7f161c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f16396af640 1 -- 172.21.15.59:0/2058062087 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16340b4170 msgr2=0x7f161c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f16396af640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16340b4170 0x7f161c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f16396af640 1 -- 172.21.15.59:0/2058062087 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f161c004d40 msgr2=0x7f161c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f16396af640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f161c004d40 0x7f161c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.996+0000 7f16396af640 1 -- 172.21.15.59:0/2058062087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161c003520 con 0x7f161c0039b0 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.997+0000 7f16396af640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f161c0039b0 0x7f161c003dd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f162c009ac0 tx=0x7f161c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3a6f1b6fc18bd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.997+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c039070 con 0x7f161c0039b0 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.997+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c02c6b0 con 0x7f161c0039b0 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.997+0000 7f1633fff640 1 -- 172.21.15.59:0/2058062087 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c009690 con 0x7f161c0039b0 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.997+0000 7f163b913640 1 -- 172.21.15.59:0/2058062087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f161c0039b0 msgr2=0x7f161c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.997+0000 7f163b913640 1 --2- 172.21.15.59:0/2058062087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f161c0039b0 0x7f161c003dd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f162c009ac0 tx=0x7f161c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:16.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.998+0000 7f1639eb0640 1 -- 172.21.15.59:0/2058062087 reap_dead start 2024-03-06T07:23:16.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.998+0000 7f163b913640 1 -- 172.21.15.59:0/2058062087 shutdown_connections 2024-03-06T07:23:16.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.998+0000 7f163b913640 1 -- 172.21.15.59:0/2058062087 >> 172.21.15.59:0/2058062087 conn(0x7f16340d3c40 msgr2=0x7f16340d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.998+0000 7f163b913640 1 -- 172.21.15.59:0/2058062087 shutdown_connections 2024-03-06T07:23:16.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.998+0000 7f163b913640 1 -- 172.21.15.59:0/2058062087 wait complete. 2024-03-06T07:23:16.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.999+0000 7f163b913640 1 Processor -- start 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.999+0000 7f163b913640 1 -- start start 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.999+0000 7f163b913640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f161c0039b0 0x7f16341f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.999+0000 7f163b913640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16341f4930 0x7f16340b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.999+0000 7f163b913640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16340b6da0 0x7f16340b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.999+0000 7f163b913640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16340b4dd0 con 0x7f16340b6da0 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.999+0000 7f163b913640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16340b4550 con 0x7f16341f4930 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:15.999+0000 7f163b913640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16340b4f50 con 0x7f161c0039b0 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1639eb0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16340b6da0 0x7f16340b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1639eb0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16340b6da0 0x7f16340b8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40088/0 (socket says 172.21.15.59:40088) 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1639eb0640 1 -- 172.21.15.59:0/1402894923 learned_addr learned my addr 172.21.15.59:0/1402894923 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f16396af640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f161c0039b0 0x7f16341f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1638eae640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16341f4930 0x7f16340b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1639eb0640 1 -- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f161c0039b0 msgr2=0x7f16341f43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1639eb0640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f161c0039b0 0x7f16341f43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1639eb0640 1 -- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16341f4930 msgr2=0x7f16340b6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1639eb0640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16341f4930 0x7f16340b6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1639eb0640 1 -- 172.21.15.59:0/1402894923 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16341f3fc0 con 0x7f16340b6da0 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1639eb0640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16340b6da0 0x7f16340b8970 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f163408df00 tx=0x7f1624011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1631ffb640 1 -- 172.21.15.59:0/1402894923 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624004440 con 0x7f16340b6da0 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1631ffb640 1 -- 172.21.15.59:0/1402894923 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162401c070 con 0x7f16340b6da0 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f163b913640 1 -- 172.21.15.59:0/1402894923 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16340b95e0 con 0x7f16340b6da0 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.000+0000 7f1631ffb640 1 -- 172.21.15.59:0/1402894923 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624003850 con 0x7f16340b6da0 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.001+0000 7f163b913640 1 -- 172.21.15.59:0/1402894923 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163415c770 con 0x7f16340b6da0 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.002+0000 7f163b913640 1 -- 172.21.15.59:0/1402894923 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16340d1060 con 0x7f16340b6da0 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.002+0000 7f1631ffb640 1 -- 172.21.15.59:0/1402894923 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16240039f0 con 0x7f16340b6da0 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.003+0000 7f1631ffb640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f162006b4d0 0x7f162006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.003+0000 7f1631ffb640 1 -- 172.21.15.59:0/1402894923 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f162408a020 con 0x7f16340b6da0 2024-03-06T07:23:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.003+0000 7f16396af640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f162006b4d0 0x7f162006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.004+0000 7f16396af640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f162006b4d0 0x7f162006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f162c036000 tx=0x7f162c030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.007+0000 7f1631ffb640 1 -- 172.21.15.59:0/1402894923 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16340d1060 con 0x7f16340b6da0 2024-03-06T07:23:16.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.263+0000 7f163b913640 1 -- 172.21.15.59:0/1402894923 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f163404f330 con 0x7f16340b6da0 2024-03-06T07:23:16.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.264+0000 7f1631ffb640 1 -- 172.21.15.59:0/1402894923 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16657 (secure 0 0 0) 0x7f1624026020 con 0x7f16340b6da0 2024-03-06T07:23:16.266 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:16.266 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":33,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:22:52.226312+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:48.848650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:49.545360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:16.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 -- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f162006b4d0 msgr2=0x7f162006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f162006b4d0 0x7f162006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f162c036000 tx=0x7f162c030000 comp rx=0 tx=0).stop 2024-03-06T07:23:16.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 -- 172.21.15.59:0/1402894923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16340b6da0 msgr2=0x7f16340b8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16340b6da0 0x7f16340b8970 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f163408df00 tx=0x7f1624011a60 comp rx=0 tx=0).stop 2024-03-06T07:23:16.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 -- 172.21.15.59:0/1402894923 shutdown_connections 2024-03-06T07:23:16.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f162006b4d0 0x7f162006d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16340b6da0 0x7f16340b8970 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16341f4930 0x7f16340b6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 --2- 172.21.15.59:0/1402894923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f161c0039b0 0x7f16341f43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 -- 172.21.15.59:0/1402894923 >> 172.21.15.59:0/1402894923 conn(0x7f16340d3c40 msgr2=0x7f16340b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 -- 172.21.15.59:0/1402894923 shutdown_connections 2024-03-06T07:23:16.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.266+0000 7f163b913640 1 -- 172.21.15.59:0/1402894923 wait complete. 2024-03-06T07:23:16.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:16.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.394+0000 7fcca1e9d640 1 Processor -- start 2024-03-06T07:23:16.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.394+0000 7fcca1e9d640 1 -- start start 2024-03-06T07:23:16.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.395+0000 7fcca1e9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c0cffa0 0x7fcc9c0d03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.395+0000 7fcca1e9d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc9c0cc8c0 con 0x7fcc9c072d90 2024-03-06T07:23:16.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.395+0000 7fcca1e9d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc9c0ccba0 con 0x7fcc9c15fc70 2024-03-06T07:23:16.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.395+0000 7fcca1e9d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c15d330 con 0x7fcc9c0cffa0 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9affd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcc9c072d90 0x7fcc9c160050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54384/0 (socket says 172.21.15.59:54384) 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9affd640 1 -- 172.21.15.59:0/3376684942 learned_addr learned my addr 172.21.15.59:0/3376684942 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9bfff640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c0cffa0 0x7fcc9c0d03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114720122 0 0) 0x7fcc9c0cc8c0 con 0x7fcc9c072d90 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc88003520 con 0x7fcc9c072d90 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2174731868 0 0) 0x7fcc88003520 con 0x7fcc9c072d90 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c0cffa0 msgr2=0x7fcc9c0d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c0cffa0 0x7fcc9c0d03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 >> v1:172.21.15.164:6789/0 conn(0x7fcc9c15fc70 legacy=0x7fcc9c15cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c15c420 con 0x7fcc9c072d90 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fcc8c001690 con 0x7fcc9c072d90 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 >> v1:172.21.15.59:6789/0 conn(0x7fcc9c072d90 legacy=0x7fcc9c160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc880039e0 0x7fcc88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc88004390 0x7fcc88004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c0cffa0 0x7fcc88005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c15d330 con 0x7fcc880039e0 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8800baa0 con 0x7fcc88004390 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8800bca0 con 0x7fcc9c0cffa0 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9b7fe640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc880039e0 0x7fcc88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9affd640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc88004390 0x7fcc88004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.396+0000 7fcc9bfff640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c0cffa0 0x7fcc88005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.397+0000 7fcc9b7fe640 1 -- 172.21.15.59:0/3376684942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c0cffa0 msgr2=0x7fcc88005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.397+0000 7fcc9b7fe640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c0cffa0 0x7fcc88005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.397+0000 7fcc9b7fe640 1 -- 172.21.15.59:0/3376684942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc88004390 msgr2=0x7fcc88004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.397+0000 7fcc9b7fe640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc88004390 0x7fcc88004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.397+0000 7fcc9b7fe640 1 -- 172.21.15.59:0/3376684942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc88003520 con 0x7fcc880039e0 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.397+0000 7fcc9b7fe640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc880039e0 0x7fcc88003dc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fcc84001860 tx=0x7fcc88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82715c61b745d80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.397+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8402f050 con 0x7fcc880039e0 2024-03-06T07:23:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.398+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc84033650 con 0x7fcc880039e0 2024-03-06T07:23:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.398+0000 7fcc9a7fc640 1 -- 172.21.15.59:0/3376684942 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84001430 con 0x7fcc880039e0 2024-03-06T07:23:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.398+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3376684942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc880039e0 msgr2=0x7fcc88003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.398+0000 7fcca1e9d640 1 --2- 172.21.15.59:0/3376684942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc880039e0 0x7fcc88003dc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fcc84001860 tx=0x7fcc88005830 comp rx=0 tx=0).stop 2024-03-06T07:23:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.398+0000 7fcc9bfff640 1 -- 172.21.15.59:0/3376684942 reap_dead start 2024-03-06T07:23:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.398+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3376684942 shutdown_connections 2024-03-06T07:23:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.398+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3376684942 >> 172.21.15.59:0/3376684942 conn(0x7fcc9c06b190 msgr2=0x7fcc9c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.398+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3376684942 shutdown_connections 2024-03-06T07:23:16.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.398+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3376684942 wait complete. 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.399+0000 7fcca1e9d640 1 Processor -- start 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.399+0000 7fcca1e9d640 1 -- start start 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.400+0000 7fcca1e9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc880039e0 0x7fcc9c14df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.400+0000 7fcca1e9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c1569d0 0x7fcc9c14d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.400+0000 7fcca1e9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9c140670 0x7fcc9c0ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.400+0000 7fcca1e9d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c0cce80 con 0x7fcc9c140670 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.400+0000 7fcca1e9d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c160760 con 0x7fcc880039e0 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.400+0000 7fcca1e9d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c0d03f0 con 0x7fcc9c1569d0 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.400+0000 7fcc9bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9c140670 0x7fcc9c0ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.400+0000 7fcc9bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9c140670 0x7fcc9c0ce3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40104/0 (socket says 172.21.15.59:40104) 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.400+0000 7fcc9bfff640 1 -- 172.21.15.59:0/3138207909 learned_addr learned my addr 172.21.15.59:0/3138207909 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:16.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcc9b7fe640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc880039e0 0x7fcc9c14df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcc9affd640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c1569d0 0x7fcc9c14d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcc9bfff640 1 -- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c1569d0 msgr2=0x7fcc9c14d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcc9bfff640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c1569d0 0x7fcc9c14d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcc9bfff640 1 -- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc880039e0 msgr2=0x7fcc9c14df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcc9bfff640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc880039e0 0x7fcc9c14df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcc9bfff640 1 -- 172.21.15.59:0/3138207909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c14d780 con 0x7fcc9c140670 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcc9bfff640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9c140670 0x7fcc9c0ce3a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c0cc660 tx=0x7fcc940115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcc93fff640 1 -- 172.21.15.59:0/3138207909 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94003990 con 0x7fcc9c140670 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcc93fff640 1 -- 172.21.15.59:0/3138207909 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9401b070 con 0x7fcc9c140670 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3138207909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9c14c480 con 0x7fcc9c140670 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcc93fff640 1 -- 172.21.15.59:0/3138207909 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94003b70 con 0x7fcc9c140670 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.401+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3138207909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9c14ca40 con 0x7fcc9c140670 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.402+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3138207909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9c0cfba0 con 0x7fcc9c140670 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.403+0000 7fcc93fff640 1 -- 172.21.15.59:0/3138207909 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc94005d10 con 0x7fcc9c140670 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.403+0000 7fcc93fff640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 0x7fcc7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.403+0000 7fcc93fff640 1 -- 172.21.15.59:0/3138207909 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc9408a5e0 con 0x7fcc9c140670 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.403+0000 7fcc9b7fe640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 0x7fcc7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.404+0000 7fcc9b7fe640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 0x7fcc7c06d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcc84036000 tx=0x7fcc84031000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.413+0000 7fcc93fff640 1 -- 172.21.15.59:0/3138207909 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc940594f0 con 0x7fcc9c140670 2024-03-06T07:23:16.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.650+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3138207909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcc9c06a1b0 con 0x7fcc9c140670 2024-03-06T07:23:16.653 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:23:16.651+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.652+0000 7fcc93fff640 1 -- 172.21.15.59:0/3138207909 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v14) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcc94058df0 con 0x7fcc9c140670 2024-03-06T07:23:16.654 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 msgr2=0x7fcc7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 0x7fcc7c06d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcc84036000 tx=0x7fcc84031000 comp rx=0 tx=0).stop 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3138207909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9c140670 msgr2=0x7fcc9c0ce3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9c140670 0x7fcc9c0ce3a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c0cc660 tx=0x7fcc940115a0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3138207909 shutdown_connections 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc7c06b4d0 0x7fcc7c06d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc9c140670 0x7fcc9c0ce3a0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc9c1569d0 0x7fcc9c14d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 --2- 172.21.15.59:0/3138207909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc880039e0 0x7fcc9c14df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3138207909 >> 172.21.15.59:0/3138207909 conn(0x7fcc9c06b190 msgr2=0x7fcc9c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3138207909 shutdown_connections 2024-03-06T07:23:16.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.656+0000 7fcca1e9d640 1 -- 172.21.15.59:0/3138207909 wait complete. 2024-03-06T07:23:16.672 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:16.697 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:23:16.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.776+0000 7f9956fb5640 1 Processor -- start 2024-03-06T07:23:16.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.776+0000 7f9956fb5640 1 -- start start 2024-03-06T07:23:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.777+0000 7f9956fb5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f99501614d0 0x7f99500d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.777+0000 7f9956fb5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f99500d1910 0x7f99500d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.777+0000 7f9956fb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d22e0 0x7f99500d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.777+0000 7f9956fb5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99500d2750 con 0x7f99500d1910 2024-03-06T07:23:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.777+0000 7f9956fb5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99501618b0 con 0x7f99501614d0 2024-03-06T07:23:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.777+0000 7f9956fb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99500cf400 con 0x7f99500d22e0 2024-03-06T07:23:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.777+0000 7f994ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f99500d1910 0x7f99500d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.777+0000 7f994ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f99500d1910 0x7f99500d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40124/0 (socket says 172.21.15.59:40124) 2024-03-06T07:23:16.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.777+0000 7f994ffff640 1 -- 172.21.15.59:0/2318347118 learned_addr learned my addr 172.21.15.59:0/2318347118 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.778+0000 7f9954d51640 1 --2- 172.21.15.59:0/2318347118 >> v2:172.21.15.164:3300/0 conn(0x7f99501614d0 0x7f99500d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.778+0000 7f9955552640 1 --2- 172.21.15.59:0/2318347118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d22e0 0x7f99500d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.778+0000 7f994ffff640 1 -- 172.21.15.59:0/2318347118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d22e0 msgr2=0x7f99500d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.778+0000 7f994ffff640 1 --2- 172.21.15.59:0/2318347118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d22e0 0x7f99500d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.778+0000 7f994ffff640 1 -- 172.21.15.59:0/2318347118 >> v2:172.21.15.164:3300/0 conn(0x7f99501614d0 msgr2=0x7f99500d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.778+0000 7f994ffff640 1 --2- 172.21.15.59:0/2318347118 >> v2:172.21.15.164:3300/0 conn(0x7f99501614d0 0x7f99500d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.778+0000 7f994ffff640 1 -- 172.21.15.59:0/2318347118 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99500cf6a0 con 0x7f99500d1910 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.778+0000 7f9954d51640 1 --2- 172.21.15.59:0/2318347118 >> v2:172.21.15.164:3300/0 conn(0x7f99501614d0 0x7f99500d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.778+0000 7f9955552640 1 --2- 172.21.15.59:0/2318347118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d22e0 0x7f99500d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.778+0000 7f994ffff640 1 --2- 172.21.15.59:0/2318347118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99500d1910 0x7f99500d1da0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f995008c470 tx=0x7f994402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4530b586ecf9db98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f994f7fe640 1 -- 172.21.15.59:0/2318347118 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9944035020 con 0x7f99500d1910 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f994f7fe640 1 -- 172.21.15.59:0/2318347118 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9944031870 con 0x7f99500d1910 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f994f7fe640 1 -- 172.21.15.59:0/2318347118 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9944031c40 con 0x7f99500d1910 2024-03-06T07:23:16.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f9956fb5640 1 -- 172.21.15.59:0/2318347118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99500d1910 msgr2=0x7f99500d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f9956fb5640 1 --2- 172.21.15.59:0/2318347118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99500d1910 0x7f99500d1da0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f995008c470 tx=0x7f994402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:16.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f9956fb5640 1 -- 172.21.15.59:0/2318347118 shutdown_connections 2024-03-06T07:23:16.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f9956fb5640 1 --2- 172.21.15.59:0/2318347118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d22e0 0x7f99500d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f9956fb5640 1 --2- 172.21.15.59:0/2318347118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99500d1910 0x7f99500d1da0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f9956fb5640 1 --2- 172.21.15.59:0/2318347118 >> v2:172.21.15.164:3300/0 conn(0x7f99501614d0 0x7f99500d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f9956fb5640 1 -- 172.21.15.59:0/2318347118 >> 172.21.15.59:0/2318347118 conn(0x7f99500d3d40 msgr2=0x7f99500d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f9956fb5640 1 -- 172.21.15.59:0/2318347118 shutdown_connections 2024-03-06T07:23:16.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.779+0000 7f9956fb5640 1 -- 172.21.15.59:0/2318347118 wait complete. 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.780+0000 7f9956fb5640 1 Processor -- start 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.780+0000 7f9956fb5640 1 -- start start 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.781+0000 7f9956fb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d1910 0x7f99500cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.781+0000 7f9956fb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99500d22e0 0x7f99501f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.781+0000 7f9956fb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99501614d0 0x7f99501f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.781+0000 7f9956fb5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99500b0910 con 0x7f99501614d0 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.781+0000 7f9956fb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99500cfc60 con 0x7f99500d22e0 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.781+0000 7f9956fb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99500d0be0 con 0x7f99500d1910 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.781+0000 7f9955552640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99501614d0 0x7f99501f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.781+0000 7f9955552640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99501614d0 0x7f99501f6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40140/0 (socket says 172.21.15.59:40140) 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.781+0000 7f9955552640 1 -- 172.21.15.59:0/3698475553 learned_addr learned my addr 172.21.15.59:0/3698475553 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.781+0000 7f994ffff640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99500d22e0 0x7f99501f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.781+0000 7f9954d51640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d1910 0x7f99500cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f9955552640 1 -- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d1910 msgr2=0x7f99500cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f9955552640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d1910 0x7f99500cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f9955552640 1 -- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99500d22e0 msgr2=0x7f99501f2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f9955552640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99500d22e0 0x7f99501f2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f9955552640 1 -- 172.21.15.59:0/3698475553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99500cdf70 con 0x7f99501614d0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f9954d51640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d1910 0x7f99500cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f9955552640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99501614d0 0x7f99501f6cb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f994800cef0 tx=0x7f994800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f994d7fa640 1 -- 172.21.15.59:0/3698475553 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994801a070 con 0x7f99501614d0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f994d7fa640 1 -- 172.21.15.59:0/3698475553 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994800d860 con 0x7f99501614d0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f9956fb5640 1 -- 172.21.15.59:0/3698475553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99500cba30 con 0x7f99501614d0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f994d7fa640 1 -- 172.21.15.59:0/3698475553 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994800dc50 con 0x7f99501614d0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.782+0000 7f9956fb5640 1 -- 172.21.15.59:0/3698475553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99500cbff0 con 0x7f99501614d0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.783+0000 7f9956fb5640 1 -- 172.21.15.59:0/3698475553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99500c7b20 con 0x7f99501614d0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.783+0000 7f994d7fa640 1 -- 172.21.15.59:0/3698475553 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f994801d070 con 0x7f99501614d0 2024-03-06T07:23:16.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.784+0000 7f994d7fa640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993c06b500 0x7f993c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.784+0000 7f994d7fa640 1 -- 172.21.15.59:0/3698475553 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f994808aaf0 con 0x7f99501614d0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.784+0000 7f9954d51640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993c06b500 0x7f993c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.785+0000 7f9954d51640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993c06b500 0x7f993c06d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9940006fd0 tx=0x7f9940008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:16.788+0000 7f994d7fa640 1 -- 172.21.15.59:0/3698475553 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99480599c0 con 0x7f99501614d0 2024-03-06T07:23:17.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.008+0000 7f9956fb5640 1 -- 172.21.15.59:0/3698475553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9950068ff0 con 0x7f99501614d0 2024-03-06T07:23:17.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.657+0000 7f994d7fa640 1 -- 172.21.15.59:0/3698475553 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f99480592c0 con 0x7f99501614d0 2024-03-06T07:23:17.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.659+0000 7f9956fb5640 1 -- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993c06b500 msgr2=0x7f993c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.659+0000 7f9956fb5640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993c06b500 0x7f993c06d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9940006fd0 tx=0x7f9940008040 comp rx=0 tx=0).stop 2024-03-06T07:23:17.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.659+0000 7f9956fb5640 1 -- 172.21.15.59:0/3698475553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99501614d0 msgr2=0x7f99501f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.659+0000 7f9956fb5640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99501614d0 0x7f99501f6cb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f994800cef0 tx=0x7f994800a710 comp rx=0 tx=0).stop 2024-03-06T07:23:17.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.659+0000 7f9956fb5640 1 -- 172.21.15.59:0/3698475553 shutdown_connections 2024-03-06T07:23:17.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.659+0000 7f9956fb5640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993c06b500 0x7f993c06d9c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.660+0000 7f9956fb5640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99501614d0 0x7f99501f6cb0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.660+0000 7f9956fb5640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99500d22e0 0x7f99501f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.660+0000 7f9956fb5640 1 --2- 172.21.15.59:0/3698475553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99500d1910 0x7f99500cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.660+0000 7f9956fb5640 1 -- 172.21.15.59:0/3698475553 >> 172.21.15.59:0/3698475553 conn(0x7f99500d3d40 msgr2=0x7f995006c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.660+0000 7f9956fb5640 1 -- 172.21.15.59:0/3698475553 shutdown_connections 2024-03-06T07:23:17.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.660+0000 7f9956fb5640 1 -- 172.21.15.59:0/3698475553 wait complete. 2024-03-06T07:23:17.675 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:17.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.827+0000 7fb02549a640 1 Processor -- start 2024-03-06T07:23:17.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.827+0000 7fb02549a640 1 -- start start 2024-03-06T07:23:17.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.828+0000 7fb02549a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb020162850 0x7fb0200d1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.828+0000 7fb02549a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0200cf3c0 con 0x7fb0200d1f90 2024-03-06T07:23:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.828+0000 7fb02549a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0200cf0e0 con 0x7fb020162470 2024-03-06T07:23:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.828+0000 7fb02549a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0200d23e0 con 0x7fb020162850 2024-03-06T07:23:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.828+0000 7fb01e7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb020162850 0x7fb0200d1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.828+0000 7fb01f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb0200d1f90 0x7fb02006c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54408/0 (socket says 172.21.15.59:54408) 2024-03-06T07:23:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.828+0000 7fb01f7fe640 1 -- 172.21.15.59:0/3138963067 learned_addr learned my addr 172.21.15.59:0/3138963067 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.829+0000 7fb01e7fc640 1 -- 172.21.15.59:0/3138963067 >> v1:172.21.15.164:6789/0 conn(0x7fb020162470 legacy=0x7fb02006f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.829+0000 7fb01e7fc640 1 -- 172.21.15.59:0/3138963067 >> v1:172.21.15.59:6789/0 conn(0x7fb0200d1f90 legacy=0x7fb02006c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.829+0000 7fb01e7fc640 1 -- 172.21.15.59:0/3138963067 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02006ec20 con 0x7fb020162850 2024-03-06T07:23:17.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.829+0000 7fb01e7fc640 1 --2- 172.21.15.59:0/3138963067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb020162850 0x7fb0200d1a50 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb0140097a0 tx=0x7fb01402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb22483e830782 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.830+0000 7fb01dffb640 1 -- 172.21.15.59:0/3138963067 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb014034070 con 0x7fb020162850 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.830+0000 7fb01dffb640 1 -- 172.21.15.59:0/3138963067 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb014030610 con 0x7fb020162850 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.830+0000 7fb01dffb640 1 -- 172.21.15.59:0/3138963067 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0140309e0 con 0x7fb020162850 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.830+0000 7fb02549a640 1 -- 172.21.15.59:0/3138963067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb020162850 msgr2=0x7fb0200d1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.830+0000 7fb02549a640 1 --2- 172.21.15.59:0/3138963067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb020162850 0x7fb0200d1a50 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb0140097a0 tx=0x7fb01402b960 comp rx=0 tx=0).stop 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.830+0000 7fb02549a640 1 -- 172.21.15.59:0/3138963067 shutdown_connections 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.830+0000 7fb02549a640 1 --2- 172.21.15.59:0/3138963067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb020162850 0x7fb0200d1a50 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.830+0000 7fb02549a640 1 -- 172.21.15.59:0/3138963067 >> 172.21.15.59:0/3138963067 conn(0x7fb02009a9f0 msgr2=0x7fb02009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.830+0000 7fb02549a640 1 -- 172.21.15.59:0/3138963067 shutdown_connections 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.830+0000 7fb02549a640 1 -- 172.21.15.59:0/3138963067 wait complete. 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.831+0000 7fb02549a640 1 Processor -- start 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.832+0000 7fb02549a640 1 -- start start 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.832+0000 7fb02549a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0200d1f90 0x7fb0201552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.832+0000 7fb02549a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb020162470 0x7fb0200c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.832+0000 7fb02549a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb020162850 0x7fb0200c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.832+0000 7fb02549a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02006fe10 con 0x7fb020162470 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.832+0000 7fb02549a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02006f270 con 0x7fb0200d1f90 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.832+0000 7fb02549a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02006ff90 con 0x7fb020162850 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.832+0000 7fb01e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb020162470 0x7fb0200c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.832+0000 7fb01e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb020162470 0x7fb0200c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40166/0 (socket says 172.21.15.59:40166) 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.832+0000 7fb01e7fc640 1 -- 172.21.15.59:0/2716061821 learned_addr learned my addr 172.21.15.59:0/2716061821 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb01e7fc640 1 -- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb020162850 msgr2=0x7fb0200c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb01e7fc640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb020162850 0x7fb0200c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb01e7fc640 1 -- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0200d1f90 msgr2=0x7fb0201552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb01effd640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0200d1f90 0x7fb0201552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb01e7fc640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0200d1f90 0x7fb0201552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb01e7fc640 1 -- 172.21.15.59:0/2716061821 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0200b59b0 con 0x7fb020162470 2024-03-06T07:23:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb01effd640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0200d1f90 0x7fb0201552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb01e7fc640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb020162470 0x7fb0200c6e10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb01402bf70 tx=0x7fb0140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb0077fe640 1 -- 172.21.15.59:0/2716061821 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb014034040 con 0x7fb020162470 2024-03-06T07:23:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb0077fe640 1 -- 172.21.15.59:0/2716061821 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb014039a10 con 0x7fb020162470 2024-03-06T07:23:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb0077fe640 1 -- 172.21.15.59:0/2716061821 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb014039e20 con 0x7fb020162470 2024-03-06T07:23:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb02549a640 1 -- 172.21.15.59:0/2716061821 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0200b5c90 con 0x7fb020162470 2024-03-06T07:23:17.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.833+0000 7fb02549a640 1 -- 172.21.15.59:0/2716061821 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0200b6250 con 0x7fb020162470 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.834+0000 7fb02549a640 1 -- 172.21.15.59:0/2716061821 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02004f330 con 0x7fb020162470 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.834+0000 7fb0077fe640 1 -- 172.21.15.59:0/2716061821 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb014037070 con 0x7fb020162470 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.835+0000 7fb0077fe640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb00c06b4d0 0x7fb00c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.835+0000 7fb0077fe640 1 -- 172.21.15.59:0/2716061821 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb0140ac440 con 0x7fb020162470 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.835+0000 7fb01effd640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb00c06b4d0 0x7fb00c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.836+0000 7fb01effd640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb00c06b4d0 0x7fb00c06d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb020161710 tx=0x7fb018009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:17.839+0000 7fb0077fe640 1 -- 172.21.15.59:0/2716061821 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb014037360 con 0x7fb020162470 2024-03-06T07:23:18.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.019+0000 7fb02549a640 1 -- 172.21.15.59:0/2716061821 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb02006c190 con 0x7fb020162470 2024-03-06T07:23:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.658+0000 7fb0077fe640 1 -- 172.21.15.59:0/2716061821 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb01407b300 con 0x7fb020162470 2024-03-06T07:23:18.660 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 -- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb00c06b4d0 msgr2=0x7fb00c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb00c06b4d0 0x7fb00c06d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb020161710 tx=0x7fb018009040 comp rx=0 tx=0).stop 2024-03-06T07:23:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 -- 172.21.15.59:0/2716061821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb020162470 msgr2=0x7fb0200c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb020162470 0x7fb0200c6e10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb01402bf70 tx=0x7fb0140046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 -- 172.21.15.59:0/2716061821 shutdown_connections 2024-03-06T07:23:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb00c06b4d0 0x7fb00c06d990 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb020162850 0x7fb0200c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb020162470 0x7fb0200c6e10 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 --2- 172.21.15.59:0/2716061821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0200d1f90 0x7fb0201552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 -- 172.21.15.59:0/2716061821 >> 172.21.15.59:0/2716061821 conn(0x7fb02009a9f0 msgr2=0x7fb02006c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 -- 172.21.15.59:0/2716061821 shutdown_connections 2024-03-06T07:23:18.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.661+0000 7fb02549a640 1 -- 172.21.15.59:0/2716061821 wait complete. 2024-03-06T07:23:18.677 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:18.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.787+0000 7f121f01c640 1 Processor -- start 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.787+0000 7f121f01c640 1 -- start start 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.788+0000 7f121f01c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f12180b57b0 0x7f121806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.788+0000 7f121f01c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121806f270 con 0x7f121815bd60 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.788+0000 7f121f01c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121806f550 con 0x7f12180cc6e0 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.788+0000 7f121f01c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f121815f750 con 0x7f12180b57b0 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.788+0000 7f121cdb8640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f121815bd60 0x7f12180b50a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54412/0 (socket says 172.21.15.59:54412) 2024-03-06T07:23:18.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.788+0000 7f121cdb8640 1 -- 172.21.15.59:0/2978207292 learned_addr learned my addr 172.21.15.59:0/2978207292 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:18.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.788+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151274648 0 0) 0x7f121806f270 con 0x7f121815bd60 2024-03-06T07:23:18.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.788+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1200003520 con 0x7f121815bd60 2024-03-06T07:23:18.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.788+0000 7f1217fff640 1 --2- 172.21.15.59:0/2978207292 >> v2:172.21.15.164:3300/0 conn(0x7f12180b57b0 0x7f121806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1391535862 0 0) 0x7f1200003520 con 0x7f121815bd60 2024-03-06T07:23:18.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 >> v2:172.21.15.164:3300/0 conn(0x7f12180b57b0 msgr2=0x7f121806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:18.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 --2- 172.21.15.59:0/2978207292 >> v2:172.21.15.164:3300/0 conn(0x7f12180b57b0 0x7f121806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 >> v1:172.21.15.164:6789/0 conn(0x7f12180cc6e0 legacy=0x7f12180ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121815f9a0 con 0x7f121815bd60 2024-03-06T07:23:18.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f12100019f0 con 0x7f121815bd60 2024-03-06T07:23:18.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 >> v1:172.21.15.59:6789/0 conn(0x7f121815bd60 legacy=0x7f12180b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 --2- 172.21.15.59:0/2978207292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12000039b0 0x7f1200003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 --2- 172.21.15.59:0/2978207292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1200004800 0x7f1200004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 --2- 172.21.15.59:0/2978207292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12000052e0 0x7f120000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121815f750 con 0x7f12000039b0 2024-03-06T07:23:18.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120000c1b0 con 0x7f12000052e0 2024-03-06T07:23:18.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120000c3b0 con 0x7f1200004800 2024-03-06T07:23:18.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f121cdb8640 1 --2- 172.21.15.59:0/2978207292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12000039b0 0x7f1200003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f121d5b9640 1 --2- 172.21.15.59:0/2978207292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12000052e0 0x7f120000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f1217fff640 1 --2- 172.21.15.59:0/2978207292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1200004800 0x7f1200004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f121cdb8640 1 -- 172.21.15.59:0/2978207292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1200004800 msgr2=0x7f1200004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:18.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f121cdb8640 1 --2- 172.21.15.59:0/2978207292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1200004800 0x7f1200004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f121cdb8640 1 -- 172.21.15.59:0/2978207292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12000052e0 msgr2=0x7f120000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:18.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f121cdb8640 1 --2- 172.21.15.59:0/2978207292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12000052e0 0x7f120000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f121d5b9640 1 -- 172.21.15.59:0/2978207292 reap_dead start 2024-03-06T07:23:18.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.789+0000 7f121cdb8640 1 -- 172.21.15.59:0/2978207292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1200003520 con 0x7f12000039b0 2024-03-06T07:23:18.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.790+0000 7f121cdb8640 1 --2- 172.21.15.59:0/2978207292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12000039b0 0x7f1200003dd0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f121000a810 tx=0x7f1200005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=513c9967fe05e9fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.790+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121000a460 con 0x7f12000039b0 2024-03-06T07:23:18.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.790+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1210034650 con 0x7f12000039b0 2024-03-06T07:23:18.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.790+0000 7f12177fe640 1 -- 172.21.15.59:0/2978207292 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1210034a40 con 0x7f12000039b0 2024-03-06T07:23:18.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.791+0000 7f121f01c640 1 -- 172.21.15.59:0/2978207292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12000039b0 msgr2=0x7f1200003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.791+0000 7f121f01c640 1 --2- 172.21.15.59:0/2978207292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12000039b0 0x7f1200003dd0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f121000a810 tx=0x7f1200005830 comp rx=0 tx=0).stop 2024-03-06T07:23:18.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.791+0000 7f121f01c640 1 -- 172.21.15.59:0/2978207292 shutdown_connections 2024-03-06T07:23:18.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.791+0000 7f121f01c640 1 --2- 172.21.15.59:0/2978207292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12000039b0 0x7f1200003dd0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.791+0000 7f121f01c640 1 -- 172.21.15.59:0/2978207292 >> 172.21.15.59:0/2978207292 conn(0x7f12180b1a90 msgr2=0x7f12180b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.791+0000 7f121f01c640 1 -- 172.21.15.59:0/2978207292 shutdown_connections 2024-03-06T07:23:18.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.791+0000 7f121f01c640 1 -- 172.21.15.59:0/2978207292 wait complete. 2024-03-06T07:23:18.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.792+0000 7f121f01c640 1 Processor -- start 2024-03-06T07:23:18.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.792+0000 7f121f01c640 1 -- start start 2024-03-06T07:23:18.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f121f01c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12000039b0 0x7f121814d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f121f01c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12181e6360 0x7f12181e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f121f01c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12180b3210 0x7f12180b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f121f01c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121806e260 con 0x7f12181e6360 2024-03-06T07:23:18.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f121f01c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121806e0e0 con 0x7f12180b3210 2024-03-06T07:23:18.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f121f01c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121815fd10 con 0x7f12000039b0 2024-03-06T07:23:18.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f1217fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12181e6360 0x7f12181e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f1217fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12181e6360 0x7f12181e67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40180/0 (socket says 172.21.15.59:40180) 2024-03-06T07:23:18.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f1217fff640 1 -- 172.21.15.59:0/4035560253 learned_addr learned my addr 172.21.15.59:0/4035560253 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:18.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f121cdb8640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12000039b0 0x7f121814d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f121d5b9640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12180b3210 0x7f12180b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f1217fff640 1 -- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12000039b0 msgr2=0x7f121814d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f1217fff640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12000039b0 0x7f121814d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f1217fff640 1 -- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12180b3210 msgr2=0x7f12180b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f1217fff640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12180b3210 0x7f12180b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f1217fff640 1 -- 172.21.15.59:0/4035560253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12180b3e20 con 0x7f12181e6360 2024-03-06T07:23:18.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f1217fff640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12181e6360 0x7f12181e67e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f120c011730 tx=0x7f120c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f12157fa640 1 -- 172.21.15.59:0/4035560253 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c00f960 con 0x7f12181e6360 2024-03-06T07:23:18.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f12157fa640 1 -- 172.21.15.59:0/4035560253 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c00fb00 con 0x7f12181e6360 2024-03-06T07:23:18.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f12157fa640 1 -- 172.21.15.59:0/4035560253 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c015460 con 0x7f12181e6360 2024-03-06T07:23:18.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.793+0000 7f121f01c640 1 -- 172.21.15.59:0/4035560253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12180b4100 con 0x7f12181e6360 2024-03-06T07:23:18.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.794+0000 7f121f01c640 1 -- 172.21.15.59:0/4035560253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12180b46c0 con 0x7f12181e6360 2024-03-06T07:23:18.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.794+0000 7f121f01c640 1 -- 172.21.15.59:0/4035560253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121804f330 con 0x7f12181e6360 2024-03-06T07:23:18.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.795+0000 7f12157fa640 1 -- 172.21.15.59:0/4035560253 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f120c015600 con 0x7f12181e6360 2024-03-06T07:23:18.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.795+0000 7f12157fa640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f120406b4d0 0x7f120406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.795+0000 7f12157fa640 1 -- 172.21.15.59:0/4035560253 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f120c08a9e0 con 0x7f12181e6360 2024-03-06T07:23:18.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.796+0000 7f121cdb8640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f120406b4d0 0x7f120406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.796+0000 7f121cdb8640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f120406b4d0 0x7f120406d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1210034000 tx=0x7f1210009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.800+0000 7f12157fa640 1 -- 172.21.15.59:0/4035560253 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f120c019030 con 0x7f12181e6360 2024-03-06T07:23:18.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:18.982+0000 7f121f01c640 1 -- 172.21.15.59:0/4035560253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f121814e6d0 con 0x7f12181e6360 2024-03-06T07:23:19.132 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:19.135 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:19.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.663+0000 7f12157fa640 1 -- 172.21.15.59:0/4035560253 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) v1 ==== 169+0+0 (secure 0 0 0) 0x7f120c0599c0 con 0x7f12181e6360 2024-03-06T07:23:19.665 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-03-06T07:23:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.670+0000 7f121f01c640 1 -- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f120406b4d0 msgr2=0x7f120406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.670+0000 7f121f01c640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f120406b4d0 0x7f120406d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1210034000 tx=0x7f1210009030 comp rx=0 tx=0).stop 2024-03-06T07:23:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.670+0000 7f121f01c640 1 -- 172.21.15.59:0/4035560253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12181e6360 msgr2=0x7f12181e67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.670+0000 7f121f01c640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12181e6360 0x7f12181e67e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f120c011730 tx=0x7f120c011bf0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.670+0000 7f121f01c640 1 -- 172.21.15.59:0/4035560253 shutdown_connections 2024-03-06T07:23:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.670+0000 7f121f01c640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f120406b4d0 0x7f120406d990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.670+0000 7f121f01c640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12180b3210 0x7f12180b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.670+0000 7f121f01c640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12181e6360 0x7f12181e67e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.670+0000 7f121f01c640 1 --2- 172.21.15.59:0/4035560253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12000039b0 0x7f121814d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.670+0000 7f121f01c640 1 -- 172.21.15.59:0/4035560253 >> 172.21.15.59:0/4035560253 conn(0x7f12180b1a90 msgr2=0x7f12180b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.670+0000 7f121f01c640 1 -- 172.21.15.59:0/4035560253 shutdown_connections 2024-03-06T07:23:19.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.671+0000 7f121f01c640 1 -- 172.21.15.59:0/4035560253 wait complete. 2024-03-06T07:23:19.691 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:23:19.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.796+0000 7fd7658e0640 1 Processor -- start 2024-03-06T07:23:19.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.796+0000 7fd7658e0640 1 -- start start 2024-03-06T07:23:19.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.797+0000 7fd7658e0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd76015c050 0x7fd76006f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.797+0000 7fd7658e0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd76006fb40 0x7fd7600cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.797+0000 7fd7658e0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7600ccca0 0x7fd7600cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.797+0000 7fd7658e0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd76006ffb0 con 0x7fd76006fb40 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.797+0000 7fd7658e0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7600cd4b0 con 0x7fd76015c050 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.797+0000 7fd7658e0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600cd790 con 0x7fd7600ccca0 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.797+0000 7fd75e7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd76006fb40 0x7fd7600cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.797+0000 7fd75e7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd76006fb40 0x7fd7600cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40194/0 (socket says 172.21.15.59:40194) 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.797+0000 7fd75e7fc640 1 -- 172.21.15.59:0/1600841316 learned_addr learned my addr 172.21.15.59:0/1600841316 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.798+0000 7fd75effd640 1 --2- 172.21.15.59:0/1600841316 >> v2:172.21.15.164:3300/0 conn(0x7fd76015c050 0x7fd76006f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.798+0000 7fd75f7fe640 1 --2- 172.21.15.59:0/1600841316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7600ccca0 0x7fd7600cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.798+0000 7fd75e7fc640 1 -- 172.21.15.59:0/1600841316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7600ccca0 msgr2=0x7fd7600cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.798+0000 7fd75e7fc640 1 --2- 172.21.15.59:0/1600841316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7600ccca0 0x7fd7600cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.798+0000 7fd75e7fc640 1 -- 172.21.15.59:0/1600841316 >> v2:172.21.15.164:3300/0 conn(0x7fd76015c050 msgr2=0x7fd76006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.798+0000 7fd75e7fc640 1 --2- 172.21.15.59:0/1600841316 >> v2:172.21.15.164:3300/0 conn(0x7fd76015c050 0x7fd76006f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.798+0000 7fd75e7fc640 1 -- 172.21.15.59:0/1600841316 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7600cda70 con 0x7fd76006fb40 2024-03-06T07:23:19.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.798+0000 7fd75effd640 1 --2- 172.21.15.59:0/1600841316 >> v2:172.21.15.164:3300/0 conn(0x7fd76015c050 0x7fd76006f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:19.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.798+0000 7fd75e7fc640 1 --2- 172.21.15.59:0/1600841316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd76006fb40 0x7fd7600cc760 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd760070ef0 tx=0x7fd74802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b392511bc7e61c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd75dffb640 1 -- 172.21.15.59:0/1600841316 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74802e070 con 0x7fd76006fb40 2024-03-06T07:23:19.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd75dffb640 1 -- 172.21.15.59:0/1600841316 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd748037070 con 0x7fd76006fb40 2024-03-06T07:23:19.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd75dffb640 1 -- 172.21.15.59:0/1600841316 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd748031a00 con 0x7fd76006fb40 2024-03-06T07:23:19.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd7658e0640 1 -- 172.21.15.59:0/1600841316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd76006fb40 msgr2=0x7fd7600cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd7658e0640 1 --2- 172.21.15.59:0/1600841316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd76006fb40 0x7fd7600cc760 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd760070ef0 tx=0x7fd74802b880 comp rx=0 tx=0).stop 2024-03-06T07:23:19.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd7658e0640 1 -- 172.21.15.59:0/1600841316 shutdown_connections 2024-03-06T07:23:19.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd7658e0640 1 --2- 172.21.15.59:0/1600841316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7600ccca0 0x7fd7600cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd7658e0640 1 --2- 172.21.15.59:0/1600841316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd76006fb40 0x7fd7600cc760 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd7658e0640 1 --2- 172.21.15.59:0/1600841316 >> v2:172.21.15.164:3300/0 conn(0x7fd76015c050 0x7fd76006f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd7658e0640 1 -- 172.21.15.59:0/1600841316 >> 172.21.15.59:0/1600841316 conn(0x7fd7600b3800 msgr2=0x7fd7600b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd7658e0640 1 -- 172.21.15.59:0/1600841316 shutdown_connections 2024-03-06T07:23:19.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.799+0000 7fd7658e0640 1 -- 172.21.15.59:0/1600841316 wait complete. 2024-03-06T07:23:19.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.800+0000 7fd7658e0640 1 Processor -- start 2024-03-06T07:23:19.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.800+0000 7fd7658e0640 1 -- start start 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.801+0000 7fd7658e0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd76006fb40 0x7fd76014a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.801+0000 7fd7658e0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7600ccca0 0x7fd7600b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.801+0000 7fd7658e0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd76015c050 0x7fd7600b6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.801+0000 7fd7658e0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600d0020 con 0x7fd76006fb40 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.801+0000 7fd7658e0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600cd130 con 0x7fd7600ccca0 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.801+0000 7fd7658e0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600d01a0 con 0x7fd76015c050 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.801+0000 7fd75effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd76006fb40 0x7fd76014a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.801+0000 7fd75effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd76006fb40 0x7fd76014a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40208/0 (socket says 172.21.15.59:40208) 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.801+0000 7fd75effd640 1 -- 172.21.15.59:0/3436278356 learned_addr learned my addr 172.21.15.59:0/3436278356 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.802+0000 7fd75effd640 1 -- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd76015c050 msgr2=0x7fd7600b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.802+0000 7fd75effd640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd76015c050 0x7fd7600b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.802+0000 7fd75effd640 1 -- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7600ccca0 msgr2=0x7fd7600b6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.802+0000 7fd75effd640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7600ccca0 0x7fd7600b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.802+0000 7fd75effd640 1 -- 172.21.15.59:0/3436278356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7600b5740 con 0x7fd76006fb40 2024-03-06T07:23:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.802+0000 7fd75effd640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd76006fb40 0x7fd76014a560 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c820 tx=0x7fd75800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.802+0000 7fd757fff640 1 -- 172.21.15.59:0/3436278356 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758015070 con 0x7fd76006fb40 2024-03-06T07:23:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.802+0000 7fd757fff640 1 -- 172.21.15.59:0/3436278356 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd758007a30 con 0x7fd76006fb40 2024-03-06T07:23:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.802+0000 7fd7658e0640 1 -- 172.21.15.59:0/3436278356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7600b5a20 con 0x7fd76006fb40 2024-03-06T07:23:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.802+0000 7fd757fff640 1 -- 172.21.15.59:0/3436278356 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758007e00 con 0x7fd76006fb40 2024-03-06T07:23:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.802+0000 7fd7658e0640 1 -- 172.21.15.59:0/3436278356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76014aaa0 con 0x7fd76006fb40 2024-03-06T07:23:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.803+0000 7fd7658e0640 1 -- 172.21.15.59:0/3436278356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76009fe60 con 0x7fd76006fb40 2024-03-06T07:23:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.803+0000 7fd757fff640 1 -- 172.21.15.59:0/3436278356 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd758018070 con 0x7fd76006fb40 2024-03-06T07:23:19.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.804+0000 7fd757fff640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd74006b4d0 0x7fd74006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.804+0000 7fd757fff640 1 -- 172.21.15.59:0/3436278356 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd758089ee0 con 0x7fd76006fb40 2024-03-06T07:23:19.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.804+0000 7fd75e7fc640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd74006b4d0 0x7fd74006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.805+0000 7fd75e7fc640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd74006b4d0 0x7fd74006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd748036000 tx=0x7fd748030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.811+0000 7fd757fff640 1 -- 172.21.15.59:0/3436278356 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd758059060 con 0x7fd76006fb40 2024-03-06T07:23:19.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:19.997+0000 7fd7658e0640 1 -- 172.21.15.59:0/3436278356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd7600b0ad0 con 0x7fd76006fb40 2024-03-06T07:23:20.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.000+0000 7fd757fff640 1 -- 172.21.15.59:0/3436278356 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd758058960 con 0x7fd76006fb40 2024-03-06T07:23:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.001+0000 7fd7658e0640 1 -- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd74006b4d0 msgr2=0x7fd74006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.001+0000 7fd7658e0640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd74006b4d0 0x7fd74006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd748036000 tx=0x7fd748030000 comp rx=0 tx=0).stop 2024-03-06T07:23:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.001+0000 7fd7658e0640 1 -- 172.21.15.59:0/3436278356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd76006fb40 msgr2=0x7fd76014a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.001+0000 7fd7658e0640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd76006fb40 0x7fd76014a560 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c820 tx=0x7fd75800cc20 comp rx=0 tx=0).stop 2024-03-06T07:23:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.002+0000 7fd7658e0640 1 -- 172.21.15.59:0/3436278356 shutdown_connections 2024-03-06T07:23:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.002+0000 7fd7658e0640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd74006b4d0 0x7fd74006d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.002+0000 7fd7658e0640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd76015c050 0x7fd7600b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.002+0000 7fd7658e0640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7600ccca0 0x7fd7600b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.002+0000 7fd7658e0640 1 --2- 172.21.15.59:0/3436278356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd76006fb40 0x7fd76014a560 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.002+0000 7fd7658e0640 1 -- 172.21.15.59:0/3436278356 >> 172.21.15.59:0/3436278356 conn(0x7fd7600b3800 msgr2=0x7fd7600afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.002+0000 7fd7658e0640 1 -- 172.21.15.59:0/3436278356 shutdown_connections 2024-03-06T07:23:20.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.002+0000 7fd7658e0640 1 -- 172.21.15.59:0/3436278356 wait complete. 2024-03-06T07:23:20.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:23:20.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.115+0000 7fc5c0c37640 1 Processor -- start 2024-03-06T07:23:20.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.115+0000 7fc5c0c37640 1 -- start start 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.116+0000 7fc5c0c37640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5bc06b220 0x7fc5bc15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.116+0000 7fc5c0c37640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc5bc06b600 0x7fc5bc162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.116+0000 7fc5c0c37640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc15ea30 con 0x7fc5bc0ce920 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.116+0000 7fc5c0c37640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5bc161b70 con 0x7fc5bc06b600 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.116+0000 7fc5c0c37640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc15ed10 con 0x7fc5bc06b220 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.116+0000 7fc5bad9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc5bc06b600 0x7fc5bc162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.116+0000 7fc5ba59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5bc06b220 0x7fc5bc15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.116+0000 7fc5b9d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc5bc0ce920 0x7fc5bc071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:55392/0 (socket says 172.21.15.59:55392) 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.116+0000 7fc5b9d9b640 1 -- 172.21.15.59:0/1824785088 learned_addr learned my addr 172.21.15.59:0/1824785088 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:20.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.117+0000 7fc5bad9d640 1 -- 172.21.15.59:0/1824785088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5bc06b220 msgr2=0x7fc5bc15e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.117+0000 7fc5bad9d640 1 --2- 172.21.15.59:0/1824785088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5bc06b220 0x7fc5bc15e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.117+0000 7fc5bad9d640 1 -- 172.21.15.59:0/1824785088 >> v1:172.21.15.164:6789/0 conn(0x7fc5bc0ce920 legacy=0x7fc5bc071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.117+0000 7fc5bad9d640 1 -- 172.21.15.59:0/1824785088 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc15eff0 con 0x7fc5bc06b600 2024-03-06T07:23:20.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.117+0000 7fc5bad9d640 1 --2- 172.21.15.59:0/1824785088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5bc06b600 0x7fc5bc162000 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac009d90 tx=0x7fc5ac02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8e5c91688556eda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.118+0000 7fc5b959a640 1 -- 172.21.15.59:0/1824785088 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac039070 con 0x7fc5bc06b600 2024-03-06T07:23:20.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.118+0000 7fc5b959a640 1 -- 172.21.15.59:0/1824785088 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac02cc50 con 0x7fc5bc06b600 2024-03-06T07:23:20.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.118+0000 7fc5b959a640 1 -- 172.21.15.59:0/1824785088 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac034440 con 0x7fc5bc06b600 2024-03-06T07:23:20.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.118+0000 7fc5c0c37640 1 -- 172.21.15.59:0/1824785088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5bc06b600 msgr2=0x7fc5bc162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.118+0000 7fc5c0c37640 1 --2- 172.21.15.59:0/1824785088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5bc06b600 0x7fc5bc162000 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac009d90 tx=0x7fc5ac02bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.118+0000 7fc5c0c37640 1 -- 172.21.15.59:0/1824785088 shutdown_connections 2024-03-06T07:23:20.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.118+0000 7fc5c0c37640 1 --2- 172.21.15.59:0/1824785088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5bc06b600 0x7fc5bc162000 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.118+0000 7fc5c0c37640 1 --2- 172.21.15.59:0/1824785088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5bc06b220 0x7fc5bc15e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.118+0000 7fc5c0c37640 1 -- 172.21.15.59:0/1824785088 >> 172.21.15.59:0/1824785088 conn(0x7fc5bc06c0c0 msgr2=0x7fc5bc06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.118+0000 7fc5c0c37640 1 -- 172.21.15.59:0/1824785088 shutdown_connections 2024-03-06T07:23:20.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.118+0000 7fc5c0c37640 1 -- 172.21.15.59:0/1824785088 wait complete. 2024-03-06T07:23:20.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.119+0000 7fc5c0c37640 1 Processor -- start 2024-03-06T07:23:20.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.120+0000 7fc5c0c37640 1 -- start start 2024-03-06T07:23:20.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.120+0000 7fc5c0c37640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5bc06b220 0x7fc5bc14a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.120+0000 7fc5c0c37640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5bc06b600 0x7fc5bc0b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.120+0000 7fc5c0c37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5bc0ce920 0x7fc5bc0b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.120+0000 7fc5c0c37640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc072090 con 0x7fc5bc0ce920 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.120+0000 7fc5c0c37640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc0ce200 con 0x7fc5bc06b600 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.120+0000 7fc5c0c37640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc0b2160 con 0x7fc5bc06b220 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.120+0000 7fc5bad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5bc0ce920 0x7fc5bc0b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.120+0000 7fc5bad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5bc0ce920 0x7fc5bc0b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40212/0 (socket says 172.21.15.59:40212) 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.120+0000 7fc5bad9d640 1 -- 172.21.15.59:0/3814913549 learned_addr learned my addr 172.21.15.59:0/3814913549 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5ba59c640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5bc06b220 0x7fc5bc14a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5b9d9b640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5bc06b600 0x7fc5bc0b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5bad9d640 1 -- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5bc06b220 msgr2=0x7fc5bc14a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5bad9d640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5bc06b220 0x7fc5bc14a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5bad9d640 1 -- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5bc06b600 msgr2=0x7fc5bc0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5bad9d640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5bc06b600 0x7fc5bc0b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5bad9d640 1 -- 172.21.15.59:0/3814913549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc14aa20 con 0x7fc5bc0ce920 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5ba59c640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5bc06b220 0x7fc5bc14a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:20.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5bad9d640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5bc0ce920 0x7fc5bc0b3b80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac009d90 tx=0x7fc5ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5b2ffd640 1 -- 172.21.15.59:0/3814913549 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac039050 con 0x7fc5bc0ce920 2024-03-06T07:23:20.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5b2ffd640 1 -- 172.21.15.59:0/3814913549 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac045070 con 0x7fc5bc0ce920 2024-03-06T07:23:20.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5c0c37640 1 -- 172.21.15.59:0/3814913549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc0b10c0 con 0x7fc5bc0ce920 2024-03-06T07:23:20.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5b2ffd640 1 -- 172.21.15.59:0/3814913549 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac009450 con 0x7fc5bc0ce920 2024-03-06T07:23:20.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.121+0000 7fc5c0c37640 1 -- 172.21.15.59:0/3814913549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc0b1680 con 0x7fc5bc0ce920 2024-03-06T07:23:20.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.122+0000 7fc5c0c37640 1 -- 172.21.15.59:0/3814913549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc072430 con 0x7fc5bc0ce920 2024-03-06T07:23:20.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.123+0000 7fc5b2ffd640 1 -- 172.21.15.59:0/3814913549 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5ac03e610 con 0x7fc5bc0ce920 2024-03-06T07:23:20.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.123+0000 7fc5b2ffd640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc59c06b4d0 0x7fc59c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.123+0000 7fc5b2ffd640 1 -- 172.21.15.59:0/3814913549 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc5ac0ac9a0 con 0x7fc5bc0ce920 2024-03-06T07:23:20.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.123+0000 7fc5ba59c640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc59c06b4d0 0x7fc59c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.124+0000 7fc5ba59c640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc59c06b4d0 0x7fc59c06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009af0 tx=0x7fc5b4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.127+0000 7fc5b2ffd640 1 -- 172.21.15.59:0/3814913549 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5bc14aa20 con 0x7fc5bc0ce920 2024-03-06T07:23:20.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.376+0000 7fc5c0c37640 1 -- 172.21.15.59:0/3814913549 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7fc5bc04f330 con 0x7fc5bc0ce920 2024-03-06T07:23:20.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.664+0000 7fc5b2ffd640 1 -- 172.21.15.59:0/3814913549 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v102) v1 ==== 132+0+0 (secure 0 0 0) 0x7fc5ac07bbb0 con 0x7fc5bc0ce920 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.666+0000 7fc5c0c37640 1 -- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc59c06b4d0 msgr2=0x7fc59c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.666+0000 7fc5c0c37640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc59c06b4d0 0x7fc59c06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009af0 tx=0x7fc5b4008040 comp rx=0 tx=0).stop 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.666+0000 7fc5c0c37640 1 -- 172.21.15.59:0/3814913549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5bc0ce920 msgr2=0x7fc5bc0b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.666+0000 7fc5c0c37640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5bc0ce920 0x7fc5bc0b3b80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac009d90 tx=0x7fc5ac004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.666+0000 7fc5c0c37640 1 -- 172.21.15.59:0/3814913549 shutdown_connections 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.667+0000 7fc5c0c37640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc59c06b4d0 0x7fc59c06d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.667+0000 7fc5c0c37640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5bc0ce920 0x7fc5bc0b3b80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.667+0000 7fc5c0c37640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5bc06b600 0x7fc5bc0b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.667+0000 7fc5c0c37640 1 --2- 172.21.15.59:0/3814913549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5bc06b220 0x7fc5bc14a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.667+0000 7fc5c0c37640 1 -- 172.21.15.59:0/3814913549 >> 172.21.15.59:0/3814913549 conn(0x7fc5bc06c0c0 msgr2=0x7fc5bc161280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.667+0000 7fc5c0c37640 1 -- 172.21.15.59:0/3814913549 shutdown_connections 2024-03-06T07:23:20.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.667+0000 7fc5c0c37640 1 -- 172.21.15.59:0/3814913549 wait complete. 2024-03-06T07:23:20.680 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-03-06T07:23:20.681 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-03-06T07:23:20.681 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:23:20.681 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:20.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.782+0000 7f355215c640 1 Processor -- start 2024-03-06T07:23:20.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.782+0000 7f355215c640 1 -- start start 2024-03-06T07:23:20.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f355215c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f354c0b51d0 0x7f354c0c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f355215c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 0x7f354c0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f355215c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f354c0d0c20 0x7f354c0d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f355215c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f354c0d1090 con 0x7f354c0b51d0 2024-03-06T07:23:20.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f355215c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f354c0b6b20 con 0x7f354c0d0c20 2024-03-06T07:23:20.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f355215c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354c0d4c50 con 0x7f354c0d47f0 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f354b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f354c0b51d0 0x7f354c0c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f354b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f354c0b51d0 0x7f354c0c6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40226/0 (socket says 172.21.15.59:40226) 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f354b7fe640 1 -- 172.21.15.59:0/3804204438 learned_addr learned my addr 172.21.15.59:0/3804204438 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f354bfff640 1 --2- 172.21.15.59:0/3804204438 >> v2:172.21.15.164:3300/0 conn(0x7f354c0d0c20 0x7f354c0d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f354affd640 1 --2- 172.21.15.59:0/3804204438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 0x7f354c0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f354b7fe640 1 -- 172.21.15.59:0/3804204438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 msgr2=0x7f354c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f354b7fe640 1 --2- 172.21.15.59:0/3804204438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 0x7f354c0d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f354b7fe640 1 -- 172.21.15.59:0/3804204438 >> v2:172.21.15.164:3300/0 conn(0x7f354c0d0c20 msgr2=0x7f354c0d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f354b7fe640 1 --2- 172.21.15.59:0/3804204438 >> v2:172.21.15.164:3300/0 conn(0x7f354c0d0c20 0x7f354c0d1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.783+0000 7f354b7fe640 1 -- 172.21.15.59:0/3804204438 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c162b20 con 0x7f354c0b51d0 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.784+0000 7f354bfff640 1 --2- 172.21.15.59:0/3804204438 >> v2:172.21.15.164:3300/0 conn(0x7f354c0d0c20 0x7f354c0d1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.784+0000 7f354affd640 1 --2- 172.21.15.59:0/3804204438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 0x7f354c0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.784+0000 7f354b7fe640 1 --2- 172.21.15.59:0/3804204438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f354c0b51d0 0x7f354c0c6cb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f354c08d120 tx=0x7f353c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de8ded70f597399 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.784+0000 7f354a7fc640 1 -- 172.21.15.59:0/3804204438 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c02e070 con 0x7f354c0b51d0 2024-03-06T07:23:20.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.784+0000 7f354a7fc640 1 -- 172.21.15.59:0/3804204438 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c037070 con 0x7f354c0b51d0 2024-03-06T07:23:20.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.784+0000 7f354a7fc640 1 -- 172.21.15.59:0/3804204438 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c031a00 con 0x7f354c0b51d0 2024-03-06T07:23:20.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.785+0000 7f355215c640 1 -- 172.21.15.59:0/3804204438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f354c0b51d0 msgr2=0x7f354c0c6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.785+0000 7f355215c640 1 --2- 172.21.15.59:0/3804204438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f354c0b51d0 0x7f354c0c6cb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f354c08d120 tx=0x7f353c02b880 comp rx=0 tx=0).stop 2024-03-06T07:23:20.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.785+0000 7f355215c640 1 -- 172.21.15.59:0/3804204438 shutdown_connections 2024-03-06T07:23:20.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.785+0000 7f355215c640 1 --2- 172.21.15.59:0/3804204438 >> v2:172.21.15.164:3300/0 conn(0x7f354c0d0c20 0x7f354c0d1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.785+0000 7f355215c640 1 --2- 172.21.15.59:0/3804204438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 0x7f354c0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.785+0000 7f355215c640 1 --2- 172.21.15.59:0/3804204438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f354c0b51d0 0x7f354c0c6cb0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.785+0000 7f355215c640 1 -- 172.21.15.59:0/3804204438 >> 172.21.15.59:0/3804204438 conn(0x7f354c0d3cd0 msgr2=0x7f354c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.785+0000 7f355215c640 1 -- 172.21.15.59:0/3804204438 shutdown_connections 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.785+0000 7f355215c640 1 -- 172.21.15.59:0/3804204438 wait complete. 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.786+0000 7f355215c640 1 Processor -- start 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.786+0000 7f355215c640 1 -- start start 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f355215c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f354c0b51d0 0x7f354c1f7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f355215c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f354c0d0c20 0x7f354c15caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f355215c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 0x7f354c06c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f355215c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c0cf6a0 con 0x7f354c0b51d0 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f355215c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c0cf400 con 0x7f354c0d0c20 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f355215c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354c0cf980 con 0x7f354c0d47f0 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f354b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f354c0b51d0 0x7f354c1f7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f354b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f354c0b51d0 0x7f354c1f7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40236/0 (socket says 172.21.15.59:40236) 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f354b7fe640 1 -- 172.21.15.59:0/4254337227 learned_addr learned my addr 172.21.15.59:0/4254337227 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:20.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f354affd640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f354c0d0c20 0x7f354c15caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f354bfff640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 0x7f354c06c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f354b7fe640 1 -- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 msgr2=0x7f354c06c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f354b7fe640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 0x7f354c06c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f354b7fe640 1 -- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f354c0d0c20 msgr2=0x7f354c15caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f354b7fe640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f354c0d0c20 0x7f354c15caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f354b7fe640 1 -- 172.21.15.59:0/4254337227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c1f8140 con 0x7f354c0b51d0 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.787+0000 7f354bfff640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 0x7f354c06c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.788+0000 7f354b7fe640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f354c0b51d0 0x7f354c1f7c00 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f353c02b8c0 tx=0x7f353c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.788+0000 7f3533fff640 1 -- 172.21.15.59:0/4254337227 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c032880 con 0x7f354c0b51d0 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.788+0000 7f3533fff640 1 -- 172.21.15.59:0/4254337227 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c032a20 con 0x7f354c0b51d0 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.788+0000 7f355215c640 1 -- 172.21.15.59:0/4254337227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354c151030 con 0x7f354c0b51d0 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.788+0000 7f3533fff640 1 -- 172.21.15.59:0/4254337227 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c032e10 con 0x7f354c0b51d0 2024-03-06T07:23:20.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.788+0000 7f355215c640 1 -- 172.21.15.59:0/4254337227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f354c1518d0 con 0x7f354c0b51d0 2024-03-06T07:23:20.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.789+0000 7f355215c640 1 -- 172.21.15.59:0/4254337227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354c151420 con 0x7f354c0b51d0 2024-03-06T07:23:20.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.789+0000 7f3533fff640 1 -- 172.21.15.59:0/4254337227 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f353c04c060 con 0x7f354c0b51d0 2024-03-06T07:23:20.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.790+0000 7f3533fff640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f353806b4d0 0x7f353806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.790+0000 7f3533fff640 1 -- 172.21.15.59:0/4254337227 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f353c0ab910 con 0x7f354c0b51d0 2024-03-06T07:23:20.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.790+0000 7f354affd640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f353806b4d0 0x7f353806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.790+0000 7f354affd640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f353806b4d0 0x7f353806d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f354c0b9b10 tx=0x7f3540008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.794+0000 7f3533fff640 1 -- 172.21.15.59:0/4254337227 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f353c040070 con 0x7f354c0b51d0 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.975+0000 7f355215c640 1 -- 172.21.15.59:0/4254337227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f354c0d2930 con 0x7f354c0b51d0 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.976+0000 7f3533fff640 1 -- 172.21.15.59:0/4254337227 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+13008 (secure 0 0 0) 0x7f353c07aa90 con 0x7f354c0b51d0 2024-03-06T07:23:20.978 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:20.978 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":36,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:23:19.659849+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6835/1359034023":"2024-03-07T07:23:16.652148+0000","172.21.15.59:6833/1359034023":"2024-03-07T07:23:16.652148+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:20.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 -- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f353806b4d0 msgr2=0x7f353806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f353806b4d0 0x7f353806d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f354c0b9b10 tx=0x7f3540008040 comp rx=0 tx=0).stop 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 -- 172.21.15.59:0/4254337227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f354c0b51d0 msgr2=0x7f354c1f7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f354c0b51d0 0x7f354c1f7c00 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f353c02b8c0 tx=0x7f353c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 -- 172.21.15.59:0/4254337227 shutdown_connections 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f353806b4d0 0x7f353806d990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f354c0d47f0 0x7f354c06c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f354c0d0c20 0x7f354c15caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 --2- 172.21.15.59:0/4254337227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f354c0b51d0 0x7f354c1f7c00 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 -- 172.21.15.59:0/4254337227 >> 172.21.15.59:0/4254337227 conn(0x7f354c0d3cd0 msgr2=0x7f354c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 -- 172.21.15.59:0/4254337227 shutdown_connections 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:20.978+0000 7f355215c640 1 -- 172.21.15.59:0/4254337227 wait complete. 2024-03-06T07:23:20.991 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:20.991 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:21.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.093+0000 7f58663d6640 1 Processor -- start 2024-03-06T07:23:21.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.093+0000 7f58663d6640 1 -- start start 2024-03-06T07:23:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f58663d6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f586015bdd0 0x7f58600b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f58663d6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f58600b15c0 0x7f58600b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f58663d6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58600d0240 con 0x7f58600ce300 2024-03-06T07:23:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f58663d6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58600cf4f0 con 0x7f586015bdd0 2024-03-06T07:23:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f58663d6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58600d0850 con 0x7f58600b15c0 2024-03-06T07:23:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f585ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f586015bdd0 0x7f58600b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f585ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f586015bdd0 0x7f58600b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40238/0 (socket says 172.21.15.59:40238) 2024-03-06T07:23:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f585ffff640 1 -- 172.21.15.59:0/3446789493 learned_addr learned my addr 172.21.15.59:0/3446789493 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f5864973640 1 --2- 172.21.15.59:0/3446789493 >> v2:172.21.15.164:3300/0 conn(0x7f58600b15c0 0x7f58600b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f585ffff640 1 -- 172.21.15.59:0/3446789493 >> v2:172.21.15.164:3300/0 conn(0x7f58600b15c0 msgr2=0x7f58600b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f585ffff640 1 --2- 172.21.15.59:0/3446789493 >> v2:172.21.15.164:3300/0 conn(0x7f58600b15c0 0x7f58600b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f585ffff640 1 -- 172.21.15.59:0/3446789493 >> v1:172.21.15.164:6789/0 conn(0x7f58600ce300 legacy=0x7f58600ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f585ffff640 1 -- 172.21.15.59:0/3446789493 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58600cf210 con 0x7f586015bdd0 2024-03-06T07:23:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.094+0000 7f5864973640 1 --2- 172.21.15.59:0/3446789493 >> v2:172.21.15.164:3300/0 conn(0x7f58600b15c0 0x7f58600b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.095+0000 7f585ffff640 1 --2- 172.21.15.59:0/3446789493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f586015bdd0 0x7f58600b41b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5860072b70 tx=0x7f585402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6dadf3d7b989f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.095+0000 7f585effd640 1 -- 172.21.15.59:0/3446789493 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5854035020 con 0x7f586015bdd0 2024-03-06T07:23:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.095+0000 7f585effd640 1 -- 172.21.15.59:0/3446789493 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5854031870 con 0x7f586015bdd0 2024-03-06T07:23:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.095+0000 7f585effd640 1 -- 172.21.15.59:0/3446789493 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5854031c40 con 0x7f586015bdd0 2024-03-06T07:23:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.096+0000 7f58663d6640 1 -- 172.21.15.59:0/3446789493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f586015bdd0 msgr2=0x7f58600b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.096+0000 7f58663d6640 1 --2- 172.21.15.59:0/3446789493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f586015bdd0 0x7f58600b41b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5860072b70 tx=0x7f585402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:21.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.096+0000 7f58663d6640 1 -- 172.21.15.59:0/3446789493 shutdown_connections 2024-03-06T07:23:21.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.096+0000 7f58663d6640 1 --2- 172.21.15.59:0/3446789493 >> v2:172.21.15.164:3300/0 conn(0x7f58600b15c0 0x7f58600b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.096+0000 7f58663d6640 1 --2- 172.21.15.59:0/3446789493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f586015bdd0 0x7f58600b41b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.096+0000 7f58663d6640 1 -- 172.21.15.59:0/3446789493 >> 172.21.15.59:0/3446789493 conn(0x7f58600b3690 msgr2=0x7f58600b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.096+0000 7f58663d6640 1 -- 172.21.15.59:0/3446789493 shutdown_connections 2024-03-06T07:23:21.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.096+0000 7f58663d6640 1 -- 172.21.15.59:0/3446789493 wait complete. 2024-03-06T07:23:21.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.097+0000 7f58663d6640 1 Processor -- start 2024-03-06T07:23:21.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.097+0000 7f58663d6640 1 -- start start 2024-03-06T07:23:21.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f58663d6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58600b15c0 0x7f5860156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f58663d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58600ce300 0x7f586006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f58663d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f586015bdd0 0x7f586006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f58663d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586015c1b0 con 0x7f58600ce300 2024-03-06T07:23:21.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f58663d6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58600ceea0 con 0x7f58600b15c0 2024-03-06T07:23:21.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f58663d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58600b10c0 con 0x7f586015bdd0 2024-03-06T07:23:21.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58600ce300 0x7f586006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f585f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58600ce300 0x7f586006d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40254/0 (socket says 172.21.15.59:40254) 2024-03-06T07:23:21.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f585f7fe640 1 -- 172.21.15.59:0/278967279 learned_addr learned my addr 172.21.15.59:0/278967279 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:21.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f5864973640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f586015bdd0 0x7f586006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f585ffff640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58600b15c0 0x7f5860156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f585f7fe640 1 -- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f586015bdd0 msgr2=0x7f586006dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f585f7fe640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f586015bdd0 0x7f586006dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f585f7fe640 1 -- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58600b15c0 msgr2=0x7f5860156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f585f7fe640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58600b15c0 0x7f5860156130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f585f7fe640 1 -- 172.21.15.59:0/278967279 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58601f3c50 con 0x7f58600ce300 2024-03-06T07:23:21.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f5864973640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f586015bdd0 0x7f586006dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:21.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.098+0000 7f585ffff640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58600b15c0 0x7f5860156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:21.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.099+0000 7f585f7fe640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58600ce300 0x7f586006d790 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f585000b3f0 tx=0x7f585000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.099+0000 7f585cff9640 1 -- 172.21.15.59:0/278967279 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5850015020 con 0x7f58600ce300 2024-03-06T07:23:21.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.099+0000 7f585cff9640 1 -- 172.21.15.59:0/278967279 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5850011830 con 0x7f58600ce300 2024-03-06T07:23:21.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.099+0000 7f58663d6640 1 -- 172.21.15.59:0/278967279 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5860156670 con 0x7f58600ce300 2024-03-06T07:23:21.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.099+0000 7f585cff9640 1 -- 172.21.15.59:0/278967279 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5850011c20 con 0x7f58600ce300 2024-03-06T07:23:21.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.099+0000 7f58663d6640 1 -- 172.21.15.59:0/278967279 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58601f41b0 con 0x7f58600ce300 2024-03-06T07:23:21.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.100+0000 7f58663d6640 1 -- 172.21.15.59:0/278967279 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f586009fcf0 con 0x7f58600ce300 2024-03-06T07:23:21.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.100+0000 7f585cff9640 1 -- 172.21.15.59:0/278967279 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5850018020 con 0x7f58600ce300 2024-03-06T07:23:21.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.101+0000 7f585cff9640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f584c06b4d0 0x7f584c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.101+0000 7f585cff9640 1 -- 172.21.15.59:0/278967279 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5850089de0 con 0x7f58600ce300 2024-03-06T07:23:21.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.101+0000 7f585ffff640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f584c06b4d0 0x7f584c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.101+0000 7f585ffff640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f584c06b4d0 0x7f584c06d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5854036000 tx=0x7f585403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.105+0000 7f585cff9640 1 -- 172.21.15.59:0/278967279 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f585001f020 con 0x7f58600ce300 2024-03-06T07:23:21.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.284+0000 7f58663d6640 1 -- 172.21.15.59:0/278967279 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f58600b30d0 con 0x7f58600ce300 2024-03-06T07:23:21.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.285+0000 7f585cff9640 1 -- 172.21.15.59:0/278967279 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f58601f3c50 con 0x7f58600ce300 2024-03-06T07:23:21.287 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:23:21.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 -- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f584c06b4d0 msgr2=0x7f584c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f584c06b4d0 0x7f584c06d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5854036000 tx=0x7f585403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:21.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 -- 172.21.15.59:0/278967279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58600ce300 msgr2=0x7f586006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58600ce300 0x7f586006d790 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f585000b3f0 tx=0x7f585000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 -- 172.21.15.59:0/278967279 shutdown_connections 2024-03-06T07:23:21.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f584c06b4d0 0x7f584c06d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f586015bdd0 0x7f586006dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58600ce300 0x7f586006d790 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 --2- 172.21.15.59:0/278967279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58600b15c0 0x7f5860156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 -- 172.21.15.59:0/278967279 >> 172.21.15.59:0/278967279 conn(0x7f58600b3690 msgr2=0x7f586006f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 -- 172.21.15.59:0/278967279 shutdown_connections 2024-03-06T07:23:21.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.287+0000 7f58663d6640 1 -- 172.21.15.59:0/278967279 wait complete. 2024-03-06T07:23:21.300 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:23:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.401+0000 7fcdd1b32640 1 Processor -- start 2024-03-06T07:23:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.401+0000 7fcdd1b32640 1 -- start start 2024-03-06T07:23:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.401+0000 7fcdd1b32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdcc15f200 0x7fcdcc0d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.401+0000 7fcdd1b32640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcdcc0d06a0 0x7fcdcc15d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.401+0000 7fcdd1b32640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdcc0cc950 con 0x7fcdcc15ee20 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.401+0000 7fcdd1b32640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdcc161fa0 con 0x7fcdcc0d06a0 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.401+0000 7fcdd1b32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc162280 con 0x7fcdcc15f200 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.402+0000 7fcdcbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcdcc0d06a0 0x7fcdcc15d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.402+0000 7fcdcbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcdcc0d06a0 0x7fcdcc15d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40262/0 (socket says 172.21.15.59:40262) 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.402+0000 7fcdcbfff640 1 -- 172.21.15.59:0/386996262 learned_addr learned my addr 172.21.15.59:0/386996262 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.402+0000 7fcdcaffd640 1 --2- 172.21.15.59:0/386996262 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdcc15f200 0x7fcdcc0d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.402+0000 7fcdcbfff640 1 -- 172.21.15.59:0/386996262 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdcc15f200 msgr2=0x7fcdcc0d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.402+0000 7fcdcbfff640 1 --2- 172.21.15.59:0/386996262 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdcc15f200 0x7fcdcc0d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.402+0000 7fcdcbfff640 1 -- 172.21.15.59:0/386996262 >> v1:172.21.15.164:6789/0 conn(0x7fcdcc15ee20 legacy=0x7fcdcc0cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.402+0000 7fcdcbfff640 1 -- 172.21.15.59:0/386996262 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc162560 con 0x7fcdcc0d06a0 2024-03-06T07:23:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdcbfff640 1 --2- 172.21.15.59:0/386996262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdcc0d06a0 0x7fcdcc15d710 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4009e50 tx=0x7fcdc402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44cc4a4d6d147012 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdca7fc640 1 -- 172.21.15.59:0/386996262 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc402cb00 con 0x7fcdcc0d06a0 2024-03-06T07:23:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdca7fc640 1 -- 172.21.15.59:0/386996262 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc402cca0 con 0x7fcdcc0d06a0 2024-03-06T07:23:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdca7fc640 1 -- 172.21.15.59:0/386996262 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4034610 con 0x7fcdcc0d06a0 2024-03-06T07:23:21.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdd1b32640 1 -- 172.21.15.59:0/386996262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdcc0d06a0 msgr2=0x7fcdcc15d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdd1b32640 1 --2- 172.21.15.59:0/386996262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdcc0d06a0 0x7fcdcc15d710 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4009e50 tx=0x7fcdc402bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdd1b32640 1 -- 172.21.15.59:0/386996262 shutdown_connections 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdd1b32640 1 --2- 172.21.15.59:0/386996262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdcc0d06a0 0x7fcdcc15d710 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdd1b32640 1 --2- 172.21.15.59:0/386996262 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdcc15f200 0x7fcdcc0d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdd1b32640 1 -- 172.21.15.59:0/386996262 >> 172.21.15.59:0/386996262 conn(0x7fcdcc0b3610 msgr2=0x7fcdcc0b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdd1b32640 1 -- 172.21.15.59:0/386996262 shutdown_connections 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.403+0000 7fcdd1b32640 1 -- 172.21.15.59:0/386996262 wait complete. 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.404+0000 7fcdd1b32640 1 Processor -- start 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.405+0000 7fcdd1b32640 1 -- start start 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.405+0000 7fcdd1b32640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdcc0d06a0 0x7fcdcc0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.405+0000 7fcdd1b32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdcc15ee20 0x7fcdcc1572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.405+0000 7fcdd1b32640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdcc15f200 0x7fcdcc0b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.405+0000 7fcdd1b32640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc161d60 con 0x7fcdcc15f200 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.405+0000 7fcdd1b32640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc161be0 con 0x7fcdcc0d06a0 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.405+0000 7fcdd1b32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc15dc50 con 0x7fcdcc15ee20 2024-03-06T07:23:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.405+0000 7fcdcbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdcc15f200 0x7fcdcc0b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.405+0000 7fcdcbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdcc15f200 0x7fcdcc0b5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40270/0 (socket says 172.21.15.59:40270) 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.405+0000 7fcdcbfff640 1 -- 172.21.15.59:0/3561191430 learned_addr learned my addr 172.21.15.59:0/3561191430 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdcb7fe640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdcc0d06a0 0x7fcdcc0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdcaffd640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdcc15ee20 0x7fcdcc1572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdcbfff640 1 -- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdcc15ee20 msgr2=0x7fcdcc1572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdcbfff640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdcc15ee20 0x7fcdcc1572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdcbfff640 1 -- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdcc0d06a0 msgr2=0x7fcdcc0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdcbfff640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdcc0d06a0 0x7fcdcc0b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdcbfff640 1 -- 172.21.15.59:0/3561191430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc0b4ea0 con 0x7fcdcc15f200 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdcaffd640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdcc15ee20 0x7fcdcc1572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdcbfff640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdcc15f200 0x7fcdcc0b5f40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4009e50 tx=0x7fcdc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdb3fff640 1 -- 172.21.15.59:0/3561191430 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4043020 con 0x7fcdcc15f200 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdb3fff640 1 -- 172.21.15.59:0/3561191430 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc402ce50 con 0x7fcdcc15f200 2024-03-06T07:23:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdd1b32640 1 -- 172.21.15.59:0/3561191430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdcc0b5180 con 0x7fcdcc15f200 2024-03-06T07:23:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdb3fff640 1 -- 172.21.15.59:0/3561191430 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4009510 con 0x7fcdcc15f200 2024-03-06T07:23:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.406+0000 7fcdd1b32640 1 -- 172.21.15.59:0/3561191430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdcc0b5a20 con 0x7fcdcc15f200 2024-03-06T07:23:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.407+0000 7fcdd1b32640 1 -- 172.21.15.59:0/3561191430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdcc0b5350 con 0x7fcdcc15f200 2024-03-06T07:23:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.407+0000 7fcdb3fff640 1 -- 172.21.15.59:0/3561191430 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdc4046020 con 0x7fcdcc15f200 2024-03-06T07:23:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.408+0000 7fcdb3fff640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcdb806b4d0 0x7fcdb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.408+0000 7fcdb3fff640 1 -- 172.21.15.59:0/3561191430 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcdc40ac4d0 con 0x7fcdcc15f200 2024-03-06T07:23:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.408+0000 7fcdcb7fe640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcdb806b4d0 0x7fcdb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.409+0000 7fcdcb7fe640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcdb806b4d0 0x7fcdb806d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc001640 tx=0x7fcdbc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.412+0000 7fcdb3fff640 1 -- 172.21.15.59:0/3561191430 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdc404d030 con 0x7fcdcc15f200 2024-03-06T07:23:21.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:21.667+0000 7fcdd1b32640 1 -- 172.21.15.59:0/3561191430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7fcdcc04f330 con 0x7fcdcc15f200 2024-03-06T07:23:22.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.670+0000 7fcdb3fff640 1 -- 172.21.15.59:0/3561191430 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v104) v1 ==== 136+0+0 (secure 0 0 0) 0x7fcdc407b770 con 0x7fcdcc15f200 2024-03-06T07:23:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 -- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcdb806b4d0 msgr2=0x7fcdb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcdb806b4d0 0x7fcdb806d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc001640 tx=0x7fcdbc009040 comp rx=0 tx=0).stop 2024-03-06T07:23:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 -- 172.21.15.59:0/3561191430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdcc15f200 msgr2=0x7fcdcc0b5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdcc15f200 0x7fcdcc0b5f40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4009e50 tx=0x7fcdc4004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 -- 172.21.15.59:0/3561191430 shutdown_connections 2024-03-06T07:23:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcdb806b4d0 0x7fcdb806d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcdcc15f200 0x7fcdcc0b5f40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdcc15ee20 0x7fcdcc1572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 --2- 172.21.15.59:0/3561191430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdcc0d06a0 0x7fcdcc0b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 -- 172.21.15.59:0/3561191430 >> 172.21.15.59:0/3561191430 conn(0x7fcdcc0b3610 msgr2=0x7fcdcc06eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 -- 172.21.15.59:0/3561191430 shutdown_connections 2024-03-06T07:23:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.673+0000 7fcdd1b32640 1 -- 172.21.15.59:0/3561191430 wait complete. 2024-03-06T07:23:22.692 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:23:22.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.801+0000 7f98b073a640 1 Processor -- start 2024-03-06T07:23:22.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.801+0000 7f98b073a640 1 -- start start 2024-03-06T07:23:22.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.802+0000 7f98b073a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b41b0 0x7f98a80cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.802+0000 7f98b073a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a80d0240 con 0x7f98a80b15c0 2024-03-06T07:23:22.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.802+0000 7f98b073a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a80cef50 con 0x7f98a815bdd0 2024-03-06T07:23:22.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.802+0000 7f98b073a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a80cf4f0 con 0x7f98a80b41b0 2024-03-06T07:23:22.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.802+0000 7f98aecd7640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f98a80b15c0 0x7f98a80b1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54416/0 (socket says 172.21.15.59:54416) 2024-03-06T07:23:22.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.802+0000 7f98aecd7640 1 -- 172.21.15.59:0/2129485725 learned_addr learned my addr 172.21.15.59:0/2129485725 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:22.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.802+0000 7f98adcd5640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b41b0 0x7f98a80cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.802+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328618208 0 0) 0x7f98a80d0240 con 0x7f98a80b15c0 2024-03-06T07:23:22.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.802+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9890003520 con 0x7f98a80b15c0 2024-03-06T07:23:22.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3683139064 0 0) 0x7f9890003520 con 0x7f98a80b15c0 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b41b0 msgr2=0x7f98a80cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b41b0 0x7f98a80cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 >> v1:172.21.15.164:6789/0 conn(0x7f98a815bdd0 legacy=0x7f98a80ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a80cf210 con 0x7f98a80b15c0 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f98980019f0 con 0x7f98a80b15c0 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98adcd5640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b41b0 0x7f98a80cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 >> v1:172.21.15.59:6789/0 conn(0x7f98a80b15c0 legacy=0x7f98a80b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98900039e0 0x7f9890003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9890004390 0x7f9890004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b41b0 0x7f9890005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a80cf4f0 con 0x7f9890004390 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989000b9c0 con 0x7f98900039e0 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989000bb70 con 0x7f98a80b41b0 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98adcd5640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9890004390 0x7f9890004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98ae4d6640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98900039e0 0x7f9890003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98aecd7640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b41b0 0x7f9890005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98adcd5640 1 -- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b41b0 msgr2=0x7f9890005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98adcd5640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a80b41b0 0x7f9890005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98adcd5640 1 -- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98900039e0 msgr2=0x7f9890003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:22.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98adcd5640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98900039e0 0x7f9890003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.803+0000 7f98adcd5640 1 -- 172.21.15.59:0/2129485725 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9890003520 con 0x7f9890004390 2024-03-06T07:23:22.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.804+0000 7f98adcd5640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9890004390 0x7f9890004860 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f989c009c90 tx=0x7f9890005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50eda154fe407b2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.804+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c039070 con 0x7f9890004390 2024-03-06T07:23:22.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.804+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c02c8f0 con 0x7f9890004390 2024-03-06T07:23:22.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.804+0000 7f98ad4d4640 1 -- 172.21.15.59:0/2129485725 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c009860 con 0x7f9890004390 2024-03-06T07:23:22.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.805+0000 7f98b073a640 1 -- 172.21.15.59:0/2129485725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9890004390 msgr2=0x7f9890004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.805+0000 7f98b073a640 1 --2- 172.21.15.59:0/2129485725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9890004390 0x7f9890004860 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f989c009c90 tx=0x7f9890005830 comp rx=0 tx=0).stop 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.805+0000 7f98aecd7640 1 -- 172.21.15.59:0/2129485725 reap_dead start 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.805+0000 7f98b073a640 1 -- 172.21.15.59:0/2129485725 shutdown_connections 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.805+0000 7f98b073a640 1 -- 172.21.15.59:0/2129485725 >> 172.21.15.59:0/2129485725 conn(0x7f98a80b3690 msgr2=0x7f98a80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.805+0000 7f98b073a640 1 -- 172.21.15.59:0/2129485725 shutdown_connections 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.805+0000 7f98b073a640 1 -- 172.21.15.59:0/2129485725 wait complete. 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.806+0000 7f98b073a640 1 Processor -- start 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.806+0000 7f98b073a640 1 -- start start 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98b073a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9890004390 0x7f98a81510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98b073a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a80c6170 0x7f98a80ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98b073a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a8159510 0x7f98a8159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98b073a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a815c1b0 con 0x7f98a80c6170 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98b073a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a80d0850 con 0x7f9890004390 2024-03-06T07:23:22.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98b073a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a80b10c0 con 0x7f98a8159510 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98adcd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a80c6170 0x7f98a80ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98adcd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a80c6170 0x7f98a80ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40284/0 (socket says 172.21.15.59:40284) 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98adcd5640 1 -- 172.21.15.59:0/2274424994 learned_addr learned my addr 172.21.15.59:0/2274424994 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98ae4d6640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9890004390 0x7f98a81510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98aecd7640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a8159510 0x7f98a8159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98adcd5640 1 -- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a8159510 msgr2=0x7f98a8159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98adcd5640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a8159510 0x7f98a8159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98adcd5640 1 -- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9890004390 msgr2=0x7f98a81510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98adcd5640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9890004390 0x7f98a81510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98adcd5640 1 -- 172.21.15.59:0/2274424994 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a80cb960 con 0x7f98a80c6170 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98aecd7640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a8159510 0x7f98a8159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98adcd5640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a80c6170 0x7f98a80ca5f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f989c009db0 tx=0x7f989c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98a6ffd640 1 -- 172.21.15.59:0/2274424994 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c039050 con 0x7f98a80c6170 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98a6ffd640 1 -- 172.21.15.59:0/2274424994 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c045070 con 0x7f98a80c6170 2024-03-06T07:23:22.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98a6ffd640 1 -- 172.21.15.59:0/2274424994 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c009860 con 0x7f98a80c6170 2024-03-06T07:23:22.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.807+0000 7f98b073a640 1 -- 172.21.15.59:0/2274424994 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a80cab30 con 0x7f98a80c6170 2024-03-06T07:23:22.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.808+0000 7f98b073a640 1 -- 172.21.15.59:0/2274424994 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a80c68e0 con 0x7f98a80c6170 2024-03-06T07:23:22.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.809+0000 7f98b073a640 1 -- 172.21.15.59:0/2274424994 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a8159ed0 con 0x7f98a80c6170 2024-03-06T07:23:22.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.809+0000 7f98a6ffd640 1 -- 172.21.15.59:0/2274424994 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f989c04a460 con 0x7f98a80c6170 2024-03-06T07:23:22.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.809+0000 7f98a6ffd640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f989406b4d0 0x7f989406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:22.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.809+0000 7f98a6ffd640 1 -- 172.21.15.59:0/2274424994 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f989c0abf70 con 0x7f98a80c6170 2024-03-06T07:23:22.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.809+0000 7f98ae4d6640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f989406b4d0 0x7f989406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:22.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.810+0000 7f98ae4d6640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f989406b4d0 0x7f989406d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f98a815a120 tx=0x7f98a0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.813+0000 7f98a6ffd640 1 -- 172.21.15.59:0/2274424994 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f989c04f020 con 0x7f98a80c6170 2024-03-06T07:23:22.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.994+0000 7f98b073a640 1 -- 172.21.15.59:0/2274424994 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f98a809fcf0 con 0x7f98a80c6170 2024-03-06T07:23:22.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.997+0000 7f98a6ffd640 1 -- 172.21.15.59:0/2274424994 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f98a80cb960 con 0x7f98a80c6170 2024-03-06T07:23:23.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 -- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f989406b4d0 msgr2=0x7f989406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f989406b4d0 0x7f989406d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f98a815a120 tx=0x7f98a0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:23.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 -- 172.21.15.59:0/2274424994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a80c6170 msgr2=0x7f98a80ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a80c6170 0x7f98a80ca5f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f989c009db0 tx=0x7f989c004730 comp rx=0 tx=0).stop 2024-03-06T07:23:23.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 -- 172.21.15.59:0/2274424994 shutdown_connections 2024-03-06T07:23:23.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f989406b4d0 0x7f989406d990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98a8159510 0x7f98a8159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98a80c6170 0x7f98a80ca5f0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 --2- 172.21.15.59:0/2274424994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9890004390 0x7f98a81510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 -- 172.21.15.59:0/2274424994 >> 172.21.15.59:0/2274424994 conn(0x7f98a80b3690 msgr2=0x7f98a806f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 -- 172.21.15.59:0/2274424994 shutdown_connections 2024-03-06T07:23:23.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:22.999+0000 7f98b073a640 1 -- 172.21.15.59:0/2274424994 wait complete. 2024-03-06T07:23:23.012 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:23.012 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:23.052 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:23.104 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:23.104 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:23:23.104 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:23.120 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:23.120 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:23:23.120 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:23.137 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:23.137 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:23:23.137 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:23.175 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:23.176 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:23:23.176 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:23.193 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:23.194 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:23:23.194 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:23.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.315+0000 7f81e3908640 1 Processor -- start 2024-03-06T07:23:23.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.316+0000 7f81e3908640 1 -- start start 2024-03-06T07:23:23.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.316+0000 7f81e3908640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81dc13dc10 0x7f81dc0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.316+0000 7f81e3908640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc15f580 con 0x7f81dc15bf30 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.316+0000 7f81e3908640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81dc15b450 con 0x7f81dc15ef60 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.317+0000 7f81e3908640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0cf3f0 con 0x7f81dc13dc10 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.317+0000 7f81e0ea3640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f81dc15bf30 0x7f81dc15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54420/0 (socket says 172.21.15.59:54420) 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.317+0000 7f81e0ea3640 1 -- 172.21.15.59:0/1362418169 learned_addr learned my addr 172.21.15.59:0/1362418169 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.317+0000 7f81e16a4640 1 --2- 172.21.15.59:0/1362418169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81dc13dc10 0x7f81dc0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.317+0000 7f81d3fff640 1 -- 172.21.15.59:0/1362418169 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358463146 0 0) 0x7f81dc15f580 con 0x7f81dc15bf30 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.317+0000 7f81d3fff640 1 -- 172.21.15.59:0/1362418169 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c8003520 con 0x7f81dc15bf30 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.317+0000 7f81d3fff640 1 -- 172.21.15.59:0/1362418169 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159533565 0 0) 0x7f81dc15b450 con 0x7f81dc15ef60 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.317+0000 7f81d3fff640 1 -- 172.21.15.59:0/1362418169 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81dc15f580 con 0x7f81dc15ef60 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.317+0000 7f81e16a4640 1 -- 172.21.15.59:0/1362418169 >> v1:172.21.15.164:6789/0 conn(0x7f81dc15ef60 legacy=0x7f81dc0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.317+0000 7f81e16a4640 1 -- 172.21.15.59:0/1362418169 >> v1:172.21.15.59:6789/0 conn(0x7f81dc15bf30 legacy=0x7f81dc15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.317+0000 7f81e16a4640 1 -- 172.21.15.59:0/1362418169 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc15b730 con 0x7f81dc13dc10 2024-03-06T07:23:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.318+0000 7f81e16a4640 1 --2- 172.21.15.59:0/1362418169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81dc13dc10 0x7f81dc0b2950 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f81d80097d0 tx=0x7f81d802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a74a766d76482c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.318+0000 7f81d3fff640 1 -- 172.21.15.59:0/1362418169 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8035020 con 0x7f81dc13dc10 2024-03-06T07:23:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.318+0000 7f81d3fff640 1 -- 172.21.15.59:0/1362418169 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d8031800 con 0x7f81dc13dc10 2024-03-06T07:23:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.318+0000 7f81d3fff640 1 -- 172.21.15.59:0/1362418169 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8031bb0 con 0x7f81dc13dc10 2024-03-06T07:23:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.319+0000 7f81e3908640 1 -- 172.21.15.59:0/1362418169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81dc13dc10 msgr2=0x7f81dc0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.319+0000 7f81e3908640 1 --2- 172.21.15.59:0/1362418169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81dc13dc10 0x7f81dc0b2950 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f81d80097d0 tx=0x7f81d802b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.319+0000 7f81e3908640 1 -- 172.21.15.59:0/1362418169 shutdown_connections 2024-03-06T07:23:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.319+0000 7f81e3908640 1 --2- 172.21.15.59:0/1362418169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81dc13dc10 0x7f81dc0b2950 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.319+0000 7f81e3908640 1 -- 172.21.15.59:0/1362418169 >> 172.21.15.59:0/1362418169 conn(0x7f81dc0cf8a0 msgr2=0x7f81dc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.319+0000 7f81e3908640 1 -- 172.21.15.59:0/1362418169 shutdown_connections 2024-03-06T07:23:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.319+0000 7f81e3908640 1 -- 172.21.15.59:0/1362418169 wait complete. 2024-03-06T07:23:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.320+0000 7f81e3908640 1 Processor -- start 2024-03-06T07:23:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.320+0000 7f81e3908640 1 -- start start 2024-03-06T07:23:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e3908640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81dc13dc10 0x7f81dc1f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e3908640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81dc15bf30 0x7f81dc13fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e3908640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81dc15ef60 0x7f81dc1f3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e3908640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0b2e90 con 0x7f81dc15ef60 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e3908640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc15b210 con 0x7f81dc15bf30 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e3908640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc0b3010 con 0x7f81dc13dc10 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e1ea5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81dc15ef60 0x7f81dc1f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e1ea5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81dc15ef60 0x7f81dc1f3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40296/0 (socket says 172.21.15.59:40296) 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e0ea3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81dc15bf30 0x7f81dc13fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e1ea5640 1 -- 172.21.15.59:0/2089797342 learned_addr learned my addr 172.21.15.59:0/2089797342 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e16a4640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81dc13dc10 0x7f81dc1f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e1ea5640 1 -- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81dc13dc10 msgr2=0x7f81dc1f09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e1ea5640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81dc13dc10 0x7f81dc1f09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e1ea5640 1 -- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81dc15bf30 msgr2=0x7f81dc13fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e1ea5640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81dc15bf30 0x7f81dc13fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e1ea5640 1 -- 172.21.15.59:0/2089797342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc1f0ef0 con 0x7f81dc15ef60 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e1ea5640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81dc15ef60 0x7f81dc1f3310 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f81d400c6e0 tx=0x7f81d400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81d1ffb640 1 -- 172.21.15.59:0/2089797342 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4003b70 con 0x7f81dc15ef60 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81d1ffb640 1 -- 172.21.15.59:0/2089797342 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d4003d10 con 0x7f81dc15ef60 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81e3908640 1 -- 172.21.15.59:0/2089797342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc1f4c90 con 0x7f81dc15ef60 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.321+0000 7f81d1ffb640 1 -- 172.21.15.59:0/2089797342 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d40052d0 con 0x7f81dc15ef60 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.322+0000 7f81e3908640 1 -- 172.21.15.59:0/2089797342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc141a60 con 0x7f81dc15ef60 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.323+0000 7f81e3908640 1 -- 172.21.15.59:0/2089797342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81dc04f330 con 0x7f81dc15ef60 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.323+0000 7f81d1ffb640 1 -- 172.21.15.59:0/2089797342 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81d4005470 con 0x7f81dc15ef60 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.323+0000 7f81d1ffb640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f81c006b4d0 0x7f81c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.323+0000 7f81d1ffb640 1 -- 172.21.15.59:0/2089797342 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f81d4089c20 con 0x7f81dc15ef60 2024-03-06T07:23:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.323+0000 7f81e16a4640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f81c006b4d0 0x7f81c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.324+0000 7f81e16a4640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f81c006b4d0 0x7f81c006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f81d8036000 tx=0x7f81d8008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.328+0000 7f81d1ffb640 1 -- 172.21.15.59:0/2089797342 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81d40587e0 con 0x7f81dc15ef60 2024-03-06T07:23:23.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.553+0000 7f81e3908640 1 -- 172.21.15.59:0/2089797342 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81dc13fa80 con 0x7f81dc15ef60 2024-03-06T07:23:23.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.553+0000 7f81d1ffb640 1 -- 172.21.15.59:0/2089797342 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f81d405c880 con 0x7f81dc15ef60 2024-03-06T07:23:23.556 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:23.556 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":15,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4970,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/887214689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":887214689},{"type":"v1","addr":"172.21.15.164:6836","nonce":887214689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4973,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3635398011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3635398011},{"type":"v1","addr":"172.21.15.164:6837","nonce":3635398011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2604270468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2604270468},{"type":"v1","addr":"172.21.15.59:6834","nonce":2604270468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/295636955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":295636955},{"type":"v1","addr":"172.21.15.59:6835","nonce":295636955}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":15}],"filesystems":[]} 2024-03-06T07:23:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 15 2024-03-06T07:23:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 -- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f81c006b4d0 msgr2=0x7f81c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f81c006b4d0 0x7f81c006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f81d8036000 tx=0x7f81d8008030 comp rx=0 tx=0).stop 2024-03-06T07:23:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 -- 172.21.15.59:0/2089797342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81dc15ef60 msgr2=0x7f81dc1f3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81dc15ef60 0x7f81dc1f3310 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f81d400c6e0 tx=0x7f81d400cba0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 -- 172.21.15.59:0/2089797342 shutdown_connections 2024-03-06T07:23:23.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f81c006b4d0 0x7f81c006d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81dc15ef60 0x7f81dc1f3310 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81dc15bf30 0x7f81dc13fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 --2- 172.21.15.59:0/2089797342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81dc13dc10 0x7f81dc1f09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 -- 172.21.15.59:0/2089797342 >> 172.21.15.59:0/2089797342 conn(0x7f81dc0cf8a0 msgr2=0x7f81dc0b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 -- 172.21.15.59:0/2089797342 shutdown_connections 2024-03-06T07:23:23.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.555+0000 7f81e3908640 1 -- 172.21.15.59:0/2089797342 wait complete. 2024-03-06T07:23:23.569 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:23:23.569 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:23:23.570 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:23:23.570 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:23.570 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:23:23.570 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:23:23.571 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:23:23.571 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:23.571 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:23:23.571 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:23:23.572 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:23:23.572 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:23.572 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:23:23.572 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:23:23.573 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:23:23.573 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:23.580 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:23:23.580 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:23:23.581 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:23:23.582 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:23:23.586 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:23:23.586 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:23:23.587 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:23:23.587 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:23:23.622 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:23:23.622 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:23.623 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:23:23.623 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:23.629 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:23:23.629 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:23.630 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:23:23.630 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:23.630 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:23:23.698 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:23:23.699 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:23:23.716 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:23:23.716 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:23:23.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.735+0000 7f2422e1c640 1 Processor -- start 2024-03-06T07:23:23.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.735+0000 7f2422e1c640 1 -- start start 2024-03-06T07:23:23.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.736+0000 7f2422e1c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f241c162400 0x7f241c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.736+0000 7f2422e1c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c0d22c0 con 0x7f241c0b4460 2024-03-06T07:23:23.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.736+0000 7f2422e1c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c0d25a0 con 0x7f241c162970 2024-03-06T07:23:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.736+0000 7f2422e1c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241c161b00 con 0x7f241c162400 2024-03-06T07:23:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.736+0000 7f24213b9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f241c162400 0x7f241c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.736+0000 7f24213b9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f241c162400 0x7f241c0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40782/0 (socket says 172.21.15.59:40782) 2024-03-06T07:23:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.736+0000 7f24213b9640 1 -- 172.21.15.59:0/2170438717 learned_addr learned my addr 172.21.15.59:0/2170438717 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.736+0000 7f2420bb8640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f241c0b4460 0x7f241c0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42852/0 (socket says 172.21.15.59:42852) 2024-03-06T07:23:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.737+0000 7f24213b9640 1 -- 172.21.15.59:0/2170438717 >> v1:172.21.15.164:6789/0 conn(0x7f241c162970 legacy=0x7f241c161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.737+0000 7f24213b9640 1 -- 172.21.15.59:0/2170438717 >> v1:172.21.15.59:6789/0 conn(0x7f241c0b4460 legacy=0x7f241c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.737+0000 7f241b7fe640 1 -- 172.21.15.59:0/2170438717 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220173541 0 0) 0x7f241c0d22c0 con 0x7f241c0b4460 2024-03-06T07:23:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.737+0000 7f24213b9640 1 -- 172.21.15.59:0/2170438717 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c0d2880 con 0x7f241c162400 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.737+0000 7f24213b9640 1 --2- 172.21.15.59:0/2170438717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c162400 0x7f241c0d4800 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f240c009870 tx=0x7f240c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91f24a4f2c4d85e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.737+0000 7f241b7fe640 1 -- 172.21.15.59:0/2170438717 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c034070 con 0x7f241c162400 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.737+0000 7f241b7fe640 1 -- 172.21.15.59:0/2170438717 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c030860 con 0x7f241c162400 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.738+0000 7f241b7fe640 1 -- 172.21.15.59:0/2170438717 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c030c30 con 0x7f241c162400 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.738+0000 7f2422e1c640 1 -- 172.21.15.59:0/2170438717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c162400 msgr2=0x7f241c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.738+0000 7f2422e1c640 1 --2- 172.21.15.59:0/2170438717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c162400 0x7f241c0d4800 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f240c009870 tx=0x7f240c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.738+0000 7f2422e1c640 1 -- 172.21.15.59:0/2170438717 shutdown_connections 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.738+0000 7f2422e1c640 1 --2- 172.21.15.59:0/2170438717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c162400 0x7f241c0d4800 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.738+0000 7f2422e1c640 1 -- 172.21.15.59:0/2170438717 >> 172.21.15.59:0/2170438717 conn(0x7f241c0d3ce0 msgr2=0x7f241c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.738+0000 7f2422e1c640 1 -- 172.21.15.59:0/2170438717 shutdown_connections 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.738+0000 7f2422e1c640 1 -- 172.21.15.59:0/2170438717 wait complete. 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.739+0000 7f2422e1c640 1 Processor -- start 2024-03-06T07:23:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.739+0000 7f2422e1c640 1 -- start start 2024-03-06T07:23:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f2422e1c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f241c0b4460 0x7f241c0c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f2422e1c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c162400 0x7f241c155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f2422e1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c162970 0x7f241c0c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f2422e1c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0ce950 con 0x7f241c162970 2024-03-06T07:23:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f2422e1c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0cecc0 con 0x7f241c162400 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f2422e1c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c0b4840 con 0x7f241c0b4460 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f24213b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c162970 0x7f241c0c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f24213b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c162970 0x7f241c0c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40790/0 (socket says 172.21.15.59:40790) 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f24213b9640 1 -- 172.21.15.59:0/3809898607 learned_addr learned my addr 172.21.15.59:0/3809898607 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f2420bb8640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f241c0b4460 0x7f241c0c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f241bfff640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c162400 0x7f241c155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f24213b9640 1 -- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f241c0b4460 msgr2=0x7f241c0c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f24213b9640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f241c0b4460 0x7f241c0c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f24213b9640 1 -- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c162400 msgr2=0x7f241c155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f24213b9640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c162400 0x7f241c155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f24213b9640 1 -- 172.21.15.59:0/3809898607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c156250 con 0x7f241c162970 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.740+0000 7f241bfff640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c162400 0x7f241c155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.741+0000 7f24213b9640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c162970 0x7f241c0c7810 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f240c02b9e0 tx=0x7f240c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.741+0000 7f24197fa640 1 -- 172.21.15.59:0/3809898607 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c048070 con 0x7f241c162970 2024-03-06T07:23:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.741+0000 7f24197fa640 1 -- 172.21.15.59:0/3809898607 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c030660 con 0x7f241c162970 2024-03-06T07:23:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.741+0000 7f2422e1c640 1 -- 172.21.15.59:0/3809898607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c0c8380 con 0x7f241c162970 2024-03-06T07:23:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.741+0000 7f24197fa640 1 -- 172.21.15.59:0/3809898607 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c034040 con 0x7f241c162970 2024-03-06T07:23:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.741+0000 7f2422e1c640 1 -- 172.21.15.59:0/3809898607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c0c8940 con 0x7f241c162970 2024-03-06T07:23:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.742+0000 7f2422e1c640 1 -- 172.21.15.59:0/3809898607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c04f330 con 0x7f241c162970 2024-03-06T07:23:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.742+0000 7f24197fa640 1 -- 172.21.15.59:0/3809898607 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f240c04b070 con 0x7f241c162970 2024-03-06T07:23:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.742+0000 7f24197fa640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f240806b4d0 0x7f240806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.743+0000 7f24197fa640 1 -- 172.21.15.59:0/3809898607 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f240c0abe80 con 0x7f241c162970 2024-03-06T07:23:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.743+0000 7f2420bb8640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f240806b4d0 0x7f240806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.743+0000 7f2420bb8640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f240806b4d0 0x7f240806d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f24140012e0 tx=0x7f2414001700 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.747+0000 7f24197fa640 1 -- 172.21.15.59:0/3809898607 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f240c07ab60 con 0x7f241c162970 2024-03-06T07:23:23.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:23.939+0000 7f2422e1c640 1 -- 172.21.15.59:0/3809898607 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f241c06b180 con 0x7f241c162970 2024-03-06T07:23:25.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.019+0000 7f24197fa640 1 -- 172.21.15.59:0/3809898607 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v41) v1 ==== 97+0+0 (secure 0 0 0) 0x7f240c07ec00 con 0x7f241c162970 2024-03-06T07:23:25.021 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-03-06T07:23:25.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 -- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f240806b4d0 msgr2=0x7f240806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f240806b4d0 0x7f240806d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f24140012e0 tx=0x7f2414001700 comp rx=0 tx=0).stop 2024-03-06T07:23:25.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 -- 172.21.15.59:0/3809898607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c162970 msgr2=0x7f241c0c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c162970 0x7f241c0c7810 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f240c02b9e0 tx=0x7f240c004730 comp rx=0 tx=0).stop 2024-03-06T07:23:25.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 -- 172.21.15.59:0/3809898607 shutdown_connections 2024-03-06T07:23:25.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f240806b4d0 0x7f240806d990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c162970 0x7f241c0c7810 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c162400 0x7f241c155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 --2- 172.21.15.59:0/3809898607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f241c0b4460 0x7f241c0c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 -- 172.21.15.59:0/3809898607 >> 172.21.15.59:0/3809898607 conn(0x7f241c0d3ce0 msgr2=0x7f241c0d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 -- 172.21.15.59:0/3809898607 shutdown_connections 2024-03-06T07:23:25.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.023+0000 7f2422e1c640 1 -- 172.21.15.59:0/3809898607 wait complete. 2024-03-06T07:23:25.038 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:23:25.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.144+0000 7efece3a2640 1 Processor -- start 2024-03-06T07:23:25.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.144+0000 7efece3a2640 1 -- start start 2024-03-06T07:23:25.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.145+0000 7efece3a2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efec80ae180 0x7efec81687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.145+0000 7efece3a2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec815e180 con 0x7efec806b6c0 2024-03-06T07:23:25.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.145+0000 7efece3a2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec80ae680 con 0x7efec806f060 2024-03-06T07:23:25.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.145+0000 7efece3a2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec815e440 con 0x7efec80ae180 2024-03-06T07:23:25.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.145+0000 7efeccb9f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7efec806b6c0 0x7efec80ada70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42862/0 (socket says 172.21.15.59:42862) 2024-03-06T07:23:25.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.145+0000 7efecdba1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efec80ae180 0x7efec81687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.145+0000 7efeccb9f640 1 -- 172.21.15.59:0/1471626589 learned_addr learned my addr 172.21.15.59:0/1471626589 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:25.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.146+0000 7efecdba1640 1 -- 172.21.15.59:0/1471626589 >> v1:172.21.15.164:6789/0 conn(0x7efec806f060 legacy=0x7efec806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.146+0000 7efecdba1640 1 -- 172.21.15.59:0/1471626589 >> v1:172.21.15.59:6789/0 conn(0x7efec806b6c0 legacy=0x7efec80ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.146+0000 7efec7fff640 1 -- 172.21.15.59:0/1471626589 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318663526 0 0) 0x7efec815e180 con 0x7efec806b6c0 2024-03-06T07:23:25.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.146+0000 7efecdba1640 1 -- 172.21.15.59:0/1471626589 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec815e720 con 0x7efec80ae180 2024-03-06T07:23:25.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.147+0000 7efecdba1640 1 --2- 172.21.15.59:0/1471626589 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efec80ae180 0x7efec81687b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efeb80097a0 tx=0x7efeb802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2169b3e2bf9ddb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.147+0000 7efec7fff640 1 -- 172.21.15.59:0/1471626589 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb80315a0 con 0x7efec80ae180 2024-03-06T07:23:25.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.147+0000 7efec7fff640 1 -- 172.21.15.59:0/1471626589 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb8037070 con 0x7efec80ae180 2024-03-06T07:23:25.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.147+0000 7efec7fff640 1 -- 172.21.15.59:0/1471626589 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb8031960 con 0x7efec80ae180 2024-03-06T07:23:25.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.147+0000 7efece3a2640 1 -- 172.21.15.59:0/1471626589 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efec80ae180 msgr2=0x7efec81687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.147+0000 7efece3a2640 1 --2- 172.21.15.59:0/1471626589 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efec80ae180 0x7efec81687b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efeb80097a0 tx=0x7efeb802b840 comp rx=0 tx=0).stop 2024-03-06T07:23:25.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.147+0000 7efece3a2640 1 -- 172.21.15.59:0/1471626589 shutdown_connections 2024-03-06T07:23:25.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.147+0000 7efece3a2640 1 --2- 172.21.15.59:0/1471626589 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efec80ae180 0x7efec81687b0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.147+0000 7efece3a2640 1 -- 172.21.15.59:0/1471626589 >> 172.21.15.59:0/1471626589 conn(0x7efec80d0860 msgr2=0x7efec80d0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.147+0000 7efece3a2640 1 -- 172.21.15.59:0/1471626589 shutdown_connections 2024-03-06T07:23:25.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.147+0000 7efece3a2640 1 -- 172.21.15.59:0/1471626589 wait complete. 2024-03-06T07:23:25.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.148+0000 7efece3a2640 1 Processor -- start 2024-03-06T07:23:25.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.149+0000 7efece3a2640 1 -- start start 2024-03-06T07:23:25.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.149+0000 7efece3a2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efec806b6c0 0x7efec81551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.149+0000 7efece3a2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efec806f060 0x7efec8140150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.149+0000 7efece3a2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efec80ae180 0x7efec8160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.149+0000 7efece3a2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec815f0a0 con 0x7efec806f060 2024-03-06T07:23:25.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.149+0000 7efece3a2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec815ef20 con 0x7efec806b6c0 2024-03-06T07:23:25.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.149+0000 7efece3a2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec8168cf0 con 0x7efec80ae180 2024-03-06T07:23:25.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.149+0000 7efeccb9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efec806f060 0x7efec8140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.149+0000 7efeccb9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efec806f060 0x7efec8140150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40800/0 (socket says 172.21.15.59:40800) 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.149+0000 7efeccb9f640 1 -- 172.21.15.59:0/4286702502 learned_addr learned my addr 172.21.15.59:0/4286702502 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efecd3a0640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efec806b6c0 0x7efec81551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efecdba1640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efec80ae180 0x7efec8160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efeccb9f640 1 -- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efec80ae180 msgr2=0x7efec8160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efeccb9f640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efec80ae180 0x7efec8160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efeccb9f640 1 -- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efec806b6c0 msgr2=0x7efec81551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efeccb9f640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efec806b6c0 0x7efec81551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efeccb9f640 1 -- 172.21.15.59:0/4286702502 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec8150c00 con 0x7efec806f060 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efeccb9f640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efec806f060 0x7efec8140150 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7efebc00c4d0 tx=0x7efebc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efec5ffb640 1 -- 172.21.15.59:0/4286702502 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc018070 con 0x7efec806f060 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efec5ffb640 1 -- 172.21.15.59:0/4286702502 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efebc007500 con 0x7efec806f060 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efec5ffb640 1 -- 172.21.15.59:0/4286702502 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc004a20 con 0x7efec806f060 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efece3a2640 1 -- 172.21.15.59:0/4286702502 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec8160fc0 con 0x7efec806f060 2024-03-06T07:23:25.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.150+0000 7efece3a2640 1 -- 172.21.15.59:0/4286702502 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec81549f0 con 0x7efec806f060 2024-03-06T07:23:25.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.151+0000 7efece3a2640 1 -- 172.21.15.59:0/4286702502 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec8068f80 con 0x7efec806f060 2024-03-06T07:23:25.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.152+0000 7efec5ffb640 1 -- 172.21.15.59:0/4286702502 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efebc01b070 con 0x7efec806f060 2024-03-06T07:23:25.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.152+0000 7efec5ffb640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efeb406b4d0 0x7efeb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.152+0000 7efec5ffb640 1 -- 172.21.15.59:0/4286702502 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efebc0891b0 con 0x7efec806f060 2024-03-06T07:23:25.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.152+0000 7efecd3a0640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efeb406b4d0 0x7efeb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.153+0000 7efecd3a0640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efeb406b4d0 0x7efeb406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efec81f70b0 tx=0x7efec0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.156+0000 7efec5ffb640 1 -- 172.21.15.59:0/4286702502 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efebc058380 con 0x7efec806f060 2024-03-06T07:23:25.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.406+0000 7efece3a2640 1 -- 172.21.15.59:0/4286702502 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7efec80af2f0 con 0x7efec806f060 2024-03-06T07:23:25.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.406+0000 7efec5ffb640 1 -- 172.21.15.59:0/4286702502 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7efebc057c80 con 0x7efec806f060 2024-03-06T07:23:25.408 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:25.408 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:23:25.408 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlODBZGBAAqV+TH2t2ZKyV9l13QEn6ew==", 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-03-06T07:23:25.409 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlR1MuJRAAp6OAQ9gZemWtwI1dh3H2SA==", 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlDXW2HxAAnKyuS1GW0wLO98/F9wdOAg==", 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:23:25.410 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehl5MO4LBAAKmt2dvKk7BtcCM4o7cSwgg==", 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.411 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBXGehlzdinFhAA0ACDAs9dtoY6EhUIveeIvA==", 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGehlH92eFRAANQceHGuQyrbI6gP1m/Gakw==", 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:23:25.412 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGehlMWpVFBAAbOG9nScbFlvKHXZMQJoQug==", 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-03-06T07:23:25.413 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGehlLCyWEhAAyfoW/FmU37QD2IohknBwWw==", 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBmGehldqv5GhAAWEloqqKgdpKPHjt7VKFodw==", 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGehlc0ENGRAA5V9Az+vxAR3OUwWpDP7w1A==", 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBsGehl5SYEFxAAPLyrHQV9SGguCV9f5VIkYg==", 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.415 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBvGehle3v0FBAA4xUPwX5couppHgi9VwSi2g==", 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.416 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA==", 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlqhlAOhAAQ4l3uI8MlcCH+NAIWo/UuA==", 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:23:25.417 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehltL/GNhAAZqSQ+UtwjI/PA95O1RGBNg==", 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.418 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGehlltx5BBAAFa/+/w8Hd8lZegfIWtpJOA==", 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBSGehlMExaNhAAQ2FD7afktMhrTbMMAK1epQ==", 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:23:25.419 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlLSTzFBAACeLdHsCX9KJGEsq44aqTyw==", 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:25.420 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-03-06T07:23:25.421 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlAgqXEBAACCHZ9Lp2lEyF93xD4Ns9Nw==", 2024-03-06T07:23:25.421 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:23:25.421 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:23:25.421 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:25.421 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:23:25.421 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.421 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:25.421 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-06T07:23:25.421 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:23:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 -- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efeb406b4d0 msgr2=0x7efeb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efeb406b4d0 0x7efeb406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efec81f70b0 tx=0x7efec0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 -- 172.21.15.59:0/4286702502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efec806f060 msgr2=0x7efec8140150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efec806f060 0x7efec8140150 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7efebc00c4d0 tx=0x7efebc00c990 comp rx=0 tx=0).stop 2024-03-06T07:23:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 -- 172.21.15.59:0/4286702502 shutdown_connections 2024-03-06T07:23:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efeb406b4d0 0x7efeb406d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efec80ae180 0x7efec8160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efec806f060 0x7efec8140150 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 --2- 172.21.15.59:0/4286702502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efec806b6c0 0x7efec81551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 -- 172.21.15.59:0/4286702502 >> 172.21.15.59:0/4286702502 conn(0x7efec80d0860 msgr2=0x7efec80b3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 -- 172.21.15.59:0/4286702502 shutdown_connections 2024-03-06T07:23:25.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.408+0000 7efece3a2640 1 -- 172.21.15.59:0/4286702502 wait complete. 2024-03-06T07:23:25.424 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:25.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.524+0000 7f26136a4640 1 Processor -- start 2024-03-06T07:23:25.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.524+0000 7f26136a4640 1 -- start start 2024-03-06T07:23:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f26136a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f260c13dc10 0x7f260c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f26136a4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f260c15bf30 0x7f260c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f26136a4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f260c15e850 0x7f260c15eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f26136a4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f260c0cf3f0 con 0x7f260c15e850 2024-03-06T07:23:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f26136a4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f260c15b210 con 0x7f260c15bf30 2024-03-06T07:23:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f26136a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260c15f580 con 0x7f260c13dc10 2024-03-06T07:23:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f2611c41640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f260c15e850 0x7f260c15eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f2611c41640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f260c15e850 0x7f260c15eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40816/0 (socket says 172.21.15.59:40816) 2024-03-06T07:23:25.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f2611c41640 1 -- 172.21.15.59:0/2265746987 learned_addr learned my addr 172.21.15.59:0/2265746987 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:25.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f2610c3f640 1 --2- 172.21.15.59:0/2265746987 >> v2:172.21.15.164:3300/0 conn(0x7f260c15bf30 0x7f260c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f2611440640 1 --2- 172.21.15.59:0/2265746987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f260c13dc10 0x7f260c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f2611c41640 1 -- 172.21.15.59:0/2265746987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f260c13dc10 msgr2=0x7f260c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f2611c41640 1 --2- 172.21.15.59:0/2265746987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f260c13dc10 0x7f260c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f2611c41640 1 -- 172.21.15.59:0/2265746987 >> v2:172.21.15.164:3300/0 conn(0x7f260c15bf30 msgr2=0x7f260c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f2611c41640 1 --2- 172.21.15.59:0/2265746987 >> v2:172.21.15.164:3300/0 conn(0x7f260c15bf30 0x7f260c0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.525+0000 7f2611c41640 1 -- 172.21.15.59:0/2265746987 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260c15b450 con 0x7f260c15e850 2024-03-06T07:23:25.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.526+0000 7f2611c41640 1 --2- 172.21.15.59:0/2265746987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f260c15e850 0x7f260c15eca0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f260c068890 tx=0x7f260402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3551ee3a5a2dd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.526+0000 7f2603fff640 1 -- 172.21.15.59:0/2265746987 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260402c9e0 con 0x7f260c15e850 2024-03-06T07:23:25.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.526+0000 7f2603fff640 1 -- 172.21.15.59:0/2265746987 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f260402cb80 con 0x7f260c15e850 2024-03-06T07:23:25.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.526+0000 7f2603fff640 1 -- 172.21.15.59:0/2265746987 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26040343f0 con 0x7f260c15e850 2024-03-06T07:23:25.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.526+0000 7f26136a4640 1 -- 172.21.15.59:0/2265746987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f260c15e850 msgr2=0x7f260c15eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.527+0000 7f26136a4640 1 --2- 172.21.15.59:0/2265746987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f260c15e850 0x7f260c15eca0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f260c068890 tx=0x7f260402be40 comp rx=0 tx=0).stop 2024-03-06T07:23:25.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.527+0000 7f26136a4640 1 -- 172.21.15.59:0/2265746987 shutdown_connections 2024-03-06T07:23:25.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.527+0000 7f26136a4640 1 --2- 172.21.15.59:0/2265746987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f260c15e850 0x7f260c15eca0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.527+0000 7f26136a4640 1 --2- 172.21.15.59:0/2265746987 >> v2:172.21.15.164:3300/0 conn(0x7f260c15bf30 0x7f260c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.527+0000 7f26136a4640 1 --2- 172.21.15.59:0/2265746987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f260c13dc10 0x7f260c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.527+0000 7f26136a4640 1 -- 172.21.15.59:0/2265746987 >> 172.21.15.59:0/2265746987 conn(0x7f260c0cf8a0 msgr2=0x7f260c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.527+0000 7f26136a4640 1 -- 172.21.15.59:0/2265746987 shutdown_connections 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.527+0000 7f26136a4640 1 -- 172.21.15.59:0/2265746987 wait complete. 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.528+0000 7f26136a4640 1 Processor -- start 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.528+0000 7f26136a4640 1 -- start start 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.528+0000 7f26136a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f260c13dc10 0x7f260c1f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.528+0000 7f26136a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f260c15bf30 0x7f260c141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.528+0000 7f26136a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f260c15e850 0x7f260c141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.528+0000 7f26136a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c15f1e0 con 0x7f260c13dc10 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f26136a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c13dff0 con 0x7f260c15e850 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f26136a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260c15f360 con 0x7f260c15bf30 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2611440640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f260c13dc10 0x7f260c1f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2611440640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f260c13dc10 0x7f260c1f1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40826/0 (socket says 172.21.15.59:40826) 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2611440640 1 -- 172.21.15.59:0/3893177876 learned_addr learned my addr 172.21.15.59:0/3893177876 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2611c41640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f260c15e850 0x7f260c141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2610c3f640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f260c15bf30 0x7f260c141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2611440640 1 -- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f260c15bf30 msgr2=0x7f260c141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2611440640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f260c15bf30 0x7f260c141510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2611440640 1 -- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f260c15e850 msgr2=0x7f260c141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2611440640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f260c15e850 0x7f260c141a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2611440640 1 -- 172.21.15.59:0/3893177876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260c13f4f0 con 0x7f260c13dc10 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2611440640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f260c13dc10 0x7f260c1f1a20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f260800d800 tx=0x7f260800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2601ffb640 1 -- 172.21.15.59:0/3893177876 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2608018030 con 0x7f260c13dc10 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2601ffb640 1 -- 172.21.15.59:0/3893177876 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26080096a0 con 0x7f260c13dc10 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f26136a4640 1 -- 172.21.15.59:0/3893177876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260c152c50 con 0x7f260c13dc10 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f2601ffb640 1 -- 172.21.15.59:0/3893177876 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2608009870 con 0x7f260c13dc10 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.529+0000 7f26136a4640 1 -- 172.21.15.59:0/3893177876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260c1534f0 con 0x7f260c13dc10 2024-03-06T07:23:25.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.530+0000 7f26136a4640 1 -- 172.21.15.59:0/3893177876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f260c04f330 con 0x7f260c13dc10 2024-03-06T07:23:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.531+0000 7f2601ffb640 1 -- 172.21.15.59:0/3893177876 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f260801b070 con 0x7f260c13dc10 2024-03-06T07:23:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.531+0000 7f2601ffb640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f25f006b4d0 0x7f25f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.531+0000 7f2601ffb640 1 -- 172.21.15.59:0/3893177876 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f260808a0f0 con 0x7f260c13dc10 2024-03-06T07:23:25.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.532+0000 7f2610c3f640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f25f006b4d0 0x7f25f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.532+0000 7f2610c3f640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f25f006b4d0 0x7f25f006d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f260c0b0590 tx=0x7f25fc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.536+0000 7f2601ffb640 1 -- 172.21.15.59:0/3893177876 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f260c13f4f0 con 0x7f260c13dc10 2024-03-06T07:23:25.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.718+0000 7f26136a4640 1 -- 172.21.15.59:0/3893177876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f260c15bcf0 con 0x7f260c13dc10 2024-03-06T07:23:25.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.719+0000 7f2601ffb640 1 -- 172.21.15.59:0/3893177876 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f26080592c0 con 0x7f260c13dc10 2024-03-06T07:23:25.721 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:25.721 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":41,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:23:25.019256+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.721+0000 7f26136a4640 1 -- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f25f006b4d0 msgr2=0x7f25f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.721+0000 7f26136a4640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f25f006b4d0 0x7f25f006d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f260c0b0590 tx=0x7f25fc008040 comp rx=0 tx=0).stop 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.722+0000 7f26136a4640 1 -- 172.21.15.59:0/3893177876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f260c13dc10 msgr2=0x7f260c1f1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.722+0000 7f26136a4640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f260c13dc10 0x7f260c1f1a20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f260800d800 tx=0x7f260800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.722+0000 7f26136a4640 1 -- 172.21.15.59:0/3893177876 shutdown_connections 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.722+0000 7f26136a4640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f25f006b4d0 0x7f25f006d990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.722+0000 7f26136a4640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f260c15e850 0x7f260c141a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.722+0000 7f26136a4640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f260c15bf30 0x7f260c141510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.722+0000 7f26136a4640 1 --2- 172.21.15.59:0/3893177876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f260c13dc10 0x7f260c1f1a20 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.722+0000 7f26136a4640 1 -- 172.21.15.59:0/3893177876 >> 172.21.15.59:0/3893177876 conn(0x7f260c0cf8a0 msgr2=0x7f260c0af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.722+0000 7f26136a4640 1 -- 172.21.15.59:0/3893177876 shutdown_connections 2024-03-06T07:23:25.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.722+0000 7f26136a4640 1 -- 172.21.15.59:0/3893177876 wait complete. 2024-03-06T07:23:25.736 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:25.736 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:25.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.838+0000 7f108390f640 1 Processor -- start 2024-03-06T07:23:25.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.839+0000 7f108390f640 1 -- start start 2024-03-06T07:23:25.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.839+0000 7f108390f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c15bb40 0x7f107c0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.839+0000 7f108390f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c15ff90 con 0x7f107c0cfa90 2024-03-06T07:23:25.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.839+0000 7f108390f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c160270 con 0x7f107c15f7d0 2024-03-06T07:23:25.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.839+0000 7f108390f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c06c910 con 0x7f107c15bb40 2024-03-06T07:23:25.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.839+0000 7f1081eac640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f107c0cfa90 0x7f107c0cdcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42866/0 (socket says 172.21.15.59:42866) 2024-03-06T07:23:25.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.839+0000 7f1081eac640 1 -- 172.21.15.59:0/2224776054 learned_addr learned my addr 172.21.15.59:0/2224776054 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:25.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f10816ab640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c15bb40 0x7f107c0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893924453 0 0) 0x7f107c15ff90 con 0x7f107c0cfa90 2024-03-06T07:23:25.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1068003520 con 0x7f107c0cfa90 2024-03-06T07:23:25.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3600925251 0 0) 0x7f1068003520 con 0x7f107c0cfa90 2024-03-06T07:23:25.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c15bb40 msgr2=0x7f107c0b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c15bb40 0x7f107c0b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 >> v1:172.21.15.164:6789/0 conn(0x7f107c15f7d0 legacy=0x7f107c06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c160550 con 0x7f107c0cfa90 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f10740019f0 con 0x7f107c0cfa90 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f10816ab640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c15bb40 0x7f107c0b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 >> v1:172.21.15.59:6789/0 conn(0x7f107c0cfa90 legacy=0x7f107c0cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10680039e0 0x7f1068003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c15bb40 0x7f1068004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1068004d40 0x7f10680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c06c910 con 0x7f10680039e0 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106800bad0 con 0x7f1068004d40 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106800bcd0 con 0x7f107c15bb40 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f10816ab640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10680039e0 0x7f1068003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1080eaa640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c15bb40 0x7f1068004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f1081eac640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1068004d40 0x7f10680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f10816ab640 1 -- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c15bb40 msgr2=0x7f1068004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f10816ab640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c15bb40 0x7f1068004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f10816ab640 1 -- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1068004d40 msgr2=0x7f10680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f10816ab640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1068004d40 0x7f10680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.840+0000 7f10816ab640 1 -- 172.21.15.59:0/2224776054 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1068003520 con 0x7f10680039e0 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.841+0000 7f10816ab640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10680039e0 0x7f1068003e00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1078006fd0 tx=0x7f1068005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfba1e09492ef0d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.841+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107800f040 con 0x7f10680039e0 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.841+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107802e070 con 0x7f10680039e0 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.841+0000 7f1073fff640 1 -- 172.21.15.59:0/2224776054 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078004450 con 0x7f10680039e0 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.842+0000 7f108390f640 1 -- 172.21.15.59:0/2224776054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10680039e0 msgr2=0x7f1068003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.842+0000 7f108390f640 1 --2- 172.21.15.59:0/2224776054 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10680039e0 0x7f1068003e00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1078006fd0 tx=0x7f1068005830 comp rx=0 tx=0).stop 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.842+0000 7f1081eac640 1 -- 172.21.15.59:0/2224776054 reap_dead start 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.842+0000 7f108390f640 1 -- 172.21.15.59:0/2224776054 shutdown_connections 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.842+0000 7f108390f640 1 -- 172.21.15.59:0/2224776054 >> 172.21.15.59:0/2224776054 conn(0x7f107c06b190 msgr2=0x7f107c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.842+0000 7f108390f640 1 -- 172.21.15.59:0/2224776054 shutdown_connections 2024-03-06T07:23:25.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.842+0000 7f108390f640 1 -- 172.21.15.59:0/2224776054 wait complete. 2024-03-06T07:23:25.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.843+0000 7f108390f640 1 Processor -- start 2024-03-06T07:23:25.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.843+0000 7f108390f640 1 -- start start 2024-03-06T07:23:25.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f108390f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10680039e0 0x7f107c070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f108390f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c15ab20 0x7f107c15e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f108390f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c149eb0 0x7f107c14a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f108390f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c06d790 con 0x7f107c15ab20 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f108390f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c15fc30 con 0x7f10680039e0 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f108390f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c15bf20 con 0x7f107c149eb0 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f1080eaa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c15ab20 0x7f107c15e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f1080eaa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c15ab20 0x7f107c15e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40842/0 (socket says 172.21.15.59:40842) 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f1080eaa640 1 -- 172.21.15.59:0/1204379075 learned_addr learned my addr 172.21.15.59:0/1204379075 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f10816ab640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10680039e0 0x7f107c070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f1081eac640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c149eb0 0x7f107c14a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f1080eaa640 1 -- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c149eb0 msgr2=0x7f107c14a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f1080eaa640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c149eb0 0x7f107c14a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f1080eaa640 1 -- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10680039e0 msgr2=0x7f107c070980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f1080eaa640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10680039e0 0x7f107c070980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f1080eaa640 1 -- 172.21.15.59:0/1204379075 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c15afa0 con 0x7f107c15ab20 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f1080eaa640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c15ab20 0x7f107c15e000 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f106c00f820 tx=0x7f106c00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.844+0000 7f1071ffb640 1 -- 172.21.15.59:0/1204379075 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c0050d0 con 0x7f107c15ab20 2024-03-06T07:23:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.845+0000 7f1071ffb640 1 -- 172.21.15.59:0/1204379075 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c018070 con 0x7f107c15ab20 2024-03-06T07:23:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.845+0000 7f108390f640 1 -- 172.21.15.59:0/1204379075 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c071080 con 0x7f107c15ab20 2024-03-06T07:23:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.845+0000 7f1071ffb640 1 -- 172.21.15.59:0/1204379075 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c005490 con 0x7f107c15ab20 2024-03-06T07:23:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.845+0000 7f108390f640 1 -- 172.21.15.59:0/1204379075 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c071640 con 0x7f107c15ab20 2024-03-06T07:23:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.846+0000 7f108390f640 1 -- 172.21.15.59:0/1204379075 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c0732b0 con 0x7f107c15ab20 2024-03-06T07:23:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.846+0000 7f1071ffb640 1 -- 172.21.15.59:0/1204379075 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f106c00d730 con 0x7f107c15ab20 2024-03-06T07:23:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.846+0000 7f1071ffb640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f106006b500 0x7f106006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.847+0000 7f1071ffb640 1 -- 172.21.15.59:0/1204379075 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f106c089d80 con 0x7f107c15ab20 2024-03-06T07:23:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.847+0000 7f10816ab640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f106006b500 0x7f106006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.847+0000 7f10816ab640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f106006b500 0x7f106006d9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1078035000 tx=0x7f1078031000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:25.851+0000 7f1071ffb640 1 -- 172.21.15.59:0/1204379075 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f106c058f50 con 0x7f107c15ab20 2024-03-06T07:23:26.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.033+0000 7f108390f640 1 -- 172.21.15.59:0/1204379075 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f107c06e130 con 0x7f107c15ab20 2024-03-06T07:23:26.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.033+0000 7f1071ffb640 1 -- 172.21.15.59:0/1204379075 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f106c058850 con 0x7f107c15ab20 2024-03-06T07:23:26.035 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:23:26.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 -- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f106006b500 msgr2=0x7f106006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f106006b500 0x7f106006d9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1078035000 tx=0x7f1078031000 comp rx=0 tx=0).stop 2024-03-06T07:23:26.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 -- 172.21.15.59:0/1204379075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c15ab20 msgr2=0x7f107c15e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c15ab20 0x7f107c15e000 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f106c00f820 tx=0x7f106c00fce0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 -- 172.21.15.59:0/1204379075 shutdown_connections 2024-03-06T07:23:26.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f106006b500 0x7f106006d9c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f107c149eb0 0x7f107c14a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f107c15ab20 0x7f107c15e000 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 --2- 172.21.15.59:0/1204379075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10680039e0 0x7f107c070980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 -- 172.21.15.59:0/1204379075 >> 172.21.15.59:0/1204379075 conn(0x7f107c06b190 msgr2=0x7f107c0b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 -- 172.21.15.59:0/1204379075 shutdown_connections 2024-03-06T07:23:26.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.035+0000 7f108390f640 1 -- 172.21.15.59:0/1204379075 wait complete. 2024-03-06T07:23:26.048 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:23:26.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.148+0000 7f19c1081640 1 Processor -- start 2024-03-06T07:23:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.148+0000 7f19c1081640 1 -- start start 2024-03-06T07:23:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.149+0000 7f19c1081640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f19bc0d00c0 0x7f19bc0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.149+0000 7f19c1081640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19bc0cf850 0x7f19bc0cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.149+0000 7f19c1081640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19bc0d09e0 con 0x7f19bc0ce1b0 2024-03-06T07:23:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.149+0000 7f19c1081640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19bc06ca90 con 0x7f19bc0d00c0 2024-03-06T07:23:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.149+0000 7f19c1081640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc06df30 con 0x7f19bc0cf850 2024-03-06T07:23:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.149+0000 7f19bad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f19bc0d00c0 0x7f19bc0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.149+0000 7f19bb59e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f19bc0ce1b0 0x7f19bc0ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42874/0 (socket says 172.21.15.59:42874) 2024-03-06T07:23:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.149+0000 7f19bb59e640 1 -- 172.21.15.59:0/2240006229 learned_addr learned my addr 172.21.15.59:0/2240006229 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.149+0000 7f19ba59c640 1 --2- 172.21.15.59:0/2240006229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19bc0cf850 0x7f19bc0cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.150+0000 7f19bad9d640 1 -- 172.21.15.59:0/2240006229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19bc0cf850 msgr2=0x7f19bc0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.150+0000 7f19bad9d640 1 --2- 172.21.15.59:0/2240006229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19bc0cf850 0x7f19bc0cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.150+0000 7f19bad9d640 1 -- 172.21.15.59:0/2240006229 >> v1:172.21.15.59:6789/0 conn(0x7f19bc0ce1b0 legacy=0x7f19bc0ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.150+0000 7f19bad9d640 1 -- 172.21.15.59:0/2240006229 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc15e380 con 0x7f19bc0d00c0 2024-03-06T07:23:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.150+0000 7f19b9d9b640 1 -- 172.21.15.59:0/2240006229 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446362964 0 0) 0x7f19bc0d09e0 con 0x7f19bc0ce1b0 2024-03-06T07:23:26.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.150+0000 7f19bad9d640 1 --2- 172.21.15.59:0/2240006229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19bc0d00c0 0x7f19bc0d04a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0097a0 tx=0x7f19ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b6bf8ee2da42a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.150+0000 7f19b9d9b640 1 -- 172.21.15.59:0/2240006229 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac035020 con 0x7f19bc0d00c0 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.150+0000 7f19b9d9b640 1 -- 172.21.15.59:0/2240006229 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac031830 con 0x7f19bc0d00c0 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.150+0000 7f19b9d9b640 1 -- 172.21.15.59:0/2240006229 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ac031c00 con 0x7f19bc0d00c0 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.151+0000 7f19c1081640 1 -- 172.21.15.59:0/2240006229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19bc0d00c0 msgr2=0x7f19bc0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.151+0000 7f19c1081640 1 --2- 172.21.15.59:0/2240006229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19bc0d00c0 0x7f19bc0d04a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0097a0 tx=0x7f19ac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.151+0000 7f19c1081640 1 -- 172.21.15.59:0/2240006229 shutdown_connections 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.151+0000 7f19c1081640 1 --2- 172.21.15.59:0/2240006229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19bc0cf850 0x7f19bc0cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.151+0000 7f19c1081640 1 --2- 172.21.15.59:0/2240006229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19bc0d00c0 0x7f19bc0d04a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.151+0000 7f19c1081640 1 -- 172.21.15.59:0/2240006229 >> 172.21.15.59:0/2240006229 conn(0x7f19bc0b35a0 msgr2=0x7f19bc0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.151+0000 7f19c1081640 1 -- 172.21.15.59:0/2240006229 shutdown_connections 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.151+0000 7f19c1081640 1 -- 172.21.15.59:0/2240006229 wait complete. 2024-03-06T07:23:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.152+0000 7f19c1081640 1 Processor -- start 2024-03-06T07:23:26.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.153+0000 7f19c1081640 1 -- start start 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.153+0000 7f19c1081640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19bc0ce1b0 0x7f19bc154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.153+0000 7f19c1081640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19bc0cf850 0x7f19bc0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.153+0000 7f19c1081640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19bc0d00c0 0x7f19bc0b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.153+0000 7f19c1081640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc06c440 con 0x7f19bc0cf850 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.153+0000 7f19c1081640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc06c160 con 0x7f19bc0ce1b0 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.153+0000 7f19c1081640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc06c720 con 0x7f19bc0d00c0 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.153+0000 7f19ba59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19bc0cf850 0x7f19bc0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.153+0000 7f19ba59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19bc0cf850 0x7f19bc0c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40872/0 (socket says 172.21.15.59:40872) 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.153+0000 7f19ba59c640 1 -- 172.21.15.59:0/248271491 learned_addr learned my addr 172.21.15.59:0/248271491 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19bb59e640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19bc0d00c0 0x7f19bc0b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19bad9d640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19bc0ce1b0 0x7f19bc154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19ba59c640 1 -- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19bc0d00c0 msgr2=0x7f19bc0b65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19ba59c640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19bc0d00c0 0x7f19bc0b65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19ba59c640 1 -- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19bc0ce1b0 msgr2=0x7f19bc154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19ba59c640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19bc0ce1b0 0x7f19bc154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19ba59c640 1 -- 172.21.15.59:0/248271491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc15d500 con 0x7f19bc0cf850 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19ba59c640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19bc0cf850 0x7f19bc0c5eb0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f19b000d830 tx=0x7f19b000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19a37fe640 1 -- 172.21.15.59:0/248271491 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b000b6c0 con 0x7f19bc0cf850 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19a37fe640 1 -- 172.21.15.59:0/248271491 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b0019070 con 0x7f19bc0cf850 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19c1081640 1 -- 172.21.15.59:0/248271491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19bc0b5540 con 0x7f19bc0cf850 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19a37fe640 1 -- 172.21.15.59:0/248271491 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b000ba80 con 0x7f19bc0cf850 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.154+0000 7f19c1081640 1 -- 172.21.15.59:0/248271491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19bc0b5b00 con 0x7f19bc0cf850 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.155+0000 7f19c1081640 1 -- 172.21.15.59:0/248271491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19bc04f330 con 0x7f19bc0cf850 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.156+0000 7f19a37fe640 1 -- 172.21.15.59:0/248271491 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19b001e460 con 0x7f19bc0cf850 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.156+0000 7f19a37fe640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19a806b500 0x7f19a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.156+0000 7f19a37fe640 1 -- 172.21.15.59:0/248271491 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f19b008a570 con 0x7f19bc0cf850 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.156+0000 7f19bad9d640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19a806b500 0x7f19a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.157+0000 7f19bad9d640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19a806b500 0x7f19a806d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f19ac036000 tx=0x7f19ac008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.160+0000 7f19a37fe640 1 -- 172.21.15.59:0/248271491 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19b0059740 con 0x7f19bc0cf850 2024-03-06T07:23:26.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.342+0000 7f19c1081640 1 -- 172.21.15.59:0/248271491 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f19bc0747a0 con 0x7f19bc0cf850 2024-03-06T07:23:26.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.342+0000 7f19a37fe640 1 -- 172.21.15.59:0/248271491 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) v1 ==== 84+0+65 (secure 0 0 0) 0x7f19b0059040 con 0x7f19bc0cf850 2024-03-06T07:23:26.345 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:26.345 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-03-06T07:23:26.345 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:23:26.345 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-03-06T07:23:26.345 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-03-06T07:23:26.345 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:26.345 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-03-06T07:23:26.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 -- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19a806b500 msgr2=0x7f19a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19a806b500 0x7f19a806d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f19ac036000 tx=0x7f19ac008030 comp rx=0 tx=0).stop 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 -- 172.21.15.59:0/248271491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19bc0cf850 msgr2=0x7f19bc0c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19bc0cf850 0x7f19bc0c5eb0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f19b000d830 tx=0x7f19b000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 -- 172.21.15.59:0/248271491 shutdown_connections 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19a806b500 0x7f19a806d9c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19bc0d00c0 0x7f19bc0b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19bc0cf850 0x7f19bc0c5eb0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 --2- 172.21.15.59:0/248271491 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19bc0ce1b0 0x7f19bc154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 -- 172.21.15.59:0/248271491 >> 172.21.15.59:0/248271491 conn(0x7f19bc0b35a0 msgr2=0x7f19bc0d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 -- 172.21.15.59:0/248271491 shutdown_connections 2024-03-06T07:23:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.344+0000 7f19c1081640 1 -- 172.21.15.59:0/248271491 wait complete. 2024-03-06T07:23:26.357 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:23:26.357 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.461+0000 7fe811637640 1 Processor -- start 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.461+0000 7fe811637640 1 -- start start 2024-03-06T07:23:26.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe811637640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe80c0cfa50 0x7fe80c0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe811637640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe80c0cf590 0x7fe80c0b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe811637640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80c161a10 con 0x7fe80c161630 2024-03-06T07:23:26.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe811637640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80c0d12e0 con 0x7fe80c0cfa50 2024-03-06T07:23:26.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe811637640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80c0d29e0 con 0x7fe80c0cf590 2024-03-06T07:23:26.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe80a7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe80c0cfa50 0x7fe80c0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe80a7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe80c0cfa50 0x7fe80c0cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40888/0 (socket says 172.21.15.59:40888) 2024-03-06T07:23:26.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe80a7fc640 1 -- 172.21.15.59:0/3992704975 learned_addr learned my addr 172.21.15.59:0/3992704975 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe80a7fc640 1 -- 172.21.15.59:0/3992704975 >> v2:172.21.15.164:3300/0 conn(0x7fe80c0cf590 msgr2=0x7fe80c0b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe80a7fc640 1 --2- 172.21.15.59:0/3992704975 >> v2:172.21.15.164:3300/0 conn(0x7fe80c0cf590 0x7fe80c0b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe80a7fc640 1 -- 172.21.15.59:0/3992704975 >> v1:172.21.15.164:6789/0 conn(0x7fe80c161630 legacy=0x7fe80c0ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.462+0000 7fe80a7fc640 1 -- 172.21.15.59:0/3992704975 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c0d2cc0 con 0x7fe80c0cfa50 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.463+0000 7fe80a7fc640 1 --2- 172.21.15.59:0/3992704975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c0cfa50 0x7fe80c0cf050 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0d1460 tx=0x7fe7f402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7e70ee865a11aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.463+0000 7fe809ffb640 1 -- 172.21.15.59:0/3992704975 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4035020 con 0x7fe80c0cfa50 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.463+0000 7fe809ffb640 1 -- 172.21.15.59:0/3992704975 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f4031800 con 0x7fe80c0cfa50 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.463+0000 7fe809ffb640 1 -- 172.21.15.59:0/3992704975 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4031bd0 con 0x7fe80c0cfa50 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.464+0000 7fe811637640 1 -- 172.21.15.59:0/3992704975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c0cfa50 msgr2=0x7fe80c0cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.464+0000 7fe811637640 1 --2- 172.21.15.59:0/3992704975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c0cfa50 0x7fe80c0cf050 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0d1460 tx=0x7fe7f402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.464+0000 7fe811637640 1 -- 172.21.15.59:0/3992704975 shutdown_connections 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.464+0000 7fe811637640 1 --2- 172.21.15.59:0/3992704975 >> v2:172.21.15.164:3300/0 conn(0x7fe80c0cf590 0x7fe80c0b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.464+0000 7fe811637640 1 --2- 172.21.15.59:0/3992704975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c0cfa50 0x7fe80c0cf050 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.464+0000 7fe811637640 1 -- 172.21.15.59:0/3992704975 >> 172.21.15.59:0/3992704975 conn(0x7fe80c0d3f30 msgr2=0x7fe80c0d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.464+0000 7fe811637640 1 -- 172.21.15.59:0/3992704975 shutdown_connections 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.464+0000 7fe811637640 1 -- 172.21.15.59:0/3992704975 wait complete. 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.465+0000 7fe811637640 1 Processor -- start 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.465+0000 7fe811637640 1 -- start start 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe811637640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe80c0cf590 0x7fe80c151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe811637640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe80c0cfa50 0x7fe80c0c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe811637640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c161630 0x7fe80c0b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe811637640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0d1050 con 0x7fe80c161630 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe811637640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0d0ed0 con 0x7fe80c0cfa50 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe811637640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0b41b0 con 0x7fe80c0cf590 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c161630 0x7fe80c0b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c161630 0x7fe80c0b6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40890/0 (socket says 172.21.15.59:40890) 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80b7fe640 1 -- 172.21.15.59:0/90550371 learned_addr learned my addr 172.21.15.59:0/90550371 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80affd640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe80c0cf590 0x7fe80c151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80a7fc640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe80c0cfa50 0x7fe80c0c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80b7fe640 1 -- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe80c0cf590 msgr2=0x7fe80c151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80b7fe640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe80c0cf590 0x7fe80c151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80b7fe640 1 -- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe80c0cfa50 msgr2=0x7fe80c0c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80b7fe640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe80c0cfa50 0x7fe80c0c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80b7fe640 1 -- 172.21.15.59:0/90550371 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c0c9f80 con 0x7fe80c161630 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80a7fc640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe80c0cfa50 0x7fe80c0c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe80b7fe640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c161630 0x7fe80c0b6af0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00b9e0 tx=0x7fe7fc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.466+0000 7fe8037fe640 1 -- 172.21.15.59:0/90550371 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc019070 con 0x7fe80c161630 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.467+0000 7fe8037fe640 1 -- 172.21.15.59:0/90550371 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7fc00cb40 con 0x7fe80c161630 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.467+0000 7fe811637640 1 -- 172.21.15.59:0/90550371 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80c0ca260 con 0x7fe80c161630 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.467+0000 7fe8037fe640 1 -- 172.21.15.59:0/90550371 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc014440 con 0x7fe80c161630 2024-03-06T07:23:26.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.467+0000 7fe811637640 1 -- 172.21.15.59:0/90550371 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80c0cc200 con 0x7fe80c161630 2024-03-06T07:23:26.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.468+0000 7fe811637640 1 -- 172.21.15.59:0/90550371 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80c04f330 con 0x7fe80c161630 2024-03-06T07:23:26.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.468+0000 7fe8037fe640 1 -- 172.21.15.59:0/90550371 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7fc01c070 con 0x7fe80c161630 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.468+0000 7fe8037fe640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7ec06b4d0 0x7fe7ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.469+0000 7fe8037fe640 1 -- 172.21.15.59:0/90550371 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe7fc089a90 con 0x7fe80c161630 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.469+0000 7fe80affd640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7ec06b4d0 0x7fe7ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.469+0000 7fe80affd640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7ec06b4d0 0x7fe7ec06d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe804006fd0 tx=0x7fe804008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.473+0000 7fe8037fe640 1 -- 172.21.15.59:0/90550371 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7fc058cf0 con 0x7fe80c161630 2024-03-06T07:23:26.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:26.653+0000 7fe811637640 1 -- 172.21.15.59:0/90550371 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe80c0671b0 con 0x7fe80c161630 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.024+0000 7fe8037fe640 1 -- 172.21.15.59:0/90550371 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe7fc0585f0 con 0x7fe80c161630 2024-03-06T07:23:27.026 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 -- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7ec06b4d0 msgr2=0x7fe7ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7ec06b4d0 0x7fe7ec06d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe804006fd0 tx=0x7fe804008040 comp rx=0 tx=0).stop 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 -- 172.21.15.59:0/90550371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c161630 msgr2=0x7fe80c0b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c161630 0x7fe80c0b6af0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00b9e0 tx=0x7fe7fc00bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 -- 172.21.15.59:0/90550371 shutdown_connections 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7ec06b4d0 0x7fe7ec06d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe80c161630 0x7fe80c0b6af0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe80c0cfa50 0x7fe80c0c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 --2- 172.21.15.59:0/90550371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe80c0cf590 0x7fe80c151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 -- 172.21.15.59:0/90550371 >> 172.21.15.59:0/90550371 conn(0x7fe80c0d3f30 msgr2=0x7fe80c15f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 -- 172.21.15.59:0/90550371 shutdown_connections 2024-03-06T07:23:27.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.026+0000 7fe811637640 1 -- 172.21.15.59:0/90550371 wait complete. 2024-03-06T07:23:27.043 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:23:27.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.151+0000 7fd88cb5e640 1 Processor -- start 2024-03-06T07:23:27.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.151+0000 7fd88cb5e640 1 -- start start 2024-03-06T07:23:27.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.151+0000 7fd88cb5e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd8880cea50 0x7fd8880b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.151+0000 7fd88cb5e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88806d030 con 0x7fd88815def0 2024-03-06T07:23:27.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.151+0000 7fd88cb5e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88806d310 con 0x7fd88806cab0 2024-03-06T07:23:27.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.151+0000 7fd88cb5e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8880cf1c0 con 0x7fd8880cea50 2024-03-06T07:23:27.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.152+0000 7fd88659c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd88815def0 0x7fd8880ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42888/0 (socket says 172.21.15.59:42888) 2024-03-06T07:23:27.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.152+0000 7fd88659c640 1 -- 172.21.15.59:0/961987502 learned_addr learned my addr 172.21.15.59:0/961987502 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:27.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.152+0000 7fd87ffff640 1 --2- 172.21.15.59:0/961987502 >> v2:172.21.15.164:3300/0 conn(0x7fd8880cea50 0x7fd8880b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.152+0000 7fd885d9b640 1 -- 172.21.15.59:0/961987502 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216482220 0 0) 0x7fd88806d030 con 0x7fd88815def0 2024-03-06T07:23:27.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.152+0000 7fd885d9b640 1 -- 172.21.15.59:0/961987502 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd874003520 con 0x7fd88815def0 2024-03-06T07:23:27.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.152+0000 7fd87ffff640 1 -- 172.21.15.59:0/961987502 >> v1:172.21.15.164:6789/0 conn(0x7fd88806cab0 legacy=0x7fd8880ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.152+0000 7fd885d9b640 1 -- 172.21.15.59:0/961987502 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1813029905 0 0) 0x7fd874003520 con 0x7fd88815def0 2024-03-06T07:23:27.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.152+0000 7fd87ffff640 1 -- 172.21.15.59:0/961987502 >> v1:172.21.15.59:6789/0 conn(0x7fd88815def0 legacy=0x7fd8880ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.152+0000 7fd87ffff640 1 -- 172.21.15.59:0/961987502 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8880cc110 con 0x7fd8880cea50 2024-03-06T07:23:27.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.153+0000 7fd87ffff640 1 --2- 172.21.15.59:0/961987502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8880cea50 0x7fd8880b41b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd8700097a0 tx=0x7fd87002b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=363c6169134d9e48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.153+0000 7fd885d9b640 1 -- 172.21.15.59:0/961987502 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd870037070 con 0x7fd8880cea50 2024-03-06T07:23:27.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.153+0000 7fd885d9b640 1 -- 172.21.15.59:0/961987502 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd870030510 con 0x7fd8880cea50 2024-03-06T07:23:27.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.153+0000 7fd885d9b640 1 -- 172.21.15.59:0/961987502 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd878001670 con 0x7fd8880cea50 2024-03-06T07:23:27.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.154+0000 7fd88cb5e640 1 -- 172.21.15.59:0/961987502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8880cea50 msgr2=0x7fd8880b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.154+0000 7fd88cb5e640 1 --2- 172.21.15.59:0/961987502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8880cea50 0x7fd8880b41b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd8700097a0 tx=0x7fd87002b680 comp rx=0 tx=0).stop 2024-03-06T07:23:27.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.154+0000 7fd88cb5e640 1 -- 172.21.15.59:0/961987502 shutdown_connections 2024-03-06T07:23:27.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.154+0000 7fd88cb5e640 1 --2- 172.21.15.59:0/961987502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8880cea50 0x7fd8880b41b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.154+0000 7fd88cb5e640 1 -- 172.21.15.59:0/961987502 >> 172.21.15.59:0/961987502 conn(0x7fd8880b3690 msgr2=0x7fd8880b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.154+0000 7fd88cb5e640 1 -- 172.21.15.59:0/961987502 shutdown_connections 2024-03-06T07:23:27.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.154+0000 7fd88cb5e640 1 -- 172.21.15.59:0/961987502 wait complete. 2024-03-06T07:23:27.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.155+0000 7fd88cb5e640 1 Processor -- start 2024-03-06T07:23:27.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.155+0000 7fd88cb5e640 1 -- start start 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd88cb5e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd88806cab0 0x7fd888157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd88cb5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8880cea50 0x7fd8881f7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd88cb5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88815def0 0x7fd8880b6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd88cb5e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88815e2d0 con 0x7fd8880cea50 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd88cb5e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8880cc760 con 0x7fd88806cab0 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd88cb5e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88815e450 con 0x7fd88815def0 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd87ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8880cea50 0x7fd8881f7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd87ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8880cea50 0x7fd8881f7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40898/0 (socket says 172.21.15.59:40898) 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd87ffff640 1 -- 172.21.15.59:0/1327743102 learned_addr learned my addr 172.21.15.59:0/1327743102 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd88659c640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd88806cab0 0x7fd888157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd886d9d640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88815def0 0x7fd8880b6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd87ffff640 1 -- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88815def0 msgr2=0x7fd8880b6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd87ffff640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88815def0 0x7fd8880b6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd87ffff640 1 -- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd88806cab0 msgr2=0x7fd888157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd87ffff640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd88806cab0 0x7fd888157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd87ffff640 1 -- 172.21.15.59:0/1327743102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8881f84f0 con 0x7fd8880cea50 2024-03-06T07:23:27.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.156+0000 7fd87ffff640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8880cea50 0x7fd8881f7fb0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd870000c00 tx=0x7fd8700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.157+0000 7fd87effd640 1 -- 172.21.15.59:0/1327743102 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd870042070 con 0x7fd8880cea50 2024-03-06T07:23:27.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.157+0000 7fd87effd640 1 -- 172.21.15.59:0/1327743102 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87003e870 con 0x7fd8880cea50 2024-03-06T07:23:27.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.157+0000 7fd88cb5e640 1 -- 172.21.15.59:0/1327743102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8880c9310 con 0x7fd8880cea50 2024-03-06T07:23:27.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.157+0000 7fd87effd640 1 -- 172.21.15.59:0/1327743102 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd870037040 con 0x7fd8880cea50 2024-03-06T07:23:27.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.157+0000 7fd88cb5e640 1 -- 172.21.15.59:0/1327743102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8880c98d0 con 0x7fd8880cea50 2024-03-06T07:23:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.158+0000 7fd88cb5e640 1 -- 172.21.15.59:0/1327743102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd88809fcf0 con 0x7fd8880cea50 2024-03-06T07:23:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.158+0000 7fd87effd640 1 -- 172.21.15.59:0/1327743102 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd870045070 con 0x7fd8880cea50 2024-03-06T07:23:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.158+0000 7fd87effd640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd86806b4d0 0x7fd86806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.159+0000 7fd87effd640 1 -- 172.21.15.59:0/1327743102 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd8700abf50 con 0x7fd8880cea50 2024-03-06T07:23:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.159+0000 7fd88659c640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd86806b4d0 0x7fd86806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.159+0000 7fd88659c640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd86806b4d0 0x7fd86806d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd8780011e0 tx=0x7fd878001530 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.163+0000 7fd87effd640 1 -- 172.21.15.59:0/1327743102 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8881f84f0 con 0x7fd8880cea50 2024-03-06T07:23:27.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:27.343+0000 7fd88cb5e640 1 -- 172.21.15.59:0/1327743102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd888068ff0 con 0x7fd8880cea50 2024-03-06T07:23:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.031+0000 7fd87effd640 1 -- 172.21.15.59:0/1327743102 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd87007b160 con 0x7fd8880cea50 2024-03-06T07:23:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-03-06T07:23:28.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.033+0000 7fd88cb5e640 1 -- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd86806b4d0 msgr2=0x7fd86806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.033+0000 7fd88cb5e640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd86806b4d0 0x7fd86806d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd8780011e0 tx=0x7fd878001530 comp rx=0 tx=0).stop 2024-03-06T07:23:28.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.033+0000 7fd88cb5e640 1 -- 172.21.15.59:0/1327743102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8880cea50 msgr2=0x7fd8881f7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.033+0000 7fd88cb5e640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8880cea50 0x7fd8881f7fb0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd870000c00 tx=0x7fd8700046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.033+0000 7fd88cb5e640 1 -- 172.21.15.59:0/1327743102 shutdown_connections 2024-03-06T07:23:28.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.033+0000 7fd88cb5e640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd86806b4d0 0x7fd86806d990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.033+0000 7fd88cb5e640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88815def0 0x7fd8880b6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.033+0000 7fd88cb5e640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8880cea50 0x7fd8881f7fb0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.033+0000 7fd88cb5e640 1 --2- 172.21.15.59:0/1327743102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd88806cab0 0x7fd888157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.033+0000 7fd88cb5e640 1 -- 172.21.15.59:0/1327743102 >> 172.21.15.59:0/1327743102 conn(0x7fd8880b3690 msgr2=0x7fd888160be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.033+0000 7fd88cb5e640 1 -- 172.21.15.59:0/1327743102 shutdown_connections 2024-03-06T07:23:28.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.034+0000 7fd88cb5e640 1 -- 172.21.15.59:0/1327743102 wait complete. 2024-03-06T07:23:28.058 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:23:28.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.167+0000 7ff5531a2640 1 Processor -- start 2024-03-06T07:23:28.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.167+0000 7ff5531a2640 1 -- start start 2024-03-06T07:23:28.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5531a2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff54c161320 0x7ff54c0b43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5531a2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff54c1619e0 0x7ff54c0d20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5531a2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54c1623a0 con 0x7ff54c161e70 2024-03-06T07:23:28.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5531a2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54c0b3f60 con 0x7ff54c161320 2024-03-06T07:23:28.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5531a2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54c162680 con 0x7ff54c1619e0 2024-03-06T07:23:28.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5521a0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff54c161320 0x7ff54c0b43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5521a0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff54c161320 0x7ff54c0b43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40906/0 (socket says 172.21.15.59:40906) 2024-03-06T07:23:28.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5521a0640 1 -- 172.21.15.59:0/2472206837 learned_addr learned my addr 172.21.15.59:0/2472206837 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:28.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff55199f640 1 --2- 172.21.15.59:0/2472206837 >> v2:172.21.15.164:3300/0 conn(0x7ff54c1619e0 0x7ff54c0d20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5521a0640 1 -- 172.21.15.59:0/2472206837 >> v2:172.21.15.164:3300/0 conn(0x7ff54c1619e0 msgr2=0x7ff54c0d20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5521a0640 1 --2- 172.21.15.59:0/2472206837 >> v2:172.21.15.164:3300/0 conn(0x7ff54c1619e0 0x7ff54c0d20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5521a0640 1 -- 172.21.15.59:0/2472206837 >> v1:172.21.15.164:6789/0 conn(0x7ff54c161e70 legacy=0x7ff54c0ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:28.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.168+0000 7ff5521a0640 1 -- 172.21.15.59:0/2472206837 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c0b3bf0 con 0x7ff54c161320 2024-03-06T07:23:28.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.169+0000 7ff5521a0640 1 --2- 172.21.15.59:0/2472206837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff54c161320 0x7ff54c0b43f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff54c0d1590 tx=0x7ff54802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7884e43d57373625 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.169+0000 7ff55119e640 1 -- 172.21.15.59:0/2472206837 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548035020 con 0x7ff54c161320 2024-03-06T07:23:28.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.169+0000 7ff55119e640 1 -- 172.21.15.59:0/2472206837 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff548031870 con 0x7ff54c161320 2024-03-06T07:23:28.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.169+0000 7ff55119e640 1 -- 172.21.15.59:0/2472206837 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548031c40 con 0x7ff54c161320 2024-03-06T07:23:28.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.169+0000 7ff5531a2640 1 -- 172.21.15.59:0/2472206837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff54c161320 msgr2=0x7ff54c0b43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.169+0000 7ff5531a2640 1 --2- 172.21.15.59:0/2472206837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff54c161320 0x7ff54c0b43f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff54c0d1590 tx=0x7ff54802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:28.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.170+0000 7ff5531a2640 1 -- 172.21.15.59:0/2472206837 shutdown_connections 2024-03-06T07:23:28.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.170+0000 7ff5531a2640 1 --2- 172.21.15.59:0/2472206837 >> v2:172.21.15.164:3300/0 conn(0x7ff54c1619e0 0x7ff54c0d20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.170+0000 7ff5531a2640 1 --2- 172.21.15.59:0/2472206837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff54c161320 0x7ff54c0b43f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.170+0000 7ff5531a2640 1 -- 172.21.15.59:0/2472206837 >> 172.21.15.59:0/2472206837 conn(0x7ff54c08d4b0 msgr2=0x7ff54c09cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:28.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.170+0000 7ff5531a2640 1 -- 172.21.15.59:0/2472206837 shutdown_connections 2024-03-06T07:23:28.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.170+0000 7ff5531a2640 1 -- 172.21.15.59:0/2472206837 wait complete. 2024-03-06T07:23:28.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.171+0000 7ff5531a2640 1 Processor -- start 2024-03-06T07:23:28.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.171+0000 7ff5531a2640 1 -- start start 2024-03-06T07:23:28.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.171+0000 7ff5531a2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff54c161320 0x7ff54c152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.171+0000 7ff5531a2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff54c1619e0 0x7ff54c0c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.171+0000 7ff5531a2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff54c161e70 0x7ff54c0b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.171+0000 7ff5531a2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0b4ab0 con 0x7ff54c161e70 2024-03-06T07:23:28.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.171+0000 7ff5531a2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0b4930 con 0x7ff54c161320 2024-03-06T07:23:28.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.171+0000 7ff5531a2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0d1a30 con 0x7ff54c1619e0 2024-03-06T07:23:28.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5529a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff54c161e70 0x7ff54c0b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5529a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff54c161e70 0x7ff54c0b88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40908/0 (socket says 172.21.15.59:40908) 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5529a1640 1 -- 172.21.15.59:0/2871900816 learned_addr learned my addr 172.21.15.59:0/2871900816 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5521a0640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff54c161320 0x7ff54c152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff55199f640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff54c1619e0 0x7ff54c0c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5529a1640 1 -- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff54c1619e0 msgr2=0x7ff54c0c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5529a1640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff54c1619e0 0x7ff54c0c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5529a1640 1 -- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff54c161320 msgr2=0x7ff54c152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5529a1640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff54c161320 0x7ff54c152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5529a1640 1 -- 172.21.15.59:0/2871900816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c0c8970 con 0x7ff54c161e70 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5529a1640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff54c161e70 0x7ff54c0b88b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff54400b9a0 tx=0x7ff54400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff542ffd640 1 -- 172.21.15.59:0/2871900816 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544019070 con 0x7ff54c161e70 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff542ffd640 1 -- 172.21.15.59:0/2871900816 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54400cba0 con 0x7ff54c161e70 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff542ffd640 1 -- 172.21.15.59:0/2871900816 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544014440 con 0x7ff54c161e70 2024-03-06T07:23:28.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5531a2640 1 -- 172.21.15.59:0/2871900816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54c0c8c50 con 0x7ff54c161e70 2024-03-06T07:23:28.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.172+0000 7ff5531a2640 1 -- 172.21.15.59:0/2871900816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54c0cc470 con 0x7ff54c161e70 2024-03-06T07:23:28.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.173+0000 7ff5531a2640 1 -- 172.21.15.59:0/2871900816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54c0cc670 con 0x7ff54c161e70 2024-03-06T07:23:28.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.174+0000 7ff542ffd640 1 -- 172.21.15.59:0/2871900816 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff54401c070 con 0x7ff54c161e70 2024-03-06T07:23:28.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.174+0000 7ff542ffd640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53006b4d0 0x7ff53006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:28.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.174+0000 7ff542ffd640 1 -- 172.21.15.59:0/2871900816 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff544089ca0 con 0x7ff54c161e70 2024-03-06T07:23:28.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.174+0000 7ff5521a0640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53006b4d0 0x7ff53006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:28.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.175+0000 7ff5521a0640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53006b4d0 0x7ff53006d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff548036000 tx=0x7ff54803c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:28.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.179+0000 7ff542ffd640 1 -- 172.21.15.59:0/2871900816 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff54c0cc670 con 0x7ff54c161e70 2024-03-06T07:23:28.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:28.398+0000 7ff5531a2640 1 -- 172.21.15.59:0/2871900816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff54c068f10 con 0x7ff54c161e70 2024-03-06T07:23:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.032+0000 7ff542ffd640 1 -- 172.21.15.59:0/2871900816 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v17) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff5440585c0 con 0x7ff54c161e70 2024-03-06T07:23:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 6 and data pool 7 2024-03-06T07:23:29.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.040+0000 7ff5531a2640 1 -- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53006b4d0 msgr2=0x7ff53006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.040+0000 7ff5531a2640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53006b4d0 0x7ff53006d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff548036000 tx=0x7ff54803c040 comp rx=0 tx=0).stop 2024-03-06T07:23:29.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.040+0000 7ff5531a2640 1 -- 172.21.15.59:0/2871900816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff54c161e70 msgr2=0x7ff54c0b88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.040+0000 7ff5531a2640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff54c161e70 0x7ff54c0b88b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff54400b9a0 tx=0x7ff54400be60 comp rx=0 tx=0).stop 2024-03-06T07:23:29.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.041+0000 7ff5531a2640 1 -- 172.21.15.59:0/2871900816 shutdown_connections 2024-03-06T07:23:29.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.041+0000 7ff5531a2640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53006b4d0 0x7ff53006d990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.041+0000 7ff5531a2640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff54c161e70 0x7ff54c0b88b0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.041+0000 7ff5531a2640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff54c1619e0 0x7ff54c0c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.041+0000 7ff5531a2640 1 --2- 172.21.15.59:0/2871900816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff54c161320 0x7ff54c152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.041+0000 7ff5531a2640 1 -- 172.21.15.59:0/2871900816 >> 172.21.15.59:0/2871900816 conn(0x7ff54c08d4b0 msgr2=0x7ff54c0d20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.044+0000 7ff5531a2640 1 -- 172.21.15.59:0/2871900816 shutdown_connections 2024-03-06T07:23:29.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.044+0000 7ff5531a2640 1 -- 172.21.15.59:0/2871900816 wait complete. 2024-03-06T07:23:29.072 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:29.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.177+0000 7f940fdc9640 1 Processor -- start 2024-03-06T07:23:29.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.177+0000 7f940fdc9640 1 -- start start 2024-03-06T07:23:29.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940fdc9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f94080cf850 0x7f94080cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940fdc9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f940806cf20 0x7f940806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940fdc9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940815e380 con 0x7f94080d00c0 2024-03-06T07:23:29.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940fdc9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940806ca90 con 0x7f94080cf850 2024-03-06T07:23:29.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940fdc9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94080cdc10 con 0x7f940806cf20 2024-03-06T07:23:29.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940d364640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f94080cf850 0x7f94080cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940db65640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f94080d00c0 0x7f94080d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42902/0 (socket says 172.21.15.59:42902) 2024-03-06T07:23:29.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940d364640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f94080cf850 0x7f94080cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40914/0 (socket says 172.21.15.59:40914) 2024-03-06T07:23:29.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940db65640 1 -- 172.21.15.59:0/4139423358 learned_addr learned my addr 172.21.15.59:0/4139423358 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:29.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940e366640 1 --2- 172.21.15.59:0/4139423358 >> v2:172.21.15.164:3300/0 conn(0x7f940806cf20 0x7f940806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940d364640 1 -- 172.21.15.59:0/4139423358 >> v2:172.21.15.164:3300/0 conn(0x7f940806cf20 msgr2=0x7f940806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940d364640 1 --2- 172.21.15.59:0/4139423358 >> v2:172.21.15.164:3300/0 conn(0x7f940806cf20 0x7f940806d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940d364640 1 -- 172.21.15.59:0/4139423358 >> v1:172.21.15.59:6789/0 conn(0x7f94080d00c0 legacy=0x7f94080d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940d364640 1 -- 172.21.15.59:0/4139423358 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940806c160 con 0x7f94080cf850 2024-03-06T07:23:29.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.178+0000 7f940cb63640 1 -- 172.21.15.59:0/4139423358 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497797701 0 0) 0x7f940815e380 con 0x7f94080d00c0 2024-03-06T07:23:29.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.179+0000 7f940d364640 1 --2- 172.21.15.59:0/4139423358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94080cf850 0x7f94080cd3c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9408071f50 tx=0x7f93f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eddb43fdf533bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.179+0000 7f940cb63640 1 -- 172.21.15.59:0/4139423358 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8035020 con 0x7f94080cf850 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.179+0000 7f940cb63640 1 -- 172.21.15.59:0/4139423358 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f8031830 con 0x7f94080cf850 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.179+0000 7f940cb63640 1 -- 172.21.15.59:0/4139423358 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8031c00 con 0x7f94080cf850 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.180+0000 7f940fdc9640 1 -- 172.21.15.59:0/4139423358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94080cf850 msgr2=0x7f94080cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.180+0000 7f940fdc9640 1 --2- 172.21.15.59:0/4139423358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94080cf850 0x7f94080cd3c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9408071f50 tx=0x7f93f802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.180+0000 7f940fdc9640 1 -- 172.21.15.59:0/4139423358 shutdown_connections 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.180+0000 7f940fdc9640 1 --2- 172.21.15.59:0/4139423358 >> v2:172.21.15.164:3300/0 conn(0x7f940806cf20 0x7f940806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.180+0000 7f940fdc9640 1 --2- 172.21.15.59:0/4139423358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94080cf850 0x7f94080cd3c0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.180+0000 7f940fdc9640 1 -- 172.21.15.59:0/4139423358 >> 172.21.15.59:0/4139423358 conn(0x7f94080b35a0 msgr2=0x7f94080b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.180+0000 7f940fdc9640 1 -- 172.21.15.59:0/4139423358 shutdown_connections 2024-03-06T07:23:29.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.180+0000 7f940fdc9640 1 -- 172.21.15.59:0/4139423358 wait complete. 2024-03-06T07:23:29.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.181+0000 7f940fdc9640 1 Processor -- start 2024-03-06T07:23:29.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.181+0000 7f940fdc9640 1 -- start start 2024-03-06T07:23:29.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940fdc9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f940806cf20 0x7f94080b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940fdc9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94080cf850 0x7f9408154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940fdc9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94080d00c0 0x7f94080c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940fdc9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94080cd900 con 0x7f94080cf850 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940fdc9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940806c720 con 0x7f94080d00c0 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940fdc9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94080b1030 con 0x7f940806cf20 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940d364640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94080cf850 0x7f9408154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940d364640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94080cf850 0x7f9408154030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40924/0 (socket says 172.21.15.59:40924) 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940d364640 1 -- 172.21.15.59:0/3890333772 learned_addr learned my addr 172.21.15.59:0/3890333772 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940db65640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f940806cf20 0x7f94080b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940e366640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94080d00c0 0x7f94080c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940d364640 1 -- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f940806cf20 msgr2=0x7f94080b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940d364640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f940806cf20 0x7f94080b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940d364640 1 -- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94080d00c0 msgr2=0x7f94080c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940d364640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94080d00c0 0x7f94080c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940d364640 1 -- 172.21.15.59:0/3890333772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94080b76f0 con 0x7f94080cf850 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940e366640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94080d00c0 0x7f94080c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940db65640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f940806cf20 0x7f94080b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.182+0000 7f940d364640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94080cf850 0x7f9408154030 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009890 tx=0x7f93f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.183+0000 7f93fe7fc640 1 -- 172.21.15.59:0/3890333772 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8004230 con 0x7f94080cf850 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.183+0000 7f93fe7fc640 1 -- 172.21.15.59:0/3890333772 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f80043d0 con 0x7f94080cf850 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.183+0000 7f940fdc9640 1 -- 172.21.15.59:0/3890333772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94080b49b0 con 0x7f94080cf850 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.183+0000 7f93fe7fc640 1 -- 172.21.15.59:0/3890333772 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f803cad0 con 0x7f94080cf850 2024-03-06T07:23:29.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.183+0000 7f940fdc9640 1 -- 172.21.15.59:0/3890333772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94080c72a0 con 0x7f94080cf850 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.184+0000 7f940fdc9640 1 -- 172.21.15.59:0/3890333772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940804f330 con 0x7f94080cf850 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.184+0000 7f93fe7fc640 1 -- 172.21.15.59:0/3890333772 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93f8038020 con 0x7f94080cf850 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.184+0000 7f93fe7fc640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93ec06b4d0 0x7f93ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.185+0000 7f93fe7fc640 1 -- 172.21.15.59:0/3890333772 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93f80ac460 con 0x7f94080cf850 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.185+0000 7f940db65640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93ec06b4d0 0x7f93ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.185+0000 7f940db65640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93ec06b4d0 0x7f93ec06d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9404000f60 tx=0x7f9404009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.189+0000 7f93fe7fc640 1 -- 172.21.15.59:0/3890333772 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93f8041070 con 0x7f94080cf850 2024-03-06T07:23:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.372+0000 7f940fdc9640 1 -- 172.21.15.59:0/3890333772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94080cea80 con 0x7f94080cf850 2024-03-06T07:23:29.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.373+0000 7f93fe7fc640 1 -- 172.21.15.59:0/3890333772 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16614 (secure 0 0 0) 0x7f93f807b2e0 con 0x7f94080cf850 2024-03-06T07:23:29.375 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:29.375 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":44,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:23:28.400113+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:26.672668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:27.362891+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 -- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93ec06b4d0 msgr2=0x7f93ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93ec06b4d0 0x7f93ec06d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9404000f60 tx=0x7f9404009040 comp rx=0 tx=0).stop 2024-03-06T07:23:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 -- 172.21.15.59:0/3890333772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94080cf850 msgr2=0x7f9408154030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94080cf850 0x7f9408154030 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009890 tx=0x7f93f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 -- 172.21.15.59:0/3890333772 shutdown_connections 2024-03-06T07:23:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93ec06b4d0 0x7f93ec06d990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94080d00c0 0x7f94080c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94080cf850 0x7f9408154030 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 --2- 172.21.15.59:0/3890333772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f940806cf20 0x7f94080b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 -- 172.21.15.59:0/3890333772 >> 172.21.15.59:0/3890333772 conn(0x7f94080b35a0 msgr2=0x7f940806f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 -- 172.21.15.59:0/3890333772 shutdown_connections 2024-03-06T07:23:29.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.375+0000 7f940fdc9640 1 -- 172.21.15.59:0/3890333772 wait complete. 2024-03-06T07:23:29.388 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.487+0000 7f5390055640 1 Processor -- start 2024-03-06T07:23:29.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.487+0000 7f5390055640 1 -- start start 2024-03-06T07:23:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.488+0000 7f5390055640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f538815dfe0 0x7f53880cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.488+0000 7f5390055640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53880cf830 0x7f53880d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.488+0000 7f5390055640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f53880d27f0 0x7f53880d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.488+0000 7f5390055640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53880d2c50 con 0x7f53880d27f0 2024-03-06T07:23:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.488+0000 7f5390055640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53880b31e0 con 0x7f538815dfe0 2024-03-06T07:23:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.488+0000 7f5390055640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53880b3d60 con 0x7f53880cf830 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.488+0000 7f538e5f2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f53880d27f0 0x7f53880d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.488+0000 7f538e5f2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f53880d27f0 0x7f53880d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40934/0 (socket says 172.21.15.59:40934) 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.488+0000 7f538e5f2640 1 -- 172.21.15.59:0/1321379371 learned_addr learned my addr 172.21.15.59:0/1321379371 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.489+0000 7f538d5f0640 1 --2- 172.21.15.59:0/1321379371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53880cf830 0x7f53880d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.489+0000 7f538ddf1640 1 --2- 172.21.15.59:0/1321379371 >> v2:172.21.15.164:3300/0 conn(0x7f538815dfe0 0x7f53880cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.489+0000 7f538e5f2640 1 -- 172.21.15.59:0/1321379371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53880cf830 msgr2=0x7f53880d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.489+0000 7f538e5f2640 1 --2- 172.21.15.59:0/1321379371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53880cf830 0x7f53880d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.489+0000 7f538e5f2640 1 -- 172.21.15.59:0/1321379371 >> v2:172.21.15.164:3300/0 conn(0x7f538815dfe0 msgr2=0x7f53880cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.489+0000 7f538e5f2640 1 --2- 172.21.15.59:0/1321379371 >> v2:172.21.15.164:3300/0 conn(0x7f538815dfe0 0x7f53880cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.489+0000 7f538e5f2640 1 -- 172.21.15.59:0/1321379371 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880b3430 con 0x7f53880d27f0 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.489+0000 7f538ddf1640 1 --2- 172.21.15.59:0/1321379371 >> v2:172.21.15.164:3300/0 conn(0x7f538815dfe0 0x7f53880cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.489+0000 7f538e5f2640 1 --2- 172.21.15.59:0/1321379371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53880d27f0 0x7f53880d0300 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5388067990 tx=0x7f537802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88b55a3651cda5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f538cdef640 1 -- 172.21.15.59:0/1321379371 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378035020 con 0x7f53880d27f0 2024-03-06T07:23:29.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f538cdef640 1 -- 172.21.15.59:0/1321379371 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5378031870 con 0x7f53880d27f0 2024-03-06T07:23:29.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f538cdef640 1 -- 172.21.15.59:0/1321379371 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378031c40 con 0x7f53880d27f0 2024-03-06T07:23:29.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f5390055640 1 -- 172.21.15.59:0/1321379371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53880d27f0 msgr2=0x7f53880d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f5390055640 1 --2- 172.21.15.59:0/1321379371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53880d27f0 0x7f53880d0300 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5388067990 tx=0x7f537802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f5390055640 1 -- 172.21.15.59:0/1321379371 shutdown_connections 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f5390055640 1 --2- 172.21.15.59:0/1321379371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53880d27f0 0x7f53880d0300 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f5390055640 1 --2- 172.21.15.59:0/1321379371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53880cf830 0x7f53880d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f5390055640 1 --2- 172.21.15.59:0/1321379371 >> v2:172.21.15.164:3300/0 conn(0x7f538815dfe0 0x7f53880cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f5390055640 1 -- 172.21.15.59:0/1321379371 >> 172.21.15.59:0/1321379371 conn(0x7f53880d3cd0 msgr2=0x7f53880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f5390055640 1 -- 172.21.15.59:0/1321379371 shutdown_connections 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.490+0000 7f5390055640 1 -- 172.21.15.59:0/1321379371 wait complete. 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.491+0000 7f5390055640 1 Processor -- start 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.491+0000 7f5390055640 1 -- start start 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.492+0000 7f5390055640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53880cf830 0x7f538814f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.492+0000 7f5390055640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53880d27f0 0x7f53880cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.492+0000 7f5390055640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f538815dfe0 0x7f53880b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.492+0000 7f5390055640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880d1c70 con 0x7f538815dfe0 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.492+0000 7f5390055640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880b39f0 con 0x7f53880cf830 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.492+0000 7f5390055640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53880d1df0 con 0x7f53880d27f0 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.492+0000 7f538e5f2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f538815dfe0 0x7f53880b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.492+0000 7f538e5f2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f538815dfe0 0x7f53880b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40948/0 (socket says 172.21.15.59:40948) 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.492+0000 7f538e5f2640 1 -- 172.21.15.59:0/3410845565 learned_addr learned my addr 172.21.15.59:0/3410845565 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.492+0000 7f538ddf1640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53880cf830 0x7f538814f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.492+0000 7f538d5f0640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53880d27f0 0x7f53880cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f538e5f2640 1 -- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53880d27f0 msgr2=0x7f53880cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f538e5f2640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53880d27f0 0x7f53880cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f538e5f2640 1 -- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53880cf830 msgr2=0x7f538814f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f538e5f2640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53880cf830 0x7f538814f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f538e5f2640 1 -- 172.21.15.59:0/3410845565 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880cbcd0 con 0x7f538815dfe0 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f538d5f0640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53880d27f0 0x7f53880cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f538ddf1640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53880cf830 0x7f538814f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f538e5f2640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f538815dfe0 0x7f53880b2180 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5378009960 tx=0x7f53780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f53867fc640 1 -- 172.21.15.59:0/3410845565 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378040070 con 0x7f538815dfe0 2024-03-06T07:23:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f53867fc640 1 -- 172.21.15.59:0/3410845565 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5378030c50 con 0x7f538815dfe0 2024-03-06T07:23:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f5390055640 1 -- 172.21.15.59:0/3410845565 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53880cb710 con 0x7f538815dfe0 2024-03-06T07:23:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f53867fc640 1 -- 172.21.15.59:0/3410845565 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378035040 con 0x7f538815dfe0 2024-03-06T07:23:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.493+0000 7f5390055640 1 -- 172.21.15.59:0/3410845565 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53880cc440 con 0x7f538815dfe0 2024-03-06T07:23:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.494+0000 7f5390055640 1 -- 172.21.15.59:0/3410845565 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53880cf5f0 con 0x7f538815dfe0 2024-03-06T07:23:29.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.494+0000 7f53867fc640 1 -- 172.21.15.59:0/3410845565 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5378038020 con 0x7f538815dfe0 2024-03-06T07:23:29.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.495+0000 7f53867fc640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537406b4d0 0x7f537406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.495+0000 7f53867fc640 1 -- 172.21.15.59:0/3410845565 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53780ac060 con 0x7f538815dfe0 2024-03-06T07:23:29.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.495+0000 7f538ddf1640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537406b4d0 0x7f537406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.495+0000 7f538ddf1640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537406b4d0 0x7f537406d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f53880ca2e0 tx=0x7f53800074b0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.499+0000 7f53867fc640 1 -- 172.21.15.59:0/3410845565 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53880cf5f0 con 0x7f538815dfe0 2024-03-06T07:23:29.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.682+0000 7f5390055640 1 -- 172.21.15.59:0/3410845565 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53880cdfe0 con 0x7f538815dfe0 2024-03-06T07:23:29.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.683+0000 7f53867fc640 1 -- 172.21.15.59:0/3410845565 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16614 (secure 0 0 0) 0x7f537807aee0 con 0x7f538815dfe0 2024-03-06T07:23:29.685 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:29.685 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":44,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:23:28.400113+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:26.672668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:27.362891+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 -- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537406b4d0 msgr2=0x7f537406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537406b4d0 0x7f537406d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f53880ca2e0 tx=0x7f53800074b0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 -- 172.21.15.59:0/3410845565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f538815dfe0 msgr2=0x7f53880b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f538815dfe0 0x7f53880b2180 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5378009960 tx=0x7f53780046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 -- 172.21.15.59:0/3410845565 shutdown_connections 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537406b4d0 0x7f537406d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f538815dfe0 0x7f53880b2180 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53880d27f0 0x7f53880cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 --2- 172.21.15.59:0/3410845565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53880cf830 0x7f538814f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 -- 172.21.15.59:0/3410845565 >> 172.21.15.59:0/3410845565 conn(0x7f53880d3cd0 msgr2=0x7f53880cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 -- 172.21.15.59:0/3410845565 shutdown_connections 2024-03-06T07:23:29.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.685+0000 7f5390055640 1 -- 172.21.15.59:0/3410845565 wait complete. 2024-03-06T07:23:29.698 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:23:29.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.800+0000 7fdd2fbf7640 1 Processor -- start 2024-03-06T07:23:29.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.800+0000 7fdd2fbf7640 1 -- start start 2024-03-06T07:23:29.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2fbf7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdd280cfa90 0x7fdd280cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2fbf7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd28158ac0 con 0x7fdd280b33a0 2024-03-06T07:23:29.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2fbf7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2815b2a0 con 0x7fdd281586e0 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2fbf7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2815b540 con 0x7fdd280cfa90 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2e194640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdd280b33a0 0x7fdd280b37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42904/0 (socket says 172.21.15.59:42904) 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2e194640 1 -- 172.21.15.59:0/2799457489 learned_addr learned my addr 172.21.15.59:0/2799457489 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165189703 0 0) 0x7fdd28158ac0 con 0x7fdd280b33a0 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2d192640 1 --2- 172.21.15.59:0/2799457489 >> v2:172.21.15.164:3300/0 conn(0x7fdd280cfa90 0x7fdd280cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd14003520 con 0x7fdd280b33a0 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3940056475 0 0) 0x7fdd14003520 con 0x7fdd280b33a0 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 >> v2:172.21.15.164:3300/0 conn(0x7fdd280cfa90 msgr2=0x7fdd280cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2c991640 1 --2- 172.21.15.59:0/2799457489 >> v2:172.21.15.164:3300/0 conn(0x7fdd280cfa90 0x7fdd280cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 >> v1:172.21.15.164:6789/0 conn(0x7fdd281586e0 legacy=0x7fdd280aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2815b820 con 0x7fdd280b33a0 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fdd200019f0 con 0x7fdd280b33a0 2024-03-06T07:23:29.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.801+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 >> v1:172.21.15.59:6789/0 conn(0x7fdd280b33a0 legacy=0x7fdd280b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2c991640 1 --2- 172.21.15.59:0/2799457489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd140039b0 0x7fdd14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2c991640 1 --2- 172.21.15.59:0/2799457489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd14004800 0x7fdd14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2c991640 1 --2- 172.21.15.59:0/2799457489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd140051c0 0x7fdd1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2815b540 con 0x7fdd140051c0 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1400e270 con 0x7fdd140039b0 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1400e470 con 0x7fdd14004800 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2e194640 1 --2- 172.21.15.59:0/2799457489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd140051c0 0x7fdd1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2d993640 1 --2- 172.21.15.59:0/2799457489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd140039b0 0x7fdd14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2d192640 1 --2- 172.21.15.59:0/2799457489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd14004800 0x7fdd14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2e194640 1 -- 172.21.15.59:0/2799457489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd14004800 msgr2=0x7fdd14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2e194640 1 --2- 172.21.15.59:0/2799457489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd14004800 0x7fdd14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2e194640 1 -- 172.21.15.59:0/2799457489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd140039b0 msgr2=0x7fdd14003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2e194640 1 --2- 172.21.15.59:0/2799457489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd140039b0 0x7fdd14003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.802+0000 7fdd2e194640 1 -- 172.21.15.59:0/2799457489 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd14003520 con 0x7fdd140051c0 2024-03-06T07:23:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.803+0000 7fdd2e194640 1 -- 172.21.15.59:0/2799457489 reap_dead start 2024-03-06T07:23:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.803+0000 7fdd2e194640 1 --2- 172.21.15.59:0/2799457489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd140051c0 0x7fdd1400b9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000a860 tx=0x7fdd14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca57835b45c36e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.803+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2000a480 con 0x7fdd140051c0 2024-03-06T07:23:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.803+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd20034630 con 0x7fdd140051c0 2024-03-06T07:23:29.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.803+0000 7fdd2c991640 1 -- 172.21.15.59:0/2799457489 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20034a20 con 0x7fdd140051c0 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.803+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/2799457489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd140051c0 msgr2=0x7fdd1400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.803+0000 7fdd2fbf7640 1 --2- 172.21.15.59:0/2799457489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd140051c0 0x7fdd1400b9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000a860 tx=0x7fdd14005830 comp rx=0 tx=0).stop 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.803+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/2799457489 shutdown_connections 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.803+0000 7fdd2fbf7640 1 --2- 172.21.15.59:0/2799457489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd140051c0 0x7fdd1400b9c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.803+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/2799457489 >> 172.21.15.59:0/2799457489 conn(0x7fdd2806b190 msgr2=0x7fdd2806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.804+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/2799457489 shutdown_connections 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.804+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/2799457489 wait complete. 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.805+0000 7fdd2fbf7640 1 Processor -- start 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.805+0000 7fdd2fbf7640 1 -- start start 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.805+0000 7fdd2fbf7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd140051c0 0x7fdd281e1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.805+0000 7fdd2fbf7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd28156690 0x7fdd281e3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.805+0000 7fdd2fbf7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd281e3cc0 0x7fdd281e4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.805+0000 7fdd2fbf7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28071350 con 0x7fdd28156690 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.805+0000 7fdd2fbf7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2815bde0 con 0x7fdd281e3cc0 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.805+0000 7fdd2fbf7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd280714d0 con 0x7fdd140051c0 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.805+0000 7fdd2d192640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd28156690 0x7fdd281e3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2d192640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd28156690 0x7fdd281e3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40970/0 (socket says 172.21.15.59:40970) 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2d192640 1 -- 172.21.15.59:0/1412843312 learned_addr learned my addr 172.21.15.59:0/1412843312 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2d993640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd140051c0 0x7fdd281e1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2e194640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd281e3cc0 0x7fdd281e4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2d192640 1 -- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd140051c0 msgr2=0x7fdd281e1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2d192640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd140051c0 0x7fdd281e1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2d192640 1 -- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd281e3cc0 msgr2=0x7fdd281e4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2d192640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd281e3cc0 0x7fdd281e4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:29.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2d192640 1 -- 172.21.15.59:0/1412843312 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28156af0 con 0x7fdd28156690 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2d192640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd28156690 0x7fdd281e3780 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fdd180118e0 tx=0x7fdd18011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd1e7fc640 1 -- 172.21.15.59:0/1412843312 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1800faf0 con 0x7fdd28156690 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd1e7fc640 1 -- 172.21.15.59:0/1412843312 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1800fc90 con 0x7fdd28156690 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd1e7fc640 1 -- 172.21.15.59:0/1412843312 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18014440 con 0x7fdd28156690 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/1412843312 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd28156dc0 con 0x7fdd28156690 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.806+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/1412843312 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd28157380 con 0x7fdd28156690 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.807+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/1412843312 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd280732b0 con 0x7fdd28156690 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.807+0000 7fdd1e7fc640 1 -- 172.21.15.59:0/1412843312 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd18012040 con 0x7fdd28156690 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.808+0000 7fdd1e7fc640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd0c06b4d0 0x7fdd0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.808+0000 7fdd1e7fc640 1 -- 172.21.15.59:0/1412843312 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd1808a930 con 0x7fdd28156690 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.808+0000 7fdd2d993640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd0c06b4d0 0x7fdd0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:29.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.809+0000 7fdd2d993640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd0c06b4d0 0x7fdd0c06d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd24000c00 tx=0x7fdd24009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:29.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:29.812+0000 7fdd1e7fc640 1 -- 172.21.15.59:0/1412843312 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd18019030 con 0x7fdd28156690 2024-03-06T07:23:30.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:30.033+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/1412843312 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdd28155900 con 0x7fdd28156690 2024-03-06T07:23:31.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.035+0000 7fdd1e7fc640 1 -- 172.21.15.59:0/1412843312 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v19) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdd18059160 con 0x7fdd28156690 2024-03-06T07:23:31.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd0c06b4d0 msgr2=0x7fdd0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd0c06b4d0 0x7fdd0c06d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd24000c00 tx=0x7fdd24009040 comp rx=0 tx=0).stop 2024-03-06T07:23:31.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/1412843312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd28156690 msgr2=0x7fdd281e3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd28156690 0x7fdd281e3780 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fdd180118e0 tx=0x7fdd18011da0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/1412843312 shutdown_connections 2024-03-06T07:23:31.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd0c06b4d0 0x7fdd0c06d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd281e3cc0 0x7fdd281e4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd28156690 0x7fdd281e3780 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 --2- 172.21.15.59:0/1412843312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd140051c0 0x7fdd281e1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/1412843312 >> 172.21.15.59:0/1412843312 conn(0x7fdd2806b190 msgr2=0x7fdd28158ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/1412843312 shutdown_connections 2024-03-06T07:23:31.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.038+0000 7fdd2fbf7640 1 -- 172.21.15.59:0/1412843312 wait complete. 2024-03-06T07:23:31.053 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:23:31.053 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:23:31.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.162+0000 7fe18326b640 1 Processor -- start 2024-03-06T07:23:31.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.162+0000 7fe18326b640 1 -- start start 2024-03-06T07:23:31.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe18326b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe17c1586e0 0x7fe17c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe18326b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0d01a0 0x7fe17c0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe18326b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17c158ac0 con 0x7fe17c0aef80 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe18326b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe17c15b2a0 con 0x7fe17c1586e0 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe18326b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c15b540 con 0x7fe17c0d01a0 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe181007640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe17c1586e0 0x7fe17c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe180806640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe17c0aef80 0x7fe17c0cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42914/0 (socket says 172.21.15.59:42914) 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe180806640 1 -- 172.21.15.59:0/1373553679 learned_addr learned my addr 172.21.15.59:0/1373553679 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe181808640 1 --2- 172.21.15.59:0/1373553679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0d01a0 0x7fe17c0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe181007640 1 -- 172.21.15.59:0/1373553679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0d01a0 msgr2=0x7fe17c0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe181007640 1 --2- 172.21.15.59:0/1373553679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0d01a0 0x7fe17c0b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe181007640 1 -- 172.21.15.59:0/1373553679 >> v1:172.21.15.59:6789/0 conn(0x7fe17c0aef80 legacy=0x7fe17c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe17bfff640 1 -- 172.21.15.59:0/1373553679 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488503360 0 0) 0x7fe17c158ac0 con 0x7fe17c0aef80 2024-03-06T07:23:31.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.163+0000 7fe181007640 1 -- 172.21.15.59:0/1373553679 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c15b820 con 0x7fe17c1586e0 2024-03-06T07:23:31.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.164+0000 7fe181007640 1 --2- 172.21.15.59:0/1373553679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c1586e0 0x7fe17c0aea40 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe1740097a0 tx=0x7fe17402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55418627ac989d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.164+0000 7fe17bfff640 1 -- 172.21.15.59:0/1373553679 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe174035020 con 0x7fe17c1586e0 2024-03-06T07:23:31.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.164+0000 7fe17bfff640 1 -- 172.21.15.59:0/1373553679 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe174031830 con 0x7fe17c1586e0 2024-03-06T07:23:31.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.164+0000 7fe17bfff640 1 -- 172.21.15.59:0/1373553679 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe174031c00 con 0x7fe17c1586e0 2024-03-06T07:23:31.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.165+0000 7fe18326b640 1 -- 172.21.15.59:0/1373553679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c1586e0 msgr2=0x7fe17c0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.165+0000 7fe18326b640 1 --2- 172.21.15.59:0/1373553679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c1586e0 0x7fe17c0aea40 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe1740097a0 tx=0x7fe17402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.165+0000 7fe18326b640 1 -- 172.21.15.59:0/1373553679 shutdown_connections 2024-03-06T07:23:31.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.165+0000 7fe18326b640 1 --2- 172.21.15.59:0/1373553679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0d01a0 0x7fe17c0b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.165+0000 7fe18326b640 1 --2- 172.21.15.59:0/1373553679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c1586e0 0x7fe17c0aea40 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.165+0000 7fe18326b640 1 -- 172.21.15.59:0/1373553679 >> 172.21.15.59:0/1373553679 conn(0x7fe17c06b190 msgr2=0x7fe17c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.165+0000 7fe18326b640 1 -- 172.21.15.59:0/1373553679 shutdown_connections 2024-03-06T07:23:31.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.165+0000 7fe18326b640 1 -- 172.21.15.59:0/1373553679 wait complete. 2024-03-06T07:23:31.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.166+0000 7fe18326b640 1 Processor -- start 2024-03-06T07:23:31.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.166+0000 7fe18326b640 1 -- start start 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe18326b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe17c0aef80 0x7fe17c1451d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe18326b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0d01a0 0x7fe17c14fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe18326b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c1586e0 0x7fe17c156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe18326b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c071350 con 0x7fe17c1586e0 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe18326b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c15bde0 con 0x7fe17c0aef80 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe18326b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c0714d0 con 0x7fe17c0d01a0 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe181808640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c1586e0 0x7fe17c156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe181808640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c1586e0 0x7fe17c156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40994/0 (socket says 172.21.15.59:40994) 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe181808640 1 -- 172.21.15.59:0/2161749200 learned_addr learned my addr 172.21.15.59:0/2161749200 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe181007640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe17c0aef80 0x7fe17c1451d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe180806640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0d01a0 0x7fe17c14fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe181808640 1 -- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0d01a0 msgr2=0x7fe17c14fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe181808640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0d01a0 0x7fe17c14fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe181808640 1 -- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe17c0aef80 msgr2=0x7fe17c1451d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe181808640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe17c0aef80 0x7fe17c1451d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe181808640 1 -- 172.21.15.59:0/2161749200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c14f200 con 0x7fe17c1586e0 2024-03-06T07:23:31.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.167+0000 7fe181808640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c1586e0 0x7fe17c156650 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009e30 tx=0x7fe16c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.168+0000 7fe179ffb640 1 -- 172.21.15.59:0/2161749200 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c018020 con 0x7fe17c1586e0 2024-03-06T07:23:31.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.168+0000 7fe179ffb640 1 -- 172.21.15.59:0/2161749200 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16c014480 con 0x7fe17c1586e0 2024-03-06T07:23:31.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.168+0000 7fe18326b640 1 -- 172.21.15.59:0/2161749200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17c1571c0 con 0x7fe17c1586e0 2024-03-06T07:23:31.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.168+0000 7fe179ffb640 1 -- 172.21.15.59:0/2161749200 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c014870 con 0x7fe17c1586e0 2024-03-06T07:23:31.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.168+0000 7fe18326b640 1 -- 172.21.15.59:0/2161749200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17c14f4a0 con 0x7fe17c1586e0 2024-03-06T07:23:31.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.169+0000 7fe18326b640 1 -- 172.21.15.59:0/2161749200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17c0732b0 con 0x7fe17c1586e0 2024-03-06T07:23:31.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.169+0000 7fe179ffb640 1 -- 172.21.15.59:0/2161749200 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe16c005d10 con 0x7fe17c1586e0 2024-03-06T07:23:31.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.170+0000 7fe179ffb640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe16806b4d0 0x7fe16806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.170+0000 7fe179ffb640 1 -- 172.21.15.59:0/2161749200 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe16c08af70 con 0x7fe17c1586e0 2024-03-06T07:23:31.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.170+0000 7fe181007640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe16806b4d0 0x7fe16806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.171+0000 7fe181007640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe16806b4d0 0x7fe16806d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe174036000 tx=0x7fe17403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.174+0000 7fe179ffb640 1 -- 172.21.15.59:0/2161749200 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe16c022030 con 0x7fe17c1586e0 2024-03-06T07:23:31.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:31.396+0000 7fe18326b640 1 -- 172.21.15.59:0/2161749200 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe17c0b3d60 con 0x7fe17c1586e0 2024-03-06T07:23:32.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.053+0000 7fe179ffb640 1 -- 172.21.15.59:0/2161749200 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe16c010040 con 0x7fe17c1586e0 2024-03-06T07:23:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 -- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe16806b4d0 msgr2=0x7fe16806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe16806b4d0 0x7fe16806d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe174036000 tx=0x7fe17403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 -- 172.21.15.59:0/2161749200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c1586e0 msgr2=0x7fe17c156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c1586e0 0x7fe17c156650 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009e30 tx=0x7fe16c00c490 comp rx=0 tx=0).stop 2024-03-06T07:23:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 -- 172.21.15.59:0/2161749200 shutdown_connections 2024-03-06T07:23:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe16806b4d0 0x7fe16806d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c1586e0 0x7fe17c156650 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0d01a0 0x7fe17c14fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 --2- 172.21.15.59:0/2161749200 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe17c0aef80 0x7fe17c1451d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 -- 172.21.15.59:0/2161749200 >> 172.21.15.59:0/2161749200 conn(0x7fe17c06b190 msgr2=0x7fe17c072380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 -- 172.21.15.59:0/2161749200 shutdown_connections 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.056+0000 7fe18326b640 1 -- 172.21.15.59:0/2161749200 wait complete. 2024-03-06T07:23:32.070 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:23:32.070 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:32.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.173+0000 7f1fd3eaf640 1 Processor -- start 2024-03-06T07:23:32.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.174+0000 7f1fd3eaf640 1 -- start start 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.174+0000 7f1fd3eaf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1fcc0b4170 0x7f1fcc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.174+0000 7f1fd3eaf640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1fcc0d4760 0x7f1fcc0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.174+0000 7f1fd3eaf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc0d25d0 con 0x7f1fcc0b4dd0 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.174+0000 7f1fd3eaf640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc0d4bf0 con 0x7f1fcc0d4760 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.174+0000 7f1fd3eaf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc0ce9e0 con 0x7f1fcc0b4170 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.175+0000 7f1fd144a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1fcc0d4760 0x7f1fcc0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.175+0000 7f1fd144a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1fcc0d4760 0x7f1fcc0d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41004/0 (socket says 172.21.15.59:41004) 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.175+0000 7f1fd144a640 1 -- 172.21.15.59:0/1412309512 learned_addr learned my addr 172.21.15.59:0/1412309512 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.175+0000 7f1fd1c4b640 1 --2- 172.21.15.59:0/1412309512 >> v2:172.21.15.164:3300/0 conn(0x7f1fcc0b4170 0x7f1fcc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.175+0000 7f1fd144a640 1 -- 172.21.15.59:0/1412309512 >> v2:172.21.15.164:3300/0 conn(0x7f1fcc0b4170 msgr2=0x7f1fcc0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.175+0000 7f1fd144a640 1 --2- 172.21.15.59:0/1412309512 >> v2:172.21.15.164:3300/0 conn(0x7f1fcc0b4170 0x7f1fcc0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.175+0000 7f1fd144a640 1 -- 172.21.15.59:0/1412309512 >> v1:172.21.15.164:6789/0 conn(0x7f1fcc0b4dd0 legacy=0x7f1fcc0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.175+0000 7f1fd144a640 1 -- 172.21.15.59:0/1412309512 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc0d2870 con 0x7f1fcc0d4760 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd144a640 1 --2- 172.21.15.59:0/1412309512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc0d4760 0x7f1fcc0d0370 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc06e030 tx=0x7f1fbc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72086bfd45b3acfe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd0c49640 1 -- 172.21.15.59:0/1412309512 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc034070 con 0x7f1fcc0d4760 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd0c49640 1 -- 172.21.15.59:0/1412309512 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fbc030860 con 0x7f1fcc0d4760 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd0c49640 1 -- 172.21.15.59:0/1412309512 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc030c30 con 0x7f1fcc0d4760 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/1412309512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc0d4760 msgr2=0x7f1fcc0d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd3eaf640 1 --2- 172.21.15.59:0/1412309512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc0d4760 0x7f1fcc0d0370 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc06e030 tx=0x7f1fbc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/1412309512 shutdown_connections 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd3eaf640 1 --2- 172.21.15.59:0/1412309512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc0d4760 0x7f1fcc0d0370 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd3eaf640 1 --2- 172.21.15.59:0/1412309512 >> v2:172.21.15.164:3300/0 conn(0x7f1fcc0b4170 0x7f1fcc0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/1412309512 >> 172.21.15.59:0/1412309512 conn(0x7f1fcc0d3c40 msgr2=0x7f1fcc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/1412309512 shutdown_connections 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.176+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/1412309512 wait complete. 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd3eaf640 1 Processor -- start 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd3eaf640 1 -- start start 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd3eaf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fcc0b4170 0x7f1fcc0cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd3eaf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fcc0b4dd0 0x7f1fcc0c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd3eaf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc0cad10 0x7f1fcc1500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd3eaf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0b47d0 con 0x7f1fcc0cad10 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd3eaf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0b4550 con 0x7f1fcc0b4dd0 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd3eaf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0b4950 con 0x7f1fcc0b4170 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd244c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc0cad10 0x7f1fcc1500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd1c4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fcc0b4170 0x7f1fcc0cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd144a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fcc0b4dd0 0x7f1fcc0c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd244c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc0cad10 0x7f1fcc1500a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41020/0 (socket says 172.21.15.59:41020) 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.178+0000 7f1fd244c640 1 -- 172.21.15.59:0/3408172909 learned_addr learned my addr 172.21.15.59:0/3408172909 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fd244c640 1 -- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fcc0b4170 msgr2=0x7f1fcc0cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fd244c640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fcc0b4170 0x7f1fcc0cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fd244c640 1 -- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fcc0b4dd0 msgr2=0x7f1fcc0c8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fd244c640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fcc0b4dd0 0x7f1fcc0c8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fd244c640 1 -- 172.21.15.59:0/3408172909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc1505e0 con 0x7f1fcc0cad10 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fd1c4b640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fcc0b4170 0x7f1fcc0cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fd144a640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fcc0b4dd0 0x7f1fcc0c8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fd244c640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc0cad10 0x7f1fcc1500a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400c7a0 tx=0x7f1fc400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fc27fc640 1 -- 172.21.15.59:0/3408172909 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4003b70 con 0x7f1fcc0cad10 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fc27fc640 1 -- 172.21.15.59:0/3408172909 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc4003d30 con 0x7f1fcc0cad10 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fc27fc640 1 -- 172.21.15.59:0/3408172909 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc400aa90 con 0x7f1fcc0cad10 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/3408172909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc0cb1c0 con 0x7f1fcc0cad10 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.179+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/3408172909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc0cb760 con 0x7f1fcc0cad10 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.180+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/3408172909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fcc04f330 con 0x7f1fcc0cad10 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.181+0000 7f1fc27fc640 1 -- 172.21.15.59:0/3408172909 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fc400ac30 con 0x7f1fcc0cad10 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.181+0000 7f1fc27fc640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fb006b4d0 0x7f1fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.181+0000 7f1fc27fc640 1 -- 172.21.15.59:0/3408172909 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fc4089f70 con 0x7f1fcc0cad10 2024-03-06T07:23:32.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.181+0000 7f1fd1c4b640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fb006b4d0 0x7f1fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.182+0000 7f1fd1c4b640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fb006b4d0 0x7f1fb006d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009f30 tx=0x7f1fc8009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.185+0000 7f1fc27fc640 1 -- 172.21.15.59:0/3408172909 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fc4058830 con 0x7f1fcc0cad10 2024-03-06T07:23:32.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.406+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/3408172909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fcc0733b0 con 0x7f1fcc0cad10 2024-03-06T07:23:32.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.406+0000 7f1fc27fc640 1 -- 172.21.15.59:0/3408172909 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f1fc405c8d0 con 0x7f1fcc0cad10 2024-03-06T07:23:32.408 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:32.409 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5557,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/690078434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":690078434},{"type":"v1","addr":"172.21.15.164:6836","nonce":690078434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1345698247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1345698247},{"type":"v1","addr":"172.21.15.59:6833","nonce":1345698247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5685,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3875776524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3875776524},{"type":"v1","addr":"172.21.15.59:6835","nonce":3875776524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:28.400413+0000","modified":"2024-03-06T07:23:31.397689+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/3749315375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3749315375},{"type":"v1","addr":"172.21.15.164:6837","nonce":3749315375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-06T07:23:32.409 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 20 2024-03-06T07:23:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fb006b4d0 msgr2=0x7f1fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fb006b4d0 0x7f1fb006d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009f30 tx=0x7f1fc8009190 comp rx=0 tx=0).stop 2024-03-06T07:23:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/3408172909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc0cad10 msgr2=0x7f1fcc1500a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc0cad10 0x7f1fcc1500a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400c7a0 tx=0x7f1fc400cc60 comp rx=0 tx=0).stop 2024-03-06T07:23:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/3408172909 shutdown_connections 2024-03-06T07:23:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fb006b4d0 0x7f1fb006d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fcc0cad10 0x7f1fcc1500a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fcc0b4dd0 0x7f1fcc0c8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 --2- 172.21.15.59:0/3408172909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fcc0b4170 0x7f1fcc0cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/3408172909 >> 172.21.15.59:0/3408172909 conn(0x7f1fcc0d3c40 msgr2=0x7f1fcc06cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/3408172909 shutdown_connections 2024-03-06T07:23:32.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.409+0000 7f1fd3eaf640 1 -- 172.21.15.59:0/3408172909 wait complete. 2024-03-06T07:23:32.422 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:32.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.521+0000 7f0f27e2a640 1 Processor -- start 2024-03-06T07:23:32.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.521+0000 7f0f27e2a640 1 -- start start 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.522+0000 7f0f27e2a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2015dfe0 0x7f0f200cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.522+0000 7f0f27e2a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0f200cf830 0x7f0f200d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.522+0000 7f0f27e2a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f200b3430 con 0x7f0f200d27f0 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.522+0000 7f0f27e2a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f200d2c50 con 0x7f0f200cf830 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.522+0000 7f0f27e2a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f200b31e0 con 0x7f0f2015dfe0 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.522+0000 7f0f263c7640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0f200d27f0 0x7f0f200d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42926/0 (socket says 172.21.15.59:42926) 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.522+0000 7f0f263c7640 1 -- 172.21.15.59:0/3620722523 learned_addr learned my addr 172.21.15.59:0/3620722523 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.522+0000 7f0f25bc6640 1 --2- 172.21.15.59:0/3620722523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2015dfe0 0x7f0f200cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.522+0000 7f0f253c5640 1 --2- 172.21.15.59:0/3620722523 >> v2:172.21.15.164:3300/0 conn(0x7f0f200cf830 0x7f0f200d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.523+0000 7f0f24bc4640 1 -- 172.21.15.59:0/3620722523 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230800175 0 0) 0x7f0f200b3430 con 0x7f0f200d27f0 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.523+0000 7f0f24bc4640 1 -- 172.21.15.59:0/3620722523 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f08003520 con 0x7f0f200d27f0 2024-03-06T07:23:32.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.523+0000 7f0f25bc6640 1 -- 172.21.15.59:0/3620722523 >> v2:172.21.15.164:3300/0 conn(0x7f0f200cf830 msgr2=0x7f0f200d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.523+0000 7f0f25bc6640 1 --2- 172.21.15.59:0/3620722523 >> v2:172.21.15.164:3300/0 conn(0x7f0f200cf830 0x7f0f200d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.523+0000 7f0f25bc6640 1 -- 172.21.15.59:0/3620722523 >> v1:172.21.15.59:6789/0 conn(0x7f0f200d27f0 legacy=0x7f0f200d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.523+0000 7f0f25bc6640 1 -- 172.21.15.59:0/3620722523 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f200b3710 con 0x7f0f2015dfe0 2024-03-06T07:23:32.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f25bc6640 1 --2- 172.21.15.59:0/3620722523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2015dfe0 0x7f0f200cf2f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0f180097a0 tx=0x7f0f1802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cfb786d0f5344b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f24bc4640 1 -- 172.21.15.59:0/3620722523 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18035020 con 0x7f0f2015dfe0 2024-03-06T07:23:32.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f24bc4640 1 -- 172.21.15.59:0/3620722523 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f18031830 con 0x7f0f2015dfe0 2024-03-06T07:23:32.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f24bc4640 1 -- 172.21.15.59:0/3620722523 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18031c00 con 0x7f0f2015dfe0 2024-03-06T07:23:32.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f27e2a640 1 -- 172.21.15.59:0/3620722523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2015dfe0 msgr2=0x7f0f200cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f27e2a640 1 --2- 172.21.15.59:0/3620722523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2015dfe0 0x7f0f200cf2f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0f180097a0 tx=0x7f0f1802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f27e2a640 1 -- 172.21.15.59:0/3620722523 shutdown_connections 2024-03-06T07:23:32.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f27e2a640 1 --2- 172.21.15.59:0/3620722523 >> v2:172.21.15.164:3300/0 conn(0x7f0f200cf830 0x7f0f200d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f27e2a640 1 --2- 172.21.15.59:0/3620722523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2015dfe0 0x7f0f200cf2f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f27e2a640 1 -- 172.21.15.59:0/3620722523 >> 172.21.15.59:0/3620722523 conn(0x7f0f200d3cd0 msgr2=0x7f0f200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f27e2a640 1 -- 172.21.15.59:0/3620722523 shutdown_connections 2024-03-06T07:23:32.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.524+0000 7f0f27e2a640 1 -- 172.21.15.59:0/3620722523 wait complete. 2024-03-06T07:23:32.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.525+0000 7f0f27e2a640 1 Processor -- start 2024-03-06T07:23:32.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.525+0000 7f0f27e2a640 1 -- start start 2024-03-06T07:23:32.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.526+0000 7f0f27e2a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f200cf830 0x7f0f201532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.526+0000 7f0f27e2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f200d27f0 0x7f0f200cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.526+0000 7f0f27e2a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2015dfe0 0x7f0f200ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.526+0000 7f0f27e2a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f200d1c70 con 0x7f0f200d27f0 2024-03-06T07:23:32.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.526+0000 7f0f27e2a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f200b3d60 con 0x7f0f200cf830 2024-03-06T07:23:32.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.526+0000 7f0f27e2a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f200d1df0 con 0x7f0f2015dfe0 2024-03-06T07:23:32.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.526+0000 7f0f253c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f200d27f0 0x7f0f200cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.526+0000 7f0f253c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f200d27f0 0x7f0f200cbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41024/0 (socket says 172.21.15.59:41024) 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.526+0000 7f0f253c5640 1 -- 172.21.15.59:0/651573681 learned_addr learned my addr 172.21.15.59:0/651573681 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.526+0000 7f0f25bc6640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f200cf830 0x7f0f201532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.526+0000 7f0f263c7640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2015dfe0 0x7f0f200ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f253c5640 1 -- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2015dfe0 msgr2=0x7f0f200ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f253c5640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2015dfe0 0x7f0f200ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f253c5640 1 -- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f200cf830 msgr2=0x7f0f201532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f253c5640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f200cf830 0x7f0f201532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f253c5640 1 -- 172.21.15.59:0/651573681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f200b9630 con 0x7f0f200d27f0 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f25bc6640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f200cf830 0x7f0f201532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f253c5640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f200d27f0 0x7f0f200cbc80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400e9b0 tx=0x7f0f1400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f1e7fc640 1 -- 172.21.15.59:0/651573681 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14017070 con 0x7f0f200d27f0 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f1e7fc640 1 -- 172.21.15.59:0/651573681 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1400c9a0 con 0x7f0f200d27f0 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f27e2a640 1 -- 172.21.15.59:0/651573681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f200b9910 con 0x7f0f200d27f0 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f1e7fc640 1 -- 172.21.15.59:0/651573681 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1400cd70 con 0x7f0f200d27f0 2024-03-06T07:23:32.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.527+0000 7f0f27e2a640 1 -- 172.21.15.59:0/651573681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2015c0b0 con 0x7f0f200d27f0 2024-03-06T07:23:32.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.528+0000 7f0f27e2a640 1 -- 172.21.15.59:0/651573681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f200ce3f0 con 0x7f0f200d27f0 2024-03-06T07:23:32.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.529+0000 7f0f1e7fc640 1 -- 172.21.15.59:0/651573681 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f1401a070 con 0x7f0f200d27f0 2024-03-06T07:23:32.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.529+0000 7f0f1e7fc640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0f0c06b500 0x7f0f0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.529+0000 7f0f1e7fc640 1 -- 172.21.15.59:0/651573681 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f1408a3c0 con 0x7f0f200d27f0 2024-03-06T07:23:32.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.529+0000 7f0f25bc6640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0f0c06b500 0x7f0f0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.529+0000 7f0f25bc6640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0f0c06b500 0x7f0f0c06d9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0f18036000 tx=0x7f0f1803c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.533+0000 7f0f1e7fc640 1 -- 172.21.15.59:0/651573681 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f200ce3f0 con 0x7f0f200d27f0 2024-03-06T07:23:32.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.718+0000 7f0f27e2a640 1 -- 172.21.15.59:0/651573681 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f2004f330 con 0x7f0f200d27f0 2024-03-06T07:23:32.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.719+0000 7f0f1e7fc640 1 -- 172.21.15.59:0/651573681 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16601 (secure 0 0 0) 0x7f0f14058c80 con 0x7f0f200d27f0 2024-03-06T07:23:32.721 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:32.721 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":45,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:23:30.044730+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:26.672668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:27.362891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:32.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 -- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0f0c06b500 msgr2=0x7f0f0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0f0c06b500 0x7f0f0c06d9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0f18036000 tx=0x7f0f1803c040 comp rx=0 tx=0).stop 2024-03-06T07:23:32.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 -- 172.21.15.59:0/651573681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f200d27f0 msgr2=0x7f0f200cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f200d27f0 0x7f0f200cbc80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400e9b0 tx=0x7f0f1400ee70 comp rx=0 tx=0).stop 2024-03-06T07:23:32.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 -- 172.21.15.59:0/651573681 shutdown_connections 2024-03-06T07:23:32.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0f0c06b500 0x7f0f0c06d9c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2015dfe0 0x7f0f200ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f200d27f0 0x7f0f200cbc80 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 --2- 172.21.15.59:0/651573681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f200cf830 0x7f0f201532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 -- 172.21.15.59:0/651573681 >> 172.21.15.59:0/651573681 conn(0x7f0f200d3cd0 msgr2=0x7f0f200b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 -- 172.21.15.59:0/651573681 shutdown_connections 2024-03-06T07:23:32.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.721+0000 7f0f27e2a640 1 -- 172.21.15.59:0/651573681 wait complete. 2024-03-06T07:23:32.734 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:32.734 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:32.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.833+0000 7f0a9880e640 1 Processor -- start 2024-03-06T07:23:32.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.833+0000 7f0a9880e640 1 -- start start 2024-03-06T07:23:32.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.834+0000 7f0a9880e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0a9413dc10 0x7f0a940b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.834+0000 7f0a9880e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9415ef60 0x7f0a940b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.834+0000 7f0a9880e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9415f580 con 0x7f0a9415bf30 2024-03-06T07:23:32.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.834+0000 7f0a9880e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a940cf3f0 con 0x7f0a9413dc10 2024-03-06T07:23:32.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.834+0000 7f0a9880e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9415b210 con 0x7f0a9415ef60 2024-03-06T07:23:32.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.834+0000 7f0a91d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0a9415bf30 0x7f0a9415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42934/0 (socket says 172.21.15.59:42934) 2024-03-06T07:23:32.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.834+0000 7f0a91d9b640 1 -- 172.21.15.59:0/619683244 learned_addr learned my addr 172.21.15.59:0/619683244 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.834+0000 7f0a92d9d640 1 --2- 172.21.15.59:0/619683244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9415ef60 0x7f0a940b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.834+0000 7f0a9259c640 1 --2- 172.21.15.59:0/619683244 >> v2:172.21.15.164:3300/0 conn(0x7f0a9413dc10 0x7f0a940b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.835+0000 7f0a9159a640 1 -- 172.21.15.59:0/619683244 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417888570 0 0) 0x7f0a9415f580 con 0x7f0a9415bf30 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.835+0000 7f0a9159a640 1 -- 172.21.15.59:0/619683244 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a7c003520 con 0x7f0a9415bf30 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.835+0000 7f0a92d9d640 1 -- 172.21.15.59:0/619683244 >> v2:172.21.15.164:3300/0 conn(0x7f0a9413dc10 msgr2=0x7f0a940b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.835+0000 7f0a92d9d640 1 --2- 172.21.15.59:0/619683244 >> v2:172.21.15.164:3300/0 conn(0x7f0a9413dc10 0x7f0a940b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.835+0000 7f0a92d9d640 1 -- 172.21.15.59:0/619683244 >> v1:172.21.15.59:6789/0 conn(0x7f0a9415bf30 legacy=0x7f0a9415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.835+0000 7f0a92d9d640 1 -- 172.21.15.59:0/619683244 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9415b450 con 0x7f0a9415ef60 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.835+0000 7f0a9259c640 1 --2- 172.21.15.59:0/619683244 >> v2:172.21.15.164:3300/0 conn(0x7f0a9413dc10 0x7f0a940b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.835+0000 7f0a92d9d640 1 --2- 172.21.15.59:0/619683244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9415ef60 0x7f0a940b2e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a840097a0 tx=0x7f0a8402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2827702949cc1356 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.836+0000 7f0a9159a640 1 -- 172.21.15.59:0/619683244 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84031780 con 0x7f0a9415ef60 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.836+0000 7f0a9159a640 1 -- 172.21.15.59:0/619683244 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a84037070 con 0x7f0a9415ef60 2024-03-06T07:23:32.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.836+0000 7f0a9159a640 1 -- 172.21.15.59:0/619683244 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84031b40 con 0x7f0a9415ef60 2024-03-06T07:23:32.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.836+0000 7f0a9880e640 1 -- 172.21.15.59:0/619683244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9415ef60 msgr2=0x7f0a940b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.836+0000 7f0a9880e640 1 --2- 172.21.15.59:0/619683244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9415ef60 0x7f0a940b2e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a840097a0 tx=0x7f0a8402b820 comp rx=0 tx=0).stop 2024-03-06T07:23:32.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.836+0000 7f0a9880e640 1 -- 172.21.15.59:0/619683244 shutdown_connections 2024-03-06T07:23:32.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.836+0000 7f0a9880e640 1 --2- 172.21.15.59:0/619683244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9415ef60 0x7f0a940b2e90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.836+0000 7f0a9880e640 1 --2- 172.21.15.59:0/619683244 >> v2:172.21.15.164:3300/0 conn(0x7f0a9413dc10 0x7f0a940b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.836+0000 7f0a9880e640 1 -- 172.21.15.59:0/619683244 >> 172.21.15.59:0/619683244 conn(0x7f0a940cf8a0 msgr2=0x7f0a940cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.836+0000 7f0a9880e640 1 -- 172.21.15.59:0/619683244 shutdown_connections 2024-03-06T07:23:32.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.836+0000 7f0a9880e640 1 -- 172.21.15.59:0/619683244 wait complete. 2024-03-06T07:23:32.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.837+0000 7f0a9880e640 1 Processor -- start 2024-03-06T07:23:32.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.837+0000 7f0a9880e640 1 -- start start 2024-03-06T07:23:32.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.838+0000 7f0a9880e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9413dc10 0x7f0a941434d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.838+0000 7f0a9880e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9415bf30 0x7f0a9413e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.838+0000 7f0a9880e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a9415ef60 0x7f0a9413e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.838+0000 7f0a9880e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a940708d0 con 0x7f0a9415ef60 2024-03-06T07:23:32.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.838+0000 7f0a9880e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9413dff0 con 0x7f0a9415bf30 2024-03-06T07:23:32.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.838+0000 7f0a9880e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a94070a50 con 0x7f0a9413dc10 2024-03-06T07:23:32.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.838+0000 7f0a92d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a9415ef60 0x7f0a9413e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.838+0000 7f0a92d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a9415ef60 0x7f0a9413e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41040/0 (socket says 172.21.15.59:41040) 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.838+0000 7f0a92d9d640 1 -- 172.21.15.59:0/3046450812 learned_addr learned my addr 172.21.15.59:0/3046450812 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.838+0000 7f0a9259c640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9413dc10 0x7f0a941434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.838+0000 7f0a91d9b640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9415bf30 0x7f0a9413e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.839+0000 7f0a92d9d640 1 -- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9413dc10 msgr2=0x7f0a941434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.839+0000 7f0a92d9d640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9413dc10 0x7f0a941434d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.839+0000 7f0a92d9d640 1 -- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9415bf30 msgr2=0x7f0a9413e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.839+0000 7f0a92d9d640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9415bf30 0x7f0a9413e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.839+0000 7f0a92d9d640 1 -- 172.21.15.59:0/3046450812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a94140590 con 0x7f0a9415ef60 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.839+0000 7f0a92d9d640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a9415ef60 0x7f0a9413e6b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0a840097a0 tx=0x7f0a840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.839+0000 7f0a82ffd640 1 -- 172.21.15.59:0/3046450812 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a84032410 con 0x7f0a9415ef60 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.839+0000 7f0a82ffd640 1 -- 172.21.15.59:0/3046450812 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a840325d0 con 0x7f0a9415ef60 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.839+0000 7f0a9880e640 1 -- 172.21.15.59:0/3046450812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a94140870 con 0x7f0a9415ef60 2024-03-06T07:23:32.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.839+0000 7f0a82ffd640 1 -- 172.21.15.59:0/3046450812 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a840329c0 con 0x7f0a9415ef60 2024-03-06T07:23:32.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.839+0000 7f0a9880e640 1 -- 172.21.15.59:0/3046450812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a94140e30 con 0x7f0a9415ef60 2024-03-06T07:23:32.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.840+0000 7f0a9880e640 1 -- 172.21.15.59:0/3046450812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a94141510 con 0x7f0a9415ef60 2024-03-06T07:23:32.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.840+0000 7f0a82ffd640 1 -- 172.21.15.59:0/3046450812 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a8404c060 con 0x7f0a9415ef60 2024-03-06T07:23:32.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.841+0000 7f0a82ffd640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a7406b500 0x7f0a7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.841+0000 7f0a82ffd640 1 -- 172.21.15.59:0/3046450812 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a840ab960 con 0x7f0a9415ef60 2024-03-06T07:23:32.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.841+0000 7f0a9259c640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a7406b500 0x7f0a7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.841+0000 7f0a9259c640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a7406b500 0x7f0a7406d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009bd0 tx=0x7f0a8c00b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:32.845+0000 7f0a82ffd640 1 -- 172.21.15.59:0/3046450812 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a94141510 con 0x7f0a9415ef60 2024-03-06T07:23:33.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.020+0000 7f0a9880e640 1 -- 172.21.15.59:0/3046450812 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a9404f330 con 0x7f0a7406b500 2024-03-06T07:23:33.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.026+0000 7f0a82ffd640 1 -- 172.21.15.59:0/3046450812 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237161 (secure 0 0 0) 0x7f0a9404f330 con 0x7f0a7406b500 2024-03-06T07:23:33.030 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:33.032 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:23:33.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.028+0000 7f0a9880e640 1 -- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a7406b500 msgr2=0x7f0a7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.028+0000 7f0a9880e640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a7406b500 0x7f0a7406d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009bd0 tx=0x7f0a8c00b040 comp rx=0 tx=0).stop 2024-03-06T07:23:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.028+0000 7f0a9880e640 1 -- 172.21.15.59:0/3046450812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a9415ef60 msgr2=0x7f0a9413e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.028+0000 7f0a9880e640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a9415ef60 0x7f0a9413e6b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0a840097a0 tx=0x7f0a840046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.028+0000 7f0a9880e640 1 -- 172.21.15.59:0/3046450812 shutdown_connections 2024-03-06T07:23:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.028+0000 7f0a9880e640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0a7406b500 0x7f0a7406d9c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.028+0000 7f0a9880e640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a9415ef60 0x7f0a9413e6b0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.028+0000 7f0a9880e640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9415bf30 0x7f0a9413e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.028+0000 7f0a9880e640 1 --2- 172.21.15.59:0/3046450812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9413dc10 0x7f0a941434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.028+0000 7f0a9880e640 1 -- 172.21.15.59:0/3046450812 >> 172.21.15.59:0/3046450812 conn(0x7f0a940cf8a0 msgr2=0x7f0a940b0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.028+0000 7f0a9880e640 1 -- 172.21.15.59:0/3046450812 shutdown_connections 2024-03-06T07:23:33.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.029+0000 7f0a9880e640 1 -- 172.21.15.59:0/3046450812 wait complete. 2024-03-06T07:23:33.041 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-03-06T07:23:31.321859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":250160,"kb_used_data":4868,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754724560,"statfs":{"total":773094113280,"available":772837949440,"internally_reserved":0,"allocated":4984832,"data_stored":2685003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.295346"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:28.025632+0000","last_change":"2024-03-06T07:23:28.025632+0000","last_active":"2024-03-06T07:23:28.025632+0000","last_peered":"2024-03-06T07:23:28.025632+0000","last_clean":"2024-03-06T07:23:28.025632+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:28.025632+0000","last_undegraded":"2024-03-06T07:23:28.025632+0000","last_fullsized":"2024-03-06T07:23:28.025632+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:27.021935+0000","last_change":"2024-03-06T07:23:27.021935+0000","last_active":"2024-03-06T07:23:27.021935+0000","last_peered":"2024-03-06T07:23:27.021935+0000","last_clean":"2024-03-06T07:23:27.021935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:27.021935+0000","last_undegraded":"2024-03-06T07:23:27.021935+0000","last_fullsized":"2024-03-06T07:23:27.021935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":207,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-06T07:23:26.209607+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:23:26.209607+0000","last_peered":"2024-03-06T07:23:26.209607+0000","last_clean":"2024-03-06T07:23:26.209607+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:23:26.209607+0000","last_undegraded":"2024-03-06T07:23:26.209607+0000","last_fullsized":"2024-03-06T07:23:26.209607+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":775197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31636,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":466944,"data_stored":184829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":189956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":189956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31660,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":479232,"data_stored":189956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27488,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":479232,"data_stored":189956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":189956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":775197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:33.045 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:33.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.181+0000 7f7ba05f7640 1 Processor -- start 2024-03-06T07:23:33.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.181+0000 7f7ba05f7640 1 -- start start 2024-03-06T07:23:33.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.182+0000 7f7ba05f7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7b980d02c0 0x7f7b980cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.182+0000 7f7ba05f7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b980cef50 con 0x7f7b980b6550 2024-03-06T07:23:33.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.182+0000 7f7ba05f7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9815c620 con 0x7f7b980d06a0 2024-03-06T07:23:33.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.182+0000 7f7ba05f7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b980b50b0 con 0x7f7b980d02c0 2024-03-06T07:23:33.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.182+0000 7f7b9e393640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7b980d02c0 0x7f7b980cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.182+0000 7f7b9eb94640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7b980b6550 0x7f7b980ce840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42940/0 (socket says 172.21.15.59:42940) 2024-03-06T07:23:33.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.182+0000 7f7b9eb94640 1 -- 172.21.15.59:0/2546832829 learned_addr learned my addr 172.21.15.59:0/2546832829 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.182+0000 7f7b9e393640 1 -- 172.21.15.59:0/2546832829 >> v1:172.21.15.164:6789/0 conn(0x7f7b980d06a0 legacy=0x7f7b980ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.182+0000 7f7b9e393640 1 -- 172.21.15.59:0/2546832829 >> v1:172.21.15.59:6789/0 conn(0x7f7b980b6550 legacy=0x7f7b980ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.182+0000 7f7b9e393640 1 -- 172.21.15.59:0/2546832829 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980cd800 con 0x7f7b980d02c0 2024-03-06T07:23:33.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.182+0000 7f7b9d391640 1 -- 172.21.15.59:0/2546832829 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352619285 0 0) 0x7f7b980cef50 con 0x7f7b980b6550 2024-03-06T07:23:33.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.183+0000 7f7b9e393640 1 --2- 172.21.15.59:0/2546832829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b980d02c0 0x7f7b980cf850 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7b940097a0 tx=0x7f7b9402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1252cd512b0020dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.183+0000 7f7b9d391640 1 -- 172.21.15.59:0/2546832829 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b94034070 con 0x7f7b980d02c0 2024-03-06T07:23:33.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.183+0000 7f7b9d391640 1 -- 172.21.15.59:0/2546832829 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b94030610 con 0x7f7b980d02c0 2024-03-06T07:23:33.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.183+0000 7f7b9d391640 1 -- 172.21.15.59:0/2546832829 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b940309e0 con 0x7f7b980d02c0 2024-03-06T07:23:33.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.184+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2546832829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b980d02c0 msgr2=0x7f7b980cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.184+0000 7f7ba05f7640 1 --2- 172.21.15.59:0/2546832829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b980d02c0 0x7f7b980cf850 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7b940097a0 tx=0x7f7b9402b960 comp rx=0 tx=0).stop 2024-03-06T07:23:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.184+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2546832829 shutdown_connections 2024-03-06T07:23:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.184+0000 7f7ba05f7640 1 --2- 172.21.15.59:0/2546832829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b980d02c0 0x7f7b980cf850 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.184+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2546832829 >> 172.21.15.59:0/2546832829 conn(0x7f7b980b35a0 msgr2=0x7f7b980b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.184+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2546832829 shutdown_connections 2024-03-06T07:23:33.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.184+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2546832829 wait complete. 2024-03-06T07:23:33.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.185+0000 7f7ba05f7640 1 Processor -- start 2024-03-06T07:23:33.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.185+0000 7f7ba05f7640 1 -- start start 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7ba05f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b980b6550 0x7f7b980c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7ba05f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b980d02c0 0x7f7b981e5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7ba05f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b980d06a0 0x7f7b98155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7ba05f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980cd0d0 con 0x7f7b980d06a0 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7ba05f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980b4d40 con 0x7f7b980d02c0 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7ba05f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980cd250 con 0x7f7b980b6550 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7b9eb94640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b980d06a0 0x7f7b98155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7b9eb94640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b980d06a0 0x7f7b98155b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41062/0 (socket says 172.21.15.59:41062) 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7b9eb94640 1 -- 172.21.15.59:0/2146094513 learned_addr learned my addr 172.21.15.59:0/2146094513 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7b9e393640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b980b6550 0x7f7b980c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7b9db92640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b980d02c0 0x7f7b981e5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7b9eb94640 1 -- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b980b6550 msgr2=0x7f7b980c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7b9eb94640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b980b6550 0x7f7b980c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7b9eb94640 1 -- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b980d02c0 msgr2=0x7f7b981e5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7b9eb94640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b980d02c0 0x7f7b981e5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7b9eb94640 1 -- 172.21.15.59:0/2146094513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b981e6100 con 0x7f7b980d06a0 2024-03-06T07:23:33.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.186+0000 7f7b9eb94640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b980d06a0 0x7f7b98155b00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000c7a0 tx=0x7f7b9000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.187+0000 7f7b8effd640 1 -- 172.21.15.59:0/2146094513 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b90018070 con 0x7f7b980d06a0 2024-03-06T07:23:33.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.187+0000 7f7b8effd640 1 -- 172.21.15.59:0/2146094513 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9000a640 con 0x7f7b980d06a0 2024-03-06T07:23:33.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.187+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2146094513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9806c240 con 0x7f7b980d06a0 2024-03-06T07:23:33.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.187+0000 7f7b8effd640 1 -- 172.21.15.59:0/2146094513 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9000aab0 con 0x7f7b980d06a0 2024-03-06T07:23:33.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.187+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2146094513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9806c800 con 0x7f7b980d06a0 2024-03-06T07:23:33.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.188+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2146094513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9804f330 con 0x7f7b980d06a0 2024-03-06T07:23:33.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.188+0000 7f7b8effd640 1 -- 172.21.15.59:0/2146094513 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b9001b070 con 0x7f7b980d06a0 2024-03-06T07:23:33.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.188+0000 7f7b8effd640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7b7c06b4d0 0x7f7b7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.188+0000 7f7b8effd640 1 -- 172.21.15.59:0/2146094513 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b900896f0 con 0x7f7b980d06a0 2024-03-06T07:23:33.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.189+0000 7f7b9e393640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7b7c06b4d0 0x7f7b7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.189+0000 7f7b9e393640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7b7c06b4d0 0x7f7b7c06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7b94036000 tx=0x7f7b9403b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.193+0000 7f7b8effd640 1 -- 172.21.15.59:0/2146094513 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b90057fb0 con 0x7f7b980d06a0 2024-03-06T07:23:33.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.371+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2146094513 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b981e5ea0 con 0x7f7b7c06b4d0 2024-03-06T07:23:33.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.377+0000 7f7b8effd640 1 -- 172.21.15.59:0/2146094513 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251956 (secure 0 0 0) 0x7f7b94039820 con 0x7f7b7c06b4d0 2024-03-06T07:23:33.381 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:23:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7b7c06b4d0 msgr2=0x7f7b7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7b7c06b4d0 0x7f7b7c06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7b94036000 tx=0x7f7b9403b040 comp rx=0 tx=0).stop 2024-03-06T07:23:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2146094513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b980d06a0 msgr2=0x7f7b98155b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b980d06a0 0x7f7b98155b00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000c7a0 tx=0x7f7b9000cc60 comp rx=0 tx=0).stop 2024-03-06T07:23:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2146094513 shutdown_connections 2024-03-06T07:23:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7b7c06b4d0 0x7f7b7c06d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7b980d06a0 0x7f7b98155b00 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b980d02c0 0x7f7b981e5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 --2- 172.21.15.59:0/2146094513 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b980b6550 0x7f7b980c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2146094513 >> 172.21.15.59:0/2146094513 conn(0x7f7b980b35a0 msgr2=0x7f7b980d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2146094513 shutdown_connections 2024-03-06T07:23:33.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.380+0000 7f7ba05f7640 1 -- 172.21.15.59:0/2146094513 wait complete. 2024-03-06T07:23:33.393 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-03-06T07:23:33.322130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":110,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":250636,"kb_used_data":5316,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754724084,"statfs":{"total":773094113280,"available":772837462016,"internally_reserved":0,"allocated":5443584,"data_stored":2887576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001187"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910607+0000","last_change":"2024-03-06T07:23:29.054824+0000","last_active":"2024-03-06T07:23:30.910607+0000","last_peered":"2024-03-06T07:23:30.910607+0000","last_clean":"2024-03-06T07:23:30.910607+0000","last_became_active":"2024-03-06T07:23:29.054565+0000","last_became_peered":"2024-03-06T07:23:29.054565+0000","last_unstale":"2024-03-06T07:23:30.910607+0000","last_undegraded":"2024-03-06T07:23:30.910607+0000","last_fullsized":"2024-03-06T07:23:30.910607+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:15.495109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.052106+0000","last_change":"2024-03-06T07:23:29.047042+0000","last_active":"2024-03-06T07:23:30.052106+0000","last_peered":"2024-03-06T07:23:30.052106+0000","last_clean":"2024-03-06T07:23:30.052106+0000","last_became_active":"2024-03-06T07:23:29.046179+0000","last_became_peered":"2024-03-06T07:23:29.046179+0000","last_unstale":"2024-03-06T07:23:30.052106+0000","last_undegraded":"2024-03-06T07:23:30.052106+0000","last_fullsized":"2024-03-06T07:23:30.052106+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:12.848240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.909865+0000","last_change":"2024-03-06T07:23:29.061210+0000","last_active":"2024-03-06T07:23:30.909865+0000","last_peered":"2024-03-06T07:23:30.909865+0000","last_clean":"2024-03-06T07:23:30.909865+0000","last_became_active":"2024-03-06T07:23:29.060933+0000","last_became_peered":"2024-03-06T07:23:29.060933+0000","last_unstale":"2024-03-06T07:23:30.909865+0000","last_undegraded":"2024-03-06T07:23:30.909865+0000","last_fullsized":"2024-03-06T07:23:30.909865+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:40.523420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050626+0000","last_change":"2024-03-06T07:23:29.057135+0000","last_active":"2024-03-06T07:23:30.050626+0000","last_peered":"2024-03-06T07:23:30.050626+0000","last_clean":"2024-03-06T07:23:30.050626+0000","last_became_active":"2024-03-06T07:23:29.056777+0000","last_became_peered":"2024-03-06T07:23:29.056777+0000","last_unstale":"2024-03-06T07:23:30.050626+0000","last_undegraded":"2024-03-06T07:23:30.050626+0000","last_fullsized":"2024-03-06T07:23:30.050626+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:35.341393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.052401+0000","last_change":"2024-03-06T07:23:29.047764+0000","last_active":"2024-03-06T07:23:30.052401+0000","last_peered":"2024-03-06T07:23:30.052401+0000","last_clean":"2024-03-06T07:23:30.052401+0000","last_became_active":"2024-03-06T07:23:29.047417+0000","last_became_peered":"2024-03-06T07:23:29.047417+0000","last_unstale":"2024-03-06T07:23:30.052401+0000","last_undegraded":"2024-03-06T07:23:30.052401+0000","last_fullsized":"2024-03-06T07:23:30.052401+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:39.156916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.911065+0000","last_change":"2024-03-06T07:23:29.046399+0000","last_active":"2024-03-06T07:23:30.911065+0000","last_peered":"2024-03-06T07:23:30.911065+0000","last_clean":"2024-03-06T07:23:30.911065+0000","last_became_active":"2024-03-06T07:23:29.045904+0000","last_became_peered":"2024-03-06T07:23:29.045904+0000","last_unstale":"2024-03-06T07:23:30.911065+0000","last_undegraded":"2024-03-06T07:23:30.911065+0000","last_fullsized":"2024-03-06T07:23:30.911065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:50.354700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910176+0000","last_change":"2024-03-06T07:23:29.048790+0000","last_active":"2024-03-06T07:23:30.910176+0000","last_peered":"2024-03-06T07:23:30.910176+0000","last_clean":"2024-03-06T07:23:30.910176+0000","last_became_active":"2024-03-06T07:23:29.048506+0000","last_became_peered":"2024-03-06T07:23:29.048506+0000","last_unstale":"2024-03-06T07:23:30.910176+0000","last_undegraded":"2024-03-06T07:23:30.910176+0000","last_fullsized":"2024-03-06T07:23:30.910176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:42.666447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050138+0000","last_change":"2024-03-06T07:23:29.057404+0000","last_active":"2024-03-06T07:23:30.050138+0000","last_peered":"2024-03-06T07:23:30.050138+0000","last_clean":"2024-03-06T07:23:30.050138+0000","last_became_active":"2024-03-06T07:23:29.055718+0000","last_became_peered":"2024-03-06T07:23:29.055718+0000","last_unstale":"2024-03-06T07:23:30.050138+0000","last_undegraded":"2024-03-06T07:23:30.050138+0000","last_fullsized":"2024-03-06T07:23:30.050138+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:23.840986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050957+0000","last_change":"2024-03-06T07:23:29.050937+0000","last_active":"2024-03-06T07:23:30.050957+0000","last_peered":"2024-03-06T07:23:30.050957+0000","last_clean":"2024-03-06T07:23:30.050957+0000","last_became_active":"2024-03-06T07:23:29.050662+0000","last_became_peered":"2024-03-06T07:23:29.050662+0000","last_unstale":"2024-03-06T07:23:30.050957+0000","last_undegraded":"2024-03-06T07:23:30.050957+0000","last_fullsized":"2024-03-06T07:23:30.050957+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:24.415779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050740+0000","last_change":"2024-03-06T07:23:29.056916+0000","last_active":"2024-03-06T07:23:30.050740+0000","last_peered":"2024-03-06T07:23:30.050740+0000","last_clean":"2024-03-06T07:23:30.050740+0000","last_became_active":"2024-03-06T07:23:29.056008+0000","last_became_peered":"2024-03-06T07:23:29.056008+0000","last_unstale":"2024-03-06T07:23:30.050740+0000","last_undegraded":"2024-03-06T07:23:30.050740+0000","last_fullsized":"2024-03-06T07:23:30.050740+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:27.610870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050407+0000","last_change":"2024-03-06T07:23:29.051069+0000","last_active":"2024-03-06T07:23:30.050407+0000","last_peered":"2024-03-06T07:23:30.050407+0000","last_clean":"2024-03-06T07:23:30.050407+0000","last_became_active":"2024-03-06T07:23:29.050794+0000","last_became_peered":"2024-03-06T07:23:29.050794+0000","last_unstale":"2024-03-06T07:23:30.050407+0000","last_undegraded":"2024-03-06T07:23:30.050407+0000","last_fullsized":"2024-03-06T07:23:30.050407+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:38.978972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.052054+0000","last_change":"2024-03-06T07:23:29.063491+0000","last_active":"2024-03-06T07:23:30.052054+0000","last_peered":"2024-03-06T07:23:30.052054+0000","last_clean":"2024-03-06T07:23:30.052054+0000","last_became_active":"2024-03-06T07:23:29.063206+0000","last_became_peered":"2024-03-06T07:23:29.063206+0000","last_unstale":"2024-03-06T07:23:30.052054+0000","last_undegraded":"2024-03-06T07:23:30.052054+0000","last_fullsized":"2024-03-06T07:23:30.052054+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:24.403922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910830+0000","last_change":"2024-03-06T07:23:29.067374+0000","last_active":"2024-03-06T07:23:30.910830+0000","last_peered":"2024-03-06T07:23:30.910830+0000","last_clean":"2024-03-06T07:23:30.910830+0000","last_became_active":"2024-03-06T07:23:29.067073+0000","last_became_peered":"2024-03-06T07:23:29.067073+0000","last_unstale":"2024-03-06T07:23:30.910830+0000","last_undegraded":"2024-03-06T07:23:30.910830+0000","last_fullsized":"2024-03-06T07:23:30.910830+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:48.728579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050657+0000","last_change":"2024-03-06T07:23:29.050590+0000","last_active":"2024-03-06T07:23:30.050657+0000","last_peered":"2024-03-06T07:23:30.050657+0000","last_clean":"2024-03-06T07:23:30.050657+0000","last_became_active":"2024-03-06T07:23:29.050307+0000","last_became_peered":"2024-03-06T07:23:29.050307+0000","last_unstale":"2024-03-06T07:23:30.050657+0000","last_undegraded":"2024-03-06T07:23:30.050657+0000","last_fullsized":"2024-03-06T07:23:30.050657+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:37.381518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910824+0000","last_change":"2024-03-06T07:23:29.059177+0000","last_active":"2024-03-06T07:23:30.910824+0000","last_peered":"2024-03-06T07:23:30.910824+0000","last_clean":"2024-03-06T07:23:30.910824+0000","last_became_active":"2024-03-06T07:23:29.058829+0000","last_became_peered":"2024-03-06T07:23:29.058829+0000","last_unstale":"2024-03-06T07:23:30.910824+0000","last_undegraded":"2024-03-06T07:23:30.910824+0000","last_fullsized":"2024-03-06T07:23:30.910824+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:52.721604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053153+0000","last_change":"2024-03-06T07:23:29.050687+0000","last_active":"2024-03-06T07:23:30.053153+0000","last_peered":"2024-03-06T07:23:30.053153+0000","last_clean":"2024-03-06T07:23:30.053153+0000","last_became_active":"2024-03-06T07:23:29.050433+0000","last_became_peered":"2024-03-06T07:23:29.050433+0000","last_unstale":"2024-03-06T07:23:30.053153+0000","last_undegraded":"2024-03-06T07:23:30.053153+0000","last_fullsized":"2024-03-06T07:23:30.053153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:37.783755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910335+0000","last_change":"2024-03-06T07:23:29.054468+0000","last_active":"2024-03-06T07:23:30.910335+0000","last_peered":"2024-03-06T07:23:30.910335+0000","last_clean":"2024-03-06T07:23:30.910335+0000","last_became_active":"2024-03-06T07:23:29.054137+0000","last_became_peered":"2024-03-06T07:23:29.054137+0000","last_unstale":"2024-03-06T07:23:30.910335+0000","last_undegraded":"2024-03-06T07:23:30.910335+0000","last_fullsized":"2024-03-06T07:23:30.910335+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:32.549563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910028+0000","last_change":"2024-03-06T07:23:29.052766+0000","last_active":"2024-03-06T07:23:30.910028+0000","last_peered":"2024-03-06T07:23:30.910028+0000","last_clean":"2024-03-06T07:23:30.910028+0000","last_became_active":"2024-03-06T07:23:29.052052+0000","last_became_peered":"2024-03-06T07:23:29.052052+0000","last_unstale":"2024-03-06T07:23:30.910028+0000","last_undegraded":"2024-03-06T07:23:30.910028+0000","last_fullsized":"2024-03-06T07:23:30.910028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:32.500489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053498+0000","last_change":"2024-03-06T07:23:29.060987+0000","last_active":"2024-03-06T07:23:30.053498+0000","last_peered":"2024-03-06T07:23:30.053498+0000","last_clean":"2024-03-06T07:23:30.053498+0000","last_became_active":"2024-03-06T07:23:29.060698+0000","last_became_peered":"2024-03-06T07:23:29.060698+0000","last_unstale":"2024-03-06T07:23:30.053498+0000","last_undegraded":"2024-03-06T07:23:30.053498+0000","last_fullsized":"2024-03-06T07:23:30.053498+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:41.646368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910515+0000","last_change":"2024-03-06T07:23:29.063335+0000","last_active":"2024-03-06T07:23:30.910515+0000","last_peered":"2024-03-06T07:23:30.910515+0000","last_clean":"2024-03-06T07:23:30.910515+0000","last_became_active":"2024-03-06T07:23:29.063035+0000","last_became_peered":"2024-03-06T07:23:29.063035+0000","last_unstale":"2024-03-06T07:23:30.910515+0000","last_undegraded":"2024-03-06T07:23:30.910515+0000","last_fullsized":"2024-03-06T07:23:30.910515+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:40.379768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910546+0000","last_change":"2024-03-06T07:23:29.054595+0000","last_active":"2024-03-06T07:23:30.910546+0000","last_peered":"2024-03-06T07:23:30.910546+0000","last_clean":"2024-03-06T07:23:30.910546+0000","last_became_active":"2024-03-06T07:23:29.054332+0000","last_became_peered":"2024-03-06T07:23:29.054332+0000","last_unstale":"2024-03-06T07:23:30.910546+0000","last_undegraded":"2024-03-06T07:23:30.910546+0000","last_fullsized":"2024-03-06T07:23:30.910546+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:43.438655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050278+0000","last_change":"2024-03-06T07:23:29.055809+0000","last_active":"2024-03-06T07:23:30.050278+0000","last_peered":"2024-03-06T07:23:30.050278+0000","last_clean":"2024-03-06T07:23:30.050278+0000","last_became_active":"2024-03-06T07:23:29.055525+0000","last_became_peered":"2024-03-06T07:23:29.055525+0000","last_unstale":"2024-03-06T07:23:30.050278+0000","last_undegraded":"2024-03-06T07:23:30.050278+0000","last_fullsized":"2024-03-06T07:23:30.050278+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:42.593912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053118+0000","last_change":"2024-03-06T07:23:29.061235+0000","last_active":"2024-03-06T07:23:30.053118+0000","last_peered":"2024-03-06T07:23:30.053118+0000","last_clean":"2024-03-06T07:23:30.053118+0000","last_became_active":"2024-03-06T07:23:29.060553+0000","last_became_peered":"2024-03-06T07:23:29.060553+0000","last_unstale":"2024-03-06T07:23:30.053118+0000","last_undegraded":"2024-03-06T07:23:30.053118+0000","last_fullsized":"2024-03-06T07:23:30.053118+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:41.727713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053018+0000","last_change":"2024-03-06T07:23:29.051633+0000","last_active":"2024-03-06T07:23:30.053018+0000","last_peered":"2024-03-06T07:23:30.053018+0000","last_clean":"2024-03-06T07:23:30.053018+0000","last_became_active":"2024-03-06T07:23:29.051383+0000","last_became_peered":"2024-03-06T07:23:29.051383+0000","last_unstale":"2024-03-06T07:23:30.053018+0000","last_undegraded":"2024-03-06T07:23:30.053018+0000","last_fullsized":"2024-03-06T07:23:30.053018+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:56.509431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053613+0000","last_change":"2024-03-06T07:23:29.061495+0000","last_active":"2024-03-06T07:23:30.053613+0000","last_peered":"2024-03-06T07:23:30.053613+0000","last_clean":"2024-03-06T07:23:30.053613+0000","last_became_active":"2024-03-06T07:23:29.061193+0000","last_became_peered":"2024-03-06T07:23:29.061193+0000","last_unstale":"2024-03-06T07:23:30.053613+0000","last_undegraded":"2024-03-06T07:23:30.053613+0000","last_fullsized":"2024-03-06T07:23:30.053613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:53:59.837257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910673+0000","last_change":"2024-03-06T07:23:29.066543+0000","last_active":"2024-03-06T07:23:30.910673+0000","last_peered":"2024-03-06T07:23:30.910673+0000","last_clean":"2024-03-06T07:23:30.910673+0000","last_became_active":"2024-03-06T07:23:29.066278+0000","last_became_peered":"2024-03-06T07:23:29.066278+0000","last_unstale":"2024-03-06T07:23:30.910673+0000","last_undegraded":"2024-03-06T07:23:30.910673+0000","last_fullsized":"2024-03-06T07:23:30.910673+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:26.428158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.909966+0000","last_change":"2024-03-06T07:23:29.053533+0000","last_active":"2024-03-06T07:23:30.909966+0000","last_peered":"2024-03-06T07:23:30.909966+0000","last_clean":"2024-03-06T07:23:30.909966+0000","last_became_active":"2024-03-06T07:23:29.053216+0000","last_became_peered":"2024-03-06T07:23:29.053216+0000","last_unstale":"2024-03-06T07:23:30.909966+0000","last_undegraded":"2024-03-06T07:23:30.909966+0000","last_fullsized":"2024-03-06T07:23:30.909966+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:48.702926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.052028+0000","last_change":"2024-03-06T07:23:29.063417+0000","last_active":"2024-03-06T07:23:30.052028+0000","last_peered":"2024-03-06T07:23:30.052028+0000","last_clean":"2024-03-06T07:23:30.052028+0000","last_became_active":"2024-03-06T07:23:29.063135+0000","last_became_peered":"2024-03-06T07:23:29.063135+0000","last_unstale":"2024-03-06T07:23:30.052028+0000","last_undegraded":"2024-03-06T07:23:30.052028+0000","last_fullsized":"2024-03-06T07:23:30.052028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:27.610870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053594+0000","last_change":"2024-03-06T07:23:29.065029+0000","last_active":"2024-03-06T07:23:30.053594+0000","last_peered":"2024-03-06T07:23:30.053594+0000","last_clean":"2024-03-06T07:23:30.053594+0000","last_became_active":"2024-03-06T07:23:29.064698+0000","last_became_peered":"2024-03-06T07:23:29.064698+0000","last_unstale":"2024-03-06T07:23:30.053594+0000","last_undegraded":"2024-03-06T07:23:30.053594+0000","last_fullsized":"2024-03-06T07:23:30.053594+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:15.652100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.909803+0000","last_change":"2024-03-06T07:23:29.055932+0000","last_active":"2024-03-06T07:23:30.909803+0000","last_peered":"2024-03-06T07:23:30.909803+0000","last_clean":"2024-03-06T07:23:30.909803+0000","last_became_active":"2024-03-06T07:23:29.055659+0000","last_became_peered":"2024-03-06T07:23:29.055659+0000","last_unstale":"2024-03-06T07:23:30.909803+0000","last_undegraded":"2024-03-06T07:23:30.909803+0000","last_fullsized":"2024-03-06T07:23:30.909803+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:37.308581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910625+0000","last_change":"2024-03-06T07:23:29.042006+0000","last_active":"2024-03-06T07:23:30.910625+0000","last_peered":"2024-03-06T07:23:30.910625+0000","last_clean":"2024-03-06T07:23:30.910625+0000","last_became_active":"2024-03-06T07:23:29.041681+0000","last_became_peered":"2024-03-06T07:23:29.041681+0000","last_unstale":"2024-03-06T07:23:30.910625+0000","last_undegraded":"2024-03-06T07:23:30.910625+0000","last_fullsized":"2024-03-06T07:23:30.910625+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:56.269198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910251+0000","last_change":"2024-03-06T07:23:29.060802+0000","last_active":"2024-03-06T07:23:30.910251+0000","last_peered":"2024-03-06T07:23:30.910251+0000","last_clean":"2024-03-06T07:23:30.910251+0000","last_became_active":"2024-03-06T07:23:29.060527+0000","last_became_peered":"2024-03-06T07:23:29.060527+0000","last_unstale":"2024-03-06T07:23:30.910251+0000","last_undegraded":"2024-03-06T07:23:30.910251+0000","last_fullsized":"2024-03-06T07:23:30.910251+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:24.410692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.617772+0000","last_change":"2024-03-06T07:23:29.066398+0000","last_active":"2024-03-06T07:23:30.617772+0000","last_peered":"2024-03-06T07:23:30.617772+0000","last_clean":"2024-03-06T07:23:30.617772+0000","last_became_active":"2024-03-06T07:23:29.066140+0000","last_became_peered":"2024-03-06T07:23:29.066140+0000","last_unstale":"2024-03-06T07:23:30.617772+0000","last_undegraded":"2024-03-06T07:23:30.617772+0000","last_fullsized":"2024-03-06T07:23:30.617772+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:59.838775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910759+0000","last_change":"2024-03-06T07:23:28.054918+0000","last_active":"2024-03-06T07:23:30.910759+0000","last_peered":"2024-03-06T07:23:30.910759+0000","last_clean":"2024-03-06T07:23:30.910759+0000","last_became_active":"2024-03-06T07:23:28.054613+0000","last_became_peered":"2024-03-06T07:23:28.054613+0000","last_unstale":"2024-03-06T07:23:30.910759+0000","last_undegraded":"2024-03-06T07:23:30.910759+0000","last_fullsized":"2024-03-06T07:23:30.910759+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:02.697278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053629+0000","last_change":"2024-03-06T07:23:29.058694+0000","last_active":"2024-03-06T07:23:30.053629+0000","last_peered":"2024-03-06T07:23:30.053629+0000","last_clean":"2024-03-06T07:23:30.053629+0000","last_became_active":"2024-03-06T07:23:29.058385+0000","last_became_peered":"2024-03-06T07:23:29.058385+0000","last_unstale":"2024-03-06T07:23:30.053629+0000","last_undegraded":"2024-03-06T07:23:30.053629+0000","last_fullsized":"2024-03-06T07:23:30.053629+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:05.333397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910288+0000","last_change":"2024-03-06T07:23:28.048972+0000","last_active":"2024-03-06T07:23:30.910288+0000","last_peered":"2024-03-06T07:23:30.910288+0000","last_clean":"2024-03-06T07:23:30.910288+0000","last_became_active":"2024-03-06T07:23:28.048697+0000","last_became_peered":"2024-03-06T07:23:28.048697+0000","last_unstale":"2024-03-06T07:23:30.910288+0000","last_undegraded":"2024-03-06T07:23:30.910288+0000","last_fullsized":"2024-03-06T07:23:30.910288+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:23.826865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.052176+0000","last_change":"2024-03-06T07:23:29.047607+0000","last_active":"2024-03-06T07:23:30.052176+0000","last_peered":"2024-03-06T07:23:30.052176+0000","last_clean":"2024-03-06T07:23:30.052176+0000","last_became_active":"2024-03-06T07:23:29.047272+0000","last_became_peered":"2024-03-06T07:23:29.047272+0000","last_unstale":"2024-03-06T07:23:30.052176+0000","last_undegraded":"2024-03-06T07:23:30.052176+0000","last_fullsized":"2024-03-06T07:23:30.052176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:24.415779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.617950+0000","last_change":"2024-03-06T07:23:28.049032+0000","last_active":"2024-03-06T07:23:30.617950+0000","last_peered":"2024-03-06T07:23:30.617950+0000","last_clean":"2024-03-06T07:23:30.617950+0000","last_became_active":"2024-03-06T07:23:28.048743+0000","last_became_peered":"2024-03-06T07:23:28.048743+0000","last_unstale":"2024-03-06T07:23:30.617950+0000","last_undegraded":"2024-03-06T07:23:30.617950+0000","last_fullsized":"2024-03-06T07:23:30.617950+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:54.482990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.617794+0000","last_change":"2024-03-06T07:23:29.061546+0000","last_active":"2024-03-06T07:23:30.617794+0000","last_peered":"2024-03-06T07:23:30.617794+0000","last_clean":"2024-03-06T07:23:30.617794+0000","last_became_active":"2024-03-06T07:23:29.061272+0000","last_became_peered":"2024-03-06T07:23:29.061272+0000","last_unstale":"2024-03-06T07:23:30.617794+0000","last_undegraded":"2024-03-06T07:23:30.617794+0000","last_fullsized":"2024-03-06T07:23:30.617794+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:42.593912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910451+0000","last_change":"2024-03-06T07:23:28.054041+0000","last_active":"2024-03-06T07:23:30.910451+0000","last_peered":"2024-03-06T07:23:30.910451+0000","last_clean":"2024-03-06T07:23:30.910451+0000","last_became_active":"2024-03-06T07:23:28.053739+0000","last_became_peered":"2024-03-06T07:23:28.053739+0000","last_unstale":"2024-03-06T07:23:30.910451+0000","last_undegraded":"2024-03-06T07:23:30.910451+0000","last_fullsized":"2024-03-06T07:23:30.910451+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:32.942286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.909873+0000","last_change":"2024-03-06T07:23:29.052519+0000","last_active":"2024-03-06T07:23:30.909873+0000","last_peered":"2024-03-06T07:23:30.909873+0000","last_clean":"2024-03-06T07:23:30.909873+0000","last_became_active":"2024-03-06T07:23:29.052186+0000","last_became_peered":"2024-03-06T07:23:29.052186+0000","last_unstale":"2024-03-06T07:23:30.909873+0000","last_undegraded":"2024-03-06T07:23:30.909873+0000","last_fullsized":"2024-03-06T07:23:30.909873+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:55.486686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910670+0000","last_change":"2024-03-06T07:23:28.056182+0000","last_active":"2024-03-06T07:23:30.910670+0000","last_peered":"2024-03-06T07:23:30.910670+0000","last_clean":"2024-03-06T07:23:30.910670+0000","last_became_active":"2024-03-06T07:23:28.055643+0000","last_became_peered":"2024-03-06T07:23:28.055643+0000","last_unstale":"2024-03-06T07:23:30.910670+0000","last_undegraded":"2024-03-06T07:23:30.910670+0000","last_fullsized":"2024-03-06T07:23:30.910670+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:07:26.607174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050298+0000","last_change":"2024-03-06T07:23:29.061008+0000","last_active":"2024-03-06T07:23:30.050298+0000","last_peered":"2024-03-06T07:23:30.050298+0000","last_clean":"2024-03-06T07:23:30.050298+0000","last_became_active":"2024-03-06T07:23:29.060729+0000","last_became_peered":"2024-03-06T07:23:29.060729+0000","last_unstale":"2024-03-06T07:23:30.050298+0000","last_undegraded":"2024-03-06T07:23:30.050298+0000","last_fullsized":"2024-03-06T07:23:30.050298+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:33.945983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.909933+0000","last_change":"2024-03-06T07:23:28.051998+0000","last_active":"2024-03-06T07:23:30.909933+0000","last_peered":"2024-03-06T07:23:30.909933+0000","last_clean":"2024-03-06T07:23:30.909933+0000","last_became_active":"2024-03-06T07:23:28.051417+0000","last_became_peered":"2024-03-06T07:23:28.051417+0000","last_unstale":"2024-03-06T07:23:30.909933+0000","last_undegraded":"2024-03-06T07:23:30.909933+0000","last_fullsized":"2024-03-06T07:23:30.909933+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:41.774950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050169+0000","last_change":"2024-03-06T07:23:29.057267+0000","last_active":"2024-03-06T07:23:30.050169+0000","last_peered":"2024-03-06T07:23:30.050169+0000","last_clean":"2024-03-06T07:23:30.050169+0000","last_became_active":"2024-03-06T07:23:29.055394+0000","last_became_peered":"2024-03-06T07:23:29.055394+0000","last_unstale":"2024-03-06T07:23:30.050169+0000","last_undegraded":"2024-03-06T07:23:30.050169+0000","last_fullsized":"2024-03-06T07:23:30.050169+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:59.838775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.051826+0000","last_change":"2024-03-06T07:23:28.047326+0000","last_active":"2024-03-06T07:23:30.051826+0000","last_peered":"2024-03-06T07:23:30.051826+0000","last_clean":"2024-03-06T07:23:30.051826+0000","last_became_active":"2024-03-06T07:23:28.046947+0000","last_became_peered":"2024-03-06T07:23:28.046947+0000","last_unstale":"2024-03-06T07:23:30.051826+0000","last_undegraded":"2024-03-06T07:23:30.051826+0000","last_fullsized":"2024-03-06T07:23:30.051826+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:38:31.496793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053335+0000","last_change":"2024-03-06T07:23:29.052602+0000","last_active":"2024-03-06T07:23:30.053335+0000","last_peered":"2024-03-06T07:23:30.053335+0000","last_clean":"2024-03-06T07:23:30.053335+0000","last_became_active":"2024-03-06T07:23:29.052353+0000","last_became_peered":"2024-03-06T07:23:29.052353+0000","last_unstale":"2024-03-06T07:23:30.053335+0000","last_undegraded":"2024-03-06T07:23:30.053335+0000","last_fullsized":"2024-03-06T07:23:30.053335+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:50.621162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050282+0000","last_change":"2024-03-06T07:23:28.046157+0000","last_active":"2024-03-06T07:23:30.050282+0000","last_peered":"2024-03-06T07:23:30.050282+0000","last_clean":"2024-03-06T07:23:30.050282+0000","last_became_active":"2024-03-06T07:23:28.045827+0000","last_became_peered":"2024-03-06T07:23:28.045827+0000","last_unstale":"2024-03-06T07:23:30.050282+0000","last_undegraded":"2024-03-06T07:23:30.050282+0000","last_fullsized":"2024-03-06T07:23:30.050282+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:02.488678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910139+0000","last_change":"2024-03-06T07:23:29.052213+0000","last_active":"2024-03-06T07:23:30.910139+0000","last_peered":"2024-03-06T07:23:30.910139+0000","last_clean":"2024-03-06T07:23:30.910139+0000","last_became_active":"2024-03-06T07:23:29.051909+0000","last_became_peered":"2024-03-06T07:23:29.051909+0000","last_unstale":"2024-03-06T07:23:30.910139+0000","last_undegraded":"2024-03-06T07:23:30.910139+0000","last_fullsized":"2024-03-06T07:23:30.910139+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:23.840986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910420+0000","last_change":"2024-03-06T07:23:28.052681+0000","last_active":"2024-03-06T07:23:30.910420+0000","last_peered":"2024-03-06T07:23:30.910420+0000","last_clean":"2024-03-06T07:23:30.910420+0000","last_became_active":"2024-03-06T07:23:28.052456+0000","last_became_peered":"2024-03-06T07:23:28.052456+0000","last_unstale":"2024-03-06T07:23:30.910420+0000","last_undegraded":"2024-03-06T07:23:30.910420+0000","last_fullsized":"2024-03-06T07:23:30.910420+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:55.505735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910673+0000","last_change":"2024-03-06T07:23:29.045762+0000","last_active":"2024-03-06T07:23:30.910673+0000","last_peered":"2024-03-06T07:23:30.910673+0000","last_clean":"2024-03-06T07:23:30.910673+0000","last_became_active":"2024-03-06T07:23:29.045488+0000","last_became_peered":"2024-03-06T07:23:29.045488+0000","last_unstale":"2024-03-06T07:23:30.910673+0000","last_undegraded":"2024-03-06T07:23:30.910673+0000","last_fullsized":"2024-03-06T07:23:30.910673+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:55.812186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050824+0000","last_change":"2024-03-06T07:23:28.053860+0000","last_active":"2024-03-06T07:23:30.050824+0000","last_peered":"2024-03-06T07:23:30.050824+0000","last_clean":"2024-03-06T07:23:30.050824+0000","last_became_active":"2024-03-06T07:23:28.053178+0000","last_became_peered":"2024-03-06T07:23:28.053178+0000","last_unstale":"2024-03-06T07:23:30.050824+0000","last_undegraded":"2024-03-06T07:23:30.050824+0000","last_fullsized":"2024-03-06T07:23:30.050824+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:55.505735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.911090+0000","last_change":"2024-03-06T07:23:29.052015+0000","last_active":"2024-03-06T07:23:30.911090+0000","last_peered":"2024-03-06T07:23:30.911090+0000","last_clean":"2024-03-06T07:23:30.911090+0000","last_became_active":"2024-03-06T07:23:29.051736+0000","last_became_peered":"2024-03-06T07:23:29.051736+0000","last_unstale":"2024-03-06T07:23:30.911090+0000","last_undegraded":"2024-03-06T07:23:30.911090+0000","last_fullsized":"2024-03-06T07:23:30.911090+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:23.730232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910685+0000","last_change":"2024-03-06T07:23:28.050628+0000","last_active":"2024-03-06T07:23:30.910685+0000","last_peered":"2024-03-06T07:23:30.910685+0000","last_clean":"2024-03-06T07:23:30.910685+0000","last_became_active":"2024-03-06T07:23:28.050327+0000","last_became_peered":"2024-03-06T07:23:28.050327+0000","last_unstale":"2024-03-06T07:23:30.910685+0000","last_undegraded":"2024-03-06T07:23:30.910685+0000","last_fullsized":"2024-03-06T07:23:30.910685+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:37.975276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910564+0000","last_change":"2024-03-06T07:23:29.050452+0000","last_active":"2024-03-06T07:23:30.910564+0000","last_peered":"2024-03-06T07:23:30.910564+0000","last_clean":"2024-03-06T07:23:30.910564+0000","last_became_active":"2024-03-06T07:23:29.050121+0000","last_became_peered":"2024-03-06T07:23:29.050121+0000","last_unstale":"2024-03-06T07:23:30.910564+0000","last_undegraded":"2024-03-06T07:23:30.910564+0000","last_fullsized":"2024-03-06T07:23:30.910564+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:52.858997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.911065+0000","last_change":"2024-03-06T07:23:28.061203+0000","last_active":"2024-03-06T07:23:30.911065+0000","last_peered":"2024-03-06T07:23:30.911065+0000","last_clean":"2024-03-06T07:23:30.911065+0000","last_became_active":"2024-03-06T07:23:28.060875+0000","last_became_peered":"2024-03-06T07:23:28.060875+0000","last_unstale":"2024-03-06T07:23:30.911065+0000","last_undegraded":"2024-03-06T07:23:30.911065+0000","last_fullsized":"2024-03-06T07:23:30.911065+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:18.999025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053494+0000","last_change":"2024-03-06T07:23:29.058421+0000","last_active":"2024-03-06T07:23:30.053494+0000","last_peered":"2024-03-06T07:23:30.053494+0000","last_clean":"2024-03-06T07:23:30.053494+0000","last_became_active":"2024-03-06T07:23:29.057985+0000","last_became_peered":"2024-03-06T07:23:29.057985+0000","last_unstale":"2024-03-06T07:23:30.053494+0000","last_undegraded":"2024-03-06T07:23:30.053494+0000","last_fullsized":"2024-03-06T07:23:30.053494+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:31.491955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910799+0000","last_change":"2024-03-06T07:23:28.052868+0000","last_active":"2024-03-06T07:23:30.910799+0000","last_peered":"2024-03-06T07:23:30.910799+0000","last_clean":"2024-03-06T07:23:30.910799+0000","last_became_active":"2024-03-06T07:23:28.052267+0000","last_became_peered":"2024-03-06T07:23:28.052267+0000","last_unstale":"2024-03-06T07:23:30.910799+0000","last_undegraded":"2024-03-06T07:23:30.910799+0000","last_fullsized":"2024-03-06T07:23:30.910799+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:23.400225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050407+0000","last_change":"2024-03-06T07:23:29.050696+0000","last_active":"2024-03-06T07:23:30.050407+0000","last_peered":"2024-03-06T07:23:30.050407+0000","last_clean":"2024-03-06T07:23:30.050407+0000","last_became_active":"2024-03-06T07:23:29.050326+0000","last_became_peered":"2024-03-06T07:23:29.050326+0000","last_unstale":"2024-03-06T07:23:30.050407+0000","last_undegraded":"2024-03-06T07:23:30.050407+0000","last_fullsized":"2024-03-06T07:23:30.050407+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:51.346838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910297+0000","last_change":"2024-03-06T07:23:28.051616+0000","last_active":"2024-03-06T07:23:30.910297+0000","last_peered":"2024-03-06T07:23:30.910297+0000","last_clean":"2024-03-06T07:23:30.910297+0000","last_became_active":"2024-03-06T07:23:28.051362+0000","last_became_peered":"2024-03-06T07:23:28.051362+0000","last_unstale":"2024-03-06T07:23:30.910297+0000","last_undegraded":"2024-03-06T07:23:30.910297+0000","last_fullsized":"2024-03-06T07:23:30.910297+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:50.343142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910290+0000","last_change":"2024-03-06T07:23:29.051695+0000","last_active":"2024-03-06T07:23:30.910290+0000","last_peered":"2024-03-06T07:23:30.910290+0000","last_clean":"2024-03-06T07:23:30.910290+0000","last_became_active":"2024-03-06T07:23:29.051414+0000","last_became_peered":"2024-03-06T07:23:29.051414+0000","last_unstale":"2024-03-06T07:23:30.910290+0000","last_undegraded":"2024-03-06T07:23:30.910290+0000","last_fullsized":"2024-03-06T07:23:30.910290+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:18.961412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910748+0000","last_change":"2024-03-06T07:23:28.046582+0000","last_active":"2024-03-06T07:23:30.910748+0000","last_peered":"2024-03-06T07:23:30.910748+0000","last_clean":"2024-03-06T07:23:30.910748+0000","last_became_active":"2024-03-06T07:23:28.046321+0000","last_became_peered":"2024-03-06T07:23:28.046321+0000","last_unstale":"2024-03-06T07:23:30.910748+0000","last_undegraded":"2024-03-06T07:23:30.910748+0000","last_fullsized":"2024-03-06T07:23:30.910748+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:34.337697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910210+0000","last_change":"2024-03-06T07:23:29.056856+0000","last_active":"2024-03-06T07:23:30.910210+0000","last_peered":"2024-03-06T07:23:30.910210+0000","last_clean":"2024-03-06T07:23:30.910210+0000","last_became_active":"2024-03-06T07:23:29.056572+0000","last_became_peered":"2024-03-06T07:23:29.056572+0000","last_unstale":"2024-03-06T07:23:30.910210+0000","last_undegraded":"2024-03-06T07:23:30.910210+0000","last_fullsized":"2024-03-06T07:23:30.910210+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:52.899579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050286+0000","last_change":"2024-03-06T07:23:28.042949+0000","last_active":"2024-03-06T07:23:30.050286+0000","last_peered":"2024-03-06T07:23:30.050286+0000","last_clean":"2024-03-06T07:23:30.050286+0000","last_became_active":"2024-03-06T07:23:28.042653+0000","last_became_peered":"2024-03-06T07:23:28.042653+0000","last_unstale":"2024-03-06T07:23:30.050286+0000","last_undegraded":"2024-03-06T07:23:30.050286+0000","last_fullsized":"2024-03-06T07:23:30.050286+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:23.826865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910791+0000","last_change":"2024-03-06T07:23:29.054265+0000","last_active":"2024-03-06T07:23:30.910791+0000","last_peered":"2024-03-06T07:23:30.910791+0000","last_clean":"2024-03-06T07:23:30.910791+0000","last_became_active":"2024-03-06T07:23:29.053992+0000","last_became_peered":"2024-03-06T07:23:29.053992+0000","last_unstale":"2024-03-06T07:23:30.910791+0000","last_undegraded":"2024-03-06T07:23:30.910791+0000","last_fullsized":"2024-03-06T07:23:30.910791+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:20.295108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910512+0000","last_change":"2024-03-06T07:23:28.059645+0000","last_active":"2024-03-06T07:23:30.910512+0000","last_peered":"2024-03-06T07:23:30.910512+0000","last_clean":"2024-03-06T07:23:30.910512+0000","last_became_active":"2024-03-06T07:23:28.059350+0000","last_became_peered":"2024-03-06T07:23:28.059350+0000","last_unstale":"2024-03-06T07:23:30.910512+0000","last_undegraded":"2024-03-06T07:23:30.910512+0000","last_fullsized":"2024-03-06T07:23:30.910512+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:38.416301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050956+0000","last_change":"2024-03-06T07:23:29.051554+0000","last_active":"2024-03-06T07:23:30.050956+0000","last_peered":"2024-03-06T07:23:30.050956+0000","last_clean":"2024-03-06T07:23:30.050956+0000","last_became_active":"2024-03-06T07:23:29.051219+0000","last_became_peered":"2024-03-06T07:23:29.051219+0000","last_unstale":"2024-03-06T07:23:30.050956+0000","last_undegraded":"2024-03-06T07:23:30.050956+0000","last_fullsized":"2024-03-06T07:23:30.050956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:12.848240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910281+0000","last_change":"2024-03-06T07:23:28.058938+0000","last_active":"2024-03-06T07:23:30.910281+0000","last_peered":"2024-03-06T07:23:30.910281+0000","last_clean":"2024-03-06T07:23:30.910281+0000","last_became_active":"2024-03-06T07:23:28.058565+0000","last_became_peered":"2024-03-06T07:23:28.058565+0000","last_unstale":"2024-03-06T07:23:30.910281+0000","last_undegraded":"2024-03-06T07:23:30.910281+0000","last_fullsized":"2024-03-06T07:23:30.910281+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:02.488678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053363+0000","last_change":"2024-03-06T07:23:29.061228+0000","last_active":"2024-03-06T07:23:30.053363+0000","last_peered":"2024-03-06T07:23:30.053363+0000","last_clean":"2024-03-06T07:23:30.053363+0000","last_became_active":"2024-03-06T07:23:29.060835+0000","last_became_peered":"2024-03-06T07:23:29.060835+0000","last_unstale":"2024-03-06T07:23:30.053363+0000","last_undegraded":"2024-03-06T07:23:30.053363+0000","last_fullsized":"2024-03-06T07:23:30.053363+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:24:30.978821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.052164+0000","last_change":"2024-03-06T07:23:28.056074+0000","last_active":"2024-03-06T07:23:30.052164+0000","last_peered":"2024-03-06T07:23:30.052164+0000","last_clean":"2024-03-06T07:23:30.052164+0000","last_became_active":"2024-03-06T07:23:28.055798+0000","last_became_peered":"2024-03-06T07:23:28.055798+0000","last_unstale":"2024-03-06T07:23:30.052164+0000","last_undegraded":"2024-03-06T07:23:30.052164+0000","last_fullsized":"2024-03-06T07:23:30.052164+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:58.835079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053688+0000","last_change":"2024-03-06T07:23:29.064408+0000","last_active":"2024-03-06T07:23:30.053688+0000","last_peered":"2024-03-06T07:23:30.053688+0000","last_clean":"2024-03-06T07:23:30.053688+0000","last_became_active":"2024-03-06T07:23:29.064118+0000","last_became_peered":"2024-03-06T07:23:29.064118+0000","last_unstale":"2024-03-06T07:23:30.053688+0000","last_undegraded":"2024-03-06T07:23:30.053688+0000","last_fullsized":"2024-03-06T07:23:30.053688+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:37.308581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.911099+0000","last_change":"2024-03-06T07:23:28.055018+0000","last_active":"2024-03-06T07:23:30.911099+0000","last_peered":"2024-03-06T07:23:30.911099+0000","last_clean":"2024-03-06T07:23:30.911099+0000","last_became_active":"2024-03-06T07:23:28.054759+0000","last_became_peered":"2024-03-06T07:23:28.054759+0000","last_unstale":"2024-03-06T07:23:30.911099+0000","last_undegraded":"2024-03-06T07:23:30.911099+0000","last_fullsized":"2024-03-06T07:23:30.911099+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:36.377822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.911185+0000","last_change":"2024-03-06T07:23:29.066859+0000","last_active":"2024-03-06T07:23:30.911185+0000","last_peered":"2024-03-06T07:23:30.911185+0000","last_clean":"2024-03-06T07:23:30.911185+0000","last_became_active":"2024-03-06T07:23:29.066539+0000","last_became_peered":"2024-03-06T07:23:29.066539+0000","last_unstale":"2024-03-06T07:23:30.911185+0000","last_undegraded":"2024-03-06T07:23:30.911185+0000","last_fullsized":"2024-03-06T07:23:30.911185+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:13.398879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.911037+0000","last_change":"2024-03-06T07:23:28.055288+0000","last_active":"2024-03-06T07:23:30.911037+0000","last_peered":"2024-03-06T07:23:30.911037+0000","last_clean":"2024-03-06T07:23:30.911037+0000","last_became_active":"2024-03-06T07:23:28.053772+0000","last_became_peered":"2024-03-06T07:23:28.053772+0000","last_unstale":"2024-03-06T07:23:30.911037+0000","last_undegraded":"2024-03-06T07:23:30.911037+0000","last_fullsized":"2024-03-06T07:23:30.911037+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:37:11.844544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.617703+0000","last_change":"2024-03-06T07:23:29.048068+0000","last_active":"2024-03-06T07:23:30.617703+0000","last_peered":"2024-03-06T07:23:30.617703+0000","last_clean":"2024-03-06T07:23:30.617703+0000","last_became_active":"2024-03-06T07:23:29.047776+0000","last_became_peered":"2024-03-06T07:23:29.047776+0000","last_unstale":"2024-03-06T07:23:30.617703+0000","last_undegraded":"2024-03-06T07:23:30.617703+0000","last_fullsized":"2024-03-06T07:23:30.617703+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:32.549563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.051848+0000","last_change":"2024-03-06T07:23:28.048978+0000","last_active":"2024-03-06T07:23:30.051848+0000","last_peered":"2024-03-06T07:23:30.051848+0000","last_clean":"2024-03-06T07:23:30.051848+0000","last_became_active":"2024-03-06T07:23:28.048667+0000","last_became_peered":"2024-03-06T07:23:28.048667+0000","last_unstale":"2024-03-06T07:23:30.051848+0000","last_undegraded":"2024-03-06T07:23:30.051848+0000","last_fullsized":"2024-03-06T07:23:30.051848+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:31.545867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910267+0000","last_change":"2024-03-06T07:23:29.060473+0000","last_active":"2024-03-06T07:23:30.910267+0000","last_peered":"2024-03-06T07:23:30.910267+0000","last_clean":"2024-03-06T07:23:30.910267+0000","last_became_active":"2024-03-06T07:23:29.060166+0000","last_became_peered":"2024-03-06T07:23:29.060166+0000","last_unstale":"2024-03-06T07:23:30.910267+0000","last_undegraded":"2024-03-06T07:23:30.910267+0000","last_fullsized":"2024-03-06T07:23:30.910267+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:29.421043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050167+0000","last_change":"2024-03-06T07:23:28.049913+0000","last_active":"2024-03-06T07:23:30.050167+0000","last_peered":"2024-03-06T07:23:30.050167+0000","last_clean":"2024-03-06T07:23:30.050167+0000","last_became_active":"2024-03-06T07:23:28.049627+0000","last_became_peered":"2024-03-06T07:23:28.049627+0000","last_unstale":"2024-03-06T07:23:30.050167+0000","last_undegraded":"2024-03-06T07:23:30.050167+0000","last_fullsized":"2024-03-06T07:23:30.050167+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:41.774950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050155+0000","last_change":"2024-03-06T07:23:29.050108+0000","last_active":"2024-03-06T07:23:30.050155+0000","last_peered":"2024-03-06T07:23:30.050155+0000","last_clean":"2024-03-06T07:23:30.050155+0000","last_became_active":"2024-03-06T07:23:29.049794+0000","last_became_peered":"2024-03-06T07:23:29.049794+0000","last_unstale":"2024-03-06T07:23:30.050155+0000","last_undegraded":"2024-03-06T07:23:30.050155+0000","last_fullsized":"2024-03-06T07:23:30.050155+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:47:32.549563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910031+0000","last_change":"2024-03-06T07:23:28.061522+0000","last_active":"2024-03-06T07:23:30.910031+0000","last_peered":"2024-03-06T07:23:30.910031+0000","last_clean":"2024-03-06T07:23:30.910031+0000","last_became_active":"2024-03-06T07:23:28.061194+0000","last_became_peered":"2024-03-06T07:23:28.061194+0000","last_unstale":"2024-03-06T07:23:30.910031+0000","last_undegraded":"2024-03-06T07:23:30.910031+0000","last_fullsized":"2024-03-06T07:23:30.910031+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:41.590215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.053766+0000","last_change":"2024-03-06T07:23:29.061624+0000","last_active":"2024-03-06T07:23:30.053766+0000","last_peered":"2024-03-06T07:23:30.053766+0000","last_clean":"2024-03-06T07:23:30.053766+0000","last_became_active":"2024-03-06T07:23:29.060998+0000","last_became_peered":"2024-03-06T07:23:29.060998+0000","last_unstale":"2024-03-06T07:23:30.053766+0000","last_undegraded":"2024-03-06T07:23:30.053766+0000","last_fullsized":"2024-03-06T07:23:30.053766+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:40.523420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910698+0000","last_change":"2024-03-06T07:23:28.059116+0000","last_active":"2024-03-06T07:23:30.910698+0000","last_peered":"2024-03-06T07:23:30.910698+0000","last_clean":"2024-03-06T07:23:30.910698+0000","last_became_active":"2024-03-06T07:23:28.058817+0000","last_became_peered":"2024-03-06T07:23:28.058817+0000","last_unstale":"2024-03-06T07:23:30.910698+0000","last_undegraded":"2024-03-06T07:23:30.910698+0000","last_fullsized":"2024-03-06T07:23:30.910698+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:37.975276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050646+0000","last_change":"2024-03-06T07:23:29.053368+0000","last_active":"2024-03-06T07:23:30.050646+0000","last_peered":"2024-03-06T07:23:30.050646+0000","last_clean":"2024-03-06T07:23:30.050646+0000","last_became_active":"2024-03-06T07:23:29.053086+0000","last_became_peered":"2024-03-06T07:23:29.053086+0000","last_unstale":"2024-03-06T07:23:30.050646+0000","last_undegraded":"2024-03-06T07:23:30.050646+0000","last_fullsized":"2024-03-06T07:23:30.050646+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:03.700974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910657+0000","last_change":"2024-03-06T07:23:28.050767+0000","last_active":"2024-03-06T07:23:30.910657+0000","last_peered":"2024-03-06T07:23:30.910657+0000","last_clean":"2024-03-06T07:23:30.910657+0000","last_became_active":"2024-03-06T07:23:28.050357+0000","last_became_peered":"2024-03-06T07:23:28.050357+0000","last_unstale":"2024-03-06T07:23:30.910657+0000","last_undegraded":"2024-03-06T07:23:30.910657+0000","last_fullsized":"2024-03-06T07:23:30.910657+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:34.337697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910594+0000","last_change":"2024-03-06T07:23:29.060684+0000","last_active":"2024-03-06T07:23:30.910594+0000","last_peered":"2024-03-06T07:23:30.910594+0000","last_clean":"2024-03-06T07:23:30.910594+0000","last_became_active":"2024-03-06T07:23:29.060284+0000","last_became_peered":"2024-03-06T07:23:29.060284+0000","last_unstale":"2024-03-06T07:23:30.910594+0000","last_undegraded":"2024-03-06T07:23:30.910594+0000","last_fullsized":"2024-03-06T07:23:30.910594+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:26:52.424550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.617629+0000","last_change":"2024-03-06T07:23:28.050491+0000","last_active":"2024-03-06T07:23:30.617629+0000","last_peered":"2024-03-06T07:23:30.617629+0000","last_clean":"2024-03-06T07:23:30.617629+0000","last_became_active":"2024-03-06T07:23:28.050224+0000","last_became_peered":"2024-03-06T07:23:28.050224+0000","last_unstale":"2024-03-06T07:23:30.617629+0000","last_undegraded":"2024-03-06T07:23:30.617629+0000","last_fullsized":"2024-03-06T07:23:30.617629+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:36.304885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.911273+0000","last_change":"2024-03-06T07:23:29.047962+0000","last_active":"2024-03-06T07:23:30.911273+0000","last_peered":"2024-03-06T07:23:30.911273+0000","last_clean":"2024-03-06T07:23:30.911273+0000","last_became_active":"2024-03-06T07:23:29.047693+0000","last_became_peered":"2024-03-06T07:23:29.047693+0000","last_unstale":"2024-03-06T07:23:30.911273+0000","last_undegraded":"2024-03-06T07:23:30.911273+0000","last_fullsized":"2024-03-06T07:23:30.911273+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:54.829577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910501+0000","last_change":"2024-03-06T07:23:28.050364+0000","last_active":"2024-03-06T07:23:30.910501+0000","last_peered":"2024-03-06T07:23:30.910501+0000","last_clean":"2024-03-06T07:23:30.910501+0000","last_became_active":"2024-03-06T07:23:28.050109+0000","last_became_peered":"2024-03-06T07:23:28.050109+0000","last_unstale":"2024-03-06T07:23:30.910501+0000","last_undegraded":"2024-03-06T07:23:30.910501+0000","last_fullsized":"2024-03-06T07:23:30.910501+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:02.697278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910566+0000","last_change":"2024-03-06T07:23:29.066297+0000","last_active":"2024-03-06T07:23:30.910566+0000","last_peered":"2024-03-06T07:23:30.910566+0000","last_clean":"2024-03-06T07:23:30.910566+0000","last_became_active":"2024-03-06T07:23:29.066000+0000","last_became_peered":"2024-03-06T07:23:29.066000+0000","last_unstale":"2024-03-06T07:23:30.910566+0000","last_undegraded":"2024-03-06T07:23:30.910566+0000","last_fullsized":"2024-03-06T07:23:30.910566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:37.856912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.909934+0000","last_change":"2024-03-06T07:23:28.059557+0000","last_active":"2024-03-06T07:23:30.909934+0000","last_peered":"2024-03-06T07:23:30.909934+0000","last_clean":"2024-03-06T07:23:30.909934+0000","last_became_active":"2024-03-06T07:23:28.059302+0000","last_became_peered":"2024-03-06T07:23:28.059302+0000","last_unstale":"2024-03-06T07:23:30.909934+0000","last_undegraded":"2024-03-06T07:23:30.909934+0000","last_fullsized":"2024-03-06T07:23:30.909934+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:56.840142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.051765+0000","last_change":"2024-03-06T07:23:29.048359+0000","last_active":"2024-03-06T07:23:30.051765+0000","last_peered":"2024-03-06T07:23:30.051765+0000","last_clean":"2024-03-06T07:23:30.051765+0000","last_became_active":"2024-03-06T07:23:29.048093+0000","last_became_peered":"2024-03-06T07:23:29.048093+0000","last_unstale":"2024-03-06T07:23:30.051765+0000","last_undegraded":"2024-03-06T07:23:30.051765+0000","last_fullsized":"2024-03-06T07:23:30.051765+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:37.381518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.617560+0000","last_change":"2024-03-06T07:23:28.048953+0000","last_active":"2024-03-06T07:23:30.617560+0000","last_peered":"2024-03-06T07:23:30.617560+0000","last_clean":"2024-03-06T07:23:30.617560+0000","last_became_active":"2024-03-06T07:23:28.048677+0000","last_became_peered":"2024-03-06T07:23:28.048677+0000","last_unstale":"2024-03-06T07:23:30.617560+0000","last_undegraded":"2024-03-06T07:23:30.617560+0000","last_fullsized":"2024-03-06T07:23:30.617560+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:04.329701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.909800+0000","last_change":"2024-03-06T07:23:29.057146+0000","last_active":"2024-03-06T07:23:30.909800+0000","last_peered":"2024-03-06T07:23:30.909800+0000","last_clean":"2024-03-06T07:23:30.909800+0000","last_became_active":"2024-03-06T07:23:29.056662+0000","last_became_peered":"2024-03-06T07:23:29.056662+0000","last_unstale":"2024-03-06T07:23:30.909800+0000","last_undegraded":"2024-03-06T07:23:30.909800+0000","last_fullsized":"2024-03-06T07:23:30.909800+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:05.333397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050385+0000","last_change":"2024-03-06T07:23:28.054899+0000","last_active":"2024-03-06T07:23:30.050385+0000","last_peered":"2024-03-06T07:23:30.050385+0000","last_clean":"2024-03-06T07:23:30.050385+0000","last_became_active":"2024-03-06T07:23:28.054603+0000","last_became_peered":"2024-03-06T07:23:28.054603+0000","last_unstale":"2024-03-06T07:23:30.050385+0000","last_undegraded":"2024-03-06T07:23:30.050385+0000","last_fullsized":"2024-03-06T07:23:30.050385+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:56.840142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.050769+0000","last_change":"2024-03-06T07:23:29.051642+0000","last_active":"2024-03-06T07:23:30.050769+0000","last_peered":"2024-03-06T07:23:30.050769+0000","last_clean":"2024-03-06T07:23:30.050769+0000","last_became_active":"2024-03-06T07:23:29.051293+0000","last_became_peered":"2024-03-06T07:23:29.051293+0000","last_unstale":"2024-03-06T07:23:30.050769+0000","last_undegraded":"2024-03-06T07:23:30.050769+0000","last_fullsized":"2024-03-06T07:23:30.050769+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:28.025632+0000","last_clean_scrub_stamp":"2024-03-06T07:23:28.025632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:24.403922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.052176+0000","last_change":"2024-03-06T07:23:28.044302+0000","last_active":"2024-03-06T07:23:30.052176+0000","last_peered":"2024-03-06T07:23:30.052176+0000","last_clean":"2024-03-06T07:23:30.052176+0000","last_became_active":"2024-03-06T07:23:28.044034+0000","last_became_peered":"2024-03-06T07:23:28.044034+0000","last_unstale":"2024-03-06T07:23:30.052176+0000","last_undegraded":"2024-03-06T07:23:30.052176+0000","last_fullsized":"2024-03-06T07:23:30.052176+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:27.021935+0000","last_clean_scrub_stamp":"2024-03-06T07:23:27.021935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:22.837290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":215,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:23:30.910078+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:23:30.910078+0000","last_peered":"2024-03-06T07:23:30.910078+0000","last_clean":"2024-03-06T07:23:30.910078+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:23:30.910078+0000","last_undegraded":"2024-03-06T07:23:30.910078+0000","last_fullsized":"2024-03-06T07:23:30.910078+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1114112,"data_stored":799273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574874,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":536576,"data_stored":215787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":532480,"data_stored":215185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":540672,"data_stored":216727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":540672,"data_stored":215753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":532480,"data_stored":210449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31692,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340148,"statfs":{"total":96636764160,"available":96604311552,"internally_reserved":0,"allocated":520192,"data_stored":208885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":805517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:33.398 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:33.398 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:23:33.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.539+0000 7f4373bf7640 1 Processor -- start 2024-03-06T07:23:33.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.539+0000 7f4373bf7640 1 -- start start 2024-03-06T07:23:33.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4373bf7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f436c15fc30 0x7f436c0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4373bf7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f436c0b4d10 0x7f436c0cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4373bf7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436c1600a0 con 0x7f436c0d06e0 2024-03-06T07:23:33.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4373bf7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f436c0ced70 con 0x7f436c15fc30 2024-03-06T07:23:33.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4373bf7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f436c0d47f0 con 0x7f436c0b4d10 2024-03-06T07:23:33.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4371192640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f436c15fc30 0x7f436c0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4371192640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f436c15fc30 0x7f436c0b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56602/0 (socket says 172.21.15.59:56602) 2024-03-06T07:23:33.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4371192640 1 -- 172.21.15.59:0/38838593 learned_addr learned my addr 172.21.15.59:0/38838593 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:33.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4372194640 1 --2- 172.21.15.59:0/38838593 >> v2:172.21.15.164:3300/0 conn(0x7f436c0b4d10 0x7f436c0cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4371192640 1 -- 172.21.15.59:0/38838593 >> v2:172.21.15.164:3300/0 conn(0x7f436c0b4d10 msgr2=0x7f436c0cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4371192640 1 --2- 172.21.15.59:0/38838593 >> v2:172.21.15.164:3300/0 conn(0x7f436c0b4d10 0x7f436c0cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4371192640 1 -- 172.21.15.59:0/38838593 >> v1:172.21.15.164:6789/0 conn(0x7f436c0d06e0 legacy=0x7f436c0d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.540+0000 7f4371192640 1 -- 172.21.15.59:0/38838593 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c0d1ab0 con 0x7f436c15fc30 2024-03-06T07:23:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.541+0000 7f4371192640 1 --2- 172.21.15.59:0/38838593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f436c15fc30 0x7f436c0b47d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f436c0d1520 tx=0x7f436002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867b517afa59d40b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.541+0000 7f4370991640 1 -- 172.21.15.59:0/38838593 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360034070 con 0x7f436c15fc30 2024-03-06T07:23:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.541+0000 7f4370991640 1 -- 172.21.15.59:0/38838593 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4360030860 con 0x7f436c15fc30 2024-03-06T07:23:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.541+0000 7f4370991640 1 -- 172.21.15.59:0/38838593 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360030c30 con 0x7f436c15fc30 2024-03-06T07:23:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.542+0000 7f4373bf7640 1 -- 172.21.15.59:0/38838593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f436c15fc30 msgr2=0x7f436c0b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.542+0000 7f4373bf7640 1 --2- 172.21.15.59:0/38838593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f436c15fc30 0x7f436c0b47d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f436c0d1520 tx=0x7f436002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.542+0000 7f4373bf7640 1 -- 172.21.15.59:0/38838593 shutdown_connections 2024-03-06T07:23:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.542+0000 7f4373bf7640 1 --2- 172.21.15.59:0/38838593 >> v2:172.21.15.164:3300/0 conn(0x7f436c0b4d10 0x7f436c0cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.542+0000 7f4373bf7640 1 --2- 172.21.15.59:0/38838593 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f436c15fc30 0x7f436c0b47d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.542+0000 7f4373bf7640 1 -- 172.21.15.59:0/38838593 >> 172.21.15.59:0/38838593 conn(0x7f436c0d3cd0 msgr2=0x7f436c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.542+0000 7f4373bf7640 1 -- 172.21.15.59:0/38838593 shutdown_connections 2024-03-06T07:23:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.542+0000 7f4373bf7640 1 -- 172.21.15.59:0/38838593 wait complete. 2024-03-06T07:23:33.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.543+0000 7f4373bf7640 1 Processor -- start 2024-03-06T07:23:33.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.543+0000 7f4373bf7640 1 -- start start 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4373bf7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f436c0b4d10 0x7f436c0b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4373bf7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436c0d06e0 0x7f436c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4373bf7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f436c15fc30 0x7f436c15f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4373bf7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c0ce990 con 0x7f436c15fc30 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4373bf7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c0d4ab0 con 0x7f436c0b4d10 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4373bf7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436c0b44d0 con 0x7f436c0d06e0 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4372194640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f436c15fc30 0x7f436c15f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4372194640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f436c15fc30 0x7f436c15f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56606/0 (socket says 172.21.15.59:56606) 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4372194640 1 -- 172.21.15.59:0/2145104781 learned_addr learned my addr 172.21.15.59:0/2145104781 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4371993640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f436c0b4d10 0x7f436c0b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4371192640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436c0d06e0 0x7f436c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4372194640 1 -- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436c0d06e0 msgr2=0x7f436c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4372194640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436c0d06e0 0x7f436c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4372194640 1 -- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f436c0b4d10 msgr2=0x7f436c0b7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4372194640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f436c0b4d10 0x7f436c0b7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4372194640 1 -- 172.21.15.59:0/2145104781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c0cdd00 con 0x7f436c15fc30 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f4372194640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f436c15fc30 0x7f436c15f170 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f435c00efc0 tx=0x7f435c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.544+0000 7f436a7fc640 1 -- 172.21.15.59:0/2145104781 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c018020 con 0x7f436c15fc30 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.545+0000 7f436a7fc640 1 -- 172.21.15.59:0/2145104781 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435c014480 con 0x7f436c15fc30 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.545+0000 7f4373bf7640 1 -- 172.21.15.59:0/2145104781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436c0b8510 con 0x7f436c15fc30 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.545+0000 7f436a7fc640 1 -- 172.21.15.59:0/2145104781 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c014870 con 0x7f436c15fc30 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.545+0000 7f4373bf7640 1 -- 172.21.15.59:0/2145104781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436c0b8ad0 con 0x7f436c15fc30 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.546+0000 7f4373bf7640 1 -- 172.21.15.59:0/2145104781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436c0d2a00 con 0x7f436c15fc30 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.546+0000 7f436a7fc640 1 -- 172.21.15.59:0/2145104781 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f435c005d10 con 0x7f436c15fc30 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.546+0000 7f436a7fc640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f435806b4d0 0x7f435806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.547+0000 7f436a7fc640 1 -- 172.21.15.59:0/2145104781 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f435c08ac00 con 0x7f436c15fc30 2024-03-06T07:23:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.547+0000 7f4371993640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f435806b4d0 0x7f435806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.547+0000 7f4371993640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f435806b4d0 0x7f435806d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f43640016e0 tx=0x7f4364009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.551+0000 7f436a7fc640 1 -- 172.21.15.59:0/2145104781 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f436c0d2a00 con 0x7f436c15fc30 2024-03-06T07:23:33.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.802+0000 7f4373bf7640 1 -- 172.21.15.59:0/2145104781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f436c068ff0 con 0x7f436c15fc30 2024-03-06T07:23:33.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.804+0000 7f436a7fc640 1 -- 172.21.15.59:0/2145104781 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f435c091080 con 0x7f436c15fc30 2024-03-06T07:23:33.806 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-06T07:23:33.806 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA== 2024-03-06T07:23:33.806 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-03-06T07:23:33.806 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-03-06T07:23:33.806 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:33.806 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-03-06T07:23:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 -- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f435806b4d0 msgr2=0x7f435806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f435806b4d0 0x7f435806d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f43640016e0 tx=0x7f4364009040 comp rx=0 tx=0).stop 2024-03-06T07:23:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 -- 172.21.15.59:0/2145104781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f436c15fc30 msgr2=0x7f436c15f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f436c15fc30 0x7f436c15f170 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f435c00efc0 tx=0x7f435c00c490 comp rx=0 tx=0).stop 2024-03-06T07:23:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 -- 172.21.15.59:0/2145104781 shutdown_connections 2024-03-06T07:23:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f435806b4d0 0x7f435806d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f436c15fc30 0x7f436c15f170 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436c0d06e0 0x7f436c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 --2- 172.21.15.59:0/2145104781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f436c0b4d10 0x7f436c0b7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 -- 172.21.15.59:0/2145104781 >> 172.21.15.59:0/2145104781 conn(0x7f436c0d3cd0 msgr2=0x7f436c06d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 -- 172.21.15.59:0/2145104781 shutdown_connections 2024-03-06T07:23:33.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.806+0000 7f4373bf7640 1 -- 172.21.15.59:0/2145104781 wait complete. 2024-03-06T07:23:33.818 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:33.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.918+0000 7f1f7e156640 1 Processor -- start 2024-03-06T07:23:33.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.918+0000 7f1f7e156640 1 -- start start 2024-03-06T07:23:33.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f7e156640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1f780d02c0 0x7f1f780cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f7e156640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f780cef50 con 0x7f1f780d06a0 2024-03-06T07:23:33.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f7e156640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7815c620 con 0x7f1f780b6550 2024-03-06T07:23:33.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f7e156640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f780b50b0 con 0x7f1f780d02c0 2024-03-06T07:23:33.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f777fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1f780d02c0 0x7f1f780cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f76ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1f780d06a0 0x7f1f780ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34808/0 (socket says 172.21.15.59:34808) 2024-03-06T07:23:33.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f777fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1f780d02c0 0x7f1f780cf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56612/0 (socket says 172.21.15.59:56612) 2024-03-06T07:23:33.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f76ffd640 1 -- 172.21.15.59:0/1649292435 learned_addr learned my addr 172.21.15.59:0/1649292435 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:33.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f777fe640 1 -- 172.21.15.59:0/1649292435 >> v1:172.21.15.164:6789/0 conn(0x7f1f780b6550 legacy=0x7f1f780ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f777fe640 1 -- 172.21.15.59:0/1649292435 >> v1:172.21.15.59:6789/0 conn(0x7f1f780d06a0 legacy=0x7f1f780ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f767fc640 1 -- 172.21.15.59:0/1649292435 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384869868 0 0) 0x7f1f780cef50 con 0x7f1f780d06a0 2024-03-06T07:23:33.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.919+0000 7f1f777fe640 1 -- 172.21.15.59:0/1649292435 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f780cd800 con 0x7f1f780d02c0 2024-03-06T07:23:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.920+0000 7f1f777fe640 1 --2- 172.21.15.59:0/1649292435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f780d02c0 0x7f1f780cf850 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1f680097a0 tx=0x7f1f6802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27dbe9e70e437df5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.920+0000 7f1f767fc640 1 -- 172.21.15.59:0/1649292435 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68034070 con 0x7f1f780d02c0 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.920+0000 7f1f767fc640 1 -- 172.21.15.59:0/1649292435 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68030610 con 0x7f1f780d02c0 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.920+0000 7f1f767fc640 1 -- 172.21.15.59:0/1649292435 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f680309e0 con 0x7f1f780d02c0 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.920+0000 7f1f7e156640 1 -- 172.21.15.59:0/1649292435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f780d02c0 msgr2=0x7f1f780cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.920+0000 7f1f7e156640 1 --2- 172.21.15.59:0/1649292435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f780d02c0 0x7f1f780cf850 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1f680097a0 tx=0x7f1f6802b960 comp rx=0 tx=0).stop 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.921+0000 7f1f7e156640 1 -- 172.21.15.59:0/1649292435 shutdown_connections 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.921+0000 7f1f7e156640 1 --2- 172.21.15.59:0/1649292435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f780d02c0 0x7f1f780cf850 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.921+0000 7f1f7e156640 1 -- 172.21.15.59:0/1649292435 >> 172.21.15.59:0/1649292435 conn(0x7f1f780b35a0 msgr2=0x7f1f780b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.921+0000 7f1f7e156640 1 -- 172.21.15.59:0/1649292435 shutdown_connections 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.921+0000 7f1f7e156640 1 -- 172.21.15.59:0/1649292435 wait complete. 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.922+0000 7f1f7e156640 1 Processor -- start 2024-03-06T07:23:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.922+0000 7f1f7e156640 1 -- start start 2024-03-06T07:23:33.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f7e156640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f780b6550 0x7f1f781f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f7e156640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f780d02c0 0x7f1f780c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f7e156640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f780d06a0 0x7f1f780c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f7e156640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780cd0d0 con 0x7f1f780d06a0 2024-03-06T07:23:33.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f7e156640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780b4d40 con 0x7f1f780b6550 2024-03-06T07:23:33.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f7e156640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780cd250 con 0x7f1f780d02c0 2024-03-06T07:23:33.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f77fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f780d06a0 0x7f1f780c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f77fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f780d06a0 0x7f1f780c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56624/0 (socket says 172.21.15.59:56624) 2024-03-06T07:23:33.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f77fff640 1 -- 172.21.15.59:0/1181908456 learned_addr learned my addr 172.21.15.59:0/1181908456 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f777fe640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f780b6550 0x7f1f781f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f76ffd640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f780d02c0 0x7f1f780c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f77fff640 1 -- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f780d02c0 msgr2=0x7f1f780c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f77fff640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f780d02c0 0x7f1f780c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f77fff640 1 -- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f780b6550 msgr2=0x7f1f781f6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f77fff640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f780b6550 0x7f1f781f6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f77fff640 1 -- 172.21.15.59:0/1181908456 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f781f7280 con 0x7f1f780d06a0 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f77fff640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f780d06a0 0x7f1f780c5ae0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000c7d0 tx=0x7f1f7000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f5ffff640 1 -- 172.21.15.59:0/1181908456 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f70003b70 con 0x7f1f780d06a0 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f5ffff640 1 -- 172.21.15.59:0/1181908456 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f70003d30 con 0x7f1f780d06a0 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f7e156640 1 -- 172.21.15.59:0/1181908456 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7806cfd0 con 0x7f1f780d06a0 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f5ffff640 1 -- 172.21.15.59:0/1181908456 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7000aa90 con 0x7f1f780d06a0 2024-03-06T07:23:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.923+0000 7f1f7e156640 1 -- 172.21.15.59:0/1181908456 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7806d590 con 0x7f1f780d06a0 2024-03-06T07:23:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.925+0000 7f1f7e156640 1 -- 172.21.15.59:0/1181908456 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7804f330 con 0x7f1f780d06a0 2024-03-06T07:23:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.925+0000 7f1f5ffff640 1 -- 172.21.15.59:0/1181908456 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f7000ac30 con 0x7f1f780d06a0 2024-03-06T07:23:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.925+0000 7f1f5ffff640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f6406b500 0x7f1f6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.925+0000 7f1f5ffff640 1 -- 172.21.15.59:0/1181908456 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f70089fd0 con 0x7f1f780d06a0 2024-03-06T07:23:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.925+0000 7f1f777fe640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f6406b500 0x7f1f6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.926+0000 7f1f777fe640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f6406b500 0x7f1f6406d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1f68036000 tx=0x7f1f68008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:33.929+0000 7f1f5ffff640 1 -- 172.21.15.59:0/1181908456 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f781f7280 con 0x7f1f780d06a0 2024-03-06T07:23:34.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.149+0000 7f1f7e156640 1 -- 172.21.15.59:0/1181908456 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f780cf3d0 con 0x7f1f780d06a0 2024-03-06T07:23:34.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.150+0000 7f1f5ffff640 1 -- 172.21.15.59:0/1181908456 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f1f70058800 con 0x7f1f780d06a0 2024-03-06T07:23:34.152 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:34.152 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5557,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/690078434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":690078434},{"type":"v1","addr":"172.21.15.164:6836","nonce":690078434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1345698247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1345698247},{"type":"v1","addr":"172.21.15.59:6833","nonce":1345698247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5685,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3875776524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3875776524},{"type":"v1","addr":"172.21.15.59:6835","nonce":3875776524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:28.400413+0000","modified":"2024-03-06T07:23:31.397689+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/3749315375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3749315375},{"type":"v1","addr":"172.21.15.164:6837","nonce":3749315375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-06T07:23:34.152 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 20 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 -- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f6406b500 msgr2=0x7f1f6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f6406b500 0x7f1f6406d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1f68036000 tx=0x7f1f68008030 comp rx=0 tx=0).stop 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 -- 172.21.15.59:0/1181908456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f780d06a0 msgr2=0x7f1f780c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f780d06a0 0x7f1f780c5ae0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000c7d0 tx=0x7f1f7000cc90 comp rx=0 tx=0).stop 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 -- 172.21.15.59:0/1181908456 shutdown_connections 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f6406b500 0x7f1f6406d9c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f780d06a0 0x7f1f780c5ae0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f780d02c0 0x7f1f780c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 --2- 172.21.15.59:0/1181908456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f780b6550 0x7f1f781f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 -- 172.21.15.59:0/1181908456 >> 172.21.15.59:0/1181908456 conn(0x7f1f780b35a0 msgr2=0x7f1f7806f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 -- 172.21.15.59:0/1181908456 shutdown_connections 2024-03-06T07:23:34.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.152+0000 7f1f7e156640 1 -- 172.21.15.59:0/1181908456 wait complete. 2024-03-06T07:23:34.569 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:28.400413+0000', 'modified': '2024-03-06T07:23:31.397689+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5697}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5697': {'gid': 5697, 'name': 'b', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.164:6837/3749315375', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 3749315375}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 3749315375}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5697, 'qdb_cluster': [5697]} 2024-03-06T07:23:34.569 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:34.570 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.674+0000 7ff55a53e640 1 Processor -- start 2024-03-06T07:23:34.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.674+0000 7ff55a53e640 1 -- start start 2024-03-06T07:23:34.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.675+0000 7ff55a53e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff554162970 0x7ff5540d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.675+0000 7ff55a53e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5540d22c0 con 0x7ff5540b4460 2024-03-06T07:23:34.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.675+0000 7ff55a53e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5540d25a0 con 0x7ff554161cf0 2024-03-06T07:23:34.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.675+0000 7ff55a53e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff554161b00 con 0x7ff554162970 2024-03-06T07:23:34.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.675+0000 7ff553fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff5540b4460 0x7ff5540b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34812/0 (socket says 172.21.15.59:34812) 2024-03-06T07:23:34.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.675+0000 7ff553fff640 1 -- 172.21.15.59:0/1879353662 learned_addr learned my addr 172.21.15.59:0/1879353662 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:34.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.675+0000 7ff5537fe640 1 --2- 172.21.15.59:0/1879353662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff554162970 0x7ff5540d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.675+0000 7ff552ffd640 1 -- 172.21.15.59:0/1879353662 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551064280 0 0) 0x7ff5540d22c0 con 0x7ff5540b4460 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.676+0000 7ff552ffd640 1 -- 172.21.15.59:0/1879353662 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff540003520 con 0x7ff5540b4460 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.676+0000 7ff552ffd640 1 -- 172.21.15.59:0/1879353662 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769034592 0 0) 0x7ff5540d25a0 con 0x7ff554161cf0 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.676+0000 7ff552ffd640 1 -- 172.21.15.59:0/1879353662 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5540d22c0 con 0x7ff554161cf0 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.676+0000 7ff5537fe640 1 -- 172.21.15.59:0/1879353662 >> v1:172.21.15.164:6789/0 conn(0x7ff554161cf0 legacy=0x7ff5540cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.676+0000 7ff5537fe640 1 -- 172.21.15.59:0/1879353662 >> v1:172.21.15.59:6789/0 conn(0x7ff5540b4460 legacy=0x7ff5540b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.676+0000 7ff5537fe640 1 -- 172.21.15.59:0/1879353662 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5540d2880 con 0x7ff554162970 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.676+0000 7ff5537fe640 1 --2- 172.21.15.59:0/1879353662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff554162970 0x7ff5540d4800 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0097d0 tx=0x7ff53c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=814e73a205dba302 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.677+0000 7ff552ffd640 1 -- 172.21.15.59:0/1879353662 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c035020 con 0x7ff554162970 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.677+0000 7ff552ffd640 1 -- 172.21.15.59:0/1879353662 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53c031890 con 0x7ff554162970 2024-03-06T07:23:34.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.677+0000 7ff552ffd640 1 -- 172.21.15.59:0/1879353662 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c031c60 con 0x7ff554162970 2024-03-06T07:23:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.677+0000 7ff55a53e640 1 -- 172.21.15.59:0/1879353662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff554162970 msgr2=0x7ff5540d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.677+0000 7ff55a53e640 1 --2- 172.21.15.59:0/1879353662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff554162970 0x7ff5540d4800 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0097d0 tx=0x7ff53c02b930 comp rx=0 tx=0).stop 2024-03-06T07:23:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.677+0000 7ff55a53e640 1 -- 172.21.15.59:0/1879353662 shutdown_connections 2024-03-06T07:23:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.677+0000 7ff55a53e640 1 --2- 172.21.15.59:0/1879353662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff554162970 0x7ff5540d4800 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.677+0000 7ff55a53e640 1 -- 172.21.15.59:0/1879353662 >> 172.21.15.59:0/1879353662 conn(0x7ff5540d3ce0 msgr2=0x7ff5540d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.677+0000 7ff55a53e640 1 -- 172.21.15.59:0/1879353662 shutdown_connections 2024-03-06T07:23:34.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.677+0000 7ff55a53e640 1 -- 172.21.15.59:0/1879353662 wait complete. 2024-03-06T07:23:34.688 INFO:tasks.ceph.mds.b.smithi164.stderr:2024-03-06T07:23:34.684+0000 7fecbef37640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:34.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.678+0000 7ff55a53e640 1 Processor -- start 2024-03-06T07:23:34.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff55a53e640 1 -- start start 2024-03-06T07:23:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff55a53e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5540b4460 0x7ff554156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff55a53e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff554161cf0 0x7ff55415e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff55a53e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff554162970 0x7ff55415ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff55a53e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540ce950 con 0x7ff5540b4460 2024-03-06T07:23:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff55a53e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540cecc0 con 0x7ff554162970 2024-03-06T07:23:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff55a53e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540b4840 con 0x7ff554161cf0 2024-03-06T07:23:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff553fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5540b4460 0x7ff554156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff553fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5540b4460 0x7ff554156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56640/0 (socket says 172.21.15.59:56640) 2024-03-06T07:23:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff553fff640 1 -- 172.21.15.59:0/1680867521 learned_addr learned my addr 172.21.15.59:0/1680867521 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff5537fe640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff554161cf0 0x7ff55415e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.679+0000 7ff558adb640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff554162970 0x7ff55415ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.680+0000 7ff553fff640 1 -- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff554161cf0 msgr2=0x7ff55415e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.680+0000 7ff553fff640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff554161cf0 0x7ff55415e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.680+0000 7ff553fff640 1 -- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff554162970 msgr2=0x7ff55415ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.680+0000 7ff553fff640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff554162970 0x7ff55415ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.680+0000 7ff553fff640 1 -- 172.21.15.59:0/1680867521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5540b7830 con 0x7ff5540b4460 2024-03-06T07:23:34.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.680+0000 7ff553fff640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5540b4460 0x7ff554156110 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff54400c410 tx=0x7ff54400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.680+0000 7ff550ff9640 1 -- 172.21.15.59:0/1680867521 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544017070 con 0x7ff5540b4460 2024-03-06T07:23:34.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.680+0000 7ff550ff9640 1 -- 172.21.15.59:0/1680867521 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff544005ce0 con 0x7ff5540b4460 2024-03-06T07:23:34.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.680+0000 7ff55a53e640 1 -- 172.21.15.59:0/1680867521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5540caf50 con 0x7ff5540b4460 2024-03-06T07:23:34.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.680+0000 7ff550ff9640 1 -- 172.21.15.59:0/1680867521 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5440072d0 con 0x7ff5540b4460 2024-03-06T07:23:34.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.680+0000 7ff55a53e640 1 -- 172.21.15.59:0/1680867521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5540cb510 con 0x7ff5540b4460 2024-03-06T07:23:34.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.682+0000 7ff550ff9640 1 -- 172.21.15.59:0/1680867521 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff54401a070 con 0x7ff5540b4460 2024-03-06T07:23:34.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.682+0000 7ff550ff9640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53406b4d0 0x7ff53406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.682+0000 7ff550ff9640 1 -- 172.21.15.59:0/1680867521 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff544089470 con 0x7ff5540b4460 2024-03-06T07:23:34.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.682+0000 7ff5537fe640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53406b4d0 0x7ff53406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.683+0000 7ff5537fe640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53406b4d0 0x7ff53406d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff53c036000 tx=0x7ff53c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.683+0000 7ff55d20a740 1 Processor -- start 2024-03-06T07:23:34.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.683+0000 7ff55d20a740 1 -- start start 2024-03-06T07:23:34.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759a8870 0x55ad759a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759ac8c0 0x55ad759ae360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759d8600 0x55ad759d8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff55d20a740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad759126b0 con 0x55ad759a8870 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff55d20a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad758bba60 con 0x55ad759ac8c0 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff55d20a740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad7590e750 con 0x55ad759d8600 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff553fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759a8870 0x55ad759a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff553fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759a8870 0x55ad759a8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56644/0 (socket says 172.21.15.59:56644) 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff553fff640 1 -- 172.21.15.59:0/2064279249 learned_addr learned my addr 172.21.15.59:0/2064279249 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff558adb640 1 --2- 172.21.15.59:0/2064279249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759d8600 0x55ad759d8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff5537fe640 1 --2- 172.21.15.59:0/2064279249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759ac8c0 0x55ad759ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff553fff640 1 -- 172.21.15.59:0/2064279249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759d8600 msgr2=0x55ad759d8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff553fff640 1 --2- 172.21.15.59:0/2064279249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759d8600 0x55ad759d8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff553fff640 1 -- 172.21.15.59:0/2064279249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759ac8c0 msgr2=0x55ad759ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff553fff640 1 --2- 172.21.15.59:0/2064279249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759ac8c0 0x55ad759ae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff553fff640 1 -- 172.21.15.59:0/2064279249 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad758d9f30 con 0x55ad759a8870 2024-03-06T07:23:34.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff553fff640 1 --2- 172.21.15.59:0/2064279249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759a8870 0x55ad759a8c50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff544087a50 tx=0x7ff544057df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4db3bee5741651 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.684+0000 7ff549ffb640 1 -- 172.21.15.59:0/2064279249 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54405e070 con 0x55ad759a8870 2024-03-06T07:23:34.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.685+0000 7ff549ffb640 1 -- 172.21.15.59:0/2064279249 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff544056030 con 0x55ad759a8870 2024-03-06T07:23:34.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.685+0000 7ff549ffb640 1 -- 172.21.15.59:0/2064279249 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54405ff00 con 0x55ad759a8870 2024-03-06T07:23:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.685+0000 7ff55d20a740 1 -- 172.21.15.59:0/2064279249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759a8870 msgr2=0x55ad759a8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.685+0000 7ff55d20a740 1 --2- 172.21.15.59:0/2064279249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759a8870 0x55ad759a8c50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff544087a50 tx=0x7ff544057df0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.685+0000 7ff55d20a740 1 -- 172.21.15.59:0/2064279249 shutdown_connections 2024-03-06T07:23:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.685+0000 7ff55d20a740 1 --2- 172.21.15.59:0/2064279249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759d8600 0x55ad759d8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.685+0000 7ff55d20a740 1 --2- 172.21.15.59:0/2064279249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759ac8c0 0x55ad759ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.685+0000 7ff55d20a740 1 --2- 172.21.15.59:0/2064279249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759a8870 0x55ad759a8c50 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.685+0000 7ff55d20a740 1 -- 172.21.15.59:0/2064279249 >> 172.21.15.59:0/2064279249 conn(0x55ad759aa8f0 msgr2=0x55ad759ac1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.685+0000 7ff55d20a740 1 -- 172.21.15.59:0/2064279249 shutdown_connections 2024-03-06T07:23:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.685+0000 7ff55d20a740 1 -- 172.21.15.59:0/2064279249 wait complete. 2024-03-06T07:23:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.686+0000 7ff55d20a740 1 -- start start 2024-03-06T07:23:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.686+0000 7ff55d20a740 1 Processor -- start 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759d8600 0x55ad759dc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759db240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759db780 0x55ad75a06d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff55d20a740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad759acd50 con 0x55ad759da150 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff55d20a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad758ceea0 con 0x55ad759d8600 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff55d20a740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad759ad030 con 0x55ad759db780 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff55d20a740 10 client.0 _ll_register_callbacks cb 0x55ad758ce870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff5537fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759db240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff553fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759d8600 0x55ad759dc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff553fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759d8600 0x55ad759dc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:39480/0 (socket says 172.21.15.59:39480) 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff558adb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759db780 0x55ad75a06d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff553fff640 1 -- 172.21.15.59:0/3665940118 learned_addr learned my addr 172.21.15.59:0/3665940118 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff5537fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759db240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56648/0 (socket says 172.21.15.59:56648) 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff553fff640 1 -- 172.21.15.59:0/3665940118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759db780 msgr2=0x55ad75a06d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff553fff640 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759db780 0x55ad75a06d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff553fff640 1 -- 172.21.15.59:0/3665940118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 msgr2=0x55ad759db240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff553fff640 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759db240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff553fff640 1 -- 172.21.15.59:0/3665940118 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad758cf270 con 0x55ad759d8600 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff5537fe640 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759db240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff553fff640 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759d8600 0x55ad759dc3c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff544089790 tx=0x7ff544000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ecb9c5c17dff013 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff532ffd640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff532ffd640 1 -- 172.21.15.59:0/3665940118 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544060100 con 0x55ad759d8600 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.687+0000 7ff532ffd640 1 -- 172.21.15.59:0/3665940118 <== mon.1 v2:172.21.15.164:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff544007470 con 0x55ad759d8600 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff55d20a740 1 -- 172.21.15.59:0/3665940118 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff544056030 con 0x55ad759d8600 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff532ffd640 1 -- 172.21.15.59:0/3665940118 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff54405e070 con 0x55ad759d8600 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff55d20a740 10 client.5662 fetch_fsmap learned FSMap version 20 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff55d20a740 1 -- 172.21.15.59:0/3665940118 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x55ad756272d0 con 0x55ad759d8600 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff532ffd640 1 -- 172.21.15.59:0/3665940118 <== mon.1 v2:172.21.15.164:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff544056030 con 0x55ad759d8600 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff532ffd640 1 -- 172.21.15.59:0/3665940118 <== mon.1 v2:172.21.15.164:3300/0 5 ==== fsmap(e 20) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff544076b50 con 0x55ad759d8600 2024-03-06T07:23:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff55d20a740 10 client.5662 fetch_fsmap finished waiting for FSMap version 20 2024-03-06T07:23:34.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff55d20a740 10 client.5662 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:34.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff55d20a740 20 client.5662 populate_metadata read hostname 'smithi059' 2024-03-06T07:23:34.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff55d20a740 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.164:6835/3749315375,v1:172.21.15.164:6837/3749315375] conn(0x55ad759eed90 0x55ad759ef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff55d20a740 4 client.5662 mds_command: new command op to 5697 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:34.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff55d20a740 1 -- 172.21.15.59:0/3665940118 --> [v2:172.21.15.164:6835/3749315375,v1:172.21.15.164:6837/3749315375] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad7591fbf0 con 0x55ad759eed90 2024-03-06T07:23:34.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.688+0000 7ff5537fe640 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.164:6835/3749315375,v1:172.21.15.164:6837/3749315375] conn(0x55ad759eed90 0x55ad759ef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff5537fe640 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.164:6835/3749315375,v1:172.21.15.164:6837/3749315375] conn(0x55ad759eed90 0x55ad759ef1b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c47b2481cf4a2bf6 server_cookie=6c2c3390632b9079 in_seq=0 out_seq=0 2024-03-06T07:23:34.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff532ffd640 10 client.5662 ms_handle_connect on v2:172.21.15.164:6835/3749315375 2024-03-06T07:23:34.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff532ffd640 1 -- 172.21.15.59:0/3665940118 <== mds.0 v2:172.21.15.164:6835/3749315375 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ad7591fbf0 con 0x55ad759eed90 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff532ffd640 10 client.5662 handle_command_reply: tid=0 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff55d20a740 1 client.5662 shutdown 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff55d20a740 1 -- 172.21.15.59:0/3665940118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759d8600 msgr2=0x55ad759dc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff55d20a740 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759d8600 0x55ad759dc3c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff544089790 tx=0x7ff544000ed0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff55d20a740 1 -- 172.21.15.59:0/3665940118 shutdown_connections 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff55d20a740 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.164:6835/3749315375,v1:172.21.15.164:6837/3749315375] conn(0x55ad759eed90 0x55ad759ef1b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff55d20a740 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759db780 0x55ad75a06d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff55d20a740 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759db240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff55d20a740 1 --2- 172.21.15.59:0/3665940118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759d8600 0x55ad759dc3c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff532ffd640 0 client.5662 ms_handle_reset on v2:172.21.15.164:6835/3749315375 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff55d20a740 1 -- 172.21.15.59:0/3665940118 >> 172.21.15.59:0/3665940118 conn(0x55ad759a8870 msgr2=0x55ad759d65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff55d20a740 1 -- 172.21.15.59:0/3665940118 shutdown_connections 2024-03-06T07:23:34.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.689+0000 7ff55d20a740 1 -- 172.21.15.59:0/3665940118 wait complete. 2024-03-06T07:23:34.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.690+0000 7ff55d20a740 20 client.5662 trim_cache size 0 max 16384 2024-03-06T07:23:34.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.706+0000 7ff55d20a740 1 Processor -- start 2024-03-06T07:23:34.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.706+0000 7ff55d20a740 1 -- start start 2024-03-06T07:23:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759d8600 0x55ad759d89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759da150 0x55ad759d8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759db780 0x55ad759d51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff55d20a740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5541ff180 con 0x55ad759d8600 2024-03-06T07:23:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff55d20a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540cca80 con 0x55ad759db780 2024-03-06T07:23:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff55d20a740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5541febb0 con 0x55ad759da150 2024-03-06T07:23:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff553fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759d8600 0x55ad759d89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff553fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759d8600 0x55ad759d89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56658/0 (socket says 172.21.15.59:56658) 2024-03-06T07:23:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff553fff640 1 -- 172.21.15.59:0/1560276428 learned_addr learned my addr 172.21.15.59:0/1560276428 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff5537fe640 1 --2- 172.21.15.59:0/1560276428 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759da150 0x55ad759d8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff558adb640 1 --2- 172.21.15.59:0/1560276428 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759db780 0x55ad759d51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff553fff640 1 -- 172.21.15.59:0/1560276428 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759da150 msgr2=0x55ad759d8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff553fff640 1 --2- 172.21.15.59:0/1560276428 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759da150 0x55ad759d8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff553fff640 1 -- 172.21.15.59:0/1560276428 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759db780 msgr2=0x55ad759d51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff553fff640 1 --2- 172.21.15.59:0/1560276428 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759db780 0x55ad759d51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff553fff640 1 -- 172.21.15.59:0/1560276428 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad756f16b0 con 0x55ad759d8600 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff5537fe640 1 --2- 172.21.15.59:0/1560276428 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759da150 0x55ad759d8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.707+0000 7ff558adb640 1 --2- 172.21.15.59:0/1560276428 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759db780 0x55ad759d51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff553fff640 1 --2- 172.21.15.59:0/1560276428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759d8600 0x55ad759d89e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff544057a30 tx=0x7ff544083000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7334094cee0053e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff5497fa640 1 -- 172.21.15.59:0/1560276428 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544056030 con 0x55ad759d8600 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff5497fa640 1 -- 172.21.15.59:0/1560276428 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54405ff00 con 0x55ad759d8600 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff5497fa640 1 -- 172.21.15.59:0/1560276428 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544007670 con 0x55ad759d8600 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff55d20a740 1 -- 172.21.15.59:0/1560276428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759d8600 msgr2=0x55ad759d89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff55d20a740 1 --2- 172.21.15.59:0/1560276428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759d8600 0x55ad759d89e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff544057a30 tx=0x7ff544083000 comp rx=0 tx=0).stop 2024-03-06T07:23:34.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff55d20a740 1 -- 172.21.15.59:0/1560276428 shutdown_connections 2024-03-06T07:23:34.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff55d20a740 1 --2- 172.21.15.59:0/1560276428 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759db780 0x55ad759d51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff55d20a740 1 --2- 172.21.15.59:0/1560276428 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759da150 0x55ad759d8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff55d20a740 1 --2- 172.21.15.59:0/1560276428 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759d8600 0x55ad759d89e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff55d20a740 1 -- 172.21.15.59:0/1560276428 >> 172.21.15.59:0/1560276428 conn(0x55ad759a8870 msgr2=0x55ad759ac1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff55d20a740 1 -- 172.21.15.59:0/1560276428 shutdown_connections 2024-03-06T07:23:34.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.708+0000 7ff55d20a740 1 -- 172.21.15.59:0/1560276428 wait complete. 2024-03-06T07:23:34.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.709+0000 7ff55d20a740 1 -- start start 2024-03-06T07:23:34.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.709+0000 7ff55d20a740 1 Processor -- start 2024-03-06T07:23:34.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759d8600 0x55ad75a61710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759d89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff55d20a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759db780 0x55ad759d8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff55d20a740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad758ceea0 con 0x55ad759da150 2024-03-06T07:23:34.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff55d20a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad759ad030 con 0x55ad759db780 2024-03-06T07:23:34.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff55d20a740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad759acd50 con 0x55ad759d8600 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff55d20a740 10 client.0 _ll_register_callbacks cb 0x7ff54c001920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff5537fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759d89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff5537fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759d89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56672/0 (socket says 172.21.15.59:56672) 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff5537fe640 1 -- 172.21.15.59:0/1628169989 learned_addr learned my addr 172.21.15.59:0/1628169989 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff553fff640 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759d8600 0x55ad75a61710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff558adb640 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759db780 0x55ad759d8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff5537fe640 1 -- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759d8600 msgr2=0x55ad75a61710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff5537fe640 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759d8600 0x55ad75a61710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff5537fe640 1 -- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759db780 msgr2=0x55ad759d8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff5537fe640 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759db780 0x55ad759d8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff5537fe640 1 -- 172.21.15.59:0/1628169989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad759d9800 con 0x55ad759da150 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff5537fe640 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759d89e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0097d0 tx=0x7ff53c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97b87a5942255e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.710+0000 7ff532ffd640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff532ffd640 1 -- 172.21.15.59:0/1628169989 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c035040 con 0x55ad759da150 2024-03-06T07:23:34.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff532ffd640 1 -- 172.21.15.59:0/1628169989 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff53c0561b0 con 0x55ad759da150 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff55d20a740 1 -- 172.21.15.59:0/1628169989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff54405ff00 con 0x55ad759da150 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff532ffd640 1 -- 172.21.15.59:0/1628169989 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff53c064050 con 0x55ad759da150 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff55d20a740 10 client.5844 fetch_fsmap learned FSMap version 20 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff55d20a740 1 -- 172.21.15.59:0/1628169989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x55ad759d9aa0 con 0x55ad759da150 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff532ffd640 1 -- 172.21.15.59:0/1628169989 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54405ff00 con 0x55ad759da150 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff532ffd640 1 -- 172.21.15.59:0/1628169989 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 20) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff53c067020 con 0x55ad759da150 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff55d20a740 10 client.5844 fetch_fsmap finished waiting for FSMap version 20 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff55d20a740 10 client.5844 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff55d20a740 20 client.5844 populate_metadata read hostname 'smithi059' 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff55d20a740 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:6835/3749315375,v1:172.21.15.164:6837/3749315375] conn(0x55ad759f0970 0x55ad759ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff55d20a740 4 client.5844 mds_command: new command op to 5697 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff55d20a740 1 -- 172.21.15.59:0/1628169989 --> [v2:172.21.15.164:6835/3749315375,v1:172.21.15.164:6837/3749315375] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ad75919f40 con 0x55ad759f0970 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.711+0000 7ff553fff640 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:6835/3749315375,v1:172.21.15.164:6837/3749315375] conn(0x55ad759f0970 0x55ad759ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff532ffd640 10 client.5844 ms_handle_connect on v2:172.21.15.164:6835/3749315375 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff553fff640 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:6835/3749315375,v1:172.21.15.164:6837/3749315375] conn(0x55ad759f0970 0x55ad759ad1b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e440febd3d427fb server_cookie=2c14c50bded6c9fc in_seq=0 out_seq=0 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff532ffd640 1 -- 172.21.15.59:0/1628169989 <== mds.0 v2:172.21.15.164:6835/3749315375 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55ad75919f40 con 0x55ad759f0970 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff532ffd640 10 client.5844 handle_command_reply: tid=0 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 1 client.5844 shutdown 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 1 -- 172.21.15.59:0/1628169989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 msgr2=0x55ad759d89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759d89e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0097d0 tx=0x7ff53c004730 comp rx=0 tx=0).stop 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 1 -- 172.21.15.59:0/1628169989 shutdown_connections 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:6835/3749315375,v1:172.21.15.164:6837/3749315375] conn(0x55ad759f0970 0x55ad759ad1b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ad759db780 0x55ad759d8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff532ffd640 0 client.5844 ms_handle_reset on v2:172.21.15.164:6835/3749315375 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55ad759da150 0x55ad759d89e0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 1 --2- 172.21.15.59:0/1628169989 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55ad759d8600 0x55ad75a61710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 1 -- 172.21.15.59:0/1628169989 >> 172.21.15.59:0/1628169989 conn(0x55ad759a8870 msgr2=0x55ad759ae6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 1 -- 172.21.15.59:0/1628169989 shutdown_connections 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 1 -- 172.21.15.59:0/1628169989 wait complete. 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.712+0000 7ff55d20a740 20 client.5844 trim_cache size 0 max 16384 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 5697, 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:23:34.721 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.6786465320000001, 2024-03-06T07:23:34.722 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 20, 2024-03-06T07:23:34.722 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 45, 2024-03-06T07:23:34.722 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 45, 2024-03-06T07:23:34.722 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.99529568 2024-03-06T07:23:34.722 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:23:34.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 -- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53406b4d0 msgr2=0x7ff53406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53406b4d0 0x7ff53406d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff53c036000 tx=0x7ff53c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:34.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 -- 172.21.15.59:0/1680867521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5540b4460 msgr2=0x7ff554156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5540b4460 0x7ff554156110 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff54400c410 tx=0x7ff54400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 -- 172.21.15.59:0/1680867521 shutdown_connections 2024-03-06T07:23:34.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff53406b4d0 0x7ff53406d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff554162970 0x7ff55415ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff554161cf0 0x7ff55415e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 --2- 172.21.15.59:0/1680867521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5540b4460 0x7ff554156110 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 -- 172.21.15.59:0/1680867521 >> 172.21.15.59:0/1680867521 conn(0x7ff5540d3ce0 msgr2=0x7ff5540d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 -- 172.21.15.59:0/1680867521 shutdown_connections 2024-03-06T07:23:34.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.713+0000 7ff55a53e640 1 -- 172.21.15.59:0/1680867521 wait complete. 2024-03-06T07:23:34.725 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:34.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.824+0000 7f67e8d7a640 1 Processor -- start 2024-03-06T07:23:34.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.824+0000 7f67e8d7a640 1 -- start start 2024-03-06T07:23:34.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.825+0000 7f67e8d7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e415bb20 0x7f67e40d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.825+0000 7f67e8d7a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f67e40cfff0 0x7f67e40d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.825+0000 7f67e8d7a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e4161980 con 0x7f67e40ce4b0 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.825+0000 7f67e8d7a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67e4161c60 con 0x7f67e40cfff0 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.825+0000 7f67e8d7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e4161f40 con 0x7f67e415bb20 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.825+0000 7f67e259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e415bb20 0x7f67e40d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.825+0000 7f67e1d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f67e40cfff0 0x7f67e40d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.825+0000 7f67e259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e415bb20 0x7f67e40d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:54874/0 (socket says 172.21.15.59:54874) 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.825+0000 7f67e2d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f67e40ce4b0 0x7f67e40ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:34630/0 (socket says 172.21.15.59:34630) 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.825+0000 7f67e1d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f67e40cfff0 0x7f67e40d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:39508/0 (socket says 172.21.15.59:39508) 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.825+0000 7f67e259c640 1 -- 172.21.15.59:0/3657509872 learned_addr learned my addr 172.21.15.59:0/3657509872 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.826+0000 7f67e259c640 1 -- 172.21.15.59:0/3657509872 >> v2:172.21.15.164:3300/0 conn(0x7f67e40cfff0 msgr2=0x7f67e40d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.826+0000 7f67e259c640 1 --2- 172.21.15.59:0/3657509872 >> v2:172.21.15.164:3300/0 conn(0x7f67e40cfff0 0x7f67e40d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.826+0000 7f67e259c640 1 -- 172.21.15.59:0/3657509872 >> v1:172.21.15.164:6789/0 conn(0x7f67e40ce4b0 legacy=0x7f67e40ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.826+0000 7f67e259c640 1 -- 172.21.15.59:0/3657509872 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e4162220 con 0x7f67e415bb20 2024-03-06T07:23:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.826+0000 7f67e259c640 1 --2- 172.21.15.59:0/3657509872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e415bb20 0x7f67e40d1df0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009870 tx=0x7f67dc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec7d608f84ab1cb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.827+0000 7f67e159a640 1 -- 172.21.15.59:0/3657509872 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc034070 con 0x7f67e415bb20 2024-03-06T07:23:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.827+0000 7f67e159a640 1 -- 172.21.15.59:0/3657509872 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67dc030840 con 0x7f67e415bb20 2024-03-06T07:23:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.827+0000 7f67e159a640 1 -- 172.21.15.59:0/3657509872 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc030c10 con 0x7f67e415bb20 2024-03-06T07:23:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.827+0000 7f67e8d7a640 1 -- 172.21.15.59:0/3657509872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e415bb20 msgr2=0x7f67e40d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.827+0000 7f67e8d7a640 1 --2- 172.21.15.59:0/3657509872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e415bb20 0x7f67e40d1df0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009870 tx=0x7f67dc02b980 comp rx=0 tx=0).stop 2024-03-06T07:23:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.827+0000 7f67e8d7a640 1 -- 172.21.15.59:0/3657509872 shutdown_connections 2024-03-06T07:23:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.827+0000 7f67e8d7a640 1 --2- 172.21.15.59:0/3657509872 >> v2:172.21.15.164:3300/0 conn(0x7f67e40cfff0 0x7f67e40d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.827+0000 7f67e8d7a640 1 --2- 172.21.15.59:0/3657509872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e415bb20 0x7f67e40d1df0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.827+0000 7f67e8d7a640 1 -- 172.21.15.59:0/3657509872 >> 172.21.15.59:0/3657509872 conn(0x7f67e409a9f0 msgr2=0x7f67e409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.827+0000 7f67e8d7a640 1 -- 172.21.15.59:0/3657509872 shutdown_connections 2024-03-06T07:23:34.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.827+0000 7f67e8d7a640 1 -- 172.21.15.59:0/3657509872 wait complete. 2024-03-06T07:23:34.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.829+0000 7f67e8d7a640 1 Processor -- start 2024-03-06T07:23:34.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.829+0000 7f67e8d7a640 1 -- start start 2024-03-06T07:23:34.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.829+0000 7f67e8d7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e40ce4b0 0x7f67e41f8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.829+0000 7f67e8d7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e40cfff0 0x7f67e41ea0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.829+0000 7f67e8d7a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67e415bb20 0x7f67e40c6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.829+0000 7f67e8d7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e40d24b0 con 0x7f67e40ce4b0 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.829+0000 7f67e8d7a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e40d2330 con 0x7f67e415bb20 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.829+0000 7f67e8d7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e40d1980 con 0x7f67e40cfff0 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.829+0000 7f67e259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e40ce4b0 0x7f67e41f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.829+0000 7f67e259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e40ce4b0 0x7f67e41f8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56680/0 (socket says 172.21.15.59:56680) 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67e259c640 1 -- 172.21.15.59:0/422648139 learned_addr learned my addr 172.21.15.59:0/422648139 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67e1d9b640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e40cfff0 0x7f67e41ea0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67e2d9d640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67e415bb20 0x7f67e40c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67e259c640 1 -- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e40cfff0 msgr2=0x7f67e41ea0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67e259c640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e40cfff0 0x7f67e41ea0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67e259c640 1 -- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67e415bb20 msgr2=0x7f67e40c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67e259c640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67e415bb20 0x7f67e40c6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67e259c640 1 -- 172.21.15.59:0/422648139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e41ea5e0 con 0x7f67e40ce4b0 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67e259c640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e40ce4b0 0x7f67e41f8290 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f67dc02b9c0 tx=0x7f67dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67daffd640 1 -- 172.21.15.59:0/422648139 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc034040 con 0x7f67e40ce4b0 2024-03-06T07:23:34.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67daffd640 1 -- 172.21.15.59:0/422648139 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67dc03ada0 con 0x7f67e40ce4b0 2024-03-06T07:23:34.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67e8d7a640 1 -- 172.21.15.59:0/422648139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e40c6d50 con 0x7f67e40ce4b0 2024-03-06T07:23:34.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67daffd640 1 -- 172.21.15.59:0/422648139 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc044610 con 0x7f67e40ce4b0 2024-03-06T07:23:34.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.830+0000 7f67e8d7a640 1 -- 172.21.15.59:0/422648139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e40c72d0 con 0x7f67e40ce4b0 2024-03-06T07:23:34.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.831+0000 7f67e8d7a640 1 -- 172.21.15.59:0/422648139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e404f330 con 0x7f67e40ce4b0 2024-03-06T07:23:34.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.832+0000 7f67daffd640 1 -- 172.21.15.59:0/422648139 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67dc04b060 con 0x7f67e40ce4b0 2024-03-06T07:23:34.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.832+0000 7f67daffd640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67c406b4d0 0x7f67c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:34.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.832+0000 7f67daffd640 1 -- 172.21.15.59:0/422648139 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67dc0ac090 con 0x7f67e40ce4b0 2024-03-06T07:23:34.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.832+0000 7f67e1d9b640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67c406b4d0 0x7f67c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:34.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.833+0000 7f67e1d9b640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67c406b4d0 0x7f67c406d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f67e415b670 tx=0x7f67cc009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:34.836+0000 7f67daffd640 1 -- 172.21.15.59:0/422648139 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67dc03f070 con 0x7f67e40ce4b0 2024-03-06T07:23:35.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.058+0000 7f67e8d7a640 1 -- 172.21.15.59:0/422648139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f67e40b1160 con 0x7f67e40ce4b0 2024-03-06T07:23:35.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.058+0000 7f67daffd640 1 -- 172.21.15.59:0/422648139 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+73 (secure 0 0 0) 0x7f67dc07a8c0 con 0x7f67e40ce4b0 2024-03-06T07:23:35.061 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:35.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 -- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67c406b4d0 msgr2=0x7f67c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67c406b4d0 0x7f67c406d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f67e415b670 tx=0x7f67cc009190 comp rx=0 tx=0).stop 2024-03-06T07:23:35.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 -- 172.21.15.59:0/422648139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e40ce4b0 msgr2=0x7f67e41f8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:35.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e40ce4b0 0x7f67e41f8290 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f67dc02b9c0 tx=0x7f67dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 -- 172.21.15.59:0/422648139 shutdown_connections 2024-03-06T07:23:35.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67c406b4d0 0x7f67c406d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67e415bb20 0x7f67e40c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67e40cfff0 0x7f67e41ea0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 --2- 172.21.15.59:0/422648139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67e40ce4b0 0x7f67e41f8290 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:35.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 -- 172.21.15.59:0/422648139 >> 172.21.15.59:0/422648139 conn(0x7f67e409a9f0 msgr2=0x7f67e409add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:35.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 -- 172.21.15.59:0/422648139 shutdown_connections 2024-03-06T07:23:35.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35.060+0000 7f67e8d7a640 1 -- 172.21.15.59:0/422648139 wait complete. 2024-03-06T07:23:35.073 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:35.073 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:35.073 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:23:35.073 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:23:35.073 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:35.073 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:35.073 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:35.073 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:35.074 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:35.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:35.074 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:23:35.090 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:35.090 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:35.090 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:35.091 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:35.092 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:35.092 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:35.092 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:35.092 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:35.093 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:23:35.093 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:35.093 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:35.093 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:23:35.093 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:35.093 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:35.094 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:23:35.094 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:23:35.094 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42564sec preferred_lft 42564sec 2024-03-06T07:23:35.094 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:23:35.094 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:35.094 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:35.095 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:35.095 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:23:35.095 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:35.095 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:35.095 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:23:35.095 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:35.096 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:23:35.096 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:35.096 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:35.096 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:35.096 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:23:35.096 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:35.096 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:35.096 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:35.097 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:23:35.097 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:35.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:35.097 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:35.156 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:23:35.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:35.157 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:23:35.213 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:23:35.213 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:35.213 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:23:35.213 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:23:35.213 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:35.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:35.214 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:23:35.214 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:23:35.214 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:23:35.214 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:23:35.214 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:23:35.214 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:23:35.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:35.309 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:23:35.324 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:23:35.325 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:23:35.325 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:35.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:35.325 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:23:35.394 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:23:35.395 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:35.395 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:35.449 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:35.449 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:35.449 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:35.503 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:23:35.573 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:35.628 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:23:35.628 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:23:35.628 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:23:35.628 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:23:35.628 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:23:35.628 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:23:35.628 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:23:35.628 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:23:35.628 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:23:35.628 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:23:35.629 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:23:35.630 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:35.630 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:35.630 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:35.630 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:35.630 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:23:35.630 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:35.630 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:35.630 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:35.630 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:35.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:35.630 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:23:35.682 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:23:35.682 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:23:35.725 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:23:35.751 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:35.807 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:23:35.807 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:23:35.807 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:23:35.808 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:23:35.808 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:23:35.808 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:23:35.808 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:23:35.808 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:23:35.808 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:23:35.808 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:23:35.808 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:23:35.809 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:23:35.810 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:35.810 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:35.810 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:35.810 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:35.810 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:23:35.810 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:35.810 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:35.810 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:35.810 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:35.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:35.810 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:23:35.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:23:35.807+0000 7f00ac3ee480 -1 init, newargv = 0x55904a395690 newargc=15 2024-03-06T07:23:35.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[55855]: starting ceph client 2024-03-06T07:23:35.814 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[55855]: starting fuse 2024-03-06T07:23:35.825 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:23:35.826 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:23:35.826 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:35.826 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:23:35.826 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:23:35.826 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:23:35.826 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:23:35.826 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:23:35.826 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:23:35.826 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:23:35.826 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:23:35.826 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:35.826 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:23:35.827 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:23:35.946 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.55855.asok 2024-03-06T07:23:35.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:35.956 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55855.asok 2024-03-06T07:23:35.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:35.956 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55855.asok status 2024-03-06T07:23:36.097 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:23:36.097 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:23:36.097 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:36.097 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "55855", 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 5856, 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 5856 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:36.098 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 4034659290 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 4034659290 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.5856 192.168.0.1:0/4034659290", 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/4034659290", 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:23:36.099 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 20, 2024-03-06T07:23:36.100 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 45, 2024-03-06T07:23:36.100 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:36.100 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:23:36.100 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:23:36.100 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:23:36.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:36.106 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:23:36.106 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:36.121 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:23:36.121 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:36.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:36.121 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:36.190 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:23:36.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.332+0000 7fbf88ea4640 1 Processor -- start 2024-03-06T07:23:36.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.332+0000 7fbf88ea4640 1 -- start start 2024-03-06T07:23:36.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.332+0000 7fbf88ea4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf840cea50 0x7fbf840b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.332+0000 7fbf88ea4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf8406cab0 0x7fbf840ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.332+0000 7fbf88ea4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8406d030 con 0x7fbf8415def0 2024-03-06T07:23:36.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.332+0000 7fbf88ea4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf840cf1c0 con 0x7fbf840cea50 2024-03-06T07:23:36.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.332+0000 7fbf88ea4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf840cc760 con 0x7fbf8406cab0 2024-03-06T07:23:36.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.333+0000 7fbf7bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf840cea50 0x7fbf840b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.333+0000 7fbf7bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf840cea50 0x7fbf840b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56692/0 (socket says 172.21.15.59:56692) 2024-03-06T07:23:36.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.333+0000 7fbf7bfff640 1 -- 172.21.15.59:0/212524333 learned_addr learned my addr 172.21.15.59:0/212524333 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:36.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.333+0000 7fbf82d9d640 1 --2- 172.21.15.59:0/212524333 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf8406cab0 0x7fbf840ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.333+0000 7fbf7bfff640 1 -- 172.21.15.59:0/212524333 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf8406cab0 msgr2=0x7fbf840ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:36.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.333+0000 7fbf7bfff640 1 --2- 172.21.15.59:0/212524333 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf8406cab0 0x7fbf840ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.333+0000 7fbf7bfff640 1 -- 172.21.15.59:0/212524333 >> v1:172.21.15.164:6789/0 conn(0x7fbf8415def0 legacy=0x7fbf840ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.333+0000 7fbf7bfff640 1 -- 172.21.15.59:0/212524333 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8406d310 con 0x7fbf840cea50 2024-03-06T07:23:36.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf7bfff640 1 --2- 172.21.15.59:0/212524333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf840cea50 0x7fbf840b41b0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009870 tx=0x7fbf6c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f204636e378ddf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf81d9b640 1 -- 172.21.15.59:0/212524333 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c02f840 con 0x7fbf840cea50 2024-03-06T07:23:36.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf81d9b640 1 -- 172.21.15.59:0/212524333 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c02f9e0 con 0x7fbf840cea50 2024-03-06T07:23:36.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf81d9b640 1 -- 172.21.15.59:0/212524333 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c02fdb0 con 0x7fbf840cea50 2024-03-06T07:23:36.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf88ea4640 1 -- 172.21.15.59:0/212524333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf840cea50 msgr2=0x7fbf840b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf88ea4640 1 --2- 172.21.15.59:0/212524333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf840cea50 0x7fbf840b41b0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009870 tx=0x7fbf6c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:23:36.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf88ea4640 1 -- 172.21.15.59:0/212524333 shutdown_connections 2024-03-06T07:23:36.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf88ea4640 1 --2- 172.21.15.59:0/212524333 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf8406cab0 0x7fbf840ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf88ea4640 1 --2- 172.21.15.59:0/212524333 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf840cea50 0x7fbf840b41b0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf88ea4640 1 -- 172.21.15.59:0/212524333 >> 172.21.15.59:0/212524333 conn(0x7fbf840b3690 msgr2=0x7fbf840b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf88ea4640 1 -- 172.21.15.59:0/212524333 shutdown_connections 2024-03-06T07:23:36.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.334+0000 7fbf88ea4640 1 -- 172.21.15.59:0/212524333 wait complete. 2024-03-06T07:23:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.336+0000 7fbf88ea4640 1 Processor -- start 2024-03-06T07:23:36.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.336+0000 7fbf88ea4640 1 -- start start 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.336+0000 7fbf88ea4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf8406cab0 0x7fbf8415bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.336+0000 7fbf88ea4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf840cea50 0x7fbf840c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.336+0000 7fbf88ea4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf8415def0 0x7fbf840b6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.336+0000 7fbf88ea4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8415e2d0 con 0x7fbf8415def0 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.336+0000 7fbf88ea4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf840cc3f0 con 0x7fbf840cea50 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.336+0000 7fbf88ea4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8415e450 con 0x7fbf8406cab0 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf82d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf8415def0 0x7fbf840b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf82d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf8415def0 0x7fbf840b6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56694/0 (socket says 172.21.15.59:56694) 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf82d9d640 1 -- 172.21.15.59:0/2668262771 learned_addr learned my addr 172.21.15.59:0/2668262771 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf7bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf840cea50 0x7fbf840c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf8259c640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf8406cab0 0x7fbf8415bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf82d9d640 1 -- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf8406cab0 msgr2=0x7fbf8415bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf82d9d640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf8406cab0 0x7fbf8415bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf82d9d640 1 -- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf840cea50 msgr2=0x7fbf840c61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf82d9d640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf840cea50 0x7fbf840c61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf82d9d640 1 -- 172.21.15.59:0/2668262771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf840b6ee0 con 0x7fbf8415def0 2024-03-06T07:23:36.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf82d9d640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf8415def0 0x7fbf840b6440 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00efc0 tx=0x7fbf7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf7affd640 1 -- 172.21.15.59:0/2668262771 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c018020 con 0x7fbf8415def0 2024-03-06T07:23:36.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf7affd640 1 -- 172.21.15.59:0/2668262771 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c014480 con 0x7fbf8415def0 2024-03-06T07:23:36.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf88ea4640 1 -- 172.21.15.59:0/2668262771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf840c56c0 con 0x7fbf8415def0 2024-03-06T07:23:36.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf7affd640 1 -- 172.21.15.59:0/2668262771 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c014870 con 0x7fbf8415def0 2024-03-06T07:23:36.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.337+0000 7fbf88ea4640 1 -- 172.21.15.59:0/2668262771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf840c5c80 con 0x7fbf8415def0 2024-03-06T07:23:36.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.338+0000 7fbf88ea4640 1 -- 172.21.15.59:0/2668262771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8409fcf0 con 0x7fbf8415def0 2024-03-06T07:23:36.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.339+0000 7fbf7affd640 1 -- 172.21.15.59:0/2668262771 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf7c005d10 con 0x7fbf8415def0 2024-03-06T07:23:36.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.339+0000 7fbf7affd640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf6406b4d0 0x7fbf6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:36.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.339+0000 7fbf7affd640 1 -- 172.21.15.59:0/2668262771 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbf7c08ab40 con 0x7fbf8415def0 2024-03-06T07:23:36.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.339+0000 7fbf8259c640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf6406b4d0 0x7fbf6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:36.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.340+0000 7fbf8259c640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf6406b4d0 0x7fbf6406d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbf740016e0 tx=0x7fbf74009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:36.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.344+0000 7fbf7affd640 1 -- 172.21.15.59:0/2668262771 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf7c022070 con 0x7fbf8415def0 2024-03-06T07:23:36.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.523+0000 7fbf88ea4640 1 -- 172.21.15.59:0/2668262771 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7fbf84068ff0 con 0x7fbf8415def0 2024-03-06T07:23:36.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.526+0000 7fbf7affd640 1 -- 172.21.15.59:0/2668262771 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7fbf7c010040 con 0x7fbf8415def0 2024-03-06T07:23:36.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 -- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf6406b4d0 msgr2=0x7fbf6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf6406b4d0 0x7fbf6406d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbf740016e0 tx=0x7fbf74009040 comp rx=0 tx=0).stop 2024-03-06T07:23:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 -- 172.21.15.59:0/2668262771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf8415def0 msgr2=0x7fbf840b6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf8415def0 0x7fbf840b6440 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00efc0 tx=0x7fbf7c00c490 comp rx=0 tx=0).stop 2024-03-06T07:23:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 -- 172.21.15.59:0/2668262771 shutdown_connections 2024-03-06T07:23:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf6406b4d0 0x7fbf6406d990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf8415def0 0x7fbf840b6440 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf840cea50 0x7fbf840c61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 --2- 172.21.15.59:0/2668262771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf8406cab0 0x7fbf8415bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 -- 172.21.15.59:0/2668262771 >> 172.21.15.59:0/2668262771 conn(0x7fbf840b3690 msgr2=0x7fbf840d0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 -- 172.21.15.59:0/2668262771 shutdown_connections 2024-03-06T07:23:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:36.528+0000 7fbf88ea4640 1 -- 172.21.15.59:0/2668262771 wait complete. 2024-03-06T07:23:46.542 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:23:46.542 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:46.561 DEBUG:teuthology.orchestra.run.smithi059:> stdin-killer -- python3 -c ' 2024-03-06T07:23:46.561 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:23:46.561 DEBUG:teuthology.orchestra.run.smithi059:> import uuid 2024-03-06T07:23:46.561 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:23:46.561 DEBUG:teuthology.orchestra.run.smithi059:> n = 51 2024-03-06T07:23:46.561 DEBUG:teuthology.orchestra.run.smithi059:> create_hard_links = False 2024-03-06T07:23:46.561 DEBUG:teuthology.orchestra.run.smithi059:> if 0 > 0: 2024-03-06T07:23:46.561 DEBUG:teuthology.orchestra.run.smithi059:> create_hard_links = True 2024-03-06T07:23:46.561 DEBUG:teuthology.orchestra.run.smithi059:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:23:46.561 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:23:46.561 DEBUG:teuthology.orchestra.run.smithi059:> dpath = os.path.dirname(path) 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> fnameprefix = os.path.basename(path) 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> try: 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> for i in range(n): 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> f.write(f"{i}") 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> f.flush() 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(f.fileno()) 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> os.unlink(fpath) 2024-03-06T07:23:46.562 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:23:46.563 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(dirfd) 2024-03-06T07:23:46.563 DEBUG:teuthology.orchestra.run.smithi059:> if create_hard_links: 2024-03-06T07:23:46.563 DEBUG:teuthology.orchestra.run.smithi059:> for j in range(0): 2024-03-06T07:23:46.563 DEBUG:teuthology.orchestra.run.smithi059:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:23:46.563 DEBUG:teuthology.orchestra.run.smithi059:> if True: 2024-03-06T07:23:46.563 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(dirfd) 2024-03-06T07:23:46.563 DEBUG:teuthology.orchestra.run.smithi059:> finally: 2024-03-06T07:23:46.563 DEBUG:teuthology.orchestra.run.smithi059:> os.close(dirfd) 2024-03-06T07:23:46.563 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:23:46.770 INFO:teuthology.orchestra.run.smithi059.stderr:Traceback (most recent call last): 2024-03-06T07:23:46.770 INFO:teuthology.orchestra.run.smithi059.stderr: File "", line 20, in 2024-03-06T07:23:46.770 INFO:teuthology.orchestra.run.smithi059.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_50' 2024-03-06T07:23:46.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:46 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:23:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:46.780 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:46.780 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:46.862 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:46.901 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:46.902 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:23:46.902 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:46.917 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:47.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.057+0000 7f6eca2c9640 1 Processor -- start 2024-03-06T07:23:47.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.057+0000 7f6eca2c9640 1 -- start start 2024-03-06T07:23:47.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6eca2c9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ec40d4920 0x7f6ec40cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6eca2c9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6ec41625a0 0x7f6ec415dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6eca2c9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ec40ceac0 con 0x7f6ec41621c0 2024-03-06T07:23:47.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6eca2c9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec40cf690 con 0x7f6ec40d4920 2024-03-06T07:23:47.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6eca2c9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec40d2dc0 con 0x7f6ec41625a0 2024-03-06T07:23:47.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6ec37fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ec40d4920 0x7f6ec40cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6ec37fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ec40d4920 0x7f6ec40cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37990/0 (socket says 172.21.15.59:37990) 2024-03-06T07:23:47.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6ec37fe640 1 -- 172.21.15.59:0/2642744337 learned_addr learned my addr 172.21.15.59:0/2642744337 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:47.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6ec8866640 1 --2- 172.21.15.59:0/2642744337 >> v2:172.21.15.164:3300/0 conn(0x7f6ec41625a0 0x7f6ec415dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6ec37fe640 1 -- 172.21.15.59:0/2642744337 >> v2:172.21.15.164:3300/0 conn(0x7f6ec41625a0 msgr2=0x7f6ec415dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6ec37fe640 1 --2- 172.21.15.59:0/2642744337 >> v2:172.21.15.164:3300/0 conn(0x7f6ec41625a0 0x7f6ec415dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6ec37fe640 1 -- 172.21.15.59:0/2642744337 >> v1:172.21.15.164:6789/0 conn(0x7f6ec41621c0 legacy=0x7f6ec40d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.058+0000 7f6ec37fe640 1 -- 172.21.15.59:0/2642744337 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec40ced60 con 0x7f6ec40d4920 2024-03-06T07:23:47.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.059+0000 7f6ec37fe640 1 --2- 172.21.15.59:0/2642744337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ec40d4920 0x7f6ec40cfb20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6ec40d3be0 tx=0x7f6eb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a63e7ec00909401 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.059+0000 7f6ec2ffd640 1 -- 172.21.15.59:0/2642744337 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4035020 con 0x7f6ec40d4920 2024-03-06T07:23:47.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.059+0000 7f6ec2ffd640 1 -- 172.21.15.59:0/2642744337 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb4031870 con 0x7f6ec40d4920 2024-03-06T07:23:47.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.059+0000 7f6ec2ffd640 1 -- 172.21.15.59:0/2642744337 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4031c40 con 0x7f6ec40d4920 2024-03-06T07:23:47.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.060+0000 7f6eca2c9640 1 -- 172.21.15.59:0/2642744337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ec40d4920 msgr2=0x7f6ec40cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.060+0000 7f6eca2c9640 1 --2- 172.21.15.59:0/2642744337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ec40d4920 0x7f6ec40cfb20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6ec40d3be0 tx=0x7f6eb402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:47.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.060+0000 7f6eca2c9640 1 -- 172.21.15.59:0/2642744337 shutdown_connections 2024-03-06T07:23:47.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.060+0000 7f6eca2c9640 1 --2- 172.21.15.59:0/2642744337 >> v2:172.21.15.164:3300/0 conn(0x7f6ec41625a0 0x7f6ec415dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.060+0000 7f6eca2c9640 1 --2- 172.21.15.59:0/2642744337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ec40d4920 0x7f6ec40cfb20 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.060+0000 7f6eca2c9640 1 -- 172.21.15.59:0/2642744337 >> 172.21.15.59:0/2642744337 conn(0x7f6ec40d3d10 msgr2=0x7f6ec40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.060+0000 7f6eca2c9640 1 -- 172.21.15.59:0/2642744337 shutdown_connections 2024-03-06T07:23:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.060+0000 7f6eca2c9640 1 -- 172.21.15.59:0/2642744337 wait complete. 2024-03-06T07:23:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.061+0000 7f6eca2c9640 1 Processor -- start 2024-03-06T07:23:47.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.061+0000 7f6eca2c9640 1 -- start start 2024-03-06T07:23:47.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6eca2c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ec40d4920 0x7f6ec409ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6eca2c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ec41621c0 0x7f6ec40b67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6eca2c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ec41625a0 0x7f6ec4151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6eca2c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec415e280 con 0x7f6ec41625a0 2024-03-06T07:23:47.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6eca2c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec40cf320 con 0x7f6ec41621c0 2024-03-06T07:23:47.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6eca2c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec415e400 con 0x7f6ec40d4920 2024-03-06T07:23:47.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec8866640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ec41625a0 0x7f6ec4151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec8866640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ec41625a0 0x7f6ec4151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37994/0 (socket says 172.21.15.59:37994) 2024-03-06T07:23:47.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec8866640 1 -- 172.21.15.59:0/4233052136 learned_addr learned my addr 172.21.15.59:0/4233052136 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:47.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec37fe640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ec41621c0 0x7f6ec40b67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec3fff640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ec40d4920 0x7f6ec409ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec8866640 1 -- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ec40d4920 msgr2=0x7f6ec409ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec8866640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ec40d4920 0x7f6ec409ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec8866640 1 -- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ec41621c0 msgr2=0x7f6ec40b67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec8866640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ec41621c0 0x7f6ec40b67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec8866640 1 -- 172.21.15.59:0/4233052136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec40b7d40 con 0x7f6ec41625a0 2024-03-06T07:23:47.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec8866640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ec41625a0 0x7f6ec4151050 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00efc0 tx=0x7f6ebc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec0ff9640 1 -- 172.21.15.59:0/4233052136 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc018020 con 0x7f6ec41625a0 2024-03-06T07:23:47.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec0ff9640 1 -- 172.21.15.59:0/4233052136 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ebc014480 con 0x7f6ec41625a0 2024-03-06T07:23:47.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6eca2c9640 1 -- 172.21.15.59:0/4233052136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec40b6de0 con 0x7f6ec41625a0 2024-03-06T07:23:47.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6ec0ff9640 1 -- 172.21.15.59:0/4233052136 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc014870 con 0x7f6ec41625a0 2024-03-06T07:23:47.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.062+0000 7f6eca2c9640 1 -- 172.21.15.59:0/4233052136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec40b7680 con 0x7f6ec41625a0 2024-03-06T07:23:47.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.063+0000 7f6eca2c9640 1 -- 172.21.15.59:0/4233052136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec404f330 con 0x7f6ec41625a0 2024-03-06T07:23:47.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.064+0000 7f6ec0ff9640 1 -- 172.21.15.59:0/4233052136 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ebc005d10 con 0x7f6ec41625a0 2024-03-06T07:23:47.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.064+0000 7f6ec0ff9640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6eb006b4d0 0x7f6eb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.064+0000 7f6ec0ff9640 1 -- 172.21.15.59:0/4233052136 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6ebc08aba0 con 0x7f6ec41625a0 2024-03-06T07:23:47.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.064+0000 7f6ec3fff640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6eb006b4d0 0x7f6eb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.065+0000 7f6ec3fff640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6eb006b4d0 0x7f6eb006d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8001780 tx=0x7f6eb8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.069+0000 7f6ec0ff9640 1 -- 172.21.15.59:0/4233052136 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ebc022070 con 0x7f6ec41625a0 2024-03-06T07:23:47.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.250+0000 7f6eca2c9640 1 -- 172.21.15.59:0/4233052136 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ec4073380 con 0x7f6ec41625a0 2024-03-06T07:23:47.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.251+0000 7f6ec0ff9640 1 -- 172.21.15.59:0/4233052136 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16752 (secure 0 0 0) 0x7f6ebc010040 con 0x7f6ec41625a0 2024-03-06T07:23:47.254 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:47.255 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":47,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:23:37.063479+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:26.672668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:27.362891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":0,"to":5}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:47.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.253+0000 7f6eca2c9640 1 -- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6eb006b4d0 msgr2=0x7f6eb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.253+0000 7f6eca2c9640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6eb006b4d0 0x7f6eb006d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8001780 tx=0x7f6eb8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:47.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.253+0000 7f6eca2c9640 1 -- 172.21.15.59:0/4233052136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ec41625a0 msgr2=0x7f6ec4151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.253+0000 7f6eca2c9640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ec41625a0 0x7f6ec4151050 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00efc0 tx=0x7f6ebc00c490 comp rx=0 tx=0).stop 2024-03-06T07:23:47.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.253+0000 7f6eca2c9640 1 -- 172.21.15.59:0/4233052136 shutdown_connections 2024-03-06T07:23:47.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.253+0000 7f6eca2c9640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6eb006b4d0 0x7f6eb006d990 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.253+0000 7f6eca2c9640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ec41625a0 0x7f6ec4151050 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.253+0000 7f6eca2c9640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ec41621c0 0x7f6ec40b67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.253+0000 7f6eca2c9640 1 --2- 172.21.15.59:0/4233052136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ec40d4920 0x7f6ec409ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.253+0000 7f6eca2c9640 1 -- 172.21.15.59:0/4233052136 >> 172.21.15.59:0/4233052136 conn(0x7f6ec40d3d10 msgr2=0x7f6ec406b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.254+0000 7f6eca2c9640 1 -- 172.21.15.59:0/4233052136 shutdown_connections 2024-03-06T07:23:47.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.254+0000 7f6eca2c9640 1 -- 172.21.15.59:0/4233052136 wait complete. 2024-03-06T07:23:47.266 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:26.672668+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:27.362891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:47.266 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:47.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.369+0000 7f98407bf640 1 Processor -- start 2024-03-06T07:23:47.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.369+0000 7f98407bf640 1 -- start start 2024-03-06T07:23:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.370+0000 7f98407bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98381614d0 0x7f98380d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.370+0000 7f98407bf640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f98380d1910 0x7f98380d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.370+0000 7f98407bf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f98380d22b0 0x7f98380d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.370+0000 7f98407bf640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98380d2740 con 0x7f98380d1910 2024-03-06T07:23:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.370+0000 7f98407bf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98381618b0 con 0x7f98380d22b0 2024-03-06T07:23:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.370+0000 7f98407bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98380cf400 con 0x7f98381614d0 2024-03-06T07:23:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.371+0000 7f983dd5a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f98380d1910 0x7f98380d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.371+0000 7f983dd5a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f98380d1910 0x7f98380d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38006/0 (socket says 172.21.15.59:38006) 2024-03-06T07:23:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.371+0000 7f983dd5a640 1 -- 172.21.15.59:0/370864938 learned_addr learned my addr 172.21.15.59:0/370864938 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.371+0000 7f983ed5c640 1 --2- 172.21.15.59:0/370864938 >> v2:172.21.15.164:3300/0 conn(0x7f98380d22b0 0x7f98380d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.371+0000 7f983e55b640 1 --2- 172.21.15.59:0/370864938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98381614d0 0x7f98380d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.371+0000 7f983dd5a640 1 -- 172.21.15.59:0/370864938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98381614d0 msgr2=0x7f98380d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.371+0000 7f983dd5a640 1 --2- 172.21.15.59:0/370864938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98381614d0 0x7f98380d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.371+0000 7f983dd5a640 1 -- 172.21.15.59:0/370864938 >> v2:172.21.15.164:3300/0 conn(0x7f98380d22b0 msgr2=0x7f98380d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.371+0000 7f983dd5a640 1 --2- 172.21.15.59:0/370864938 >> v2:172.21.15.164:3300/0 conn(0x7f98380d22b0 0x7f98380d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.371+0000 7f983dd5a640 1 -- 172.21.15.59:0/370864938 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98380cf6a0 con 0x7f98380d1910 2024-03-06T07:23:47.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f983dd5a640 1 --2- 172.21.15.59:0/370864938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98380d1910 0x7f98380d1d70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f98380b57d0 tx=0x7f982802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbfb793a85bdcf04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f983d559640 1 -- 172.21.15.59:0/370864938 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9828035020 con 0x7f98380d1910 2024-03-06T07:23:47.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f983d559640 1 -- 172.21.15.59:0/370864938 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9828031870 con 0x7f98380d1910 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f983d559640 1 -- 172.21.15.59:0/370864938 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9828031c40 con 0x7f98380d1910 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f98407bf640 1 -- 172.21.15.59:0/370864938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98380d1910 msgr2=0x7f98380d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f98407bf640 1 --2- 172.21.15.59:0/370864938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98380d1910 0x7f98380d1d70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f98380b57d0 tx=0x7f982802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f98407bf640 1 -- 172.21.15.59:0/370864938 shutdown_connections 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f98407bf640 1 --2- 172.21.15.59:0/370864938 >> v2:172.21.15.164:3300/0 conn(0x7f98380d22b0 0x7f98380d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f98407bf640 1 --2- 172.21.15.59:0/370864938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98380d1910 0x7f98380d1d70 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f98407bf640 1 --2- 172.21.15.59:0/370864938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98381614d0 0x7f98380d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f98407bf640 1 -- 172.21.15.59:0/370864938 >> 172.21.15.59:0/370864938 conn(0x7f98380d3d40 msgr2=0x7f98380d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f98407bf640 1 -- 172.21.15.59:0/370864938 shutdown_connections 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.372+0000 7f98407bf640 1 -- 172.21.15.59:0/370864938 wait complete. 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.373+0000 7f98407bf640 1 Processor -- start 2024-03-06T07:23:47.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.373+0000 7f98407bf640 1 -- start start 2024-03-06T07:23:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.374+0000 7f98407bf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98380d1910 0x7f98380b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.374+0000 7f98407bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98380d22b0 0x7f98380b9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.374+0000 7f98407bf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98381614d0 0x7f98381f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.374+0000 7f98407bf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98380b0910 con 0x7f98381614d0 2024-03-06T07:23:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.374+0000 7f98407bf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98380cfc60 con 0x7f98380d1910 2024-03-06T07:23:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.374+0000 7f98407bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98380d0be0 con 0x7f98380d22b0 2024-03-06T07:23:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.374+0000 7f983ed5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98381614d0 0x7f98381f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.374+0000 7f983ed5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98381614d0 0x7f98381f6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38018/0 (socket says 172.21.15.59:38018) 2024-03-06T07:23:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.374+0000 7f983ed5c640 1 -- 172.21.15.59:0/2459860562 learned_addr learned my addr 172.21.15.59:0/2459860562 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.374+0000 7f983e55b640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98380d1910 0x7f98380b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.374+0000 7f983dd5a640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98380d22b0 0x7f98380b9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.375+0000 7f983ed5c640 1 -- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98380d22b0 msgr2=0x7f98380b9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.375+0000 7f983ed5c640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98380d22b0 0x7f98380b9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.375+0000 7f983ed5c640 1 -- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98380d1910 msgr2=0x7f98380b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.375+0000 7f983ed5c640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98380d1910 0x7f98380b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.375+0000 7f983ed5c640 1 -- 172.21.15.59:0/2459860562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983815d120 con 0x7f98381614d0 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.375+0000 7f983ed5c640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98381614d0 0x7f98381f6e90 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f983000efc0 tx=0x7f983000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.375+0000 7f982effd640 1 -- 172.21.15.59:0/2459860562 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830018020 con 0x7f98381614d0 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.375+0000 7f982effd640 1 -- 172.21.15.59:0/2459860562 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9830014480 con 0x7f98381614d0 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.375+0000 7f98407bf640 1 -- 172.21.15.59:0/2459860562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983815d400 con 0x7f98381614d0 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.375+0000 7f982effd640 1 -- 172.21.15.59:0/2459860562 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830014870 con 0x7f98381614d0 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.375+0000 7f98407bf640 1 -- 172.21.15.59:0/2459860562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983815d9c0 con 0x7f98381614d0 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.376+0000 7f98407bf640 1 -- 172.21.15.59:0/2459860562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9838068ff0 con 0x7f98381614d0 2024-03-06T07:23:47.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.376+0000 7f982effd640 1 -- 172.21.15.59:0/2459860562 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9830005d10 con 0x7f98381614d0 2024-03-06T07:23:47.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.377+0000 7f982effd640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f981c06b4d0 0x7f981c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.377+0000 7f982effd640 1 -- 172.21.15.59:0/2459860562 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f983008aee0 con 0x7f98381614d0 2024-03-06T07:23:47.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.377+0000 7f983e55b640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f981c06b4d0 0x7f981c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.377+0000 7f983e55b640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f981c06b4d0 0x7f981c06d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f98380cdba0 tx=0x7f9834009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.381+0000 7f982effd640 1 -- 172.21.15.59:0/2459860562 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9830022030 con 0x7f98381614d0 2024-03-06T07:23:47.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.564+0000 7f98407bf640 1 -- 172.21.15.59:0/2459860562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f98380a1c10 con 0x7f98381614d0 2024-03-06T07:23:47.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.564+0000 7f982effd640 1 -- 172.21.15.59:0/2459860562 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9830010040 con 0x7f98381614d0 2024-03-06T07:23:47.567 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:23:47.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.566+0000 7f98407bf640 1 -- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f981c06b4d0 msgr2=0x7f981c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.566+0000 7f98407bf640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f981c06b4d0 0x7f981c06d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f98380cdba0 tx=0x7f9834009190 comp rx=0 tx=0).stop 2024-03-06T07:23:47.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.566+0000 7f98407bf640 1 -- 172.21.15.59:0/2459860562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98381614d0 msgr2=0x7f98381f6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.566+0000 7f98407bf640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98381614d0 0x7f98381f6e90 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f983000efc0 tx=0x7f983000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.566+0000 7f98407bf640 1 -- 172.21.15.59:0/2459860562 shutdown_connections 2024-03-06T07:23:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.566+0000 7f98407bf640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f981c06b4d0 0x7f981c06d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.566+0000 7f98407bf640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98381614d0 0x7f98381f6e90 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.566+0000 7f98407bf640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98380d22b0 0x7f98380b9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.567+0000 7f98407bf640 1 --2- 172.21.15.59:0/2459860562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98380d1910 0x7f98380b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.567+0000 7f98407bf640 1 -- 172.21.15.59:0/2459860562 >> 172.21.15.59:0/2459860562 conn(0x7f98380d3d40 msgr2=0x7f98380b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.567+0000 7f98407bf640 1 -- 172.21.15.59:0/2459860562 shutdown_connections 2024-03-06T07:23:47.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.567+0000 7f98407bf640 1 -- 172.21.15.59:0/2459860562 wait complete. 2024-03-06T07:23:47.579 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.681+0000 7f5750066640 1 Processor -- start 2024-03-06T07:23:47.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.682+0000 7f5750066640 1 -- start start 2024-03-06T07:23:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.682+0000 7f5750066640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f57481586e0 0x7f57480aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.682+0000 7f5750066640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57480d01a0 0x7f57480b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.682+0000 7f5750066640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5748158ac0 con 0x7f57480aef80 2024-03-06T07:23:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.683+0000 7f5750066640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574815b2a0 con 0x7f57481586e0 2024-03-06T07:23:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.683+0000 7f5750066640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574815b540 con 0x7f57480d01a0 2024-03-06T07:23:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.683+0000 7f574de02640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f57481586e0 0x7f57480aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.683+0000 7f574d601640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f57480aef80 0x7f57480cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50440/0 (socket says 172.21.15.59:50440) 2024-03-06T07:23:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.683+0000 7f574d601640 1 -- 172.21.15.59:0/190821858 learned_addr learned my addr 172.21.15.59:0/190821858 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:47.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.683+0000 7f574e603640 1 --2- 172.21.15.59:0/190821858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57480d01a0 0x7f57480b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.683+0000 7f574de02640 1 -- 172.21.15.59:0/190821858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57480d01a0 msgr2=0x7f57480b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.683+0000 7f574de02640 1 --2- 172.21.15.59:0/190821858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57480d01a0 0x7f57480b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.683+0000 7f574de02640 1 -- 172.21.15.59:0/190821858 >> v1:172.21.15.59:6789/0 conn(0x7f57480aef80 legacy=0x7f57480cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.683+0000 7f574ce00640 1 -- 172.21.15.59:0/190821858 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152564200 0 0) 0x7f5748158ac0 con 0x7f57480aef80 2024-03-06T07:23:47.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.683+0000 7f574de02640 1 -- 172.21.15.59:0/190821858 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574815b820 con 0x7f57481586e0 2024-03-06T07:23:47.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.684+0000 7f574de02640 1 --2- 172.21.15.59:0/190821858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57481586e0 0x7f57480aea40 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f57440097a0 tx=0x7f574402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2b0c03c99a5670 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.684+0000 7f574ce00640 1 -- 172.21.15.59:0/190821858 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744035020 con 0x7f57481586e0 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.684+0000 7f574ce00640 1 -- 172.21.15.59:0/190821858 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5744031830 con 0x7f57481586e0 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.684+0000 7f574ce00640 1 -- 172.21.15.59:0/190821858 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744031c00 con 0x7f57481586e0 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.685+0000 7f5750066640 1 -- 172.21.15.59:0/190821858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57481586e0 msgr2=0x7f57480aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.685+0000 7f5750066640 1 --2- 172.21.15.59:0/190821858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57481586e0 0x7f57480aea40 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f57440097a0 tx=0x7f574402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.685+0000 7f5750066640 1 -- 172.21.15.59:0/190821858 shutdown_connections 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.685+0000 7f5750066640 1 --2- 172.21.15.59:0/190821858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57480d01a0 0x7f57480b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.685+0000 7f5750066640 1 --2- 172.21.15.59:0/190821858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57481586e0 0x7f57480aea40 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.685+0000 7f5750066640 1 -- 172.21.15.59:0/190821858 >> 172.21.15.59:0/190821858 conn(0x7f574806b190 msgr2=0x7f574806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.685+0000 7f5750066640 1 -- 172.21.15.59:0/190821858 shutdown_connections 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.685+0000 7f5750066640 1 -- 172.21.15.59:0/190821858 wait complete. 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.686+0000 7f5750066640 1 Processor -- start 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.686+0000 7f5750066640 1 -- start start 2024-03-06T07:23:47.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f5750066640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57480aef80 0x7f5748154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f5750066640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57480d01a0 0x7f57481f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f5750066640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57481586e0 0x7f57481f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f5750066640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5748071350 con 0x7f57480d01a0 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f5750066640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574815bde0 con 0x7f57481586e0 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f5750066640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480714d0 con 0x7f57480aef80 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f574d601640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57480d01a0 0x7f57481f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f574d601640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57480d01a0 0x7f57481f22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38034/0 (socket says 172.21.15.59:38034) 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f574d601640 1 -- 172.21.15.59:0/3592902696 learned_addr learned my addr 172.21.15.59:0/3592902696 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f574de02640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57480aef80 0x7f5748154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f574e603640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57481586e0 0x7f57481f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f574d601640 1 -- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57480aef80 msgr2=0x7f5748154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f574d601640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57480aef80 0x7f5748154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f574d601640 1 -- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57481586e0 msgr2=0x7f57481f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f574d601640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57481586e0 0x7f57481f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f574d601640 1 -- 172.21.15.59:0/3592902696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57481f2e10 con 0x7f57480d01a0 2024-03-06T07:23:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.687+0000 7f574d601640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57480d01a0 0x7f57481f22f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f573800c4b0 tx=0x7f573800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.688+0000 7f573e7fc640 1 -- 172.21.15.59:0/3592902696 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738018070 con 0x7f57480d01a0 2024-03-06T07:23:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.688+0000 7f573e7fc640 1 -- 172.21.15.59:0/3592902696 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5738007500 con 0x7f57480d01a0 2024-03-06T07:23:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.688+0000 7f5750066640 1 -- 172.21.15.59:0/3592902696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57481f30f0 con 0x7f57480d01a0 2024-03-06T07:23:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.688+0000 7f573e7fc640 1 -- 172.21.15.59:0/3592902696 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738004a20 con 0x7f57480d01a0 2024-03-06T07:23:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.688+0000 7f5750066640 1 -- 172.21.15.59:0/3592902696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57481f36b0 con 0x7f57480d01a0 2024-03-06T07:23:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.689+0000 7f5750066640 1 -- 172.21.15.59:0/3592902696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574815c250 con 0x7f57480d01a0 2024-03-06T07:23:47.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.689+0000 7f573e7fc640 1 -- 172.21.15.59:0/3592902696 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f573801b070 con 0x7f57480d01a0 2024-03-06T07:23:47.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.689+0000 7f573e7fc640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b500 0x7f572c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.689+0000 7f573e7fc640 1 -- 172.21.15.59:0/3592902696 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5738089480 con 0x7f57480d01a0 2024-03-06T07:23:47.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.690+0000 7f574de02640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b500 0x7f572c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.690+0000 7f574de02640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b500 0x7f572c06d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5744036000 tx=0x7f574403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.694+0000 7f573e7fc640 1 -- 172.21.15.59:0/3592902696 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f574815c250 con 0x7f57480d01a0 2024-03-06T07:23:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.875+0000 7f5750066640 1 -- 172.21.15.59:0/3592902696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f57480b4220 con 0x7f57480d01a0 2024-03-06T07:23:47.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.876+0000 7f573e7fc640 1 -- 172.21.15.59:0/3592902696 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v47) v1 ==== 104+0+11 (secure 0 0 0) 0x7f57480b4220 con 0x7f57480d01a0 2024-03-06T07:23:47.878 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:23:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 -- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b500 msgr2=0x7f572c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b500 0x7f572c06d9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5744036000 tx=0x7f574403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 -- 172.21.15.59:0/3592902696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57480d01a0 msgr2=0x7f57481f22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57480d01a0 0x7f57481f22f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f573800c4b0 tx=0x7f573800c970 comp rx=0 tx=0).stop 2024-03-06T07:23:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 -- 172.21.15.59:0/3592902696 shutdown_connections 2024-03-06T07:23:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b500 0x7f572c06d9c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57481586e0 0x7f57481f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57480d01a0 0x7f57481f22f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 --2- 172.21.15.59:0/3592902696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57480aef80 0x7f5748154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 -- 172.21.15.59:0/3592902696 >> 172.21.15.59:0/3592902696 conn(0x7f574806b190 msgr2=0x7f57480b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 -- 172.21.15.59:0/3592902696 shutdown_connections 2024-03-06T07:23:47.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.878+0000 7f5750066640 1 -- 172.21.15.59:0/3592902696 wait complete. 2024-03-06T07:23:47.890 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:47.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.993+0000 7f1cb0df7640 1 Processor -- start 2024-03-06T07:23:47.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.993+0000 7f1cb0df7640 1 -- start start 2024-03-06T07:23:47.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.993+0000 7f1cb0df7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1cac0d4860 0x7f1cac0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.993+0000 7f1cb0df7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1cac0b0e30 0x7f1cac0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.993+0000 7f1cb0df7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cac1618b0 con 0x7f1cac1614d0 2024-03-06T07:23:47.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.993+0000 7f1cb0df7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cac0d4cf0 con 0x7f1cac0d4860 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.993+0000 7f1cb0df7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cac0cf400 con 0x7f1cac0b0e30 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.993+0000 7f1ca3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1cac0d4860 0x7f1cac0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.993+0000 7f1caa59c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1cac1614d0 0x7f1cac0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50446/0 (socket says 172.21.15.59:50446) 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.993+0000 7f1ca3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1cac0d4860 0x7f1cac0d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38050/0 (socket says 172.21.15.59:38050) 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.993+0000 7f1caa59c640 1 -- 172.21.15.59:0/3731114881 learned_addr learned my addr 172.21.15.59:0/3731114881 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.994+0000 7f1caad9d640 1 --2- 172.21.15.59:0/3731114881 >> v2:172.21.15.164:3300/0 conn(0x7f1cac0b0e30 0x7f1cac0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.994+0000 7f1ca3fff640 1 -- 172.21.15.59:0/3731114881 >> v2:172.21.15.164:3300/0 conn(0x7f1cac0b0e30 msgr2=0x7f1cac0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.994+0000 7f1ca3fff640 1 --2- 172.21.15.59:0/3731114881 >> v2:172.21.15.164:3300/0 conn(0x7f1cac0b0e30 0x7f1cac0b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.994+0000 7f1ca3fff640 1 -- 172.21.15.59:0/3731114881 >> v1:172.21.15.59:6789/0 conn(0x7f1cac1614d0 legacy=0x7f1cac0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.994+0000 7f1ca9d9b640 1 -- 172.21.15.59:0/3731114881 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674592518 0 0) 0x7f1cac1618b0 con 0x7f1cac1614d0 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.994+0000 7f1ca3fff640 1 -- 172.21.15.59:0/3731114881 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac0cf6a0 con 0x7f1cac0d4860 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1ca3fff640 1 --2- 172.21.15.59:0/3731114881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac0d4860 0x7f1cac0d2020 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0b2a70 tx=0x7f1c9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540197dd992d9b4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1ca9d9b640 1 -- 172.21.15.59:0/3731114881 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94035020 con 0x7f1cac0d4860 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1ca9d9b640 1 -- 172.21.15.59:0/3731114881 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c94031830 con 0x7f1cac0d4860 2024-03-06T07:23:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1ca9d9b640 1 -- 172.21.15.59:0/3731114881 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94031c00 con 0x7f1cac0d4860 2024-03-06T07:23:48.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1cb0df7640 1 -- 172.21.15.59:0/3731114881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac0d4860 msgr2=0x7f1cac0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1cb0df7640 1 --2- 172.21.15.59:0/3731114881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac0d4860 0x7f1cac0d2020 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0b2a70 tx=0x7f1c9402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1cb0df7640 1 -- 172.21.15.59:0/3731114881 shutdown_connections 2024-03-06T07:23:48.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1cb0df7640 1 --2- 172.21.15.59:0/3731114881 >> v2:172.21.15.164:3300/0 conn(0x7f1cac0b0e30 0x7f1cac0b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1cb0df7640 1 --2- 172.21.15.59:0/3731114881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac0d4860 0x7f1cac0d2020 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1cb0df7640 1 -- 172.21.15.59:0/3731114881 >> 172.21.15.59:0/3731114881 conn(0x7f1cac0d3d40 msgr2=0x7f1cac0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1cb0df7640 1 -- 172.21.15.59:0/3731114881 shutdown_connections 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.995+0000 7f1cb0df7640 1 -- 172.21.15.59:0/3731114881 wait complete. 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.997+0000 7f1cb0df7640 1 Processor -- start 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.997+0000 7f1cb0df7640 1 -- start start 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.997+0000 7f1cb0df7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1cac0b0e30 0x7f1cac0b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.997+0000 7f1cb0df7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cac0d4860 0x7f1cac15dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.997+0000 7f1cb0df7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac1614d0 0x7f1cac15b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.997+0000 7f1cb0df7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0b0910 con 0x7f1cac1614d0 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.997+0000 7f1cb0df7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0cfc60 con 0x7f1cac0d4860 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.997+0000 7f1cb0df7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0d2560 con 0x7f1cac0b0e30 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.997+0000 7f1caad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac1614d0 0x7f1cac15b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.997+0000 7f1caad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac1614d0 0x7f1cac15b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38064/0 (socket says 172.21.15.59:38064) 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.997+0000 7f1caad9d640 1 -- 172.21.15.59:0/533939260 learned_addr learned my addr 172.21.15.59:0/533939260 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1ca3fff640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cac0d4860 0x7f1cac15dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1caa59c640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1cac0b0e30 0x7f1cac0b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1caad9d640 1 -- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1cac0b0e30 msgr2=0x7f1cac0b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1caad9d640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1cac0b0e30 0x7f1cac0b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1caad9d640 1 -- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cac0d4860 msgr2=0x7f1cac15dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1caad9d640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cac0d4860 0x7f1cac15dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1caad9d640 1 -- 172.21.15.59:0/533939260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac0b5dd0 con 0x7f1cac1614d0 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1caad9d640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac1614d0 0x7f1cac15b7c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400efc0 tx=0x7f1ca400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1ca2ffd640 1 -- 172.21.15.59:0/533939260 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4018020 con 0x7f1cac1614d0 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1ca2ffd640 1 -- 172.21.15.59:0/533939260 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca4014480 con 0x7f1cac1614d0 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1cb0df7640 1 -- 172.21.15.59:0/533939260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cac0b60b0 con 0x7f1cac1614d0 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1ca2ffd640 1 -- 172.21.15.59:0/533939260 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4014870 con 0x7f1cac1614d0 2024-03-06T07:23:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.998+0000 7f1cb0df7640 1 -- 172.21.15.59:0/533939260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cac0cafb0 con 0x7f1cac1614d0 2024-03-06T07:23:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:47.999+0000 7f1cb0df7640 1 -- 172.21.15.59:0/533939260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cac068ff0 con 0x7f1cac1614d0 2024-03-06T07:23:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.000+0000 7f1ca2ffd640 1 -- 172.21.15.59:0/533939260 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ca4005d10 con 0x7f1cac1614d0 2024-03-06T07:23:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.000+0000 7f1ca2ffd640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c8c06b4d0 0x7f1c8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.000+0000 7f1ca2ffd640 1 -- 172.21.15.59:0/533939260 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1ca408ab80 con 0x7f1cac1614d0 2024-03-06T07:23:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.000+0000 7f1caa59c640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c8c06b4d0 0x7f1c8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.000+0000 7f1caa59c640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c8c06b4d0 0x7f1c8c06d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c000f60 tx=0x7f1c9c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.004+0000 7f1ca2ffd640 1 -- 172.21.15.59:0/533939260 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ca4022070 con 0x7f1cac1614d0 2024-03-06T07:23:48.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.184+0000 7f1cb0df7640 1 -- 172.21.15.59:0/533939260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1cac0ce9e0 con 0x7f1cac1614d0 2024-03-06T07:23:48.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.185+0000 7f1ca2ffd640 1 -- 172.21.15.59:0/533939260 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v47) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ca4010040 con 0x7f1cac1614d0 2024-03-06T07:23:48.187 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:23:48.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 -- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c8c06b4d0 msgr2=0x7f1c8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c8c06b4d0 0x7f1c8c06d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c000f60 tx=0x7f1c9c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:48.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 -- 172.21.15.59:0/533939260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac1614d0 msgr2=0x7f1cac15b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac1614d0 0x7f1cac15b7c0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1ca400efc0 tx=0x7f1ca400c490 comp rx=0 tx=0).stop 2024-03-06T07:23:48.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 -- 172.21.15.59:0/533939260 shutdown_connections 2024-03-06T07:23:48.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c8c06b4d0 0x7f1c8c06d990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac1614d0 0x7f1cac15b7c0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cac0d4860 0x7f1cac15dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 --2- 172.21.15.59:0/533939260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1cac0b0e30 0x7f1cac0b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 -- 172.21.15.59:0/533939260 >> 172.21.15.59:0/533939260 conn(0x7f1cac0d3d40 msgr2=0x7f1cac06c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 -- 172.21.15.59:0/533939260 shutdown_connections 2024-03-06T07:23:48.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.187+0000 7f1cb0df7640 1 -- 172.21.15.59:0/533939260 wait complete. 2024-03-06T07:23:48.199 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:48.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.300+0000 7f9fd8266640 1 Processor -- start 2024-03-06T07:23:48.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.300+0000 7f9fd8266640 1 -- start start 2024-03-06T07:23:48.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd8266640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd00b3240 0x7f9fd00b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd8266640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9fd00b2910 0x7f9fd00d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd8266640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd015e3f0 con 0x7f9fd015e010 2024-03-06T07:23:48.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd8266640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd00b1f40 con 0x7f9fd00b2910 2024-03-06T07:23:48.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd8266640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd00b1c10 con 0x7f9fd00b3240 2024-03-06T07:23:48.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd5801640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd00b3240 0x7f9fd00b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd5801640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd00b3240 0x7f9fd00b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:53768/0 (socket says 172.21.15.59:53768) 2024-03-06T07:23:48.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd5801640 1 -- 172.21.15.59:0/2270443597 learned_addr learned my addr 172.21.15.59:0/2270443597 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:48.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd6803640 1 --2- 172.21.15.59:0/2270443597 >> v2:172.21.15.164:3300/0 conn(0x7f9fd00b2910 0x7f9fd00d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd5801640 1 -- 172.21.15.59:0/2270443597 >> v2:172.21.15.164:3300/0 conn(0x7f9fd00b2910 msgr2=0x7f9fd00d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd5801640 1 --2- 172.21.15.59:0/2270443597 >> v2:172.21.15.164:3300/0 conn(0x7f9fd00b2910 0x7f9fd00d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd5801640 1 -- 172.21.15.59:0/2270443597 >> v1:172.21.15.164:6789/0 conn(0x7f9fd015e010 legacy=0x7f9fd00cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd5801640 1 -- 172.21.15.59:0/2270443597 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd006d1b0 con 0x7f9fd00b3240 2024-03-06T07:23:48.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.301+0000 7f9fd6803640 1 --2- 172.21.15.59:0/2270443597 >> v2:172.21.15.164:3300/0 conn(0x7f9fd00b2910 0x7f9fd00d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:48.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.302+0000 7f9fd5801640 1 --2- 172.21.15.59:0/2270443597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd00b3240 0x7f9fd00b23d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00098a0 tx=0x7f9fc002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=107c3cd8570cb2a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.302+0000 7f9fd5000640 1 -- 172.21.15.59:0/2270443597 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0035020 con 0x7f9fd00b3240 2024-03-06T07:23:48.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.302+0000 7f9fd5000640 1 -- 172.21.15.59:0/2270443597 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc0031640 con 0x7f9fd00b3240 2024-03-06T07:23:48.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.302+0000 7f9fd5000640 1 -- 172.21.15.59:0/2270443597 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc002d040 con 0x7f9fd00b3240 2024-03-06T07:23:48.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.303+0000 7f9fd8266640 1 -- 172.21.15.59:0/2270443597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd00b3240 msgr2=0x7f9fd00b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.303+0000 7f9fd8266640 1 --2- 172.21.15.59:0/2270443597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd00b3240 0x7f9fd00b23d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00098a0 tx=0x7f9fc002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.303+0000 7f9fd8266640 1 -- 172.21.15.59:0/2270443597 shutdown_connections 2024-03-06T07:23:48.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.303+0000 7f9fd8266640 1 --2- 172.21.15.59:0/2270443597 >> v2:172.21.15.164:3300/0 conn(0x7f9fd00b2910 0x7f9fd00d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.303+0000 7f9fd8266640 1 --2- 172.21.15.59:0/2270443597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd00b3240 0x7f9fd00b23d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.303+0000 7f9fd8266640 1 -- 172.21.15.59:0/2270443597 >> 172.21.15.59:0/2270443597 conn(0x7f9fd009a9f0 msgr2=0x7f9fd009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.303+0000 7f9fd8266640 1 -- 172.21.15.59:0/2270443597 shutdown_connections 2024-03-06T07:23:48.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.303+0000 7f9fd8266640 1 -- 172.21.15.59:0/2270443597 wait complete. 2024-03-06T07:23:48.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.304+0000 7f9fd8266640 1 Processor -- start 2024-03-06T07:23:48.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.305+0000 7f9fd8266640 1 -- start start 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.305+0000 7f9fd8266640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd015e010 0x7f9fd0156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.305+0000 7f9fd8266640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fd00b4710 0x7f9fd00c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.305+0000 7f9fd8266640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fd00b4b90 0x7f9fd00b6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.305+0000 7f9fd8266640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd00d2010 con 0x7f9fd00b4710 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.305+0000 7f9fd8266640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd006ced0 con 0x7f9fd00b4b90 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.305+0000 7f9fd8266640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd00d2190 con 0x7f9fd015e010 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.305+0000 7f9fd5801640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fd00b4710 0x7f9fd00c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.305+0000 7f9fd5801640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fd00b4710 0x7f9fd00c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38076/0 (socket says 172.21.15.59:38076) 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.305+0000 7f9fd5801640 1 -- 172.21.15.59:0/794610216 learned_addr learned my addr 172.21.15.59:0/794610216 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.306+0000 7f9fd6002640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd015e010 0x7f9fd0156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.306+0000 7f9fd6803640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fd00b4b90 0x7f9fd00b6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.306+0000 7f9fd5801640 1 -- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd015e010 msgr2=0x7f9fd0156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.306+0000 7f9fd5801640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd015e010 0x7f9fd0156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.306+0000 7f9fd5801640 1 -- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fd00b4b90 msgr2=0x7f9fd00b6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.306+0000 7f9fd5801640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fd00b4b90 0x7f9fd00b6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.306+0000 7f9fd5801640 1 -- 172.21.15.59:0/794610216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd0157100 con 0x7f9fd00b4710 2024-03-06T07:23:48.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.306+0000 7f9fd5801640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fd00b4710 0x7f9fd00c5670 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0000c00 tx=0x7f9fc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.308+0000 7f9fc67fc640 1 -- 172.21.15.59:0/794610216 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0035040 con 0x7f9fd00b4710 2024-03-06T07:23:48.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.308+0000 7f9fc67fc640 1 -- 172.21.15.59:0/794610216 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc00399f0 con 0x7f9fd00b4710 2024-03-06T07:23:48.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.308+0000 7f9fc67fc640 1 -- 172.21.15.59:0/794610216 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0039dc0 con 0x7f9fd00b4710 2024-03-06T07:23:48.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.308+0000 7f9fd8266640 1 -- 172.21.15.59:0/794610216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd00c6130 con 0x7f9fd00b4710 2024-03-06T07:23:48.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.308+0000 7f9fd8266640 1 -- 172.21.15.59:0/794610216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd0157480 con 0x7f9fd00b4710 2024-03-06T07:23:48.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.309+0000 7f9fd8266640 1 -- 172.21.15.59:0/794610216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd004f330 con 0x7f9fd00b4710 2024-03-06T07:23:48.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.309+0000 7f9fc67fc640 1 -- 172.21.15.59:0/794610216 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fc0038020 con 0x7f9fd00b4710 2024-03-06T07:23:48.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.309+0000 7f9fc67fc640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9fb406b4d0 0x7f9fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.309+0000 7f9fc67fc640 1 -- 172.21.15.59:0/794610216 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9fc00abf00 con 0x7f9fd00b4710 2024-03-06T07:23:48.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.310+0000 7f9fd6002640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9fb406b4d0 0x7f9fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.310+0000 7f9fd6002640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9fb406b4d0 0x7f9fb406d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0016e0 tx=0x7f9fcc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.314+0000 7f9fc67fc640 1 -- 172.21.15.59:0/794610216 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9fc007a780 con 0x7f9fd00b4710 2024-03-06T07:23:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.510+0000 7f9fd8266640 1 -- 172.21.15.59:0/794610216 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9fd00d2980 con 0x7f9fd00b4710 2024-03-06T07:23:48.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.511+0000 7f9fc67fc640 1 -- 172.21.15.59:0/794610216 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v47) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9fc007e820 con 0x7f9fd00b4710 2024-03-06T07:23:48.513 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 -- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9fb406b4d0 msgr2=0x7f9fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9fb406b4d0 0x7f9fb406d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0016e0 tx=0x7f9fcc009040 comp rx=0 tx=0).stop 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 -- 172.21.15.59:0/794610216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fd00b4710 msgr2=0x7f9fd00c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fd00b4710 0x7f9fd00c5670 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0000c00 tx=0x7f9fc00046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 -- 172.21.15.59:0/794610216 shutdown_connections 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9fb406b4d0 0x7f9fb406d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fd00b4b90 0x7f9fd00b6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fd00b4710 0x7f9fd00c5670 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 --2- 172.21.15.59:0/794610216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fd015e010 0x7f9fd0156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 -- 172.21.15.59:0/794610216 >> 172.21.15.59:0/794610216 conn(0x7f9fd009a9f0 msgr2=0x7f9fd00af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 -- 172.21.15.59:0/794610216 shutdown_connections 2024-03-06T07:23:48.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:48.513+0000 7f9fd8266640 1 -- 172.21.15.59:0/794610216 wait complete. 2024-03-06T07:23:48.525 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:23:48.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:48.526 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:48.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[55855]: fuse finished with error 0 and tester_r 0 2024-03-06T07:23:48.552 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:48.552 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:23:48.717 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:23:48.718 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:48.733 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:48.733 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:23:48.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:48.734 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:23:48.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:48.789 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:23:48.789 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:23:48.789 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:48.806 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:48.806 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:23:48.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:48.806 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:23:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:23:48.861 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:23:48.861 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:48.878 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:48.878 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:23:48.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:48.879 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:23:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:48.933 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:23:48.934 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:49.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.033+0000 7f65db1dd640 1 Processor -- start 2024-03-06T07:23:49.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.033+0000 7f65db1dd640 1 -- start start 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65db1dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 0x7f65d40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65db1dd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f65d415bf30 0x7f65d40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65db1dd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f65d415e850 0x7f65d415eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65db1dd640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d40cf3f0 con 0x7f65d415bf30 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65db1dd640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d415b210 con 0x7f65d415e850 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65db1dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d415f580 con 0x7f65d413dc10 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65d3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f65d415bf30 0x7f65d40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65d3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f65d415bf30 0x7f65d40b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38080/0 (socket says 172.21.15.59:38080) 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65d3fff640 1 -- 172.21.15.59:0/2672208624 learned_addr learned my addr 172.21.15.59:0/2672208624 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65d977a640 1 --2- 172.21.15.59:0/2672208624 >> v2:172.21.15.164:3300/0 conn(0x7f65d415e850 0x7f65d415eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65d8f79640 1 --2- 172.21.15.59:0/2672208624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 0x7f65d40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65d3fff640 1 -- 172.21.15.59:0/2672208624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 msgr2=0x7f65d40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65d3fff640 1 --2- 172.21.15.59:0/2672208624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 0x7f65d40b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65d3fff640 1 -- 172.21.15.59:0/2672208624 >> v2:172.21.15.164:3300/0 conn(0x7f65d415e850 msgr2=0x7f65d415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65d3fff640 1 --2- 172.21.15.59:0/2672208624 >> v2:172.21.15.164:3300/0 conn(0x7f65d415e850 0x7f65d415eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65d3fff640 1 -- 172.21.15.59:0/2672208624 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d415b450 con 0x7f65d415bf30 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.034+0000 7f65d8f79640 1 --2- 172.21.15.59:0/2672208624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 0x7f65d40b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.035+0000 7f65d3fff640 1 --2- 172.21.15.59:0/2672208624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d415bf30 0x7f65d40b2e90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f65d406c790 tx=0x7f65c802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e87b03ac26a139 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.035+0000 7f65d37fe640 1 -- 172.21.15.59:0/2672208624 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c80317c0 con 0x7f65d415bf30 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.035+0000 7f65d37fe640 1 -- 172.21.15.59:0/2672208624 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c8037070 con 0x7f65d415bf30 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.035+0000 7f65d37fe640 1 -- 172.21.15.59:0/2672208624 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8031b80 con 0x7f65d415bf30 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.036+0000 7f65db1dd640 1 -- 172.21.15.59:0/2672208624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d415bf30 msgr2=0x7f65d40b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.036+0000 7f65db1dd640 1 --2- 172.21.15.59:0/2672208624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d415bf30 0x7f65d40b2e90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f65d406c790 tx=0x7f65c802b860 comp rx=0 tx=0).stop 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.036+0000 7f65db1dd640 1 -- 172.21.15.59:0/2672208624 shutdown_connections 2024-03-06T07:23:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.036+0000 7f65db1dd640 1 --2- 172.21.15.59:0/2672208624 >> v2:172.21.15.164:3300/0 conn(0x7f65d415e850 0x7f65d415eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.036+0000 7f65db1dd640 1 --2- 172.21.15.59:0/2672208624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d415bf30 0x7f65d40b2e90 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.036+0000 7f65db1dd640 1 --2- 172.21.15.59:0/2672208624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 0x7f65d40b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.036+0000 7f65db1dd640 1 -- 172.21.15.59:0/2672208624 >> 172.21.15.59:0/2672208624 conn(0x7f65d40cf8a0 msgr2=0x7f65d40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.036+0000 7f65db1dd640 1 -- 172.21.15.59:0/2672208624 shutdown_connections 2024-03-06T07:23:49.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.036+0000 7f65db1dd640 1 -- 172.21.15.59:0/2672208624 wait complete. 2024-03-06T07:23:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.037+0000 7f65db1dd640 1 Processor -- start 2024-03-06T07:23:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.037+0000 7f65db1dd640 1 -- start start 2024-03-06T07:23:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65db1dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 0x7f65d41f5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65db1dd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d415bf30 0x7f65d413b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65db1dd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65d415e850 0x7f65d4139820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65db1dd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d415f1e0 con 0x7f65d415bf30 2024-03-06T07:23:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65db1dd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d413dff0 con 0x7f65d415e850 2024-03-06T07:23:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65db1dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d415f360 con 0x7f65d413dc10 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d415bf30 0x7f65d413b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d415bf30 0x7f65d413b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38082/0 (socket says 172.21.15.59:38082) 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d3fff640 1 -- 172.21.15.59:0/2716617985 learned_addr learned my addr 172.21.15.59:0/2716617985 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d977a640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65d415e850 0x7f65d4139820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d8f79640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 0x7f65d41f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d3fff640 1 -- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 msgr2=0x7f65d41f5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d3fff640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 0x7f65d41f5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d3fff640 1 -- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65d415e850 msgr2=0x7f65d4139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d3fff640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65d415e850 0x7f65d4139820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d3fff640 1 -- 172.21.15.59:0/2716617985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d413c170 con 0x7f65d415bf30 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d8f79640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 0x7f65d41f5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d3fff640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d415bf30 0x7f65d413b710 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f65c8000c00 tx=0x7f65c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.038+0000 7f65d17fa640 1 -- 172.21.15.59:0/2716617985 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8032650 con 0x7f65d415bf30 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.039+0000 7f65d17fa640 1 -- 172.21.15.59:0/2716617985 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c80327f0 con 0x7f65d415bf30 2024-03-06T07:23:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.039+0000 7f65db1dd640 1 -- 172.21.15.59:0/2716617985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d413c450 con 0x7f65d415bf30 2024-03-06T07:23:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.039+0000 7f65d17fa640 1 -- 172.21.15.59:0/2716617985 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8032be0 con 0x7f65d415bf30 2024-03-06T07:23:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.039+0000 7f65db1dd640 1 -- 172.21.15.59:0/2716617985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d41e5570 con 0x7f65d415bf30 2024-03-06T07:23:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.040+0000 7f65db1dd640 1 -- 172.21.15.59:0/2716617985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d40b3b30 con 0x7f65d415bf30 2024-03-06T07:23:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.040+0000 7f65d17fa640 1 -- 172.21.15.59:0/2716617985 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65c804c060 con 0x7f65d415bf30 2024-03-06T07:23:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.040+0000 7f65d17fa640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65c006b4d0 0x7f65c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.041+0000 7f65d17fa640 1 -- 172.21.15.59:0/2716617985 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f65c80ab8a0 con 0x7f65d415bf30 2024-03-06T07:23:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.041+0000 7f65d8f79640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65c006b4d0 0x7f65c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.041+0000 7f65d8f79640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65c006b4d0 0x7f65c006d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f65c400ba50 tx=0x7f65c4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.045+0000 7f65d17fa640 1 -- 172.21.15.59:0/2716617985 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65d40b3b30 con 0x7f65d415bf30 2024-03-06T07:23:49.254 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:49.257 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:49.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.263+0000 7f65db1dd640 1 -- 172.21.15.59:0/2716617985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65d404f330 con 0x7f65d415bf30 2024-03-06T07:23:49.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.264+0000 7f65d17fa640 1 -- 172.21.15.59:0/2716617985 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f65c80ab070 con 0x7f65d415bf30 2024-03-06T07:23:49.266 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:49.266 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5557,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/690078434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":690078434},{"type":"v1","addr":"172.21.15.164:6836","nonce":690078434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1345698247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1345698247},{"type":"v1","addr":"172.21.15.59:6833","nonce":1345698247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5685,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3875776524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3875776524},{"type":"v1","addr":"172.21.15.59:6835","nonce":3875776524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:28.400413+0000","modified":"2024-03-06T07:23:31.397689+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/3749315375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3749315375},{"type":"v1","addr":"172.21.15.164:6837","nonce":3749315375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-06T07:23:49.266 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 20 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 -- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65c006b4d0 msgr2=0x7f65c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65c006b4d0 0x7f65c006d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f65c400ba50 tx=0x7f65c4008040 comp rx=0 tx=0).stop 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 -- 172.21.15.59:0/2716617985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d415bf30 msgr2=0x7f65d413b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d415bf30 0x7f65d413b710 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f65c8000c00 tx=0x7f65c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 -- 172.21.15.59:0/2716617985 shutdown_connections 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65c006b4d0 0x7f65c006d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65d415e850 0x7f65d4139820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d415bf30 0x7f65d413b710 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 --2- 172.21.15.59:0/2716617985 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65d413dc10 0x7f65d41f5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 -- 172.21.15.59:0/2716617985 >> 172.21.15.59:0/2716617985 conn(0x7f65d40cf8a0 msgr2=0x7f65d406c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 -- 172.21.15.59:0/2716617985 shutdown_connections 2024-03-06T07:23:49.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.266+0000 7f65db1dd640 1 -- 172.21.15.59:0/2716617985 wait complete. 2024-03-06T07:23:49.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:49.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.445+0000 7f1275d4b640 1 Processor -- start 2024-03-06T07:23:49.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.445+0000 7f1275d4b640 1 -- start start 2024-03-06T07:23:49.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f1275d4b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f12700d00c0 0x7f12700d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f1275d4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12700cf850 0x7f12700cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f1275d4b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f12700ce1b0 0x7f12700ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f1275d4b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f127006ca90 con 0x7f12700d00c0 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f1275d4b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12700d09e0 con 0x7f12700ce1b0 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f1275d4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127015e380 con 0x7f12700cf850 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f126f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f12700d00c0 0x7f12700d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f126f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f12700d00c0 0x7f12700d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38086/0 (socket says 172.21.15.59:38086) 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f126f7fe640 1 -- 172.21.15.59:0/3221571020 learned_addr learned my addr 172.21.15.59:0/3221571020 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f126effd640 1 --2- 172.21.15.59:0/3221571020 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12700cf850 0x7f12700cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f126ffff640 1 --2- 172.21.15.59:0/3221571020 >> v2:172.21.15.164:3300/0 conn(0x7f12700ce1b0 0x7f12700ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f126f7fe640 1 -- 172.21.15.59:0/3221571020 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12700cf850 msgr2=0x7f12700cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f126f7fe640 1 --2- 172.21.15.59:0/3221571020 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12700cf850 0x7f12700cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f126f7fe640 1 -- 172.21.15.59:0/3221571020 >> v2:172.21.15.164:3300/0 conn(0x7f12700ce1b0 msgr2=0x7f12700ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f126f7fe640 1 --2- 172.21.15.59:0/3221571020 >> v2:172.21.15.164:3300/0 conn(0x7f12700ce1b0 0x7f12700ce640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.446+0000 7f126f7fe640 1 -- 172.21.15.59:0/3221571020 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12700cdc10 con 0x7f12700d00c0 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.447+0000 7f126f7fe640 1 --2- 172.21.15.59:0/3221571020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12700d00c0 0x7f12700d04a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1270071cf0 tx=0x7f126002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab08ecdf02011f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.447+0000 7f126e7fc640 1 -- 172.21.15.59:0/3221571020 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260035020 con 0x7f12700d00c0 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.447+0000 7f126e7fc640 1 -- 172.21.15.59:0/3221571020 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1260031870 con 0x7f12700d00c0 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.447+0000 7f126e7fc640 1 -- 172.21.15.59:0/3221571020 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260031c40 con 0x7f12700d00c0 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.448+0000 7f1275d4b640 1 -- 172.21.15.59:0/3221571020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12700d00c0 msgr2=0x7f12700d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.448+0000 7f1275d4b640 1 --2- 172.21.15.59:0/3221571020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12700d00c0 0x7f12700d04a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1270071cf0 tx=0x7f126002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.448+0000 7f1275d4b640 1 -- 172.21.15.59:0/3221571020 shutdown_connections 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.448+0000 7f1275d4b640 1 --2- 172.21.15.59:0/3221571020 >> v2:172.21.15.164:3300/0 conn(0x7f12700ce1b0 0x7f12700ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.448+0000 7f1275d4b640 1 --2- 172.21.15.59:0/3221571020 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12700cf850 0x7f12700cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.448+0000 7f1275d4b640 1 --2- 172.21.15.59:0/3221571020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12700d00c0 0x7f12700d04a0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.448+0000 7f1275d4b640 1 -- 172.21.15.59:0/3221571020 >> 172.21.15.59:0/3221571020 conn(0x7f12700b35a0 msgr2=0x7f12700b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.448+0000 7f1275d4b640 1 -- 172.21.15.59:0/3221571020 shutdown_connections 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.448+0000 7f1275d4b640 1 -- 172.21.15.59:0/3221571020 wait complete. 2024-03-06T07:23:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.449+0000 7f1275d4b640 1 Processor -- start 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.449+0000 7f1275d4b640 1 -- start start 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f1275d4b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12700ce1b0 0x7f12701537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f1275d4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12700cf850 0x7f12700c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f1275d4b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12700d00c0 0x7f12700ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f1275d4b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127006c720 con 0x7f12700ce1b0 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f1275d4b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127006c440 con 0x7f12700d00c0 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f1275d4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12700cd900 con 0x7f12700cf850 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f126f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12700ce1b0 0x7f12701537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f126f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12700ce1b0 0x7f12701537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38094/0 (socket says 172.21.15.59:38094) 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f126f7fe640 1 -- 172.21.15.59:0/4263245980 learned_addr learned my addr 172.21.15.59:0/4263245980 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f126effd640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12700cf850 0x7f12700c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f126ffff640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12700d00c0 0x7f12700ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f126f7fe640 1 -- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12700cf850 msgr2=0x7f12700c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f126f7fe640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12700cf850 0x7f12700c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f126f7fe640 1 -- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12700d00c0 msgr2=0x7f12700ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f126f7fe640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12700d00c0 0x7f12700ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.450+0000 7f126f7fe640 1 -- 172.21.15.59:0/4263245980 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12701f8750 con 0x7f12700ce1b0 2024-03-06T07:23:49.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.451+0000 7f126f7fe640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12700ce1b0 0x7f12701537b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f126002b950 tx=0x7f12600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.451+0000 7f1257fff640 1 -- 172.21.15.59:0/4263245980 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260040070 con 0x7f12700ce1b0 2024-03-06T07:23:49.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.451+0000 7f1257fff640 1 -- 172.21.15.59:0/4263245980 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1260031e30 con 0x7f12700ce1b0 2024-03-06T07:23:49.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.451+0000 7f1257fff640 1 -- 172.21.15.59:0/4263245980 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260035040 con 0x7f12700ce1b0 2024-03-06T07:23:49.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.451+0000 7f1275d4b640 1 -- 172.21.15.59:0/4263245980 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12701f78a0 con 0x7f12700ce1b0 2024-03-06T07:23:49.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.451+0000 7f1275d4b640 1 -- 172.21.15.59:0/4263245980 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12701f7e20 con 0x7f12700ce1b0 2024-03-06T07:23:49.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.452+0000 7f1275d4b640 1 -- 172.21.15.59:0/4263245980 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127004f330 con 0x7f12700ce1b0 2024-03-06T07:23:49.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.452+0000 7f1257fff640 1 -- 172.21.15.59:0/4263245980 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f126004b060 con 0x7f12700ce1b0 2024-03-06T07:23:49.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.452+0000 7f1257fff640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f125c06b500 0x7f125c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.453+0000 7f1257fff640 1 -- 172.21.15.59:0/4263245980 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f12600abcb0 con 0x7f12700ce1b0 2024-03-06T07:23:49.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.453+0000 7f126effd640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f125c06b500 0x7f125c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.453+0000 7f126effd640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f125c06b500 0x7f125c06d9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f12701548b0 tx=0x7f1264009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.457+0000 7f1257fff640 1 -- 172.21.15.59:0/4263245980 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f126007a4a0 con 0x7f12700ce1b0 2024-03-06T07:23:49.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.637+0000 7f1275d4b640 1 -- 172.21.15.59:0/4263245980 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f127006fc10 con 0x7f12700ce1b0 2024-03-06T07:23:49.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.638+0000 7f1257fff640 1 -- 172.21.15.59:0/4263245980 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16752 (secure 0 0 0) 0x7f126007e540 con 0x7f12700ce1b0 2024-03-06T07:23:49.640 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:49.641 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":47,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:23:37.063479+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:26.672668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:27.362891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":0,"to":5}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:49.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 -- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f125c06b500 msgr2=0x7f125c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f125c06b500 0x7f125c06d9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f12701548b0 tx=0x7f1264009190 comp rx=0 tx=0).stop 2024-03-06T07:23:49.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 -- 172.21.15.59:0/4263245980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12700ce1b0 msgr2=0x7f12701537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12700ce1b0 0x7f12701537b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f126002b950 tx=0x7f12600046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 -- 172.21.15.59:0/4263245980 shutdown_connections 2024-03-06T07:23:49.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f125c06b500 0x7f125c06d9c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12700d00c0 0x7f12700ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12700cf850 0x7f12700c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 --2- 172.21.15.59:0/4263245980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12700ce1b0 0x7f12701537b0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 -- 172.21.15.59:0/4263245980 >> 172.21.15.59:0/4263245980 conn(0x7f12700b35a0 msgr2=0x7f12700d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 -- 172.21.15.59:0/4263245980 shutdown_connections 2024-03-06T07:23:49.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.640+0000 7f1275d4b640 1 -- 172.21.15.59:0/4263245980 wait complete. 2024-03-06T07:23:49.654 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:26.672668+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:27.362891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:49.655 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.757+0000 7f831ceb5640 1 Processor -- start 2024-03-06T07:23:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.757+0000 7f831ceb5640 1 -- start start 2024-03-06T07:23:49.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f831ceb5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f83181586e0 0x7f83180aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f831ceb5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8318158ac0 con 0x7f83180d01a0 2024-03-06T07:23:49.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f831ceb5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831815b2a0 con 0x7f83180aef80 2024-03-06T07:23:49.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f831ceb5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f831815b540 con 0x7f83181586e0 2024-03-06T07:23:49.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f8316d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f83180d01a0 0x7f83180b33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50460/0 (socket says 172.21.15.59:50460) 2024-03-06T07:23:49.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f8316d9d640 1 -- 172.21.15.59:0/3673565098 learned_addr learned my addr 172.21.15.59:0/3673565098 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:49.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f831659c640 1 --2- 172.21.15.59:0/3673565098 >> v2:172.21.15.164:3300/0 conn(0x7f83181586e0 0x7f83180aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f8315d9b640 1 -- 172.21.15.59:0/3673565098 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165795318 0 0) 0x7f8318158ac0 con 0x7f83180d01a0 2024-03-06T07:23:49.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f8315d9b640 1 -- 172.21.15.59:0/3673565098 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8304003520 con 0x7f83180d01a0 2024-03-06T07:23:49.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f831659c640 1 -- 172.21.15.59:0/3673565098 >> v1:172.21.15.164:6789/0 conn(0x7f83180aef80 legacy=0x7f83180cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f831659c640 1 -- 172.21.15.59:0/3673565098 >> v1:172.21.15.59:6789/0 conn(0x7f83180d01a0 legacy=0x7f83180b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.758+0000 7f831659c640 1 -- 172.21.15.59:0/3673565098 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831815b820 con 0x7f83181586e0 2024-03-06T07:23:49.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.759+0000 7f831659c640 1 --2- 172.21.15.59:0/3673565098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83181586e0 0x7f83180aea40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f83080097a0 tx=0x7f830802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1d57ce246a116a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.759+0000 7f8315d9b640 1 -- 172.21.15.59:0/3673565098 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8308035020 con 0x7f83181586e0 2024-03-06T07:23:49.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.759+0000 7f8315d9b640 1 -- 172.21.15.59:0/3673565098 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8308031830 con 0x7f83181586e0 2024-03-06T07:23:49.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.759+0000 7f8315d9b640 1 -- 172.21.15.59:0/3673565098 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8308031c00 con 0x7f83181586e0 2024-03-06T07:23:49.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.760+0000 7f831ceb5640 1 -- 172.21.15.59:0/3673565098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83181586e0 msgr2=0x7f83180aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.760+0000 7f831ceb5640 1 --2- 172.21.15.59:0/3673565098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83181586e0 0x7f83180aea40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f83080097a0 tx=0x7f830802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.760+0000 7f831ceb5640 1 -- 172.21.15.59:0/3673565098 shutdown_connections 2024-03-06T07:23:49.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.760+0000 7f831ceb5640 1 --2- 172.21.15.59:0/3673565098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83181586e0 0x7f83180aea40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.760+0000 7f831ceb5640 1 -- 172.21.15.59:0/3673565098 >> 172.21.15.59:0/3673565098 conn(0x7f831806b190 msgr2=0x7f831806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.760+0000 7f831ceb5640 1 -- 172.21.15.59:0/3673565098 shutdown_connections 2024-03-06T07:23:49.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.760+0000 7f831ceb5640 1 -- 172.21.15.59:0/3673565098 wait complete. 2024-03-06T07:23:49.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.761+0000 7f831ceb5640 1 Processor -- start 2024-03-06T07:23:49.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.761+0000 7f831ceb5640 1 -- start start 2024-03-06T07:23:49.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f831ceb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83180aef80 0x7f8318143cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f831ceb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83180d01a0 0x7f8318141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f831ceb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83181586e0 0x7f8318142380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f831ceb5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8318071350 con 0x7f83180d01a0 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f831ceb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831815bde0 con 0x7f83180aef80 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f831ceb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83180714d0 con 0x7f83181586e0 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f830ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83180d01a0 0x7f8318141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f831659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83180aef80 0x7f8318143cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f830ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83180d01a0 0x7f8318141ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38104/0 (socket says 172.21.15.59:38104) 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f830ffff640 1 -- 172.21.15.59:0/4225191632 learned_addr learned my addr 172.21.15.59:0/4225191632 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f8316d9d640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83181586e0 0x7f8318142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f830ffff640 1 -- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83181586e0 msgr2=0x7f8318142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f830ffff640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83181586e0 0x7f8318142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f830ffff640 1 -- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83180aef80 msgr2=0x7f8318143cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f830ffff640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83180aef80 0x7f8318143cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f830ffff640 1 -- 172.21.15.59:0/4225191632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8318142c20 con 0x7f83180d01a0 2024-03-06T07:23:49.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f831659c640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83180aef80 0x7f8318143cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:49.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.762+0000 7f830ffff640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83180d01a0 0x7f8318141ba0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f830000c470 tx=0x7f8300001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.763+0000 7f830effd640 1 -- 172.21.15.59:0/4225191632 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83000052d0 con 0x7f83180d01a0 2024-03-06T07:23:49.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.763+0000 7f830effd640 1 -- 172.21.15.59:0/4225191632 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8300005470 con 0x7f83180d01a0 2024-03-06T07:23:49.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.763+0000 7f830effd640 1 -- 172.21.15.59:0/4225191632 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8300007260 con 0x7f83180d01a0 2024-03-06T07:23:49.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.763+0000 7f831ceb5640 1 -- 172.21.15.59:0/4225191632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8318142f00 con 0x7f83180d01a0 2024-03-06T07:23:49.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.763+0000 7f831ceb5640 1 -- 172.21.15.59:0/4225191632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83181434c0 con 0x7f83180d01a0 2024-03-06T07:23:49.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.764+0000 7f831ceb5640 1 -- 172.21.15.59:0/4225191632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83180732b0 con 0x7f83180d01a0 2024-03-06T07:23:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.764+0000 7f830effd640 1 -- 172.21.15.59:0/4225191632 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8300007400 con 0x7f83180d01a0 2024-03-06T07:23:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.764+0000 7f830effd640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82f806b500 0x7f82f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.765+0000 7f830effd640 1 -- 172.21.15.59:0/4225191632 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f83000896a0 con 0x7f83180d01a0 2024-03-06T07:23:49.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.765+0000 7f831659c640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82f806b500 0x7f82f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.765+0000 7f831659c640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82f806b500 0x7f82f806d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8308036000 tx=0x7f830803c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.769+0000 7f830effd640 1 -- 172.21.15.59:0/4225191632 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8300057e90 con 0x7f83180d01a0 2024-03-06T07:23:49.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.947+0000 7f831ceb5640 1 -- 172.21.15.59:0/4225191632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f831806a130 con 0x7f83180d01a0 2024-03-06T07:23:49.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.948+0000 7f830effd640 1 -- 172.21.15.59:0/4225191632 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f830005bf30 con 0x7f83180d01a0 2024-03-06T07:23:49.950 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:23:49.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 -- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82f806b500 msgr2=0x7f82f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82f806b500 0x7f82f806d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8308036000 tx=0x7f830803c040 comp rx=0 tx=0).stop 2024-03-06T07:23:49.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 -- 172.21.15.59:0/4225191632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83180d01a0 msgr2=0x7f8318141ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83180d01a0 0x7f8318141ba0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f830000c470 tx=0x7f8300001e70 comp rx=0 tx=0).stop 2024-03-06T07:23:49.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 -- 172.21.15.59:0/4225191632 shutdown_connections 2024-03-06T07:23:49.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82f806b500 0x7f82f806d9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83181586e0 0x7f8318142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83180d01a0 0x7f8318141ba0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 --2- 172.21.15.59:0/4225191632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83180aef80 0x7f8318143cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 -- 172.21.15.59:0/4225191632 >> 172.21.15.59:0/4225191632 conn(0x7f831806b190 msgr2=0x7f83180afbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 -- 172.21.15.59:0/4225191632 shutdown_connections 2024-03-06T07:23:49.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:49.950+0000 7f831ceb5640 1 -- 172.21.15.59:0/4225191632 wait complete. 2024-03-06T07:23:49.962 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:50.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.065+0000 7f82c2078640 1 Processor -- start 2024-03-06T07:23:50.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.065+0000 7f82c2078640 1 -- start start 2024-03-06T07:23:50.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.066+0000 7f82c2078640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f82bc071a70 0x7f82bc071e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.066+0000 7f82c2078640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82bc15ad90 0x7f82bc0b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.066+0000 7f82c2078640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f82bc0b4110 0x7f82bc06da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.066+0000 7f82c2078640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82bc15c020 con 0x7f82bc0b4110 2024-03-06T07:23:50.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.066+0000 7f82c2078640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82bc072390 con 0x7f82bc071a70 2024-03-06T07:23:50.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.066+0000 7f82c2078640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc15b260 con 0x7f82bc15ad90 2024-03-06T07:23:50.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.066+0000 7f82bbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f82bc0b4110 0x7f82bc06da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.066+0000 7f82bbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f82bc0b4110 0x7f82bc06da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38112/0 (socket says 172.21.15.59:38112) 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.066+0000 7f82bbfff640 1 -- 172.21.15.59:0/534060062 learned_addr learned my addr 172.21.15.59:0/534060062 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.066+0000 7f82bb7fe640 1 --2- 172.21.15.59:0/534060062 >> v2:172.21.15.164:3300/0 conn(0x7f82bc071a70 0x7f82bc071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.066+0000 7f82baffd640 1 --2- 172.21.15.59:0/534060062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82bc15ad90 0x7f82bc0b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.067+0000 7f82bbfff640 1 -- 172.21.15.59:0/534060062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82bc15ad90 msgr2=0x7f82bc0b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.067+0000 7f82bbfff640 1 --2- 172.21.15.59:0/534060062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82bc15ad90 0x7f82bc0b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.067+0000 7f82bbfff640 1 -- 172.21.15.59:0/534060062 >> v2:172.21.15.164:3300/0 conn(0x7f82bc071a70 msgr2=0x7f82bc071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.067+0000 7f82bbfff640 1 --2- 172.21.15.59:0/534060062 >> v2:172.21.15.164:3300/0 conn(0x7f82bc071a70 0x7f82bc071e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.067+0000 7f82bbfff640 1 -- 172.21.15.59:0/534060062 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc15b540 con 0x7f82bc0b4110 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.067+0000 7f82bbfff640 1 --2- 172.21.15.59:0/534060062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82bc0b4110 0x7f82bc06da00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f82bc06d050 tx=0x7f82b402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90047f3e6e6c22d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.067+0000 7f82ba7fc640 1 -- 172.21.15.59:0/534060062 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b402c9e0 con 0x7f82bc0b4110 2024-03-06T07:23:50.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.067+0000 7f82ba7fc640 1 -- 172.21.15.59:0/534060062 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b402cb80 con 0x7f82bc0b4110 2024-03-06T07:23:50.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.067+0000 7f82ba7fc640 1 -- 172.21.15.59:0/534060062 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b40343f0 con 0x7f82bc0b4110 2024-03-06T07:23:50.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.068+0000 7f82c2078640 1 -- 172.21.15.59:0/534060062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82bc0b4110 msgr2=0x7f82bc06da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.068+0000 7f82c2078640 1 --2- 172.21.15.59:0/534060062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82bc0b4110 0x7f82bc06da00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f82bc06d050 tx=0x7f82b402be40 comp rx=0 tx=0).stop 2024-03-06T07:23:50.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.068+0000 7f82c2078640 1 -- 172.21.15.59:0/534060062 shutdown_connections 2024-03-06T07:23:50.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.068+0000 7f82c2078640 1 --2- 172.21.15.59:0/534060062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82bc0b4110 0x7f82bc06da00 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.068+0000 7f82c2078640 1 --2- 172.21.15.59:0/534060062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82bc15ad90 0x7f82bc0b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.068+0000 7f82c2078640 1 --2- 172.21.15.59:0/534060062 >> v2:172.21.15.164:3300/0 conn(0x7f82bc071a70 0x7f82bc071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.068+0000 7f82c2078640 1 -- 172.21.15.59:0/534060062 >> 172.21.15.59:0/534060062 conn(0x7f82bc0b2660 msgr2=0x7f82bc0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.068+0000 7f82c2078640 1 -- 172.21.15.59:0/534060062 shutdown_connections 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.068+0000 7f82c2078640 1 -- 172.21.15.59:0/534060062 wait complete. 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.069+0000 7f82c2078640 1 Processor -- start 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.069+0000 7f82c2078640 1 -- start start 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82c2078640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82bc071a70 0x7f82bc1575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82c2078640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82bc0b4110 0x7f82bc0c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82c2078640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82bc15ad90 0x7f82bc0c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82c2078640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0b4560 con 0x7f82bc071a70 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82c2078640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc15bb00 con 0x7f82bc15ad90 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82c2078640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc0b46e0 con 0x7f82bc0b4110 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82bc071a70 0x7f82bc1575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82bc071a70 0x7f82bc1575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38122/0 (socket says 172.21.15.59:38122) 2024-03-06T07:23:50.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82bb7fe640 1 -- 172.21.15.59:0/1471836963 learned_addr learned my addr 172.21.15.59:0/1471836963 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82bbfff640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82bc15ad90 0x7f82bc0c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82baffd640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82bc0b4110 0x7f82bc0c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82bb7fe640 1 -- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82bc0b4110 msgr2=0x7f82bc0c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82bb7fe640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82bc0b4110 0x7f82bc0c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82bb7fe640 1 -- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82bc15ad90 msgr2=0x7f82bc0c58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82bb7fe640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82bc15ad90 0x7f82bc0c58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.070+0000 7f82bb7fe640 1 -- 172.21.15.59:0/1471836963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0c6290 con 0x7f82bc071a70 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.071+0000 7f82bb7fe640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82bc071a70 0x7f82bc1575b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00e870 tx=0x7f82ac00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.071+0000 7f82a3fff640 1 -- 172.21.15.59:0/1471836963 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac017070 con 0x7f82bc071a70 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.071+0000 7f82a3fff640 1 -- 172.21.15.59:0/1471836963 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ac00ee60 con 0x7f82bc071a70 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.071+0000 7f82c2078640 1 -- 172.21.15.59:0/1471836963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc0c6570 con 0x7f82bc071a70 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.071+0000 7f82a3fff640 1 -- 172.21.15.59:0/1471836963 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac00cb80 con 0x7f82bc071a70 2024-03-06T07:23:50.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.071+0000 7f82c2078640 1 -- 172.21.15.59:0/1471836963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc0c6e10 con 0x7f82bc071a70 2024-03-06T07:23:50.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.072+0000 7f82c2078640 1 -- 172.21.15.59:0/1471836963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82bc073390 con 0x7f82bc071a70 2024-03-06T07:23:50.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.072+0000 7f82a3fff640 1 -- 172.21.15.59:0/1471836963 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82ac01a070 con 0x7f82bc071a70 2024-03-06T07:23:50.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.073+0000 7f82a3fff640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82a806b4d0 0x7f82a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.073+0000 7f82a3fff640 1 -- 172.21.15.59:0/1471836963 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f82ac08a3a0 con 0x7f82bc071a70 2024-03-06T07:23:50.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.073+0000 7f82baffd640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82a806b4d0 0x7f82a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.073+0000 7f82baffd640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82a806b4d0 0x7f82a806d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f82bc155900 tx=0x7f82b0009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.077+0000 7f82a3fff640 1 -- 172.21.15.59:0/1471836963 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82ac058b90 con 0x7f82bc071a70 2024-03-06T07:23:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.257+0000 7f82c2078640 1 -- 172.21.15.59:0/1471836963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f82bc0cc720 con 0x7f82bc071a70 2024-03-06T07:23:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.257+0000 7f82a3fff640 1 -- 172.21.15.59:0/1471836963 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v47) v1 ==== 104+0+11 (secure 0 0 0) 0x7f82ac05cc30 con 0x7f82bc071a70 2024-03-06T07:23:50.259 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:23:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 -- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82a806b4d0 msgr2=0x7f82a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82a806b4d0 0x7f82a806d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f82bc155900 tx=0x7f82b0009190 comp rx=0 tx=0).stop 2024-03-06T07:23:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 -- 172.21.15.59:0/1471836963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82bc071a70 msgr2=0x7f82bc1575b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82bc071a70 0x7f82bc1575b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00e870 tx=0x7f82ac00ed30 comp rx=0 tx=0).stop 2024-03-06T07:23:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 -- 172.21.15.59:0/1471836963 shutdown_connections 2024-03-06T07:23:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82a806b4d0 0x7f82a806d990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82bc15ad90 0x7f82bc0c58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82bc0b4110 0x7f82bc0c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 --2- 172.21.15.59:0/1471836963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82bc071a70 0x7f82bc1575b0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 -- 172.21.15.59:0/1471836963 >> 172.21.15.59:0/1471836963 conn(0x7f82bc0b2660 msgr2=0x7f82bc15dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 -- 172.21.15.59:0/1471836963 shutdown_connections 2024-03-06T07:23:50.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.259+0000 7f82c2078640 1 -- 172.21.15.59:0/1471836963 wait complete. 2024-03-06T07:23:50.272 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:50.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.375+0000 7f6603037640 1 Processor -- start 2024-03-06T07:23:50.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.375+0000 7f6603037640 1 -- start start 2024-03-06T07:23:50.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f6603037640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f65fc1614d0 0x7f65fc0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f6603037640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65fc0d24b0 0x7f65fc0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f6603037640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65fc1618b0 con 0x7f65fc0d1910 2024-03-06T07:23:50.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f6603037640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65fc0cf400 con 0x7f65fc1614d0 2024-03-06T07:23:50.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f6603037640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0cf6a0 con 0x7f65fc0d24b0 2024-03-06T07:23:50.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f65fbfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f65fc0d1910 0x7f65fc0d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50476/0 (socket says 172.21.15.59:50476) 2024-03-06T07:23:50.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f65fbfff640 1 -- 172.21.15.59:0/1725224292 learned_addr learned my addr 172.21.15.59:0/1725224292 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:50.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f6600dd3640 1 --2- 172.21.15.59:0/1725224292 >> v2:172.21.15.164:3300/0 conn(0x7f65fc1614d0 0x7f65fc0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f66015d4640 1 --2- 172.21.15.59:0/1725224292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65fc0d24b0 0x7f65fc0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f65fb7fe640 1 -- 172.21.15.59:0/1725224292 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561099449 0 0) 0x7f65fc1618b0 con 0x7f65fc0d1910 2024-03-06T07:23:50.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f65fb7fe640 1 -- 172.21.15.59:0/1725224292 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65e4003520 con 0x7f65fc0d1910 2024-03-06T07:23:50.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f6600dd3640 1 -- 172.21.15.59:0/1725224292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65fc0d24b0 msgr2=0x7f65fc0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f6600dd3640 1 --2- 172.21.15.59:0/1725224292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65fc0d24b0 0x7f65fc0b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f6600dd3640 1 -- 172.21.15.59:0/1725224292 >> v1:172.21.15.59:6789/0 conn(0x7f65fc0d1910 legacy=0x7f65fc0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.376+0000 7f6600dd3640 1 -- 172.21.15.59:0/1725224292 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc0cf980 con 0x7f65fc1614d0 2024-03-06T07:23:50.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.377+0000 7f6600dd3640 1 --2- 172.21.15.59:0/1725224292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65fc1614d0 0x7f65fc0d4860 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0097a0 tx=0x7f65ec02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d192f7ff4966d5d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.377+0000 7f65fb7fe640 1 -- 172.21.15.59:0/1725224292 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec035020 con 0x7f65fc1614d0 2024-03-06T07:23:50.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.377+0000 7f65fb7fe640 1 -- 172.21.15.59:0/1725224292 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec031830 con 0x7f65fc1614d0 2024-03-06T07:23:50.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.377+0000 7f65fb7fe640 1 -- 172.21.15.59:0/1725224292 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec031be0 con 0x7f65fc1614d0 2024-03-06T07:23:50.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.378+0000 7f6603037640 1 -- 172.21.15.59:0/1725224292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65fc1614d0 msgr2=0x7f65fc0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.378+0000 7f6603037640 1 --2- 172.21.15.59:0/1725224292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65fc1614d0 0x7f65fc0d4860 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0097a0 tx=0x7f65ec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.378+0000 7f6603037640 1 -- 172.21.15.59:0/1725224292 shutdown_connections 2024-03-06T07:23:50.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.378+0000 7f6603037640 1 --2- 172.21.15.59:0/1725224292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65fc0d24b0 0x7f65fc0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.378+0000 7f6603037640 1 --2- 172.21.15.59:0/1725224292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65fc1614d0 0x7f65fc0d4860 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.378+0000 7f6603037640 1 -- 172.21.15.59:0/1725224292 >> 172.21.15.59:0/1725224292 conn(0x7f65fc0d3d40 msgr2=0x7f65fc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.378+0000 7f6603037640 1 -- 172.21.15.59:0/1725224292 shutdown_connections 2024-03-06T07:23:50.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.378+0000 7f6603037640 1 -- 172.21.15.59:0/1725224292 wait complete. 2024-03-06T07:23:50.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.379+0000 7f6603037640 1 Processor -- start 2024-03-06T07:23:50.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.379+0000 7f6603037640 1 -- start start 2024-03-06T07:23:50.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f6603037640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65fc0d1910 0x7f65fc0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f6603037640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65fc0d24b0 0x7f65fc1f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f6603037640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65fc1614d0 0x7f65fc06bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f6603037640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0b1370 con 0x7f65fc1614d0 2024-03-06T07:23:50.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f6603037640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0b0910 con 0x7f65fc0d1910 2024-03-06T07:23:50.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f6603037640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0b14f0 con 0x7f65fc0d24b0 2024-03-06T07:23:50.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f66015d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65fc1614d0 0x7f65fc06bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f66015d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65fc1614d0 0x7f65fc06bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38128/0 (socket says 172.21.15.59:38128) 2024-03-06T07:23:50.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f66015d4640 1 -- 172.21.15.59:0/3230132035 learned_addr learned my addr 172.21.15.59:0/3230132035 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:50.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f6600dd3640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65fc0d1910 0x7f65fc0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f65fbfff640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65fc0d24b0 0x7f65fc1f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f66015d4640 1 -- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65fc0d24b0 msgr2=0x7f65fc1f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f66015d4640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65fc0d24b0 0x7f65fc1f46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f66015d4640 1 -- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65fc0d1910 msgr2=0x7f65fc0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f66015d4640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65fc0d1910 0x7f65fc0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.380+0000 7f66015d4640 1 -- 172.21.15.59:0/3230132035 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc1f4c20 con 0x7f65fc1614d0 2024-03-06T07:23:50.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.381+0000 7f66015d4640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65fc1614d0 0x7f65fc06bba0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009dd0 tx=0x7f65f400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.381+0000 7f65f97fa640 1 -- 172.21.15.59:0/3230132035 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4018020 con 0x7f65fc1614d0 2024-03-06T07:23:50.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.381+0000 7f65f97fa640 1 -- 172.21.15.59:0/3230132035 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f4014480 con 0x7f65fc1614d0 2024-03-06T07:23:50.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.381+0000 7f6603037640 1 -- 172.21.15.59:0/3230132035 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65fc0b8490 con 0x7f65fc1614d0 2024-03-06T07:23:50.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.381+0000 7f65f97fa640 1 -- 172.21.15.59:0/3230132035 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f4014870 con 0x7f65fc1614d0 2024-03-06T07:23:50.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.381+0000 7f6603037640 1 -- 172.21.15.59:0/3230132035 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65fc0b8cd0 con 0x7f65fc1614d0 2024-03-06T07:23:50.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.382+0000 7f6603037640 1 -- 172.21.15.59:0/3230132035 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65fc068ff0 con 0x7f65fc1614d0 2024-03-06T07:23:50.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.382+0000 7f65f97fa640 1 -- 172.21.15.59:0/3230132035 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65f4005d10 con 0x7f65fc1614d0 2024-03-06T07:23:50.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.382+0000 7f65f97fa640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65e806b4d0 0x7f65e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.383+0000 7f65f97fa640 1 -- 172.21.15.59:0/3230132035 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f65f408ac10 con 0x7f65fc1614d0 2024-03-06T07:23:50.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.383+0000 7f6600dd3640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65e806b4d0 0x7f65e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.383+0000 7f6600dd3640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65e806b4d0 0x7f65e806d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f65ec036000 tx=0x7f65ec03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.387+0000 7f65f97fa640 1 -- 172.21.15.59:0/3230132035 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65f4022070 con 0x7f65fc1614d0 2024-03-06T07:23:50.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.567+0000 7f6603037640 1 -- 172.21.15.59:0/3230132035 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f65fc06cb70 con 0x7f65fc1614d0 2024-03-06T07:23:50.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.567+0000 7f65f97fa640 1 -- 172.21.15.59:0/3230132035 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v47) v1 ==== 100+0+11 (secure 0 0 0) 0x7f65fc0b8490 con 0x7f65fc1614d0 2024-03-06T07:23:50.569 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:23:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 -- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65e806b4d0 msgr2=0x7f65e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65e806b4d0 0x7f65e806d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f65ec036000 tx=0x7f65ec03c040 comp rx=0 tx=0).stop 2024-03-06T07:23:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 -- 172.21.15.59:0/3230132035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65fc1614d0 msgr2=0x7f65fc06bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65fc1614d0 0x7f65fc06bba0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009dd0 tx=0x7f65f400c490 comp rx=0 tx=0).stop 2024-03-06T07:23:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 -- 172.21.15.59:0/3230132035 shutdown_connections 2024-03-06T07:23:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65e806b4d0 0x7f65e806d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65fc1614d0 0x7f65fc06bba0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65fc0d24b0 0x7f65fc1f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 --2- 172.21.15.59:0/3230132035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65fc0d1910 0x7f65fc0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 -- 172.21.15.59:0/3230132035 >> 172.21.15.59:0/3230132035 conn(0x7f65fc0d3d40 msgr2=0x7f65fc06cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 -- 172.21.15.59:0/3230132035 shutdown_connections 2024-03-06T07:23:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.569+0000 7f6603037640 1 -- 172.21.15.59:0/3230132035 wait complete. 2024-03-06T07:23:50.582 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:50.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.681+0000 7f3b07c2a640 1 Processor -- start 2024-03-06T07:23:50.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.681+0000 7f3b07c2a640 1 -- start start 2024-03-06T07:23:50.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.682+0000 7f3b07c2a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3b000d00c0 0x7f3b000d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.682+0000 7f3b07c2a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b000d09e0 con 0x7f3b0006cf20 2024-03-06T07:23:50.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.682+0000 7f3b07c2a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0015e380 con 0x7f3b000cf850 2024-03-06T07:23:50.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.682+0000 7f3b07c2a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0006ca90 con 0x7f3b000d00c0 2024-03-06T07:23:50.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.682+0000 7f3b061c7640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3b0006cf20 0x7f3b0006d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50480/0 (socket says 172.21.15.59:50480) 2024-03-06T07:23:50.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.682+0000 7f3b059c6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3b000d00c0 0x7f3b000d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.682+0000 7f3b061c7640 1 -- 172.21.15.59:0/2407217123 learned_addr learned my addr 172.21.15.59:0/2407217123 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:50.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.682+0000 7f3b059c6640 1 -- 172.21.15.59:0/2407217123 >> v1:172.21.15.164:6789/0 conn(0x7f3b000cf850 legacy=0x7f3b000cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.683+0000 7f3b059c6640 1 -- 172.21.15.59:0/2407217123 >> v1:172.21.15.59:6789/0 conn(0x7f3b0006cf20 legacy=0x7f3b0006d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.683+0000 7f3b049c4640 1 -- 172.21.15.59:0/2407217123 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201807192 0 0) 0x7f3b0015e380 con 0x7f3b000cf850 2024-03-06T07:23:50.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.683+0000 7f3b059c6640 1 -- 172.21.15.59:0/2407217123 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b000cdc10 con 0x7f3b000d00c0 2024-03-06T07:23:50.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.683+0000 7f3b059c6640 1 --2- 172.21.15.59:0/2407217123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b000d00c0 0x7f3b000d04a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3af80097a0 tx=0x7f3af802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6f0ca4a995eed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.683+0000 7f3b049c4640 1 -- 172.21.15.59:0/2407217123 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af8037070 con 0x7f3b000d00c0 2024-03-06T07:23:50.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.683+0000 7f3b049c4640 1 -- 172.21.15.59:0/2407217123 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af8030710 con 0x7f3b000d00c0 2024-03-06T07:23:50.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.683+0000 7f3b049c4640 1 -- 172.21.15.59:0/2407217123 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af8030ae0 con 0x7f3b000d00c0 2024-03-06T07:23:50.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.684+0000 7f3b07c2a640 1 -- 172.21.15.59:0/2407217123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b000d00c0 msgr2=0x7f3b000d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.684+0000 7f3b07c2a640 1 --2- 172.21.15.59:0/2407217123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b000d00c0 0x7f3b000d04a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3af80097a0 tx=0x7f3af802b790 comp rx=0 tx=0).stop 2024-03-06T07:23:50.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.684+0000 7f3b07c2a640 1 -- 172.21.15.59:0/2407217123 shutdown_connections 2024-03-06T07:23:50.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.684+0000 7f3b07c2a640 1 --2- 172.21.15.59:0/2407217123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b000d00c0 0x7f3b000d04a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.684+0000 7f3b07c2a640 1 -- 172.21.15.59:0/2407217123 >> 172.21.15.59:0/2407217123 conn(0x7f3b000b35a0 msgr2=0x7f3b000b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.684+0000 7f3b07c2a640 1 -- 172.21.15.59:0/2407217123 shutdown_connections 2024-03-06T07:23:50.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.684+0000 7f3b07c2a640 1 -- 172.21.15.59:0/2407217123 wait complete. 2024-03-06T07:23:50.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.685+0000 7f3b07c2a640 1 Processor -- start 2024-03-06T07:23:50.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.685+0000 7f3b07c2a640 1 -- start start 2024-03-06T07:23:50.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b07c2a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b0006cf20 0x7f3b0015a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b07c2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b000cf850 0x7f3b000c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b07c2a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b000d00c0 0x7f3b000b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b07c2a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0006c720 con 0x7f3b000cf850 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b07c2a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0006c440 con 0x7f3b000d00c0 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b07c2a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b000b1030 con 0x7f3b0006cf20 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b051c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b000cf850 0x7f3b000c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b051c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b000cf850 0x7f3b000c73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38138/0 (socket says 172.21.15.59:38138) 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b051c5640 1 -- 172.21.15.59:0/1183263421 learned_addr learned my addr 172.21.15.59:0/1183263421 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b061c7640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b000d00c0 0x7f3b000b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b059c6640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b0006cf20 0x7f3b0015a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b051c5640 1 -- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b0006cf20 msgr2=0x7f3b0015a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b051c5640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b0006cf20 0x7f3b0015a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b051c5640 1 -- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b000d00c0 msgr2=0x7f3b000b03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b051c5640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b000d00c0 0x7f3b000b03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.686+0000 7f3b051c5640 1 -- 172.21.15.59:0/1183263421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b000b5790 con 0x7f3b000cf850 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.687+0000 7f3b051c5640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b000cf850 0x7f3b000c73d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3af400c4b0 tx=0x7f3af400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.687+0000 7f3afe7fc640 1 -- 172.21.15.59:0/1183263421 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4017070 con 0x7f3b000cf850 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.687+0000 7f3afe7fc640 1 -- 172.21.15.59:0/1183263421 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af4005480 con 0x7f3b000cf850 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.687+0000 7f3b07c2a640 1 -- 172.21.15.59:0/1183263421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b000b46f0 con 0x7f3b000cf850 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.687+0000 7f3afe7fc640 1 -- 172.21.15.59:0/1183263421 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4003830 con 0x7f3b000cf850 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.687+0000 7f3b07c2a640 1 -- 172.21.15.59:0/1183263421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b000b4cb0 con 0x7f3b000cf850 2024-03-06T07:23:50.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.688+0000 7f3b07c2a640 1 -- 172.21.15.59:0/1183263421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b000ca380 con 0x7f3b000cf850 2024-03-06T07:23:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.688+0000 7f3afe7fc640 1 -- 172.21.15.59:0/1183263421 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3af401a070 con 0x7f3b000cf850 2024-03-06T07:23:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.688+0000 7f3afe7fc640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aec06b500 0x7f3aec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.689+0000 7f3afe7fc640 1 -- 172.21.15.59:0/1183263421 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3af4089510 con 0x7f3b000cf850 2024-03-06T07:23:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.689+0000 7f3b059c6640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aec06b500 0x7f3aec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.689+0000 7f3b059c6640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aec06b500 0x7f3aec06d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3af8034000 tx=0x7f3af802f000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.693+0000 7f3afe7fc640 1 -- 172.21.15.59:0/1183263421 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3af4057d00 con 0x7f3b000cf850 2024-03-06T07:23:50.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.915+0000 7f3b07c2a640 1 -- 172.21.15.59:0/1183263421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b000cbf30 con 0x7f3b000cf850 2024-03-06T07:23:50.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.916+0000 7f3afe7fc640 1 -- 172.21.15.59:0/1183263421 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f3af4087e20 con 0x7f3b000cf850 2024-03-06T07:23:50.918 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:50.918 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5557,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/690078434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":690078434},{"type":"v1","addr":"172.21.15.164:6836","nonce":690078434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1345698247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1345698247},{"type":"v1","addr":"172.21.15.59:6833","nonce":1345698247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5685,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3875776524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3875776524},{"type":"v1","addr":"172.21.15.59:6835","nonce":3875776524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:28.400413+0000","modified":"2024-03-06T07:23:31.397689+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/3749315375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3749315375},{"type":"v1","addr":"172.21.15.164:6837","nonce":3749315375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-06T07:23:50.918 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 20 2024-03-06T07:23:50.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 -- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aec06b500 msgr2=0x7f3aec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aec06b500 0x7f3aec06d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3af8034000 tx=0x7f3af802f000 comp rx=0 tx=0).stop 2024-03-06T07:23:50.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 -- 172.21.15.59:0/1183263421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b000cf850 msgr2=0x7f3b000c73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b000cf850 0x7f3b000c73d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3af400c4b0 tx=0x7f3af400c970 comp rx=0 tx=0).stop 2024-03-06T07:23:50.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 -- 172.21.15.59:0/1183263421 shutdown_connections 2024-03-06T07:23:50.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aec06b500 0x7f3aec06d9c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b000d00c0 0x7f3b000b03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3b000cf850 0x7f3b000c73d0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 --2- 172.21.15.59:0/1183263421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b0006cf20 0x7f3b0015a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 -- 172.21.15.59:0/1183263421 >> 172.21.15.59:0/1183263421 conn(0x7f3b000b35a0 msgr2=0x7f3b000cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 -- 172.21.15.59:0/1183263421 shutdown_connections 2024-03-06T07:23:50.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:50.918+0000 7f3b07c2a640 1 -- 172.21.15.59:0/1183263421 wait complete. 2024-03-06T07:23:50.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:51.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.032+0000 7f26e5a04640 1 Processor -- start 2024-03-06T07:23:51.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.032+0000 7f26e5a04640 1 -- start start 2024-03-06T07:23:51.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.032+0000 7f26e5a04640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f26e006fa30 0x7f26e00d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.032+0000 7f26e5a04640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26e00d1ce0 0x7f26e00d0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.032+0000 7f26e5a04640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e006fec0 con 0x7f26e015dec0 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.032+0000 7f26e5a04640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e015e2a0 con 0x7f26e006fa30 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.032+0000 7f26e5a04640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e006cf70 con 0x7f26e00d1ce0 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.033+0000 7f26d7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f26e006fa30 0x7f26e00d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.033+0000 7f26deffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f26e015dec0 0x7f26e006c7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50496/0 (socket says 172.21.15.59:50496) 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.033+0000 7f26deffd640 1 -- 172.21.15.59:0/124088506 learned_addr learned my addr 172.21.15.59:0/124088506 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.033+0000 7f26df7fe640 1 --2- 172.21.15.59:0/124088506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26e00d1ce0 0x7f26e00d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.033+0000 7f26d7fff640 1 -- 172.21.15.59:0/124088506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26e00d1ce0 msgr2=0x7f26e00d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.033+0000 7f26d7fff640 1 --2- 172.21.15.59:0/124088506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26e00d1ce0 0x7f26e00d0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.033+0000 7f26d7fff640 1 -- 172.21.15.59:0/124088506 >> v1:172.21.15.59:6789/0 conn(0x7f26e015dec0 legacy=0x7f26e006c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.033+0000 7f26de7fc640 1 -- 172.21.15.59:0/124088506 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60674622 0 0) 0x7f26e006fec0 con 0x7f26e015dec0 2024-03-06T07:23:51.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.033+0000 7f26d7fff640 1 -- 172.21.15.59:0/124088506 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e006d250 con 0x7f26e006fa30 2024-03-06T07:23:51.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26d7fff640 1 --2- 172.21.15.59:0/124088506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e006fa30 0x7f26e00d17a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f26c80097a0 tx=0x7f26c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de02d851a187f1ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26de7fc640 1 -- 172.21.15.59:0/124088506 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8035020 con 0x7f26e006fa30 2024-03-06T07:23:51.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26de7fc640 1 -- 172.21.15.59:0/124088506 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c8031830 con 0x7f26e006fa30 2024-03-06T07:23:51.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26de7fc640 1 -- 172.21.15.59:0/124088506 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8031c00 con 0x7f26e006fa30 2024-03-06T07:23:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26e5a04640 1 -- 172.21.15.59:0/124088506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e006fa30 msgr2=0x7f26e00d17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26e5a04640 1 --2- 172.21.15.59:0/124088506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e006fa30 0x7f26e00d17a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f26c80097a0 tx=0x7f26c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26e5a04640 1 -- 172.21.15.59:0/124088506 shutdown_connections 2024-03-06T07:23:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26e5a04640 1 --2- 172.21.15.59:0/124088506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26e00d1ce0 0x7f26e00d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26e5a04640 1 --2- 172.21.15.59:0/124088506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e006fa30 0x7f26e00d17a0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26e5a04640 1 -- 172.21.15.59:0/124088506 >> 172.21.15.59:0/124088506 conn(0x7f26e009a9f0 msgr2=0x7f26e009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26e5a04640 1 -- 172.21.15.59:0/124088506 shutdown_connections 2024-03-06T07:23:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.034+0000 7f26e5a04640 1 -- 172.21.15.59:0/124088506 wait complete. 2024-03-06T07:23:51.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.036+0000 7f26e5a04640 1 Processor -- start 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.036+0000 7f26e5a04640 1 -- start start 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.036+0000 7f26e5a04640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e006fa30 0x7f26e00b5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.036+0000 7f26e5a04640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26e00d1ce0 0x7f26e01f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.036+0000 7f26e5a04640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26e015dec0 0x7f26e015c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.036+0000 7f26e5a04640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00cfc60 con 0x7f26e006fa30 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.036+0000 7f26e5a04640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e006d810 con 0x7f26e015dec0 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.036+0000 7f26e5a04640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00cfde0 con 0x7f26e00d1ce0 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.036+0000 7f26deffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e006fa30 0x7f26e00b5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.036+0000 7f26deffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e006fa30 0x7f26e00b5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38164/0 (socket says 172.21.15.59:38164) 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.036+0000 7f26deffd640 1 -- 172.21.15.59:0/2313875116 learned_addr learned my addr 172.21.15.59:0/2313875116 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26d7fff640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26e00d1ce0 0x7f26e01f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26df7fe640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26e015dec0 0x7f26e015c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26deffd640 1 -- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26e00d1ce0 msgr2=0x7f26e01f43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26deffd640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26e00d1ce0 0x7f26e01f43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26deffd640 1 -- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26e015dec0 msgr2=0x7f26e015c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26deffd640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26e015dec0 0x7f26e015c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26deffd640 1 -- 172.21.15.59:0/2313875116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e00b4510 con 0x7f26e006fa30 2024-03-06T07:23:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26deffd640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e006fa30 0x7f26e00b5ef0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f26d000c460 tx=0x7f26d000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26dcff9640 1 -- 172.21.15.59:0/2313875116 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0018070 con 0x7f26e006fa30 2024-03-06T07:23:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26dcff9640 1 -- 172.21.15.59:0/2313875116 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d0007500 con 0x7f26e006fa30 2024-03-06T07:23:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26e5a04640 1 -- 172.21.15.59:0/2313875116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e00b47f0 con 0x7f26e006fa30 2024-03-06T07:23:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26dcff9640 1 -- 172.21.15.59:0/2313875116 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0004a20 con 0x7f26e006fa30 2024-03-06T07:23:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.037+0000 7f26e5a04640 1 -- 172.21.15.59:0/2313875116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e00b4db0 con 0x7f26e006fa30 2024-03-06T07:23:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.038+0000 7f26e5a04640 1 -- 172.21.15.59:0/2313875116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e0068ff0 con 0x7f26e006fa30 2024-03-06T07:23:51.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.039+0000 7f26dcff9640 1 -- 172.21.15.59:0/2313875116 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26d001b070 con 0x7f26e006fa30 2024-03-06T07:23:51.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.039+0000 7f26dcff9640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f26c006b4d0 0x7f26c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.039+0000 7f26dcff9640 1 -- 172.21.15.59:0/2313875116 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f26d0089470 con 0x7f26e006fa30 2024-03-06T07:23:51.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.039+0000 7f26d7fff640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f26c006b4d0 0x7f26c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.040+0000 7f26d7fff640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f26c006b4d0 0x7f26c006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f26c8036000 tx=0x7f26c803d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.043+0000 7f26dcff9640 1 -- 172.21.15.59:0/2313875116 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26d0057c50 con 0x7f26e006fa30 2024-03-06T07:23:51.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.224+0000 7f26e5a04640 1 -- 172.21.15.59:0/2313875116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26e006c360 con 0x7f26e006fa30 2024-03-06T07:23:51.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.225+0000 7f26dcff9640 1 -- 172.21.15.59:0/2313875116 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16752 (secure 0 0 0) 0x7f26d005bcf0 con 0x7f26e006fa30 2024-03-06T07:23:51.227 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:51.227 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":47,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:23:37.063479+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:26.672668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:27.362891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":0,"to":5}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 -- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f26c006b4d0 msgr2=0x7f26c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f26c006b4d0 0x7f26c006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f26c8036000 tx=0x7f26c803d040 comp rx=0 tx=0).stop 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 -- 172.21.15.59:0/2313875116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e006fa30 msgr2=0x7f26e00b5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e006fa30 0x7f26e00b5ef0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f26d000c460 tx=0x7f26d000c920 comp rx=0 tx=0).stop 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 -- 172.21.15.59:0/2313875116 shutdown_connections 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f26c006b4d0 0x7f26c006d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26e015dec0 0x7f26e015c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26e00d1ce0 0x7f26e01f43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 --2- 172.21.15.59:0/2313875116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e006fa30 0x7f26e00b5ef0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 -- 172.21.15.59:0/2313875116 >> 172.21.15.59:0/2313875116 conn(0x7f26e009a9f0 msgr2=0x7f26e00b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 -- 172.21.15.59:0/2313875116 shutdown_connections 2024-03-06T07:23:51.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.227+0000 7f26e5a04640 1 -- 172.21.15.59:0/2313875116 wait complete. 2024-03-06T07:23:51.240 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:51.240 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:51.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.341+0000 7f5f3d6f4640 1 Processor -- start 2024-03-06T07:23:51.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.341+0000 7f5f3d6f4640 1 -- start start 2024-03-06T07:23:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.341+0000 7f5f3d6f4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5f3813dc10 0x7f5f380b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.341+0000 7f5f3d6f4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5f3815ef60 0x7f5f380b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.341+0000 7f5f3d6f4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3815f580 con 0x7f5f3815bf30 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.341+0000 7f5f3d6f4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f380cf3f0 con 0x7f5f3815ef60 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.341+0000 7f5f3d6f4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3815b210 con 0x7f5f3813dc10 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.342+0000 7f5f377fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5f3815ef60 0x7f5f380b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.342+0000 7f5f377fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5f3815ef60 0x7f5f380b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38180/0 (socket says 172.21.15.59:38180) 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.342+0000 7f5f377fe640 1 -- 172.21.15.59:0/1166625248 learned_addr learned my addr 172.21.15.59:0/1166625248 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.342+0000 7f5f36ffd640 1 --2- 172.21.15.59:0/1166625248 >> v2:172.21.15.164:3300/0 conn(0x7f5f3813dc10 0x7f5f380b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.342+0000 7f5f377fe640 1 -- 172.21.15.59:0/1166625248 >> v2:172.21.15.164:3300/0 conn(0x7f5f3813dc10 msgr2=0x7f5f380b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.342+0000 7f5f377fe640 1 --2- 172.21.15.59:0/1166625248 >> v2:172.21.15.164:3300/0 conn(0x7f5f3813dc10 0x7f5f380b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.342+0000 7f5f377fe640 1 -- 172.21.15.59:0/1166625248 >> v1:172.21.15.164:6789/0 conn(0x7f5f3815bf30 legacy=0x7f5f3815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.342+0000 7f5f377fe640 1 -- 172.21.15.59:0/1166625248 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3815b450 con 0x7f5f3815ef60 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f377fe640 1 --2- 172.21.15.59:0/1166625248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3815ef60 0x7f5f380b2e90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5f380731a0 tx=0x7f5f3002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37e89e36326afb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f367fc640 1 -- 172.21.15.59:0/1166625248 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3002ca80 con 0x7f5f3815ef60 2024-03-06T07:23:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f367fc640 1 -- 172.21.15.59:0/1166625248 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3002cc20 con 0x7f5f3815ef60 2024-03-06T07:23:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f367fc640 1 -- 172.21.15.59:0/1166625248 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f300343f0 con 0x7f5f3815ef60 2024-03-06T07:23:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/1166625248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3815ef60 msgr2=0x7f5f380b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f3d6f4640 1 --2- 172.21.15.59:0/1166625248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3815ef60 0x7f5f380b2e90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5f380731a0 tx=0x7f5f3002bee0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/1166625248 shutdown_connections 2024-03-06T07:23:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f3d6f4640 1 --2- 172.21.15.59:0/1166625248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3815ef60 0x7f5f380b2e90 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f3d6f4640 1 --2- 172.21.15.59:0/1166625248 >> v2:172.21.15.164:3300/0 conn(0x7f5f3813dc10 0x7f5f380b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/1166625248 >> 172.21.15.59:0/1166625248 conn(0x7f5f380cf8a0 msgr2=0x7f5f380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/1166625248 shutdown_connections 2024-03-06T07:23:51.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.343+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/1166625248 wait complete. 2024-03-06T07:23:51.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f3d6f4640 1 Processor -- start 2024-03-06T07:23:51.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f3d6f4640 1 -- start start 2024-03-06T07:23:51.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f3d6f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3813dc10 0x7f5f3806dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f3d6f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f3815bf30 0x7f5f38153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f3d6f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f3815ef60 0x7f5f38143df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f3d6f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f380708d0 con 0x7f5f3813dc10 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f3d6f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3813dff0 con 0x7f5f3815ef60 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f3d6f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f38070a50 con 0x7f5f3815bf30 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f36ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3813dc10 0x7f5f3806dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f36ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3813dc10 0x7f5f3806dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38194/0 (socket says 172.21.15.59:38194) 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f36ffd640 1 -- 172.21.15.59:0/132758977 learned_addr learned my addr 172.21.15.59:0/132758977 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f377fe640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f3815ef60 0x7f5f38143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.345+0000 7f5f2ffff640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f3815bf30 0x7f5f38153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f36ffd640 1 -- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f3815bf30 msgr2=0x7f5f38153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f36ffd640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f3815bf30 0x7f5f38153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f36ffd640 1 -- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f3815ef60 msgr2=0x7f5f38143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f36ffd640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f3815ef60 0x7f5f38143df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f36ffd640 1 -- 172.21.15.59:0/132758977 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f38138e30 con 0x7f5f3813dc10 2024-03-06T07:23:51.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f2ffff640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f3815bf30 0x7f5f38153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:51.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f36ffd640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3813dc10 0x7f5f3806dbf0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800e9b0 tx=0x7f5f2800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f2f7fe640 1 -- 172.21.15.59:0/132758977 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f28017070 con 0x7f5f3813dc10 2024-03-06T07:23:51.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f2f7fe640 1 -- 172.21.15.59:0/132758977 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2800c9a0 con 0x7f5f3813dc10 2024-03-06T07:23:51.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/132758977 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f38139110 con 0x7f5f3813dc10 2024-03-06T07:23:51.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f2f7fe640 1 -- 172.21.15.59:0/132758977 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2800cd90 con 0x7f5f3813dc10 2024-03-06T07:23:51.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.346+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/132758977 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f381396d0 con 0x7f5f3813dc10 2024-03-06T07:23:51.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.347+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/132758977 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3804f330 con 0x7f5f3813dc10 2024-03-06T07:23:51.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.347+0000 7f5f2f7fe640 1 -- 172.21.15.59:0/132758977 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f2801a070 con 0x7f5f3813dc10 2024-03-06T07:23:51.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.348+0000 7f5f2f7fe640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1806b4d0 0x7f5f1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.348+0000 7f5f2f7fe640 1 -- 172.21.15.59:0/132758977 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5f2808a570 con 0x7f5f3813dc10 2024-03-06T07:23:51.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.348+0000 7f5f2ffff640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1806b4d0 0x7f5f1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.349+0000 7f5f2ffff640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1806b4d0 0x7f5f1806d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5f3813d400 tx=0x7f5f20009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.352+0000 7f5f2f7fe640 1 -- 172.21.15.59:0/132758977 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f28058d60 con 0x7f5f3813dc10 2024-03-06T07:23:51.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.570+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/132758977 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f3806eb00 con 0x7f5f3813dc10 2024-03-06T07:23:51.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.571+0000 7f5f2f7fe640 1 -- 172.21.15.59:0/132758977 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f5f2805ce00 con 0x7f5f3813dc10 2024-03-06T07:23:51.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:51.573 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5557,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/690078434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":690078434},{"type":"v1","addr":"172.21.15.164:6836","nonce":690078434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1345698247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1345698247},{"type":"v1","addr":"172.21.15.59:6833","nonce":1345698247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5685,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3875776524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3875776524},{"type":"v1","addr":"172.21.15.59:6835","nonce":3875776524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:28.400413+0000","modified":"2024-03-06T07:23:31.397689+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/3749315375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3749315375},{"type":"v1","addr":"172.21.15.164:6837","nonce":3749315375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-06T07:23:51.573 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 20 2024-03-06T07:23:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1806b4d0 msgr2=0x7f5f1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1806b4d0 0x7f5f1806d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5f3813d400 tx=0x7f5f20009040 comp rx=0 tx=0).stop 2024-03-06T07:23:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/132758977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3813dc10 msgr2=0x7f5f3806dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3813dc10 0x7f5f3806dbf0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800e9b0 tx=0x7f5f2800ee70 comp rx=0 tx=0).stop 2024-03-06T07:23:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/132758977 shutdown_connections 2024-03-06T07:23:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1806b4d0 0x7f5f1806d990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f3815ef60 0x7f5f38143df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f3815bf30 0x7f5f38153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 --2- 172.21.15.59:0/132758977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f3813dc10 0x7f5f3806dbf0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/132758977 >> 172.21.15.59:0/132758977 conn(0x7f5f380cf8a0 msgr2=0x7f5f380af980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/132758977 shutdown_connections 2024-03-06T07:23:51.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.573+0000 7f5f3d6f4640 1 -- 172.21.15.59:0/132758977 wait complete. 2024-03-06T07:23:51.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.685+0000 7ff70b9c9640 1 Processor -- start 2024-03-06T07:23:51.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.685+0000 7ff70b9c9640 1 -- start start 2024-03-06T07:23:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff70b9c9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff7040cf850 0x7ff7040cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff70b9c9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70415e380 con 0x7ff7040d00c0 2024-03-06T07:23:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff70b9c9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7040cdc10 con 0x7ff70406cf20 2024-03-06T07:23:51.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff70b9c9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70406ca90 con 0x7ff7040cf850 2024-03-06T07:23:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff709765640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff7040d00c0 0x7ff7040d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50508/0 (socket says 172.21.15.59:50508) 2024-03-06T07:23:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff709765640 1 -- 172.21.15.59:0/3563094416 learned_addr learned my addr 172.21.15.59:0/3563094416 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff708f64640 1 --2- 172.21.15.59:0/3563094416 >> v2:172.21.15.164:3300/0 conn(0x7ff7040cf850 0x7ff7040cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff6fbfff640 1 -- 172.21.15.59:0/3563094416 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47375699 0 0) 0x7ff70415e380 con 0x7ff7040d00c0 2024-03-06T07:23:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff6fbfff640 1 -- 172.21.15.59:0/3563094416 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f0003520 con 0x7ff7040d00c0 2024-03-06T07:23:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff708f64640 1 -- 172.21.15.59:0/3563094416 >> v1:172.21.15.164:6789/0 conn(0x7ff70406cf20 legacy=0x7ff70406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff708f64640 1 -- 172.21.15.59:0/3563094416 >> v1:172.21.15.59:6789/0 conn(0x7ff7040d00c0 legacy=0x7ff7040d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff6fbfff640 1 -- 172.21.15.59:0/3563094416 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975203412 0 0) 0x7ff7040cdc10 con 0x7ff70406cf20 2024-03-06T07:23:51.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.686+0000 7ff708f64640 1 -- 172.21.15.59:0/3563094416 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70406c160 con 0x7ff7040cf850 2024-03-06T07:23:51.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.687+0000 7ff708f64640 1 --2- 172.21.15.59:0/3563094416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7040cf850 0x7ff7040cd3c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40097a0 tx=0x7ff6f402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=632afe172a74e59d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.687+0000 7ff6fbfff640 1 -- 172.21.15.59:0/3563094416 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4034070 con 0x7ff7040cf850 2024-03-06T07:23:51.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.687+0000 7ff6fbfff640 1 -- 172.21.15.59:0/3563094416 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f4030610 con 0x7ff7040cf850 2024-03-06T07:23:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.687+0000 7ff6fbfff640 1 -- 172.21.15.59:0/3563094416 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f40309c0 con 0x7ff7040cf850 2024-03-06T07:23:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.688+0000 7ff70b9c9640 1 -- 172.21.15.59:0/3563094416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7040cf850 msgr2=0x7ff7040cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.688+0000 7ff70b9c9640 1 --2- 172.21.15.59:0/3563094416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7040cf850 0x7ff7040cd3c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40097a0 tx=0x7ff6f402b960 comp rx=0 tx=0).stop 2024-03-06T07:23:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.688+0000 7ff70b9c9640 1 -- 172.21.15.59:0/3563094416 shutdown_connections 2024-03-06T07:23:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.688+0000 7ff70b9c9640 1 --2- 172.21.15.59:0/3563094416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7040cf850 0x7ff7040cd3c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.688+0000 7ff70b9c9640 1 -- 172.21.15.59:0/3563094416 >> 172.21.15.59:0/3563094416 conn(0x7ff7040b35a0 msgr2=0x7ff7040b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.688+0000 7ff70b9c9640 1 -- 172.21.15.59:0/3563094416 shutdown_connections 2024-03-06T07:23:51.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.688+0000 7ff70b9c9640 1 -- 172.21.15.59:0/3563094416 wait complete. 2024-03-06T07:23:51.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.689+0000 7ff70b9c9640 1 Processor -- start 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.689+0000 7ff70b9c9640 1 -- start start 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff70b9c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff70406cf20 0x7ff7040cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff70b9c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7040cf850 0x7ff704154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff70b9c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7040d00c0 0x7ff7040c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff70b9c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040cd900 con 0x7ff70406cf20 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff70b9c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70406c720 con 0x7ff7040d00c0 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff70b9c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040b1030 con 0x7ff7040cf850 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff709765640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff70406cf20 0x7ff7040cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff709765640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff70406cf20 0x7ff7040cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38208/0 (socket says 172.21.15.59:38208) 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff709765640 1 -- 172.21.15.59:0/264824685 learned_addr learned my addr 172.21.15.59:0/264824685 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff708f64640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7040cf850 0x7ff704154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff709f66640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7040d00c0 0x7ff7040c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff709765640 1 -- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7040cf850 msgr2=0x7ff704154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff709765640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7040cf850 0x7ff704154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff709765640 1 -- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7040d00c0 msgr2=0x7ff7040c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff709765640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7040d00c0 0x7ff7040c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff709765640 1 -- 172.21.15.59:0/264824685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff704152a00 con 0x7ff70406cf20 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.690+0000 7ff709765640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff70406cf20 0x7ff7040cb2e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c3c0 tx=0x7ff6fc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.691+0000 7ff6f9ffb640 1 -- 172.21.15.59:0/264824685 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc017070 con 0x7ff70406cf20 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.691+0000 7ff6f9ffb640 1 -- 172.21.15.59:0/264824685 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6fc005ce0 con 0x7ff70406cf20 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.691+0000 7ff70b9c9640 1 -- 172.21.15.59:0/264824685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70415a7b0 con 0x7ff70406cf20 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.691+0000 7ff6f9ffb640 1 -- 172.21.15.59:0/264824685 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc0072f0 con 0x7ff70406cf20 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.691+0000 7ff70b9c9640 1 -- 172.21.15.59:0/264824685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff704152f50 con 0x7ff70406cf20 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.692+0000 7ff70b9c9640 1 -- 172.21.15.59:0/264824685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff70404f330 con 0x7ff70406cf20 2024-03-06T07:23:51.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.692+0000 7ff6f9ffb640 1 -- 172.21.15.59:0/264824685 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6fc01a070 con 0x7ff70406cf20 2024-03-06T07:23:51.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.692+0000 7ff6f9ffb640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6e806b500 0x7ff6e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.693+0000 7ff6f9ffb640 1 -- 172.21.15.59:0/264824685 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff6fc0894e0 con 0x7ff70406cf20 2024-03-06T07:23:51.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.693+0000 7ff708f64640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6e806b500 0x7ff6e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.693+0000 7ff708f64640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6e806b500 0x7ff6e806d9c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4036000 tx=0x7ff6f4008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.697+0000 7ff6f9ffb640 1 -- 172.21.15.59:0/264824685 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6fc057cd0 con 0x7ff70406cf20 2024-03-06T07:23:51.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.912+0000 7ff70b9c9640 1 -- 172.21.15.59:0/264824685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7040008d0 con 0x7ff70406cf20 2024-03-06T07:23:51.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.912+0000 7ff6f9ffb640 1 -- 172.21.15.59:0/264824685 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7ff6fc05bd70 con 0x7ff70406cf20 2024-03-06T07:23:51.915 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:51.915 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5557,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/690078434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":690078434},{"type":"v1","addr":"172.21.15.164:6836","nonce":690078434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1345698247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1345698247},{"type":"v1","addr":"172.21.15.59:6833","nonce":1345698247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5685,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3875776524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3875776524},{"type":"v1","addr":"172.21.15.59:6835","nonce":3875776524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:28.400413+0000","modified":"2024-03-06T07:23:31.397689+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5697":{"gid":5697,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/3749315375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3749315375},{"type":"v1","addr":"172.21.15.164:6837","nonce":3749315375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5697,"qdb_cluster":[5697]},"id":3}]} 2024-03-06T07:23:51.916 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 20 2024-03-06T07:23:51.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 -- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6e806b500 msgr2=0x7ff6e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6e806b500 0x7ff6e806d9c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4036000 tx=0x7ff6f4008030 comp rx=0 tx=0).stop 2024-03-06T07:23:51.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 -- 172.21.15.59:0/264824685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff70406cf20 msgr2=0x7ff7040cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff70406cf20 0x7ff7040cb2e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c3c0 tx=0x7ff6fc00c880 comp rx=0 tx=0).stop 2024-03-06T07:23:51.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 -- 172.21.15.59:0/264824685 shutdown_connections 2024-03-06T07:23:51.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6e806b500 0x7ff6e806d9c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7040d00c0 0x7ff7040c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7040cf850 0x7ff704154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 --2- 172.21.15.59:0/264824685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff70406cf20 0x7ff7040cb2e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 -- 172.21.15.59:0/264824685 >> 172.21.15.59:0/264824685 conn(0x7ff7040b35a0 msgr2=0x7ff7040d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 -- 172.21.15.59:0/264824685 shutdown_connections 2024-03-06T07:23:51.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:51.914+0000 7ff70b9c9640 1 -- 172.21.15.59:0/264824685 wait complete. 2024-03-06T07:23:51.929 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.028+0000 7f31696c2640 1 Processor -- start 2024-03-06T07:23:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.028+0000 7f31696c2640 1 -- start start 2024-03-06T07:23:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.028+0000 7f31696c2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f316406c6d0 0x7f316415ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.028+0000 7f31696c2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31640cd450 con 0x7f31640b4f70 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.028+0000 7f31696c2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31640cd730 con 0x7f3164066d90 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.028+0000 7f31696c2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316415e9f0 con 0x7f316406c6d0 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.028+0000 7f31627fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f316406c6d0 0x7f316415ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.028+0000 7f3162ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f31640b4f70 0x7f31640b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50510/0 (socket says 172.21.15.59:50510) 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.028+0000 7f3162ffd640 1 -- 172.21.15.59:0/63549846 learned_addr learned my addr 172.21.15.59:0/63549846 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.029+0000 7f31627fc640 1 -- 172.21.15.59:0/63549846 >> v1:172.21.15.164:6789/0 conn(0x7f3164066d90 legacy=0x7f31640ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.029+0000 7f31627fc640 1 -- 172.21.15.59:0/63549846 >> v1:172.21.15.59:6789/0 conn(0x7f31640b4f70 legacy=0x7f31640b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.029+0000 7f3161ffb640 1 -- 172.21.15.59:0/63549846 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627399739 0 0) 0x7f31640cd730 con 0x7f3164066d90 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.029+0000 7f31627fc640 1 -- 172.21.15.59:0/63549846 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316415e680 con 0x7f316406c6d0 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.029+0000 7f31627fc640 1 --2- 172.21.15.59:0/63549846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f316406c6d0 0x7f316415ee80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f31580097a0 tx=0x7f315802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c6a601ea606d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.030+0000 7f3161ffb640 1 -- 172.21.15.59:0/63549846 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158037070 con 0x7f316406c6d0 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.030+0000 7f3161ffb640 1 -- 172.21.15.59:0/63549846 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3158030710 con 0x7f316406c6d0 2024-03-06T07:23:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.030+0000 7f3161ffb640 1 -- 172.21.15.59:0/63549846 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158030ae0 con 0x7f316406c6d0 2024-03-06T07:23:52.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.030+0000 7f31696c2640 1 -- 172.21.15.59:0/63549846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f316406c6d0 msgr2=0x7f316415ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.030+0000 7f31696c2640 1 --2- 172.21.15.59:0/63549846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f316406c6d0 0x7f316415ee80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f31580097a0 tx=0x7f315802b790 comp rx=0 tx=0).stop 2024-03-06T07:23:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.030+0000 7f31696c2640 1 -- 172.21.15.59:0/63549846 shutdown_connections 2024-03-06T07:23:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.030+0000 7f31696c2640 1 --2- 172.21.15.59:0/63549846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f316406c6d0 0x7f316415ee80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.030+0000 7f31696c2640 1 -- 172.21.15.59:0/63549846 >> 172.21.15.59:0/63549846 conn(0x7f31640724d0 msgr2=0x7f31640728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.030+0000 7f31696c2640 1 -- 172.21.15.59:0/63549846 shutdown_connections 2024-03-06T07:23:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.030+0000 7f31696c2640 1 -- 172.21.15.59:0/63549846 wait complete. 2024-03-06T07:23:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.032+0000 7f31696c2640 1 Processor -- start 2024-03-06T07:23:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.032+0000 7f31696c2640 1 -- start start 2024-03-06T07:23:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.032+0000 7f31696c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3164066d90 0x7f31641f4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.032+0000 7f31696c2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31640b4f70 0x7f31641edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.032+0000 7f31696c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31640ce690 0x7f31640afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.032+0000 7f31696c2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3164161ad0 con 0x7f31640b4f70 2024-03-06T07:23:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.032+0000 7f31696c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31640cda10 con 0x7f31640ce690 2024-03-06T07:23:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.032+0000 7f31696c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3164161c50 con 0x7f3164066d90 2024-03-06T07:23:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31627fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31640b4f70 0x7f31641edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31627fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31640b4f70 0x7f31641edfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38226/0 (socket says 172.21.15.59:38226) 2024-03-06T07:23:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31627fc640 1 -- 172.21.15.59:0/1773508657 learned_addr learned my addr 172.21.15.59:0/1773508657 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f3162ffd640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3164066d90 0x7f31641f4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31637fe640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31640ce690 0x7f31640afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31627fc640 1 -- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3164066d90 msgr2=0x7f31641f4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31627fc640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3164066d90 0x7f31641f4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31627fc640 1 -- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31640ce690 msgr2=0x7f31640afc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31627fc640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31640ce690 0x7f31640afc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31627fc640 1 -- 172.21.15.59:0/1773508657 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31640ceee0 con 0x7f31640b4f70 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31627fc640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31640b4f70 0x7f31641edfa0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3158035040 tx=0x7f31580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f314b7fe640 1 -- 172.21.15.59:0/1773508657 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158037040 con 0x7f31640b4f70 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f314b7fe640 1 -- 172.21.15.59:0/1773508657 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3158004030 con 0x7f31640b4f70 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f314b7fe640 1 -- 172.21.15.59:0/1773508657 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158004420 con 0x7f31640b4f70 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31696c2640 1 -- 172.21.15.59:0/1773508657 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31640cf1c0 con 0x7f31640b4f70 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.033+0000 7f31696c2640 1 -- 172.21.15.59:0/1773508657 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31640cf780 con 0x7f31640b4f70 2024-03-06T07:23:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.034+0000 7f31696c2640 1 -- 172.21.15.59:0/1773508657 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31640cf950 con 0x7f31640b4f70 2024-03-06T07:23:52.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.035+0000 7f314b7fe640 1 -- 172.21.15.59:0/1773508657 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3158032940 con 0x7f31640b4f70 2024-03-06T07:23:52.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.035+0000 7f314b7fe640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f315006b4d0 0x7f315006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.035+0000 7f314b7fe640 1 -- 172.21.15.59:0/1773508657 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f31580ac040 con 0x7f31640b4f70 2024-03-06T07:23:52.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.035+0000 7f3162ffd640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f315006b4d0 0x7f315006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.036+0000 7f3162ffd640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f315006b4d0 0x7f315006d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f315c000f60 tx=0x7f315c007e20 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.039+0000 7f314b7fe640 1 -- 172.21.15.59:0/1773508657 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3158042020 con 0x7f31640b4f70 2024-03-06T07:23:52.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.222+0000 7f31696c2640 1 -- 172.21.15.59:0/1773508657 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f316404f330 con 0x7f31640b4f70 2024-03-06T07:23:52.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.224+0000 7f314b7fe640 1 -- 172.21.15.59:0/1773508657 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16752 (secure 0 0 0) 0x7f315807a8c0 con 0x7f31640b4f70 2024-03-06T07:23:52.227 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:52.227 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":47,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:23:37.063479+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:26.672668+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:27.362891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.9","mappings":[{"from":0,"to":5}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:52.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 -- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f315006b4d0 msgr2=0x7f315006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f315006b4d0 0x7f315006d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f315c000f60 tx=0x7f315c007e20 comp rx=0 tx=0).stop 2024-03-06T07:23:52.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 -- 172.21.15.59:0/1773508657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31640b4f70 msgr2=0x7f31641edfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31640b4f70 0x7f31641edfa0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3158035040 tx=0x7f31580046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 -- 172.21.15.59:0/1773508657 shutdown_connections 2024-03-06T07:23:52.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f315006b4d0 0x7f315006d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31640ce690 0x7f31640afc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31640b4f70 0x7f31641edfa0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 --2- 172.21.15.59:0/1773508657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3164066d90 0x7f31641f4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 -- 172.21.15.59:0/1773508657 >> 172.21.15.59:0/1773508657 conn(0x7f31640724d0 msgr2=0x7f316406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 -- 172.21.15.59:0/1773508657 shutdown_connections 2024-03-06T07:23:52.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.226+0000 7f31696c2640 1 -- 172.21.15.59:0/1773508657 wait complete. 2024-03-06T07:23:52.240 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:52.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.339+0000 7f5002eb5640 1 Processor -- start 2024-03-06T07:23:52.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.339+0000 7f5002eb5640 1 -- start start 2024-03-06T07:23:52.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.340+0000 7f5002eb5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4ffc161670 0x7f4ffc072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.340+0000 7f5002eb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc072a60 0x7f4ffc072ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.340+0000 7f5002eb5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4ffc0adf10 0x7f4ffc0ae380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.340+0000 7f5002eb5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc0b6170 con 0x7f4ffc0adf10 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.340+0000 7f5002eb5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc0ada80 con 0x7f4ffc161670 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.340+0000 7f5002eb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc161a50 con 0x7f4ffc072a60 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.340+0000 7f5001452640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4ffc0adf10 0x7f4ffc0ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.340+0000 7f5001452640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4ffc0adf10 0x7f4ffc0ae380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38238/0 (socket says 172.21.15.59:38238) 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.340+0000 7f5001452640 1 -- 172.21.15.59:0/3160674933 learned_addr learned my addr 172.21.15.59:0/3160674933 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.340+0000 7f4ffbfff640 1 --2- 172.21.15.59:0/3160674933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc072a60 0x7f4ffc072ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.340+0000 7f5000c51640 1 --2- 172.21.15.59:0/3160674933 >> v2:172.21.15.164:3300/0 conn(0x7f4ffc161670 0x7f4ffc072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.341+0000 7f5001452640 1 -- 172.21.15.59:0/3160674933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc072a60 msgr2=0x7f4ffc072ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.341+0000 7f5001452640 1 --2- 172.21.15.59:0/3160674933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc072a60 0x7f4ffc072ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.341+0000 7f5001452640 1 -- 172.21.15.59:0/3160674933 >> v2:172.21.15.164:3300/0 conn(0x7f4ffc161670 msgr2=0x7f4ffc072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.341+0000 7f5001452640 1 --2- 172.21.15.59:0/3160674933 >> v2:172.21.15.164:3300/0 conn(0x7f4ffc161670 0x7f4ffc072520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.341+0000 7f5001452640 1 -- 172.21.15.59:0/3160674933 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc066d00 con 0x7f4ffc0adf10 2024-03-06T07:23:52.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.341+0000 7f5001452640 1 --2- 172.21.15.59:0/3160674933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc0adf10 0x7f4ffc0ae380 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc070a80 tx=0x7f4fec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4faf4a45bbaea2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f4ffb7fe640 1 -- 172.21.15.59:0/3160674933 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec035020 con 0x7f4ffc0adf10 2024-03-06T07:23:52.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f4ffb7fe640 1 -- 172.21.15.59:0/3160674933 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec031640 con 0x7f4ffc0adf10 2024-03-06T07:23:52.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f4ffb7fe640 1 -- 172.21.15.59:0/3160674933 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec02d040 con 0x7f4ffc0adf10 2024-03-06T07:23:52.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f5002eb5640 1 -- 172.21.15.59:0/3160674933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc0adf10 msgr2=0x7f4ffc0ae380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f5002eb5640 1 --2- 172.21.15.59:0/3160674933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc0adf10 0x7f4ffc0ae380 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc070a80 tx=0x7f4fec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f5002eb5640 1 -- 172.21.15.59:0/3160674933 shutdown_connections 2024-03-06T07:23:52.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f5002eb5640 1 --2- 172.21.15.59:0/3160674933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc0adf10 0x7f4ffc0ae380 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f5002eb5640 1 --2- 172.21.15.59:0/3160674933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc072a60 0x7f4ffc072ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f5002eb5640 1 --2- 172.21.15.59:0/3160674933 >> v2:172.21.15.164:3300/0 conn(0x7f4ffc161670 0x7f4ffc072520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f5002eb5640 1 -- 172.21.15.59:0/3160674933 >> 172.21.15.59:0/3160674933 conn(0x7f4ffc0b4e60 msgr2=0x7f4ffc0b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f5002eb5640 1 -- 172.21.15.59:0/3160674933 shutdown_connections 2024-03-06T07:23:52.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.342+0000 7f5002eb5640 1 -- 172.21.15.59:0/3160674933 wait complete. 2024-03-06T07:23:52.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.343+0000 7f5002eb5640 1 Processor -- start 2024-03-06T07:23:52.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.343+0000 7f5002eb5640 1 -- start start 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5002eb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc072a60 0x7f4ffc14f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5002eb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ffc0adf10 0x7f4ffc0ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5002eb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc161670 0x7f4ffc14fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5002eb5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc06b330 con 0x7f4ffc161670 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5002eb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc06b050 con 0x7f4ffc0adf10 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5002eb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc06b610 con 0x7f4ffc072a60 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5001452640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc161670 0x7f4ffc14fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5001452640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc161670 0x7f4ffc14fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38244/0 (socket says 172.21.15.59:38244) 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5001452640 1 -- 172.21.15.59:0/3895519887 learned_addr learned my addr 172.21.15.59:0/3895519887 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5000c51640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc072a60 0x7f4ffc14f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f4ffbfff640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ffc0adf10 0x7f4ffc0ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5001452640 1 -- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc072a60 msgr2=0x7f4ffc14f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.344+0000 7f5001452640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc072a60 0x7f4ffc14f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.345+0000 7f5001452640 1 -- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ffc0adf10 msgr2=0x7f4ffc0ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.345+0000 7f5001452640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ffc0adf10 0x7f4ffc0ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.345+0000 7f5001452640 1 -- 172.21.15.59:0/3895519887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc0cafe0 con 0x7f4ffc161670 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.345+0000 7f5001452640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc161670 0x7f4ffc14fe00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4fec02b930 tx=0x7f4fec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.345+0000 7f4ff97fa640 1 -- 172.21.15.59:0/3895519887 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec035040 con 0x7f4ffc161670 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.345+0000 7f4ff97fa640 1 -- 172.21.15.59:0/3895519887 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec0399f0 con 0x7f4ffc161670 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.345+0000 7f5002eb5640 1 -- 172.21.15.59:0/3895519887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc0cb2c0 con 0x7f4ffc161670 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.345+0000 7f4ff97fa640 1 -- 172.21.15.59:0/3895519887 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec039dc0 con 0x7f4ffc161670 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.345+0000 7f5002eb5640 1 -- 172.21.15.59:0/3895519887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc0cb880 con 0x7f4ffc161670 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.346+0000 7f5002eb5640 1 -- 172.21.15.59:0/3895519887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc04f330 con 0x7f4ffc161670 2024-03-06T07:23:52.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.346+0000 7f4ff97fa640 1 -- 172.21.15.59:0/3895519887 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fec038020 con 0x7f4ffc161670 2024-03-06T07:23:52.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.348+0000 7f4ff97fa640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe806b4d0 0x7f4fe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.348+0000 7f4ff97fa640 1 -- 172.21.15.59:0/3895519887 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4fec0abe80 con 0x7f4ffc161670 2024-03-06T07:23:52.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.348+0000 7f5000c51640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe806b4d0 0x7f4fe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.348+0000 7f5000c51640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe806b4d0 0x7f4fe806d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4011340 tx=0x7f4ff400b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.352+0000 7f4ff97fa640 1 -- 172.21.15.59:0/3895519887 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fec07a670 con 0x7f4ffc161670 2024-03-06T07:23:52.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.569+0000 7f5002eb5640 1 -- 172.21.15.59:0/3895519887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4ffc06c570 con 0x7f4ffc161670 2024-03-06T07:23:52.573 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:23:52.571+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:52.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.571+0000 7f4ff97fa640 1 -- 172.21.15.59:0/3895519887 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v21) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4fec07e710 con 0x7f4ffc161670 2024-03-06T07:23:52.574 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.574+0000 7f5002eb5640 1 -- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe806b4d0 msgr2=0x7f4fe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.574+0000 7f5002eb5640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe806b4d0 0x7f4fe806d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4011340 tx=0x7f4ff400b040 comp rx=0 tx=0).stop 2024-03-06T07:23:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.574+0000 7f5002eb5640 1 -- 172.21.15.59:0/3895519887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc161670 msgr2=0x7f4ffc14fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.574+0000 7f5002eb5640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc161670 0x7f4ffc14fe00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4fec02b930 tx=0x7f4fec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.575+0000 7f5002eb5640 1 -- 172.21.15.59:0/3895519887 shutdown_connections 2024-03-06T07:23:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.575+0000 7f5002eb5640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe806b4d0 0x7f4fe806d990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.575+0000 7f5002eb5640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc161670 0x7f4ffc14fe00 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.575+0000 7f5002eb5640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ffc0adf10 0x7f4ffc0ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.575+0000 7f5002eb5640 1 --2- 172.21.15.59:0/3895519887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc072a60 0x7f4ffc14f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.575+0000 7f5002eb5640 1 -- 172.21.15.59:0/3895519887 >> 172.21.15.59:0/3895519887 conn(0x7f4ffc0b4e60 msgr2=0x7f4ffc06ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.575+0000 7f5002eb5640 1 -- 172.21.15.59:0/3895519887 shutdown_connections 2024-03-06T07:23:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.575+0000 7f5002eb5640 1 -- 172.21.15.59:0/3895519887 wait complete. 2024-03-06T07:23:52.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:52.613 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:23:52.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38c0d79640 1 Processor -- start 2024-03-06T07:23:52.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38c0d79640 1 -- start start 2024-03-06T07:23:52.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38c0d79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38bc15bb40 0x7f38bc0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38c0d79640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38bc0cfa90 0x7f38bc0cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38c0d79640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38bc15ff90 con 0x7f38bc15f7d0 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38c0d79640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38bc06c910 con 0x7f38bc0cfa90 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38c0d79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc160270 con 0x7f38bc15bb40 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38bad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38bc0cfa90 0x7f38bc0cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38b3fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f38bc15f7d0 0x7f38bc06d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50524/0 (socket says 172.21.15.59:50524) 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38b3fff640 1 -- 172.21.15.59:0/3028918442 learned_addr learned my addr 172.21.15.59:0/3028918442 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38ba59c640 1 --2- 172.21.15.59:0/3028918442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38bc15bb40 0x7f38bc0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38bad9d640 1 -- 172.21.15.59:0/3028918442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38bc15bb40 msgr2=0x7f38bc0b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38bad9d640 1 --2- 172.21.15.59:0/3028918442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38bc15bb40 0x7f38bc0b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38bad9d640 1 -- 172.21.15.59:0/3028918442 >> v1:172.21.15.59:6789/0 conn(0x7f38bc15f7d0 legacy=0x7f38bc06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38b9d9b640 1 -- 172.21.15.59:0/3028918442 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892790985 0 0) 0x7f38bc15ff90 con 0x7f38bc15f7d0 2024-03-06T07:23:52.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38bad9d640 1 -- 172.21.15.59:0/3028918442 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc160550 con 0x7f38bc0cfa90 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.690+0000 7f38ba59c640 1 --2- 172.21.15.59:0/3028918442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38bc15bb40 0x7f38bc0b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.691+0000 7f38bad9d640 1 --2- 172.21.15.59:0/3028918442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38bc0cfa90 0x7f38bc0cff20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f38b4009d90 tx=0x7f38b402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec91ee610f5b305 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.691+0000 7f38b9d9b640 1 -- 172.21.15.59:0/3028918442 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4039070 con 0x7f38bc0cfa90 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.691+0000 7f38b9d9b640 1 -- 172.21.15.59:0/3028918442 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b402cc50 con 0x7f38bc0cfa90 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.691+0000 7f38b9d9b640 1 -- 172.21.15.59:0/3028918442 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b4034440 con 0x7f38bc0cfa90 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.692+0000 7f38c0d79640 1 -- 172.21.15.59:0/3028918442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38bc0cfa90 msgr2=0x7f38bc0cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.692+0000 7f38c0d79640 1 --2- 172.21.15.59:0/3028918442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38bc0cfa90 0x7f38bc0cff20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f38b4009d90 tx=0x7f38b402bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.692+0000 7f38c0d79640 1 -- 172.21.15.59:0/3028918442 shutdown_connections 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.692+0000 7f38c0d79640 1 --2- 172.21.15.59:0/3028918442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38bc0cfa90 0x7f38bc0cff20 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.692+0000 7f38c0d79640 1 --2- 172.21.15.59:0/3028918442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38bc15bb40 0x7f38bc0b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.692+0000 7f38c0d79640 1 -- 172.21.15.59:0/3028918442 >> 172.21.15.59:0/3028918442 conn(0x7f38bc06b190 msgr2=0x7f38bc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.692+0000 7f38c0d79640 1 -- 172.21.15.59:0/3028918442 shutdown_connections 2024-03-06T07:23:52.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.692+0000 7f38c0d79640 1 -- 172.21.15.59:0/3028918442 wait complete. 2024-03-06T07:23:52.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.693+0000 7f38c0d79640 1 Processor -- start 2024-03-06T07:23:52.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.693+0000 7f38c0d79640 1 -- start start 2024-03-06T07:23:52.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38c0d79640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38bc0cfa90 0x7f38bc072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38c0d79640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38bc15bb40 0x7f38bc151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38c0d79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38bc15f7d0 0x7f38bc1e5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38c0d79640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc06d790 con 0x7f38bc0cfa90 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38c0d79640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc15fc30 con 0x7f38bc15bb40 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38c0d79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc15bf20 con 0x7f38bc15f7d0 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38ba59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38bc0cfa90 0x7f38bc072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38ba59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38bc0cfa90 0x7f38bc072570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38258/0 (socket says 172.21.15.59:38258) 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38ba59c640 1 -- 172.21.15.59:0/3490900539 learned_addr learned my addr 172.21.15.59:0/3490900539 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38bad9d640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38bc15f7d0 0x7f38bc1e5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38b3fff640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38bc15bb40 0x7f38bc151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38ba59c640 1 -- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38bc15f7d0 msgr2=0x7f38bc1e5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38ba59c640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38bc15f7d0 0x7f38bc1e5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38ba59c640 1 -- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38bc15bb40 msgr2=0x7f38bc151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38ba59c640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38bc15bb40 0x7f38bc151100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38ba59c640 1 -- 172.21.15.59:0/3490900539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc1e5a80 con 0x7f38bc0cfa90 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38b3fff640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38bc15bb40 0x7f38bc151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38ba59c640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38bc0cfa90 0x7f38bc072570 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00e730 tx=0x7f38ac00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38b2ffd640 1 -- 172.21.15.59:0/3490900539 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac00f040 con 0x7f38bc0cfa90 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38b2ffd640 1 -- 172.21.15.59:0/3490900539 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac004230 con 0x7f38bc0cfa90 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38c0d79640 1 -- 172.21.15.59:0/3490900539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38bc071140 con 0x7f38bc0cfa90 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38b2ffd640 1 -- 172.21.15.59:0/3490900539 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac00ee40 con 0x7f38bc0cfa90 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38c0d79640 1 -- 172.21.15.59:0/3490900539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38bc071700 con 0x7f38bc0cfa90 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.695+0000 7f38c0d79640 1 -- 172.21.15.59:0/3490900539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38bc0cee20 con 0x7f38bc0cfa90 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.696+0000 7f38b2ffd640 1 -- 172.21.15.59:0/3490900539 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38ac00c6d0 con 0x7f38bc0cfa90 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.696+0000 7f38b2ffd640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f389c06b4d0 0x7f389c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.696+0000 7f38b2ffd640 1 -- 172.21.15.59:0/3490900539 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f38ac089f10 con 0x7f38bc0cfa90 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.696+0000 7f38b3fff640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f389c06b4d0 0x7f389c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.697+0000 7f38b3fff640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f389c06b4d0 0x7f389c06d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f38bc1f6e60 tx=0x7f38a4007ca0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.701+0000 7f38b2ffd640 1 -- 172.21.15.59:0/3490900539 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38bc0cee20 con 0x7f38bc0cfa90 2024-03-06T07:23:52.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:52.918+0000 7f38c0d79640 1 -- 172.21.15.59:0/3490900539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f38bc0732b0 con 0x7f38bc0cfa90 2024-03-06T07:23:53.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.574+0000 7f38b2ffd640 1 -- 172.21.15.59:0/3490900539 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v22) v1 ==== 104+0+0 (secure 0 0 0) 0x7f38ac017020 con 0x7f38bc0cfa90 2024-03-06T07:23:53.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 -- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f389c06b4d0 msgr2=0x7f389c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f389c06b4d0 0x7f389c06d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f38bc1f6e60 tx=0x7f38a4007ca0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 -- 172.21.15.59:0/3490900539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38bc0cfa90 msgr2=0x7f38bc072570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38bc0cfa90 0x7f38bc072570 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f38ac00e730 tx=0x7f38ac00ebf0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 -- 172.21.15.59:0/3490900539 shutdown_connections 2024-03-06T07:23:53.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f389c06b4d0 0x7f389c06d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38bc15f7d0 0x7f38bc1e5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38bc15bb40 0x7f38bc151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 --2- 172.21.15.59:0/3490900539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38bc0cfa90 0x7f38bc072570 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 -- 172.21.15.59:0/3490900539 >> 172.21.15.59:0/3490900539 conn(0x7f38bc06b190 msgr2=0x7f38bc0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 -- 172.21.15.59:0/3490900539 shutdown_connections 2024-03-06T07:23:53.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.576+0000 7f38c0d79640 1 -- 172.21.15.59:0/3490900539 wait complete. 2024-03-06T07:23:53.591 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:53.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.701+0000 7fb0a86df640 1 Processor -- start 2024-03-06T07:23:53.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.701+0000 7fb0a86df640 1 -- start start 2024-03-06T07:23:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a86df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0a015a490 0x7fb0a006df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a86df640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb0a00b3fd0 0x7fb0a00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a86df640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb0a00b2e90 0x7fb0a00708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a86df640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a006e4d0 con 0x7fb0a00b3fd0 2024-03-06T07:23:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a86df640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a00b4490 con 0x7fb0a00b2e90 2024-03-06T07:23:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a86df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00cf4a0 con 0x7fb0a015a490 2024-03-06T07:23:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a5c7a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb0a00b3fd0 0x7fb0a00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a5c7a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb0a00b3fd0 0x7fb0a00b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57818/0 (socket says 172.21.15.59:57818) 2024-03-06T07:23:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a5c7a640 1 -- 172.21.15.59:0/673202217 learned_addr learned my addr 172.21.15.59:0/673202217 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:53.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a6c7c640 1 --2- 172.21.15.59:0/673202217 >> v2:172.21.15.164:3300/0 conn(0x7fb0a00b2e90 0x7fb0a00708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a647b640 1 --2- 172.21.15.59:0/673202217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0a015a490 0x7fb0a006df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a5c7a640 1 -- 172.21.15.59:0/673202217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0a015a490 msgr2=0x7fb0a006df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a5c7a640 1 --2- 172.21.15.59:0/673202217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0a015a490 0x7fb0a006df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a5c7a640 1 -- 172.21.15.59:0/673202217 >> v2:172.21.15.164:3300/0 conn(0x7fb0a00b2e90 msgr2=0x7fb0a00708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a5c7a640 1 --2- 172.21.15.59:0/673202217 >> v2:172.21.15.164:3300/0 conn(0x7fb0a00b2e90 0x7fb0a00708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.702+0000 7fb0a5c7a640 1 -- 172.21.15.59:0/673202217 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a00b4750 con 0x7fb0a00b3fd0 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.703+0000 7fb0a5c7a640 1 --2- 172.21.15.59:0/673202217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0a00b3fd0 0x7fb0a00b2950 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0070560 tx=0x7fb09002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924915802f231898 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.703+0000 7fb0a5479640 1 -- 172.21.15.59:0/673202217 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090035020 con 0x7fb0a00b3fd0 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.703+0000 7fb0a5479640 1 -- 172.21.15.59:0/673202217 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb090031870 con 0x7fb0a00b3fd0 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.703+0000 7fb0a5479640 1 -- 172.21.15.59:0/673202217 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090031c40 con 0x7fb0a00b3fd0 2024-03-06T07:23:53.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.704+0000 7fb0a86df640 1 -- 172.21.15.59:0/673202217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0a00b3fd0 msgr2=0x7fb0a00b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.704+0000 7fb0a86df640 1 --2- 172.21.15.59:0/673202217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0a00b3fd0 0x7fb0a00b2950 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0070560 tx=0x7fb09002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:53.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.704+0000 7fb0a86df640 1 -- 172.21.15.59:0/673202217 shutdown_connections 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.704+0000 7fb0a86df640 1 --2- 172.21.15.59:0/673202217 >> v2:172.21.15.164:3300/0 conn(0x7fb0a00b2e90 0x7fb0a00708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.704+0000 7fb0a86df640 1 --2- 172.21.15.59:0/673202217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0a00b3fd0 0x7fb0a00b2950 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.704+0000 7fb0a86df640 1 --2- 172.21.15.59:0/673202217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0a015a490 0x7fb0a006df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.704+0000 7fb0a86df640 1 -- 172.21.15.59:0/673202217 >> 172.21.15.59:0/673202217 conn(0x7fb0a00b1650 msgr2=0x7fb0a00b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.704+0000 7fb0a86df640 1 -- 172.21.15.59:0/673202217 shutdown_connections 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.704+0000 7fb0a86df640 1 -- 172.21.15.59:0/673202217 wait complete. 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.705+0000 7fb0a86df640 1 Processor -- start 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.705+0000 7fb0a86df640 1 -- start start 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a86df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0a00b2e90 0x7fb0a0141cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a86df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0a00b3fd0 0x7fb0a00ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a86df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0a015a490 0x7fb0a00b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a86df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00cf260 con 0x7fb0a00b2e90 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a86df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00cf0e0 con 0x7fb0a00b3fd0 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a86df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00b3320 con 0x7fb0a015a490 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a647b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0a00b2e90 0x7fb0a0141cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a647b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0a00b2e90 0x7fb0a0141cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57824/0 (socket says 172.21.15.59:57824) 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a647b640 1 -- 172.21.15.59:0/3890982052 learned_addr learned my addr 172.21.15.59:0/3890982052 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a5c7a640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0a00b3fd0 0x7fb0a00ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a6c7c640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0a015a490 0x7fb0a00b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a647b640 1 -- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0a015a490 msgr2=0x7fb0a00b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a647b640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0a015a490 0x7fb0a00b0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a647b640 1 -- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0a00b3fd0 msgr2=0x7fb0a00ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a647b640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0a00b3fd0 0x7fb0a00ae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a647b640 1 -- 172.21.15.59:0/3890982052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a015aac0 con 0x7fb0a00b2e90 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a647b640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0a00b2e90 0x7fb0a0141cb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00d5f0 tx=0x7fb09c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb096ffd640 1 -- 172.21.15.59:0/3890982052 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c0125f0 con 0x7fb0a00b2e90 2024-03-06T07:23:53.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb096ffd640 1 -- 172.21.15.59:0/3890982052 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c019070 con 0x7fb0a00b2e90 2024-03-06T07:23:53.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb096ffd640 1 -- 172.21.15.59:0/3890982052 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c0129d0 con 0x7fb0a00b2e90 2024-03-06T07:23:53.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a86df640 1 -- 172.21.15.59:0/3890982052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a015ada0 con 0x7fb0a00b2e90 2024-03-06T07:23:53.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.706+0000 7fb0a86df640 1 -- 172.21.15.59:0/3890982052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a015b360 con 0x7fb0a00b2e90 2024-03-06T07:23:53.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.707+0000 7fb0a86df640 1 -- 172.21.15.59:0/3890982052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a00687d0 con 0x7fb0a00b2e90 2024-03-06T07:23:53.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.708+0000 7fb096ffd640 1 -- 172.21.15.59:0/3890982052 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb09c012b70 con 0x7fb0a00b2e90 2024-03-06T07:23:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.708+0000 7fb096ffd640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b4d0 0x7fb08406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.708+0000 7fb096ffd640 1 -- 172.21.15.59:0/3890982052 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb09c08a680 con 0x7fb0a00b2e90 2024-03-06T07:23:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.709+0000 7fb0a5c7a640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b4d0 0x7fb08406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.709+0000 7fb0a5c7a640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b4d0 0x7fb08406d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb090036000 tx=0x7fb09003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.713+0000 7fb096ffd640 1 -- 172.21.15.59:0/3890982052 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb09c0593d0 con 0x7fb0a00b2e90 2024-03-06T07:23:53.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:53.893+0000 7fb0a86df640 1 -- 172.21.15.59:0/3890982052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb0a004f330 con 0x7fb0a00b2e90 2024-03-06T07:23:54.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.575+0000 7fb096ffd640 1 -- 172.21.15.59:0/3890982052 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v49) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb09c058cd0 con 0x7fb0a00b2e90 2024-03-06T07:23:54.577 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:54.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.580+0000 7fb0a86df640 1 -- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b4d0 msgr2=0x7fb08406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.580+0000 7fb0a86df640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b4d0 0x7fb08406d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb090036000 tx=0x7fb09003c040 comp rx=0 tx=0).stop 2024-03-06T07:23:54.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.580+0000 7fb0a86df640 1 -- 172.21.15.59:0/3890982052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0a00b2e90 msgr2=0x7fb0a0141cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.580+0000 7fb0a86df640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0a00b2e90 0x7fb0a0141cb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00d5f0 tx=0x7fb09c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.580+0000 7fb0a86df640 1 -- 172.21.15.59:0/3890982052 shutdown_connections 2024-03-06T07:23:54.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.580+0000 7fb0a86df640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b4d0 0x7fb08406d990 secure :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb090036000 tx=0x7fb09003c040 comp rx=0 tx=0).stop 2024-03-06T07:23:54.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.580+0000 7fb0a86df640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0a015a490 0x7fb0a00b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.580+0000 7fb0a86df640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0a00b3fd0 0x7fb0a00ae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.580+0000 7fb0a86df640 1 --2- 172.21.15.59:0/3890982052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0a00b2e90 0x7fb0a0141cb0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.580+0000 7fb0a86df640 1 -- 172.21.15.59:0/3890982052 >> 172.21.15.59:0/3890982052 conn(0x7fb0a00b1650 msgr2=0x7fb0a0150620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.581+0000 7fb0a86df640 1 -- 172.21.15.59:0/3890982052 shutdown_connections 2024-03-06T07:23:54.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.581+0000 7fb0a86df640 1 -- 172.21.15.59:0/3890982052 wait complete. 2024-03-06T07:23:54.598 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.707+0000 7f8000f65640 1 Processor -- start 2024-03-06T07:23:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.707+0000 7f8000f65640 1 -- start start 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.708+0000 7f8000f65640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ffc161a90 0x7f7ffc161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.708+0000 7f8000f65640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ffc0d2720 0x7f7ffc0cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.708+0000 7f8000f65640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc06be00 0x7f7ffc06c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.708+0000 7f8000f65640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ffc0d00c0 con 0x7f7ffc161a90 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.708+0000 7f8000f65640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ffc1623b0 con 0x7f7ffc0d2720 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.708+0000 7f8000f65640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc0cf790 con 0x7f7ffc06be00 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.708+0000 7f7ffa59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ffc161a90 0x7f7ffc161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.708+0000 7f7ffa59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ffc161a90 0x7f7ffc161e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57832/0 (socket says 172.21.15.59:57832) 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.708+0000 7f7ffa59c640 1 -- 172.21.15.59:0/2824739464 learned_addr learned my addr 172.21.15.59:0/2824739464 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.709+0000 7f7ffad9d640 1 --2- 172.21.15.59:0/2824739464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc06be00 0x7f7ffc06c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.709+0000 7f7ff9d9b640 1 --2- 172.21.15.59:0/2824739464 >> v2:172.21.15.164:3300/0 conn(0x7f7ffc0d2720 0x7f7ffc0cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.709+0000 7f7ffa59c640 1 -- 172.21.15.59:0/2824739464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc06be00 msgr2=0x7f7ffc06c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.709+0000 7f7ffa59c640 1 --2- 172.21.15.59:0/2824739464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc06be00 0x7f7ffc06c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.709+0000 7f7ffa59c640 1 -- 172.21.15.59:0/2824739464 >> v2:172.21.15.164:3300/0 conn(0x7f7ffc0d2720 msgr2=0x7f7ffc0cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.709+0000 7f7ffa59c640 1 --2- 172.21.15.59:0/2824739464 >> v2:172.21.15.164:3300/0 conn(0x7f7ffc0d2720 0x7f7ffc0cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.709+0000 7f7ffa59c640 1 -- 172.21.15.59:0/2824739464 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc0cfa70 con 0x7f7ffc161a90 2024-03-06T07:23:54.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.709+0000 7f7ff9d9b640 1 --2- 172.21.15.59:0/2824739464 >> v2:172.21.15.164:3300/0 conn(0x7f7ffc0d2720 0x7f7ffc0cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:54.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.709+0000 7f7ffa59c640 1 --2- 172.21.15.59:0/2824739464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc161a90 0x7f7ffc161e70 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009870 tx=0x7f7ff402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c96c5ef2b15e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f7ff959a640 1 -- 172.21.15.59:0/2824739464 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4035020 con 0x7f7ffc161a90 2024-03-06T07:23:54.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f7ff959a640 1 -- 172.21.15.59:0/2824739464 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff4031640 con 0x7f7ffc161a90 2024-03-06T07:23:54.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f7ff959a640 1 -- 172.21.15.59:0/2824739464 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff402d040 con 0x7f7ffc161a90 2024-03-06T07:23:54.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f8000f65640 1 -- 172.21.15.59:0/2824739464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc161a90 msgr2=0x7f7ffc161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f8000f65640 1 --2- 172.21.15.59:0/2824739464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc161a90 0x7f7ffc161e70 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009870 tx=0x7f7ff402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f8000f65640 1 -- 172.21.15.59:0/2824739464 shutdown_connections 2024-03-06T07:23:54.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f8000f65640 1 --2- 172.21.15.59:0/2824739464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc06be00 0x7f7ffc06c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f8000f65640 1 --2- 172.21.15.59:0/2824739464 >> v2:172.21.15.164:3300/0 conn(0x7f7ffc0d2720 0x7f7ffc0cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f8000f65640 1 --2- 172.21.15.59:0/2824739464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc161a90 0x7f7ffc161e70 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f8000f65640 1 -- 172.21.15.59:0/2824739464 >> 172.21.15.59:0/2824739464 conn(0x7f7ffc0d1b20 msgr2=0x7f7ffc0d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f8000f65640 1 -- 172.21.15.59:0/2824739464 shutdown_connections 2024-03-06T07:23:54.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.710+0000 7f8000f65640 1 -- 172.21.15.59:0/2824739464 wait complete. 2024-03-06T07:23:54.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.712+0000 7f8000f65640 1 Processor -- start 2024-03-06T07:23:54.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.712+0000 7f8000f65640 1 -- start start 2024-03-06T07:23:54.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.712+0000 7f8000f65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc06be00 0x7f7ffc0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.712+0000 7f8000f65640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ffc0d2720 0x7f7ffc152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.712+0000 7f8000f65640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc161a90 0x7f7ffc153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.712+0000 7f8000f65640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc0cdd10 con 0x7f7ffc06be00 2024-03-06T07:23:54.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.712+0000 7f8000f65640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc161790 con 0x7f7ffc0d2720 2024-03-06T07:23:54.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.712+0000 7f8000f65640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc0cd9b0 con 0x7f7ffc161a90 2024-03-06T07:23:54.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.712+0000 7f7ffa59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc06be00 0x7f7ffc0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.712+0000 7f7ffa59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc06be00 0x7f7ffc0b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57840/0 (socket says 172.21.15.59:57840) 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.712+0000 7f7ffa59c640 1 -- 172.21.15.59:0/1172374676 learned_addr learned my addr 172.21.15.59:0/1172374676 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ff9d9b640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ffc0d2720 0x7f7ffc152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ffad9d640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc161a90 0x7f7ffc153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ffa59c640 1 -- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc161a90 msgr2=0x7f7ffc153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ffa59c640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc161a90 0x7f7ffc153170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ffa59c640 1 -- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ffc0d2720 msgr2=0x7f7ffc152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ffa59c640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ffc0d2720 0x7f7ffc152c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ffa59c640 1 -- 172.21.15.59:0/1172374676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc153ac0 con 0x7f7ffc06be00 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ff9d9b640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ffc0d2720 0x7f7ffc152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ffa59c640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc06be00 0x7f7ffc0b4970 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40097c0 tx=0x7f7ff40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ff2ffd640 1 -- 172.21.15.59:0/1172374676 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4035040 con 0x7f7ffc06be00 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ff2ffd640 1 -- 172.21.15.59:0/1172374676 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff40399f0 con 0x7f7ffc06be00 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f8000f65640 1 -- 172.21.15.59:0/1172374676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ffc156280 con 0x7f7ffc06be00 2024-03-06T07:23:54.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f7ff2ffd640 1 -- 172.21.15.59:0/1172374676 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4039de0 con 0x7f7ffc06be00 2024-03-06T07:23:54.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.713+0000 7f8000f65640 1 -- 172.21.15.59:0/1172374676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ffc156840 con 0x7f7ffc06be00 2024-03-06T07:23:54.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.714+0000 7f8000f65640 1 -- 172.21.15.59:0/1172374676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ffc04f330 con 0x7f7ffc06be00 2024-03-06T07:23:54.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.715+0000 7f7ff2ffd640 1 -- 172.21.15.59:0/1172374676 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ff4038020 con 0x7f7ffc06be00 2024-03-06T07:23:54.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.715+0000 7f7ff2ffd640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fdc06b4d0 0x7f7fdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.715+0000 7f7ff2ffd640 1 -- 172.21.15.59:0/1172374676 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f7ff40abe70 con 0x7f7ffc06be00 2024-03-06T07:23:54.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.715+0000 7f7ff9d9b640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fdc06b4d0 0x7f7fdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.716+0000 7f7ff9d9b640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fdc06b4d0 0x7f7fdc06d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0ce610 tx=0x7f7fe4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.719+0000 7f7ff2ffd640 1 -- 172.21.15.59:0/1172374676 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ff4042330 con 0x7f7ffc06be00 2024-03-06T07:23:54.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:54.899+0000 7f8000f65640 1 -- 172.21.15.59:0/1172374676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7ffc073f80 con 0x7f7ffc06be00 2024-03-06T07:23:55.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.581+0000 7f7ff2ffd640 1 -- 172.21.15.59:0/1172374676 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v50) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7ff407aea0 con 0x7f7ffc06be00 2024-03-06T07:23:55.583 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-03-06T07:23:55.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.586+0000 7f8000f65640 1 -- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fdc06b4d0 msgr2=0x7f7fdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.586+0000 7f8000f65640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fdc06b4d0 0x7f7fdc06d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0ce610 tx=0x7f7fe4008040 comp rx=0 tx=0).stop 2024-03-06T07:23:55.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.586+0000 7f8000f65640 1 -- 172.21.15.59:0/1172374676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc06be00 msgr2=0x7f7ffc0b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.586+0000 7f8000f65640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc06be00 0x7f7ffc0b4970 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40097c0 tx=0x7f7ff40046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.587+0000 7f8000f65640 1 -- 172.21.15.59:0/1172374676 shutdown_connections 2024-03-06T07:23:55.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.587+0000 7f8000f65640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fdc06b4d0 0x7f7fdc06d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.587+0000 7f8000f65640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc161a90 0x7f7ffc153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.587+0000 7f8000f65640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ffc0d2720 0x7f7ffc152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.587+0000 7f8000f65640 1 --2- 172.21.15.59:0/1172374676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ffc06be00 0x7f7ffc0b4970 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.587+0000 7f8000f65640 1 -- 172.21.15.59:0/1172374676 >> 172.21.15.59:0/1172374676 conn(0x7f7ffc0d1b20 msgr2=0x7f7ffc06eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.587+0000 7f8000f65640 1 -- 172.21.15.59:0/1172374676 shutdown_connections 2024-03-06T07:23:55.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.588+0000 7f8000f65640 1 -- 172.21.15.59:0/1172374676 wait complete. 2024-03-06T07:23:55.605 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:23:55.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.713+0000 7fc1d269d640 1 Processor -- start 2024-03-06T07:23:55.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.713+0000 7fc1d269d640 1 -- start start 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.713+0000 7fc1d269d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc1cc0d10c0 0x7fc1cc0d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.713+0000 7fc1d269d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc0cfa50 con 0x7fc1cc15ecd0 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.713+0000 7fc1d269d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc0d0a30 con 0x7fc1cc0d14a0 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.713+0000 7fc1d269d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1cc15c050 con 0x7fc1cc0d10c0 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1d0c3a640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc1cc15ecd0 0x7fc1cc0cae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38346/0 (socket says 172.21.15.59:38346) 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1d0c3a640 1 -- 172.21.15.59:0/2290004300 learned_addr learned my addr 172.21.15.59:0/2290004300 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465189759 0 0) 0x7fc1cc0cfa50 con 0x7fc1cc15ecd0 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1cbfff640 1 --2- 172.21.15.59:0/2290004300 >> v2:172.21.15.164:3300/0 conn(0x7fc1cc0d10c0 0x7fc1cc0d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b4003520 con 0x7fc1cc15ecd0 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2040160371 0 0) 0x7fc1b4003520 con 0x7fc1cc15ecd0 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 >> v2:172.21.15.164:3300/0 conn(0x7fc1cc0d10c0 msgr2=0x7fc1cc0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 --2- 172.21.15.59:0/2290004300 >> v2:172.21.15.164:3300/0 conn(0x7fc1cc0d10c0 0x7fc1cc0d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 >> v1:172.21.15.164:6789/0 conn(0x7fc1cc0d14a0 legacy=0x7fc1cc15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc0b45c0 con 0x7fc1cc15ecd0 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fc1c40019f0 con 0x7fc1cc15ecd0 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 >> v1:172.21.15.59:6789/0 conn(0x7fc1cc15ecd0 legacy=0x7fc1cc0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1b40039b0 0x7fc1b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1b4004800 0x7fc1b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b40051c0 0x7fc1b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b400c030 con 0x7fc1b40039b0 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b400c1e0 con 0x7fc1b40051c0 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b400c3e0 con 0x7fc1b4004800 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1cbfff640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1b40039b0 0x7fc1b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1cb7fe640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1b4004800 0x7fc1b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.714+0000 7fc1d0c3a640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b40051c0 0x7fc1b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.715+0000 7fc1cbfff640 1 -- 172.21.15.59:0/2290004300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1b4004800 msgr2=0x7fc1b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.715+0000 7fc1cbfff640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1b4004800 0x7fc1b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.715+0000 7fc1cbfff640 1 -- 172.21.15.59:0/2290004300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b40051c0 msgr2=0x7fc1b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.715+0000 7fc1cbfff640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b40051c0 0x7fc1b400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.715+0000 7fc1cbfff640 1 -- 172.21.15.59:0/2290004300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4003520 con 0x7fc1b40039b0 2024-03-06T07:23:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.715+0000 7fc1d0c3a640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b40051c0 0x7fc1b400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.715+0000 7fc1d0c3a640 1 -- 172.21.15.59:0/2290004300 reap_dead start 2024-03-06T07:23:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.715+0000 7fc1cbfff640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1b40039b0 0x7fc1b4003dd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009ac0 tx=0x7fc1b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c1c1ea6e40ce2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.715+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc039070 con 0x7fc1b40039b0 2024-03-06T07:23:55.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.716+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc02c6b0 con 0x7fc1b40039b0 2024-03-06T07:23:55.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.716+0000 7fc1caffd640 1 -- 172.21.15.59:0/2290004300 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc009690 con 0x7fc1b40039b0 2024-03-06T07:23:55.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.716+0000 7fc1d269d640 1 -- 172.21.15.59:0/2290004300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1b40039b0 msgr2=0x7fc1b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.716+0000 7fc1d269d640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1b40039b0 0x7fc1b4003dd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009ac0 tx=0x7fc1b4005830 comp rx=0 tx=0).stop 2024-03-06T07:23:55.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.716+0000 7fc1d269d640 1 -- 172.21.15.59:0/2290004300 shutdown_connections 2024-03-06T07:23:55.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.716+0000 7fc1d269d640 1 --2- 172.21.15.59:0/2290004300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1b40039b0 0x7fc1b4003dd0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.716+0000 7fc1d269d640 1 -- 172.21.15.59:0/2290004300 >> 172.21.15.59:0/2290004300 conn(0x7fc1cc0b3800 msgr2=0x7fc1cc0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.716+0000 7fc1d269d640 1 -- 172.21.15.59:0/2290004300 shutdown_connections 2024-03-06T07:23:55.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.716+0000 7fc1d269d640 1 -- 172.21.15.59:0/2290004300 wait complete. 2024-03-06T07:23:55.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.717+0000 7fc1d269d640 1 Processor -- start 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.717+0000 7fc1d269d640 1 -- start start 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.718+0000 7fc1d269d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b40039b0 0x7fc1cc154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.718+0000 7fc1d269d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1cc0cdb00 0x7fc1cc0cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.718+0000 7fc1d269d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1cc0ce4c0 0x7fc1cc1f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.718+0000 7fc1d269d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc15f440 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.718+0000 7fc1d269d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0d0060 con 0x7fc1b40039b0 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.718+0000 7fc1d269d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0cfcf0 con 0x7fc1cc0cdb00 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.718+0000 7fc1d0c3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1cc0ce4c0 0x7fc1cc1f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.718+0000 7fc1d0c3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1cc0ce4c0 0x7fc1cc1f7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57868/0 (socket says 172.21.15.59:57868) 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.718+0000 7fc1d0c3a640 1 -- 172.21.15.59:0/3045777971 learned_addr learned my addr 172.21.15.59:0/3045777971 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.718+0000 7fc1cbfff640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b40039b0 0x7fc1cc154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.718+0000 7fc1cb7fe640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1cc0cdb00 0x7fc1cc0cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.719+0000 7fc1d0c3a640 1 -- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1cc0cdb00 msgr2=0x7fc1cc0cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.719+0000 7fc1d0c3a640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1cc0cdb00 0x7fc1cc0cdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.719+0000 7fc1d0c3a640 1 -- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b40039b0 msgr2=0x7fc1cc154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.719+0000 7fc1d0c3a640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b40039b0 0x7fc1cc154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.719+0000 7fc1d0c3a640 1 -- 172.21.15.59:0/3045777971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc0ce940 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.719+0000 7fc1d0c3a640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1cc0ce4c0 0x7fc1cc1f7760 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc072610 tx=0x7fc1c400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.719+0000 7fc1c8ff9640 1 -- 172.21.15.59:0/3045777971 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c400ebc0 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.719+0000 7fc1c8ff9640 1 -- 172.21.15.59:0/3045777971 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c400ed60 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.719+0000 7fc1d269d640 1 -- 172.21.15.59:0/3045777971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc0cd5a0 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.719+0000 7fc1c8ff9640 1 -- 172.21.15.59:0/3045777971 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4014650 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.719+0000 7fc1d269d640 1 -- 172.21.15.59:0/3045777971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc09c750 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.720+0000 7fc1d269d640 1 -- 172.21.15.59:0/3045777971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1cc0d0820 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.720+0000 7fc1c8ff9640 1 -- 172.21.15.59:0/3045777971 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1c40147f0 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.721+0000 7fc1c8ff9640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1b806b4d0 0x7fc1b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.721+0000 7fc1c8ff9640 1 -- 172.21.15.59:0/3045777971 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc1c4089d40 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.721+0000 7fc1cbfff640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1b806b4d0 0x7fc1b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.722+0000 7fc1cbfff640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1b806b4d0 0x7fc1b806d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc036000 tx=0x7fc1bc030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.725+0000 7fc1c8ff9640 1 -- 172.21.15.59:0/3045777971 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1cc0d0820 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.905+0000 7fc1d269d640 1 -- 172.21.15.59:0/3045777971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc1cc161580 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.908+0000 7fc1c8ff9640 1 -- 172.21.15.59:0/3045777971 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc1c4058870 con 0x7fc1cc0ce4c0 2024-03-06T07:23:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 -- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1b806b4d0 msgr2=0x7fc1b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1b806b4d0 0x7fc1b806d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc036000 tx=0x7fc1bc030000 comp rx=0 tx=0).stop 2024-03-06T07:23:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 -- 172.21.15.59:0/3045777971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1cc0ce4c0 msgr2=0x7fc1cc1f7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1cc0ce4c0 0x7fc1cc1f7760 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc072610 tx=0x7fc1c400e530 comp rx=0 tx=0).stop 2024-03-06T07:23:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 -- 172.21.15.59:0/3045777971 shutdown_connections 2024-03-06T07:23:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1b806b4d0 0x7fc1b806d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1cc0ce4c0 0x7fc1cc1f7760 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1cc0cdb00 0x7fc1cc0cdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 --2- 172.21.15.59:0/3045777971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b40039b0 0x7fc1cc154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 -- 172.21.15.59:0/3045777971 >> 172.21.15.59:0/3045777971 conn(0x7fc1cc0b3800 msgr2=0x7fc1cc0b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 -- 172.21.15.59:0/3045777971 shutdown_connections 2024-03-06T07:23:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:55.910+0000 7fc1d269d640 1 -- 172.21.15.59:0/3045777971 wait complete. 2024-03-06T07:23:55.923 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:23:56.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.029+0000 7fef2c876640 1 Processor -- start 2024-03-06T07:23:56.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.029+0000 7fef2c876640 1 -- start start 2024-03-06T07:23:56.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.030+0000 7fef2c876640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fef280b5e20 0x7fef2806bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.030+0000 7fef2c876640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fef280cff60 0x7fef280d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.030+0000 7fef2c876640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef280d08b0 con 0x7fef281581c0 2024-03-06T07:23:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.030+0000 7fef2c876640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef2815b680 con 0x7fef280cff60 2024-03-06T07:23:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.030+0000 7fef2c876640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef2806b050 con 0x7fef280b5e20 2024-03-06T07:23:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.031+0000 7fef26d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fef280cff60 0x7fef280d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.031+0000 7fef26d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fef280cff60 0x7fef280d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57880/0 (socket says 172.21.15.59:57880) 2024-03-06T07:23:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.031+0000 7fef26d9d640 1 -- 172.21.15.59:0/2814304064 learned_addr learned my addr 172.21.15.59:0/2814304064 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.031+0000 7fef26d9d640 1 -- 172.21.15.59:0/2814304064 >> v2:172.21.15.164:3300/0 conn(0x7fef280b5e20 msgr2=0x7fef2806bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.031+0000 7fef26d9d640 1 --2- 172.21.15.59:0/2814304064 >> v2:172.21.15.164:3300/0 conn(0x7fef280b5e20 0x7fef2806bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.031+0000 7fef26d9d640 1 -- 172.21.15.59:0/2814304064 >> v1:172.21.15.164:6789/0 conn(0x7fef281581c0 legacy=0x7fef2806c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.031+0000 7fef26d9d640 1 -- 172.21.15.59:0/2814304064 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2806b330 con 0x7fef280cff60 2024-03-06T07:23:56.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.032+0000 7fef26d9d640 1 --2- 172.21.15.59:0/2814304064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef280cff60 0x7fef280d0370 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fef2806fb90 tx=0x7fef1002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3976f6007c461be3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.032+0000 7fef2559a640 1 -- 172.21.15.59:0/2814304064 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1002e070 con 0x7fef280cff60 2024-03-06T07:23:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.033+0000 7fef2559a640 1 -- 172.21.15.59:0/2814304064 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef10037070 con 0x7fef280cff60 2024-03-06T07:23:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.033+0000 7fef2559a640 1 -- 172.21.15.59:0/2814304064 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef10031a00 con 0x7fef280cff60 2024-03-06T07:23:56.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.034+0000 7fef2c876640 1 -- 172.21.15.59:0/2814304064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef280cff60 msgr2=0x7fef280d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.034+0000 7fef2c876640 1 --2- 172.21.15.59:0/2814304064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef280cff60 0x7fef280d0370 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fef2806fb90 tx=0x7fef1002b880 comp rx=0 tx=0).stop 2024-03-06T07:23:56.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.034+0000 7fef2c876640 1 -- 172.21.15.59:0/2814304064 shutdown_connections 2024-03-06T07:23:56.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.034+0000 7fef2c876640 1 --2- 172.21.15.59:0/2814304064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef280cff60 0x7fef280d0370 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.034+0000 7fef2c876640 1 --2- 172.21.15.59:0/2814304064 >> v2:172.21.15.164:3300/0 conn(0x7fef280b5e20 0x7fef2806bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.034+0000 7fef2c876640 1 -- 172.21.15.59:0/2814304064 >> 172.21.15.59:0/2814304064 conn(0x7fef280b4fe0 msgr2=0x7fef280b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.034+0000 7fef2c876640 1 -- 172.21.15.59:0/2814304064 shutdown_connections 2024-03-06T07:23:56.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.034+0000 7fef2c876640 1 -- 172.21.15.59:0/2814304064 wait complete. 2024-03-06T07:23:56.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.035+0000 7fef2c876640 1 Processor -- start 2024-03-06T07:23:56.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.036+0000 7fef2c876640 1 -- start start 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.036+0000 7fef2c876640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef280b5e20 0x7fef280cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.036+0000 7fef2c876640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef280cff60 0x7fef281506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.036+0000 7fef2c876640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef281581c0 0x7fef2814f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.036+0000 7fef2c876640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef28066d00 con 0x7fef280b5e20 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.036+0000 7fef2c876640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2815b960 con 0x7fef280cff60 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.036+0000 7fef2c876640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef28066e80 con 0x7fef281581c0 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.036+0000 7fef2659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef280b5e20 0x7fef280cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef2659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef280b5e20 0x7fef280cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57892/0 (socket says 172.21.15.59:57892) 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef2659c640 1 -- 172.21.15.59:0/3745800229 learned_addr learned my addr 172.21.15.59:0/3745800229 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef25d9b640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef280cff60 0x7fef281506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef26d9d640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef281581c0 0x7fef2814f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef2659c640 1 -- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef281581c0 msgr2=0x7fef2814f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef2659c640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef281581c0 0x7fef2814f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef2659c640 1 -- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef280cff60 msgr2=0x7fef281506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef2659c640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef280cff60 0x7fef281506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef2659c640 1 -- 172.21.15.59:0/3745800229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef280cc8a0 con 0x7fef280b5e20 2024-03-06T07:23:56.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef25d9b640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef280cff60 0x7fef281506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:56.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef2659c640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef280b5e20 0x7fef280cd1e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fef1800b3e0 tx=0x7fef1800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef1effd640 1 -- 172.21.15.59:0/3745800229 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18015020 con 0x7fef280b5e20 2024-03-06T07:23:56.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef1effd640 1 -- 172.21.15.59:0/3745800229 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef18011850 con 0x7fef280b5e20 2024-03-06T07:23:56.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef2c876640 1 -- 172.21.15.59:0/3745800229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef280ccb10 con 0x7fef280b5e20 2024-03-06T07:23:56.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.037+0000 7fef1effd640 1 -- 172.21.15.59:0/3745800229 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18011c40 con 0x7fef280b5e20 2024-03-06T07:23:56.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.038+0000 7fef2c876640 1 -- 172.21.15.59:0/3745800229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef280ae480 con 0x7fef280b5e20 2024-03-06T07:23:56.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.039+0000 7fef2c876640 1 -- 172.21.15.59:0/3745800229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2809e4e0 con 0x7fef280b5e20 2024-03-06T07:23:56.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.039+0000 7fef1effd640 1 -- 172.21.15.59:0/3745800229 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef18018020 con 0x7fef280b5e20 2024-03-06T07:23:56.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.039+0000 7fef1effd640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef0806b4d0 0x7fef0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.039+0000 7fef1effd640 1 -- 172.21.15.59:0/3745800229 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fef18089d90 con 0x7fef280b5e20 2024-03-06T07:23:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.040+0000 7fef25d9b640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef0806b4d0 0x7fef0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.040+0000 7fef25d9b640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef0806b4d0 0x7fef0806d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fef2815a7a0 tx=0x7fef20009570 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.044+0000 7fef1effd640 1 -- 172.21.15.59:0/3745800229 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef1801f020 con 0x7fef280b5e20 2024-03-06T07:23:56.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.226+0000 7fef2c876640 1 -- 172.21.15.59:0/3745800229 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fef28068ff0 con 0x7fef280b5e20 2024-03-06T07:23:56.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.229+0000 7fef1effd640 1 -- 172.21.15.59:0/3745800229 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v15) v1 ==== 108+0+0 (secure 0 0 0) 0x7fef180172c0 con 0x7fef280b5e20 2024-03-06T07:23:56.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 -- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef0806b4d0 msgr2=0x7fef0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef0806b4d0 0x7fef0806d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fef2815a7a0 tx=0x7fef20009570 comp rx=0 tx=0).stop 2024-03-06T07:23:56.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 -- 172.21.15.59:0/3745800229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef280b5e20 msgr2=0x7fef280cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef280b5e20 0x7fef280cd1e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fef1800b3e0 tx=0x7fef1800b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 -- 172.21.15.59:0/3745800229 shutdown_connections 2024-03-06T07:23:56.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef0806b4d0 0x7fef0806d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef281581c0 0x7fef2814f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef280cff60 0x7fef281506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 --2- 172.21.15.59:0/3745800229 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef280b5e20 0x7fef280cd1e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 -- 172.21.15.59:0/3745800229 >> 172.21.15.59:0/3745800229 conn(0x7fef280b4fe0 msgr2=0x7fef28151640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 -- 172.21.15.59:0/3745800229 shutdown_connections 2024-03-06T07:23:56.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.231+0000 7fef2c876640 1 -- 172.21.15.59:0/3745800229 wait complete. 2024-03-06T07:23:56.244 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:23:56.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.344+0000 7f398a8f4640 1 Processor -- start 2024-03-06T07:23:56.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.344+0000 7f398a8f4640 1 -- start start 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.344+0000 7f398a8f4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f39840b4170 0x7f39840b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.344+0000 7f398a8f4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f39840d4760 0x7f39840d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.344+0000 7f398a8f4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39840d25d0 con 0x7f39840b4dd0 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.344+0000 7f398a8f4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39840d4bf0 con 0x7f39840b4170 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.344+0000 7f398a8f4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39840ce9e0 con 0x7f39840d4760 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.344+0000 7f3983fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f39840b4170 0x7f39840b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.344+0000 7f3983fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f39840b4170 0x7f39840b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57908/0 (socket says 172.21.15.59:57908) 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.344+0000 7f3983fff640 1 -- 172.21.15.59:0/112953419 learned_addr learned my addr 172.21.15.59:0/112953419 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.345+0000 7f39837fe640 1 --2- 172.21.15.59:0/112953419 >> v2:172.21.15.164:3300/0 conn(0x7f39840d4760 0x7f39840d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.345+0000 7f3983fff640 1 -- 172.21.15.59:0/112953419 >> v2:172.21.15.164:3300/0 conn(0x7f39840d4760 msgr2=0x7f39840d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.345+0000 7f3983fff640 1 --2- 172.21.15.59:0/112953419 >> v2:172.21.15.164:3300/0 conn(0x7f39840d4760 0x7f39840d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.345+0000 7f3983fff640 1 -- 172.21.15.59:0/112953419 >> v1:172.21.15.164:6789/0 conn(0x7f39840b4dd0 legacy=0x7f39840cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.345+0000 7f3983fff640 1 -- 172.21.15.59:0/112953419 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39840d2870 con 0x7f39840b4170 2024-03-06T07:23:56.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f3983fff640 1 --2- 172.21.15.59:0/112953419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39840b4170 0x7f39840b6a90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f398408d0c0 tx=0x7f397402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80946c251e673a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f3982ffd640 1 -- 172.21.15.59:0/112953419 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974034070 con 0x7f39840b4170 2024-03-06T07:23:56.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f3982ffd640 1 -- 172.21.15.59:0/112953419 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3974030860 con 0x7f39840b4170 2024-03-06T07:23:56.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f3982ffd640 1 -- 172.21.15.59:0/112953419 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974030c30 con 0x7f39840b4170 2024-03-06T07:23:56.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f398a8f4640 1 -- 172.21.15.59:0/112953419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39840b4170 msgr2=0x7f39840b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f398a8f4640 1 --2- 172.21.15.59:0/112953419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39840b4170 0x7f39840b6a90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f398408d0c0 tx=0x7f397402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f398a8f4640 1 -- 172.21.15.59:0/112953419 shutdown_connections 2024-03-06T07:23:56.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f398a8f4640 1 --2- 172.21.15.59:0/112953419 >> v2:172.21.15.164:3300/0 conn(0x7f39840d4760 0x7f39840d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f398a8f4640 1 --2- 172.21.15.59:0/112953419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39840b4170 0x7f39840b6a90 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f398a8f4640 1 -- 172.21.15.59:0/112953419 >> 172.21.15.59:0/112953419 conn(0x7f39840d3c40 msgr2=0x7f39840d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f398a8f4640 1 -- 172.21.15.59:0/112953419 shutdown_connections 2024-03-06T07:23:56.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.346+0000 7f398a8f4640 1 -- 172.21.15.59:0/112953419 wait complete. 2024-03-06T07:23:56.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.347+0000 7f398a8f4640 1 Processor -- start 2024-03-06T07:23:56.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.348+0000 7f398a8f4640 1 -- start start 2024-03-06T07:23:56.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.348+0000 7f398a8f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39840b4170 0x7f398415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.348+0000 7f398a8f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39840b4dd0 0x7f398415eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.348+0000 7f398a8f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f398415bbe0 0x7f398415f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.348+0000 7f398a8f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39840b47d0 con 0x7f398415bbe0 2024-03-06T07:23:56.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.348+0000 7f398a8f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39840b4550 con 0x7f39840b4170 2024-03-06T07:23:56.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.348+0000 7f398a8f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39840b4950 con 0x7f39840b4dd0 2024-03-06T07:23:56.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.348+0000 7f3988e91640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f398415bbe0 0x7f398415f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.348+0000 7f3988e91640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f398415bbe0 0x7f398415f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57910/0 (socket says 172.21.15.59:57910) 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.348+0000 7f3988e91640 1 -- 172.21.15.59:0/2244796750 learned_addr learned my addr 172.21.15.59:0/2244796750 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f3983fff640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39840b4170 0x7f398415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f39837fe640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39840b4dd0 0x7f398415eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f3988e91640 1 -- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39840b4dd0 msgr2=0x7f398415eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f3988e91640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39840b4dd0 0x7f398415eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f3988e91640 1 -- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39840b4170 msgr2=0x7f398415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f3988e91640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39840b4170 0x7f398415b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f3988e91640 1 -- 172.21.15.59:0/2244796750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398415c0f0 con 0x7f398415bbe0 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f3988e91640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f398415bbe0 0x7f398415f410 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f397c00c7d0 tx=0x7f397c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f3980ff9640 1 -- 172.21.15.59:0/2244796750 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c003b70 con 0x7f398415bbe0 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f3980ff9640 1 -- 172.21.15.59:0/2244796750 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397c003d30 con 0x7f398415bbe0 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f398a8f4640 1 -- 172.21.15.59:0/2244796750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398415c3d0 con 0x7f398415bbe0 2024-03-06T07:23:56.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f3980ff9640 1 -- 172.21.15.59:0/2244796750 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397c00aa90 con 0x7f398415bbe0 2024-03-06T07:23:56.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.349+0000 7f398a8f4640 1 -- 172.21.15.59:0/2244796750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398415c990 con 0x7f398415bbe0 2024-03-06T07:23:56.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.350+0000 7f398a8f4640 1 -- 172.21.15.59:0/2244796750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398404f330 con 0x7f398415bbe0 2024-03-06T07:23:56.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.351+0000 7f3980ff9640 1 -- 172.21.15.59:0/2244796750 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f397c00ac30 con 0x7f398415bbe0 2024-03-06T07:23:56.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.351+0000 7f3980ff9640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f397006b500 0x7f397006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.351+0000 7f3980ff9640 1 -- 172.21.15.59:0/2244796750 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f397c089da0 con 0x7f398415bbe0 2024-03-06T07:23:56.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.351+0000 7f3983fff640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f397006b500 0x7f397006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.352+0000 7f3983fff640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f397006b500 0x7f397006d9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3974036000 tx=0x7f397403b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.355+0000 7f3980ff9640 1 -- 172.21.15.59:0/2244796750 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f397c058fb0 con 0x7f398415bbe0 2024-03-06T07:23:56.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:56.605+0000 7f398a8f4640 1 -- 172.21.15.59:0/2244796750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f39840733b0 con 0x7f398415bbe0 2024-03-06T07:23:57.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.590+0000 7f3980ff9640 1 -- 172.21.15.59:0/2244796750 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v135) v1 ==== 133+0+0 (secure 0 0 0) 0x7f397c0588b0 con 0x7f398415bbe0 2024-03-06T07:23:57.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.592+0000 7f398a8f4640 1 -- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f397006b500 msgr2=0x7f397006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.592+0000 7f398a8f4640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f397006b500 0x7f397006d9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3974036000 tx=0x7f397403b040 comp rx=0 tx=0).stop 2024-03-06T07:23:57.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.592+0000 7f398a8f4640 1 -- 172.21.15.59:0/2244796750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f398415bbe0 msgr2=0x7f398415f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.592+0000 7f398a8f4640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f398415bbe0 0x7f398415f410 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f397c00c7d0 tx=0x7f397c00cc90 comp rx=0 tx=0).stop 2024-03-06T07:23:57.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.592+0000 7f398a8f4640 1 -- 172.21.15.59:0/2244796750 shutdown_connections 2024-03-06T07:23:57.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.592+0000 7f398a8f4640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f397006b500 0x7f397006d9c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.592+0000 7f398a8f4640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f398415bbe0 0x7f398415f410 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.592+0000 7f398a8f4640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39840b4dd0 0x7f398415eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.592+0000 7f398a8f4640 1 --2- 172.21.15.59:0/2244796750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39840b4170 0x7f398415b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.592+0000 7f398a8f4640 1 -- 172.21.15.59:0/2244796750 >> 172.21.15.59:0/2244796750 conn(0x7f39840d3c40 msgr2=0x7f398406b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.593+0000 7f398a8f4640 1 -- 172.21.15.59:0/2244796750 shutdown_connections 2024-03-06T07:23:57.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.593+0000 7f398a8f4640 1 -- 172.21.15.59:0/2244796750 wait complete. 2024-03-06T07:23:57.608 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-03-06T07:23:57.608 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-03-06T07:23:57.608 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:23:57.609 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:57.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.715+0000 7fd05d66b640 1 Processor -- start 2024-03-06T07:23:57.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.715+0000 7fd05d66b640 1 -- start start 2024-03-06T07:23:57.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.716+0000 7fd05d66b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd05815e0f0 0x7fd0580cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.716+0000 7fd05d66b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd0580d0660 0x7fd0580b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.716+0000 7fd05d66b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05815e4d0 con 0x7fd0580cf850 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.716+0000 7fd05d66b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0580b1410 con 0x7fd0580d0660 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.716+0000 7fd05d66b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05806c8b0 con 0x7fd05815e0f0 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.716+0000 7fd04e7fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd0580cf850 0x7fd0580cff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38362/0 (socket says 172.21.15.59:38362) 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.716+0000 7fd04e7fc640 1 -- 172.21.15.59:0/3348874764 learned_addr learned my addr 172.21.15.59:0/3348874764 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.716+0000 7fd0577fe640 1 --2- 172.21.15.59:0/3348874764 >> v2:172.21.15.164:3300/0 conn(0x7fd0580d0660 0x7fd0580b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.716+0000 7fd056ffd640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd05815e0f0 0x7fd0580cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.716+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519972907 0 0) 0x7fd05815e4d0 con 0x7fd0580cf850 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.716+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd044003520 con 0x7fd0580cf850 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2960643499 0 0) 0x7fd044003520 con 0x7fd0580cf850 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd05815e0f0 msgr2=0x7fd0580cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd05815e0f0 0x7fd0580cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 >> v2:172.21.15.164:3300/0 conn(0x7fd0580d0660 msgr2=0x7fd0580b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 --2- 172.21.15.59:0/3348874764 >> v2:172.21.15.164:3300/0 conn(0x7fd0580d0660 0x7fd0580b25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0580cdac0 con 0x7fd0580cf850 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fd040001670 con 0x7fd0580cf850 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 >> v1:172.21.15.59:6789/0 conn(0x7fd0580cf850 legacy=0x7fd0580cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0440039b0 0x7fd044003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd05815e0f0 0x7fd044004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd044004d40 0x7fd0440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05806c8b0 con 0x7fd044004d40 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04400b9c0 con 0x7fd0440039b0 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04400bb40 con 0x7fd05815e0f0 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0577fe640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd044004d40 0x7fd0440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd056ffd640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0440039b0 0x7fd044003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd04e7fc640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd05815e0f0 0x7fd044004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0577fe640 1 -- 172.21.15.59:0/3348874764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd05815e0f0 msgr2=0x7fd044004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0577fe640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd05815e0f0 0x7fd044004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0577fe640 1 -- 172.21.15.59:0/3348874764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0440039b0 msgr2=0x7fd044003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0577fe640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0440039b0 0x7fd044003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.717+0000 7fd0577fe640 1 -- 172.21.15.59:0/3348874764 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd044003520 con 0x7fd044004d40 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.718+0000 7fd0577fe640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd044004d40 0x7fd0440051c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd050009fc0 tx=0x7fd044005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d0d255fe019899 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.718+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050039070 con 0x7fd044004d40 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.718+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05002cc60 con 0x7fd044004d40 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.718+0000 7fd0567fc640 1 -- 172.21.15.59:0/3348874764 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050009c10 con 0x7fd044004d40 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.719+0000 7fd05d66b640 1 -- 172.21.15.59:0/3348874764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd044004d40 msgr2=0x7fd0440051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.719+0000 7fd05d66b640 1 --2- 172.21.15.59:0/3348874764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd044004d40 0x7fd0440051c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd050009fc0 tx=0x7fd044005830 comp rx=0 tx=0).stop 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.719+0000 7fd0577fe640 1 -- 172.21.15.59:0/3348874764 reap_dead start 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.719+0000 7fd05d66b640 1 -- 172.21.15.59:0/3348874764 shutdown_connections 2024-03-06T07:23:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.719+0000 7fd05d66b640 1 -- 172.21.15.59:0/3348874764 >> 172.21.15.59:0/3348874764 conn(0x7fd0580b35a0 msgr2=0x7fd0580b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.719+0000 7fd05d66b640 1 -- 172.21.15.59:0/3348874764 shutdown_connections 2024-03-06T07:23:57.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.719+0000 7fd05d66b640 1 -- 172.21.15.59:0/3348874764 wait complete. 2024-03-06T07:23:57.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.720+0000 7fd05d66b640 1 Processor -- start 2024-03-06T07:23:57.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.720+0000 7fd05d66b640 1 -- start start 2024-03-06T07:23:57.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd05d66b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd044004d40 0x7fd05815d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd05d66b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd058156b10 0x7fd05809c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd05d66b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd05809c900 0x7fd0580c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd05d66b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0580d0cc0 con 0x7fd05809c900 2024-03-06T07:23:57.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd05d66b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05806c540 con 0x7fd044004d40 2024-03-06T07:23:57.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd05d66b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0580cd3c0 con 0x7fd058156b10 2024-03-06T07:23:57.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd0577fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd05809c900 0x7fd0580c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd0577fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd05809c900 0x7fd0580c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57938/0 (socket says 172.21.15.59:57938) 2024-03-06T07:23:57.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd0577fe640 1 -- 172.21.15.59:0/1142085969 learned_addr learned my addr 172.21.15.59:0/1142085969 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:57.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd056ffd640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd044004d40 0x7fd05815d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd04e7fc640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd058156b10 0x7fd05809c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd0577fe640 1 -- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd058156b10 msgr2=0x7fd05809c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd0577fe640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd058156b10 0x7fd05809c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd0577fe640 1 -- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd044004d40 msgr2=0x7fd05815d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd0577fe640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd044004d40 0x7fd05815d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd0577fe640 1 -- 172.21.15.59:0/1142085969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0580b60d0 con 0x7fd05809c900 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd0577fe640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd05809c900 0x7fd0580c9e10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd05000e230 tx=0x7fd050009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.721+0000 7fd04ffff640 1 -- 172.21.15.59:0/1142085969 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050039030 con 0x7fd05809c900 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.722+0000 7fd04ffff640 1 -- 172.21.15.59:0/1142085969 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05003a040 con 0x7fd05809c900 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.722+0000 7fd05d66b640 1 -- 172.21.15.59:0/1142085969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd058156f50 con 0x7fd05809c900 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.722+0000 7fd04ffff640 1 -- 172.21.15.59:0/1142085969 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd050009c10 con 0x7fd05809c900 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.722+0000 7fd05d66b640 1 -- 172.21.15.59:0/1142085969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0580b6650 con 0x7fd05809c900 2024-03-06T07:23:57.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.722+0000 7fd05d66b640 1 -- 172.21.15.59:0/1142085969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05804f330 con 0x7fd05809c900 2024-03-06T07:23:57.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.723+0000 7fd04ffff640 1 -- 172.21.15.59:0/1142085969 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd050045030 con 0x7fd05809c900 2024-03-06T07:23:57.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.723+0000 7fd04ffff640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd03806b4d0 0x7fd03806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.723+0000 7fd04ffff640 1 -- 172.21.15.59:0/1142085969 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd0500abcb0 con 0x7fd05809c900 2024-03-06T07:23:57.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.724+0000 7fd056ffd640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd03806b4d0 0x7fd03806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.724+0000 7fd056ffd640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd03806b4d0 0x7fd03806d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd048009b40 tx=0x7fd048009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.728+0000 7fd04ffff640 1 -- 172.21.15.59:0/1142085969 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd05004c060 con 0x7fd05809c900 2024-03-06T07:23:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.909+0000 7fd05d66b640 1 -- 172.21.15.59:0/1142085969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0580b7930 con 0x7fd05809c900 2024-03-06T07:23:57.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.910+0000 7fd04ffff640 1 -- 172.21.15.59:0/1142085969 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+13010 (secure 0 0 0) 0x7fd05007ae30 con 0x7fd05809c900 2024-03-06T07:23:57.911 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:23:57.912 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":50,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:23:55.577139+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:6837/3749315375":"2024-03-07T07:23:52.572229+0000","172.21.15.164:6835/3749315375":"2024-03-07T07:23:52.572229+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 -- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd03806b4d0 msgr2=0x7fd03806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd03806b4d0 0x7fd03806d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd048009b40 tx=0x7fd048009190 comp rx=0 tx=0).stop 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 -- 172.21.15.59:0/1142085969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd05809c900 msgr2=0x7fd0580c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd05809c900 0x7fd0580c9e10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd05000e230 tx=0x7fd050009750 comp rx=0 tx=0).stop 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 -- 172.21.15.59:0/1142085969 shutdown_connections 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd03806b4d0 0x7fd03806d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd05809c900 0x7fd0580c9e10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd058156b10 0x7fd05809c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 --2- 172.21.15.59:0/1142085969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd044004d40 0x7fd05815d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 -- 172.21.15.59:0/1142085969 >> 172.21.15.59:0/1142085969 conn(0x7fd0580b35a0 msgr2=0x7fd0580700f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 -- 172.21.15.59:0/1142085969 shutdown_connections 2024-03-06T07:23:57.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:57.912+0000 7fd05d66b640 1 -- 172.21.15.59:0/1142085969 wait complete. 2024-03-06T07:23:57.925 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:57.925 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:58.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.027+0000 7f9949c1c640 1 Processor -- start 2024-03-06T07:23:58.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.027+0000 7f9949c1c640 1 -- start start 2024-03-06T07:23:58.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.028+0000 7f9949c1c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99440d10c0 0x7f99440d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.028+0000 7f9949c1c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f99440d14a0 0x7f994415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.028+0000 7f9949c1c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f994415eb00 0x7f99440cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.028+0000 7f9949c1c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994415c050 con 0x7f994415eb00 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.028+0000 7f9949c1c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99440d0060 con 0x7f99440d14a0 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.028+0000 7f9949c1c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99440cfa50 con 0x7f99440d10c0 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.028+0000 7f9943fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f994415eb00 0x7f99440cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.028+0000 7f9943fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f994415eb00 0x7f99440cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57940/0 (socket says 172.21.15.59:57940) 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.028+0000 7f9943fff640 1 -- 172.21.15.59:0/3027425267 learned_addr learned my addr 172.21.15.59:0/3027425267 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.029+0000 7f99437fe640 1 --2- 172.21.15.59:0/3027425267 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99440d10c0 0x7f99440d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.029+0000 7f9942ffd640 1 --2- 172.21.15.59:0/3027425267 >> v2:172.21.15.164:3300/0 conn(0x7f99440d14a0 0x7f994415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.029+0000 7f9943fff640 1 -- 172.21.15.59:0/3027425267 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99440d10c0 msgr2=0x7f99440d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.029+0000 7f9943fff640 1 --2- 172.21.15.59:0/3027425267 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99440d10c0 0x7f99440d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.029+0000 7f9943fff640 1 -- 172.21.15.59:0/3027425267 >> v2:172.21.15.164:3300/0 conn(0x7f99440d14a0 msgr2=0x7f994415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.029+0000 7f9943fff640 1 --2- 172.21.15.59:0/3027425267 >> v2:172.21.15.164:3300/0 conn(0x7f99440d14a0 0x7f994415e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.029+0000 7f9943fff640 1 -- 172.21.15.59:0/3027425267 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99440d0a30 con 0x7f994415eb00 2024-03-06T07:23:58.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.029+0000 7f9943fff640 1 --2- 172.21.15.59:0/3027425267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994415eb00 0x7f99440cae80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f993c009e50 tx=0x7f993c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51df37e4e422352 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f99427fc640 1 -- 172.21.15.59:0/3027425267 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c02ca70 con 0x7f994415eb00 2024-03-06T07:23:58.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f99427fc640 1 -- 172.21.15.59:0/3027425267 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c02cc10 con 0x7f994415eb00 2024-03-06T07:23:58.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f99427fc640 1 -- 172.21.15.59:0/3027425267 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c0343f0 con 0x7f994415eb00 2024-03-06T07:23:58.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f9949c1c640 1 -- 172.21.15.59:0/3027425267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994415eb00 msgr2=0x7f99440cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f9949c1c640 1 --2- 172.21.15.59:0/3027425267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994415eb00 0x7f99440cae80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f993c009e50 tx=0x7f993c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f9949c1c640 1 -- 172.21.15.59:0/3027425267 shutdown_connections 2024-03-06T07:23:58.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f9949c1c640 1 --2- 172.21.15.59:0/3027425267 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994415eb00 0x7f99440cae80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f9949c1c640 1 --2- 172.21.15.59:0/3027425267 >> v2:172.21.15.164:3300/0 conn(0x7f99440d14a0 0x7f994415e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f9949c1c640 1 --2- 172.21.15.59:0/3027425267 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99440d10c0 0x7f99440d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f9949c1c640 1 -- 172.21.15.59:0/3027425267 >> 172.21.15.59:0/3027425267 conn(0x7f99440b3800 msgr2=0x7f99440b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f9949c1c640 1 -- 172.21.15.59:0/3027425267 shutdown_connections 2024-03-06T07:23:58.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.030+0000 7f9949c1c640 1 -- 172.21.15.59:0/3027425267 wait complete. 2024-03-06T07:23:58.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.031+0000 7f9949c1c640 1 Processor -- start 2024-03-06T07:23:58.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.031+0000 7f9949c1c640 1 -- start start 2024-03-06T07:23:58.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9949c1c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99440d10c0 0x7f99440cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9949c1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99440d14a0 0x7f99440cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9949c1c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f994415eb00 0x7f99440cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9949c1c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994415f440 con 0x7f99440d14a0 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9949c1c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994415f160 con 0x7f99440d10c0 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9949c1c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99440cfcf0 con 0x7f994415eb00 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9942ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99440d14a0 0x7f99440cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9942ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99440d14a0 0x7f99440cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57950/0 (socket says 172.21.15.59:57950) 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9942ffd640 1 -- 172.21.15.59:0/1582408283 learned_addr learned my addr 172.21.15.59:0/1582408283 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f99437fe640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99440d10c0 0x7f99440cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9943fff640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f994415eb00 0x7f99440cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9942ffd640 1 -- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f994415eb00 msgr2=0x7f99440cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.032+0000 7f9942ffd640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f994415eb00 0x7f99440cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.033+0000 7f9942ffd640 1 -- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99440d10c0 msgr2=0x7f99440cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.033+0000 7f9942ffd640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99440d10c0 0x7f99440cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.033+0000 7f9942ffd640 1 -- 172.21.15.59:0/1582408283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99440cd1c0 con 0x7f99440d14a0 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.033+0000 7f99437fe640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99440d10c0 0x7f99440cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:58.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.033+0000 7f9942ffd640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99440d14a0 0x7f99440cc430 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f993800d5f0 tx=0x7f993800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.033+0000 7f992bfff640 1 -- 172.21.15.59:0/1582408283 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99380125f0 con 0x7f99440d14a0 2024-03-06T07:23:58.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.033+0000 7f992bfff640 1 -- 172.21.15.59:0/1582408283 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9938019070 con 0x7f99440d14a0 2024-03-06T07:23:58.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.033+0000 7f9949c1c640 1 -- 172.21.15.59:0/1582408283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99440cd4a0 con 0x7f99440d14a0 2024-03-06T07:23:58.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.033+0000 7f992bfff640 1 -- 172.21.15.59:0/1582408283 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99380129d0 con 0x7f99440d14a0 2024-03-06T07:23:58.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.033+0000 7f9949c1c640 1 -- 172.21.15.59:0/1582408283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99440b4d70 con 0x7f99440d14a0 2024-03-06T07:23:58.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.034+0000 7f9949c1c640 1 -- 172.21.15.59:0/1582408283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994409fe60 con 0x7f99440d14a0 2024-03-06T07:23:58.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.034+0000 7f992bfff640 1 -- 172.21.15.59:0/1582408283 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9938012b70 con 0x7f99440d14a0 2024-03-06T07:23:58.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.035+0000 7f992bfff640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993006b4d0 0x7f993006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.035+0000 7f992bfff640 1 -- 172.21.15.59:0/1582408283 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f993808a360 con 0x7f99440d14a0 2024-03-06T07:23:58.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.035+0000 7f99437fe640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993006b4d0 0x7f993006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.035+0000 7f99437fe640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993006b4d0 0x7f993006d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f993400ba50 tx=0x7f9934008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.039+0000 7f992bfff640 1 -- 172.21.15.59:0/1582408283 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9938059450 con 0x7f99440d14a0 2024-03-06T07:23:58.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.220+0000 7f9949c1c640 1 -- 172.21.15.59:0/1582408283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9944068ff0 con 0x7f99440d14a0 2024-03-06T07:23:58.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.220+0000 7f992bfff640 1 -- 172.21.15.59:0/1582408283 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9938058d50 con 0x7f99440d14a0 2024-03-06T07:23:58.222 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:23:58.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 -- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993006b4d0 msgr2=0x7f993006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993006b4d0 0x7f993006d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f993400ba50 tx=0x7f9934008040 comp rx=0 tx=0).stop 2024-03-06T07:23:58.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 -- 172.21.15.59:0/1582408283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99440d14a0 msgr2=0x7f99440cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99440d14a0 0x7f99440cc430 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f993800d5f0 tx=0x7f993800dab0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 -- 172.21.15.59:0/1582408283 shutdown_connections 2024-03-06T07:23:58.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993006b4d0 0x7f993006d990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f994415eb00 0x7f99440cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99440d14a0 0x7f99440cc430 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 --2- 172.21.15.59:0/1582408283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99440d10c0 0x7f99440cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 -- 172.21.15.59:0/1582408283 >> 172.21.15.59:0/1582408283 conn(0x7f99440b3800 msgr2=0x7f994415db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 -- 172.21.15.59:0/1582408283 shutdown_connections 2024-03-06T07:23:58.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.222+0000 7f9949c1c640 1 -- 172.21.15.59:0/1582408283 wait complete. 2024-03-06T07:23:58.235 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:23:58.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.335+0000 7fdbaeff6640 1 Processor -- start 2024-03-06T07:23:58.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.335+0000 7fdbaeff6640 1 -- start start 2024-03-06T07:23:58.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.336+0000 7fdbaeff6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdba80d2710 0x7fdba80b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.336+0000 7fdbaeff6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba80b3430 con 0x7fdba80d22b0 2024-03-06T07:23:58.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.336+0000 7fdbaeff6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba80b3710 con 0x7fdba815dfe0 2024-03-06T07:23:58.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.336+0000 7fdbaeff6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba80b31e0 con 0x7fdba80d2710 2024-03-06T07:23:58.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.336+0000 7fdba7fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdba80d22b0 0x7fdba80b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38370/0 (socket says 172.21.15.59:38370) 2024-03-06T07:23:58.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.336+0000 7fdba7fff640 1 -- 172.21.15.59:0/2246698719 learned_addr learned my addr 172.21.15.59:0/2246698719 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:58.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416175554 0 0) 0x7fdba80b3430 con 0x7fdba80d22b0 2024-03-06T07:23:58.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb90003520 con 0x7fdba80d22b0 2024-03-06T07:23:58.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdbad593640 1 --2- 172.21.15.59:0/2246698719 >> v2:172.21.15.164:3300/0 conn(0x7fdba80d2710 0x7fdba80b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3795352323 0 0) 0x7fdb90003520 con 0x7fdba80d22b0 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 >> v2:172.21.15.164:3300/0 conn(0x7fdba80d2710 msgr2=0x7fdba80b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 --2- 172.21.15.59:0/2246698719 >> v2:172.21.15.164:3300/0 conn(0x7fdba80d2710 0x7fdba80b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 >> v1:172.21.15.164:6789/0 conn(0x7fdba815dfe0 legacy=0x7fdba80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba80b39f0 con 0x7fdba80d22b0 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fdb9c001670 con 0x7fdba80d22b0 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 >> v1:172.21.15.59:6789/0 conn(0x7fdba80d22b0 legacy=0x7fdba80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 --2- 172.21.15.59:0/2246698719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb900039e0 0x7fdb90003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 --2- 172.21.15.59:0/2246698719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb90004800 0x7fdb90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 --2- 172.21.15.59:0/2246698719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb900051c0 0x7fdb9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba80b31e0 con 0x7fdb900039e0 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9000c090 con 0x7fdb900051c0 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9000c290 con 0x7fdb90004800 2024-03-06T07:23:58.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdbacd92640 1 --2- 172.21.15.59:0/2246698719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb900039e0 0x7fdb90003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdba7fff640 1 --2- 172.21.15.59:0/2246698719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb90004800 0x7fdb90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdbad593640 1 --2- 172.21.15.59:0/2246698719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb900051c0 0x7fdb9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdbacd92640 1 -- 172.21.15.59:0/2246698719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb90004800 msgr2=0x7fdb90004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdbacd92640 1 --2- 172.21.15.59:0/2246698719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb90004800 0x7fdb90004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdbacd92640 1 -- 172.21.15.59:0/2246698719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb900051c0 msgr2=0x7fdb9000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdbacd92640 1 --2- 172.21.15.59:0/2246698719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb900051c0 0x7fdb9000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdbacd92640 1 -- 172.21.15.59:0/2246698719 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb90003520 con 0x7fdb900039e0 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.337+0000 7fdbad593640 1 -- 172.21.15.59:0/2246698719 reap_dead start 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.338+0000 7fdbacd92640 1 --2- 172.21.15.59:0/2246698719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb900039e0 0x7fdb90003e00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdba0001e50 tx=0x7fdb90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4619049410945d48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.338+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0001a70 con 0x7fdb900039e0 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.338+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba0003b70 con 0x7fdb900039e0 2024-03-06T07:23:58.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.338+0000 7fdba77fe640 1 -- 172.21.15.59:0/2246698719 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba00050d0 con 0x7fdb900039e0 2024-03-06T07:23:58.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.339+0000 7fdbaeff6640 1 -- 172.21.15.59:0/2246698719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb900039e0 msgr2=0x7fdb90003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.339+0000 7fdbaeff6640 1 --2- 172.21.15.59:0/2246698719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb900039e0 0x7fdb90003e00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdba0001e50 tx=0x7fdb90005830 comp rx=0 tx=0).stop 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.339+0000 7fdbaeff6640 1 -- 172.21.15.59:0/2246698719 shutdown_connections 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.339+0000 7fdbaeff6640 1 --2- 172.21.15.59:0/2246698719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb900039e0 0x7fdb90003e00 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.339+0000 7fdbaeff6640 1 -- 172.21.15.59:0/2246698719 >> 172.21.15.59:0/2246698719 conn(0x7fdba80d3cd0 msgr2=0x7fdba80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.339+0000 7fdbaeff6640 1 -- 172.21.15.59:0/2246698719 shutdown_connections 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.339+0000 7fdbaeff6640 1 -- 172.21.15.59:0/2246698719 wait complete. 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.340+0000 7fdbaeff6640 1 Processor -- start 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.340+0000 7fdbaeff6640 1 -- start start 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbaeff6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb900039e0 0x7fdba815c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbaeff6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba80b83a0 0x7fdba80b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbaeff6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdba80b6e10 0x7fdba80b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbaeff6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba80d1c70 con 0x7fdba80b6e10 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbaeff6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba80b3d60 con 0x7fdb900039e0 2024-03-06T07:23:58.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbaeff6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba80d1df0 con 0x7fdba80b83a0 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbad593640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdba80b6e10 0x7fdba80b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbad593640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdba80b6e10 0x7fdba80b8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57968/0 (socket says 172.21.15.59:57968) 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbad593640 1 -- 172.21.15.59:0/4153131966 learned_addr learned my addr 172.21.15.59:0/4153131966 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbacd92640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb900039e0 0x7fdba815c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdba7fff640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba80b83a0 0x7fdba80b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbad593640 1 -- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba80b83a0 msgr2=0x7fdba80b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbad593640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba80b83a0 0x7fdba80b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbad593640 1 -- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb900039e0 msgr2=0x7fdba815c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbad593640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb900039e0 0x7fdba815c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbad593640 1 -- 172.21.15.59:0/4153131966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba80b7300 con 0x7fdba80b6e10 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.341+0000 7fdbacd92640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb900039e0 0x7fdba815c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.342+0000 7fdbad593640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdba80b6e10 0x7fdba80b8c90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdba80cfd40 tx=0x7fdb98011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.342+0000 7fdba57fa640 1 -- 172.21.15.59:0/4153131966 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98019020 con 0x7fdba80b6e10 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.342+0000 7fdba57fa640 1 -- 172.21.15.59:0/4153131966 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb98004030 con 0x7fdba80b6e10 2024-03-06T07:23:58.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.342+0000 7fdbaeff6640 1 -- 172.21.15.59:0/4153131966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba80b75e0 con 0x7fdba80b6e10 2024-03-06T07:23:58.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.342+0000 7fdba57fa640 1 -- 172.21.15.59:0/4153131966 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb98004440 con 0x7fdba80b6e10 2024-03-06T07:23:58.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.342+0000 7fdbaeff6640 1 -- 172.21.15.59:0/4153131966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba80b7ba0 con 0x7fdba80b6e10 2024-03-06T07:23:58.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.343+0000 7fdbaeff6640 1 -- 172.21.15.59:0/4153131966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba804f330 con 0x7fdba80b6e10 2024-03-06T07:23:58.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.343+0000 7fdba57fa640 1 -- 172.21.15.59:0/4153131966 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb9801c020 con 0x7fdba80b6e10 2024-03-06T07:23:58.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.344+0000 7fdba57fa640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb9406b4d0 0x7fdb9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.344+0000 7fdba57fa640 1 -- 172.21.15.59:0/4153131966 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdb9808a390 con 0x7fdba80b6e10 2024-03-06T07:23:58.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.344+0000 7fdbacd92640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb9406b4d0 0x7fdb9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.344+0000 7fdbacd92640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb9406b4d0 0x7fdb9406d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdba0034000 tx=0x7fdba0009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.348+0000 7fdba57fa640 1 -- 172.21.15.59:0/4153131966 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb98023020 con 0x7fdba80b6e10 2024-03-06T07:23:58.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:58.600+0000 7fdbaeff6640 1 -- 172.21.15.59:0/4153131966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7fdba806c210 con 0x7fdba80b6e10 2024-03-06T07:23:59.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.596+0000 7fdba57fa640 1 -- 172.21.15.59:0/4153131966 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v137) v1 ==== 147+0+0 (secure 0 0 0) 0x7fdb980595a0 con 0x7fdba80b6e10 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 -- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb9406b4d0 msgr2=0x7fdb9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb9406b4d0 0x7fdb9406d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdba0034000 tx=0x7fdba0009030 comp rx=0 tx=0).stop 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 -- 172.21.15.59:0/4153131966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdba80b6e10 msgr2=0x7fdba80b8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdba80b6e10 0x7fdba80b8c90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdba80cfd40 tx=0x7fdb98011cf0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 -- 172.21.15.59:0/4153131966 shutdown_connections 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb9406b4d0 0x7fdb9406d990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdba80b6e10 0x7fdba80b8c90 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba80b83a0 0x7fdba80b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 --2- 172.21.15.59:0/4153131966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb900039e0 0x7fdba815c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 -- 172.21.15.59:0/4153131966 >> 172.21.15.59:0/4153131966 conn(0x7fdba80d3cd0 msgr2=0x7fdba80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 -- 172.21.15.59:0/4153131966 shutdown_connections 2024-03-06T07:23:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.599+0000 7fdbaeff6640 1 -- 172.21.15.59:0/4153131966 wait complete. 2024-03-06T07:23:59.614 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:23:59.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.723+0000 7fb617081640 1 Processor -- start 2024-03-06T07:23:59.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.723+0000 7fb617081640 1 -- start start 2024-03-06T07:23:59.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb617081640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb61015a220 0x7fb6100cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb617081640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6100b3320 con 0x7fb61006da00 2024-03-06T07:23:59.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb617081640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61015ec80 con 0x7fb6100712d0 2024-03-06T07:23:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb617081640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6100cf610 con 0x7fb61015a220 2024-03-06T07:23:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb61561e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb61006da00 0x7fb61015e570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38386/0 (socket says 172.21.15.59:38386) 2024-03-06T07:23:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb61561e640 1 -- 172.21.15.59:0/2448608504 learned_addr learned my addr 172.21.15.59:0/2448608504 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:23:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb614e1d640 1 --2- 172.21.15.59:0/2448608504 >> v2:172.21.15.164:3300/0 conn(0x7fb61015a220 0x7fb6100cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234379327 0 0) 0x7fb6100b3320 con 0x7fb61006da00 2024-03-06T07:23:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5f8003520 con 0x7fb61006da00 2024-03-06T07:23:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 588686797 0 0) 0x7fb5f8003520 con 0x7fb61006da00 2024-03-06T07:23:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 >> v2:172.21.15.164:3300/0 conn(0x7fb61015a220 msgr2=0x7fb6100cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 --2- 172.21.15.59:0/2448608504 >> v2:172.21.15.164:3300/0 conn(0x7fb61015a220 0x7fb6100cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 >> v1:172.21.15.164:6789/0 conn(0x7fb6100712d0 legacy=0x7fb61015de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6100728d0 con 0x7fb61006da00 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fb6080019f0 con 0x7fb61006da00 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 >> v1:172.21.15.59:6789/0 conn(0x7fb61006da00 legacy=0x7fb61015e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 --2- 172.21.15.59:0/2448608504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5f80039b0 0x7fb5f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 --2- 172.21.15.59:0/2448608504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5f8004800 0x7fb5f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 --2- 172.21.15.59:0/2448608504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5f80051c0 0x7fb5f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6100cf610 con 0x7fb5f8004800 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f800c160 con 0x7fb5f80039b0 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.724+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f800c360 con 0x7fb5f80051c0 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.725+0000 7fb607fff640 1 --2- 172.21.15.59:0/2448608504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5f8004800 0x7fb5f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.725+0000 7fb614e1d640 1 --2- 172.21.15.59:0/2448608504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5f80039b0 0x7fb5f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.725+0000 7fb61561e640 1 --2- 172.21.15.59:0/2448608504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5f80051c0 0x7fb5f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.725+0000 7fb607fff640 1 -- 172.21.15.59:0/2448608504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5f80051c0 msgr2=0x7fb5f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.725+0000 7fb607fff640 1 --2- 172.21.15.59:0/2448608504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5f80051c0 0x7fb5f800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.725+0000 7fb607fff640 1 -- 172.21.15.59:0/2448608504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5f80039b0 msgr2=0x7fb5f8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.725+0000 7fb607fff640 1 --2- 172.21.15.59:0/2448608504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5f80039b0 0x7fb5f8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.725+0000 7fb607fff640 1 -- 172.21.15.59:0/2448608504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f8003520 con 0x7fb5f8004800 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.725+0000 7fb61561e640 1 -- 172.21.15.59:0/2448608504 reap_dead start 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.726+0000 7fb607fff640 1 --2- 172.21.15.59:0/2448608504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5f8004800 0x7fb5f8004c80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb600001330 tx=0x7fb5f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4acaa1394f4f13f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.726+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb600001370 con 0x7fb5f8004800 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.726+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6000052d0 con 0x7fb5f8004800 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.726+0000 7fb6077fe640 1 -- 172.21.15.59:0/2448608504 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb600003830 con 0x7fb5f8004800 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.726+0000 7fb617081640 1 -- 172.21.15.59:0/2448608504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5f8004800 msgr2=0x7fb5f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.726+0000 7fb617081640 1 --2- 172.21.15.59:0/2448608504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5f8004800 0x7fb5f8004c80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb600001330 tx=0x7fb5f8005830 comp rx=0 tx=0).stop 2024-03-06T07:23:59.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.726+0000 7fb617081640 1 -- 172.21.15.59:0/2448608504 shutdown_connections 2024-03-06T07:23:59.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.726+0000 7fb617081640 1 --2- 172.21.15.59:0/2448608504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5f8004800 0x7fb5f8004c80 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.726+0000 7fb617081640 1 -- 172.21.15.59:0/2448608504 >> 172.21.15.59:0/2448608504 conn(0x7fb61006b190 msgr2=0x7fb61006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.726+0000 7fb617081640 1 -- 172.21.15.59:0/2448608504 shutdown_connections 2024-03-06T07:23:59.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.726+0000 7fb617081640 1 -- 172.21.15.59:0/2448608504 wait complete. 2024-03-06T07:23:59.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.727+0000 7fb617081640 1 Processor -- start 2024-03-06T07:23:59.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.727+0000 7fb617081640 1 -- start start 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.728+0000 7fb617081640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5f8004800 0x7fb6101486e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.728+0000 7fb617081640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6101547e0 0x7fb610154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.728+0000 7fb617081640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb610152cf0 0x7fb610153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.728+0000 7fb617081640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb610072030 con 0x7fb6101547e0 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.728+0000 7fb617081640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb610072bb0 con 0x7fb5f8004800 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.728+0000 7fb617081640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb610072530 con 0x7fb610152cf0 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.728+0000 7fb607fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6101547e0 0x7fb610154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.728+0000 7fb607fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6101547e0 0x7fb610154c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57990/0 (socket says 172.21.15.59:57990) 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.728+0000 7fb607fff640 1 -- 172.21.15.59:0/2229872046 learned_addr learned my addr 172.21.15.59:0/2229872046 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.729+0000 7fb607fff640 1 -- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb610152cf0 msgr2=0x7fb610153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.729+0000 7fb607fff640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb610152cf0 0x7fb610153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.729+0000 7fb607fff640 1 -- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5f8004800 msgr2=0x7fb6101486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.729+0000 7fb607fff640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5f8004800 0x7fb6101486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.729+0000 7fb607fff640 1 -- 172.21.15.59:0/2229872046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb610145b20 con 0x7fb6101547e0 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.729+0000 7fb607fff640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6101547e0 0x7fb610154c60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fb600001840 tx=0x7fb600001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.729+0000 7fb6057fa640 1 -- 172.21.15.59:0/2229872046 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb600042070 con 0x7fb6101547e0 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.729+0000 7fb6057fa640 1 -- 172.21.15.59:0/2229872046 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb600005480 con 0x7fb6101547e0 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.729+0000 7fb617081640 1 -- 172.21.15.59:0/2229872046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb610153a20 con 0x7fb6101547e0 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.729+0000 7fb6057fa640 1 -- 172.21.15.59:0/2229872046 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb600007a30 con 0x7fb6101547e0 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.729+0000 7fb617081640 1 -- 172.21.15.59:0/2229872046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb610153fe0 con 0x7fb6101547e0 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.730+0000 7fb617081640 1 -- 172.21.15.59:0/2229872046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6100732b0 con 0x7fb6101547e0 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.730+0000 7fb6057fa640 1 -- 172.21.15.59:0/2229872046 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb600045070 con 0x7fb6101547e0 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.731+0000 7fb6057fa640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb5fc06b500 0x7fb5fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.731+0000 7fb6057fa640 1 -- 172.21.15.59:0/2229872046 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb6000ada10 con 0x7fb6101547e0 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.731+0000 7fb614e1d640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb5fc06b500 0x7fb5fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.732+0000 7fb614e1d640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb5fc06b500 0x7fb5fc06d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb60c006e80 tx=0x7fb60c006df0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.735+0000 7fb6057fa640 1 -- 172.21.15.59:0/2229872046 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb60007cb90 con 0x7fb6101547e0 2024-03-06T07:23:59.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.916+0000 7fb617081640 1 -- 172.21.15.59:0/2229872046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb6100cc060 con 0x7fb6101547e0 2024-03-06T07:23:59.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.919+0000 7fb6057fa640 1 -- 172.21.15.59:0/2229872046 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb60007c490 con 0x7fb6101547e0 2024-03-06T07:23:59.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 -- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb5fc06b500 msgr2=0x7fb5fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb5fc06b500 0x7fb5fc06d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb60c006e80 tx=0x7fb60c006df0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 -- 172.21.15.59:0/2229872046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6101547e0 msgr2=0x7fb610154c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6101547e0 0x7fb610154c60 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fb600001840 tx=0x7fb600001120 comp rx=0 tx=0).stop 2024-03-06T07:23:59.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 -- 172.21.15.59:0/2229872046 shutdown_connections 2024-03-06T07:23:59.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb5fc06b500 0x7fb5fc06d9c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb610152cf0 0x7fb610153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6101547e0 0x7fb610154c60 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 --2- 172.21.15.59:0/2229872046 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5f8004800 0x7fb6101486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 -- 172.21.15.59:0/2229872046 >> 172.21.15.59:0/2229872046 conn(0x7fb61006b190 msgr2=0x7fb6100cc300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 -- 172.21.15.59:0/2229872046 shutdown_connections 2024-03-06T07:23:59.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:23:59.921+0000 7fb617081640 1 -- 172.21.15.59:0/2229872046 wait complete. 2024-03-06T07:23:59.934 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:59.934 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:59.989 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:00.042 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:00.043 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:24:00.043 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:00.057 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:00.058 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:24:00.058 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:00.075 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:00.075 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:24:00.075 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:00.112 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:00.112 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:24:00.112 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:00.132 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:00.132 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:24:00.133 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:00.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.252+0000 7fec78688640 1 Processor -- start 2024-03-06T07:24:00.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.252+0000 7fec78688640 1 -- start start 2024-03-06T07:24:00.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.253+0000 7fec78688640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fec701621c0 0x7fec700cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.253+0000 7fec78688640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec701625a0 0x7fec7015dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.253+0000 7fec78688640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec700ceac0 con 0x7fec700d4920 2024-03-06T07:24:00.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.253+0000 7fec78688640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec700cf690 con 0x7fec701621c0 2024-03-06T07:24:00.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.253+0000 7fec78688640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec700d2dc0 con 0x7fec701625a0 2024-03-06T07:24:00.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.253+0000 7fec76c25640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec701625a0 0x7fec7015dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.253+0000 7fec76424640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fec701621c0 0x7fec700cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.253+0000 7fec75c23640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fec700d4920 0x7fec700d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:51466/0 (socket says 172.21.15.59:51466) 2024-03-06T07:24:00.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.253+0000 7fec76424640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fec701621c0 0x7fec700cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:57644/0 (socket says 172.21.15.59:57644) 2024-03-06T07:24:00.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.253+0000 7fec75c23640 1 -- 172.21.15.59:0/3464012691 learned_addr learned my addr 172.21.15.59:0/3464012691 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:00.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.254+0000 7fec76424640 1 -- 172.21.15.59:0/3464012691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec701625a0 msgr2=0x7fec7015dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.254+0000 7fec76424640 1 --2- 172.21.15.59:0/3464012691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec701625a0 0x7fec7015dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.254+0000 7fec76424640 1 -- 172.21.15.59:0/3464012691 >> v1:172.21.15.164:6789/0 conn(0x7fec700d4920 legacy=0x7fec700d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.254+0000 7fec75422640 1 -- 172.21.15.59:0/3464012691 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037003262 0 0) 0x7fec700ceac0 con 0x7fec700d4920 2024-03-06T07:24:00.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.254+0000 7fec76424640 1 -- 172.21.15.59:0/3464012691 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700ced60 con 0x7fec701621c0 2024-03-06T07:24:00.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.254+0000 7fec76c25640 1 --2- 172.21.15.59:0/3464012691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec701625a0 0x7fec7015dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:00.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec76424640 1 --2- 172.21.15.59:0/3464012691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec701621c0 0x7fec700cfb20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fec68009d90 tx=0x7fec6802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c96ccc68e6175ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec75422640 1 -- 172.21.15.59:0/3464012691 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68039070 con 0x7fec701621c0 2024-03-06T07:24:00.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec75422640 1 -- 172.21.15.59:0/3464012691 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6802cc50 con 0x7fec701621c0 2024-03-06T07:24:00.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec75422640 1 -- 172.21.15.59:0/3464012691 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68034440 con 0x7fec701621c0 2024-03-06T07:24:00.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec78688640 1 -- 172.21.15.59:0/3464012691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec701621c0 msgr2=0x7fec700cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec78688640 1 --2- 172.21.15.59:0/3464012691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec701621c0 0x7fec700cfb20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fec68009d90 tx=0x7fec6802bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec78688640 1 -- 172.21.15.59:0/3464012691 shutdown_connections 2024-03-06T07:24:00.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec78688640 1 --2- 172.21.15.59:0/3464012691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec701625a0 0x7fec7015dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec78688640 1 --2- 172.21.15.59:0/3464012691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec701621c0 0x7fec700cfb20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec78688640 1 -- 172.21.15.59:0/3464012691 >> 172.21.15.59:0/3464012691 conn(0x7fec700d3d10 msgr2=0x7fec700d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec78688640 1 -- 172.21.15.59:0/3464012691 shutdown_connections 2024-03-06T07:24:00.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.255+0000 7fec78688640 1 -- 172.21.15.59:0/3464012691 wait complete. 2024-03-06T07:24:00.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.256+0000 7fec78688640 1 Processor -- start 2024-03-06T07:24:00.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.256+0000 7fec78688640 1 -- start start 2024-03-06T07:24:00.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.257+0000 7fec78688640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec700d4920 0x7fec70143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.257+0000 7fec78688640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec701621c0 0x7fec700b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.257+0000 7fec78688640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec701625a0 0x7fec700cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.257+0000 7fec78688640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7015e280 con 0x7fec701621c0 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.257+0000 7fec78688640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec700cf320 con 0x7fec700d4920 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.257+0000 7fec78688640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7015e400 con 0x7fec701625a0 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.257+0000 7fec75c23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec701621c0 0x7fec700b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.257+0000 7fec75c23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec701621c0 0x7fec700b6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58004/0 (socket says 172.21.15.59:58004) 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.257+0000 7fec75c23640 1 -- 172.21.15.59:0/437921554 learned_addr learned my addr 172.21.15.59:0/437921554 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.257+0000 7fec76424640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec700d4920 0x7fec70143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.257+0000 7fec76c25640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec701625a0 0x7fec700cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.258+0000 7fec75c23640 1 -- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec701625a0 msgr2=0x7fec700cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.258+0000 7fec75c23640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec701625a0 0x7fec700cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.258+0000 7fec75c23640 1 -- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec700d4920 msgr2=0x7fec70143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.258+0000 7fec75c23640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec700d4920 0x7fec70143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.258+0000 7fec75c23640 1 -- 172.21.15.59:0/437921554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec70144370 con 0x7fec701621c0 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.258+0000 7fec75c23640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec701621c0 0x7fec700b6f90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fec6400c4d0 tx=0x7fec6400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.258+0000 7fec6effd640 1 -- 172.21.15.59:0/437921554 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec640074e0 con 0x7fec701621c0 2024-03-06T07:24:00.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.258+0000 7fec6effd640 1 -- 172.21.15.59:0/437921554 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec640047e0 con 0x7fec701621c0 2024-03-06T07:24:00.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.258+0000 7fec78688640 1 -- 172.21.15.59:0/437921554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec70146350 con 0x7fec701621c0 2024-03-06T07:24:00.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.258+0000 7fec6effd640 1 -- 172.21.15.59:0/437921554 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64004bd0 con 0x7fec701621c0 2024-03-06T07:24:00.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.258+0000 7fec78688640 1 -- 172.21.15.59:0/437921554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec70146910 con 0x7fec701621c0 2024-03-06T07:24:00.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.259+0000 7fec78688640 1 -- 172.21.15.59:0/437921554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7004f330 con 0x7fec701621c0 2024-03-06T07:24:00.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.259+0000 7fec6effd640 1 -- 172.21.15.59:0/437921554 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec64011070 con 0x7fec701621c0 2024-03-06T07:24:00.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.260+0000 7fec6effd640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fec5c06b4d0 0x7fec5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.260+0000 7fec6effd640 1 -- 172.21.15.59:0/437921554 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fec64089a40 con 0x7fec701621c0 2024-03-06T07:24:00.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.260+0000 7fec76424640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fec5c06b4d0 0x7fec5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.261+0000 7fec76424640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fec5c06b4d0 0x7fec5c06d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fec68036000 tx=0x7fec68030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.264+0000 7fec6effd640 1 -- 172.21.15.59:0/437921554 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec64058bc0 con 0x7fec701621c0 2024-03-06T07:24:00.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.486+0000 7fec78688640 1 -- 172.21.15.59:0/437921554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec700d2640 con 0x7fec701621c0 2024-03-06T07:24:00.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.487+0000 7fec6effd640 1 -- 172.21.15.59:0/437921554 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+3702 (secure 0 0 0) 0x7fec64089e20 con 0x7fec701621c0 2024-03-06T07:24:00.489 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:00.489 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":22,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1492280032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1492280032},{"type":"v1","addr":"172.21.15.164:6837","nonce":1492280032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5557,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/690078434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":690078434},{"type":"v1","addr":"172.21.15.164:6836","nonce":690078434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5682,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1345698247","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1345698247},{"type":"v1","addr":"172.21.15.59:6833","nonce":1345698247}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16},{"gid":5685,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3875776524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3875776524},{"type":"v1","addr":"172.21.15.59:6835","nonce":3875776524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[]} 2024-03-06T07:24:00.489 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 22 2024-03-06T07:24:00.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 -- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fec5c06b4d0 msgr2=0x7fec5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fec5c06b4d0 0x7fec5c06d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fec68036000 tx=0x7fec68030000 comp rx=0 tx=0).stop 2024-03-06T07:24:00.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 -- 172.21.15.59:0/437921554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec701621c0 msgr2=0x7fec700b6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec701621c0 0x7fec700b6f90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fec6400c4d0 tx=0x7fec6400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 -- 172.21.15.59:0/437921554 shutdown_connections 2024-03-06T07:24:00.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fec5c06b4d0 0x7fec5c06d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec701625a0 0x7fec700cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec701621c0 0x7fec700b6f90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 --2- 172.21.15.59:0/437921554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec700d4920 0x7fec70143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 -- 172.21.15.59:0/437921554 >> 172.21.15.59:0/437921554 conn(0x7fec700d3d10 msgr2=0x7fec700d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 -- 172.21.15.59:0/437921554 shutdown_connections 2024-03-06T07:24:00.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:00.489+0000 7fec78688640 1 -- 172.21.15.59:0/437921554 wait complete. 2024-03-06T07:24:00.502 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:24:00.502 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:24:00.502 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:24:00.502 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:00.502 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:24:00.502 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:24:00.502 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:24:00.503 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:00.503 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:24:00.503 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:24:00.503 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:24:00.503 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:00.503 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:24:00.503 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:24:00.503 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:24:00.504 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:00.512 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:24:00.512 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:24:00.515 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:24:00.515 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:24:00.516 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:24:00.516 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:24:00.555 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:24:00.555 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:00.558 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:24:00.558 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:00.559 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:24:00.559 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:00.631 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:24:00.642 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:24:00.642 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:24:01.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-06T07:24:06.512 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:24:06.513 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:24:06.516 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:24:06.516 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:06.516 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:24:06.603 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:24:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.630+0000 7f996dcd4640 1 Processor -- start 2024-03-06T07:24:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.630+0000 7f996dcd4640 1 -- start start 2024-03-06T07:24:06.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.631+0000 7f996dcd4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f996806e7b0 0x7f99680cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.631+0000 7f996dcd4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99680b4450 con 0x7f996806ee10 2024-03-06T07:24:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.631+0000 7f996dcd4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99680b5b60 con 0x7f9968144030 2024-03-06T07:24:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.631+0000 7f996dcd4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99680b4a60 con 0x7f996806e7b0 2024-03-06T07:24:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.631+0000 7f9967fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f996806e7b0 0x7f99680cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.631+0000 7f99677fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f996806ee10 0x7f996806f1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38414/0 (socket says 172.21.15.59:38414) 2024-03-06T07:24:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.631+0000 7f99677fe640 1 -- 172.21.15.59:0/3738061351 learned_addr learned my addr 172.21.15.59:0/3738061351 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.632+0000 7f9967fff640 1 -- 172.21.15.59:0/3738061351 >> v1:172.21.15.164:6789/0 conn(0x7f9968144030 legacy=0x7f99680b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.632+0000 7f9967fff640 1 -- 172.21.15.59:0/3738061351 >> v1:172.21.15.59:6789/0 conn(0x7f996806ee10 legacy=0x7f996806f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.632+0000 7f99667fc640 1 -- 172.21.15.59:0/3738061351 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631509363 0 0) 0x7f99680b4450 con 0x7f996806ee10 2024-03-06T07:24:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.632+0000 7f9967fff640 1 -- 172.21.15.59:0/3738061351 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99680b5e40 con 0x7f996806e7b0 2024-03-06T07:24:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.632+0000 7f9967fff640 1 --2- 172.21.15.59:0/3738061351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f996806e7b0 0x7f99680cdde0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9960009d90 tx=0x7f996002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75887985cfa4738a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.633+0000 7f99667fc640 1 -- 172.21.15.59:0/3738061351 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960039070 con 0x7f996806e7b0 2024-03-06T07:24:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.633+0000 7f99667fc640 1 -- 172.21.15.59:0/3738061351 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996002cd00 con 0x7f996806e7b0 2024-03-06T07:24:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.633+0000 7f99667fc640 1 -- 172.21.15.59:0/3738061351 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960034660 con 0x7f996806e7b0 2024-03-06T07:24:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.633+0000 7f996dcd4640 1 -- 172.21.15.59:0/3738061351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f996806e7b0 msgr2=0x7f99680cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.633+0000 7f996dcd4640 1 --2- 172.21.15.59:0/3738061351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f996806e7b0 0x7f99680cdde0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9960009d90 tx=0x7f996002bf50 comp rx=0 tx=0).stop 2024-03-06T07:24:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.633+0000 7f996dcd4640 1 -- 172.21.15.59:0/3738061351 shutdown_connections 2024-03-06T07:24:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.633+0000 7f996dcd4640 1 --2- 172.21.15.59:0/3738061351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f996806e7b0 0x7f99680cdde0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.633+0000 7f996dcd4640 1 -- 172.21.15.59:0/3738061351 >> 172.21.15.59:0/3738061351 conn(0x7f99680d0a10 msgr2=0x7f99680d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.633+0000 7f996dcd4640 1 -- 172.21.15.59:0/3738061351 shutdown_connections 2024-03-06T07:24:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.633+0000 7f996dcd4640 1 -- 172.21.15.59:0/3738061351 wait complete. 2024-03-06T07:24:06.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.634+0000 7f996dcd4640 1 Processor -- start 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.635+0000 7f996dcd4640 1 -- start start 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.635+0000 7f996dcd4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f996806e7b0 0x7f99681d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.635+0000 7f996dcd4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f996806ee10 0x7f99681e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.635+0000 7f996dcd4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9968144030 0x7f99681e3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.635+0000 7f996dcd4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9968143c40 con 0x7f996806e7b0 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.635+0000 7f996dcd4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9968144840 con 0x7f9968144030 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.635+0000 7f996dcd4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9968143dc0 con 0x7f996806ee10 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.635+0000 7f99677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f996806e7b0 0x7f99681d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f99677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f996806e7b0 0x7f99681d1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37846/0 (socket says 172.21.15.59:37846) 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f99677fe640 1 -- 172.21.15.59:0/4267765391 learned_addr learned my addr 172.21.15.59:0/4267765391 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f99677fe640 1 -- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f996806ee10 msgr2=0x7f99681e84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f9967fff640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9968144030 0x7f99681e3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f9966ffd640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f996806ee10 0x7f99681e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f99677fe640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f996806ee10 0x7f99681e84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f99677fe640 1 -- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9968144030 msgr2=0x7f99681e3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f99677fe640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9968144030 0x7f99681e3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f99677fe640 1 -- 172.21.15.59:0/4267765391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99681d23a0 con 0x7f996806e7b0 2024-03-06T07:24:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f9967fff640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9968144030 0x7f99681e3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f99677fe640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f996806e7b0 0x7f99681d1d80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f995800c4a0 tx=0x7f995800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f994ffff640 1 -- 172.21.15.59:0/4267765391 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958018070 con 0x7f996806e7b0 2024-03-06T07:24:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f994ffff640 1 -- 172.21.15.59:0/4267765391 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99580074e0 con 0x7f996806e7b0 2024-03-06T07:24:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f996dcd4640 1 -- 172.21.15.59:0/4267765391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99681d2680 con 0x7f996806e7b0 2024-03-06T07:24:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f994ffff640 1 -- 172.21.15.59:0/4267765391 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958004a20 con 0x7f996806e7b0 2024-03-06T07:24:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.636+0000 7f996dcd4640 1 -- 172.21.15.59:0/4267765391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99681d2c40 con 0x7f996806e7b0 2024-03-06T07:24:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.637+0000 7f996dcd4640 1 -- 172.21.15.59:0/4267765391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996804f330 con 0x7f996806e7b0 2024-03-06T07:24:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.638+0000 7f994ffff640 1 -- 172.21.15.59:0/4267765391 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f995801b070 con 0x7f996806e7b0 2024-03-06T07:24:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.638+0000 7f994ffff640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f995406b500 0x7f995406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.638+0000 7f994ffff640 1 -- 172.21.15.59:0/4267765391 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9958089290 con 0x7f996806e7b0 2024-03-06T07:24:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.638+0000 7f9966ffd640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f995406b500 0x7f995406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.639+0000 7f9966ffd640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f995406b500 0x7f995406d9c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f99681e3b30 tx=0x7f995c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:06.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.643+0000 7f994ffff640 1 -- 172.21.15.59:0/4267765391 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9958058380 con 0x7f996806e7b0 2024-03-06T07:24:06.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:06.839+0000 7f996dcd4640 1 -- 172.21.15.59:0/4267765391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f99681e8be0 con 0x7f996806e7b0 2024-03-06T07:24:07.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.630+0000 7f994ffff640 1 -- 172.21.15.59:0/4267765391 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v55) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9958057c80 con 0x7f996806e7b0 2024-03-06T07:24:07.632 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-03-06T07:24:07.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 -- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f995406b500 msgr2=0x7f995406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f995406b500 0x7f995406d9c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f99681e3b30 tx=0x7f995c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:07.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 -- 172.21.15.59:0/4267765391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f996806e7b0 msgr2=0x7f99681d1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f996806e7b0 0x7f99681d1d80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f995800c4a0 tx=0x7f995800c960 comp rx=0 tx=0).stop 2024-03-06T07:24:07.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 -- 172.21.15.59:0/4267765391 shutdown_connections 2024-03-06T07:24:07.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f995406b500 0x7f995406d9c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9968144030 0x7f99681e3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f996806ee10 0x7f99681e84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 --2- 172.21.15.59:0/4267765391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f996806e7b0 0x7f99681d1d80 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 -- 172.21.15.59:0/4267765391 >> 172.21.15.59:0/4267765391 conn(0x7f99680d0a10 msgr2=0x7f996806f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 -- 172.21.15.59:0/4267765391 shutdown_connections 2024-03-06T07:24:07.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.633+0000 7f996dcd4640 1 -- 172.21.15.59:0/4267765391 wait complete. 2024-03-06T07:24:07.649 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:24:07.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.755+0000 7f39bfef4640 1 Processor -- start 2024-03-06T07:24:07.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.755+0000 7f39bfef4640 1 -- start start 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.756+0000 7f39bfef4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f39b80b3060 0x7f39b815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.756+0000 7f39bfef4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b815f580 con 0x7f39b813dc10 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.756+0000 7f39bfef4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b815b450 con 0x7f39b815bf30 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.756+0000 7f39bfef4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80cf3f0 con 0x7f39b80b3060 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.756+0000 7f39bdc90640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f39b813dc10 0x7f39b80b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38418/0 (socket says 172.21.15.59:38418) 2024-03-06T07:24:07.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.756+0000 7f39bdc90640 1 -- 172.21.15.59:0/1922117166 learned_addr learned my addr 172.21.15.59:0/1922117166 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.756+0000 7f39bd48f640 1 --2- 172.21.15.59:0/1922117166 >> v2:172.21.15.164:3300/0 conn(0x7f39b80b3060 0x7f39b815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.756+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954880564 0 0) 0x7f39b815f580 con 0x7f39b813dc10 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.756+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a4003520 con 0x7f39b813dc10 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132503136 0 0) 0x7f39b815b450 con 0x7f39b815bf30 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b815f580 con 0x7f39b815bf30 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1578868908 0 0) 0x7f39a4003520 con 0x7f39b813dc10 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 >> v2:172.21.15.164:3300/0 conn(0x7f39b80b3060 msgr2=0x7f39b815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 --2- 172.21.15.59:0/1922117166 >> v2:172.21.15.164:3300/0 conn(0x7f39b80b3060 0x7f39b815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 >> v1:172.21.15.164:6789/0 conn(0x7f39b815bf30 legacy=0x7f39b815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b815b730 con 0x7f39b813dc10 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f39b0001670 con 0x7f39b813dc10 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 >> v1:172.21.15.59:6789/0 conn(0x7f39b813dc10 legacy=0x7f39b80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 --2- 172.21.15.59:0/1922117166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39a4003b40 0x7f39a4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 --2- 172.21.15.59:0/1922117166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39a4004800 0x7f39a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 --2- 172.21.15.59:0/1922117166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 0x7f39a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80cf3f0 con 0x7f39a40051c0 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a400c0f0 con 0x7f39a4003b40 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a400c2f0 con 0x7f39a4004800 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39be491640 1 --2- 172.21.15.59:0/1922117166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 0x7f39a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bd48f640 1 --2- 172.21.15.59:0/1922117166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39a4004800 0x7f39a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39bdc90640 1 --2- 172.21.15.59:0/1922117166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39a4003b40 0x7f39a4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39be491640 1 -- 172.21.15.59:0/1922117166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39a4004800 msgr2=0x7f39a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39be491640 1 --2- 172.21.15.59:0/1922117166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39a4004800 0x7f39a4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39be491640 1 -- 172.21.15.59:0/1922117166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39a4003b40 msgr2=0x7f39a4003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39be491640 1 --2- 172.21.15.59:0/1922117166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39a4003b40 0x7f39a4003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.757+0000 7f39be491640 1 -- 172.21.15.59:0/1922117166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a4003520 con 0x7f39a40051c0 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.758+0000 7f39be491640 1 -- 172.21.15.59:0/1922117166 reap_dead start 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.758+0000 7f39be491640 1 --2- 172.21.15.59:0/1922117166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 0x7f39a400b9c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f39b4001fd0 tx=0x7f39a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e7731ed8d391ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.758+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4001bf0 con 0x7f39a40051c0 2024-03-06T07:24:07.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.758+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b4003b70 con 0x7f39a40051c0 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.758+0000 7f39bcc8e640 1 -- 172.21.15.59:0/1922117166 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b40050d0 con 0x7f39a40051c0 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.759+0000 7f39bfef4640 1 -- 172.21.15.59:0/1922117166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 msgr2=0x7f39a400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.759+0000 7f39bfef4640 1 --2- 172.21.15.59:0/1922117166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 0x7f39a400b9c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f39b4001fd0 tx=0x7f39a4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.759+0000 7f39bfef4640 1 -- 172.21.15.59:0/1922117166 shutdown_connections 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.759+0000 7f39bfef4640 1 --2- 172.21.15.59:0/1922117166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 0x7f39a400b9c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.759+0000 7f39bfef4640 1 -- 172.21.15.59:0/1922117166 >> 172.21.15.59:0/1922117166 conn(0x7f39b80cf8a0 msgr2=0x7f39b80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.759+0000 7f39bfef4640 1 -- 172.21.15.59:0/1922117166 shutdown_connections 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.759+0000 7f39bfef4640 1 -- 172.21.15.59:0/1922117166 wait complete. 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.760+0000 7f39bfef4640 1 Processor -- start 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.760+0000 7f39bfef4640 1 -- start start 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bfef4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 0x7f39b8143f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bfef4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39b80ae270 0x7f39b81f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bfef4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39b81f4ad0 0x7f39b8071e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bfef4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80b2950 con 0x7f39a40051c0 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bfef4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b815b210 con 0x7f39b81f4ad0 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bfef4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80b2ad0 con 0x7f39b80ae270 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bdc90640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 0x7f39b8143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bdc90640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 0x7f39b8143f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37858/0 (socket says 172.21.15.59:37858) 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bdc90640 1 -- 172.21.15.59:0/959481534 learned_addr learned my addr 172.21.15.59:0/959481534 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bd48f640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39b80ae270 0x7f39b81f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39be491640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39b81f4ad0 0x7f39b8071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bdc90640 1 -- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39b80ae270 msgr2=0x7f39b81f0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bdc90640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39b80ae270 0x7f39b81f0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bdc90640 1 -- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39b81f4ad0 msgr2=0x7f39b8071e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bdc90640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39b81f4ad0 0x7f39b8071e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bdc90640 1 -- 172.21.15.59:0/959481534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80ae7c0 con 0x7f39a40051c0 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39be491640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39b81f4ad0 0x7f39b8071e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:07.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bdc90640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 0x7f39b8143f60 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f39b0004f70 tx=0x7f39b00074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39ae7fc640 1 -- 172.21.15.59:0/959481534 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b0001080 con 0x7f39a40051c0 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39ae7fc640 1 -- 172.21.15.59:0/959481534 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b0018070 con 0x7f39a40051c0 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bfef4640 1 -- 172.21.15.59:0/959481534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b80aeaa0 con 0x7f39a40051c0 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39ae7fc640 1 -- 172.21.15.59:0/959481534 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b000f970 con 0x7f39a40051c0 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.761+0000 7f39bfef4640 1 -- 172.21.15.59:0/959481534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b8141a60 con 0x7f39a40051c0 2024-03-06T07:24:07.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.763+0000 7f39bfef4640 1 -- 172.21.15.59:0/959481534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b804f330 con 0x7f39a40051c0 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.763+0000 7f39ae7fc640 1 -- 172.21.15.59:0/959481534 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39b000fb10 con 0x7f39a40051c0 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.763+0000 7f39ae7fc640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f399c06b670 0x7f399c06db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.763+0000 7f39ae7fc640 1 -- 172.21.15.59:0/959481534 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f39b0089df0 con 0x7f39a40051c0 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.764+0000 7f39bd48f640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f399c06b670 0x7f399c06db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.764+0000 7f39bd48f640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f399c06b670 0x7f399c06db30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f39b81414a0 tx=0x7f39a8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:07.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:07.768+0000 7f39ae7fc640 1 -- 172.21.15.59:0/959481534 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39b00112b0 con 0x7f39a40051c0 2024-03-06T07:24:07.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-06T07:24:08.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.019+0000 7f39bfef4640 1 -- 172.21.15.59:0/959481534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f39b8141c30 con 0x7f39a40051c0 2024-03-06T07:24:08.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.020+0000 7f39ae7fc640 1 -- 172.21.15.59:0/959481534 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f39b0059050 con 0x7f39a40051c0 2024-03-06T07:24:08.022 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:08.022 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:24:08.022 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-03-06T07:24:08.022 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.022 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-03-06T07:24:08.022 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlODBZGBAAqV+TH2t2ZKyV9l13QEn6ew==", 2024-03-06T07:24:08.022 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.022 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:24:08.022 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:24:08.022 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlR1MuJRAAp6OAQ9gZemWtwI1dh3H2SA==", 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.023 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlDXW2HxAAnKyuS1GW0wLO98/F9wdOAg==", 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehl5MO4LBAAKmt2dvKk7BtcCM4o7cSwgg==", 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:24:08.024 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBXGehlzdinFhAA0ACDAs9dtoY6EhUIveeIvA==", 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.025 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGehlH92eFRAANQceHGuQyrbI6gP1m/Gakw==", 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGehlMWpVFBAAbOG9nScbFlvKHXZMQJoQug==", 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:08.026 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGehlLCyWEhAAyfoW/FmU37QD2IohknBwWw==", 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.027 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBmGehldqv5GhAAWEloqqKgdpKPHjt7VKFodw==", 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGehlc0ENGRAA5V9Az+vxAR3OUwWpDP7w1A==", 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.028 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBsGehl5SYEFxAAPLyrHQV9SGguCV9f5VIkYg==", 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBvGehle3v0FBAA4xUPwX5couppHgi9VwSi2g==", 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.029 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA==", 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.030 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlqhlAOhAAQ4l3uI8MlcCH+NAIWo/UuA==", 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehltL/GNhAAZqSQ+UtwjI/PA95O1RGBNg==", 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.031 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGehlltx5BBAAFa/+/w8Hd8lZegfIWtpJOA==", 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.032 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBSGehlMExaNhAAQ2FD7afktMhrTbMMAK1epQ==", 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlLSTzFBAACeLdHsCX9KJGEsq44aqTyw==", 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.033 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlAgqXEBAACCHZ9Lp2lEyF93xD4Ns9Nw==", 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-06T07:24:08.034 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:24:08.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.021+0000 7f39bfef4640 1 -- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f399c06b670 msgr2=0x7f399c06db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.021+0000 7f39bfef4640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f399c06b670 0x7f399c06db30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f39b81414a0 tx=0x7f39a8008040 comp rx=0 tx=0).stop 2024-03-06T07:24:08.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.021+0000 7f39bfef4640 1 -- 172.21.15.59:0/959481534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 msgr2=0x7f39b8143f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.021+0000 7f39bfef4640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 0x7f39b8143f60 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f39b0004f70 tx=0x7f39b00074f0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.022+0000 7f39bfef4640 1 -- 172.21.15.59:0/959481534 shutdown_connections 2024-03-06T07:24:08.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.022+0000 7f39bfef4640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f399c06b670 0x7f399c06db30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.022+0000 7f39bfef4640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39b81f4ad0 0x7f39b8071e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.022+0000 7f39bfef4640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39b80ae270 0x7f39b81f0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.022+0000 7f39bfef4640 1 --2- 172.21.15.59:0/959481534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39a40051c0 0x7f39b8143f60 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.022+0000 7f39bfef4640 1 -- 172.21.15.59:0/959481534 >> 172.21.15.59:0/959481534 conn(0x7f39b80cf8a0 msgr2=0x7f39b8159210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.022+0000 7f39bfef4640 1 -- 172.21.15.59:0/959481534 shutdown_connections 2024-03-06T07:24:08.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.022+0000 7f39bfef4640 1 -- 172.21.15.59:0/959481534 wait complete. 2024-03-06T07:24:08.037 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:08.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.137+0000 7ff8f533a640 1 Processor -- start 2024-03-06T07:24:08.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.137+0000 7ff8f533a640 1 -- start start 2024-03-06T07:24:08.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8f533a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff8f00ce920 0x7ff8f015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8f533a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8f006b600 0x7ff8f0162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8f533a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f015ea30 con 0x7ff8f006b220 2024-03-06T07:24:08.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8f533a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f0161b70 con 0x7ff8f00ce920 2024-03-06T07:24:08.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8f533a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f015ed10 con 0x7ff8f006b600 2024-03-06T07:24:08.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8eeffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff8f006b220 0x7ff8f0071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38420/0 (socket says 172.21.15.59:38420) 2024-03-06T07:24:08.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8eeffd640 1 -- 172.21.15.59:0/61412794 learned_addr learned my addr 172.21.15.59:0/61412794 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:08.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8ee7fc640 1 --2- 172.21.15.59:0/61412794 >> v2:172.21.15.164:3300/0 conn(0x7ff8f00ce920 0x7ff8f015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8ef7fe640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8f006b600 0x7ff8f0162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233602442 0 0) 0x7ff8f015ea30 con 0x7ff8f006b220 2024-03-06T07:24:08.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc003520 con 0x7ff8f006b220 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4185573109 0 0) 0x7ff8dc003520 con 0x7ff8f006b220 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8f006b600 msgr2=0x7ff8f0162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8edffb640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8f006b600 0x7ff8f0162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 >> v2:172.21.15.164:3300/0 conn(0x7ff8f00ce920 msgr2=0x7ff8f015e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8edffb640 1 --2- 172.21.15.59:0/61412794 >> v2:172.21.15.164:3300/0 conn(0x7ff8f00ce920 0x7ff8f015e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f015eff0 con 0x7ff8f006b220 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7ff8e0001a10 con 0x7ff8f006b220 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.138+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 >> v1:172.21.15.59:6789/0 conn(0x7ff8f006b220 legacy=0x7ff8f0071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8edffb640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8f006b600 0x7ff8dc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8edffb640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8dc003f30 0x7ff8dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8edffb640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc004d40 0x7ff8dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0161b70 con 0x7ff8dc004d40 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f015ed10 con 0x7ff8dc003f30 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc00ba10 con 0x7ff8f006b600 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8ef7fe640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc004d40 0x7ff8dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8eeffd640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8f006b600 0x7ff8dc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8ee7fc640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8dc003f30 0x7ff8dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8ef7fe640 1 -- 172.21.15.59:0/61412794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8f006b600 msgr2=0x7ff8dc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8ef7fe640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8f006b600 0x7ff8dc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8ef7fe640 1 -- 172.21.15.59:0/61412794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8dc003f30 msgr2=0x7ff8dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8ef7fe640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8dc003f30 0x7ff8dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.139+0000 7ff8ef7fe640 1 -- 172.21.15.59:0/61412794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc003520 con 0x7ff8dc004d40 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.140+0000 7ff8ef7fe640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc004d40 0x7ff8dc0051c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009af0 tx=0x7ff8dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ce20cd9260038a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.140+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8039070 con 0x7ff8dc004d40 2024-03-06T07:24:08.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.140+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e802c6b0 con 0x7ff8dc004d40 2024-03-06T07:24:08.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.140+0000 7ff8edffb640 1 -- 172.21.15.59:0/61412794 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8009690 con 0x7ff8dc004d40 2024-03-06T07:24:08.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.140+0000 7ff8f533a640 1 -- 172.21.15.59:0/61412794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc004d40 msgr2=0x7ff8dc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.140+0000 7ff8f533a640 1 --2- 172.21.15.59:0/61412794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8dc004d40 0x7ff8dc0051c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009af0 tx=0x7ff8dc005830 comp rx=0 tx=0).stop 2024-03-06T07:24:08.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.140+0000 7ff8ef7fe640 1 -- 172.21.15.59:0/61412794 reap_dead start 2024-03-06T07:24:08.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.140+0000 7ff8f533a640 1 -- 172.21.15.59:0/61412794 shutdown_connections 2024-03-06T07:24:08.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.140+0000 7ff8f533a640 1 -- 172.21.15.59:0/61412794 >> 172.21.15.59:0/61412794 conn(0x7ff8f006c0c0 msgr2=0x7ff8f006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.140+0000 7ff8f533a640 1 -- 172.21.15.59:0/61412794 shutdown_connections 2024-03-06T07:24:08.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.140+0000 7ff8f533a640 1 -- 172.21.15.59:0/61412794 wait complete. 2024-03-06T07:24:08.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.142+0000 7ff8f533a640 1 Processor -- start 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.142+0000 7ff8f533a640 1 -- start start 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.142+0000 7ff8f533a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8dc004d40 0x7ff8f00b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.142+0000 7ff8f533a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8f014bb30 0x7ff8f00adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.142+0000 7ff8f533a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8f014afa0 0x7ff8f014b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.142+0000 7ff8f533a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0072090 con 0x7ff8f014bb30 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.142+0000 7ff8f533a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00ce200 con 0x7ff8f014afa0 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.142+0000 7ff8f533a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f014b990 con 0x7ff8dc004d40 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.142+0000 7ff8ee7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8f014bb30 0x7ff8f00adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8ee7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8f014bb30 0x7ff8f00adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37872/0 (socket says 172.21.15.59:37872) 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8ee7fc640 1 -- 172.21.15.59:0/899350466 learned_addr learned my addr 172.21.15.59:0/899350466 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8ef7fe640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8f014afa0 0x7ff8f014b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8eeffd640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8dc004d40 0x7ff8f00b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8ee7fc640 1 -- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8dc004d40 msgr2=0x7ff8f00b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8ee7fc640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8dc004d40 0x7ff8f00b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8ee7fc640 1 -- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8f014afa0 msgr2=0x7ff8f014b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8ee7fc640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8f014afa0 0x7ff8f014b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8ee7fc640 1 -- 172.21.15.59:0/899350466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00b4ee0 con 0x7ff8f014bb30 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8eeffd640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8dc004d40 0x7ff8f00b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8ee7fc640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8f014bb30 0x7ff8f00adad0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8011960 tx=0x7ff8d8011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8e77fe640 1 -- 172.21.15.59:0/899350466 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d800faf0 con 0x7ff8f014bb30 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8e77fe640 1 -- 172.21.15.59:0/899350466 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d800fc90 con 0x7ff8f014bb30 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8e77fe640 1 -- 172.21.15.59:0/899350466 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8014440 con 0x7ff8f014bb30 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8f533a640 1 -- 172.21.15.59:0/899350466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f00b51c0 con 0x7ff8f014bb30 2024-03-06T07:24:08.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.143+0000 7ff8f533a640 1 -- 172.21.15.59:0/899350466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f0145a90 con 0x7ff8f014bb30 2024-03-06T07:24:08.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.144+0000 7ff8f533a640 1 -- 172.21.15.59:0/899350466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f004f330 con 0x7ff8f014bb30 2024-03-06T07:24:08.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.145+0000 7ff8e77fe640 1 -- 172.21.15.59:0/899350466 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8d8012040 con 0x7ff8f014bb30 2024-03-06T07:24:08.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.145+0000 7ff8e77fe640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8d006b500 0x7ff8d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.145+0000 7ff8e77fe640 1 -- 172.21.15.59:0/899350466 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff8d808a4f0 con 0x7ff8f014bb30 2024-03-06T07:24:08.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.145+0000 7ff8eeffd640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8d006b500 0x7ff8d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.146+0000 7ff8eeffd640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8d006b500 0x7ff8d006d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff8f006c070 tx=0x7ff8e0006040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.149+0000 7ff8e77fe640 1 -- 172.21.15.59:0/899350466 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8d8019030 con 0x7ff8f014bb30 2024-03-06T07:24:08.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.330+0000 7ff8f533a640 1 -- 172.21.15.59:0/899350466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8f0071c40 con 0x7ff8f014bb30 2024-03-06T07:24:08.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.331+0000 7ff8e77fe640 1 -- 172.21.15.59:0/899350466 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+12879 (secure 0 0 0) 0x7ff8d80596c0 con 0x7ff8f014bb30 2024-03-06T07:24:08.336 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:08.336 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":55,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:24:07.626568+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:08.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 -- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8d006b500 msgr2=0x7ff8d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8d006b500 0x7ff8d006d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff8f006c070 tx=0x7ff8e0006040 comp rx=0 tx=0).stop 2024-03-06T07:24:08.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 -- 172.21.15.59:0/899350466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8f014bb30 msgr2=0x7ff8f00adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8f014bb30 0x7ff8f00adad0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8011960 tx=0x7ff8d8011e20 comp rx=0 tx=0).stop 2024-03-06T07:24:08.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 -- 172.21.15.59:0/899350466 shutdown_connections 2024-03-06T07:24:08.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8d006b500 0x7ff8d006d9c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8f014afa0 0x7ff8f014b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8f014bb30 0x7ff8f00adad0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 --2- 172.21.15.59:0/899350466 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8dc004d40 0x7ff8f00b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 -- 172.21.15.59:0/899350466 >> 172.21.15.59:0/899350466 conn(0x7ff8f006c0c0 msgr2=0x7ff8f006f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 -- 172.21.15.59:0/899350466 shutdown_connections 2024-03-06T07:24:08.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.333+0000 7ff8f533a640 1 -- 172.21.15.59:0/899350466 wait complete. 2024-03-06T07:24:08.346 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:08.346 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:08.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.448+0000 7fe8986f4640 1 Processor -- start 2024-03-06T07:24:08.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.448+0000 7fe8986f4640 1 -- start start 2024-03-06T07:24:08.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.449+0000 7fe8986f4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe8901614d0 0x7fe8900d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.449+0000 7fe8986f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8901618b0 con 0x7fe8900d1910 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.449+0000 7fe8986f4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8900cf400 con 0x7fe8900d24b0 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.449+0000 7fe8986f4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8900cf6a0 con 0x7fe8901614d0 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.449+0000 7fe895c8f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe8900d1910 0x7fe8900d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38434/0 (socket says 172.21.15.59:38434) 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.449+0000 7fe895c8f640 1 -- 172.21.15.59:0/1227540287 learned_addr learned my addr 172.21.15.59:0/1227540287 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.449+0000 7fe896490640 1 --2- 172.21.15.59:0/1227540287 >> v2:172.21.15.164:3300/0 conn(0x7fe8901614d0 0x7fe8900d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164607193 0 0) 0x7fe8901618b0 con 0x7fe8900d1910 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe878003520 con 0x7fe8900d1910 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524565787 0 0) 0x7fe8900cf400 con 0x7fe8900d24b0 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8901618b0 con 0x7fe8900d24b0 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4010477507 0 0) 0x7fe878003520 con 0x7fe8900d1910 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 >> v2:172.21.15.164:3300/0 conn(0x7fe8901614d0 msgr2=0x7fe8900d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 --2- 172.21.15.59:0/1227540287 >> v2:172.21.15.164:3300/0 conn(0x7fe8901614d0 0x7fe8900d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 >> v1:172.21.15.164:6789/0 conn(0x7fe8900d24b0 legacy=0x7fe8900b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900cf980 con 0x7fe8900d1910 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fe884001670 con 0x7fe8900d1910 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 >> v1:172.21.15.59:6789/0 conn(0x7fe8900d1910 legacy=0x7fe8900d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe878003b40 0x7fe878003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe878004800 0x7fe878004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8780051c0 0x7fe87800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900cf6a0 con 0x7fe8780051c0 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87800c040 con 0x7fe878003b40 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87800c240 con 0x7fe878004800 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe896c91640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8780051c0 0x7fe87800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe896490640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe878003b40 0x7fe878003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe895c8f640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe878004800 0x7fe878004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe896c91640 1 -- 172.21.15.59:0/1227540287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe878004800 msgr2=0x7fe878004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe896c91640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe878004800 0x7fe878004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe896c91640 1 -- 172.21.15.59:0/1227540287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe878003b40 msgr2=0x7fe878003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe896c91640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe878003b40 0x7fe878003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe896c91640 1 -- 172.21.15.59:0/1227540287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe878003520 con 0x7fe8780051c0 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.450+0000 7fe896490640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe878003b40 0x7fe878003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.451+0000 7fe896c91640 1 -- 172.21.15.59:0/1227540287 reap_dead start 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.451+0000 7fe896c91640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8780051c0 0x7fe87800b9c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe8880018b0 tx=0x7fe878005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f2d54e0f5ae98a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.451+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880009c10 con 0x7fe8780051c0 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.451+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8880052d0 con 0x7fe8780051c0 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.451+0000 7fe89548e640 1 -- 172.21.15.59:0/1227540287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888001500 con 0x7fe8780051c0 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.452+0000 7fe8986f4640 1 -- 172.21.15.59:0/1227540287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8780051c0 msgr2=0x7fe87800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.452+0000 7fe8986f4640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8780051c0 0x7fe87800b9c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe8880018b0 tx=0x7fe878005830 comp rx=0 tx=0).stop 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.452+0000 7fe8986f4640 1 -- 172.21.15.59:0/1227540287 shutdown_connections 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.452+0000 7fe8986f4640 1 --2- 172.21.15.59:0/1227540287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8780051c0 0x7fe87800b9c0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.452+0000 7fe8986f4640 1 -- 172.21.15.59:0/1227540287 >> 172.21.15.59:0/1227540287 conn(0x7fe8900d3d40 msgr2=0x7fe8900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.452+0000 7fe8986f4640 1 -- 172.21.15.59:0/1227540287 shutdown_connections 2024-03-06T07:24:08.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.452+0000 7fe8986f4640 1 -- 172.21.15.59:0/1227540287 wait complete. 2024-03-06T07:24:08.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.453+0000 7fe8986f4640 1 Processor -- start 2024-03-06T07:24:08.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.453+0000 7fe8986f4640 1 -- start start 2024-03-06T07:24:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe8986f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8780051c0 0x7fe8900ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe8986f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8901562c0 0x7fe8900b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe8986f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8900b22c0 0x7fe8900cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe8986f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900d4860 con 0x7fe8900b22c0 2024-03-06T07:24:08.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe8986f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900b0910 con 0x7fe8901562c0 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe8986f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900d49e0 con 0x7fe8780051c0 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe896c91640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8900b22c0 0x7fe8900cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe896c91640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8900b22c0 0x7fe8900cafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37892/0 (socket says 172.21.15.59:37892) 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe896c91640 1 -- 172.21.15.59:0/1046992875 learned_addr learned my addr 172.21.15.59:0/1046992875 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe895c8f640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8901562c0 0x7fe8900b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe896490640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8780051c0 0x7fe8900ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe896c91640 1 -- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8780051c0 msgr2=0x7fe8900ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe896c91640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8780051c0 0x7fe8900ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe896c91640 1 -- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8901562c0 msgr2=0x7fe8900b1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe896c91640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8901562c0 0x7fe8900b1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe896c91640 1 -- 172.21.15.59:0/1046992875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe890156740 con 0x7fe8900b22c0 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe896c91640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8900b22c0 0x7fe8900cafd0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe8880018b0 tx=0x7fe8880012f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe88effd640 1 -- 172.21.15.59:0/1046992875 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888043070 con 0x7fe8900b22c0 2024-03-06T07:24:08.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe88effd640 1 -- 172.21.15.59:0/1046992875 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe888037040 con 0x7fe8900b22c0 2024-03-06T07:24:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe88effd640 1 -- 172.21.15.59:0/1046992875 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888036070 con 0x7fe8900b22c0 2024-03-06T07:24:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.454+0000 7fe8986f4640 1 -- 172.21.15.59:0/1046992875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8900cbd40 con 0x7fe8900b22c0 2024-03-06T07:24:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.455+0000 7fe8986f4640 1 -- 172.21.15.59:0/1046992875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8900cc300 con 0x7fe8900b22c0 2024-03-06T07:24:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.456+0000 7fe8986f4640 1 -- 172.21.15.59:0/1046992875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8900ca7a0 con 0x7fe8900b22c0 2024-03-06T07:24:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.456+0000 7fe88effd640 1 -- 172.21.15.59:0/1046992875 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe888046070 con 0x7fe8900b22c0 2024-03-06T07:24:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.456+0000 7fe88effd640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe87c06b500 0x7fe87c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.457+0000 7fe88effd640 1 -- 172.21.15.59:0/1046992875 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe8880aa840 con 0x7fe8900b22c0 2024-03-06T07:24:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.457+0000 7fe896490640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe87c06b500 0x7fe87c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.457+0000 7fe896490640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe87c06b500 0x7fe87c06d9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe880009fd0 tx=0x7fe88000b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.461+0000 7fe88effd640 1 -- 172.21.15.59:0/1046992875 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe890156740 con 0x7fe8900b22c0 2024-03-06T07:24:08.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.644+0000 7fe8986f4640 1 -- 172.21.15.59:0/1046992875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe890068ff0 con 0x7fe8900b22c0 2024-03-06T07:24:08.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.644+0000 7fe88effd640 1 -- 172.21.15.59:0/1046992875 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe888079a20 con 0x7fe8900b22c0 2024-03-06T07:24:08.646 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:24:08.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 -- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe87c06b500 msgr2=0x7fe87c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe87c06b500 0x7fe87c06d9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe880009fd0 tx=0x7fe88000b040 comp rx=0 tx=0).stop 2024-03-06T07:24:08.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 -- 172.21.15.59:0/1046992875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8900b22c0 msgr2=0x7fe8900cafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8900b22c0 0x7fe8900cafd0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe8880018b0 tx=0x7fe8880012f0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 -- 172.21.15.59:0/1046992875 shutdown_connections 2024-03-06T07:24:08.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe87c06b500 0x7fe87c06d9c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8900b22c0 0x7fe8900cafd0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8901562c0 0x7fe8900b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 --2- 172.21.15.59:0/1046992875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8780051c0 0x7fe8900ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 -- 172.21.15.59:0/1046992875 >> 172.21.15.59:0/1046992875 conn(0x7fe8900d3d40 msgr2=0x7fe8900b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 -- 172.21.15.59:0/1046992875 shutdown_connections 2024-03-06T07:24:08.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.646+0000 7fe8986f4640 1 -- 172.21.15.59:0/1046992875 wait complete. 2024-03-06T07:24:08.658 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:08.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.758+0000 7fbf8eb1e640 1 Processor -- start 2024-03-06T07:24:08.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.758+0000 7fbf8eb1e640 1 -- start start 2024-03-06T07:24:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8eb1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf880b4f70 0x7fbf880b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8eb1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf88066d90 0x7fbf880ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8eb1e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf880b5890 con 0x7fbf8806c6d0 2024-03-06T07:24:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8eb1e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8815e9f0 con 0x7fbf880b4f70 2024-03-06T07:24:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8eb1e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf880cda10 con 0x7fbf88066d90 2024-03-06T07:24:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8c8ba640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf880b4f70 0x7fbf880b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8c8ba640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf880b4f70 0x7fbf880b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37904/0 (socket says 172.21.15.59:37904) 2024-03-06T07:24:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8c8ba640 1 -- 172.21.15.59:0/141081634 learned_addr learned my addr 172.21.15.59:0/141081634 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8d0bb640 1 --2- 172.21.15.59:0/141081634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf88066d90 0x7fbf880ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8c8ba640 1 -- 172.21.15.59:0/141081634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf88066d90 msgr2=0x7fbf880ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8c8ba640 1 --2- 172.21.15.59:0/141081634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf88066d90 0x7fbf880ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8c8ba640 1 -- 172.21.15.59:0/141081634 >> v1:172.21.15.164:6789/0 conn(0x7fbf8806c6d0 legacy=0x7fbf8815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.759+0000 7fbf8c8ba640 1 -- 172.21.15.59:0/141081634 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf880cd450 con 0x7fbf880b4f70 2024-03-06T07:24:08.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.760+0000 7fbf8c8ba640 1 --2- 172.21.15.59:0/141081634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf880b4f70 0x7fbf880b5350 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009e50 tx=0x7fbf8002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09fcedf55137e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.760+0000 7fbf877fe640 1 -- 172.21.15.59:0/141081634 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8002cb00 con 0x7fbf880b4f70 2024-03-06T07:24:08.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.760+0000 7fbf877fe640 1 -- 172.21.15.59:0/141081634 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8002cca0 con 0x7fbf880b4f70 2024-03-06T07:24:08.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.760+0000 7fbf877fe640 1 -- 172.21.15.59:0/141081634 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf80034610 con 0x7fbf880b4f70 2024-03-06T07:24:08.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.761+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/141081634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf880b4f70 msgr2=0x7fbf880b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.761+0000 7fbf8eb1e640 1 --2- 172.21.15.59:0/141081634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf880b4f70 0x7fbf880b5350 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009e50 tx=0x7fbf8002bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:08.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.761+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/141081634 shutdown_connections 2024-03-06T07:24:08.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.761+0000 7fbf8eb1e640 1 --2- 172.21.15.59:0/141081634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf88066d90 0x7fbf880ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.761+0000 7fbf8eb1e640 1 --2- 172.21.15.59:0/141081634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf880b4f70 0x7fbf880b5350 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.761+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/141081634 >> 172.21.15.59:0/141081634 conn(0x7fbf880724d0 msgr2=0x7fbf880728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.761+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/141081634 shutdown_connections 2024-03-06T07:24:08.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.761+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/141081634 wait complete. 2024-03-06T07:24:08.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.762+0000 7fbf8eb1e640 1 Processor -- start 2024-03-06T07:24:08.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.762+0000 7fbf8eb1e640 1 -- start start 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8eb1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88066d90 0x7fbf8814df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8eb1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf880b4f70 0x7fbf881e8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8eb1e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf8815f5e0 0x7fbf8815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8eb1e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf880cd090 con 0x7fbf88066d90 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8eb1e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8815e680 con 0x7fbf8815f5e0 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8eb1e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf880cd210 con 0x7fbf880b4f70 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8c8ba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88066d90 0x7fbf8814df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8c8ba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88066d90 0x7fbf8814df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37918/0 (socket says 172.21.15.59:37918) 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8c8ba640 1 -- 172.21.15.59:0/3776922159 learned_addr learned my addr 172.21.15.59:0/3776922159 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf87fff640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf880b4f70 0x7fbf881e8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8d0bb640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf8815f5e0 0x7fbf8815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8c8ba640 1 -- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf880b4f70 msgr2=0x7fbf881e8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8c8ba640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf880b4f70 0x7fbf881e8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8c8ba640 1 -- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf8815f5e0 msgr2=0x7fbf8815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8c8ba640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf8815f5e0 0x7fbf8815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8c8ba640 1 -- 172.21.15.59:0/3776922159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8814e4c0 con 0x7fbf88066d90 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf8c8ba640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88066d90 0x7fbf8814df80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009f70 tx=0x7fbf80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.763+0000 7fbf857fa640 1 -- 172.21.15.59:0/3776922159 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf80043020 con 0x7fbf88066d90 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.764+0000 7fbf857fa640 1 -- 172.21.15.59:0/3776922159 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8002ce50 con 0x7fbf88066d90 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.764+0000 7fbf857fa640 1 -- 172.21.15.59:0/3776922159 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8003e690 con 0x7fbf88066d90 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.764+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/3776922159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8815fff0 con 0x7fbf88066d90 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.764+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/3776922159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf881e7a20 con 0x7fbf88066d90 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.765+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/3776922159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8804f330 con 0x7fbf88066d90 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.765+0000 7fbf857fa640 1 -- 172.21.15.59:0/3776922159 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf80046020 con 0x7fbf88066d90 2024-03-06T07:24:08.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.765+0000 7fbf857fa640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf7406b4d0 0x7fbf7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.765+0000 7fbf857fa640 1 -- 172.21.15.59:0/3776922159 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbf800ac650 con 0x7fbf88066d90 2024-03-06T07:24:08.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.766+0000 7fbf87fff640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf7406b4d0 0x7fbf7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.766+0000 7fbf87fff640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf7406b4d0 0x7fbf7406d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c007fd0 tx=0x7fbf7c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.770+0000 7fbf857fa640 1 -- 172.21.15.59:0/3776922159 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf8004d030 con 0x7fbf88066d90 2024-03-06T07:24:08.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.956+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/3776922159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbf8806e060 con 0x7fbf88066d90 2024-03-06T07:24:08.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.957+0000 7fbf857fa640 1 -- 172.21.15.59:0/3776922159 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v55) v1 ==== 84+0+65 (secure 0 0 0) 0x7fbf8007b8b0 con 0x7fbf88066d90 2024-03-06T07:24:08.959 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:08.959 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-03-06T07:24:08.959 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:24:08.959 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-03-06T07:24:08.959 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-03-06T07:24:08.959 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:08.959 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-03-06T07:24:08.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf7406b4d0 msgr2=0x7fbf7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf7406b4d0 0x7fbf7406d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c007fd0 tx=0x7fbf7c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:08.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/3776922159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88066d90 msgr2=0x7fbf8814df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88066d90 0x7fbf8814df80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009f70 tx=0x7fbf80004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:08.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/3776922159 shutdown_connections 2024-03-06T07:24:08.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf7406b4d0 0x7fbf7406d990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf8815f5e0 0x7fbf8815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf880b4f70 0x7fbf881e8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 --2- 172.21.15.59:0/3776922159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf88066d90 0x7fbf8814df80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/3776922159 >> 172.21.15.59:0/3776922159 conn(0x7fbf880724d0 msgr2=0x7fbf8806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/3776922159 shutdown_connections 2024-03-06T07:24:08.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:08.959+0000 7fbf8eb1e640 1 -- 172.21.15.59:0/3776922159 wait complete. 2024-03-06T07:24:08.971 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:24:08.972 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:24:09.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.075+0000 7f7c8dfd0640 1 Processor -- start 2024-03-06T07:24:09.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.075+0000 7f7c8dfd0640 1 -- start start 2024-03-06T07:24:09.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.076+0000 7f7c8dfd0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c880d4630 0x7f7c880d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.076+0000 7f7c8dfd0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7c880d2820 0x7f7c880cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.076+0000 7f7c8dfd0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8815e380 con 0x7f7c8815dfa0 2024-03-06T07:24:09.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.076+0000 7f7c8dfd0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c880d2100 con 0x7f7c880d2820 2024-03-06T07:24:09.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.076+0000 7f7c8dfd0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880cf470 con 0x7f7c880d4630 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.076+0000 7f7c87fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7c880d2820 0x7f7c880cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.076+0000 7f7c87fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7c880d2820 0x7f7c880cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37924/0 (socket says 172.21.15.59:37924) 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.076+0000 7f7c87fff640 1 -- 172.21.15.59:0/3694777690 learned_addr learned my addr 172.21.15.59:0/3694777690 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.077+0000 7f7c86ffd640 1 --2- 172.21.15.59:0/3694777690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c880d4630 0x7f7c880d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.077+0000 7f7c87fff640 1 -- 172.21.15.59:0/3694777690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c880d4630 msgr2=0x7f7c880d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.077+0000 7f7c87fff640 1 --2- 172.21.15.59:0/3694777690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c880d4630 0x7f7c880d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.077+0000 7f7c87fff640 1 -- 172.21.15.59:0/3694777690 >> v1:172.21.15.164:6789/0 conn(0x7f7c8815dfa0 legacy=0x7f7c880d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.077+0000 7f7c87fff640 1 -- 172.21.15.59:0/3694777690 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c880cf750 con 0x7f7c880d2820 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.077+0000 7f7c87fff640 1 --2- 172.21.15.59:0/3694777690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c880d2820 0x7f7c880cb190 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7c80009e50 tx=0x7f7c8002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5a0d6f9aeb02d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.078+0000 7f7c867fc640 1 -- 172.21.15.59:0/3694777690 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8002c9e0 con 0x7f7c880d2820 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.078+0000 7f7c867fc640 1 -- 172.21.15.59:0/3694777690 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8002cb80 con 0x7f7c880d2820 2024-03-06T07:24:09.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.078+0000 7f7c867fc640 1 -- 172.21.15.59:0/3694777690 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c800343f0 con 0x7f7c880d2820 2024-03-06T07:24:09.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.078+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/3694777690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c880d2820 msgr2=0x7f7c880cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.078+0000 7f7c8dfd0640 1 --2- 172.21.15.59:0/3694777690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c880d2820 0x7f7c880cb190 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7c80009e50 tx=0x7f7c8002be40 comp rx=0 tx=0).stop 2024-03-06T07:24:09.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.078+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/3694777690 shutdown_connections 2024-03-06T07:24:09.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.078+0000 7f7c8dfd0640 1 --2- 172.21.15.59:0/3694777690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c880d2820 0x7f7c880cb190 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.078+0000 7f7c8dfd0640 1 --2- 172.21.15.59:0/3694777690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c880d4630 0x7f7c880d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.078+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/3694777690 >> 172.21.15.59:0/3694777690 conn(0x7f7c8809d000 msgr2=0x7f7c880d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.078+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/3694777690 shutdown_connections 2024-03-06T07:24:09.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.078+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/3694777690 wait complete. 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.079+0000 7f7c8dfd0640 1 Processor -- start 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c8dfd0640 1 -- start start 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c8dfd0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c880d2820 0x7f7c8809ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c8dfd0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c880d4630 0x7f7c880b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c8dfd0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c8815dfa0 0x7f7c880c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c8dfd0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880cb850 con 0x7f7c8815dfa0 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c8dfd0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880cb6d0 con 0x7f7c880d2820 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c8dfd0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880cb9d0 con 0x7f7c880d4630 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c87fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c8815dfa0 0x7f7c880c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c87fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c8815dfa0 0x7f7c880c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37938/0 (socket says 172.21.15.59:37938) 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c87fff640 1 -- 172.21.15.59:0/1245136906 learned_addr learned my addr 172.21.15.59:0/1245136906 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c877fe640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c880d2820 0x7f7c8809ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.080+0000 7f7c86ffd640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c880d4630 0x7f7c880b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.081+0000 7f7c87fff640 1 -- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c880d4630 msgr2=0x7f7c880b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.081+0000 7f7c87fff640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c880d4630 0x7f7c880b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.081+0000 7f7c87fff640 1 -- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c880d2820 msgr2=0x7f7c8809ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.081+0000 7f7c87fff640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c880d2820 0x7f7c8809ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.081+0000 7f7c87fff640 1 -- 172.21.15.59:0/1245136906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c880b9790 con 0x7f7c8815dfa0 2024-03-06T07:24:09.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.081+0000 7f7c87fff640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c8815dfa0 0x7f7c880c96c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7c80036040 tx=0x7f7c80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.081+0000 7f7c6ffff640 1 -- 172.21.15.59:0/1245136906 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c80043020 con 0x7f7c8815dfa0 2024-03-06T07:24:09.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.081+0000 7f7c6ffff640 1 -- 172.21.15.59:0/1245136906 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8003e450 con 0x7f7c8815dfa0 2024-03-06T07:24:09.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.081+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/1245136906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c880b94b0 con 0x7f7c8815dfa0 2024-03-06T07:24:09.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.081+0000 7f7c6ffff640 1 -- 172.21.15.59:0/1245136906 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8003e820 con 0x7f7c8815dfa0 2024-03-06T07:24:09.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.081+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/1245136906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8809e690 con 0x7f7c8815dfa0 2024-03-06T07:24:09.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.082+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/1245136906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8804f330 con 0x7f7c8815dfa0 2024-03-06T07:24:09.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.083+0000 7f7c6ffff640 1 -- 172.21.15.59:0/1245136906 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c80046020 con 0x7f7c8815dfa0 2024-03-06T07:24:09.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.083+0000 7f7c6ffff640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c7406b4d0 0x7f7c7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.083+0000 7f7c6ffff640 1 -- 172.21.15.59:0/1245136906 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7c800ac5f0 con 0x7f7c8815dfa0 2024-03-06T07:24:09.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.083+0000 7f7c877fe640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c7406b4d0 0x7f7c7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.084+0000 7f7c877fe640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c7406b4d0 0x7f7c7406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7c880b73c0 tx=0x7f7c78009060 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.087+0000 7f7c6ffff640 1 -- 172.21.15.59:0/1245136906 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c8004d030 con 0x7f7c8815dfa0 2024-03-06T07:24:09.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.268+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/1245136906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7c881610d0 con 0x7f7c8815dfa0 2024-03-06T07:24:09.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.638+0000 7f7c6ffff640 1 -- 172.21.15.59:0/1245136906 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v56) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7c8007b200 con 0x7f7c8815dfa0 2024-03-06T07:24:09.640 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-03-06T07:24:09.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c7406b4d0 msgr2=0x7f7c7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c7406b4d0 0x7f7c7406d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7c880b73c0 tx=0x7f7c78009060 comp rx=0 tx=0).stop 2024-03-06T07:24:09.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/1245136906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c8815dfa0 msgr2=0x7f7c880c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c8815dfa0 0x7f7c880c96c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7c80036040 tx=0x7f7c80004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:09.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/1245136906 shutdown_connections 2024-03-06T07:24:09.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7c7406b4d0 0x7f7c7406d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c8815dfa0 0x7f7c880c96c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c880d4630 0x7f7c880b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 --2- 172.21.15.59:0/1245136906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c880d2820 0x7f7c8809ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/1245136906 >> 172.21.15.59:0/1245136906 conn(0x7f7c8809d000 msgr2=0x7f7c88160560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/1245136906 shutdown_connections 2024-03-06T07:24:09.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.640+0000 7f7c8dfd0640 1 -- 172.21.15.59:0/1245136906 wait complete. 2024-03-06T07:24:09.657 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:09.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.765+0000 7fbed435c640 1 Processor -- start 2024-03-06T07:24:09.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.765+0000 7fbed435c640 1 -- start start 2024-03-06T07:24:09.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed435c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbecc0b41b0 0x7fbecc0cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed435c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbecc0b15c0 0x7fbecc0b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed435c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbecc0d0240 con 0x7fbecc15bdd0 2024-03-06T07:24:09.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed435c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbecc0cf4f0 con 0x7fbecc0b41b0 2024-03-06T07:24:09.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed435c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc0d0850 con 0x7fbecc0b15c0 2024-03-06T07:24:09.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed18f7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbecc0b41b0 0x7fbecc0cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed20f8640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbecc15bdd0 0x7fbecc0ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38440/0 (socket says 172.21.15.59:38440) 2024-03-06T07:24:09.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed20f8640 1 -- 172.21.15.59:0/2280819272 learned_addr learned my addr 172.21.15.59:0/2280819272 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:09.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed18f7640 1 -- 172.21.15.59:0/2280819272 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbecc0b15c0 msgr2=0x7fbecc0b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed18f7640 1 --2- 172.21.15.59:0/2280819272 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbecc0b15c0 0x7fbecc0b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed18f7640 1 -- 172.21.15.59:0/2280819272 >> v1:172.21.15.59:6789/0 conn(0x7fbecc15bdd0 legacy=0x7fbecc0ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed10f6640 1 -- 172.21.15.59:0/2280819272 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076832221 0 0) 0x7fbecc0d0240 con 0x7fbecc15bdd0 2024-03-06T07:24:09.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.766+0000 7fbed18f7640 1 -- 172.21.15.59:0/2280819272 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc0cef50 con 0x7fbecc0b41b0 2024-03-06T07:24:09.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.767+0000 7fbed18f7640 1 --2- 172.21.15.59:0/2280819272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbecc0b41b0 0x7fbecc0cea10 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fbec00097a0 tx=0x7fbec002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927c76cc88c75e6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.767+0000 7fbed10f6640 1 -- 172.21.15.59:0/2280819272 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec0035020 con 0x7fbecc0b41b0 2024-03-06T07:24:09.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.767+0000 7fbed10f6640 1 -- 172.21.15.59:0/2280819272 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec0031830 con 0x7fbecc0b41b0 2024-03-06T07:24:09.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.767+0000 7fbed10f6640 1 -- 172.21.15.59:0/2280819272 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec0031c00 con 0x7fbecc0b41b0 2024-03-06T07:24:09.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.768+0000 7fbed435c640 1 -- 172.21.15.59:0/2280819272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbecc0b41b0 msgr2=0x7fbecc0cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.768+0000 7fbed435c640 1 --2- 172.21.15.59:0/2280819272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbecc0b41b0 0x7fbecc0cea10 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fbec00097a0 tx=0x7fbec002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.768+0000 7fbed435c640 1 -- 172.21.15.59:0/2280819272 shutdown_connections 2024-03-06T07:24:09.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.768+0000 7fbed435c640 1 --2- 172.21.15.59:0/2280819272 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbecc0b15c0 0x7fbecc0b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.768+0000 7fbed435c640 1 --2- 172.21.15.59:0/2280819272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbecc0b41b0 0x7fbecc0cea10 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.768+0000 7fbed435c640 1 -- 172.21.15.59:0/2280819272 >> 172.21.15.59:0/2280819272 conn(0x7fbecc0b3690 msgr2=0x7fbecc0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.768+0000 7fbed435c640 1 -- 172.21.15.59:0/2280819272 shutdown_connections 2024-03-06T07:24:09.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.768+0000 7fbed435c640 1 -- 172.21.15.59:0/2280819272 wait complete. 2024-03-06T07:24:09.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.769+0000 7fbed435c640 1 Processor -- start 2024-03-06T07:24:09.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.769+0000 7fbed435c640 1 -- start start 2024-03-06T07:24:09.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed435c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbecc0b15c0 0x7fbecc0ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed435c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbecc0b41b0 0x7fbecc06cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed435c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbecc15bdd0 0x7fbecc06d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed435c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc15c1b0 con 0x7fbecc0b15c0 2024-03-06T07:24:09.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed435c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc0d04e0 con 0x7fbecc0b41b0 2024-03-06T07:24:09.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed435c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc0b10c0 con 0x7fbecc15bdd0 2024-03-06T07:24:09.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed20f8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbecc0b15c0 0x7fbecc0ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed20f8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbecc0b15c0 0x7fbecc0ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37966/0 (socket says 172.21.15.59:37966) 2024-03-06T07:24:09.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed20f8640 1 -- 172.21.15.59:0/2908915209 learned_addr learned my addr 172.21.15.59:0/2908915209 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed28f9640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbecc15bdd0 0x7fbecc06d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed18f7640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbecc0b41b0 0x7fbecc06cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed20f8640 1 -- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbecc15bdd0 msgr2=0x7fbecc06d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed20f8640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbecc15bdd0 0x7fbecc06d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed20f8640 1 -- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbecc0b41b0 msgr2=0x7fbecc06cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed20f8640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbecc0b41b0 0x7fbecc06cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed20f8640 1 -- 172.21.15.59:0/2908915209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc0cab30 con 0x7fbecc0b15c0 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.770+0000 7fbed20f8640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbecc0b15c0 0x7fbecc0ca5f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbec400c430 tx=0x7fbec400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.771+0000 7fbecaffd640 1 -- 172.21.15.59:0/2908915209 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec40074e0 con 0x7fbecc0b15c0 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.771+0000 7fbecaffd640 1 -- 172.21.15.59:0/2908915209 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec40047e0 con 0x7fbecc0b15c0 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.771+0000 7fbecaffd640 1 -- 172.21.15.59:0/2908915209 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec4004bd0 con 0x7fbecc0b15c0 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.771+0000 7fbed435c640 1 -- 172.21.15.59:0/2908915209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbecc0cadb0 con 0x7fbecc0b15c0 2024-03-06T07:24:09.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.771+0000 7fbed435c640 1 -- 172.21.15.59:0/2908915209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbecc0cb370 con 0x7fbecc0b15c0 2024-03-06T07:24:09.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.772+0000 7fbed435c640 1 -- 172.21.15.59:0/2908915209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbecc09fcf0 con 0x7fbecc0b15c0 2024-03-06T07:24:09.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.772+0000 7fbecaffd640 1 -- 172.21.15.59:0/2908915209 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbec4011070 con 0x7fbecc0b15c0 2024-03-06T07:24:09.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.772+0000 7fbecaffd640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbeb806b4d0 0x7fbeb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.773+0000 7fbecaffd640 1 -- 172.21.15.59:0/2908915209 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fbec4089c50 con 0x7fbecc0b15c0 2024-03-06T07:24:09.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.773+0000 7fbed18f7640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbeb806b4d0 0x7fbeb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.773+0000 7fbed18f7640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbeb806b4d0 0x7fbeb806d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbec0036000 tx=0x7fbec0008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.777+0000 7fbecaffd640 1 -- 172.21.15.59:0/2908915209 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbec408f020 con 0x7fbecc0b15c0 2024-03-06T07:24:09.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:09.959+0000 7fbed435c640 1 -- 172.21.15.59:0/2908915209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbecc068ff0 con 0x7fbecc0b15c0 2024-03-06T07:24:10.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.641+0000 7fbecaffd640 1 -- 172.21.15.59:0/2908915209 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v57) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbecc0cab30 con 0x7fbecc0b15c0 2024-03-06T07:24:10.643 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-03-06T07:24:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.645+0000 7fbed435c640 1 -- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbeb806b4d0 msgr2=0x7fbeb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.645+0000 7fbed435c640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbeb806b4d0 0x7fbeb806d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbec0036000 tx=0x7fbec0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.645+0000 7fbed435c640 1 -- 172.21.15.59:0/2908915209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbecc0b15c0 msgr2=0x7fbecc0ca5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.645+0000 7fbed435c640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbecc0b15c0 0x7fbecc0ca5f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbec400c430 tx=0x7fbec400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.645+0000 7fbed435c640 1 -- 172.21.15.59:0/2908915209 shutdown_connections 2024-03-06T07:24:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.645+0000 7fbed435c640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbeb806b4d0 0x7fbeb806d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.645+0000 7fbed435c640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbecc15bdd0 0x7fbecc06d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.645+0000 7fbed435c640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbecc0b41b0 0x7fbecc06cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.645+0000 7fbed435c640 1 --2- 172.21.15.59:0/2908915209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbecc0b15c0 0x7fbecc0ca5f0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.645+0000 7fbed435c640 1 -- 172.21.15.59:0/2908915209 >> 172.21.15.59:0/2908915209 conn(0x7fbecc0b3690 msgr2=0x7fbecc06ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.645+0000 7fbed435c640 1 -- 172.21.15.59:0/2908915209 shutdown_connections 2024-03-06T07:24:10.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.646+0000 7fbed435c640 1 -- 172.21.15.59:0/2908915209 wait complete. 2024-03-06T07:24:10.673 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:24:10.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.780+0000 7f5e61bc7640 1 Processor -- start 2024-03-06T07:24:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.780+0000 7f5e61bc7640 1 -- start start 2024-03-06T07:24:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.780+0000 7f5e61bc7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e5c158420 0x7f5e5c0b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.780+0000 7f5e61bc7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5e5c158800 0x7f5e5c0b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.780+0000 7f5e61bc7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c15e8a0 con 0x7f5e5c15b370 2024-03-06T07:24:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.780+0000 7f5e61bc7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5c15e650 con 0x7f5e5c158800 2024-03-06T07:24:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.780+0000 7f5e61bc7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c15eb80 con 0x7f5e5c158420 2024-03-06T07:24:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.781+0000 7f5e5affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5e5c158800 0x7f5e5c0b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.781+0000 7f5e5affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5e5c158800 0x7f5e5c0b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37974/0 (socket says 172.21.15.59:37974) 2024-03-06T07:24:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.781+0000 7f5e5affd640 1 -- 172.21.15.59:0/237385494 learned_addr learned my addr 172.21.15.59:0/237385494 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:10.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.781+0000 7f5e5affd640 1 -- 172.21.15.59:0/237385494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e5c158420 msgr2=0x7f5e5c0b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.781+0000 7f5e5b7fe640 1 --2- 172.21.15.59:0/237385494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e5c158420 0x7f5e5c0b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.781+0000 7f5e5affd640 1 --2- 172.21.15.59:0/237385494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e5c158420 0x7f5e5c0b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.781+0000 7f5e5affd640 1 -- 172.21.15.59:0/237385494 >> v1:172.21.15.164:6789/0 conn(0x7f5e5c15b370 legacy=0x7f5e5c15b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.781+0000 7f5e5affd640 1 -- 172.21.15.59:0/237385494 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c15ee60 con 0x7f5e5c158800 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.781+0000 7f5e5b7fe640 1 --2- 172.21.15.59:0/237385494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e5c158420 0x7f5e5c0b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e5affd640 1 --2- 172.21.15.59:0/237385494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e5c158800 0x7f5e5c0b3b00 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009870 tx=0x7f5e5002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814204d22590732 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e5a7fc640 1 -- 172.21.15.59:0/237385494 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e500317c0 con 0x7f5e5c158800 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e5a7fc640 1 -- 172.21.15.59:0/237385494 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e50037070 con 0x7f5e5c158800 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e5a7fc640 1 -- 172.21.15.59:0/237385494 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50031b80 con 0x7f5e5c158800 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e61bc7640 1 -- 172.21.15.59:0/237385494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e5c158800 msgr2=0x7f5e5c0b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e61bc7640 1 --2- 172.21.15.59:0/237385494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e5c158800 0x7f5e5c0b3b00 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009870 tx=0x7f5e5002b860 comp rx=0 tx=0).stop 2024-03-06T07:24:10.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e61bc7640 1 -- 172.21.15.59:0/237385494 shutdown_connections 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e61bc7640 1 --2- 172.21.15.59:0/237385494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e5c158800 0x7f5e5c0b3b00 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e61bc7640 1 --2- 172.21.15.59:0/237385494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e5c158420 0x7f5e5c0b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e61bc7640 1 -- 172.21.15.59:0/237385494 >> 172.21.15.59:0/237385494 conn(0x7f5e5c06afb0 msgr2=0x7f5e5c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e61bc7640 1 -- 172.21.15.59:0/237385494 shutdown_connections 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.782+0000 7f5e61bc7640 1 -- 172.21.15.59:0/237385494 wait complete. 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.783+0000 7f5e61bc7640 1 Processor -- start 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.784+0000 7f5e61bc7640 1 -- start start 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.784+0000 7f5e61bc7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e5c158420 0x7f5e5c0a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.784+0000 7f5e61bc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e5c15b370 0x7f5e5c143340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.784+0000 7f5e61bc7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e5c1e56e0 0x7f5e5c146460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.784+0000 7f5e61bc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c071ba0 con 0x7f5e5c15b370 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.784+0000 7f5e61bc7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c071a20 con 0x7f5e5c1e56e0 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.784+0000 7f5e61bc7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c071d20 con 0x7f5e5c158420 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.784+0000 7f5e5affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e5c15b370 0x7f5e5c143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.784+0000 7f5e5affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e5c15b370 0x7f5e5c143340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37984/0 (socket says 172.21.15.59:37984) 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.784+0000 7f5e5affd640 1 -- 172.21.15.59:0/310666147 learned_addr learned my addr 172.21.15.59:0/310666147 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e5b7fe640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e5c158420 0x7f5e5c0a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e5bfff640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e5c1e56e0 0x7f5e5c146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e5affd640 1 -- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e5c158420 msgr2=0x7f5e5c0a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e5affd640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e5c158420 0x7f5e5c0a95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e5affd640 1 -- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e5c1e56e0 msgr2=0x7f5e5c146460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e5affd640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e5c1e56e0 0x7f5e5c146460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e5affd640 1 -- 172.21.15.59:0/310666147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c1e5b90 con 0x7f5e5c15b370 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e5bfff640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e5c1e56e0 0x7f5e5c146460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e5affd640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e5c15b370 0x7f5e5c143340 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009870 tx=0x7f5e500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e43fff640 1 -- 172.21.15.59:0/310666147 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50032860 con 0x7f5e5c15b370 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e43fff640 1 -- 172.21.15.59:0/310666147 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e50032a00 con 0x7f5e5c15b370 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e61bc7640 1 -- 172.21.15.59:0/310666147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c1469a0 con 0x7f5e5c15b370 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e43fff640 1 -- 172.21.15.59:0/310666147 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50032df0 con 0x7f5e5c15b370 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.785+0000 7f5e61bc7640 1 -- 172.21.15.59:0/310666147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c143e70 con 0x7f5e5c15b370 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.786+0000 7f5e61bc7640 1 -- 172.21.15.59:0/310666147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c04f330 con 0x7f5e5c15b370 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.786+0000 7f5e43fff640 1 -- 172.21.15.59:0/310666147 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e5004d3f0 con 0x7f5e5c15b370 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.787+0000 7f5e43fff640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5e4806b4d0 0x7f5e4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.787+0000 7f5e43fff640 1 -- 172.21.15.59:0/310666147 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f5e500abe10 con 0x7f5e5c15b370 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.787+0000 7f5e5b7fe640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5e4806b4d0 0x7f5e4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.788+0000 7f5e5b7fe640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5e4806b4d0 0x7f5e4806d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00a990 tx=0x7f5e4c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:10.791+0000 7f5e43fff640 1 -- 172.21.15.59:0/310666147 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e50040070 con 0x7f5e5c15b370 2024-03-06T07:24:11.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.009+0000 7f5e61bc7640 1 -- 172.21.15.59:0/310666147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5e5c0af150 con 0x7f5e5c15b370 2024-03-06T07:24:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.666+0000 7f5e43fff640 1 -- 172.21.15.59:0/310666147 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v26) v1 ==== 172+0+0 (secure 0 0 0) 0x7f5e5007ad80 con 0x7f5e5c15b370 2024-03-06T07:24:11.668 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 8 and data pool 9 2024-03-06T07:24:11.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.674+0000 7f5e61bc7640 1 -- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5e4806b4d0 msgr2=0x7f5e4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.674+0000 7f5e61bc7640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5e4806b4d0 0x7f5e4806d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00a990 tx=0x7f5e4c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:11.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.674+0000 7f5e61bc7640 1 -- 172.21.15.59:0/310666147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e5c15b370 msgr2=0x7f5e5c143340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.674+0000 7f5e61bc7640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e5c15b370 0x7f5e5c143340 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009870 tx=0x7f5e500046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.677+0000 7f5e61bc7640 1 -- 172.21.15.59:0/310666147 shutdown_connections 2024-03-06T07:24:11.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.677+0000 7f5e61bc7640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5e4806b4d0 0x7f5e4806d990 secure :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00a990 tx=0x7f5e4c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:11.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.677+0000 7f5e61bc7640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e5c1e56e0 0x7f5e5c146460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.677+0000 7f5e61bc7640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e5c15b370 0x7f5e5c143340 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.677+0000 7f5e61bc7640 1 --2- 172.21.15.59:0/310666147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e5c158420 0x7f5e5c0a95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.677+0000 7f5e61bc7640 1 -- 172.21.15.59:0/310666147 >> 172.21.15.59:0/310666147 conn(0x7f5e5c06afb0 msgr2=0x7f5e5c0cfcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.681+0000 7f5e61bc7640 1 -- 172.21.15.59:0/310666147 shutdown_connections 2024-03-06T07:24:11.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.683+0000 7f5e61bc7640 1 -- 172.21.15.59:0/310666147 wait complete. 2024-03-06T07:24:11.701 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:11.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.805+0000 7f48a0535640 1 Processor -- start 2024-03-06T07:24:11.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.805+0000 7f48a0535640 1 -- start start 2024-03-06T07:24:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.806+0000 7f48a0535640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f48980b6550 0x7f48980cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.806+0000 7f48a0535640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980d06a0 0x7f48980cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.806+0000 7f48a0535640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489815c620 con 0x7f48980d02c0 2024-03-06T07:24:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.806+0000 7f48a0535640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48980b50b0 con 0x7f48980b6550 2024-03-06T07:24:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.806+0000 7f48a0535640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48980cd800 con 0x7f48980d06a0 2024-03-06T07:24:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.806+0000 7f489ead2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980d06a0 0x7f48980cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.806+0000 7f489dad0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f48980b6550 0x7f48980cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.806+0000 7f489e2d1640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f48980d02c0 0x7f48980ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:45716/0 (socket says 172.21.15.59:45716) 2024-03-06T07:24:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.806+0000 7f489ead2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980d06a0 0x7f48980cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:47344/0 (socket says 172.21.15.59:47344) 2024-03-06T07:24:11.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.806+0000 7f489e2d1640 1 -- 172.21.15.59:0/923721196 learned_addr learned my addr 172.21.15.59:0/923721196 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:11.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.807+0000 7f489ead2640 1 -- 172.21.15.59:0/923721196 >> v2:172.21.15.164:3300/0 conn(0x7f48980b6550 msgr2=0x7f48980cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.807+0000 7f489ead2640 1 --2- 172.21.15.59:0/923721196 >> v2:172.21.15.164:3300/0 conn(0x7f48980b6550 0x7f48980cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.807+0000 7f489ead2640 1 -- 172.21.15.59:0/923721196 >> v1:172.21.15.164:6789/0 conn(0x7f48980d02c0 legacy=0x7f48980ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.807+0000 7f489ead2640 1 -- 172.21.15.59:0/923721196 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48980b4a60 con 0x7f48980d06a0 2024-03-06T07:24:11.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.807+0000 7f489dad0640 1 --2- 172.21.15.59:0/923721196 >> v2:172.21.15.164:3300/0 conn(0x7f48980b6550 0x7f48980cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:11.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.807+0000 7f489ead2640 1 --2- 172.21.15.59:0/923721196 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980d06a0 0x7f48980cd0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f48880097a0 tx=0x7f488802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe770b2ff926a307 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.808+0000 7f489d2cf640 1 -- 172.21.15.59:0/923721196 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48880315a0 con 0x7f48980d06a0 2024-03-06T07:24:11.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.808+0000 7f489d2cf640 1 -- 172.21.15.59:0/923721196 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4888037070 con 0x7f48980d06a0 2024-03-06T07:24:11.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.808+0000 7f489d2cf640 1 -- 172.21.15.59:0/923721196 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888031960 con 0x7f48980d06a0 2024-03-06T07:24:11.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.808+0000 7f48a0535640 1 -- 172.21.15.59:0/923721196 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980d06a0 msgr2=0x7f48980cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.808+0000 7f48a0535640 1 --2- 172.21.15.59:0/923721196 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980d06a0 0x7f48980cd0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f48880097a0 tx=0x7f488802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:11.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.808+0000 7f48a0535640 1 -- 172.21.15.59:0/923721196 shutdown_connections 2024-03-06T07:24:11.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.808+0000 7f48a0535640 1 --2- 172.21.15.59:0/923721196 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980d06a0 0x7f48980cd0d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.808+0000 7f48a0535640 1 --2- 172.21.15.59:0/923721196 >> v2:172.21.15.164:3300/0 conn(0x7f48980b6550 0x7f48980cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.808+0000 7f48a0535640 1 -- 172.21.15.59:0/923721196 >> 172.21.15.59:0/923721196 conn(0x7f48980b35a0 msgr2=0x7f48980b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.808+0000 7f48a0535640 1 -- 172.21.15.59:0/923721196 shutdown_connections 2024-03-06T07:24:11.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.808+0000 7f48a0535640 1 -- 172.21.15.59:0/923721196 wait complete. 2024-03-06T07:24:11.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.810+0000 7f48a0535640 1 Processor -- start 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.810+0000 7f48a0535640 1 -- start start 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.810+0000 7f48a0535640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980b6550 0x7f489814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.810+0000 7f48a0535640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48980d02c0 0x7f48980c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.810+0000 7f48a0535640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48980d06a0 0x7f48980ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.810+0000 7f48a0535640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48980ce9c0 con 0x7f48980d06a0 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.810+0000 7f48a0535640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48980ce840 con 0x7f48980d02c0 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.810+0000 7f48a0535640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48980ceb40 con 0x7f48980b6550 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.810+0000 7f489ead2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48980d06a0 0x7f48980ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.810+0000 7f489ead2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48980d06a0 0x7f48980ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38000/0 (socket says 172.21.15.59:38000) 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.810+0000 7f489ead2640 1 -- 172.21.15.59:0/3506248485 learned_addr learned my addr 172.21.15.59:0/3506248485 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f489dad0640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48980d02c0 0x7f48980c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f489e2d1640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980b6550 0x7f489814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f489ead2640 1 -- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980b6550 msgr2=0x7f489814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f489ead2640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980b6550 0x7f489814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f489ead2640 1 -- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48980d02c0 msgr2=0x7f48980c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f489ead2640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48980d02c0 0x7f48980c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f489ead2640 1 -- 172.21.15.59:0/3506248485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48980c8200 con 0x7f48980d06a0 2024-03-06T07:24:11.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f489ead2640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48980d06a0 0x7f48980ca8b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4888034040 tx=0x7f48880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f4896ffd640 1 -- 172.21.15.59:0/3506248485 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888032630 con 0x7f48980d06a0 2024-03-06T07:24:11.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f4896ffd640 1 -- 172.21.15.59:0/3506248485 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48880327d0 con 0x7f48980d06a0 2024-03-06T07:24:11.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f48a0535640 1 -- 172.21.15.59:0/3506248485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48980c6050 con 0x7f48980d06a0 2024-03-06T07:24:11.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f4896ffd640 1 -- 172.21.15.59:0/3506248485 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888032bc0 con 0x7f48980d06a0 2024-03-06T07:24:11.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.811+0000 7f48a0535640 1 -- 172.21.15.59:0/3506248485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48980c6610 con 0x7f48980d06a0 2024-03-06T07:24:11.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.812+0000 7f48a0535640 1 -- 172.21.15.59:0/3506248485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489804f330 con 0x7f48980d06a0 2024-03-06T07:24:11.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.812+0000 7f4896ffd640 1 -- 172.21.15.59:0/3506248485 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f488804c020 con 0x7f48980d06a0 2024-03-06T07:24:11.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.813+0000 7f4896ffd640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f488406b4d0 0x7f488406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.813+0000 7f4896ffd640 1 -- 172.21.15.59:0/3506248485 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f488802f020 con 0x7f48980d06a0 2024-03-06T07:24:11.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.813+0000 7f489e2d1640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f488406b4d0 0x7f488406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.814+0000 7f489e2d1640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f488406b4d0 0x7f488406d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f48900012e0 tx=0x7f4890001a30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.817+0000 7f4896ffd640 1 -- 172.21.15.59:0/3506248485 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4888040070 con 0x7f48980d06a0 2024-03-06T07:24:12.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:11.999+0000 7f48a0535640 1 -- 172.21.15.59:0/3506248485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48980cf3d0 con 0x7f48980d06a0 2024-03-06T07:24:12.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.000+0000 7f4896ffd640 1 -- 172.21.15.59:0/3506248485 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f488807a490 con 0x7f48980d06a0 2024-03-06T07:24:12.002 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:12.002 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":58,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:24:11.010490+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:09.288329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.978574+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:12.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 -- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f488406b4d0 msgr2=0x7f488406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f488406b4d0 0x7f488406d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f48900012e0 tx=0x7f4890001a30 comp rx=0 tx=0).stop 2024-03-06T07:24:12.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 -- 172.21.15.59:0/3506248485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48980d06a0 msgr2=0x7f48980ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48980d06a0 0x7f48980ca8b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4888034040 tx=0x7f48880046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 -- 172.21.15.59:0/3506248485 shutdown_connections 2024-03-06T07:24:12.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f488406b4d0 0x7f488406d990 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48980d06a0 0x7f48980ca8b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48980d02c0 0x7f48980c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 --2- 172.21.15.59:0/3506248485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48980b6550 0x7f489814fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 -- 172.21.15.59:0/3506248485 >> 172.21.15.59:0/3506248485 conn(0x7f48980b35a0 msgr2=0x7f489806ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 -- 172.21.15.59:0/3506248485 shutdown_connections 2024-03-06T07:24:12.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.002+0000 7f48a0535640 1 -- 172.21.15.59:0/3506248485 wait complete. 2024-03-06T07:24:12.015 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:12.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.115+0000 7f6de318a640 1 Processor -- start 2024-03-06T07:24:12.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.115+0000 7f6de318a640 1 -- start start 2024-03-06T07:24:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.116+0000 7f6de318a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ddc15bdd0 0x7f6ddc0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.116+0000 7f6de318a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc0cebe0 con 0x7f6ddc15fa10 2024-03-06T07:24:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.116+0000 7f6de318a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc0ceec0 con 0x7f6ddc0d16e0 2024-03-06T07:24:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.116+0000 7f6de318a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ddc0ce980 con 0x7f6ddc15bdd0 2024-03-06T07:24:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.116+0000 7f6de0f26640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ddc15bdd0 0x7f6ddc0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.116+0000 7f6ddbfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6ddc15fa10 0x7f6ddc0ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38456/0 (socket says 172.21.15.59:38456) 2024-03-06T07:24:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.116+0000 7f6ddbfff640 1 -- 172.21.15.59:0/1253148272 learned_addr learned my addr 172.21.15.59:0/1253148272 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.116+0000 7f6de0f26640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ddc15bdd0 0x7f6ddc0c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38004/0 (socket says 172.21.15.59:38004) 2024-03-06T07:24:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.117+0000 7f6de0f26640 1 -- 172.21.15.59:0/1253148272 >> v1:172.21.15.164:6789/0 conn(0x7f6ddc0d16e0 legacy=0x7f6ddc0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.117+0000 7f6de0f26640 1 -- 172.21.15.59:0/1253148272 >> v1:172.21.15.59:6789/0 conn(0x7f6ddc15fa10 legacy=0x7f6ddc0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.117+0000 7f6de0f26640 1 -- 172.21.15.59:0/1253148272 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc0cf1a0 con 0x7f6ddc15bdd0 2024-03-06T07:24:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.117+0000 7f6de0f26640 1 --2- 172.21.15.59:0/1253148272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ddc15bdd0 0x7f6ddc0c5610 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009d90 tx=0x7f6dd402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e1b0e85ac5dee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.117+0000 7f6ddb7fe640 1 -- 172.21.15.59:0/1253148272 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4039070 con 0x7f6ddc15bdd0 2024-03-06T07:24:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.117+0000 7f6ddb7fe640 1 -- 172.21.15.59:0/1253148272 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd402cbd0 con 0x7f6ddc15bdd0 2024-03-06T07:24:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.117+0000 7f6ddb7fe640 1 -- 172.21.15.59:0/1253148272 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4034440 con 0x7f6ddc15bdd0 2024-03-06T07:24:12.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.118+0000 7f6de318a640 1 -- 172.21.15.59:0/1253148272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ddc15bdd0 msgr2=0x7f6ddc0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.118+0000 7f6de318a640 1 --2- 172.21.15.59:0/1253148272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ddc15bdd0 0x7f6ddc0c5610 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009d90 tx=0x7f6dd402be20 comp rx=0 tx=0).stop 2024-03-06T07:24:12.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.118+0000 7f6de318a640 1 -- 172.21.15.59:0/1253148272 shutdown_connections 2024-03-06T07:24:12.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.118+0000 7f6de318a640 1 --2- 172.21.15.59:0/1253148272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ddc15bdd0 0x7f6ddc0c5610 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.118+0000 7f6de318a640 1 -- 172.21.15.59:0/1253148272 >> 172.21.15.59:0/1253148272 conn(0x7f6ddc09a9f0 msgr2=0x7f6ddc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.118+0000 7f6de318a640 1 -- 172.21.15.59:0/1253148272 shutdown_connections 2024-03-06T07:24:12.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.118+0000 7f6de318a640 1 -- 172.21.15.59:0/1253148272 wait complete. 2024-03-06T07:24:12.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.119+0000 7f6de318a640 1 Processor -- start 2024-03-06T07:24:12.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.119+0000 7f6de318a640 1 -- start start 2024-03-06T07:24:12.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.120+0000 7f6de318a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ddc0d16e0 0x7f6ddc15c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.120+0000 7f6de318a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ddc15bdd0 0x7f6ddc06dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.120+0000 7f6de318a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ddc15fa10 0x7f6ddc155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.120+0000 7f6de318a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc0cc5d0 con 0x7f6ddc15fa10 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.120+0000 7f6de318a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc0c5b50 con 0x7f6ddc15bdd0 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.120+0000 7f6de318a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc15c1b0 con 0x7f6ddc0d16e0 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.120+0000 7f6de1727640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ddc15fa10 0x7f6ddc155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.120+0000 7f6de1727640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ddc15fa10 0x7f6ddc155450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38020/0 (socket says 172.21.15.59:38020) 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.120+0000 7f6de1727640 1 -- 172.21.15.59:0/2768272219 learned_addr learned my addr 172.21.15.59:0/2768272219 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.120+0000 7f6de0f26640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ddc0d16e0 0x7f6ddc15c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.120+0000 7f6ddbfff640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ddc15bdd0 0x7f6ddc06dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.121+0000 7f6de1727640 1 -- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ddc0d16e0 msgr2=0x7f6ddc15c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.121+0000 7f6de1727640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ddc0d16e0 0x7f6ddc15c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.121+0000 7f6de1727640 1 -- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ddc15bdd0 msgr2=0x7f6ddc06dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.121+0000 7f6de1727640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ddc15bdd0 0x7f6ddc06dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.121+0000 7f6de1727640 1 -- 172.21.15.59:0/2768272219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc15ca80 con 0x7f6ddc15fa10 2024-03-06T07:24:12.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.121+0000 7f6de1727640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ddc15fa10 0x7f6ddc155450 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00c440 tx=0x7f6dcc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.121+0000 7f6dd97fa640 1 -- 172.21.15.59:0/2768272219 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc011790 con 0x7f6ddc15fa10 2024-03-06T07:24:12.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.121+0000 7f6dd97fa640 1 -- 172.21.15.59:0/2768272219 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dcc017070 con 0x7f6ddc15fa10 2024-03-06T07:24:12.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.121+0000 7f6de318a640 1 -- 172.21.15.59:0/2768272219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ddc15cd60 con 0x7f6ddc15fa10 2024-03-06T07:24:12.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.121+0000 7f6dd97fa640 1 -- 172.21.15.59:0/2768272219 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc011b70 con 0x7f6ddc15fa10 2024-03-06T07:24:12.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.121+0000 7f6de318a640 1 -- 172.21.15.59:0/2768272219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ddc15d320 con 0x7f6ddc15fa10 2024-03-06T07:24:12.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.122+0000 7f6de318a640 1 -- 172.21.15.59:0/2768272219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ddc04f330 con 0x7f6ddc15fa10 2024-03-06T07:24:12.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.122+0000 7f6dd97fa640 1 -- 172.21.15.59:0/2768272219 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6dcc011d10 con 0x7f6ddc15fa10 2024-03-06T07:24:12.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.123+0000 7f6dd97fa640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6dc806b500 0x7f6dc806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.123+0000 7f6dd97fa640 1 -- 172.21.15.59:0/2768272219 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6dcc089f90 con 0x7f6ddc15fa10 2024-03-06T07:24:12.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.123+0000 7f6de0f26640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6dc806b500 0x7f6dc806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.123+0000 7f6de0f26640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6dc806b500 0x7f6dc806d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4036000 tx=0x7f6dd4030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.127+0000 7f6dd97fa640 1 -- 172.21.15.59:0/2768272219 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6dcc058e10 con 0x7f6ddc15fa10 2024-03-06T07:24:12.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.309+0000 7f6de318a640 1 -- 172.21.15.59:0/2768272219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ddc0701c0 con 0x7f6ddc15fa10 2024-03-06T07:24:12.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.310+0000 7f6dd97fa640 1 -- 172.21.15.59:0/2768272219 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16666 (secure 0 0 0) 0x7f6dcc081e20 con 0x7f6ddc15fa10 2024-03-06T07:24:12.312 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:12.312 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":58,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:24:11.010490+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:09.288329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.978574+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:12.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 -- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6dc806b500 msgr2=0x7f6dc806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6dc806b500 0x7f6dc806d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4036000 tx=0x7f6dd4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:12.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 -- 172.21.15.59:0/2768272219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ddc15fa10 msgr2=0x7f6ddc155450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ddc15fa10 0x7f6ddc155450 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00c440 tx=0x7f6dcc00c900 comp rx=0 tx=0).stop 2024-03-06T07:24:12.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 -- 172.21.15.59:0/2768272219 shutdown_connections 2024-03-06T07:24:12.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6dc806b500 0x7f6dc806d9c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ddc15fa10 0x7f6ddc155450 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ddc15bdd0 0x7f6ddc06dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 --2- 172.21.15.59:0/2768272219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ddc0d16e0 0x7f6ddc15c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 -- 172.21.15.59:0/2768272219 >> 172.21.15.59:0/2768272219 conn(0x7f6ddc09a9f0 msgr2=0x7f6ddc0703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 -- 172.21.15.59:0/2768272219 shutdown_connections 2024-03-06T07:24:12.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.312+0000 7f6de318a640 1 -- 172.21.15.59:0/2768272219 wait complete. 2024-03-06T07:24:12.325 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.428+0000 7fb333c78640 1 Processor -- start 2024-03-06T07:24:12.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.428+0000 7fb333c78640 1 -- start start 2024-03-06T07:24:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb333c78640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb32c15def0 0x7fb32c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb333c78640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb32c0cea50 0x7fb32c0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb333c78640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32c06d030 con 0x7fb32c06cab0 2024-03-06T07:24:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb333c78640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c0cf1c0 con 0x7fb32c0cea50 2024-03-06T07:24:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb333c78640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0cc760 con 0x7fb32c15def0 2024-03-06T07:24:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb332215640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb32c0cea50 0x7fb32c0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb331213640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb32c06cab0 0x7fb32c0ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38470/0 (socket says 172.21.15.59:38470) 2024-03-06T07:24:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb332215640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb32c0cea50 0x7fb32c0ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38036/0 (socket says 172.21.15.59:38036) 2024-03-06T07:24:12.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb331213640 1 -- 172.21.15.59:0/175322987 learned_addr learned my addr 172.21.15.59:0/175322987 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:12.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb331a14640 1 --2- 172.21.15.59:0/175322987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb32c15def0 0x7fb32c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb332215640 1 -- 172.21.15.59:0/175322987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb32c15def0 msgr2=0x7fb32c0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:12.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb332215640 1 --2- 172.21.15.59:0/175322987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb32c15def0 0x7fb32c0b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb332215640 1 -- 172.21.15.59:0/175322987 >> v1:172.21.15.59:6789/0 conn(0x7fb32c06cab0 legacy=0x7fb32c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb332215640 1 -- 172.21.15.59:0/175322987 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c06d310 con 0x7fb32c0cea50 2024-03-06T07:24:12.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.429+0000 7fb330a12640 1 -- 172.21.15.59:0/175322987 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811781415 0 0) 0x7fb32c06d030 con 0x7fb32c06cab0 2024-03-06T07:24:12.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.430+0000 7fb332215640 1 --2- 172.21.15.59:0/175322987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb32c0cea50 0x7fb32c0ccbf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb324009d90 tx=0x7fb32402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f17f981819c27f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.430+0000 7fb330a12640 1 -- 172.21.15.59:0/175322987 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324039070 con 0x7fb32c0cea50 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.430+0000 7fb330a12640 1 -- 172.21.15.59:0/175322987 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32402cce0 con 0x7fb32c0cea50 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.430+0000 7fb330a12640 1 -- 172.21.15.59:0/175322987 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324034660 con 0x7fb32c0cea50 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.431+0000 7fb333c78640 1 -- 172.21.15.59:0/175322987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb32c0cea50 msgr2=0x7fb32c0ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.431+0000 7fb333c78640 1 --2- 172.21.15.59:0/175322987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb32c0cea50 0x7fb32c0ccbf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb324009d90 tx=0x7fb32402bf30 comp rx=0 tx=0).stop 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.431+0000 7fb333c78640 1 -- 172.21.15.59:0/175322987 shutdown_connections 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.431+0000 7fb333c78640 1 --2- 172.21.15.59:0/175322987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb32c0cea50 0x7fb32c0ccbf0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.431+0000 7fb333c78640 1 --2- 172.21.15.59:0/175322987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb32c15def0 0x7fb32c0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.431+0000 7fb333c78640 1 -- 172.21.15.59:0/175322987 >> 172.21.15.59:0/175322987 conn(0x7fb32c0b3690 msgr2=0x7fb32c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.431+0000 7fb333c78640 1 -- 172.21.15.59:0/175322987 shutdown_connections 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.431+0000 7fb333c78640 1 -- 172.21.15.59:0/175322987 wait complete. 2024-03-06T07:24:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.432+0000 7fb333c78640 1 Processor -- start 2024-03-06T07:24:12.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.432+0000 7fb333c78640 1 -- start start 2024-03-06T07:24:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.432+0000 7fb333c78640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb32c06cab0 0x7fb32c0c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.432+0000 7fb333c78640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb32c0cea50 0x7fb32c1f3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb333c78640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb32c15def0 0x7fb32c1f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb333c78640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c15e2d0 con 0x7fb32c06cab0 2024-03-06T07:24:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb333c78640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0cc3f0 con 0x7fb32c15def0 2024-03-06T07:24:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb333c78640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c15e450 con 0x7fb32c0cea50 2024-03-06T07:24:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb331a14640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb32c06cab0 0x7fb32c0c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb331a14640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb32c06cab0 0x7fb32c0c9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38042/0 (socket says 172.21.15.59:38042) 2024-03-06T07:24:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb331a14640 1 -- 172.21.15.59:0/2844825930 learned_addr learned my addr 172.21.15.59:0/2844825930 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb332215640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb32c15def0 0x7fb32c1f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb331213640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb32c0cea50 0x7fb32c1f3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb331a14640 1 -- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb32c0cea50 msgr2=0x7fb32c1f3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb331a14640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb32c0cea50 0x7fb32c1f3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb331a14640 1 -- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb32c15def0 msgr2=0x7fb32c1f69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb331a14640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb32c15def0 0x7fb32c1f69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb331a14640 1 -- 172.21.15.59:0/2844825930 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c0b4920 con 0x7fb32c06cab0 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb331a14640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb32c06cab0 0x7fb32c0c9860 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb32800d5f0 tx=0x7fb32800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb3227fc640 1 -- 172.21.15.59:0/2844825930 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3280125f0 con 0x7fb32c06cab0 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb3227fc640 1 -- 172.21.15.59:0/2844825930 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb328019070 con 0x7fb32c06cab0 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb333c78640 1 -- 172.21.15.59:0/2844825930 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32c0b4c00 con 0x7fb32c06cab0 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb3227fc640 1 -- 172.21.15.59:0/2844825930 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3280129d0 con 0x7fb32c06cab0 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.433+0000 7fb333c78640 1 -- 172.21.15.59:0/2844825930 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32c0b51c0 con 0x7fb32c06cab0 2024-03-06T07:24:12.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.434+0000 7fb333c78640 1 -- 172.21.15.59:0/2844825930 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32c09fcf0 con 0x7fb32c06cab0 2024-03-06T07:24:12.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.435+0000 7fb3227fc640 1 -- 172.21.15.59:0/2844825930 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb328012b70 con 0x7fb32c06cab0 2024-03-06T07:24:12.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.435+0000 7fb3227fc640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb31006b500 0x7fb31006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.435+0000 7fb3227fc640 1 -- 172.21.15.59:0/2844825930 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb32808a620 con 0x7fb32c06cab0 2024-03-06T07:24:12.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.435+0000 7fb331213640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb31006b500 0x7fb31006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.436+0000 7fb331213640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb31006b500 0x7fb31006d9c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0c9da0 tx=0x7fb31c007b50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.439+0000 7fb3227fc640 1 -- 172.21.15.59:0/2844825930 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3280594a0 con 0x7fb32c06cab0 2024-03-06T07:24:12.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.658+0000 7fb333c78640 1 -- 172.21.15.59:0/2844825930 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb32c068ff0 con 0x7fb32c06cab0 2024-03-06T07:24:12.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.663+0000 7fb3227fc640 1 -- 172.21.15.59:0/2844825930 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v27) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb328058da0 con 0x7fb32c06cab0 2024-03-06T07:24:12.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 -- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb31006b500 msgr2=0x7fb31006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb31006b500 0x7fb31006d9c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0c9da0 tx=0x7fb31c007b50 comp rx=0 tx=0).stop 2024-03-06T07:24:12.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 -- 172.21.15.59:0/2844825930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb32c06cab0 msgr2=0x7fb32c0c9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb32c06cab0 0x7fb32c0c9860 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb32800d5f0 tx=0x7fb32800dab0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 -- 172.21.15.59:0/2844825930 shutdown_connections 2024-03-06T07:24:12.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb31006b500 0x7fb31006d9c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb32c15def0 0x7fb32c1f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb32c0cea50 0x7fb32c1f3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 --2- 172.21.15.59:0/2844825930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb32c06cab0 0x7fb32c0c9860 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 -- 172.21.15.59:0/2844825930 >> 172.21.15.59:0/2844825930 conn(0x7fb32c0b3690 msgr2=0x7fb32c160c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 -- 172.21.15.59:0/2844825930 shutdown_connections 2024-03-06T07:24:12.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.666+0000 7fb333c78640 1 -- 172.21.15.59:0/2844825930 wait complete. 2024-03-06T07:24:12.680 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:12.681 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:24:12.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.783+0000 7f6045713640 1 Processor -- start 2024-03-06T07:24:12.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.783+0000 7f6045713640 1 -- start start 2024-03-06T07:24:12.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.783+0000 7f6045713640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f60401614d0 0x7f60400d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.783+0000 7f6045713640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60400d24b0 0x7f60400b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.783+0000 7f6045713640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60401618b0 con 0x7f60400d1910 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.783+0000 7f6045713640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60400cf400 con 0x7f60400d24b0 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.783+0000 7f6045713640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60400cf6a0 con 0x7f60401614d0 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.784+0000 7f603f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60400d24b0 0x7f60400b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.784+0000 7f6037fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f60400d1910 0x7f60400d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38482/0 (socket says 172.21.15.59:38482) 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.784+0000 7f6037fff640 1 -- 172.21.15.59:0/530368032 learned_addr learned my addr 172.21.15.59:0/530368032 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.784+0000 7f603f7fe640 1 -- 172.21.15.59:0/530368032 >> v2:172.21.15.164:3300/0 conn(0x7f60401614d0 msgr2=0x7f60400d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.784+0000 7f603effd640 1 --2- 172.21.15.59:0/530368032 >> v2:172.21.15.164:3300/0 conn(0x7f60401614d0 0x7f60400d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.784+0000 7f603f7fe640 1 --2- 172.21.15.59:0/530368032 >> v2:172.21.15.164:3300/0 conn(0x7f60401614d0 0x7f60400d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.784+0000 7f603f7fe640 1 -- 172.21.15.59:0/530368032 >> v1:172.21.15.59:6789/0 conn(0x7f60400d1910 legacy=0x7f60400d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.784+0000 7f603e7fc640 1 -- 172.21.15.59:0/530368032 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984197960 0 0) 0x7f60401618b0 con 0x7f60400d1910 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.784+0000 7f603f7fe640 1 -- 172.21.15.59:0/530368032 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60400cf980 con 0x7f60400d24b0 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f603f7fe640 1 --2- 172.21.15.59:0/530368032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60400d24b0 0x7f60400b0e30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f604008c470 tx=0x7f603802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728a998888a2f8b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f603e7fc640 1 -- 172.21.15.59:0/530368032 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6038039070 con 0x7f60400d24b0 2024-03-06T07:24:12.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f603e7fc640 1 -- 172.21.15.59:0/530368032 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603802cc80 con 0x7f60400d24b0 2024-03-06T07:24:12.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f603e7fc640 1 -- 172.21.15.59:0/530368032 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6038034660 con 0x7f60400d24b0 2024-03-06T07:24:12.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f6045713640 1 -- 172.21.15.59:0/530368032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60400d24b0 msgr2=0x7f60400b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f6045713640 1 --2- 172.21.15.59:0/530368032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60400d24b0 0x7f60400b0e30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f604008c470 tx=0x7f603802bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f6045713640 1 -- 172.21.15.59:0/530368032 shutdown_connections 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f6045713640 1 --2- 172.21.15.59:0/530368032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60400d24b0 0x7f60400b0e30 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f6045713640 1 --2- 172.21.15.59:0/530368032 >> v2:172.21.15.164:3300/0 conn(0x7f60401614d0 0x7f60400d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f6045713640 1 -- 172.21.15.59:0/530368032 >> 172.21.15.59:0/530368032 conn(0x7f60400d3d40 msgr2=0x7f60400d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f6045713640 1 -- 172.21.15.59:0/530368032 shutdown_connections 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.785+0000 7f6045713640 1 -- 172.21.15.59:0/530368032 wait complete. 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.787+0000 7f6045713640 1 Processor -- start 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.787+0000 7f6045713640 1 -- start start 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.787+0000 7f6045713640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60400d1910 0x7f60401521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.787+0000 7f6045713640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60400d24b0 0x7f60400b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.787+0000 7f6045713640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60401614d0 0x7f60400b9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.787+0000 7f6045713640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60400b1370 con 0x7f60400d24b0 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.787+0000 7f6045713640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60400b0910 con 0x7f60401614d0 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.787+0000 7f6045713640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60400b14f0 con 0x7f60400d1910 2024-03-06T07:24:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.787+0000 7f6037fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60400d24b0 0x7f60400b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f6037fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60400d24b0 0x7f60400b8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38068/0 (socket says 172.21.15.59:38068) 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f6037fff640 1 -- 172.21.15.59:0/1934577389 learned_addr learned my addr 172.21.15.59:0/1934577389 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f603f7fe640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60401614d0 0x7f60400b9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f603effd640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60400d1910 0x7f60401521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f6037fff640 1 -- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60400d1910 msgr2=0x7f60401521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f6037fff640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60400d1910 0x7f60401521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f6037fff640 1 -- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60401614d0 msgr2=0x7f60400b9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f6037fff640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60401614d0 0x7f60400b9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f6037fff640 1 -- 172.21.15.59:0/1934577389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6040152700 con 0x7f60400d24b0 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f603f7fe640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60401614d0 0x7f60400b9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f6037fff640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60400d24b0 0x7f60400b8b00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f602800c4d0 tx=0x7f602800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f60377fe640 1 -- 172.21.15.59:0/1934577389 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028018070 con 0x7f60400d24b0 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f60377fe640 1 -- 172.21.15.59:0/1934577389 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6028007500 con 0x7f60400d24b0 2024-03-06T07:24:12.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f60377fe640 1 -- 172.21.15.59:0/1934577389 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028004a20 con 0x7f60400d24b0 2024-03-06T07:24:12.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f6045713640 1 -- 172.21.15.59:0/1934577389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60401f7860 con 0x7f60400d24b0 2024-03-06T07:24:12.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.788+0000 7f6045713640 1 -- 172.21.15.59:0/1934577389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60401f7de0 con 0x7f60400d24b0 2024-03-06T07:24:12.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.789+0000 7f6045713640 1 -- 172.21.15.59:0/1934577389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60400b9c00 con 0x7f60400d24b0 2024-03-06T07:24:12.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.790+0000 7f60377fe640 1 -- 172.21.15.59:0/1934577389 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f602801b070 con 0x7f60400d24b0 2024-03-06T07:24:12.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.790+0000 7f60377fe640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f602006b4d0 0x7f602006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:12.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.790+0000 7f60377fe640 1 -- 172.21.15.59:0/1934577389 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6028089470 con 0x7f60400d24b0 2024-03-06T07:24:12.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.790+0000 7f603effd640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f602006b4d0 0x7f602006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:12.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.791+0000 7f603effd640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f602006b4d0 0x7f602006d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6030006fd0 tx=0x7f6030008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:12.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:12.794+0000 7f60377fe640 1 -- 172.21.15.59:0/1934577389 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6040152700 con 0x7f60400d24b0 2024-03-06T07:24:13.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.051+0000 7f6045713640 1 -- 172.21.15.59:0/1934577389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6040068ff0 con 0x7f60400d24b0 2024-03-06T07:24:13.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.675+0000 7f60377fe640 1 -- 172.21.15.59:0/1934577389 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6028058380 con 0x7f60400d24b0 2024-03-06T07:24:13.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 -- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f602006b4d0 msgr2=0x7f602006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f602006b4d0 0x7f602006d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6030006fd0 tx=0x7f6030008040 comp rx=0 tx=0).stop 2024-03-06T07:24:13.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 -- 172.21.15.59:0/1934577389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60400d24b0 msgr2=0x7f60400b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60400d24b0 0x7f60400b8b00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f602800c4d0 tx=0x7f602800c990 comp rx=0 tx=0).stop 2024-03-06T07:24:13.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 -- 172.21.15.59:0/1934577389 shutdown_connections 2024-03-06T07:24:13.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f602006b4d0 0x7f602006d990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60401614d0 0x7f60400b9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60400d24b0 0x7f60400b8b00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 --2- 172.21.15.59:0/1934577389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60400d1910 0x7f60401521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 -- 172.21.15.59:0/1934577389 >> 172.21.15.59:0/1934577389 conn(0x7f60400d3d40 msgr2=0x7f60400b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 -- 172.21.15.59:0/1934577389 shutdown_connections 2024-03-06T07:24:13.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.677+0000 7f6045713640 1 -- 172.21.15.59:0/1934577389 wait complete. 2024-03-06T07:24:13.690 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:24:13.690 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:13.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.789+0000 7f6d8a7a0640 1 Processor -- start 2024-03-06T07:24:13.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.789+0000 7f6d8a7a0640 1 -- start start 2024-03-06T07:24:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8a7a0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6d8415dd30 0x7f6d840b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8a7a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d8406afb0 0x7f6d8406b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8a7a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d840b4360 con 0x7f6d8406f710 2024-03-06T07:24:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8a7a0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8406b950 con 0x7f6d8415dd30 2024-03-06T07:24:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8a7a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8406d670 con 0x7f6d8406afb0 2024-03-06T07:24:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8979e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6d8415dd30 0x7f6d840b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8979e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6d8415dd30 0x7f6d840b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50866/0 (socket says 172.21.15.59:50866) 2024-03-06T07:24:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8979e640 1 -- 172.21.15.59:0/3536109508 learned_addr learned my addr 172.21.15.59:0/3536109508 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d88f9d640 1 --2- 172.21.15.59:0/3536109508 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d8406afb0 0x7f6d8406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8979e640 1 -- 172.21.15.59:0/3536109508 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d8406afb0 msgr2=0x7f6d8406b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8979e640 1 --2- 172.21.15.59:0/3536109508 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d8406afb0 0x7f6d8406b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8979e640 1 -- 172.21.15.59:0/3536109508 >> v1:172.21.15.164:6789/0 conn(0x7f6d8406f710 legacy=0x7f6d8406fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.790+0000 7f6d8979e640 1 -- 172.21.15.59:0/3536109508 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8406d950 con 0x7f6d8415dd30 2024-03-06T07:24:13.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.791+0000 7f6d8979e640 1 --2- 172.21.15.59:0/3536109508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d8415dd30 0x7f6d840b3e20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009870 tx=0x7f6d8002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6698844833c8c64b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.791+0000 7f6d7bfff640 1 -- 172.21.15.59:0/3536109508 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d80034070 con 0x7f6d8415dd30 2024-03-06T07:24:13.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.791+0000 7f6d7bfff640 1 -- 172.21.15.59:0/3536109508 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d80030840 con 0x7f6d8415dd30 2024-03-06T07:24:13.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.791+0000 7f6d7bfff640 1 -- 172.21.15.59:0/3536109508 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d80030c10 con 0x7f6d8415dd30 2024-03-06T07:24:13.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.791+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/3536109508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d8415dd30 msgr2=0x7f6d840b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.791+0000 7f6d8a7a0640 1 --2- 172.21.15.59:0/3536109508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d8415dd30 0x7f6d840b3e20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009870 tx=0x7f6d8002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.791+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/3536109508 shutdown_connections 2024-03-06T07:24:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.791+0000 7f6d8a7a0640 1 --2- 172.21.15.59:0/3536109508 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d8406afb0 0x7f6d8406b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.791+0000 7f6d8a7a0640 1 --2- 172.21.15.59:0/3536109508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d8415dd30 0x7f6d840b3e20 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.792+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/3536109508 >> 172.21.15.59:0/3536109508 conn(0x7f6d840d0860 msgr2=0x7f6d840d0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.792+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/3536109508 shutdown_connections 2024-03-06T07:24:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.792+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/3536109508 wait complete. 2024-03-06T07:24:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.793+0000 7f6d8a7a0640 1 Processor -- start 2024-03-06T07:24:13.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.793+0000 7f6d8a7a0640 1 -- start start 2024-03-06T07:24:13.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d8a7a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d8406afb0 0x7f6d841f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d8a7a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d8406f710 0x7f6d841f8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d8a7a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d8415dd30 0x7f6d8415f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d8a7a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8406e1f0 con 0x7f6d8406f710 2024-03-06T07:24:13.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d8a7a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8406df10 con 0x7f6d8415dd30 2024-03-06T07:24:13.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d8a7a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8406e710 con 0x7f6d8406afb0 2024-03-06T07:24:13.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d88f9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d8406f710 0x7f6d841f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d88f9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d8406f710 0x7f6d841f8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50868/0 (socket says 172.21.15.59:50868) 2024-03-06T07:24:13.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d88f9d640 1 -- 172.21.15.59:0/2531377665 learned_addr learned my addr 172.21.15.59:0/2531377665 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d8979e640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d8406afb0 0x7f6d841f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d89f9f640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d8415dd30 0x7f6d8415f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d88f9d640 1 -- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d8406afb0 msgr2=0x7f6d841f6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d88f9d640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d8406afb0 0x7f6d841f6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d88f9d640 1 -- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d8415dd30 msgr2=0x7f6d8415f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d88f9d640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d8415dd30 0x7f6d8415f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.794+0000 7f6d88f9d640 1 -- 172.21.15.59:0/2531377665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d841f8560 con 0x7f6d8406f710 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.795+0000 7f6d88f9d640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d8406f710 0x7f6d841f8020 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400d800 tx=0x7f6d7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.795+0000 7f6d79ffb640 1 -- 172.21.15.59:0/2531377665 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7400b6c0 con 0x7f6d8406f710 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.795+0000 7f6d79ffb640 1 -- 172.21.15.59:0/2531377665 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d74019070 con 0x7f6d8406f710 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.795+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/2531377665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8415e4b0 con 0x7f6d8406f710 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.795+0000 7f6d79ffb640 1 -- 172.21.15.59:0/2531377665 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7400bac0 con 0x7f6d8406f710 2024-03-06T07:24:13.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.795+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/2531377665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8415ea70 con 0x7f6d8406f710 2024-03-06T07:24:13.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.796+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/2531377665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d84068f80 con 0x7f6d8406f710 2024-03-06T07:24:13.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.796+0000 7f6d79ffb640 1 -- 172.21.15.59:0/2531377665 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d7400bc60 con 0x7f6d8406f710 2024-03-06T07:24:13.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.797+0000 7f6d79ffb640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d6806b4d0 0x7f6d6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:13.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.797+0000 7f6d79ffb640 1 -- 172.21.15.59:0/2531377665 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d7408a590 con 0x7f6d8406f710 2024-03-06T07:24:13.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.797+0000 7f6d8979e640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d6806b4d0 0x7f6d6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:13.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.798+0000 7f6d8979e640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d6806b4d0 0x7f6d6806d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6d80036000 tx=0x7f6d8003b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:13.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:13.801+0000 7f6d79ffb640 1 -- 172.21.15.59:0/2531377665 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d841f8560 con 0x7f6d8406f710 2024-03-06T07:24:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.019+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/2531377665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d841614b0 con 0x7f6d8406f710 2024-03-06T07:24:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.020+0000 7f6d79ffb640 1 -- 172.21.15.59:0/2531377665 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f6d74058dc0 con 0x7f6d8406f710 2024-03-06T07:24:14.022 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:14.022 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2205562350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2205562350},{"type":"v1","addr":"172.21.15.164:6837","nonce":2205562350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2606750481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2606750481},{"type":"v1","addr":"172.21.15.164:6836","nonce":2606750481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6048,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3685052110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3685052110},{"type":"v1","addr":"172.21.15.59:6833","nonce":3685052110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:11.010795+0000","modified":"2024-03-06T07:24:13.053115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/750409746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":750409746},{"type":"v1","addr":"172.21.15.59:6835","nonce":750409746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":4}]} 2024-03-06T07:24:14.022 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 28 2024-03-06T07:24:14.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d6806b4d0 msgr2=0x7f6d6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d6806b4d0 0x7f6d6806d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6d80036000 tx=0x7f6d8003b040 comp rx=0 tx=0).stop 2024-03-06T07:24:14.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/2531377665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d8406f710 msgr2=0x7f6d841f8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d8406f710 0x7f6d841f8020 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400d800 tx=0x7f6d7400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/2531377665 shutdown_connections 2024-03-06T07:24:14.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d6806b4d0 0x7f6d6806d990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d8415dd30 0x7f6d8415f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d8406f710 0x7f6d841f8020 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 --2- 172.21.15.59:0/2531377665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d8406afb0 0x7f6d841f6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/2531377665 >> 172.21.15.59:0/2531377665 conn(0x7f6d840d0860 msgr2=0x7f6d8415ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/2531377665 shutdown_connections 2024-03-06T07:24:14.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.022+0000 7f6d8a7a0640 1 -- 172.21.15.59:0/2531377665 wait complete. 2024-03-06T07:24:14.034 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:14.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.134+0000 7f32781e9640 1 Processor -- start 2024-03-06T07:24:14.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.134+0000 7f32781e9640 1 -- start start 2024-03-06T07:24:14.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.135+0000 7f32781e9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f327015bdd0 0x7f32700c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.135+0000 7f32781e9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f327015fa10 0x7f32700ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.135+0000 7f32781e9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32700cebe0 con 0x7f32700d16e0 2024-03-06T07:24:14.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.135+0000 7f32781e9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32700ceec0 con 0x7f327015bdd0 2024-03-06T07:24:14.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.135+0000 7f32781e9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32700cf1a0 con 0x7f327015fa10 2024-03-06T07:24:14.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.135+0000 7f3275f85640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f327015bdd0 0x7f32700c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.135+0000 7f3276786640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f32700d16e0 0x7f32700d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37662/0 (socket says 172.21.15.59:37662) 2024-03-06T07:24:14.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.135+0000 7f3276786640 1 -- 172.21.15.59:0/767339343 learned_addr learned my addr 172.21.15.59:0/767339343 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:14.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.135+0000 7f3275784640 1 --2- 172.21.15.59:0/767339343 >> v2:172.21.15.164:3300/0 conn(0x7f327015fa10 0x7f32700ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.136+0000 7f3275f85640 1 -- 172.21.15.59:0/767339343 >> v2:172.21.15.164:3300/0 conn(0x7f327015fa10 msgr2=0x7f32700ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.136+0000 7f3275f85640 1 --2- 172.21.15.59:0/767339343 >> v2:172.21.15.164:3300/0 conn(0x7f327015fa10 0x7f32700ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.136+0000 7f3275f85640 1 -- 172.21.15.59:0/767339343 >> v1:172.21.15.59:6789/0 conn(0x7f32700d16e0 legacy=0x7f32700d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.136+0000 7f3275f85640 1 -- 172.21.15.59:0/767339343 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32700cc8e0 con 0x7f327015bdd0 2024-03-06T07:24:14.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.136+0000 7f3274f83640 1 -- 172.21.15.59:0/767339343 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092606625 0 0) 0x7f32700cebe0 con 0x7f32700d16e0 2024-03-06T07:24:14.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.136+0000 7f3275f85640 1 --2- 172.21.15.59:0/767339343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f327015bdd0 0x7f32700c5610 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f32700cdfd0 tx=0x7f326802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f501ebc0cf6308 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.136+0000 7f3274f83640 1 -- 172.21.15.59:0/767339343 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268039070 con 0x7f327015bdd0 2024-03-06T07:24:14.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.136+0000 7f3274f83640 1 -- 172.21.15.59:0/767339343 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326802cbf0 con 0x7f327015bdd0 2024-03-06T07:24:14.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.136+0000 7f3274f83640 1 -- 172.21.15.59:0/767339343 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268034440 con 0x7f327015bdd0 2024-03-06T07:24:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.137+0000 7f32781e9640 1 -- 172.21.15.59:0/767339343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f327015bdd0 msgr2=0x7f32700c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.137+0000 7f32781e9640 1 --2- 172.21.15.59:0/767339343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f327015bdd0 0x7f32700c5610 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f32700cdfd0 tx=0x7f326802be40 comp rx=0 tx=0).stop 2024-03-06T07:24:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.137+0000 7f32781e9640 1 -- 172.21.15.59:0/767339343 shutdown_connections 2024-03-06T07:24:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.137+0000 7f32781e9640 1 --2- 172.21.15.59:0/767339343 >> v2:172.21.15.164:3300/0 conn(0x7f327015fa10 0x7f32700ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.137+0000 7f32781e9640 1 --2- 172.21.15.59:0/767339343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f327015bdd0 0x7f32700c5610 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.137+0000 7f32781e9640 1 -- 172.21.15.59:0/767339343 >> 172.21.15.59:0/767339343 conn(0x7f327009a9f0 msgr2=0x7f327009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.137+0000 7f32781e9640 1 -- 172.21.15.59:0/767339343 shutdown_connections 2024-03-06T07:24:14.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.137+0000 7f32781e9640 1 -- 172.21.15.59:0/767339343 wait complete. 2024-03-06T07:24:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.138+0000 7f32781e9640 1 Processor -- start 2024-03-06T07:24:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.138+0000 7f32781e9640 1 -- start start 2024-03-06T07:24:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f32781e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32700d16e0 0x7f3270154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f32781e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f327015bdd0 0x7f327015d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f32781e9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f327015fa10 0x7f3270158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f32781e9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327015c1b0 con 0x7f32700d16e0 2024-03-06T07:24:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f32781e9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32700cc5d0 con 0x7f327015fa10 2024-03-06T07:24:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f32781e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32700ce7b0 con 0x7f327015bdd0 2024-03-06T07:24:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f3275f85640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32700d16e0 0x7f3270154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f3275f85640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32700d16e0 0x7f3270154a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50884/0 (socket says 172.21.15.59:50884) 2024-03-06T07:24:14.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f3275f85640 1 -- 172.21.15.59:0/4217579072 learned_addr learned my addr 172.21.15.59:0/4217579072 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f3276786640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f327015fa10 0x7f3270158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f3275784640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f327015bdd0 0x7f327015d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f3275f85640 1 -- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f327015bdd0 msgr2=0x7f327015d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f3275f85640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f327015bdd0 0x7f327015d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f3275f85640 1 -- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f327015fa10 msgr2=0x7f3270158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f3275f85640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f327015fa10 0x7f3270158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f3275f85640 1 -- 172.21.15.59:0/4217579072 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32700b6050 con 0x7f32700d16e0 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.139+0000 7f3275784640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f327015bdd0 0x7f327015d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.140+0000 7f3275f85640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32700d16e0 0x7f3270154a20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3268037040 tx=0x7f3268004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.140+0000 7f32667fc640 1 -- 172.21.15.59:0/4217579072 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268039050 con 0x7f32700d16e0 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.140+0000 7f32667fc640 1 -- 172.21.15.59:0/4217579072 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3268045070 con 0x7f32700d16e0 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.140+0000 7f32781e9640 1 -- 172.21.15.59:0/4217579072 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32700b6330 con 0x7f32700d16e0 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.140+0000 7f32667fc640 1 -- 172.21.15.59:0/4217579072 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326803e850 con 0x7f32700d16e0 2024-03-06T07:24:14.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.140+0000 7f32781e9640 1 -- 172.21.15.59:0/4217579072 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32700b68f0 con 0x7f32700d16e0 2024-03-06T07:24:14.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.141+0000 7f32781e9640 1 -- 172.21.15.59:0/4217579072 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32700c5610 con 0x7f32700d16e0 2024-03-06T07:24:14.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.141+0000 7f32667fc640 1 -- 172.21.15.59:0/4217579072 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f326803e9f0 con 0x7f32700d16e0 2024-03-06T07:24:14.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.142+0000 7f32667fc640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f325406b500 0x7f325406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.142+0000 7f32667fc640 1 -- 172.21.15.59:0/4217579072 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32680ad060 con 0x7f32700d16e0 2024-03-06T07:24:14.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.142+0000 7f3275784640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f325406b500 0x7f325406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.143+0000 7f3275784640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f325406b500 0x7f325406d9c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f32701557d0 tx=0x7f3260008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.146+0000 7f32667fc640 1 -- 172.21.15.59:0/4217579072 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32700c5610 con 0x7f32700d16e0 2024-03-06T07:24:14.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.409+0000 7f32781e9640 1 -- 172.21.15.59:0/4217579072 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3270069e80 con 0x7f32700d16e0 2024-03-06T07:24:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.410+0000 7f32667fc640 1 -- 172.21.15.59:0/4217579072 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f3270069e80 con 0x7f32700d16e0 2024-03-06T07:24:14.412 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:14.413 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:24:12.652576+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:09.288329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.978574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 -- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f325406b500 msgr2=0x7f325406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f325406b500 0x7f325406d9c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f32701557d0 tx=0x7f3260008040 comp rx=0 tx=0).stop 2024-03-06T07:24:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 -- 172.21.15.59:0/4217579072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32700d16e0 msgr2=0x7f3270154a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32700d16e0 0x7f3270154a20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3268037040 tx=0x7f3268004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 -- 172.21.15.59:0/4217579072 shutdown_connections 2024-03-06T07:24:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f325406b500 0x7f325406d9c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f327015fa10 0x7f3270158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f327015bdd0 0x7f327015d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 --2- 172.21.15.59:0/4217579072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32700d16e0 0x7f3270154a20 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 -- 172.21.15.59:0/4217579072 >> 172.21.15.59:0/4217579072 conn(0x7f327009a9f0 msgr2=0x7f32700c7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 -- 172.21.15.59:0/4217579072 shutdown_connections 2024-03-06T07:24:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.413+0000 7f32781e9640 1 -- 172.21.15.59:0/4217579072 wait complete. 2024-03-06T07:24:14.426 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:14.426 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:14.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.526+0000 7f774e393640 1 Processor -- start 2024-03-06T07:24:14.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.526+0000 7f774e393640 1 -- start start 2024-03-06T07:24:14.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.526+0000 7f774e393640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f774815dfe0 0x7f77480cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.526+0000 7f774e393640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77480d29c0 0x7f77480b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.526+0000 7f774e393640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77480b3430 con 0x7f77480cf830 2024-03-06T07:24:14.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.526+0000 7f774e393640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77480b31e0 con 0x7f774815dfe0 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.526+0000 7f774e393640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77480b3d60 con 0x7f77480d29c0 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.527+0000 7f77477fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f77480cf830 0x7f77480d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37672/0 (socket says 172.21.15.59:37672) 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.527+0000 7f77477fe640 1 -- 172.21.15.59:0/2396911282 learned_addr learned my addr 172.21.15.59:0/2396911282 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.527+0000 7f7747fff640 1 --2- 172.21.15.59:0/2396911282 >> v2:172.21.15.164:3300/0 conn(0x7f774815dfe0 0x7f77480cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.527+0000 7f774c930640 1 --2- 172.21.15.59:0/2396911282 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77480d29c0 0x7f77480b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.527+0000 7f7746ffd640 1 -- 172.21.15.59:0/2396911282 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465583811 0 0) 0x7f77480b3430 con 0x7f77480cf830 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.527+0000 7f7746ffd640 1 -- 172.21.15.59:0/2396911282 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7730003520 con 0x7f77480cf830 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.527+0000 7f7747fff640 1 -- 172.21.15.59:0/2396911282 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77480d29c0 msgr2=0x7f77480b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.527+0000 7f7747fff640 1 --2- 172.21.15.59:0/2396911282 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77480d29c0 0x7f77480b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.527+0000 7f7747fff640 1 -- 172.21.15.59:0/2396911282 >> v1:172.21.15.59:6789/0 conn(0x7f77480cf830 legacy=0x7f77480d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.527+0000 7f7746ffd640 1 -- 172.21.15.59:0/2396911282 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2349545702 0 0) 0x7f7730003520 con 0x7f77480cf830 2024-03-06T07:24:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.527+0000 7f7747fff640 1 -- 172.21.15.59:0/2396911282 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77480b3710 con 0x7f774815dfe0 2024-03-06T07:24:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.528+0000 7f7747fff640 1 --2- 172.21.15.59:0/2396911282 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f774815dfe0 0x7f77480cf2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f77380097a0 tx=0x7f773802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=798e86f31243202c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.528+0000 7f7746ffd640 1 -- 172.21.15.59:0/2396911282 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7738037070 con 0x7f774815dfe0 2024-03-06T07:24:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.528+0000 7f7746ffd640 1 -- 172.21.15.59:0/2396911282 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7738030510 con 0x7f774815dfe0 2024-03-06T07:24:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.528+0000 7f7746ffd640 1 -- 172.21.15.59:0/2396911282 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c001670 con 0x7f774815dfe0 2024-03-06T07:24:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.529+0000 7f774e393640 1 -- 172.21.15.59:0/2396911282 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f774815dfe0 msgr2=0x7f77480cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.529+0000 7f774e393640 1 --2- 172.21.15.59:0/2396911282 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f774815dfe0 0x7f77480cf2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f77380097a0 tx=0x7f773802b790 comp rx=0 tx=0).stop 2024-03-06T07:24:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.529+0000 7f774e393640 1 -- 172.21.15.59:0/2396911282 shutdown_connections 2024-03-06T07:24:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.529+0000 7f774e393640 1 --2- 172.21.15.59:0/2396911282 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77480d29c0 0x7f77480b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.529+0000 7f774e393640 1 --2- 172.21.15.59:0/2396911282 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f774815dfe0 0x7f77480cf2f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.529+0000 7f774e393640 1 -- 172.21.15.59:0/2396911282 >> 172.21.15.59:0/2396911282 conn(0x7f77480d3cd0 msgr2=0x7f77480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.529+0000 7f774e393640 1 -- 172.21.15.59:0/2396911282 shutdown_connections 2024-03-06T07:24:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.529+0000 7f774e393640 1 -- 172.21.15.59:0/2396911282 wait complete. 2024-03-06T07:24:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.530+0000 7f774e393640 1 Processor -- start 2024-03-06T07:24:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.530+0000 7f774e393640 1 -- start start 2024-03-06T07:24:14.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f774e393640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77480cf830 0x7f77480a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f774e393640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77480d29c0 0x7f77481500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f774e393640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f774815dfe0 0x7f7748150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f774e393640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77480d1df0 con 0x7f77480d29c0 2024-03-06T07:24:14.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f774e393640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77480d1c70 con 0x7f77480cf830 2024-03-06T07:24:14.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f774e393640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f774815e3c0 con 0x7f774815dfe0 2024-03-06T07:24:14.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f77477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77480d29c0 0x7f77481500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f77477fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77480d29c0 0x7f77481500c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50896/0 (socket says 172.21.15.59:50896) 2024-03-06T07:24:14.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f77477fe640 1 -- 172.21.15.59:0/2141705849 learned_addr learned my addr 172.21.15.59:0/2141705849 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f7747fff640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77480cf830 0x7f77480a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f774c930640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f774815dfe0 0x7f7748150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f77477fe640 1 -- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f774815dfe0 msgr2=0x7f7748150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f77477fe640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f774815dfe0 0x7f7748150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f77477fe640 1 -- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77480cf830 msgr2=0x7f77480a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f77477fe640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77480cf830 0x7f77480a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f77477fe640 1 -- 172.21.15.59:0/2141705849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77480b9230 con 0x7f77480d29c0 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.531+0000 7f774c930640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f774815dfe0 0x7f7748150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.532+0000 7f77477fe640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77480d29c0 0x7f77481500c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f773c0017e0 tx=0x7f773c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.532+0000 7f7744ff9640 1 -- 172.21.15.59:0/2141705849 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c005ad0 con 0x7f77480d29c0 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.532+0000 7f7744ff9640 1 -- 172.21.15.59:0/2141705849 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773c005c70 con 0x7f77480d29c0 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.532+0000 7f774e393640 1 -- 172.21.15.59:0/2141705849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f774814e0d0 con 0x7f77480d29c0 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.532+0000 7f7744ff9640 1 -- 172.21.15.59:0/2141705849 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c005e60 con 0x7f77480d29c0 2024-03-06T07:24:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.532+0000 7f774e393640 1 -- 172.21.15.59:0/2141705849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f774814e690 con 0x7f77480d29c0 2024-03-06T07:24:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.533+0000 7f774e393640 1 -- 172.21.15.59:0/2141705849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f774804f330 con 0x7f77480d29c0 2024-03-06T07:24:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.533+0000 7f7744ff9640 1 -- 172.21.15.59:0/2141705849 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f773c007290 con 0x7f77480d29c0 2024-03-06T07:24:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.534+0000 7f7744ff9640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f773406b4d0 0x7f773406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.534+0000 7f7744ff9640 1 -- 172.21.15.59:0/2141705849 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f773c089f90 con 0x7f77480d29c0 2024-03-06T07:24:14.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.534+0000 7f7747fff640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f773406b4d0 0x7f773406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.535+0000 7f7747fff640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f773406b4d0 0x7f773406d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7738034000 tx=0x7f773802f000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.541+0000 7f7744ff9640 1 -- 172.21.15.59:0/2141705849 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f773c058e10 con 0x7f77480d29c0 2024-03-06T07:24:14.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.758+0000 7f774e393640 1 -- 172.21.15.59:0/2141705849 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77480b0d80 con 0x7f773406b4d0 2024-03-06T07:24:14.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.764+0000 7f7744ff9640 1 -- 172.21.15.59:0/2141705849 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246638 (secure 0 0 0) 0x7f77480b0d80 con 0x7f773406b4d0 2024-03-06T07:24:14.769 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:14.771 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:24:14.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 -- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f773406b4d0 msgr2=0x7f773406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f773406b4d0 0x7f773406d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7738034000 tx=0x7f773802f000 comp rx=0 tx=0).stop 2024-03-06T07:24:14.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 -- 172.21.15.59:0/2141705849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77480d29c0 msgr2=0x7f77481500c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77480d29c0 0x7f77481500c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f773c0017e0 tx=0x7f773c00cbe0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 -- 172.21.15.59:0/2141705849 shutdown_connections 2024-03-06T07:24:14.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f773406b4d0 0x7f773406d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f774815dfe0 0x7f7748150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77480d29c0 0x7f77481500c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 --2- 172.21.15.59:0/2141705849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77480cf830 0x7f77480a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 -- 172.21.15.59:0/2141705849 >> 172.21.15.59:0/2141705849 conn(0x7f77480d3cd0 msgr2=0x7f774806d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 -- 172.21.15.59:0/2141705849 shutdown_connections 2024-03-06T07:24:14.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.767+0000 7f774e393640 1 -- 172.21.15.59:0/2141705849 wait complete. 2024-03-06T07:24:14.779 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-03-06T07:24:13.329617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255916,"kb_used_data":6456,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718804,"statfs":{"total":773094113280,"available":772832055296,"internally_reserved":0,"allocated":6610944,"data_stored":3417244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000962"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653222+0000","last_change":"2024-03-06T07:24:10.653222+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653222+0000","last_undegraded":"2024-03-06T07:24:10.653222+0000","last_fullsized":"2024-03-06T07:24:10.653222+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654247+0000","last_change":"2024-03-06T07:24:10.654247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654247+0000","last_undegraded":"2024-03-06T07:24:10.654247+0000","last_fullsized":"2024-03-06T07:24:10.654247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653845+0000","last_change":"2024-03-06T07:24:10.653845+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653845+0000","last_undegraded":"2024-03-06T07:24:10.653845+0000","last_fullsized":"2024-03-06T07:24:10.653845+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649798+0000","last_change":"2024-03-06T07:24:10.649798+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649798+0000","last_undegraded":"2024-03-06T07:24:10.649798+0000","last_fullsized":"2024-03-06T07:24:10.649798+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653052+0000","last_change":"2024-03-06T07:24:10.653052+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653052+0000","last_undegraded":"2024-03-06T07:24:10.653052+0000","last_fullsized":"2024-03-06T07:24:10.653052+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649635+0000","last_change":"2024-03-06T07:24:10.649635+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649635+0000","last_undegraded":"2024-03-06T07:24:10.649635+0000","last_fullsized":"2024-03-06T07:24:10.649635+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654247+0000","last_change":"2024-03-06T07:24:10.654247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654247+0000","last_undegraded":"2024-03-06T07:24:10.654247+0000","last_fullsized":"2024-03-06T07:24:10.654247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653163+0000","last_change":"2024-03-06T07:24:10.653163+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653163+0000","last_undegraded":"2024-03-06T07:24:10.653163+0000","last_fullsized":"2024-03-06T07:24:10.653163+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.655335+0000","last_change":"2024-03-06T07:24:10.655335+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.655335+0000","last_undegraded":"2024-03-06T07:24:10.655335+0000","last_fullsized":"2024-03-06T07:24:10.655335+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648323+0000","last_change":"2024-03-06T07:24:10.648323+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648323+0000","last_undegraded":"2024-03-06T07:24:10.648323+0000","last_fullsized":"2024-03-06T07:24:10.648323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648785+0000","last_change":"2024-03-06T07:24:10.648785+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648785+0000","last_undegraded":"2024-03-06T07:24:10.648785+0000","last_fullsized":"2024-03-06T07:24:10.648785+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649164+0000","last_change":"2024-03-06T07:24:10.649164+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649164+0000","last_undegraded":"2024-03-06T07:24:10.649164+0000","last_fullsized":"2024-03-06T07:24:10.649164+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.646039+0000","last_change":"2024-03-06T07:24:10.646039+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.646039+0000","last_undegraded":"2024-03-06T07:24:10.646039+0000","last_fullsized":"2024-03-06T07:24:10.646039+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653571+0000","last_change":"2024-03-06T07:24:10.653571+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653571+0000","last_undegraded":"2024-03-06T07:24:10.653571+0000","last_fullsized":"2024-03-06T07:24:10.653571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653868+0000","last_change":"2024-03-06T07:24:10.653868+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653868+0000","last_undegraded":"2024-03-06T07:24:10.653868+0000","last_fullsized":"2024-03-06T07:24:10.653868+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.650079+0000","last_change":"2024-03-06T07:24:10.650079+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.650079+0000","last_undegraded":"2024-03-06T07:24:10.650079+0000","last_fullsized":"2024-03-06T07:24:10.650079+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652629+0000","last_change":"2024-03-06T07:24:10.652629+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652629+0000","last_undegraded":"2024-03-06T07:24:10.652629+0000","last_fullsized":"2024-03-06T07:24:10.652629+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653025+0000","last_change":"2024-03-06T07:24:10.653025+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653025+0000","last_undegraded":"2024-03-06T07:24:10.653025+0000","last_fullsized":"2024-03-06T07:24:10.653025+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653633+0000","last_change":"2024-03-06T07:24:10.653633+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653633+0000","last_undegraded":"2024-03-06T07:24:10.653633+0000","last_fullsized":"2024-03-06T07:24:10.653633+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.655332+0000","last_change":"2024-03-06T07:24:10.655332+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.655332+0000","last_undegraded":"2024-03-06T07:24:10.655332+0000","last_fullsized":"2024-03-06T07:24:10.655332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652655+0000","last_change":"2024-03-06T07:24:10.652655+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652655+0000","last_undegraded":"2024-03-06T07:24:10.652655+0000","last_fullsized":"2024-03-06T07:24:10.652655+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653247+0000","last_change":"2024-03-06T07:24:10.653247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653247+0000","last_undegraded":"2024-03-06T07:24:10.653247+0000","last_fullsized":"2024-03-06T07:24:10.653247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651973+0000","last_change":"2024-03-06T07:24:10.651973+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651973+0000","last_undegraded":"2024-03-06T07:24:10.651973+0000","last_fullsized":"2024-03-06T07:24:10.651973+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647134+0000","last_change":"2024-03-06T07:24:10.647134+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647134+0000","last_undegraded":"2024-03-06T07:24:10.647134+0000","last_fullsized":"2024-03-06T07:24:10.647134+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652823+0000","last_change":"2024-03-06T07:24:10.652823+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652823+0000","last_undegraded":"2024-03-06T07:24:10.652823+0000","last_fullsized":"2024-03-06T07:24:10.652823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649078+0000","last_change":"2024-03-06T07:24:10.649078+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649078+0000","last_undegraded":"2024-03-06T07:24:10.649078+0000","last_fullsized":"2024-03-06T07:24:10.649078+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649913+0000","last_change":"2024-03-06T07:24:10.649913+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649913+0000","last_undegraded":"2024-03-06T07:24:10.649913+0000","last_fullsized":"2024-03-06T07:24:10.649913+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653647+0000","last_change":"2024-03-06T07:24:10.653647+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653647+0000","last_undegraded":"2024-03-06T07:24:10.653647+0000","last_fullsized":"2024-03-06T07:24:10.653647+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653928+0000","last_change":"2024-03-06T07:24:10.653928+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653928+0000","last_undegraded":"2024-03-06T07:24:10.653928+0000","last_fullsized":"2024-03-06T07:24:10.653928+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651185+0000","last_change":"2024-03-06T07:24:10.651185+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651185+0000","last_undegraded":"2024-03-06T07:24:10.651185+0000","last_fullsized":"2024-03-06T07:24:10.651185+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651671+0000","last_change":"2024-03-06T07:24:10.651671+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651671+0000","last_undegraded":"2024-03-06T07:24:10.651671+0000","last_fullsized":"2024-03-06T07:24:10.651671+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647547+0000","last_change":"2024-03-06T07:24:10.647547+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647547+0000","last_undegraded":"2024-03-06T07:24:10.647547+0000","last_fullsized":"2024-03-06T07:24:10.647547+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648981+0000","last_change":"2024-03-06T07:24:10.648981+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648981+0000","last_undegraded":"2024-03-06T07:24:10.648981+0000","last_fullsized":"2024-03-06T07:24:10.648981+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661442+0000","last_change":"2024-03-06T07:24:10.658033+0000","last_active":"2024-03-06T07:24:10.661442+0000","last_peered":"2024-03-06T07:24:10.661442+0000","last_clean":"2024-03-06T07:24:10.661442+0000","last_became_active":"2024-03-06T07:24:10.657766+0000","last_became_peered":"2024-03-06T07:24:10.657766+0000","last_unstale":"2024-03-06T07:24:10.661442+0000","last_undegraded":"2024-03-06T07:24:10.661442+0000","last_fullsized":"2024-03-06T07:24:10.661442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648210+0000","last_change":"2024-03-06T07:24:10.648210+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648210+0000","last_undegraded":"2024-03-06T07:24:10.648210+0000","last_fullsized":"2024-03-06T07:24:10.648210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660926+0000","last_change":"2024-03-06T07:24:10.659965+0000","last_active":"2024-03-06T07:24:10.660926+0000","last_peered":"2024-03-06T07:24:10.660926+0000","last_clean":"2024-03-06T07:24:10.660926+0000","last_became_active":"2024-03-06T07:24:10.659581+0000","last_became_peered":"2024-03-06T07:24:10.659581+0000","last_unstale":"2024-03-06T07:24:10.660926+0000","last_undegraded":"2024-03-06T07:24:10.660926+0000","last_fullsized":"2024-03-06T07:24:10.660926+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:39.589222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648969+0000","last_change":"2024-03-06T07:24:10.648969+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648969+0000","last_undegraded":"2024-03-06T07:24:10.648969+0000","last_fullsized":"2024-03-06T07:24:10.648969+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666485+0000","last_change":"2024-03-06T07:24:10.662366+0000","last_active":"2024-03-06T07:24:10.666485+0000","last_peered":"2024-03-06T07:24:10.666485+0000","last_clean":"2024-03-06T07:24:10.666485+0000","last_became_active":"2024-03-06T07:24:10.662075+0000","last_became_peered":"2024-03-06T07:24:10.662075+0000","last_unstale":"2024-03-06T07:24:10.666485+0000","last_undegraded":"2024-03-06T07:24:10.666485+0000","last_fullsized":"2024-03-06T07:24:10.666485+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.249754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649490+0000","last_change":"2024-03-06T07:24:10.649490+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649490+0000","last_undegraded":"2024-03-06T07:24:10.649490+0000","last_fullsized":"2024-03-06T07:24:10.649490+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660644+0000","last_change":"2024-03-06T07:24:10.659614+0000","last_active":"2024-03-06T07:24:10.660644+0000","last_peered":"2024-03-06T07:24:10.660644+0000","last_clean":"2024-03-06T07:24:10.660644+0000","last_became_active":"2024-03-06T07:24:10.659301+0000","last_became_peered":"2024-03-06T07:24:10.659301+0000","last_unstale":"2024-03-06T07:24:10.660644+0000","last_undegraded":"2024-03-06T07:24:10.660644+0000","last_fullsized":"2024-03-06T07:24:10.660644+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652779+0000","last_change":"2024-03-06T07:24:10.652779+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652779+0000","last_undegraded":"2024-03-06T07:24:10.652779+0000","last_fullsized":"2024-03-06T07:24:10.652779+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663858+0000","last_change":"2024-03-06T07:24:10.661362+0000","last_active":"2024-03-06T07:24:10.663858+0000","last_peered":"2024-03-06T07:24:10.663858+0000","last_clean":"2024-03-06T07:24:10.663858+0000","last_became_active":"2024-03-06T07:24:10.661069+0000","last_became_peered":"2024-03-06T07:24:10.661069+0000","last_unstale":"2024-03-06T07:24:10.663858+0000","last_undegraded":"2024-03-06T07:24:10.663858+0000","last_fullsized":"2024-03-06T07:24:10.663858+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:47.652320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654658+0000","last_change":"2024-03-06T07:24:10.654658+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654658+0000","last_undegraded":"2024-03-06T07:24:10.654658+0000","last_fullsized":"2024-03-06T07:24:10.654658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.655589+0000","last_change":"2024-03-06T07:24:10.653845+0000","last_active":"2024-03-06T07:24:10.655589+0000","last_peered":"2024-03-06T07:24:10.655589+0000","last_clean":"2024-03-06T07:24:10.655589+0000","last_became_active":"2024-03-06T07:24:10.653600+0000","last_became_peered":"2024-03-06T07:24:10.653600+0000","last_unstale":"2024-03-06T07:24:10.655589+0000","last_undegraded":"2024-03-06T07:24:10.655589+0000","last_fullsized":"2024-03-06T07:24:10.655589+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:05.337286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.646559+0000","last_change":"2024-03-06T07:24:10.646559+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.646559+0000","last_undegraded":"2024-03-06T07:24:10.646559+0000","last_fullsized":"2024-03-06T07:24:10.646559+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663129+0000","last_change":"2024-03-06T07:24:10.660990+0000","last_active":"2024-03-06T07:24:10.663129+0000","last_peered":"2024-03-06T07:24:10.663129+0000","last_clean":"2024-03-06T07:24:10.663129+0000","last_became_active":"2024-03-06T07:24:10.660680+0000","last_became_peered":"2024-03-06T07:24:10.660680+0000","last_unstale":"2024-03-06T07:24:10.663129+0000","last_undegraded":"2024-03-06T07:24:10.663129+0000","last_fullsized":"2024-03-06T07:24:10.663129+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.249754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652835+0000","last_change":"2024-03-06T07:24:10.652835+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652835+0000","last_undegraded":"2024-03-06T07:24:10.652835+0000","last_fullsized":"2024-03-06T07:24:10.652835+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660821+0000","last_change":"2024-03-06T07:24:10.659499+0000","last_active":"2024-03-06T07:24:10.660821+0000","last_peered":"2024-03-06T07:24:10.660821+0000","last_clean":"2024-03-06T07:24:10.660821+0000","last_became_active":"2024-03-06T07:24:10.659224+0000","last_became_peered":"2024-03-06T07:24:10.659224+0000","last_unstale":"2024-03-06T07:24:10.660821+0000","last_undegraded":"2024-03-06T07:24:10.660821+0000","last_fullsized":"2024-03-06T07:24:10.660821+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:37.419239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654768+0000","last_change":"2024-03-06T07:24:10.654768+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654768+0000","last_undegraded":"2024-03-06T07:24:10.654768+0000","last_fullsized":"2024-03-06T07:24:10.654768+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664029+0000","last_change":"2024-03-06T07:24:10.662290+0000","last_active":"2024-03-06T07:24:10.664029+0000","last_peered":"2024-03-06T07:24:10.664029+0000","last_clean":"2024-03-06T07:24:10.664029+0000","last_became_active":"2024-03-06T07:24:10.662003+0000","last_became_peered":"2024-03-06T07:24:10.662003+0000","last_unstale":"2024-03-06T07:24:10.664029+0000","last_undegraded":"2024-03-06T07:24:10.664029+0000","last_fullsized":"2024-03-06T07:24:10.664029+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652707+0000","last_change":"2024-03-06T07:24:10.652707+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652707+0000","last_undegraded":"2024-03-06T07:24:10.652707+0000","last_fullsized":"2024-03-06T07:24:10.652707+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666379+0000","last_change":"2024-03-06T07:24:10.663647+0000","last_active":"2024-03-06T07:24:10.666379+0000","last_peered":"2024-03-06T07:24:10.666379+0000","last_clean":"2024-03-06T07:24:10.666379+0000","last_became_active":"2024-03-06T07:24:10.663378+0000","last_became_peered":"2024-03-06T07:24:10.663378+0000","last_unstale":"2024-03-06T07:24:10.666379+0000","last_undegraded":"2024-03-06T07:24:10.666379+0000","last_fullsized":"2024-03-06T07:24:10.666379+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:21.084766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648169+0000","last_change":"2024-03-06T07:24:10.648169+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648169+0000","last_undegraded":"2024-03-06T07:24:10.648169+0000","last_fullsized":"2024-03-06T07:24:10.648169+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.668926+0000","last_change":"2024-03-06T07:24:10.665110+0000","last_active":"2024-03-06T07:24:10.668926+0000","last_peered":"2024-03-06T07:24:10.668926+0000","last_clean":"2024-03-06T07:24:10.668926+0000","last_became_active":"2024-03-06T07:24:10.664833+0000","last_became_peered":"2024-03-06T07:24:10.664833+0000","last_unstale":"2024-03-06T07:24:10.668926+0000","last_undegraded":"2024-03-06T07:24:10.668926+0000","last_fullsized":"2024-03-06T07:24:10.668926+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653401+0000","last_change":"2024-03-06T07:24:10.653401+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653401+0000","last_undegraded":"2024-03-06T07:24:10.653401+0000","last_fullsized":"2024-03-06T07:24:10.653401+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664163+0000","last_change":"2024-03-06T07:24:10.662566+0000","last_active":"2024-03-06T07:24:10.664163+0000","last_peered":"2024-03-06T07:24:10.664163+0000","last_clean":"2024-03-06T07:24:10.664163+0000","last_became_active":"2024-03-06T07:24:10.662201+0000","last_became_peered":"2024-03-06T07:24:10.662201+0000","last_unstale":"2024-03-06T07:24:10.664163+0000","last_undegraded":"2024-03-06T07:24:10.664163+0000","last_fullsized":"2024-03-06T07:24:10.664163+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:00.568465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648596+0000","last_change":"2024-03-06T07:24:10.648596+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648596+0000","last_undegraded":"2024-03-06T07:24:10.648596+0000","last_fullsized":"2024-03-06T07:24:10.648596+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666591+0000","last_change":"2024-03-06T07:24:10.664502+0000","last_active":"2024-03-06T07:24:10.666591+0000","last_peered":"2024-03-06T07:24:10.666591+0000","last_clean":"2024-03-06T07:24:10.666591+0000","last_became_active":"2024-03-06T07:24:10.664248+0000","last_became_peered":"2024-03-06T07:24:10.664248+0000","last_unstale":"2024-03-06T07:24:10.666591+0000","last_undegraded":"2024-03-06T07:24:10.666591+0000","last_fullsized":"2024-03-06T07:24:10.666591+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:07.375756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652181+0000","last_change":"2024-03-06T07:24:10.652181+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652181+0000","last_undegraded":"2024-03-06T07:24:10.652181+0000","last_fullsized":"2024-03-06T07:24:10.652181+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658397+0000","last_change":"2024-03-06T07:24:10.655420+0000","last_active":"2024-03-06T07:24:10.658397+0000","last_peered":"2024-03-06T07:24:10.658397+0000","last_clean":"2024-03-06T07:24:10.658397+0000","last_became_active":"2024-03-06T07:24:10.655148+0000","last_became_peered":"2024-03-06T07:24:10.655148+0000","last_unstale":"2024-03-06T07:24:10.658397+0000","last_undegraded":"2024-03-06T07:24:10.658397+0000","last_fullsized":"2024-03-06T07:24:10.658397+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:16.948447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652559+0000","last_change":"2024-03-06T07:24:10.652559+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652559+0000","last_undegraded":"2024-03-06T07:24:10.652559+0000","last_fullsized":"2024-03-06T07:24:10.652559+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661144+0000","last_change":"2024-03-06T07:24:10.659555+0000","last_active":"2024-03-06T07:24:10.661144+0000","last_peered":"2024-03-06T07:24:10.661144+0000","last_clean":"2024-03-06T07:24:10.661144+0000","last_became_active":"2024-03-06T07:24:10.659285+0000","last_became_peered":"2024-03-06T07:24:10.659285+0000","last_unstale":"2024-03-06T07:24:10.661144+0000","last_undegraded":"2024-03-06T07:24:10.661144+0000","last_fullsized":"2024-03-06T07:24:10.661144+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651408+0000","last_change":"2024-03-06T07:24:10.651408+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651408+0000","last_undegraded":"2024-03-06T07:24:10.651408+0000","last_fullsized":"2024-03-06T07:24:10.651408+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664001+0000","last_change":"2024-03-06T07:24:10.661441+0000","last_active":"2024-03-06T07:24:10.664001+0000","last_peered":"2024-03-06T07:24:10.664001+0000","last_clean":"2024-03-06T07:24:10.664001+0000","last_became_active":"2024-03-06T07:24:10.661157+0000","last_became_peered":"2024-03-06T07:24:10.661157+0000","last_unstale":"2024-03-06T07:24:10.664001+0000","last_undegraded":"2024-03-06T07:24:10.664001+0000","last_fullsized":"2024-03-06T07:24:10.664001+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652018+0000","last_change":"2024-03-06T07:24:10.652018+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652018+0000","last_undegraded":"2024-03-06T07:24:10.652018+0000","last_fullsized":"2024-03-06T07:24:10.652018+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.659358+0000","last_change":"2024-03-06T07:24:10.658071+0000","last_active":"2024-03-06T07:24:10.659358+0000","last_peered":"2024-03-06T07:24:10.659358+0000","last_clean":"2024-03-06T07:24:10.659358+0000","last_became_active":"2024-03-06T07:24:10.657790+0000","last_became_peered":"2024-03-06T07:24:10.657790+0000","last_unstale":"2024-03-06T07:24:10.659358+0000","last_undegraded":"2024-03-06T07:24:10.659358+0000","last_fullsized":"2024-03-06T07:24:10.659358+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:33.615309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649490+0000","last_change":"2024-03-06T07:24:10.649490+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649490+0000","last_undegraded":"2024-03-06T07:24:10.649490+0000","last_fullsized":"2024-03-06T07:24:10.649490+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661030+0000","last_change":"2024-03-06T07:24:10.658700+0000","last_active":"2024-03-06T07:24:10.661030+0000","last_peered":"2024-03-06T07:24:10.661030+0000","last_clean":"2024-03-06T07:24:10.661030+0000","last_became_active":"2024-03-06T07:24:10.658430+0000","last_became_peered":"2024-03-06T07:24:10.658430+0000","last_unstale":"2024-03-06T07:24:10.661030+0000","last_undegraded":"2024-03-06T07:24:10.661030+0000","last_fullsized":"2024-03-06T07:24:10.661030+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654047+0000","last_change":"2024-03-06T07:24:10.654047+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654047+0000","last_undegraded":"2024-03-06T07:24:10.654047+0000","last_fullsized":"2024-03-06T07:24:10.654047+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661932+0000","last_change":"2024-03-06T07:24:10.659862+0000","last_active":"2024-03-06T07:24:10.661932+0000","last_peered":"2024-03-06T07:24:10.661932+0000","last_clean":"2024-03-06T07:24:10.661932+0000","last_became_active":"2024-03-06T07:24:10.659578+0000","last_became_peered":"2024-03-06T07:24:10.659578+0000","last_unstale":"2024-03-06T07:24:10.661932+0000","last_undegraded":"2024-03-06T07:24:10.661932+0000","last_fullsized":"2024-03-06T07:24:10.661932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652981+0000","last_change":"2024-03-06T07:24:10.652981+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652981+0000","last_undegraded":"2024-03-06T07:24:10.652981+0000","last_fullsized":"2024-03-06T07:24:10.652981+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.659947+0000","last_change":"2024-03-06T07:24:10.658442+0000","last_active":"2024-03-06T07:24:10.659947+0000","last_peered":"2024-03-06T07:24:10.659947+0000","last_clean":"2024-03-06T07:24:10.659947+0000","last_became_active":"2024-03-06T07:24:10.658200+0000","last_became_peered":"2024-03-06T07:24:10.658200+0000","last_unstale":"2024-03-06T07:24:10.659947+0000","last_undegraded":"2024-03-06T07:24:10.659947+0000","last_fullsized":"2024-03-06T07:24:10.659947+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647966+0000","last_change":"2024-03-06T07:24:10.647966+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647966+0000","last_undegraded":"2024-03-06T07:24:10.647966+0000","last_fullsized":"2024-03-06T07:24:10.647966+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666422+0000","last_change":"2024-03-06T07:24:10.662365+0000","last_active":"2024-03-06T07:24:10.666422+0000","last_peered":"2024-03-06T07:24:10.666422+0000","last_clean":"2024-03-06T07:24:10.666422+0000","last_became_active":"2024-03-06T07:24:10.662073+0000","last_became_peered":"2024-03-06T07:24:10.662073+0000","last_unstale":"2024-03-06T07:24:10.666422+0000","last_undegraded":"2024-03-06T07:24:10.666422+0000","last_fullsized":"2024-03-06T07:24:10.666422+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:59.892449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647790+0000","last_change":"2024-03-06T07:24:10.647790+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647790+0000","last_undegraded":"2024-03-06T07:24:10.647790+0000","last_fullsized":"2024-03-06T07:24:10.647790+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660165+0000","last_change":"2024-03-06T07:24:10.659174+0000","last_active":"2024-03-06T07:24:10.660165+0000","last_peered":"2024-03-06T07:24:10.660165+0000","last_clean":"2024-03-06T07:24:10.660165+0000","last_became_active":"2024-03-06T07:24:10.658834+0000","last_became_peered":"2024-03-06T07:24:10.658834+0000","last_unstale":"2024-03-06T07:24:10.660165+0000","last_undegraded":"2024-03-06T07:24:10.660165+0000","last_fullsized":"2024-03-06T07:24:10.660165+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:37.876251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648001+0000","last_change":"2024-03-06T07:24:10.648001+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648001+0000","last_undegraded":"2024-03-06T07:24:10.648001+0000","last_fullsized":"2024-03-06T07:24:10.648001+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657498+0000","last_change":"2024-03-06T07:24:10.656628+0000","last_active":"2024-03-06T07:24:10.657498+0000","last_peered":"2024-03-06T07:24:10.657498+0000","last_clean":"2024-03-06T07:24:10.657498+0000","last_became_active":"2024-03-06T07:24:10.656271+0000","last_became_peered":"2024-03-06T07:24:10.656271+0000","last_unstale":"2024-03-06T07:24:10.657498+0000","last_undegraded":"2024-03-06T07:24:10.657498+0000","last_fullsized":"2024-03-06T07:24:10.657498+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:55.005932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648025+0000","last_change":"2024-03-06T07:24:10.648025+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648025+0000","last_undegraded":"2024-03-06T07:24:10.648025+0000","last_fullsized":"2024-03-06T07:24:10.648025+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664836+0000","last_change":"2024-03-06T07:24:10.663013+0000","last_active":"2024-03-06T07:24:10.664836+0000","last_peered":"2024-03-06T07:24:10.664836+0000","last_clean":"2024-03-06T07:24:10.664836+0000","last_became_active":"2024-03-06T07:24:10.662736+0000","last_became_peered":"2024-03-06T07:24:10.662736+0000","last_unstale":"2024-03-06T07:24:10.664836+0000","last_undegraded":"2024-03-06T07:24:10.664836+0000","last_fullsized":"2024-03-06T07:24:10.664836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:34.328658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648442+0000","last_change":"2024-03-06T07:24:10.648442+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648442+0000","last_undegraded":"2024-03-06T07:24:10.648442+0000","last_fullsized":"2024-03-06T07:24:10.648442+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658858+0000","last_change":"2024-03-06T07:24:10.654750+0000","last_active":"2024-03-06T07:24:10.658858+0000","last_peered":"2024-03-06T07:24:10.658858+0000","last_clean":"2024-03-06T07:24:10.658858+0000","last_became_active":"2024-03-06T07:24:10.654476+0000","last_became_peered":"2024-03-06T07:24:10.654476+0000","last_unstale":"2024-03-06T07:24:10.658858+0000","last_undegraded":"2024-03-06T07:24:10.658858+0000","last_fullsized":"2024-03-06T07:24:10.658858+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:55.005932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647211+0000","last_change":"2024-03-06T07:24:10.647211+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647211+0000","last_undegraded":"2024-03-06T07:24:10.647211+0000","last_fullsized":"2024-03-06T07:24:10.647211+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658862+0000","last_change":"2024-03-06T07:24:10.657981+0000","last_active":"2024-03-06T07:24:10.658862+0000","last_peered":"2024-03-06T07:24:10.658862+0000","last_clean":"2024-03-06T07:24:10.658862+0000","last_became_active":"2024-03-06T07:24:10.657616+0000","last_became_peered":"2024-03-06T07:24:10.657616+0000","last_unstale":"2024-03-06T07:24:10.658862+0000","last_undegraded":"2024-03-06T07:24:10.658862+0000","last_fullsized":"2024-03-06T07:24:10.658862+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:45.308028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649057+0000","last_change":"2024-03-06T07:24:10.649057+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649057+0000","last_undegraded":"2024-03-06T07:24:10.649057+0000","last_fullsized":"2024-03-06T07:24:10.649057+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663759+0000","last_change":"2024-03-06T07:24:10.661755+0000","last_active":"2024-03-06T07:24:10.663759+0000","last_peered":"2024-03-06T07:24:10.663759+0000","last_clean":"2024-03-06T07:24:10.663759+0000","last_became_active":"2024-03-06T07:24:10.661432+0000","last_became_peered":"2024-03-06T07:24:10.661432+0000","last_unstale":"2024-03-06T07:24:10.663759+0000","last_undegraded":"2024-03-06T07:24:10.663759+0000","last_fullsized":"2024-03-06T07:24:10.663759+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:15.720920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652228+0000","last_change":"2024-03-06T07:24:10.652228+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652228+0000","last_undegraded":"2024-03-06T07:24:10.652228+0000","last_fullsized":"2024-03-06T07:24:10.652228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657656+0000","last_change":"2024-03-06T07:24:10.656363+0000","last_active":"2024-03-06T07:24:10.657656+0000","last_peered":"2024-03-06T07:24:10.657656+0000","last_clean":"2024-03-06T07:24:10.657656+0000","last_became_active":"2024-03-06T07:24:10.656056+0000","last_became_peered":"2024-03-06T07:24:10.656056+0000","last_unstale":"2024-03-06T07:24:10.657656+0000","last_undegraded":"2024-03-06T07:24:10.657656+0000","last_fullsized":"2024-03-06T07:24:10.657656+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:19.463965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651712+0000","last_change":"2024-03-06T07:24:10.651712+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651712+0000","last_undegraded":"2024-03-06T07:24:10.651712+0000","last_fullsized":"2024-03-06T07:24:10.651712+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664178+0000","last_change":"2024-03-06T07:24:10.662788+0000","last_active":"2024-03-06T07:24:10.664178+0000","last_peered":"2024-03-06T07:24:10.664178+0000","last_clean":"2024-03-06T07:24:10.664178+0000","last_became_active":"2024-03-06T07:24:10.662446+0000","last_became_peered":"2024-03-06T07:24:10.662446+0000","last_unstale":"2024-03-06T07:24:10.664178+0000","last_undegraded":"2024-03-06T07:24:10.664178+0000","last_fullsized":"2024-03-06T07:24:10.664178+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:25.045708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647210+0000","last_change":"2024-03-06T07:24:10.647210+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647210+0000","last_undegraded":"2024-03-06T07:24:10.647210+0000","last_fullsized":"2024-03-06T07:24:10.647210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658501+0000","last_change":"2024-03-06T07:24:10.649819+0000","last_active":"2024-03-06T07:24:10.658501+0000","last_peered":"2024-03-06T07:24:10.658501+0000","last_clean":"2024-03-06T07:24:10.658501+0000","last_became_active":"2024-03-06T07:24:10.649508+0000","last_became_peered":"2024-03-06T07:24:10.649508+0000","last_unstale":"2024-03-06T07:24:10.658501+0000","last_undegraded":"2024-03-06T07:24:10.658501+0000","last_fullsized":"2024-03-06T07:24:10.658501+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:25.045708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652286+0000","last_change":"2024-03-06T07:24:10.652286+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652286+0000","last_undegraded":"2024-03-06T07:24:10.652286+0000","last_fullsized":"2024-03-06T07:24:10.652286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657611+0000","last_change":"2024-03-06T07:24:10.656290+0000","last_active":"2024-03-06T07:24:10.657611+0000","last_peered":"2024-03-06T07:24:10.657611+0000","last_clean":"2024-03-06T07:24:10.657611+0000","last_became_active":"2024-03-06T07:24:10.656023+0000","last_became_peered":"2024-03-06T07:24:10.656023+0000","last_unstale":"2024-03-06T07:24:10.657611+0000","last_undegraded":"2024-03-06T07:24:10.657611+0000","last_fullsized":"2024-03-06T07:24:10.657611+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:35.493530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651834+0000","last_change":"2024-03-06T07:24:10.651834+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651834+0000","last_undegraded":"2024-03-06T07:24:10.651834+0000","last_fullsized":"2024-03-06T07:24:10.651834+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.652350+0000","last_change":"2024-03-06T07:24:10.650351+0000","last_active":"2024-03-06T07:24:10.652350+0000","last_peered":"2024-03-06T07:24:10.652350+0000","last_clean":"2024-03-06T07:24:10.652350+0000","last_became_active":"2024-03-06T07:24:10.649966+0000","last_became_peered":"2024-03-06T07:24:10.649966+0000","last_unstale":"2024-03-06T07:24:10.652350+0000","last_undegraded":"2024-03-06T07:24:10.652350+0000","last_fullsized":"2024-03-06T07:24:10.652350+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:20.586025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":238,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.653875+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:24:10.653875+0000","last_peered":"2024-03-06T07:24:10.653875+0000","last_clean":"2024-03-06T07:24:10.653875+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:24:10.653875+0000","last_undegraded":"2024-03-06T07:24:10.653875+0000","last_fullsized":"2024-03-06T07:24:10.653875+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1265664,"data_stored":866798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":671744,"data_stored":276430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":671744,"data_stored":276430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1265664,"data_stored":866798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:14.784 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:14.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.919+0000 7f3fb9a71640 1 Processor -- start 2024-03-06T07:24:14.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.919+0000 7f3fb9a71640 1 -- start start 2024-03-06T07:24:14.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.919+0000 7f3fb9a71640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3fb415bb20 0x7f3fb40d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.919+0000 7f3fb9a71640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb4161980 con 0x7f3fb40cfff0 2024-03-06T07:24:14.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.919+0000 7f3fb9a71640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb4161c60 con 0x7f3fb40cd8c0 2024-03-06T07:24:14.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.919+0000 7f3fb9a71640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fb4161f40 con 0x7f3fb415bb20 2024-03-06T07:24:14.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb2ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3fb40cfff0 0x7f3fb40ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37674/0 (socket says 172.21.15.59:37674) 2024-03-06T07:24:14.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb2ffd640 1 -- 172.21.15.59:0/3907559383 learned_addr learned my addr 172.21.15.59:0/3907559383 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:14.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb37fe640 1 --2- 172.21.15.59:0/3907559383 >> v2:172.21.15.164:3300/0 conn(0x7f3fb415bb20 0x7f3fb40d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003307392 0 0) 0x7f3fb4161980 con 0x7f3fb40cfff0 2024-03-06T07:24:14.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f9c003520 con 0x7f3fb40cfff0 2024-03-06T07:24:14.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 444038483 0 0) 0x7f3f9c003520 con 0x7f3fb40cfff0 2024-03-06T07:24:14.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 >> v2:172.21.15.164:3300/0 conn(0x7f3fb415bb20 msgr2=0x7f3fb40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 --2- 172.21.15.59:0/3907559383 >> v2:172.21.15.164:3300/0 conn(0x7f3fb415bb20 0x7f3fb40d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 >> v1:172.21.15.164:6789/0 conn(0x7f3fb40cd8c0 legacy=0x7f3fb40cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb4162220 con 0x7f3fb40cfff0 2024-03-06T07:24:14.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f3fa8001670 con 0x7f3fb40cfff0 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 >> v1:172.21.15.59:6789/0 conn(0x7f3fb40cfff0 legacy=0x7f3fb40ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f9c0039b0 0x7f3f9c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9c004800 0x7f3f9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9c0051c0 0x7f3f9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb4161f40 con 0x7f3f9c0039b0 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c00c060 con 0x7f3f9c004800 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c00c260 con 0x7f3f9c0051c0 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb37fe640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f9c0039b0 0x7f3f9c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.920+0000 7f3fb2ffd640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9c004800 0x7f3f9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.921+0000 7f3fb3fff640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9c0051c0 0x7f3f9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.921+0000 7f3fb37fe640 1 -- 172.21.15.59:0/3907559383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9c0051c0 msgr2=0x7f3f9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.921+0000 7f3fb37fe640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9c0051c0 0x7f3f9c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.921+0000 7f3fb37fe640 1 -- 172.21.15.59:0/3907559383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9c004800 msgr2=0x7f3f9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.921+0000 7f3fb37fe640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9c004800 0x7f3f9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.921+0000 7f3fb37fe640 1 -- 172.21.15.59:0/3907559383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c003520 con 0x7f3f9c0039b0 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.921+0000 7f3fb3fff640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9c0051c0 0x7f3f9c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.921+0000 7f3fb2ffd640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9c004800 0x7f3f9c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.921+0000 7f3fb3fff640 1 -- 172.21.15.59:0/3907559383 reap_dead start 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.921+0000 7f3fb37fe640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f9c0039b0 0x7f3f9c003dd0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009ac0 tx=0x7f3f9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838f44235749ed9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.922+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4039070 con 0x7f3f9c0039b0 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.922+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa402c6b0 con 0x7f3f9c0039b0 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.922+0000 7f3fb27fc640 1 -- 172.21.15.59:0/3907559383 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4009690 con 0x7f3f9c0039b0 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.922+0000 7f3fb9a71640 1 -- 172.21.15.59:0/3907559383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f9c0039b0 msgr2=0x7f3f9c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.922+0000 7f3fb9a71640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f9c0039b0 0x7f3f9c003dd0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009ac0 tx=0x7f3f9c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.922+0000 7f3fb9a71640 1 -- 172.21.15.59:0/3907559383 shutdown_connections 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.922+0000 7f3fb9a71640 1 --2- 172.21.15.59:0/3907559383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f9c0039b0 0x7f3f9c003dd0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.922+0000 7f3fb9a71640 1 -- 172.21.15.59:0/3907559383 >> 172.21.15.59:0/3907559383 conn(0x7f3fb409a9f0 msgr2=0x7f3fb409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.922+0000 7f3fb9a71640 1 -- 172.21.15.59:0/3907559383 shutdown_connections 2024-03-06T07:24:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.922+0000 7f3fb9a71640 1 -- 172.21.15.59:0/3907559383 wait complete. 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.923+0000 7f3fb9a71640 1 Processor -- start 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.923+0000 7f3fb9a71640 1 -- start start 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.924+0000 7f3fb9a71640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9c0039b0 0x7f3fb4156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.924+0000 7f3fb9a71640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fb4157260 0x7f3fb40b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.924+0000 7f3fb9a71640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb40b6f00 0x7f3fb406be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.924+0000 7f3fb9a71640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb40d1b00 con 0x7f3fb40b6f00 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.924+0000 7f3fb9a71640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb40d1980 con 0x7f3f9c0039b0 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.924+0000 7f3fb9a71640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb40d04a0 con 0x7f3fb4157260 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.924+0000 7f3fb3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb40b6f00 0x7f3fb406be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.924+0000 7f3fb3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb40b6f00 0x7f3fb406be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50916/0 (socket says 172.21.15.59:50916) 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.924+0000 7f3fb3fff640 1 -- 172.21.15.59:0/2837092785 learned_addr learned my addr 172.21.15.59:0/2837092785 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.924+0000 7f3fb2ffd640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fb4157260 0x7f3fb40b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.924+0000 7f3fb37fe640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9c0039b0 0x7f3fb4156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.925+0000 7f3fb3fff640 1 -- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fb4157260 msgr2=0x7f3fb40b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.925+0000 7f3fb3fff640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fb4157260 0x7f3fb40b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.925+0000 7f3fb3fff640 1 -- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9c0039b0 msgr2=0x7f3fb4156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.925+0000 7f3fb3fff640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9c0039b0 0x7f3fb4156560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.925+0000 7f3fb3fff640 1 -- 172.21.15.59:0/2837092785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb40c8370 con 0x7f3fb40b6f00 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.925+0000 7f3fb3fff640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb40b6f00 0x7f3fb406be60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f3fb406bd80 tx=0x7f3fac00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.925+0000 7f3f9bfff640 1 -- 172.21.15.59:0/2837092785 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac003970 con 0x7f3fb40b6f00 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.925+0000 7f3f9bfff640 1 -- 172.21.15.59:0/2837092785 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac018070 con 0x7f3fb40b6f00 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.925+0000 7f3fb9a71640 1 -- 172.21.15.59:0/2837092785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb40b5ee0 con 0x7f3fb40b6f00 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.925+0000 7f3f9bfff640 1 -- 172.21.15.59:0/2837092785 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac003d50 con 0x7f3fb40b6f00 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.925+0000 7f3fb9a71640 1 -- 172.21.15.59:0/2837092785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb4156aa0 con 0x7f3fb40b6f00 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.926+0000 7f3fb9a71640 1 -- 172.21.15.59:0/2837092785 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb404f330 con 0x7f3fb40b6f00 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.926+0000 7f3f9bfff640 1 -- 172.21.15.59:0/2837092785 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fac0050d0 con 0x7f3fb40b6f00 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.927+0000 7f3f9bfff640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3fa006b500 0x7f3fa006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.927+0000 7f3f9bfff640 1 -- 172.21.15.59:0/2837092785 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fac089a50 con 0x7f3fb40b6f00 2024-03-06T07:24:14.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.927+0000 7f3fb37fe640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3fa006b500 0x7f3fa006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:14.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.928+0000 7f3fb37fe640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3fa006b500 0x7f3fa006d9c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4036000 tx=0x7f3fa4030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:14.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:14.931+0000 7f3f9bfff640 1 -- 172.21.15.59:0/2837092785 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fac022020 con 0x7f3fb40b6f00 2024-03-06T07:24:15.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.120+0000 7f3fb9a71640 1 -- 172.21.15.59:0/2837092785 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3fb415a540 con 0x7f3fa006b500 2024-03-06T07:24:15.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.126+0000 7f3f9bfff640 1 -- 172.21.15.59:0/2837092785 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246638 (secure 0 0 0) 0x7f3fb415a540 con 0x7f3fa006b500 2024-03-06T07:24:15.130 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:15.131 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 -- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3fa006b500 msgr2=0x7f3fa006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3fa006b500 0x7f3fa006d9c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4036000 tx=0x7f3fa4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 -- 172.21.15.59:0/2837092785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb40b6f00 msgr2=0x7f3fb406be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb40b6f00 0x7f3fb406be60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f3fb406bd80 tx=0x7f3fac00eb90 comp rx=0 tx=0).stop 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 -- 172.21.15.59:0/2837092785 shutdown_connections 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3fa006b500 0x7f3fa006d9c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb40b6f00 0x7f3fb406be60 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fb4157260 0x7f3fb40b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 --2- 172.21.15.59:0/2837092785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9c0039b0 0x7f3fb4156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 -- 172.21.15.59:0/2837092785 >> 172.21.15.59:0/2837092785 conn(0x7f3fb409a9f0 msgr2=0x7f3fb40b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 -- 172.21.15.59:0/2837092785 shutdown_connections 2024-03-06T07:24:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.129+0000 7f3fb9a71640 1 -- 172.21.15.59:0/2837092785 wait complete. 2024-03-06T07:24:15.141 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-03-06T07:24:13.329617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255916,"kb_used_data":6456,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718804,"statfs":{"total":773094113280,"available":772832055296,"internally_reserved":0,"allocated":6610944,"data_stored":3417244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000962"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653222+0000","last_change":"2024-03-06T07:24:10.653222+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653222+0000","last_undegraded":"2024-03-06T07:24:10.653222+0000","last_fullsized":"2024-03-06T07:24:10.653222+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654247+0000","last_change":"2024-03-06T07:24:10.654247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654247+0000","last_undegraded":"2024-03-06T07:24:10.654247+0000","last_fullsized":"2024-03-06T07:24:10.654247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653845+0000","last_change":"2024-03-06T07:24:10.653845+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653845+0000","last_undegraded":"2024-03-06T07:24:10.653845+0000","last_fullsized":"2024-03-06T07:24:10.653845+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649798+0000","last_change":"2024-03-06T07:24:10.649798+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649798+0000","last_undegraded":"2024-03-06T07:24:10.649798+0000","last_fullsized":"2024-03-06T07:24:10.649798+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653052+0000","last_change":"2024-03-06T07:24:10.653052+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653052+0000","last_undegraded":"2024-03-06T07:24:10.653052+0000","last_fullsized":"2024-03-06T07:24:10.653052+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649635+0000","last_change":"2024-03-06T07:24:10.649635+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649635+0000","last_undegraded":"2024-03-06T07:24:10.649635+0000","last_fullsized":"2024-03-06T07:24:10.649635+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654247+0000","last_change":"2024-03-06T07:24:10.654247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654247+0000","last_undegraded":"2024-03-06T07:24:10.654247+0000","last_fullsized":"2024-03-06T07:24:10.654247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653163+0000","last_change":"2024-03-06T07:24:10.653163+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653163+0000","last_undegraded":"2024-03-06T07:24:10.653163+0000","last_fullsized":"2024-03-06T07:24:10.653163+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.655335+0000","last_change":"2024-03-06T07:24:10.655335+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.655335+0000","last_undegraded":"2024-03-06T07:24:10.655335+0000","last_fullsized":"2024-03-06T07:24:10.655335+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648323+0000","last_change":"2024-03-06T07:24:10.648323+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648323+0000","last_undegraded":"2024-03-06T07:24:10.648323+0000","last_fullsized":"2024-03-06T07:24:10.648323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648785+0000","last_change":"2024-03-06T07:24:10.648785+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648785+0000","last_undegraded":"2024-03-06T07:24:10.648785+0000","last_fullsized":"2024-03-06T07:24:10.648785+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649164+0000","last_change":"2024-03-06T07:24:10.649164+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649164+0000","last_undegraded":"2024-03-06T07:24:10.649164+0000","last_fullsized":"2024-03-06T07:24:10.649164+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.646039+0000","last_change":"2024-03-06T07:24:10.646039+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.646039+0000","last_undegraded":"2024-03-06T07:24:10.646039+0000","last_fullsized":"2024-03-06T07:24:10.646039+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653571+0000","last_change":"2024-03-06T07:24:10.653571+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653571+0000","last_undegraded":"2024-03-06T07:24:10.653571+0000","last_fullsized":"2024-03-06T07:24:10.653571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653868+0000","last_change":"2024-03-06T07:24:10.653868+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653868+0000","last_undegraded":"2024-03-06T07:24:10.653868+0000","last_fullsized":"2024-03-06T07:24:10.653868+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.650079+0000","last_change":"2024-03-06T07:24:10.650079+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.650079+0000","last_undegraded":"2024-03-06T07:24:10.650079+0000","last_fullsized":"2024-03-06T07:24:10.650079+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652629+0000","last_change":"2024-03-06T07:24:10.652629+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652629+0000","last_undegraded":"2024-03-06T07:24:10.652629+0000","last_fullsized":"2024-03-06T07:24:10.652629+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653025+0000","last_change":"2024-03-06T07:24:10.653025+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653025+0000","last_undegraded":"2024-03-06T07:24:10.653025+0000","last_fullsized":"2024-03-06T07:24:10.653025+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653633+0000","last_change":"2024-03-06T07:24:10.653633+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653633+0000","last_undegraded":"2024-03-06T07:24:10.653633+0000","last_fullsized":"2024-03-06T07:24:10.653633+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.655332+0000","last_change":"2024-03-06T07:24:10.655332+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.655332+0000","last_undegraded":"2024-03-06T07:24:10.655332+0000","last_fullsized":"2024-03-06T07:24:10.655332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652655+0000","last_change":"2024-03-06T07:24:10.652655+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652655+0000","last_undegraded":"2024-03-06T07:24:10.652655+0000","last_fullsized":"2024-03-06T07:24:10.652655+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653247+0000","last_change":"2024-03-06T07:24:10.653247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653247+0000","last_undegraded":"2024-03-06T07:24:10.653247+0000","last_fullsized":"2024-03-06T07:24:10.653247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651973+0000","last_change":"2024-03-06T07:24:10.651973+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651973+0000","last_undegraded":"2024-03-06T07:24:10.651973+0000","last_fullsized":"2024-03-06T07:24:10.651973+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647134+0000","last_change":"2024-03-06T07:24:10.647134+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647134+0000","last_undegraded":"2024-03-06T07:24:10.647134+0000","last_fullsized":"2024-03-06T07:24:10.647134+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652823+0000","last_change":"2024-03-06T07:24:10.652823+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652823+0000","last_undegraded":"2024-03-06T07:24:10.652823+0000","last_fullsized":"2024-03-06T07:24:10.652823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649078+0000","last_change":"2024-03-06T07:24:10.649078+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649078+0000","last_undegraded":"2024-03-06T07:24:10.649078+0000","last_fullsized":"2024-03-06T07:24:10.649078+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649913+0000","last_change":"2024-03-06T07:24:10.649913+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649913+0000","last_undegraded":"2024-03-06T07:24:10.649913+0000","last_fullsized":"2024-03-06T07:24:10.649913+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653647+0000","last_change":"2024-03-06T07:24:10.653647+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653647+0000","last_undegraded":"2024-03-06T07:24:10.653647+0000","last_fullsized":"2024-03-06T07:24:10.653647+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653928+0000","last_change":"2024-03-06T07:24:10.653928+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653928+0000","last_undegraded":"2024-03-06T07:24:10.653928+0000","last_fullsized":"2024-03-06T07:24:10.653928+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651185+0000","last_change":"2024-03-06T07:24:10.651185+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651185+0000","last_undegraded":"2024-03-06T07:24:10.651185+0000","last_fullsized":"2024-03-06T07:24:10.651185+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651671+0000","last_change":"2024-03-06T07:24:10.651671+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651671+0000","last_undegraded":"2024-03-06T07:24:10.651671+0000","last_fullsized":"2024-03-06T07:24:10.651671+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647547+0000","last_change":"2024-03-06T07:24:10.647547+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647547+0000","last_undegraded":"2024-03-06T07:24:10.647547+0000","last_fullsized":"2024-03-06T07:24:10.647547+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648981+0000","last_change":"2024-03-06T07:24:10.648981+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648981+0000","last_undegraded":"2024-03-06T07:24:10.648981+0000","last_fullsized":"2024-03-06T07:24:10.648981+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661442+0000","last_change":"2024-03-06T07:24:10.658033+0000","last_active":"2024-03-06T07:24:10.661442+0000","last_peered":"2024-03-06T07:24:10.661442+0000","last_clean":"2024-03-06T07:24:10.661442+0000","last_became_active":"2024-03-06T07:24:10.657766+0000","last_became_peered":"2024-03-06T07:24:10.657766+0000","last_unstale":"2024-03-06T07:24:10.661442+0000","last_undegraded":"2024-03-06T07:24:10.661442+0000","last_fullsized":"2024-03-06T07:24:10.661442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648210+0000","last_change":"2024-03-06T07:24:10.648210+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648210+0000","last_undegraded":"2024-03-06T07:24:10.648210+0000","last_fullsized":"2024-03-06T07:24:10.648210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660926+0000","last_change":"2024-03-06T07:24:10.659965+0000","last_active":"2024-03-06T07:24:10.660926+0000","last_peered":"2024-03-06T07:24:10.660926+0000","last_clean":"2024-03-06T07:24:10.660926+0000","last_became_active":"2024-03-06T07:24:10.659581+0000","last_became_peered":"2024-03-06T07:24:10.659581+0000","last_unstale":"2024-03-06T07:24:10.660926+0000","last_undegraded":"2024-03-06T07:24:10.660926+0000","last_fullsized":"2024-03-06T07:24:10.660926+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:39.589222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648969+0000","last_change":"2024-03-06T07:24:10.648969+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648969+0000","last_undegraded":"2024-03-06T07:24:10.648969+0000","last_fullsized":"2024-03-06T07:24:10.648969+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666485+0000","last_change":"2024-03-06T07:24:10.662366+0000","last_active":"2024-03-06T07:24:10.666485+0000","last_peered":"2024-03-06T07:24:10.666485+0000","last_clean":"2024-03-06T07:24:10.666485+0000","last_became_active":"2024-03-06T07:24:10.662075+0000","last_became_peered":"2024-03-06T07:24:10.662075+0000","last_unstale":"2024-03-06T07:24:10.666485+0000","last_undegraded":"2024-03-06T07:24:10.666485+0000","last_fullsized":"2024-03-06T07:24:10.666485+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.249754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649490+0000","last_change":"2024-03-06T07:24:10.649490+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649490+0000","last_undegraded":"2024-03-06T07:24:10.649490+0000","last_fullsized":"2024-03-06T07:24:10.649490+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660644+0000","last_change":"2024-03-06T07:24:10.659614+0000","last_active":"2024-03-06T07:24:10.660644+0000","last_peered":"2024-03-06T07:24:10.660644+0000","last_clean":"2024-03-06T07:24:10.660644+0000","last_became_active":"2024-03-06T07:24:10.659301+0000","last_became_peered":"2024-03-06T07:24:10.659301+0000","last_unstale":"2024-03-06T07:24:10.660644+0000","last_undegraded":"2024-03-06T07:24:10.660644+0000","last_fullsized":"2024-03-06T07:24:10.660644+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652779+0000","last_change":"2024-03-06T07:24:10.652779+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652779+0000","last_undegraded":"2024-03-06T07:24:10.652779+0000","last_fullsized":"2024-03-06T07:24:10.652779+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663858+0000","last_change":"2024-03-06T07:24:10.661362+0000","last_active":"2024-03-06T07:24:10.663858+0000","last_peered":"2024-03-06T07:24:10.663858+0000","last_clean":"2024-03-06T07:24:10.663858+0000","last_became_active":"2024-03-06T07:24:10.661069+0000","last_became_peered":"2024-03-06T07:24:10.661069+0000","last_unstale":"2024-03-06T07:24:10.663858+0000","last_undegraded":"2024-03-06T07:24:10.663858+0000","last_fullsized":"2024-03-06T07:24:10.663858+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:47.652320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654658+0000","last_change":"2024-03-06T07:24:10.654658+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654658+0000","last_undegraded":"2024-03-06T07:24:10.654658+0000","last_fullsized":"2024-03-06T07:24:10.654658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.655589+0000","last_change":"2024-03-06T07:24:10.653845+0000","last_active":"2024-03-06T07:24:10.655589+0000","last_peered":"2024-03-06T07:24:10.655589+0000","last_clean":"2024-03-06T07:24:10.655589+0000","last_became_active":"2024-03-06T07:24:10.653600+0000","last_became_peered":"2024-03-06T07:24:10.653600+0000","last_unstale":"2024-03-06T07:24:10.655589+0000","last_undegraded":"2024-03-06T07:24:10.655589+0000","last_fullsized":"2024-03-06T07:24:10.655589+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:05.337286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.646559+0000","last_change":"2024-03-06T07:24:10.646559+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.646559+0000","last_undegraded":"2024-03-06T07:24:10.646559+0000","last_fullsized":"2024-03-06T07:24:10.646559+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663129+0000","last_change":"2024-03-06T07:24:10.660990+0000","last_active":"2024-03-06T07:24:10.663129+0000","last_peered":"2024-03-06T07:24:10.663129+0000","last_clean":"2024-03-06T07:24:10.663129+0000","last_became_active":"2024-03-06T07:24:10.660680+0000","last_became_peered":"2024-03-06T07:24:10.660680+0000","last_unstale":"2024-03-06T07:24:10.663129+0000","last_undegraded":"2024-03-06T07:24:10.663129+0000","last_fullsized":"2024-03-06T07:24:10.663129+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.249754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652835+0000","last_change":"2024-03-06T07:24:10.652835+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652835+0000","last_undegraded":"2024-03-06T07:24:10.652835+0000","last_fullsized":"2024-03-06T07:24:10.652835+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660821+0000","last_change":"2024-03-06T07:24:10.659499+0000","last_active":"2024-03-06T07:24:10.660821+0000","last_peered":"2024-03-06T07:24:10.660821+0000","last_clean":"2024-03-06T07:24:10.660821+0000","last_became_active":"2024-03-06T07:24:10.659224+0000","last_became_peered":"2024-03-06T07:24:10.659224+0000","last_unstale":"2024-03-06T07:24:10.660821+0000","last_undegraded":"2024-03-06T07:24:10.660821+0000","last_fullsized":"2024-03-06T07:24:10.660821+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:37.419239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654768+0000","last_change":"2024-03-06T07:24:10.654768+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654768+0000","last_undegraded":"2024-03-06T07:24:10.654768+0000","last_fullsized":"2024-03-06T07:24:10.654768+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664029+0000","last_change":"2024-03-06T07:24:10.662290+0000","last_active":"2024-03-06T07:24:10.664029+0000","last_peered":"2024-03-06T07:24:10.664029+0000","last_clean":"2024-03-06T07:24:10.664029+0000","last_became_active":"2024-03-06T07:24:10.662003+0000","last_became_peered":"2024-03-06T07:24:10.662003+0000","last_unstale":"2024-03-06T07:24:10.664029+0000","last_undegraded":"2024-03-06T07:24:10.664029+0000","last_fullsized":"2024-03-06T07:24:10.664029+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652707+0000","last_change":"2024-03-06T07:24:10.652707+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652707+0000","last_undegraded":"2024-03-06T07:24:10.652707+0000","last_fullsized":"2024-03-06T07:24:10.652707+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666379+0000","last_change":"2024-03-06T07:24:10.663647+0000","last_active":"2024-03-06T07:24:10.666379+0000","last_peered":"2024-03-06T07:24:10.666379+0000","last_clean":"2024-03-06T07:24:10.666379+0000","last_became_active":"2024-03-06T07:24:10.663378+0000","last_became_peered":"2024-03-06T07:24:10.663378+0000","last_unstale":"2024-03-06T07:24:10.666379+0000","last_undegraded":"2024-03-06T07:24:10.666379+0000","last_fullsized":"2024-03-06T07:24:10.666379+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:21.084766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648169+0000","last_change":"2024-03-06T07:24:10.648169+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648169+0000","last_undegraded":"2024-03-06T07:24:10.648169+0000","last_fullsized":"2024-03-06T07:24:10.648169+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.668926+0000","last_change":"2024-03-06T07:24:10.665110+0000","last_active":"2024-03-06T07:24:10.668926+0000","last_peered":"2024-03-06T07:24:10.668926+0000","last_clean":"2024-03-06T07:24:10.668926+0000","last_became_active":"2024-03-06T07:24:10.664833+0000","last_became_peered":"2024-03-06T07:24:10.664833+0000","last_unstale":"2024-03-06T07:24:10.668926+0000","last_undegraded":"2024-03-06T07:24:10.668926+0000","last_fullsized":"2024-03-06T07:24:10.668926+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653401+0000","last_change":"2024-03-06T07:24:10.653401+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653401+0000","last_undegraded":"2024-03-06T07:24:10.653401+0000","last_fullsized":"2024-03-06T07:24:10.653401+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664163+0000","last_change":"2024-03-06T07:24:10.662566+0000","last_active":"2024-03-06T07:24:10.664163+0000","last_peered":"2024-03-06T07:24:10.664163+0000","last_clean":"2024-03-06T07:24:10.664163+0000","last_became_active":"2024-03-06T07:24:10.662201+0000","last_became_peered":"2024-03-06T07:24:10.662201+0000","last_unstale":"2024-03-06T07:24:10.664163+0000","last_undegraded":"2024-03-06T07:24:10.664163+0000","last_fullsized":"2024-03-06T07:24:10.664163+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:00.568465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648596+0000","last_change":"2024-03-06T07:24:10.648596+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648596+0000","last_undegraded":"2024-03-06T07:24:10.648596+0000","last_fullsized":"2024-03-06T07:24:10.648596+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666591+0000","last_change":"2024-03-06T07:24:10.664502+0000","last_active":"2024-03-06T07:24:10.666591+0000","last_peered":"2024-03-06T07:24:10.666591+0000","last_clean":"2024-03-06T07:24:10.666591+0000","last_became_active":"2024-03-06T07:24:10.664248+0000","last_became_peered":"2024-03-06T07:24:10.664248+0000","last_unstale":"2024-03-06T07:24:10.666591+0000","last_undegraded":"2024-03-06T07:24:10.666591+0000","last_fullsized":"2024-03-06T07:24:10.666591+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:07.375756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652181+0000","last_change":"2024-03-06T07:24:10.652181+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652181+0000","last_undegraded":"2024-03-06T07:24:10.652181+0000","last_fullsized":"2024-03-06T07:24:10.652181+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658397+0000","last_change":"2024-03-06T07:24:10.655420+0000","last_active":"2024-03-06T07:24:10.658397+0000","last_peered":"2024-03-06T07:24:10.658397+0000","last_clean":"2024-03-06T07:24:10.658397+0000","last_became_active":"2024-03-06T07:24:10.655148+0000","last_became_peered":"2024-03-06T07:24:10.655148+0000","last_unstale":"2024-03-06T07:24:10.658397+0000","last_undegraded":"2024-03-06T07:24:10.658397+0000","last_fullsized":"2024-03-06T07:24:10.658397+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:16.948447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652559+0000","last_change":"2024-03-06T07:24:10.652559+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652559+0000","last_undegraded":"2024-03-06T07:24:10.652559+0000","last_fullsized":"2024-03-06T07:24:10.652559+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661144+0000","last_change":"2024-03-06T07:24:10.659555+0000","last_active":"2024-03-06T07:24:10.661144+0000","last_peered":"2024-03-06T07:24:10.661144+0000","last_clean":"2024-03-06T07:24:10.661144+0000","last_became_active":"2024-03-06T07:24:10.659285+0000","last_became_peered":"2024-03-06T07:24:10.659285+0000","last_unstale":"2024-03-06T07:24:10.661144+0000","last_undegraded":"2024-03-06T07:24:10.661144+0000","last_fullsized":"2024-03-06T07:24:10.661144+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651408+0000","last_change":"2024-03-06T07:24:10.651408+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651408+0000","last_undegraded":"2024-03-06T07:24:10.651408+0000","last_fullsized":"2024-03-06T07:24:10.651408+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664001+0000","last_change":"2024-03-06T07:24:10.661441+0000","last_active":"2024-03-06T07:24:10.664001+0000","last_peered":"2024-03-06T07:24:10.664001+0000","last_clean":"2024-03-06T07:24:10.664001+0000","last_became_active":"2024-03-06T07:24:10.661157+0000","last_became_peered":"2024-03-06T07:24:10.661157+0000","last_unstale":"2024-03-06T07:24:10.664001+0000","last_undegraded":"2024-03-06T07:24:10.664001+0000","last_fullsized":"2024-03-06T07:24:10.664001+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652018+0000","last_change":"2024-03-06T07:24:10.652018+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652018+0000","last_undegraded":"2024-03-06T07:24:10.652018+0000","last_fullsized":"2024-03-06T07:24:10.652018+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.659358+0000","last_change":"2024-03-06T07:24:10.658071+0000","last_active":"2024-03-06T07:24:10.659358+0000","last_peered":"2024-03-06T07:24:10.659358+0000","last_clean":"2024-03-06T07:24:10.659358+0000","last_became_active":"2024-03-06T07:24:10.657790+0000","last_became_peered":"2024-03-06T07:24:10.657790+0000","last_unstale":"2024-03-06T07:24:10.659358+0000","last_undegraded":"2024-03-06T07:24:10.659358+0000","last_fullsized":"2024-03-06T07:24:10.659358+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:33.615309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649490+0000","last_change":"2024-03-06T07:24:10.649490+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649490+0000","last_undegraded":"2024-03-06T07:24:10.649490+0000","last_fullsized":"2024-03-06T07:24:10.649490+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661030+0000","last_change":"2024-03-06T07:24:10.658700+0000","last_active":"2024-03-06T07:24:10.661030+0000","last_peered":"2024-03-06T07:24:10.661030+0000","last_clean":"2024-03-06T07:24:10.661030+0000","last_became_active":"2024-03-06T07:24:10.658430+0000","last_became_peered":"2024-03-06T07:24:10.658430+0000","last_unstale":"2024-03-06T07:24:10.661030+0000","last_undegraded":"2024-03-06T07:24:10.661030+0000","last_fullsized":"2024-03-06T07:24:10.661030+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654047+0000","last_change":"2024-03-06T07:24:10.654047+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654047+0000","last_undegraded":"2024-03-06T07:24:10.654047+0000","last_fullsized":"2024-03-06T07:24:10.654047+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661932+0000","last_change":"2024-03-06T07:24:10.659862+0000","last_active":"2024-03-06T07:24:10.661932+0000","last_peered":"2024-03-06T07:24:10.661932+0000","last_clean":"2024-03-06T07:24:10.661932+0000","last_became_active":"2024-03-06T07:24:10.659578+0000","last_became_peered":"2024-03-06T07:24:10.659578+0000","last_unstale":"2024-03-06T07:24:10.661932+0000","last_undegraded":"2024-03-06T07:24:10.661932+0000","last_fullsized":"2024-03-06T07:24:10.661932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652981+0000","last_change":"2024-03-06T07:24:10.652981+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652981+0000","last_undegraded":"2024-03-06T07:24:10.652981+0000","last_fullsized":"2024-03-06T07:24:10.652981+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.659947+0000","last_change":"2024-03-06T07:24:10.658442+0000","last_active":"2024-03-06T07:24:10.659947+0000","last_peered":"2024-03-06T07:24:10.659947+0000","last_clean":"2024-03-06T07:24:10.659947+0000","last_became_active":"2024-03-06T07:24:10.658200+0000","last_became_peered":"2024-03-06T07:24:10.658200+0000","last_unstale":"2024-03-06T07:24:10.659947+0000","last_undegraded":"2024-03-06T07:24:10.659947+0000","last_fullsized":"2024-03-06T07:24:10.659947+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647966+0000","last_change":"2024-03-06T07:24:10.647966+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647966+0000","last_undegraded":"2024-03-06T07:24:10.647966+0000","last_fullsized":"2024-03-06T07:24:10.647966+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666422+0000","last_change":"2024-03-06T07:24:10.662365+0000","last_active":"2024-03-06T07:24:10.666422+0000","last_peered":"2024-03-06T07:24:10.666422+0000","last_clean":"2024-03-06T07:24:10.666422+0000","last_became_active":"2024-03-06T07:24:10.662073+0000","last_became_peered":"2024-03-06T07:24:10.662073+0000","last_unstale":"2024-03-06T07:24:10.666422+0000","last_undegraded":"2024-03-06T07:24:10.666422+0000","last_fullsized":"2024-03-06T07:24:10.666422+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:59.892449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647790+0000","last_change":"2024-03-06T07:24:10.647790+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647790+0000","last_undegraded":"2024-03-06T07:24:10.647790+0000","last_fullsized":"2024-03-06T07:24:10.647790+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660165+0000","last_change":"2024-03-06T07:24:10.659174+0000","last_active":"2024-03-06T07:24:10.660165+0000","last_peered":"2024-03-06T07:24:10.660165+0000","last_clean":"2024-03-06T07:24:10.660165+0000","last_became_active":"2024-03-06T07:24:10.658834+0000","last_became_peered":"2024-03-06T07:24:10.658834+0000","last_unstale":"2024-03-06T07:24:10.660165+0000","last_undegraded":"2024-03-06T07:24:10.660165+0000","last_fullsized":"2024-03-06T07:24:10.660165+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:37.876251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648001+0000","last_change":"2024-03-06T07:24:10.648001+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648001+0000","last_undegraded":"2024-03-06T07:24:10.648001+0000","last_fullsized":"2024-03-06T07:24:10.648001+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657498+0000","last_change":"2024-03-06T07:24:10.656628+0000","last_active":"2024-03-06T07:24:10.657498+0000","last_peered":"2024-03-06T07:24:10.657498+0000","last_clean":"2024-03-06T07:24:10.657498+0000","last_became_active":"2024-03-06T07:24:10.656271+0000","last_became_peered":"2024-03-06T07:24:10.656271+0000","last_unstale":"2024-03-06T07:24:10.657498+0000","last_undegraded":"2024-03-06T07:24:10.657498+0000","last_fullsized":"2024-03-06T07:24:10.657498+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:55.005932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648025+0000","last_change":"2024-03-06T07:24:10.648025+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648025+0000","last_undegraded":"2024-03-06T07:24:10.648025+0000","last_fullsized":"2024-03-06T07:24:10.648025+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664836+0000","last_change":"2024-03-06T07:24:10.663013+0000","last_active":"2024-03-06T07:24:10.664836+0000","last_peered":"2024-03-06T07:24:10.664836+0000","last_clean":"2024-03-06T07:24:10.664836+0000","last_became_active":"2024-03-06T07:24:10.662736+0000","last_became_peered":"2024-03-06T07:24:10.662736+0000","last_unstale":"2024-03-06T07:24:10.664836+0000","last_undegraded":"2024-03-06T07:24:10.664836+0000","last_fullsized":"2024-03-06T07:24:10.664836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:34.328658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648442+0000","last_change":"2024-03-06T07:24:10.648442+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648442+0000","last_undegraded":"2024-03-06T07:24:10.648442+0000","last_fullsized":"2024-03-06T07:24:10.648442+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658858+0000","last_change":"2024-03-06T07:24:10.654750+0000","last_active":"2024-03-06T07:24:10.658858+0000","last_peered":"2024-03-06T07:24:10.658858+0000","last_clean":"2024-03-06T07:24:10.658858+0000","last_became_active":"2024-03-06T07:24:10.654476+0000","last_became_peered":"2024-03-06T07:24:10.654476+0000","last_unstale":"2024-03-06T07:24:10.658858+0000","last_undegraded":"2024-03-06T07:24:10.658858+0000","last_fullsized":"2024-03-06T07:24:10.658858+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:55.005932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647211+0000","last_change":"2024-03-06T07:24:10.647211+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647211+0000","last_undegraded":"2024-03-06T07:24:10.647211+0000","last_fullsized":"2024-03-06T07:24:10.647211+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658862+0000","last_change":"2024-03-06T07:24:10.657981+0000","last_active":"2024-03-06T07:24:10.658862+0000","last_peered":"2024-03-06T07:24:10.658862+0000","last_clean":"2024-03-06T07:24:10.658862+0000","last_became_active":"2024-03-06T07:24:10.657616+0000","last_became_peered":"2024-03-06T07:24:10.657616+0000","last_unstale":"2024-03-06T07:24:10.658862+0000","last_undegraded":"2024-03-06T07:24:10.658862+0000","last_fullsized":"2024-03-06T07:24:10.658862+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:45.308028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649057+0000","last_change":"2024-03-06T07:24:10.649057+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649057+0000","last_undegraded":"2024-03-06T07:24:10.649057+0000","last_fullsized":"2024-03-06T07:24:10.649057+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663759+0000","last_change":"2024-03-06T07:24:10.661755+0000","last_active":"2024-03-06T07:24:10.663759+0000","last_peered":"2024-03-06T07:24:10.663759+0000","last_clean":"2024-03-06T07:24:10.663759+0000","last_became_active":"2024-03-06T07:24:10.661432+0000","last_became_peered":"2024-03-06T07:24:10.661432+0000","last_unstale":"2024-03-06T07:24:10.663759+0000","last_undegraded":"2024-03-06T07:24:10.663759+0000","last_fullsized":"2024-03-06T07:24:10.663759+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:15.720920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652228+0000","last_change":"2024-03-06T07:24:10.652228+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652228+0000","last_undegraded":"2024-03-06T07:24:10.652228+0000","last_fullsized":"2024-03-06T07:24:10.652228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657656+0000","last_change":"2024-03-06T07:24:10.656363+0000","last_active":"2024-03-06T07:24:10.657656+0000","last_peered":"2024-03-06T07:24:10.657656+0000","last_clean":"2024-03-06T07:24:10.657656+0000","last_became_active":"2024-03-06T07:24:10.656056+0000","last_became_peered":"2024-03-06T07:24:10.656056+0000","last_unstale":"2024-03-06T07:24:10.657656+0000","last_undegraded":"2024-03-06T07:24:10.657656+0000","last_fullsized":"2024-03-06T07:24:10.657656+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:19.463965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651712+0000","last_change":"2024-03-06T07:24:10.651712+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651712+0000","last_undegraded":"2024-03-06T07:24:10.651712+0000","last_fullsized":"2024-03-06T07:24:10.651712+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664178+0000","last_change":"2024-03-06T07:24:10.662788+0000","last_active":"2024-03-06T07:24:10.664178+0000","last_peered":"2024-03-06T07:24:10.664178+0000","last_clean":"2024-03-06T07:24:10.664178+0000","last_became_active":"2024-03-06T07:24:10.662446+0000","last_became_peered":"2024-03-06T07:24:10.662446+0000","last_unstale":"2024-03-06T07:24:10.664178+0000","last_undegraded":"2024-03-06T07:24:10.664178+0000","last_fullsized":"2024-03-06T07:24:10.664178+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:25.045708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647210+0000","last_change":"2024-03-06T07:24:10.647210+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647210+0000","last_undegraded":"2024-03-06T07:24:10.647210+0000","last_fullsized":"2024-03-06T07:24:10.647210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658501+0000","last_change":"2024-03-06T07:24:10.649819+0000","last_active":"2024-03-06T07:24:10.658501+0000","last_peered":"2024-03-06T07:24:10.658501+0000","last_clean":"2024-03-06T07:24:10.658501+0000","last_became_active":"2024-03-06T07:24:10.649508+0000","last_became_peered":"2024-03-06T07:24:10.649508+0000","last_unstale":"2024-03-06T07:24:10.658501+0000","last_undegraded":"2024-03-06T07:24:10.658501+0000","last_fullsized":"2024-03-06T07:24:10.658501+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:25.045708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652286+0000","last_change":"2024-03-06T07:24:10.652286+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652286+0000","last_undegraded":"2024-03-06T07:24:10.652286+0000","last_fullsized":"2024-03-06T07:24:10.652286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657611+0000","last_change":"2024-03-06T07:24:10.656290+0000","last_active":"2024-03-06T07:24:10.657611+0000","last_peered":"2024-03-06T07:24:10.657611+0000","last_clean":"2024-03-06T07:24:10.657611+0000","last_became_active":"2024-03-06T07:24:10.656023+0000","last_became_peered":"2024-03-06T07:24:10.656023+0000","last_unstale":"2024-03-06T07:24:10.657611+0000","last_undegraded":"2024-03-06T07:24:10.657611+0000","last_fullsized":"2024-03-06T07:24:10.657611+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:35.493530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651834+0000","last_change":"2024-03-06T07:24:10.651834+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651834+0000","last_undegraded":"2024-03-06T07:24:10.651834+0000","last_fullsized":"2024-03-06T07:24:10.651834+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.652350+0000","last_change":"2024-03-06T07:24:10.650351+0000","last_active":"2024-03-06T07:24:10.652350+0000","last_peered":"2024-03-06T07:24:10.652350+0000","last_clean":"2024-03-06T07:24:10.652350+0000","last_became_active":"2024-03-06T07:24:10.649966+0000","last_became_peered":"2024-03-06T07:24:10.649966+0000","last_unstale":"2024-03-06T07:24:10.652350+0000","last_undegraded":"2024-03-06T07:24:10.652350+0000","last_fullsized":"2024-03-06T07:24:10.652350+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:20.586025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":238,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.653875+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:24:10.653875+0000","last_peered":"2024-03-06T07:24:10.653875+0000","last_clean":"2024-03-06T07:24:10.653875+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:24:10.653875+0000","last_undegraded":"2024-03-06T07:24:10.653875+0000","last_fullsized":"2024-03-06T07:24:10.653875+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1265664,"data_stored":866798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":671744,"data_stored":276430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":671744,"data_stored":276430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1265664,"data_stored":866798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:15.145 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:15.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.295+0000 7f14b29d0640 1 Processor -- start 2024-03-06T07:24:15.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.295+0000 7f14b29d0640 1 -- start start 2024-03-06T07:24:15.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14b29d0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f14ac15dfe0 0x7f14ac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14b29d0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac0d29c0 0x7f14ac0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14b29d0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ac0b3430 con 0x7f14ac0cf830 2024-03-06T07:24:15.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14b29d0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ac0b31e0 con 0x7f14ac15dfe0 2024-03-06T07:24:15.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14b29d0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0b3d60 con 0x7f14ac0d29c0 2024-03-06T07:24:15.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14ab7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f14ac0cf830 0x7f14ac0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37690/0 (socket says 172.21.15.59:37690) 2024-03-06T07:24:15.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14ab7fe640 1 -- 172.21.15.59:0/3723182682 learned_addr learned my addr 172.21.15.59:0/3723182682 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14b0f6d640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac0d29c0 0x7f14ac0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14abfff640 1 --2- 172.21.15.59:0/3723182682 >> v2:172.21.15.164:3300/0 conn(0x7f14ac15dfe0 0x7f14ac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908940409 0 0) 0x7f14ac0b3430 con 0x7f14ac0cf830 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1498003520 con 0x7f14ac0cf830 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2817039318 0 0) 0x7f1498003520 con 0x7f14ac0cf830 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac0d29c0 msgr2=0x7f14ac0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14aaffd640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac0d29c0 0x7f14ac0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 >> v2:172.21.15.164:3300/0 conn(0x7f14ac15dfe0 msgr2=0x7f14ac0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14aaffd640 1 --2- 172.21.15.59:0/3723182682 >> v2:172.21.15.164:3300/0 conn(0x7f14ac15dfe0 0x7f14ac0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac0b3710 con 0x7f14ac0cf830 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f1494001670 con 0x7f14ac0cf830 2024-03-06T07:24:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 >> v1:172.21.15.59:6789/0 conn(0x7f14ac0cf830 legacy=0x7f14ac0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.296+0000 7f14aaffd640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14980039b0 0x7f1498003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14aaffd640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac0d29c0 0x7f1498004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14aaffd640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1498004d40 0x7f14980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0b31e0 con 0x7f1498004d40 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0b3d60 con 0x7f14980039b0 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149800b9c0 con 0x7f14ac0d29c0 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14b0f6d640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1498004d40 0x7f14980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14ab7fe640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac0d29c0 0x7f1498004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14abfff640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14980039b0 0x7f1498003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14b0f6d640 1 -- 172.21.15.59:0/3723182682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac0d29c0 msgr2=0x7f1498004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14b0f6d640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac0d29c0 0x7f1498004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14b0f6d640 1 -- 172.21.15.59:0/3723182682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14980039b0 msgr2=0x7f1498003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14b0f6d640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14980039b0 0x7f1498003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.297+0000 7f14b0f6d640 1 -- 172.21.15.59:0/3723182682 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1498003520 con 0x7f1498004d40 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.298+0000 7f14b0f6d640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1498004d40 0x7f14980051c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f14a400e170 tx=0x7f1498005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf604b8f9896d67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.298+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4039070 con 0x7f1498004d40 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.298+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a402cbf0 con 0x7f1498004d40 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.298+0000 7f14aaffd640 1 -- 172.21.15.59:0/3723182682 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4009c10 con 0x7f1498004d40 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.299+0000 7f14b29d0640 1 -- 172.21.15.59:0/3723182682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1498004d40 msgr2=0x7f14980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.299+0000 7f14b29d0640 1 --2- 172.21.15.59:0/3723182682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1498004d40 0x7f14980051c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f14a400e170 tx=0x7f1498005830 comp rx=0 tx=0).stop 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.299+0000 7f14b0f6d640 1 -- 172.21.15.59:0/3723182682 reap_dead start 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.299+0000 7f14b29d0640 1 -- 172.21.15.59:0/3723182682 shutdown_connections 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.299+0000 7f14b29d0640 1 -- 172.21.15.59:0/3723182682 >> 172.21.15.59:0/3723182682 conn(0x7f14ac0d3cd0 msgr2=0x7f14ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.299+0000 7f14b29d0640 1 -- 172.21.15.59:0/3723182682 shutdown_connections 2024-03-06T07:24:15.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.299+0000 7f14b29d0640 1 -- 172.21.15.59:0/3723182682 wait complete. 2024-03-06T07:24:15.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.300+0000 7f14b29d0640 1 Processor -- start 2024-03-06T07:24:15.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.300+0000 7f14b29d0640 1 -- start start 2024-03-06T07:24:15.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14b29d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14ac156420 0x7f14ac15a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14b29d0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14ac15a790 0x7f14ac0a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14b29d0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac156800 0x7f14ac0a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14b29d0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0d1df0 con 0x7f14ac156420 2024-03-06T07:24:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14b29d0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0d1c70 con 0x7f14ac15a790 2024-03-06T07:24:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14b29d0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac15e3c0 con 0x7f14ac156800 2024-03-06T07:24:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14abfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14ac156420 0x7f14ac15a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14abfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14ac156420 0x7f14ac15a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50936/0 (socket says 172.21.15.59:50936) 2024-03-06T07:24:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14abfff640 1 -- 172.21.15.59:0/1023843086 learned_addr learned my addr 172.21.15.59:0/1023843086 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14ab7fe640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14ac15a790 0x7f14ac0a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14b0f6d640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac156800 0x7f14ac0a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14abfff640 1 -- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac156800 msgr2=0x7f14ac0a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14abfff640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac156800 0x7f14ac0a0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14abfff640 1 -- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14ac15a790 msgr2=0x7f14ac0a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14abfff640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14ac15a790 0x7f14ac0a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14abfff640 1 -- 172.21.15.59:0/1023843086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac09e340 con 0x7f14ac156420 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14abfff640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14ac156420 0x7f14ac15a250 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f149c011960 tx=0x7f149c011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14a8ff9640 1 -- 172.21.15.59:0/1023843086 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c00faf0 con 0x7f14ac156420 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.301+0000 7f14a8ff9640 1 -- 172.21.15.59:0/1023843086 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149c00fc90 con 0x7f14ac156420 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.302+0000 7f14b29d0640 1 -- 172.21.15.59:0/1023843086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac09e620 con 0x7f14ac156420 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.302+0000 7f14a8ff9640 1 -- 172.21.15.59:0/1023843086 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c014440 con 0x7f14ac156420 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.302+0000 7f14b29d0640 1 -- 172.21.15.59:0/1023843086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac09ebe0 con 0x7f14ac156420 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.303+0000 7f14b29d0640 1 -- 172.21.15.59:0/1023843086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ac04f330 con 0x7f14ac156420 2024-03-06T07:24:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.303+0000 7f14a8ff9640 1 -- 172.21.15.59:0/1023843086 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f149c012040 con 0x7f14ac156420 2024-03-06T07:24:15.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.303+0000 7f14a8ff9640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f148c06b500 0x7f148c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.304+0000 7f14a8ff9640 1 -- 172.21.15.59:0/1023843086 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f149c08a8d0 con 0x7f14ac156420 2024-03-06T07:24:15.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.304+0000 7f14ab7fe640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f148c06b500 0x7f148c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.304+0000 7f14ab7fe640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f148c06b500 0x7f148c06d9c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1494004840 tx=0x7f1494009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.312+0000 7f14a8ff9640 1 -- 172.21.15.59:0/1023843086 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f149c019030 con 0x7f14ac156420 2024-03-06T07:24:15.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.488+0000 7f14b29d0640 1 -- 172.21.15.59:0/1023843086 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14ac0cf280 con 0x7f148c06b500 2024-03-06T07:24:15.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.494+0000 7f14a8ff9640 1 -- 172.21.15.59:0/1023843086 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246638 (secure 0 0 0) 0x7f14ac0cf280 con 0x7f148c06b500 2024-03-06T07:24:15.498 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:15.499 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:24:15.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.496+0000 7f14b29d0640 1 -- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f148c06b500 msgr2=0x7f148c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.496+0000 7f14b29d0640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f148c06b500 0x7f148c06d9c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1494004840 tx=0x7f1494009040 comp rx=0 tx=0).stop 2024-03-06T07:24:15.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.496+0000 7f14b29d0640 1 -- 172.21.15.59:0/1023843086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14ac156420 msgr2=0x7f14ac15a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.496+0000 7f14b29d0640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14ac156420 0x7f14ac15a250 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f149c011960 tx=0x7f149c011e20 comp rx=0 tx=0).stop 2024-03-06T07:24:15.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.497+0000 7f14b29d0640 1 -- 172.21.15.59:0/1023843086 shutdown_connections 2024-03-06T07:24:15.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.497+0000 7f14b29d0640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f148c06b500 0x7f148c06d9c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.497+0000 7f14b29d0640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ac156800 0x7f14ac0a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.497+0000 7f14b29d0640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14ac15a790 0x7f14ac0a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.497+0000 7f14b29d0640 1 --2- 172.21.15.59:0/1023843086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14ac156420 0x7f14ac15a250 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.497+0000 7f14b29d0640 1 -- 172.21.15.59:0/1023843086 >> 172.21.15.59:0/1023843086 conn(0x7f14ac0d3cd0 msgr2=0x7f14ac15d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.497+0000 7f14b29d0640 1 -- 172.21.15.59:0/1023843086 shutdown_connections 2024-03-06T07:24:15.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.497+0000 7f14b29d0640 1 -- 172.21.15.59:0/1023843086 wait complete. 2024-03-06T07:24:15.512 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-03-06T07:24:15.330142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255916,"kb_used_data":6456,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718804,"statfs":{"total":773094113280,"available":772832055296,"internally_reserved":0,"allocated":6610944,"data_stored":3417244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.690206"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653222+0000","last_change":"2024-03-06T07:24:10.653222+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653222+0000","last_undegraded":"2024-03-06T07:24:10.653222+0000","last_fullsized":"2024-03-06T07:24:10.653222+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654247+0000","last_change":"2024-03-06T07:24:10.654247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654247+0000","last_undegraded":"2024-03-06T07:24:10.654247+0000","last_fullsized":"2024-03-06T07:24:10.654247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653845+0000","last_change":"2024-03-06T07:24:10.653845+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653845+0000","last_undegraded":"2024-03-06T07:24:10.653845+0000","last_fullsized":"2024-03-06T07:24:10.653845+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649798+0000","last_change":"2024-03-06T07:24:10.649798+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649798+0000","last_undegraded":"2024-03-06T07:24:10.649798+0000","last_fullsized":"2024-03-06T07:24:10.649798+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653052+0000","last_change":"2024-03-06T07:24:10.653052+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653052+0000","last_undegraded":"2024-03-06T07:24:10.653052+0000","last_fullsized":"2024-03-06T07:24:10.653052+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649635+0000","last_change":"2024-03-06T07:24:10.649635+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649635+0000","last_undegraded":"2024-03-06T07:24:10.649635+0000","last_fullsized":"2024-03-06T07:24:10.649635+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654247+0000","last_change":"2024-03-06T07:24:10.654247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654247+0000","last_undegraded":"2024-03-06T07:24:10.654247+0000","last_fullsized":"2024-03-06T07:24:10.654247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653163+0000","last_change":"2024-03-06T07:24:10.653163+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653163+0000","last_undegraded":"2024-03-06T07:24:10.653163+0000","last_fullsized":"2024-03-06T07:24:10.653163+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.655335+0000","last_change":"2024-03-06T07:24:10.655335+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.655335+0000","last_undegraded":"2024-03-06T07:24:10.655335+0000","last_fullsized":"2024-03-06T07:24:10.655335+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648323+0000","last_change":"2024-03-06T07:24:10.648323+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648323+0000","last_undegraded":"2024-03-06T07:24:10.648323+0000","last_fullsized":"2024-03-06T07:24:10.648323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648785+0000","last_change":"2024-03-06T07:24:10.648785+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648785+0000","last_undegraded":"2024-03-06T07:24:10.648785+0000","last_fullsized":"2024-03-06T07:24:10.648785+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649164+0000","last_change":"2024-03-06T07:24:10.649164+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649164+0000","last_undegraded":"2024-03-06T07:24:10.649164+0000","last_fullsized":"2024-03-06T07:24:10.649164+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.646039+0000","last_change":"2024-03-06T07:24:10.646039+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.646039+0000","last_undegraded":"2024-03-06T07:24:10.646039+0000","last_fullsized":"2024-03-06T07:24:10.646039+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653571+0000","last_change":"2024-03-06T07:24:10.653571+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653571+0000","last_undegraded":"2024-03-06T07:24:10.653571+0000","last_fullsized":"2024-03-06T07:24:10.653571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653868+0000","last_change":"2024-03-06T07:24:10.653868+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653868+0000","last_undegraded":"2024-03-06T07:24:10.653868+0000","last_fullsized":"2024-03-06T07:24:10.653868+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.650079+0000","last_change":"2024-03-06T07:24:10.650079+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.650079+0000","last_undegraded":"2024-03-06T07:24:10.650079+0000","last_fullsized":"2024-03-06T07:24:10.650079+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652629+0000","last_change":"2024-03-06T07:24:10.652629+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652629+0000","last_undegraded":"2024-03-06T07:24:10.652629+0000","last_fullsized":"2024-03-06T07:24:10.652629+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653025+0000","last_change":"2024-03-06T07:24:10.653025+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653025+0000","last_undegraded":"2024-03-06T07:24:10.653025+0000","last_fullsized":"2024-03-06T07:24:10.653025+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653633+0000","last_change":"2024-03-06T07:24:10.653633+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653633+0000","last_undegraded":"2024-03-06T07:24:10.653633+0000","last_fullsized":"2024-03-06T07:24:10.653633+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.655332+0000","last_change":"2024-03-06T07:24:10.655332+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.655332+0000","last_undegraded":"2024-03-06T07:24:10.655332+0000","last_fullsized":"2024-03-06T07:24:10.655332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652655+0000","last_change":"2024-03-06T07:24:10.652655+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652655+0000","last_undegraded":"2024-03-06T07:24:10.652655+0000","last_fullsized":"2024-03-06T07:24:10.652655+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653247+0000","last_change":"2024-03-06T07:24:10.653247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653247+0000","last_undegraded":"2024-03-06T07:24:10.653247+0000","last_fullsized":"2024-03-06T07:24:10.653247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651973+0000","last_change":"2024-03-06T07:24:10.651973+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651973+0000","last_undegraded":"2024-03-06T07:24:10.651973+0000","last_fullsized":"2024-03-06T07:24:10.651973+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647134+0000","last_change":"2024-03-06T07:24:10.647134+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647134+0000","last_undegraded":"2024-03-06T07:24:10.647134+0000","last_fullsized":"2024-03-06T07:24:10.647134+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652823+0000","last_change":"2024-03-06T07:24:10.652823+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652823+0000","last_undegraded":"2024-03-06T07:24:10.652823+0000","last_fullsized":"2024-03-06T07:24:10.652823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649078+0000","last_change":"2024-03-06T07:24:10.649078+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649078+0000","last_undegraded":"2024-03-06T07:24:10.649078+0000","last_fullsized":"2024-03-06T07:24:10.649078+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649913+0000","last_change":"2024-03-06T07:24:10.649913+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649913+0000","last_undegraded":"2024-03-06T07:24:10.649913+0000","last_fullsized":"2024-03-06T07:24:10.649913+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653647+0000","last_change":"2024-03-06T07:24:10.653647+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653647+0000","last_undegraded":"2024-03-06T07:24:10.653647+0000","last_fullsized":"2024-03-06T07:24:10.653647+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653928+0000","last_change":"2024-03-06T07:24:10.653928+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653928+0000","last_undegraded":"2024-03-06T07:24:10.653928+0000","last_fullsized":"2024-03-06T07:24:10.653928+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651185+0000","last_change":"2024-03-06T07:24:10.651185+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651185+0000","last_undegraded":"2024-03-06T07:24:10.651185+0000","last_fullsized":"2024-03-06T07:24:10.651185+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651671+0000","last_change":"2024-03-06T07:24:10.651671+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651671+0000","last_undegraded":"2024-03-06T07:24:10.651671+0000","last_fullsized":"2024-03-06T07:24:10.651671+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647547+0000","last_change":"2024-03-06T07:24:10.647547+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647547+0000","last_undegraded":"2024-03-06T07:24:10.647547+0000","last_fullsized":"2024-03-06T07:24:10.647547+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648981+0000","last_change":"2024-03-06T07:24:10.648981+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648981+0000","last_undegraded":"2024-03-06T07:24:10.648981+0000","last_fullsized":"2024-03-06T07:24:10.648981+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661442+0000","last_change":"2024-03-06T07:24:10.658033+0000","last_active":"2024-03-06T07:24:10.661442+0000","last_peered":"2024-03-06T07:24:10.661442+0000","last_clean":"2024-03-06T07:24:10.661442+0000","last_became_active":"2024-03-06T07:24:10.657766+0000","last_became_peered":"2024-03-06T07:24:10.657766+0000","last_unstale":"2024-03-06T07:24:10.661442+0000","last_undegraded":"2024-03-06T07:24:10.661442+0000","last_fullsized":"2024-03-06T07:24:10.661442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648210+0000","last_change":"2024-03-06T07:24:10.648210+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648210+0000","last_undegraded":"2024-03-06T07:24:10.648210+0000","last_fullsized":"2024-03-06T07:24:10.648210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660926+0000","last_change":"2024-03-06T07:24:10.659965+0000","last_active":"2024-03-06T07:24:10.660926+0000","last_peered":"2024-03-06T07:24:10.660926+0000","last_clean":"2024-03-06T07:24:10.660926+0000","last_became_active":"2024-03-06T07:24:10.659581+0000","last_became_peered":"2024-03-06T07:24:10.659581+0000","last_unstale":"2024-03-06T07:24:10.660926+0000","last_undegraded":"2024-03-06T07:24:10.660926+0000","last_fullsized":"2024-03-06T07:24:10.660926+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:39.589222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648969+0000","last_change":"2024-03-06T07:24:10.648969+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648969+0000","last_undegraded":"2024-03-06T07:24:10.648969+0000","last_fullsized":"2024-03-06T07:24:10.648969+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666485+0000","last_change":"2024-03-06T07:24:10.662366+0000","last_active":"2024-03-06T07:24:10.666485+0000","last_peered":"2024-03-06T07:24:10.666485+0000","last_clean":"2024-03-06T07:24:10.666485+0000","last_became_active":"2024-03-06T07:24:10.662075+0000","last_became_peered":"2024-03-06T07:24:10.662075+0000","last_unstale":"2024-03-06T07:24:10.666485+0000","last_undegraded":"2024-03-06T07:24:10.666485+0000","last_fullsized":"2024-03-06T07:24:10.666485+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.249754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649490+0000","last_change":"2024-03-06T07:24:10.649490+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649490+0000","last_undegraded":"2024-03-06T07:24:10.649490+0000","last_fullsized":"2024-03-06T07:24:10.649490+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660644+0000","last_change":"2024-03-06T07:24:10.659614+0000","last_active":"2024-03-06T07:24:10.660644+0000","last_peered":"2024-03-06T07:24:10.660644+0000","last_clean":"2024-03-06T07:24:10.660644+0000","last_became_active":"2024-03-06T07:24:10.659301+0000","last_became_peered":"2024-03-06T07:24:10.659301+0000","last_unstale":"2024-03-06T07:24:10.660644+0000","last_undegraded":"2024-03-06T07:24:10.660644+0000","last_fullsized":"2024-03-06T07:24:10.660644+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652779+0000","last_change":"2024-03-06T07:24:10.652779+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652779+0000","last_undegraded":"2024-03-06T07:24:10.652779+0000","last_fullsized":"2024-03-06T07:24:10.652779+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663858+0000","last_change":"2024-03-06T07:24:10.661362+0000","last_active":"2024-03-06T07:24:10.663858+0000","last_peered":"2024-03-06T07:24:10.663858+0000","last_clean":"2024-03-06T07:24:10.663858+0000","last_became_active":"2024-03-06T07:24:10.661069+0000","last_became_peered":"2024-03-06T07:24:10.661069+0000","last_unstale":"2024-03-06T07:24:10.663858+0000","last_undegraded":"2024-03-06T07:24:10.663858+0000","last_fullsized":"2024-03-06T07:24:10.663858+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:47.652320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654658+0000","last_change":"2024-03-06T07:24:10.654658+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654658+0000","last_undegraded":"2024-03-06T07:24:10.654658+0000","last_fullsized":"2024-03-06T07:24:10.654658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.655589+0000","last_change":"2024-03-06T07:24:10.653845+0000","last_active":"2024-03-06T07:24:10.655589+0000","last_peered":"2024-03-06T07:24:10.655589+0000","last_clean":"2024-03-06T07:24:10.655589+0000","last_became_active":"2024-03-06T07:24:10.653600+0000","last_became_peered":"2024-03-06T07:24:10.653600+0000","last_unstale":"2024-03-06T07:24:10.655589+0000","last_undegraded":"2024-03-06T07:24:10.655589+0000","last_fullsized":"2024-03-06T07:24:10.655589+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:05.337286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.646559+0000","last_change":"2024-03-06T07:24:10.646559+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.646559+0000","last_undegraded":"2024-03-06T07:24:10.646559+0000","last_fullsized":"2024-03-06T07:24:10.646559+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663129+0000","last_change":"2024-03-06T07:24:10.660990+0000","last_active":"2024-03-06T07:24:10.663129+0000","last_peered":"2024-03-06T07:24:10.663129+0000","last_clean":"2024-03-06T07:24:10.663129+0000","last_became_active":"2024-03-06T07:24:10.660680+0000","last_became_peered":"2024-03-06T07:24:10.660680+0000","last_unstale":"2024-03-06T07:24:10.663129+0000","last_undegraded":"2024-03-06T07:24:10.663129+0000","last_fullsized":"2024-03-06T07:24:10.663129+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.249754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652835+0000","last_change":"2024-03-06T07:24:10.652835+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652835+0000","last_undegraded":"2024-03-06T07:24:10.652835+0000","last_fullsized":"2024-03-06T07:24:10.652835+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660821+0000","last_change":"2024-03-06T07:24:10.659499+0000","last_active":"2024-03-06T07:24:10.660821+0000","last_peered":"2024-03-06T07:24:10.660821+0000","last_clean":"2024-03-06T07:24:10.660821+0000","last_became_active":"2024-03-06T07:24:10.659224+0000","last_became_peered":"2024-03-06T07:24:10.659224+0000","last_unstale":"2024-03-06T07:24:10.660821+0000","last_undegraded":"2024-03-06T07:24:10.660821+0000","last_fullsized":"2024-03-06T07:24:10.660821+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:37.419239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654768+0000","last_change":"2024-03-06T07:24:10.654768+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654768+0000","last_undegraded":"2024-03-06T07:24:10.654768+0000","last_fullsized":"2024-03-06T07:24:10.654768+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664029+0000","last_change":"2024-03-06T07:24:10.662290+0000","last_active":"2024-03-06T07:24:10.664029+0000","last_peered":"2024-03-06T07:24:10.664029+0000","last_clean":"2024-03-06T07:24:10.664029+0000","last_became_active":"2024-03-06T07:24:10.662003+0000","last_became_peered":"2024-03-06T07:24:10.662003+0000","last_unstale":"2024-03-06T07:24:10.664029+0000","last_undegraded":"2024-03-06T07:24:10.664029+0000","last_fullsized":"2024-03-06T07:24:10.664029+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652707+0000","last_change":"2024-03-06T07:24:10.652707+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652707+0000","last_undegraded":"2024-03-06T07:24:10.652707+0000","last_fullsized":"2024-03-06T07:24:10.652707+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666379+0000","last_change":"2024-03-06T07:24:10.663647+0000","last_active":"2024-03-06T07:24:10.666379+0000","last_peered":"2024-03-06T07:24:10.666379+0000","last_clean":"2024-03-06T07:24:10.666379+0000","last_became_active":"2024-03-06T07:24:10.663378+0000","last_became_peered":"2024-03-06T07:24:10.663378+0000","last_unstale":"2024-03-06T07:24:10.666379+0000","last_undegraded":"2024-03-06T07:24:10.666379+0000","last_fullsized":"2024-03-06T07:24:10.666379+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:21.084766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648169+0000","last_change":"2024-03-06T07:24:10.648169+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648169+0000","last_undegraded":"2024-03-06T07:24:10.648169+0000","last_fullsized":"2024-03-06T07:24:10.648169+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.668926+0000","last_change":"2024-03-06T07:24:10.665110+0000","last_active":"2024-03-06T07:24:10.668926+0000","last_peered":"2024-03-06T07:24:10.668926+0000","last_clean":"2024-03-06T07:24:10.668926+0000","last_became_active":"2024-03-06T07:24:10.664833+0000","last_became_peered":"2024-03-06T07:24:10.664833+0000","last_unstale":"2024-03-06T07:24:10.668926+0000","last_undegraded":"2024-03-06T07:24:10.668926+0000","last_fullsized":"2024-03-06T07:24:10.668926+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653401+0000","last_change":"2024-03-06T07:24:10.653401+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653401+0000","last_undegraded":"2024-03-06T07:24:10.653401+0000","last_fullsized":"2024-03-06T07:24:10.653401+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664163+0000","last_change":"2024-03-06T07:24:10.662566+0000","last_active":"2024-03-06T07:24:10.664163+0000","last_peered":"2024-03-06T07:24:10.664163+0000","last_clean":"2024-03-06T07:24:10.664163+0000","last_became_active":"2024-03-06T07:24:10.662201+0000","last_became_peered":"2024-03-06T07:24:10.662201+0000","last_unstale":"2024-03-06T07:24:10.664163+0000","last_undegraded":"2024-03-06T07:24:10.664163+0000","last_fullsized":"2024-03-06T07:24:10.664163+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:00.568465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648596+0000","last_change":"2024-03-06T07:24:10.648596+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648596+0000","last_undegraded":"2024-03-06T07:24:10.648596+0000","last_fullsized":"2024-03-06T07:24:10.648596+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666591+0000","last_change":"2024-03-06T07:24:10.664502+0000","last_active":"2024-03-06T07:24:10.666591+0000","last_peered":"2024-03-06T07:24:10.666591+0000","last_clean":"2024-03-06T07:24:10.666591+0000","last_became_active":"2024-03-06T07:24:10.664248+0000","last_became_peered":"2024-03-06T07:24:10.664248+0000","last_unstale":"2024-03-06T07:24:10.666591+0000","last_undegraded":"2024-03-06T07:24:10.666591+0000","last_fullsized":"2024-03-06T07:24:10.666591+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:07.375756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652181+0000","last_change":"2024-03-06T07:24:10.652181+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652181+0000","last_undegraded":"2024-03-06T07:24:10.652181+0000","last_fullsized":"2024-03-06T07:24:10.652181+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658397+0000","last_change":"2024-03-06T07:24:10.655420+0000","last_active":"2024-03-06T07:24:10.658397+0000","last_peered":"2024-03-06T07:24:10.658397+0000","last_clean":"2024-03-06T07:24:10.658397+0000","last_became_active":"2024-03-06T07:24:10.655148+0000","last_became_peered":"2024-03-06T07:24:10.655148+0000","last_unstale":"2024-03-06T07:24:10.658397+0000","last_undegraded":"2024-03-06T07:24:10.658397+0000","last_fullsized":"2024-03-06T07:24:10.658397+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:16.948447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652559+0000","last_change":"2024-03-06T07:24:10.652559+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652559+0000","last_undegraded":"2024-03-06T07:24:10.652559+0000","last_fullsized":"2024-03-06T07:24:10.652559+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661144+0000","last_change":"2024-03-06T07:24:10.659555+0000","last_active":"2024-03-06T07:24:10.661144+0000","last_peered":"2024-03-06T07:24:10.661144+0000","last_clean":"2024-03-06T07:24:10.661144+0000","last_became_active":"2024-03-06T07:24:10.659285+0000","last_became_peered":"2024-03-06T07:24:10.659285+0000","last_unstale":"2024-03-06T07:24:10.661144+0000","last_undegraded":"2024-03-06T07:24:10.661144+0000","last_fullsized":"2024-03-06T07:24:10.661144+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651408+0000","last_change":"2024-03-06T07:24:10.651408+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651408+0000","last_undegraded":"2024-03-06T07:24:10.651408+0000","last_fullsized":"2024-03-06T07:24:10.651408+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664001+0000","last_change":"2024-03-06T07:24:10.661441+0000","last_active":"2024-03-06T07:24:10.664001+0000","last_peered":"2024-03-06T07:24:10.664001+0000","last_clean":"2024-03-06T07:24:10.664001+0000","last_became_active":"2024-03-06T07:24:10.661157+0000","last_became_peered":"2024-03-06T07:24:10.661157+0000","last_unstale":"2024-03-06T07:24:10.664001+0000","last_undegraded":"2024-03-06T07:24:10.664001+0000","last_fullsized":"2024-03-06T07:24:10.664001+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652018+0000","last_change":"2024-03-06T07:24:10.652018+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652018+0000","last_undegraded":"2024-03-06T07:24:10.652018+0000","last_fullsized":"2024-03-06T07:24:10.652018+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.659358+0000","last_change":"2024-03-06T07:24:10.658071+0000","last_active":"2024-03-06T07:24:10.659358+0000","last_peered":"2024-03-06T07:24:10.659358+0000","last_clean":"2024-03-06T07:24:10.659358+0000","last_became_active":"2024-03-06T07:24:10.657790+0000","last_became_peered":"2024-03-06T07:24:10.657790+0000","last_unstale":"2024-03-06T07:24:10.659358+0000","last_undegraded":"2024-03-06T07:24:10.659358+0000","last_fullsized":"2024-03-06T07:24:10.659358+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:33.615309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649490+0000","last_change":"2024-03-06T07:24:10.649490+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649490+0000","last_undegraded":"2024-03-06T07:24:10.649490+0000","last_fullsized":"2024-03-06T07:24:10.649490+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661030+0000","last_change":"2024-03-06T07:24:10.658700+0000","last_active":"2024-03-06T07:24:10.661030+0000","last_peered":"2024-03-06T07:24:10.661030+0000","last_clean":"2024-03-06T07:24:10.661030+0000","last_became_active":"2024-03-06T07:24:10.658430+0000","last_became_peered":"2024-03-06T07:24:10.658430+0000","last_unstale":"2024-03-06T07:24:10.661030+0000","last_undegraded":"2024-03-06T07:24:10.661030+0000","last_fullsized":"2024-03-06T07:24:10.661030+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654047+0000","last_change":"2024-03-06T07:24:10.654047+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654047+0000","last_undegraded":"2024-03-06T07:24:10.654047+0000","last_fullsized":"2024-03-06T07:24:10.654047+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661932+0000","last_change":"2024-03-06T07:24:10.659862+0000","last_active":"2024-03-06T07:24:10.661932+0000","last_peered":"2024-03-06T07:24:10.661932+0000","last_clean":"2024-03-06T07:24:10.661932+0000","last_became_active":"2024-03-06T07:24:10.659578+0000","last_became_peered":"2024-03-06T07:24:10.659578+0000","last_unstale":"2024-03-06T07:24:10.661932+0000","last_undegraded":"2024-03-06T07:24:10.661932+0000","last_fullsized":"2024-03-06T07:24:10.661932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652981+0000","last_change":"2024-03-06T07:24:10.652981+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652981+0000","last_undegraded":"2024-03-06T07:24:10.652981+0000","last_fullsized":"2024-03-06T07:24:10.652981+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.659947+0000","last_change":"2024-03-06T07:24:10.658442+0000","last_active":"2024-03-06T07:24:10.659947+0000","last_peered":"2024-03-06T07:24:10.659947+0000","last_clean":"2024-03-06T07:24:10.659947+0000","last_became_active":"2024-03-06T07:24:10.658200+0000","last_became_peered":"2024-03-06T07:24:10.658200+0000","last_unstale":"2024-03-06T07:24:10.659947+0000","last_undegraded":"2024-03-06T07:24:10.659947+0000","last_fullsized":"2024-03-06T07:24:10.659947+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647966+0000","last_change":"2024-03-06T07:24:10.647966+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647966+0000","last_undegraded":"2024-03-06T07:24:10.647966+0000","last_fullsized":"2024-03-06T07:24:10.647966+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666422+0000","last_change":"2024-03-06T07:24:10.662365+0000","last_active":"2024-03-06T07:24:10.666422+0000","last_peered":"2024-03-06T07:24:10.666422+0000","last_clean":"2024-03-06T07:24:10.666422+0000","last_became_active":"2024-03-06T07:24:10.662073+0000","last_became_peered":"2024-03-06T07:24:10.662073+0000","last_unstale":"2024-03-06T07:24:10.666422+0000","last_undegraded":"2024-03-06T07:24:10.666422+0000","last_fullsized":"2024-03-06T07:24:10.666422+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:59.892449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647790+0000","last_change":"2024-03-06T07:24:10.647790+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647790+0000","last_undegraded":"2024-03-06T07:24:10.647790+0000","last_fullsized":"2024-03-06T07:24:10.647790+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660165+0000","last_change":"2024-03-06T07:24:10.659174+0000","last_active":"2024-03-06T07:24:10.660165+0000","last_peered":"2024-03-06T07:24:10.660165+0000","last_clean":"2024-03-06T07:24:10.660165+0000","last_became_active":"2024-03-06T07:24:10.658834+0000","last_became_peered":"2024-03-06T07:24:10.658834+0000","last_unstale":"2024-03-06T07:24:10.660165+0000","last_undegraded":"2024-03-06T07:24:10.660165+0000","last_fullsized":"2024-03-06T07:24:10.660165+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:37.876251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648001+0000","last_change":"2024-03-06T07:24:10.648001+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648001+0000","last_undegraded":"2024-03-06T07:24:10.648001+0000","last_fullsized":"2024-03-06T07:24:10.648001+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657498+0000","last_change":"2024-03-06T07:24:10.656628+0000","last_active":"2024-03-06T07:24:10.657498+0000","last_peered":"2024-03-06T07:24:10.657498+0000","last_clean":"2024-03-06T07:24:10.657498+0000","last_became_active":"2024-03-06T07:24:10.656271+0000","last_became_peered":"2024-03-06T07:24:10.656271+0000","last_unstale":"2024-03-06T07:24:10.657498+0000","last_undegraded":"2024-03-06T07:24:10.657498+0000","last_fullsized":"2024-03-06T07:24:10.657498+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:55.005932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648025+0000","last_change":"2024-03-06T07:24:10.648025+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648025+0000","last_undegraded":"2024-03-06T07:24:10.648025+0000","last_fullsized":"2024-03-06T07:24:10.648025+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664836+0000","last_change":"2024-03-06T07:24:10.663013+0000","last_active":"2024-03-06T07:24:10.664836+0000","last_peered":"2024-03-06T07:24:10.664836+0000","last_clean":"2024-03-06T07:24:10.664836+0000","last_became_active":"2024-03-06T07:24:10.662736+0000","last_became_peered":"2024-03-06T07:24:10.662736+0000","last_unstale":"2024-03-06T07:24:10.664836+0000","last_undegraded":"2024-03-06T07:24:10.664836+0000","last_fullsized":"2024-03-06T07:24:10.664836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:34.328658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648442+0000","last_change":"2024-03-06T07:24:10.648442+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648442+0000","last_undegraded":"2024-03-06T07:24:10.648442+0000","last_fullsized":"2024-03-06T07:24:10.648442+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658858+0000","last_change":"2024-03-06T07:24:10.654750+0000","last_active":"2024-03-06T07:24:10.658858+0000","last_peered":"2024-03-06T07:24:10.658858+0000","last_clean":"2024-03-06T07:24:10.658858+0000","last_became_active":"2024-03-06T07:24:10.654476+0000","last_became_peered":"2024-03-06T07:24:10.654476+0000","last_unstale":"2024-03-06T07:24:10.658858+0000","last_undegraded":"2024-03-06T07:24:10.658858+0000","last_fullsized":"2024-03-06T07:24:10.658858+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:55.005932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647211+0000","last_change":"2024-03-06T07:24:10.647211+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647211+0000","last_undegraded":"2024-03-06T07:24:10.647211+0000","last_fullsized":"2024-03-06T07:24:10.647211+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658862+0000","last_change":"2024-03-06T07:24:10.657981+0000","last_active":"2024-03-06T07:24:10.658862+0000","last_peered":"2024-03-06T07:24:10.658862+0000","last_clean":"2024-03-06T07:24:10.658862+0000","last_became_active":"2024-03-06T07:24:10.657616+0000","last_became_peered":"2024-03-06T07:24:10.657616+0000","last_unstale":"2024-03-06T07:24:10.658862+0000","last_undegraded":"2024-03-06T07:24:10.658862+0000","last_fullsized":"2024-03-06T07:24:10.658862+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:45.308028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649057+0000","last_change":"2024-03-06T07:24:10.649057+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649057+0000","last_undegraded":"2024-03-06T07:24:10.649057+0000","last_fullsized":"2024-03-06T07:24:10.649057+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663759+0000","last_change":"2024-03-06T07:24:10.661755+0000","last_active":"2024-03-06T07:24:10.663759+0000","last_peered":"2024-03-06T07:24:10.663759+0000","last_clean":"2024-03-06T07:24:10.663759+0000","last_became_active":"2024-03-06T07:24:10.661432+0000","last_became_peered":"2024-03-06T07:24:10.661432+0000","last_unstale":"2024-03-06T07:24:10.663759+0000","last_undegraded":"2024-03-06T07:24:10.663759+0000","last_fullsized":"2024-03-06T07:24:10.663759+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:15.720920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652228+0000","last_change":"2024-03-06T07:24:10.652228+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652228+0000","last_undegraded":"2024-03-06T07:24:10.652228+0000","last_fullsized":"2024-03-06T07:24:10.652228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657656+0000","last_change":"2024-03-06T07:24:10.656363+0000","last_active":"2024-03-06T07:24:10.657656+0000","last_peered":"2024-03-06T07:24:10.657656+0000","last_clean":"2024-03-06T07:24:10.657656+0000","last_became_active":"2024-03-06T07:24:10.656056+0000","last_became_peered":"2024-03-06T07:24:10.656056+0000","last_unstale":"2024-03-06T07:24:10.657656+0000","last_undegraded":"2024-03-06T07:24:10.657656+0000","last_fullsized":"2024-03-06T07:24:10.657656+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:19.463965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651712+0000","last_change":"2024-03-06T07:24:10.651712+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651712+0000","last_undegraded":"2024-03-06T07:24:10.651712+0000","last_fullsized":"2024-03-06T07:24:10.651712+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664178+0000","last_change":"2024-03-06T07:24:10.662788+0000","last_active":"2024-03-06T07:24:10.664178+0000","last_peered":"2024-03-06T07:24:10.664178+0000","last_clean":"2024-03-06T07:24:10.664178+0000","last_became_active":"2024-03-06T07:24:10.662446+0000","last_became_peered":"2024-03-06T07:24:10.662446+0000","last_unstale":"2024-03-06T07:24:10.664178+0000","last_undegraded":"2024-03-06T07:24:10.664178+0000","last_fullsized":"2024-03-06T07:24:10.664178+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:25.045708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647210+0000","last_change":"2024-03-06T07:24:10.647210+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647210+0000","last_undegraded":"2024-03-06T07:24:10.647210+0000","last_fullsized":"2024-03-06T07:24:10.647210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658501+0000","last_change":"2024-03-06T07:24:10.649819+0000","last_active":"2024-03-06T07:24:10.658501+0000","last_peered":"2024-03-06T07:24:10.658501+0000","last_clean":"2024-03-06T07:24:10.658501+0000","last_became_active":"2024-03-06T07:24:10.649508+0000","last_became_peered":"2024-03-06T07:24:10.649508+0000","last_unstale":"2024-03-06T07:24:10.658501+0000","last_undegraded":"2024-03-06T07:24:10.658501+0000","last_fullsized":"2024-03-06T07:24:10.658501+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:25.045708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652286+0000","last_change":"2024-03-06T07:24:10.652286+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652286+0000","last_undegraded":"2024-03-06T07:24:10.652286+0000","last_fullsized":"2024-03-06T07:24:10.652286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657611+0000","last_change":"2024-03-06T07:24:10.656290+0000","last_active":"2024-03-06T07:24:10.657611+0000","last_peered":"2024-03-06T07:24:10.657611+0000","last_clean":"2024-03-06T07:24:10.657611+0000","last_became_active":"2024-03-06T07:24:10.656023+0000","last_became_peered":"2024-03-06T07:24:10.656023+0000","last_unstale":"2024-03-06T07:24:10.657611+0000","last_undegraded":"2024-03-06T07:24:10.657611+0000","last_fullsized":"2024-03-06T07:24:10.657611+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:35.493530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651834+0000","last_change":"2024-03-06T07:24:10.651834+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651834+0000","last_undegraded":"2024-03-06T07:24:10.651834+0000","last_fullsized":"2024-03-06T07:24:10.651834+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.652350+0000","last_change":"2024-03-06T07:24:10.650351+0000","last_active":"2024-03-06T07:24:10.652350+0000","last_peered":"2024-03-06T07:24:10.652350+0000","last_clean":"2024-03-06T07:24:10.652350+0000","last_became_active":"2024-03-06T07:24:10.649966+0000","last_became_peered":"2024-03-06T07:24:10.649966+0000","last_unstale":"2024-03-06T07:24:10.652350+0000","last_undegraded":"2024-03-06T07:24:10.652350+0000","last_fullsized":"2024-03-06T07:24:10.652350+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:20.586025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":238,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.653875+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:24:10.653875+0000","last_peered":"2024-03-06T07:24:10.653875+0000","last_clean":"2024-03-06T07:24:10.653875+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:24:10.653875+0000","last_undegraded":"2024-03-06T07:24:10.653875+0000","last_fullsized":"2024-03-06T07:24:10.653875+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1265664,"data_stored":866798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":671744,"data_stored":276430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":671744,"data_stored":276430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1265664,"data_stored":866798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:15.517 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3f is not active+clean 2024-03-06T07:24:15.517 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653222+0000', 'last_change': '2024-03-06T07:24:10.653222+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653222+0000', 'last_undegraded': '2024-03-06T07:24:10.653222+0000', 'last_fullsized': '2024-03-06T07:24:10.653222+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:15.517 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3e is not active+clean 2024-03-06T07:24:15.517 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.654247+0000', 'last_change': '2024-03-06T07:24:10.654247+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.654247+0000', 'last_undegraded': '2024-03-06T07:24:10.654247+0000', 'last_fullsized': '2024-03-06T07:24:10.654247+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.517 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3d is not active+clean 2024-03-06T07:24:15.517 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653845+0000', 'last_change': '2024-03-06T07:24:10.653845+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653845+0000', 'last_undegraded': '2024-03-06T07:24:10.653845+0000', 'last_fullsized': '2024-03-06T07:24:10.653845+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.517 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3c is not active+clean 2024-03-06T07:24:15.517 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.649798+0000', 'last_change': '2024-03-06T07:24:10.649798+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.649798+0000', 'last_undegraded': '2024-03-06T07:24:10.649798+0000', 'last_fullsized': '2024-03-06T07:24:10.649798+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:15.518 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3b is not active+clean 2024-03-06T07:24:15.518 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653052+0000', 'last_change': '2024-03-06T07:24:10.653052+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653052+0000', 'last_undegraded': '2024-03-06T07:24:10.653052+0000', 'last_fullsized': '2024-03-06T07:24:10.653052+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:15.518 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3a is not active+clean 2024-03-06T07:24:15.518 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.649635+0000', 'last_change': '2024-03-06T07:24:10.649635+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.649635+0000', 'last_undegraded': '2024-03-06T07:24:10.649635+0000', 'last_fullsized': '2024-03-06T07:24:10.649635+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:15.518 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.39 is not active+clean 2024-03-06T07:24:15.518 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.654247+0000', 'last_change': '2024-03-06T07:24:10.654247+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.654247+0000', 'last_undegraded': '2024-03-06T07:24:10.654247+0000', 'last_fullsized': '2024-03-06T07:24:10.654247+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:15.518 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.38 is not active+clean 2024-03-06T07:24:15.518 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653163+0000', 'last_change': '2024-03-06T07:24:10.653163+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653163+0000', 'last_undegraded': '2024-03-06T07:24:10.653163+0000', 'last_fullsized': '2024-03-06T07:24:10.653163+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.37 is not active+clean 2024-03-06T07:24:15.519 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.655335+0000', 'last_change': '2024-03-06T07:24:10.655335+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.655335+0000', 'last_undegraded': '2024-03-06T07:24:10.655335+0000', 'last_fullsized': '2024-03-06T07:24:10.655335+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:15.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.36 is not active+clean 2024-03-06T07:24:15.519 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.648323+0000', 'last_change': '2024-03-06T07:24:10.648323+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.648323+0000', 'last_undegraded': '2024-03-06T07:24:10.648323+0000', 'last_fullsized': '2024-03-06T07:24:10.648323+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:15.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.35 is not active+clean 2024-03-06T07:24:15.519 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.648785+0000', 'last_change': '2024-03-06T07:24:10.648785+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.648785+0000', 'last_undegraded': '2024-03-06T07:24:10.648785+0000', 'last_fullsized': '2024-03-06T07:24:10.648785+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:15.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.34 is not active+clean 2024-03-06T07:24:15.519 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.649164+0000', 'last_change': '2024-03-06T07:24:10.649164+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.649164+0000', 'last_undegraded': '2024-03-06T07:24:10.649164+0000', 'last_fullsized': '2024-03-06T07:24:10.649164+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:15.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.33 is not active+clean 2024-03-06T07:24:15.519 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.646039+0000', 'last_change': '2024-03-06T07:24:10.646039+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.646039+0000', 'last_undegraded': '2024-03-06T07:24:10.646039+0000', 'last_fullsized': '2024-03-06T07:24:10.646039+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:15.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.32 is not active+clean 2024-03-06T07:24:15.520 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653571+0000', 'last_change': '2024-03-06T07:24:10.653571+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653571+0000', 'last_undegraded': '2024-03-06T07:24:10.653571+0000', 'last_fullsized': '2024-03-06T07:24:10.653571+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:15.520 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.31 is not active+clean 2024-03-06T07:24:15.520 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653868+0000', 'last_change': '2024-03-06T07:24:10.653868+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653868+0000', 'last_undegraded': '2024-03-06T07:24:10.653868+0000', 'last_fullsized': '2024-03-06T07:24:10.653868+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:15.520 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.30 is not active+clean 2024-03-06T07:24:15.520 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.650079+0000', 'last_change': '2024-03-06T07:24:10.650079+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.650079+0000', 'last_undegraded': '2024-03-06T07:24:10.650079+0000', 'last_fullsized': '2024-03-06T07:24:10.650079+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:15.520 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2f is not active+clean 2024-03-06T07:24:15.520 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652629+0000', 'last_change': '2024-03-06T07:24:10.652629+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652629+0000', 'last_undegraded': '2024-03-06T07:24:10.652629+0000', 'last_fullsized': '2024-03-06T07:24:10.652629+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.521 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2e is not active+clean 2024-03-06T07:24:15.521 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653025+0000', 'last_change': '2024-03-06T07:24:10.653025+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653025+0000', 'last_undegraded': '2024-03-06T07:24:10.653025+0000', 'last_fullsized': '2024-03-06T07:24:10.653025+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.521 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2d is not active+clean 2024-03-06T07:24:15.521 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653633+0000', 'last_change': '2024-03-06T07:24:10.653633+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653633+0000', 'last_undegraded': '2024-03-06T07:24:10.653633+0000', 'last_fullsized': '2024-03-06T07:24:10.653633+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:15.521 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2c is not active+clean 2024-03-06T07:24:15.521 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.655332+0000', 'last_change': '2024-03-06T07:24:10.655332+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.655332+0000', 'last_undegraded': '2024-03-06T07:24:10.655332+0000', 'last_fullsized': '2024-03-06T07:24:10.655332+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.521 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2b is not active+clean 2024-03-06T07:24:15.521 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652655+0000', 'last_change': '2024-03-06T07:24:10.652655+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652655+0000', 'last_undegraded': '2024-03-06T07:24:10.652655+0000', 'last_fullsized': '2024-03-06T07:24:10.652655+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.522 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2a is not active+clean 2024-03-06T07:24:15.522 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653247+0000', 'last_change': '2024-03-06T07:24:10.653247+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653247+0000', 'last_undegraded': '2024-03-06T07:24:10.653247+0000', 'last_fullsized': '2024-03-06T07:24:10.653247+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.522 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.29 is not active+clean 2024-03-06T07:24:15.522 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.651973+0000', 'last_change': '2024-03-06T07:24:10.651973+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.651973+0000', 'last_undegraded': '2024-03-06T07:24:10.651973+0000', 'last_fullsized': '2024-03-06T07:24:10.651973+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.522 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.28 is not active+clean 2024-03-06T07:24:15.522 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.647134+0000', 'last_change': '2024-03-06T07:24:10.647134+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.647134+0000', 'last_undegraded': '2024-03-06T07:24:10.647134+0000', 'last_fullsized': '2024-03-06T07:24:10.647134+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:15.523 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.27 is not active+clean 2024-03-06T07:24:15.523 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652823+0000', 'last_change': '2024-03-06T07:24:10.652823+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652823+0000', 'last_undegraded': '2024-03-06T07:24:10.652823+0000', 'last_fullsized': '2024-03-06T07:24:10.652823+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:15.523 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.26 is not active+clean 2024-03-06T07:24:15.523 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.649078+0000', 'last_change': '2024-03-06T07:24:10.649078+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.649078+0000', 'last_undegraded': '2024-03-06T07:24:10.649078+0000', 'last_fullsized': '2024-03-06T07:24:10.649078+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:15.523 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.25 is not active+clean 2024-03-06T07:24:15.523 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.649913+0000', 'last_change': '2024-03-06T07:24:10.649913+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.649913+0000', 'last_undegraded': '2024-03-06T07:24:10.649913+0000', 'last_fullsized': '2024-03-06T07:24:10.649913+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:15.523 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.24 is not active+clean 2024-03-06T07:24:15.523 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653647+0000', 'last_change': '2024-03-06T07:24:10.653647+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653647+0000', 'last_undegraded': '2024-03-06T07:24:10.653647+0000', 'last_fullsized': '2024-03-06T07:24:10.653647+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:15.524 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.23 is not active+clean 2024-03-06T07:24:15.524 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653928+0000', 'last_change': '2024-03-06T07:24:10.653928+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653928+0000', 'last_undegraded': '2024-03-06T07:24:10.653928+0000', 'last_fullsized': '2024-03-06T07:24:10.653928+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.524 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.22 is not active+clean 2024-03-06T07:24:15.524 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.651185+0000', 'last_change': '2024-03-06T07:24:10.651185+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.651185+0000', 'last_undegraded': '2024-03-06T07:24:10.651185+0000', 'last_fullsized': '2024-03-06T07:24:10.651185+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.524 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.21 is not active+clean 2024-03-06T07:24:15.524 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.651671+0000', 'last_change': '2024-03-06T07:24:10.651671+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.651671+0000', 'last_undegraded': '2024-03-06T07:24:10.651671+0000', 'last_fullsized': '2024-03-06T07:24:10.651671+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:15.524 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.20 is not active+clean 2024-03-06T07:24:15.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.647547+0000', 'last_change': '2024-03-06T07:24:10.647547+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.647547+0000', 'last_undegraded': '2024-03-06T07:24:10.647547+0000', 'last_fullsized': '2024-03-06T07:24:10.647547+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:15.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2024-03-06T07:24:15.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.648981+0000', 'last_change': '2024-03-06T07:24:10.648981+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.648981+0000', 'last_undegraded': '2024-03-06T07:24:10.648981+0000', 'last_fullsized': '2024-03-06T07:24:10.648981+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:15.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1f is not active+clean 2024-03-06T07:24:15.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.648210+0000', 'last_change': '2024-03-06T07:24:10.648210+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.648210+0000', 'last_undegraded': '2024-03-06T07:24:10.648210+0000', 'last_fullsized': '2024-03-06T07:24:10.648210+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:15.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2024-03-06T07:24:15.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.648969+0000', 'last_change': '2024-03-06T07:24:10.648969+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.648969+0000', 'last_undegraded': '2024-03-06T07:24:10.648969+0000', 'last_fullsized': '2024-03-06T07:24:10.648969+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:15.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2024-03-06T07:24:15.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.649490+0000', 'last_change': '2024-03-06T07:24:10.649490+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.649490+0000', 'last_undegraded': '2024-03-06T07:24:10.649490+0000', 'last_fullsized': '2024-03-06T07:24:10.649490+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:15.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1a is not active+clean 2024-03-06T07:24:15.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652779+0000', 'last_change': '2024-03-06T07:24:10.652779+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652779+0000', 'last_undegraded': '2024-03-06T07:24:10.652779+0000', 'last_fullsized': '2024-03-06T07:24:10.652779+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1b is not active+clean 2024-03-06T07:24:15.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.654658+0000', 'last_change': '2024-03-06T07:24:10.654658+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.654658+0000', 'last_undegraded': '2024-03-06T07:24:10.654658+0000', 'last_fullsized': '2024-03-06T07:24:10.654658+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.18 is not active+clean 2024-03-06T07:24:15.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.646559+0000', 'last_change': '2024-03-06T07:24:10.646559+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.646559+0000', 'last_undegraded': '2024-03-06T07:24:10.646559+0000', 'last_fullsized': '2024-03-06T07:24:10.646559+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:15.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.19 is not active+clean 2024-03-06T07:24:15.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652835+0000', 'last_change': '2024-03-06T07:24:10.652835+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652835+0000', 'last_undegraded': '2024-03-06T07:24:10.652835+0000', 'last_fullsized': '2024-03-06T07:24:10.652835+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.16 is not active+clean 2024-03-06T07:24:15.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.654768+0000', 'last_change': '2024-03-06T07:24:10.654768+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.654768+0000', 'last_undegraded': '2024-03-06T07:24:10.654768+0000', 'last_fullsized': '2024-03-06T07:24:10.654768+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:15.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.17 is not active+clean 2024-03-06T07:24:15.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652707+0000', 'last_change': '2024-03-06T07:24:10.652707+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652707+0000', 'last_undegraded': '2024-03-06T07:24:10.652707+0000', 'last_fullsized': '2024-03-06T07:24:10.652707+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.14 is not active+clean 2024-03-06T07:24:15.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.648169+0000', 'last_change': '2024-03-06T07:24:10.648169+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.648169+0000', 'last_undegraded': '2024-03-06T07:24:10.648169+0000', 'last_fullsized': '2024-03-06T07:24:10.648169+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:15.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.15 is not active+clean 2024-03-06T07:24:15.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.653401+0000', 'last_change': '2024-03-06T07:24:10.653401+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.653401+0000', 'last_undegraded': '2024-03-06T07:24:10.653401+0000', 'last_fullsized': '2024-03-06T07:24:10.653401+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2024-03-06T07:24:15.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.648596+0000', 'last_change': '2024-03-06T07:24:10.648596+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.648596+0000', 'last_undegraded': '2024-03-06T07:24:10.648596+0000', 'last_fullsized': '2024-03-06T07:24:10.648596+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:15.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.13 is not active+clean 2024-03-06T07:24:15.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652181+0000', 'last_change': '2024-03-06T07:24:10.652181+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652181+0000', 'last_undegraded': '2024-03-06T07:24:10.652181+0000', 'last_fullsized': '2024-03-06T07:24:10.652181+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.10 is not active+clean 2024-03-06T07:24:15.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652559+0000', 'last_change': '2024-03-06T07:24:10.652559+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652559+0000', 'last_undegraded': '2024-03-06T07:24:10.652559+0000', 'last_fullsized': '2024-03-06T07:24:10.652559+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:15.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.11 is not active+clean 2024-03-06T07:24:15.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.651408+0000', 'last_change': '2024-03-06T07:24:10.651408+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.651408+0000', 'last_undegraded': '2024-03-06T07:24:10.651408+0000', 'last_fullsized': '2024-03-06T07:24:10.651408+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.e is not active+clean 2024-03-06T07:24:15.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652018+0000', 'last_change': '2024-03-06T07:24:10.652018+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652018+0000', 'last_undegraded': '2024-03-06T07:24:10.652018+0000', 'last_fullsized': '2024-03-06T07:24:10.652018+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2024-03-06T07:24:15.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.649490+0000', 'last_change': '2024-03-06T07:24:10.649490+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.649490+0000', 'last_undegraded': '2024-03-06T07:24:10.649490+0000', 'last_fullsized': '2024-03-06T07:24:10.649490+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:15.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.c is not active+clean 2024-03-06T07:24:15.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.654047+0000', 'last_change': '2024-03-06T07:24:10.654047+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.654047+0000', 'last_undegraded': '2024-03-06T07:24:10.654047+0000', 'last_fullsized': '2024-03-06T07:24:10.654047+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.d is not active+clean 2024-03-06T07:24:15.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652981+0000', 'last_change': '2024-03-06T07:24:10.652981+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652981+0000', 'last_undegraded': '2024-03-06T07:24:10.652981+0000', 'last_fullsized': '2024-03-06T07:24:10.652981+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2024-03-06T07:24:15.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.647966+0000', 'last_change': '2024-03-06T07:24:10.647966+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.647966+0000', 'last_undegraded': '2024-03-06T07:24:10.647966+0000', 'last_fullsized': '2024-03-06T07:24:10.647966+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:15.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2024-03-06T07:24:15.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.647790+0000', 'last_change': '2024-03-06T07:24:10.647790+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.647790+0000', 'last_undegraded': '2024-03-06T07:24:10.647790+0000', 'last_fullsized': '2024-03-06T07:24:10.647790+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:15.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2024-03-06T07:24:15.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.648001+0000', 'last_change': '2024-03-06T07:24:10.648001+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.648001+0000', 'last_undegraded': '2024-03-06T07:24:10.648001+0000', 'last_fullsized': '2024-03-06T07:24:10.648001+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:15.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2024-03-06T07:24:15.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.648025+0000', 'last_change': '2024-03-06T07:24:10.648025+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.648025+0000', 'last_undegraded': '2024-03-06T07:24:10.648025+0000', 'last_fullsized': '2024-03-06T07:24:10.648025+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:15.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2024-03-06T07:24:15.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.648442+0000', 'last_change': '2024-03-06T07:24:10.648442+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.648442+0000', 'last_undegraded': '2024-03-06T07:24:10.648442+0000', 'last_fullsized': '2024-03-06T07:24:10.648442+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:15.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.4 is not active+clean 2024-03-06T07:24:15.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.647211+0000', 'last_change': '2024-03-06T07:24:10.647211+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.647211+0000', 'last_undegraded': '2024-03-06T07:24:10.647211+0000', 'last_fullsized': '2024-03-06T07:24:10.647211+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:15.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2024-03-06T07:24:15.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.649057+0000', 'last_change': '2024-03-06T07:24:10.649057+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.649057+0000', 'last_undegraded': '2024-03-06T07:24:10.649057+0000', 'last_fullsized': '2024-03-06T07:24:10.649057+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:15.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2 is not active+clean 2024-03-06T07:24:15.532 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652228+0000', 'last_change': '2024-03-06T07:24:10.652228+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652228+0000', 'last_undegraded': '2024-03-06T07:24:10.652228+0000', 'last_fullsized': '2024-03-06T07:24:10.652228+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3 is not active+clean 2024-03-06T07:24:15.532 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.651712+0000', 'last_change': '2024-03-06T07:24:10.651712+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.651712+0000', 'last_undegraded': '2024-03-06T07:24:10.651712+0000', 'last_fullsized': '2024-03-06T07:24:10.651712+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:15.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.0 is not active+clean 2024-03-06T07:24:15.532 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.647210+0000', 'last_change': '2024-03-06T07:24:10.647210+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.647210+0000', 'last_undegraded': '2024-03-06T07:24:10.647210+0000', 'last_fullsized': '2024-03-06T07:24:10.647210+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:15.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1 is not active+clean 2024-03-06T07:24:15.532 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.652286+0000', 'last_change': '2024-03-06T07:24:10.652286+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.652286+0000', 'last_undegraded': '2024-03-06T07:24:10.652286+0000', 'last_fullsized': '2024-03-06T07:24:10.652286+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.8 is not active+clean 2024-03-06T07:24:15.532 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 57, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:10.651834+0000', 'last_change': '2024-03-06T07:24:10.651834+0000', 'last_active': '2024-03-06T07:24:10.639170+0000', 'last_peered': '2024-03-06T07:24:10.639170+0000', 'last_clean': '2024-03-06T07:24:10.639170+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:10.651834+0000', 'last_undegraded': '2024-03-06T07:24:10.651834+0000', 'last_fullsized': '2024-03-06T07:24:10.651834+0000', 'mapping_epoch': 57, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 57, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:10.639170+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:15.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:24:15.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.655+0000 7fae8173e640 1 Processor -- start 2024-03-06T07:24:15.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.656+0000 7fae8173e640 1 -- start start 2024-03-06T07:24:15.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.656+0000 7fae8173e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae7c162970 0x7fae7c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.656+0000 7fae8173e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fae7c161cf0 0x7fae7c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.656+0000 7fae8173e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c0d22c0 con 0x7fae7c0b4460 2024-03-06T07:24:15.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.656+0000 7fae8173e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7c161b00 con 0x7fae7c161cf0 2024-03-06T07:24:15.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.656+0000 7fae8173e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c0cecc0 con 0x7fae7c162970 2024-03-06T07:24:15.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.657+0000 7fae7b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fae7c161cf0 0x7fae7c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.657+0000 7fae7a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae7c162970 0x7fae7c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.657+0000 7fae7affd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fae7c0b4460 0x7fae7c0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:46682/0 (socket says 172.21.15.59:46682) 2024-03-06T07:24:15.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.657+0000 7fae7affd640 1 -- 172.21.15.59:0/3337488772 learned_addr learned my addr 172.21.15.59:0/3337488772 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:15.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.657+0000 7fae7b7fe640 1 -- 172.21.15.59:0/3337488772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae7c162970 msgr2=0x7fae7c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:15.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.657+0000 7fae7b7fe640 1 --2- 172.21.15.59:0/3337488772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae7c162970 0x7fae7c0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.657+0000 7fae7b7fe640 1 -- 172.21.15.59:0/3337488772 >> v1:172.21.15.164:6789/0 conn(0x7fae7c0b4460 legacy=0x7fae7c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.657+0000 7fae7b7fe640 1 -- 172.21.15.59:0/3337488772 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c0d25a0 con 0x7fae7c161cf0 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.657+0000 7fae7a7fc640 1 --2- 172.21.15.59:0/3337488772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae7c162970 0x7fae7c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.658+0000 7fae7b7fe640 1 --2- 172.21.15.59:0/3337488772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae7c161cf0 0x7fae7c162180 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fae74009d90 tx=0x7fae7402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd297a100776db8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.658+0000 7fae79ffb640 1 -- 172.21.15.59:0/3337488772 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74039070 con 0x7fae7c161cf0 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.658+0000 7fae79ffb640 1 -- 172.21.15.59:0/3337488772 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7402cc50 con 0x7fae7c161cf0 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.658+0000 7fae79ffb640 1 -- 172.21.15.59:0/3337488772 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74034440 con 0x7fae7c161cf0 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.658+0000 7fae8173e640 1 -- 172.21.15.59:0/3337488772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae7c161cf0 msgr2=0x7fae7c162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.658+0000 7fae8173e640 1 --2- 172.21.15.59:0/3337488772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae7c161cf0 0x7fae7c162180 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fae74009d90 tx=0x7fae7402bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.659+0000 7fae8173e640 1 -- 172.21.15.59:0/3337488772 shutdown_connections 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.659+0000 7fae8173e640 1 --2- 172.21.15.59:0/3337488772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae7c161cf0 0x7fae7c162180 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.659+0000 7fae8173e640 1 --2- 172.21.15.59:0/3337488772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae7c162970 0x7fae7c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.659+0000 7fae8173e640 1 -- 172.21.15.59:0/3337488772 >> 172.21.15.59:0/3337488772 conn(0x7fae7c0d3ce0 msgr2=0x7fae7c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.659+0000 7fae8173e640 1 -- 172.21.15.59:0/3337488772 shutdown_connections 2024-03-06T07:24:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.659+0000 7fae8173e640 1 -- 172.21.15.59:0/3337488772 wait complete. 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.660+0000 7fae8173e640 1 Processor -- start 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.660+0000 7fae8173e640 1 -- start start 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae8173e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae7c0b4460 0x7fae7c1562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae8173e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae7c161cf0 0x7fae7c0c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae8173e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae7c162970 0x7fae7c0c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae8173e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c0ce950 con 0x7fae7c161cf0 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae8173e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c0d2b60 con 0x7fae7c0b4460 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae8173e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c0b4840 con 0x7fae7c162970 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae7a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae7c161cf0 0x7fae7c0c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae7a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae7c161cf0 0x7fae7c0c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50944/0 (socket says 172.21.15.59:50944) 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae7a7fc640 1 -- 172.21.15.59:0/3912934100 learned_addr learned my addr 172.21.15.59:0/3912934100 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae7affd640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae7c0b4460 0x7fae7c1562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae7b7fe640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae7c162970 0x7fae7c0c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae7a7fc640 1 -- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae7c162970 msgr2=0x7fae7c0c8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae7a7fc640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae7c162970 0x7fae7c0c8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae7a7fc640 1 -- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae7c0b4460 msgr2=0x7fae7c1562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae7a7fc640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae7c0b4460 0x7fae7c1562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae7a7fc640 1 -- 172.21.15.59:0/3912934100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c1567e0 con 0x7fae7c161cf0 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.661+0000 7fae7a7fc640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae7c161cf0 0x7fae7c0c8430 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00e9b0 tx=0x7fae6c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.662+0000 7fae637fe640 1 -- 172.21.15.59:0/3912934100 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c017070 con 0x7fae7c161cf0 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.662+0000 7fae637fe640 1 -- 172.21.15.59:0/3912934100 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6c00c9a0 con 0x7fae7c161cf0 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.662+0000 7fae8173e640 1 -- 172.21.15.59:0/3912934100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c0cdcf0 con 0x7fae7c161cf0 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.662+0000 7fae637fe640 1 -- 172.21.15.59:0/3912934100 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c00cd90 con 0x7fae7c161cf0 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.662+0000 7fae8173e640 1 -- 172.21.15.59:0/3912934100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7c0cd3c0 con 0x7fae7c161cf0 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.663+0000 7fae8173e640 1 -- 172.21.15.59:0/3912934100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7c04f330 con 0x7fae7c161cf0 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.663+0000 7fae637fe640 1 -- 172.21.15.59:0/3912934100 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae6c01a070 con 0x7fae7c161cf0 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.663+0000 7fae637fe640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae6806b4d0 0x7fae6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.664+0000 7fae637fe640 1 -- 172.21.15.59:0/3912934100 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae6c089c00 con 0x7fae7c161cf0 2024-03-06T07:24:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.664+0000 7fae7affd640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae6806b4d0 0x7fae6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.664+0000 7fae7affd640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae6806b4d0 0x7fae6806d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fae70007fd0 tx=0x7fae70007e80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:15.668+0000 7fae637fe640 1 -- 172.21.15.59:0/3912934100 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae6c090080 con 0x7fae7c161cf0 2024-03-06T07:24:16.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.006+0000 7fae8173e640 1 -- 172.21.15.59:0/3912934100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fae7c0b6900 con 0x7fae7c161cf0 2024-03-06T07:24:16.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.006+0000 7fae637fe640 1 -- 172.21.15.59:0/3912934100 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1453 (secure 0 0 0) 0x7fae7c1567e0 con 0x7fae7c161cf0 2024-03-06T07:24:16.008 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:16.008 INFO:teuthology.orchestra.run.smithi059.stdout:{"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":165,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709701,"num_in_osds":8,"osd_in_since":1709709692,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":262057984,"bytes_avail":772832055296,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":28,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"a","status":"up:active","gid":6075}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-03-06T07:23:53.325507+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:24:16.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 -- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae6806b4d0 msgr2=0x7fae6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae6806b4d0 0x7fae6806d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fae70007fd0 tx=0x7fae70007e80 comp rx=0 tx=0).stop 2024-03-06T07:24:16.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 -- 172.21.15.59:0/3912934100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae7c161cf0 msgr2=0x7fae7c0c8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae7c161cf0 0x7fae7c0c8430 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00e9b0 tx=0x7fae6c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:24:16.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 -- 172.21.15.59:0/3912934100 shutdown_connections 2024-03-06T07:24:16.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae6806b4d0 0x7fae6806d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae7c162970 0x7fae7c0c8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae7c161cf0 0x7fae7c0c8430 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 --2- 172.21.15.59:0/3912934100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae7c0b4460 0x7fae7c1562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 -- 172.21.15.59:0/3912934100 >> 172.21.15.59:0/3912934100 conn(0x7fae7c0d3ce0 msgr2=0x7fae7c0d06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 -- 172.21.15.59:0/3912934100 shutdown_connections 2024-03-06T07:24:16.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.008+0000 7fae8173e640 1 -- 172.21.15.59:0/3912934100 wait complete. 2024-03-06T07:24:16.021 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:24:16.021 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:16.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.121+0000 7fc279884640 1 Processor -- start 2024-03-06T07:24:16.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.121+0000 7fc279884640 1 -- start start 2024-03-06T07:24:16.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc279884640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc2740cf790 0x7fc2740ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc279884640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2740adda0 0x7fc274071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc279884640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc274072220 0x7fc2741619b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc279884640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc27406cf70 con 0x7fc274072220 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc279884640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2740ae240 con 0x7fc2740cf790 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc279884640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2740ae520 con 0x7fc2740adda0 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc2737fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc274072220 0x7fc2741619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc2737fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc274072220 0x7fc2741619b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50946/0 (socket says 172.21.15.59:50946) 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc2737fe640 1 -- 172.21.15.59:0/709531732 learned_addr learned my addr 172.21.15.59:0/709531732 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc2727fc640 1 --2- 172.21.15.59:0/709531732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2740adda0 0x7fc274071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc272ffd640 1 --2- 172.21.15.59:0/709531732 >> v2:172.21.15.164:3300/0 conn(0x7fc2740cf790 0x7fc2740ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc2737fe640 1 -- 172.21.15.59:0/709531732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2740adda0 msgr2=0x7fc274071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc2737fe640 1 --2- 172.21.15.59:0/709531732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2740adda0 0x7fc274071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc2737fe640 1 -- 172.21.15.59:0/709531732 >> v2:172.21.15.164:3300/0 conn(0x7fc2740cf790 msgr2=0x7fc2740ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc2737fe640 1 --2- 172.21.15.59:0/709531732 >> v2:172.21.15.164:3300/0 conn(0x7fc2740cf790 0x7fc2740ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc2737fe640 1 -- 172.21.15.59:0/709531732 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2740ae800 con 0x7fc274072220 2024-03-06T07:24:16.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.122+0000 7fc272ffd640 1 --2- 172.21.15.59:0/709531732 >> v2:172.21.15.164:3300/0 conn(0x7fc2740cf790 0x7fc2740ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.123+0000 7fc2737fe640 1 --2- 172.21.15.59:0/709531732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc274072220 0x7fc2741619b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc2740b27b0 tx=0x7fc26c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f10ab14825e5989 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.123+0000 7fc271ffb640 1 -- 172.21.15.59:0/709531732 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c02ca70 con 0x7fc274072220 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.123+0000 7fc271ffb640 1 -- 172.21.15.59:0/709531732 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26c02cc10 con 0x7fc274072220 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.123+0000 7fc271ffb640 1 -- 172.21.15.59:0/709531732 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c0343f0 con 0x7fc274072220 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.123+0000 7fc279884640 1 -- 172.21.15.59:0/709531732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc274072220 msgr2=0x7fc2741619b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.123+0000 7fc279884640 1 --2- 172.21.15.59:0/709531732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc274072220 0x7fc2741619b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc2740b27b0 tx=0x7fc26c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.124+0000 7fc279884640 1 -- 172.21.15.59:0/709531732 shutdown_connections 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.124+0000 7fc279884640 1 --2- 172.21.15.59:0/709531732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc274072220 0x7fc2741619b0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.124+0000 7fc279884640 1 --2- 172.21.15.59:0/709531732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2740adda0 0x7fc274071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.124+0000 7fc279884640 1 --2- 172.21.15.59:0/709531732 >> v2:172.21.15.164:3300/0 conn(0x7fc2740cf790 0x7fc2740ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.124+0000 7fc279884640 1 -- 172.21.15.59:0/709531732 >> 172.21.15.59:0/709531732 conn(0x7fc2740b1a90 msgr2=0x7fc2740b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.124+0000 7fc279884640 1 -- 172.21.15.59:0/709531732 shutdown_connections 2024-03-06T07:24:16.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.124+0000 7fc279884640 1 -- 172.21.15.59:0/709531732 wait complete. 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.125+0000 7fc279884640 1 Processor -- start 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.125+0000 7fc279884640 1 -- start start 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc279884640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc274072220 0x7fc274147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc279884640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2740adda0 0x7fc2741f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc279884640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2740cf790 0x7fc274152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc279884640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27415e700 con 0x7fc2740adda0 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc279884640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27415e580 con 0x7fc2740cf790 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc279884640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc274161ef0 con 0x7fc274072220 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc2727fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2740adda0 0x7fc2741f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc2727fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2740adda0 0x7fc2741f3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50956/0 (socket says 172.21.15.59:50956) 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc2727fc640 1 -- 172.21.15.59:0/1500326581 learned_addr learned my addr 172.21.15.59:0/1500326581 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc272ffd640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc274072220 0x7fc274147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc2737fe640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2740cf790 0x7fc274152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc2727fc640 1 -- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc274072220 msgr2=0x7fc274147960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc2727fc640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc274072220 0x7fc274147960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc2727fc640 1 -- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2740cf790 msgr2=0x7fc274152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc2727fc640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2740cf790 0x7fc274152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc2727fc640 1 -- 172.21.15.59:0/1500326581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2741f4840 con 0x7fc2740adda0 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc2737fe640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2740cf790 0x7fc274152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc2727fc640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2740adda0 0x7fc2741f3e70 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc26800e8c0 tx=0x7fc26800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc25bfff640 1 -- 172.21.15.59:0/1500326581 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc268017070 con 0x7fc2740adda0 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc25bfff640 1 -- 172.21.15.59:0/1500326581 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26800c940 con 0x7fc2740adda0 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.126+0000 7fc279884640 1 -- 172.21.15.59:0/1500326581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2741f4b20 con 0x7fc2740adda0 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.127+0000 7fc25bfff640 1 -- 172.21.15.59:0/1500326581 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26800cd10 con 0x7fc2740adda0 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.127+0000 7fc279884640 1 -- 172.21.15.59:0/1500326581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc274152150 con 0x7fc2740adda0 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.128+0000 7fc279884640 1 -- 172.21.15.59:0/1500326581 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27404f330 con 0x7fc2740adda0 2024-03-06T07:24:16.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.128+0000 7fc25bfff640 1 -- 172.21.15.59:0/1500326581 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc26801a070 con 0x7fc2740adda0 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.128+0000 7fc25bfff640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc26006b4d0 0x7fc26006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.129+0000 7fc25bfff640 1 -- 172.21.15.59:0/1500326581 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc26808a050 con 0x7fc2740adda0 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.129+0000 7fc272ffd640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc26006b4d0 0x7fc26006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:16.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.129+0000 7fc272ffd640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc26006b4d0 0x7fc26006d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc264006fd0 tx=0x7fc264008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:16.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.133+0000 7fc25bfff640 1 -- 172.21.15.59:0/1500326581 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc268058910 con 0x7fc2740adda0 2024-03-06T07:24:16.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.311+0000 7fc279884640 1 -- 172.21.15.59:0/1500326581 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2740afa00 con 0x7fc26006b4d0 2024-03-06T07:24:16.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.318+0000 7fc25bfff640 1 -- 172.21.15.59:0/1500326581 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246638 (secure 0 0 0) 0x7fc2740afa00 con 0x7fc26006b4d0 2024-03-06T07:24:16.322 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:16.323 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:24:16.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 -- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc26006b4d0 msgr2=0x7fc26006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc26006b4d0 0x7fc26006d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc264006fd0 tx=0x7fc264008040 comp rx=0 tx=0).stop 2024-03-06T07:24:16.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 -- 172.21.15.59:0/1500326581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2740adda0 msgr2=0x7fc2741f3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:16.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2740adda0 0x7fc2741f3e70 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc26800e8c0 tx=0x7fc26800ed80 comp rx=0 tx=0).stop 2024-03-06T07:24:16.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 -- 172.21.15.59:0/1500326581 shutdown_connections 2024-03-06T07:24:16.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc26006b4d0 0x7fc26006d990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2740cf790 0x7fc274152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc2740adda0 0x7fc2741f3e70 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 --2- 172.21.15.59:0/1500326581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc274072220 0x7fc274147960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:16.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 -- 172.21.15.59:0/1500326581 >> 172.21.15.59:0/1500326581 conn(0x7fc2740b1a90 msgr2=0x7fc27415f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:16.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 -- 172.21.15.59:0/1500326581 shutdown_connections 2024-03-06T07:24:16.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:16.321+0000 7fc279884640 1 -- 172.21.15.59:0/1500326581 wait complete. 2024-03-06T07:24:16.337 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-03-06T07:24:15.330142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255916,"kb_used_data":6456,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718804,"statfs":{"total":773094113280,"available":772832055296,"internally_reserved":0,"allocated":6610944,"data_stored":3417244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.690206"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653222+0000","last_change":"2024-03-06T07:24:10.653222+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653222+0000","last_undegraded":"2024-03-06T07:24:10.653222+0000","last_fullsized":"2024-03-06T07:24:10.653222+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654247+0000","last_change":"2024-03-06T07:24:10.654247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654247+0000","last_undegraded":"2024-03-06T07:24:10.654247+0000","last_fullsized":"2024-03-06T07:24:10.654247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653845+0000","last_change":"2024-03-06T07:24:10.653845+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653845+0000","last_undegraded":"2024-03-06T07:24:10.653845+0000","last_fullsized":"2024-03-06T07:24:10.653845+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649798+0000","last_change":"2024-03-06T07:24:10.649798+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649798+0000","last_undegraded":"2024-03-06T07:24:10.649798+0000","last_fullsized":"2024-03-06T07:24:10.649798+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653052+0000","last_change":"2024-03-06T07:24:10.653052+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653052+0000","last_undegraded":"2024-03-06T07:24:10.653052+0000","last_fullsized":"2024-03-06T07:24:10.653052+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649635+0000","last_change":"2024-03-06T07:24:10.649635+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649635+0000","last_undegraded":"2024-03-06T07:24:10.649635+0000","last_fullsized":"2024-03-06T07:24:10.649635+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654247+0000","last_change":"2024-03-06T07:24:10.654247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654247+0000","last_undegraded":"2024-03-06T07:24:10.654247+0000","last_fullsized":"2024-03-06T07:24:10.654247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653163+0000","last_change":"2024-03-06T07:24:10.653163+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653163+0000","last_undegraded":"2024-03-06T07:24:10.653163+0000","last_fullsized":"2024-03-06T07:24:10.653163+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.655335+0000","last_change":"2024-03-06T07:24:10.655335+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.655335+0000","last_undegraded":"2024-03-06T07:24:10.655335+0000","last_fullsized":"2024-03-06T07:24:10.655335+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648323+0000","last_change":"2024-03-06T07:24:10.648323+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648323+0000","last_undegraded":"2024-03-06T07:24:10.648323+0000","last_fullsized":"2024-03-06T07:24:10.648323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648785+0000","last_change":"2024-03-06T07:24:10.648785+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648785+0000","last_undegraded":"2024-03-06T07:24:10.648785+0000","last_fullsized":"2024-03-06T07:24:10.648785+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649164+0000","last_change":"2024-03-06T07:24:10.649164+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649164+0000","last_undegraded":"2024-03-06T07:24:10.649164+0000","last_fullsized":"2024-03-06T07:24:10.649164+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.646039+0000","last_change":"2024-03-06T07:24:10.646039+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.646039+0000","last_undegraded":"2024-03-06T07:24:10.646039+0000","last_fullsized":"2024-03-06T07:24:10.646039+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653571+0000","last_change":"2024-03-06T07:24:10.653571+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653571+0000","last_undegraded":"2024-03-06T07:24:10.653571+0000","last_fullsized":"2024-03-06T07:24:10.653571+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653868+0000","last_change":"2024-03-06T07:24:10.653868+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653868+0000","last_undegraded":"2024-03-06T07:24:10.653868+0000","last_fullsized":"2024-03-06T07:24:10.653868+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.650079+0000","last_change":"2024-03-06T07:24:10.650079+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.650079+0000","last_undegraded":"2024-03-06T07:24:10.650079+0000","last_fullsized":"2024-03-06T07:24:10.650079+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652629+0000","last_change":"2024-03-06T07:24:10.652629+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652629+0000","last_undegraded":"2024-03-06T07:24:10.652629+0000","last_fullsized":"2024-03-06T07:24:10.652629+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653025+0000","last_change":"2024-03-06T07:24:10.653025+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653025+0000","last_undegraded":"2024-03-06T07:24:10.653025+0000","last_fullsized":"2024-03-06T07:24:10.653025+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653633+0000","last_change":"2024-03-06T07:24:10.653633+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653633+0000","last_undegraded":"2024-03-06T07:24:10.653633+0000","last_fullsized":"2024-03-06T07:24:10.653633+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.655332+0000","last_change":"2024-03-06T07:24:10.655332+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.655332+0000","last_undegraded":"2024-03-06T07:24:10.655332+0000","last_fullsized":"2024-03-06T07:24:10.655332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652655+0000","last_change":"2024-03-06T07:24:10.652655+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652655+0000","last_undegraded":"2024-03-06T07:24:10.652655+0000","last_fullsized":"2024-03-06T07:24:10.652655+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653247+0000","last_change":"2024-03-06T07:24:10.653247+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653247+0000","last_undegraded":"2024-03-06T07:24:10.653247+0000","last_fullsized":"2024-03-06T07:24:10.653247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651973+0000","last_change":"2024-03-06T07:24:10.651973+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651973+0000","last_undegraded":"2024-03-06T07:24:10.651973+0000","last_fullsized":"2024-03-06T07:24:10.651973+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647134+0000","last_change":"2024-03-06T07:24:10.647134+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647134+0000","last_undegraded":"2024-03-06T07:24:10.647134+0000","last_fullsized":"2024-03-06T07:24:10.647134+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652823+0000","last_change":"2024-03-06T07:24:10.652823+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652823+0000","last_undegraded":"2024-03-06T07:24:10.652823+0000","last_fullsized":"2024-03-06T07:24:10.652823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649078+0000","last_change":"2024-03-06T07:24:10.649078+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649078+0000","last_undegraded":"2024-03-06T07:24:10.649078+0000","last_fullsized":"2024-03-06T07:24:10.649078+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649913+0000","last_change":"2024-03-06T07:24:10.649913+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649913+0000","last_undegraded":"2024-03-06T07:24:10.649913+0000","last_fullsized":"2024-03-06T07:24:10.649913+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653647+0000","last_change":"2024-03-06T07:24:10.653647+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653647+0000","last_undegraded":"2024-03-06T07:24:10.653647+0000","last_fullsized":"2024-03-06T07:24:10.653647+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653928+0000","last_change":"2024-03-06T07:24:10.653928+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653928+0000","last_undegraded":"2024-03-06T07:24:10.653928+0000","last_fullsized":"2024-03-06T07:24:10.653928+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651185+0000","last_change":"2024-03-06T07:24:10.651185+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651185+0000","last_undegraded":"2024-03-06T07:24:10.651185+0000","last_fullsized":"2024-03-06T07:24:10.651185+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651671+0000","last_change":"2024-03-06T07:24:10.651671+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651671+0000","last_undegraded":"2024-03-06T07:24:10.651671+0000","last_fullsized":"2024-03-06T07:24:10.651671+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647547+0000","last_change":"2024-03-06T07:24:10.647547+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647547+0000","last_undegraded":"2024-03-06T07:24:10.647547+0000","last_fullsized":"2024-03-06T07:24:10.647547+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648981+0000","last_change":"2024-03-06T07:24:10.648981+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648981+0000","last_undegraded":"2024-03-06T07:24:10.648981+0000","last_fullsized":"2024-03-06T07:24:10.648981+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661442+0000","last_change":"2024-03-06T07:24:10.658033+0000","last_active":"2024-03-06T07:24:10.661442+0000","last_peered":"2024-03-06T07:24:10.661442+0000","last_clean":"2024-03-06T07:24:10.661442+0000","last_became_active":"2024-03-06T07:24:10.657766+0000","last_became_peered":"2024-03-06T07:24:10.657766+0000","last_unstale":"2024-03-06T07:24:10.661442+0000","last_undegraded":"2024-03-06T07:24:10.661442+0000","last_fullsized":"2024-03-06T07:24:10.661442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648210+0000","last_change":"2024-03-06T07:24:10.648210+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648210+0000","last_undegraded":"2024-03-06T07:24:10.648210+0000","last_fullsized":"2024-03-06T07:24:10.648210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660926+0000","last_change":"2024-03-06T07:24:10.659965+0000","last_active":"2024-03-06T07:24:10.660926+0000","last_peered":"2024-03-06T07:24:10.660926+0000","last_clean":"2024-03-06T07:24:10.660926+0000","last_became_active":"2024-03-06T07:24:10.659581+0000","last_became_peered":"2024-03-06T07:24:10.659581+0000","last_unstale":"2024-03-06T07:24:10.660926+0000","last_undegraded":"2024-03-06T07:24:10.660926+0000","last_fullsized":"2024-03-06T07:24:10.660926+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:39.589222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648969+0000","last_change":"2024-03-06T07:24:10.648969+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648969+0000","last_undegraded":"2024-03-06T07:24:10.648969+0000","last_fullsized":"2024-03-06T07:24:10.648969+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666485+0000","last_change":"2024-03-06T07:24:10.662366+0000","last_active":"2024-03-06T07:24:10.666485+0000","last_peered":"2024-03-06T07:24:10.666485+0000","last_clean":"2024-03-06T07:24:10.666485+0000","last_became_active":"2024-03-06T07:24:10.662075+0000","last_became_peered":"2024-03-06T07:24:10.662075+0000","last_unstale":"2024-03-06T07:24:10.666485+0000","last_undegraded":"2024-03-06T07:24:10.666485+0000","last_fullsized":"2024-03-06T07:24:10.666485+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.249754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649490+0000","last_change":"2024-03-06T07:24:10.649490+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649490+0000","last_undegraded":"2024-03-06T07:24:10.649490+0000","last_fullsized":"2024-03-06T07:24:10.649490+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660644+0000","last_change":"2024-03-06T07:24:10.659614+0000","last_active":"2024-03-06T07:24:10.660644+0000","last_peered":"2024-03-06T07:24:10.660644+0000","last_clean":"2024-03-06T07:24:10.660644+0000","last_became_active":"2024-03-06T07:24:10.659301+0000","last_became_peered":"2024-03-06T07:24:10.659301+0000","last_unstale":"2024-03-06T07:24:10.660644+0000","last_undegraded":"2024-03-06T07:24:10.660644+0000","last_fullsized":"2024-03-06T07:24:10.660644+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652779+0000","last_change":"2024-03-06T07:24:10.652779+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652779+0000","last_undegraded":"2024-03-06T07:24:10.652779+0000","last_fullsized":"2024-03-06T07:24:10.652779+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663858+0000","last_change":"2024-03-06T07:24:10.661362+0000","last_active":"2024-03-06T07:24:10.663858+0000","last_peered":"2024-03-06T07:24:10.663858+0000","last_clean":"2024-03-06T07:24:10.663858+0000","last_became_active":"2024-03-06T07:24:10.661069+0000","last_became_peered":"2024-03-06T07:24:10.661069+0000","last_unstale":"2024-03-06T07:24:10.663858+0000","last_undegraded":"2024-03-06T07:24:10.663858+0000","last_fullsized":"2024-03-06T07:24:10.663858+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:47.652320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654658+0000","last_change":"2024-03-06T07:24:10.654658+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654658+0000","last_undegraded":"2024-03-06T07:24:10.654658+0000","last_fullsized":"2024-03-06T07:24:10.654658+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.655589+0000","last_change":"2024-03-06T07:24:10.653845+0000","last_active":"2024-03-06T07:24:10.655589+0000","last_peered":"2024-03-06T07:24:10.655589+0000","last_clean":"2024-03-06T07:24:10.655589+0000","last_became_active":"2024-03-06T07:24:10.653600+0000","last_became_peered":"2024-03-06T07:24:10.653600+0000","last_unstale":"2024-03-06T07:24:10.655589+0000","last_undegraded":"2024-03-06T07:24:10.655589+0000","last_fullsized":"2024-03-06T07:24:10.655589+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:05.337286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.646559+0000","last_change":"2024-03-06T07:24:10.646559+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.646559+0000","last_undegraded":"2024-03-06T07:24:10.646559+0000","last_fullsized":"2024-03-06T07:24:10.646559+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663129+0000","last_change":"2024-03-06T07:24:10.660990+0000","last_active":"2024-03-06T07:24:10.663129+0000","last_peered":"2024-03-06T07:24:10.663129+0000","last_clean":"2024-03-06T07:24:10.663129+0000","last_became_active":"2024-03-06T07:24:10.660680+0000","last_became_peered":"2024-03-06T07:24:10.660680+0000","last_unstale":"2024-03-06T07:24:10.663129+0000","last_undegraded":"2024-03-06T07:24:10.663129+0000","last_fullsized":"2024-03-06T07:24:10.663129+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.249754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652835+0000","last_change":"2024-03-06T07:24:10.652835+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652835+0000","last_undegraded":"2024-03-06T07:24:10.652835+0000","last_fullsized":"2024-03-06T07:24:10.652835+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660821+0000","last_change":"2024-03-06T07:24:10.659499+0000","last_active":"2024-03-06T07:24:10.660821+0000","last_peered":"2024-03-06T07:24:10.660821+0000","last_clean":"2024-03-06T07:24:10.660821+0000","last_became_active":"2024-03-06T07:24:10.659224+0000","last_became_peered":"2024-03-06T07:24:10.659224+0000","last_unstale":"2024-03-06T07:24:10.660821+0000","last_undegraded":"2024-03-06T07:24:10.660821+0000","last_fullsized":"2024-03-06T07:24:10.660821+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:37.419239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654768+0000","last_change":"2024-03-06T07:24:10.654768+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654768+0000","last_undegraded":"2024-03-06T07:24:10.654768+0000","last_fullsized":"2024-03-06T07:24:10.654768+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664029+0000","last_change":"2024-03-06T07:24:10.662290+0000","last_active":"2024-03-06T07:24:10.664029+0000","last_peered":"2024-03-06T07:24:10.664029+0000","last_clean":"2024-03-06T07:24:10.664029+0000","last_became_active":"2024-03-06T07:24:10.662003+0000","last_became_peered":"2024-03-06T07:24:10.662003+0000","last_unstale":"2024-03-06T07:24:10.664029+0000","last_undegraded":"2024-03-06T07:24:10.664029+0000","last_fullsized":"2024-03-06T07:24:10.664029+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652707+0000","last_change":"2024-03-06T07:24:10.652707+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652707+0000","last_undegraded":"2024-03-06T07:24:10.652707+0000","last_fullsized":"2024-03-06T07:24:10.652707+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666379+0000","last_change":"2024-03-06T07:24:10.663647+0000","last_active":"2024-03-06T07:24:10.666379+0000","last_peered":"2024-03-06T07:24:10.666379+0000","last_clean":"2024-03-06T07:24:10.666379+0000","last_became_active":"2024-03-06T07:24:10.663378+0000","last_became_peered":"2024-03-06T07:24:10.663378+0000","last_unstale":"2024-03-06T07:24:10.666379+0000","last_undegraded":"2024-03-06T07:24:10.666379+0000","last_fullsized":"2024-03-06T07:24:10.666379+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:21.084766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648169+0000","last_change":"2024-03-06T07:24:10.648169+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648169+0000","last_undegraded":"2024-03-06T07:24:10.648169+0000","last_fullsized":"2024-03-06T07:24:10.648169+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.668926+0000","last_change":"2024-03-06T07:24:10.665110+0000","last_active":"2024-03-06T07:24:10.668926+0000","last_peered":"2024-03-06T07:24:10.668926+0000","last_clean":"2024-03-06T07:24:10.668926+0000","last_became_active":"2024-03-06T07:24:10.664833+0000","last_became_peered":"2024-03-06T07:24:10.664833+0000","last_unstale":"2024-03-06T07:24:10.668926+0000","last_undegraded":"2024-03-06T07:24:10.668926+0000","last_fullsized":"2024-03-06T07:24:10.668926+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.653401+0000","last_change":"2024-03-06T07:24:10.653401+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.653401+0000","last_undegraded":"2024-03-06T07:24:10.653401+0000","last_fullsized":"2024-03-06T07:24:10.653401+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664163+0000","last_change":"2024-03-06T07:24:10.662566+0000","last_active":"2024-03-06T07:24:10.664163+0000","last_peered":"2024-03-06T07:24:10.664163+0000","last_clean":"2024-03-06T07:24:10.664163+0000","last_became_active":"2024-03-06T07:24:10.662201+0000","last_became_peered":"2024-03-06T07:24:10.662201+0000","last_unstale":"2024-03-06T07:24:10.664163+0000","last_undegraded":"2024-03-06T07:24:10.664163+0000","last_fullsized":"2024-03-06T07:24:10.664163+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:00.568465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648596+0000","last_change":"2024-03-06T07:24:10.648596+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648596+0000","last_undegraded":"2024-03-06T07:24:10.648596+0000","last_fullsized":"2024-03-06T07:24:10.648596+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666591+0000","last_change":"2024-03-06T07:24:10.664502+0000","last_active":"2024-03-06T07:24:10.666591+0000","last_peered":"2024-03-06T07:24:10.666591+0000","last_clean":"2024-03-06T07:24:10.666591+0000","last_became_active":"2024-03-06T07:24:10.664248+0000","last_became_peered":"2024-03-06T07:24:10.664248+0000","last_unstale":"2024-03-06T07:24:10.666591+0000","last_undegraded":"2024-03-06T07:24:10.666591+0000","last_fullsized":"2024-03-06T07:24:10.666591+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:07.375756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652181+0000","last_change":"2024-03-06T07:24:10.652181+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652181+0000","last_undegraded":"2024-03-06T07:24:10.652181+0000","last_fullsized":"2024-03-06T07:24:10.652181+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658397+0000","last_change":"2024-03-06T07:24:10.655420+0000","last_active":"2024-03-06T07:24:10.658397+0000","last_peered":"2024-03-06T07:24:10.658397+0000","last_clean":"2024-03-06T07:24:10.658397+0000","last_became_active":"2024-03-06T07:24:10.655148+0000","last_became_peered":"2024-03-06T07:24:10.655148+0000","last_unstale":"2024-03-06T07:24:10.658397+0000","last_undegraded":"2024-03-06T07:24:10.658397+0000","last_fullsized":"2024-03-06T07:24:10.658397+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:16.948447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652559+0000","last_change":"2024-03-06T07:24:10.652559+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652559+0000","last_undegraded":"2024-03-06T07:24:10.652559+0000","last_fullsized":"2024-03-06T07:24:10.652559+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661144+0000","last_change":"2024-03-06T07:24:10.659555+0000","last_active":"2024-03-06T07:24:10.661144+0000","last_peered":"2024-03-06T07:24:10.661144+0000","last_clean":"2024-03-06T07:24:10.661144+0000","last_became_active":"2024-03-06T07:24:10.659285+0000","last_became_peered":"2024-03-06T07:24:10.659285+0000","last_unstale":"2024-03-06T07:24:10.661144+0000","last_undegraded":"2024-03-06T07:24:10.661144+0000","last_fullsized":"2024-03-06T07:24:10.661144+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651408+0000","last_change":"2024-03-06T07:24:10.651408+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651408+0000","last_undegraded":"2024-03-06T07:24:10.651408+0000","last_fullsized":"2024-03-06T07:24:10.651408+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664001+0000","last_change":"2024-03-06T07:24:10.661441+0000","last_active":"2024-03-06T07:24:10.664001+0000","last_peered":"2024-03-06T07:24:10.664001+0000","last_clean":"2024-03-06T07:24:10.664001+0000","last_became_active":"2024-03-06T07:24:10.661157+0000","last_became_peered":"2024-03-06T07:24:10.661157+0000","last_unstale":"2024-03-06T07:24:10.664001+0000","last_undegraded":"2024-03-06T07:24:10.664001+0000","last_fullsized":"2024-03-06T07:24:10.664001+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652018+0000","last_change":"2024-03-06T07:24:10.652018+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652018+0000","last_undegraded":"2024-03-06T07:24:10.652018+0000","last_fullsized":"2024-03-06T07:24:10.652018+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.659358+0000","last_change":"2024-03-06T07:24:10.658071+0000","last_active":"2024-03-06T07:24:10.659358+0000","last_peered":"2024-03-06T07:24:10.659358+0000","last_clean":"2024-03-06T07:24:10.659358+0000","last_became_active":"2024-03-06T07:24:10.657790+0000","last_became_peered":"2024-03-06T07:24:10.657790+0000","last_unstale":"2024-03-06T07:24:10.659358+0000","last_undegraded":"2024-03-06T07:24:10.659358+0000","last_fullsized":"2024-03-06T07:24:10.659358+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:33.615309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649490+0000","last_change":"2024-03-06T07:24:10.649490+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649490+0000","last_undegraded":"2024-03-06T07:24:10.649490+0000","last_fullsized":"2024-03-06T07:24:10.649490+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661030+0000","last_change":"2024-03-06T07:24:10.658700+0000","last_active":"2024-03-06T07:24:10.661030+0000","last_peered":"2024-03-06T07:24:10.661030+0000","last_clean":"2024-03-06T07:24:10.661030+0000","last_became_active":"2024-03-06T07:24:10.658430+0000","last_became_peered":"2024-03-06T07:24:10.658430+0000","last_unstale":"2024-03-06T07:24:10.661030+0000","last_undegraded":"2024-03-06T07:24:10.661030+0000","last_fullsized":"2024-03-06T07:24:10.661030+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.654047+0000","last_change":"2024-03-06T07:24:10.654047+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.654047+0000","last_undegraded":"2024-03-06T07:24:10.654047+0000","last_fullsized":"2024-03-06T07:24:10.654047+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.661932+0000","last_change":"2024-03-06T07:24:10.659862+0000","last_active":"2024-03-06T07:24:10.661932+0000","last_peered":"2024-03-06T07:24:10.661932+0000","last_clean":"2024-03-06T07:24:10.661932+0000","last_became_active":"2024-03-06T07:24:10.659578+0000","last_became_peered":"2024-03-06T07:24:10.659578+0000","last_unstale":"2024-03-06T07:24:10.661932+0000","last_undegraded":"2024-03-06T07:24:10.661932+0000","last_fullsized":"2024-03-06T07:24:10.661932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:59.529864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652981+0000","last_change":"2024-03-06T07:24:10.652981+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652981+0000","last_undegraded":"2024-03-06T07:24:10.652981+0000","last_fullsized":"2024-03-06T07:24:10.652981+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.659947+0000","last_change":"2024-03-06T07:24:10.658442+0000","last_active":"2024-03-06T07:24:10.659947+0000","last_peered":"2024-03-06T07:24:10.659947+0000","last_clean":"2024-03-06T07:24:10.659947+0000","last_became_active":"2024-03-06T07:24:10.658200+0000","last_became_peered":"2024-03-06T07:24:10.658200+0000","last_unstale":"2024-03-06T07:24:10.659947+0000","last_undegraded":"2024-03-06T07:24:10.659947+0000","last_fullsized":"2024-03-06T07:24:10.659947+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:01.152640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647966+0000","last_change":"2024-03-06T07:24:10.647966+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647966+0000","last_undegraded":"2024-03-06T07:24:10.647966+0000","last_fullsized":"2024-03-06T07:24:10.647966+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.666422+0000","last_change":"2024-03-06T07:24:10.662365+0000","last_active":"2024-03-06T07:24:10.666422+0000","last_peered":"2024-03-06T07:24:10.666422+0000","last_clean":"2024-03-06T07:24:10.666422+0000","last_became_active":"2024-03-06T07:24:10.662073+0000","last_became_peered":"2024-03-06T07:24:10.662073+0000","last_unstale":"2024-03-06T07:24:10.666422+0000","last_undegraded":"2024-03-06T07:24:10.666422+0000","last_fullsized":"2024-03-06T07:24:10.666422+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:59.892449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647790+0000","last_change":"2024-03-06T07:24:10.647790+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647790+0000","last_undegraded":"2024-03-06T07:24:10.647790+0000","last_fullsized":"2024-03-06T07:24:10.647790+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.660165+0000","last_change":"2024-03-06T07:24:10.659174+0000","last_active":"2024-03-06T07:24:10.660165+0000","last_peered":"2024-03-06T07:24:10.660165+0000","last_clean":"2024-03-06T07:24:10.660165+0000","last_became_active":"2024-03-06T07:24:10.658834+0000","last_became_peered":"2024-03-06T07:24:10.658834+0000","last_unstale":"2024-03-06T07:24:10.660165+0000","last_undegraded":"2024-03-06T07:24:10.660165+0000","last_fullsized":"2024-03-06T07:24:10.660165+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:37.876251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648001+0000","last_change":"2024-03-06T07:24:10.648001+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648001+0000","last_undegraded":"2024-03-06T07:24:10.648001+0000","last_fullsized":"2024-03-06T07:24:10.648001+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657498+0000","last_change":"2024-03-06T07:24:10.656628+0000","last_active":"2024-03-06T07:24:10.657498+0000","last_peered":"2024-03-06T07:24:10.657498+0000","last_clean":"2024-03-06T07:24:10.657498+0000","last_became_active":"2024-03-06T07:24:10.656271+0000","last_became_peered":"2024-03-06T07:24:10.656271+0000","last_unstale":"2024-03-06T07:24:10.657498+0000","last_undegraded":"2024-03-06T07:24:10.657498+0000","last_fullsized":"2024-03-06T07:24:10.657498+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:55.005932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648025+0000","last_change":"2024-03-06T07:24:10.648025+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648025+0000","last_undegraded":"2024-03-06T07:24:10.648025+0000","last_fullsized":"2024-03-06T07:24:10.648025+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664836+0000","last_change":"2024-03-06T07:24:10.663013+0000","last_active":"2024-03-06T07:24:10.664836+0000","last_peered":"2024-03-06T07:24:10.664836+0000","last_clean":"2024-03-06T07:24:10.664836+0000","last_became_active":"2024-03-06T07:24:10.662736+0000","last_became_peered":"2024-03-06T07:24:10.662736+0000","last_unstale":"2024-03-06T07:24:10.664836+0000","last_undegraded":"2024-03-06T07:24:10.664836+0000","last_fullsized":"2024-03-06T07:24:10.664836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:34.328658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.648442+0000","last_change":"2024-03-06T07:24:10.648442+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.648442+0000","last_undegraded":"2024-03-06T07:24:10.648442+0000","last_fullsized":"2024-03-06T07:24:10.648442+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658858+0000","last_change":"2024-03-06T07:24:10.654750+0000","last_active":"2024-03-06T07:24:10.658858+0000","last_peered":"2024-03-06T07:24:10.658858+0000","last_clean":"2024-03-06T07:24:10.658858+0000","last_became_active":"2024-03-06T07:24:10.654476+0000","last_became_peered":"2024-03-06T07:24:10.654476+0000","last_unstale":"2024-03-06T07:24:10.658858+0000","last_undegraded":"2024-03-06T07:24:10.658858+0000","last_fullsized":"2024-03-06T07:24:10.658858+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:55.005932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647211+0000","last_change":"2024-03-06T07:24:10.647211+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647211+0000","last_undegraded":"2024-03-06T07:24:10.647211+0000","last_fullsized":"2024-03-06T07:24:10.647211+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658862+0000","last_change":"2024-03-06T07:24:10.657981+0000","last_active":"2024-03-06T07:24:10.658862+0000","last_peered":"2024-03-06T07:24:10.658862+0000","last_clean":"2024-03-06T07:24:10.658862+0000","last_became_active":"2024-03-06T07:24:10.657616+0000","last_became_peered":"2024-03-06T07:24:10.657616+0000","last_unstale":"2024-03-06T07:24:10.658862+0000","last_undegraded":"2024-03-06T07:24:10.658862+0000","last_fullsized":"2024-03-06T07:24:10.658862+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:45.308028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.649057+0000","last_change":"2024-03-06T07:24:10.649057+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.649057+0000","last_undegraded":"2024-03-06T07:24:10.649057+0000","last_fullsized":"2024-03-06T07:24:10.649057+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.663759+0000","last_change":"2024-03-06T07:24:10.661755+0000","last_active":"2024-03-06T07:24:10.663759+0000","last_peered":"2024-03-06T07:24:10.663759+0000","last_clean":"2024-03-06T07:24:10.663759+0000","last_became_active":"2024-03-06T07:24:10.661432+0000","last_became_peered":"2024-03-06T07:24:10.661432+0000","last_unstale":"2024-03-06T07:24:10.663759+0000","last_undegraded":"2024-03-06T07:24:10.663759+0000","last_fullsized":"2024-03-06T07:24:10.663759+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:15.720920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652228+0000","last_change":"2024-03-06T07:24:10.652228+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652228+0000","last_undegraded":"2024-03-06T07:24:10.652228+0000","last_fullsized":"2024-03-06T07:24:10.652228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657656+0000","last_change":"2024-03-06T07:24:10.656363+0000","last_active":"2024-03-06T07:24:10.657656+0000","last_peered":"2024-03-06T07:24:10.657656+0000","last_clean":"2024-03-06T07:24:10.657656+0000","last_became_active":"2024-03-06T07:24:10.656056+0000","last_became_peered":"2024-03-06T07:24:10.656056+0000","last_unstale":"2024-03-06T07:24:10.657656+0000","last_undegraded":"2024-03-06T07:24:10.657656+0000","last_fullsized":"2024-03-06T07:24:10.657656+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:19.463965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651712+0000","last_change":"2024-03-06T07:24:10.651712+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651712+0000","last_undegraded":"2024-03-06T07:24:10.651712+0000","last_fullsized":"2024-03-06T07:24:10.651712+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.664178+0000","last_change":"2024-03-06T07:24:10.662788+0000","last_active":"2024-03-06T07:24:10.664178+0000","last_peered":"2024-03-06T07:24:10.664178+0000","last_clean":"2024-03-06T07:24:10.664178+0000","last_became_active":"2024-03-06T07:24:10.662446+0000","last_became_peered":"2024-03-06T07:24:10.662446+0000","last_unstale":"2024-03-06T07:24:10.664178+0000","last_undegraded":"2024-03-06T07:24:10.664178+0000","last_fullsized":"2024-03-06T07:24:10.664178+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:25.045708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.647210+0000","last_change":"2024-03-06T07:24:10.647210+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.647210+0000","last_undegraded":"2024-03-06T07:24:10.647210+0000","last_fullsized":"2024-03-06T07:24:10.647210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.658501+0000","last_change":"2024-03-06T07:24:10.649819+0000","last_active":"2024-03-06T07:24:10.658501+0000","last_peered":"2024-03-06T07:24:10.658501+0000","last_clean":"2024-03-06T07:24:10.658501+0000","last_became_active":"2024-03-06T07:24:10.649508+0000","last_became_peered":"2024-03-06T07:24:10.649508+0000","last_unstale":"2024-03-06T07:24:10.658501+0000","last_undegraded":"2024-03-06T07:24:10.658501+0000","last_fullsized":"2024-03-06T07:24:10.658501+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:25.045708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.652286+0000","last_change":"2024-03-06T07:24:10.652286+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.652286+0000","last_undegraded":"2024-03-06T07:24:10.652286+0000","last_fullsized":"2024-03-06T07:24:10.652286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.657611+0000","last_change":"2024-03-06T07:24:10.656290+0000","last_active":"2024-03-06T07:24:10.657611+0000","last_peered":"2024-03-06T07:24:10.657611+0000","last_clean":"2024-03-06T07:24:10.657611+0000","last_became_active":"2024-03-06T07:24:10.656023+0000","last_became_peered":"2024-03-06T07:24:10.656023+0000","last_unstale":"2024-03-06T07:24:10.657611+0000","last_undegraded":"2024-03-06T07:24:10.657611+0000","last_fullsized":"2024-03-06T07:24:10.657611+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:35.493530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":57,"state":"creating+peering","last_fresh":"2024-03-06T07:24:10.651834+0000","last_change":"2024-03-06T07:24:10.651834+0000","last_active":"2024-03-06T07:24:10.639170+0000","last_peered":"2024-03-06T07:24:10.639170+0000","last_clean":"2024-03-06T07:24:10.639170+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:10.651834+0000","last_undegraded":"2024-03-06T07:24:10.651834+0000","last_fullsized":"2024-03-06T07:24:10.651834+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.652350+0000","last_change":"2024-03-06T07:24:10.650351+0000","last_active":"2024-03-06T07:24:10.652350+0000","last_peered":"2024-03-06T07:24:10.652350+0000","last_clean":"2024-03-06T07:24:10.652350+0000","last_became_active":"2024-03-06T07:24:10.649966+0000","last_became_peered":"2024-03-06T07:24:10.649966+0000","last_unstale":"2024-03-06T07:24:10.652350+0000","last_undegraded":"2024-03-06T07:24:10.652350+0000","last_fullsized":"2024-03-06T07:24:10.652350+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:20.586025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":238,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:24:10.653875+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:24:10.653875+0000","last_peered":"2024-03-06T07:24:10.653875+0000","last_clean":"2024-03-06T07:24:10.653875+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:24:10.653875+0000","last_undegraded":"2024-03-06T07:24:10.653875+0000","last_fullsized":"2024-03-06T07:24:10.653875+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1265664,"data_stored":866798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":671744,"data_stored":276430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":671744,"data_stored":276430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":684032,"data_stored":282697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339424,"statfs":{"total":96636764160,"available":96603570176,"internally_reserved":0,"allocated":1265664,"data_stored":866798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:16.627 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-03-06T07:24:16.626+0000 7f4a6e039640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:19.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:19.402 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:19.404 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:19.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.442+0000 7f5f38979640 1 Processor -- start 2024-03-06T07:24:19.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.442+0000 7f5f38979640 1 -- start start 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.443+0000 7f5f38979640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5f34158920 0x7f5f3415c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.443+0000 7f5f38979640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340d14f0 0x7f5f3406f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.443+0000 7f5f38979640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5f340af6c0 0x7f5f340afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.443+0000 7f5f38979640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3406fae0 con 0x7f5f340af6c0 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.443+0000 7f5f38979640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3415f5f0 con 0x7f5f34158920 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.443+0000 7f5f38979640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3415e740 con 0x7f5f340d14f0 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.444+0000 7f5f32d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5f340af6c0 0x7f5f340afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.444+0000 7f5f32d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5f340af6c0 0x7f5f340afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50960/0 (socket says 172.21.15.59:50960) 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.444+0000 7f5f32d9d640 1 -- 172.21.15.59:0/3519897979 learned_addr learned my addr 172.21.15.59:0/3519897979 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.444+0000 7f5f3259c640 1 --2- 172.21.15.59:0/3519897979 >> v2:172.21.15.164:3300/0 conn(0x7f5f34158920 0x7f5f3415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.444+0000 7f5f2bfff640 1 --2- 172.21.15.59:0/3519897979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340d14f0 0x7f5f3406f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.444+0000 7f5f32d9d640 1 -- 172.21.15.59:0/3519897979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340d14f0 msgr2=0x7f5f3406f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.444+0000 7f5f32d9d640 1 --2- 172.21.15.59:0/3519897979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340d14f0 0x7f5f3406f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.444+0000 7f5f32d9d640 1 -- 172.21.15.59:0/3519897979 >> v2:172.21.15.164:3300/0 conn(0x7f5f34158920 msgr2=0x7f5f3415c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.444+0000 7f5f32d9d640 1 --2- 172.21.15.59:0/3519897979 >> v2:172.21.15.164:3300/0 conn(0x7f5f34158920 0x7f5f3415c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.444+0000 7f5f32d9d640 1 -- 172.21.15.59:0/3519897979 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3415e9e0 con 0x7f5f340af6c0 2024-03-06T07:24:19.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.444+0000 7f5f2bfff640 1 --2- 172.21.15.59:0/3519897979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340d14f0 0x7f5f3406f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:19.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.445+0000 7f5f32d9d640 1 --2- 172.21.15.59:0/3519897979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f340af6c0 0x7f5f340afb50 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f5f340689f0 tx=0x7f5f2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc40b3a0c6cac32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.445+0000 7f5f31d9b640 1 -- 172.21.15.59:0/3519897979 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c02ca70 con 0x7f5f340af6c0 2024-03-06T07:24:19.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.445+0000 7f5f31d9b640 1 -- 172.21.15.59:0/3519897979 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c02cc10 con 0x7f5f340af6c0 2024-03-06T07:24:19.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.445+0000 7f5f31d9b640 1 -- 172.21.15.59:0/3519897979 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0343f0 con 0x7f5f340af6c0 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.445+0000 7f5f38979640 1 -- 172.21.15.59:0/3519897979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f340af6c0 msgr2=0x7f5f340afb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.445+0000 7f5f38979640 1 --2- 172.21.15.59:0/3519897979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f340af6c0 0x7f5f340afb50 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f5f340689f0 tx=0x7f5f2c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.445+0000 7f5f38979640 1 -- 172.21.15.59:0/3519897979 shutdown_connections 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.445+0000 7f5f38979640 1 --2- 172.21.15.59:0/3519897979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f340af6c0 0x7f5f340afb50 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.445+0000 7f5f38979640 1 --2- 172.21.15.59:0/3519897979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340d14f0 0x7f5f3406f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.445+0000 7f5f38979640 1 --2- 172.21.15.59:0/3519897979 >> v2:172.21.15.164:3300/0 conn(0x7f5f34158920 0x7f5f3415c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.445+0000 7f5f38979640 1 -- 172.21.15.59:0/3519897979 >> 172.21.15.59:0/3519897979 conn(0x7f5f340d0960 msgr2=0x7f5f340d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.446+0000 7f5f38979640 1 -- 172.21.15.59:0/3519897979 shutdown_connections 2024-03-06T07:24:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.446+0000 7f5f38979640 1 -- 172.21.15.59:0/3519897979 wait complete. 2024-03-06T07:24:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.447+0000 7f5f38979640 1 Processor -- start 2024-03-06T07:24:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.447+0000 7f5f38979640 1 -- start start 2024-03-06T07:24:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f38979640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340af6c0 0x7f5f34155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f38979640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f340d14f0 0x7f5f340cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f38979640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f34158920 0x7f5f340ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f38979640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3415f280 con 0x7f5f34158920 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f38979640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3415efa0 con 0x7f5f340d14f0 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f38979640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3415b400 con 0x7f5f340af6c0 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f32d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f34158920 0x7f5f340ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f32d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f34158920 0x7f5f340ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50974/0 (socket says 172.21.15.59:50974) 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f32d9d640 1 -- 172.21.15.59:0/2586177535 learned_addr learned my addr 172.21.15.59:0/2586177535 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f2bfff640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f340d14f0 0x7f5f340cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f3259c640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340af6c0 0x7f5f34155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f32d9d640 1 -- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340af6c0 msgr2=0x7f5f34155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f32d9d640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340af6c0 0x7f5f34155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f32d9d640 1 -- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f340d14f0 msgr2=0x7f5f340cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f32d9d640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f340d14f0 0x7f5f340cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f32d9d640 1 -- 172.21.15.59:0/2586177535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f340ce900 con 0x7f5f34158920 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f3259c640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340af6c0 0x7f5f34155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f32d9d640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f34158920 0x7f5f340ce230 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c036040 tx=0x7f5f2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f2affd640 1 -- 172.21.15.59:0/2586177535 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c043020 con 0x7f5f34158920 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f2affd640 1 -- 172.21.15.59:0/2586177535 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c03e450 con 0x7f5f34158920 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f38979640 1 -- 172.21.15.59:0/2586177535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f340cebe0 con 0x7f5f34158920 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f2affd640 1 -- 172.21.15.59:0/2586177535 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c009510 con 0x7f5f34158920 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.448+0000 7f5f38979640 1 -- 172.21.15.59:0/2586177535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f340cf1a0 con 0x7f5f34158920 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.450+0000 7f5f38979640 1 -- 172.21.15.59:0/2586177535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3404f330 con 0x7f5f34158920 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.450+0000 7f5f2affd640 1 -- 172.21.15.59:0/2586177535 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f2c046020 con 0x7f5f34158920 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.450+0000 7f5f2affd640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1406b4d0 0x7f5f1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.450+0000 7f5f2affd640 1 -- 172.21.15.59:0/2586177535 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f2c0ac990 con 0x7f5f34158920 2024-03-06T07:24:19.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.451+0000 7f5f3259c640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1406b4d0 0x7f5f1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.451+0000 7f5f3259c640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1406b4d0 0x7f5f1406d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5f24006fd0 tx=0x7f5f24008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.459+0000 7f5f2affd640 1 -- 172.21.15.59:0/2586177535 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f2c04d030 con 0x7f5f34158920 2024-03-06T07:24:19.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.635+0000 7f5f38979640 1 -- 172.21.15.59:0/2586177535 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f34072b80 con 0x7f5f1406b4d0 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.642+0000 7f5f2affd640 1 -- 172.21.15.59:0/2586177535 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251955 (secure 0 0 0) 0x7f5f34072b80 con 0x7f5f1406b4d0 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 -- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1406b4d0 msgr2=0x7f5f1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1406b4d0 0x7f5f1406d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5f24006fd0 tx=0x7f5f24008040 comp rx=0 tx=0).stop 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 -- 172.21.15.59:0/2586177535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f34158920 msgr2=0x7f5f340ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f34158920 0x7f5f340ce230 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c036040 tx=0x7f5f2c004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 -- 172.21.15.59:0/2586177535 shutdown_connections 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f1406b4d0 0x7f5f1406d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f34158920 0x7f5f340ce230 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f340d14f0 0x7f5f340cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 --2- 172.21.15.59:0/2586177535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f340af6c0 0x7f5f34155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 -- 172.21.15.59:0/2586177535 >> 172.21.15.59:0/2586177535 conn(0x7f5f340d0960 msgr2=0x7f5f340aee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 -- 172.21.15.59:0/2586177535 shutdown_connections 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.645+0000 7f5f38979640 1 -- 172.21.15.59:0/2586177535 wait complete. 2024-03-06T07:24:19.657 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-03-06T07:24:19.331313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256232,"kb_used_data":6760,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718488,"statfs":{"total":773094113280,"available":772831731712,"internally_reserved":0,"allocated":6922240,"data_stored":3551732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002203"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.655879+0000","last_change":"2024-03-06T07:24:11.680522+0000","last_active":"2024-03-06T07:24:12.655879+0000","last_peered":"2024-03-06T07:24:12.655879+0000","last_clean":"2024-03-06T07:24:12.655879+0000","last_became_active":"2024-03-06T07:24:11.680249+0000","last_became_peered":"2024-03-06T07:24:11.680249+0000","last_unstale":"2024-03-06T07:24:12.655879+0000","last_undegraded":"2024-03-06T07:24:12.655879+0000","last_fullsized":"2024-03-06T07:24:12.655879+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:33.960376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656765+0000","last_change":"2024-03-06T07:24:11.682126+0000","last_active":"2024-03-06T07:24:12.656765+0000","last_peered":"2024-03-06T07:24:12.656765+0000","last_clean":"2024-03-06T07:24:12.656765+0000","last_became_active":"2024-03-06T07:24:11.681780+0000","last_became_peered":"2024-03-06T07:24:11.681780+0000","last_unstale":"2024-03-06T07:24:12.656765+0000","last_undegraded":"2024-03-06T07:24:12.656765+0000","last_fullsized":"2024-03-06T07:24:12.656765+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:13.642422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.915127+0000","last_change":"2024-03-06T07:24:11.676114+0000","last_active":"2024-03-06T07:24:12.915127+0000","last_peered":"2024-03-06T07:24:12.915127+0000","last_clean":"2024-03-06T07:24:12.915127+0000","last_became_active":"2024-03-06T07:24:11.675796+0000","last_became_peered":"2024-03-06T07:24:11.675796+0000","last_unstale":"2024-03-06T07:24:12.915127+0000","last_undegraded":"2024-03-06T07:24:12.915127+0000","last_fullsized":"2024-03-06T07:24:12.915127+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:48.491217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.659058+0000","last_change":"2024-03-06T07:24:11.666841+0000","last_active":"2024-03-06T07:24:12.659058+0000","last_peered":"2024-03-06T07:24:12.659058+0000","last_clean":"2024-03-06T07:24:12.659058+0000","last_became_active":"2024-03-06T07:24:11.666464+0000","last_became_peered":"2024-03-06T07:24:11.666464+0000","last_unstale":"2024-03-06T07:24:12.659058+0000","last_undegraded":"2024-03-06T07:24:12.659058+0000","last_fullsized":"2024-03-06T07:24:12.659058+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:08.382241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.655724+0000","last_change":"2024-03-06T07:24:11.667913+0000","last_active":"2024-03-06T07:24:12.655724+0000","last_peered":"2024-03-06T07:24:12.655724+0000","last_clean":"2024-03-06T07:24:12.655724+0000","last_became_active":"2024-03-06T07:24:11.667641+0000","last_became_peered":"2024-03-06T07:24:11.667641+0000","last_unstale":"2024-03-06T07:24:12.655724+0000","last_undegraded":"2024-03-06T07:24:12.655724+0000","last_fullsized":"2024-03-06T07:24:12.655724+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:16.559521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.674937+0000","last_change":"2024-03-06T07:24:11.661430+0000","last_active":"2024-03-06T07:24:12.674937+0000","last_peered":"2024-03-06T07:24:12.674937+0000","last_clean":"2024-03-06T07:24:12.674937+0000","last_became_active":"2024-03-06T07:24:11.661088+0000","last_became_peered":"2024-03-06T07:24:11.661088+0000","last_unstale":"2024-03-06T07:24:12.674937+0000","last_undegraded":"2024-03-06T07:24:12.674937+0000","last_fullsized":"2024-03-06T07:24:12.674937+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:13.027051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914127+0000","last_change":"2024-03-06T07:24:11.671522+0000","last_active":"2024-03-06T07:24:12.914127+0000","last_peered":"2024-03-06T07:24:12.914127+0000","last_clean":"2024-03-06T07:24:12.914127+0000","last_became_active":"2024-03-06T07:24:11.671257+0000","last_became_peered":"2024-03-06T07:24:11.671257+0000","last_unstale":"2024-03-06T07:24:12.914127+0000","last_undegraded":"2024-03-06T07:24:12.914127+0000","last_fullsized":"2024-03-06T07:24:12.914127+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:32.577425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656161+0000","last_change":"2024-03-06T07:24:11.678311+0000","last_active":"2024-03-06T07:24:12.656161+0000","last_peered":"2024-03-06T07:24:12.656161+0000","last_clean":"2024-03-06T07:24:12.656161+0000","last_became_active":"2024-03-06T07:24:11.677972+0000","last_became_peered":"2024-03-06T07:24:11.677972+0000","last_unstale":"2024-03-06T07:24:12.656161+0000","last_undegraded":"2024-03-06T07:24:12.656161+0000","last_fullsized":"2024-03-06T07:24:12.656161+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:31:25.430993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914646+0000","last_change":"2024-03-06T07:24:11.666602+0000","last_active":"2024-03-06T07:24:12.914646+0000","last_peered":"2024-03-06T07:24:12.914646+0000","last_clean":"2024-03-06T07:24:12.914646+0000","last_became_active":"2024-03-06T07:24:11.666357+0000","last_became_peered":"2024-03-06T07:24:11.666357+0000","last_unstale":"2024-03-06T07:24:12.914646+0000","last_undegraded":"2024-03-06T07:24:12.914646+0000","last_fullsized":"2024-03-06T07:24:12.914646+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:27.680745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914650+0000","last_change":"2024-03-06T07:24:11.675980+0000","last_active":"2024-03-06T07:24:12.914650+0000","last_peered":"2024-03-06T07:24:12.914650+0000","last_clean":"2024-03-06T07:24:12.914650+0000","last_became_active":"2024-03-06T07:24:11.675273+0000","last_became_peered":"2024-03-06T07:24:11.675273+0000","last_unstale":"2024-03-06T07:24:12.914650+0000","last_undegraded":"2024-03-06T07:24:12.914650+0000","last_fullsized":"2024-03-06T07:24:12.914650+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:04.631037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914621+0000","last_change":"2024-03-06T07:24:11.681071+0000","last_active":"2024-03-06T07:24:12.914621+0000","last_peered":"2024-03-06T07:24:12.914621+0000","last_clean":"2024-03-06T07:24:12.914621+0000","last_became_active":"2024-03-06T07:24:11.680646+0000","last_became_peered":"2024-03-06T07:24:11.680646+0000","last_unstale":"2024-03-06T07:24:12.914621+0000","last_undegraded":"2024-03-06T07:24:12.914621+0000","last_fullsized":"2024-03-06T07:24:12.914621+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:13.357699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.659148+0000","last_change":"2024-03-06T07:24:11.678829+0000","last_active":"2024-03-06T07:24:12.659148+0000","last_peered":"2024-03-06T07:24:12.659148+0000","last_clean":"2024-03-06T07:24:12.659148+0000","last_became_active":"2024-03-06T07:24:11.678557+0000","last_became_peered":"2024-03-06T07:24:11.678557+0000","last_unstale":"2024-03-06T07:24:12.659148+0000","last_undegraded":"2024-03-06T07:24:12.659148+0000","last_fullsized":"2024-03-06T07:24:12.659148+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:38.918588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.677625+0000","last_change":"2024-03-06T07:24:11.674541+0000","last_active":"2024-03-06T07:24:12.677625+0000","last_peered":"2024-03-06T07:24:12.677625+0000","last_clean":"2024-03-06T07:24:12.677625+0000","last_became_active":"2024-03-06T07:24:11.674238+0000","last_became_peered":"2024-03-06T07:24:11.674238+0000","last_unstale":"2024-03-06T07:24:12.677625+0000","last_undegraded":"2024-03-06T07:24:12.677625+0000","last_fullsized":"2024-03-06T07:24:12.677625+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:35.472535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914773+0000","last_change":"2024-03-06T07:24:11.672781+0000","last_active":"2024-03-06T07:24:12.914773+0000","last_peered":"2024-03-06T07:24:12.914773+0000","last_clean":"2024-03-06T07:24:12.914773+0000","last_became_active":"2024-03-06T07:24:11.672319+0000","last_became_peered":"2024-03-06T07:24:11.672319+0000","last_unstale":"2024-03-06T07:24:12.914773+0000","last_undegraded":"2024-03-06T07:24:12.914773+0000","last_fullsized":"2024-03-06T07:24:12.914773+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:13.772481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914211+0000","last_change":"2024-03-06T07:24:11.673411+0000","last_active":"2024-03-06T07:24:12.914211+0000","last_peered":"2024-03-06T07:24:12.914211+0000","last_clean":"2024-03-06T07:24:12.914211+0000","last_became_active":"2024-03-06T07:24:11.673102+0000","last_became_peered":"2024-03-06T07:24:11.673102+0000","last_unstale":"2024-03-06T07:24:12.914211+0000","last_undegraded":"2024-03-06T07:24:12.914211+0000","last_fullsized":"2024-03-06T07:24:12.914211+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:06.456699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.674734+0000","last_change":"2024-03-06T07:24:11.677467+0000","last_active":"2024-03-06T07:24:12.674734+0000","last_peered":"2024-03-06T07:24:12.674734+0000","last_clean":"2024-03-06T07:24:12.674734+0000","last_became_active":"2024-03-06T07:24:11.677188+0000","last_became_peered":"2024-03-06T07:24:11.677188+0000","last_unstale":"2024-03-06T07:24:12.674734+0000","last_undegraded":"2024-03-06T07:24:12.674734+0000","last_fullsized":"2024-03-06T07:24:12.674734+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:22.091251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656373+0000","last_change":"2024-03-06T07:24:11.675825+0000","last_active":"2024-03-06T07:24:12.656373+0000","last_peered":"2024-03-06T07:24:12.656373+0000","last_clean":"2024-03-06T07:24:12.656373+0000","last_became_active":"2024-03-06T07:24:11.675456+0000","last_became_peered":"2024-03-06T07:24:11.675456+0000","last_unstale":"2024-03-06T07:24:12.656373+0000","last_undegraded":"2024-03-06T07:24:12.656373+0000","last_fullsized":"2024-03-06T07:24:12.656373+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:24.539649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914953+0000","last_change":"2024-03-06T07:24:11.687571+0000","last_active":"2024-03-06T07:24:12.914953+0000","last_peered":"2024-03-06T07:24:12.914953+0000","last_clean":"2024-03-06T07:24:12.914953+0000","last_became_active":"2024-03-06T07:24:11.687337+0000","last_became_peered":"2024-03-06T07:24:11.687337+0000","last_unstale":"2024-03-06T07:24:12.914953+0000","last_undegraded":"2024-03-06T07:24:12.914953+0000","last_fullsized":"2024-03-06T07:24:12.914953+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:13.027051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.655723+0000","last_change":"2024-03-06T07:24:11.681139+0000","last_active":"2024-03-06T07:24:12.655723+0000","last_peered":"2024-03-06T07:24:12.655723+0000","last_clean":"2024-03-06T07:24:12.655723+0000","last_became_active":"2024-03-06T07:24:11.680890+0000","last_became_peered":"2024-03-06T07:24:11.680890+0000","last_unstale":"2024-03-06T07:24:12.655723+0000","last_undegraded":"2024-03-06T07:24:12.655723+0000","last_fullsized":"2024-03-06T07:24:12.655723+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:25.207450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656352+0000","last_change":"2024-03-06T07:24:11.679760+0000","last_active":"2024-03-06T07:24:12.656352+0000","last_peered":"2024-03-06T07:24:12.656352+0000","last_clean":"2024-03-06T07:24:12.656352+0000","last_became_active":"2024-03-06T07:24:11.677770+0000","last_became_peered":"2024-03-06T07:24:11.677770+0000","last_unstale":"2024-03-06T07:24:12.656352+0000","last_undegraded":"2024-03-06T07:24:12.656352+0000","last_fullsized":"2024-03-06T07:24:12.656352+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:50.043632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914795+0000","last_change":"2024-03-06T07:24:11.676777+0000","last_active":"2024-03-06T07:24:12.914795+0000","last_peered":"2024-03-06T07:24:12.914795+0000","last_clean":"2024-03-06T07:24:12.914795+0000","last_became_active":"2024-03-06T07:24:11.676166+0000","last_became_peered":"2024-03-06T07:24:11.676166+0000","last_unstale":"2024-03-06T07:24:12.914795+0000","last_undegraded":"2024-03-06T07:24:12.914795+0000","last_fullsized":"2024-03-06T07:24:12.914795+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:38.918588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656605+0000","last_change":"2024-03-06T07:24:11.679770+0000","last_active":"2024-03-06T07:24:12.656605+0000","last_peered":"2024-03-06T07:24:12.656605+0000","last_clean":"2024-03-06T07:24:12.656605+0000","last_became_active":"2024-03-06T07:24:11.679336+0000","last_became_peered":"2024-03-06T07:24:11.679336+0000","last_unstale":"2024-03-06T07:24:12.656605+0000","last_undegraded":"2024-03-06T07:24:12.656605+0000","last_fullsized":"2024-03-06T07:24:12.656605+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:05:25.123300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914522+0000","last_change":"2024-03-06T07:24:11.681555+0000","last_active":"2024-03-06T07:24:12.914522+0000","last_peered":"2024-03-06T07:24:12.914522+0000","last_clean":"2024-03-06T07:24:12.914522+0000","last_became_active":"2024-03-06T07:24:11.681265+0000","last_became_peered":"2024-03-06T07:24:11.681265+0000","last_unstale":"2024-03-06T07:24:12.914522+0000","last_undegraded":"2024-03-06T07:24:12.914522+0000","last_fullsized":"2024-03-06T07:24:12.914522+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:00.898933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.677439+0000","last_change":"2024-03-06T07:24:11.670268+0000","last_active":"2024-03-06T07:24:12.677439+0000","last_peered":"2024-03-06T07:24:12.677439+0000","last_clean":"2024-03-06T07:24:12.677439+0000","last_became_active":"2024-03-06T07:24:11.670010+0000","last_became_peered":"2024-03-06T07:24:11.670010+0000","last_unstale":"2024-03-06T07:24:12.677439+0000","last_undegraded":"2024-03-06T07:24:12.677439+0000","last_fullsized":"2024-03-06T07:24:12.677439+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:25.279985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656090+0000","last_change":"2024-03-06T07:24:11.661496+0000","last_active":"2024-03-06T07:24:12.656090+0000","last_peered":"2024-03-06T07:24:12.656090+0000","last_clean":"2024-03-06T07:24:12.656090+0000","last_became_active":"2024-03-06T07:24:11.661182+0000","last_became_peered":"2024-03-06T07:24:11.661182+0000","last_unstale":"2024-03-06T07:24:12.656090+0000","last_undegraded":"2024-03-06T07:24:12.656090+0000","last_fullsized":"2024-03-06T07:24:12.656090+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:21.592510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.659200+0000","last_change":"2024-03-06T07:24:11.666079+0000","last_active":"2024-03-06T07:24:12.659200+0000","last_peered":"2024-03-06T07:24:12.659200+0000","last_clean":"2024-03-06T07:24:12.659200+0000","last_became_active":"2024-03-06T07:24:11.665810+0000","last_became_peered":"2024-03-06T07:24:11.665810+0000","last_unstale":"2024-03-06T07:24:12.659200+0000","last_undegraded":"2024-03-06T07:24:12.659200+0000","last_fullsized":"2024-03-06T07:24:12.659200+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:13.027051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.674961+0000","last_change":"2024-03-06T07:24:11.677643+0000","last_active":"2024-03-06T07:24:12.674961+0000","last_peered":"2024-03-06T07:24:12.674961+0000","last_clean":"2024-03-06T07:24:12.674961+0000","last_became_active":"2024-03-06T07:24:11.677352+0000","last_became_peered":"2024-03-06T07:24:11.677352+0000","last_unstale":"2024-03-06T07:24:12.674961+0000","last_undegraded":"2024-03-06T07:24:12.674961+0000","last_fullsized":"2024-03-06T07:24:12.674961+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:23.795264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656106+0000","last_change":"2024-03-06T07:24:11.668996+0000","last_active":"2024-03-06T07:24:12.656106+0000","last_peered":"2024-03-06T07:24:12.656106+0000","last_clean":"2024-03-06T07:24:12.656106+0000","last_became_active":"2024-03-06T07:24:11.668694+0000","last_became_peered":"2024-03-06T07:24:11.668694+0000","last_unstale":"2024-03-06T07:24:12.656106+0000","last_undegraded":"2024-03-06T07:24:12.656106+0000","last_fullsized":"2024-03-06T07:24:12.656106+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:17.954932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656891+0000","last_change":"2024-03-06T07:24:11.684395+0000","last_active":"2024-03-06T07:24:12.656891+0000","last_peered":"2024-03-06T07:24:12.656891+0000","last_clean":"2024-03-06T07:24:12.656891+0000","last_became_active":"2024-03-06T07:24:11.683390+0000","last_became_peered":"2024-03-06T07:24:11.683390+0000","last_unstale":"2024-03-06T07:24:12.656891+0000","last_undegraded":"2024-03-06T07:24:12.656891+0000","last_fullsized":"2024-03-06T07:24:12.656891+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:02.362195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914506+0000","last_change":"2024-03-06T07:24:11.676151+0000","last_active":"2024-03-06T07:24:12.914506+0000","last_peered":"2024-03-06T07:24:12.914506+0000","last_clean":"2024-03-06T07:24:12.914506+0000","last_became_active":"2024-03-06T07:24:11.675754+0000","last_became_peered":"2024-03-06T07:24:11.675754+0000","last_unstale":"2024-03-06T07:24:12.914506+0000","last_undegraded":"2024-03-06T07:24:12.914506+0000","last_fullsized":"2024-03-06T07:24:12.914506+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:40.595707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.655516+0000","last_change":"2024-03-06T07:24:11.661453+0000","last_active":"2024-03-06T07:24:12.655516+0000","last_peered":"2024-03-06T07:24:12.655516+0000","last_clean":"2024-03-06T07:24:12.655516+0000","last_became_active":"2024-03-06T07:24:11.661156+0000","last_became_peered":"2024-03-06T07:24:11.661156+0000","last_unstale":"2024-03-06T07:24:12.655516+0000","last_undegraded":"2024-03-06T07:24:12.655516+0000","last_fullsized":"2024-03-06T07:24:12.655516+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:42.452313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.674776+0000","last_change":"2024-03-06T07:24:11.680648+0000","last_active":"2024-03-06T07:24:12.674776+0000","last_peered":"2024-03-06T07:24:12.674776+0000","last_clean":"2024-03-06T07:24:12.674776+0000","last_became_active":"2024-03-06T07:24:11.680181+0000","last_became_peered":"2024-03-06T07:24:11.680181+0000","last_unstale":"2024-03-06T07:24:12.674776+0000","last_undegraded":"2024-03-06T07:24:12.674776+0000","last_fullsized":"2024-03-06T07:24:12.674776+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:38.918588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.675217+0000","last_change":"2024-03-06T07:24:11.682433+0000","last_active":"2024-03-06T07:24:12.675217+0000","last_peered":"2024-03-06T07:24:12.675217+0000","last_clean":"2024-03-06T07:24:12.675217+0000","last_became_active":"2024-03-06T07:24:11.682159+0000","last_became_peered":"2024-03-06T07:24:11.682159+0000","last_unstale":"2024-03-06T07:24:12.675217+0000","last_undegraded":"2024-03-06T07:24:12.675217+0000","last_fullsized":"2024-03-06T07:24:12.675217+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:11.078896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"58'4","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.675731+0000","last_change":"2024-03-06T07:24:10.658033+0000","last_active":"2024-03-06T07:24:12.675731+0000","last_peered":"2024-03-06T07:24:12.675731+0000","last_clean":"2024-03-06T07:24:12.675731+0000","last_became_active":"2024-03-06T07:24:10.657766+0000","last_became_peered":"2024-03-06T07:24:10.657766+0000","last_unstale":"2024-03-06T07:24:12.675731+0000","last_undegraded":"2024-03-06T07:24:12.675731+0000","last_fullsized":"2024-03-06T07:24:12.675731+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:15.720920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914758+0000","last_change":"2024-03-06T07:24:11.680485+0000","last_active":"2024-03-06T07:24:12.914758+0000","last_peered":"2024-03-06T07:24:12.914758+0000","last_clean":"2024-03-06T07:24:12.914758+0000","last_became_active":"2024-03-06T07:24:11.680161+0000","last_became_peered":"2024-03-06T07:24:11.680161+0000","last_unstale":"2024-03-06T07:24:12.914758+0000","last_undegraded":"2024-03-06T07:24:12.914758+0000","last_fullsized":"2024-03-06T07:24:12.914758+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:21.859220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656717+0000","last_change":"2024-03-06T07:24:10.659965+0000","last_active":"2024-03-06T07:24:12.656717+0000","last_peered":"2024-03-06T07:24:12.656717+0000","last_clean":"2024-03-06T07:24:12.656717+0000","last_became_active":"2024-03-06T07:24:10.659581+0000","last_became_peered":"2024-03-06T07:24:10.659581+0000","last_unstale":"2024-03-06T07:24:12.656717+0000","last_undegraded":"2024-03-06T07:24:12.656717+0000","last_fullsized":"2024-03-06T07:24:12.656717+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:40.264112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.674959+0000","last_change":"2024-03-06T07:24:11.661626+0000","last_active":"2024-03-06T07:24:12.674959+0000","last_peered":"2024-03-06T07:24:12.674959+0000","last_clean":"2024-03-06T07:24:12.674959+0000","last_became_active":"2024-03-06T07:24:11.661201+0000","last_became_peered":"2024-03-06T07:24:11.661201+0000","last_unstale":"2024-03-06T07:24:12.674959+0000","last_undegraded":"2024-03-06T07:24:12.674959+0000","last_fullsized":"2024-03-06T07:24:12.674959+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:41.270597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.675088+0000","last_change":"2024-03-06T07:24:10.662366+0000","last_active":"2024-03-06T07:24:12.675088+0000","last_peered":"2024-03-06T07:24:12.675088+0000","last_clean":"2024-03-06T07:24:12.675088+0000","last_became_active":"2024-03-06T07:24:10.662075+0000","last_became_peered":"2024-03-06T07:24:10.662075+0000","last_unstale":"2024-03-06T07:24:12.675088+0000","last_undegraded":"2024-03-06T07:24:12.675088+0000","last_fullsized":"2024-03-06T07:24:12.675088+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:26.674260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.674979+0000","last_change":"2024-03-06T07:24:11.667199+0000","last_active":"2024-03-06T07:24:12.674979+0000","last_peered":"2024-03-06T07:24:12.674979+0000","last_clean":"2024-03-06T07:24:12.674979+0000","last_became_active":"2024-03-06T07:24:11.666884+0000","last_became_peered":"2024-03-06T07:24:11.666884+0000","last_unstale":"2024-03-06T07:24:12.674979+0000","last_undegraded":"2024-03-06T07:24:12.674979+0000","last_fullsized":"2024-03-06T07:24:12.674979+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:48.749848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.659301+0000","last_change":"2024-03-06T07:24:10.659614+0000","last_active":"2024-03-06T07:24:12.659301+0000","last_peered":"2024-03-06T07:24:12.659301+0000","last_clean":"2024-03-06T07:24:12.659301+0000","last_became_active":"2024-03-06T07:24:10.659301+0000","last_became_peered":"2024-03-06T07:24:10.659301+0000","last_unstale":"2024-03-06T07:24:12.659301+0000","last_undegraded":"2024-03-06T07:24:12.659301+0000","last_fullsized":"2024-03-06T07:24:12.659301+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:31.570941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656685+0000","last_change":"2024-03-06T07:24:11.679802+0000","last_active":"2024-03-06T07:24:12.656685+0000","last_peered":"2024-03-06T07:24:12.656685+0000","last_clean":"2024-03-06T07:24:12.656685+0000","last_became_active":"2024-03-06T07:24:11.679287+0000","last_became_peered":"2024-03-06T07:24:11.679287+0000","last_unstale":"2024-03-06T07:24:12.656685+0000","last_undegraded":"2024-03-06T07:24:12.656685+0000","last_fullsized":"2024-03-06T07:24:12.656685+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:59.974334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914251+0000","last_change":"2024-03-06T07:24:10.661362+0000","last_active":"2024-03-06T07:24:12.914251+0000","last_peered":"2024-03-06T07:24:12.914251+0000","last_clean":"2024-03-06T07:24:12.914251+0000","last_became_active":"2024-03-06T07:24:10.661069+0000","last_became_peered":"2024-03-06T07:24:10.661069+0000","last_unstale":"2024-03-06T07:24:12.914251+0000","last_undegraded":"2024-03-06T07:24:12.914251+0000","last_fullsized":"2024-03-06T07:24:12.914251+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:34.328658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656623+0000","last_change":"2024-03-06T07:24:11.683681+0000","last_active":"2024-03-06T07:24:12.656623+0000","last_peered":"2024-03-06T07:24:12.656623+0000","last_clean":"2024-03-06T07:24:12.656623+0000","last_became_active":"2024-03-06T07:24:11.683069+0000","last_became_peered":"2024-03-06T07:24:11.683069+0000","last_unstale":"2024-03-06T07:24:12.656623+0000","last_undegraded":"2024-03-06T07:24:12.656623+0000","last_fullsized":"2024-03-06T07:24:12.656623+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:45.363024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.675051+0000","last_change":"2024-03-06T07:24:10.653845+0000","last_active":"2024-03-06T07:24:12.675051+0000","last_peered":"2024-03-06T07:24:12.675051+0000","last_clean":"2024-03-06T07:24:12.675051+0000","last_became_active":"2024-03-06T07:24:10.653600+0000","last_became_peered":"2024-03-06T07:24:10.653600+0000","last_unstale":"2024-03-06T07:24:12.675051+0000","last_undegraded":"2024-03-06T07:24:12.675051+0000","last_fullsized":"2024-03-06T07:24:12.675051+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:12.765996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.677577+0000","last_change":"2024-03-06T07:24:11.677855+0000","last_active":"2024-03-06T07:24:12.677577+0000","last_peered":"2024-03-06T07:24:12.677577+0000","last_clean":"2024-03-06T07:24:12.677577+0000","last_became_active":"2024-03-06T07:24:11.677533+0000","last_became_peered":"2024-03-06T07:24:11.677533+0000","last_unstale":"2024-03-06T07:24:12.677577+0000","last_undegraded":"2024-03-06T07:24:12.677577+0000","last_fullsized":"2024-03-06T07:24:12.677577+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:35.513117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.658747+0000","last_change":"2024-03-06T07:24:10.660990+0000","last_active":"2024-03-06T07:24:12.658747+0000","last_peered":"2024-03-06T07:24:12.658747+0000","last_clean":"2024-03-06T07:24:12.658747+0000","last_became_active":"2024-03-06T07:24:10.660680+0000","last_became_peered":"2024-03-06T07:24:10.660680+0000","last_unstale":"2024-03-06T07:24:12.658747+0000","last_undegraded":"2024-03-06T07:24:12.658747+0000","last_fullsized":"2024-03-06T07:24:12.658747+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:05.450215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656371+0000","last_change":"2024-03-06T07:24:11.679016+0000","last_active":"2024-03-06T07:24:12.656371+0000","last_peered":"2024-03-06T07:24:12.656371+0000","last_clean":"2024-03-06T07:24:12.656371+0000","last_became_active":"2024-03-06T07:24:11.676671+0000","last_became_peered":"2024-03-06T07:24:11.676671+0000","last_unstale":"2024-03-06T07:24:12.656371+0000","last_undegraded":"2024-03-06T07:24:12.656371+0000","last_fullsized":"2024-03-06T07:24:12.656371+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:29.242415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914124+0000","last_change":"2024-03-06T07:24:10.659499+0000","last_active":"2024-03-06T07:24:12.914124+0000","last_peered":"2024-03-06T07:24:12.914124+0000","last_clean":"2024-03-06T07:24:12.914124+0000","last_became_active":"2024-03-06T07:24:10.659224+0000","last_became_peered":"2024-03-06T07:24:10.659224+0000","last_unstale":"2024-03-06T07:24:12.914124+0000","last_undegraded":"2024-03-06T07:24:12.914124+0000","last_fullsized":"2024-03-06T07:24:12.914124+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:19.463965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.915098+0000","last_change":"2024-03-06T07:24:11.674211+0000","last_active":"2024-03-06T07:24:12.915098+0000","last_peered":"2024-03-06T07:24:12.915098+0000","last_clean":"2024-03-06T07:24:12.915098+0000","last_became_active":"2024-03-06T07:24:11.673766+0000","last_became_peered":"2024-03-06T07:24:11.673766+0000","last_unstale":"2024-03-06T07:24:12.915098+0000","last_undegraded":"2024-03-06T07:24:12.915098+0000","last_fullsized":"2024-03-06T07:24:12.915098+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:08.252768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914970+0000","last_change":"2024-03-06T07:24:10.662290+0000","last_active":"2024-03-06T07:24:12.914970+0000","last_peered":"2024-03-06T07:24:12.914970+0000","last_clean":"2024-03-06T07:24:12.914970+0000","last_became_active":"2024-03-06T07:24:10.662003+0000","last_became_peered":"2024-03-06T07:24:10.662003+0000","last_unstale":"2024-03-06T07:24:12.914970+0000","last_undegraded":"2024-03-06T07:24:12.914970+0000","last_fullsized":"2024-03-06T07:24:12.914970+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:05.450215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914989+0000","last_change":"2024-03-06T07:24:11.685983+0000","last_active":"2024-03-06T07:24:12.914989+0000","last_peered":"2024-03-06T07:24:12.914989+0000","last_clean":"2024-03-06T07:24:12.914989+0000","last_became_active":"2024-03-06T07:24:11.685706+0000","last_became_peered":"2024-03-06T07:24:11.685706+0000","last_unstale":"2024-03-06T07:24:12.914989+0000","last_undegraded":"2024-03-06T07:24:12.914989+0000","last_fullsized":"2024-03-06T07:24:12.914989+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:48.749848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656815+0000","last_change":"2024-03-06T07:24:10.663647+0000","last_active":"2024-03-06T07:24:12.656815+0000","last_peered":"2024-03-06T07:24:12.656815+0000","last_clean":"2024-03-06T07:24:12.656815+0000","last_became_active":"2024-03-06T07:24:10.663378+0000","last_became_peered":"2024-03-06T07:24:10.663378+0000","last_unstale":"2024-03-06T07:24:12.656815+0000","last_undegraded":"2024-03-06T07:24:12.656815+0000","last_fullsized":"2024-03-06T07:24:12.656815+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:47.743364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914439+0000","last_change":"2024-03-06T07:24:11.668895+0000","last_active":"2024-03-06T07:24:12.914439+0000","last_peered":"2024-03-06T07:24:12.914439+0000","last_clean":"2024-03-06T07:24:12.914439+0000","last_became_active":"2024-03-06T07:24:11.668609+0000","last_became_peered":"2024-03-06T07:24:11.668609+0000","last_unstale":"2024-03-06T07:24:12.914439+0000","last_undegraded":"2024-03-06T07:24:12.914439+0000","last_fullsized":"2024-03-06T07:24:12.914439+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:08.252768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"58'3","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.915347+0000","last_change":"2024-03-06T07:24:10.665110+0000","last_active":"2024-03-06T07:24:12.915347+0000","last_peered":"2024-03-06T07:24:12.915347+0000","last_clean":"2024-03-06T07:24:12.915347+0000","last_became_active":"2024-03-06T07:24:10.664833+0000","last_became_peered":"2024-03-06T07:24:10.664833+0000","last_unstale":"2024-03-06T07:24:12.915347+0000","last_undegraded":"2024-03-06T07:24:12.915347+0000","last_fullsized":"2024-03-06T07:24:12.915347+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:15.720920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.915230+0000","last_change":"2024-03-06T07:24:11.681452+0000","last_active":"2024-03-06T07:24:12.915230+0000","last_peered":"2024-03-06T07:24:12.915230+0000","last_clean":"2024-03-06T07:24:12.915230+0000","last_became_active":"2024-03-06T07:24:11.681164+0000","last_became_peered":"2024-03-06T07:24:11.681164+0000","last_unstale":"2024-03-06T07:24:12.915230+0000","last_undegraded":"2024-03-06T07:24:12.915230+0000","last_fullsized":"2024-03-06T07:24:12.915230+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:11.078896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"58'6","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914581+0000","last_change":"2024-03-06T07:24:10.662566+0000","last_active":"2024-03-06T07:24:12.914581+0000","last_peered":"2024-03-06T07:24:12.914581+0000","last_clean":"2024-03-06T07:24:12.914581+0000","last_became_active":"2024-03-06T07:24:10.662201+0000","last_became_peered":"2024-03-06T07:24:10.662201+0000","last_unstale":"2024-03-06T07:24:12.914581+0000","last_undegraded":"2024-03-06T07:24:12.914581+0000","last_fullsized":"2024-03-06T07:24:12.914581+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:37.876251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.658804+0000","last_change":"2024-03-06T07:24:11.668850+0000","last_active":"2024-03-06T07:24:12.658804+0000","last_peered":"2024-03-06T07:24:12.658804+0000","last_clean":"2024-03-06T07:24:12.658804+0000","last_became_active":"2024-03-06T07:24:11.668200+0000","last_became_peered":"2024-03-06T07:24:11.668200+0000","last_unstale":"2024-03-06T07:24:12.658804+0000","last_undegraded":"2024-03-06T07:24:12.658804+0000","last_fullsized":"2024-03-06T07:24:12.658804+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:00.898933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656724+0000","last_change":"2024-03-06T07:24:10.664502+0000","last_active":"2024-03-06T07:24:12.656724+0000","last_peered":"2024-03-06T07:24:12.656724+0000","last_clean":"2024-03-06T07:24:12.656724+0000","last_became_active":"2024-03-06T07:24:10.664248+0000","last_became_peered":"2024-03-06T07:24:10.664248+0000","last_unstale":"2024-03-06T07:24:12.656724+0000","last_undegraded":"2024-03-06T07:24:12.656724+0000","last_fullsized":"2024-03-06T07:24:12.656724+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:47.484732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914547+0000","last_change":"2024-03-06T07:24:11.677331+0000","last_active":"2024-03-06T07:24:12.914547+0000","last_peered":"2024-03-06T07:24:12.914547+0000","last_clean":"2024-03-06T07:24:12.914547+0000","last_became_active":"2024-03-06T07:24:11.677038+0000","last_became_peered":"2024-03-06T07:24:11.677038+0000","last_unstale":"2024-03-06T07:24:12.914547+0000","last_undegraded":"2024-03-06T07:24:12.914547+0000","last_fullsized":"2024-03-06T07:24:12.914547+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:22.091251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.677406+0000","last_change":"2024-03-06T07:24:10.655420+0000","last_active":"2024-03-06T07:24:12.677406+0000","last_peered":"2024-03-06T07:24:12.677406+0000","last_clean":"2024-03-06T07:24:12.677406+0000","last_became_active":"2024-03-06T07:24:10.655148+0000","last_became_peered":"2024-03-06T07:24:10.655148+0000","last_unstale":"2024-03-06T07:24:12.677406+0000","last_undegraded":"2024-03-06T07:24:12.677406+0000","last_fullsized":"2024-03-06T07:24:12.677406+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:18.988572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914117+0000","last_change":"2024-03-06T07:24:11.678650+0000","last_active":"2024-03-06T07:24:12.914117+0000","last_peered":"2024-03-06T07:24:12.914117+0000","last_clean":"2024-03-06T07:24:12.914117+0000","last_became_active":"2024-03-06T07:24:11.678347+0000","last_became_peered":"2024-03-06T07:24:11.678347+0000","last_unstale":"2024-03-06T07:24:12.914117+0000","last_undegraded":"2024-03-06T07:24:12.914117+0000","last_fullsized":"2024-03-06T07:24:12.914117+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:16.727405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.658781+0000","last_change":"2024-03-06T07:24:10.659555+0000","last_active":"2024-03-06T07:24:12.658781+0000","last_peered":"2024-03-06T07:24:12.658781+0000","last_clean":"2024-03-06T07:24:12.658781+0000","last_became_active":"2024-03-06T07:24:10.659285+0000","last_became_peered":"2024-03-06T07:24:10.659285+0000","last_unstale":"2024-03-06T07:24:12.658781+0000","last_undegraded":"2024-03-06T07:24:12.658781+0000","last_fullsized":"2024-03-06T07:24:12.658781+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:12.765996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914715+0000","last_change":"2024-03-06T07:24:11.683053+0000","last_active":"2024-03-06T07:24:12.914715+0000","last_peered":"2024-03-06T07:24:12.914715+0000","last_clean":"2024-03-06T07:24:12.914715+0000","last_became_active":"2024-03-06T07:24:11.682027+0000","last_became_peered":"2024-03-06T07:24:11.682027+0000","last_unstale":"2024-03-06T07:24:12.914715+0000","last_undegraded":"2024-03-06T07:24:12.914715+0000","last_fullsized":"2024-03-06T07:24:12.914715+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:23.795264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"58'8","reported_seq":22,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.915009+0000","last_change":"2024-03-06T07:24:10.661441+0000","last_active":"2024-03-06T07:24:12.915009+0000","last_peered":"2024-03-06T07:24:12.915009+0000","last_clean":"2024-03-06T07:24:12.915009+0000","last_became_active":"2024-03-06T07:24:10.661157+0000","last_became_peered":"2024-03-06T07:24:10.661157+0000","last_unstale":"2024-03-06T07:24:12.915009+0000","last_undegraded":"2024-03-06T07:24:12.915009+0000","last_fullsized":"2024-03-06T07:24:12.915009+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:08.249754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.915104+0000","last_change":"2024-03-06T07:24:11.680566+0000","last_active":"2024-03-06T07:24:12.915104+0000","last_peered":"2024-03-06T07:24:12.915104+0000","last_clean":"2024-03-06T07:24:12.915104+0000","last_became_active":"2024-03-06T07:24:11.680268+0000","last_became_peered":"2024-03-06T07:24:11.680268+0000","last_unstale":"2024-03-06T07:24:12.915104+0000","last_undegraded":"2024-03-06T07:24:12.915104+0000","last_fullsized":"2024-03-06T07:24:12.915104+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:41.270597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656600+0000","last_change":"2024-03-06T07:24:10.658071+0000","last_active":"2024-03-06T07:24:12.656600+0000","last_peered":"2024-03-06T07:24:12.656600+0000","last_clean":"2024-03-06T07:24:12.656600+0000","last_became_active":"2024-03-06T07:24:10.657790+0000","last_became_peered":"2024-03-06T07:24:10.657790+0000","last_unstale":"2024-03-06T07:24:12.656600+0000","last_undegraded":"2024-03-06T07:24:12.656600+0000","last_fullsized":"2024-03-06T07:24:12.656600+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:12.020566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.659285+0000","last_change":"2024-03-06T07:24:11.668290+0000","last_active":"2024-03-06T07:24:12.659285+0000","last_peered":"2024-03-06T07:24:12.659285+0000","last_clean":"2024-03-06T07:24:12.659285+0000","last_became_active":"2024-03-06T07:24:11.667886+0000","last_became_peered":"2024-03-06T07:24:11.667886+0000","last_unstale":"2024-03-06T07:24:12.659285+0000","last_undegraded":"2024-03-06T07:24:12.659285+0000","last_fullsized":"2024-03-06T07:24:12.659285+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:48.749848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.675299+0000","last_change":"2024-03-06T07:24:10.658700+0000","last_active":"2024-03-06T07:24:12.675299+0000","last_peered":"2024-03-06T07:24:12.675299+0000","last_clean":"2024-03-06T07:24:12.675299+0000","last_became_active":"2024-03-06T07:24:10.658430+0000","last_became_peered":"2024-03-06T07:24:10.658430+0000","last_unstale":"2024-03-06T07:24:12.675299+0000","last_undegraded":"2024-03-06T07:24:12.675299+0000","last_fullsized":"2024-03-06T07:24:12.675299+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:05.450215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656476+0000","last_change":"2024-03-06T07:24:11.678030+0000","last_active":"2024-03-06T07:24:12.656476+0000","last_peered":"2024-03-06T07:24:12.656476+0000","last_clean":"2024-03-06T07:24:12.656476+0000","last_became_active":"2024-03-06T07:24:11.677464+0000","last_became_peered":"2024-03-06T07:24:11.677464+0000","last_unstale":"2024-03-06T07:24:12.656476+0000","last_undegraded":"2024-03-06T07:24:12.656476+0000","last_fullsized":"2024-03-06T07:24:12.656476+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:55.103260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.659163+0000","last_change":"2024-03-06T07:24:10.659862+0000","last_active":"2024-03-06T07:24:12.659163+0000","last_peered":"2024-03-06T07:24:12.659163+0000","last_clean":"2024-03-06T07:24:12.659163+0000","last_became_active":"2024-03-06T07:24:10.659578+0000","last_became_peered":"2024-03-06T07:24:10.659578+0000","last_unstale":"2024-03-06T07:24:12.659163+0000","last_undegraded":"2024-03-06T07:24:12.659163+0000","last_fullsized":"2024-03-06T07:24:12.659163+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:36.030127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.915257+0000","last_change":"2024-03-06T07:24:11.686554+0000","last_active":"2024-03-06T07:24:12.915257+0000","last_peered":"2024-03-06T07:24:12.915257+0000","last_clean":"2024-03-06T07:24:12.915257+0000","last_became_active":"2024-03-06T07:24:11.686335+0000","last_became_peered":"2024-03-06T07:24:11.686335+0000","last_unstale":"2024-03-06T07:24:12.915257+0000","last_undegraded":"2024-03-06T07:24:12.915257+0000","last_fullsized":"2024-03-06T07:24:12.915257+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:36.975198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.675734+0000","last_change":"2024-03-06T07:24:10.658442+0000","last_active":"2024-03-06T07:24:12.675734+0000","last_peered":"2024-03-06T07:24:12.675734+0000","last_clean":"2024-03-06T07:24:12.675734+0000","last_became_active":"2024-03-06T07:24:10.658200+0000","last_became_peered":"2024-03-06T07:24:10.658200+0000","last_unstale":"2024-03-06T07:24:12.675734+0000","last_undegraded":"2024-03-06T07:24:12.675734+0000","last_fullsized":"2024-03-06T07:24:12.675734+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:07.246283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.658846+0000","last_change":"2024-03-06T07:24:11.668587+0000","last_active":"2024-03-06T07:24:12.658846+0000","last_peered":"2024-03-06T07:24:12.658846+0000","last_clean":"2024-03-06T07:24:12.658846+0000","last_became_active":"2024-03-06T07:24:11.667851+0000","last_became_peered":"2024-03-06T07:24:11.667851+0000","last_unstale":"2024-03-06T07:24:12.658846+0000","last_undegraded":"2024-03-06T07:24:12.658846+0000","last_fullsized":"2024-03-06T07:24:12.658846+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:38:22.091251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656868+0000","last_change":"2024-03-06T07:24:10.662365+0000","last_active":"2024-03-06T07:24:12.656868+0000","last_peered":"2024-03-06T07:24:12.656868+0000","last_clean":"2024-03-06T07:24:12.656868+0000","last_became_active":"2024-03-06T07:24:10.662073+0000","last_became_peered":"2024-03-06T07:24:10.662073+0000","last_unstale":"2024-03-06T07:24:12.656868+0000","last_undegraded":"2024-03-06T07:24:12.656868+0000","last_fullsized":"2024-03-06T07:24:12.656868+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:10.072412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.675371+0000","last_change":"2024-03-06T07:24:11.672676+0000","last_active":"2024-03-06T07:24:12.675371+0000","last_peered":"2024-03-06T07:24:12.675371+0000","last_clean":"2024-03-06T07:24:12.675371+0000","last_became_active":"2024-03-06T07:24:11.672386+0000","last_became_peered":"2024-03-06T07:24:11.672386+0000","last_unstale":"2024-03-06T07:24:12.675371+0000","last_undegraded":"2024-03-06T07:24:12.675371+0000","last_fullsized":"2024-03-06T07:24:12.675371+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:36.975198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914315+0000","last_change":"2024-03-06T07:24:10.659174+0000","last_active":"2024-03-06T07:24:12.914315+0000","last_peered":"2024-03-06T07:24:12.914315+0000","last_clean":"2024-03-06T07:24:12.914315+0000","last_became_active":"2024-03-06T07:24:10.658834+0000","last_became_peered":"2024-03-06T07:24:10.658834+0000","last_unstale":"2024-03-06T07:24:12.914315+0000","last_undegraded":"2024-03-06T07:24:12.914315+0000","last_fullsized":"2024-03-06T07:24:12.914315+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:05.337286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.674771+0000","last_change":"2024-03-06T07:24:11.683559+0000","last_active":"2024-03-06T07:24:12.674771+0000","last_peered":"2024-03-06T07:24:12.674771+0000","last_clean":"2024-03-06T07:24:12.674771+0000","last_became_active":"2024-03-06T07:24:11.683306+0000","last_became_peered":"2024-03-06T07:24:11.683306+0000","last_unstale":"2024-03-06T07:24:12.674771+0000","last_undegraded":"2024-03-06T07:24:12.674771+0000","last_fullsized":"2024-03-06T07:24:12.674771+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:08.382241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.658655+0000","last_change":"2024-03-06T07:24:10.656628+0000","last_active":"2024-03-06T07:24:12.658655+0000","last_peered":"2024-03-06T07:24:12.658655+0000","last_clean":"2024-03-06T07:24:12.658655+0000","last_became_active":"2024-03-06T07:24:10.656271+0000","last_became_peered":"2024-03-06T07:24:10.656271+0000","last_unstale":"2024-03-06T07:24:12.658655+0000","last_undegraded":"2024-03-06T07:24:12.658655+0000","last_fullsized":"2024-03-06T07:24:12.658655+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:26.674260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.659226+0000","last_change":"2024-03-06T07:24:11.665534+0000","last_active":"2024-03-06T07:24:12.659226+0000","last_peered":"2024-03-06T07:24:12.659226+0000","last_clean":"2024-03-06T07:24:12.659226+0000","last_became_active":"2024-03-06T07:24:11.665238+0000","last_became_peered":"2024-03-06T07:24:11.665238+0000","last_unstale":"2024-03-06T07:24:12.659226+0000","last_undegraded":"2024-03-06T07:24:12.659226+0000","last_fullsized":"2024-03-06T07:24:12.659226+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:23.795264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914775+0000","last_change":"2024-03-06T07:24:10.663013+0000","last_active":"2024-03-06T07:24:12.914775+0000","last_peered":"2024-03-06T07:24:12.914775+0000","last_clean":"2024-03-06T07:24:12.914775+0000","last_became_active":"2024-03-06T07:24:10.662736+0000","last_became_peered":"2024-03-06T07:24:10.662736+0000","last_unstale":"2024-03-06T07:24:12.914775+0000","last_undegraded":"2024-03-06T07:24:12.914775+0000","last_fullsized":"2024-03-06T07:24:12.914775+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.436630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.659327+0000","last_change":"2024-03-06T07:24:11.678056+0000","last_active":"2024-03-06T07:24:12.659327+0000","last_peered":"2024-03-06T07:24:12.659327+0000","last_clean":"2024-03-06T07:24:12.659327+0000","last_became_active":"2024-03-06T07:24:11.677776+0000","last_became_peered":"2024-03-06T07:24:11.677776+0000","last_unstale":"2024-03-06T07:24:12.659327+0000","last_undegraded":"2024-03-06T07:24:12.659327+0000","last_fullsized":"2024-03-06T07:24:12.659327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:41.270597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"59'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.675390+0000","last_change":"2024-03-06T07:24:10.654750+0000","last_active":"2024-03-06T07:24:12.675390+0000","last_peered":"2024-03-06T07:24:12.675390+0000","last_clean":"2024-03-06T07:24:12.675390+0000","last_became_active":"2024-03-06T07:24:10.654476+0000","last_became_peered":"2024-03-06T07:24:10.654476+0000","last_unstale":"2024-03-06T07:24:12.675390+0000","last_undegraded":"2024-03-06T07:24:12.675390+0000","last_fullsized":"2024-03-06T07:24:12.675390+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:31.570941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914491+0000","last_change":"2024-03-06T07:24:11.669159+0000","last_active":"2024-03-06T07:24:12.914491+0000","last_peered":"2024-03-06T07:24:12.914491+0000","last_clean":"2024-03-06T07:24:12.914491+0000","last_became_active":"2024-03-06T07:24:11.664674+0000","last_became_peered":"2024-03-06T07:24:11.664674+0000","last_unstale":"2024-03-06T07:24:12.914491+0000","last_undegraded":"2024-03-06T07:24:12.914491+0000","last_fullsized":"2024-03-06T07:24:12.914491+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:37.036611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.677578+0000","last_change":"2024-03-06T07:24:10.657981+0000","last_active":"2024-03-06T07:24:12.677578+0000","last_peered":"2024-03-06T07:24:12.677578+0000","last_clean":"2024-03-06T07:24:12.677578+0000","last_became_active":"2024-03-06T07:24:10.657616+0000","last_became_peered":"2024-03-06T07:24:10.657616+0000","last_unstale":"2024-03-06T07:24:12.677578+0000","last_undegraded":"2024-03-06T07:24:12.677578+0000","last_fullsized":"2024-03-06T07:24:12.677578+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:09.217923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.675110+0000","last_change":"2024-03-06T07:24:11.666476+0000","last_active":"2024-03-06T07:24:12.675110+0000","last_peered":"2024-03-06T07:24:12.675110+0000","last_clean":"2024-03-06T07:24:12.675110+0000","last_became_active":"2024-03-06T07:24:11.666170+0000","last_became_peered":"2024-03-06T07:24:11.666170+0000","last_unstale":"2024-03-06T07:24:12.675110+0000","last_undegraded":"2024-03-06T07:24:12.675110+0000","last_fullsized":"2024-03-06T07:24:12.675110+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:48.491217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.659261+0000","last_change":"2024-03-06T07:24:10.661755+0000","last_active":"2024-03-06T07:24:12.659261+0000","last_peered":"2024-03-06T07:24:12.659261+0000","last_clean":"2024-03-06T07:24:12.659261+0000","last_became_active":"2024-03-06T07:24:10.661432+0000","last_became_peered":"2024-03-06T07:24:10.661432+0000","last_unstale":"2024-03-06T07:24:12.659261+0000","last_undegraded":"2024-03-06T07:24:12.659261+0000","last_fullsized":"2024-03-06T07:24:12.659261+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:07.246283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656760+0000","last_change":"2024-03-06T07:24:11.682710+0000","last_active":"2024-03-06T07:24:12.656760+0000","last_peered":"2024-03-06T07:24:12.656760+0000","last_clean":"2024-03-06T07:24:12.656760+0000","last_became_active":"2024-03-06T07:24:11.681402+0000","last_became_peered":"2024-03-06T07:24:11.681402+0000","last_unstale":"2024-03-06T07:24:12.656760+0000","last_undegraded":"2024-03-06T07:24:12.656760+0000","last_fullsized":"2024-03-06T07:24:12.656760+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:22.866371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914333+0000","last_change":"2024-03-06T07:24:10.656363+0000","last_active":"2024-03-06T07:24:12.914333+0000","last_peered":"2024-03-06T07:24:12.914333+0000","last_clean":"2024-03-06T07:24:12.914333+0000","last_became_active":"2024-03-06T07:24:10.656056+0000","last_became_peered":"2024-03-06T07:24:10.656056+0000","last_unstale":"2024-03-06T07:24:12.914333+0000","last_undegraded":"2024-03-06T07:24:12.914333+0000","last_fullsized":"2024-03-06T07:24:12.914333+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:31.961753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914752+0000","last_change":"2024-03-06T07:24:11.684648+0000","last_active":"2024-03-06T07:24:12.914752+0000","last_peered":"2024-03-06T07:24:12.914752+0000","last_clean":"2024-03-06T07:24:12.914752+0000","last_became_active":"2024-03-06T07:24:11.684396+0000","last_became_peered":"2024-03-06T07:24:11.684396+0000","last_unstale":"2024-03-06T07:24:12.914752+0000","last_undegraded":"2024-03-06T07:24:12.914752+0000","last_fullsized":"2024-03-06T07:24:12.914752+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:08.382241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914327+0000","last_change":"2024-03-06T07:24:10.662788+0000","last_active":"2024-03-06T07:24:12.914327+0000","last_peered":"2024-03-06T07:24:12.914327+0000","last_clean":"2024-03-06T07:24:12.914327+0000","last_became_active":"2024-03-06T07:24:10.662446+0000","last_became_peered":"2024-03-06T07:24:10.662446+0000","last_unstale":"2024-03-06T07:24:12.914327+0000","last_undegraded":"2024-03-06T07:24:12.914327+0000","last_fullsized":"2024-03-06T07:24:12.914327+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:31.961753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914243+0000","last_change":"2024-03-06T07:24:11.674727+0000","last_active":"2024-03-06T07:24:12.914243+0000","last_peered":"2024-03-06T07:24:12.914243+0000","last_clean":"2024-03-06T07:24:12.914243+0000","last_became_active":"2024-03-06T07:24:11.674424+0000","last_became_peered":"2024-03-06T07:24:11.674424+0000","last_unstale":"2024-03-06T07:24:12.914243+0000","last_undegraded":"2024-03-06T07:24:12.914243+0000","last_fullsized":"2024-03-06T07:24:12.914243+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:13.772481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.914534+0000","last_change":"2024-03-06T07:24:10.649819+0000","last_active":"2024-03-06T07:24:12.914534+0000","last_peered":"2024-03-06T07:24:12.914534+0000","last_clean":"2024-03-06T07:24:12.914534+0000","last_became_active":"2024-03-06T07:24:10.649508+0000","last_became_peered":"2024-03-06T07:24:10.649508+0000","last_unstale":"2024-03-06T07:24:12.914534+0000","last_undegraded":"2024-03-06T07:24:12.914534+0000","last_fullsized":"2024-03-06T07:24:12.914534+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:39:55.005932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656391+0000","last_change":"2024-03-06T07:24:11.676951+0000","last_active":"2024-03-06T07:24:12.656391+0000","last_peered":"2024-03-06T07:24:12.656391+0000","last_clean":"2024-03-06T07:24:12.656391+0000","last_became_active":"2024-03-06T07:24:11.676371+0000","last_became_peered":"2024-03-06T07:24:11.676371+0000","last_unstale":"2024-03-06T07:24:12.656391+0000","last_undegraded":"2024-03-06T07:24:12.656391+0000","last_fullsized":"2024-03-06T07:24:12.656391+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:55.166783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656613+0000","last_change":"2024-03-06T07:24:10.656290+0000","last_active":"2024-03-06T07:24:12.656613+0000","last_peered":"2024-03-06T07:24:12.656613+0000","last_clean":"2024-03-06T07:24:12.656613+0000","last_became_active":"2024-03-06T07:24:10.656023+0000","last_became_peered":"2024-03-06T07:24:10.656023+0000","last_unstale":"2024-03-06T07:24:12.656613+0000","last_undegraded":"2024-03-06T07:24:12.656613+0000","last_fullsized":"2024-03-06T07:24:12.656613+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:50:22.788779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.656474+0000","last_change":"2024-03-06T07:24:11.678505+0000","last_active":"2024-03-06T07:24:12.656474+0000","last_peered":"2024-03-06T07:24:12.656474+0000","last_clean":"2024-03-06T07:24:12.656474+0000","last_became_active":"2024-03-06T07:24:11.678254+0000","last_became_peered":"2024-03-06T07:24:11.678254+0000","last_unstale":"2024-03-06T07:24:12.656474+0000","last_undegraded":"2024-03-06T07:24:12.656474+0000","last_fullsized":"2024-03-06T07:24:12.656474+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:10.639170+0000","last_clean_scrub_stamp":"2024-03-06T07:24:10.639170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:48.522273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.677405+0000","last_change":"2024-03-06T07:24:10.650351+0000","last_active":"2024-03-06T07:24:12.677405+0000","last_peered":"2024-03-06T07:24:12.677405+0000","last_clean":"2024-03-06T07:24:12.677405+0000","last_became_active":"2024-03-06T07:24:10.649966+0000","last_became_peered":"2024-03-06T07:24:10.649966+0000","last_unstale":"2024-03-06T07:24:12.677405+0000","last_undegraded":"2024-03-06T07:24:12.677405+0000","last_fullsized":"2024-03-06T07:24:12.677405+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:09.632685+0000","last_clean_scrub_stamp":"2024-03-06T07:24:09.632685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:06.010975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":242,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:12.915131+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:24:12.915131+0000","last_peered":"2024-03-06T07:24:12.915131+0000","last_clean":"2024-03-06T07:24:12.915131+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:24:12.915131+0000","last_undegraded":"2024-03-06T07:24:12.915131+0000","last_fullsized":"2024-03-06T07:24:12.915131+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574882,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1310720,"data_stored":886300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":708608,"data_stored":295772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":296424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574882,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":720896,"data_stored":297384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":716800,"data_stored":295828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":295828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31884,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339956,"statfs":{"total":96636764160,"available":96604114944,"internally_reserved":0,"allocated":712704,"data_stored":295862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":888334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:19.661 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:24:19.662 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:24:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.804+0000 7fd5a73a1640 1 Processor -- start 2024-03-06T07:24:19.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.804+0000 7fd5a73a1640 1 -- start start 2024-03-06T07:24:19.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.805+0000 7fd5a73a1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd5a015ecd0 0x7fd5a00cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.805+0000 7fd5a73a1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a00cfa50 con 0x7fd5a00d14a0 2024-03-06T07:24:19.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.805+0000 7fd5a73a1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a00b45c0 con 0x7fd5a00d10c0 2024-03-06T07:24:19.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.805+0000 7fd5a73a1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a015c050 con 0x7fd5a015ecd0 2024-03-06T07:24:19.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.805+0000 7fd5a593e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd5a015ecd0 0x7fd5a00cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.805+0000 7fd5a493c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd5a00d14a0 0x7fd5a015e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37694/0 (socket says 172.21.15.59:37694) 2024-03-06T07:24:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.805+0000 7fd5a493c640 1 -- 172.21.15.59:0/1122261306 learned_addr learned my addr 172.21.15.59:0/1122261306 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.805+0000 7fd5a593e640 1 -- 172.21.15.59:0/1122261306 >> v1:172.21.15.164:6789/0 conn(0x7fd5a00d10c0 legacy=0x7fd5a00d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.805+0000 7fd59ffff640 1 -- 172.21.15.59:0/1122261306 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863242886 0 0) 0x7fd5a00cfa50 con 0x7fd5a00d14a0 2024-03-06T07:24:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.806+0000 7fd5a593e640 1 -- 172.21.15.59:0/1122261306 >> v1:172.21.15.59:6789/0 conn(0x7fd5a00d14a0 legacy=0x7fd5a015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.806+0000 7fd5a593e640 1 -- 172.21.15.59:0/1122261306 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a015f160 con 0x7fd5a015ecd0 2024-03-06T07:24:19.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.806+0000 7fd5a593e640 1 --2- 172.21.15.59:0/1122261306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5a015ecd0 0x7fd5a00cae80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd5980097a0 tx=0x7fd59802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95f54b5bdea17e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.806+0000 7fd59ffff640 1 -- 172.21.15.59:0/1122261306 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598034070 con 0x7fd5a015ecd0 2024-03-06T07:24:19.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.806+0000 7fd59ffff640 1 -- 172.21.15.59:0/1122261306 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd598030610 con 0x7fd5a015ecd0 2024-03-06T07:24:19.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.806+0000 7fd59ffff640 1 -- 172.21.15.59:0/1122261306 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5980309e0 con 0x7fd5a015ecd0 2024-03-06T07:24:19.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.807+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1122261306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5a015ecd0 msgr2=0x7fd5a00cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.807+0000 7fd5a73a1640 1 --2- 172.21.15.59:0/1122261306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5a015ecd0 0x7fd5a00cae80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd5980097a0 tx=0x7fd59802b960 comp rx=0 tx=0).stop 2024-03-06T07:24:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.807+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1122261306 shutdown_connections 2024-03-06T07:24:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.807+0000 7fd5a73a1640 1 --2- 172.21.15.59:0/1122261306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5a015ecd0 0x7fd5a00cae80 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.807+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1122261306 >> 172.21.15.59:0/1122261306 conn(0x7fd5a00b3800 msgr2=0x7fd5a00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.807+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1122261306 shutdown_connections 2024-03-06T07:24:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.807+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1122261306 wait complete. 2024-03-06T07:24:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.809+0000 7fd5a73a1640 1 Processor -- start 2024-03-06T07:24:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.809+0000 7fd5a73a1640 1 -- start start 2024-03-06T07:24:19.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a73a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5a00d10c0 0x7fd5a014c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a73a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5a00d14a0 0x7fd5a00b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a73a1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5a00b5d00 0x7fd5a00b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a73a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00cfcf0 con 0x7fd5a00d10c0 2024-03-06T07:24:19.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a73a1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00d0060 con 0x7fd5a00d14a0 2024-03-06T07:24:19.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a73a1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00cb3c0 con 0x7fd5a00b5d00 2024-03-06T07:24:19.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a513d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5a00d10c0 0x7fd5a014c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a513d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5a00d10c0 0x7fd5a014c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50992/0 (socket says 172.21.15.59:50992) 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a513d640 1 -- 172.21.15.59:0/1995274671 learned_addr learned my addr 172.21.15.59:0/1995274671 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a493c640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5a00d14a0 0x7fd5a00b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a593e640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5a00b5d00 0x7fd5a00b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a513d640 1 -- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5a00b5d00 msgr2=0x7fd5a00b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a513d640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5a00b5d00 0x7fd5a00b63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a513d640 1 -- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5a00d14a0 msgr2=0x7fd5a00b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.810+0000 7fd5a513d640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5a00d14a0 0x7fd5a00b57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.811+0000 7fd5a513d640 1 -- 172.21.15.59:0/1995274671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a00b0c50 con 0x7fd5a00d10c0 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.811+0000 7fd5a513d640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5a00d10c0 0x7fd5a014c680 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c870 tx=0x7fd59000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.811+0000 7fd59dffb640 1 -- 172.21.15.59:0/1995274671 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590003b70 con 0x7fd5a00d10c0 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.811+0000 7fd59dffb640 1 -- 172.21.15.59:0/1995274671 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd590003d30 con 0x7fd5a00d10c0 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.811+0000 7fd59dffb640 1 -- 172.21.15.59:0/1995274671 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59000aa30 con 0x7fd5a00d10c0 2024-03-06T07:24:19.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.811+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1995274671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a014cbc0 con 0x7fd5a00d10c0 2024-03-06T07:24:19.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.811+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1995274671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a00b4ce0 con 0x7fd5a00d10c0 2024-03-06T07:24:19.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.812+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1995274671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a009fe60 con 0x7fd5a00d10c0 2024-03-06T07:24:19.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.812+0000 7fd59dffb640 1 -- 172.21.15.59:0/1995274671 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd59000abd0 con 0x7fd5a00d10c0 2024-03-06T07:24:19.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.813+0000 7fd59dffb640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd58c06b4d0 0x7fd58c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:19.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.813+0000 7fd59dffb640 1 -- 172.21.15.59:0/1995274671 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd59008a1a0 con 0x7fd5a00d10c0 2024-03-06T07:24:19.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.813+0000 7fd5a493c640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd58c06b4d0 0x7fd58c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:19.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.814+0000 7fd5a493c640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd58c06b4d0 0x7fd58c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd5a015dcc0 tx=0x7fd594007bb0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:19.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:19.817+0000 7fd59dffb640 1 -- 172.21.15.59:0/1995274671 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd590059020 con 0x7fd5a00d10c0 2024-03-06T07:24:20.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.071+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1995274671 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd5a0068ff0 con 0x7fd5a00d10c0 2024-03-06T07:24:20.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.075+0000 7fd59dffb640 1 -- 172.21.15.59:0/1995274671 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd590058920 con 0x7fd5a00d10c0 2024-03-06T07:24:20.077 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-06T07:24:20.077 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA== 2024-03-06T07:24:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-03-06T07:24:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-03-06T07:24:20.078 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:20.078 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-03-06T07:24:20.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd58c06b4d0 msgr2=0x7fd58c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd58c06b4d0 0x7fd58c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd5a015dcc0 tx=0x7fd594007bb0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1995274671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5a00d10c0 msgr2=0x7fd5a014c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5a00d10c0 0x7fd5a014c680 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c870 tx=0x7fd59000cd30 comp rx=0 tx=0).stop 2024-03-06T07:24:20.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1995274671 shutdown_connections 2024-03-06T07:24:20.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd58c06b4d0 0x7fd58c06d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5a00b5d00 0x7fd5a00b63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5a00d14a0 0x7fd5a00b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 --2- 172.21.15.59:0/1995274671 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5a00d10c0 0x7fd5a014c680 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1995274671 >> 172.21.15.59:0/1995274671 conn(0x7fd5a00b3800 msgr2=0x7fd5a00b2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1995274671 shutdown_connections 2024-03-06T07:24:20.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.078+0000 7fd5a73a1640 1 -- 172.21.15.59:0/1995274671 wait complete. 2024-03-06T07:24:20.091 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.191+0000 7f83b99b6640 1 Processor -- start 2024-03-06T07:24:20.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.191+0000 7f83b99b6640 1 -- start start 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.191+0000 7f83b99b6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f83b40b4170 0x7f83b40b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.191+0000 7f83b99b6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f83b40d4760 0x7f83b40d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.191+0000 7f83b99b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83b40b4dd0 0x7f83b40d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.191+0000 7f83b99b6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b40d4bf0 con 0x7f83b40d4760 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.191+0000 7f83b99b6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b40ce9e0 con 0x7f83b40b4170 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.191+0000 7f83b99b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40d25d0 con 0x7f83b40b4dd0 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.191+0000 7f83b27fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f83b40d4760 0x7f83b40d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.191+0000 7f83b27fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f83b40d4760 0x7f83b40d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50996/0 (socket says 172.21.15.59:50996) 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.191+0000 7f83b27fc640 1 -- 172.21.15.59:0/662312154 learned_addr learned my addr 172.21.15.59:0/662312154 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.192+0000 7f83b2ffd640 1 --2- 172.21.15.59:0/662312154 >> v2:172.21.15.164:3300/0 conn(0x7f83b40b4170 0x7f83b40b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.192+0000 7f83b37fe640 1 --2- 172.21.15.59:0/662312154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83b40b4dd0 0x7f83b40d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.192+0000 7f83b27fc640 1 -- 172.21.15.59:0/662312154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83b40b4dd0 msgr2=0x7f83b40d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.192+0000 7f83b27fc640 1 --2- 172.21.15.59:0/662312154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83b40b4dd0 0x7f83b40d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.192+0000 7f83b27fc640 1 -- 172.21.15.59:0/662312154 >> v2:172.21.15.164:3300/0 conn(0x7f83b40b4170 msgr2=0x7f83b40b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.192+0000 7f83b27fc640 1 --2- 172.21.15.59:0/662312154 >> v2:172.21.15.164:3300/0 conn(0x7f83b40b4170 0x7f83b40b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.192+0000 7f83b27fc640 1 -- 172.21.15.59:0/662312154 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b40d2870 con 0x7f83b40d4760 2024-03-06T07:24:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.192+0000 7f83b27fc640 1 --2- 172.21.15.59:0/662312154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83b40d4760 0x7f83b40d0370 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f83b406e030 tx=0x7f83a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2970e722c3862c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b1ffb640 1 -- 172.21.15.59:0/662312154 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8035020 con 0x7f83b40d4760 2024-03-06T07:24:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b1ffb640 1 -- 172.21.15.59:0/662312154 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8031870 con 0x7f83b40d4760 2024-03-06T07:24:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b1ffb640 1 -- 172.21.15.59:0/662312154 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8031c40 con 0x7f83b40d4760 2024-03-06T07:24:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b99b6640 1 -- 172.21.15.59:0/662312154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83b40d4760 msgr2=0x7f83b40d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b99b6640 1 --2- 172.21.15.59:0/662312154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83b40d4760 0x7f83b40d0370 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f83b406e030 tx=0x7f83a802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b99b6640 1 -- 172.21.15.59:0/662312154 shutdown_connections 2024-03-06T07:24:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b99b6640 1 --2- 172.21.15.59:0/662312154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83b40b4dd0 0x7f83b40d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b99b6640 1 --2- 172.21.15.59:0/662312154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83b40d4760 0x7f83b40d0370 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b99b6640 1 --2- 172.21.15.59:0/662312154 >> v2:172.21.15.164:3300/0 conn(0x7f83b40b4170 0x7f83b40b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b99b6640 1 -- 172.21.15.59:0/662312154 >> 172.21.15.59:0/662312154 conn(0x7f83b40d3c40 msgr2=0x7f83b40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b99b6640 1 -- 172.21.15.59:0/662312154 shutdown_connections 2024-03-06T07:24:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.193+0000 7f83b99b6640 1 -- 172.21.15.59:0/662312154 wait complete. 2024-03-06T07:24:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.195+0000 7f83b99b6640 1 Processor -- start 2024-03-06T07:24:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.195+0000 7f83b99b6640 1 -- start start 2024-03-06T07:24:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.195+0000 7f83b99b6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83b40b4170 0x7f83b40a1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b99b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83b40b4dd0 0x7f83b40c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b99b6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83b409d970 0x7f83b409f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b99b6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40b47d0 con 0x7f83b409d970 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b99b6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40b4550 con 0x7f83b40b4170 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b99b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40b4950 con 0x7f83b40b4dd0 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83b409d970 0x7f83b409f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83b409d970 0x7f83b409f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51012/0 (socket says 172.21.15.59:51012) 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b37fe640 1 -- 172.21.15.59:0/1538475684 learned_addr learned my addr 172.21.15.59:0/1538475684 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b27fc640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83b40b4dd0 0x7f83b40c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b2ffd640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83b40b4170 0x7f83b40a1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b37fe640 1 -- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83b40b4dd0 msgr2=0x7f83b40c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b37fe640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83b40b4dd0 0x7f83b40c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b37fe640 1 -- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83b40b4170 msgr2=0x7f83b40a1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b37fe640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83b40b4170 0x7f83b40a1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b37fe640 1 -- 172.21.15.59:0/1538475684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b40a1970 con 0x7f83b409d970 2024-03-06T07:24:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b37fe640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83b409d970 0x7f83b409f820 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f83a4007e40 tx=0x7f83a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f839bfff640 1 -- 172.21.15.59:0/1538475684 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a4014620 con 0x7f83b409d970 2024-03-06T07:24:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f839bfff640 1 -- 172.21.15.59:0/1538475684 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a4019070 con 0x7f83b409d970 2024-03-06T07:24:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b99b6640 1 -- 172.21.15.59:0/1538475684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b40c8ed0 con 0x7f83b409d970 2024-03-06T07:24:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f839bfff640 1 -- 172.21.15.59:0/1538475684 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a4014a20 con 0x7f83b409d970 2024-03-06T07:24:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.196+0000 7f83b99b6640 1 -- 172.21.15.59:0/1538475684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b40c9490 con 0x7f83b409d970 2024-03-06T07:24:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.197+0000 7f83b99b6640 1 -- 172.21.15.59:0/1538475684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b404f330 con 0x7f83b409d970 2024-03-06T07:24:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.198+0000 7f839bfff640 1 -- 172.21.15.59:0/1538475684 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83a4014bc0 con 0x7f83b409d970 2024-03-06T07:24:20.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.198+0000 7f839bfff640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.199+0000 7f839bfff640 1 -- 172.21.15.59:0/1538475684 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83a408ab40 con 0x7f83b409d970 2024-03-06T07:24:20.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.199+0000 7f83b2ffd640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.199+0000 7f83b2ffd640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f83ac006fd0 tx=0x7f83ac008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.203+0000 7f839bfff640 1 -- 172.21.15.59:0/1538475684 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83a40599c0 con 0x7f83b409d970 2024-03-06T07:24:20.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.422+0000 7f83b99b6640 1 -- 172.21.15.59:0/1538475684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83b40733b0 con 0x7f83b409d970 2024-03-06T07:24:20.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.423+0000 7f839bfff640 1 -- 172.21.15.59:0/1538475684 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f83a40592c0 con 0x7f83b409d970 2024-03-06T07:24:20.425 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:20.425 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2205562350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2205562350},{"type":"v1","addr":"172.21.15.164:6837","nonce":2205562350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2606750481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2606750481},{"type":"v1","addr":"172.21.15.164:6836","nonce":2606750481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6048,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3685052110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3685052110},{"type":"v1","addr":"172.21.15.59:6833","nonce":3685052110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:11.010795+0000","modified":"2024-03-06T07:24:13.053115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/750409746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":750409746},{"type":"v1","addr":"172.21.15.59:6835","nonce":750409746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":4}]} 2024-03-06T07:24:20.425 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 28 2024-03-06T07:24:20.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 -- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 msgr2=0x7f83a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f83ac006fd0 tx=0x7f83ac008040 comp rx=0 tx=0).stop 2024-03-06T07:24:20.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 -- 172.21.15.59:0/1538475684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83b409d970 msgr2=0x7f83b409f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83b409d970 0x7f83b409f820 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f83a4007e40 tx=0x7f83a400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 -- 172.21.15.59:0/1538475684 shutdown_connections 2024-03-06T07:24:20.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83b409d970 0x7f83b409f820 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83b40b4dd0 0x7f83b40c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 --2- 172.21.15.59:0/1538475684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83b40b4170 0x7f83b40a1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 -- 172.21.15.59:0/1538475684 >> 172.21.15.59:0/1538475684 conn(0x7f83b40d3c40 msgr2=0x7f83b40b7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 -- 172.21.15.59:0/1538475684 shutdown_connections 2024-03-06T07:24:20.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.425+0000 7f83b99b6640 1 -- 172.21.15.59:0/1538475684 wait complete. 2024-03-06T07:24:20.841 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:11.010795+0000', 'modified': '2024-03-06T07:24:13.053115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6075}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6075': {'gid': 6075, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/750409746', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 750409746}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 750409746}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6075, 'qdb_cluster': [6075]} 2024-03-06T07:24:20.841 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:20.841 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:24:20.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.943+0000 7f6fbf488640 1 Processor -- start 2024-03-06T07:24:20.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.943+0000 7f6fbf488640 1 -- start start 2024-03-06T07:24:20.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbf488640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6fb80b4460 0x7f6fb80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbf488640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6fb8162970 0x7f6fb80b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbf488640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80d22c0 con 0x7f6fb8161cf0 2024-03-06T07:24:20.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbf488640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb8161b00 con 0x7f6fb80b4460 2024-03-06T07:24:20.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbf488640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb80cecc0 con 0x7f6fb8162970 2024-03-06T07:24:20.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbd224640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6fb80b4460 0x7f6fb80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbda25640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6fb8161cf0 0x7f6fb80cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37698/0 (socket says 172.21.15.59:37698) 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbda25640 1 -- 172.21.15.59:0/3227928065 learned_addr learned my addr 172.21.15.59:0/3227928065 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbca23640 1 --2- 172.21.15.59:0/3227928065 >> v2:172.21.15.164:3300/0 conn(0x7f6fb8162970 0x7f6fb80b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbd224640 1 -- 172.21.15.59:0/3227928065 >> v2:172.21.15.164:3300/0 conn(0x7f6fb8162970 msgr2=0x7f6fb80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbd224640 1 --2- 172.21.15.59:0/3227928065 >> v2:172.21.15.164:3300/0 conn(0x7f6fb8162970 0x7f6fb80b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbd224640 1 -- 172.21.15.59:0/3227928065 >> v1:172.21.15.59:6789/0 conn(0x7f6fb8161cf0 legacy=0x7f6fb80cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbd224640 1 -- 172.21.15.59:0/3227928065 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb80d25a0 con 0x7f6fb80b4460 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6faffff640 1 -- 172.21.15.59:0/3227928065 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729470802 0 0) 0x7f6fb80d22c0 con 0x7f6fb8161cf0 2024-03-06T07:24:20.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.944+0000 7f6fbca23640 1 --2- 172.21.15.59:0/3227928065 >> v2:172.21.15.164:3300/0 conn(0x7f6fb8162970 0x7f6fb80b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.945+0000 7f6fbd224640 1 --2- 172.21.15.59:0/3227928065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb80b4460 0x7f6fb80d4800 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80d17e0 tx=0x7f6fb402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bdbf659bca5b2a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.945+0000 7f6faffff640 1 -- 172.21.15.59:0/3227928065 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4035020 con 0x7f6fb80b4460 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.945+0000 7f6faffff640 1 -- 172.21.15.59:0/3227928065 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb4031830 con 0x7f6fb80b4460 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.945+0000 7f6faffff640 1 -- 172.21.15.59:0/3227928065 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4031c00 con 0x7f6fb80b4460 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.946+0000 7f6fbf488640 1 -- 172.21.15.59:0/3227928065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb80b4460 msgr2=0x7f6fb80d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.946+0000 7f6fbf488640 1 --2- 172.21.15.59:0/3227928065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb80b4460 0x7f6fb80d4800 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80d17e0 tx=0x7f6fb402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.946+0000 7f6fbf488640 1 -- 172.21.15.59:0/3227928065 shutdown_connections 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.946+0000 7f6fbf488640 1 --2- 172.21.15.59:0/3227928065 >> v2:172.21.15.164:3300/0 conn(0x7f6fb8162970 0x7f6fb80b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.946+0000 7f6fbf488640 1 --2- 172.21.15.59:0/3227928065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb80b4460 0x7f6fb80d4800 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.946+0000 7f6fbf488640 1 -- 172.21.15.59:0/3227928065 >> 172.21.15.59:0/3227928065 conn(0x7f6fb80d3ce0 msgr2=0x7f6fb80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.946+0000 7f6fbf488640 1 -- 172.21.15.59:0/3227928065 shutdown_connections 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.946+0000 7f6fbf488640 1 -- 172.21.15.59:0/3227928065 wait complete. 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.947+0000 7f6fbf488640 1 Processor -- start 2024-03-06T07:24:20.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.947+0000 7f6fbf488640 1 -- start start 2024-03-06T07:24:20.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbf488640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb80b4460 0x7f6fb80c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbf488640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb8161cf0 0x7f6fb80cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbf488640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb8162970 0x7f6fb8156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbf488640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80ce950 con 0x7f6fb8162970 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbf488640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80d2b60 con 0x7f6fb8161cf0 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbf488640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80b3ac0 con 0x7f6fb80b4460 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbda25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb8162970 0x7f6fb8156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbda25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb8162970 0x7f6fb8156070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51020/0 (socket says 172.21.15.59:51020) 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbda25640 1 -- 172.21.15.59:0/140669876 learned_addr learned my addr 172.21.15.59:0/140669876 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbca23640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb8161cf0 0x7f6fb80cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbd224640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb80b4460 0x7f6fb80c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbda25640 1 -- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb80b4460 msgr2=0x7f6fb80c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbda25640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb80b4460 0x7f6fb80c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbda25640 1 -- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb8161cf0 msgr2=0x7f6fb80cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbda25640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb8161cf0 0x7f6fb80cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbda25640 1 -- 172.21.15.59:0/140669876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb80c7e80 con 0x7f6fb8162970 2024-03-06T07:24:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbca23640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb8161cf0 0x7f6fb80cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbda25640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb8162970 0x7f6fb8156070 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000c7a0 tx=0x7f6fb000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fadffb640 1 -- 172.21.15.59:0/140669876 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0018070 con 0x7f6fb8162970 2024-03-06T07:24:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fadffb640 1 -- 172.21.15.59:0/140669876 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb000a640 con 0x7f6fb8162970 2024-03-06T07:24:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fbf488640 1 -- 172.21.15.59:0/140669876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb80c8160 con 0x7f6fb8162970 2024-03-06T07:24:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.948+0000 7f6fadffb640 1 -- 172.21.15.59:0/140669876 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb000aa70 con 0x7f6fb8162970 2024-03-06T07:24:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.949+0000 7f6fbf488640 1 -- 172.21.15.59:0/140669876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb80c8720 con 0x7f6fb8162970 2024-03-06T07:24:20.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.950+0000 7f6fadffb640 1 -- 172.21.15.59:0/140669876 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fb001b070 con 0x7f6fb8162970 2024-03-06T07:24:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.950+0000 7f6fadffb640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 0x7f6f9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.950+0000 7f6fadffb640 1 -- 172.21.15.59:0/140669876 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fb0089790 con 0x7f6fb8162970 2024-03-06T07:24:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.951+0000 7f6fbd224640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 0x7f6f9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.951+0000 7f6fbd224640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 0x7f6f9c06d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4036000 tx=0x7f6fb4008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.952+0000 7f6fc2075740 1 Processor -- start 2024-03-06T07:24:20.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.953+0000 7f6fc2075740 1 -- start start 2024-03-06T07:24:20.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e6ff0870 0x5652e6ff0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e6ff48c0 0x5652e6ff6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fc2075740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e6f5a6b0 con 0x5652e7020600 2024-03-06T07:24:20.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fc2075740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e6f03a60 con 0x5652e6ff48c0 2024-03-06T07:24:20.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fc2075740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652e6f56750 con 0x5652e6ff0870 2024-03-06T07:24:20.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbda25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbda25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51024/0 (socket says 172.21.15.59:51024) 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbda25640 1 -- 172.21.15.59:0/3567468233 learned_addr learned my addr 172.21.15.59:0/3567468233 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbd224640 1 --2- 172.21.15.59:0/3567468233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e6ff0870 0x5652e6ff0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbca23640 1 --2- 172.21.15.59:0/3567468233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e6ff48c0 0x5652e6ff6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbda25640 1 -- 172.21.15.59:0/3567468233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e6ff0870 msgr2=0x5652e6ff0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbda25640 1 --2- 172.21.15.59:0/3567468233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e6ff0870 0x5652e6ff0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbda25640 1 -- 172.21.15.59:0/3567468233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e6ff48c0 msgr2=0x5652e6ff6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbda25640 1 --2- 172.21.15.59:0/3567468233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e6ff48c0 0x5652e6ff6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbda25640 1 -- 172.21.15.59:0/3567468233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652e6f21f30 con 0x5652e7020600 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbca23640 1 --2- 172.21.15.59:0/3567468233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e6ff48c0 0x5652e6ff6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fbda25640 1 --2- 172.21.15.59:0/3567468233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0087cf0 tx=0x7f6fb00586a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c50e062b293de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.954+0000 7f6fa2ffd640 1 -- 172.21.15.59:0/3567468233 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb00606b0 con 0x5652e7020600 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.955+0000 7f6fa2ffd640 1 -- 172.21.15.59:0/3567468233 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb00562a0 con 0x5652e7020600 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.955+0000 7f6fa2ffd640 1 -- 172.21.15.59:0/3567468233 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0060220 con 0x5652e7020600 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.955+0000 7f6fc2075740 1 -- 172.21.15.59:0/3567468233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 msgr2=0x5652e7020a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.955+0000 7f6fc2075740 1 --2- 172.21.15.59:0/3567468233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0087cf0 tx=0x7f6fb00586a0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.955+0000 7f6fc2075740 1 -- 172.21.15.59:0/3567468233 shutdown_connections 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.955+0000 7f6fc2075740 1 --2- 172.21.15.59:0/3567468233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a80 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.955+0000 7f6fc2075740 1 --2- 172.21.15.59:0/3567468233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e6ff48c0 0x5652e6ff6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.955+0000 7f6fc2075740 1 --2- 172.21.15.59:0/3567468233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e6ff0870 0x5652e6ff0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.955+0000 7f6fc2075740 1 -- 172.21.15.59:0/3567468233 >> 172.21.15.59:0/3567468233 conn(0x5652e6ff28f0 msgr2=0x5652e6ff41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.955+0000 7f6fc2075740 1 -- 172.21.15.59:0/3567468233 shutdown_connections 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.955+0000 7f6fc2075740 1 -- 172.21.15.59:0/3567468233 wait complete. 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.956+0000 7f6fc2075740 1 -- start start 2024-03-06T07:24:20.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.956+0000 7f6fc2075740 1 Processor -- start 2024-03-06T07:24:20.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.957+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e7020600 0x5652e70186b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.957+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e701cd40 0x5652e7024200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.957+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e704f5a0 0x5652e704fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.957+0000 7f6fc2075740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e6ff4d50 con 0x5652e701cd40 2024-03-06T07:24:20.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.957+0000 7f6fc2075740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e6f16ea0 con 0x5652e7020600 2024-03-06T07:24:20.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.957+0000 7f6fc2075740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652e6ff5030 con 0x5652e704f5a0 2024-03-06T07:24:20.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.957+0000 7f6fc2075740 10 client.0 _ll_register_callbacks cb 0x5652e6f16870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:20.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.957+0000 7f6fbca23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e701cd40 0x5652e7024200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.957+0000 7f6fbca23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e701cd40 0x5652e7024200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51036/0 (socket says 172.21.15.59:51036) 2024-03-06T07:24:20.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.957+0000 7f6fbca23640 1 -- 172.21.15.59:0/2802149038 learned_addr learned my addr 172.21.15.59:0/2802149038 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:20.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fbda25640 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e704f5a0 0x5652e704fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fbca23640 1 -- 172.21.15.59:0/2802149038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e704f5a0 msgr2=0x5652e704fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fbd224640 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e7020600 0x5652e70186b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fbca23640 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e704f5a0 0x5652e704fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fbca23640 1 -- 172.21.15.59:0/2802149038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e7020600 msgr2=0x5652e70186b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fbca23640 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e7020600 0x5652e70186b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fbca23640 1 -- 172.21.15.59:0/2802149038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652e6f17270 con 0x5652e701cd40 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fbca23640 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e701cd40 0x5652e7024200 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6fb815d440 tx=0x7f6fa80117c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d70448bb58836ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6f97fff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6f97fff640 1 -- 172.21.15.59:0/2802149038 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8019020 con 0x5652e701cd40 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6f97fff640 1 -- 172.21.15.59:0/2802149038 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fa8004240 con 0x5652e701cd40 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fc2075740 1 -- 172.21.15.59:0/2802149038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fb00562a0 con 0x5652e701cd40 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6f97fff640 1 -- 172.21.15.59:0/2802149038 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fa8023050 con 0x5652e701cd40 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fc2075740 10 client.6237 fetch_fsmap learned FSMap version 28 2024-03-06T07:24:20.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fc2075740 1 -- 172.21.15.59:0/2802149038 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x5652e6c6f2d0 con 0x5652e701cd40 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6f97fff640 1 -- 172.21.15.59:0/2802149038 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb00562a0 con 0x5652e701cd40 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6f97fff640 1 -- 172.21.15.59:0/2802149038 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 28) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6fa8026020 con 0x5652e701cd40 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fc2075740 10 client.6237 fetch_fsmap finished waiting for FSMap version 28 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.958+0000 7f6fc2075740 10 client.6237 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.959+0000 7f6fc2075740 20 client.6237 populate_metadata read hostname 'smithi059' 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.959+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.59:6834/750409746,v1:172.21.15.59:6835/750409746] conn(0x5652e7021fd0 0x5652e705a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.959+0000 7f6fc2075740 4 client.6237 mds_command: new command op to 6075 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.959+0000 7f6fc2075740 1 -- 172.21.15.59:0/2802149038 --> [v2:172.21.15.59:6834/750409746,v1:172.21.15.59:6835/750409746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652e6f67bf0 con 0x5652e7021fd0 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.959+0000 7f6fbd224640 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.59:6834/750409746,v1:172.21.15.59:6835/750409746] conn(0x5652e7021fd0 0x5652e705a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.959+0000 7f6f97fff640 10 client.6237 ms_handle_connect on v2:172.21.15.59:6834/750409746 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.959+0000 7f6fbd224640 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.59:6834/750409746,v1:172.21.15.59:6835/750409746] conn(0x5652e7021fd0 0x5652e705a450 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7da11403bdcf9e44 server_cookie=851a12a5511626c4 in_seq=0 out_seq=0 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6f97fff640 1 -- 172.21.15.59:0/2802149038 <== mds.0 v2:172.21.15.59:6834/750409746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5652e6f67bf0 con 0x5652e7021fd0 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6f97fff640 10 client.6237 handle_command_reply: tid=0 2024-03-06T07:24:20.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 1 client.6237 shutdown 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 1 -- 172.21.15.59:0/2802149038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e701cd40 msgr2=0x5652e7024200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e701cd40 0x5652e7024200 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6fb815d440 tx=0x7f6fa80117c0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 1 -- 172.21.15.59:0/2802149038 shutdown_connections 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.59:6834/750409746,v1:172.21.15.59:6835/750409746] conn(0x5652e7021fd0 0x5652e705a450 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e704f5a0 0x5652e704fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e701cd40 0x5652e7024200 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6f97fff640 0 client.6237 ms_handle_reset on v2:172.21.15.59:6834/750409746 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2802149038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e7020600 0x5652e70186b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 1 -- 172.21.15.59:0/2802149038 >> 172.21.15.59:0/2802149038 conn(0x5652e6ff0870 msgr2=0x5652e701e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 1 -- 172.21.15.59:0/2802149038 shutdown_connections 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 1 -- 172.21.15.59:0/2802149038 wait complete. 2024-03-06T07:24:20.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.960+0000 7f6fc2075740 20 client.6237 trim_cache size 0 max 16384 2024-03-06T07:24:20.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.993+0000 7f6fc2075740 1 Processor -- start 2024-03-06T07:24:20.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.993+0000 7f6fc2075740 1 -- start start 2024-03-06T07:24:20.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e701cd40 0x5652e704fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e703b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e704f5a0 0x5652e6fef960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fc2075740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80ca1a0 con 0x5652e7020600 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fc2075740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb81f7dc0 con 0x5652e704f5a0 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fc2075740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80c9ba0 con 0x5652e701cd40 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fbca23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e703b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fbca23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e703b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51044/0 (socket says 172.21.15.59:51044) 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fbca23640 1 -- 172.21.15.59:0/2449074505 learned_addr learned my addr 172.21.15.59:0/2449074505 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fbca23640 1 -- 172.21.15.59:0/2449074505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e701cd40 msgr2=0x5652e704fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fbca23640 1 --2- 172.21.15.59:0/2449074505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e701cd40 0x5652e704fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fbca23640 1 -- 172.21.15.59:0/2449074505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e704f5a0 msgr2=0x5652e6fef960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fbca23640 1 --2- 172.21.15.59:0/2449074505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e704f5a0 0x5652e6fef960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.994+0000 7f6fbca23640 1 -- 172.21.15.59:0/2449074505 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652e6d396b0 con 0x5652e7020600 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fbca23640 1 --2- 172.21.15.59:0/2449074505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e703b110 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800ac20 tx=0x7f6fa800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed77965cd6c109d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:20.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fa27fc640 1 -- 172.21.15.59:0/2449074505 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8019070 con 0x5652e7020600 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fa27fc640 1 -- 172.21.15.59:0/2449074505 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa802b020 con 0x5652e7020600 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fa27fc640 1 -- 172.21.15.59:0/2449074505 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8003610 con 0x5652e7020600 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fc2075740 1 -- 172.21.15.59:0/2449074505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 msgr2=0x5652e703b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2449074505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e703b110 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800ac20 tx=0x7f6fa800f000 comp rx=0 tx=0).stop 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fc2075740 1 -- 172.21.15.59:0/2449074505 shutdown_connections 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2449074505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e704f5a0 0x5652e6fef960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2449074505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e703b110 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2449074505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e701cd40 0x5652e704fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fc2075740 1 -- 172.21.15.59:0/2449074505 >> 172.21.15.59:0/2449074505 conn(0x5652e6ff0870 msgr2=0x5652e70190b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fc2075740 1 -- 172.21.15.59:0/2449074505 shutdown_connections 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.995+0000 7f6fc2075740 1 -- 172.21.15.59:0/2449074505 wait complete. 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.996+0000 7f6fc2075740 1 -- start start 2024-03-06T07:24:20.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.996+0000 7f6fc2075740 1 Processor -- start 2024-03-06T07:24:21.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e701cd40 0x5652e70ab040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fc2075740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e704f5a0 0x5652e7020f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fc2075740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e704f080 con 0x5652e7020600 2024-03-06T07:24:21.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fc2075740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e704c320 con 0x5652e701cd40 2024-03-06T07:24:21.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fc2075740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652e701d1a0 con 0x5652e704f5a0 2024-03-06T07:24:21.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fc2075740 10 client.0 _ll_register_callbacks cb 0x7f6fa800ac50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fbca23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fbca23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51054/0 (socket says 172.21.15.59:51054) 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fbca23640 1 -- 172.21.15.59:0/2450126940 learned_addr learned my addr 172.21.15.59:0/2450126940 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fbd224640 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e701cd40 0x5652e70ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fbda25640 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e704f5a0 0x5652e7020f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fbca23640 1 -- 172.21.15.59:0/2450126940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e704f5a0 msgr2=0x5652e7020f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fbca23640 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e704f5a0 0x5652e7020f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fbca23640 1 -- 172.21.15.59:0/2450126940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e701cd40 msgr2=0x5652e70ab040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fbca23640 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e701cd40 0x5652e70ab040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fbca23640 1 -- 172.21.15.59:0/2450126940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652e701dd10 con 0x5652e7020600 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6fbca23640 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6fa802d040 tx=0x7f6fa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48d53dce2c6c937 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.997+0000 7f6f97fff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6f97fff640 1 -- 172.21.15.59:0/2450126940 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8019040 con 0x5652e7020600 2024-03-06T07:24:21.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6f97fff640 1 -- 172.21.15.59:0/2450126940 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fa8015690 con 0x5652e7020600 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6fc2075740 1 -- 172.21.15.59:0/2450126940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fa802b020 con 0x5652e7020600 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6f97fff640 1 -- 172.21.15.59:0/2450126940 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fa803d050 con 0x5652e7020600 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6fc2075740 10 client.6243 fetch_fsmap learned FSMap version 28 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6fc2075740 1 -- 172.21.15.59:0/2450126940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x5652e705f190 con 0x5652e7020600 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6f97fff640 1 -- 172.21.15.59:0/2450126940 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa802b020 con 0x5652e7020600 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6f97fff640 1 -- 172.21.15.59:0/2450126940 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 28) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6fa8040020 con 0x5652e7020600 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6fc2075740 10 client.6243 fetch_fsmap finished waiting for FSMap version 28 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6fc2075740 10 client.6243 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6fc2075740 20 client.6243 populate_metadata read hostname 'smithi059' 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.59:6834/750409746,v1:172.21.15.59:6835/750409746] conn(0x5652e701ee40 0x5652e7048040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6fc2075740 4 client.6243 mds_command: new command op to 6075 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6fc2075740 1 -- 172.21.15.59:0/2450126940 --> [v2:172.21.15.59:6834/750409746,v1:172.21.15.59:6835/750409746] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5652e6ff3050 con 0x5652e701ee40 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.998+0000 7f6fbd224640 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.59:6834/750409746,v1:172.21.15.59:6835/750409746] conn(0x5652e701ee40 0x5652e7048040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6f97fff640 10 client.6243 ms_handle_connect on v2:172.21.15.59:6834/750409746 2024-03-06T07:24:21.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fbd224640 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.59:6834/750409746,v1:172.21.15.59:6835/750409746] conn(0x5652e701ee40 0x5652e7048040 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd9f9e5cfe31342c server_cookie=c2a913d0ca06c3ee in_seq=0 out_seq=0 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6f97fff640 1 -- 172.21.15.59:0/2450126940 <== mds.0 v2:172.21.15.59:6834/750409746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5652e6ff3050 con 0x5652e701ee40 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6f97fff640 10 client.6243 handle_command_reply: tid=0 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 1 client.6243 shutdown 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 1 -- 172.21.15.59:0/2450126940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 msgr2=0x5652e7020a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6fa802d040 tx=0x7f6fa80046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 1 -- 172.21.15.59:0/2450126940 shutdown_connections 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.59:6834/750409746,v1:172.21.15.59:6835/750409746] conn(0x5652e701ee40 0x5652e7048040 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5652e704f5a0 0x5652e7020f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5652e7020600 0x5652e7020a20 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6f97fff640 0 client.6243 ms_handle_reset on v2:172.21.15.59:6834/750409746 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 1 --2- 172.21.15.59:0/2450126940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5652e701cd40 0x5652e70ab040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 1 -- 172.21.15.59:0/2450126940 >> 172.21.15.59:0/2450126940 conn(0x5652e6ff0870 msgr2=0x5652e6febc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 1 -- 172.21.15.59:0/2450126940 shutdown_connections 2024-03-06T07:24:21.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 1 -- 172.21.15.59:0/2450126940 wait complete. 2024-03-06T07:24:21.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:20.999+0000 7f6fc2075740 20 client.6243 trim_cache size 0 max 16384 2024-03-06T07:24:21.007 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:24:21.007 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:24:21.007 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:24:21.007 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 6075, 2024-03-06T07:24:21.007 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:24:21.007 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:24:21.007 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:24:21.007 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 9.3348421930000001, 2024-03-06T07:24:21.008 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 28, 2024-03-06T07:24:21.008 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 59, 2024-03-06T07:24:21.008 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 59, 2024-03-06T07:24:21.008 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 14.396530778000001 2024-03-06T07:24:21.008 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:24:21.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 -- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 msgr2=0x7f6f9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 0x7f6f9c06d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4036000 tx=0x7f6fb4008030 comp rx=0 tx=0).stop 2024-03-06T07:24:21.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 -- 172.21.15.59:0/140669876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb8162970 msgr2=0x7f6fb8156070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb8162970 0x7f6fb8156070 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000c7a0 tx=0x7f6fb000cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:21.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 -- 172.21.15.59:0/140669876 shutdown_connections 2024-03-06T07:24:21.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 0x7f6f9c06d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb8162970 0x7f6fb8156070 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb8161cf0 0x7f6fb80cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 --2- 172.21.15.59:0/140669876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb80b4460 0x7f6fb80c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 -- 172.21.15.59:0/140669876 >> 172.21.15.59:0/140669876 conn(0x7f6fb80d3ce0 msgr2=0x7f6fb80d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 -- 172.21.15.59:0/140669876 shutdown_connections 2024-03-06T07:24:21.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.000+0000 7f6fbf488640 1 -- 172.21.15.59:0/140669876 wait complete. 2024-03-06T07:24:21.012 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:21.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.115+0000 7f5c2758a640 1 Processor -- start 2024-03-06T07:24:21.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.115+0000 7f5c2758a640 1 -- start start 2024-03-06T07:24:21.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c2758a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5c2015fa10 0x7f5c200ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c2758a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5c200d16e0 0x7f5c200d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c2758a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c200cebe0 con 0x7f5c2015bdd0 2024-03-06T07:24:21.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c2758a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c200ceec0 con 0x7f5c2015fa10 2024-03-06T07:24:21.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c2758a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c200cf1a0 con 0x7f5c200d16e0 2024-03-06T07:24:21.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c24b25640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5c2015fa10 0x7f5c200ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c24b25640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5c2015fa10 0x7f5c200ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51058/0 (socket says 172.21.15.59:51058) 2024-03-06T07:24:21.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c24b25640 1 -- 172.21.15.59:0/2237951238 learned_addr learned my addr 172.21.15.59:0/2237951238 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:21.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c25b27640 1 --2- 172.21.15.59:0/2237951238 >> v2:172.21.15.164:3300/0 conn(0x7f5c200d16e0 0x7f5c200d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c24b25640 1 -- 172.21.15.59:0/2237951238 >> v2:172.21.15.164:3300/0 conn(0x7f5c200d16e0 msgr2=0x7f5c200d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c24b25640 1 --2- 172.21.15.59:0/2237951238 >> v2:172.21.15.164:3300/0 conn(0x7f5c200d16e0 0x7f5c200d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c24b25640 1 -- 172.21.15.59:0/2237951238 >> v1:172.21.15.164:6789/0 conn(0x7f5c2015bdd0 legacy=0x7f5c200c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:21.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.116+0000 7f5c24b25640 1 -- 172.21.15.59:0/2237951238 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c200cc8e0 con 0x7f5c2015fa10 2024-03-06T07:24:21.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.117+0000 7f5c24b25640 1 --2- 172.21.15.59:0/2237951238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c2015fa10 0x7f5c200ce270 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5c200cdfd0 tx=0x7f5c1002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a447c4f23737db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.117+0000 7f5c17fff640 1 -- 172.21.15.59:0/2237951238 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c100317c0 con 0x7f5c2015fa10 2024-03-06T07:24:21.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.117+0000 7f5c17fff640 1 -- 172.21.15.59:0/2237951238 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c10037070 con 0x7f5c2015fa10 2024-03-06T07:24:21.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.117+0000 7f5c17fff640 1 -- 172.21.15.59:0/2237951238 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c10031b80 con 0x7f5c2015fa10 2024-03-06T07:24:21.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.118+0000 7f5c2758a640 1 -- 172.21.15.59:0/2237951238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c2015fa10 msgr2=0x7f5c200ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.118+0000 7f5c2758a640 1 --2- 172.21.15.59:0/2237951238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c2015fa10 0x7f5c200ce270 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5c200cdfd0 tx=0x7f5c1002b860 comp rx=0 tx=0).stop 2024-03-06T07:24:21.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.118+0000 7f5c2758a640 1 -- 172.21.15.59:0/2237951238 shutdown_connections 2024-03-06T07:24:21.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.118+0000 7f5c2758a640 1 --2- 172.21.15.59:0/2237951238 >> v2:172.21.15.164:3300/0 conn(0x7f5c200d16e0 0x7f5c200d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.118+0000 7f5c2758a640 1 --2- 172.21.15.59:0/2237951238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c2015fa10 0x7f5c200ce270 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.118+0000 7f5c2758a640 1 -- 172.21.15.59:0/2237951238 >> 172.21.15.59:0/2237951238 conn(0x7f5c2009a9f0 msgr2=0x7f5c2009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.118+0000 7f5c2758a640 1 -- 172.21.15.59:0/2237951238 shutdown_connections 2024-03-06T07:24:21.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.118+0000 7f5c2758a640 1 -- 172.21.15.59:0/2237951238 wait complete. 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.119+0000 7f5c2758a640 1 Processor -- start 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.119+0000 7f5c2758a640 1 -- start start 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c2758a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c200d16e0 0x7f5c2014ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c2758a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c2015bdd0 0x7f5c2006c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c2758a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c2015fa10 0x7f5c201e63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c2758a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c200d2030 con 0x7f5c2015bdd0 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c2758a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2015c1b0 con 0x7f5c2015fa10 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c2758a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c200ce7b0 con 0x7f5c200d16e0 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c24b25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c2015bdd0 0x7f5c2006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c24b25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c2015bdd0 0x7f5c2006c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51074/0 (socket says 172.21.15.59:51074) 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c24b25640 1 -- 172.21.15.59:0/4153453088 learned_addr learned my addr 172.21.15.59:0/4153453088 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c25326640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c200d16e0 0x7f5c2014ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c25b27640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c2015fa10 0x7f5c201e63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c24b25640 1 -- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c200d16e0 msgr2=0x7f5c2014ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c24b25640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c200d16e0 0x7f5c2014ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c24b25640 1 -- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c2015fa10 msgr2=0x7f5c201e63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c24b25640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c2015fa10 0x7f5c201e63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c24b25640 1 -- 172.21.15.59:0/4153453088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c200ca790 con 0x7f5c2015bdd0 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c25326640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c200d16e0 0x7f5c2014ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c25b27640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c2015fa10 0x7f5c201e63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c24b25640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c2015bdd0 0x7f5c2006c640 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5c1002b8a0 tx=0x7f5c100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c15ffb640 1 -- 172.21.15.59:0/4153453088 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c10032650 con 0x7f5c2015bdd0 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c15ffb640 1 -- 172.21.15.59:0/4153453088 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c10032810 con 0x7f5c2015bdd0 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c2758a640 1 -- 172.21.15.59:0/4153453088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c200caa70 con 0x7f5c2015bdd0 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.120+0000 7f5c15ffb640 1 -- 172.21.15.59:0/4153453088 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c10032c00 con 0x7f5c2015bdd0 2024-03-06T07:24:21.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.121+0000 7f5c2758a640 1 -- 172.21.15.59:0/4153453088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c200cb030 con 0x7f5c2015bdd0 2024-03-06T07:24:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.122+0000 7f5c2758a640 1 -- 172.21.15.59:0/4153453088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2004f330 con 0x7f5c2015bdd0 2024-03-06T07:24:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.123+0000 7f5c15ffb640 1 -- 172.21.15.59:0/4153453088 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c1004c060 con 0x7f5c2015bdd0 2024-03-06T07:24:21.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.123+0000 7f5c15ffb640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c0406b500 0x7f5c0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.123+0000 7f5c15ffb640 1 -- 172.21.15.59:0/4153453088 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c100abd10 con 0x7f5c2015bdd0 2024-03-06T07:24:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.123+0000 7f5c25326640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c0406b500 0x7f5c0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.124+0000 7f5c25326640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c0406b500 0x7f5c0406d9c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c003eb0 tx=0x7f5c1c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:21.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.127+0000 7f5c15ffb640 1 -- 172.21.15.59:0/4153453088 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c10040070 con 0x7f5c2015bdd0 2024-03-06T07:24:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.349+0000 7f5c2758a640 1 -- 172.21.15.59:0/4153453088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5c200c56f0 con 0x7f5c2015bdd0 2024-03-06T07:24:21.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.349+0000 7f5c15ffb640 1 -- 172.21.15.59:0/4153453088 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5c1007ab90 con 0x7f5c2015bdd0 2024-03-06T07:24:21.351 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.351+0000 7f5c2758a640 1 -- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c0406b500 msgr2=0x7f5c0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.351+0000 7f5c2758a640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c0406b500 0x7f5c0406d9c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c003eb0 tx=0x7f5c1c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.351+0000 7f5c2758a640 1 -- 172.21.15.59:0/4153453088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c2015bdd0 msgr2=0x7f5c2006c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.351+0000 7f5c2758a640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c2015bdd0 0x7f5c2006c640 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5c1002b8a0 tx=0x7f5c100046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.351+0000 7f5c2758a640 1 -- 172.21.15.59:0/4153453088 shutdown_connections 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.352+0000 7f5c2758a640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5c0406b500 0x7f5c0406d9c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.352+0000 7f5c2758a640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c2015fa10 0x7f5c201e63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.352+0000 7f5c2758a640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c2015bdd0 0x7f5c2006c640 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.352+0000 7f5c2758a640 1 --2- 172.21.15.59:0/4153453088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c200d16e0 0x7f5c2014ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.352+0000 7f5c2758a640 1 -- 172.21.15.59:0/4153453088 >> 172.21.15.59:0/4153453088 conn(0x7f5c2009a9f0 msgr2=0x7f5c2009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.352+0000 7f5c2758a640 1 -- 172.21.15.59:0/4153453088 shutdown_connections 2024-03-06T07:24:21.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:21.352+0000 7f5c2758a640 1 -- 172.21.15.59:0/4153453088 wait complete. 2024-03-06T07:24:21.364 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:21.364 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:21.364 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:24:21.364 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:21.364 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:21.365 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:21.365 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:21.365 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:21.365 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:21.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:21.365 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:21.380 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42517sec preferred_lft 42517sec 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:21.381 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:24:21.382 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:21.382 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:21.382 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:21.382 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:24:21.382 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:21.382 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:21.382 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:21.382 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:24:21.382 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:21.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:21.382 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:21.448 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:24:21.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:21.450 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:24:21.505 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:24:21.505 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:21.505 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:24:21.505 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:24:21.505 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:21.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:21.505 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:24:21.505 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:24:21.506 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:24:21.506 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:24:21.506 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:24:21.506 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:24:21.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:21.601 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:24:21.616 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:21.616 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:21.616 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:21.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:21.616 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:24:21.683 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:21.684 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:21.684 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:21.739 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:21.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:21.740 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:21.793 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:24:21.860 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:21.915 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:24:21.916 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:24:21.916 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:24:21.916 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:24:21.916 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:24:21.916 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:24:21.916 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:24:21.916 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:24:21.916 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:24:21.916 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:24:21.916 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:24:21.916 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:21.917 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:21.918 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:21.918 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:21.918 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:24:21.918 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:21.918 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:21.918 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:21.918 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:21.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:21.918 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:24:21.970 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:21.970 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:22.013 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:24:22.040 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:24:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:24:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:24:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:24:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:24:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:24:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:24:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:24:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:24:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:24:22.096 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:24:22.097 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:22.098 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:22.098 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:22.098 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:22.098 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:24:22.098 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:22.098 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:22.098 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:22.098 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:22.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:22.098 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:24:22.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:24:22.100+0000 7f9d3c03e480 -1 init, newargv = 0x55f36042b810 newargc=15 2024-03-06T07:24:22.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[58761]: starting ceph client 2024-03-06T07:24:22.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[58761]: starting fuse 2024-03-06T07:24:22.116 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:24:22.116 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:24:22.116 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:22.116 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:24:22.116 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:24:22.116 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:22.117 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:22.118 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:24:22.118 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:24:22.118 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:24:22.118 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:22.118 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:22.118 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:22.118 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:24:22.239 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.58761.asok 2024-03-06T07:24:22.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:22.247 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.58761.asok 2024-03-06T07:24:22.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:22.247 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.58761.asok status 2024-03-06T07:24:22.384 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "58761", 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:22.385 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 6255, 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 6255 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3759887992 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:22.386 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3759887992 2024-03-06T07:24:22.387 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:22.387 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.6255 192.168.0.1:0/3759887992", 2024-03-06T07:24:22.387 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/3759887992", 2024-03-06T07:24:22.387 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:24:22.387 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 28, 2024-03-06T07:24:22.387 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 59, 2024-03-06T07:24:22.387 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:22.387 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:24:22.387 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:24:22.387 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:24:22.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:22.393 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:24:22.393 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:22.409 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:24:22.409 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:22.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:22.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:22.487 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:24:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.628+0000 7fa4b6f35640 1 Processor -- start 2024-03-06T07:24:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.628+0000 7fa4b6f35640 1 -- start start 2024-03-06T07:24:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.629+0000 7fa4b6f35640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa4b00cb1b0 0x7fa4b0066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.629+0000 7fa4b6f35640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b006afb0 con 0x7fa4b01616d0 2024-03-06T07:24:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.629+0000 7fa4b6f35640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b00b5090 con 0x7fa4b0161ab0 2024-03-06T07:24:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.629+0000 7fa4b6f35640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b00b53c0 con 0x7fa4b00cb1b0 2024-03-06T07:24:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.629+0000 7fa4b54d2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa4b00cb1b0 0x7fa4b0066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.629+0000 7fa4b4cd1640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa4b01616d0 0x7fa4b00b5a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37708/0 (socket says 172.21.15.59:37708) 2024-03-06T07:24:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.629+0000 7fa4b4cd1640 1 -- 172.21.15.59:0/758199968 learned_addr learned my addr 172.21.15.59:0/758199968 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.629+0000 7fa4b54d2640 1 -- 172.21.15.59:0/758199968 >> v1:172.21.15.164:6789/0 conn(0x7fa4b0161ab0 legacy=0x7fa4b00caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.629+0000 7fa4b54d2640 1 -- 172.21.15.59:0/758199968 >> v1:172.21.15.59:6789/0 conn(0x7fa4b01616d0 legacy=0x7fa4b00b5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.629+0000 7fa4b54d2640 1 -- 172.21.15.59:0/758199968 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b006b250 con 0x7fa4b00cb1b0 2024-03-06T07:24:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.630+0000 7fa4b54d2640 1 --2- 172.21.15.59:0/758199968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4b00cb1b0 0x7fa4b0066d00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009d90 tx=0x7fa4a802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da3f479977f89bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.631+0000 7fa4a77fe640 1 -- 172.21.15.59:0/758199968 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8039070 con 0x7fa4b00cb1b0 2024-03-06T07:24:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.631+0000 7fa4a77fe640 1 -- 172.21.15.59:0/758199968 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a802cc60 con 0x7fa4b00cb1b0 2024-03-06T07:24:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.631+0000 7fa4a77fe640 1 -- 172.21.15.59:0/758199968 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8034660 con 0x7fa4b00cb1b0 2024-03-06T07:24:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.631+0000 7fa4b6f35640 1 -- 172.21.15.59:0/758199968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4b00cb1b0 msgr2=0x7fa4b0066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.631+0000 7fa4b6f35640 1 --2- 172.21.15.59:0/758199968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4b00cb1b0 0x7fa4b0066d00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009d90 tx=0x7fa4a802beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.631+0000 7fa4b6f35640 1 -- 172.21.15.59:0/758199968 shutdown_connections 2024-03-06T07:24:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.631+0000 7fa4b6f35640 1 --2- 172.21.15.59:0/758199968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4b00cb1b0 0x7fa4b0066d00 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.631+0000 7fa4b6f35640 1 -- 172.21.15.59:0/758199968 >> 172.21.15.59:0/758199968 conn(0x7fa4b00d0120 msgr2=0x7fa4b00d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.631+0000 7fa4b6f35640 1 -- 172.21.15.59:0/758199968 shutdown_connections 2024-03-06T07:24:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.631+0000 7fa4b6f35640 1 -- 172.21.15.59:0/758199968 wait complete. 2024-03-06T07:24:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.632+0000 7fa4b6f35640 1 Processor -- start 2024-03-06T07:24:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.632+0000 7fa4b6f35640 1 -- start start 2024-03-06T07:24:22.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.633+0000 7fa4b6f35640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b00cb1b0 0x7fa4b00ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.633+0000 7fa4b6f35640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4b01616d0 0x7fa4b00c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.633+0000 7fa4b6f35640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b0161ab0 0x7fa4b014fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.633+0000 7fa4b6f35640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b00aea70 con 0x7fa4b01616d0 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.633+0000 7fa4b6f35640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b006b530 con 0x7fa4b00cb1b0 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.633+0000 7fa4b6f35640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b00aebf0 con 0x7fa4b0161ab0 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.633+0000 7fa4a7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4b01616d0 0x7fa4b00c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.633+0000 7fa4a7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4b01616d0 0x7fa4b00c9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51082/0 (socket says 172.21.15.59:51082) 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.633+0000 7fa4a7fff640 1 -- 172.21.15.59:0/2858037184 learned_addr learned my addr 172.21.15.59:0/2858037184 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4b4cd1640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b00cb1b0 0x7fa4b00ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4b54d2640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b0161ab0 0x7fa4b014fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4a7fff640 1 -- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b0161ab0 msgr2=0x7fa4b014fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4a7fff640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b0161ab0 0x7fa4b014fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4a7fff640 1 -- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b00cb1b0 msgr2=0x7fa4b00ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4a7fff640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b00cb1b0 0x7fa4b00ad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4a7fff640 1 -- 172.21.15.59:0/2858037184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b00c82c0 con 0x7fa4b01616d0 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4a7fff640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4b01616d0 0x7fa4b00c9320 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000c430 tx=0x7fa4a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4a57fa640 1 -- 172.21.15.59:0/2858037184 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0017070 con 0x7fa4b01616d0 2024-03-06T07:24:22.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4a57fa640 1 -- 172.21.15.59:0/2858037184 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a00052d0 con 0x7fa4b01616d0 2024-03-06T07:24:22.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4b6f35640 1 -- 172.21.15.59:0/2858037184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b00c85c0 con 0x7fa4b01616d0 2024-03-06T07:24:22.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4a57fa640 1 -- 172.21.15.59:0/2858037184 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0003850 con 0x7fa4b01616d0 2024-03-06T07:24:22.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.634+0000 7fa4b6f35640 1 -- 172.21.15.59:0/2858037184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b00c8e00 con 0x7fa4b01616d0 2024-03-06T07:24:22.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.635+0000 7fa4b6f35640 1 -- 172.21.15.59:0/2858037184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b009e3d0 con 0x7fa4b01616d0 2024-03-06T07:24:22.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.636+0000 7fa4a57fa640 1 -- 172.21.15.59:0/2858037184 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4a001a070 con 0x7fa4b01616d0 2024-03-06T07:24:22.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.636+0000 7fa4a57fa640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa49c06b4d0 0x7fa49c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.636+0000 7fa4a57fa640 1 -- 172.21.15.59:0/2858037184 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4a0089680 con 0x7fa4b01616d0 2024-03-06T07:24:22.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.636+0000 7fa4b4cd1640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa49c06b4d0 0x7fa49c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.637+0000 7fa4b4cd1640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa49c06b4d0 0x7fa49c06d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac005880 tx=0x7fa4ac009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.640+0000 7fa4a57fa640 1 -- 172.21.15.59:0/2858037184 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4b00c85c0 con 0x7fa4b01616d0 2024-03-06T07:24:22.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.820+0000 7fa4b6f35640 1 -- 172.21.15.59:0/2858037184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7fa4b0068ff0 con 0x7fa4b01616d0 2024-03-06T07:24:22.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.824+0000 7fa4a57fa640 1 -- 172.21.15.59:0/2858037184 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa4a0058470 con 0x7fa4b01616d0 2024-03-06T07:24:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 -- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa49c06b4d0 msgr2=0x7fa49c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa49c06b4d0 0x7fa49c06d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac005880 tx=0x7fa4ac009040 comp rx=0 tx=0).stop 2024-03-06T07:24:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 -- 172.21.15.59:0/2858037184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4b01616d0 msgr2=0x7fa4b00c9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4b01616d0 0x7fa4b00c9320 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa4a000c430 tx=0x7fa4a000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 -- 172.21.15.59:0/2858037184 shutdown_connections 2024-03-06T07:24:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa49c06b4d0 0x7fa49c06d990 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b0161ab0 0x7fa4b014fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4b01616d0 0x7fa4b00c9320 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 --2- 172.21.15.59:0/2858037184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b00cb1b0 0x7fa4b00ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 -- 172.21.15.59:0/2858037184 >> 172.21.15.59:0/2858037184 conn(0x7fa4b00d0120 msgr2=0x7fa4b00caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 -- 172.21.15.59:0/2858037184 shutdown_connections 2024-03-06T07:24:22.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.826+0000 7fa4b6f35640 1 -- 172.21.15.59:0/2858037184 wait complete. 2024-03-06T07:24:22.838 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-03-06T07:24:22.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.941+0000 7f55da15e640 1 Processor -- start 2024-03-06T07:24:22.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.941+0000 7f55da15e640 1 -- start start 2024-03-06T07:24:22.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.941+0000 7f55da15e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55d41614d0 0x7f55d40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.941+0000 7f55da15e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f55d40d1910 0x7f55d40d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.941+0000 7f55da15e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f55d40d22b0 0x7f55d40d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.941+0000 7f55da15e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d40d2740 con 0x7f55d40d22b0 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.941+0000 7f55da15e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d41618b0 con 0x7f55d40d1910 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.941+0000 7f55da15e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40cf400 con 0x7f55d41614d0 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55d3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f55d40d22b0 0x7f55d40d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55d3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f55d40d22b0 0x7f55d40d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51086/0 (socket says 172.21.15.59:51086) 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55d3fff640 1 -- 172.21.15.59:0/1619614727 learned_addr learned my addr 172.21.15.59:0/1619614727 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55cbfff640 1 --2- 172.21.15.59:0/1619614727 >> v2:172.21.15.164:3300/0 conn(0x7f55d40d1910 0x7f55d40d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55d37fe640 1 --2- 172.21.15.59:0/1619614727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55d41614d0 0x7f55d40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55d3fff640 1 -- 172.21.15.59:0/1619614727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55d41614d0 msgr2=0x7f55d40d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55d3fff640 1 --2- 172.21.15.59:0/1619614727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55d41614d0 0x7f55d40d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55d3fff640 1 -- 172.21.15.59:0/1619614727 >> v2:172.21.15.164:3300/0 conn(0x7f55d40d1910 msgr2=0x7f55d40d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55d3fff640 1 --2- 172.21.15.59:0/1619614727 >> v2:172.21.15.164:3300/0 conn(0x7f55d40d1910 0x7f55d40d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55d3fff640 1 -- 172.21.15.59:0/1619614727 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d40cf6a0 con 0x7f55d40d22b0 2024-03-06T07:24:22.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55cbfff640 1 --2- 172.21.15.59:0/1619614727 >> v2:172.21.15.164:3300/0 conn(0x7f55d40d1910 0x7f55d40d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:22.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.942+0000 7f55d37fe640 1 --2- 172.21.15.59:0/1619614727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55d41614d0 0x7f55d40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:22.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.943+0000 7f55d3fff640 1 --2- 172.21.15.59:0/1619614727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55d40d22b0 0x7f55d40d06a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f55d408c4f0 tx=0x7f55cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a2ab2cfe9fe70a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.943+0000 7f55d2ffd640 1 -- 172.21.15.59:0/1619614727 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc02ca70 con 0x7f55d40d22b0 2024-03-06T07:24:22.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.943+0000 7f55d2ffd640 1 -- 172.21.15.59:0/1619614727 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc02cc10 con 0x7f55d40d22b0 2024-03-06T07:24:22.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.943+0000 7f55d2ffd640 1 -- 172.21.15.59:0/1619614727 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0343f0 con 0x7f55d40d22b0 2024-03-06T07:24:22.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.943+0000 7f55da15e640 1 -- 172.21.15.59:0/1619614727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55d40d22b0 msgr2=0x7f55d40d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.943+0000 7f55da15e640 1 --2- 172.21.15.59:0/1619614727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55d40d22b0 0x7f55d40d06a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f55d408c4f0 tx=0x7f55cc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.943+0000 7f55da15e640 1 -- 172.21.15.59:0/1619614727 shutdown_connections 2024-03-06T07:24:22.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.943+0000 7f55da15e640 1 --2- 172.21.15.59:0/1619614727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55d40d22b0 0x7f55d40d06a0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.943+0000 7f55da15e640 1 --2- 172.21.15.59:0/1619614727 >> v2:172.21.15.164:3300/0 conn(0x7f55d40d1910 0x7f55d40d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.943+0000 7f55da15e640 1 --2- 172.21.15.59:0/1619614727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55d41614d0 0x7f55d40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.943+0000 7f55da15e640 1 -- 172.21.15.59:0/1619614727 >> 172.21.15.59:0/1619614727 conn(0x7f55d40d3d40 msgr2=0x7f55d40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.944+0000 7f55da15e640 1 -- 172.21.15.59:0/1619614727 shutdown_connections 2024-03-06T07:24:22.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.944+0000 7f55da15e640 1 -- 172.21.15.59:0/1619614727 wait complete. 2024-03-06T07:24:22.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.945+0000 7f55da15e640 1 Processor -- start 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.945+0000 7f55da15e640 1 -- start start 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.945+0000 7f55da15e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55d40d1910 0x7f55d415e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.945+0000 7f55da15e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55d40d22b0 0x7f55d40cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.945+0000 7f55da15e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55d41614d0 0x7f55d41600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.945+0000 7f55da15e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40b0910 con 0x7f55d41614d0 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.945+0000 7f55da15e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40cfc60 con 0x7f55d40d1910 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.945+0000 7f55da15e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40d0be0 con 0x7f55d40d22b0 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.945+0000 7f55d3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55d41614d0 0x7f55d41600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55d41614d0 0x7f55d41600b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51096/0 (socket says 172.21.15.59:51096) 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d3fff640 1 -- 172.21.15.59:0/4219326543 learned_addr learned my addr 172.21.15.59:0/4219326543 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d37fe640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55d40d1910 0x7f55d415e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55cbfff640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55d40d22b0 0x7f55d40cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d3fff640 1 -- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55d40d22b0 msgr2=0x7f55d40cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d3fff640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55d40d22b0 0x7f55d40cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d3fff640 1 -- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55d40d1910 msgr2=0x7f55d415e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d3fff640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55d40d1910 0x7f55d415e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d3fff640 1 -- 172.21.15.59:0/4219326543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d40cbba0 con 0x7f55d41614d0 2024-03-06T07:24:22.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d37fe640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55d40d1910 0x7f55d415e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:22.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d3fff640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55d41614d0 0x7f55d41600b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009f40 tx=0x7f55cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d0ff9640 1 -- 172.21.15.59:0/4219326543 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc043020 con 0x7f55d41614d0 2024-03-06T07:24:22.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d0ff9640 1 -- 172.21.15.59:0/4219326543 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc03e450 con 0x7f55d41614d0 2024-03-06T07:24:22.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55d0ff9640 1 -- 172.21.15.59:0/4219326543 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc03e840 con 0x7f55d41614d0 2024-03-06T07:24:22.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55da15e640 1 -- 172.21.15.59:0/4219326543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d40cd290 con 0x7f55d41614d0 2024-03-06T07:24:22.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.946+0000 7f55da15e640 1 -- 172.21.15.59:0/4219326543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d40cdb30 con 0x7f55d41614d0 2024-03-06T07:24:22.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.947+0000 7f55da15e640 1 -- 172.21.15.59:0/4219326543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d4068ff0 con 0x7f55d41614d0 2024-03-06T07:24:22.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.948+0000 7f55d0ff9640 1 -- 172.21.15.59:0/4219326543 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55cc046020 con 0x7f55d41614d0 2024-03-06T07:24:22.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.948+0000 7f55d0ff9640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55b406b4d0 0x7f55b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.948+0000 7f55d0ff9640 1 -- 172.21.15.59:0/4219326543 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55cc0accb0 con 0x7f55d41614d0 2024-03-06T07:24:22.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.948+0000 7f55d37fe640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55b406b4d0 0x7f55b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.949+0000 7f55d37fe640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55b406b4d0 0x7f55b406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f55c400a910 tx=0x7f55c4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:22.952+0000 7f55d0ff9640 1 -- 172.21.15.59:0/4219326543 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55cc04d030 con 0x7f55d41614d0 2024-03-06T07:24:23.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.133+0000 7f55da15e640 1 -- 172.21.15.59:0/4219326543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f55d40a1c10 con 0x7f55d41614d0 2024-03-06T07:24:23.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.136+0000 7f55d0ff9640 1 -- 172.21.15.59:0/4219326543 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7f55cc07ba90 con 0x7f55d41614d0 2024-03-06T07:24:23.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 -- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55b406b4d0 msgr2=0x7f55b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55b406b4d0 0x7f55b406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f55c400a910 tx=0x7f55c4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 -- 172.21.15.59:0/4219326543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55d41614d0 msgr2=0x7f55d41600b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55d41614d0 0x7f55d41600b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009f40 tx=0x7f55cc004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 -- 172.21.15.59:0/4219326543 shutdown_connections 2024-03-06T07:24:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55b406b4d0 0x7f55b406d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55d41614d0 0x7f55d41600b0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55d40d22b0 0x7f55d40cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 --2- 172.21.15.59:0/4219326543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55d40d1910 0x7f55d415e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 -- 172.21.15.59:0/4219326543 >> 172.21.15.59:0/4219326543 conn(0x7f55d40d3d40 msgr2=0x7f55d406bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 -- 172.21.15.59:0/4219326543 shutdown_connections 2024-03-06T07:24:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:23.138+0000 7f55da15e640 1 -- 172.21.15.59:0/4219326543 wait complete. 2024-03-06T07:24:33.152 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:24:33.152 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> stdin-killer -- python3 -c ' 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> import uuid 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> n = 50 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> create_hard_links = False 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> if 0 > 0: 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> create_hard_links = True 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> dpath = os.path.dirname(path) 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:33.168 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> try: 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> for i in range(n): 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> f.write(f"{i}") 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> f.flush() 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(f.fileno()) 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> os.unlink(fpath) 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(dirfd) 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> if create_hard_links: 2024-03-06T07:24:33.169 DEBUG:teuthology.orchestra.run.smithi059:> for j in range(0): 2024-03-06T07:24:33.170 DEBUG:teuthology.orchestra.run.smithi059:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:33.170 DEBUG:teuthology.orchestra.run.smithi059:> if True: 2024-03-06T07:24:33.170 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(dirfd) 2024-03-06T07:24:33.170 DEBUG:teuthology.orchestra.run.smithi059:> finally: 2024-03-06T07:24:33.170 DEBUG:teuthology.orchestra.run.smithi059:> os.close(dirfd) 2024-03-06T07:24:33.170 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:24:33.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:33.393 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:24:33.393 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:33.449 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:33.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.588+0000 7f655378a640 1 Processor -- start 2024-03-06T07:24:33.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.589+0000 7f655378a640 1 -- start start 2024-03-06T07:24:33.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.589+0000 7f655378a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f654c15dfe0 0x7f654c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.589+0000 7f655378a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f654c0cf830 0x7f654c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.589+0000 7f655378a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f654c0d27f0 0x7f654c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.589+0000 7f655378a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654c0d2c50 con 0x7f654c0d27f0 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.589+0000 7f655378a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654c0b31e0 con 0x7f654c0cf830 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.589+0000 7f655378a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654c0b3d60 con 0x7f654c15dfe0 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6551d27640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f654c0d27f0 0x7f654c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6551d27640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f654c0d27f0 0x7f654c0d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38994/0 (socket says 172.21.15.59:38994) 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6551d27640 1 -- 172.21.15.59:0/1195448130 learned_addr learned my addr 172.21.15.59:0/1195448130 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6550d25640 1 --2- 172.21.15.59:0/1195448130 >> v2:172.21.15.164:3300/0 conn(0x7f654c0cf830 0x7f654c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6551d27640 1 -- 172.21.15.59:0/1195448130 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f654c15dfe0 msgr2=0x7f654c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6551526640 1 --2- 172.21.15.59:0/1195448130 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f654c15dfe0 0x7f654c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6551d27640 1 --2- 172.21.15.59:0/1195448130 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f654c15dfe0 0x7f654c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6551d27640 1 -- 172.21.15.59:0/1195448130 >> v2:172.21.15.164:3300/0 conn(0x7f654c0cf830 msgr2=0x7f654c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6551d27640 1 --2- 172.21.15.59:0/1195448130 >> v2:172.21.15.164:3300/0 conn(0x7f654c0cf830 0x7f654c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6551d27640 1 -- 172.21.15.59:0/1195448130 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654c0b3430 con 0x7f654c0d27f0 2024-03-06T07:24:33.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6550d25640 1 --2- 172.21.15.59:0/1195448130 >> v2:172.21.15.164:3300/0 conn(0x7f654c0cf830 0x7f654c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:33.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.590+0000 7f6551526640 1 --2- 172.21.15.59:0/1195448130 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f654c15dfe0 0x7f654c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:33.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f6551d27640 1 --2- 172.21.15.59:0/1195448130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f654c0d27f0 0x7f654c0d0300 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f654c08cba0 tx=0x7f654402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9e15a9e0caac4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f654bfff640 1 -- 172.21.15.59:0/1195448130 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544035020 con 0x7f654c0d27f0 2024-03-06T07:24:33.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f654bfff640 1 -- 172.21.15.59:0/1195448130 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6544031870 con 0x7f654c0d27f0 2024-03-06T07:24:33.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f654bfff640 1 -- 172.21.15.59:0/1195448130 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544031c40 con 0x7f654c0d27f0 2024-03-06T07:24:33.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f655378a640 1 -- 172.21.15.59:0/1195448130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f654c0d27f0 msgr2=0x7f654c0d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f655378a640 1 --2- 172.21.15.59:0/1195448130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f654c0d27f0 0x7f654c0d0300 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f654c08cba0 tx=0x7f654402b910 comp rx=0 tx=0).stop 2024-03-06T07:24:33.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f655378a640 1 -- 172.21.15.59:0/1195448130 shutdown_connections 2024-03-06T07:24:33.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f655378a640 1 --2- 172.21.15.59:0/1195448130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f654c0d27f0 0x7f654c0d0300 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f655378a640 1 --2- 172.21.15.59:0/1195448130 >> v2:172.21.15.164:3300/0 conn(0x7f654c0cf830 0x7f654c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f655378a640 1 --2- 172.21.15.59:0/1195448130 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f654c15dfe0 0x7f654c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f655378a640 1 -- 172.21.15.59:0/1195448130 >> 172.21.15.59:0/1195448130 conn(0x7f654c0d3cd0 msgr2=0x7f654c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f655378a640 1 -- 172.21.15.59:0/1195448130 shutdown_connections 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.591+0000 7f655378a640 1 -- 172.21.15.59:0/1195448130 wait complete. 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.593+0000 7f655378a640 1 Processor -- start 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.593+0000 7f655378a640 1 -- start start 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.593+0000 7f655378a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f654c0cf830 0x7f654c15a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.593+0000 7f655378a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f654c0d27f0 0x7f654c0b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.593+0000 7f655378a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f654c15dfe0 0x7f654c0ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.593+0000 7f655378a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654c0d1c70 con 0x7f654c0d27f0 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.593+0000 7f655378a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654c0b39f0 con 0x7f654c15dfe0 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.593+0000 7f655378a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654c0d1df0 con 0x7f654c0cf830 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.593+0000 7f6550d25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f654c0d27f0 0x7f654c0b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.593+0000 7f6550d25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f654c0d27f0 0x7f654c0b7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39008/0 (socket says 172.21.15.59:39008) 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.593+0000 7f6550d25640 1 -- 172.21.15.59:0/4133668346 learned_addr learned my addr 172.21.15.59:0/4133668346 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6551526640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f654c0cf830 0x7f654c15a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6551d27640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f654c15dfe0 0x7f654c0ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6550d25640 1 -- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f654c0cf830 msgr2=0x7f654c15a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6550d25640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f654c0cf830 0x7f654c15a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6550d25640 1 -- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f654c15dfe0 msgr2=0x7f654c0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6550d25640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f654c15dfe0 0x7f654c0ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6550d25640 1 -- 172.21.15.59:0/4133668346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654c15e490 con 0x7f654c0d27f0 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6551d27640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f654c15dfe0 0x7f654c0ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6550d25640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f654c0d27f0 0x7f654c0b7f90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f654000c7c0 tx=0x7f654000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6549ffb640 1 -- 172.21.15.59:0/4133668346 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540015070 con 0x7f654c0d27f0 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6549ffb640 1 -- 172.21.15.59:0/4133668346 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6540007820 con 0x7f654c0d27f0 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f655378a640 1 -- 172.21.15.59:0/4133668346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654c15e770 con 0x7f654c0d27f0 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f6549ffb640 1 -- 172.21.15.59:0/4133668346 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540007bf0 con 0x7f654c0d27f0 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.594+0000 7f655378a640 1 -- 172.21.15.59:0/4133668346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654c15ed30 con 0x7f654c0d27f0 2024-03-06T07:24:33.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.595+0000 7f655378a640 1 -- 172.21.15.59:0/4133668346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f654c04f330 con 0x7f654c0d27f0 2024-03-06T07:24:33.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.596+0000 7f6549ffb640 1 -- 172.21.15.59:0/4133668346 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6540018070 con 0x7f654c0d27f0 2024-03-06T07:24:33.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.597+0000 7f6549ffb640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f653806b4d0 0x7f653806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.597+0000 7f6549ffb640 1 -- 172.21.15.59:0/4133668346 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f654008a180 con 0x7f654c0d27f0 2024-03-06T07:24:33.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.597+0000 7f6551526640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f653806b4d0 0x7f653806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.597+0000 7f6551526640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f653806b4d0 0x7f653806d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f653c007510 tx=0x7f653c0074a0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.601+0000 7f6549ffb640 1 -- 172.21.15.59:0/4133668346 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6540058f60 con 0x7f654c0d27f0 2024-03-06T07:24:33.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.783+0000 7f655378a640 1 -- 172.21.15.59:0/4133668346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f654c15aa30 con 0x7f654c0d27f0 2024-03-06T07:24:33.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.784+0000 7f6549ffb640 1 -- 172.21.15.59:0/4133668346 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f6540058860 con 0x7f654c0d27f0 2024-03-06T07:24:33.786 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:33.787 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:24:12.652576+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:09.288329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.978574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:33.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 -- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f653806b4d0 msgr2=0x7f653806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f653806b4d0 0x7f653806d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f653c007510 tx=0x7f653c0074a0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 -- 172.21.15.59:0/4133668346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f654c0d27f0 msgr2=0x7f654c0b7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f654c0d27f0 0x7f654c0b7f90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f654000c7c0 tx=0x7f654000cc80 comp rx=0 tx=0).stop 2024-03-06T07:24:33.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 -- 172.21.15.59:0/4133668346 shutdown_connections 2024-03-06T07:24:33.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f653806b4d0 0x7f653806d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f654c15dfe0 0x7f654c0ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f654c0d27f0 0x7f654c0b7f90 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 --2- 172.21.15.59:0/4133668346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f654c0cf830 0x7f654c15a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 -- 172.21.15.59:0/4133668346 >> 172.21.15.59:0/4133668346 conn(0x7f654c0d3cd0 msgr2=0x7f654c06cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 -- 172.21.15.59:0/4133668346 shutdown_connections 2024-03-06T07:24:33.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.786+0000 7f655378a640 1 -- 172.21.15.59:0/4133668346 wait complete. 2024-03-06T07:24:33.799 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:09.288329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:09.978574+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:33.799 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:33.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.920+0000 7fdbc6b1e640 1 Processor -- start 2024-03-06T07:24:33.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.920+0000 7fdbc6b1e640 1 -- start start 2024-03-06T07:24:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.920+0000 7fdbc6b1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdbc006cab0 0x7fdbc00b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.920+0000 7fdbc6b1e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc006d030 con 0x7fdbc00cea50 2024-03-06T07:24:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.920+0000 7fdbc6b1e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc006d310 con 0x7fdbc015def0 2024-03-06T07:24:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.920+0000 7fdbc6b1e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc00cf1c0 con 0x7fdbc006cab0 2024-03-06T07:24:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.921+0000 7fdbc50bb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdbc006cab0 0x7fdbc00b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.921+0000 7fdbbffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdbc00cea50 0x7fdbc00ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42888/0 (socket says 172.21.15.59:42888) 2024-03-06T07:24:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.921+0000 7fdbbffff640 1 -- 172.21.15.59:0/766904399 learned_addr learned my addr 172.21.15.59:0/766904399 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.921+0000 7fdbc50bb640 1 -- 172.21.15.59:0/766904399 >> v1:172.21.15.164:6789/0 conn(0x7fdbc015def0 legacy=0x7fdbc00ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.921+0000 7fdbbf7fe640 1 -- 172.21.15.59:0/766904399 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934670837 0 0) 0x7fdbc006d030 con 0x7fdbc00cea50 2024-03-06T07:24:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.921+0000 7fdbc50bb640 1 -- 172.21.15.59:0/766904399 >> v1:172.21.15.59:6789/0 conn(0x7fdbc00cea50 legacy=0x7fdbc00ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.921+0000 7fdbc50bb640 1 -- 172.21.15.59:0/766904399 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00cc110 con 0x7fdbc006cab0 2024-03-06T07:24:33.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.922+0000 7fdbc50bb640 1 --2- 172.21.15.59:0/766904399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc006cab0 0x7fdbc00b41b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009d90 tx=0x7fdbb802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35292963ee14dcbc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.922+0000 7fdbbf7fe640 1 -- 172.21.15.59:0/766904399 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8039070 con 0x7fdbc006cab0 2024-03-06T07:24:33.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.922+0000 7fdbbf7fe640 1 -- 172.21.15.59:0/766904399 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb802cd00 con 0x7fdbc006cab0 2024-03-06T07:24:33.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.922+0000 7fdbbf7fe640 1 -- 172.21.15.59:0/766904399 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8034660 con 0x7fdbc006cab0 2024-03-06T07:24:33.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.922+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/766904399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc006cab0 msgr2=0x7fdbc00b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.922+0000 7fdbc6b1e640 1 --2- 172.21.15.59:0/766904399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc006cab0 0x7fdbc00b41b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009d90 tx=0x7fdbb802bf50 comp rx=0 tx=0).stop 2024-03-06T07:24:33.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.922+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/766904399 shutdown_connections 2024-03-06T07:24:33.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.922+0000 7fdbc6b1e640 1 --2- 172.21.15.59:0/766904399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc006cab0 0x7fdbc00b41b0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.922+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/766904399 >> 172.21.15.59:0/766904399 conn(0x7fdbc00b3690 msgr2=0x7fdbc00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.922+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/766904399 shutdown_connections 2024-03-06T07:24:33.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.922+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/766904399 wait complete. 2024-03-06T07:24:33.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.924+0000 7fdbc6b1e640 1 Processor -- start 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.924+0000 7fdbc6b1e640 1 -- start start 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.924+0000 7fdbc6b1e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc006cab0 0x7fdbc015c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.924+0000 7fdbc6b1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc00cea50 0x7fdbc00c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.924+0000 7fdbc6b1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc015def0 0x7fdbc00d12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.924+0000 7fdbc6b1e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc015e2d0 con 0x7fdbc015def0 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.924+0000 7fdbc6b1e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00cc760 con 0x7fdbc006cab0 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.924+0000 7fdbc6b1e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc015e450 con 0x7fdbc00cea50 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc50bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc015def0 0x7fdbc00d12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc50bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc015def0 0x7fdbc00d12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39036/0 (socket says 172.21.15.59:39036) 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc50bb640 1 -- 172.21.15.59:0/3599857184 learned_addr learned my addr 172.21.15.59:0/3599857184 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbbffff640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc00cea50 0x7fdbc00c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc48ba640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc006cab0 0x7fdbc015c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc50bb640 1 -- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc00cea50 msgr2=0x7fdbc00c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc50bb640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc00cea50 0x7fdbc00c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc50bb640 1 -- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc006cab0 msgr2=0x7fdbc015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc50bb640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc006cab0 0x7fdbc015c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc50bb640 1 -- 172.21.15.59:0/3599857184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc01f4fc0 con 0x7fdbc015def0 2024-03-06T07:24:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbbffff640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc00cea50 0x7fdbc00c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc48ba640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc006cab0 0x7fdbc015c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc50bb640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc015def0 0x7fdbc00d12d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8037040 tx=0x7fdbb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbbd7fa640 1 -- 172.21.15.59:0/3599857184 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8039050 con 0x7fdbc015def0 2024-03-06T07:24:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbbd7fa640 1 -- 172.21.15.59:0/3599857184 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb803a040 con 0x7fdbc015def0 2024-03-06T07:24:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/3599857184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc01f4200 con 0x7fdbc015def0 2024-03-06T07:24:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbbd7fa640 1 -- 172.21.15.59:0/3599857184 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb803e610 con 0x7fdbc015def0 2024-03-06T07:24:33.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.925+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/3599857184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc01f47c0 con 0x7fdbc015def0 2024-03-06T07:24:33.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.926+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/3599857184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc006c300 con 0x7fdbc015def0 2024-03-06T07:24:33.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.927+0000 7fdbbd7fa640 1 -- 172.21.15.59:0/3599857184 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbb8047020 con 0x7fdbc015def0 2024-03-06T07:24:33.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.927+0000 7fdbbd7fa640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdba406b4d0 0x7fdba406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:33.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.927+0000 7fdbbd7fa640 1 -- 172.21.15.59:0/3599857184 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbb80ac5d0 con 0x7fdbc015def0 2024-03-06T07:24:33.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.927+0000 7fdbc48ba640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdba406b4d0 0x7fdba406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:33.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.928+0000 7fdbc48ba640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdba406b4d0 0x7fdba406d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00d1e20 tx=0x7fdbb4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:33.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:33.931+0000 7fdbbd7fa640 1 -- 172.21.15.59:0/3599857184 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbc006c300 con 0x7fdbc015def0 2024-03-06T07:24:34.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.112+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/3599857184 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdbc009fcf0 con 0x7fdbc015def0 2024-03-06T07:24:34.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.113+0000 7fdbbd7fa640 1 -- 172.21.15.59:0/3599857184 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdbb804e020 con 0x7fdbc015def0 2024-03-06T07:24:34.115 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:24:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdba406b4d0 msgr2=0x7fdba406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdba406b4d0 0x7fdba406d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00d1e20 tx=0x7fdbb4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/3599857184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc015def0 msgr2=0x7fdbc00d12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc015def0 0x7fdbc00d12d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8037040 tx=0x7fdbb8004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/3599857184 shutdown_connections 2024-03-06T07:24:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdba406b4d0 0x7fdba406d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc015def0 0x7fdbc00d12d0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc00cea50 0x7fdbc00c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 --2- 172.21.15.59:0/3599857184 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc006cab0 0x7fdbc015c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/3599857184 >> 172.21.15.59:0/3599857184 conn(0x7fdbc00b3690 msgr2=0x7fdbc00d0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/3599857184 shutdown_connections 2024-03-06T07:24:34.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.115+0000 7fdbc6b1e640 1 -- 172.21.15.59:0/3599857184 wait complete. 2024-03-06T07:24:34.127 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:34.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.229+0000 7f4f1b24e640 1 Processor -- start 2024-03-06T07:24:34.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.229+0000 7f4f1b24e640 1 -- start start 2024-03-06T07:24:34.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.230+0000 7f4f1b24e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4f140d10c0 0x7f4f140d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.230+0000 7f4f1b24e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f140cfa50 con 0x7f4f1415ecd0 2024-03-06T07:24:34.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.230+0000 7f4f1b24e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f140d0a30 con 0x7f4f140d14a0 2024-03-06T07:24:34.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.230+0000 7f4f1b24e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1415c050 con 0x7f4f140d10c0 2024-03-06T07:24:34.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.230+0000 7f4f197eb640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4f1415ecd0 0x7f4f140cae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42904/0 (socket says 172.21.15.59:42904) 2024-03-06T07:24:34.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.230+0000 7f4f197eb640 1 -- 172.21.15.59:0/1017671792 learned_addr learned my addr 172.21.15.59:0/1017671792 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:34.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.230+0000 7f4f18fea640 1 --2- 172.21.15.59:0/1017671792 >> v2:172.21.15.59:3300/0 conn(0x7f4f140d10c0 0x7f4f140d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.230+0000 7f4f18fea640 1 -- 172.21.15.59:0/1017671792 >> v1:172.21.15.164:6789/0 conn(0x7f4f140d14a0 legacy=0x7f4f1415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.230+0000 7f4f18fea640 1 -- 172.21.15.59:0/1017671792 >> v1:172.21.15.59:6789/0 conn(0x7f4f1415ecd0 legacy=0x7f4f140cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.230+0000 7f4f137fe640 1 -- 172.21.15.59:0/1017671792 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660884404 0 0) 0x7f4f140cfa50 con 0x7f4f1415ecd0 2024-03-06T07:24:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.230+0000 7f4f18fea640 1 -- 172.21.15.59:0/1017671792 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f140b45c0 con 0x7f4f140d10c0 2024-03-06T07:24:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.231+0000 7f4f18fea640 1 --2- 172.21.15.59:0/1017671792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f140d10c0 0x7f4f140d04f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4f080097a0 tx=0x7f4f0802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218aa9dea7029b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.231+0000 7f4f137fe640 1 -- 172.21.15.59:0/1017671792 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08034070 con 0x7f4f140d10c0 2024-03-06T07:24:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.231+0000 7f4f137fe640 1 -- 172.21.15.59:0/1017671792 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f08030610 con 0x7f4f140d10c0 2024-03-06T07:24:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.231+0000 7f4f137fe640 1 -- 172.21.15.59:0/1017671792 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f080309e0 con 0x7f4f140d10c0 2024-03-06T07:24:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.232+0000 7f4f1b24e640 1 -- 172.21.15.59:0/1017671792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f140d10c0 msgr2=0x7f4f140d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.232+0000 7f4f1b24e640 1 --2- 172.21.15.59:0/1017671792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f140d10c0 0x7f4f140d04f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4f080097a0 tx=0x7f4f0802b960 comp rx=0 tx=0).stop 2024-03-06T07:24:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.232+0000 7f4f1b24e640 1 -- 172.21.15.59:0/1017671792 shutdown_connections 2024-03-06T07:24:34.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.232+0000 7f4f1b24e640 1 --2- 172.21.15.59:0/1017671792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f140d10c0 0x7f4f140d04f0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.232+0000 7f4f1b24e640 1 -- 172.21.15.59:0/1017671792 >> 172.21.15.59:0/1017671792 conn(0x7f4f140b3800 msgr2=0x7f4f140b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.232+0000 7f4f1b24e640 1 -- 172.21.15.59:0/1017671792 shutdown_connections 2024-03-06T07:24:34.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.232+0000 7f4f1b24e640 1 -- 172.21.15.59:0/1017671792 wait complete. 2024-03-06T07:24:34.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.233+0000 7f4f1b24e640 1 Processor -- start 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.233+0000 7f4f1b24e640 1 -- start start 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f1b24e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f140d10c0 0x7f4f140b76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f1b24e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f140d14a0 0x7f4f140b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f1b24e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f14146cd0 0x7f4f141f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f1b24e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1415f440 con 0x7f4f140d10c0 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f1b24e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f140d0060 con 0x7f4f14146cd0 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f1b24e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f140cfcf0 con 0x7f4f140d14a0 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f18fea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f140d10c0 0x7f4f140b76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f18fea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f140d10c0 0x7f4f140b76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39058/0 (socket says 172.21.15.59:39058) 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f18fea640 1 -- 172.21.15.59:0/2189420299 learned_addr learned my addr 172.21.15.59:0/2189420299 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f18fea640 1 -- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f140d14a0 msgr2=0x7f4f140b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f18fea640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f140d14a0 0x7f4f140b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f18fea640 1 -- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f14146cd0 msgr2=0x7f4f141f73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f18fea640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f14146cd0 0x7f4f141f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.234+0000 7f4f18fea640 1 -- 172.21.15.59:0/2189420299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f140b6900 con 0x7f4f140d10c0 2024-03-06T07:24:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.235+0000 7f4f18fea640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f140d10c0 0x7f4f140b76c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4f08000c00 tx=0x7f4f080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.235+0000 7f4f117fa640 1 -- 172.21.15.59:0/2189420299 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08047070 con 0x7f4f140d10c0 2024-03-06T07:24:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.235+0000 7f4f117fa640 1 -- 172.21.15.59:0/2189420299 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f08004030 con 0x7f4f140d10c0 2024-03-06T07:24:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.235+0000 7f4f117fa640 1 -- 172.21.15.59:0/2189420299 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08034040 con 0x7f4f140d10c0 2024-03-06T07:24:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.235+0000 7f4f1b24e640 1 -- 172.21.15.59:0/2189420299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f140b6be0 con 0x7f4f140d10c0 2024-03-06T07:24:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.235+0000 7f4f1b24e640 1 -- 172.21.15.59:0/2189420299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f140b65d0 con 0x7f4f140d10c0 2024-03-06T07:24:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.236+0000 7f4f1b24e640 1 -- 172.21.15.59:0/2189420299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1409fe60 con 0x7f4f140d10c0 2024-03-06T07:24:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.237+0000 7f4f117fa640 1 -- 172.21.15.59:0/2189420299 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f08037070 con 0x7f4f140d10c0 2024-03-06T07:24:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.237+0000 7f4f117fa640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ef806b4d0 0x7f4ef806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.237+0000 7f4f117fa640 1 -- 172.21.15.59:0/2189420299 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f080ab8b0 con 0x7f4f140d10c0 2024-03-06T07:24:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.237+0000 7f4f13fff640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ef806b4d0 0x7f4ef806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.238+0000 7f4f13fff640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ef806b4d0 0x7f4ef806d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4f00000f60 tx=0x7f4f000079b0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.241+0000 7f4f117fa640 1 -- 172.21.15.59:0/2189420299 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f0807a7c0 con 0x7f4f140d10c0 2024-03-06T07:24:34.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.594+0000 7f4f1b24e640 1 -- 172.21.15.59:0/2189420299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4f14068ff0 con 0x7f4f140d10c0 2024-03-06T07:24:34.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.595+0000 7f4f117fa640 1 -- 172.21.15.59:0/2189420299 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4f0807a0c0 con 0x7f4f140d10c0 2024-03-06T07:24:34.701 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:24:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 -- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ef806b4d0 msgr2=0x7f4ef806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ef806b4d0 0x7f4ef806d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4f00000f60 tx=0x7f4f000079b0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 -- 172.21.15.59:0/2189420299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f140d10c0 msgr2=0x7f4f140b76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f140d10c0 0x7f4f140b76c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4f08000c00 tx=0x7f4f080046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 -- 172.21.15.59:0/2189420299 shutdown_connections 2024-03-06T07:24:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ef806b4d0 0x7f4ef806d990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f14146cd0 0x7f4f141f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f140d14a0 0x7f4f140b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 --2- 172.21.15.59:0/2189420299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f140d10c0 0x7f4f140b76c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 -- 172.21.15.59:0/2189420299 >> 172.21.15.59:0/2189420299 conn(0x7f4f140b3800 msgr2=0x7f4f140cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 -- 172.21.15.59:0/2189420299 shutdown_connections 2024-03-06T07:24:34.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.597+0000 7f4f1b24e640 1 -- 172.21.15.59:0/2189420299 wait complete. 2024-03-06T07:24:34.703 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:34.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.806+0000 7f252af77640 1 Processor -- start 2024-03-06T07:24:34.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.806+0000 7f252af77640 1 -- start start 2024-03-06T07:24:34.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f252af77640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f25241614d0 0x7f25240d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f252af77640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f25240d1910 0x7f25240d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f252af77640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25241618b0 con 0x7f25240d22e0 2024-03-06T07:24:34.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f252af77640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25240cf400 con 0x7f25240d1910 2024-03-06T07:24:34.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f252af77640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25240d2710 con 0x7f25241614d0 2024-03-06T07:24:34.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f2523fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f25240d1910 0x7f25240d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f2523fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f25240d1910 0x7f25240d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39074/0 (socket says 172.21.15.59:39074) 2024-03-06T07:24:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f2523fff640 1 -- 172.21.15.59:0/592974924 learned_addr learned my addr 172.21.15.59:0/592974924 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f2528d13640 1 --2- 172.21.15.59:0/592974924 >> v2:172.21.15.164:3300/0 conn(0x7f25241614d0 0x7f25240d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f2523fff640 1 -- 172.21.15.59:0/592974924 >> v2:172.21.15.164:3300/0 conn(0x7f25241614d0 msgr2=0x7f25240d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f2523fff640 1 --2- 172.21.15.59:0/592974924 >> v2:172.21.15.164:3300/0 conn(0x7f25241614d0 0x7f25240d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f2523fff640 1 -- 172.21.15.59:0/592974924 >> v1:172.21.15.164:6789/0 conn(0x7f25240d22e0 legacy=0x7f25240d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.807+0000 7f2523fff640 1 -- 172.21.15.59:0/592974924 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25240cf6a0 con 0x7f25240d1910 2024-03-06T07:24:34.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.808+0000 7f2523fff640 1 --2- 172.21.15.59:0/592974924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25240d1910 0x7f25240d1da0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f252408c470 tx=0x7f251002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be779894c1a92aa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.808+0000 7f25237fe640 1 -- 172.21.15.59:0/592974924 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2510034070 con 0x7f25240d1910 2024-03-06T07:24:34.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.808+0000 7f25237fe640 1 -- 172.21.15.59:0/592974924 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2510030860 con 0x7f25240d1910 2024-03-06T07:24:34.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.808+0000 7f25237fe640 1 -- 172.21.15.59:0/592974924 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2510030c30 con 0x7f25240d1910 2024-03-06T07:24:34.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.809+0000 7f252af77640 1 -- 172.21.15.59:0/592974924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25240d1910 msgr2=0x7f25240d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.809+0000 7f252af77640 1 --2- 172.21.15.59:0/592974924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25240d1910 0x7f25240d1da0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f252408c470 tx=0x7f251002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.809+0000 7f252af77640 1 -- 172.21.15.59:0/592974924 shutdown_connections 2024-03-06T07:24:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.809+0000 7f252af77640 1 --2- 172.21.15.59:0/592974924 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25240d1910 0x7f25240d1da0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.809+0000 7f252af77640 1 --2- 172.21.15.59:0/592974924 >> v2:172.21.15.164:3300/0 conn(0x7f25241614d0 0x7f25240d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.809+0000 7f252af77640 1 -- 172.21.15.59:0/592974924 >> 172.21.15.59:0/592974924 conn(0x7f25240d3d40 msgr2=0x7f25240d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.809+0000 7f252af77640 1 -- 172.21.15.59:0/592974924 shutdown_connections 2024-03-06T07:24:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.809+0000 7f252af77640 1 -- 172.21.15.59:0/592974924 wait complete. 2024-03-06T07:24:34.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.810+0000 7f252af77640 1 Processor -- start 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.810+0000 7f252af77640 1 -- start start 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f252af77640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25240d1910 0x7f25240cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f252af77640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25240d22e0 0x7f252414f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f252af77640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25241614d0 0x7f25240b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f252af77640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25240b0910 con 0x7f25240d22e0 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f252af77640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25240cfc60 con 0x7f25241614d0 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f252af77640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25240b1e40 con 0x7f25240d1910 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f2523fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25240d22e0 0x7f252414f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f2523fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25240d22e0 0x7f252414f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39090/0 (socket says 172.21.15.59:39090) 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f2523fff640 1 -- 172.21.15.59:0/3607184454 learned_addr learned my addr 172.21.15.59:0/3607184454 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f2528d13640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25240d1910 0x7f25240cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f2529514640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25241614d0 0x7f25240b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f2523fff640 1 -- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25240d1910 msgr2=0x7f25240cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f2523fff640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25240d1910 0x7f25240cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f2523fff640 1 -- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25241614d0 msgr2=0x7f25240b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f2523fff640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25241614d0 0x7f25240b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f2523fff640 1 -- 172.21.15.59:0/3607184454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25240b63e0 con 0x7f25240d22e0 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f2523fff640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25240d22e0 0x7f252414f290 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f2510000c00 tx=0x7f2510004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.811+0000 7f25217fa640 1 -- 172.21.15.59:0/3607184454 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2510048070 con 0x7f25240d22e0 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.812+0000 7f25217fa640 1 -- 172.21.15.59:0/3607184454 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2510030660 con 0x7f25240d22e0 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.812+0000 7f252af77640 1 -- 172.21.15.59:0/3607184454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25240b5340 con 0x7f25240d22e0 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.812+0000 7f25217fa640 1 -- 172.21.15.59:0/3607184454 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2510034040 con 0x7f25240d22e0 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.812+0000 7f252af77640 1 -- 172.21.15.59:0/3607184454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25240b5900 con 0x7f25240d22e0 2024-03-06T07:24:34.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.812+0000 7f252af77640 1 -- 172.21.15.59:0/3607184454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2524068ff0 con 0x7f25240d22e0 2024-03-06T07:24:34.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.814+0000 7f25217fa640 1 -- 172.21.15.59:0/3607184454 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f251004b070 con 0x7f25240d22e0 2024-03-06T07:24:34.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.814+0000 7f25217fa640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f250806b500 0x7f250806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:34.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.814+0000 7f25217fa640 1 -- 172.21.15.59:0/3607184454 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25100ac400 con 0x7f25240d22e0 2024-03-06T07:24:34.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.814+0000 7f2528d13640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f250806b500 0x7f250806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:34.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.815+0000 7f2528d13640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f250806b500 0x7f250806d9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f25180079a0 tx=0x7f2518008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:34.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:34.818+0000 7f25217fa640 1 -- 172.21.15.59:0/3607184454 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f251007acb0 con 0x7f25240d22e0 2024-03-06T07:24:35.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.001+0000 7f252af77640 1 -- 172.21.15.59:0/3607184454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f25240caab0 con 0x7f25240d22e0 2024-03-06T07:24:35.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.001+0000 7f25217fa640 1 -- 172.21.15.59:0/3607184454 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7f251007ed50 con 0x7f25240d22e0 2024-03-06T07:24:35.003 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:24:35.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.003+0000 7f252af77640 1 -- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f250806b500 msgr2=0x7f250806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.003+0000 7f252af77640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f250806b500 0x7f250806d9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f25180079a0 tx=0x7f2518008040 comp rx=0 tx=0).stop 2024-03-06T07:24:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.003+0000 7f252af77640 1 -- 172.21.15.59:0/3607184454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25240d22e0 msgr2=0x7f252414f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.003+0000 7f252af77640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25240d22e0 0x7f252414f290 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f2510000c00 tx=0x7f2510004730 comp rx=0 tx=0).stop 2024-03-06T07:24:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.003+0000 7f252af77640 1 -- 172.21.15.59:0/3607184454 shutdown_connections 2024-03-06T07:24:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.003+0000 7f252af77640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f250806b500 0x7f250806d9c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.003+0000 7f252af77640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25241614d0 0x7f25240b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.003+0000 7f252af77640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25240d22e0 0x7f252414f290 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.003+0000 7f252af77640 1 --2- 172.21.15.59:0/3607184454 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25240d1910 0x7f25240cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.003+0000 7f252af77640 1 -- 172.21.15.59:0/3607184454 >> 172.21.15.59:0/3607184454 conn(0x7f25240d3d40 msgr2=0x7f25240cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.004+0000 7f252af77640 1 -- 172.21.15.59:0/3607184454 shutdown_connections 2024-03-06T07:24:35.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.004+0000 7f252af77640 1 -- 172.21.15.59:0/3607184454 wait complete. 2024-03-06T07:24:35.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.120+0000 7fc169655640 1 Processor -- start 2024-03-06T07:24:35.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.120+0000 7fc169655640 1 -- start start 2024-03-06T07:24:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.120+0000 7fc169655640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc1640c90f0 0x7fc1640ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.120+0000 7fc169655640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1640d25e0 con 0x7fc1640d0d90 2024-03-06T07:24:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.120+0000 7fc169655640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1640d2880 con 0x7fc16415e000 2024-03-06T07:24:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.120+0000 7fc169655640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1640c7c50 con 0x7fc1640c90f0 2024-03-06T07:24:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.121+0000 7fc1637fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc1640c90f0 0x7fc1640ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.121+0000 7fc1627fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc1640d0d90 0x7fc1640ce7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42910/0 (socket says 172.21.15.59:42910) 2024-03-06T07:24:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.121+0000 7fc1627fc640 1 -- 172.21.15.59:0/885521885 learned_addr learned my addr 172.21.15.59:0/885521885 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.121+0000 7fc1637fe640 1 -- 172.21.15.59:0/885521885 >> v1:172.21.15.164:6789/0 conn(0x7fc16415e000 legacy=0x7fc1640d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.121+0000 7fc1637fe640 1 -- 172.21.15.59:0/885521885 >> v1:172.21.15.59:6789/0 conn(0x7fc1640d0d90 legacy=0x7fc1640ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.121+0000 7fc161ffb640 1 -- 172.21.15.59:0/885521885 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869950973 0 0) 0x7fc1640d25e0 con 0x7fc1640d0d90 2024-03-06T07:24:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.121+0000 7fc1637fe640 1 -- 172.21.15.59:0/885521885 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1640d2b40 con 0x7fc1640c90f0 2024-03-06T07:24:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.122+0000 7fc1637fe640 1 --2- 172.21.15.59:0/885521885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640c90f0 0x7fc1640ceed0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0097a0 tx=0x7fc15c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6637efbd2bdab4ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.122+0000 7fc161ffb640 1 -- 172.21.15.59:0/885521885 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c02f810 con 0x7fc1640c90f0 2024-03-06T07:24:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.122+0000 7fc161ffb640 1 -- 172.21.15.59:0/885521885 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c02f9b0 con 0x7fc1640c90f0 2024-03-06T07:24:35.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.122+0000 7fc161ffb640 1 -- 172.21.15.59:0/885521885 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c02fd80 con 0x7fc1640c90f0 2024-03-06T07:24:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.122+0000 7fc169655640 1 -- 172.21.15.59:0/885521885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640c90f0 msgr2=0x7fc1640ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.122+0000 7fc169655640 1 --2- 172.21.15.59:0/885521885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640c90f0 0x7fc1640ceed0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0097a0 tx=0x7fc15c02b9f0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.122+0000 7fc169655640 1 -- 172.21.15.59:0/885521885 shutdown_connections 2024-03-06T07:24:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.122+0000 7fc169655640 1 --2- 172.21.15.59:0/885521885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640c90f0 0x7fc1640ceed0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.122+0000 7fc169655640 1 -- 172.21.15.59:0/885521885 >> 172.21.15.59:0/885521885 conn(0x7fc1640d3cf0 msgr2=0x7fc1640d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.122+0000 7fc169655640 1 -- 172.21.15.59:0/885521885 shutdown_connections 2024-03-06T07:24:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.122+0000 7fc169655640 1 -- 172.21.15.59:0/885521885 wait complete. 2024-03-06T07:24:35.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.124+0000 7fc169655640 1 Processor -- start 2024-03-06T07:24:35.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.124+0000 7fc169655640 1 -- start start 2024-03-06T07:24:35.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.124+0000 7fc169655640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640c90f0 0x7fc1640cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.124+0000 7fc169655640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1640d0d90 0x7fc1640a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.124+0000 7fc169655640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc16415e000 0x7fc1641f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.124+0000 7fc169655640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16415e3e0 con 0x7fc1640c90f0 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.124+0000 7fc169655640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640d22d0 con 0x7fc16415e000 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.124+0000 7fc169655640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640d4810 con 0x7fc1640d0d90 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.124+0000 7fc162ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640c90f0 0x7fc1640cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc162ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640c90f0 0x7fc1640cd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39106/0 (socket says 172.21.15.59:39106) 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc162ffd640 1 -- 172.21.15.59:0/1010547485 learned_addr learned my addr 172.21.15.59:0/1010547485 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc1627fc640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1640d0d90 0x7fc1640a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc1637fe640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc16415e000 0x7fc1641f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc162ffd640 1 -- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1640d0d90 msgr2=0x7fc1640a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc162ffd640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1640d0d90 0x7fc1640a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc162ffd640 1 -- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc16415e000 msgr2=0x7fc1641f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc162ffd640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc16415e000 0x7fc1641f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc162ffd640 1 -- 172.21.15.59:0/1010547485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1641f4930 con 0x7fc1640c90f0 2024-03-06T07:24:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc162ffd640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640c90f0 0x7fc1640cd200 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc15400c730 tx=0x7fc15400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc14b7fe640 1 -- 172.21.15.59:0/1010547485 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154003b70 con 0x7fc1640c90f0 2024-03-06T07:24:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc14b7fe640 1 -- 172.21.15.59:0/1010547485 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc154003d30 con 0x7fc1640c90f0 2024-03-06T07:24:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc169655640 1 -- 172.21.15.59:0/1010547485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1641f4c10 con 0x7fc1640c90f0 2024-03-06T07:24:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc14b7fe640 1 -- 172.21.15.59:0/1010547485 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15400aa90 con 0x7fc1640c90f0 2024-03-06T07:24:35.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.125+0000 7fc169655640 1 -- 172.21.15.59:0/1010547485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1641f54b0 con 0x7fc1640c90f0 2024-03-06T07:24:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.126+0000 7fc169655640 1 -- 172.21.15.59:0/1010547485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16404f330 con 0x7fc1640c90f0 2024-03-06T07:24:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.127+0000 7fc14b7fe640 1 -- 172.21.15.59:0/1010547485 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc15400ac30 con 0x7fc1640c90f0 2024-03-06T07:24:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.127+0000 7fc14b7fe640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc15006b4d0 0x7fc15006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.128+0000 7fc14b7fe640 1 -- 172.21.15.59:0/1010547485 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc154089fd0 con 0x7fc1640c90f0 2024-03-06T07:24:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.128+0000 7fc1627fc640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc15006b4d0 0x7fc15006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.128+0000 7fc1627fc640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc15006b4d0 0x7fc15006d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc158007f70 tx=0x7fc158007f00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.132+0000 7fc14b7fe640 1 -- 172.21.15.59:0/1010547485 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc154058800 con 0x7fc1640c90f0 2024-03-06T07:24:35.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.324+0000 7fc169655640 1 -- 172.21.15.59:0/1010547485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc1641f5680 con 0x7fc1640c90f0 2024-03-06T07:24:35.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.325+0000 7fc14b7fe640 1 -- 172.21.15.59:0/1010547485 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v59) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc15405c8a0 con 0x7fc1640c90f0 2024-03-06T07:24:35.327 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.326+0000 7fc169655640 1 -- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc15006b4d0 msgr2=0x7fc15006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.327+0000 7fc169655640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc15006b4d0 0x7fc15006d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc158007f70 tx=0x7fc158007f00 comp rx=0 tx=0).stop 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.327+0000 7fc169655640 1 -- 172.21.15.59:0/1010547485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640c90f0 msgr2=0x7fc1640cd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.327+0000 7fc169655640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640c90f0 0x7fc1640cd200 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc15400c730 tx=0x7fc15400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.327+0000 7fc169655640 1 -- 172.21.15.59:0/1010547485 shutdown_connections 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.327+0000 7fc169655640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc15006b4d0 0x7fc15006d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.327+0000 7fc169655640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc16415e000 0x7fc1641f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.327+0000 7fc169655640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1640d0d90 0x7fc1640a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.327+0000 7fc169655640 1 --2- 172.21.15.59:0/1010547485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1640c90f0 0x7fc1640cd200 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.327+0000 7fc169655640 1 -- 172.21.15.59:0/1010547485 >> 172.21.15.59:0/1010547485 conn(0x7fc1640d3cf0 msgr2=0x7fc16406cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.327+0000 7fc169655640 1 -- 172.21.15.59:0/1010547485 shutdown_connections 2024-03-06T07:24:35.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.327+0000 7fc169655640 1 -- 172.21.15.59:0/1010547485 wait complete. 2024-03-06T07:24:35.339 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:24:35.340 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:35.354 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.494+0000 7f6da260e640 1 Processor -- start 2024-03-06T07:24:35.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.494+0000 7f6da260e640 1 -- start start 2024-03-06T07:24:35.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6da260e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6d9c15dfe0 0x7f6d9c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6da260e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d9c0d29c0 0x7f6d9c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6da260e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0b3430 con 0x7f6d9c0cf830 2024-03-06T07:24:35.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6da260e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d9c0b31e0 con 0x7f6d9c15dfe0 2024-03-06T07:24:35.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6da260e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0b3d60 con 0x7f6d9c0d29c0 2024-03-06T07:24:35.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6d9bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6d9c15dfe0 0x7f6d9c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6d9b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6d9c0cf830 0x7f6d9c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42926/0 (socket says 172.21.15.59:42926) 2024-03-06T07:24:35.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6d9b7fe640 1 -- 172.21.15.59:0/3398919841 learned_addr learned my addr 172.21.15.59:0/3398919841 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:35.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6d9affd640 1 -- 172.21.15.59:0/3398919841 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889358783 0 0) 0x7f6d9c0b3430 con 0x7f6d9c0cf830 2024-03-06T07:24:35.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6da0bab640 1 --2- 172.21.15.59:0/3398919841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d9c0d29c0 0x7f6d9c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6d9affd640 1 -- 172.21.15.59:0/3398919841 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d84003520 con 0x7f6d9c0cf830 2024-03-06T07:24:35.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6d9bfff640 1 -- 172.21.15.59:0/3398919841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d9c0d29c0 msgr2=0x7f6d9c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6d9bfff640 1 --2- 172.21.15.59:0/3398919841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d9c0d29c0 0x7f6d9c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6d9bfff640 1 -- 172.21.15.59:0/3398919841 >> v1:172.21.15.59:6789/0 conn(0x7f6d9c0cf830 legacy=0x7f6d9c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.498 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.495+0000 7f6d9bfff640 1 -- 172.21.15.59:0/3398919841 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c0b3710 con 0x7f6d9c15dfe0 2024-03-06T07:24:35.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.496+0000 7f6d9bfff640 1 --2- 172.21.15.59:0/3398919841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d9c15dfe0 0x7f6d9c0cf2f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c0097a0 tx=0x7f6d8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ba81d37a5c7c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.496+0000 7f6d9affd640 1 -- 172.21.15.59:0/3398919841 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c035020 con 0x7f6d9c15dfe0 2024-03-06T07:24:35.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.496+0000 7f6d9affd640 1 -- 172.21.15.59:0/3398919841 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8c031830 con 0x7f6d9c15dfe0 2024-03-06T07:24:35.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.496+0000 7f6d9affd640 1 -- 172.21.15.59:0/3398919841 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c031c00 con 0x7f6d9c15dfe0 2024-03-06T07:24:35.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.497+0000 7f6da260e640 1 -- 172.21.15.59:0/3398919841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d9c15dfe0 msgr2=0x7f6d9c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.497+0000 7f6da260e640 1 --2- 172.21.15.59:0/3398919841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d9c15dfe0 0x7f6d9c0cf2f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c0097a0 tx=0x7f6d8c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.497+0000 7f6da260e640 1 -- 172.21.15.59:0/3398919841 shutdown_connections 2024-03-06T07:24:35.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.497+0000 7f6da260e640 1 --2- 172.21.15.59:0/3398919841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d9c0d29c0 0x7f6d9c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.497+0000 7f6da260e640 1 --2- 172.21.15.59:0/3398919841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d9c15dfe0 0x7f6d9c0cf2f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.497+0000 7f6da260e640 1 -- 172.21.15.59:0/3398919841 >> 172.21.15.59:0/3398919841 conn(0x7f6d9c0d3cd0 msgr2=0x7f6d9c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.497+0000 7f6da260e640 1 -- 172.21.15.59:0/3398919841 shutdown_connections 2024-03-06T07:24:35.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.497+0000 7f6da260e640 1 -- 172.21.15.59:0/3398919841 wait complete. 2024-03-06T07:24:35.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.498+0000 7f6da260e640 1 Processor -- start 2024-03-06T07:24:35.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.498+0000 7f6da260e640 1 -- start start 2024-03-06T07:24:35.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6da260e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d9c0cf830 0x7f6d9c1f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6da260e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d9c0d29c0 0x7f6d9c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6da260e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d9c15dfe0 0x7f6d9c0ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6da260e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0d1df0 con 0x7f6d9c0cf830 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6da260e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0d1c70 con 0x7f6d9c0d29c0 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6da260e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c15e3c0 con 0x7f6d9c15dfe0 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d9bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d9c0cf830 0x7f6d9c1f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d9bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d9c0cf830 0x7f6d9c1f3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39124/0 (socket says 172.21.15.59:39124) 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d9bfff640 1 -- 172.21.15.59:0/38317315 learned_addr learned my addr 172.21.15.59:0/38317315 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6da0bab640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d9c15dfe0 0x7f6d9c0ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d9b7fe640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d9c0d29c0 0x7f6d9c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d9bfff640 1 -- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d9c15dfe0 msgr2=0x7f6d9c0ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d9bfff640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d9c15dfe0 0x7f6d9c0ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d9bfff640 1 -- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d9c0d29c0 msgr2=0x7f6d9c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d9bfff640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d9c0d29c0 0x7f6d9c0cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d9bfff640 1 -- 172.21.15.59:0/38317315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c0ca520 con 0x7f6d9c0cf830 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d9b7fe640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d9c0d29c0 0x7f6d9c0cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d9bfff640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d9c0cf830 0x7f6d9c1f3ed0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c034040 tx=0x7f6d8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d98ff9640 1 -- 172.21.15.59:0/38317315 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c040070 con 0x7f6d9c0cf830 2024-03-06T07:24:35.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.499+0000 7f6d98ff9640 1 -- 172.21.15.59:0/38317315 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8c03a910 con 0x7f6d9c0cf830 2024-03-06T07:24:35.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.500+0000 7f6d98ff9640 1 -- 172.21.15.59:0/38317315 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c035040 con 0x7f6d9c0cf830 2024-03-06T07:24:35.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.500+0000 7f6da260e640 1 -- 172.21.15.59:0/38317315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c0ca800 con 0x7f6d9c0cf830 2024-03-06T07:24:35.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.500+0000 7f6da260e640 1 -- 172.21.15.59:0/38317315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c15f240 con 0x7f6d9c0cf830 2024-03-06T07:24:35.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.500+0000 7f6da260e640 1 -- 172.21.15.59:0/38317315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c04f330 con 0x7f6d9c0cf830 2024-03-06T07:24:35.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.501+0000 7f6d98ff9640 1 -- 172.21.15.59:0/38317315 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d8c038020 con 0x7f6d9c0cf830 2024-03-06T07:24:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.501+0000 7f6d98ff9640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d8806b500 0x7f6d8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.501+0000 7f6d98ff9640 1 -- 172.21.15.59:0/38317315 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d8c0abab0 con 0x7f6d9c0cf830 2024-03-06T07:24:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.502+0000 7f6d9b7fe640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d8806b500 0x7f6d8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.502+0000 7f6d9b7fe640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d8806b500 0x7f6d8806d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6d90007fd0 tx=0x7f6d90009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.506+0000 7f6d98ff9640 1 -- 172.21.15.59:0/38317315 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d8c031800 con 0x7f6d9c0cf830 2024-03-06T07:24:35.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.689+0000 7f6da260e640 1 -- 172.21.15.59:0/38317315 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d9c0b9ae0 con 0x7f6d9c0cf830 2024-03-06T07:24:35.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.690+0000 7f6d98ff9640 1 -- 172.21.15.59:0/38317315 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f6d8c07a9c0 con 0x7f6d9c0cf830 2024-03-06T07:24:35.692 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:35.692 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:24:12.652576+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:09.288329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.978574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 -- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d8806b500 msgr2=0x7f6d8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d8806b500 0x7f6d8806d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6d90007fd0 tx=0x7f6d90009040 comp rx=0 tx=0).stop 2024-03-06T07:24:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 -- 172.21.15.59:0/38317315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d9c0cf830 msgr2=0x7f6d9c1f3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d9c0cf830 0x7f6d9c1f3ed0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c034040 tx=0x7f6d8c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 -- 172.21.15.59:0/38317315 shutdown_connections 2024-03-06T07:24:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6d8806b500 0x7f6d8806d9c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d9c15dfe0 0x7f6d9c0ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d9c0d29c0 0x7f6d9c0cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 --2- 172.21.15.59:0/38317315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6d9c0cf830 0x7f6d9c1f3ed0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 -- 172.21.15.59:0/38317315 >> 172.21.15.59:0/38317315 conn(0x7f6d9c0d3cd0 msgr2=0x7f6d9c0cfd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 -- 172.21.15.59:0/38317315 shutdown_connections 2024-03-06T07:24:35.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.692+0000 7f6da260e640 1 -- 172.21.15.59:0/38317315 wait complete. 2024-03-06T07:24:35.706 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:09.288329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:09.978574+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:35.706 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.808+0000 7f72962b5640 1 Processor -- start 2024-03-06T07:24:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.808+0000 7f72962b5640 1 -- start start 2024-03-06T07:24:35.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f72962b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f72900d4860 0x7f72900d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f72962b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900b0e30 0x7f72900b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f72962b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72901618b0 con 0x7f72901614d0 2024-03-06T07:24:35.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f72962b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72900d4cf0 con 0x7f72900d4860 2024-03-06T07:24:35.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f72962b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72900cf400 con 0x7f72900b0e30 2024-03-06T07:24:35.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f7294852640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900b0e30 0x7f72900b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f7294852640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900b0e30 0x7f72900b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:44548/0 (socket says 172.21.15.59:44548) 2024-03-06T07:24:35.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f7294852640 1 -- 172.21.15.59:0/2345156432 learned_addr learned my addr 172.21.15.59:0/2345156432 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:35.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f728f7fe640 1 --2- 172.21.15.59:0/2345156432 >> v2:172.21.15.164:3300/0 conn(0x7f72900d4860 0x7f72900d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f7294852640 1 -- 172.21.15.59:0/2345156432 >> v2:172.21.15.164:3300/0 conn(0x7f72900d4860 msgr2=0x7f72900d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f7294852640 1 --2- 172.21.15.59:0/2345156432 >> v2:172.21.15.164:3300/0 conn(0x7f72900d4860 0x7f72900d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f7294852640 1 -- 172.21.15.59:0/2345156432 >> v1:172.21.15.164:6789/0 conn(0x7f72901614d0 legacy=0x7f72900d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.810+0000 7f7294852640 1 -- 172.21.15.59:0/2345156432 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72900cf6a0 con 0x7f72900b0e30 2024-03-06T07:24:35.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.811+0000 7f7294852640 1 --2- 172.21.15.59:0/2345156432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900b0e30 0x7f72900b12a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7288009e50 tx=0x7f728802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c920b0dbddf71264 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.811+0000 7f728effd640 1 -- 172.21.15.59:0/2345156432 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728802cb00 con 0x7f72900b0e30 2024-03-06T07:24:35.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.811+0000 7f728effd640 1 -- 172.21.15.59:0/2345156432 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728802cca0 con 0x7f72900b0e30 2024-03-06T07:24:35.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.811+0000 7f728effd640 1 -- 172.21.15.59:0/2345156432 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288034610 con 0x7f72900b0e30 2024-03-06T07:24:35.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.812+0000 7f72962b5640 1 -- 172.21.15.59:0/2345156432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900b0e30 msgr2=0x7f72900b12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.812+0000 7f72962b5640 1 --2- 172.21.15.59:0/2345156432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900b0e30 0x7f72900b12a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7288009e50 tx=0x7f728802bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:35.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.812+0000 7f72962b5640 1 -- 172.21.15.59:0/2345156432 shutdown_connections 2024-03-06T07:24:35.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.812+0000 7f72962b5640 1 --2- 172.21.15.59:0/2345156432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900b0e30 0x7f72900b12a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.812+0000 7f72962b5640 1 --2- 172.21.15.59:0/2345156432 >> v2:172.21.15.164:3300/0 conn(0x7f72900d4860 0x7f72900d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.812+0000 7f72962b5640 1 -- 172.21.15.59:0/2345156432 >> 172.21.15.59:0/2345156432 conn(0x7f72900d3d40 msgr2=0x7f72900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.812+0000 7f72962b5640 1 -- 172.21.15.59:0/2345156432 shutdown_connections 2024-03-06T07:24:35.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.812+0000 7f72962b5640 1 -- 172.21.15.59:0/2345156432 wait complete. 2024-03-06T07:24:35.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.814+0000 7f72962b5640 1 Processor -- start 2024-03-06T07:24:35.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.814+0000 7f72962b5640 1 -- start start 2024-03-06T07:24:35.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.815+0000 7f72962b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72900b0e30 0x7f72900ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.815+0000 7f72962b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900d4860 0x7f729014a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.815+0000 7f72962b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72901614d0 0x7f72900c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.815+0000 7f72962b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72900b0910 con 0x7f72900b0e30 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.815+0000 7f72962b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72900cfc60 con 0x7f72901614d0 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.815+0000 7f72962b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72900d2560 con 0x7f72900d4860 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.815+0000 7f728ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72900b0e30 0x7f72900ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.815+0000 7f728ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72900b0e30 0x7f72900ca660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39132/0 (socket says 172.21.15.59:39132) 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.815+0000 7f728ffff640 1 -- 172.21.15.59:0/2335293955 learned_addr learned my addr 172.21.15.59:0/2335293955 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f7294852640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72901614d0 0x7f72900c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f728f7fe640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900d4860 0x7f729014a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f728ffff640 1 -- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900d4860 msgr2=0x7f729014a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f728ffff640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900d4860 0x7f729014a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f728ffff640 1 -- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72901614d0 msgr2=0x7f72900c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f728ffff640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72901614d0 0x7f72900c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f728ffff640 1 -- 172.21.15.59:0/2335293955 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72900caba0 con 0x7f72900b0e30 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f728ffff640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72900b0e30 0x7f72900ca660 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f728000c400 tx=0x7f728000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f728cff9640 1 -- 172.21.15.59:0/2335293955 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72800052d0 con 0x7f72900b0e30 2024-03-06T07:24:35.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f728cff9640 1 -- 172.21.15.59:0/2335293955 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7280005490 con 0x7f72900b0e30 2024-03-06T07:24:35.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f728cff9640 1 -- 172.21.15.59:0/2335293955 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280007260 con 0x7f72900b0e30 2024-03-06T07:24:35.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f72962b5640 1 -- 172.21.15.59:0/2335293955 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72900c98a0 con 0x7f72900b0e30 2024-03-06T07:24:35.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.816+0000 7f72962b5640 1 -- 172.21.15.59:0/2335293955 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72900c9e60 con 0x7f72900b0e30 2024-03-06T07:24:35.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.817+0000 7f72962b5640 1 -- 172.21.15.59:0/2335293955 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7290068ff0 con 0x7f72900b0e30 2024-03-06T07:24:35.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.818+0000 7f728cff9640 1 -- 172.21.15.59:0/2335293955 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7280007400 con 0x7f72900b0e30 2024-03-06T07:24:35.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.818+0000 7f728cff9640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f727c06b4d0 0x7f727c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.818+0000 7f728cff9640 1 -- 172.21.15.59:0/2335293955 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72800896c0 con 0x7f72900b0e30 2024-03-06T07:24:35.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.818+0000 7f728f7fe640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f727c06b4d0 0x7f727c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.819+0000 7f728f7fe640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f727c06b4d0 0x7f727c06d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f72900ceb90 tx=0x7f7284008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:35.822+0000 7f728cff9640 1 -- 172.21.15.59:0/2335293955 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7280058540 con 0x7f72900b0e30 2024-03-06T07:24:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.001+0000 7f72962b5640 1 -- 172.21.15.59:0/2335293955 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72900d4590 con 0x7f72900b0e30 2024-03-06T07:24:36.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.001+0000 7f728cff9640 1 -- 172.21.15.59:0/2335293955 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7280057e40 con 0x7f72900b0e30 2024-03-06T07:24:36.003 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:24:36.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 -- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f727c06b4d0 msgr2=0x7f727c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f727c06b4d0 0x7f727c06d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f72900ceb90 tx=0x7f7284008040 comp rx=0 tx=0).stop 2024-03-06T07:24:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 -- 172.21.15.59:0/2335293955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72900b0e30 msgr2=0x7f72900ca660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72900b0e30 0x7f72900ca660 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f728000c400 tx=0x7f728000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 -- 172.21.15.59:0/2335293955 shutdown_connections 2024-03-06T07:24:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f727c06b4d0 0x7f727c06d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72901614d0 0x7f72900c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72900d4860 0x7f729014a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 --2- 172.21.15.59:0/2335293955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72900b0e30 0x7f72900ca660 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 -- 172.21.15.59:0/2335293955 >> 172.21.15.59:0/2335293955 conn(0x7f72900d3d40 msgr2=0x7f729006d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 -- 172.21.15.59:0/2335293955 shutdown_connections 2024-03-06T07:24:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.003+0000 7f72962b5640 1 -- 172.21.15.59:0/2335293955 wait complete. 2024-03-06T07:24:36.016 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:36.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.118+0000 7f7db31b6640 1 Processor -- start 2024-03-06T07:24:36.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.118+0000 7f7db31b6640 1 -- start start 2024-03-06T07:24:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7db31b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dac0d4860 0x7f7dac0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7db31b6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7dac0b0e30 0x7f7dac0b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7db31b6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac1618b0 con 0x7f7dac1614d0 2024-03-06T07:24:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7db31b6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dac0d4cc0 con 0x7f7dac0b0e30 2024-03-06T07:24:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7db31b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac0cf400 con 0x7f7dac0d4860 2024-03-06T07:24:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7dabfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dac0d4860 0x7f7dac0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7db1753640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7dac0b0e30 0x7f7dac0b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7db0f52640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7dac1614d0 0x7f7dac0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:54928/0 (socket says 172.21.15.59:54928) 2024-03-06T07:24:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7db0f52640 1 -- 172.21.15.59:0/2120497213 learned_addr learned my addr 172.21.15.59:0/2120497213 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7dabfff640 1 -- 172.21.15.59:0/2120497213 >> v2:172.21.15.164:3300/0 conn(0x7f7dac0b0e30 msgr2=0x7f7dac0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7dabfff640 1 --2- 172.21.15.59:0/2120497213 >> v2:172.21.15.164:3300/0 conn(0x7f7dac0b0e30 0x7f7dac0b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7dabfff640 1 -- 172.21.15.59:0/2120497213 >> v1:172.21.15.164:6789/0 conn(0x7f7dac1614d0 legacy=0x7f7dac0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7dabfff640 1 -- 172.21.15.59:0/2120497213 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac0cf6a0 con 0x7f7dac0d4860 2024-03-06T07:24:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.119+0000 7f7db1753640 1 --2- 172.21.15.59:0/2120497213 >> v2:172.21.15.164:3300/0 conn(0x7f7dac0b0e30 0x7f7dac0b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.120+0000 7f7dabfff640 1 --2- 172.21.15.59:0/2120497213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dac0d4860 0x7f7dac0d2020 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7d980097a0 tx=0x7f7d9802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21b412fa63c41022 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.120+0000 7f7dab7fe640 1 -- 172.21.15.59:0/2120497213 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98035020 con 0x7f7dac0d4860 2024-03-06T07:24:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.120+0000 7f7dab7fe640 1 -- 172.21.15.59:0/2120497213 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d98031830 con 0x7f7dac0d4860 2024-03-06T07:24:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.120+0000 7f7dab7fe640 1 -- 172.21.15.59:0/2120497213 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98031c00 con 0x7f7dac0d4860 2024-03-06T07:24:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.121+0000 7f7db31b6640 1 -- 172.21.15.59:0/2120497213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dac0d4860 msgr2=0x7f7dac0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.121+0000 7f7db31b6640 1 --2- 172.21.15.59:0/2120497213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dac0d4860 0x7f7dac0d2020 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7d980097a0 tx=0x7f7d9802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.121+0000 7f7db31b6640 1 -- 172.21.15.59:0/2120497213 shutdown_connections 2024-03-06T07:24:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.121+0000 7f7db31b6640 1 --2- 172.21.15.59:0/2120497213 >> v2:172.21.15.164:3300/0 conn(0x7f7dac0b0e30 0x7f7dac0b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.121+0000 7f7db31b6640 1 --2- 172.21.15.59:0/2120497213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dac0d4860 0x7f7dac0d2020 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.121+0000 7f7db31b6640 1 -- 172.21.15.59:0/2120497213 >> 172.21.15.59:0/2120497213 conn(0x7f7dac0d3d40 msgr2=0x7f7dac0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.121+0000 7f7db31b6640 1 -- 172.21.15.59:0/2120497213 shutdown_connections 2024-03-06T07:24:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.121+0000 7f7db31b6640 1 -- 172.21.15.59:0/2120497213 wait complete. 2024-03-06T07:24:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.122+0000 7f7db31b6640 1 Processor -- start 2024-03-06T07:24:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.122+0000 7f7db31b6640 1 -- start start 2024-03-06T07:24:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7db31b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dac0b0e30 0x7f7dac1f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7db31b6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dac0d4860 0x7f7dac0cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7db31b6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac1614d0 0x7f7dac15ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7db31b6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac0b0910 con 0x7f7dac0d4860 2024-03-06T07:24:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7db31b6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac0cfc60 con 0x7f7dac1614d0 2024-03-06T07:24:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7db31b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac0d2560 con 0x7f7dac0b0e30 2024-03-06T07:24:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7dabfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dac0d4860 0x7f7dac0cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7dabfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dac0d4860 0x7f7dac0cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39144/0 (socket says 172.21.15.59:39144) 2024-03-06T07:24:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7dabfff640 1 -- 172.21.15.59:0/1091899165 learned_addr learned my addr 172.21.15.59:0/1091899165 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7db0f52640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dac0b0e30 0x7f7dac1f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7db1753640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac1614d0 0x7f7dac15ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7dabfff640 1 -- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dac0b0e30 msgr2=0x7f7dac1f4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7dabfff640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dac0b0e30 0x7f7dac1f4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7dabfff640 1 -- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac1614d0 msgr2=0x7f7dac15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7dabfff640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac1614d0 0x7f7dac15ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.123+0000 7f7dabfff640 1 -- 172.21.15.59:0/1091899165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac0cc8d0 con 0x7f7dac0d4860 2024-03-06T07:24:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.124+0000 7f7dabfff640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dac0d4860 0x7f7dac0cc390 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7d9802bee0 tx=0x7f7d980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.124+0000 7f7da97fa640 1 -- 172.21.15.59:0/1091899165 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98035040 con 0x7f7dac0d4860 2024-03-06T07:24:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.124+0000 7f7da97fa640 1 -- 172.21.15.59:0/1091899165 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9803b860 con 0x7f7dac0d4860 2024-03-06T07:24:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.124+0000 7f7db31b6640 1 -- 172.21.15.59:0/1091899165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dac15b6a0 con 0x7f7dac0d4860 2024-03-06T07:24:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.124+0000 7f7da97fa640 1 -- 172.21.15.59:0/1091899165 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9803bc50 con 0x7f7dac0d4860 2024-03-06T07:24:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.124+0000 7f7db31b6640 1 -- 172.21.15.59:0/1091899165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dac0cb5d0 con 0x7f7dac0d4860 2024-03-06T07:24:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.125+0000 7f7db31b6640 1 -- 172.21.15.59:0/1091899165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dac068ff0 con 0x7f7dac0d4860 2024-03-06T07:24:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.126+0000 7f7da97fa640 1 -- 172.21.15.59:0/1091899165 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d98038020 con 0x7f7dac0d4860 2024-03-06T07:24:36.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.126+0000 7f7da97fa640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 0x7f7d9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.126+0000 7f7da97fa640 1 -- 172.21.15.59:0/1091899165 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d980ac1e0 con 0x7f7dac0d4860 2024-03-06T07:24:36.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.126+0000 7f7db0f52640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 0x7f7d9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.127+0000 7f7db0f52640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 0x7f7d9006d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7da0007fd0 tx=0x7f7da0007e80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.130+0000 7f7da97fa640 1 -- 172.21.15.59:0/1091899165 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d98031800 con 0x7f7dac0d4860 2024-03-06T07:24:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.309+0000 7f7db31b6640 1 -- 172.21.15.59:0/1091899165 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7dac06d1e0 con 0x7f7dac0d4860 2024-03-06T07:24:36.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.309+0000 7f7da97fa640 1 -- 172.21.15.59:0/1091899165 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d9807b070 con 0x7f7dac0d4860 2024-03-06T07:24:36.311 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:24:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 -- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 msgr2=0x7f7d9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 0x7f7d9006d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7da0007fd0 tx=0x7f7da0007e80 comp rx=0 tx=0).stop 2024-03-06T07:24:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 -- 172.21.15.59:0/1091899165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dac0d4860 msgr2=0x7f7dac0cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dac0d4860 0x7f7dac0cc390 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7d9802bee0 tx=0x7f7d980046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 -- 172.21.15.59:0/1091899165 shutdown_connections 2024-03-06T07:24:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 0x7f7d9006d990 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac1614d0 0x7f7dac15ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dac0d4860 0x7f7dac0cc390 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 --2- 172.21.15.59:0/1091899165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dac0b0e30 0x7f7dac1f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 -- 172.21.15.59:0/1091899165 >> 172.21.15.59:0/1091899165 conn(0x7f7dac0d3d40 msgr2=0x7f7dac0b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 -- 172.21.15.59:0/1091899165 shutdown_connections 2024-03-06T07:24:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.312+0000 7f7db31b6640 1 -- 172.21.15.59:0/1091899165 wait complete. 2024-03-06T07:24:36.324 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:36.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.428+0000 7f67df688640 1 Processor -- start 2024-03-06T07:24:36.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.428+0000 7f67df688640 1 -- start start 2024-03-06T07:24:36.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.429+0000 7f67df688640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67d806c5b0 0x7f67d80b0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.429+0000 7f67df688640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f67d80b0760 0x7f67d806b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.429+0000 7f67df688640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d80b0b70 con 0x7f67d815aec0 2024-03-06T07:24:36.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.429+0000 7f67df688640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d815c520 con 0x7f67d806c5b0 2024-03-06T07:24:36.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.429+0000 7f67df688640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d80b0f70 con 0x7f67d80b0760 2024-03-06T07:24:36.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.429+0000 7f67dcc23640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67d806c5b0 0x7f67d80b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.429+0000 7f67dd424640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f67d815aec0 0x7f67d80ad720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42938/0 (socket says 172.21.15.59:42938) 2024-03-06T07:24:36.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.429+0000 7f67dd424640 1 -- 172.21.15.59:0/2245993448 learned_addr learned my addr 172.21.15.59:0/2245993448 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:36.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.430+0000 7f67dcc23640 1 -- 172.21.15.59:0/2245993448 >> v2:172.21.15.164:3300/0 conn(0x7f67d80b0760 msgr2=0x7f67d806b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.430+0000 7f67ddc25640 1 --2- 172.21.15.59:0/2245993448 >> v2:172.21.15.164:3300/0 conn(0x7f67d80b0760 0x7f67d806b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.430+0000 7f67dcc23640 1 --2- 172.21.15.59:0/2245993448 >> v2:172.21.15.164:3300/0 conn(0x7f67d80b0760 0x7f67d806b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.430+0000 7f67dcc23640 1 -- 172.21.15.59:0/2245993448 >> v1:172.21.15.59:6789/0 conn(0x7f67d815aec0 legacy=0x7f67d80ad720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.430+0000 7f67dcc23640 1 -- 172.21.15.59:0/2245993448 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d815bf60 con 0x7f67d806c5b0 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.430+0000 7f67cffff640 1 -- 172.21.15.59:0/2245993448 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422558368 0 0) 0x7f67d4001110 con 0x7f67d815aec0 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.430+0000 7f67ddc25640 1 --2- 172.21.15.59:0/2245993448 >> v2:172.21.15.164:3300/0 conn(0x7f67d80b0760 0x7f67d806b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.430+0000 7f67dcc23640 1 --2- 172.21.15.59:0/2245993448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67d806c5b0 0x7f67d80b0220 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f67d8070800 tx=0x7f67c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943616f285631d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.431+0000 7f67cffff640 1 -- 172.21.15.59:0/2245993448 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c8035020 con 0x7f67d806c5b0 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.431+0000 7f67cffff640 1 -- 172.21.15.59:0/2245993448 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c8031830 con 0x7f67d806c5b0 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.431+0000 7f67cffff640 1 -- 172.21.15.59:0/2245993448 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c8031c00 con 0x7f67d806c5b0 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.431+0000 7f67df688640 1 -- 172.21.15.59:0/2245993448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67d806c5b0 msgr2=0x7f67d80b0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.431+0000 7f67df688640 1 --2- 172.21.15.59:0/2245993448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67d806c5b0 0x7f67d80b0220 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f67d8070800 tx=0x7f67c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.431+0000 7f67df688640 1 -- 172.21.15.59:0/2245993448 shutdown_connections 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.431+0000 7f67df688640 1 --2- 172.21.15.59:0/2245993448 >> v2:172.21.15.164:3300/0 conn(0x7f67d80b0760 0x7f67d806b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.431+0000 7f67df688640 1 --2- 172.21.15.59:0/2245993448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67d806c5b0 0x7f67d80b0220 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.431+0000 7f67df688640 1 -- 172.21.15.59:0/2245993448 >> 172.21.15.59:0/2245993448 conn(0x7f67d80b4fe0 msgr2=0x7f67d80b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.431+0000 7f67df688640 1 -- 172.21.15.59:0/2245993448 shutdown_connections 2024-03-06T07:24:36.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.431+0000 7f67df688640 1 -- 172.21.15.59:0/2245993448 wait complete. 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.433+0000 7f67df688640 1 Processor -- start 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.433+0000 7f67df688640 1 -- start start 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.434+0000 7f67df688640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67d806c5b0 0x7f67d80d0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.434+0000 7f67df688640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67d80b0760 0x7f67d809a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.434+0000 7f67df688640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67d815aec0 0x7f67d809acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.434+0000 7f67df688640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d80b5db0 con 0x7f67d80b0760 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.434+0000 7f67df688640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d806b150 con 0x7f67d806c5b0 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.434+0000 7f67df688640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d80b5f30 con 0x7f67d815aec0 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.434+0000 7f67dcc23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67d80b0760 0x7f67d809a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.434+0000 7f67dcc23640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67d80b0760 0x7f67d809a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39160/0 (socket says 172.21.15.59:39160) 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.434+0000 7f67dcc23640 1 -- 172.21.15.59:0/1733925292 learned_addr learned my addr 172.21.15.59:0/1733925292 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67dd424640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67d806c5b0 0x7f67d80d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67ddc25640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67d815aec0 0x7f67d809acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67dcc23640 1 -- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67d815aec0 msgr2=0x7f67d809acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67dcc23640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67d815aec0 0x7f67d809acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67dcc23640 1 -- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67d806c5b0 msgr2=0x7f67d80d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67dcc23640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67d806c5b0 0x7f67d80d0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67dcc23640 1 -- 172.21.15.59:0/1733925292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d81e5a70 con 0x7f67d80b0760 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67dcc23640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67d80b0760 0x7f67d809a7b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f67c8009890 tx=0x7f67c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67cdffb640 1 -- 172.21.15.59:0/1733925292 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c8041070 con 0x7f67d80b0760 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67cdffb640 1 -- 172.21.15.59:0/1733925292 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c803aa30 con 0x7f67d80b0760 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67df688640 1 -- 172.21.15.59:0/1733925292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d81e5d50 con 0x7f67d80b0760 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67cdffb640 1 -- 172.21.15.59:0/1733925292 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c8035040 con 0x7f67d80b0760 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.435+0000 7f67df688640 1 -- 172.21.15.59:0/1733925292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d80d09c0 con 0x7f67d80b0760 2024-03-06T07:24:36.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.436+0000 7f67df688640 1 -- 172.21.15.59:0/1733925292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d815bba0 con 0x7f67d80b0760 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.437+0000 7f67cdffb640 1 -- 172.21.15.59:0/1733925292 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67c8038020 con 0x7f67d80b0760 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.437+0000 7f67cdffb640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67bc06b500 0x7f67bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.437+0000 7f67cdffb640 1 -- 172.21.15.59:0/1733925292 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67c80abd50 con 0x7f67d80b0760 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.437+0000 7f67dd424640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67bc06b500 0x7f67bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.438+0000 7f67dd424640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67bc06b500 0x7f67bc06d9c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f67d4000f60 tx=0x7f67d4007c00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.441+0000 7f67cdffb640 1 -- 172.21.15.59:0/1733925292 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67c80af030 con 0x7f67d80b0760 2024-03-06T07:24:36.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.622+0000 7f67df688640 1 -- 172.21.15.59:0/1733925292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f67d8068ff0 con 0x7f67d80b0760 2024-03-06T07:24:36.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.622+0000 7f67cdffb640 1 -- 172.21.15.59:0/1733925292 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7f67c807abd0 con 0x7f67d80b0760 2024-03-06T07:24:36.624 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:24:36.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 -- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67bc06b500 msgr2=0x7f67bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67bc06b500 0x7f67bc06d9c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f67d4000f60 tx=0x7f67d4007c00 comp rx=0 tx=0).stop 2024-03-06T07:24:36.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 -- 172.21.15.59:0/1733925292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67d80b0760 msgr2=0x7f67d809a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67d80b0760 0x7f67d809a7b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f67c8009890 tx=0x7f67c80046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 -- 172.21.15.59:0/1733925292 shutdown_connections 2024-03-06T07:24:36.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f67bc06b500 0x7f67bc06d9c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67d815aec0 0x7f67d809acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67d80b0760 0x7f67d809a7b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 --2- 172.21.15.59:0/1733925292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67d806c5b0 0x7f67d80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 -- 172.21.15.59:0/1733925292 >> 172.21.15.59:0/1733925292 conn(0x7f67d80b4fe0 msgr2=0x7f67d8072590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 -- 172.21.15.59:0/1733925292 shutdown_connections 2024-03-06T07:24:36.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.624+0000 7f67df688640 1 -- 172.21.15.59:0/1733925292 wait complete. 2024-03-06T07:24:36.636 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:36.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.737+0000 7ff04a6c9640 1 Processor -- start 2024-03-06T07:24:36.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.737+0000 7ff04a6c9640 1 -- start start 2024-03-06T07:24:36.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.737+0000 7ff04a6c9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff0440b4c00 0x7ff0440b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff04a6c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff04415ba50 0x7ff04415be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff04a6c9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04415d510 con 0x7ff0440b28a0 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff04a6c9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0440b3420 con 0x7ff0440b4c00 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff04a6c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04415e0b0 con 0x7ff04415ba50 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff043fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff0440b4c00 0x7ff0440b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff043fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff0440b4c00 0x7ff0440b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39164/0 (socket says 172.21.15.59:39164) 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff043fff640 1 -- 172.21.15.59:0/2626460858 learned_addr learned my addr 172.21.15.59:0/2626460858 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff048c66640 1 --2- 172.21.15.59:0/2626460858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff04415ba50 0x7ff04415be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff043fff640 1 -- 172.21.15.59:0/2626460858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff04415ba50 msgr2=0x7ff04415be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff043fff640 1 --2- 172.21.15.59:0/2626460858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff04415ba50 0x7ff04415be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff043fff640 1 -- 172.21.15.59:0/2626460858 >> v1:172.21.15.164:6789/0 conn(0x7ff0440b28a0 legacy=0x7ff0440b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.738+0000 7ff043fff640 1 -- 172.21.15.59:0/2626460858 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04415d780 con 0x7ff0440b4c00 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.739+0000 7ff043fff640 1 --2- 172.21.15.59:0/2626460858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0440b4c00 0x7ff0440b4fe0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff034009870 tx=0x7ff03402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7405e5a98e3bdec0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.739+0000 7ff042ffd640 1 -- 172.21.15.59:0/2626460858 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03402f840 con 0x7ff0440b4c00 2024-03-06T07:24:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.739+0000 7ff042ffd640 1 -- 172.21.15.59:0/2626460858 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03402f9e0 con 0x7ff0440b4c00 2024-03-06T07:24:36.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.739+0000 7ff042ffd640 1 -- 172.21.15.59:0/2626460858 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03402fdb0 con 0x7ff0440b4c00 2024-03-06T07:24:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.739+0000 7ff04a6c9640 1 -- 172.21.15.59:0/2626460858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0440b4c00 msgr2=0x7ff0440b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.739+0000 7ff04a6c9640 1 --2- 172.21.15.59:0/2626460858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0440b4c00 0x7ff0440b4fe0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff034009870 tx=0x7ff03402ba20 comp rx=0 tx=0).stop 2024-03-06T07:24:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.739+0000 7ff04a6c9640 1 -- 172.21.15.59:0/2626460858 shutdown_connections 2024-03-06T07:24:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.739+0000 7ff04a6c9640 1 --2- 172.21.15.59:0/2626460858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff04415ba50 0x7ff04415be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.739+0000 7ff04a6c9640 1 --2- 172.21.15.59:0/2626460858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0440b4c00 0x7ff0440b4fe0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.739+0000 7ff04a6c9640 1 -- 172.21.15.59:0/2626460858 >> 172.21.15.59:0/2626460858 conn(0x7ff04409a9f0 msgr2=0x7ff04409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.739+0000 7ff04a6c9640 1 -- 172.21.15.59:0/2626460858 shutdown_connections 2024-03-06T07:24:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.740+0000 7ff04a6c9640 1 -- 172.21.15.59:0/2626460858 wait complete. 2024-03-06T07:24:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.741+0000 7ff04a6c9640 1 Processor -- start 2024-03-06T07:24:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.741+0000 7ff04a6c9640 1 -- start start 2024-03-06T07:24:36.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.741+0000 7ff04a6c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0440b28a0 0x7ff04414e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.741+0000 7ff04a6c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0440b4c00 0x7ff04414f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.741+0000 7ff04a6c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff04415ba50 0x7ff0440b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.741+0000 7ff04a6c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04415f550 con 0x7ff0440b4c00 2024-03-06T07:24:36.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.741+0000 7ff04a6c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04415dd40 con 0x7ff0440b28a0 2024-03-06T07:24:36.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.741+0000 7ff04a6c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04415f6d0 con 0x7ff04415ba50 2024-03-06T07:24:36.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff0437fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0440b4c00 0x7ff04414f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff0437fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0440b4c00 0x7ff04414f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39174/0 (socket says 172.21.15.59:39174) 2024-03-06T07:24:36.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff0437fe640 1 -- 172.21.15.59:0/3494849376 learned_addr learned my addr 172.21.15.59:0/3494849376 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff043fff640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0440b28a0 0x7ff04414e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff048c66640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff04415ba50 0x7ff0440b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff0437fe640 1 -- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff04415ba50 msgr2=0x7ff0440b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff0437fe640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff04415ba50 0x7ff0440b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff0437fe640 1 -- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0440b28a0 msgr2=0x7ff04414e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff0437fe640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0440b28a0 0x7ff04414e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff0437fe640 1 -- 172.21.15.59:0/3494849376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff044152940 con 0x7ff0440b4c00 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff0437fe640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0440b4c00 0x7ff04414f5f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff03800c400 tx=0x7ff03800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff040ff9640 1 -- 172.21.15.59:0/3494849376 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0380052d0 con 0x7ff0440b4c00 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff040ff9640 1 -- 172.21.15.59:0/3494849376 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff038005490 con 0x7ff0440b4c00 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff04a6c9640 1 -- 172.21.15.59:0/3494849376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0441520a0 con 0x7ff0440b4c00 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff040ff9640 1 -- 172.21.15.59:0/3494849376 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038007260 con 0x7ff0440b4c00 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.742+0000 7ff04a6c9640 1 -- 172.21.15.59:0/3494849376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff044152660 con 0x7ff0440b4c00 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.743+0000 7ff04a6c9640 1 -- 172.21.15.59:0/3494849376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04404f330 con 0x7ff0440b4c00 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.744+0000 7ff040ff9640 1 -- 172.21.15.59:0/3494849376 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff038007400 con 0x7ff0440b4c00 2024-03-06T07:24:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.744+0000 7ff040ff9640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff03006b4d0 0x7ff03006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.744+0000 7ff040ff9640 1 -- 172.21.15.59:0/3494849376 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff038089610 con 0x7ff0440b4c00 2024-03-06T07:24:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.744+0000 7ff043fff640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff03006b4d0 0x7ff03006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.745+0000 7ff043fff640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff03006b4d0 0x7ff03006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff034036000 tx=0x7ff034008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.749+0000 7ff040ff9640 1 -- 172.21.15.59:0/3494849376 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff038057ed0 con 0x7ff0440b4c00 2024-03-06T07:24:36.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.943+0000 7ff04a6c9640 1 -- 172.21.15.59:0/3494849376 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff0440b5d60 con 0x7ff0440b4c00 2024-03-06T07:24:36.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.943+0000 7ff040ff9640 1 -- 172.21.15.59:0/3494849376 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v59) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff03805bf70 con 0x7ff0440b4c00 2024-03-06T07:24:36.945 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:24:36.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 -- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff03006b4d0 msgr2=0x7ff03006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff03006b4d0 0x7ff03006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff034036000 tx=0x7ff034008030 comp rx=0 tx=0).stop 2024-03-06T07:24:36.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 -- 172.21.15.59:0/3494849376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0440b4c00 msgr2=0x7ff04414f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0440b4c00 0x7ff04414f5f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff03800c400 tx=0x7ff03800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 -- 172.21.15.59:0/3494849376 shutdown_connections 2024-03-06T07:24:36.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff03006b4d0 0x7ff03006d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff04415ba50 0x7ff0440b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0440b4c00 0x7ff04414f5f0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 --2- 172.21.15.59:0/3494849376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0440b28a0 0x7ff04414e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 -- 172.21.15.59:0/3494849376 >> 172.21.15.59:0/3494849376 conn(0x7ff04409a9f0 msgr2=0x7ff044159040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 -- 172.21.15.59:0/3494849376 shutdown_connections 2024-03-06T07:24:36.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:36.946+0000 7ff04a6c9640 1 -- 172.21.15.59:0/3494849376 wait complete. 2024-03-06T07:24:36.958 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:24:36.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:36.959 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:36.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[58761]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:36.985 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:43.087 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:24:43.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:43.087 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:43.102 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:43.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.242+0000 7f7e1a53a640 1 Processor -- start 2024-03-06T07:24:43.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.242+0000 7f7e1a53a640 1 -- start start 2024-03-06T07:24:43.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e1a53a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e140d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e1a53a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e140b3430 con 0x7f7e140b41f0 2024-03-06T07:24:43.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e1a53a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e140b3710 con 0x7f7e1415dfe0 2024-03-06T07:24:43.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e1a53a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140d2c50 con 0x7f7e140d22b0 2024-03-06T07:24:43.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e18ad7640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7e140b41f0 0x7f7e140b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42942/0 (socket says 172.21.15.59:42942) 2024-03-06T07:24:43.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e18ad7640 1 -- 172.21.15.59:0/3273364500 learned_addr learned my addr 172.21.15.59:0/3273364500 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:43.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e137fe640 1 --2- 172.21.15.59:0/3273364500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e140d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e12ffd640 1 -- 172.21.15.59:0/3273364500 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913171351 0 0) 0x7f7e140b3430 con 0x7f7e140b41f0 2024-03-06T07:24:43.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e12ffd640 1 -- 172.21.15.59:0/3273364500 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e00003520 con 0x7f7e140b41f0 2024-03-06T07:24:43.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e12ffd640 1 -- 172.21.15.59:0/3273364500 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898271788 0 0) 0x7f7e140b3710 con 0x7f7e1415dfe0 2024-03-06T07:24:43.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e12ffd640 1 -- 172.21.15.59:0/3273364500 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e140b3430 con 0x7f7e1415dfe0 2024-03-06T07:24:43.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e137fe640 1 -- 172.21.15.59:0/3273364500 >> v1:172.21.15.164:6789/0 conn(0x7f7e1415dfe0 legacy=0x7f7e140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e137fe640 1 -- 172.21.15.59:0/3273364500 >> v1:172.21.15.59:6789/0 conn(0x7f7e140b41f0 legacy=0x7f7e140b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.243+0000 7f7e137fe640 1 -- 172.21.15.59:0/3273364500 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e140b39f0 con 0x7f7e140d22b0 2024-03-06T07:24:43.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.244+0000 7f7e137fe640 1 --2- 172.21.15.59:0/3273364500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e140d2710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7e040097d0 tx=0x7f7e0402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b5fedab605ac9fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.244+0000 7f7e12ffd640 1 -- 172.21.15.59:0/3273364500 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04034070 con 0x7f7e140d22b0 2024-03-06T07:24:43.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.244+0000 7f7e12ffd640 1 -- 172.21.15.59:0/3273364500 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e04030800 con 0x7f7e140d22b0 2024-03-06T07:24:43.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.244+0000 7f7e12ffd640 1 -- 172.21.15.59:0/3273364500 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04030bb0 con 0x7f7e140d22b0 2024-03-06T07:24:43.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.245+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3273364500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 msgr2=0x7f7e140d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.245+0000 7f7e1a53a640 1 --2- 172.21.15.59:0/3273364500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e140d2710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7e040097d0 tx=0x7f7e0402b940 comp rx=0 tx=0).stop 2024-03-06T07:24:43.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.245+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3273364500 shutdown_connections 2024-03-06T07:24:43.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.245+0000 7f7e1a53a640 1 --2- 172.21.15.59:0/3273364500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e140d2710 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.245+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3273364500 >> 172.21.15.59:0/3273364500 conn(0x7f7e140d3cd0 msgr2=0x7f7e140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.245+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3273364500 shutdown_connections 2024-03-06T07:24:43.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.245+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3273364500 wait complete. 2024-03-06T07:24:43.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.246+0000 7f7e1a53a640 1 Processor -- start 2024-03-06T07:24:43.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.246+0000 7f7e1a53a640 1 -- start start 2024-03-06T07:24:43.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e1a53a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140b41f0 0x7f7e1415ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e1a53a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e1414c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e1a53a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e1415dfe0 0x7f7e1415fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e1a53a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140b3d60 con 0x7f7e140b41f0 2024-03-06T07:24:43.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e1a53a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140b31e0 con 0x7f7e1415dfe0 2024-03-06T07:24:43.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e1a53a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140d1c70 con 0x7f7e140d22b0 2024-03-06T07:24:43.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e13fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140b41f0 0x7f7e1415ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e13fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140b41f0 0x7f7e1415ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39188/0 (socket says 172.21.15.59:39188) 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e13fff640 1 -- 172.21.15.59:0/3967748736 learned_addr learned my addr 172.21.15.59:0/3967748736 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e18ad7640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e1415dfe0 0x7f7e1415fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e137fe640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e1414c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e13fff640 1 -- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 msgr2=0x7f7e1414c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e13fff640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e1414c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e13fff640 1 -- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e1415dfe0 msgr2=0x7f7e1415fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e13fff640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e1415dfe0 0x7f7e1415fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.247+0000 7f7e13fff640 1 -- 172.21.15.59:0/3967748736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e141607d0 con 0x7f7e140b41f0 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.248+0000 7f7e13fff640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140b41f0 0x7f7e1415ef40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00c430 tx=0x7f7dfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.248+0000 7f7e10ff9640 1 -- 172.21.15.59:0/3967748736 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc017070 con 0x7f7e140b41f0 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.248+0000 7f7e10ff9640 1 -- 172.21.15.59:0/3967748736 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc0052f0 con 0x7f7e140b41f0 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.248+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3967748736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e14160ab0 con 0x7f7e140b41f0 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.248+0000 7f7e10ff9640 1 -- 172.21.15.59:0/3967748736 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc003830 con 0x7f7e140b41f0 2024-03-06T07:24:43.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.248+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3967748736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1415e740 con 0x7f7e140b41f0 2024-03-06T07:24:43.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.249+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3967748736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1404f330 con 0x7f7e140b41f0 2024-03-06T07:24:43.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.249+0000 7f7e10ff9640 1 -- 172.21.15.59:0/3967748736 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7dfc01a070 con 0x7f7e140b41f0 2024-03-06T07:24:43.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.250+0000 7f7e10ff9640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7df406b4d0 0x7f7df406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.250+0000 7f7e10ff9640 1 -- 172.21.15.59:0/3967748736 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7dfc089600 con 0x7f7e140b41f0 2024-03-06T07:24:43.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.250+0000 7f7e137fe640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7df406b4d0 0x7f7df406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.250+0000 7f7e137fe640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7df406b4d0 0x7f7df406d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7e04036000 tx=0x7f7e04008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.254+0000 7f7e10ff9640 1 -- 172.21.15.59:0/3967748736 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7dfc058480 con 0x7f7e140b41f0 2024-03-06T07:24:43.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.473+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3967748736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e140ceae0 con 0x7f7e140b41f0 2024-03-06T07:24:43.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.474+0000 7f7e10ff9640 1 -- 172.21.15.59:0/3967748736 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f7dfc057d80 con 0x7f7e140b41f0 2024-03-06T07:24:43.475 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:43.476 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2205562350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2205562350},{"type":"v1","addr":"172.21.15.164:6837","nonce":2205562350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2606750481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2606750481},{"type":"v1","addr":"172.21.15.164:6836","nonce":2606750481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6048,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3685052110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3685052110},{"type":"v1","addr":"172.21.15.59:6833","nonce":3685052110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:11.010795+0000","modified":"2024-03-06T07:24:13.053115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/750409746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":750409746},{"type":"v1","addr":"172.21.15.59:6835","nonce":750409746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":4}]} 2024-03-06T07:24:43.476 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 28 2024-03-06T07:24:43.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7df406b4d0 msgr2=0x7f7df406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7df406b4d0 0x7f7df406d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7e04036000 tx=0x7f7e04008030 comp rx=0 tx=0).stop 2024-03-06T07:24:43.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3967748736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140b41f0 msgr2=0x7f7e1415ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140b41f0 0x7f7e1415ef40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00c430 tx=0x7f7dfc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3967748736 shutdown_connections 2024-03-06T07:24:43.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7df406b4d0 0x7f7df406d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e1415dfe0 0x7f7e1415fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e140d22b0 0x7f7e1414c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 --2- 172.21.15.59:0/3967748736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e140b41f0 0x7f7e1415ef40 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3967748736 >> 172.21.15.59:0/3967748736 conn(0x7f7e140d3cd0 msgr2=0x7f7e1406ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3967748736 shutdown_connections 2024-03-06T07:24:43.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.476+0000 7f7e1a53a640 1 -- 172.21.15.59:0/3967748736 wait complete. 2024-03-06T07:24:43.489 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dirfrag split /subdir 0/0 1 2024-03-06T07:24:43.607 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-03-06T07:24:43.608 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:43.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.711+0000 7ff20d3d0640 1 Processor -- start 2024-03-06T07:24:43.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.711+0000 7ff20d3d0640 1 -- start start 2024-03-06T07:24:43.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.711+0000 7ff20d3d0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff20815bdd0 0x7ff2080c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.711+0000 7ff20d3d0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2080d16e0 0x7ff2080d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.711+0000 7ff20d3d0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2080cebe0 con 0x7ff20815fa10 2024-03-06T07:24:43.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.711+0000 7ff20d3d0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2080ce980 con 0x7ff20815bdd0 2024-03-06T07:24:43.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.711+0000 7ff20d3d0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080ceec0 con 0x7ff2080d16e0 2024-03-06T07:24:43.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.712+0000 7ff206ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff20815bdd0 0x7ff2080c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.712+0000 7ff1fffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff20815fa10 0x7ff2080ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47302/0 (socket says 172.21.15.59:47302) 2024-03-06T07:24:43.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.712+0000 7ff1fffff640 1 -- 172.21.15.59:0/4247038728 learned_addr learned my addr 172.21.15.59:0/4247038728 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:43.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.712+0000 7ff2077fe640 1 --2- 172.21.15.59:0/4247038728 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2080d16e0 0x7ff2080d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.712+0000 7ff206ffd640 1 -- 172.21.15.59:0/4247038728 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2080d16e0 msgr2=0x7ff2080d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.712+0000 7ff206ffd640 1 --2- 172.21.15.59:0/4247038728 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2080d16e0 0x7ff2080d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.712+0000 7ff206ffd640 1 -- 172.21.15.59:0/4247038728 >> v1:172.21.15.59:6789/0 conn(0x7ff20815fa10 legacy=0x7ff2080ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.712+0000 7ff206ffd640 1 -- 172.21.15.59:0/4247038728 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080cf1a0 con 0x7ff20815bdd0 2024-03-06T07:24:43.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.712+0000 7ff2067fc640 1 -- 172.21.15.59:0/4247038728 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028189664 0 0) 0x7ff2080cebe0 con 0x7ff20815fa10 2024-03-06T07:24:43.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.712+0000 7ff2077fe640 1 --2- 172.21.15.59:0/4247038728 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2080d16e0 0x7ff2080d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.713+0000 7ff206ffd640 1 --2- 172.21.15.59:0/4247038728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff20815bdd0 0x7ff2080c5610 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80097a0 tx=0x7ff1f802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a24e1804e33ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.713+0000 7ff2067fc640 1 -- 172.21.15.59:0/4247038728 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80315a0 con 0x7ff20815bdd0 2024-03-06T07:24:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.713+0000 7ff2067fc640 1 -- 172.21.15.59:0/4247038728 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f8037070 con 0x7ff20815bdd0 2024-03-06T07:24:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.713+0000 7ff2067fc640 1 -- 172.21.15.59:0/4247038728 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8031960 con 0x7ff20815bdd0 2024-03-06T07:24:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.714+0000 7ff20d3d0640 1 -- 172.21.15.59:0/4247038728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff20815bdd0 msgr2=0x7ff2080c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.714+0000 7ff20d3d0640 1 --2- 172.21.15.59:0/4247038728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff20815bdd0 0x7ff2080c5610 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80097a0 tx=0x7ff1f802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.714+0000 7ff20d3d0640 1 -- 172.21.15.59:0/4247038728 shutdown_connections 2024-03-06T07:24:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.714+0000 7ff20d3d0640 1 --2- 172.21.15.59:0/4247038728 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2080d16e0 0x7ff2080d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.714+0000 7ff20d3d0640 1 --2- 172.21.15.59:0/4247038728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff20815bdd0 0x7ff2080c5610 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.714+0000 7ff20d3d0640 1 -- 172.21.15.59:0/4247038728 >> 172.21.15.59:0/4247038728 conn(0x7ff20809a9f0 msgr2=0x7ff20809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.714+0000 7ff20d3d0640 1 -- 172.21.15.59:0/4247038728 shutdown_connections 2024-03-06T07:24:43.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.714+0000 7ff20d3d0640 1 -- 172.21.15.59:0/4247038728 wait complete. 2024-03-06T07:24:43.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.716+0000 7ff20d3d0640 1 Processor -- start 2024-03-06T07:24:43.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.716+0000 7ff20d3d0640 1 -- start start 2024-03-06T07:24:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff20d3d0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2080d16e0 0x7ff20815eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff20d3d0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815bdd0 0x7ff20814ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff20d3d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff20815fa10 0x7ff2081e79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff20d3d0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080d2050 con 0x7ff20815fa10 2024-03-06T07:24:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff20d3d0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080c5b50 con 0x7ff2080d16e0 2024-03-06T07:24:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff20d3d0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080cc5d0 con 0x7ff20815bdd0 2024-03-06T07:24:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff2077fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff20815fa10 0x7ff2081e79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff2077fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff20815fa10 0x7ff2081e79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60976/0 (socket says 172.21.15.59:60976) 2024-03-06T07:24:43.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff2077fe640 1 -- 172.21.15.59:0/763514053 learned_addr learned my addr 172.21.15.59:0/763514053 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff206ffd640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2080d16e0 0x7ff20815eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff1fffff640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815bdd0 0x7ff20814ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff2077fe640 1 -- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815bdd0 msgr2=0x7ff20814ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff2077fe640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815bdd0 0x7ff20814ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff2077fe640 1 -- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2080d16e0 msgr2=0x7ff20815eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff2077fe640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2080d16e0 0x7ff20815eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff2077fe640 1 -- 172.21.15.59:0/763514053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff208150480 con 0x7ff20815fa10 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff206ffd640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2080d16e0 0x7ff20815eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff1fffff640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815bdd0 0x7ff20814ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.717+0000 7ff2077fe640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff20815fa10 0x7ff2081e79a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff20000cd50 tx=0x7ff20000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.718+0000 7ff1ff7fe640 1 -- 172.21.15.59:0/763514053 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000f680 con 0x7ff20815fa10 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.718+0000 7ff1ff7fe640 1 -- 172.21.15.59:0/763514053 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20000f840 con 0x7ff20815fa10 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.718+0000 7ff20d3d0640 1 -- 172.21.15.59:0/763514053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2081e6900 con 0x7ff20815fa10 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.718+0000 7ff1ff7fe640 1 -- 172.21.15.59:0/763514053 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000fc30 con 0x7ff20815fa10 2024-03-06T07:24:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.718+0000 7ff20d3d0640 1 -- 172.21.15.59:0/763514053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2081e6ec0 con 0x7ff20815fa10 2024-03-06T07:24:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.719+0000 7ff20d3d0640 1 -- 172.21.15.59:0/763514053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff208068300 con 0x7ff20815fa10 2024-03-06T07:24:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.719+0000 7ff1ff7fe640 1 -- 172.21.15.59:0/763514053 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2000213f0 con 0x7ff20815fa10 2024-03-06T07:24:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.720+0000 7ff1ff7fe640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff1e806b4d0 0x7ff1e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.720+0000 7ff1ff7fe640 1 -- 172.21.15.59:0/763514053 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff20008afd0 con 0x7ff20815fa10 2024-03-06T07:24:43.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.720+0000 7ff206ffd640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff1e806b4d0 0x7ff1e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.721+0000 7ff206ffd640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff1e806b4d0 0x7ff1e806d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8036000 tx=0x7ff1f8008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.724+0000 7ff1ff7fe640 1 -- 172.21.15.59:0/763514053 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff208068300 con 0x7ff20815fa10 2024-03-06T07:24:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.944+0000 7ff20d3d0640 1 -- 172.21.15.59:0/763514053 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff20804f330 con 0x7ff20815fa10 2024-03-06T07:24:43.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.944+0000 7ff1ff7fe640 1 -- 172.21.15.59:0/763514053 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4980 (secure 0 0 0) 0x7ff208150480 con 0x7ff20815fa10 2024-03-06T07:24:43.946 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:43.946 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2205562350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2205562350},{"type":"v1","addr":"172.21.15.164:6837","nonce":2205562350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2606750481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2606750481},{"type":"v1","addr":"172.21.15.164:6836","nonce":2606750481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6048,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3685052110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3685052110},{"type":"v1","addr":"172.21.15.59:6833","nonce":3685052110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:11.010795+0000","modified":"2024-03-06T07:24:13.053115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/750409746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":750409746},{"type":"v1","addr":"172.21.15.59:6835","nonce":750409746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":4}]} 2024-03-06T07:24:43.947 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 28 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 -- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff1e806b4d0 msgr2=0x7ff1e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff1e806b4d0 0x7ff1e806d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8036000 tx=0x7ff1f8008030 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 -- 172.21.15.59:0/763514053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff20815fa10 msgr2=0x7ff2081e79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff20815fa10 0x7ff2081e79a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff20000cd50 tx=0x7ff20000a650 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 -- 172.21.15.59:0/763514053 shutdown_connections 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff1e806b4d0 0x7ff1e806d990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff20815fa10 0x7ff2081e79a0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff20815bdd0 0x7ff20814ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 --2- 172.21.15.59:0/763514053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2080d16e0 0x7ff20815eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 -- 172.21.15.59:0/763514053 >> 172.21.15.59:0/763514053 conn(0x7ff20809a9f0 msgr2=0x7ff2080caba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 -- 172.21.15.59:0/763514053 shutdown_connections 2024-03-06T07:24:43.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:43.946+0000 7ff20d3d0640 1 -- 172.21.15.59:0/763514053 wait complete. 2024-03-06T07:24:43.962 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:24:44.093 INFO:teuthology.orchestra.run.smithi059.stdout:{"message":"","return_code":0} 2024-03-06T07:24:44.093 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:24:44.093 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:44.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.196+0000 7fca566d4640 1 Processor -- start 2024-03-06T07:24:44.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.197+0000 7fca566d4640 1 -- start start 2024-03-06T07:24:44.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.197+0000 7fca566d4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fca501424f0 0x7fca500cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.197+0000 7fca566d4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fca5006da40 0x7fca5015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.197+0000 7fca566d4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca500cc980 con 0x7fca500cd7a0 2024-03-06T07:24:44.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.197+0000 7fca566d4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca500cd220 con 0x7fca5006da40 2024-03-06T07:24:44.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.197+0000 7fca566d4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca500ccc60 con 0x7fca501424f0 2024-03-06T07:24:44.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.197+0000 7fca54c71640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fca5006da40 0x7fca5015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.197+0000 7fca4f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fca500cd7a0 0x7fca5006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47308/0 (socket says 172.21.15.59:47308) 2024-03-06T07:24:44.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.197+0000 7fca4f7fe640 1 -- 172.21.15.59:0/1547036576 learned_addr learned my addr 172.21.15.59:0/1547036576 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:44.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.198+0000 7fca54c71640 1 -- 172.21.15.59:0/1547036576 >> v2:172.21.15.164:3300/0 conn(0x7fca501424f0 msgr2=0x7fca500cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.198+0000 7fca54c71640 1 --2- 172.21.15.59:0/1547036576 >> v2:172.21.15.164:3300/0 conn(0x7fca501424f0 0x7fca500cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.198+0000 7fca54c71640 1 -- 172.21.15.59:0/1547036576 >> v1:172.21.15.59:6789/0 conn(0x7fca500cd7a0 legacy=0x7fca5006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.198+0000 7fca54c71640 1 -- 172.21.15.59:0/1547036576 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca500ccf40 con 0x7fca5006da40 2024-03-06T07:24:44.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.198+0000 7fca54c71640 1 --2- 172.21.15.59:0/1547036576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5006da40 0x7fca5015cef0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fca5006c6d0 tx=0x7fca4802bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db99b88ca3e2f181 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.199+0000 7fca4effd640 1 -- 172.21.15.59:0/1547036576 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48039070 con 0x7fca5006da40 2024-03-06T07:24:44.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.199+0000 7fca4effd640 1 -- 172.21.15.59:0/1547036576 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4802ca30 con 0x7fca5006da40 2024-03-06T07:24:44.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.199+0000 7fca4effd640 1 -- 172.21.15.59:0/1547036576 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4802ce00 con 0x7fca5006da40 2024-03-06T07:24:44.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.199+0000 7fca566d4640 1 -- 172.21.15.59:0/1547036576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5006da40 msgr2=0x7fca5015cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.199+0000 7fca566d4640 1 --2- 172.21.15.59:0/1547036576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5006da40 0x7fca5015cef0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fca5006c6d0 tx=0x7fca4802bd00 comp rx=0 tx=0).stop 2024-03-06T07:24:44.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.199+0000 7fca566d4640 1 -- 172.21.15.59:0/1547036576 shutdown_connections 2024-03-06T07:24:44.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.199+0000 7fca566d4640 1 --2- 172.21.15.59:0/1547036576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5006da40 0x7fca5015cef0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.199+0000 7fca566d4640 1 --2- 172.21.15.59:0/1547036576 >> v2:172.21.15.164:3300/0 conn(0x7fca501424f0 0x7fca500cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.199+0000 7fca566d4640 1 -- 172.21.15.59:0/1547036576 >> 172.21.15.59:0/1547036576 conn(0x7fca500cf8a0 msgr2=0x7fca500cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.199+0000 7fca566d4640 1 -- 172.21.15.59:0/1547036576 shutdown_connections 2024-03-06T07:24:44.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.199+0000 7fca566d4640 1 -- 172.21.15.59:0/1547036576 wait complete. 2024-03-06T07:24:44.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.200+0000 7fca566d4640 1 Processor -- start 2024-03-06T07:24:44.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.201+0000 7fca566d4640 1 -- start start 2024-03-06T07:24:44.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca566d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5006da40 0x7fca501f3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca566d4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca500cd7a0 0x7fca501f1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca566d4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca501424f0 0x7fca501f2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca566d4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca50071ce0 con 0x7fca5006da40 2024-03-06T07:24:44.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca566d4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca50071b60 con 0x7fca500cd7a0 2024-03-06T07:24:44.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca566d4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca50071e60 con 0x7fca501424f0 2024-03-06T07:24:44.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca4ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5006da40 0x7fca501f3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca4ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5006da40 0x7fca501f3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60988/0 (socket says 172.21.15.59:60988) 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca4ffff640 1 -- 172.21.15.59:0/1980729566 learned_addr learned my addr 172.21.15.59:0/1980729566 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca54c71640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca501424f0 0x7fca501f2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca4f7fe640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca500cd7a0 0x7fca501f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca4ffff640 1 -- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca501424f0 msgr2=0x7fca501f2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca4ffff640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca501424f0 0x7fca501f2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca4ffff640 1 -- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca500cd7a0 msgr2=0x7fca501f1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca4ffff640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca500cd7a0 0x7fca501f1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca4ffff640 1 -- 172.21.15.59:0/1980729566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca501f2e70 con 0x7fca5006da40 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca54c71640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca501424f0 0x7fca501f2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.202+0000 7fca4ffff640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5006da40 0x7fca501f3070 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fca4000b410 tx=0x7fca4000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.203+0000 7fca4cff9640 1 -- 172.21.15.59:0/1980729566 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40015020 con 0x7fca5006da40 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.203+0000 7fca4cff9640 1 -- 172.21.15.59:0/1980729566 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca40011850 con 0x7fca5006da40 2024-03-06T07:24:44.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.203+0000 7fca566d4640 1 -- 172.21.15.59:0/1980729566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca501f3650 con 0x7fca5006da40 2024-03-06T07:24:44.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.203+0000 7fca4cff9640 1 -- 172.21.15.59:0/1980729566 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40011c40 con 0x7fca5006da40 2024-03-06T07:24:44.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.203+0000 7fca566d4640 1 -- 172.21.15.59:0/1980729566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca501f3c10 con 0x7fca5006da40 2024-03-06T07:24:44.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.204+0000 7fca566d4640 1 -- 172.21.15.59:0/1980729566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5004f330 con 0x7fca5006da40 2024-03-06T07:24:44.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.204+0000 7fca4cff9640 1 -- 172.21.15.59:0/1980729566 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca40018020 con 0x7fca5006da40 2024-03-06T07:24:44.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.205+0000 7fca4cff9640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3006b4d0 0x7fca3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.205+0000 7fca4cff9640 1 -- 172.21.15.59:0/1980729566 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca4008a110 con 0x7fca5006da40 2024-03-06T07:24:44.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.205+0000 7fca4f7fe640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3006b4d0 0x7fca3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.205+0000 7fca4f7fe640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3006b4d0 0x7fca3006d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fca38007fd0 tx=0x7fca38009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.209+0000 7fca4cff9640 1 -- 172.21.15.59:0/1980729566 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca4001f020 con 0x7fca5006da40 2024-03-06T07:24:44.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.431+0000 7fca566d4640 1 -- 172.21.15.59:0/1980729566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fca500d0620 con 0x7fca5006da40 2024-03-06T07:24:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.431+0000 7fca4cff9640 1 -- 172.21.15.59:0/1980729566 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+73 (secure 0 0 0) 0x7fca501f2e70 con 0x7fca5006da40 2024-03-06T07:24:44.433 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 -- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3006b4d0 msgr2=0x7fca3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3006b4d0 0x7fca3006d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fca38007fd0 tx=0x7fca38009040 comp rx=0 tx=0).stop 2024-03-06T07:24:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 -- 172.21.15.59:0/1980729566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5006da40 msgr2=0x7fca501f3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5006da40 0x7fca501f3070 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fca4000b410 tx=0x7fca4000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 -- 172.21.15.59:0/1980729566 shutdown_connections 2024-03-06T07:24:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3006b4d0 0x7fca3006d990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca501424f0 0x7fca501f2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca500cd7a0 0x7fca501f1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 --2- 172.21.15.59:0/1980729566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5006da40 0x7fca501f3070 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 -- 172.21.15.59:0/1980729566 >> 172.21.15.59:0/1980729566 conn(0x7fca500cf8a0 msgr2=0x7fca50072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 -- 172.21.15.59:0/1980729566 shutdown_connections 2024-03-06T07:24:44.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:44.433+0000 7fca566d4640 1 -- 172.21.15.59:0/1980729566 wait complete. 2024-03-06T07:24:44.445 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:44.445 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:44.445 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:24:44.446 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:44.446 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:44.446 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:44.446 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:44.446 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:44.446 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:44.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:44.446 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:24:44.462 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:44.462 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42494sec preferred_lft 42494sec 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:24:44.463 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:44.464 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:24:44.465 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:44.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:44.465 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:44.527 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:24:44.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:44.529 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:24:44.583 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:24:44.583 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:44.583 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:24:44.584 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:24:44.584 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:44.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:44.584 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:24:44.584 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:24:44.584 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:24:44.584 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:24:44.584 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:24:44.584 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:24:44.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:44.678 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:24:44.692 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:44.693 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:44.693 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:44.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:44.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:24:44.760 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:44.760 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:44.760 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:44.815 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:44.816 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:44.816 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:44.870 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:24:44.938 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:44.995 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:24:44.995 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:24:44.995 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:24:44.996 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:24:44.996 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:24:44.996 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:24:44.996 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:24:44.996 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:24:44.996 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:24:44.996 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:24:44.997 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:24:44.997 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:24:44.997 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:44.997 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:24:44.997 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:24:44.997 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:24:44.997 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:24:44.997 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:44.997 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:44.998 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:24:44.998 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:44.998 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:44.998 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:44.998 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:24:44.998 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:24:44.998 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:44.999 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:44.999 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:44.999 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:44.999 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:24:44.999 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:44.999 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:44.999 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:45.000 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:45.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:45.000 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:24:45.050 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:45.051 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:45.095 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:24:45.124 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:45.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:24:45.179+0000 7f38c0f3f480 -1 init, newargv = 0x5622b6990780 newargc=15 2024-03-06T07:24:45.181 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[59907]: starting ceph client 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:24:45.182 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:45.184 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:45.184 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:24:45.184 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:45.184 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:45.184 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:45.184 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:45.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:45.184 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:24:45.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[59907]: starting fuse 2024-03-06T07:24:45.200 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:24:45.200 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:24:45.200 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:45.200 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:24:45.200 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:24:45.200 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:24:45.200 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:24:45.200 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:45.200 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:24:45.201 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:24:45.202 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:24:45.202 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:45.202 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:45.202 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:45.202 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:24:45.320 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.59907.asok 2024-03-06T07:24:45.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:45.331 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59907.asok 2024-03-06T07:24:45.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:45.331 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59907.asok status 2024-03-06T07:24:45.467 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:24:45.467 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:24:45.467 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:45.467 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:45.467 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:24:45.467 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:24:45.467 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:45.467 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "59907", 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 6369, 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 6369 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:24:45.468 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3852551908 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3852551908 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.6369 192.168.0.1:0/3852551908", 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/3852551908", 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:24:45.469 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 28, 2024-03-06T07:24:45.470 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 59, 2024-03-06T07:24:45.470 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:45.470 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:24:45.470 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:24:45.470 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:24:45.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:45.475 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:24:45.475 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:45.490 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:24:45.491 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:45.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:45.491 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:45.557 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:24:45.557 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:45.613 DEBUG:teuthology.orchestra.run.smithi059:> stdin-killer -- python3 -c ' 2024-03-06T07:24:45.614 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:24:45.614 DEBUG:teuthology.orchestra.run.smithi059:> import uuid 2024-03-06T07:24:45.614 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:45.614 DEBUG:teuthology.orchestra.run.smithi059:> n = 75 2024-03-06T07:24:45.614 DEBUG:teuthology.orchestra.run.smithi059:> create_hard_links = False 2024-03-06T07:24:45.614 DEBUG:teuthology.orchestra.run.smithi059:> if 0 > 0: 2024-03-06T07:24:45.614 DEBUG:teuthology.orchestra.run.smithi059:> create_hard_links = True 2024-03-06T07:24:45.615 DEBUG:teuthology.orchestra.run.smithi059:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:45.615 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:45.615 DEBUG:teuthology.orchestra.run.smithi059:> dpath = os.path.dirname(path) 2024-03-06T07:24:45.615 DEBUG:teuthology.orchestra.run.smithi059:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:45.615 DEBUG:teuthology.orchestra.run.smithi059:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:45.615 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:45.616 DEBUG:teuthology.orchestra.run.smithi059:> try: 2024-03-06T07:24:45.616 DEBUG:teuthology.orchestra.run.smithi059:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:45.616 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:24:45.616 DEBUG:teuthology.orchestra.run.smithi059:> for i in range(n): 2024-03-06T07:24:45.616 DEBUG:teuthology.orchestra.run.smithi059:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:45.616 DEBUG:teuthology.orchestra.run.smithi059:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:45.617 DEBUG:teuthology.orchestra.run.smithi059:> f.write(f"{i}") 2024-03-06T07:24:45.617 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:24:45.617 DEBUG:teuthology.orchestra.run.smithi059:> f.flush() 2024-03-06T07:24:45.617 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(f.fileno()) 2024-03-06T07:24:45.617 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:24:45.617 DEBUG:teuthology.orchestra.run.smithi059:> os.unlink(fpath) 2024-03-06T07:24:45.617 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:24:45.618 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(dirfd) 2024-03-06T07:24:45.618 DEBUG:teuthology.orchestra.run.smithi059:> if create_hard_links: 2024-03-06T07:24:45.618 DEBUG:teuthology.orchestra.run.smithi059:> for j in range(0): 2024-03-06T07:24:45.618 DEBUG:teuthology.orchestra.run.smithi059:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:45.618 DEBUG:teuthology.orchestra.run.smithi059:> if True: 2024-03-06T07:24:45.618 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(dirfd) 2024-03-06T07:24:45.619 DEBUG:teuthology.orchestra.run.smithi059:> finally: 2024-03-06T07:24:45.619 DEBUG:teuthology.orchestra.run.smithi059:> os.close(dirfd) 2024-03-06T07:24:45.619 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:24:46.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:46.121 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:46.121 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:46.206 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:46.252 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:46.252 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:24:46.252 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:46.267 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:46.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.406+0000 7fac5ec84640 1 Processor -- start 2024-03-06T07:24:46.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.406+0000 7fac5ec84640 1 -- start start 2024-03-06T07:24:46.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.407+0000 7fac5ec84640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fac5815dfe0 0x7fac580cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.407+0000 7fac5ec84640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac580d29c0 0x7fac580b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.407+0000 7fac5ec84640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac580b3430 con 0x7fac580cf830 2024-03-06T07:24:46.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.407+0000 7fac5ec84640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac580b31e0 con 0x7fac5815dfe0 2024-03-06T07:24:46.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.407+0000 7fac5ec84640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580b3d60 con 0x7fac580d29c0 2024-03-06T07:24:46.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.407+0000 7fac5d221640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac580d29c0 0x7fac580b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.408+0000 7fac5d221640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac580d29c0 0x7fac580b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:49476/0 (socket says 172.21.15.59:49476) 2024-03-06T07:24:46.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.408+0000 7fac5ca20640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fac5815dfe0 0x7fac580cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.408+0000 7fac57fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fac580cf830 0x7fac580d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:49878/0 (socket says 172.21.15.59:49878) 2024-03-06T07:24:46.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.408+0000 7fac5d221640 1 -- 172.21.15.59:0/1268120111 learned_addr learned my addr 172.21.15.59:0/1268120111 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:46.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.408+0000 7fac5ca20640 1 -- 172.21.15.59:0/1268120111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac580d29c0 msgr2=0x7fac580b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.408+0000 7fac5ca20640 1 --2- 172.21.15.59:0/1268120111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac580d29c0 0x7fac580b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.408+0000 7fac5ca20640 1 -- 172.21.15.59:0/1268120111 >> v1:172.21.15.164:6789/0 conn(0x7fac580cf830 legacy=0x7fac580d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.408+0000 7fac577fe640 1 -- 172.21.15.59:0/1268120111 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467516094 0 0) 0x7fac580b3430 con 0x7fac580cf830 2024-03-06T07:24:46.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.408+0000 7fac5ca20640 1 -- 172.21.15.59:0/1268120111 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580b3710 con 0x7fac5815dfe0 2024-03-06T07:24:46.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.408+0000 7fac5d221640 1 --2- 172.21.15.59:0/1268120111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac580d29c0 0x7fac580b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac5ca20640 1 --2- 172.21.15.59:0/1268120111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac5815dfe0 0x7fac580cf2f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0097a0 tx=0x7fac4c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fef01e9c1116c661 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac577fe640 1 -- 172.21.15.59:0/1268120111 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c035020 con 0x7fac5815dfe0 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac577fe640 1 -- 172.21.15.59:0/1268120111 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c031830 con 0x7fac5815dfe0 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac577fe640 1 -- 172.21.15.59:0/1268120111 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c031c00 con 0x7fac5815dfe0 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac5ec84640 1 -- 172.21.15.59:0/1268120111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac5815dfe0 msgr2=0x7fac580cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac5ec84640 1 --2- 172.21.15.59:0/1268120111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac5815dfe0 0x7fac580cf2f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0097a0 tx=0x7fac4c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac5ec84640 1 -- 172.21.15.59:0/1268120111 shutdown_connections 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac5ec84640 1 --2- 172.21.15.59:0/1268120111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac580d29c0 0x7fac580b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac5ec84640 1 --2- 172.21.15.59:0/1268120111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac5815dfe0 0x7fac580cf2f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac5ec84640 1 -- 172.21.15.59:0/1268120111 >> 172.21.15.59:0/1268120111 conn(0x7fac580d3cd0 msgr2=0x7fac580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac5ec84640 1 -- 172.21.15.59:0/1268120111 shutdown_connections 2024-03-06T07:24:46.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.409+0000 7fac5ec84640 1 -- 172.21.15.59:0/1268120111 wait complete. 2024-03-06T07:24:46.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.410+0000 7fac5ec84640 1 Processor -- start 2024-03-06T07:24:46.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.410+0000 7fac5ec84640 1 -- start start 2024-03-06T07:24:46.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac5ec84640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac580cf830 0x7fac580c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac5ec84640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac580d29c0 0x7fac5814d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac5ec84640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac5815dfe0 0x7fac5814d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac5ec84640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580d1df0 con 0x7fac580d29c0 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac5ec84640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580d1c70 con 0x7fac580cf830 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac5ec84640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5815e3c0 con 0x7fac5815dfe0 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac57fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac580d29c0 0x7fac5814d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac5ca20640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac580cf830 0x7fac580c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac5d221640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac5815dfe0 0x7fac5814d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac5ca20640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac580cf830 0x7fac580c9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:50046/0 (socket says 172.21.15.59:50046) 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac5d221640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac5815dfe0 0x7fac5814d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:49492/0 (socket says 172.21.15.59:49492) 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac5ca20640 1 -- 172.21.15.59:0/2644168947 learned_addr learned my addr 172.21.15.59:0/2644168947 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.411+0000 7fac57fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac580d29c0 0x7fac5814d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32772/0 (socket says 172.21.15.59:32772) 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac5ca20640 1 -- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac5815dfe0 msgr2=0x7fac5814d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac5ca20640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac5815dfe0 0x7fac5814d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac5ca20640 1 -- 172.21.15.59:0/2644168947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac580d29c0 msgr2=0x7fac5814d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac5ca20640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac580d29c0 0x7fac5814d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac5ca20640 1 -- 172.21.15.59:0/2644168947 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac58161120 con 0x7fac580cf830 2024-03-06T07:24:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac5ca20640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac580cf830 0x7fac580c9f50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0097a0 tx=0x7fac4c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac557fa640 1 -- 172.21.15.59:0/2644168947 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c035040 con 0x7fac580cf830 2024-03-06T07:24:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac557fa640 1 -- 172.21.15.59:0/2644168947 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c004230 con 0x7fac580cf830 2024-03-06T07:24:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac5ec84640 1 -- 172.21.15.59:0/2644168947 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5814b150 con 0x7fac580cf830 2024-03-06T07:24:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac557fa640 1 -- 172.21.15.59:0/2644168947 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c03a640 con 0x7fac580cf830 2024-03-06T07:24:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.412+0000 7fac5ec84640 1 -- 172.21.15.59:0/2644168947 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5814b710 con 0x7fac580cf830 2024-03-06T07:24:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.413+0000 7fac5ec84640 1 -- 172.21.15.59:0/2644168947 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5804f330 con 0x7fac580cf830 2024-03-06T07:24:46.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.414+0000 7fac557fa640 1 -- 172.21.15.59:0/2644168947 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac4c038020 con 0x7fac580cf830 2024-03-06T07:24:46.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.414+0000 7fac557fa640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac3c06b4d0 0x7fac3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.414+0000 7fac557fa640 1 -- 172.21.15.59:0/2644168947 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac4c0abd90 con 0x7fac580cf830 2024-03-06T07:24:46.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.414+0000 7fac57fff640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac3c06b4d0 0x7fac3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.415+0000 7fac57fff640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac3c06b4d0 0x7fac3c06d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fac580cad50 tx=0x7fac44007b50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.419+0000 7fac557fa640 1 -- 172.21.15.59:0/2644168947 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac4c07aca0 con 0x7fac580cf830 2024-03-06T07:24:46.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.600+0000 7fac5ec84640 1 -- 172.21.15.59:0/2644168947 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac580b51a0 con 0x7fac580cf830 2024-03-06T07:24:46.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.601+0000 7fac557fa640 1 -- 172.21.15.59:0/2644168947 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fac4c07a5a0 con 0x7fac580cf830 2024-03-06T07:24:46.603 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:46.603 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:24:12.652576+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:09.288329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.978574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.603+0000 7fac5ec84640 1 -- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac3c06b4d0 msgr2=0x7fac3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.603+0000 7fac5ec84640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac3c06b4d0 0x7fac3c06d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fac580cad50 tx=0x7fac44007b50 comp rx=0 tx=0).stop 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.603+0000 7fac5ec84640 1 -- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac580cf830 msgr2=0x7fac580c9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.603+0000 7fac5ec84640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac580cf830 0x7fac580c9f50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0097a0 tx=0x7fac4c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.604+0000 7fac5ec84640 1 -- 172.21.15.59:0/2644168947 shutdown_connections 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.604+0000 7fac5ec84640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac3c06b4d0 0x7fac3c06d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.604+0000 7fac5ec84640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac5815dfe0 0x7fac5814d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.604+0000 7fac5ec84640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac580d29c0 0x7fac5814d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.604+0000 7fac5ec84640 1 --2- 172.21.15.59:0/2644168947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac580cf830 0x7fac580c9f50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.604+0000 7fac5ec84640 1 -- 172.21.15.59:0/2644168947 >> 172.21.15.59:0/2644168947 conn(0x7fac580d3cd0 msgr2=0x7fac5806cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.604+0000 7fac5ec84640 1 -- 172.21.15.59:0/2644168947 shutdown_connections 2024-03-06T07:24:46.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.604+0000 7fac5ec84640 1 -- 172.21.15.59:0/2644168947 wait complete. 2024-03-06T07:24:46.620 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:09.288329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:09.978574+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:46.620 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.722+0000 7f0254ed4640 1 Processor -- start 2024-03-06T07:24:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.723+0000 7f0254ed4640 1 -- start start 2024-03-06T07:24:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.723+0000 7f0254ed4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02500d4860 0x7f02500d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.723+0000 7f0254ed4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f02500b0e30 0x7f02500b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.723+0000 7f0254ed4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02501618b0 con 0x7f02501614d0 2024-03-06T07:24:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.723+0000 7f0254ed4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02500d4cc0 con 0x7f02500b0e30 2024-03-06T07:24:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.723+0000 7f0254ed4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02500cf400 con 0x7f02500d4860 2024-03-06T07:24:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.724+0000 7f024ed9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f02500b0e30 0x7f02500b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.724+0000 7f0245d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02500d4860 0x7f02500d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.724+0000 7f024e59c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f02501614d0 0x7f02500d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:49882/0 (socket says 172.21.15.59:49882) 2024-03-06T07:24:46.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.724+0000 7f024e59c640 1 -- 172.21.15.59:0/1766411983 learned_addr learned my addr 172.21.15.59:0/1766411983 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:46.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.724+0000 7f024ed9d640 1 -- 172.21.15.59:0/1766411983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02500d4860 msgr2=0x7f02500d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.724+0000 7f024ed9d640 1 --2- 172.21.15.59:0/1766411983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02500d4860 0x7f02500d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.724+0000 7f024ed9d640 1 -- 172.21.15.59:0/1766411983 >> v1:172.21.15.164:6789/0 conn(0x7f02501614d0 legacy=0x7f02500d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.724+0000 7f024ed9d640 1 -- 172.21.15.59:0/1766411983 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02500cf6a0 con 0x7f02500b0e30 2024-03-06T07:24:46.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.724+0000 7f024dd9b640 1 -- 172.21.15.59:0/1766411983 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256044670 0 0) 0x7f02501618b0 con 0x7f02501614d0 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f024ed9d640 1 --2- 172.21.15.59:0/1766411983 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02500b0e30 0x7f02500b12c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0248009d90 tx=0x7f024802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a8e402977f52bfb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f024dd9b640 1 -- 172.21.15.59:0/1766411983 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248039070 con 0x7f02500b0e30 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f024dd9b640 1 -- 172.21.15.59:0/1766411983 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024802cc50 con 0x7f02500b0e30 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f024dd9b640 1 -- 172.21.15.59:0/1766411983 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248034440 con 0x7f02500b0e30 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f0254ed4640 1 -- 172.21.15.59:0/1766411983 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02500b0e30 msgr2=0x7f02500b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f0254ed4640 1 --2- 172.21.15.59:0/1766411983 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02500b0e30 0x7f02500b12c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0248009d90 tx=0x7f024802bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f0254ed4640 1 -- 172.21.15.59:0/1766411983 shutdown_connections 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f0254ed4640 1 --2- 172.21.15.59:0/1766411983 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02500b0e30 0x7f02500b12c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f0254ed4640 1 --2- 172.21.15.59:0/1766411983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02500d4860 0x7f02500d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f0254ed4640 1 -- 172.21.15.59:0/1766411983 >> 172.21.15.59:0/1766411983 conn(0x7f02500d3d40 msgr2=0x7f02500d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f0254ed4640 1 -- 172.21.15.59:0/1766411983 shutdown_connections 2024-03-06T07:24:46.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.725+0000 7f0254ed4640 1 -- 172.21.15.59:0/1766411983 wait complete. 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.727+0000 7f0254ed4640 1 Processor -- start 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.727+0000 7f0254ed4640 1 -- start start 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.727+0000 7f0254ed4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02500b0e30 0x7f02501542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.727+0000 7f0254ed4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02500d4860 0x7f02500c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.727+0000 7f0254ed4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02501614d0 0x7f02500c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.727+0000 7f0254ed4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02500b0910 con 0x7f02500d4860 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.727+0000 7f0254ed4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02500cfc60 con 0x7f02501614d0 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.727+0000 7f0254ed4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02500d2560 con 0x7f02500b0e30 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f0245d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02500d4860 0x7f02500c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f0245d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02500d4860 0x7f02500c9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32788/0 (socket says 172.21.15.59:32788) 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f0245d9b640 1 -- 172.21.15.59:0/253528052 learned_addr learned my addr 172.21.15.59:0/253528052 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f024e59c640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02500b0e30 0x7f02501542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f024ed9d640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02501614d0 0x7f02500c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f0245d9b640 1 -- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02500b0e30 msgr2=0x7f02501542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f0245d9b640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02500b0e30 0x7f02501542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f0245d9b640 1 -- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02501614d0 msgr2=0x7f02500c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f0245d9b640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02501614d0 0x7f02500c97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f0245d9b640 1 -- 172.21.15.59:0/253528052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02500c7c50 con 0x7f02500d4860 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f0245d9b640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02500d4860 0x7f02500c9270 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f023800e9b0 tx=0x7f023800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f02477fe640 1 -- 172.21.15.59:0/253528052 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238017070 con 0x7f02500d4860 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f02477fe640 1 -- 172.21.15.59:0/253528052 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023800c9a0 con 0x7f02500d4860 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f0254ed4640 1 -- 172.21.15.59:0/253528052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02500c7f30 con 0x7f02500d4860 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f02477fe640 1 -- 172.21.15.59:0/253528052 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023800cd90 con 0x7f02500d4860 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.728+0000 7f0254ed4640 1 -- 172.21.15.59:0/253528052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02500c84f0 con 0x7f02500d4860 2024-03-06T07:24:46.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.729+0000 7f0254ed4640 1 -- 172.21.15.59:0/253528052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0250068ff0 con 0x7f02500d4860 2024-03-06T07:24:46.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.730+0000 7f02477fe640 1 -- 172.21.15.59:0/253528052 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f023801a070 con 0x7f02500d4860 2024-03-06T07:24:46.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.731+0000 7f02477fe640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f023006b4d0 0x7f023006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.731+0000 7f02477fe640 1 -- 172.21.15.59:0/253528052 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f023808a4e0 con 0x7f02500d4860 2024-03-06T07:24:46.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.731+0000 7f024e59c640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f023006b4d0 0x7f023006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.731+0000 7f024e59c640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f023006b4d0 0x7f023006d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0240007fd0 tx=0x7f0240007de0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.739+0000 7f02477fe640 1 -- 172.21.15.59:0/253528052 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0238058d10 con 0x7f02500d4860 2024-03-06T07:24:46.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.921+0000 7f0254ed4640 1 -- 172.21.15.59:0/253528052 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f02500747a0 con 0x7f02500d4860 2024-03-06T07:24:46.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.921+0000 7f02477fe640 1 -- 172.21.15.59:0/253528052 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f023805cdb0 con 0x7f02500d4860 2024-03-06T07:24:46.923 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:24:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.923+0000 7f0254ed4640 1 -- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f023006b4d0 msgr2=0x7f023006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.923+0000 7f0254ed4640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f023006b4d0 0x7f023006d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0240007fd0 tx=0x7f0240007de0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.923+0000 7f0254ed4640 1 -- 172.21.15.59:0/253528052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02500d4860 msgr2=0x7f02500c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.923+0000 7f0254ed4640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02500d4860 0x7f02500c9270 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f023800e9b0 tx=0x7f023800ee70 comp rx=0 tx=0).stop 2024-03-06T07:24:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.923+0000 7f0254ed4640 1 -- 172.21.15.59:0/253528052 shutdown_connections 2024-03-06T07:24:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.923+0000 7f0254ed4640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f023006b4d0 0x7f023006d990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.923+0000 7f0254ed4640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02501614d0 0x7f02500c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.923+0000 7f0254ed4640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02500d4860 0x7f02500c9270 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.923+0000 7f0254ed4640 1 --2- 172.21.15.59:0/253528052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02500b0e30 0x7f02501542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.923+0000 7f0254ed4640 1 -- 172.21.15.59:0/253528052 >> 172.21.15.59:0/253528052 conn(0x7f02500d3d40 msgr2=0x7f025006b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.923+0000 7f0254ed4640 1 -- 172.21.15.59:0/253528052 shutdown_connections 2024-03-06T07:24:46.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:46.924+0000 7f0254ed4640 1 -- 172.21.15.59:0/253528052 wait complete. 2024-03-06T07:24:46.936 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:47.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.038+0000 7feba53a2640 1 Processor -- start 2024-03-06T07:24:47.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.038+0000 7feba53a2640 1 -- start start 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.039+0000 7feba53a2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feba015e0d0 0x7feba00b4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.039+0000 7feba53a2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba015e690 con 0x7feba00b4e90 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.039+0000 7feba53a2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba015e970 con 0x7feba0158030 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.039+0000 7feba53a2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba015ec50 con 0x7feba015e0d0 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.039+0000 7feba4ba1640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7feba00b4e90 0x7feba006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47320/0 (socket says 172.21.15.59:47320) 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.039+0000 7feba4ba1640 1 -- 172.21.15.59:0/1661780941 learned_addr learned my addr 172.21.15.59:0/1661780941 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.039+0000 7feb9f7fe640 1 --2- 172.21.15.59:0/1661780941 >> v2:172.21.15.164:3300/0 conn(0x7feba015e0d0 0x7feba00b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.039+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237703473 0 0) 0x7feba015e690 con 0x7feba00b4e90 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.039+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb88003520 con 0x7feba00b4e90 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2343142643 0 0) 0x7feb88003520 con 0x7feba00b4e90 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 >> v2:172.21.15.164:3300/0 conn(0x7feba015e0d0 msgr2=0x7feba00b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 --2- 172.21.15.59:0/1661780941 >> v2:172.21.15.164:3300/0 conn(0x7feba015e0d0 0x7feba00b4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 >> v1:172.21.15.164:6789/0 conn(0x7feba0158030 legacy=0x7feba0072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba015ef30 con 0x7feba00b4e90 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7feb980019f0 con 0x7feba00b4e90 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 >> v1:172.21.15.59:6789/0 conn(0x7feba00b4e90 legacy=0x7feba006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 0x7feb88003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb88004800 0x7feb88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb880051c0 0x7feb8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba015ec50 con 0x7feb880039b0 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8800c060 con 0x7feb88004800 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8800c260 con 0x7feb880051c0 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9ffff640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 0x7feb88003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9f7fe640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb88004800 0x7feb88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feba4ba1640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb880051c0 0x7feb8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9ffff640 1 -- 172.21.15.59:0/1661780941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb880051c0 msgr2=0x7feb8800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9ffff640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb880051c0 0x7feb8800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9ffff640 1 -- 172.21.15.59:0/1661780941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb88004800 msgr2=0x7feb88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9ffff640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb88004800 0x7feb88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feba4ba1640 1 -- 172.21.15.59:0/1661780941 reap_dead start 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9ffff640 1 -- 172.21.15.59:0/1661780941 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb88003520 con 0x7feb880039b0 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.040+0000 7feb9f7fe640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb88004800 0x7feb88004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.041+0000 7feb9ffff640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 0x7feb88003dd0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7feb90001750 tx=0x7feb88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee23cc3f280ada5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.041+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90001370 con 0x7feb880039b0 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.041+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb900052d0 con 0x7feb880039b0 2024-03-06T07:24:47.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.041+0000 7feb9effd640 1 -- 172.21.15.59:0/1661780941 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90003830 con 0x7feb880039b0 2024-03-06T07:24:47.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.042+0000 7feba53a2640 1 -- 172.21.15.59:0/1661780941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 msgr2=0x7feb88003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.042+0000 7feba53a2640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 0x7feb88003dd0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7feb90001750 tx=0x7feb88005830 comp rx=0 tx=0).stop 2024-03-06T07:24:47.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.042+0000 7feba53a2640 1 -- 172.21.15.59:0/1661780941 shutdown_connections 2024-03-06T07:24:47.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.042+0000 7feba53a2640 1 --2- 172.21.15.59:0/1661780941 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 0x7feb88003dd0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.042+0000 7feba53a2640 1 -- 172.21.15.59:0/1661780941 >> 172.21.15.59:0/1661780941 conn(0x7feba00b2510 msgr2=0x7feba00b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.042+0000 7feba53a2640 1 -- 172.21.15.59:0/1661780941 shutdown_connections 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.042+0000 7feba53a2640 1 -- 172.21.15.59:0/1661780941 wait complete. 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.043+0000 7feba53a2640 1 Processor -- start 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.043+0000 7feba53a2640 1 -- start start 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feba53a2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 0x7feba01e7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feba53a2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feba0156e60 0x7feba0156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feba53a2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feba00ae9e0 0x7feba00aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feba53a2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba0158590 con 0x7feb880039b0 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feba53a2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba0158410 con 0x7feba0156e60 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feba53a2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba00b4950 con 0x7feba00ae9e0 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feb9ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 0x7feba01e7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feb9ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 0x7feba01e7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32802/0 (socket says 172.21.15.59:32802) 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feb9ffff640 1 -- 172.21.15.59:0/3835013408 learned_addr learned my addr 172.21.15.59:0/3835013408 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feb9f7fe640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feba0156e60 0x7feba0156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feb9ffff640 1 -- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feba00ae9e0 msgr2=0x7feba00aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feba4ba1640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feba00ae9e0 0x7feba00aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feb9ffff640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feba00ae9e0 0x7feba00aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feb9ffff640 1 -- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feba0156e60 msgr2=0x7feba0156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feb9ffff640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feba0156e60 0x7feba0156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feb9ffff640 1 -- 172.21.15.59:0/3835013408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba01e9040 con 0x7feb880039b0 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feb9f7fe640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feba0156e60 0x7feba0156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.044+0000 7feb9ffff640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 0x7feba01e7f90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7feb90000c00 tx=0x7feb90001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.045+0000 7feb9cff9640 1 -- 172.21.15.59:0/3835013408 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90041020 con 0x7feb880039b0 2024-03-06T07:24:47.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.045+0000 7feb9cff9640 1 -- 172.21.15.59:0/3835013408 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb90005480 con 0x7feb880039b0 2024-03-06T07:24:47.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.045+0000 7feba53a2640 1 -- 172.21.15.59:0/3835013408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba01e84d0 con 0x7feb880039b0 2024-03-06T07:24:47.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.045+0000 7feb9cff9640 1 -- 172.21.15.59:0/3835013408 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb90007a30 con 0x7feb880039b0 2024-03-06T07:24:47.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.045+0000 7feba53a2640 1 -- 172.21.15.59:0/3835013408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba01e8a80 con 0x7feb880039b0 2024-03-06T07:24:47.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.046+0000 7feba53a2640 1 -- 172.21.15.59:0/3835013408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba0068f10 con 0x7feb880039b0 2024-03-06T07:24:47.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.046+0000 7feb9cff9640 1 -- 172.21.15.59:0/3835013408 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb90044020 con 0x7feb880039b0 2024-03-06T07:24:47.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.046+0000 7feb9cff9640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb8c06b4d0 0x7feb8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.047+0000 7feb9cff9640 1 -- 172.21.15.59:0/3835013408 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb900abc80 con 0x7feb880039b0 2024-03-06T07:24:47.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.047+0000 7feb9f7fe640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb8c06b4d0 0x7feb8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.047+0000 7feb9f7fe640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb8c06b4d0 0x7feb8c06d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7feb94006d40 tx=0x7feb94008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.051+0000 7feb9cff9640 1 -- 172.21.15.59:0/3835013408 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb9004b030 con 0x7feb880039b0 2024-03-06T07:24:47.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.231+0000 7feba53a2640 1 -- 172.21.15.59:0/3835013408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feba00b1210 con 0x7feb880039b0 2024-03-06T07:24:47.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.231+0000 7feb9cff9640 1 -- 172.21.15.59:0/3835013408 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7feb9007aa70 con 0x7feb880039b0 2024-03-06T07:24:47.233 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 -- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb8c06b4d0 msgr2=0x7feb8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb8c06b4d0 0x7feb8c06d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7feb94006d40 tx=0x7feb94008040 comp rx=0 tx=0).stop 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 -- 172.21.15.59:0/3835013408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 msgr2=0x7feba01e7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 0x7feba01e7f90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7feb90000c00 tx=0x7feb90001120 comp rx=0 tx=0).stop 2024-03-06T07:24:47.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 -- 172.21.15.59:0/3835013408 shutdown_connections 2024-03-06T07:24:47.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb8c06b4d0 0x7feb8c06d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feba00ae9e0 0x7feba00aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feba0156e60 0x7feba0156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 --2- 172.21.15.59:0/3835013408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb880039b0 0x7feba01e7f90 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 -- 172.21.15.59:0/3835013408 >> 172.21.15.59:0/3835013408 conn(0x7feba00b2510 msgr2=0x7feba00d05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 -- 172.21.15.59:0/3835013408 shutdown_connections 2024-03-06T07:24:47.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.234+0000 7feba53a2640 1 -- 172.21.15.59:0/3835013408 wait complete. 2024-03-06T07:24:47.246 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.351+0000 7faddc9e2640 1 Processor -- start 2024-03-06T07:24:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.351+0000 7faddc9e2640 1 -- start start 2024-03-06T07:24:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.351+0000 7faddc9e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd815fc30 0x7fadd80b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.351+0000 7faddc9e2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fadd80b4d10 0x7fadd80cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.351+0000 7faddc9e2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd80d47f0 con 0x7fadd80d06e0 2024-03-06T07:24:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.351+0000 7faddc9e2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd80ced70 con 0x7fadd80b4d10 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.351+0000 7faddc9e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd8160070 con 0x7fadd815fc30 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd659c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fadd80d06e0 0x7fadd80d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47328/0 (socket says 172.21.15.59:47328) 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd659c640 1 -- 172.21.15.59:0/1356881126 learned_addr learned my addr 172.21.15.59:0/1356881126 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd6d9d640 1 --2- 172.21.15.59:0/1356881126 >> v2:172.21.15.164:3300/0 conn(0x7fadd80b4d10 0x7fadd80cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd5d9b640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd815fc30 0x7fadd80b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369713220 0 0) 0x7fadd80d47f0 con 0x7fadd80d06e0 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc0003520 con 0x7fadd80d06e0 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3688306351 0 0) 0x7fadc0003520 con 0x7fadd80d06e0 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd815fc30 msgr2=0x7fadd80b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd815fc30 0x7fadd80b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 >> v2:172.21.15.164:3300/0 conn(0x7fadd80b4d10 msgr2=0x7fadd80cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 --2- 172.21.15.59:0/1356881126 >> v2:172.21.15.164:3300/0 conn(0x7fadd80b4d10 0x7fadd80cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80d1ab0 con 0x7fadd80d06e0 2024-03-06T07:24:47.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fadc80019f0 con 0x7fadd80d06e0 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 >> v1:172.21.15.59:6789/0 conn(0x7fadd80d06e0 legacy=0x7fadd80d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd815fc30 0x7fadc00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadc0003f30 0x7fadc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc0004d40 0x7fadc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80ced70 con 0x7fadc0003f30 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd8160070 con 0x7fadc0004d40 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc000b9c0 con 0x7fadd815fc30 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd5d9b640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadc0003f30 0x7fadc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd659c640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd815fc30 0x7fadc00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.352+0000 7fadd6d9d640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc0004d40 0x7fadc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.353+0000 7fadd5d9b640 1 -- 172.21.15.59:0/1356881126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd815fc30 msgr2=0x7fadc00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.353+0000 7fadd5d9b640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd815fc30 0x7fadc00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.353+0000 7fadd5d9b640 1 -- 172.21.15.59:0/1356881126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc0004d40 msgr2=0x7fadc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.353+0000 7fadd5d9b640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc0004d40 0x7fadc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.353+0000 7fadd5d9b640 1 -- 172.21.15.59:0/1356881126 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc0003520 con 0x7fadc0003f30 2024-03-06T07:24:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.354+0000 7fadd5d9b640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadc0003f30 0x7fadc0004800 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009ac0 tx=0x7fadc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72bdbb1e41d6720 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.354+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc039070 con 0x7fadc0003f30 2024-03-06T07:24:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.354+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc02c6b0 con 0x7fadc0003f30 2024-03-06T07:24:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.354+0000 7fadd559a640 1 -- 172.21.15.59:0/1356881126 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc009690 con 0x7fadc0003f30 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.355+0000 7faddc9e2640 1 -- 172.21.15.59:0/1356881126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadc0003f30 msgr2=0x7fadc0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.355+0000 7faddc9e2640 1 --2- 172.21.15.59:0/1356881126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadc0003f30 0x7fadc0004800 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009ac0 tx=0x7fadc0005830 comp rx=0 tx=0).stop 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.355+0000 7fadd6d9d640 1 -- 172.21.15.59:0/1356881126 reap_dead start 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.355+0000 7faddc9e2640 1 -- 172.21.15.59:0/1356881126 shutdown_connections 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.355+0000 7faddc9e2640 1 -- 172.21.15.59:0/1356881126 >> 172.21.15.59:0/1356881126 conn(0x7fadd80d3cd0 msgr2=0x7fadd80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.355+0000 7faddc9e2640 1 -- 172.21.15.59:0/1356881126 shutdown_connections 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.355+0000 7faddc9e2640 1 -- 172.21.15.59:0/1356881126 wait complete. 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.357+0000 7faddc9e2640 1 Processor -- start 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.357+0000 7faddc9e2640 1 -- start start 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7faddc9e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadc0003f30 0x7fadd80b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7faddc9e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd81f7590 0x7fadd81f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7faddc9e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd80b2960 0x7fadd80b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7faddc9e2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80ce990 con 0x7fadd80b2960 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7faddc9e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80d4ab0 con 0x7fadd81f7590 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7faddc9e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80b44d0 con 0x7fadc0003f30 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7fadd6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd80b2960 0x7fadd80b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7fadd6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd80b2960 0x7fadd80b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32816/0 (socket says 172.21.15.59:32816) 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7fadd6d9d640 1 -- 172.21.15.59:0/2270984677 learned_addr learned my addr 172.21.15.59:0/2270984677 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7fadd659c640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadc0003f30 0x7fadd80b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7fadd5d9b640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd81f7590 0x7fadd81f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7fadd6d9d640 1 -- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadc0003f30 msgr2=0x7fadd80b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7fadd6d9d640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadc0003f30 0x7fadd80b8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7fadd6d9d640 1 -- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd81f7590 msgr2=0x7fadd81f7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7fadd6d9d640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd81f7590 0x7fadd81f7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7fadd6d9d640 1 -- 172.21.15.59:0/2270984677 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80b8770 con 0x7fadd80b2960 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.358+0000 7fadd659c640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadc0003f30 0x7fadd80b8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.359+0000 7fadd6d9d640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd80b2960 0x7fadd80b2de0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fadd0011860 tx=0x7fadd0011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.359+0000 7fadc6ffd640 1 -- 172.21.15.59:0/2270984677 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0019020 con 0x7fadd80b2960 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.359+0000 7fadc6ffd640 1 -- 172.21.15.59:0/2270984677 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd0011e50 con 0x7fadd80b2960 2024-03-06T07:24:47.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.359+0000 7faddc9e2640 1 -- 172.21.15.59:0/2270984677 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd80b8a50 con 0x7fadd80b2960 2024-03-06T07:24:47.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.359+0000 7fadc6ffd640 1 -- 172.21.15.59:0/2270984677 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0004460 con 0x7fadd80b2960 2024-03-06T07:24:47.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.359+0000 7faddc9e2640 1 -- 172.21.15.59:0/2270984677 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd80b9010 con 0x7fadd80b2960 2024-03-06T07:24:47.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.360+0000 7faddc9e2640 1 -- 172.21.15.59:0/2270984677 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd80a1c80 con 0x7fadd80b2960 2024-03-06T07:24:47.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.360+0000 7fadc6ffd640 1 -- 172.21.15.59:0/2270984677 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fadd001c020 con 0x7fadd80b2960 2024-03-06T07:24:47.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.361+0000 7fadc6ffd640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadb806b500 0x7fadb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.361+0000 7fadc6ffd640 1 -- 172.21.15.59:0/2270984677 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadd008a4f0 con 0x7fadd80b2960 2024-03-06T07:24:47.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.361+0000 7fadd659c640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadb806b500 0x7fadb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.361+0000 7fadd659c640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadb806b500 0x7fadb806d9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fadc8001330 tx=0x7fadc8006040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.365+0000 7fadc6ffd640 1 -- 172.21.15.59:0/2270984677 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fadd0023020 con 0x7fadd80b2960 2024-03-06T07:24:47.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.544+0000 7faddc9e2640 1 -- 172.21.15.59:0/2270984677 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fadd8068ff0 con 0x7fadd80b2960 2024-03-06T07:24:47.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.544+0000 7fadc6ffd640 1 -- 172.21.15.59:0/2270984677 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7fadd0059400 con 0x7fadd80b2960 2024-03-06T07:24:47.547 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:24:47.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 -- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadb806b500 msgr2=0x7fadb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadb806b500 0x7fadb806d9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fadc8001330 tx=0x7fadc8006040 comp rx=0 tx=0).stop 2024-03-06T07:24:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 -- 172.21.15.59:0/2270984677 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd80b2960 msgr2=0x7fadd80b2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd80b2960 0x7fadd80b2de0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fadd0011860 tx=0x7fadd0011d20 comp rx=0 tx=0).stop 2024-03-06T07:24:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 -- 172.21.15.59:0/2270984677 shutdown_connections 2024-03-06T07:24:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadb806b500 0x7fadb806d9c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd80b2960 0x7fadd80b2de0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd81f7590 0x7fadd81f7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 --2- 172.21.15.59:0/2270984677 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadc0003f30 0x7fadd80b8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 -- 172.21.15.59:0/2270984677 >> 172.21.15.59:0/2270984677 conn(0x7fadd80d3cd0 msgr2=0x7fadd815df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 -- 172.21.15.59:0/2270984677 shutdown_connections 2024-03-06T07:24:47.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.546+0000 7faddc9e2640 1 -- 172.21.15.59:0/2270984677 wait complete. 2024-03-06T07:24:47.559 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:47.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.659+0000 7f9947d5e640 1 Processor -- start 2024-03-06T07:24:47.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.659+0000 7f9947d5e640 1 -- start start 2024-03-06T07:24:47.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f9947d5e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f99400b4460 0x7f99400d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f9947d5e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9940162970 0x7f99400b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f9947d5e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99400d22c0 con 0x7f9940161cf0 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f9947d5e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9940161b00 con 0x7f9940162970 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f9947d5e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99400cecc0 con 0x7f99400b4460 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f99452f9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9940162970 0x7f99400b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f99462fb640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9940161cf0 0x7f99400cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47336/0 (socket says 172.21.15.59:47336) 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f99462fb640 1 -- 172.21.15.59:0/1275634232 learned_addr learned my addr 172.21.15.59:0/1275634232 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f9945afa640 1 --2- 172.21.15.59:0/1275634232 >> v2:172.21.15.164:3300/0 conn(0x7f99400b4460 0x7f99400d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f99452f9640 1 -- 172.21.15.59:0/1275634232 >> v2:172.21.15.164:3300/0 conn(0x7f99400b4460 msgr2=0x7f99400d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f99452f9640 1 --2- 172.21.15.59:0/1275634232 >> v2:172.21.15.164:3300/0 conn(0x7f99400b4460 0x7f99400d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f99452f9640 1 -- 172.21.15.59:0/1275634232 >> v1:172.21.15.59:6789/0 conn(0x7f9940161cf0 legacy=0x7f99400cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f99452f9640 1 -- 172.21.15.59:0/1275634232 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99400d25a0 con 0x7f9940162970 2024-03-06T07:24:47.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.660+0000 7f9944af8640 1 -- 172.21.15.59:0/1275634232 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236979592 0 0) 0x7f99400d22c0 con 0x7f9940161cf0 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.661+0000 7f99452f9640 1 --2- 172.21.15.59:0/1275634232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9940162970 0x7f99400b3580 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f994008ddd0 tx=0x7f993002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3915e0aeb24d3d71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.661+0000 7f9944af8640 1 -- 172.21.15.59:0/1275634232 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930035020 con 0x7f9940162970 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.661+0000 7f9944af8640 1 -- 172.21.15.59:0/1275634232 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9930031830 con 0x7f9940162970 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.661+0000 7f9944af8640 1 -- 172.21.15.59:0/1275634232 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930031c00 con 0x7f9940162970 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.662+0000 7f9947d5e640 1 -- 172.21.15.59:0/1275634232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9940162970 msgr2=0x7f99400b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.662+0000 7f9947d5e640 1 --2- 172.21.15.59:0/1275634232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9940162970 0x7f99400b3580 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f994008ddd0 tx=0x7f993002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.662+0000 7f9947d5e640 1 -- 172.21.15.59:0/1275634232 shutdown_connections 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.662+0000 7f9947d5e640 1 --2- 172.21.15.59:0/1275634232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9940162970 0x7f99400b3580 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.662+0000 7f9947d5e640 1 --2- 172.21.15.59:0/1275634232 >> v2:172.21.15.164:3300/0 conn(0x7f99400b4460 0x7f99400d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.662+0000 7f9947d5e640 1 -- 172.21.15.59:0/1275634232 >> 172.21.15.59:0/1275634232 conn(0x7f99400d3ce0 msgr2=0x7f99400d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.662+0000 7f9947d5e640 1 -- 172.21.15.59:0/1275634232 shutdown_connections 2024-03-06T07:24:47.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.662+0000 7f9947d5e640 1 -- 172.21.15.59:0/1275634232 wait complete. 2024-03-06T07:24:47.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.664+0000 7f9947d5e640 1 Processor -- start 2024-03-06T07:24:47.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.664+0000 7f9947d5e640 1 -- start start 2024-03-06T07:24:47.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.664+0000 7f9947d5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99400b4460 0x7f99400b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.664+0000 7f9947d5e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9940161cf0 0x7f99400b75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.664+0000 7f9947d5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9940162970 0x7f99400b7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.664+0000 7f9947d5e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99400ce950 con 0x7f9940162970 2024-03-06T07:24:47.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.664+0000 7f9947d5e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99400d2b60 con 0x7f9940161cf0 2024-03-06T07:24:47.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.664+0000 7f9947d5e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99400b3ac0 con 0x7f99400b4460 2024-03-06T07:24:47.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.664+0000 7f99462fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9940162970 0x7f99400b7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.664+0000 7f99462fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9940162970 0x7f99400b7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32842/0 (socket says 172.21.15.59:32842) 2024-03-06T07:24:47.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.664+0000 7f99462fb640 1 -- 172.21.15.59:0/2877073686 learned_addr learned my addr 172.21.15.59:0/2877073686 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:47.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f9945afa640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99400b4460 0x7f99400b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f99462fb640 1 -- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99400b4460 msgr2=0x7f99400b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f99452f9640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9940161cf0 0x7f99400b75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f99462fb640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99400b4460 0x7f99400b1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f99462fb640 1 -- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9940161cf0 msgr2=0x7f99400b75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f99462fb640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9940161cf0 0x7f99400b75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f99462fb640 1 -- 172.21.15.59:0/2877073686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99401f49c0 con 0x7f9940162970 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f99462fb640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9940162970 0x7f99400b7b00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f993800c7d0 tx=0x7f993800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f99367fc640 1 -- 172.21.15.59:0/2877073686 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938018070 con 0x7f9940162970 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f99367fc640 1 -- 172.21.15.59:0/2877073686 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993800a640 con 0x7f9940162970 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f9947d5e640 1 -- 172.21.15.59:0/2877073686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99400cb000 con 0x7f9940162970 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f99367fc640 1 -- 172.21.15.59:0/2877073686 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993800aa70 con 0x7f9940162970 2024-03-06T07:24:47.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.665+0000 7f9947d5e640 1 -- 172.21.15.59:0/2877073686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99400cb5c0 con 0x7f9940162970 2024-03-06T07:24:47.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.666+0000 7f9947d5e640 1 -- 172.21.15.59:0/2877073686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99400cae10 con 0x7f9940162970 2024-03-06T07:24:47.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.667+0000 7f99367fc640 1 -- 172.21.15.59:0/2877073686 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f993801b070 con 0x7f9940162970 2024-03-06T07:24:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.667+0000 7f99367fc640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f992406b4d0 0x7f992406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.668+0000 7f99367fc640 1 -- 172.21.15.59:0/2877073686 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9938089790 con 0x7f9940162970 2024-03-06T07:24:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.668+0000 7f9945afa640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f992406b4d0 0x7f992406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.668+0000 7f9945afa640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f992406b4d0 0x7f992406d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f994006d6b0 tx=0x7f993c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.672+0000 7f99367fc640 1 -- 172.21.15.59:0/2877073686 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99380586a0 con 0x7f9940162970 2024-03-06T07:24:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.872+0000 7f9947d5e640 1 -- 172.21.15.59:0/2877073686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f994004f330 con 0x7f9940162970 2024-03-06T07:24:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.872+0000 7f99367fc640 1 -- 172.21.15.59:0/2877073686 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v59) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9938057fa0 con 0x7f9940162970 2024-03-06T07:24:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:24:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 -- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f992406b4d0 msgr2=0x7f992406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f992406b4d0 0x7f992406d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f994006d6b0 tx=0x7f993c008040 comp rx=0 tx=0).stop 2024-03-06T07:24:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 -- 172.21.15.59:0/2877073686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9940162970 msgr2=0x7f99400b7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9940162970 0x7f99400b7b00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f993800c7d0 tx=0x7f993800cc90 comp rx=0 tx=0).stop 2024-03-06T07:24:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 -- 172.21.15.59:0/2877073686 shutdown_connections 2024-03-06T07:24:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f992406b4d0 0x7f992406d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9940162970 0x7f99400b7b00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9940161cf0 0x7f99400b75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 --2- 172.21.15.59:0/2877073686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99400b4460 0x7f99400b1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 -- 172.21.15.59:0/2877073686 >> 172.21.15.59:0/2877073686 conn(0x7f99400d3ce0 msgr2=0x7f99400d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 -- 172.21.15.59:0/2877073686 shutdown_connections 2024-03-06T07:24:47.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:47.875+0000 7f9947d5e640 1 -- 172.21.15.59:0/2877073686 wait complete. 2024-03-06T07:24:47.887 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:24:47.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:47.887 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:47.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[59907]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:47.915 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:47.915 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:48.093 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:24:48.093 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:48.110 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:48.110 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:24:48.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:48.110 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:24:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:48.167 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:24:48.167 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:24:48.167 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:48.182 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:48.182 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:24:48.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:48.183 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:24:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:48.237 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:24:48.237 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:24:48.237 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:48.254 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:48.254 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:24:48.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:48.254 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:24:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:48.310 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:24:48.311 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:48.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.426+0000 7fd078781640 1 Processor -- start 2024-03-06T07:24:48.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.426+0000 7fd078781640 1 -- start start 2024-03-06T07:24:48.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.427+0000 7fd078781640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd07015fa10 0x7fd0700ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.427+0000 7fd078781640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd0700d16e0 0x7fd0700d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.427+0000 7fd078781640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0700cebe0 con 0x7fd07015bdd0 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.427+0000 7fd078781640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0700ceec0 con 0x7fd0700d16e0 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.427+0000 7fd078781640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0700cf1a0 con 0x7fd07015fa10 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.427+0000 7fd076d1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd0700d16e0 0x7fd0700d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.427+0000 7fd07651d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd07015bdd0 0x7fd0700c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47338/0 (socket says 172.21.15.59:47338) 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.427+0000 7fd07651d640 1 -- 172.21.15.59:0/2358149971 learned_addr learned my addr 172.21.15.59:0/2358149971 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.428+0000 7fd076d1e640 1 -- 172.21.15.59:0/2358149971 >> v2:172.21.15.164:3300/0 conn(0x7fd07015fa10 msgr2=0x7fd0700ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.428+0000 7fd075d1c640 1 --2- 172.21.15.59:0/2358149971 >> v2:172.21.15.164:3300/0 conn(0x7fd07015fa10 0x7fd0700ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.428+0000 7fd076d1e640 1 --2- 172.21.15.59:0/2358149971 >> v2:172.21.15.164:3300/0 conn(0x7fd07015fa10 0x7fd0700ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.428+0000 7fd076d1e640 1 -- 172.21.15.59:0/2358149971 >> v1:172.21.15.59:6789/0 conn(0x7fd07015bdd0 legacy=0x7fd0700c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.428+0000 7fd076d1e640 1 -- 172.21.15.59:0/2358149971 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0700cc8e0 con 0x7fd0700d16e0 2024-03-06T07:24:48.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.428+0000 7fd07551b640 1 -- 172.21.15.59:0/2358149971 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159028734 0 0) 0x7fd0700cebe0 con 0x7fd07015bdd0 2024-03-06T07:24:48.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.428+0000 7fd076d1e640 1 --2- 172.21.15.59:0/2358149971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0700d16e0 0x7fd0700d1af0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd0700cdfd0 tx=0x7fd06802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75b2186c282e15b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.428+0000 7fd07551b640 1 -- 172.21.15.59:0/2358149971 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd068039070 con 0x7fd0700d16e0 2024-03-06T07:24:48.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.429+0000 7fd07551b640 1 -- 172.21.15.59:0/2358149971 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06802cbf0 con 0x7fd0700d16e0 2024-03-06T07:24:48.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.429+0000 7fd07551b640 1 -- 172.21.15.59:0/2358149971 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd068034440 con 0x7fd0700d16e0 2024-03-06T07:24:48.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.429+0000 7fd078781640 1 -- 172.21.15.59:0/2358149971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0700d16e0 msgr2=0x7fd0700d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.429+0000 7fd078781640 1 --2- 172.21.15.59:0/2358149971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0700d16e0 0x7fd0700d1af0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd0700cdfd0 tx=0x7fd06802be40 comp rx=0 tx=0).stop 2024-03-06T07:24:48.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.429+0000 7fd078781640 1 -- 172.21.15.59:0/2358149971 shutdown_connections 2024-03-06T07:24:48.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.429+0000 7fd078781640 1 --2- 172.21.15.59:0/2358149971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0700d16e0 0x7fd0700d1af0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.429+0000 7fd078781640 1 --2- 172.21.15.59:0/2358149971 >> v2:172.21.15.164:3300/0 conn(0x7fd07015fa10 0x7fd0700ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.429+0000 7fd078781640 1 -- 172.21.15.59:0/2358149971 >> 172.21.15.59:0/2358149971 conn(0x7fd07009a9f0 msgr2=0x7fd07009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.429+0000 7fd078781640 1 -- 172.21.15.59:0/2358149971 shutdown_connections 2024-03-06T07:24:48.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.429+0000 7fd078781640 1 -- 172.21.15.59:0/2358149971 wait complete. 2024-03-06T07:24:48.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.430+0000 7fd078781640 1 Processor -- start 2024-03-06T07:24:48.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.430+0000 7fd078781640 1 -- start start 2024-03-06T07:24:48.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd078781640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0700d16e0 0x7fd07015e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd078781640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07015bdd0 0x7fd0700c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd078781640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07015fa10 0x7fd070152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd078781640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700d2030 con 0x7fd07015bdd0 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd078781640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07015c1b0 con 0x7fd0700d16e0 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd078781640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700ce7b0 con 0x7fd07015fa10 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd075d1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07015bdd0 0x7fd0700c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd075d1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07015bdd0 0x7fd0700c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32864/0 (socket says 172.21.15.59:32864) 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd075d1c640 1 -- 172.21.15.59:0/237195197 learned_addr learned my addr 172.21.15.59:0/237195197 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd07651d640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0700d16e0 0x7fd07015e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd076d1e640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07015fa10 0x7fd070152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd075d1c640 1 -- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07015fa10 msgr2=0x7fd070152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd075d1c640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07015fa10 0x7fd070152310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd075d1c640 1 -- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0700d16e0 msgr2=0x7fd07015e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd075d1c640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0700d16e0 0x7fd07015e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd075d1c640 1 -- 172.21.15.59:0/237195197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07015f220 con 0x7fd07015bdd0 2024-03-06T07:24:48.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.431+0000 7fd07651d640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0700d16e0 0x7fd07015e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:48.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.432+0000 7fd075d1c640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07015bdd0 0x7fd0700c8b90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c820 tx=0x7fd06c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.432+0000 7fd062ffd640 1 -- 172.21.15.59:0/237195197 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c004030 con 0x7fd07015bdd0 2024-03-06T07:24:48.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.432+0000 7fd062ffd640 1 -- 172.21.15.59:0/237195197 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c0041d0 con 0x7fd07015bdd0 2024-03-06T07:24:48.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.432+0000 7fd078781640 1 -- 172.21.15.59:0/237195197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07015ef40 con 0x7fd07015bdd0 2024-03-06T07:24:48.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.432+0000 7fd062ffd640 1 -- 172.21.15.59:0/237195197 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c007830 con 0x7fd07015bdd0 2024-03-06T07:24:48.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.432+0000 7fd078781640 1 -- 172.21.15.59:0/237195197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0701537a0 con 0x7fd07015bdd0 2024-03-06T07:24:48.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.433+0000 7fd078781640 1 -- 172.21.15.59:0/237195197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07004f330 con 0x7fd07015bdd0 2024-03-06T07:24:48.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.434+0000 7fd062ffd640 1 -- 172.21.15.59:0/237195197 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd06c0079d0 con 0x7fd07015bdd0 2024-03-06T07:24:48.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.434+0000 7fd062ffd640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd05406b500 0x7fd05406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.434+0000 7fd062ffd640 1 -- 172.21.15.59:0/237195197 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd06c08a110 con 0x7fd07015bdd0 2024-03-06T07:24:48.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.434+0000 7fd07651d640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd05406b500 0x7fd05406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.435+0000 7fd07651d640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd05406b500 0x7fd05406d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd064007fd0 tx=0x7fd064009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.438+0000 7fd062ffd640 1 -- 172.21.15.59:0/237195197 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd06c015020 con 0x7fd07015bdd0 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.660+0000 7fd078781640 1 -- 172.21.15.59:0/237195197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0700b7a40 con 0x7fd07015bdd0 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.660+0000 7fd062ffd640 1 -- 172.21.15.59:0/237195197 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fd06c058f90 con 0x7fd07015bdd0 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:48.662 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2205562350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2205562350},{"type":"v1","addr":"172.21.15.164:6837","nonce":2205562350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2606750481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2606750481},{"type":"v1","addr":"172.21.15.164:6836","nonce":2606750481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6048,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3685052110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3685052110},{"type":"v1","addr":"172.21.15.59:6833","nonce":3685052110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:11.010795+0000","modified":"2024-03-06T07:24:13.053115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/750409746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":750409746},{"type":"v1","addr":"172.21.15.59:6835","nonce":750409746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":4}]} 2024-03-06T07:24:48.663 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 28 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 -- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd05406b500 msgr2=0x7fd05406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd05406b500 0x7fd05406d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd064007fd0 tx=0x7fd064009040 comp rx=0 tx=0).stop 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 -- 172.21.15.59:0/237195197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07015bdd0 msgr2=0x7fd0700c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07015bdd0 0x7fd0700c8b90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c820 tx=0x7fd06c00cce0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 -- 172.21.15.59:0/237195197 shutdown_connections 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd05406b500 0x7fd05406d9c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07015fa10 0x7fd070152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07015bdd0 0x7fd0700c8b90 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 --2- 172.21.15.59:0/237195197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0700d16e0 0x7fd07015e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 -- 172.21.15.59:0/237195197 >> 172.21.15.59:0/237195197 conn(0x7fd07009a9f0 msgr2=0x7fd0700cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 -- 172.21.15.59:0/237195197 shutdown_connections 2024-03-06T07:24:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.662+0000 7fd078781640 1 -- 172.21.15.59:0/237195197 wait complete. 2024-03-06T07:24:48.676 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.775+0000 7f6fd9c8f640 1 Processor -- start 2024-03-06T07:24:48.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.775+0000 7f6fd9c8f640 1 -- start start 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.776+0000 7f6fd9c8f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6fd40d22b0 0x7f6fd40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.776+0000 7f6fd9c8f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd40b41f0 0x7f6fd40b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fd9c8f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd40b3430 con 0x7f6fd415dfe0 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fd9c8f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd40d2c50 con 0x7f6fd40d22b0 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fd9c8f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40b31e0 con 0x7f6fd40b41f0 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fd37fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6fd415dfe0 0x7f6fd40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47344/0 (socket says 172.21.15.59:47344) 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fcbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6fd40d22b0 0x7f6fd40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fd37fe640 1 -- 172.21.15.59:0/1270026378 learned_addr learned my addr 172.21.15.59:0/1270026378 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fd3fff640 1 --2- 172.21.15.59:0/1270026378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd40b41f0 0x7f6fd40b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fcbfff640 1 -- 172.21.15.59:0/1270026378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd40b41f0 msgr2=0x7f6fd40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fcbfff640 1 --2- 172.21.15.59:0/1270026378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd40b41f0 0x7f6fd40b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fcbfff640 1 -- 172.21.15.59:0/1270026378 >> v1:172.21.15.59:6789/0 conn(0x7f6fd415dfe0 legacy=0x7f6fd40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fcbfff640 1 -- 172.21.15.59:0/1270026378 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40b3710 con 0x7f6fd40d22b0 2024-03-06T07:24:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.777+0000 7f6fd2ffd640 1 -- 172.21.15.59:0/1270026378 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931845006 0 0) 0x7f6fd40b3430 con 0x7f6fd415dfe0 2024-03-06T07:24:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.778+0000 7f6fcbfff640 1 --2- 172.21.15.59:0/1270026378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fd40d22b0 0x7f6fd40d2710 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0097a0 tx=0x7f6fbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a32e5c63e824ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.778+0000 7f6fd2ffd640 1 -- 172.21.15.59:0/1270026378 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc035020 con 0x7f6fd40d22b0 2024-03-06T07:24:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.778+0000 7f6fd2ffd640 1 -- 172.21.15.59:0/1270026378 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc031830 con 0x7f6fd40d22b0 2024-03-06T07:24:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.778+0000 7f6fd2ffd640 1 -- 172.21.15.59:0/1270026378 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc031c00 con 0x7f6fd40d22b0 2024-03-06T07:24:48.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.778+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/1270026378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fd40d22b0 msgr2=0x7f6fd40d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.778+0000 7f6fd9c8f640 1 --2- 172.21.15.59:0/1270026378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fd40d22b0 0x7f6fd40d2710 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0097a0 tx=0x7f6fbc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.778+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/1270026378 shutdown_connections 2024-03-06T07:24:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.778+0000 7f6fd9c8f640 1 --2- 172.21.15.59:0/1270026378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd40b41f0 0x7f6fd40b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.778+0000 7f6fd9c8f640 1 --2- 172.21.15.59:0/1270026378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fd40d22b0 0x7f6fd40d2710 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.778+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/1270026378 >> 172.21.15.59:0/1270026378 conn(0x7f6fd40d3cd0 msgr2=0x7f6fd40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.778+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/1270026378 shutdown_connections 2024-03-06T07:24:48.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.779+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/1270026378 wait complete. 2024-03-06T07:24:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.780+0000 7f6fd9c8f640 1 Processor -- start 2024-03-06T07:24:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.780+0000 7f6fd9c8f640 1 -- start start 2024-03-06T07:24:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.780+0000 7f6fd9c8f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fd40b41f0 0x7f6fd40b9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.780+0000 7f6fd9c8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fd40d22b0 0x7f6fd40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.780+0000 7f6fd9c8f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd415dfe0 0x7f6fd415b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.780+0000 7f6fd9c8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40cfa00 con 0x7f6fd40d22b0 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.780+0000 7f6fd9c8f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40b3d60 con 0x7f6fd40b41f0 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.780+0000 7f6fd9c8f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd40d1c70 con 0x7f6fd415dfe0 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.780+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fd40d22b0 0x7f6fd40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.780+0000 7f6fcbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fd40d22b0 0x7f6fd40cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32892/0 (socket says 172.21.15.59:32892) 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.780+0000 7f6fcbfff640 1 -- 172.21.15.59:0/2744623896 learned_addr learned my addr 172.21.15.59:0/2744623896 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fd3fff640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd415dfe0 0x7f6fd415b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fd37fe640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fd40b41f0 0x7f6fd40b9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fcbfff640 1 -- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd415dfe0 msgr2=0x7f6fd415b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fcbfff640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd415dfe0 0x7f6fd415b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fcbfff640 1 -- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fd40b41f0 msgr2=0x7f6fd40b9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fcbfff640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fd40b41f0 0x7f6fd40b9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fcbfff640 1 -- 172.21.15.59:0/2744623896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40cb510 con 0x7f6fd40d22b0 2024-03-06T07:24:48.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fcbfff640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fd40d22b0 0x7f6fd40cada0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc02bee0 tx=0x7f6fbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fd0ff9640 1 -- 172.21.15.59:0/2744623896 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc041070 con 0x7f6fd40d22b0 2024-03-06T07:24:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fd0ff9640 1 -- 172.21.15.59:0/2744623896 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc03a950 con 0x7f6fd40d22b0 2024-03-06T07:24:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/2744623896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd40cb7f0 con 0x7f6fd40d22b0 2024-03-06T07:24:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fd0ff9640 1 -- 172.21.15.59:0/2744623896 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc035040 con 0x7f6fd40d22b0 2024-03-06T07:24:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.781+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/2744623896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd40cc330 con 0x7f6fd40d22b0 2024-03-06T07:24:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.782+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/2744623896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd40cbd10 con 0x7f6fd40d22b0 2024-03-06T07:24:48.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.783+0000 7f6fd0ff9640 1 -- 172.21.15.59:0/2744623896 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fbc04c060 con 0x7f6fd40d22b0 2024-03-06T07:24:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.783+0000 7f6fd0ff9640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fb406b500 0x7f6fb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.783+0000 7f6fd0ff9640 1 -- 172.21.15.59:0/2744623896 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fbc0abb80 con 0x7f6fd40d22b0 2024-03-06T07:24:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.783+0000 7f6fd37fe640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fb406b500 0x7f6fb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:48.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.784+0000 7f6fd37fe640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fb406b500 0x7f6fb406d9c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4000f60 tx=0x7f6fc4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:48.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.787+0000 7f6fd0ff9640 1 -- 172.21.15.59:0/2744623896 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fd40cbd10 con 0x7f6fd40d22b0 2024-03-06T07:24:48.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.970+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/2744623896 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fd404f330 con 0x7f6fd40d22b0 2024-03-06T07:24:48.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.971+0000 7f6fd0ff9640 1 -- 172.21.15.59:0/2744623896 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16653 (secure 0 0 0) 0x7f6fbc07aa90 con 0x7f6fd40d22b0 2024-03-06T07:24:48.973 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:48.974 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:24:12.652576+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:09.288329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.978574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fb406b500 msgr2=0x7f6fb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fb406b500 0x7f6fb406d9c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4000f60 tx=0x7f6fc4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/2744623896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fd40d22b0 msgr2=0x7f6fd40cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fd40d22b0 0x7f6fd40cada0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc02bee0 tx=0x7f6fbc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/2744623896 shutdown_connections 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6fb406b500 0x7f6fb406d9c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fd415dfe0 0x7f6fd415b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fd40d22b0 0x7f6fd40cada0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 --2- 172.21.15.59:0/2744623896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fd40b41f0 0x7f6fd40b9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/2744623896 >> 172.21.15.59:0/2744623896 conn(0x7f6fd40d3cd0 msgr2=0x7f6fd406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/2744623896 shutdown_connections 2024-03-06T07:24:48.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:48.973+0000 7f6fd9c8f640 1 -- 172.21.15.59:0/2744623896 wait complete. 2024-03-06T07:24:48.986 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:09.288329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:09.978574+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:48.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:49.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.089+0000 7f4809155640 1 Processor -- start 2024-03-06T07:24:49.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.089+0000 7f4809155640 1 -- start start 2024-03-06T07:24:49.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f4809155640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f480415c050 0x7f480406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f4809155640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f480406fb40 0x7f48040cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f4809155640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48040ccca0 0x7f48040cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f4809155640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f480406ffb0 con 0x7f480406fb40 2024-03-06T07:24:49.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f4809155640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48040cd4b0 con 0x7f480415c050 2024-03-06T07:24:49.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f4809155640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48040cd790 con 0x7f48040ccca0 2024-03-06T07:24:49.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f47fbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f480406fb40 0x7f48040cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f47fbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f480406fb40 0x7f48040cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32908/0 (socket says 172.21.15.59:32908) 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f47fbfff640 1 -- 172.21.15.59:0/3204791332 learned_addr learned my addr 172.21.15.59:0/3204791332 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f480359e640 1 --2- 172.21.15.59:0/3204791332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48040ccca0 0x7f48040cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f4802d9d640 1 --2- 172.21.15.59:0/3204791332 >> v2:172.21.15.164:3300/0 conn(0x7f480415c050 0x7f480406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f47fbfff640 1 -- 172.21.15.59:0/3204791332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48040ccca0 msgr2=0x7f48040cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f47fbfff640 1 --2- 172.21.15.59:0/3204791332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48040ccca0 0x7f48040cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f47fbfff640 1 -- 172.21.15.59:0/3204791332 >> v2:172.21.15.164:3300/0 conn(0x7f480415c050 msgr2=0x7f480406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f47fbfff640 1 --2- 172.21.15.59:0/3204791332 >> v2:172.21.15.164:3300/0 conn(0x7f480415c050 0x7f480406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.090+0000 7f47fbfff640 1 -- 172.21.15.59:0/3204791332 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48040cda70 con 0x7f480406fb40 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.091+0000 7f47fbfff640 1 --2- 172.21.15.59:0/3204791332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f480406fb40 0x7f48040cc760 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4804070ef0 tx=0x7f47ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4313b76a7af622 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.091+0000 7f480259c640 1 -- 172.21.15.59:0/3204791332 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec02e070 con 0x7f480406fb40 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.091+0000 7f480259c640 1 -- 172.21.15.59:0/3204791332 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec037070 con 0x7f480406fb40 2024-03-06T07:24:49.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.091+0000 7f480259c640 1 -- 172.21.15.59:0/3204791332 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec031a00 con 0x7f480406fb40 2024-03-06T07:24:49.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.092+0000 7f4809155640 1 -- 172.21.15.59:0/3204791332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f480406fb40 msgr2=0x7f48040cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.092+0000 7f4809155640 1 --2- 172.21.15.59:0/3204791332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f480406fb40 0x7f48040cc760 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4804070ef0 tx=0x7f47ec02b880 comp rx=0 tx=0).stop 2024-03-06T07:24:49.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.092+0000 7f4809155640 1 -- 172.21.15.59:0/3204791332 shutdown_connections 2024-03-06T07:24:49.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.092+0000 7f4809155640 1 --2- 172.21.15.59:0/3204791332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48040ccca0 0x7f48040cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.092+0000 7f4809155640 1 --2- 172.21.15.59:0/3204791332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f480406fb40 0x7f48040cc760 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.092+0000 7f4809155640 1 --2- 172.21.15.59:0/3204791332 >> v2:172.21.15.164:3300/0 conn(0x7f480415c050 0x7f480406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.092+0000 7f4809155640 1 -- 172.21.15.59:0/3204791332 >> 172.21.15.59:0/3204791332 conn(0x7f48040b3800 msgr2=0x7f48040b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.092+0000 7f4809155640 1 -- 172.21.15.59:0/3204791332 shutdown_connections 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.092+0000 7f4809155640 1 -- 172.21.15.59:0/3204791332 wait complete. 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.093+0000 7f4809155640 1 Processor -- start 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.093+0000 7f4809155640 1 -- start start 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.094+0000 7f4809155640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f480406fb40 0x7f48041e5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.094+0000 7f4809155640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040ccca0 0x7f4804144040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.094+0000 7f4809155640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f480415c050 0x7f4804144580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.094+0000 7f4809155640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48040d0020 con 0x7f48040ccca0 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.094+0000 7f4809155640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48040cd130 con 0x7f480406fb40 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.094+0000 7f4809155640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48040d01a0 con 0x7f480415c050 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.094+0000 7f47fbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040ccca0 0x7f4804144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.094+0000 7f47fbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040ccca0 0x7f4804144040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32914/0 (socket says 172.21.15.59:32914) 2024-03-06T07:24:49.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.094+0000 7f47fbfff640 1 -- 172.21.15.59:0/1790581142 learned_addr learned my addr 172.21.15.59:0/1790581142 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.094+0000 7f480359e640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f480415c050 0x7f4804144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.094+0000 7f4802d9d640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f480406fb40 0x7f48041e5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.095+0000 7f47fbfff640 1 -- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f480415c050 msgr2=0x7f4804144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.095+0000 7f47fbfff640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f480415c050 0x7f4804144580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.095+0000 7f47fbfff640 1 -- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f480406fb40 msgr2=0x7f48041e5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.095+0000 7f47fbfff640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f480406fb40 0x7f48041e5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.095+0000 7f47fbfff640 1 -- 172.21.15.59:0/1790581142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4804143030 con 0x7f48040ccca0 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.095+0000 7f47fbfff640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040ccca0 0x7f4804144040 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f47ec000c00 tx=0x7f47ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.095+0000 7f47fb7fe640 1 -- 172.21.15.59:0/1790581142 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec032670 con 0x7f48040ccca0 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.095+0000 7f47fb7fe640 1 -- 172.21.15.59:0/1790581142 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec032850 con 0x7f48040ccca0 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.095+0000 7f4809155640 1 -- 172.21.15.59:0/1790581142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4804143310 con 0x7f48040ccca0 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.095+0000 7f47fb7fe640 1 -- 172.21.15.59:0/1790581142 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec032c40 con 0x7f48040ccca0 2024-03-06T07:24:49.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.095+0000 7f4809155640 1 -- 172.21.15.59:0/1790581142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48041438d0 con 0x7f48040ccca0 2024-03-06T07:24:49.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.096+0000 7f4809155640 1 -- 172.21.15.59:0/1790581142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f480409fe60 con 0x7f48040ccca0 2024-03-06T07:24:49.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.096+0000 7f47fb7fe640 1 -- 172.21.15.59:0/1790581142 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f47ec0453f0 con 0x7f48040ccca0 2024-03-06T07:24:49.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.097+0000 7f47fb7fe640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f47e406b4d0 0x7f47e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.097+0000 7f47fb7fe640 1 -- 172.21.15.59:0/1790581142 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47ec0ac090 con 0x7f48040ccca0 2024-03-06T07:24:49.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.097+0000 7f4802d9d640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f47e406b4d0 0x7f47e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.098+0000 7f4802d9d640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f47e406b4d0 0x7f47e406d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f47f4006fd0 tx=0x7f47f4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.101+0000 7f47fb7fe640 1 -- 172.21.15.59:0/1790581142 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f47ec040070 con 0x7f48040ccca0 2024-03-06T07:24:49.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.326+0000 7f4809155640 1 -- 172.21.15.59:0/1790581142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f48040b0c60 con 0x7f48040ccca0 2024-03-06T07:24:49.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.326+0000 7f47fb7fe640 1 -- 172.21.15.59:0/1790581142 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f47ec07ae70 con 0x7f48040ccca0 2024-03-06T07:24:49.328 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:24:49.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 -- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f47e406b4d0 msgr2=0x7f47e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f47e406b4d0 0x7f47e406d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f47f4006fd0 tx=0x7f47f4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 -- 172.21.15.59:0/1790581142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040ccca0 msgr2=0x7f4804144040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040ccca0 0x7f4804144040 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f47ec000c00 tx=0x7f47ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 -- 172.21.15.59:0/1790581142 shutdown_connections 2024-03-06T07:24:49.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f47e406b4d0 0x7f47e406d990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f480415c050 0x7f4804144580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48040ccca0 0x7f4804144040 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 --2- 172.21.15.59:0/1790581142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f480406fb40 0x7f48041e5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 -- 172.21.15.59:0/1790581142 >> 172.21.15.59:0/1790581142 conn(0x7f48040b3800 msgr2=0x7f48040aebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 -- 172.21.15.59:0/1790581142 shutdown_connections 2024-03-06T07:24:49.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.328+0000 7f4809155640 1 -- 172.21.15.59:0/1790581142 wait complete. 2024-03-06T07:24:49.341 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:49.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.444+0000 7f4323193640 1 Processor -- start 2024-03-06T07:24:49.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.444+0000 7f4323193640 1 -- start start 2024-03-06T07:24:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f4323193640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f431c0d4860 0x7f431c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f4323193640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f431c0b0e30 0x7f431c0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f4323193640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c1618b0 con 0x7f431c1614d0 2024-03-06T07:24:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f4323193640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431c0d4cf0 con 0x7f431c0d4860 2024-03-06T07:24:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f4323193640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431c0cf400 con 0x7f431c0b0e30 2024-03-06T07:24:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f431bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f431c0d4860 0x7f431c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f4320f2f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f431c1614d0 0x7f431c0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47346/0 (socket says 172.21.15.59:47346) 2024-03-06T07:24:49.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f431bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f431c0d4860 0x7f431c0d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32930/0 (socket says 172.21.15.59:32930) 2024-03-06T07:24:49.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f4320f2f640 1 -- 172.21.15.59:0/3142742107 learned_addr learned my addr 172.21.15.59:0/3142742107 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:49.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f4321730640 1 --2- 172.21.15.59:0/3142742107 >> v2:172.21.15.164:3300/0 conn(0x7f431c0b0e30 0x7f431c0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f431bfff640 1 -- 172.21.15.59:0/3142742107 >> v2:172.21.15.164:3300/0 conn(0x7f431c0b0e30 msgr2=0x7f431c0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f431bfff640 1 --2- 172.21.15.59:0/3142742107 >> v2:172.21.15.164:3300/0 conn(0x7f431c0b0e30 0x7f431c0b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f431bfff640 1 -- 172.21.15.59:0/3142742107 >> v1:172.21.15.59:6789/0 conn(0x7f431c1614d0 legacy=0x7f431c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.445+0000 7f431bfff640 1 -- 172.21.15.59:0/3142742107 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c0cf6a0 con 0x7f431c0d4860 2024-03-06T07:24:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.446+0000 7f431bfff640 1 --2- 172.21.15.59:0/3142742107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f431c0d4860 0x7f431c0d2020 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f431c0b2a70 tx=0x7f430802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d45cd8dce5dc6c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.446+0000 7f431b7fe640 1 -- 172.21.15.59:0/3142742107 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308035020 con 0x7f431c0d4860 2024-03-06T07:24:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.446+0000 7f431b7fe640 1 -- 172.21.15.59:0/3142742107 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4308031830 con 0x7f431c0d4860 2024-03-06T07:24:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.446+0000 7f431b7fe640 1 -- 172.21.15.59:0/3142742107 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308031c00 con 0x7f431c0d4860 2024-03-06T07:24:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.447+0000 7f4323193640 1 -- 172.21.15.59:0/3142742107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f431c0d4860 msgr2=0x7f431c0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.447+0000 7f4323193640 1 --2- 172.21.15.59:0/3142742107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f431c0d4860 0x7f431c0d2020 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f431c0b2a70 tx=0x7f430802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.447+0000 7f4323193640 1 -- 172.21.15.59:0/3142742107 shutdown_connections 2024-03-06T07:24:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.447+0000 7f4323193640 1 --2- 172.21.15.59:0/3142742107 >> v2:172.21.15.164:3300/0 conn(0x7f431c0b0e30 0x7f431c0b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.447+0000 7f4323193640 1 --2- 172.21.15.59:0/3142742107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f431c0d4860 0x7f431c0d2020 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.447+0000 7f4323193640 1 -- 172.21.15.59:0/3142742107 >> 172.21.15.59:0/3142742107 conn(0x7f431c0d3d40 msgr2=0x7f431c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.447+0000 7f4323193640 1 -- 172.21.15.59:0/3142742107 shutdown_connections 2024-03-06T07:24:49.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.447+0000 7f4323193640 1 -- 172.21.15.59:0/3142742107 wait complete. 2024-03-06T07:24:49.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.448+0000 7f4323193640 1 Processor -- start 2024-03-06T07:24:49.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.448+0000 7f4323193640 1 -- start start 2024-03-06T07:24:49.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f4323193640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f431c0b0e30 0x7f431c0b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f4323193640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f431c0d4860 0x7f431c159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f4323193640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f431c1614d0 0x7f431c1e9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f4323193640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c0b0910 con 0x7f431c0d4860 2024-03-06T07:24:49.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f4323193640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c0cfc60 con 0x7f431c1614d0 2024-03-06T07:24:49.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f4323193640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431c0d2560 con 0x7f431c0b0e30 2024-03-06T07:24:49.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f431bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f431c0d4860 0x7f431c159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f431bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f431c0d4860 0x7f431c159df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32942/0 (socket says 172.21.15.59:32942) 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f431bfff640 1 -- 172.21.15.59:0/2198967532 learned_addr learned my addr 172.21.15.59:0/2198967532 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f4321730640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f431c1614d0 0x7f431c1e9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f4320f2f640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f431c0b0e30 0x7f431c0b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f431bfff640 1 -- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f431c0b0e30 msgr2=0x7f431c0b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f431bfff640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f431c0b0e30 0x7f431c0b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f431bfff640 1 -- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f431c1614d0 msgr2=0x7f431c1e9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f431bfff640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f431c1614d0 0x7f431c1e9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f431bfff640 1 -- 172.21.15.59:0/2198967532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c0cb310 con 0x7f431c0d4860 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.449+0000 7f431bfff640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f431c0d4860 0x7f431c159df0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4308009890 tx=0x7f43080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.450+0000 7f43197fa640 1 -- 172.21.15.59:0/2198967532 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308041070 con 0x7f431c0d4860 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.450+0000 7f43197fa640 1 -- 172.21.15.59:0/2198967532 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430803aa30 con 0x7f431c0d4860 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.450+0000 7f4323193640 1 -- 172.21.15.59:0/2198967532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431c0cb5f0 con 0x7f431c0d4860 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.450+0000 7f43197fa640 1 -- 172.21.15.59:0/2198967532 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308035040 con 0x7f431c0d4860 2024-03-06T07:24:49.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.450+0000 7f4323193640 1 -- 172.21.15.59:0/2198967532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431c0cbbb0 con 0x7f431c0d4860 2024-03-06T07:24:49.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.451+0000 7f4323193640 1 -- 172.21.15.59:0/2198967532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431c068ff0 con 0x7f431c0d4860 2024-03-06T07:24:49.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.451+0000 7f43197fa640 1 -- 172.21.15.59:0/2198967532 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4308038020 con 0x7f431c0d4860 2024-03-06T07:24:49.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.451+0000 7f43197fa640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f430006b4d0 0x7f430006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.452+0000 7f43197fa640 1 -- 172.21.15.59:0/2198967532 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43080abc60 con 0x7f431c0d4860 2024-03-06T07:24:49.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.452+0000 7f4320f2f640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f430006b4d0 0x7f430006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.452+0000 7f4320f2f640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f430006b4d0 0x7f430006d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4310000f60 tx=0x7f4310009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.456+0000 7f43197fa640 1 -- 172.21.15.59:0/2198967532 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f430807aae0 con 0x7f431c0d4860 2024-03-06T07:24:49.466 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:49.468 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:49.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.679+0000 7f4323193640 1 -- 172.21.15.59:0/2198967532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f431c0d0fd0 con 0x7f431c0d4860 2024-03-06T07:24:49.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.680+0000 7f43197fa640 1 -- 172.21.15.59:0/2198967532 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7f430807a3e0 con 0x7f431c0d4860 2024-03-06T07:24:49.682 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.681+0000 7f4323193640 1 -- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f430006b4d0 msgr2=0x7f430006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.681+0000 7f4323193640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f430006b4d0 0x7f430006d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4310000f60 tx=0x7f4310009040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.681+0000 7f4323193640 1 -- 172.21.15.59:0/2198967532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f431c0d4860 msgr2=0x7f431c159df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.681+0000 7f4323193640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f431c0d4860 0x7f431c159df0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4308009890 tx=0x7f43080046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.682+0000 7f4323193640 1 -- 172.21.15.59:0/2198967532 shutdown_connections 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.682+0000 7f4323193640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f430006b4d0 0x7f430006d990 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.682+0000 7f4323193640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f431c1614d0 0x7f431c1e9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.682+0000 7f4323193640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f431c0d4860 0x7f431c159df0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.682+0000 7f4323193640 1 --2- 172.21.15.59:0/2198967532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f431c0b0e30 0x7f431c0b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.682+0000 7f4323193640 1 -- 172.21.15.59:0/2198967532 >> 172.21.15.59:0/2198967532 conn(0x7f431c0d3d40 msgr2=0x7f431c0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.682+0000 7f4323193640 1 -- 172.21.15.59:0/2198967532 shutdown_connections 2024-03-06T07:24:49.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.682+0000 7f4323193640 1 -- 172.21.15.59:0/2198967532 wait complete. 2024-03-06T07:24:49.694 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:49.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.811+0000 7f335c678640 1 Processor -- start 2024-03-06T07:24:49.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.811+0000 7f335c678640 1 -- start start 2024-03-06T07:24:49.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f335c678640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f335406efd0 0x7f335406f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f335c678640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33540cc5c0 0x7f33540cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f335c678640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f335406b6c0 con 0x7f3354145590 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f335c678640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f335406f970 con 0x7f335406efd0 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f335c678640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3354145970 con 0x7f33540cc5c0 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f335a414640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3354145590 0x7f335406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47360/0 (socket says 172.21.15.59:47360) 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f335a414640 1 -- 172.21.15.59:0/1909716520 learned_addr learned my addr 172.21.15.59:0/1909716520 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880586269 0 0) 0x7f335406b6c0 con 0x7f3354145590 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f3359c13640 1 --2- 172.21.15.59:0/1909716520 >> v2:172.21.15.164:3300/0 conn(0x7f335406efd0 0x7f335406f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3340003520 con 0x7f3354145590 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f335ac15640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33540cc5c0 0x7f33540cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 564791236 0 0) 0x7f3340003520 con 0x7f3354145590 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33540cc5c0 msgr2=0x7f33540cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f3359412640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33540cc5c0 0x7f33540cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 >> v2:172.21.15.164:3300/0 conn(0x7f335406efd0 msgr2=0x7f335406f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f3359412640 1 --2- 172.21.15.59:0/1909716520 >> v2:172.21.15.164:3300/0 conn(0x7f335406efd0 0x7f335406f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.812+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335406b970 con 0x7f3354145590 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f3350001670 con 0x7f3354145590 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 >> v1:172.21.15.59:6789/0 conn(0x7f3354145590 legacy=0x7f335406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f3359412640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33400039b0 0x7f3340003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f3359412640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3340004800 0x7f3340004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f3359412640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33540cc5c0 0x7f33400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3354145970 con 0x7f33400039b0 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334000b9c0 con 0x7f3340004800 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334000bb40 con 0x7f33540cc5c0 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f335a414640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33400039b0 0x7f3340003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f3359c13640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3340004800 0x7f3340004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f335ac15640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33540cc5c0 0x7f33400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f335a414640 1 -- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33540cc5c0 msgr2=0x7f33400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f335a414640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33540cc5c0 0x7f33400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f335a414640 1 -- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3340004800 msgr2=0x7f3340004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f335a414640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3340004800 0x7f3340004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f335a414640 1 -- 172.21.15.59:0/1909716520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3340003520 con 0x7f33400039b0 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.813+0000 7f335ac15640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33540cc5c0 0x7f33400051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.814+0000 7f335a414640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33400039b0 0x7f3340003dd0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f335000c240 tx=0x7f3340005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430509a9f367d2d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.814+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3350007d30 con 0x7f33400039b0 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.814+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3350030070 con 0x7f33400039b0 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.814+0000 7f3359412640 1 -- 172.21.15.59:0/1909716520 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335002cd40 con 0x7f33400039b0 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.814+0000 7f335c678640 1 -- 172.21.15.59:0/1909716520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33400039b0 msgr2=0x7f3340003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.814+0000 7f335c678640 1 --2- 172.21.15.59:0/1909716520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33400039b0 0x7f3340003dd0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f335000c240 tx=0x7f3340005830 comp rx=0 tx=0).stop 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.814+0000 7f335ac15640 1 -- 172.21.15.59:0/1909716520 reap_dead start 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.815+0000 7f335c678640 1 -- 172.21.15.59:0/1909716520 shutdown_connections 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.815+0000 7f335c678640 1 -- 172.21.15.59:0/1909716520 >> 172.21.15.59:0/1909716520 conn(0x7f33540b2660 msgr2=0x7f33540b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.815+0000 7f335c678640 1 -- 172.21.15.59:0/1909716520 shutdown_connections 2024-03-06T07:24:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.815+0000 7f335c678640 1 -- 172.21.15.59:0/1909716520 wait complete. 2024-03-06T07:24:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.816+0000 7f335c678640 1 Processor -- start 2024-03-06T07:24:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.816+0000 7f335c678640 1 -- start start 2024-03-06T07:24:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.816+0000 7f335c678640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400039b0 0x7f33540d0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.816+0000 7f335c678640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33541e54c0 0x7f33540b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.816+0000 7f335c678640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33541e6c90 0x7f33541425f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.816+0000 7f335c678640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335415fac0 con 0x7f33541e6c90 2024-03-06T07:24:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.816+0000 7f335c678640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3354160110 con 0x7f33400039b0 2024-03-06T07:24:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.816+0000 7f335c678640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335415fda0 con 0x7f33541e54c0 2024-03-06T07:24:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335ac15640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33541e6c90 0x7f33541425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335ac15640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33541e6c90 0x7f33541425f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32954/0 (socket says 172.21.15.59:32954) 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335ac15640 1 -- 172.21.15.59:0/4047076757 learned_addr learned my addr 172.21.15.59:0/4047076757 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f3359c13640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33541e54c0 0x7f33540b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335a414640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400039b0 0x7f33540d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335ac15640 1 -- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33541e54c0 msgr2=0x7f33540b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335ac15640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33541e54c0 0x7f33540b5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335ac15640 1 -- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400039b0 msgr2=0x7f33540d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335ac15640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400039b0 0x7f33540d0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335ac15640 1 -- 172.21.15.59:0/4047076757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33541e5940 con 0x7f33541e6c90 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335ac15640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33541e6c90 0x7f33541425f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3354071cc0 tx=0x7f334c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f334affd640 1 -- 172.21.15.59:0/4047076757 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c010e80 con 0x7f33541e6c90 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f334affd640 1 -- 172.21.15.59:0/4047076757 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c003990 con 0x7f33541e6c90 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f334affd640 1 -- 172.21.15.59:0/4047076757 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c003da0 con 0x7f33541e6c90 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335c678640 1 -- 172.21.15.59:0/4047076757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33541e5bf0 con 0x7f33541e6c90 2024-03-06T07:24:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.817+0000 7f335c678640 1 -- 172.21.15.59:0/4047076757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33541e61b0 con 0x7f33541e6c90 2024-03-06T07:24:49.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.818+0000 7f335c678640 1 -- 172.21.15.59:0/4047076757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33540733b0 con 0x7f33541e6c90 2024-03-06T07:24:49.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.819+0000 7f334affd640 1 -- 172.21.15.59:0/4047076757 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f334c005d10 con 0x7f33541e6c90 2024-03-06T07:24:49.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.819+0000 7f334affd640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f333806b500 0x7f333806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.820+0000 7f334affd640 1 -- 172.21.15.59:0/4047076757 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f334c08aaa0 con 0x7f33541e6c90 2024-03-06T07:24:49.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.820+0000 7f335a414640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f333806b500 0x7f333806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.820+0000 7f335a414640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f333806b500 0x7f333806d9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3350034000 tx=0x7f3350007000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:49.824+0000 7f334affd640 1 -- 172.21.15.59:0/4047076757 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f334c025020 con 0x7f33541e6c90 2024-03-06T07:24:50.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.006+0000 7f335c678640 1 -- 172.21.15.59:0/4047076757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3354068ff0 con 0x7f33541e6c90 2024-03-06T07:24:50.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.007+0000 7f334affd640 1 -- 172.21.15.59:0/4047076757 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7f334c059880 con 0x7f33541e6c90 2024-03-06T07:24:50.009 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:24:50.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.008+0000 7f335c678640 1 -- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f333806b500 msgr2=0x7f333806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.008+0000 7f335c678640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f333806b500 0x7f333806d9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3350034000 tx=0x7f3350007000 comp rx=0 tx=0).stop 2024-03-06T07:24:50.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.008+0000 7f335c678640 1 -- 172.21.15.59:0/4047076757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33541e6c90 msgr2=0x7f33541425f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.008+0000 7f335c678640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33541e6c90 0x7f33541425f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3354071cc0 tx=0x7f334c009f90 comp rx=0 tx=0).stop 2024-03-06T07:24:50.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.008+0000 7f335c678640 1 -- 172.21.15.59:0/4047076757 shutdown_connections 2024-03-06T07:24:50.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.008+0000 7f335c678640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f333806b500 0x7f333806d9c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.008+0000 7f335c678640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33541e6c90 0x7f33541425f0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.008+0000 7f335c678640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33541e54c0 0x7f33540b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.008+0000 7f335c678640 1 --2- 172.21.15.59:0/4047076757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400039b0 0x7f33540d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.008+0000 7f335c678640 1 -- 172.21.15.59:0/4047076757 >> 172.21.15.59:0/4047076757 conn(0x7f33540b2660 msgr2=0x7f33540b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.008+0000 7f335c678640 1 -- 172.21.15.59:0/4047076757 shutdown_connections 2024-03-06T07:24:50.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.009+0000 7f335c678640 1 -- 172.21.15.59:0/4047076757 wait complete. 2024-03-06T07:24:50.021 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.121+0000 7fadd7fa1640 1 Processor -- start 2024-03-06T07:24:50.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.122+0000 7fadd7fa1640 1 -- start start 2024-03-06T07:24:50.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd7fa1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd015ddb0 0x7fadd0072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd7fa1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fadd006b6c0 0x7fadd006d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd7fa1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd0066fe0 con 0x7fadd015e190 2024-03-06T07:24:50.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd7fa1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd0073400 con 0x7fadd006b6c0 2024-03-06T07:24:50.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd7fa1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd00b5200 con 0x7fadd015ddb0 2024-03-06T07:24:50.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd653e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fadd006b6c0 0x7fadd006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd653e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fadd006b6c0 0x7fadd006d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32964/0 (socket says 172.21.15.59:32964) 2024-03-06T07:24:50.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd653e640 1 -- 172.21.15.59:0/1829790386 learned_addr learned my addr 172.21.15.59:0/1829790386 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:50.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd5d3d640 1 --2- 172.21.15.59:0/1829790386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd015ddb0 0x7fadd0072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd653e640 1 -- 172.21.15.59:0/1829790386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd015ddb0 msgr2=0x7fadd0072ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd653e640 1 --2- 172.21.15.59:0/1829790386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd015ddb0 0x7fadd0072ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd653e640 1 -- 172.21.15.59:0/1829790386 >> v1:172.21.15.164:6789/0 conn(0x7fadd015e190 legacy=0x7fadd006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.123+0000 7fadd653e640 1 -- 172.21.15.59:0/1829790386 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd00b54e0 con 0x7fadd006b6c0 2024-03-06T07:24:50.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.124+0000 7fadd653e640 1 --2- 172.21.15.59:0/1829790386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd006b6c0 0x7fadd006d960 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009870 tx=0x7fadc802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7ceb70e89ab6ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.124+0000 7fadd4d3b640 1 -- 172.21.15.59:0/1829790386 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc802f840 con 0x7fadd006b6c0 2024-03-06T07:24:50.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.124+0000 7fadd4d3b640 1 -- 172.21.15.59:0/1829790386 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc802f9e0 con 0x7fadd006b6c0 2024-03-06T07:24:50.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.124+0000 7fadd4d3b640 1 -- 172.21.15.59:0/1829790386 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc802fdb0 con 0x7fadd006b6c0 2024-03-06T07:24:50.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.125+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1829790386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd006b6c0 msgr2=0x7fadd006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.125+0000 7fadd7fa1640 1 --2- 172.21.15.59:0/1829790386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd006b6c0 0x7fadd006d960 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009870 tx=0x7fadc802ba20 comp rx=0 tx=0).stop 2024-03-06T07:24:50.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.125+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1829790386 shutdown_connections 2024-03-06T07:24:50.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.125+0000 7fadd7fa1640 1 --2- 172.21.15.59:0/1829790386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd006b6c0 0x7fadd006d960 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.125+0000 7fadd7fa1640 1 --2- 172.21.15.59:0/1829790386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd015ddb0 0x7fadd0072ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.125+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1829790386 >> 172.21.15.59:0/1829790386 conn(0x7fadd00b1d00 msgr2=0x7fadd00b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.125+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1829790386 shutdown_connections 2024-03-06T07:24:50.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.125+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1829790386 wait complete. 2024-03-06T07:24:50.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.126+0000 7fadd7fa1640 1 Processor -- start 2024-03-06T07:24:50.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.126+0000 7fadd7fa1640 1 -- start start 2024-03-06T07:24:50.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd7fa1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd006b6c0 0x7fadd00999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd7fa1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd015ddb0 0x7fadd01611e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd7fa1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd00adf10 0x7fadd00ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd7fa1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd00b5d80 con 0x7fadd006b6c0 2024-03-06T07:24:50.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd7fa1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd00b5aa0 con 0x7fadd015ddb0 2024-03-06T07:24:50.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd7fa1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd00b4e60 con 0x7fadd00adf10 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd5d3d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd006b6c0 0x7fadd00999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd5d3d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd006b6c0 0x7fadd00999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32976/0 (socket says 172.21.15.59:32976) 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd5d3d640 1 -- 172.21.15.59:0/1828036166 learned_addr learned my addr 172.21.15.59:0/1828036166 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd5d3d640 1 -- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd00adf10 msgr2=0x7fadd00ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd653e640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd00adf10 0x7fadd00ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd5d3d640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd00adf10 0x7fadd00ae3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd5d3d640 1 -- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd015ddb0 msgr2=0x7fadd01611e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd5d3d640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd015ddb0 0x7fadd01611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd5d3d640 1 -- 172.21.15.59:0/1828036166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd0161720 con 0x7fadd006b6c0 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.127+0000 7fadd5d3d640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd006b6c0 0x7fadd00999b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fadc0007ef0 tx=0x7fadc000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.128+0000 7fadce7fc640 1 -- 172.21.15.59:0/1828036166 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc0014620 con 0x7fadd006b6c0 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.128+0000 7fadce7fc640 1 -- 172.21.15.59:0/1828036166 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc0019070 con 0x7fadd006b6c0 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.128+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1828036166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd0157a00 con 0x7fadd006b6c0 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.128+0000 7fadce7fc640 1 -- 172.21.15.59:0/1828036166 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc0014a00 con 0x7fadd006b6c0 2024-03-06T07:24:50.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.128+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1828036166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd0157fc0 con 0x7fadd006b6c0 2024-03-06T07:24:50.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.129+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1828036166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd004f330 con 0x7fadd006b6c0 2024-03-06T07:24:50.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.129+0000 7fadce7fc640 1 -- 172.21.15.59:0/1828036166 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fadc0014ba0 con 0x7fadd006b6c0 2024-03-06T07:24:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.130+0000 7fadce7fc640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadbc06b4d0 0x7fadbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.130+0000 7fadce7fc640 1 -- 172.21.15.59:0/1828036166 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadc008aa50 con 0x7fadd006b6c0 2024-03-06T07:24:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.130+0000 7fadd553c640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadbc06b4d0 0x7fadbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.131+0000 7fadd553c640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadbc06b4d0 0x7fadbc06d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fadd00afb70 tx=0x7fadc4007970 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.134+0000 7fadce7fc640 1 -- 172.21.15.59:0/1828036166 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fadc0059960 con 0x7fadd006b6c0 2024-03-06T07:24:50.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.353+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1828036166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadd00713a0 con 0x7fadd006b6c0 2024-03-06T07:24:50.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.354+0000 7fadce7fc640 1 -- 172.21.15.59:0/1828036166 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fadc0059260 con 0x7fadd006b6c0 2024-03-06T07:24:50.356 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:50.356 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2205562350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2205562350},{"type":"v1","addr":"172.21.15.164:6837","nonce":2205562350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2606750481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2606750481},{"type":"v1","addr":"172.21.15.164:6836","nonce":2606750481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6048,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3685052110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3685052110},{"type":"v1","addr":"172.21.15.59:6833","nonce":3685052110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:11.010795+0000","modified":"2024-03-06T07:24:13.053115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/750409746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":750409746},{"type":"v1","addr":"172.21.15.59:6835","nonce":750409746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":4}]} 2024-03-06T07:24:50.356 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 28 2024-03-06T07:24:50.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.355+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadbc06b4d0 msgr2=0x7fadbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.355+0000 7fadd7fa1640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadbc06b4d0 0x7fadbc06d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fadd00afb70 tx=0x7fadc4007970 comp rx=0 tx=0).stop 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.355+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1828036166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd006b6c0 msgr2=0x7fadd00999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.355+0000 7fadd7fa1640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd006b6c0 0x7fadd00999b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fadc0007ef0 tx=0x7fadc000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.355+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1828036166 shutdown_connections 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.355+0000 7fadd7fa1640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fadbc06b4d0 0x7fadbc06d990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.355+0000 7fadd7fa1640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd00adf10 0x7fadd00ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.355+0000 7fadd7fa1640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd015ddb0 0x7fadd01611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.355+0000 7fadd7fa1640 1 --2- 172.21.15.59:0/1828036166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd006b6c0 0x7fadd00999b0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.355+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1828036166 >> 172.21.15.59:0/1828036166 conn(0x7fadd00b1d00 msgr2=0x7fadd00afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.355+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1828036166 shutdown_connections 2024-03-06T07:24:50.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.356+0000 7fadd7fa1640 1 -- 172.21.15.59:0/1828036166 wait complete. 2024-03-06T07:24:50.369 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:50.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.468+0000 7fd9e7fac640 1 Processor -- start 2024-03-06T07:24:50.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.469+0000 7fd9e7fac640 1 -- start start 2024-03-06T07:24:50.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.469+0000 7fd9e7fac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e00afb20 0x7fd9e015c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.469+0000 7fd9e7fac640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e00b4e30 con 0x7fd9e00af6c0 2024-03-06T07:24:50.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.469+0000 7fd9e7fac640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e00b5110 con 0x7fd9e015ab00 2024-03-06T07:24:50.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.469+0000 7fd9e7fac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e015b310 con 0x7fd9e00afb20 2024-03-06T07:24:50.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e5547640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd9e00af6c0 0x7fd9e015bb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47376/0 (socket says 172.21.15.59:47376) 2024-03-06T07:24:50.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e5547640 1 -- 172.21.15.59:0/422168811 learned_addr learned my addr 172.21.15.59:0/422168811 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:50.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e6549640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e00afb20 0x7fd9e015c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879686451 0 0) 0x7fd9e00b4e30 con 0x7fd9e00af6c0 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c8003520 con 0x7fd9e00af6c0 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494138636 0 0) 0x7fd9e00b5110 con 0x7fd9e015ab00 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e00b4e30 con 0x7fd9e015ab00 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 741886168 0 0) 0x7fd9c8003520 con 0x7fd9e00af6c0 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e00afb20 msgr2=0x7fd9e015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e00afb20 0x7fd9e015c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 >> v1:172.21.15.164:6789/0 conn(0x7fd9e015ab00 legacy=0x7fd9e00b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e015b5f0 con 0x7fd9e00af6c0 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fd9d4001690 con 0x7fd9e00af6c0 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 >> v1:172.21.15.59:6789/0 conn(0x7fd9e00af6c0 legacy=0x7fd9e015bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9c8003b70 0x7fd9c8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e00afb20 0x7fd9c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9c8004d40 0x7fd9c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e015b310 con 0x7fd9c8003b70 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c800ba70 con 0x7fd9c8004d40 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c800bc70 con 0x7fd9e00afb20 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e5d48640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9c8003b70 0x7fd9c8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e6549640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9c8004d40 0x7fd9c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.470+0000 7fd9e5547640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e00afb20 0x7fd9c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.471+0000 7fd9e5d48640 1 -- 172.21.15.59:0/422168811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e00afb20 msgr2=0x7fd9c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.471+0000 7fd9e5d48640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e00afb20 0x7fd9c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.471+0000 7fd9e5d48640 1 -- 172.21.15.59:0/422168811 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9c8004d40 msgr2=0x7fd9c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.471+0000 7fd9e5d48640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9c8004d40 0x7fd9c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.471+0000 7fd9e5d48640 1 -- 172.21.15.59:0/422168811 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c8003520 con 0x7fd9c8003b70 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.471+0000 7fd9e5d48640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9c8003b70 0x7fd9c8003f70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8001fd0 tx=0x7fd9c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85efe52b799fd601 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.472+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8001bf0 con 0x7fd9c8003b70 2024-03-06T07:24:50.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.472+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d8003b70 con 0x7fd9c8003b70 2024-03-06T07:24:50.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.472+0000 7fd9e4d46640 1 -- 172.21.15.59:0/422168811 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d80050d0 con 0x7fd9c8003b70 2024-03-06T07:24:50.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.472+0000 7fd9e7fac640 1 -- 172.21.15.59:0/422168811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9c8003b70 msgr2=0x7fd9c8003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.472+0000 7fd9e7fac640 1 --2- 172.21.15.59:0/422168811 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9c8003b70 0x7fd9c8003f70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8001fd0 tx=0x7fd9c8005830 comp rx=0 tx=0).stop 2024-03-06T07:24:50.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.472+0000 7fd9e6549640 1 -- 172.21.15.59:0/422168811 reap_dead start 2024-03-06T07:24:50.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.472+0000 7fd9e7fac640 1 -- 172.21.15.59:0/422168811 shutdown_connections 2024-03-06T07:24:50.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.472+0000 7fd9e7fac640 1 -- 172.21.15.59:0/422168811 >> 172.21.15.59:0/422168811 conn(0x7fd9e006c0c0 msgr2=0x7fd9e006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.472+0000 7fd9e7fac640 1 -- 172.21.15.59:0/422168811 shutdown_connections 2024-03-06T07:24:50.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.472+0000 7fd9e7fac640 1 -- 172.21.15.59:0/422168811 wait complete. 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.473+0000 7fd9e7fac640 1 Processor -- start 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.473+0000 7fd9e7fac640 1 -- start start 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.474+0000 7fd9e7fac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9c8003b70 0x7fd9e0153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.474+0000 7fd9e7fac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e015ca60 0x7fd9e015cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.474+0000 7fd9e7fac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e01f4000 0x7fd9e01f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.474+0000 7fd9e7fac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e0071b20 con 0x7fd9e01f4000 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.474+0000 7fd9e7fac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e00b4ac0 con 0x7fd9c8003b70 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.474+0000 7fd9e7fac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e0071ca0 con 0x7fd9e015ca60 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.474+0000 7fd9e6549640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e01f4000 0x7fd9e01f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.474+0000 7fd9e6549640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e01f4000 0x7fd9e01f4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:32988/0 (socket says 172.21.15.59:32988) 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.474+0000 7fd9e6549640 1 -- 172.21.15.59:0/1933370659 learned_addr learned my addr 172.21.15.59:0/1933370659 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:50.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.474+0000 7fd9e5547640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e015ca60 0x7fd9e015cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.474+0000 7fd9e5d48640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9c8003b70 0x7fd9e0153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.475+0000 7fd9e6549640 1 -- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e015ca60 msgr2=0x7fd9e015cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.475+0000 7fd9e6549640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e015ca60 0x7fd9e015cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.475+0000 7fd9e6549640 1 -- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9c8003b70 msgr2=0x7fd9e0153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.475+0000 7fd9e6549640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9c8003b70 0x7fd9e0153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.475+0000 7fd9e6549640 1 -- 172.21.15.59:0/1933370659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e01f0700 con 0x7fd9e01f4000 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.475+0000 7fd9e6549640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e01f4000 0x7fd9e01f4480 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0073750 tx=0x7fd9d0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.475+0000 7fd9de7fc640 1 -- 172.21.15.59:0/1933370659 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0004300 con 0x7fd9e01f4000 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.475+0000 7fd9de7fc640 1 -- 172.21.15.59:0/1933370659 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d00044a0 con 0x7fd9e01f4000 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.475+0000 7fd9de7fc640 1 -- 172.21.15.59:0/1933370659 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d000e660 con 0x7fd9e01f4000 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.475+0000 7fd9e7fac640 1 -- 172.21.15.59:0/1933370659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e01543a0 con 0x7fd9e01f4000 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.475+0000 7fd9e7fac640 1 -- 172.21.15.59:0/1933370659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e01f0c60 con 0x7fd9e01f4000 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.476+0000 7fd9e7fac640 1 -- 172.21.15.59:0/1933370659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e004f330 con 0x7fd9e01f4000 2024-03-06T07:24:50.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.477+0000 7fd9de7fc640 1 -- 172.21.15.59:0/1933370659 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9d000e800 con 0x7fd9e01f4000 2024-03-06T07:24:50.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.477+0000 7fd9de7fc640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9cc06b4d0 0x7fd9cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.477+0000 7fd9de7fc640 1 -- 172.21.15.59:0/1933370659 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9d008bbd0 con 0x7fd9e01f4000 2024-03-06T07:24:50.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.477+0000 7fd9e5d48640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9cc06b4d0 0x7fd9cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.478+0000 7fd9e5d48640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9cc06b4d0 0x7fd9cc06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8034000 tx=0x7fd9d8009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.482+0000 7fd9de7fc640 1 -- 172.21.15.59:0/1933370659 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9e01543a0 con 0x7fd9e01f4000 2024-03-06T07:24:50.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.666+0000 7fd9e7fac640 1 -- 172.21.15.59:0/1933370659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9e0159c00 con 0x7fd9e01f4000 2024-03-06T07:24:50.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.667+0000 7fd9de7fc640 1 -- 172.21.15.59:0/1933370659 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16653 (secure 0 0 0) 0x7fd9e01f0700 con 0x7fd9e01f4000 2024-03-06T07:24:50.669 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:50.669 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:24:12.652576+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:09.288329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.978574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 -- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9cc06b4d0 msgr2=0x7fd9cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9cc06b4d0 0x7fd9cc06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8034000 tx=0x7fd9d8009030 comp rx=0 tx=0).stop 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 -- 172.21.15.59:0/1933370659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e01f4000 msgr2=0x7fd9e01f4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e01f4000 0x7fd9e01f4480 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0073750 tx=0x7fd9d0011e80 comp rx=0 tx=0).stop 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 -- 172.21.15.59:0/1933370659 shutdown_connections 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9cc06b4d0 0x7fd9cc06d990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e01f4000 0x7fd9e01f4480 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e015ca60 0x7fd9e015cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 --2- 172.21.15.59:0/1933370659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9c8003b70 0x7fd9e0153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 -- 172.21.15.59:0/1933370659 >> 172.21.15.59:0/1933370659 conn(0x7fd9e006c0c0 msgr2=0x7fd9e01577e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 -- 172.21.15.59:0/1933370659 shutdown_connections 2024-03-06T07:24:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.669+0000 7fd9e7fac640 1 -- 172.21.15.59:0/1933370659 wait complete. 2024-03-06T07:24:50.682 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:24:50.682 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:50.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.806+0000 7f88ebcc2640 1 Processor -- start 2024-03-06T07:24:50.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.807+0000 7f88ebcc2640 1 -- start start 2024-03-06T07:24:50.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.807+0000 7f88ebcc2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f88e415dfe0 0x7f88e40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.807+0000 7f88ebcc2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f88e40d29c0 0x7f88e40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.807+0000 7f88ebcc2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e40b3430 con 0x7f88e40cf830 2024-03-06T07:24:50.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.807+0000 7f88ebcc2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e40b31e0 con 0x7f88e415dfe0 2024-03-06T07:24:50.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.807+0000 7f88ebcc2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88e40b3d60 con 0x7f88e40d29c0 2024-03-06T07:24:50.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.807+0000 7f88e9a5e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f88e415dfe0 0x7f88e40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.807+0000 7f88e925d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f88e40cf830 0x7f88e40d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47392/0 (socket says 172.21.15.59:47392) 2024-03-06T07:24:50.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.807+0000 7f88e925d640 1 -- 172.21.15.59:0/2349473636 learned_addr learned my addr 172.21.15.59:0/2349473636 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:50.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.808+0000 7f88ea25f640 1 --2- 172.21.15.59:0/2349473636 >> v2:172.21.15.164:3300/0 conn(0x7f88e40d29c0 0x7f88e40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.808+0000 7f88e9a5e640 1 -- 172.21.15.59:0/2349473636 >> v2:172.21.15.164:3300/0 conn(0x7f88e40d29c0 msgr2=0x7f88e40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.808+0000 7f88e9a5e640 1 --2- 172.21.15.59:0/2349473636 >> v2:172.21.15.164:3300/0 conn(0x7f88e40d29c0 0x7f88e40b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.808+0000 7f88e9a5e640 1 -- 172.21.15.59:0/2349473636 >> v1:172.21.15.59:6789/0 conn(0x7f88e40cf830 legacy=0x7f88e40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.808+0000 7f88e9a5e640 1 -- 172.21.15.59:0/2349473636 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e40b3710 con 0x7f88e415dfe0 2024-03-06T07:24:50.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.808+0000 7f88e8a5c640 1 -- 172.21.15.59:0/2349473636 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008323751 0 0) 0x7f88e40b3430 con 0x7f88e40cf830 2024-03-06T07:24:50.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.808+0000 7f88ea25f640 1 --2- 172.21.15.59:0/2349473636 >> v2:172.21.15.164:3300/0 conn(0x7f88e40d29c0 0x7f88e40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:50.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88e9a5e640 1 --2- 172.21.15.59:0/2349473636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e415dfe0 0x7f88e40cf2f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f88e40b3cd0 tx=0x7f88e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9ecf2f27f10a92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88e8a5c640 1 -- 172.21.15.59:0/2349473636 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0035020 con 0x7f88e415dfe0 2024-03-06T07:24:50.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88e8a5c640 1 -- 172.21.15.59:0/2349473636 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e0031830 con 0x7f88e415dfe0 2024-03-06T07:24:50.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88e8a5c640 1 -- 172.21.15.59:0/2349473636 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0031c00 con 0x7f88e415dfe0 2024-03-06T07:24:50.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88ebcc2640 1 -- 172.21.15.59:0/2349473636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e415dfe0 msgr2=0x7f88e40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88ebcc2640 1 --2- 172.21.15.59:0/2349473636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e415dfe0 0x7f88e40cf2f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f88e40b3cd0 tx=0x7f88e002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88ebcc2640 1 -- 172.21.15.59:0/2349473636 shutdown_connections 2024-03-06T07:24:50.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88ebcc2640 1 --2- 172.21.15.59:0/2349473636 >> v2:172.21.15.164:3300/0 conn(0x7f88e40d29c0 0x7f88e40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88ebcc2640 1 --2- 172.21.15.59:0/2349473636 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e415dfe0 0x7f88e40cf2f0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88ebcc2640 1 -- 172.21.15.59:0/2349473636 >> 172.21.15.59:0/2349473636 conn(0x7f88e40d3cd0 msgr2=0x7f88e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88ebcc2640 1 -- 172.21.15.59:0/2349473636 shutdown_connections 2024-03-06T07:24:50.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.809+0000 7f88ebcc2640 1 -- 172.21.15.59:0/2349473636 wait complete. 2024-03-06T07:24:50.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.810+0000 7f88ebcc2640 1 Processor -- start 2024-03-06T07:24:50.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.811+0000 7f88ebcc2640 1 -- start start 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.811+0000 7f88ebcc2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e40cf830 0x7f88e4160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.811+0000 7f88ebcc2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88e40d29c0 0x7f88e40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.811+0000 7f88ebcc2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88e415dfe0 0x7f88e40cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.811+0000 7f88ebcc2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e40d1df0 con 0x7f88e40cf830 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.811+0000 7f88ebcc2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e40d1c70 con 0x7f88e40d29c0 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.811+0000 7f88ebcc2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e415e3c0 con 0x7f88e415dfe0 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.811+0000 7f88e9a5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e40cf830 0x7f88e4160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.811+0000 7f88e9a5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e40cf830 0x7f88e4160600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33008/0 (socket says 172.21.15.59:33008) 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.811+0000 7f88e9a5e640 1 -- 172.21.15.59:0/1629852934 learned_addr learned my addr 172.21.15.59:0/1629852934 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88e9a5e640 1 -- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88e415dfe0 msgr2=0x7f88e40cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88e925d640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88e40d29c0 0x7f88e40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88ea25f640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88e415dfe0 0x7f88e40cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88e9a5e640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88e415dfe0 0x7f88e40cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88e9a5e640 1 -- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88e40d29c0 msgr2=0x7f88e40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88e9a5e640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88e40d29c0 0x7f88e40cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88e9a5e640 1 -- 172.21.15.59:0/1629852934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e415f560 con 0x7f88e40cf830 2024-03-06T07:24:50.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88e925d640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88e40d29c0 0x7f88e40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:50.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88ea25f640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88e415dfe0 0x7f88e40cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:50.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88e9a5e640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e40cf830 0x7f88e4160600 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f88e0000c00 tx=0x7f88e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88da7fc640 1 -- 172.21.15.59:0/1629852934 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0040070 con 0x7f88e40cf830 2024-03-06T07:24:50.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88da7fc640 1 -- 172.21.15.59:0/1629852934 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e0039940 con 0x7f88e40cf830 2024-03-06T07:24:50.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88ebcc2640 1 -- 172.21.15.59:0/1629852934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e415f840 con 0x7f88e40cf830 2024-03-06T07:24:50.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88da7fc640 1 -- 172.21.15.59:0/1629852934 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0035040 con 0x7f88e40cf830 2024-03-06T07:24:50.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.812+0000 7f88ebcc2640 1 -- 172.21.15.59:0/1629852934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e415fe00 con 0x7f88e40cf830 2024-03-06T07:24:50.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.813+0000 7f88ebcc2640 1 -- 172.21.15.59:0/1629852934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e404f330 con 0x7f88e40cf830 2024-03-06T07:24:50.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.814+0000 7f88da7fc640 1 -- 172.21.15.59:0/1629852934 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88e004b060 con 0x7f88e40cf830 2024-03-06T07:24:50.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.814+0000 7f88da7fc640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88c806b4d0 0x7f88c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.814+0000 7f88da7fc640 1 -- 172.21.15.59:0/1629852934 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88e00abc60 con 0x7f88e40cf830 2024-03-06T07:24:50.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.814+0000 7f88e925d640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88c806b4d0 0x7f88c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.815+0000 7f88e925d640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88c806b4d0 0x7f88c806d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f88e415c5f0 tx=0x7f88d4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:50.818+0000 7f88da7fc640 1 -- 172.21.15.59:0/1629852934 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88e007aae0 con 0x7f88e40cf830 2024-03-06T07:24:51.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.039+0000 7f88ebcc2640 1 -- 172.21.15.59:0/1629852934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88e40b48e0 con 0x7f88e40cf830 2024-03-06T07:24:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.040+0000 7f88da7fc640 1 -- 172.21.15.59:0/1629852934 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f88e007a3e0 con 0x7f88e40cf830 2024-03-06T07:24:51.042 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:51.042 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2205562350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2205562350},{"type":"v1","addr":"172.21.15.164:6837","nonce":2205562350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2606750481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2606750481},{"type":"v1","addr":"172.21.15.164:6836","nonce":2606750481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6048,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3685052110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3685052110},{"type":"v1","addr":"172.21.15.59:6833","nonce":3685052110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:11.010795+0000","modified":"2024-03-06T07:24:13.053115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/750409746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":750409746},{"type":"v1","addr":"172.21.15.59:6835","nonce":750409746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":4}]} 2024-03-06T07:24:51.042 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 28 2024-03-06T07:24:51.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 -- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88c806b4d0 msgr2=0x7f88c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88c806b4d0 0x7f88c806d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f88e415c5f0 tx=0x7f88d4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:51.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 -- 172.21.15.59:0/1629852934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e40cf830 msgr2=0x7f88e4160600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e40cf830 0x7f88e4160600 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f88e0000c00 tx=0x7f88e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 -- 172.21.15.59:0/1629852934 shutdown_connections 2024-03-06T07:24:51.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88c806b4d0 0x7f88c806d990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88e415dfe0 0x7f88e40cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88e40d29c0 0x7f88e40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 --2- 172.21.15.59:0/1629852934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88e40cf830 0x7f88e4160600 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 -- 172.21.15.59:0/1629852934 >> 172.21.15.59:0/1629852934 conn(0x7f88e40d3cd0 msgr2=0x7f88e40b4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 -- 172.21.15.59:0/1629852934 shutdown_connections 2024-03-06T07:24:51.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.042+0000 7f88ebcc2640 1 -- 172.21.15.59:0/1629852934 wait complete. 2024-03-06T07:24:51.055 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:51.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.155+0000 7f152740e640 1 Processor -- start 2024-03-06T07:24:51.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.155+0000 7f152740e640 1 -- start start 2024-03-06T07:24:51.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f152740e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f15200d22b0 0x7f15200d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f152740e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f15200b41f0 0x7f15200b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f152740e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15200b3430 con 0x7f152015dfe0 2024-03-06T07:24:51.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f152740e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15200d2c50 con 0x7f15200d22b0 2024-03-06T07:24:51.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f152740e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15200b31e0 con 0x7f15200b41f0 2024-03-06T07:24:51.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f15249a9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f15200d22b0 0x7f15200d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f15251aa640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f152015dfe0 0x7f15200cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47400/0 (socket says 172.21.15.59:47400) 2024-03-06T07:24:51.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f15251aa640 1 -- 172.21.15.59:0/3751164667 learned_addr learned my addr 172.21.15.59:0/3751164667 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:51.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f15249a9640 1 -- 172.21.15.59:0/3751164667 >> v2:172.21.15.164:3300/0 conn(0x7f15200b41f0 msgr2=0x7f15200b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f15259ab640 1 --2- 172.21.15.59:0/3751164667 >> v2:172.21.15.164:3300/0 conn(0x7f15200b41f0 0x7f15200b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f1513fff640 1 -- 172.21.15.59:0/3751164667 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260939684 0 0) 0x7f15200b3430 con 0x7f152015dfe0 2024-03-06T07:24:51.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f15249a9640 1 --2- 172.21.15.59:0/3751164667 >> v2:172.21.15.164:3300/0 conn(0x7f15200b41f0 0x7f15200b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f15249a9640 1 -- 172.21.15.59:0/3751164667 >> v1:172.21.15.59:6789/0 conn(0x7f152015dfe0 legacy=0x7f15200cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f15249a9640 1 -- 172.21.15.59:0/3751164667 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15200b3710 con 0x7f15200d22b0 2024-03-06T07:24:51.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.156+0000 7f15259ab640 1 --2- 172.21.15.59:0/3751164667 >> v2:172.21.15.164:3300/0 conn(0x7f15200b41f0 0x7f15200b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:51.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.157+0000 7f15249a9640 1 --2- 172.21.15.59:0/3751164667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15200d22b0 0x7f15200d2710 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f15140097a0 tx=0x7f151402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af220a3f0b21b2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.157+0000 7f1513fff640 1 -- 172.21.15.59:0/3751164667 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514035020 con 0x7f15200d22b0 2024-03-06T07:24:51.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.157+0000 7f1513fff640 1 -- 172.21.15.59:0/3751164667 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1514031830 con 0x7f15200d22b0 2024-03-06T07:24:51.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.157+0000 7f1513fff640 1 -- 172.21.15.59:0/3751164667 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514031c00 con 0x7f15200d22b0 2024-03-06T07:24:51.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.158+0000 7f152740e640 1 -- 172.21.15.59:0/3751164667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15200d22b0 msgr2=0x7f15200d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.158+0000 7f152740e640 1 --2- 172.21.15.59:0/3751164667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15200d22b0 0x7f15200d2710 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f15140097a0 tx=0x7f151402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.158+0000 7f152740e640 1 -- 172.21.15.59:0/3751164667 shutdown_connections 2024-03-06T07:24:51.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.158+0000 7f152740e640 1 --2- 172.21.15.59:0/3751164667 >> v2:172.21.15.164:3300/0 conn(0x7f15200b41f0 0x7f15200b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.158+0000 7f152740e640 1 --2- 172.21.15.59:0/3751164667 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15200d22b0 0x7f15200d2710 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.158+0000 7f152740e640 1 -- 172.21.15.59:0/3751164667 >> 172.21.15.59:0/3751164667 conn(0x7f15200d3cd0 msgr2=0x7f15200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.158+0000 7f152740e640 1 -- 172.21.15.59:0/3751164667 shutdown_connections 2024-03-06T07:24:51.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.158+0000 7f152740e640 1 -- 172.21.15.59:0/3751164667 wait complete. 2024-03-06T07:24:51.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.159+0000 7f152740e640 1 Processor -- start 2024-03-06T07:24:51.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.159+0000 7f152740e640 1 -- start start 2024-03-06T07:24:51.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f152740e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15200b41f0 0x7f152015e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f152740e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15200d22b0 0x7f15200b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f152740e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f152015dfe0 0x7f15200b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f152740e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15200cfa00 con 0x7f15200b41f0 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f152740e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15200b3d60 con 0x7f15200d22b0 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f152740e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15200d1c70 con 0x7f152015dfe0 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f15251aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15200b41f0 0x7f152015e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f15251aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15200b41f0 0x7f152015e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33030/0 (socket says 172.21.15.59:33030) 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f15251aa640 1 -- 172.21.15.59:0/3180594421 learned_addr learned my addr 172.21.15.59:0/3180594421 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f15249a9640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15200d22b0 0x7f15200b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f15259ab640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f152015dfe0 0x7f15200b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f15251aa640 1 -- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f152015dfe0 msgr2=0x7f15200b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f15251aa640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f152015dfe0 0x7f15200b7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f15251aa640 1 -- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15200d22b0 msgr2=0x7f15200b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f15251aa640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15200d22b0 0x7f15200b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f15251aa640 1 -- 172.21.15.59:0/3180594421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152015e900 con 0x7f15200b41f0 2024-03-06T07:24:51.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f15251aa640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15200b41f0 0x7f152015e3c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f151c00c430 tx=0x7f151c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.160+0000 7f1511ffb640 1 -- 172.21.15.59:0/3180594421 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c0074e0 con 0x7f15200b41f0 2024-03-06T07:24:51.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.161+0000 7f1511ffb640 1 -- 172.21.15.59:0/3180594421 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151c0047e0 con 0x7f15200b41f0 2024-03-06T07:24:51.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.161+0000 7f152740e640 1 -- 172.21.15.59:0/3180594421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152015ec30 con 0x7f15200b41f0 2024-03-06T07:24:51.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.161+0000 7f1511ffb640 1 -- 172.21.15.59:0/3180594421 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151c004bd0 con 0x7f15200b41f0 2024-03-06T07:24:51.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.161+0000 7f152740e640 1 -- 172.21.15.59:0/3180594421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152015f470 con 0x7f15200b41f0 2024-03-06T07:24:51.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.162+0000 7f152740e640 1 -- 172.21.15.59:0/3180594421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152004f330 con 0x7f15200b41f0 2024-03-06T07:24:51.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.162+0000 7f1511ffb640 1 -- 172.21.15.59:0/3180594421 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f151c011070 con 0x7f15200b41f0 2024-03-06T07:24:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.162+0000 7f1511ffb640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f150406b4d0 0x7f150406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.162+0000 7f1511ffb640 1 -- 172.21.15.59:0/3180594421 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f151c089d20 con 0x7f15200b41f0 2024-03-06T07:24:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.163+0000 7f15249a9640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f150406b4d0 0x7f150406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.163+0000 7f15249a9640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f150406b4d0 0x7f150406d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1514036000 tx=0x7f1514008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.167+0000 7f1511ffb640 1 -- 172.21.15.59:0/3180594421 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f152015ec30 con 0x7f15200b41f0 2024-03-06T07:24:51.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.387+0000 7f152740e640 1 -- 172.21.15.59:0/3180594421 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15200b1e60 con 0x7f15200b41f0 2024-03-06T07:24:51.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.387+0000 7f1511ffb640 1 -- 172.21.15.59:0/3180594421 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f151c058c30 con 0x7f15200b41f0 2024-03-06T07:24:51.389 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:51.389 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2205562350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2205562350},{"type":"v1","addr":"172.21.15.164:6837","nonce":2205562350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2606750481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2606750481},{"type":"v1","addr":"172.21.15.164:6836","nonce":2606750481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6048,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3685052110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3685052110},{"type":"v1","addr":"172.21.15.59:6833","nonce":3685052110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:11.010795+0000","modified":"2024-03-06T07:24:13.053115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/750409746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":750409746},{"type":"v1","addr":"172.21.15.59:6835","nonce":750409746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":4}]} 2024-03-06T07:24:51.390 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 28 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 -- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f150406b4d0 msgr2=0x7f150406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f150406b4d0 0x7f150406d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1514036000 tx=0x7f1514008030 comp rx=0 tx=0).stop 2024-03-06T07:24:51.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 -- 172.21.15.59:0/3180594421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15200b41f0 msgr2=0x7f152015e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15200b41f0 0x7f152015e3c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f151c00c430 tx=0x7f151c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 -- 172.21.15.59:0/3180594421 shutdown_connections 2024-03-06T07:24:51.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f150406b4d0 0x7f150406d990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f152015dfe0 0x7f15200b7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15200d22b0 0x7f15200b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 --2- 172.21.15.59:0/3180594421 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15200b41f0 0x7f152015e3c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 -- 172.21.15.59:0/3180594421 >> 172.21.15.59:0/3180594421 conn(0x7f15200d3cd0 msgr2=0x7f152006cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 -- 172.21.15.59:0/3180594421 shutdown_connections 2024-03-06T07:24:51.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.389+0000 7f152740e640 1 -- 172.21.15.59:0/3180594421 wait complete. 2024-03-06T07:24:51.402 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:51.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.501+0000 7feaf6b56640 1 Processor -- start 2024-03-06T07:24:51.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.501+0000 7feaf6b56640 1 -- start start 2024-03-06T07:24:51.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf6b56640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feaf015bb20 0x7feaf00d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf6b56640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf00cfff0 0x7feaf00d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf6b56640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feaf00ce4b0 0x7feaf00ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf6b56640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaf0161980 con 0x7feaf00ce4b0 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf6b56640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaf0161c60 con 0x7feaf015bb20 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf6b56640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf0161f40 con 0x7feaf00cfff0 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf50f3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feaf00ce4b0 0x7feaf00ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf50f3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feaf00ce4b0 0x7feaf00ce940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33044/0 (socket says 172.21.15.59:33044) 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf50f3640 1 -- 172.21.15.59:0/2674561614 learned_addr learned my addr 172.21.15.59:0/2674561614 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaeffff640 1 --2- 172.21.15.59:0/2674561614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf00cfff0 0x7feaf00d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf48f2640 1 --2- 172.21.15.59:0/2674561614 >> v2:172.21.15.164:3300/0 conn(0x7feaf015bb20 0x7feaf00d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf50f3640 1 -- 172.21.15.59:0/2674561614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf00cfff0 msgr2=0x7feaf00d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf50f3640 1 --2- 172.21.15.59:0/2674561614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf00cfff0 0x7feaf00d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf50f3640 1 -- 172.21.15.59:0/2674561614 >> v2:172.21.15.164:3300/0 conn(0x7feaf015bb20 msgr2=0x7feaf00d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf50f3640 1 --2- 172.21.15.59:0/2674561614 >> v2:172.21.15.164:3300/0 conn(0x7feaf015bb20 0x7feaf00d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.502+0000 7feaf50f3640 1 -- 172.21.15.59:0/2674561614 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf0162220 con 0x7feaf00ce4b0 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.503+0000 7feaf50f3640 1 --2- 172.21.15.59:0/2674561614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feaf00ce4b0 0x7feaf00ce940 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7feaf00d2bc0 tx=0x7feae802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1250963089c682 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.503+0000 7feaef7fe640 1 -- 172.21.15.59:0/2674561614 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae802ca70 con 0x7feaf00ce4b0 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.503+0000 7feaef7fe640 1 -- 172.21.15.59:0/2674561614 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae802cc10 con 0x7feaf00ce4b0 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.503+0000 7feaef7fe640 1 -- 172.21.15.59:0/2674561614 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae80343f0 con 0x7feaf00ce4b0 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.504+0000 7feaf6b56640 1 -- 172.21.15.59:0/2674561614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feaf00ce4b0 msgr2=0x7feaf00ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.504+0000 7feaf6b56640 1 --2- 172.21.15.59:0/2674561614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feaf00ce4b0 0x7feaf00ce940 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7feaf00d2bc0 tx=0x7feae802bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.504+0000 7feaf6b56640 1 -- 172.21.15.59:0/2674561614 shutdown_connections 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.504+0000 7feaf6b56640 1 --2- 172.21.15.59:0/2674561614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feaf00ce4b0 0x7feaf00ce940 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.504+0000 7feaf6b56640 1 --2- 172.21.15.59:0/2674561614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf00cfff0 0x7feaf00d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.504+0000 7feaf6b56640 1 --2- 172.21.15.59:0/2674561614 >> v2:172.21.15.164:3300/0 conn(0x7feaf015bb20 0x7feaf00d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.504+0000 7feaf6b56640 1 -- 172.21.15.59:0/2674561614 >> 172.21.15.59:0/2674561614 conn(0x7feaf009a9f0 msgr2=0x7feaf009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.504+0000 7feaf6b56640 1 -- 172.21.15.59:0/2674561614 shutdown_connections 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.504+0000 7feaf6b56640 1 -- 172.21.15.59:0/2674561614 wait complete. 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.505+0000 7feaf6b56640 1 Processor -- start 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.505+0000 7feaf6b56640 1 -- start start 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf6b56640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf00ce4b0 0x7feaf0158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf6b56640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf00cfff0 0x7feaf006ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf6b56640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feaf015bb20 0x7feaf006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf6b56640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf00d24b0 con 0x7feaf015bb20 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf6b56640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf00d2330 con 0x7feaf00cfff0 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf6b56640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf00d1980 con 0x7feaf00ce4b0 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf50f3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feaf015bb20 0x7feaf006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf50f3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feaf015bb20 0x7feaf006d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33046/0 (socket says 172.21.15.59:33046) 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf50f3640 1 -- 172.21.15.59:0/1595275963 learned_addr learned my addr 172.21.15.59:0/1595275963 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf48f2640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf00ce4b0 0x7feaf0158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaeffff640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf00cfff0 0x7feaf006ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf50f3640 1 -- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf00ce4b0 msgr2=0x7feaf0158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf50f3640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf00ce4b0 0x7feaf0158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf50f3640 1 -- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf00cfff0 msgr2=0x7feaf006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf50f3640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf00cfff0 0x7feaf006ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.506+0000 7feaf50f3640 1 -- 172.21.15.59:0/1595275963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf015c150 con 0x7feaf015bb20 2024-03-06T07:24:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.507+0000 7feaf50f3640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feaf015bb20 0x7feaf006d3f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7feae8009f40 tx=0x7feae8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.507+0000 7feaed7fa640 1 -- 172.21.15.59:0/1595275963 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae8043020 con 0x7feaf015bb20 2024-03-06T07:24:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.507+0000 7feaed7fa640 1 -- 172.21.15.59:0/1595275963 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae803e450 con 0x7feaf015bb20 2024-03-06T07:24:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.507+0000 7feaf6b56640 1 -- 172.21.15.59:0/1595275963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf015c430 con 0x7feaf015bb20 2024-03-06T07:24:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.507+0000 7feaed7fa640 1 -- 172.21.15.59:0/1595275963 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae8009510 con 0x7feaf015bb20 2024-03-06T07:24:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.507+0000 7feaf6b56640 1 -- 172.21.15.59:0/1595275963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf015c9f0 con 0x7feaf015bb20 2024-03-06T07:24:51.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.508+0000 7feaf6b56640 1 -- 172.21.15.59:0/1595275963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf006b500 con 0x7feaf015bb20 2024-03-06T07:24:51.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.508+0000 7feaed7fa640 1 -- 172.21.15.59:0/1595275963 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feae8046020 con 0x7feaf015bb20 2024-03-06T07:24:51.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.508+0000 7feaed7fa640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fead406b4d0 0x7fead406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.509+0000 7feaed7fa640 1 -- 172.21.15.59:0/1595275963 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feae80ac810 con 0x7feaf015bb20 2024-03-06T07:24:51.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.509+0000 7feaf48f2640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fead406b4d0 0x7fead406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.509+0000 7feaf48f2640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fead406b4d0 0x7fead406d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feaf015d6a0 tx=0x7feae4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.517+0000 7feaed7fa640 1 -- 172.21.15.59:0/1595275963 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feaf006b500 con 0x7feaf015bb20 2024-03-06T07:24:51.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.707+0000 7feaf6b56640 1 -- 172.21.15.59:0/1595275963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feaf004f330 con 0x7feaf015bb20 2024-03-06T07:24:51.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.708+0000 7feaed7fa640 1 -- 172.21.15.59:0/1595275963 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16653 (secure 0 0 0) 0x7feae804d030 con 0x7feaf015bb20 2024-03-06T07:24:51.710 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:51.710 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:24:12.652576+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:09.288329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:09.978574+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 -- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fead406b4d0 msgr2=0x7fead406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fead406b4d0 0x7fead406d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feaf015d6a0 tx=0x7feae4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 -- 172.21.15.59:0/1595275963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feaf015bb20 msgr2=0x7feaf006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feaf015bb20 0x7feaf006d3f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7feae8009f40 tx=0x7feae8004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 -- 172.21.15.59:0/1595275963 shutdown_connections 2024-03-06T07:24:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fead406b4d0 0x7fead406d990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feaf015bb20 0x7feaf006d3f0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf00cfff0 0x7feaf006ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 --2- 172.21.15.59:0/1595275963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf00ce4b0 0x7feaf0158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 -- 172.21.15.59:0/1595275963 >> 172.21.15.59:0/1595275963 conn(0x7feaf009a9f0 msgr2=0x7feaf00cdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 -- 172.21.15.59:0/1595275963 shutdown_connections 2024-03-06T07:24:51.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.710+0000 7feaf6b56640 1 -- 172.21.15.59:0/1595275963 wait complete. 2024-03-06T07:24:51.726 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:24:51.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.826+0000 7ff137808640 1 Processor -- start 2024-03-06T07:24:51.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.826+0000 7ff137808640 1 -- start start 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.827+0000 7ff137808640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff1301614a0 0x7ff13006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.827+0000 7ff137808640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1300b0270 0x7ff1300b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.827+0000 7ff137808640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff1300b1850 0x7ff13006c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.827+0000 7ff137808640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1300af4b0 con 0x7ff1300b1850 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.827+0000 7ff137808640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1300af790 con 0x7ff1301614a0 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.827+0000 7ff137808640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1300afa70 con 0x7ff1300b0270 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.827+0000 7ff135da5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff1300b1850 0x7ff13006c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.827+0000 7ff135da5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff1300b1850 0x7ff13006c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33058/0 (socket says 172.21.15.59:33058) 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.827+0000 7ff135da5640 1 -- 172.21.15.59:0/3087308240 learned_addr learned my addr 172.21.15.59:0/3087308240 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.828+0000 7ff1355a4640 1 --2- 172.21.15.59:0/3087308240 >> v2:172.21.15.164:3300/0 conn(0x7ff1301614a0 0x7ff13006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.828+0000 7ff134da3640 1 --2- 172.21.15.59:0/3087308240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1300b0270 0x7ff1300b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.828+0000 7ff135da5640 1 -- 172.21.15.59:0/3087308240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1300b0270 msgr2=0x7ff1300b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.828+0000 7ff135da5640 1 --2- 172.21.15.59:0/3087308240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1300b0270 0x7ff1300b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.828+0000 7ff135da5640 1 -- 172.21.15.59:0/3087308240 >> v2:172.21.15.164:3300/0 conn(0x7ff1301614a0 msgr2=0x7ff13006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:51.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.828+0000 7ff135da5640 1 --2- 172.21.15.59:0/3087308240 >> v2:172.21.15.164:3300/0 conn(0x7ff1301614a0 0x7ff13006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.828+0000 7ff135da5640 1 -- 172.21.15.59:0/3087308240 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1300afd50 con 0x7ff1300b1850 2024-03-06T07:24:51.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.828+0000 7ff135da5640 1 --2- 172.21.15.59:0/3087308240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1300b1850 0x7ff13006c040 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff1300730b0 tx=0x7ff12802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e33421f35a9054d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.829+0000 7ff127fff640 1 -- 172.21.15.59:0/3087308240 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12802c9e0 con 0x7ff1300b1850 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.829+0000 7ff127fff640 1 -- 172.21.15.59:0/3087308240 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12802cb80 con 0x7ff1300b1850 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.829+0000 7ff127fff640 1 -- 172.21.15.59:0/3087308240 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1280343f0 con 0x7ff1300b1850 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.829+0000 7ff137808640 1 -- 172.21.15.59:0/3087308240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1300b1850 msgr2=0x7ff13006c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.829+0000 7ff137808640 1 --2- 172.21.15.59:0/3087308240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1300b1850 0x7ff13006c040 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff1300730b0 tx=0x7ff12802be40 comp rx=0 tx=0).stop 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.829+0000 7ff137808640 1 -- 172.21.15.59:0/3087308240 shutdown_connections 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.829+0000 7ff137808640 1 --2- 172.21.15.59:0/3087308240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1300b1850 0x7ff13006c040 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.829+0000 7ff137808640 1 --2- 172.21.15.59:0/3087308240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1300b0270 0x7ff1300b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.829+0000 7ff137808640 1 --2- 172.21.15.59:0/3087308240 >> v2:172.21.15.164:3300/0 conn(0x7ff1301614a0 0x7ff13006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.829+0000 7ff137808640 1 -- 172.21.15.59:0/3087308240 >> 172.21.15.59:0/3087308240 conn(0x7ff13009a9f0 msgr2=0x7ff13009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.830+0000 7ff137808640 1 -- 172.21.15.59:0/3087308240 shutdown_connections 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.830+0000 7ff137808640 1 -- 172.21.15.59:0/3087308240 wait complete. 2024-03-06T07:24:51.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.831+0000 7ff137808640 1 Processor -- start 2024-03-06T07:24:51.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.831+0000 7ff137808640 1 -- start start 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.831+0000 7ff137808640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1300b0270 0x7ff1300cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.831+0000 7ff137808640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1300b1850 0x7ff130147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.831+0000 7ff137808640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1301614a0 0x7ff13006ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.831+0000 7ff137808640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1300b1c80 con 0x7ff1300b0270 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.831+0000 7ff137808640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1300b06b0 con 0x7ff1300b1850 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.831+0000 7ff137808640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1300b1e00 con 0x7ff1301614a0 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.831+0000 7ff1355a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1300b0270 0x7ff1300cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff1355a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1300b0270 0x7ff1300cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33066/0 (socket says 172.21.15.59:33066) 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff1355a4640 1 -- 172.21.15.59:0/2290293752 learned_addr learned my addr 172.21.15.59:0/2290293752 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff134da3640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1300b1850 0x7ff130147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff135da5640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1301614a0 0x7ff13006ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff1355a4640 1 -- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1301614a0 msgr2=0x7ff13006ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff1355a4640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1301614a0 0x7ff13006ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff1355a4640 1 -- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1300b1850 msgr2=0x7ff130147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff1355a4640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1300b1850 0x7ff130147420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff1355a4640 1 -- 172.21.15.59:0/2290293752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1300cdbe0 con 0x7ff1300b0270 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff134da3640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1300b1850 0x7ff130147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff1355a4640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1300b0270 0x7ff1300cd6a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00d5f0 tx=0x7ff12c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff125ffb640 1 -- 172.21.15.59:0/2290293752 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c0125f0 con 0x7ff1300b0270 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff125ffb640 1 -- 172.21.15.59:0/2290293752 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12c019070 con 0x7ff1300b0270 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff125ffb640 1 -- 172.21.15.59:0/2290293752 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c0129d0 con 0x7ff1300b0270 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff137808640 1 -- 172.21.15.59:0/2290293752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1300cc600 con 0x7ff1300b0270 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.832+0000 7ff137808640 1 -- 172.21.15.59:0/2290293752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1300ccbc0 con 0x7ff1300b0270 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.834+0000 7ff125ffb640 1 -- 172.21.15.59:0/2290293752 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff12c012b70 con 0x7ff1300b0270 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.834+0000 7ff137808640 1 -- 172.21.15.59:0/2290293752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13004f330 con 0x7ff1300b0270 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.834+0000 7ff125ffb640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff11406b4d0 0x7ff11406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.834+0000 7ff125ffb640 1 -- 172.21.15.59:0/2290293752 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff12c08a5d0 con 0x7ff1300b0270 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.834+0000 7ff134da3640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff11406b4d0 0x7ff11406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.835+0000 7ff134da3640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff11406b4d0 0x7ff11406d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff120006fd0 tx=0x7ff120008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:51.838+0000 7ff125ffb640 1 -- 172.21.15.59:0/2290293752 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff12c059450 con 0x7ff1300b0270 2024-03-06T07:24:52.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.059+0000 7ff137808640 1 -- 172.21.15.59:0/2290293752 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff1300b6210 con 0x7ff1300b0270 2024-03-06T07:24:52.061 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:24:52.059+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.061+0000 7ff125ffb640 1 -- 172.21.15.59:0/2290293752 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff12c058d50 con 0x7ff1300b0270 2024-03-06T07:24:52.062 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:24:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 -- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff11406b4d0 msgr2=0x7ff11406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff11406b4d0 0x7ff11406d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff120006fd0 tx=0x7ff120008040 comp rx=0 tx=0).stop 2024-03-06T07:24:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 -- 172.21.15.59:0/2290293752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1300b0270 msgr2=0x7ff1300cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1300b0270 0x7ff1300cd6a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00d5f0 tx=0x7ff12c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 -- 172.21.15.59:0/2290293752 shutdown_connections 2024-03-06T07:24:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff11406b4d0 0x7ff11406d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1301614a0 0x7ff13006ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1300b1850 0x7ff130147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 --2- 172.21.15.59:0/2290293752 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1300b0270 0x7ff1300cd6a0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 -- 172.21.15.59:0/2290293752 >> 172.21.15.59:0/2290293752 conn(0x7ff13009a9f0 msgr2=0x7ff1300b0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 -- 172.21.15.59:0/2290293752 shutdown_connections 2024-03-06T07:24:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.064+0000 7ff137808640 1 -- 172.21.15.59:0/2290293752 wait complete. 2024-03-06T07:24:52.077 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:24:52.104 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:24:52.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.180+0000 7fc1aeea4640 1 Processor -- start 2024-03-06T07:24:52.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.180+0000 7fc1aeea4640 1 -- start start 2024-03-06T07:24:52.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.181+0000 7fc1aeea4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1a815def0 0x7fc1a80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.181+0000 7fc1aeea4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a806d030 con 0x7fc1a80cea50 2024-03-06T07:24:52.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.181+0000 7fc1aeea4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a806d310 con 0x7fc1a806cab0 2024-03-06T07:24:52.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.181+0000 7fc1aeea4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a80cf1c0 con 0x7fc1a815def0 2024-03-06T07:24:52.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.181+0000 7fc1ad441640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc1a80cea50 0x7fc1a80ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47408/0 (socket says 172.21.15.59:47408) 2024-03-06T07:24:52.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.181+0000 7fc1ad441640 1 -- 172.21.15.59:0/3963964109 learned_addr learned my addr 172.21.15.59:0/3963964109 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:52.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.181+0000 7fc1acc40640 1 --2- 172.21.15.59:0/3963964109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1a815def0 0x7fc1a80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.182+0000 7fc1a77fe640 1 -- 172.21.15.59:0/3963964109 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285279052 0 0) 0x7fc1a806d030 con 0x7fc1a80cea50 2024-03-06T07:24:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.182+0000 7fc1a77fe640 1 -- 172.21.15.59:0/3963964109 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc194003520 con 0x7fc1a80cea50 2024-03-06T07:24:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.182+0000 7fc1a77fe640 1 -- 172.21.15.59:0/3963964109 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676332890 0 0) 0x7fc1a806d310 con 0x7fc1a806cab0 2024-03-06T07:24:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.182+0000 7fc1acc40640 1 -- 172.21.15.59:0/3963964109 >> v1:172.21.15.164:6789/0 conn(0x7fc1a806cab0 legacy=0x7fc1a80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.182+0000 7fc1acc40640 1 -- 172.21.15.59:0/3963964109 >> v1:172.21.15.59:6789/0 conn(0x7fc1a80cea50 legacy=0x7fc1a80ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.182+0000 7fc1acc40640 1 -- 172.21.15.59:0/3963964109 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a80cc110 con 0x7fc1a815def0 2024-03-06T07:24:52.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.183+0000 7fc1acc40640 1 --2- 172.21.15.59:0/3963964109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1a815def0 0x7fc1a80b41b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc1a00097d0 tx=0x7fc1a002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc7f0a74bf16577c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.183+0000 7fc1a77fe640 1 -- 172.21.15.59:0/3963964109 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0034070 con 0x7fc1a815def0 2024-03-06T07:24:52.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.183+0000 7fc1a77fe640 1 -- 172.21.15.59:0/3963964109 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a0030880 con 0x7fc1a815def0 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.183+0000 7fc1a77fe640 1 -- 172.21.15.59:0/3963964109 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0030c30 con 0x7fc1a815def0 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.183+0000 7fc1aeea4640 1 -- 172.21.15.59:0/3963964109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1a815def0 msgr2=0x7fc1a80b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.183+0000 7fc1aeea4640 1 --2- 172.21.15.59:0/3963964109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1a815def0 0x7fc1a80b41b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc1a00097d0 tx=0x7fc1a002b9c0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.183+0000 7fc1aeea4640 1 -- 172.21.15.59:0/3963964109 shutdown_connections 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.183+0000 7fc1aeea4640 1 --2- 172.21.15.59:0/3963964109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1a815def0 0x7fc1a80b41b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.183+0000 7fc1aeea4640 1 -- 172.21.15.59:0/3963964109 >> 172.21.15.59:0/3963964109 conn(0x7fc1a80b3690 msgr2=0x7fc1a80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.183+0000 7fc1aeea4640 1 -- 172.21.15.59:0/3963964109 shutdown_connections 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.183+0000 7fc1aeea4640 1 -- 172.21.15.59:0/3963964109 wait complete. 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.184+0000 7fc1aeea4640 1 Processor -- start 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.184+0000 7fc1aeea4640 1 -- start start 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.185+0000 7fc1aeea4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1a806cab0 0x7fc1a8159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.185+0000 7fc1aeea4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a80cea50 0x7fc1a80c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.185+0000 7fc1aeea4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1a815def0 0x7fc1a80c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.185+0000 7fc1aeea4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a815e2d0 con 0x7fc1a80cea50 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.185+0000 7fc1aeea4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a80cc760 con 0x7fc1a806cab0 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.185+0000 7fc1aeea4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a815e450 con 0x7fc1a815def0 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.185+0000 7fc1a7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a80cea50 0x7fc1a80c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.185+0000 7fc1a7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a80cea50 0x7fc1a80c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33102/0 (socket says 172.21.15.59:33102) 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.185+0000 7fc1a7fff640 1 -- 172.21.15.59:0/894240067 learned_addr learned my addr 172.21.15.59:0/894240067 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1acc40640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1a806cab0 0x7fc1a8159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1ad441640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1a815def0 0x7fc1a80c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1a7fff640 1 -- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1a815def0 msgr2=0x7fc1a80c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1a7fff640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1a815def0 0x7fc1a80c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1a7fff640 1 -- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1a806cab0 msgr2=0x7fc1a8159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1a7fff640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1a806cab0 0x7fc1a8159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1a7fff640 1 -- 172.21.15.59:0/894240067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a80c7040 con 0x7fc1a80cea50 2024-03-06T07:24:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1a7fff640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a80cea50 0x7fc1a80c65f0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc19800c890 tx=0x7fc19800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1a57fa640 1 -- 172.21.15.59:0/894240067 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198017070 con 0x7fc1a80cea50 2024-03-06T07:24:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1a57fa640 1 -- 172.21.15.59:0/894240067 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc198003dc0 con 0x7fc1a80cea50 2024-03-06T07:24:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1aeea4640 1 -- 172.21.15.59:0/894240067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a80c7320 con 0x7fc1a80cea50 2024-03-06T07:24:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1a57fa640 1 -- 172.21.15.59:0/894240067 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1980052b0 con 0x7fc1a80cea50 2024-03-06T07:24:52.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.186+0000 7fc1aeea4640 1 -- 172.21.15.59:0/894240067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a80c78e0 con 0x7fc1a80cea50 2024-03-06T07:24:52.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.187+0000 7fc1aeea4640 1 -- 172.21.15.59:0/894240067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a815a440 con 0x7fc1a80cea50 2024-03-06T07:24:52.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.188+0000 7fc1a57fa640 1 -- 172.21.15.59:0/894240067 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc19801a070 con 0x7fc1a80cea50 2024-03-06T07:24:52.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.188+0000 7fc1a57fa640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc18806b4d0 0x7fc18806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.188+0000 7fc1a57fa640 1 -- 172.21.15.59:0/894240067 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc198089890 con 0x7fc1a80cea50 2024-03-06T07:24:52.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.188+0000 7fc1acc40640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc18806b4d0 0x7fc18806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.189+0000 7fc1acc40640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc18806b4d0 0x7fc18806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0036000 tx=0x7fc1a0008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.192+0000 7fc1a57fa640 1 -- 172.21.15.59:0/894240067 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1980580f0 con 0x7fc1a80cea50 2024-03-06T07:24:52.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:52.410+0000 7fc1aeea4640 1 -- 172.21.15.59:0/894240067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc1a809fcf0 con 0x7fc1a80cea50 2024-03-06T07:24:53.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.062+0000 7fc1a57fa640 1 -- 172.21.15.59:0/894240067 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc19805c190 con 0x7fc1a80cea50 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 -- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc18806b4d0 msgr2=0x7fc18806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc18806b4d0 0x7fc18806d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0036000 tx=0x7fc1a0008030 comp rx=0 tx=0).stop 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 -- 172.21.15.59:0/894240067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a80cea50 msgr2=0x7fc1a80c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a80cea50 0x7fc1a80c65f0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc19800c890 tx=0x7fc19800cd50 comp rx=0 tx=0).stop 2024-03-06T07:24:53.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 -- 172.21.15.59:0/894240067 shutdown_connections 2024-03-06T07:24:53.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc18806b4d0 0x7fc18806d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1a815def0 0x7fc1a80c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1a80cea50 0x7fc1a80c65f0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 --2- 172.21.15.59:0/894240067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1a806cab0 0x7fc1a8159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 -- 172.21.15.59:0/894240067 >> 172.21.15.59:0/894240067 conn(0x7fc1a80b3690 msgr2=0x7fc1a806f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 -- 172.21.15.59:0/894240067 shutdown_connections 2024-03-06T07:24:53.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.065+0000 7fc1aeea4640 1 -- 172.21.15.59:0/894240067 wait complete. 2024-03-06T07:24:53.082 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:24:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.233+0000 7f740104e640 1 Processor -- start 2024-03-06T07:24:53.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.233+0000 7f740104e640 1 -- start start 2024-03-06T07:24:53.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.234+0000 7f740104e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f73fc15bdd0 0x7f73fc1602b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.234+0000 7f740104e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0d0130 0x7f73fc15d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.234+0000 7f740104e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc06f420 con 0x7f73fc0701e0 2024-03-06T07:24:53.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.234+0000 7f740104e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73fc06f700 con 0x7f73fc15bdd0 2024-03-06T07:24:53.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.234+0000 7f740104e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc06f9e0 con 0x7f73fc0d0130 2024-03-06T07:24:53.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.234+0000 7f73fb59e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f73fc0701e0 0x7f73fc070630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47412/0 (socket says 172.21.15.59:47412) 2024-03-06T07:24:53.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.234+0000 7f73fb59e640 1 -- 172.21.15.59:0/3235611536 learned_addr learned my addr 172.21.15.59:0/3235611536 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:53.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744520214 0 0) 0x7f73fc06f420 con 0x7f73fc0701e0 2024-03-06T07:24:53.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e8003520 con 0x7f73fc0701e0 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fa59c640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0d0130 0x7f73fc15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fad9d640 1 --2- 172.21.15.59:0/3235611536 >> v2:172.21.15.164:3300/0 conn(0x7f73fc15bdd0 0x7f73fc1602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4201745207 0 0) 0x7f73e8003520 con 0x7f73fc0701e0 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0d0130 msgr2=0x7f73fc15d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0d0130 0x7f73fc15d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 >> v2:172.21.15.164:3300/0 conn(0x7f73fc15bdd0 msgr2=0x7f73fc1602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 --2- 172.21.15.59:0/3235611536 >> v2:172.21.15.164:3300/0 conn(0x7f73fc15bdd0 0x7f73fc1602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc06fcc0 con 0x7f73fc0701e0 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f73ec0019f0 con 0x7f73fc0701e0 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fa59c640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0d0130 0x7f73fc15d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fad9d640 1 --2- 172.21.15.59:0/3235611536 >> v2:172.21.15.164:3300/0 conn(0x7f73fc15bdd0 0x7f73fc1602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 >> v1:172.21.15.59:6789/0 conn(0x7f73fc0701e0 legacy=0x7f73fc070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 0x7f73e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0d0130 0x7f73e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e8004d40 0x7f73e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc06f700 con 0x7f73e80039b0 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc06f9e0 con 0x7f73e8004d40 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e800ba70 con 0x7f73fc0d0130 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fad9d640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 0x7f73e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fa59c640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0d0130 0x7f73e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fb59e640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e8004d40 0x7f73e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fad9d640 1 -- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0d0130 msgr2=0x7f73e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fad9d640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0d0130 0x7f73e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fad9d640 1 -- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e8004d40 msgr2=0x7f73e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fad9d640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e8004d40 0x7f73e80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.235+0000 7f73fad9d640 1 -- 172.21.15.59:0/3235611536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e8003520 con 0x7f73e80039b0 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.236+0000 7f73fad9d640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 0x7f73e8003dd0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f73f40098a0 tx=0x7f73e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b872d162136d6c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.236+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f400f040 con 0x7f73e80039b0 2024-03-06T07:24:53.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.236+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f4004230 con 0x7f73e80039b0 2024-03-06T07:24:53.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.236+0000 7f73f9d9b640 1 -- 172.21.15.59:0/3235611536 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4003610 con 0x7f73e80039b0 2024-03-06T07:24:53.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.237+0000 7f740104e640 1 -- 172.21.15.59:0/3235611536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 msgr2=0x7f73e8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.237+0000 7f740104e640 1 --2- 172.21.15.59:0/3235611536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 0x7f73e8003dd0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f73f40098a0 tx=0x7f73e8005830 comp rx=0 tx=0).stop 2024-03-06T07:24:53.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.237+0000 7f73fb59e640 1 -- 172.21.15.59:0/3235611536 reap_dead start 2024-03-06T07:24:53.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.237+0000 7f740104e640 1 -- 172.21.15.59:0/3235611536 shutdown_connections 2024-03-06T07:24:53.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.237+0000 7f740104e640 1 -- 172.21.15.59:0/3235611536 >> 172.21.15.59:0/3235611536 conn(0x7f73fc09a9f0 msgr2=0x7f73fc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.237+0000 7f740104e640 1 -- 172.21.15.59:0/3235611536 shutdown_connections 2024-03-06T07:24:53.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.237+0000 7f740104e640 1 -- 172.21.15.59:0/3235611536 wait complete. 2024-03-06T07:24:53.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.238+0000 7f740104e640 1 Processor -- start 2024-03-06T07:24:53.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.238+0000 7f740104e640 1 -- start start 2024-03-06T07:24:53.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.239+0000 7f740104e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 0x7f73fc0c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.239+0000 7f740104e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73fc0cee50 0x7f73fc0b67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.239+0000 7f740104e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0b6ce0 0x7f73fc0b7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.239+0000 7f740104e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0d0a50 con 0x7f73e80039b0 2024-03-06T07:24:53.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.239+0000 7f740104e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0d08d0 con 0x7f73fc0cee50 2024-03-06T07:24:53.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.239+0000 7f740104e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc15d330 con 0x7f73fc0b6ce0 2024-03-06T07:24:53.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.239+0000 7f73fad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 0x7f73fc0c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.239+0000 7f73fad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 0x7f73fc0c9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33122/0 (socket says 172.21.15.59:33122) 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.239+0000 7f73fad9d640 1 -- 172.21.15.59:0/3915220300 learned_addr learned my addr 172.21.15.59:0/3915220300 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73fb59e640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0b6ce0 0x7f73fc0b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73fa59c640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73fc0cee50 0x7f73fc0b67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73fad9d640 1 -- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0b6ce0 msgr2=0x7f73fc0b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73fad9d640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0b6ce0 0x7f73fc0b7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73fad9d640 1 -- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73fc0cee50 msgr2=0x7f73fc0b67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73fad9d640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73fc0cee50 0x7f73fc0b67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73fad9d640 1 -- 172.21.15.59:0/3915220300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc0b76a0 con 0x7f73e80039b0 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73fa59c640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73fc0cee50 0x7f73fc0b67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73fad9d640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 0x7f73fc0c9c70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f73f402bff0 tx=0x7f73f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73f37fe640 1 -- 172.21.15.59:0/3915220300 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4047070 con 0x7f73e80039b0 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73f37fe640 1 -- 172.21.15.59:0/3915220300 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f4042400 con 0x7f73e80039b0 2024-03-06T07:24:53.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f740104e640 1 -- 172.21.15.59:0/3915220300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73fc0ca370 con 0x7f73e80039b0 2024-03-06T07:24:53.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f73f37fe640 1 -- 172.21.15.59:0/3915220300 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f40427d0 con 0x7f73e80039b0 2024-03-06T07:24:53.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.240+0000 7f740104e640 1 -- 172.21.15.59:0/3915220300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73fc0ca930 con 0x7f73e80039b0 2024-03-06T07:24:53.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.241+0000 7f740104e640 1 -- 172.21.15.59:0/3915220300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73fc04f330 con 0x7f73e80039b0 2024-03-06T07:24:53.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.241+0000 7f73f37fe640 1 -- 172.21.15.59:0/3915220300 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73f404a070 con 0x7f73e80039b0 2024-03-06T07:24:53.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.242+0000 7f73f37fe640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73dc06b500 0x7f73dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.242+0000 7f73f37fe640 1 -- 172.21.15.59:0/3915220300 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f73f40aba80 con 0x7f73e80039b0 2024-03-06T07:24:53.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.242+0000 7f73fa59c640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73dc06b500 0x7f73dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.243+0000 7f73fa59c640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73dc06b500 0x7f73dc06d9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f73fc15c7a0 tx=0x7f73e4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.246+0000 7f73f37fe640 1 -- 172.21.15.59:0/3915220300 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73f4033070 con 0x7f73e80039b0 2024-03-06T07:24:53.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:53.426+0000 7f740104e640 1 -- 172.21.15.59:0/3915220300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f73fc0cbc50 con 0x7f73e80039b0 2024-03-06T07:24:54.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.071+0000 7f73f37fe640 1 -- 172.21.15.59:0/3915220300 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v61) v1 ==== 181+0+0 (secure 0 0 0) 0x7f73f407a8a0 con 0x7f73e80039b0 2024-03-06T07:24:54.073 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:24:54.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.074+0000 7f740104e640 1 -- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73dc06b500 msgr2=0x7f73dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.074+0000 7f740104e640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73dc06b500 0x7f73dc06d9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f73fc15c7a0 tx=0x7f73e4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:54.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.074+0000 7f740104e640 1 -- 172.21.15.59:0/3915220300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 msgr2=0x7f73fc0c9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.074+0000 7f740104e640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 0x7f73fc0c9c70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f73f402bff0 tx=0x7f73f4004730 comp rx=0 tx=0).stop 2024-03-06T07:24:54.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.075+0000 7f740104e640 1 -- 172.21.15.59:0/3915220300 shutdown_connections 2024-03-06T07:24:54.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.075+0000 7f740104e640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73dc06b500 0x7f73dc06d9c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.075+0000 7f740104e640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73fc0b6ce0 0x7f73fc0b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.075+0000 7f740104e640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73fc0cee50 0x7f73fc0b67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.075+0000 7f740104e640 1 --2- 172.21.15.59:0/3915220300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73e80039b0 0x7f73fc0c9c70 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.075+0000 7f740104e640 1 -- 172.21.15.59:0/3915220300 >> 172.21.15.59:0/3915220300 conn(0x7f73fc09a9f0 msgr2=0x7f73fc0b0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.075+0000 7f740104e640 1 -- 172.21.15.59:0/3915220300 shutdown_connections 2024-03-06T07:24:54.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.075+0000 7f740104e640 1 -- 172.21.15.59:0/3915220300 wait complete. 2024-03-06T07:24:54.091 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:24:54.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.199+0000 7f18da9a0640 1 Processor -- start 2024-03-06T07:24:54.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.199+0000 7f18da9a0640 1 -- start start 2024-03-06T07:24:54.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.200+0000 7f18da9a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f18d415e0a0 0x7f18d40ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.200+0000 7f18da9a0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d40cf280 con 0x7f18d40d0db0 2024-03-06T07:24:54.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.200+0000 7f18da9a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d40cf560 con 0x7f18d415efa0 2024-03-06T07:24:54.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.200+0000 7f18da9a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d40cf840 con 0x7f18d415e0a0 2024-03-06T07:24:54.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.200+0000 7f18da19f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f18d40d0db0 0x7f18d40d1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59354/0 (socket says 172.21.15.59:59354) 2024-03-06T07:24:54.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.200+0000 7f18da19f640 1 -- 172.21.15.59:0/2530961678 learned_addr learned my addr 172.21.15.59:0/2530961678 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:54.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.200+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787906291 0 0) 0x7f18d40cf280 con 0x7f18d40d0db0 2024-03-06T07:24:54.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.200+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18bc003520 con 0x7f18d40d0db0 2024-03-06T07:24:54.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d999e640 1 --2- 172.21.15.59:0/2530961678 >> v2:172.21.15.164:3300/0 conn(0x7f18d415e0a0 0x7f18d40ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069050978 0 0) 0x7f18bc003520 con 0x7f18d40d0db0 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 >> v2:172.21.15.164:3300/0 conn(0x7f18d415e0a0 msgr2=0x7f18d40ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 --2- 172.21.15.59:0/2530961678 >> v2:172.21.15.164:3300/0 conn(0x7f18d415e0a0 0x7f18d40ceb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 >> v1:172.21.15.164:6789/0 conn(0x7f18d415efa0 legacy=0x7f18d40d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d40cfb20 con 0x7f18d40d0db0 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f18c40019f0 con 0x7f18d40d0db0 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 >> v1:172.21.15.59:6789/0 conn(0x7f18d40d0db0 legacy=0x7f18d40d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18bc0039b0 0x7f18bc003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18bc004800 0x7f18bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18bc0052e0 0x7f18bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d40cf840 con 0x7f18bc004800 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18bc00c190 con 0x7f18bc0039b0 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18bc00c390 con 0x7f18bc0052e0 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d919d640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18bc004800 0x7f18bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d999e640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18bc0039b0 0x7f18bc003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18da19f640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18bc0052e0 0x7f18bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d919d640 1 -- 172.21.15.59:0/2530961678 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18bc0052e0 msgr2=0x7f18bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d919d640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18bc0052e0 0x7f18bc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d919d640 1 -- 172.21.15.59:0/2530961678 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18bc0039b0 msgr2=0x7f18bc003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d919d640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18bc0039b0 0x7f18bc003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d919d640 1 -- 172.21.15.59:0/2530961678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18bc003520 con 0x7f18bc004800 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18d999e640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18bc0039b0 0x7f18bc003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.201+0000 7f18da19f640 1 -- 172.21.15.59:0/2530961678 reap_dead start 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.202+0000 7f18d919d640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18bc004800 0x7f18bc004c80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f18c8003d80 tx=0x7f18bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3816519f2238f370 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.202+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8037070 con 0x7f18bc004800 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.202+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c8032600 con 0x7f18bc004800 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.202+0000 7f18d899c640 1 -- 172.21.15.59:0/2530961678 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8003980 con 0x7f18bc004800 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.202+0000 7f18da9a0640 1 -- 172.21.15.59:0/2530961678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18bc004800 msgr2=0x7f18bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.203+0000 7f18da9a0640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18bc004800 0x7f18bc004c80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f18c8003d80 tx=0x7f18bc005830 comp rx=0 tx=0).stop 2024-03-06T07:24:54.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.203+0000 7f18da9a0640 1 -- 172.21.15.59:0/2530961678 shutdown_connections 2024-03-06T07:24:54.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.203+0000 7f18da9a0640 1 --2- 172.21.15.59:0/2530961678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18bc004800 0x7f18bc004c80 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.203+0000 7f18da9a0640 1 -- 172.21.15.59:0/2530961678 >> 172.21.15.59:0/2530961678 conn(0x7f18d40d3c90 msgr2=0x7f18d40d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.203+0000 7f18da9a0640 1 -- 172.21.15.59:0/2530961678 shutdown_connections 2024-03-06T07:24:54.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.203+0000 7f18da9a0640 1 -- 172.21.15.59:0/2530961678 wait complete. 2024-03-06T07:24:54.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.204+0000 7f18da9a0640 1 Processor -- start 2024-03-06T07:24:54.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.204+0000 7f18da9a0640 1 -- start start 2024-03-06T07:24:54.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.204+0000 7f18da9a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18bc004800 0x7f18d415b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.204+0000 7f18da9a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18d4161e20 0x7f18d40a1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da9a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18d406c6c0 0x7f18d406cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da9a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d40cece0 con 0x7f18d406c6c0 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da9a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d40ceb60 con 0x7f18bc004800 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da9a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d40cee60 con 0x7f18d4161e20 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da19f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18d406c6c0 0x7f18d406cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da19f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18d406c6c0 0x7f18d406cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47660/0 (socket says 172.21.15.59:47660) 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da19f640 1 -- 172.21.15.59:0/4035625770 learned_addr learned my addr 172.21.15.59:0/4035625770 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18d999e640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18bc004800 0x7f18d415b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18d919d640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18d4161e20 0x7f18d40a1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da19f640 1 -- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18d4161e20 msgr2=0x7f18d40a1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da19f640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18d4161e20 0x7f18d40a1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da19f640 1 -- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18bc004800 msgr2=0x7f18d415b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da19f640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18bc004800 0x7f18d415b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da19f640 1 -- 172.21.15.59:0/4035625770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d41622a0 con 0x7f18d406c6c0 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18d999e640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18bc004800 0x7f18d415b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da19f640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18d406c6c0 0x7f18d406cb40 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f18d408e390 tx=0x7f18c400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18d27fc640 1 -- 172.21.15.59:0/4035625770 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c400eb90 con 0x7f18d406c6c0 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18d27fc640 1 -- 172.21.15.59:0/4035625770 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c4018070 con 0x7f18d406c6c0 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da9a0640 1 -- 172.21.15.59:0/4035625770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d40c78f0 con 0x7f18d406c6c0 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18d27fc640 1 -- 172.21.15.59:0/4035625770 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4013420 con 0x7f18d406c6c0 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.205+0000 7f18da9a0640 1 -- 172.21.15.59:0/4035625770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d40c7eb0 con 0x7f18d406c6c0 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.206+0000 7f18da9a0640 1 -- 172.21.15.59:0/4035625770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d406d080 con 0x7f18d406c6c0 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.207+0000 7f18d27fc640 1 -- 172.21.15.59:0/4035625770 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18c40135c0 con 0x7f18d406c6c0 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.207+0000 7f18d27fc640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f18c006b4d0 0x7f18c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.207+0000 7f18d27fc640 1 -- 172.21.15.59:0/4035625770 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f18c408a2f0 con 0x7f18d406c6c0 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.207+0000 7f18d999e640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f18c006b4d0 0x7f18c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.208+0000 7f18d999e640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f18c006b4d0 0x7f18c006d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f18cc006fd0 tx=0x7f18cc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.212+0000 7f18d27fc640 1 -- 172.21.15.59:0/4035625770 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18c4059360 con 0x7f18d406c6c0 2024-03-06T07:24:54.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:54.393+0000 7f18da9a0640 1 -- 172.21.15.59:0/4035625770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f18d4068f10 con 0x7f18d406c6c0 2024-03-06T07:24:55.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.080+0000 7f18d27fc640 1 -- 172.21.15.59:0/4035625770 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v62) v1 ==== 169+0+0 (secure 0 0 0) 0x7f18c4058c60 con 0x7f18d406c6c0 2024-03-06T07:24:55.082 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-03-06T07:24:55.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.083+0000 7f18da9a0640 1 -- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f18c006b4d0 msgr2=0x7f18c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.083+0000 7f18da9a0640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f18c006b4d0 0x7f18c006d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f18cc006fd0 tx=0x7f18cc008040 comp rx=0 tx=0).stop 2024-03-06T07:24:55.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.083+0000 7f18da9a0640 1 -- 172.21.15.59:0/4035625770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18d406c6c0 msgr2=0x7f18d406cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.083+0000 7f18da9a0640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18d406c6c0 0x7f18d406cb40 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f18d408e390 tx=0x7f18c400e530 comp rx=0 tx=0).stop 2024-03-06T07:24:55.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.084+0000 7f18da9a0640 1 -- 172.21.15.59:0/4035625770 shutdown_connections 2024-03-06T07:24:55.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.084+0000 7f18da9a0640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f18c006b4d0 0x7f18c006d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.084+0000 7f18da9a0640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f18d406c6c0 0x7f18d406cb40 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.084+0000 7f18da9a0640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18d4161e20 0x7f18d40a1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.084+0000 7f18da9a0640 1 --2- 172.21.15.59:0/4035625770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18bc004800 0x7f18d415b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.084+0000 7f18da9a0640 1 -- 172.21.15.59:0/4035625770 >> 172.21.15.59:0/4035625770 conn(0x7f18d40d3c90 msgr2=0x7f18d40b9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.084+0000 7f18da9a0640 1 -- 172.21.15.59:0/4035625770 shutdown_connections 2024-03-06T07:24:55.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.084+0000 7f18da9a0640 1 -- 172.21.15.59:0/4035625770 wait complete. 2024-03-06T07:24:55.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-03-06T07:24:55.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.226+0000 7f93ccac2640 1 Processor -- start 2024-03-06T07:24:55.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.226+0000 7f93ccac2640 1 -- start start 2024-03-06T07:24:55.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.227+0000 7f93ccac2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f93c80d4860 0x7f93c80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.227+0000 7f93ccac2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c80b0e30 0x7f93c80b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.227+0000 7f93ccac2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c81618b0 con 0x7f93c81614d0 2024-03-06T07:24:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.227+0000 7f93ccac2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c80d4cf0 con 0x7f93c80d4860 2024-03-06T07:24:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.227+0000 7f93ccac2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c80cf400 con 0x7f93c80b0e30 2024-03-06T07:24:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c6d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c80b0e30 0x7f93c80b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c5d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f93c80d4860 0x7f93c80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c5d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f93c80d4860 0x7f93c80d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:37630/0 (socket says 172.21.15.59:37630) 2024-03-06T07:24:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c6d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c80b0e30 0x7f93c80b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:54616/0 (socket says 172.21.15.59:54616) 2024-03-06T07:24:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c5d9b640 1 -- 172.21.15.59:0/244059349 learned_addr learned my addr 172.21.15.59:0/244059349 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c659c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f93c81614d0 0x7f93c80d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:49500/0 (socket says 172.21.15.59:49500) 2024-03-06T07:24:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c6d9d640 1 -- 172.21.15.59:0/244059349 >> v2:172.21.15.164:3300/0 conn(0x7f93c80d4860 msgr2=0x7f93c80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c6d9d640 1 --2- 172.21.15.59:0/244059349 >> v2:172.21.15.164:3300/0 conn(0x7f93c80d4860 0x7f93c80d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c6d9d640 1 -- 172.21.15.59:0/244059349 >> v1:172.21.15.164:6789/0 conn(0x7f93c81614d0 legacy=0x7f93c80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c559a640 1 -- 172.21.15.59:0/244059349 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601355990 0 0) 0x7f93c81618b0 con 0x7f93c81614d0 2024-03-06T07:24:55.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c6d9d640 1 -- 172.21.15.59:0/244059349 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c80cf6a0 con 0x7f93c80b0e30 2024-03-06T07:24:55.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.228+0000 7f93c5d9b640 1 --2- 172.21.15.59:0/244059349 >> v2:172.21.15.164:3300/0 conn(0x7f93c80d4860 0x7f93c80d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:55.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.229+0000 7f93c6d9d640 1 --2- 172.21.15.59:0/244059349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c80b0e30 0x7f93c80b12a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f93c00097a0 tx=0x7f93c002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd3e7615ea7ed7ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.229+0000 7f93c559a640 1 -- 172.21.15.59:0/244059349 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c0035020 con 0x7f93c80b0e30 2024-03-06T07:24:55.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.229+0000 7f93c559a640 1 -- 172.21.15.59:0/244059349 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c0031810 con 0x7f93c80b0e30 2024-03-06T07:24:55.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.229+0000 7f93c559a640 1 -- 172.21.15.59:0/244059349 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c0031be0 con 0x7f93c80b0e30 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.230+0000 7f93ccac2640 1 -- 172.21.15.59:0/244059349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c80b0e30 msgr2=0x7f93c80b12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.230+0000 7f93ccac2640 1 --2- 172.21.15.59:0/244059349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c80b0e30 0x7f93c80b12a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f93c00097a0 tx=0x7f93c002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.230+0000 7f93ccac2640 1 -- 172.21.15.59:0/244059349 shutdown_connections 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.230+0000 7f93ccac2640 1 --2- 172.21.15.59:0/244059349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c80b0e30 0x7f93c80b12a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.230+0000 7f93ccac2640 1 --2- 172.21.15.59:0/244059349 >> v2:172.21.15.164:3300/0 conn(0x7f93c80d4860 0x7f93c80d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.230+0000 7f93ccac2640 1 -- 172.21.15.59:0/244059349 >> 172.21.15.59:0/244059349 conn(0x7f93c80d3d40 msgr2=0x7f93c80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.230+0000 7f93ccac2640 1 -- 172.21.15.59:0/244059349 shutdown_connections 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.230+0000 7f93ccac2640 1 -- 172.21.15.59:0/244059349 wait complete. 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.231+0000 7f93ccac2640 1 Processor -- start 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.231+0000 7f93ccac2640 1 -- start start 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93ccac2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93c80b0e30 0x7f93c8156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93ccac2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93c80d4860 0x7f93c80b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93ccac2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c81614d0 0x7f93c80b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93ccac2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c80b0910 con 0x7f93c80d4860 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93ccac2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c80cfc60 con 0x7f93c80b0e30 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93ccac2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c80d2560 con 0x7f93c81614d0 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93c5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93c80d4860 0x7f93c80b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93c5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93c80d4860 0x7f93c80b6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47672/0 (socket says 172.21.15.59:47672) 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93c5d9b640 1 -- 172.21.15.59:0/2653016687 learned_addr learned my addr 172.21.15.59:0/2653016687 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93c6d9d640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c81614d0 0x7f93c80b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93c659c640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93c80b0e30 0x7f93c8156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93c5d9b640 1 -- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c81614d0 msgr2=0x7f93c80b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93c5d9b640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c81614d0 0x7f93c80b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93c5d9b640 1 -- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93c80b0e30 msgr2=0x7f93c8156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93c5d9b640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93c80b0e30 0x7f93c8156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.232+0000 7f93c5d9b640 1 -- 172.21.15.59:0/2653016687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c80b72a0 con 0x7f93c80d4860 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.233+0000 7f93c5d9b640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93c80d4860 0x7f93c80b6930 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f93b000e9d0 tx=0x7f93b000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.233+0000 7f93beffd640 1 -- 172.21.15.59:0/2653016687 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b0017070 con 0x7f93c80d4860 2024-03-06T07:24:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.233+0000 7f93beffd640 1 -- 172.21.15.59:0/2653016687 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b000ca60 con 0x7f93c80d4860 2024-03-06T07:24:55.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.233+0000 7f93ccac2640 1 -- 172.21.15.59:0/2653016687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c80b7580 con 0x7f93c80d4860 2024-03-06T07:24:55.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.233+0000 7f93beffd640 1 -- 172.21.15.59:0/2653016687 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b000ce10 con 0x7f93c80d4860 2024-03-06T07:24:55.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.233+0000 7f93ccac2640 1 -- 172.21.15.59:0/2653016687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c80b7b40 con 0x7f93c80d4860 2024-03-06T07:24:55.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.234+0000 7f93ccac2640 1 -- 172.21.15.59:0/2653016687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c8068ff0 con 0x7f93c80d4860 2024-03-06T07:24:55.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.234+0000 7f93beffd640 1 -- 172.21.15.59:0/2653016687 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93b001a070 con 0x7f93c80d4860 2024-03-06T07:24:55.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.235+0000 7f93beffd640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93a806b4d0 0x7f93a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.235+0000 7f93beffd640 1 -- 172.21.15.59:0/2653016687 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f93b008a170 con 0x7f93c80d4860 2024-03-06T07:24:55.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.235+0000 7f93c659c640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93a806b4d0 0x7f93a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.236+0000 7f93c659c640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93a806b4d0 0x7f93a806d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f93b80012e0 tx=0x7f93b8001630 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.239+0000 7f93beffd640 1 -- 172.21.15.59:0/2653016687 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93b0058ca0 con 0x7f93c80d4860 2024-03-06T07:24:55.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.420+0000 7f93ccac2640 1 -- 172.21.15.59:0/2653016687 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7f93c80d0f10 con 0x7f93c80d4860 2024-03-06T07:24:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.423+0000 7f93beffd640 1 -- 172.21.15.59:0/2653016687 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v19) v1 ==== 101+0+0 (secure 0 0 0) 0x7f93b005cd40 con 0x7f93c80d4860 2024-03-06T07:24:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 -- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93a806b4d0 msgr2=0x7f93a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93a806b4d0 0x7f93a806d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f93b80012e0 tx=0x7f93b8001630 comp rx=0 tx=0).stop 2024-03-06T07:24:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 -- 172.21.15.59:0/2653016687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93c80d4860 msgr2=0x7f93c80b6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93c80d4860 0x7f93c80b6930 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f93b000e9d0 tx=0x7f93b000ee90 comp rx=0 tx=0).stop 2024-03-06T07:24:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 -- 172.21.15.59:0/2653016687 shutdown_connections 2024-03-06T07:24:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f93a806b4d0 0x7f93a806d990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c81614d0 0x7f93c80b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93c80d4860 0x7f93c80b6930 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 --2- 172.21.15.59:0/2653016687 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93c80b0e30 0x7f93c8156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 -- 172.21.15.59:0/2653016687 >> 172.21.15.59:0/2653016687 conn(0x7f93c80d3d40 msgr2=0x7f93c806bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 -- 172.21.15.59:0/2653016687 shutdown_connections 2024-03-06T07:24:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.425+0000 7f93ccac2640 1 -- 172.21.15.59:0/2653016687 wait complete. 2024-03-06T07:24:55.438 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:24:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.539+0000 7fc6a7b20640 1 Processor -- start 2024-03-06T07:24:55.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.539+0000 7fc6a7b20640 1 -- start start 2024-03-06T07:24:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.540+0000 7fc6a7b20640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc6a01614d0 0x7fc6a00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.540+0000 7fc6a7b20640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6a00d1910 0x7fc6a00d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.540+0000 7fc6a7b20640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a00d2740 con 0x7fc6a00d22b0 2024-03-06T07:24:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.540+0000 7fc6a7b20640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a01618b0 con 0x7fc6a01614d0 2024-03-06T07:24:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.540+0000 7fc6a7b20640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a00cf400 con 0x7fc6a00d1910 2024-03-06T07:24:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.540+0000 7fc6a58bc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc6a01614d0 0x7fc6a00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.540+0000 7fc6a60bd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc6a00d22b0 0x7fc6a00d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59356/0 (socket says 172.21.15.59:59356) 2024-03-06T07:24:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.540+0000 7fc6a60bd640 1 -- 172.21.15.59:0/572854841 learned_addr learned my addr 172.21.15.59:0/572854841 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.540+0000 7fc6a58bc640 1 -- 172.21.15.59:0/572854841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6a00d1910 msgr2=0x7fc6a00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.540+0000 7fc6a50bb640 1 --2- 172.21.15.59:0/572854841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6a00d1910 0x7fc6a00d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.541+0000 7fc6a58bc640 1 --2- 172.21.15.59:0/572854841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6a00d1910 0x7fc6a00d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.541+0000 7fc6a48ba640 1 -- 172.21.15.59:0/572854841 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206202411 0 0) 0x7fc69c001490 con 0x7fc6a00d22b0 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.541+0000 7fc6a58bc640 1 -- 172.21.15.59:0/572854841 >> v1:172.21.15.59:6789/0 conn(0x7fc6a00d22b0 legacy=0x7fc6a00d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.541+0000 7fc6a58bc640 1 -- 172.21.15.59:0/572854841 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a00cf6a0 con 0x7fc6a01614d0 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.541+0000 7fc6a50bb640 1 --2- 172.21.15.59:0/572854841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6a00d1910 0x7fc6a00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.541+0000 7fc6a58bc640 1 --2- 172.21.15.59:0/572854841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a01614d0 0x7fc6a00d4860 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc6980097a0 tx=0x7fc69802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dfe30e0be7eab0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.541+0000 7fc6a48ba640 1 -- 172.21.15.59:0/572854841 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698035020 con 0x7fc6a01614d0 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.541+0000 7fc6a48ba640 1 -- 172.21.15.59:0/572854841 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc698031810 con 0x7fc6a01614d0 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.541+0000 7fc6a48ba640 1 -- 172.21.15.59:0/572854841 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698031be0 con 0x7fc6a01614d0 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.542+0000 7fc6a7b20640 1 -- 172.21.15.59:0/572854841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a01614d0 msgr2=0x7fc6a00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.542+0000 7fc6a7b20640 1 --2- 172.21.15.59:0/572854841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a01614d0 0x7fc6a00d4860 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc6980097a0 tx=0x7fc69802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.542+0000 7fc6a7b20640 1 -- 172.21.15.59:0/572854841 shutdown_connections 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.542+0000 7fc6a7b20640 1 --2- 172.21.15.59:0/572854841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6a00d1910 0x7fc6a00d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.542+0000 7fc6a7b20640 1 --2- 172.21.15.59:0/572854841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a01614d0 0x7fc6a00d4860 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.542+0000 7fc6a7b20640 1 -- 172.21.15.59:0/572854841 >> 172.21.15.59:0/572854841 conn(0x7fc6a00d3d40 msgr2=0x7fc6a00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.542+0000 7fc6a7b20640 1 -- 172.21.15.59:0/572854841 shutdown_connections 2024-03-06T07:24:55.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.542+0000 7fc6a7b20640 1 -- 172.21.15.59:0/572854841 wait complete. 2024-03-06T07:24:55.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.543+0000 7fc6a7b20640 1 Processor -- start 2024-03-06T07:24:55.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.543+0000 7fc6a7b20640 1 -- start start 2024-03-06T07:24:55.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a7b20640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6a00d1910 0x7fc6a00ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a7b20640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6a00d22b0 0x7fc6a0154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a7b20640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a01614d0 0x7fc6a00cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a7b20640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a00b0910 con 0x7fc6a01614d0 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a7b20640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a00cfc60 con 0x7fc6a00d1910 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a7b20640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a00b1e40 con 0x7fc6a00d22b0 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a60bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a01614d0 0x7fc6a00cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a60bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a01614d0 0x7fc6a00cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47700/0 (socket says 172.21.15.59:47700) 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a60bd640 1 -- 172.21.15.59:0/4041393162 learned_addr learned my addr 172.21.15.59:0/4041393162 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a60bd640 1 -- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6a00d22b0 msgr2=0x7fc6a0154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a60bd640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6a00d22b0 0x7fc6a0154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a60bd640 1 -- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6a00d1910 msgr2=0x7fc6a00ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a58bc640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6a00d1910 0x7fc6a00ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a60bd640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6a00d1910 0x7fc6a00ca3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a60bd640 1 -- 172.21.15.59:0/4041393162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a00cb040 con 0x7fc6a01614d0 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.544+0000 7fc6a60bd640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a01614d0 0x7fc6a00cce60 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00c820 tx=0x7fc69c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.545+0000 7fc6967fc640 1 -- 172.21.15.59:0/4041393162 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c018070 con 0x7fc6a01614d0 2024-03-06T07:24:55.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.545+0000 7fc6967fc640 1 -- 172.21.15.59:0/4041393162 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69c00a5e0 con 0x7fc6a01614d0 2024-03-06T07:24:55.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.545+0000 7fc6a7b20640 1 -- 172.21.15.59:0/4041393162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a00cb4f0 con 0x7fc6a01614d0 2024-03-06T07:24:55.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.545+0000 7fc6967fc640 1 -- 172.21.15.59:0/4041393162 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c00a9d0 con 0x7fc6a01614d0 2024-03-06T07:24:55.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.545+0000 7fc6a7b20640 1 -- 172.21.15.59:0/4041393162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a00cbab0 con 0x7fc6a01614d0 2024-03-06T07:24:55.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.546+0000 7fc6a7b20640 1 -- 172.21.15.59:0/4041393162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a00cc280 con 0x7fc6a01614d0 2024-03-06T07:24:55.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.546+0000 7fc6967fc640 1 -- 172.21.15.59:0/4041393162 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc69c01b070 con 0x7fc6a01614d0 2024-03-06T07:24:55.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.546+0000 7fc6967fc640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc68406b4d0 0x7fc68406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.547+0000 7fc6967fc640 1 -- 172.21.15.59:0/4041393162 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc69c089410 con 0x7fc6a01614d0 2024-03-06T07:24:55.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.547+0000 7fc6a58bc640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc68406b4d0 0x7fc68406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.547+0000 7fc6a58bc640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc68406b4d0 0x7fc68406d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc698036000 tx=0x7fc698008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.551+0000 7fc6967fc640 1 -- 172.21.15.59:0/4041393162 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6a00d2740 con 0x7fc6a01614d0 2024-03-06T07:24:55.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.731+0000 7fc6a7b20640 1 -- 172.21.15.59:0/4041393162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc6a0068ff0 con 0x7fc6a01614d0 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.735+0000 7fc6967fc640 1 -- 172.21.15.59:0/4041393162 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v20) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc69c058620 con 0x7fc6a01614d0 2024-03-06T07:24:55.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 -- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc68406b4d0 msgr2=0x7fc68406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc68406b4d0 0x7fc68406d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc698036000 tx=0x7fc698008030 comp rx=0 tx=0).stop 2024-03-06T07:24:55.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 -- 172.21.15.59:0/4041393162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a01614d0 msgr2=0x7fc6a00cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a01614d0 0x7fc6a00cce60 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00c820 tx=0x7fc69c00cce0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 -- 172.21.15.59:0/4041393162 shutdown_connections 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc68406b4d0 0x7fc68406d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a01614d0 0x7fc6a00cce60 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6a00d22b0 0x7fc6a0154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 --2- 172.21.15.59:0/4041393162 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6a00d1910 0x7fc6a00ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 -- 172.21.15.59:0/4041393162 >> 172.21.15.59:0/4041393162 conn(0x7fc6a00d3d40 msgr2=0x7fc6a006ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 -- 172.21.15.59:0/4041393162 shutdown_connections 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.737+0000 7fc6a7b20640 1 -- 172.21.15.59:0/4041393162 wait complete. 2024-03-06T07:24:55.750 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:24:55.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.851+0000 7f009aafa640 1 Processor -- start 2024-03-06T07:24:55.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.851+0000 7f009aafa640 1 -- start start 2024-03-06T07:24:55.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f009aafa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f00940b41b0 0x7f00940cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f009aafa640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f00940b15c0 0x7f00940b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f009aafa640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00940d0240 con 0x7f009415bdd0 2024-03-06T07:24:55.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f009aafa640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00940cf4f0 con 0x7f00940b41b0 2024-03-06T07:24:55.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f009aafa640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00940d0850 con 0x7f00940b15c0 2024-03-06T07:24:55.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f0093fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f00940b41b0 0x7f00940cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f0093fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f00940b41b0 0x7f00940cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47710/0 (socket says 172.21.15.59:47710) 2024-03-06T07:24:55.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f0093fff640 1 -- 172.21.15.59:0/256734237 learned_addr learned my addr 172.21.15.59:0/256734237 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:55.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f0099097640 1 --2- 172.21.15.59:0/256734237 >> v2:172.21.15.164:3300/0 conn(0x7f00940b15c0 0x7f00940b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f0093fff640 1 -- 172.21.15.59:0/256734237 >> v2:172.21.15.164:3300/0 conn(0x7f00940b15c0 msgr2=0x7f00940b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f0093fff640 1 --2- 172.21.15.59:0/256734237 >> v2:172.21.15.164:3300/0 conn(0x7f00940b15c0 0x7f00940b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.852+0000 7f0093fff640 1 -- 172.21.15.59:0/256734237 >> v1:172.21.15.164:6789/0 conn(0x7f009415bdd0 legacy=0x7f00940ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.853+0000 7f0093fff640 1 -- 172.21.15.59:0/256734237 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00940cef50 con 0x7f00940b41b0 2024-03-06T07:24:55.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.853+0000 7f0093fff640 1 --2- 172.21.15.59:0/256734237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00940b41b0 0x7f00940cea10 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0094072b70 tx=0x7f008802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad76563db27d4de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.853+0000 7f00937fe640 1 -- 172.21.15.59:0/256734237 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0088035020 con 0x7f00940b41b0 2024-03-06T07:24:55.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.853+0000 7f00937fe640 1 -- 172.21.15.59:0/256734237 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0088031870 con 0x7f00940b41b0 2024-03-06T07:24:55.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.853+0000 7f00937fe640 1 -- 172.21.15.59:0/256734237 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0088031c40 con 0x7f00940b41b0 2024-03-06T07:24:55.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.854+0000 7f009aafa640 1 -- 172.21.15.59:0/256734237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00940b41b0 msgr2=0x7f00940cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.854+0000 7f009aafa640 1 --2- 172.21.15.59:0/256734237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00940b41b0 0x7f00940cea10 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0094072b70 tx=0x7f008802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:55.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.854+0000 7f009aafa640 1 -- 172.21.15.59:0/256734237 shutdown_connections 2024-03-06T07:24:55.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.854+0000 7f009aafa640 1 --2- 172.21.15.59:0/256734237 >> v2:172.21.15.164:3300/0 conn(0x7f00940b15c0 0x7f00940b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.854+0000 7f009aafa640 1 --2- 172.21.15.59:0/256734237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00940b41b0 0x7f00940cea10 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.854+0000 7f009aafa640 1 -- 172.21.15.59:0/256734237 >> 172.21.15.59:0/256734237 conn(0x7f00940b3690 msgr2=0x7f00940b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.854+0000 7f009aafa640 1 -- 172.21.15.59:0/256734237 shutdown_connections 2024-03-06T07:24:55.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.854+0000 7f009aafa640 1 -- 172.21.15.59:0/256734237 wait complete. 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.855+0000 7f009aafa640 1 Processor -- start 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.855+0000 7f009aafa640 1 -- start start 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f009aafa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00940b15c0 0x7f00941f1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f009aafa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00940b41b0 0x7f00940c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f009aafa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f009415bdd0 0x7f009415e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f009aafa640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009415c1b0 con 0x7f00940b41b0 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f009aafa640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00940d04e0 con 0x7f009415bdd0 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f009aafa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00940b10c0 con 0x7f00940b15c0 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f0093fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00940b41b0 0x7f00940c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f0093fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00940b41b0 0x7f00940c5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47712/0 (socket says 172.21.15.59:47712) 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f0093fff640 1 -- 172.21.15.59:0/142664047 learned_addr learned my addr 172.21.15.59:0/142664047 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:55.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f0099097640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f009415bdd0 0x7f009415e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f0098896640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00940b15c0 0x7f00941f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f0093fff640 1 -- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00940b15c0 msgr2=0x7f00941f1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f0093fff640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00940b15c0 0x7f00941f1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f0093fff640 1 -- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f009415bdd0 msgr2=0x7f009415e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f0093fff640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f009415bdd0 0x7f009415e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f0093fff640 1 -- 172.21.15.59:0/142664047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009415df10 con 0x7f00940b41b0 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.856+0000 7f0099097640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f009415bdd0 0x7f009415e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.857+0000 7f0093fff640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00940b41b0 0x7f00940c5360 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f008802b950 tx=0x7f00880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.857+0000 7f00917fa640 1 -- 172.21.15.59:0/142664047 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0088040070 con 0x7f00940b41b0 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.857+0000 7f00917fa640 1 -- 172.21.15.59:0/142664047 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008803a420 con 0x7f00940b41b0 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.857+0000 7f00917fa640 1 -- 172.21.15.59:0/142664047 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0088035040 con 0x7f00940b41b0 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.857+0000 7f009aafa640 1 -- 172.21.15.59:0/142664047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00940c7f00 con 0x7f00940b41b0 2024-03-06T07:24:55.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.857+0000 7f009aafa640 1 -- 172.21.15.59:0/142664047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00941f1ec0 con 0x7f00940b41b0 2024-03-06T07:24:55.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.858+0000 7f009aafa640 1 -- 172.21.15.59:0/142664047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009415dd40 con 0x7f00940b41b0 2024-03-06T07:24:55.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.858+0000 7f00917fa640 1 -- 172.21.15.59:0/142664047 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0088038020 con 0x7f00940b41b0 2024-03-06T07:24:55.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.859+0000 7f00917fa640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f008006b4d0 0x7f008006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.859+0000 7f00917fa640 1 -- 172.21.15.59:0/142664047 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f00880abf60 con 0x7f00940b41b0 2024-03-06T07:24:55.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.859+0000 7f0098896640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f008006b4d0 0x7f008006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.859+0000 7f0098896640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f008006b4d0 0x7f008006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f008c009f60 tx=0x7f008c0094f0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:55.863+0000 7f00917fa640 1 -- 172.21.15.59:0/142664047 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f008807b0e0 con 0x7f00940b41b0 2024-03-06T07:24:56.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.044+0000 7f009aafa640 1 -- 172.21.15.59:0/142664047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f009415e0e0 con 0x7f00940b41b0 2024-03-06T07:24:56.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.047+0000 7f00917fa640 1 -- 172.21.15.59:0/142664047 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v21) v1 ==== 108+0+0 (secure 0 0 0) 0x7f008807a9e0 con 0x7f00940b41b0 2024-03-06T07:24:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.049+0000 7f009aafa640 1 -- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f008006b4d0 msgr2=0x7f008006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.049+0000 7f009aafa640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f008006b4d0 0x7f008006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f008c009f60 tx=0x7f008c0094f0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.049+0000 7f009aafa640 1 -- 172.21.15.59:0/142664047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00940b41b0 msgr2=0x7f00940c5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.049+0000 7f009aafa640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00940b41b0 0x7f00940c5360 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f008802b950 tx=0x7f00880046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.049+0000 7f009aafa640 1 -- 172.21.15.59:0/142664047 shutdown_connections 2024-03-06T07:24:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.049+0000 7f009aafa640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f008006b4d0 0x7f008006d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.049+0000 7f009aafa640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f009415bdd0 0x7f009415e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.049+0000 7f009aafa640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f00940b41b0 0x7f00940c5360 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.049+0000 7f009aafa640 1 --2- 172.21.15.59:0/142664047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00940b15c0 0x7f00941f1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.049+0000 7f009aafa640 1 -- 172.21.15.59:0/142664047 >> 172.21.15.59:0/142664047 conn(0x7f00940b3690 msgr2=0x7f00940cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.050+0000 7f009aafa640 1 -- 172.21.15.59:0/142664047 shutdown_connections 2024-03-06T07:24:56.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.050+0000 7f009aafa640 1 -- 172.21.15.59:0/142664047 wait complete. 2024-03-06T07:24:56.062 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:24:56.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.162+0000 7f265bda1640 1 Processor -- start 2024-03-06T07:24:56.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.162+0000 7f265bda1640 1 -- start start 2024-03-06T07:24:56.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.163+0000 7f265bda1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f265415ab00 0x7f26540b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.163+0000 7f265bda1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f26540b48f0 0x7f26540af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.163+0000 7f265bda1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26540afc00 0x7f265406e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.163+0000 7f265bda1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26540b4e30 con 0x7f265415ab00 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.163+0000 7f265bda1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26540b5110 con 0x7f26540b48f0 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.163+0000 7f265bda1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265415b310 con 0x7f26540afc00 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.163+0000 7f2659b3d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f265415ab00 0x7f26540b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.163+0000 7f2659b3d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f265415ab00 0x7f26540b43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47718/0 (socket says 172.21.15.59:47718) 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.163+0000 7f2659b3d640 1 -- 172.21.15.59:0/2782951431 learned_addr learned my addr 172.21.15.59:0/2782951431 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.163+0000 7f265a33e640 1 --2- 172.21.15.59:0/2782951431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26540afc00 0x7f265406e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.163+0000 7f265933c640 1 --2- 172.21.15.59:0/2782951431 >> v2:172.21.15.164:3300/0 conn(0x7f26540b48f0 0x7f26540af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.164+0000 7f2659b3d640 1 -- 172.21.15.59:0/2782951431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26540afc00 msgr2=0x7f265406e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.164+0000 7f2659b3d640 1 --2- 172.21.15.59:0/2782951431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26540afc00 0x7f265406e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.164+0000 7f2659b3d640 1 -- 172.21.15.59:0/2782951431 >> v2:172.21.15.164:3300/0 conn(0x7f26540b48f0 msgr2=0x7f26540af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.164+0000 7f2659b3d640 1 --2- 172.21.15.59:0/2782951431 >> v2:172.21.15.164:3300/0 conn(0x7f26540b48f0 0x7f26540af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.164+0000 7f2659b3d640 1 -- 172.21.15.59:0/2782951431 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265415b5f0 con 0x7f265415ab00 2024-03-06T07:24:56.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.164+0000 7f2659b3d640 1 --2- 172.21.15.59:0/2782951431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f265415ab00 0x7f26540b43b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2654073720 tx=0x7f264402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92c3a27088d7efe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.164+0000 7f2658b3b640 1 -- 172.21.15.59:0/2782951431 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264402ca70 con 0x7f265415ab00 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.165+0000 7f2658b3b640 1 -- 172.21.15.59:0/2782951431 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264402cc10 con 0x7f265415ab00 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.165+0000 7f2658b3b640 1 -- 172.21.15.59:0/2782951431 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26440343f0 con 0x7f265415ab00 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.165+0000 7f265bda1640 1 -- 172.21.15.59:0/2782951431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f265415ab00 msgr2=0x7f26540b43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.165+0000 7f265bda1640 1 --2- 172.21.15.59:0/2782951431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f265415ab00 0x7f26540b43b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2654073720 tx=0x7f264402bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.165+0000 7f265bda1640 1 -- 172.21.15.59:0/2782951431 shutdown_connections 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.165+0000 7f265bda1640 1 --2- 172.21.15.59:0/2782951431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f26540afc00 0x7f265406e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.165+0000 7f265bda1640 1 --2- 172.21.15.59:0/2782951431 >> v2:172.21.15.164:3300/0 conn(0x7f26540b48f0 0x7f26540af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.165+0000 7f265bda1640 1 --2- 172.21.15.59:0/2782951431 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f265415ab00 0x7f26540b43b0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.165+0000 7f265bda1640 1 -- 172.21.15.59:0/2782951431 >> 172.21.15.59:0/2782951431 conn(0x7f265406c0c0 msgr2=0x7f265406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.165+0000 7f265bda1640 1 -- 172.21.15.59:0/2782951431 shutdown_connections 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.165+0000 7f265bda1640 1 -- 172.21.15.59:0/2782951431 wait complete. 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.166+0000 7f265bda1640 1 Processor -- start 2024-03-06T07:24:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.166+0000 7f265bda1640 1 -- start start 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f265bda1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26540afc00 0x7f2654158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f265bda1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26540b48f0 0x7f2654142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f265bda1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f265415ab00 0x7f265413f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f265bda1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265415b060 con 0x7f26540afc00 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f265bda1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265415aee0 con 0x7f26540b48f0 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f265bda1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265406ecc0 con 0x7f265415ab00 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f2659b3d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26540afc00 0x7f2654158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f2659b3d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26540afc00 0x7f2654158700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47732/0 (socket says 172.21.15.59:47732) 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f2659b3d640 1 -- 172.21.15.59:0/349337909 learned_addr learned my addr 172.21.15.59:0/349337909 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f265a33e640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f265415ab00 0x7f265413f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f265933c640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26540b48f0 0x7f2654142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f2659b3d640 1 -- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f265415ab00 msgr2=0x7f265413f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f2659b3d640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f265415ab00 0x7f265413f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f2659b3d640 1 -- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26540b48f0 msgr2=0x7f2654142cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f2659b3d640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26540b48f0 0x7f2654142cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f2659b3d640 1 -- 172.21.15.59:0/349337909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2654154690 con 0x7f26540afc00 2024-03-06T07:24:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.167+0000 7f265933c640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26540b48f0 0x7f2654142cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.168+0000 7f2659b3d640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26540afc00 0x7f2654158700 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f2644036040 tx=0x7f2644004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.168+0000 7f26527fc640 1 -- 172.21.15.59:0/349337909 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2644043020 con 0x7f26540afc00 2024-03-06T07:24:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.168+0000 7f26527fc640 1 -- 172.21.15.59:0/349337909 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264403e450 con 0x7f26540afc00 2024-03-06T07:24:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.168+0000 7f26527fc640 1 -- 172.21.15.59:0/349337909 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2644009510 con 0x7f26540afc00 2024-03-06T07:24:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.168+0000 7f265bda1640 1 -- 172.21.15.59:0/349337909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2654154970 con 0x7f26540afc00 2024-03-06T07:24:56.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.168+0000 7f265bda1640 1 -- 172.21.15.59:0/349337909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2654154f30 con 0x7f26540afc00 2024-03-06T07:24:56.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.169+0000 7f265bda1640 1 -- 172.21.15.59:0/349337909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265404f330 con 0x7f26540afc00 2024-03-06T07:24:56.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.169+0000 7f26527fc640 1 -- 172.21.15.59:0/349337909 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2644046020 con 0x7f26540afc00 2024-03-06T07:24:56.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.170+0000 7f26527fc640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f264006b4d0 0x7f264006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.170+0000 7f26527fc640 1 -- 172.21.15.59:0/349337909 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f26440ac6a0 con 0x7f26540afc00 2024-03-06T07:24:56.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.170+0000 7f265933c640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f264006b4d0 0x7f264006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.170+0000 7f265933c640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f264006b4d0 0x7f264006d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f26540cdcc0 tx=0x7f2648008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.174+0000 7f26527fc640 1 -- 172.21.15.59:0/349337909 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f264404d030 con 0x7f26540afc00 2024-03-06T07:24:56.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:56.427+0000 7f265bda1640 1 -- 172.21.15.59:0/349337909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f26540b4490 con 0x7f26540afc00 2024-03-06T07:24:57.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.086+0000 7f26527fc640 1 -- 172.21.15.59:0/349337909 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v185) v1 ==== 144+0+0 (secure 0 0 0) 0x7f264407b8b0 con 0x7f26540afc00 2024-03-06T07:24:57.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 -- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f264006b4d0 msgr2=0x7f264006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f264006b4d0 0x7f264006d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f26540cdcc0 tx=0x7f2648008040 comp rx=0 tx=0).stop 2024-03-06T07:24:57.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 -- 172.21.15.59:0/349337909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26540afc00 msgr2=0x7f2654158700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26540afc00 0x7f2654158700 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f2644036040 tx=0x7f2644004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:57.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 -- 172.21.15.59:0/349337909 shutdown_connections 2024-03-06T07:24:57.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f264006b4d0 0x7f264006d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f265415ab00 0x7f265413f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f26540b48f0 0x7f2654142cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 --2- 172.21.15.59:0/349337909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26540afc00 0x7f2654158700 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 -- 172.21.15.59:0/349337909 >> 172.21.15.59:0/349337909 conn(0x7f265406c0c0 msgr2=0x7f2654071ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 -- 172.21.15.59:0/349337909 shutdown_connections 2024-03-06T07:24:57.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.089+0000 7f265bda1640 1 -- 172.21.15.59:0/349337909 wait complete. 2024-03-06T07:24:57.104 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-03-06T07:24:57.105 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-03-06T07:24:57.105 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:24:57.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:57.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.212+0000 7fcda565c640 1 Processor -- start 2024-03-06T07:24:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.212+0000 7fcda565c640 1 -- start start 2024-03-06T07:24:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.212+0000 7fcda565c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcda00cfb10 0x7fcda00d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.212+0000 7fcda565c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda00b1070 con 0x7fcda0162020 2024-03-06T07:24:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.212+0000 7fcda565c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda00d2500 con 0x7fcda0162a00 2024-03-06T07:24:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.212+0000 7fcda565c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda00ce9d0 con 0x7fcda00cfb10 2024-03-06T07:24:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.213+0000 7fcd9f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcda00cfb10 0x7fcda00d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.213+0000 7fcd9f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcda00cfb10 0x7fcda00d4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47738/0 (socket says 172.21.15.59:47738) 2024-03-06T07:24:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.213+0000 7fcd9f7fe640 1 -- 172.21.15.59:0/101926969 learned_addr learned my addr 172.21.15.59:0/101926969 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.213+0000 7fcd9effd640 1 --1- 172.21.15.59:0/101926969 >> v1:172.21.15.59:6789/0 conn(0x7fcda0162020 0x7fcda00b0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59372/0 (socket says 172.21.15.59:59372) 2024-03-06T07:24:57.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.213+0000 7fcd9f7fe640 1 -- 172.21.15.59:0/101926969 >> v1:172.21.15.164:6789/0 conn(0x7fcda0162a00 legacy=0x7fcda00cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.213+0000 7fcd9f7fe640 1 -- 172.21.15.59:0/101926969 >> v1:172.21.15.59:6789/0 conn(0x7fcda0162020 legacy=0x7fcda00b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.213+0000 7fcd9f7fe640 1 -- 172.21.15.59:0/101926969 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda00d27e0 con 0x7fcda00cfb10 2024-03-06T07:24:57.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.214+0000 7fcd9f7fe640 1 --2- 172.21.15.59:0/101926969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcda00cfb10 0x7fcda00d4830 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009e50 tx=0x7fcd9002c090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5886ad3481915014 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.214+0000 7fcd9dffb640 1 -- 172.21.15.59:0/101926969 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9002cc30 con 0x7fcda00cfb10 2024-03-06T07:24:57.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.214+0000 7fcd9dffb640 1 -- 172.21.15.59:0/101926969 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9002cdd0 con 0x7fcda00cfb10 2024-03-06T07:24:57.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.214+0000 7fcd9dffb640 1 -- 172.21.15.59:0/101926969 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90034610 con 0x7fcda00cfb10 2024-03-06T07:24:57.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.214+0000 7fcda565c640 1 -- 172.21.15.59:0/101926969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcda00cfb10 msgr2=0x7fcda00d4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.214+0000 7fcda565c640 1 --2- 172.21.15.59:0/101926969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcda00cfb10 0x7fcda00d4830 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009e50 tx=0x7fcd9002c090 comp rx=0 tx=0).stop 2024-03-06T07:24:57.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.214+0000 7fcda565c640 1 -- 172.21.15.59:0/101926969 shutdown_connections 2024-03-06T07:24:57.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.214+0000 7fcda565c640 1 --2- 172.21.15.59:0/101926969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcda00cfb10 0x7fcda00d4830 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.214+0000 7fcda565c640 1 -- 172.21.15.59:0/101926969 >> 172.21.15.59:0/101926969 conn(0x7fcda00d3d10 msgr2=0x7fcda00d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.214+0000 7fcda565c640 1 -- 172.21.15.59:0/101926969 shutdown_connections 2024-03-06T07:24:57.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.214+0000 7fcda565c640 1 -- 172.21.15.59:0/101926969 wait complete. 2024-03-06T07:24:57.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.216+0000 7fcda565c640 1 Processor -- start 2024-03-06T07:24:57.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.216+0000 7fcda565c640 1 -- start start 2024-03-06T07:24:57.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.216+0000 7fcda565c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcda00cfb10 0x7fcda00b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.216+0000 7fcda565c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcda0162020 0x7fcda015ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.216+0000 7fcda565c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcda0162a00 0x7fcda01604c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.216+0000 7fcda565c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda00cec00 con 0x7fcda0162020 2024-03-06T07:24:57.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.216+0000 7fcda565c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda00d2da0 con 0x7fcda00cfb10 2024-03-06T07:24:57.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.216+0000 7fcda565c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda0162400 con 0x7fcda0162a00 2024-03-06T07:24:57.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd9e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcda0162020 0x7fcda015ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd9e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcda0162020 0x7fcda015ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47740/0 (socket says 172.21.15.59:47740) 2024-03-06T07:24:57.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd9e7fc640 1 -- 172.21.15.59:0/2929711959 learned_addr learned my addr 172.21.15.59:0/2929711959 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:57.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd9effd640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcda00cfb10 0x7fcda00b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd9f7fe640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcda0162a00 0x7fcda01604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd9e7fc640 1 -- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcda0162a00 msgr2=0x7fcda01604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd9e7fc640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcda0162a00 0x7fcda01604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd9e7fc640 1 -- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcda00cfb10 msgr2=0x7fcda00b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd9e7fc640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcda00cfb10 0x7fcda00b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd9e7fc640 1 -- 172.21.15.59:0/2929711959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda00c79e0 con 0x7fcda0162020 2024-03-06T07:24:57.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd9e7fc640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcda0162020 0x7fcda015ff80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800c440 tx=0x7fcd8800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd977fe640 1 -- 172.21.15.59:0/2929711959 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd880052d0 con 0x7fcda0162020 2024-03-06T07:24:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd977fe640 1 -- 172.21.15.59:0/2929711959 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd88005470 con 0x7fcda0162020 2024-03-06T07:24:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcda565c640 1 -- 172.21.15.59:0/2929711959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda00c7cc0 con 0x7fcda0162020 2024-03-06T07:24:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcd977fe640 1 -- 172.21.15.59:0/2929711959 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd88007260 con 0x7fcda0162020 2024-03-06T07:24:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.217+0000 7fcda565c640 1 -- 172.21.15.59:0/2929711959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda00b4e20 con 0x7fcda0162020 2024-03-06T07:24:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.218+0000 7fcda565c640 1 -- 172.21.15.59:0/2929711959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda00c82d0 con 0x7fcda0162020 2024-03-06T07:24:57.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.219+0000 7fcd977fe640 1 -- 172.21.15.59:0/2929711959 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd88007400 con 0x7fcda0162020 2024-03-06T07:24:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.219+0000 7fcd977fe640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8006b500 0x7fcd8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.219+0000 7fcd977fe640 1 -- 172.21.15.59:0/2929711959 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcd880893a0 con 0x7fcda0162020 2024-03-06T07:24:57.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.219+0000 7fcd9effd640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8006b500 0x7fcd8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.220+0000 7fcd9effd640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8006b500 0x7fcd8006d9c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcd98000f60 tx=0x7fcd98007c80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.224+0000 7fcd977fe640 1 -- 172.21.15.59:0/2929711959 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd88058490 con 0x7fcda0162020 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.406+0000 7fcda565c640 1 -- 172.21.15.59:0/2929711959 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcda004f330 con 0x7fcda0162020 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.406+0000 7fcd977fe640 1 -- 172.21.15.59:0/2929711959 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+13006 (secure 0 0 0) 0x7fcd88057d90 con 0x7fcda0162020 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:57.408 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":62,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:24:55.072851+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6835/750409746":"2024-03-07T07:24:52.060517+0000","172.21.15.59:6834/750409746":"2024-03-07T07:24:52.060517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:57.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 -- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8006b500 msgr2=0x7fcd8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8006b500 0x7fcd8006d9c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcd98000f60 tx=0x7fcd98007c80 comp rx=0 tx=0).stop 2024-03-06T07:24:57.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 -- 172.21.15.59:0/2929711959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcda0162020 msgr2=0x7fcda015ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcda0162020 0x7fcda015ff80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800c440 tx=0x7fcd8800c900 comp rx=0 tx=0).stop 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 -- 172.21.15.59:0/2929711959 shutdown_connections 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8006b500 0x7fcd8006d9c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcda0162a00 0x7fcda01604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcda0162020 0x7fcda015ff80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 --2- 172.21.15.59:0/2929711959 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcda00cfb10 0x7fcda00b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 -- 172.21.15.59:0/2929711959 >> 172.21.15.59:0/2929711959 conn(0x7fcda00d3d10 msgr2=0x7fcda006ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 -- 172.21.15.59:0/2929711959 shutdown_connections 2024-03-06T07:24:57.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.408+0000 7fcda565c640 1 -- 172.21.15.59:0/2929711959 wait complete. 2024-03-06T07:24:57.422 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:57.422 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.524+0000 7fa6c740f640 1 Processor -- start 2024-03-06T07:24:57.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.524+0000 7fa6c740f640 1 -- start start 2024-03-06T07:24:57.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.525+0000 7fa6c740f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa6c00cea50 0x7fa6c00b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.525+0000 7fa6c740f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6c006cab0 0x7fa6c00ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.525+0000 7fa6c740f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c006d030 con 0x7fa6c015def0 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.525+0000 7fa6c740f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c00cf1c0 con 0x7fa6c00cea50 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.525+0000 7fa6c740f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c00cc760 con 0x7fa6c006cab0 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.525+0000 7fa6c49aa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa6c00cea50 0x7fa6c00b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.525+0000 7fa6c49aa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa6c00cea50 0x7fa6c00b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47742/0 (socket says 172.21.15.59:47742) 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.525+0000 7fa6c49aa640 1 -- 172.21.15.59:0/1018684541 learned_addr learned my addr 172.21.15.59:0/1018684541 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.525+0000 7fa6c59ac640 1 --2- 172.21.15.59:0/1018684541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6c006cab0 0x7fa6c00ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.526+0000 7fa6c49aa640 1 -- 172.21.15.59:0/1018684541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6c006cab0 msgr2=0x7fa6c00ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.526+0000 7fa6c49aa640 1 --2- 172.21.15.59:0/1018684541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6c006cab0 0x7fa6c00ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.526+0000 7fa6c49aa640 1 -- 172.21.15.59:0/1018684541 >> v1:172.21.15.164:6789/0 conn(0x7fa6c015def0 legacy=0x7fa6c00ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.526+0000 7fa6c49aa640 1 -- 172.21.15.59:0/1018684541 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c006d310 con 0x7fa6c00cea50 2024-03-06T07:24:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.526+0000 7fa6c49aa640 1 --2- 172.21.15.59:0/1018684541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6c00cea50 0x7fa6c00b41b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009870 tx=0x7fa6b402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255567c8fecc84b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.526+0000 7fa6bffff640 1 -- 172.21.15.59:0/1018684541 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b402f840 con 0x7fa6c00cea50 2024-03-06T07:24:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.527+0000 7fa6bffff640 1 -- 172.21.15.59:0/1018684541 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b402f9e0 con 0x7fa6c00cea50 2024-03-06T07:24:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.527+0000 7fa6bffff640 1 -- 172.21.15.59:0/1018684541 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b402fdb0 con 0x7fa6c00cea50 2024-03-06T07:24:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.527+0000 7fa6c740f640 1 -- 172.21.15.59:0/1018684541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6c00cea50 msgr2=0x7fa6c00b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.527+0000 7fa6c740f640 1 --2- 172.21.15.59:0/1018684541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6c00cea50 0x7fa6c00b41b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009870 tx=0x7fa6b402ba20 comp rx=0 tx=0).stop 2024-03-06T07:24:57.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.527+0000 7fa6c740f640 1 -- 172.21.15.59:0/1018684541 shutdown_connections 2024-03-06T07:24:57.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.527+0000 7fa6c740f640 1 --2- 172.21.15.59:0/1018684541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6c006cab0 0x7fa6c00ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.527+0000 7fa6c740f640 1 --2- 172.21.15.59:0/1018684541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6c00cea50 0x7fa6c00b41b0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.527+0000 7fa6c740f640 1 -- 172.21.15.59:0/1018684541 >> 172.21.15.59:0/1018684541 conn(0x7fa6c00b3690 msgr2=0x7fa6c00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.527+0000 7fa6c740f640 1 -- 172.21.15.59:0/1018684541 shutdown_connections 2024-03-06T07:24:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.527+0000 7fa6c740f640 1 -- 172.21.15.59:0/1018684541 wait complete. 2024-03-06T07:24:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.528+0000 7fa6c740f640 1 Processor -- start 2024-03-06T07:24:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.528+0000 7fa6c740f640 1 -- start start 2024-03-06T07:24:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c740f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6c006cab0 0x7fa6c015bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c740f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6c00cea50 0x7fa6c00c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c740f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6c015def0 0x7fa6c00b6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c740f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c015e2d0 con 0x7fa6c00cea50 2024-03-06T07:24:57.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c740f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c00cc3f0 con 0x7fa6c006cab0 2024-03-06T07:24:57.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c740f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c015e450 con 0x7fa6c015def0 2024-03-06T07:24:57.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c49aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6c00cea50 0x7fa6c00c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c49aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6c00cea50 0x7fa6c00c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47758/0 (socket says 172.21.15.59:47758) 2024-03-06T07:24:57.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c49aa640 1 -- 172.21.15.59:0/2805030036 learned_addr learned my addr 172.21.15.59:0/2805030036 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:57.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c51ab640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6c006cab0 0x7fa6c015bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c59ac640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6c015def0 0x7fa6c00b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c49aa640 1 -- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6c015def0 msgr2=0x7fa6c00b6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c49aa640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6c015def0 0x7fa6c00b6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c49aa640 1 -- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6c006cab0 msgr2=0x7fa6c015bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c49aa640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6c006cab0 0x7fa6c015bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c49aa640 1 -- 172.21.15.59:0/2805030036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c00b6ee0 con 0x7fa6c00cea50 2024-03-06T07:24:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.529+0000 7fa6c59ac640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6c015def0 0x7fa6c00b6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.530+0000 7fa6c49aa640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6c00cea50 0x7fa6c00c61a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4000c00 tx=0x7fa6b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.530+0000 7fa6bdffb640 1 -- 172.21.15.59:0/2805030036 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4037c90 con 0x7fa6c00cea50 2024-03-06T07:24:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.530+0000 7fa6bdffb640 1 -- 172.21.15.59:0/2805030036 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b4046070 con 0x7fa6c00cea50 2024-03-06T07:24:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.530+0000 7fa6c740f640 1 -- 172.21.15.59:0/2805030036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c00c56c0 con 0x7fa6c00cea50 2024-03-06T07:24:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.530+0000 7fa6bdffb640 1 -- 172.21.15.59:0/2805030036 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4041640 con 0x7fa6c00cea50 2024-03-06T07:24:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.530+0000 7fa6c740f640 1 -- 172.21.15.59:0/2805030036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c00c5c80 con 0x7fa6c00cea50 2024-03-06T07:24:57.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.531+0000 7fa6c740f640 1 -- 172.21.15.59:0/2805030036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c009fcf0 con 0x7fa6c00cea50 2024-03-06T07:24:57.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.531+0000 7fa6bdffb640 1 -- 172.21.15.59:0/2805030036 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6b4040070 con 0x7fa6c00cea50 2024-03-06T07:24:57.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.532+0000 7fa6bdffb640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa6ac06b4d0 0x7fa6ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.532+0000 7fa6bdffb640 1 -- 172.21.15.59:0/2805030036 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa6b40ac5d0 con 0x7fa6c00cea50 2024-03-06T07:24:57.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.532+0000 7fa6c51ab640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa6ac06b4d0 0x7fa6ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.532+0000 7fa6c51ab640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa6ac06b4d0 0x7fa6ac06d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8007fd0 tx=0x7fa6b8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.536+0000 7fa6bdffb640 1 -- 172.21.15.59:0/2805030036 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6b4033030 con 0x7fa6c00cea50 2024-03-06T07:24:57.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.717+0000 7fa6c740f640 1 -- 172.21.15.59:0/2805030036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa6c0068ff0 con 0x7fa6c00cea50 2024-03-06T07:24:57.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.717+0000 7fa6bdffb640 1 -- 172.21.15.59:0/2805030036 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa6b407b7e0 con 0x7fa6c00cea50 2024-03-06T07:24:57.720 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 -- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa6ac06b4d0 msgr2=0x7fa6ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa6ac06b4d0 0x7fa6ac06d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8007fd0 tx=0x7fa6b8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 -- 172.21.15.59:0/2805030036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6c00cea50 msgr2=0x7fa6c00c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6c00cea50 0x7fa6c00c61a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4000c00 tx=0x7fa6b4004730 comp rx=0 tx=0).stop 2024-03-06T07:24:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 -- 172.21.15.59:0/2805030036 shutdown_connections 2024-03-06T07:24:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa6ac06b4d0 0x7fa6ac06d990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6c015def0 0x7fa6c00b6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6c00cea50 0x7fa6c00c61a0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 --2- 172.21.15.59:0/2805030036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6c006cab0 0x7fa6c015bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 -- 172.21.15.59:0/2805030036 >> 172.21.15.59:0/2805030036 conn(0x7fa6c00b3690 msgr2=0x7fa6c00d0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 -- 172.21.15.59:0/2805030036 shutdown_connections 2024-03-06T07:24:57.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.720+0000 7fa6c740f640 1 -- 172.21.15.59:0/2805030036 wait complete. 2024-03-06T07:24:57.733 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:24:57.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.833+0000 7f7f193a0640 1 Processor -- start 2024-03-06T07:24:57.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.833+0000 7f7f193a0640 1 -- start start 2024-03-06T07:24:57.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.833+0000 7f7f193a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f1415ddc0 0x7f7f140d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.833+0000 7f7f193a0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7f140cf370 0x7f7f14161a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.833+0000 7f7f193a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f14161f40 0x7f7f140d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.833+0000 7f7f193a0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f141623b0 con 0x7f7f140cf370 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.833+0000 7f7f193a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f140cfae0 con 0x7f7f14161f40 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.833+0000 7f7f193a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140b6a10 con 0x7f7f1415ddc0 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.834+0000 7f7f137fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7f140cf370 0x7f7f14161a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.834+0000 7f7f137fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7f140cf370 0x7f7f14161a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47772/0 (socket says 172.21.15.59:47772) 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.834+0000 7f7f137fe640 1 -- 172.21.15.59:0/1003658626 learned_addr learned my addr 172.21.15.59:0/1003658626 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.834+0000 7f7f18b9f640 1 --2- 172.21.15.59:0/1003658626 >> v2:172.21.15.164:3300/0 conn(0x7f7f14161f40 0x7f7f140d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.834+0000 7f7f13fff640 1 --2- 172.21.15.59:0/1003658626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f1415ddc0 0x7f7f140d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.834+0000 7f7f137fe640 1 -- 172.21.15.59:0/1003658626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f1415ddc0 msgr2=0x7f7f140d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.834+0000 7f7f137fe640 1 --2- 172.21.15.59:0/1003658626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f1415ddc0 0x7f7f140d44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.834+0000 7f7f137fe640 1 -- 172.21.15.59:0/1003658626 >> v2:172.21.15.164:3300/0 conn(0x7f7f14161f40 msgr2=0x7f7f140d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.834+0000 7f7f137fe640 1 --2- 172.21.15.59:0/1003658626 >> v2:172.21.15.164:3300/0 conn(0x7f7f14161f40 0x7f7f140d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.834+0000 7f7f137fe640 1 -- 172.21.15.59:0/1003658626 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f140b6cf0 con 0x7f7f140cf370 2024-03-06T07:24:57.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.834+0000 7f7f18b9f640 1 --2- 172.21.15.59:0/1003658626 >> v2:172.21.15.164:3300/0 conn(0x7f7f14161f40 0x7f7f140d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:57.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.835+0000 7f7f137fe640 1 --2- 172.21.15.59:0/1003658626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f140cf370 0x7f7f14161a00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7f140d14d0 tx=0x7f7f0802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2852d7a80c83d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.835+0000 7f7f12ffd640 1 -- 172.21.15.59:0/1003658626 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0802e070 con 0x7f7f140cf370 2024-03-06T07:24:57.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.835+0000 7f7f12ffd640 1 -- 172.21.15.59:0/1003658626 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f08037070 con 0x7f7f140cf370 2024-03-06T07:24:57.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.835+0000 7f7f12ffd640 1 -- 172.21.15.59:0/1003658626 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08031a00 con 0x7f7f140cf370 2024-03-06T07:24:57.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.835+0000 7f7f193a0640 1 -- 172.21.15.59:0/1003658626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f140cf370 msgr2=0x7f7f14161a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.835+0000 7f7f193a0640 1 --2- 172.21.15.59:0/1003658626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f140cf370 0x7f7f14161a00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7f140d14d0 tx=0x7f7f0802b880 comp rx=0 tx=0).stop 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.835+0000 7f7f193a0640 1 -- 172.21.15.59:0/1003658626 shutdown_connections 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.835+0000 7f7f193a0640 1 --2- 172.21.15.59:0/1003658626 >> v2:172.21.15.164:3300/0 conn(0x7f7f14161f40 0x7f7f140d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.835+0000 7f7f193a0640 1 --2- 172.21.15.59:0/1003658626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f140cf370 0x7f7f14161a00 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.835+0000 7f7f193a0640 1 --2- 172.21.15.59:0/1003658626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f1415ddc0 0x7f7f140d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.835+0000 7f7f193a0640 1 -- 172.21.15.59:0/1003658626 >> 172.21.15.59:0/1003658626 conn(0x7f7f1409ce20 msgr2=0x7f7f140d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.836+0000 7f7f193a0640 1 -- 172.21.15.59:0/1003658626 shutdown_connections 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.836+0000 7f7f193a0640 1 -- 172.21.15.59:0/1003658626 wait complete. 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.837+0000 7f7f193a0640 1 Processor -- start 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.837+0000 7f7f193a0640 1 -- start start 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.837+0000 7f7f193a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f140cf370 0x7f7f1415e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f193a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f1415ddc0 0x7f7f1415c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f193a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f14161f40 0x7f7f1415c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f193a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1415e1a0 con 0x7f7f14161f40 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f193a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140cf7d0 con 0x7f7f1415ddc0 2024-03-06T07:24:57.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f193a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1415e320 con 0x7f7f140cf370 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f18b9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f14161f40 0x7f7f1415c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f18b9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f14161f40 0x7f7f1415c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47788/0 (socket says 172.21.15.59:47788) 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f18b9f640 1 -- 172.21.15.59:0/1968171206 learned_addr learned my addr 172.21.15.59:0/1968171206 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f137fe640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f1415ddc0 0x7f7f1415c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f13fff640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f140cf370 0x7f7f1415e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f18b9f640 1 -- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f140cf370 msgr2=0x7f7f1415e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f18b9f640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f140cf370 0x7f7f1415e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f18b9f640 1 -- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f1415ddc0 msgr2=0x7f7f1415c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f18b9f640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f1415ddc0 0x7f7f1415c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f18b9f640 1 -- 172.21.15.59:0/1968171206 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f14147a90 con 0x7f7f14161f40 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f137fe640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f1415ddc0 0x7f7f1415c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f18b9f640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f14161f40 0x7f7f1415c610 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c007e70 tx=0x7f7f0c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f10ff9640 1 -- 172.21.15.59:0/1968171206 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c01a070 con 0x7f7f14161f40 2024-03-06T07:24:57.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f10ff9640 1 -- 172.21.15.59:0/1968171206 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0c003bb0 con 0x7f7f14161f40 2024-03-06T07:24:57.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f193a0640 1 -- 172.21.15.59:0/1968171206 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f14147d00 con 0x7f7f14161f40 2024-03-06T07:24:57.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f10ff9640 1 -- 172.21.15.59:0/1968171206 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c004030 con 0x7f7f14161f40 2024-03-06T07:24:57.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.838+0000 7f7f193a0640 1 -- 172.21.15.59:0/1968171206 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f141482c0 con 0x7f7f14161f40 2024-03-06T07:24:57.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.839+0000 7f7f193a0640 1 -- 172.21.15.59:0/1968171206 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f141f7db0 con 0x7f7f14161f40 2024-03-06T07:24:57.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.840+0000 7f7f10ff9640 1 -- 172.21.15.59:0/1968171206 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f0c01d070 con 0x7f7f14161f40 2024-03-06T07:24:57.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.840+0000 7f7f10ff9640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0006b4d0 0x7f7f0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.840+0000 7f7f10ff9640 1 -- 172.21.15.59:0/1968171206 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7f0c08a130 con 0x7f7f14161f40 2024-03-06T07:24:57.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.841+0000 7f7f13fff640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0006b4d0 0x7f7f0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.841+0000 7f7f13fff640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0006b4d0 0x7f7f0006d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7f04006fd0 tx=0x7f7f04008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:57.845+0000 7f7f10ff9640 1 -- 172.21.15.59:0/1968171206 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f0c010070 con 0x7f7f14161f40 2024-03-06T07:24:58.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:58.094+0000 7f7f193a0640 1 -- 172.21.15.59:0/1968171206 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f7f14068f80 con 0x7f7f14161f40 2024-03-06T07:24:59.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.097+0000 7f7f10ff9640 1 -- 172.21.15.59:0/1968171206 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v187) v1 ==== 143+0+0 (secure 0 0 0) 0x7f7f0c0592b0 con 0x7f7f14161f40 2024-03-06T07:24:59.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 -- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0006b4d0 msgr2=0x7f7f0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0006b4d0 0x7f7f0006d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7f04006fd0 tx=0x7f7f04008040 comp rx=0 tx=0).stop 2024-03-06T07:24:59.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 -- 172.21.15.59:0/1968171206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f14161f40 msgr2=0x7f7f1415c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f14161f40 0x7f7f1415c610 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c007e70 tx=0x7f7f0c00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 -- 172.21.15.59:0/1968171206 shutdown_connections 2024-03-06T07:24:59.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0006b4d0 0x7f7f0006d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f14161f40 0x7f7f1415c610 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f1415ddc0 0x7f7f1415c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 --2- 172.21.15.59:0/1968171206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f140cf370 0x7f7f1415e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 -- 172.21.15.59:0/1968171206 >> 172.21.15.59:0/1968171206 conn(0x7f7f1409ce20 msgr2=0x7f7f140b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 -- 172.21.15.59:0/1968171206 shutdown_connections 2024-03-06T07:24:59.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.100+0000 7f7f193a0640 1 -- 172.21.15.59:0/1968171206 wait complete. 2024-03-06T07:24:59.116 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:24:59.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.225+0000 7f6143caf640 1 Processor -- start 2024-03-06T07:24:59.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.225+0000 7f6143caf640 1 -- start start 2024-03-06T07:24:59.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6143caf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f613c1614d0 0x7f613c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6143caf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f613c0d1910 0x7f613c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6143caf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613c0d2740 con 0x7f613c0d22b0 2024-03-06T07:24:59.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6143caf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f613c1618b0 con 0x7f613c1614d0 2024-03-06T07:24:59.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6143caf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613c0cf400 con 0x7f613c0d1910 2024-03-06T07:24:59.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f614224c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f613c0d22b0 0x7f613c0d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59388/0 (socket says 172.21.15.59:59388) 2024-03-06T07:24:59.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f614224c640 1 -- 172.21.15.59:0/3052426271 learned_addr learned my addr 172.21.15.59:0/3052426271 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:24:59.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f614124a640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f613c0d1910 0x7f613c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6141a4b640 1 --2- 172.21.15.59:0/3052426271 >> v2:172.21.15.164:3300/0 conn(0x7f613c1614d0 0x7f613c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864601329 0 0) 0x7f6134001490 con 0x7f613c0d22b0 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6128003520 con 0x7f613c0d22b0 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2907547809 0 0) 0x7f6128003520 con 0x7f613c0d22b0 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f613c0d1910 msgr2=0x7f613c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6140a49640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f613c0d1910 0x7f613c0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 >> v2:172.21.15.164:3300/0 conn(0x7f613c1614d0 msgr2=0x7f613c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6140a49640 1 --2- 172.21.15.59:0/3052426271 >> v2:172.21.15.164:3300/0 conn(0x7f613c1614d0 0x7f613c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c0cf6a0 con 0x7f613c0d22b0 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f6134001bc0 con 0x7f613c0d22b0 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 >> v1:172.21.15.59:6789/0 conn(0x7f613c0d22b0 legacy=0x7f613c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.226+0000 7f6140a49640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61280039b0 0x7f6128003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f6140a49640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6128004800 0x7f6128004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f6140a49640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f613c0d1910 0x7f61280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c1618b0 con 0x7f61280039b0 2024-03-06T07:24:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c0cf400 con 0x7f6128004800 2024-03-06T07:24:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612800b9c0 con 0x7f613c0d1910 2024-03-06T07:24:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f6141a4b640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61280039b0 0x7f6128003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f614224c640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f613c0d1910 0x7f61280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f614124a640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6128004800 0x7f6128004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f6141a4b640 1 -- 172.21.15.59:0/3052426271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f613c0d1910 msgr2=0x7f61280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f6141a4b640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f613c0d1910 0x7f61280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f6141a4b640 1 -- 172.21.15.59:0/3052426271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6128004800 msgr2=0x7f6128004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f6141a4b640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6128004800 0x7f6128004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.227+0000 7f6141a4b640 1 -- 172.21.15.59:0/3052426271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6128003520 con 0x7f61280039b0 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.228+0000 7f6141a4b640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61280039b0 0x7f6128003dd0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6138009ac0 tx=0x7f6128005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5534113145daaace server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.228+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138039070 con 0x7f61280039b0 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.228+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613802c710 con 0x7f61280039b0 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.228+0000 7f6140a49640 1 -- 172.21.15.59:0/3052426271 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138009690 con 0x7f61280039b0 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.228+0000 7f6143caf640 1 -- 172.21.15.59:0/3052426271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61280039b0 msgr2=0x7f6128003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.228+0000 7f6143caf640 1 --2- 172.21.15.59:0/3052426271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61280039b0 0x7f6128003dd0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6138009ac0 tx=0x7f6128005830 comp rx=0 tx=0).stop 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.228+0000 7f614224c640 1 -- 172.21.15.59:0/3052426271 reap_dead start 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.228+0000 7f6143caf640 1 -- 172.21.15.59:0/3052426271 shutdown_connections 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.228+0000 7f6143caf640 1 -- 172.21.15.59:0/3052426271 >> 172.21.15.59:0/3052426271 conn(0x7f613c0d3d40 msgr2=0x7f613c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.228+0000 7f6143caf640 1 -- 172.21.15.59:0/3052426271 shutdown_connections 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.229+0000 7f6143caf640 1 -- 172.21.15.59:0/3052426271 wait complete. 2024-03-06T07:24:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.230+0000 7f6143caf640 1 Processor -- start 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.230+0000 7f6143caf640 1 -- start start 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.230+0000 7f6143caf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61280039b0 0x7f613c1581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.230+0000 7f6143caf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f613c0ca510 0x7f613c0b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.230+0000 7f6143caf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f613c0ca990 0x7f613c0b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.230+0000 7f6143caf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c0b0910 con 0x7f613c0ca990 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.230+0000 7f6143caf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c0cfc60 con 0x7f613c0ca510 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.230+0000 7f6143caf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613c0b1e40 con 0x7f61280039b0 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f614224c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f613c0ca990 0x7f613c0b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f614224c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f613c0ca990 0x7f613c0b82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47812/0 (socket says 172.21.15.59:47812) 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f614224c640 1 -- 172.21.15.59:0/3669454375 learned_addr learned my addr 172.21.15.59:0/3669454375 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f614124a640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f613c0ca510 0x7f613c0b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f6141a4b640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61280039b0 0x7f613c1581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f614224c640 1 -- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61280039b0 msgr2=0x7f613c1581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f614224c640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61280039b0 0x7f613c1581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f614224c640 1 -- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f613c0ca510 msgr2=0x7f613c0b7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f614224c640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f613c0ca510 0x7f613c0b7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f614224c640 1 -- 172.21.15.59:0/3669454375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c0b6d10 con 0x7f613c0ca990 2024-03-06T07:24:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f614124a640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f613c0ca510 0x7f613c0b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f614224c640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f613c0ca990 0x7f613c0b82f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f61340013d0 tx=0x7f6134007060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f61327fc640 1 -- 172.21.15.59:0/3669454375 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134019070 con 0x7f613c0ca990 2024-03-06T07:24:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f61327fc640 1 -- 172.21.15.59:0/3669454375 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6134007930 con 0x7f613c0ca990 2024-03-06T07:24:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f6143caf640 1 -- 172.21.15.59:0/3669454375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c0b6ff0 con 0x7f613c0ca990 2024-03-06T07:24:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f61327fc640 1 -- 172.21.15.59:0/3669454375 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134013610 con 0x7f613c0ca990 2024-03-06T07:24:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.231+0000 7f6143caf640 1 -- 172.21.15.59:0/3669454375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613c0b75b0 con 0x7f613c0ca990 2024-03-06T07:24:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.232+0000 7f6143caf640 1 -- 172.21.15.59:0/3669454375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613c0ce100 con 0x7f613c0ca990 2024-03-06T07:24:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.233+0000 7f61327fc640 1 -- 172.21.15.59:0/3669454375 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f613401c070 con 0x7f613c0ca990 2024-03-06T07:24:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.233+0000 7f61327fc640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f612006b4d0 0x7f612006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.233+0000 7f61327fc640 1 -- 172.21.15.59:0/3669454375 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6134089c10 con 0x7f613c0ca990 2024-03-06T07:24:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.233+0000 7f6141a4b640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f612006b4d0 0x7f612006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.234+0000 7f6141a4b640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f612006b4d0 0x7f612006d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6138036000 tx=0x7f6138030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.237+0000 7f61327fc640 1 -- 172.21.15.59:0/3669454375 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f613c0d2740 con 0x7f613c0ca990 2024-03-06T07:24:59.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.419+0000 7f6143caf640 1 -- 172.21.15.59:0/3669454375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f613c068ff0 con 0x7f613c0ca990 2024-03-06T07:24:59.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.425+0000 7f61327fc640 1 -- 172.21.15.59:0/3669454375 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f613408d070 con 0x7f613c0ca990 2024-03-06T07:24:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 -- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f612006b4d0 msgr2=0x7f612006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f612006b4d0 0x7f612006d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6138036000 tx=0x7f6138030000 comp rx=0 tx=0).stop 2024-03-06T07:24:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 -- 172.21.15.59:0/3669454375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f613c0ca990 msgr2=0x7f613c0b82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f613c0ca990 0x7f613c0b82f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f61340013d0 tx=0x7f6134007060 comp rx=0 tx=0).stop 2024-03-06T07:24:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 -- 172.21.15.59:0/3669454375 shutdown_connections 2024-03-06T07:24:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f612006b4d0 0x7f612006d990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f613c0ca990 0x7f613c0b82f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f613c0ca510 0x7f613c0b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 --2- 172.21.15.59:0/3669454375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61280039b0 0x7f613c1581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 -- 172.21.15.59:0/3669454375 >> 172.21.15.59:0/3669454375 conn(0x7f613c0d3d40 msgr2=0x7f613c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 -- 172.21.15.59:0/3669454375 shutdown_connections 2024-03-06T07:24:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.427+0000 7f6143caf640 1 -- 172.21.15.59:0/3669454375 wait complete. 2024-03-06T07:24:59.439 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:59.440 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:59.489 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:59.533 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:59.533 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:24:59.533 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:59.548 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:59.548 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:24:59.548 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:59.591 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:59.591 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:24:59.591 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:59.605 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:59.605 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:24:59.606 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:59.647 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:59.647 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:24:59.647 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:59.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.747+0000 7f11251ac640 1 Processor -- start 2024-03-06T07:24:59.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.747+0000 7f11251ac640 1 -- start start 2024-03-06T07:24:59.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f11251ac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f11200cf850 0x7f11200cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f11251ac640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f112006cf20 0x7f112006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f11251ac640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112015e380 con 0x7f11200d00c0 2024-03-06T07:24:59.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f11251ac640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f112006ca90 con 0x7f11200cf850 2024-03-06T07:24:59.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f11251ac640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11200cdc10 con 0x7f112006cf20 2024-03-06T07:24:59.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f111e59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f11200cf850 0x7f11200cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f111e59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f11200cf850 0x7f11200cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47828/0 (socket says 172.21.15.59:47828) 2024-03-06T07:24:59.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f111e59c640 1 -- 172.21.15.59:0/3483280196 learned_addr learned my addr 172.21.15.59:0/3483280196 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:59.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f111f59e640 1 --2- 172.21.15.59:0/3483280196 >> v2:172.21.15.164:3300/0 conn(0x7f112006cf20 0x7f112006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f111e59c640 1 -- 172.21.15.59:0/3483280196 >> v2:172.21.15.164:3300/0 conn(0x7f112006cf20 msgr2=0x7f112006d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:59.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f111e59c640 1 --2- 172.21.15.59:0/3483280196 >> v2:172.21.15.164:3300/0 conn(0x7f112006cf20 0x7f112006d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f111e59c640 1 -- 172.21.15.59:0/3483280196 >> v1:172.21.15.164:6789/0 conn(0x7f11200d00c0 legacy=0x7f11200d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.748+0000 7f111e59c640 1 -- 172.21.15.59:0/3483280196 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112006c160 con 0x7f11200cf850 2024-03-06T07:24:59.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.749+0000 7f111e59c640 1 --2- 172.21.15.59:0/3483280196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11200cf850 0x7f11200cd3c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1120071f50 tx=0x7f110802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1579bfe402637b96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.749+0000 7f111dd9b640 1 -- 172.21.15.59:0/3483280196 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108034070 con 0x7f11200cf850 2024-03-06T07:24:59.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.749+0000 7f111dd9b640 1 -- 172.21.15.59:0/3483280196 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1108030860 con 0x7f11200cf850 2024-03-06T07:24:59.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.749+0000 7f111dd9b640 1 -- 172.21.15.59:0/3483280196 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108030c30 con 0x7f11200cf850 2024-03-06T07:24:59.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.750+0000 7f11251ac640 1 -- 172.21.15.59:0/3483280196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11200cf850 msgr2=0x7f11200cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.750+0000 7f11251ac640 1 --2- 172.21.15.59:0/3483280196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11200cf850 0x7f11200cd3c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1120071f50 tx=0x7f110802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.750+0000 7f11251ac640 1 -- 172.21.15.59:0/3483280196 shutdown_connections 2024-03-06T07:24:59.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.750+0000 7f11251ac640 1 --2- 172.21.15.59:0/3483280196 >> v2:172.21.15.164:3300/0 conn(0x7f112006cf20 0x7f112006d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.750+0000 7f11251ac640 1 --2- 172.21.15.59:0/3483280196 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f11200cf850 0x7f11200cd3c0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.750+0000 7f11251ac640 1 -- 172.21.15.59:0/3483280196 >> 172.21.15.59:0/3483280196 conn(0x7f11200b35a0 msgr2=0x7f11200b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.750+0000 7f11251ac640 1 -- 172.21.15.59:0/3483280196 shutdown_connections 2024-03-06T07:24:59.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.750+0000 7f11251ac640 1 -- 172.21.15.59:0/3483280196 wait complete. 2024-03-06T07:24:59.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.751+0000 7f11251ac640 1 Processor -- start 2024-03-06T07:24:59.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.751+0000 7f11251ac640 1 -- start start 2024-03-06T07:24:59.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f11251ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f112006cf20 0x7f1120159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f11251ac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11200cf850 0x7f112015a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f11251ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11200d00c0 0x7f11200ae810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f11251ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11200cd900 con 0x7f112006cf20 2024-03-06T07:24:59.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f11251ac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112006c720 con 0x7f11200cf850 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f11251ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11200b1030 con 0x7f11200d00c0 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f111ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f112006cf20 0x7f1120159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f111ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f112006cf20 0x7f1120159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47840/0 (socket says 172.21.15.59:47840) 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f111ed9d640 1 -- 172.21.15.59:0/1560994331 learned_addr learned my addr 172.21.15.59:0/1560994331 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f111f59e640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11200d00c0 0x7f11200ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f111ed9d640 1 -- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11200d00c0 msgr2=0x7f11200ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f111e59c640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11200cf850 0x7f112015a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f111ed9d640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11200d00c0 0x7f11200ae810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f111ed9d640 1 -- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11200cf850 msgr2=0x7f112015a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f111ed9d640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11200cf850 0x7f112015a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.752+0000 7f111ed9d640 1 -- 172.21.15.59:0/1560994331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11201f5050 con 0x7f112006cf20 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.753+0000 7f111ed9d640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f112006cf20 0x7f1120159b40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f111800c3c0 tx=0x7f111800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.753+0000 7f11177fe640 1 -- 172.21.15.59:0/1560994331 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11180052d0 con 0x7f112006cf20 2024-03-06T07:24:59.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.753+0000 7f11177fe640 1 -- 172.21.15.59:0/1560994331 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1118005490 con 0x7f112006cf20 2024-03-06T07:24:59.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.753+0000 7f11251ac640 1 -- 172.21.15.59:0/1560994331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11201f5330 con 0x7f112006cf20 2024-03-06T07:24:59.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.753+0000 7f11177fe640 1 -- 172.21.15.59:0/1560994331 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118007260 con 0x7f112006cf20 2024-03-06T07:24:59.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.753+0000 7f11251ac640 1 -- 172.21.15.59:0/1560994331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11201f58f0 con 0x7f112006cf20 2024-03-06T07:24:59.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.754+0000 7f11251ac640 1 -- 172.21.15.59:0/1560994331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112004f330 con 0x7f112006cf20 2024-03-06T07:24:59.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.754+0000 7f11177fe640 1 -- 172.21.15.59:0/1560994331 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1118007400 con 0x7f112006cf20 2024-03-06T07:24:59.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.755+0000 7f11177fe640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f110006b500 0x7f110006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:59.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.755+0000 7f11177fe640 1 -- 172.21.15.59:0/1560994331 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f11180892c0 con 0x7f112006cf20 2024-03-06T07:24:59.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.755+0000 7f111e59c640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f110006b500 0x7f110006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:59.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.755+0000 7f111e59c640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f110006b500 0x7f110006d9c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1108036000 tx=0x7f110803b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:59.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.759+0000 7f11177fe640 1 -- 172.21.15.59:0/1560994331 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11180584d0 con 0x7f112006cf20 2024-03-06T07:24:59.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.980+0000 7f11251ac640 1 -- 172.21.15.59:0/1560994331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11200ced60 con 0x7f112006cf20 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.981+0000 7f11177fe640 1 -- 172.21.15.59:0/1560994331 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f1118057dd0 con 0x7f112006cf20 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":30,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5429,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2205562350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2205562350},{"type":"v1","addr":"172.21.15.164:6837","nonce":2205562350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5872,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2606750481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2606750481},{"type":"v1","addr":"172.21.15.164:6836","nonce":2606750481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6048,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3685052110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3685052110},{"type":"v1","addr":"172.21.15.59:6833","nonce":3685052110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1055310172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1055310172},{"type":"v1","addr":"172.21.15.59:6835","nonce":1055310172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[]} 2024-03-06T07:24:59.983 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 30 2024-03-06T07:24:59.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 -- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f110006b500 msgr2=0x7f110006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f110006b500 0x7f110006d9c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1108036000 tx=0x7f110803b040 comp rx=0 tx=0).stop 2024-03-06T07:24:59.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 -- 172.21.15.59:0/1560994331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f112006cf20 msgr2=0x7f1120159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:59.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f112006cf20 0x7f1120159b40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f111800c3c0 tx=0x7f111800c880 comp rx=0 tx=0).stop 2024-03-06T07:24:59.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 -- 172.21.15.59:0/1560994331 shutdown_connections 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f110006b500 0x7f110006d9c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11200d00c0 0x7f11200ae810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11200cf850 0x7f112015a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 --2- 172.21.15.59:0/1560994331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f112006cf20 0x7f1120159b40 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 -- 172.21.15.59:0/1560994331 >> 172.21.15.59:0/1560994331 conn(0x7f11200b35a0 msgr2=0x7f11200702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 -- 172.21.15.59:0/1560994331 shutdown_connections 2024-03-06T07:24:59.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:24:59.983+0000 7f11251ac640 1 -- 172.21.15.59:0/1560994331 wait complete. 2024-03-06T07:24:59.996 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:24:59.996 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:24:59.996 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:24:59.996 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:59.997 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:24:59.997 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:24:59.997 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:24:59.997 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:59.997 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:24:59.997 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:24:59.997 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:24:59.997 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:59.998 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:24:59.998 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:24:59.998 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:24:59.998 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:00.006 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:25:00.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:25:00.007 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:25:00.007 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:25:00.010 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:25:00.010 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:25:00.012 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:25:00.012 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:25:00.049 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:25:00.049 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:00.050 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:25:00.050 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:00.053 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:25:00.054 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:00.055 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:25:00.055 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:00.055 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:25:00.126 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:25:00.127 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:25:00.139 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:25:00.140 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:25:00.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.160+0000 7f5a44a01640 1 Processor -- start 2024-03-06T07:25:00.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.160+0000 7f5a44a01640 1 -- start start 2024-03-06T07:25:00.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.161+0000 7f5a44a01640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4015bb20 0x7f5a4006c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.161+0000 7f5a44a01640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a400ce430 0x7f5a400ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.161+0000 7f5a44a01640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4015f260 con 0x7f5a400cfa20 2024-03-06T07:25:00.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.161+0000 7f5a44a01640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a4015bf00 con 0x7f5a400ce430 2024-03-06T07:25:00.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.161+0000 7f5a44a01640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400b3480 con 0x7f5a4015bb20 2024-03-06T07:25:00.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.161+0000 7f5a3f59e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a400ce430 0x7f5a400ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.161+0000 7f5a3f59e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a400ce430 0x7f5a400ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47880/0 (socket says 172.21.15.59:47880) 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.161+0000 7f5a3f59e640 1 -- 172.21.15.59:0/411920131 learned_addr learned my addr 172.21.15.59:0/411920131 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.161+0000 7f5a3ed9d640 1 --2- 172.21.15.59:0/411920131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4015bb20 0x7f5a4006c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.162+0000 7f5a3f59e640 1 -- 172.21.15.59:0/411920131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4015bb20 msgr2=0x7f5a4006c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.162+0000 7f5a3f59e640 1 --2- 172.21.15.59:0/411920131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4015bb20 0x7f5a4006c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.162+0000 7f5a3f59e640 1 -- 172.21.15.59:0/411920131 >> v1:172.21.15.164:6789/0 conn(0x7f5a400cfa20 legacy=0x7f5a400d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.162+0000 7f5a3f59e640 1 -- 172.21.15.59:0/411920131 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400b2e30 con 0x7f5a400ce430 2024-03-06T07:25:00.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.162+0000 7f5a3f59e640 1 --2- 172.21.15.59:0/411920131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a400ce430 0x7f5a400ce8c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009e50 tx=0x7f5a3802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61ef585fc9a5954 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.162+0000 7f5a3e59c640 1 -- 172.21.15.59:0/411920131 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3802cb00 con 0x7f5a400ce430 2024-03-06T07:25:00.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.162+0000 7f5a3e59c640 1 -- 172.21.15.59:0/411920131 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3802cca0 con 0x7f5a400ce430 2024-03-06T07:25:00.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.163+0000 7f5a3e59c640 1 -- 172.21.15.59:0/411920131 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38034610 con 0x7f5a400ce430 2024-03-06T07:25:00.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.163+0000 7f5a44a01640 1 -- 172.21.15.59:0/411920131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a400ce430 msgr2=0x7f5a400ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.163+0000 7f5a44a01640 1 --2- 172.21.15.59:0/411920131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a400ce430 0x7f5a400ce8c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009e50 tx=0x7f5a3802bf60 comp rx=0 tx=0).stop 2024-03-06T07:25:00.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.163+0000 7f5a44a01640 1 -- 172.21.15.59:0/411920131 shutdown_connections 2024-03-06T07:25:00.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.163+0000 7f5a44a01640 1 --2- 172.21.15.59:0/411920131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a400ce430 0x7f5a400ce8c0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.163+0000 7f5a44a01640 1 --2- 172.21.15.59:0/411920131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4015bb20 0x7f5a4006c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.163+0000 7f5a44a01640 1 -- 172.21.15.59:0/411920131 >> 172.21.15.59:0/411920131 conn(0x7f5a400b6e30 msgr2=0x7f5a400b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.163+0000 7f5a44a01640 1 -- 172.21.15.59:0/411920131 shutdown_connections 2024-03-06T07:25:00.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.163+0000 7f5a44a01640 1 -- 172.21.15.59:0/411920131 wait complete. 2024-03-06T07:25:00.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.164+0000 7f5a44a01640 1 Processor -- start 2024-03-06T07:25:00.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.164+0000 7f5a44a01640 1 -- start start 2024-03-06T07:25:00.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a44a01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a400ce430 0x7f5a4014d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a44a01640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a400cfa20 0x7f5a400b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a44a01640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4015bb20 0x7f5a401557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a44a01640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a40161d40 con 0x7f5a400ce430 2024-03-06T07:25:00.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a44a01640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4015eef0 con 0x7f5a400cfa20 2024-03-06T07:25:00.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a44a01640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a40161ec0 con 0x7f5a4015bb20 2024-03-06T07:25:00.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a3ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a400ce430 0x7f5a4014d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a3ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a400ce430 0x7f5a4014d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47886/0 (socket says 172.21.15.59:47886) 2024-03-06T07:25:00.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a3ed9d640 1 -- 172.21.15.59:0/1175253327 learned_addr learned my addr 172.21.15.59:0/1175253327 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:00.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a37fff640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a400cfa20 0x7f5a400b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a3f59e640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4015bb20 0x7f5a401557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a3ed9d640 1 -- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4015bb20 msgr2=0x7f5a401557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a3ed9d640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4015bb20 0x7f5a401557f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a3ed9d640 1 -- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a400cfa20 msgr2=0x7f5a400b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a3ed9d640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a400cfa20 0x7f5a400b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a3ed9d640 1 -- 172.21.15.59:0/1175253327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4015f640 con 0x7f5a400ce430 2024-03-06T07:25:00.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.165+0000 7f5a3f59e640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4015bb20 0x7f5a401557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:00.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.166+0000 7f5a3ed9d640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a400ce430 0x7f5a4014d0a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000e870 tx=0x7f5a3000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.166+0000 7f5a377fe640 1 -- 172.21.15.59:0/1175253327 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a30017070 con 0x7f5a400ce430 2024-03-06T07:25:00.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.166+0000 7f5a377fe640 1 -- 172.21.15.59:0/1175253327 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3000c940 con 0x7f5a400ce430 2024-03-06T07:25:00.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.166+0000 7f5a44a01640 1 -- 172.21.15.59:0/1175253327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4015f920 con 0x7f5a400ce430 2024-03-06T07:25:00.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.166+0000 7f5a377fe640 1 -- 172.21.15.59:0/1175253327 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3000cd30 con 0x7f5a400ce430 2024-03-06T07:25:00.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.166+0000 7f5a44a01640 1 -- 172.21.15.59:0/1175253327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a400b5e40 con 0x7f5a400ce430 2024-03-06T07:25:00.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.167+0000 7f5a44a01640 1 -- 172.21.15.59:0/1175253327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a40068f80 con 0x7f5a400ce430 2024-03-06T07:25:00.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.167+0000 7f5a377fe640 1 -- 172.21.15.59:0/1175253327 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a3001a070 con 0x7f5a400ce430 2024-03-06T07:25:00.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.168+0000 7f5a377fe640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a2006b4d0 0x7f5a2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.168+0000 7f5a377fe640 1 -- 172.21.15.59:0/1175253327 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5a30089db0 con 0x7f5a400ce430 2024-03-06T07:25:00.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.168+0000 7f5a37fff640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a2006b4d0 0x7f5a2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.168+0000 7f5a37fff640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a2006b4d0 0x7f5a2006d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5a40157270 tx=0x7f5a28009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.172+0000 7f5a377fe640 1 -- 172.21.15.59:0/1175253327 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a3008e070 con 0x7f5a400ce430 2024-03-06T07:25:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:00.364+0000 7f5a44a01640 1 -- 172.21.15.59:0/1175253327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5a400ce210 con 0x7f5a400ce430 2024-03-06T07:25:01.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.126+0000 7f5a377fe640 1 -- 172.21.15.59:0/1175253327 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5a30058840 con 0x7f5a400ce430 2024-03-06T07:25:01.128 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-03-06T07:25:01.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 -- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a2006b4d0 msgr2=0x7f5a2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a2006b4d0 0x7f5a2006d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5a40157270 tx=0x7f5a28009040 comp rx=0 tx=0).stop 2024-03-06T07:25:01.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 -- 172.21.15.59:0/1175253327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a400ce430 msgr2=0x7f5a4014d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a400ce430 0x7f5a4014d0a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000e870 tx=0x7f5a3000ed30 comp rx=0 tx=0).stop 2024-03-06T07:25:01.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 -- 172.21.15.59:0/1175253327 shutdown_connections 2024-03-06T07:25:01.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a2006b4d0 0x7f5a2006d990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4015bb20 0x7f5a401557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a400cfa20 0x7f5a400b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 --2- 172.21.15.59:0/1175253327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a400ce430 0x7f5a4014d0a0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 -- 172.21.15.59:0/1175253327 >> 172.21.15.59:0/1175253327 conn(0x7f5a400b6e30 msgr2=0x7f5a4006ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 -- 172.21.15.59:0/1175253327 shutdown_connections 2024-03-06T07:25:01.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.129+0000 7f5a44a01640 1 -- 172.21.15.59:0/1175253327 wait complete. 2024-03-06T07:25:01.145 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:25:01.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.250+0000 7f466dd32640 1 Processor -- start 2024-03-06T07:25:01.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.250+0000 7f466dd32640 1 -- start start 2024-03-06T07:25:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.251+0000 7f466dd32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680cf850 0x7f46680cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.251+0000 7f466dd32640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f466806cf20 0x7f466806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.251+0000 7f466dd32640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466815e380 con 0x7f46680d00c0 2024-03-06T07:25:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.251+0000 7f466dd32640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466806ca90 con 0x7f466806cf20 2024-03-06T07:25:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.251+0000 7f466dd32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46680cdc10 con 0x7f46680cf850 2024-03-06T07:25:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.251+0000 7f46677fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f46680d00c0 0x7f46680d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59404/0 (socket says 172.21.15.59:59404) 2024-03-06T07:25:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.251+0000 7f46677fe640 1 -- 172.21.15.59:0/2745076884 learned_addr learned my addr 172.21.15.59:0/2745076884 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.251+0000 7f4667fff640 1 --2- 172.21.15.59:0/2745076884 >> v2:172.21.15.164:3300/0 conn(0x7f466806cf20 0x7f466806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.251+0000 7f4666ffd640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680cf850 0x7f46680cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.251+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497716562 0 0) 0x7f466815e380 con 0x7f46680d00c0 2024-03-06T07:25:01.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.251+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4650003520 con 0x7f46680d00c0 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2831210509 0 0) 0x7f4650003520 con 0x7f46680d00c0 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680cf850 msgr2=0x7f46680cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680cf850 0x7f46680cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 >> v2:172.21.15.164:3300/0 conn(0x7f466806cf20 msgr2=0x7f466806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 --2- 172.21.15.59:0/2745076884 >> v2:172.21.15.164:3300/0 conn(0x7f466806cf20 0x7f466806d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466806c160 con 0x7f46680d00c0 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f4658001670 con 0x7f46680d00c0 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 >> v1:172.21.15.59:6789/0 conn(0x7f46680d00c0 legacy=0x7f46680d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680cf850 0x7f46500039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4650003f30 0x7f4650004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650004d40 0x7f46500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466806ca90 con 0x7f4650004d40 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46680cdc10 con 0x7f4650003f30 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465000b9c0 con 0x7f46680cf850 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f4667fff640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650004d40 0x7f46500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f4666ffd640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4650003f30 0x7f4650004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f46677fe640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680cf850 0x7f46500039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f4667fff640 1 -- 172.21.15.59:0/2745076884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680cf850 msgr2=0x7f46500039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f4667fff640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680cf850 0x7f46500039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f4667fff640 1 -- 172.21.15.59:0/2745076884 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4650003f30 msgr2=0x7f4650004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f4667fff640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4650003f30 0x7f4650004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.252+0000 7f4667fff640 1 -- 172.21.15.59:0/2745076884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4650003520 con 0x7f4650004d40 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.253+0000 7f4667fff640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650004d40 0x7f46500051c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4660009fc0 tx=0x7f4650005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe086eef3f387671 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.253+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660039070 con 0x7f4650004d40 2024-03-06T07:25:01.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.253+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466002cc40 con 0x7f4650004d40 2024-03-06T07:25:01.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.253+0000 7f46667fc640 1 -- 172.21.15.59:0/2745076884 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660009c10 con 0x7f4650004d40 2024-03-06T07:25:01.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.254+0000 7f466dd32640 1 -- 172.21.15.59:0/2745076884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650004d40 msgr2=0x7f46500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.254+0000 7f466dd32640 1 --2- 172.21.15.59:0/2745076884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4650004d40 0x7f46500051c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4660009fc0 tx=0x7f4650005830 comp rx=0 tx=0).stop 2024-03-06T07:25:01.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.254+0000 7f4667fff640 1 -- 172.21.15.59:0/2745076884 reap_dead start 2024-03-06T07:25:01.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.254+0000 7f466dd32640 1 -- 172.21.15.59:0/2745076884 shutdown_connections 2024-03-06T07:25:01.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.254+0000 7f466dd32640 1 -- 172.21.15.59:0/2745076884 >> 172.21.15.59:0/2745076884 conn(0x7f46680b35a0 msgr2=0x7f46680b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.254+0000 7f466dd32640 1 -- 172.21.15.59:0/2745076884 shutdown_connections 2024-03-06T07:25:01.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.254+0000 7f466dd32640 1 -- 172.21.15.59:0/2745076884 wait complete. 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.255+0000 7f466dd32640 1 Processor -- start 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.255+0000 7f466dd32640 1 -- start start 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f466dd32640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46681f3940 0x7f46680c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f466dd32640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46680c7e70 0x7f46680b5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f466dd32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680b5a90 0x7f46680b5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f466dd32640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46680b1030 con 0x7f46680c7e70 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f466dd32640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466806c720 con 0x7f46681f3940 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f466dd32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466814f7b0 con 0x7f46680b5a90 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f4666ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46680c7e70 0x7f46680b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f4666ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46680c7e70 0x7f46680b5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47908/0 (socket says 172.21.15.59:47908) 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f4666ffd640 1 -- 172.21.15.59:0/1780059152 learned_addr learned my addr 172.21.15.59:0/1780059152 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:01.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f4666ffd640 1 -- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680b5a90 msgr2=0x7f46680b5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f4666ffd640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680b5a90 0x7f46680b5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f4666ffd640 1 -- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46681f3940 msgr2=0x7f46680c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f4666ffd640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46681f3940 0x7f46680c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f4666ffd640 1 -- 172.21.15.59:0/1780059152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46680c8330 con 0x7f46680c7e70 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f4666ffd640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46680c7e70 0x7f46680b5550 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f465c011910 tx=0x7f465c011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.256+0000 7f464ffff640 1 -- 172.21.15.59:0/1780059152 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c00fa70 con 0x7f46680c7e70 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.257+0000 7f464ffff640 1 -- 172.21.15.59:0/1780059152 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465c00fc10 con 0x7f46680c7e70 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.257+0000 7f466dd32640 1 -- 172.21.15.59:0/1780059152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46680b4a70 con 0x7f46680c7e70 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.257+0000 7f464ffff640 1 -- 172.21.15.59:0/1780059152 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c014440 con 0x7f46680c7e70 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.257+0000 7f466dd32640 1 -- 172.21.15.59:0/1780059152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46680b5030 con 0x7f46680c7e70 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.258+0000 7f466dd32640 1 -- 172.21.15.59:0/1780059152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466804f330 con 0x7f46680c7e70 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.258+0000 7f464ffff640 1 -- 172.21.15.59:0/1780059152 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f465c012040 con 0x7f46680c7e70 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.258+0000 7f464ffff640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f465406b500 0x7f465406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.258+0000 7f464ffff640 1 -- 172.21.15.59:0/1780059152 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f465c08a460 con 0x7f46680c7e70 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.259+0000 7f46677fe640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f465406b500 0x7f465406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.259+0000 7f46677fe640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f465406b500 0x7f465406d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4658001eb0 tx=0x7f46580047e0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.263+0000 7f464ffff640 1 -- 172.21.15.59:0/1780059152 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f465c019030 con 0x7f46680c7e70 2024-03-06T07:25:01.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.516+0000 7f466dd32640 1 -- 172.21.15.59:0/1780059152 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4668070880 con 0x7f46680c7e70 2024-03-06T07:25:01.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.516+0000 7f464ffff640 1 -- 172.21.15.59:0/1780059152 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f465c059630 con 0x7f46680c7e70 2024-03-06T07:25:01.519 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:01.519 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:25:01.519 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-03-06T07:25:01.519 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.519 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-03-06T07:25:01.519 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlODBZGBAAqV+TH2t2ZKyV9l13QEn6ew==", 2024-03-06T07:25:01.519 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.519 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:01.519 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:25:01.519 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:25:01.519 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlR1MuJRAAp6OAQ9gZemWtwI1dh3H2SA==", 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.520 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlDXW2HxAAnKyuS1GW0wLO98/F9wdOAg==", 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehl5MO4LBAAKmt2dvKk7BtcCM4o7cSwgg==", 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:25:01.521 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:25:01.522 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.522 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.522 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.522 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.522 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-03-06T07:25:01.522 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBXGehlzdinFhAA0ACDAs9dtoY6EhUIveeIvA==", 2024-03-06T07:25:01.522 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.522 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:01.522 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:01.522 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.522 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGehlH92eFRAANQceHGuQyrbI6gP1m/Gakw==", 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGehlMWpVFBAAbOG9nScbFlvKHXZMQJoQug==", 2024-03-06T07:25:01.523 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.524 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:01.524 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:01.524 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.524 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.524 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.524 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.524 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-03-06T07:25:01.524 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGehlLCyWEhAAyfoW/FmU37QD2IohknBwWw==", 2024-03-06T07:25:01.524 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.524 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:01.524 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:01.525 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.525 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.525 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.525 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.525 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-03-06T07:25:01.525 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBmGehldqv5GhAAWEloqqKgdpKPHjt7VKFodw==", 2024-03-06T07:25:01.525 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.525 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:01.525 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGehlc0ENGRAA5V9Az+vxAR3OUwWpDP7w1A==", 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.526 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBsGehl5SYEFxAAPLyrHQV9SGguCV9f5VIkYg==", 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBvGehle3v0FBAA4xUPwX5couppHgi9VwSi2g==", 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.527 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:01.528 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:01.528 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.528 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.528 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.528 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.528 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-03-06T07:25:01.528 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA==", 2024-03-06T07:25:01.528 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.528 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:01.528 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-03-06T07:25:01.528 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlqhlAOhAAQ4l3uI8MlcCH+NAIWo/UuA==", 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.529 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-03-06T07:25:01.530 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehltL/GNhAAZqSQ+UtwjI/PA95O1RGBNg==", 2024-03-06T07:25:01.530 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.530 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:01.530 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:25:01.530 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:25:01.530 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:25:01.530 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.530 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.530 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.530 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-03-06T07:25:01.530 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGehlltx5BBAAFa/+/w8Hd8lZegfIWtpJOA==", 2024-03-06T07:25:01.531 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.531 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:01.531 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:25:01.531 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:25:01.531 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:25:01.531 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.531 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.531 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.531 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-03-06T07:25:01.531 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBSGehlMExaNhAAQ2FD7afktMhrTbMMAK1epQ==", 2024-03-06T07:25:01.531 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlLSTzFBAACeLdHsCX9KJGEsq44aqTyw==", 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:25:01.532 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlAgqXEBAACCHZ9Lp2lEyF93xD4Ns9Nw==", 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:01.533 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-06T07:25:01.534 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:25:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 -- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f465406b500 msgr2=0x7f465406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f465406b500 0x7f465406d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4658001eb0 tx=0x7f46580047e0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 -- 172.21.15.59:0/1780059152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46680c7e70 msgr2=0x7f46680b5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46680c7e70 0x7f46680b5550 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f465c011910 tx=0x7f465c011dd0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 -- 172.21.15.59:0/1780059152 shutdown_connections 2024-03-06T07:25:01.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f465406b500 0x7f465406d9c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46680b5a90 0x7f46680b5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46680c7e70 0x7f46680b5550 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 --2- 172.21.15.59:0/1780059152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46681f3940 0x7f46680c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 -- 172.21.15.59:0/1780059152 >> 172.21.15.59:0/1780059152 conn(0x7f46680b35a0 msgr2=0x7f46680d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 -- 172.21.15.59:0/1780059152 shutdown_connections 2024-03-06T07:25:01.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.518+0000 7f466dd32640 1 -- 172.21.15.59:0/1780059152 wait complete. 2024-03-06T07:25:01.537 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:01.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.636+0000 7f1f0558a640 1 Processor -- start 2024-03-06T07:25:01.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.636+0000 7f1f0558a640 1 -- start start 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1f0558a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f000cf850 0x7f1f000cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1f0558a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1f0006cf20 0x7f1f0006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1f0558a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0015e380 con 0x7f1f000d00c0 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1f0558a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f0006ca90 con 0x7f1f0006cf20 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1f0558a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f000cdc10 con 0x7f1f000cf850 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1eff7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1f0006cf20 0x7f1f0006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1efeffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1f000d00c0 0x7f1f000d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59420/0 (socket says 172.21.15.59:59420) 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1efeffd640 1 -- 172.21.15.59:0/3708695310 learned_addr learned my addr 172.21.15.59:0/3708695310 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1efe7fc640 1 --2- 172.21.15.59:0/3708695310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f000cf850 0x7f1f000cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1eff7fe640 1 -- 172.21.15.59:0/3708695310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f000cf850 msgr2=0x7f1f000cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1eff7fe640 1 --2- 172.21.15.59:0/3708695310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f000cf850 0x7f1f000cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1eff7fe640 1 -- 172.21.15.59:0/3708695310 >> v1:172.21.15.59:6789/0 conn(0x7f1f000d00c0 legacy=0x7f1f000d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1eff7fe640 1 -- 172.21.15.59:0/3708695310 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0006c160 con 0x7f1f0006cf20 2024-03-06T07:25:01.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.637+0000 7f1efdffb640 1 -- 172.21.15.59:0/3708695310 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916114817 0 0) 0x7f1f0015e380 con 0x7f1f000d00c0 2024-03-06T07:25:01.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.638+0000 7f1eff7fe640 1 --2- 172.21.15.59:0/3708695310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f0006cf20 0x7f1f0006d3b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80097a0 tx=0x7f1ef802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b96beff1d8c541 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.638+0000 7f1efdffb640 1 -- 172.21.15.59:0/3708695310 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8035020 con 0x7f1f0006cf20 2024-03-06T07:25:01.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.638+0000 7f1efdffb640 1 -- 172.21.15.59:0/3708695310 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef8031830 con 0x7f1f0006cf20 2024-03-06T07:25:01.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.638+0000 7f1efdffb640 1 -- 172.21.15.59:0/3708695310 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8031c00 con 0x7f1f0006cf20 2024-03-06T07:25:01.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.638+0000 7f1f0558a640 1 -- 172.21.15.59:0/3708695310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f0006cf20 msgr2=0x7f1f0006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.638+0000 7f1f0558a640 1 --2- 172.21.15.59:0/3708695310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f0006cf20 0x7f1f0006d3b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1ef80097a0 tx=0x7f1ef802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.638+0000 7f1f0558a640 1 -- 172.21.15.59:0/3708695310 shutdown_connections 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.638+0000 7f1f0558a640 1 --2- 172.21.15.59:0/3708695310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f0006cf20 0x7f1f0006d3b0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.638+0000 7f1f0558a640 1 --2- 172.21.15.59:0/3708695310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f000cf850 0x7f1f000cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.638+0000 7f1f0558a640 1 -- 172.21.15.59:0/3708695310 >> 172.21.15.59:0/3708695310 conn(0x7f1f000b35a0 msgr2=0x7f1f000b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.639+0000 7f1f0558a640 1 -- 172.21.15.59:0/3708695310 shutdown_connections 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.639+0000 7f1f0558a640 1 -- 172.21.15.59:0/3708695310 wait complete. 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.640+0000 7f1f0558a640 1 Processor -- start 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.640+0000 7f1f0558a640 1 -- start start 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.640+0000 7f1f0558a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f0006cf20 0x7f1f000b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.640+0000 7f1f0558a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f000cf850 0x7f1f000c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.640+0000 7f1f0558a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f000d00c0 0x7f1f000cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.640+0000 7f1f0558a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f000cd900 con 0x7f1f000cf850 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.640+0000 7f1f0558a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0006c720 con 0x7f1f000d00c0 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.640+0000 7f1f0558a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f000b1030 con 0x7f1f0006cf20 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1efe7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f000cf850 0x7f1f000c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1efe7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f000cf850 0x7f1f000c9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47926/0 (socket says 172.21.15.59:47926) 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1efe7fc640 1 -- 172.21.15.59:0/2366799079 learned_addr learned my addr 172.21.15.59:0/2366799079 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1efeffd640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f0006cf20 0x7f1f000b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1eff7fe640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f000d00c0 0x7f1f000cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1efe7fc640 1 -- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f0006cf20 msgr2=0x7f1f000b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1efe7fc640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f0006cf20 0x7f1f000b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1efe7fc640 1 -- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f000d00c0 msgr2=0x7f1f000cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1efe7fc640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f000d00c0 0x7f1f000cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1efe7fc640 1 -- 172.21.15.59:0/2366799079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f000ca0d0 con 0x7f1f000cf850 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1eff7fe640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f000d00c0 0x7f1f000cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1efeffd640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f0006cf20 0x7f1f000b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1efe7fc640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f000cf850 0x7f1f000c9b90 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400d800 tx=0x7f1ef400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1ee77fe640 1 -- 172.21.15.59:0/2366799079 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef400b6c0 con 0x7f1f000cf850 2024-03-06T07:25:01.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1ee77fe640 1 -- 172.21.15.59:0/2366799079 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef4019070 con 0x7f1f000cf850 2024-03-06T07:25:01.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1f0558a640 1 -- 172.21.15.59:0/2366799079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f000ca3a0 con 0x7f1f000cf850 2024-03-06T07:25:01.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1ee77fe640 1 -- 172.21.15.59:0/2366799079 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef400ba80 con 0x7f1f000cf850 2024-03-06T07:25:01.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.641+0000 7f1f0558a640 1 -- 172.21.15.59:0/2366799079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f000ca960 con 0x7f1f000cf850 2024-03-06T07:25:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.642+0000 7f1f0558a640 1 -- 172.21.15.59:0/2366799079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f000cab30 con 0x7f1f000cf850 2024-03-06T07:25:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.643+0000 7f1ee77fe640 1 -- 172.21.15.59:0/2366799079 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ef400bc20 con 0x7f1f000cf850 2024-03-06T07:25:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.643+0000 7f1ee77fe640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b4d0 0x7f1eec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.643+0000 7f1ee77fe640 1 -- 172.21.15.59:0/2366799079 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1ef408a300 con 0x7f1f000cf850 2024-03-06T07:25:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.644+0000 7f1efeffd640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b4d0 0x7f1eec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.644+0000 7f1efeffd640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b4d0 0x7f1eec06d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00012e0 tx=0x7f1ef0005b90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.647+0000 7f1ee77fe640 1 -- 172.21.15.59:0/2366799079 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ef401c360 con 0x7f1f000cf850 2024-03-06T07:25:01.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.828+0000 7f1f0558a640 1 -- 172.21.15.59:0/2366799079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f0004f330 con 0x7f1f000cf850 2024-03-06T07:25:01.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.828+0000 7f1ee77fe640 1 -- 172.21.15.59:0/2366799079 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+12879 (secure 0 0 0) 0x7f1ef4059560 con 0x7f1f000cf850 2024-03-06T07:25:01.830 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:01.830 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":66,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:01.135492+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:01.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.830+0000 7f1f0558a640 1 -- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b4d0 msgr2=0x7f1eec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.830+0000 7f1f0558a640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b4d0 0x7f1eec06d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00012e0 tx=0x7f1ef0005b90 comp rx=0 tx=0).stop 2024-03-06T07:25:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.830+0000 7f1f0558a640 1 -- 172.21.15.59:0/2366799079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f000cf850 msgr2=0x7f1f000c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.830+0000 7f1f0558a640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f000cf850 0x7f1f000c9b90 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400d800 tx=0x7f1ef400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.830+0000 7f1f0558a640 1 -- 172.21.15.59:0/2366799079 shutdown_connections 2024-03-06T07:25:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.830+0000 7f1f0558a640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b4d0 0x7f1eec06d990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.830+0000 7f1f0558a640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f000d00c0 0x7f1f000cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.830+0000 7f1f0558a640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f000cf850 0x7f1f000c9b90 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.830+0000 7f1f0558a640 1 --2- 172.21.15.59:0/2366799079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f0006cf20 0x7f1f000b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.830+0000 7f1f0558a640 1 -- 172.21.15.59:0/2366799079 >> 172.21.15.59:0/2366799079 conn(0x7f1f000b35a0 msgr2=0x7f1f000b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.831+0000 7f1f0558a640 1 -- 172.21.15.59:0/2366799079 shutdown_connections 2024-03-06T07:25:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.831+0000 7f1f0558a640 1 -- 172.21.15.59:0/2366799079 wait complete. 2024-03-06T07:25:01.843 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:01.844 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:01.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.946+0000 7ff3bd9f6640 1 Processor -- start 2024-03-06T07:25:01.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.946+0000 7ff3bd9f6640 1 -- start start 2024-03-06T07:25:01.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3bd9f6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff3b80cea50 0x7ff3b80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3bd9f6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff3b806cab0 0x7ff3b80ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3bd9f6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b806d030 con 0x7ff3b815def0 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3bd9f6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b80cf1c0 con 0x7ff3b806cab0 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3bd9f6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b80cc760 con 0x7ff3b80cea50 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3b77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff3b806cab0 0x7ff3b80ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3b6ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff3b815def0 0x7ff3b80ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59422/0 (socket says 172.21.15.59:59422) 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3b6ffd640 1 -- 172.21.15.59:0/661560152 learned_addr learned my addr 172.21.15.59:0/661560152 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3b67fc640 1 --2- 172.21.15.59:0/661560152 >> v2:172.21.15.164:3300/0 conn(0x7ff3b80cea50 0x7ff3b80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3b5ffb640 1 -- 172.21.15.59:0/661560152 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708586508 0 0) 0x7ff3b806d030 con 0x7ff3b815def0 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3b5ffb640 1 -- 172.21.15.59:0/661560152 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a4003520 con 0x7ff3b815def0 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3b77fe640 1 -- 172.21.15.59:0/661560152 >> v2:172.21.15.164:3300/0 conn(0x7ff3b80cea50 msgr2=0x7ff3b80b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3b77fe640 1 --2- 172.21.15.59:0/661560152 >> v2:172.21.15.164:3300/0 conn(0x7ff3b80cea50 0x7ff3b80b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3b77fe640 1 -- 172.21.15.59:0/661560152 >> v1:172.21.15.59:6789/0 conn(0x7ff3b815def0 legacy=0x7ff3b80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.947+0000 7ff3b77fe640 1 -- 172.21.15.59:0/661560152 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b806d310 con 0x7ff3b806cab0 2024-03-06T07:25:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.948+0000 7ff3b77fe640 1 --2- 172.21.15.59:0/661560152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3b806cab0 0x7ff3b80ccbf0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff3b80b3660 tx=0x7ff3b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b8e22285295a54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.948+0000 7ff3b5ffb640 1 -- 172.21.15.59:0/661560152 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0035020 con 0x7ff3b806cab0 2024-03-06T07:25:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.948+0000 7ff3b5ffb640 1 -- 172.21.15.59:0/661560152 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b0031830 con 0x7ff3b806cab0 2024-03-06T07:25:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.948+0000 7ff3b5ffb640 1 -- 172.21.15.59:0/661560152 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0031c00 con 0x7ff3b806cab0 2024-03-06T07:25:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.949+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/661560152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3b806cab0 msgr2=0x7ff3b80ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.949+0000 7ff3bd9f6640 1 --2- 172.21.15.59:0/661560152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3b806cab0 0x7ff3b80ccbf0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff3b80b3660 tx=0x7ff3b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.949+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/661560152 shutdown_connections 2024-03-06T07:25:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.949+0000 7ff3bd9f6640 1 --2- 172.21.15.59:0/661560152 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3b806cab0 0x7ff3b80ccbf0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.949+0000 7ff3bd9f6640 1 --2- 172.21.15.59:0/661560152 >> v2:172.21.15.164:3300/0 conn(0x7ff3b80cea50 0x7ff3b80b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.949+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/661560152 >> 172.21.15.59:0/661560152 conn(0x7ff3b80b3690 msgr2=0x7ff3b80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.949+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/661560152 shutdown_connections 2024-03-06T07:25:01.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.949+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/661560152 wait complete. 2024-03-06T07:25:01.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.950+0000 7ff3bd9f6640 1 Processor -- start 2024-03-06T07:25:01.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.950+0000 7ff3bd9f6640 1 -- start start 2024-03-06T07:25:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3bd9f6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3b806cab0 0x7ff3b80b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3bd9f6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3b80cea50 0x7ff3b814fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3bd9f6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3b815def0 0x7ff3b80cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3bd9f6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b815e2d0 con 0x7ff3b815def0 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3bd9f6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b80cc3f0 con 0x7ff3b80cea50 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3bd9f6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b815e450 con 0x7ff3b806cab0 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3b77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3b815def0 0x7ff3b80cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3b77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3b815def0 0x7ff3b80cb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47954/0 (socket says 172.21.15.59:47954) 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3b77fe640 1 -- 172.21.15.59:0/531076861 learned_addr learned my addr 172.21.15.59:0/531076861 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3b77fe640 1 -- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3b806cab0 msgr2=0x7ff3b80b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3b67fc640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3b80cea50 0x7ff3b814fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3b6ffd640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3b806cab0 0x7ff3b80b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3b77fe640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3b806cab0 0x7ff3b80b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3b77fe640 1 -- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3b80cea50 msgr2=0x7ff3b814fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3b77fe640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3b80cea50 0x7ff3b814fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3b77fe640 1 -- 172.21.15.59:0/531076861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b8150350 con 0x7ff3b815def0 2024-03-06T07:25:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.951+0000 7ff3b77fe640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3b815def0 0x7ff3b80cb280 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff3b002bee0 tx=0x7ff3b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.952+0000 7ff3affff640 1 -- 172.21.15.59:0/531076861 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0035040 con 0x7ff3b815def0 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.952+0000 7ff3affff640 1 -- 172.21.15.59:0/531076861 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b003b860 con 0x7ff3b815def0 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.952+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/531076861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b80ca1e0 con 0x7ff3b815def0 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.952+0000 7ff3affff640 1 -- 172.21.15.59:0/531076861 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b003bc50 con 0x7ff3b815def0 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.952+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/531076861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b80ca7a0 con 0x7ff3b815def0 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.953+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/531076861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b809fcf0 con 0x7ff3b815def0 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.953+0000 7ff3affff640 1 -- 172.21.15.59:0/531076861 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3b0038020 con 0x7ff3b815def0 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.953+0000 7ff3affff640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff39806b4d0 0x7ff39806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.954+0000 7ff3affff640 1 -- 172.21.15.59:0/531076861 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff3b00abc00 con 0x7ff3b815def0 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.954+0000 7ff3b6ffd640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff39806b4d0 0x7ff39806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.954+0000 7ff3b6ffd640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff39806b4d0 0x7ff39806d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff3a80012e0 tx=0x7ff3a8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:01.958+0000 7ff3affff640 1 -- 172.21.15.59:0/531076861 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3b00b2080 con 0x7ff3b815def0 2024-03-06T07:25:02.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.141+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/531076861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff3b80d22e0 con 0x7ff3b815def0 2024-03-06T07:25:02.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.142+0000 7ff3affff640 1 -- 172.21.15.59:0/531076861 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff3b0031800 con 0x7ff3b815def0 2024-03-06T07:25:02.144 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:25:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff39806b4d0 msgr2=0x7ff39806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff39806b4d0 0x7ff39806d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff3a80012e0 tx=0x7ff3a8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/531076861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3b815def0 msgr2=0x7ff3b80cb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3b815def0 0x7ff3b80cb280 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff3b002bee0 tx=0x7ff3b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/531076861 shutdown_connections 2024-03-06T07:25:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff39806b4d0 0x7ff39806d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3b815def0 0x7ff3b80cb280 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3b80cea50 0x7ff3b814fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 --2- 172.21.15.59:0/531076861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3b806cab0 0x7ff3b80b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/531076861 >> 172.21.15.59:0/531076861 conn(0x7ff3b80b3690 msgr2=0x7ff3b806f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/531076861 shutdown_connections 2024-03-06T07:25:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.144+0000 7ff3bd9f6640 1 -- 172.21.15.59:0/531076861 wait complete. 2024-03-06T07:25:02.157 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:25:02.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.257+0000 7fd48306f640 1 Processor -- start 2024-03-06T07:25:02.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.257+0000 7fd48306f640 1 -- start start 2024-03-06T07:25:02.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.258+0000 7fd48306f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd47c1621c0 0x7fd47c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.258+0000 7fd48306f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd47c1625a0 0x7fd47c15dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.258+0000 7fd48306f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c0ceac0 con 0x7fd47c0d4920 2024-03-06T07:25:02.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.258+0000 7fd48306f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47c0cf690 con 0x7fd47c1621c0 2024-03-06T07:25:02.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.258+0000 7fd48306f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c0d2dc0 con 0x7fd47c1625a0 2024-03-06T07:25:02.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.258+0000 7fd480e0b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd47c1621c0 0x7fd47c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.258+0000 7fd47bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd47c0d4920 0x7fd47c0d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59436/0 (socket says 172.21.15.59:59436) 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.258+0000 7fd47bfff640 1 -- 172.21.15.59:0/2051779890 learned_addr learned my addr 172.21.15.59:0/2051779890 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.258+0000 7fd480e0b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd47c1621c0 0x7fd47c0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47958/0 (socket says 172.21.15.59:47958) 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.258+0000 7fd48160c640 1 --2- 172.21.15.59:0/2051779890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd47c1625a0 0x7fd47c15dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.258+0000 7fd480e0b640 1 -- 172.21.15.59:0/2051779890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd47c1625a0 msgr2=0x7fd47c15dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.259+0000 7fd480e0b640 1 --2- 172.21.15.59:0/2051779890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd47c1625a0 0x7fd47c15dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.259+0000 7fd480e0b640 1 -- 172.21.15.59:0/2051779890 >> v1:172.21.15.59:6789/0 conn(0x7fd47c0d4920 legacy=0x7fd47c0d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.259+0000 7fd480e0b640 1 -- 172.21.15.59:0/2051779890 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c0ced60 con 0x7fd47c1621c0 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.259+0000 7fd47b7fe640 1 -- 172.21.15.59:0/2051779890 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656835459 0 0) 0x7fd47c0ceac0 con 0x7fd47c0d4920 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.259+0000 7fd480e0b640 1 --2- 172.21.15.59:0/2051779890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd47c1621c0 0x7fd47c0cfb20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0097a0 tx=0x7fd46c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9611b1da70868f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.259+0000 7fd47b7fe640 1 -- 172.21.15.59:0/2051779890 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c035020 con 0x7fd47c1621c0 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.259+0000 7fd47b7fe640 1 -- 172.21.15.59:0/2051779890 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c031830 con 0x7fd47c1621c0 2024-03-06T07:25:02.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.259+0000 7fd47b7fe640 1 -- 172.21.15.59:0/2051779890 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c031c00 con 0x7fd47c1621c0 2024-03-06T07:25:02.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.260+0000 7fd48306f640 1 -- 172.21.15.59:0/2051779890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd47c1621c0 msgr2=0x7fd47c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.260+0000 7fd48306f640 1 --2- 172.21.15.59:0/2051779890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd47c1621c0 0x7fd47c0cfb20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0097a0 tx=0x7fd46c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.260+0000 7fd48306f640 1 -- 172.21.15.59:0/2051779890 shutdown_connections 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.260+0000 7fd48306f640 1 --2- 172.21.15.59:0/2051779890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd47c1625a0 0x7fd47c15dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.260+0000 7fd48306f640 1 --2- 172.21.15.59:0/2051779890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd47c1621c0 0x7fd47c0cfb20 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.260+0000 7fd48306f640 1 -- 172.21.15.59:0/2051779890 >> 172.21.15.59:0/2051779890 conn(0x7fd47c0d3d10 msgr2=0x7fd47c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.260+0000 7fd48306f640 1 -- 172.21.15.59:0/2051779890 shutdown_connections 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.260+0000 7fd48306f640 1 -- 172.21.15.59:0/2051779890 wait complete. 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.261+0000 7fd48306f640 1 Processor -- start 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.261+0000 7fd48306f640 1 -- start start 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd48306f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd47c0d4920 0x7fd47c0b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd48306f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd47c1621c0 0x7fd47c151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd48306f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd47c1625a0 0x7fd47c0b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd48306f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c15e280 con 0x7fd47c1621c0 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd48306f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c0cf320 con 0x7fd47c1625a0 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd48306f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c15e400 con 0x7fd47c0d4920 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd47bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd47c1621c0 0x7fd47c151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd47bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd47c1621c0 0x7fd47c151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47966/0 (socket says 172.21.15.59:47966) 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd47bfff640 1 -- 172.21.15.59:0/693868942 learned_addr learned my addr 172.21.15.59:0/693868942 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd48160c640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd47c1625a0 0x7fd47c0b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd480e0b640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd47c0d4920 0x7fd47c0b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd47bfff640 1 -- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd47c0d4920 msgr2=0x7fd47c0b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd47bfff640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd47c0d4920 0x7fd47c0b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd47bfff640 1 -- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd47c1625a0 msgr2=0x7fd47c0b14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd47bfff640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd47c1625a0 0x7fd47c0b14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd47bfff640 1 -- 172.21.15.59:0/693868942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c0b40d0 con 0x7fd47c1621c0 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.262+0000 7fd480e0b640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd47c0d4920 0x7fd47c0b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.263+0000 7fd47bfff640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd47c1621c0 0x7fd47c151280 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd47000c500 tx=0x7fd47000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.263+0000 7fd4797fa640 1 -- 172.21.15.59:0/693868942 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470018070 con 0x7fd47c1621c0 2024-03-06T07:25:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.263+0000 7fd4797fa640 1 -- 172.21.15.59:0/693868942 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd470007580 con 0x7fd47c1621c0 2024-03-06T07:25:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.263+0000 7fd48306f640 1 -- 172.21.15.59:0/693868942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47c0b43b0 con 0x7fd47c1621c0 2024-03-06T07:25:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.263+0000 7fd4797fa640 1 -- 172.21.15.59:0/693868942 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470004a20 con 0x7fd47c1621c0 2024-03-06T07:25:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.263+0000 7fd48306f640 1 -- 172.21.15.59:0/693868942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47c0b1a20 con 0x7fd47c1621c0 2024-03-06T07:25:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.264+0000 7fd48306f640 1 -- 172.21.15.59:0/693868942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47c04f330 con 0x7fd47c1621c0 2024-03-06T07:25:02.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.264+0000 7fd4797fa640 1 -- 172.21.15.59:0/693868942 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd470004bc0 con 0x7fd47c1621c0 2024-03-06T07:25:02.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.265+0000 7fd4797fa640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd46806b4d0 0x7fd46806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.265+0000 7fd4797fa640 1 -- 172.21.15.59:0/693868942 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd4700891c0 con 0x7fd47c1621c0 2024-03-06T07:25:02.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.265+0000 7fd480e0b640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd46806b4d0 0x7fd46806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.265+0000 7fd480e0b640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd46806b4d0 0x7fd46806d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd46c036000 tx=0x7fd46c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.269+0000 7fd4797fa640 1 -- 172.21.15.59:0/693868942 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4700583a0 con 0x7fd47c1621c0 2024-03-06T07:25:02.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.450+0000 7fd48306f640 1 -- 172.21.15.59:0/693868942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd47c073380 con 0x7fd47c1621c0 2024-03-06T07:25:02.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.450+0000 7fd4797fa640 1 -- 172.21.15.59:0/693868942 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v66) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd470057ca0 con 0x7fd47c1621c0 2024-03-06T07:25:02.453 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:02.453 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-03-06T07:25:02.453 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:02.453 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-03-06T07:25:02.453 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-03-06T07:25:02.453 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:02.453 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-03-06T07:25:02.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 -- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd46806b4d0 msgr2=0x7fd46806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd46806b4d0 0x7fd46806d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd46c036000 tx=0x7fd46c008030 comp rx=0 tx=0).stop 2024-03-06T07:25:02.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 -- 172.21.15.59:0/693868942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd47c1621c0 msgr2=0x7fd47c151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd47c1621c0 0x7fd47c151280 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd47000c500 tx=0x7fd47000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 -- 172.21.15.59:0/693868942 shutdown_connections 2024-03-06T07:25:02.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd46806b4d0 0x7fd46806d990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd47c1625a0 0x7fd47c0b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd47c1621c0 0x7fd47c151280 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 --2- 172.21.15.59:0/693868942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd47c0d4920 0x7fd47c0b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 -- 172.21.15.59:0/693868942 >> 172.21.15.59:0/693868942 conn(0x7fd47c0d3d10 msgr2=0x7fd47c06d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 -- 172.21.15.59:0/693868942 shutdown_connections 2024-03-06T07:25:02.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.452+0000 7fd48306f640 1 -- 172.21.15.59:0/693868942 wait complete. 2024-03-06T07:25:02.465 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:25:02.465 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:25:02.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.569+0000 7fd297e0e640 1 Processor -- start 2024-03-06T07:25:02.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.569+0000 7fd297e0e640 1 -- start start 2024-03-06T07:25:02.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.570+0000 7fd297e0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 0x7fd29015c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.570+0000 7fd297e0e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd29015c980 0x7fd29006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.570+0000 7fd297e0e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29006f490 con 0x7fd29006b4f0 2024-03-06T07:25:02.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.570+0000 7fd297e0e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd290071db0 con 0x7fd29015c980 2024-03-06T07:25:02.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.570+0000 7fd297e0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29006ebf0 con 0x7fd290145980 2024-03-06T07:25:02.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.570+0000 7fd2953a9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd29015c980 0x7fd29006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.570+0000 7fd2953a9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd29015c980 0x7fd29006afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47976/0 (socket says 172.21.15.59:47976) 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.570+0000 7fd2953a9640 1 -- 172.21.15.59:0/1396932073 learned_addr learned my addr 172.21.15.59:0/1396932073 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.571+0000 7fd295baa640 1 --2- 172.21.15.59:0/1396932073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 0x7fd29015c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.571+0000 7fd2953a9640 1 -- 172.21.15.59:0/1396932073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 msgr2=0x7fd29015c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.571+0000 7fd2953a9640 1 --2- 172.21.15.59:0/1396932073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 0x7fd29015c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.571+0000 7fd2953a9640 1 -- 172.21.15.59:0/1396932073 >> v1:172.21.15.164:6789/0 conn(0x7fd29006b4f0 legacy=0x7fd2900cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.571+0000 7fd2953a9640 1 -- 172.21.15.59:0/1396932073 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29006eed0 con 0x7fd29015c980 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.571+0000 7fd295baa640 1 --2- 172.21.15.59:0/1396932073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 0x7fd29015c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.571+0000 7fd2953a9640 1 --2- 172.21.15.59:0/1396932073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd29015c980 0x7fd29006afb0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd284009870 tx=0x7fd28402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c6292dea5c804f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.571+0000 7fd294ba8640 1 -- 172.21.15.59:0/1396932073 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284035020 con 0x7fd29015c980 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.572+0000 7fd294ba8640 1 -- 172.21.15.59:0/1396932073 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd284031640 con 0x7fd29015c980 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.572+0000 7fd294ba8640 1 -- 172.21.15.59:0/1396932073 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28402d040 con 0x7fd29015c980 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.572+0000 7fd297e0e640 1 -- 172.21.15.59:0/1396932073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd29015c980 msgr2=0x7fd29006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.572+0000 7fd297e0e640 1 --2- 172.21.15.59:0/1396932073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd29015c980 0x7fd29006afb0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd284009870 tx=0x7fd28402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.572+0000 7fd297e0e640 1 -- 172.21.15.59:0/1396932073 shutdown_connections 2024-03-06T07:25:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.572+0000 7fd297e0e640 1 --2- 172.21.15.59:0/1396932073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd29015c980 0x7fd29006afb0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.572+0000 7fd297e0e640 1 --2- 172.21.15.59:0/1396932073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 0x7fd29015c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.572+0000 7fd297e0e640 1 -- 172.21.15.59:0/1396932073 >> 172.21.15.59:0/1396932073 conn(0x7fd2900d0a30 msgr2=0x7fd2900d0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.572+0000 7fd297e0e640 1 -- 172.21.15.59:0/1396932073 shutdown_connections 2024-03-06T07:25:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.572+0000 7fd297e0e640 1 -- 172.21.15.59:0/1396932073 wait complete. 2024-03-06T07:25:02.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.573+0000 7fd297e0e640 1 Processor -- start 2024-03-06T07:25:02.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.573+0000 7fd297e0e640 1 -- start start 2024-03-06T07:25:02.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd297e0e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd29006b4f0 0x7fd29015b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd297e0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 0x7fd2900b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd297e0e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2900b1380 0x7fd2900b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd297e0e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900cfa20 con 0x7fd29006b4f0 2024-03-06T07:25:02.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd297e0e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd290071a40 con 0x7fd2900b1380 2024-03-06T07:25:02.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd297e0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900cfba0 con 0x7fd290145980 2024-03-06T07:25:02.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd295baa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd29006b4f0 0x7fd29015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd295baa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd29006b4f0 0x7fd29015b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47980/0 (socket says 172.21.15.59:47980) 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd295baa640 1 -- 172.21.15.59:0/1272448567 learned_addr learned my addr 172.21.15.59:0/1272448567 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd2953a9640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 0x7fd2900b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd2963ab640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2900b1380 0x7fd2900b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd295baa640 1 -- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 msgr2=0x7fd2900b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd295baa640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 0x7fd2900b3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd295baa640 1 -- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2900b1380 msgr2=0x7fd2900b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd295baa640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2900b1380 0x7fd2900b1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd295baa640 1 -- 172.21.15.59:0/1272448567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2900b4840 con 0x7fd29006b4f0 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd2953a9640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 0x7fd2900b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.574+0000 7fd2963ab640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2900b1380 0x7fd2900b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.575+0000 7fd295baa640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd29006b4f0 0x7fd29015b690 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd28800c7f0 tx=0x7fd28800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.575+0000 7fd28e7fc640 1 -- 172.21.15.59:0/1272448567 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd288015070 con 0x7fd29006b4f0 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.575+0000 7fd28e7fc640 1 -- 172.21.15.59:0/1272448567 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd288007a30 con 0x7fd29006b4f0 2024-03-06T07:25:02.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.575+0000 7fd297e0e640 1 -- 172.21.15.59:0/1272448567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2900b4b20 con 0x7fd29006b4f0 2024-03-06T07:25:02.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.575+0000 7fd28e7fc640 1 -- 172.21.15.59:0/1272448567 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd288007e00 con 0x7fd29006b4f0 2024-03-06T07:25:02.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.575+0000 7fd297e0e640 1 -- 172.21.15.59:0/1272448567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2900b50e0 con 0x7fd29006b4f0 2024-03-06T07:25:02.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.576+0000 7fd297e0e640 1 -- 172.21.15.59:0/1272448567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29004f330 con 0x7fd29006b4f0 2024-03-06T07:25:02.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.576+0000 7fd28e7fc640 1 -- 172.21.15.59:0/1272448567 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd288018070 con 0x7fd29006b4f0 2024-03-06T07:25:02.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.577+0000 7fd28e7fc640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd27c06b4d0 0x7fd27c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.577+0000 7fd28e7fc640 1 -- 172.21.15.59:0/1272448567 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd288089e60 con 0x7fd29006b4f0 2024-03-06T07:25:02.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.577+0000 7fd2953a9640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd27c06b4d0 0x7fd27c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.577+0000 7fd2953a9640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd27c06b4d0 0x7fd27c06d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd284036000 tx=0x7fd28403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.581+0000 7fd28e7fc640 1 -- 172.21.15.59:0/1272448567 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd288059030 con 0x7fd29006b4f0 2024-03-06T07:25:02.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:02.759+0000 7fd297e0e640 1 -- 172.21.15.59:0/1272448567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd29006ce60 con 0x7fd29006b4f0 2024-03-06T07:25:03.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.143+0000 7fd28e7fc640 1 -- 172.21.15.59:0/1272448567 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v67) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd288058930 con 0x7fd29006b4f0 2024-03-06T07:25:03.145 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-03-06T07:25:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 -- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd27c06b4d0 msgr2=0x7fd27c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd27c06b4d0 0x7fd27c06d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd284036000 tx=0x7fd28403c040 comp rx=0 tx=0).stop 2024-03-06T07:25:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 -- 172.21.15.59:0/1272448567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd29006b4f0 msgr2=0x7fd29015b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd29006b4f0 0x7fd29015b690 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd28800c7f0 tx=0x7fd28800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 -- 172.21.15.59:0/1272448567 shutdown_connections 2024-03-06T07:25:03.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd27c06b4d0 0x7fd27c06d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2900b1380 0x7fd2900b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd290145980 0x7fd2900b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 --2- 172.21.15.59:0/1272448567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd29006b4f0 0x7fd29015b690 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 -- 172.21.15.59:0/1272448567 >> 172.21.15.59:0/1272448567 conn(0x7fd2900d0a30 msgr2=0x7fd29015ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 -- 172.21.15.59:0/1272448567 shutdown_connections 2024-03-06T07:25:03.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.145+0000 7fd297e0e640 1 -- 172.21.15.59:0/1272448567 wait complete. 2024-03-06T07:25:03.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:25:03.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.270+0000 7fd8657bc640 1 Processor -- start 2024-03-06T07:25:03.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.270+0000 7fd8657bc640 1 -- start start 2024-03-06T07:25:03.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.270+0000 7fd8657bc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd86015bbb0 0x7fd86006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.270+0000 7fd8657bc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd86006b4f0 0x7fd86015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.270+0000 7fd8657bc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8600cea90 con 0x7fd86015ec40 2024-03-06T07:25:03.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.270+0000 7fd8657bc640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd860161970 con 0x7fd86015bbb0 2024-03-06T07:25:03.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.270+0000 7fd8657bc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8601624f0 con 0x7fd86006b4f0 2024-03-06T07:25:03.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.271+0000 7fd85effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd86015bbb0 0x7fd86006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.271+0000 7fd85effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd86015bbb0 0x7fd86006afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47988/0 (socket says 172.21.15.59:47988) 2024-03-06T07:25:03.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.271+0000 7fd85effd640 1 -- 172.21.15.59:0/66664595 learned_addr learned my addr 172.21.15.59:0/66664595 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:03.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.271+0000 7fd85e7fc640 1 --2- 172.21.15.59:0/66664595 >> v2:172.21.15.164:3300/0 conn(0x7fd86006b4f0 0x7fd86015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.271+0000 7fd85effd640 1 -- 172.21.15.59:0/66664595 >> v2:172.21.15.164:3300/0 conn(0x7fd86006b4f0 msgr2=0x7fd86015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:03.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.271+0000 7fd85effd640 1 --2- 172.21.15.59:0/66664595 >> v2:172.21.15.164:3300/0 conn(0x7fd86006b4f0 0x7fd86015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.271+0000 7fd85effd640 1 -- 172.21.15.59:0/66664595 >> v1:172.21.15.164:6789/0 conn(0x7fd86015ec40 legacy=0x7fd8600cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:03.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.271+0000 7fd85effd640 1 -- 172.21.15.59:0/66664595 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860161bc0 con 0x7fd86015bbb0 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd85effd640 1 --2- 172.21.15.59:0/66664595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd86015bbb0 0x7fd86006afb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd86006dcc0 tx=0x7fd85002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af04307bbc07f6e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd85dffb640 1 -- 172.21.15.59:0/66664595 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850034070 con 0x7fd86015bbb0 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd85dffb640 1 -- 172.21.15.59:0/66664595 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd850030860 con 0x7fd86015bbb0 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd85dffb640 1 -- 172.21.15.59:0/66664595 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850030c30 con 0x7fd86015bbb0 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd8657bc640 1 -- 172.21.15.59:0/66664595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd86015bbb0 msgr2=0x7fd86006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd8657bc640 1 --2- 172.21.15.59:0/66664595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd86015bbb0 0x7fd86006afb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd86006dcc0 tx=0x7fd85002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd8657bc640 1 -- 172.21.15.59:0/66664595 shutdown_connections 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd8657bc640 1 --2- 172.21.15.59:0/66664595 >> v2:172.21.15.164:3300/0 conn(0x7fd86006b4f0 0x7fd86015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd8657bc640 1 --2- 172.21.15.59:0/66664595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd86015bbb0 0x7fd86006afb0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd8657bc640 1 -- 172.21.15.59:0/66664595 >> 172.21.15.59:0/66664595 conn(0x7fd8600b2660 msgr2=0x7fd8600b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd8657bc640 1 -- 172.21.15.59:0/66664595 shutdown_connections 2024-03-06T07:25:03.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.272+0000 7fd8657bc640 1 -- 172.21.15.59:0/66664595 wait complete. 2024-03-06T07:25:03.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.274+0000 7fd8657bc640 1 Processor -- start 2024-03-06T07:25:03.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.274+0000 7fd8657bc640 1 -- start start 2024-03-06T07:25:03.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.274+0000 7fd8657bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd86006b4f0 0x7fd8601f8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.274+0000 7fd8657bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd86015ec40 0x7fd860143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.274+0000 7fd8657bc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd860160ec0 0x7fd86015cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.274+0000 7fd8657bc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860072cb0 con 0x7fd86006b4f0 2024-03-06T07:25:03.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.274+0000 7fd8657bc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860162180 con 0x7fd860160ec0 2024-03-06T07:25:03.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.274+0000 7fd8657bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86015f0d0 con 0x7fd86015ec40 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd85effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd86006b4f0 0x7fd8601f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd85effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd86006b4f0 0x7fd8601f8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47990/0 (socket says 172.21.15.59:47990) 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd85effd640 1 -- 172.21.15.59:0/60926442 learned_addr learned my addr 172.21.15.59:0/60926442 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd85f7fe640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd860160ec0 0x7fd86015cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd85e7fc640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd86015ec40 0x7fd860143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd85effd640 1 -- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd86015ec40 msgr2=0x7fd860143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd85effd640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd86015ec40 0x7fd860143fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd85effd640 1 -- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd860160ec0 msgr2=0x7fd86015cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd85effd640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd860160ec0 0x7fd86015cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd85effd640 1 -- 172.21.15.59:0/60926442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860159ea0 con 0x7fd86006b4f0 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd85effd640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd86006b4f0 0x7fd8601f8000 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd850000c00 tx=0x7fd8500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd8477fe640 1 -- 172.21.15.59:0/60926442 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850034040 con 0x7fd86006b4f0 2024-03-06T07:25:03.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd8477fe640 1 -- 172.21.15.59:0/60926442 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd850039d90 con 0x7fd86006b4f0 2024-03-06T07:25:03.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd8477fe640 1 -- 172.21.15.59:0/60926442 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850043610 con 0x7fd86006b4f0 2024-03-06T07:25:03.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd8657bc640 1 -- 172.21.15.59:0/60926442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86015a110 con 0x7fd86006b4f0 2024-03-06T07:25:03.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.275+0000 7fd8657bc640 1 -- 172.21.15.59:0/60926442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd860159850 con 0x7fd86006b4f0 2024-03-06T07:25:03.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.276+0000 7fd8657bc640 1 -- 172.21.15.59:0/60926442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8600733b0 con 0x7fd86006b4f0 2024-03-06T07:25:03.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.277+0000 7fd8477fe640 1 -- 172.21.15.59:0/60926442 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd850037070 con 0x7fd86006b4f0 2024-03-06T07:25:03.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.278+0000 7fd8477fe640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84c06b4d0 0x7fd84c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.278+0000 7fd8477fe640 1 -- 172.21.15.59:0/60926442 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd8500ac880 con 0x7fd86006b4f0 2024-03-06T07:25:03.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.278+0000 7fd85e7fc640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84c06b4d0 0x7fd84c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.279+0000 7fd85e7fc640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84c06b4d0 0x7fd84c06d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd8600b3520 tx=0x7fd854008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.282+0000 7fd8477fe640 1 -- 172.21.15.59:0/60926442 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd85003e070 con 0x7fd86006b4f0 2024-03-06T07:25:03.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:03.462+0000 7fd8657bc640 1 -- 172.21.15.59:0/60926442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd860068ff0 con 0x7fd86006b4f0 2024-03-06T07:25:04.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.149+0000 7fd8477fe640 1 -- 172.21.15.59:0/60926442 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v68) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd85007b370 con 0x7fd86006b4f0 2024-03-06T07:25:04.151 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 -- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84c06b4d0 msgr2=0x7fd84c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84c06b4d0 0x7fd84c06d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd8600b3520 tx=0x7fd854008040 comp rx=0 tx=0).stop 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 -- 172.21.15.59:0/60926442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd86006b4f0 msgr2=0x7fd8601f8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd86006b4f0 0x7fd8601f8000 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd850000c00 tx=0x7fd8500046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 -- 172.21.15.59:0/60926442 shutdown_connections 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84c06b4d0 0x7fd84c06d990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd860160ec0 0x7fd86015cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd86015ec40 0x7fd860143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 --2- 172.21.15.59:0/60926442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd86006b4f0 0x7fd8601f8000 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 -- 172.21.15.59:0/60926442 >> 172.21.15.59:0/60926442 conn(0x7fd8600b2660 msgr2=0x7fd86014d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 -- 172.21.15.59:0/60926442 shutdown_connections 2024-03-06T07:25:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.152+0000 7fd8657bc640 1 -- 172.21.15.59:0/60926442 wait complete. 2024-03-06T07:25:04.178 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:25:04.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.284+0000 7f2439037640 1 Processor -- start 2024-03-06T07:25:04.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.284+0000 7f2439037640 1 -- start start 2024-03-06T07:25:04.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2439037640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2434072a30 0x7f243415ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2439037640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f243415e8a0 con 0x7f2434158800 2024-03-06T07:25:04.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2439037640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f243415eb80 con 0x7f2434158420 2024-03-06T07:25:04.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2439037640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f243415e650 con 0x7f2434072a30 2024-03-06T07:25:04.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f243259c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2434158800 0x7f243415b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58740/0 (socket says 172.21.15.59:58740) 2024-03-06T07:25:04.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f243259c640 1 -- 172.21.15.59:0/2670901004 learned_addr learned my addr 172.21.15.59:0/2670901004 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:04.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f243359e640 1 --2- 172.21.15.59:0/2670901004 >> v2:172.21.15.164:3300/0 conn(0x7f2434072a30 0x7f243415ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433747893 0 0) 0x7f243415e8a0 con 0x7f2434158800 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f241c003520 con 0x7f2434158800 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47085684 0 0) 0x7f241c003520 con 0x7f2434158800 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 >> v2:172.21.15.164:3300/0 conn(0x7f2434072a30 msgr2=0x7f243415ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2431d9b640 1 --2- 172.21.15.59:0/2670901004 >> v2:172.21.15.164:3300/0 conn(0x7f2434072a30 0x7f243415ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 >> v1:172.21.15.164:6789/0 conn(0x7f2434158420 legacy=0x7f24340b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243415ee60 con 0x7f2434158800 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f2428001670 con 0x7f2434158800 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.285+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 >> v1:172.21.15.59:6789/0 conn(0x7f2434158800 legacy=0x7f243415b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2431d9b640 1 --2- 172.21.15.59:0/2670901004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c0039b0 0x7f241c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2431d9b640 1 --2- 172.21.15.59:0/2670901004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c0043b0 0x7f241c004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2431d9b640 1 --2- 172.21.15.59:0/2670901004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f241c005120 0x7f241c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243415e650 con 0x7f241c0039b0 2024-03-06T07:25:04.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c00c060 con 0x7f241c0043b0 2024-03-06T07:25:04.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c00e470 con 0x7f241c005120 2024-03-06T07:25:04.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2432d9d640 1 --2- 172.21.15.59:0/2670901004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c0039b0 0x7f241c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f243259c640 1 --2- 172.21.15.59:0/2670901004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c0043b0 0x7f241c004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f243359e640 1 --2- 172.21.15.59:0/2670901004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f241c005120 0x7f241c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2432d9d640 1 -- 172.21.15.59:0/2670901004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f241c005120 msgr2=0x7f241c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2432d9d640 1 --2- 172.21.15.59:0/2670901004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f241c005120 0x7f241c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2432d9d640 1 -- 172.21.15.59:0/2670901004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c0043b0 msgr2=0x7f241c004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:04.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2432d9d640 1 --2- 172.21.15.59:0/2670901004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c0043b0 0x7f241c004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f2432d9d640 1 -- 172.21.15.59:0/2670901004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c003520 con 0x7f241c0039b0 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.286+0000 7f243359e640 1 -- 172.21.15.59:0/2670901004 reap_dead start 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.287+0000 7f2432d9d640 1 --2- 172.21.15.59:0/2670901004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c0039b0 0x7f241c003d90 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f242c001860 tx=0x7f241c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1847135163aa231 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.287+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c038070 con 0x7f241c0039b0 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.287+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242c033410 con 0x7f241c0039b0 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.287+0000 7f2431d9b640 1 -- 172.21.15.59:0/2670901004 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c001430 con 0x7f241c0039b0 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.287+0000 7f2439037640 1 -- 172.21.15.59:0/2670901004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c0039b0 msgr2=0x7f241c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.287+0000 7f2439037640 1 --2- 172.21.15.59:0/2670901004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c0039b0 0x7f241c003d90 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f242c001860 tx=0x7f241c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.287+0000 7f2439037640 1 -- 172.21.15.59:0/2670901004 shutdown_connections 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.287+0000 7f2439037640 1 --2- 172.21.15.59:0/2670901004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f241c0039b0 0x7f241c003d90 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.287+0000 7f2439037640 1 -- 172.21.15.59:0/2670901004 >> 172.21.15.59:0/2670901004 conn(0x7f243406afb0 msgr2=0x7f243406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.287+0000 7f2439037640 1 -- 172.21.15.59:0/2670901004 shutdown_connections 2024-03-06T07:25:04.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.287+0000 7f2439037640 1 -- 172.21.15.59:0/2670901004 wait complete. 2024-03-06T07:25:04.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.289+0000 7f2439037640 1 Processor -- start 2024-03-06T07:25:04.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.289+0000 7f2439037640 1 -- start start 2024-03-06T07:25:04.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.289+0000 7f2439037640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c0039b0 0x7f2434153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.289+0000 7f2439037640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2434151f70 0x7f24341f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f2439037640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24341f3c70 0x7f24341f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f2439037640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340b3740 con 0x7f2434151f70 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f2439037640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340b35c0 con 0x7f241c0039b0 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f2439037640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24340b38c0 con 0x7f24341f3c70 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f243259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2434151f70 0x7f24341f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f243259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2434151f70 0x7f24341f30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54598/0 (socket says 172.21.15.59:54598) 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f243259c640 1 -- 172.21.15.59:0/2374828532 learned_addr learned my addr 172.21.15.59:0/2374828532 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f2432d9d640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c0039b0 0x7f2434153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f243359e640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24341f3c70 0x7f24341f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f243259c640 1 -- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24341f3c70 msgr2=0x7f24341f3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f243259c640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24341f3c70 0x7f24341f3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f243259c640 1 -- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c0039b0 msgr2=0x7f2434153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f243259c640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c0039b0 0x7f2434153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f243259c640 1 -- 172.21.15.59:0/2374828532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2434153b40 con 0x7f2434151f70 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f243359e640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24341f3c70 0x7f24341f3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f2432d9d640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c0039b0 0x7f2434153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f243259c640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2434151f70 0x7f24341f30d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f2428007fd0 tx=0x7f2428007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f241b7fe640 1 -- 172.21.15.59:0/2374828532 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242800f750 con 0x7f2434151f70 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f241b7fe640 1 -- 172.21.15.59:0/2374828532 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2428018070 con 0x7f2434151f70 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f241b7fe640 1 -- 172.21.15.59:0/2374828532 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242800fb50 con 0x7f2434151f70 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f2439037640 1 -- 172.21.15.59:0/2374828532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24340cd8d0 con 0x7f2434151f70 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.290+0000 7f2439037640 1 -- 172.21.15.59:0/2374828532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24340cde90 con 0x7f2434151f70 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.291+0000 7f2439037640 1 -- 172.21.15.59:0/2374828532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f243404f330 con 0x7f2434151f70 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.292+0000 7f241b7fe640 1 -- 172.21.15.59:0/2374828532 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f242800fcf0 con 0x7f2434151f70 2024-03-06T07:25:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.292+0000 7f241b7fe640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f242006b4d0 0x7f242006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:04.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.292+0000 7f241b7fe640 1 -- 172.21.15.59:0/2374828532 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f242808a0d0 con 0x7f2434151f70 2024-03-06T07:25:04.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.292+0000 7f2432d9d640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f242006b4d0 0x7f242006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:04.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.293+0000 7f2432d9d640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f242006b4d0 0x7f242006d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f242c035000 tx=0x7f242c009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:04.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.297+0000 7f241b7fe640 1 -- 172.21.15.59:0/2374828532 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2428058fb0 con 0x7f2434151f70 2024-03-06T07:25:04.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:04.515+0000 7f2439037640 1 -- 172.21.15.59:0/2374828532 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f24340cc1d0 con 0x7f2434151f70 2024-03-06T07:25:05.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.151+0000 7f241b7fe640 1 -- 172.21.15.59:0/2374828532 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v33) v1 ==== 174+0+0 (secure 0 0 0) 0x7f24280588b0 con 0x7f2434151f70 2024-03-06T07:25:05.153 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 10 and data pool 11 2024-03-06T07:25:05.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.155+0000 7f2439037640 1 -- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f242006b4d0 msgr2=0x7f242006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.155+0000 7f2439037640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f242006b4d0 0x7f242006d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f242c035000 tx=0x7f242c009030 comp rx=0 tx=0).stop 2024-03-06T07:25:05.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.155+0000 7f2439037640 1 -- 172.21.15.59:0/2374828532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2434151f70 msgr2=0x7f24341f30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.155+0000 7f2439037640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2434151f70 0x7f24341f30d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f2428007fd0 tx=0x7f2428007430 comp rx=0 tx=0).stop 2024-03-06T07:25:05.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.156+0000 7f2439037640 1 -- 172.21.15.59:0/2374828532 shutdown_connections 2024-03-06T07:25:05.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.156+0000 7f2439037640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f242006b4d0 0x7f242006d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.156+0000 7f2439037640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24341f3c70 0x7f24341f3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.156+0000 7f2439037640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2434151f70 0x7f24341f30d0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.156+0000 7f2439037640 1 --2- 172.21.15.59:0/2374828532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f241c0039b0 0x7f2434153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.156+0000 7f2439037640 1 -- 172.21.15.59:0/2374828532 >> 172.21.15.59:0/2374828532 conn(0x7f243406afb0 msgr2=0x7f24341550c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.158+0000 7f2439037640 1 -- 172.21.15.59:0/2374828532 shutdown_connections 2024-03-06T07:25:05.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.158+0000 7f2439037640 1 -- 172.21.15.59:0/2374828532 wait complete. 2024-03-06T07:25:05.185 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:05.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.290+0000 7fd9eca81640 1 Processor -- start 2024-03-06T07:25:05.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.290+0000 7fd9eca81640 1 -- start start 2024-03-06T07:25:05.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.290+0000 7fd9eca81640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd9e80d00c0 0x7fd9e80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.291+0000 7fd9eca81640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e80d09e0 con 0x7fd9e80cf850 2024-03-06T07:25:05.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.291+0000 7fd9eca81640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e815e380 con 0x7fd9e806cf20 2024-03-06T07:25:05.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.291+0000 7fd9eca81640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e806ca90 con 0x7fd9e80d00c0 2024-03-06T07:25:05.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.291+0000 7fd9e659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd9e80d00c0 0x7fd9e80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.291+0000 7fd9e659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd9e80d00c0 0x7fd9e80d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54614/0 (socket says 172.21.15.59:54614) 2024-03-06T07:25:05.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.291+0000 7fd9e5d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd9e80cf850 0x7fd9e80cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58742/0 (socket says 172.21.15.59:58742) 2024-03-06T07:25:05.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.291+0000 7fd9e659c640 1 -- 172.21.15.59:0/3118553064 learned_addr learned my addr 172.21.15.59:0/3118553064 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:05.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.291+0000 7fd9e659c640 1 -- 172.21.15.59:0/3118553064 >> v1:172.21.15.164:6789/0 conn(0x7fd9e806cf20 legacy=0x7fd9e806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.291+0000 7fd9e659c640 1 -- 172.21.15.59:0/3118553064 >> v1:172.21.15.59:6789/0 conn(0x7fd9e80cf850 legacy=0x7fd9e80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.291+0000 7fd9e659c640 1 -- 172.21.15.59:0/3118553064 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e80cdc10 con 0x7fd9e80d00c0 2024-03-06T07:25:05.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.292+0000 7fd9e659c640 1 --2- 172.21.15.59:0/3118553064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e80d00c0 0x7fd9e80d04a0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009e50 tx=0x7fd9d802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6db949b7887f454 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.292+0000 7fd9e559a640 1 -- 172.21.15.59:0/3118553064 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d802c960 con 0x7fd9e80d00c0 2024-03-06T07:25:05.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.292+0000 7fd9e559a640 1 -- 172.21.15.59:0/3118553064 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d802cb00 con 0x7fd9e80d00c0 2024-03-06T07:25:05.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.292+0000 7fd9e559a640 1 -- 172.21.15.59:0/3118553064 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d80343f0 con 0x7fd9e80d00c0 2024-03-06T07:25:05.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.292+0000 7fd9eca81640 1 -- 172.21.15.59:0/3118553064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e80d00c0 msgr2=0x7fd9e80d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.292+0000 7fd9eca81640 1 --2- 172.21.15.59:0/3118553064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e80d00c0 0x7fd9e80d04a0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009e50 tx=0x7fd9d802be40 comp rx=0 tx=0).stop 2024-03-06T07:25:05.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.292+0000 7fd9eca81640 1 -- 172.21.15.59:0/3118553064 shutdown_connections 2024-03-06T07:25:05.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.292+0000 7fd9eca81640 1 --2- 172.21.15.59:0/3118553064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e80d00c0 0x7fd9e80d04a0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.293+0000 7fd9eca81640 1 -- 172.21.15.59:0/3118553064 >> 172.21.15.59:0/3118553064 conn(0x7fd9e80b35a0 msgr2=0x7fd9e80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.293+0000 7fd9eca81640 1 -- 172.21.15.59:0/3118553064 shutdown_connections 2024-03-06T07:25:05.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.293+0000 7fd9eca81640 1 -- 172.21.15.59:0/3118553064 wait complete. 2024-03-06T07:25:05.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.294+0000 7fd9eca81640 1 Processor -- start 2024-03-06T07:25:05.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.294+0000 7fd9eca81640 1 -- start start 2024-03-06T07:25:05.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9eca81640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e806cf20 0x7fd9e80c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9eca81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e80cf850 0x7fd9e80b6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9eca81640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e80d00c0 0x7fd9e80b6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9eca81640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e806c720 con 0x7fd9e80cf850 2024-03-06T07:25:05.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9eca81640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e806c440 con 0x7fd9e80d00c0 2024-03-06T07:25:05.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9eca81640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e80b1030 con 0x7fd9e806cf20 2024-03-06T07:25:05.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9e5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e80cf850 0x7fd9e80b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9e5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e80cf850 0x7fd9e80b6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54616/0 (socket says 172.21.15.59:54616) 2024-03-06T07:25:05.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9e5d9b640 1 -- 172.21.15.59:0/3567368696 learned_addr learned my addr 172.21.15.59:0/3567368696 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:05.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9e659c640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e806cf20 0x7fd9e80c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9e6d9d640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e80d00c0 0x7fd9e80b6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9e5d9b640 1 -- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e806cf20 msgr2=0x7fd9e80c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9e5d9b640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e806cf20 0x7fd9e80c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9e5d9b640 1 -- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e80d00c0 msgr2=0x7fd9e80b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9e5d9b640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e80d00c0 0x7fd9e80b6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9e5d9b640 1 -- 172.21.15.59:0/3567368696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e80b0160 con 0x7fd9e80cf850 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9e5d9b640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e80cf850 0x7fd9e80b6800 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c800 tx=0x7fd9dc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.295+0000 7fd9d6ffd640 1 -- 172.21.15.59:0/3567368696 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc018070 con 0x7fd9e80cf850 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.296+0000 7fd9d6ffd640 1 -- 172.21.15.59:0/3567368696 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc0072d0 con 0x7fd9e80cf850 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.296+0000 7fd9d6ffd640 1 -- 172.21.15.59:0/3567368696 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc0047e0 con 0x7fd9e80cf850 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.296+0000 7fd9eca81640 1 -- 172.21.15.59:0/3567368696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e80b5380 con 0x7fd9e80cf850 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.296+0000 7fd9eca81640 1 -- 172.21.15.59:0/3567368696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e80af3a0 con 0x7fd9e80cf850 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.296+0000 7fd9eca81640 1 -- 172.21.15.59:0/3567368696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e804f330 con 0x7fd9e80cf850 2024-03-06T07:25:05.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.297+0000 7fd9d6ffd640 1 -- 172.21.15.59:0/3567368696 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9dc01b070 con 0x7fd9e80cf850 2024-03-06T07:25:05.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.297+0000 7fd9d6ffd640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9c806b4d0 0x7fd9c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.298+0000 7fd9d6ffd640 1 -- 172.21.15.59:0/3567368696 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9dc089650 con 0x7fd9e80cf850 2024-03-06T07:25:05.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.298+0000 7fd9e659c640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9c806b4d0 0x7fd9c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.298+0000 7fd9e659c640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9c806b4d0 0x7fd9c806d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8035000 tx=0x7fd9d8030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.302+0000 7fd9d6ffd640 1 -- 172.21.15.59:0/3567368696 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9dc057f10 con 0x7fd9e80cf850 2024-03-06T07:25:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.487+0000 7fd9eca81640 1 -- 172.21.15.59:0/3567368696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9e80b3bd0 con 0x7fd9e80cf850 2024-03-06T07:25:05.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.488+0000 7fd9d6ffd640 1 -- 172.21.15.59:0/3567368696 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16669 (secure 0 0 0) 0x7fd9dc05bfb0 con 0x7fd9e80cf850 2024-03-06T07:25:05.491 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:05.491 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":69,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:04.517055+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:02.779485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:03.482047+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.490+0000 7fd9eca81640 1 -- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9c806b4d0 msgr2=0x7fd9c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.490+0000 7fd9eca81640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9c806b4d0 0x7fd9c806d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8035000 tx=0x7fd9d8030000 comp rx=0 tx=0).stop 2024-03-06T07:25:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.490+0000 7fd9eca81640 1 -- 172.21.15.59:0/3567368696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e80cf850 msgr2=0x7fd9e80b6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.490+0000 7fd9eca81640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e80cf850 0x7fd9e80b6800 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c800 tx=0x7fd9dc00ccc0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.490+0000 7fd9eca81640 1 -- 172.21.15.59:0/3567368696 shutdown_connections 2024-03-06T07:25:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.490+0000 7fd9eca81640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd9c806b4d0 0x7fd9c806d990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.490+0000 7fd9eca81640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e80d00c0 0x7fd9e80b6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.490+0000 7fd9eca81640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd9e80cf850 0x7fd9e80b6800 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.490+0000 7fd9eca81640 1 --2- 172.21.15.59:0/3567368696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e806cf20 0x7fd9e80c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.490+0000 7fd9eca81640 1 -- 172.21.15.59:0/3567368696 >> 172.21.15.59:0/3567368696 conn(0x7fd9e80b35a0 msgr2=0x7fd9e8070110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.490+0000 7fd9eca81640 1 -- 172.21.15.59:0/3567368696 shutdown_connections 2024-03-06T07:25:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.491+0000 7fd9eca81640 1 -- 172.21.15.59:0/3567368696 wait complete. 2024-03-06T07:25:05.504 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:05.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.604+0000 7fbad4e1c640 1 Processor -- start 2024-03-06T07:25:05.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.604+0000 7fbad4e1c640 1 -- start start 2024-03-06T07:25:05.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbad4e1c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbad006b220 0x7fbad015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbad4e1c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbad00ce920 0x7fbad0071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbad4e1c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbad0071ec0 0x7fbad0162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbad4e1c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad0161b70 con 0x7fbad0071ec0 2024-03-06T07:25:05.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbad4e1c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad015ea30 con 0x7fbad006b220 2024-03-06T07:25:05.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbad4e1c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad015ed10 con 0x7fbad00ce920 2024-03-06T07:25:05.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbaced9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbad0071ec0 0x7fbad0162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbaced9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbad0071ec0 0x7fbad0162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54624/0 (socket says 172.21.15.59:54624) 2024-03-06T07:25:05.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbaced9d640 1 -- 172.21.15.59:0/4165711337 learned_addr learned my addr 172.21.15.59:0/4165711337 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:05.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbace59c640 1 --2- 172.21.15.59:0/4165711337 >> v2:172.21.15.164:3300/0 conn(0x7fbad006b220 0x7fbad015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbaced9d640 1 -- 172.21.15.59:0/4165711337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbad00ce920 msgr2=0x7fbad0071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:05.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbaced9d640 1 --2- 172.21.15.59:0/4165711337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbad00ce920 0x7fbad0071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbaced9d640 1 -- 172.21.15.59:0/4165711337 >> v2:172.21.15.164:3300/0 conn(0x7fbad006b220 msgr2=0x7fbad015e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:05.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbaced9d640 1 --2- 172.21.15.59:0/4165711337 >> v2:172.21.15.164:3300/0 conn(0x7fbad006b220 0x7fbad015e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.605+0000 7fbaced9d640 1 -- 172.21.15.59:0/4165711337 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad015eff0 con 0x7fbad0071ec0 2024-03-06T07:25:05.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.606+0000 7fbaced9d640 1 --2- 172.21.15.59:0/4165711337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbad0071ec0 0x7fbad0162000 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fbad00d0980 tx=0x7fbac802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cafc69d7d7238124 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.606+0000 7fbacdd9b640 1 -- 172.21.15.59:0/4165711337 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac802ca70 con 0x7fbad0071ec0 2024-03-06T07:25:05.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.606+0000 7fbacdd9b640 1 -- 172.21.15.59:0/4165711337 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac802cc10 con 0x7fbad0071ec0 2024-03-06T07:25:05.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.606+0000 7fbacdd9b640 1 -- 172.21.15.59:0/4165711337 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac80343f0 con 0x7fbad0071ec0 2024-03-06T07:25:05.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.606+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4165711337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbad0071ec0 msgr2=0x7fbad0162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.606+0000 7fbad4e1c640 1 --2- 172.21.15.59:0/4165711337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbad0071ec0 0x7fbad0162000 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fbad00d0980 tx=0x7fbac802bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.607+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4165711337 shutdown_connections 2024-03-06T07:25:05.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.607+0000 7fbad4e1c640 1 --2- 172.21.15.59:0/4165711337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbad0071ec0 0x7fbad0162000 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.607+0000 7fbad4e1c640 1 --2- 172.21.15.59:0/4165711337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbad00ce920 0x7fbad0071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.607+0000 7fbad4e1c640 1 --2- 172.21.15.59:0/4165711337 >> v2:172.21.15.164:3300/0 conn(0x7fbad006b220 0x7fbad015e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.607+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4165711337 >> 172.21.15.59:0/4165711337 conn(0x7fbad006c0c0 msgr2=0x7fbad006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.607+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4165711337 shutdown_connections 2024-03-06T07:25:05.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.607+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4165711337 wait complete. 2024-03-06T07:25:05.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.608+0000 7fbad4e1c640 1 Processor -- start 2024-03-06T07:25:05.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.608+0000 7fbad4e1c640 1 -- start start 2024-03-06T07:25:05.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbad4e1c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbad006b220 0x7fbad014e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbad4e1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbad0071ec0 0x7fbad00c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbad4e1c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbad00ce920 0x7fbad00c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbad4e1c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad00ce380 con 0x7fbad0071ec0 2024-03-06T07:25:05.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbad4e1c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad00ce200 con 0x7fbad006b220 2024-03-06T07:25:05.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbad4e1c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad006b600 con 0x7fbad00ce920 2024-03-06T07:25:05.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbad0071ec0 0x7fbad00c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbad0071ec0 0x7fbad00c7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54630/0 (socket says 172.21.15.59:54630) 2024-03-06T07:25:05.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac7fff640 1 -- 172.21.15.59:0/4076189770 learned_addr learned my addr 172.21.15.59:0/4076189770 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbace59c640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbad006b220 0x7fbad014e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbaced9d640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbad00ce920 0x7fbad00c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac7fff640 1 -- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbad00ce920 msgr2=0x7fbad00c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac7fff640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbad00ce920 0x7fbad00c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac7fff640 1 -- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbad006b220 msgr2=0x7fbad014e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac7fff640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbad006b220 0x7fbad014e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac7fff640 1 -- 172.21.15.59:0/4076189770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad00c64b0 con 0x7fbad0071ec0 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac7fff640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbad0071ec0 0x7fbad00c7550 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbab800b410 tx=0x7fbab800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac6ffd640 1 -- 172.21.15.59:0/4076189770 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8015020 con 0x7fbad0071ec0 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac6ffd640 1 -- 172.21.15.59:0/4076189770 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab8011850 con 0x7fbad0071ec0 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4076189770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad00c6790 con 0x7fbad0071ec0 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbac6ffd640 1 -- 172.21.15.59:0/4076189770 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8011c40 con 0x7fbad0071ec0 2024-03-06T07:25:05.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.609+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4076189770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad00c7030 con 0x7fbad0071ec0 2024-03-06T07:25:05.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.610+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4076189770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad00c7230 con 0x7fbad0071ec0 2024-03-06T07:25:05.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.611+0000 7fbac6ffd640 1 -- 172.21.15.59:0/4076189770 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbab8018020 con 0x7fbad0071ec0 2024-03-06T07:25:05.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.611+0000 7fbac6ffd640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbab006b4d0 0x7fbab006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.611+0000 7fbac6ffd640 1 -- 172.21.15.59:0/4076189770 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbab808a010 con 0x7fbad0071ec0 2024-03-06T07:25:05.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.611+0000 7fbace59c640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbab006b4d0 0x7fbab006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.612+0000 7fbace59c640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbab006b4d0 0x7fbab006d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbad00b1330 tx=0x7fbac0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.615+0000 7fbac6ffd640 1 -- 172.21.15.59:0/4076189770 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbad00c7230 con 0x7fbad0071ec0 2024-03-06T07:25:05.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.796+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4076189770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbad004f330 con 0x7fbad0071ec0 2024-03-06T07:25:05.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.798+0000 7fbac6ffd640 1 -- 172.21.15.59:0/4076189770 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16669 (secure 0 0 0) 0x7fbab801f020 con 0x7fbad0071ec0 2024-03-06T07:25:05.800 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:05.800 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":69,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:04.517055+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:02.779485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:03.482047+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:05.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.799+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbab006b4d0 msgr2=0x7fbab006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.799+0000 7fbad4e1c640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbab006b4d0 0x7fbab006d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbad00b1330 tx=0x7fbac0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.799+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4076189770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbad0071ec0 msgr2=0x7fbad00c7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.799+0000 7fbad4e1c640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbad0071ec0 0x7fbad00c7550 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbab800b410 tx=0x7fbab800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.799+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4076189770 shutdown_connections 2024-03-06T07:25:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.799+0000 7fbad4e1c640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbab006b4d0 0x7fbab006d990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.799+0000 7fbad4e1c640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbad00ce920 0x7fbad00c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.799+0000 7fbad4e1c640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbad0071ec0 0x7fbad00c7550 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.799+0000 7fbad4e1c640 1 --2- 172.21.15.59:0/4076189770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbad006b220 0x7fbad014e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.800+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4076189770 >> 172.21.15.59:0/4076189770 conn(0x7fbad006c0c0 msgr2=0x7fbad00b42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.800+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4076189770 shutdown_connections 2024-03-06T07:25:05.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.800+0000 7fbad4e1c640 1 -- 172.21.15.59:0/4076189770 wait complete. 2024-03-06T07:25:05.813 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:25:05.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.914+0000 7fcc9bb63640 1 Processor -- start 2024-03-06T07:25:05.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.915+0000 7fcc9bb63640 1 -- start start 2024-03-06T07:25:05.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.915+0000 7fcc9bb63640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc940d4860 0x7fcc940d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.915+0000 7fcc9bb63640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcc940b0e30 0x7fcc940b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.915+0000 7fcc9bb63640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc941618b0 con 0x7fcc941614d0 2024-03-06T07:25:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.915+0000 7fcc9bb63640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc940d4cc0 con 0x7fcc940b0e30 2024-03-06T07:25:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.915+0000 7fcc9bb63640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc940cf400 con 0x7fcc940d4860 2024-03-06T07:25:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.915+0000 7fcc9a100640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcc940b0e30 0x7fcc940b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.915+0000 7fcc998ff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcc941614d0 0x7fcc940d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58746/0 (socket says 172.21.15.59:58746) 2024-03-06T07:25:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.915+0000 7fcc998ff640 1 -- 172.21.15.59:0/2049476719 learned_addr learned my addr 172.21.15.59:0/2049476719 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.916+0000 7fcc990fe640 1 --2- 172.21.15.59:0/2049476719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc940d4860 0x7fcc940d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.916+0000 7fcc9a100640 1 -- 172.21.15.59:0/2049476719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc940d4860 msgr2=0x7fcc940d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.916+0000 7fcc9a100640 1 --2- 172.21.15.59:0/2049476719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc940d4860 0x7fcc940d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.916+0000 7fcc9a100640 1 -- 172.21.15.59:0/2049476719 >> v1:172.21.15.59:6789/0 conn(0x7fcc941614d0 legacy=0x7fcc940d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.916+0000 7fcc9a100640 1 -- 172.21.15.59:0/2049476719 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc940cf6a0 con 0x7fcc940b0e30 2024-03-06T07:25:05.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.916+0000 7fcc990fe640 1 --2- 172.21.15.59:0/2049476719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc940d4860 0x7fcc940d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:05.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.916+0000 7fcc9a100640 1 --2- 172.21.15.59:0/2049476719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc940b0e30 0x7fcc940b12c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fcc84009d90 tx=0x7fcc8402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cefffee8d5744f2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.917+0000 7fcc988fd640 1 -- 172.21.15.59:0/2049476719 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84039070 con 0x7fcc940b0e30 2024-03-06T07:25:05.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.917+0000 7fcc988fd640 1 -- 172.21.15.59:0/2049476719 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8402cc60 con 0x7fcc940b0e30 2024-03-06T07:25:05.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.917+0000 7fcc988fd640 1 -- 172.21.15.59:0/2049476719 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84034660 con 0x7fcc940b0e30 2024-03-06T07:25:05.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.917+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2049476719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc940b0e30 msgr2=0x7fcc940b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.917+0000 7fcc9bb63640 1 --2- 172.21.15.59:0/2049476719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc940b0e30 0x7fcc940b12c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fcc84009d90 tx=0x7fcc8402beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.917+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2049476719 shutdown_connections 2024-03-06T07:25:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.917+0000 7fcc9bb63640 1 --2- 172.21.15.59:0/2049476719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc940b0e30 0x7fcc940b12c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.917+0000 7fcc9bb63640 1 --2- 172.21.15.59:0/2049476719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc940d4860 0x7fcc940d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.917+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2049476719 >> 172.21.15.59:0/2049476719 conn(0x7fcc940d3d40 msgr2=0x7fcc940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.917+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2049476719 shutdown_connections 2024-03-06T07:25:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.917+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2049476719 wait complete. 2024-03-06T07:25:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.918+0000 7fcc9bb63640 1 Processor -- start 2024-03-06T07:25:05.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.918+0000 7fcc9bb63640 1 -- start start 2024-03-06T07:25:05.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.919+0000 7fcc9bb63640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc940b0e30 0x7fcc940b9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.919+0000 7fcc9bb63640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc940d4860 0x7fcc940b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.919+0000 7fcc9bb63640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc941614d0 0x7fcc940b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.919+0000 7fcc9bb63640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc940b0910 con 0x7fcc940d4860 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.919+0000 7fcc9bb63640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc940cfc60 con 0x7fcc941614d0 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.919+0000 7fcc9bb63640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc940d2560 con 0x7fcc940b0e30 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.919+0000 7fcc990fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc940d4860 0x7fcc940b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.919+0000 7fcc990fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc940d4860 0x7fcc940b6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54654/0 (socket says 172.21.15.59:54654) 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.919+0000 7fcc990fe640 1 -- 172.21.15.59:0/2747825662 learned_addr learned my addr 172.21.15.59:0/2747825662 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.919+0000 7fcc998ff640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc940b0e30 0x7fcc940b9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.919+0000 7fcc9a100640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc941614d0 0x7fcc940b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.920+0000 7fcc990fe640 1 -- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc940b0e30 msgr2=0x7fcc940b9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.920+0000 7fcc990fe640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc940b0e30 0x7fcc940b9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.920+0000 7fcc990fe640 1 -- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc941614d0 msgr2=0x7fcc940b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.920+0000 7fcc990fe640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc941614d0 0x7fcc940b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.920+0000 7fcc990fe640 1 -- 172.21.15.59:0/2747825662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc940b3ab0 con 0x7fcc940d4860 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.920+0000 7fcc990fe640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc940d4860 0x7fcc940b6930 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fcc8800c7f0 tx=0x7fcc8800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.920+0000 7fcc927fc640 1 -- 172.21.15.59:0/2747825662 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc88015070 con 0x7fcc940d4860 2024-03-06T07:25:05.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.920+0000 7fcc927fc640 1 -- 172.21.15.59:0/2747825662 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc88007a30 con 0x7fcc940d4860 2024-03-06T07:25:05.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.920+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2747825662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc940b3d90 con 0x7fcc940d4860 2024-03-06T07:25:05.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.920+0000 7fcc927fc640 1 -- 172.21.15.59:0/2747825662 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc88007e00 con 0x7fcc940d4860 2024-03-06T07:25:05.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.920+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2747825662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc940b4350 con 0x7fcc940d4860 2024-03-06T07:25:05.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.921+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2747825662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc94068ff0 con 0x7fcc940d4860 2024-03-06T07:25:05.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.921+0000 7fcc927fc640 1 -- 172.21.15.59:0/2747825662 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc88018070 con 0x7fcc940d4860 2024-03-06T07:25:05.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.922+0000 7fcc927fc640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:05.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.922+0000 7fcc927fc640 1 -- 172.21.15.59:0/2747825662 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc8808a1b0 con 0x7fcc940d4860 2024-03-06T07:25:05.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.922+0000 7fcc998ff640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:05.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.922+0000 7fcc998ff640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc8006b4d0 0x7fcc8006d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c000f60 tx=0x7fcc8c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:05.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:05.926+0000 7fcc927fc640 1 -- 172.21.15.59:0/2747825662 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc88059030 con 0x7fcc940d4860 2024-03-06T07:25:06.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.144+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2747825662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fcc940b24c0 con 0x7fcc940d4860 2024-03-06T07:25:06.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.165+0000 7fcc927fc640 1 -- 172.21.15.59:0/2747825662 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcc88058930 con 0x7fcc940d4860 2024-03-06T07:25:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc8006b4d0 msgr2=0x7fcc8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc8006b4d0 0x7fcc8006d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c000f60 tx=0x7fcc8c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2747825662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc940d4860 msgr2=0x7fcc940b6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc940d4860 0x7fcc940b6930 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fcc8800c7f0 tx=0x7fcc8800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2747825662 shutdown_connections 2024-03-06T07:25:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc8006b4d0 0x7fcc8006d990 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc941614d0 0x7fcc940b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc940d4860 0x7fcc940b6930 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 --2- 172.21.15.59:0/2747825662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc940b0e30 0x7fcc940b9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2747825662 >> 172.21.15.59:0/2747825662 conn(0x7fcc940d3d40 msgr2=0x7fcc940b1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2747825662 shutdown_connections 2024-03-06T07:25:06.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.167+0000 7fcc9bb63640 1 -- 172.21.15.59:0/2747825662 wait complete. 2024-03-06T07:25:06.180 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:06.180 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:06.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.282+0000 7f980c3f7640 1 Processor -- start 2024-03-06T07:25:06.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.282+0000 7f980c3f7640 1 -- start start 2024-03-06T07:25:06.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.283+0000 7f980c3f7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f98040d4860 0x7f98040d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.283+0000 7f980c3f7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f98040b0e30 0x7f98040b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.283+0000 7f980c3f7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98041618b0 con 0x7f98041614d0 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.283+0000 7f980c3f7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98040d4cf0 con 0x7f98040d4860 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.283+0000 7f980c3f7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98040cf400 con 0x7f98040b0e30 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.283+0000 7f9809992640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f98040d4860 0x7f98040d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.283+0000 7f980a193640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f98041614d0 0x7f98040d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58752/0 (socket says 172.21.15.59:58752) 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.283+0000 7f980a193640 1 -- 172.21.15.59:0/400160949 learned_addr learned my addr 172.21.15.59:0/400160949 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.283+0000 7f980a994640 1 --2- 172.21.15.59:0/400160949 >> v2:172.21.15.164:3300/0 conn(0x7f98040b0e30 0x7f98040b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.284+0000 7f9809992640 1 -- 172.21.15.59:0/400160949 >> v2:172.21.15.164:3300/0 conn(0x7f98040b0e30 msgr2=0x7f98040b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.284+0000 7f9809992640 1 --2- 172.21.15.59:0/400160949 >> v2:172.21.15.164:3300/0 conn(0x7f98040b0e30 0x7f98040b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.284+0000 7f9809992640 1 -- 172.21.15.59:0/400160949 >> v1:172.21.15.59:6789/0 conn(0x7f98041614d0 legacy=0x7f98040d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.284+0000 7f9809191640 1 -- 172.21.15.59:0/400160949 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922889899 0 0) 0x7f98041618b0 con 0x7f98041614d0 2024-03-06T07:25:06.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.284+0000 7f9809992640 1 -- 172.21.15.59:0/400160949 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98040cf6a0 con 0x7f98040d4860 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.284+0000 7f9809992640 1 --2- 172.21.15.59:0/400160949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98040d4860 0x7f98040d2020 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f98040b2a70 tx=0x7f97f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856e4dc283af5622 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.285+0000 7f9809191640 1 -- 172.21.15.59:0/400160949 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4035020 con 0x7f98040d4860 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.285+0000 7f9809191640 1 -- 172.21.15.59:0/400160949 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f4031830 con 0x7f98040d4860 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.285+0000 7f9809191640 1 -- 172.21.15.59:0/400160949 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f4031c00 con 0x7f98040d4860 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.285+0000 7f980c3f7640 1 -- 172.21.15.59:0/400160949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98040d4860 msgr2=0x7f98040d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.285+0000 7f980c3f7640 1 --2- 172.21.15.59:0/400160949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98040d4860 0x7f98040d2020 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f98040b2a70 tx=0x7f97f402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.285+0000 7f980c3f7640 1 -- 172.21.15.59:0/400160949 shutdown_connections 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.285+0000 7f980c3f7640 1 --2- 172.21.15.59:0/400160949 >> v2:172.21.15.164:3300/0 conn(0x7f98040b0e30 0x7f98040b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.285+0000 7f980c3f7640 1 --2- 172.21.15.59:0/400160949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98040d4860 0x7f98040d2020 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.285+0000 7f980c3f7640 1 -- 172.21.15.59:0/400160949 >> 172.21.15.59:0/400160949 conn(0x7f98040d3d40 msgr2=0x7f98040d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.285+0000 7f980c3f7640 1 -- 172.21.15.59:0/400160949 shutdown_connections 2024-03-06T07:25:06.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.285+0000 7f980c3f7640 1 -- 172.21.15.59:0/400160949 wait complete. 2024-03-06T07:25:06.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.286+0000 7f980c3f7640 1 Processor -- start 2024-03-06T07:25:06.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.286+0000 7f980c3f7640 1 -- start start 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f980c3f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98040b0e30 0x7f98040b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f980c3f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98040d4860 0x7f980415dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f980c3f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98041614d0 0x7f980415b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f980c3f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98040b0910 con 0x7f98041614d0 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f980c3f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98040cfc60 con 0x7f98040d4860 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f980c3f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98040d2560 con 0x7f98040b0e30 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f980a994640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98041614d0 0x7f980415b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f980a994640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98041614d0 0x7f980415b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54672/0 (socket says 172.21.15.59:54672) 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f980a994640 1 -- 172.21.15.59:0/2144407086 learned_addr learned my addr 172.21.15.59:0/2144407086 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f980a193640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98040b0e30 0x7f98040b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f9809992640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98040d4860 0x7f980415dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.287+0000 7f980a994640 1 -- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98040b0e30 msgr2=0x7f98040b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.288+0000 7f980a994640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98040b0e30 0x7f98040b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.288+0000 7f980a994640 1 -- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98040d4860 msgr2=0x7f980415dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.288+0000 7f980a994640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98040d4860 0x7f980415dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.288+0000 7f980a994640 1 -- 172.21.15.59:0/2144407086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98040b5dd0 con 0x7f98041614d0 2024-03-06T07:25:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.288+0000 7f980a994640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98041614d0 0x7f980415b7c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00efc0 tx=0x7f97fc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.288+0000 7f97faffd640 1 -- 172.21.15.59:0/2144407086 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc018020 con 0x7f98041614d0 2024-03-06T07:25:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.288+0000 7f97faffd640 1 -- 172.21.15.59:0/2144407086 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc014480 con 0x7f98041614d0 2024-03-06T07:25:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.288+0000 7f980c3f7640 1 -- 172.21.15.59:0/2144407086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98040b60b0 con 0x7f98041614d0 2024-03-06T07:25:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.288+0000 7f97faffd640 1 -- 172.21.15.59:0/2144407086 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc014870 con 0x7f98041614d0 2024-03-06T07:25:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.288+0000 7f980c3f7640 1 -- 172.21.15.59:0/2144407086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98040cafb0 con 0x7f98041614d0 2024-03-06T07:25:06.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.289+0000 7f980c3f7640 1 -- 172.21.15.59:0/2144407086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9804068ff0 con 0x7f98041614d0 2024-03-06T07:25:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.289+0000 7f97faffd640 1 -- 172.21.15.59:0/2144407086 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97fc005d10 con 0x7f98041614d0 2024-03-06T07:25:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.290+0000 7f97faffd640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97e806b4d0 0x7f97e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.290+0000 7f97faffd640 1 -- 172.21.15.59:0/2144407086 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97fc08ab40 con 0x7f98041614d0 2024-03-06T07:25:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.290+0000 7f980a193640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97e806b4d0 0x7f97e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:06.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.290+0000 7f980a193640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97e806b4d0 0x7f97e806d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9800000f60 tx=0x7f9800009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:06.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.294+0000 7f97faffd640 1 -- 172.21.15.59:0/2144407086 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97fc022070 con 0x7f98041614d0 2024-03-06T07:25:06.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:06.516+0000 7f980c3f7640 1 -- 172.21.15.59:0/2144407086 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f98040ce9e0 con 0x7f98041614d0 2024-03-06T07:25:07.127 INFO:tasks.ceph.mds.d.smithi164.stderr:2024-03-06T07:25:07.125+0000 7fb23003b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:07.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.166+0000 7f97faffd640 1 -- 172.21.15.59:0/2144407086 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7f97fc010040 con 0x7f98041614d0 2024-03-06T07:25:07.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 -- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97e806b4d0 msgr2=0x7f97e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97e806b4d0 0x7f97e806d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9800000f60 tx=0x7f9800009040 comp rx=0 tx=0).stop 2024-03-06T07:25:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 -- 172.21.15.59:0/2144407086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98041614d0 msgr2=0x7f980415b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98041614d0 0x7f980415b7c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f97fc00efc0 tx=0x7f97fc00c490 comp rx=0 tx=0).stop 2024-03-06T07:25:07.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 -- 172.21.15.59:0/2144407086 shutdown_connections 2024-03-06T07:25:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97e806b4d0 0x7f97e806d990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98041614d0 0x7f980415b7c0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98040d4860 0x7f980415dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 --2- 172.21.15.59:0/2144407086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98040b0e30 0x7f98040b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 -- 172.21.15.59:0/2144407086 >> 172.21.15.59:0/2144407086 conn(0x7f98040d3d40 msgr2=0x7f980406c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 -- 172.21.15.59:0/2144407086 shutdown_connections 2024-03-06T07:25:07.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.169+0000 7f980c3f7640 1 -- 172.21.15.59:0/2144407086 wait complete. 2024-03-06T07:25:07.185 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:07.185 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:07.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.291+0000 7fd199db5640 1 Processor -- start 2024-03-06T07:25:07.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.292+0000 7fd199db5640 1 -- start start 2024-03-06T07:25:07.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.292+0000 7fd199db5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd194072140 0x7fd19406db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.292+0000 7fd199db5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1940adf70 con 0x7fd1940cf790 2024-03-06T07:25:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.292+0000 7fd199db5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1940ae240 con 0x7fd194071ce0 2024-03-06T07:25:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.292+0000 7fd199db5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd19406cf70 con 0x7fd194072140 2024-03-06T07:25:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.292+0000 7fd193fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd194072140 0x7fd19406db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.292+0000 7fd1937fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd1940cf790 0x7fd1940ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58754/0 (socket says 172.21.15.59:58754) 2024-03-06T07:25:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.292+0000 7fd193fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd194072140 0x7fd19406db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54688/0 (socket says 172.21.15.59:54688) 2024-03-06T07:25:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.292+0000 7fd1937fe640 1 -- 172.21.15.59:0/1038016210 learned_addr learned my addr 172.21.15.59:0/1038016210 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.293+0000 7fd1927fc640 1 -- 172.21.15.59:0/1038016210 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271648012 0 0) 0x7fd1940ae240 con 0x7fd194071ce0 2024-03-06T07:25:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.293+0000 7fd193fff640 1 -- 172.21.15.59:0/1038016210 >> v1:172.21.15.164:6789/0 conn(0x7fd194071ce0 legacy=0x7fd19406d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.293+0000 7fd193fff640 1 -- 172.21.15.59:0/1038016210 >> v1:172.21.15.59:6789/0 conn(0x7fd1940cf790 legacy=0x7fd1940ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.293+0000 7fd193fff640 1 -- 172.21.15.59:0/1038016210 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1940ae520 con 0x7fd194072140 2024-03-06T07:25:07.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.294+0000 7fd193fff640 1 --2- 172.21.15.59:0/1038016210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd194072140 0x7fd19406db10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009d90 tx=0x7fd18c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19701097b2e2efaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.294+0000 7fd1927fc640 1 -- 172.21.15.59:0/1038016210 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c039070 con 0x7fd194072140 2024-03-06T07:25:07.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.294+0000 7fd1927fc640 1 -- 172.21.15.59:0/1038016210 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18c02cab0 con 0x7fd194072140 2024-03-06T07:25:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.294+0000 7fd1927fc640 1 -- 172.21.15.59:0/1038016210 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c034440 con 0x7fd194072140 2024-03-06T07:25:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.294+0000 7fd199db5640 1 -- 172.21.15.59:0/1038016210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd194072140 msgr2=0x7fd19406db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.294+0000 7fd199db5640 1 --2- 172.21.15.59:0/1038016210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd194072140 0x7fd19406db10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009d90 tx=0x7fd18c02bd80 comp rx=0 tx=0).stop 2024-03-06T07:25:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.294+0000 7fd199db5640 1 -- 172.21.15.59:0/1038016210 shutdown_connections 2024-03-06T07:25:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.294+0000 7fd199db5640 1 --2- 172.21.15.59:0/1038016210 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd194072140 0x7fd19406db10 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.294+0000 7fd199db5640 1 -- 172.21.15.59:0/1038016210 >> 172.21.15.59:0/1038016210 conn(0x7fd1940b1a90 msgr2=0x7fd1940b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.294+0000 7fd199db5640 1 -- 172.21.15.59:0/1038016210 shutdown_connections 2024-03-06T07:25:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.294+0000 7fd199db5640 1 -- 172.21.15.59:0/1038016210 wait complete. 2024-03-06T07:25:07.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.295+0000 7fd199db5640 1 Processor -- start 2024-03-06T07:25:07.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.296+0000 7fd199db5640 1 -- start start 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.296+0000 7fd199db5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd194071ce0 0x7fd1941e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.296+0000 7fd199db5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd194072140 0x7fd19413c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.296+0000 7fd199db5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1940cf790 0x7fd1941570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.296+0000 7fd199db5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1940725d0 con 0x7fd1940cf790 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.296+0000 7fd199db5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1940aeae0 con 0x7fd194071ce0 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.296+0000 7fd199db5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19415e580 con 0x7fd194072140 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd193fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1940cf790 0x7fd1941570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd193fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1940cf790 0x7fd1941570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54694/0 (socket says 172.21.15.59:54694) 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd1937fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd194071ce0 0x7fd1941e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd193fff640 1 -- 172.21.15.59:0/2672895699 learned_addr learned my addr 172.21.15.59:0/2672895699 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd192ffd640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd194072140 0x7fd19413c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd193fff640 1 -- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd194072140 msgr2=0x7fd19413c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd193fff640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd194072140 0x7fd19413c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd193fff640 1 -- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd194071ce0 msgr2=0x7fd1941e81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd193fff640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd194071ce0 0x7fd1941e81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd193fff640 1 -- 172.21.15.59:0/2672895699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1941526f0 con 0x7fd1940cf790 2024-03-06T07:25:07.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd193fff640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1940cf790 0x7fd1941570e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd18c037040 tx=0x7fd18c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd17bfff640 1 -- 172.21.15.59:0/2672895699 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c044020 con 0x7fd1940cf790 2024-03-06T07:25:07.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd17bfff640 1 -- 172.21.15.59:0/2672895699 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18c03a040 con 0x7fd1940cf790 2024-03-06T07:25:07.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd199db5640 1 -- 172.21.15.59:0/2672895699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1941529d0 con 0x7fd1940cf790 2024-03-06T07:25:07.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd17bfff640 1 -- 172.21.15.59:0/2672895699 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c039050 con 0x7fd1940cf790 2024-03-06T07:25:07.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.297+0000 7fd199db5640 1 -- 172.21.15.59:0/2672895699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd194152f90 con 0x7fd1940cf790 2024-03-06T07:25:07.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.298+0000 7fd199db5640 1 -- 172.21.15.59:0/2672895699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19404f330 con 0x7fd1940cf790 2024-03-06T07:25:07.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.299+0000 7fd17bfff640 1 -- 172.21.15.59:0/2672895699 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd18c02c930 con 0x7fd1940cf790 2024-03-06T07:25:07.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.299+0000 7fd17bfff640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd18006b4d0 0x7fd18006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.299+0000 7fd17bfff640 1 -- 172.21.15.59:0/2672895699 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd18c0ac850 con 0x7fd1940cf790 2024-03-06T07:25:07.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.299+0000 7fd1937fe640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd18006b4d0 0x7fd18006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.300+0000 7fd1937fe640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd18006b4d0 0x7fd18006d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd184000f60 tx=0x7fd184009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.304+0000 7fd17bfff640 1 -- 172.21.15.59:0/2672895699 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd18c04e060 con 0x7fd1940cf790 2024-03-06T07:25:07.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.521+0000 7fd199db5640 1 -- 172.21.15.59:0/2672895699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1940b1870 con 0x7fd1940cf790 2024-03-06T07:25:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.521+0000 7fd17bfff640 1 -- 172.21.15.59:0/2672895699 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fd18c07b070 con 0x7fd1940cf790 2024-03-06T07:25:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:07.523 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:04.517350+0000","modified":"2024-03-06T07:25:06.518231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/250576205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":250576205},{"type":"v1","addr":"172.21.15.164:6837","nonce":250576205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 35 2024-03-06T07:25:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 -- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd18006b4d0 msgr2=0x7fd18006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd18006b4d0 0x7fd18006d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd184000f60 tx=0x7fd184009040 comp rx=0 tx=0).stop 2024-03-06T07:25:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 -- 172.21.15.59:0/2672895699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1940cf790 msgr2=0x7fd1941570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1940cf790 0x7fd1941570e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd18c037040 tx=0x7fd18c004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 -- 172.21.15.59:0/2672895699 shutdown_connections 2024-03-06T07:25:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd18006b4d0 0x7fd18006d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1940cf790 0x7fd1941570e0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd194072140 0x7fd19413c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 --2- 172.21.15.59:0/2672895699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd194071ce0 0x7fd1941e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 -- 172.21.15.59:0/2672895699 >> 172.21.15.59:0/2672895699 conn(0x7fd1940b1a90 msgr2=0x7fd1941602f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 -- 172.21.15.59:0/2672895699 shutdown_connections 2024-03-06T07:25:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.523+0000 7fd199db5640 1 -- 172.21.15.59:0/2672895699 wait complete. 2024-03-06T07:25:07.536 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:07.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.635+0000 7f0ba420e640 1 Processor -- start 2024-03-06T07:25:07.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.635+0000 7f0ba420e640 1 -- start start 2024-03-06T07:25:07.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.636+0000 7f0ba420e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0b9c0cdeb0 0x7f0b9c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.636+0000 7f0ba420e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c071f40 con 0x7f0b9c1515e0 2024-03-06T07:25:07.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.636+0000 7f0ba420e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c070a70 con 0x7f0b9c0cce40 2024-03-06T07:25:07.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.636+0000 7f0ba420e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9c0cd2f0 con 0x7f0b9c0cdeb0 2024-03-06T07:25:07.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.636+0000 7f0ba27ab640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0b9c0cdeb0 0x7f0b9c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.636+0000 7f0ba1faa640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0b9c1515e0 0x7f0b9c071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58770/0 (socket says 172.21.15.59:58770) 2024-03-06T07:25:07.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.636+0000 7f0ba1faa640 1 -- 172.21.15.59:0/2652875391 learned_addr learned my addr 172.21.15.59:0/2652875391 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:07.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.636+0000 7f0ba27ab640 1 -- 172.21.15.59:0/2652875391 >> v1:172.21.15.164:6789/0 conn(0x7f0b9c0cce40 legacy=0x7f0b9c0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.636+0000 7f0ba27ab640 1 -- 172.21.15.59:0/2652875391 >> v1:172.21.15.59:6789/0 conn(0x7f0b9c1515e0 legacy=0x7f0b9c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.636+0000 7f0ba0fa8640 1 -- 172.21.15.59:0/2652875391 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843358844 0 0) 0x7f0b9c071f40 con 0x7f0b9c1515e0 2024-03-06T07:25:07.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.636+0000 7f0ba27ab640 1 -- 172.21.15.59:0/2652875391 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c070d50 con 0x7f0b9c0cdeb0 2024-03-06T07:25:07.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.637+0000 7f0ba27ab640 1 --2- 172.21.15.59:0/2652875391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b9c0cdeb0 0x7f0b9c15b430 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400aea0 tx=0x7f0b9402d050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ffd1055beaffdd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.637+0000 7f0ba0fa8640 1 -- 172.21.15.59:0/2652875391 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9402dbf0 con 0x7f0b9c0cdeb0 2024-03-06T07:25:07.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.637+0000 7f0ba0fa8640 1 -- 172.21.15.59:0/2652875391 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9402dd90 con 0x7f0b9c0cdeb0 2024-03-06T07:25:07.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.637+0000 7f0ba0fa8640 1 -- 172.21.15.59:0/2652875391 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94008660 con 0x7f0b9c0cdeb0 2024-03-06T07:25:07.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.638+0000 7f0ba420e640 1 -- 172.21.15.59:0/2652875391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b9c0cdeb0 msgr2=0x7f0b9c15b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.638+0000 7f0ba420e640 1 --2- 172.21.15.59:0/2652875391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b9c0cdeb0 0x7f0b9c15b430 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400aea0 tx=0x7f0b9402d050 comp rx=0 tx=0).stop 2024-03-06T07:25:07.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.638+0000 7f0ba420e640 1 -- 172.21.15.59:0/2652875391 shutdown_connections 2024-03-06T07:25:07.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.638+0000 7f0ba420e640 1 --2- 172.21.15.59:0/2652875391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b9c0cdeb0 0x7f0b9c15b430 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.638+0000 7f0ba420e640 1 -- 172.21.15.59:0/2652875391 >> 172.21.15.59:0/2652875391 conn(0x7f0b9c0cf8a0 msgr2=0x7f0b9c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.638+0000 7f0ba420e640 1 -- 172.21.15.59:0/2652875391 shutdown_connections 2024-03-06T07:25:07.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.638+0000 7f0ba420e640 1 -- 172.21.15.59:0/2652875391 wait complete. 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.639+0000 7f0ba420e640 1 Processor -- start 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.639+0000 7f0ba420e640 1 -- start start 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba420e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c0cce40 0x7f0b9c148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba420e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b9c0cdeb0 0x7f0b9c14d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba420e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b9c1515e0 0x7f0b9c151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba420e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c0cc3a0 con 0x7f0b9c1515e0 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba420e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c071310 con 0x7f0b9c0cdeb0 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba420e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c0cc520 con 0x7f0b9c0cce40 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba27ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b9c1515e0 0x7f0b9c151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba27ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b9c1515e0 0x7f0b9c151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54718/0 (socket says 172.21.15.59:54718) 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba27ab640 1 -- 172.21.15.59:0/1057777197 learned_addr learned my addr 172.21.15.59:0/1057777197 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:07.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba1faa640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c0cce40 0x7f0b9c148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba17a9640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b9c0cdeb0 0x7f0b9c14d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba27ab640 1 -- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c0cce40 msgr2=0x7f0b9c148170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba27ab640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c0cce40 0x7f0b9c148170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba27ab640 1 -- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b9c0cdeb0 msgr2=0x7f0b9c14d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba27ab640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b9c0cdeb0 0x7f0b9c14d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba27ab640 1 -- 172.21.15.59:0/1057777197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c146c30 con 0x7f0b9c1515e0 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba27ab640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b9c1515e0 0x7f0b9c151a40 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400aea0 tx=0x7f0b94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0b927fc640 1 -- 172.21.15.59:0/1057777197 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94044020 con 0x7f0b9c1515e0 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0b927fc640 1 -- 172.21.15.59:0/1057777197 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b94040690 con 0x7f0b9c1515e0 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba420e640 1 -- 172.21.15.59:0/1057777197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c15ee30 con 0x7f0b9c1515e0 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0b927fc640 1 -- 172.21.15.59:0/1057777197 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9400a560 con 0x7f0b9c1515e0 2024-03-06T07:25:07.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.640+0000 7f0ba420e640 1 -- 172.21.15.59:0/1057777197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c14e390 con 0x7f0b9c1515e0 2024-03-06T07:25:07.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.641+0000 7f0ba420e640 1 -- 172.21.15.59:0/1057777197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9c068840 con 0x7f0b9c1515e0 2024-03-06T07:25:07.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.642+0000 7f0b927fc640 1 -- 172.21.15.59:0/1057777197 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b94047020 con 0x7f0b9c1515e0 2024-03-06T07:25:07.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.642+0000 7f0b927fc640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b8006b4d0 0x7f0b8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.642+0000 7f0b927fc640 1 -- 172.21.15.59:0/1057777197 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b940acf30 con 0x7f0b9c1515e0 2024-03-06T07:25:07.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.642+0000 7f0ba1faa640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b8006b4d0 0x7f0b8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.643+0000 7f0ba1faa640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b8006b4d0 0x7f0b8006d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0b98007c30 tx=0x7f0b98007b80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.646+0000 7f0b927fc640 1 -- 172.21.15.59:0/1057777197 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b9404e030 con 0x7f0b9c1515e0 2024-03-06T07:25:07.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.830+0000 7f0ba420e640 1 -- 172.21.15.59:0/1057777197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b9c0b4c30 con 0x7f0b9c1515e0 2024-03-06T07:25:07.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.831+0000 7f0b927fc640 1 -- 172.21.15.59:0/1057777197 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16656 (secure 0 0 0) 0x7f0b9407b910 con 0x7f0b9c1515e0 2024-03-06T07:25:07.833 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:07.833 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":70,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:06.152587+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:02.779485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:03.482047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:07.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 -- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b8006b4d0 msgr2=0x7f0b8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b8006b4d0 0x7f0b8006d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0b98007c30 tx=0x7f0b98007b80 comp rx=0 tx=0).stop 2024-03-06T07:25:07.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 -- 172.21.15.59:0/1057777197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b9c1515e0 msgr2=0x7f0b9c151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b9c1515e0 0x7f0b9c151a40 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400aea0 tx=0x7f0b94004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:07.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 -- 172.21.15.59:0/1057777197 shutdown_connections 2024-03-06T07:25:07.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b8006b4d0 0x7f0b8006d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b9c1515e0 0x7f0b9c151a40 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b9c0cdeb0 0x7f0b9c14d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 --2- 172.21.15.59:0/1057777197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c0cce40 0x7f0b9c148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 -- 172.21.15.59:0/1057777197 >> 172.21.15.59:0/1057777197 conn(0x7f0b9c0cf8a0 msgr2=0x7f0b9c0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 -- 172.21.15.59:0/1057777197 shutdown_connections 2024-03-06T07:25:07.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.833+0000 7f0ba420e640 1 -- 172.21.15.59:0/1057777197 wait complete. 2024-03-06T07:25:07.846 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:07.846 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.945+0000 7fe316aa4640 1 Processor -- start 2024-03-06T07:25:07.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.945+0000 7fe316aa4640 1 -- start start 2024-03-06T07:25:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.946+0000 7fe316aa4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe31013dc10 0x7fe3100b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.946+0000 7fe316aa4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe31015bf30 0x7fe3100b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.946+0000 7fe316aa4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31015e850 0x7fe31015eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.946+0000 7fe316aa4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3100cf3f0 con 0x7fe31015bf30 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.946+0000 7fe316aa4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe31015b210 con 0x7fe31013dc10 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.946+0000 7fe316aa4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31015f580 con 0x7fe31015e850 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.946+0000 7fe30ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe31015bf30 0x7fe3100b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.946+0000 7fe30ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe31015bf30 0x7fe3100b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54730/0 (socket says 172.21.15.59:54730) 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.946+0000 7fe30ffff640 1 -- 172.21.15.59:0/1351425613 learned_addr learned my addr 172.21.15.59:0/1351425613 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.946+0000 7fe314840640 1 --2- 172.21.15.59:0/1351425613 >> v2:172.21.15.164:3300/0 conn(0x7fe31013dc10 0x7fe3100b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.946+0000 7fe315041640 1 --2- 172.21.15.59:0/1351425613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31015e850 0x7fe31015eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.947+0000 7fe30ffff640 1 -- 172.21.15.59:0/1351425613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31015e850 msgr2=0x7fe31015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.947+0000 7fe30ffff640 1 --2- 172.21.15.59:0/1351425613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31015e850 0x7fe31015eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.947+0000 7fe30ffff640 1 -- 172.21.15.59:0/1351425613 >> v2:172.21.15.164:3300/0 conn(0x7fe31013dc10 msgr2=0x7fe3100b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:07.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.947+0000 7fe30ffff640 1 --2- 172.21.15.59:0/1351425613 >> v2:172.21.15.164:3300/0 conn(0x7fe31013dc10 0x7fe3100b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.947+0000 7fe30ffff640 1 -- 172.21.15.59:0/1351425613 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31015b450 con 0x7fe31015bf30 2024-03-06T07:25:07.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.947+0000 7fe30ffff640 1 --2- 172.21.15.59:0/1351425613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe31015bf30 0x7fe3100b2e90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe3100731a0 tx=0x7fe2fc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b71f68a802460d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe30f7fe640 1 -- 172.21.15.59:0/1351425613 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc02e070 con 0x7fe31015bf30 2024-03-06T07:25:07.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe30f7fe640 1 -- 172.21.15.59:0/1351425613 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc037070 con 0x7fe31015bf30 2024-03-06T07:25:07.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe30f7fe640 1 -- 172.21.15.59:0/1351425613 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc031a00 con 0x7fe31015bf30 2024-03-06T07:25:07.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe316aa4640 1 -- 172.21.15.59:0/1351425613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe31015bf30 msgr2=0x7fe3100b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe316aa4640 1 --2- 172.21.15.59:0/1351425613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe31015bf30 0x7fe3100b2e90 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe3100731a0 tx=0x7fe2fc02b880 comp rx=0 tx=0).stop 2024-03-06T07:25:07.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe316aa4640 1 -- 172.21.15.59:0/1351425613 shutdown_connections 2024-03-06T07:25:07.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe316aa4640 1 --2- 172.21.15.59:0/1351425613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31015e850 0x7fe31015eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe316aa4640 1 --2- 172.21.15.59:0/1351425613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe31015bf30 0x7fe3100b2e90 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe316aa4640 1 --2- 172.21.15.59:0/1351425613 >> v2:172.21.15.164:3300/0 conn(0x7fe31013dc10 0x7fe3100b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe316aa4640 1 -- 172.21.15.59:0/1351425613 >> 172.21.15.59:0/1351425613 conn(0x7fe3100cf8a0 msgr2=0x7fe3100cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:07.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe316aa4640 1 -- 172.21.15.59:0/1351425613 shutdown_connections 2024-03-06T07:25:07.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.948+0000 7fe316aa4640 1 -- 172.21.15.59:0/1351425613 wait complete. 2024-03-06T07:25:07.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.949+0000 7fe316aa4640 1 Processor -- start 2024-03-06T07:25:07.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.949+0000 7fe316aa4640 1 -- start start 2024-03-06T07:25:07.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.950+0000 7fe316aa4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31013dc10 0x7fe31015dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.950+0000 7fe316aa4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe31015bf30 0x7fe3101408b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.950+0000 7fe316aa4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe31015e850 0x7fe310140df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.950+0000 7fe316aa4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31015f1e0 con 0x7fe31015bf30 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.950+0000 7fe316aa4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31013dff0 con 0x7fe31015e850 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.950+0000 7fe316aa4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31015f360 con 0x7fe31013dc10 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.950+0000 7fe30ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe31015bf30 0x7fe3101408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.950+0000 7fe30ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe31015bf30 0x7fe3101408b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54732/0 (socket says 172.21.15.59:54732) 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.950+0000 7fe30ffff640 1 -- 172.21.15.59:0/381032463 learned_addr learned my addr 172.21.15.59:0/381032463 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.950+0000 7fe315041640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe31015e850 0x7fe310140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe314840640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31013dc10 0x7fe31015dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe30ffff640 1 -- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31013dc10 msgr2=0x7fe31015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe30ffff640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31013dc10 0x7fe31015dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe30ffff640 1 -- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe31015e850 msgr2=0x7fe310140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe30ffff640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe31015e850 0x7fe310140df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe30ffff640 1 -- 172.21.15.59:0/381032463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31013cc80 con 0x7fe31015bf30 2024-03-06T07:25:07.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe30ffff640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe31015bf30 0x7fe3101408b0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc02b8c0 tx=0x7fe2fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe30d7fa640 1 -- 172.21.15.59:0/381032463 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc032670 con 0x7fe31015bf30 2024-03-06T07:25:07.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe30d7fa640 1 -- 172.21.15.59:0/381032463 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc032830 con 0x7fe31015bf30 2024-03-06T07:25:07.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe316aa4640 1 -- 172.21.15.59:0/381032463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3101f3bb0 con 0x7fe31015bf30 2024-03-06T07:25:07.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe30d7fa640 1 -- 172.21.15.59:0/381032463 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc032c20 con 0x7fe31015bf30 2024-03-06T07:25:07.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.951+0000 7fe316aa4640 1 -- 172.21.15.59:0/381032463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3101f4170 con 0x7fe31015bf30 2024-03-06T07:25:07.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.952+0000 7fe316aa4640 1 -- 172.21.15.59:0/381032463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31004f330 con 0x7fe31015bf30 2024-03-06T07:25:07.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.952+0000 7fe30d7fa640 1 -- 172.21.15.59:0/381032463 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2fc04c060 con 0x7fe31015bf30 2024-03-06T07:25:07.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.953+0000 7fe30d7fa640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:07.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.953+0000 7fe30d7fa640 1 -- 172.21.15.59:0/381032463 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2fc0ab9b0 con 0x7fe31015bf30 2024-03-06T07:25:07.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.953+0000 7fe314840640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:07.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.954+0000 7fe314840640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2f406b4d0 0x7fe2f406d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe304009a20 tx=0x7fe304009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:07.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:07.957+0000 7fe30d7fa640 1 -- 172.21.15.59:0/381032463 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2fc040070 con 0x7fe31015bf30 2024-03-06T07:25:08.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.134+0000 7fe316aa4640 1 -- 172.21.15.59:0/381032463 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe310144040 con 0x7fe2f406b4d0 2024-03-06T07:25:08.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.141+0000 7fe30d7fa640 1 -- 172.21.15.59:0/381032463 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252026 (secure 0 0 0) 0x7fe310144040 con 0x7fe2f406b4d0 2024-03-06T07:25:08.145 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:08.146 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.143+0000 7fe316aa4640 1 -- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2f406b4d0 msgr2=0x7fe2f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.143+0000 7fe316aa4640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2f406b4d0 0x7fe2f406d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe304009a20 tx=0x7fe304009190 comp rx=0 tx=0).stop 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.143+0000 7fe316aa4640 1 -- 172.21.15.59:0/381032463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe31015bf30 msgr2=0x7fe3101408b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.143+0000 7fe316aa4640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe31015bf30 0x7fe3101408b0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc02b8c0 tx=0x7fe2fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.143+0000 7fe316aa4640 1 -- 172.21.15.59:0/381032463 shutdown_connections 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.143+0000 7fe316aa4640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2f406b4d0 0x7fe2f406d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.143+0000 7fe316aa4640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe31015e850 0x7fe310140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.143+0000 7fe316aa4640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe31015bf30 0x7fe3101408b0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.143+0000 7fe316aa4640 1 --2- 172.21.15.59:0/381032463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe31013dc10 0x7fe31015dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.143+0000 7fe316aa4640 1 -- 172.21.15.59:0/381032463 >> 172.21.15.59:0/381032463 conn(0x7fe3100cf8a0 msgr2=0x7fe3100711b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.144+0000 7fe316aa4640 1 -- 172.21.15.59:0/381032463 shutdown_connections 2024-03-06T07:25:08.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.144+0000 7fe316aa4640 1 -- 172.21.15.59:0/381032463 wait complete. 2024-03-06T07:25:08.156 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-03-06T07:25:07.343488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257248,"kb_used_data":7784,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717472,"statfs":{"total":773094113280,"available":772830691328,"internally_reserved":0,"allocated":7970816,"data_stored":4006932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.198583"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176274+0000","last_change":"2024-03-06T07:25:05.172938+0000","last_active":"2024-03-06T07:25:05.176274+0000","last_peered":"2024-03-06T07:25:05.176274+0000","last_clean":"2024-03-06T07:25:05.176274+0000","last_became_active":"2024-03-06T07:25:05.172632+0000","last_became_peered":"2024-03-06T07:25:05.172632+0000","last_unstale":"2024-03-06T07:25:05.176274+0000","last_undegraded":"2024-03-06T07:25:05.176274+0000","last_fullsized":"2024-03-06T07:25:05.176274+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:45.258201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.185040+0000","last_change":"2024-03-06T07:25:05.177682+0000","last_active":"2024-03-06T07:25:05.185040+0000","last_peered":"2024-03-06T07:25:05.185040+0000","last_clean":"2024-03-06T07:25:05.185040+0000","last_became_active":"2024-03-06T07:25:05.177317+0000","last_became_peered":"2024-03-06T07:25:05.177317+0000","last_unstale":"2024-03-06T07:25:05.185040+0000","last_undegraded":"2024-03-06T07:25:05.185040+0000","last_fullsized":"2024-03-06T07:25:05.185040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:06.532171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156409+0000","last_change":"2024-03-06T07:25:05.168424+0000","last_active":"2024-03-06T07:25:06.156409+0000","last_peered":"2024-03-06T07:25:06.156409+0000","last_clean":"2024-03-06T07:25:06.156409+0000","last_became_active":"2024-03-06T07:25:05.167489+0000","last_became_peered":"2024-03-06T07:25:05.167489+0000","last_unstale":"2024-03-06T07:25:06.156409+0000","last_undegraded":"2024-03-06T07:25:06.156409+0000","last_fullsized":"2024-03-06T07:25:06.156409+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:14.728644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155459+0000","last_change":"2024-03-06T07:25:05.181199+0000","last_active":"2024-03-06T07:25:06.155459+0000","last_peered":"2024-03-06T07:25:06.155459+0000","last_clean":"2024-03-06T07:25:06.155459+0000","last_became_active":"2024-03-06T07:25:05.180890+0000","last_became_peered":"2024-03-06T07:25:05.180890+0000","last_unstale":"2024-03-06T07:25:06.155459+0000","last_undegraded":"2024-03-06T07:25:06.155459+0000","last_fullsized":"2024-03-06T07:25:06.155459+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:53.479454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176253+0000","last_change":"2024-03-06T07:25:05.171737+0000","last_active":"2024-03-06T07:25:05.176253+0000","last_peered":"2024-03-06T07:25:05.176253+0000","last_clean":"2024-03-06T07:25:05.176253+0000","last_became_active":"2024-03-06T07:25:05.171458+0000","last_became_peered":"2024-03-06T07:25:05.171458+0000","last_unstale":"2024-03-06T07:25:05.176253+0000","last_undegraded":"2024-03-06T07:25:05.176253+0000","last_fullsized":"2024-03-06T07:25:05.176253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:17:06.862819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.178281+0000","last_change":"2024-03-06T07:25:05.174102+0000","last_active":"2024-03-06T07:25:05.178281+0000","last_peered":"2024-03-06T07:25:05.178281+0000","last_clean":"2024-03-06T07:25:05.178281+0000","last_became_active":"2024-03-06T07:25:05.173840+0000","last_became_peered":"2024-03-06T07:25:05.173840+0000","last_unstale":"2024-03-06T07:25:05.178281+0000","last_undegraded":"2024-03-06T07:25:05.178281+0000","last_fullsized":"2024-03-06T07:25:05.178281+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:32.423709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156897+0000","last_change":"2024-03-06T07:25:05.171597+0000","last_active":"2024-03-06T07:25:06.156897+0000","last_peered":"2024-03-06T07:25:06.156897+0000","last_clean":"2024-03-06T07:25:06.156897+0000","last_became_active":"2024-03-06T07:25:05.171326+0000","last_became_peered":"2024-03-06T07:25:05.171326+0000","last_unstale":"2024-03-06T07:25:06.156897+0000","last_undegraded":"2024-03-06T07:25:06.156897+0000","last_fullsized":"2024-03-06T07:25:06.156897+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:04.740636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187632+0000","last_change":"2024-03-06T07:25:05.182681+0000","last_active":"2024-03-06T07:25:05.187632+0000","last_peered":"2024-03-06T07:25:05.187632+0000","last_clean":"2024-03-06T07:25:05.187632+0000","last_became_active":"2024-03-06T07:25:05.182392+0000","last_became_peered":"2024-03-06T07:25:05.182392+0000","last_unstale":"2024-03-06T07:25:05.187632+0000","last_undegraded":"2024-03-06T07:25:05.187632+0000","last_fullsized":"2024-03-06T07:25:05.187632+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:41.996337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176159+0000","last_change":"2024-03-06T07:25:05.168655+0000","last_active":"2024-03-06T07:25:05.176159+0000","last_peered":"2024-03-06T07:25:05.176159+0000","last_clean":"2024-03-06T07:25:05.176159+0000","last_became_active":"2024-03-06T07:25:05.168154+0000","last_became_peered":"2024-03-06T07:25:05.168154+0000","last_unstale":"2024-03-06T07:25:05.176159+0000","last_undegraded":"2024-03-06T07:25:05.176159+0000","last_fullsized":"2024-03-06T07:25:05.176159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:07.277601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179991+0000","last_change":"2024-03-06T07:25:05.174638+0000","last_active":"2024-03-06T07:25:05.179991+0000","last_peered":"2024-03-06T07:25:05.179991+0000","last_clean":"2024-03-06T07:25:05.179991+0000","last_became_active":"2024-03-06T07:25:05.174336+0000","last_became_peered":"2024-03-06T07:25:05.174336+0000","last_unstale":"2024-03-06T07:25:05.179991+0000","last_undegraded":"2024-03-06T07:25:05.179991+0000","last_fullsized":"2024-03-06T07:25:05.179991+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:42.027393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.173919+0000","last_change":"2024-03-06T07:25:05.170458+0000","last_active":"2024-03-06T07:25:05.173919+0000","last_peered":"2024-03-06T07:25:05.173919+0000","last_clean":"2024-03-06T07:25:05.173919+0000","last_became_active":"2024-03-06T07:25:05.170193+0000","last_became_peered":"2024-03-06T07:25:05.170193+0000","last_unstale":"2024-03-06T07:25:05.173919+0000","last_undegraded":"2024-03-06T07:25:05.173919+0000","last_fullsized":"2024-03-06T07:25:05.173919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:55.867315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155653+0000","last_change":"2024-03-06T07:25:05.176864+0000","last_active":"2024-03-06T07:25:06.155653+0000","last_peered":"2024-03-06T07:25:06.155653+0000","last_clean":"2024-03-06T07:25:06.155653+0000","last_became_active":"2024-03-06T07:25:05.176577+0000","last_became_peered":"2024-03-06T07:25:05.176577+0000","last_unstale":"2024-03-06T07:25:06.155653+0000","last_undegraded":"2024-03-06T07:25:06.155653+0000","last_fullsized":"2024-03-06T07:25:06.155653+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:16.371492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172765+0000","last_change":"2024-03-06T07:25:05.175851+0000","last_active":"2024-03-06T07:25:06.172765+0000","last_peered":"2024-03-06T07:25:06.172765+0000","last_clean":"2024-03-06T07:25:06.172765+0000","last_became_active":"2024-03-06T07:25:05.175502+0000","last_became_peered":"2024-03-06T07:25:05.175502+0000","last_unstale":"2024-03-06T07:25:06.172765+0000","last_undegraded":"2024-03-06T07:25:06.172765+0000","last_fullsized":"2024-03-06T07:25:06.172765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:52.667362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.169510+0000","last_change":"2024-03-06T07:25:05.164708+0000","last_active":"2024-03-06T07:25:05.169510+0000","last_peered":"2024-03-06T07:25:05.169510+0000","last_clean":"2024-03-06T07:25:05.169510+0000","last_became_active":"2024-03-06T07:25:05.164375+0000","last_became_peered":"2024-03-06T07:25:05.164375+0000","last_unstale":"2024-03-06T07:25:05.169510+0000","last_undegraded":"2024-03-06T07:25:05.169510+0000","last_fullsized":"2024-03-06T07:25:05.169510+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:10.232525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172604+0000","last_change":"2024-03-06T07:25:05.170080+0000","last_active":"2024-03-06T07:25:06.172604+0000","last_peered":"2024-03-06T07:25:06.172604+0000","last_clean":"2024-03-06T07:25:06.172604+0000","last_became_active":"2024-03-06T07:25:05.169816+0000","last_became_peered":"2024-03-06T07:25:05.169816+0000","last_unstale":"2024-03-06T07:25:06.172604+0000","last_undegraded":"2024-03-06T07:25:06.172604+0000","last_fullsized":"2024-03-06T07:25:06.172604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:34.634891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.181163+0000","last_change":"2024-03-06T07:25:05.176069+0000","last_active":"2024-03-06T07:25:05.181163+0000","last_peered":"2024-03-06T07:25:05.181163+0000","last_clean":"2024-03-06T07:25:05.181163+0000","last_became_active":"2024-03-06T07:25:05.175585+0000","last_became_peered":"2024-03-06T07:25:05.175585+0000","last_unstale":"2024-03-06T07:25:05.181163+0000","last_undegraded":"2024-03-06T07:25:05.181163+0000","last_fullsized":"2024-03-06T07:25:05.181163+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:53.479454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176135+0000","last_change":"2024-03-06T07:25:05.168538+0000","last_active":"2024-03-06T07:25:05.176135+0000","last_peered":"2024-03-06T07:25:05.176135+0000","last_clean":"2024-03-06T07:25:05.176135+0000","last_became_active":"2024-03-06T07:25:05.167805+0000","last_became_peered":"2024-03-06T07:25:05.167805+0000","last_unstale":"2024-03-06T07:25:05.176135+0000","last_undegraded":"2024-03-06T07:25:05.176135+0000","last_fullsized":"2024-03-06T07:25:05.176135+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:21.185865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155709+0000","last_change":"2024-03-06T07:25:05.181216+0000","last_active":"2024-03-06T07:25:06.155709+0000","last_peered":"2024-03-06T07:25:06.155709+0000","last_clean":"2024-03-06T07:25:06.155709+0000","last_became_active":"2024-03-06T07:25:05.180875+0000","last_became_peered":"2024-03-06T07:25:05.180875+0000","last_unstale":"2024-03-06T07:25:06.155709+0000","last_undegraded":"2024-03-06T07:25:06.155709+0000","last_fullsized":"2024-03-06T07:25:06.155709+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:55.867315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187768+0000","last_change":"2024-03-06T07:25:05.183636+0000","last_active":"2024-03-06T07:25:05.187768+0000","last_peered":"2024-03-06T07:25:05.187768+0000","last_clean":"2024-03-06T07:25:05.187768+0000","last_became_active":"2024-03-06T07:25:05.183333+0000","last_became_peered":"2024-03-06T07:25:05.183333+0000","last_unstale":"2024-03-06T07:25:05.187768+0000","last_undegraded":"2024-03-06T07:25:05.187768+0000","last_fullsized":"2024-03-06T07:25:05.187768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:04.584017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172788+0000","last_change":"2024-03-06T07:25:05.165481+0000","last_active":"2024-03-06T07:25:06.172788+0000","last_peered":"2024-03-06T07:25:06.172788+0000","last_clean":"2024-03-06T07:25:06.172788+0000","last_became_active":"2024-03-06T07:25:05.165073+0000","last_became_peered":"2024-03-06T07:25:05.165073+0000","last_unstale":"2024-03-06T07:25:06.172788+0000","last_undegraded":"2024-03-06T07:25:06.172788+0000","last_fullsized":"2024-03-06T07:25:06.172788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:26.601256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.174837+0000","last_change":"2024-03-06T07:25:05.168055+0000","last_active":"2024-03-06T07:25:05.174837+0000","last_peered":"2024-03-06T07:25:05.174837+0000","last_clean":"2024-03-06T07:25:05.174837+0000","last_became_active":"2024-03-06T07:25:05.167764+0000","last_became_peered":"2024-03-06T07:25:05.167764+0000","last_unstale":"2024-03-06T07:25:05.174837+0000","last_undegraded":"2024-03-06T07:25:05.174837+0000","last_fullsized":"2024-03-06T07:25:05.174837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:18.044770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179727+0000","last_change":"2024-03-06T07:25:05.174182+0000","last_active":"2024-03-06T07:25:05.179727+0000","last_peered":"2024-03-06T07:25:05.179727+0000","last_clean":"2024-03-06T07:25:05.179727+0000","last_became_active":"2024-03-06T07:25:05.173885+0000","last_became_peered":"2024-03-06T07:25:05.173885+0000","last_unstale":"2024-03-06T07:25:05.179727+0000","last_undegraded":"2024-03-06T07:25:05.179727+0000","last_fullsized":"2024-03-06T07:25:05.179727+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:48.608381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.182055+0000","last_change":"2024-03-06T07:25:05.176762+0000","last_active":"2024-03-06T07:25:05.182055+0000","last_peered":"2024-03-06T07:25:05.182055+0000","last_clean":"2024-03-06T07:25:05.182055+0000","last_became_active":"2024-03-06T07:25:05.176507+0000","last_became_peered":"2024-03-06T07:25:05.176507+0000","last_unstale":"2024-03-06T07:25:05.182055+0000","last_undegraded":"2024-03-06T07:25:05.182055+0000","last_fullsized":"2024-03-06T07:25:05.182055+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:07.379633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187508+0000","last_change":"2024-03-06T07:25:05.181937+0000","last_active":"2024-03-06T07:25:05.187508+0000","last_peered":"2024-03-06T07:25:05.187508+0000","last_clean":"2024-03-06T07:25:05.187508+0000","last_became_active":"2024-03-06T07:25:05.181676+0000","last_became_peered":"2024-03-06T07:25:05.181676+0000","last_unstale":"2024-03-06T07:25:05.187508+0000","last_undegraded":"2024-03-06T07:25:05.187508+0000","last_fullsized":"2024-03-06T07:25:05.187508+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:43.727360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179999+0000","last_change":"2024-03-06T07:25:05.174088+0000","last_active":"2024-03-06T07:25:05.179999+0000","last_peered":"2024-03-06T07:25:05.179999+0000","last_clean":"2024-03-06T07:25:05.179999+0000","last_became_active":"2024-03-06T07:25:05.173799+0000","last_became_peered":"2024-03-06T07:25:05.173799+0000","last_unstale":"2024-03-06T07:25:05.179999+0000","last_undegraded":"2024-03-06T07:25:05.179999+0000","last_fullsized":"2024-03-06T07:25:05.179999+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:48.671903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156730+0000","last_change":"2024-03-06T07:25:05.175328+0000","last_active":"2024-03-06T07:25:06.156730+0000","last_peered":"2024-03-06T07:25:06.156730+0000","last_clean":"2024-03-06T07:25:06.156730+0000","last_became_active":"2024-03-06T07:25:05.175035+0000","last_became_peered":"2024-03-06T07:25:05.175035+0000","last_unstale":"2024-03-06T07:25:06.156730+0000","last_undegraded":"2024-03-06T07:25:06.156730+0000","last_fullsized":"2024-03-06T07:25:06.156730+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:38.802280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156373+0000","last_change":"2024-03-06T07:25:05.173245+0000","last_active":"2024-03-06T07:25:06.156373+0000","last_peered":"2024-03-06T07:25:06.156373+0000","last_clean":"2024-03-06T07:25:06.156373+0000","last_became_active":"2024-03-06T07:25:05.172920+0000","last_became_peered":"2024-03-06T07:25:05.172920+0000","last_unstale":"2024-03-06T07:25:06.156373+0000","last_undegraded":"2024-03-06T07:25:06.156373+0000","last_fullsized":"2024-03-06T07:25:06.156373+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:44.912192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179938+0000","last_change":"2024-03-06T07:25:05.174748+0000","last_active":"2024-03-06T07:25:05.179938+0000","last_peered":"2024-03-06T07:25:05.179938+0000","last_clean":"2024-03-06T07:25:05.179938+0000","last_became_active":"2024-03-06T07:25:05.174493+0000","last_became_peered":"2024-03-06T07:25:05.174493+0000","last_unstale":"2024-03-06T07:25:05.179938+0000","last_undegraded":"2024-03-06T07:25:05.179938+0000","last_fullsized":"2024-03-06T07:25:05.179938+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:38.868144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156031+0000","last_change":"2024-03-06T07:25:05.168584+0000","last_active":"2024-03-06T07:25:06.156031+0000","last_peered":"2024-03-06T07:25:06.156031+0000","last_clean":"2024-03-06T07:25:06.156031+0000","last_became_active":"2024-03-06T07:25:05.168291+0000","last_became_peered":"2024-03-06T07:25:05.168291+0000","last_unstale":"2024-03-06T07:25:06.156031+0000","last_undegraded":"2024-03-06T07:25:06.156031+0000","last_fullsized":"2024-03-06T07:25:06.156031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:02.657258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156621+0000","last_change":"2024-03-06T07:25:05.174070+0000","last_active":"2024-03-06T07:25:06.156621+0000","last_peered":"2024-03-06T07:25:06.156621+0000","last_clean":"2024-03-06T07:25:06.156621+0000","last_became_active":"2024-03-06T07:25:05.173795+0000","last_became_peered":"2024-03-06T07:25:05.173795+0000","last_unstale":"2024-03-06T07:25:06.156621+0000","last_undegraded":"2024-03-06T07:25:06.156621+0000","last_fullsized":"2024-03-06T07:25:06.156621+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:10.690693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156751+0000","last_change":"2024-03-06T07:25:05.170463+0000","last_active":"2024-03-06T07:25:06.156751+0000","last_peered":"2024-03-06T07:25:06.156751+0000","last_clean":"2024-03-06T07:25:06.156751+0000","last_became_active":"2024-03-06T07:25:05.170199+0000","last_became_peered":"2024-03-06T07:25:05.170199+0000","last_unstale":"2024-03-06T07:25:06.156751+0000","last_undegraded":"2024-03-06T07:25:06.156751+0000","last_fullsized":"2024-03-06T07:25:06.156751+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:52.667362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187547+0000","last_change":"2024-03-06T07:25:05.180187+0000","last_active":"2024-03-06T07:25:05.187547+0000","last_peered":"2024-03-06T07:25:05.187547+0000","last_clean":"2024-03-06T07:25:05.187547+0000","last_became_active":"2024-03-06T07:25:05.179807+0000","last_became_peered":"2024-03-06T07:25:05.179807+0000","last_unstale":"2024-03-06T07:25:05.187547+0000","last_undegraded":"2024-03-06T07:25:05.187547+0000","last_fullsized":"2024-03-06T07:25:05.187547+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:42.254969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.184822+0000","last_change":"2024-03-06T07:25:05.180668+0000","last_active":"2024-03-06T07:25:05.184822+0000","last_peered":"2024-03-06T07:25:05.184822+0000","last_clean":"2024-03-06T07:25:05.184822+0000","last_became_active":"2024-03-06T07:25:05.180435+0000","last_became_peered":"2024-03-06T07:25:05.180435+0000","last_unstale":"2024-03-06T07:25:05.184822+0000","last_undegraded":"2024-03-06T07:25:05.184822+0000","last_fullsized":"2024-03-06T07:25:05.184822+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:52.834751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"69'4","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.182218+0000","last_change":"2024-03-06T07:25:04.171525+0000","last_active":"2024-03-06T07:25:05.182218+0000","last_peered":"2024-03-06T07:25:05.182218+0000","last_clean":"2024-03-06T07:25:05.182218+0000","last_became_active":"2024-03-06T07:25:04.171243+0000","last_became_peered":"2024-03-06T07:25:04.171243+0000","last_unstale":"2024-03-06T07:25:05.182218+0000","last_undegraded":"2024-03-06T07:25:05.182218+0000","last_fullsized":"2024-03-06T07:25:05.182218+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:06.145388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.173937+0000","last_change":"2024-03-06T07:25:05.168417+0000","last_active":"2024-03-06T07:25:05.173937+0000","last_peered":"2024-03-06T07:25:05.173937+0000","last_clean":"2024-03-06T07:25:05.173937+0000","last_became_active":"2024-03-06T07:25:05.167429+0000","last_became_peered":"2024-03-06T07:25:05.167429+0000","last_unstale":"2024-03-06T07:25:05.173937+0000","last_undegraded":"2024-03-06T07:25:05.173937+0000","last_fullsized":"2024-03-06T07:25:05.173937+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:26.082546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172845+0000","last_change":"2024-03-06T07:25:04.165893+0000","last_active":"2024-03-06T07:25:06.172845+0000","last_peered":"2024-03-06T07:25:06.172845+0000","last_clean":"2024-03-06T07:25:06.172845+0000","last_became_active":"2024-03-06T07:25:04.165612+0000","last_became_peered":"2024-03-06T07:25:04.165612+0000","last_unstale":"2024-03-06T07:25:06.172845+0000","last_undegraded":"2024-03-06T07:25:06.172845+0000","last_fullsized":"2024-03-06T07:25:06.172845+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:54.865160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187826+0000","last_change":"2024-03-06T07:25:05.184206+0000","last_active":"2024-03-06T07:25:05.187826+0000","last_peered":"2024-03-06T07:25:05.187826+0000","last_clean":"2024-03-06T07:25:05.187826+0000","last_became_active":"2024-03-06T07:25:05.183741+0000","last_became_peered":"2024-03-06T07:25:05.183741+0000","last_unstale":"2024-03-06T07:25:05.187826+0000","last_undegraded":"2024-03-06T07:25:05.187826+0000","last_fullsized":"2024-03-06T07:25:05.187826+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:30.480318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.180251+0000","last_change":"2024-03-06T07:25:04.168953+0000","last_active":"2024-03-06T07:25:05.180251+0000","last_peered":"2024-03-06T07:25:05.180251+0000","last_clean":"2024-03-06T07:25:05.180251+0000","last_became_active":"2024-03-06T07:25:04.168671+0000","last_became_peered":"2024-03-06T07:25:04.168671+0000","last_unstale":"2024-03-06T07:25:05.180251+0000","last_undegraded":"2024-03-06T07:25:05.180251+0000","last_fullsized":"2024-03-06T07:25:05.180251+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:42.546597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179277+0000","last_change":"2024-03-06T07:25:05.173770+0000","last_active":"2024-03-06T07:25:05.179277+0000","last_peered":"2024-03-06T07:25:05.179277+0000","last_clean":"2024-03-06T07:25:05.179277+0000","last_became_active":"2024-03-06T07:25:05.173515+0000","last_became_peered":"2024-03-06T07:25:05.173515+0000","last_unstale":"2024-03-06T07:25:05.179277+0000","last_undegraded":"2024-03-06T07:25:05.179277+0000","last_fullsized":"2024-03-06T07:25:05.179277+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:26.498743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.183271+0000","last_change":"2024-03-06T07:25:04.172181+0000","last_active":"2024-03-06T07:25:05.183271+0000","last_peered":"2024-03-06T07:25:05.183271+0000","last_clean":"2024-03-06T07:25:05.183271+0000","last_became_active":"2024-03-06T07:25:04.171880+0000","last_became_peered":"2024-03-06T07:25:04.171880+0000","last_unstale":"2024-03-06T07:25:05.183271+0000","last_undegraded":"2024-03-06T07:25:05.183271+0000","last_fullsized":"2024-03-06T07:25:05.183271+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:14.594216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156644+0000","last_change":"2024-03-06T07:25:05.167977+0000","last_active":"2024-03-06T07:25:06.156644+0000","last_peered":"2024-03-06T07:25:06.156644+0000","last_clean":"2024-03-06T07:25:06.156644+0000","last_became_active":"2024-03-06T07:25:05.167376+0000","last_became_peered":"2024-03-06T07:25:05.167376+0000","last_unstale":"2024-03-06T07:25:06.156644+0000","last_undegraded":"2024-03-06T07:25:06.156644+0000","last_fullsized":"2024-03-06T07:25:06.156644+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:43.727360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156129+0000","last_change":"2024-03-06T07:25:04.166968+0000","last_active":"2024-03-06T07:25:06.156129+0000","last_peered":"2024-03-06T07:25:06.156129+0000","last_clean":"2024-03-06T07:25:06.156129+0000","last_became_active":"2024-03-06T07:25:04.166438+0000","last_became_peered":"2024-03-06T07:25:04.166438+0000","last_unstale":"2024-03-06T07:25:06.156129+0000","last_undegraded":"2024-03-06T07:25:06.156129+0000","last_fullsized":"2024-03-06T07:25:06.156129+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:18.530284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179854+0000","last_change":"2024-03-06T07:25:05.174645+0000","last_active":"2024-03-06T07:25:05.179854+0000","last_peered":"2024-03-06T07:25:05.179854+0000","last_clean":"2024-03-06T07:25:05.179854+0000","last_became_active":"2024-03-06T07:25:05.173522+0000","last_became_peered":"2024-03-06T07:25:05.173522+0000","last_unstale":"2024-03-06T07:25:05.179854+0000","last_undegraded":"2024-03-06T07:25:05.179854+0000","last_fullsized":"2024-03-06T07:25:05.179854+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:18.628420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179237+0000","last_change":"2024-03-06T07:25:04.172436+0000","last_active":"2024-03-06T07:25:05.179237+0000","last_peered":"2024-03-06T07:25:05.179237+0000","last_clean":"2024-03-06T07:25:05.179237+0000","last_became_active":"2024-03-06T07:25:04.172154+0000","last_became_peered":"2024-03-06T07:25:04.172154+0000","last_unstale":"2024-03-06T07:25:05.179237+0000","last_undegraded":"2024-03-06T07:25:05.179237+0000","last_fullsized":"2024-03-06T07:25:05.179237+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:14.362185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.169290+0000","last_change":"2024-03-06T07:25:05.162264+0000","last_active":"2024-03-06T07:25:05.169290+0000","last_peered":"2024-03-06T07:25:05.169290+0000","last_clean":"2024-03-06T07:25:05.169290+0000","last_became_active":"2024-03-06T07:25:05.161977+0000","last_became_peered":"2024-03-06T07:25:05.161977+0000","last_unstale":"2024-03-06T07:25:05.169290+0000","last_undegraded":"2024-03-06T07:25:05.169290+0000","last_fullsized":"2024-03-06T07:25:05.169290+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:02.761359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155285+0000","last_change":"2024-03-06T07:25:04.171825+0000","last_active":"2024-03-06T07:25:06.155285+0000","last_peered":"2024-03-06T07:25:06.155285+0000","last_clean":"2024-03-06T07:25:06.155285+0000","last_became_active":"2024-03-06T07:25:04.171576+0000","last_became_peered":"2024-03-06T07:25:04.171576+0000","last_unstale":"2024-03-06T07:25:06.155285+0000","last_undegraded":"2024-03-06T07:25:06.155285+0000","last_fullsized":"2024-03-06T07:25:06.155285+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:10.525330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172564+0000","last_change":"2024-03-06T07:25:05.171716+0000","last_active":"2024-03-06T07:25:06.172564+0000","last_peered":"2024-03-06T07:25:06.172564+0000","last_clean":"2024-03-06T07:25:06.172564+0000","last_became_active":"2024-03-06T07:25:05.171449+0000","last_became_peered":"2024-03-06T07:25:05.171449+0000","last_unstale":"2024-03-06T07:25:06.172564+0000","last_undegraded":"2024-03-06T07:25:06.172564+0000","last_fullsized":"2024-03-06T07:25:06.172564+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:38.802280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.155809+0000","last_change":"2024-03-06T07:25:04.163693+0000","last_active":"2024-03-06T07:25:05.155809+0000","last_peered":"2024-03-06T07:25:05.155809+0000","last_clean":"2024-03-06T07:25:05.155809+0000","last_became_active":"2024-03-06T07:25:04.162812+0000","last_became_peered":"2024-03-06T07:25:04.162812+0000","last_unstale":"2024-03-06T07:25:05.155809+0000","last_undegraded":"2024-03-06T07:25:05.155809+0000","last_fullsized":"2024-03-06T07:25:05.155809+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:53.039314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156864+0000","last_change":"2024-03-06T07:25:05.172330+0000","last_active":"2024-03-06T07:25:06.156864+0000","last_peered":"2024-03-06T07:25:06.156864+0000","last_clean":"2024-03-06T07:25:06.156864+0000","last_became_active":"2024-03-06T07:25:05.172071+0000","last_became_peered":"2024-03-06T07:25:05.172071+0000","last_unstale":"2024-03-06T07:25:06.156864+0000","last_undegraded":"2024-03-06T07:25:06.156864+0000","last_fullsized":"2024-03-06T07:25:06.156864+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:15.620644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.158825+0000","last_change":"2024-03-06T07:25:04.162855+0000","last_active":"2024-03-06T07:25:05.158825+0000","last_peered":"2024-03-06T07:25:05.158825+0000","last_clean":"2024-03-06T07:25:05.158825+0000","last_became_active":"2024-03-06T07:25:04.162574+0000","last_became_peered":"2024-03-06T07:25:04.162574+0000","last_unstale":"2024-03-06T07:25:05.158825+0000","last_undegraded":"2024-03-06T07:25:05.158825+0000","last_fullsized":"2024-03-06T07:25:05.158825+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:17.933958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.184900+0000","last_change":"2024-03-06T07:25:05.181471+0000","last_active":"2024-03-06T07:25:05.184900+0000","last_peered":"2024-03-06T07:25:05.184900+0000","last_clean":"2024-03-06T07:25:05.184900+0000","last_became_active":"2024-03-06T07:25:05.181208+0000","last_became_peered":"2024-03-06T07:25:05.181208+0000","last_unstale":"2024-03-06T07:25:05.184900+0000","last_undegraded":"2024-03-06T07:25:05.184900+0000","last_fullsized":"2024-03-06T07:25:05.184900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:10.690693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.169692+0000","last_change":"2024-03-06T07:25:04.166491+0000","last_active":"2024-03-06T07:25:05.169692+0000","last_peered":"2024-03-06T07:25:05.169692+0000","last_clean":"2024-03-06T07:25:05.169692+0000","last_became_active":"2024-03-06T07:25:04.166179+0000","last_became_peered":"2024-03-06T07:25:04.166179+0000","last_unstale":"2024-03-06T07:25:05.169692+0000","last_undegraded":"2024-03-06T07:25:05.169692+0000","last_fullsized":"2024-03-06T07:25:05.169692+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:00.885206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.184899+0000","last_change":"2024-03-06T07:25:05.177728+0000","last_active":"2024-03-06T07:25:05.184899+0000","last_peered":"2024-03-06T07:25:05.184899+0000","last_clean":"2024-03-06T07:25:05.184899+0000","last_became_active":"2024-03-06T07:25:05.177466+0000","last_became_peered":"2024-03-06T07:25:05.177466+0000","last_unstale":"2024-03-06T07:25:05.184899+0000","last_undegraded":"2024-03-06T07:25:05.184899+0000","last_fullsized":"2024-03-06T07:25:05.184899+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:17.300384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"69'3","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156432+0000","last_change":"2024-03-06T07:25:04.158336+0000","last_active":"2024-03-06T07:25:06.156432+0000","last_peered":"2024-03-06T07:25:06.156432+0000","last_clean":"2024-03-06T07:25:06.156432+0000","last_became_active":"2024-03-06T07:25:04.158042+0000","last_became_peered":"2024-03-06T07:25:04.158042+0000","last_unstale":"2024-03-06T07:25:06.156432+0000","last_undegraded":"2024-03-06T07:25:06.156432+0000","last_fullsized":"2024-03-06T07:25:06.156432+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:50.416138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176227+0000","last_change":"2024-03-06T07:25:05.169972+0000","last_active":"2024-03-06T07:25:05.176227+0000","last_peered":"2024-03-06T07:25:05.176227+0000","last_clean":"2024-03-06T07:25:05.176227+0000","last_became_active":"2024-03-06T07:25:05.169690+0000","last_became_peered":"2024-03-06T07:25:05.169690+0000","last_unstale":"2024-03-06T07:25:05.176227+0000","last_undegraded":"2024-03-06T07:25:05.176227+0000","last_fullsized":"2024-03-06T07:25:05.176227+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:30.541732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"69'6","reported_seq":20,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155749+0000","last_change":"2024-03-06T07:25:04.170811+0000","last_active":"2024-03-06T07:25:06.155749+0000","last_peered":"2024-03-06T07:25:06.155749+0000","last_clean":"2024-03-06T07:25:06.155749+0000","last_became_active":"2024-03-06T07:25:04.170480+0000","last_became_peered":"2024-03-06T07:25:04.170480+0000","last_unstale":"2024-03-06T07:25:06.155749+0000","last_undegraded":"2024-03-06T07:25:06.155749+0000","last_fullsized":"2024-03-06T07:25:06.155749+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:21.745380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.173078+0000","last_change":"2024-03-06T07:25:05.172095+0000","last_active":"2024-03-06T07:25:05.173078+0000","last_peered":"2024-03-06T07:25:05.173078+0000","last_clean":"2024-03-06T07:25:05.173078+0000","last_became_active":"2024-03-06T07:25:05.171834+0000","last_became_peered":"2024-03-06T07:25:05.171834+0000","last_unstale":"2024-03-06T07:25:05.173078+0000","last_undegraded":"2024-03-06T07:25:05.173078+0000","last_fullsized":"2024-03-06T07:25:05.173078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:29.527566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155438+0000","last_change":"2024-03-06T07:25:04.172900+0000","last_active":"2024-03-06T07:25:06.155438+0000","last_peered":"2024-03-06T07:25:06.155438+0000","last_clean":"2024-03-06T07:25:06.155438+0000","last_became_active":"2024-03-06T07:25:04.172598+0000","last_became_peered":"2024-03-06T07:25:04.172598+0000","last_unstale":"2024-03-06T07:25:06.155438+0000","last_undegraded":"2024-03-06T07:25:06.155438+0000","last_fullsized":"2024-03-06T07:25:06.155438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:17.933958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156824+0000","last_change":"2024-03-06T07:25:05.176530+0000","last_active":"2024-03-06T07:25:06.156824+0000","last_peered":"2024-03-06T07:25:06.156824+0000","last_clean":"2024-03-06T07:25:06.156824+0000","last_became_active":"2024-03-06T07:25:05.176242+0000","last_became_peered":"2024-03-06T07:25:05.176242+0000","last_unstale":"2024-03-06T07:25:06.156824+0000","last_undegraded":"2024-03-06T07:25:06.156824+0000","last_fullsized":"2024-03-06T07:25:06.156824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:26.601256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.165525+0000","last_change":"2024-03-06T07:25:04.168096+0000","last_active":"2024-03-06T07:25:05.165525+0000","last_peered":"2024-03-06T07:25:05.165525+0000","last_clean":"2024-03-06T07:25:05.165525+0000","last_became_active":"2024-03-06T07:25:04.167649+0000","last_became_peered":"2024-03-06T07:25:04.167649+0000","last_unstale":"2024-03-06T07:25:05.165525+0000","last_undegraded":"2024-03-06T07:25:05.165525+0000","last_fullsized":"2024-03-06T07:25:05.165525+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:02.250812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172562+0000","last_change":"2024-03-06T07:25:05.175345+0000","last_active":"2024-03-06T07:25:06.172562+0000","last_peered":"2024-03-06T07:25:06.172562+0000","last_clean":"2024-03-06T07:25:06.172562+0000","last_became_active":"2024-03-06T07:25:05.175071+0000","last_became_peered":"2024-03-06T07:25:05.175071+0000","last_unstale":"2024-03-06T07:25:06.172562+0000","last_undegraded":"2024-03-06T07:25:06.172562+0000","last_fullsized":"2024-03-06T07:25:06.172562+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:43.727360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156471+0000","last_change":"2024-03-06T07:25:04.166090+0000","last_active":"2024-03-06T07:25:06.156471+0000","last_peered":"2024-03-06T07:25:06.156471+0000","last_clean":"2024-03-06T07:25:06.156471+0000","last_became_active":"2024-03-06T07:25:04.165783+0000","last_became_peered":"2024-03-06T07:25:04.165783+0000","last_unstale":"2024-03-06T07:25:06.156471+0000","last_undegraded":"2024-03-06T07:25:06.156471+0000","last_fullsized":"2024-03-06T07:25:06.156471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:15.369337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.169310+0000","last_change":"2024-03-06T07:25:05.166736+0000","last_active":"2024-03-06T07:25:05.169310+0000","last_peered":"2024-03-06T07:25:05.169310+0000","last_clean":"2024-03-06T07:25:05.169310+0000","last_became_active":"2024-03-06T07:25:05.166321+0000","last_became_peered":"2024-03-06T07:25:05.166321+0000","last_unstale":"2024-03-06T07:25:05.169310+0000","last_undegraded":"2024-03-06T07:25:05.169310+0000","last_fullsized":"2024-03-06T07:25:05.169310+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:59.961820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"69'8","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.177826+0000","last_change":"2024-03-06T07:25:04.168166+0000","last_active":"2024-03-06T07:25:05.177826+0000","last_peered":"2024-03-06T07:25:05.177826+0000","last_clean":"2024-03-06T07:25:05.177826+0000","last_became_active":"2024-03-06T07:25:04.167871+0000","last_became_peered":"2024-03-06T07:25:04.167871+0000","last_unstale":"2024-03-06T07:25:05.177826+0000","last_undegraded":"2024-03-06T07:25:05.177826+0000","last_fullsized":"2024-03-06T07:25:05.177826+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:10.525330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.183838+0000","last_change":"2024-03-06T07:25:05.180594+0000","last_active":"2024-03-06T07:25:05.183838+0000","last_peered":"2024-03-06T07:25:05.183838+0000","last_clean":"2024-03-06T07:25:05.183838+0000","last_became_active":"2024-03-06T07:25:05.180296+0000","last_became_peered":"2024-03-06T07:25:05.180296+0000","last_unstale":"2024-03-06T07:25:05.183838+0000","last_undegraded":"2024-03-06T07:25:05.183838+0000","last_fullsized":"2024-03-06T07:25:05.183838+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:16.371492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179978+0000","last_change":"2024-03-06T07:25:04.170917+0000","last_active":"2024-03-06T07:25:05.179978+0000","last_peered":"2024-03-06T07:25:05.179978+0000","last_clean":"2024-03-06T07:25:05.179978+0000","last_became_active":"2024-03-06T07:25:04.170632+0000","last_became_peered":"2024-03-06T07:25:04.170632+0000","last_unstale":"2024-03-06T07:25:05.179978+0000","last_undegraded":"2024-03-06T07:25:05.179978+0000","last_fullsized":"2024-03-06T07:25:05.179978+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:27.124759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.184884+0000","last_change":"2024-03-06T07:25:05.180901+0000","last_active":"2024-03-06T07:25:05.184884+0000","last_peered":"2024-03-06T07:25:05.184884+0000","last_clean":"2024-03-06T07:25:05.184884+0000","last_became_active":"2024-03-06T07:25:05.180660+0000","last_became_peered":"2024-03-06T07:25:05.180660+0000","last_unstale":"2024-03-06T07:25:05.184884+0000","last_undegraded":"2024-03-06T07:25:05.184884+0000","last_fullsized":"2024-03-06T07:25:05.184884+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:30.604907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179634+0000","last_change":"2024-03-06T07:25:04.171879+0000","last_active":"2024-03-06T07:25:05.179634+0000","last_peered":"2024-03-06T07:25:05.179634+0000","last_clean":"2024-03-06T07:25:05.179634+0000","last_became_active":"2024-03-06T07:25:04.171605+0000","last_became_peered":"2024-03-06T07:25:04.171605+0000","last_unstale":"2024-03-06T07:25:05.179634+0000","last_undegraded":"2024-03-06T07:25:05.179634+0000","last_fullsized":"2024-03-06T07:25:05.179634+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:21.745380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155679+0000","last_change":"2024-03-06T07:25:05.175642+0000","last_active":"2024-03-06T07:25:06.155679+0000","last_peered":"2024-03-06T07:25:06.155679+0000","last_clean":"2024-03-06T07:25:06.155679+0000","last_became_active":"2024-03-06T07:25:05.175303+0000","last_became_peered":"2024-03-06T07:25:05.175303+0000","last_unstale":"2024-03-06T07:25:06.155679+0000","last_undegraded":"2024-03-06T07:25:06.155679+0000","last_fullsized":"2024-03-06T07:25:06.155679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:07.147543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.181645+0000","last_change":"2024-03-06T07:25:04.174153+0000","last_active":"2024-03-06T07:25:05.181645+0000","last_peered":"2024-03-06T07:25:05.181645+0000","last_clean":"2024-03-06T07:25:05.181645+0000","last_became_active":"2024-03-06T07:25:04.173868+0000","last_became_peered":"2024-03-06T07:25:04.173868+0000","last_unstale":"2024-03-06T07:25:05.181645+0000","last_undegraded":"2024-03-06T07:25:05.181645+0000","last_fullsized":"2024-03-06T07:25:05.181645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:53.401899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176371+0000","last_change":"2024-03-06T07:25:05.174743+0000","last_active":"2024-03-06T07:25:05.176371+0000","last_peered":"2024-03-06T07:25:05.176371+0000","last_clean":"2024-03-06T07:25:05.176371+0000","last_became_active":"2024-03-06T07:25:05.174441+0000","last_became_peered":"2024-03-06T07:25:05.174441+0000","last_unstale":"2024-03-06T07:25:05.176371+0000","last_undegraded":"2024-03-06T07:25:05.176371+0000","last_fullsized":"2024-03-06T07:25:05.176371+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.136158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.163309+0000","last_change":"2024-03-06T07:25:04.174253+0000","last_active":"2024-03-06T07:25:05.163309+0000","last_peered":"2024-03-06T07:25:05.163309+0000","last_clean":"2024-03-06T07:25:05.163309+0000","last_became_active":"2024-03-06T07:25:04.173938+0000","last_became_peered":"2024-03-06T07:25:04.173938+0000","last_unstale":"2024-03-06T07:25:05.163309+0000","last_undegraded":"2024-03-06T07:25:05.163309+0000","last_fullsized":"2024-03-06T07:25:05.163309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:15.369337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156159+0000","last_change":"2024-03-06T07:25:05.179646+0000","last_active":"2024-03-06T07:25:06.156159+0000","last_peered":"2024-03-06T07:25:06.156159+0000","last_clean":"2024-03-06T07:25:06.156159+0000","last_became_active":"2024-03-06T07:25:05.179317+0000","last_became_peered":"2024-03-06T07:25:05.179317+0000","last_unstale":"2024-03-06T07:25:06.156159+0000","last_undegraded":"2024-03-06T07:25:06.156159+0000","last_fullsized":"2024-03-06T07:25:06.156159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:28.040514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.165650+0000","last_change":"2024-03-06T07:25:04.172205+0000","last_active":"2024-03-06T07:25:05.165650+0000","last_peered":"2024-03-06T07:25:05.165650+0000","last_clean":"2024-03-06T07:25:05.165650+0000","last_became_active":"2024-03-06T07:25:04.171932+0000","last_became_peered":"2024-03-06T07:25:04.171932+0000","last_unstale":"2024-03-06T07:25:05.165650+0000","last_undegraded":"2024-03-06T07:25:05.165650+0000","last_fullsized":"2024-03-06T07:25:05.165650+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:08.265287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155995+0000","last_change":"2024-03-06T07:25:05.182790+0000","last_active":"2024-03-06T07:25:06.155995+0000","last_peered":"2024-03-06T07:25:06.155995+0000","last_clean":"2024-03-06T07:25:06.155995+0000","last_became_active":"2024-03-06T07:25:05.182322+0000","last_became_peered":"2024-03-06T07:25:05.182322+0000","last_unstale":"2024-03-06T07:25:06.155995+0000","last_undegraded":"2024-03-06T07:25:06.155995+0000","last_fullsized":"2024-03-06T07:25:06.155995+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:00.651166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156855+0000","last_change":"2024-03-06T07:25:04.164589+0000","last_active":"2024-03-06T07:25:06.156855+0000","last_peered":"2024-03-06T07:25:06.156855+0000","last_clean":"2024-03-06T07:25:06.156855+0000","last_became_active":"2024-03-06T07:25:04.164235+0000","last_became_peered":"2024-03-06T07:25:04.164235+0000","last_unstale":"2024-03-06T07:25:06.156855+0000","last_undegraded":"2024-03-06T07:25:06.156855+0000","last_fullsized":"2024-03-06T07:25:06.156855+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:06.145388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156051+0000","last_change":"2024-03-06T07:25:05.180865+0000","last_active":"2024-03-06T07:25:06.156051+0000","last_peered":"2024-03-06T07:25:06.156051+0000","last_clean":"2024-03-06T07:25:06.156051+0000","last_became_active":"2024-03-06T07:25:05.180568+0000","last_became_peered":"2024-03-06T07:25:05.180568+0000","last_unstale":"2024-03-06T07:25:06.156051+0000","last_undegraded":"2024-03-06T07:25:06.156051+0000","last_fullsized":"2024-03-06T07:25:06.156051+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:03.798281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172511+0000","last_change":"2024-03-06T07:25:04.166345+0000","last_active":"2024-03-06T07:25:06.172511+0000","last_peered":"2024-03-06T07:25:06.172511+0000","last_clean":"2024-03-06T07:25:06.172511+0000","last_became_active":"2024-03-06T07:25:04.166053+0000","last_became_peered":"2024-03-06T07:25:04.166053+0000","last_unstale":"2024-03-06T07:25:06.172511+0000","last_undegraded":"2024-03-06T07:25:06.172511+0000","last_fullsized":"2024-03-06T07:25:06.172511+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:06.145388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172828+0000","last_change":"2024-03-06T07:25:05.170943+0000","last_active":"2024-03-06T07:25:06.172828+0000","last_peered":"2024-03-06T07:25:06.172828+0000","last_clean":"2024-03-06T07:25:06.172828+0000","last_became_active":"2024-03-06T07:25:05.170576+0000","last_became_peered":"2024-03-06T07:25:05.170576+0000","last_unstale":"2024-03-06T07:25:06.172828+0000","last_undegraded":"2024-03-06T07:25:06.172828+0000","last_fullsized":"2024-03-06T07:25:06.172828+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:15.620644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179721+0000","last_change":"2024-03-06T07:25:04.170274+0000","last_active":"2024-03-06T07:25:05.179721+0000","last_peered":"2024-03-06T07:25:05.179721+0000","last_clean":"2024-03-06T07:25:05.179721+0000","last_became_active":"2024-03-06T07:25:04.170001+0000","last_became_peered":"2024-03-06T07:25:04.170001+0000","last_unstale":"2024-03-06T07:25:05.179721+0000","last_undegraded":"2024-03-06T07:25:05.179721+0000","last_fullsized":"2024-03-06T07:25:05.179721+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:33.098672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156739+0000","last_change":"2024-03-06T07:25:05.181381+0000","last_active":"2024-03-06T07:25:06.156739+0000","last_peered":"2024-03-06T07:25:06.156739+0000","last_clean":"2024-03-06T07:25:06.156739+0000","last_became_active":"2024-03-06T07:25:05.181054+0000","last_became_peered":"2024-03-06T07:25:05.181054+0000","last_unstale":"2024-03-06T07:25:06.156739+0000","last_undegraded":"2024-03-06T07:25:06.156739+0000","last_fullsized":"2024-03-06T07:25:06.156739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:34.634891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156849+0000","last_change":"2024-03-06T07:25:04.165832+0000","last_active":"2024-03-06T07:25:06.156849+0000","last_peered":"2024-03-06T07:25:06.156849+0000","last_clean":"2024-03-06T07:25:06.156849+0000","last_became_active":"2024-03-06T07:25:04.165536+0000","last_became_peered":"2024-03-06T07:25:04.165536+0000","last_unstale":"2024-03-06T07:25:06.156849+0000","last_undegraded":"2024-03-06T07:25:06.156849+0000","last_fullsized":"2024-03-06T07:25:06.156849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:54.865160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.184859+0000","last_change":"2024-03-06T07:25:05.181291+0000","last_active":"2024-03-06T07:25:05.184859+0000","last_peered":"2024-03-06T07:25:05.184859+0000","last_clean":"2024-03-06T07:25:05.184859+0000","last_became_active":"2024-03-06T07:25:05.181044+0000","last_became_peered":"2024-03-06T07:25:05.181044+0000","last_unstale":"2024-03-06T07:25:05.184859+0000","last_undegraded":"2024-03-06T07:25:05.184859+0000","last_fullsized":"2024-03-06T07:25:05.184859+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:12.941579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.155524+0000","last_change":"2024-03-06T07:25:04.166343+0000","last_active":"2024-03-06T07:25:05.155524+0000","last_peered":"2024-03-06T07:25:05.155524+0000","last_clean":"2024-03-06T07:25:05.155524+0000","last_became_active":"2024-03-06T07:25:04.166086+0000","last_became_peered":"2024-03-06T07:25:04.166086+0000","last_unstale":"2024-03-06T07:25:05.155524+0000","last_undegraded":"2024-03-06T07:25:05.155524+0000","last_fullsized":"2024-03-06T07:25:05.155524+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:29.946081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.182856+0000","last_change":"2024-03-06T07:25:05.179127+0000","last_active":"2024-03-06T07:25:05.182856+0000","last_peered":"2024-03-06T07:25:05.182856+0000","last_clean":"2024-03-06T07:25:05.182856+0000","last_became_active":"2024-03-06T07:25:05.178480+0000","last_became_peered":"2024-03-06T07:25:05.178480+0000","last_unstale":"2024-03-06T07:25:05.182856+0000","last_undegraded":"2024-03-06T07:25:05.182856+0000","last_fullsized":"2024-03-06T07:25:05.182856+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:45.697928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172725+0000","last_change":"2024-03-06T07:25:04.172405+0000","last_active":"2024-03-06T07:25:06.172725+0000","last_peered":"2024-03-06T07:25:06.172725+0000","last_clean":"2024-03-06T07:25:06.172725+0000","last_became_active":"2024-03-06T07:25:04.172070+0000","last_became_peered":"2024-03-06T07:25:04.172070+0000","last_unstale":"2024-03-06T07:25:06.172725+0000","last_undegraded":"2024-03-06T07:25:06.172725+0000","last_fullsized":"2024-03-06T07:25:06.172725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:28.525411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176192+0000","last_change":"2024-03-06T07:25:05.169212+0000","last_active":"2024-03-06T07:25:05.176192+0000","last_peered":"2024-03-06T07:25:05.176192+0000","last_clean":"2024-03-06T07:25:05.176192+0000","last_became_active":"2024-03-06T07:25:05.168919+0000","last_became_peered":"2024-03-06T07:25:05.168919+0000","last_unstale":"2024-03-06T07:25:05.176192+0000","last_undegraded":"2024-03-06T07:25:05.176192+0000","last_fullsized":"2024-03-06T07:25:05.176192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:01.757888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.155664+0000","last_change":"2024-03-06T07:25:04.160814+0000","last_active":"2024-03-06T07:25:05.155664+0000","last_peered":"2024-03-06T07:25:05.155664+0000","last_clean":"2024-03-06T07:25:05.155664+0000","last_became_active":"2024-03-06T07:25:04.160521+0000","last_became_peered":"2024-03-06T07:25:04.160521+0000","last_unstale":"2024-03-06T07:25:05.155664+0000","last_undegraded":"2024-03-06T07:25:05.155664+0000","last_fullsized":"2024-03-06T07:25:05.155664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:54.662090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187840+0000","last_change":"2024-03-06T07:25:05.184235+0000","last_active":"2024-03-06T07:25:05.187840+0000","last_peered":"2024-03-06T07:25:05.187840+0000","last_clean":"2024-03-06T07:25:05.187840+0000","last_became_active":"2024-03-06T07:25:05.183943+0000","last_became_peered":"2024-03-06T07:25:05.183943+0000","last_unstale":"2024-03-06T07:25:05.187840+0000","last_undegraded":"2024-03-06T07:25:05.187840+0000","last_fullsized":"2024-03-06T07:25:05.187840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:27.369584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155283+0000","last_change":"2024-03-06T07:25:04.171677+0000","last_active":"2024-03-06T07:25:06.155283+0000","last_peered":"2024-03-06T07:25:06.155283+0000","last_clean":"2024-03-06T07:25:06.155283+0000","last_became_active":"2024-03-06T07:25:04.171415+0000","last_became_peered":"2024-03-06T07:25:04.171415+0000","last_unstale":"2024-03-06T07:25:06.155283+0000","last_undegraded":"2024-03-06T07:25:06.155283+0000","last_fullsized":"2024-03-06T07:25:06.155283+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:42.546597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155377+0000","last_change":"2024-03-06T07:25:05.167404+0000","last_active":"2024-03-06T07:25:06.155377+0000","last_peered":"2024-03-06T07:25:06.155377+0000","last_clean":"2024-03-06T07:25:06.155377+0000","last_became_active":"2024-03-06T07:25:05.167076+0000","last_became_peered":"2024-03-06T07:25:05.167076+0000","last_unstale":"2024-03-06T07:25:06.155377+0000","last_undegraded":"2024-03-06T07:25:06.155377+0000","last_fullsized":"2024-03-06T07:25:06.155377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:38.868144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.158252+0000","last_change":"2024-03-06T07:25:04.168113+0000","last_active":"2024-03-06T07:25:05.158252+0000","last_peered":"2024-03-06T07:25:05.158252+0000","last_clean":"2024-03-06T07:25:05.158252+0000","last_became_active":"2024-03-06T07:25:04.167821+0000","last_became_peered":"2024-03-06T07:25:04.167821+0000","last_unstale":"2024-03-06T07:25:05.158252+0000","last_undegraded":"2024-03-06T07:25:05.158252+0000","last_fullsized":"2024-03-06T07:25:05.158252+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:48.554709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187563+0000","last_change":"2024-03-06T07:25:05.180999+0000","last_active":"2024-03-06T07:25:05.187563+0000","last_peered":"2024-03-06T07:25:05.187563+0000","last_clean":"2024-03-06T07:25:05.187563+0000","last_became_active":"2024-03-06T07:25:05.180739+0000","last_became_peered":"2024-03-06T07:25:05.180739+0000","last_unstale":"2024-03-06T07:25:05.187563+0000","last_undegraded":"2024-03-06T07:25:05.187563+0000","last_fullsized":"2024-03-06T07:25:05.187563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:34.775717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.164943+0000","last_change":"2024-03-06T07:25:04.169366+0000","last_active":"2024-03-06T07:25:05.164943+0000","last_peered":"2024-03-06T07:25:05.164943+0000","last_clean":"2024-03-06T07:25:05.164943+0000","last_became_active":"2024-03-06T07:25:04.168918+0000","last_became_peered":"2024-03-06T07:25:04.168918+0000","last_unstale":"2024-03-06T07:25:05.164943+0000","last_undegraded":"2024-03-06T07:25:05.164943+0000","last_fullsized":"2024-03-06T07:25:05.164943+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:57.134003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156064+0000","last_change":"2024-03-06T07:25:05.181711+0000","last_active":"2024-03-06T07:25:06.156064+0000","last_peered":"2024-03-06T07:25:06.156064+0000","last_clean":"2024-03-06T07:25:06.156064+0000","last_became_active":"2024-03-06T07:25:05.181416+0000","last_became_peered":"2024-03-06T07:25:05.181416+0000","last_unstale":"2024-03-06T07:25:06.156064+0000","last_undegraded":"2024-03-06T07:25:06.156064+0000","last_fullsized":"2024-03-06T07:25:06.156064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:52.374211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.182171+0000","last_change":"2024-03-06T07:25:04.168606+0000","last_active":"2024-03-06T07:25:05.182171+0000","last_peered":"2024-03-06T07:25:05.182171+0000","last_clean":"2024-03-06T07:25:05.182171+0000","last_became_active":"2024-03-06T07:25:04.168127+0000","last_became_peered":"2024-03-06T07:25:04.168127+0000","last_unstale":"2024-03-06T07:25:05.182171+0000","last_undegraded":"2024-03-06T07:25:05.182171+0000","last_fullsized":"2024-03-06T07:25:05.182171+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:29.002981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":258,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.164116+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:25:05.164116+0000","last_peered":"2024-03-06T07:25:05.164116+0000","last_clean":"2024-03-06T07:25:05.164116+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:25:05.164116+0000","last_undegraded":"2024-03-06T07:25:05.164116+0000","last_fullsized":"2024-03-06T07:25:05.164116+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32592,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339248,"statfs":{"total":96636764160,"available":96603389952,"internally_reserved":0,"allocated":1441792,"data_stored":943656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":847872,"data_stored":352764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":843776,"data_stored":352762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":843776,"data_stored":352742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":847872,"data_stored":352784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":851968,"data_stored":354284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":354900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574893,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1433600,"data_stored":943040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:08.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:08.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.296+0000 7f8a3a8fb640 1 Processor -- start 2024-03-06T07:25:08.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.296+0000 7f8a3a8fb640 1 -- start start 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.297+0000 7f8a3a8fb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8a3415bb20 0x7f8a340d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.297+0000 7f8a3a8fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340cd8c0 0x7f8a340d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.297+0000 7f8a3a8fb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a34161980 con 0x7f8a340cfff0 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.297+0000 7f8a3a8fb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a34161c60 con 0x7f8a3415bb20 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.297+0000 7f8a3a8fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a34161f40 con 0x7f8a340cd8c0 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.297+0000 7f8a33fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8a3415bb20 0x7f8a340d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.297+0000 7f8a337fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8a340cfff0 0x7f8a340ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58784/0 (socket says 172.21.15.59:58784) 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.297+0000 7f8a337fe640 1 -- 172.21.15.59:0/846401093 learned_addr learned my addr 172.21.15.59:0/846401093 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.298+0000 7f8a38e98640 1 --2- 172.21.15.59:0/846401093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340cd8c0 0x7f8a340d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.298+0000 7f8a33fff640 1 -- 172.21.15.59:0/846401093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340cd8c0 msgr2=0x7f8a340d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.298+0000 7f8a33fff640 1 --2- 172.21.15.59:0/846401093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340cd8c0 0x7f8a340d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.298+0000 7f8a33fff640 1 -- 172.21.15.59:0/846401093 >> v1:172.21.15.59:6789/0 conn(0x7f8a340cfff0 legacy=0x7f8a340ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.298+0000 7f8a32ffd640 1 -- 172.21.15.59:0/846401093 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408157398 0 0) 0x7f8a34161980 con 0x7f8a340cfff0 2024-03-06T07:25:08.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.298+0000 7f8a33fff640 1 -- 172.21.15.59:0/846401093 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a34162220 con 0x7f8a3415bb20 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.298+0000 7f8a33fff640 1 --2- 172.21.15.59:0/846401093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a3415bb20 0x7f8a340d1df0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0097a0 tx=0x7f8a1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e6f2be30fb1090 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.299+0000 7f8a32ffd640 1 -- 172.21.15.59:0/846401093 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c035020 con 0x7f8a3415bb20 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.299+0000 7f8a32ffd640 1 -- 172.21.15.59:0/846401093 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c031830 con 0x7f8a3415bb20 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.299+0000 7f8a32ffd640 1 -- 172.21.15.59:0/846401093 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c031c00 con 0x7f8a3415bb20 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.299+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/846401093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a3415bb20 msgr2=0x7f8a340d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.299+0000 7f8a3a8fb640 1 --2- 172.21.15.59:0/846401093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a3415bb20 0x7f8a340d1df0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0097a0 tx=0x7f8a1c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.299+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/846401093 shutdown_connections 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.299+0000 7f8a3a8fb640 1 --2- 172.21.15.59:0/846401093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340cd8c0 0x7f8a340d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.299+0000 7f8a3a8fb640 1 --2- 172.21.15.59:0/846401093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a3415bb20 0x7f8a340d1df0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.299+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/846401093 >> 172.21.15.59:0/846401093 conn(0x7f8a3409a9f0 msgr2=0x7f8a3409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.299+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/846401093 shutdown_connections 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.299+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/846401093 wait complete. 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.300+0000 7f8a3a8fb640 1 Processor -- start 2024-03-06T07:25:08.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.301+0000 7f8a3a8fb640 1 -- start start 2024-03-06T07:25:08.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.301+0000 7f8a3a8fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a340cd8c0 0x7f8a341f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.301+0000 7f8a3a8fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340cfff0 0x7f8a341f4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.301+0000 7f8a3a8fb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a3415bb20 0x7f8a3415a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.301+0000 7f8a3a8fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340d24b0 con 0x7f8a340cd8c0 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.301+0000 7f8a3a8fb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340d2330 con 0x7f8a3415bb20 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.301+0000 7f8a3a8fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340d1980 con 0x7f8a340cfff0 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.301+0000 7f8a33fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a340cd8c0 0x7f8a341f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.301+0000 7f8a33fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a340cd8c0 0x7f8a341f0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54742/0 (socket says 172.21.15.59:54742) 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.301+0000 7f8a33fff640 1 -- 172.21.15.59:0/3923212036 learned_addr learned my addr 172.21.15.59:0/3923212036 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.301+0000 7f8a337fe640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340cfff0 0x7f8a341f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a38e98640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a3415bb20 0x7f8a3415a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a33fff640 1 -- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340cfff0 msgr2=0x7f8a341f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a33fff640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340cfff0 0x7f8a341f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a33fff640 1 -- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a3415bb20 msgr2=0x7f8a3415a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a33fff640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a3415bb20 0x7f8a3415a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a33fff640 1 -- 172.21.15.59:0/3923212036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a341f14f0 con 0x7f8a340cd8c0 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a33fff640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a340cd8c0 0x7f8a341f0fb0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c034040 tx=0x7f8a1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a30ff9640 1 -- 172.21.15.59:0/3923212036 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c040070 con 0x7f8a340cd8c0 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a30ff9640 1 -- 172.21.15.59:0/3923212036 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c03a910 con 0x7f8a340cd8c0 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a30ff9640 1 -- 172.21.15.59:0/3923212036 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c035040 con 0x7f8a340cd8c0 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/3923212036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a341f4a50 con 0x7f8a340cd8c0 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.302+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/3923212036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3415fc00 con 0x7f8a340cd8c0 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.303+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/3923212036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3404f330 con 0x7f8a340cd8c0 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.303+0000 7f8a30ff9640 1 -- 172.21.15.59:0/3923212036 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a1c038020 con 0x7f8a340cd8c0 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.304+0000 7f8a30ff9640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a1406b500 0x7f8a1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.304+0000 7f8a30ff9640 1 -- 172.21.15.59:0/3923212036 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a1c0abb90 con 0x7f8a340cd8c0 2024-03-06T07:25:08.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.304+0000 7f8a337fe640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a1406b500 0x7f8a1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.305+0000 7f8a337fe640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a1406b500 0x7f8a1406d9c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8a340d1950 tx=0x7f8a24007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.308+0000 7f8a30ff9640 1 -- 172.21.15.59:0/3923212036 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a1c031800 con 0x7f8a340cd8c0 2024-03-06T07:25:08.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.487+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/3923212036 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a340d1e80 con 0x7f8a1406b500 2024-03-06T07:25:08.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.494+0000 7f8a30ff9640 1 -- 172.21.15.59:0/3923212036 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252026 (secure 0 0 0) 0x7f8a340d1e80 con 0x7f8a1406b500 2024-03-06T07:25:08.498 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:08.499 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:25:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a1406b500 msgr2=0x7f8a1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a1406b500 0x7f8a1406d9c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8a340d1950 tx=0x7f8a24007d50 comp rx=0 tx=0).stop 2024-03-06T07:25:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/3923212036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a340cd8c0 msgr2=0x7f8a341f0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a340cd8c0 0x7f8a341f0fb0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c034040 tx=0x7f8a1c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/3923212036 shutdown_connections 2024-03-06T07:25:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a1406b500 0x7f8a1406d9c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a3415bb20 0x7f8a3415a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340cfff0 0x7f8a341f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 --2- 172.21.15.59:0/3923212036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a340cd8c0 0x7f8a341f0fb0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/3923212036 >> 172.21.15.59:0/3923212036 conn(0x7f8a3409a9f0 msgr2=0x7f8a340ce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/3923212036 shutdown_connections 2024-03-06T07:25:08.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.497+0000 7f8a3a8fb640 1 -- 172.21.15.59:0/3923212036 wait complete. 2024-03-06T07:25:08.509 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-03-06T07:25:07.343488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257248,"kb_used_data":7784,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717472,"statfs":{"total":773094113280,"available":772830691328,"internally_reserved":0,"allocated":7970816,"data_stored":4006932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.198583"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176274+0000","last_change":"2024-03-06T07:25:05.172938+0000","last_active":"2024-03-06T07:25:05.176274+0000","last_peered":"2024-03-06T07:25:05.176274+0000","last_clean":"2024-03-06T07:25:05.176274+0000","last_became_active":"2024-03-06T07:25:05.172632+0000","last_became_peered":"2024-03-06T07:25:05.172632+0000","last_unstale":"2024-03-06T07:25:05.176274+0000","last_undegraded":"2024-03-06T07:25:05.176274+0000","last_fullsized":"2024-03-06T07:25:05.176274+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:45.258201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.185040+0000","last_change":"2024-03-06T07:25:05.177682+0000","last_active":"2024-03-06T07:25:05.185040+0000","last_peered":"2024-03-06T07:25:05.185040+0000","last_clean":"2024-03-06T07:25:05.185040+0000","last_became_active":"2024-03-06T07:25:05.177317+0000","last_became_peered":"2024-03-06T07:25:05.177317+0000","last_unstale":"2024-03-06T07:25:05.185040+0000","last_undegraded":"2024-03-06T07:25:05.185040+0000","last_fullsized":"2024-03-06T07:25:05.185040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:06.532171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156409+0000","last_change":"2024-03-06T07:25:05.168424+0000","last_active":"2024-03-06T07:25:06.156409+0000","last_peered":"2024-03-06T07:25:06.156409+0000","last_clean":"2024-03-06T07:25:06.156409+0000","last_became_active":"2024-03-06T07:25:05.167489+0000","last_became_peered":"2024-03-06T07:25:05.167489+0000","last_unstale":"2024-03-06T07:25:06.156409+0000","last_undegraded":"2024-03-06T07:25:06.156409+0000","last_fullsized":"2024-03-06T07:25:06.156409+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:14.728644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155459+0000","last_change":"2024-03-06T07:25:05.181199+0000","last_active":"2024-03-06T07:25:06.155459+0000","last_peered":"2024-03-06T07:25:06.155459+0000","last_clean":"2024-03-06T07:25:06.155459+0000","last_became_active":"2024-03-06T07:25:05.180890+0000","last_became_peered":"2024-03-06T07:25:05.180890+0000","last_unstale":"2024-03-06T07:25:06.155459+0000","last_undegraded":"2024-03-06T07:25:06.155459+0000","last_fullsized":"2024-03-06T07:25:06.155459+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:53.479454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176253+0000","last_change":"2024-03-06T07:25:05.171737+0000","last_active":"2024-03-06T07:25:05.176253+0000","last_peered":"2024-03-06T07:25:05.176253+0000","last_clean":"2024-03-06T07:25:05.176253+0000","last_became_active":"2024-03-06T07:25:05.171458+0000","last_became_peered":"2024-03-06T07:25:05.171458+0000","last_unstale":"2024-03-06T07:25:05.176253+0000","last_undegraded":"2024-03-06T07:25:05.176253+0000","last_fullsized":"2024-03-06T07:25:05.176253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:17:06.862819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.178281+0000","last_change":"2024-03-06T07:25:05.174102+0000","last_active":"2024-03-06T07:25:05.178281+0000","last_peered":"2024-03-06T07:25:05.178281+0000","last_clean":"2024-03-06T07:25:05.178281+0000","last_became_active":"2024-03-06T07:25:05.173840+0000","last_became_peered":"2024-03-06T07:25:05.173840+0000","last_unstale":"2024-03-06T07:25:05.178281+0000","last_undegraded":"2024-03-06T07:25:05.178281+0000","last_fullsized":"2024-03-06T07:25:05.178281+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:32.423709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156897+0000","last_change":"2024-03-06T07:25:05.171597+0000","last_active":"2024-03-06T07:25:06.156897+0000","last_peered":"2024-03-06T07:25:06.156897+0000","last_clean":"2024-03-06T07:25:06.156897+0000","last_became_active":"2024-03-06T07:25:05.171326+0000","last_became_peered":"2024-03-06T07:25:05.171326+0000","last_unstale":"2024-03-06T07:25:06.156897+0000","last_undegraded":"2024-03-06T07:25:06.156897+0000","last_fullsized":"2024-03-06T07:25:06.156897+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:04.740636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187632+0000","last_change":"2024-03-06T07:25:05.182681+0000","last_active":"2024-03-06T07:25:05.187632+0000","last_peered":"2024-03-06T07:25:05.187632+0000","last_clean":"2024-03-06T07:25:05.187632+0000","last_became_active":"2024-03-06T07:25:05.182392+0000","last_became_peered":"2024-03-06T07:25:05.182392+0000","last_unstale":"2024-03-06T07:25:05.187632+0000","last_undegraded":"2024-03-06T07:25:05.187632+0000","last_fullsized":"2024-03-06T07:25:05.187632+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:41.996337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176159+0000","last_change":"2024-03-06T07:25:05.168655+0000","last_active":"2024-03-06T07:25:05.176159+0000","last_peered":"2024-03-06T07:25:05.176159+0000","last_clean":"2024-03-06T07:25:05.176159+0000","last_became_active":"2024-03-06T07:25:05.168154+0000","last_became_peered":"2024-03-06T07:25:05.168154+0000","last_unstale":"2024-03-06T07:25:05.176159+0000","last_undegraded":"2024-03-06T07:25:05.176159+0000","last_fullsized":"2024-03-06T07:25:05.176159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:33:07.277601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179991+0000","last_change":"2024-03-06T07:25:05.174638+0000","last_active":"2024-03-06T07:25:05.179991+0000","last_peered":"2024-03-06T07:25:05.179991+0000","last_clean":"2024-03-06T07:25:05.179991+0000","last_became_active":"2024-03-06T07:25:05.174336+0000","last_became_peered":"2024-03-06T07:25:05.174336+0000","last_unstale":"2024-03-06T07:25:05.179991+0000","last_undegraded":"2024-03-06T07:25:05.179991+0000","last_fullsized":"2024-03-06T07:25:05.179991+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:42.027393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.173919+0000","last_change":"2024-03-06T07:25:05.170458+0000","last_active":"2024-03-06T07:25:05.173919+0000","last_peered":"2024-03-06T07:25:05.173919+0000","last_clean":"2024-03-06T07:25:05.173919+0000","last_became_active":"2024-03-06T07:25:05.170193+0000","last_became_peered":"2024-03-06T07:25:05.170193+0000","last_unstale":"2024-03-06T07:25:05.173919+0000","last_undegraded":"2024-03-06T07:25:05.173919+0000","last_fullsized":"2024-03-06T07:25:05.173919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:55.867315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155653+0000","last_change":"2024-03-06T07:25:05.176864+0000","last_active":"2024-03-06T07:25:06.155653+0000","last_peered":"2024-03-06T07:25:06.155653+0000","last_clean":"2024-03-06T07:25:06.155653+0000","last_became_active":"2024-03-06T07:25:05.176577+0000","last_became_peered":"2024-03-06T07:25:05.176577+0000","last_unstale":"2024-03-06T07:25:06.155653+0000","last_undegraded":"2024-03-06T07:25:06.155653+0000","last_fullsized":"2024-03-06T07:25:06.155653+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:16.371492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172765+0000","last_change":"2024-03-06T07:25:05.175851+0000","last_active":"2024-03-06T07:25:06.172765+0000","last_peered":"2024-03-06T07:25:06.172765+0000","last_clean":"2024-03-06T07:25:06.172765+0000","last_became_active":"2024-03-06T07:25:05.175502+0000","last_became_peered":"2024-03-06T07:25:05.175502+0000","last_unstale":"2024-03-06T07:25:06.172765+0000","last_undegraded":"2024-03-06T07:25:06.172765+0000","last_fullsized":"2024-03-06T07:25:06.172765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:52.667362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.169510+0000","last_change":"2024-03-06T07:25:05.164708+0000","last_active":"2024-03-06T07:25:05.169510+0000","last_peered":"2024-03-06T07:25:05.169510+0000","last_clean":"2024-03-06T07:25:05.169510+0000","last_became_active":"2024-03-06T07:25:05.164375+0000","last_became_peered":"2024-03-06T07:25:05.164375+0000","last_unstale":"2024-03-06T07:25:05.169510+0000","last_undegraded":"2024-03-06T07:25:05.169510+0000","last_fullsized":"2024-03-06T07:25:05.169510+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:10.232525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172604+0000","last_change":"2024-03-06T07:25:05.170080+0000","last_active":"2024-03-06T07:25:06.172604+0000","last_peered":"2024-03-06T07:25:06.172604+0000","last_clean":"2024-03-06T07:25:06.172604+0000","last_became_active":"2024-03-06T07:25:05.169816+0000","last_became_peered":"2024-03-06T07:25:05.169816+0000","last_unstale":"2024-03-06T07:25:06.172604+0000","last_undegraded":"2024-03-06T07:25:06.172604+0000","last_fullsized":"2024-03-06T07:25:06.172604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:34.634891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.181163+0000","last_change":"2024-03-06T07:25:05.176069+0000","last_active":"2024-03-06T07:25:05.181163+0000","last_peered":"2024-03-06T07:25:05.181163+0000","last_clean":"2024-03-06T07:25:05.181163+0000","last_became_active":"2024-03-06T07:25:05.175585+0000","last_became_peered":"2024-03-06T07:25:05.175585+0000","last_unstale":"2024-03-06T07:25:05.181163+0000","last_undegraded":"2024-03-06T07:25:05.181163+0000","last_fullsized":"2024-03-06T07:25:05.181163+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:53.479454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176135+0000","last_change":"2024-03-06T07:25:05.168538+0000","last_active":"2024-03-06T07:25:05.176135+0000","last_peered":"2024-03-06T07:25:05.176135+0000","last_clean":"2024-03-06T07:25:05.176135+0000","last_became_active":"2024-03-06T07:25:05.167805+0000","last_became_peered":"2024-03-06T07:25:05.167805+0000","last_unstale":"2024-03-06T07:25:05.176135+0000","last_undegraded":"2024-03-06T07:25:05.176135+0000","last_fullsized":"2024-03-06T07:25:05.176135+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:21.185865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155709+0000","last_change":"2024-03-06T07:25:05.181216+0000","last_active":"2024-03-06T07:25:06.155709+0000","last_peered":"2024-03-06T07:25:06.155709+0000","last_clean":"2024-03-06T07:25:06.155709+0000","last_became_active":"2024-03-06T07:25:05.180875+0000","last_became_peered":"2024-03-06T07:25:05.180875+0000","last_unstale":"2024-03-06T07:25:06.155709+0000","last_undegraded":"2024-03-06T07:25:06.155709+0000","last_fullsized":"2024-03-06T07:25:06.155709+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:55.867315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187768+0000","last_change":"2024-03-06T07:25:05.183636+0000","last_active":"2024-03-06T07:25:05.187768+0000","last_peered":"2024-03-06T07:25:05.187768+0000","last_clean":"2024-03-06T07:25:05.187768+0000","last_became_active":"2024-03-06T07:25:05.183333+0000","last_became_peered":"2024-03-06T07:25:05.183333+0000","last_unstale":"2024-03-06T07:25:05.187768+0000","last_undegraded":"2024-03-06T07:25:05.187768+0000","last_fullsized":"2024-03-06T07:25:05.187768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:04.584017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172788+0000","last_change":"2024-03-06T07:25:05.165481+0000","last_active":"2024-03-06T07:25:06.172788+0000","last_peered":"2024-03-06T07:25:06.172788+0000","last_clean":"2024-03-06T07:25:06.172788+0000","last_became_active":"2024-03-06T07:25:05.165073+0000","last_became_peered":"2024-03-06T07:25:05.165073+0000","last_unstale":"2024-03-06T07:25:06.172788+0000","last_undegraded":"2024-03-06T07:25:06.172788+0000","last_fullsized":"2024-03-06T07:25:06.172788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:26.601256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.174837+0000","last_change":"2024-03-06T07:25:05.168055+0000","last_active":"2024-03-06T07:25:05.174837+0000","last_peered":"2024-03-06T07:25:05.174837+0000","last_clean":"2024-03-06T07:25:05.174837+0000","last_became_active":"2024-03-06T07:25:05.167764+0000","last_became_peered":"2024-03-06T07:25:05.167764+0000","last_unstale":"2024-03-06T07:25:05.174837+0000","last_undegraded":"2024-03-06T07:25:05.174837+0000","last_fullsized":"2024-03-06T07:25:05.174837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:18.044770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179727+0000","last_change":"2024-03-06T07:25:05.174182+0000","last_active":"2024-03-06T07:25:05.179727+0000","last_peered":"2024-03-06T07:25:05.179727+0000","last_clean":"2024-03-06T07:25:05.179727+0000","last_became_active":"2024-03-06T07:25:05.173885+0000","last_became_peered":"2024-03-06T07:25:05.173885+0000","last_unstale":"2024-03-06T07:25:05.179727+0000","last_undegraded":"2024-03-06T07:25:05.179727+0000","last_fullsized":"2024-03-06T07:25:05.179727+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:48.608381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.182055+0000","last_change":"2024-03-06T07:25:05.176762+0000","last_active":"2024-03-06T07:25:05.182055+0000","last_peered":"2024-03-06T07:25:05.182055+0000","last_clean":"2024-03-06T07:25:05.182055+0000","last_became_active":"2024-03-06T07:25:05.176507+0000","last_became_peered":"2024-03-06T07:25:05.176507+0000","last_unstale":"2024-03-06T07:25:05.182055+0000","last_undegraded":"2024-03-06T07:25:05.182055+0000","last_fullsized":"2024-03-06T07:25:05.182055+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:07.379633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187508+0000","last_change":"2024-03-06T07:25:05.181937+0000","last_active":"2024-03-06T07:25:05.187508+0000","last_peered":"2024-03-06T07:25:05.187508+0000","last_clean":"2024-03-06T07:25:05.187508+0000","last_became_active":"2024-03-06T07:25:05.181676+0000","last_became_peered":"2024-03-06T07:25:05.181676+0000","last_unstale":"2024-03-06T07:25:05.187508+0000","last_undegraded":"2024-03-06T07:25:05.187508+0000","last_fullsized":"2024-03-06T07:25:05.187508+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:43.727360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179999+0000","last_change":"2024-03-06T07:25:05.174088+0000","last_active":"2024-03-06T07:25:05.179999+0000","last_peered":"2024-03-06T07:25:05.179999+0000","last_clean":"2024-03-06T07:25:05.179999+0000","last_became_active":"2024-03-06T07:25:05.173799+0000","last_became_peered":"2024-03-06T07:25:05.173799+0000","last_unstale":"2024-03-06T07:25:05.179999+0000","last_undegraded":"2024-03-06T07:25:05.179999+0000","last_fullsized":"2024-03-06T07:25:05.179999+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:48.671903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156730+0000","last_change":"2024-03-06T07:25:05.175328+0000","last_active":"2024-03-06T07:25:06.156730+0000","last_peered":"2024-03-06T07:25:06.156730+0000","last_clean":"2024-03-06T07:25:06.156730+0000","last_became_active":"2024-03-06T07:25:05.175035+0000","last_became_peered":"2024-03-06T07:25:05.175035+0000","last_unstale":"2024-03-06T07:25:06.156730+0000","last_undegraded":"2024-03-06T07:25:06.156730+0000","last_fullsized":"2024-03-06T07:25:06.156730+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:38.802280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156373+0000","last_change":"2024-03-06T07:25:05.173245+0000","last_active":"2024-03-06T07:25:06.156373+0000","last_peered":"2024-03-06T07:25:06.156373+0000","last_clean":"2024-03-06T07:25:06.156373+0000","last_became_active":"2024-03-06T07:25:05.172920+0000","last_became_peered":"2024-03-06T07:25:05.172920+0000","last_unstale":"2024-03-06T07:25:06.156373+0000","last_undegraded":"2024-03-06T07:25:06.156373+0000","last_fullsized":"2024-03-06T07:25:06.156373+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:44.912192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179938+0000","last_change":"2024-03-06T07:25:05.174748+0000","last_active":"2024-03-06T07:25:05.179938+0000","last_peered":"2024-03-06T07:25:05.179938+0000","last_clean":"2024-03-06T07:25:05.179938+0000","last_became_active":"2024-03-06T07:25:05.174493+0000","last_became_peered":"2024-03-06T07:25:05.174493+0000","last_unstale":"2024-03-06T07:25:05.179938+0000","last_undegraded":"2024-03-06T07:25:05.179938+0000","last_fullsized":"2024-03-06T07:25:05.179938+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:38.868144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156031+0000","last_change":"2024-03-06T07:25:05.168584+0000","last_active":"2024-03-06T07:25:06.156031+0000","last_peered":"2024-03-06T07:25:06.156031+0000","last_clean":"2024-03-06T07:25:06.156031+0000","last_became_active":"2024-03-06T07:25:05.168291+0000","last_became_peered":"2024-03-06T07:25:05.168291+0000","last_unstale":"2024-03-06T07:25:06.156031+0000","last_undegraded":"2024-03-06T07:25:06.156031+0000","last_fullsized":"2024-03-06T07:25:06.156031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:02.657258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156621+0000","last_change":"2024-03-06T07:25:05.174070+0000","last_active":"2024-03-06T07:25:06.156621+0000","last_peered":"2024-03-06T07:25:06.156621+0000","last_clean":"2024-03-06T07:25:06.156621+0000","last_became_active":"2024-03-06T07:25:05.173795+0000","last_became_peered":"2024-03-06T07:25:05.173795+0000","last_unstale":"2024-03-06T07:25:06.156621+0000","last_undegraded":"2024-03-06T07:25:06.156621+0000","last_fullsized":"2024-03-06T07:25:06.156621+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:10.690693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156751+0000","last_change":"2024-03-06T07:25:05.170463+0000","last_active":"2024-03-06T07:25:06.156751+0000","last_peered":"2024-03-06T07:25:06.156751+0000","last_clean":"2024-03-06T07:25:06.156751+0000","last_became_active":"2024-03-06T07:25:05.170199+0000","last_became_peered":"2024-03-06T07:25:05.170199+0000","last_unstale":"2024-03-06T07:25:06.156751+0000","last_undegraded":"2024-03-06T07:25:06.156751+0000","last_fullsized":"2024-03-06T07:25:06.156751+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:52.667362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187547+0000","last_change":"2024-03-06T07:25:05.180187+0000","last_active":"2024-03-06T07:25:05.187547+0000","last_peered":"2024-03-06T07:25:05.187547+0000","last_clean":"2024-03-06T07:25:05.187547+0000","last_became_active":"2024-03-06T07:25:05.179807+0000","last_became_peered":"2024-03-06T07:25:05.179807+0000","last_unstale":"2024-03-06T07:25:05.187547+0000","last_undegraded":"2024-03-06T07:25:05.187547+0000","last_fullsized":"2024-03-06T07:25:05.187547+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:42.254969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.184822+0000","last_change":"2024-03-06T07:25:05.180668+0000","last_active":"2024-03-06T07:25:05.184822+0000","last_peered":"2024-03-06T07:25:05.184822+0000","last_clean":"2024-03-06T07:25:05.184822+0000","last_became_active":"2024-03-06T07:25:05.180435+0000","last_became_peered":"2024-03-06T07:25:05.180435+0000","last_unstale":"2024-03-06T07:25:05.184822+0000","last_undegraded":"2024-03-06T07:25:05.184822+0000","last_fullsized":"2024-03-06T07:25:05.184822+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:07:52.834751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"69'4","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.182218+0000","last_change":"2024-03-06T07:25:04.171525+0000","last_active":"2024-03-06T07:25:05.182218+0000","last_peered":"2024-03-06T07:25:05.182218+0000","last_clean":"2024-03-06T07:25:05.182218+0000","last_became_active":"2024-03-06T07:25:04.171243+0000","last_became_peered":"2024-03-06T07:25:04.171243+0000","last_unstale":"2024-03-06T07:25:05.182218+0000","last_undegraded":"2024-03-06T07:25:05.182218+0000","last_fullsized":"2024-03-06T07:25:05.182218+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:06.145388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.173937+0000","last_change":"2024-03-06T07:25:05.168417+0000","last_active":"2024-03-06T07:25:05.173937+0000","last_peered":"2024-03-06T07:25:05.173937+0000","last_clean":"2024-03-06T07:25:05.173937+0000","last_became_active":"2024-03-06T07:25:05.167429+0000","last_became_peered":"2024-03-06T07:25:05.167429+0000","last_unstale":"2024-03-06T07:25:05.173937+0000","last_undegraded":"2024-03-06T07:25:05.173937+0000","last_fullsized":"2024-03-06T07:25:05.173937+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:26.082546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172845+0000","last_change":"2024-03-06T07:25:04.165893+0000","last_active":"2024-03-06T07:25:06.172845+0000","last_peered":"2024-03-06T07:25:06.172845+0000","last_clean":"2024-03-06T07:25:06.172845+0000","last_became_active":"2024-03-06T07:25:04.165612+0000","last_became_peered":"2024-03-06T07:25:04.165612+0000","last_unstale":"2024-03-06T07:25:06.172845+0000","last_undegraded":"2024-03-06T07:25:06.172845+0000","last_fullsized":"2024-03-06T07:25:06.172845+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:54.865160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187826+0000","last_change":"2024-03-06T07:25:05.184206+0000","last_active":"2024-03-06T07:25:05.187826+0000","last_peered":"2024-03-06T07:25:05.187826+0000","last_clean":"2024-03-06T07:25:05.187826+0000","last_became_active":"2024-03-06T07:25:05.183741+0000","last_became_peered":"2024-03-06T07:25:05.183741+0000","last_unstale":"2024-03-06T07:25:05.187826+0000","last_undegraded":"2024-03-06T07:25:05.187826+0000","last_fullsized":"2024-03-06T07:25:05.187826+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:30.480318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.180251+0000","last_change":"2024-03-06T07:25:04.168953+0000","last_active":"2024-03-06T07:25:05.180251+0000","last_peered":"2024-03-06T07:25:05.180251+0000","last_clean":"2024-03-06T07:25:05.180251+0000","last_became_active":"2024-03-06T07:25:04.168671+0000","last_became_peered":"2024-03-06T07:25:04.168671+0000","last_unstale":"2024-03-06T07:25:05.180251+0000","last_undegraded":"2024-03-06T07:25:05.180251+0000","last_fullsized":"2024-03-06T07:25:05.180251+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:42.546597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179277+0000","last_change":"2024-03-06T07:25:05.173770+0000","last_active":"2024-03-06T07:25:05.179277+0000","last_peered":"2024-03-06T07:25:05.179277+0000","last_clean":"2024-03-06T07:25:05.179277+0000","last_became_active":"2024-03-06T07:25:05.173515+0000","last_became_peered":"2024-03-06T07:25:05.173515+0000","last_unstale":"2024-03-06T07:25:05.179277+0000","last_undegraded":"2024-03-06T07:25:05.179277+0000","last_fullsized":"2024-03-06T07:25:05.179277+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:26.498743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.183271+0000","last_change":"2024-03-06T07:25:04.172181+0000","last_active":"2024-03-06T07:25:05.183271+0000","last_peered":"2024-03-06T07:25:05.183271+0000","last_clean":"2024-03-06T07:25:05.183271+0000","last_became_active":"2024-03-06T07:25:04.171880+0000","last_became_peered":"2024-03-06T07:25:04.171880+0000","last_unstale":"2024-03-06T07:25:05.183271+0000","last_undegraded":"2024-03-06T07:25:05.183271+0000","last_fullsized":"2024-03-06T07:25:05.183271+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:14.594216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156644+0000","last_change":"2024-03-06T07:25:05.167977+0000","last_active":"2024-03-06T07:25:06.156644+0000","last_peered":"2024-03-06T07:25:06.156644+0000","last_clean":"2024-03-06T07:25:06.156644+0000","last_became_active":"2024-03-06T07:25:05.167376+0000","last_became_peered":"2024-03-06T07:25:05.167376+0000","last_unstale":"2024-03-06T07:25:06.156644+0000","last_undegraded":"2024-03-06T07:25:06.156644+0000","last_fullsized":"2024-03-06T07:25:06.156644+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:43.727360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156129+0000","last_change":"2024-03-06T07:25:04.166968+0000","last_active":"2024-03-06T07:25:06.156129+0000","last_peered":"2024-03-06T07:25:06.156129+0000","last_clean":"2024-03-06T07:25:06.156129+0000","last_became_active":"2024-03-06T07:25:04.166438+0000","last_became_peered":"2024-03-06T07:25:04.166438+0000","last_unstale":"2024-03-06T07:25:06.156129+0000","last_undegraded":"2024-03-06T07:25:06.156129+0000","last_fullsized":"2024-03-06T07:25:06.156129+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:18.530284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179854+0000","last_change":"2024-03-06T07:25:05.174645+0000","last_active":"2024-03-06T07:25:05.179854+0000","last_peered":"2024-03-06T07:25:05.179854+0000","last_clean":"2024-03-06T07:25:05.179854+0000","last_became_active":"2024-03-06T07:25:05.173522+0000","last_became_peered":"2024-03-06T07:25:05.173522+0000","last_unstale":"2024-03-06T07:25:05.179854+0000","last_undegraded":"2024-03-06T07:25:05.179854+0000","last_fullsized":"2024-03-06T07:25:05.179854+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:18.628420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179237+0000","last_change":"2024-03-06T07:25:04.172436+0000","last_active":"2024-03-06T07:25:05.179237+0000","last_peered":"2024-03-06T07:25:05.179237+0000","last_clean":"2024-03-06T07:25:05.179237+0000","last_became_active":"2024-03-06T07:25:04.172154+0000","last_became_peered":"2024-03-06T07:25:04.172154+0000","last_unstale":"2024-03-06T07:25:05.179237+0000","last_undegraded":"2024-03-06T07:25:05.179237+0000","last_fullsized":"2024-03-06T07:25:05.179237+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:14.362185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.169290+0000","last_change":"2024-03-06T07:25:05.162264+0000","last_active":"2024-03-06T07:25:05.169290+0000","last_peered":"2024-03-06T07:25:05.169290+0000","last_clean":"2024-03-06T07:25:05.169290+0000","last_became_active":"2024-03-06T07:25:05.161977+0000","last_became_peered":"2024-03-06T07:25:05.161977+0000","last_unstale":"2024-03-06T07:25:05.169290+0000","last_undegraded":"2024-03-06T07:25:05.169290+0000","last_fullsized":"2024-03-06T07:25:05.169290+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:02.761359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155285+0000","last_change":"2024-03-06T07:25:04.171825+0000","last_active":"2024-03-06T07:25:06.155285+0000","last_peered":"2024-03-06T07:25:06.155285+0000","last_clean":"2024-03-06T07:25:06.155285+0000","last_became_active":"2024-03-06T07:25:04.171576+0000","last_became_peered":"2024-03-06T07:25:04.171576+0000","last_unstale":"2024-03-06T07:25:06.155285+0000","last_undegraded":"2024-03-06T07:25:06.155285+0000","last_fullsized":"2024-03-06T07:25:06.155285+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:10.525330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172564+0000","last_change":"2024-03-06T07:25:05.171716+0000","last_active":"2024-03-06T07:25:06.172564+0000","last_peered":"2024-03-06T07:25:06.172564+0000","last_clean":"2024-03-06T07:25:06.172564+0000","last_became_active":"2024-03-06T07:25:05.171449+0000","last_became_peered":"2024-03-06T07:25:05.171449+0000","last_unstale":"2024-03-06T07:25:06.172564+0000","last_undegraded":"2024-03-06T07:25:06.172564+0000","last_fullsized":"2024-03-06T07:25:06.172564+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:38.802280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.155809+0000","last_change":"2024-03-06T07:25:04.163693+0000","last_active":"2024-03-06T07:25:05.155809+0000","last_peered":"2024-03-06T07:25:05.155809+0000","last_clean":"2024-03-06T07:25:05.155809+0000","last_became_active":"2024-03-06T07:25:04.162812+0000","last_became_peered":"2024-03-06T07:25:04.162812+0000","last_unstale":"2024-03-06T07:25:05.155809+0000","last_undegraded":"2024-03-06T07:25:05.155809+0000","last_fullsized":"2024-03-06T07:25:05.155809+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:53.039314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156864+0000","last_change":"2024-03-06T07:25:05.172330+0000","last_active":"2024-03-06T07:25:06.156864+0000","last_peered":"2024-03-06T07:25:06.156864+0000","last_clean":"2024-03-06T07:25:06.156864+0000","last_became_active":"2024-03-06T07:25:05.172071+0000","last_became_peered":"2024-03-06T07:25:05.172071+0000","last_unstale":"2024-03-06T07:25:06.156864+0000","last_undegraded":"2024-03-06T07:25:06.156864+0000","last_fullsized":"2024-03-06T07:25:06.156864+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:15.620644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.158825+0000","last_change":"2024-03-06T07:25:04.162855+0000","last_active":"2024-03-06T07:25:05.158825+0000","last_peered":"2024-03-06T07:25:05.158825+0000","last_clean":"2024-03-06T07:25:05.158825+0000","last_became_active":"2024-03-06T07:25:04.162574+0000","last_became_peered":"2024-03-06T07:25:04.162574+0000","last_unstale":"2024-03-06T07:25:05.158825+0000","last_undegraded":"2024-03-06T07:25:05.158825+0000","last_fullsized":"2024-03-06T07:25:05.158825+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:17.933958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.184900+0000","last_change":"2024-03-06T07:25:05.181471+0000","last_active":"2024-03-06T07:25:05.184900+0000","last_peered":"2024-03-06T07:25:05.184900+0000","last_clean":"2024-03-06T07:25:05.184900+0000","last_became_active":"2024-03-06T07:25:05.181208+0000","last_became_peered":"2024-03-06T07:25:05.181208+0000","last_unstale":"2024-03-06T07:25:05.184900+0000","last_undegraded":"2024-03-06T07:25:05.184900+0000","last_fullsized":"2024-03-06T07:25:05.184900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:10.690693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.169692+0000","last_change":"2024-03-06T07:25:04.166491+0000","last_active":"2024-03-06T07:25:05.169692+0000","last_peered":"2024-03-06T07:25:05.169692+0000","last_clean":"2024-03-06T07:25:05.169692+0000","last_became_active":"2024-03-06T07:25:04.166179+0000","last_became_peered":"2024-03-06T07:25:04.166179+0000","last_unstale":"2024-03-06T07:25:05.169692+0000","last_undegraded":"2024-03-06T07:25:05.169692+0000","last_fullsized":"2024-03-06T07:25:05.169692+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:00.885206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.184899+0000","last_change":"2024-03-06T07:25:05.177728+0000","last_active":"2024-03-06T07:25:05.184899+0000","last_peered":"2024-03-06T07:25:05.184899+0000","last_clean":"2024-03-06T07:25:05.184899+0000","last_became_active":"2024-03-06T07:25:05.177466+0000","last_became_peered":"2024-03-06T07:25:05.177466+0000","last_unstale":"2024-03-06T07:25:05.184899+0000","last_undegraded":"2024-03-06T07:25:05.184899+0000","last_fullsized":"2024-03-06T07:25:05.184899+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:17.300384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"69'3","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156432+0000","last_change":"2024-03-06T07:25:04.158336+0000","last_active":"2024-03-06T07:25:06.156432+0000","last_peered":"2024-03-06T07:25:06.156432+0000","last_clean":"2024-03-06T07:25:06.156432+0000","last_became_active":"2024-03-06T07:25:04.158042+0000","last_became_peered":"2024-03-06T07:25:04.158042+0000","last_unstale":"2024-03-06T07:25:06.156432+0000","last_undegraded":"2024-03-06T07:25:06.156432+0000","last_fullsized":"2024-03-06T07:25:06.156432+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:50.416138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176227+0000","last_change":"2024-03-06T07:25:05.169972+0000","last_active":"2024-03-06T07:25:05.176227+0000","last_peered":"2024-03-06T07:25:05.176227+0000","last_clean":"2024-03-06T07:25:05.176227+0000","last_became_active":"2024-03-06T07:25:05.169690+0000","last_became_peered":"2024-03-06T07:25:05.169690+0000","last_unstale":"2024-03-06T07:25:05.176227+0000","last_undegraded":"2024-03-06T07:25:05.176227+0000","last_fullsized":"2024-03-06T07:25:05.176227+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:30.541732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"69'6","reported_seq":20,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155749+0000","last_change":"2024-03-06T07:25:04.170811+0000","last_active":"2024-03-06T07:25:06.155749+0000","last_peered":"2024-03-06T07:25:06.155749+0000","last_clean":"2024-03-06T07:25:06.155749+0000","last_became_active":"2024-03-06T07:25:04.170480+0000","last_became_peered":"2024-03-06T07:25:04.170480+0000","last_unstale":"2024-03-06T07:25:06.155749+0000","last_undegraded":"2024-03-06T07:25:06.155749+0000","last_fullsized":"2024-03-06T07:25:06.155749+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:21.745380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.173078+0000","last_change":"2024-03-06T07:25:05.172095+0000","last_active":"2024-03-06T07:25:05.173078+0000","last_peered":"2024-03-06T07:25:05.173078+0000","last_clean":"2024-03-06T07:25:05.173078+0000","last_became_active":"2024-03-06T07:25:05.171834+0000","last_became_peered":"2024-03-06T07:25:05.171834+0000","last_unstale":"2024-03-06T07:25:05.173078+0000","last_undegraded":"2024-03-06T07:25:05.173078+0000","last_fullsized":"2024-03-06T07:25:05.173078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:29.527566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155438+0000","last_change":"2024-03-06T07:25:04.172900+0000","last_active":"2024-03-06T07:25:06.155438+0000","last_peered":"2024-03-06T07:25:06.155438+0000","last_clean":"2024-03-06T07:25:06.155438+0000","last_became_active":"2024-03-06T07:25:04.172598+0000","last_became_peered":"2024-03-06T07:25:04.172598+0000","last_unstale":"2024-03-06T07:25:06.155438+0000","last_undegraded":"2024-03-06T07:25:06.155438+0000","last_fullsized":"2024-03-06T07:25:06.155438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:17.933958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156824+0000","last_change":"2024-03-06T07:25:05.176530+0000","last_active":"2024-03-06T07:25:06.156824+0000","last_peered":"2024-03-06T07:25:06.156824+0000","last_clean":"2024-03-06T07:25:06.156824+0000","last_became_active":"2024-03-06T07:25:05.176242+0000","last_became_peered":"2024-03-06T07:25:05.176242+0000","last_unstale":"2024-03-06T07:25:06.156824+0000","last_undegraded":"2024-03-06T07:25:06.156824+0000","last_fullsized":"2024-03-06T07:25:06.156824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:26.601256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.165525+0000","last_change":"2024-03-06T07:25:04.168096+0000","last_active":"2024-03-06T07:25:05.165525+0000","last_peered":"2024-03-06T07:25:05.165525+0000","last_clean":"2024-03-06T07:25:05.165525+0000","last_became_active":"2024-03-06T07:25:04.167649+0000","last_became_peered":"2024-03-06T07:25:04.167649+0000","last_unstale":"2024-03-06T07:25:05.165525+0000","last_undegraded":"2024-03-06T07:25:05.165525+0000","last_fullsized":"2024-03-06T07:25:05.165525+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:02.250812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172562+0000","last_change":"2024-03-06T07:25:05.175345+0000","last_active":"2024-03-06T07:25:06.172562+0000","last_peered":"2024-03-06T07:25:06.172562+0000","last_clean":"2024-03-06T07:25:06.172562+0000","last_became_active":"2024-03-06T07:25:05.175071+0000","last_became_peered":"2024-03-06T07:25:05.175071+0000","last_unstale":"2024-03-06T07:25:06.172562+0000","last_undegraded":"2024-03-06T07:25:06.172562+0000","last_fullsized":"2024-03-06T07:25:06.172562+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:01:43.727360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156471+0000","last_change":"2024-03-06T07:25:04.166090+0000","last_active":"2024-03-06T07:25:06.156471+0000","last_peered":"2024-03-06T07:25:06.156471+0000","last_clean":"2024-03-06T07:25:06.156471+0000","last_became_active":"2024-03-06T07:25:04.165783+0000","last_became_peered":"2024-03-06T07:25:04.165783+0000","last_unstale":"2024-03-06T07:25:06.156471+0000","last_undegraded":"2024-03-06T07:25:06.156471+0000","last_fullsized":"2024-03-06T07:25:06.156471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:15.369337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.169310+0000","last_change":"2024-03-06T07:25:05.166736+0000","last_active":"2024-03-06T07:25:05.169310+0000","last_peered":"2024-03-06T07:25:05.169310+0000","last_clean":"2024-03-06T07:25:05.169310+0000","last_became_active":"2024-03-06T07:25:05.166321+0000","last_became_peered":"2024-03-06T07:25:05.166321+0000","last_unstale":"2024-03-06T07:25:05.169310+0000","last_undegraded":"2024-03-06T07:25:05.169310+0000","last_fullsized":"2024-03-06T07:25:05.169310+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:59.961820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"69'8","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.177826+0000","last_change":"2024-03-06T07:25:04.168166+0000","last_active":"2024-03-06T07:25:05.177826+0000","last_peered":"2024-03-06T07:25:05.177826+0000","last_clean":"2024-03-06T07:25:05.177826+0000","last_became_active":"2024-03-06T07:25:04.167871+0000","last_became_peered":"2024-03-06T07:25:04.167871+0000","last_unstale":"2024-03-06T07:25:05.177826+0000","last_undegraded":"2024-03-06T07:25:05.177826+0000","last_fullsized":"2024-03-06T07:25:05.177826+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:10.525330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.183838+0000","last_change":"2024-03-06T07:25:05.180594+0000","last_active":"2024-03-06T07:25:05.183838+0000","last_peered":"2024-03-06T07:25:05.183838+0000","last_clean":"2024-03-06T07:25:05.183838+0000","last_became_active":"2024-03-06T07:25:05.180296+0000","last_became_peered":"2024-03-06T07:25:05.180296+0000","last_unstale":"2024-03-06T07:25:05.183838+0000","last_undegraded":"2024-03-06T07:25:05.183838+0000","last_fullsized":"2024-03-06T07:25:05.183838+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:16.371492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179978+0000","last_change":"2024-03-06T07:25:04.170917+0000","last_active":"2024-03-06T07:25:05.179978+0000","last_peered":"2024-03-06T07:25:05.179978+0000","last_clean":"2024-03-06T07:25:05.179978+0000","last_became_active":"2024-03-06T07:25:04.170632+0000","last_became_peered":"2024-03-06T07:25:04.170632+0000","last_unstale":"2024-03-06T07:25:05.179978+0000","last_undegraded":"2024-03-06T07:25:05.179978+0000","last_fullsized":"2024-03-06T07:25:05.179978+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:40:27.124759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.184884+0000","last_change":"2024-03-06T07:25:05.180901+0000","last_active":"2024-03-06T07:25:05.184884+0000","last_peered":"2024-03-06T07:25:05.184884+0000","last_clean":"2024-03-06T07:25:05.184884+0000","last_became_active":"2024-03-06T07:25:05.180660+0000","last_became_peered":"2024-03-06T07:25:05.180660+0000","last_unstale":"2024-03-06T07:25:05.184884+0000","last_undegraded":"2024-03-06T07:25:05.184884+0000","last_fullsized":"2024-03-06T07:25:05.184884+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:30.604907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179634+0000","last_change":"2024-03-06T07:25:04.171879+0000","last_active":"2024-03-06T07:25:05.179634+0000","last_peered":"2024-03-06T07:25:05.179634+0000","last_clean":"2024-03-06T07:25:05.179634+0000","last_became_active":"2024-03-06T07:25:04.171605+0000","last_became_peered":"2024-03-06T07:25:04.171605+0000","last_unstale":"2024-03-06T07:25:05.179634+0000","last_undegraded":"2024-03-06T07:25:05.179634+0000","last_fullsized":"2024-03-06T07:25:05.179634+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:21.745380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155679+0000","last_change":"2024-03-06T07:25:05.175642+0000","last_active":"2024-03-06T07:25:06.155679+0000","last_peered":"2024-03-06T07:25:06.155679+0000","last_clean":"2024-03-06T07:25:06.155679+0000","last_became_active":"2024-03-06T07:25:05.175303+0000","last_became_peered":"2024-03-06T07:25:05.175303+0000","last_unstale":"2024-03-06T07:25:06.155679+0000","last_undegraded":"2024-03-06T07:25:06.155679+0000","last_fullsized":"2024-03-06T07:25:06.155679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:07.147543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.181645+0000","last_change":"2024-03-06T07:25:04.174153+0000","last_active":"2024-03-06T07:25:05.181645+0000","last_peered":"2024-03-06T07:25:05.181645+0000","last_clean":"2024-03-06T07:25:05.181645+0000","last_became_active":"2024-03-06T07:25:04.173868+0000","last_became_peered":"2024-03-06T07:25:04.173868+0000","last_unstale":"2024-03-06T07:25:05.181645+0000","last_undegraded":"2024-03-06T07:25:05.181645+0000","last_fullsized":"2024-03-06T07:25:05.181645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:53.401899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176371+0000","last_change":"2024-03-06T07:25:05.174743+0000","last_active":"2024-03-06T07:25:05.176371+0000","last_peered":"2024-03-06T07:25:05.176371+0000","last_clean":"2024-03-06T07:25:05.176371+0000","last_became_active":"2024-03-06T07:25:05.174441+0000","last_became_peered":"2024-03-06T07:25:05.174441+0000","last_unstale":"2024-03-06T07:25:05.176371+0000","last_undegraded":"2024-03-06T07:25:05.176371+0000","last_fullsized":"2024-03-06T07:25:05.176371+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:58.136158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.163309+0000","last_change":"2024-03-06T07:25:04.174253+0000","last_active":"2024-03-06T07:25:05.163309+0000","last_peered":"2024-03-06T07:25:05.163309+0000","last_clean":"2024-03-06T07:25:05.163309+0000","last_became_active":"2024-03-06T07:25:04.173938+0000","last_became_peered":"2024-03-06T07:25:04.173938+0000","last_unstale":"2024-03-06T07:25:05.163309+0000","last_undegraded":"2024-03-06T07:25:05.163309+0000","last_fullsized":"2024-03-06T07:25:05.163309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:15.369337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156159+0000","last_change":"2024-03-06T07:25:05.179646+0000","last_active":"2024-03-06T07:25:06.156159+0000","last_peered":"2024-03-06T07:25:06.156159+0000","last_clean":"2024-03-06T07:25:06.156159+0000","last_became_active":"2024-03-06T07:25:05.179317+0000","last_became_peered":"2024-03-06T07:25:05.179317+0000","last_unstale":"2024-03-06T07:25:06.156159+0000","last_undegraded":"2024-03-06T07:25:06.156159+0000","last_fullsized":"2024-03-06T07:25:06.156159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:28.040514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.165650+0000","last_change":"2024-03-06T07:25:04.172205+0000","last_active":"2024-03-06T07:25:05.165650+0000","last_peered":"2024-03-06T07:25:05.165650+0000","last_clean":"2024-03-06T07:25:05.165650+0000","last_became_active":"2024-03-06T07:25:04.171932+0000","last_became_peered":"2024-03-06T07:25:04.171932+0000","last_unstale":"2024-03-06T07:25:05.165650+0000","last_undegraded":"2024-03-06T07:25:05.165650+0000","last_fullsized":"2024-03-06T07:25:05.165650+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:52:08.265287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155995+0000","last_change":"2024-03-06T07:25:05.182790+0000","last_active":"2024-03-06T07:25:06.155995+0000","last_peered":"2024-03-06T07:25:06.155995+0000","last_clean":"2024-03-06T07:25:06.155995+0000","last_became_active":"2024-03-06T07:25:05.182322+0000","last_became_peered":"2024-03-06T07:25:05.182322+0000","last_unstale":"2024-03-06T07:25:06.155995+0000","last_undegraded":"2024-03-06T07:25:06.155995+0000","last_fullsized":"2024-03-06T07:25:06.155995+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:00.651166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156855+0000","last_change":"2024-03-06T07:25:04.164589+0000","last_active":"2024-03-06T07:25:06.156855+0000","last_peered":"2024-03-06T07:25:06.156855+0000","last_clean":"2024-03-06T07:25:06.156855+0000","last_became_active":"2024-03-06T07:25:04.164235+0000","last_became_peered":"2024-03-06T07:25:04.164235+0000","last_unstale":"2024-03-06T07:25:06.156855+0000","last_undegraded":"2024-03-06T07:25:06.156855+0000","last_fullsized":"2024-03-06T07:25:06.156855+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:06.145388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156051+0000","last_change":"2024-03-06T07:25:05.180865+0000","last_active":"2024-03-06T07:25:06.156051+0000","last_peered":"2024-03-06T07:25:06.156051+0000","last_clean":"2024-03-06T07:25:06.156051+0000","last_became_active":"2024-03-06T07:25:05.180568+0000","last_became_peered":"2024-03-06T07:25:05.180568+0000","last_unstale":"2024-03-06T07:25:06.156051+0000","last_undegraded":"2024-03-06T07:25:06.156051+0000","last_fullsized":"2024-03-06T07:25:06.156051+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:03.798281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172511+0000","last_change":"2024-03-06T07:25:04.166345+0000","last_active":"2024-03-06T07:25:06.172511+0000","last_peered":"2024-03-06T07:25:06.172511+0000","last_clean":"2024-03-06T07:25:06.172511+0000","last_became_active":"2024-03-06T07:25:04.166053+0000","last_became_peered":"2024-03-06T07:25:04.166053+0000","last_unstale":"2024-03-06T07:25:06.172511+0000","last_undegraded":"2024-03-06T07:25:06.172511+0000","last_fullsized":"2024-03-06T07:25:06.172511+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:06.145388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172828+0000","last_change":"2024-03-06T07:25:05.170943+0000","last_active":"2024-03-06T07:25:06.172828+0000","last_peered":"2024-03-06T07:25:06.172828+0000","last_clean":"2024-03-06T07:25:06.172828+0000","last_became_active":"2024-03-06T07:25:05.170576+0000","last_became_peered":"2024-03-06T07:25:05.170576+0000","last_unstale":"2024-03-06T07:25:06.172828+0000","last_undegraded":"2024-03-06T07:25:06.172828+0000","last_fullsized":"2024-03-06T07:25:06.172828+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:15.620644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.179721+0000","last_change":"2024-03-06T07:25:04.170274+0000","last_active":"2024-03-06T07:25:05.179721+0000","last_peered":"2024-03-06T07:25:05.179721+0000","last_clean":"2024-03-06T07:25:05.179721+0000","last_became_active":"2024-03-06T07:25:04.170001+0000","last_became_peered":"2024-03-06T07:25:04.170001+0000","last_unstale":"2024-03-06T07:25:05.179721+0000","last_undegraded":"2024-03-06T07:25:05.179721+0000","last_fullsized":"2024-03-06T07:25:05.179721+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:33.098672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156739+0000","last_change":"2024-03-06T07:25:05.181381+0000","last_active":"2024-03-06T07:25:06.156739+0000","last_peered":"2024-03-06T07:25:06.156739+0000","last_clean":"2024-03-06T07:25:06.156739+0000","last_became_active":"2024-03-06T07:25:05.181054+0000","last_became_peered":"2024-03-06T07:25:05.181054+0000","last_unstale":"2024-03-06T07:25:06.156739+0000","last_undegraded":"2024-03-06T07:25:06.156739+0000","last_fullsized":"2024-03-06T07:25:06.156739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:34.634891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156849+0000","last_change":"2024-03-06T07:25:04.165832+0000","last_active":"2024-03-06T07:25:06.156849+0000","last_peered":"2024-03-06T07:25:06.156849+0000","last_clean":"2024-03-06T07:25:06.156849+0000","last_became_active":"2024-03-06T07:25:04.165536+0000","last_became_peered":"2024-03-06T07:25:04.165536+0000","last_unstale":"2024-03-06T07:25:06.156849+0000","last_undegraded":"2024-03-06T07:25:06.156849+0000","last_fullsized":"2024-03-06T07:25:06.156849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:54.865160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.184859+0000","last_change":"2024-03-06T07:25:05.181291+0000","last_active":"2024-03-06T07:25:05.184859+0000","last_peered":"2024-03-06T07:25:05.184859+0000","last_clean":"2024-03-06T07:25:05.184859+0000","last_became_active":"2024-03-06T07:25:05.181044+0000","last_became_peered":"2024-03-06T07:25:05.181044+0000","last_unstale":"2024-03-06T07:25:05.184859+0000","last_undegraded":"2024-03-06T07:25:05.184859+0000","last_fullsized":"2024-03-06T07:25:05.184859+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:12.941579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.155524+0000","last_change":"2024-03-06T07:25:04.166343+0000","last_active":"2024-03-06T07:25:05.155524+0000","last_peered":"2024-03-06T07:25:05.155524+0000","last_clean":"2024-03-06T07:25:05.155524+0000","last_became_active":"2024-03-06T07:25:04.166086+0000","last_became_peered":"2024-03-06T07:25:04.166086+0000","last_unstale":"2024-03-06T07:25:05.155524+0000","last_undegraded":"2024-03-06T07:25:05.155524+0000","last_fullsized":"2024-03-06T07:25:05.155524+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:29.946081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.182856+0000","last_change":"2024-03-06T07:25:05.179127+0000","last_active":"2024-03-06T07:25:05.182856+0000","last_peered":"2024-03-06T07:25:05.182856+0000","last_clean":"2024-03-06T07:25:05.182856+0000","last_became_active":"2024-03-06T07:25:05.178480+0000","last_became_peered":"2024-03-06T07:25:05.178480+0000","last_unstale":"2024-03-06T07:25:05.182856+0000","last_undegraded":"2024-03-06T07:25:05.182856+0000","last_fullsized":"2024-03-06T07:25:05.182856+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:31:45.697928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.172725+0000","last_change":"2024-03-06T07:25:04.172405+0000","last_active":"2024-03-06T07:25:06.172725+0000","last_peered":"2024-03-06T07:25:06.172725+0000","last_clean":"2024-03-06T07:25:06.172725+0000","last_became_active":"2024-03-06T07:25:04.172070+0000","last_became_peered":"2024-03-06T07:25:04.172070+0000","last_unstale":"2024-03-06T07:25:06.172725+0000","last_undegraded":"2024-03-06T07:25:06.172725+0000","last_fullsized":"2024-03-06T07:25:06.172725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:28.525411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.176192+0000","last_change":"2024-03-06T07:25:05.169212+0000","last_active":"2024-03-06T07:25:05.176192+0000","last_peered":"2024-03-06T07:25:05.176192+0000","last_clean":"2024-03-06T07:25:05.176192+0000","last_became_active":"2024-03-06T07:25:05.168919+0000","last_became_peered":"2024-03-06T07:25:05.168919+0000","last_unstale":"2024-03-06T07:25:05.176192+0000","last_undegraded":"2024-03-06T07:25:05.176192+0000","last_fullsized":"2024-03-06T07:25:05.176192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:01.757888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.155664+0000","last_change":"2024-03-06T07:25:04.160814+0000","last_active":"2024-03-06T07:25:05.155664+0000","last_peered":"2024-03-06T07:25:05.155664+0000","last_clean":"2024-03-06T07:25:05.155664+0000","last_became_active":"2024-03-06T07:25:04.160521+0000","last_became_peered":"2024-03-06T07:25:04.160521+0000","last_unstale":"2024-03-06T07:25:05.155664+0000","last_undegraded":"2024-03-06T07:25:05.155664+0000","last_fullsized":"2024-03-06T07:25:05.155664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:54.662090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187840+0000","last_change":"2024-03-06T07:25:05.184235+0000","last_active":"2024-03-06T07:25:05.187840+0000","last_peered":"2024-03-06T07:25:05.187840+0000","last_clean":"2024-03-06T07:25:05.187840+0000","last_became_active":"2024-03-06T07:25:05.183943+0000","last_became_peered":"2024-03-06T07:25:05.183943+0000","last_unstale":"2024-03-06T07:25:05.187840+0000","last_undegraded":"2024-03-06T07:25:05.187840+0000","last_fullsized":"2024-03-06T07:25:05.187840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:27.369584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155283+0000","last_change":"2024-03-06T07:25:04.171677+0000","last_active":"2024-03-06T07:25:06.155283+0000","last_peered":"2024-03-06T07:25:06.155283+0000","last_clean":"2024-03-06T07:25:06.155283+0000","last_became_active":"2024-03-06T07:25:04.171415+0000","last_became_peered":"2024-03-06T07:25:04.171415+0000","last_unstale":"2024-03-06T07:25:06.155283+0000","last_undegraded":"2024-03-06T07:25:06.155283+0000","last_fullsized":"2024-03-06T07:25:06.155283+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:58:42.546597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.155377+0000","last_change":"2024-03-06T07:25:05.167404+0000","last_active":"2024-03-06T07:25:06.155377+0000","last_peered":"2024-03-06T07:25:06.155377+0000","last_clean":"2024-03-06T07:25:06.155377+0000","last_became_active":"2024-03-06T07:25:05.167076+0000","last_became_peered":"2024-03-06T07:25:05.167076+0000","last_unstale":"2024-03-06T07:25:06.155377+0000","last_undegraded":"2024-03-06T07:25:06.155377+0000","last_fullsized":"2024-03-06T07:25:06.155377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:01:38.868144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.158252+0000","last_change":"2024-03-06T07:25:04.168113+0000","last_active":"2024-03-06T07:25:05.158252+0000","last_peered":"2024-03-06T07:25:05.158252+0000","last_clean":"2024-03-06T07:25:05.158252+0000","last_became_active":"2024-03-06T07:25:04.167821+0000","last_became_peered":"2024-03-06T07:25:04.167821+0000","last_unstale":"2024-03-06T07:25:05.158252+0000","last_undegraded":"2024-03-06T07:25:05.158252+0000","last_fullsized":"2024-03-06T07:25:05.158252+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:48.554709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.187563+0000","last_change":"2024-03-06T07:25:05.180999+0000","last_active":"2024-03-06T07:25:05.187563+0000","last_peered":"2024-03-06T07:25:05.187563+0000","last_clean":"2024-03-06T07:25:05.187563+0000","last_became_active":"2024-03-06T07:25:05.180739+0000","last_became_peered":"2024-03-06T07:25:05.180739+0000","last_unstale":"2024-03-06T07:25:05.187563+0000","last_undegraded":"2024-03-06T07:25:05.187563+0000","last_fullsized":"2024-03-06T07:25:05.187563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:34.775717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.164943+0000","last_change":"2024-03-06T07:25:04.169366+0000","last_active":"2024-03-06T07:25:05.164943+0000","last_peered":"2024-03-06T07:25:05.164943+0000","last_clean":"2024-03-06T07:25:05.164943+0000","last_became_active":"2024-03-06T07:25:04.168918+0000","last_became_peered":"2024-03-06T07:25:04.168918+0000","last_unstale":"2024-03-06T07:25:05.164943+0000","last_undegraded":"2024-03-06T07:25:05.164943+0000","last_fullsized":"2024-03-06T07:25:05.164943+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:57.134003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-06T07:25:06.156064+0000","last_change":"2024-03-06T07:25:05.181711+0000","last_active":"2024-03-06T07:25:06.156064+0000","last_peered":"2024-03-06T07:25:06.156064+0000","last_clean":"2024-03-06T07:25:06.156064+0000","last_became_active":"2024-03-06T07:25:05.181416+0000","last_became_peered":"2024-03-06T07:25:05.181416+0000","last_unstale":"2024-03-06T07:25:06.156064+0000","last_undegraded":"2024-03-06T07:25:06.156064+0000","last_fullsized":"2024-03-06T07:25:06.156064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:04.144290+0000","last_clean_scrub_stamp":"2024-03-06T07:25:04.144290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:45:52.374211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.182171+0000","last_change":"2024-03-06T07:25:04.168606+0000","last_active":"2024-03-06T07:25:05.182171+0000","last_peered":"2024-03-06T07:25:05.182171+0000","last_clean":"2024-03-06T07:25:05.182171+0000","last_became_active":"2024-03-06T07:25:04.168127+0000","last_became_peered":"2024-03-06T07:25:04.168127+0000","last_unstale":"2024-03-06T07:25:05.182171+0000","last_undegraded":"2024-03-06T07:25:05.182171+0000","last_fullsized":"2024-03-06T07:25:05.182171+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:03.142135+0000","last_clean_scrub_stamp":"2024-03-06T07:25:03.142135+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:29.002981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":258,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:25:05.164116+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:25:05.164116+0000","last_peered":"2024-03-06T07:25:05.164116+0000","last_clean":"2024-03-06T07:25:05.164116+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:25:05.164116+0000","last_undegraded":"2024-03-06T07:25:05.164116+0000","last_fullsized":"2024-03-06T07:25:05.164116+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32592,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339248,"statfs":{"total":96636764160,"available":96603389952,"internally_reserved":0,"allocated":1441792,"data_stored":943656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":847872,"data_stored":352764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":843776,"data_stored":352762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":843776,"data_stored":352742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":847872,"data_stored":352784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":851968,"data_stored":354284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":354900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574893,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1433600,"data_stored":943040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:08.514 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:08.514 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:08.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.657+0000 7f6999956640 1 Processor -- start 2024-03-06T07:25:08.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.657+0000 7f6999956640 1 -- start start 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f6999956640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f69940d4860 0x7f69940d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f6999956640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69940b0e30 0x7f69940b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f6999956640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69941618b0 con 0x7f69941614d0 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f6999956640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69940d4cf0 con 0x7f69940d4860 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f6999956640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69940cf400 con 0x7f69940b0e30 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f6992ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f69941614d0 0x7f69940d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58800/0 (socket says 172.21.15.59:58800) 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f6992ffd640 1 -- 172.21.15.59:0/2333535654 learned_addr learned my addr 172.21.15.59:0/2333535654 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f69937fe640 1 --2- 172.21.15.59:0/2333535654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69940b0e30 0x7f69940b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f69927fc640 1 --2- 172.21.15.59:0/2333535654 >> v2:172.21.15.164:3300/0 conn(0x7f69940d4860 0x7f69940d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f6991ffb640 1 -- 172.21.15.59:0/2333535654 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244774590 0 0) 0x7f69941618b0 con 0x7f69941614d0 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f6991ffb640 1 -- 172.21.15.59:0/2333535654 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6980003520 con 0x7f69941614d0 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f6991ffb640 1 -- 172.21.15.59:0/2333535654 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2044875823 0 0) 0x7f6980003520 con 0x7f69941614d0 2024-03-06T07:25:08.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f69927fc640 1 -- 172.21.15.59:0/2333535654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69940b0e30 msgr2=0x7f69940b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:08.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f69927fc640 1 --2- 172.21.15.59:0/2333535654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69940b0e30 0x7f69940b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f69927fc640 1 -- 172.21.15.59:0/2333535654 >> v1:172.21.15.59:6789/0 conn(0x7f69941614d0 legacy=0x7f69940d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f69927fc640 1 -- 172.21.15.59:0/2333535654 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69940cf6a0 con 0x7f69940d4860 2024-03-06T07:25:08.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.658+0000 7f69937fe640 1 --2- 172.21.15.59:0/2333535654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69940b0e30 0x7f69940b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:08.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.659+0000 7f69927fc640 1 --2- 172.21.15.59:0/2333535654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69940d4860 0x7f69940d2020 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f697c0097a0 tx=0x7f697c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68dcc076464db7ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.659+0000 7f6991ffb640 1 -- 172.21.15.59:0/2333535654 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c037070 con 0x7f69940d4860 2024-03-06T07:25:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.659+0000 7f6991ffb640 1 -- 172.21.15.59:0/2333535654 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697c030510 con 0x7f69940d4860 2024-03-06T07:25:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.659+0000 7f6991ffb640 1 -- 172.21.15.59:0/2333535654 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c001670 con 0x7f69940d4860 2024-03-06T07:25:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.660+0000 7f6999956640 1 -- 172.21.15.59:0/2333535654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69940d4860 msgr2=0x7f69940d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.660+0000 7f6999956640 1 --2- 172.21.15.59:0/2333535654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69940d4860 0x7f69940d2020 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f697c0097a0 tx=0x7f697c02b790 comp rx=0 tx=0).stop 2024-03-06T07:25:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.660+0000 7f6999956640 1 -- 172.21.15.59:0/2333535654 shutdown_connections 2024-03-06T07:25:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.660+0000 7f6999956640 1 --2- 172.21.15.59:0/2333535654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69940b0e30 0x7f69940b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.660+0000 7f6999956640 1 --2- 172.21.15.59:0/2333535654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69940d4860 0x7f69940d2020 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.660+0000 7f6999956640 1 -- 172.21.15.59:0/2333535654 >> 172.21.15.59:0/2333535654 conn(0x7f69940d3d40 msgr2=0x7f69940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.660+0000 7f6999956640 1 -- 172.21.15.59:0/2333535654 shutdown_connections 2024-03-06T07:25:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.660+0000 7f6999956640 1 -- 172.21.15.59:0/2333535654 wait complete. 2024-03-06T07:25:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.661+0000 7f6999956640 1 Processor -- start 2024-03-06T07:25:08.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.661+0000 7f6999956640 1 -- start start 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6999956640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69940b0e30 0x7f699415b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6999956640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69940d4860 0x7f69940ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6999956640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69941614d0 0x7f69940cb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6999956640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69940b0910 con 0x7f69940b0e30 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6999956640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69940cfc60 con 0x7f69941614d0 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6999956640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69940d2560 con 0x7f69940d4860 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6992ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69940b0e30 0x7f699415b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6992ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69940b0e30 0x7f699415b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54754/0 (socket says 172.21.15.59:54754) 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6992ffd640 1 -- 172.21.15.59:0/4131629602 learned_addr learned my addr 172.21.15.59:0/4131629602 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f69937fe640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69941614d0 0x7f69940cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f69927fc640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69940d4860 0x7f69940ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6992ffd640 1 -- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69940d4860 msgr2=0x7f69940ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6992ffd640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69940d4860 0x7f69940ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6992ffd640 1 -- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69941614d0 msgr2=0x7f69940cb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6992ffd640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69941614d0 0x7f69940cb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.662+0000 7f6992ffd640 1 -- 172.21.15.59:0/4131629602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69940cafb0 con 0x7f69940b0e30 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.663+0000 7f6992ffd640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69940b0e30 0x7f699415b2e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f698c00c750 tx=0x7f698c00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.663+0000 7f698bfff640 1 -- 172.21.15.59:0/4131629602 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c005ad0 con 0x7f69940b0e30 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.663+0000 7f698bfff640 1 -- 172.21.15.59:0/4131629602 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c005c90 con 0x7f69940b0e30 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.663+0000 7f6999956640 1 -- 172.21.15.59:0/4131629602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69940cd410 con 0x7f69940b0e30 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.663+0000 7f698bfff640 1 -- 172.21.15.59:0/4131629602 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c0072b0 con 0x7f69940b0e30 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.663+0000 7f6999956640 1 -- 172.21.15.59:0/4131629602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69940cb250 con 0x7f69940b0e30 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.664+0000 7f6999956640 1 -- 172.21.15.59:0/4131629602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69940cd930 con 0x7f69940b0e30 2024-03-06T07:25:08.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.664+0000 7f698bfff640 1 -- 172.21.15.59:0/4131629602 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f698c007450 con 0x7f69940b0e30 2024-03-06T07:25:08.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.665+0000 7f698bfff640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f697406b4d0 0x7f697406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:08.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.665+0000 7f698bfff640 1 -- 172.21.15.59:0/4131629602 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f698c089e60 con 0x7f69940b0e30 2024-03-06T07:25:08.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.665+0000 7f69927fc640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f697406b4d0 0x7f697406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:08.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.665+0000 7f69927fc640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f697406b4d0 0x7f697406d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f697c034000 tx=0x7f697c02f000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:08.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.669+0000 7f698bfff640 1 -- 172.21.15.59:0/4131629602 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f698c058d70 con 0x7f69940b0e30 2024-03-06T07:25:08.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.924+0000 7f6999956640 1 -- 172.21.15.59:0/4131629602 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f69940b80c0 con 0x7f69940b0e30 2024-03-06T07:25:08.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.926+0000 7f698bfff640 1 -- 172.21.15.59:0/4131629602 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f698c058670 con 0x7f69940b0e30 2024-03-06T07:25:08.928 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-06T07:25:08.928 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA== 2024-03-06T07:25:08.928 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-03-06T07:25:08.928 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-03-06T07:25:08.928 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:08.928 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-03-06T07:25:08.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.928+0000 7f6999956640 1 -- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f697406b4d0 msgr2=0x7f697406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.928+0000 7f6999956640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f697406b4d0 0x7f697406d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f697c034000 tx=0x7f697c02f000 comp rx=0 tx=0).stop 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.928+0000 7f6999956640 1 -- 172.21.15.59:0/4131629602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69940b0e30 msgr2=0x7f699415b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.928+0000 7f6999956640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69940b0e30 0x7f699415b2e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f698c00c750 tx=0x7f698c00cb50 comp rx=0 tx=0).stop 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.928+0000 7f6999956640 1 -- 172.21.15.59:0/4131629602 shutdown_connections 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.928+0000 7f6999956640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f697406b4d0 0x7f697406d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.928+0000 7f6999956640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69941614d0 0x7f69940cb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.928+0000 7f6999956640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69940d4860 0x7f69940ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.928+0000 7f6999956640 1 --2- 172.21.15.59:0/4131629602 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69940b0e30 0x7f699415b2e0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.928+0000 7f6999956640 1 -- 172.21.15.59:0/4131629602 >> 172.21.15.59:0/4131629602 conn(0x7f69940d3d40 msgr2=0x7f69940d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.929+0000 7f6999956640 1 -- 172.21.15.59:0/4131629602 shutdown_connections 2024-03-06T07:25:08.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:08.929+0000 7f6999956640 1 -- 172.21.15.59:0/4131629602 wait complete. 2024-03-06T07:25:08.941 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:09.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.041+0000 7f583b671640 1 Processor -- start 2024-03-06T07:25:09.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.041+0000 7f583b671640 1 -- start start 2024-03-06T07:25:09.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.041+0000 7f583b671640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f583413dc10 0x7f58340b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f583b671640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583415f580 con 0x7f583415bf30 2024-03-06T07:25:09.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f583b671640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583415b450 con 0x7f583415ef60 2024-03-06T07:25:09.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f583b671640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340cf3f0 con 0x7f583413dc10 2024-03-06T07:25:09.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f5838c0c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f583415bf30 0x7f583415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58812/0 (socket says 172.21.15.59:58812) 2024-03-06T07:25:09.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f5838c0c640 1 -- 172.21.15.59:0/1418860747 learned_addr learned my addr 172.21.15.59:0/1418860747 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:09.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f583940d640 1 --2- 172.21.15.59:0/1418860747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f583413dc10 0x7f58340b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f582bfff640 1 -- 172.21.15.59:0/1418860747 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368687640 0 0) 0x7f583415f580 con 0x7f583415bf30 2024-03-06T07:25:09.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f582bfff640 1 -- 172.21.15.59:0/1418860747 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5820003520 con 0x7f583415bf30 2024-03-06T07:25:09.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f582bfff640 1 -- 172.21.15.59:0/1418860747 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465182552 0 0) 0x7f583415b450 con 0x7f583415ef60 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f583940d640 1 -- 172.21.15.59:0/1418860747 >> v1:172.21.15.164:6789/0 conn(0x7f583415ef60 legacy=0x7f58340708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f583940d640 1 -- 172.21.15.59:0/1418860747 >> v1:172.21.15.59:6789/0 conn(0x7f583415bf30 legacy=0x7f583415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.042+0000 7f583940d640 1 -- 172.21.15.59:0/1418860747 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583415b730 con 0x7f583413dc10 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.043+0000 7f583940d640 1 --2- 172.21.15.59:0/1418860747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f583413dc10 0x7f58340b2950 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f58300097d0 tx=0x7f583002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=835f84375a46af32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.043+0000 7f582bfff640 1 -- 172.21.15.59:0/1418860747 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830035020 con 0x7f583413dc10 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.043+0000 7f582bfff640 1 -- 172.21.15.59:0/1418860747 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5830031800 con 0x7f583413dc10 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.043+0000 7f582bfff640 1 -- 172.21.15.59:0/1418860747 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830031bb0 con 0x7f583413dc10 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.044+0000 7f583b671640 1 -- 172.21.15.59:0/1418860747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f583413dc10 msgr2=0x7f58340b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.044+0000 7f583b671640 1 --2- 172.21.15.59:0/1418860747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f583413dc10 0x7f58340b2950 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f58300097d0 tx=0x7f583002b8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.044+0000 7f583b671640 1 -- 172.21.15.59:0/1418860747 shutdown_connections 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.044+0000 7f583b671640 1 --2- 172.21.15.59:0/1418860747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f583413dc10 0x7f58340b2950 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.044+0000 7f583b671640 1 -- 172.21.15.59:0/1418860747 >> 172.21.15.59:0/1418860747 conn(0x7f58340cf8a0 msgr2=0x7f58340cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.044+0000 7f583b671640 1 -- 172.21.15.59:0/1418860747 shutdown_connections 2024-03-06T07:25:09.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.044+0000 7f583b671640 1 -- 172.21.15.59:0/1418860747 wait complete. 2024-03-06T07:25:09.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.045+0000 7f583b671640 1 Processor -- start 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.045+0000 7f583b671640 1 -- start start 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583b671640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583413dc10 0x7f5834138780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583b671640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f583415bf30 0x7f58340cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583b671640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f583415ef60 0x7f583413a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583b671640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340b2e90 con 0x7f583413dc10 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583b671640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583415b210 con 0x7f583415ef60 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583b671640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340b3010 con 0x7f583415bf30 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583940d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583413dc10 0x7f5834138780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583940d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583413dc10 0x7f5834138780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54762/0 (socket says 172.21.15.59:54762) 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583940d640 1 -- 172.21.15.59:0/2809651672 learned_addr learned my addr 172.21.15.59:0/2809651672 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f5838c0c640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f583415bf30 0x7f58340cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f5839c0e640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f583415ef60 0x7f583413a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583940d640 1 -- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f583415bf30 msgr2=0x7f58340cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583940d640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f583415bf30 0x7f58340cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583940d640 1 -- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f583415ef60 msgr2=0x7f583413a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583940d640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f583415ef60 0x7f583413a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583940d640 1 -- 172.21.15.59:0/2809651672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58341438d0 con 0x7f583413dc10 2024-03-06T07:25:09.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583940d640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583413dc10 0x7f5834138780 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5830000c00 tx=0x7f58300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f5829ffb640 1 -- 172.21.15.59:0/2809651672 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830035040 con 0x7f583413dc10 2024-03-06T07:25:09.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f5829ffb640 1 -- 172.21.15.59:0/2809651672 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5830039cf0 con 0x7f583413dc10 2024-03-06T07:25:09.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f5829ffb640 1 -- 172.21.15.59:0/2809651672 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830039ea0 con 0x7f583413dc10 2024-03-06T07:25:09.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.046+0000 7f583b671640 1 -- 172.21.15.59:0/2809651672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5834141f10 con 0x7f583413dc10 2024-03-06T07:25:09.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.047+0000 7f583b671640 1 -- 172.21.15.59:0/2809651672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583415a460 con 0x7f583413dc10 2024-03-06T07:25:09.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.047+0000 7f583b671640 1 -- 172.21.15.59:0/2809651672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583404f330 con 0x7f583413dc10 2024-03-06T07:25:09.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.048+0000 7f5829ffb640 1 -- 172.21.15.59:0/2809651672 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5830038020 con 0x7f583413dc10 2024-03-06T07:25:09.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.048+0000 7f5829ffb640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f581806b4d0 0x7f581806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.048+0000 7f5829ffb640 1 -- 172.21.15.59:0/2809651672 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58300abff0 con 0x7f583413dc10 2024-03-06T07:25:09.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.049+0000 7f5838c0c640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f581806b4d0 0x7f581806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.049+0000 7f5838c0c640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f581806b4d0 0x7f581806d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5834138cc0 tx=0x7f5824009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.053+0000 7f5829ffb640 1 -- 172.21.15.59:0/2809651672 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58341438d0 con 0x7f583413dc10 2024-03-06T07:25:09.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.273+0000 7f583b671640 1 -- 172.21.15.59:0/2809651672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f583406eb00 con 0x7f583413dc10 2024-03-06T07:25:09.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.273+0000 7f5829ffb640 1 -- 172.21.15.59:0/2809651672 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f583007a930 con 0x7f583413dc10 2024-03-06T07:25:09.275 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:09.276 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:04.517350+0000","modified":"2024-03-06T07:25:06.518231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/250576205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":250576205},{"type":"v1","addr":"172.21.15.164:6837","nonce":250576205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:09.276 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 35 2024-03-06T07:25:09.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.275+0000 7f583b671640 1 -- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f581806b4d0 msgr2=0x7f581806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.275+0000 7f583b671640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f581806b4d0 0x7f581806d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5834138cc0 tx=0x7f5824009040 comp rx=0 tx=0).stop 2024-03-06T07:25:09.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.275+0000 7f583b671640 1 -- 172.21.15.59:0/2809651672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583413dc10 msgr2=0x7f5834138780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.275+0000 7f583b671640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583413dc10 0x7f5834138780 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5830000c00 tx=0x7f58300046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.275+0000 7f583b671640 1 -- 172.21.15.59:0/2809651672 shutdown_connections 2024-03-06T07:25:09.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.275+0000 7f583b671640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f581806b4d0 0x7f581806d990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.275+0000 7f583b671640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f583415ef60 0x7f583413a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.275+0000 7f583b671640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f583415bf30 0x7f58340cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.275+0000 7f583b671640 1 --2- 172.21.15.59:0/2809651672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f583413dc10 0x7f5834138780 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.275+0000 7f583b671640 1 -- 172.21.15.59:0/2809651672 >> 172.21.15.59:0/2809651672 conn(0x7f58340cf8a0 msgr2=0x7f583413bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.275+0000 7f583b671640 1 -- 172.21.15.59:0/2809651672 shutdown_connections 2024-03-06T07:25:09.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.276+0000 7f583b671640 1 -- 172.21.15.59:0/2809651672 wait complete. 2024-03-06T07:25:09.690 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:04.517350+0000', 'modified': '2024-03-06T07:25:06.518231+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6579}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6579': {'gid': 6579, 'name': 'd', 'rank': 0, 'incarnation': 33, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.164:6837/250576205', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 250576205}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 250576205}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6579, 'qdb_cluster': [6579]} 2024-03-06T07:25:09.690 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:09.690 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:25:09.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.795+0000 7f986d581640 1 Processor -- start 2024-03-06T07:25:09.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.795+0000 7f986d581640 1 -- start start 2024-03-06T07:25:09.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.796+0000 7f986d581640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f986806d3c0 0x7f98680b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.796+0000 7f986d581640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f98680b1850 0x7f98680b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.796+0000 7f986d581640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98680af4b0 con 0x7f98681614a0 2024-03-06T07:25:09.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.796+0000 7f986d581640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986806d800 con 0x7f98680b1850 2024-03-06T07:25:09.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.796+0000 7f986d581640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98680af790 con 0x7f986806d3c0 2024-03-06T07:25:09.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.796+0000 7f9866ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f98681614a0 0x7f98680b0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58818/0 (socket says 172.21.15.59:58818) 2024-03-06T07:25:09.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.796+0000 7f9866ffd640 1 -- 172.21.15.59:0/3003744299 learned_addr learned my addr 172.21.15.59:0/3003744299 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:09.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.796+0000 7f98677fe640 1 --2- 172.21.15.59:0/3003744299 >> v2:172.21.15.164:3300/0 conn(0x7f98680b1850 0x7f98680b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.796+0000 7f98667fc640 1 --2- 172.21.15.59:0/3003744299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f986806d3c0 0x7f98680b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.796+0000 7f9865ffb640 1 -- 172.21.15.59:0/3003744299 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018775210 0 0) 0x7f98680af4b0 con 0x7f98681614a0 2024-03-06T07:25:09.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.796+0000 7f9865ffb640 1 -- 172.21.15.59:0/3003744299 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9850003520 con 0x7f98681614a0 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.797+0000 7f98677fe640 1 -- 172.21.15.59:0/3003744299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f986806d3c0 msgr2=0x7f98680b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.797+0000 7f98677fe640 1 --2- 172.21.15.59:0/3003744299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f986806d3c0 0x7f98680b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.797+0000 7f98677fe640 1 -- 172.21.15.59:0/3003744299 >> v1:172.21.15.59:6789/0 conn(0x7f98681614a0 legacy=0x7f98680b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.797+0000 7f98677fe640 1 -- 172.21.15.59:0/3003744299 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98680afa70 con 0x7f98680b1850 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.797+0000 7f98667fc640 1 --2- 172.21.15.59:0/3003744299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f986806d3c0 0x7f98680b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.797+0000 7f98677fe640 1 --2- 172.21.15.59:0/3003744299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98680b1850 0x7f98680b5510 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9858009d90 tx=0x7f985802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3de3d6197aa5e8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.798+0000 7f9865ffb640 1 -- 172.21.15.59:0/3003744299 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858039070 con 0x7f98680b1850 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.798+0000 7f9865ffb640 1 -- 172.21.15.59:0/3003744299 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985802cc50 con 0x7f98680b1850 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.798+0000 7f9865ffb640 1 -- 172.21.15.59:0/3003744299 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858034440 con 0x7f98680b1850 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.798+0000 7f986d581640 1 -- 172.21.15.59:0/3003744299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98680b1850 msgr2=0x7f98680b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.798+0000 7f986d581640 1 --2- 172.21.15.59:0/3003744299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98680b1850 0x7f98680b5510 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9858009d90 tx=0x7f985802bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.798+0000 7f986d581640 1 -- 172.21.15.59:0/3003744299 shutdown_connections 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.798+0000 7f986d581640 1 --2- 172.21.15.59:0/3003744299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f98680b1850 0x7f98680b5510 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.798+0000 7f986d581640 1 --2- 172.21.15.59:0/3003744299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f986806d3c0 0x7f98680b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.798+0000 7f986d581640 1 -- 172.21.15.59:0/3003744299 >> 172.21.15.59:0/3003744299 conn(0x7f986809a9f0 msgr2=0x7f986809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.798+0000 7f986d581640 1 -- 172.21.15.59:0/3003744299 shutdown_connections 2024-03-06T07:25:09.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.798+0000 7f986d581640 1 -- 172.21.15.59:0/3003744299 wait complete. 2024-03-06T07:25:09.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.799+0000 7f986d581640 1 Processor -- start 2024-03-06T07:25:09.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f986d581640 1 -- start start 2024-03-06T07:25:09.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f986d581640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f986806d3c0 0x7f986814aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f986d581640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98680b1850 0x7f986814a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f986d581640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98681614a0 0x7f986815bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f986d581640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98680b1c80 con 0x7f98681614a0 2024-03-06T07:25:09.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f986d581640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9868161880 con 0x7f986806d3c0 2024-03-06T07:25:09.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f986d581640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98680b1e00 con 0x7f98680b1850 2024-03-06T07:25:09.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f98677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98681614a0 0x7f986815bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f98677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98681614a0 0x7f986815bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54766/0 (socket says 172.21.15.59:54766) 2024-03-06T07:25:09.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f98677fe640 1 -- 172.21.15.59:0/733177007 learned_addr learned my addr 172.21.15.59:0/733177007 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:09.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f98667fc640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98680b1850 0x7f986814a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.800+0000 7f9866ffd640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f986806d3c0 0x7f986814aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.801+0000 7f98677fe640 1 -- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98680b1850 msgr2=0x7f986814a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.801+0000 7f98677fe640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98680b1850 0x7f986814a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.801+0000 7f98677fe640 1 -- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f986806d3c0 msgr2=0x7f986814aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.801+0000 7f98677fe640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f986806d3c0 0x7f986814aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.801+0000 7f98677fe640 1 -- 172.21.15.59:0/733177007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9868159380 con 0x7f98681614a0 2024-03-06T07:25:09.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.801+0000 7f98677fe640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98681614a0 0x7f986815bbd0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9858009eb0 tx=0x7f9858004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.801+0000 7f984f7fe640 1 -- 172.21.15.59:0/733177007 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858039050 con 0x7f98681614a0 2024-03-06T07:25:09.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.801+0000 7f984f7fe640 1 -- 172.21.15.59:0/733177007 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9858045070 con 0x7f98681614a0 2024-03-06T07:25:09.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.801+0000 7f986d581640 1 -- 172.21.15.59:0/733177007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98680cc570 con 0x7f98681614a0 2024-03-06T07:25:09.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.801+0000 7f984f7fe640 1 -- 172.21.15.59:0/733177007 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858009450 con 0x7f98681614a0 2024-03-06T07:25:09.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.801+0000 7f986d581640 1 -- 172.21.15.59:0/733177007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986814a990 con 0x7f98681614a0 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.803+0000 7f984f7fe640 1 -- 172.21.15.59:0/733177007 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f985803e610 con 0x7f98681614a0 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.803+0000 7f984f7fe640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f985406b4d0 0x7f985406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.803+0000 7f984f7fe640 1 -- 172.21.15.59:0/733177007 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98580acab0 con 0x7f98681614a0 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.803+0000 7f9866ffd640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f985406b4d0 0x7f985406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.804+0000 7f9866ffd640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f985406b4d0 0x7f985406d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9860005880 tx=0x7f9860009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.804+0000 7f987016d740 1 Processor -- start 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.804+0000 7f987016d740 1 -- start start 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f987016d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b78972b870 0x55b78972bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f987016d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78972f8c0 0x55b789731360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f987016d740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b78975b600 0x55b78975ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f987016d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7896956b0 con 0x55b78972f8c0 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f987016d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b78963ea60 con 0x55b78972b870 2024-03-06T07:25:09.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f987016d740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b789691750 con 0x55b78975b600 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f98667fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78972f8c0 0x55b789731360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f98667fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78972f8c0 0x55b789731360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54776/0 (socket says 172.21.15.59:54776) 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f98667fc640 1 -- 172.21.15.59:0/601337285 learned_addr learned my addr 172.21.15.59:0/601337285 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f98677fe640 1 --2- 172.21.15.59:0/601337285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b78975b600 0x55b78975ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f9866ffd640 1 --2- 172.21.15.59:0/601337285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b78972b870 0x55b78972bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f98667fc640 1 -- 172.21.15.59:0/601337285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b78975b600 msgr2=0x55b78975ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f98667fc640 1 --2- 172.21.15.59:0/601337285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b78975b600 0x55b78975ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f98667fc640 1 -- 172.21.15.59:0/601337285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b78972b870 msgr2=0x55b78972bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f98667fc640 1 --2- 172.21.15.59:0/601337285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b78972b870 0x55b78972bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f98667fc640 1 -- 172.21.15.59:0/601337285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b78965cf30 con 0x55b78972f8c0 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f98667fc640 1 --2- 172.21.15.59:0/601337285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78972f8c0 0x55b789731360 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f985c009bf0 tx=0x7f985c0112c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b176258e416189ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.805+0000 7f984cff9640 1 -- 172.21.15.59:0/601337285 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c019070 con 0x55b78972f8c0 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.806+0000 7f984cff9640 1 -- 172.21.15.59:0/601337285 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c011dc0 con 0x55b78972f8c0 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.806+0000 7f984cff9640 1 -- 172.21.15.59:0/601337285 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c004270 con 0x55b78972f8c0 2024-03-06T07:25:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.806+0000 7f987016d740 1 -- 172.21.15.59:0/601337285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78972f8c0 msgr2=0x55b789731360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.806+0000 7f987016d740 1 --2- 172.21.15.59:0/601337285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78972f8c0 0x55b789731360 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f985c009bf0 tx=0x7f985c0112c0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.806+0000 7f987016d740 1 -- 172.21.15.59:0/601337285 shutdown_connections 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.806+0000 7f987016d740 1 --2- 172.21.15.59:0/601337285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b78975b600 0x55b78975ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.806+0000 7f987016d740 1 --2- 172.21.15.59:0/601337285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78972f8c0 0x55b789731360 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.806+0000 7f987016d740 1 --2- 172.21.15.59:0/601337285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b78972b870 0x55b78972bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.806+0000 7f987016d740 1 -- 172.21.15.59:0/601337285 >> 172.21.15.59:0/601337285 conn(0x55b78972d8f0 msgr2=0x55b78972f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.806+0000 7f987016d740 1 -- 172.21.15.59:0/601337285 shutdown_connections 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.806+0000 7f987016d740 1 -- 172.21.15.59:0/601337285 wait complete. 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.807+0000 7f987016d740 1 -- start start 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.807+0000 7f987016d740 1 Processor -- start 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f987016d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b78975b600 0x55b78975ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f987016d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789774560 0x55b789772e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f987016d740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b789773390 0x55b789776ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f987016d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b78972fd50 con 0x55b789774560 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f987016d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b789651ea0 con 0x55b78975b600 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f987016d740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b789730030 con 0x55b789773390 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f987016d740 10 client.0 _ll_register_callbacks cb 0x55b789651870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f98667fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789774560 0x55b789772e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f98667fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789774560 0x55b789772e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54784/0 (socket says 172.21.15.59:54784) 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f98667fc640 1 -- 172.21.15.59:0/434403550 learned_addr learned my addr 172.21.15.59:0/434403550 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f9866ffd640 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b78975b600 0x55b78975ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f98677fe640 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b789773390 0x55b789776ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f98667fc640 1 -- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b789773390 msgr2=0x55b789776ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f98667fc640 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b789773390 0x55b789776ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f98667fc640 1 -- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b78975b600 msgr2=0x55b78975ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f98667fc640 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b78975b600 0x55b78975ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f98667fc640 1 -- 172.21.15.59:0/434403550 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b789652270 con 0x55b789774560 2024-03-06T07:25:09.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f98667fc640 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789774560 0x55b789772e50 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f985c018000 tx=0x7f985c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712bf91b37308f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f9845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f9845ffb640 1 -- 172.21.15.59:0/434403550 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c019050 con 0x55b789774560 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f9845ffb640 1 -- 172.21.15.59:0/434403550 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f985c003820 con 0x55b789774560 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.808+0000 7f987016d740 1 -- 172.21.15.59:0/434403550 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f985c011dc0 con 0x55b789774560 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f9845ffb640 1 -- 172.21.15.59:0/434403550 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f985c030050 con 0x55b789774560 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f987016d740 10 client.6726 fetch_fsmap learned FSMap version 35 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f987016d740 1 -- 172.21.15.59:0/434403550 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55b7893aa2d0 con 0x55b789774560 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f9845ffb640 1 -- 172.21.15.59:0/434403550 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c011dc0 con 0x55b789774560 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f9845ffb640 1 -- 172.21.15.59:0/434403550 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 35) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f985c033020 con 0x55b789774560 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f987016d740 10 client.6726 fetch_fsmap finished waiting for FSMap version 35 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f987016d740 10 client.6726 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f987016d740 20 client.6726 populate_metadata read hostname 'smithi059' 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f987016d740 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:6835/250576205,v1:172.21.15.164:6837/250576205] conn(0x55b789792f40 0x55b789795330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f987016d740 4 client.6726 mds_command: new command op to 6579 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:09.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f987016d740 1 -- 172.21.15.59:0/434403550 --> [v2:172.21.15.164:6835/250576205,v1:172.21.15.164:6837/250576205] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b7896a2bf0 con 0x55b789792f40 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f9866ffd640 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:6835/250576205,v1:172.21.15.164:6837/250576205] conn(0x55b789792f40 0x55b789795330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f9866ffd640 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:6835/250576205,v1:172.21.15.164:6837/250576205] conn(0x55b789792f40 0x55b789795330 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c56e7c8c9bb4a3c server_cookie=9f18f894446e472a in_seq=0 out_seq=0 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.809+0000 7f9845ffb640 10 client.6726 ms_handle_connect on v2:172.21.15.164:6835/250576205 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.810+0000 7f9845ffb640 1 -- 172.21.15.59:0/434403550 <== mds.0 v2:172.21.15.164:6835/250576205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b7896a2bf0 con 0x55b789792f40 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.810+0000 7f9845ffb640 10 client.6726 handle_command_reply: tid=0 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.810+0000 7f987016d740 1 client.6726 shutdown 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.810+0000 7f987016d740 1 -- 172.21.15.59:0/434403550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789774560 msgr2=0x55b789772e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.811+0000 7f987016d740 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789774560 0x55b789772e50 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f985c018000 tx=0x7f985c004730 comp rx=0 tx=0).stop 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.811+0000 7f987016d740 1 -- 172.21.15.59:0/434403550 shutdown_connections 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.811+0000 7f987016d740 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:6835/250576205,v1:172.21.15.164:6837/250576205] conn(0x55b789792f40 0x55b789795330 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.811+0000 7f987016d740 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b789773390 0x55b789776ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.811+0000 7f987016d740 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789774560 0x55b789772e50 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.811+0000 7f987016d740 1 --2- 172.21.15.59:0/434403550 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b78975b600 0x55b78975ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.811+0000 7f9845ffb640 0 client.6726 ms_handle_reset on v2:172.21.15.164:6835/250576205 2024-03-06T07:25:09.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.811+0000 7f987016d740 1 -- 172.21.15.59:0/434403550 >> 172.21.15.59:0/434403550 conn(0x55b78972b870 msgr2=0x55b78972a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.811+0000 7f987016d740 1 -- 172.21.15.59:0/434403550 shutdown_connections 2024-03-06T07:25:09.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.811+0000 7f987016d740 1 -- 172.21.15.59:0/434403550 wait complete. 2024-03-06T07:25:09.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.811+0000 7f987016d740 20 client.6726 trim_cache size 0 max 16384 2024-03-06T07:25:09.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.828+0000 7f987016d740 1 Processor -- start 2024-03-06T07:25:09.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.828+0000 7f987016d740 1 -- start start 2024-03-06T07:25:09.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.828+0000 7f987016d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78975b600 0x55b78972aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.828+0000 7f987016d740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b789773390 0x55b78975b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.828+0000 7f987016d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b789774560 0x55b78975bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.828+0000 7f987016d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98680cb740 con 0x55b78975b600 2024-03-06T07:25:09.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.828+0000 7f987016d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98680cdc90 con 0x55b789774560 2024-03-06T07:25:09.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.828+0000 7f987016d740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98680cb140 con 0x55b789773390 2024-03-06T07:25:09.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9866ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78975b600 0x55b78972aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9866ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78975b600 0x55b78972aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54792/0 (socket says 172.21.15.59:54792) 2024-03-06T07:25:09.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9866ffd640 1 -- 172.21.15.59:0/3301857123 learned_addr learned my addr 172.21.15.59:0/3301857123 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f98667fc640 1 --2- 172.21.15.59:0/3301857123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b789773390 0x55b78975b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f98677fe640 1 --2- 172.21.15.59:0/3301857123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b789774560 0x55b78975bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9866ffd640 1 -- 172.21.15.59:0/3301857123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b789773390 msgr2=0x55b78975b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9866ffd640 1 --2- 172.21.15.59:0/3301857123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b789773390 0x55b78975b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9866ffd640 1 -- 172.21.15.59:0/3301857123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b789774560 msgr2=0x55b78975bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9866ffd640 1 --2- 172.21.15.59:0/3301857123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b789774560 0x55b78975bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9866ffd640 1 -- 172.21.15.59:0/3301857123 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7894746b0 con 0x55b78975b600 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9866ffd640 1 --2- 172.21.15.59:0/3301857123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78975b600 0x55b78972aa10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x55b789773790 tx=0x7f9860000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b50ef72101719d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9847fff640 1 -- 172.21.15.59:0/3301857123 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986002e070 con 0x55b78975b600 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9847fff640 1 -- 172.21.15.59:0/3301857123 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9860028d50 con 0x55b78975b600 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.829+0000 7f9847fff640 1 -- 172.21.15.59:0/3301857123 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9860028150 con 0x55b78975b600 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.830+0000 7f987016d740 1 -- 172.21.15.59:0/3301857123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78975b600 msgr2=0x55b78972aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.830+0000 7f987016d740 1 --2- 172.21.15.59:0/3301857123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78975b600 0x55b78972aa10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x55b789773790 tx=0x7f9860000ed0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.830+0000 7f987016d740 1 -- 172.21.15.59:0/3301857123 shutdown_connections 2024-03-06T07:25:09.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.830+0000 7f987016d740 1 --2- 172.21.15.59:0/3301857123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b789774560 0x55b78975bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.830+0000 7f987016d740 1 --2- 172.21.15.59:0/3301857123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b789773390 0x55b78975b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.830+0000 7f987016d740 1 --2- 172.21.15.59:0/3301857123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b78975b600 0x55b78972aa10 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.830+0000 7f987016d740 1 -- 172.21.15.59:0/3301857123 >> 172.21.15.59:0/3301857123 conn(0x55b78972b870 msgr2=0x55b78972f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.830+0000 7f987016d740 1 -- 172.21.15.59:0/3301857123 shutdown_connections 2024-03-06T07:25:09.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.830+0000 7f987016d740 1 -- 172.21.15.59:0/3301857123 wait complete. 2024-03-06T07:25:09.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.830+0000 7f987016d740 1 -- start start 2024-03-06T07:25:09.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.830+0000 7f987016d740 1 Processor -- start 2024-03-06T07:25:09.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f987016d740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b78975b600 0x55b78975c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f987016d740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789773390 0x55b78975cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f987016d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b789774560 0x55b789757190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f987016d740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b78975d640 con 0x55b789773390 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f987016d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b789772930 con 0x55b789774560 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f987016d740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b789744da0 con 0x55b78975b600 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f987016d740 10 client.0 _ll_register_callbacks cb 0x7f985c004360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f98667fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789773390 0x55b78975cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f98667fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789773390 0x55b78975cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54804/0 (socket says 172.21.15.59:54804) 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f98667fc640 1 -- 172.21.15.59:0/2146548411 learned_addr learned my addr 172.21.15.59:0/2146548411 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f9866ffd640 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b78975b600 0x55b78975c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f98677fe640 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b789774560 0x55b789757190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f98667fc640 1 -- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b78975b600 msgr2=0x55b78975c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f98667fc640 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b78975b600 0x55b78975c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f98667fc640 1 -- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b789774560 msgr2=0x55b789757190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f98667fc640 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b789774560 0x55b789757190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.831+0000 7f98667fc640 1 -- 172.21.15.59:0/2146548411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b78975a770 con 0x55b789773390 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f98667fc640 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789773390 0x55b78975cee0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f985c0092a0 tx=0x7f985c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1fabd86a19e3d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f9845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:25:09.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f9845ffb640 1 -- 172.21.15.59:0/2146548411 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c019070 con 0x55b789773390 2024-03-06T07:25:09.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f9845ffb640 1 -- 172.21.15.59:0/2146548411 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f985c024040 con 0x55b789773390 2024-03-06T07:25:09.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f987016d740 1 -- 172.21.15.59:0/2146548411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9860028d50 con 0x55b789773390 2024-03-06T07:25:09.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f9845ffb640 1 -- 172.21.15.59:0/2146548411 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f985c02d020 con 0x55b789773390 2024-03-06T07:25:09.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f987016d740 10 client.6732 fetch_fsmap learned FSMap version 35 2024-03-06T07:25:09.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f987016d740 1 -- 172.21.15.59:0/2146548411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55b789789600 con 0x55b789773390 2024-03-06T07:25:09.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f9845ffb640 1 -- 172.21.15.59:0/2146548411 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9860028d50 con 0x55b789773390 2024-03-06T07:25:09.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f9845ffb640 1 -- 172.21.15.59:0/2146548411 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 35) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f985c033070 con 0x55b789773390 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f987016d740 10 client.6732 fetch_fsmap finished waiting for FSMap version 35 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f987016d740 10 client.6732 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f987016d740 20 client.6732 populate_metadata read hostname 'smithi059' 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f987016d740 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:6835/250576205,v1:172.21.15.164:6837/250576205] conn(0x55b789757d40 0x55b789730bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f987016d740 4 client.6732 mds_command: new command op to 6579 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f987016d740 1 -- 172.21.15.59:0/2146548411 --> [v2:172.21.15.164:6835/250576205,v1:172.21.15.164:6837/250576205] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b78969cf40 con 0x55b789757d40 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.832+0000 7f9866ffd640 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:6835/250576205,v1:172.21.15.164:6837/250576205] conn(0x55b789757d40 0x55b789730bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f9866ffd640 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:6835/250576205,v1:172.21.15.164:6837/250576205] conn(0x55b789757d40 0x55b789730bc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad8c6d1a866ed85e server_cookie=500fdfd2c42104b7 in_seq=0 out_seq=0 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f9845ffb640 10 client.6732 ms_handle_connect on v2:172.21.15.164:6835/250576205 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f9845ffb640 1 -- 172.21.15.59:0/2146548411 <== mds.0 v2:172.21.15.164:6835/250576205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55b78969cf40 con 0x55b789757d40 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f9845ffb640 10 client.6732 handle_command_reply: tid=0 2024-03-06T07:25:09.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 1 client.6732 shutdown 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 1 -- 172.21.15.59:0/2146548411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789773390 msgr2=0x55b78975cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789773390 0x55b78975cee0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f985c0092a0 tx=0x7f985c009190 comp rx=0 tx=0).stop 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 1 -- 172.21.15.59:0/2146548411 shutdown_connections 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b789774560 0x55b789757190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55b789773390 0x55b78975cee0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:6835/250576205,v1:172.21.15.164:6837/250576205] conn(0x55b789757d40 0x55b789730bc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 1 --2- 172.21.15.59:0/2146548411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55b78975b600 0x55b78975c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f9845ffb640 0 client.6732 ms_handle_reset on v2:172.21.15.164:6835/250576205 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 1 -- 172.21.15.59:0/2146548411 >> 172.21.15.59:0/2146548411 conn(0x55b78972b870 msgr2=0x55b78975bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 1 -- 172.21.15.59:0/2146548411 shutdown_connections 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 1 -- 172.21.15.59:0/2146548411 wait complete. 2024-03-06T07:25:09.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.833+0000 7f987016d740 20 client.6732 trim_cache size 0 max 16384 2024-03-06T07:25:09.842 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:25:09.842 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:25:09.842 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:25:09.842 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 6579, 2024-03-06T07:25:09.842 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:25:09.842 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:25:09.842 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:25:09.842 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 4.681114354, 2024-03-06T07:25:09.842 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 35, 2024-03-06T07:25:09.842 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 70, 2024-03-06T07:25:09.843 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 70, 2024-03-06T07:25:09.843 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 9.6933117039999992 2024-03-06T07:25:09.843 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:25:09.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 -- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f985406b4d0 msgr2=0x7f985406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f985406b4d0 0x7f985406d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9860005880 tx=0x7f9860009040 comp rx=0 tx=0).stop 2024-03-06T07:25:09.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 -- 172.21.15.59:0/733177007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98681614a0 msgr2=0x7f986815bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98681614a0 0x7f986815bbd0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9858009eb0 tx=0x7f9858004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:09.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 -- 172.21.15.59:0/733177007 shutdown_connections 2024-03-06T07:25:09.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f985406b4d0 0x7f985406d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98681614a0 0x7f986815bbd0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f98680b1850 0x7f986814a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 --2- 172.21.15.59:0/733177007 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f986806d3c0 0x7f986814aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 -- 172.21.15.59:0/733177007 >> 172.21.15.59:0/733177007 conn(0x7f986809a9f0 msgr2=0x7f98681603f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 -- 172.21.15.59:0/733177007 shutdown_connections 2024-03-06T07:25:09.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.834+0000 7f986d581640 1 -- 172.21.15.59:0/733177007 wait complete. 2024-03-06T07:25:09.847 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:25:09.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.946+0000 7f051285c640 1 Processor -- start 2024-03-06T07:25:09.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.946+0000 7f051285c640 1 -- start start 2024-03-06T07:25:09.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.947+0000 7f051285c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f050c0cf850 0x7f050c0cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.947+0000 7f051285c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c15e380 con 0x7f050c06cf20 2024-03-06T07:25:09.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.947+0000 7f051285c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c0cdc10 con 0x7f050c0d00c0 2024-03-06T07:25:09.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.947+0000 7f051285c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f050c06ca90 con 0x7f050c0cf850 2024-03-06T07:25:09.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.947+0000 7f050b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f050c0cf850 0x7f050c0cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.947+0000 7f0510df9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f050c06cf20 0x7f050c06d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58820/0 (socket says 172.21.15.59:58820) 2024-03-06T07:25:09.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.947+0000 7f0510df9640 1 -- 172.21.15.59:0/317442501 learned_addr learned my addr 172.21.15.59:0/317442501 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:09.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.948+0000 7f050b7fe640 1 -- 172.21.15.59:0/317442501 >> v1:172.21.15.164:6789/0 conn(0x7f050c0d00c0 legacy=0x7f050c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.948+0000 7f050b7fe640 1 -- 172.21.15.59:0/317442501 >> v1:172.21.15.59:6789/0 conn(0x7f050c06cf20 legacy=0x7f050c06d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.948+0000 7f050affd640 1 -- 172.21.15.59:0/317442501 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124574072 0 0) 0x7f050c0cdc10 con 0x7f050c0d00c0 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.948+0000 7f050b7fe640 1 -- 172.21.15.59:0/317442501 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c06c160 con 0x7f050c0cf850 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.948+0000 7f050b7fe640 1 --2- 172.21.15.59:0/317442501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f050c0cf850 0x7f050c0cd3c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0097a0 tx=0x7f04fc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33156efcbbfe43d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.948+0000 7f050affd640 1 -- 172.21.15.59:0/317442501 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc037070 con 0x7f050c0cf850 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.948+0000 7f050affd640 1 -- 172.21.15.59:0/317442501 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc030710 con 0x7f050c0cf850 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.949+0000 7f050affd640 1 -- 172.21.15.59:0/317442501 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc030ae0 con 0x7f050c0cf850 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.949+0000 7f051285c640 1 -- 172.21.15.59:0/317442501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f050c0cf850 msgr2=0x7f050c0cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.949+0000 7f051285c640 1 --2- 172.21.15.59:0/317442501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f050c0cf850 0x7f050c0cd3c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0097a0 tx=0x7f04fc02b790 comp rx=0 tx=0).stop 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.949+0000 7f051285c640 1 -- 172.21.15.59:0/317442501 shutdown_connections 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.949+0000 7f051285c640 1 --2- 172.21.15.59:0/317442501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f050c0cf850 0x7f050c0cd3c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.949+0000 7f051285c640 1 -- 172.21.15.59:0/317442501 >> 172.21.15.59:0/317442501 conn(0x7f050c0b35a0 msgr2=0x7f050c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.949+0000 7f051285c640 1 -- 172.21.15.59:0/317442501 shutdown_connections 2024-03-06T07:25:09.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.949+0000 7f051285c640 1 -- 172.21.15.59:0/317442501 wait complete. 2024-03-06T07:25:09.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.950+0000 7f051285c640 1 Processor -- start 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.950+0000 7f051285c640 1 -- start start 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.951+0000 7f051285c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f050c06cf20 0x7f050c151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.951+0000 7f051285c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f050c0cf850 0x7f050c0b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.951+0000 7f051285c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f050c0d00c0 0x7f050c0c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.951+0000 7f051285c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0cd900 con 0x7f050c06cf20 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.951+0000 7f051285c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c06c720 con 0x7f050c0d00c0 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.951+0000 7f051285c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0b1030 con 0x7f050c0cf850 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.951+0000 7f050bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f050c06cf20 0x7f050c151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.951+0000 7f050bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f050c06cf20 0x7f050c151890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54810/0 (socket says 172.21.15.59:54810) 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.951+0000 7f050bfff640 1 -- 172.21.15.59:0/1434578453 learned_addr learned my addr 172.21.15.59:0/1434578453 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.951+0000 7f0510df9640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f050c0d00c0 0x7f050c0c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.951+0000 7f050b7fe640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f050c0cf850 0x7f050c0b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.952+0000 7f050bfff640 1 -- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f050c0cf850 msgr2=0x7f050c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.952+0000 7f050bfff640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f050c0cf850 0x7f050c0b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.952+0000 7f050bfff640 1 -- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f050c0d00c0 msgr2=0x7f050c0c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.952+0000 7f050bfff640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f050c0d00c0 0x7f050c0c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.952+0000 7f050bfff640 1 -- 172.21.15.59:0/1434578453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c0b4c70 con 0x7f050c06cf20 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.952+0000 7f050bfff640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f050c06cf20 0x7f050c151890 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f04f400c410 tx=0x7f04f400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.952+0000 7f0508ff9640 1 -- 172.21.15.59:0/1434578453 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f4017070 con 0x7f050c06cf20 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.952+0000 7f0508ff9640 1 -- 172.21.15.59:0/1434578453 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f40035f0 con 0x7f050c06cf20 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.952+0000 7f051285c640 1 -- 172.21.15.59:0/1434578453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050c0b4f50 con 0x7f050c06cf20 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.952+0000 7f0508ff9640 1 -- 172.21.15.59:0/1434578453 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f40039e0 con 0x7f050c06cf20 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.952+0000 7f051285c640 1 -- 172.21.15.59:0/1434578453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050c151dd0 con 0x7f050c06cf20 2024-03-06T07:25:09.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.953+0000 7f051285c640 1 -- 172.21.15.59:0/1434578453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050c0b5120 con 0x7f050c06cf20 2024-03-06T07:25:09.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.953+0000 7f0508ff9640 1 -- 172.21.15.59:0/1434578453 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04f401a070 con 0x7f050c06cf20 2024-03-06T07:25:09.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.954+0000 7f0508ff9640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ec06b500 0x7f04ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:09.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.954+0000 7f0508ff9640 1 -- 172.21.15.59:0/1434578453 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04f4089600 con 0x7f050c06cf20 2024-03-06T07:25:09.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.954+0000 7f050b7fe640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ec06b500 0x7f04ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:09.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.954+0000 7f050b7fe640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ec06b500 0x7f04ec06d9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f04fc034000 tx=0x7f04fc02f000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:09.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:09.958+0000 7f0508ff9640 1 -- 172.21.15.59:0/1434578453 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f050c0b5120 con 0x7f050c06cf20 2024-03-06T07:25:10.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.179+0000 7f051285c640 1 -- 172.21.15.59:0/1434578453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f050c04f330 con 0x7f050c06cf20 2024-03-06T07:25:10.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.180+0000 7f0508ff9640 1 -- 172.21.15.59:0/1434578453 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) v1 ==== 53+0+73 (secure 0 0 0) 0x7f04f4057da0 con 0x7f050c06cf20 2024-03-06T07:25:10.182 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:25:10.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 -- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ec06b500 msgr2=0x7f04ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ec06b500 0x7f04ec06d9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f04fc034000 tx=0x7f04fc02f000 comp rx=0 tx=0).stop 2024-03-06T07:25:10.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 -- 172.21.15.59:0/1434578453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f050c06cf20 msgr2=0x7f050c151890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:10.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f050c06cf20 0x7f050c151890 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f04f400c410 tx=0x7f04f400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 -- 172.21.15.59:0/1434578453 shutdown_connections 2024-03-06T07:25:10.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ec06b500 0x7f04ec06d9c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f050c0d00c0 0x7f050c0c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f050c0cf850 0x7f050c0b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 --2- 172.21.15.59:0/1434578453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f050c06cf20 0x7f050c151890 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:10.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 -- 172.21.15.59:0/1434578453 >> 172.21.15.59:0/1434578453 conn(0x7f050c0b35a0 msgr2=0x7f050c0b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:10.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 -- 172.21.15.59:0/1434578453 shutdown_connections 2024-03-06T07:25:10.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:10.182+0000 7f051285c640 1 -- 172.21.15.59:0/1434578453 wait complete. 2024-03-06T07:25:10.194 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:25:10.194 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:25:10.194 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:25:10.195 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:25:10.195 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:10.195 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:25:10.195 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:25:10.195 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:25:10.195 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:10.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:10.195 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:25:10.210 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:10.211 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42469sec preferred_lft 42469sec 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:10.212 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:10.213 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:10.213 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:25:10.213 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:10.213 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:10.213 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:10.213 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:25:10.213 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:10.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:10.214 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:25:10.276 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:25:10.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:10.278 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:25:10.332 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:25:10.332 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:25:10.332 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:25:10.332 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:25:10.332 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:25:10.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:10.333 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:25:10.333 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:25:10.333 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:25:10.333 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:25:10.333 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:25:10.333 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:25:10.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:10.427 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:25:10.441 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:25:10.441 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:25:10.442 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:10.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:10.442 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:25:10.508 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:25:10.508 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:10.508 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:10.563 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:10.563 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:10.563 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:10.616 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:25:10.684 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:10.738 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:25:10.738 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:25:10.738 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:25:10.739 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:10.740 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:10.741 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:10.741 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:25:10.741 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:10.741 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:10.741 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:10.741 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:10.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:10.742 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:25:10.796 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:25:10.797 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:25:10.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:25:10.866 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:25:10.921 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:10.922 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:10.923 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:10.923 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:25:10.923 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:10.923 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:10.923 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:10.923 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:10.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:10.923 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:25:10.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:25:10.923+0000 7fab9effb480 -1 init, newargv = 0x557a4f7eb8b0 newargc=15 2024-03-06T07:25:10.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[62632]: starting ceph client 2024-03-06T07:25:10.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[62632]: starting fuse 2024-03-06T07:25:10.939 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:25:10.939 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:25:10.939 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:25:10.940 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:25:10.941 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:25:11.059 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.62632.asok 2024-03-06T07:25:11.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:11.070 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.62632.asok 2024-03-06T07:25:11.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:11.070 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.62632.asok status 2024-03-06T07:25:11.207 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:25:11.207 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "62632", 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 6343, 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:25:11.208 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 6343 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:0", 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3920053473 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "172.21.15.59:0", 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3920053473 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:11.209 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.6343 172.21.15.59:0/3920053473", 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "172.21.15.59:0/3920053473", 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 35, 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 70, 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:25:11.210 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:25:11.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:11.216 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:25:11.216 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:11.230 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:25:11.231 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:11.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:11.231 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:11.297 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-03-06T07:25:11.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.439+0000 7f2435488640 1 Processor -- start 2024-03-06T07:25:11.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.439+0000 7f2435488640 1 -- start start 2024-03-06T07:25:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f2435488640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f24300cea50 0x7f24300b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f2435488640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f243006cab0 0x7f24300ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f2435488640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f243006d030 con 0x7f243015def0 2024-03-06T07:25:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f2435488640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24300cf1c0 con 0x7f243006cab0 2024-03-06T07:25:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f2435488640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24300cc760 con 0x7f24300cea50 2024-03-06T07:25:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f242f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f243006cab0 0x7f24300ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f242f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f243006cab0 0x7f24300ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54812/0 (socket says 172.21.15.59:54812) 2024-03-06T07:25:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f242effd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f243015def0 0x7f24300ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58826/0 (socket says 172.21.15.59:58826) 2024-03-06T07:25:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f242f7fe640 1 -- 172.21.15.59:0/2786886982 learned_addr learned my addr 172.21.15.59:0/2786886982 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f2427fff640 1 --2- 172.21.15.59:0/2786886982 >> v2:172.21.15.164:3300/0 conn(0x7f24300cea50 0x7f24300b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f242f7fe640 1 -- 172.21.15.59:0/2786886982 >> v2:172.21.15.164:3300/0 conn(0x7f24300cea50 msgr2=0x7f24300b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f242f7fe640 1 --2- 172.21.15.59:0/2786886982 >> v2:172.21.15.164:3300/0 conn(0x7f24300cea50 0x7f24300b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f242f7fe640 1 -- 172.21.15.59:0/2786886982 >> v1:172.21.15.59:6789/0 conn(0x7f243015def0 legacy=0x7f24300ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.440+0000 7f242e7fc640 1 -- 172.21.15.59:0/2786886982 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193354399 0 0) 0x7f243006d030 con 0x7f243015def0 2024-03-06T07:25:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.441+0000 7f242f7fe640 1 -- 172.21.15.59:0/2786886982 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243006d310 con 0x7f243006cab0 2024-03-06T07:25:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.441+0000 7f242f7fe640 1 --2- 172.21.15.59:0/2786886982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f243006cab0 0x7f24300ccbf0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f24300b3660 tx=0x7f242802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea1d6618b7f4120 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.441+0000 7f242e7fc640 1 -- 172.21.15.59:0/2786886982 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242802cb20 con 0x7f243006cab0 2024-03-06T07:25:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.441+0000 7f242e7fc640 1 -- 172.21.15.59:0/2786886982 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242802ccc0 con 0x7f243006cab0 2024-03-06T07:25:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.441+0000 7f242e7fc640 1 -- 172.21.15.59:0/2786886982 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428034610 con 0x7f243006cab0 2024-03-06T07:25:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.442+0000 7f2435488640 1 -- 172.21.15.59:0/2786886982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f243006cab0 msgr2=0x7f24300ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.442+0000 7f2435488640 1 --2- 172.21.15.59:0/2786886982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f243006cab0 0x7f24300ccbf0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f24300b3660 tx=0x7f242802bf80 comp rx=0 tx=0).stop 2024-03-06T07:25:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.442+0000 7f2435488640 1 -- 172.21.15.59:0/2786886982 shutdown_connections 2024-03-06T07:25:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.442+0000 7f2435488640 1 --2- 172.21.15.59:0/2786886982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f243006cab0 0x7f24300ccbf0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.442+0000 7f2435488640 1 --2- 172.21.15.59:0/2786886982 >> v2:172.21.15.164:3300/0 conn(0x7f24300cea50 0x7f24300b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.442+0000 7f2435488640 1 -- 172.21.15.59:0/2786886982 >> 172.21.15.59:0/2786886982 conn(0x7f24300b3690 msgr2=0x7f24300b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.442+0000 7f2435488640 1 -- 172.21.15.59:0/2786886982 shutdown_connections 2024-03-06T07:25:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.442+0000 7f2435488640 1 -- 172.21.15.59:0/2786886982 wait complete. 2024-03-06T07:25:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.443+0000 7f2435488640 1 Processor -- start 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.443+0000 7f2435488640 1 -- start start 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f2435488640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f243006cab0 0x7f24300b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f2435488640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24300cea50 0x7f24300cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f2435488640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f243015def0 0x7f24300c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f2435488640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243015e2d0 con 0x7f243006cab0 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f2435488640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24300cc3f0 con 0x7f243015def0 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f2435488640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f243015e450 con 0x7f24300cea50 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f242effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f243006cab0 0x7f24300b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f242effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f243006cab0 0x7f24300b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54826/0 (socket says 172.21.15.59:54826) 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f242effd640 1 -- 172.21.15.59:0/1382110228 learned_addr learned my addr 172.21.15.59:0/1382110228 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f2427fff640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24300cea50 0x7f24300cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f242f7fe640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f243015def0 0x7f24300c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f242effd640 1 -- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24300cea50 msgr2=0x7f24300cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f242effd640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24300cea50 0x7f24300cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f242effd640 1 -- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f243015def0 msgr2=0x7f24300c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f242effd640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f243015def0 0x7f24300c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.444+0000 7f242effd640 1 -- 172.21.15.59:0/1382110228 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243015c6d0 con 0x7f243006cab0 2024-03-06T07:25:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.445+0000 7f242effd640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f243006cab0 0x7f24300b4780 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f242000c430 tx=0x7f242000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.445+0000 7f24277fe640 1 -- 172.21.15.59:0/1382110228 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24200074e0 con 0x7f243006cab0 2024-03-06T07:25:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.445+0000 7f24277fe640 1 -- 172.21.15.59:0/1382110228 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24200047e0 con 0x7f243006cab0 2024-03-06T07:25:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.445+0000 7f24277fe640 1 -- 172.21.15.59:0/1382110228 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420004bd0 con 0x7f243006cab0 2024-03-06T07:25:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.445+0000 7f2435488640 1 -- 172.21.15.59:0/1382110228 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24300b58e0 con 0x7f243006cab0 2024-03-06T07:25:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.445+0000 7f2435488640 1 -- 172.21.15.59:0/1382110228 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24300b5ea0 con 0x7f243006cab0 2024-03-06T07:25:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.446+0000 7f2435488640 1 -- 172.21.15.59:0/1382110228 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f243009fcf0 con 0x7f243006cab0 2024-03-06T07:25:11.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.446+0000 7f24277fe640 1 -- 172.21.15.59:0/1382110228 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2420011070 con 0x7f243006cab0 2024-03-06T07:25:11.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.447+0000 7f24277fe640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f241006b4d0 0x7f241006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:11.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.447+0000 7f24277fe640 1 -- 172.21.15.59:0/1382110228 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2420089ca0 con 0x7f243006cab0 2024-03-06T07:25:11.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.447+0000 7f2427fff640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f241006b4d0 0x7f241006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:11.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.447+0000 7f2427fff640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f241006b4d0 0x7f241006d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f24300b50b0 tx=0x7f2418009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:11.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.451+0000 7f24277fe640 1 -- 172.21.15.59:0/1382110228 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24200584c0 con 0x7f243006cab0 2024-03-06T07:25:11.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.629+0000 7f2435488640 1 -- 172.21.15.59:0/1382110228 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f2430068ff0 con 0x7f243006cab0 2024-03-06T07:25:11.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.632+0000 7f24277fe640 1 -- 172.21.15.59:0/1382110228 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7f242005c560 con 0x7f243006cab0 2024-03-06T07:25:11.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 -- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f241006b4d0 msgr2=0x7f241006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f241006b4d0 0x7f241006d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f24300b50b0 tx=0x7f2418009190 comp rx=0 tx=0).stop 2024-03-06T07:25:11.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 -- 172.21.15.59:0/1382110228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f243006cab0 msgr2=0x7f24300b4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:11.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f243006cab0 0x7f24300b4780 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f242000c430 tx=0x7f242000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 -- 172.21.15.59:0/1382110228 shutdown_connections 2024-03-06T07:25:11.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f241006b4d0 0x7f241006d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f243015def0 0x7f24300c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24300cea50 0x7f24300cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 --2- 172.21.15.59:0/1382110228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f243006cab0 0x7f24300b4780 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:11.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 -- 172.21.15.59:0/1382110228 >> 172.21.15.59:0/1382110228 conn(0x7f24300b3690 msgr2=0x7f243006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:11.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 -- 172.21.15.59:0/1382110228 shutdown_connections 2024-03-06T07:25:11.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:11.634+0000 7f2435488640 1 -- 172.21.15.59:0/1382110228 wait complete. 2024-03-06T07:25:19.604 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:19.607 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:21.649 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.760+0000 7fa851c66640 1 Processor -- start 2024-03-06T07:25:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.760+0000 7fa851c66640 1 -- start start 2024-03-06T07:25:21.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa851c66640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa84c1621c0 0x7fa84c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa851c66640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c1625a0 0x7fa84c15dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa851c66640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84c0ceac0 con 0x7fa84c0d4920 2024-03-06T07:25:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa851c66640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c0cf690 con 0x7fa84c1621c0 2024-03-06T07:25:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa851c66640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c0d2dc0 con 0x7fa84c1625a0 2024-03-06T07:25:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa84b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa84c1621c0 0x7fa84c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa84bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c1625a0 0x7fa84c15dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa84affd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa84c0d4920 0x7fa84c0d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:50038/0 (socket says 172.21.15.59:50038) 2024-03-06T07:25:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa84affd640 1 -- 172.21.15.59:0/556222865 learned_addr learned my addr 172.21.15.59:0/556222865 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa84bfff640 1 --2- 172.21.15.59:0/556222865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c1625a0 0x7fa84c15dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:57000/0 (socket says 172.21.15.59:57000) 2024-03-06T07:25:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa84bfff640 1 -- 172.21.15.59:0/556222865 >> v2:172.21.15.164:3300/0 conn(0x7fa84c1621c0 msgr2=0x7fa84c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa84bfff640 1 --2- 172.21.15.59:0/556222865 >> v2:172.21.15.164:3300/0 conn(0x7fa84c1621c0 0x7fa84c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa84bfff640 1 -- 172.21.15.59:0/556222865 >> v1:172.21.15.164:6789/0 conn(0x7fa84c0d4920 legacy=0x7fa84c0d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa84bfff640 1 -- 172.21.15.59:0/556222865 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c0ced60 con 0x7fa84c1625a0 2024-03-06T07:25:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.761+0000 7fa84b7fe640 1 --2- 172.21.15.59:0/556222865 >> v2:172.21.15.164:3300/0 conn(0x7fa84c1621c0 0x7fa84c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:21.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.762+0000 7fa84bfff640 1 --2- 172.21.15.59:0/556222865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c1625a0 0x7fa84c15dd40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa844009d90 tx=0x7fa84402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e933a518d67e741 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.762+0000 7fa84a7fc640 1 -- 172.21.15.59:0/556222865 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844039070 con 0x7fa84c1625a0 2024-03-06T07:25:21.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.762+0000 7fa84a7fc640 1 -- 172.21.15.59:0/556222865 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84402cc50 con 0x7fa84c1625a0 2024-03-06T07:25:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.762+0000 7fa84a7fc640 1 -- 172.21.15.59:0/556222865 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844034440 con 0x7fa84c1625a0 2024-03-06T07:25:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.763+0000 7fa851c66640 1 -- 172.21.15.59:0/556222865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c1625a0 msgr2=0x7fa84c15dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.763+0000 7fa851c66640 1 --2- 172.21.15.59:0/556222865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c1625a0 0x7fa84c15dd40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa844009d90 tx=0x7fa84402bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.763+0000 7fa851c66640 1 -- 172.21.15.59:0/556222865 shutdown_connections 2024-03-06T07:25:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.763+0000 7fa851c66640 1 --2- 172.21.15.59:0/556222865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c1625a0 0x7fa84c15dd40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.763+0000 7fa851c66640 1 --2- 172.21.15.59:0/556222865 >> v2:172.21.15.164:3300/0 conn(0x7fa84c1621c0 0x7fa84c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.763+0000 7fa851c66640 1 -- 172.21.15.59:0/556222865 >> 172.21.15.59:0/556222865 conn(0x7fa84c0d3d10 msgr2=0x7fa84c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.763+0000 7fa851c66640 1 -- 172.21.15.59:0/556222865 shutdown_connections 2024-03-06T07:25:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.763+0000 7fa851c66640 1 -- 172.21.15.59:0/556222865 wait complete. 2024-03-06T07:25:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.764+0000 7fa851c66640 1 Processor -- start 2024-03-06T07:25:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.764+0000 7fa851c66640 1 -- start start 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.765+0000 7fa851c66640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c0d4920 0x7fa84c14c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.765+0000 7fa851c66640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa84c1621c0 0x7fa84c14a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.765+0000 7fa851c66640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1625a0 0x7fa84c14a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.765+0000 7fa851c66640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c15e280 con 0x7fa84c1625a0 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.765+0000 7fa851c66640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c0cf320 con 0x7fa84c1621c0 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.765+0000 7fa851c66640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c15e400 con 0x7fa84c0d4920 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.765+0000 7fa84bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1625a0 0x7fa84c14a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.765+0000 7fa84bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1625a0 0x7fa84c14a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53548/0 (socket says 172.21.15.59:53548) 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.765+0000 7fa84b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c0d4920 0x7fa84c14c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.765+0000 7fa84bfff640 1 -- 172.21.15.59:0/1164140076 learned_addr learned my addr 172.21.15.59:0/1164140076 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.765+0000 7fa84affd640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa84c1621c0 0x7fa84c14a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa84bfff640 1 -- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c0d4920 msgr2=0x7fa84c14c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa84bfff640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c0d4920 0x7fa84c14c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa84bfff640 1 -- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa84c1621c0 msgr2=0x7fa84c14a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa84bfff640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa84c1621c0 0x7fa84c14a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa84bfff640 1 -- 172.21.15.59:0/1164140076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c1592b0 con 0x7fa84c1625a0 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa84affd640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa84c1621c0 0x7fa84c14a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa84b7fe640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c0d4920 0x7fa84c14c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa84bfff640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1625a0 0x7fa84c14a970 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa844009eb0 tx=0x7fa844004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa833fff640 1 -- 172.21.15.59:0/1164140076 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844039050 con 0x7fa84c1625a0 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa833fff640 1 -- 172.21.15.59:0/1164140076 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa844045070 con 0x7fa84c1625a0 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa833fff640 1 -- 172.21.15.59:0/1164140076 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa844009450 con 0x7fa84c1625a0 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa851c66640 1 -- 172.21.15.59:0/1164140076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84c149390 con 0x7fa84c1625a0 2024-03-06T07:25:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.766+0000 7fa851c66640 1 -- 172.21.15.59:0/1164140076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c149950 con 0x7fa84c1625a0 2024-03-06T07:25:21.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.767+0000 7fa851c66640 1 -- 172.21.15.59:0/1164140076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c04f330 con 0x7fa84c1625a0 2024-03-06T07:25:21.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.767+0000 7fa833fff640 1 -- 172.21.15.59:0/1164140076 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa84403e610 con 0x7fa84c1625a0 2024-03-06T07:25:21.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.768+0000 7fa833fff640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa83806b4d0 0x7fa83806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.768+0000 7fa833fff640 1 -- 172.21.15.59:0/1164140076 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8440acb70 con 0x7fa84c1625a0 2024-03-06T07:25:21.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.768+0000 7fa84b7fe640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa83806b4d0 0x7fa83806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.769+0000 7fa84b7fe640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa83806b4d0 0x7fa83806d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009560 tx=0x7fa83c009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.772+0000 7fa833fff640 1 -- 172.21.15.59:0/1164140076 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa84407b3a0 con 0x7fa84c1625a0 2024-03-06T07:25:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.994+0000 7fa851c66640 1 -- 172.21.15.59:0/1164140076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa84c15cbf0 con 0x7fa84c1625a0 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.994+0000 7fa833fff640 1 -- 172.21.15.59:0/1164140076 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fa84407f440 con 0x7fa84c1625a0 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:04.517350+0000","modified":"2024-03-06T07:25:06.518231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/250576205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":250576205},{"type":"v1","addr":"172.21.15.164:6837","nonce":250576205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 35 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 -- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa83806b4d0 msgr2=0x7fa83806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa83806b4d0 0x7fa83806d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009560 tx=0x7fa83c009190 comp rx=0 tx=0).stop 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 -- 172.21.15.59:0/1164140076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1625a0 msgr2=0x7fa84c14a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1625a0 0x7fa84c14a970 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa844009eb0 tx=0x7fa844004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 -- 172.21.15.59:0/1164140076 shutdown_connections 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa83806b4d0 0x7fa83806d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1625a0 0x7fa84c14a970 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa84c1621c0 0x7fa84c14a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 --2- 172.21.15.59:0/1164140076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa84c0d4920 0x7fa84c14c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 -- 172.21.15.59:0/1164140076 >> 172.21.15.59:0/1164140076 conn(0x7fa84c0d3d10 msgr2=0x7fa84c15fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 -- 172.21.15.59:0/1164140076 shutdown_connections 2024-03-06T07:25:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:21.996+0000 7fa851c66640 1 -- 172.21.15.59:0/1164140076 wait complete. 2024-03-06T07:25:22.009 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:25:22.125 INFO:teuthology.orchestra.run.smithi164.stdout:{"mds_cache":{"strays_created":0}} 2024-03-06T07:25:22.125 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-03-06T07:25:22.125 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:25:22.125 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> stdin-killer -- python3 -c ' 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> import uuid 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> n = 100 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> create_hard_links = False 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> if 0 > 0: 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> create_hard_links = True 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> dpath = os.path.dirname(path) 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:22.141 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> try: 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> for i in range(n): 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> f.write(f"{i}") 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> f.flush() 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(f.fileno()) 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> if True: 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> os.unlink(fpath) 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(dirfd) 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> if create_hard_links: 2024-03-06T07:25:22.142 DEBUG:teuthology.orchestra.run.smithi059:> for j in range(0): 2024-03-06T07:25:22.143 DEBUG:teuthology.orchestra.run.smithi059:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:22.143 DEBUG:teuthology.orchestra.run.smithi059:> if True: 2024-03-06T07:25:22.143 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(dirfd) 2024-03-06T07:25:22.143 DEBUG:teuthology.orchestra.run.smithi059:> finally: 2024-03-06T07:25:22.143 DEBUG:teuthology.orchestra.run.smithi059:> os.close(dirfd) 2024-03-06T07:25:22.143 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:25:22.290 INFO:teuthology.orchestra.run.smithi059.stderr:Traceback (most recent call last): 2024-03-06T07:25:22.290 INFO:teuthology.orchestra.run.smithi059.stderr: File "", line 26, in 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi059.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-03-06T07:25:22.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:25:22.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:22.300 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:22.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.439+0000 7f2a06ad4640 1 Processor -- start 2024-03-06T07:25:22.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.439+0000 7f2a06ad4640 1 -- start start 2024-03-06T07:25:22.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.440+0000 7f2a06ad4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2a0006b220 0x7f2a0015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.440+0000 7f2a06ad4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2a000ce920 0x7f2a00071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.440+0000 7f2a06ad4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0015ea30 con 0x7f2a00071ec0 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.440+0000 7f2a06ad4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a00161b70 con 0x7f2a000ce920 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.440+0000 7f2a06ad4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0015ed10 con 0x7f2a0006b220 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.440+0000 7f29fffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2a000ce920 0x7f2a00071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.440+0000 7f2a05071640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2a00071ec0 0x7f2a00162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41952/0 (socket says 172.21.15.59:41952) 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.440+0000 7f2a05071640 1 -- 172.21.15.59:0/4011482045 learned_addr learned my addr 172.21.15.59:0/4011482045 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.440+0000 7f2a04870640 1 --2- 172.21.15.59:0/4011482045 >> v2:172.21.15.164:3300/0 conn(0x7f2a0006b220 0x7f2a0015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.441+0000 7f29fffff640 1 -- 172.21.15.59:0/4011482045 >> v2:172.21.15.164:3300/0 conn(0x7f2a0006b220 msgr2=0x7f2a0015e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.441+0000 7f29fffff640 1 --2- 172.21.15.59:0/4011482045 >> v2:172.21.15.164:3300/0 conn(0x7f2a0006b220 0x7f2a0015e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.441+0000 7f29fffff640 1 -- 172.21.15.59:0/4011482045 >> v1:172.21.15.59:6789/0 conn(0x7f2a00071ec0 legacy=0x7f2a00162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.441+0000 7f29ff7fe640 1 -- 172.21.15.59:0/4011482045 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680416485 0 0) 0x7f2a0015ea30 con 0x7f2a00071ec0 2024-03-06T07:25:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.441+0000 7f29fffff640 1 -- 172.21.15.59:0/4011482045 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0015eff0 con 0x7f2a000ce920 2024-03-06T07:25:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.441+0000 7f29fffff640 1 --2- 172.21.15.59:0/4011482045 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a000ce920 0x7f2a00071980 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2a000d0980 tx=0x7f29f002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed3269375c90f98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.441+0000 7f29ff7fe640 1 -- 172.21.15.59:0/4011482045 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0039070 con 0x7f2a000ce920 2024-03-06T07:25:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.441+0000 7f29ff7fe640 1 -- 172.21.15.59:0/4011482045 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f002cc80 con 0x7f2a000ce920 2024-03-06T07:25:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.441+0000 7f29ff7fe640 1 -- 172.21.15.59:0/4011482045 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0034660 con 0x7f2a000ce920 2024-03-06T07:25:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.442+0000 7f2a06ad4640 1 -- 172.21.15.59:0/4011482045 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a000ce920 msgr2=0x7f2a00071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.442+0000 7f2a06ad4640 1 --2- 172.21.15.59:0/4011482045 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a000ce920 0x7f2a00071980 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f2a000d0980 tx=0x7f29f002bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.442+0000 7f2a06ad4640 1 -- 172.21.15.59:0/4011482045 shutdown_connections 2024-03-06T07:25:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.442+0000 7f2a06ad4640 1 --2- 172.21.15.59:0/4011482045 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a000ce920 0x7f2a00071980 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.442+0000 7f2a06ad4640 1 --2- 172.21.15.59:0/4011482045 >> v2:172.21.15.164:3300/0 conn(0x7f2a0006b220 0x7f2a0015e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.442+0000 7f2a06ad4640 1 -- 172.21.15.59:0/4011482045 >> 172.21.15.59:0/4011482045 conn(0x7f2a0006c0c0 msgr2=0x7f2a0006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.442+0000 7f2a06ad4640 1 -- 172.21.15.59:0/4011482045 shutdown_connections 2024-03-06T07:25:22.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.442+0000 7f2a06ad4640 1 -- 172.21.15.59:0/4011482045 wait complete. 2024-03-06T07:25:22.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.443+0000 7f2a06ad4640 1 Processor -- start 2024-03-06T07:25:22.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.443+0000 7f2a06ad4640 1 -- start start 2024-03-06T07:25:22.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a06ad4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a0006b220 0x7f2a0014c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a06ad4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a00071ec0 0x7f2a00160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a06ad4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a000ce920 0x7f2a000cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a06ad4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000ce380 con 0x7f2a0006b220 2024-03-06T07:25:22.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a06ad4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000ce200 con 0x7f2a00071ec0 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a06ad4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a00162710 con 0x7f2a000ce920 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a04870640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a0006b220 0x7f2a0014c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a04870640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a0006b220 0x7f2a0014c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53568/0 (socket says 172.21.15.59:53568) 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a04870640 1 -- 172.21.15.59:0/2821360648 learned_addr learned my addr 172.21.15.59:0/2821360648 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f29fffff640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a00071ec0 0x7f2a00160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a05071640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a000ce920 0x7f2a000cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a04870640 1 -- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a000ce920 msgr2=0x7f2a000cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a04870640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a000ce920 0x7f2a000cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a04870640 1 -- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a00071ec0 msgr2=0x7f2a00160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a04870640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a00071ec0 0x7f2a00160840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a04870640 1 -- 172.21.15.59:0/2821360648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a001493f0 con 0x7f2a0006b220 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.444+0000 7f2a05071640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a000ce920 0x7f2a000cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.445+0000 7f2a04870640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a0006b220 0x7f2a0014c120 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f29f800e9e0 tx=0x7f29f800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.445+0000 7f29fd7fa640 1 -- 172.21.15.59:0/2821360648 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8017070 con 0x7f2a0006b220 2024-03-06T07:25:22.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.445+0000 7f29fd7fa640 1 -- 172.21.15.59:0/2821360648 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f800c9d0 con 0x7f2a0006b220 2024-03-06T07:25:22.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.445+0000 7f2a06ad4640 1 -- 172.21.15.59:0/2821360648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0014c660 con 0x7f2a0006b220 2024-03-06T07:25:22.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.445+0000 7f29fd7fa640 1 -- 172.21.15.59:0/2821360648 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f800cdc0 con 0x7f2a0006b220 2024-03-06T07:25:22.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.445+0000 7f2a06ad4640 1 -- 172.21.15.59:0/2821360648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a000c5e30 con 0x7f2a0006b220 2024-03-06T07:25:22.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.446+0000 7f2a06ad4640 1 -- 172.21.15.59:0/2821360648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0004f330 con 0x7f2a0006b220 2024-03-06T07:25:22.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.446+0000 7f29fd7fa640 1 -- 172.21.15.59:0/2821360648 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29f801a070 con 0x7f2a0006b220 2024-03-06T07:25:22.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.447+0000 7f29fd7fa640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f29e006b4d0 0x7f29e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.447+0000 7f29fd7fa640 1 -- 172.21.15.59:0/2821360648 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29f808a1f0 con 0x7f2a0006b220 2024-03-06T07:25:22.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.447+0000 7f29fffff640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f29e006b4d0 0x7f29e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.447+0000 7f29fffff640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f29e006b4d0 0x7f29e006d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f29f0036000 tx=0x7f29f0030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.451+0000 7f29fd7fa640 1 -- 172.21.15.59:0/2821360648 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29f8058990 con 0x7f2a0006b220 2024-03-06T07:25:22.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.671+0000 7f2a06ad4640 1 -- 172.21.15.59:0/2821360648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a000cdf00 con 0x7f2a0006b220 2024-03-06T07:25:22.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.671+0000 7f29fd7fa640 1 -- 172.21.15.59:0/2821360648 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f29f805ca30 con 0x7f2a0006b220 2024-03-06T07:25:22.673 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:22.673 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:04.517350+0000","modified":"2024-03-06T07:25:06.518231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/250576205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":250576205},{"type":"v1","addr":"172.21.15.164:6837","nonce":250576205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:22.674 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 35 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.673+0000 7f2a06ad4640 1 -- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f29e006b4d0 msgr2=0x7f29e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.673+0000 7f2a06ad4640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f29e006b4d0 0x7f29e006d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f29f0036000 tx=0x7f29f0030000 comp rx=0 tx=0).stop 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.673+0000 7f2a06ad4640 1 -- 172.21.15.59:0/2821360648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a0006b220 msgr2=0x7f2a0014c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.673+0000 7f2a06ad4640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a0006b220 0x7f2a0014c120 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f29f800e9e0 tx=0x7f29f800eea0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.674+0000 7f2a06ad4640 1 -- 172.21.15.59:0/2821360648 shutdown_connections 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.674+0000 7f2a06ad4640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f29e006b4d0 0x7f29e006d990 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.674+0000 7f2a06ad4640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a000ce920 0x7f2a000cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.674+0000 7f2a06ad4640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a00071ec0 0x7f2a00160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.674+0000 7f2a06ad4640 1 --2- 172.21.15.59:0/2821360648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2a0006b220 0x7f2a0014c120 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.674+0000 7f2a06ad4640 1 -- 172.21.15.59:0/2821360648 >> 172.21.15.59:0/2821360648 conn(0x7f2a0006c0c0 msgr2=0x7f2a0006e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.674+0000 7f2a06ad4640 1 -- 172.21.15.59:0/2821360648 shutdown_connections 2024-03-06T07:25:22.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.674+0000 7f2a06ad4640 1 -- 172.21.15.59:0/2821360648 wait complete. 2024-03-06T07:25:22.686 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:25:22.804 INFO:teuthology.orchestra.run.smithi164.stdout:{"mds_cache":{"strays_created":10}} 2024-03-06T07:25:22.804 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-03-06T07:25:22.804 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:22.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.904+0000 7f1d82066640 1 Processor -- start 2024-03-06T07:25:22.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.904+0000 7f1d82066640 1 -- start start 2024-03-06T07:25:22.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d82066640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1d7c0b4f70 0x7f1d7c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d82066640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1d7c066d90 0x7f1d7c0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d82066640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c0b5890 con 0x7f1d7c06c6d0 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d82066640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c15e9f0 con 0x7f1d7c066d90 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d82066640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c0cda10 con 0x7f1d7c0b4f70 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d7bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1d7c066d90 0x7f1d7c0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d7affd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1d7c06c6d0 0x7f1d7c15ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41954/0 (socket says 172.21.15.59:41954) 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d7affd640 1 -- 172.21.15.59:0/2614172443 learned_addr learned my addr 172.21.15.59:0/2614172443 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d7b7fe640 1 --2- 172.21.15.59:0/2614172443 >> v2:172.21.15.164:3300/0 conn(0x7f1d7c0b4f70 0x7f1d7c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d7bfff640 1 -- 172.21.15.59:0/2614172443 >> v2:172.21.15.164:3300/0 conn(0x7f1d7c0b4f70 msgr2=0x7f1d7c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d7bfff640 1 --2- 172.21.15.59:0/2614172443 >> v2:172.21.15.164:3300/0 conn(0x7f1d7c0b4f70 0x7f1d7c0b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d7bfff640 1 -- 172.21.15.59:0/2614172443 >> v1:172.21.15.59:6789/0 conn(0x7f1d7c06c6d0 legacy=0x7f1d7c15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d7a7fc640 1 -- 172.21.15.59:0/2614172443 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626800740 0 0) 0x7f1d7c0b5890 con 0x7f1d7c06c6d0 2024-03-06T07:25:22.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.905+0000 7f1d7bfff640 1 -- 172.21.15.59:0/2614172443 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c0cd450 con 0x7f1d7c066d90 2024-03-06T07:25:22.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.906+0000 7f1d7bfff640 1 --2- 172.21.15.59:0/2614172443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7c066d90 0x7f1d7c0ccb50 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c06cac0 tx=0x7f1d7402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35375f6cb7d71665 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.906+0000 7f1d7a7fc640 1 -- 172.21.15.59:0/2614172443 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74039070 con 0x7f1d7c066d90 2024-03-06T07:25:22.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.906+0000 7f1d7a7fc640 1 -- 172.21.15.59:0/2614172443 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7402cc80 con 0x7f1d7c066d90 2024-03-06T07:25:22.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.906+0000 7f1d7a7fc640 1 -- 172.21.15.59:0/2614172443 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74034660 con 0x7f1d7c066d90 2024-03-06T07:25:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.906+0000 7f1d82066640 1 -- 172.21.15.59:0/2614172443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7c066d90 msgr2=0x7f1d7c0ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.906+0000 7f1d82066640 1 --2- 172.21.15.59:0/2614172443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7c066d90 0x7f1d7c0ccb50 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c06cac0 tx=0x7f1d7402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.906+0000 7f1d82066640 1 -- 172.21.15.59:0/2614172443 shutdown_connections 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.906+0000 7f1d82066640 1 --2- 172.21.15.59:0/2614172443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7c066d90 0x7f1d7c0ccb50 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.906+0000 7f1d82066640 1 --2- 172.21.15.59:0/2614172443 >> v2:172.21.15.164:3300/0 conn(0x7f1d7c0b4f70 0x7f1d7c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.906+0000 7f1d82066640 1 -- 172.21.15.59:0/2614172443 >> 172.21.15.59:0/2614172443 conn(0x7f1d7c0724d0 msgr2=0x7f1d7c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.907+0000 7f1d82066640 1 -- 172.21.15.59:0/2614172443 shutdown_connections 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.907+0000 7f1d82066640 1 -- 172.21.15.59:0/2614172443 wait complete. 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.908+0000 7f1d82066640 1 Processor -- start 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.908+0000 7f1d82066640 1 -- start start 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d82066640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d7c066d90 0x7f1d7c0cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d82066640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d7c0b4f70 0x7f1d7c0d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d82066640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7c14d540 0x7f1d7c0ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d82066640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0cd090 con 0x7f1d7c14d540 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d82066640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c15e680 con 0x7f1d7c066d90 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d82066640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0cd210 con 0x7f1d7c0b4f70 2024-03-06T07:25:22.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d7bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7c14d540 0x7f1d7c0ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d7bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7c14d540 0x7f1d7c0ae6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53576/0 (socket says 172.21.15.59:53576) 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d7bfff640 1 -- 172.21.15.59:0/3339726823 learned_addr learned my addr 172.21.15.59:0/3339726823 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d7b7fe640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d7c066d90 0x7f1d7c0cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d7affd640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d7c0b4f70 0x7f1d7c0d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d7bfff640 1 -- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d7c0b4f70 msgr2=0x7f1d7c0d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d7bfff640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d7c0b4f70 0x7f1d7c0d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d7bfff640 1 -- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d7c066d90 msgr2=0x7f1d7c0cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d7bfff640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d7c066d90 0x7f1d7c0cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d7bfff640 1 -- 172.21.15.59:0/3339726823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c14e090 con 0x7f1d7c14d540 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d7bfff640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7c14d540 0x7f1d7c0ae6e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1d74037040 tx=0x7f1d74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d63fff640 1 -- 172.21.15.59:0/3339726823 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74039050 con 0x7f1d7c14d540 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d63fff640 1 -- 172.21.15.59:0/3339726823 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d74045070 con 0x7f1d7c14d540 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d82066640 1 -- 172.21.15.59:0/3339726823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c0cf070 con 0x7f1d7c14d540 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d63fff640 1 -- 172.21.15.59:0/3339726823 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74009450 con 0x7f1d7c14d540 2024-03-06T07:25:22.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.909+0000 7f1d82066640 1 -- 172.21.15.59:0/3339726823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c0caa10 con 0x7f1d7c14d540 2024-03-06T07:25:22.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.910+0000 7f1d82066640 1 -- 172.21.15.59:0/3339726823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c04f330 con 0x7f1d7c14d540 2024-03-06T07:25:22.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.911+0000 7f1d63fff640 1 -- 172.21.15.59:0/3339726823 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d7403e850 con 0x7f1d7c14d540 2024-03-06T07:25:22.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.911+0000 7f1d63fff640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d6806b4d0 0x7f1d6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.911+0000 7f1d63fff640 1 -- 172.21.15.59:0/3339726823 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d740ad1d0 con 0x7f1d7c14d540 2024-03-06T07:25:22.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.911+0000 7f1d7b7fe640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d6806b4d0 0x7f1d6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.912+0000 7f1d7b7fe640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d6806b4d0 0x7f1d6806d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009b40 tx=0x7f1d70009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:22.915+0000 7f1d63fff640 1 -- 172.21.15.59:0/3339726823 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d7407bb20 con 0x7f1d7c14d540 2024-03-06T07:25:23.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.135+0000 7f1d82066640 1 -- 172.21.15.59:0/3339726823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d7c06a190 con 0x7f1d7c14d540 2024-03-06T07:25:23.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.136+0000 7f1d63fff640 1 -- 172.21.15.59:0/3339726823 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f1d7407fbc0 con 0x7f1d7c14d540 2024-03-06T07:25:23.138 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:23.138 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:04.517350+0000","modified":"2024-03-06T07:25:06.518231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/250576205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":250576205},{"type":"v1","addr":"172.21.15.164:6837","nonce":250576205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:23.138 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 35 2024-03-06T07:25:23.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 -- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d6806b4d0 msgr2=0x7f1d6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d6806b4d0 0x7f1d6806d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009b40 tx=0x7f1d70009190 comp rx=0 tx=0).stop 2024-03-06T07:25:23.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 -- 172.21.15.59:0/3339726823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7c14d540 msgr2=0x7f1d7c0ae6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7c14d540 0x7f1d7c0ae6e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1d74037040 tx=0x7f1d74004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:23.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 -- 172.21.15.59:0/3339726823 shutdown_connections 2024-03-06T07:25:23.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d6806b4d0 0x7f1d6806d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d7c14d540 0x7f1d7c0ae6e0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d7c0b4f70 0x7f1d7c0d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 --2- 172.21.15.59:0/3339726823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d7c066d90 0x7f1d7c0cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 -- 172.21.15.59:0/3339726823 >> 172.21.15.59:0/3339726823 conn(0x7f1d7c0724d0 msgr2=0x7f1d7c0728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 -- 172.21.15.59:0/3339726823 shutdown_connections 2024-03-06T07:25:23.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:23.138+0000 7f1d82066640 1 -- 172.21.15.59:0/3339726823 wait complete. 2024-03-06T07:25:23.151 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:25:23.266 INFO:teuthology.orchestra.run.smithi164.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:25:23.266 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:25:23.266 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:25:28.267 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:28.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.379+0000 7fe727f7a640 1 Processor -- start 2024-03-06T07:25:28.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.380+0000 7fe727f7a640 1 -- start start 2024-03-06T07:25:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.380+0000 7fe727f7a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe72015dfe0 0x7fe7200cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.380+0000 7fe727f7a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe7200cf830 0x7fe7200d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.380+0000 7fe727f7a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7200b3430 con 0x7fe7200d27f0 2024-03-06T07:25:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.380+0000 7fe727f7a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7200b31e0 con 0x7fe7200cf830 2024-03-06T07:25:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.380+0000 7fe727f7a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7200b3d60 con 0x7fe72015dfe0 2024-03-06T07:25:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.381+0000 7fe725515640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe7200cf830 0x7fe7200d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.381+0000 7fe725515640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe7200cf830 0x7fe7200d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58664/0 (socket says 172.21.15.59:58664) 2024-03-06T07:25:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.381+0000 7fe725515640 1 -- 172.21.15.59:0/876109053 learned_addr learned my addr 172.21.15.59:0/876109053 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.381+0000 7fe725d16640 1 --2- 172.21.15.59:0/876109053 >> v2:172.21.15.164:3300/0 conn(0x7fe72015dfe0 0x7fe7200cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.381+0000 7fe725515640 1 -- 172.21.15.59:0/876109053 >> v2:172.21.15.164:3300/0 conn(0x7fe72015dfe0 msgr2=0x7fe7200cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.381+0000 7fe725515640 1 --2- 172.21.15.59:0/876109053 >> v2:172.21.15.164:3300/0 conn(0x7fe72015dfe0 0x7fe7200cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.381+0000 7fe725515640 1 -- 172.21.15.59:0/876109053 >> v1:172.21.15.164:6789/0 conn(0x7fe7200d27f0 legacy=0x7fe7200d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.381+0000 7fe725515640 1 -- 172.21.15.59:0/876109053 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7200b3710 con 0x7fe7200cf830 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.382+0000 7fe725515640 1 --2- 172.21.15.59:0/876109053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7200cf830 0x7fe7200d22b0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe710009870 tx=0x7fe71002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfa1bb8127dd85f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.382+0000 7fe724d14640 1 -- 172.21.15.59:0/876109053 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710034070 con 0x7fe7200cf830 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.382+0000 7fe724d14640 1 -- 172.21.15.59:0/876109053 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe710030860 con 0x7fe7200cf830 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.382+0000 7fe724d14640 1 -- 172.21.15.59:0/876109053 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710030c30 con 0x7fe7200cf830 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.382+0000 7fe727f7a640 1 -- 172.21.15.59:0/876109053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7200cf830 msgr2=0x7fe7200d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.382+0000 7fe727f7a640 1 --2- 172.21.15.59:0/876109053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7200cf830 0x7fe7200d22b0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe710009870 tx=0x7fe71002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.382+0000 7fe727f7a640 1 -- 172.21.15.59:0/876109053 shutdown_connections 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.382+0000 7fe727f7a640 1 --2- 172.21.15.59:0/876109053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7200cf830 0x7fe7200d22b0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.382+0000 7fe727f7a640 1 --2- 172.21.15.59:0/876109053 >> v2:172.21.15.164:3300/0 conn(0x7fe72015dfe0 0x7fe7200cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.382+0000 7fe727f7a640 1 -- 172.21.15.59:0/876109053 >> 172.21.15.59:0/876109053 conn(0x7fe7200d3cd0 msgr2=0x7fe7200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.383+0000 7fe727f7a640 1 -- 172.21.15.59:0/876109053 shutdown_connections 2024-03-06T07:25:28.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.383+0000 7fe727f7a640 1 -- 172.21.15.59:0/876109053 wait complete. 2024-03-06T07:25:28.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.384+0000 7fe727f7a640 1 Processor -- start 2024-03-06T07:25:28.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.384+0000 7fe727f7a640 1 -- start start 2024-03-06T07:25:28.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.384+0000 7fe727f7a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7200cf830 0x7fe72015c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.384+0000 7fe727f7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7200d27f0 0x7fe72015f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.384+0000 7fe727f7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe72015dfe0 0x7fe7201f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.384+0000 7fe727f7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200d1df0 con 0x7fe7200d27f0 2024-03-06T07:25:28.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.384+0000 7fe727f7a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7200d1c70 con 0x7fe7200cf830 2024-03-06T07:25:28.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.384+0000 7fe727f7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72015e3c0 con 0x7fe72015dfe0 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe725515640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7200d27f0 0x7fe72015f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe725515640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7200d27f0 0x7fe72015f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58674/0 (socket says 172.21.15.59:58674) 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe725515640 1 -- 172.21.15.59:0/3777079705 learned_addr learned my addr 172.21.15.59:0/3777079705 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe725d16640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7200cf830 0x7fe72015c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe726517640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe72015dfe0 0x7fe7201f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe725515640 1 -- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe72015dfe0 msgr2=0x7fe7201f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe725515640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe72015dfe0 0x7fe7201f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe725515640 1 -- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7200cf830 msgr2=0x7fe72015c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe725515640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7200cf830 0x7fe72015c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe725515640 1 -- 172.21.15.59:0/3777079705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72015c950 con 0x7fe7200d27f0 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe725d16640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7200cf830 0x7fe72015c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:28.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe725515640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7200d27f0 0x7fe72015f0d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fe710000c00 tx=0x7fe7100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe7167fc640 1 -- 172.21.15.59:0/3777079705 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710034040 con 0x7fe7200d27f0 2024-03-06T07:25:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe7167fc640 1 -- 172.21.15.59:0/3777079705 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe710039d90 con 0x7fe7200d27f0 2024-03-06T07:25:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe727f7a640 1 -- 172.21.15.59:0/3777079705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72015fa00 con 0x7fe7200d27f0 2024-03-06T07:25:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe7167fc640 1 -- 172.21.15.59:0/3777079705 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710043630 con 0x7fe7200d27f0 2024-03-06T07:25:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.385+0000 7fe727f7a640 1 -- 172.21.15.59:0/3777079705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72015ffc0 con 0x7fe7200d27f0 2024-03-06T07:25:28.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.386+0000 7fe727f7a640 1 -- 172.21.15.59:0/3777079705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72004f330 con 0x7fe7200d27f0 2024-03-06T07:25:28.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.387+0000 7fe7167fc640 1 -- 172.21.15.59:0/3777079705 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe71004a060 con 0x7fe7200d27f0 2024-03-06T07:25:28.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.387+0000 7fe7167fc640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe70406b4d0 0x7fe70406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.388+0000 7fe7167fc640 1 -- 172.21.15.59:0/3777079705 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7100ac050 con 0x7fe7200d27f0 2024-03-06T07:25:28.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.388+0000 7fe725d16640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe70406b4d0 0x7fe70406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.388+0000 7fe725d16640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe70406b4d0 0x7fe70406d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009f30 tx=0x7fe71c009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.392+0000 7fe7167fc640 1 -- 172.21.15.59:0/3777079705 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe71003e070 con 0x7fe7200d27f0 2024-03-06T07:25:28.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.615+0000 7fe727f7a640 1 -- 172.21.15.59:0/3777079705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe72006c560 con 0x7fe7200d27f0 2024-03-06T07:25:28.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.615+0000 7fe7167fc640 1 -- 172.21.15.59:0/3777079705 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fe71007a910 con 0x7fe7200d27f0 2024-03-06T07:25:28.617 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:28.617 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:04.517350+0000","modified":"2024-03-06T07:25:06.518231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/250576205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":250576205},{"type":"v1","addr":"172.21.15.164:6837","nonce":250576205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:28.618 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 35 2024-03-06T07:25:28.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 -- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe70406b4d0 msgr2=0x7fe70406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe70406b4d0 0x7fe70406d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe71c009f30 tx=0x7fe71c009190 comp rx=0 tx=0).stop 2024-03-06T07:25:28.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 -- 172.21.15.59:0/3777079705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7200d27f0 msgr2=0x7fe72015f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7200d27f0 0x7fe72015f0d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fe710000c00 tx=0x7fe7100046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 -- 172.21.15.59:0/3777079705 shutdown_connections 2024-03-06T07:25:28.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe70406b4d0 0x7fe70406d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe72015dfe0 0x7fe7201f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7200d27f0 0x7fe72015f0d0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 --2- 172.21.15.59:0/3777079705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7200cf830 0x7fe72015c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 -- 172.21.15.59:0/3777079705 >> 172.21.15.59:0/3777079705 conn(0x7fe7200d3cd0 msgr2=0x7fe72006cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 -- 172.21.15.59:0/3777079705 shutdown_connections 2024-03-06T07:25:28.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.617+0000 7fe727f7a640 1 -- 172.21.15.59:0/3777079705 wait complete. 2024-03-06T07:25:28.630 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:25:28.749 INFO:teuthology.orchestra.run.smithi164.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-03-06T07:25:28.749 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-03-06T07:25:28.749 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:28.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.849+0000 7f136631e640 1 Processor -- start 2024-03-06T07:25:28.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.849+0000 7f136631e640 1 -- start start 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f136631e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1360162020 0x7f13600d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f136631e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1360162400 0x7f13600cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f136631e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13600b1070 con 0x7f1360162a00 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f136631e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13600ce9d0 con 0x7f1360162020 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f136631e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13600d2da0 con 0x7f1360162400 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f135f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1360162a00 0x7f13600b0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38592/0 (socket says 172.21.15.59:38592) 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f135f7fe640 1 -- 172.21.15.59:0/2877562070 learned_addr learned my addr 172.21.15.59:0/2877562070 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f135ffff640 1 --2- 172.21.15.59:0/2877562070 >> v2:172.21.15.164:3300/0 conn(0x7f1360162020 0x7f13600d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f13648bb640 1 --2- 172.21.15.59:0/2877562070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1360162400 0x7f13600cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f135effd640 1 -- 172.21.15.59:0/2877562070 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533278622 0 0) 0x7f13600b1070 con 0x7f1360162a00 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f135effd640 1 -- 172.21.15.59:0/2877562070 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1348003520 con 0x7f1360162a00 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f135ffff640 1 -- 172.21.15.59:0/2877562070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1360162400 msgr2=0x7f13600cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f135ffff640 1 --2- 172.21.15.59:0/2877562070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1360162400 0x7f13600cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f135ffff640 1 -- 172.21.15.59:0/2877562070 >> v1:172.21.15.59:6789/0 conn(0x7f1360162a00 legacy=0x7f13600b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f135ffff640 1 -- 172.21.15.59:0/2877562070 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13600d2500 con 0x7f1360162020 2024-03-06T07:25:28.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.850+0000 7f13648bb640 1 --2- 172.21.15.59:0/2877562070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1360162400 0x7f13600cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:28.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.851+0000 7f135ffff640 1 --2- 172.21.15.59:0/2877562070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1360162020 0x7f13600d4830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f13600d2f20 tx=0x7f135402b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c51896a80e36df39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.851+0000 7f135effd640 1 -- 172.21.15.59:0/2877562070 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354035020 con 0x7f1360162020 2024-03-06T07:25:28.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.851+0000 7f135effd640 1 -- 172.21.15.59:0/2877562070 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1354031890 con 0x7f1360162020 2024-03-06T07:25:28.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.851+0000 7f135effd640 1 -- 172.21.15.59:0/2877562070 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354031c40 con 0x7f1360162020 2024-03-06T07:25:28.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.852+0000 7f136631e640 1 -- 172.21.15.59:0/2877562070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1360162020 msgr2=0x7f13600d4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.852+0000 7f136631e640 1 --2- 172.21.15.59:0/2877562070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1360162020 0x7f13600d4830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f13600d2f20 tx=0x7f135402b930 comp rx=0 tx=0).stop 2024-03-06T07:25:28.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.852+0000 7f136631e640 1 -- 172.21.15.59:0/2877562070 shutdown_connections 2024-03-06T07:25:28.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.852+0000 7f136631e640 1 --2- 172.21.15.59:0/2877562070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1360162400 0x7f13600cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.852+0000 7f136631e640 1 --2- 172.21.15.59:0/2877562070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1360162020 0x7f13600d4830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.852+0000 7f136631e640 1 -- 172.21.15.59:0/2877562070 >> 172.21.15.59:0/2877562070 conn(0x7f13600d3d10 msgr2=0x7f13600d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.852+0000 7f136631e640 1 -- 172.21.15.59:0/2877562070 shutdown_connections 2024-03-06T07:25:28.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.852+0000 7f136631e640 1 -- 172.21.15.59:0/2877562070 wait complete. 2024-03-06T07:25:28.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.853+0000 7f136631e640 1 Processor -- start 2024-03-06T07:25:28.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.853+0000 7f136631e640 1 -- start start 2024-03-06T07:25:28.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f136631e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1360162020 0x7f13601611e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f136631e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1360162400 0x7f136006c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f136631e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1360162a00 0x7f13600b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f136631e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13600cec00 con 0x7f1360162a00 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f136631e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13600d2ac0 con 0x7f1360162400 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f136631e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13600cf940 con 0x7f1360162020 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f13648bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1360162a00 0x7f13600b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f13648bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1360162a00 0x7f13600b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58678/0 (socket says 172.21.15.59:58678) 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f13648bb640 1 -- 172.21.15.59:0/2085835601 learned_addr learned my addr 172.21.15.59:0/2085835601 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f135f7fe640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1360162400 0x7f136006c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f135ffff640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1360162020 0x7f13601611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f13648bb640 1 -- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1360162020 msgr2=0x7f13601611e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f13648bb640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1360162020 0x7f13601611e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f13648bb640 1 -- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1360162400 msgr2=0x7f136006c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f13648bb640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1360162400 0x7f136006c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.854+0000 7f13648bb640 1 -- 172.21.15.59:0/2085835601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136014cf60 con 0x7f1360162a00 2024-03-06T07:25:28.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.855+0000 7f13648bb640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1360162a00 0x7f13600b1290 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f135800ef90 tx=0x7f1358009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.855+0000 7f135cff9640 1 -- 172.21.15.59:0/2085835601 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358018020 con 0x7f1360162a00 2024-03-06T07:25:28.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.855+0000 7f135cff9640 1 -- 172.21.15.59:0/2085835601 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1358014480 con 0x7f1360162a00 2024-03-06T07:25:28.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.855+0000 7f135cff9640 1 -- 172.21.15.59:0/2085835601 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358014870 con 0x7f1360162a00 2024-03-06T07:25:28.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.855+0000 7f136631e640 1 -- 172.21.15.59:0/2085835601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136014d240 con 0x7f1360162a00 2024-03-06T07:25:28.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.855+0000 7f136631e640 1 -- 172.21.15.59:0/2085835601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136014d800 con 0x7f1360162a00 2024-03-06T07:25:28.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.856+0000 7f136631e640 1 -- 172.21.15.59:0/2085835601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136004f330 con 0x7f1360162a00 2024-03-06T07:25:28.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.856+0000 7f135cff9640 1 -- 172.21.15.59:0/2085835601 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1358005d10 con 0x7f1360162a00 2024-03-06T07:25:28.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.857+0000 7f135cff9640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134c06b4d0 0x7f134c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.857+0000 7f135cff9640 1 -- 172.21.15.59:0/2085835601 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f135808af10 con 0x7f1360162a00 2024-03-06T07:25:28.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.857+0000 7f135ffff640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134c06b4d0 0x7f134c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.857+0000 7f135ffff640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134c06b4d0 0x7f134c06d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1354036000 tx=0x7f1354008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:28.861+0000 7f135cff9640 1 -- 172.21.15.59:0/2085835601 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1358022030 con 0x7f1360162a00 2024-03-06T07:25:29.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.078+0000 7f136631e640 1 -- 172.21.15.59:0/2085835601 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13600b9b60 con 0x7f1360162a00 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.079+0000 7f135cff9640 1 -- 172.21.15.59:0/2085835601 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f1358010040 con 0x7f1360162a00 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:04.517350+0000","modified":"2024-03-06T07:25:06.518231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/250576205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":250576205},{"type":"v1","addr":"172.21.15.164:6837","nonce":250576205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:29.081 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 35 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 -- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134c06b4d0 msgr2=0x7f134c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134c06b4d0 0x7f134c06d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1354036000 tx=0x7f1354008030 comp rx=0 tx=0).stop 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 -- 172.21.15.59:0/2085835601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1360162a00 msgr2=0x7f13600b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1360162a00 0x7f13600b1290 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f135800ef90 tx=0x7f1358009f90 comp rx=0 tx=0).stop 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 -- 172.21.15.59:0/2085835601 shutdown_connections 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f134c06b4d0 0x7f134c06d990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1360162a00 0x7f13600b1290 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1360162400 0x7f136006c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 --2- 172.21.15.59:0/2085835601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1360162020 0x7f13601611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 -- 172.21.15.59:0/2085835601 >> 172.21.15.59:0/2085835601 conn(0x7f13600d3d10 msgr2=0x7f13600d01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 -- 172.21.15.59:0/2085835601 shutdown_connections 2024-03-06T07:25:29.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.081+0000 7f136631e640 1 -- 172.21.15.59:0/2085835601 wait complete. 2024-03-06T07:25:29.094 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:25:29.213 INFO:teuthology.orchestra.run.smithi164.stdout:{"purge_queue":{"pq_executed":10}} 2024-03-06T07:25:29.213 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-03-06T07:25:29.213 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:25:29.213 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> stdin-killer -- python3 -c ' 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> import uuid 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> n = 10 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> create_hard_links = False 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> if 0 > 0: 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> create_hard_links = True 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> dpath = os.path.dirname(path) 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:29.229 DEBUG:teuthology.orchestra.run.smithi059:> try: 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> for i in range(n): 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> f.write(f"{i}") 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> f.flush() 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(f.fileno()) 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> if True: 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> os.unlink(fpath) 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> if True: 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(dirfd) 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> if create_hard_links: 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> for j in range(0): 2024-03-06T07:25:29.230 DEBUG:teuthology.orchestra.run.smithi059:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:29.231 DEBUG:teuthology.orchestra.run.smithi059:> if False: 2024-03-06T07:25:29.231 DEBUG:teuthology.orchestra.run.smithi059:> os.fsync(dirfd) 2024-03-06T07:25:29.231 DEBUG:teuthology.orchestra.run.smithi059:> finally: 2024-03-06T07:25:29.231 DEBUG:teuthology.orchestra.run.smithi059:> os.close(dirfd) 2024-03-06T07:25:29.231 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:25:29.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:29.404 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:29.404 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:29.491 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:29.532 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:29.532 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:25:29.532 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:29.547 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:29.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.687+0000 7fdbc773e640 1 Processor -- start 2024-03-06T07:25:29.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.687+0000 7fdbc773e640 1 -- start start 2024-03-06T07:25:29.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.688+0000 7fdbc773e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc01621c0 0x7fdbc00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.688+0000 7fdbc773e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdbc01625a0 0x7fdbc015dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.688+0000 7fdbc773e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc00ceac0 con 0x7fdbc00d4920 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.688+0000 7fdbc773e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc00cf690 con 0x7fdbc01625a0 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.688+0000 7fdbc773e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00d2dc0 con 0x7fdbc01621c0 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.688+0000 7fdbc5cdb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdbc01625a0 0x7fdbc015dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.688+0000 7fdbc5cdb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdbc01625a0 0x7fdbc015dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58690/0 (socket says 172.21.15.59:58690) 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.688+0000 7fdbc5cdb640 1 -- 172.21.15.59:0/3639800828 learned_addr learned my addr 172.21.15.59:0/3639800828 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.688+0000 7fdbc54da640 1 --2- 172.21.15.59:0/3639800828 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc01621c0 0x7fdbc00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.689+0000 7fdbc5cdb640 1 -- 172.21.15.59:0/3639800828 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc01621c0 msgr2=0x7fdbc00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.689+0000 7fdbc5cdb640 1 --2- 172.21.15.59:0/3639800828 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc01621c0 0x7fdbc00cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.689+0000 7fdbc5cdb640 1 -- 172.21.15.59:0/3639800828 >> v1:172.21.15.164:6789/0 conn(0x7fdbc00d4920 legacy=0x7fdbc00d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.689+0000 7fdbc5cdb640 1 -- 172.21.15.59:0/3639800828 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00ced60 con 0x7fdbc01625a0 2024-03-06T07:25:29.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.689+0000 7fdbc5cdb640 1 --2- 172.21.15.59:0/3639800828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc01625a0 0x7fdbc015dd40 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009e50 tx=0x7fdbb002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da242e7a12414d83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.689+0000 7fdbbffff640 1 -- 172.21.15.59:0/3639800828 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb002cb00 con 0x7fdbc01625a0 2024-03-06T07:25:29.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.689+0000 7fdbbffff640 1 -- 172.21.15.59:0/3639800828 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb002cca0 con 0x7fdbc01625a0 2024-03-06T07:25:29.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.689+0000 7fdbbffff640 1 -- 172.21.15.59:0/3639800828 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0034610 con 0x7fdbc01625a0 2024-03-06T07:25:29.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.690+0000 7fdbc773e640 1 -- 172.21.15.59:0/3639800828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc01625a0 msgr2=0x7fdbc015dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.690+0000 7fdbc773e640 1 --2- 172.21.15.59:0/3639800828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc01625a0 0x7fdbc015dd40 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009e50 tx=0x7fdbb002bf60 comp rx=0 tx=0).stop 2024-03-06T07:25:29.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.690+0000 7fdbc773e640 1 -- 172.21.15.59:0/3639800828 shutdown_connections 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.690+0000 7fdbc773e640 1 --2- 172.21.15.59:0/3639800828 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc01625a0 0x7fdbc015dd40 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.690+0000 7fdbc773e640 1 --2- 172.21.15.59:0/3639800828 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc01621c0 0x7fdbc00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.690+0000 7fdbc773e640 1 -- 172.21.15.59:0/3639800828 >> 172.21.15.59:0/3639800828 conn(0x7fdbc00d3d10 msgr2=0x7fdbc00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.690+0000 7fdbc773e640 1 -- 172.21.15.59:0/3639800828 shutdown_connections 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.690+0000 7fdbc773e640 1 -- 172.21.15.59:0/3639800828 wait complete. 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.691+0000 7fdbc773e640 1 Processor -- start 2024-03-06T07:25:29.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.691+0000 7fdbc773e640 1 -- start start 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc773e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc00d4920 0x7fdbc00b9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc773e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc01621c0 0x7fdbc01f7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc773e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc01625a0 0x7fdbc01f7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc773e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc015e280 con 0x7fdbc01621c0 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc773e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00cf320 con 0x7fdbc01625a0 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc773e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc015e400 con 0x7fdbc00d4920 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc4cd9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc01621c0 0x7fdbc01f7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc4cd9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc01621c0 0x7fdbc01f7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58702/0 (socket says 172.21.15.59:58702) 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc4cd9640 1 -- 172.21.15.59:0/166325758 learned_addr learned my addr 172.21.15.59:0/166325758 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc54da640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc00d4920 0x7fdbc00b9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc5cdb640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc01625a0 0x7fdbc01f7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc4cd9640 1 -- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc00d4920 msgr2=0x7fdbc00b9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc4cd9640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc00d4920 0x7fdbc00b9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc4cd9640 1 -- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc01625a0 msgr2=0x7fdbc01f7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc4cd9640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc01625a0 0x7fdbc01f7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.692+0000 7fdbc4cd9640 1 -- 172.21.15.59:0/166325758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00a1ab0 con 0x7fdbc01621c0 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.693+0000 7fdbc4cd9640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc01621c0 0x7fdbc01f7300 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400c430 tx=0x7fdbb400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.693+0000 7fdbbdffb640 1 -- 172.21.15.59:0/166325758 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb40052d0 con 0x7fdbc01621c0 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.693+0000 7fdbbdffb640 1 -- 172.21.15.59:0/166325758 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb4005490 con 0x7fdbc01621c0 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.693+0000 7fdbc773e640 1 -- 172.21.15.59:0/166325758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc00b6bc0 con 0x7fdbc01621c0 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.693+0000 7fdbbdffb640 1 -- 172.21.15.59:0/166325758 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4007260 con 0x7fdbc01621c0 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.693+0000 7fdbc773e640 1 -- 172.21.15.59:0/166325758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc00b7460 con 0x7fdbc01621c0 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.694+0000 7fdbc773e640 1 -- 172.21.15.59:0/166325758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc004f330 con 0x7fdbc01621c0 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.694+0000 7fdbbdffb640 1 -- 172.21.15.59:0/166325758 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbb4007400 con 0x7fdbc01621c0 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.695+0000 7fdbbdffb640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbac06b4d0 0x7fdbac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.695+0000 7fdbbdffb640 1 -- 172.21.15.59:0/166325758 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbb40895b0 con 0x7fdbc01621c0 2024-03-06T07:25:29.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.695+0000 7fdbc54da640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbac06b4d0 0x7fdbac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.695+0000 7fdbc54da640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbac06b4d0 0x7fdbac06d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009b40 tx=0x7fdbb8009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.699+0000 7fdbbdffb640 1 -- 172.21.15.59:0/166325758 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbb4057e70 con 0x7fdbc01621c0 2024-03-06T07:25:29.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.882+0000 7fdbc773e640 1 -- 172.21.15.59:0/166325758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbc0073380 con 0x7fdbc01621c0 2024-03-06T07:25:29.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.883+0000 7fdbbdffb640 1 -- 172.21.15.59:0/166325758 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16656 (secure 0 0 0) 0x7fdbb405bf10 con 0x7fdbc01621c0 2024-03-06T07:25:29.885 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:29.885 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":70,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:06.152587+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:02.779485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:03.482047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.885+0000 7fdbc773e640 1 -- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbac06b4d0 msgr2=0x7fdbac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.885+0000 7fdbc773e640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbac06b4d0 0x7fdbac06d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009b40 tx=0x7fdbb8009190 comp rx=0 tx=0).stop 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.885+0000 7fdbc773e640 1 -- 172.21.15.59:0/166325758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc01621c0 msgr2=0x7fdbc01f7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.885+0000 7fdbc773e640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc01621c0 0x7fdbc01f7300 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400c430 tx=0x7fdbb400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.886+0000 7fdbc773e640 1 -- 172.21.15.59:0/166325758 shutdown_connections 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.886+0000 7fdbc773e640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbac06b4d0 0x7fdbac06d990 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.886+0000 7fdbc773e640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc01625a0 0x7fdbc01f7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.886+0000 7fdbc773e640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbc01621c0 0x7fdbc01f7300 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.886+0000 7fdbc773e640 1 --2- 172.21.15.59:0/166325758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc00d4920 0x7fdbc00b9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.886+0000 7fdbc773e640 1 -- 172.21.15.59:0/166325758 >> 172.21.15.59:0/166325758 conn(0x7fdbc00d3d10 msgr2=0x7fdbc015e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.886+0000 7fdbc773e640 1 -- 172.21.15.59:0/166325758 shutdown_connections 2024-03-06T07:25:29.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:29.886+0000 7fdbc773e640 1 -- 172.21.15.59:0/166325758 wait complete. 2024-03-06T07:25:29.898 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:02.779485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:03.482047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:29.898 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.001+0000 7f0d8922a640 1 Processor -- start 2024-03-06T07:25:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.001+0000 7f0d8922a640 1 -- start start 2024-03-06T07:25:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.002+0000 7f0d8922a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0d8415a220 0x7f0d840cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.002+0000 7f0d8922a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d840b3320 con 0x7f0d8406da00 2024-03-06T07:25:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.002+0000 7f0d8922a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8415ec80 con 0x7f0d840712d0 2024-03-06T07:25:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.002+0000 7f0d8922a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d840cf610 con 0x7f0d8415a220 2024-03-06T07:25:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.002+0000 7f0d82d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0d8415a220 0x7f0d840cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.002+0000 7f0d8359e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0d8406da00 0x7f0d8415e570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38602/0 (socket says 172.21.15.59:38602) 2024-03-06T07:25:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.002+0000 7f0d8359e640 1 -- 172.21.15.59:0/2569051759 learned_addr learned my addr 172.21.15.59:0/2569051759 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.002+0000 7f0d82d9d640 1 -- 172.21.15.59:0/2569051759 >> v1:172.21.15.164:6789/0 conn(0x7f0d840712d0 legacy=0x7f0d8415de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.002+0000 7f0d82d9d640 1 -- 172.21.15.59:0/2569051759 >> v1:172.21.15.59:6789/0 conn(0x7f0d8406da00 legacy=0x7f0d8415e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.002+0000 7f0d82d9d640 1 -- 172.21.15.59:0/2569051759 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d840728d0 con 0x7f0d8415a220 2024-03-06T07:25:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.003+0000 7f0d82d9d640 1 --2- 172.21.15.59:0/2569051759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d8415a220 0x7f0d840cf0d0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0097a0 tx=0x7f0d7c02c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974dae3a7c9eb327 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.003+0000 7f0d81d9b640 1 -- 172.21.15.59:0/2569051759 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c030800 con 0x7f0d8415a220 2024-03-06T07:25:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.003+0000 7f0d81d9b640 1 -- 172.21.15.59:0/2569051759 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7c0309a0 con 0x7f0d8415a220 2024-03-06T07:25:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.003+0000 7f0d81d9b640 1 -- 172.21.15.59:0/2569051759 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c030d70 con 0x7f0d8415a220 2024-03-06T07:25:30.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.004+0000 7f0d8922a640 1 -- 172.21.15.59:0/2569051759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d8415a220 msgr2=0x7f0d840cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.004+0000 7f0d8922a640 1 --2- 172.21.15.59:0/2569051759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d8415a220 0x7f0d840cf0d0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0097a0 tx=0x7f0d7c02c9e0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.004+0000 7f0d8922a640 1 -- 172.21.15.59:0/2569051759 shutdown_connections 2024-03-06T07:25:30.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.004+0000 7f0d8922a640 1 --2- 172.21.15.59:0/2569051759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d8415a220 0x7f0d840cf0d0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.004+0000 7f0d8922a640 1 -- 172.21.15.59:0/2569051759 >> 172.21.15.59:0/2569051759 conn(0x7f0d8406b190 msgr2=0x7f0d8406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.004+0000 7f0d8922a640 1 -- 172.21.15.59:0/2569051759 shutdown_connections 2024-03-06T07:25:30.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.004+0000 7f0d8922a640 1 -- 172.21.15.59:0/2569051759 wait complete. 2024-03-06T07:25:30.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.005+0000 7f0d8922a640 1 Processor -- start 2024-03-06T07:25:30.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.005+0000 7f0d8922a640 1 -- start start 2024-03-06T07:25:30.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8922a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d8406da00 0x7f0d841552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8922a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d840712d0 0x7f0d8415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8922a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d8415a220 0x7f0d84141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8922a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d84072030 con 0x7f0d8415a220 2024-03-06T07:25:30.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8922a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d84072bb0 con 0x7f0d8406da00 2024-03-06T07:25:30.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8922a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d84072530 con 0x7f0d840712d0 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d8415a220 0x7f0d84141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d8415a220 0x7f0d84141c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58726/0 (socket says 172.21.15.59:58726) 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8359e640 1 -- 172.21.15.59:0/2045612652 learned_addr learned my addr 172.21.15.59:0/2045612652 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d82d9d640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d8406da00 0x7f0d841552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8259c640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d840712d0 0x7f0d8415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8359e640 1 -- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d840712d0 msgr2=0x7f0d8415abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8359e640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d840712d0 0x7f0d8415abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8359e640 1 -- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d8406da00 msgr2=0x7f0d841552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8359e640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d8406da00 0x7f0d841552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d8359e640 1 -- 172.21.15.59:0/2045612652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8415b130 con 0x7f0d8415a220 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.006+0000 7f0d82d9d640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d8406da00 0x7f0d841552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.007+0000 7f0d8359e640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d8415a220 0x7f0d84141c50 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0d7400cbc0 tx=0x7f0d7400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.007+0000 7f0d7b7fe640 1 -- 172.21.15.59:0/2045612652 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d74017070 con 0x7f0d8415a220 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.007+0000 7f0d7b7fe640 1 -- 172.21.15.59:0/2045612652 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7400ad40 con 0x7f0d8415a220 2024-03-06T07:25:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.007+0000 7f0d8922a640 1 -- 172.21.15.59:0/2045612652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8415b3b0 con 0x7f0d8415a220 2024-03-06T07:25:30.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.007+0000 7f0d7b7fe640 1 -- 172.21.15.59:0/2045612652 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d74012600 con 0x7f0d8415a220 2024-03-06T07:25:30.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.007+0000 7f0d8922a640 1 -- 172.21.15.59:0/2045612652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8415b970 con 0x7f0d8415a220 2024-03-06T07:25:30.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.008+0000 7f0d8922a640 1 -- 172.21.15.59:0/2045612652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8415c140 con 0x7f0d8415a220 2024-03-06T07:25:30.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.008+0000 7f0d7b7fe640 1 -- 172.21.15.59:0/2045612652 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d7401a070 con 0x7f0d8415a220 2024-03-06T07:25:30.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.009+0000 7f0d7b7fe640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d6406b4d0 0x7f0d6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.009+0000 7f0d7b7fe640 1 -- 172.21.15.59:0/2045612652 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d74089e70 con 0x7f0d8415a220 2024-03-06T07:25:30.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.009+0000 7f0d82d9d640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d6406b4d0 0x7f0d6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.009+0000 7f0d82d9d640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d6406b4d0 0x7f0d6406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c037000 tx=0x7f0d7c03a040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.013+0000 7f0d7b7fe640 1 -- 172.21.15.59:0/2045612652 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d8415b130 con 0x7f0d8415a220 2024-03-06T07:25:30.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.193+0000 7f0d8922a640 1 -- 172.21.15.59:0/2045612652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d840732b0 con 0x7f0d8415a220 2024-03-06T07:25:30.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.194+0000 7f0d7b7fe640 1 -- 172.21.15.59:0/2045612652 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0d740586a0 con 0x7f0d8415a220 2024-03-06T07:25:30.196 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:25:30.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 -- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d6406b4d0 msgr2=0x7f0d6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d6406b4d0 0x7f0d6406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c037000 tx=0x7f0d7c03a040 comp rx=0 tx=0).stop 2024-03-06T07:25:30.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 -- 172.21.15.59:0/2045612652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d8415a220 msgr2=0x7f0d84141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d8415a220 0x7f0d84141c50 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0d7400cbc0 tx=0x7f0d7400a430 comp rx=0 tx=0).stop 2024-03-06T07:25:30.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 -- 172.21.15.59:0/2045612652 shutdown_connections 2024-03-06T07:25:30.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0d6406b4d0 0x7f0d6406d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0d8415a220 0x7f0d84141c50 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d840712d0 0x7f0d8415abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 --2- 172.21.15.59:0/2045612652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d8406da00 0x7f0d841552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 -- 172.21.15.59:0/2045612652 >> 172.21.15.59:0/2045612652 conn(0x7f0d8406b190 msgr2=0x7f0d84159330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 -- 172.21.15.59:0/2045612652 shutdown_connections 2024-03-06T07:25:30.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.196+0000 7f0d8922a640 1 -- 172.21.15.59:0/2045612652 wait complete. 2024-03-06T07:25:30.208 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.310+0000 7fee70076640 1 Processor -- start 2024-03-06T07:25:30.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.311+0000 7fee70076640 1 -- start start 2024-03-06T07:25:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.311+0000 7fee70076640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee681614d0 0x7fee680d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.311+0000 7fee70076640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fee680d1910 0x7fee680d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.311+0000 7fee70076640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee680d2740 con 0x7fee680d22b0 2024-03-06T07:25:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.311+0000 7fee70076640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee681618b0 con 0x7fee680d1910 2024-03-06T07:25:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.311+0000 7fee70076640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee680cf400 con 0x7fee681614d0 2024-03-06T07:25:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.311+0000 7fee6d611640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fee680d1910 0x7fee680d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.311+0000 7fee6d611640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fee680d1910 0x7fee680d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58742/0 (socket says 172.21.15.59:58742) 2024-03-06T07:25:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.311+0000 7fee6e613640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fee680d22b0 0x7fee680d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38612/0 (socket says 172.21.15.59:38612) 2024-03-06T07:25:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.311+0000 7fee6d611640 1 -- 172.21.15.59:0/1359846343 learned_addr learned my addr 172.21.15.59:0/1359846343 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.312+0000 7fee6de12640 1 --2- 172.21.15.59:0/1359846343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee681614d0 0x7fee680d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.312+0000 7fee6d611640 1 -- 172.21.15.59:0/1359846343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee681614d0 msgr2=0x7fee680d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.312+0000 7fee6d611640 1 --2- 172.21.15.59:0/1359846343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee681614d0 0x7fee680d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.312+0000 7fee6d611640 1 -- 172.21.15.59:0/1359846343 >> v1:172.21.15.59:6789/0 conn(0x7fee680d22b0 legacy=0x7fee680d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.312+0000 7fee6ce10640 1 -- 172.21.15.59:0/1359846343 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296814694 0 0) 0x7fee60001490 con 0x7fee680d22b0 2024-03-06T07:25:30.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.312+0000 7fee6d611640 1 -- 172.21.15.59:0/1359846343 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee680cf6a0 con 0x7fee680d1910 2024-03-06T07:25:30.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.312+0000 7fee6d611640 1 --2- 172.21.15.59:0/1359846343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee680d1910 0x7fee680d1d70 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fee58009870 tx=0x7fee5802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12665f4cb45c0623 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.313+0000 7fee6ce10640 1 -- 172.21.15.59:0/1359846343 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58034070 con 0x7fee680d1910 2024-03-06T07:25:30.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.313+0000 7fee6ce10640 1 -- 172.21.15.59:0/1359846343 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee58030840 con 0x7fee680d1910 2024-03-06T07:25:30.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.313+0000 7fee6ce10640 1 -- 172.21.15.59:0/1359846343 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58030c10 con 0x7fee680d1910 2024-03-06T07:25:30.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.313+0000 7fee70076640 1 -- 172.21.15.59:0/1359846343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee680d1910 msgr2=0x7fee680d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.313+0000 7fee70076640 1 --2- 172.21.15.59:0/1359846343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee680d1910 0x7fee680d1d70 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fee58009870 tx=0x7fee5802b980 comp rx=0 tx=0).stop 2024-03-06T07:25:30.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.313+0000 7fee70076640 1 -- 172.21.15.59:0/1359846343 shutdown_connections 2024-03-06T07:25:30.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.313+0000 7fee70076640 1 --2- 172.21.15.59:0/1359846343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee680d1910 0x7fee680d1d70 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.313+0000 7fee70076640 1 --2- 172.21.15.59:0/1359846343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee681614d0 0x7fee680d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.313+0000 7fee70076640 1 -- 172.21.15.59:0/1359846343 >> 172.21.15.59:0/1359846343 conn(0x7fee680d3d40 msgr2=0x7fee680d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.313+0000 7fee70076640 1 -- 172.21.15.59:0/1359846343 shutdown_connections 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.313+0000 7fee70076640 1 -- 172.21.15.59:0/1359846343 wait complete. 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.314+0000 7fee70076640 1 Processor -- start 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.314+0000 7fee70076640 1 -- start start 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.315+0000 7fee70076640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee680d1910 0x7fee680b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.315+0000 7fee70076640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee680d22b0 0x7fee680cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.315+0000 7fee70076640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee681614d0 0x7fee680cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.315+0000 7fee70076640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680b0910 con 0x7fee681614d0 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.315+0000 7fee70076640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680cfc60 con 0x7fee680d22b0 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.315+0000 7fee70076640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee680b1e40 con 0x7fee680d1910 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.315+0000 7fee6e613640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee681614d0 0x7fee680cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.315+0000 7fee6e613640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee681614d0 0x7fee680cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58748/0 (socket says 172.21.15.59:58748) 2024-03-06T07:25:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.315+0000 7fee6e613640 1 -- 172.21.15.59:0/677280864 learned_addr learned my addr 172.21.15.59:0/677280864 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.315+0000 7fee6d611640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee680d22b0 0x7fee680cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.315+0000 7fee6de12640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee680d1910 0x7fee680b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.316+0000 7fee6e613640 1 -- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee680d1910 msgr2=0x7fee680b7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.316+0000 7fee6e613640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee680d1910 0x7fee680b7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.316+0000 7fee6e613640 1 -- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee680d22b0 msgr2=0x7fee680cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.316+0000 7fee6e613640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee680d22b0 0x7fee680cc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.316+0000 7fee6e613640 1 -- 172.21.15.59:0/677280864 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6815ca80 con 0x7fee681614d0 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.316+0000 7fee6e613640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee681614d0 0x7fee680cae10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fee6000c7f0 tx=0x7fee6000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.316+0000 7fee5e7fc640 1 -- 172.21.15.59:0/677280864 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60018070 con 0x7fee681614d0 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.316+0000 7fee5e7fc640 1 -- 172.21.15.59:0/677280864 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6000a5e0 con 0x7fee681614d0 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.316+0000 7fee5e7fc640 1 -- 172.21.15.59:0/677280864 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6000aa10 con 0x7fee681614d0 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.316+0000 7fee70076640 1 -- 172.21.15.59:0/677280864 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee680cb510 con 0x7fee681614d0 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.316+0000 7fee70076640 1 -- 172.21.15.59:0/677280864 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee680cbad0 con 0x7fee681614d0 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.317+0000 7fee70076640 1 -- 172.21.15.59:0/677280864 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee680ce170 con 0x7fee681614d0 2024-03-06T07:25:30.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.317+0000 7fee5e7fc640 1 -- 172.21.15.59:0/677280864 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee6001b070 con 0x7fee681614d0 2024-03-06T07:25:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.318+0000 7fee5e7fc640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee4c06b4d0 0x7fee4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.318+0000 7fee5e7fc640 1 -- 172.21.15.59:0/677280864 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee60089720 con 0x7fee681614d0 2024-03-06T07:25:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.318+0000 7fee6de12640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee4c06b4d0 0x7fee4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.318+0000 7fee6de12640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee4c06b4d0 0x7fee4c06d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fee6815c6e0 tx=0x7fee64009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.322+0000 7fee5e7fc640 1 -- 172.21.15.59:0/677280864 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee680d2740 con 0x7fee681614d0 2024-03-06T07:25:30.502 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.501+0000 7fee70076640 1 -- 172.21.15.59:0/677280864 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fee68068ff0 con 0x7fee681614d0 2024-03-06T07:25:30.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.501+0000 7fee5e7fc640 1 -- 172.21.15.59:0/677280864 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7fee60057f50 con 0x7fee681614d0 2024-03-06T07:25:30.503 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:25:30.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 -- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee4c06b4d0 msgr2=0x7fee4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee4c06b4d0 0x7fee4c06d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fee6815c6e0 tx=0x7fee64009190 comp rx=0 tx=0).stop 2024-03-06T07:25:30.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 -- 172.21.15.59:0/677280864 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee681614d0 msgr2=0x7fee680cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee681614d0 0x7fee680cae10 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fee6000c7f0 tx=0x7fee6000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 -- 172.21.15.59:0/677280864 shutdown_connections 2024-03-06T07:25:30.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee4c06b4d0 0x7fee4c06d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee681614d0 0x7fee680cae10 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee680d22b0 0x7fee680cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 --2- 172.21.15.59:0/677280864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee680d1910 0x7fee680b7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 -- 172.21.15.59:0/677280864 >> 172.21.15.59:0/677280864 conn(0x7fee680d3d40 msgr2=0x7fee680cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 -- 172.21.15.59:0/677280864 shutdown_connections 2024-03-06T07:25:30.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.503+0000 7fee70076640 1 -- 172.21.15.59:0/677280864 wait complete. 2024-03-06T07:25:30.515 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:30.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.618+0000 7f4d73015640 1 Processor -- start 2024-03-06T07:25:30.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.618+0000 7f4d73015640 1 -- start start 2024-03-06T07:25:30.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.618+0000 7f4d73015640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4d6c1614d0 0x7f4d6c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.618+0000 7f4d73015640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4d6c0d1910 0x7f4d6c0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.618+0000 7f4d73015640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6c1618b0 con 0x7f4d6c0d22e0 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.618+0000 7f4d73015640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d6c0cf400 con 0x7f4d6c1614d0 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.618+0000 7f4d73015640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d6c0d2710 con 0x7f4d6c0d1910 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.619+0000 7f4d70db1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4d6c1614d0 0x7f4d6c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.619+0000 7f4d6bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4d6c0d1910 0x7f4d6c0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.619+0000 7f4d70db1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4d6c1614d0 0x7f4d6c0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58756/0 (socket says 172.21.15.59:58756) 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.619+0000 7f4d6bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4d6c0d1910 0x7f4d6c0d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:35130/0 (socket says 172.21.15.59:35130) 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.619+0000 7f4d70db1640 1 -- 172.21.15.59:0/2422523036 learned_addr learned my addr 172.21.15.59:0/2422523036 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.619+0000 7f4d6bfff640 1 -- 172.21.15.59:0/2422523036 >> v2:172.21.15.59:3300/0 conn(0x7f4d6c1614d0 msgr2=0x7f4d6c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.619+0000 7f4d6bfff640 1 --2- 172.21.15.59:0/2422523036 >> v2:172.21.15.59:3300/0 conn(0x7f4d6c1614d0 0x7f4d6c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.619+0000 7f4d6bfff640 1 -- 172.21.15.59:0/2422523036 >> v1:172.21.15.164:6789/0 conn(0x7f4d6c0d22e0 legacy=0x7f4d6c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.619+0000 7f4d6bfff640 1 -- 172.21.15.59:0/2422523036 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c0cf6a0 con 0x7f4d6c0d1910 2024-03-06T07:25:30.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d6bfff640 1 --2- 172.21.15.59:0/2422523036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6c0d1910 0x7f4d6c0d1da0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0ce550 tx=0x7f4d5802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a215ce6e030d54b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d6b7fe640 1 -- 172.21.15.59:0/2422523036 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58035020 con 0x7f4d6c0d1910 2024-03-06T07:25:30.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d6b7fe640 1 -- 172.21.15.59:0/2422523036 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d58031830 con 0x7f4d6c0d1910 2024-03-06T07:25:30.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d6b7fe640 1 -- 172.21.15.59:0/2422523036 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58031c00 con 0x7f4d6c0d1910 2024-03-06T07:25:30.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d73015640 1 -- 172.21.15.59:0/2422523036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6c0d1910 msgr2=0x7f4d6c0d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d73015640 1 --2- 172.21.15.59:0/2422523036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6c0d1910 0x7f4d6c0d1da0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0ce550 tx=0x7f4d5802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d73015640 1 -- 172.21.15.59:0/2422523036 shutdown_connections 2024-03-06T07:25:30.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d73015640 1 --2- 172.21.15.59:0/2422523036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6c0d1910 0x7f4d6c0d1da0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d73015640 1 --2- 172.21.15.59:0/2422523036 >> v2:172.21.15.59:3300/0 conn(0x7f4d6c1614d0 0x7f4d6c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d73015640 1 -- 172.21.15.59:0/2422523036 >> 172.21.15.59:0/2422523036 conn(0x7f4d6c0d3d40 msgr2=0x7f4d6c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d73015640 1 -- 172.21.15.59:0/2422523036 shutdown_connections 2024-03-06T07:25:30.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.620+0000 7f4d73015640 1 -- 172.21.15.59:0/2422523036 wait complete. 2024-03-06T07:25:30.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.621+0000 7f4d73015640 1 Processor -- start 2024-03-06T07:25:30.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.622+0000 7f4d73015640 1 -- start start 2024-03-06T07:25:30.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.622+0000 7f4d73015640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6c0d1910 0x7f4d6c0b7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.622+0000 7f4d73015640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d6c0d22e0 0x7f4d6c15dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.622+0000 7f4d73015640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6c1614d0 0x7f4d6c154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.622+0000 7f4d73015640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c0b0910 con 0x7f4d6c0d1910 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.622+0000 7f4d73015640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c0cfc60 con 0x7f4d6c1614d0 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.622+0000 7f4d73015640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c0b1e40 con 0x7f4d6c0d22e0 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d70db1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6c0d1910 0x7f4d6c0b7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d70db1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6c0d1910 0x7f4d6c0b7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58768/0 (socket says 172.21.15.59:58768) 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d70db1640 1 -- 172.21.15.59:0/3978069195 learned_addr learned my addr 172.21.15.59:0/3978069195 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d6bfff640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d6c0d22e0 0x7f4d6c15dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d715b2640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6c1614d0 0x7f4d6c154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d70db1640 1 -- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d6c0d22e0 msgr2=0x7f4d6c15dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d70db1640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d6c0d22e0 0x7f4d6c15dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d70db1640 1 -- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6c1614d0 msgr2=0x7f4d6c154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d70db1640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6c1614d0 0x7f4d6c154de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d70db1640 1 -- 172.21.15.59:0/3978069195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c0b7b90 con 0x7f4d6c0d1910 2024-03-06T07:25:30.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d70db1640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6c0d1910 0x7f4d6c0b7340 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4d6000e840 tx=0x7f4d6000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d697fa640 1 -- 172.21.15.59:0/3978069195 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d60017070 con 0x7f4d6c0d1910 2024-03-06T07:25:30.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d697fa640 1 -- 172.21.15.59:0/3978069195 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d6000c8e0 con 0x7f4d6c0d1910 2024-03-06T07:25:30.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d697fa640 1 -- 172.21.15.59:0/3978069195 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6000ccd0 con 0x7f4d6c0d1910 2024-03-06T07:25:30.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d73015640 1 -- 172.21.15.59:0/3978069195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6c0b7e70 con 0x7f4d6c0d1910 2024-03-06T07:25:30.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.623+0000 7f4d73015640 1 -- 172.21.15.59:0/3978069195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6c0b8430 con 0x7f4d6c0d1910 2024-03-06T07:25:30.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.624+0000 7f4d73015640 1 -- 172.21.15.59:0/3978069195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6c068ff0 con 0x7f4d6c0d1910 2024-03-06T07:25:30.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.625+0000 7f4d697fa640 1 -- 172.21.15.59:0/3978069195 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d6001a070 con 0x7f4d6c0d1910 2024-03-06T07:25:30.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.625+0000 7f4d697fa640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b500 0x7f4d5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.625+0000 7f4d697fa640 1 -- 172.21.15.59:0/3978069195 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d6008a2c0 con 0x7f4d6c0d1910 2024-03-06T07:25:30.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.625+0000 7f4d6bfff640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b500 0x7f4d5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.626+0000 7f4d6bfff640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b500 0x7f4d5006d9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4d58036000 tx=0x7f4d58008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.630+0000 7f4d697fa640 1 -- 172.21.15.59:0/3978069195 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d60058b80 con 0x7f4d6c0d1910 2024-03-06T07:25:30.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.809+0000 7f4d73015640 1 -- 172.21.15.59:0/3978069195 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4d6c0cea60 con 0x7f4d6c0d1910 2024-03-06T07:25:30.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.810+0000 7f4d697fa640 1 -- 172.21.15.59:0/3978069195 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4d6005cc20 con 0x7f4d6c0d1910 2024-03-06T07:25:30.812 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:25:30.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 -- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b500 msgr2=0x7f4d5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b500 0x7f4d5006d9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4d58036000 tx=0x7f4d58008030 comp rx=0 tx=0).stop 2024-03-06T07:25:30.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 -- 172.21.15.59:0/3978069195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6c0d1910 msgr2=0x7f4d6c0b7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6c0d1910 0x7f4d6c0b7340 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4d6000e840 tx=0x7f4d6000ed00 comp rx=0 tx=0).stop 2024-03-06T07:25:30.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 -- 172.21.15.59:0/3978069195 shutdown_connections 2024-03-06T07:25:30.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b500 0x7f4d5006d9c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6c1614d0 0x7f4d6c154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d6c0d22e0 0x7f4d6c15dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 --2- 172.21.15.59:0/3978069195 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6c0d1910 0x7f4d6c0b7340 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 -- 172.21.15.59:0/3978069195 >> 172.21.15.59:0/3978069195 conn(0x7f4d6c0d3d40 msgr2=0x7f4d6c06cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 -- 172.21.15.59:0/3978069195 shutdown_connections 2024-03-06T07:25:30.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.812+0000 7f4d73015640 1 -- 172.21.15.59:0/3978069195 wait complete. 2024-03-06T07:25:30.824 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:25:30.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.925+0000 7fd8ee7bd640 1 Processor -- start 2024-03-06T07:25:30.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.925+0000 7fd8ee7bd640 1 -- start start 2024-03-06T07:25:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8ee7bd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd8e81586c0 0x7fd8e815efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8ee7bd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd8e80b0240 0x7fd8e806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8ee7bd640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e815f520 con 0x7fd8e806b4f0 2024-03-06T07:25:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8ee7bd640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e8158aa0 con 0x7fd8e80b0240 2024-03-06T07:25:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8ee7bd640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e806b960 con 0x7fd8e81586c0 2024-03-06T07:25:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8e77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd8e80b0240 0x7fd8e806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8ecd5a640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd8e806b4f0 0x7fd8e80cf050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38622/0 (socket says 172.21.15.59:38622) 2024-03-06T07:25:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8ecd5a640 1 -- 172.21.15.59:0/171769633 learned_addr learned my addr 172.21.15.59:0/171769633 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8e7fff640 1 --2- 172.21.15.59:0/171769633 >> v2:172.21.15.164:3300/0 conn(0x7fd8e81586c0 0x7fd8e815efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8e77fe640 1 -- 172.21.15.59:0/171769633 >> v2:172.21.15.164:3300/0 conn(0x7fd8e81586c0 msgr2=0x7fd8e815efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8e77fe640 1 --2- 172.21.15.59:0/171769633 >> v2:172.21.15.164:3300/0 conn(0x7fd8e81586c0 0x7fd8e815efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8e77fe640 1 -- 172.21.15.59:0/171769633 >> v1:172.21.15.59:6789/0 conn(0x7fd8e806b4f0 legacy=0x7fd8e80cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8e77fe640 1 -- 172.21.15.59:0/171769633 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e80ce510 con 0x7fd8e80b0240 2024-03-06T07:25:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.926+0000 7fd8e6ffd640 1 -- 172.21.15.59:0/171769633 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482485662 0 0) 0x7fd8e0001490 con 0x7fd8e806b4f0 2024-03-06T07:25:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.927+0000 7fd8e77fe640 1 --2- 172.21.15.59:0/171769633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e80b0240 0x7fd8e806afb0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8073730 tx=0x7fd8d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f9d9455e6972f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.927+0000 7fd8e6ffd640 1 -- 172.21.15.59:0/171769633 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8035020 con 0x7fd8e80b0240 2024-03-06T07:25:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.927+0000 7fd8e6ffd640 1 -- 172.21.15.59:0/171769633 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d8031830 con 0x7fd8e80b0240 2024-03-06T07:25:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.927+0000 7fd8e6ffd640 1 -- 172.21.15.59:0/171769633 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8031c00 con 0x7fd8e80b0240 2024-03-06T07:25:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.928+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/171769633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e80b0240 msgr2=0x7fd8e806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.928+0000 7fd8ee7bd640 1 --2- 172.21.15.59:0/171769633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e80b0240 0x7fd8e806afb0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8073730 tx=0x7fd8d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.928+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/171769633 shutdown_connections 2024-03-06T07:25:30.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.928+0000 7fd8ee7bd640 1 --2- 172.21.15.59:0/171769633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e80b0240 0x7fd8e806afb0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.928+0000 7fd8ee7bd640 1 --2- 172.21.15.59:0/171769633 >> v2:172.21.15.164:3300/0 conn(0x7fd8e81586c0 0x7fd8e815efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.928+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/171769633 >> 172.21.15.59:0/171769633 conn(0x7fd8e80d09d0 msgr2=0x7fd8e80d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.928+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/171769633 shutdown_connections 2024-03-06T07:25:30.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.928+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/171769633 wait complete. 2024-03-06T07:25:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.929+0000 7fd8ee7bd640 1 Processor -- start 2024-03-06T07:25:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.929+0000 7fd8ee7bd640 1 -- start start 2024-03-06T07:25:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ee7bd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8e806b4f0 0x7fd8e80b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ee7bd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8e80b0240 0x7fd8e815b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ee7bd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e81586c0 0x7fd8e815cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ee7bd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e81622b0 con 0x7fd8e81586c0 2024-03-06T07:25:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ee7bd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e8161fd0 con 0x7fd8e806b4f0 2024-03-06T07:25:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ee7bd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e8162590 con 0x7fd8e80b0240 2024-03-06T07:25:30.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ecd5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e81586c0 0x7fd8e815cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ecd5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e81586c0 0x7fd8e815cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58794/0 (socket says 172.21.15.59:58794) 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ecd5a640 1 -- 172.21.15.59:0/184339380 learned_addr learned my addr 172.21.15.59:0/184339380 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8e7fff640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8e806b4f0 0x7fd8e80b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8e77fe640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8e80b0240 0x7fd8e815b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ecd5a640 1 -- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8e80b0240 msgr2=0x7fd8e815b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ecd5a640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8e80b0240 0x7fd8e815b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ecd5a640 1 -- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8e806b4f0 msgr2=0x7fd8e80b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ecd5a640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8e806b4f0 0x7fd8e80b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ecd5a640 1 -- 172.21.15.59:0/184339380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e815d0d0 con 0x7fd8e81586c0 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.930+0000 7fd8ecd5a640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e81586c0 0x7fd8e815cb90 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000c7d0 tx=0x7fd8e000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.931+0000 7fd8e4ff9640 1 -- 172.21.15.59:0/184339380 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0018070 con 0x7fd8e81586c0 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.931+0000 7fd8e4ff9640 1 -- 172.21.15.59:0/184339380 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e000a640 con 0x7fd8e81586c0 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.931+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/184339380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e815d370 con 0x7fd8e81586c0 2024-03-06T07:25:30.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.931+0000 7fd8e4ff9640 1 -- 172.21.15.59:0/184339380 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000aab0 con 0x7fd8e81586c0 2024-03-06T07:25:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.931+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/184339380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e815d930 con 0x7fd8e81586c0 2024-03-06T07:25:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.932+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/184339380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e804f330 con 0x7fd8e81586c0 2024-03-06T07:25:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.932+0000 7fd8e4ff9640 1 -- 172.21.15.59:0/184339380 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8e001b070 con 0x7fd8e81586c0 2024-03-06T07:25:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.933+0000 7fd8e4ff9640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8d406b4d0 0x7fd8d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.933+0000 7fd8e4ff9640 1 -- 172.21.15.59:0/184339380 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8e00896d0 con 0x7fd8e81586c0 2024-03-06T07:25:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.933+0000 7fd8e7fff640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8d406b4d0 0x7fd8d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.933+0000 7fd8e7fff640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8d406b4d0 0x7fd8d406d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009aa0 tx=0x7fd8dc009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:30.937+0000 7fd8e4ff9640 1 -- 172.21.15.59:0/184339380 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8e815f520 con 0x7fd8e81586c0 2024-03-06T07:25:31.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.131+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/184339380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd8e8162860 con 0x7fd8e81586c0 2024-03-06T07:25:31.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.131+0000 7fd8e4ff9640 1 -- 172.21.15.59:0/184339380 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v70) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd8e0057f90 con 0x7fd8e81586c0 2024-03-06T07:25:31.133 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:25:31.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8d406b4d0 msgr2=0x7fd8d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8d406b4d0 0x7fd8d406d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009aa0 tx=0x7fd8dc009190 comp rx=0 tx=0).stop 2024-03-06T07:25:31.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/184339380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e81586c0 msgr2=0x7fd8e815cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e81586c0 0x7fd8e815cb90 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000c7d0 tx=0x7fd8e000cc90 comp rx=0 tx=0).stop 2024-03-06T07:25:31.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/184339380 shutdown_connections 2024-03-06T07:25:31.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8d406b4d0 0x7fd8d406d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e81586c0 0x7fd8e815cb90 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8e80b0240 0x7fd8e815b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 --2- 172.21.15.59:0/184339380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8e806b4f0 0x7fd8e80b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/184339380 >> 172.21.15.59:0/184339380 conn(0x7fd8e80d09d0 msgr2=0x7fd8e80cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/184339380 shutdown_connections 2024-03-06T07:25:31.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.133+0000 7fd8ee7bd640 1 -- 172.21.15.59:0/184339380 wait complete. 2024-03-06T07:25:31.146 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:25:31.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:31.146 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:31.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[62632]: fuse finished with error 0 and tester_r 0 2024-03-06T07:25:31.173 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:25:31.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:31.173 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:25:31.338 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:25:31.338 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:31.353 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:31.353 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:25:31.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:31.354 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:25:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:31.409 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:25:31.409 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:25:31.409 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:31.424 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:31.425 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:25:31.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:31.425 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:25:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:31.479 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:25:31.479 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:25:31.479 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:31.494 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:31.494 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:25:31.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:31.494 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:25:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:25:31.548 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:31.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.648+0000 7f07e97f7640 1 Processor -- start 2024-03-06T07:25:31.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.648+0000 7f07e97f7640 1 -- start start 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e97f7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f07e415dfe0 0x7f07e40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e97f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07e40cf830 0x7f07e40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e97f7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e40b3430 con 0x7f07e40d27f0 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e97f7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e40d2c50 con 0x7f07e415dfe0 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e97f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e40b31e0 con 0x7f07e40cf830 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e2ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f07e415dfe0 0x7f07e40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e37fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f07e40d27f0 0x7f07e40d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38628/0 (socket says 172.21.15.59:38628) 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e37fe640 1 -- 172.21.15.59:0/2277688805 learned_addr learned my addr 172.21.15.59:0/2277688805 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e27fc640 1 --2- 172.21.15.59:0/2277688805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07e40cf830 0x7f07e40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e2ffd640 1 -- 172.21.15.59:0/2277688805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07e40cf830 msgr2=0x7f07e40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e2ffd640 1 --2- 172.21.15.59:0/2277688805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07e40cf830 0x7f07e40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e2ffd640 1 -- 172.21.15.59:0/2277688805 >> v1:172.21.15.59:6789/0 conn(0x7f07e40d27f0 legacy=0x7f07e40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e2ffd640 1 -- 172.21.15.59:0/2277688805 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e40b3710 con 0x7f07e415dfe0 2024-03-06T07:25:31.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.649+0000 7f07e1ffb640 1 -- 172.21.15.59:0/2277688805 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772855586 0 0) 0x7f07e40b3430 con 0x7f07e40d27f0 2024-03-06T07:25:31.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.650+0000 7f07e2ffd640 1 --2- 172.21.15.59:0/2277688805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07e415dfe0 0x7f07e40cf2f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0097a0 tx=0x7f07dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922ba4a395dfa33d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.650+0000 7f07e1ffb640 1 -- 172.21.15.59:0/2277688805 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc035020 con 0x7f07e415dfe0 2024-03-06T07:25:31.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.650+0000 7f07e1ffb640 1 -- 172.21.15.59:0/2277688805 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07dc031830 con 0x7f07e415dfe0 2024-03-06T07:25:31.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.650+0000 7f07e1ffb640 1 -- 172.21.15.59:0/2277688805 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc031c00 con 0x7f07e415dfe0 2024-03-06T07:25:31.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.651+0000 7f07e97f7640 1 -- 172.21.15.59:0/2277688805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07e415dfe0 msgr2=0x7f07e40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.651+0000 7f07e97f7640 1 --2- 172.21.15.59:0/2277688805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07e415dfe0 0x7f07e40cf2f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0097a0 tx=0x7f07dc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.651+0000 7f07e97f7640 1 -- 172.21.15.59:0/2277688805 shutdown_connections 2024-03-06T07:25:31.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.651+0000 7f07e97f7640 1 --2- 172.21.15.59:0/2277688805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07e40cf830 0x7f07e40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.651+0000 7f07e97f7640 1 --2- 172.21.15.59:0/2277688805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07e415dfe0 0x7f07e40cf2f0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.651+0000 7f07e97f7640 1 -- 172.21.15.59:0/2277688805 >> 172.21.15.59:0/2277688805 conn(0x7f07e40d3cd0 msgr2=0x7f07e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.651+0000 7f07e97f7640 1 -- 172.21.15.59:0/2277688805 shutdown_connections 2024-03-06T07:25:31.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.651+0000 7f07e97f7640 1 -- 172.21.15.59:0/2277688805 wait complete. 2024-03-06T07:25:31.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.652+0000 7f07e97f7640 1 Processor -- start 2024-03-06T07:25:31.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.652+0000 7f07e97f7640 1 -- start start 2024-03-06T07:25:31.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e97f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07e40cf830 0x7f07e40cce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e97f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07e40d27f0 0x7f07e415e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e97f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07e415dfe0 0x7f07e415ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e97f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e40d1c70 con 0x7f07e40d27f0 2024-03-06T07:25:31.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e97f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e40b3d60 con 0x7f07e415dfe0 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e97f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e40d1df0 con 0x7f07e40cf830 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07e40d27f0 0x7f07e415e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e27fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07e40d27f0 0x7f07e415e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58808/0 (socket says 172.21.15.59:58808) 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e27fc640 1 -- 172.21.15.59:0/3830188910 learned_addr learned my addr 172.21.15.59:0/3830188910 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e37fe640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07e415dfe0 0x7f07e415ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e2ffd640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07e40cf830 0x7f07e40cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e27fc640 1 -- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07e40cf830 msgr2=0x7f07e40cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e27fc640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07e40cf830 0x7f07e40cce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e27fc640 1 -- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07e415dfe0 msgr2=0x7f07e415ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e27fc640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07e415dfe0 0x7f07e415ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.653+0000 7f07e27fc640 1 -- 172.21.15.59:0/3830188910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e40b6a40 con 0x7f07e40d27f0 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.654+0000 7f07e27fc640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07e40d27f0 0x7f07e415e4e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f07cc00e980 tx=0x7f07cc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.654+0000 7f07db7fe640 1 -- 172.21.15.59:0/3830188910 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc017070 con 0x7f07e40d27f0 2024-03-06T07:25:31.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.654+0000 7f07db7fe640 1 -- 172.21.15.59:0/3830188910 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc00ca00 con 0x7f07e40d27f0 2024-03-06T07:25:31.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.654+0000 7f07db7fe640 1 -- 172.21.15.59:0/3830188910 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc00cdd0 con 0x7f07e40d27f0 2024-03-06T07:25:31.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.654+0000 7f07e97f7640 1 -- 172.21.15.59:0/3830188910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e415f050 con 0x7f07e40d27f0 2024-03-06T07:25:31.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.654+0000 7f07e97f7640 1 -- 172.21.15.59:0/3830188910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e40b6ce0 con 0x7f07e40d27f0 2024-03-06T07:25:31.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.655+0000 7f07e97f7640 1 -- 172.21.15.59:0/3830188910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e404f330 con 0x7f07e40d27f0 2024-03-06T07:25:31.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.655+0000 7f07db7fe640 1 -- 172.21.15.59:0/3830188910 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07cc01a070 con 0x7f07e40d27f0 2024-03-06T07:25:31.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.655+0000 7f07db7fe640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07c406b500 0x7f07c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.656+0000 7f07db7fe640 1 -- 172.21.15.59:0/3830188910 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07cc08a0a0 con 0x7f07e40d27f0 2024-03-06T07:25:31.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.656+0000 7f07e2ffd640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07c406b500 0x7f07c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.656+0000 7f07e2ffd640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07c406b500 0x7f07c406d9c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f07dc036000 tx=0x7f07dc03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.660+0000 7f07db7fe640 1 -- 172.21.15.59:0/3830188910 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07cc058960 con 0x7f07e40d27f0 2024-03-06T07:25:31.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.879+0000 7f07e97f7640 1 -- 172.21.15.59:0/3830188910 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07e40b0a60 con 0x7f07e40d27f0 2024-03-06T07:25:31.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.879+0000 7f07db7fe640 1 -- 172.21.15.59:0/3830188910 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f07cc05ca00 con 0x7f07e40d27f0 2024-03-06T07:25:31.882 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:31.882 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:04.517350+0000","modified":"2024-03-06T07:25:06.518231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/250576205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":250576205},{"type":"v1","addr":"172.21.15.164:6837","nonce":250576205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:31.882 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 35 2024-03-06T07:25:31.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 -- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07c406b500 msgr2=0x7f07c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07c406b500 0x7f07c406d9c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f07dc036000 tx=0x7f07dc03c040 comp rx=0 tx=0).stop 2024-03-06T07:25:31.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 -- 172.21.15.59:0/3830188910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07e40d27f0 msgr2=0x7f07e415e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07e40d27f0 0x7f07e415e4e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f07cc00e980 tx=0x7f07cc00ee40 comp rx=0 tx=0).stop 2024-03-06T07:25:31.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 -- 172.21.15.59:0/3830188910 shutdown_connections 2024-03-06T07:25:31.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f07c406b500 0x7f07c406d9c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07e415dfe0 0x7f07e415ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07e40d27f0 0x7f07e415e4e0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 --2- 172.21.15.59:0/3830188910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07e40cf830 0x7f07e40cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 -- 172.21.15.59:0/3830188910 >> 172.21.15.59:0/3830188910 conn(0x7f07e40d3cd0 msgr2=0x7f07e406cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 -- 172.21.15.59:0/3830188910 shutdown_connections 2024-03-06T07:25:31.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.881+0000 7f07e97f7640 1 -- 172.21.15.59:0/3830188910 wait complete. 2024-03-06T07:25:31.894 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.993+0000 7ff3c6d81640 1 Processor -- start 2024-03-06T07:25:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.993+0000 7ff3c6d81640 1 -- start start 2024-03-06T07:25:31.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3c6d81640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3c006da40 0x7ff3c00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3c6d81640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff3c00cd7a0 0x7ff3c015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3c6d81640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00cc980 con 0x7ff3c01424f0 2024-03-06T07:25:31.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3c6d81640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c00cd220 con 0x7ff3c00cd7a0 2024-03-06T07:25:31.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3c6d81640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c00ccc60 con 0x7ff3c006da40 2024-03-06T07:25:31.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3c4b1d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff3c01424f0 0x7ff3c006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38644/0 (socket says 172.21.15.59:38644) 2024-03-06T07:25:31.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3c4b1d640 1 -- 172.21.15.59:0/2265526438 learned_addr learned my addr 172.21.15.59:0/2265526438 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:31.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3bffff640 1 --2- 172.21.15.59:0/2265526438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3c006da40 0x7ff3c00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3c531e640 1 --2- 172.21.15.59:0/2265526438 >> v2:172.21.15.164:3300/0 conn(0x7ff3c00cd7a0 0x7ff3c015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3bf7fe640 1 -- 172.21.15.59:0/2265526438 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953603293 0 0) 0x7ff3c00cc980 con 0x7ff3c01424f0 2024-03-06T07:25:31.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3bf7fe640 1 -- 172.21.15.59:0/2265526438 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a8003520 con 0x7ff3c01424f0 2024-03-06T07:25:31.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3bffff640 1 -- 172.21.15.59:0/2265526438 >> v2:172.21.15.164:3300/0 conn(0x7ff3c00cd7a0 msgr2=0x7ff3c015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3bffff640 1 --2- 172.21.15.59:0/2265526438 >> v2:172.21.15.164:3300/0 conn(0x7ff3c00cd7a0 0x7ff3c015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.994+0000 7ff3bffff640 1 -- 172.21.15.59:0/2265526438 >> v1:172.21.15.59:6789/0 conn(0x7ff3c01424f0 legacy=0x7ff3c006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.995+0000 7ff3bffff640 1 -- 172.21.15.59:0/2265526438 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c00ccf40 con 0x7ff3c006da40 2024-03-06T07:25:32.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.995+0000 7ff3bffff640 1 --2- 172.21.15.59:0/2265526438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3c006da40 0x7ff3c00cc2f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40097a0 tx=0x7ff3b402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0edd615d950eda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.995+0000 7ff3bf7fe640 1 -- 172.21.15.59:0/2265526438 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4035020 con 0x7ff3c006da40 2024-03-06T07:25:32.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.995+0000 7ff3bf7fe640 1 -- 172.21.15.59:0/2265526438 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b4031830 con 0x7ff3c006da40 2024-03-06T07:25:32.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.995+0000 7ff3bf7fe640 1 -- 172.21.15.59:0/2265526438 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4031c00 con 0x7ff3c006da40 2024-03-06T07:25:32.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.996+0000 7ff3c6d81640 1 -- 172.21.15.59:0/2265526438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3c006da40 msgr2=0x7ff3c00cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.996+0000 7ff3c6d81640 1 --2- 172.21.15.59:0/2265526438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3c006da40 0x7ff3c00cc2f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40097a0 tx=0x7ff3b402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.996+0000 7ff3c6d81640 1 -- 172.21.15.59:0/2265526438 shutdown_connections 2024-03-06T07:25:32.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.996+0000 7ff3c6d81640 1 --2- 172.21.15.59:0/2265526438 >> v2:172.21.15.164:3300/0 conn(0x7ff3c00cd7a0 0x7ff3c015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.996+0000 7ff3c6d81640 1 --2- 172.21.15.59:0/2265526438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3c006da40 0x7ff3c00cc2f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.996+0000 7ff3c6d81640 1 -- 172.21.15.59:0/2265526438 >> 172.21.15.59:0/2265526438 conn(0x7ff3c00cf8a0 msgr2=0x7ff3c00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.996+0000 7ff3c6d81640 1 -- 172.21.15.59:0/2265526438 shutdown_connections 2024-03-06T07:25:32.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.996+0000 7ff3c6d81640 1 -- 172.21.15.59:0/2265526438 wait complete. 2024-03-06T07:25:32.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.997+0000 7ff3c6d81640 1 Processor -- start 2024-03-06T07:25:32.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.997+0000 7ff3c6d81640 1 -- start start 2024-03-06T07:25:32.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.998+0000 7ff3c6d81640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3c006da40 0x7ff3c0143580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.998+0000 7ff3c6d81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3c00cd7a0 0x7ff3c0151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.998+0000 7ff3c6d81640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3c01424f0 0x7ff3c00a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.998+0000 7ff3c6d81640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c0071ce0 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.998+0000 7ff3c6d81640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c0071b60 con 0x7ff3c01424f0 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.998+0000 7ff3c6d81640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c0071e60 con 0x7ff3c006da40 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.998+0000 7ff3bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3c00cd7a0 0x7ff3c0151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.998+0000 7ff3bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3c00cd7a0 0x7ff3c0151500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58816/0 (socket says 172.21.15.59:58816) 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.998+0000 7ff3bffff640 1 -- 172.21.15.59:0/3848883704 learned_addr learned my addr 172.21.15.59:0/3848883704 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.998+0000 7ff3c531e640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3c01424f0 0x7ff3c00a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.998+0000 7ff3c4b1d640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3c006da40 0x7ff3c0143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.999+0000 7ff3bffff640 1 -- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3c006da40 msgr2=0x7ff3c0143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.999+0000 7ff3bffff640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3c006da40 0x7ff3c0143580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.999+0000 7ff3bffff640 1 -- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3c01424f0 msgr2=0x7ff3c00a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.999+0000 7ff3bffff640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3c01424f0 0x7ff3c00a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.999+0000 7ff3bffff640 1 -- 172.21.15.59:0/3848883704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c0143ac0 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.999+0000 7ff3bffff640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3c00cd7a0 0x7ff3c0151500 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40098c0 tx=0x7ff3b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.999+0000 7ff3bd7fa640 1 -- 172.21.15.59:0/3848883704 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4048070 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.999+0000 7ff3bd7fa640 1 -- 172.21.15.59:0/3848883704 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b403a6f0 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.999+0000 7ff3bd7fa640 1 -- 172.21.15.59:0/3848883704 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b403aae0 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.999+0000 7ff3c6d81640 1 -- 172.21.15.59:0/3848883704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c0143d40 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:31.999+0000 7ff3c6d81640 1 -- 172.21.15.59:0/3848883704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c0144300 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.000+0000 7ff3c6d81640 1 -- 172.21.15.59:0/3848883704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c004f330 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.000+0000 7ff3bd7fa640 1 -- 172.21.15.59:0/3848883704 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3b404b070 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.001+0000 7ff3bd7fa640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff3ac06b4d0 0x7ff3ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.001+0000 7ff3bd7fa640 1 -- 172.21.15.59:0/3848883704 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3b40ab7b0 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.001+0000 7ff3c4b1d640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff3ac06b4d0 0x7ff3ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.002+0000 7ff3c4b1d640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff3ac06b4d0 0x7ff3ac06d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8005c00 tx=0x7ff3b8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.005+0000 7ff3bd7fa640 1 -- 172.21.15.59:0/3848883704 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3c0143ac0 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.188+0000 7ff3c6d81640 1 -- 172.21.15.59:0/3848883704 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3c01419e0 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.189+0000 7ff3bd7fa640 1 -- 172.21.15.59:0/3848883704 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16656 (secure 0 0 0) 0x7ff3b407a070 con 0x7ff3c00cd7a0 2024-03-06T07:25:32.191 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:32.192 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":70,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:06.152587+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:02.779485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:03.482047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.191+0000 7ff3c6d81640 1 -- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff3ac06b4d0 msgr2=0x7ff3ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.191+0000 7ff3c6d81640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff3ac06b4d0 0x7ff3ac06d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8005c00 tx=0x7ff3b8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.191+0000 7ff3c6d81640 1 -- 172.21.15.59:0/3848883704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3c00cd7a0 msgr2=0x7ff3c0151500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.191+0000 7ff3c6d81640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3c00cd7a0 0x7ff3c0151500 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40098c0 tx=0x7ff3b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.192+0000 7ff3c6d81640 1 -- 172.21.15.59:0/3848883704 shutdown_connections 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.192+0000 7ff3c6d81640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff3ac06b4d0 0x7ff3ac06d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.192+0000 7ff3c6d81640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3c01424f0 0x7ff3c00a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.192+0000 7ff3c6d81640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3c00cd7a0 0x7ff3c0151500 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.192+0000 7ff3c6d81640 1 --2- 172.21.15.59:0/3848883704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3c006da40 0x7ff3c0143580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.192+0000 7ff3c6d81640 1 -- 172.21.15.59:0/3848883704 >> 172.21.15.59:0/3848883704 conn(0x7ff3c00cf8a0 msgr2=0x7ff3c0140080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.192+0000 7ff3c6d81640 1 -- 172.21.15.59:0/3848883704 shutdown_connections 2024-03-06T07:25:32.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.192+0000 7ff3c6d81640 1 -- 172.21.15.59:0/3848883704 wait complete. 2024-03-06T07:25:32.205 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:02.779485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:03.482047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:32.205 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:32.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.308+0000 7ffafdfae640 1 Processor -- start 2024-03-06T07:25:32.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.308+0000 7ffafdfae640 1 -- start start 2024-03-06T07:25:32.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffafdfae640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffaf815def0 0x7ffaf80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffafdfae640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffaf806cab0 0x7ffaf80ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffafdfae640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf806d030 con 0x7ffaf80ce880 2024-03-06T07:25:32.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffafdfae640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf80cf1c0 con 0x7ffaf815def0 2024-03-06T07:25:32.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffafdfae640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf80cc760 con 0x7ffaf806cab0 2024-03-06T07:25:32.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffaf77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffaf815def0 0x7ffaf80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffaf77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffaf815def0 0x7ffaf80b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58832/0 (socket says 172.21.15.59:58832) 2024-03-06T07:25:32.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffaf77fe640 1 -- 172.21.15.59:0/54359214 learned_addr learned my addr 172.21.15.59:0/54359214 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:32.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffaf6ffd640 1 --2- 172.21.15.59:0/54359214 >> v2:172.21.15.164:3300/0 conn(0x7ffaf806cab0 0x7ffaf80ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffaf77fe640 1 -- 172.21.15.59:0/54359214 >> v2:172.21.15.164:3300/0 conn(0x7ffaf806cab0 msgr2=0x7ffaf80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffaf77fe640 1 --2- 172.21.15.59:0/54359214 >> v2:172.21.15.164:3300/0 conn(0x7ffaf806cab0 0x7ffaf80ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffaf77fe640 1 -- 172.21.15.59:0/54359214 >> v1:172.21.15.164:6789/0 conn(0x7ffaf80ce880 legacy=0x7ffaf80cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.309+0000 7ffaf77fe640 1 -- 172.21.15.59:0/54359214 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf806d310 con 0x7ffaf815def0 2024-03-06T07:25:32.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.310+0000 7ffaf77fe640 1 --2- 172.21.15.59:0/54359214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf815def0 0x7ffaf80b41b0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80b3660 tx=0x7ffae802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc80cfa267abdb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.310+0000 7ffaf67fc640 1 -- 172.21.15.59:0/54359214 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae802f850 con 0x7ffaf815def0 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.310+0000 7ffaf67fc640 1 -- 172.21.15.59:0/54359214 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae802f9f0 con 0x7ffaf815def0 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.310+0000 7ffaf67fc640 1 -- 172.21.15.59:0/54359214 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae802fdc0 con 0x7ffaf815def0 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.311+0000 7ffafdfae640 1 -- 172.21.15.59:0/54359214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf815def0 msgr2=0x7ffaf80b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.311+0000 7ffafdfae640 1 --2- 172.21.15.59:0/54359214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf815def0 0x7ffaf80b41b0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7ffaf80b3660 tx=0x7ffae802ba30 comp rx=0 tx=0).stop 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.311+0000 7ffafdfae640 1 -- 172.21.15.59:0/54359214 shutdown_connections 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.311+0000 7ffafdfae640 1 --2- 172.21.15.59:0/54359214 >> v2:172.21.15.164:3300/0 conn(0x7ffaf806cab0 0x7ffaf80ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.311+0000 7ffafdfae640 1 --2- 172.21.15.59:0/54359214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf815def0 0x7ffaf80b41b0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.311+0000 7ffafdfae640 1 -- 172.21.15.59:0/54359214 >> 172.21.15.59:0/54359214 conn(0x7ffaf80b3690 msgr2=0x7ffaf80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.311+0000 7ffafdfae640 1 -- 172.21.15.59:0/54359214 shutdown_connections 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.311+0000 7ffafdfae640 1 -- 172.21.15.59:0/54359214 wait complete. 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.312+0000 7ffafdfae640 1 Processor -- start 2024-03-06T07:25:32.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.312+0000 7ffafdfae640 1 -- start start 2024-03-06T07:25:32.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffafdfae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf806cab0 0x7ffaf814bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffafdfae640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffaf80ce880 0x7ffaf80c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffafdfae640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffaf815def0 0x7ffaf80c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffafdfae640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80cc3f0 con 0x7ffaf806cab0 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffafdfae640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80cecb0 con 0x7ffaf815def0 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffafdfae640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815e2d0 con 0x7ffaf80ce880 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf806cab0 0x7ffaf814bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf806cab0 0x7ffaf814bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58840/0 (socket says 172.21.15.59:58840) 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf77fe640 1 -- 172.21.15.59:0/3501927685 learned_addr learned my addr 172.21.15.59:0/3501927685 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf6ffd640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffaf80ce880 0x7ffaf80c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf7fff640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffaf815def0 0x7ffaf80c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf77fe640 1 -- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffaf80ce880 msgr2=0x7ffaf80c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf77fe640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffaf80ce880 0x7ffaf80c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf77fe640 1 -- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffaf815def0 msgr2=0x7ffaf80c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf77fe640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffaf815def0 0x7ffaf80c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf77fe640 1 -- 172.21.15.59:0/3501927685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf814c1a0 con 0x7ffaf806cab0 2024-03-06T07:25:32.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf6ffd640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffaf80ce880 0x7ffaf80c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:32.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffaf77fe640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf806cab0 0x7ffaf814bc60 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ffae802ba70 tx=0x7ffae8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.313+0000 7ffadffff640 1 -- 172.21.15.59:0/3501927685 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8037d10 con 0x7ffaf806cab0 2024-03-06T07:25:32.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.314+0000 7ffadffff640 1 -- 172.21.15.59:0/3501927685 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae8046070 con 0x7ffaf806cab0 2024-03-06T07:25:32.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.314+0000 7ffadffff640 1 -- 172.21.15.59:0/3501927685 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8041640 con 0x7ffaf806cab0 2024-03-06T07:25:32.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.314+0000 7ffafdfae640 1 -- 172.21.15.59:0/3501927685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf80c6ce0 con 0x7ffaf806cab0 2024-03-06T07:25:32.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.314+0000 7ffafdfae640 1 -- 172.21.15.59:0/3501927685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf80c72a0 con 0x7ffaf806cab0 2024-03-06T07:25:32.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.314+0000 7ffafdfae640 1 -- 172.21.15.59:0/3501927685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf809fcf0 con 0x7ffaf806cab0 2024-03-06T07:25:32.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.315+0000 7ffadffff640 1 -- 172.21.15.59:0/3501927685 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffae8040070 con 0x7ffaf806cab0 2024-03-06T07:25:32.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.315+0000 7ffadffff640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffae406b4d0 0x7ffae406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.315+0000 7ffadffff640 1 -- 172.21.15.59:0/3501927685 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffae80ac840 con 0x7ffaf806cab0 2024-03-06T07:25:32.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.315+0000 7ffaf6ffd640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffae406b4d0 0x7ffae406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.316+0000 7ffaf6ffd640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffae406b4d0 0x7ffae406d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ffaf81f7290 tx=0x7ffaec008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.319+0000 7ffadffff640 1 -- 172.21.15.59:0/3501927685 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffae8033030 con 0x7ffaf806cab0 2024-03-06T07:25:32.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.498+0000 7ffafdfae640 1 -- 172.21.15.59:0/3501927685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffaf80b78d0 con 0x7ffaf806cab0 2024-03-06T07:25:32.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.499+0000 7ffadffff640 1 -- 172.21.15.59:0/3501927685 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffae807b6c0 con 0x7ffaf806cab0 2024-03-06T07:25:32.501 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:25:32.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 -- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffae406b4d0 msgr2=0x7ffae406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffae406b4d0 0x7ffae406d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ffaf81f7290 tx=0x7ffaec008040 comp rx=0 tx=0).stop 2024-03-06T07:25:32.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 -- 172.21.15.59:0/3501927685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf806cab0 msgr2=0x7ffaf814bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf806cab0 0x7ffaf814bc60 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7ffae802ba70 tx=0x7ffae8004730 comp rx=0 tx=0).stop 2024-03-06T07:25:32.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 -- 172.21.15.59:0/3501927685 shutdown_connections 2024-03-06T07:25:32.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffae406b4d0 0x7ffae406d990 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffaf815def0 0x7ffaf80c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffaf80ce880 0x7ffaf80c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 --2- 172.21.15.59:0/3501927685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaf806cab0 0x7ffaf814bc60 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 -- 172.21.15.59:0/3501927685 >> 172.21.15.59:0/3501927685 conn(0x7ffaf80b3690 msgr2=0x7ffaf80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 -- 172.21.15.59:0/3501927685 shutdown_connections 2024-03-06T07:25:32.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.501+0000 7ffafdfae640 1 -- 172.21.15.59:0/3501927685 wait complete. 2024-03-06T07:25:32.513 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:32.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.616+0000 7f3886dc7640 1 Processor -- start 2024-03-06T07:25:32.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.616+0000 7f3886dc7640 1 -- start start 2024-03-06T07:25:32.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3886dc7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38800d4860 0x7f38800d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3886dc7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f38800b0e30 0x7f38800b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3886dc7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38801618b0 con 0x7f38801614d0 2024-03-06T07:25:32.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3886dc7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38800d4cf0 con 0x7f38800d4860 2024-03-06T07:25:32.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3886dc7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38800cf400 con 0x7f38800b0e30 2024-03-06T07:25:32.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f387ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38800d4860 0x7f38800d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3885364640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f38800b0e30 0x7f38800b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3885364640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f38800b0e30 0x7f38800b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:35222/0 (socket says 172.21.15.59:35222) 2024-03-06T07:25:32.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f387ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38800d4860 0x7f38800d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58842/0 (socket says 172.21.15.59:58842) 2024-03-06T07:25:32.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3885364640 1 -- 172.21.15.59:0/2655482713 learned_addr learned my addr 172.21.15.59:0/2655482713 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:32.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3884b63640 1 --1- 172.21.15.59:0/2655482713 >> v1:172.21.15.164:6789/0 conn(0x7f38801614d0 0x7f38800d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:42898/0 (socket says 172.21.15.59:42898) 2024-03-06T07:25:32.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3885364640 1 -- 172.21.15.59:0/2655482713 >> v2:172.21.15.59:3300/0 conn(0x7f38800d4860 msgr2=0x7f38800d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3885364640 1 --2- 172.21.15.59:0/2655482713 >> v2:172.21.15.59:3300/0 conn(0x7f38800d4860 0x7f38800d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3885364640 1 -- 172.21.15.59:0/2655482713 >> v1:172.21.15.164:6789/0 conn(0x7f38801614d0 legacy=0x7f38800d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f3885364640 1 -- 172.21.15.59:0/2655482713 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38800cf6a0 con 0x7f38800b0e30 2024-03-06T07:25:32.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.617+0000 7f387ffff640 1 --2- 172.21.15.59:0/2655482713 >> v2:172.21.15.59:3300/0 conn(0x7f38800d4860 0x7f38800d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:32.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.618+0000 7f3885364640 1 --2- 172.21.15.59:0/2655482713 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800b0e30 0x7f38800b1260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f38800b2a70 tx=0x7f387802bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff5f8f3237ea142f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.618+0000 7f387f7fe640 1 -- 172.21.15.59:0/2655482713 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387802cb10 con 0x7f38800b0e30 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.618+0000 7f387f7fe640 1 -- 172.21.15.59:0/2655482713 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387802ccb0 con 0x7f38800b0e30 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.618+0000 7f387f7fe640 1 -- 172.21.15.59:0/2655482713 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387802ce60 con 0x7f38800b0e30 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.619+0000 7f3886dc7640 1 -- 172.21.15.59:0/2655482713 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800b0e30 msgr2=0x7f38800b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.619+0000 7f3886dc7640 1 --2- 172.21.15.59:0/2655482713 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800b0e30 0x7f38800b1260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f38800b2a70 tx=0x7f387802bf70 comp rx=0 tx=0).stop 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.619+0000 7f3886dc7640 1 -- 172.21.15.59:0/2655482713 shutdown_connections 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.619+0000 7f3886dc7640 1 --2- 172.21.15.59:0/2655482713 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800b0e30 0x7f38800b1260 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.619+0000 7f3886dc7640 1 --2- 172.21.15.59:0/2655482713 >> v2:172.21.15.59:3300/0 conn(0x7f38800d4860 0x7f38800d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.619+0000 7f3886dc7640 1 -- 172.21.15.59:0/2655482713 >> 172.21.15.59:0/2655482713 conn(0x7f38800d3d40 msgr2=0x7f38800d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.619+0000 7f3886dc7640 1 -- 172.21.15.59:0/2655482713 shutdown_connections 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.619+0000 7f3886dc7640 1 -- 172.21.15.59:0/2655482713 wait complete. 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.620+0000 7f3886dc7640 1 Processor -- start 2024-03-06T07:25:32.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.620+0000 7f3886dc7640 1 -- start start 2024-03-06T07:25:32.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3886dc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38800b0e30 0x7f38800cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3886dc7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800d4860 0x7f38800b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3886dc7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38801614d0 0x7f388015ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3886dc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38800b0910 con 0x7f38800b0e30 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3886dc7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38800cfc60 con 0x7f38800d4860 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3886dc7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38800d2560 con 0x7f38801614d0 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3884b63640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38800b0e30 0x7f38800cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3884b63640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38800b0e30 0x7f38800cd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58846/0 (socket says 172.21.15.59:58846) 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3884b63640 1 -- 172.21.15.59:0/3592503889 learned_addr learned my addr 172.21.15.59:0/3592503889 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f387ffff640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800d4860 0x7f38800b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3885364640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38801614d0 0x7f388015ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3884b63640 1 -- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38801614d0 msgr2=0x7f388015ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3884b63640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38801614d0 0x7f388015ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3884b63640 1 -- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800d4860 msgr2=0x7f38800b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3884b63640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800d4860 0x7f38800b8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3884b63640 1 -- 172.21.15.59:0/3592503889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38800b7c20 con 0x7f38800b0e30 2024-03-06T07:25:32.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3884b63640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38800b0e30 0x7f38800cd790 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f387400c3c0 tx=0x7f387400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f387d7fa640 1 -- 172.21.15.59:0/3592503889 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38740074e0 con 0x7f38800b0e30 2024-03-06T07:25:32.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f387d7fa640 1 -- 172.21.15.59:0/3592503889 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3874007680 con 0x7f38800b0e30 2024-03-06T07:25:32.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.621+0000 7f3886dc7640 1 -- 172.21.15.59:0/3592503889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38800b87d0 con 0x7f38800b0e30 2024-03-06T07:25:32.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.622+0000 7f387d7fa640 1 -- 172.21.15.59:0/3592503889 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3874004a00 con 0x7f38800b0e30 2024-03-06T07:25:32.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.622+0000 7f3886dc7640 1 -- 172.21.15.59:0/3592503889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38800cbd10 con 0x7f38800b0e30 2024-03-06T07:25:32.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.623+0000 7f3886dc7640 1 -- 172.21.15.59:0/3592503889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3880068ff0 con 0x7f38800b0e30 2024-03-06T07:25:32.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.623+0000 7f387d7fa640 1 -- 172.21.15.59:0/3592503889 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f387401a020 con 0x7f38800b0e30 2024-03-06T07:25:32.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.623+0000 7f387d7fa640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386406b4d0 0x7f386406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.623+0000 7f387d7fa640 1 -- 172.21.15.59:0/3592503889 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3874089870 con 0x7f38800b0e30 2024-03-06T07:25:32.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.624+0000 7f387ffff640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386406b4d0 0x7f386406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.624+0000 7f387ffff640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386406b4d0 0x7f386406d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f386c009560 tx=0x7f386c009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.628+0000 7f387d7fa640 1 -- 172.21.15.59:0/3592503889 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3874058020 con 0x7f38800b0e30 2024-03-06T07:25:32.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.808+0000 7f3886dc7640 1 -- 172.21.15.59:0/3592503889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f38800b25b0 con 0x7f38800b0e30 2024-03-06T07:25:32.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.808+0000 7f387d7fa640 1 -- 172.21.15.59:0/3592503889 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f387405c0c0 con 0x7f38800b0e30 2024-03-06T07:25:32.810 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:25:32.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 -- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386406b4d0 msgr2=0x7f386406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386406b4d0 0x7f386406d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f386c009560 tx=0x7f386c009190 comp rx=0 tx=0).stop 2024-03-06T07:25:32.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 -- 172.21.15.59:0/3592503889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38800b0e30 msgr2=0x7f38800cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38800b0e30 0x7f38800cd790 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f387400c3c0 tx=0x7f387400c880 comp rx=0 tx=0).stop 2024-03-06T07:25:32.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 -- 172.21.15.59:0/3592503889 shutdown_connections 2024-03-06T07:25:32.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386406b4d0 0x7f386406d990 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38801614d0 0x7f388015ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800d4860 0x7f38800b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 --2- 172.21.15.59:0/3592503889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38800b0e30 0x7f38800cd790 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 -- 172.21.15.59:0/3592503889 >> 172.21.15.59:0/3592503889 conn(0x7f38800d3d40 msgr2=0x7f38800cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 -- 172.21.15.59:0/3592503889 shutdown_connections 2024-03-06T07:25:32.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.810+0000 7f3886dc7640 1 -- 172.21.15.59:0/3592503889 wait complete. 2024-03-06T07:25:32.823 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:32.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.925+0000 7fe3963f7640 1 Processor -- start 2024-03-06T07:25:32.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.925+0000 7fe3963f7640 1 -- start start 2024-03-06T07:25:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.926+0000 7fe3963f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe39006cab0 0x7fe3900b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.926+0000 7fe3963f7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39006d030 con 0x7fe3900cea50 2024-03-06T07:25:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.926+0000 7fe3963f7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39006d310 con 0x7fe39015def0 2024-03-06T07:25:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.926+0000 7fe3963f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3900cf1c0 con 0x7fe39006cab0 2024-03-06T07:25:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.926+0000 7fe38f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe3900cea50 0x7fe3900ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38652/0 (socket says 172.21.15.59:38652) 2024-03-06T07:25:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.926+0000 7fe38f7fe640 1 -- 172.21.15.59:0/3330420276 learned_addr learned my addr 172.21.15.59:0/3330420276 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.926+0000 7fe394994640 1 --2- 172.21.15.59:0/3330420276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe39006cab0 0x7fe3900b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.926+0000 7fe38effd640 1 -- 172.21.15.59:0/3330420276 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695447194 0 0) 0x7fe39006d030 con 0x7fe3900cea50 2024-03-06T07:25:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.926+0000 7fe38effd640 1 -- 172.21.15.59:0/3330420276 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe37c003520 con 0x7fe3900cea50 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.926+0000 7fe38effd640 1 -- 172.21.15.59:0/3330420276 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974119782 0 0) 0x7fe39006d310 con 0x7fe39015def0 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.927+0000 7fe38effd640 1 -- 172.21.15.59:0/3330420276 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe39006d030 con 0x7fe39015def0 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.927+0000 7fe394994640 1 -- 172.21.15.59:0/3330420276 >> v1:172.21.15.164:6789/0 conn(0x7fe39015def0 legacy=0x7fe3900ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.927+0000 7fe394994640 1 -- 172.21.15.59:0/3330420276 >> v1:172.21.15.59:6789/0 conn(0x7fe3900cea50 legacy=0x7fe3900ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.927+0000 7fe394994640 1 -- 172.21.15.59:0/3330420276 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3900cc110 con 0x7fe39006cab0 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.927+0000 7fe394994640 1 --2- 172.21.15.59:0/3330420276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe39006cab0 0x7fe3900b41b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe388009dc0 tx=0x7fe38802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=627f10bdfe4dbe33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.927+0000 7fe38effd640 1 -- 172.21.15.59:0/3330420276 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388039070 con 0x7fe39006cab0 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.928+0000 7fe38effd640 1 -- 172.21.15.59:0/3330420276 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38802cd50 con 0x7fe39006cab0 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.928+0000 7fe38effd640 1 -- 172.21.15.59:0/3330420276 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388034660 con 0x7fe39006cab0 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.928+0000 7fe3963f7640 1 -- 172.21.15.59:0/3330420276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe39006cab0 msgr2=0x7fe3900b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.928+0000 7fe3963f7640 1 --2- 172.21.15.59:0/3330420276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe39006cab0 0x7fe3900b41b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe388009dc0 tx=0x7fe38802bfa0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.928+0000 7fe3963f7640 1 -- 172.21.15.59:0/3330420276 shutdown_connections 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.928+0000 7fe3963f7640 1 --2- 172.21.15.59:0/3330420276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe39006cab0 0x7fe3900b41b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.928+0000 7fe3963f7640 1 -- 172.21.15.59:0/3330420276 >> 172.21.15.59:0/3330420276 conn(0x7fe3900b3690 msgr2=0x7fe3900b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.928+0000 7fe3963f7640 1 -- 172.21.15.59:0/3330420276 shutdown_connections 2024-03-06T07:25:32.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.928+0000 7fe3963f7640 1 -- 172.21.15.59:0/3330420276 wait complete. 2024-03-06T07:25:32.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.929+0000 7fe3963f7640 1 Processor -- start 2024-03-06T07:25:32.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.929+0000 7fe3963f7640 1 -- start start 2024-03-06T07:25:32.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe3963f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe39006cab0 0x7fe390160c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe3963f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3900cea50 0x7fe3900c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe3963f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe39015def0 0x7fe390157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe3963f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39015e2d0 con 0x7fe39015def0 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe3963f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3900cc760 con 0x7fe39006cab0 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe3963f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39015e450 con 0x7fe3900cea50 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe394994640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe39015def0 0x7fe390157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe394994640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe39015def0 0x7fe390157940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58862/0 (socket says 172.21.15.59:58862) 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe394994640 1 -- 172.21.15.59:0/291624566 learned_addr learned my addr 172.21.15.59:0/291624566 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe38ffff640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe39006cab0 0x7fe390160c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe38f7fe640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3900cea50 0x7fe3900c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe394994640 1 -- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3900cea50 msgr2=0x7fe3900c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe394994640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3900cea50 0x7fe3900c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe394994640 1 -- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe39006cab0 msgr2=0x7fe390160c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe394994640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe39006cab0 0x7fe390160c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.930+0000 7fe394994640 1 -- 172.21.15.59:0/291624566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3901f8280 con 0x7fe39015def0 2024-03-06T07:25:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.931+0000 7fe394994640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe39015def0 0x7fe390157940 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe388009ee0 tx=0x7fe388004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.931+0000 7fe38cff9640 1 -- 172.21.15.59:0/291624566 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388039050 con 0x7fe39015def0 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.931+0000 7fe38cff9640 1 -- 172.21.15.59:0/291624566 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe388045070 con 0x7fe39015def0 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.931+0000 7fe3963f7640 1 -- 172.21.15.59:0/291624566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3900c5670 con 0x7fe39015def0 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.931+0000 7fe38cff9640 1 -- 172.21.15.59:0/291624566 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388009450 con 0x7fe39015def0 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.931+0000 7fe3963f7640 1 -- 172.21.15.59:0/291624566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3900c5bf0 con 0x7fe39015def0 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.932+0000 7fe3963f7640 1 -- 172.21.15.59:0/291624566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39009fcf0 con 0x7fe39015def0 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.932+0000 7fe38cff9640 1 -- 172.21.15.59:0/291624566 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe38803e850 con 0x7fe39015def0 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.933+0000 7fe38cff9640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe37006b4d0 0x7fe37006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.933+0000 7fe38cff9640 1 -- 172.21.15.59:0/291624566 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3880acf30 con 0x7fe39015def0 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.933+0000 7fe38ffff640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe37006b4d0 0x7fe37006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.933+0000 7fe38ffff640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe37006b4d0 0x7fe37006d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe378007fd0 tx=0x7fe378009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:32.937+0000 7fe38cff9640 1 -- 172.21.15.59:0/291624566 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe38807b9d0 con 0x7fe39015def0 2024-03-06T07:25:33.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.117+0000 7fe3963f7640 1 -- 172.21.15.59:0/291624566 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe39015e5d0 con 0x7fe39015def0 2024-03-06T07:25:33.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.117+0000 7fe38cff9640 1 -- 172.21.15.59:0/291624566 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe38807fa70 con 0x7fe39015def0 2024-03-06T07:25:33.119 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:25:33.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 -- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe37006b4d0 msgr2=0x7fe37006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe37006b4d0 0x7fe37006d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe378007fd0 tx=0x7fe378009040 comp rx=0 tx=0).stop 2024-03-06T07:25:33.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 -- 172.21.15.59:0/291624566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe39015def0 msgr2=0x7fe390157940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe39015def0 0x7fe390157940 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fe388009ee0 tx=0x7fe388004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:33.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 -- 172.21.15.59:0/291624566 shutdown_connections 2024-03-06T07:25:33.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe37006b4d0 0x7fe37006d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe39015def0 0x7fe390157940 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3900cea50 0x7fe3900c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 --2- 172.21.15.59:0/291624566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe39006cab0 0x7fe390160c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 -- 172.21.15.59:0/291624566 >> 172.21.15.59:0/291624566 conn(0x7fe3900b3690 msgr2=0x7fe3900cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 -- 172.21.15.59:0/291624566 shutdown_connections 2024-03-06T07:25:33.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.119+0000 7fe3963f7640 1 -- 172.21.15.59:0/291624566 wait complete. 2024-03-06T07:25:33.132 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:33.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.231+0000 7f9cad02a640 1 Processor -- start 2024-03-06T07:25:33.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.231+0000 7f9cad02a640 1 -- start start 2024-03-06T07:25:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9cad02a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca80cf850 0x7f9ca80cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9cad02a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9ca806cf20 0x7f9ca806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9cad02a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca815e380 con 0x7f9ca80d00c0 2024-03-06T07:25:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9cad02a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca806ca90 con 0x7f9ca806cf20 2024-03-06T07:25:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9cad02a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80cdc10 con 0x7f9ca80cf850 2024-03-06T07:25:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9ca6d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9ca80d00c0 0x7f9ca80d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38656/0 (socket says 172.21.15.59:38656) 2024-03-06T07:25:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9ca6d9d640 1 -- 172.21.15.59:0/110677902 learned_addr learned my addr 172.21.15.59:0/110677902 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:33.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9ca759e640 1 --2- 172.21.15.59:0/110677902 >> v2:172.21.15.59:3300/0 conn(0x7f9ca806cf20 0x7f9ca806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9ca659c640 1 --2- 172.21.15.59:0/110677902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca80cf850 0x7f9ca80cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9ca5d9b640 1 -- 172.21.15.59:0/110677902 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807473599 0 0) 0x7f9ca815e380 con 0x7f9ca80d00c0 2024-03-06T07:25:33.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9ca5d9b640 1 -- 172.21.15.59:0/110677902 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c94003520 con 0x7f9ca80d00c0 2024-03-06T07:25:33.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9ca759e640 1 -- 172.21.15.59:0/110677902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca80cf850 msgr2=0x7f9ca80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9ca759e640 1 --2- 172.21.15.59:0/110677902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca80cf850 0x7f9ca80cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9ca759e640 1 -- 172.21.15.59:0/110677902 >> v1:172.21.15.59:6789/0 conn(0x7f9ca80d00c0 legacy=0x7f9ca80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9ca759e640 1 -- 172.21.15.59:0/110677902 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca806c160 con 0x7f9ca806cf20 2024-03-06T07:25:33.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.232+0000 7f9ca659c640 1 --2- 172.21.15.59:0/110677902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca80cf850 0x7f9ca80cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.233+0000 7f9ca759e640 1 --2- 172.21.15.59:0/110677902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca806cf20 0x7f9ca806d3b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009d90 tx=0x7f9c9802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29cc71b3b348668f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.233+0000 7f9ca5d9b640 1 -- 172.21.15.59:0/110677902 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98039070 con 0x7f9ca806cf20 2024-03-06T07:25:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.233+0000 7f9ca5d9b640 1 -- 172.21.15.59:0/110677902 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9802cc50 con 0x7f9ca806cf20 2024-03-06T07:25:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.233+0000 7f9ca5d9b640 1 -- 172.21.15.59:0/110677902 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98034440 con 0x7f9ca806cf20 2024-03-06T07:25:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.234+0000 7f9cad02a640 1 -- 172.21.15.59:0/110677902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca806cf20 msgr2=0x7f9ca806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.234+0000 7f9cad02a640 1 --2- 172.21.15.59:0/110677902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca806cf20 0x7f9ca806d3b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009d90 tx=0x7f9c9802bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.234+0000 7f9cad02a640 1 -- 172.21.15.59:0/110677902 shutdown_connections 2024-03-06T07:25:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.234+0000 7f9cad02a640 1 --2- 172.21.15.59:0/110677902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca806cf20 0x7f9ca806d3b0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.234+0000 7f9cad02a640 1 --2- 172.21.15.59:0/110677902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca80cf850 0x7f9ca80cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.234+0000 7f9cad02a640 1 -- 172.21.15.59:0/110677902 >> 172.21.15.59:0/110677902 conn(0x7f9ca80b35a0 msgr2=0x7f9ca80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.234+0000 7f9cad02a640 1 -- 172.21.15.59:0/110677902 shutdown_connections 2024-03-06T07:25:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.234+0000 7f9cad02a640 1 -- 172.21.15.59:0/110677902 wait complete. 2024-03-06T07:25:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.235+0000 7f9cad02a640 1 Processor -- start 2024-03-06T07:25:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.235+0000 7f9cad02a640 1 -- start start 2024-03-06T07:25:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9cad02a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca806cf20 0x7f9ca8158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9cad02a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca80cf850 0x7f9ca80b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9cad02a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca80d00c0 0x7f9ca80caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9cad02a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80cd900 con 0x7f9ca80d00c0 2024-03-06T07:25:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9cad02a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca806c720 con 0x7f9ca806cf20 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9cad02a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80b1030 con 0x7f9ca80cf850 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9ca759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca80d00c0 0x7f9ca80caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9ca759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca80d00c0 0x7f9ca80caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58866/0 (socket says 172.21.15.59:58866) 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9ca759e640 1 -- 172.21.15.59:0/3505495358 learned_addr learned my addr 172.21.15.59:0/3505495358 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9ca6d9d640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca806cf20 0x7f9ca8158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9ca659c640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca80cf850 0x7f9ca80b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9ca759e640 1 -- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca80cf850 msgr2=0x7f9ca80b5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9ca759e640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca80cf850 0x7f9ca80b5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9ca759e640 1 -- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca806cf20 msgr2=0x7f9ca8158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9ca759e640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca806cf20 0x7f9ca8158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9ca759e640 1 -- 172.21.15.59:0/3505495358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca80ca1a0 con 0x7f9ca80d00c0 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.236+0000 7f9ca759e640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca80d00c0 0x7f9ca80caaf0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009eb0 tx=0x7f9c98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.237+0000 7f9c9f7fe640 1 -- 172.21.15.59:0/3505495358 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98039050 con 0x7f9ca80d00c0 2024-03-06T07:25:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.237+0000 7f9c9f7fe640 1 -- 172.21.15.59:0/3505495358 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c98045070 con 0x7f9ca80d00c0 2024-03-06T07:25:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.237+0000 7f9cad02a640 1 -- 172.21.15.59:0/3505495358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca80c56b0 con 0x7f9ca80d00c0 2024-03-06T07:25:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.237+0000 7f9c9f7fe640 1 -- 172.21.15.59:0/3505495358 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98009450 con 0x7f9ca80d00c0 2024-03-06T07:25:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.237+0000 7f9cad02a640 1 -- 172.21.15.59:0/3505495358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca80c5c70 con 0x7f9ca80d00c0 2024-03-06T07:25:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.237+0000 7f9cad02a640 1 -- 172.21.15.59:0/3505495358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca8068150 con 0x7f9ca80d00c0 2024-03-06T07:25:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.238+0000 7f9c9f7fe640 1 -- 172.21.15.59:0/3505495358 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c9803e610 con 0x7f9ca80d00c0 2024-03-06T07:25:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.238+0000 7f9c9f7fe640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.238+0000 7f9c9f7fe640 1 -- 172.21.15.59:0/3505495358 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c980aca40 con 0x7f9ca80d00c0 2024-03-06T07:25:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.239+0000 7f9ca6d9d640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.239+0000 7f9ca6d9d640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b4d0 0x7f9c8806d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005880 tx=0x7f9ca0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.243+0000 7f9c9f7fe640 1 -- 172.21.15.59:0/3505495358 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ca8068150 con 0x7f9ca80d00c0 2024-03-06T07:25:33.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.461+0000 7f9cad02a640 1 -- 172.21.15.59:0/3505495358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ca804f330 con 0x7f9ca80d00c0 2024-03-06T07:25:33.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.462+0000 7f9c9f7fe640 1 -- 172.21.15.59:0/3505495358 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f9c980ace20 con 0x7f9ca80d00c0 2024-03-06T07:25:33.464 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:33.464 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:04.517350+0000","modified":"2024-03-06T07:25:06.518231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/250576205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":250576205},{"type":"v1","addr":"172.21.15.164:6837","nonce":250576205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:33.464 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 35 2024-03-06T07:25:33.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 -- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b4d0 msgr2=0x7f9c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b4d0 0x7f9c8806d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005880 tx=0x7f9ca0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:33.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 -- 172.21.15.59:0/3505495358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca80d00c0 msgr2=0x7f9ca80caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca80d00c0 0x7f9ca80caaf0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f9c98009eb0 tx=0x7f9c98004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:33.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 -- 172.21.15.59:0/3505495358 shutdown_connections 2024-03-06T07:25:33.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ca80d00c0 0x7f9ca80caaf0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ca80cf850 0x7f9ca80b5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 --2- 172.21.15.59:0/3505495358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ca806cf20 0x7f9ca8158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 -- 172.21.15.59:0/3505495358 >> 172.21.15.59:0/3505495358 conn(0x7f9ca80b35a0 msgr2=0x7f9ca80b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 -- 172.21.15.59:0/3505495358 shutdown_connections 2024-03-06T07:25:33.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.463+0000 7f9cad02a640 1 -- 172.21.15.59:0/3505495358 wait complete. 2024-03-06T07:25:33.476 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:33.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.575+0000 7fdb2a247640 1 Processor -- start 2024-03-06T07:25:33.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.575+0000 7fdb2a247640 1 -- start start 2024-03-06T07:25:33.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.576+0000 7fdb2a247640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb240b4420 0x7fdb240cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.576+0000 7fdb2a247640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdb240ce0d0 0x7fdb2415c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.576+0000 7fdb2a247640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240d0a70 con 0x7fdb2406afb0 2024-03-06T07:25:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.576+0000 7fdb2a247640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb240d0850 con 0x7fdb240ce0d0 2024-03-06T07:25:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.576+0000 7fdb2a247640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240d0d10 con 0x7fdb240b4420 2024-03-06T07:25:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.576+0000 7fdb23fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdb240ce0d0 0x7fdb2415c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.576+0000 7fdb237fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdb2406afb0 0x7fdb2406b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55274/0 (socket says 172.21.15.59:55274) 2024-03-06T07:25:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.576+0000 7fdb237fe640 1 -- 172.21.15.59:0/1223883463 learned_addr learned my addr 172.21.15.59:0/1223883463 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.576+0000 7fdb22ffd640 1 --2- 172.21.15.59:0/1223883463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb240b4420 0x7fdb240cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.577+0000 7fdb23fff640 1 -- 172.21.15.59:0/1223883463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb240b4420 msgr2=0x7fdb240cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.577+0000 7fdb23fff640 1 --2- 172.21.15.59:0/1223883463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb240b4420 0x7fdb240cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.577+0000 7fdb23fff640 1 -- 172.21.15.59:0/1223883463 >> v1:172.21.15.59:6789/0 conn(0x7fdb2406afb0 legacy=0x7fdb2406b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.577+0000 7fdb23fff640 1 -- 172.21.15.59:0/1223883463 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb240d0ff0 con 0x7fdb240ce0d0 2024-03-06T07:25:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.577+0000 7fdb23fff640 1 --2- 172.21.15.59:0/1223883463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb240ce0d0 0x7fdb2415c490 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009d90 tx=0x7fdb1c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5959381a7bb4318 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.578+0000 7fdb227fc640 1 -- 172.21.15.59:0/1223883463 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c039070 con 0x7fdb240ce0d0 2024-03-06T07:25:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.578+0000 7fdb227fc640 1 -- 172.21.15.59:0/1223883463 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c02cbc0 con 0x7fdb240ce0d0 2024-03-06T07:25:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.578+0000 7fdb227fc640 1 -- 172.21.15.59:0/1223883463 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c034440 con 0x7fdb240ce0d0 2024-03-06T07:25:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.578+0000 7fdb2a247640 1 -- 172.21.15.59:0/1223883463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb240ce0d0 msgr2=0x7fdb2415c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.578+0000 7fdb2a247640 1 --2- 172.21.15.59:0/1223883463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb240ce0d0 0x7fdb2415c490 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009d90 tx=0x7fdb1c02be10 comp rx=0 tx=0).stop 2024-03-06T07:25:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.578+0000 7fdb2a247640 1 -- 172.21.15.59:0/1223883463 shutdown_connections 2024-03-06T07:25:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.578+0000 7fdb2a247640 1 --2- 172.21.15.59:0/1223883463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb240ce0d0 0x7fdb2415c490 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.578+0000 7fdb2a247640 1 --2- 172.21.15.59:0/1223883463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb240b4420 0x7fdb240cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.578+0000 7fdb2a247640 1 -- 172.21.15.59:0/1223883463 >> 172.21.15.59:0/1223883463 conn(0x7fdb2406c0c0 msgr2=0x7fdb2406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.578+0000 7fdb2a247640 1 -- 172.21.15.59:0/1223883463 shutdown_connections 2024-03-06T07:25:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.578+0000 7fdb2a247640 1 -- 172.21.15.59:0/1223883463 wait complete. 2024-03-06T07:25:33.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.579+0000 7fdb2a247640 1 Processor -- start 2024-03-06T07:25:33.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.579+0000 7fdb2a247640 1 -- start start 2024-03-06T07:25:33.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.580+0000 7fdb2a247640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb2406afb0 0x7fdb24155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.580+0000 7fdb2a247640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb240b4420 0x7fdb2415de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.580+0000 7fdb2a247640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb240ce0d0 0x7fdb2415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.580+0000 7fdb2a247640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240ceb90 con 0x7fdb240ce0d0 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.580+0000 7fdb2a247640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24145840 con 0x7fdb240b4420 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.580+0000 7fdb2a247640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2415c9d0 con 0x7fdb2406afb0 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.580+0000 7fdb23fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb240ce0d0 0x7fdb2415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.580+0000 7fdb23fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb240ce0d0 0x7fdb2415e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44662/0 (socket says 172.21.15.59:44662) 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.580+0000 7fdb23fff640 1 -- 172.21.15.59:0/4033140472 learned_addr learned my addr 172.21.15.59:0/4033140472 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.580+0000 7fdb22ffd640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb240b4420 0x7fdb2415de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.580+0000 7fdb237fe640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb2406afb0 0x7fdb24155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb22ffd640 1 -- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb2406afb0 msgr2=0x7fdb24155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb22ffd640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb2406afb0 0x7fdb24155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb22ffd640 1 -- 172.21.15.59:0/4033140472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb240ce0d0 msgr2=0x7fdb2415e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb22ffd640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb240ce0d0 0x7fdb2415e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb22ffd640 1 -- 172.21.15.59:0/4033140472 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb24133d00 con 0x7fdb240b4420 2024-03-06T07:25:33.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb23fff640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb240ce0d0 0x7fdb2415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:33.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb22ffd640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb240b4420 0x7fdb2415de80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800e880 tx=0x7fdb1800ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb0bfff640 1 -- 172.21.15.59:0/4033140472 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb18017070 con 0x7fdb240b4420 2024-03-06T07:25:33.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb0bfff640 1 -- 172.21.15.59:0/4033140472 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1800c940 con 0x7fdb240b4420 2024-03-06T07:25:33.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb2a247640 1 -- 172.21.15.59:0/4033140472 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb24133fe0 con 0x7fdb240b4420 2024-03-06T07:25:33.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb0bfff640 1 -- 172.21.15.59:0/4033140472 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1800cd30 con 0x7fdb240b4420 2024-03-06T07:25:33.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.581+0000 7fdb2a247640 1 -- 172.21.15.59:0/4033140472 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb241345a0 con 0x7fdb240b4420 2024-03-06T07:25:33.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.582+0000 7fdb2a247640 1 -- 172.21.15.59:0/4033140472 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2404f330 con 0x7fdb240b4420 2024-03-06T07:25:33.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.582+0000 7fdb0bfff640 1 -- 172.21.15.59:0/4033140472 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb1801a070 con 0x7fdb240b4420 2024-03-06T07:25:33.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.583+0000 7fdb0bfff640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb1006b4d0 0x7fdb1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.583+0000 7fdb0bfff640 1 -- 172.21.15.59:0/4033140472 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb1808a490 con 0x7fdb240b4420 2024-03-06T07:25:33.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.583+0000 7fdb237fe640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb1006b4d0 0x7fdb1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.583+0000 7fdb237fe640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb1006b4d0 0x7fdb1006d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdb14007fd0 tx=0x7fdb14007e80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.588+0000 7fdb0bfff640 1 -- 172.21.15.59:0/4033140472 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb18058cc0 con 0x7fdb240b4420 2024-03-06T07:25:33.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.770+0000 7fdb2a247640 1 -- 172.21.15.59:0/4033140472 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb240b5cb0 con 0x7fdb240b4420 2024-03-06T07:25:33.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.772+0000 7fdb0bfff640 1 -- 172.21.15.59:0/4033140472 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16656 (secure 0 0 0) 0x7fdb1805cd60 con 0x7fdb240b4420 2024-03-06T07:25:33.774 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:33.774 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":70,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:06.152587+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:02.779485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:03.482047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.773+0000 7fdb2a247640 1 -- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb1006b4d0 msgr2=0x7fdb1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.773+0000 7fdb2a247640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb1006b4d0 0x7fdb1006d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdb14007fd0 tx=0x7fdb14007e80 comp rx=0 tx=0).stop 2024-03-06T07:25:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.774+0000 7fdb2a247640 1 -- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb240b4420 msgr2=0x7fdb2415de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.774+0000 7fdb2a247640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb240b4420 0x7fdb2415de80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800e880 tx=0x7fdb1800ed40 comp rx=0 tx=0).stop 2024-03-06T07:25:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.774+0000 7fdb2a247640 1 -- 172.21.15.59:0/4033140472 shutdown_connections 2024-03-06T07:25:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.774+0000 7fdb2a247640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb1006b4d0 0x7fdb1006d990 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.774+0000 7fdb2a247640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb240ce0d0 0x7fdb2415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.774+0000 7fdb2a247640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb240b4420 0x7fdb2415de80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.774+0000 7fdb2a247640 1 --2- 172.21.15.59:0/4033140472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb2406afb0 0x7fdb24155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.774+0000 7fdb2a247640 1 -- 172.21.15.59:0/4033140472 >> 172.21.15.59:0/4033140472 conn(0x7fdb2406c0c0 msgr2=0x7fdb24142c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.774+0000 7fdb2a247640 1 -- 172.21.15.59:0/4033140472 shutdown_connections 2024-03-06T07:25:33.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.774+0000 7fdb2a247640 1 -- 172.21.15.59:0/4033140472 wait complete. 2024-03-06T07:25:33.787 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:25:33.787 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:33.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.886+0000 7f66c668d640 1 Processor -- start 2024-03-06T07:25:33.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.886+0000 7f66c668d640 1 -- start start 2024-03-06T07:25:33.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.887+0000 7f66c668d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66c015fa10 0x7f66c00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.887+0000 7f66c668d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f66c00d16e0 0x7f66c00d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.887+0000 7f66c668d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c00cebe0 con 0x7f66c015bdd0 2024-03-06T07:25:33.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.887+0000 7f66c668d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c00ceec0 con 0x7f66c00d16e0 2024-03-06T07:25:33.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.887+0000 7f66c668d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c00cf1a0 con 0x7f66c015fa10 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.887+0000 7f66c4c2a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f66c00d16e0 0x7f66c00d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.887+0000 7f66c4c2a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f66c00d16e0 0x7f66c00d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44670/0 (socket says 172.21.15.59:44670) 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.887+0000 7f66c4c2a640 1 -- 172.21.15.59:0/1336316091 learned_addr learned my addr 172.21.15.59:0/1336316091 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.887+0000 7f66bf7fe640 1 --2- 172.21.15.59:0/1336316091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66c015fa10 0x7f66c00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.888+0000 7f66c4c2a640 1 -- 172.21.15.59:0/1336316091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66c015fa10 msgr2=0x7f66c00ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.888+0000 7f66c4c2a640 1 --2- 172.21.15.59:0/1336316091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66c015fa10 0x7f66c00ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.888+0000 7f66c4c2a640 1 -- 172.21.15.59:0/1336316091 >> v1:172.21.15.164:6789/0 conn(0x7f66c015bdd0 legacy=0x7f66c00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.888+0000 7f66c4c2a640 1 -- 172.21.15.59:0/1336316091 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c00cc8e0 con 0x7f66c00d16e0 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.888+0000 7f66c4c2a640 1 --2- 172.21.15.59:0/1336316091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66c00d16e0 0x7f66c00d1b30 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009e50 tx=0x7f66b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3826db901dd1dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.888+0000 7f66beffd640 1 -- 172.21.15.59:0/1336316091 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b802ca70 con 0x7f66c00d16e0 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.888+0000 7f66beffd640 1 -- 172.21.15.59:0/1336316091 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b802cc10 con 0x7f66c00d16e0 2024-03-06T07:25:33.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.889+0000 7f66beffd640 1 -- 172.21.15.59:0/1336316091 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b80343f0 con 0x7f66c00d16e0 2024-03-06T07:25:33.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.889+0000 7f66c668d640 1 -- 172.21.15.59:0/1336316091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66c00d16e0 msgr2=0x7f66c00d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.889+0000 7f66c668d640 1 --2- 172.21.15.59:0/1336316091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66c00d16e0 0x7f66c00d1b30 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009e50 tx=0x7f66b802bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.889+0000 7f66c668d640 1 -- 172.21.15.59:0/1336316091 shutdown_connections 2024-03-06T07:25:33.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.889+0000 7f66c668d640 1 --2- 172.21.15.59:0/1336316091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66c00d16e0 0x7f66c00d1b30 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.889+0000 7f66c668d640 1 --2- 172.21.15.59:0/1336316091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66c015fa10 0x7f66c00ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.889+0000 7f66c668d640 1 -- 172.21.15.59:0/1336316091 >> 172.21.15.59:0/1336316091 conn(0x7f66c009a9f0 msgr2=0x7f66c009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.889+0000 7f66c668d640 1 -- 172.21.15.59:0/1336316091 shutdown_connections 2024-03-06T07:25:33.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.889+0000 7f66c668d640 1 -- 172.21.15.59:0/1336316091 wait complete. 2024-03-06T07:25:33.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.890+0000 7f66c668d640 1 Processor -- start 2024-03-06T07:25:33.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.890+0000 7f66c668d640 1 -- start start 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66c668d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66c00d16e0 0x7f66c01544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66c668d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66c015bdd0 0x7f66c00b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66c668d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66c015fa10 0x7f66c01e7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66c668d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c015c1b0 con 0x7f66c00d16e0 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66c668d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c00cc5d0 con 0x7f66c015bdd0 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66c668d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c00ce7b0 con 0x7f66c015fa10 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66c00d16e0 0x7f66c01544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66bffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66c00d16e0 0x7f66c01544d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44678/0 (socket says 172.21.15.59:44678) 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66bffff640 1 -- 172.21.15.59:0/2540051819 learned_addr learned my addr 172.21.15.59:0/2540051819 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66bf7fe640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66c015bdd0 0x7f66c00b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66c4c2a640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66c015fa10 0x7f66c01e7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66bffff640 1 -- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66c015fa10 msgr2=0x7f66c01e7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66bffff640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66c015fa10 0x7f66c01e7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66bffff640 1 -- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66c015bdd0 msgr2=0x7f66c00b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66bffff640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66c015bdd0 0x7f66c00b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.891+0000 7f66bffff640 1 -- 172.21.15.59:0/2540051819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c006fd50 con 0x7f66c00d16e0 2024-03-06T07:25:33.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.892+0000 7f66bffff640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66c00d16e0 0x7f66c01544d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f66b000c400 tx=0x7f66b000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.892+0000 7f66bcff9640 1 -- 172.21.15.59:0/2540051819 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00052d0 con 0x7f66c00d16e0 2024-03-06T07:25:33.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.892+0000 7f66bcff9640 1 -- 172.21.15.59:0/2540051819 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b0005490 con 0x7f66c00d16e0 2024-03-06T07:25:33.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.892+0000 7f66c668d640 1 -- 172.21.15.59:0/2540051819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c006d420 con 0x7f66c00d16e0 2024-03-06T07:25:33.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.892+0000 7f66bcff9640 1 -- 172.21.15.59:0/2540051819 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0007260 con 0x7f66c00d16e0 2024-03-06T07:25:33.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.892+0000 7f66c668d640 1 -- 172.21.15.59:0/2540051819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c0154a10 con 0x7f66c00d16e0 2024-03-06T07:25:33.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.893+0000 7f66c668d640 1 -- 172.21.15.59:0/2540051819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c004f330 con 0x7f66c00d16e0 2024-03-06T07:25:33.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.893+0000 7f66bcff9640 1 -- 172.21.15.59:0/2540051819 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66b0007400 con 0x7f66c00d16e0 2024-03-06T07:25:33.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.894+0000 7f66bcff9640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66ac06b4d0 0x7f66ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:33.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.894+0000 7f66bcff9640 1 -- 172.21.15.59:0/2540051819 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66b00896d0 con 0x7f66c00d16e0 2024-03-06T07:25:33.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.894+0000 7f66bf7fe640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66ac06b4d0 0x7f66ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:33.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.894+0000 7f66bf7fe640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66ac06b4d0 0x7f66ac06d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f66c01556d0 tx=0x7f66b4009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:33.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:33.898+0000 7f66bcff9640 1 -- 172.21.15.59:0/2540051819 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66b0057f00 con 0x7f66c00d16e0 2024-03-06T07:25:34.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.115+0000 7f66c668d640 1 -- 172.21.15.59:0/2540051819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66c00b7a40 con 0x7f66c00d16e0 2024-03-06T07:25:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.115+0000 7f66bcff9640 1 -- 172.21.15.59:0/2540051819 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f66b005bfa0 con 0x7f66c00d16e0 2024-03-06T07:25:34.117 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:34.117 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:04.517350+0000","modified":"2024-03-06T07:25:06.518231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/250576205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":250576205},{"type":"v1","addr":"172.21.15.164:6837","nonce":250576205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:34.117 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 35 2024-03-06T07:25:34.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 -- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66ac06b4d0 msgr2=0x7f66ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66ac06b4d0 0x7f66ac06d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f66c01556d0 tx=0x7f66b4009190 comp rx=0 tx=0).stop 2024-03-06T07:25:34.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 -- 172.21.15.59:0/2540051819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66c00d16e0 msgr2=0x7f66c01544d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66c00d16e0 0x7f66c01544d0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f66b000c400 tx=0x7f66b000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 -- 172.21.15.59:0/2540051819 shutdown_connections 2024-03-06T07:25:34.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66ac06b4d0 0x7f66ac06d990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66c015fa10 0x7f66c01e7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66c015bdd0 0x7f66c00b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 --2- 172.21.15.59:0/2540051819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f66c00d16e0 0x7f66c01544d0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 -- 172.21.15.59:0/2540051819 >> 172.21.15.59:0/2540051819 conn(0x7f66c009a9f0 msgr2=0x7f66c00c5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 -- 172.21.15.59:0/2540051819 shutdown_connections 2024-03-06T07:25:34.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.117+0000 7f66c668d640 1 -- 172.21.15.59:0/2540051819 wait complete. 2024-03-06T07:25:34.130 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:34.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.229+0000 7fd798cb5640 1 Processor -- start 2024-03-06T07:25:34.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.229+0000 7fd798cb5640 1 -- start start 2024-03-06T07:25:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.230+0000 7fd798cb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940b6550 0x7fd7940cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.230+0000 7fd798cb5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7940cef50 con 0x7fd7940d06a0 2024-03-06T07:25:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.230+0000 7fd798cb5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79415c620 con 0x7fd7940d02c0 2024-03-06T07:25:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.230+0000 7fd798cb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7940b50b0 con 0x7fd7940b6550 2024-03-06T07:25:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.230+0000 7fd792d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd7940d06a0 0x7fd7940ce840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55288/0 (socket says 172.21.15.59:55288) 2024-03-06T07:25:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.230+0000 7fd792d9d640 1 -- 172.21.15.59:0/3529112580 learned_addr learned my addr 172.21.15.59:0/3529112580 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.230+0000 7fd791d9b640 1 --2- 172.21.15.59:0/3529112580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940b6550 0x7fd7940cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.231+0000 7fd79159a640 1 -- 172.21.15.59:0/3529112580 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122729886 0 0) 0x7fd7940cef50 con 0x7fd7940d06a0 2024-03-06T07:25:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.231+0000 7fd79159a640 1 -- 172.21.15.59:0/3529112580 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd77c003520 con 0x7fd7940d06a0 2024-03-06T07:25:34.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.231+0000 7fd79159a640 1 -- 172.21.15.59:0/3529112580 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762706909 0 0) 0x7fd79415c620 con 0x7fd7940d02c0 2024-03-06T07:25:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.231+0000 7fd79159a640 1 -- 172.21.15.59:0/3529112580 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7940cef50 con 0x7fd7940d02c0 2024-03-06T07:25:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.231+0000 7fd791d9b640 1 -- 172.21.15.59:0/3529112580 >> v1:172.21.15.164:6789/0 conn(0x7fd7940d02c0 legacy=0x7fd7940ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.231+0000 7fd791d9b640 1 -- 172.21.15.59:0/3529112580 >> v1:172.21.15.59:6789/0 conn(0x7fd7940d06a0 legacy=0x7fd7940ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.231+0000 7fd791d9b640 1 -- 172.21.15.59:0/3529112580 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7940cd800 con 0x7fd7940b6550 2024-03-06T07:25:34.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.231+0000 7fd791d9b640 1 --2- 172.21.15.59:0/3529112580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940b6550 0x7fd7940cf850 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd7880097d0 tx=0x7fd78802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f2af059bb6952b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.232+0000 7fd79159a640 1 -- 172.21.15.59:0/3529112580 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd788035020 con 0x7fd7940b6550 2024-03-06T07:25:34.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.232+0000 7fd79159a640 1 -- 172.21.15.59:0/3529112580 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd788031890 con 0x7fd7940b6550 2024-03-06T07:25:34.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.232+0000 7fd79159a640 1 -- 172.21.15.59:0/3529112580 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd788031c60 con 0x7fd7940b6550 2024-03-06T07:25:34.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.232+0000 7fd798cb5640 1 -- 172.21.15.59:0/3529112580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940b6550 msgr2=0x7fd7940cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.232+0000 7fd798cb5640 1 --2- 172.21.15.59:0/3529112580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940b6550 0x7fd7940cf850 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd7880097d0 tx=0x7fd78802b930 comp rx=0 tx=0).stop 2024-03-06T07:25:34.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.232+0000 7fd798cb5640 1 -- 172.21.15.59:0/3529112580 shutdown_connections 2024-03-06T07:25:34.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.232+0000 7fd798cb5640 1 --2- 172.21.15.59:0/3529112580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940b6550 0x7fd7940cf850 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.232+0000 7fd798cb5640 1 -- 172.21.15.59:0/3529112580 >> 172.21.15.59:0/3529112580 conn(0x7fd7940b35a0 msgr2=0x7fd7940b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.232+0000 7fd798cb5640 1 -- 172.21.15.59:0/3529112580 shutdown_connections 2024-03-06T07:25:34.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.232+0000 7fd798cb5640 1 -- 172.21.15.59:0/3529112580 wait complete. 2024-03-06T07:25:34.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.233+0000 7fd798cb5640 1 Processor -- start 2024-03-06T07:25:34.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd798cb5640 1 -- start start 2024-03-06T07:25:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd798cb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7940b6550 0x7fd7941f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd798cb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7940d02c0 0x7fd794158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd798cb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940d06a0 0x7fd7940caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd798cb5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7940cd0d0 con 0x7fd7940b6550 2024-03-06T07:25:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd798cb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7940b4d40 con 0x7fd7940d02c0 2024-03-06T07:25:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd798cb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7940cd250 con 0x7fd7940d06a0 2024-03-06T07:25:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd79259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7940b6550 0x7fd7941f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd79259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7940b6550 0x7fd7941f4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44692/0 (socket says 172.21.15.59:44692) 2024-03-06T07:25:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd79259c640 1 -- 172.21.15.59:0/3354872690 learned_addr learned my addr 172.21.15.59:0/3354872690 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd792d9d640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940d06a0 0x7fd7940caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.234+0000 7fd791d9b640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7940d02c0 0x7fd794158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd79259c640 1 -- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940d06a0 msgr2=0x7fd7940caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd79259c640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940d06a0 0x7fd7940caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd79259c640 1 -- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7940d02c0 msgr2=0x7fd794158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd79259c640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7940d02c0 0x7fd794158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd79259c640 1 -- 172.21.15.59:0/3354872690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7940cbde0 con 0x7fd7940b6550 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd792d9d640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940d06a0 0x7fd7940caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd79259c640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7940b6550 0x7fd7941f4dd0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fd78c00c500 tx=0x7fd78c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd782ffd640 1 -- 172.21.15.59:0/3354872690 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c017070 con 0x7fd7940b6550 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd782ffd640 1 -- 172.21.15.59:0/3354872690 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c005310 con 0x7fd7940b6550 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd798cb5640 1 -- 172.21.15.59:0/3354872690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7940c9480 con 0x7fd7940b6550 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd782ffd640 1 -- 172.21.15.59:0/3354872690 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c003830 con 0x7fd7940b6550 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.235+0000 7fd798cb5640 1 -- 172.21.15.59:0/3354872690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7940c9a40 con 0x7fd7940b6550 2024-03-06T07:25:34.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.236+0000 7fd798cb5640 1 -- 172.21.15.59:0/3354872690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79404f330 con 0x7fd7940b6550 2024-03-06T07:25:34.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.237+0000 7fd782ffd640 1 -- 172.21.15.59:0/3354872690 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd78c01a070 con 0x7fd7940b6550 2024-03-06T07:25:34.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.237+0000 7fd782ffd640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd77406b4d0 0x7fd77406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.237+0000 7fd782ffd640 1 -- 172.21.15.59:0/3354872690 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd78c0894c0 con 0x7fd7940b6550 2024-03-06T07:25:34.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.237+0000 7fd791d9b640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd77406b4d0 0x7fd77406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.238+0000 7fd791d9b640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd77406b4d0 0x7fd77406d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd788036000 tx=0x7fd788008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.241+0000 7fd782ffd640 1 -- 172.21.15.59:0/3354872690 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd78c057d00 con 0x7fd7940b6550 2024-03-06T07:25:34.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.460+0000 7fd798cb5640 1 -- 172.21.15.59:0/3354872690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7941595f0 con 0x7fd7940b6550 2024-03-06T07:25:34.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.461+0000 7fd782ffd640 1 -- 172.21.15.59:0/3354872690 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fd78c05bda0 con 0x7fd7940b6550 2024-03-06T07:25:34.463 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:34.463 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:04.517350+0000","modified":"2024-03-06T07:25:06.518231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/250576205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":250576205},{"type":"v1","addr":"172.21.15.164:6837","nonce":250576205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:34.463 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 35 2024-03-06T07:25:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 -- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd77406b4d0 msgr2=0x7fd77406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd77406b4d0 0x7fd77406d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd788036000 tx=0x7fd788008030 comp rx=0 tx=0).stop 2024-03-06T07:25:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 -- 172.21.15.59:0/3354872690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7940b6550 msgr2=0x7fd7941f4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7940b6550 0x7fd7941f4dd0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fd78c00c500 tx=0x7fd78c00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 -- 172.21.15.59:0/3354872690 shutdown_connections 2024-03-06T07:25:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd77406b4d0 0x7fd77406d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7940d06a0 0x7fd7940caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7940d02c0 0x7fd794158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 --2- 172.21.15.59:0/3354872690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7940b6550 0x7fd7941f4dd0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 -- 172.21.15.59:0/3354872690 >> 172.21.15.59:0/3354872690 conn(0x7fd7940b35a0 msgr2=0x7fd794070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 -- 172.21.15.59:0/3354872690 shutdown_connections 2024-03-06T07:25:34.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.463+0000 7fd798cb5640 1 -- 172.21.15.59:0/3354872690 wait complete. 2024-03-06T07:25:34.476 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:34.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.575+0000 7fc3a838a640 1 Processor -- start 2024-03-06T07:25:34.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.575+0000 7fc3a838a640 1 -- start start 2024-03-06T07:25:34.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a838a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc3a01621c0 0x7fc3a00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a838a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a00ceac0 con 0x7fc3a00d4920 2024-03-06T07:25:34.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a838a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a00d2dc0 con 0x7fc3a01625a0 2024-03-06T07:25:34.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a838a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a00cf690 con 0x7fc3a01621c0 2024-03-06T07:25:34.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a5925640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc3a00d4920 0x7fc3a00d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55296/0 (socket says 172.21.15.59:55296) 2024-03-06T07:25:34.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a5925640 1 -- 172.21.15.59:0/1839540249 learned_addr learned my addr 172.21.15.59:0/1839540249 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:34.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a6126640 1 --2- 172.21.15.59:0/1839540249 >> v2:172.21.15.164:3300/0 conn(0x7fc3a01621c0 0x7fc3a00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a5124640 1 -- 172.21.15.59:0/1839540249 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532467247 0 0) 0x7fc3a00ceac0 con 0x7fc3a00d4920 2024-03-06T07:25:34.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a5124640 1 -- 172.21.15.59:0/1839540249 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc38c003520 con 0x7fc3a00d4920 2024-03-06T07:25:34.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a6126640 1 -- 172.21.15.59:0/1839540249 >> v1:172.21.15.164:6789/0 conn(0x7fc3a01625a0 legacy=0x7fc3a015dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a5124640 1 -- 172.21.15.59:0/1839540249 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852690384 0 0) 0x7fc3a00d2dc0 con 0x7fc3a01625a0 2024-03-06T07:25:34.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a6126640 1 -- 172.21.15.59:0/1839540249 >> v1:172.21.15.59:6789/0 conn(0x7fc3a00d4920 legacy=0x7fc3a00d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.576+0000 7fc3a6126640 1 -- 172.21.15.59:0/1839540249 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a00ced60 con 0x7fc3a01621c0 2024-03-06T07:25:34.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.577+0000 7fc3a6126640 1 --2- 172.21.15.59:0/1839540249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3a01621c0 0x7fc3a00cfb20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc39c0097a0 tx=0x7fc39c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d14bee2c5ad7058a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.577+0000 7fc3a5124640 1 -- 172.21.15.59:0/1839540249 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c035020 con 0x7fc3a01621c0 2024-03-06T07:25:34.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.577+0000 7fc3a5124640 1 -- 172.21.15.59:0/1839540249 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c031830 con 0x7fc3a01621c0 2024-03-06T07:25:34.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.577+0000 7fc3a5124640 1 -- 172.21.15.59:0/1839540249 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c031be0 con 0x7fc3a01621c0 2024-03-06T07:25:34.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.578+0000 7fc3a838a640 1 -- 172.21.15.59:0/1839540249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3a01621c0 msgr2=0x7fc3a00cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.578+0000 7fc3a838a640 1 --2- 172.21.15.59:0/1839540249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3a01621c0 0x7fc3a00cfb20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc39c0097a0 tx=0x7fc39c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.578+0000 7fc3a838a640 1 -- 172.21.15.59:0/1839540249 shutdown_connections 2024-03-06T07:25:34.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.578+0000 7fc3a838a640 1 --2- 172.21.15.59:0/1839540249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3a01621c0 0x7fc3a00cfb20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.578+0000 7fc3a838a640 1 -- 172.21.15.59:0/1839540249 >> 172.21.15.59:0/1839540249 conn(0x7fc3a00d3d10 msgr2=0x7fc3a00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.578+0000 7fc3a838a640 1 -- 172.21.15.59:0/1839540249 shutdown_connections 2024-03-06T07:25:34.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.578+0000 7fc3a838a640 1 -- 172.21.15.59:0/1839540249 wait complete. 2024-03-06T07:25:34.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.579+0000 7fc3a838a640 1 Processor -- start 2024-03-06T07:25:34.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.579+0000 7fc3a838a640 1 -- start start 2024-03-06T07:25:34.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a838a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3a00d4920 0x7fc3a00cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a838a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a01621c0 0x7fc3a01572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a838a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3a01625a0 0x7fc3a015a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a838a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a015e450 con 0x7fc3a01621c0 2024-03-06T07:25:34.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a838a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00cf320 con 0x7fc3a00d4920 2024-03-06T07:25:34.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a838a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a015e5d0 con 0x7fc3a01625a0 2024-03-06T07:25:34.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a5925640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a01621c0 0x7fc3a01572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a5925640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a01621c0 0x7fc3a01572e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44708/0 (socket says 172.21.15.59:44708) 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a6126640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3a00d4920 0x7fc3a00cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a5925640 1 -- 172.21.15.59:0/434429108 learned_addr learned my addr 172.21.15.59:0/434429108 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a6927640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3a01625a0 0x7fc3a015a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a5925640 1 -- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3a01625a0 msgr2=0x7fc3a015a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a5925640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3a01625a0 0x7fc3a015a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a5925640 1 -- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3a00d4920 msgr2=0x7fc3a00cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a5925640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3a00d4920 0x7fc3a00cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a5925640 1 -- 172.21.15.59:0/434429108 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a00b8d20 con 0x7fc3a01621c0 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.580+0000 7fc3a5925640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a01621c0 0x7fc3a01572e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc39000c430 tx=0x7fc39000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.581+0000 7fc396ffd640 1 -- 172.21.15.59:0/434429108 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc390017070 con 0x7fc3a01621c0 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.581+0000 7fc396ffd640 1 -- 172.21.15.59:0/434429108 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc390005ce0 con 0x7fc3a01621c0 2024-03-06T07:25:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.581+0000 7fc3a838a640 1 -- 172.21.15.59:0/434429108 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a00b8f90 con 0x7fc3a01621c0 2024-03-06T07:25:34.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.581+0000 7fc396ffd640 1 -- 172.21.15.59:0/434429108 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3900072f0 con 0x7fc3a01621c0 2024-03-06T07:25:34.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.581+0000 7fc3a838a640 1 -- 172.21.15.59:0/434429108 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a00b9550 con 0x7fc3a01621c0 2024-03-06T07:25:34.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.582+0000 7fc3a838a640 1 -- 172.21.15.59:0/434429108 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a004f330 con 0x7fc3a01621c0 2024-03-06T07:25:34.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.582+0000 7fc396ffd640 1 -- 172.21.15.59:0/434429108 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc39001a070 con 0x7fc3a01621c0 2024-03-06T07:25:34.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.583+0000 7fc396ffd640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc38406b4d0 0x7fc38406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.583+0000 7fc396ffd640 1 -- 172.21.15.59:0/434429108 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3900895f0 con 0x7fc3a01621c0 2024-03-06T07:25:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.583+0000 7fc3a6126640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc38406b4d0 0x7fc38406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.583+0000 7fc3a6126640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc38406b4d0 0x7fc38406d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc39c036000 tx=0x7fc39c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.587+0000 7fc396ffd640 1 -- 172.21.15.59:0/434429108 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc390057e30 con 0x7fc3a01621c0 2024-03-06T07:25:34.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.769+0000 7fc3a838a640 1 -- 172.21.15.59:0/434429108 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3a0073380 con 0x7fc3a01621c0 2024-03-06T07:25:34.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.770+0000 7fc396ffd640 1 -- 172.21.15.59:0/434429108 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16656 (secure 0 0 0) 0x7fc39005bed0 con 0x7fc3a01621c0 2024-03-06T07:25:34.772 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:34.772 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":70,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:06.152587+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:02.779485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:03.482047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:34.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.772+0000 7fc3a838a640 1 -- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc38406b4d0 msgr2=0x7fc38406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.772+0000 7fc3a838a640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc38406b4d0 0x7fc38406d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc39c036000 tx=0x7fc39c008030 comp rx=0 tx=0).stop 2024-03-06T07:25:34.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.772+0000 7fc3a838a640 1 -- 172.21.15.59:0/434429108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a01621c0 msgr2=0x7fc3a01572e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.772+0000 7fc3a838a640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a01621c0 0x7fc3a01572e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc39000c430 tx=0x7fc39000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.773+0000 7fc3a838a640 1 -- 172.21.15.59:0/434429108 shutdown_connections 2024-03-06T07:25:34.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.773+0000 7fc3a838a640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc38406b4d0 0x7fc38406d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.773+0000 7fc3a838a640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3a01625a0 0x7fc3a015a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.773+0000 7fc3a838a640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3a01621c0 0x7fc3a01572e0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.773+0000 7fc3a838a640 1 --2- 172.21.15.59:0/434429108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3a00d4920 0x7fc3a00cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.773+0000 7fc3a838a640 1 -- 172.21.15.59:0/434429108 >> 172.21.15.59:0/434429108 conn(0x7fc3a00d3d10 msgr2=0x7fc3a015f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.773+0000 7fc3a838a640 1 -- 172.21.15.59:0/434429108 shutdown_connections 2024-03-06T07:25:34.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.773+0000 7fc3a838a640 1 -- 172.21.15.59:0/434429108 wait complete. 2024-03-06T07:25:34.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:25:34.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.886+0000 7f01f1d93640 1 Processor -- start 2024-03-06T07:25:34.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.886+0000 7f01f1d93640 1 -- start start 2024-03-06T07:25:34.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01f1d93640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f01ec1456f0 0x7f01ec070940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01f1d93640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01ec070e80 0x7f01ec0b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01f1d93640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec0b3680 con 0x7f01ec15c2c0 2024-03-06T07:25:34.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01f1d93640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec15c760 con 0x7f01ec070e80 2024-03-06T07:25:34.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01f1d93640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec15ca40 con 0x7f01ec1456f0 2024-03-06T07:25:34.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01ebfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01ec070e80 0x7f01ec0b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01ebfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01ec070e80 0x7f01ec0b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44724/0 (socket says 172.21.15.59:44724) 2024-03-06T07:25:34.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01ebfff640 1 -- 172.21.15.59:0/875169628 learned_addr learned my addr 172.21.15.59:0/875169628 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:34.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01eb7fe640 1 --2- 172.21.15.59:0/875169628 >> v2:172.21.15.164:3300/0 conn(0x7f01ec1456f0 0x7f01ec070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01ebfff640 1 -- 172.21.15.59:0/875169628 >> v2:172.21.15.164:3300/0 conn(0x7f01ec1456f0 msgr2=0x7f01ec070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01ebfff640 1 --2- 172.21.15.59:0/875169628 >> v2:172.21.15.164:3300/0 conn(0x7f01ec1456f0 0x7f01ec070940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01ebfff640 1 -- 172.21.15.59:0/875169628 >> v1:172.21.15.164:6789/0 conn(0x7f01ec15c2c0 legacy=0x7f01ec0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01ebfff640 1 -- 172.21.15.59:0/875169628 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec15cd20 con 0x7f01ec070e80 2024-03-06T07:25:34.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.887+0000 7f01ea7fc640 1 -- 172.21.15.59:0/875169628 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197330085 0 0) 0x7f01e0001110 con 0x7f01ec15c2c0 2024-03-06T07:25:34.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.888+0000 7f01ebfff640 1 --2- 172.21.15.59:0/875169628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ec070e80 0x7f01ec0b3b40 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f01ec06ea30 tx=0x7f01e402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9371ffddf4e8487 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.888+0000 7f01ea7fc640 1 -- 172.21.15.59:0/875169628 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e402ca80 con 0x7f01ec070e80 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.888+0000 7f01ea7fc640 1 -- 172.21.15.59:0/875169628 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e402cc20 con 0x7f01ec070e80 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.888+0000 7f01ea7fc640 1 -- 172.21.15.59:0/875169628 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40343f0 con 0x7f01ec070e80 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.889+0000 7f01f1d93640 1 -- 172.21.15.59:0/875169628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ec070e80 msgr2=0x7f01ec0b3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.889+0000 7f01f1d93640 1 --2- 172.21.15.59:0/875169628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ec070e80 0x7f01ec0b3b40 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f01ec06ea30 tx=0x7f01e402bee0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.889+0000 7f01f1d93640 1 -- 172.21.15.59:0/875169628 shutdown_connections 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.889+0000 7f01f1d93640 1 --2- 172.21.15.59:0/875169628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ec070e80 0x7f01ec0b3b40 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.889+0000 7f01f1d93640 1 --2- 172.21.15.59:0/875169628 >> v2:172.21.15.164:3300/0 conn(0x7f01ec1456f0 0x7f01ec070940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.889+0000 7f01f1d93640 1 -- 172.21.15.59:0/875169628 >> 172.21.15.59:0/875169628 conn(0x7f01ec0b1c80 msgr2=0x7f01ec0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.889+0000 7f01f1d93640 1 -- 172.21.15.59:0/875169628 shutdown_connections 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.889+0000 7f01f1d93640 1 -- 172.21.15.59:0/875169628 wait complete. 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.890+0000 7f01f1d93640 1 Processor -- start 2024-03-06T07:25:34.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.890+0000 7f01f1d93640 1 -- start start 2024-03-06T07:25:34.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01f1d93640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01ec070e80 0x7f01ec1f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01f1d93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ec1456f0 0x7f01ec0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01f1d93640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01ec15c2c0 0x7f01ec141ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01f1d93640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec15d980 con 0x7f01ec1456f0 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01f1d93640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec15d800 con 0x7f01ec15c2c0 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01f1d93640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0cca00 con 0x7f01ec070e80 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01eaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ec1456f0 0x7f01ec0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01eb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01ec070e80 0x7f01ec1f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01eaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ec1456f0 0x7f01ec0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44734/0 (socket says 172.21.15.59:44734) 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01eaffd640 1 -- 172.21.15.59:0/2018204014 learned_addr learned my addr 172.21.15.59:0/2018204014 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01ebfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01ec15c2c0 0x7f01ec141ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01eaffd640 1 -- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01ec070e80 msgr2=0x7f01ec1f0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01eaffd640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01ec070e80 0x7f01ec1f0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01eaffd640 1 -- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01ec15c2c0 msgr2=0x7f01ec141ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01eaffd640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01ec15c2c0 0x7f01ec141ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01eaffd640 1 -- 172.21.15.59:0/2018204014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec0ce210 con 0x7f01ec1456f0 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01eaffd640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ec1456f0 0x7f01ec0cd7c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f01e000c440 tx=0x7f01e000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.891+0000 7f01d3fff640 1 -- 172.21.15.59:0/2018204014 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0017070 con 0x7f01ec1456f0 2024-03-06T07:25:34.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.892+0000 7f01d3fff640 1 -- 172.21.15.59:0/2018204014 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e00035f0 con 0x7f01ec1456f0 2024-03-06T07:25:34.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.892+0000 7f01f1d93640 1 -- 172.21.15.59:0/2018204014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec0ce4f0 con 0x7f01ec1456f0 2024-03-06T07:25:34.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.892+0000 7f01d3fff640 1 -- 172.21.15.59:0/2018204014 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e00039e0 con 0x7f01ec1456f0 2024-03-06T07:25:34.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.892+0000 7f01f1d93640 1 -- 172.21.15.59:0/2018204014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec1f1450 con 0x7f01ec1456f0 2024-03-06T07:25:34.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.892+0000 7f01f1d93640 1 -- 172.21.15.59:0/2018204014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ec04f330 con 0x7f01ec1456f0 2024-03-06T07:25:34.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.893+0000 7f01d3fff640 1 -- 172.21.15.59:0/2018204014 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01e001a070 con 0x7f01ec1456f0 2024-03-06T07:25:34.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.893+0000 7f01d3fff640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f01d806b4d0 0x7f01d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.893+0000 7f01d3fff640 1 -- 172.21.15.59:0/2018204014 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01e0089550 con 0x7f01ec1456f0 2024-03-06T07:25:34.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.894+0000 7f01eb7fe640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f01d806b4d0 0x7f01d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.894+0000 7f01eb7fe640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f01d806b4d0 0x7f01d806d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009b40 tx=0x7f01dc009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:34.898+0000 7f01d3fff640 1 -- 172.21.15.59:0/2018204014 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01ec0b3680 con 0x7f01ec1456f0 2024-03-06T07:25:35.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.116+0000 7f01f1d93640 1 -- 172.21.15.59:0/2018204014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f01ec0748c0 con 0x7f01ec1456f0 2024-03-06T07:25:35.119 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:25:35.116+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.117+0000 7f01d3fff640 1 -- 172.21.15.59:0/2018204014 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) v1 ==== 161+0+0 (secure 0 0 0) 0x7f01e0057d90 con 0x7f01ec1456f0 2024-03-06T07:25:35.119 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:25:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.120+0000 7f01f1d93640 1 -- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f01d806b4d0 msgr2=0x7f01d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.120+0000 7f01f1d93640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f01d806b4d0 0x7f01d806d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009b40 tx=0x7f01dc009190 comp rx=0 tx=0).stop 2024-03-06T07:25:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.120+0000 7f01f1d93640 1 -- 172.21.15.59:0/2018204014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ec1456f0 msgr2=0x7f01ec0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.120+0000 7f01f1d93640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ec1456f0 0x7f01ec0cd7c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f01e000c440 tx=0x7f01e000c900 comp rx=0 tx=0).stop 2024-03-06T07:25:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.120+0000 7f01f1d93640 1 -- 172.21.15.59:0/2018204014 shutdown_connections 2024-03-06T07:25:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.120+0000 7f01f1d93640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f01d806b4d0 0x7f01d806d990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.120+0000 7f01f1d93640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01ec15c2c0 0x7f01ec141ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.120+0000 7f01f1d93640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01ec1456f0 0x7f01ec0cd7c0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.120+0000 7f01f1d93640 1 --2- 172.21.15.59:0/2018204014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01ec070e80 0x7f01ec1f0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.120+0000 7f01f1d93640 1 -- 172.21.15.59:0/2018204014 >> 172.21.15.59:0/2018204014 conn(0x7f01ec0b1c80 msgr2=0x7f01ec160290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.120+0000 7f01f1d93640 1 -- 172.21.15.59:0/2018204014 shutdown_connections 2024-03-06T07:25:35.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.121+0000 7f01f1d93640 1 -- 172.21.15.59:0/2018204014 wait complete. 2024-03-06T07:25:35.133 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:25:35.167 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:25:35.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.236+0000 7efcbaf35640 1 Processor -- start 2024-03-06T07:25:35.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.236+0000 7efcbaf35640 1 -- start start 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcbaf35640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efcb40b25d0 0x7efcb41669e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcbaf35640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb40d0240 con 0x7efcb415bdd0 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcbaf35640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb40cf210 con 0x7efcb40b41b0 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcbaf35640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb40cf4f0 con 0x7efcb40b25d0 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcb94d2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efcb40b25d0 0x7efcb41669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcb8cd1640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7efcb415bdd0 0x7efcb40ce300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55308/0 (socket says 172.21.15.59:55308) 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcb94d2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efcb40b25d0 0x7efcb41669e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44746/0 (socket says 172.21.15.59:44746) 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcb8cd1640 1 -- 172.21.15.59:0/2041355873 learned_addr learned my addr 172.21.15.59:0/2041355873 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcb94d2640 1 -- 172.21.15.59:0/2041355873 >> v1:172.21.15.164:6789/0 conn(0x7efcb40b41b0 legacy=0x7efcb40cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcb94d2640 1 -- 172.21.15.59:0/2041355873 >> v1:172.21.15.59:6789/0 conn(0x7efcb415bdd0 legacy=0x7efcb40ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcb94d2640 1 -- 172.21.15.59:0/2041355873 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb40d04e0 con 0x7efcb40b25d0 2024-03-06T07:25:35.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.237+0000 7efcb37fe640 1 -- 172.21.15.59:0/2041355873 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810905138 0 0) 0x7efcb40d0240 con 0x7efcb415bdd0 2024-03-06T07:25:35.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.238+0000 7efcb94d2640 1 --2- 172.21.15.59:0/2041355873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcb40b25d0 0x7efcb41669e0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7efcac009d90 tx=0x7efcac02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30efb232cdaea19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.238+0000 7efcb37fe640 1 -- 172.21.15.59:0/2041355873 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac039070 con 0x7efcb40b25d0 2024-03-06T07:25:35.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.238+0000 7efcb37fe640 1 -- 172.21.15.59:0/2041355873 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac02cd00 con 0x7efcb40b25d0 2024-03-06T07:25:35.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.238+0000 7efcb37fe640 1 -- 172.21.15.59:0/2041355873 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac034660 con 0x7efcb40b25d0 2024-03-06T07:25:35.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.238+0000 7efcbaf35640 1 -- 172.21.15.59:0/2041355873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcb40b25d0 msgr2=0x7efcb41669e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.238+0000 7efcbaf35640 1 --2- 172.21.15.59:0/2041355873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcb40b25d0 0x7efcb41669e0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7efcac009d90 tx=0x7efcac02bf50 comp rx=0 tx=0).stop 2024-03-06T07:25:35.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.238+0000 7efcbaf35640 1 -- 172.21.15.59:0/2041355873 shutdown_connections 2024-03-06T07:25:35.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.238+0000 7efcbaf35640 1 --2- 172.21.15.59:0/2041355873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcb40b25d0 0x7efcb41669e0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.238+0000 7efcbaf35640 1 -- 172.21.15.59:0/2041355873 >> 172.21.15.59:0/2041355873 conn(0x7efcb40b3690 msgr2=0x7efcb40b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.239+0000 7efcbaf35640 1 -- 172.21.15.59:0/2041355873 shutdown_connections 2024-03-06T07:25:35.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.239+0000 7efcbaf35640 1 -- 172.21.15.59:0/2041355873 wait complete. 2024-03-06T07:25:35.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.240+0000 7efcbaf35640 1 Processor -- start 2024-03-06T07:25:35.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.240+0000 7efcbaf35640 1 -- start start 2024-03-06T07:25:35.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.240+0000 7efcbaf35640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcb40b25d0 0x7efcb40b7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.240+0000 7efcbaf35640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcb40b41b0 0x7efcb41510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.240+0000 7efcbaf35640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcb415bdd0 0x7efcb40c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcbaf35640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40b10c0 con 0x7efcb415bdd0 2024-03-06T07:25:35.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcbaf35640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40d0850 con 0x7efcb40b25d0 2024-03-06T07:25:35.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcbaf35640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40b1240 con 0x7efcb40b41b0 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb94d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcb415bdd0 0x7efcb40c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb94d2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcb415bdd0 0x7efcb40c68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44758/0 (socket says 172.21.15.59:44758) 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb94d2640 1 -- 172.21.15.59:0/2078297067 learned_addr learned my addr 172.21.15.59:0/2078297067 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb8cd1640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcb40b25d0 0x7efcb40b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb3fff640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcb40b41b0 0x7efcb41510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb94d2640 1 -- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcb40b41b0 msgr2=0x7efcb41510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb94d2640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcb40b41b0 0x7efcb41510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb94d2640 1 -- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcb40b25d0 msgr2=0x7efcb40b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb94d2640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcb40b25d0 0x7efcb40b7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb94d2640 1 -- 172.21.15.59:0/2078297067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb40c6e20 con 0x7efcb415bdd0 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb3fff640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcb40b41b0 0x7efcb41510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb94d2640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcb415bdd0 0x7efcb40c68e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7efcac037040 tx=0x7efcac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb17fa640 1 -- 172.21.15.59:0/2078297067 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac044020 con 0x7efcb415bdd0 2024-03-06T07:25:35.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb17fa640 1 -- 172.21.15.59:0/2078297067 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac03a040 con 0x7efcb415bdd0 2024-03-06T07:25:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcbaf35640 1 -- 172.21.15.59:0/2078297067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb40c70e0 con 0x7efcb415bdd0 2024-03-06T07:25:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcb17fa640 1 -- 172.21.15.59:0/2078297067 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac039050 con 0x7efcb415bdd0 2024-03-06T07:25:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.241+0000 7efcbaf35640 1 -- 172.21.15.59:0/2078297067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb40c76a0 con 0x7efcb415bdd0 2024-03-06T07:25:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.242+0000 7efcbaf35640 1 -- 172.21.15.59:0/2078297067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb409fcf0 con 0x7efcb415bdd0 2024-03-06T07:25:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.243+0000 7efcb17fa640 1 -- 172.21.15.59:0/2078297067 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcac02cb80 con 0x7efcb415bdd0 2024-03-06T07:25:35.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.243+0000 7efcb17fa640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc9806b4d0 0x7efc9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.243+0000 7efcb17fa640 1 -- 172.21.15.59:0/2078297067 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efcac0aca30 con 0x7efcb415bdd0 2024-03-06T07:25:35.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.243+0000 7efcb8cd1640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc9806b4d0 0x7efc9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.244+0000 7efcb8cd1640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc9806b4d0 0x7efc9806d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7efca8000f60 tx=0x7efca8007b90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.248+0000 7efcb17fa640 1 -- 172.21.15.59:0/2078297067 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcac04e060 con 0x7efcb415bdd0 2024-03-06T07:25:35.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:35.465+0000 7efcbaf35640 1 -- 172.21.15.59:0/2078297067 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efcb4068ff0 con 0x7efcb415bdd0 2024-03-06T07:25:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.122+0000 7efcb17fa640 1 -- 172.21.15.59:0/2078297067 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7efcac081070 con 0x7efcb415bdd0 2024-03-06T07:25:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.124+0000 7efcbaf35640 1 -- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc9806b4d0 msgr2=0x7efc9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.124+0000 7efcbaf35640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc9806b4d0 0x7efc9806d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7efca8000f60 tx=0x7efca8007b90 comp rx=0 tx=0).stop 2024-03-06T07:25:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.124+0000 7efcbaf35640 1 -- 172.21.15.59:0/2078297067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcb415bdd0 msgr2=0x7efcb40c68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.124+0000 7efcbaf35640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcb415bdd0 0x7efcb40c68e0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7efcac037040 tx=0x7efcac004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.124+0000 7efcbaf35640 1 -- 172.21.15.59:0/2078297067 shutdown_connections 2024-03-06T07:25:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.124+0000 7efcbaf35640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efc9806b4d0 0x7efc9806d990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.124+0000 7efcbaf35640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcb415bdd0 0x7efcb40c68e0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.124+0000 7efcbaf35640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcb40b41b0 0x7efcb41510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.124+0000 7efcbaf35640 1 --2- 172.21.15.59:0/2078297067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcb40b25d0 0x7efcb40b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.124+0000 7efcbaf35640 1 -- 172.21.15.59:0/2078297067 >> 172.21.15.59:0/2078297067 conn(0x7efcb40b3690 msgr2=0x7efcb40b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.125+0000 7efcbaf35640 1 -- 172.21.15.59:0/2078297067 shutdown_connections 2024-03-06T07:25:36.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.125+0000 7efcbaf35640 1 -- 172.21.15.59:0/2078297067 wait complete. 2024-03-06T07:25:36.140 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:25:36.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.250+0000 7f36db4b5640 1 Processor -- start 2024-03-06T07:25:36.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.250+0000 7f36db4b5640 1 -- start start 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.250+0000 7f36db4b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d415e180 0x7f36d40cd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.250+0000 7f36db4b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f36d40cfe00 0x7f36d40ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.250+0000 7f36db4b5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d415f7c0 con 0x7f36d415f290 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36db4b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d40ccc60 con 0x7f36d40cfe00 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36db4b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40ccf40 con 0x7f36d415e180 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d8a50640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f36d415f290 0x7f36d40cf6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55310/0 (socket says 172.21.15.59:55310) 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d8a50640 1 -- 172.21.15.59:0/2604835520 learned_addr learned my addr 172.21.15.59:0/2604835520 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d9251640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d415e180 0x7f36d40cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d9a52640 1 --2- 172.21.15.59:0/2604835520 >> v2:172.21.15.164:3300/0 conn(0x7f36d40cfe00 0x7f36d40ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959711826 0 0) 0x7f36d415f7c0 con 0x7f36d415f290 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc003520 con 0x7f36d415f290 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2626159863 0 0) 0x7f36bc003520 con 0x7f36d415f290 2024-03-06T07:25:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d415e180 msgr2=0x7f36d40cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d415e180 0x7f36d40cd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 >> v2:172.21.15.164:3300/0 conn(0x7f36d40cfe00 msgr2=0x7f36d40ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 --2- 172.21.15.59:0/2604835520 >> v2:172.21.15.164:3300/0 conn(0x7f36d40cfe00 0x7f36d40ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d40cd220 con 0x7f36d415f290 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f36c8001670 con 0x7f36d415f290 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 >> v1:172.21.15.59:6789/0 conn(0x7f36d415f290 legacy=0x7f36d40cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d415e180 0x7f36bc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36bc003f30 0x7f36bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36bc004d40 0x7f36bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40ccc60 con 0x7f36bc004d40 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40ccf40 con 0x7f36bc003f30 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36bc00ba10 con 0x7f36d415e180 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d9251640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d415e180 0x7f36bc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d9a52640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36bc004d40 0x7f36bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.251+0000 7f36d8a50640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36bc003f30 0x7f36bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.252+0000 7f36d9a52640 1 -- 172.21.15.59:0/2604835520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d415e180 msgr2=0x7f36bc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.252+0000 7f36d9a52640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d415e180 0x7f36bc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.252+0000 7f36d9a52640 1 -- 172.21.15.59:0/2604835520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36bc003f30 msgr2=0x7f36bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.252+0000 7f36d9a52640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36bc003f30 0x7f36bc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.252+0000 7f36d9a52640 1 -- 172.21.15.59:0/2604835520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36bc003520 con 0x7f36bc004d40 2024-03-06T07:25:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.252+0000 7f36d9a52640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36bc004d40 0x7f36bc0051c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009af0 tx=0x7f36bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8cbd219b771dadb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.253+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc039070 con 0x7f36bc004d40 2024-03-06T07:25:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.253+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc02c710 con 0x7f36bc004d40 2024-03-06T07:25:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.253+0000 7f36d3fff640 1 -- 172.21.15.59:0/2604835520 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc009690 con 0x7f36bc004d40 2024-03-06T07:25:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.253+0000 7f36db4b5640 1 -- 172.21.15.59:0/2604835520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36bc004d40 msgr2=0x7f36bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.253+0000 7f36db4b5640 1 --2- 172.21.15.59:0/2604835520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36bc004d40 0x7f36bc0051c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009af0 tx=0x7f36bc005830 comp rx=0 tx=0).stop 2024-03-06T07:25:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.253+0000 7f36d9a52640 1 -- 172.21.15.59:0/2604835520 reap_dead start 2024-03-06T07:25:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.253+0000 7f36db4b5640 1 -- 172.21.15.59:0/2604835520 shutdown_connections 2024-03-06T07:25:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.253+0000 7f36db4b5640 1 -- 172.21.15.59:0/2604835520 >> 172.21.15.59:0/2604835520 conn(0x7f36d40d1b20 msgr2=0x7f36d40d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.253+0000 7f36db4b5640 1 -- 172.21.15.59:0/2604835520 shutdown_connections 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.253+0000 7f36db4b5640 1 -- 172.21.15.59:0/2604835520 wait complete. 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.254+0000 7f36db4b5640 1 Processor -- start 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.254+0000 7f36db4b5640 1 -- start start 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.255+0000 7f36db4b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36bc004d40 0x7f36d415bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.255+0000 7f36db4b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36d40b6ec0 0x7f36d415fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.255+0000 7f36db4b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36d41601c0 0x7f36d414d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.255+0000 7f36db4b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d415e560 con 0x7f36d40b6ec0 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.255+0000 7f36db4b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40b2ee0 con 0x7f36d41601c0 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.255+0000 7f36db4b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d415e6e0 con 0x7f36bc004d40 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.255+0000 7f36d8a50640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36d40b6ec0 0x7f36d415fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.255+0000 7f36d8a50640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36d40b6ec0 0x7f36d415fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44774/0 (socket says 172.21.15.59:44774) 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.255+0000 7f36d8a50640 1 -- 172.21.15.59:0/2336142135 learned_addr learned my addr 172.21.15.59:0/2336142135 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.255+0000 7f36d9251640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36bc004d40 0x7f36d415bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.255+0000 7f36d9a52640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36d41601c0 0x7f36d414d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36d8a50640 1 -- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36bc004d40 msgr2=0x7f36d415bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36d8a50640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36bc004d40 0x7f36d415bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36d8a50640 1 -- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36d41601c0 msgr2=0x7f36d414d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36d8a50640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36d41601c0 0x7f36d414d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36d8a50640 1 -- 172.21.15.59:0/2336142135 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d40b5b40 con 0x7f36d40b6ec0 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36d9a52640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36d41601c0 0x7f36d414d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36d8a50640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36d40b6ec0 0x7f36d415fc80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f36c8007310 tx=0x7f36c80077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36d1ffb640 1 -- 172.21.15.59:0/2336142135 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c800f9c0 con 0x7f36d40b6ec0 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36d1ffb640 1 -- 172.21.15.59:0/2336142135 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c8018070 con 0x7f36d40b6ec0 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36db4b5640 1 -- 172.21.15.59:0/2336142135 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d40b5e20 con 0x7f36d40b6ec0 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36d1ffb640 1 -- 172.21.15.59:0/2336142135 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c800fd80 con 0x7f36d40b6ec0 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.256+0000 7f36db4b5640 1 -- 172.21.15.59:0/2336142135 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d40b63e0 con 0x7f36d40b6ec0 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.257+0000 7f36db4b5640 1 -- 172.21.15.59:0/2336142135 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d404f330 con 0x7f36d40b6ec0 2024-03-06T07:25:36.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.257+0000 7f36d1ffb640 1 -- 172.21.15.59:0/2336142135 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36c801d460 con 0x7f36d40b6ec0 2024-03-06T07:25:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.258+0000 7f36d1ffb640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36c006b4d0 0x7f36c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.258+0000 7f36d1ffb640 1 -- 172.21.15.59:0/2336142135 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f36c808a3e0 con 0x7f36d40b6ec0 2024-03-06T07:25:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.258+0000 7f36d9251640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36c006b4d0 0x7f36c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.258+0000 7f36d9251640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36c006b4d0 0x7f36c006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f36c4007520 tx=0x7f36c400b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.262+0000 7f36d1ffb640 1 -- 172.21.15.59:0/2336142135 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36c8059200 con 0x7f36d40b6ec0 2024-03-06T07:25:36.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.442+0000 7f36db4b5640 1 -- 172.21.15.59:0/2336142135 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f36d4072d80 con 0x7f36d40b6ec0 2024-03-06T07:25:36.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.686+0000 7f36d1ffb640 1 -- 172.21.15.59:0/2336142135 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v72) v1 ==== 181+0+0 (secure 0 0 0) 0x7f36c8058b00 con 0x7f36d40b6ec0 2024-03-06T07:25:36.688 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:25:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 -- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36c006b4d0 msgr2=0x7f36c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36c006b4d0 0x7f36c006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f36c4007520 tx=0x7f36c400b040 comp rx=0 tx=0).stop 2024-03-06T07:25:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 -- 172.21.15.59:0/2336142135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36d40b6ec0 msgr2=0x7f36d415fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36d40b6ec0 0x7f36d415fc80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f36c8007310 tx=0x7f36c80077d0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 -- 172.21.15.59:0/2336142135 shutdown_connections 2024-03-06T07:25:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36c006b4d0 0x7f36c006d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36d41601c0 0x7f36d414d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36d40b6ec0 0x7f36d415fc80 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 --2- 172.21.15.59:0/2336142135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36bc004d40 0x7f36d415bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 -- 172.21.15.59:0/2336142135 >> 172.21.15.59:0/2336142135 conn(0x7f36d40d1b20 msgr2=0x7f36d415d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 -- 172.21.15.59:0/2336142135 shutdown_connections 2024-03-06T07:25:36.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.689+0000 7f36db4b5640 1 -- 172.21.15.59:0/2336142135 wait complete. 2024-03-06T07:25:36.704 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:25:36.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.807+0000 7f8a2bced640 1 Processor -- start 2024-03-06T07:25:36.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.808+0000 7f8a2bced640 1 -- start start 2024-03-06T07:25:36.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.808+0000 7f8a2bced640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24161a90 0x7f8a24161e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.808+0000 7f8a2bced640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8a240d2720 0x7f8a240cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.808+0000 7f8a2bced640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a241623b0 con 0x7f8a2406be00 2024-03-06T07:25:36.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.808+0000 7f8a2bced640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a240d00c0 con 0x7f8a240d2720 2024-03-06T07:25:36.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.808+0000 7f8a2bced640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240cf790 con 0x7f8a24161a90 2024-03-06T07:25:36.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a2a28a640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8a2406be00 0x7f8a240d0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55316/0 (socket says 172.21.15.59:55316) 2024-03-06T07:25:36.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a2a28a640 1 -- 172.21.15.59:0/1840104245 learned_addr learned my addr 172.21.15.59:0/1840104245 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:36.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a29a89640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24161a90 0x7f8a24161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a29288640 1 --2- 172.21.15.59:0/1840104245 >> v2:172.21.15.164:3300/0 conn(0x7f8a240d2720 0x7f8a240cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920837803 0 0) 0x7f8a241623b0 con 0x7f8a2406be00 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a10003520 con 0x7f8a2406be00 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2647166779 0 0) 0x7f8a10003520 con 0x7f8a2406be00 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24161a90 msgr2=0x7f8a24161e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24161a90 0x7f8a24161e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 >> v2:172.21.15.164:3300/0 conn(0x7f8a240d2720 msgr2=0x7f8a240cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 --2- 172.21.15.59:0/1840104245 >> v2:172.21.15.164:3300/0 conn(0x7f8a240d2720 0x7f8a240cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a240cfa70 con 0x7f8a2406be00 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f8a1c0019f0 con 0x7f8a2406be00 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 >> v1:172.21.15.59:6789/0 conn(0x7f8a2406be00 legacy=0x7f8a240d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a100039b0 0x7f8a10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24161a90 0x7f8a10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 0x7f8a100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240d00c0 con 0x7f8a10004d40 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240cf790 con 0x7f8a100039b0 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1000b9c0 con 0x7f8a24161a90 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a2a28a640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 0x7f8a100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a29a89640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a100039b0 0x7f8a10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.809+0000 7f8a29288640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24161a90 0x7f8a10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.810+0000 7f8a2a28a640 1 -- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24161a90 msgr2=0x7f8a10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.810+0000 7f8a2a28a640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24161a90 0x7f8a10004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.810+0000 7f8a2a28a640 1 -- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a100039b0 msgr2=0x7f8a10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.810+0000 7f8a2a28a640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a100039b0 0x7f8a10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.810+0000 7f8a2a28a640 1 -- 172.21.15.59:0/1840104245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a10003520 con 0x7f8a10004d40 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.810+0000 7f8a29288640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24161a90 0x7f8a10004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.810+0000 7f8a2a28a640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 0x7f8a100051c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0013d0 tx=0x7f8a10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbdc45e405893ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.811+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c00a4e0 con 0x7f8a10004d40 2024-03-06T07:25:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.811+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c034650 con 0x7f8a10004d40 2024-03-06T07:25:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.811+0000 7f8a28a87640 1 -- 172.21.15.59:0/1840104245 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c034a40 con 0x7f8a10004d40 2024-03-06T07:25:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.811+0000 7f8a2bced640 1 -- 172.21.15.59:0/1840104245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 msgr2=0x7f8a100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.811+0000 7f8a2bced640 1 --2- 172.21.15.59:0/1840104245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 0x7f8a100051c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0013d0 tx=0x7f8a10005830 comp rx=0 tx=0).stop 2024-03-06T07:25:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.811+0000 7f8a2a28a640 1 -- 172.21.15.59:0/1840104245 reap_dead start 2024-03-06T07:25:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.811+0000 7f8a2bced640 1 -- 172.21.15.59:0/1840104245 shutdown_connections 2024-03-06T07:25:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.811+0000 7f8a2bced640 1 -- 172.21.15.59:0/1840104245 >> 172.21.15.59:0/1840104245 conn(0x7f8a240d1b20 msgr2=0x7f8a240d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.811+0000 7f8a2bced640 1 -- 172.21.15.59:0/1840104245 shutdown_connections 2024-03-06T07:25:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.811+0000 7f8a2bced640 1 -- 172.21.15.59:0/1840104245 wait complete. 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.812+0000 7f8a2bced640 1 Processor -- start 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.812+0000 7f8a2bced640 1 -- start start 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a2bced640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 0x7f8a2415fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a2bced640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24146b70 0x7f8a24146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a2bced640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a24156c10 0x7f8a24157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a2bced640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240cdd10 con 0x7f8a10004d40 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a2bced640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a24161790 con 0x7f8a24156c10 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a2bced640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240cd9b0 con 0x7f8a24146b70 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a29a89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 0x7f8a2415fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a29a89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 0x7f8a2415fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44790/0 (socket says 172.21.15.59:44790) 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a29a89640 1 -- 172.21.15.59:0/2967911373 learned_addr learned my addr 172.21.15.59:0/2967911373 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a29288640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24146b70 0x7f8a24146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a2a28a640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a24156c10 0x7f8a24157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a29a89640 1 -- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24146b70 msgr2=0x7f8a24146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a29a89640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24146b70 0x7f8a24146fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a29a89640 1 -- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a24156c10 msgr2=0x7f8a24157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a29a89640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a24156c10 0x7f8a24157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.813+0000 7f8a29a89640 1 -- 172.21.15.59:0/2967911373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a241579e0 con 0x7f8a10004d40 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.814+0000 7f8a29a89640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 0x7f8a2415fa10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8a200118e0 tx=0x7f8a20011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.814+0000 7f8a1a7fc640 1 -- 172.21.15.59:0/2967911373 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2000faf0 con 0x7f8a10004d40 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.814+0000 7f8a1a7fc640 1 -- 172.21.15.59:0/2967911373 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2000fc90 con 0x7f8a10004d40 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.814+0000 7f8a2bced640 1 -- 172.21.15.59:0/2967911373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a24157cc0 con 0x7f8a10004d40 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.814+0000 7f8a1a7fc640 1 -- 172.21.15.59:0/2967911373 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20014440 con 0x7f8a10004d40 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.814+0000 7f8a2bced640 1 -- 172.21.15.59:0/2967911373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a24158280 con 0x7f8a10004d40 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.815+0000 7f8a2bced640 1 -- 172.21.15.59:0/2967911373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2404f330 con 0x7f8a10004d40 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.815+0000 7f8a1a7fc640 1 -- 172.21.15.59:0/2967911373 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a20012040 con 0x7f8a10004d40 2024-03-06T07:25:36.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.816+0000 7f8a1a7fc640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a0806b4d0 0x7f8a0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.816+0000 7f8a1a7fc640 1 -- 172.21.15.59:0/2967911373 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f8a2008a780 con 0x7f8a10004d40 2024-03-06T07:25:36.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.816+0000 7f8a29288640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a0806b4d0 0x7f8a0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.816+0000 7f8a29288640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a0806b4d0 0x7f8a0806d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8a241565a0 tx=0x7f8a14008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:36.820+0000 7f8a1a7fc640 1 -- 172.21.15.59:0/2967911373 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a20019030 con 0x7f8a10004d40 2024-03-06T07:25:37.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.000+0000 7f8a2bced640 1 -- 172.21.15.59:0/2967911373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8a240b7aa0 con 0x7f8a10004d40 2024-03-06T07:25:37.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.690+0000 7f8a1a7fc640 1 -- 172.21.15.59:0/2967911373 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v73) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8a20059160 con 0x7f8a10004d40 2024-03-06T07:25:37.692 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-03-06T07:25:37.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 -- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a0806b4d0 msgr2=0x7f8a0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a0806b4d0 0x7f8a0806d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8a241565a0 tx=0x7f8a14008040 comp rx=0 tx=0).stop 2024-03-06T07:25:37.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 -- 172.21.15.59:0/2967911373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 msgr2=0x7f8a2415fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 0x7f8a2415fa10 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8a200118e0 tx=0x7f8a20011da0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 -- 172.21.15.59:0/2967911373 shutdown_connections 2024-03-06T07:25:37.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8a0806b4d0 0x7f8a0806d990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a24156c10 0x7f8a24157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a24146b70 0x7f8a24146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 --2- 172.21.15.59:0/2967911373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a10004d40 0x7f8a2415fa10 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 -- 172.21.15.59:0/2967911373 >> 172.21.15.59:0/2967911373 conn(0x7f8a240d1b20 msgr2=0x7f8a2406cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 -- 172.21.15.59:0/2967911373 shutdown_connections 2024-03-06T07:25:37.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.692+0000 7f8a2bced640 1 -- 172.21.15.59:0/2967911373 wait complete. 2024-03-06T07:25:37.708 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:25:37.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.811+0000 7f6eaa24e640 1 Processor -- start 2024-03-06T07:25:37.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.811+0000 7f6eaa24e640 1 -- start start 2024-03-06T07:25:37.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.812+0000 7f6eaa24e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6ea415c050 0x7f6ea406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.812+0000 7f6eaa24e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ea40d0af0 0x7f6ea40cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.812+0000 7f6eaa24e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea40cd4b0 con 0x7f6ea406fb40 2024-03-06T07:25:37.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.812+0000 7f6eaa24e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea406ffb0 con 0x7f6ea40d0af0 2024-03-06T07:25:37.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.812+0000 7f6eaa24e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea40cd790 con 0x7f6ea415c050 2024-03-06T07:25:37.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.812+0000 7f6ea3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ea40d0af0 0x7f6ea40cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.812+0000 7f6ea3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ea40d0af0 0x7f6ea40cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44798/0 (socket says 172.21.15.59:44798) 2024-03-06T07:25:37.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.812+0000 7f6ea2ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6ea406fb40 0x7f6ea40cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55326/0 (socket says 172.21.15.59:55326) 2024-03-06T07:25:37.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.812+0000 7f6ea3fff640 1 -- 172.21.15.59:0/1866031257 learned_addr learned my addr 172.21.15.59:0/1866031257 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:37.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.813+0000 7f6ea37fe640 1 --2- 172.21.15.59:0/1866031257 >> v2:172.21.15.164:3300/0 conn(0x7f6ea415c050 0x7f6ea406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.813+0000 7f6ea3fff640 1 -- 172.21.15.59:0/1866031257 >> v2:172.21.15.164:3300/0 conn(0x7f6ea415c050 msgr2=0x7f6ea406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.813+0000 7f6ea3fff640 1 --2- 172.21.15.59:0/1866031257 >> v2:172.21.15.164:3300/0 conn(0x7f6ea415c050 0x7f6ea406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.813+0000 7f6ea3fff640 1 -- 172.21.15.59:0/1866031257 >> v1:172.21.15.59:6789/0 conn(0x7f6ea406fb40 legacy=0x7f6ea40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.813+0000 7f6ea3fff640 1 -- 172.21.15.59:0/1866031257 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea40cda70 con 0x7f6ea40d0af0 2024-03-06T07:25:37.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.813+0000 7f6ea27fc640 1 -- 172.21.15.59:0/1866031257 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037370349 0 0) 0x7f6ea40cd4b0 con 0x7f6ea406fb40 2024-03-06T07:25:37.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.813+0000 7f6ea3fff640 1 --2- 172.21.15.59:0/1866031257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ea40d0af0 0x7f6ea40cce70 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4070ef0 tx=0x7f6e9c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86b58b13d6e48c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.814+0000 7f6ea27fc640 1 -- 172.21.15.59:0/1866031257 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c02ca80 con 0x7f6ea40d0af0 2024-03-06T07:25:37.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.814+0000 7f6ea27fc640 1 -- 172.21.15.59:0/1866031257 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c02cc20 con 0x7f6ea40d0af0 2024-03-06T07:25:37.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.814+0000 7f6ea27fc640 1 -- 172.21.15.59:0/1866031257 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0343f0 con 0x7f6ea40d0af0 2024-03-06T07:25:37.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.814+0000 7f6eaa24e640 1 -- 172.21.15.59:0/1866031257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ea40d0af0 msgr2=0x7f6ea40cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.814+0000 7f6eaa24e640 1 --2- 172.21.15.59:0/1866031257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ea40d0af0 0x7f6ea40cce70 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4070ef0 tx=0x7f6e9c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.814+0000 7f6eaa24e640 1 -- 172.21.15.59:0/1866031257 shutdown_connections 2024-03-06T07:25:37.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.814+0000 7f6eaa24e640 1 --2- 172.21.15.59:0/1866031257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ea40d0af0 0x7f6ea40cce70 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.814+0000 7f6eaa24e640 1 --2- 172.21.15.59:0/1866031257 >> v2:172.21.15.164:3300/0 conn(0x7f6ea415c050 0x7f6ea406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.814+0000 7f6eaa24e640 1 -- 172.21.15.59:0/1866031257 >> 172.21.15.59:0/1866031257 conn(0x7f6ea40b3800 msgr2=0x7f6ea40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.814+0000 7f6eaa24e640 1 -- 172.21.15.59:0/1866031257 shutdown_connections 2024-03-06T07:25:37.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.814+0000 7f6eaa24e640 1 -- 172.21.15.59:0/1866031257 wait complete. 2024-03-06T07:25:37.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.815+0000 7f6eaa24e640 1 Processor -- start 2024-03-06T07:25:37.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.815+0000 7f6eaa24e640 1 -- start start 2024-03-06T07:25:37.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.816+0000 7f6eaa24e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ea406fb40 0x7f6ea4142440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.816+0000 7f6eaa24e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ea40d0af0 0x7f6ea406d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.816+0000 7f6eaa24e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea415c050 0x7f6ea40cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.816+0000 7f6eaa24e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4165c30 con 0x7f6ea40d0af0 2024-03-06T07:25:37.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.816+0000 7f6eaa24e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4165ab0 con 0x7f6ea415c050 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.816+0000 7f6eaa24e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4165db0 con 0x7f6ea406fb40 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.816+0000 7f6ea2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ea40d0af0 0x7f6ea406d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.816+0000 7f6ea2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ea40d0af0 0x7f6ea406d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44812/0 (socket says 172.21.15.59:44812) 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.816+0000 7f6ea2ffd640 1 -- 172.21.15.59:0/835317859 learned_addr learned my addr 172.21.15.59:0/835317859 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6ea3fff640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea415c050 0x7f6ea40cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6ea37fe640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ea406fb40 0x7f6ea4142440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6ea2ffd640 1 -- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ea406fb40 msgr2=0x7f6ea4142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6ea2ffd640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ea406fb40 0x7f6ea4142440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6ea2ffd640 1 -- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea415c050 msgr2=0x7f6ea40cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6ea2ffd640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea415c050 0x7f6ea40cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6ea2ffd640 1 -- 172.21.15.59:0/835317859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea415abb0 con 0x7f6ea40d0af0 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6ea3fff640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea415c050 0x7f6ea40cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6ea37fe640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ea406fb40 0x7f6ea4142440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6ea2ffd640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ea40d0af0 0x7f6ea406d130 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800c460 tx=0x7f6e9800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6e8bfff640 1 -- 172.21.15.59:0/835317859 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98018070 con 0x7f6ea40d0af0 2024-03-06T07:25:37.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6e8bfff640 1 -- 172.21.15.59:0/835317859 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e98007500 con 0x7f6ea40d0af0 2024-03-06T07:25:37.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6e8bfff640 1 -- 172.21.15.59:0/835317859 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98004a20 con 0x7f6ea40d0af0 2024-03-06T07:25:37.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6eaa24e640 1 -- 172.21.15.59:0/835317859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea415ae90 con 0x7f6ea40d0af0 2024-03-06T07:25:37.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.817+0000 7f6eaa24e640 1 -- 172.21.15.59:0/835317859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea415b450 con 0x7f6ea40d0af0 2024-03-06T07:25:37.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.818+0000 7f6eaa24e640 1 -- 172.21.15.59:0/835317859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea40b4650 con 0x7f6ea40d0af0 2024-03-06T07:25:37.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.818+0000 7f6e8bfff640 1 -- 172.21.15.59:0/835317859 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e98004bc0 con 0x7f6ea40d0af0 2024-03-06T07:25:37.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.819+0000 7f6e8bfff640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6e9006b4d0 0x7f6e9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.819+0000 7f6e8bfff640 1 -- 172.21.15.59:0/835317859 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6e98089210 con 0x7f6ea40d0af0 2024-03-06T07:25:37.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.819+0000 7f6ea37fe640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6e9006b4d0 0x7f6e9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.820+0000 7f6ea37fe640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6e9006b4d0 0x7f6e9006d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6e94006fd0 tx=0x7f6e94008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:37.823+0000 7f6e8bfff640 1 -- 172.21.15.59:0/835317859 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ea40b4650 con 0x7f6ea40d0af0 2024-03-06T07:25:38.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.013+0000 7f6eaa24e640 1 -- 172.21.15.59:0/835317859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6ea4068ff0 con 0x7f6ea40d0af0 2024-03-06T07:25:38.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.016+0000 7f6e8bfff640 1 -- 172.21.15.59:0/835317859 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6e98058390 con 0x7f6ea40d0af0 2024-03-06T07:25:38.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 -- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6e9006b4d0 msgr2=0x7f6e9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6e9006b4d0 0x7f6e9006d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6e94006fd0 tx=0x7f6e94008040 comp rx=0 tx=0).stop 2024-03-06T07:25:38.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 -- 172.21.15.59:0/835317859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ea40d0af0 msgr2=0x7f6ea406d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ea40d0af0 0x7f6ea406d130 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800c460 tx=0x7f6e9800c920 comp rx=0 tx=0).stop 2024-03-06T07:25:38.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 -- 172.21.15.59:0/835317859 shutdown_connections 2024-03-06T07:25:38.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6e9006b4d0 0x7f6e9006d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea415c050 0x7f6ea40cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ea40d0af0 0x7f6ea406d130 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 --2- 172.21.15.59:0/835317859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ea406fb40 0x7f6ea4142440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 -- 172.21.15.59:0/835317859 >> 172.21.15.59:0/835317859 conn(0x7f6ea40b3800 msgr2=0x7f6ea4070250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 -- 172.21.15.59:0/835317859 shutdown_connections 2024-03-06T07:25:38.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.018+0000 7f6eaa24e640 1 -- 172.21.15.59:0/835317859 wait complete. 2024-03-06T07:25:38.031 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:25:38.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.132+0000 7f4112834640 1 Processor -- start 2024-03-06T07:25:38.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.132+0000 7f4112834640 1 -- start start 2024-03-06T07:25:38.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.133+0000 7f4112834640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f410c0cff40 0x7f410c0b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.133+0000 7f4112834640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c158ac0 con 0x7f410c1586e0 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.133+0000 7f4112834640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c15b2a0 con 0x7f410c0cfa90 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.133+0000 7f4112834640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410c15b540 con 0x7f410c0cff40 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.133+0000 7f410bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f410c1586e0 0x7f410c0aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55332/0 (socket says 172.21.15.59:55332) 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.133+0000 7f410bfff640 1 -- 172.21.15.59:0/1680117736 learned_addr learned my addr 172.21.15.59:0/1680117736 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.133+0000 7f4110dd1640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f410c0cff40 0x7f410c0b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.133+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437895950 0 0) 0x7f410c158ac0 con 0x7f410c1586e0 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.133+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f4003520 con 0x7f410c1586e0 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2976068873 0 0) 0x7f40f4003520 con 0x7f410c1586e0 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f410c0cff40 msgr2=0x7f410c0b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f410c0cff40 0x7f410c0b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 >> v1:172.21.15.164:6789/0 conn(0x7f410c0cfa90 legacy=0x7f410c0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c15b820 con 0x7f410c1586e0 2024-03-06T07:25:38.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f4100001670 con 0x7f410c1586e0 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 >> v1:172.21.15.59:6789/0 conn(0x7f410c1586e0 legacy=0x7f410c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f40039e0 0x7f40f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f410c0cff40 0x7f40f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40f4004d40 0x7f40f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c15b540 con 0x7f40f40039e0 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f400b9c0 con 0x7f40f4004d40 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f400bbc0 con 0x7f410c0cff40 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410bfff640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f40039e0 0x7f40f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410b7fe640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f410c0cff40 0x7f40f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f4110dd1640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40f4004d40 0x7f40f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410bfff640 1 -- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f410c0cff40 msgr2=0x7f40f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410bfff640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f410c0cff40 0x7f40f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410bfff640 1 -- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40f4004d40 msgr2=0x7f40f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410bfff640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40f4004d40 0x7f40f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f410bfff640 1 -- 172.21.15.59:0/1680117736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4003520 con 0x7f40f40039e0 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.134+0000 7f4110dd1640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40f4004d40 0x7f40f40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.135+0000 7f410bfff640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f40039e0 0x7f40f4003dc0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f410000c240 tx=0x7f40f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf0106a55cdfcde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.135+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100007d10 con 0x7f40f40039e0 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.135+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4100030070 con 0x7f40f40039e0 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.135+0000 7f410affd640 1 -- 172.21.15.59:0/1680117736 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100005d50 con 0x7f40f40039e0 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.136+0000 7f4112834640 1 -- 172.21.15.59:0/1680117736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f40039e0 msgr2=0x7f40f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.136+0000 7f4112834640 1 --2- 172.21.15.59:0/1680117736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f40039e0 0x7f40f4003dc0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f410000c240 tx=0x7f40f4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.136+0000 7f4110dd1640 1 -- 172.21.15.59:0/1680117736 reap_dead start 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.136+0000 7f4112834640 1 -- 172.21.15.59:0/1680117736 shutdown_connections 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.136+0000 7f4112834640 1 -- 172.21.15.59:0/1680117736 >> 172.21.15.59:0/1680117736 conn(0x7f410c06b190 msgr2=0x7f410c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.136+0000 7f4112834640 1 -- 172.21.15.59:0/1680117736 shutdown_connections 2024-03-06T07:25:38.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.136+0000 7f4112834640 1 -- 172.21.15.59:0/1680117736 wait complete. 2024-03-06T07:25:38.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.137+0000 7f4112834640 1 Processor -- start 2024-03-06T07:25:38.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.137+0000 7f4112834640 1 -- start start 2024-03-06T07:25:38.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f4112834640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40f40039e0 0x7f410c0adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f4112834640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f410c14e960 0x7f410c14be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f4112834640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f410c14c380 0x7f410c0ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f4112834640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c0d0360 con 0x7f410c14e960 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f4112834640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410c15bde0 con 0x7f410c14c380 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f4112834640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410c0af150 con 0x7f40f40039e0 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f410b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f410c14e960 0x7f410c14be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f410b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f410c14e960 0x7f410c14be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44834/0 (socket says 172.21.15.59:44834) 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f410b7fe640 1 -- 172.21.15.59:0/2032877872 learned_addr learned my addr 172.21.15.59:0/2032877872 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f410bfff640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40f40039e0 0x7f410c0adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f4110dd1640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f410c14c380 0x7f410c0ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f410b7fe640 1 -- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40f40039e0 msgr2=0x7f410c0adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f410b7fe640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40f40039e0 0x7f410c0adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f410b7fe640 1 -- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f410c14c380 msgr2=0x7f410c0ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f410b7fe640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f410c14c380 0x7f410c0ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f410b7fe640 1 -- 172.21.15.59:0/2032877872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c0b1030 con 0x7f410c14e960 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f410b7fe640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f410c14e960 0x7f410c14be40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00f830 tx=0x7f40fc00fcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f4108ff9640 1 -- 172.21.15.59:0/2032877872 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc0052e0 con 0x7f410c14e960 2024-03-06T07:25:38.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f4108ff9640 1 -- 172.21.15.59:0/2032877872 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc018070 con 0x7f410c14e960 2024-03-06T07:25:38.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.138+0000 7f4112834640 1 -- 172.21.15.59:0/2032877872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410c0cdce0 con 0x7f410c14e960 2024-03-06T07:25:38.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.139+0000 7f4108ff9640 1 -- 172.21.15.59:0/2032877872 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc00d910 con 0x7f410c14e960 2024-03-06T07:25:38.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.139+0000 7f4112834640 1 -- 172.21.15.59:0/2032877872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410c0ce2a0 con 0x7f410c14e960 2024-03-06T07:25:38.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.140+0000 7f4112834640 1 -- 172.21.15.59:0/2032877872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410c0732b0 con 0x7f410c14e960 2024-03-06T07:25:38.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.140+0000 7f4108ff9640 1 -- 172.21.15.59:0/2032877872 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40fc00dab0 con 0x7f410c14e960 2024-03-06T07:25:38.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.140+0000 7f4108ff9640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40f806b4d0 0x7f40f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.140+0000 7f4108ff9640 1 -- 172.21.15.59:0/2032877872 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f40fc089d60 con 0x7f410c14e960 2024-03-06T07:25:38.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.141+0000 7f410bfff640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40f806b4d0 0x7f40f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.141+0000 7f410bfff640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40f806b4d0 0x7f40f806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4100034000 tx=0x7f4100007000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.145+0000 7f4108ff9640 1 -- 172.21.15.59:0/2032877872 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40fc058890 con 0x7f410c14e960 2024-03-06T07:25:38.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.327+0000 7f4112834640 1 -- 172.21.15.59:0/2032877872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f410c068ff0 con 0x7f410c14e960 2024-03-06T07:25:38.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.330+0000 7f4108ff9640 1 -- 172.21.15.59:0/2032877872 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v25) v1 ==== 108+0+0 (secure 0 0 0) 0x7f40fc05c930 con 0x7f410c14e960 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 -- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40f806b4d0 msgr2=0x7f40f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40f806b4d0 0x7f40f806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4100034000 tx=0x7f4100007000 comp rx=0 tx=0).stop 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 -- 172.21.15.59:0/2032877872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f410c14e960 msgr2=0x7f410c14be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f410c14e960 0x7f410c14be40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00f830 tx=0x7f40fc00fcf0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 -- 172.21.15.59:0/2032877872 shutdown_connections 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f40f806b4d0 0x7f40f806d990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f410c14c380 0x7f410c0ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f410c14e960 0x7f410c14be40 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 --2- 172.21.15.59:0/2032877872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40f40039e0 0x7f410c0adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 -- 172.21.15.59:0/2032877872 >> 172.21.15.59:0/2032877872 conn(0x7f410c06b190 msgr2=0x7f410c0b3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 -- 172.21.15.59:0/2032877872 shutdown_connections 2024-03-06T07:25:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.332+0000 7f4112834640 1 -- 172.21.15.59:0/2032877872 wait complete. 2024-03-06T07:25:38.345 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:25:38.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.445+0000 7fa82dac3640 1 Processor -- start 2024-03-06T07:25:38.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.445+0000 7fa82dac3640 1 -- start start 2024-03-06T07:25:38.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.445+0000 7fa82dac3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa8281515e0 0x7fa8280cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.445+0000 7fa82dac3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8280cd7a0 0x7fa8280cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.445+0000 7fa82dac3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa828071f40 con 0x7fa82815b430 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.445+0000 7fa82dac3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82815b8e0 con 0x7fa8281515e0 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.445+0000 7fa82dac3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa828070a70 con 0x7fa8280cd7a0 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa826ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa82815b430 0x7fa828071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55342/0 (socket says 172.21.15.59:55342) 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa826ffd640 1 -- 172.21.15.59:0/4042010977 learned_addr learned my addr 172.21.15.59:0/4042010977 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa827fff640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8280cd7a0 0x7fa8280cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8277fe640 1 --2- 172.21.15.59:0/4042010977 >> v2:172.21.15.164:3300/0 conn(0x7fa8281515e0 0x7fa8280cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236276749 0 0) 0x7fa828071f40 con 0x7fa82815b430 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa810003520 con 0x7fa82815b430 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 86402602 0 0) 0x7fa810003520 con 0x7fa82815b430 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8280cd7a0 msgr2=0x7fa8280cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8280cd7a0 0x7fa8280cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 >> v2:172.21.15.164:3300/0 conn(0x7fa8281515e0 msgr2=0x7fa8280cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 --2- 172.21.15.59:0/4042010977 >> v2:172.21.15.164:3300/0 conn(0x7fa8281515e0 0x7fa8280cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa828070d50 con 0x7fa82815b430 2024-03-06T07:25:38.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fa81c001690 con 0x7fa82815b430 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 >> v1:172.21.15.59:6789/0 conn(0x7fa82815b430 legacy=0x7fa828071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8280cd7a0 0x7fa8100039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa810003f60 0x7fa810004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa810004d40 0x7fa8100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82815b8e0 con 0x7fa810004d40 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa828070a70 con 0x7fa810003f60 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81000b9c0 con 0x7fa8280cd7a0 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa8277fe640 1 --2- 172.21.15.59:0/4042010977 >> v2:172.21.15.164:3300/0 conn(0x7fa8281515e0 0x7fa8280cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa827fff640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa810004d40 0x7fa8100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.446+0000 7fa826ffd640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa810003f60 0x7fa810004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.447+0000 7fa8277fe640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8280cd7a0 0x7fa8100039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.447+0000 7fa827fff640 1 -- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8280cd7a0 msgr2=0x7fa8100039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.447+0000 7fa827fff640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8280cd7a0 0x7fa8100039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.447+0000 7fa827fff640 1 -- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa810003f60 msgr2=0x7fa810004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.447+0000 7fa827fff640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa810003f60 0x7fa810004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.447+0000 7fa827fff640 1 -- 172.21.15.59:0/4042010977 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa810003520 con 0x7fa810004d40 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.447+0000 7fa826ffd640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa810003f60 0x7fa810004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.447+0000 7fa827fff640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa810004d40 0x7fa8100051c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa820009fc0 tx=0x7fa810005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3f385d9a1f774f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.448+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820039070 con 0x7fa810004d40 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.448+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82002cc20 con 0x7fa810004d40 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.448+0000 7fa8267fc640 1 -- 172.21.15.59:0/4042010977 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820009c10 con 0x7fa810004d40 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.448+0000 7fa82dac3640 1 -- 172.21.15.59:0/4042010977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa810004d40 msgr2=0x7fa8100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.448+0000 7fa82dac3640 1 --2- 172.21.15.59:0/4042010977 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa810004d40 0x7fa8100051c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa820009fc0 tx=0x7fa810005830 comp rx=0 tx=0).stop 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.448+0000 7fa827fff640 1 -- 172.21.15.59:0/4042010977 reap_dead start 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.448+0000 7fa82dac3640 1 -- 172.21.15.59:0/4042010977 shutdown_connections 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.448+0000 7fa82dac3640 1 -- 172.21.15.59:0/4042010977 >> 172.21.15.59:0/4042010977 conn(0x7fa8280cf8a0 msgr2=0x7fa8280cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.448+0000 7fa82dac3640 1 -- 172.21.15.59:0/4042010977 shutdown_connections 2024-03-06T07:25:38.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.448+0000 7fa82dac3640 1 -- 172.21.15.59:0/4042010977 wait complete. 2024-03-06T07:25:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.449+0000 7fa82dac3640 1 Processor -- start 2024-03-06T07:25:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.449+0000 7fa82dac3640 1 -- start start 2024-03-06T07:25:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.450+0000 7fa82dac3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa828148350 0x7fa8281440e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.450+0000 7fa82dac3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa828148730 0x7fa828144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.450+0000 7fa82dac3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa828151a20 0x7fa828151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.450+0000 7fa82dac3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8280cdbb0 con 0x7fa828148350 2024-03-06T07:25:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.450+0000 7fa82dac3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa828071310 con 0x7fa828148730 2024-03-06T07:25:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.450+0000 7fa82dac3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8280cdd30 con 0x7fa828151a20 2024-03-06T07:25:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.450+0000 7fa8277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa828148350 0x7fa8281440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.450+0000 7fa8277fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa828148350 0x7fa8281440e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44850/0 (socket says 172.21.15.59:44850) 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.450+0000 7fa8277fe640 1 -- 172.21.15.59:0/445927882 learned_addr learned my addr 172.21.15.59:0/445927882 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.450+0000 7fa827fff640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa828151a20 0x7fa828151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.450+0000 7fa826ffd640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa828148730 0x7fa828144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.451+0000 7fa8277fe640 1 -- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa828151a20 msgr2=0x7fa828151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.451+0000 7fa8277fe640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa828151a20 0x7fa828151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.451+0000 7fa8277fe640 1 -- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa828148730 msgr2=0x7fa828144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.451+0000 7fa8277fe640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa828148730 0x7fa828144620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.451+0000 7fa8277fe640 1 -- 172.21.15.59:0/445927882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8281532c0 con 0x7fa828148350 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.451+0000 7fa8277fe640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa828148350 0x7fa8281440e0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fa818011930 tx=0x7fa818011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.451+0000 7fa80ffff640 1 -- 172.21.15.59:0/445927882 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81800fbd0 con 0x7fa828148350 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.451+0000 7fa80ffff640 1 -- 172.21.15.59:0/445927882 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81800fd70 con 0x7fa828148350 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.451+0000 7fa82dac3640 1 -- 172.21.15.59:0/445927882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa828144c50 con 0x7fa828148350 2024-03-06T07:25:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.451+0000 7fa80ffff640 1 -- 172.21.15.59:0/445927882 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818015660 con 0x7fa828148350 2024-03-06T07:25:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.451+0000 7fa82dac3640 1 -- 172.21.15.59:0/445927882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa828145210 con 0x7fa828148350 2024-03-06T07:25:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.452+0000 7fa82dac3640 1 -- 172.21.15.59:0/445927882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa828068840 con 0x7fa828148350 2024-03-06T07:25:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.452+0000 7fa80ffff640 1 -- 172.21.15.59:0/445927882 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa818015800 con 0x7fa828148350 2024-03-06T07:25:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.453+0000 7fa80ffff640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81406b4d0 0x7fa81406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.453+0000 7fa80ffff640 1 -- 172.21.15.59:0/445927882 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa81808aad0 con 0x7fa828148350 2024-03-06T07:25:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.453+0000 7fa826ffd640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81406b4d0 0x7fa81406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.453+0000 7fa826ffd640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81406b4d0 0x7fa81406d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa828072d00 tx=0x7fa81c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.457+0000 7fa80ffff640 1 -- 172.21.15.59:0/445927882 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa818019030 con 0x7fa828148350 2024-03-06T07:25:38.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:38.707+0000 7fa82dac3640 1 -- 172.21.15.59:0/445927882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7fa82804f330 con 0x7fa828148350 2024-03-06T07:25:39.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.696+0000 7fa80ffff640 1 -- 172.21.15.59:0/445927882 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v220) v1 ==== 140+0+0 (secure 0 0 0) 0x7fa818059c50 con 0x7fa828148350 2024-03-06T07:25:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 -- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81406b4d0 msgr2=0x7fa81406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81406b4d0 0x7fa81406d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa828072d00 tx=0x7fa81c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 -- 172.21.15.59:0/445927882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa828148350 msgr2=0x7fa8281440e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa828148350 0x7fa8281440e0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fa818011930 tx=0x7fa818011df0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 -- 172.21.15.59:0/445927882 shutdown_connections 2024-03-06T07:25:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81406b4d0 0x7fa81406d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa828151a20 0x7fa828151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa828148730 0x7fa828144620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 --2- 172.21.15.59:0/445927882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa828148350 0x7fa8281440e0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 -- 172.21.15.59:0/445927882 >> 172.21.15.59:0/445927882 conn(0x7fa8280cf8a0 msgr2=0x7fa82814e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 -- 172.21.15.59:0/445927882 shutdown_connections 2024-03-06T07:25:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.699+0000 7fa82dac3640 1 -- 172.21.15.59:0/445927882 wait complete. 2024-03-06T07:25:39.714 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-03-06T07:25:39.715 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-03-06T07:25:39.715 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:25:39.715 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:39.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.820+0000 7fc134447640 1 Processor -- start 2024-03-06T07:25:39.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.820+0000 7fc134447640 1 -- start start 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.821+0000 7fc134447640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc12c0d22b0 0x7fc12c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.821+0000 7fc134447640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc12c0b41f0 0x7fc12c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.821+0000 7fc134447640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c0b3430 con 0x7fc12c15dfe0 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.821+0000 7fc134447640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c0d2c50 con 0x7fc12c0d22b0 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.821+0000 7fc134447640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c0b31e0 con 0x7fc12c0b41f0 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.821+0000 7fc1319e2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc12c0d22b0 0x7fc12c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.821+0000 7fc1321e3640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc12c15dfe0 0x7fc12c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55344/0 (socket says 172.21.15.59:55344) 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.821+0000 7fc1321e3640 1 -- 172.21.15.59:0/3713624763 learned_addr learned my addr 172.21.15.59:0/3713624763 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.822+0000 7fc1329e4640 1 --2- 172.21.15.59:0/3713624763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc12c0b41f0 0x7fc12c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.822+0000 7fc1319e2640 1 -- 172.21.15.59:0/3713624763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc12c0b41f0 msgr2=0x7fc12c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.822+0000 7fc1319e2640 1 --2- 172.21.15.59:0/3713624763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc12c0b41f0 0x7fc12c0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.822+0000 7fc1319e2640 1 -- 172.21.15.59:0/3713624763 >> v1:172.21.15.59:6789/0 conn(0x7fc12c15dfe0 legacy=0x7fc12c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.822+0000 7fc1319e2640 1 -- 172.21.15.59:0/3713624763 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c0b3710 con 0x7fc12c0d22b0 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc1319e2640 1 --2- 172.21.15.59:0/3713624763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc12c0d22b0 0x7fc12c0d2710 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc1200097a0 tx=0x7fc12002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65e591cfee261a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc1311e1640 1 -- 172.21.15.59:0/3713624763 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc120035020 con 0x7fc12c0d22b0 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc1311e1640 1 -- 172.21.15.59:0/3713624763 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc120031830 con 0x7fc12c0d22b0 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc1311e1640 1 -- 172.21.15.59:0/3713624763 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc120031c00 con 0x7fc12c0d22b0 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc134447640 1 -- 172.21.15.59:0/3713624763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc12c0d22b0 msgr2=0x7fc12c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc134447640 1 --2- 172.21.15.59:0/3713624763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc12c0d22b0 0x7fc12c0d2710 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc1200097a0 tx=0x7fc12002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc134447640 1 -- 172.21.15.59:0/3713624763 shutdown_connections 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc134447640 1 --2- 172.21.15.59:0/3713624763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc12c0b41f0 0x7fc12c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc134447640 1 --2- 172.21.15.59:0/3713624763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc12c0d22b0 0x7fc12c0d2710 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc134447640 1 -- 172.21.15.59:0/3713624763 >> 172.21.15.59:0/3713624763 conn(0x7fc12c0d3cd0 msgr2=0x7fc12c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc134447640 1 -- 172.21.15.59:0/3713624763 shutdown_connections 2024-03-06T07:25:39.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.823+0000 7fc134447640 1 -- 172.21.15.59:0/3713624763 wait complete. 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.824+0000 7fc134447640 1 Processor -- start 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.825+0000 7fc134447640 1 -- start start 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.825+0000 7fc134447640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc12c0b41f0 0x7fc12c0b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.825+0000 7fc134447640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc12c0d22b0 0x7fc12c1f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.825+0000 7fc134447640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc12c15dfe0 0x7fc12c1f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.825+0000 7fc134447640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c0cfa00 con 0x7fc12c0d22b0 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.825+0000 7fc134447640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c0b3d60 con 0x7fc12c15dfe0 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.825+0000 7fc134447640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c0d1c70 con 0x7fc12c0b41f0 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.825+0000 7fc1319e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc12c0d22b0 0x7fc12c1f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.825+0000 7fc1319e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc12c0d22b0 0x7fc12c1f3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44868/0 (socket says 172.21.15.59:44868) 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.825+0000 7fc1319e2640 1 -- 172.21.15.59:0/2772944587 learned_addr learned my addr 172.21.15.59:0/2772944587 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc1321e3640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc12c0b41f0 0x7fc12c0b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc1319e2640 1 -- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc12c0b41f0 msgr2=0x7fc12c0b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc1319e2640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc12c0b41f0 0x7fc12c0b6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc1319e2640 1 -- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc12c15dfe0 msgr2=0x7fc12c1f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc1319e2640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc12c15dfe0 0x7fc12c1f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc1319e2640 1 -- 172.21.15.59:0/2772944587 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c0b9490 con 0x7fc12c0d22b0 2024-03-06T07:25:39.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc1319e2640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc12c0d22b0 0x7fc12c1f3900 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fc120009890 tx=0x7fc1200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc12affd640 1 -- 172.21.15.59:0/2772944587 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc120041070 con 0x7fc12c0d22b0 2024-03-06T07:25:39.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc12affd640 1 -- 172.21.15.59:0/2772944587 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12003b920 con 0x7fc12c0d22b0 2024-03-06T07:25:39.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc134447640 1 -- 172.21.15.59:0/2772944587 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c0b9770 con 0x7fc12c0d22b0 2024-03-06T07:25:39.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc12affd640 1 -- 172.21.15.59:0/2772944587 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc120035040 con 0x7fc12c0d22b0 2024-03-06T07:25:39.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.826+0000 7fc134447640 1 -- 172.21.15.59:0/2772944587 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c0b1130 con 0x7fc12c0d22b0 2024-03-06T07:25:39.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.827+0000 7fc134447640 1 -- 172.21.15.59:0/2772944587 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c04f330 con 0x7fc12c0d22b0 2024-03-06T07:25:39.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.827+0000 7fc12affd640 1 -- 172.21.15.59:0/2772944587 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc12004c060 con 0x7fc12c0d22b0 2024-03-06T07:25:39.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.828+0000 7fc12affd640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc11806b4d0 0x7fc11806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.828+0000 7fc12affd640 1 -- 172.21.15.59:0/2772944587 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc1200ab860 con 0x7fc12c0d22b0 2024-03-06T07:25:39.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.828+0000 7fc1321e3640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc11806b4d0 0x7fc11806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.829+0000 7fc1321e3640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc11806b4d0 0x7fc11806d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc12c15e9e0 tx=0x7fc124009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:39.832+0000 7fc12affd640 1 -- 172.21.15.59:0/2772944587 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc12007a9f0 con 0x7fc12c0d22b0 2024-03-06T07:25:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.016+0000 7fc134447640 1 -- 172.21.15.59:0/2772944587 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc12c160470 con 0x7fc12c0d22b0 2024-03-06T07:25:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.017+0000 7fc12affd640 1 -- 172.21.15.59:0/2772944587 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+13009 (secure 0 0 0) 0x7fc12007a2f0 con 0x7fc12c0d22b0 2024-03-06T07:25:40.019 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:40.019 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:37.687219+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:6837/250576205":"2024-03-07T07:25:35.117519+0000","172.21.15.164:6835/250576205":"2024-03-07T07:25:35.117519+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 -- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc11806b4d0 msgr2=0x7fc11806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc11806b4d0 0x7fc11806d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc12c15e9e0 tx=0x7fc124009040 comp rx=0 tx=0).stop 2024-03-06T07:25:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 -- 172.21.15.59:0/2772944587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc12c0d22b0 msgr2=0x7fc12c1f3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc12c0d22b0 0x7fc12c1f3900 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fc120009890 tx=0x7fc1200046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 -- 172.21.15.59:0/2772944587 shutdown_connections 2024-03-06T07:25:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc11806b4d0 0x7fc11806d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc12c15dfe0 0x7fc12c1f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc12c0d22b0 0x7fc12c1f3900 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 --2- 172.21.15.59:0/2772944587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc12c0b41f0 0x7fc12c0b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 -- 172.21.15.59:0/2772944587 >> 172.21.15.59:0/2772944587 conn(0x7fc12c0d3cd0 msgr2=0x7fc12c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 -- 172.21.15.59:0/2772944587 shutdown_connections 2024-03-06T07:25:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.019+0000 7fc134447640 1 -- 172.21.15.59:0/2772944587 wait complete. 2024-03-06T07:25:40.032 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:40.032 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:40.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.134+0000 7f7528029640 1 Processor -- start 2024-03-06T07:25:40.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.135+0000 7f7528029640 1 -- start start 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.135+0000 7f7528029640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f752006e9d0 0x7f752015c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.135+0000 7f7528029640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7520070f50 con 0x7f75200cdd10 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.135+0000 7f7528029640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7520071230 con 0x7f7520145700 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.135+0000 7f7528029640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7520071b60 con 0x7f752006e9d0 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.135+0000 7f75265c6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f752006e9d0 0x7f752015c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.136+0000 7f75265c6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f752006e9d0 0x7f752015c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44876/0 (socket says 172.21.15.59:44876) 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.136+0000 7f75255c4640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f75200cdd10 0x7f752006e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55352/0 (socket says 172.21.15.59:55352) 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.136+0000 7f75265c6640 1 -- 172.21.15.59:0/2156369572 learned_addr learned my addr 172.21.15.59:0/2156369572 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.136+0000 7f75265c6640 1 -- 172.21.15.59:0/2156369572 >> v1:172.21.15.164:6789/0 conn(0x7f7520145700 legacy=0x7f75200cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.136+0000 7f75265c6640 1 -- 172.21.15.59:0/2156369572 >> v1:172.21.15.59:6789/0 conn(0x7f75200cdd10 legacy=0x7f752006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.136+0000 7f7524dc3640 1 -- 172.21.15.59:0/2156369572 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645073594 0 0) 0x7f7520070f50 con 0x7f75200cdd10 2024-03-06T07:25:40.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.136+0000 7f75265c6640 1 -- 172.21.15.59:0/2156369572 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7520071510 con 0x7f752006e9d0 2024-03-06T07:25:40.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.137+0000 7f75265c6640 1 --2- 172.21.15.59:0/2156369572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f752006e9d0 0x7f752015c400 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f7510009e50 tx=0x7f751002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f719b2de93f95f6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.137+0000 7f7524dc3640 1 -- 172.21.15.59:0/2156369572 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751002cb20 con 0x7f752006e9d0 2024-03-06T07:25:40.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.137+0000 7f7524dc3640 1 -- 172.21.15.59:0/2156369572 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751002ccc0 con 0x7f752006e9d0 2024-03-06T07:25:40.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.137+0000 7f7524dc3640 1 -- 172.21.15.59:0/2156369572 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510034610 con 0x7f752006e9d0 2024-03-06T07:25:40.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.137+0000 7f7528029640 1 -- 172.21.15.59:0/2156369572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f752006e9d0 msgr2=0x7f752015c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.137+0000 7f7528029640 1 --2- 172.21.15.59:0/2156369572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f752006e9d0 0x7f752015c400 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f7510009e50 tx=0x7f751002bf80 comp rx=0 tx=0).stop 2024-03-06T07:25:40.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.137+0000 7f7528029640 1 -- 172.21.15.59:0/2156369572 shutdown_connections 2024-03-06T07:25:40.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.137+0000 7f7528029640 1 --2- 172.21.15.59:0/2156369572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f752006e9d0 0x7f752015c400 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.137+0000 7f7528029640 1 -- 172.21.15.59:0/2156369572 >> 172.21.15.59:0/2156369572 conn(0x7f752006b830 msgr2=0x7f75200b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.137+0000 7f7528029640 1 -- 172.21.15.59:0/2156369572 shutdown_connections 2024-03-06T07:25:40.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.137+0000 7f7528029640 1 -- 172.21.15.59:0/2156369572 wait complete. 2024-03-06T07:25:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.138+0000 7f7528029640 1 Processor -- start 2024-03-06T07:25:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.138+0000 7f7528029640 1 -- start start 2024-03-06T07:25:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.139+0000 7f7528029640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f752006e9d0 0x7f75200ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.139+0000 7f7528029640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75200cdd10 0x7f75200aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.139+0000 7f7528029640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7520145700 0x7f75200c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.139+0000 7f7528029640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7520145ae0 con 0x7f752006e9d0 2024-03-06T07:25:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.139+0000 7f7528029640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75200cfbc0 con 0x7f7520145700 2024-03-06T07:25:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.139+0000 7f7528029640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7520145c60 con 0x7f75200cdd10 2024-03-06T07:25:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.139+0000 7f7525dc5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f752006e9d0 0x7f75200ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.139+0000 7f7525dc5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f752006e9d0 0x7f75200ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44890/0 (socket says 172.21.15.59:44890) 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.139+0000 7f7525dc5640 1 -- 172.21.15.59:0/1431453928 learned_addr learned my addr 172.21.15.59:0/1431453928 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f75255c4640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75200cdd10 0x7f75200aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f75265c6640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7520145700 0x7f75200c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f7525dc5640 1 -- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75200cdd10 msgr2=0x7f75200aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f7525dc5640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75200cdd10 0x7f75200aef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f7525dc5640 1 -- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7520145700 msgr2=0x7f75200c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f7525dc5640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7520145700 0x7f75200c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f7525dc5640 1 -- 172.21.15.59:0/1431453928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75200c6800 con 0x7f752006e9d0 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f75255c4640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75200cdd10 0x7f75200aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f75265c6640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7520145700 0x7f75200c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f7525dc5640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f752006e9d0 0x7f75200ae9e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f751800c430 tx=0x7f751800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f751e7fc640 1 -- 172.21.15.59:0/1431453928 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7518017070 con 0x7f752006e9d0 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f751e7fc640 1 -- 172.21.15.59:0/1431453928 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75180052d0 con 0x7f752006e9d0 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f7528029640 1 -- 172.21.15.59:0/1431453928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75200c6ae0 con 0x7f752006e9d0 2024-03-06T07:25:40.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f751e7fc640 1 -- 172.21.15.59:0/1431453928 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7518003830 con 0x7f752006e9d0 2024-03-06T07:25:40.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.140+0000 7f7528029640 1 -- 172.21.15.59:0/1431453928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75200c70a0 con 0x7f752006e9d0 2024-03-06T07:25:40.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.141+0000 7f7528029640 1 -- 172.21.15.59:0/1431453928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752006e0b0 con 0x7f752006e9d0 2024-03-06T07:25:40.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.141+0000 7f751e7fc640 1 -- 172.21.15.59:0/1431453928 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f751801a070 con 0x7f752006e9d0 2024-03-06T07:25:40.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.142+0000 7f751e7fc640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f750c06b4d0 0x7f750c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.142+0000 7f751e7fc640 1 -- 172.21.15.59:0/1431453928 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7518089280 con 0x7f752006e9d0 2024-03-06T07:25:40.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.142+0000 7f75255c4640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f750c06b4d0 0x7f750c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.143+0000 7f75255c4640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f750c06b4d0 0x7f750c06d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f75200b4b80 tx=0x7f7514009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.146+0000 7f751e7fc640 1 -- 172.21.15.59:0/1431453928 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7518058400 con 0x7f752006e9d0 2024-03-06T07:25:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.326+0000 7f7528029640 1 -- 172.21.15.59:0/1431453928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f752015f4c0 con 0x7f752006e9d0 2024-03-06T07:25:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.326+0000 7f751e7fc640 1 -- 172.21.15.59:0/1431453928 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7518057d00 con 0x7f752006e9d0 2024-03-06T07:25:40.328 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:25:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 -- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f750c06b4d0 msgr2=0x7f750c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f750c06b4d0 0x7f750c06d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f75200b4b80 tx=0x7f7514009040 comp rx=0 tx=0).stop 2024-03-06T07:25:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 -- 172.21.15.59:0/1431453928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f752006e9d0 msgr2=0x7f75200ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f752006e9d0 0x7f75200ae9e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f751800c430 tx=0x7f751800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 -- 172.21.15.59:0/1431453928 shutdown_connections 2024-03-06T07:25:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f750c06b4d0 0x7f750c06d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7520145700 0x7f75200c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75200cdd10 0x7f75200aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 --2- 172.21.15.59:0/1431453928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f752006e9d0 0x7f75200ae9e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 -- 172.21.15.59:0/1431453928 >> 172.21.15.59:0/1431453928 conn(0x7f752006b830 msgr2=0x7f7520142a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 -- 172.21.15.59:0/1431453928 shutdown_connections 2024-03-06T07:25:40.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.329+0000 7f7528029640 1 -- 172.21.15.59:0/1431453928 wait complete. 2024-03-06T07:25:40.341 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.441+0000 7f4b0f615640 1 Processor -- start 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.441+0000 7f4b0f615640 1 -- start start 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.442+0000 7f4b0f615640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b080d2380 0x7f4b080d27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.442+0000 7f4b0f615640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4b080cee00 0x7f4b080cf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.442+0000 7f4b0f615640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b080b3fc0 con 0x7f4b0815e0b0 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.442+0000 7f4b0f615640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b080ce990 con 0x7f4b080cee00 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.442+0000 7f4b0f615640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080d2d20 con 0x7f4b080d2380 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.442+0000 7f4b0dbb2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4b080cee00 0x7f4b080cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.442+0000 7f4b0dbb2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4b080cee00 0x7f4b080cf280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44898/0 (socket says 172.21.15.59:44898) 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.442+0000 7f4b0dbb2640 1 -- 172.21.15.59:0/1105003518 learned_addr learned my addr 172.21.15.59:0/1105003518 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.442+0000 7f4b0d3b1640 1 --1- 172.21.15.59:0/1105003518 >> v1:172.21.15.59:6789/0 conn(0x7f4b0815e0b0 0x7f4b080b38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55368/0 (socket says 172.21.15.59:55368) 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.443+0000 7f4b0cbb0640 1 --2- 172.21.15.59:0/1105003518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b080d2380 0x7f4b080d27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.443+0000 7f4b0dbb2640 1 -- 172.21.15.59:0/1105003518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b080d2380 msgr2=0x7f4b080d27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.443+0000 7f4b0dbb2640 1 --2- 172.21.15.59:0/1105003518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b080d2380 0x7f4b080d27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.443+0000 7f4b0dbb2640 1 -- 172.21.15.59:0/1105003518 >> v1:172.21.15.59:6789/0 conn(0x7f4b0815e0b0 legacy=0x7f4b080b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.443+0000 7f4b0dbb2640 1 -- 172.21.15.59:0/1105003518 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b080b4290 con 0x7f4b080cee00 2024-03-06T07:25:40.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.443+0000 7f4b0dbb2640 1 --2- 172.21.15.59:0/1105003518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b080cee00 0x7f4b080cf280 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f4af8009e50 tx=0x7f4af802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51226d3191d53ea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.443+0000 7f4b07fff640 1 -- 172.21.15.59:0/1105003518 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af802cb00 con 0x7f4b080cee00 2024-03-06T07:25:40.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.444+0000 7f4b07fff640 1 -- 172.21.15.59:0/1105003518 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af802cca0 con 0x7f4b080cee00 2024-03-06T07:25:40.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.444+0000 7f4b07fff640 1 -- 172.21.15.59:0/1105003518 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8034610 con 0x7f4b080cee00 2024-03-06T07:25:40.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.444+0000 7f4b0f615640 1 -- 172.21.15.59:0/1105003518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b080cee00 msgr2=0x7f4b080cf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.444+0000 7f4b0f615640 1 --2- 172.21.15.59:0/1105003518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b080cee00 0x7f4b080cf280 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f4af8009e50 tx=0x7f4af802bf60 comp rx=0 tx=0).stop 2024-03-06T07:25:40.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.444+0000 7f4b0f615640 1 -- 172.21.15.59:0/1105003518 shutdown_connections 2024-03-06T07:25:40.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.444+0000 7f4b0f615640 1 --2- 172.21.15.59:0/1105003518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b080cee00 0x7f4b080cf280 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.444+0000 7f4b0f615640 1 --2- 172.21.15.59:0/1105003518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b080d2380 0x7f4b080d27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.444+0000 7f4b0f615640 1 -- 172.21.15.59:0/1105003518 >> 172.21.15.59:0/1105003518 conn(0x7f4b0808d6f0 msgr2=0x7f4b0809d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.444+0000 7f4b0f615640 1 -- 172.21.15.59:0/1105003518 shutdown_connections 2024-03-06T07:25:40.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.444+0000 7f4b0f615640 1 -- 172.21.15.59:0/1105003518 wait complete. 2024-03-06T07:25:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.445+0000 7f4b0f615640 1 Processor -- start 2024-03-06T07:25:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.445+0000 7f4b0f615640 1 -- start start 2024-03-06T07:25:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0f615640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b080cee00 0x7f4b0814dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0f615640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b080d2380 0x7f4b080b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0f615640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b0815e0b0 0x7f4b080cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0f615640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080b4b30 con 0x7f4b0815e0b0 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0f615640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080b4850 con 0x7f4b080d2380 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0f615640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0815e490 con 0x7f4b080cee00 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0dbb2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b0815e0b0 0x7f4b080cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0dbb2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b0815e0b0 0x7f4b080cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44906/0 (socket says 172.21.15.59:44906) 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0dbb2640 1 -- 172.21.15.59:0/3627061180 learned_addr learned my addr 172.21.15.59:0/3627061180 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0d3b1640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b080cee00 0x7f4b0814dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0dbb2640 1 -- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b080cee00 msgr2=0x7f4b0814dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0cbb0640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b080d2380 0x7f4b080b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0dbb2640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b080cee00 0x7f4b0814dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0dbb2640 1 -- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b080d2380 msgr2=0x7f4b080b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0dbb2640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b080d2380 0x7f4b080b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0dbb2640 1 -- 172.21.15.59:0/3627061180 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b080b6f00 con 0x7f4b0815e0b0 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0d3b1640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b080cee00 0x7f4b0814dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.446+0000 7f4b0cbb0640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b080d2380 0x7f4b080b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.447+0000 7f4b0dbb2640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b0815e0b0 0x7f4b080cd710 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f4af8036040 tx=0x7f4af8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.447+0000 7f4b05ffb640 1 -- 172.21.15.59:0/3627061180 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8043020 con 0x7f4b0815e0b0 2024-03-06T07:25:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.447+0000 7f4b05ffb640 1 -- 172.21.15.59:0/3627061180 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af802ce50 con 0x7f4b0815e0b0 2024-03-06T07:25:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.447+0000 7f4b0f615640 1 -- 172.21.15.59:0/3627061180 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0815f880 con 0x7f4b0815e0b0 2024-03-06T07:25:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.447+0000 7f4b05ffb640 1 -- 172.21.15.59:0/3627061180 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af8009510 con 0x7f4b0815e0b0 2024-03-06T07:25:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.447+0000 7f4b0f615640 1 -- 172.21.15.59:0/3627061180 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0815fe40 con 0x7f4b0815e0b0 2024-03-06T07:25:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.448+0000 7f4b0f615640 1 -- 172.21.15.59:0/3627061180 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0804f330 con 0x7f4b0815e0b0 2024-03-06T07:25:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.449+0000 7f4b05ffb640 1 -- 172.21.15.59:0/3627061180 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4af8046020 con 0x7f4b0815e0b0 2024-03-06T07:25:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.449+0000 7f4b05ffb640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4af406b500 0x7f4af406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.449+0000 7f4b05ffb640 1 -- 172.21.15.59:0/3627061180 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4af80ac550 con 0x7f4b0815e0b0 2024-03-06T07:25:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.449+0000 7f4b0d3b1640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4af406b500 0x7f4af406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.450+0000 7f4b0d3b1640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4af406b500 0x7f4af406d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4b00000f60 tx=0x7f4b00009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.453+0000 7f4b05ffb640 1 -- 172.21.15.59:0/3627061180 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4af804d030 con 0x7f4b0815e0b0 2024-03-06T07:25:40.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:40.707+0000 7f4b0f615640 1 -- 172.21.15.59:0/3627061180 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f4b080d46d0 con 0x7f4b0815e0b0 2024-03-06T07:25:41.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.702+0000 7f4b05ffb640 1 -- 172.21.15.59:0/3627061180 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v222) v1 ==== 135+0+0 (secure 0 0 0) 0x7f4af807b6d0 con 0x7f4b0815e0b0 2024-03-06T07:25:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 -- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4af406b500 msgr2=0x7f4af406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4af406b500 0x7f4af406d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4b00000f60 tx=0x7f4b00009040 comp rx=0 tx=0).stop 2024-03-06T07:25:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 -- 172.21.15.59:0/3627061180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b0815e0b0 msgr2=0x7f4b080cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b0815e0b0 0x7f4b080cd710 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f4af8036040 tx=0x7f4af8004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 -- 172.21.15.59:0/3627061180 shutdown_connections 2024-03-06T07:25:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4af406b500 0x7f4af406d9c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b0815e0b0 0x7f4b080cd710 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b080d2380 0x7f4b080b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 --2- 172.21.15.59:0/3627061180 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b080cee00 0x7f4b0814dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 -- 172.21.15.59:0/3627061180 >> 172.21.15.59:0/3627061180 conn(0x7f4b0808d6f0 msgr2=0x7f4b080b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 -- 172.21.15.59:0/3627061180 shutdown_connections 2024-03-06T07:25:41.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.705+0000 7f4b0f615640 1 -- 172.21.15.59:0/3627061180 wait complete. 2024-03-06T07:25:41.720 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:25:41.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.829+0000 7f405e9f7640 1 Processor -- start 2024-03-06T07:25:41.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.830+0000 7f405e9f7640 1 -- start start 2024-03-06T07:25:41.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.830+0000 7f405e9f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40581614d0 0x7f40580d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.830+0000 7f405e9f7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f40580d2480 0x7f40580b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.830+0000 7f405e9f7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40581618b0 con 0x7f40580d1910 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.830+0000 7f405e9f7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40580cf400 con 0x7f40580d2480 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.830+0000 7f405e9f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40580cf6a0 con 0x7f40581614d0 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.830+0000 7f40577fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f40580d1910 0x7f40580d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55378/0 (socket says 172.21.15.59:55378) 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f40577fe640 1 -- 172.21.15.59:0/2855106109 learned_addr learned my addr 172.21.15.59:0/2855106109 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f405cf94640 1 --2- 172.21.15.59:0/2855106109 >> v2:172.21.15.164:3300/0 conn(0x7f40580d2480 0x7f40580b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4057fff640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40581614d0 0x7f40580d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287240100 0 0) 0x7f40581618b0 con 0x7f40580d1910 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4044003520 con 0x7f40580d1910 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2422956782 0 0) 0x7f4044003520 con 0x7f40580d1910 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40581614d0 msgr2=0x7f40580d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40581614d0 0x7f40580d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 >> v2:172.21.15.164:3300/0 conn(0x7f40580d2480 msgr2=0x7f40580b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 --2- 172.21.15.59:0/2855106109 >> v2:172.21.15.164:3300/0 conn(0x7f40580d2480 0x7f40580b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40580cf980 con 0x7f40580d1910 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f4048001670 con 0x7f40580d1910 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 >> v1:172.21.15.59:6789/0 conn(0x7f40580d1910 legacy=0x7f40580d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 0x7f4044003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4044004800 0x7f4044004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40581614d0 0x7f40440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40580cf400 con 0x7f40440039b0 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40580cf6a0 con 0x7f4044004800 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404400b9c0 con 0x7f40581614d0 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f4057fff640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 0x7f4044003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f40577fe640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4044004800 0x7f4044004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.831+0000 7f405cf94640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40581614d0 0x7f40440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.832+0000 7f4057fff640 1 -- 172.21.15.59:0/2855106109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40581614d0 msgr2=0x7f40440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.832+0000 7f4057fff640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40581614d0 0x7f40440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.832+0000 7f4057fff640 1 -- 172.21.15.59:0/2855106109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4044004800 msgr2=0x7f4044004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.832+0000 7f4057fff640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4044004800 0x7f4044004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.832+0000 7f4057fff640 1 -- 172.21.15.59:0/2855106109 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4044003520 con 0x7f40440039b0 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.832+0000 7f4057fff640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 0x7f4044003dd0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f4040009ac0 tx=0x7f4044005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec374887121bd075 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.832+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4040039070 con 0x7f40440039b0 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.832+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404002c6b0 con 0x7f40440039b0 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.832+0000 7f4056ffd640 1 -- 172.21.15.59:0/2855106109 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4040009690 con 0x7f40440039b0 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.833+0000 7f405e9f7640 1 -- 172.21.15.59:0/2855106109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 msgr2=0x7f4044003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.833+0000 7f405e9f7640 1 --2- 172.21.15.59:0/2855106109 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 0x7f4044003dd0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f4040009ac0 tx=0x7f4044005830 comp rx=0 tx=0).stop 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.833+0000 7f405cf94640 1 -- 172.21.15.59:0/2855106109 reap_dead start 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.833+0000 7f405e9f7640 1 -- 172.21.15.59:0/2855106109 shutdown_connections 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.833+0000 7f405e9f7640 1 -- 172.21.15.59:0/2855106109 >> 172.21.15.59:0/2855106109 conn(0x7f40580d3d40 msgr2=0x7f40580d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.833+0000 7f405e9f7640 1 -- 172.21.15.59:0/2855106109 shutdown_connections 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.833+0000 7f405e9f7640 1 -- 172.21.15.59:0/2855106109 wait complete. 2024-03-06T07:25:41.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.834+0000 7f405e9f7640 1 Processor -- start 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.834+0000 7f405e9f7640 1 -- start start 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.835+0000 7f405e9f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 0x7f40581f8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.835+0000 7f405e9f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f405815a240 0x7f40581f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.835+0000 7f405e9f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f405815cd50 0x7f405815d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.835+0000 7f405e9f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40580d4860 con 0x7f40440039b0 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.835+0000 7f405e9f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40580b0910 con 0x7f405815cd50 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.835+0000 7f405e9f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40580d49e0 con 0x7f405815a240 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.835+0000 7f4057fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 0x7f40581f8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.835+0000 7f4057fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 0x7f40581f8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44930/0 (socket says 172.21.15.59:44930) 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.835+0000 7f4057fff640 1 -- 172.21.15.59:0/649114526 learned_addr learned my addr 172.21.15.59:0/649114526 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.835+0000 7f40577fe640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f405815a240 0x7f40581f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.835+0000 7f405cf94640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f405815cd50 0x7f405815d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f4057fff640 1 -- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f405815a240 msgr2=0x7f40581f3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f4057fff640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f405815a240 0x7f40581f3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f4057fff640 1 -- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f405815cd50 msgr2=0x7f405815d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f4057fff640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f405815cd50 0x7f405815d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f4057fff640 1 -- 172.21.15.59:0/649114526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40580c7870 con 0x7f40440039b0 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f405cf94640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f405815cd50 0x7f405815d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f4057fff640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 0x7f40581f8040 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f4040009be0 tx=0x7f4040004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f4054ff9640 1 -- 172.21.15.59:0/649114526 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4040042020 con 0x7f40440039b0 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f4054ff9640 1 -- 172.21.15.59:0/649114526 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404003a040 con 0x7f40440039b0 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f405e9f7640 1 -- 172.21.15.59:0/649114526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40581f4800 con 0x7f40440039b0 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f4054ff9640 1 -- 172.21.15.59:0/649114526 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4040039070 con 0x7f40440039b0 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.836+0000 7f405e9f7640 1 -- 172.21.15.59:0/649114526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40581f4dc0 con 0x7f40440039b0 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.837+0000 7f405e9f7640 1 -- 172.21.15.59:0/649114526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4058068ff0 con 0x7f40440039b0 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.837+0000 7f4054ff9640 1 -- 172.21.15.59:0/649114526 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4040045020 con 0x7f40440039b0 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.838+0000 7f4054ff9640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f403806b500 0x7f403806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.838+0000 7f4054ff9640 1 -- 172.21.15.59:0/649114526 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f40400ab250 con 0x7f40440039b0 2024-03-06T07:25:41.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.838+0000 7f40577fe640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f403806b500 0x7f403806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.839+0000 7f40577fe640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f403806b500 0x7f403806d9c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4048007fd0 tx=0x7f40480078f0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:41.842+0000 7f4054ff9640 1 -- 172.21.15.59:0/649114526 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f404004c070 con 0x7f40440039b0 2024-03-06T07:25:42.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.023+0000 7f405e9f7640 1 -- 172.21.15.59:0/649114526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f40580b1770 con 0x7f40440039b0 2024-03-06T07:25:42.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.026+0000 7f4054ff9640 1 -- 172.21.15.59:0/649114526 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f404007a3d0 con 0x7f40440039b0 2024-03-06T07:25:42.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 -- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f403806b500 msgr2=0x7f403806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f403806b500 0x7f403806d9c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4048007fd0 tx=0x7f40480078f0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 -- 172.21.15.59:0/649114526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 msgr2=0x7f40581f8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 0x7f40581f8040 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f4040009be0 tx=0x7f4040004730 comp rx=0 tx=0).stop 2024-03-06T07:25:42.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 -- 172.21.15.59:0/649114526 shutdown_connections 2024-03-06T07:25:42.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f403806b500 0x7f403806d9c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f405815cd50 0x7f405815d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f405815a240 0x7f40581f3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 --2- 172.21.15.59:0/649114526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40440039b0 0x7f40581f8040 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 -- 172.21.15.59:0/649114526 >> 172.21.15.59:0/649114526 conn(0x7f40580d3d40 msgr2=0x7f40580b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 -- 172.21.15.59:0/649114526 shutdown_connections 2024-03-06T07:25:42.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.028+0000 7f405e9f7640 1 -- 172.21.15.59:0/649114526 wait complete. 2024-03-06T07:25:42.041 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:42.041 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:42.089 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:42.142 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:42.143 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:25:42.143 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:42.157 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:42.158 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:25:42.158 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:42.174 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:42.175 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:25:42.175 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:42.212 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:42.212 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:25:42.212 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:42.231 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:42.231 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:25:42.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:42.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.352+0000 7fc485d7a640 1 Processor -- start 2024-03-06T07:25:42.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.352+0000 7fc485d7a640 1 -- start start 2024-03-06T07:25:42.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc485d7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc48006da40 0x7fc4800cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc485d7a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4800cd7a0 0x7fc48015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc485d7a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4800cc980 con 0x7fc4801424f0 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc485d7a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4800cd220 con 0x7fc4800cd7a0 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc485d7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800ccc60 con 0x7fc48006da40 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc47ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4800cd7a0 0x7fc48015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc47ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4800cd7a0 0x7fc48015cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44932/0 (socket says 172.21.15.59:44932) 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc47ffff640 1 -- 172.21.15.59:0/2643809954 learned_addr learned my addr 172.21.15.59:0/2643809954 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc47effd640 1 --2- 172.21.15.59:0/2643809954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc48006da40 0x7fc4800cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc47ffff640 1 -- 172.21.15.59:0/2643809954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc48006da40 msgr2=0x7fc4800cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc47ffff640 1 --2- 172.21.15.59:0/2643809954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc48006da40 0x7fc4800cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc47ffff640 1 -- 172.21.15.59:0/2643809954 >> v1:172.21.15.164:6789/0 conn(0x7fc4801424f0 legacy=0x7fc48006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc47ffff640 1 -- 172.21.15.59:0/2643809954 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4800ccf40 con 0x7fc4800cd7a0 2024-03-06T07:25:42.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.353+0000 7fc47effd640 1 --2- 172.21.15.59:0/2643809954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc48006da40 0x7fc4800cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.354+0000 7fc47ffff640 1 --2- 172.21.15.59:0/2643809954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4800cd7a0 0x7fc48015cef0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc478009e50 tx=0x7fc47802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef3684fe0c1c43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.354+0000 7fc47e7fc640 1 -- 172.21.15.59:0/2643809954 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47802cb00 con 0x7fc4800cd7a0 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.354+0000 7fc47e7fc640 1 -- 172.21.15.59:0/2643809954 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47802cca0 con 0x7fc4800cd7a0 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.354+0000 7fc47e7fc640 1 -- 172.21.15.59:0/2643809954 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc478034610 con 0x7fc4800cd7a0 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.355+0000 7fc485d7a640 1 -- 172.21.15.59:0/2643809954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4800cd7a0 msgr2=0x7fc48015cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.355+0000 7fc485d7a640 1 --2- 172.21.15.59:0/2643809954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4800cd7a0 0x7fc48015cef0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc478009e50 tx=0x7fc47802bf60 comp rx=0 tx=0).stop 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.355+0000 7fc485d7a640 1 -- 172.21.15.59:0/2643809954 shutdown_connections 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.355+0000 7fc485d7a640 1 --2- 172.21.15.59:0/2643809954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4800cd7a0 0x7fc48015cef0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.355+0000 7fc485d7a640 1 --2- 172.21.15.59:0/2643809954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc48006da40 0x7fc4800cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.355+0000 7fc485d7a640 1 -- 172.21.15.59:0/2643809954 >> 172.21.15.59:0/2643809954 conn(0x7fc4800cf8a0 msgr2=0x7fc4800cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.355+0000 7fc485d7a640 1 -- 172.21.15.59:0/2643809954 shutdown_connections 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.355+0000 7fc485d7a640 1 -- 172.21.15.59:0/2643809954 wait complete. 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.356+0000 7fc485d7a640 1 Processor -- start 2024-03-06T07:25:42.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.356+0000 7fc485d7a640 1 -- start start 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc485d7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc48006da40 0x7fc4801428d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc485d7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4800cd7a0 0x7fc4801404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc485d7a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4801424f0 0x7fc48015e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc485d7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc480071ce0 con 0x7fc48006da40 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc485d7a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc480071b60 con 0x7fc4801424f0 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc485d7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc480071e60 con 0x7fc4800cd7a0 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc47f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc48006da40 0x7fc4801428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc47f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc48006da40 0x7fc4801428d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44948/0 (socket says 172.21.15.59:44948) 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc47f7fe640 1 -- 172.21.15.59:0/3871800141 learned_addr learned my addr 172.21.15.59:0/3871800141 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc47f7fe640 1 -- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4800cd7a0 msgr2=0x7fc4801404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc47f7fe640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4800cd7a0 0x7fc4801404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc47f7fe640 1 -- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4801424f0 msgr2=0x7fc48015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc47ffff640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4801424f0 0x7fc48015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc47f7fe640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4801424f0 0x7fc48015e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc47f7fe640 1 -- 172.21.15.59:0/3871800141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4801f5310 con 0x7fc48006da40 2024-03-06T07:25:42.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.357+0000 7fc47f7fe640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc48006da40 0x7fc4801428d0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc47000c3f0 tx=0x7fc47000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.358+0000 7fc467fff640 1 -- 172.21.15.59:0/3871800141 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4700052d0 con 0x7fc48006da40 2024-03-06T07:25:42.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.358+0000 7fc467fff640 1 -- 172.21.15.59:0/3871800141 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc470005490 con 0x7fc48006da40 2024-03-06T07:25:42.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.358+0000 7fc485d7a640 1 -- 172.21.15.59:0/3871800141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4801f55f0 con 0x7fc48006da40 2024-03-06T07:25:42.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.358+0000 7fc467fff640 1 -- 172.21.15.59:0/3871800141 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470007260 con 0x7fc48006da40 2024-03-06T07:25:42.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.358+0000 7fc485d7a640 1 -- 172.21.15.59:0/3871800141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4801f5bb0 con 0x7fc48006da40 2024-03-06T07:25:42.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.359+0000 7fc485d7a640 1 -- 172.21.15.59:0/3871800141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4801f19b0 con 0x7fc48006da40 2024-03-06T07:25:42.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.359+0000 7fc467fff640 1 -- 172.21.15.59:0/3871800141 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc470007400 con 0x7fc48006da40 2024-03-06T07:25:42.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.359+0000 7fc467fff640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc46c06b4d0 0x7fc46c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.360+0000 7fc467fff640 1 -- 172.21.15.59:0/3871800141 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc470089350 con 0x7fc48006da40 2024-03-06T07:25:42.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.360+0000 7fc47effd640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc46c06b4d0 0x7fc46c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.360+0000 7fc47effd640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc46c06b4d0 0x7fc46c06d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc474006fd0 tx=0x7fc474008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.364+0000 7fc467fff640 1 -- 172.21.15.59:0/3871800141 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc470058560 con 0x7fc48006da40 2024-03-06T07:25:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.586+0000 7fc485d7a640 1 -- 172.21.15.59:0/3871800141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc48004f330 con 0x7fc48006da40 2024-03-06T07:25:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.586+0000 7fc467fff640 1 -- 172.21.15.59:0/3871800141 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fc470057e60 con 0x7fc48006da40 2024-03-06T07:25:42.588 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:42.589 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":37,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3948750038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3948750038},{"type":"v1","addr":"172.21.15.59:6835","nonce":3948750038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2166308105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2166308105},{"type":"v1","addr":"172.21.15.164:6836","nonce":2166308105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/2515696959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":2515696959},{"type":"v1","addr":"172.21.15.164:6837","nonce":2515696959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":37},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1858043839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1858043839},{"type":"v1","addr":"172.21.15.59:6834","nonce":1858043839}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[]} 2024-03-06T07:25:42.589 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 37 2024-03-06T07:25:42.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 -- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc46c06b4d0 msgr2=0x7fc46c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc46c06b4d0 0x7fc46c06d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc474006fd0 tx=0x7fc474008040 comp rx=0 tx=0).stop 2024-03-06T07:25:42.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 -- 172.21.15.59:0/3871800141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc48006da40 msgr2=0x7fc4801428d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc48006da40 0x7fc4801428d0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc47000c3f0 tx=0x7fc47000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 -- 172.21.15.59:0/3871800141 shutdown_connections 2024-03-06T07:25:42.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc46c06b4d0 0x7fc46c06d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4801424f0 0x7fc48015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4800cd7a0 0x7fc4801404b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 --2- 172.21.15.59:0/3871800141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc48006da40 0x7fc4801428d0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 -- 172.21.15.59:0/3871800141 >> 172.21.15.59:0/3871800141 conn(0x7fc4800cf8a0 msgr2=0x7fc480072400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 -- 172.21.15.59:0/3871800141 shutdown_connections 2024-03-06T07:25:42.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.588+0000 7fc485d7a640 1 -- 172.21.15.59:0/3871800141 wait complete. 2024-03-06T07:25:42.601 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:25:42.601 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:25:42.601 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:25:42.602 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:42.602 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:25:42.602 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:25:42.602 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:25:42.602 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:42.602 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:25:42.602 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:25:42.603 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:25:42.603 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:42.603 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:25:42.603 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:25:42.603 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:25:42.603 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:42.612 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:25:42.612 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:25:42.613 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:25:42.613 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:25:42.618 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:25:42.618 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:25:42.619 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:25:42.619 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:25:42.654 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:25:42.655 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:42.655 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:25:42.656 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:42.661 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:25:42.661 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:42.662 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:25:42.662 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:42.662 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:25:42.730 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:25:42.731 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:25:42.746 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:25:42.747 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:25:42.747 INFO:tasks.ceph.mds.d.smithi164.stderr:2024-03-06T07:25:42.745+0000 7fdc181c9640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:25:42.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.767+0000 7f404c9ae640 1 Processor -- start 2024-03-06T07:25:42.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.767+0000 7f404c9ae640 1 -- start start 2024-03-06T07:25:42.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f404c9ae640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404815c2c0 0x7f40480cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f404c9ae640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40480b3680 con 0x7f40481456f0 2024-03-06T07:25:42.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f404c9ae640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404815c760 con 0x7f40480cc830 2024-03-06T07:25:42.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f404c9ae640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404815ca40 con 0x7f404815c2c0 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f404659c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f40481456f0 0x7f4048070940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55396/0 (socket says 172.21.15.59:55396) 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f404659c640 1 -- 172.21.15.59:0/2609607548 learned_addr learned my addr 172.21.15.59:0/2609607548 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f4045d9b640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404815c2c0 0x7f40480cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231239666 0 0) 0x7f403c001110 con 0x7f40481456f0 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4030003520 con 0x7f40481456f0 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1337022328 0 0) 0x7f4030003520 con 0x7f40481456f0 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404815c2c0 msgr2=0x7f40480cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.768+0000 7f404559a640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404815c2c0 0x7f40480cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 >> v1:172.21.15.164:6789/0 conn(0x7f40480cc830 legacy=0x7f40480b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404815cd20 con 0x7f40481456f0 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f403c001840 con 0x7f40481456f0 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 >> v1:172.21.15.59:6789/0 conn(0x7f40481456f0 legacy=0x7f4048070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404559a640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40300039e0 0x7f4030003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404559a640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4030004800 0x7f4030004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404559a640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404815c2c0 0x7f40300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404815ca40 con 0x7f40300039e0 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403000baa0 con 0x7f4030004800 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403000bca0 con 0x7f404815c2c0 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404659c640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40300039e0 0x7f4030003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f4045d9b640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4030004800 0x7f4030004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f4046d9d640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404815c2c0 0x7f40300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404659c640 1 -- 172.21.15.59:0/2609607548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404815c2c0 msgr2=0x7f40300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404659c640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404815c2c0 0x7f40300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404659c640 1 -- 172.21.15.59:0/2609607548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4030004800 msgr2=0x7f4030004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404659c640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4030004800 0x7f4030004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.769+0000 7f404659c640 1 -- 172.21.15.59:0/2609607548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4030003520 con 0x7f40300039e0 2024-03-06T07:25:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.770+0000 7f404659c640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40300039e0 0x7f4030003e00 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f403c00c2f0 tx=0x7f4030005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10af8aca26c3cf28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.770+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c007e10 con 0x7f40300039e0 2024-03-06T07:25:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.770+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403c030070 con 0x7f40300039e0 2024-03-06T07:25:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.770+0000 7f404559a640 1 -- 172.21.15.59:0/2609607548 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c02ce30 con 0x7f40300039e0 2024-03-06T07:25:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.770+0000 7f404c9ae640 1 -- 172.21.15.59:0/2609607548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40300039e0 msgr2=0x7f4030003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.770+0000 7f404c9ae640 1 --2- 172.21.15.59:0/2609607548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40300039e0 0x7f4030003e00 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f403c00c2f0 tx=0x7f4030005830 comp rx=0 tx=0).stop 2024-03-06T07:25:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.770+0000 7f4046d9d640 1 -- 172.21.15.59:0/2609607548 reap_dead start 2024-03-06T07:25:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.770+0000 7f404c9ae640 1 -- 172.21.15.59:0/2609607548 shutdown_connections 2024-03-06T07:25:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.770+0000 7f404c9ae640 1 -- 172.21.15.59:0/2609607548 >> 172.21.15.59:0/2609607548 conn(0x7f40480b1c80 msgr2=0x7f40480cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.771+0000 7f404c9ae640 1 -- 172.21.15.59:0/2609607548 shutdown_connections 2024-03-06T07:25:42.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.771+0000 7f404c9ae640 1 -- 172.21.15.59:0/2609607548 wait complete. 2024-03-06T07:25:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.772+0000 7f404c9ae640 1 Processor -- start 2024-03-06T07:25:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.772+0000 7f404c9ae640 1 -- start start 2024-03-06T07:25:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f404c9ae640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40300039e0 0x7f40480aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f404c9ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40480cf5f0 0x7f40480cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f404c9ae640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404813f950 0x7f404813fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f404c9ae640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404815d800 con 0x7f40480cf5f0 2024-03-06T07:25:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f404c9ae640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4048071050 con 0x7f40300039e0 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f404c9ae640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404815d980 con 0x7f404813f950 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4045d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40480cf5f0 0x7f40480cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4045d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40480cf5f0 0x7f40480cfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45006/0 (socket says 172.21.15.59:45006) 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4045d9b640 1 -- 172.21.15.59:0/3431507066 learned_addr learned my addr 172.21.15.59:0/3431507066 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f404659c640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40300039e0 0x7f40480aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4046d9d640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404813f950 0x7f404813fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4045d9b640 1 -- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404813f950 msgr2=0x7f404813fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4045d9b640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404813f950 0x7f404813fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4045d9b640 1 -- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40300039e0 msgr2=0x7f40480aeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4045d9b640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40300039e0 0x7f40480aeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4045d9b640 1 -- 172.21.15.59:0/3431507066 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40481f4ca0 con 0x7f40480cf5f0 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f404659c640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40300039e0 0x7f40480aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4045d9b640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40480cf5f0 0x7f40480cfa70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f40380119d0 tx=0x7f4038011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4036ffd640 1 -- 172.21.15.59:0/3431507066 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403800fb50 con 0x7f40480cf5f0 2024-03-06T07:25:42.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4036ffd640 1 -- 172.21.15.59:0/3431507066 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403800fcf0 con 0x7f40480cf5f0 2024-03-06T07:25:42.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f4036ffd640 1 -- 172.21.15.59:0/3431507066 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403800fea0 con 0x7f40480cf5f0 2024-03-06T07:25:42.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f404c9ae640 1 -- 172.21.15.59:0/3431507066 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40481f4f60 con 0x7f40480cf5f0 2024-03-06T07:25:42.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.773+0000 7f404c9ae640 1 -- 172.21.15.59:0/3431507066 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40481f5520 con 0x7f40480cf5f0 2024-03-06T07:25:42.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.774+0000 7f404c9ae640 1 -- 172.21.15.59:0/3431507066 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404804f330 con 0x7f40480cf5f0 2024-03-06T07:25:42.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.775+0000 7f4036ffd640 1 -- 172.21.15.59:0/3431507066 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4038012040 con 0x7f40480cf5f0 2024-03-06T07:25:42.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.775+0000 7f4036ffd640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f402806b500 0x7f402806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.776+0000 7f4036ffd640 1 -- 172.21.15.59:0/3431507066 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f403808a280 con 0x7f40480cf5f0 2024-03-06T07:25:42.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.776+0000 7f404659c640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f402806b500 0x7f402806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.776+0000 7f404659c640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f402806b500 0x7f402806d9c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f403c034000 tx=0x7f403c009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.780+0000 7f4036ffd640 1 -- 172.21.15.59:0/3431507066 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4038019030 con 0x7f40480cf5f0 2024-03-06T07:25:42.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:42.973+0000 7f404c9ae640 1 -- 172.21.15.59:0/3431507066 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f40480008d0 con 0x7f40480cf5f0 2024-03-06T07:25:43.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.717+0000 7f4036ffd640 1 -- 172.21.15.59:0/3431507066 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v75) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4038058e40 con 0x7f40480cf5f0 2024-03-06T07:25:43.719 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-03-06T07:25:43.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 -- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f402806b500 msgr2=0x7f402806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f402806b500 0x7f402806d9c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f403c034000 tx=0x7f403c009030 comp rx=0 tx=0).stop 2024-03-06T07:25:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 -- 172.21.15.59:0/3431507066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40480cf5f0 msgr2=0x7f40480cfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40480cf5f0 0x7f40480cfa70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f40380119d0 tx=0x7f4038011e90 comp rx=0 tx=0).stop 2024-03-06T07:25:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 -- 172.21.15.59:0/3431507066 shutdown_connections 2024-03-06T07:25:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f402806b500 0x7f402806d9c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f404813f950 0x7f404813fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40480cf5f0 0x7f40480cfa70 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 --2- 172.21.15.59:0/3431507066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40300039e0 0x7f40480aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 -- 172.21.15.59:0/3431507066 >> 172.21.15.59:0/3431507066 conn(0x7f40480b1c80 msgr2=0x7f40480cebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 -- 172.21.15.59:0/3431507066 shutdown_connections 2024-03-06T07:25:43.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.720+0000 7f404c9ae640 1 -- 172.21.15.59:0/3431507066 wait complete. 2024-03-06T07:25:43.736 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:25:43.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.840+0000 7f0adab25640 1 Processor -- start 2024-03-06T07:25:43.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.840+0000 7f0adab25640 1 -- start start 2024-03-06T07:25:43.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0adab25640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ad406b220 0x7f0ad415e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0adab25640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0ad406b600 0x7f0ad4162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0adab25640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad415ea30 con 0x7f0ad40ce920 2024-03-06T07:25:43.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0adab25640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad4161b70 con 0x7f0ad406b600 2024-03-06T07:25:43.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0adab25640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad415ed10 con 0x7f0ad406b220 2024-03-06T07:25:43.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0ad90c2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0ad406b600 0x7f0ad4162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0ad3fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0ad40ce920 0x7f0ad4071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49920/0 (socket says 172.21.15.59:49920) 2024-03-06T07:25:43.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0ad3fff640 1 -- 172.21.15.59:0/1254053867 learned_addr learned my addr 172.21.15.59:0/1254053867 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:43.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0ad88c1640 1 --2- 172.21.15.59:0/1254053867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ad406b220 0x7f0ad415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0ad90c2640 1 -- 172.21.15.59:0/1254053867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ad406b220 msgr2=0x7f0ad415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:43.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0ad90c2640 1 --2- 172.21.15.59:0/1254053867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ad406b220 0x7f0ad415e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0ad90c2640 1 -- 172.21.15.59:0/1254053867 >> v1:172.21.15.59:6789/0 conn(0x7f0ad40ce920 legacy=0x7f0ad4071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0ad90c2640 1 -- 172.21.15.59:0/1254053867 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad415eff0 con 0x7f0ad406b600 2024-03-06T07:25:43.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.841+0000 7f0ad37fe640 1 -- 172.21.15.59:0/1254053867 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197955725 0 0) 0x7f0ad415ea30 con 0x7f0ad40ce920 2024-03-06T07:25:43.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.842+0000 7f0ad90c2640 1 --2- 172.21.15.59:0/1254053867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ad406b600 0x7f0ad4162000 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40097a0 tx=0x7f0ac402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e039711036cc59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.842+0000 7f0ad37fe640 1 -- 172.21.15.59:0/1254053867 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4035020 con 0x7f0ad406b600 2024-03-06T07:25:43.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.842+0000 7f0ad37fe640 1 -- 172.21.15.59:0/1254053867 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac4031830 con 0x7f0ad406b600 2024-03-06T07:25:43.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.842+0000 7f0ad37fe640 1 -- 172.21.15.59:0/1254053867 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4031c00 con 0x7f0ad406b600 2024-03-06T07:25:43.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.843+0000 7f0adab25640 1 -- 172.21.15.59:0/1254053867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ad406b600 msgr2=0x7f0ad4162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.843+0000 7f0adab25640 1 --2- 172.21.15.59:0/1254053867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ad406b600 0x7f0ad4162000 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40097a0 tx=0x7f0ac402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.843+0000 7f0adab25640 1 -- 172.21.15.59:0/1254053867 shutdown_connections 2024-03-06T07:25:43.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.843+0000 7f0adab25640 1 --2- 172.21.15.59:0/1254053867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ad406b600 0x7f0ad4162000 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.843+0000 7f0adab25640 1 --2- 172.21.15.59:0/1254053867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ad406b220 0x7f0ad415e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.843+0000 7f0adab25640 1 -- 172.21.15.59:0/1254053867 >> 172.21.15.59:0/1254053867 conn(0x7f0ad406c0c0 msgr2=0x7f0ad406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:43.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.843+0000 7f0adab25640 1 -- 172.21.15.59:0/1254053867 shutdown_connections 2024-03-06T07:25:43.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.843+0000 7f0adab25640 1 -- 172.21.15.59:0/1254053867 wait complete. 2024-03-06T07:25:43.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.844+0000 7f0adab25640 1 Processor -- start 2024-03-06T07:25:43.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.844+0000 7f0adab25640 1 -- start start 2024-03-06T07:25:43.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0adab25640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ad406b220 0x7f0ad41f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0adab25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ad406b600 0x7f0ad4149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0adab25640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ad40ce920 0x7f0ad4156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0adab25640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4072090 con 0x7f0ad406b600 2024-03-06T07:25:43.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0adab25640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40ce200 con 0x7f0ad406b220 2024-03-06T07:25:43.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0adab25640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad40b2e40 con 0x7f0ad40ce920 2024-03-06T07:25:43.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ad406b600 0x7f0ad4149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ad406b600 0x7f0ad4149440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46366/0 (socket says 172.21.15.59:46366) 2024-03-06T07:25:43.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad3fff640 1 -- 172.21.15.59:0/3106197099 learned_addr learned my addr 172.21.15.59:0/3106197099 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad90c2640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ad40ce920 0x7f0ad4156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad88c1640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ad406b220 0x7f0ad41f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad3fff640 1 -- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ad40ce920 msgr2=0x7f0ad4156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad3fff640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ad40ce920 0x7f0ad4156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad3fff640 1 -- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ad406b220 msgr2=0x7f0ad41f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad3fff640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ad406b220 0x7f0ad41f3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad3fff640 1 -- 172.21.15.59:0/3106197099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad40b3120 con 0x7f0ad406b600 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad90c2640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ad40ce920 0x7f0ad4156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.845+0000 7f0ad3fff640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ad406b600 0x7f0ad4149440 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0ac800c4d0 tx=0x7f0ac800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.846+0000 7f0ad17fa640 1 -- 172.21.15.59:0/3106197099 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8018070 con 0x7f0ad406b600 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.846+0000 7f0ad17fa640 1 -- 172.21.15.59:0/3106197099 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac8007500 con 0x7f0ad406b600 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.846+0000 7f0adab25640 1 -- 172.21.15.59:0/3106197099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad40b3400 con 0x7f0ad406b600 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.846+0000 7f0ad17fa640 1 -- 172.21.15.59:0/3106197099 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8004a20 con 0x7f0ad406b600 2024-03-06T07:25:43.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.846+0000 7f0adab25640 1 -- 172.21.15.59:0/3106197099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad40b39c0 con 0x7f0ad406b600 2024-03-06T07:25:43.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.847+0000 7f0adab25640 1 -- 172.21.15.59:0/3106197099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad404f330 con 0x7f0ad406b600 2024-03-06T07:25:43.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.847+0000 7f0ad17fa640 1 -- 172.21.15.59:0/3106197099 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ac801b070 con 0x7f0ad406b600 2024-03-06T07:25:43.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.847+0000 7f0ad17fa640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ac006b4d0 0x7f0ac006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:43.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.848+0000 7f0ad17fa640 1 -- 172.21.15.59:0/3106197099 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0ac8089290 con 0x7f0ad406b600 2024-03-06T07:25:43.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.848+0000 7f0ad88c1640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ac006b4d0 0x7f0ac006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:43.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.848+0000 7f0ad88c1640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ac006b4d0 0x7f0ac006d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0ad415e5a0 tx=0x7f0acc00b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:43.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:43.852+0000 7f0ad17fa640 1 -- 172.21.15.59:0/3106197099 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ac80583c0 con 0x7f0ad406b600 2024-03-06T07:25:44.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.104+0000 7f0adab25640 1 -- 172.21.15.59:0/3106197099 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0ad40cdf00 con 0x7f0ad406b600 2024-03-06T07:25:44.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.105+0000 7f0ad17fa640 1 -- 172.21.15.59:0/3106197099 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0ac805e0c0 con 0x7f0ad406b600 2024-03-06T07:25:44.106 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:44.106 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlODBZGBAAqV+TH2t2ZKyV9l13QEn6ew==", 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlR1MuJRAAp6OAQ9gZemWtwI1dh3H2SA==", 2024-03-06T07:25:44.107 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlDXW2HxAAnKyuS1GW0wLO98/F9wdOAg==", 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.108 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehl5MO4LBAAKmt2dvKk7BtcCM4o7cSwgg==", 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBXGehlzdinFhAA0ACDAs9dtoY6EhUIveeIvA==", 2024-03-06T07:25:44.109 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGehlH92eFRAANQceHGuQyrbI6gP1m/Gakw==", 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.110 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGehlMWpVFBAAbOG9nScbFlvKHXZMQJoQug==", 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGehlLCyWEhAAyfoW/FmU37QD2IohknBwWw==", 2024-03-06T07:25:44.111 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBmGehldqv5GhAAWEloqqKgdpKPHjt7VKFodw==", 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.112 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGehlc0ENGRAA5V9Az+vxAR3OUwWpDP7w1A==", 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBsGehl5SYEFxAAPLyrHQV9SGguCV9f5VIkYg==", 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.113 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBvGehle3v0FBAA4xUPwX5couppHgi9VwSi2g==", 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA==", 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:44.114 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlqhlAOhAAQ4l3uI8MlcCH+NAIWo/UuA==", 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-03-06T07:25:44.115 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehltL/GNhAAZqSQ+UtwjI/PA95O1RGBNg==", 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGehlltx5BBAAFa/+/w8Hd8lZegfIWtpJOA==", 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:25:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBSGehlMExaNhAAQ2FD7afktMhrTbMMAK1epQ==", 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.117 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlLSTzFBAACeLdHsCX9KJGEsq44aqTyw==", 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlAgqXEBAACCHZ9Lp2lEyF93xD4Ns9Nw==", 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:44.118 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:25:44.119 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.119 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:44.119 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-06T07:25:44.119 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:25:44.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 -- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ac006b4d0 msgr2=0x7f0ac006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ac006b4d0 0x7f0ac006d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0ad415e5a0 tx=0x7f0acc00b040 comp rx=0 tx=0).stop 2024-03-06T07:25:44.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 -- 172.21.15.59:0/3106197099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ad406b600 msgr2=0x7f0ad4149440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ad406b600 0x7f0ad4149440 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f0ac800c4d0 tx=0x7f0ac800c990 comp rx=0 tx=0).stop 2024-03-06T07:25:44.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 -- 172.21.15.59:0/3106197099 shutdown_connections 2024-03-06T07:25:44.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ac006b4d0 0x7f0ac006d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ad40ce920 0x7f0ad4156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ad406b600 0x7f0ad4149440 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 --2- 172.21.15.59:0/3106197099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ad406b220 0x7f0ad41f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 -- 172.21.15.59:0/3106197099 >> 172.21.15.59:0/3106197099 conn(0x7f0ad406c0c0 msgr2=0x7f0ad4071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 -- 172.21.15.59:0/3106197099 shutdown_connections 2024-03-06T07:25:44.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.107+0000 7f0adab25640 1 -- 172.21.15.59:0/3106197099 wait complete. 2024-03-06T07:25:44.121 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:44.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.221+0000 7fa833e8d640 1 Processor -- start 2024-03-06T07:25:44.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.221+0000 7fa833e8d640 1 -- start start 2024-03-06T07:25:44.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.222+0000 7fa833e8d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa82c1621c0 0x7fa82c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.222+0000 7fa833e8d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa82c1625a0 0x7fa82c15dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.222+0000 7fa833e8d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c0ceac0 con 0x7fa82c0d4920 2024-03-06T07:25:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.222+0000 7fa833e8d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82c0cf690 con 0x7fa82c1625a0 2024-03-06T07:25:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.222+0000 7fa833e8d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0d2dc0 con 0x7fa82c1621c0 2024-03-06T07:25:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.222+0000 7fa83242a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa82c1625a0 0x7fa82c15dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.222+0000 7fa831428640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa82c0d4920 0x7fa82c0d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49930/0 (socket says 172.21.15.59:49930) 2024-03-06T07:25:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.222+0000 7fa831428640 1 -- 172.21.15.59:0/2824629960 learned_addr learned my addr 172.21.15.59:0/2824629960 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.222+0000 7fa831c29640 1 --2- 172.21.15.59:0/2824629960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa82c1621c0 0x7fa82c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.222+0000 7fa83242a640 1 -- 172.21.15.59:0/2824629960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa82c1621c0 msgr2=0x7fa82c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.222+0000 7fa83242a640 1 --2- 172.21.15.59:0/2824629960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa82c1621c0 0x7fa82c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.223+0000 7fa83242a640 1 -- 172.21.15.59:0/2824629960 >> v1:172.21.15.59:6789/0 conn(0x7fa82c0d4920 legacy=0x7fa82c0d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.223+0000 7fa83242a640 1 -- 172.21.15.59:0/2824629960 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c0ced60 con 0x7fa82c1625a0 2024-03-06T07:25:44.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.223+0000 7fa830c27640 1 -- 172.21.15.59:0/2824629960 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902850777 0 0) 0x7fa82c0ceac0 con 0x7fa82c0d4920 2024-03-06T07:25:44.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.223+0000 7fa83242a640 1 --2- 172.21.15.59:0/2824629960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa82c1625a0 0x7fa82c15dd40 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fa828009d90 tx=0x7fa82802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e4e760049361eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.223+0000 7fa830c27640 1 -- 172.21.15.59:0/2824629960 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828039070 con 0x7fa82c1625a0 2024-03-06T07:25:44.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.223+0000 7fa830c27640 1 -- 172.21.15.59:0/2824629960 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82802cc50 con 0x7fa82c1625a0 2024-03-06T07:25:44.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.223+0000 7fa830c27640 1 -- 172.21.15.59:0/2824629960 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828034440 con 0x7fa82c1625a0 2024-03-06T07:25:44.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.224+0000 7fa833e8d640 1 -- 172.21.15.59:0/2824629960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa82c1625a0 msgr2=0x7fa82c15dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.224+0000 7fa833e8d640 1 --2- 172.21.15.59:0/2824629960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa82c1625a0 0x7fa82c15dd40 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fa828009d90 tx=0x7fa82802bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.224+0000 7fa833e8d640 1 -- 172.21.15.59:0/2824629960 shutdown_connections 2024-03-06T07:25:44.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.224+0000 7fa833e8d640 1 --2- 172.21.15.59:0/2824629960 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa82c1625a0 0x7fa82c15dd40 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.224+0000 7fa833e8d640 1 --2- 172.21.15.59:0/2824629960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa82c1621c0 0x7fa82c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.224+0000 7fa833e8d640 1 -- 172.21.15.59:0/2824629960 >> 172.21.15.59:0/2824629960 conn(0x7fa82c0d3d10 msgr2=0x7fa82c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.224+0000 7fa833e8d640 1 -- 172.21.15.59:0/2824629960 shutdown_connections 2024-03-06T07:25:44.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.224+0000 7fa833e8d640 1 -- 172.21.15.59:0/2824629960 wait complete. 2024-03-06T07:25:44.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.225+0000 7fa833e8d640 1 Processor -- start 2024-03-06T07:25:44.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.225+0000 7fa833e8d640 1 -- start start 2024-03-06T07:25:44.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa833e8d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa82c0d4920 0x7fa82c0cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa833e8d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa82c1621c0 0x7fa82c1f0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa833e8d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa82c1625a0 0x7fa82c15ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa833e8d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c15e280 con 0x7fa82c1625a0 2024-03-06T07:25:44.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa833e8d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0cf320 con 0x7fa82c1621c0 2024-03-06T07:25:44.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa833e8d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c15e400 con 0x7fa82c0d4920 2024-03-06T07:25:44.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa83242a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa82c1625a0 0x7fa82c15ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa83242a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa82c1625a0 0x7fa82c15ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46392/0 (socket says 172.21.15.59:46392) 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa83242a640 1 -- 172.21.15.59:0/3641885904 learned_addr learned my addr 172.21.15.59:0/3641885904 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa83242a640 1 -- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa82c0d4920 msgr2=0x7fa82c0cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa83242a640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa82c0d4920 0x7fa82c0cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa83242a640 1 -- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa82c1621c0 msgr2=0x7fa82c1f0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa83242a640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa82c1621c0 0x7fa82c1f0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.226+0000 7fa83242a640 1 -- 172.21.15.59:0/3641885904 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c1f4710 con 0x7fa82c1625a0 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.227+0000 7fa83242a640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa82c1625a0 0x7fa82c15ff70 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa828037040 tx=0x7fa828004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.227+0000 7fa8227fc640 1 -- 172.21.15.59:0/3641885904 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828044030 con 0x7fa82c1625a0 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.227+0000 7fa8227fc640 1 -- 172.21.15.59:0/3641885904 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82802ca50 con 0x7fa82c1625a0 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.227+0000 7fa833e8d640 1 -- 172.21.15.59:0/3641885904 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82c0cb030 con 0x7fa82c1625a0 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.227+0000 7fa8227fc640 1 -- 172.21.15.59:0/3641885904 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828039040 con 0x7fa82c1625a0 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.227+0000 7fa833e8d640 1 -- 172.21.15.59:0/3641885904 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82c0cb5f0 con 0x7fa82c1625a0 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.228+0000 7fa833e8d640 1 -- 172.21.15.59:0/3641885904 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82c04f330 con 0x7fa82c1625a0 2024-03-06T07:25:44.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.228+0000 7fa8227fc640 1 -- 172.21.15.59:0/3641885904 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa82803f470 con 0x7fa82c1625a0 2024-03-06T07:25:44.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.228+0000 7fa8227fc640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81006b4d0 0x7fa81006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.229+0000 7fa8227fc640 1 -- 172.21.15.59:0/3641885904 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa8280ac100 con 0x7fa82c1625a0 2024-03-06T07:25:44.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.229+0000 7fa831c29640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81006b4d0 0x7fa81006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.229+0000 7fa831c29640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81006b4d0 0x7fa81006d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0b91e0 tx=0x7fa824008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.233+0000 7fa8227fc640 1 -- 172.21.15.59:0/3641885904 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa82804e060 con 0x7fa82c1625a0 2024-03-06T07:25:44.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.414+0000 7fa833e8d640 1 -- 172.21.15.59:0/3641885904 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa82c073380 con 0x7fa82c1625a0 2024-03-06T07:25:44.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.415+0000 7fa8227fc640 1 -- 172.21.15.59:0/3641885904 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+12880 (secure 0 0 0) 0x7fa82807b360 con 0x7fa82c1625a0 2024-03-06T07:25:44.417 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:44.417 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":77,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:43.731265+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:44.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 -- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81006b4d0 msgr2=0x7fa81006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81006b4d0 0x7fa81006d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0b91e0 tx=0x7fa824008040 comp rx=0 tx=0).stop 2024-03-06T07:25:44.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 -- 172.21.15.59:0/3641885904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa82c1625a0 msgr2=0x7fa82c15ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa82c1625a0 0x7fa82c15ff70 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa828037040 tx=0x7fa828004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:44.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 -- 172.21.15.59:0/3641885904 shutdown_connections 2024-03-06T07:25:44.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81006b4d0 0x7fa81006d990 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa82c1625a0 0x7fa82c15ff70 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa82c1621c0 0x7fa82c1f0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 --2- 172.21.15.59:0/3641885904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa82c0d4920 0x7fa82c0cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 -- 172.21.15.59:0/3641885904 >> 172.21.15.59:0/3641885904 conn(0x7fa82c0d3d10 msgr2=0x7fa82c0d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 -- 172.21.15.59:0/3641885904 shutdown_connections 2024-03-06T07:25:44.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.417+0000 7fa833e8d640 1 -- 172.21.15.59:0/3641885904 wait complete. 2024-03-06T07:25:44.430 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:44.430 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:44.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.532+0000 7f4d35702640 1 Processor -- start 2024-03-06T07:25:44.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.532+0000 7f4d35702640 1 -- start start 2024-03-06T07:25:44.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d35702640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4d3015bb40 0x7f4d300b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d35702640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d3015ff90 con 0x7f4d300cfa90 2024-03-06T07:25:44.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d35702640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d30160270 con 0x7f4d3015f7d0 2024-03-06T07:25:44.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d35702640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d3006c910 con 0x7f4d3015bb40 2024-03-06T07:25:44.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d2f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4d300cfa90 0x7f4d300cdcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49940/0 (socket says 172.21.15.59:49940) 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d2f7fe640 1 -- 172.21.15.59:0/617604664 learned_addr learned my addr 172.21.15.59:0/617604664 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371640221 0 0) 0x7f4d3015ff90 con 0x7f4d300cfa90 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d2effd640 1 --2- 172.21.15.59:0/617604664 >> v2:172.21.15.164:3300/0 conn(0x7f4d3015bb40 0x7f4d300b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d1c003520 con 0x7f4d300cfa90 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517752157 0 0) 0x7f4d1c003520 con 0x7f4d300cfa90 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 >> v2:172.21.15.164:3300/0 conn(0x7f4d3015bb40 msgr2=0x7f4d300b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d2dffb640 1 --2- 172.21.15.59:0/617604664 >> v2:172.21.15.164:3300/0 conn(0x7f4d3015bb40 0x7f4d300b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.533+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 >> v1:172.21.15.164:6789/0 conn(0x7f4d3015f7d0 legacy=0x7f4d3006d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d30160550 con 0x7f4d300cfa90 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f4d200019f0 con 0x7f4d300cfa90 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 >> v1:172.21.15.59:6789/0 conn(0x7f4d300cfa90 legacy=0x7f4d300cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2dffb640 1 --2- 172.21.15.59:0/617604664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0039b0 0x7f4d1c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2dffb640 1 --2- 172.21.15.59:0/617604664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d1c004800 0x7f4d1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2dffb640 1 --2- 172.21.15.59:0/617604664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d1c0051c0 0x7f4d1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3006c910 con 0x7f4d1c0039b0 2024-03-06T07:25:44.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c00c090 con 0x7f4d1c004800 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c00c290 con 0x7f4d1c0051c0 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2effd640 1 --2- 172.21.15.59:0/617604664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0039b0 0x7f4d1c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2e7fc640 1 --2- 172.21.15.59:0/617604664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d1c004800 0x7f4d1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2f7fe640 1 --2- 172.21.15.59:0/617604664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d1c0051c0 0x7f4d1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2effd640 1 -- 172.21.15.59:0/617604664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d1c0051c0 msgr2=0x7f4d1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2effd640 1 --2- 172.21.15.59:0/617604664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d1c0051c0 0x7f4d1c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2effd640 1 -- 172.21.15.59:0/617604664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d1c004800 msgr2=0x7f4d1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2effd640 1 --2- 172.21.15.59:0/617604664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d1c004800 0x7f4d1c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2effd640 1 -- 172.21.15.59:0/617604664 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c003520 con 0x7f4d1c0039b0 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.534+0000 7f4d2f7fe640 1 -- 172.21.15.59:0/617604664 reap_dead start 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.535+0000 7f4d2effd640 1 --2- 172.21.15.59:0/617604664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0039b0 0x7f4d1c003df0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009aa0 tx=0x7f4d1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53523e27ab9589fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.535+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28039070 con 0x7f4d1c0039b0 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.535+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2802c6b0 con 0x7f4d1c0039b0 2024-03-06T07:25:44.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.535+0000 7f4d2dffb640 1 -- 172.21.15.59:0/617604664 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28009670 con 0x7f4d1c0039b0 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.535+0000 7f4d35702640 1 -- 172.21.15.59:0/617604664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0039b0 msgr2=0x7f4d1c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.535+0000 7f4d35702640 1 --2- 172.21.15.59:0/617604664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0039b0 0x7f4d1c003df0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009aa0 tx=0x7f4d1c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.535+0000 7f4d35702640 1 -- 172.21.15.59:0/617604664 shutdown_connections 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.535+0000 7f4d35702640 1 --2- 172.21.15.59:0/617604664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d1c0039b0 0x7f4d1c003df0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.535+0000 7f4d35702640 1 -- 172.21.15.59:0/617604664 >> 172.21.15.59:0/617604664 conn(0x7f4d3006b190 msgr2=0x7f4d3006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.535+0000 7f4d35702640 1 -- 172.21.15.59:0/617604664 shutdown_connections 2024-03-06T07:25:44.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.535+0000 7f4d35702640 1 -- 172.21.15.59:0/617604664 wait complete. 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.537+0000 7f4d35702640 1 Processor -- start 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.537+0000 7f4d35702640 1 -- start start 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.537+0000 7f4d35702640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d1c0039b0 0x7f4d301f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.537+0000 7f4d35702640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3015a150 0x7f4d300cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.537+0000 7f4d35702640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d30157bd0 0x7f4d30158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.537+0000 7f4d35702640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3015bf20 con 0x7f4d3015a150 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.537+0000 7f4d35702640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3015fc30 con 0x7f4d1c0039b0 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.537+0000 7f4d35702640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3015c0a0 con 0x7f4d30157bd0 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3015a150 0x7f4d300cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3015a150 0x7f4d300cc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46404/0 (socket says 172.21.15.59:46404) 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2e7fc640 1 -- 172.21.15.59:0/3381904543 learned_addr learned my addr 172.21.15.59:0/3381904543 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:44.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2effd640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d1c0039b0 0x7f4d301f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2f7fe640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d30157bd0 0x7f4d30158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2e7fc640 1 -- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d30157bd0 msgr2=0x7f4d30158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2e7fc640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d30157bd0 0x7f4d30158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2e7fc640 1 -- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d1c0039b0 msgr2=0x7f4d301f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2e7fc640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d1c0039b0 0x7f4d301f41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2e7fc640 1 -- 172.21.15.59:0/3381904543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d300cc780 con 0x7f4d3015a150 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2f7fe640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d30157bd0 0x7f4d30158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d2e7fc640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3015a150 0x7f4d300cc240 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800f850 tx=0x7f4d1800fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d277fe640 1 -- 172.21.15.59:0/3381904543 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1800fe40 con 0x7f4d3015a150 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d277fe640 1 -- 172.21.15.59:0/3381904543 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d18018070 con 0x7f4d3015a150 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d35702640 1 -- 172.21.15.59:0/3381904543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d301f47d0 con 0x7f4d3015a150 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d277fe640 1 -- 172.21.15.59:0/3381904543 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1800d6d0 con 0x7f4d3015a150 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.538+0000 7f4d35702640 1 -- 172.21.15.59:0/3381904543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d301f4d90 con 0x7f4d3015a150 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.539+0000 7f4d35702640 1 -- 172.21.15.59:0/3381904543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d300732b0 con 0x7f4d3015a150 2024-03-06T07:25:44.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.540+0000 7f4d277fe640 1 -- 172.21.15.59:0/3381904543 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d1800d870 con 0x7f4d3015a150 2024-03-06T07:25:44.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.540+0000 7f4d277fe640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1006b4d0 0x7f4d1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.540+0000 7f4d277fe640 1 -- 172.21.15.59:0/3381904543 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4d18089b90 con 0x7f4d3015a150 2024-03-06T07:25:44.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.540+0000 7f4d2effd640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1006b4d0 0x7f4d1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.541+0000 7f4d2effd640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1006b4d0 0x7f4d1006d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009aa0 tx=0x7f4d28030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.544+0000 7f4d277fe640 1 -- 172.21.15.59:0/3381904543 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d300cc780 con 0x7f4d3015a150 2024-03-06T07:25:44.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.728+0000 7f4d35702640 1 -- 172.21.15.59:0/3381904543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4d30068ff0 con 0x7f4d3015a150 2024-03-06T07:25:44.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.728+0000 7f4d277fe640 1 -- 172.21.15.59:0/3381904543 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4d18058d60 con 0x7f4d3015a150 2024-03-06T07:25:44.730 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:25:44.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 -- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1006b4d0 msgr2=0x7f4d1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1006b4d0 0x7f4d1006d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009aa0 tx=0x7f4d28030000 comp rx=0 tx=0).stop 2024-03-06T07:25:44.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 -- 172.21.15.59:0/3381904543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3015a150 msgr2=0x7f4d300cc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3015a150 0x7f4d300cc240 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800f850 tx=0x7f4d1800fd10 comp rx=0 tx=0).stop 2024-03-06T07:25:44.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 -- 172.21.15.59:0/3381904543 shutdown_connections 2024-03-06T07:25:44.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d1006b4d0 0x7f4d1006d990 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d30157bd0 0x7f4d30158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d3015a150 0x7f4d300cc240 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 --2- 172.21.15.59:0/3381904543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d1c0039b0 0x7f4d301f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 -- 172.21.15.59:0/3381904543 >> 172.21.15.59:0/3381904543 conn(0x7f4d3006b190 msgr2=0x7f4d300b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 -- 172.21.15.59:0/3381904543 shutdown_connections 2024-03-06T07:25:44.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.730+0000 7f4d35702640 1 -- 172.21.15.59:0/3381904543 wait complete. 2024-03-06T07:25:44.743 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:25:44.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.843+0000 7fe1d6640640 1 Processor -- start 2024-03-06T07:25:44.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.843+0000 7fe1d6640640 1 -- start start 2024-03-06T07:25:44.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1d6640640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe1d00d00c0 0x7fe1d00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1d6640640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe1d00cf850 0x7fe1d00cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1d6640640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d00d09e0 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1d6640640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d006ca90 con 0x7fe1d00cf850 2024-03-06T07:25:44.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1d6640640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d006df30 con 0x7fe1d00d00c0 2024-03-06T07:25:44.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1cf7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe1d00cf850 0x7fe1d00cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1d4bdd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe1d00ce1b0 0x7fe1d00ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49952/0 (socket says 172.21.15.59:49952) 2024-03-06T07:25:44.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1d4bdd640 1 -- 172.21.15.59:0/1745125053 learned_addr learned my addr 172.21.15.59:0/1745125053 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:44.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1cf7fe640 1 -- 172.21.15.59:0/1745125053 >> v2:172.21.15.164:3300/0 conn(0x7fe1d00d00c0 msgr2=0x7fe1d00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:44.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1cffff640 1 --2- 172.21.15.59:0/1745125053 >> v2:172.21.15.164:3300/0 conn(0x7fe1d00d00c0 0x7fe1d00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1ceffd640 1 -- 172.21.15.59:0/1745125053 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231579861 0 0) 0x7fe1d00d09e0 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1cf7fe640 1 --2- 172.21.15.59:0/1745125053 >> v2:172.21.15.164:3300/0 conn(0x7fe1d00d00c0 0x7fe1d00d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1cf7fe640 1 -- 172.21.15.59:0/1745125053 >> v1:172.21.15.59:6789/0 conn(0x7fe1d00ce1b0 legacy=0x7fe1d00ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.844+0000 7fe1cf7fe640 1 -- 172.21.15.59:0/1745125053 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d015e380 con 0x7fe1d00cf850 2024-03-06T07:25:44.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.845+0000 7fe1cf7fe640 1 --2- 172.21.15.59:0/1745125053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d00cf850 0x7fe1d00cd3c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00d1a40 tx=0x7fe1c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b2843d1348cdf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.845+0000 7fe1ceffd640 1 -- 172.21.15.59:0/1745125053 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4035020 con 0x7fe1d00cf850 2024-03-06T07:25:44.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.845+0000 7fe1ceffd640 1 -- 172.21.15.59:0/1745125053 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c4031830 con 0x7fe1d00cf850 2024-03-06T07:25:44.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.845+0000 7fe1ceffd640 1 -- 172.21.15.59:0/1745125053 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4031c00 con 0x7fe1d00cf850 2024-03-06T07:25:44.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.845+0000 7fe1d6640640 1 -- 172.21.15.59:0/1745125053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d00cf850 msgr2=0x7fe1d00cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.845+0000 7fe1d6640640 1 --2- 172.21.15.59:0/1745125053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d00cf850 0x7fe1d00cd3c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00d1a40 tx=0x7fe1c402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.846+0000 7fe1d6640640 1 -- 172.21.15.59:0/1745125053 shutdown_connections 2024-03-06T07:25:44.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.846+0000 7fe1d6640640 1 --2- 172.21.15.59:0/1745125053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d00cf850 0x7fe1d00cd3c0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.846+0000 7fe1d6640640 1 --2- 172.21.15.59:0/1745125053 >> v2:172.21.15.164:3300/0 conn(0x7fe1d00d00c0 0x7fe1d00d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.846+0000 7fe1d6640640 1 -- 172.21.15.59:0/1745125053 >> 172.21.15.59:0/1745125053 conn(0x7fe1d00b35a0 msgr2=0x7fe1d00b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:44.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.846+0000 7fe1d6640640 1 -- 172.21.15.59:0/1745125053 shutdown_connections 2024-03-06T07:25:44.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.846+0000 7fe1d6640640 1 -- 172.21.15.59:0/1745125053 wait complete. 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.847+0000 7fe1d6640640 1 Processor -- start 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.847+0000 7fe1d6640640 1 -- start start 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.847+0000 7fe1d6640640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d00ce1b0 0x7fe1d014ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.847+0000 7fe1d6640640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d00cf850 0x7fe1d006ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1d6640640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1d00d00c0 0x7fe1d006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1d6640640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d006c440 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1d6640640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d006c160 con 0x7fe1d00cf850 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1d6640640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d006c720 con 0x7fe1d00d00c0 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1cffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d00ce1b0 0x7fe1d014ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1cffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d00ce1b0 0x7fe1d014ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46420/0 (socket says 172.21.15.59:46420) 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1cffff640 1 -- 172.21.15.59:0/1306979182 learned_addr learned my addr 172.21.15.59:0/1306979182 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1d4bdd640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1d00d00c0 0x7fe1d006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1cf7fe640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d00cf850 0x7fe1d006ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1cffff640 1 -- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1d00d00c0 msgr2=0x7fe1d006d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1cffff640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1d00d00c0 0x7fe1d006d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1cffff640 1 -- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d00cf850 msgr2=0x7fe1d006ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1cffff640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d00cf850 0x7fe1d006ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1cffff640 1 -- 172.21.15.59:0/1306979182 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d014fa50 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1cf7fe640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d00cf850 0x7fe1d006ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1d4bdd640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1d00d00c0 0x7fe1d006d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1cffff640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d00ce1b0 0x7fe1d014ef70 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c820 tx=0x7fe1c000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1ccff9640 1 -- 172.21.15.59:0/1306979182 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c0015070 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1ccff9640 1 -- 172.21.15.59:0/1306979182 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c0004020 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1d6640640 1 -- 172.21.15.59:0/1306979182 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d014fd10 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1ccff9640 1 -- 172.21.15.59:0/1306979182 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00043f0 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.848+0000 7fe1d6640640 1 -- 172.21.15.59:0/1306979182 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d01502d0 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.849+0000 7fe1d6640640 1 -- 172.21.15.59:0/1306979182 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d00cd6a0 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.850+0000 7fe1ccff9640 1 -- 172.21.15.59:0/1306979182 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1c0007830 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.850+0000 7fe1ccff9640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.850+0000 7fe1ccff9640 1 -- 172.21.15.59:0/1306979182 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe1c0089c70 con 0x7fe1d00ce1b0 2024-03-06T07:25:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.850+0000 7fe1cf7fe640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:44.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.851+0000 7fe1cf7fe640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4036000 tx=0x7fe1c4008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:44.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:44.854+0000 7fe1ccff9640 1 -- 172.21.15.59:0/1306979182 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1d00cd6a0 con 0x7fe1d00ce1b0 2024-03-06T07:25:45.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.038+0000 7fe1d6640640 1 -- 172.21.15.59:0/1306979182 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe1d00d08b0 con 0x7fe1d00ce1b0 2024-03-06T07:25:45.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.038+0000 7fe1ccff9640 1 -- 172.21.15.59:0/1306979182 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe1c0058e40 con 0x7fe1d00ce1b0 2024-03-06T07:25:45.040 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:45.041 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-03-06T07:25:45.041 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:45.041 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-03-06T07:25:45.041 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-03-06T07:25:45.041 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:45.041 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-03-06T07:25:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 -- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 msgr2=0x7fe1bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4036000 tx=0x7fe1c4008030 comp rx=0 tx=0).stop 2024-03-06T07:25:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 -- 172.21.15.59:0/1306979182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d00ce1b0 msgr2=0x7fe1d014ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d00ce1b0 0x7fe1d014ef70 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c820 tx=0x7fe1c000cce0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 -- 172.21.15.59:0/1306979182 shutdown_connections 2024-03-06T07:25:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1d00d00c0 0x7fe1d006d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d00cf850 0x7fe1d006ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 --2- 172.21.15.59:0/1306979182 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d00ce1b0 0x7fe1d014ef70 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 -- 172.21.15.59:0/1306979182 >> 172.21.15.59:0/1306979182 conn(0x7fe1d00b35a0 msgr2=0x7fe1d006fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 -- 172.21.15.59:0/1306979182 shutdown_connections 2024-03-06T07:25:45.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.041+0000 7fe1d6640640 1 -- 172.21.15.59:0/1306979182 wait complete. 2024-03-06T07:25:45.053 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:25:45.053 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:25:45.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.156+0000 7f8b41688640 1 Processor -- start 2024-03-06T07:25:45.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.156+0000 7f8b41688640 1 -- start start 2024-03-06T07:25:45.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.157+0000 7f8b41688640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8b3c0d1560 0x7f8b3c0cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.157+0000 7f8b41688640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8b3c06b6c0 0x7f8b3c167b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.157+0000 7f8b41688640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c0b0390 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.157+0000 7f8b41688640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c0cdba0 con 0x7f8b3c06b6c0 2024-03-06T07:25:45.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.157+0000 7f8b41688640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c0b00b0 con 0x7f8b3c0d1560 2024-03-06T07:25:45.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.157+0000 7f8b3b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8b3c06b6c0 0x7f8b3c167b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.157+0000 7f8b3b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8b3c06b6c0 0x7f8b3c167b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46432/0 (socket says 172.21.15.59:46432) 2024-03-06T07:25:45.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.157+0000 7f8b3b7fe640 1 -- 172.21.15.59:0/591676807 learned_addr learned my addr 172.21.15.59:0/591676807 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:45.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.158+0000 7f8b3b7fe640 1 -- 172.21.15.59:0/591676807 >> v2:172.21.15.164:3300/0 conn(0x7f8b3c0d1560 msgr2=0x7f8b3c0cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.158+0000 7f8b3a7fc640 1 --2- 172.21.15.59:0/591676807 >> v2:172.21.15.164:3300/0 conn(0x7f8b3c0d1560 0x7f8b3c0cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.158+0000 7f8b3b7fe640 1 --2- 172.21.15.59:0/591676807 >> v2:172.21.15.164:3300/0 conn(0x7f8b3c0d1560 0x7f8b3c0cfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.158+0000 7f8b3b7fe640 1 -- 172.21.15.59:0/591676807 >> v1:172.21.15.164:6789/0 conn(0x7f8b3c15e1c0 legacy=0x7f8b3c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.158+0000 7f8b3b7fe640 1 -- 172.21.15.59:0/591676807 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0cd550 con 0x7f8b3c06b6c0 2024-03-06T07:25:45.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.158+0000 7f8b3a7fc640 1 --2- 172.21.15.59:0/591676807 >> v2:172.21.15.164:3300/0 conn(0x7f8b3c0d1560 0x7f8b3c0cfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:45.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.158+0000 7f8b3b7fe640 1 --2- 172.21.15.59:0/591676807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b3c06b6c0 0x7f8b3c167b80 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c0b0560 tx=0x7f8b3402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412a126b5d98e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.158+0000 7f8b39ffb640 1 -- 172.21.15.59:0/591676807 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34038070 con 0x7f8b3c06b6c0 2024-03-06T07:25:45.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.158+0000 7f8b39ffb640 1 -- 172.21.15.59:0/591676807 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3402f050 con 0x7f8b3c06b6c0 2024-03-06T07:25:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.159+0000 7f8b39ffb640 1 -- 172.21.15.59:0/591676807 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34031970 con 0x7f8b3c06b6c0 2024-03-06T07:25:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.159+0000 7f8b41688640 1 -- 172.21.15.59:0/591676807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b3c06b6c0 msgr2=0x7f8b3c167b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.159+0000 7f8b41688640 1 --2- 172.21.15.59:0/591676807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b3c06b6c0 0x7f8b3c167b80 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c0b0560 tx=0x7f8b3402b7f0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.159+0000 7f8b41688640 1 -- 172.21.15.59:0/591676807 shutdown_connections 2024-03-06T07:25:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.159+0000 7f8b41688640 1 --2- 172.21.15.59:0/591676807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b3c06b6c0 0x7f8b3c167b80 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.159+0000 7f8b41688640 1 --2- 172.21.15.59:0/591676807 >> v2:172.21.15.164:3300/0 conn(0x7f8b3c0d1560 0x7f8b3c0cfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.159+0000 7f8b41688640 1 -- 172.21.15.59:0/591676807 >> 172.21.15.59:0/591676807 conn(0x7f8b3c0d09b0 msgr2=0x7f8b3c0d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.159+0000 7f8b41688640 1 -- 172.21.15.59:0/591676807 shutdown_connections 2024-03-06T07:25:45.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.159+0000 7f8b41688640 1 -- 172.21.15.59:0/591676807 wait complete. 2024-03-06T07:25:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.160+0000 7f8b41688640 1 Processor -- start 2024-03-06T07:25:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.160+0000 7f8b41688640 1 -- start start 2024-03-06T07:25:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b41688640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b3c06b6c0 0x7f8b3c153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b41688640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b3c0d1560 0x7f8b3c160af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b41688640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b3c15e1c0 0x7f8b3c161030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b41688640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0afd30 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b41688640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0afbb0 con 0x7f8b3c06b6c0 2024-03-06T07:25:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b41688640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c0ce030 con 0x7f8b3c0d1560 2024-03-06T07:25:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b3b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b3c15e1c0 0x7f8b3c161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b3b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b3c15e1c0 0x7f8b3c161030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46434/0 (socket says 172.21.15.59:46434) 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b3b7fe640 1 -- 172.21.15.59:0/2408934467 learned_addr learned my addr 172.21.15.59:0/2408934467 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b3affd640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b3c06b6c0 0x7f8b3c153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b3a7fc640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b3c0d1560 0x7f8b3c160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b3b7fe640 1 -- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b3c0d1560 msgr2=0x7f8b3c160af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b3b7fe640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b3c0d1560 0x7f8b3c160af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b3b7fe640 1 -- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b3c06b6c0 msgr2=0x7f8b3c153470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b3b7fe640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b3c06b6c0 0x7f8b3c153470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b3b7fe640 1 -- 172.21.15.59:0/2408934467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c15f340 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.161+0000 7f8b3affd640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b3c06b6c0 0x7f8b3c153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.162+0000 7f8b3b7fe640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b3c15e1c0 0x7f8b3c161030 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8b340090b0 tx=0x7f8b340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.162+0000 7f8b337fe640 1 -- 172.21.15.59:0/2408934467 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34042070 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.162+0000 7f8b337fe640 1 -- 172.21.15.59:0/2408934467 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3403d890 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.162+0000 7f8b41688640 1 -- 172.21.15.59:0/2408934467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c15f620 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.162+0000 7f8b337fe640 1 -- 172.21.15.59:0/2408934467 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34038040 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.162+0000 7f8b41688640 1 -- 172.21.15.59:0/2408934467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c15fbe0 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.163+0000 7f8b41688640 1 -- 172.21.15.59:0/2408934467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c04f330 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.163+0000 7f8b337fe640 1 -- 172.21.15.59:0/2408934467 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b34045070 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.164+0000 7f8b337fe640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.164+0000 7f8b337fe640 1 -- 172.21.15.59:0/2408934467 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8b340aba60 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.164+0000 7f8b3affd640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.164+0000 7f8b3affd640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c1f75c0 tx=0x7f8b2c009570 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.168+0000 7f8b337fe640 1 -- 172.21.15.59:0/2408934467 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b3407ac30 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.348+0000 7f8b41688640 1 -- 172.21.15.59:0/2408934467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8b3c0ce450 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.717+0000 7f8b337fe640 1 -- 172.21.15.59:0/2408934467 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8b3407a530 con 0x7f8b3c15e1c0 2024-03-06T07:25:45.719 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-03-06T07:25:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 -- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b1c06b4d0 msgr2=0x7f8b1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c1f75c0 tx=0x7f8b2c009570 comp rx=0 tx=0).stop 2024-03-06T07:25:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 -- 172.21.15.59:0/2408934467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b3c15e1c0 msgr2=0x7f8b3c161030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b3c15e1c0 0x7f8b3c161030 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8b340090b0 tx=0x7f8b340046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 -- 172.21.15.59:0/2408934467 shutdown_connections 2024-03-06T07:25:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b1c06b4d0 0x7f8b1c06d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b3c15e1c0 0x7f8b3c161030 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b3c0d1560 0x7f8b3c160af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 --2- 172.21.15.59:0/2408934467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b3c06b6c0 0x7f8b3c153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 -- 172.21.15.59:0/2408934467 >> 172.21.15.59:0/2408934467 conn(0x7f8b3c0d09b0 msgr2=0x7f8b3c0b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 -- 172.21.15.59:0/2408934467 shutdown_connections 2024-03-06T07:25:45.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.720+0000 7f8b41688640 1 -- 172.21.15.59:0/2408934467 wait complete. 2024-03-06T07:25:45.736 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:25:45.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.844+0000 7fdbeacd6640 1 Processor -- start 2024-03-06T07:25:45.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.844+0000 7fdbeacd6640 1 -- start start 2024-03-06T07:25:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.845+0000 7fdbeacd6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdbe40cfa90 0x7fdbe40cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.845+0000 7fdbeacd6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe4158ac0 con 0x7fdbe41586e0 2024-03-06T07:25:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.845+0000 7fdbeacd6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe415b2a0 con 0x7fdbe40b33a0 2024-03-06T07:25:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.845+0000 7fdbeacd6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe415b540 con 0x7fdbe40cfa90 2024-03-06T07:25:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.845+0000 7fdbdbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdbe40cfa90 0x7fdbe40cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.845+0000 7fdbe8a72640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdbe41586e0 0x7fdbe40aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49958/0 (socket says 172.21.15.59:49958) 2024-03-06T07:25:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.845+0000 7fdbe8a72640 1 -- 172.21.15.59:0/1946927346 learned_addr learned my addr 172.21.15.59:0/1946927346 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.845+0000 7fdbdbfff640 1 -- 172.21.15.59:0/1946927346 >> v1:172.21.15.164:6789/0 conn(0x7fdbe40b33a0 legacy=0x7fdbe40b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.845+0000 7fdbdbfff640 1 -- 172.21.15.59:0/1946927346 >> v1:172.21.15.59:6789/0 conn(0x7fdbe41586e0 legacy=0x7fdbe40aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.845+0000 7fdbdbfff640 1 -- 172.21.15.59:0/1946927346 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe415b820 con 0x7fdbe40cfa90 2024-03-06T07:25:45.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.845+0000 7fdbdb7fe640 1 -- 172.21.15.59:0/1946927346 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246315808 0 0) 0x7fdbe4158ac0 con 0x7fdbe41586e0 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.846+0000 7fdbdbfff640 1 --2- 172.21.15.59:0/1946927346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbe40cfa90 0x7fdbe40cff40 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40097a0 tx=0x7fdbd402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1280b5ff370824 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.846+0000 7fdbdb7fe640 1 -- 172.21.15.59:0/1946927346 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4035020 con 0x7fdbe40cfa90 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.846+0000 7fdbdb7fe640 1 -- 172.21.15.59:0/1946927346 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd4031830 con 0x7fdbe40cfa90 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.846+0000 7fdbdb7fe640 1 -- 172.21.15.59:0/1946927346 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4031c00 con 0x7fdbe40cfa90 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.847+0000 7fdbeacd6640 1 -- 172.21.15.59:0/1946927346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbe40cfa90 msgr2=0x7fdbe40cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.847+0000 7fdbeacd6640 1 --2- 172.21.15.59:0/1946927346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbe40cfa90 0x7fdbe40cff40 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40097a0 tx=0x7fdbd402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.847+0000 7fdbeacd6640 1 -- 172.21.15.59:0/1946927346 shutdown_connections 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.847+0000 7fdbeacd6640 1 --2- 172.21.15.59:0/1946927346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbe40cfa90 0x7fdbe40cff40 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.847+0000 7fdbeacd6640 1 -- 172.21.15.59:0/1946927346 >> 172.21.15.59:0/1946927346 conn(0x7fdbe406b190 msgr2=0x7fdbe406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.847+0000 7fdbeacd6640 1 -- 172.21.15.59:0/1946927346 shutdown_connections 2024-03-06T07:25:45.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.847+0000 7fdbeacd6640 1 -- 172.21.15.59:0/1946927346 wait complete. 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.848+0000 7fdbeacd6640 1 Processor -- start 2024-03-06T07:25:45.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.848+0000 7fdbeacd6640 1 -- start start 2024-03-06T07:25:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbeacd6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbe40b33a0 0x7fdbe41ef660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbeacd6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbe40cfa90 0x7fdbe40ab7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbeacd6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbe41586e0 0x7fdbe41f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbeacd6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe40af150 con 0x7fdbe40cfa90 2024-03-06T07:25:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbeacd6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe415bde0 con 0x7fdbe40b33a0 2024-03-06T07:25:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbeacd6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe4071350 con 0x7fdbe41586e0 2024-03-06T07:25:45.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbdbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbe40cfa90 0x7fdbe40ab7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbdbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbe40cfa90 0x7fdbe40ab7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46448/0 (socket says 172.21.15.59:46448) 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbdbfff640 1 -- 172.21.15.59:0/114762156 learned_addr learned my addr 172.21.15.59:0/114762156 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbe8a72640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbe40b33a0 0x7fdbe41ef660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbe9273640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbe41586e0 0x7fdbe41f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbdbfff640 1 -- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbe41586e0 msgr2=0x7fdbe41f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbdbfff640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbe41586e0 0x7fdbe41f2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbdbfff640 1 -- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbe40b33a0 msgr2=0x7fdbe41ef660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbdbfff640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbe40b33a0 0x7fdbe41ef660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbdbfff640 1 -- 172.21.15.59:0/114762156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe41f57c0 con 0x7fdbe40cfa90 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbe9273640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbe41586e0 0x7fdbe41f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbdbfff640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbe40cfa90 0x7fdbe40ab7f0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fdbd402bee0 tx=0x7fdbd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.849+0000 7fdbd97fa640 1 -- 172.21.15.59:0/114762156 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd403ba80 con 0x7fdbe40cfa90 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.850+0000 7fdbd97fa640 1 -- 172.21.15.59:0/114762156 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd403bc20 con 0x7fdbe40cfa90 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.850+0000 7fdbeacd6640 1 -- 172.21.15.59:0/114762156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe41f5aa0 con 0x7fdbe40cfa90 2024-03-06T07:25:45.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.850+0000 7fdbd97fa640 1 -- 172.21.15.59:0/114762156 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40453f0 con 0x7fdbe40cfa90 2024-03-06T07:25:45.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.850+0000 7fdbeacd6640 1 -- 172.21.15.59:0/114762156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe40ac160 con 0x7fdbe40cfa90 2024-03-06T07:25:45.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.850+0000 7fdbeacd6640 1 -- 172.21.15.59:0/114762156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe40732b0 con 0x7fdbe40cfa90 2024-03-06T07:25:45.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.851+0000 7fdbd97fa640 1 -- 172.21.15.59:0/114762156 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbd4038020 con 0x7fdbe40cfa90 2024-03-06T07:25:45.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.851+0000 7fdbd97fa640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd006b4d0 0x7fdbd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.852+0000 7fdbd97fa640 1 -- 172.21.15.59:0/114762156 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fdbd40abd50 con 0x7fdbe40cfa90 2024-03-06T07:25:45.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.852+0000 7fdbe8a72640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd006b4d0 0x7fdbd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.852+0000 7fdbe8a72640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd006b4d0 0x7fdbd006d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0007fd0 tx=0x7fdbe0007de0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:45.856+0000 7fdbd97fa640 1 -- 172.21.15.59:0/114762156 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbd4040070 con 0x7fdbe40cfa90 2024-03-06T07:25:46.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.036+0000 7fdbeacd6640 1 -- 172.21.15.59:0/114762156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdbe40abf40 con 0x7fdbe40cfa90 2024-03-06T07:25:46.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.720+0000 7fdbd97fa640 1 -- 172.21.15.59:0/114762156 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) v1 ==== 170+0+0 (secure 0 0 0) 0x7fdbd407a750 con 0x7fdbe40cfa90 2024-03-06T07:25:46.722 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-03-06T07:25:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.722+0000 7fdbeacd6640 1 -- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd006b4d0 msgr2=0x7fdbd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.722+0000 7fdbeacd6640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd006b4d0 0x7fdbd006d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0007fd0 tx=0x7fdbe0007de0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.722+0000 7fdbeacd6640 1 -- 172.21.15.59:0/114762156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbe40cfa90 msgr2=0x7fdbe40ab7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.722+0000 7fdbeacd6640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbe40cfa90 0x7fdbe40ab7f0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fdbd402bee0 tx=0x7fdbd40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.723+0000 7fdbeacd6640 1 -- 172.21.15.59:0/114762156 shutdown_connections 2024-03-06T07:25:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.723+0000 7fdbeacd6640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd006b4d0 0x7fdbd006d990 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.723+0000 7fdbeacd6640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbe41586e0 0x7fdbe41f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.723+0000 7fdbeacd6640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbe40cfa90 0x7fdbe40ab7f0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.723+0000 7fdbeacd6640 1 --2- 172.21.15.59:0/114762156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbe40b33a0 0x7fdbe41ef660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.723+0000 7fdbeacd6640 1 -- 172.21.15.59:0/114762156 >> 172.21.15.59:0/114762156 conn(0x7fdbe406b190 msgr2=0x7fdbe41558f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.723+0000 7fdbeacd6640 1 -- 172.21.15.59:0/114762156 shutdown_connections 2024-03-06T07:25:46.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.723+0000 7fdbeacd6640 1 -- 172.21.15.59:0/114762156 wait complete. 2024-03-06T07:25:46.749 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:25:46.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.857+0000 7fcb931e2640 1 Processor -- start 2024-03-06T07:25:46.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.857+0000 7fcb931e2640 1 -- start start 2024-03-06T07:25:46.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb931e2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcb8c06cc90 0x7fcb8c06d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb931e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c06c290 0x7fcb8c06fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb931e2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb8c06d5b0 con 0x7fcb8c0b5ee0 2024-03-06T07:25:46.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb931e2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb8c06ebf0 con 0x7fcb8c06cc90 2024-03-06T07:25:46.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb931e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c0b6430 con 0x7fcb8c06c290 2024-03-06T07:25:46.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb90f7e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcb8c06cc90 0x7fcb8c06d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb90f7e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcb8c06cc90 0x7fcb8c06d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46460/0 (socket says 172.21.15.59:46460) 2024-03-06T07:25:46.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb90f7e640 1 -- 172.21.15.59:0/2599163901 learned_addr learned my addr 172.21.15.59:0/2599163901 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:46.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb9177f640 1 --2- 172.21.15.59:0/2599163901 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c06c290 0x7fcb8c06fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb90f7e640 1 -- 172.21.15.59:0/2599163901 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c06c290 msgr2=0x7fcb8c06fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:46.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb90f7e640 1 --2- 172.21.15.59:0/2599163901 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c06c290 0x7fcb8c06fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb90f7e640 1 -- 172.21.15.59:0/2599163901 >> v1:172.21.15.164:6789/0 conn(0x7fcb8c0b5ee0 legacy=0x7fcb8c06f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:46.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.858+0000 7fcb90f7e640 1 -- 172.21.15.59:0/2599163901 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c0b6710 con 0x7fcb8c06cc90 2024-03-06T07:25:46.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.859+0000 7fcb90f7e640 1 --2- 172.21.15.59:0/2599163901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c06cc90 0x7fcb8c06d070 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fcb84009e50 tx=0x7fcb8402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea6d514db471e01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.859+0000 7fcb8b7fe640 1 -- 172.21.15.59:0/2599163901 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8402cb00 con 0x7fcb8c06cc90 2024-03-06T07:25:46.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.859+0000 7fcb8b7fe640 1 -- 172.21.15.59:0/2599163901 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8402cca0 con 0x7fcb8c06cc90 2024-03-06T07:25:46.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.859+0000 7fcb8b7fe640 1 -- 172.21.15.59:0/2599163901 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb84034610 con 0x7fcb8c06cc90 2024-03-06T07:25:46.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.860+0000 7fcb931e2640 1 -- 172.21.15.59:0/2599163901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c06cc90 msgr2=0x7fcb8c06d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.860+0000 7fcb931e2640 1 --2- 172.21.15.59:0/2599163901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c06cc90 0x7fcb8c06d070 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fcb84009e50 tx=0x7fcb8402bf60 comp rx=0 tx=0).stop 2024-03-06T07:25:46.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.860+0000 7fcb931e2640 1 -- 172.21.15.59:0/2599163901 shutdown_connections 2024-03-06T07:25:46.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.860+0000 7fcb931e2640 1 --2- 172.21.15.59:0/2599163901 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c06c290 0x7fcb8c06fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.860+0000 7fcb931e2640 1 --2- 172.21.15.59:0/2599163901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c06cc90 0x7fcb8c06d070 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.860+0000 7fcb931e2640 1 -- 172.21.15.59:0/2599163901 >> 172.21.15.59:0/2599163901 conn(0x7fcb8c0b6ea0 msgr2=0x7fcb8c0b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.860+0000 7fcb931e2640 1 -- 172.21.15.59:0/2599163901 shutdown_connections 2024-03-06T07:25:46.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.860+0000 7fcb931e2640 1 -- 172.21.15.59:0/2599163901 wait complete. 2024-03-06T07:25:46.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.861+0000 7fcb931e2640 1 Processor -- start 2024-03-06T07:25:46.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.861+0000 7fcb931e2640 1 -- start start 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb931e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c06c290 0x7fcb8c1494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb931e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c06cc90 0x7fcb8c0ce480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb931e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0b5ee0 0x7fcb8c09e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb931e2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c06c850 con 0x7fcb8c0b5ee0 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb931e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c06f120 con 0x7fcb8c06cc90 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb931e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8c06c9d0 con 0x7fcb8c06c290 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb9177f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0b5ee0 0x7fcb8c09e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb9177f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0b5ee0 0x7fcb8c09e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46462/0 (socket says 172.21.15.59:46462) 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb9177f640 1 -- 172.21.15.59:0/3882568553 learned_addr learned my addr 172.21.15.59:0/3882568553 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb90f7e640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c06c290 0x7fcb8c1494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb8bfff640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c06cc90 0x7fcb8c0ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb9177f640 1 -- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c06c290 msgr2=0x7fcb8c1494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb9177f640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c06c290 0x7fcb8c1494f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb9177f640 1 -- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c06cc90 msgr2=0x7fcb8c0ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb9177f640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c06cc90 0x7fcb8c0ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.862+0000 7fcb9177f640 1 -- 172.21.15.59:0/3882568553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c09d900 con 0x7fcb8c0b5ee0 2024-03-06T07:25:46.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.863+0000 7fcb9177f640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0b5ee0 0x7fcb8c09e9a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00d800 tx=0x7fcb7c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.863+0000 7fcb897fa640 1 -- 172.21.15.59:0/3882568553 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c00b6c0 con 0x7fcb8c0b5ee0 2024-03-06T07:25:46.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.863+0000 7fcb897fa640 1 -- 172.21.15.59:0/3882568553 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7c019070 con 0x7fcb8c0b5ee0 2024-03-06T07:25:46.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.863+0000 7fcb897fa640 1 -- 172.21.15.59:0/3882568553 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c00bac0 con 0x7fcb8c0b5ee0 2024-03-06T07:25:46.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.863+0000 7fcb931e2640 1 -- 172.21.15.59:0/3882568553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8c09dbe0 con 0x7fcb8c0b5ee0 2024-03-06T07:25:46.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.863+0000 7fcb931e2640 1 -- 172.21.15.59:0/3882568553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8c09e1a0 con 0x7fcb8c0b5ee0 2024-03-06T07:25:46.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.864+0000 7fcb931e2640 1 -- 172.21.15.59:0/3882568553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8c04f330 con 0x7fcb8c0b5ee0 2024-03-06T07:25:46.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.864+0000 7fcb897fa640 1 -- 172.21.15.59:0/3882568553 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb7c00bc60 con 0x7fcb8c0b5ee0 2024-03-06T07:25:46.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.865+0000 7fcb897fa640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb7806b500 0x7fcb7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.865+0000 7fcb897fa640 1 -- 172.21.15.59:0/3882568553 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fcb7c08a3e0 con 0x7fcb8c0b5ee0 2024-03-06T07:25:46.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.865+0000 7fcb90f7e640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb7806b500 0x7fcb7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.865+0000 7fcb90f7e640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb7806b500 0x7fcb7806d9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fcb84035000 tx=0x7fcb84030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:46.869+0000 7fcb897fa640 1 -- 172.21.15.59:0/3882568553 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb7c058d00 con 0x7fcb8c0b5ee0 2024-03-06T07:25:47.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.089+0000 7fcb931e2640 1 -- 172.21.15.59:0/3882568553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcb8c0ce1d0 con 0x7fcb8c0b5ee0 2024-03-06T07:25:47.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.743+0000 7fcb897fa640 1 -- 172.21.15.59:0/3882568553 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v40) v1 ==== 174+0+0 (secure 0 0 0) 0x7fcb7c05cda0 con 0x7fcb8c0b5ee0 2024-03-06T07:25:47.745 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 12 and data pool 13 2024-03-06T07:25:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.752+0000 7fcb931e2640 1 -- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb7806b500 msgr2=0x7fcb7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.752+0000 7fcb931e2640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb7806b500 0x7fcb7806d9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fcb84035000 tx=0x7fcb84030000 comp rx=0 tx=0).stop 2024-03-06T07:25:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.753+0000 7fcb931e2640 1 -- 172.21.15.59:0/3882568553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0b5ee0 msgr2=0x7fcb8c09e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.753+0000 7fcb931e2640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0b5ee0 0x7fcb8c09e9a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00d800 tx=0x7fcb7c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.753+0000 7fcb931e2640 1 -- 172.21.15.59:0/3882568553 shutdown_connections 2024-03-06T07:25:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.753+0000 7fcb931e2640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb7806b500 0x7fcb7806d9c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.753+0000 7fcb931e2640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb8c0b5ee0 0x7fcb8c09e9a0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.754+0000 7fcb931e2640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c06cc90 0x7fcb8c0ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.754+0000 7fcb931e2640 1 --2- 172.21.15.59:0/3882568553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c06c290 0x7fcb8c1494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.754+0000 7fcb931e2640 1 -- 172.21.15.59:0/3882568553 >> 172.21.15.59:0/3882568553 conn(0x7fcb8c0b6ea0 msgr2=0x7fcb8c0b2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.755+0000 7fcb931e2640 1 -- 172.21.15.59:0/3882568553 shutdown_connections 2024-03-06T07:25:47.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.755+0000 7fcb931e2640 1 -- 172.21.15.59:0/3882568553 wait complete. 2024-03-06T07:25:47.780 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.883+0000 7f3ac4d5e640 1 Processor -- start 2024-03-06T07:25:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.883+0000 7f3ac4d5e640 1 -- start start 2024-03-06T07:25:47.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.884+0000 7f3ac4d5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac00b3060 0x7f3ac015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.884+0000 7f3ac4d5e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac015f580 con 0x7f3ac015bf30 2024-03-06T07:25:47.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.884+0000 7f3ac4d5e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac015b450 con 0x7f3ac013dc10 2024-03-06T07:25:47.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.884+0000 7f3ac4d5e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac00cf3f0 con 0x7f3ac00b3060 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.884+0000 7f3abed9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3ac015bf30 0x7f3ac015ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49974/0 (socket says 172.21.15.59:49974) 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.884+0000 7f3abed9d640 1 -- 172.21.15.59:0/2256839163 learned_addr learned my addr 172.21.15.59:0/2256839163 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333739069 0 0) 0x7f3ac015f580 con 0x7f3ac015bf30 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3ab7fff640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac00b3060 0x7f3ac015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aac003520 con 0x7f3ac015bf30 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1298674045 0 0) 0x7f3aac003520 con 0x7f3ac015bf30 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac00b3060 msgr2=0x7f3ac015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac00b3060 0x7f3ac015e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 >> v1:172.21.15.164:6789/0 conn(0x7f3ac013dc10 legacy=0x7f3ac00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac015b730 con 0x7f3ac015bf30 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f3ab80019f0 con 0x7f3ac015bf30 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 >> v1:172.21.15.59:6789/0 conn(0x7f3ac015bf30 legacy=0x7f3ac015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac00b3060 0x7f3aac0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3aac003f60 0x7f3aac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3aac004d40 0x7f3aac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac00cf3f0 con 0x7f3aac004d40 2024-03-06T07:25:47.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac00bb50 con 0x7f3aac003f60 2024-03-06T07:25:47.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac00df60 con 0x7f3ac00b3060 2024-03-06T07:25:47.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abed9d640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3aac004d40 0x7f3aac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abe59c640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac00b3060 0x7f3aac0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3ab7fff640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3aac003f60 0x7f3aac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abed9d640 1 -- 172.21.15.59:0/2256839163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac00b3060 msgr2=0x7f3aac0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abed9d640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac00b3060 0x7f3aac0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abed9d640 1 -- 172.21.15.59:0/2256839163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3aac003f60 msgr2=0x7f3aac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abed9d640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3aac003f60 0x7f3aac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.885+0000 7f3abed9d640 1 -- 172.21.15.59:0/2256839163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac003520 con 0x7f3aac004d40 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.886+0000 7f3abed9d640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3aac004d40 0x7f3aac0051c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800a840 tx=0x7f3aac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f897b4befd5bcd57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.886+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab800a460 con 0x7f3aac004d40 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.886+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab8034650 con 0x7f3aac004d40 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.886+0000 7f3abdd9b640 1 -- 172.21.15.59:0/2256839163 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8034a40 con 0x7f3aac004d40 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.887+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/2256839163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3aac004d40 msgr2=0x7f3aac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.887+0000 7f3ac4d5e640 1 --2- 172.21.15.59:0/2256839163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3aac004d40 0x7f3aac0051c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800a840 tx=0x7f3aac005830 comp rx=0 tx=0).stop 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.887+0000 7f3abed9d640 1 -- 172.21.15.59:0/2256839163 reap_dead start 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.887+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/2256839163 shutdown_connections 2024-03-06T07:25:47.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.887+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/2256839163 >> 172.21.15.59:0/2256839163 conn(0x7f3ac00cf8a0 msgr2=0x7f3ac00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.887+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/2256839163 shutdown_connections 2024-03-06T07:25:47.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.887+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/2256839163 wait complete. 2024-03-06T07:25:47.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.888+0000 7f3ac4d5e640 1 Processor -- start 2024-03-06T07:25:47.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.888+0000 7f3ac4d5e640 1 -- start start 2024-03-06T07:25:47.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ac4d5e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ac013cc00 0x7f3ac01f4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ac4d5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ac0143dc0 0x7f3ac006d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ac4d5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac006dd90 0x7f3ac0150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ac4d5e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac015e850 con 0x7f3ac0143dc0 2024-03-06T07:25:47.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ac4d5e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac015b210 con 0x7f3ac013cc00 2024-03-06T07:25:47.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ac4d5e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac015e9d0 con 0x7f3ac006dd90 2024-03-06T07:25:47.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ab7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ac0143dc0 0x7f3ac006d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ab7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ac0143dc0 0x7f3ac006d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46480/0 (socket says 172.21.15.59:46480) 2024-03-06T07:25:47.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ab7fff640 1 -- 172.21.15.59:0/788882808 learned_addr learned my addr 172.21.15.59:0/788882808 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3abe59c640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ac013cc00 0x7f3ac01f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3abed9d640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac006dd90 0x7f3ac0150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ab7fff640 1 -- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac006dd90 msgr2=0x7f3ac0150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ab7fff640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac006dd90 0x7f3ac0150440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ab7fff640 1 -- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ac013cc00 msgr2=0x7f3ac01f4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ab7fff640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ac013cc00 0x7f3ac01f4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.889+0000 7f3ab7fff640 1 -- 172.21.15.59:0/788882808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac0153420 con 0x7f3ac0143dc0 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.890+0000 7f3ab7fff640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ac0143dc0 0x7f3ac006d850 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80119a0 tx=0x7f3aa8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.890+0000 7f3ab6ffd640 1 -- 172.21.15.59:0/788882808 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa800faf0 con 0x7f3ac0143dc0 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.890+0000 7f3ab6ffd640 1 -- 172.21.15.59:0/788882808 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa800fc90 con 0x7f3ac0143dc0 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.890+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/788882808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac0153700 con 0x7f3ac0143dc0 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.890+0000 7f3ab6ffd640 1 -- 172.21.15.59:0/788882808 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8014440 con 0x7f3ac0143dc0 2024-03-06T07:25:47.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.890+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/788882808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac01f3d70 con 0x7f3ac0143dc0 2024-03-06T07:25:47.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.891+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/788882808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac006a120 con 0x7f3ac0143dc0 2024-03-06T07:25:47.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.891+0000 7f3ab6ffd640 1 -- 172.21.15.59:0/788882808 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3aa8012040 con 0x7f3ac0143dc0 2024-03-06T07:25:47.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.892+0000 7f3ab6ffd640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aa006b4d0 0x7f3aa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.892+0000 7f3ab6ffd640 1 -- 172.21.15.59:0/788882808 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3aa808a8b0 con 0x7f3ac0143dc0 2024-03-06T07:25:47.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.892+0000 7f3abe59c640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aa006b4d0 0x7f3aa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.892+0000 7f3abe59c640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aa006b4d0 0x7f3aa006d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0005590 tx=0x7f3ab0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:47.896+0000 7f3ab6ffd640 1 -- 172.21.15.59:0/788882808 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ac006a120 con 0x7f3ac0143dc0 2024-03-06T07:25:48.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.076+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/788882808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ac004f330 con 0x7f3ac0143dc0 2024-03-06T07:25:48.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.077+0000 7f3ab6ffd640 1 -- 172.21.15.59:0/788882808 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16661 (secure 0 0 0) 0x7f3aa8019030 con 0x7f3ac0143dc0 2024-03-06T07:25:48.079 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:48.079 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":80,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:47.090639+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:45.368088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:46.056543+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:48.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aa006b4d0 msgr2=0x7f3aa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aa006b4d0 0x7f3aa006d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0005590 tx=0x7f3ab0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/788882808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ac0143dc0 msgr2=0x7f3ac006d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ac0143dc0 0x7f3ac006d850 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f3aa80119a0 tx=0x7f3aa8011e60 comp rx=0 tx=0).stop 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/788882808 shutdown_connections 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3aa006b4d0 0x7f3aa006d990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ac006dd90 0x7f3ac0150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ac0143dc0 0x7f3ac006d850 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 --2- 172.21.15.59:0/788882808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ac013cc00 0x7f3ac01f4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/788882808 >> 172.21.15.59:0/788882808 conn(0x7f3ac00cf8a0 msgr2=0x7f3ac00712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/788882808 shutdown_connections 2024-03-06T07:25:48.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.079+0000 7f3ac4d5e640 1 -- 172.21.15.59:0/788882808 wait complete. 2024-03-06T07:25:48.092 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:48.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.191+0000 7f9c367a0640 1 Processor -- start 2024-03-06T07:25:48.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.192+0000 7f9c367a0640 1 -- start start 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.192+0000 7f9c367a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c30154fa0 0x7f9c300cd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.192+0000 7f9c367a0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9c300cd7d0 0x7f9c3015b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.192+0000 7f9c367a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c30155380 con 0x7f9c3015b5b0 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.192+0000 7f9c367a0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c300702d0 con 0x7f9c300cd7d0 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.192+0000 7f9c367a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c300cdc10 con 0x7f9c30154fa0 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.193+0000 7f9c34f9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9c300cd7d0 0x7f9c3015b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.193+0000 7f9c34f9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9c300cd7d0 0x7f9c3015b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46494/0 (socket says 172.21.15.59:46494) 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.193+0000 7f9c34f9d640 1 -- 172.21.15.59:0/3817754265 learned_addr learned my addr 172.21.15.59:0/3817754265 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.193+0000 7f9c3579e640 1 --2- 172.21.15.59:0/3817754265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c30154fa0 0x7f9c300cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.193+0000 7f9c34f9d640 1 -- 172.21.15.59:0/3817754265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c30154fa0 msgr2=0x7f9c300cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.193+0000 7f9c34f9d640 1 --2- 172.21.15.59:0/3817754265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c30154fa0 0x7f9c300cd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.193+0000 7f9c34f9d640 1 -- 172.21.15.59:0/3817754265 >> v1:172.21.15.164:6789/0 conn(0x7f9c3015b5b0 legacy=0x7f9c3015e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.193+0000 7f9c34f9d640 1 -- 172.21.15.59:0/3817754265 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c300726d0 con 0x7f9c300cd7d0 2024-03-06T07:25:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c34f9d640 1 --2- 172.21.15.59:0/3817754265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c300cd7d0 0x7f9c3015b070 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400bb10 tx=0x7f9c2402e1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaec285340e9e419 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c2ffff640 1 -- 172.21.15.59:0/3817754265 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2403b070 con 0x7f9c300cd7d0 2024-03-06T07:25:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c2ffff640 1 -- 172.21.15.59:0/3817754265 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c24036440 con 0x7f9c300cd7d0 2024-03-06T07:25:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c2ffff640 1 -- 172.21.15.59:0/3817754265 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24036810 con 0x7f9c300cd7d0 2024-03-06T07:25:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c367a0640 1 -- 172.21.15.59:0/3817754265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c300cd7d0 msgr2=0x7f9c3015b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c367a0640 1 --2- 172.21.15.59:0/3817754265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c300cd7d0 0x7f9c3015b070 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400bb10 tx=0x7f9c2402e1f0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c367a0640 1 -- 172.21.15.59:0/3817754265 shutdown_connections 2024-03-06T07:25:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c367a0640 1 --2- 172.21.15.59:0/3817754265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c300cd7d0 0x7f9c3015b070 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c367a0640 1 --2- 172.21.15.59:0/3817754265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c30154fa0 0x7f9c300cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c367a0640 1 -- 172.21.15.59:0/3817754265 >> 172.21.15.59:0/3817754265 conn(0x7f9c3006c9d0 msgr2=0x7f9c300b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c367a0640 1 -- 172.21.15.59:0/3817754265 shutdown_connections 2024-03-06T07:25:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.194+0000 7f9c367a0640 1 -- 172.21.15.59:0/3817754265 wait complete. 2024-03-06T07:25:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.196+0000 7f9c367a0640 1 Processor -- start 2024-03-06T07:25:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.196+0000 7f9c367a0640 1 -- start start 2024-03-06T07:25:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.196+0000 7f9c367a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c300cd7d0 0x7f9c3015a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.196+0000 7f9c367a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c30154fa0 0x7f9c3013d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.196+0000 7f9c367a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c3015b5b0 0x7f9c301402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.196+0000 7f9c367a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30071bf0 con 0x7f9c3015b5b0 2024-03-06T07:25:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.196+0000 7f9c367a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30071910 con 0x7f9c300cd7d0 2024-03-06T07:25:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.196+0000 7f9c367a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c30071ed0 con 0x7f9c30154fa0 2024-03-06T07:25:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.196+0000 7f9c35f9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c3015b5b0 0x7f9c301402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.196+0000 7f9c35f9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c3015b5b0 0x7f9c301402c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46498/0 (socket says 172.21.15.59:46498) 2024-03-06T07:25:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.196+0000 7f9c35f9f640 1 -- 172.21.15.59:0/2757959332 learned_addr learned my addr 172.21.15.59:0/2757959332 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c3579e640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c300cd7d0 0x7f9c3015a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c34f9d640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c30154fa0 0x7f9c3013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c35f9f640 1 -- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c30154fa0 msgr2=0x7f9c3013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c35f9f640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c30154fa0 0x7f9c3013d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c35f9f640 1 -- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c300cd7d0 msgr2=0x7f9c3015a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c35f9f640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c300cd7d0 0x7f9c3015a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c35f9f640 1 -- 172.21.15.59:0/2757959332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3013df00 con 0x7f9c3015b5b0 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c34f9d640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c30154fa0 0x7f9c3013d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c35f9f640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c3015b5b0 0x7f9c301402c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000c6b0 tx=0x7f9c2000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c2dffb640 1 -- 172.21.15.59:0/2757959332 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20003b70 con 0x7f9c3015b5b0 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c2dffb640 1 -- 172.21.15.59:0/2757959332 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c20003d30 con 0x7f9c3015b5b0 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c2dffb640 1 -- 172.21.15.59:0/2757959332 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20005310 con 0x7f9c3015b5b0 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c367a0640 1 -- 172.21.15.59:0/2757959332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c30155f20 con 0x7f9c3015b5b0 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.197+0000 7f9c367a0640 1 -- 172.21.15.59:0/2757959332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3013e890 con 0x7f9c3015b5b0 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.198+0000 7f9c367a0640 1 -- 172.21.15.59:0/2757959332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c30068f80 con 0x7f9c3015b5b0 2024-03-06T07:25:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.199+0000 7f9c2dffb640 1 -- 172.21.15.59:0/2757959332 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c200054b0 con 0x7f9c3015b5b0 2024-03-06T07:25:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.199+0000 7f9c2dffb640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c1c06b4d0 0x7f9c1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.199+0000 7f9c2dffb640 1 -- 172.21.15.59:0/2757959332 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c2008a0f0 con 0x7f9c3015b5b0 2024-03-06T07:25:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.199+0000 7f9c3579e640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c1c06b4d0 0x7f9c1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.200+0000 7f9c3579e640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c1c06b4d0 0x7f9c1c06d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009ad0 tx=0x7f9c28008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.203+0000 7f9c2dffb640 1 -- 172.21.15.59:0/2757959332 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c20058ee0 con 0x7f9c3015b5b0 2024-03-06T07:25:48.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.385+0000 7f9c367a0640 1 -- 172.21.15.59:0/2757959332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c3013e380 con 0x7f9c3015b5b0 2024-03-06T07:25:48.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.386+0000 7f9c2dffb640 1 -- 172.21.15.59:0/2757959332 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16661 (secure 0 0 0) 0x7f9c200587e0 con 0x7f9c3015b5b0 2024-03-06T07:25:48.388 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:48.388 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":80,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:47.090639+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:45.368088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:46.056543+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 -- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c1c06b4d0 msgr2=0x7f9c1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c1c06b4d0 0x7f9c1c06d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009ad0 tx=0x7f9c28008040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 -- 172.21.15.59:0/2757959332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c3015b5b0 msgr2=0x7f9c301402c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c3015b5b0 0x7f9c301402c0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f9c2000c6b0 tx=0x7f9c2000cb70 comp rx=0 tx=0).stop 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 -- 172.21.15.59:0/2757959332 shutdown_connections 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c1c06b4d0 0x7f9c1c06d990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c3015b5b0 0x7f9c301402c0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c30154fa0 0x7f9c3013d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 --2- 172.21.15.59:0/2757959332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c300cd7d0 0x7f9c3015a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 -- 172.21.15.59:0/2757959332 >> 172.21.15.59:0/2757959332 conn(0x7f9c3006c9d0 msgr2=0x7f9c30152820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 -- 172.21.15.59:0/2757959332 shutdown_connections 2024-03-06T07:25:48.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.388+0000 7f9c367a0640 1 -- 172.21.15.59:0/2757959332 wait complete. 2024-03-06T07:25:48.401 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:25:48.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.503+0000 7fa99449d640 1 Processor -- start 2024-03-06T07:25:48.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.503+0000 7fa99449d640 1 -- start start 2024-03-06T07:25:48.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa99449d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa98c15def0 0x7fa98c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa99449d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa98c0cea50 0x7fa98c0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa99449d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c06d030 con 0x7fa98c06cab0 2024-03-06T07:25:48.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa99449d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c0cf1c0 con 0x7fa98c15def0 2024-03-06T07:25:48.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa99449d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c0cc760 con 0x7fa98c0cea50 2024-03-06T07:25:48.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa992239640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa98c15def0 0x7fa98c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa992239640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa98c15def0 0x7fa98c0b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46500/0 (socket says 172.21.15.59:46500) 2024-03-06T07:25:48.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa992239640 1 -- 172.21.15.59:0/2753479230 learned_addr learned my addr 172.21.15.59:0/2753479230 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:48.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa992a3a640 1 --2- 172.21.15.59:0/2753479230 >> v2:172.21.15.164:3300/0 conn(0x7fa98c0cea50 0x7fa98c0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa992239640 1 -- 172.21.15.59:0/2753479230 >> v2:172.21.15.164:3300/0 conn(0x7fa98c0cea50 msgr2=0x7fa98c0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa992239640 1 --2- 172.21.15.59:0/2753479230 >> v2:172.21.15.164:3300/0 conn(0x7fa98c0cea50 0x7fa98c0ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa992239640 1 -- 172.21.15.59:0/2753479230 >> v1:172.21.15.164:6789/0 conn(0x7fa98c06cab0 legacy=0x7fa98c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.504+0000 7fa992239640 1 -- 172.21.15.59:0/2753479230 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c06d310 con 0x7fa98c15def0 2024-03-06T07:25:48.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.505+0000 7fa992239640 1 --2- 172.21.15.59:0/2753479230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa98c15def0 0x7fa98c0b41b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0729a0 tx=0x7fa98802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4415771f2a2a9865 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.505+0000 7fa991237640 1 -- 172.21.15.59:0/2753479230 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988034070 con 0x7fa98c15def0 2024-03-06T07:25:48.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.505+0000 7fa991237640 1 -- 172.21.15.59:0/2753479230 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988030860 con 0x7fa98c15def0 2024-03-06T07:25:48.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.505+0000 7fa991237640 1 -- 172.21.15.59:0/2753479230 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988030c30 con 0x7fa98c15def0 2024-03-06T07:25:48.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.506+0000 7fa99449d640 1 -- 172.21.15.59:0/2753479230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa98c15def0 msgr2=0x7fa98c0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.506+0000 7fa99449d640 1 --2- 172.21.15.59:0/2753479230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa98c15def0 0x7fa98c0b41b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0729a0 tx=0x7fa98802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.506+0000 7fa99449d640 1 -- 172.21.15.59:0/2753479230 shutdown_connections 2024-03-06T07:25:48.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.506+0000 7fa99449d640 1 --2- 172.21.15.59:0/2753479230 >> v2:172.21.15.164:3300/0 conn(0x7fa98c0cea50 0x7fa98c0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.506+0000 7fa99449d640 1 --2- 172.21.15.59:0/2753479230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa98c15def0 0x7fa98c0b41b0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.506+0000 7fa99449d640 1 -- 172.21.15.59:0/2753479230 >> 172.21.15.59:0/2753479230 conn(0x7fa98c0b3690 msgr2=0x7fa98c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.506+0000 7fa99449d640 1 -- 172.21.15.59:0/2753479230 shutdown_connections 2024-03-06T07:25:48.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.506+0000 7fa99449d640 1 -- 172.21.15.59:0/2753479230 wait complete. 2024-03-06T07:25:48.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.507+0000 7fa99449d640 1 Processor -- start 2024-03-06T07:25:48.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.507+0000 7fa99449d640 1 -- start start 2024-03-06T07:25:48.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa99449d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa98c06cab0 0x7fa98c14b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa99449d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa98c0cea50 0x7fa98c0c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa99449d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa98c15def0 0x7fa98c0c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa99449d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c15e2d0 con 0x7fa98c06cab0 2024-03-06T07:25:48.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa99449d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0cc3f0 con 0x7fa98c0cea50 2024-03-06T07:25:48.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa99449d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c15e450 con 0x7fa98c15def0 2024-03-06T07:25:48.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa992239640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa98c06cab0 0x7fa98c14b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa992239640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa98c06cab0 0x7fa98c14b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46516/0 (socket says 172.21.15.59:46516) 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa992239640 1 -- 172.21.15.59:0/949796534 learned_addr learned my addr 172.21.15.59:0/949796534 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa991a38640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa98c0cea50 0x7fa98c0c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa992a3a640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa98c15def0 0x7fa98c0c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa992239640 1 -- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa98c15def0 msgr2=0x7fa98c0c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa992239640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa98c15def0 0x7fa98c0c73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa992239640 1 -- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa98c0cea50 msgr2=0x7fa98c0c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa992239640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa98c0cea50 0x7fa98c0c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa992239640 1 -- 172.21.15.59:0/949796534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c14bdc0 con 0x7fa98c06cab0 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa991a38640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa98c0cea50 0x7fa98c0c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa992239640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa98c06cab0 0x7fa98c14b880 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa988000c00 tx=0x7fa9880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa982ffd640 1 -- 172.21.15.59:0/949796534 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988034040 con 0x7fa98c06cab0 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa982ffd640 1 -- 172.21.15.59:0/949796534 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa988039d90 con 0x7fa98c06cab0 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa982ffd640 1 -- 172.21.15.59:0/949796534 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa988043610 con 0x7fa98c06cab0 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa99449d640 1 -- 172.21.15.59:0/949796534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98c0c5e40 con 0x7fa98c06cab0 2024-03-06T07:25:48.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.508+0000 7fa99449d640 1 -- 172.21.15.59:0/949796534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98c0c6400 con 0x7fa98c06cab0 2024-03-06T07:25:48.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.509+0000 7fa99449d640 1 -- 172.21.15.59:0/949796534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98c09fcf0 con 0x7fa98c06cab0 2024-03-06T07:25:48.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.510+0000 7fa982ffd640 1 -- 172.21.15.59:0/949796534 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa98804a060 con 0x7fa98c06cab0 2024-03-06T07:25:48.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.510+0000 7fa982ffd640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa97006b500 0x7fa97006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.510+0000 7fa982ffd640 1 -- 172.21.15.59:0/949796534 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9880ac2e0 con 0x7fa98c06cab0 2024-03-06T07:25:48.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.510+0000 7fa991a38640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa97006b500 0x7fa97006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.511+0000 7fa991a38640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa97006b500 0x7fa97006d9c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa98c1601d0 tx=0x7fa97c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.515+0000 7fa982ffd640 1 -- 172.21.15.59:0/949796534 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa98803e070 con 0x7fa98c06cab0 2024-03-06T07:25:48.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.733+0000 7fa99449d640 1 -- 172.21.15.59:0/949796534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa98c0c87a0 con 0x7fa98c06cab0 2024-03-06T07:25:48.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.743+0000 7fa982ffd640 1 -- 172.21.15.59:0/949796534 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa988037360 con 0x7fa98c06cab0 2024-03-06T07:25:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 -- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa97006b500 msgr2=0x7fa97006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa97006b500 0x7fa97006d9c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa98c1601d0 tx=0x7fa97c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 -- 172.21.15.59:0/949796534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa98c06cab0 msgr2=0x7fa98c14b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa98c06cab0 0x7fa98c14b880 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa988000c00 tx=0x7fa9880046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 -- 172.21.15.59:0/949796534 shutdown_connections 2024-03-06T07:25:48.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa97006b500 0x7fa97006d9c0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa98c15def0 0x7fa98c0c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa98c0cea50 0x7fa98c0c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 --2- 172.21.15.59:0/949796534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa98c06cab0 0x7fa98c14b880 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 -- 172.21.15.59:0/949796534 >> 172.21.15.59:0/949796534 conn(0x7fa98c0b3690 msgr2=0x7fa98c0ccbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 -- 172.21.15.59:0/949796534 shutdown_connections 2024-03-06T07:25:48.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.745+0000 7fa99449d640 1 -- 172.21.15.59:0/949796534 wait complete. 2024-03-06T07:25:48.758 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:48.758 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:48.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.861+0000 7f36feb5e640 1 Processor -- start 2024-03-06T07:25:48.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.861+0000 7f36feb5e640 1 -- start start 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.862+0000 7f36feb5e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f36f81614d0 0x7f36f80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.862+0000 7f36feb5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36f80d24b0 0x7f36f80b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.862+0000 7f36feb5e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f81618b0 con 0x7f36f80d1910 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.862+0000 7f36feb5e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f80cf400 con 0x7f36f81614d0 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.862+0000 7f36feb5e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80cf6a0 con 0x7f36f80d24b0 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.862+0000 7f36fc8fa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f36f81614d0 0x7f36f80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.862+0000 7f36fc8fa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f36f81614d0 0x7f36f80d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46530/0 (socket says 172.21.15.59:46530) 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.862+0000 7f36fc8fa640 1 -- 172.21.15.59:0/2403114293 learned_addr learned my addr 172.21.15.59:0/2403114293 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.862+0000 7f36f7fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f36f80d1910 0x7f36f80d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49986/0 (socket says 172.21.15.59:49986) 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.863+0000 7f36fd0fb640 1 --2- 172.21.15.59:0/2403114293 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36f80d24b0 0x7f36f80b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.863+0000 7f36fc8fa640 1 -- 172.21.15.59:0/2403114293 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36f80d24b0 msgr2=0x7f36f80b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.863+0000 7f36fc8fa640 1 --2- 172.21.15.59:0/2403114293 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36f80d24b0 0x7f36f80b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.863+0000 7f36fc8fa640 1 -- 172.21.15.59:0/2403114293 >> v1:172.21.15.59:6789/0 conn(0x7f36f80d1910 legacy=0x7f36f80d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.863+0000 7f36fc8fa640 1 -- 172.21.15.59:0/2403114293 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f80cf980 con 0x7f36f81614d0 2024-03-06T07:25:48.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.863+0000 7f36f77fe640 1 -- 172.21.15.59:0/2403114293 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586465972 0 0) 0x7f36f81618b0 con 0x7f36f80d1910 2024-03-06T07:25:48.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.863+0000 7f36fc8fa640 1 --2- 172.21.15.59:0/2403114293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36f81614d0 0x7f36f80d4860 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009870 tx=0x7f36ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=415d6b6032f47365 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.864+0000 7f36f77fe640 1 -- 172.21.15.59:0/2403114293 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ec034070 con 0x7f36f81614d0 2024-03-06T07:25:48.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.864+0000 7f36f77fe640 1 -- 172.21.15.59:0/2403114293 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ec030840 con 0x7f36f81614d0 2024-03-06T07:25:48.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.864+0000 7f36f77fe640 1 -- 172.21.15.59:0/2403114293 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ec030c10 con 0x7f36f81614d0 2024-03-06T07:25:48.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.864+0000 7f36feb5e640 1 -- 172.21.15.59:0/2403114293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36f81614d0 msgr2=0x7f36f80d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.864+0000 7f36feb5e640 1 --2- 172.21.15.59:0/2403114293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36f81614d0 0x7f36f80d4860 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009870 tx=0x7f36ec02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:48.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.864+0000 7f36feb5e640 1 -- 172.21.15.59:0/2403114293 shutdown_connections 2024-03-06T07:25:48.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.864+0000 7f36feb5e640 1 --2- 172.21.15.59:0/2403114293 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36f80d24b0 0x7f36f80b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.864+0000 7f36feb5e640 1 --2- 172.21.15.59:0/2403114293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36f81614d0 0x7f36f80d4860 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.864+0000 7f36feb5e640 1 -- 172.21.15.59:0/2403114293 >> 172.21.15.59:0/2403114293 conn(0x7f36f80d3d40 msgr2=0x7f36f80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:48.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.864+0000 7f36feb5e640 1 -- 172.21.15.59:0/2403114293 shutdown_connections 2024-03-06T07:25:48.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.864+0000 7f36feb5e640 1 -- 172.21.15.59:0/2403114293 wait complete. 2024-03-06T07:25:48.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.865+0000 7f36feb5e640 1 Processor -- start 2024-03-06T07:25:48.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.865+0000 7f36feb5e640 1 -- start start 2024-03-06T07:25:48.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.866+0000 7f36feb5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36f80d1910 0x7f36f8155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.866+0000 7f36feb5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36f80d24b0 0x7f36f80cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.866+0000 7f36feb5e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36f81614d0 0x7f36f80cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.866+0000 7f36feb5e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80b1370 con 0x7f36f80d24b0 2024-03-06T07:25:48.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.866+0000 7f36feb5e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80b0910 con 0x7f36f81614d0 2024-03-06T07:25:48.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.866+0000 7f36feb5e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80b14f0 con 0x7f36f80d1910 2024-03-06T07:25:48.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.866+0000 7f36f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36f80d24b0 0x7f36f80cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.866+0000 7f36f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36f80d24b0 0x7f36f80cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46534/0 (socket says 172.21.15.59:46534) 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.866+0000 7f36f7fff640 1 -- 172.21.15.59:0/2887328772 learned_addr learned my addr 172.21.15.59:0/2887328772 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36fc8fa640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36f80d1910 0x7f36f8155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36fd0fb640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36f81614d0 0x7f36f80cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36f7fff640 1 -- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36f80d1910 msgr2=0x7f36f8155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36f7fff640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36f80d1910 0x7f36f8155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36f7fff640 1 -- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36f81614d0 msgr2=0x7f36f80cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36f7fff640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36f81614d0 0x7f36f80cb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36f7fff640 1 -- 172.21.15.59:0/2887328772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f80cd0a0 con 0x7f36f80d24b0 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36fc8fa640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36f80d1910 0x7f36f8155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36f7fff640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36f80d24b0 0x7f36f80cc730 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f36e400c4d0 tx=0x7f36e400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36f57fa640 1 -- 172.21.15.59:0/2887328772 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4018070 con 0x7f36f80d24b0 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36f57fa640 1 -- 172.21.15.59:0/2887328772 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e4007500 con 0x7f36f80d24b0 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36feb5e640 1 -- 172.21.15.59:0/2887328772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f80cd380 con 0x7f36f80d24b0 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36f57fa640 1 -- 172.21.15.59:0/2887328772 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4004a20 con 0x7f36f80d24b0 2024-03-06T07:25:48.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.867+0000 7f36feb5e640 1 -- 172.21.15.59:0/2887328772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f80cb300 con 0x7f36f80d24b0 2024-03-06T07:25:48.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.868+0000 7f36feb5e640 1 -- 172.21.15.59:0/2887328772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f80cd550 con 0x7f36f80d24b0 2024-03-06T07:25:48.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.868+0000 7f36f57fa640 1 -- 172.21.15.59:0/2887328772 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36e4004bc0 con 0x7f36f80d24b0 2024-03-06T07:25:48.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.869+0000 7f36f57fa640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36dc06b4d0 0x7f36dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:48.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.869+0000 7f36f57fa640 1 -- 172.21.15.59:0/2887328772 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36e4089570 con 0x7f36f80d24b0 2024-03-06T07:25:48.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.869+0000 7f36fc8fa640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36dc06b4d0 0x7f36dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:48.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.870+0000 7f36fc8fa640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36dc06b4d0 0x7f36dc06d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f36ec036000 tx=0x7f36ec008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:48.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:48.873+0000 7f36f57fa640 1 -- 172.21.15.59:0/2887328772 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36e4058370 con 0x7f36f80d24b0 2024-03-06T07:25:49.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.092+0000 7f36feb5e640 1 -- 172.21.15.59:0/2887328772 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f36f8068ff0 con 0x7f36f80d24b0 2024-03-06T07:25:49.670 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:49.672 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:49.718 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-03-06T07:25:49.717+0000 7f67bd2b8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:49.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.754+0000 7f36f57fa640 1 -- 172.21.15.59:0/2887328772 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f36e4057c70 con 0x7f36f80d24b0 2024-03-06T07:25:49.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 -- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36dc06b4d0 msgr2=0x7f36dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36dc06b4d0 0x7f36dc06d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f36ec036000 tx=0x7f36ec008030 comp rx=0 tx=0).stop 2024-03-06T07:25:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 -- 172.21.15.59:0/2887328772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36f80d24b0 msgr2=0x7f36f80cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36f80d24b0 0x7f36f80cc730 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f36e400c4d0 tx=0x7f36e400c990 comp rx=0 tx=0).stop 2024-03-06T07:25:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 -- 172.21.15.59:0/2887328772 shutdown_connections 2024-03-06T07:25:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36dc06b4d0 0x7f36dc06d990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36f81614d0 0x7f36f80cb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36f80d24b0 0x7f36f80cc730 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 --2- 172.21.15.59:0/2887328772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36f80d1910 0x7f36f8155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 -- 172.21.15.59:0/2887328772 >> 172.21.15.59:0/2887328772 conn(0x7f36f80d3d40 msgr2=0x7f36f806cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 -- 172.21.15.59:0/2887328772 shutdown_connections 2024-03-06T07:25:49.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.756+0000 7f36feb5e640 1 -- 172.21.15.59:0/2887328772 wait complete. 2024-03-06T07:25:49.769 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:49.769 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:49.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.909+0000 7f4429156640 1 Processor -- start 2024-03-06T07:25:49.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.909+0000 7f4429156640 1 -- start start 2024-03-06T07:25:49.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.910+0000 7f4429156640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f442413dc10 0x7f44240b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.910+0000 7f4429156640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442415f580 con 0x7f442415ef60 2024-03-06T07:25:49.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.910+0000 7f4429156640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442415b450 con 0x7f442415bf30 2024-03-06T07:25:49.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.910+0000 7f4429156640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44240cf3f0 con 0x7f442413dc10 2024-03-06T07:25:49.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.910+0000 7f4422d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f442413dc10 0x7f44240b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.910+0000 7f442359e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f442415ef60 0x7f44240708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50000/0 (socket says 172.21.15.59:50000) 2024-03-06T07:25:49.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.910+0000 7f442359e640 1 -- 172.21.15.59:0/3032558387 learned_addr learned my addr 172.21.15.59:0/3032558387 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:49.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.911+0000 7f4422d9d640 1 -- 172.21.15.59:0/3032558387 >> v1:172.21.15.164:6789/0 conn(0x7f442415bf30 legacy=0x7f442415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.911+0000 7f4421d9b640 1 -- 172.21.15.59:0/3032558387 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739166279 0 0) 0x7f442415f580 con 0x7f442415ef60 2024-03-06T07:25:49.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.911+0000 7f4422d9d640 1 -- 172.21.15.59:0/3032558387 >> v1:172.21.15.59:6789/0 conn(0x7f442415ef60 legacy=0x7f44240708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.911+0000 7f4422d9d640 1 -- 172.21.15.59:0/3032558387 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442415b730 con 0x7f442413dc10 2024-03-06T07:25:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.911+0000 7f4422d9d640 1 --2- 172.21.15.59:0/3032558387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f442413dc10 0x7f44240b2950 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f440c0097a0 tx=0x7f440c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53e8a4b64dba52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.911+0000 7f4421d9b640 1 -- 172.21.15.59:0/3032558387 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c035020 con 0x7f442413dc10 2024-03-06T07:25:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.911+0000 7f4421d9b640 1 -- 172.21.15.59:0/3032558387 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c031830 con 0x7f442413dc10 2024-03-06T07:25:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.911+0000 7f4421d9b640 1 -- 172.21.15.59:0/3032558387 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c031c00 con 0x7f442413dc10 2024-03-06T07:25:49.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.912+0000 7f4429156640 1 -- 172.21.15.59:0/3032558387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f442413dc10 msgr2=0x7f44240b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.912+0000 7f4429156640 1 --2- 172.21.15.59:0/3032558387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f442413dc10 0x7f44240b2950 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f440c0097a0 tx=0x7f440c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.912+0000 7f4429156640 1 -- 172.21.15.59:0/3032558387 shutdown_connections 2024-03-06T07:25:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.912+0000 7f4429156640 1 --2- 172.21.15.59:0/3032558387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f442413dc10 0x7f44240b2950 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.912+0000 7f4429156640 1 -- 172.21.15.59:0/3032558387 >> 172.21.15.59:0/3032558387 conn(0x7f44240cf8a0 msgr2=0x7f44240cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.912+0000 7f4429156640 1 -- 172.21.15.59:0/3032558387 shutdown_connections 2024-03-06T07:25:49.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.912+0000 7f4429156640 1 -- 172.21.15.59:0/3032558387 wait complete. 2024-03-06T07:25:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.914+0000 7f4429156640 1 Processor -- start 2024-03-06T07:25:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.914+0000 7f4429156640 1 -- start start 2024-03-06T07:25:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.914+0000 7f4429156640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f442413dc10 0x7f4424144360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.914+0000 7f4429156640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f442415bf30 0x7f442413b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.914+0000 7f4429156640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f442415ef60 0x7f4424156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.914+0000 7f4429156640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44240b2e90 con 0x7f442415ef60 2024-03-06T07:25:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.914+0000 7f4429156640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442415b210 con 0x7f442413dc10 2024-03-06T07:25:49.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.914+0000 7f4429156640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44240b3010 con 0x7f442415bf30 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.914+0000 7f442359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f442415ef60 0x7f4424156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.914+0000 7f442359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f442415ef60 0x7f4424156330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46556/0 (socket says 172.21.15.59:46556) 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.914+0000 7f442359e640 1 -- 172.21.15.59:0/84917275 learned_addr learned my addr 172.21.15.59:0/84917275 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f4422d9d640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f442413dc10 0x7f4424144360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f442259c640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f442415bf30 0x7f442413b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f442359e640 1 -- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f442415bf30 msgr2=0x7f442413b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f442359e640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f442415bf30 0x7f442413b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f442359e640 1 -- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f442413dc10 msgr2=0x7f4424144360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f442359e640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f442413dc10 0x7f4424144360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f442359e640 1 -- 172.21.15.59:0/84917275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442413bac0 con 0x7f442415ef60 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f4422d9d640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f442413dc10 0x7f4424144360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f442359e640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f442415ef60 0x7f4424156330 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f441c00c700 tx=0x7f441c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f441b7fe640 1 -- 172.21.15.59:0/84917275 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c018070 con 0x7f442415ef60 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f441b7fe640 1 -- 172.21.15.59:0/84917275 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441c00ce10 con 0x7f442415ef60 2024-03-06T07:25:49.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f4429156640 1 -- 172.21.15.59:0/84917275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44241432c0 con 0x7f442415ef60 2024-03-06T07:25:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f441b7fe640 1 -- 172.21.15.59:0/84917275 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c00a880 con 0x7f442415ef60 2024-03-06T07:25:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.915+0000 7f4429156640 1 -- 172.21.15.59:0/84917275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4424143880 con 0x7f442415ef60 2024-03-06T07:25:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.916+0000 7f4429156640 1 -- 172.21.15.59:0/84917275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f442404f330 con 0x7f442415ef60 2024-03-06T07:25:49.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.917+0000 7f441b7fe640 1 -- 172.21.15.59:0/84917275 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f441c00aa20 con 0x7f442415ef60 2024-03-06T07:25:49.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.917+0000 7f441b7fe640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f440406b4d0 0x7f440406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.917+0000 7f441b7fe640 1 -- 172.21.15.59:0/84917275 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f441c089820 con 0x7f442415ef60 2024-03-06T07:25:49.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.917+0000 7f4422d9d640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f440406b4d0 0x7f440406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.918+0000 7f4422d9d640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f440406b4d0 0x7f440406d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f440c036000 tx=0x7f440c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:49.926+0000 7f441b7fe640 1 -- 172.21.15.59:0/84917275 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f441c058620 con 0x7f442415ef60 2024-03-06T07:25:50.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.143+0000 7f4429156640 1 -- 172.21.15.59:0/84917275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f442406eb00 con 0x7f442415ef60 2024-03-06T07:25:50.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.144+0000 7f441b7fe640 1 -- 172.21.15.59:0/84917275 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f441c057f20 con 0x7f442415ef60 2024-03-06T07:25:50.146 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:50.146 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":42,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2865818497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2865818497},{"type":"v1","addr":"172.21.15.164:6836","nonce":2865818497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1805325807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1805325807},{"type":"v1","addr":"172.21.15.164:6837","nonce":1805325807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6999,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3345771250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3345771250},{"type":"v1","addr":"172.21.15.59:6835","nonce":3345771250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:47.090938+0000","modified":"2024-03-06T07:25:49.093664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2521311582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2521311582},{"type":"v1","addr":"172.21.15.59:6834","nonce":2521311582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7002,"qdb_cluster":[7002]},"id":6}]} 2024-03-06T07:25:50.146 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 42 2024-03-06T07:25:50.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 -- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f440406b4d0 msgr2=0x7f440406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f440406b4d0 0x7f440406d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f440c036000 tx=0x7f440c008030 comp rx=0 tx=0).stop 2024-03-06T07:25:50.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 -- 172.21.15.59:0/84917275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f442415ef60 msgr2=0x7f4424156330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f442415ef60 0x7f4424156330 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f441c00c700 tx=0x7f441c00cbc0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 -- 172.21.15.59:0/84917275 shutdown_connections 2024-03-06T07:25:50.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f440406b4d0 0x7f440406d990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f442415ef60 0x7f4424156330 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f442415bf30 0x7f442413b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 --2- 172.21.15.59:0/84917275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f442413dc10 0x7f4424144360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 -- 172.21.15.59:0/84917275 >> 172.21.15.59:0/84917275 conn(0x7f44240cf8a0 msgr2=0x7f44240cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 -- 172.21.15.59:0/84917275 shutdown_connections 2024-03-06T07:25:50.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.146+0000 7f4429156640 1 -- 172.21.15.59:0/84917275 wait complete. 2024-03-06T07:25:50.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:50.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.258+0000 7f72f0fd0640 1 Processor -- start 2024-03-06T07:25:50.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.258+0000 7f72f0fd0640 1 -- start start 2024-03-06T07:25:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.258+0000 7f72f0fd0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f72ec15dfe0 0x7f72ec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.258+0000 7f72f0fd0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f72ec0d29c0 0x7f72ec0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72f0fd0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec0b3430 con 0x7f72ec0cf830 2024-03-06T07:25:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72f0fd0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec0b31e0 con 0x7f72ec15dfe0 2024-03-06T07:25:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72f0fd0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec0b3d60 con 0x7f72ec0d29c0 2024-03-06T07:25:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72ea59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f72ec15dfe0 0x7f72ec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72e9d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f72ec0cf830 0x7f72ec0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50002/0 (socket says 172.21.15.59:50002) 2024-03-06T07:25:50.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72e9d9b640 1 -- 172.21.15.59:0/1347593641 learned_addr learned my addr 172.21.15.59:0/1347593641 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:50.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72ead9d640 1 --2- 172.21.15.59:0/1347593641 >> v2:172.21.15.164:3300/0 conn(0x7f72ec0d29c0 0x7f72ec0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72ea59c640 1 -- 172.21.15.59:0/1347593641 >> v2:172.21.15.164:3300/0 conn(0x7f72ec0d29c0 msgr2=0x7f72ec0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72ea59c640 1 --2- 172.21.15.59:0/1347593641 >> v2:172.21.15.164:3300/0 conn(0x7f72ec0d29c0 0x7f72ec0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72ea59c640 1 -- 172.21.15.59:0/1347593641 >> v1:172.21.15.59:6789/0 conn(0x7f72ec0cf830 legacy=0x7f72ec0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72ea59c640 1 -- 172.21.15.59:0/1347593641 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0b3710 con 0x7f72ec15dfe0 2024-03-06T07:25:50.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.259+0000 7f72ead9d640 1 --2- 172.21.15.59:0/1347593641 >> v2:172.21.15.164:3300/0 conn(0x7f72ec0d29c0 0x7f72ec0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:50.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.260+0000 7f72ea59c640 1 --2- 172.21.15.59:0/1347593641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ec15dfe0 0x7f72ec0cf2f0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0b3cd0 tx=0x7f72dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3d14da4b55f068 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.260+0000 7f72e959a640 1 -- 172.21.15.59:0/1347593641 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc039070 con 0x7f72ec15dfe0 2024-03-06T07:25:50.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.260+0000 7f72e959a640 1 -- 172.21.15.59:0/1347593641 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc02cc80 con 0x7f72ec15dfe0 2024-03-06T07:25:50.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.260+0000 7f72e959a640 1 -- 172.21.15.59:0/1347593641 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc034660 con 0x7f72ec15dfe0 2024-03-06T07:25:50.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.260+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1347593641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ec15dfe0 msgr2=0x7f72ec0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.260+0000 7f72f0fd0640 1 --2- 172.21.15.59:0/1347593641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ec15dfe0 0x7f72ec0cf2f0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0b3cd0 tx=0x7f72dc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.261+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1347593641 shutdown_connections 2024-03-06T07:25:50.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.261+0000 7f72f0fd0640 1 --2- 172.21.15.59:0/1347593641 >> v2:172.21.15.164:3300/0 conn(0x7f72ec0d29c0 0x7f72ec0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.261+0000 7f72f0fd0640 1 --2- 172.21.15.59:0/1347593641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ec15dfe0 0x7f72ec0cf2f0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.261+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1347593641 >> 172.21.15.59:0/1347593641 conn(0x7f72ec0d3cd0 msgr2=0x7f72ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.261+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1347593641 shutdown_connections 2024-03-06T07:25:50.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.261+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1347593641 wait complete. 2024-03-06T07:25:50.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.263+0000 7f72f0fd0640 1 Processor -- start 2024-03-06T07:25:50.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.263+0000 7f72f0fd0640 1 -- start start 2024-03-06T07:25:50.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72f0fd0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72ec0cf830 0x7f72ec1f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72f0fd0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ec0d29c0 0x7f72ec14ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72f0fd0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec15dfe0 0x7f72ec0cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72f0fd0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0d1df0 con 0x7f72ec0d29c0 2024-03-06T07:25:50.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72f0fd0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0d1c70 con 0x7f72ec0cf830 2024-03-06T07:25:50.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72f0fd0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec15e3c0 con 0x7f72ec15dfe0 2024-03-06T07:25:50.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72e9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ec0d29c0 0x7f72ec14ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72e9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ec0d29c0 0x7f72ec14ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46576/0 (socket says 172.21.15.59:46576) 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72e9d9b640 1 -- 172.21.15.59:0/1160512783 learned_addr learned my addr 172.21.15.59:0/1160512783 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72ea59c640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72ec0cf830 0x7f72ec1f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72ead9d640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec15dfe0 0x7f72ec0cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72e9d9b640 1 -- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec15dfe0 msgr2=0x7f72ec0cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72e9d9b640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec15dfe0 0x7f72ec0cd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72e9d9b640 1 -- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72ec0cf830 msgr2=0x7f72ec1f3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72e9d9b640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72ec0cf830 0x7f72ec1f3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72e9d9b640 1 -- 172.21.15.59:0/1160512783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec15f8f0 con 0x7f72ec0d29c0 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72ea59c640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72ec0cf830 0x7f72ec1f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.264+0000 7f72e9d9b640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ec0d29c0 0x7f72ec14ef80 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f72e000c440 tx=0x7f72e0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.265+0000 7f72daffd640 1 -- 172.21.15.59:0/1160512783 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0017070 con 0x7f72ec0d29c0 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.265+0000 7f72daffd640 1 -- 172.21.15.59:0/1160512783 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e0005ce0 con 0x7f72ec0d29c0 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.265+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1160512783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec15fbd0 con 0x7f72ec0d29c0 2024-03-06T07:25:50.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.265+0000 7f72daffd640 1 -- 172.21.15.59:0/1160512783 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e00072f0 con 0x7f72ec0d29c0 2024-03-06T07:25:50.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.265+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1160512783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec160190 con 0x7f72ec0d29c0 2024-03-06T07:25:50.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.266+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1160512783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec04f330 con 0x7f72ec0d29c0 2024-03-06T07:25:50.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.266+0000 7f72daffd640 1 -- 172.21.15.59:0/1160512783 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72e001a070 con 0x7f72ec0d29c0 2024-03-06T07:25:50.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.266+0000 7f72daffd640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72cc06b4d0 0x7f72cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.267+0000 7f72daffd640 1 -- 172.21.15.59:0/1160512783 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72e0089680 con 0x7f72ec0d29c0 2024-03-06T07:25:50.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.267+0000 7f72ea59c640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72cc06b4d0 0x7f72cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.267+0000 7f72ea59c640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72cc06b4d0 0x7f72cc06d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f72dc036000 tx=0x7f72dc030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.271+0000 7f72daffd640 1 -- 172.21.15.59:0/1160512783 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72e0058470 con 0x7f72ec0d29c0 2024-03-06T07:25:50.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.453+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1160512783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72ec0caa10 con 0x7f72ec0d29c0 2024-03-06T07:25:50.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.454+0000 7f72daffd640 1 -- 172.21.15.59:0/1160512783 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f72e0057d70 con 0x7f72ec0d29c0 2024-03-06T07:25:50.457 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:50.457 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":81,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:48.732524+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:45.368088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:46.056543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:50.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72cc06b4d0 msgr2=0x7f72cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72cc06b4d0 0x7f72cc06d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f72dc036000 tx=0x7f72dc030000 comp rx=0 tx=0).stop 2024-03-06T07:25:50.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1160512783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ec0d29c0 msgr2=0x7f72ec14ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ec0d29c0 0x7f72ec14ef80 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f72e000c440 tx=0x7f72e0001e70 comp rx=0 tx=0).stop 2024-03-06T07:25:50.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1160512783 shutdown_connections 2024-03-06T07:25:50.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72cc06b4d0 0x7f72cc06d990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72ec15dfe0 0x7f72ec0cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72ec0d29c0 0x7f72ec14ef80 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 --2- 172.21.15.59:0/1160512783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72ec0cf830 0x7f72ec1f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1160512783 >> 172.21.15.59:0/1160512783 conn(0x7f72ec0d3cd0 msgr2=0x7f72ec06b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1160512783 shutdown_connections 2024-03-06T07:25:50.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.456+0000 7f72f0fd0640 1 -- 172.21.15.59:0/1160512783 wait complete. 2024-03-06T07:25:50.469 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:50.469 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.569+0000 7ff2e873a640 1 Processor -- start 2024-03-06T07:25:50.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.569+0000 7ff2e873a640 1 -- start start 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.570+0000 7ff2e873a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff2e015bb20 0x7ff2e00d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.570+0000 7ff2e873a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e00cd8c0 0x7ff2e00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.570+0000 7ff2e873a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e0161980 con 0x7ff2e00cfff0 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.570+0000 7ff2e873a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e0161c60 con 0x7ff2e015bb20 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.570+0000 7ff2e873a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e0161f40 con 0x7ff2e00cd8c0 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.571+0000 7ff2e64d6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff2e015bb20 0x7ff2e00d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.571+0000 7ff2e64d6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff2e015bb20 0x7ff2e00d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46578/0 (socket says 172.21.15.59:46578) 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.571+0000 7ff2e64d6640 1 -- 172.21.15.59:0/4133520719 learned_addr learned my addr 172.21.15.59:0/4133520719 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.571+0000 7ff2e6cd7640 1 --2- 172.21.15.59:0/4133520719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e00cd8c0 0x7ff2e00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.571+0000 7ff2e64d6640 1 -- 172.21.15.59:0/4133520719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e00cd8c0 msgr2=0x7ff2e00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.571+0000 7ff2e64d6640 1 --2- 172.21.15.59:0/4133520719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e00cd8c0 0x7ff2e00d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.571+0000 7ff2e64d6640 1 -- 172.21.15.59:0/4133520719 >> v1:172.21.15.164:6789/0 conn(0x7ff2e00cfff0 legacy=0x7ff2e00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.571+0000 7ff2e64d6640 1 -- 172.21.15.59:0/4133520719 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e0162220 con 0x7ff2e015bb20 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.572+0000 7ff2e64d6640 1 --2- 172.21.15.59:0/4133520719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e015bb20 0x7ff2e00d1df0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc009870 tx=0x7ff2dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea7b7efcf9dbbb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.572+0000 7ff2e54d4640 1 -- 172.21.15.59:0/4133520719 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc034070 con 0x7ff2e015bb20 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.572+0000 7ff2e54d4640 1 -- 172.21.15.59:0/4133520719 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2dc030840 con 0x7ff2e015bb20 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.572+0000 7ff2e54d4640 1 -- 172.21.15.59:0/4133520719 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc030c10 con 0x7ff2e015bb20 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.572+0000 7ff2e873a640 1 -- 172.21.15.59:0/4133520719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e015bb20 msgr2=0x7ff2e00d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.572+0000 7ff2e873a640 1 --2- 172.21.15.59:0/4133520719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e015bb20 0x7ff2e00d1df0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc009870 tx=0x7ff2dc02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.573+0000 7ff2e873a640 1 -- 172.21.15.59:0/4133520719 shutdown_connections 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.573+0000 7ff2e873a640 1 --2- 172.21.15.59:0/4133520719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e00cd8c0 0x7ff2e00d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.573+0000 7ff2e873a640 1 --2- 172.21.15.59:0/4133520719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e015bb20 0x7ff2e00d1df0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.573+0000 7ff2e873a640 1 -- 172.21.15.59:0/4133520719 >> 172.21.15.59:0/4133520719 conn(0x7ff2e009a9f0 msgr2=0x7ff2e009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.573+0000 7ff2e873a640 1 -- 172.21.15.59:0/4133520719 shutdown_connections 2024-03-06T07:25:50.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.573+0000 7ff2e873a640 1 -- 172.21.15.59:0/4133520719 wait complete. 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.574+0000 7ff2e873a640 1 Processor -- start 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.574+0000 7ff2e873a640 1 -- start start 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.574+0000 7ff2e873a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e00cd8c0 0x7ff2e015e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.574+0000 7ff2e873a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00cfff0 0x7ff2e00c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e873a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e015bb20 0x7ff2e014ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e873a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e00d24b0 con 0x7ff2e00cfff0 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e873a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e00d2330 con 0x7ff2e015bb20 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e873a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e00d1980 con 0x7ff2e00cd8c0 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e5cd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00cfff0 0x7ff2e00c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e5cd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00cfff0 0x7ff2e00c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46586/0 (socket says 172.21.15.59:46586) 2024-03-06T07:25:50.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e5cd5640 1 -- 172.21.15.59:0/2889998829 learned_addr learned my addr 172.21.15.59:0/2889998829 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e64d6640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e00cd8c0 0x7ff2e015e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e6cd7640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e015bb20 0x7ff2e014ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e5cd5640 1 -- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e00cd8c0 msgr2=0x7ff2e015e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e5cd5640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e00cd8c0 0x7ff2e015e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e5cd5640 1 -- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e015bb20 msgr2=0x7ff2e014ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e5cd5640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e015bb20 0x7ff2e014ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e5cd5640 1 -- 172.21.15.59:0/2889998829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e00c84b0 con 0x7ff2e00cfff0 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e64d6640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e00cd8c0 0x7ff2e015e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e5cd5640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00cfff0 0x7ff2e00c7f70 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000c400 tx=0x7ff2d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2d6ffd640 1 -- 172.21.15.59:0/2889998829 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d00052d0 con 0x7ff2e00cfff0 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2d6ffd640 1 -- 172.21.15.59:0/2889998829 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d0005490 con 0x7ff2e00cfff0 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e873a640 1 -- 172.21.15.59:0/2889998829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e015ea30 con 0x7ff2e00cfff0 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2d6ffd640 1 -- 172.21.15.59:0/2889998829 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d0007260 con 0x7ff2e00cfff0 2024-03-06T07:25:50.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.575+0000 7ff2e873a640 1 -- 172.21.15.59:0/2889998829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e015f2d0 con 0x7ff2e00cfff0 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.576+0000 7ff2e873a640 1 -- 172.21.15.59:0/2889998829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e004f330 con 0x7ff2e00cfff0 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.577+0000 7ff2d6ffd640 1 -- 172.21.15.59:0/2889998829 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2d0007400 con 0x7ff2e00cfff0 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.577+0000 7ff2d6ffd640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2c406b4d0 0x7ff2c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.577+0000 7ff2d6ffd640 1 -- 172.21.15.59:0/2889998829 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2d0089730 con 0x7ff2e00cfff0 2024-03-06T07:25:50.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.578+0000 7ff2e64d6640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2c406b4d0 0x7ff2c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.578+0000 7ff2e64d6640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2c406b4d0 0x7ff2c406d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc036000 tx=0x7ff2dc03b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.581+0000 7ff2d6ffd640 1 -- 172.21.15.59:0/2889998829 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2d00585b0 con 0x7ff2e00cfff0 2024-03-06T07:25:50.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.759+0000 7ff2e873a640 1 -- 172.21.15.59:0/2889998829 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2e00b1820 con 0x7ff2c406b4d0 2024-03-06T07:25:50.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.765+0000 7ff2d6ffd640 1 -- 172.21.15.59:0/2889998829 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240121 (secure 0 0 0) 0x7ff2e00b1820 con 0x7ff2c406b4d0 2024-03-06T07:25:50.770 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:50.771 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:25:50.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.768+0000 7ff2e873a640 1 -- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2c406b4d0 msgr2=0x7ff2c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.768+0000 7ff2e873a640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2c406b4d0 0x7ff2c406d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc036000 tx=0x7ff2dc03b040 comp rx=0 tx=0).stop 2024-03-06T07:25:50.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.768+0000 7ff2e873a640 1 -- 172.21.15.59:0/2889998829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00cfff0 msgr2=0x7ff2e00c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.768+0000 7ff2e873a640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00cfff0 0x7ff2e00c7f70 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000c400 tx=0x7ff2d000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.768+0000 7ff2e873a640 1 -- 172.21.15.59:0/2889998829 shutdown_connections 2024-03-06T07:25:50.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.768+0000 7ff2e873a640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2c406b4d0 0x7ff2c406d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.768+0000 7ff2e873a640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e015bb20 0x7ff2e014ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.768+0000 7ff2e873a640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2e00cfff0 0x7ff2e00c7f70 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.768+0000 7ff2e873a640 1 --2- 172.21.15.59:0/2889998829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e00cd8c0 0x7ff2e015e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.768+0000 7ff2e873a640 1 -- 172.21.15.59:0/2889998829 >> 172.21.15.59:0/2889998829 conn(0x7ff2e009a9f0 msgr2=0x7ff2e015b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.769+0000 7ff2e873a640 1 -- 172.21.15.59:0/2889998829 shutdown_connections 2024-03-06T07:25:50.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.769+0000 7ff2e873a640 1 -- 172.21.15.59:0/2889998829 wait complete. 2024-03-06T07:25:50.781 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-03-06T07:25:49.351481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258044,"kb_used_data":8552,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716676,"statfs":{"total":773094113280,"available":772829876224,"internally_reserved":0,"allocated":8757248,"data_stored":4360384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000931"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729992+0000","last_change":"2024-03-06T07:25:45.729992+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729992+0000","last_undegraded":"2024-03-06T07:25:45.729992+0000","last_fullsized":"2024-03-06T07:25:45.729992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729021+0000","last_change":"2024-03-06T07:25:45.729021+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729021+0000","last_undegraded":"2024-03-06T07:25:45.729021+0000","last_fullsized":"2024-03-06T07:25:45.729021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724803+0000","last_change":"2024-03-06T07:25:45.724803+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724803+0000","last_undegraded":"2024-03-06T07:25:45.724803+0000","last_fullsized":"2024-03-06T07:25:45.724803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.725364+0000","last_change":"2024-03-06T07:25:45.725364+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.725364+0000","last_undegraded":"2024-03-06T07:25:45.725364+0000","last_fullsized":"2024-03-06T07:25:45.725364+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729009+0000","last_change":"2024-03-06T07:25:45.729009+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729009+0000","last_undegraded":"2024-03-06T07:25:45.729009+0000","last_fullsized":"2024-03-06T07:25:45.729009+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728203+0000","last_change":"2024-03-06T07:25:45.728203+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728203+0000","last_undegraded":"2024-03-06T07:25:45.728203+0000","last_fullsized":"2024-03-06T07:25:45.728203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724767+0000","last_change":"2024-03-06T07:25:45.724767+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724767+0000","last_undegraded":"2024-03-06T07:25:45.724767+0000","last_fullsized":"2024-03-06T07:25:45.724767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.723194+0000","last_change":"2024-03-06T07:25:45.723194+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.723194+0000","last_undegraded":"2024-03-06T07:25:45.723194+0000","last_fullsized":"2024-03-06T07:25:45.723194+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729420+0000","last_change":"2024-03-06T07:25:45.729420+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729420+0000","last_undegraded":"2024-03-06T07:25:45.729420+0000","last_fullsized":"2024-03-06T07:25:45.729420+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728800+0000","last_change":"2024-03-06T07:25:45.728800+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728800+0000","last_undegraded":"2024-03-06T07:25:45.728800+0000","last_fullsized":"2024-03-06T07:25:45.728800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728545+0000","last_change":"2024-03-06T07:25:45.728545+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728545+0000","last_undegraded":"2024-03-06T07:25:45.728545+0000","last_fullsized":"2024-03-06T07:25:45.728545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.726711+0000","last_change":"2024-03-06T07:25:45.726711+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.726711+0000","last_undegraded":"2024-03-06T07:25:45.726711+0000","last_fullsized":"2024-03-06T07:25:45.726711+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.723201+0000","last_change":"2024-03-06T07:25:45.723201+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.723201+0000","last_undegraded":"2024-03-06T07:25:45.723201+0000","last_fullsized":"2024-03-06T07:25:45.723201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.727839+0000","last_change":"2024-03-06T07:25:45.727839+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.727839+0000","last_undegraded":"2024-03-06T07:25:45.727839+0000","last_fullsized":"2024-03-06T07:25:45.727839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.726680+0000","last_change":"2024-03-06T07:25:45.726680+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.726680+0000","last_undegraded":"2024-03-06T07:25:45.726680+0000","last_fullsized":"2024-03-06T07:25:45.726680+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724779+0000","last_change":"2024-03-06T07:25:45.724779+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724779+0000","last_undegraded":"2024-03-06T07:25:45.724779+0000","last_fullsized":"2024-03-06T07:25:45.724779+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728561+0000","last_change":"2024-03-06T07:25:45.728561+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728561+0000","last_undegraded":"2024-03-06T07:25:45.728561+0000","last_fullsized":"2024-03-06T07:25:45.728561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724849+0000","last_change":"2024-03-06T07:25:45.724849+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724849+0000","last_undegraded":"2024-03-06T07:25:45.724849+0000","last_fullsized":"2024-03-06T07:25:45.724849+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724769+0000","last_change":"2024-03-06T07:25:45.724769+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724769+0000","last_undegraded":"2024-03-06T07:25:45.724769+0000","last_fullsized":"2024-03-06T07:25:45.724769+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.727213+0000","last_change":"2024-03-06T07:25:45.727213+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.727213+0000","last_undegraded":"2024-03-06T07:25:45.727213+0000","last_fullsized":"2024-03-06T07:25:45.727213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.727070+0000","last_change":"2024-03-06T07:25:45.727070+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.727070+0000","last_undegraded":"2024-03-06T07:25:45.727070+0000","last_fullsized":"2024-03-06T07:25:45.727070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.727027+0000","last_change":"2024-03-06T07:25:45.727027+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.727027+0000","last_undegraded":"2024-03-06T07:25:45.727027+0000","last_fullsized":"2024-03-06T07:25:45.727027+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728235+0000","last_change":"2024-03-06T07:25:45.728235+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728235+0000","last_undegraded":"2024-03-06T07:25:45.728235+0000","last_fullsized":"2024-03-06T07:25:45.728235+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.723194+0000","last_change":"2024-03-06T07:25:45.723194+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.723194+0000","last_undegraded":"2024-03-06T07:25:45.723194+0000","last_fullsized":"2024-03-06T07:25:45.723194+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.723177+0000","last_change":"2024-03-06T07:25:45.723177+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.723177+0000","last_undegraded":"2024-03-06T07:25:45.723177+0000","last_fullsized":"2024-03-06T07:25:45.723177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729143+0000","last_change":"2024-03-06T07:25:45.729143+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729143+0000","last_undegraded":"2024-03-06T07:25:45.729143+0000","last_fullsized":"2024-03-06T07:25:45.729143+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728826+0000","last_change":"2024-03-06T07:25:45.728826+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728826+0000","last_undegraded":"2024-03-06T07:25:45.728826+0000","last_fullsized":"2024-03-06T07:25:45.728826+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724795+0000","last_change":"2024-03-06T07:25:45.724795+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724795+0000","last_undegraded":"2024-03-06T07:25:45.724795+0000","last_fullsized":"2024-03-06T07:25:45.724795+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.727769+0000","last_change":"2024-03-06T07:25:45.727769+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.727769+0000","last_undegraded":"2024-03-06T07:25:45.727769+0000","last_fullsized":"2024-03-06T07:25:45.727769+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728506+0000","last_change":"2024-03-06T07:25:45.728506+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728506+0000","last_undegraded":"2024-03-06T07:25:45.728506+0000","last_fullsized":"2024-03-06T07:25:45.728506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.726152+0000","last_change":"2024-03-06T07:25:45.726152+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.726152+0000","last_undegraded":"2024-03-06T07:25:45.726152+0000","last_fullsized":"2024-03-06T07:25:45.726152+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.725861+0000","last_change":"2024-03-06T07:25:45.725861+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.725861+0000","last_undegraded":"2024-03-06T07:25:45.725861+0000","last_fullsized":"2024-03-06T07:25:45.725861+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":275,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:25:45.725226+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:25:45.725226+0000","last_peered":"2024-03-06T07:25:45.725226+0000","last_clean":"2024-03-06T07:25:45.725226+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:25:45.725226+0000","last_undegraded":"2024-03-06T07:25:45.725226+0000","last_fullsized":"2024-03-06T07:25:45.725226+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1540096,"data_stored":987539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":402988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":958464,"data_stored":402988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32084,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339756,"statfs":{"total":96636764160,"available":96603910144,"internally_reserved":0,"allocated":921600,"data_stored":387057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":933888,"data_stored":392114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":946176,"data_stored":397171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":958464,"data_stored":402988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1540096,"data_stored":987539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:50.785 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:50.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.924+0000 7ffb7b6c3640 1 Processor -- start 2024-03-06T07:25:50.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.924+0000 7ffb7b6c3640 1 -- start start 2024-03-06T07:25:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.925+0000 7ffb7b6c3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffb740d22b0 0x7ffb740d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.925+0000 7ffb7b6c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb740b41f0 0x7ffb740b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.925+0000 7ffb7b6c3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb740b3430 con 0x7ffb7415dfe0 2024-03-06T07:25:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.925+0000 7ffb7b6c3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb740d2c50 con 0x7ffb740d22b0 2024-03-06T07:25:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.925+0000 7ffb7b6c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740b31e0 con 0x7ffb740b41f0 2024-03-06T07:25:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.925+0000 7ffb78c5e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffb740d22b0 0x7ffb740d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.925+0000 7ffb7945f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ffb7415dfe0 0x7ffb740cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50010/0 (socket says 172.21.15.59:50010) 2024-03-06T07:25:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.925+0000 7ffb7945f640 1 -- 172.21.15.59:0/1820301096 learned_addr learned my addr 172.21.15.59:0/1820301096 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.926+0000 7ffb78c5e640 1 -- 172.21.15.59:0/1820301096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb740b41f0 msgr2=0x7ffb740b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.926+0000 7ffb79c60640 1 --2- 172.21.15.59:0/1820301096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb740b41f0 0x7ffb740b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.926+0000 7ffb6bfff640 1 -- 172.21.15.59:0/1820301096 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712722943 0 0) 0x7ffb740b3430 con 0x7ffb7415dfe0 2024-03-06T07:25:50.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.926+0000 7ffb78c5e640 1 --2- 172.21.15.59:0/1820301096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb740b41f0 0x7ffb740b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.926+0000 7ffb78c5e640 1 -- 172.21.15.59:0/1820301096 >> v1:172.21.15.59:6789/0 conn(0x7ffb7415dfe0 legacy=0x7ffb740cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.926+0000 7ffb78c5e640 1 -- 172.21.15.59:0/1820301096 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb740b3710 con 0x7ffb740d22b0 2024-03-06T07:25:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.927+0000 7ffb78c5e640 1 --2- 172.21.15.59:0/1820301096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb740d22b0 0x7ffb740d2710 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ffb640097a0 tx=0x7ffb6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d3ac25684ab553 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.927+0000 7ffb6bfff640 1 -- 172.21.15.59:0/1820301096 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64035020 con 0x7ffb740d22b0 2024-03-06T07:25:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.927+0000 7ffb6bfff640 1 -- 172.21.15.59:0/1820301096 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb64031830 con 0x7ffb740d22b0 2024-03-06T07:25:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.927+0000 7ffb6bfff640 1 -- 172.21.15.59:0/1820301096 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64031c00 con 0x7ffb740d22b0 2024-03-06T07:25:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.927+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1820301096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb740d22b0 msgr2=0x7ffb740d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.927+0000 7ffb7b6c3640 1 --2- 172.21.15.59:0/1820301096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb740d22b0 0x7ffb740d2710 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ffb640097a0 tx=0x7ffb6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.927+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1820301096 shutdown_connections 2024-03-06T07:25:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.927+0000 7ffb7b6c3640 1 --2- 172.21.15.59:0/1820301096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb740b41f0 0x7ffb740b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.927+0000 7ffb7b6c3640 1 --2- 172.21.15.59:0/1820301096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb740d22b0 0x7ffb740d2710 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.927+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1820301096 >> 172.21.15.59:0/1820301096 conn(0x7ffb740d3cd0 msgr2=0x7ffb740d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.928+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1820301096 shutdown_connections 2024-03-06T07:25:50.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.928+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1820301096 wait complete. 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.929+0000 7ffb7b6c3640 1 Processor -- start 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.929+0000 7ffb7b6c3640 1 -- start start 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.929+0000 7ffb7b6c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb740b41f0 0x7ffb740b9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.929+0000 7ffb7b6c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb740d22b0 0x7ffb740cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.929+0000 7ffb7b6c3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb7415dfe0 0x7ffb7415b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.929+0000 7ffb7b6c3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740cfa00 con 0x7ffb740d22b0 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.929+0000 7ffb7b6c3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740b3d60 con 0x7ffb7415dfe0 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.929+0000 7ffb7b6c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740d1c70 con 0x7ffb740b41f0 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.929+0000 7ffb78c5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb740d22b0 0x7ffb740cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb78c5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb740d22b0 0x7ffb740cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46604/0 (socket says 172.21.15.59:46604) 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb78c5e640 1 -- 172.21.15.59:0/1243390644 learned_addr learned my addr 172.21.15.59:0/1243390644 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:50.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb7945f640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb740b41f0 0x7ffb740b9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb79c60640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb7415dfe0 0x7ffb7415b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb78c5e640 1 -- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb740b41f0 msgr2=0x7ffb740b9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb78c5e640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb740b41f0 0x7ffb740b9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb78c5e640 1 -- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb7415dfe0 msgr2=0x7ffb7415b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb78c5e640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb7415dfe0 0x7ffb7415b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb78c5e640 1 -- 172.21.15.59:0/1243390644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb740cb510 con 0x7ffb740d22b0 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb78c5e640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb740d22b0 0x7ffb740cada0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ffb6402bee0 tx=0x7ffb640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb69ffb640 1 -- 172.21.15.59:0/1243390644 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64041070 con 0x7ffb740d22b0 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb69ffb640 1 -- 172.21.15.59:0/1243390644 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6403a950 con 0x7ffb740d22b0 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1243390644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb740cb7f0 con 0x7ffb740d22b0 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb69ffb640 1 -- 172.21.15.59:0/1243390644 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64035040 con 0x7ffb740d22b0 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.930+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1243390644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb740cc330 con 0x7ffb740d22b0 2024-03-06T07:25:50.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.931+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1243390644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb740cbd10 con 0x7ffb740d22b0 2024-03-06T07:25:50.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.932+0000 7ffb69ffb640 1 -- 172.21.15.59:0/1243390644 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb6404c060 con 0x7ffb740d22b0 2024-03-06T07:25:50.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.932+0000 7ffb69ffb640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb5806b500 0x7ffb5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.933+0000 7ffb69ffb640 1 -- 172.21.15.59:0/1243390644 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb640abb80 con 0x7ffb740d22b0 2024-03-06T07:25:50.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.933+0000 7ffb7945f640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb5806b500 0x7ffb5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.933+0000 7ffb7945f640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb5806b500 0x7ffb5806d9c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ffb70000f60 tx=0x7ffb70007c00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:50.937+0000 7ffb69ffb640 1 -- 172.21.15.59:0/1243390644 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb740cbd10 con 0x7ffb740d22b0 2024-03-06T07:25:51.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.115+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1243390644 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb7404f330 con 0x7ffb5806b500 2024-03-06T07:25:51.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.121+0000 7ffb69ffb640 1 -- 172.21.15.59:0/1243390644 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240121 (secure 0 0 0) 0x7ffb7404f330 con 0x7ffb5806b500 2024-03-06T07:25:51.125 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:51.126 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:25:51.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.124+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb5806b500 msgr2=0x7ffb5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.124+0000 7ffb7b6c3640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb5806b500 0x7ffb5806d9c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ffb70000f60 tx=0x7ffb70007c00 comp rx=0 tx=0).stop 2024-03-06T07:25:51.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.125+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1243390644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb740d22b0 msgr2=0x7ffb740cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.125+0000 7ffb7b6c3640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb740d22b0 0x7ffb740cada0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ffb6402bee0 tx=0x7ffb640046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.125+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1243390644 shutdown_connections 2024-03-06T07:25:51.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.125+0000 7ffb7b6c3640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb5806b500 0x7ffb5806d9c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.125+0000 7ffb7b6c3640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb7415dfe0 0x7ffb7415b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.125+0000 7ffb7b6c3640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb740d22b0 0x7ffb740cada0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.125+0000 7ffb7b6c3640 1 --2- 172.21.15.59:0/1243390644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb740b41f0 0x7ffb740b9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.125+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1243390644 >> 172.21.15.59:0/1243390644 conn(0x7ffb740d3cd0 msgr2=0x7ffb7406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.125+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1243390644 shutdown_connections 2024-03-06T07:25:51.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.125+0000 7ffb7b6c3640 1 -- 172.21.15.59:0/1243390644 wait complete. 2024-03-06T07:25:51.137 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-03-06T07:25:49.351481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258044,"kb_used_data":8552,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716676,"statfs":{"total":773094113280,"available":772829876224,"internally_reserved":0,"allocated":8757248,"data_stored":4360384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000931"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729992+0000","last_change":"2024-03-06T07:25:45.729992+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729992+0000","last_undegraded":"2024-03-06T07:25:45.729992+0000","last_fullsized":"2024-03-06T07:25:45.729992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729021+0000","last_change":"2024-03-06T07:25:45.729021+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729021+0000","last_undegraded":"2024-03-06T07:25:45.729021+0000","last_fullsized":"2024-03-06T07:25:45.729021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724803+0000","last_change":"2024-03-06T07:25:45.724803+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724803+0000","last_undegraded":"2024-03-06T07:25:45.724803+0000","last_fullsized":"2024-03-06T07:25:45.724803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.725364+0000","last_change":"2024-03-06T07:25:45.725364+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.725364+0000","last_undegraded":"2024-03-06T07:25:45.725364+0000","last_fullsized":"2024-03-06T07:25:45.725364+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729009+0000","last_change":"2024-03-06T07:25:45.729009+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729009+0000","last_undegraded":"2024-03-06T07:25:45.729009+0000","last_fullsized":"2024-03-06T07:25:45.729009+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728203+0000","last_change":"2024-03-06T07:25:45.728203+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728203+0000","last_undegraded":"2024-03-06T07:25:45.728203+0000","last_fullsized":"2024-03-06T07:25:45.728203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724767+0000","last_change":"2024-03-06T07:25:45.724767+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724767+0000","last_undegraded":"2024-03-06T07:25:45.724767+0000","last_fullsized":"2024-03-06T07:25:45.724767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.723194+0000","last_change":"2024-03-06T07:25:45.723194+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.723194+0000","last_undegraded":"2024-03-06T07:25:45.723194+0000","last_fullsized":"2024-03-06T07:25:45.723194+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729420+0000","last_change":"2024-03-06T07:25:45.729420+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729420+0000","last_undegraded":"2024-03-06T07:25:45.729420+0000","last_fullsized":"2024-03-06T07:25:45.729420+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728800+0000","last_change":"2024-03-06T07:25:45.728800+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728800+0000","last_undegraded":"2024-03-06T07:25:45.728800+0000","last_fullsized":"2024-03-06T07:25:45.728800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728545+0000","last_change":"2024-03-06T07:25:45.728545+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728545+0000","last_undegraded":"2024-03-06T07:25:45.728545+0000","last_fullsized":"2024-03-06T07:25:45.728545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.726711+0000","last_change":"2024-03-06T07:25:45.726711+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.726711+0000","last_undegraded":"2024-03-06T07:25:45.726711+0000","last_fullsized":"2024-03-06T07:25:45.726711+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.723201+0000","last_change":"2024-03-06T07:25:45.723201+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.723201+0000","last_undegraded":"2024-03-06T07:25:45.723201+0000","last_fullsized":"2024-03-06T07:25:45.723201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.727839+0000","last_change":"2024-03-06T07:25:45.727839+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.727839+0000","last_undegraded":"2024-03-06T07:25:45.727839+0000","last_fullsized":"2024-03-06T07:25:45.727839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.726680+0000","last_change":"2024-03-06T07:25:45.726680+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.726680+0000","last_undegraded":"2024-03-06T07:25:45.726680+0000","last_fullsized":"2024-03-06T07:25:45.726680+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724779+0000","last_change":"2024-03-06T07:25:45.724779+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724779+0000","last_undegraded":"2024-03-06T07:25:45.724779+0000","last_fullsized":"2024-03-06T07:25:45.724779+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728561+0000","last_change":"2024-03-06T07:25:45.728561+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728561+0000","last_undegraded":"2024-03-06T07:25:45.728561+0000","last_fullsized":"2024-03-06T07:25:45.728561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724849+0000","last_change":"2024-03-06T07:25:45.724849+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724849+0000","last_undegraded":"2024-03-06T07:25:45.724849+0000","last_fullsized":"2024-03-06T07:25:45.724849+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724769+0000","last_change":"2024-03-06T07:25:45.724769+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724769+0000","last_undegraded":"2024-03-06T07:25:45.724769+0000","last_fullsized":"2024-03-06T07:25:45.724769+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.727213+0000","last_change":"2024-03-06T07:25:45.727213+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.727213+0000","last_undegraded":"2024-03-06T07:25:45.727213+0000","last_fullsized":"2024-03-06T07:25:45.727213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.727070+0000","last_change":"2024-03-06T07:25:45.727070+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.727070+0000","last_undegraded":"2024-03-06T07:25:45.727070+0000","last_fullsized":"2024-03-06T07:25:45.727070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.727027+0000","last_change":"2024-03-06T07:25:45.727027+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.727027+0000","last_undegraded":"2024-03-06T07:25:45.727027+0000","last_fullsized":"2024-03-06T07:25:45.727027+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728235+0000","last_change":"2024-03-06T07:25:45.728235+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728235+0000","last_undegraded":"2024-03-06T07:25:45.728235+0000","last_fullsized":"2024-03-06T07:25:45.728235+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.723194+0000","last_change":"2024-03-06T07:25:45.723194+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.723194+0000","last_undegraded":"2024-03-06T07:25:45.723194+0000","last_fullsized":"2024-03-06T07:25:45.723194+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.723177+0000","last_change":"2024-03-06T07:25:45.723177+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.723177+0000","last_undegraded":"2024-03-06T07:25:45.723177+0000","last_fullsized":"2024-03-06T07:25:45.723177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729143+0000","last_change":"2024-03-06T07:25:45.729143+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729143+0000","last_undegraded":"2024-03-06T07:25:45.729143+0000","last_fullsized":"2024-03-06T07:25:45.729143+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728826+0000","last_change":"2024-03-06T07:25:45.728826+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728826+0000","last_undegraded":"2024-03-06T07:25:45.728826+0000","last_fullsized":"2024-03-06T07:25:45.728826+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.724795+0000","last_change":"2024-03-06T07:25:45.724795+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.724795+0000","last_undegraded":"2024-03-06T07:25:45.724795+0000","last_fullsized":"2024-03-06T07:25:45.724795+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.727769+0000","last_change":"2024-03-06T07:25:45.727769+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.727769+0000","last_undegraded":"2024-03-06T07:25:45.727769+0000","last_fullsized":"2024-03-06T07:25:45.727769+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728506+0000","last_change":"2024-03-06T07:25:45.728506+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728506+0000","last_undegraded":"2024-03-06T07:25:45.728506+0000","last_fullsized":"2024-03-06T07:25:45.728506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.726152+0000","last_change":"2024-03-06T07:25:45.726152+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.726152+0000","last_undegraded":"2024-03-06T07:25:45.726152+0000","last_fullsized":"2024-03-06T07:25:45.726152+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.725861+0000","last_change":"2024-03-06T07:25:45.725861+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.725861+0000","last_undegraded":"2024-03-06T07:25:45.725861+0000","last_fullsized":"2024-03-06T07:25:45.725861+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":275,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-06T07:25:45.725226+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:25:45.725226+0000","last_peered":"2024-03-06T07:25:45.725226+0000","last_clean":"2024-03-06T07:25:45.725226+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:25:45.725226+0000","last_undegraded":"2024-03-06T07:25:45.725226+0000","last_fullsized":"2024-03-06T07:25:45.725226+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1540096,"data_stored":987539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":402988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":958464,"data_stored":402988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32084,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339756,"statfs":{"total":96636764160,"available":96603910144,"internally_reserved":0,"allocated":921600,"data_stored":387057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":933888,"data_stored":392114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32112,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339728,"statfs":{"total":96636764160,"available":96603881472,"internally_reserved":0,"allocated":946176,"data_stored":397171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":958464,"data_stored":402988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32688,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339152,"statfs":{"total":96636764160,"available":96603291648,"internally_reserved":0,"allocated":1540096,"data_stored":987539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:51.141 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:51.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.277+0000 7f9c4f8f4640 1 Processor -- start 2024-03-06T07:25:51.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.277+0000 7f9c4f8f4640 1 -- start start 2024-03-06T07:25:51.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.278+0000 7f9c4f8f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c480d04a0 0x7f9c480ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.278+0000 7f9c4f8f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c480d0910 con 0x7f9c480cfff0 2024-03-06T07:25:51.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.278+0000 7f9c4f8f4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c48161980 con 0x7f9c4815bb20 2024-03-06T07:25:51.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.278+0000 7f9c4f8f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480d2500 con 0x7f9c480d04a0 2024-03-06T07:25:51.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.278+0000 7f9c4ce8f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9c480cfff0 0x7f9c480cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50024/0 (socket says 172.21.15.59:50024) 2024-03-06T07:25:51.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.278+0000 7f9c4ce8f640 1 -- 172.21.15.59:0/1905502135 learned_addr learned my addr 172.21.15.59:0/1905502135 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:51.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.278+0000 7f9c4de91640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c480d04a0 0x7f9c480ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.278+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431873480 0 0) 0x7f9c480d0910 con 0x7f9c480cfff0 2024-03-06T07:25:51.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.278+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c34003520 con 0x7f9c480cfff0 2024-03-06T07:25:51.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915020999 0 0) 0x7f9c48161980 con 0x7f9c4815bb20 2024-03-06T07:25:51.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c480d0910 con 0x7f9c4815bb20 2024-03-06T07:25:51.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2616340275 0 0) 0x7f9c34003520 con 0x7f9c480cfff0 2024-03-06T07:25:51.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c480d04a0 msgr2=0x7f9c480ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c480d04a0 0x7f9c480ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 >> v1:172.21.15.164:6789/0 conn(0x7f9c4815bb20 legacy=0x7f9c480d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c48161c60 con 0x7f9c480cfff0 2024-03-06T07:25:51.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f9c3c001670 con 0x7f9c480cfff0 2024-03-06T07:25:51.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c4de91640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c480d04a0 0x7f9c480ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:51.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 >> v1:172.21.15.59:6789/0 conn(0x7f9c480cfff0 legacy=0x7f9c480cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c34003b70 0x7f9c34003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c480d04a0 0x7f9c34004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c34004d40 0x7f9c340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480d2500 con 0x7f9c34004d40 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3400ba70 con 0x7f9c34003b70 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3400bbf0 con 0x7f9c480d04a0 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c4de91640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c34004d40 0x7f9c340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c4ce8f640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c480d04a0 0x7f9c34004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c4d690640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c34003b70 0x7f9c34003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c4de91640 1 -- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c480d04a0 msgr2=0x7f9c34004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c4de91640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c480d04a0 0x7f9c34004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c4de91640 1 -- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c34003b70 msgr2=0x7f9c34003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c4de91640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c34003b70 0x7f9c34003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.279+0000 7f9c4de91640 1 -- 172.21.15.59:0/1905502135 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c34003520 con 0x7f9c34004d40 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.280+0000 7f9c4de91640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c34004d40 0x7f9c340051c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000e380 tx=0x7f9c34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e92a70a61df098c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.280+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40039070 con 0x7f9c34004d40 2024-03-06T07:25:51.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.280+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4002cd70 con 0x7f9c34004d40 2024-03-06T07:25:51.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.280+0000 7f9c3bfff640 1 -- 172.21.15.59:0/1905502135 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40009de0 con 0x7f9c34004d40 2024-03-06T07:25:51.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.281+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/1905502135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c34004d40 msgr2=0x7f9c340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.281+0000 7f9c4f8f4640 1 --2- 172.21.15.59:0/1905502135 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c34004d40 0x7f9c340051c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000e380 tx=0x7f9c34005830 comp rx=0 tx=0).stop 2024-03-06T07:25:51.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.281+0000 7f9c4de91640 1 -- 172.21.15.59:0/1905502135 reap_dead start 2024-03-06T07:25:51.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.281+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/1905502135 shutdown_connections 2024-03-06T07:25:51.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.281+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/1905502135 >> 172.21.15.59:0/1905502135 conn(0x7f9c4809a9f0 msgr2=0x7f9c4809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.281+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/1905502135 shutdown_connections 2024-03-06T07:25:51.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.281+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/1905502135 wait complete. 2024-03-06T07:25:51.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.282+0000 7f9c4f8f4640 1 Processor -- start 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.282+0000 7f9c4f8f4640 1 -- start start 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4f8f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c34004d40 0x7f9c480b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4f8f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c481e81f0 0x7f9c480afbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4f8f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c481584c0 0x7f9c48158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4f8f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48162500 con 0x7f9c481584c0 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4f8f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48162220 con 0x7f9c481e81f0 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4f8f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c480d1980 con 0x7f9c34004d40 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4de91640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c481584c0 0x7f9c48158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4de91640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c481584c0 0x7f9c48158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46626/0 (socket says 172.21.15.59:46626) 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4de91640 1 -- 172.21.15.59:0/844353505 learned_addr learned my addr 172.21.15.59:0/844353505 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4d690640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c34004d40 0x7f9c480b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4ce8f640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c481e81f0 0x7f9c480afbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4de91640 1 -- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c34004d40 msgr2=0x7f9c480b4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4de91640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c34004d40 0x7f9c480b4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4de91640 1 -- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c481e81f0 msgr2=0x7f9c480afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4de91640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c481e81f0 0x7f9c480afbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4de91640 1 -- 172.21.15.59:0/844353505 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c481e8670 con 0x7f9c481584c0 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4de91640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c481584c0 0x7f9c48158940 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f9c40037040 tx=0x7f9c40009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c39ffb640 1 -- 172.21.15.59:0/844353505 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40039030 con 0x7f9c481584c0 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c39ffb640 1 -- 172.21.15.59:0/844353505 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4003a040 con 0x7f9c481584c0 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.283+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/844353505 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c480b4da0 con 0x7f9c481584c0 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.284+0000 7f9c39ffb640 1 -- 172.21.15.59:0/844353505 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40009de0 con 0x7f9c481584c0 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.284+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/844353505 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c480b5360 con 0x7f9c481584c0 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.285+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/844353505 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4804f330 con 0x7f9c481584c0 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.285+0000 7f9c39ffb640 1 -- 172.21.15.59:0/844353505 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c40045030 con 0x7f9c481584c0 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.285+0000 7f9c39ffb640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c2c06b4d0 0x7f9c2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.285+0000 7f9c39ffb640 1 -- 172.21.15.59:0/844353505 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c400abba0 con 0x7f9c481584c0 2024-03-06T07:25:51.290 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.286+0000 7f9c4d690640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c2c06b4d0 0x7f9c2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.286+0000 7f9c4d690640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c2c06b4d0 0x7f9c2c06d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9c440018d0 tx=0x7f9c44009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.290+0000 7f9c39ffb640 1 -- 172.21.15.59:0/844353505 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c4004c060 con 0x7f9c481584c0 2024-03-06T07:25:51.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.468+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/844353505 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9c480b1820 con 0x7f9c2c06b4d0 2024-03-06T07:25:51.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.474+0000 7f9c39ffb640 1 -- 172.21.15.59:0/844353505 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+249919 (secure 0 0 0) 0x7f9c480b1820 con 0x7f9c2c06b4d0 2024-03-06T07:25:51.478 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:51.480 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:25:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c2c06b4d0 msgr2=0x7f9c2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c2c06b4d0 0x7f9c2c06d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9c440018d0 tx=0x7f9c44009040 comp rx=0 tx=0).stop 2024-03-06T07:25:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/844353505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c481584c0 msgr2=0x7f9c48158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c481584c0 0x7f9c48158940 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f9c40037040 tx=0x7f9c40009750 comp rx=0 tx=0).stop 2024-03-06T07:25:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/844353505 shutdown_connections 2024-03-06T07:25:51.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c2c06b4d0 0x7f9c2c06d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c481584c0 0x7f9c48158940 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c481e81f0 0x7f9c480afbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 --2- 172.21.15.59:0/844353505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c34004d40 0x7f9c480b4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/844353505 >> 172.21.15.59:0/844353505 conn(0x7f9c4809a9f0 msgr2=0x7f9c480b3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/844353505 shutdown_connections 2024-03-06T07:25:51.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.477+0000 7f9c4f8f4640 1 -- 172.21.15.59:0/844353505 wait complete. 2024-03-06T07:25:51.489 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-03-06T07:25:51.351996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592084,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":195,"num_write_kb":2427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":172,"acting":172,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258424,"kb_used_data":8952,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716296,"statfs":{"total":773094113280,"available":772829487104,"internally_reserved":0,"allocated":9166848,"data_stored":4544216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1716,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.627688"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760083+0000","last_change":"2024-03-06T07:25:47.754942+0000","last_active":"2024-03-06T07:25:48.760083+0000","last_peered":"2024-03-06T07:25:48.760083+0000","last_clean":"2024-03-06T07:25:48.760083+0000","last_became_active":"2024-03-06T07:25:47.753877+0000","last_became_peered":"2024-03-06T07:25:47.753877+0000","last_unstale":"2024-03-06T07:25:48.760083+0000","last_undegraded":"2024-03-06T07:25:48.760083+0000","last_fullsized":"2024-03-06T07:25:48.760083+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:52.540026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040115+0000","last_change":"2024-03-06T07:25:47.750010+0000","last_active":"2024-03-06T07:25:49.040115+0000","last_peered":"2024-03-06T07:25:49.040115+0000","last_clean":"2024-03-06T07:25:49.040115+0000","last_became_active":"2024-03-06T07:25:47.749735+0000","last_became_peered":"2024-03-06T07:25:47.749735+0000","last_unstale":"2024-03-06T07:25:49.040115+0000","last_undegraded":"2024-03-06T07:25:49.040115+0000","last_fullsized":"2024-03-06T07:25:49.040115+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:41.897733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736087+0000","last_change":"2024-03-06T07:25:47.750010+0000","last_active":"2024-03-06T07:25:48.736087+0000","last_peered":"2024-03-06T07:25:48.736087+0000","last_clean":"2024-03-06T07:25:48.736087+0000","last_became_active":"2024-03-06T07:25:47.749708+0000","last_became_peered":"2024-03-06T07:25:47.749708+0000","last_unstale":"2024-03-06T07:25:48.736087+0000","last_undegraded":"2024-03-06T07:25:48.736087+0000","last_fullsized":"2024-03-06T07:25:48.736087+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:49.954151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737396+0000","last_change":"2024-03-06T07:25:47.747729+0000","last_active":"2024-03-06T07:25:48.737396+0000","last_peered":"2024-03-06T07:25:48.737396+0000","last_clean":"2024-03-06T07:25:48.737396+0000","last_became_active":"2024-03-06T07:25:47.747442+0000","last_became_peered":"2024-03-06T07:25:47.747442+0000","last_unstale":"2024-03-06T07:25:48.737396+0000","last_undegraded":"2024-03-06T07:25:48.737396+0000","last_fullsized":"2024-03-06T07:25:48.737396+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:53.366551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737419+0000","last_change":"2024-03-06T07:25:47.751731+0000","last_active":"2024-03-06T07:25:48.737419+0000","last_peered":"2024-03-06T07:25:48.737419+0000","last_clean":"2024-03-06T07:25:48.737419+0000","last_became_active":"2024-03-06T07:25:47.751445+0000","last_became_peered":"2024-03-06T07:25:47.751445+0000","last_unstale":"2024-03-06T07:25:48.737419+0000","last_undegraded":"2024-03-06T07:25:48.737419+0000","last_fullsized":"2024-03-06T07:25:48.737419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:08.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735762+0000","last_change":"2024-03-06T07:25:47.747479+0000","last_active":"2024-03-06T07:25:48.735762+0000","last_peered":"2024-03-06T07:25:48.735762+0000","last_clean":"2024-03-06T07:25:48.735762+0000","last_became_active":"2024-03-06T07:25:47.747186+0000","last_became_peered":"2024-03-06T07:25:47.747186+0000","last_unstale":"2024-03-06T07:25:48.735762+0000","last_undegraded":"2024-03-06T07:25:48.735762+0000","last_fullsized":"2024-03-06T07:25:48.735762+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:38.441834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736301+0000","last_change":"2024-03-06T07:25:47.742812+0000","last_active":"2024-03-06T07:25:48.736301+0000","last_peered":"2024-03-06T07:25:48.736301+0000","last_clean":"2024-03-06T07:25:48.736301+0000","last_became_active":"2024-03-06T07:25:47.739811+0000","last_became_peered":"2024-03-06T07:25:47.739811+0000","last_unstale":"2024-03-06T07:25:48.736301+0000","last_undegraded":"2024-03-06T07:25:48.736301+0000","last_fullsized":"2024-03-06T07:25:48.736301+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:17.837597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736920+0000","last_change":"2024-03-06T07:25:47.760422+0000","last_active":"2024-03-06T07:25:48.736920+0000","last_peered":"2024-03-06T07:25:48.736920+0000","last_clean":"2024-03-06T07:25:48.736920+0000","last_became_active":"2024-03-06T07:25:47.760129+0000","last_became_peered":"2024-03-06T07:25:47.760129+0000","last_unstale":"2024-03-06T07:25:48.736920+0000","last_undegraded":"2024-03-06T07:25:48.736920+0000","last_fullsized":"2024-03-06T07:25:48.736920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:52.495587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760008+0000","last_change":"2024-03-06T07:25:47.745385+0000","last_active":"2024-03-06T07:25:48.760008+0000","last_peered":"2024-03-06T07:25:48.760008+0000","last_clean":"2024-03-06T07:25:48.760008+0000","last_became_active":"2024-03-06T07:25:47.745117+0000","last_became_peered":"2024-03-06T07:25:47.745117+0000","last_unstale":"2024-03-06T07:25:48.760008+0000","last_undegraded":"2024-03-06T07:25:48.760008+0000","last_fullsized":"2024-03-06T07:25:48.760008+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:40.310991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039620+0000","last_change":"2024-03-06T07:25:47.749258+0000","last_active":"2024-03-06T07:25:49.039620+0000","last_peered":"2024-03-06T07:25:49.039620+0000","last_clean":"2024-03-06T07:25:49.039620+0000","last_became_active":"2024-03-06T07:25:47.748993+0000","last_became_peered":"2024-03-06T07:25:47.748993+0000","last_unstale":"2024-03-06T07:25:49.039620+0000","last_undegraded":"2024-03-06T07:25:49.039620+0000","last_fullsized":"2024-03-06T07:25:49.039620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:53.366551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735907+0000","last_change":"2024-03-06T07:25:47.750890+0000","last_active":"2024-03-06T07:25:48.735907+0000","last_peered":"2024-03-06T07:25:48.735907+0000","last_clean":"2024-03-06T07:25:48.735907+0000","last_became_active":"2024-03-06T07:25:47.750579+0000","last_became_peered":"2024-03-06T07:25:47.750579+0000","last_unstale":"2024-03-06T07:25:48.735907+0000","last_undegraded":"2024-03-06T07:25:48.735907+0000","last_fullsized":"2024-03-06T07:25:48.735907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:12.102084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039116+0000","last_change":"2024-03-06T07:25:47.749794+0000","last_active":"2024-03-06T07:25:49.039116+0000","last_peered":"2024-03-06T07:25:49.039116+0000","last_clean":"2024-03-06T07:25:49.039116+0000","last_became_active":"2024-03-06T07:25:47.749189+0000","last_became_peered":"2024-03-06T07:25:47.749189+0000","last_unstale":"2024-03-06T07:25:49.039116+0000","last_undegraded":"2024-03-06T07:25:49.039116+0000","last_fullsized":"2024-03-06T07:25:49.039116+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:49.406968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736570+0000","last_change":"2024-03-06T07:25:47.753561+0000","last_active":"2024-03-06T07:25:48.736570+0000","last_peered":"2024-03-06T07:25:48.736570+0000","last_clean":"2024-03-06T07:25:48.736570+0000","last_became_active":"2024-03-06T07:25:47.753272+0000","last_became_peered":"2024-03-06T07:25:47.753272+0000","last_unstale":"2024-03-06T07:25:48.736570+0000","last_undegraded":"2024-03-06T07:25:48.736570+0000","last_fullsized":"2024-03-06T07:25:48.736570+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:23.030627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737208+0000","last_change":"2024-03-06T07:25:47.752307+0000","last_active":"2024-03-06T07:25:48.737208+0000","last_peered":"2024-03-06T07:25:48.737208+0000","last_clean":"2024-03-06T07:25:48.737208+0000","last_became_active":"2024-03-06T07:25:47.752046+0000","last_became_peered":"2024-03-06T07:25:47.752046+0000","last_unstale":"2024-03-06T07:25:48.737208+0000","last_undegraded":"2024-03-06T07:25:48.737208+0000","last_fullsized":"2024-03-06T07:25:48.737208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.822094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736117+0000","last_change":"2024-03-06T07:25:47.750762+0000","last_active":"2024-03-06T07:25:48.736117+0000","last_peered":"2024-03-06T07:25:48.736117+0000","last_clean":"2024-03-06T07:25:48.736117+0000","last_became_active":"2024-03-06T07:25:47.750456+0000","last_became_peered":"2024-03-06T07:25:47.750456+0000","last_unstale":"2024-03-06T07:25:48.736117+0000","last_undegraded":"2024-03-06T07:25:48.736117+0000","last_fullsized":"2024-03-06T07:25:48.736117+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:28.272446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.762159+0000","last_change":"2024-03-06T07:25:47.754357+0000","last_active":"2024-03-06T07:25:48.762159+0000","last_peered":"2024-03-06T07:25:48.762159+0000","last_clean":"2024-03-06T07:25:48.762159+0000","last_became_active":"2024-03-06T07:25:47.754043+0000","last_became_peered":"2024-03-06T07:25:47.754043+0000","last_unstale":"2024-03-06T07:25:48.762159+0000","last_undegraded":"2024-03-06T07:25:48.762159+0000","last_fullsized":"2024-03-06T07:25:48.762159+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.822094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736225+0000","last_change":"2024-03-06T07:25:47.756728+0000","last_active":"2024-03-06T07:25:48.736225+0000","last_peered":"2024-03-06T07:25:48.736225+0000","last_clean":"2024-03-06T07:25:48.736225+0000","last_became_active":"2024-03-06T07:25:47.756364+0000","last_became_peered":"2024-03-06T07:25:47.756364+0000","last_unstale":"2024-03-06T07:25:48.736225+0000","last_undegraded":"2024-03-06T07:25:48.736225+0000","last_fullsized":"2024-03-06T07:25:48.736225+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:35.409269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735980+0000","last_change":"2024-03-06T07:25:47.760935+0000","last_active":"2024-03-06T07:25:48.735980+0000","last_peered":"2024-03-06T07:25:48.735980+0000","last_clean":"2024-03-06T07:25:48.735980+0000","last_became_active":"2024-03-06T07:25:47.760648+0000","last_became_peered":"2024-03-06T07:25:47.760648+0000","last_unstale":"2024-03-06T07:25:48.735980+0000","last_undegraded":"2024-03-06T07:25:48.735980+0000","last_fullsized":"2024-03-06T07:25:48.735980+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:09.073261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759561+0000","last_change":"2024-03-06T07:25:47.760858+0000","last_active":"2024-03-06T07:25:48.759561+0000","last_peered":"2024-03-06T07:25:48.759561+0000","last_clean":"2024-03-06T07:25:48.759561+0000","last_became_active":"2024-03-06T07:25:47.760570+0000","last_became_peered":"2024-03-06T07:25:47.760570+0000","last_unstale":"2024-03-06T07:25:48.759561+0000","last_undegraded":"2024-03-06T07:25:48.759561+0000","last_fullsized":"2024-03-06T07:25:48.759561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:27.486710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735731+0000","last_change":"2024-03-06T07:25:47.761368+0000","last_active":"2024-03-06T07:25:48.735731+0000","last_peered":"2024-03-06T07:25:48.735731+0000","last_clean":"2024-03-06T07:25:48.735731+0000","last_became_active":"2024-03-06T07:25:47.761104+0000","last_became_peered":"2024-03-06T07:25:47.761104+0000","last_unstale":"2024-03-06T07:25:48.735731+0000","last_undegraded":"2024-03-06T07:25:48.735731+0000","last_fullsized":"2024-03-06T07:25:48.735731+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:58.946010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760082+0000","last_change":"2024-03-06T07:25:47.756504+0000","last_active":"2024-03-06T07:25:48.760082+0000","last_peered":"2024-03-06T07:25:48.760082+0000","last_clean":"2024-03-06T07:25:48.760082+0000","last_became_active":"2024-03-06T07:25:47.756231+0000","last_became_peered":"2024-03-06T07:25:47.756231+0000","last_unstale":"2024-03-06T07:25:48.760082+0000","last_undegraded":"2024-03-06T07:25:48.760082+0000","last_fullsized":"2024-03-06T07:25:48.760082+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:17.837597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039678+0000","last_change":"2024-03-06T07:25:47.755618+0000","last_active":"2024-03-06T07:25:49.039678+0000","last_peered":"2024-03-06T07:25:49.039678+0000","last_clean":"2024-03-06T07:25:49.039678+0000","last_became_active":"2024-03-06T07:25:47.755092+0000","last_became_peered":"2024-03-06T07:25:47.755092+0000","last_unstale":"2024-03-06T07:25:49.039678+0000","last_undegraded":"2024-03-06T07:25:49.039678+0000","last_fullsized":"2024-03-06T07:25:49.039678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:24.401568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736402+0000","last_change":"2024-03-06T07:25:47.751435+0000","last_active":"2024-03-06T07:25:48.736402+0000","last_peered":"2024-03-06T07:25:48.736402+0000","last_clean":"2024-03-06T07:25:48.736402+0000","last_became_active":"2024-03-06T07:25:47.751146+0000","last_became_peered":"2024-03-06T07:25:47.751146+0000","last_unstale":"2024-03-06T07:25:48.736402+0000","last_undegraded":"2024-03-06T07:25:48.736402+0000","last_fullsized":"2024-03-06T07:25:48.736402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:52.495587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759695+0000","last_change":"2024-03-06T07:25:47.739643+0000","last_active":"2024-03-06T07:25:48.759695+0000","last_peered":"2024-03-06T07:25:48.759695+0000","last_clean":"2024-03-06T07:25:48.759695+0000","last_became_active":"2024-03-06T07:25:47.739353+0000","last_became_peered":"2024-03-06T07:25:47.739353+0000","last_unstale":"2024-03-06T07:25:48.759695+0000","last_undegraded":"2024-03-06T07:25:48.759695+0000","last_fullsized":"2024-03-06T07:25:48.759695+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:09.040328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759629+0000","last_change":"2024-03-06T07:25:47.753200+0000","last_active":"2024-03-06T07:25:48.759629+0000","last_peered":"2024-03-06T07:25:48.759629+0000","last_clean":"2024-03-06T07:25:48.759629+0000","last_became_active":"2024-03-06T07:25:47.752393+0000","last_became_peered":"2024-03-06T07:25:47.752393+0000","last_unstale":"2024-03-06T07:25:48.759629+0000","last_undegraded":"2024-03-06T07:25:48.759629+0000","last_fullsized":"2024-03-06T07:25:48.759629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:57.303162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736327+0000","last_change":"2024-03-06T07:25:47.761829+0000","last_active":"2024-03-06T07:25:48.736327+0000","last_peered":"2024-03-06T07:25:48.736327+0000","last_clean":"2024-03-06T07:25:48.736327+0000","last_became_active":"2024-03-06T07:25:47.761583+0000","last_became_peered":"2024-03-06T07:25:47.761583+0000","last_unstale":"2024-03-06T07:25:48.736327+0000","last_undegraded":"2024-03-06T07:25:48.736327+0000","last_fullsized":"2024-03-06T07:25:48.736327+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:19.894041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737151+0000","last_change":"2024-03-06T07:25:47.750725+0000","last_active":"2024-03-06T07:25:48.737151+0000","last_peered":"2024-03-06T07:25:48.737151+0000","last_clean":"2024-03-06T07:25:48.737151+0000","last_became_active":"2024-03-06T07:25:47.750427+0000","last_became_peered":"2024-03-06T07:25:47.750427+0000","last_unstale":"2024-03-06T07:25:48.737151+0000","last_undegraded":"2024-03-06T07:25:48.737151+0000","last_fullsized":"2024-03-06T07:25:48.737151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:09.013439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729992+0000","last_change":"2024-03-06T07:25:45.729992+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729992+0000","last_undegraded":"2024-03-06T07:25:45.729992+0000","last_fullsized":"2024-03-06T07:25:45.729992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737231+0000","last_change":"2024-03-06T07:25:46.741560+0000","last_active":"2024-03-06T07:25:48.737231+0000","last_peered":"2024-03-06T07:25:48.737231+0000","last_clean":"2024-03-06T07:25:48.737231+0000","last_became_active":"2024-03-06T07:25:46.741225+0000","last_became_peered":"2024-03-06T07:25:46.741225+0000","last_unstale":"2024-03-06T07:25:48.737231+0000","last_undegraded":"2024-03-06T07:25:48.737231+0000","last_fullsized":"2024-03-06T07:25:48.737231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:42.221129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.762098+0000","last_change":"2024-03-06T07:25:46.741655+0000","last_active":"2024-03-06T07:25:48.762098+0000","last_peered":"2024-03-06T07:25:48.762098+0000","last_clean":"2024-03-06T07:25:48.762098+0000","last_became_active":"2024-03-06T07:25:46.741374+0000","last_became_peered":"2024-03-06T07:25:46.741374+0000","last_unstale":"2024-03-06T07:25:48.762098+0000","last_undegraded":"2024-03-06T07:25:48.762098+0000","last_fullsized":"2024-03-06T07:25:48.762098+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:16.833042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040274+0000","last_change":"2024-03-06T07:25:47.748531+0000","last_active":"2024-03-06T07:25:49.040274+0000","last_peered":"2024-03-06T07:25:49.040274+0000","last_clean":"2024-03-06T07:25:49.040274+0000","last_became_active":"2024-03-06T07:25:47.748272+0000","last_became_peered":"2024-03-06T07:25:47.748272+0000","last_unstale":"2024-03-06T07:25:49.040274+0000","last_undegraded":"2024-03-06T07:25:49.040274+0000","last_fullsized":"2024-03-06T07:25:49.040274+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:52.756249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761801+0000","last_change":"2024-03-06T07:25:46.742202+0000","last_active":"2024-03-06T07:25:48.761801+0000","last_peered":"2024-03-06T07:25:48.761801+0000","last_clean":"2024-03-06T07:25:48.761801+0000","last_became_active":"2024-03-06T07:25:46.741325+0000","last_became_peered":"2024-03-06T07:25:46.741325+0000","last_unstale":"2024-03-06T07:25:48.761801+0000","last_undegraded":"2024-03-06T07:25:48.761801+0000","last_fullsized":"2024-03-06T07:25:48.761801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:44.989067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759670+0000","last_change":"2024-03-06T07:25:47.747094+0000","last_active":"2024-03-06T07:25:48.759670+0000","last_peered":"2024-03-06T07:25:48.759670+0000","last_clean":"2024-03-06T07:25:48.759670+0000","last_became_active":"2024-03-06T07:25:47.746796+0000","last_became_peered":"2024-03-06T07:25:47.746796+0000","last_unstale":"2024-03-06T07:25:48.759670+0000","last_undegraded":"2024-03-06T07:25:48.759670+0000","last_fullsized":"2024-03-06T07:25:48.759670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:13.850354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737511+0000","last_change":"2024-03-06T07:25:46.741705+0000","last_active":"2024-03-06T07:25:48.737511+0000","last_peered":"2024-03-06T07:25:48.737511+0000","last_clean":"2024-03-06T07:25:48.737511+0000","last_became_active":"2024-03-06T07:25:46.741411+0000","last_became_peered":"2024-03-06T07:25:46.741411+0000","last_unstale":"2024-03-06T07:25:48.737511+0000","last_undegraded":"2024-03-06T07:25:48.737511+0000","last_fullsized":"2024-03-06T07:25:48.737511+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:52.885965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737431+0000","last_change":"2024-03-06T07:25:47.753035+0000","last_active":"2024-03-06T07:25:48.737431+0000","last_peered":"2024-03-06T07:25:48.737431+0000","last_clean":"2024-03-06T07:25:48.737431+0000","last_became_active":"2024-03-06T07:25:47.752710+0000","last_became_peered":"2024-03-06T07:25:47.752710+0000","last_unstale":"2024-03-06T07:25:48.737431+0000","last_undegraded":"2024-03-06T07:25:48.737431+0000","last_fullsized":"2024-03-06T07:25:48.737431+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:24.401568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736619+0000","last_change":"2024-03-06T07:25:46.741681+0000","last_active":"2024-03-06T07:25:48.736619+0000","last_peered":"2024-03-06T07:25:48.736619+0000","last_clean":"2024-03-06T07:25:48.736619+0000","last_became_active":"2024-03-06T07:25:46.741397+0000","last_became_peered":"2024-03-06T07:25:46.741397+0000","last_unstale":"2024-03-06T07:25:48.736619+0000","last_undegraded":"2024-03-06T07:25:48.736619+0000","last_fullsized":"2024-03-06T07:25:48.736619+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:32.000019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039562+0000","last_change":"2024-03-06T07:25:47.759466+0000","last_active":"2024-03-06T07:25:49.039562+0000","last_peered":"2024-03-06T07:25:49.039562+0000","last_clean":"2024-03-06T07:25:49.039562+0000","last_became_active":"2024-03-06T07:25:47.759185+0000","last_became_peered":"2024-03-06T07:25:47.759185+0000","last_unstale":"2024-03-06T07:25:49.039562+0000","last_undegraded":"2024-03-06T07:25:49.039562+0000","last_fullsized":"2024-03-06T07:25:49.039562+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.822094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761641+0000","last_change":"2024-03-06T07:25:46.737587+0000","last_active":"2024-03-06T07:25:48.761641+0000","last_peered":"2024-03-06T07:25:48.761641+0000","last_clean":"2024-03-06T07:25:48.761641+0000","last_became_active":"2024-03-06T07:25:46.737289+0000","last_became_peered":"2024-03-06T07:25:46.737289+0000","last_unstale":"2024-03-06T07:25:48.761641+0000","last_undegraded":"2024-03-06T07:25:48.761641+0000","last_fullsized":"2024-03-06T07:25:48.761641+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:08.035773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039138+0000","last_change":"2024-03-06T07:25:47.749535+0000","last_active":"2024-03-06T07:25:49.039138+0000","last_peered":"2024-03-06T07:25:49.039138+0000","last_clean":"2024-03-06T07:25:49.039138+0000","last_became_active":"2024-03-06T07:25:47.748786+0000","last_became_peered":"2024-03-06T07:25:47.748786+0000","last_unstale":"2024-03-06T07:25:49.039138+0000","last_undegraded":"2024-03-06T07:25:49.039138+0000","last_fullsized":"2024-03-06T07:25:49.039138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:57.368763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759500+0000","last_change":"2024-03-06T07:25:46.741890+0000","last_active":"2024-03-06T07:25:48.759500+0000","last_peered":"2024-03-06T07:25:48.759500+0000","last_clean":"2024-03-06T07:25:48.759500+0000","last_became_active":"2024-03-06T07:25:46.741566+0000","last_became_peered":"2024-03-06T07:25:46.741566+0000","last_unstale":"2024-03-06T07:25:48.759500+0000","last_undegraded":"2024-03-06T07:25:48.759500+0000","last_fullsized":"2024-03-06T07:25:48.759500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:53.583814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039409+0000","last_change":"2024-03-06T07:25:47.741722+0000","last_active":"2024-03-06T07:25:49.039409+0000","last_peered":"2024-03-06T07:25:49.039409+0000","last_clean":"2024-03-06T07:25:49.039409+0000","last_became_active":"2024-03-06T07:25:47.741384+0000","last_became_peered":"2024-03-06T07:25:47.741384+0000","last_unstale":"2024-03-06T07:25:49.039409+0000","last_undegraded":"2024-03-06T07:25:49.039409+0000","last_fullsized":"2024-03-06T07:25:49.039409+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:26.696427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729420+0000","last_change":"2024-03-06T07:25:45.729420+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729420+0000","last_undegraded":"2024-03-06T07:25:45.729420+0000","last_fullsized":"2024-03-06T07:25:45.729420+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737099+0000","last_change":"2024-03-06T07:25:47.752574+0000","last_active":"2024-03-06T07:25:48.737099+0000","last_peered":"2024-03-06T07:25:48.737099+0000","last_clean":"2024-03-06T07:25:48.737099+0000","last_became_active":"2024-03-06T07:25:47.752246+0000","last_became_peered":"2024-03-06T07:25:47.752246+0000","last_unstale":"2024-03-06T07:25:48.737099+0000","last_undegraded":"2024-03-06T07:25:48.737099+0000","last_fullsized":"2024-03-06T07:25:48.737099+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:23.030627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728800+0000","last_change":"2024-03-06T07:25:45.728800+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728800+0000","last_undegraded":"2024-03-06T07:25:45.728800+0000","last_fullsized":"2024-03-06T07:25:45.728800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.762062+0000","last_change":"2024-03-06T07:25:47.755416+0000","last_active":"2024-03-06T07:25:48.762062+0000","last_peered":"2024-03-06T07:25:48.762062+0000","last_clean":"2024-03-06T07:25:48.762062+0000","last_became_active":"2024-03-06T07:25:47.755154+0000","last_became_peered":"2024-03-06T07:25:47.755154+0000","last_unstale":"2024-03-06T07:25:48.762062+0000","last_undegraded":"2024-03-06T07:25:48.762062+0000","last_fullsized":"2024-03-06T07:25:48.762062+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:09.013439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728545+0000","last_change":"2024-03-06T07:25:45.728545+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728545+0000","last_undegraded":"2024-03-06T07:25:45.728545+0000","last_fullsized":"2024-03-06T07:25:45.728545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736937+0000","last_change":"2024-03-06T07:25:47.761129+0000","last_active":"2024-03-06T07:25:48.736937+0000","last_peered":"2024-03-06T07:25:48.736937+0000","last_clean":"2024-03-06T07:25:48.736937+0000","last_became_active":"2024-03-06T07:25:47.760855+0000","last_became_peered":"2024-03-06T07:25:47.760855+0000","last_unstale":"2024-03-06T07:25:48.736937+0000","last_undegraded":"2024-03-06T07:25:48.736937+0000","last_fullsized":"2024-03-06T07:25:48.736937+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:08.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736791+0000","last_change":"2024-03-06T07:25:46.732588+0000","last_active":"2024-03-06T07:25:48.736791+0000","last_peered":"2024-03-06T07:25:48.736791+0000","last_clean":"2024-03-06T07:25:48.736791+0000","last_became_active":"2024-03-06T07:25:46.732252+0000","last_became_peered":"2024-03-06T07:25:46.732252+0000","last_unstale":"2024-03-06T07:25:48.736791+0000","last_undegraded":"2024-03-06T07:25:48.736791+0000","last_fullsized":"2024-03-06T07:25:48.736791+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:45.368244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761800+0000","last_change":"2024-03-06T07:25:47.759623+0000","last_active":"2024-03-06T07:25:48.761800+0000","last_peered":"2024-03-06T07:25:48.761800+0000","last_clean":"2024-03-06T07:25:48.761800+0000","last_became_active":"2024-03-06T07:25:47.759263+0000","last_became_peered":"2024-03-06T07:25:47.759263+0000","last_unstale":"2024-03-06T07:25:48.761800+0000","last_undegraded":"2024-03-06T07:25:48.761800+0000","last_fullsized":"2024-03-06T07:25:48.761800+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.952554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759773+0000","last_change":"2024-03-06T07:25:46.746659+0000","last_active":"2024-03-06T07:25:48.759773+0000","last_peered":"2024-03-06T07:25:48.759773+0000","last_clean":"2024-03-06T07:25:48.759773+0000","last_became_active":"2024-03-06T07:25:46.746246+0000","last_became_peered":"2024-03-06T07:25:46.746246+0000","last_unstale":"2024-03-06T07:25:48.759773+0000","last_undegraded":"2024-03-06T07:25:48.759773+0000","last_fullsized":"2024-03-06T07:25:48.759773+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:32.988256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039919+0000","last_change":"2024-03-06T07:25:47.760007+0000","last_active":"2024-03-06T07:25:49.039919+0000","last_peered":"2024-03-06T07:25:49.039919+0000","last_clean":"2024-03-06T07:25:49.039919+0000","last_became_active":"2024-03-06T07:25:47.759722+0000","last_became_peered":"2024-03-06T07:25:47.759722+0000","last_unstale":"2024-03-06T07:25:49.039919+0000","last_undegraded":"2024-03-06T07:25:49.039919+0000","last_fullsized":"2024-03-06T07:25:49.039919+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:30.829345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039266+0000","last_change":"2024-03-06T07:25:46.741231+0000","last_active":"2024-03-06T07:25:49.039266+0000","last_peered":"2024-03-06T07:25:49.039266+0000","last_clean":"2024-03-06T07:25:49.039266+0000","last_became_active":"2024-03-06T07:25:46.740951+0000","last_became_peered":"2024-03-06T07:25:46.740951+0000","last_unstale":"2024-03-06T07:25:49.039266+0000","last_undegraded":"2024-03-06T07:25:49.039266+0000","last_fullsized":"2024-03-06T07:25:49.039266+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:40.893178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759533+0000","last_change":"2024-03-06T07:25:47.746838+0000","last_active":"2024-03-06T07:25:48.759533+0000","last_peered":"2024-03-06T07:25:48.759533+0000","last_clean":"2024-03-06T07:25:48.759533+0000","last_became_active":"2024-03-06T07:25:47.746563+0000","last_became_peered":"2024-03-06T07:25:47.746563+0000","last_unstale":"2024-03-06T07:25:48.759533+0000","last_undegraded":"2024-03-06T07:25:48.759533+0000","last_fullsized":"2024-03-06T07:25:48.759533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:10.615032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736434+0000","last_change":"2024-03-06T07:25:46.741981+0000","last_active":"2024-03-06T07:25:48.736434+0000","last_peered":"2024-03-06T07:25:48.736434+0000","last_clean":"2024-03-06T07:25:48.736434+0000","last_became_active":"2024-03-06T07:25:46.741709+0000","last_became_peered":"2024-03-06T07:25:46.741709+0000","last_unstale":"2024-03-06T07:25:48.736434+0000","last_undegraded":"2024-03-06T07:25:48.736434+0000","last_fullsized":"2024-03-06T07:25:48.736434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:33.944174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759470+0000","last_change":"2024-03-06T07:25:47.755828+0000","last_active":"2024-03-06T07:25:48.759470+0000","last_peered":"2024-03-06T07:25:48.759470+0000","last_clean":"2024-03-06T07:25:48.759470+0000","last_became_active":"2024-03-06T07:25:47.755559+0000","last_became_peered":"2024-03-06T07:25:47.755559+0000","last_unstale":"2024-03-06T07:25:48.759470+0000","last_undegraded":"2024-03-06T07:25:48.759470+0000","last_fullsized":"2024-03-06T07:25:48.759470+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:34.948730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761798+0000","last_change":"2024-03-06T07:25:46.743744+0000","last_active":"2024-03-06T07:25:48.761798+0000","last_peered":"2024-03-06T07:25:48.761798+0000","last_clean":"2024-03-06T07:25:48.761798+0000","last_became_active":"2024-03-06T07:25:46.743447+0000","last_became_peered":"2024-03-06T07:25:46.743447+0000","last_unstale":"2024-03-06T07:25:48.761798+0000","last_undegraded":"2024-03-06T07:25:48.761798+0000","last_fullsized":"2024-03-06T07:25:48.761798+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:56.298607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039836+0000","last_change":"2024-03-06T07:25:47.747720+0000","last_active":"2024-03-06T07:25:49.039836+0000","last_peered":"2024-03-06T07:25:49.039836+0000","last_clean":"2024-03-06T07:25:49.039836+0000","last_became_active":"2024-03-06T07:25:47.747412+0000","last_became_peered":"2024-03-06T07:25:47.747412+0000","last_unstale":"2024-03-06T07:25:49.039836+0000","last_undegraded":"2024-03-06T07:25:49.039836+0000","last_fullsized":"2024-03-06T07:25:49.039836+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:10.096097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728561+0000","last_change":"2024-03-06T07:25:45.728561+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728561+0000","last_undegraded":"2024-03-06T07:25:45.728561+0000","last_fullsized":"2024-03-06T07:25:45.728561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761941+0000","last_change":"2024-03-06T07:25:46.744401+0000","last_active":"2024-03-06T07:25:48.761941+0000","last_peered":"2024-03-06T07:25:48.761941+0000","last_clean":"2024-03-06T07:25:48.761941+0000","last_became_active":"2024-03-06T07:25:46.744090+0000","last_became_peered":"2024-03-06T07:25:46.744090+0000","last_unstale":"2024-03-06T07:25:48.761941+0000","last_undegraded":"2024-03-06T07:25:48.761941+0000","last_fullsized":"2024-03-06T07:25:48.761941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:51.535471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737048+0000","last_change":"2024-03-06T07:25:47.748162+0000","last_active":"2024-03-06T07:25:48.737048+0000","last_peered":"2024-03-06T07:25:48.737048+0000","last_clean":"2024-03-06T07:25:48.737048+0000","last_became_active":"2024-03-06T07:25:47.747901+0000","last_became_peered":"2024-03-06T07:25:47.747901+0000","last_unstale":"2024-03-06T07:25:48.737048+0000","last_undegraded":"2024-03-06T07:25:48.737048+0000","last_fullsized":"2024-03-06T07:25:48.737048+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.952554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761902+0000","last_change":"2024-03-06T07:25:46.736412+0000","last_active":"2024-03-06T07:25:48.761902+0000","last_peered":"2024-03-06T07:25:48.761902+0000","last_clean":"2024-03-06T07:25:48.761902+0000","last_became_active":"2024-03-06T07:25:46.735973+0000","last_became_peered":"2024-03-06T07:25:46.735973+0000","last_unstale":"2024-03-06T07:25:48.761902+0000","last_undegraded":"2024-03-06T07:25:48.761902+0000","last_fullsized":"2024-03-06T07:25:48.761902+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:39.306436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735938+0000","last_change":"2024-03-06T07:25:47.748114+0000","last_active":"2024-03-06T07:25:48.735938+0000","last_peered":"2024-03-06T07:25:48.735938+0000","last_clean":"2024-03-06T07:25:48.735938+0000","last_became_active":"2024-03-06T07:25:47.747802+0000","last_became_peered":"2024-03-06T07:25:47.747802+0000","last_unstale":"2024-03-06T07:25:48.735938+0000","last_undegraded":"2024-03-06T07:25:48.735938+0000","last_fullsized":"2024-03-06T07:25:48.735938+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:27.832719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039237+0000","last_change":"2024-03-06T07:25:46.740390+0000","last_active":"2024-03-06T07:25:49.039237+0000","last_peered":"2024-03-06T07:25:49.039237+0000","last_clean":"2024-03-06T07:25:49.039237+0000","last_became_active":"2024-03-06T07:25:46.740119+0000","last_became_peered":"2024-03-06T07:25:46.740119+0000","last_unstale":"2024-03-06T07:25:49.039237+0000","last_undegraded":"2024-03-06T07:25:49.039237+0000","last_fullsized":"2024-03-06T07:25:49.039237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:52.685641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736602+0000","last_change":"2024-03-06T07:25:47.753494+0000","last_active":"2024-03-06T07:25:48.736602+0000","last_peered":"2024-03-06T07:25:48.736602+0000","last_clean":"2024-03-06T07:25:48.736602+0000","last_became_active":"2024-03-06T07:25:47.753128+0000","last_became_peered":"2024-03-06T07:25:47.753128+0000","last_unstale":"2024-03-06T07:25:48.736602+0000","last_undegraded":"2024-03-06T07:25:48.736602+0000","last_fullsized":"2024-03-06T07:25:48.736602+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:19.894041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736006+0000","last_change":"2024-03-06T07:25:46.746719+0000","last_active":"2024-03-06T07:25:48.736006+0000","last_peered":"2024-03-06T07:25:48.736006+0000","last_clean":"2024-03-06T07:25:48.736006+0000","last_became_active":"2024-03-06T07:25:46.746405+0000","last_became_peered":"2024-03-06T07:25:46.746405+0000","last_unstale":"2024-03-06T07:25:48.736006+0000","last_undegraded":"2024-03-06T07:25:48.736006+0000","last_fullsized":"2024-03-06T07:25:48.736006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:00.506076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735741+0000","last_change":"2024-03-06T07:25:47.746644+0000","last_active":"2024-03-06T07:25:48.735741+0000","last_peered":"2024-03-06T07:25:48.735741+0000","last_clean":"2024-03-06T07:25:48.735741+0000","last_became_active":"2024-03-06T07:25:47.746338+0000","last_became_peered":"2024-03-06T07:25:47.746338+0000","last_unstale":"2024-03-06T07:25:48.735741+0000","last_undegraded":"2024-03-06T07:25:48.735741+0000","last_fullsized":"2024-03-06T07:25:48.735741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:49.722061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735738+0000","last_change":"2024-03-06T07:25:46.742152+0000","last_active":"2024-03-06T07:25:48.735738+0000","last_peered":"2024-03-06T07:25:48.735738+0000","last_clean":"2024-03-06T07:25:48.735738+0000","last_became_active":"2024-03-06T07:25:46.741827+0000","last_became_peered":"2024-03-06T07:25:46.741827+0000","last_unstale":"2024-03-06T07:25:48.735738+0000","last_undegraded":"2024-03-06T07:25:48.735738+0000","last_fullsized":"2024-03-06T07:25:48.735738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:53.097448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736315+0000","last_change":"2024-03-06T07:25:47.760666+0000","last_active":"2024-03-06T07:25:48.736315+0000","last_peered":"2024-03-06T07:25:48.736315+0000","last_clean":"2024-03-06T07:25:48.736315+0000","last_became_active":"2024-03-06T07:25:47.760210+0000","last_became_peered":"2024-03-06T07:25:47.760210+0000","last_unstale":"2024-03-06T07:25:48.736315+0000","last_undegraded":"2024-03-06T07:25:48.736315+0000","last_fullsized":"2024-03-06T07:25:48.736315+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:52.540026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.038963+0000","last_change":"2024-03-06T07:25:46.735528+0000","last_active":"2024-03-06T07:25:49.038963+0000","last_peered":"2024-03-06T07:25:49.038963+0000","last_clean":"2024-03-06T07:25:49.038963+0000","last_became_active":"2024-03-06T07:25:46.735242+0000","last_became_peered":"2024-03-06T07:25:46.735242+0000","last_unstale":"2024-03-06T07:25:49.038963+0000","last_undegraded":"2024-03-06T07:25:49.038963+0000","last_fullsized":"2024-03-06T07:25:49.038963+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:29.824789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737538+0000","last_change":"2024-03-06T07:25:47.746068+0000","last_active":"2024-03-06T07:25:48.737538+0000","last_peered":"2024-03-06T07:25:48.737538+0000","last_clean":"2024-03-06T07:25:48.737538+0000","last_became_active":"2024-03-06T07:25:47.745758+0000","last_became_peered":"2024-03-06T07:25:47.745758+0000","last_unstale":"2024-03-06T07:25:48.737538+0000","last_undegraded":"2024-03-06T07:25:48.737538+0000","last_fullsized":"2024-03-06T07:25:48.737538+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:10.096097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759998+0000","last_change":"2024-03-06T07:25:46.746740+0000","last_active":"2024-03-06T07:25:48.759998+0000","last_peered":"2024-03-06T07:25:48.759998+0000","last_clean":"2024-03-06T07:25:48.759998+0000","last_became_active":"2024-03-06T07:25:46.746272+0000","last_became_peered":"2024-03-06T07:25:46.746272+0000","last_unstale":"2024-03-06T07:25:48.759998+0000","last_undegraded":"2024-03-06T07:25:48.759998+0000","last_fullsized":"2024-03-06T07:25:48.759998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:04.573822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039946+0000","last_change":"2024-03-06T07:25:47.754144+0000","last_active":"2024-03-06T07:25:49.039946+0000","last_peered":"2024-03-06T07:25:49.039946+0000","last_clean":"2024-03-06T07:25:49.039946+0000","last_became_active":"2024-03-06T07:25:47.753738+0000","last_became_peered":"2024-03-06T07:25:47.753738+0000","last_unstale":"2024-03-06T07:25:49.039946+0000","last_undegraded":"2024-03-06T07:25:49.039946+0000","last_fullsized":"2024-03-06T07:25:49.039946+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:53.690196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760064+0000","last_change":"2024-03-06T07:25:46.746681+0000","last_active":"2024-03-06T07:25:48.760064+0000","last_peered":"2024-03-06T07:25:48.760064+0000","last_clean":"2024-03-06T07:25:48.760064+0000","last_became_active":"2024-03-06T07:25:46.746393+0000","last_became_peered":"2024-03-06T07:25:46.746393+0000","last_unstale":"2024-03-06T07:25:48.760064+0000","last_undegraded":"2024-03-06T07:25:48.760064+0000","last_fullsized":"2024-03-06T07:25:48.760064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:01.102402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729143+0000","last_change":"2024-03-06T07:25:45.729143+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729143+0000","last_undegraded":"2024-03-06T07:25:45.729143+0000","last_fullsized":"2024-03-06T07:25:45.729143+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737109+0000","last_change":"2024-03-06T07:25:47.759956+0000","last_active":"2024-03-06T07:25:48.737109+0000","last_peered":"2024-03-06T07:25:48.737109+0000","last_clean":"2024-03-06T07:25:48.737109+0000","last_became_active":"2024-03-06T07:25:47.759705+0000","last_became_peered":"2024-03-06T07:25:47.759705+0000","last_unstale":"2024-03-06T07:25:48.737109+0000","last_undegraded":"2024-03-06T07:25:48.737109+0000","last_fullsized":"2024-03-06T07:25:48.737109+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:34.507002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728826+0000","last_change":"2024-03-06T07:25:45.728826+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728826+0000","last_undegraded":"2024-03-06T07:25:45.728826+0000","last_fullsized":"2024-03-06T07:25:45.728826+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737427+0000","last_change":"2024-03-06T07:25:47.738425+0000","last_active":"2024-03-06T07:25:48.737427+0000","last_peered":"2024-03-06T07:25:48.737427+0000","last_clean":"2024-03-06T07:25:48.737427+0000","last_became_active":"2024-03-06T07:25:47.738221+0000","last_became_peered":"2024-03-06T07:25:47.738221+0000","last_unstale":"2024-03-06T07:25:48.737427+0000","last_undegraded":"2024-03-06T07:25:48.737427+0000","last_fullsized":"2024-03-06T07:25:48.737427+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.952554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761806+0000","last_change":"2024-03-06T07:25:46.742402+0000","last_active":"2024-03-06T07:25:48.761806+0000","last_peered":"2024-03-06T07:25:48.761806+0000","last_clean":"2024-03-06T07:25:48.761806+0000","last_became_active":"2024-03-06T07:25:46.742144+0000","last_became_peered":"2024-03-06T07:25:46.742144+0000","last_unstale":"2024-03-06T07:25:48.761806+0000","last_undegraded":"2024-03-06T07:25:48.761806+0000","last_fullsized":"2024-03-06T07:25:48.761806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:12.845799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761869+0000","last_change":"2024-03-06T07:25:47.758008+0000","last_active":"2024-03-06T07:25:48.761869+0000","last_peered":"2024-03-06T07:25:48.761869+0000","last_clean":"2024-03-06T07:25:48.761869+0000","last_became_active":"2024-03-06T07:25:47.754667+0000","last_became_peered":"2024-03-06T07:25:47.754667+0000","last_unstale":"2024-03-06T07:25:48.761869+0000","last_undegraded":"2024-03-06T07:25:48.761869+0000","last_fullsized":"2024-03-06T07:25:48.761869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:34.507002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736719+0000","last_change":"2024-03-06T07:25:46.741357+0000","last_active":"2024-03-06T07:25:48.736719+0000","last_peered":"2024-03-06T07:25:48.736719+0000","last_clean":"2024-03-06T07:25:48.736719+0000","last_became_active":"2024-03-06T07:25:46.741039+0000","last_became_peered":"2024-03-06T07:25:46.741039+0000","last_unstale":"2024-03-06T07:25:48.736719+0000","last_undegraded":"2024-03-06T07:25:48.736719+0000","last_fullsized":"2024-03-06T07:25:48.736719+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:42.364991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039438+0000","last_change":"2024-03-06T07:25:47.755852+0000","last_active":"2024-03-06T07:25:49.039438+0000","last_peered":"2024-03-06T07:25:49.039438+0000","last_clean":"2024-03-06T07:25:49.039438+0000","last_became_active":"2024-03-06T07:25:47.755099+0000","last_became_peered":"2024-03-06T07:25:47.755099+0000","last_unstale":"2024-03-06T07:25:49.039438+0000","last_undegraded":"2024-03-06T07:25:49.039438+0000","last_fullsized":"2024-03-06T07:25:49.039438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:09.013439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728506+0000","last_change":"2024-03-06T07:25:45.728506+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728506+0000","last_undegraded":"2024-03-06T07:25:45.728506+0000","last_fullsized":"2024-03-06T07:25:45.728506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736571+0000","last_change":"2024-03-06T07:25:46.739746+0000","last_active":"2024-03-06T07:25:48.736571+0000","last_peered":"2024-03-06T07:25:48.736571+0000","last_clean":"2024-03-06T07:25:48.736571+0000","last_became_active":"2024-03-06T07:25:46.739449+0000","last_became_peered":"2024-03-06T07:25:46.739449+0000","last_unstale":"2024-03-06T07:25:48.736571+0000","last_undegraded":"2024-03-06T07:25:48.736571+0000","last_fullsized":"2024-03-06T07:25:48.736571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:07.641157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759838+0000","last_change":"2024-03-06T07:25:47.744991+0000","last_active":"2024-03-06T07:25:48.759838+0000","last_peered":"2024-03-06T07:25:48.759838+0000","last_clean":"2024-03-06T07:25:48.759838+0000","last_became_active":"2024-03-06T07:25:47.744673+0000","last_became_peered":"2024-03-06T07:25:47.744673+0000","last_unstale":"2024-03-06T07:25:48.759838+0000","last_undegraded":"2024-03-06T07:25:48.759838+0000","last_fullsized":"2024-03-06T07:25:48.759838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:45.993623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040220+0000","last_change":"2024-03-06T07:25:46.746841+0000","last_active":"2024-03-06T07:25:49.040220+0000","last_peered":"2024-03-06T07:25:49.040220+0000","last_clean":"2024-03-06T07:25:49.040220+0000","last_became_active":"2024-03-06T07:25:46.746592+0000","last_became_peered":"2024-03-06T07:25:46.746592+0000","last_unstale":"2024-03-06T07:25:49.040220+0000","last_undegraded":"2024-03-06T07:25:49.040220+0000","last_fullsized":"2024-03-06T07:25:49.040220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:12.845799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":281,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039662+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:25:49.039662+0000","last_peered":"2024-03-06T07:25:49.039662+0000","last_clean":"2024-03-06T07:25:49.039662+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:25:49.039662+0000","last_undegraded":"2024-03-06T07:25:49.039662+0000","last_fullsized":"2024-03-06T07:25:49.039662+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":106,"acting":106,"num_store_stats":7},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":1716,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1593344,"data_stored":1012776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":402988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1011712,"data_stored":423972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":422338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1007616,"data_stored":423950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":422920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1007616,"data_stored":422532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1593344,"data_stored":1012740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:51.493 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3f is not active+clean 2024-03-06T07:25:51.493 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:25:46.718808+0000', 'last_change': '2024-03-06T07:25:46.718808+0000', 'last_active': '2024-03-06T07:25:46.718808+0000', 'last_peered': '2024-03-06T07:25:46.718808+0000', 'last_clean': '2024-03-06T07:25:46.718808+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:46.718808+0000', 'last_undegraded': '2024-03-06T07:25:46.718808+0000', 'last_fullsized': '2024-03-06T07:25:46.718808+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:51.494 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.36 is not active+clean 2024-03-06T07:25:51.494 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:25:46.718808+0000', 'last_change': '2024-03-06T07:25:46.718808+0000', 'last_active': '2024-03-06T07:25:46.718808+0000', 'last_peered': '2024-03-06T07:25:46.718808+0000', 'last_clean': '2024-03-06T07:25:46.718808+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:46.718808+0000', 'last_undegraded': '2024-03-06T07:25:46.718808+0000', 'last_fullsized': '2024-03-06T07:25:46.718808+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:51.494 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.34 is not active+clean 2024-03-06T07:25:51.494 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:25:46.718808+0000', 'last_change': '2024-03-06T07:25:46.718808+0000', 'last_active': '2024-03-06T07:25:46.718808+0000', 'last_peered': '2024-03-06T07:25:46.718808+0000', 'last_clean': '2024-03-06T07:25:46.718808+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:46.718808+0000', 'last_undegraded': '2024-03-06T07:25:46.718808+0000', 'last_fullsized': '2024-03-06T07:25:46.718808+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:51.494 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.29 is not active+clean 2024-03-06T07:25:51.494 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:25:46.718808+0000', 'last_change': '2024-03-06T07:25:46.718808+0000', 'last_active': '2024-03-06T07:25:46.718808+0000', 'last_peered': '2024-03-06T07:25:46.718808+0000', 'last_clean': '2024-03-06T07:25:46.718808+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:46.718808+0000', 'last_undegraded': '2024-03-06T07:25:46.718808+0000', 'last_fullsized': '2024-03-06T07:25:46.718808+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:51.494 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.24 is not active+clean 2024-03-06T07:25:51.495 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:25:46.718808+0000', 'last_change': '2024-03-06T07:25:46.718808+0000', 'last_active': '2024-03-06T07:25:46.718808+0000', 'last_peered': '2024-03-06T07:25:46.718808+0000', 'last_clean': '2024-03-06T07:25:46.718808+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:46.718808+0000', 'last_undegraded': '2024-03-06T07:25:46.718808+0000', 'last_fullsized': '2024-03-06T07:25:46.718808+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:51.495 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.20 is not active+clean 2024-03-06T07:25:51.495 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:25:46.718808+0000', 'last_change': '2024-03-06T07:25:46.718808+0000', 'last_active': '2024-03-06T07:25:46.718808+0000', 'last_peered': '2024-03-06T07:25:46.718808+0000', 'last_clean': '2024-03-06T07:25:46.718808+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:46.718808+0000', 'last_undegraded': '2024-03-06T07:25:46.718808+0000', 'last_fullsized': '2024-03-06T07:25:46.718808+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:51.495 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1f is not active+clean 2024-03-06T07:25:51.495 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:45.729992+0000', 'last_change': '2024-03-06T07:25:45.729992+0000', 'last_active': '2024-03-06T07:25:45.714253+0000', 'last_peered': '2024-03-06T07:25:45.714253+0000', 'last_clean': '2024-03-06T07:25:45.714253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:45.729992+0000', 'last_undegraded': '2024-03-06T07:25:45.729992+0000', 'last_fullsized': '2024-03-06T07:25:45.729992+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:51.495 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1f is not active+clean 2024-03-06T07:25:51.495 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:25:46.718808+0000', 'last_change': '2024-03-06T07:25:46.718808+0000', 'last_active': '2024-03-06T07:25:46.718808+0000', 'last_peered': '2024-03-06T07:25:46.718808+0000', 'last_clean': '2024-03-06T07:25:46.718808+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:46.718808+0000', 'last_undegraded': '2024-03-06T07:25:46.718808+0000', 'last_fullsized': '2024-03-06T07:25:46.718808+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:51.495 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1c is not active+clean 2024-03-06T07:25:51.495 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:25:46.718808+0000', 'last_change': '2024-03-06T07:25:46.718808+0000', 'last_active': '2024-03-06T07:25:46.718808+0000', 'last_peered': '2024-03-06T07:25:46.718808+0000', 'last_clean': '2024-03-06T07:25:46.718808+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:46.718808+0000', 'last_undegraded': '2024-03-06T07:25:46.718808+0000', 'last_fullsized': '2024-03-06T07:25:46.718808+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.17 is not active+clean 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:45.729420+0000', 'last_change': '2024-03-06T07:25:45.729420+0000', 'last_active': '2024-03-06T07:25:45.714253+0000', 'last_peered': '2024-03-06T07:25:45.714253+0000', 'last_clean': '2024-03-06T07:25:45.714253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:45.729420+0000', 'last_undegraded': '2024-03-06T07:25:45.729420+0000', 'last_fullsized': '2024-03-06T07:25:45.729420+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.16 is not active+clean 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:45.728800+0000', 'last_change': '2024-03-06T07:25:45.728800+0000', 'last_active': '2024-03-06T07:25:45.714253+0000', 'last_peered': '2024-03-06T07:25:45.714253+0000', 'last_clean': '2024-03-06T07:25:45.714253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:45.728800+0000', 'last_undegraded': '2024-03-06T07:25:45.728800+0000', 'last_fullsized': '2024-03-06T07:25:45.728800+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.15 is not active+clean 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:45.728545+0000', 'last_change': '2024-03-06T07:25:45.728545+0000', 'last_active': '2024-03-06T07:25:45.714253+0000', 'last_peered': '2024-03-06T07:25:45.714253+0000', 'last_clean': '2024-03-06T07:25:45.714253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:45.728545+0000', 'last_undegraded': '2024-03-06T07:25:45.728545+0000', 'last_fullsized': '2024-03-06T07:25:45.728545+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.f is not active+clean 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:45.728561+0000', 'last_change': '2024-03-06T07:25:45.728561+0000', 'last_active': '2024-03-06T07:25:45.714253+0000', 'last_peered': '2024-03-06T07:25:45.714253+0000', 'last_clean': '2024-03-06T07:25:45.714253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:45.728561+0000', 'last_undegraded': '2024-03-06T07:25:45.728561+0000', 'last_fullsized': '2024-03-06T07:25:45.728561+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.f is not active+clean 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:25:46.718808+0000', 'last_change': '2024-03-06T07:25:46.718808+0000', 'last_active': '2024-03-06T07:25:46.718808+0000', 'last_peered': '2024-03-06T07:25:46.718808+0000', 'last_clean': '2024-03-06T07:25:46.718808+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:46.718808+0000', 'last_undegraded': '2024-03-06T07:25:46.718808+0000', 'last_fullsized': '2024-03-06T07:25:46.718808+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.4 is not active+clean 2024-03-06T07:25:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:25:46.718808+0000', 'last_change': '2024-03-06T07:25:46.718808+0000', 'last_active': '2024-03-06T07:25:46.718808+0000', 'last_peered': '2024-03-06T07:25:46.718808+0000', 'last_clean': '2024-03-06T07:25:46.718808+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:46.718808+0000', 'last_undegraded': '2024-03-06T07:25:46.718808+0000', 'last_fullsized': '2024-03-06T07:25:46.718808+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:51.497 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.5 is not active+clean 2024-03-06T07:25:51.497 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:45.729143+0000', 'last_change': '2024-03-06T07:25:45.729143+0000', 'last_active': '2024-03-06T07:25:45.714253+0000', 'last_peered': '2024-03-06T07:25:45.714253+0000', 'last_clean': '2024-03-06T07:25:45.714253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:45.729143+0000', 'last_undegraded': '2024-03-06T07:25:45.729143+0000', 'last_fullsized': '2024-03-06T07:25:45.729143+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:51.497 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.4 is not active+clean 2024-03-06T07:25:51.497 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:45.728826+0000', 'last_change': '2024-03-06T07:25:45.728826+0000', 'last_active': '2024-03-06T07:25:45.714253+0000', 'last_peered': '2024-03-06T07:25:45.714253+0000', 'last_clean': '2024-03-06T07:25:45.714253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:45.728826+0000', 'last_undegraded': '2024-03-06T07:25:45.728826+0000', 'last_fullsized': '2024-03-06T07:25:45.728826+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:51.497 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1 is not active+clean 2024-03-06T07:25:51.497 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:45.728506+0000', 'last_change': '2024-03-06T07:25:45.728506+0000', 'last_active': '2024-03-06T07:25:45.714253+0000', 'last_peered': '2024-03-06T07:25:45.714253+0000', 'last_clean': '2024-03-06T07:25:45.714253+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:45.728506+0000', 'last_undegraded': '2024-03-06T07:25:45.728506+0000', 'last_fullsized': '2024-03-06T07:25:45.728506+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:45.714253+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:51.497 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1 is not active+clean 2024-03-06T07:25:51.497 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:25:46.718808+0000', 'last_change': '2024-03-06T07:25:46.718808+0000', 'last_active': '2024-03-06T07:25:46.718808+0000', 'last_peered': '2024-03-06T07:25:46.718808+0000', 'last_clean': '2024-03-06T07:25:46.718808+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:46.718808+0000', 'last_undegraded': '2024-03-06T07:25:46.718808+0000', 'last_fullsized': '2024-03-06T07:25:46.718808+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:46.718808+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:25:51.498 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:25:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.627+0000 7f326cdd0640 1 Processor -- start 2024-03-06T07:25:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.627+0000 7f326cdd0640 1 -- start start 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.628+0000 7f326cdd0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3268161670 0x7f3268072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.628+0000 7f326cdd0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f32680b0ea0 0x7f3268072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.628+0000 7f326cdd0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3268066d00 con 0x7f3268072a60 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.628+0000 7f326cdd0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32680b6170 con 0x7f32680b0ea0 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.628+0000 7f326cdd0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32680ada80 con 0x7f3268161670 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.628+0000 7f3266d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f32680b0ea0 0x7f3268072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.628+0000 7f3266d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f32680b0ea0 0x7f3268072ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46634/0 (socket says 172.21.15.59:46634) 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.628+0000 7f3266d9d640 1 -- 172.21.15.59:0/999604329 learned_addr learned my addr 172.21.15.59:0/999604329 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.629+0000 7f326659c640 1 --2- 172.21.15.59:0/999604329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3268161670 0x7f3268072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.629+0000 7f3266d9d640 1 -- 172.21.15.59:0/999604329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3268161670 msgr2=0x7f3268072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.629+0000 7f3266d9d640 1 --2- 172.21.15.59:0/999604329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3268161670 0x7f3268072520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.629+0000 7f3266d9d640 1 -- 172.21.15.59:0/999604329 >> v1:172.21.15.164:6789/0 conn(0x7f3268072a60 legacy=0x7f32680adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.629+0000 7f3266d9d640 1 -- 172.21.15.59:0/999604329 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3268066fc0 con 0x7f32680b0ea0 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f3266d9d640 1 --2- 172.21.15.59:0/999604329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32680b0ea0 0x7f3268072ec0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f3260009e50 tx=0x7f326002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e9eabd3cbe4477 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f326559a640 1 -- 172.21.15.59:0/999604329 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326002c9e0 con 0x7f32680b0ea0 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f326559a640 1 -- 172.21.15.59:0/999604329 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326002cb80 con 0x7f32680b0ea0 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f326559a640 1 -- 172.21.15.59:0/999604329 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32600343f0 con 0x7f32680b0ea0 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f326cdd0640 1 -- 172.21.15.59:0/999604329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32680b0ea0 msgr2=0x7f3268072ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f326cdd0640 1 --2- 172.21.15.59:0/999604329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32680b0ea0 0x7f3268072ec0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f3260009e50 tx=0x7f326002be40 comp rx=0 tx=0).stop 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f326cdd0640 1 -- 172.21.15.59:0/999604329 shutdown_connections 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f326cdd0640 1 --2- 172.21.15.59:0/999604329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32680b0ea0 0x7f3268072ec0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f326cdd0640 1 --2- 172.21.15.59:0/999604329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3268161670 0x7f3268072520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f326cdd0640 1 -- 172.21.15.59:0/999604329 >> 172.21.15.59:0/999604329 conn(0x7f32680b4e60 msgr2=0x7f32680b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f326cdd0640 1 -- 172.21.15.59:0/999604329 shutdown_connections 2024-03-06T07:25:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.630+0000 7f326cdd0640 1 -- 172.21.15.59:0/999604329 wait complete. 2024-03-06T07:25:51.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.632+0000 7f326cdd0640 1 Processor -- start 2024-03-06T07:25:51.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.632+0000 7f326cdd0640 1 -- start start 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.632+0000 7f326cdd0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3268072a60 0x7f32681f6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.632+0000 7f326cdd0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32680b0ea0 0x7f3268157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.632+0000 7f326cdd0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268161670 0x7f3268158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.632+0000 7f326cdd0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326806b330 con 0x7f3268072a60 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.632+0000 7f326cdd0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3268161a50 con 0x7f3268161670 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.632+0000 7f326cdd0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326806b610 con 0x7f32680b0ea0 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.632+0000 7f326659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3268072a60 0x7f32681f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f326659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3268072a60 0x7f32681f6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46646/0 (socket says 172.21.15.59:46646) 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f326659c640 1 -- 172.21.15.59:0/700057503 learned_addr learned my addr 172.21.15.59:0/700057503 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f3266d9d640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268161670 0x7f3268158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f3265d9b640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32680b0ea0 0x7f3268157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f326659c640 1 -- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32680b0ea0 msgr2=0x7f3268157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f326659c640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32680b0ea0 0x7f3268157c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f326659c640 1 -- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268161670 msgr2=0x7f3268158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f326659c640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268161670 0x7f3268158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f326659c640 1 -- 172.21.15.59:0/700057503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32681f5890 con 0x7f3268072a60 2024-03-06T07:25:51.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f3266d9d640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268161670 0x7f3268158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:51.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f326659c640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3268072a60 0x7f32681f6930 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f325000d5f0 tx=0x7f325000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f325effd640 1 -- 172.21.15.59:0/700057503 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32500125f0 con 0x7f3268072a60 2024-03-06T07:25:51.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f325effd640 1 -- 172.21.15.59:0/700057503 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3250019070 con 0x7f3268072a60 2024-03-06T07:25:51.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f326cdd0640 1 -- 172.21.15.59:0/700057503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32681f5b70 con 0x7f3268072a60 2024-03-06T07:25:51.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f325effd640 1 -- 172.21.15.59:0/700057503 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32500129d0 con 0x7f3268072a60 2024-03-06T07:25:51.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.633+0000 7f326cdd0640 1 -- 172.21.15.59:0/700057503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32681f6130 con 0x7f3268072a60 2024-03-06T07:25:51.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.634+0000 7f326cdd0640 1 -- 172.21.15.59:0/700057503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326804f330 con 0x7f3268072a60 2024-03-06T07:25:51.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.635+0000 7f325effd640 1 -- 172.21.15.59:0/700057503 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3250012b70 con 0x7f3268072a60 2024-03-06T07:25:51.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.635+0000 7f325effd640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f324806b4d0 0x7f324806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.635+0000 7f325effd640 1 -- 172.21.15.59:0/700057503 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f325008a610 con 0x7f3268072a60 2024-03-06T07:25:51.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.636+0000 7f3265d9b640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f324806b4d0 0x7f324806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.636+0000 7f3265d9b640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f324806b4d0 0x7f324806d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3268159060 tx=0x7f3258009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.640+0000 7f325effd640 1 -- 172.21.15.59:0/700057503 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3250059490 con 0x7f3268072a60 2024-03-06T07:25:51.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.890+0000 7f326cdd0640 1 -- 172.21.15.59:0/700057503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f32680b25a0 con 0x7f3268072a60 2024-03-06T07:25:51.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.891+0000 7f325effd640 1 -- 172.21.15.59:0/700057503 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1594 (secure 0 0 0) 0x7f3250058d90 con 0x7f3268072a60 2024-03-06T07:25:51.893 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:51.893 INFO:teuthology.orchestra.run.smithi059.stdout:{"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":261,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":81,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709701,"num_in_osds":8,"osd_in_since":1709709692,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":78},{"state_name":"unknown","count":11},{"state_name":"creating+peering","count":8}],"num_pgs":97,"num_pools":3,"num_objects":19,"data_bytes":592084,"bytes_used":264626176,"bytes_avail":772829487104,"bytes_total":773094113280,"unknown_pgs_ratio":0.11340206116437912,"inactive_pgs_ratio":0.082474224269390106,"write_bytes_sec":2001,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":42,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"a","status":"up:active","gid":7002}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-03-06T07:25:35.348800+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:25:51.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 -- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f324806b4d0 msgr2=0x7f324806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f324806b4d0 0x7f324806d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3268159060 tx=0x7f3258009040 comp rx=0 tx=0).stop 2024-03-06T07:25:51.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 -- 172.21.15.59:0/700057503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3268072a60 msgr2=0x7f32681f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3268072a60 0x7f32681f6930 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f325000d5f0 tx=0x7f325000dab0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 -- 172.21.15.59:0/700057503 shutdown_connections 2024-03-06T07:25:51.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f324806b4d0 0x7f324806d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268161670 0x7f3268158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32680b0ea0 0x7f3268157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 --2- 172.21.15.59:0/700057503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3268072a60 0x7f32681f6930 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 -- 172.21.15.59:0/700057503 >> 172.21.15.59:0/700057503 conn(0x7f32680b4e60 msgr2=0x7f326806d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 -- 172.21.15.59:0/700057503 shutdown_connections 2024-03-06T07:25:51.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:51.893+0000 7f326cdd0640 1 -- 172.21.15.59:0/700057503 wait complete. 2024-03-06T07:25:51.905 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:25:51.906 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:52.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.043+0000 7efe500df640 1 Processor -- start 2024-03-06T07:25:52.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.044+0000 7efe500df640 1 -- start start 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.044+0000 7efe500df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe4815dfe0 0x7efe480cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.044+0000 7efe500df640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe480b3430 con 0x7efe480cf830 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.044+0000 7efe500df640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe480b3710 con 0x7efe480d29c0 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.044+0000 7efe500df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe480b31e0 con 0x7efe4815dfe0 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.044+0000 7efe4d67a640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7efe480cf830 0x7efe480d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50028/0 (socket says 172.21.15.59:50028) 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.044+0000 7efe4d67a640 1 -- 172.21.15.59:0/2000551141 learned_addr learned my addr 172.21.15.59:0/2000551141 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808489230 0 0) 0x7efe480b3430 con 0x7efe480cf830 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe30003520 con 0x7efe480cf830 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4de7b640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe4815dfe0 0x7efe480cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2699645846 0 0) 0x7efe30003520 con 0x7efe480cf830 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe4815dfe0 msgr2=0x7efe480cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe4815dfe0 0x7efe480cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 >> v1:172.21.15.164:6789/0 conn(0x7efe480d29c0 legacy=0x7efe480b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe480b39f0 con 0x7efe480cf830 2024-03-06T07:25:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7efe3c001670 con 0x7efe480cf830 2024-03-06T07:25:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 >> v1:172.21.15.59:6789/0 conn(0x7efe480cf830 legacy=0x7efe480d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe300039e0 0x7efe30003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe30004800 0x7efe30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe300051c0 0x7efe3000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe480b31e0 con 0x7efe30004800 2024-03-06T07:25:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3000c010 con 0x7efe300051c0 2024-03-06T07:25:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3000c210 con 0x7efe300039e0 2024-03-06T07:25:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.045+0000 7efe4d67a640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe30004800 0x7efe30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.046+0000 7efe4e67c640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe300051c0 0x7efe3000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.046+0000 7efe4de7b640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe300039e0 0x7efe30003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.046+0000 7efe4d67a640 1 -- 172.21.15.59:0/2000551141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe300039e0 msgr2=0x7efe30003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.046+0000 7efe4d67a640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe300039e0 0x7efe30003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.046+0000 7efe4d67a640 1 -- 172.21.15.59:0/2000551141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe300051c0 msgr2=0x7efe3000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.046+0000 7efe4d67a640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe300051c0 0x7efe3000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.046+0000 7efe4d67a640 1 -- 172.21.15.59:0/2000551141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe30003520 con 0x7efe30004800 2024-03-06T07:25:52.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.046+0000 7efe4d67a640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe30004800 0x7efe30004c80 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7efe3c00c0e0 tx=0x7efe30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c5758bdc3aafac6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.047+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c007c40 con 0x7efe30004800 2024-03-06T07:25:52.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.047+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3c030070 con 0x7efe30004800 2024-03-06T07:25:52.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.047+0000 7efe4ce79640 1 -- 172.21.15.59:0/2000551141 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c005d70 con 0x7efe30004800 2024-03-06T07:25:52.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.047+0000 7efe500df640 1 -- 172.21.15.59:0/2000551141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe30004800 msgr2=0x7efe30004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.047+0000 7efe500df640 1 --2- 172.21.15.59:0/2000551141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe30004800 0x7efe30004c80 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7efe3c00c0e0 tx=0x7efe30005830 comp rx=0 tx=0).stop 2024-03-06T07:25:52.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.047+0000 7efe4e67c640 1 -- 172.21.15.59:0/2000551141 reap_dead start 2024-03-06T07:25:52.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.047+0000 7efe500df640 1 -- 172.21.15.59:0/2000551141 shutdown_connections 2024-03-06T07:25:52.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.047+0000 7efe500df640 1 -- 172.21.15.59:0/2000551141 >> 172.21.15.59:0/2000551141 conn(0x7efe480d3cd0 msgr2=0x7efe480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.047+0000 7efe500df640 1 -- 172.21.15.59:0/2000551141 shutdown_connections 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.047+0000 7efe500df640 1 -- 172.21.15.59:0/2000551141 wait complete. 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.049+0000 7efe500df640 1 Processor -- start 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.049+0000 7efe500df640 1 -- start start 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.050+0000 7efe500df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe30004800 0x7efe4815b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.050+0000 7efe500df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe480b91b0 0x7efe480b9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.050+0000 7efe500df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe480b0e50 0x7efe480b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.050+0000 7efe500df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4815e3c0 con 0x7efe480b91b0 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.050+0000 7efe500df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe480b3d60 con 0x7efe30004800 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.050+0000 7efe500df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4815e540 con 0x7efe480b0e50 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.050+0000 7efe4d67a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe480b91b0 0x7efe480b9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe4d67a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe480b91b0 0x7efe480b9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46652/0 (socket says 172.21.15.59:46652) 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe4d67a640 1 -- 172.21.15.59:0/373037448 learned_addr learned my addr 172.21.15.59:0/373037448 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe4de7b640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe30004800 0x7efe4815b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe4e67c640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe480b0e50 0x7efe480b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe4d67a640 1 -- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe480b0e50 msgr2=0x7efe480b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe4d67a640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe480b0e50 0x7efe480b12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe4d67a640 1 -- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe30004800 msgr2=0x7efe4815b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe4d67a640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe30004800 0x7efe4815b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe4d67a640 1 -- 172.21.15.59:0/373037448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe481f3a70 con 0x7efe480b91b0 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe4e67c640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe480b0e50 0x7efe480b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe4d67a640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe480b91b0 0x7efe480b9630 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7efe3c00c0e0 tx=0x7efe480cfd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe467fc640 1 -- 172.21.15.59:0/373037448 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c004bf0 con 0x7efe480b91b0 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe467fc640 1 -- 172.21.15.59:0/373037448 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3c000ed0 con 0x7efe480b91b0 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe500df640 1 -- 172.21.15.59:0/373037448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe481f4ad0 con 0x7efe480b91b0 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe467fc640 1 -- 172.21.15.59:0/373037448 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c007070 con 0x7efe480b91b0 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.051+0000 7efe500df640 1 -- 172.21.15.59:0/373037448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe481f3ff0 con 0x7efe480b91b0 2024-03-06T07:25:52.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.052+0000 7efe500df640 1 -- 172.21.15.59:0/373037448 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4804f330 con 0x7efe480b91b0 2024-03-06T07:25:52.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.053+0000 7efe467fc640 1 -- 172.21.15.59:0/373037448 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe3c047020 con 0x7efe480b91b0 2024-03-06T07:25:52.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.053+0000 7efe467fc640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efe3406b4d0 0x7efe3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.053+0000 7efe467fc640 1 -- 172.21.15.59:0/373037448 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe3c0ab270 con 0x7efe480b91b0 2024-03-06T07:25:52.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.053+0000 7efe4de7b640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efe3406b4d0 0x7efe3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.054+0000 7efe4de7b640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efe3406b4d0 0x7efe3406d990 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7efe48067990 tx=0x7efe40008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.057+0000 7efe467fc640 1 -- 172.21.15.59:0/373037448 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe3c07a0f0 con 0x7efe480b91b0 2024-03-06T07:25:52.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.236+0000 7efe500df640 1 -- 172.21.15.59:0/373037448 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe480d2500 con 0x7efe3406b4d0 2024-03-06T07:25:52.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.242+0000 7efe467fc640 1 -- 172.21.15.59:0/373037448 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+249919 (secure 0 0 0) 0x7efe480d2500 con 0x7efe3406b4d0 2024-03-06T07:25:52.246 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:52.248 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:25:52.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.245+0000 7efe500df640 1 -- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efe3406b4d0 msgr2=0x7efe3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.245+0000 7efe500df640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efe3406b4d0 0x7efe3406d990 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7efe48067990 tx=0x7efe40008040 comp rx=0 tx=0).stop 2024-03-06T07:25:52.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.245+0000 7efe500df640 1 -- 172.21.15.59:0/373037448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe480b91b0 msgr2=0x7efe480b9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.245+0000 7efe500df640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe480b91b0 0x7efe480b9630 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7efe3c00c0e0 tx=0x7efe480cfd40 comp rx=0 tx=0).stop 2024-03-06T07:25:52.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.245+0000 7efe500df640 1 -- 172.21.15.59:0/373037448 shutdown_connections 2024-03-06T07:25:52.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.245+0000 7efe500df640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efe3406b4d0 0x7efe3406d990 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.245+0000 7efe500df640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe480b0e50 0x7efe480b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.245+0000 7efe500df640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe480b91b0 0x7efe480b9630 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.245+0000 7efe500df640 1 --2- 172.21.15.59:0/373037448 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe30004800 0x7efe4815b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.245+0000 7efe500df640 1 -- 172.21.15.59:0/373037448 >> 172.21.15.59:0/373037448 conn(0x7efe480d3cd0 msgr2=0x7efe4806ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.246+0000 7efe500df640 1 -- 172.21.15.59:0/373037448 shutdown_connections 2024-03-06T07:25:52.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:52.246+0000 7efe500df640 1 -- 172.21.15.59:0/373037448 wait complete. 2024-03-06T07:25:52.261 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-03-06T07:25:51.351996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592084,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":195,"num_write_kb":2427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":172,"acting":172,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258424,"kb_used_data":8952,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716296,"statfs":{"total":773094113280,"available":772829487104,"internally_reserved":0,"allocated":9166848,"data_stored":4544216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1716,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.627688"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760083+0000","last_change":"2024-03-06T07:25:47.754942+0000","last_active":"2024-03-06T07:25:48.760083+0000","last_peered":"2024-03-06T07:25:48.760083+0000","last_clean":"2024-03-06T07:25:48.760083+0000","last_became_active":"2024-03-06T07:25:47.753877+0000","last_became_peered":"2024-03-06T07:25:47.753877+0000","last_unstale":"2024-03-06T07:25:48.760083+0000","last_undegraded":"2024-03-06T07:25:48.760083+0000","last_fullsized":"2024-03-06T07:25:48.760083+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:52.540026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040115+0000","last_change":"2024-03-06T07:25:47.750010+0000","last_active":"2024-03-06T07:25:49.040115+0000","last_peered":"2024-03-06T07:25:49.040115+0000","last_clean":"2024-03-06T07:25:49.040115+0000","last_became_active":"2024-03-06T07:25:47.749735+0000","last_became_peered":"2024-03-06T07:25:47.749735+0000","last_unstale":"2024-03-06T07:25:49.040115+0000","last_undegraded":"2024-03-06T07:25:49.040115+0000","last_fullsized":"2024-03-06T07:25:49.040115+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:41.897733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736087+0000","last_change":"2024-03-06T07:25:47.750010+0000","last_active":"2024-03-06T07:25:48.736087+0000","last_peered":"2024-03-06T07:25:48.736087+0000","last_clean":"2024-03-06T07:25:48.736087+0000","last_became_active":"2024-03-06T07:25:47.749708+0000","last_became_peered":"2024-03-06T07:25:47.749708+0000","last_unstale":"2024-03-06T07:25:48.736087+0000","last_undegraded":"2024-03-06T07:25:48.736087+0000","last_fullsized":"2024-03-06T07:25:48.736087+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:49.954151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737396+0000","last_change":"2024-03-06T07:25:47.747729+0000","last_active":"2024-03-06T07:25:48.737396+0000","last_peered":"2024-03-06T07:25:48.737396+0000","last_clean":"2024-03-06T07:25:48.737396+0000","last_became_active":"2024-03-06T07:25:47.747442+0000","last_became_peered":"2024-03-06T07:25:47.747442+0000","last_unstale":"2024-03-06T07:25:48.737396+0000","last_undegraded":"2024-03-06T07:25:48.737396+0000","last_fullsized":"2024-03-06T07:25:48.737396+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:53.366551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737419+0000","last_change":"2024-03-06T07:25:47.751731+0000","last_active":"2024-03-06T07:25:48.737419+0000","last_peered":"2024-03-06T07:25:48.737419+0000","last_clean":"2024-03-06T07:25:48.737419+0000","last_became_active":"2024-03-06T07:25:47.751445+0000","last_became_peered":"2024-03-06T07:25:47.751445+0000","last_unstale":"2024-03-06T07:25:48.737419+0000","last_undegraded":"2024-03-06T07:25:48.737419+0000","last_fullsized":"2024-03-06T07:25:48.737419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:08.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735762+0000","last_change":"2024-03-06T07:25:47.747479+0000","last_active":"2024-03-06T07:25:48.735762+0000","last_peered":"2024-03-06T07:25:48.735762+0000","last_clean":"2024-03-06T07:25:48.735762+0000","last_became_active":"2024-03-06T07:25:47.747186+0000","last_became_peered":"2024-03-06T07:25:47.747186+0000","last_unstale":"2024-03-06T07:25:48.735762+0000","last_undegraded":"2024-03-06T07:25:48.735762+0000","last_fullsized":"2024-03-06T07:25:48.735762+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:38.441834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736301+0000","last_change":"2024-03-06T07:25:47.742812+0000","last_active":"2024-03-06T07:25:48.736301+0000","last_peered":"2024-03-06T07:25:48.736301+0000","last_clean":"2024-03-06T07:25:48.736301+0000","last_became_active":"2024-03-06T07:25:47.739811+0000","last_became_peered":"2024-03-06T07:25:47.739811+0000","last_unstale":"2024-03-06T07:25:48.736301+0000","last_undegraded":"2024-03-06T07:25:48.736301+0000","last_fullsized":"2024-03-06T07:25:48.736301+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:17.837597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736920+0000","last_change":"2024-03-06T07:25:47.760422+0000","last_active":"2024-03-06T07:25:48.736920+0000","last_peered":"2024-03-06T07:25:48.736920+0000","last_clean":"2024-03-06T07:25:48.736920+0000","last_became_active":"2024-03-06T07:25:47.760129+0000","last_became_peered":"2024-03-06T07:25:47.760129+0000","last_unstale":"2024-03-06T07:25:48.736920+0000","last_undegraded":"2024-03-06T07:25:48.736920+0000","last_fullsized":"2024-03-06T07:25:48.736920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:52.495587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760008+0000","last_change":"2024-03-06T07:25:47.745385+0000","last_active":"2024-03-06T07:25:48.760008+0000","last_peered":"2024-03-06T07:25:48.760008+0000","last_clean":"2024-03-06T07:25:48.760008+0000","last_became_active":"2024-03-06T07:25:47.745117+0000","last_became_peered":"2024-03-06T07:25:47.745117+0000","last_unstale":"2024-03-06T07:25:48.760008+0000","last_undegraded":"2024-03-06T07:25:48.760008+0000","last_fullsized":"2024-03-06T07:25:48.760008+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:40.310991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039620+0000","last_change":"2024-03-06T07:25:47.749258+0000","last_active":"2024-03-06T07:25:49.039620+0000","last_peered":"2024-03-06T07:25:49.039620+0000","last_clean":"2024-03-06T07:25:49.039620+0000","last_became_active":"2024-03-06T07:25:47.748993+0000","last_became_peered":"2024-03-06T07:25:47.748993+0000","last_unstale":"2024-03-06T07:25:49.039620+0000","last_undegraded":"2024-03-06T07:25:49.039620+0000","last_fullsized":"2024-03-06T07:25:49.039620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:53.366551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735907+0000","last_change":"2024-03-06T07:25:47.750890+0000","last_active":"2024-03-06T07:25:48.735907+0000","last_peered":"2024-03-06T07:25:48.735907+0000","last_clean":"2024-03-06T07:25:48.735907+0000","last_became_active":"2024-03-06T07:25:47.750579+0000","last_became_peered":"2024-03-06T07:25:47.750579+0000","last_unstale":"2024-03-06T07:25:48.735907+0000","last_undegraded":"2024-03-06T07:25:48.735907+0000","last_fullsized":"2024-03-06T07:25:48.735907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:12.102084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039116+0000","last_change":"2024-03-06T07:25:47.749794+0000","last_active":"2024-03-06T07:25:49.039116+0000","last_peered":"2024-03-06T07:25:49.039116+0000","last_clean":"2024-03-06T07:25:49.039116+0000","last_became_active":"2024-03-06T07:25:47.749189+0000","last_became_peered":"2024-03-06T07:25:47.749189+0000","last_unstale":"2024-03-06T07:25:49.039116+0000","last_undegraded":"2024-03-06T07:25:49.039116+0000","last_fullsized":"2024-03-06T07:25:49.039116+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:49.406968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736570+0000","last_change":"2024-03-06T07:25:47.753561+0000","last_active":"2024-03-06T07:25:48.736570+0000","last_peered":"2024-03-06T07:25:48.736570+0000","last_clean":"2024-03-06T07:25:48.736570+0000","last_became_active":"2024-03-06T07:25:47.753272+0000","last_became_peered":"2024-03-06T07:25:47.753272+0000","last_unstale":"2024-03-06T07:25:48.736570+0000","last_undegraded":"2024-03-06T07:25:48.736570+0000","last_fullsized":"2024-03-06T07:25:48.736570+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:23.030627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737208+0000","last_change":"2024-03-06T07:25:47.752307+0000","last_active":"2024-03-06T07:25:48.737208+0000","last_peered":"2024-03-06T07:25:48.737208+0000","last_clean":"2024-03-06T07:25:48.737208+0000","last_became_active":"2024-03-06T07:25:47.752046+0000","last_became_peered":"2024-03-06T07:25:47.752046+0000","last_unstale":"2024-03-06T07:25:48.737208+0000","last_undegraded":"2024-03-06T07:25:48.737208+0000","last_fullsized":"2024-03-06T07:25:48.737208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.822094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736117+0000","last_change":"2024-03-06T07:25:47.750762+0000","last_active":"2024-03-06T07:25:48.736117+0000","last_peered":"2024-03-06T07:25:48.736117+0000","last_clean":"2024-03-06T07:25:48.736117+0000","last_became_active":"2024-03-06T07:25:47.750456+0000","last_became_peered":"2024-03-06T07:25:47.750456+0000","last_unstale":"2024-03-06T07:25:48.736117+0000","last_undegraded":"2024-03-06T07:25:48.736117+0000","last_fullsized":"2024-03-06T07:25:48.736117+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:28.272446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.762159+0000","last_change":"2024-03-06T07:25:47.754357+0000","last_active":"2024-03-06T07:25:48.762159+0000","last_peered":"2024-03-06T07:25:48.762159+0000","last_clean":"2024-03-06T07:25:48.762159+0000","last_became_active":"2024-03-06T07:25:47.754043+0000","last_became_peered":"2024-03-06T07:25:47.754043+0000","last_unstale":"2024-03-06T07:25:48.762159+0000","last_undegraded":"2024-03-06T07:25:48.762159+0000","last_fullsized":"2024-03-06T07:25:48.762159+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.822094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736225+0000","last_change":"2024-03-06T07:25:47.756728+0000","last_active":"2024-03-06T07:25:48.736225+0000","last_peered":"2024-03-06T07:25:48.736225+0000","last_clean":"2024-03-06T07:25:48.736225+0000","last_became_active":"2024-03-06T07:25:47.756364+0000","last_became_peered":"2024-03-06T07:25:47.756364+0000","last_unstale":"2024-03-06T07:25:48.736225+0000","last_undegraded":"2024-03-06T07:25:48.736225+0000","last_fullsized":"2024-03-06T07:25:48.736225+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:35.409269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735980+0000","last_change":"2024-03-06T07:25:47.760935+0000","last_active":"2024-03-06T07:25:48.735980+0000","last_peered":"2024-03-06T07:25:48.735980+0000","last_clean":"2024-03-06T07:25:48.735980+0000","last_became_active":"2024-03-06T07:25:47.760648+0000","last_became_peered":"2024-03-06T07:25:47.760648+0000","last_unstale":"2024-03-06T07:25:48.735980+0000","last_undegraded":"2024-03-06T07:25:48.735980+0000","last_fullsized":"2024-03-06T07:25:48.735980+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:09.073261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759561+0000","last_change":"2024-03-06T07:25:47.760858+0000","last_active":"2024-03-06T07:25:48.759561+0000","last_peered":"2024-03-06T07:25:48.759561+0000","last_clean":"2024-03-06T07:25:48.759561+0000","last_became_active":"2024-03-06T07:25:47.760570+0000","last_became_peered":"2024-03-06T07:25:47.760570+0000","last_unstale":"2024-03-06T07:25:48.759561+0000","last_undegraded":"2024-03-06T07:25:48.759561+0000","last_fullsized":"2024-03-06T07:25:48.759561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:27.486710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735731+0000","last_change":"2024-03-06T07:25:47.761368+0000","last_active":"2024-03-06T07:25:48.735731+0000","last_peered":"2024-03-06T07:25:48.735731+0000","last_clean":"2024-03-06T07:25:48.735731+0000","last_became_active":"2024-03-06T07:25:47.761104+0000","last_became_peered":"2024-03-06T07:25:47.761104+0000","last_unstale":"2024-03-06T07:25:48.735731+0000","last_undegraded":"2024-03-06T07:25:48.735731+0000","last_fullsized":"2024-03-06T07:25:48.735731+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:58.946010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760082+0000","last_change":"2024-03-06T07:25:47.756504+0000","last_active":"2024-03-06T07:25:48.760082+0000","last_peered":"2024-03-06T07:25:48.760082+0000","last_clean":"2024-03-06T07:25:48.760082+0000","last_became_active":"2024-03-06T07:25:47.756231+0000","last_became_peered":"2024-03-06T07:25:47.756231+0000","last_unstale":"2024-03-06T07:25:48.760082+0000","last_undegraded":"2024-03-06T07:25:48.760082+0000","last_fullsized":"2024-03-06T07:25:48.760082+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:17.837597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039678+0000","last_change":"2024-03-06T07:25:47.755618+0000","last_active":"2024-03-06T07:25:49.039678+0000","last_peered":"2024-03-06T07:25:49.039678+0000","last_clean":"2024-03-06T07:25:49.039678+0000","last_became_active":"2024-03-06T07:25:47.755092+0000","last_became_peered":"2024-03-06T07:25:47.755092+0000","last_unstale":"2024-03-06T07:25:49.039678+0000","last_undegraded":"2024-03-06T07:25:49.039678+0000","last_fullsized":"2024-03-06T07:25:49.039678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:24.401568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736402+0000","last_change":"2024-03-06T07:25:47.751435+0000","last_active":"2024-03-06T07:25:48.736402+0000","last_peered":"2024-03-06T07:25:48.736402+0000","last_clean":"2024-03-06T07:25:48.736402+0000","last_became_active":"2024-03-06T07:25:47.751146+0000","last_became_peered":"2024-03-06T07:25:47.751146+0000","last_unstale":"2024-03-06T07:25:48.736402+0000","last_undegraded":"2024-03-06T07:25:48.736402+0000","last_fullsized":"2024-03-06T07:25:48.736402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:52.495587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759695+0000","last_change":"2024-03-06T07:25:47.739643+0000","last_active":"2024-03-06T07:25:48.759695+0000","last_peered":"2024-03-06T07:25:48.759695+0000","last_clean":"2024-03-06T07:25:48.759695+0000","last_became_active":"2024-03-06T07:25:47.739353+0000","last_became_peered":"2024-03-06T07:25:47.739353+0000","last_unstale":"2024-03-06T07:25:48.759695+0000","last_undegraded":"2024-03-06T07:25:48.759695+0000","last_fullsized":"2024-03-06T07:25:48.759695+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:09.040328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759629+0000","last_change":"2024-03-06T07:25:47.753200+0000","last_active":"2024-03-06T07:25:48.759629+0000","last_peered":"2024-03-06T07:25:48.759629+0000","last_clean":"2024-03-06T07:25:48.759629+0000","last_became_active":"2024-03-06T07:25:47.752393+0000","last_became_peered":"2024-03-06T07:25:47.752393+0000","last_unstale":"2024-03-06T07:25:48.759629+0000","last_undegraded":"2024-03-06T07:25:48.759629+0000","last_fullsized":"2024-03-06T07:25:48.759629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:57.303162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736327+0000","last_change":"2024-03-06T07:25:47.761829+0000","last_active":"2024-03-06T07:25:48.736327+0000","last_peered":"2024-03-06T07:25:48.736327+0000","last_clean":"2024-03-06T07:25:48.736327+0000","last_became_active":"2024-03-06T07:25:47.761583+0000","last_became_peered":"2024-03-06T07:25:47.761583+0000","last_unstale":"2024-03-06T07:25:48.736327+0000","last_undegraded":"2024-03-06T07:25:48.736327+0000","last_fullsized":"2024-03-06T07:25:48.736327+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:19.894041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737151+0000","last_change":"2024-03-06T07:25:47.750725+0000","last_active":"2024-03-06T07:25:48.737151+0000","last_peered":"2024-03-06T07:25:48.737151+0000","last_clean":"2024-03-06T07:25:48.737151+0000","last_became_active":"2024-03-06T07:25:47.750427+0000","last_became_peered":"2024-03-06T07:25:47.750427+0000","last_unstale":"2024-03-06T07:25:48.737151+0000","last_undegraded":"2024-03-06T07:25:48.737151+0000","last_fullsized":"2024-03-06T07:25:48.737151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:09.013439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729992+0000","last_change":"2024-03-06T07:25:45.729992+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729992+0000","last_undegraded":"2024-03-06T07:25:45.729992+0000","last_fullsized":"2024-03-06T07:25:45.729992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737231+0000","last_change":"2024-03-06T07:25:46.741560+0000","last_active":"2024-03-06T07:25:48.737231+0000","last_peered":"2024-03-06T07:25:48.737231+0000","last_clean":"2024-03-06T07:25:48.737231+0000","last_became_active":"2024-03-06T07:25:46.741225+0000","last_became_peered":"2024-03-06T07:25:46.741225+0000","last_unstale":"2024-03-06T07:25:48.737231+0000","last_undegraded":"2024-03-06T07:25:48.737231+0000","last_fullsized":"2024-03-06T07:25:48.737231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:42.221129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.762098+0000","last_change":"2024-03-06T07:25:46.741655+0000","last_active":"2024-03-06T07:25:48.762098+0000","last_peered":"2024-03-06T07:25:48.762098+0000","last_clean":"2024-03-06T07:25:48.762098+0000","last_became_active":"2024-03-06T07:25:46.741374+0000","last_became_peered":"2024-03-06T07:25:46.741374+0000","last_unstale":"2024-03-06T07:25:48.762098+0000","last_undegraded":"2024-03-06T07:25:48.762098+0000","last_fullsized":"2024-03-06T07:25:48.762098+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:16.833042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040274+0000","last_change":"2024-03-06T07:25:47.748531+0000","last_active":"2024-03-06T07:25:49.040274+0000","last_peered":"2024-03-06T07:25:49.040274+0000","last_clean":"2024-03-06T07:25:49.040274+0000","last_became_active":"2024-03-06T07:25:47.748272+0000","last_became_peered":"2024-03-06T07:25:47.748272+0000","last_unstale":"2024-03-06T07:25:49.040274+0000","last_undegraded":"2024-03-06T07:25:49.040274+0000","last_fullsized":"2024-03-06T07:25:49.040274+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:52.756249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761801+0000","last_change":"2024-03-06T07:25:46.742202+0000","last_active":"2024-03-06T07:25:48.761801+0000","last_peered":"2024-03-06T07:25:48.761801+0000","last_clean":"2024-03-06T07:25:48.761801+0000","last_became_active":"2024-03-06T07:25:46.741325+0000","last_became_peered":"2024-03-06T07:25:46.741325+0000","last_unstale":"2024-03-06T07:25:48.761801+0000","last_undegraded":"2024-03-06T07:25:48.761801+0000","last_fullsized":"2024-03-06T07:25:48.761801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:44.989067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759670+0000","last_change":"2024-03-06T07:25:47.747094+0000","last_active":"2024-03-06T07:25:48.759670+0000","last_peered":"2024-03-06T07:25:48.759670+0000","last_clean":"2024-03-06T07:25:48.759670+0000","last_became_active":"2024-03-06T07:25:47.746796+0000","last_became_peered":"2024-03-06T07:25:47.746796+0000","last_unstale":"2024-03-06T07:25:48.759670+0000","last_undegraded":"2024-03-06T07:25:48.759670+0000","last_fullsized":"2024-03-06T07:25:48.759670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:13.850354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737511+0000","last_change":"2024-03-06T07:25:46.741705+0000","last_active":"2024-03-06T07:25:48.737511+0000","last_peered":"2024-03-06T07:25:48.737511+0000","last_clean":"2024-03-06T07:25:48.737511+0000","last_became_active":"2024-03-06T07:25:46.741411+0000","last_became_peered":"2024-03-06T07:25:46.741411+0000","last_unstale":"2024-03-06T07:25:48.737511+0000","last_undegraded":"2024-03-06T07:25:48.737511+0000","last_fullsized":"2024-03-06T07:25:48.737511+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:52.885965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737431+0000","last_change":"2024-03-06T07:25:47.753035+0000","last_active":"2024-03-06T07:25:48.737431+0000","last_peered":"2024-03-06T07:25:48.737431+0000","last_clean":"2024-03-06T07:25:48.737431+0000","last_became_active":"2024-03-06T07:25:47.752710+0000","last_became_peered":"2024-03-06T07:25:47.752710+0000","last_unstale":"2024-03-06T07:25:48.737431+0000","last_undegraded":"2024-03-06T07:25:48.737431+0000","last_fullsized":"2024-03-06T07:25:48.737431+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:24.401568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736619+0000","last_change":"2024-03-06T07:25:46.741681+0000","last_active":"2024-03-06T07:25:48.736619+0000","last_peered":"2024-03-06T07:25:48.736619+0000","last_clean":"2024-03-06T07:25:48.736619+0000","last_became_active":"2024-03-06T07:25:46.741397+0000","last_became_peered":"2024-03-06T07:25:46.741397+0000","last_unstale":"2024-03-06T07:25:48.736619+0000","last_undegraded":"2024-03-06T07:25:48.736619+0000","last_fullsized":"2024-03-06T07:25:48.736619+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:32.000019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039562+0000","last_change":"2024-03-06T07:25:47.759466+0000","last_active":"2024-03-06T07:25:49.039562+0000","last_peered":"2024-03-06T07:25:49.039562+0000","last_clean":"2024-03-06T07:25:49.039562+0000","last_became_active":"2024-03-06T07:25:47.759185+0000","last_became_peered":"2024-03-06T07:25:47.759185+0000","last_unstale":"2024-03-06T07:25:49.039562+0000","last_undegraded":"2024-03-06T07:25:49.039562+0000","last_fullsized":"2024-03-06T07:25:49.039562+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.822094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761641+0000","last_change":"2024-03-06T07:25:46.737587+0000","last_active":"2024-03-06T07:25:48.761641+0000","last_peered":"2024-03-06T07:25:48.761641+0000","last_clean":"2024-03-06T07:25:48.761641+0000","last_became_active":"2024-03-06T07:25:46.737289+0000","last_became_peered":"2024-03-06T07:25:46.737289+0000","last_unstale":"2024-03-06T07:25:48.761641+0000","last_undegraded":"2024-03-06T07:25:48.761641+0000","last_fullsized":"2024-03-06T07:25:48.761641+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:08.035773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039138+0000","last_change":"2024-03-06T07:25:47.749535+0000","last_active":"2024-03-06T07:25:49.039138+0000","last_peered":"2024-03-06T07:25:49.039138+0000","last_clean":"2024-03-06T07:25:49.039138+0000","last_became_active":"2024-03-06T07:25:47.748786+0000","last_became_peered":"2024-03-06T07:25:47.748786+0000","last_unstale":"2024-03-06T07:25:49.039138+0000","last_undegraded":"2024-03-06T07:25:49.039138+0000","last_fullsized":"2024-03-06T07:25:49.039138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:57.368763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759500+0000","last_change":"2024-03-06T07:25:46.741890+0000","last_active":"2024-03-06T07:25:48.759500+0000","last_peered":"2024-03-06T07:25:48.759500+0000","last_clean":"2024-03-06T07:25:48.759500+0000","last_became_active":"2024-03-06T07:25:46.741566+0000","last_became_peered":"2024-03-06T07:25:46.741566+0000","last_unstale":"2024-03-06T07:25:48.759500+0000","last_undegraded":"2024-03-06T07:25:48.759500+0000","last_fullsized":"2024-03-06T07:25:48.759500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:53.583814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039409+0000","last_change":"2024-03-06T07:25:47.741722+0000","last_active":"2024-03-06T07:25:49.039409+0000","last_peered":"2024-03-06T07:25:49.039409+0000","last_clean":"2024-03-06T07:25:49.039409+0000","last_became_active":"2024-03-06T07:25:47.741384+0000","last_became_peered":"2024-03-06T07:25:47.741384+0000","last_unstale":"2024-03-06T07:25:49.039409+0000","last_undegraded":"2024-03-06T07:25:49.039409+0000","last_fullsized":"2024-03-06T07:25:49.039409+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:26.696427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729420+0000","last_change":"2024-03-06T07:25:45.729420+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729420+0000","last_undegraded":"2024-03-06T07:25:45.729420+0000","last_fullsized":"2024-03-06T07:25:45.729420+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737099+0000","last_change":"2024-03-06T07:25:47.752574+0000","last_active":"2024-03-06T07:25:48.737099+0000","last_peered":"2024-03-06T07:25:48.737099+0000","last_clean":"2024-03-06T07:25:48.737099+0000","last_became_active":"2024-03-06T07:25:47.752246+0000","last_became_peered":"2024-03-06T07:25:47.752246+0000","last_unstale":"2024-03-06T07:25:48.737099+0000","last_undegraded":"2024-03-06T07:25:48.737099+0000","last_fullsized":"2024-03-06T07:25:48.737099+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:23.030627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728800+0000","last_change":"2024-03-06T07:25:45.728800+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728800+0000","last_undegraded":"2024-03-06T07:25:45.728800+0000","last_fullsized":"2024-03-06T07:25:45.728800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.762062+0000","last_change":"2024-03-06T07:25:47.755416+0000","last_active":"2024-03-06T07:25:48.762062+0000","last_peered":"2024-03-06T07:25:48.762062+0000","last_clean":"2024-03-06T07:25:48.762062+0000","last_became_active":"2024-03-06T07:25:47.755154+0000","last_became_peered":"2024-03-06T07:25:47.755154+0000","last_unstale":"2024-03-06T07:25:48.762062+0000","last_undegraded":"2024-03-06T07:25:48.762062+0000","last_fullsized":"2024-03-06T07:25:48.762062+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:09.013439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728545+0000","last_change":"2024-03-06T07:25:45.728545+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728545+0000","last_undegraded":"2024-03-06T07:25:45.728545+0000","last_fullsized":"2024-03-06T07:25:45.728545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736937+0000","last_change":"2024-03-06T07:25:47.761129+0000","last_active":"2024-03-06T07:25:48.736937+0000","last_peered":"2024-03-06T07:25:48.736937+0000","last_clean":"2024-03-06T07:25:48.736937+0000","last_became_active":"2024-03-06T07:25:47.760855+0000","last_became_peered":"2024-03-06T07:25:47.760855+0000","last_unstale":"2024-03-06T07:25:48.736937+0000","last_undegraded":"2024-03-06T07:25:48.736937+0000","last_fullsized":"2024-03-06T07:25:48.736937+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:08.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736791+0000","last_change":"2024-03-06T07:25:46.732588+0000","last_active":"2024-03-06T07:25:48.736791+0000","last_peered":"2024-03-06T07:25:48.736791+0000","last_clean":"2024-03-06T07:25:48.736791+0000","last_became_active":"2024-03-06T07:25:46.732252+0000","last_became_peered":"2024-03-06T07:25:46.732252+0000","last_unstale":"2024-03-06T07:25:48.736791+0000","last_undegraded":"2024-03-06T07:25:48.736791+0000","last_fullsized":"2024-03-06T07:25:48.736791+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:45.368244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761800+0000","last_change":"2024-03-06T07:25:47.759623+0000","last_active":"2024-03-06T07:25:48.761800+0000","last_peered":"2024-03-06T07:25:48.761800+0000","last_clean":"2024-03-06T07:25:48.761800+0000","last_became_active":"2024-03-06T07:25:47.759263+0000","last_became_peered":"2024-03-06T07:25:47.759263+0000","last_unstale":"2024-03-06T07:25:48.761800+0000","last_undegraded":"2024-03-06T07:25:48.761800+0000","last_fullsized":"2024-03-06T07:25:48.761800+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.952554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759773+0000","last_change":"2024-03-06T07:25:46.746659+0000","last_active":"2024-03-06T07:25:48.759773+0000","last_peered":"2024-03-06T07:25:48.759773+0000","last_clean":"2024-03-06T07:25:48.759773+0000","last_became_active":"2024-03-06T07:25:46.746246+0000","last_became_peered":"2024-03-06T07:25:46.746246+0000","last_unstale":"2024-03-06T07:25:48.759773+0000","last_undegraded":"2024-03-06T07:25:48.759773+0000","last_fullsized":"2024-03-06T07:25:48.759773+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:32.988256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039919+0000","last_change":"2024-03-06T07:25:47.760007+0000","last_active":"2024-03-06T07:25:49.039919+0000","last_peered":"2024-03-06T07:25:49.039919+0000","last_clean":"2024-03-06T07:25:49.039919+0000","last_became_active":"2024-03-06T07:25:47.759722+0000","last_became_peered":"2024-03-06T07:25:47.759722+0000","last_unstale":"2024-03-06T07:25:49.039919+0000","last_undegraded":"2024-03-06T07:25:49.039919+0000","last_fullsized":"2024-03-06T07:25:49.039919+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:30.829345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039266+0000","last_change":"2024-03-06T07:25:46.741231+0000","last_active":"2024-03-06T07:25:49.039266+0000","last_peered":"2024-03-06T07:25:49.039266+0000","last_clean":"2024-03-06T07:25:49.039266+0000","last_became_active":"2024-03-06T07:25:46.740951+0000","last_became_peered":"2024-03-06T07:25:46.740951+0000","last_unstale":"2024-03-06T07:25:49.039266+0000","last_undegraded":"2024-03-06T07:25:49.039266+0000","last_fullsized":"2024-03-06T07:25:49.039266+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:40.893178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759533+0000","last_change":"2024-03-06T07:25:47.746838+0000","last_active":"2024-03-06T07:25:48.759533+0000","last_peered":"2024-03-06T07:25:48.759533+0000","last_clean":"2024-03-06T07:25:48.759533+0000","last_became_active":"2024-03-06T07:25:47.746563+0000","last_became_peered":"2024-03-06T07:25:47.746563+0000","last_unstale":"2024-03-06T07:25:48.759533+0000","last_undegraded":"2024-03-06T07:25:48.759533+0000","last_fullsized":"2024-03-06T07:25:48.759533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:10.615032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736434+0000","last_change":"2024-03-06T07:25:46.741981+0000","last_active":"2024-03-06T07:25:48.736434+0000","last_peered":"2024-03-06T07:25:48.736434+0000","last_clean":"2024-03-06T07:25:48.736434+0000","last_became_active":"2024-03-06T07:25:46.741709+0000","last_became_peered":"2024-03-06T07:25:46.741709+0000","last_unstale":"2024-03-06T07:25:48.736434+0000","last_undegraded":"2024-03-06T07:25:48.736434+0000","last_fullsized":"2024-03-06T07:25:48.736434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:33.944174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759470+0000","last_change":"2024-03-06T07:25:47.755828+0000","last_active":"2024-03-06T07:25:48.759470+0000","last_peered":"2024-03-06T07:25:48.759470+0000","last_clean":"2024-03-06T07:25:48.759470+0000","last_became_active":"2024-03-06T07:25:47.755559+0000","last_became_peered":"2024-03-06T07:25:47.755559+0000","last_unstale":"2024-03-06T07:25:48.759470+0000","last_undegraded":"2024-03-06T07:25:48.759470+0000","last_fullsized":"2024-03-06T07:25:48.759470+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:34.948730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761798+0000","last_change":"2024-03-06T07:25:46.743744+0000","last_active":"2024-03-06T07:25:48.761798+0000","last_peered":"2024-03-06T07:25:48.761798+0000","last_clean":"2024-03-06T07:25:48.761798+0000","last_became_active":"2024-03-06T07:25:46.743447+0000","last_became_peered":"2024-03-06T07:25:46.743447+0000","last_unstale":"2024-03-06T07:25:48.761798+0000","last_undegraded":"2024-03-06T07:25:48.761798+0000","last_fullsized":"2024-03-06T07:25:48.761798+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:56.298607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039836+0000","last_change":"2024-03-06T07:25:47.747720+0000","last_active":"2024-03-06T07:25:49.039836+0000","last_peered":"2024-03-06T07:25:49.039836+0000","last_clean":"2024-03-06T07:25:49.039836+0000","last_became_active":"2024-03-06T07:25:47.747412+0000","last_became_peered":"2024-03-06T07:25:47.747412+0000","last_unstale":"2024-03-06T07:25:49.039836+0000","last_undegraded":"2024-03-06T07:25:49.039836+0000","last_fullsized":"2024-03-06T07:25:49.039836+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:10.096097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728561+0000","last_change":"2024-03-06T07:25:45.728561+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728561+0000","last_undegraded":"2024-03-06T07:25:45.728561+0000","last_fullsized":"2024-03-06T07:25:45.728561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761941+0000","last_change":"2024-03-06T07:25:46.744401+0000","last_active":"2024-03-06T07:25:48.761941+0000","last_peered":"2024-03-06T07:25:48.761941+0000","last_clean":"2024-03-06T07:25:48.761941+0000","last_became_active":"2024-03-06T07:25:46.744090+0000","last_became_peered":"2024-03-06T07:25:46.744090+0000","last_unstale":"2024-03-06T07:25:48.761941+0000","last_undegraded":"2024-03-06T07:25:48.761941+0000","last_fullsized":"2024-03-06T07:25:48.761941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:51.535471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737048+0000","last_change":"2024-03-06T07:25:47.748162+0000","last_active":"2024-03-06T07:25:48.737048+0000","last_peered":"2024-03-06T07:25:48.737048+0000","last_clean":"2024-03-06T07:25:48.737048+0000","last_became_active":"2024-03-06T07:25:47.747901+0000","last_became_peered":"2024-03-06T07:25:47.747901+0000","last_unstale":"2024-03-06T07:25:48.737048+0000","last_undegraded":"2024-03-06T07:25:48.737048+0000","last_fullsized":"2024-03-06T07:25:48.737048+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.952554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761902+0000","last_change":"2024-03-06T07:25:46.736412+0000","last_active":"2024-03-06T07:25:48.761902+0000","last_peered":"2024-03-06T07:25:48.761902+0000","last_clean":"2024-03-06T07:25:48.761902+0000","last_became_active":"2024-03-06T07:25:46.735973+0000","last_became_peered":"2024-03-06T07:25:46.735973+0000","last_unstale":"2024-03-06T07:25:48.761902+0000","last_undegraded":"2024-03-06T07:25:48.761902+0000","last_fullsized":"2024-03-06T07:25:48.761902+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:39.306436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735938+0000","last_change":"2024-03-06T07:25:47.748114+0000","last_active":"2024-03-06T07:25:48.735938+0000","last_peered":"2024-03-06T07:25:48.735938+0000","last_clean":"2024-03-06T07:25:48.735938+0000","last_became_active":"2024-03-06T07:25:47.747802+0000","last_became_peered":"2024-03-06T07:25:47.747802+0000","last_unstale":"2024-03-06T07:25:48.735938+0000","last_undegraded":"2024-03-06T07:25:48.735938+0000","last_fullsized":"2024-03-06T07:25:48.735938+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:27.832719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039237+0000","last_change":"2024-03-06T07:25:46.740390+0000","last_active":"2024-03-06T07:25:49.039237+0000","last_peered":"2024-03-06T07:25:49.039237+0000","last_clean":"2024-03-06T07:25:49.039237+0000","last_became_active":"2024-03-06T07:25:46.740119+0000","last_became_peered":"2024-03-06T07:25:46.740119+0000","last_unstale":"2024-03-06T07:25:49.039237+0000","last_undegraded":"2024-03-06T07:25:49.039237+0000","last_fullsized":"2024-03-06T07:25:49.039237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:52.685641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736602+0000","last_change":"2024-03-06T07:25:47.753494+0000","last_active":"2024-03-06T07:25:48.736602+0000","last_peered":"2024-03-06T07:25:48.736602+0000","last_clean":"2024-03-06T07:25:48.736602+0000","last_became_active":"2024-03-06T07:25:47.753128+0000","last_became_peered":"2024-03-06T07:25:47.753128+0000","last_unstale":"2024-03-06T07:25:48.736602+0000","last_undegraded":"2024-03-06T07:25:48.736602+0000","last_fullsized":"2024-03-06T07:25:48.736602+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:19.894041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736006+0000","last_change":"2024-03-06T07:25:46.746719+0000","last_active":"2024-03-06T07:25:48.736006+0000","last_peered":"2024-03-06T07:25:48.736006+0000","last_clean":"2024-03-06T07:25:48.736006+0000","last_became_active":"2024-03-06T07:25:46.746405+0000","last_became_peered":"2024-03-06T07:25:46.746405+0000","last_unstale":"2024-03-06T07:25:48.736006+0000","last_undegraded":"2024-03-06T07:25:48.736006+0000","last_fullsized":"2024-03-06T07:25:48.736006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:00.506076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735741+0000","last_change":"2024-03-06T07:25:47.746644+0000","last_active":"2024-03-06T07:25:48.735741+0000","last_peered":"2024-03-06T07:25:48.735741+0000","last_clean":"2024-03-06T07:25:48.735741+0000","last_became_active":"2024-03-06T07:25:47.746338+0000","last_became_peered":"2024-03-06T07:25:47.746338+0000","last_unstale":"2024-03-06T07:25:48.735741+0000","last_undegraded":"2024-03-06T07:25:48.735741+0000","last_fullsized":"2024-03-06T07:25:48.735741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:49.722061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735738+0000","last_change":"2024-03-06T07:25:46.742152+0000","last_active":"2024-03-06T07:25:48.735738+0000","last_peered":"2024-03-06T07:25:48.735738+0000","last_clean":"2024-03-06T07:25:48.735738+0000","last_became_active":"2024-03-06T07:25:46.741827+0000","last_became_peered":"2024-03-06T07:25:46.741827+0000","last_unstale":"2024-03-06T07:25:48.735738+0000","last_undegraded":"2024-03-06T07:25:48.735738+0000","last_fullsized":"2024-03-06T07:25:48.735738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:53.097448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736315+0000","last_change":"2024-03-06T07:25:47.760666+0000","last_active":"2024-03-06T07:25:48.736315+0000","last_peered":"2024-03-06T07:25:48.736315+0000","last_clean":"2024-03-06T07:25:48.736315+0000","last_became_active":"2024-03-06T07:25:47.760210+0000","last_became_peered":"2024-03-06T07:25:47.760210+0000","last_unstale":"2024-03-06T07:25:48.736315+0000","last_undegraded":"2024-03-06T07:25:48.736315+0000","last_fullsized":"2024-03-06T07:25:48.736315+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:52.540026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.038963+0000","last_change":"2024-03-06T07:25:46.735528+0000","last_active":"2024-03-06T07:25:49.038963+0000","last_peered":"2024-03-06T07:25:49.038963+0000","last_clean":"2024-03-06T07:25:49.038963+0000","last_became_active":"2024-03-06T07:25:46.735242+0000","last_became_peered":"2024-03-06T07:25:46.735242+0000","last_unstale":"2024-03-06T07:25:49.038963+0000","last_undegraded":"2024-03-06T07:25:49.038963+0000","last_fullsized":"2024-03-06T07:25:49.038963+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:29.824789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737538+0000","last_change":"2024-03-06T07:25:47.746068+0000","last_active":"2024-03-06T07:25:48.737538+0000","last_peered":"2024-03-06T07:25:48.737538+0000","last_clean":"2024-03-06T07:25:48.737538+0000","last_became_active":"2024-03-06T07:25:47.745758+0000","last_became_peered":"2024-03-06T07:25:47.745758+0000","last_unstale":"2024-03-06T07:25:48.737538+0000","last_undegraded":"2024-03-06T07:25:48.737538+0000","last_fullsized":"2024-03-06T07:25:48.737538+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:10.096097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759998+0000","last_change":"2024-03-06T07:25:46.746740+0000","last_active":"2024-03-06T07:25:48.759998+0000","last_peered":"2024-03-06T07:25:48.759998+0000","last_clean":"2024-03-06T07:25:48.759998+0000","last_became_active":"2024-03-06T07:25:46.746272+0000","last_became_peered":"2024-03-06T07:25:46.746272+0000","last_unstale":"2024-03-06T07:25:48.759998+0000","last_undegraded":"2024-03-06T07:25:48.759998+0000","last_fullsized":"2024-03-06T07:25:48.759998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:04.573822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039946+0000","last_change":"2024-03-06T07:25:47.754144+0000","last_active":"2024-03-06T07:25:49.039946+0000","last_peered":"2024-03-06T07:25:49.039946+0000","last_clean":"2024-03-06T07:25:49.039946+0000","last_became_active":"2024-03-06T07:25:47.753738+0000","last_became_peered":"2024-03-06T07:25:47.753738+0000","last_unstale":"2024-03-06T07:25:49.039946+0000","last_undegraded":"2024-03-06T07:25:49.039946+0000","last_fullsized":"2024-03-06T07:25:49.039946+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:53.690196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760064+0000","last_change":"2024-03-06T07:25:46.746681+0000","last_active":"2024-03-06T07:25:48.760064+0000","last_peered":"2024-03-06T07:25:48.760064+0000","last_clean":"2024-03-06T07:25:48.760064+0000","last_became_active":"2024-03-06T07:25:46.746393+0000","last_became_peered":"2024-03-06T07:25:46.746393+0000","last_unstale":"2024-03-06T07:25:48.760064+0000","last_undegraded":"2024-03-06T07:25:48.760064+0000","last_fullsized":"2024-03-06T07:25:48.760064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:01.102402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.729143+0000","last_change":"2024-03-06T07:25:45.729143+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.729143+0000","last_undegraded":"2024-03-06T07:25:45.729143+0000","last_fullsized":"2024-03-06T07:25:45.729143+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737109+0000","last_change":"2024-03-06T07:25:47.759956+0000","last_active":"2024-03-06T07:25:48.737109+0000","last_peered":"2024-03-06T07:25:48.737109+0000","last_clean":"2024-03-06T07:25:48.737109+0000","last_became_active":"2024-03-06T07:25:47.759705+0000","last_became_peered":"2024-03-06T07:25:47.759705+0000","last_unstale":"2024-03-06T07:25:48.737109+0000","last_undegraded":"2024-03-06T07:25:48.737109+0000","last_fullsized":"2024-03-06T07:25:48.737109+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:34.507002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728826+0000","last_change":"2024-03-06T07:25:45.728826+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728826+0000","last_undegraded":"2024-03-06T07:25:45.728826+0000","last_fullsized":"2024-03-06T07:25:45.728826+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737427+0000","last_change":"2024-03-06T07:25:47.738425+0000","last_active":"2024-03-06T07:25:48.737427+0000","last_peered":"2024-03-06T07:25:48.737427+0000","last_clean":"2024-03-06T07:25:48.737427+0000","last_became_active":"2024-03-06T07:25:47.738221+0000","last_became_peered":"2024-03-06T07:25:47.738221+0000","last_unstale":"2024-03-06T07:25:48.737427+0000","last_undegraded":"2024-03-06T07:25:48.737427+0000","last_fullsized":"2024-03-06T07:25:48.737427+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.952554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761806+0000","last_change":"2024-03-06T07:25:46.742402+0000","last_active":"2024-03-06T07:25:48.761806+0000","last_peered":"2024-03-06T07:25:48.761806+0000","last_clean":"2024-03-06T07:25:48.761806+0000","last_became_active":"2024-03-06T07:25:46.742144+0000","last_became_peered":"2024-03-06T07:25:46.742144+0000","last_unstale":"2024-03-06T07:25:48.761806+0000","last_undegraded":"2024-03-06T07:25:48.761806+0000","last_fullsized":"2024-03-06T07:25:48.761806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:12.845799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761869+0000","last_change":"2024-03-06T07:25:47.758008+0000","last_active":"2024-03-06T07:25:48.761869+0000","last_peered":"2024-03-06T07:25:48.761869+0000","last_clean":"2024-03-06T07:25:48.761869+0000","last_became_active":"2024-03-06T07:25:47.754667+0000","last_became_peered":"2024-03-06T07:25:47.754667+0000","last_unstale":"2024-03-06T07:25:48.761869+0000","last_undegraded":"2024-03-06T07:25:48.761869+0000","last_fullsized":"2024-03-06T07:25:48.761869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:34.507002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736719+0000","last_change":"2024-03-06T07:25:46.741357+0000","last_active":"2024-03-06T07:25:48.736719+0000","last_peered":"2024-03-06T07:25:48.736719+0000","last_clean":"2024-03-06T07:25:48.736719+0000","last_became_active":"2024-03-06T07:25:46.741039+0000","last_became_peered":"2024-03-06T07:25:46.741039+0000","last_unstale":"2024-03-06T07:25:48.736719+0000","last_undegraded":"2024-03-06T07:25:48.736719+0000","last_fullsized":"2024-03-06T07:25:48.736719+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:42.364991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039438+0000","last_change":"2024-03-06T07:25:47.755852+0000","last_active":"2024-03-06T07:25:49.039438+0000","last_peered":"2024-03-06T07:25:49.039438+0000","last_clean":"2024-03-06T07:25:49.039438+0000","last_became_active":"2024-03-06T07:25:47.755099+0000","last_became_peered":"2024-03-06T07:25:47.755099+0000","last_unstale":"2024-03-06T07:25:49.039438+0000","last_undegraded":"2024-03-06T07:25:49.039438+0000","last_fullsized":"2024-03-06T07:25:49.039438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:09.013439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-06T07:25:45.728506+0000","last_change":"2024-03-06T07:25:45.728506+0000","last_active":"2024-03-06T07:25:45.714253+0000","last_peered":"2024-03-06T07:25:45.714253+0000","last_clean":"2024-03-06T07:25:45.714253+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:45.728506+0000","last_undegraded":"2024-03-06T07:25:45.728506+0000","last_fullsized":"2024-03-06T07:25:45.728506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:25:46.718808+0000","last_change":"2024-03-06T07:25:46.718808+0000","last_active":"2024-03-06T07:25:46.718808+0000","last_peered":"2024-03-06T07:25:46.718808+0000","last_clean":"2024-03-06T07:25:46.718808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:46.718808+0000","last_undegraded":"2024-03-06T07:25:46.718808+0000","last_fullsized":"2024-03-06T07:25:46.718808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736571+0000","last_change":"2024-03-06T07:25:46.739746+0000","last_active":"2024-03-06T07:25:48.736571+0000","last_peered":"2024-03-06T07:25:48.736571+0000","last_clean":"2024-03-06T07:25:48.736571+0000","last_became_active":"2024-03-06T07:25:46.739449+0000","last_became_peered":"2024-03-06T07:25:46.739449+0000","last_unstale":"2024-03-06T07:25:48.736571+0000","last_undegraded":"2024-03-06T07:25:48.736571+0000","last_fullsized":"2024-03-06T07:25:48.736571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:07.641157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759838+0000","last_change":"2024-03-06T07:25:47.744991+0000","last_active":"2024-03-06T07:25:48.759838+0000","last_peered":"2024-03-06T07:25:48.759838+0000","last_clean":"2024-03-06T07:25:48.759838+0000","last_became_active":"2024-03-06T07:25:47.744673+0000","last_became_peered":"2024-03-06T07:25:47.744673+0000","last_unstale":"2024-03-06T07:25:48.759838+0000","last_undegraded":"2024-03-06T07:25:48.759838+0000","last_fullsized":"2024-03-06T07:25:48.759838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:45.993623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040220+0000","last_change":"2024-03-06T07:25:46.746841+0000","last_active":"2024-03-06T07:25:49.040220+0000","last_peered":"2024-03-06T07:25:49.040220+0000","last_clean":"2024-03-06T07:25:49.040220+0000","last_became_active":"2024-03-06T07:25:46.746592+0000","last_became_peered":"2024-03-06T07:25:46.746592+0000","last_unstale":"2024-03-06T07:25:49.040220+0000","last_undegraded":"2024-03-06T07:25:49.040220+0000","last_fullsized":"2024-03-06T07:25:49.040220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:12.845799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":281,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039662+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:25:49.039662+0000","last_peered":"2024-03-06T07:25:49.039662+0000","last_clean":"2024-03-06T07:25:49.039662+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:25:49.039662+0000","last_undegraded":"2024-03-06T07:25:49.039662+0000","last_fullsized":"2024-03-06T07:25:49.039662+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":106,"acting":106,"num_store_stats":7},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":1716,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1593344,"data_stored":1012776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":402988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1011712,"data_stored":423972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":422338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1007616,"data_stored":423950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":422920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1007616,"data_stored":422532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1593344,"data_stored":1012740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:55.265 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.366+0000 7f1fa52a4640 1 Processor -- start 2024-03-06T07:25:55.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.366+0000 7f1fa52a4640 1 -- start start 2024-03-06T07:25:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1fa52a4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1fa015e090 0x7f1fa00d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1fa52a4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1fa00d0650 0x7f1fa00d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1fa52a4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa015e470 con 0x7f1fa00d2920 2024-03-06T07:25:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1fa52a4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa00d0fe0 con 0x7f1fa015e090 2024-03-06T07:25:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1fa52a4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa00d18f0 con 0x7f1fa00d0650 2024-03-06T07:25:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1f9effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1fa015e090 0x7f1fa00d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1f9e7fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1fa00d2920 0x7f1fa00b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60580/0 (socket says 172.21.15.59:60580) 2024-03-06T07:25:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1f9e7fc640 1 -- 172.21.15.59:0/2830651558 learned_addr learned my addr 172.21.15.59:0/2830651558 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1f9effd640 1 -- 172.21.15.59:0/2830651558 >> v2:172.21.15.164:3300/0 conn(0x7f1fa00d0650 msgr2=0x7f1fa00d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1f9effd640 1 --2- 172.21.15.59:0/2830651558 >> v2:172.21.15.164:3300/0 conn(0x7f1fa00d0650 0x7f1fa00d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1f9effd640 1 -- 172.21.15.59:0/2830651558 >> v1:172.21.15.59:6789/0 conn(0x7f1fa00d2920 legacy=0x7f1fa00b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1f9dffb640 1 -- 172.21.15.59:0/2830651558 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338061902 0 0) 0x7f1fa015e470 con 0x7f1fa00d2920 2024-03-06T07:25:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.367+0000 7f1f9effd640 1 -- 172.21.15.59:0/2830651558 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa00b3cf0 con 0x7f1fa015e090 2024-03-06T07:25:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.368+0000 7f1f9effd640 1 --2- 172.21.15.59:0/2830651558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fa015e090 0x7f1fa00d47a0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f1fa008c550 tx=0x7f1f9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86f7ea11e0a6c09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.368+0000 7f1f9dffb640 1 -- 172.21.15.59:0/2830651558 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f90035020 con 0x7f1fa015e090 2024-03-06T07:25:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.368+0000 7f1f9dffb640 1 -- 172.21.15.59:0/2830651558 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f90031830 con 0x7f1fa015e090 2024-03-06T07:25:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.368+0000 7f1f9dffb640 1 -- 172.21.15.59:0/2830651558 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f90031c00 con 0x7f1fa015e090 2024-03-06T07:25:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.369+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2830651558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fa015e090 msgr2=0x7f1fa00d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.369+0000 7f1fa52a4640 1 --2- 172.21.15.59:0/2830651558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fa015e090 0x7f1fa00d47a0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f1fa008c550 tx=0x7f1f9002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.369+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2830651558 shutdown_connections 2024-03-06T07:25:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.369+0000 7f1fa52a4640 1 --2- 172.21.15.59:0/2830651558 >> v2:172.21.15.164:3300/0 conn(0x7f1fa00d0650 0x7f1fa00d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.369+0000 7f1fa52a4640 1 --2- 172.21.15.59:0/2830651558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fa015e090 0x7f1fa00d47a0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.369+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2830651558 >> 172.21.15.59:0/2830651558 conn(0x7f1fa00d3c80 msgr2=0x7f1fa00d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.369+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2830651558 shutdown_connections 2024-03-06T07:25:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.369+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2830651558 wait complete. 2024-03-06T07:25:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.370+0000 7f1fa52a4640 1 Processor -- start 2024-03-06T07:25:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.370+0000 7f1fa52a4640 1 -- start start 2024-03-06T07:25:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1fa52a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fa00d0650 0x7f1fa0161100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1fa52a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fa00d2920 0x7f1fa00cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1fa52a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fa015e090 0x7f1fa00b4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1fa52a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa00cee30 con 0x7f1fa015e090 2024-03-06T07:25:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1fa52a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa00ceb50 con 0x7f1fa00d2920 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1fa52a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa00b2df0 con 0x7f1fa00d0650 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1f9f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fa015e090 0x7f1fa00b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1f9f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fa015e090 0x7f1fa00b4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59638/0 (socket says 172.21.15.59:59638) 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1f9f7fe640 1 -- 172.21.15.59:0/2132992742 learned_addr learned my addr 172.21.15.59:0/2132992742 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1f9effd640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fa00d0650 0x7f1fa0161100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1f9e7fc640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fa00d2920 0x7f1fa00cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1f9f7fe640 1 -- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fa00d0650 msgr2=0x7f1fa0161100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1f9f7fe640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fa00d0650 0x7f1fa0161100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1f9f7fe640 1 -- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fa00d2920 msgr2=0x7f1fa00cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1f9f7fe640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fa00d2920 0x7f1fa00cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.371+0000 7f1f9f7fe640 1 -- 172.21.15.59:0/2132992742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa00cb760 con 0x7f1fa015e090 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.372+0000 7f1f9f7fe640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fa015e090 0x7f1fa00b4500 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f1f9800b970 tx=0x7f1f9800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.372+0000 7f1f877fe640 1 -- 172.21.15.59:0/2132992742 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f98019070 con 0x7f1fa015e090 2024-03-06T07:25:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.372+0000 7f1f877fe640 1 -- 172.21.15.59:0/2132992742 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9800cb60 con 0x7f1fa015e090 2024-03-06T07:25:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.372+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2132992742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa00cba40 con 0x7f1fa015e090 2024-03-06T07:25:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.372+0000 7f1f877fe640 1 -- 172.21.15.59:0/2132992742 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f98014440 con 0x7f1fa015e090 2024-03-06T07:25:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.372+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2132992742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa00cc000 con 0x7f1fa015e090 2024-03-06T07:25:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.373+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2132992742 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa004f330 con 0x7f1fa015e090 2024-03-06T07:25:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.373+0000 7f1f877fe640 1 -- 172.21.15.59:0/2132992742 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f9801c070 con 0x7f1fa015e090 2024-03-06T07:25:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.374+0000 7f1f877fe640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f8c06b4d0 0x7f1f8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.374+0000 7f1f877fe640 1 -- 172.21.15.59:0/2132992742 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f98089f20 con 0x7f1fa015e090 2024-03-06T07:25:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.374+0000 7f1f9effd640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f8c06b4d0 0x7f1f8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.375+0000 7f1f9effd640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f8c06b4d0 0x7f1f8c06d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1f90036000 tx=0x7f1f90008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.378+0000 7f1f877fe640 1 -- 172.21.15.59:0/2132992742 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f98058da0 con 0x7f1fa015e090 2024-03-06T07:25:55.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.591+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2132992742 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1fa00d1230 con 0x7f1f8c06b4d0 2024-03-06T07:25:55.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.598+0000 7f1f877fe640 1 -- 172.21.15.59:0/2132992742 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252078 (secure 0 0 0) 0x7f1fa00d1230 con 0x7f1f8c06b4d0 2024-03-06T07:25:55.602 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:55.603 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:25:55.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.600+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f8c06b4d0 msgr2=0x7f1f8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.600+0000 7f1fa52a4640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f8c06b4d0 0x7f1f8c06d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1f90036000 tx=0x7f1f90008030 comp rx=0 tx=0).stop 2024-03-06T07:25:55.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.600+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2132992742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fa015e090 msgr2=0x7f1fa00b4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.600+0000 7f1fa52a4640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fa015e090 0x7f1fa00b4500 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f1f9800b970 tx=0x7f1f9800be30 comp rx=0 tx=0).stop 2024-03-06T07:25:55.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.601+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2132992742 shutdown_connections 2024-03-06T07:25:55.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.601+0000 7f1fa52a4640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1f8c06b4d0 0x7f1f8c06d990 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.601+0000 7f1fa52a4640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fa015e090 0x7f1fa00b4500 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.601+0000 7f1fa52a4640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fa00d2920 0x7f1fa00cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.601+0000 7f1fa52a4640 1 --2- 172.21.15.59:0/2132992742 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fa00d0650 0x7f1fa0161100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.601+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2132992742 >> 172.21.15.59:0/2132992742 conn(0x7f1fa00d3c80 msgr2=0x7f1fa00b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.601+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2132992742 shutdown_connections 2024-03-06T07:25:55.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.601+0000 7f1fa52a4640 1 -- 172.21.15.59:0/2132992742 wait complete. 2024-03-06T07:25:55.613 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-03-06T07:25:55.352990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":190,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258472,"kb_used_data":9000,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716248,"statfs":{"total":773094113280,"available":772829437952,"internally_reserved":0,"allocated":9216000,"data_stored":4564260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001962"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040421+0000","last_change":"2024-03-06T07:25:47.755539+0000","last_active":"2024-03-06T07:25:49.040421+0000","last_peered":"2024-03-06T07:25:49.040421+0000","last_clean":"2024-03-06T07:25:49.040421+0000","last_became_active":"2024-03-06T07:25:47.755265+0000","last_became_peered":"2024-03-06T07:25:47.755265+0000","last_unstale":"2024-03-06T07:25:49.040421+0000","last_undegraded":"2024-03-06T07:25:49.040421+0000","last_fullsized":"2024-03-06T07:25:49.040421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:45.993623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760083+0000","last_change":"2024-03-06T07:25:47.754942+0000","last_active":"2024-03-06T07:25:48.760083+0000","last_peered":"2024-03-06T07:25:48.760083+0000","last_clean":"2024-03-06T07:25:48.760083+0000","last_became_active":"2024-03-06T07:25:47.753877+0000","last_became_peered":"2024-03-06T07:25:47.753877+0000","last_unstale":"2024-03-06T07:25:48.760083+0000","last_undegraded":"2024-03-06T07:25:48.760083+0000","last_fullsized":"2024-03-06T07:25:48.760083+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:52.540026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040115+0000","last_change":"2024-03-06T07:25:47.750010+0000","last_active":"2024-03-06T07:25:49.040115+0000","last_peered":"2024-03-06T07:25:49.040115+0000","last_clean":"2024-03-06T07:25:49.040115+0000","last_became_active":"2024-03-06T07:25:47.749735+0000","last_became_peered":"2024-03-06T07:25:47.749735+0000","last_unstale":"2024-03-06T07:25:49.040115+0000","last_undegraded":"2024-03-06T07:25:49.040115+0000","last_fullsized":"2024-03-06T07:25:49.040115+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:41.897733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736087+0000","last_change":"2024-03-06T07:25:47.750010+0000","last_active":"2024-03-06T07:25:48.736087+0000","last_peered":"2024-03-06T07:25:48.736087+0000","last_clean":"2024-03-06T07:25:48.736087+0000","last_became_active":"2024-03-06T07:25:47.749708+0000","last_became_peered":"2024-03-06T07:25:47.749708+0000","last_unstale":"2024-03-06T07:25:48.736087+0000","last_undegraded":"2024-03-06T07:25:48.736087+0000","last_fullsized":"2024-03-06T07:25:48.736087+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:49.954151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737396+0000","last_change":"2024-03-06T07:25:47.747729+0000","last_active":"2024-03-06T07:25:48.737396+0000","last_peered":"2024-03-06T07:25:48.737396+0000","last_clean":"2024-03-06T07:25:48.737396+0000","last_became_active":"2024-03-06T07:25:47.747442+0000","last_became_peered":"2024-03-06T07:25:47.747442+0000","last_unstale":"2024-03-06T07:25:48.737396+0000","last_undegraded":"2024-03-06T07:25:48.737396+0000","last_fullsized":"2024-03-06T07:25:48.737396+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:53.366551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737419+0000","last_change":"2024-03-06T07:25:47.751731+0000","last_active":"2024-03-06T07:25:48.737419+0000","last_peered":"2024-03-06T07:25:48.737419+0000","last_clean":"2024-03-06T07:25:48.737419+0000","last_became_active":"2024-03-06T07:25:47.751445+0000","last_became_peered":"2024-03-06T07:25:47.751445+0000","last_unstale":"2024-03-06T07:25:48.737419+0000","last_undegraded":"2024-03-06T07:25:48.737419+0000","last_fullsized":"2024-03-06T07:25:48.737419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:08.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735762+0000","last_change":"2024-03-06T07:25:47.747479+0000","last_active":"2024-03-06T07:25:48.735762+0000","last_peered":"2024-03-06T07:25:48.735762+0000","last_clean":"2024-03-06T07:25:48.735762+0000","last_became_active":"2024-03-06T07:25:47.747186+0000","last_became_peered":"2024-03-06T07:25:47.747186+0000","last_unstale":"2024-03-06T07:25:48.735762+0000","last_undegraded":"2024-03-06T07:25:48.735762+0000","last_fullsized":"2024-03-06T07:25:48.735762+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:48:38.441834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736301+0000","last_change":"2024-03-06T07:25:47.742812+0000","last_active":"2024-03-06T07:25:48.736301+0000","last_peered":"2024-03-06T07:25:48.736301+0000","last_clean":"2024-03-06T07:25:48.736301+0000","last_became_active":"2024-03-06T07:25:47.739811+0000","last_became_peered":"2024-03-06T07:25:47.739811+0000","last_unstale":"2024-03-06T07:25:48.736301+0000","last_undegraded":"2024-03-06T07:25:48.736301+0000","last_fullsized":"2024-03-06T07:25:48.736301+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:17.837597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736920+0000","last_change":"2024-03-06T07:25:47.760422+0000","last_active":"2024-03-06T07:25:48.736920+0000","last_peered":"2024-03-06T07:25:48.736920+0000","last_clean":"2024-03-06T07:25:48.736920+0000","last_became_active":"2024-03-06T07:25:47.760129+0000","last_became_peered":"2024-03-06T07:25:47.760129+0000","last_unstale":"2024-03-06T07:25:48.736920+0000","last_undegraded":"2024-03-06T07:25:48.736920+0000","last_fullsized":"2024-03-06T07:25:48.736920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:52.495587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040103+0000","last_change":"2024-03-06T07:25:47.757129+0000","last_active":"2024-03-06T07:25:49.040103+0000","last_peered":"2024-03-06T07:25:49.040103+0000","last_clean":"2024-03-06T07:25:49.040103+0000","last_became_active":"2024-03-06T07:25:47.756823+0000","last_became_peered":"2024-03-06T07:25:47.756823+0000","last_unstale":"2024-03-06T07:25:49.040103+0000","last_undegraded":"2024-03-06T07:25:49.040103+0000","last_fullsized":"2024-03-06T07:25:49.040103+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:10.615032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760008+0000","last_change":"2024-03-06T07:25:47.745385+0000","last_active":"2024-03-06T07:25:48.760008+0000","last_peered":"2024-03-06T07:25:48.760008+0000","last_clean":"2024-03-06T07:25:48.760008+0000","last_became_active":"2024-03-06T07:25:47.745117+0000","last_became_peered":"2024-03-06T07:25:47.745117+0000","last_unstale":"2024-03-06T07:25:48.760008+0000","last_undegraded":"2024-03-06T07:25:48.760008+0000","last_fullsized":"2024-03-06T07:25:48.760008+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:40.310991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040117+0000","last_change":"2024-03-06T07:25:47.750804+0000","last_active":"2024-03-06T07:25:49.040117+0000","last_peered":"2024-03-06T07:25:49.040117+0000","last_clean":"2024-03-06T07:25:49.040117+0000","last_became_active":"2024-03-06T07:25:47.750490+0000","last_became_peered":"2024-03-06T07:25:47.750490+0000","last_unstale":"2024-03-06T07:25:49.040117+0000","last_undegraded":"2024-03-06T07:25:49.040117+0000","last_fullsized":"2024-03-06T07:25:49.040117+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:27.486710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039620+0000","last_change":"2024-03-06T07:25:47.749258+0000","last_active":"2024-03-06T07:25:49.039620+0000","last_peered":"2024-03-06T07:25:49.039620+0000","last_clean":"2024-03-06T07:25:49.039620+0000","last_became_active":"2024-03-06T07:25:47.748993+0000","last_became_peered":"2024-03-06T07:25:47.748993+0000","last_unstale":"2024-03-06T07:25:49.039620+0000","last_undegraded":"2024-03-06T07:25:49.039620+0000","last_fullsized":"2024-03-06T07:25:49.039620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:53.366551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735907+0000","last_change":"2024-03-06T07:25:47.750890+0000","last_active":"2024-03-06T07:25:48.735907+0000","last_peered":"2024-03-06T07:25:48.735907+0000","last_clean":"2024-03-06T07:25:48.735907+0000","last_became_active":"2024-03-06T07:25:47.750579+0000","last_became_peered":"2024-03-06T07:25:47.750579+0000","last_unstale":"2024-03-06T07:25:48.735907+0000","last_undegraded":"2024-03-06T07:25:48.735907+0000","last_fullsized":"2024-03-06T07:25:48.735907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:12.102084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039116+0000","last_change":"2024-03-06T07:25:47.749794+0000","last_active":"2024-03-06T07:25:49.039116+0000","last_peered":"2024-03-06T07:25:49.039116+0000","last_clean":"2024-03-06T07:25:49.039116+0000","last_became_active":"2024-03-06T07:25:47.749189+0000","last_became_peered":"2024-03-06T07:25:47.749189+0000","last_unstale":"2024-03-06T07:25:49.039116+0000","last_undegraded":"2024-03-06T07:25:49.039116+0000","last_fullsized":"2024-03-06T07:25:49.039116+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:49.406968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736570+0000","last_change":"2024-03-06T07:25:47.753561+0000","last_active":"2024-03-06T07:25:48.736570+0000","last_peered":"2024-03-06T07:25:48.736570+0000","last_clean":"2024-03-06T07:25:48.736570+0000","last_became_active":"2024-03-06T07:25:47.753272+0000","last_became_peered":"2024-03-06T07:25:47.753272+0000","last_unstale":"2024-03-06T07:25:48.736570+0000","last_undegraded":"2024-03-06T07:25:48.736570+0000","last_fullsized":"2024-03-06T07:25:48.736570+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:23.030627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737208+0000","last_change":"2024-03-06T07:25:47.752307+0000","last_active":"2024-03-06T07:25:48.737208+0000","last_peered":"2024-03-06T07:25:48.737208+0000","last_clean":"2024-03-06T07:25:48.737208+0000","last_became_active":"2024-03-06T07:25:47.752046+0000","last_became_peered":"2024-03-06T07:25:47.752046+0000","last_unstale":"2024-03-06T07:25:48.737208+0000","last_undegraded":"2024-03-06T07:25:48.737208+0000","last_fullsized":"2024-03-06T07:25:48.737208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.822094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736117+0000","last_change":"2024-03-06T07:25:47.750762+0000","last_active":"2024-03-06T07:25:48.736117+0000","last_peered":"2024-03-06T07:25:48.736117+0000","last_clean":"2024-03-06T07:25:48.736117+0000","last_became_active":"2024-03-06T07:25:47.750456+0000","last_became_peered":"2024-03-06T07:25:47.750456+0000","last_unstale":"2024-03-06T07:25:48.736117+0000","last_undegraded":"2024-03-06T07:25:48.736117+0000","last_fullsized":"2024-03-06T07:25:48.736117+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:32:28.272446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.762159+0000","last_change":"2024-03-06T07:25:47.754357+0000","last_active":"2024-03-06T07:25:48.762159+0000","last_peered":"2024-03-06T07:25:48.762159+0000","last_clean":"2024-03-06T07:25:48.762159+0000","last_became_active":"2024-03-06T07:25:47.754043+0000","last_became_peered":"2024-03-06T07:25:47.754043+0000","last_unstale":"2024-03-06T07:25:48.762159+0000","last_undegraded":"2024-03-06T07:25:48.762159+0000","last_fullsized":"2024-03-06T07:25:48.762159+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.822094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736225+0000","last_change":"2024-03-06T07:25:47.756728+0000","last_active":"2024-03-06T07:25:48.736225+0000","last_peered":"2024-03-06T07:25:48.736225+0000","last_clean":"2024-03-06T07:25:48.736225+0000","last_became_active":"2024-03-06T07:25:47.756364+0000","last_became_peered":"2024-03-06T07:25:47.756364+0000","last_unstale":"2024-03-06T07:25:48.736225+0000","last_undegraded":"2024-03-06T07:25:48.736225+0000","last_fullsized":"2024-03-06T07:25:48.736225+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:08:35.409269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735980+0000","last_change":"2024-03-06T07:25:47.760935+0000","last_active":"2024-03-06T07:25:48.735980+0000","last_peered":"2024-03-06T07:25:48.735980+0000","last_clean":"2024-03-06T07:25:48.735980+0000","last_became_active":"2024-03-06T07:25:47.760648+0000","last_became_peered":"2024-03-06T07:25:47.760648+0000","last_unstale":"2024-03-06T07:25:48.735980+0000","last_undegraded":"2024-03-06T07:25:48.735980+0000","last_fullsized":"2024-03-06T07:25:48.735980+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:09.073261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759561+0000","last_change":"2024-03-06T07:25:47.760858+0000","last_active":"2024-03-06T07:25:48.759561+0000","last_peered":"2024-03-06T07:25:48.759561+0000","last_clean":"2024-03-06T07:25:48.759561+0000","last_became_active":"2024-03-06T07:25:47.760570+0000","last_became_peered":"2024-03-06T07:25:47.760570+0000","last_unstale":"2024-03-06T07:25:48.759561+0000","last_undegraded":"2024-03-06T07:25:48.759561+0000","last_fullsized":"2024-03-06T07:25:48.759561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:27.486710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039648+0000","last_change":"2024-03-06T07:25:47.761416+0000","last_active":"2024-03-06T07:25:49.039648+0000","last_peered":"2024-03-06T07:25:49.039648+0000","last_clean":"2024-03-06T07:25:49.039648+0000","last_became_active":"2024-03-06T07:25:47.760825+0000","last_became_peered":"2024-03-06T07:25:47.760825+0000","last_unstale":"2024-03-06T07:25:49.039648+0000","last_undegraded":"2024-03-06T07:25:49.039648+0000","last_fullsized":"2024-03-06T07:25:49.039648+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:46.372799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735731+0000","last_change":"2024-03-06T07:25:47.761368+0000","last_active":"2024-03-06T07:25:48.735731+0000","last_peered":"2024-03-06T07:25:48.735731+0000","last_clean":"2024-03-06T07:25:48.735731+0000","last_became_active":"2024-03-06T07:25:47.761104+0000","last_became_peered":"2024-03-06T07:25:47.761104+0000","last_unstale":"2024-03-06T07:25:48.735731+0000","last_undegraded":"2024-03-06T07:25:48.735731+0000","last_fullsized":"2024-03-06T07:25:48.735731+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:58.946010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760082+0000","last_change":"2024-03-06T07:25:47.756504+0000","last_active":"2024-03-06T07:25:48.760082+0000","last_peered":"2024-03-06T07:25:48.760082+0000","last_clean":"2024-03-06T07:25:48.760082+0000","last_became_active":"2024-03-06T07:25:47.756231+0000","last_became_peered":"2024-03-06T07:25:47.756231+0000","last_unstale":"2024-03-06T07:25:48.760082+0000","last_undegraded":"2024-03-06T07:25:48.760082+0000","last_fullsized":"2024-03-06T07:25:48.760082+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:17.837597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039678+0000","last_change":"2024-03-06T07:25:47.755618+0000","last_active":"2024-03-06T07:25:49.039678+0000","last_peered":"2024-03-06T07:25:49.039678+0000","last_clean":"2024-03-06T07:25:49.039678+0000","last_became_active":"2024-03-06T07:25:47.755092+0000","last_became_peered":"2024-03-06T07:25:47.755092+0000","last_unstale":"2024-03-06T07:25:49.039678+0000","last_undegraded":"2024-03-06T07:25:49.039678+0000","last_fullsized":"2024-03-06T07:25:49.039678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:24.401568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736402+0000","last_change":"2024-03-06T07:25:47.751435+0000","last_active":"2024-03-06T07:25:48.736402+0000","last_peered":"2024-03-06T07:25:48.736402+0000","last_clean":"2024-03-06T07:25:48.736402+0000","last_became_active":"2024-03-06T07:25:47.751146+0000","last_became_peered":"2024-03-06T07:25:47.751146+0000","last_unstale":"2024-03-06T07:25:48.736402+0000","last_undegraded":"2024-03-06T07:25:48.736402+0000","last_fullsized":"2024-03-06T07:25:48.736402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:52.495587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040179+0000","last_change":"2024-03-06T07:25:47.755562+0000","last_active":"2024-03-06T07:25:49.040179+0000","last_peered":"2024-03-06T07:25:49.040179+0000","last_clean":"2024-03-06T07:25:49.040179+0000","last_became_active":"2024-03-06T07:25:47.755278+0000","last_became_peered":"2024-03-06T07:25:47.755278+0000","last_unstale":"2024-03-06T07:25:49.040179+0000","last_undegraded":"2024-03-06T07:25:49.040179+0000","last_fullsized":"2024-03-06T07:25:49.040179+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:57.303162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759695+0000","last_change":"2024-03-06T07:25:47.739643+0000","last_active":"2024-03-06T07:25:48.759695+0000","last_peered":"2024-03-06T07:25:48.759695+0000","last_clean":"2024-03-06T07:25:48.759695+0000","last_became_active":"2024-03-06T07:25:47.739353+0000","last_became_peered":"2024-03-06T07:25:47.739353+0000","last_unstale":"2024-03-06T07:25:48.759695+0000","last_undegraded":"2024-03-06T07:25:48.759695+0000","last_fullsized":"2024-03-06T07:25:48.759695+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:09.040328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759629+0000","last_change":"2024-03-06T07:25:47.753200+0000","last_active":"2024-03-06T07:25:48.759629+0000","last_peered":"2024-03-06T07:25:48.759629+0000","last_clean":"2024-03-06T07:25:48.759629+0000","last_became_active":"2024-03-06T07:25:47.752393+0000","last_became_peered":"2024-03-06T07:25:47.752393+0000","last_unstale":"2024-03-06T07:25:48.759629+0000","last_undegraded":"2024-03-06T07:25:48.759629+0000","last_fullsized":"2024-03-06T07:25:48.759629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:57.303162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736327+0000","last_change":"2024-03-06T07:25:47.761829+0000","last_active":"2024-03-06T07:25:48.736327+0000","last_peered":"2024-03-06T07:25:48.736327+0000","last_clean":"2024-03-06T07:25:48.736327+0000","last_became_active":"2024-03-06T07:25:47.761583+0000","last_became_peered":"2024-03-06T07:25:47.761583+0000","last_unstale":"2024-03-06T07:25:48.736327+0000","last_undegraded":"2024-03-06T07:25:48.736327+0000","last_fullsized":"2024-03-06T07:25:48.736327+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:19.894041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039590+0000","last_change":"2024-03-06T07:25:47.764339+0000","last_active":"2024-03-06T07:25:49.039590+0000","last_peered":"2024-03-06T07:25:49.039590+0000","last_clean":"2024-03-06T07:25:49.039590+0000","last_became_active":"2024-03-06T07:25:47.763925+0000","last_became_peered":"2024-03-06T07:25:47.763925+0000","last_unstale":"2024-03-06T07:25:49.039590+0000","last_undegraded":"2024-03-06T07:25:49.039590+0000","last_fullsized":"2024-03-06T07:25:49.039590+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:45.231776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737151+0000","last_change":"2024-03-06T07:25:47.750725+0000","last_active":"2024-03-06T07:25:48.737151+0000","last_peered":"2024-03-06T07:25:48.737151+0000","last_clean":"2024-03-06T07:25:48.737151+0000","last_became_active":"2024-03-06T07:25:47.750427+0000","last_became_peered":"2024-03-06T07:25:47.750427+0000","last_unstale":"2024-03-06T07:25:48.737151+0000","last_undegraded":"2024-03-06T07:25:48.737151+0000","last_fullsized":"2024-03-06T07:25:48.737151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:09.013439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040475+0000","last_change":"2024-03-06T07:25:46.746340+0000","last_active":"2024-03-06T07:25:49.040475+0000","last_peered":"2024-03-06T07:25:49.040475+0000","last_clean":"2024-03-06T07:25:49.040475+0000","last_became_active":"2024-03-06T07:25:46.746033+0000","last_became_peered":"2024-03-06T07:25:46.746033+0000","last_unstale":"2024-03-06T07:25:49.040475+0000","last_undegraded":"2024-03-06T07:25:49.040475+0000","last_fullsized":"2024-03-06T07:25:49.040475+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:51:57.190607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040430+0000","last_change":"2024-03-06T07:25:47.760799+0000","last_active":"2024-03-06T07:25:49.040430+0000","last_peered":"2024-03-06T07:25:49.040430+0000","last_clean":"2024-03-06T07:25:49.040430+0000","last_became_active":"2024-03-06T07:25:47.760500+0000","last_became_peered":"2024-03-06T07:25:47.760500+0000","last_unstale":"2024-03-06T07:25:49.040430+0000","last_undegraded":"2024-03-06T07:25:49.040430+0000","last_fullsized":"2024-03-06T07:25:49.040430+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:40.310991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737231+0000","last_change":"2024-03-06T07:25:46.741560+0000","last_active":"2024-03-06T07:25:48.737231+0000","last_peered":"2024-03-06T07:25:48.737231+0000","last_clean":"2024-03-06T07:25:48.737231+0000","last_became_active":"2024-03-06T07:25:46.741225+0000","last_became_peered":"2024-03-06T07:25:46.741225+0000","last_unstale":"2024-03-06T07:25:48.737231+0000","last_undegraded":"2024-03-06T07:25:48.737231+0000","last_fullsized":"2024-03-06T07:25:48.737231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:42.221129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040351+0000","last_change":"2024-03-06T07:25:47.764179+0000","last_active":"2024-03-06T07:25:49.040351+0000","last_peered":"2024-03-06T07:25:49.040351+0000","last_clean":"2024-03-06T07:25:49.040351+0000","last_became_active":"2024-03-06T07:25:47.763857+0000","last_became_peered":"2024-03-06T07:25:47.763857+0000","last_unstale":"2024-03-06T07:25:49.040351+0000","last_undegraded":"2024-03-06T07:25:49.040351+0000","last_fullsized":"2024-03-06T07:25:49.040351+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:09.040328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.762098+0000","last_change":"2024-03-06T07:25:46.741655+0000","last_active":"2024-03-06T07:25:48.762098+0000","last_peered":"2024-03-06T07:25:48.762098+0000","last_clean":"2024-03-06T07:25:48.762098+0000","last_became_active":"2024-03-06T07:25:46.741374+0000","last_became_peered":"2024-03-06T07:25:46.741374+0000","last_unstale":"2024-03-06T07:25:48.762098+0000","last_undegraded":"2024-03-06T07:25:48.762098+0000","last_fullsized":"2024-03-06T07:25:48.762098+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:16.833042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040274+0000","last_change":"2024-03-06T07:25:47.748531+0000","last_active":"2024-03-06T07:25:49.040274+0000","last_peered":"2024-03-06T07:25:49.040274+0000","last_clean":"2024-03-06T07:25:49.040274+0000","last_became_active":"2024-03-06T07:25:47.748272+0000","last_became_peered":"2024-03-06T07:25:47.748272+0000","last_unstale":"2024-03-06T07:25:49.040274+0000","last_undegraded":"2024-03-06T07:25:49.040274+0000","last_fullsized":"2024-03-06T07:25:49.040274+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:52.756249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761801+0000","last_change":"2024-03-06T07:25:46.742202+0000","last_active":"2024-03-06T07:25:48.761801+0000","last_peered":"2024-03-06T07:25:48.761801+0000","last_clean":"2024-03-06T07:25:48.761801+0000","last_became_active":"2024-03-06T07:25:46.741325+0000","last_became_peered":"2024-03-06T07:25:46.741325+0000","last_unstale":"2024-03-06T07:25:48.761801+0000","last_undegraded":"2024-03-06T07:25:48.761801+0000","last_fullsized":"2024-03-06T07:25:48.761801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:44.989067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759670+0000","last_change":"2024-03-06T07:25:47.747094+0000","last_active":"2024-03-06T07:25:48.759670+0000","last_peered":"2024-03-06T07:25:48.759670+0000","last_clean":"2024-03-06T07:25:48.759670+0000","last_became_active":"2024-03-06T07:25:47.746796+0000","last_became_peered":"2024-03-06T07:25:47.746796+0000","last_unstale":"2024-03-06T07:25:48.759670+0000","last_undegraded":"2024-03-06T07:25:48.759670+0000","last_fullsized":"2024-03-06T07:25:48.759670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:13.850354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737511+0000","last_change":"2024-03-06T07:25:46.741705+0000","last_active":"2024-03-06T07:25:48.737511+0000","last_peered":"2024-03-06T07:25:48.737511+0000","last_clean":"2024-03-06T07:25:48.737511+0000","last_became_active":"2024-03-06T07:25:46.741411+0000","last_became_peered":"2024-03-06T07:25:46.741411+0000","last_unstale":"2024-03-06T07:25:48.737511+0000","last_undegraded":"2024-03-06T07:25:48.737511+0000","last_fullsized":"2024-03-06T07:25:48.737511+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:52.885965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737431+0000","last_change":"2024-03-06T07:25:47.753035+0000","last_active":"2024-03-06T07:25:48.737431+0000","last_peered":"2024-03-06T07:25:48.737431+0000","last_clean":"2024-03-06T07:25:48.737431+0000","last_became_active":"2024-03-06T07:25:47.752710+0000","last_became_peered":"2024-03-06T07:25:47.752710+0000","last_unstale":"2024-03-06T07:25:48.737431+0000","last_undegraded":"2024-03-06T07:25:48.737431+0000","last_fullsized":"2024-03-06T07:25:48.737431+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:24.401568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736619+0000","last_change":"2024-03-06T07:25:46.741681+0000","last_active":"2024-03-06T07:25:48.736619+0000","last_peered":"2024-03-06T07:25:48.736619+0000","last_clean":"2024-03-06T07:25:48.736619+0000","last_became_active":"2024-03-06T07:25:46.741397+0000","last_became_peered":"2024-03-06T07:25:46.741397+0000","last_unstale":"2024-03-06T07:25:48.736619+0000","last_undegraded":"2024-03-06T07:25:48.736619+0000","last_fullsized":"2024-03-06T07:25:48.736619+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:32.000019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039562+0000","last_change":"2024-03-06T07:25:47.759466+0000","last_active":"2024-03-06T07:25:49.039562+0000","last_peered":"2024-03-06T07:25:49.039562+0000","last_clean":"2024-03-06T07:25:49.039562+0000","last_became_active":"2024-03-06T07:25:47.759185+0000","last_became_peered":"2024-03-06T07:25:47.759185+0000","last_unstale":"2024-03-06T07:25:49.039562+0000","last_undegraded":"2024-03-06T07:25:49.039562+0000","last_fullsized":"2024-03-06T07:25:49.039562+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.822094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761641+0000","last_change":"2024-03-06T07:25:46.737587+0000","last_active":"2024-03-06T07:25:48.761641+0000","last_peered":"2024-03-06T07:25:48.761641+0000","last_clean":"2024-03-06T07:25:48.761641+0000","last_became_active":"2024-03-06T07:25:46.737289+0000","last_became_peered":"2024-03-06T07:25:46.737289+0000","last_unstale":"2024-03-06T07:25:48.761641+0000","last_undegraded":"2024-03-06T07:25:48.761641+0000","last_fullsized":"2024-03-06T07:25:48.761641+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:08.035773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039138+0000","last_change":"2024-03-06T07:25:47.749535+0000","last_active":"2024-03-06T07:25:49.039138+0000","last_peered":"2024-03-06T07:25:49.039138+0000","last_clean":"2024-03-06T07:25:49.039138+0000","last_became_active":"2024-03-06T07:25:47.748786+0000","last_became_peered":"2024-03-06T07:25:47.748786+0000","last_unstale":"2024-03-06T07:25:49.039138+0000","last_undegraded":"2024-03-06T07:25:49.039138+0000","last_fullsized":"2024-03-06T07:25:49.039138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:57.368763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759500+0000","last_change":"2024-03-06T07:25:46.741890+0000","last_active":"2024-03-06T07:25:48.759500+0000","last_peered":"2024-03-06T07:25:48.759500+0000","last_clean":"2024-03-06T07:25:48.759500+0000","last_became_active":"2024-03-06T07:25:46.741566+0000","last_became_peered":"2024-03-06T07:25:46.741566+0000","last_unstale":"2024-03-06T07:25:48.759500+0000","last_undegraded":"2024-03-06T07:25:48.759500+0000","last_fullsized":"2024-03-06T07:25:48.759500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:53.583814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039409+0000","last_change":"2024-03-06T07:25:47.741722+0000","last_active":"2024-03-06T07:25:49.039409+0000","last_peered":"2024-03-06T07:25:49.039409+0000","last_clean":"2024-03-06T07:25:49.039409+0000","last_became_active":"2024-03-06T07:25:47.741384+0000","last_became_peered":"2024-03-06T07:25:47.741384+0000","last_unstale":"2024-03-06T07:25:49.039409+0000","last_undegraded":"2024-03-06T07:25:49.039409+0000","last_fullsized":"2024-03-06T07:25:49.039409+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:26.696427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040541+0000","last_change":"2024-03-06T07:25:46.744840+0000","last_active":"2024-03-06T07:25:49.040541+0000","last_peered":"2024-03-06T07:25:49.040541+0000","last_clean":"2024-03-06T07:25:49.040541+0000","last_became_active":"2024-03-06T07:25:46.744575+0000","last_became_peered":"2024-03-06T07:25:46.744575+0000","last_unstale":"2024-03-06T07:25:49.040541+0000","last_undegraded":"2024-03-06T07:25:49.040541+0000","last_fullsized":"2024-03-06T07:25:49.040541+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:46.310599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737099+0000","last_change":"2024-03-06T07:25:47.752574+0000","last_active":"2024-03-06T07:25:48.737099+0000","last_peered":"2024-03-06T07:25:48.737099+0000","last_clean":"2024-03-06T07:25:48.737099+0000","last_became_active":"2024-03-06T07:25:47.752246+0000","last_became_peered":"2024-03-06T07:25:47.752246+0000","last_unstale":"2024-03-06T07:25:48.737099+0000","last_undegraded":"2024-03-06T07:25:48.737099+0000","last_fullsized":"2024-03-06T07:25:48.737099+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:23.030627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040162+0000","last_change":"2024-03-06T07:25:46.744403+0000","last_active":"2024-03-06T07:25:49.040162+0000","last_peered":"2024-03-06T07:25:49.040162+0000","last_clean":"2024-03-06T07:25:49.040162+0000","last_became_active":"2024-03-06T07:25:46.744009+0000","last_became_peered":"2024-03-06T07:25:46.744009+0000","last_unstale":"2024-03-06T07:25:49.040162+0000","last_undegraded":"2024-03-06T07:25:49.040162+0000","last_fullsized":"2024-03-06T07:25:49.040162+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:34.237325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.762062+0000","last_change":"2024-03-06T07:25:47.755416+0000","last_active":"2024-03-06T07:25:48.762062+0000","last_peered":"2024-03-06T07:25:48.762062+0000","last_clean":"2024-03-06T07:25:48.762062+0000","last_became_active":"2024-03-06T07:25:47.755154+0000","last_became_peered":"2024-03-06T07:25:47.755154+0000","last_unstale":"2024-03-06T07:25:48.762062+0000","last_undegraded":"2024-03-06T07:25:48.762062+0000","last_fullsized":"2024-03-06T07:25:48.762062+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:09.013439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039904+0000","last_change":"2024-03-06T07:25:46.742641+0000","last_active":"2024-03-06T07:25:49.039904+0000","last_peered":"2024-03-06T07:25:49.039904+0000","last_clean":"2024-03-06T07:25:49.039904+0000","last_became_active":"2024-03-06T07:25:46.742383+0000","last_became_peered":"2024-03-06T07:25:46.742383+0000","last_unstale":"2024-03-06T07:25:49.039904+0000","last_undegraded":"2024-03-06T07:25:49.039904+0000","last_fullsized":"2024-03-06T07:25:49.039904+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:20.372243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736937+0000","last_change":"2024-03-06T07:25:47.761129+0000","last_active":"2024-03-06T07:25:48.736937+0000","last_peered":"2024-03-06T07:25:48.736937+0000","last_clean":"2024-03-06T07:25:48.736937+0000","last_became_active":"2024-03-06T07:25:47.760855+0000","last_became_peered":"2024-03-06T07:25:47.760855+0000","last_unstale":"2024-03-06T07:25:48.736937+0000","last_undegraded":"2024-03-06T07:25:48.736937+0000","last_fullsized":"2024-03-06T07:25:48.736937+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:08.417093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736791+0000","last_change":"2024-03-06T07:25:46.732588+0000","last_active":"2024-03-06T07:25:48.736791+0000","last_peered":"2024-03-06T07:25:48.736791+0000","last_clean":"2024-03-06T07:25:48.736791+0000","last_became_active":"2024-03-06T07:25:46.732252+0000","last_became_peered":"2024-03-06T07:25:46.732252+0000","last_unstale":"2024-03-06T07:25:48.736791+0000","last_undegraded":"2024-03-06T07:25:48.736791+0000","last_fullsized":"2024-03-06T07:25:48.736791+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:45.368244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761800+0000","last_change":"2024-03-06T07:25:47.759623+0000","last_active":"2024-03-06T07:25:48.761800+0000","last_peered":"2024-03-06T07:25:48.761800+0000","last_clean":"2024-03-06T07:25:48.761800+0000","last_became_active":"2024-03-06T07:25:47.759263+0000","last_became_peered":"2024-03-06T07:25:47.759263+0000","last_unstale":"2024-03-06T07:25:48.761800+0000","last_undegraded":"2024-03-06T07:25:48.761800+0000","last_fullsized":"2024-03-06T07:25:48.761800+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.952554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759773+0000","last_change":"2024-03-06T07:25:46.746659+0000","last_active":"2024-03-06T07:25:48.759773+0000","last_peered":"2024-03-06T07:25:48.759773+0000","last_clean":"2024-03-06T07:25:48.759773+0000","last_became_active":"2024-03-06T07:25:46.746246+0000","last_became_peered":"2024-03-06T07:25:46.746246+0000","last_unstale":"2024-03-06T07:25:48.759773+0000","last_undegraded":"2024-03-06T07:25:48.759773+0000","last_fullsized":"2024-03-06T07:25:48.759773+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:09:32.988256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039919+0000","last_change":"2024-03-06T07:25:47.760007+0000","last_active":"2024-03-06T07:25:49.039919+0000","last_peered":"2024-03-06T07:25:49.039919+0000","last_clean":"2024-03-06T07:25:49.039919+0000","last_became_active":"2024-03-06T07:25:47.759722+0000","last_became_peered":"2024-03-06T07:25:47.759722+0000","last_unstale":"2024-03-06T07:25:49.039919+0000","last_undegraded":"2024-03-06T07:25:49.039919+0000","last_fullsized":"2024-03-06T07:25:49.039919+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:30.829345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039266+0000","last_change":"2024-03-06T07:25:46.741231+0000","last_active":"2024-03-06T07:25:49.039266+0000","last_peered":"2024-03-06T07:25:49.039266+0000","last_clean":"2024-03-06T07:25:49.039266+0000","last_became_active":"2024-03-06T07:25:46.740951+0000","last_became_peered":"2024-03-06T07:25:46.740951+0000","last_unstale":"2024-03-06T07:25:49.039266+0000","last_undegraded":"2024-03-06T07:25:49.039266+0000","last_fullsized":"2024-03-06T07:25:49.039266+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:40.893178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759533+0000","last_change":"2024-03-06T07:25:47.746838+0000","last_active":"2024-03-06T07:25:48.759533+0000","last_peered":"2024-03-06T07:25:48.759533+0000","last_clean":"2024-03-06T07:25:48.759533+0000","last_became_active":"2024-03-06T07:25:47.746563+0000","last_became_peered":"2024-03-06T07:25:47.746563+0000","last_unstale":"2024-03-06T07:25:48.759533+0000","last_undegraded":"2024-03-06T07:25:48.759533+0000","last_fullsized":"2024-03-06T07:25:48.759533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:30:10.615032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736434+0000","last_change":"2024-03-06T07:25:46.741981+0000","last_active":"2024-03-06T07:25:48.736434+0000","last_peered":"2024-03-06T07:25:48.736434+0000","last_clean":"2024-03-06T07:25:48.736434+0000","last_became_active":"2024-03-06T07:25:46.741709+0000","last_became_peered":"2024-03-06T07:25:46.741709+0000","last_unstale":"2024-03-06T07:25:48.736434+0000","last_undegraded":"2024-03-06T07:25:48.736434+0000","last_fullsized":"2024-03-06T07:25:48.736434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:33.944174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759470+0000","last_change":"2024-03-06T07:25:47.755828+0000","last_active":"2024-03-06T07:25:48.759470+0000","last_peered":"2024-03-06T07:25:48.759470+0000","last_clean":"2024-03-06T07:25:48.759470+0000","last_became_active":"2024-03-06T07:25:47.755559+0000","last_became_peered":"2024-03-06T07:25:47.755559+0000","last_unstale":"2024-03-06T07:25:48.759470+0000","last_undegraded":"2024-03-06T07:25:48.759470+0000","last_fullsized":"2024-03-06T07:25:48.759470+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:34.948730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761798+0000","last_change":"2024-03-06T07:25:46.743744+0000","last_active":"2024-03-06T07:25:48.761798+0000","last_peered":"2024-03-06T07:25:48.761798+0000","last_clean":"2024-03-06T07:25:48.761798+0000","last_became_active":"2024-03-06T07:25:46.743447+0000","last_became_peered":"2024-03-06T07:25:46.743447+0000","last_unstale":"2024-03-06T07:25:48.761798+0000","last_undegraded":"2024-03-06T07:25:48.761798+0000","last_fullsized":"2024-03-06T07:25:48.761798+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:56.298607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039836+0000","last_change":"2024-03-06T07:25:47.747720+0000","last_active":"2024-03-06T07:25:49.039836+0000","last_peered":"2024-03-06T07:25:49.039836+0000","last_clean":"2024-03-06T07:25:49.039836+0000","last_became_active":"2024-03-06T07:25:47.747412+0000","last_became_peered":"2024-03-06T07:25:47.747412+0000","last_unstale":"2024-03-06T07:25:49.039836+0000","last_undegraded":"2024-03-06T07:25:49.039836+0000","last_fullsized":"2024-03-06T07:25:49.039836+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:10.096097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040604+0000","last_change":"2024-03-06T07:25:46.738811+0000","last_active":"2024-03-06T07:25:49.040604+0000","last_peered":"2024-03-06T07:25:49.040604+0000","last_clean":"2024-03-06T07:25:49.040604+0000","last_became_active":"2024-03-06T07:25:46.738504+0000","last_became_peered":"2024-03-06T07:25:46.738504+0000","last_unstale":"2024-03-06T07:25:49.040604+0000","last_undegraded":"2024-03-06T07:25:49.040604+0000","last_fullsized":"2024-03-06T07:25:49.040604+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:53.583814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040533+0000","last_change":"2024-03-06T07:25:47.752862+0000","last_active":"2024-03-06T07:25:49.040533+0000","last_peered":"2024-03-06T07:25:49.040533+0000","last_clean":"2024-03-06T07:25:49.040533+0000","last_became_active":"2024-03-06T07:25:47.751652+0000","last_became_peered":"2024-03-06T07:25:47.751652+0000","last_unstale":"2024-03-06T07:25:49.040533+0000","last_undegraded":"2024-03-06T07:25:49.040533+0000","last_fullsized":"2024-03-06T07:25:49.040533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:17.837597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761941+0000","last_change":"2024-03-06T07:25:46.744401+0000","last_active":"2024-03-06T07:25:48.761941+0000","last_peered":"2024-03-06T07:25:48.761941+0000","last_clean":"2024-03-06T07:25:48.761941+0000","last_became_active":"2024-03-06T07:25:46.744090+0000","last_became_peered":"2024-03-06T07:25:46.744090+0000","last_unstale":"2024-03-06T07:25:48.761941+0000","last_undegraded":"2024-03-06T07:25:48.761941+0000","last_fullsized":"2024-03-06T07:25:48.761941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:51.535471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737048+0000","last_change":"2024-03-06T07:25:47.748162+0000","last_active":"2024-03-06T07:25:48.737048+0000","last_peered":"2024-03-06T07:25:48.737048+0000","last_clean":"2024-03-06T07:25:48.737048+0000","last_became_active":"2024-03-06T07:25:47.747901+0000","last_became_peered":"2024-03-06T07:25:47.747901+0000","last_unstale":"2024-03-06T07:25:48.737048+0000","last_undegraded":"2024-03-06T07:25:48.737048+0000","last_fullsized":"2024-03-06T07:25:48.737048+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.952554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761902+0000","last_change":"2024-03-06T07:25:46.736412+0000","last_active":"2024-03-06T07:25:48.761902+0000","last_peered":"2024-03-06T07:25:48.761902+0000","last_clean":"2024-03-06T07:25:48.761902+0000","last_became_active":"2024-03-06T07:25:46.735973+0000","last_became_peered":"2024-03-06T07:25:46.735973+0000","last_unstale":"2024-03-06T07:25:48.761902+0000","last_undegraded":"2024-03-06T07:25:48.761902+0000","last_fullsized":"2024-03-06T07:25:48.761902+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:39.306436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735938+0000","last_change":"2024-03-06T07:25:47.748114+0000","last_active":"2024-03-06T07:25:48.735938+0000","last_peered":"2024-03-06T07:25:48.735938+0000","last_clean":"2024-03-06T07:25:48.735938+0000","last_became_active":"2024-03-06T07:25:47.747802+0000","last_became_peered":"2024-03-06T07:25:47.747802+0000","last_unstale":"2024-03-06T07:25:48.735938+0000","last_undegraded":"2024-03-06T07:25:48.735938+0000","last_fullsized":"2024-03-06T07:25:48.735938+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:27.832719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039237+0000","last_change":"2024-03-06T07:25:46.740390+0000","last_active":"2024-03-06T07:25:49.039237+0000","last_peered":"2024-03-06T07:25:49.039237+0000","last_clean":"2024-03-06T07:25:49.039237+0000","last_became_active":"2024-03-06T07:25:46.740119+0000","last_became_peered":"2024-03-06T07:25:46.740119+0000","last_unstale":"2024-03-06T07:25:49.039237+0000","last_undegraded":"2024-03-06T07:25:49.039237+0000","last_fullsized":"2024-03-06T07:25:49.039237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:52.685641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736602+0000","last_change":"2024-03-06T07:25:47.753494+0000","last_active":"2024-03-06T07:25:48.736602+0000","last_peered":"2024-03-06T07:25:48.736602+0000","last_clean":"2024-03-06T07:25:48.736602+0000","last_became_active":"2024-03-06T07:25:47.753128+0000","last_became_peered":"2024-03-06T07:25:47.753128+0000","last_unstale":"2024-03-06T07:25:48.736602+0000","last_undegraded":"2024-03-06T07:25:48.736602+0000","last_fullsized":"2024-03-06T07:25:48.736602+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:19.894041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736006+0000","last_change":"2024-03-06T07:25:46.746719+0000","last_active":"2024-03-06T07:25:48.736006+0000","last_peered":"2024-03-06T07:25:48.736006+0000","last_clean":"2024-03-06T07:25:48.736006+0000","last_became_active":"2024-03-06T07:25:46.746405+0000","last_became_peered":"2024-03-06T07:25:46.746405+0000","last_unstale":"2024-03-06T07:25:48.736006+0000","last_undegraded":"2024-03-06T07:25:48.736006+0000","last_fullsized":"2024-03-06T07:25:48.736006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:00.506076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735741+0000","last_change":"2024-03-06T07:25:47.746644+0000","last_active":"2024-03-06T07:25:48.735741+0000","last_peered":"2024-03-06T07:25:48.735741+0000","last_clean":"2024-03-06T07:25:48.735741+0000","last_became_active":"2024-03-06T07:25:47.746338+0000","last_became_peered":"2024-03-06T07:25:47.746338+0000","last_unstale":"2024-03-06T07:25:48.735741+0000","last_undegraded":"2024-03-06T07:25:48.735741+0000","last_fullsized":"2024-03-06T07:25:48.735741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:49.722061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.735738+0000","last_change":"2024-03-06T07:25:46.742152+0000","last_active":"2024-03-06T07:25:48.735738+0000","last_peered":"2024-03-06T07:25:48.735738+0000","last_clean":"2024-03-06T07:25:48.735738+0000","last_became_active":"2024-03-06T07:25:46.741827+0000","last_became_peered":"2024-03-06T07:25:46.741827+0000","last_unstale":"2024-03-06T07:25:48.735738+0000","last_undegraded":"2024-03-06T07:25:48.735738+0000","last_fullsized":"2024-03-06T07:25:48.735738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:53.097448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736315+0000","last_change":"2024-03-06T07:25:47.760666+0000","last_active":"2024-03-06T07:25:48.736315+0000","last_peered":"2024-03-06T07:25:48.736315+0000","last_clean":"2024-03-06T07:25:48.736315+0000","last_became_active":"2024-03-06T07:25:47.760210+0000","last_became_peered":"2024-03-06T07:25:47.760210+0000","last_unstale":"2024-03-06T07:25:48.736315+0000","last_undegraded":"2024-03-06T07:25:48.736315+0000","last_fullsized":"2024-03-06T07:25:48.736315+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:52.540026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.038963+0000","last_change":"2024-03-06T07:25:46.735528+0000","last_active":"2024-03-06T07:25:49.038963+0000","last_peered":"2024-03-06T07:25:49.038963+0000","last_clean":"2024-03-06T07:25:49.038963+0000","last_became_active":"2024-03-06T07:25:46.735242+0000","last_became_peered":"2024-03-06T07:25:46.735242+0000","last_unstale":"2024-03-06T07:25:49.038963+0000","last_undegraded":"2024-03-06T07:25:49.038963+0000","last_fullsized":"2024-03-06T07:25:49.038963+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:29.824789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737538+0000","last_change":"2024-03-06T07:25:47.746068+0000","last_active":"2024-03-06T07:25:48.737538+0000","last_peered":"2024-03-06T07:25:48.737538+0000","last_clean":"2024-03-06T07:25:48.737538+0000","last_became_active":"2024-03-06T07:25:47.745758+0000","last_became_peered":"2024-03-06T07:25:47.745758+0000","last_unstale":"2024-03-06T07:25:48.737538+0000","last_undegraded":"2024-03-06T07:25:48.737538+0000","last_fullsized":"2024-03-06T07:25:48.737538+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:10.096097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759998+0000","last_change":"2024-03-06T07:25:46.746740+0000","last_active":"2024-03-06T07:25:48.759998+0000","last_peered":"2024-03-06T07:25:48.759998+0000","last_clean":"2024-03-06T07:25:48.759998+0000","last_became_active":"2024-03-06T07:25:46.746272+0000","last_became_peered":"2024-03-06T07:25:46.746272+0000","last_unstale":"2024-03-06T07:25:48.759998+0000","last_undegraded":"2024-03-06T07:25:48.759998+0000","last_fullsized":"2024-03-06T07:25:48.759998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:16:04.573822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039946+0000","last_change":"2024-03-06T07:25:47.754144+0000","last_active":"2024-03-06T07:25:49.039946+0000","last_peered":"2024-03-06T07:25:49.039946+0000","last_clean":"2024-03-06T07:25:49.039946+0000","last_became_active":"2024-03-06T07:25:47.753738+0000","last_became_peered":"2024-03-06T07:25:47.753738+0000","last_unstale":"2024-03-06T07:25:49.039946+0000","last_undegraded":"2024-03-06T07:25:49.039946+0000","last_fullsized":"2024-03-06T07:25:49.039946+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:53.690196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.760064+0000","last_change":"2024-03-06T07:25:46.746681+0000","last_active":"2024-03-06T07:25:48.760064+0000","last_peered":"2024-03-06T07:25:48.760064+0000","last_clean":"2024-03-06T07:25:48.760064+0000","last_became_active":"2024-03-06T07:25:46.746393+0000","last_became_peered":"2024-03-06T07:25:46.746393+0000","last_unstale":"2024-03-06T07:25:48.760064+0000","last_undegraded":"2024-03-06T07:25:48.760064+0000","last_fullsized":"2024-03-06T07:25:48.760064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:01.102402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040348+0000","last_change":"2024-03-06T07:25:47.759138+0000","last_active":"2024-03-06T07:25:49.040348+0000","last_peered":"2024-03-06T07:25:49.040348+0000","last_clean":"2024-03-06T07:25:49.040348+0000","last_became_active":"2024-03-06T07:25:47.758792+0000","last_became_peered":"2024-03-06T07:25:47.758792+0000","last_unstale":"2024-03-06T07:25:49.040348+0000","last_undegraded":"2024-03-06T07:25:49.040348+0000","last_fullsized":"2024-03-06T07:25:49.040348+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:13.850354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039796+0000","last_change":"2024-03-06T07:25:46.747435+0000","last_active":"2024-03-06T07:25:49.039796+0000","last_peered":"2024-03-06T07:25:49.039796+0000","last_clean":"2024-03-06T07:25:49.039796+0000","last_became_active":"2024-03-06T07:25:46.747130+0000","last_became_peered":"2024-03-06T07:25:46.747130+0000","last_unstale":"2024-03-06T07:25:49.039796+0000","last_undegraded":"2024-03-06T07:25:49.039796+0000","last_fullsized":"2024-03-06T07:25:49.039796+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:25.297323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737109+0000","last_change":"2024-03-06T07:25:47.759956+0000","last_active":"2024-03-06T07:25:48.737109+0000","last_peered":"2024-03-06T07:25:48.737109+0000","last_clean":"2024-03-06T07:25:48.737109+0000","last_became_active":"2024-03-06T07:25:47.759705+0000","last_became_peered":"2024-03-06T07:25:47.759705+0000","last_unstale":"2024-03-06T07:25:48.737109+0000","last_undegraded":"2024-03-06T07:25:48.737109+0000","last_fullsized":"2024-03-06T07:25:48.737109+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:34.507002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040420+0000","last_change":"2024-03-06T07:25:46.745467+0000","last_active":"2024-03-06T07:25:49.040420+0000","last_peered":"2024-03-06T07:25:49.040420+0000","last_clean":"2024-03-06T07:25:49.040420+0000","last_became_active":"2024-03-06T07:25:46.745204+0000","last_became_peered":"2024-03-06T07:25:46.745204+0000","last_unstale":"2024-03-06T07:25:49.040420+0000","last_undegraded":"2024-03-06T07:25:49.040420+0000","last_fullsized":"2024-03-06T07:25:49.040420+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:08.171219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.737427+0000","last_change":"2024-03-06T07:25:47.738425+0000","last_active":"2024-03-06T07:25:48.737427+0000","last_peered":"2024-03-06T07:25:48.737427+0000","last_clean":"2024-03-06T07:25:48.737427+0000","last_became_active":"2024-03-06T07:25:47.738221+0000","last_became_peered":"2024-03-06T07:25:47.738221+0000","last_unstale":"2024-03-06T07:25:48.737427+0000","last_undegraded":"2024-03-06T07:25:48.737427+0000","last_fullsized":"2024-03-06T07:25:48.737427+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:14.952554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761806+0000","last_change":"2024-03-06T07:25:46.742402+0000","last_active":"2024-03-06T07:25:48.761806+0000","last_peered":"2024-03-06T07:25:48.761806+0000","last_clean":"2024-03-06T07:25:48.761806+0000","last_became_active":"2024-03-06T07:25:46.742144+0000","last_became_peered":"2024-03-06T07:25:46.742144+0000","last_unstale":"2024-03-06T07:25:48.761806+0000","last_undegraded":"2024-03-06T07:25:48.761806+0000","last_fullsized":"2024-03-06T07:25:48.761806+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:12.845799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.761869+0000","last_change":"2024-03-06T07:25:47.758008+0000","last_active":"2024-03-06T07:25:48.761869+0000","last_peered":"2024-03-06T07:25:48.761869+0000","last_clean":"2024-03-06T07:25:48.761869+0000","last_became_active":"2024-03-06T07:25:47.754667+0000","last_became_peered":"2024-03-06T07:25:47.754667+0000","last_unstale":"2024-03-06T07:25:48.761869+0000","last_undegraded":"2024-03-06T07:25:48.761869+0000","last_fullsized":"2024-03-06T07:25:48.761869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:34.507002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736719+0000","last_change":"2024-03-06T07:25:46.741357+0000","last_active":"2024-03-06T07:25:48.736719+0000","last_peered":"2024-03-06T07:25:48.736719+0000","last_clean":"2024-03-06T07:25:48.736719+0000","last_became_active":"2024-03-06T07:25:46.741039+0000","last_became_peered":"2024-03-06T07:25:46.741039+0000","last_unstale":"2024-03-06T07:25:48.736719+0000","last_undegraded":"2024-03-06T07:25:48.736719+0000","last_fullsized":"2024-03-06T07:25:48.736719+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:42.364991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039438+0000","last_change":"2024-03-06T07:25:47.755852+0000","last_active":"2024-03-06T07:25:49.039438+0000","last_peered":"2024-03-06T07:25:49.039438+0000","last_clean":"2024-03-06T07:25:49.039438+0000","last_became_active":"2024-03-06T07:25:47.755099+0000","last_became_peered":"2024-03-06T07:25:47.755099+0000","last_unstale":"2024-03-06T07:25:49.039438+0000","last_undegraded":"2024-03-06T07:25:49.039438+0000","last_fullsized":"2024-03-06T07:25:49.039438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:09.013439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039802+0000","last_change":"2024-03-06T07:25:46.746404+0000","last_active":"2024-03-06T07:25:49.039802+0000","last_peered":"2024-03-06T07:25:49.039802+0000","last_clean":"2024-03-06T07:25:49.039802+0000","last_became_active":"2024-03-06T07:25:46.746085+0000","last_became_peered":"2024-03-06T07:25:46.746085+0000","last_unstale":"2024-03-06T07:25:49.039802+0000","last_undegraded":"2024-03-06T07:25:49.039802+0000","last_fullsized":"2024-03-06T07:25:49.039802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:16.204854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039820+0000","last_change":"2024-03-06T07:25:47.751674+0000","last_active":"2024-03-06T07:25:49.039820+0000","last_peered":"2024-03-06T07:25:49.039820+0000","last_clean":"2024-03-06T07:25:49.039820+0000","last_became_active":"2024-03-06T07:25:47.751400+0000","last_became_peered":"2024-03-06T07:25:47.751400+0000","last_unstale":"2024-03-06T07:25:49.039820+0000","last_undegraded":"2024-03-06T07:25:49.039820+0000","last_fullsized":"2024-03-06T07:25:49.039820+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:34.948730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.736571+0000","last_change":"2024-03-06T07:25:46.739746+0000","last_active":"2024-03-06T07:25:48.736571+0000","last_peered":"2024-03-06T07:25:48.736571+0000","last_clean":"2024-03-06T07:25:48.736571+0000","last_became_active":"2024-03-06T07:25:46.739449+0000","last_became_peered":"2024-03-06T07:25:46.739449+0000","last_unstale":"2024-03-06T07:25:48.736571+0000","last_undegraded":"2024-03-06T07:25:48.736571+0000","last_fullsized":"2024-03-06T07:25:48.736571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:07.641157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:48.759838+0000","last_change":"2024-03-06T07:25:47.744991+0000","last_active":"2024-03-06T07:25:48.759838+0000","last_peered":"2024-03-06T07:25:48.759838+0000","last_clean":"2024-03-06T07:25:48.759838+0000","last_became_active":"2024-03-06T07:25:47.744673+0000","last_became_peered":"2024-03-06T07:25:47.744673+0000","last_unstale":"2024-03-06T07:25:48.759838+0000","last_undegraded":"2024-03-06T07:25:48.759838+0000","last_fullsized":"2024-03-06T07:25:48.759838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:46.718808+0000","last_clean_scrub_stamp":"2024-03-06T07:25:46.718808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:45.993623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.040220+0000","last_change":"2024-03-06T07:25:46.746841+0000","last_active":"2024-03-06T07:25:49.040220+0000","last_peered":"2024-03-06T07:25:49.040220+0000","last_clean":"2024-03-06T07:25:49.040220+0000","last_became_active":"2024-03-06T07:25:46.746592+0000","last_became_peered":"2024-03-06T07:25:46.746592+0000","last_unstale":"2024-03-06T07:25:49.040220+0000","last_undegraded":"2024-03-06T07:25:49.040220+0000","last_fullsized":"2024-03-06T07:25:49.040220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:45.714253+0000","last_clean_scrub_stamp":"2024-03-06T07:25:45.714253+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:12.845799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":281,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:49.039662+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:25:49.039662+0000","last_peered":"2024-03-06T07:25:49.039662+0000","last_clean":"2024-03-06T07:25:49.039662+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:25:49.039662+0000","last_undegraded":"2024-03-06T07:25:49.039662+0000","last_fullsized":"2024-03-06T07:25:49.039662+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1593344,"data_stored":1012776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1007616,"data_stored":423032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1011712,"data_stored":423972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574901,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":422338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1007616,"data_stored":423950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":422920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1007616,"data_stored":422532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32744,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339096,"statfs":{"total":96636764160,"available":96603234304,"internally_reserved":0,"allocated":1593344,"data_stored":1012740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:55.617 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:55.617 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:55.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.762+0000 7ff90d408640 1 Processor -- start 2024-03-06T07:25:55.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.762+0000 7ff90d408640 1 -- start start 2024-03-06T07:25:55.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.762+0000 7ff90d408640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff90815c050 0x7ff90806f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff90d408640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff90806fb40 0x7ff9080cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff90d408640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9080cd4b0 con 0x7ff9080ccca0 2024-03-06T07:25:55.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff90d408640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90806ff80 con 0x7ff90806fb40 2024-03-06T07:25:55.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff90d408640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9080cd790 con 0x7ff90815c050 2024-03-06T07:25:55.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff9067fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff90806fb40 0x7ff9080cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff906ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff90815c050 0x7ff90806f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff9067fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff90806fb40 0x7ff9080cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:45878/0 (socket says 172.21.15.59:45878) 2024-03-06T07:25:55.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff906ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff90815c050 0x7ff90806f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:60954/0 (socket says 172.21.15.59:60954) 2024-03-06T07:25:55.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff9077fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff9080ccca0 0x7ff9080cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:60014/0 (socket says 172.21.15.59:60014) 2024-03-06T07:25:55.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff9067fc640 1 -- 172.21.15.59:0/932761556 learned_addr learned my addr 172.21.15.59:0/932761556 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:55.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff906ffd640 1 -- 172.21.15.59:0/932761556 >> v2:172.21.15.164:3300/0 conn(0x7ff90806fb40 msgr2=0x7ff9080cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff906ffd640 1 --2- 172.21.15.59:0/932761556 >> v2:172.21.15.164:3300/0 conn(0x7ff90806fb40 0x7ff9080cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff906ffd640 1 -- 172.21.15.59:0/932761556 >> v1:172.21.15.164:6789/0 conn(0x7ff9080ccca0 legacy=0x7ff9080cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.763+0000 7ff906ffd640 1 -- 172.21.15.59:0/932761556 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9080cda70 con 0x7ff90815c050 2024-03-06T07:25:55.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.764+0000 7ff906ffd640 1 --2- 172.21.15.59:0/932761556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff90815c050 0x7ff90806f600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80097a0 tx=0x7ff8f802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c9513758c0f2c9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.764+0000 7ff905ffb640 1 -- 172.21.15.59:0/932761556 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f80315a0 con 0x7ff90815c050 2024-03-06T07:25:55.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.764+0000 7ff905ffb640 1 -- 172.21.15.59:0/932761556 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f8037070 con 0x7ff90815c050 2024-03-06T07:25:55.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.764+0000 7ff905ffb640 1 -- 172.21.15.59:0/932761556 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f8031960 con 0x7ff90815c050 2024-03-06T07:25:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.765+0000 7ff90d408640 1 -- 172.21.15.59:0/932761556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff90815c050 msgr2=0x7ff90806f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.765+0000 7ff90d408640 1 --2- 172.21.15.59:0/932761556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff90815c050 0x7ff90806f600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80097a0 tx=0x7ff8f802b840 comp rx=0 tx=0).stop 2024-03-06T07:25:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.765+0000 7ff90d408640 1 -- 172.21.15.59:0/932761556 shutdown_connections 2024-03-06T07:25:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.765+0000 7ff90d408640 1 --2- 172.21.15.59:0/932761556 >> v2:172.21.15.164:3300/0 conn(0x7ff90806fb40 0x7ff9080cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.765+0000 7ff90d408640 1 --2- 172.21.15.59:0/932761556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff90815c050 0x7ff90806f600 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.765+0000 7ff90d408640 1 -- 172.21.15.59:0/932761556 >> 172.21.15.59:0/932761556 conn(0x7ff9080b3800 msgr2=0x7ff9080b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.765+0000 7ff90d408640 1 -- 172.21.15.59:0/932761556 shutdown_connections 2024-03-06T07:25:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.765+0000 7ff90d408640 1 -- 172.21.15.59:0/932761556 wait complete. 2024-03-06T07:25:55.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.767+0000 7ff90d408640 1 Processor -- start 2024-03-06T07:25:55.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.767+0000 7ff90d408640 1 -- start start 2024-03-06T07:25:55.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.768+0000 7ff90d408640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff90806fb40 0x7ff90814d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.768+0000 7ff90d408640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9080ccca0 0x7ff9080b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.768+0000 7ff90d408640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90815c050 0x7ff9081558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.768+0000 7ff90d408640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9080d01f0 con 0x7ff90815c050 2024-03-06T07:25:55.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.768+0000 7ff90d408640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9080cd130 con 0x7ff90806fb40 2024-03-06T07:25:55.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.768+0000 7ff90d408640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9080d0370 con 0x7ff9080ccca0 2024-03-06T07:25:55.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.768+0000 7ff9077fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90815c050 0x7ff9081558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.768+0000 7ff906ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff90806fb40 0x7ff90814d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.768+0000 7ff9077fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90815c050 0x7ff9081558e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59650/0 (socket says 172.21.15.59:59650) 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.768+0000 7ff9077fe640 1 -- 172.21.15.59:0/3213639073 learned_addr learned my addr 172.21.15.59:0/3213639073 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.768+0000 7ff9067fc640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9080ccca0 0x7ff9080b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.769+0000 7ff9077fe640 1 -- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9080ccca0 msgr2=0x7ff9080b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.769+0000 7ff9077fe640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9080ccca0 0x7ff9080b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.769+0000 7ff9077fe640 1 -- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff90806fb40 msgr2=0x7ff90814d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.769+0000 7ff9077fe640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff90806fb40 0x7ff90814d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.769+0000 7ff9077fe640 1 -- 172.21.15.59:0/3213639073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90814dc80 con 0x7ff90815c050 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.769+0000 7ff9077fe640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90815c050 0x7ff9081558e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff90000c780 tx=0x7ff90000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.769+0000 7ff8ef7fe640 1 -- 172.21.15.59:0/3213639073 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff900017070 con 0x7ff90815c050 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.769+0000 7ff8ef7fe640 1 -- 172.21.15.59:0/3213639073 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90000a640 con 0x7ff90815c050 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.769+0000 7ff8ef7fe640 1 -- 172.21.15.59:0/3213639073 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90000aa30 con 0x7ff90815c050 2024-03-06T07:25:55.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.769+0000 7ff90d408640 1 -- 172.21.15.59:0/3213639073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff908154c20 con 0x7ff90815c050 2024-03-06T07:25:55.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.769+0000 7ff90d408640 1 -- 172.21.15.59:0/3213639073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9080b4a90 con 0x7ff90815c050 2024-03-06T07:25:55.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.770+0000 7ff90d408640 1 -- 172.21.15.59:0/3213639073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90809fe60 con 0x7ff90815c050 2024-03-06T07:25:55.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.771+0000 7ff8ef7fe640 1 -- 172.21.15.59:0/3213639073 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff90001a070 con 0x7ff90815c050 2024-03-06T07:25:55.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.771+0000 7ff8ef7fe640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8f406b4d0 0x7ff8f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.772+0000 7ff8ef7fe640 1 -- 172.21.15.59:0/3213639073 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff900089720 con 0x7ff90815c050 2024-03-06T07:25:55.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.772+0000 7ff906ffd640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8f406b4d0 0x7ff8f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.772+0000 7ff906ffd640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8f406b4d0 0x7ff8f406d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8036000 tx=0x7ff8f8008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:55.776+0000 7ff8ef7fe640 1 -- 172.21.15.59:0/3213639073 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff90814dc80 con 0x7ff90815c050 2024-03-06T07:25:56.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.025+0000 7ff90d408640 1 -- 172.21.15.59:0/3213639073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff908068ff0 con 0x7ff90815c050 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.027+0000 7ff8ef7fe640 1 -- 172.21.15.59:0/3213639073 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff900057f40 con 0x7ff90815c050 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA== 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:56.030 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-03-06T07:25:56.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.030+0000 7ff90d408640 1 -- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8f406b4d0 msgr2=0x7ff8f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.030+0000 7ff90d408640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8f406b4d0 0x7ff8f406d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8036000 tx=0x7ff8f8008030 comp rx=0 tx=0).stop 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.030+0000 7ff90d408640 1 -- 172.21.15.59:0/3213639073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90815c050 msgr2=0x7ff9081558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.030+0000 7ff90d408640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90815c050 0x7ff9081558e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff90000c780 tx=0x7ff90000cc40 comp rx=0 tx=0).stop 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.030+0000 7ff90d408640 1 -- 172.21.15.59:0/3213639073 shutdown_connections 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.030+0000 7ff90d408640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff8f406b4d0 0x7ff8f406d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.031+0000 7ff90d408640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff90815c050 0x7ff9081558e0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.031+0000 7ff90d408640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9080ccca0 0x7ff9080b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.031+0000 7ff90d408640 1 --2- 172.21.15.59:0/3213639073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff90806fb40 0x7ff90814d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.031+0000 7ff90d408640 1 -- 172.21.15.59:0/3213639073 >> 172.21.15.59:0/3213639073 conn(0x7ff9080b3800 msgr2=0x7ff908070230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.031+0000 7ff90d408640 1 -- 172.21.15.59:0/3213639073 shutdown_connections 2024-03-06T07:25:56.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.031+0000 7ff90d408640 1 -- 172.21.15.59:0/3213639073 wait complete. 2024-03-06T07:25:56.047 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.150+0000 7fc4258fb640 1 Processor -- start 2024-03-06T07:25:56.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.150+0000 7fc4258fb640 1 -- start start 2024-03-06T07:25:56.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc4258fb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc42015ab00 0x7fc4200b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc4258fb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4200b48f0 0x7fc4200af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc4258fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4200afc00 0x7fc42006e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc4258fb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4200b4e30 con 0x7fc4200b48f0 2024-03-06T07:25:56.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc4258fb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4200b5110 con 0x7fc42015ab00 2024-03-06T07:25:56.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc4258fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42015b310 con 0x7fc4200afc00 2024-03-06T07:25:56.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc41e7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4200b48f0 0x7fc4200af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc41e7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4200b48f0 0x7fc4200af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59664/0 (socket says 172.21.15.59:59664) 2024-03-06T07:25:56.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc41e7fc640 1 -- 172.21.15.59:0/1885967413 learned_addr learned my addr 172.21.15.59:0/1885967413 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:56.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc41e7fc640 1 -- 172.21.15.59:0/1885967413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4200afc00 msgr2=0x7fc42006e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc41e7fc640 1 --2- 172.21.15.59:0/1885967413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4200afc00 0x7fc42006e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc41e7fc640 1 -- 172.21.15.59:0/1885967413 >> v2:172.21.15.164:3300/0 conn(0x7fc42015ab00 msgr2=0x7fc4200b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.151+0000 7fc41e7fc640 1 --2- 172.21.15.59:0/1885967413 >> v2:172.21.15.164:3300/0 conn(0x7fc42015ab00 0x7fc4200b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.152+0000 7fc41e7fc640 1 -- 172.21.15.59:0/1885967413 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42015b5f0 con 0x7fc4200b48f0 2024-03-06T07:25:56.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.152+0000 7fc41e7fc640 1 --2- 172.21.15.59:0/1885967413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4200b48f0 0x7fc4200af6c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fc4200732b0 tx=0x7fc41402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08e0d3219d41eed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.152+0000 7fc41dffb640 1 -- 172.21.15.59:0/1885967413 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41403b070 con 0x7fc4200b48f0 2024-03-06T07:25:56.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.152+0000 7fc41dffb640 1 -- 172.21.15.59:0/1885967413 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc414036440 con 0x7fc4200b48f0 2024-03-06T07:25:56.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.152+0000 7fc41dffb640 1 -- 172.21.15.59:0/1885967413 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414036810 con 0x7fc4200b48f0 2024-03-06T07:25:56.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.153+0000 7fc4258fb640 1 -- 172.21.15.59:0/1885967413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4200b48f0 msgr2=0x7fc4200af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.153+0000 7fc4258fb640 1 --2- 172.21.15.59:0/1885967413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4200b48f0 0x7fc4200af6c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fc4200732b0 tx=0x7fc41402e0e0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.153+0000 7fc4258fb640 1 -- 172.21.15.59:0/1885967413 shutdown_connections 2024-03-06T07:25:56.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.153+0000 7fc4258fb640 1 --2- 172.21.15.59:0/1885967413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4200afc00 0x7fc42006e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.153+0000 7fc4258fb640 1 --2- 172.21.15.59:0/1885967413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4200b48f0 0x7fc4200af6c0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.153+0000 7fc4258fb640 1 --2- 172.21.15.59:0/1885967413 >> v2:172.21.15.164:3300/0 conn(0x7fc42015ab00 0x7fc4200b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.153+0000 7fc4258fb640 1 -- 172.21.15.59:0/1885967413 >> 172.21.15.59:0/1885967413 conn(0x7fc42006c0c0 msgr2=0x7fc42006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.153+0000 7fc4258fb640 1 -- 172.21.15.59:0/1885967413 shutdown_connections 2024-03-06T07:25:56.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.153+0000 7fc4258fb640 1 -- 172.21.15.59:0/1885967413 wait complete. 2024-03-06T07:25:56.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.155+0000 7fc4258fb640 1 Processor -- start 2024-03-06T07:25:56.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.155+0000 7fc4258fb640 1 -- start start 2024-03-06T07:25:56.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.156+0000 7fc4258fb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4200afc00 0x7fc420141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.156+0000 7fc4258fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4200b48f0 0x7fc4200b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.156+0000 7fc4258fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc42015ab00 0x7fc4201f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.156+0000 7fc4258fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42015b060 con 0x7fc4200b48f0 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.156+0000 7fc4258fb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42015aee0 con 0x7fc4200afc00 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.156+0000 7fc4258fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42006ecc0 con 0x7fc42015ab00 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.156+0000 7fc41e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4200b48f0 0x7fc4200b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.156+0000 7fc41e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4200b48f0 0x7fc4200b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59678/0 (socket says 172.21.15.59:59678) 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.156+0000 7fc41e7fc640 1 -- 172.21.15.59:0/1035908179 learned_addr learned my addr 172.21.15.59:0/1035908179 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.156+0000 7fc41effd640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4200afc00 0x7fc420141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.156+0000 7fc41f7fe640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc42015ab00 0x7fc4201f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.157+0000 7fc41e7fc640 1 -- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc42015ab00 msgr2=0x7fc4201f2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.157+0000 7fc41e7fc640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc42015ab00 0x7fc4201f2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.157+0000 7fc41e7fc640 1 -- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4200afc00 msgr2=0x7fc420141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.157+0000 7fc41e7fc640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4200afc00 0x7fc420141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.157+0000 7fc41e7fc640 1 -- 172.21.15.59:0/1035908179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4201ef6d0 con 0x7fc4200b48f0 2024-03-06T07:25:56.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.157+0000 7fc41e7fc640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4200b48f0 0x7fc4200b0440 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fc41402e120 tx=0x7fc4140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.157+0000 7fc407fff640 1 -- 172.21.15.59:0/1035908179 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414045070 con 0x7fc4200b48f0 2024-03-06T07:25:56.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.157+0000 7fc407fff640 1 -- 172.21.15.59:0/1035908179 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc414040890 con 0x7fc4200b48f0 2024-03-06T07:25:56.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.157+0000 7fc407fff640 1 -- 172.21.15.59:0/1035908179 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41403b040 con 0x7fc4200b48f0 2024-03-06T07:25:56.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.157+0000 7fc4258fb640 1 -- 172.21.15.59:0/1035908179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4200b0f00 con 0x7fc4200b48f0 2024-03-06T07:25:56.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.157+0000 7fc4258fb640 1 -- 172.21.15.59:0/1035908179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4200b14c0 con 0x7fc4200b48f0 2024-03-06T07:25:56.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.158+0000 7fc4258fb640 1 -- 172.21.15.59:0/1035908179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc42004f330 con 0x7fc4200b48f0 2024-03-06T07:25:56.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.159+0000 7fc407fff640 1 -- 172.21.15.59:0/1035908179 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc414048070 con 0x7fc4200b48f0 2024-03-06T07:25:56.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.159+0000 7fc407fff640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc40c06b4d0 0x7fc40c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.159+0000 7fc407fff640 1 -- 172.21.15.59:0/1035908179 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4140aefc0 con 0x7fc4200b48f0 2024-03-06T07:25:56.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.159+0000 7fc41effd640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc40c06b4d0 0x7fc40c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.160+0000 7fc41effd640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc40c06b4d0 0x7fc40c06d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc41000b810 tx=0x7fc410008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.163+0000 7fc407fff640 1 -- 172.21.15.59:0/1035908179 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc41407dff0 con 0x7fc4200b48f0 2024-03-06T07:25:56.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.384+0000 7fc4258fb640 1 -- 172.21.15.59:0/1035908179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc420157b20 con 0x7fc4200b48f0 2024-03-06T07:25:56.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.384+0000 7fc407fff640 1 -- 172.21.15.59:0/1035908179 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fc41407d8f0 con 0x7fc4200b48f0 2024-03-06T07:25:56.387 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:56.387 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":42,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2865818497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2865818497},{"type":"v1","addr":"172.21.15.164:6836","nonce":2865818497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1805325807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1805325807},{"type":"v1","addr":"172.21.15.164:6837","nonce":1805325807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6999,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3345771250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3345771250},{"type":"v1","addr":"172.21.15.59:6835","nonce":3345771250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:47.090938+0000","modified":"2024-03-06T07:25:49.093664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2521311582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2521311582},{"type":"v1","addr":"172.21.15.59:6834","nonce":2521311582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7002,"qdb_cluster":[7002]},"id":6}]} 2024-03-06T07:25:56.387 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 42 2024-03-06T07:25:56.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 -- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc40c06b4d0 msgr2=0x7fc40c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc40c06b4d0 0x7fc40c06d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc41000b810 tx=0x7fc410008040 comp rx=0 tx=0).stop 2024-03-06T07:25:56.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 -- 172.21.15.59:0/1035908179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4200b48f0 msgr2=0x7fc4200b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4200b48f0 0x7fc4200b0440 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fc41402e120 tx=0x7fc4140046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 -- 172.21.15.59:0/1035908179 shutdown_connections 2024-03-06T07:25:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc40c06b4d0 0x7fc40c06d990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc42015ab00 0x7fc4201f2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4200b48f0 0x7fc4200b0440 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 --2- 172.21.15.59:0/1035908179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4200afc00 0x7fc420141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 -- 172.21.15.59:0/1035908179 >> 172.21.15.59:0/1035908179 conn(0x7fc42006c0c0 msgr2=0x7fc4200b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 -- 172.21.15.59:0/1035908179 shutdown_connections 2024-03-06T07:25:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.386+0000 7fc4258fb640 1 -- 172.21.15.59:0/1035908179 wait complete. 2024-03-06T07:25:56.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:47.090938+0000', 'modified': '2024-03-06T07:25:49.093664+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7002}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7002': {'gid': 7002, 'name': 'a', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6834/2521311582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6833', 'nonce': 2521311582}, {'type': 'v1', 'addr': '172.21.15.59:6834', 'nonce': 2521311582}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7002, 'qdb_cluster': [7002]} 2024-03-06T07:25:56.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:56.801 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:25:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.902+0000 7f06cbcaf640 1 Processor -- start 2024-03-06T07:25:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.903+0000 7f06cbcaf640 1 -- start start 2024-03-06T07:25:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.903+0000 7f06cbcaf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f06c4162400 0x7f06c40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.903+0000 7f06cbcaf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c40d22c0 con 0x7f06c4162970 2024-03-06T07:25:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.903+0000 7f06cbcaf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c40d25a0 con 0x7f06c40b4460 2024-03-06T07:25:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.903+0000 7f06cbcaf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c4161b00 con 0x7f06c4162400 2024-03-06T07:25:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c924a640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f06c4162970 0x7f06c4161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60590/0 (socket says 172.21.15.59:60590) 2024-03-06T07:25:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c924a640 1 -- 172.21.15.59:0/1088233881 learned_addr learned my addr 172.21.15.59:0/1088233881 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228472805 0 0) 0x7f06c40d22c0 con 0x7f06c4162970 2024-03-06T07:25:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06ca24c640 1 --2- 172.21.15.59:0/1088233881 >> v2:172.21.15.164:3300/0 conn(0x7f06c4162400 0x7f06c40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b0003520 con 0x7f06c4162970 2024-03-06T07:25:56.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3700696121 0 0) 0x7f06b0003520 con 0x7f06c4162970 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 >> v2:172.21.15.164:3300/0 conn(0x7f06c4162400 msgr2=0x7f06c40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 --2- 172.21.15.59:0/1088233881 >> v2:172.21.15.164:3300/0 conn(0x7f06c4162400 0x7f06c40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 >> v1:172.21.15.164:6789/0 conn(0x7f06c40b4460 legacy=0x7f06c40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c40d2880 con 0x7f06c4162970 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f06b4001670 con 0x7f06c4162970 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 >> v1:172.21.15.59:6789/0 conn(0x7f06c4162970 legacy=0x7f06c4161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 --2- 172.21.15.59:0/1088233881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 0x7f06b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 --2- 172.21.15.59:0/1088233881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06b0004800 0x7f06b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 --2- 172.21.15.59:0/1088233881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06b00051c0 0x7f06b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c4161b00 con 0x7f06b00039b0 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b000c060 con 0x7f06b00051c0 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b000c260 con 0x7f06b0004800 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/1088233881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 0x7f06b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06c924a640 1 --2- 172.21.15.59:0/1088233881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06b0004800 0x7f06b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.904+0000 7f06ca24c640 1 --2- 172.21.15.59:0/1088233881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06b00051c0 0x7f06b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.905+0000 7f06c9a4b640 1 -- 172.21.15.59:0/1088233881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06b0004800 msgr2=0x7f06b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.905+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/1088233881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06b0004800 0x7f06b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.905+0000 7f06c9a4b640 1 -- 172.21.15.59:0/1088233881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06b00051c0 msgr2=0x7f06b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.905+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/1088233881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06b00051c0 0x7f06b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.905+0000 7f06c9a4b640 1 -- 172.21.15.59:0/1088233881 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b0003520 con 0x7f06b00039b0 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.905+0000 7f06ca24c640 1 -- 172.21.15.59:0/1088233881 reap_dead start 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.905+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/1088233881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 0x7f06b0003dd0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f06c0001750 tx=0x7f06b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d46f3ada56d344 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.906+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0001370 con 0x7f06b00039b0 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.906+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c00052d0 con 0x7f06b00039b0 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.906+0000 7f06c8a49640 1 -- 172.21.15.59:0/1088233881 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0003830 con 0x7f06b00039b0 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.906+0000 7f06cbcaf640 1 -- 172.21.15.59:0/1088233881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 msgr2=0x7f06b0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.906+0000 7f06cbcaf640 1 --2- 172.21.15.59:0/1088233881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 0x7f06b0003dd0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f06c0001750 tx=0x7f06b0005830 comp rx=0 tx=0).stop 2024-03-06T07:25:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.906+0000 7f06cbcaf640 1 -- 172.21.15.59:0/1088233881 shutdown_connections 2024-03-06T07:25:56.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.906+0000 7f06cbcaf640 1 --2- 172.21.15.59:0/1088233881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 0x7f06b0003dd0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.906+0000 7f06cbcaf640 1 -- 172.21.15.59:0/1088233881 >> 172.21.15.59:0/1088233881 conn(0x7f06c40d3ce0 msgr2=0x7f06c40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.906+0000 7f06cbcaf640 1 -- 172.21.15.59:0/1088233881 shutdown_connections 2024-03-06T07:25:56.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.906+0000 7f06cbcaf640 1 -- 172.21.15.59:0/1088233881 wait complete. 2024-03-06T07:25:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.907+0000 7f06cbcaf640 1 Processor -- start 2024-03-06T07:25:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.907+0000 7f06cbcaf640 1 -- start start 2024-03-06T07:25:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06cbcaf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 0x7f06c41561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06cbcaf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06c40c9790 0x7f06c40c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06cbcaf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06c40cc4b0 0x7f06c40cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06cbcaf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40ce950 con 0x7f06b00039b0 2024-03-06T07:25:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06cbcaf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40cecc0 con 0x7f06c40cc4b0 2024-03-06T07:25:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06cbcaf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40b4840 con 0x7f06c40c9790 2024-03-06T07:25:56.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06c9a4b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 0x7f06c41561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06c9a4b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 0x7f06c41561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59690/0 (socket says 172.21.15.59:59690) 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06c9a4b640 1 -- 172.21.15.59:0/2168401178 learned_addr learned my addr 172.21.15.59:0/2168401178 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06c924a640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06c40c9790 0x7f06c40c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06ca24c640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06c40cc4b0 0x7f06c40cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06c9a4b640 1 -- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06c40c9790 msgr2=0x7f06c40c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06c40c9790 0x7f06c40c9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06c9a4b640 1 -- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06c40cc4b0 msgr2=0x7f06c40cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.908+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06c40cc4b0 0x7f06c40cc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.909+0000 7f06c9a4b640 1 -- 172.21.15.59:0/2168401178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c41566e0 con 0x7f06b00039b0 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.909+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 0x7f06c41561a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f06c0000c00 tx=0x7f06c0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.909+0000 7f06ba7fc640 1 -- 172.21.15.59:0/2168401178 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0041020 con 0x7f06b00039b0 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.909+0000 7f06ba7fc640 1 -- 172.21.15.59:0/2168401178 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c0005480 con 0x7f06b00039b0 2024-03-06T07:25:56.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.909+0000 7f06cbcaf640 1 -- 172.21.15.59:0/2168401178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c40cb000 con 0x7f06b00039b0 2024-03-06T07:25:56.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.909+0000 7f06ba7fc640 1 -- 172.21.15.59:0/2168401178 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0007a30 con 0x7f06b00039b0 2024-03-06T07:25:56.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.909+0000 7f06cbcaf640 1 -- 172.21.15.59:0/2168401178 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c40cb5c0 con 0x7f06b00039b0 2024-03-06T07:25:56.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.911+0000 7f06ba7fc640 1 -- 172.21.15.59:0/2168401178 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06c0044020 con 0x7f06b00039b0 2024-03-06T07:25:56.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.911+0000 7f06ba7fc640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f06a806b500 0x7f06a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.911+0000 7f06ba7fc640 1 -- 172.21.15.59:0/2168401178 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06c00abc70 con 0x7f06b00039b0 2024-03-06T07:25:56.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.911+0000 7f06c924a640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f06a806b500 0x7f06a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.912+0000 7f06ce974740 1 Processor -- start 2024-03-06T07:25:56.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.912+0000 7f06ce974740 1 -- start start 2024-03-06T07:25:56.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.912+0000 7f06c924a640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f06a806b500 0x7f06a806d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f06b4001050 tx=0x7f06b4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06ce974740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e17008e870 0x55e17008ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06ce974740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700928c0 0x55e170094360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06ce974740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700be600 0x55e1700bea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06ce974740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e16fff86b0 con 0x55e1700928c0 2024-03-06T07:25:56.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06ce974740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e16ffa1a60 con 0x55e1700be600 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06ce974740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e16fff4750 con 0x55e17008e870 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06c924a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700928c0 0x55e170094360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06c924a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700928c0 0x55e170094360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59694/0 (socket says 172.21.15.59:59694) 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06c924a640 1 -- 172.21.15.59:0/1694711415 learned_addr learned my addr 172.21.15.59:0/1694711415 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/1694711415 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e17008e870 0x55e17008ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06ca24c640 1 --2- 172.21.15.59:0/1694711415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700be600 0x55e1700bea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06c924a640 1 -- 172.21.15.59:0/1694711415 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e17008e870 msgr2=0x55e17008ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06c924a640 1 --2- 172.21.15.59:0/1694711415 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e17008e870 0x55e17008ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06c924a640 1 -- 172.21.15.59:0/1694711415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700be600 msgr2=0x55e1700bea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06c924a640 1 --2- 172.21.15.59:0/1694711415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700be600 0x55e1700bea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06c924a640 1 -- 172.21.15.59:0/1694711415 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e16ffbff30 con 0x55e1700928c0 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06c924a640 1 --2- 172.21.15.59:0/1694711415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700928c0 0x55e170094360 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f06b4007370 tx=0x7f06b4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc7d7b0bf2500e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06a77fe640 1 -- 172.21.15.59:0/1694711415 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4028070 con 0x55e1700928c0 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.913+0000 7f06a77fe640 1 -- 172.21.15.59:0/1694711415 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b4015c00 con 0x55e1700928c0 2024-03-06T07:25:56.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.914+0000 7f06a77fe640 1 -- 172.21.15.59:0/1694711415 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b40143d0 con 0x55e1700928c0 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.914+0000 7f06ce974740 1 -- 172.21.15.59:0/1694711415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700928c0 msgr2=0x55e170094360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.914+0000 7f06ce974740 1 --2- 172.21.15.59:0/1694711415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700928c0 0x55e170094360 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f06b4007370 tx=0x7f06b4014000 comp rx=0 tx=0).stop 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.914+0000 7f06ce974740 1 -- 172.21.15.59:0/1694711415 shutdown_connections 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.914+0000 7f06ce974740 1 --2- 172.21.15.59:0/1694711415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700be600 0x55e1700bea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.914+0000 7f06ce974740 1 --2- 172.21.15.59:0/1694711415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700928c0 0x55e170094360 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.914+0000 7f06ce974740 1 --2- 172.21.15.59:0/1694711415 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e17008e870 0x55e17008ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.914+0000 7f06ce974740 1 -- 172.21.15.59:0/1694711415 >> 172.21.15.59:0/1694711415 conn(0x55e1700908f0 msgr2=0x55e1700921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.914+0000 7f06ce974740 1 -- 172.21.15.59:0/1694711415 shutdown_connections 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.914+0000 7f06ce974740 1 -- 172.21.15.59:0/1694711415 wait complete. 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.915+0000 7f06ce974740 1 -- start start 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.915+0000 7f06ce974740 1 Processor -- start 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.916+0000 7f06ce974740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700be600 0x55e1700d6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.916+0000 7f06ce974740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700b1d90 0x55e1700bb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.916+0000 7f06ce974740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700d7960 0x55e1700b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.916+0000 7f06ce974740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e170092d50 con 0x55e1700d7960 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.916+0000 7f06ce974740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e16ffb4ea0 con 0x55e1700b1d90 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.916+0000 7f06ce974740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e170093030 con 0x55e1700be600 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.916+0000 7f06ce974740 10 client.0 _ll_register_callbacks cb 0x55e16ffb4870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.916+0000 7f06ca24c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700d7960 0x55e1700b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.916+0000 7f06ca24c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700d7960 0x55e1700b2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59704/0 (socket says 172.21.15.59:59704) 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.916+0000 7f06ca24c640 1 -- 172.21.15.59:0/3059296259 learned_addr learned my addr 172.21.15.59:0/3059296259 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700be600 0x55e1700d6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06c924a640 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700b1d90 0x55e1700bb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06ca24c640 1 -- 172.21.15.59:0/3059296259 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700be600 msgr2=0x55e1700d6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06ca24c640 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700be600 0x55e1700d6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06ca24c640 1 -- 172.21.15.59:0/3059296259 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700b1d90 msgr2=0x55e1700bb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06ca24c640 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700b1d90 0x55e1700bb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06ca24c640 1 -- 172.21.15.59:0/3059296259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e16ffb5270 con 0x55e1700d7960 2024-03-06T07:25:56.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06ca24c640 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700d7960 0x55e1700b2330 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f06bc004270 tx=0x7f06bc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6954af92c8546c22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06a4ff9640 1 -- 172.21.15.59:0/3059296259 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc017820 con 0x55e1700d7960 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06a4ff9640 1 -- 172.21.15.59:0/3059296259 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06bc0179c0 con 0x55e1700d7960 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06ce974740 1 -- 172.21.15.59:0/3059296259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06b4015c00 con 0x55e1700d7960 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06a4ff9640 1 -- 172.21.15.59:0/3059296259 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06bc026050 con 0x55e1700d7960 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06ce974740 10 client.7185 fetch_fsmap learned FSMap version 42 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06ce974740 1 -- 172.21.15.59:0/3059296259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55e16fd0d2d0 con 0x55e1700d7960 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.917+0000 7f06a4ff9640 1 -- 172.21.15.59:0/3059296259 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b4015c00 con 0x55e1700d7960 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.918+0000 7f06a4ff9640 1 -- 172.21.15.59:0/3059296259 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 42) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f06bc029020 con 0x55e1700d7960 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.918+0000 7f06ce974740 10 client.7185 fetch_fsmap finished waiting for FSMap version 42 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.918+0000 7f06ce974740 10 client.7185 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.918+0000 7f06ce974740 20 client.7185 populate_metadata read hostname 'smithi059' 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.918+0000 7f06ce974740 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.59:6833/2521311582,v1:172.21.15.59:6834/2521311582] conn(0x55e1700b69f0 0x55e1700f8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.918+0000 7f06ce974740 4 client.7185 mds_command: new command op to 7002 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.918+0000 7f06ce974740 1 -- 172.21.15.59:0/3059296259 --> [v2:172.21.15.59:6833/2521311582,v1:172.21.15.59:6834/2521311582] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e170005bf0 con 0x55e1700b69f0 2024-03-06T07:25:56.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.918+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.59:6833/2521311582,v1:172.21.15.59:6834/2521311582] conn(0x55e1700b69f0 0x55e1700f8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.918+0000 7f06a4ff9640 10 client.7185 ms_handle_connect on v2:172.21.15.59:6833/2521311582 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.918+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.59:6833/2521311582,v1:172.21.15.59:6834/2521311582] conn(0x55e1700b69f0 0x55e1700f8ce0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=993758441c2cfa02 server_cookie=93d546335e950892 in_seq=0 out_seq=0 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06a4ff9640 1 -- 172.21.15.59:0/3059296259 <== mds.0 v2:172.21.15.59:6833/2521311582 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e170005bf0 con 0x55e1700b69f0 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06a4ff9640 10 client.7185 handle_command_reply: tid=0 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 1 client.7185 shutdown 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 1 -- 172.21.15.59:0/3059296259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700d7960 msgr2=0x55e1700b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700d7960 0x55e1700b2330 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f06bc004270 tx=0x7f06bc004360 comp rx=0 tx=0).stop 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 1 -- 172.21.15.59:0/3059296259 shutdown_connections 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.59:6833/2521311582,v1:172.21.15.59:6834/2521311582] conn(0x55e1700b69f0 0x55e1700f8ce0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700d7960 0x55e1700b2330 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700b1d90 0x55e1700bb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06a4ff9640 0 client.7185 ms_handle_reset on v2:172.21.15.59:6833/2521311582 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 1 --2- 172.21.15.59:0/3059296259 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700be600 0x55e1700d6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 1 -- 172.21.15.59:0/3059296259 >> 172.21.15.59:0/3059296259 conn(0x55e17008e870 msgr2=0x55e17008d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 1 -- 172.21.15.59:0/3059296259 shutdown_connections 2024-03-06T07:25:56.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 1 -- 172.21.15.59:0/3059296259 wait complete. 2024-03-06T07:25:56.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.919+0000 7f06ce974740 20 client.7185 trim_cache size 0 max 16384 2024-03-06T07:25:56.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.952+0000 7f06ce974740 1 Processor -- start 2024-03-06T07:25:56.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.952+0000 7f06ce974740 1 -- start start 2024-03-06T07:25:56.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06ce974740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700b1d90 0x55e1700bb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06ce974740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700be600 0x55e1700b1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06ce974740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700d7960 0x55e17008d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06ce974740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c41ffca0 con 0x55e1700b1d90 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06ce974740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c41f86b0 con 0x55e1700d7960 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06ce974740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c40c84d0 con 0x55e1700be600 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06c9a4b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700b1d90 0x55e1700bb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06c9a4b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700b1d90 0x55e1700bb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59706/0 (socket says 172.21.15.59:59706) 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06c9a4b640 1 -- 172.21.15.59:0/440102842 learned_addr learned my addr 172.21.15.59:0/440102842 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06c9a4b640 1 -- 172.21.15.59:0/440102842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700be600 msgr2=0x55e1700b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/440102842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700be600 0x55e1700b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06c9a4b640 1 -- 172.21.15.59:0/440102842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700d7960 msgr2=0x55e17008d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/440102842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700d7960 0x55e17008d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.953+0000 7f06c9a4b640 1 -- 172.21.15.59:0/440102842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e16fdd76b0 con 0x55e1700b1d90 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/440102842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700b1d90 0x55e1700bb970 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f06c00767c0 tx=0x7f06c007d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76f77b4b07d54bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06a6ffd640 1 -- 172.21.15.59:0/440102842 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0082b00 con 0x55e1700b1d90 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06a6ffd640 1 -- 172.21.15.59:0/440102842 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c0093020 con 0x55e1700b1d90 2024-03-06T07:25:56.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06a6ffd640 1 -- 172.21.15.59:0/440102842 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0006e10 con 0x55e1700b1d90 2024-03-06T07:25:56.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06ce974740 1 -- 172.21.15.59:0/440102842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700b1d90 msgr2=0x55e1700bb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06ce974740 1 --2- 172.21.15.59:0/440102842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700b1d90 0x55e1700bb970 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f06c00767c0 tx=0x7f06c007d040 comp rx=0 tx=0).stop 2024-03-06T07:25:56.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06ce974740 1 -- 172.21.15.59:0/440102842 shutdown_connections 2024-03-06T07:25:56.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06ce974740 1 --2- 172.21.15.59:0/440102842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700d7960 0x55e17008d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06ce974740 1 --2- 172.21.15.59:0/440102842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700be600 0x55e1700b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06ce974740 1 --2- 172.21.15.59:0/440102842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700b1d90 0x55e1700bb970 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06ce974740 1 -- 172.21.15.59:0/440102842 >> 172.21.15.59:0/440102842 conn(0x55e17008e870 msgr2=0x55e170089df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06ce974740 1 -- 172.21.15.59:0/440102842 shutdown_connections 2024-03-06T07:25:56.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.954+0000 7f06ce974740 1 -- 172.21.15.59:0/440102842 wait complete. 2024-03-06T07:25:56.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.955+0000 7f06ce974740 1 -- start start 2024-03-06T07:25:56.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.955+0000 7f06ce974740 1 Processor -- start 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06ce974740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700b1d90 0x55e170147b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06ce974740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700be600 0x55e1700b8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06ce974740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700d7960 0x55e1700b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06ce974740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1700d82b0 con 0x55e1700be600 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06ce974740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1700bb020 con 0x55e1700b1d90 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06ce974740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e1700b3020 con 0x55e1700d7960 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06ce974740 10 client.0 _ll_register_callbacks cb 0x7f06bc003d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06c924a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700be600 0x55e1700b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06c924a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700be600 0x55e1700b8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59718/0 (socket says 172.21.15.59:59718) 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06c924a640 1 -- 172.21.15.59:0/2040824987 learned_addr learned my addr 172.21.15.59:0/2040824987 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700b1d90 0x55e170147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06ca24c640 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700d7960 0x55e1700b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06c924a640 1 -- 172.21.15.59:0/2040824987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700d7960 msgr2=0x55e1700b86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06c924a640 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700d7960 0x55e1700b86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06c924a640 1 -- 172.21.15.59:0/2040824987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700b1d90 msgr2=0x55e170147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06c924a640 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700b1d90 0x55e170147b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06c924a640 1 -- 172.21.15.59:0/2040824987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e17014a4e0 con 0x55e1700be600 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06ca24c640 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700d7960 0x55e1700b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06c924a640 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700be600 0x55e1700b8170 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f06b4015840 tx=0x7f06b40078f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91573c1d26fc9802 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.956+0000 7f06a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06a4ff9640 1 -- 172.21.15.59:0/2040824987 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4033060 con 0x55e1700be600 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06a4ff9640 1 -- 172.21.15.59:0/2040824987 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06b4022c10 con 0x55e1700be600 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06ce974740 1 -- 172.21.15.59:0/2040824987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06c0093020 con 0x55e1700be600 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06a4ff9640 1 -- 172.21.15.59:0/2040824987 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06b4039050 con 0x55e1700be600 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06ce974740 10 client.7191 fetch_fsmap learned FSMap version 42 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06ce974740 1 -- 172.21.15.59:0/2040824987 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55e1700d9280 con 0x55e1700be600 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06a4ff9640 1 -- 172.21.15.59:0/2040824987 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c0093020 con 0x55e1700be600 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06a4ff9640 1 -- 172.21.15.59:0/2040824987 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 42) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f06b403c020 con 0x55e1700be600 2024-03-06T07:25:56.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06ce974740 10 client.7191 fetch_fsmap finished waiting for FSMap version 42 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06ce974740 10 client.7191 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06ce974740 20 client.7191 populate_metadata read hostname 'smithi059' 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06ce974740 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.59:6833/2521311582,v1:172.21.15.59:6834/2521311582] conn(0x55e1700ecec0 0x55e1700bd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06ce974740 4 client.7191 mds_command: new command op to 7002 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06ce974740 1 -- 172.21.15.59:0/2040824987 --> [v2:172.21.15.59:6833/2521311582,v1:172.21.15.59:6834/2521311582] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e1700d74d0 con 0x55e1700ecec0 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.59:6833/2521311582,v1:172.21.15.59:6834/2521311582] conn(0x55e1700ecec0 0x55e1700bd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06a4ff9640 10 client.7191 ms_handle_connect on v2:172.21.15.59:6833/2521311582 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.957+0000 7f06c9a4b640 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.59:6833/2521311582,v1:172.21.15.59:6834/2521311582] conn(0x55e1700ecec0 0x55e1700bd3e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f0d90342ab8573f server_cookie=dbfa8fcdce076dae in_seq=0 out_seq=0 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06a4ff9640 1 -- 172.21.15.59:0/2040824987 <== mds.0 v2:172.21.15.59:6833/2521311582 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55e1700d74d0 con 0x55e1700ecec0 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06a4ff9640 10 client.7191 handle_command_reply: tid=0 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 1 client.7191 shutdown 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 1 -- 172.21.15.59:0/2040824987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700be600 msgr2=0x55e1700b8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700be600 0x55e1700b8170 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f06b4015840 tx=0x7f06b40078f0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 1 -- 172.21.15.59:0/2040824987 shutdown_connections 2024-03-06T07:25:56.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.59:6833/2521311582,v1:172.21.15.59:6834/2521311582] conn(0x55e1700ecec0 0x55e1700bd3e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55e1700d7960 0x55e1700b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55e1700be600 0x55e1700b8170 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06a4ff9640 0 client.7191 ms_handle_reset on v2:172.21.15.59:6833/2521311582 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 1 --2- 172.21.15.59:0/2040824987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55e1700b1d90 0x55e170147b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 1 -- 172.21.15.59:0/2040824987 >> 172.21.15.59:0/2040824987 conn(0x55e17008e870 msgr2=0x55e1700fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 1 -- 172.21.15.59:0/2040824987 shutdown_connections 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 1 -- 172.21.15.59:0/2040824987 wait complete. 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.958+0000 7f06ce974740 20 client.7191 trim_cache size 0 max 16384 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:25:56.965 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 7002, 2024-03-06T07:25:56.966 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:25:56.966 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:25:56.966 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:25:56.966 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 9.2111571619999992, 2024-03-06T07:25:56.966 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 42, 2024-03-06T07:25:56.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 81, 2024-03-06T07:25:56.966 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 81, 2024-03-06T07:25:56.966 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 14.227486595 2024-03-06T07:25:56.966 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:25:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 -- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f06a806b500 msgr2=0x7f06a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f06a806b500 0x7f06a806d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f06b4001050 tx=0x7f06b4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 -- 172.21.15.59:0/2168401178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 msgr2=0x7f06c41561a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 0x7f06c41561a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f06c0000c00 tx=0x7f06c0001120 comp rx=0 tx=0).stop 2024-03-06T07:25:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 -- 172.21.15.59:0/2168401178 shutdown_connections 2024-03-06T07:25:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f06a806b500 0x7f06a806d9c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06c40cc4b0 0x7f06c40cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06c40c9790 0x7f06c40c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 --2- 172.21.15.59:0/2168401178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06b00039b0 0x7f06c41561a0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 -- 172.21.15.59:0/2168401178 >> 172.21.15.59:0/2168401178 conn(0x7f06c40d3ce0 msgr2=0x7f06c406d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 -- 172.21.15.59:0/2168401178 shutdown_connections 2024-03-06T07:25:56.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:56.959+0000 7f06cbcaf640 1 -- 172.21.15.59:0/2168401178 wait complete. 2024-03-06T07:25:56.971 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:25:57.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.070+0000 7ff26020e640 1 Processor -- start 2024-03-06T07:25:57.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.071+0000 7ff26020e640 1 -- start start 2024-03-06T07:25:57.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.071+0000 7ff26020e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581597c0 0x7ff25815cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.071+0000 7ff26020e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff25806b6c0 0x7ff2580d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.071+0000 7ff26020e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2580cd440 con 0x7ff25806f290 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.071+0000 7ff26020e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff258159ba0 con 0x7ff25806b6c0 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.071+0000 7ff26020e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25815c7d0 con 0x7ff2581597c0 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.071+0000 7ff25d7a9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff25806f290 0x7ff25806afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60606/0 (socket says 172.21.15.59:60606) 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.071+0000 7ff25d7a9640 1 -- 172.21.15.59:0/823681554 learned_addr learned my addr 172.21.15.59:0/823681554 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172309879 0 0) 0x7ff2580cd440 con 0x7ff25806f290 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25dfaa640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581597c0 0x7ff25815cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25e7ab640 1 --2- 172.21.15.59:0/823681554 >> v2:172.21.15.164:3300/0 conn(0x7ff25806b6c0 0x7ff2580d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff244003520 con 0x7ff25806f290 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3845638436 0 0) 0x7ff244003520 con 0x7ff25806f290 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581597c0 msgr2=0x7ff25815cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581597c0 0x7ff25815cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 >> v2:172.21.15.164:3300/0 conn(0x7ff25806b6c0 msgr2=0x7ff2580d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 --2- 172.21.15.59:0/823681554 >> v2:172.21.15.164:3300/0 conn(0x7ff25806b6c0 0x7ff2580d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2580cd720 con 0x7ff25806f290 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7ff248001670 con 0x7ff25806f290 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 >> v1:172.21.15.59:6789/0 conn(0x7ff25806f290 legacy=0x7ff25806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2440039b0 0x7ff244003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581597c0 0x7ff244004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff244004d40 0x7ff2440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25815c7d0 con 0x7ff2440039b0 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24400b9c0 con 0x7ff244004d40 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24400bb40 con 0x7ff2581597c0 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25dfaa640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2440039b0 0x7ff244003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25d7a9640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581597c0 0x7ff244004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.072+0000 7ff25e7ab640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff244004d40 0x7ff2440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.073+0000 7ff25dfaa640 1 -- 172.21.15.59:0/823681554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581597c0 msgr2=0x7ff244004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.073+0000 7ff25dfaa640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581597c0 0x7ff244004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.073+0000 7ff25dfaa640 1 -- 172.21.15.59:0/823681554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff244004d40 msgr2=0x7ff2440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:57.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.073+0000 7ff25dfaa640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff244004d40 0x7ff2440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.073+0000 7ff25dfaa640 1 -- 172.21.15.59:0/823681554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff244003520 con 0x7ff2440039b0 2024-03-06T07:25:57.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.074+0000 7ff25dfaa640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2440039b0 0x7ff244003df0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7ff254009ac0 tx=0x7ff244005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa3293d8344a58dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.074+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff254039070 con 0x7ff2440039b0 2024-03-06T07:25:57.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.074+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25402c6b0 con 0x7ff2440039b0 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.074+0000 7ff25cfa8640 1 -- 172.21.15.59:0/823681554 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff254009690 con 0x7ff2440039b0 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.074+0000 7ff26020e640 1 -- 172.21.15.59:0/823681554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2440039b0 msgr2=0x7ff244003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.074+0000 7ff26020e640 1 --2- 172.21.15.59:0/823681554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2440039b0 0x7ff244003df0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7ff254009ac0 tx=0x7ff244005830 comp rx=0 tx=0).stop 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.074+0000 7ff25e7ab640 1 -- 172.21.15.59:0/823681554 reap_dead start 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.074+0000 7ff26020e640 1 -- 172.21.15.59:0/823681554 shutdown_connections 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.074+0000 7ff26020e640 1 -- 172.21.15.59:0/823681554 >> 172.21.15.59:0/823681554 conn(0x7ff25806c0c0 msgr2=0x7ff25806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.075+0000 7ff26020e640 1 -- 172.21.15.59:0/823681554 shutdown_connections 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.075+0000 7ff26020e640 1 -- 172.21.15.59:0/823681554 wait complete. 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.076+0000 7ff26020e640 1 Processor -- start 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.076+0000 7ff26020e640 1 -- start start 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.076+0000 7ff26020e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2440039b0 0x7ff2581f1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.076+0000 7ff26020e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff258155250 0x7ff2580b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.076+0000 7ff26020e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581472c0 0x7ff25806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff26020e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580cd0c0 con 0x7ff258155250 2024-03-06T07:25:57.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff26020e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25815c150 con 0x7ff2440039b0 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff26020e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580728d0 con 0x7ff2581472c0 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25d7a9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff258155250 0x7ff2580b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25d7a9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff258155250 0x7ff2580b25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59728/0 (socket says 172.21.15.59:59728) 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25d7a9640 1 -- 172.21.15.59:0/2568020562 learned_addr learned my addr 172.21.15.59:0/2568020562 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25e7ab640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581472c0 0x7ff25806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25dfaa640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2440039b0 0x7ff2581f1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25d7a9640 1 -- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581472c0 msgr2=0x7ff25806e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25d7a9640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581472c0 0x7ff25806e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25d7a9640 1 -- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2440039b0 msgr2=0x7ff2581f1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25d7a9640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2440039b0 0x7ff2581f1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25d7a9640 1 -- 172.21.15.59:0/2568020562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2581557f0 con 0x7ff258155250 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25dfaa640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2440039b0 0x7ff2581f1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff25d7a9640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff258155250 0x7ff2580b25b0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7ff248007310 tx=0x7ff2480077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff24e7fc640 1 -- 172.21.15.59:0/2568020562 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24800f960 con 0x7ff258155250 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff24e7fc640 1 -- 172.21.15.59:0/2568020562 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff248018070 con 0x7ff258155250 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff26020e640 1 -- 172.21.15.59:0/2568020562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff258155ad0 con 0x7ff258155250 2024-03-06T07:25:57.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff24e7fc640 1 -- 172.21.15.59:0/2568020562 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24800fd60 con 0x7ff258155250 2024-03-06T07:25:57.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.077+0000 7ff26020e640 1 -- 172.21.15.59:0/2568020562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff258156090 con 0x7ff258155250 2024-03-06T07:25:57.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.079+0000 7ff26020e640 1 -- 172.21.15.59:0/2568020562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25804f330 con 0x7ff258155250 2024-03-06T07:25:57.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.079+0000 7ff24e7fc640 1 -- 172.21.15.59:0/2568020562 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff24801d460 con 0x7ff258155250 2024-03-06T07:25:57.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.079+0000 7ff24e7fc640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff23c06b500 0x7ff23c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.079+0000 7ff24e7fc640 1 -- 172.21.15.59:0/2568020562 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff24808a1a0 con 0x7ff258155250 2024-03-06T07:25:57.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.079+0000 7ff25dfaa640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff23c06b500 0x7ff23c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.080+0000 7ff25dfaa640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff23c06b500 0x7ff23c06d9c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff254036000 tx=0x7ff254030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.083+0000 7ff24e7fc640 1 -- 172.21.15.59:0/2568020562 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2480590b0 con 0x7ff258155250 2024-03-06T07:25:57.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.344+0000 7ff26020e640 1 -- 172.21.15.59:0/2568020562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff25806c720 con 0x7ff258155250 2024-03-06T07:25:57.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.345+0000 7ff24e7fc640 1 -- 172.21.15.59:0/2568020562 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v42) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff2480589b0 con 0x7ff258155250 2024-03-06T07:25:57.347 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:25:57.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 -- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff23c06b500 msgr2=0x7ff23c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff23c06b500 0x7ff23c06d9c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff254036000 tx=0x7ff254030000 comp rx=0 tx=0).stop 2024-03-06T07:25:57.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 -- 172.21.15.59:0/2568020562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff258155250 msgr2=0x7ff2580b25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff258155250 0x7ff2580b25b0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7ff248007310 tx=0x7ff2480077d0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 -- 172.21.15.59:0/2568020562 shutdown_connections 2024-03-06T07:25:57.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff23c06b500 0x7ff23c06d9c0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2581472c0 0x7ff25806e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff258155250 0x7ff2580b25b0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 --2- 172.21.15.59:0/2568020562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2440039b0 0x7ff2581f1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 -- 172.21.15.59:0/2568020562 >> 172.21.15.59:0/2568020562 conn(0x7ff25806c0c0 msgr2=0x7ff25806d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 -- 172.21.15.59:0/2568020562 shutdown_connections 2024-03-06T07:25:57.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:57.347+0000 7ff26020e640 1 -- 172.21.15.59:0/2568020562 wait complete. 2024-03-06T07:25:57.359 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:25:57.360 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:25:57.360 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:25:57.360 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:25:57.360 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:57.360 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:25:57.360 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:25:57.360 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:25:57.360 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:57.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:57.360 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:25:57.375 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:25:57.375 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:25:57.375 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:25:57.375 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:57.375 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:25:57.375 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:57.375 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42421sec preferred_lft 42421sec 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:57.376 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:25:57.377 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:57.378 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:57.378 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:57.378 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:25:57.378 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:57.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:57.378 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:25:57.440 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:25:57.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:57.442 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:25:57.497 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:25:57.497 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:25:57.497 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:25:57.497 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:25:57.497 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:25:57.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:57.497 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:25:57.498 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:25:57.498 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:25:57.498 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:25:57.498 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:25:57.498 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:25:57.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:57.592 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:25:57.606 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:25:57.606 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:25:57.607 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:57.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:57.607 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:25:57.673 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:25:57.673 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:57.673 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:57.726 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:57.727 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:57.727 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:57.780 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:25:57.847 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:57.902 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:25:57.902 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:25:57.902 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:25:57.902 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:25:57.902 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:25:57.902 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:57.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:57.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:57.905 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:25:57.956 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:25:57.956 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:25:57.998 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:25:58.027 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:58.084 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:25:58.084 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:25:58.084 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:25:58.084 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:25:58.084 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:25:58.084 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:25:58.084 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:58.085 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:25:58.086 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:25:58.086 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:58.086 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:58.086 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:58.086 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:58.086 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:25:58.086 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:58.086 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:58.086 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:58.086 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:58.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:25:58.083+0000 7fdbf1666480 -1 init, newargv = 0x558cd3163760 newargc=15 2024-03-06T07:25:58.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:58.087 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:25:58.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[65781]: starting ceph client 2024-03-06T07:25:58.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[65781]: starting fuse 2024-03-06T07:25:58.102 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:25:58.103 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:25:58.103 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:25:58.103 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:25:58.103 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:25:58.103 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:25:58.103 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:25:58.103 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:58.103 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:25:58.103 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:25:58.103 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:25:58.103 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:25:58.103 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:25:58.104 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:25:58.226 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.65781.asok 2024-03-06T07:25:58.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:58.237 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.65781.asok 2024-03-06T07:25:58.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:58.237 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.65781.asok status 2024-03-06T07:25:58.374 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:25:58.374 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:25:58.374 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "65781", 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 7203, 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:25:58.375 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 7203 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 486864753 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 486864753 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.7203 192.168.0.1:0/486864753", 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/486864753", 2024-03-06T07:25:58.376 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:25:58.377 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 42, 2024-03-06T07:25:58.377 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 81, 2024-03-06T07:25:58.377 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:25:58.377 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:25:58.377 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:25:58.377 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:25:58.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:58.383 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:25:58.383 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:58.397 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:25:58.397 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:58.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:58.397 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:58.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:58.466 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-03-06T07:25:58.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:58.526 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-03-06T07:25:58.600 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:25:58.600 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:58.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:58.655 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi059.stderr:35+0 records in 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi059.stderr:35+0 records out 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run.smithi059.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.161189 s, 228 MB/s 2024-03-06T07:25:58.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:58.875 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-03-06T07:25:58.896 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:59.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.035+0000 7faeb915e640 1 Processor -- start 2024-03-06T07:25:59.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.035+0000 7faeb915e640 1 -- start start 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.036+0000 7faeb915e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7faeb415dfe0 0x7faeb40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.036+0000 7faeb915e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faeb40cf830 0x7faeb40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.036+0000 7faeb915e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faeb40d27f0 0x7faeb40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.036+0000 7faeb915e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb40d2c50 con 0x7faeb415dfe0 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.036+0000 7faeb915e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb40b31e0 con 0x7faeb40d27f0 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.036+0000 7faeb915e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb40b3d60 con 0x7faeb40cf830 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.036+0000 7faeb2d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7faeb415dfe0 0x7faeb40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.036+0000 7faeb2d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7faeb415dfe0 0x7faeb40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59736/0 (socket says 172.21.15.59:59736) 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.036+0000 7faeb2d9d640 1 -- 172.21.15.59:0/2489519903 learned_addr learned my addr 172.21.15.59:0/2489519903 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.037+0000 7faeabfff640 1 --2- 172.21.15.59:0/2489519903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faeb40cf830 0x7faeb40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.037+0000 7faeb359e640 1 --2- 172.21.15.59:0/2489519903 >> v2:172.21.15.164:3300/0 conn(0x7faeb40d27f0 0x7faeb40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.037+0000 7faeb2d9d640 1 -- 172.21.15.59:0/2489519903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faeb40cf830 msgr2=0x7faeb40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.037+0000 7faeb2d9d640 1 --2- 172.21.15.59:0/2489519903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faeb40cf830 0x7faeb40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.037+0000 7faeb2d9d640 1 -- 172.21.15.59:0/2489519903 >> v2:172.21.15.164:3300/0 conn(0x7faeb40d27f0 msgr2=0x7faeb40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.037+0000 7faeb2d9d640 1 --2- 172.21.15.59:0/2489519903 >> v2:172.21.15.164:3300/0 conn(0x7faeb40d27f0 0x7faeb40d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.037+0000 7faeb2d9d640 1 -- 172.21.15.59:0/2489519903 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb40b3430 con 0x7faeb415dfe0 2024-03-06T07:25:59.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.038+0000 7faeb2d9d640 1 --2- 172.21.15.59:0/2489519903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faeb415dfe0 0x7faeb40cf2f0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7faeb4067ad0 tx=0x7faea402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57c36d485fa64de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.038+0000 7faeb259c640 1 -- 172.21.15.59:0/2489519903 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4035020 con 0x7faeb415dfe0 2024-03-06T07:25:59.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.038+0000 7faeb259c640 1 -- 172.21.15.59:0/2489519903 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea4031870 con 0x7faeb415dfe0 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.038+0000 7faeb259c640 1 -- 172.21.15.59:0/2489519903 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4031c40 con 0x7faeb415dfe0 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.039+0000 7faeb915e640 1 -- 172.21.15.59:0/2489519903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faeb415dfe0 msgr2=0x7faeb40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.039+0000 7faeb915e640 1 --2- 172.21.15.59:0/2489519903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faeb415dfe0 0x7faeb40cf2f0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7faeb4067ad0 tx=0x7faea402b910 comp rx=0 tx=0).stop 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.039+0000 7faeb915e640 1 -- 172.21.15.59:0/2489519903 shutdown_connections 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.039+0000 7faeb915e640 1 --2- 172.21.15.59:0/2489519903 >> v2:172.21.15.164:3300/0 conn(0x7faeb40d27f0 0x7faeb40d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.039+0000 7faeb915e640 1 --2- 172.21.15.59:0/2489519903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faeb40cf830 0x7faeb40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.039+0000 7faeb915e640 1 --2- 172.21.15.59:0/2489519903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faeb415dfe0 0x7faeb40cf2f0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.039+0000 7faeb915e640 1 -- 172.21.15.59:0/2489519903 >> 172.21.15.59:0/2489519903 conn(0x7faeb40d3cd0 msgr2=0x7faeb40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.039+0000 7faeb915e640 1 -- 172.21.15.59:0/2489519903 shutdown_connections 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.039+0000 7faeb915e640 1 -- 172.21.15.59:0/2489519903 wait complete. 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.040+0000 7faeb915e640 1 Processor -- start 2024-03-06T07:25:59.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.040+0000 7faeb915e640 1 -- start start 2024-03-06T07:25:59.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb915e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faeb40cf830 0x7faeb4150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb915e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faeb40d27f0 0x7faeb40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb915e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faeb415dfe0 0x7faeb40cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb915e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb40d1c70 con 0x7faeb40cf830 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb915e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb40b39f0 con 0x7faeb415dfe0 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb915e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb40d1df0 con 0x7faeb40d27f0 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faeb40cf830 0x7faeb4150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faeb40cf830 0x7faeb4150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59748/0 (socket says 172.21.15.59:59748) 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb2d9d640 1 -- 172.21.15.59:0/3996917282 learned_addr learned my addr 172.21.15.59:0/3996917282 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeabfff640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faeb40d27f0 0x7faeb40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb359e640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faeb415dfe0 0x7faeb40cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb2d9d640 1 -- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faeb40d27f0 msgr2=0x7faeb40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb2d9d640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faeb40d27f0 0x7faeb40cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb2d9d640 1 -- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faeb415dfe0 msgr2=0x7faeb40cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb2d9d640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faeb415dfe0 0x7faeb40cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb2d9d640 1 -- 172.21.15.59:0/3996917282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb41514e0 con 0x7faeb40cf830 2024-03-06T07:25:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb2d9d640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faeb40cf830 0x7faeb4150fa0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7faea4000c00 tx=0x7faea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeab7fe640 1 -- 172.21.15.59:0/3996917282 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4035040 con 0x7faeb40cf830 2024-03-06T07:25:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeab7fe640 1 -- 172.21.15.59:0/3996917282 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea4045480 con 0x7faeb40cf830 2024-03-06T07:25:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeb915e640 1 -- 172.21.15.59:0/3996917282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb40cd660 con 0x7faeb40cf830 2024-03-06T07:25:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.041+0000 7faeab7fe640 1 -- 172.21.15.59:0/3996917282 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4045870 con 0x7faeb40cf830 2024-03-06T07:25:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.042+0000 7faeb915e640 1 -- 172.21.15.59:0/3996917282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb40cdc20 con 0x7faeb40cf830 2024-03-06T07:25:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.042+0000 7faeb915e640 1 -- 172.21.15.59:0/3996917282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb404f330 con 0x7faeb40cf830 2024-03-06T07:25:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.043+0000 7faeab7fe640 1 -- 172.21.15.59:0/3996917282 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faea4038020 con 0x7faeb40cf830 2024-03-06T07:25:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.043+0000 7faeab7fe640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae9406b4d0 0x7fae9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.043+0000 7faeab7fe640 1 -- 172.21.15.59:0/3996917282 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faea40ac4a0 con 0x7faeb40cf830 2024-03-06T07:25:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.044+0000 7faeabfff640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae9406b4d0 0x7fae9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.044+0000 7faeabfff640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae9406b4d0 0x7fae9406d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7faeb415a440 tx=0x7fae9c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.048+0000 7faeab7fe640 1 -- 172.21.15.59:0/3996917282 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faea407ad60 con 0x7faeb40cf830 2024-03-06T07:25:59.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.265+0000 7faeb915e640 1 -- 172.21.15.59:0/3996917282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faeb40ceac0 con 0x7faeb40cf830 2024-03-06T07:25:59.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.266+0000 7faeab7fe640 1 -- 172.21.15.59:0/3996917282 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4985 (secure 0 0 0) 0x7faea407ee00 con 0x7faeb40cf830 2024-03-06T07:25:59.268 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":42,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2865818497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2865818497},{"type":"v1","addr":"172.21.15.164:6836","nonce":2865818497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1805325807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1805325807},{"type":"v1","addr":"172.21.15.164:6837","nonce":1805325807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6999,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3345771250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3345771250},{"type":"v1","addr":"172.21.15.59:6835","nonce":3345771250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:47.090938+0000","modified":"2024-03-06T07:25:49.093664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2521311582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2521311582},{"type":"v1","addr":"172.21.15.59:6834","nonce":2521311582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7002,"qdb_cluster":[7002]},"id":6}]} 2024-03-06T07:25:59.268 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 42 2024-03-06T07:25:59.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 -- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae9406b4d0 msgr2=0x7fae9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae9406b4d0 0x7fae9406d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7faeb415a440 tx=0x7fae9c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:59.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 -- 172.21.15.59:0/3996917282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faeb40cf830 msgr2=0x7faeb4150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faeb40cf830 0x7faeb4150fa0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7faea4000c00 tx=0x7faea40046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 -- 172.21.15.59:0/3996917282 shutdown_connections 2024-03-06T07:25:59.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fae9406b4d0 0x7fae9406d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faeb415dfe0 0x7faeb40cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faeb40d27f0 0x7faeb40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 --2- 172.21.15.59:0/3996917282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faeb40cf830 0x7faeb4150fa0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 -- 172.21.15.59:0/3996917282 >> 172.21.15.59:0/3996917282 conn(0x7faeb40d3cd0 msgr2=0x7faeb406cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 -- 172.21.15.59:0/3996917282 shutdown_connections 2024-03-06T07:25:59.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.268+0000 7faeb915e640 1 -- 172.21.15.59:0/3996917282 wait complete. 2024-03-06T07:25:59.284 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:25:59.410 INFO:teuthology.orchestra.run.smithi059.stdout:{"message":"","return_code":0} 2024-03-06T07:25:59.411 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:25:59.411 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:25:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.509+0000 7fb65ce33640 1 Processor -- start 2024-03-06T07:25:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.509+0000 7fb65ce33640 1 -- start start 2024-03-06T07:25:59.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.510+0000 7fb65ce33640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6580cc920 0x7fb65815eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.510+0000 7fb65ce33640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb6580b4a50 0x7fb658150b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.510+0000 7fb65ce33640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65806e770 con 0x7fb65806e000 2024-03-06T07:25:59.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.510+0000 7fb65ce33640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb65815e660 con 0x7fb6580b4a50 2024-03-06T07:25:59.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.510+0000 7fb65ce33640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65806e490 con 0x7fb6580cc920 2024-03-06T07:25:59.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.510+0000 7fb656d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb6580b4a50 0x7fb658150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.510+0000 7fb656d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb6580b4a50 0x7fb658150b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59752/0 (socket says 172.21.15.59:59752) 2024-03-06T07:25:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.510+0000 7fb656d9d640 1 -- 172.21.15.59:0/3435445917 learned_addr learned my addr 172.21.15.59:0/3435445917 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.510+0000 7fb65659c640 1 --2- 172.21.15.59:0/3435445917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6580cc920 0x7fb65815eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.511+0000 7fb656d9d640 1 -- 172.21.15.59:0/3435445917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6580cc920 msgr2=0x7fb65815eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.511+0000 7fb656d9d640 1 --2- 172.21.15.59:0/3435445917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6580cc920 0x7fb65815eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.511+0000 7fb656d9d640 1 -- 172.21.15.59:0/3435445917 >> v1:172.21.15.164:6789/0 conn(0x7fb65806e000 legacy=0x7fb658072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.511+0000 7fb656d9d640 1 -- 172.21.15.59:0/3435445917 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65815e010 con 0x7fb6580b4a50 2024-03-06T07:25:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.511+0000 7fb656d9d640 1 --2- 172.21.15.59:0/3435445917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6580b4a50 0x7fb658150b10 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fb65815e7e0 tx=0x7fb65002cfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce25c748ec0b33b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.511+0000 7fb65559a640 1 -- 172.21.15.59:0/3435445917 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65002d970 con 0x7fb6580b4a50 2024-03-06T07:25:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.512+0000 7fb65559a640 1 -- 172.21.15.59:0/3435445917 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65002db10 con 0x7fb6580b4a50 2024-03-06T07:25:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.512+0000 7fb65559a640 1 -- 172.21.15.59:0/3435445917 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb650008500 con 0x7fb6580b4a50 2024-03-06T07:25:59.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.512+0000 7fb65ce33640 1 -- 172.21.15.59:0/3435445917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6580b4a50 msgr2=0x7fb658150b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.512+0000 7fb65ce33640 1 --2- 172.21.15.59:0/3435445917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6580b4a50 0x7fb658150b10 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fb65815e7e0 tx=0x7fb65002cfd0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.512+0000 7fb65ce33640 1 -- 172.21.15.59:0/3435445917 shutdown_connections 2024-03-06T07:25:59.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.512+0000 7fb65ce33640 1 --2- 172.21.15.59:0/3435445917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6580b4a50 0x7fb658150b10 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.512+0000 7fb65ce33640 1 --2- 172.21.15.59:0/3435445917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6580cc920 0x7fb65815eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.512+0000 7fb65ce33640 1 -- 172.21.15.59:0/3435445917 >> 172.21.15.59:0/3435445917 conn(0x7fb6580b1c80 msgr2=0x7fb6580cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.512+0000 7fb65ce33640 1 -- 172.21.15.59:0/3435445917 shutdown_connections 2024-03-06T07:25:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.512+0000 7fb65ce33640 1 -- 172.21.15.59:0/3435445917 wait complete. 2024-03-06T07:25:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.514+0000 7fb65ce33640 1 Processor -- start 2024-03-06T07:25:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.514+0000 7fb65ce33640 1 -- start start 2024-03-06T07:25:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.515+0000 7fb65ce33640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6580b4a50 0x7fb658147620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65ce33640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6580cc920 0x7fb658158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65ce33640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6581efa50 0x7fb6580b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65ce33640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580cce80 con 0x7fb6580b4a50 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65ce33640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6580ccd00 con 0x7fb6580cc920 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65ce33640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb658070940 con 0x7fb6581efa50 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6580b4a50 0x7fb658147620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6580b4a50 0x7fb658147620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59758/0 (socket says 172.21.15.59:59758) 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65659c640 1 -- 172.21.15.59:0/887924193 learned_addr learned my addr 172.21.15.59:0/887924193 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb655d9b640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6580cc920 0x7fb658158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb656d9d640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6581efa50 0x7fb6580b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65659c640 1 -- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6581efa50 msgr2=0x7fb6580b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65659c640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6581efa50 0x7fb6580b0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65659c640 1 -- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6580cc920 msgr2=0x7fb658158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65659c640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6580cc920 0x7fb658158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65659c640 1 -- 172.21.15.59:0/887924193 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6581594c0 con 0x7fb6580b4a50 2024-03-06T07:25:59.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb655d9b640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6580cc920 0x7fb658158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:59.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65659c640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6580b4a50 0x7fb658147620 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb64800d800 tx=0x7fb64800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb646ffd640 1 -- 172.21.15.59:0/887924193 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64800b6c0 con 0x7fb6580b4a50 2024-03-06T07:25:59.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb646ffd640 1 -- 172.21.15.59:0/887924193 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb648019070 con 0x7fb6580b4a50 2024-03-06T07:25:59.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb646ffd640 1 -- 172.21.15.59:0/887924193 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64800ba80 con 0x7fb6580b4a50 2024-03-06T07:25:59.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65ce33640 1 -- 172.21.15.59:0/887924193 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb658159760 con 0x7fb6580b4a50 2024-03-06T07:25:59.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.516+0000 7fb65ce33640 1 -- 172.21.15.59:0/887924193 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb658159d20 con 0x7fb6580b4a50 2024-03-06T07:25:59.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.517+0000 7fb65ce33640 1 -- 172.21.15.59:0/887924193 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb658155040 con 0x7fb6580b4a50 2024-03-06T07:25:59.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.518+0000 7fb646ffd640 1 -- 172.21.15.59:0/887924193 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb64800bc20 con 0x7fb6580b4a50 2024-03-06T07:25:59.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.518+0000 7fb646ffd640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb63806b4d0 0x7fb63806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.518+0000 7fb646ffd640 1 -- 172.21.15.59:0/887924193 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb64808a6b0 con 0x7fb6580b4a50 2024-03-06T07:25:59.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.519+0000 7fb655d9b640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb63806b4d0 0x7fb63806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.519+0000 7fb655d9b640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb63806b4d0 0x7fb63806d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb658147da0 tx=0x7fb64c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.523+0000 7fb646ffd640 1 -- 172.21.15.59:0/887924193 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb648059530 con 0x7fb6580b4a50 2024-03-06T07:25:59.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.776+0000 7fb65ce33640 1 -- 172.21.15.59:0/887924193 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fb65804f330 con 0x7fb6580b4a50 2024-03-06T07:25:59.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.778+0000 7fb646ffd640 1 -- 172.21.15.59:0/887924193 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1651 (secure 0 0 0) 0x7fb648058e30 con 0x7fb6580b4a50 2024-03-06T07:25:59.779 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: "stats": { 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: "total_bytes": 773094113280, 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: "total_avail_bytes": 772829437952, 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_bytes": 264675328, 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_raw_bytes": 264675328, 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_raw_ratio": 0.0003423584857955575, 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: "num_osds": 8, 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: "num_per_pool_osds": 8, 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: "stats_by_class": { 2024-03-06T07:25:59.780 INFO:teuthology.orchestra.run.smithi059.stdout: "ssd": { 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: "total_bytes": 773094113280, 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: "total_avail_bytes": 772829437952, 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_bytes": 264675328, 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_raw_bytes": 264675328, 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: "total_used_raw_ratio": 0.0003423584857955575 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: "pools": [ 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: "name": ".mgr", 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 1, 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: "stats": { 2024-03-06T07:25:59.781 INFO:teuthology.orchestra.run.smithi059.stdout: "stored": 590368, 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: "objects": 2, 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: "kb_used": 1160, 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: "bytes_used": 1187840, 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: "percent_used": 1.6179307067432092e-06, 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: "max_avail": 367085584384 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs_metadata", 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 12, 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: "stats": { 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: "stored": 2410, 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: "objects": 22, 2024-03-06T07:25:59.782 INFO:teuthology.orchestra.run.smithi059.stdout: "kb_used": 64, 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "bytes_used": 65536, 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "percent_used": 8.9265277836148016e-08, 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "max_avail": 367085584384 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cephfs_data", 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 13, 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "stats": { 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "stored": 0, 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "objects": 0, 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "kb_used": 0, 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "bytes_used": 0, 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "percent_used": 0, 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: "max_avail": 367085584384 2024-03-06T07:25:59.783 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:59.784 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:25:59.784 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-06T07:25:59.784 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:25:59.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 -- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb63806b4d0 msgr2=0x7fb63806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb63806b4d0 0x7fb63806d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb658147da0 tx=0x7fb64c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:59.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 -- 172.21.15.59:0/887924193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6580b4a50 msgr2=0x7fb658147620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6580b4a50 0x7fb658147620 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fb64800d800 tx=0x7fb64800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 -- 172.21.15.59:0/887924193 shutdown_connections 2024-03-06T07:25:59.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb63806b4d0 0x7fb63806d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6581efa50 0x7fb6580b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6580cc920 0x7fb658158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 --2- 172.21.15.59:0/887924193 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6580b4a50 0x7fb658147620 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 -- 172.21.15.59:0/887924193 >> 172.21.15.59:0/887924193 conn(0x7fb6580b1c80 msgr2=0x7fb6580b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 -- 172.21.15.59:0/887924193 shutdown_connections 2024-03-06T07:25:59.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:25:59.780+0000 7fb65ce33640 1 -- 172.21.15.59:0/887924193 wait complete. 2024-03-06T07:25:59.793 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:25:59.793 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:59.793 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:59.842 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:59.889 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:59.890 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:25:59.890 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:59.905 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.043+0000 7f5ba8f16640 1 Processor -- start 2024-03-06T07:26:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.043+0000 7f5ba8f16640 1 -- start start 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.044+0000 7f5ba8f16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ba415dfe0 0x7f5ba40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.044+0000 7f5ba8f16640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5ba40d29c0 0x7f5ba40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.044+0000 7f5ba8f16640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba40b3430 con 0x7f5ba40cf830 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.044+0000 7f5ba8f16640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba40b31e0 con 0x7f5ba40d29c0 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.044+0000 7f5ba8f16640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba40b3d60 con 0x7f5ba415dfe0 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.044+0000 7f5ba2d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5ba40d29c0 0x7f5ba40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.045+0000 7f5ba2d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5ba40d29c0 0x7f5ba40b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59762/0 (socket says 172.21.15.59:59762) 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.045+0000 7f5ba2d9d640 1 -- 172.21.15.59:0/562293374 learned_addr learned my addr 172.21.15.59:0/562293374 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.045+0000 7f5ba259c640 1 --2- 172.21.15.59:0/562293374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ba415dfe0 0x7f5ba40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.045+0000 7f5ba2d9d640 1 -- 172.21.15.59:0/562293374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ba415dfe0 msgr2=0x7f5ba40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.045+0000 7f5ba2d9d640 1 --2- 172.21.15.59:0/562293374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ba415dfe0 0x7f5ba40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.045+0000 7f5ba2d9d640 1 -- 172.21.15.59:0/562293374 >> v1:172.21.15.164:6789/0 conn(0x7f5ba40cf830 legacy=0x7f5ba40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.045+0000 7f5ba2d9d640 1 -- 172.21.15.59:0/562293374 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba40b3710 con 0x7f5ba40d29c0 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba2d9d640 1 --2- 172.21.15.59:0/562293374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ba40d29c0 0x7f5ba40b41f0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009870 tx=0x7f5b9c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b01f5346f0c74af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba159a640 1 -- 172.21.15.59:0/562293374 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c034070 con 0x7f5ba40d29c0 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba159a640 1 -- 172.21.15.59:0/562293374 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9c030840 con 0x7f5ba40d29c0 2024-03-06T07:26:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba159a640 1 -- 172.21.15.59:0/562293374 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c030c10 con 0x7f5ba40d29c0 2024-03-06T07:26:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba8f16640 1 -- 172.21.15.59:0/562293374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ba40d29c0 msgr2=0x7f5ba40b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba8f16640 1 --2- 172.21.15.59:0/562293374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ba40d29c0 0x7f5ba40b41f0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009870 tx=0x7f5b9c02b980 comp rx=0 tx=0).stop 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba8f16640 1 -- 172.21.15.59:0/562293374 shutdown_connections 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba8f16640 1 --2- 172.21.15.59:0/562293374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ba40d29c0 0x7f5ba40b41f0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba8f16640 1 --2- 172.21.15.59:0/562293374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ba415dfe0 0x7f5ba40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba8f16640 1 -- 172.21.15.59:0/562293374 >> 172.21.15.59:0/562293374 conn(0x7f5ba40d3cd0 msgr2=0x7f5ba40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba8f16640 1 -- 172.21.15.59:0/562293374 shutdown_connections 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.046+0000 7f5ba8f16640 1 -- 172.21.15.59:0/562293374 wait complete. 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.048+0000 7f5ba8f16640 1 Processor -- start 2024-03-06T07:26:00.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.048+0000 7f5ba8f16640 1 -- start start 2024-03-06T07:26:00.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.048+0000 7f5ba8f16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ba40cf830 0x7f5ba40a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.048+0000 7f5ba8f16640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ba40d29c0 0x7f5ba40cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.048+0000 7f5ba8f16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ba415dfe0 0x7f5ba415be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.048+0000 7f5ba8f16640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba40d1df0 con 0x7f5ba415dfe0 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.048+0000 7f5ba8f16640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba40d1c70 con 0x7f5ba40d29c0 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.048+0000 7f5ba8f16640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba415e3c0 con 0x7f5ba40cf830 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ba415dfe0 0x7f5ba415be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ba415dfe0 0x7f5ba415be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59774/0 (socket says 172.21.15.59:59774) 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba2d9d640 1 -- 172.21.15.59:0/2735458227 learned_addr learned my addr 172.21.15.59:0/2735458227 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba259c640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ba40cf830 0x7f5ba40a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba1d9b640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ba40d29c0 0x7f5ba40cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba2d9d640 1 -- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ba40cf830 msgr2=0x7f5ba40a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba2d9d640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ba40cf830 0x7f5ba40a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba2d9d640 1 -- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ba40d29c0 msgr2=0x7f5ba40cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba2d9d640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ba40d29c0 0x7f5ba40cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba2d9d640 1 -- 172.21.15.59:0/2735458227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba40caf80 con 0x7f5ba415dfe0 2024-03-06T07:26:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba259c640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ba40cf830 0x7f5ba40a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba2d9d640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ba415dfe0 0x7f5ba415be30 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009870 tx=0x7f5b9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5b9affd640 1 -- 172.21.15.59:0/2735458227 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c048070 con 0x7f5ba415dfe0 2024-03-06T07:26:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5b9affd640 1 -- 172.21.15.59:0/2735458227 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9c030640 con 0x7f5ba415dfe0 2024-03-06T07:26:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba8f16640 1 -- 172.21.15.59:0/2735458227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba40cb260 con 0x7f5ba415dfe0 2024-03-06T07:26:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5b9affd640 1 -- 172.21.15.59:0/2735458227 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c034040 con 0x7f5ba415dfe0 2024-03-06T07:26:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.049+0000 7f5ba8f16640 1 -- 172.21.15.59:0/2735458227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba40cd7a0 con 0x7f5ba415dfe0 2024-03-06T07:26:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.050+0000 7f5ba8f16640 1 -- 172.21.15.59:0/2735458227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba404f330 con 0x7f5ba415dfe0 2024-03-06T07:26:00.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.051+0000 7f5b9affd640 1 -- 172.21.15.59:0/2735458227 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b9c04b070 con 0x7f5ba415dfe0 2024-03-06T07:26:00.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.051+0000 7f5b9affd640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b8406b4d0 0x7f5b8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.052+0000 7f5b9affd640 1 -- 172.21.15.59:0/2735458227 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b9c0abfa0 con 0x7f5ba415dfe0 2024-03-06T07:26:00.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.052+0000 7f5ba259c640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b8406b4d0 0x7f5b8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.052+0000 7f5ba259c640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b8406b4d0 0x7f5b8406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5b94009a80 tx=0x7f5b940039d0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.056+0000 7f5b9affd640 1 -- 172.21.15.59:0/2735458227 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b9c07a980 con 0x7f5ba415dfe0 2024-03-06T07:26:00.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.238+0000 7f5ba8f16640 1 -- 172.21.15.59:0/2735458227 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ba40ca380 con 0x7f5ba415dfe0 2024-03-06T07:26:00.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.239+0000 7f5b9affd640 1 -- 172.21.15.59:0/2735458227 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f5b9c07ea20 con 0x7f5ba415dfe0 2024-03-06T07:26:00.241 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:00.241 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":81,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:48.732524+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:45.368088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:46.056543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 -- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b8406b4d0 msgr2=0x7f5b8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b8406b4d0 0x7f5b8406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5b94009a80 tx=0x7f5b940039d0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 -- 172.21.15.59:0/2735458227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ba415dfe0 msgr2=0x7f5ba415be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ba415dfe0 0x7f5ba415be30 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009870 tx=0x7f5b9c004730 comp rx=0 tx=0).stop 2024-03-06T07:26:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 -- 172.21.15.59:0/2735458227 shutdown_connections 2024-03-06T07:26:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b8406b4d0 0x7f5b8406d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5ba415dfe0 0x7f5ba415be30 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ba40d29c0 0x7f5ba40cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 --2- 172.21.15.59:0/2735458227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ba40cf830 0x7f5ba40a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 -- 172.21.15.59:0/2735458227 >> 172.21.15.59:0/2735458227 conn(0x7f5ba40d3cd0 msgr2=0x7f5ba40b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 -- 172.21.15.59:0/2735458227 shutdown_connections 2024-03-06T07:26:00.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.241+0000 7f5ba8f16640 1 -- 172.21.15.59:0/2735458227 wait complete. 2024-03-06T07:26:00.253 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:45.368088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:46.056543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:00.254 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.356+0000 7f7e81981640 1 Processor -- start 2024-03-06T07:26:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.356+0000 7f7e81981640 1 -- start start 2024-03-06T07:26:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e81981640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e7c0d06e0 0x7f7e7c15fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e81981640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7e7c0cf200 0x7f7e7c0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e81981640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7c0d47f0 con 0x7f7e7c0d0ac0 2024-03-06T07:26:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e81981640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7c0ced70 con 0x7f7e7c0cf200 2024-03-06T07:26:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e81981640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0d1ab0 con 0x7f7e7c0d06e0 2024-03-06T07:26:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e7b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7e7c0cf200 0x7f7e7c0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e7a7fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7e7c0d0ac0 0x7f7e7c0b47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60608/0 (socket says 172.21.15.59:60608) 2024-03-06T07:26:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e7a7fc640 1 -- 172.21.15.59:0/3719352296 learned_addr learned my addr 172.21.15.59:0/3719352296 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e7affd640 1 --2- 172.21.15.59:0/3719352296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e7c0d06e0 0x7f7e7c15fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e7b7fe640 1 -- 172.21.15.59:0/3719352296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e7c0d06e0 msgr2=0x7f7e7c15fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e7b7fe640 1 --2- 172.21.15.59:0/3719352296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e7c0d06e0 0x7f7e7c15fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e7b7fe640 1 -- 172.21.15.59:0/3719352296 >> v1:172.21.15.59:6789/0 conn(0x7f7e7c0d0ac0 legacy=0x7f7e7c0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e7b7fe640 1 -- 172.21.15.59:0/3719352296 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c0d1d90 con 0x7f7e7c0cf200 2024-03-06T07:26:00.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.357+0000 7f7e7affd640 1 --2- 172.21.15.59:0/3719352296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e7c0d06e0 0x7f7e7c15fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:00.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.358+0000 7f7e7b7fe640 1 --2- 172.21.15.59:0/3719352296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7c0cf200 0x7f7e7c0cf690 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009d90 tx=0x7f7e7402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7dec716267def1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.358+0000 7f7e79ffb640 1 -- 172.21.15.59:0/3719352296 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e74039070 con 0x7f7e7c0cf200 2024-03-06T07:26:00.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.358+0000 7f7e79ffb640 1 -- 172.21.15.59:0/3719352296 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7402cc50 con 0x7f7e7c0cf200 2024-03-06T07:26:00.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.358+0000 7f7e79ffb640 1 -- 172.21.15.59:0/3719352296 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e74034440 con 0x7f7e7c0cf200 2024-03-06T07:26:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.359+0000 7f7e81981640 1 -- 172.21.15.59:0/3719352296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7c0cf200 msgr2=0x7f7e7c0cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.359+0000 7f7e81981640 1 --2- 172.21.15.59:0/3719352296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7c0cf200 0x7f7e7c0cf690 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009d90 tx=0x7f7e7402bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.359+0000 7f7e81981640 1 -- 172.21.15.59:0/3719352296 shutdown_connections 2024-03-06T07:26:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.359+0000 7f7e81981640 1 --2- 172.21.15.59:0/3719352296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7c0cf200 0x7f7e7c0cf690 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.359+0000 7f7e81981640 1 --2- 172.21.15.59:0/3719352296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e7c0d06e0 0x7f7e7c15fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.359+0000 7f7e81981640 1 -- 172.21.15.59:0/3719352296 >> 172.21.15.59:0/3719352296 conn(0x7f7e7c0d3cd0 msgr2=0x7f7e7c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.359+0000 7f7e81981640 1 -- 172.21.15.59:0/3719352296 shutdown_connections 2024-03-06T07:26:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.359+0000 7f7e81981640 1 -- 172.21.15.59:0/3719352296 wait complete. 2024-03-06T07:26:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.360+0000 7f7e81981640 1 Processor -- start 2024-03-06T07:26:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.360+0000 7f7e81981640 1 -- start start 2024-03-06T07:26:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e81981640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7c0cf200 0x7f7e7c14e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e81981640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e7c0d06e0 0x7f7e7c0b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e81981640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e7c0d0ac0 0x7f7e7c0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e81981640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0b44d0 con 0x7f7e7c0cf200 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e81981640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0ce990 con 0x7f7e7c0d0ac0 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e81981640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0d0f00 con 0x7f7e7c0d06e0 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7c0cf200 0x7f7e7c14e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7c0cf200 0x7f7e7c14e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59794/0 (socket says 172.21.15.59:59794) 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7affd640 1 -- 172.21.15.59:0/3670265657 learned_addr learned my addr 172.21.15.59:0/3670265657 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7b7fe640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e7c0d0ac0 0x7f7e7c0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7a7fc640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e7c0d06e0 0x7f7e7c0b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7affd640 1 -- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e7c0d06e0 msgr2=0x7f7e7c0b3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7affd640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e7c0d06e0 0x7f7e7c0b3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7affd640 1 -- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e7c0d0ac0 msgr2=0x7f7e7c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7affd640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e7c0d0ac0 0x7f7e7c0ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7affd640 1 -- 172.21.15.59:0/3670265657 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c0b6d50 con 0x7f7e7c0cf200 2024-03-06T07:26:00.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7b7fe640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e7c0d0ac0 0x7f7e7c0ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.361+0000 7f7e7affd640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7c0cf200 0x7f7e7c14e210 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c7f0 tx=0x7f7e6400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.362+0000 7f7e73fff640 1 -- 172.21.15.59:0/3670265657 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64015070 con 0x7f7e7c0cf200 2024-03-06T07:26:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.362+0000 7f7e73fff640 1 -- 172.21.15.59:0/3670265657 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e64007820 con 0x7f7e7c0cf200 2024-03-06T07:26:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.362+0000 7f7e73fff640 1 -- 172.21.15.59:0/3670265657 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64007bd0 con 0x7f7e7c0cf200 2024-03-06T07:26:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.362+0000 7f7e81981640 1 -- 172.21.15.59:0/3670265657 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7c0b7030 con 0x7f7e7c0cf200 2024-03-06T07:26:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.362+0000 7f7e81981640 1 -- 172.21.15.59:0/3670265657 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7c0b75f0 con 0x7f7e7c0cf200 2024-03-06T07:26:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.363+0000 7f7e81981640 1 -- 172.21.15.59:0/3670265657 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7c0a1c80 con 0x7f7e7c0cf200 2024-03-06T07:26:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.363+0000 7f7e73fff640 1 -- 172.21.15.59:0/3670265657 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e64018070 con 0x7f7e7c0cf200 2024-03-06T07:26:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.363+0000 7f7e73fff640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.363+0000 7f7e73fff640 1 -- 172.21.15.59:0/3670265657 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e6408a370 con 0x7f7e7c0cf200 2024-03-06T07:26:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.364+0000 7f7e7a7fc640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.364+0000 7f7e7a7fc640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c1e8580 tx=0x7f7e6c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.372+0000 7f7e73fff640 1 -- 172.21.15.59:0/3670265657 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e640591f0 con 0x7f7e7c0cf200 2024-03-06T07:26:00.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.551+0000 7f7e81981640 1 -- 172.21.15.59:0/3670265657 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e7c0ce300 con 0x7f7e7c0cf200 2024-03-06T07:26:00.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.552+0000 7f7e73fff640 1 -- 172.21.15.59:0/3670265657 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e64058af0 con 0x7f7e7c0cf200 2024-03-06T07:26:00.554 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 -- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e5c06b4d0 msgr2=0x7f7e5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c1e8580 tx=0x7f7e6c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 -- 172.21.15.59:0/3670265657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7c0cf200 msgr2=0x7f7e7c14e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7c0cf200 0x7f7e7c14e210 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c7f0 tx=0x7f7e6400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 -- 172.21.15.59:0/3670265657 shutdown_connections 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e5c06b4d0 0x7f7e5c06d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e7c0d0ac0 0x7f7e7c0ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e7c0d06e0 0x7f7e7c0b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 --2- 172.21.15.59:0/3670265657 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e7c0cf200 0x7f7e7c14e210 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 -- 172.21.15.59:0/3670265657 >> 172.21.15.59:0/3670265657 conn(0x7f7e7c0d3cd0 msgr2=0x7f7e7c15eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 -- 172.21.15.59:0/3670265657 shutdown_connections 2024-03-06T07:26:00.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.554+0000 7f7e81981640 1 -- 172.21.15.59:0/3670265657 wait complete. 2024-03-06T07:26:00.567 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:00.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.669+0000 7f9cbed1e640 1 Processor -- start 2024-03-06T07:26:00.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.669+0000 7f9cbed1e640 1 -- start start 2024-03-06T07:26:00.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.670+0000 7f9cbed1e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9cb806fd10 0x7f9cb80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.670+0000 7f9cbed1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9cb80d0af0 0x7f9cb80ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.670+0000 7f9cbed1e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb80cd1e0 con 0x7f9cb815c050 2024-03-06T07:26:00.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.670+0000 7f9cbed1e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb80cd4b0 con 0x7f9cb80d0af0 2024-03-06T07:26:00.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.670+0000 7f9cbed1e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb80cd790 con 0x7f9cb806fd10 2024-03-06T07:26:00.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.670+0000 7f9cbd2bb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9cb80d0af0 0x7f9cb80ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.670+0000 7f9cbd2bb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9cb80d0af0 0x7f9cb80ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59798/0 (socket says 172.21.15.59:59798) 2024-03-06T07:26:00.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.671+0000 7f9cbd2bb640 1 -- 172.21.15.59:0/2347007057 learned_addr learned my addr 172.21.15.59:0/2347007057 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:00.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.671+0000 7f9cb7fff640 1 --2- 172.21.15.59:0/2347007057 >> v2:172.21.15.164:3300/0 conn(0x7f9cb806fd10 0x7f9cb80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.671+0000 7f9cbd2bb640 1 -- 172.21.15.59:0/2347007057 >> v2:172.21.15.164:3300/0 conn(0x7f9cb806fd10 msgr2=0x7f9cb80cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.671+0000 7f9cbd2bb640 1 --2- 172.21.15.59:0/2347007057 >> v2:172.21.15.164:3300/0 conn(0x7f9cb806fd10 0x7f9cb80cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.671+0000 7f9cbd2bb640 1 -- 172.21.15.59:0/2347007057 >> v1:172.21.15.164:6789/0 conn(0x7f9cb815c050 legacy=0x7f9cb806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.671+0000 7f9cbd2bb640 1 -- 172.21.15.59:0/2347007057 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb80cda70 con 0x7f9cb80d0af0 2024-03-06T07:26:00.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.671+0000 7f9cb7fff640 1 --2- 172.21.15.59:0/2347007057 >> v2:172.21.15.164:3300/0 conn(0x7f9cb806fd10 0x7f9cb80cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:00.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.672+0000 7f9cbd2bb640 1 --2- 172.21.15.59:0/2347007057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cb80d0af0 0x7f9cb80ccca0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80b7dc0 tx=0x7f9cb002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9303803b4ce868 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.672+0000 7f9cb77fe640 1 -- 172.21.15.59:0/2347007057 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb002ca80 con 0x7f9cb80d0af0 2024-03-06T07:26:00.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.672+0000 7f9cb77fe640 1 -- 172.21.15.59:0/2347007057 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb002cc20 con 0x7f9cb80d0af0 2024-03-06T07:26:00.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.672+0000 7f9cb77fe640 1 -- 172.21.15.59:0/2347007057 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb00343f0 con 0x7f9cb80d0af0 2024-03-06T07:26:00.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.673+0000 7f9cbed1e640 1 -- 172.21.15.59:0/2347007057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cb80d0af0 msgr2=0x7f9cb80ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.673+0000 7f9cbed1e640 1 --2- 172.21.15.59:0/2347007057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cb80d0af0 0x7f9cb80ccca0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80b7dc0 tx=0x7f9cb002bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.673+0000 7f9cbed1e640 1 -- 172.21.15.59:0/2347007057 shutdown_connections 2024-03-06T07:26:00.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.673+0000 7f9cbed1e640 1 --2- 172.21.15.59:0/2347007057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cb80d0af0 0x7f9cb80ccca0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.673+0000 7f9cbed1e640 1 --2- 172.21.15.59:0/2347007057 >> v2:172.21.15.164:3300/0 conn(0x7f9cb806fd10 0x7f9cb80cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.673+0000 7f9cbed1e640 1 -- 172.21.15.59:0/2347007057 >> 172.21.15.59:0/2347007057 conn(0x7f9cb80b3800 msgr2=0x7f9cb80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.673+0000 7f9cbed1e640 1 -- 172.21.15.59:0/2347007057 shutdown_connections 2024-03-06T07:26:00.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.673+0000 7f9cbed1e640 1 -- 172.21.15.59:0/2347007057 wait complete. 2024-03-06T07:26:00.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.674+0000 7f9cbed1e640 1 Processor -- start 2024-03-06T07:26:00.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.674+0000 7f9cbed1e640 1 -- start start 2024-03-06T07:26:00.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.675+0000 7f9cbed1e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cb806fd10 0x7f9cb8145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.675+0000 7f9cbed1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cb80d0af0 0x7f9cb8156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.675+0000 7f9cbed1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb815c050 0x7f9cb809bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.675+0000 7f9cbed1e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb8165c30 con 0x7f9cb80d0af0 2024-03-06T07:26:00.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.675+0000 7f9cbed1e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb8165ab0 con 0x7f9cb806fd10 2024-03-06T07:26:00.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.675+0000 7f9cbed1e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb8165db0 con 0x7f9cb815c050 2024-03-06T07:26:00.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.675+0000 7f9cb7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cb80d0af0 0x7f9cb8156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.675+0000 7f9cb7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cb80d0af0 0x7f9cb8156ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59810/0 (socket says 172.21.15.59:59810) 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.675+0000 7f9cb7fff640 1 -- 172.21.15.59:0/353093316 learned_addr learned my addr 172.21.15.59:0/353093316 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.675+0000 7f9cbcaba640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cb806fd10 0x7f9cb8145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.675+0000 7f9cbd2bb640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb815c050 0x7f9cb809bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.676+0000 7f9cb7fff640 1 -- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb815c050 msgr2=0x7f9cb809bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.676+0000 7f9cb7fff640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb815c050 0x7f9cb809bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.676+0000 7f9cb7fff640 1 -- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cb806fd10 msgr2=0x7f9cb8145730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.676+0000 7f9cb7fff640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cb806fd10 0x7f9cb8145730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.676+0000 7f9cb7fff640 1 -- 172.21.15.59:0/353093316 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb80b5c10 con 0x7f9cb80d0af0 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.676+0000 7f9cb7fff640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cb80d0af0 0x7f9cb8156ab0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400d7d0 tx=0x7f9ca400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.676+0000 7f9cb57fa640 1 -- 172.21.15.59:0/353093316 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4004030 con 0x7f9cb80d0af0 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.676+0000 7f9cb57fa640 1 -- 172.21.15.59:0/353093316 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca40041d0 con 0x7f9cb80d0af0 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.676+0000 7f9cbed1e640 1 -- 172.21.15.59:0/353093316 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb80b5ef0 con 0x7f9cb80d0af0 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.676+0000 7f9cb57fa640 1 -- 172.21.15.59:0/353093316 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40096a0 con 0x7f9cb80d0af0 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.676+0000 7f9cbed1e640 1 -- 172.21.15.59:0/353093316 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb8156ff0 con 0x7f9cb80d0af0 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.677+0000 7f9cbed1e640 1 -- 172.21.15.59:0/353093316 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb809fe60 con 0x7f9cb80d0af0 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.677+0000 7f9cb57fa640 1 -- 172.21.15.59:0/353093316 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ca4009840 con 0x7f9cb80d0af0 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.678+0000 7f9cb57fa640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c9c06b4d0 0x7f9c9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.678+0000 7f9cb57fa640 1 -- 172.21.15.59:0/353093316 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ca408a4a0 con 0x7f9cb80d0af0 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.678+0000 7f9cbcaba640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c9c06b4d0 0x7f9c9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.679+0000 7f9cbcaba640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c9c06b4d0 0x7f9c9c06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9cac000f60 tx=0x7f9cac009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.682+0000 7f9cb57fa640 1 -- 172.21.15.59:0/353093316 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ca40593b0 con 0x7f9cb80d0af0 2024-03-06T07:26:00.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.863+0000 7f9cbed1e640 1 -- 172.21.15.59:0/353093316 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9cb8068ff0 con 0x7f9cb80d0af0 2024-03-06T07:26:00.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.863+0000 7f9cb57fa640 1 -- 172.21.15.59:0/353093316 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9ca4058cb0 con 0x7f9cb80d0af0 2024-03-06T07:26:00.865 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:26:00.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 -- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c9c06b4d0 msgr2=0x7f9c9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c9c06b4d0 0x7f9c9c06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9cac000f60 tx=0x7f9cac009040 comp rx=0 tx=0).stop 2024-03-06T07:26:00.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 -- 172.21.15.59:0/353093316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cb80d0af0 msgr2=0x7f9cb8156ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cb80d0af0 0x7f9cb8156ab0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400d7d0 tx=0x7f9ca400dc90 comp rx=0 tx=0).stop 2024-03-06T07:26:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 -- 172.21.15.59:0/353093316 shutdown_connections 2024-03-06T07:26:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c9c06b4d0 0x7f9c9c06d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb815c050 0x7f9cb809bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cb80d0af0 0x7f9cb8156ab0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 --2- 172.21.15.59:0/353093316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cb806fd10 0x7f9cb8145730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 -- 172.21.15.59:0/353093316 >> 172.21.15.59:0/353093316 conn(0x7f9cb80b3800 msgr2=0x7f9cb806eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 -- 172.21.15.59:0/353093316 shutdown_connections 2024-03-06T07:26:00.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.866+0000 7f9cbed1e640 1 -- 172.21.15.59:0/353093316 wait complete. 2024-03-06T07:26:00.879 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:00.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.985+0000 7fe8bc447640 1 Processor -- start 2024-03-06T07:26:00.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.985+0000 7fe8bc447640 1 -- start start 2024-03-06T07:26:00.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8bc447640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe8b41614d0 0x7fe8b40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8bc447640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b41618b0 con 0x7fe8b40d24b0 2024-03-06T07:26:00.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8bc447640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b40cf400 con 0x7fe8b40d1910 2024-03-06T07:26:00.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8bc447640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b40cf6a0 con 0x7fe8b41614d0 2024-03-06T07:26:00.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8ba9e4640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe8b40d24b0 0x7fe8b40b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60614/0 (socket says 172.21.15.59:60614) 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8ba9e4640 1 -- 172.21.15.59:0/2322873380 learned_addr learned my addr 172.21.15.59:0/2322873380 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8ba1e3640 1 --2- 172.21.15.59:0/2322873380 >> v2:172.21.15.164:3300/0 conn(0x7fe8b41614d0 0x7fe8b40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437442275 0 0) 0x7fe8b41618b0 con 0x7fe8b40d24b0 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a0003520 con 0x7fe8b40d24b0 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133505603 0 0) 0x7fe8b40cf400 con 0x7fe8b40d1910 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b41618b0 con 0x7fe8b40d1910 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3572990657 0 0) 0x7fe8a0003520 con 0x7fe8b40d24b0 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 >> v2:172.21.15.164:3300/0 conn(0x7fe8b41614d0 msgr2=0x7fe8b40d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8b91e1640 1 --2- 172.21.15.59:0/2322873380 >> v2:172.21.15.164:3300/0 conn(0x7fe8b41614d0 0x7fe8b40d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 >> v1:172.21.15.164:6789/0 conn(0x7fe8b40d1910 legacy=0x7fe8b40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b40cf980 con 0x7fe8b40d24b0 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fe8ac0019f0 con 0x7fe8b40d24b0 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.986+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 >> v1:172.21.15.59:6789/0 conn(0x7fe8b40d24b0 legacy=0x7fe8b40b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b91e1640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8a0003b40 0x7fe8a0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b91e1640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8a0004800 0x7fe8a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b91e1640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a00051c0 0x7fe8a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b40cf6a0 con 0x7fe8a0004800 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a000c040 con 0x7fe8a00051c0 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a000c240 con 0x7fe8a0003b40 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b99e2640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8a0004800 0x7fe8a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8ba1e3640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8a0003b40 0x7fe8a0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8ba9e4640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a00051c0 0x7fe8a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b99e2640 1 -- 172.21.15.59:0/2322873380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8a0003b40 msgr2=0x7fe8a0003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b99e2640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8a0003b40 0x7fe8a0003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b99e2640 1 -- 172.21.15.59:0/2322873380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a00051c0 msgr2=0x7fe8a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b99e2640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a00051c0 0x7fe8a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8b99e2640 1 -- 172.21.15.59:0/2322873380 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a0003520 con 0x7fe8a0004800 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8ba9e4640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a00051c0 0x7fe8a000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:00.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.987+0000 7fe8ba9e4640 1 -- 172.21.15.59:0/2322873380 reap_dead start 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.988+0000 7fe8b99e2640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8a0004800 0x7fe8a0004c80 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fe8a40018b0 tx=0x7fe8a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb643e0ef6bf7156 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.988+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4037070 con 0x7fe8a0004800 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.988+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a40052d0 con 0x7fe8a0004800 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.988+0000 7fe8b91e1640 1 -- 172.21.15.59:0/2322873380 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4001500 con 0x7fe8a0004800 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.988+0000 7fe8bc447640 1 -- 172.21.15.59:0/2322873380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8a0004800 msgr2=0x7fe8a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.988+0000 7fe8bc447640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8a0004800 0x7fe8a0004c80 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fe8a40018b0 tx=0x7fe8a0005830 comp rx=0 tx=0).stop 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.989+0000 7fe8bc447640 1 -- 172.21.15.59:0/2322873380 shutdown_connections 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.989+0000 7fe8bc447640 1 --2- 172.21.15.59:0/2322873380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8a0004800 0x7fe8a0004c80 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.989+0000 7fe8bc447640 1 -- 172.21.15.59:0/2322873380 >> 172.21.15.59:0/2322873380 conn(0x7fe8b40d3d40 msgr2=0x7fe8b40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.989+0000 7fe8bc447640 1 -- 172.21.15.59:0/2322873380 shutdown_connections 2024-03-06T07:26:00.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.989+0000 7fe8bc447640 1 -- 172.21.15.59:0/2322873380 wait complete. 2024-03-06T07:26:00.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.990+0000 7fe8bc447640 1 Processor -- start 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.990+0000 7fe8bc447640 1 -- start start 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.990+0000 7fe8bc447640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a0004800 0x7fe8b40b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.990+0000 7fe8bc447640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8b40cdd40 0x7fe8b4152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.990+0000 7fe8bc447640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b40cc310 0x7fe8b40b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.990+0000 7fe8bc447640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b40d4860 con 0x7fe8b40cc310 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.990+0000 7fe8bc447640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b40b0910 con 0x7fe8a0004800 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.990+0000 7fe8bc447640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b40d49e0 con 0x7fe8b40cdd40 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8ba9e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b40cc310 0x7fe8b40b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8ba9e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b40cc310 0x7fe8b40b1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59832/0 (socket says 172.21.15.59:59832) 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8ba9e4640 1 -- 172.21.15.59:0/2140165406 learned_addr learned my addr 172.21.15.59:0/2140165406 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8b99e2640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8b40cdd40 0x7fe8b4152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8ba1e3640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a0004800 0x7fe8b40b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8ba9e4640 1 -- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8b40cdd40 msgr2=0x7fe8b4152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8ba9e4640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8b40cdd40 0x7fe8b4152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8ba9e4640 1 -- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a0004800 msgr2=0x7fe8b40b7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8ba9e4640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a0004800 0x7fe8b40b7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8ba9e4640 1 -- 172.21.15.59:0/2140165406 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b40ccca0 con 0x7fe8b40cc310 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8ba9e4640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b40cc310 0x7fe8b40b1240 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac000f80 tx=0x7fe8ac00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8aaffd640 1 -- 172.21.15.59:0/2140165406 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac00eb90 con 0x7fe8b40cc310 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8aaffd640 1 -- 172.21.15.59:0/2140165406 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ac00ed30 con 0x7fe8b40cc310 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8bc447640 1 -- 172.21.15.59:0/2140165406 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b40ccf80 con 0x7fe8b40cc310 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8aaffd640 1 -- 172.21.15.59:0/2140165406 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac014650 con 0x7fe8b40cc310 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.991+0000 7fe8bc447640 1 -- 172.21.15.59:0/2140165406 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b40cd540 con 0x7fe8b40cc310 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.992+0000 7fe8bc447640 1 -- 172.21.15.59:0/2140165406 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b4068ff0 con 0x7fe8b40cc310 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.993+0000 7fe8aaffd640 1 -- 172.21.15.59:0/2140165406 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8ac01b060 con 0x7fe8b40cc310 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.993+0000 7fe8aaffd640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe89806b4d0 0x7fe89806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.993+0000 7fe8aaffd640 1 -- 172.21.15.59:0/2140165406 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8ac089d00 con 0x7fe8b40cc310 2024-03-06T07:26:00.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.994+0000 7fe8ba1e3640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe89806b4d0 0x7fe89806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.994+0000 7fe8ba1e3640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe89806b4d0 0x7fe89806d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe8b00099a0 tx=0x7fe8b0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:00.998+0000 7fe8aaffd640 1 -- 172.21.15.59:0/2140165406 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8ac058b80 con 0x7fe8b40cc310 2024-03-06T07:26:01.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.180+0000 7fe8bc447640 1 -- 172.21.15.59:0/2140165406 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe8b40cebe0 con 0x7fe8b40cc310 2024-03-06T07:26:01.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.180+0000 7fe8aaffd640 1 -- 172.21.15.59:0/2140165406 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe8ac058480 con 0x7fe8b40cc310 2024-03-06T07:26:01.182 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:26:01.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.183+0000 7fe8bc447640 1 -- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe89806b4d0 msgr2=0x7fe89806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.183+0000 7fe8bc447640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe89806b4d0 0x7fe89806d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe8b00099a0 tx=0x7fe8b0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:01.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.183+0000 7fe8bc447640 1 -- 172.21.15.59:0/2140165406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b40cc310 msgr2=0x7fe8b40b1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.183+0000 7fe8bc447640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b40cc310 0x7fe8b40b1240 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac000f80 tx=0x7fe8ac00e530 comp rx=0 tx=0).stop 2024-03-06T07:26:01.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.183+0000 7fe8bc447640 1 -- 172.21.15.59:0/2140165406 shutdown_connections 2024-03-06T07:26:01.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.183+0000 7fe8bc447640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe89806b4d0 0x7fe89806d990 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.183+0000 7fe8bc447640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8b40cc310 0x7fe8b40b1240 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.183+0000 7fe8bc447640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe8b40cdd40 0x7fe8b4152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.183+0000 7fe8bc447640 1 --2- 172.21.15.59:0/2140165406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a0004800 0x7fe8b40b7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.183+0000 7fe8bc447640 1 -- 172.21.15.59:0/2140165406 >> 172.21.15.59:0/2140165406 conn(0x7fe8b40d3d40 msgr2=0x7fe8b406c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.184+0000 7fe8bc447640 1 -- 172.21.15.59:0/2140165406 shutdown_connections 2024-03-06T07:26:01.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.184+0000 7fe8bc447640 1 -- 172.21.15.59:0/2140165406 wait complete. 2024-03-06T07:26:01.196 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:26:01.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.296+0000 7f56e0da0640 1 Processor -- start 2024-03-06T07:26:01.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.296+0000 7f56e0da0640 1 -- start start 2024-03-06T07:26:01.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.297+0000 7f56e0da0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f56dc06b460 0x7f56dc06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.297+0000 7f56e0da0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc06b8f0 con 0x7f56dc146ac0 2024-03-06T07:26:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.297+0000 7f56e0da0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56dc0b5830 con 0x7f56dc06afb0 2024-03-06T07:26:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.297+0000 7f56e0da0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56dc0b5b10 con 0x7f56dc06b460 2024-03-06T07:26:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.297+0000 7f56db7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f56dc146ac0 0x7f56dc0b38f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60622/0 (socket says 172.21.15.59:60622) 2024-03-06T07:26:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.297+0000 7f56db7fe640 1 -- 172.21.15.59:0/4241831223 learned_addr learned my addr 172.21.15.59:0/4241831223 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.297+0000 7f56dbfff640 1 --2- 172.21.15.59:0/4241831223 >> v2:172.21.15.164:3300/0 conn(0x7f56dc06b460 0x7f56dc06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.297+0000 7f56da7fc640 1 -- 172.21.15.59:0/4241831223 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661498430 0 0) 0x7f56dc06b8f0 con 0x7f56dc146ac0 2024-03-06T07:26:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.297+0000 7f56da7fc640 1 -- 172.21.15.59:0/4241831223 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56c4003520 con 0x7f56dc146ac0 2024-03-06T07:26:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.298+0000 7f56da7fc640 1 -- 172.21.15.59:0/4241831223 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2229384343 0 0) 0x7f56c4003520 con 0x7f56dc146ac0 2024-03-06T07:26:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.298+0000 7f56dbfff640 1 -- 172.21.15.59:0/4241831223 >> v1:172.21.15.164:6789/0 conn(0x7f56dc06afb0 legacy=0x7f56dc06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.298+0000 7f56dbfff640 1 -- 172.21.15.59:0/4241831223 >> v1:172.21.15.59:6789/0 conn(0x7f56dc146ac0 legacy=0x7f56dc0b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.298+0000 7f56dbfff640 1 -- 172.21.15.59:0/4241831223 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc0724b0 con 0x7f56dc06b460 2024-03-06T07:26:01.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.299+0000 7f56dbfff640 1 --2- 172.21.15.59:0/4241831223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56dc06b460 0x7f56dc06ec00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009d90 tx=0x7f56d402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=869d1ae7ce30930a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.299+0000 7f56da7fc640 1 -- 172.21.15.59:0/4241831223 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4039070 con 0x7f56dc06b460 2024-03-06T07:26:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.299+0000 7f56da7fc640 1 -- 172.21.15.59:0/4241831223 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d402c940 con 0x7f56dc06b460 2024-03-06T07:26:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.299+0000 7f56da7fc640 1 -- 172.21.15.59:0/4241831223 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc001670 con 0x7f56dc06b460 2024-03-06T07:26:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.299+0000 7f56e0da0640 1 -- 172.21.15.59:0/4241831223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56dc06b460 msgr2=0x7f56dc06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.299+0000 7f56e0da0640 1 --2- 172.21.15.59:0/4241831223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56dc06b460 0x7f56dc06ec00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009d90 tx=0x7f56d402be10 comp rx=0 tx=0).stop 2024-03-06T07:26:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.299+0000 7f56e0da0640 1 -- 172.21.15.59:0/4241831223 shutdown_connections 2024-03-06T07:26:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.299+0000 7f56e0da0640 1 --2- 172.21.15.59:0/4241831223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56dc06b460 0x7f56dc06ec00 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.299+0000 7f56e0da0640 1 -- 172.21.15.59:0/4241831223 >> 172.21.15.59:0/4241831223 conn(0x7f56dc06c0c0 msgr2=0x7f56dc06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.299+0000 7f56e0da0640 1 -- 172.21.15.59:0/4241831223 shutdown_connections 2024-03-06T07:26:01.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.299+0000 7f56e0da0640 1 -- 172.21.15.59:0/4241831223 wait complete. 2024-03-06T07:26:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.300+0000 7f56e0da0640 1 Processor -- start 2024-03-06T07:26:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.301+0000 7f56e0da0640 1 -- start start 2024-03-06T07:26:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.301+0000 7f56e0da0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56dc06afb0 0x7f56dc15ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.301+0000 7f56e0da0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56dc06b460 0x7f56dc137f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.301+0000 7f56e0da0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56dc146ac0 0x7f56dc15fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.301+0000 7f56e0da0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc071f20 con 0x7f56dc146ac0 2024-03-06T07:26:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.301+0000 7f56e0da0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc072a70 con 0x7f56dc06b460 2024-03-06T07:26:01.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.301+0000 7f56e0da0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc0720a0 con 0x7f56dc06afb0 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.301+0000 7f56dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56dc146ac0 0x7f56dc15fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56dc146ac0 0x7f56dc15fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59840/0 (socket says 172.21.15.59:59840) 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56dbfff640 1 -- 172.21.15.59:0/4095303429 learned_addr learned my addr 172.21.15.59:0/4095303429 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56dbfff640 1 -- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56dc06afb0 msgr2=0x7f56dc15ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56daffd640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56dc06b460 0x7f56dc137f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56db7fe640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56dc06afb0 0x7f56dc15ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56dbfff640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56dc06afb0 0x7f56dc15ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56dbfff640 1 -- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56dc06b460 msgr2=0x7f56dc137f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56dbfff640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56dc06b460 0x7f56dc137f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56dbfff640 1 -- 172.21.15.59:0/4095303429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56dc1f5040 con 0x7f56dc146ac0 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56dbfff640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56dc146ac0 0x7f56dc15fe30 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f56d4037040 tx=0x7f56d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56c3fff640 1 -- 172.21.15.59:0/4095303429 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4044020 con 0x7f56dc146ac0 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56c3fff640 1 -- 172.21.15.59:0/4095303429 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d4034cf0 con 0x7f56dc146ac0 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56e0da0640 1 -- 172.21.15.59:0/4095303429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56dc1f5320 con 0x7f56dc146ac0 2024-03-06T07:26:01.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56c3fff640 1 -- 172.21.15.59:0/4095303429 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4039040 con 0x7f56dc146ac0 2024-03-06T07:26:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.302+0000 7f56e0da0640 1 -- 172.21.15.59:0/4095303429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56dc1f58e0 con 0x7f56dc146ac0 2024-03-06T07:26:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.303+0000 7f56e0da0640 1 -- 172.21.15.59:0/4095303429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56dc069f30 con 0x7f56dc146ac0 2024-03-06T07:26:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.304+0000 7f56c3fff640 1 -- 172.21.15.59:0/4095303429 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56d4047020 con 0x7f56dc146ac0 2024-03-06T07:26:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.304+0000 7f56c3fff640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56c806b4d0 0x7f56c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.304+0000 7f56c3fff640 1 -- 172.21.15.59:0/4095303429 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56d40ac9e0 con 0x7f56dc146ac0 2024-03-06T07:26:01.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.304+0000 7f56db7fe640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56c806b4d0 0x7f56c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.305+0000 7f56db7fe640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56c806b4d0 0x7f56c806d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f56cc005880 tx=0x7f56cc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.308+0000 7f56c3fff640 1 -- 172.21.15.59:0/4095303429 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56dc069f30 con 0x7f56dc146ac0 2024-03-06T07:26:01.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.517+0000 7f56e0da0640 1 -- 172.21.15.59:0/4095303429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f56dc0af950 con 0x7f56dc146ac0 2024-03-06T07:26:01.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.517+0000 7f56c3fff640 1 -- 172.21.15.59:0/4095303429 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v81) v1 ==== 80+0+0 (secure 0 0 0) 0x7f56dc0af950 con 0x7f56dc146ac0 2024-03-06T07:26:01.519 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.519+0000 7f56e0da0640 1 -- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56c806b4d0 msgr2=0x7f56c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.519+0000 7f56e0da0640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56c806b4d0 0x7f56c806d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f56cc005880 tx=0x7f56cc009040 comp rx=0 tx=0).stop 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.519+0000 7f56e0da0640 1 -- 172.21.15.59:0/4095303429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56dc146ac0 msgr2=0x7f56dc15fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.519+0000 7f56e0da0640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56dc146ac0 0x7f56dc15fe30 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f56d4037040 tx=0x7f56d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.519+0000 7f56e0da0640 1 -- 172.21.15.59:0/4095303429 shutdown_connections 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.519+0000 7f56e0da0640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56c806b4d0 0x7f56c806d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.519+0000 7f56e0da0640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56dc146ac0 0x7f56dc15fe30 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.519+0000 7f56e0da0640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56dc06b460 0x7f56dc137f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.519+0000 7f56e0da0640 1 --2- 172.21.15.59:0/4095303429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56dc06afb0 0x7f56dc15ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.519+0000 7f56e0da0640 1 -- 172.21.15.59:0/4095303429 >> 172.21.15.59:0/4095303429 conn(0x7f56dc06c0c0 msgr2=0x7f56dc141e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.520+0000 7f56e0da0640 1 -- 172.21.15.59:0/4095303429 shutdown_connections 2024-03-06T07:26:01.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.520+0000 7f56e0da0640 1 -- 172.21.15.59:0/4095303429 wait complete. 2024-03-06T07:26:01.532 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:26:01.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.532 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:01.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[65781]: fuse finished with error 0 and tester_r 0 2024-03-06T07:26:01.559 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:26:01.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.559 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:26:01.675 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:26:01.676 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:01.694 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:01.694 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:26:01.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.695 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:26:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:26:01.752 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:26:01.752 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:01.769 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:01.769 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:26:01.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.769 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:26:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:01.824 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:26:01.824 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:26:01.825 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:01.842 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:01.842 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:26:01.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:01.842 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:26:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:26:01.899 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.999+0000 7f699ad93640 1 Processor -- start 2024-03-06T07:26:02.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:01.999+0000 7f699ad93640 1 -- start start 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.000+0000 7f699ad93640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f69941515e0 0x7f69940cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.000+0000 7f699ad93640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6994071f40 con 0x7f69940cd7a0 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.000+0000 7f699ad93640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699415b8e0 con 0x7f699415b430 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.000+0000 7f699ad93640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6994070a70 con 0x7f69941515e0 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.000+0000 7f6999330640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f69940cd7a0 0x7f69940cdb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60624/0 (socket says 172.21.15.59:60624) 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.000+0000 7f6999330640 1 -- 172.21.15.59:0/1991285153 learned_addr learned my addr 172.21.15.59:0/1991285153 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f6998b2f640 1 --2- 172.21.15.59:0/1991285153 >> v2:172.21.15.164:3300/0 conn(0x7f69941515e0 0x7f69940cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255261129 0 0) 0x7f6994071f40 con 0x7f69940cd7a0 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f697c004630 con 0x7f69940cd7a0 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4094906535 0 0) 0x7f697c004630 con 0x7f69940cd7a0 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 >> v2:172.21.15.164:3300/0 conn(0x7f69941515e0 msgr2=0x7f69940cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 --2- 172.21.15.59:0/1991285153 >> v2:172.21.15.164:3300/0 conn(0x7f69941515e0 0x7f69940cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 >> v1:172.21.15.164:6789/0 conn(0x7f699415b430 legacy=0x7f6994071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6994070d50 con 0x7f69940cd7a0 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f698c001a10 con 0x7f69940cd7a0 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 >> v1:172.21.15.59:6789/0 conn(0x7f69940cd7a0 legacy=0x7f69940cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 --2- 172.21.15.59:0/1991285153 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f697c004ac0 0x7f697c004ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 --2- 172.21.15.59:0/1991285153 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f697c0054c0 0x7f697c005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 --2- 172.21.15.59:0/1991285153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 0x7f697c00cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6994070a70 con 0x7f697c006190 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c00d0f0 con 0x7f697c004ac0 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697c00d2f0 con 0x7f697c0054c0 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f6999330640 1 --2- 172.21.15.59:0/1991285153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 0x7f697c00cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f6998b2f640 1 --2- 172.21.15.59:0/1991285153 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f697c004ac0 0x7f697c004ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f6993fff640 1 --2- 172.21.15.59:0/1991285153 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f697c0054c0 0x7f697c005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f6999330640 1 -- 172.21.15.59:0/1991285153 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f697c0054c0 msgr2=0x7f697c005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f6999330640 1 --2- 172.21.15.59:0/1991285153 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f697c0054c0 0x7f697c005990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f6999330640 1 -- 172.21.15.59:0/1991285153 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f697c004ac0 msgr2=0x7f697c004ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f6999330640 1 --2- 172.21.15.59:0/1991285153 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f697c004ac0 0x7f697c004ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.001+0000 7f6999330640 1 -- 172.21.15.59:0/1991285153 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c004630 con 0x7f697c006190 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.002+0000 7f6999330640 1 -- 172.21.15.59:0/1991285153 reap_dead start 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.002+0000 7f6999330640 1 --2- 172.21.15.59:0/1991285153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 0x7f697c00cad0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f698c00a850 tx=0x7f697c006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a105c9a081e40f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.002+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c00a4a0 con 0x7f697c006190 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.002+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c034630 con 0x7f697c006190 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.002+0000 7f69937fe640 1 -- 172.21.15.59:0/1991285153 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c0349e0 con 0x7f697c006190 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.003+0000 7f699ad93640 1 -- 172.21.15.59:0/1991285153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 msgr2=0x7f697c00cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.003+0000 7f699ad93640 1 --2- 172.21.15.59:0/1991285153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 0x7f697c00cad0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f698c00a850 tx=0x7f697c006940 comp rx=0 tx=0).stop 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.003+0000 7f699ad93640 1 -- 172.21.15.59:0/1991285153 shutdown_connections 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.003+0000 7f699ad93640 1 --2- 172.21.15.59:0/1991285153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 0x7f697c00cad0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.003+0000 7f699ad93640 1 -- 172.21.15.59:0/1991285153 >> 172.21.15.59:0/1991285153 conn(0x7f69940cf8a0 msgr2=0x7f69940cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.003+0000 7f699ad93640 1 -- 172.21.15.59:0/1991285153 shutdown_connections 2024-03-06T07:26:02.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.003+0000 7f699ad93640 1 -- 172.21.15.59:0/1991285153 wait complete. 2024-03-06T07:26:02.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.004+0000 7f699ad93640 1 Processor -- start 2024-03-06T07:26:02.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.004+0000 7f699ad93640 1 -- start start 2024-03-06T07:26:02.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f699ad93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 0x7f6994143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f699ad93640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f699415d8a0 0x7f699414deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f699ad93640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f699414bbf0 0x7f699414c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f699ad93640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69940cce40 con 0x7f697c006190 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f699ad93640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6994071310 con 0x7f699414bbf0 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f699ad93640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69940ccfc0 con 0x7f699415d8a0 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f6998b2f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 0x7f6994143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f6998b2f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 0x7f6994143370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59856/0 (socket says 172.21.15.59:59856) 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f6998b2f640 1 -- 172.21.15.59:0/2750220476 learned_addr learned my addr 172.21.15.59:0/2750220476 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f6998b2f640 1 -- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f699415d8a0 msgr2=0x7f699414deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f6998b2f640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f699415d8a0 0x7f699414deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f6998b2f640 1 -- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f699414bbf0 msgr2=0x7f699414c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f6999330640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f699414bbf0 0x7f699414c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f6998b2f640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f699414bbf0 0x7f699414c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f6998b2f640 1 -- 172.21.15.59:0/2750220476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699415dd80 con 0x7f697c006190 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.005+0000 7f6998b2f640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 0x7f6994143370 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f69840118e0 tx=0x7f6984011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.006+0000 7f69917fa640 1 -- 172.21.15.59:0/2750220476 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698400faf0 con 0x7f697c006190 2024-03-06T07:26:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.006+0000 7f69917fa640 1 -- 172.21.15.59:0/2750220476 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698400fc90 con 0x7f697c006190 2024-03-06T07:26:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.006+0000 7f699ad93640 1 -- 172.21.15.59:0/2750220476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699415e060 con 0x7f697c006190 2024-03-06T07:26:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.006+0000 7f69917fa640 1 -- 172.21.15.59:0/2750220476 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6984014440 con 0x7f697c006190 2024-03-06T07:26:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.006+0000 7f699ad93640 1 -- 172.21.15.59:0/2750220476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699415e620 con 0x7f697c006190 2024-03-06T07:26:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.007+0000 7f699ad93640 1 -- 172.21.15.59:0/2750220476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6994068840 con 0x7f697c006190 2024-03-06T07:26:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.007+0000 7f69917fa640 1 -- 172.21.15.59:0/2750220476 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6984012040 con 0x7f697c006190 2024-03-06T07:26:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.008+0000 7f69917fa640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f698006b500 0x7f698006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.008+0000 7f69917fa640 1 -- 172.21.15.59:0/2750220476 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f698408a910 con 0x7f697c006190 2024-03-06T07:26:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.008+0000 7f6993fff640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f698006b500 0x7f698006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.008+0000 7f6993fff640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f698006b500 0x7f698006d9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6994151a90 tx=0x7f6988009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.012+0000 7f69917fa640 1 -- 172.21.15.59:0/2750220476 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6984019030 con 0x7f697c006190 2024-03-06T07:26:02.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.230+0000 7f699ad93640 1 -- 172.21.15.59:0/2750220476 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f699404f330 con 0x7f697c006190 2024-03-06T07:26:02.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.230+0000 7f69917fa640 1 -- 172.21.15.59:0/2750220476 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f69840591d0 con 0x7f697c006190 2024-03-06T07:26:02.233 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:02.233 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":42,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2865818497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2865818497},{"type":"v1","addr":"172.21.15.164:6836","nonce":2865818497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1805325807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1805325807},{"type":"v1","addr":"172.21.15.164:6837","nonce":1805325807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6999,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3345771250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3345771250},{"type":"v1","addr":"172.21.15.59:6835","nonce":3345771250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:47.090938+0000","modified":"2024-03-06T07:25:49.093664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2521311582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2521311582},{"type":"v1","addr":"172.21.15.59:6834","nonce":2521311582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7002,"qdb_cluster":[7002]},"id":6}]} 2024-03-06T07:26:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 42 2024-03-06T07:26:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 -- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f698006b500 msgr2=0x7f698006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f698006b500 0x7f698006d9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6994151a90 tx=0x7f6988009040 comp rx=0 tx=0).stop 2024-03-06T07:26:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 -- 172.21.15.59:0/2750220476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 msgr2=0x7f6994143370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 0x7f6994143370 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f69840118e0 tx=0x7f6984011da0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 -- 172.21.15.59:0/2750220476 shutdown_connections 2024-03-06T07:26:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f698006b500 0x7f698006d9c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f699414bbf0 0x7f699414c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f699415d8a0 0x7f699414deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 --2- 172.21.15.59:0/2750220476 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f697c006190 0x7f6994143370 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 -- 172.21.15.59:0/2750220476 >> 172.21.15.59:0/2750220476 conn(0x7f69940cf8a0 msgr2=0x7f69940cebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 -- 172.21.15.59:0/2750220476 shutdown_connections 2024-03-06T07:26:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.232+0000 7f699ad93640 1 -- 172.21.15.59:0/2750220476 wait complete. 2024-03-06T07:26:02.248 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.347+0000 7f87473bc640 1 Processor -- start 2024-03-06T07:26:02.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.347+0000 7f87473bc640 1 -- start start 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.348+0000 7f87473bc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f87400cfff0 0x7f87400d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.348+0000 7f87473bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87400cc8b0 0x7f87400ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.348+0000 7f87473bc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8740161980 con 0x7f874015bb20 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.348+0000 7f87473bc640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87400d2500 con 0x7f87400cfff0 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.348+0000 7f87473bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8740161c60 con 0x7f87400cc8b0 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.349+0000 7f8744957640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f87400cfff0 0x7f87400d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.349+0000 7f8744957640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f87400cfff0 0x7f87400d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59872/0 (socket says 172.21.15.59:59872) 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.349+0000 7f8744957640 1 -- 172.21.15.59:0/1113796339 learned_addr learned my addr 172.21.15.59:0/1113796339 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.349+0000 7f8745959640 1 --2- 172.21.15.59:0/1113796339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87400cc8b0 0x7f87400ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.349+0000 7f8744957640 1 -- 172.21.15.59:0/1113796339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87400cc8b0 msgr2=0x7f87400ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.349+0000 7f8744957640 1 --2- 172.21.15.59:0/1113796339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87400cc8b0 0x7f87400ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.349+0000 7f8744957640 1 -- 172.21.15.59:0/1113796339 >> v1:172.21.15.164:6789/0 conn(0x7f874015bb20 legacy=0x7f87400d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.349+0000 7f8744957640 1 -- 172.21.15.59:0/1113796339 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8740161f40 con 0x7f87400cfff0 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.350+0000 7f8744957640 1 --2- 172.21.15.59:0/1113796339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87400cfff0 0x7f87400d04a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f8734009870 tx=0x7f873402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546945b3451a5d92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.350+0000 7f873ffff640 1 -- 172.21.15.59:0/1113796339 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8734034070 con 0x7f87400cfff0 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.350+0000 7f873ffff640 1 -- 172.21.15.59:0/1113796339 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8734030840 con 0x7f87400cfff0 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.350+0000 7f873ffff640 1 -- 172.21.15.59:0/1113796339 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8734030c10 con 0x7f87400cfff0 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.350+0000 7f87473bc640 1 -- 172.21.15.59:0/1113796339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87400cfff0 msgr2=0x7f87400d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.350+0000 7f87473bc640 1 --2- 172.21.15.59:0/1113796339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87400cfff0 0x7f87400d04a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f8734009870 tx=0x7f873402b980 comp rx=0 tx=0).stop 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.350+0000 7f87473bc640 1 -- 172.21.15.59:0/1113796339 shutdown_connections 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.350+0000 7f87473bc640 1 --2- 172.21.15.59:0/1113796339 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87400cc8b0 0x7f87400ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.350+0000 7f87473bc640 1 --2- 172.21.15.59:0/1113796339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87400cfff0 0x7f87400d04a0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.350+0000 7f87473bc640 1 -- 172.21.15.59:0/1113796339 >> 172.21.15.59:0/1113796339 conn(0x7f874009a9f0 msgr2=0x7f874009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.351+0000 7f87473bc640 1 -- 172.21.15.59:0/1113796339 shutdown_connections 2024-03-06T07:26:02.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.351+0000 7f87473bc640 1 -- 172.21.15.59:0/1113796339 wait complete. 2024-03-06T07:26:02.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.352+0000 7f87473bc640 1 Processor -- start 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.352+0000 7f87473bc640 1 -- start start 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.352+0000 7f87473bc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87400cc8b0 0x7f8740150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.352+0000 7f87473bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87400cfff0 0x7f87400ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.352+0000 7f87473bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874015bb20 0x7f874015eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.352+0000 7f87473bc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87400d1980 con 0x7f874015bb20 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.352+0000 7f87473bc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8740162500 con 0x7f87400cc8b0 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.352+0000 7f87473bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87400d1b00 con 0x7f87400cfff0 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8745959640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874015bb20 0x7f874015eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8745959640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874015bb20 0x7f874015eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59886/0 (socket says 172.21.15.59:59886) 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8745959640 1 -- 172.21.15.59:0/549033443 learned_addr learned my addr 172.21.15.59:0/549033443 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8745158640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87400cc8b0 0x7f8740150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8744957640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87400cfff0 0x7f87400ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8745959640 1 -- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87400cfff0 msgr2=0x7f87400ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8745959640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87400cfff0 0x7f87400ca7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8745959640 1 -- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87400cc8b0 msgr2=0x7f8740150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8745959640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87400cc8b0 0x7f8740150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8745959640 1 -- 172.21.15.59:0/549033443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8740151490 con 0x7f874015bb20 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8745158640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87400cc8b0 0x7f8740150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f8745959640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874015bb20 0x7f874015eaa0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f8738007e60 tx=0x7f873800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f873dffb640 1 -- 172.21.15.59:0/549033443 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738014420 con 0x7f874015bb20 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f873dffb640 1 -- 172.21.15.59:0/549033443 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8738019070 con 0x7f874015bb20 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f873dffb640 1 -- 172.21.15.59:0/549033443 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8738014820 con 0x7f874015bb20 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f87473bc640 1 -- 172.21.15.59:0/549033443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87400c72b0 con 0x7f874015bb20 2024-03-06T07:26:02.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.353+0000 7f87473bc640 1 -- 172.21.15.59:0/549033443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87400c5720 con 0x7f874015bb20 2024-03-06T07:26:02.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.355+0000 7f87473bc640 1 -- 172.21.15.59:0/549033443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874004f330 con 0x7f874015bb20 2024-03-06T07:26:02.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.355+0000 7f873dffb640 1 -- 172.21.15.59:0/549033443 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87380149c0 con 0x7f874015bb20 2024-03-06T07:26:02.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.355+0000 7f873dffb640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.355+0000 7f873dffb640 1 -- 172.21.15.59:0/549033443 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f873808a9d0 con 0x7f874015bb20 2024-03-06T07:26:02.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.356+0000 7f8745158640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.356+0000 7f8745158640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f872c06b4d0 0x7f872c06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8730000f60 tx=0x7f8730009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.360+0000 7f873dffb640 1 -- 172.21.15.59:0/549033443 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8740151490 con 0x7f874015bb20 2024-03-06T07:26:02.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.539+0000 7f87473bc640 1 -- 172.21.15.59:0/549033443 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87400b7820 con 0x7f874015bb20 2024-03-06T07:26:02.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.540+0000 7f873dffb640 1 -- 172.21.15.59:0/549033443 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f87380598e0 con 0x7f874015bb20 2024-03-06T07:26:02.542 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:02.542 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":81,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:48.732524+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:45.368088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:46.056543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:02.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 -- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f872c06b4d0 msgr2=0x7f872c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f872c06b4d0 0x7f872c06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8730000f60 tx=0x7f8730009040 comp rx=0 tx=0).stop 2024-03-06T07:26:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 -- 172.21.15.59:0/549033443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874015bb20 msgr2=0x7f874015eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874015bb20 0x7f874015eaa0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f8738007e60 tx=0x7f873800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 -- 172.21.15.59:0/549033443 shutdown_connections 2024-03-06T07:26:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f874015bb20 0x7f874015eaa0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87400cfff0 0x7f87400ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 --2- 172.21.15.59:0/549033443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87400cc8b0 0x7f8740150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 -- 172.21.15.59:0/549033443 >> 172.21.15.59:0/549033443 conn(0x7f874009a9f0 msgr2=0x7f87400ae9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 -- 172.21.15.59:0/549033443 shutdown_connections 2024-03-06T07:26:02.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.542+0000 7f87473bc640 1 -- 172.21.15.59:0/549033443 wait complete. 2024-03-06T07:26:02.555 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:45.368088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:46.056543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:02.556 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:02.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.657+0000 7f22c9d8a640 1 Processor -- start 2024-03-06T07:26:02.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.657+0000 7f22c9d8a640 1 -- start start 2024-03-06T07:26:02.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.658+0000 7f22c9d8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c40d14a0 0x7f22c415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.658+0000 7f22c9d8a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c40cfa50 con 0x7f22c40d10c0 2024-03-06T07:26:02.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.658+0000 7f22c9d8a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c40b45c0 con 0x7f22c415eb00 2024-03-06T07:26:02.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.658+0000 7f22c9d8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c415c050 con 0x7f22c40d14a0 2024-03-06T07:26:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.658+0000 7f22c37fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f22c40d10c0 0x7f22c40d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60640/0 (socket says 172.21.15.59:60640) 2024-03-06T07:26:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.658+0000 7f22c37fe640 1 -- 172.21.15.59:0/272708524 learned_addr learned my addr 172.21.15.59:0/272708524 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291646302 0 0) 0x7f22c40cfa50 con 0x7f22c40d10c0 2024-03-06T07:26:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c2ffd640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c40d14a0 0x7f22c415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ac003520 con 0x7f22c40d10c0 2024-03-06T07:26:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2347838741 0 0) 0x7f22ac003520 con 0x7f22c40d10c0 2024-03-06T07:26:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c40d14a0 msgr2=0x7f22c415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c40d14a0 0x7f22c415e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 >> v1:172.21.15.164:6789/0 conn(0x7f22c415eb00 legacy=0x7f22c40cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c415f160 con 0x7f22c40d10c0 2024-03-06T07:26:02.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f22b4001670 con 0x7f22c40d10c0 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 >> v1:172.21.15.59:6789/0 conn(0x7f22c40d10c0 legacy=0x7f22c40d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c40d14a0 0x7f22ac0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22ac003f60 0x7f22ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ac004d40 0x7f22ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac00ba70 con 0x7f22ac003f60 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac00bc20 con 0x7f22ac004d40 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22ac00be20 con 0x7f22c40d14a0 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c2ffd640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22ac003f60 0x7f22ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c37fe640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c40d14a0 0x7f22ac0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.659+0000 7f22c3fff640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ac004d40 0x7f22ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.660+0000 7f22c2ffd640 1 -- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c40d14a0 msgr2=0x7f22ac0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.660+0000 7f22c2ffd640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c40d14a0 0x7f22ac0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.660+0000 7f22c2ffd640 1 -- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ac004d40 msgr2=0x7f22ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.660+0000 7f22c2ffd640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ac004d40 0x7f22ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.660+0000 7f22c2ffd640 1 -- 172.21.15.59:0/272708524 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac003520 con 0x7f22ac003f60 2024-03-06T07:26:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.660+0000 7f22c3fff640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ac004d40 0x7f22ac0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.660+0000 7f22c2ffd640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22ac003f60 0x7f22ac004800 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f22b8009ac0 tx=0x7f22ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f35fb23654cafa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.660+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b8039070 con 0x7f22ac003f60 2024-03-06T07:26:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.660+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b802c6b0 con 0x7f22ac003f60 2024-03-06T07:26:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.660+0000 7f22c27fc640 1 -- 172.21.15.59:0/272708524 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b8009690 con 0x7f22ac003f60 2024-03-06T07:26:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.661+0000 7f22c9d8a640 1 -- 172.21.15.59:0/272708524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22ac003f60 msgr2=0x7f22ac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.661+0000 7f22c9d8a640 1 --2- 172.21.15.59:0/272708524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22ac003f60 0x7f22ac004800 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f22b8009ac0 tx=0x7f22ac005830 comp rx=0 tx=0).stop 2024-03-06T07:26:02.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.661+0000 7f22c3fff640 1 -- 172.21.15.59:0/272708524 reap_dead start 2024-03-06T07:26:02.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.661+0000 7f22c9d8a640 1 -- 172.21.15.59:0/272708524 shutdown_connections 2024-03-06T07:26:02.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.661+0000 7f22c9d8a640 1 -- 172.21.15.59:0/272708524 >> 172.21.15.59:0/272708524 conn(0x7f22c40b3800 msgr2=0x7f22c40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.661+0000 7f22c9d8a640 1 -- 172.21.15.59:0/272708524 shutdown_connections 2024-03-06T07:26:02.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.661+0000 7f22c9d8a640 1 -- 172.21.15.59:0/272708524 wait complete. 2024-03-06T07:26:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.662+0000 7f22c9d8a640 1 Processor -- start 2024-03-06T07:26:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.662+0000 7f22c9d8a640 1 -- start start 2024-03-06T07:26:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c9d8a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ac003f60 0x7f22c41f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c9d8a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c409f7c0 0x7f22c40cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c9d8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c409bbf0 0x7f22c409c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c9d8a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c40cfcf0 con 0x7f22c409f7c0 2024-03-06T07:26:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c9d8a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c40d0060 con 0x7f22ac003f60 2024-03-06T07:26:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c9d8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c415e5c0 con 0x7f22c409bbf0 2024-03-06T07:26:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c409f7c0 0x7f22c40cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c409f7c0 0x7f22c40cd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59910/0 (socket says 172.21.15.59:59910) 2024-03-06T07:26:02.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c2ffd640 1 -- 172.21.15.59:0/1783900518 learned_addr learned my addr 172.21.15.59:0/1783900518 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c37fe640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ac003f60 0x7f22c41f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c3fff640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c409bbf0 0x7f22c409c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c2ffd640 1 -- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c409bbf0 msgr2=0x7f22c409c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c2ffd640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c409bbf0 0x7f22c409c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c2ffd640 1 -- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ac003f60 msgr2=0x7f22c41f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c2ffd640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ac003f60 0x7f22c41f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.663+0000 7f22c2ffd640 1 -- 172.21.15.59:0/1783900518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c40cd710 con 0x7f22c409f7c0 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.664+0000 7f22c2ffd640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c409f7c0 0x7f22c40cd1d0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f22b8037040 tx=0x7f22b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.664+0000 7f22abfff640 1 -- 172.21.15.59:0/1783900518 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b8042020 con 0x7f22c409f7c0 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.664+0000 7f22abfff640 1 -- 172.21.15.59:0/1783900518 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b803a040 con 0x7f22c409f7c0 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.664+0000 7f22abfff640 1 -- 172.21.15.59:0/1783900518 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b8039070 con 0x7f22c409f7c0 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.664+0000 7f22c9d8a640 1 -- 172.21.15.59:0/1783900518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c40cd9b0 con 0x7f22c409f7c0 2024-03-06T07:26:02.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.664+0000 7f22c9d8a640 1 -- 172.21.15.59:0/1783900518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c40cdf70 con 0x7f22c409f7c0 2024-03-06T07:26:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.665+0000 7f22c9d8a640 1 -- 172.21.15.59:0/1783900518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c415d970 con 0x7f22c409f7c0 2024-03-06T07:26:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.666+0000 7f22abfff640 1 -- 172.21.15.59:0/1783900518 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f22b8045020 con 0x7f22c409f7c0 2024-03-06T07:26:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.666+0000 7f22abfff640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f22b006b4d0 0x7f22b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.666+0000 7f22abfff640 1 -- 172.21.15.59:0/1783900518 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f22b80ab6a0 con 0x7f22c409f7c0 2024-03-06T07:26:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.666+0000 7f22c37fe640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f22b006b4d0 0x7f22b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.667+0000 7f22c37fe640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f22b006b4d0 0x7f22b006d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f22c41f74d0 tx=0x7f22b4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.670+0000 7f22abfff640 1 -- 172.21.15.59:0/1783900518 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f22b804c070 con 0x7f22c409f7c0 2024-03-06T07:26:02.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.855+0000 7f22c9d8a640 1 -- 172.21.15.59:0/1783900518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f22c4068ff0 con 0x7f22c409f7c0 2024-03-06T07:26:02.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.856+0000 7f22abfff640 1 -- 172.21.15.59:0/1783900518 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f22b807a520 con 0x7f22c409f7c0 2024-03-06T07:26:02.859 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:26:02.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 -- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f22b006b4d0 msgr2=0x7f22b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f22b006b4d0 0x7f22b006d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f22c41f74d0 tx=0x7f22b4009040 comp rx=0 tx=0).stop 2024-03-06T07:26:02.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 -- 172.21.15.59:0/1783900518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c409f7c0 msgr2=0x7f22c40cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c409f7c0 0x7f22c40cd1d0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f22b8037040 tx=0x7f22b8004730 comp rx=0 tx=0).stop 2024-03-06T07:26:02.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 -- 172.21.15.59:0/1783900518 shutdown_connections 2024-03-06T07:26:02.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f22b006b4d0 0x7f22b006d990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c409bbf0 0x7f22c409c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22c409f7c0 0x7f22c40cd1d0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 --2- 172.21.15.59:0/1783900518 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22ac003f60 0x7f22c41f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 -- 172.21.15.59:0/1783900518 >> 172.21.15.59:0/1783900518 conn(0x7f22c40b3800 msgr2=0x7f22c415fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 -- 172.21.15.59:0/1783900518 shutdown_connections 2024-03-06T07:26:02.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.858+0000 7f22c9d8a640 1 -- 172.21.15.59:0/1783900518 wait complete. 2024-03-06T07:26:02.870 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:02.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.973+0000 7fa36b38c640 1 Processor -- start 2024-03-06T07:26:02.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.973+0000 7fa36b38c640 1 -- start start 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.974+0000 7fa36b38c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3641614d0 0x7fa3640d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.974+0000 7fa36b38c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa3640d2480 0x7fa3640b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.974+0000 7fa36b38c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3641618b0 con 0x7fa3640d1910 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.974+0000 7fa36b38c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3640cf400 con 0x7fa3640d2480 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.974+0000 7fa36b38c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3640cf6a0 con 0x7fa3641614d0 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.974+0000 7fa369929640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa3640d2480 0x7fa3640b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.974+0000 7fa369929640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa3640d2480 0x7fa3640b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59918/0 (socket says 172.21.15.59:59918) 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.974+0000 7fa369929640 1 -- 172.21.15.59:0/3197875798 learned_addr learned my addr 172.21.15.59:0/3197875798 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.975+0000 7fa369128640 1 --2- 172.21.15.59:0/3197875798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3641614d0 0x7fa3640d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.975+0000 7fa369929640 1 -- 172.21.15.59:0/3197875798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3641614d0 msgr2=0x7fa3640d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.975+0000 7fa369929640 1 --2- 172.21.15.59:0/3197875798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3641614d0 0x7fa3640d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.975+0000 7fa369929640 1 -- 172.21.15.59:0/3197875798 >> v1:172.21.15.164:6789/0 conn(0x7fa3640d1910 legacy=0x7fa3640d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.975+0000 7fa369929640 1 -- 172.21.15.59:0/3197875798 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3640cf980 con 0x7fa3640d2480 2024-03-06T07:26:02.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.975+0000 7fa369929640 1 --2- 172.21.15.59:0/3197875798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3640d2480 0x7fa3640b0e30 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009e50 tx=0x7fa35c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d50dfe41c1aa40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.976+0000 7fa35bfff640 1 -- 172.21.15.59:0/3197875798 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c02cb00 con 0x7fa3640d2480 2024-03-06T07:26:02.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.976+0000 7fa35bfff640 1 -- 172.21.15.59:0/3197875798 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35c02cca0 con 0x7fa3640d2480 2024-03-06T07:26:02.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.976+0000 7fa35bfff640 1 -- 172.21.15.59:0/3197875798 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c034610 con 0x7fa3640d2480 2024-03-06T07:26:02.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.976+0000 7fa36b38c640 1 -- 172.21.15.59:0/3197875798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3640d2480 msgr2=0x7fa3640b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.976+0000 7fa36b38c640 1 --2- 172.21.15.59:0/3197875798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3640d2480 0x7fa3640b0e30 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009e50 tx=0x7fa35c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:02.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.976+0000 7fa36b38c640 1 -- 172.21.15.59:0/3197875798 shutdown_connections 2024-03-06T07:26:02.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.976+0000 7fa36b38c640 1 --2- 172.21.15.59:0/3197875798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3640d2480 0x7fa3640b0e30 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.976+0000 7fa36b38c640 1 --2- 172.21.15.59:0/3197875798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3641614d0 0x7fa3640d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.976+0000 7fa36b38c640 1 -- 172.21.15.59:0/3197875798 >> 172.21.15.59:0/3197875798 conn(0x7fa3640d3d40 msgr2=0x7fa3640d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.976+0000 7fa36b38c640 1 -- 172.21.15.59:0/3197875798 shutdown_connections 2024-03-06T07:26:02.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.976+0000 7fa36b38c640 1 -- 172.21.15.59:0/3197875798 wait complete. 2024-03-06T07:26:02.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.977+0000 7fa36b38c640 1 Processor -- start 2024-03-06T07:26:02.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.977+0000 7fa36b38c640 1 -- start start 2024-03-06T07:26:02.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.978+0000 7fa36b38c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3640d1910 0x7fa3641f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.978+0000 7fa36b38c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3640d2480 0x7fa3641f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.978+0000 7fa36b38c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3641614d0 0x7fa3641f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.978+0000 7fa36b38c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3640b1370 con 0x7fa3641614d0 2024-03-06T07:26:02.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.978+0000 7fa36b38c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3640b0910 con 0x7fa3640d1910 2024-03-06T07:26:02.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.978+0000 7fa36b38c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3640b14f0 con 0x7fa3640d2480 2024-03-06T07:26:02.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.978+0000 7fa369929640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3641614d0 0x7fa3641f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.978+0000 7fa369929640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3641614d0 0x7fa3641f4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59934/0 (socket says 172.21.15.59:59934) 2024-03-06T07:26:02.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.978+0000 7fa369929640 1 -- 172.21.15.59:0/1380072670 learned_addr learned my addr 172.21.15.59:0/1380072670 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:02.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.978+0000 7fa369128640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3640d1910 0x7fa3641f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.978+0000 7fa368927640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3640d2480 0x7fa3641f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.979+0000 7fa369929640 1 -- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3640d2480 msgr2=0x7fa3641f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.979+0000 7fa369929640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3640d2480 0x7fa3641f7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.979+0000 7fa369929640 1 -- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3640d1910 msgr2=0x7fa3641f54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.979+0000 7fa369929640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3640d1910 0x7fa3641f54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.979+0000 7fa369929640 1 -- 172.21.15.59:0/1380072670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3641f7640 con 0x7fa3641614d0 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.979+0000 7fa369929640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3641614d0 0x7fa3641f4330 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009e50 tx=0x7fa35c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.979+0000 7fa359ffb640 1 -- 172.21.15.59:0/1380072670 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c043020 con 0x7fa3641614d0 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.979+0000 7fa359ffb640 1 -- 172.21.15.59:0/1380072670 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35c02ce50 con 0x7fa3641614d0 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.979+0000 7fa359ffb640 1 -- 172.21.15.59:0/1380072670 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c009510 con 0x7fa3641614d0 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.979+0000 7fa36b38c640 1 -- 172.21.15.59:0/1380072670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3640b80c0 con 0x7fa3641614d0 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.979+0000 7fa36b38c640 1 -- 172.21.15.59:0/1380072670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3640c9580 con 0x7fa3641614d0 2024-03-06T07:26:02.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.980+0000 7fa36b38c640 1 -- 172.21.15.59:0/1380072670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3640c9750 con 0x7fa3641614d0 2024-03-06T07:26:02.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.981+0000 7fa359ffb640 1 -- 172.21.15.59:0/1380072670 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa35c046020 con 0x7fa3641614d0 2024-03-06T07:26:02.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.981+0000 7fa359ffb640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa34806b4d0 0x7fa34806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.981+0000 7fa359ffb640 1 -- 172.21.15.59:0/1380072670 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa35c0ac7f0 con 0x7fa3641614d0 2024-03-06T07:26:02.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.981+0000 7fa369128640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa34806b4d0 0x7fa34806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.982+0000 7fa369128640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa34806b4d0 0x7fa34806d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa360006fd0 tx=0x7fa360008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:02.989+0000 7fa359ffb640 1 -- 172.21.15.59:0/1380072670 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa35c04d030 con 0x7fa3641614d0 2024-03-06T07:26:03.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.169+0000 7fa36b38c640 1 -- 172.21.15.59:0/1380072670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa364068ff0 con 0x7fa3641614d0 2024-03-06T07:26:03.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.170+0000 7fa359ffb640 1 -- 172.21.15.59:0/1380072670 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa35c07b700 con 0x7fa3641614d0 2024-03-06T07:26:03.172 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:26:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.171+0000 7fa36b38c640 1 -- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa34806b4d0 msgr2=0x7fa34806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.171+0000 7fa36b38c640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa34806b4d0 0x7fa34806d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa360006fd0 tx=0x7fa360008040 comp rx=0 tx=0).stop 2024-03-06T07:26:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.171+0000 7fa36b38c640 1 -- 172.21.15.59:0/1380072670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3641614d0 msgr2=0x7fa3641f4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.171+0000 7fa36b38c640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3641614d0 0x7fa3641f4330 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009e50 tx=0x7fa35c004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.171+0000 7fa36b38c640 1 -- 172.21.15.59:0/1380072670 shutdown_connections 2024-03-06T07:26:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.171+0000 7fa36b38c640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa34806b4d0 0x7fa34806d990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.171+0000 7fa36b38c640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3641614d0 0x7fa3641f4330 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.171+0000 7fa36b38c640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3640d2480 0x7fa3641f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.171+0000 7fa36b38c640 1 --2- 172.21.15.59:0/1380072670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3640d1910 0x7fa3641f54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.172+0000 7fa36b38c640 1 -- 172.21.15.59:0/1380072670 >> 172.21.15.59:0/1380072670 conn(0x7fa3640d3d40 msgr2=0x7fa3640d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.172+0000 7fa36b38c640 1 -- 172.21.15.59:0/1380072670 shutdown_connections 2024-03-06T07:26:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.172+0000 7fa36b38c640 1 -- 172.21.15.59:0/1380072670 wait complete. 2024-03-06T07:26:03.188 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:03.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.290+0000 7f581939a640 1 Processor -- start 2024-03-06T07:26:03.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.290+0000 7f581939a640 1 -- start start 2024-03-06T07:26:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f581939a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f58140cea50 0x7f58140b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f581939a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f581406cab0 0x7f58140ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f581939a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581406d030 con 0x7f581415def0 2024-03-06T07:26:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f581939a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58140cf1c0 con 0x7f581406cab0 2024-03-06T07:26:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f581939a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58140cc760 con 0x7f58140cea50 2024-03-06T07:26:03.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f58137fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f581406cab0 0x7f58140ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f58137fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f581406cab0 0x7f58140ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59950/0 (socket says 172.21.15.59:59950) 2024-03-06T07:26:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f58137fe640 1 -- 172.21.15.59:0/1999586586 learned_addr learned my addr 172.21.15.59:0/1999586586 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f58127fc640 1 --2- 172.21.15.59:0/1999586586 >> v2:172.21.15.164:3300/0 conn(0x7f58140cea50 0x7f58140b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f58137fe640 1 -- 172.21.15.59:0/1999586586 >> v2:172.21.15.164:3300/0 conn(0x7f58140cea50 msgr2=0x7f58140b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f58137fe640 1 --2- 172.21.15.59:0/1999586586 >> v2:172.21.15.164:3300/0 conn(0x7f58140cea50 0x7f58140b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f58137fe640 1 -- 172.21.15.59:0/1999586586 >> v1:172.21.15.164:6789/0 conn(0x7f581415def0 legacy=0x7f58140ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f58137fe640 1 -- 172.21.15.59:0/1999586586 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581406d310 con 0x7f581406cab0 2024-03-06T07:26:03.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.291+0000 7f58127fc640 1 --2- 172.21.15.59:0/1999586586 >> v2:172.21.15.164:3300/0 conn(0x7f58140cea50 0x7f58140b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:03.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.292+0000 7f58137fe640 1 --2- 172.21.15.59:0/1999586586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f581406cab0 0x7f58140ccbf0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f58140b3660 tx=0x7f580c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12cf672e91c0f3e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.292+0000 7f5811ffb640 1 -- 172.21.15.59:0/1999586586 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c035020 con 0x7f581406cab0 2024-03-06T07:26:03.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.292+0000 7f5811ffb640 1 -- 172.21.15.59:0/1999586586 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580c031870 con 0x7f581406cab0 2024-03-06T07:26:03.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.292+0000 7f5811ffb640 1 -- 172.21.15.59:0/1999586586 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c031c40 con 0x7f581406cab0 2024-03-06T07:26:03.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.293+0000 7f581939a640 1 -- 172.21.15.59:0/1999586586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f581406cab0 msgr2=0x7f58140ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.293+0000 7f581939a640 1 --2- 172.21.15.59:0/1999586586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f581406cab0 0x7f58140ccbf0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f58140b3660 tx=0x7f580c02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:03.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.293+0000 7f581939a640 1 -- 172.21.15.59:0/1999586586 shutdown_connections 2024-03-06T07:26:03.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.293+0000 7f581939a640 1 --2- 172.21.15.59:0/1999586586 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f581406cab0 0x7f58140ccbf0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.293+0000 7f581939a640 1 --2- 172.21.15.59:0/1999586586 >> v2:172.21.15.164:3300/0 conn(0x7f58140cea50 0x7f58140b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.293+0000 7f581939a640 1 -- 172.21.15.59:0/1999586586 >> 172.21.15.59:0/1999586586 conn(0x7f58140b3690 msgr2=0x7f58140b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.293+0000 7f581939a640 1 -- 172.21.15.59:0/1999586586 shutdown_connections 2024-03-06T07:26:03.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.293+0000 7f581939a640 1 -- 172.21.15.59:0/1999586586 wait complete. 2024-03-06T07:26:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.294+0000 7f581939a640 1 Processor -- start 2024-03-06T07:26:03.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.294+0000 7f581939a640 1 -- start start 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f581939a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f581406cab0 0x7f581415cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f581939a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58140cea50 0x7f58140c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f581939a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f581415def0 0x7f58140c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f581939a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581415e2d0 con 0x7f581415def0 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f581939a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58140cc3f0 con 0x7f581406cab0 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f581939a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581415e450 con 0x7f58140cea50 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f58137fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f581415def0 0x7f58140c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f58137fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f581415def0 0x7f58140c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59960/0 (socket says 172.21.15.59:59960) 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f58137fe640 1 -- 172.21.15.59:0/2843518438 learned_addr learned my addr 172.21.15.59:0/2843518438 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f58127fc640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58140cea50 0x7f58140c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f5812ffd640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f581406cab0 0x7f581415cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f58137fe640 1 -- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58140cea50 msgr2=0x7f58140c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f58137fe640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58140cea50 0x7f58140c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f58137fe640 1 -- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f581406cab0 msgr2=0x7f581415cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f58137fe640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f581406cab0 0x7f581415cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.295+0000 7f58137fe640 1 -- 172.21.15.59:0/2843518438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58140c60b0 con 0x7f581415def0 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.296+0000 7f58137fe640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f581415def0 0x7f58140c5a80 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f580c009960 tx=0x7f580c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.296+0000 7f580b7fe640 1 -- 172.21.15.59:0/2843518438 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c040070 con 0x7f581415def0 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.296+0000 7f580b7fe640 1 -- 172.21.15.59:0/2843518438 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580c03a420 con 0x7f581415def0 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.296+0000 7f581939a640 1 -- 172.21.15.59:0/2843518438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58140c6390 con 0x7f581415def0 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.296+0000 7f580b7fe640 1 -- 172.21.15.59:0/2843518438 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c035040 con 0x7f581415def0 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.296+0000 7f581939a640 1 -- 172.21.15.59:0/2843518438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58140c6950 con 0x7f581415def0 2024-03-06T07:26:03.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.297+0000 7f581939a640 1 -- 172.21.15.59:0/2843518438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581409fcf0 con 0x7f581415def0 2024-03-06T07:26:03.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.298+0000 7f580b7fe640 1 -- 172.21.15.59:0/2843518438 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f580c038020 con 0x7f581415def0 2024-03-06T07:26:03.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.298+0000 7f580b7fe640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57f406b4d0 0x7f57f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.298+0000 7f580b7fe640 1 -- 172.21.15.59:0/2843518438 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f580c0ac320 con 0x7f581415def0 2024-03-06T07:26:03.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.299+0000 7f5812ffd640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57f406b4d0 0x7f57f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.299+0000 7f5812ffd640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57f406b4d0 0x7f57f406d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5804009f20 tx=0x7f58040094d0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.302+0000 7f580b7fe640 1 -- 172.21.15.59:0/2843518438 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f580c07b070 con 0x7f581415def0 2024-03-06T07:26:03.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.487+0000 7f581939a640 1 -- 172.21.15.59:0/2843518438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5814068ff0 con 0x7f581415def0 2024-03-06T07:26:03.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.487+0000 7f580b7fe640 1 -- 172.21.15.59:0/2843518438 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f580c07a970 con 0x7f581415def0 2024-03-06T07:26:03.489 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:26:03.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 -- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57f406b4d0 msgr2=0x7f57f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57f406b4d0 0x7f57f406d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5804009f20 tx=0x7f58040094d0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 -- 172.21.15.59:0/2843518438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f581415def0 msgr2=0x7f58140c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f581415def0 0x7f58140c5a80 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f580c009960 tx=0x7f580c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 -- 172.21.15.59:0/2843518438 shutdown_connections 2024-03-06T07:26:03.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57f406b4d0 0x7f57f406d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f581415def0 0x7f58140c5a80 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58140cea50 0x7f58140c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 --2- 172.21.15.59:0/2843518438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f581406cab0 0x7f581415cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 -- 172.21.15.59:0/2843518438 >> 172.21.15.59:0/2843518438 conn(0x7f58140b3690 msgr2=0x7f581406fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 -- 172.21.15.59:0/2843518438 shutdown_connections 2024-03-06T07:26:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.489+0000 7f581939a640 1 -- 172.21.15.59:0/2843518438 wait complete. 2024-03-06T07:26:03.502 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:03.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.628+0000 7f9051402640 1 Processor -- start 2024-03-06T07:26:03.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.628+0000 7f9051402640 1 -- start start 2024-03-06T07:26:03.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f9051402640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f904c072160 0x7f904c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f9051402640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f904c15b870 0x7f904c0aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f9051402640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c0b2a10 con 0x7f904c0ae6b0 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f9051402640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f904c0af050 con 0x7f904c072160 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f9051402640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904c0b2cf0 con 0x7f904c15b870 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f904affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f904c072160 0x7f904c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f904a7fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f904c0ae6b0 0x7f904c15b160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42964/0 (socket says 172.21.15.59:42964) 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f904affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f904c072160 0x7f904c0b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35498/0 (socket says 172.21.15.59:35498) 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f904a7fc640 1 -- 172.21.15.59:0/1833138269 learned_addr learned my addr 172.21.15.59:0/1833138269 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f904b7fe640 1 --2- 172.21.15.59:0/1833138269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f904c15b870 0x7f904c0aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f904affd640 1 -- 172.21.15.59:0/1833138269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f904c15b870 msgr2=0x7f904c0aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f904affd640 1 --2- 172.21.15.59:0/1833138269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f904c15b870 0x7f904c0aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f904affd640 1 -- 172.21.15.59:0/1833138269 >> v1:172.21.15.59:6789/0 conn(0x7f904c0ae6b0 legacy=0x7f904c15b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f904affd640 1 -- 172.21.15.59:0/1833138269 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c0b2fd0 con 0x7f904c072160 2024-03-06T07:26:03.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.630+0000 7f9049ffb640 1 -- 172.21.15.59:0/1833138269 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743152245 0 0) 0x7f904c0b2a10 con 0x7f904c0ae6b0 2024-03-06T07:26:03.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.631+0000 7f904affd640 1 --2- 172.21.15.59:0/1833138269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c072160 0x7f904c0b37d0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f90340097a0 tx=0x7f903402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806f6382202098dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.631+0000 7f9049ffb640 1 -- 172.21.15.59:0/1833138269 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034035020 con 0x7f904c072160 2024-03-06T07:26:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.631+0000 7f9049ffb640 1 -- 172.21.15.59:0/1833138269 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9034031830 con 0x7f904c072160 2024-03-06T07:26:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.632+0000 7f9049ffb640 1 -- 172.21.15.59:0/1833138269 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034031c00 con 0x7f904c072160 2024-03-06T07:26:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.632+0000 7f9051402640 1 -- 172.21.15.59:0/1833138269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c072160 msgr2=0x7f904c0b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.632+0000 7f9051402640 1 --2- 172.21.15.59:0/1833138269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c072160 0x7f904c0b37d0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f90340097a0 tx=0x7f903402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.632+0000 7f9051402640 1 -- 172.21.15.59:0/1833138269 shutdown_connections 2024-03-06T07:26:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.632+0000 7f9051402640 1 --2- 172.21.15.59:0/1833138269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f904c15b870 0x7f904c0aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.632+0000 7f9051402640 1 --2- 172.21.15.59:0/1833138269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c072160 0x7f904c0b37d0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.632+0000 7f9051402640 1 -- 172.21.15.59:0/1833138269 >> 172.21.15.59:0/1833138269 conn(0x7f904c06b190 msgr2=0x7f904c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.632+0000 7f9051402640 1 -- 172.21.15.59:0/1833138269 shutdown_connections 2024-03-06T07:26:03.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.632+0000 7f9051402640 1 -- 172.21.15.59:0/1833138269 wait complete. 2024-03-06T07:26:03.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.633+0000 7f9051402640 1 Processor -- start 2024-03-06T07:26:03.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.633+0000 7f9051402640 1 -- start start 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.634+0000 7f9051402640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f904c072160 0x7f904c152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.634+0000 7f9051402640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c0ae6b0 0x7f904c1516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.634+0000 7f9051402640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f904c15b870 0x7f904c15cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.634+0000 7f9051402640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c0cdc90 con 0x7f904c0ae6b0 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.634+0000 7f9051402640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c0cdb10 con 0x7f904c072160 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.634+0000 7f9051402640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904c0cde10 con 0x7f904c15b870 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.634+0000 7f904a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c0ae6b0 0x7f904c1516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.634+0000 7f904a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c0ae6b0 0x7f904c1516a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35504/0 (socket says 172.21.15.59:35504) 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.634+0000 7f904a7fc640 1 -- 172.21.15.59:0/1808521733 learned_addr learned my addr 172.21.15.59:0/1808521733 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f904b7fe640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f904c15b870 0x7f904c15cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f904affd640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f904c072160 0x7f904c152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f904a7fc640 1 -- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f904c15b870 msgr2=0x7f904c15cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f904a7fc640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f904c15b870 0x7f904c15cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f904a7fc640 1 -- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f904c072160 msgr2=0x7f904c152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f904a7fc640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f904c072160 0x7f904c152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f904a7fc640 1 -- 172.21.15.59:0/1808521733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c06dcc0 con 0x7f904c0ae6b0 2024-03-06T07:26:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f904b7fe640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f904c15b870 0x7f904c15cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f904affd640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f904c072160 0x7f904c152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f904a7fc640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c0ae6b0 0x7f904c1516a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f903c00c4a0 tx=0x7f903c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f90437fe640 1 -- 172.21.15.59:0/1808521733 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c017070 con 0x7f904c0ae6b0 2024-03-06T07:26:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f90437fe640 1 -- 172.21.15.59:0/1808521733 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903c005ce0 con 0x7f904c0ae6b0 2024-03-06T07:26:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f9051402640 1 -- 172.21.15.59:0/1808521733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904c152dd0 con 0x7f904c0ae6b0 2024-03-06T07:26:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f90437fe640 1 -- 172.21.15.59:0/1808521733 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c0072b0 con 0x7f904c0ae6b0 2024-03-06T07:26:03.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.635+0000 7f9051402640 1 -- 172.21.15.59:0/1808521733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c153390 con 0x7f904c0ae6b0 2024-03-06T07:26:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.636+0000 7f9051402640 1 -- 172.21.15.59:0/1808521733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904c0732b0 con 0x7f904c0ae6b0 2024-03-06T07:26:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.636+0000 7f90437fe640 1 -- 172.21.15.59:0/1808521733 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f903c01a070 con 0x7f904c0ae6b0 2024-03-06T07:26:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.637+0000 7f90437fe640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f902c06b4d0 0x7f902c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.637+0000 7f90437fe640 1 -- 172.21.15.59:0/1808521733 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f903c089610 con 0x7f904c0ae6b0 2024-03-06T07:26:03.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.637+0000 7f904affd640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f902c06b4d0 0x7f902c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.638+0000 7f904affd640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f902c06b4d0 0x7f902c06d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9034036000 tx=0x7f9034008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.645+0000 7f90437fe640 1 -- 172.21.15.59:0/1808521733 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f903c058400 con 0x7f904c0ae6b0 2024-03-06T07:26:03.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.869+0000 7f9051402640 1 -- 172.21.15.59:0/1808521733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f904c155800 con 0x7f904c0ae6b0 2024-03-06T07:26:03.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.870+0000 7f90437fe640 1 -- 172.21.15.59:0/1808521733 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f903c057d00 con 0x7f904c0ae6b0 2024-03-06T07:26:03.872 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:03.872 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":42,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2865818497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2865818497},{"type":"v1","addr":"172.21.15.164:6836","nonce":2865818497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1805325807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1805325807},{"type":"v1","addr":"172.21.15.164:6837","nonce":1805325807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6999,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3345771250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3345771250},{"type":"v1","addr":"172.21.15.59:6835","nonce":3345771250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:47.090938+0000","modified":"2024-03-06T07:25:49.093664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2521311582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2521311582},{"type":"v1","addr":"172.21.15.59:6834","nonce":2521311582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7002,"qdb_cluster":[7002]},"id":6}]} 2024-03-06T07:26:03.872 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 42 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.871+0000 7f9051402640 1 -- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f902c06b4d0 msgr2=0x7f902c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.871+0000 7f9051402640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f902c06b4d0 0x7f902c06d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9034036000 tx=0x7f9034008030 comp rx=0 tx=0).stop 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.872+0000 7f9051402640 1 -- 172.21.15.59:0/1808521733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c0ae6b0 msgr2=0x7f904c1516a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.872+0000 7f9051402640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c0ae6b0 0x7f904c1516a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f903c00c4a0 tx=0x7f903c00c960 comp rx=0 tx=0).stop 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.872+0000 7f9051402640 1 -- 172.21.15.59:0/1808521733 shutdown_connections 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.872+0000 7f9051402640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f902c06b4d0 0x7f902c06d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.872+0000 7f9051402640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f904c15b870 0x7f904c15cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.872+0000 7f9051402640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f904c0ae6b0 0x7f904c1516a0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.872+0000 7f9051402640 1 --2- 172.21.15.59:0/1808521733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f904c072160 0x7f904c152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.872+0000 7f9051402640 1 -- 172.21.15.59:0/1808521733 >> 172.21.15.59:0/1808521733 conn(0x7f904c06b190 msgr2=0x7f904c0ce1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.872+0000 7f9051402640 1 -- 172.21.15.59:0/1808521733 shutdown_connections 2024-03-06T07:26:03.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.872+0000 7f9051402640 1 -- 172.21.15.59:0/1808521733 wait complete. 2024-03-06T07:26:03.885 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:03.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.984+0000 7fcbffceb640 1 Processor -- start 2024-03-06T07:26:03.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.985+0000 7fcbffceb640 1 -- start start 2024-03-06T07:26:03.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.985+0000 7fcbffceb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcbf815dfe0 0x7fcbf80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.985+0000 7fcbffceb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcbf80d29c0 0x7fcbf80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.985+0000 7fcbffceb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf80b3430 con 0x7fcbf80cf830 2024-03-06T07:26:03.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.985+0000 7fcbffceb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf80b31e0 con 0x7fcbf80d29c0 2024-03-06T07:26:03.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.985+0000 7fcbffceb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf80b3d60 con 0x7fcbf815dfe0 2024-03-06T07:26:03.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.986+0000 7fcbfe288640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcbf80d29c0 0x7fcbf80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.986+0000 7fcbfe288640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcbf80d29c0 0x7fcbf80b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35510/0 (socket says 172.21.15.59:35510) 2024-03-06T07:26:03.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.986+0000 7fcbfe288640 1 -- 172.21.15.59:0/752449207 learned_addr learned my addr 172.21.15.59:0/752449207 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:03.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.986+0000 7fcbfda87640 1 --2- 172.21.15.59:0/752449207 >> v2:172.21.15.164:3300/0 conn(0x7fcbf815dfe0 0x7fcbf80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.986+0000 7fcbfe288640 1 -- 172.21.15.59:0/752449207 >> v2:172.21.15.164:3300/0 conn(0x7fcbf815dfe0 msgr2=0x7fcbf80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.986+0000 7fcbfe288640 1 --2- 172.21.15.59:0/752449207 >> v2:172.21.15.164:3300/0 conn(0x7fcbf815dfe0 0x7fcbf80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.986+0000 7fcbfe288640 1 -- 172.21.15.59:0/752449207 >> v1:172.21.15.164:6789/0 conn(0x7fcbf80cf830 legacy=0x7fcbf80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.986+0000 7fcbfe288640 1 -- 172.21.15.59:0/752449207 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf80b3710 con 0x7fcbf80d29c0 2024-03-06T07:26:03.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.987+0000 7fcbfe288640 1 --2- 172.21.15.59:0/752449207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf80d29c0 0x7fcbf80b41f0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8067990 tx=0x7fcbf002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc52f711846cc1a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.987+0000 7fcbfca85640 1 -- 172.21.15.59:0/752449207 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf002ca80 con 0x7fcbf80d29c0 2024-03-06T07:26:03.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.987+0000 7fcbfca85640 1 -- 172.21.15.59:0/752449207 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf002cc20 con 0x7fcbf80d29c0 2024-03-06T07:26:03.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.987+0000 7fcbfca85640 1 -- 172.21.15.59:0/752449207 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf00343f0 con 0x7fcbf80d29c0 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.987+0000 7fcbffceb640 1 -- 172.21.15.59:0/752449207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf80d29c0 msgr2=0x7fcbf80b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.987+0000 7fcbffceb640 1 --2- 172.21.15.59:0/752449207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf80d29c0 0x7fcbf80b41f0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8067990 tx=0x7fcbf002bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.987+0000 7fcbffceb640 1 -- 172.21.15.59:0/752449207 shutdown_connections 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.987+0000 7fcbffceb640 1 --2- 172.21.15.59:0/752449207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf80d29c0 0x7fcbf80b41f0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.987+0000 7fcbffceb640 1 --2- 172.21.15.59:0/752449207 >> v2:172.21.15.164:3300/0 conn(0x7fcbf815dfe0 0x7fcbf80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.987+0000 7fcbffceb640 1 -- 172.21.15.59:0/752449207 >> 172.21.15.59:0/752449207 conn(0x7fcbf80d3cd0 msgr2=0x7fcbf80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.988+0000 7fcbffceb640 1 -- 172.21.15.59:0/752449207 shutdown_connections 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.988+0000 7fcbffceb640 1 -- 172.21.15.59:0/752449207 wait complete. 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.989+0000 7fcbffceb640 1 Processor -- start 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.989+0000 7fcbffceb640 1 -- start start 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.989+0000 7fcbffceb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbf80cf830 0x7fcbf80cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.989+0000 7fcbffceb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf80d29c0 0x7fcbf815b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.989+0000 7fcbffceb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbf815dfe0 0x7fcbf806ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.989+0000 7fcbffceb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80d1df0 con 0x7fcbf80d29c0 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.989+0000 7fcbffceb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80d1c70 con 0x7fcbf80cf830 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.989+0000 7fcbffceb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf815e3c0 con 0x7fcbf815dfe0 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.989+0000 7fcbfd286640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf80d29c0 0x7fcbf815b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.989+0000 7fcbfd286640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf80d29c0 0x7fcbf815b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35522/0 (socket says 172.21.15.59:35522) 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.989+0000 7fcbfd286640 1 -- 172.21.15.59:0/4202626191 learned_addr learned my addr 172.21.15.59:0/4202626191 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbfda87640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbf80cf830 0x7fcbf80cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbfe288640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbf815dfe0 0x7fcbf806ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbfd286640 1 -- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbf815dfe0 msgr2=0x7fcbf806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbfd286640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbf815dfe0 0x7fcbf806ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbfd286640 1 -- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbf80cf830 msgr2=0x7fcbf80cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbfd286640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbf80cf830 0x7fcbf80cdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbfd286640 1 -- 172.21.15.59:0/4202626191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf815e7f0 con 0x7fcbf80d29c0 2024-03-06T07:26:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbfd286640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf80d29c0 0x7fcbf815b150 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c400 tx=0x7fcbf400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbee7fc640 1 -- 172.21.15.59:0/4202626191 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4011790 con 0x7fcbf80d29c0 2024-03-06T07:26:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbee7fc640 1 -- 172.21.15.59:0/4202626191 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf4017070 con 0x7fcbf80d29c0 2024-03-06T07:26:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbffceb640 1 -- 172.21.15.59:0/4202626191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf80cb110 con 0x7fcbf80d29c0 2024-03-06T07:26:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbee7fc640 1 -- 172.21.15.59:0/4202626191 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4011bb0 con 0x7fcbf80d29c0 2024-03-06T07:26:03.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.990+0000 7fcbffceb640 1 -- 172.21.15.59:0/4202626191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf80cb3f0 con 0x7fcbf80d29c0 2024-03-06T07:26:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.991+0000 7fcbffceb640 1 -- 172.21.15.59:0/4202626191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf804f330 con 0x7fcbf80d29c0 2024-03-06T07:26:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.992+0000 7fcbee7fc640 1 -- 172.21.15.59:0/4202626191 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbf401c460 con 0x7fcbf80d29c0 2024-03-06T07:26:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.992+0000 7fcbee7fc640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 0x7fcbdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.992+0000 7fcbee7fc640 1 -- 172.21.15.59:0/4202626191 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcbf4089fc0 con 0x7fcbf80d29c0 2024-03-06T07:26:03.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.992+0000 7fcbfda87640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 0x7fcbdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.993+0000 7fcbfda87640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 0x7fcbdc06d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8006fd0 tx=0x7fcbe8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:03.996+0000 7fcbee7fc640 1 -- 172.21.15.59:0/4202626191 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbf4058ed0 con 0x7fcbf80d29c0 2024-03-06T07:26:04.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.180+0000 7fcbffceb640 1 -- 172.21.15.59:0/4202626191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbf80d2350 con 0x7fcbf80d29c0 2024-03-06T07:26:04.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.181+0000 7fcbee7fc640 1 -- 172.21.15.59:0/4202626191 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16648 (secure 0 0 0) 0x7fcbf40587d0 con 0x7fcbf80d29c0 2024-03-06T07:26:04.183 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:04.183 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":81,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:48.732524+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:45.368088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:46.056543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:04.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 -- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 msgr2=0x7fcbdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 0x7fcbdc06d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8006fd0 tx=0x7fcbe8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:04.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 -- 172.21.15.59:0/4202626191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf80d29c0 msgr2=0x7fcbf815b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf80d29c0 0x7fcbf815b150 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c400 tx=0x7fcbf400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 -- 172.21.15.59:0/4202626191 shutdown_connections 2024-03-06T07:26:04.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 0x7fcbdc06d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbf815dfe0 0x7fcbf806ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf80d29c0 0x7fcbf815b150 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 --2- 172.21.15.59:0/4202626191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbf80cf830 0x7fcbf80cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 -- 172.21.15.59:0/4202626191 >> 172.21.15.59:0/4202626191 conn(0x7fcbf80d3cd0 msgr2=0x7fcbf80b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 -- 172.21.15.59:0/4202626191 shutdown_connections 2024-03-06T07:26:04.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.183+0000 7fcbffceb640 1 -- 172.21.15.59:0/4202626191 wait complete. 2024-03-06T07:26:04.196 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:26:04.196 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.296+0000 7fb185f20640 1 Processor -- start 2024-03-06T07:26:04.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.296+0000 7fb185f20640 1 -- start start 2024-03-06T07:26:04.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb185f20640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb18015dfe0 0x7fb1800cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb185f20640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1800cf830 0x7fb1800d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb185f20640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1800b3430 con 0x7fb1800d27f0 2024-03-06T07:26:04.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb185f20640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1800d2c50 con 0x7fb18015dfe0 2024-03-06T07:26:04.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb185f20640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800b31e0 con 0x7fb1800cf830 2024-03-06T07:26:04.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb17f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb18015dfe0 0x7fb1800cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb17f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb18015dfe0 0x7fb1800cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35534/0 (socket says 172.21.15.59:35534) 2024-03-06T07:26:04.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb17f7fe640 1 -- 172.21.15.59:0/1251440024 learned_addr learned my addr 172.21.15.59:0/1251440024 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:04.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb17effd640 1 --2- 172.21.15.59:0/1251440024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1800cf830 0x7fb1800d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb17f7fe640 1 -- 172.21.15.59:0/1251440024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1800cf830 msgr2=0x7fb1800d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb17f7fe640 1 --2- 172.21.15.59:0/1251440024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1800cf830 0x7fb1800d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb17f7fe640 1 -- 172.21.15.59:0/1251440024 >> v1:172.21.15.164:6789/0 conn(0x7fb1800d27f0 legacy=0x7fb1800d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.297+0000 7fb17f7fe640 1 -- 172.21.15.59:0/1251440024 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1800b3710 con 0x7fb18015dfe0 2024-03-06T07:26:04.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.298+0000 7fb17f7fe640 1 --2- 172.21.15.59:0/1251440024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb18015dfe0 0x7fb1800cf2f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fb170009870 tx=0x7fb17002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c2bc3a7a9e1a79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.298+0000 7fb17e7fc640 1 -- 172.21.15.59:0/1251440024 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170034070 con 0x7fb18015dfe0 2024-03-06T07:26:04.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.298+0000 7fb17e7fc640 1 -- 172.21.15.59:0/1251440024 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb170030840 con 0x7fb18015dfe0 2024-03-06T07:26:04.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.298+0000 7fb17e7fc640 1 -- 172.21.15.59:0/1251440024 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170030c10 con 0x7fb18015dfe0 2024-03-06T07:26:04.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.299+0000 7fb185f20640 1 -- 172.21.15.59:0/1251440024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb18015dfe0 msgr2=0x7fb1800cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.299+0000 7fb185f20640 1 --2- 172.21.15.59:0/1251440024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb18015dfe0 0x7fb1800cf2f0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fb170009870 tx=0x7fb17002b980 comp rx=0 tx=0).stop 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.299+0000 7fb185f20640 1 -- 172.21.15.59:0/1251440024 shutdown_connections 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.299+0000 7fb185f20640 1 --2- 172.21.15.59:0/1251440024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1800cf830 0x7fb1800d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.299+0000 7fb185f20640 1 --2- 172.21.15.59:0/1251440024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb18015dfe0 0x7fb1800cf2f0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.299+0000 7fb185f20640 1 -- 172.21.15.59:0/1251440024 >> 172.21.15.59:0/1251440024 conn(0x7fb1800d3cd0 msgr2=0x7fb1800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.299+0000 7fb185f20640 1 -- 172.21.15.59:0/1251440024 shutdown_connections 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.299+0000 7fb185f20640 1 -- 172.21.15.59:0/1251440024 wait complete. 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.300+0000 7fb185f20640 1 Processor -- start 2024-03-06T07:26:04.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.301+0000 7fb185f20640 1 -- start start 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb185f20640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1800cf830 0x7fb1801f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb185f20640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1800d27f0 0x7fb18014d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb185f20640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb18015dfe0 0x7fb18014d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb185f20640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800d1c70 con 0x7fb1800cf830 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb185f20640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800b3d60 con 0x7fb1800d27f0 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb185f20640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800d1df0 con 0x7fb18015dfe0 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1800cf830 0x7fb1801f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1800cf830 0x7fb1801f39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35536/0 (socket says 172.21.15.59:35536) 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17f7fe640 1 -- 172.21.15.59:0/495256057 learned_addr learned my addr 172.21.15.59:0/495256057 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17effd640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1800d27f0 0x7fb18014d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17ffff640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb18015dfe0 0x7fb18014d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17f7fe640 1 -- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb18015dfe0 msgr2=0x7fb18014d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17f7fe640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb18015dfe0 0x7fb18014d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17f7fe640 1 -- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1800d27f0 msgr2=0x7fb18014d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17f7fe640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1800d27f0 0x7fb18014d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17f7fe640 1 -- 172.21.15.59:0/495256057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1801f40b0 con 0x7fb1800cf830 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17effd640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1800d27f0 0x7fb18014d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.302+0000 7fb17f7fe640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1800cf830 0x7fb1801f39b0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fb170000c00 tx=0x7fb1700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.303+0000 7fb167fff640 1 -- 172.21.15.59:0/495256057 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170034040 con 0x7fb1800cf830 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.303+0000 7fb167fff640 1 -- 172.21.15.59:0/495256057 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb170039d90 con 0x7fb1800cf830 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.303+0000 7fb185f20640 1 -- 172.21.15.59:0/495256057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1801f4390 con 0x7fb1800cf830 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.303+0000 7fb167fff640 1 -- 172.21.15.59:0/495256057 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170043610 con 0x7fb1800cf830 2024-03-06T07:26:04.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.303+0000 7fb185f20640 1 -- 172.21.15.59:0/495256057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18014ce00 con 0x7fb1800cf830 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.304+0000 7fb185f20640 1 -- 172.21.15.59:0/495256057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18004f330 con 0x7fb1800cf830 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.304+0000 7fb167fff640 1 -- 172.21.15.59:0/495256057 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb17004a060 con 0x7fb1800cf830 2024-03-06T07:26:04.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.304+0000 7fb167fff640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb16c06b4d0 0x7fb16c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.305+0000 7fb167fff640 1 -- 172.21.15.59:0/495256057 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1700ac1e0 con 0x7fb1800cf830 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.305+0000 7fb17effd640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb16c06b4d0 0x7fb16c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.305+0000 7fb17effd640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb16c06b4d0 0x7fb16c06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb1801615c0 tx=0x7fb174008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.309+0000 7fb167fff640 1 -- 172.21.15.59:0/495256057 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb17003e070 con 0x7fb1800cf830 2024-03-06T07:26:04.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.629+0000 7fb185f20640 1 -- 172.21.15.59:0/495256057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1800b0d40 con 0x7fb1800cf830 2024-03-06T07:26:04.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.629+0000 7fb167fff640 1 -- 172.21.15.59:0/495256057 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fb17002d330 con 0x7fb1800cf830 2024-03-06T07:26:04.632 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:04.632 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":42,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2865818497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2865818497},{"type":"v1","addr":"172.21.15.164:6836","nonce":2865818497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1805325807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1805325807},{"type":"v1","addr":"172.21.15.164:6837","nonce":1805325807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6999,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3345771250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3345771250},{"type":"v1","addr":"172.21.15.59:6835","nonce":3345771250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:47.090938+0000","modified":"2024-03-06T07:25:49.093664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2521311582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2521311582},{"type":"v1","addr":"172.21.15.59:6834","nonce":2521311582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7002,"qdb_cluster":[7002]},"id":6}]} 2024-03-06T07:26:04.632 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 42 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.632+0000 7fb185f20640 1 -- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb16c06b4d0 msgr2=0x7fb16c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.632+0000 7fb185f20640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb16c06b4d0 0x7fb16c06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb1801615c0 tx=0x7fb174008040 comp rx=0 tx=0).stop 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.632+0000 7fb185f20640 1 -- 172.21.15.59:0/495256057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1800cf830 msgr2=0x7fb1801f39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.632+0000 7fb185f20640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1800cf830 0x7fb1801f39b0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fb170000c00 tx=0x7fb1700046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.632+0000 7fb185f20640 1 -- 172.21.15.59:0/495256057 shutdown_connections 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.632+0000 7fb185f20640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb16c06b4d0 0x7fb16c06d990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.632+0000 7fb185f20640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb18015dfe0 0x7fb18014d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.632+0000 7fb185f20640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1800d27f0 0x7fb18014d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.632+0000 7fb185f20640 1 --2- 172.21.15.59:0/495256057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1800cf830 0x7fb1801f39b0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.632+0000 7fb185f20640 1 -- 172.21.15.59:0/495256057 >> 172.21.15.59:0/495256057 conn(0x7fb1800d3cd0 msgr2=0x7fb1800cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.633+0000 7fb185f20640 1 -- 172.21.15.59:0/495256057 shutdown_connections 2024-03-06T07:26:04.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.633+0000 7fb185f20640 1 -- 172.21.15.59:0/495256057 wait complete. 2024-03-06T07:26:04.649 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:04.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.749+0000 7f1ab8a88640 1 Processor -- start 2024-03-06T07:26:04.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.749+0000 7f1ab8a88640 1 -- start start 2024-03-06T07:26:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.750+0000 7f1ab8a88640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1ab40cfff0 0x7f1ab40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.750+0000 7f1ab8a88640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab4161980 con 0x7f1ab415bb20 2024-03-06T07:26:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.750+0000 7f1ab8a88640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab4161c60 con 0x7f1ab40cc8b0 2024-03-06T07:26:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.750+0000 7f1ab8a88640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab40d2500 con 0x7f1ab40cfff0 2024-03-06T07:26:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.750+0000 7f1ab1d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1ab40cfff0 0x7f1ab40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.750+0000 7f1ab259c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1ab415bb20 0x7f1ab40d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42974/0 (socket says 172.21.15.59:42974) 2024-03-06T07:26:04.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.750+0000 7f1ab259c640 1 -- 172.21.15.59:0/2476274253 learned_addr learned my addr 172.21.15.59:0/2476274253 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:04.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.750+0000 7f1ab1d9b640 1 -- 172.21.15.59:0/2476274253 >> v1:172.21.15.164:6789/0 conn(0x7f1ab40cc8b0 legacy=0x7f1ab40ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.750+0000 7f1ab1d9b640 1 -- 172.21.15.59:0/2476274253 >> v1:172.21.15.59:6789/0 conn(0x7f1ab415bb20 legacy=0x7f1ab40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.751+0000 7f1ab1d9b640 1 -- 172.21.15.59:0/2476274253 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4161f40 con 0x7f1ab40cfff0 2024-03-06T07:26:04.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.751+0000 7f1ab1d9b640 1 --2- 172.21.15.59:0/2476274253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ab40cfff0 0x7f1ab40d04a0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80097a0 tx=0x7f1aa802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a1fe248de3f63f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.751+0000 7f1ab159a640 1 -- 172.21.15.59:0/2476274253 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8035020 con 0x7f1ab40cfff0 2024-03-06T07:26:04.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.751+0000 7f1ab159a640 1 -- 172.21.15.59:0/2476274253 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa8031830 con 0x7f1ab40cfff0 2024-03-06T07:26:04.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.751+0000 7f1ab159a640 1 -- 172.21.15.59:0/2476274253 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8031c00 con 0x7f1ab40cfff0 2024-03-06T07:26:04.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.752+0000 7f1ab8a88640 1 -- 172.21.15.59:0/2476274253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ab40cfff0 msgr2=0x7f1ab40d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.752+0000 7f1ab8a88640 1 --2- 172.21.15.59:0/2476274253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ab40cfff0 0x7f1ab40d04a0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80097a0 tx=0x7f1aa802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.752+0000 7f1ab8a88640 1 -- 172.21.15.59:0/2476274253 shutdown_connections 2024-03-06T07:26:04.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.752+0000 7f1ab8a88640 1 --2- 172.21.15.59:0/2476274253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ab40cfff0 0x7f1ab40d04a0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.752+0000 7f1ab8a88640 1 -- 172.21.15.59:0/2476274253 >> 172.21.15.59:0/2476274253 conn(0x7f1ab409a9f0 msgr2=0x7f1ab409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.752+0000 7f1ab8a88640 1 -- 172.21.15.59:0/2476274253 shutdown_connections 2024-03-06T07:26:04.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.752+0000 7f1ab8a88640 1 -- 172.21.15.59:0/2476274253 wait complete. 2024-03-06T07:26:04.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.753+0000 7f1ab8a88640 1 Processor -- start 2024-03-06T07:26:04.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.753+0000 7f1ab8a88640 1 -- start start 2024-03-06T07:26:04.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab8a88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab40cc8b0 0x7f1ab41f3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab8a88640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40cfff0 0x7f1ab40c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab8a88640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ab415bb20 0x7f1ab406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab8a88640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40d1980 con 0x7f1ab415bb20 2024-03-06T07:26:04.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab8a88640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab4162500 con 0x7f1ab40cfff0 2024-03-06T07:26:04.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab8a88640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab40d1b00 con 0x7f1ab40cc8b0 2024-03-06T07:26:04.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ab415bb20 0x7f1ab406d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ab415bb20 0x7f1ab406d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35554/0 (socket says 172.21.15.59:35554) 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab2d9d640 1 -- 172.21.15.59:0/3252394609 learned_addr learned my addr 172.21.15.59:0/3252394609 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab259c640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab40cc8b0 0x7f1ab41f3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab1d9b640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40cfff0 0x7f1ab40c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab2d9d640 1 -- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab40cc8b0 msgr2=0x7f1ab41f3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab2d9d640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab40cc8b0 0x7f1ab41f3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab2d9d640 1 -- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40cfff0 msgr2=0x7f1ab40c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.754+0000 7f1ab2d9d640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40cfff0 0x7f1ab40c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.755+0000 7f1ab2d9d640 1 -- 172.21.15.59:0/3252394609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab41f4050 con 0x7f1ab415bb20 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.755+0000 7f1ab2d9d640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ab415bb20 0x7f1ab406d5f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00b970 tx=0x7f1aac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.755+0000 7f1aa2ffd640 1 -- 172.21.15.59:0/3252394609 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac019070 con 0x7f1ab415bb20 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.755+0000 7f1aa2ffd640 1 -- 172.21.15.59:0/3252394609 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac00cb40 con 0x7f1ab415bb20 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.755+0000 7f1ab8a88640 1 -- 172.21.15.59:0/3252394609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab40cb790 con 0x7f1ab415bb20 2024-03-06T07:26:04.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.755+0000 7f1aa2ffd640 1 -- 172.21.15.59:0/3252394609 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac014440 con 0x7f1ab415bb20 2024-03-06T07:26:04.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.755+0000 7f1ab8a88640 1 -- 172.21.15.59:0/3252394609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab40c5a70 con 0x7f1ab415bb20 2024-03-06T07:26:04.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.756+0000 7f1ab8a88640 1 -- 172.21.15.59:0/3252394609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab404f330 con 0x7f1ab415bb20 2024-03-06T07:26:04.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.756+0000 7f1aa2ffd640 1 -- 172.21.15.59:0/3252394609 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1aac01c070 con 0x7f1ab415bb20 2024-03-06T07:26:04.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.757+0000 7f1aa2ffd640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a9406b4d0 0x7f1a9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.757+0000 7f1aa2ffd640 1 -- 172.21.15.59:0/3252394609 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1aac089d90 con 0x7f1ab415bb20 2024-03-06T07:26:04.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.757+0000 7f1ab259c640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a9406b4d0 0x7f1a9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.757+0000 7f1ab259c640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a9406b4d0 0x7f1a9406d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4000f60 tx=0x7f1aa4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:04.761+0000 7f1aa2ffd640 1 -- 172.21.15.59:0/3252394609 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1aac058ca0 con 0x7f1ab415bb20 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.068+0000 7f1ab8a88640 1 -- 172.21.15.59:0/3252394609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ab40b1fe0 con 0x7f1ab415bb20 2024-03-06T07:26:05.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.069+0000 7f1aa2ffd640 1 -- 172.21.15.59:0/3252394609 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f1aac0585a0 con 0x7f1ab415bb20 2024-03-06T07:26:05.071 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:05.071 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":42,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2865818497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2865818497},{"type":"v1","addr":"172.21.15.164:6836","nonce":2865818497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1805325807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1805325807},{"type":"v1","addr":"172.21.15.164:6837","nonce":1805325807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6999,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3345771250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3345771250},{"type":"v1","addr":"172.21.15.59:6835","nonce":3345771250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:47.090938+0000","modified":"2024-03-06T07:25:49.093664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"a","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6834/2521311582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":2521311582},{"type":"v1","addr":"172.21.15.59:6834","nonce":2521311582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7002,"qdb_cluster":[7002]},"id":6}]} 2024-03-06T07:26:05.071 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 42 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 -- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a9406b4d0 msgr2=0x7f1a9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a9406b4d0 0x7f1a9406d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4000f60 tx=0x7f1aa4009040 comp rx=0 tx=0).stop 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 -- 172.21.15.59:0/3252394609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ab415bb20 msgr2=0x7f1ab406d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ab415bb20 0x7f1ab406d5f0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00b970 tx=0x7f1aac00be30 comp rx=0 tx=0).stop 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 -- 172.21.15.59:0/3252394609 shutdown_connections 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a9406b4d0 0x7f1a9406d990 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ab415bb20 0x7f1ab406d5f0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40cfff0 0x7f1ab40c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 --2- 172.21.15.59:0/3252394609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab40cc8b0 0x7f1ab41f3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 -- 172.21.15.59:0/3252394609 >> 172.21.15.59:0/3252394609 conn(0x7f1ab409a9f0 msgr2=0x7f1ab406fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 -- 172.21.15.59:0/3252394609 shutdown_connections 2024-03-06T07:26:05.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.071+0000 7f1ab8a88640 1 -- 172.21.15.59:0/3252394609 wait complete. 2024-03-06T07:26:05.084 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:05.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.183+0000 7fe263288640 1 Processor -- start 2024-03-06T07:26:05.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.183+0000 7fe263288640 1 -- start start 2024-03-06T07:26:05.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.184+0000 7fe263288640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe25c0cfff0 0x7fe25c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.184+0000 7fe263288640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c0cc8b0 0x7fe25c0ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.184+0000 7fe263288640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe25c161980 con 0x7fe25c15bb20 2024-03-06T07:26:05.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.184+0000 7fe263288640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe25c0d2500 con 0x7fe25c0cfff0 2024-03-06T07:26:05.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.184+0000 7fe263288640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c161c60 con 0x7fe25c0cc8b0 2024-03-06T07:26:05.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.184+0000 7fe261024640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe25c15bb20 0x7fe25c0d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:42980/0 (socket says 172.21.15.59:42980) 2024-03-06T07:26:05.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.184+0000 7fe261024640 1 -- 172.21.15.59:0/4141655383 learned_addr learned my addr 172.21.15.59:0/4141655383 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:05.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe260823640 1 --2- 172.21.15.59:0/4141655383 >> v2:172.21.15.164:3300/0 conn(0x7fe25c0cfff0 0x7fe25c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe261825640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c0cc8b0 0x7fe25c0ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599501801 0 0) 0x7fe25c161980 con 0x7fe25c15bb20 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe248003520 con 0x7fe25c15bb20 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1135760713 0 0) 0x7fe248003520 con 0x7fe25c15bb20 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c0cc8b0 msgr2=0x7fe25c0ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c0cc8b0 0x7fe25c0ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 >> v2:172.21.15.164:3300/0 conn(0x7fe25c0cfff0 msgr2=0x7fe25c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 --2- 172.21.15.59:0/4141655383 >> v2:172.21.15.164:3300/0 conn(0x7fe25c0cfff0 0x7fe25c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c161f40 con 0x7fe25c15bb20 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fe258001670 con 0x7fe25c15bb20 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 >> v1:172.21.15.59:6789/0 conn(0x7fe25c15bb20 legacy=0x7fe25c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2480039b0 0x7fe248003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c0cc8b0 0x7fe248004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe248004d40 0x7fe2480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c0d2500 con 0x7fe2480039b0 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c161c60 con 0x7fe248004d40 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24800b9c0 con 0x7fe25c0cc8b0 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe261024640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2480039b0 0x7fe248003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe260823640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c0cc8b0 0x7fe248004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe261825640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe248004d40 0x7fe2480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe261024640 1 -- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c0cc8b0 msgr2=0x7fe248004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe261024640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c0cc8b0 0x7fe248004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe261024640 1 -- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe248004d40 msgr2=0x7fe2480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe261024640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe248004d40 0x7fe2480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe261024640 1 -- 172.21.15.59:0/4141655383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe248003520 con 0x7fe2480039b0 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.185+0000 7fe261825640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe248004d40 0x7fe2480051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.186+0000 7fe261024640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2480039b0 0x7fe248003dd0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fe258001050 tx=0x7fe248005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06a233b6eb8b5e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.186+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258007d70 con 0x7fe2480039b0 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.186+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe258030070 con 0x7fe2480039b0 2024-03-06T07:26:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.186+0000 7fe253fff640 1 -- 172.21.15.59:0/4141655383 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25802cd80 con 0x7fe2480039b0 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.187+0000 7fe263288640 1 -- 172.21.15.59:0/4141655383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2480039b0 msgr2=0x7fe248003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.187+0000 7fe263288640 1 --2- 172.21.15.59:0/4141655383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2480039b0 0x7fe248003dd0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fe258001050 tx=0x7fe248005830 comp rx=0 tx=0).stop 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.187+0000 7fe261825640 1 -- 172.21.15.59:0/4141655383 reap_dead start 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.187+0000 7fe263288640 1 -- 172.21.15.59:0/4141655383 shutdown_connections 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.187+0000 7fe263288640 1 -- 172.21.15.59:0/4141655383 >> 172.21.15.59:0/4141655383 conn(0x7fe25c09a9f0 msgr2=0x7fe25c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.187+0000 7fe263288640 1 -- 172.21.15.59:0/4141655383 shutdown_connections 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.187+0000 7fe263288640 1 -- 172.21.15.59:0/4141655383 wait complete. 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.188+0000 7fe263288640 1 Processor -- start 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.188+0000 7fe263288640 1 -- start start 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe263288640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2480039b0 0x7fe25c15f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe263288640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c150cb0 0x7fe25c0cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe263288640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c151110 0x7fe25c0af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe263288640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c0d1980 con 0x7fe25c151110 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe263288640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c162500 con 0x7fe2480039b0 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe263288640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c0d1b00 con 0x7fe25c150cb0 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe261825640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c151110 0x7fe25c0af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe261825640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c151110 0x7fe25c0af940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35570/0 (socket says 172.21.15.59:35570) 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe261825640 1 -- 172.21.15.59:0/102362312 learned_addr learned my addr 172.21.15.59:0/102362312 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe260823640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c150cb0 0x7fe25c0cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe261024640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2480039b0 0x7fe25c15f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe261825640 1 -- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c150cb0 msgr2=0x7fe25c0cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe261825640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c150cb0 0x7fe25c0cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe261825640 1 -- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2480039b0 msgr2=0x7fe25c15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe261825640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2480039b0 0x7fe25c15f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe261825640 1 -- 172.21.15.59:0/102362312 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c0ca390 con 0x7fe25c151110 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.189+0000 7fe261024640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2480039b0 0x7fe25c15f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.190+0000 7fe261825640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c151110 0x7fe25c0af940 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0cf5e0 tx=0x7fe254010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.190+0000 7fe251ffb640 1 -- 172.21.15.59:0/102362312 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe254007ea0 con 0x7fe25c151110 2024-03-06T07:26:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.190+0000 7fe251ffb640 1 -- 172.21.15.59:0/102362312 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe254010d20 con 0x7fe25c151110 2024-03-06T07:26:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.190+0000 7fe263288640 1 -- 172.21.15.59:0/102362312 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25c0ca670 con 0x7fe25c151110 2024-03-06T07:26:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.190+0000 7fe251ffb640 1 -- 172.21.15.59:0/102362312 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe254003bd0 con 0x7fe25c151110 2024-03-06T07:26:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.190+0000 7fe263288640 1 -- 172.21.15.59:0/102362312 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25c0cac30 con 0x7fe25c151110 2024-03-06T07:26:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.191+0000 7fe263288640 1 -- 172.21.15.59:0/102362312 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25c04f330 con 0x7fe25c151110 2024-03-06T07:26:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.192+0000 7fe251ffb640 1 -- 172.21.15.59:0/102362312 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe254005d10 con 0x7fe25c151110 2024-03-06T07:26:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.192+0000 7fe251ffb640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe24006b4d0 0x7fe24006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.192+0000 7fe251ffb640 1 -- 172.21.15.59:0/102362312 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe25408a870 con 0x7fe25c151110 2024-03-06T07:26:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.192+0000 7fe261024640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe24006b4d0 0x7fe24006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.193+0000 7fe261024640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe24006b4d0 0x7fe24006d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe258034000 tx=0x7fe258007000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.196+0000 7fe251ffb640 1 -- 172.21.15.59:0/102362312 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe254025020 con 0x7fe25c151110 2024-03-06T07:26:05.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.392+0000 7fe263288640 1 -- 172.21.15.59:0/102362312 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe25c0b1820 con 0x7fe25c151110 2024-03-06T07:26:05.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.394+0000 7fe251ffb640 1 -- 172.21.15.59:0/102362312 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16648 (secure 0 0 0) 0x7fe254059660 con 0x7fe25c151110 2024-03-06T07:26:05.396 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:05.396 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":81,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:25:48.732524+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:45.368088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:46.056543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 -- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe24006b4d0 msgr2=0x7fe24006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe24006b4d0 0x7fe24006d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe258034000 tx=0x7fe258007000 comp rx=0 tx=0).stop 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 -- 172.21.15.59:0/102362312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c151110 msgr2=0x7fe25c0af940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c151110 0x7fe25c0af940 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0cf5e0 tx=0x7fe254010590 comp rx=0 tx=0).stop 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 -- 172.21.15.59:0/102362312 shutdown_connections 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe24006b4d0 0x7fe24006d990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe25c151110 0x7fe25c0af940 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe25c150cb0 0x7fe25c0cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 --2- 172.21.15.59:0/102362312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2480039b0 0x7fe25c15f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 -- 172.21.15.59:0/102362312 >> 172.21.15.59:0/102362312 conn(0x7fe25c09a9f0 msgr2=0x7fe25c06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 -- 172.21.15.59:0/102362312 shutdown_connections 2024-03-06T07:26:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.396+0000 7fe263288640 1 -- 172.21.15.59:0/102362312 wait complete. 2024-03-06T07:26:05.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:05.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.509+0000 7f6344755640 1 Processor -- start 2024-03-06T07:26:05.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.509+0000 7f6344755640 1 -- start start 2024-03-06T07:26:05.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.510+0000 7f6344755640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c06d3c0 0x7f633c0b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.510+0000 7f6344755640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f633c0b1850 0x7f633c0b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.510+0000 7f6344755640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c0af4b0 con 0x7f633c1614a0 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.510+0000 7f6344755640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f633c06d800 con 0x7f633c0b1850 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.510+0000 7f6344755640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633c0af790 con 0x7f633c06d3c0 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.510+0000 7f6342cf2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f633c0b1850 0x7f633c0b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.510+0000 7f6341cf0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c06d3c0 0x7f633c0b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.510+0000 7f6341cf0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c06d3c0 0x7f633c0b1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:55444/0 (socket says 172.21.15.59:55444) 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.510+0000 7f6342cf2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f633c0b1850 0x7f633c0b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:33136/0 (socket says 172.21.15.59:33136) 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.510+0000 7f6341cf0640 1 -- 172.21.15.59:0/2124998389 learned_addr learned my addr 172.21.15.59:0/2124998389 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.510+0000 7f63424f1640 1 --1- 172.21.15.59:0/2124998389 >> v1:172.21.15.164:6789/0 conn(0x7f633c1614a0 0x7f633c0b0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:36252/0 (socket says 172.21.15.59:36252) 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.511+0000 7f6341cf0640 1 -- 172.21.15.59:0/2124998389 >> v2:172.21.15.164:3300/0 conn(0x7f633c0b1850 msgr2=0x7f633c0b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.511+0000 7f6341cf0640 1 --2- 172.21.15.59:0/2124998389 >> v2:172.21.15.164:3300/0 conn(0x7f633c0b1850 0x7f633c0b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.511+0000 7f6341cf0640 1 -- 172.21.15.59:0/2124998389 >> v1:172.21.15.164:6789/0 conn(0x7f633c1614a0 legacy=0x7f633c0b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.511+0000 7f6341cf0640 1 -- 172.21.15.59:0/2124998389 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c0afa70 con 0x7f633c06d3c0 2024-03-06T07:26:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.512+0000 7f6341cf0640 1 --2- 172.21.15.59:0/2124998389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c06d3c0 0x7f633c0b1310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f632c009870 tx=0x7f632c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45db6aff792efa9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.512+0000 7f63414ef640 1 -- 172.21.15.59:0/2124998389 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632c034070 con 0x7f633c06d3c0 2024-03-06T07:26:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.512+0000 7f63414ef640 1 -- 172.21.15.59:0/2124998389 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632c030840 con 0x7f633c06d3c0 2024-03-06T07:26:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.512+0000 7f63414ef640 1 -- 172.21.15.59:0/2124998389 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632c030c10 con 0x7f633c06d3c0 2024-03-06T07:26:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.512+0000 7f6344755640 1 -- 172.21.15.59:0/2124998389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c06d3c0 msgr2=0x7f633c0b1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.512+0000 7f6344755640 1 --2- 172.21.15.59:0/2124998389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c06d3c0 0x7f633c0b1310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f632c009870 tx=0x7f632c02b980 comp rx=0 tx=0).stop 2024-03-06T07:26:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.513+0000 7f6344755640 1 -- 172.21.15.59:0/2124998389 shutdown_connections 2024-03-06T07:26:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.513+0000 7f6344755640 1 --2- 172.21.15.59:0/2124998389 >> v2:172.21.15.164:3300/0 conn(0x7f633c0b1850 0x7f633c0b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.513+0000 7f6344755640 1 --2- 172.21.15.59:0/2124998389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c06d3c0 0x7f633c0b1310 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.515 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.513+0000 7f6344755640 1 -- 172.21.15.59:0/2124998389 >> 172.21.15.59:0/2124998389 conn(0x7f633c09a9f0 msgr2=0x7f633c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.513+0000 7f6344755640 1 -- 172.21.15.59:0/2124998389 shutdown_connections 2024-03-06T07:26:05.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.513+0000 7f6344755640 1 -- 172.21.15.59:0/2124998389 wait complete. 2024-03-06T07:26:05.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.514+0000 7f6344755640 1 Processor -- start 2024-03-06T07:26:05.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.514+0000 7f6344755640 1 -- start start 2024-03-06T07:26:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6344755640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f633c06d3c0 0x7f633c0cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6344755640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c0b1850 0x7f633c14a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6344755640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f633c1614a0 0x7f633c15b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6344755640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c0b1c80 con 0x7f633c1614a0 2024-03-06T07:26:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6344755640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f633c161880 con 0x7f633c06d3c0 2024-03-06T07:26:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6344755640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f633c0b1e00 con 0x7f633c0b1850 2024-03-06T07:26:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6342cf2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f633c1614a0 0x7f633c15b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6342cf2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f633c1614a0 0x7f633c15b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35580/0 (socket says 172.21.15.59:35580) 2024-03-06T07:26:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6342cf2640 1 -- 172.21.15.59:0/2351568251 learned_addr learned my addr 172.21.15.59:0/2351568251 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:05.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f63424f1640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f633c06d3c0 0x7f633c0cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6341cf0640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c0b1850 0x7f633c14a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6342cf2640 1 -- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c0b1850 msgr2=0x7f633c14a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6342cf2640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c0b1850 0x7f633c14a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6342cf2640 1 -- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f633c06d3c0 msgr2=0x7f633c0cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6342cf2640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f633c06d3c0 0x7f633c0cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6342cf2640 1 -- 172.21.15.59:0/2351568251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c15a650 con 0x7f633c1614a0 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6342cf2640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f633c1614a0 0x7f633c15b6f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f633800efc0 tx=0x7f633800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6332ffd640 1 -- 172.21.15.59:0/2351568251 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338018020 con 0x7f633c1614a0 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.516+0000 7f6332ffd640 1 -- 172.21.15.59:0/2351568251 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6338014480 con 0x7f633c1614a0 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.517+0000 7f6344755640 1 -- 172.21.15.59:0/2351568251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633c15a930 con 0x7f633c1614a0 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.517+0000 7f6332ffd640 1 -- 172.21.15.59:0/2351568251 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338014870 con 0x7f633c1614a0 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.517+0000 7f6344755640 1 -- 172.21.15.59:0/2351568251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f633c15b1d0 con 0x7f633c1614a0 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.518+0000 7f6344755640 1 -- 172.21.15.59:0/2351568251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633c15b3d0 con 0x7f633c1614a0 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.518+0000 7f6332ffd640 1 -- 172.21.15.59:0/2351568251 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6338005d10 con 0x7f633c1614a0 2024-03-06T07:26:05.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.519+0000 7f6332ffd640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f632006b4d0 0x7f632006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.519+0000 7f6332ffd640 1 -- 172.21.15.59:0/2351568251 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f633808ad30 con 0x7f633c1614a0 2024-03-06T07:26:05.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.519+0000 7f63424f1640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f632006b4d0 0x7f632006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.520+0000 7f63424f1640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f632006b4d0 0x7f632006d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6334000f60 tx=0x7f6334007ce0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.523+0000 7f6332ffd640 1 -- 172.21.15.59:0/2351568251 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6338022070 con 0x7f633c1614a0 2024-03-06T07:26:05.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.745+0000 7f6344755640 1 -- 172.21.15.59:0/2351568251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f633c04f330 con 0x7f633c1614a0 2024-03-06T07:26:05.747 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:26:05.745+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:05.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.746+0000 7f6332ffd640 1 -- 172.21.15.59:0/2351568251 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6338010040 con 0x7f633c1614a0 2024-03-06T07:26:05.748 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:05.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 -- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f632006b4d0 msgr2=0x7f632006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f632006b4d0 0x7f632006d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6334000f60 tx=0x7f6334007ce0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 -- 172.21.15.59:0/2351568251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f633c1614a0 msgr2=0x7f633c15b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f633c1614a0 0x7f633c15b6f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f633800efc0 tx=0x7f633800c490 comp rx=0 tx=0).stop 2024-03-06T07:26:05.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 -- 172.21.15.59:0/2351568251 shutdown_connections 2024-03-06T07:26:05.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f632006b4d0 0x7f632006d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f633c1614a0 0x7f633c15b6f0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f633c0b1850 0x7f633c14a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 --2- 172.21.15.59:0/2351568251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f633c06d3c0 0x7f633c0cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 -- 172.21.15.59:0/2351568251 >> 172.21.15.59:0/2351568251 conn(0x7f633c09a9f0 msgr2=0x7f633c09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 -- 172.21.15.59:0/2351568251 shutdown_connections 2024-03-06T07:26:05.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.749+0000 7f6344755640 1 -- 172.21.15.59:0/2351568251 wait complete. 2024-03-06T07:26:05.752 INFO:tasks.ceph.mds.a.smithi059.stderr: -2918> 2024-03-06T07:25:49.717+0000 7f67bd2b8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:05.763 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:05.780 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.868+0000 7f703caa4640 1 Processor -- start 2024-03-06T07:26:05.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.868+0000 7f703caa4640 1 -- start start 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.868+0000 7f703caa4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70381586e0 0x7f70380aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.868+0000 7f703caa4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f70380aef80 0x7f70380cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.868+0000 7f703caa4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f70380cffd0 0x7f703806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.868+0000 7f703caa4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70380d03f0 con 0x7f70380aef80 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.868+0000 7f703caa4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7038158ac0 con 0x7f70380cffd0 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.868+0000 7f703caa4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703815b2a0 con 0x7f70381586e0 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.869+0000 7f702dd9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f70380aef80 0x7f70380cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.869+0000 7f702dd9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f70380aef80 0x7f70380cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35618/0 (socket says 172.21.15.59:35618) 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.869+0000 7f702dd9b640 1 -- 172.21.15.59:0/2567408398 learned_addr learned my addr 172.21.15.59:0/2567408398 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.869+0000 7f703659c640 1 --2- 172.21.15.59:0/2567408398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70381586e0 0x7f70380aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.869+0000 7f7036d9d640 1 --2- 172.21.15.59:0/2567408398 >> v2:172.21.15.164:3300/0 conn(0x7f70380cffd0 0x7f703806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.869+0000 7f702dd9b640 1 -- 172.21.15.59:0/2567408398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70381586e0 msgr2=0x7f70380aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.869+0000 7f702dd9b640 1 --2- 172.21.15.59:0/2567408398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70381586e0 0x7f70380aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.869+0000 7f702dd9b640 1 -- 172.21.15.59:0/2567408398 >> v2:172.21.15.164:3300/0 conn(0x7f70380cffd0 msgr2=0x7f703806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.869+0000 7f702dd9b640 1 --2- 172.21.15.59:0/2567408398 >> v2:172.21.15.164:3300/0 conn(0x7f70380cffd0 0x7f703806da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.869+0000 7f702dd9b640 1 -- 172.21.15.59:0/2567408398 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703815b540 con 0x7f70380aef80 2024-03-06T07:26:05.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f702dd9b640 1 --2- 172.21.15.59:0/2567408398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70380aef80 0x7f70380cfa90 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f7038072e00 tx=0x7f702002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e5d59e5ac358a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f7035d9b640 1 -- 172.21.15.59:0/2567408398 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7020035020 con 0x7f70380aef80 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f7035d9b640 1 -- 172.21.15.59:0/2567408398 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7020031870 con 0x7f70380aef80 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f7035d9b640 1 -- 172.21.15.59:0/2567408398 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7020031c40 con 0x7f70380aef80 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f703caa4640 1 -- 172.21.15.59:0/2567408398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70380aef80 msgr2=0x7f70380cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f703caa4640 1 --2- 172.21.15.59:0/2567408398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70380aef80 0x7f70380cfa90 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f7038072e00 tx=0x7f702002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f703caa4640 1 -- 172.21.15.59:0/2567408398 shutdown_connections 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f703caa4640 1 --2- 172.21.15.59:0/2567408398 >> v2:172.21.15.164:3300/0 conn(0x7f70380cffd0 0x7f703806da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f703caa4640 1 --2- 172.21.15.59:0/2567408398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70380aef80 0x7f70380cfa90 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f703caa4640 1 --2- 172.21.15.59:0/2567408398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70381586e0 0x7f70380aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f703caa4640 1 -- 172.21.15.59:0/2567408398 >> 172.21.15.59:0/2567408398 conn(0x7f703806b190 msgr2=0x7f703806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f703caa4640 1 -- 172.21.15.59:0/2567408398 shutdown_connections 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.870+0000 7f703caa4640 1 -- 172.21.15.59:0/2567408398 wait complete. 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.872+0000 7f703caa4640 1 Processor -- start 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.872+0000 7f703caa4640 1 -- start start 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.872+0000 7f703caa4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70380aef80 0x7f703814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.872+0000 7f703caa4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70380cffd0 0x7f7038152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f703caa4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70381586e0 0x7f703813fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f703caa4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703815bde0 con 0x7f70381586e0 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f703caa4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703815bb00 con 0x7f70380cffd0 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f703caa4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7038071350 con 0x7f70380aef80 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f7036d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70381586e0 0x7f703813fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f7036d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70381586e0 0x7f703813fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35624/0 (socket says 172.21.15.59:35624) 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f7036d9d640 1 -- 172.21.15.59:0/1047514281 learned_addr learned my addr 172.21.15.59:0/1047514281 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f703659c640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70380aef80 0x7f703814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f702dd9b640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70380cffd0 0x7f7038152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f7036d9d640 1 -- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70380aef80 msgr2=0x7f703814ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f7036d9d640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70380aef80 0x7f703814ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f7036d9d640 1 -- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70380cffd0 msgr2=0x7f7038152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f7036d9d640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70380cffd0 0x7f7038152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f7036d9d640 1 -- 172.21.15.59:0/1047514281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70381505d0 con 0x7f70381586e0 2024-03-06T07:26:05.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f7036d9d640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70381586e0 0x7f703813fac0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f7030007e90 tx=0x7f703000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f702f7fe640 1 -- 172.21.15.59:0/1047514281 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030014420 con 0x7f70381586e0 2024-03-06T07:26:05.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f702f7fe640 1 -- 172.21.15.59:0/1047514281 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7030019070 con 0x7f70381586e0 2024-03-06T07:26:05.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f702f7fe640 1 -- 172.21.15.59:0/1047514281 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030014820 con 0x7f70381586e0 2024-03-06T07:26:05.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f703caa4640 1 -- 172.21.15.59:0/1047514281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70380af380 con 0x7f70381586e0 2024-03-06T07:26:05.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.873+0000 7f703caa4640 1 -- 172.21.15.59:0/1047514281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70380af940 con 0x7f70381586e0 2024-03-06T07:26:05.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.874+0000 7f703caa4640 1 -- 172.21.15.59:0/1047514281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70380732b0 con 0x7f70381586e0 2024-03-06T07:26:05.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.875+0000 7f702f7fe640 1 -- 172.21.15.59:0/1047514281 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70300149c0 con 0x7f70381586e0 2024-03-06T07:26:05.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.875+0000 7f702f7fe640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f701806b4d0 0x7f701806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.875+0000 7f702f7fe640 1 -- 172.21.15.59:0/1047514281 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f703008aa30 con 0x7f70381586e0 2024-03-06T07:26:05.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.876+0000 7f703659c640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f701806b4d0 0x7f701806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.876+0000 7f703659c640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f701806b4d0 0x7f701806d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f70280079a0 tx=0x7f7028008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:05.880+0000 7f702f7fe640 1 -- 172.21.15.59:0/1047514281 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7030059170 con 0x7f70381586e0 2024-03-06T07:26:06.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.097+0000 7f703caa4640 1 -- 172.21.15.59:0/1047514281 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7038068ff0 con 0x7f70381586e0 2024-03-06T07:26:06.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.793+0000 7f702f7fe640 1 -- 172.21.15.59:0/1047514281 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f703005d210 con 0x7f70381586e0 2024-03-06T07:26:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 -- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f701806b4d0 msgr2=0x7f701806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f701806b4d0 0x7f701806d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f70280079a0 tx=0x7f7028008040 comp rx=0 tx=0).stop 2024-03-06T07:26:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 -- 172.21.15.59:0/1047514281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70381586e0 msgr2=0x7f703813fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70381586e0 0x7f703813fac0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f7030007e90 tx=0x7f703000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 -- 172.21.15.59:0/1047514281 shutdown_connections 2024-03-06T07:26:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f701806b4d0 0x7f701806d990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70381586e0 0x7f703813fac0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f70380cffd0 0x7f7038152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 --2- 172.21.15.59:0/1047514281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70380aef80 0x7f703814ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 -- 172.21.15.59:0/1047514281 >> 172.21.15.59:0/1047514281 conn(0x7f703806b190 msgr2=0x7f7038071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 -- 172.21.15.59:0/1047514281 shutdown_connections 2024-03-06T07:26:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.796+0000 7f703caa4640 1 -- 172.21.15.59:0/1047514281 wait complete. 2024-03-06T07:26:06.813 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:26:06.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.964+0000 7fb6c3579640 1 Processor -- start 2024-03-06T07:26:06.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.964+0000 7fb6c3579640 1 -- start start 2024-03-06T07:26:06.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.964+0000 7fb6c3579640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6bc161820 0x7fb6bc0d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.964+0000 7fb6c3579640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb6bc0d1300 0x7fb6bc0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c3579640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6bc0cf520 con 0x7fb6bc161440 2024-03-06T07:26:06.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c3579640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6bc0cf2d0 con 0x7fb6bc0d1300 2024-03-06T07:26:06.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c3579640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0b6b00 con 0x7fb6bc161820 2024-03-06T07:26:06.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c1b16640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb6bc0d1300 0x7fb6bc0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c1315640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb6bc161440 0x7fb6bc0b4be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:43006/0 (socket says 172.21.15.59:43006) 2024-03-06T07:26:06.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c1315640 1 -- 172.21.15.59:0/1963812177 learned_addr learned my addr 172.21.15.59:0/1963812177 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:06.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c1b16640 1 --2- 172.21.15.59:0/1963812177 >> v2:172.21.15.59:3300/0 conn(0x7fb6bc0d1300 0x7fb6bc0d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35630/0 (socket says 172.21.15.59:35630) 2024-03-06T07:26:06.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c0b14640 1 --2- 172.21.15.59:0/1963812177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6bc161820 0x7fb6bc0d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c1b16640 1 -- 172.21.15.59:0/1963812177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6bc161820 msgr2=0x7fb6bc0d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c1b16640 1 --2- 172.21.15.59:0/1963812177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6bc161820 0x7fb6bc0d0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c1b16640 1 -- 172.21.15.59:0/1963812177 >> v1:172.21.15.59:6789/0 conn(0x7fb6bc161440 legacy=0x7fb6bc0b4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6c1b16640 1 -- 172.21.15.59:0/1963812177 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc0cf800 con 0x7fb6bc0d1300 2024-03-06T07:26:06.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.965+0000 7fb6b3fff640 1 -- 172.21.15.59:0/1963812177 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255598029 0 0) 0x7fb6bc0cf520 con 0x7fb6bc161440 2024-03-06T07:26:06.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.966+0000 7fb6c1b16640 1 --2- 172.21.15.59:0/1963812177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6bc0d1300 0x7fb6bc0d0b50 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009d90 tx=0x7fb6b402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65504909de96a969 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.966+0000 7fb6b3fff640 1 -- 172.21.15.59:0/1963812177 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4039070 con 0x7fb6bc0d1300 2024-03-06T07:26:06.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.966+0000 7fb6b3fff640 1 -- 172.21.15.59:0/1963812177 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b402cbc0 con 0x7fb6bc0d1300 2024-03-06T07:26:06.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.966+0000 7fb6b3fff640 1 -- 172.21.15.59:0/1963812177 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4034440 con 0x7fb6bc0d1300 2024-03-06T07:26:06.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.966+0000 7fb6c3579640 1 -- 172.21.15.59:0/1963812177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6bc0d1300 msgr2=0x7fb6bc0d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.966+0000 7fb6c3579640 1 --2- 172.21.15.59:0/1963812177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6bc0d1300 0x7fb6bc0d0b50 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009d90 tx=0x7fb6b402be10 comp rx=0 tx=0).stop 2024-03-06T07:26:06.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.966+0000 7fb6c3579640 1 -- 172.21.15.59:0/1963812177 shutdown_connections 2024-03-06T07:26:06.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.966+0000 7fb6c3579640 1 --2- 172.21.15.59:0/1963812177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6bc0d1300 0x7fb6bc0d0b50 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.966+0000 7fb6c3579640 1 --2- 172.21.15.59:0/1963812177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6bc161820 0x7fb6bc0d0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.967+0000 7fb6c3579640 1 -- 172.21.15.59:0/1963812177 >> 172.21.15.59:0/1963812177 conn(0x7fb6bc0d3cb0 msgr2=0x7fb6bc0d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.967+0000 7fb6c3579640 1 -- 172.21.15.59:0/1963812177 shutdown_connections 2024-03-06T07:26:06.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.967+0000 7fb6c3579640 1 -- 172.21.15.59:0/1963812177 wait complete. 2024-03-06T07:26:06.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.968+0000 7fb6c3579640 1 Processor -- start 2024-03-06T07:26:06.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.968+0000 7fb6c3579640 1 -- start start 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.968+0000 7fb6c3579640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6bc0d1300 0x7fb6bc151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.968+0000 7fb6c3579640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6bc161440 0x7fb6bc0d4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.968+0000 7fb6c3579640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6bc161820 0x7fb6bc0d45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.968+0000 7fb6c3579640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0b7110 con 0x7fb6bc161820 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c3579640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0b6f90 con 0x7fb6bc161440 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c3579640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0b7290 con 0x7fb6bc0d1300 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c1b16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6bc161820 0x7fb6bc0d45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c1b16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6bc161820 0x7fb6bc0d45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35640/0 (socket says 172.21.15.59:35640) 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c1b16640 1 -- 172.21.15.59:0/379162729 learned_addr learned my addr 172.21.15.59:0/379162729 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c1b16640 1 -- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6bc0d1300 msgr2=0x7fb6bc151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c1b16640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6bc0d1300 0x7fb6bc151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c1b16640 1 -- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6bc161440 msgr2=0x7fb6bc0d4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c1b16640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6bc161440 0x7fb6bc0d4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c1b16640 1 -- 172.21.15.59:0/379162729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc0cbc40 con 0x7fb6bc161820 2024-03-06T07:26:06.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c1b16640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6bc161820 0x7fb6bc0d45d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4037040 tx=0x7fb6b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6b1ffb640 1 -- 172.21.15.59:0/379162729 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4039050 con 0x7fb6bc161820 2024-03-06T07:26:06.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6b1ffb640 1 -- 172.21.15.59:0/379162729 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b403a040 con 0x7fb6bc161820 2024-03-06T07:26:06.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c3579640 1 -- 172.21.15.59:0/379162729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6bc0cbf20 con 0x7fb6bc161820 2024-03-06T07:26:06.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6b1ffb640 1 -- 172.21.15.59:0/379162729 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4009450 con 0x7fb6bc161820 2024-03-06T07:26:06.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.969+0000 7fb6c3579640 1 -- 172.21.15.59:0/379162729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6bc0cc4e0 con 0x7fb6bc161820 2024-03-06T07:26:06.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.970+0000 7fb6c3579640 1 -- 172.21.15.59:0/379162729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6bc04f330 con 0x7fb6bc161820 2024-03-06T07:26:06.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.971+0000 7fb6b1ffb640 1 -- 172.21.15.59:0/379162729 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6b4047030 con 0x7fb6bc161820 2024-03-06T07:26:06.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.971+0000 7fb6b1ffb640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb6a006b4d0 0x7fb6a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.971+0000 7fb6b1ffb640 1 -- 172.21.15.59:0/379162729 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb6b40ac430 con 0x7fb6bc161820 2024-03-06T07:26:06.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.972+0000 7fb6c1315640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb6a006b4d0 0x7fb6a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.972+0000 7fb6c1315640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb6a006b4d0 0x7fb6a006d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8007fd0 tx=0x7fb6b8007920 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:06.976+0000 7fb6b1ffb640 1 -- 172.21.15.59:0/379162729 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6b404e060 con 0x7fb6bc161820 2024-03-06T07:26:07.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.157+0000 7fb6c3579640 1 -- 172.21.15.59:0/379162729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb6bc0d1b40 con 0x7fb6bc161820 2024-03-06T07:26:07.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.805+0000 7fb6b1ffb640 1 -- 172.21.15.59:0/379162729 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v83) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb6b407b2e0 con 0x7fb6bc161820 2024-03-06T07:26:07.807 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:26:07.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 -- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb6a006b4d0 msgr2=0x7fb6a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb6a006b4d0 0x7fb6a006d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8007fd0 tx=0x7fb6b8007920 comp rx=0 tx=0).stop 2024-03-06T07:26:07.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 -- 172.21.15.59:0/379162729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6bc161820 msgr2=0x7fb6bc0d45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6bc161820 0x7fb6bc0d45d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4037040 tx=0x7fb6b4004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:07.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 -- 172.21.15.59:0/379162729 shutdown_connections 2024-03-06T07:26:07.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb6a006b4d0 0x7fb6a006d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6bc161820 0x7fb6bc0d45d0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6bc161440 0x7fb6bc0d4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 --2- 172.21.15.59:0/379162729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6bc0d1300 0x7fb6bc151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 -- 172.21.15.59:0/379162729 >> 172.21.15.59:0/379162729 conn(0x7fb6bc0d3cb0 msgr2=0x7fb6bc0b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 -- 172.21.15.59:0/379162729 shutdown_connections 2024-03-06T07:26:07.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.809+0000 7fb6c3579640 1 -- 172.21.15.59:0/379162729 wait complete. 2024-03-06T07:26:07.826 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:26:07.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.936+0000 7f2960dd7640 1 Processor -- start 2024-03-06T07:26:07.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.936+0000 7f2960dd7640 1 -- start start 2024-03-06T07:26:07.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.937+0000 7f2960dd7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0d4630 0x7f295c0d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.937+0000 7f2960dd7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295c15e380 con 0x7f295c15dfa0 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.937+0000 7f2960dd7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295c0cf470 con 0x7f295c0d2820 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.937+0000 7f2960dd7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295c0d2100 con 0x7f295c0d4630 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.937+0000 7f295a59c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f295c15dfa0 0x7f295c0d18f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:43012/0 (socket says 172.21.15.59:43012) 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.937+0000 7f295a59c640 1 -- 172.21.15.59:0/732801764 learned_addr learned my addr 172.21.15.59:0/732801764 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585748915 0 0) 0x7f295c15e380 con 0x7f295c15dfa0 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2951d9b640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0d4630 0x7f295c0d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2948003520 con 0x7f295c15dfa0 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 16067143 0 0) 0x7f2948003520 con 0x7f295c15dfa0 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0d4630 msgr2=0x7f295c0d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0d4630 0x7f295c0d22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 >> v1:172.21.15.164:6789/0 conn(0x7f295c0d2820 legacy=0x7f295c0cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c0cf750 con 0x7f295c15dfa0 2024-03-06T07:26:07.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f294c001670 con 0x7f295c15dfa0 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2951d9b640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0d4630 0x7f295c0d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 >> v1:172.21.15.59:6789/0 conn(0x7f295c15dfa0 legacy=0x7f295c0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29480039e0 0x7f2948003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2948004800 0x7f2948004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0d4630 0x7f294800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c0d2100 con 0x7f29480039e0 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29480054e0 con 0x7f2948004800 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294800bf00 con 0x7f295c0d4630 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f295a59c640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29480039e0 0x7f2948003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f2951d9b640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2948004800 0x7f2948004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f295ad9d640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0d4630 0x7f294800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f295a59c640 1 -- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0d4630 msgr2=0x7f294800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f295a59c640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0d4630 0x7f294800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f295a59c640 1 -- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2948004800 msgr2=0x7f2948004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f295a59c640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2948004800 0x7f2948004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f295a59c640 1 -- 172.21.15.59:0/732801764 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2948003520 con 0x7f29480039e0 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.938+0000 7f295ad9d640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0d4630 0x7f294800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.939+0000 7f295a59c640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29480039e0 0x7f2948003e00 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f294c00c240 tx=0x7f2948005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ff1b2310a31722 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.939+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c007d10 con 0x7f29480039e0 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.939+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294c030070 con 0x7f29480039e0 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.939+0000 7f2959d9b640 1 -- 172.21.15.59:0/732801764 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c02cd60 con 0x7f29480039e0 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.940+0000 7f2960dd7640 1 -- 172.21.15.59:0/732801764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29480039e0 msgr2=0x7f2948003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.940+0000 7f2960dd7640 1 --2- 172.21.15.59:0/732801764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29480039e0 0x7f2948003e00 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f294c00c240 tx=0x7f2948005830 comp rx=0 tx=0).stop 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.940+0000 7f295ad9d640 1 -- 172.21.15.59:0/732801764 reap_dead start 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.940+0000 7f2960dd7640 1 -- 172.21.15.59:0/732801764 shutdown_connections 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.940+0000 7f2960dd7640 1 -- 172.21.15.59:0/732801764 >> 172.21.15.59:0/732801764 conn(0x7f295c09d000 msgr2=0x7f295c0d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.940+0000 7f2960dd7640 1 -- 172.21.15.59:0/732801764 shutdown_connections 2024-03-06T07:26:07.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.940+0000 7f2960dd7640 1 -- 172.21.15.59:0/732801764 wait complete. 2024-03-06T07:26:07.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.941+0000 7f2960dd7640 1 Processor -- start 2024-03-06T07:26:07.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.941+0000 7f2960dd7640 1 -- start start 2024-03-06T07:26:07.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2960dd7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29480039e0 0x7f295c1f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2960dd7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f295c1f3b60 0x7f295c1f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2960dd7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c15e530 0x7f295c15e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2960dd7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c0d2460 con 0x7f295c1f3b60 2024-03-06T07:26:07.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2960dd7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c0d22e0 con 0x7f295c15e530 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2960dd7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295c0d25e0 con 0x7f29480039e0 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2951d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f295c1f3b60 0x7f295c1f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2951d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f295c1f3b60 0x7f295c1f4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35658/0 (socket says 172.21.15.59:35658) 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2951d9b640 1 -- 172.21.15.59:0/2655800284 learned_addr learned my addr 172.21.15.59:0/2655800284 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f295ad9d640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c15e530 0x7f295c15e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2951d9b640 1 -- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29480039e0 msgr2=0x7f295c1f0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f295a59c640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29480039e0 0x7f295c1f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2951d9b640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29480039e0 0x7f295c1f0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2951d9b640 1 -- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c15e530 msgr2=0x7f295c15e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2951d9b640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c15e530 0x7f295c15e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.942+0000 7f2951d9b640 1 -- 172.21.15.59:0/2655800284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c1605f0 con 0x7f295c1f3b60 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.943+0000 7f2951d9b640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f295c1f3b60 0x7f295c1f4190 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2944011620 tx=0x7f2944011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.943+0000 7f29537fe640 1 -- 172.21.15.59:0/2655800284 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944019070 con 0x7f295c1f3b60 2024-03-06T07:26:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.943+0000 7f29537fe640 1 -- 172.21.15.59:0/2655800284 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2944004220 con 0x7f295c1f3b60 2024-03-06T07:26:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.943+0000 7f2960dd7640 1 -- 172.21.15.59:0/2655800284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295c1608d0 con 0x7f295c1f3b60 2024-03-06T07:26:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.943+0000 7f29537fe640 1 -- 172.21.15.59:0/2655800284 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294400f040 con 0x7f295c1f3b60 2024-03-06T07:26:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.943+0000 7f2960dd7640 1 -- 172.21.15.59:0/2655800284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295c160e90 con 0x7f295c1f3b60 2024-03-06T07:26:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.944+0000 7f2960dd7640 1 -- 172.21.15.59:0/2655800284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295c04f330 con 0x7f295c1f3b60 2024-03-06T07:26:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.944+0000 7f29537fe640 1 -- 172.21.15.59:0/2655800284 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f294400f720 con 0x7f295c1f3b60 2024-03-06T07:26:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.944+0000 7f29537fe640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f293c06b4d0 0x7f293c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.945+0000 7f29537fe640 1 -- 172.21.15.59:0/2655800284 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f294408a2f0 con 0x7f295c1f3b60 2024-03-06T07:26:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.945+0000 7f295a59c640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f293c06b4d0 0x7f293c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.945+0000 7f295a59c640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f293c06b4d0 0x7f293c06d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f294c034000 tx=0x7f294c009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:07.949+0000 7f29537fe640 1 -- 172.21.15.59:0/2655800284 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29440592d0 con 0x7f295c1f3b60 2024-03-06T07:26:08.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.129+0000 7f2960dd7640 1 -- 172.21.15.59:0/2655800284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f295c0ccba0 con 0x7f295c1f3b60 2024-03-06T07:26:08.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.812+0000 7f29537fe640 1 -- 172.21.15.59:0/2655800284 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v84) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2944058bd0 con 0x7f295c1f3b60 2024-03-06T07:26:08.814 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-03-06T07:26:08.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.815+0000 7f2960dd7640 1 -- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f293c06b4d0 msgr2=0x7f293c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.815+0000 7f2960dd7640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f293c06b4d0 0x7f293c06d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f294c034000 tx=0x7f294c009030 comp rx=0 tx=0).stop 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.815+0000 7f2960dd7640 1 -- 172.21.15.59:0/2655800284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f295c1f3b60 msgr2=0x7f295c1f4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.815+0000 7f2960dd7640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f295c1f3b60 0x7f295c1f4190 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2944011620 tx=0x7f2944011ae0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.816+0000 7f2960dd7640 1 -- 172.21.15.59:0/2655800284 shutdown_connections 2024-03-06T07:26:08.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.816+0000 7f2960dd7640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f293c06b4d0 0x7f293c06d990 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.816+0000 7f2960dd7640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c15e530 0x7f295c15e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.816+0000 7f2960dd7640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f295c1f3b60 0x7f295c1f4190 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.816+0000 7f2960dd7640 1 --2- 172.21.15.59:0/2655800284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29480039e0 0x7f295c1f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.817+0000 7f2960dd7640 1 -- 172.21.15.59:0/2655800284 >> 172.21.15.59:0/2655800284 conn(0x7f295c09d000 msgr2=0x7f295c0d03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.817+0000 7f2960dd7640 1 -- 172.21.15.59:0/2655800284 shutdown_connections 2024-03-06T07:26:08.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.818+0000 7f2960dd7640 1 -- 172.21.15.59:0/2655800284 wait complete. 2024-03-06T07:26:08.837 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:26:08.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.947+0000 7fba64455640 1 Processor -- start 2024-03-06T07:26:08.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.947+0000 7fba64455640 1 -- start start 2024-03-06T07:26:08.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba64455640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fba5c0d4860 0x7fba5c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba64455640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fba5c0b0e30 0x7fba5c0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba64455640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5c1618b0 con 0x7fba5c1614d0 2024-03-06T07:26:08.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba64455640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5c0d4cf0 con 0x7fba5c0b0e30 2024-03-06T07:26:08.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba64455640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5c0cf400 con 0x7fba5c0d4860 2024-03-06T07:26:08.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba621f1640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fba5c1614d0 0x7fba5c0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:43026/0 (socket says 172.21.15.59:43026) 2024-03-06T07:26:08.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba629f2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fba5c0b0e30 0x7fba5c0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba621f1640 1 -- 172.21.15.59:0/118685798 learned_addr learned my addr 172.21.15.59:0/118685798 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:08.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba629f2640 1 -- 172.21.15.59:0/118685798 >> v2:172.21.15.164:3300/0 conn(0x7fba5c0d4860 msgr2=0x7fba5c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba629f2640 1 --2- 172.21.15.59:0/118685798 >> v2:172.21.15.164:3300/0 conn(0x7fba5c0d4860 0x7fba5c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba629f2640 1 -- 172.21.15.59:0/118685798 >> v1:172.21.15.59:6789/0 conn(0x7fba5c1614d0 legacy=0x7fba5c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba629f2640 1 -- 172.21.15.59:0/118685798 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c0cf6a0 con 0x7fba5c0b0e30 2024-03-06T07:26:08.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.948+0000 7fba611ef640 1 -- 172.21.15.59:0/118685798 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456995812 0 0) 0x7fba5c1618b0 con 0x7fba5c1614d0 2024-03-06T07:26:08.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.949+0000 7fba629f2640 1 --2- 172.21.15.59:0/118685798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba5c0b0e30 0x7fba5c0b1260 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fba5c0ce550 tx=0x7fba5402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d2abb8b0c6af29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.949+0000 7fba611ef640 1 -- 172.21.15.59:0/118685798 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54039070 con 0x7fba5c0b0e30 2024-03-06T07:26:08.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.949+0000 7fba611ef640 1 -- 172.21.15.59:0/118685798 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5402cc80 con 0x7fba5c0b0e30 2024-03-06T07:26:08.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.949+0000 7fba611ef640 1 -- 172.21.15.59:0/118685798 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54034660 con 0x7fba5c0b0e30 2024-03-06T07:26:08.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.950+0000 7fba64455640 1 -- 172.21.15.59:0/118685798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba5c0b0e30 msgr2=0x7fba5c0b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.950+0000 7fba64455640 1 --2- 172.21.15.59:0/118685798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba5c0b0e30 0x7fba5c0b1260 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fba5c0ce550 tx=0x7fba5402bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.950+0000 7fba64455640 1 -- 172.21.15.59:0/118685798 shutdown_connections 2024-03-06T07:26:08.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.950+0000 7fba64455640 1 --2- 172.21.15.59:0/118685798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba5c0b0e30 0x7fba5c0b1260 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.950+0000 7fba64455640 1 --2- 172.21.15.59:0/118685798 >> v2:172.21.15.164:3300/0 conn(0x7fba5c0d4860 0x7fba5c0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.950+0000 7fba64455640 1 -- 172.21.15.59:0/118685798 >> 172.21.15.59:0/118685798 conn(0x7fba5c0d3d40 msgr2=0x7fba5c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.950+0000 7fba64455640 1 -- 172.21.15.59:0/118685798 shutdown_connections 2024-03-06T07:26:08.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.950+0000 7fba64455640 1 -- 172.21.15.59:0/118685798 wait complete. 2024-03-06T07:26:08.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.951+0000 7fba64455640 1 Processor -- start 2024-03-06T07:26:08.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.951+0000 7fba64455640 1 -- start start 2024-03-06T07:26:08.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba64455640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba5c0b0e30 0x7fba5c0cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba64455640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba5c0d4860 0x7fba5c0cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba64455640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba5c1614d0 0x7fba5c0cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba64455640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c0b0910 con 0x7fba5c0b0e30 2024-03-06T07:26:08.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba64455640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c0cfc60 con 0x7fba5c1614d0 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba64455640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c0d2560 con 0x7fba5c0d4860 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba621f1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba5c0b0e30 0x7fba5c0cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba621f1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba5c0b0e30 0x7fba5c0cc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35670/0 (socket says 172.21.15.59:35670) 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba621f1640 1 -- 172.21.15.59:0/3993756986 learned_addr learned my addr 172.21.15.59:0/3993756986 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba629f2640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba5c1614d0 0x7fba5c0cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba619f0640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba5c0d4860 0x7fba5c0cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba621f1640 1 -- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba5c0d4860 msgr2=0x7fba5c0cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba621f1640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba5c0d4860 0x7fba5c0cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba621f1640 1 -- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba5c1614d0 msgr2=0x7fba5c0cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba621f1640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba5c1614d0 0x7fba5c0cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba621f1640 1 -- 172.21.15.59:0/3993756986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c0cc890 con 0x7fba5c0b0e30 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba629f2640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba5c1614d0 0x7fba5c0cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:08.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba619f0640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba5c0d4860 0x7fba5c0cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:08.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba621f1640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba5c0b0e30 0x7fba5c0cc350 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fba5800c430 tx=0x7fba5800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba52ffd640 1 -- 172.21.15.59:0/3993756986 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58017070 con 0x7fba5c0b0e30 2024-03-06T07:26:08.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba52ffd640 1 -- 172.21.15.59:0/3993756986 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba58005ce0 con 0x7fba5c0b0e30 2024-03-06T07:26:08.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba52ffd640 1 -- 172.21.15.59:0/3993756986 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58007310 con 0x7fba5c0b0e30 2024-03-06T07:26:08.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba64455640 1 -- 172.21.15.59:0/3993756986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c0c7870 con 0x7fba5c0b0e30 2024-03-06T07:26:08.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.952+0000 7fba64455640 1 -- 172.21.15.59:0/3993756986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c0c7e30 con 0x7fba5c0b0e30 2024-03-06T07:26:08.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.953+0000 7fba64455640 1 -- 172.21.15.59:0/3993756986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5c068ff0 con 0x7fba5c0b0e30 2024-03-06T07:26:08.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.954+0000 7fba52ffd640 1 -- 172.21.15.59:0/3993756986 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba5801a070 con 0x7fba5c0b0e30 2024-03-06T07:26:08.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.954+0000 7fba52ffd640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fba4006b4d0 0x7fba4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.954+0000 7fba52ffd640 1 -- 172.21.15.59:0/3993756986 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fba58089270 con 0x7fba5c0b0e30 2024-03-06T07:26:08.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.955+0000 7fba619f0640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fba4006b4d0 0x7fba4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.955+0000 7fba619f0640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fba4006b4d0 0x7fba4006d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009930 tx=0x7fba4c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:08.959+0000 7fba52ffd640 1 -- 172.21.15.59:0/3993756986 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba58058480 con 0x7fba5c0b0e30 2024-03-06T07:26:09.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.140+0000 7fba64455640 1 -- 172.21.15.59:0/3993756986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fba5c0c84e0 con 0x7fba5c0b0e30 2024-03-06T07:26:09.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.143+0000 7fba52ffd640 1 -- 172.21.15.59:0/3993756986 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fba58057d80 con 0x7fba5c0b0e30 2024-03-06T07:26:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 -- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fba4006b4d0 msgr2=0x7fba4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fba4006b4d0 0x7fba4006d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009930 tx=0x7fba4c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:09.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 -- 172.21.15.59:0/3993756986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba5c0b0e30 msgr2=0x7fba5c0cc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba5c0b0e30 0x7fba5c0cc350 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fba5800c430 tx=0x7fba5800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 -- 172.21.15.59:0/3993756986 shutdown_connections 2024-03-06T07:26:09.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fba4006b4d0 0x7fba4006d990 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba5c1614d0 0x7fba5c0cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba5c0d4860 0x7fba5c0cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 --2- 172.21.15.59:0/3993756986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fba5c0b0e30 0x7fba5c0cc350 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 -- 172.21.15.59:0/3993756986 >> 172.21.15.59:0/3993756986 conn(0x7fba5c0d3d40 msgr2=0x7fba5c06b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 -- 172.21.15.59:0/3993756986 shutdown_connections 2024-03-06T07:26:09.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.145+0000 7fba64455640 1 -- 172.21.15.59:0/3993756986 wait complete. 2024-03-06T07:26:09.158 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:26:09.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.257+0000 7f166fb8a640 1 Processor -- start 2024-03-06T07:26:09.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.257+0000 7f166fb8a640 1 -- start start 2024-03-06T07:26:09.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166fb8a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f16681424f0 0x7f16680cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166fb8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16680cd7a0 0x7f166815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166fb8a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f166806d330 0x7f166806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166fb8a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16680cd220 con 0x7f16681424f0 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166fb8a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16680cc980 con 0x7f166806d330 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166fb8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16680ccc60 con 0x7f16680cd7a0 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166d926640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f16681424f0 0x7f16680cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166d926640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f16681424f0 0x7f16680cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35686/0 (socket says 172.21.15.59:35686) 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166d926640 1 -- 172.21.15.59:0/2674304488 learned_addr learned my addr 172.21.15.59:0/2674304488 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166e127640 1 --2- 172.21.15.59:0/2674304488 >> v2:172.21.15.164:3300/0 conn(0x7f166806d330 0x7f166806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166d125640 1 --2- 172.21.15.59:0/2674304488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16680cd7a0 0x7f166815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166d926640 1 -- 172.21.15.59:0/2674304488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16680cd7a0 msgr2=0x7f166815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166d926640 1 --2- 172.21.15.59:0/2674304488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16680cd7a0 0x7f166815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166d926640 1 -- 172.21.15.59:0/2674304488 >> v2:172.21.15.164:3300/0 conn(0x7f166806d330 msgr2=0x7f166806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166d926640 1 --2- 172.21.15.59:0/2674304488 >> v2:172.21.15.164:3300/0 conn(0x7f166806d330 0x7f166806d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.258+0000 7f166d926640 1 -- 172.21.15.59:0/2674304488 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16680ccf40 con 0x7f16681424f0 2024-03-06T07:26:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.259+0000 7f166d926640 1 --2- 172.21.15.59:0/2674304488 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16681424f0 0x7f16680cc2f0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f1668070860 tx=0x7f166002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e0389ae5ca2278 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.259+0000 7f166c924640 1 -- 172.21.15.59:0/2674304488 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1660035020 con 0x7f16681424f0 2024-03-06T07:26:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.259+0000 7f166c924640 1 -- 172.21.15.59:0/2674304488 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1660031870 con 0x7f16681424f0 2024-03-06T07:26:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.259+0000 7f166c924640 1 -- 172.21.15.59:0/2674304488 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1660031c40 con 0x7f16681424f0 2024-03-06T07:26:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.260+0000 7f166fb8a640 1 -- 172.21.15.59:0/2674304488 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16681424f0 msgr2=0x7f16680cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.260+0000 7f166fb8a640 1 --2- 172.21.15.59:0/2674304488 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16681424f0 0x7f16680cc2f0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f1668070860 tx=0x7f166002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.260+0000 7f166fb8a640 1 -- 172.21.15.59:0/2674304488 shutdown_connections 2024-03-06T07:26:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.260+0000 7f166fb8a640 1 --2- 172.21.15.59:0/2674304488 >> v2:172.21.15.164:3300/0 conn(0x7f166806d330 0x7f166806d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.260+0000 7f166fb8a640 1 --2- 172.21.15.59:0/2674304488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16680cd7a0 0x7f166815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.260+0000 7f166fb8a640 1 --2- 172.21.15.59:0/2674304488 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16681424f0 0x7f16680cc2f0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.260+0000 7f166fb8a640 1 -- 172.21.15.59:0/2674304488 >> 172.21.15.59:0/2674304488 conn(0x7f16680cf8a0 msgr2=0x7f16680cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.260+0000 7f166fb8a640 1 -- 172.21.15.59:0/2674304488 shutdown_connections 2024-03-06T07:26:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.260+0000 7f166fb8a640 1 -- 172.21.15.59:0/2674304488 wait complete. 2024-03-06T07:26:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.261+0000 7f166fb8a640 1 Processor -- start 2024-03-06T07:26:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.261+0000 7f166fb8a640 1 -- start start 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166fb8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f166806d330 0x7f1668140780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166fb8a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16680cd7a0 0x7f166813fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166fb8a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16681424f0 0x7f166815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166fb8a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1668071ce0 con 0x7f16681424f0 2024-03-06T07:26:09.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166fb8a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1668071b60 con 0x7f16680cd7a0 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166fb8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1668071e60 con 0x7f166806d330 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166e127640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16681424f0 0x7f166815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166e127640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16681424f0 0x7f166815d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35698/0 (socket says 172.21.15.59:35698) 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166e127640 1 -- 172.21.15.59:0/2128596163 learned_addr learned my addr 172.21.15.59:0/2128596163 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166d926640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f166806d330 0x7f1668140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166d125640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16680cd7a0 0x7f166813fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166e127640 1 -- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f166806d330 msgr2=0x7f1668140780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166e127640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f166806d330 0x7f1668140780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166e127640 1 -- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16680cd7a0 msgr2=0x7f166813fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166e127640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16680cd7a0 0x7f166813fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166e127640 1 -- 172.21.15.59:0/2128596163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16680cbf00 con 0x7f16681424f0 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.262+0000 7f166e127640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16681424f0 0x7f166815d9b0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f1658007e60 tx=0x7f165800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.263+0000 7f16667fc640 1 -- 172.21.15.59:0/2128596163 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658014620 con 0x7f16681424f0 2024-03-06T07:26:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.263+0000 7f16667fc640 1 -- 172.21.15.59:0/2128596163 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1658019070 con 0x7f16681424f0 2024-03-06T07:26:09.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.263+0000 7f166fb8a640 1 -- 172.21.15.59:0/2128596163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16680aa040 con 0x7f16681424f0 2024-03-06T07:26:09.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.263+0000 7f16667fc640 1 -- 172.21.15.59:0/2128596163 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658014a20 con 0x7f16681424f0 2024-03-06T07:26:09.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.263+0000 7f166fb8a640 1 -- 172.21.15.59:0/2128596163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16680aa320 con 0x7f16681424f0 2024-03-06T07:26:09.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.263+0000 7f166fb8a640 1 -- 172.21.15.59:0/2128596163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166804f330 con 0x7f16681424f0 2024-03-06T07:26:09.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.264+0000 7f16667fc640 1 -- 172.21.15.59:0/2128596163 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1658014bc0 con 0x7f16681424f0 2024-03-06T07:26:09.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.264+0000 7f16667fc640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f165406b4d0 0x7f165406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.264+0000 7f16667fc640 1 -- 172.21.15.59:0/2128596163 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f165808a700 con 0x7f16681424f0 2024-03-06T07:26:09.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.265+0000 7f166d926640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f165406b4d0 0x7f165406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.265+0000 7f166d926640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f165406b4d0 0x7f165406d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1660036000 tx=0x7f166003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.269+0000 7f16667fc640 1 -- 172.21.15.59:0/2128596163 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1658059910 con 0x7f16681424f0 2024-03-06T07:26:09.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.525+0000 7f166fb8a640 1 -- 172.21.15.59:0/2128596163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f16680d0620 con 0x7f16681424f0 2024-03-06T07:26:09.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.813+0000 7f16667fc640 1 -- 172.21.15.59:0/2128596163 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v249) v1 ==== 132+0+0 (secure 0 0 0) 0x7f1658059210 con 0x7f16681424f0 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.815+0000 7f166fb8a640 1 -- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f165406b4d0 msgr2=0x7f165406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.815+0000 7f166fb8a640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f165406b4d0 0x7f165406d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1660036000 tx=0x7f166003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.815+0000 7f166fb8a640 1 -- 172.21.15.59:0/2128596163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16681424f0 msgr2=0x7f166815d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.815+0000 7f166fb8a640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16681424f0 0x7f166815d9b0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f1658007e60 tx=0x7f165800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.816+0000 7f166fb8a640 1 -- 172.21.15.59:0/2128596163 shutdown_connections 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.816+0000 7f166fb8a640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f165406b4d0 0x7f165406d990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.816+0000 7f166fb8a640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16681424f0 0x7f166815d9b0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.816+0000 7f166fb8a640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16680cd7a0 0x7f166813fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.816+0000 7f166fb8a640 1 --2- 172.21.15.59:0/2128596163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f166806d330 0x7f1668140780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.816+0000 7f166fb8a640 1 -- 172.21.15.59:0/2128596163 >> 172.21.15.59:0/2128596163 conn(0x7f16680cf8a0 msgr2=0x7f166815a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.816+0000 7f166fb8a640 1 -- 172.21.15.59:0/2128596163 shutdown_connections 2024-03-06T07:26:09.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.816+0000 7f166fb8a640 1 -- 172.21.15.59:0/2128596163 wait complete. 2024-03-06T07:26:09.829 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-03-06T07:26:09.830 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:26:09.830 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:09.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.931+0000 7f671f8aa640 1 Processor -- start 2024-03-06T07:26:09.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.931+0000 7f671f8aa640 1 -- start start 2024-03-06T07:26:09.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671f8aa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67180d00c0 0x7f67180d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671f8aa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f671806cf20 0x7f671806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671f8aa640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67180d09e0 con 0x7f67180cf850 2024-03-06T07:26:09.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671f8aa640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671806ca90 con 0x7f67180d00c0 2024-03-06T07:26:09.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671f8aa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671815e380 con 0x7f671806cf20 2024-03-06T07:26:09.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671d646640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67180d00c0 0x7f67180d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671ce45640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f67180cf850 0x7f67180cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:43038/0 (socket says 172.21.15.59:43038) 2024-03-06T07:26:09.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671d646640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67180d00c0 0x7f67180d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35706/0 (socket says 172.21.15.59:35706) 2024-03-06T07:26:09.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671ce45640 1 -- 172.21.15.59:0/2791424967 learned_addr learned my addr 172.21.15.59:0/2791424967 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:09.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671d646640 1 -- 172.21.15.59:0/2791424967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f671806cf20 msgr2=0x7f671806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671d646640 1 --2- 172.21.15.59:0/2791424967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f671806cf20 0x7f671806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671d646640 1 -- 172.21.15.59:0/2791424967 >> v1:172.21.15.59:6789/0 conn(0x7f67180cf850 legacy=0x7f67180cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f671d646640 1 -- 172.21.15.59:0/2791424967 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67180cdc10 con 0x7f67180d00c0 2024-03-06T07:26:09.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.932+0000 7f670ffff640 1 -- 172.21.15.59:0/2791424967 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703214720 0 0) 0x7f67180d09e0 con 0x7f67180cf850 2024-03-06T07:26:09.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.933+0000 7f671d646640 1 --2- 172.21.15.59:0/2791424967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180d00c0 0x7f67180d04a0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f67140097a0 tx=0x7f671402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327e603fc3c94998 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.933+0000 7f670ffff640 1 -- 172.21.15.59:0/2791424967 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6714035020 con 0x7f67180d00c0 2024-03-06T07:26:09.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.933+0000 7f670ffff640 1 -- 172.21.15.59:0/2791424967 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6714031830 con 0x7f67180d00c0 2024-03-06T07:26:09.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.933+0000 7f670ffff640 1 -- 172.21.15.59:0/2791424967 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6714031c00 con 0x7f67180d00c0 2024-03-06T07:26:09.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.934+0000 7f671f8aa640 1 -- 172.21.15.59:0/2791424967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180d00c0 msgr2=0x7f67180d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.934+0000 7f671f8aa640 1 --2- 172.21.15.59:0/2791424967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180d00c0 0x7f67180d04a0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f67140097a0 tx=0x7f671402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.934+0000 7f671f8aa640 1 -- 172.21.15.59:0/2791424967 shutdown_connections 2024-03-06T07:26:09.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.934+0000 7f671f8aa640 1 --2- 172.21.15.59:0/2791424967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f671806cf20 0x7f671806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.934+0000 7f671f8aa640 1 --2- 172.21.15.59:0/2791424967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180d00c0 0x7f67180d04a0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.934+0000 7f671f8aa640 1 -- 172.21.15.59:0/2791424967 >> 172.21.15.59:0/2791424967 conn(0x7f67180b35a0 msgr2=0x7f67180b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.934+0000 7f671f8aa640 1 -- 172.21.15.59:0/2791424967 shutdown_connections 2024-03-06T07:26:09.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.934+0000 7f671f8aa640 1 -- 172.21.15.59:0/2791424967 wait complete. 2024-03-06T07:26:09.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.935+0000 7f671f8aa640 1 Processor -- start 2024-03-06T07:26:09.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.935+0000 7f671f8aa640 1 -- start start 2024-03-06T07:26:09.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671f8aa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f671806cf20 0x7f67180c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671f8aa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67180cf850 0x7f67180b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671f8aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180d00c0 0x7f67180b2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671f8aa640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671806c720 con 0x7f67180d00c0 2024-03-06T07:26:09.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671f8aa640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671806c440 con 0x7f67180cf850 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671f8aa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67180b1030 con 0x7f671806cf20 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671de47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180d00c0 0x7f67180b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671de47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180d00c0 0x7f67180b2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35710/0 (socket says 172.21.15.59:35710) 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671de47640 1 -- 172.21.15.59:0/3409495019 learned_addr learned my addr 172.21.15.59:0/3409495019 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671ce45640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67180cf850 0x7f67180b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671d646640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f671806cf20 0x7f67180c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671de47640 1 -- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f671806cf20 msgr2=0x7f67180c7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671de47640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f671806cf20 0x7f67180c7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671de47640 1 -- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67180cf850 msgr2=0x7f67180b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671de47640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67180cf850 0x7f67180b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671de47640 1 -- 172.21.15.59:0/3409495019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67180c84c0 con 0x7f67180d00c0 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671ce45640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67180cf850 0x7f67180b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671de47640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180d00c0 0x7f67180b2520 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f671000b970 tx=0x7f671000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f670dffb640 1 -- 172.21.15.59:0/3409495019 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710019070 con 0x7f67180d00c0 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f670dffb640 1 -- 172.21.15.59:0/3409495019 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f671000cb60 con 0x7f67180d00c0 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f671f8aa640 1 -- 172.21.15.59:0/3409495019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671815ea60 con 0x7f67180d00c0 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.936+0000 7f670dffb640 1 -- 172.21.15.59:0/3409495019 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710014440 con 0x7f67180d00c0 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.937+0000 7f671f8aa640 1 -- 172.21.15.59:0/3409495019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671815f020 con 0x7f67180d00c0 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.938+0000 7f671f8aa640 1 -- 172.21.15.59:0/3409495019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67180b3d60 con 0x7f67180d00c0 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.938+0000 7f670dffb640 1 -- 172.21.15.59:0/3409495019 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f671001c070 con 0x7f67180d00c0 2024-03-06T07:26:09.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.938+0000 7f670dffb640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66fc06b4d0 0x7f66fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.938+0000 7f670dffb640 1 -- 172.21.15.59:0/3409495019 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6710089bf0 con 0x7f67180d00c0 2024-03-06T07:26:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.939+0000 7f671d646640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66fc06b4d0 0x7f66fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.939+0000 7f671d646640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66fc06b4d0 0x7f66fc06d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6714036000 tx=0x7f6714008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:09.943+0000 7f670dffb640 1 -- 172.21.15.59:0/3409495019 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67180b3d60 con 0x7f67180d00c0 2024-03-06T07:26:10.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.126+0000 7f671f8aa640 1 -- 172.21.15.59:0/3409495019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f671804f330 con 0x7f67180d00c0 2024-03-06T07:26:10.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.127+0000 7f670dffb640 1 -- 172.21.15.59:0/3409495019 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+13009 (secure 0 0 0) 0x7f6710058d70 con 0x7f67180d00c0 2024-03-06T07:26:10.129 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:10.129 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":84,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:26:08.806498+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6834/2521311582":"2024-03-07T07:26:05.746333+0000","172.21.15.59:6833/2521311582":"2024-03-07T07:26:05.746333+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 -- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66fc06b4d0 msgr2=0x7f66fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66fc06b4d0 0x7f66fc06d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6714036000 tx=0x7f6714008030 comp rx=0 tx=0).stop 2024-03-06T07:26:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 -- 172.21.15.59:0/3409495019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180d00c0 msgr2=0x7f67180b2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180d00c0 0x7f67180b2520 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f671000b970 tx=0x7f671000be30 comp rx=0 tx=0).stop 2024-03-06T07:26:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 -- 172.21.15.59:0/3409495019 shutdown_connections 2024-03-06T07:26:10.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f66fc06b4d0 0x7f66fc06d990 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67180d00c0 0x7f67180b2520 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67180cf850 0x7f67180b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 --2- 172.21.15.59:0/3409495019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f671806cf20 0x7f67180c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 -- 172.21.15.59:0/3409495019 >> 172.21.15.59:0/3409495019 conn(0x7f67180b35a0 msgr2=0x7f67180cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 -- 172.21.15.59:0/3409495019 shutdown_connections 2024-03-06T07:26:10.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.129+0000 7f671f8aa640 1 -- 172.21.15.59:0/3409495019 wait complete. 2024-03-06T07:26:10.142 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:10.142 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:10.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.244+0000 7fc30cbac640 1 Processor -- start 2024-03-06T07:26:10.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.245+0000 7fc30cbac640 1 -- start start 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.245+0000 7fc30cbac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc30806cab0 0x7fc3080b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.245+0000 7fc30cbac640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30806d030 con 0x7fc30815def0 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.245+0000 7fc30cbac640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30806d310 con 0x7fc3080cea50 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.245+0000 7fc30cbac640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3080cf1c0 con 0x7fc30806cab0 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.245+0000 7fc306d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc30806cab0 0x7fc3080b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.246+0000 7fc30659c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc30815def0 0x7fc3080ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:43048/0 (socket says 172.21.15.59:43048) 2024-03-06T07:26:10.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.246+0000 7fc30659c640 1 -- 172.21.15.59:0/1113027787 learned_addr learned my addr 172.21.15.59:0/1113027787 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.246+0000 7fc30559a640 1 -- 172.21.15.59:0/1113027787 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016738120 0 0) 0x7fc30806d030 con 0x7fc30815def0 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.246+0000 7fc30559a640 1 -- 172.21.15.59:0/1113027787 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f4003520 con 0x7fc30815def0 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.246+0000 7fc306d9d640 1 -- 172.21.15.59:0/1113027787 >> v1:172.21.15.164:6789/0 conn(0x7fc3080cea50 legacy=0x7fc3080ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.246+0000 7fc306d9d640 1 -- 172.21.15.59:0/1113027787 >> v1:172.21.15.59:6789/0 conn(0x7fc30815def0 legacy=0x7fc3080ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.246+0000 7fc306d9d640 1 -- 172.21.15.59:0/1113027787 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3080cc110 con 0x7fc30806cab0 2024-03-06T07:26:10.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.247+0000 7fc306d9d640 1 --2- 172.21.15.59:0/1113027787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30806cab0 0x7fc3080b41b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009d90 tx=0x7fc2f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e470483780073e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.247+0000 7fc30559a640 1 -- 172.21.15.59:0/1113027787 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8039070 con 0x7fc30806cab0 2024-03-06T07:26:10.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.247+0000 7fc30559a640 1 -- 172.21.15.59:0/1113027787 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f802cc60 con 0x7fc30806cab0 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.247+0000 7fc30559a640 1 -- 172.21.15.59:0/1113027787 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8034640 con 0x7fc30806cab0 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.247+0000 7fc30cbac640 1 -- 172.21.15.59:0/1113027787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30806cab0 msgr2=0x7fc3080b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.247+0000 7fc30cbac640 1 --2- 172.21.15.59:0/1113027787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30806cab0 0x7fc3080b41b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009d90 tx=0x7fc2f802beb0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.247+0000 7fc30cbac640 1 -- 172.21.15.59:0/1113027787 shutdown_connections 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.247+0000 7fc30cbac640 1 --2- 172.21.15.59:0/1113027787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30806cab0 0x7fc3080b41b0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.247+0000 7fc30cbac640 1 -- 172.21.15.59:0/1113027787 >> 172.21.15.59:0/1113027787 conn(0x7fc3080b3690 msgr2=0x7fc3080b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.247+0000 7fc30cbac640 1 -- 172.21.15.59:0/1113027787 shutdown_connections 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.247+0000 7fc30cbac640 1 -- 172.21.15.59:0/1113027787 wait complete. 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.249+0000 7fc30cbac640 1 Processor -- start 2024-03-06T07:26:10.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.249+0000 7fc30cbac640 1 -- start start 2024-03-06T07:26:10.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.249+0000 7fc30cbac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30806cab0 0x7fc30815ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.249+0000 7fc30cbac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3080cea50 0x7fc30814fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.249+0000 7fc30cbac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc30815def0 0x7fc3081503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.249+0000 7fc30cbac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30815e2d0 con 0x7fc30806cab0 2024-03-06T07:26:10.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.249+0000 7fc30cbac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3080cc760 con 0x7fc3080cea50 2024-03-06T07:26:10.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.249+0000 7fc30cbac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc30815e450 con 0x7fc30815def0 2024-03-06T07:26:10.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.249+0000 7fc30659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30806cab0 0x7fc30815ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc30659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30806cab0 0x7fc30815ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35738/0 (socket says 172.21.15.59:35738) 2024-03-06T07:26:10.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc30659c640 1 -- 172.21.15.59:0/2259381711 learned_addr learned my addr 172.21.15.59:0/2259381711 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:10.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc30659c640 1 -- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc30815def0 msgr2=0x7fc3081503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc30659c640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc30815def0 0x7fc3081503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc30659c640 1 -- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3080cea50 msgr2=0x7fc30814fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc305d9b640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3080cea50 0x7fc30814fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc30659c640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3080cea50 0x7fc30814fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc30659c640 1 -- 172.21.15.59:0/2259381711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3080c8b90 con 0x7fc30806cab0 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc30659c640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30806cab0 0x7fc30815ced0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc30000c3c0 tx=0x7fc30000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc2feffd640 1 -- 172.21.15.59:0/2259381711 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300017070 con 0x7fc30806cab0 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc2feffd640 1 -- 172.21.15.59:0/2259381711 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc300005ce0 con 0x7fc30806cab0 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc30cbac640 1 -- 172.21.15.59:0/2259381711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3080c6d10 con 0x7fc30806cab0 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc2feffd640 1 -- 172.21.15.59:0/2259381711 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3000072f0 con 0x7fc30806cab0 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.250+0000 7fc30cbac640 1 -- 172.21.15.59:0/2259381711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc30814e300 con 0x7fc30806cab0 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.251+0000 7fc30cbac640 1 -- 172.21.15.59:0/2259381711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc30809fcf0 con 0x7fc30806cab0 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.252+0000 7fc2feffd640 1 -- 172.21.15.59:0/2259381711 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc30001a070 con 0x7fc30806cab0 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.252+0000 7fc2feffd640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc2e806b4d0 0x7fc2e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.252+0000 7fc2feffd640 1 -- 172.21.15.59:0/2259381711 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc3000892a0 con 0x7fc30806cab0 2024-03-06T07:26:10.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.252+0000 7fc305d9b640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc2e806b4d0 0x7fc2e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.253+0000 7fc305d9b640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc2e806b4d0 0x7fc2e806d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc3080b3080 tx=0x7fc2f0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.256+0000 7fc2feffd640 1 -- 172.21.15.59:0/2259381711 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3080c8b90 con 0x7fc30806cab0 2024-03-06T07:26:10.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.438+0000 7fc30cbac640 1 -- 172.21.15.59:0/2259381711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc30806fa70 con 0x7fc30806cab0 2024-03-06T07:26:10.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.438+0000 7fc2feffd640 1 -- 172.21.15.59:0/2259381711 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc300058420 con 0x7fc30806cab0 2024-03-06T07:26:10.440 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:26:10.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 -- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc2e806b4d0 msgr2=0x7fc2e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc2e806b4d0 0x7fc2e806d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc3080b3080 tx=0x7fc2f0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 -- 172.21.15.59:0/2259381711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30806cab0 msgr2=0x7fc30815ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30806cab0 0x7fc30815ced0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc30000c3c0 tx=0x7fc30000c880 comp rx=0 tx=0).stop 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 -- 172.21.15.59:0/2259381711 shutdown_connections 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc2e806b4d0 0x7fc2e806d990 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc30815def0 0x7fc3081503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3080cea50 0x7fc30814fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 --2- 172.21.15.59:0/2259381711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc30806cab0 0x7fc30815ced0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 -- 172.21.15.59:0/2259381711 >> 172.21.15.59:0/2259381711 conn(0x7fc3080b3690 msgr2=0x7fc30815e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 -- 172.21.15.59:0/2259381711 shutdown_connections 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.440+0000 7fc30cbac640 1 -- 172.21.15.59:0/2259381711 wait complete. 2024-03-06T07:26:10.453 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:26:10.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.552+0000 7f9bfc381640 1 Processor -- start 2024-03-06T07:26:10.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.552+0000 7f9bfc381640 1 -- start start 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.553+0000 7f9bfc381640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9bf4161440 0x7f9bf40d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.553+0000 7f9bfc381640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf40d4480 0x7f9bf40d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.553+0000 7f9bfc381640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf40d4cb0 con 0x7f9bf40d2870 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.553+0000 7f9bfc381640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf40cf420 con 0x7f9bf4161440 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.553+0000 7f9bfc381640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf40d49d0 con 0x7f9bf40d4480 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.554+0000 7f9bfa11d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9bf4161440 0x7f9bf40d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.554+0000 7f9bfa11d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9bf4161440 0x7f9bf40d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35746/0 (socket says 172.21.15.59:35746) 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.554+0000 7f9bfa11d640 1 -- 172.21.15.59:0/1225717164 learned_addr learned my addr 172.21.15.59:0/1225717164 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.554+0000 7f9bf991c640 1 --2- 172.21.15.59:0/1225717164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf40d4480 0x7f9bf40d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.554+0000 7f9bfa11d640 1 -- 172.21.15.59:0/1225717164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf40d4480 msgr2=0x7f9bf40d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.554+0000 7f9bfa11d640 1 --2- 172.21.15.59:0/1225717164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf40d4480 0x7f9bf40d2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.554+0000 7f9bfa11d640 1 -- 172.21.15.59:0/1225717164 >> v1:172.21.15.164:6789/0 conn(0x7f9bf40d2870 legacy=0x7f9bf40d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.554+0000 7f9bfa11d640 1 -- 172.21.15.59:0/1225717164 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf40cf6c0 con 0x7f9bf4161440 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bfa11d640 1 --2- 172.21.15.59:0/1225717164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf4161440 0x7f9bf40d1df0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009870 tx=0x7f9bec02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5fd2f366a8b67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bf911b640 1 -- 172.21.15.59:0/1225717164 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec038070 con 0x7f9bf4161440 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bf911b640 1 -- 172.21.15.59:0/1225717164 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec031740 con 0x7f9bf4161440 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bf911b640 1 -- 172.21.15.59:0/1225717164 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec02f050 con 0x7f9bf4161440 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bfc381640 1 -- 172.21.15.59:0/1225717164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf4161440 msgr2=0x7f9bf40d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bfc381640 1 --2- 172.21.15.59:0/1225717164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf4161440 0x7f9bf40d1df0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009870 tx=0x7f9bec02b7e0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bfc381640 1 -- 172.21.15.59:0/1225717164 shutdown_connections 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bfc381640 1 --2- 172.21.15.59:0/1225717164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf40d4480 0x7f9bf40d2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bfc381640 1 --2- 172.21.15.59:0/1225717164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf4161440 0x7f9bf40d1df0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bfc381640 1 -- 172.21.15.59:0/1225717164 >> 172.21.15.59:0/1225717164 conn(0x7f9bf409d030 msgr2=0x7f9bf40d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bfc381640 1 -- 172.21.15.59:0/1225717164 shutdown_connections 2024-03-06T07:26:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.555+0000 7f9bfc381640 1 -- 172.21.15.59:0/1225717164 wait complete. 2024-03-06T07:26:10.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bfc381640 1 Processor -- start 2024-03-06T07:26:10.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bfc381640 1 -- start start 2024-03-06T07:26:10.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bfc381640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf40d2870 0x7f9bf415d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bfc381640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf40d4480 0x7f9bf40a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bfc381640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4161440 0x7f9bf40a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bfc381640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf41619a0 con 0x7f9bf40d4480 2024-03-06T07:26:10.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bfc381640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4161820 con 0x7f9bf40d2870 2024-03-06T07:26:10.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bfc381640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf40d1590 con 0x7f9bf4161440 2024-03-06T07:26:10.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bf991c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf40d4480 0x7f9bf40a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bf991c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf40d4480 0x7f9bf40a1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35754/0 (socket says 172.21.15.59:35754) 2024-03-06T07:26:10.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bf991c640 1 -- 172.21.15.59:0/194498568 learned_addr learned my addr 172.21.15.59:0/194498568 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bfa11d640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf40d2870 0x7f9bf415d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.557+0000 7f9bfa91e640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4161440 0x7f9bf40a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.558+0000 7f9bf991c640 1 -- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4161440 msgr2=0x7f9bf40a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.558+0000 7f9bf991c640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4161440 0x7f9bf40a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.558+0000 7f9bf991c640 1 -- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf40d2870 msgr2=0x7f9bf415d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.558+0000 7f9bf991c640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf40d2870 0x7f9bf415d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.558+0000 7f9bf991c640 1 -- 172.21.15.59:0/194498568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf40cb090 con 0x7f9bf40d4480 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.558+0000 7f9bf991c640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf40d4480 0x7f9bf40a1250 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f9be800c7f0 tx=0x7f9be800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.558+0000 7f9bf2ffd640 1 -- 172.21.15.59:0/194498568 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be8015070 con 0x7f9bf40d4480 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.558+0000 7f9bf2ffd640 1 -- 172.21.15.59:0/194498568 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be8004220 con 0x7f9bf40d4480 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.558+0000 7f9bfc381640 1 -- 172.21.15.59:0/194498568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf40cb370 con 0x7f9bf40d4480 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.558+0000 7f9bf2ffd640 1 -- 172.21.15.59:0/194498568 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be8007620 con 0x7f9bf40d4480 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.558+0000 7f9bfc381640 1 -- 172.21.15.59:0/194498568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf40cb930 con 0x7f9bf40d4480 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.559+0000 7f9bfc381640 1 -- 172.21.15.59:0/194498568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf404f330 con 0x7f9bf40d4480 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.560+0000 7f9bf2ffd640 1 -- 172.21.15.59:0/194498568 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9be80079d0 con 0x7f9bf40d4480 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.560+0000 7f9bf2ffd640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9be006b4d0 0x7f9be006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.560+0000 7f9bf2ffd640 1 -- 172.21.15.59:0/194498568 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9be8089e30 con 0x7f9bf40d4480 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.560+0000 7f9bfa11d640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9be006b4d0 0x7f9be006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.561+0000 7f9bfa11d640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9be006b4d0 0x7f9be006d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9bec035000 tx=0x7f9bec008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.564+0000 7f9bf2ffd640 1 -- 172.21.15.59:0/194498568 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9be80589f0 con 0x7f9bf40d4480 2024-03-06T07:26:10.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:10.817+0000 7f9bfc381640 1 -- 172.21.15.59:0/194498568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f9bf4160640 con 0x7f9bf40d4480 2024-03-06T07:26:11.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.824+0000 7f9bf2ffd640 1 -- 172.21.15.59:0/194498568 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v251) v1 ==== 137+0+0 (secure 0 0 0) 0x7f9be805ca90 con 0x7f9bf40d4480 2024-03-06T07:26:11.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 -- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9be006b4d0 msgr2=0x7f9be006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9be006b4d0 0x7f9be006d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9bec035000 tx=0x7f9bec008030 comp rx=0 tx=0).stop 2024-03-06T07:26:11.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 -- 172.21.15.59:0/194498568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf40d4480 msgr2=0x7f9bf40a1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf40d4480 0x7f9bf40a1250 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f9be800c7f0 tx=0x7f9be800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 -- 172.21.15.59:0/194498568 shutdown_connections 2024-03-06T07:26:11.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9be006b4d0 0x7f9be006d990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4161440 0x7f9bf40a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf40d4480 0x7f9bf40a1250 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 --2- 172.21.15.59:0/194498568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf40d2870 0x7f9bf415d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 -- 172.21.15.59:0/194498568 >> 172.21.15.59:0/194498568 conn(0x7f9bf409d030 msgr2=0x7f9bf406caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 -- 172.21.15.59:0/194498568 shutdown_connections 2024-03-06T07:26:11.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.827+0000 7f9bfc381640 1 -- 172.21.15.59:0/194498568 wait complete. 2024-03-06T07:26:11.842 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:26:11.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.950+0000 7fd8d40c3640 1 Processor -- start 2024-03-06T07:26:11.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.951+0000 7fd8d40c3640 1 -- start start 2024-03-06T07:26:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.951+0000 7fd8d40c3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd8cc15def0 0x7fd8cc0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.951+0000 7fd8d40c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8cc06cab0 0x7fd8cc0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.951+0000 7fd8d40c3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd8cc0ce880 0x7fd8cc0cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.951+0000 7fd8d40c3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8cc0cecf0 con 0x7fd8cc15def0 2024-03-06T07:26:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.951+0000 7fd8d40c3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8cc0cf1c0 con 0x7fd8cc0ce880 2024-03-06T07:26:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.951+0000 7fd8d40c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc0cc760 con 0x7fd8cc06cab0 2024-03-06T07:26:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.952+0000 7fd8d1e5f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd8cc15def0 0x7fd8cc0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.952+0000 7fd8d1e5f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd8cc15def0 0x7fd8cc0b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35770/0 (socket says 172.21.15.59:35770) 2024-03-06T07:26:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.952+0000 7fd8d1e5f640 1 -- 172.21.15.59:0/2784086506 learned_addr learned my addr 172.21.15.59:0/2784086506 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.952+0000 7fd8d2660640 1 --2- 172.21.15.59:0/2784086506 >> v2:172.21.15.164:3300/0 conn(0x7fd8cc0ce880 0x7fd8cc0cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.952+0000 7fd8d165e640 1 --2- 172.21.15.59:0/2784086506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8cc06cab0 0x7fd8cc0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.952+0000 7fd8d1e5f640 1 -- 172.21.15.59:0/2784086506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8cc06cab0 msgr2=0x7fd8cc0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.952+0000 7fd8d1e5f640 1 --2- 172.21.15.59:0/2784086506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8cc06cab0 0x7fd8cc0ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.952+0000 7fd8d1e5f640 1 -- 172.21.15.59:0/2784086506 >> v2:172.21.15.164:3300/0 conn(0x7fd8cc0ce880 msgr2=0x7fd8cc0cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.952+0000 7fd8d1e5f640 1 --2- 172.21.15.59:0/2784086506 >> v2:172.21.15.164:3300/0 conn(0x7fd8cc0ce880 0x7fd8cc0cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.952+0000 7fd8d1e5f640 1 -- 172.21.15.59:0/2784086506 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc06d030 con 0x7fd8cc15def0 2024-03-06T07:26:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d1e5f640 1 --2- 172.21.15.59:0/2784086506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8cc15def0 0x7fd8cc0b41b0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc071e70 tx=0x7fd8c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8395521375927 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d0e5d640 1 -- 172.21.15.59:0/2784086506 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8034070 con 0x7fd8cc15def0 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d0e5d640 1 -- 172.21.15.59:0/2784086506 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c8030860 con 0x7fd8cc15def0 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d0e5d640 1 -- 172.21.15.59:0/2784086506 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8030c30 con 0x7fd8cc15def0 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d40c3640 1 -- 172.21.15.59:0/2784086506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8cc15def0 msgr2=0x7fd8cc0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d40c3640 1 --2- 172.21.15.59:0/2784086506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8cc15def0 0x7fd8cc0b41b0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc071e70 tx=0x7fd8c802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d40c3640 1 -- 172.21.15.59:0/2784086506 shutdown_connections 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d40c3640 1 --2- 172.21.15.59:0/2784086506 >> v2:172.21.15.164:3300/0 conn(0x7fd8cc0ce880 0x7fd8cc0cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d40c3640 1 --2- 172.21.15.59:0/2784086506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8cc06cab0 0x7fd8cc0ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d40c3640 1 --2- 172.21.15.59:0/2784086506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8cc15def0 0x7fd8cc0b41b0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d40c3640 1 -- 172.21.15.59:0/2784086506 >> 172.21.15.59:0/2784086506 conn(0x7fd8cc0b3690 msgr2=0x7fd8cc0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.953+0000 7fd8d40c3640 1 -- 172.21.15.59:0/2784086506 shutdown_connections 2024-03-06T07:26:11.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.954+0000 7fd8d40c3640 1 -- 172.21.15.59:0/2784086506 wait complete. 2024-03-06T07:26:11.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.955+0000 7fd8d40c3640 1 Processor -- start 2024-03-06T07:26:11.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.955+0000 7fd8d40c3640 1 -- start start 2024-03-06T07:26:11.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.955+0000 7fd8d40c3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8cc06cab0 0x7fd8cc0cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.955+0000 7fd8d40c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8cc0ce880 0x7fd8cc0b6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.955+0000 7fd8d40c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8cc15def0 0x7fd8cc0b6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.955+0000 7fd8d40c3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc0cc3f0 con 0x7fd8cc0ce880 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.955+0000 7fd8d40c3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc0cc110 con 0x7fd8cc06cab0 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.955+0000 7fd8d40c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc15e2d0 con 0x7fd8cc15def0 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d165e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8cc0ce880 0x7fd8cc0b6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d165e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8cc0ce880 0x7fd8cc0b6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35772/0 (socket says 172.21.15.59:35772) 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d165e640 1 -- 172.21.15.59:0/3980118957 learned_addr learned my addr 172.21.15.59:0/3980118957 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d2660640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8cc15def0 0x7fd8cc0b6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d1e5f640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8cc06cab0 0x7fd8cc0cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d165e640 1 -- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8cc15def0 msgr2=0x7fd8cc0b6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d165e640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8cc15def0 0x7fd8cc0b6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d165e640 1 -- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8cc06cab0 msgr2=0x7fd8cc0cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d165e640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8cc06cab0 0x7fd8cc0cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d165e640 1 -- 172.21.15.59:0/3980118957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc0c6230 con 0x7fd8cc0ce880 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d165e640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8cc0ce880 0x7fd8cc0b6740 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00d800 tx=0x7fd8bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8c27fc640 1 -- 172.21.15.59:0/3980118957 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc00b6c0 con 0x7fd8cc0ce880 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8c27fc640 1 -- 172.21.15.59:0/3980118957 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8bc019070 con 0x7fd8cc0ce880 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d40c3640 1 -- 172.21.15.59:0/3980118957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8cc0c6510 con 0x7fd8cc0ce880 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8c27fc640 1 -- 172.21.15.59:0/3980118957 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc00ba80 con 0x7fd8cc0ce880 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.956+0000 7fd8d40c3640 1 -- 172.21.15.59:0/3980118957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8cc0c6ad0 con 0x7fd8cc0ce880 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.957+0000 7fd8d40c3640 1 -- 172.21.15.59:0/3980118957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8cc09fcf0 con 0x7fd8cc0ce880 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.958+0000 7fd8c27fc640 1 -- 172.21.15.59:0/3980118957 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8bc00bc20 con 0x7fd8cc0ce880 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.958+0000 7fd8c27fc640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8b006b4d0 0x7fd8b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.958+0000 7fd8c27fc640 1 -- 172.21.15.59:0/3980118957 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd8bc08a3b0 con 0x7fd8cc0ce880 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.958+0000 7fd8d1e5f640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8b006b4d0 0x7fd8b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.959+0000 7fd8d1e5f640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8b006b4d0 0x7fd8b006d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8036000 tx=0x7fd8c803b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:11.962+0000 7fd8c27fc640 1 -- 172.21.15.59:0/3980118957 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8bc059530 con 0x7fd8cc0ce880 2024-03-06T07:26:12.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.142+0000 7fd8d40c3640 1 -- 172.21.15.59:0/3980118957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd8cc068ff0 con 0x7fd8cc0ce880 2024-03-06T07:26:12.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.145+0000 7fd8c27fc640 1 -- 172.21.15.59:0/3980118957 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd8bc058e30 con 0x7fd8cc0ce880 2024-03-06T07:26:12.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 -- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8b006b4d0 msgr2=0x7fd8b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8b006b4d0 0x7fd8b006d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8036000 tx=0x7fd8c803b040 comp rx=0 tx=0).stop 2024-03-06T07:26:12.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 -- 172.21.15.59:0/3980118957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8cc0ce880 msgr2=0x7fd8cc0b6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8cc0ce880 0x7fd8cc0b6740 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00d800 tx=0x7fd8bc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 -- 172.21.15.59:0/3980118957 shutdown_connections 2024-03-06T07:26:12.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd8b006b4d0 0x7fd8b006d990 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8cc15def0 0x7fd8cc0b6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8cc0ce880 0x7fd8cc0b6740 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 --2- 172.21.15.59:0/3980118957 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8cc06cab0 0x7fd8cc0cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 -- 172.21.15.59:0/3980118957 >> 172.21.15.59:0/3980118957 conn(0x7fd8cc0b3690 msgr2=0x7fd8cc06ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 -- 172.21.15.59:0/3980118957 shutdown_connections 2024-03-06T07:26:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.147+0000 7fd8d40c3640 1 -- 172.21.15.59:0/3980118957 wait complete. 2024-03-06T07:26:12.160 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:12.160 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:12.206 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:12.252 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:12.253 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:26:12.253 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:12.267 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:12.267 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:26:12.268 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:12.284 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:12.285 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:26:12.285 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:12.322 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:12.323 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:26:12.323 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:12.341 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:12.342 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:26:12.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:12.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.463+0000 7f0c9b0af640 1 Processor -- start 2024-03-06T07:26:12.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.463+0000 7f0c9b0af640 1 -- start start 2024-03-06T07:26:12.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.464+0000 7f0c9b0af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c940d00c0 0x7f0c940d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.464+0000 7f0c9b0af640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0c940cf850 0x7f0c940cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.464+0000 7f0c9b0af640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c940d09e0 con 0x7f0c940ce1b0 2024-03-06T07:26:12.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.464+0000 7f0c9b0af640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9406ca90 con 0x7f0c940cf850 2024-03-06T07:26:12.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.464+0000 7f0c9b0af640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9406df30 con 0x7f0c940d00c0 2024-03-06T07:26:12.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.464+0000 7f0c93fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0c940cf850 0x7f0c940cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.464+0000 7f0c9964c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0c940ce1b0 0x7f0c940ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:36284/0 (socket says 172.21.15.59:36284) 2024-03-06T07:26:12.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.464+0000 7f0c9964c640 1 -- 172.21.15.59:0/1085127655 learned_addr learned my addr 172.21.15.59:0/1085127655 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:12.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.464+0000 7f0c93fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0c940cf850 0x7f0c940cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:33276/0 (socket says 172.21.15.59:33276) 2024-03-06T07:26:12.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.464+0000 7f0c98e4b640 1 --2- 172.21.15.59:0/1085127655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c940d00c0 0x7f0c940d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.465+0000 7f0c93fff640 1 -- 172.21.15.59:0/1085127655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c940d00c0 msgr2=0x7f0c940d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.465+0000 7f0c93fff640 1 --2- 172.21.15.59:0/1085127655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c940d00c0 0x7f0c940d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.465+0000 7f0c93fff640 1 -- 172.21.15.59:0/1085127655 >> v1:172.21.15.164:6789/0 conn(0x7f0c940ce1b0 legacy=0x7f0c940ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.465+0000 7f0c937fe640 1 -- 172.21.15.59:0/1085127655 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033395582 0 0) 0x7f0c940d09e0 con 0x7f0c940ce1b0 2024-03-06T07:26:12.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.465+0000 7f0c93fff640 1 -- 172.21.15.59:0/1085127655 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9415e380 con 0x7f0c940cf850 2024-03-06T07:26:12.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.465+0000 7f0c98e4b640 1 --2- 172.21.15.59:0/1085127655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c940d00c0 0x7f0c940d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:12.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.465+0000 7f0c93fff640 1 --2- 172.21.15.59:0/1085127655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c940cf850 0x7f0c940cd3c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0c880097a0 tx=0x7f0c8802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=372c49349c43c291 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.466+0000 7f0c937fe640 1 -- 172.21.15.59:0/1085127655 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88035020 con 0x7f0c940cf850 2024-03-06T07:26:12.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.466+0000 7f0c937fe640 1 -- 172.21.15.59:0/1085127655 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c88031830 con 0x7f0c940cf850 2024-03-06T07:26:12.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.466+0000 7f0c937fe640 1 -- 172.21.15.59:0/1085127655 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88031c00 con 0x7f0c940cf850 2024-03-06T07:26:12.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.466+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1085127655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c940cf850 msgr2=0x7f0c940cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.466+0000 7f0c9b0af640 1 --2- 172.21.15.59:0/1085127655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c940cf850 0x7f0c940cd3c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0c880097a0 tx=0x7f0c8802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.466+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1085127655 shutdown_connections 2024-03-06T07:26:12.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.466+0000 7f0c9b0af640 1 --2- 172.21.15.59:0/1085127655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c940cf850 0x7f0c940cd3c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.466+0000 7f0c9b0af640 1 --2- 172.21.15.59:0/1085127655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c940d00c0 0x7f0c940d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.466+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1085127655 >> 172.21.15.59:0/1085127655 conn(0x7f0c940b35a0 msgr2=0x7f0c940b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.466+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1085127655 shutdown_connections 2024-03-06T07:26:12.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.466+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1085127655 wait complete. 2024-03-06T07:26:12.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.467+0000 7f0c9b0af640 1 Processor -- start 2024-03-06T07:26:12.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.467+0000 7f0c9b0af640 1 -- start start 2024-03-06T07:26:12.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.468+0000 7f0c9b0af640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c940ce1b0 0x7f0c941584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.468+0000 7f0c9b0af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c940cf850 0x7f0c940b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.468+0000 7f0c9b0af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c940d00c0 0x7f0c940b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.468+0000 7f0c9b0af640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9406c440 con 0x7f0c940d00c0 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.468+0000 7f0c9b0af640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9406c160 con 0x7f0c940ce1b0 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.468+0000 7f0c9b0af640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9406c720 con 0x7f0c940cf850 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.468+0000 7f0c9964c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c940d00c0 0x7f0c940b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.468+0000 7f0c9964c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c940d00c0 0x7f0c940b6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35774/0 (socket says 172.21.15.59:35774) 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.468+0000 7f0c9964c640 1 -- 172.21.15.59:0/1719149464 learned_addr learned my addr 172.21.15.59:0/1719149464 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.468+0000 7f0c98e4b640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c940ce1b0 0x7f0c941584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.468+0000 7f0c93fff640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c940cf850 0x7f0c940b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c9964c640 1 -- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c940cf850 msgr2=0x7f0c940b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c9964c640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c940cf850 0x7f0c940b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c9964c640 1 -- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c940ce1b0 msgr2=0x7f0c941584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c9964c640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c940ce1b0 0x7f0c941584c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c9964c640 1 -- 172.21.15.59:0/1719149464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c940c9ac0 con 0x7f0c940d00c0 2024-03-06T07:26:12.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c98e4b640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c940ce1b0 0x7f0c941584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:12.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c9964c640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c940d00c0 0x7f0c940b6bb0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400c450 tx=0x7f0c8400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c917fa640 1 -- 172.21.15.59:0/1719149464 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c840074e0 con 0x7f0c940d00c0 2024-03-06T07:26:12.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c917fa640 1 -- 172.21.15.59:0/1719149464 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c840047e0 con 0x7f0c940d00c0 2024-03-06T07:26:12.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1719149464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c940c9da0 con 0x7f0c940d00c0 2024-03-06T07:26:12.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c917fa640 1 -- 172.21.15.59:0/1719149464 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c84004bd0 con 0x7f0c940d00c0 2024-03-06T07:26:12.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.469+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1719149464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c940ca360 con 0x7f0c940d00c0 2024-03-06T07:26:12.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.470+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1719149464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9404f330 con 0x7f0c940d00c0 2024-03-06T07:26:12.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.470+0000 7f0c917fa640 1 -- 172.21.15.59:0/1719149464 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c84011070 con 0x7f0c940d00c0 2024-03-06T07:26:12.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.471+0000 7f0c917fa640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c8006b4d0 0x7f0c8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.471+0000 7f0c917fa640 1 -- 172.21.15.59:0/1719149464 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0c84089a80 con 0x7f0c940d00c0 2024-03-06T07:26:12.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.471+0000 7f0c98e4b640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c8006b4d0 0x7f0c8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.471+0000 7f0c98e4b640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c8006b4d0 0x7f0c8006d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0c941f7420 tx=0x7f0c8c00b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.475+0000 7f0c917fa640 1 -- 172.21.15.59:0/1719149464 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c8401a2b0 con 0x7f0c940d00c0 2024-03-06T07:26:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.696+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1719149464 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c940cbf30 con 0x7f0c940d00c0 2024-03-06T07:26:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.696+0000 7f0c917fa640 1 -- 172.21.15.59:0/1719149464 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f0c84058c90 con 0x7f0c940d00c0 2024-03-06T07:26:12.698 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:12.698 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5957,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/2865818497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2865818497},{"type":"v1","addr":"172.21.15.164:6836","nonce":2865818497}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/1805325807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":1805325807},{"type":"v1","addr":"172.21.15.164:6837","nonce":1805325807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6999,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3345771250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3345771250},{"type":"v1","addr":"172.21.15.59:6835","nonce":3345771250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":7326,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/3927785477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":3927785477},{"type":"v1","addr":"172.21.15.59:6834","nonce":3927785477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[]} 2024-03-06T07:26:12.699 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 44 2024-03-06T07:26:12.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c8006b4d0 msgr2=0x7f0c8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c8006b4d0 0x7f0c8006d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0c941f7420 tx=0x7f0c8c00b040 comp rx=0 tx=0).stop 2024-03-06T07:26:12.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1719149464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c940d00c0 msgr2=0x7f0c940b6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c940d00c0 0x7f0c940b6bb0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400c450 tx=0x7f0c8400c910 comp rx=0 tx=0).stop 2024-03-06T07:26:12.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1719149464 shutdown_connections 2024-03-06T07:26:12.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c8006b4d0 0x7f0c8006d990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c940d00c0 0x7f0c940b6bb0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c940cf850 0x7f0c940b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 --2- 172.21.15.59:0/1719149464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c940ce1b0 0x7f0c941584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1719149464 >> 172.21.15.59:0/1719149464 conn(0x7f0c940b35a0 msgr2=0x7f0c9406fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1719149464 shutdown_connections 2024-03-06T07:26:12.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.698+0000 7f0c9b0af640 1 -- 172.21.15.59:0/1719149464 wait complete. 2024-03-06T07:26:12.711 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:12.711 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:26:12.712 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:12.712 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:12.712 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:12.712 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:26:12.712 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:12.712 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:12.712 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:12.712 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:26:12.713 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:12.713 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:12.713 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:12.713 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:26:12.713 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:12.713 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:12.721 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:12.721 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:12.722 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:12.722 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:12.725 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:12.725 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:12.726 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:12.726 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:12.763 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:12.763 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:12.764 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:12.764 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:12.768 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:12.768 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:12.769 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:12.769 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:12.769 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:26:12.840 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:26:12.840 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:26:12.851 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:26:12.851 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.874+0000 7efd942b5640 1 Processor -- start 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.874+0000 7efd942b5640 1 -- start start 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd942b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efd8c162400 0x7efd8c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd942b5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd8c0d22c0 con 0x7efd8c0b4460 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd942b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd8c0d25a0 con 0x7efd8c162970 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd942b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd8c161b00 con 0x7efd8c162400 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd92051640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7efd8c0b4460 0x7efd8c0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:43068/0 (socket says 172.21.15.59:43068) 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd92051640 1 -- 172.21.15.59:0/1458474764 learned_addr learned my addr 172.21.15.59:0/1458474764 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd92852640 1 --2- 172.21.15.59:0/1458474764 >> v2:172.21.15.164:3300/0 conn(0x7efd8c162400 0x7efd8c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401029113 0 0) 0x7efd8c0d22c0 con 0x7efd8c0b4460 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd78003520 con 0x7efd8c0b4460 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2814101841 0 0) 0x7efd78003520 con 0x7efd8c0b4460 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 >> v2:172.21.15.164:3300/0 conn(0x7efd8c162400 msgr2=0x7efd8c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 --2- 172.21.15.59:0/1458474764 >> v2:172.21.15.164:3300/0 conn(0x7efd8c162400 0x7efd8c0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 >> v1:172.21.15.164:6789/0 conn(0x7efd8c162970 legacy=0x7efd8c161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c0d2880 con 0x7efd8c0b4460 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7efd88001670 con 0x7efd8c0b4460 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd92852640 1 --2- 172.21.15.59:0/1458474764 >> v2:172.21.15.164:3300/0 conn(0x7efd8c162400 0x7efd8c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 >> v1:172.21.15.59:6789/0 conn(0x7efd8c0b4460 legacy=0x7efd8c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 --2- 172.21.15.59:0/1458474764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd780039b0 0x7efd78003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 --2- 172.21.15.59:0/1458474764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd780043b0 0x7efd78004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.875+0000 7efd9104f640 1 --2- 172.21.15.59:0/1458474764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 0x7efd7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.876+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c161b00 con 0x7efd78005080 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.876+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7800bfe0 con 0x7efd780043b0 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.876+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7800c1e0 con 0x7efd780039b0 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.876+0000 7efd92852640 1 --2- 172.21.15.59:0/1458474764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 0x7efd7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.876+0000 7efd92051640 1 --2- 172.21.15.59:0/1458474764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd780039b0 0x7efd78003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.876+0000 7efd91850640 1 --2- 172.21.15.59:0/1458474764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd780043b0 0x7efd78004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.876+0000 7efd92852640 1 -- 172.21.15.59:0/1458474764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd780039b0 msgr2=0x7efd78003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.876+0000 7efd92852640 1 --2- 172.21.15.59:0/1458474764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd780039b0 0x7efd78003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.876+0000 7efd92852640 1 -- 172.21.15.59:0/1458474764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd780043b0 msgr2=0x7efd78004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.876+0000 7efd92852640 1 --2- 172.21.15.59:0/1458474764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd780043b0 0x7efd78004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.876+0000 7efd92852640 1 -- 172.21.15.59:0/1458474764 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd78003520 con 0x7efd78005080 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd92852640 1 -- 172.21.15.59:0/1458474764 reap_dead start 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd92852640 1 --2- 172.21.15.59:0/1458474764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 0x7efd7800b9c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7efd8400e320 tx=0x7efd78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bada8b6dd89ce89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd84009de0 con 0x7efd78005080 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8402cde0 con 0x7efd78005080 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd9104f640 1 -- 172.21.15.59:0/1458474764 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd84003bd0 con 0x7efd78005080 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd942b5640 1 -- 172.21.15.59:0/1458474764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 msgr2=0x7efd7800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd942b5640 1 --2- 172.21.15.59:0/1458474764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 0x7efd7800b9c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7efd8400e320 tx=0x7efd78005830 comp rx=0 tx=0).stop 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd942b5640 1 -- 172.21.15.59:0/1458474764 shutdown_connections 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd942b5640 1 --2- 172.21.15.59:0/1458474764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 0x7efd7800b9c0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd942b5640 1 -- 172.21.15.59:0/1458474764 >> 172.21.15.59:0/1458474764 conn(0x7efd8c0d3ce0 msgr2=0x7efd8c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd942b5640 1 -- 172.21.15.59:0/1458474764 shutdown_connections 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.877+0000 7efd942b5640 1 -- 172.21.15.59:0/1458474764 wait complete. 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.879+0000 7efd942b5640 1 Processor -- start 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.879+0000 7efd942b5640 1 -- start start 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.879+0000 7efd942b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 0x7efd8c153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.879+0000 7efd942b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd8c0b23d0 0x7efd8c1f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.879+0000 7efd942b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd8c1f3e40 0x7efd8c157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.879+0000 7efd942b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c0ce950 con 0x7efd78005080 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.879+0000 7efd942b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c0cecc0 con 0x7efd8c1f3e40 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.879+0000 7efd942b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c0b4840 con 0x7efd8c0b23d0 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.879+0000 7efd92051640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 0x7efd8c153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.879+0000 7efd92051640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 0x7efd8c153490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35822/0 (socket says 172.21.15.59:35822) 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.879+0000 7efd92051640 1 -- 172.21.15.59:0/879377010 learned_addr learned my addr 172.21.15.59:0/879377010 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd91850640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd8c0b23d0 0x7efd8c1f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd92852640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd8c1f3e40 0x7efd8c157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd92051640 1 -- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd8c0b23d0 msgr2=0x7efd8c1f7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd92051640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd8c0b23d0 0x7efd8c1f7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd92051640 1 -- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd8c1f3e40 msgr2=0x7efd8c157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd92051640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd8c1f3e40 0x7efd8c157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd92051640 1 -- 172.21.15.59:0/879377010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c1f47e0 con 0x7efd78005080 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd92852640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd8c1f3e40 0x7efd8c157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd92051640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 0x7efd8c153490 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7efd880072f0 tx=0x7efd880077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd82ffd640 1 -- 172.21.15.59:0/879377010 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8801a070 con 0x7efd78005080 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd82ffd640 1 -- 172.21.15.59:0/879377010 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8800f960 con 0x7efd78005080 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.880+0000 7efd942b5640 1 -- 172.21.15.59:0/879377010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8c1f4ac0 con 0x7efd78005080 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.881+0000 7efd82ffd640 1 -- 172.21.15.59:0/879377010 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8800fd30 con 0x7efd78005080 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.881+0000 7efd942b5640 1 -- 172.21.15.59:0/879377010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd8c1f77e0 con 0x7efd78005080 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.881+0000 7efd82ffd640 1 -- 172.21.15.59:0/879377010 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd8801d070 con 0x7efd78005080 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.881+0000 7efd942b5640 1 -- 172.21.15.59:0/879377010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8c04f330 con 0x7efd78005080 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.882+0000 7efd82ffd640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd7006b4d0 0x7efd7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.882+0000 7efd82ffd640 1 -- 172.21.15.59:0/879377010 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7efd88089820 con 0x7efd78005080 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.882+0000 7efd91850640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd7006b4d0 0x7efd7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.882+0000 7efd91850640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd7006b4d0 0x7efd7006d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0c7930 tx=0x7efd7c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:12.886+0000 7efd82ffd640 1 -- 172.21.15.59:0/879377010 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd88058a30 con 0x7efd78005080 2024-03-06T07:26:13.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.078+0000 7efd942b5640 1 -- 172.21.15.59:0/879377010 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7efd8c06b180 con 0x7efd78005080 2024-03-06T07:26:13.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.836+0000 7efd82ffd640 1 -- 172.21.15.59:0/879377010 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v86) v1 ==== 97+0+0 (secure 0 0 0) 0x7efd88058330 con 0x7efd78005080 2024-03-06T07:26:13.838 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-03-06T07:26:13.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 -- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd7006b4d0 msgr2=0x7efd7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd7006b4d0 0x7efd7006d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0c7930 tx=0x7efd7c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:13.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 -- 172.21.15.59:0/879377010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 msgr2=0x7efd8c153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 0x7efd8c153490 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7efd880072f0 tx=0x7efd880077b0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 -- 172.21.15.59:0/879377010 shutdown_connections 2024-03-06T07:26:13.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd7006b4d0 0x7efd7006d990 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd8c1f3e40 0x7efd8c157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd8c0b23d0 0x7efd8c1f7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 --2- 172.21.15.59:0/879377010 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd78005080 0x7efd8c153490 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 -- 172.21.15.59:0/879377010 >> 172.21.15.59:0/879377010 conn(0x7efd8c0d3ce0 msgr2=0x7efd8c06c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 -- 172.21.15.59:0/879377010 shutdown_connections 2024-03-06T07:26:13.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.839+0000 7efd942b5640 1 -- 172.21.15.59:0/879377010 wait complete. 2024-03-06T07:26:13.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:26:13.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.962+0000 7f3311d8a640 1 Processor -- start 2024-03-06T07:26:13.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.962+0000 7f3311d8a640 1 -- start start 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f3311d8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1424f0 0x7f330c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f3311d8a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f330c06da40 0x7f330c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f3311d8a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330c0cc980 con 0x7f330c0cd7a0 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f3311d8a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f330c0cd220 con 0x7f330c06da40 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f3311d8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330c0ccc60 con 0x7f330c1424f0 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330affd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f330c0cd7a0 0x7f330c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34326/0 (socket says 172.21.15.59:34326) 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330affd640 1 -- 172.21.15.59:0/719828942 learned_addr learned my addr 172.21.15.59:0/719828942 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927910543 0 0) 0x7f330c0cc980 con 0x7f330c0cd7a0 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32f4003520 con 0x7f330c0cd7a0 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330b7fe640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1424f0 0x7f330c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330bfff640 1 --2- 172.21.15.59:0/719828942 >> v2:172.21.15.164:3300/0 conn(0x7f330c06da40 0x7f330c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1788923339 0 0) 0x7f32f4003520 con 0x7f330c0cd7a0 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1424f0 msgr2=0x7f330c0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330a7fc640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1424f0 0x7f330c0cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 >> v2:172.21.15.164:3300/0 conn(0x7f330c06da40 msgr2=0x7f330c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330a7fc640 1 --2- 172.21.15.59:0/719828942 >> v2:172.21.15.164:3300/0 conn(0x7f330c06da40 0x7f330c15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.963+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c0ccf40 con 0x7f330c0cd7a0 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f3300001670 con 0x7f330c0cd7a0 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330bfff640 1 --2- 172.21.15.59:0/719828942 >> v2:172.21.15.164:3300/0 conn(0x7f330c06da40 0x7f330c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330b7fe640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1424f0 0x7f330c0cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 >> v1:172.21.15.59:6789/0 conn(0x7f330c0cd7a0 legacy=0x7f330c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330a7fc640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1424f0 0x7f32f40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330a7fc640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32f4003fa0 0x7f32f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330a7fc640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f4004d40 0x7f32f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c0cd220 con 0x7f32f4004d40 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c0ccc60 con 0x7f32f4003fa0 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f400baa0 con 0x7f330c1424f0 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330bfff640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f4004d40 0x7f32f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330affd640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32f4003fa0 0x7f32f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330b7fe640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1424f0 0x7f32f40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330bfff640 1 -- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1424f0 msgr2=0x7f32f40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330bfff640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1424f0 0x7f32f40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330bfff640 1 -- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32f4003fa0 msgr2=0x7f32f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330bfff640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32f4003fa0 0x7f32f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330bfff640 1 -- 172.21.15.59:0/719828942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f4003520 con 0x7f32f4004d40 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330affd640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32f4003fa0 0x7f32f4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.964+0000 7f330b7fe640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1424f0 0x7f32f40039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.965+0000 7f330bfff640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f4004d40 0x7f32f40051c0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f33040073e0 tx=0x7f32f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f683f8804a5365a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.965+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304039070 con 0x7f32f4004d40 2024-03-06T07:26:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.965+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330402cdc0 con 0x7f32f4004d40 2024-03-06T07:26:13.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.965+0000 7f330a7fc640 1 -- 172.21.15.59:0/719828942 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304009c80 con 0x7f32f4004d40 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.965+0000 7f3311d8a640 1 -- 172.21.15.59:0/719828942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f4004d40 msgr2=0x7f32f40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.965+0000 7f3311d8a640 1 --2- 172.21.15.59:0/719828942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32f4004d40 0x7f32f40051c0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f33040073e0 tx=0x7f32f4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.965+0000 7f330bfff640 1 -- 172.21.15.59:0/719828942 reap_dead start 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.965+0000 7f3311d8a640 1 -- 172.21.15.59:0/719828942 shutdown_connections 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.965+0000 7f3311d8a640 1 -- 172.21.15.59:0/719828942 >> 172.21.15.59:0/719828942 conn(0x7f330c0cf8a0 msgr2=0x7f330c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.965+0000 7f3311d8a640 1 -- 172.21.15.59:0/719828942 shutdown_connections 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.966+0000 7f3311d8a640 1 -- 172.21.15.59:0/719828942 wait complete. 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.967+0000 7f3311d8a640 1 Processor -- start 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.967+0000 7f3311d8a640 1 -- start start 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.967+0000 7f3311d8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1595b0 0x7f330c0c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.967+0000 7f3311d8a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f330c06afb0 0x7f330c06b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f3311d8a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f330c1f7410 0x7f330c1f7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f3311d8a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c0cc470 con 0x7f330c06afb0 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f3311d8a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c0cc2f0 con 0x7f330c1f7410 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f3311d8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330c0cc5f0 con 0x7f330c1595b0 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f330affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f330c06afb0 0x7f330c06b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f330affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f330c06afb0 0x7f330c06b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47184/0 (socket says 172.21.15.59:47184) 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f330affd640 1 -- 172.21.15.59:0/2911950538 learned_addr learned my addr 172.21.15.59:0/2911950538 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f330b7fe640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1595b0 0x7f330c0c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f330bfff640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f330c1f7410 0x7f330c1f7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f330affd640 1 -- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1595b0 msgr2=0x7f330c0c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f330affd640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1595b0 0x7f330c0c48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f330affd640 1 -- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f330c1f7410 msgr2=0x7f330c1f7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f330affd640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f330c1f7410 0x7f330c1f7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f330affd640 1 -- 172.21.15.59:0/2911950538 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c0c5650 con 0x7f330c06afb0 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f330affd640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f330c06afb0 0x7f330c06b430 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f330000e880 tx=0x7f330000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f32f3fff640 1 -- 172.21.15.59:0/2911950538 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330000c8d0 con 0x7f330c06afb0 2024-03-06T07:26:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f32f3fff640 1 -- 172.21.15.59:0/2911950538 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3300018070 con 0x7f330c06afb0 2024-03-06T07:26:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f3311d8a640 1 -- 172.21.15.59:0/2911950538 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330c0c5930 con 0x7f330c06afb0 2024-03-06T07:26:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f32f3fff640 1 -- 172.21.15.59:0/2911950538 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330000cc70 con 0x7f330c06afb0 2024-03-06T07:26:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.968+0000 7f3311d8a640 1 -- 172.21.15.59:0/2911950538 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330c0b2ca0 con 0x7f330c06afb0 2024-03-06T07:26:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.969+0000 7f3311d8a640 1 -- 172.21.15.59:0/2911950538 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330c04f330 con 0x7f330c06afb0 2024-03-06T07:26:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.970+0000 7f32f3fff640 1 -- 172.21.15.59:0/2911950538 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f330001d460 con 0x7f330c06afb0 2024-03-06T07:26:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.970+0000 7f32f3fff640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32f806b4d0 0x7f32f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.970+0000 7f32f3fff640 1 -- 172.21.15.59:0/2911950538 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3300089da0 con 0x7f330c06afb0 2024-03-06T07:26:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.970+0000 7f330b7fe640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32f806b4d0 0x7f32f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.971+0000 7f330b7fe640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32f806b4d0 0x7f32f806d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f32fc006fd0 tx=0x7f32fc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:13.975+0000 7f32f3fff640 1 -- 172.21.15.59:0/2911950538 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3300058920 con 0x7f330c06afb0 2024-03-06T07:26:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.225+0000 7f3311d8a640 1 -- 172.21.15.59:0/2911950538 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f330c0b2e70 con 0x7f330c06afb0 2024-03-06T07:26:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.225+0000 7f32f3fff640 1 -- 172.21.15.59:0/2911950538 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f330005c9c0 con 0x7f330c06afb0 2024-03-06T07:26:14.227 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:14.227 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:26:14.227 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlODBZGBAAqV+TH2t2ZKyV9l13QEn6ew==", 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-03-06T07:26:14.228 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlR1MuJRAAp6OAQ9gZemWtwI1dh3H2SA==", 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlDXW2HxAAnKyuS1GW0wLO98/F9wdOAg==", 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:26:14.229 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehl5MO4LBAAKmt2dvKk7BtcCM4o7cSwgg==", 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.230 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBXGehlzdinFhAA0ACDAs9dtoY6EhUIveeIvA==", 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGehlH92eFRAANQceHGuQyrbI6gP1m/Gakw==", 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:26:14.231 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGehlMWpVFBAAbOG9nScbFlvKHXZMQJoQug==", 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-03-06T07:26:14.232 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGehlLCyWEhAAyfoW/FmU37QD2IohknBwWw==", 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBmGehldqv5GhAAWEloqqKgdpKPHjt7VKFodw==", 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGehlc0ENGRAA5V9Az+vxAR3OUwWpDP7w1A==", 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBsGehl5SYEFxAAPLyrHQV9SGguCV9f5VIkYg==", 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.234 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBvGehle3v0FBAA4xUPwX5couppHgi9VwSi2g==", 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.235 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA==", 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlqhlAOhAAQ4l3uI8MlcCH+NAIWo/UuA==", 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:26:14.236 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehltL/GNhAAZqSQ+UtwjI/PA95O1RGBNg==", 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.237 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGehlltx5BBAAFa/+/w8Hd8lZegfIWtpJOA==", 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBSGehlMExaNhAAQ2FD7afktMhrTbMMAK1epQ==", 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:26:14.238 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlLSTzFBAACeLdHsCX9KJGEsq44aqTyw==", 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:14.239 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-03-06T07:26:14.240 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlAgqXEBAACCHZ9Lp2lEyF93xD4Ns9Nw==", 2024-03-06T07:26:14.240 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:26:14.240 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:26:14.240 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:14.240 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:26:14.240 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.240 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:14.240 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-06T07:26:14.240 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:26:14.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 -- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32f806b4d0 msgr2=0x7f32f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32f806b4d0 0x7f32f806d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f32fc006fd0 tx=0x7f32fc008040 comp rx=0 tx=0).stop 2024-03-06T07:26:14.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 -- 172.21.15.59:0/2911950538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f330c06afb0 msgr2=0x7f330c06b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f330c06afb0 0x7f330c06b430 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f330000e880 tx=0x7f330000ed40 comp rx=0 tx=0).stop 2024-03-06T07:26:14.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 -- 172.21.15.59:0/2911950538 shutdown_connections 2024-03-06T07:26:14.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32f806b4d0 0x7f32f806d990 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f330c1f7410 0x7f330c1f7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f330c06afb0 0x7f330c06b430 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 --2- 172.21.15.59:0/2911950538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f330c1595b0 0x7f330c0c48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 -- 172.21.15.59:0/2911950538 >> 172.21.15.59:0/2911950538 conn(0x7f330c0cf8a0 msgr2=0x7f330c0cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 -- 172.21.15.59:0/2911950538 shutdown_connections 2024-03-06T07:26:14.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.227+0000 7f3311d8a640 1 -- 172.21.15.59:0/2911950538 wait complete. 2024-03-06T07:26:14.243 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:14.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.342+0000 7f2623a15640 1 Processor -- start 2024-03-06T07:26:14.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.342+0000 7f2623a15640 1 -- start start 2024-03-06T07:26:14.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f2623a15640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f261c0d00c0 0x7f261c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f2623a15640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f261c06cf20 0x7f261c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f2623a15640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261c0d09e0 con 0x7f261c0cf850 2024-03-06T07:26:14.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f2623a15640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f261c06ca90 con 0x7f261c0d00c0 2024-03-06T07:26:14.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f2623a15640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f261c15e380 con 0x7f261c06cf20 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f26217b1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f261c0d00c0 0x7f261c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f26217b1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f261c0d00c0 0x7f261c0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47192/0 (socket says 172.21.15.59:47192) 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f26217b1640 1 -- 172.21.15.59:0/3454838876 learned_addr learned my addr 172.21.15.59:0/3454838876 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f2621fb2640 1 --2- 172.21.15.59:0/3454838876 >> v2:172.21.15.164:3300/0 conn(0x7f261c06cf20 0x7f261c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f26217b1640 1 -- 172.21.15.59:0/3454838876 >> v2:172.21.15.164:3300/0 conn(0x7f261c06cf20 msgr2=0x7f261c06d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f26217b1640 1 --2- 172.21.15.59:0/3454838876 >> v2:172.21.15.164:3300/0 conn(0x7f261c06cf20 0x7f261c06d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.343+0000 7f26217b1640 1 -- 172.21.15.59:0/3454838876 >> v1:172.21.15.164:6789/0 conn(0x7f261c0cf850 legacy=0x7f261c0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.344+0000 7f26217b1640 1 -- 172.21.15.59:0/3454838876 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c0cdc10 con 0x7f261c0d00c0 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.344+0000 7f2621fb2640 1 --2- 172.21.15.59:0/3454838876 >> v2:172.21.15.164:3300/0 conn(0x7f261c06cf20 0x7f261c06d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.344+0000 7f26217b1640 1 --2- 172.21.15.59:0/3454838876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f261c0d00c0 0x7f261c0d04a0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f261c0cfda0 tx=0x7f260c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383ea9fed31b5db4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.344+0000 7f2613fff640 1 -- 172.21.15.59:0/3454838876 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260c035020 con 0x7f261c0d00c0 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.344+0000 7f2613fff640 1 -- 172.21.15.59:0/3454838876 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f260c031870 con 0x7f261c0d00c0 2024-03-06T07:26:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.344+0000 7f2613fff640 1 -- 172.21.15.59:0/3454838876 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260c031c40 con 0x7f261c0d00c0 2024-03-06T07:26:14.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.345+0000 7f2623a15640 1 -- 172.21.15.59:0/3454838876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f261c0d00c0 msgr2=0x7f261c0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.345+0000 7f2623a15640 1 --2- 172.21.15.59:0/3454838876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f261c0d00c0 0x7f261c0d04a0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f261c0cfda0 tx=0x7f260c02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:14.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.345+0000 7f2623a15640 1 -- 172.21.15.59:0/3454838876 shutdown_connections 2024-03-06T07:26:14.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.345+0000 7f2623a15640 1 --2- 172.21.15.59:0/3454838876 >> v2:172.21.15.164:3300/0 conn(0x7f261c06cf20 0x7f261c06d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.345+0000 7f2623a15640 1 --2- 172.21.15.59:0/3454838876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f261c0d00c0 0x7f261c0d04a0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.345+0000 7f2623a15640 1 -- 172.21.15.59:0/3454838876 >> 172.21.15.59:0/3454838876 conn(0x7f261c0b35a0 msgr2=0x7f261c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.345+0000 7f2623a15640 1 -- 172.21.15.59:0/3454838876 shutdown_connections 2024-03-06T07:26:14.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.345+0000 7f2623a15640 1 -- 172.21.15.59:0/3454838876 wait complete. 2024-03-06T07:26:14.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.346+0000 7f2623a15640 1 Processor -- start 2024-03-06T07:26:14.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.346+0000 7f2623a15640 1 -- start start 2024-03-06T07:26:14.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2623a15640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f261c06cf20 0x7f261c0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2623a15640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f261c0cf850 0x7f261c15a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2623a15640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f261c0d00c0 0x7f261c15efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2623a15640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c06c720 con 0x7f261c0d00c0 2024-03-06T07:26:14.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2623a15640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c06c440 con 0x7f261c0cf850 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2623a15640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261c0b1030 con 0x7f261c06cf20 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2621fb2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f261c0d00c0 0x7f261c15efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2621fb2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f261c0d00c0 0x7f261c15efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47204/0 (socket says 172.21.15.59:47204) 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2621fb2640 1 -- 172.21.15.59:0/2302923133 learned_addr learned my addr 172.21.15.59:0/2302923133 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2620fb0640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f261c0cf850 0x7f261c15a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f26217b1640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f261c06cf20 0x7f261c0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2621fb2640 1 -- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f261c06cf20 msgr2=0x7f261c0b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2621fb2640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f261c06cf20 0x7f261c0b01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2621fb2640 1 -- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f261c0cf850 msgr2=0x7f261c15a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2621fb2640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f261c0cf850 0x7f261c15a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.347+0000 7f2621fb2640 1 -- 172.21.15.59:0/2302923133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c15f4e0 con 0x7f261c0d00c0 2024-03-06T07:26:14.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.348+0000 7f2621fb2640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f261c0d00c0 0x7f261c15efa0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f261400cd50 tx=0x7f261400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.348+0000 7f2611ffb640 1 -- 172.21.15.59:0/2302923133 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261400f680 con 0x7f261c0d00c0 2024-03-06T07:26:14.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.348+0000 7f2611ffb640 1 -- 172.21.15.59:0/2302923133 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261400f840 con 0x7f261c0d00c0 2024-03-06T07:26:14.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.348+0000 7f2623a15640 1 -- 172.21.15.59:0/2302923133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261c0c8be0 con 0x7f261c0d00c0 2024-03-06T07:26:14.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.348+0000 7f2611ffb640 1 -- 172.21.15.59:0/2302923133 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261400fc30 con 0x7f261c0d00c0 2024-03-06T07:26:14.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.348+0000 7f2623a15640 1 -- 172.21.15.59:0/2302923133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f261c0c91a0 con 0x7f261c0d00c0 2024-03-06T07:26:14.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.349+0000 7f2623a15640 1 -- 172.21.15.59:0/2302923133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f261c04f330 con 0x7f261c0d00c0 2024-03-06T07:26:14.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.349+0000 7f2611ffb640 1 -- 172.21.15.59:0/2302923133 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26140213f0 con 0x7f261c0d00c0 2024-03-06T07:26:14.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.350+0000 7f2611ffb640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f260006b4d0 0x7f260006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.350+0000 7f2611ffb640 1 -- 172.21.15.59:0/2302923133 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f261408ab30 con 0x7f261c0d00c0 2024-03-06T07:26:14.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.350+0000 7f26217b1640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f260006b4d0 0x7f260006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.350+0000 7f26217b1640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f260006b4d0 0x7f260006d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f260c036000 tx=0x7f260c03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.354+0000 7f2611ffb640 1 -- 172.21.15.59:0/2302923133 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2614014030 con 0x7f261c0d00c0 2024-03-06T07:26:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.535+0000 7f2623a15640 1 -- 172.21.15.59:0/2302923133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f261c0cd560 con 0x7f261c0d00c0 2024-03-06T07:26:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.536+0000 7f2611ffb640 1 -- 172.21.15.59:0/2302923133 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+12880 (secure 0 0 0) 0x7f2614059d90 con 0x7f261c0d00c0 2024-03-06T07:26:14.538 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:14.538 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:26:13.845392+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 -- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f260006b4d0 msgr2=0x7f260006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f260006b4d0 0x7f260006d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f260c036000 tx=0x7f260c03c040 comp rx=0 tx=0).stop 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 -- 172.21.15.59:0/2302923133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f261c0d00c0 msgr2=0x7f261c15efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f261c0d00c0 0x7f261c15efa0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f261400cd50 tx=0x7f261400a650 comp rx=0 tx=0).stop 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 -- 172.21.15.59:0/2302923133 shutdown_connections 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f260006b4d0 0x7f260006d990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f261c0d00c0 0x7f261c15efa0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f261c0cf850 0x7f261c15a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 --2- 172.21.15.59:0/2302923133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f261c06cf20 0x7f261c0b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 -- 172.21.15.59:0/2302923133 >> 172.21.15.59:0/2302923133 conn(0x7f261c0b35a0 msgr2=0x7f261c0cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 -- 172.21.15.59:0/2302923133 shutdown_connections 2024-03-06T07:26:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.538+0000 7f2623a15640 1 -- 172.21.15.59:0/2302923133 wait complete. 2024-03-06T07:26:14.551 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:14.551 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:14.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.654+0000 7fa84041c640 1 Processor -- start 2024-03-06T07:26:14.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.654+0000 7fa84041c640 1 -- start start 2024-03-06T07:26:14.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.655+0000 7fa84041c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa838066d90 0x7fa8380b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.655+0000 7fa84041c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8380b5410 0x7fa8380ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.655+0000 7fa84041c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83806b710 con 0x7fa8380cf9a0 2024-03-06T07:26:14.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.655+0000 7fa84041c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8380b61c0 con 0x7fa838066d90 2024-03-06T07:26:14.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.655+0000 7fa84041c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83806b150 con 0x7fa8380b5410 2024-03-06T07:26:14.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.655+0000 7fa83e9b9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8380b5410 0x7fa8380ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.655+0000 7fa83d9b7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa838066d90 0x7fa8380b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.655+0000 7fa83e1b8640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa8380cf9a0 0x7fa8380b5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:46920/0 (socket says 172.21.15.59:46920) 2024-03-06T07:26:14.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.655+0000 7fa83d9b7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa838066d90 0x7fa8380b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:54432/0 (socket says 172.21.15.59:54432) 2024-03-06T07:26:14.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.655+0000 7fa83e9b9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8380b5410 0x7fa8380ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:54236/0 (socket says 172.21.15.59:54236) 2024-03-06T07:26:14.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.655+0000 7fa83e1b8640 1 -- 172.21.15.59:0/3618140265 learned_addr learned my addr 172.21.15.59:0/3618140265 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:14.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.656+0000 7fa83d9b7640 1 -- 172.21.15.59:0/3618140265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8380b5410 msgr2=0x7fa8380ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.656+0000 7fa83d9b7640 1 --2- 172.21.15.59:0/3618140265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8380b5410 0x7fa8380ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.656+0000 7fa83d9b7640 1 -- 172.21.15.59:0/3618140265 >> v1:172.21.15.164:6789/0 conn(0x7fa8380cf9a0 legacy=0x7fa8380b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.656+0000 7fa83d9b7640 1 -- 172.21.15.59:0/3618140265 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83806b430 con 0x7fa838066d90 2024-03-06T07:26:14.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.656+0000 7fa83e9b9640 1 --2- 172.21.15.59:0/3618140265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8380b5410 0x7fa8380ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:14.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.656+0000 7fa83d9b7640 1 --2- 172.21.15.59:0/3618140265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa838066d90 0x7fa8380b4ed0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa8280097a0 tx=0x7fa82802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c9832d76603ba3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.657+0000 7fa83d1b6640 1 -- 172.21.15.59:0/3618140265 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828035020 con 0x7fa838066d90 2024-03-06T07:26:14.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.657+0000 7fa83d1b6640 1 -- 172.21.15.59:0/3618140265 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa828031830 con 0x7fa838066d90 2024-03-06T07:26:14.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.657+0000 7fa83d1b6640 1 -- 172.21.15.59:0/3618140265 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828031c00 con 0x7fa838066d90 2024-03-06T07:26:14.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.657+0000 7fa84041c640 1 -- 172.21.15.59:0/3618140265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa838066d90 msgr2=0x7fa8380b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.657+0000 7fa84041c640 1 --2- 172.21.15.59:0/3618140265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa838066d90 0x7fa8380b4ed0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa8280097a0 tx=0x7fa82802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.657+0000 7fa84041c640 1 -- 172.21.15.59:0/3618140265 shutdown_connections 2024-03-06T07:26:14.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.657+0000 7fa84041c640 1 --2- 172.21.15.59:0/3618140265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8380b5410 0x7fa8380ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.657+0000 7fa84041c640 1 --2- 172.21.15.59:0/3618140265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa838066d90 0x7fa8380b4ed0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.657+0000 7fa84041c640 1 -- 172.21.15.59:0/3618140265 >> 172.21.15.59:0/3618140265 conn(0x7fa8380b1d70 msgr2=0x7fa8380b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.657+0000 7fa84041c640 1 -- 172.21.15.59:0/3618140265 shutdown_connections 2024-03-06T07:26:14.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.657+0000 7fa84041c640 1 -- 172.21.15.59:0/3618140265 wait complete. 2024-03-06T07:26:14.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.658+0000 7fa84041c640 1 Processor -- start 2024-03-06T07:26:14.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.658+0000 7fa84041c640 1 -- start start 2024-03-06T07:26:14.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa84041c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa838066d90 0x7fa838152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa84041c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8380b5410 0x7fa8380b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa84041c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8380cf9a0 0x7fa8380b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa84041c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380cfd80 con 0x7fa8380b5410 2024-03-06T07:26:14.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa84041c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380b0c50 con 0x7fa8380cf9a0 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa84041c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380cff00 con 0x7fa838066d90 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa83d9b7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8380b5410 0x7fa8380b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa83d9b7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8380b5410 0x7fa8380b0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47220/0 (socket says 172.21.15.59:47220) 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa83d9b7640 1 -- 172.21.15.59:0/3516086422 learned_addr learned my addr 172.21.15.59:0/3516086422 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa83e9b9640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8380cf9a0 0x7fa8380b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa83e1b8640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa838066d90 0x7fa838152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa83d9b7640 1 -- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa838066d90 msgr2=0x7fa838152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa83d9b7640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa838066d90 0x7fa838152600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa83d9b7640 1 -- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8380cf9a0 msgr2=0x7fa8380b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa83d9b7640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8380cf9a0 0x7fa8380b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa83d9b7640 1 -- 172.21.15.59:0/3516086422 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa838152e30 con 0x7fa8380b5410 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.659+0000 7fa83e9b9640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8380cf9a0 0x7fa8380b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.660+0000 7fa83d9b7640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8380b5410 0x7fa8380b0fc0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa82802bee0 tx=0x7fa8280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.660+0000 7fa82effd640 1 -- 172.21.15.59:0/3516086422 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828035040 con 0x7fa8380b5410 2024-03-06T07:26:14.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.660+0000 7fa82effd640 1 -- 172.21.15.59:0/3516086422 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82803b920 con 0x7fa8380b5410 2024-03-06T07:26:14.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.660+0000 7fa84041c640 1 -- 172.21.15.59:0/3516086422 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa838153110 con 0x7fa8380b5410 2024-03-06T07:26:14.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.660+0000 7fa82effd640 1 -- 172.21.15.59:0/3516086422 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82803bd10 con 0x7fa8380b5410 2024-03-06T07:26:14.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.660+0000 7fa84041c640 1 -- 172.21.15.59:0/3516086422 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8381536d0 con 0x7fa8380b5410 2024-03-06T07:26:14.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.661+0000 7fa84041c640 1 -- 172.21.15.59:0/3516086422 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa83809e3d0 con 0x7fa8380b5410 2024-03-06T07:26:14.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.661+0000 7fa82effd640 1 -- 172.21.15.59:0/3516086422 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa828038020 con 0x7fa8380b5410 2024-03-06T07:26:14.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.662+0000 7fa82effd640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81c06b4d0 0x7fa81c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.662+0000 7fa82effd640 1 -- 172.21.15.59:0/3516086422 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa8280ac0c0 con 0x7fa8380b5410 2024-03-06T07:26:14.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.662+0000 7fa83e1b8640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81c06b4d0 0x7fa81c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.662+0000 7fa83e1b8640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81c06b4d0 0x7fa81c06d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa834000f60 tx=0x7fa834009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.666+0000 7fa82effd640 1 -- 172.21.15.59:0/3516086422 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa82807b210 con 0x7fa8380b5410 2024-03-06T07:26:14.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.848+0000 7fa84041c640 1 -- 172.21.15.59:0/3516086422 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa838161a70 con 0x7fa8380b5410 2024-03-06T07:26:14.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.849+0000 7fa82effd640 1 -- 172.21.15.59:0/3516086422 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa828031800 con 0x7fa8380b5410 2024-03-06T07:26:14.851 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 -- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81c06b4d0 msgr2=0x7fa81c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81c06b4d0 0x7fa81c06d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa834000f60 tx=0x7fa834009040 comp rx=0 tx=0).stop 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 -- 172.21.15.59:0/3516086422 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8380b5410 msgr2=0x7fa8380b0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8380b5410 0x7fa8380b0fc0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa82802bee0 tx=0x7fa8280046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 -- 172.21.15.59:0/3516086422 shutdown_connections 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa81c06b4d0 0x7fa81c06d990 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8380cf9a0 0x7fa8380b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8380b5410 0x7fa8380b0fc0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 --2- 172.21.15.59:0/3516086422 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa838066d90 0x7fa838152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 -- 172.21.15.59:0/3516086422 >> 172.21.15.59:0/3516086422 conn(0x7fa8380b1d70 msgr2=0x7fa838072550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 -- 172.21.15.59:0/3516086422 shutdown_connections 2024-03-06T07:26:14.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.851+0000 7fa84041c640 1 -- 172.21.15.59:0/3516086422 wait complete. 2024-03-06T07:26:14.863 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:26:14.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.963+0000 7f2975dbc640 1 Processor -- start 2024-03-06T07:26:14.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.963+0000 7f2975dbc640 1 -- start start 2024-03-06T07:26:14.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f2975dbc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f297006b220 0x7f297015e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f2975dbc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f297006b600 0x7f2970162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f2975dbc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f297015ea30 con 0x7f29700ce920 2024-03-06T07:26:14.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f2975dbc640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2970161b70 con 0x7f297006b600 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f2975dbc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f297015ed10 con 0x7f297006b220 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f296ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f297006b600 0x7f2970162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f296ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f297006b600 0x7f2970162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47234/0 (socket says 172.21.15.59:47234) 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f296ffff640 1 -- 172.21.15.59:0/2745749215 learned_addr learned my addr 172.21.15.59:0/2745749215 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f296f7fe640 1 --2- 172.21.15.59:0/2745749215 >> v2:172.21.15.164:3300/0 conn(0x7f297006b220 0x7f297015e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f296ffff640 1 -- 172.21.15.59:0/2745749215 >> v2:172.21.15.164:3300/0 conn(0x7f297006b220 msgr2=0x7f297015e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f296ffff640 1 --2- 172.21.15.59:0/2745749215 >> v2:172.21.15.164:3300/0 conn(0x7f297006b220 0x7f297015e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f296ffff640 1 -- 172.21.15.59:0/2745749215 >> v1:172.21.15.164:6789/0 conn(0x7f29700ce920 legacy=0x7f2970071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f296ffff640 1 -- 172.21.15.59:0/2745749215 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297015eff0 con 0x7f297006b600 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.964+0000 7f296f7fe640 1 --2- 172.21.15.59:0/2745749215 >> v2:172.21.15.164:3300/0 conn(0x7f297006b220 0x7f297015e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.965+0000 7f296ffff640 1 --2- 172.21.15.59:0/2745749215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f297006b600 0x7f2970162000 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f29700d0980 tx=0x7f296802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d51e58f224b803 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.965+0000 7f296e7fc640 1 -- 172.21.15.59:0/2745749215 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296802ca80 con 0x7f297006b600 2024-03-06T07:26:14.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.965+0000 7f296e7fc640 1 -- 172.21.15.59:0/2745749215 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296802cc20 con 0x7f297006b600 2024-03-06T07:26:14.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.965+0000 7f296e7fc640 1 -- 172.21.15.59:0/2745749215 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29680343f0 con 0x7f297006b600 2024-03-06T07:26:14.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.965+0000 7f2975dbc640 1 -- 172.21.15.59:0/2745749215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f297006b600 msgr2=0x7f2970162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.965+0000 7f2975dbc640 1 --2- 172.21.15.59:0/2745749215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f297006b600 0x7f2970162000 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f29700d0980 tx=0x7f296802bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.965+0000 7f2975dbc640 1 -- 172.21.15.59:0/2745749215 shutdown_connections 2024-03-06T07:26:14.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.965+0000 7f2975dbc640 1 --2- 172.21.15.59:0/2745749215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f297006b600 0x7f2970162000 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.965+0000 7f2975dbc640 1 --2- 172.21.15.59:0/2745749215 >> v2:172.21.15.164:3300/0 conn(0x7f297006b220 0x7f297015e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.965+0000 7f2975dbc640 1 -- 172.21.15.59:0/2745749215 >> 172.21.15.59:0/2745749215 conn(0x7f297006c0c0 msgr2=0x7f297006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:14.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.966+0000 7f2975dbc640 1 -- 172.21.15.59:0/2745749215 shutdown_connections 2024-03-06T07:26:14.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.966+0000 7f2975dbc640 1 -- 172.21.15.59:0/2745749215 wait complete. 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.967+0000 7f2975dbc640 1 Processor -- start 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.967+0000 7f2975dbc640 1 -- start start 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.967+0000 7f2975dbc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f297006b220 0x7f2970148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.967+0000 7f2975dbc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f297006b600 0x7f2970149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.967+0000 7f2975dbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29700ce920 0x7f297015c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.967+0000 7f2975dbc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2970072090 con 0x7f297006b220 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.967+0000 7f2975dbc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29700ce200 con 0x7f297006b600 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.967+0000 7f2975dbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297014ca00 con 0x7f29700ce920 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f296f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f297006b220 0x7f2970148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f296f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f297006b220 0x7f2970148be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47244/0 (socket says 172.21.15.59:47244) 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f296f7fe640 1 -- 172.21.15.59:0/3662702691 learned_addr learned my addr 172.21.15.59:0/3662702691 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f296ffff640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29700ce920 0x7f297015c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f296effd640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f297006b600 0x7f2970149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f296f7fe640 1 -- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29700ce920 msgr2=0x7f297015c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f296f7fe640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29700ce920 0x7f297015c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f296f7fe640 1 -- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f297006b600 msgr2=0x7f2970149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f296f7fe640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f297006b600 0x7f2970149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f296f7fe640 1 -- 172.21.15.59:0/3662702691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297014e190 con 0x7f297006b220 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f296f7fe640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f297006b220 0x7f2970148be0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f296400c7f0 tx=0x7f296400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f2957fff640 1 -- 172.21.15.59:0/3662702691 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2964015070 con 0x7f297006b220 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f2957fff640 1 -- 172.21.15.59:0/3662702691 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2964007820 con 0x7f297006b220 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f2975dbc640 1 -- 172.21.15.59:0/3662702691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297014a970 con 0x7f297006b220 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f2957fff640 1 -- 172.21.15.59:0/3662702691 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2964007bd0 con 0x7f297006b220 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.968+0000 7f2975dbc640 1 -- 172.21.15.59:0/3662702691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f297014af30 con 0x7f297006b220 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.969+0000 7f2975dbc640 1 -- 172.21.15.59:0/3662702691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297004f330 con 0x7f297006b220 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.970+0000 7f2957fff640 1 -- 172.21.15.59:0/3662702691 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2964018070 con 0x7f297006b220 2024-03-06T07:26:14.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.970+0000 7f2957fff640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f295c06b4d0 0x7f295c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.970+0000 7f2957fff640 1 -- 172.21.15.59:0/3662702691 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2964089cd0 con 0x7f297006b220 2024-03-06T07:26:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.970+0000 7f296effd640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f295c06b4d0 0x7f295c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.971+0000 7f296effd640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f295c06b4d0 0x7f295c06d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f297014c660 tx=0x7f2960007e30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:14.975+0000 7f2957fff640 1 -- 172.21.15.59:0/3662702691 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2964058840 con 0x7f297006b220 2024-03-06T07:26:15.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.157+0000 7f2975dbc640 1 -- 172.21.15.59:0/3662702691 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f29700cdf00 con 0x7f297006b220 2024-03-06T07:26:15.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.157+0000 7f2957fff640 1 -- 172.21.15.59:0/3662702691 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v88) v1 ==== 84+0+65 (secure 0 0 0) 0x7f296405c8e0 con 0x7f297006b220 2024-03-06T07:26:15.159 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:15.159 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-03-06T07:26:15.159 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:26:15.159 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-03-06T07:26:15.159 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-03-06T07:26:15.159 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:15.159 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-03-06T07:26:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 -- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f295c06b4d0 msgr2=0x7f295c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f295c06b4d0 0x7f295c06d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f297014c660 tx=0x7f2960007e30 comp rx=0 tx=0).stop 2024-03-06T07:26:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 -- 172.21.15.59:0/3662702691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f297006b220 msgr2=0x7f2970148be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f297006b220 0x7f2970148be0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f296400c7f0 tx=0x7f296400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 -- 172.21.15.59:0/3662702691 shutdown_connections 2024-03-06T07:26:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f295c06b4d0 0x7f295c06d990 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29700ce920 0x7f297015c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f297006b600 0x7f2970149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 --2- 172.21.15.59:0/3662702691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f297006b220 0x7f2970148be0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 -- 172.21.15.59:0/3662702691 >> 172.21.15.59:0/3662702691 conn(0x7f297006c0c0 msgr2=0x7f29700d06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 -- 172.21.15.59:0/3662702691 shutdown_connections 2024-03-06T07:26:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.159+0000 7f2975dbc640 1 -- 172.21.15.59:0/3662702691 wait complete. 2024-03-06T07:26:15.172 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:26:15.172 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:26:15.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.275+0000 7f3d7a239640 1 Processor -- start 2024-03-06T07:26:15.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.275+0000 7f3d7a239640 1 -- start start 2024-03-06T07:26:15.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.276+0000 7f3d7a239640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3d740cfa50 0x7f3d740cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.276+0000 7f3d7a239640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3d740cf590 0x7f3d740b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.276+0000 7f3d7a239640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d74161a10 con 0x7f3d74161630 2024-03-06T07:26:15.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.276+0000 7f3d7a239640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d740d12e0 con 0x7f3d740cfa50 2024-03-06T07:26:15.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.276+0000 7f3d7a239640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d740d29e0 con 0x7f3d740cf590 2024-03-06T07:26:15.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.276+0000 7f3d72ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3d740cfa50 0x7f3d740cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.276+0000 7f3d72ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3d740cfa50 0x7f3d740cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47250/0 (socket says 172.21.15.59:47250) 2024-03-06T07:26:15.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.276+0000 7f3d72ffd640 1 -- 172.21.15.59:0/3657787626 learned_addr learned my addr 172.21.15.59:0/3657787626 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:15.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.276+0000 7f3d73fff640 1 --2- 172.21.15.59:0/3657787626 >> v2:172.21.15.164:3300/0 conn(0x7f3d740cf590 0x7f3d740b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.276+0000 7f3d72ffd640 1 -- 172.21.15.59:0/3657787626 >> v2:172.21.15.164:3300/0 conn(0x7f3d740cf590 msgr2=0x7f3d740b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.277+0000 7f3d72ffd640 1 --2- 172.21.15.59:0/3657787626 >> v2:172.21.15.164:3300/0 conn(0x7f3d740cf590 0x7f3d740b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.277+0000 7f3d72ffd640 1 -- 172.21.15.59:0/3657787626 >> v1:172.21.15.164:6789/0 conn(0x7f3d74161630 legacy=0x7f3d740ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.277+0000 7f3d72ffd640 1 -- 172.21.15.59:0/3657787626 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d740d2cc0 con 0x7f3d740cfa50 2024-03-06T07:26:15.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.277+0000 7f3d73fff640 1 --2- 172.21.15.59:0/3657787626 >> v2:172.21.15.164:3300/0 conn(0x7f3d740cf590 0x7f3d740b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.277+0000 7f3d72ffd640 1 --2- 172.21.15.59:0/3657787626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d740cfa50 0x7f3d740cf050 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3d740d1460 tx=0x7f3d6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee9551a7600e32c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.277+0000 7f3d727fc640 1 -- 172.21.15.59:0/3657787626 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68035020 con 0x7f3d740cfa50 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.277+0000 7f3d727fc640 1 -- 172.21.15.59:0/3657787626 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d68031870 con 0x7f3d740cfa50 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.277+0000 7f3d727fc640 1 -- 172.21.15.59:0/3657787626 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68031c40 con 0x7f3d740cfa50 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.278+0000 7f3d7a239640 1 -- 172.21.15.59:0/3657787626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d740cfa50 msgr2=0x7f3d740cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.278+0000 7f3d7a239640 1 --2- 172.21.15.59:0/3657787626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d740cfa50 0x7f3d740cf050 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3d740d1460 tx=0x7f3d6802b910 comp rx=0 tx=0).stop 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.278+0000 7f3d7a239640 1 -- 172.21.15.59:0/3657787626 shutdown_connections 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.278+0000 7f3d7a239640 1 --2- 172.21.15.59:0/3657787626 >> v2:172.21.15.164:3300/0 conn(0x7f3d740cf590 0x7f3d740b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.278+0000 7f3d7a239640 1 --2- 172.21.15.59:0/3657787626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d740cfa50 0x7f3d740cf050 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.278+0000 7f3d7a239640 1 -- 172.21.15.59:0/3657787626 >> 172.21.15.59:0/3657787626 conn(0x7f3d740d3f30 msgr2=0x7f3d740d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.278+0000 7f3d7a239640 1 -- 172.21.15.59:0/3657787626 shutdown_connections 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.278+0000 7f3d7a239640 1 -- 172.21.15.59:0/3657787626 wait complete. 2024-03-06T07:26:15.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.279+0000 7f3d7a239640 1 Processor -- start 2024-03-06T07:26:15.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.279+0000 7f3d7a239640 1 -- start start 2024-03-06T07:26:15.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d7a239640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d740cf590 0x7f3d740b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d7a239640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d740cfa50 0x7f3d740ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d7a239640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d74161630 0x7f3d741f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d7a239640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740d1050 con 0x7f3d74161630 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d7a239640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740d0ed0 con 0x7f3d740cf590 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d7a239640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740b41b0 con 0x7f3d740cfa50 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d73fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d74161630 0x7f3d741f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d73fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d74161630 0x7f3d741f4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47264/0 (socket says 172.21.15.59:47264) 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d73fff640 1 -- 172.21.15.59:0/3430544644 learned_addr learned my addr 172.21.15.59:0/3430544644 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d72ffd640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d740cfa50 0x7f3d740ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d737fe640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d740cf590 0x7f3d740b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d73fff640 1 -- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d740cfa50 msgr2=0x7f3d740ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d73fff640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d740cfa50 0x7f3d740ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d73fff640 1 -- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d740cf590 msgr2=0x7f3d740b8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d73fff640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d740cf590 0x7f3d740b8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.280+0000 7f3d73fff640 1 -- 172.21.15.59:0/3430544644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d740b9310 con 0x7f3d74161630 2024-03-06T07:26:15.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.281+0000 7f3d73fff640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d74161630 0x7f3d741f4120 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c007e60 tx=0x7f3d6c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.281+0000 7f3d5bfff640 1 -- 172.21.15.59:0/3430544644 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c01a070 con 0x7f3d74161630 2024-03-06T07:26:15.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.281+0000 7f3d5bfff640 1 -- 172.21.15.59:0/3430544644 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6c003b90 con 0x7f3d74161630 2024-03-06T07:26:15.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.281+0000 7f3d7a239640 1 -- 172.21.15.59:0/3430544644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d740b95f0 con 0x7f3d74161630 2024-03-06T07:26:15.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.281+0000 7f3d5bfff640 1 -- 172.21.15.59:0/3430544644 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c00de80 con 0x7f3d74161630 2024-03-06T07:26:15.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.281+0000 7f3d7a239640 1 -- 172.21.15.59:0/3430544644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7415dd80 con 0x7f3d74161630 2024-03-06T07:26:15.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.282+0000 7f3d7a239640 1 -- 172.21.15.59:0/3430544644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7404f330 con 0x7f3d74161630 2024-03-06T07:26:15.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.282+0000 7f3d5bfff640 1 -- 172.21.15.59:0/3430544644 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d6c01d070 con 0x7f3d74161630 2024-03-06T07:26:15.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.282+0000 7f3d5bfff640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d6006b4d0 0x7f3d6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.283+0000 7f3d5bfff640 1 -- 172.21.15.59:0/3430544644 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3d6c08a0c0 con 0x7f3d74161630 2024-03-06T07:26:15.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.283+0000 7f3d737fe640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d6006b4d0 0x7f3d6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.283+0000 7f3d737fe640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d6006b4d0 0x7f3d6006d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3d74160440 tx=0x7f3d64009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.287+0000 7f3d5bfff640 1 -- 172.21.15.59:0/3430544644 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d6c010070 con 0x7f3d74161630 2024-03-06T07:26:15.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.469+0000 7f3d7a239640 1 -- 172.21.15.59:0/3430544644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3d741f4660 con 0x7f3d74161630 2024-03-06T07:26:15.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.842+0000 7f3d5bfff640 1 -- 172.21.15.59:0/3430544644 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3d6c058c40 con 0x7f3d74161630 2024-03-06T07:26:15.844 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-03-06T07:26:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.845+0000 7f3d7a239640 1 -- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d6006b4d0 msgr2=0x7f3d6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.845+0000 7f3d7a239640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d6006b4d0 0x7f3d6006d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3d74160440 tx=0x7f3d64009040 comp rx=0 tx=0).stop 2024-03-06T07:26:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.845+0000 7f3d7a239640 1 -- 172.21.15.59:0/3430544644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d74161630 msgr2=0x7f3d741f4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.845+0000 7f3d7a239640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d74161630 0x7f3d741f4120 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c007e60 tx=0x7f3d6c00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.845+0000 7f3d7a239640 1 -- 172.21.15.59:0/3430544644 shutdown_connections 2024-03-06T07:26:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.845+0000 7f3d7a239640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d6006b4d0 0x7f3d6006d990 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.845+0000 7f3d7a239640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d74161630 0x7f3d741f4120 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.845+0000 7f3d7a239640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d740cfa50 0x7f3d740ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.845+0000 7f3d7a239640 1 --2- 172.21.15.59:0/3430544644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d740cf590 0x7f3d740b8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.845+0000 7f3d7a239640 1 -- 172.21.15.59:0/3430544644 >> 172.21.15.59:0/3430544644 conn(0x7f3d740d3f30 msgr2=0x7f3d7415fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.845+0000 7f3d7a239640 1 -- 172.21.15.59:0/3430544644 shutdown_connections 2024-03-06T07:26:15.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.846+0000 7f3d7a239640 1 -- 172.21.15.59:0/3430544644 wait complete. 2024-03-06T07:26:15.863 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:26:15.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.971+0000 7f2dc0e8f640 1 Processor -- start 2024-03-06T07:26:15.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.971+0000 7f2dc0e8f640 1 -- start start 2024-03-06T07:26:15.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.971+0000 7f2dc0e8f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2dbc15bb50 0x7f2dbc1625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.971+0000 7f2dc0e8f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2dbc15f140 0x7f2dbc071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.971+0000 7f2dc0e8f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2dbc06e2c0 0x7f2dbc06e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.971+0000 7f2dc0e8f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dbc162160 con 0x7f2dbc15f140 2024-03-06T07:26:15.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.971+0000 7f2dc0e8f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dbc072870 con 0x7f2dbc06e2c0 2024-03-06T07:26:15.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.971+0000 7f2dc0e8f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc072b50 con 0x7f2dbc15bb50 2024-03-06T07:26:15.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.972+0000 7f2db3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2dbc15f140 0x7f2dbc071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.972+0000 7f2db3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2dbc15f140 0x7f2dbc071fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47276/0 (socket says 172.21.15.59:47276) 2024-03-06T07:26:15.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.972+0000 7f2db3fff640 1 -- 172.21.15.59:0/4179813690 learned_addr learned my addr 172.21.15.59:0/4179813690 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:15.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.972+0000 7f2dbad9d640 1 --2- 172.21.15.59:0/4179813690 >> v2:172.21.15.164:3300/0 conn(0x7f2dbc06e2c0 0x7f2dbc06e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.972+0000 7f2dba59c640 1 --2- 172.21.15.59:0/4179813690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2dbc15bb50 0x7f2dbc1625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.972+0000 7f2db3fff640 1 -- 172.21.15.59:0/4179813690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2dbc15bb50 msgr2=0x7f2dbc1625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.972+0000 7f2db3fff640 1 --2- 172.21.15.59:0/4179813690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2dbc15bb50 0x7f2dbc1625f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.972+0000 7f2db3fff640 1 -- 172.21.15.59:0/4179813690 >> v2:172.21.15.164:3300/0 conn(0x7f2dbc06e2c0 msgr2=0x7f2dbc06e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:15.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.972+0000 7f2db3fff640 1 --2- 172.21.15.59:0/4179813690 >> v2:172.21.15.164:3300/0 conn(0x7f2dbc06e2c0 0x7f2dbc06e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.972+0000 7f2db3fff640 1 -- 172.21.15.59:0/4179813690 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc161b10 con 0x7f2dbc15f140 2024-03-06T07:26:15.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.972+0000 7f2dbad9d640 1 --2- 172.21.15.59:0/4179813690 >> v2:172.21.15.164:3300/0 conn(0x7f2dbc06e2c0 0x7f2dbc06e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2db3fff640 1 --2- 172.21.15.59:0/4179813690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc15f140 0x7f2dbc071fb0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0af070 tx=0x7f2da402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b51f44595b12e62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2db9d9b640 1 -- 172.21.15.59:0/4179813690 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da402e070 con 0x7f2dbc15f140 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2db9d9b640 1 -- 172.21.15.59:0/4179813690 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da4037070 con 0x7f2dbc15f140 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2db9d9b640 1 -- 172.21.15.59:0/4179813690 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4031a00 con 0x7f2dbc15f140 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/4179813690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc15f140 msgr2=0x7f2dbc071fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2dc0e8f640 1 --2- 172.21.15.59:0/4179813690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc15f140 0x7f2dbc071fb0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0af070 tx=0x7f2da402b880 comp rx=0 tx=0).stop 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/4179813690 shutdown_connections 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2dc0e8f640 1 --2- 172.21.15.59:0/4179813690 >> v2:172.21.15.164:3300/0 conn(0x7f2dbc06e2c0 0x7f2dbc06e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2dc0e8f640 1 --2- 172.21.15.59:0/4179813690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc15f140 0x7f2dbc071fb0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2dc0e8f640 1 --2- 172.21.15.59:0/4179813690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2dbc15bb50 0x7f2dbc1625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/4179813690 >> 172.21.15.59:0/4179813690 conn(0x7f2dbc06b830 msgr2=0x7f2dbc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/4179813690 shutdown_connections 2024-03-06T07:26:15.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.973+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/4179813690 wait complete. 2024-03-06T07:26:15.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.975+0000 7f2dc0e8f640 1 Processor -- start 2024-03-06T07:26:15.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.975+0000 7f2dc0e8f640 1 -- start start 2024-03-06T07:26:15.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.975+0000 7f2dc0e8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc06e2c0 0x7f2dbc1e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.975+0000 7f2dc0e8f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2dbc15bb50 0x7f2dbc141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.975+0000 7f2dc0e8f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2dbc15f140 0x7f2dbc143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.975+0000 7f2dc0e8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc0ad860 con 0x7f2dbc06e2c0 2024-03-06T07:26:15.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.975+0000 7f2dc0e8f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc0724f0 con 0x7f2dbc15bb50 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.975+0000 7f2dc0e8f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc0ad9e0 con 0x7f2dbc15f140 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dba59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc06e2c0 0x7f2dbc1e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dba59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc06e2c0 0x7f2dbc1e78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47288/0 (socket says 172.21.15.59:47288) 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dba59c640 1 -- 172.21.15.59:0/1842488526 learned_addr learned my addr 172.21.15.59:0/1842488526 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2db3fff640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2dbc15bb50 0x7f2dbc141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dbad9d640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2dbc15f140 0x7f2dbc143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dba59c640 1 -- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2dbc15f140 msgr2=0x7f2dbc143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dba59c640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2dbc15f140 0x7f2dbc143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dba59c640 1 -- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2dbc15bb50 msgr2=0x7f2dbc141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dba59c640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2dbc15bb50 0x7f2dbc141450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dba59c640 1 -- 172.21.15.59:0/1842488526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc141f40 con 0x7f2dbc06e2c0 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dbad9d640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2dbc15f140 0x7f2dbc143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dba59c640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc06e2c0 0x7f2dbc1e78d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00d830 tx=0x7f2dac00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2db2ffd640 1 -- 172.21.15.59:0/1842488526 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac00b6c0 con 0x7f2dbc06e2c0 2024-03-06T07:26:15.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2db2ffd640 1 -- 172.21.15.59:0/1842488526 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac019070 con 0x7f2dbc06e2c0 2024-03-06T07:26:15.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/1842488526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dbc142220 con 0x7f2dbc06e2c0 2024-03-06T07:26:15.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2db2ffd640 1 -- 172.21.15.59:0/1842488526 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac00ba80 con 0x7f2dbc06e2c0 2024-03-06T07:26:15.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.976+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/1842488526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dbc1427e0 con 0x7f2dbc06e2c0 2024-03-06T07:26:15.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.977+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/1842488526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dbc06e0b0 con 0x7f2dbc06e2c0 2024-03-06T07:26:15.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.978+0000 7f2db2ffd640 1 -- 172.21.15.59:0/1842488526 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2dac01e460 con 0x7f2dbc06e2c0 2024-03-06T07:26:15.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.978+0000 7f2db2ffd640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2d9c06b4d0 0x7f2d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:15.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.978+0000 7f2db2ffd640 1 -- 172.21.15.59:0/1842488526 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f2dac08a790 con 0x7f2dbc06e2c0 2024-03-06T07:26:15.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.978+0000 7f2db3fff640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2d9c06b4d0 0x7f2d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:15.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.979+0000 7f2db3fff640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2d9c06b4d0 0x7f2d9c06d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2da4036000 tx=0x7f2da403d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:15.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:15.983+0000 7f2db2ffd640 1 -- 172.21.15.59:0/1842488526 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2dac059760 con 0x7f2dbc06e2c0 2024-03-06T07:26:16.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.164+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/1842488526 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2dbc068ff0 con 0x7f2dbc06e2c0 2024-03-06T07:26:16.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.848+0000 7f2db2ffd640 1 -- 172.21.15.59:0/1842488526 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v90) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2dac059060 con 0x7f2dbc06e2c0 2024-03-06T07:26:16.850 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-03-06T07:26:16.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.852+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2d9c06b4d0 msgr2=0x7f2d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.852+0000 7f2dc0e8f640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2d9c06b4d0 0x7f2d9c06d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2da4036000 tx=0x7f2da403d040 comp rx=0 tx=0).stop 2024-03-06T07:26:16.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.852+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/1842488526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc06e2c0 msgr2=0x7f2dbc1e78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.852+0000 7f2dc0e8f640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc06e2c0 0x7f2dbc1e78d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00d830 tx=0x7f2dac00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.852+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/1842488526 shutdown_connections 2024-03-06T07:26:16.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.852+0000 7f2dc0e8f640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2d9c06b4d0 0x7f2d9c06d990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.852+0000 7f2dc0e8f640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2dbc15f140 0x7f2dbc143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.852+0000 7f2dc0e8f640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2dbc15bb50 0x7f2dbc141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.852+0000 7f2dc0e8f640 1 --2- 172.21.15.59:0/1842488526 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2dbc06e2c0 0x7f2dbc1e78d0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.852+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/1842488526 >> 172.21.15.59:0/1842488526 conn(0x7f2dbc06b830 msgr2=0x7f2dbc0b2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.853+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/1842488526 shutdown_connections 2024-03-06T07:26:16.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.853+0000 7f2dc0e8f640 1 -- 172.21.15.59:0/1842488526 wait complete. 2024-03-06T07:26:16.881 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:26:16.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.987+0000 7f392cc8f640 1 Processor -- start 2024-03-06T07:26:16.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.987+0000 7f392cc8f640 1 -- start start 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.988+0000 7f392cc8f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f392806e470 0x7f3928161a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.988+0000 7f392cc8f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39280b3050 con 0x7f3928161fb0 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.988+0000 7f392cc8f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39280b3330 con 0x7f3928147840 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.988+0000 7f392cc8f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39280b2e00 con 0x7f392806e470 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.988+0000 7f3926d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3928161fb0 0x7f39280ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34340/0 (socket says 172.21.15.59:34340) 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.988+0000 7f3926d9d640 1 -- 172.21.15.59:0/1090110233 learned_addr learned my addr 172.21.15.59:0/1090110233 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.988+0000 7f3925d9b640 1 --2- 172.21.15.59:0/1090110233 >> v2:172.21.15.164:3300/0 conn(0x7f392806e470 0x7f3928161a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.988+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827802395 0 0) 0x7f39280b3050 con 0x7f3928161fb0 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.988+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3914003520 con 0x7f3928161fb0 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1464121567 0 0) 0x7f3914003520 con 0x7f3928161fb0 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 >> v2:172.21.15.164:3300/0 conn(0x7f392806e470 msgr2=0x7f3928161a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 --2- 172.21.15.59:0/1090110233 >> v2:172.21.15.164:3300/0 conn(0x7f392806e470 0x7f3928161a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 >> v1:172.21.15.164:6789/0 conn(0x7f3928147840 legacy=0x7f392806f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392806d990 con 0x7f3928161fb0 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f39180019f0 con 0x7f3928161fb0 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 >> v1:172.21.15.59:6789/0 conn(0x7f3928161fb0 legacy=0x7f39280ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39140039b0 0x7f3914003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3914004800 0x7f3914004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39140051c0 0x7f391400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39280b2e00 con 0x7f3914004800 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391400c160 con 0x7f39140039b0 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f391400c360 con 0x7f39140051c0 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f3925d9b640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3914004800 0x7f3914004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392659c640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39140039b0 0x7f3914003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f3926d9d640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39140051c0 0x7f391400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f3925d9b640 1 -- 172.21.15.59:0/1090110233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39140051c0 msgr2=0x7f391400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f3925d9b640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39140051c0 0x7f391400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f3925d9b640 1 -- 172.21.15.59:0/1090110233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39140039b0 msgr2=0x7f3914003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:16.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f3925d9b640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39140039b0 0x7f3914003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f3925d9b640 1 -- 172.21.15.59:0/1090110233 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3914003520 con 0x7f3914004800 2024-03-06T07:26:16.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f392659c640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39140039b0 0x7f3914003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:16.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.989+0000 7f3926d9d640 1 -- 172.21.15.59:0/1090110233 reap_dead start 2024-03-06T07:26:16.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.990+0000 7f3925d9b640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3914004800 0x7f3914004c80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f3910009ac0 tx=0x7f3914005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23a5ee223707b75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.990+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3910039070 con 0x7f3914004800 2024-03-06T07:26:16.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.990+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391002c6b0 con 0x7f3914004800 2024-03-06T07:26:16.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.990+0000 7f392559a640 1 -- 172.21.15.59:0/1090110233 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3910009690 con 0x7f3914004800 2024-03-06T07:26:16.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.990+0000 7f392cc8f640 1 -- 172.21.15.59:0/1090110233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3914004800 msgr2=0x7f3914004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.990+0000 7f392cc8f640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3914004800 0x7f3914004c80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f3910009ac0 tx=0x7f3914005830 comp rx=0 tx=0).stop 2024-03-06T07:26:16.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.990+0000 7f392cc8f640 1 -- 172.21.15.59:0/1090110233 shutdown_connections 2024-03-06T07:26:16.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.990+0000 7f392cc8f640 1 --2- 172.21.15.59:0/1090110233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3914004800 0x7f3914004c80 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.990+0000 7f392cc8f640 1 -- 172.21.15.59:0/1090110233 >> 172.21.15.59:0/1090110233 conn(0x7f39280afa70 msgr2=0x7f3928072db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:16.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.991+0000 7f392cc8f640 1 -- 172.21.15.59:0/1090110233 shutdown_connections 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.991+0000 7f392cc8f640 1 -- 172.21.15.59:0/1090110233 wait complete. 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.992+0000 7f392cc8f640 1 Processor -- start 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.992+0000 7f392cc8f640 1 -- start start 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.992+0000 7f392cc8f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3914004800 0x7f39281f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.992+0000 7f392cc8f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f392815faa0 0x7f392815ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.992+0000 7f392cc8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3928140fe0 0x7f3928141460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.992+0000 7f392cc8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39281623e0 con 0x7f3928140fe0 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f392cc8f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392806e8d0 con 0x7f392815faa0 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f392cc8f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392806df50 con 0x7f3914004800 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f3926d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3928140fe0 0x7f3928141460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f3926d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3928140fe0 0x7f3928141460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47312/0 (socket says 172.21.15.59:47312) 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f3926d9d640 1 -- 172.21.15.59:0/1218163928 learned_addr learned my addr 172.21.15.59:0/1218163928 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f3925d9b640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f392815faa0 0x7f392815ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f392659c640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3914004800 0x7f39281f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:16.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f3926d9d640 1 -- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3914004800 msgr2=0x7f39281f3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f3926d9d640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3914004800 0x7f39281f3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f3926d9d640 1 -- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f392815faa0 msgr2=0x7f392815ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f3926d9d640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f392815faa0 0x7f392815ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f3926d9d640 1 -- 172.21.15.59:0/1218163928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39281302d0 con 0x7f3928140fe0 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f3926d9d640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3928140fe0 0x7f3928141460 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f39281619d0 tx=0x7f391800e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f391effd640 1 -- 172.21.15.59:0/1218163928 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391800eb70 con 0x7f3928140fe0 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f391effd640 1 -- 172.21.15.59:0/1218163928 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391800ed50 con 0x7f3928140fe0 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f392cc8f640 1 -- 172.21.15.59:0/1218163928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39281305b0 con 0x7f3928140fe0 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f391effd640 1 -- 172.21.15.59:0/1218163928 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39180146c0 con 0x7f3928140fe0 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.993+0000 7f392cc8f640 1 -- 172.21.15.59:0/1218163928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3928130b70 con 0x7f3928140fe0 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.994+0000 7f392cc8f640 1 -- 172.21.15.59:0/1218163928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392804f330 con 0x7f3928140fe0 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.995+0000 7f391effd640 1 -- 172.21.15.59:0/1218163928 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3918014860 con 0x7f3928140fe0 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.995+0000 7f391effd640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f390806b4d0 0x7f390806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.996+0000 7f391effd640 1 -- 172.21.15.59:0/1218163928 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f391808a040 con 0x7f3928140fe0 2024-03-06T07:26:16.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.996+0000 7f392659c640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f390806b4d0 0x7f390806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:17.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:16.996+0000 7f392659c640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f390806b4d0 0x7f390806d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3920000c00 tx=0x7f3920009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:17.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.000+0000 7f391effd640 1 -- 172.21.15.59:0/1218163928 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3918058f20 con 0x7f3928140fe0 2024-03-06T07:26:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.218+0000 7f392cc8f640 1 -- 172.21.15.59:0/1218163928 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f39280b1590 con 0x7f3928140fe0 2024-03-06T07:26:17.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.865+0000 7f391effd640 1 -- 172.21.15.59:0/1218163928 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v47) v1 ==== 174+0+0 (secure 0 0 0) 0x7f3918058820 con 0x7f3928140fe0 2024-03-06T07:26:17.869 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 14 and data pool 15 2024-03-06T07:26:17.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.877+0000 7f392cc8f640 1 -- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f390806b4d0 msgr2=0x7f390806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.877+0000 7f392cc8f640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f390806b4d0 0x7f390806d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3920000c00 tx=0x7f3920009040 comp rx=0 tx=0).stop 2024-03-06T07:26:17.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.877+0000 7f392cc8f640 1 -- 172.21.15.59:0/1218163928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3928140fe0 msgr2=0x7f3928141460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:17.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.877+0000 7f392cc8f640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3928140fe0 0x7f3928141460 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f39281619d0 tx=0x7f391800e160 comp rx=0 tx=0).stop 2024-03-06T07:26:17.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.878+0000 7f392cc8f640 1 -- 172.21.15.59:0/1218163928 shutdown_connections 2024-03-06T07:26:17.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.878+0000 7f392cc8f640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f390806b4d0 0x7f390806d990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.878+0000 7f392cc8f640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3928140fe0 0x7f3928141460 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.878+0000 7f392cc8f640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f392815faa0 0x7f392815ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.878+0000 7f392cc8f640 1 --2- 172.21.15.59:0/1218163928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3914004800 0x7f39281f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.878+0000 7f392cc8f640 1 -- 172.21.15.59:0/1218163928 >> 172.21.15.59:0/1218163928 conn(0x7f39280afa70 msgr2=0x7f3928145b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:17.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.879+0000 7f392cc8f640 1 -- 172.21.15.59:0/1218163928 shutdown_connections 2024-03-06T07:26:17.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:17.880+0000 7f392cc8f640 1 -- 172.21.15.59:0/1218163928 wait complete. 2024-03-06T07:26:17.896 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.000+0000 7fb9322c9640 1 Processor -- start 2024-03-06T07:26:18.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.000+0000 7fb9322c9640 1 -- start start 2024-03-06T07:26:18.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.001+0000 7fb9322c9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb92c1621c0 0x7fb92c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.001+0000 7fb9322c9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb92c1625a0 0x7fb92c15dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.001+0000 7fb9322c9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb92c0ceac0 con 0x7fb92c0d4920 2024-03-06T07:26:18.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.001+0000 7fb9322c9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92c0cf690 con 0x7fb92c1625a0 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.001+0000 7fb9322c9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92c0d2dc0 con 0x7fb92c1621c0 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.001+0000 7fb930866640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb92c1625a0 0x7fb92c15dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.001+0000 7fb930866640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb92c1625a0 0x7fb92c15dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47320/0 (socket says 172.21.15.59:47320) 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.001+0000 7fb930866640 1 -- 172.21.15.59:0/4083098897 learned_addr learned my addr 172.21.15.59:0/4083098897 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.001+0000 7fb92bfff640 1 --2- 172.21.15.59:0/4083098897 >> v2:172.21.15.164:3300/0 conn(0x7fb92c1621c0 0x7fb92c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.002+0000 7fb930866640 1 -- 172.21.15.59:0/4083098897 >> v2:172.21.15.164:3300/0 conn(0x7fb92c1621c0 msgr2=0x7fb92c0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.002+0000 7fb930866640 1 --2- 172.21.15.59:0/4083098897 >> v2:172.21.15.164:3300/0 conn(0x7fb92c1621c0 0x7fb92c0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.002+0000 7fb930866640 1 -- 172.21.15.59:0/4083098897 >> v1:172.21.15.164:6789/0 conn(0x7fb92c0d4920 legacy=0x7fb92c0d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.002+0000 7fb930866640 1 -- 172.21.15.59:0/4083098897 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c0ced60 con 0x7fb92c1625a0 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.002+0000 7fb930866640 1 --2- 172.21.15.59:0/4083098897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb92c1625a0 0x7fb92c15dd40 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0d3ce0 tx=0x7fb92402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d693a6d9951f017 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.002+0000 7fb92affd640 1 -- 172.21.15.59:0/4083098897 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92402ca80 con 0x7fb92c1625a0 2024-03-06T07:26:18.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.002+0000 7fb92affd640 1 -- 172.21.15.59:0/4083098897 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92402cc20 con 0x7fb92c1625a0 2024-03-06T07:26:18.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.003+0000 7fb92affd640 1 -- 172.21.15.59:0/4083098897 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9240343f0 con 0x7fb92c1625a0 2024-03-06T07:26:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.003+0000 7fb9322c9640 1 -- 172.21.15.59:0/4083098897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb92c1625a0 msgr2=0x7fb92c15dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.003+0000 7fb9322c9640 1 --2- 172.21.15.59:0/4083098897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb92c1625a0 0x7fb92c15dd40 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0d3ce0 tx=0x7fb92402bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.003+0000 7fb9322c9640 1 -- 172.21.15.59:0/4083098897 shutdown_connections 2024-03-06T07:26:18.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.003+0000 7fb9322c9640 1 --2- 172.21.15.59:0/4083098897 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb92c1625a0 0x7fb92c15dd40 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.003+0000 7fb9322c9640 1 --2- 172.21.15.59:0/4083098897 >> v2:172.21.15.164:3300/0 conn(0x7fb92c1621c0 0x7fb92c0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.003+0000 7fb9322c9640 1 -- 172.21.15.59:0/4083098897 >> 172.21.15.59:0/4083098897 conn(0x7fb92c0d3d10 msgr2=0x7fb92c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.003+0000 7fb9322c9640 1 -- 172.21.15.59:0/4083098897 shutdown_connections 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.003+0000 7fb9322c9640 1 -- 172.21.15.59:0/4083098897 wait complete. 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.004+0000 7fb9322c9640 1 Processor -- start 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.004+0000 7fb9322c9640 1 -- start start 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb9322c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb92c0d4920 0x7fb92c0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb9322c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb92c1621c0 0x7fb92c09da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb9322c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb92c1625a0 0x7fb92c09eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb9322c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c15e280 con 0x7fb92c0d4920 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb9322c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c0cf320 con 0x7fb92c1621c0 2024-03-06T07:26:18.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb9322c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c15e400 con 0x7fb92c1625a0 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb92bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb92c0d4920 0x7fb92c0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb92bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb92c0d4920 0x7fb92c0d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47324/0 (socket says 172.21.15.59:47324) 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb92bfff640 1 -- 172.21.15.59:0/2909955927 learned_addr learned my addr 172.21.15.59:0/2909955927 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb92bfff640 1 -- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb92c1625a0 msgr2=0x7fb92c09eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb92bfff640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb92c1625a0 0x7fb92c09eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb92b7fe640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb92c1621c0 0x7fb92c09da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb92bfff640 1 -- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb92c1621c0 msgr2=0x7fb92c09da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb92bfff640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb92c1621c0 0x7fb92c09da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.005+0000 7fb92bfff640 1 -- 172.21.15.59:0/2909955927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c0a0b20 con 0x7fb92c0d4920 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.006+0000 7fb92bfff640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb92c0d4920 0x7fb92c0d0590 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb92000d830 tx=0x7fb92000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.006+0000 7fb928ff9640 1 -- 172.21.15.59:0/2909955927 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92000b6c0 con 0x7fb92c0d4920 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.006+0000 7fb928ff9640 1 -- 172.21.15.59:0/2909955927 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb920019070 con 0x7fb92c0d4920 2024-03-06T07:26:18.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.006+0000 7fb9322c9640 1 -- 172.21.15.59:0/2909955927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92c0a0e00 con 0x7fb92c0d4920 2024-03-06T07:26:18.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.006+0000 7fb928ff9640 1 -- 172.21.15.59:0/2909955927 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92000ba80 con 0x7fb92c0d4920 2024-03-06T07:26:18.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.006+0000 7fb9322c9640 1 -- 172.21.15.59:0/2909955927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92c0a13c0 con 0x7fb92c0d4920 2024-03-06T07:26:18.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.007+0000 7fb9322c9640 1 -- 172.21.15.59:0/2909955927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92c04f330 con 0x7fb92c0d4920 2024-03-06T07:26:18.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.007+0000 7fb928ff9640 1 -- 172.21.15.59:0/2909955927 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb92001e460 con 0x7fb92c0d4920 2024-03-06T07:26:18.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.007+0000 7fb928ff9640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb91806b500 0x7fb91806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.008+0000 7fb928ff9640 1 -- 172.21.15.59:0/2909955927 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb92008a950 con 0x7fb92c0d4920 2024-03-06T07:26:18.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.008+0000 7fb92b7fe640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb91806b500 0x7fb91806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.008+0000 7fb92b7fe640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb91806b500 0x7fb91806d9c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0d1320 tx=0x7fb91c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.012+0000 7fb928ff9640 1 -- 172.21.15.59:0/2909955927 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb920059740 con 0x7fb92c0d4920 2024-03-06T07:26:18.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.194+0000 7fb9322c9640 1 -- 172.21.15.59:0/2909955927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb92c06bd20 con 0x7fb92c0d4920 2024-03-06T07:26:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.195+0000 7fb928ff9640 1 -- 172.21.15.59:0/2909955927 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16609 (secure 0 0 0) 0x7fb920059040 con 0x7fb92c0d4920 2024-03-06T07:26:18.197 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:18.197 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":91,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:26:17.219416+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.489280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:16.183614+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 -- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb91806b500 msgr2=0x7fb91806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb91806b500 0x7fb91806d9c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0d1320 tx=0x7fb91c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 -- 172.21.15.59:0/2909955927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb92c0d4920 msgr2=0x7fb92c0d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb92c0d4920 0x7fb92c0d0590 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb92000d830 tx=0x7fb92000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 -- 172.21.15.59:0/2909955927 shutdown_connections 2024-03-06T07:26:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb91806b500 0x7fb91806d9c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb92c1625a0 0x7fb92c09eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb92c1621c0 0x7fb92c09da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 --2- 172.21.15.59:0/2909955927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb92c0d4920 0x7fb92c0d0590 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 -- 172.21.15.59:0/2909955927 >> 172.21.15.59:0/2909955927 conn(0x7fb92c0d3d10 msgr2=0x7fb92c15e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 -- 172.21.15.59:0/2909955927 shutdown_connections 2024-03-06T07:26:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.197+0000 7fb9322c9640 1 -- 172.21.15.59:0/2909955927 wait complete. 2024-03-06T07:26:18.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:18.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.309+0000 7f62abf25640 1 Processor -- start 2024-03-06T07:26:18.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.309+0000 7f62abf25640 1 -- start start 2024-03-06T07:26:18.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.310+0000 7f62abf25640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f62a40afdd0 0x7f62a415c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.310+0000 7f62abf25640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415e740 con 0x7f62a40d14f0 2024-03-06T07:26:18.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.310+0000 7f62abf25640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a415e9e0 con 0x7f62a4158920 2024-03-06T07:26:18.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.310+0000 7f62abf25640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a415f5f0 con 0x7f62a40afdd0 2024-03-06T07:26:18.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.310+0000 7f62a94c0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f62a40afdd0 0x7f62a415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.310+0000 7f62aa4c2640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f62a40d14f0 0x7f62a415b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34348/0 (socket says 172.21.15.59:34348) 2024-03-06T07:26:18.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.310+0000 7f62aa4c2640 1 -- 172.21.15.59:0/1077525484 learned_addr learned my addr 172.21.15.59:0/1077525484 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:18.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.311+0000 7f62a94c0640 1 -- 172.21.15.59:0/1077525484 >> v1:172.21.15.164:6789/0 conn(0x7f62a4158920 legacy=0x7f62a40af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.311+0000 7f62a94c0640 1 -- 172.21.15.59:0/1077525484 >> v1:172.21.15.59:6789/0 conn(0x7f62a40d14f0 legacy=0x7f62a415b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.311+0000 7f62a94c0640 1 -- 172.21.15.59:0/1077525484 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a415ecc0 con 0x7f62a40afdd0 2024-03-06T07:26:18.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.311+0000 7f62a8cbf640 1 -- 172.21.15.59:0/1077525484 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357052753 0 0) 0x7f62a415e740 con 0x7f62a40d14f0 2024-03-06T07:26:18.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.311+0000 7f62a94c0640 1 --2- 172.21.15.59:0/1077525484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a40afdd0 0x7f62a415c310 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f62980097a0 tx=0x7f629802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba146cf3bdab2698 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.311+0000 7f62a8cbf640 1 -- 172.21.15.59:0/1077525484 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6298035020 con 0x7f62a40afdd0 2024-03-06T07:26:18.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.311+0000 7f62a8cbf640 1 -- 172.21.15.59:0/1077525484 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6298031830 con 0x7f62a40afdd0 2024-03-06T07:26:18.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.311+0000 7f62a8cbf640 1 -- 172.21.15.59:0/1077525484 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6298031c00 con 0x7f62a40afdd0 2024-03-06T07:26:18.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.312+0000 7f62abf25640 1 -- 172.21.15.59:0/1077525484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a40afdd0 msgr2=0x7f62a415c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.312+0000 7f62abf25640 1 --2- 172.21.15.59:0/1077525484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a40afdd0 0x7f62a415c310 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f62980097a0 tx=0x7f629802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.312+0000 7f62abf25640 1 -- 172.21.15.59:0/1077525484 shutdown_connections 2024-03-06T07:26:18.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.312+0000 7f62abf25640 1 --2- 172.21.15.59:0/1077525484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a40afdd0 0x7f62a415c310 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.312+0000 7f62abf25640 1 -- 172.21.15.59:0/1077525484 >> 172.21.15.59:0/1077525484 conn(0x7f62a40d0960 msgr2=0x7f62a40d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.312+0000 7f62abf25640 1 -- 172.21.15.59:0/1077525484 shutdown_connections 2024-03-06T07:26:18.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.312+0000 7f62abf25640 1 -- 172.21.15.59:0/1077525484 wait complete. 2024-03-06T07:26:18.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.313+0000 7f62abf25640 1 Processor -- start 2024-03-06T07:26:18.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.313+0000 7f62abf25640 1 -- start start 2024-03-06T07:26:18.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62abf25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a40afdd0 0x7f62a4157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62abf25640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62a40d14f0 0x7f62a40b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62abf25640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62a4158920 0x7f62a41f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62abf25640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a406f5a0 con 0x7f62a40afdd0 2024-03-06T07:26:18.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62abf25640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a415f280 con 0x7f62a40d14f0 2024-03-06T07:26:18.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62abf25640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a406f720 con 0x7f62a4158920 2024-03-06T07:26:18.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62a9cc1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a40afdd0 0x7f62a4157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62a9cc1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a40afdd0 0x7f62a4157b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47348/0 (socket says 172.21.15.59:47348) 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62a9cc1640 1 -- 172.21.15.59:0/1478134329 learned_addr learned my addr 172.21.15.59:0/1478134329 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62a9cc1640 1 -- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62a4158920 msgr2=0x7f62a41f5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62a9cc1640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62a4158920 0x7f62a41f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62a9cc1640 1 -- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62a40d14f0 msgr2=0x7f62a40b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62a9cc1640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62a40d14f0 0x7f62a40b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.314+0000 7f62a9cc1640 1 -- 172.21.15.59:0/1478134329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a40b3c20 con 0x7f62a40afdd0 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.315+0000 7f62a9cc1640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a40afdd0 0x7f62a4157b50 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f629c00b3e0 tx=0x7f629c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.315+0000 7f62a27fc640 1 -- 172.21.15.59:0/1478134329 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c015020 con 0x7f62a40afdd0 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.315+0000 7f62a27fc640 1 -- 172.21.15.59:0/1478134329 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629c011850 con 0x7f62a40afdd0 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.315+0000 7f62abf25640 1 -- 172.21.15.59:0/1478134329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a415dc10 con 0x7f62a40afdd0 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.315+0000 7f62a27fc640 1 -- 172.21.15.59:0/1478134329 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c011c40 con 0x7f62a40afdd0 2024-03-06T07:26:18.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.315+0000 7f62abf25640 1 -- 172.21.15.59:0/1478134329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a415d930 con 0x7f62a40afdd0 2024-03-06T07:26:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.316+0000 7f62abf25640 1 -- 172.21.15.59:0/1478134329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a4070240 con 0x7f62a40afdd0 2024-03-06T07:26:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.316+0000 7f62a27fc640 1 -- 172.21.15.59:0/1478134329 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f629c018020 con 0x7f62a40afdd0 2024-03-06T07:26:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.316+0000 7f62a27fc640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f629006b500 0x7f629006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.317+0000 7f62a27fc640 1 -- 172.21.15.59:0/1478134329 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f629c08a070 con 0x7f62a40afdd0 2024-03-06T07:26:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.317+0000 7f62a94c0640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f629006b500 0x7f629006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.317+0000 7f62a94c0640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f629006b500 0x7f629006d9c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6298036000 tx=0x7f6298030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.321+0000 7f62a27fc640 1 -- 172.21.15.59:0/1478134329 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62a4070240 con 0x7f62a40afdd0 2024-03-06T07:26:18.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.501+0000 7f62abf25640 1 -- 172.21.15.59:0/1478134329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62a404f330 con 0x7f62a40afdd0 2024-03-06T07:26:18.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.502+0000 7f62a27fc640 1 -- 172.21.15.59:0/1478134329 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f629c01f020 con 0x7f62a40afdd0 2024-03-06T07:26:18.504 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:18.504 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":91,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:26:17.219416+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.489280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:16.183614+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.504+0000 7f62abf25640 1 -- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f629006b500 msgr2=0x7f629006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.504+0000 7f62abf25640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f629006b500 0x7f629006d9c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6298036000 tx=0x7f6298030000 comp rx=0 tx=0).stop 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.504+0000 7f62abf25640 1 -- 172.21.15.59:0/1478134329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a40afdd0 msgr2=0x7f62a4157b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.504+0000 7f62abf25640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a40afdd0 0x7f62a4157b50 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f629c00b3e0 tx=0x7f629c00b8a0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.505+0000 7f62abf25640 1 -- 172.21.15.59:0/1478134329 shutdown_connections 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.505+0000 7f62abf25640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f629006b500 0x7f629006d9c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.505+0000 7f62abf25640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62a4158920 0x7f62a41f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.505+0000 7f62abf25640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62a40d14f0 0x7f62a40b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.505+0000 7f62abf25640 1 --2- 172.21.15.59:0/1478134329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a40afdd0 0x7f62a4157b50 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.505+0000 7f62abf25640 1 -- 172.21.15.59:0/1478134329 >> 172.21.15.59:0/1478134329 conn(0x7f62a40d0960 msgr2=0x7f62a415a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.505+0000 7f62abf25640 1 -- 172.21.15.59:0/1478134329 shutdown_connections 2024-03-06T07:26:18.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.505+0000 7f62abf25640 1 -- 172.21.15.59:0/1478134329 wait complete. 2024-03-06T07:26:18.518 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:26:18.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.620+0000 7f44edcd4640 1 Processor -- start 2024-03-06T07:26:18.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.620+0000 7f44edcd4640 1 -- start start 2024-03-06T07:26:18.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.621+0000 7f44edcd4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f44e815def0 0x7f44e80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.621+0000 7f44edcd4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e806d030 con 0x7f44e80cea50 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.621+0000 7f44edcd4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e806d310 con 0x7f44e806cab0 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.621+0000 7f44edcd4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e80cf1c0 con 0x7f44e815def0 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.621+0000 7f44e7fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f44e80cea50 0x7f44e80ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34358/0 (socket says 172.21.15.59:34358) 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.621+0000 7f44e7fff640 1 -- 172.21.15.59:0/2738437919 learned_addr learned my addr 172.21.15.59:0/2738437919 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.621+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654190549 0 0) 0x7f44e806d030 con 0x7f44e80cea50 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.621+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d0003520 con 0x7f44e80cea50 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.621+0000 7f44e77fe640 1 --2- 172.21.15.59:0/2738437919 >> v2:172.21.15.164:3300/0 conn(0x7f44e815def0 0x7f44e80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3176089666 0 0) 0x7f44d0003520 con 0x7f44e80cea50 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 >> v2:172.21.15.164:3300/0 conn(0x7f44e815def0 msgr2=0x7f44e80b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 --2- 172.21.15.59:0/2738437919 >> v2:172.21.15.164:3300/0 conn(0x7f44e815def0 0x7f44e80b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 >> v1:172.21.15.164:6789/0 conn(0x7f44e806cab0 legacy=0x7f44e80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80cc110 con 0x7f44e80cea50 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f44e00019f0 con 0x7f44e80cea50 2024-03-06T07:26:18.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e77fe640 1 --2- 172.21.15.59:0/2738437919 >> v2:172.21.15.164:3300/0 conn(0x7f44e815def0 0x7f44e80b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 >> v1:172.21.15.59:6789/0 conn(0x7f44e80cea50 legacy=0x7f44e80ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 --2- 172.21.15.59:0/2738437919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44d00039b0 0x7f44d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 --2- 172.21.15.59:0/2738437919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44d0004800 0x7f44d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 --2- 172.21.15.59:0/2738437919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 0x7f44d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d000c0c0 con 0x7f44d00051c0 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d000c270 con 0x7f44d0004800 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d000c470 con 0x7f44d00039b0 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e7fff640 1 --2- 172.21.15.59:0/2738437919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 0x7f44d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e77fe640 1 --2- 172.21.15.59:0/2738437919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44d00039b0 0x7f44d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e6ffd640 1 --2- 172.21.15.59:0/2738437919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44d0004800 0x7f44d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e7fff640 1 -- 172.21.15.59:0/2738437919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44d00039b0 msgr2=0x7f44d0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e7fff640 1 --2- 172.21.15.59:0/2738437919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44d00039b0 0x7f44d0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e7fff640 1 -- 172.21.15.59:0/2738437919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44d0004800 msgr2=0x7f44d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e7fff640 1 --2- 172.21.15.59:0/2738437919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44d0004800 0x7f44d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.622+0000 7f44e7fff640 1 -- 172.21.15.59:0/2738437919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d0003520 con 0x7f44d00051c0 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.623+0000 7f44e7fff640 1 -- 172.21.15.59:0/2738437919 reap_dead start 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.623+0000 7f44e7fff640 1 --2- 172.21.15.59:0/2738437919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 0x7f44d000b9c0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f44e80b7c50 tx=0x7f44d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f18c6e86b18ad7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.623+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e000a4a0 con 0x7f44d00051c0 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.623+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e0034650 con 0x7f44d00051c0 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.623+0000 7f44e67fc640 1 -- 172.21.15.59:0/2738437919 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0034a40 con 0x7f44d00051c0 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.624+0000 7f44edcd4640 1 -- 172.21.15.59:0/2738437919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 msgr2=0x7f44d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.624+0000 7f44edcd4640 1 --2- 172.21.15.59:0/2738437919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 0x7f44d000b9c0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f44e80b7c50 tx=0x7f44d0005830 comp rx=0 tx=0).stop 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.624+0000 7f44edcd4640 1 -- 172.21.15.59:0/2738437919 shutdown_connections 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.624+0000 7f44edcd4640 1 --2- 172.21.15.59:0/2738437919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 0x7f44d000b9c0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.624+0000 7f44edcd4640 1 -- 172.21.15.59:0/2738437919 >> 172.21.15.59:0/2738437919 conn(0x7f44e80b3690 msgr2=0x7f44e80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.624+0000 7f44edcd4640 1 -- 172.21.15.59:0/2738437919 shutdown_connections 2024-03-06T07:26:18.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.624+0000 7f44edcd4640 1 -- 172.21.15.59:0/2738437919 wait complete. 2024-03-06T07:26:18.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.625+0000 7f44edcd4640 1 Processor -- start 2024-03-06T07:26:18.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.625+0000 7f44edcd4640 1 -- start start 2024-03-06T07:26:18.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44edcd4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 0x7f44e815cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44edcd4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80b6680 0x7f44e80c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44edcd4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e80b4780 0x7f44e80b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44edcd4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e815e2d0 con 0x7f44d00051c0 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44edcd4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80cc760 con 0x7f44e80b6680 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44edcd4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e815e450 con 0x7f44e80b4780 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 0x7f44e815cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 0x7f44e815cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47376/0 (socket says 172.21.15.59:47376) 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e77fe640 1 -- 172.21.15.59:0/3834187655 learned_addr learned my addr 172.21.15.59:0/3834187655 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e6ffd640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80b6680 0x7f44e80c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e7fff640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e80b4780 0x7f44e80b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e77fe640 1 -- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e80b4780 msgr2=0x7f44e80b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e77fe640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e80b4780 0x7f44e80b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e77fe640 1 -- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80b6680 msgr2=0x7f44e80c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e77fe640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80b6680 0x7f44e80c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e77fe640 1 -- 172.21.15.59:0/3834187655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80b6bd0 con 0x7f44d00051c0 2024-03-06T07:26:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e6ffd640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80b6680 0x7f44e80c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44e77fe640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 0x7f44e815cbc0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f44d80115f0 tx=0x7f44d8011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44cffff640 1 -- 172.21.15.59:0/3834187655 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8019070 con 0x7f44d00051c0 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44cffff640 1 -- 172.21.15.59:0/3834187655 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d8004220 con 0x7f44d00051c0 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44edcd4640 1 -- 172.21.15.59:0/3834187655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e80b6eb0 con 0x7f44d00051c0 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44cffff640 1 -- 172.21.15.59:0/3834187655 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d800f040 con 0x7f44d00051c0 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.626+0000 7f44edcd4640 1 -- 172.21.15.59:0/3834187655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e80c58c0 con 0x7f44d00051c0 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.628+0000 7f44edcd4640 1 -- 172.21.15.59:0/3834187655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e80b3a70 con 0x7f44d00051c0 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.628+0000 7f44cffff640 1 -- 172.21.15.59:0/3834187655 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44d800f6f0 con 0x7f44d00051c0 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.628+0000 7f44cffff640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 0x7f44d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.629+0000 7f44cffff640 1 -- 172.21.15.59:0/3834187655 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44d808a430 con 0x7f44d00051c0 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.629+0000 7f44e6ffd640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 0x7f44d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.629+0000 7f44e6ffd640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 0x7f44d406d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f44dc000c00 tx=0x7f44dc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.633+0000 7f44cffff640 1 -- 172.21.15.59:0/3834187655 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44e80b3a70 con 0x7f44d00051c0 2024-03-06T07:26:18.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.855+0000 7f44edcd4640 1 -- 172.21.15.59:0/3834187655 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f44e809fcf0 con 0x7f44d00051c0 2024-03-06T07:26:18.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.862+0000 7f44cffff640 1 -- 172.21.15.59:0/3834187655 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v48) v1 ==== 118+0+0 (secure 0 0 0) 0x7f44d80592b0 con 0x7f44d00051c0 2024-03-06T07:26:18.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 -- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 msgr2=0x7f44d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 0x7f44d406d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f44dc000c00 tx=0x7f44dc009040 comp rx=0 tx=0).stop 2024-03-06T07:26:18.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 -- 172.21.15.59:0/3834187655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 msgr2=0x7f44e815cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 0x7f44e815cbc0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f44d80115f0 tx=0x7f44d8011ab0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 -- 172.21.15.59:0/3834187655 shutdown_connections 2024-03-06T07:26:18.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 0x7f44d406d990 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44e80b4780 0x7f44e80b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44e80b6680 0x7f44e80c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 --2- 172.21.15.59:0/3834187655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44d00051c0 0x7f44e815cbc0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 -- 172.21.15.59:0/3834187655 >> 172.21.15.59:0/3834187655 conn(0x7f44e80b3690 msgr2=0x7f44e80caba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 -- 172.21.15.59:0/3834187655 shutdown_connections 2024-03-06T07:26:18.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.865+0000 7f44edcd4640 1 -- 172.21.15.59:0/3834187655 wait complete. 2024-03-06T07:26:18.880 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:26:18.880 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.982+0000 7ff99b7ac640 1 Processor -- start 2024-03-06T07:26:18.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.982+0000 7ff99b7ac640 1 -- start start 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.983+0000 7ff99b7ac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff9940d10c0 0x7ff9940d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.983+0000 7ff99b7ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99415ecd0 0x7ff9940cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.983+0000 7ff99b7ac640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9940cfa50 con 0x7ff9940d14a0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.983+0000 7ff99b7ac640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff99415c050 con 0x7ff9940d10c0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.983+0000 7ff99b7ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9940d0060 con 0x7ff99415ecd0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.983+0000 7ff999548640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff9940d10c0 0x7ff9940d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.983+0000 7ff998d47640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff9940d14a0 0x7ff99415e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34366/0 (socket says 172.21.15.59:34366) 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.983+0000 7ff998d47640 1 -- 172.21.15.59:0/1246493332 learned_addr learned my addr 172.21.15.59:0/1246493332 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.983+0000 7ff999d49640 1 --2- 172.21.15.59:0/1246493332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99415ecd0 0x7ff9940cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.984+0000 7ff999548640 1 -- 172.21.15.59:0/1246493332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99415ecd0 msgr2=0x7ff9940cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.984+0000 7ff999548640 1 --2- 172.21.15.59:0/1246493332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99415ecd0 0x7ff9940cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.984+0000 7ff999548640 1 -- 172.21.15.59:0/1246493332 >> v1:172.21.15.59:6789/0 conn(0x7ff9940d14a0 legacy=0x7ff99415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.984+0000 7ff999548640 1 -- 172.21.15.59:0/1246493332 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9940d0a30 con 0x7ff9940d10c0 2024-03-06T07:26:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.984+0000 7ff993fff640 1 -- 172.21.15.59:0/1246493332 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691467041 0 0) 0x7ff9940cfa50 con 0x7ff9940d14a0 2024-03-06T07:26:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.984+0000 7ff999548640 1 --2- 172.21.15.59:0/1246493332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9940d10c0 0x7ff9940d04f0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff984009d90 tx=0x7ff98402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc0cd1ed6b390c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.985+0000 7ff993fff640 1 -- 172.21.15.59:0/1246493332 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984039070 con 0x7ff9940d10c0 2024-03-06T07:26:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.985+0000 7ff993fff640 1 -- 172.21.15.59:0/1246493332 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98402cc50 con 0x7ff9940d10c0 2024-03-06T07:26:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.985+0000 7ff993fff640 1 -- 172.21.15.59:0/1246493332 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984034440 con 0x7ff9940d10c0 2024-03-06T07:26:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.985+0000 7ff99b7ac640 1 -- 172.21.15.59:0/1246493332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9940d10c0 msgr2=0x7ff9940d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.985+0000 7ff99b7ac640 1 --2- 172.21.15.59:0/1246493332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9940d10c0 0x7ff9940d04f0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff984009d90 tx=0x7ff98402bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.985+0000 7ff99b7ac640 1 -- 172.21.15.59:0/1246493332 shutdown_connections 2024-03-06T07:26:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.985+0000 7ff99b7ac640 1 --2- 172.21.15.59:0/1246493332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99415ecd0 0x7ff9940cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.985+0000 7ff99b7ac640 1 --2- 172.21.15.59:0/1246493332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9940d10c0 0x7ff9940d04f0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.985+0000 7ff99b7ac640 1 -- 172.21.15.59:0/1246493332 >> 172.21.15.59:0/1246493332 conn(0x7ff9940b3800 msgr2=0x7ff9940b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:18.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.985+0000 7ff99b7ac640 1 -- 172.21.15.59:0/1246493332 shutdown_connections 2024-03-06T07:26:18.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.985+0000 7ff99b7ac640 1 -- 172.21.15.59:0/1246493332 wait complete. 2024-03-06T07:26:18.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.986+0000 7ff99b7ac640 1 Processor -- start 2024-03-06T07:26:18.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.986+0000 7ff99b7ac640 1 -- start start 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff99b7ac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9940d10c0 0x7ff99414a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff99b7ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9940d14a0 0x7ff99414e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff99b7ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99415f760 0x7ff99415fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff99b7ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99415f440 con 0x7ff9940d14a0 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff99b7ac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99415f160 con 0x7ff9940d10c0 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff99b7ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9940cfcf0 con 0x7ff99415f760 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff998d47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9940d14a0 0x7ff99414e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff998d47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9940d14a0 0x7ff99414e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47396/0 (socket says 172.21.15.59:47396) 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff998d47640 1 -- 172.21.15.59:0/394637275 learned_addr learned my addr 172.21.15.59:0/394637275 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff999d49640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99415f760 0x7ff99415fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff999548640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9940d10c0 0x7ff99414a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff998d47640 1 -- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99415f760 msgr2=0x7ff99415fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff998d47640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99415f760 0x7ff99415fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff998d47640 1 -- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9940d10c0 msgr2=0x7ff99414a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff998d47640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9940d10c0 0x7ff99414a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.987+0000 7ff998d47640 1 -- 172.21.15.59:0/394637275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9940cc320 con 0x7ff9940d14a0 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.988+0000 7ff998d47640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9940d14a0 0x7ff99414e270 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ff98800c430 tx=0x7ff98800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.988+0000 7ff991ffb640 1 -- 172.21.15.59:0/394637275 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9880074e0 con 0x7ff9940d14a0 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.988+0000 7ff991ffb640 1 -- 172.21.15.59:0/394637275 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9880047e0 con 0x7ff9940d14a0 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.988+0000 7ff99b7ac640 1 -- 172.21.15.59:0/394637275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9940cc600 con 0x7ff9940d14a0 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.988+0000 7ff991ffb640 1 -- 172.21.15.59:0/394637275 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff988004bd0 con 0x7ff9940d14a0 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.988+0000 7ff99b7ac640 1 -- 172.21.15.59:0/394637275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9940ccbc0 con 0x7ff9940d14a0 2024-03-06T07:26:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.989+0000 7ff99b7ac640 1 -- 172.21.15.59:0/394637275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99409fe60 con 0x7ff9940d14a0 2024-03-06T07:26:18.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.989+0000 7ff991ffb640 1 -- 172.21.15.59:0/394637275 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff988011070 con 0x7ff9940d14a0 2024-03-06T07:26:18.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.990+0000 7ff991ffb640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff98006b4d0 0x7ff98006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:18.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.990+0000 7ff991ffb640 1 -- 172.21.15.59:0/394637275 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff988089ca0 con 0x7ff9940d14a0 2024-03-06T07:26:18.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.990+0000 7ff999548640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff98006b4d0 0x7ff98006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:18.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.990+0000 7ff999548640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff98006b4d0 0x7ff98006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff984036000 tx=0x7ff984030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:18.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:18.994+0000 7ff991ffb640 1 -- 172.21.15.59:0/394637275 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff988058560 con 0x7ff9940d14a0 2024-03-06T07:26:19.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.213+0000 7ff99b7ac640 1 -- 172.21.15.59:0/394637275 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff994068ff0 con 0x7ff9940d14a0 2024-03-06T07:26:19.816 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:19.819 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:19.844 INFO:tasks.ceph.mds.d.smithi164.stderr:2024-03-06T07:26:19.842+0000 7fbab3521640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:19.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.870+0000 7ff991ffb640 1 -- 172.21.15.59:0/394637275 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v49) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff98805c600 con 0x7ff9940d14a0 2024-03-06T07:26:19.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 -- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff98006b4d0 msgr2=0x7ff98006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff98006b4d0 0x7ff98006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff984036000 tx=0x7ff984030000 comp rx=0 tx=0).stop 2024-03-06T07:26:19.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 -- 172.21.15.59:0/394637275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9940d14a0 msgr2=0x7ff99414e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9940d14a0 0x7ff99414e270 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ff98800c430 tx=0x7ff98800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 -- 172.21.15.59:0/394637275 shutdown_connections 2024-03-06T07:26:19.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff98006b4d0 0x7ff98006d990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99415f760 0x7ff99415fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff9940d14a0 0x7ff99414e270 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 --2- 172.21.15.59:0/394637275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9940d10c0 0x7ff99414a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 -- 172.21.15.59:0/394637275 >> 172.21.15.59:0/394637275 conn(0x7ff9940b3800 msgr2=0x7ff99415d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 -- 172.21.15.59:0/394637275 shutdown_connections 2024-03-06T07:26:19.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:19.873+0000 7ff99b7ac640 1 -- 172.21.15.59:0/394637275 wait complete. 2024-03-06T07:26:19.889 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:26:19.889 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:20.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.035+0000 7f8fef0af640 1 Processor -- start 2024-03-06T07:26:20.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.035+0000 7f8fef0af640 1 -- start start 2024-03-06T07:26:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fef0af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe80d00c0 0x7f8fe80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fef0af640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8fe806cf20 0x7f8fe806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fef0af640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe80d09e0 con 0x7f8fe80cf850 2024-03-06T07:26:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fef0af640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe806ca90 con 0x7f8fe806cf20 2024-03-06T07:26:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fef0af640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe815e380 con 0x7f8fe80d00c0 2024-03-06T07:26:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fed64c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8fe806cf20 0x7f8fe806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fe7fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8fe80cf850 0x7f8fe80cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34370/0 (socket says 172.21.15.59:34370) 2024-03-06T07:26:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fe7fff640 1 -- 172.21.15.59:0/3252918129 learned_addr learned my addr 172.21.15.59:0/3252918129 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fece4b640 1 --2- 172.21.15.59:0/3252918129 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe80d00c0 0x7f8fe80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fed64c640 1 -- 172.21.15.59:0/3252918129 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe80d00c0 msgr2=0x7f8fe80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fed64c640 1 --2- 172.21.15.59:0/3252918129 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe80d00c0 0x7f8fe80d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fed64c640 1 -- 172.21.15.59:0/3252918129 >> v1:172.21.15.59:6789/0 conn(0x7f8fe80cf850 legacy=0x7f8fe80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fed64c640 1 -- 172.21.15.59:0/3252918129 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe80cdc10 con 0x7f8fe806cf20 2024-03-06T07:26:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.036+0000 7f8fe77fe640 1 -- 172.21.15.59:0/3252918129 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952662444 0 0) 0x7f8fe80d09e0 con 0x7f8fe80cf850 2024-03-06T07:26:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.037+0000 7f8fed64c640 1 --2- 172.21.15.59:0/3252918129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe806cf20 0x7f8fe806d3b0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0009d90 tx=0x7f8fe002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19636f32b8a59970 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.037+0000 7f8fe77fe640 1 -- 172.21.15.59:0/3252918129 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0039070 con 0x7f8fe806cf20 2024-03-06T07:26:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.037+0000 7f8fe77fe640 1 -- 172.21.15.59:0/3252918129 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe002cc50 con 0x7f8fe806cf20 2024-03-06T07:26:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.037+0000 7f8fe77fe640 1 -- 172.21.15.59:0/3252918129 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0034440 con 0x7f8fe806cf20 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.038+0000 7f8fef0af640 1 -- 172.21.15.59:0/3252918129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe806cf20 msgr2=0x7f8fe806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.038+0000 7f8fef0af640 1 --2- 172.21.15.59:0/3252918129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe806cf20 0x7f8fe806d3b0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0009d90 tx=0x7f8fe002bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.038+0000 7f8fef0af640 1 -- 172.21.15.59:0/3252918129 shutdown_connections 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.038+0000 7f8fef0af640 1 --2- 172.21.15.59:0/3252918129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe806cf20 0x7f8fe806d3b0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.038+0000 7f8fef0af640 1 --2- 172.21.15.59:0/3252918129 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe80d00c0 0x7f8fe80d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.038+0000 7f8fef0af640 1 -- 172.21.15.59:0/3252918129 >> 172.21.15.59:0/3252918129 conn(0x7f8fe80b35a0 msgr2=0x7f8fe80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.038+0000 7f8fef0af640 1 -- 172.21.15.59:0/3252918129 shutdown_connections 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.038+0000 7f8fef0af640 1 -- 172.21.15.59:0/3252918129 wait complete. 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.039+0000 7f8fef0af640 1 Processor -- start 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.039+0000 7f8fef0af640 1 -- start start 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fef0af640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe806cf20 0x7f8fe81f7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fef0af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe80cf850 0x7f8fe80c7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fef0af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe80d00c0 0x7f8fe815c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fef0af640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe806c720 con 0x7f8fe80cf850 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fef0af640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe806c440 con 0x7f8fe806cf20 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fef0af640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80b1030 con 0x7f8fe80d00c0 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fe7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe80cf850 0x7f8fe80c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fe7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe80cf850 0x7f8fe80c7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47424/0 (socket says 172.21.15.59:47424) 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fe7fff640 1 -- 172.21.15.59:0/1972869358 learned_addr learned my addr 172.21.15.59:0/1972869358 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fece4b640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe806cf20 0x7f8fe81f7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fed64c640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe80d00c0 0x7f8fe815c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fe7fff640 1 -- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe80d00c0 msgr2=0x7f8fe815c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fe7fff640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe80d00c0 0x7f8fe815c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fe7fff640 1 -- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe806cf20 msgr2=0x7f8fe81f7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fe7fff640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe806cf20 0x7f8fe81f7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fe7fff640 1 -- 172.21.15.59:0/1972869358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe81f7d80 con 0x7f8fe80cf850 2024-03-06T07:26:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.040+0000 7f8fe7fff640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe80cf850 0x7f8fe80c7830 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400c4d0 tx=0x7f8fd400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.041+0000 7f8fe57fa640 1 -- 172.21.15.59:0/1972869358 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4018070 con 0x7f8fe80cf850 2024-03-06T07:26:20.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.041+0000 7f8fe57fa640 1 -- 172.21.15.59:0/1972869358 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd4007500 con 0x7f8fe80cf850 2024-03-06T07:26:20.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.041+0000 7f8fe57fa640 1 -- 172.21.15.59:0/1972869358 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4004a20 con 0x7f8fe80cf850 2024-03-06T07:26:20.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.041+0000 7f8fef0af640 1 -- 172.21.15.59:0/1972869358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe81f8060 con 0x7f8fe80cf850 2024-03-06T07:26:20.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.041+0000 7f8fef0af640 1 -- 172.21.15.59:0/1972869358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe80b4690 con 0x7f8fe80cf850 2024-03-06T07:26:20.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.042+0000 7f8fef0af640 1 -- 172.21.15.59:0/1972869358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe804f330 con 0x7f8fe80cf850 2024-03-06T07:26:20.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.042+0000 7f8fe57fa640 1 -- 172.21.15.59:0/1972869358 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fd401b070 con 0x7f8fe80cf850 2024-03-06T07:26:20.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.043+0000 7f8fe57fa640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fcc06b4d0 0x7f8fcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.043+0000 7f8fe57fa640 1 -- 172.21.15.59:0/1972869358 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fd4089500 con 0x7f8fe80cf850 2024-03-06T07:26:20.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.043+0000 7f8fece4b640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fcc06b4d0 0x7f8fcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.043+0000 7f8fece4b640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fcc06b4d0 0x7f8fcc06d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc009b40 tx=0x7f8fdc009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.047+0000 7f8fe57fa640 1 -- 172.21.15.59:0/1972869358 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fd4057c90 con 0x7f8fe80cf850 2024-03-06T07:26:20.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.272+0000 7f8fef0af640 1 -- 172.21.15.59:0/1972869358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fe80704e0 con 0x7f8fe80cf850 2024-03-06T07:26:20.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.273+0000 7f8fe57fa640 1 -- 172.21.15.59:0/1972869358 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f8fd405bd30 con 0x7f8fe80cf850 2024-03-06T07:26:20.274 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:20.275 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":49,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6173,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/386662975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":386662975},{"type":"v1","addr":"172.21.15.164:6836","nonce":386662975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2799249020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2799249020},{"type":"v1","addr":"172.21.15.59:6834","nonce":2799249020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7410,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1784120157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1784120157},{"type":"v1","addr":"172.21.15.59:6835","nonce":1784120157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:19.214384+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7422":{"gid":7422,"name":"d","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/4292771408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4292771408},{"type":"v1","addr":"172.21.15.164:6837","nonce":4292771408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7422,"qdb_cluster":[7422]},"id":7}]} 2024-03-06T07:26:20.275 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 49 2024-03-06T07:26:20.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.274+0000 7f8fef0af640 1 -- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fcc06b4d0 msgr2=0x7f8fcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.274+0000 7f8fef0af640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fcc06b4d0 0x7f8fcc06d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc009b40 tx=0x7f8fdc009190 comp rx=0 tx=0).stop 2024-03-06T07:26:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.274+0000 7f8fef0af640 1 -- 172.21.15.59:0/1972869358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe80cf850 msgr2=0x7f8fe80c7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.274+0000 7f8fef0af640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe80cf850 0x7f8fe80c7830 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400c4d0 tx=0x7f8fd400c990 comp rx=0 tx=0).stop 2024-03-06T07:26:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.275+0000 7f8fef0af640 1 -- 172.21.15.59:0/1972869358 shutdown_connections 2024-03-06T07:26:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.275+0000 7f8fef0af640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fcc06b4d0 0x7f8fcc06d990 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.275+0000 7f8fef0af640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe80d00c0 0x7f8fe815c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.275+0000 7f8fef0af640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe80cf850 0x7f8fe80c7830 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.275+0000 7f8fef0af640 1 --2- 172.21.15.59:0/1972869358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe806cf20 0x7f8fe81f7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.275+0000 7f8fef0af640 1 -- 172.21.15.59:0/1972869358 >> 172.21.15.59:0/1972869358 conn(0x7f8fe80b35a0 msgr2=0x7f8fe80b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.275+0000 7f8fef0af640 1 -- 172.21.15.59:0/1972869358 shutdown_connections 2024-03-06T07:26:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.275+0000 7f8fef0af640 1 -- 172.21.15.59:0/1972869358 wait complete. 2024-03-06T07:26:20.287 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:20.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.387+0000 7fc511264640 1 Processor -- start 2024-03-06T07:26:20.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.387+0000 7fc511264640 1 -- start start 2024-03-06T07:26:20.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.388+0000 7fc511264640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc50c0d0a80 0x7fc50c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.388+0000 7fc511264640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50c0d25d0 con 0x7fc50c0d4760 2024-03-06T07:26:20.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.388+0000 7fc511264640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50c0d2870 con 0x7fc50c0b4170 2024-03-06T07:26:20.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.388+0000 7fc511264640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc50c0d4bf0 con 0x7fc50c0d0a80 2024-03-06T07:26:20.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.388+0000 7fc50a7fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc50c0d4760 0x7fc50c0cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34376/0 (socket says 172.21.15.59:34376) 2024-03-06T07:26:20.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.388+0000 7fc50a7fc640 1 -- 172.21.15.59:0/3401412971 learned_addr learned my addr 172.21.15.59:0/3401412971 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:20.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.388+0000 7fc50b7fe640 1 --2- 172.21.15.59:0/3401412971 >> v2:172.21.15.164:3300/0 conn(0x7fc50c0d0a80 0x7fc50c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.389+0000 7fc509ffb640 1 -- 172.21.15.59:0/3401412971 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961508643 0 0) 0x7fc50c0d25d0 con 0x7fc50c0d4760 2024-03-06T07:26:20.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.389+0000 7fc509ffb640 1 -- 172.21.15.59:0/3401412971 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4f8003520 con 0x7fc50c0d4760 2024-03-06T07:26:20.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.389+0000 7fc50b7fe640 1 -- 172.21.15.59:0/3401412971 >> v1:172.21.15.164:6789/0 conn(0x7fc50c0b4170 legacy=0x7fc50c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.389+0000 7fc50b7fe640 1 -- 172.21.15.59:0/3401412971 >> v1:172.21.15.59:6789/0 conn(0x7fc50c0d4760 legacy=0x7fc50c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.389+0000 7fc50b7fe640 1 -- 172.21.15.59:0/3401412971 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c0d2b50 con 0x7fc50c0d0a80 2024-03-06T07:26:20.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.389+0000 7fc50b7fe640 1 --2- 172.21.15.59:0/3401412971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc50c0d0a80 0x7fc50c0b6a90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009d90 tx=0x7fc4fc02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3beea286705d2e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.390+0000 7fc509ffb640 1 -- 172.21.15.59:0/3401412971 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc039070 con 0x7fc50c0d0a80 2024-03-06T07:26:20.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.390+0000 7fc509ffb640 1 -- 172.21.15.59:0/3401412971 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc02cd00 con 0x7fc50c0d0a80 2024-03-06T07:26:20.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.390+0000 7fc509ffb640 1 -- 172.21.15.59:0/3401412971 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc034640 con 0x7fc50c0d0a80 2024-03-06T07:26:20.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.390+0000 7fc511264640 1 -- 172.21.15.59:0/3401412971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc50c0d0a80 msgr2=0x7fc50c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.390+0000 7fc511264640 1 --2- 172.21.15.59:0/3401412971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc50c0d0a80 0x7fc50c0b6a90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009d90 tx=0x7fc4fc02bf50 comp rx=0 tx=0).stop 2024-03-06T07:26:20.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.390+0000 7fc511264640 1 -- 172.21.15.59:0/3401412971 shutdown_connections 2024-03-06T07:26:20.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.390+0000 7fc511264640 1 --2- 172.21.15.59:0/3401412971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc50c0d0a80 0x7fc50c0b6a90 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.390+0000 7fc511264640 1 -- 172.21.15.59:0/3401412971 >> 172.21.15.59:0/3401412971 conn(0x7fc50c0d3c40 msgr2=0x7fc50c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.390+0000 7fc511264640 1 -- 172.21.15.59:0/3401412971 shutdown_connections 2024-03-06T07:26:20.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.390+0000 7fc511264640 1 -- 172.21.15.59:0/3401412971 wait complete. 2024-03-06T07:26:20.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.391+0000 7fc511264640 1 Processor -- start 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc511264640 1 -- start start 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc511264640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc50c0b4170 0x7fc50c15e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc511264640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc50c0d0a80 0x7fc50c06c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc511264640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc50c15dee0 0x7fc50c06cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc511264640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c0b4dd0 con 0x7fc50c0b4170 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc511264640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c0ce9e0 con 0x7fc50c0d0a80 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc511264640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c0b4f50 con 0x7fc50c15dee0 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc50affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc50c0b4170 0x7fc50c15e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc50affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc50c0b4170 0x7fc50c15e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47434/0 (socket says 172.21.15.59:47434) 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc50affd640 1 -- 172.21.15.59:0/675956976 learned_addr learned my addr 172.21.15.59:0/675956976 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc50a7fc640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc50c0d0a80 0x7fc50c06c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.392+0000 7fc50b7fe640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc50c15dee0 0x7fc50c06cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc50affd640 1 -- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc50c15dee0 msgr2=0x7fc50c06cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc50affd640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc50c15dee0 0x7fc50c06cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc50affd640 1 -- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc50c0d0a80 msgr2=0x7fc50c06c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc50affd640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc50c0d0a80 0x7fc50c06c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc50affd640 1 -- 172.21.15.59:0/675956976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c0c7840 con 0x7fc50c0b4170 2024-03-06T07:26:20.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc50a7fc640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc50c0d0a80 0x7fc50c06c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:20.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc50affd640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc50c0b4170 0x7fc50c15e5f0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc50400c400 tx=0x7fc50400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc5037fe640 1 -- 172.21.15.59:0/675956976 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5040052d0 con 0x7fc50c0b4170 2024-03-06T07:26:20.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc5037fe640 1 -- 172.21.15.59:0/675956976 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc504005490 con 0x7fc50c0b4170 2024-03-06T07:26:20.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc511264640 1 -- 172.21.15.59:0/675956976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc50c0c7b20 con 0x7fc50c0b4170 2024-03-06T07:26:20.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc5037fe640 1 -- 172.21.15.59:0/675956976 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504007260 con 0x7fc50c0b4170 2024-03-06T07:26:20.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.393+0000 7fc511264640 1 -- 172.21.15.59:0/675956976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc50c0c80e0 con 0x7fc50c0b4170 2024-03-06T07:26:20.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.394+0000 7fc511264640 1 -- 172.21.15.59:0/675956976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc50c04f330 con 0x7fc50c0b4170 2024-03-06T07:26:20.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.394+0000 7fc5037fe640 1 -- 172.21.15.59:0/675956976 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc504007400 con 0x7fc50c0b4170 2024-03-06T07:26:20.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.395+0000 7fc5037fe640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4ec06b4d0 0x7fc4ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.395+0000 7fc5037fe640 1 -- 172.21.15.59:0/675956976 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc504089650 con 0x7fc50c0b4170 2024-03-06T07:26:20.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.395+0000 7fc50a7fc640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4ec06b4d0 0x7fc4ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.395+0000 7fc50a7fc640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4ec06b4d0 0x7fc4ec06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0a1b40 tx=0x7fc4f4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.401 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.399+0000 7fc5037fe640 1 -- 172.21.15.59:0/675956976 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc504057e90 con 0x7fc50c0b4170 2024-03-06T07:26:20.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.580+0000 7fc511264640 1 -- 172.21.15.59:0/675956976 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc50c15e3d0 con 0x7fc50c0b4170 2024-03-06T07:26:20.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.581+0000 7fc5037fe640 1 -- 172.21.15.59:0/675956976 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16596 (secure 0 0 0) 0x7fc50405bf30 con 0x7fc50c0b4170 2024-03-06T07:26:20.583 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:20.584 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":92,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:26:18.851026+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.489280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:16.183614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:20.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 -- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4ec06b4d0 msgr2=0x7fc4ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4ec06b4d0 0x7fc4ec06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0a1b40 tx=0x7fc4f4009040 comp rx=0 tx=0).stop 2024-03-06T07:26:20.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 -- 172.21.15.59:0/675956976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc50c0b4170 msgr2=0x7fc50c15e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc50c0b4170 0x7fc50c15e5f0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc50400c400 tx=0x7fc50400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 -- 172.21.15.59:0/675956976 shutdown_connections 2024-03-06T07:26:20.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4ec06b4d0 0x7fc4ec06d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc50c15dee0 0x7fc50c06cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc50c0d0a80 0x7fc50c06c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 --2- 172.21.15.59:0/675956976 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc50c0b4170 0x7fc50c15e5f0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 -- 172.21.15.59:0/675956976 >> 172.21.15.59:0/675956976 conn(0x7fc50c0d3c40 msgr2=0x7fc50c0b7280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 -- 172.21.15.59:0/675956976 shutdown_connections 2024-03-06T07:26:20.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.583+0000 7fc511264640 1 -- 172.21.15.59:0/675956976 wait complete. 2024-03-06T07:26:20.596 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:26:20.597 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.696+0000 7fa5740c9640 1 Processor -- start 2024-03-06T07:26:20.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.696+0000 7fa5740c9640 1 -- start start 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa5740c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa56c13dc10 0x7fa56c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa5740c9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c15f580 con 0x7fa56c15ef60 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa5740c9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c15b450 con 0x7fa56c15bf30 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa5740c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0cf3f0 con 0x7fa56c13dc10 2024-03-06T07:26:20.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa572666640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa56c15ef60 0x7fa56c0708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34390/0 (socket says 172.21.15.59:34390) 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa572666640 1 -- 172.21.15.59:0/1516030686 learned_addr learned my addr 172.21.15.59:0/1516030686 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa571e65640 1 --2- 172.21.15.59:0/1516030686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa56c13dc10 0x7fa56c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa570e63640 1 -- 172.21.15.59:0/1516030686 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231288675 0 0) 0x7fa56c15f580 con 0x7fa56c15ef60 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa570e63640 1 -- 172.21.15.59:0/1516030686 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa558003520 con 0x7fa56c15ef60 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa571e65640 1 -- 172.21.15.59:0/1516030686 >> v1:172.21.15.164:6789/0 conn(0x7fa56c15bf30 legacy=0x7fa56c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa571e65640 1 -- 172.21.15.59:0/1516030686 >> v1:172.21.15.59:6789/0 conn(0x7fa56c15ef60 legacy=0x7fa56c0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa570e63640 1 -- 172.21.15.59:0/1516030686 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140392187 0 0) 0x7fa56c15b450 con 0x7fa56c15bf30 2024-03-06T07:26:20.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.697+0000 7fa571e65640 1 -- 172.21.15.59:0/1516030686 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c15b730 con 0x7fa56c13dc10 2024-03-06T07:26:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.698+0000 7fa571e65640 1 --2- 172.21.15.59:0/1516030686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa56c13dc10 0x7fa56c0b2950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa5680097d0 tx=0x7fa56802dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38a638c1ef186b7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.698+0000 7fa570e63640 1 -- 172.21.15.59:0/1516030686 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56802f870 con 0x7fa56c13dc10 2024-03-06T07:26:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.698+0000 7fa570e63640 1 -- 172.21.15.59:0/1516030686 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56802fa10 con 0x7fa56c13dc10 2024-03-06T07:26:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.698+0000 7fa570e63640 1 -- 172.21.15.59:0/1516030686 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56802fdc0 con 0x7fa56c13dc10 2024-03-06T07:26:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.699+0000 7fa5740c9640 1 -- 172.21.15.59:0/1516030686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa56c13dc10 msgr2=0x7fa56c0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.699+0000 7fa5740c9640 1 --2- 172.21.15.59:0/1516030686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa56c13dc10 0x7fa56c0b2950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa5680097d0 tx=0x7fa56802dab0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.699+0000 7fa5740c9640 1 -- 172.21.15.59:0/1516030686 shutdown_connections 2024-03-06T07:26:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.699+0000 7fa5740c9640 1 --2- 172.21.15.59:0/1516030686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa56c13dc10 0x7fa56c0b2950 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.699+0000 7fa5740c9640 1 -- 172.21.15.59:0/1516030686 >> 172.21.15.59:0/1516030686 conn(0x7fa56c0cf8a0 msgr2=0x7fa56c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.699+0000 7fa5740c9640 1 -- 172.21.15.59:0/1516030686 shutdown_connections 2024-03-06T07:26:20.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.699+0000 7fa5740c9640 1 -- 172.21.15.59:0/1516030686 wait complete. 2024-03-06T07:26:20.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.700+0000 7fa5740c9640 1 Processor -- start 2024-03-06T07:26:20.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.700+0000 7fa5740c9640 1 -- start start 2024-03-06T07:26:20.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa5740c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa56c13dc10 0x7fa56c155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa5740c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa56c15bf30 0x7fa56c1ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa5740c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa56c15ef60 0x7fa56c0cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa5740c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0b2e90 con 0x7fa56c13dc10 2024-03-06T07:26:20.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa5740c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c15b210 con 0x7fa56c15bf30 2024-03-06T07:26:20.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa5740c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0b3010 con 0x7fa56c15ef60 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa571e65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa56c13dc10 0x7fa56c155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa571e65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa56c13dc10 0x7fa56c155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47440/0 (socket says 172.21.15.59:47440) 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa571e65640 1 -- 172.21.15.59:0/1607304714 learned_addr learned my addr 172.21.15.59:0/1607304714 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa572666640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa56c15ef60 0x7fa56c0cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa571664640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa56c15bf30 0x7fa56c1ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa571e65640 1 -- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa56c15ef60 msgr2=0x7fa56c0cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa571e65640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa56c15ef60 0x7fa56c0cfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa571e65640 1 -- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa56c15bf30 msgr2=0x7fa56c1ef7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa571e65640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa56c15bf30 0x7fa56c1ef7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa571e65640 1 -- 172.21.15.59:0/1607304714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c13fe90 con 0x7fa56c13dc10 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa571664640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa56c15bf30 0x7fa56c1ef7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa571e65640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa56c13dc10 0x7fa56c155320 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fa568000c00 tx=0x7fa5680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.701+0000 7fa5627fc640 1 -- 172.21.15.59:0/1607304714 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568030c00 con 0x7fa56c13dc10 2024-03-06T07:26:20.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.702+0000 7fa5627fc640 1 -- 172.21.15.59:0/1607304714 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56803f040 con 0x7fa56c13dc10 2024-03-06T07:26:20.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.702+0000 7fa5627fc640 1 -- 172.21.15.59:0/1607304714 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568045440 con 0x7fa56c13dc10 2024-03-06T07:26:20.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.702+0000 7fa5740c9640 1 -- 172.21.15.59:0/1607304714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c140170 con 0x7fa56c13dc10 2024-03-06T07:26:20.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.702+0000 7fa5740c9640 1 -- 172.21.15.59:0/1607304714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c140730 con 0x7fa56c13dc10 2024-03-06T07:26:20.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.702+0000 7fa5740c9640 1 -- 172.21.15.59:0/1607304714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c04f330 con 0x7fa56c13dc10 2024-03-06T07:26:20.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.703+0000 7fa5627fc640 1 -- 172.21.15.59:0/1607304714 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5680455e0 con 0x7fa56c13dc10 2024-03-06T07:26:20.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.703+0000 7fa5627fc640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa55006b4d0 0x7fa55006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:20.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.703+0000 7fa5627fc640 1 -- 172.21.15.59:0/1607304714 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5680aebc0 con 0x7fa56c13dc10 2024-03-06T07:26:20.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.703+0000 7fa571664640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa55006b4d0 0x7fa55006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:20.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.704+0000 7fa571664640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa55006b4d0 0x7fa55006d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0afcb0 tx=0x7fa55c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:20.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.708+0000 7fa5627fc640 1 -- 172.21.15.59:0/1607304714 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa568035030 con 0x7fa56c13dc10 2024-03-06T07:26:20.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.887+0000 7fa5740c9640 1 -- 172.21.15.59:0/1607304714 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa56c0af660 con 0x7fa55006b4d0 2024-03-06T07:26:20.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.893+0000 7fa5627fc640 1 -- 172.21.15.59:0/1607304714 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240122 (secure 0 0 0) 0x7fa56c0af660 con 0x7fa55006b4d0 2024-03-06T07:26:20.897 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:20.898 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:26:20.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.895+0000 7fa5740c9640 1 -- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa55006b4d0 msgr2=0x7fa55006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.896+0000 7fa5740c9640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa55006b4d0 0x7fa55006d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0afcb0 tx=0x7fa55c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:20.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.896+0000 7fa5740c9640 1 -- 172.21.15.59:0/1607304714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa56c13dc10 msgr2=0x7fa56c155320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.896+0000 7fa5740c9640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa56c13dc10 0x7fa56c155320 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fa568000c00 tx=0x7fa5680046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.896+0000 7fa5740c9640 1 -- 172.21.15.59:0/1607304714 shutdown_connections 2024-03-06T07:26:20.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.896+0000 7fa5740c9640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa55006b4d0 0x7fa55006d990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.896+0000 7fa5740c9640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa56c15ef60 0x7fa56c0cfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.896+0000 7fa5740c9640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa56c15bf30 0x7fa56c1ef7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.896+0000 7fa5740c9640 1 --2- 172.21.15.59:0/1607304714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa56c13dc10 0x7fa56c155320 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.896+0000 7fa5740c9640 1 -- 172.21.15.59:0/1607304714 >> 172.21.15.59:0/1607304714 conn(0x7fa56c0cf8a0 msgr2=0x7fa56c13bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.896+0000 7fa5740c9640 1 -- 172.21.15.59:0/1607304714 shutdown_connections 2024-03-06T07:26:20.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:20.896+0000 7fa5740c9640 1 -- 172.21.15.59:0/1607304714 wait complete. 2024-03-06T07:26:20.908 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-03-06T07:26:19.358912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259208,"kb_used_data":9720,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715512,"statfs":{"total":773094113280,"available":772828684288,"internally_reserved":0,"allocated":9953280,"data_stored":4886972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000710"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850403+0000","last_change":"2024-03-06T07:26:15.850403+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850403+0000","last_undegraded":"2024-03-06T07:26:15.850403+0000","last_fullsized":"2024-03-06T07:26:15.850403+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849894+0000","last_change":"2024-03-06T07:26:15.849894+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849894+0000","last_undegraded":"2024-03-06T07:26:15.849894+0000","last_fullsized":"2024-03-06T07:26:15.849894+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850385+0000","last_change":"2024-03-06T07:26:15.850385+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850385+0000","last_undegraded":"2024-03-06T07:26:15.850385+0000","last_fullsized":"2024-03-06T07:26:15.850385+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850344+0000","last_change":"2024-03-06T07:26:15.850344+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850344+0000","last_undegraded":"2024-03-06T07:26:15.850344+0000","last_fullsized":"2024-03-06T07:26:15.850344+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.848404+0000","last_change":"2024-03-06T07:26:15.848404+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.848404+0000","last_undegraded":"2024-03-06T07:26:15.848404+0000","last_fullsized":"2024-03-06T07:26:15.848404+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849998+0000","last_change":"2024-03-06T07:26:15.849998+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849998+0000","last_undegraded":"2024-03-06T07:26:15.849998+0000","last_fullsized":"2024-03-06T07:26:15.849998+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850220+0000","last_change":"2024-03-06T07:26:15.850220+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850220+0000","last_undegraded":"2024-03-06T07:26:15.850220+0000","last_fullsized":"2024-03-06T07:26:15.850220+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.848320+0000","last_change":"2024-03-06T07:26:15.848320+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.848320+0000","last_undegraded":"2024-03-06T07:26:15.848320+0000","last_fullsized":"2024-03-06T07:26:15.848320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847839+0000","last_change":"2024-03-06T07:26:15.847839+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847839+0000","last_undegraded":"2024-03-06T07:26:15.847839+0000","last_fullsized":"2024-03-06T07:26:15.847839+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.845422+0000","last_change":"2024-03-06T07:26:15.845422+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.845422+0000","last_undegraded":"2024-03-06T07:26:15.845422+0000","last_fullsized":"2024-03-06T07:26:15.845422+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849881+0000","last_change":"2024-03-06T07:26:15.849881+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849881+0000","last_undegraded":"2024-03-06T07:26:15.849881+0000","last_fullsized":"2024-03-06T07:26:15.849881+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847799+0000","last_change":"2024-03-06T07:26:15.847799+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847799+0000","last_undegraded":"2024-03-06T07:26:15.847799+0000","last_fullsized":"2024-03-06T07:26:15.847799+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.848548+0000","last_change":"2024-03-06T07:26:15.848548+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.848548+0000","last_undegraded":"2024-03-06T07:26:15.848548+0000","last_fullsized":"2024-03-06T07:26:15.848548+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.848106+0000","last_change":"2024-03-06T07:26:15.848106+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.848106+0000","last_undegraded":"2024-03-06T07:26:15.848106+0000","last_fullsized":"2024-03-06T07:26:15.848106+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847798+0000","last_change":"2024-03-06T07:26:15.847798+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847798+0000","last_undegraded":"2024-03-06T07:26:15.847798+0000","last_fullsized":"2024-03-06T07:26:15.847798+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847824+0000","last_change":"2024-03-06T07:26:15.847824+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847824+0000","last_undegraded":"2024-03-06T07:26:15.847824+0000","last_fullsized":"2024-03-06T07:26:15.847824+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847880+0000","last_change":"2024-03-06T07:26:15.847880+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847880+0000","last_undegraded":"2024-03-06T07:26:15.847880+0000","last_fullsized":"2024-03-06T07:26:15.847880+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850317+0000","last_change":"2024-03-06T07:26:15.850317+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850317+0000","last_undegraded":"2024-03-06T07:26:15.850317+0000","last_fullsized":"2024-03-06T07:26:15.850317+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849910+0000","last_change":"2024-03-06T07:26:15.849910+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849910+0000","last_undegraded":"2024-03-06T07:26:15.849910+0000","last_fullsized":"2024-03-06T07:26:15.849910+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849543+0000","last_change":"2024-03-06T07:26:15.849543+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849543+0000","last_undegraded":"2024-03-06T07:26:15.849543+0000","last_fullsized":"2024-03-06T07:26:15.849543+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.845909+0000","last_change":"2024-03-06T07:26:15.845909+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.845909+0000","last_undegraded":"2024-03-06T07:26:15.845909+0000","last_fullsized":"2024-03-06T07:26:15.845909+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.845904+0000","last_change":"2024-03-06T07:26:15.845904+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.845904+0000","last_undegraded":"2024-03-06T07:26:15.845904+0000","last_fullsized":"2024-03-06T07:26:15.845904+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849657+0000","last_change":"2024-03-06T07:26:15.849657+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849657+0000","last_undegraded":"2024-03-06T07:26:15.849657+0000","last_fullsized":"2024-03-06T07:26:15.849657+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.845872+0000","last_change":"2024-03-06T07:26:15.845872+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.845872+0000","last_undegraded":"2024-03-06T07:26:15.845872+0000","last_fullsized":"2024-03-06T07:26:15.845872+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849029+0000","last_change":"2024-03-06T07:26:15.849029+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849029+0000","last_undegraded":"2024-03-06T07:26:15.849029+0000","last_fullsized":"2024-03-06T07:26:15.849029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.848399+0000","last_change":"2024-03-06T07:26:15.848399+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.848399+0000","last_undegraded":"2024-03-06T07:26:15.848399+0000","last_fullsized":"2024-03-06T07:26:15.848399+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850394+0000","last_change":"2024-03-06T07:26:15.850394+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850394+0000","last_undegraded":"2024-03-06T07:26:15.850394+0000","last_fullsized":"2024-03-06T07:26:15.850394+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847905+0000","last_change":"2024-03-06T07:26:15.847905+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847905+0000","last_undegraded":"2024-03-06T07:26:15.847905+0000","last_fullsized":"2024-03-06T07:26:15.847905+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850379+0000","last_change":"2024-03-06T07:26:15.850379+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850379+0000","last_undegraded":"2024-03-06T07:26:15.850379+0000","last_fullsized":"2024-03-06T07:26:15.850379+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.845422+0000","last_change":"2024-03-06T07:26:15.845422+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.845422+0000","last_undegraded":"2024-03-06T07:26:15.845422+0000","last_fullsized":"2024-03-06T07:26:15.845422+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847954+0000","last_change":"2024-03-06T07:26:15.847954+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847954+0000","last_undegraded":"2024-03-06T07:26:15.847954+0000","last_fullsized":"2024-03-06T07:26:15.847954+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849909+0000","last_change":"2024-03-06T07:26:15.849909+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849909+0000","last_undegraded":"2024-03-06T07:26:15.849909+0000","last_fullsized":"2024-03-06T07:26:15.849909+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":295,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-06T07:26:15.850906+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:26:15.850906+0000","last_peered":"2024-03-06T07:26:15.850906+0000","last_clean":"2024-03-06T07:26:15.850906+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:26:15.850906+0000","last_undegraded":"2024-03-06T07:26:15.850906+0000","last_fullsized":"2024-03-06T07:26:15.850906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32824,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339016,"statfs":{"total":96636764160,"available":96603152384,"internally_reserved":0,"allocated":1683456,"data_stored":1050739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1101824,"data_stored":466188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1101824,"data_stored":466188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":1089536,"data_stored":460371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":460371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":460371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1101824,"data_stored":466188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1695744,"data_stored":1056556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:20.913 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:21.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.048+0000 7f4d6b41c640 1 Processor -- start 2024-03-06T07:26:21.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.049+0000 7f4d6b41c640 1 -- start start 2024-03-06T07:26:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.049+0000 7f4d6b41c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4d6413dc10 0x7f4d640b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.049+0000 7f4d6b41c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4d6415ef60 0x7f4d640b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.049+0000 7f4d6b41c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6415f580 con 0x7f4d6415bf30 2024-03-06T07:26:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.049+0000 7f4d6b41c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d640cf3f0 con 0x7f4d6413dc10 2024-03-06T07:26:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.049+0000 7f4d6b41c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d6415b210 con 0x7f4d6415ef60 2024-03-06T07:26:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.050+0000 7f4d691b8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4d6413dc10 0x7f4d640b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.050+0000 7f4d691b8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4d6413dc10 0x7f4d640b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47456/0 (socket says 172.21.15.59:47456) 2024-03-06T07:26:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.050+0000 7f4d691b8640 1 -- 172.21.15.59:0/2830674702 learned_addr learned my addr 172.21.15.59:0/2830674702 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:21.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.050+0000 7f4d699b9640 1 --2- 172.21.15.59:0/2830674702 >> v2:172.21.15.164:3300/0 conn(0x7f4d6415ef60 0x7f4d640b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.050+0000 7f4d691b8640 1 -- 172.21.15.59:0/2830674702 >> v2:172.21.15.164:3300/0 conn(0x7f4d6415ef60 msgr2=0x7f4d640b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.050+0000 7f4d691b8640 1 --2- 172.21.15.59:0/2830674702 >> v2:172.21.15.164:3300/0 conn(0x7f4d6415ef60 0x7f4d640b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.050+0000 7f4d691b8640 1 -- 172.21.15.59:0/2830674702 >> v1:172.21.15.164:6789/0 conn(0x7f4d6415bf30 legacy=0x7f4d6415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.050+0000 7f4d691b8640 1 -- 172.21.15.59:0/2830674702 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6415b450 con 0x7f4d6413dc10 2024-03-06T07:26:21.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d691b8640 1 --2- 172.21.15.59:0/2830674702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6413dc10 0x7f4d640b2950 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f4d64068700 tx=0x7f4d5c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2eae118491813d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d63fff640 1 -- 172.21.15.59:0/2830674702 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c03b070 con 0x7f4d6413dc10 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d63fff640 1 -- 172.21.15.59:0/2830674702 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5c036440 con 0x7f4d6413dc10 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d63fff640 1 -- 172.21.15.59:0/2830674702 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c036810 con 0x7f4d6413dc10 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d6b41c640 1 -- 172.21.15.59:0/2830674702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6413dc10 msgr2=0x7f4d640b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d6b41c640 1 --2- 172.21.15.59:0/2830674702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6413dc10 0x7f4d640b2950 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f4d64068700 tx=0x7f4d5c02e0e0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d6b41c640 1 -- 172.21.15.59:0/2830674702 shutdown_connections 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d6b41c640 1 --2- 172.21.15.59:0/2830674702 >> v2:172.21.15.164:3300/0 conn(0x7f4d6415ef60 0x7f4d640b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d6b41c640 1 --2- 172.21.15.59:0/2830674702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6413dc10 0x7f4d640b2950 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d6b41c640 1 -- 172.21.15.59:0/2830674702 >> 172.21.15.59:0/2830674702 conn(0x7f4d640cf8a0 msgr2=0x7f4d640cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d6b41c640 1 -- 172.21.15.59:0/2830674702 shutdown_connections 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.051+0000 7f4d6b41c640 1 -- 172.21.15.59:0/2830674702 wait complete. 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.052+0000 7f4d6b41c640 1 Processor -- start 2024-03-06T07:26:21.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.053+0000 7f4d6b41c640 1 -- start start 2024-03-06T07:26:21.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.053+0000 7f4d6b41c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6413dc10 0x7f4d641f67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.053+0000 7f4d6b41c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d6415bf30 0x7f4d640cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.053+0000 7f4d6b41c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6415ef60 0x7f4d640ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.053+0000 7f4d6b41c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d640cd7a0 con 0x7f4d6413dc10 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.053+0000 7f4d6b41c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6413dff0 con 0x7f4d6415ef60 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.053+0000 7f4d6b41c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d640cd920 con 0x7f4d6415bf30 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.053+0000 7f4d691b8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6413dc10 0x7f4d641f67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.053+0000 7f4d691b8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6413dc10 0x7f4d641f67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47468/0 (socket says 172.21.15.59:47468) 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.053+0000 7f4d691b8640 1 -- 172.21.15.59:0/3555299114 learned_addr learned my addr 172.21.15.59:0/3555299114 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.053+0000 7f4d699b9640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6415ef60 0x7f4d640ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d689b7640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d6415bf30 0x7f4d640cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d691b8640 1 -- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d6415bf30 msgr2=0x7f4d640cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d691b8640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d6415bf30 0x7f4d640cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d691b8640 1 -- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6415ef60 msgr2=0x7f4d640ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d691b8640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6415ef60 0x7f4d640ce1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d691b8640 1 -- 172.21.15.59:0/3555299114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d641f7070 con 0x7f4d6413dc10 2024-03-06T07:26:21.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d699b9640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6415ef60 0x7f4d640ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:21.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d691b8640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6413dc10 0x7f4d641f67f0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c02e120 tx=0x7f4d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d61ffb640 1 -- 172.21.15.59:0/3555299114 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c03b070 con 0x7f4d6413dc10 2024-03-06T07:26:21.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d61ffb640 1 -- 172.21.15.59:0/3555299114 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5c03d050 con 0x7f4d6413dc10 2024-03-06T07:26:21.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d6b41c640 1 -- 172.21.15.59:0/3555299114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d641f7350 con 0x7f4d6413dc10 2024-03-06T07:26:21.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d61ffb640 1 -- 172.21.15.59:0/3555299114 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c034a90 con 0x7f4d6413dc10 2024-03-06T07:26:21.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.054+0000 7f4d6b41c640 1 -- 172.21.15.59:0/3555299114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d641f7910 con 0x7f4d6413dc10 2024-03-06T07:26:21.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.055+0000 7f4d6b41c640 1 -- 172.21.15.59:0/3555299114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6404f330 con 0x7f4d6413dc10 2024-03-06T07:26:21.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.056+0000 7f4d61ffb640 1 -- 172.21.15.59:0/3555299114 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d5c048060 con 0x7f4d6413dc10 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.056+0000 7f4d61ffb640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b4d0 0x7f4d5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.056+0000 7f4d61ffb640 1 -- 172.21.15.59:0/3555299114 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d5c0abd30 con 0x7f4d6413dc10 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.056+0000 7f4d689b7640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b4d0 0x7f4d5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.057+0000 7f4d689b7640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b4d0 0x7f4d5006d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4d6415a7e0 tx=0x7f4d58009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.060+0000 7f4d61ffb640 1 -- 172.21.15.59:0/3555299114 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d5c07ac40 con 0x7f4d6413dc10 2024-03-06T07:26:21.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.239+0000 7f4d6b41c640 1 -- 172.21.15.59:0/3555299114 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d6406eb00 con 0x7f4d5006b4d0 2024-03-06T07:26:21.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.245+0000 7f4d61ffb640 1 -- 172.21.15.59:0/3555299114 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240122 (secure 0 0 0) 0x7f4d6406eb00 con 0x7f4d5006b4d0 2024-03-06T07:26:21.249 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:26:21.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 -- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b4d0 msgr2=0x7f4d5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b4d0 0x7f4d5006d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4d6415a7e0 tx=0x7f4d58009040 comp rx=0 tx=0).stop 2024-03-06T07:26:21.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 -- 172.21.15.59:0/3555299114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6413dc10 msgr2=0x7f4d641f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6413dc10 0x7f4d641f67f0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c02e120 tx=0x7f4d5c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 -- 172.21.15.59:0/3555299114 shutdown_connections 2024-03-06T07:26:21.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4d5006b4d0 0x7f4d5006d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d6415ef60 0x7f4d640ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d6415bf30 0x7f4d640cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 --2- 172.21.15.59:0/3555299114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d6413dc10 0x7f4d641f67f0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 -- 172.21.15.59:0/3555299114 >> 172.21.15.59:0/3555299114 conn(0x7f4d640cf8a0 msgr2=0x7f4d6413cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 -- 172.21.15.59:0/3555299114 shutdown_connections 2024-03-06T07:26:21.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.248+0000 7f4d6b41c640 1 -- 172.21.15.59:0/3555299114 wait complete. 2024-03-06T07:26:21.261 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-03-06T07:26:19.358912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259208,"kb_used_data":9720,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715512,"statfs":{"total":773094113280,"available":772828684288,"internally_reserved":0,"allocated":9953280,"data_stored":4886972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000710"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850403+0000","last_change":"2024-03-06T07:26:15.850403+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850403+0000","last_undegraded":"2024-03-06T07:26:15.850403+0000","last_fullsized":"2024-03-06T07:26:15.850403+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849894+0000","last_change":"2024-03-06T07:26:15.849894+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849894+0000","last_undegraded":"2024-03-06T07:26:15.849894+0000","last_fullsized":"2024-03-06T07:26:15.849894+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850385+0000","last_change":"2024-03-06T07:26:15.850385+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850385+0000","last_undegraded":"2024-03-06T07:26:15.850385+0000","last_fullsized":"2024-03-06T07:26:15.850385+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850344+0000","last_change":"2024-03-06T07:26:15.850344+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850344+0000","last_undegraded":"2024-03-06T07:26:15.850344+0000","last_fullsized":"2024-03-06T07:26:15.850344+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.848404+0000","last_change":"2024-03-06T07:26:15.848404+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.848404+0000","last_undegraded":"2024-03-06T07:26:15.848404+0000","last_fullsized":"2024-03-06T07:26:15.848404+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849998+0000","last_change":"2024-03-06T07:26:15.849998+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849998+0000","last_undegraded":"2024-03-06T07:26:15.849998+0000","last_fullsized":"2024-03-06T07:26:15.849998+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850220+0000","last_change":"2024-03-06T07:26:15.850220+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850220+0000","last_undegraded":"2024-03-06T07:26:15.850220+0000","last_fullsized":"2024-03-06T07:26:15.850220+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.848320+0000","last_change":"2024-03-06T07:26:15.848320+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.848320+0000","last_undegraded":"2024-03-06T07:26:15.848320+0000","last_fullsized":"2024-03-06T07:26:15.848320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847839+0000","last_change":"2024-03-06T07:26:15.847839+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847839+0000","last_undegraded":"2024-03-06T07:26:15.847839+0000","last_fullsized":"2024-03-06T07:26:15.847839+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.845422+0000","last_change":"2024-03-06T07:26:15.845422+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.845422+0000","last_undegraded":"2024-03-06T07:26:15.845422+0000","last_fullsized":"2024-03-06T07:26:15.845422+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849881+0000","last_change":"2024-03-06T07:26:15.849881+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849881+0000","last_undegraded":"2024-03-06T07:26:15.849881+0000","last_fullsized":"2024-03-06T07:26:15.849881+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847799+0000","last_change":"2024-03-06T07:26:15.847799+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847799+0000","last_undegraded":"2024-03-06T07:26:15.847799+0000","last_fullsized":"2024-03-06T07:26:15.847799+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.848548+0000","last_change":"2024-03-06T07:26:15.848548+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.848548+0000","last_undegraded":"2024-03-06T07:26:15.848548+0000","last_fullsized":"2024-03-06T07:26:15.848548+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.848106+0000","last_change":"2024-03-06T07:26:15.848106+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.848106+0000","last_undegraded":"2024-03-06T07:26:15.848106+0000","last_fullsized":"2024-03-06T07:26:15.848106+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847798+0000","last_change":"2024-03-06T07:26:15.847798+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847798+0000","last_undegraded":"2024-03-06T07:26:15.847798+0000","last_fullsized":"2024-03-06T07:26:15.847798+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847824+0000","last_change":"2024-03-06T07:26:15.847824+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847824+0000","last_undegraded":"2024-03-06T07:26:15.847824+0000","last_fullsized":"2024-03-06T07:26:15.847824+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847880+0000","last_change":"2024-03-06T07:26:15.847880+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847880+0000","last_undegraded":"2024-03-06T07:26:15.847880+0000","last_fullsized":"2024-03-06T07:26:15.847880+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850317+0000","last_change":"2024-03-06T07:26:15.850317+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850317+0000","last_undegraded":"2024-03-06T07:26:15.850317+0000","last_fullsized":"2024-03-06T07:26:15.850317+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849910+0000","last_change":"2024-03-06T07:26:15.849910+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849910+0000","last_undegraded":"2024-03-06T07:26:15.849910+0000","last_fullsized":"2024-03-06T07:26:15.849910+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849543+0000","last_change":"2024-03-06T07:26:15.849543+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849543+0000","last_undegraded":"2024-03-06T07:26:15.849543+0000","last_fullsized":"2024-03-06T07:26:15.849543+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.845909+0000","last_change":"2024-03-06T07:26:15.845909+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.845909+0000","last_undegraded":"2024-03-06T07:26:15.845909+0000","last_fullsized":"2024-03-06T07:26:15.845909+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.845904+0000","last_change":"2024-03-06T07:26:15.845904+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.845904+0000","last_undegraded":"2024-03-06T07:26:15.845904+0000","last_fullsized":"2024-03-06T07:26:15.845904+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849657+0000","last_change":"2024-03-06T07:26:15.849657+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849657+0000","last_undegraded":"2024-03-06T07:26:15.849657+0000","last_fullsized":"2024-03-06T07:26:15.849657+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.845872+0000","last_change":"2024-03-06T07:26:15.845872+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.845872+0000","last_undegraded":"2024-03-06T07:26:15.845872+0000","last_fullsized":"2024-03-06T07:26:15.845872+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849029+0000","last_change":"2024-03-06T07:26:15.849029+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849029+0000","last_undegraded":"2024-03-06T07:26:15.849029+0000","last_fullsized":"2024-03-06T07:26:15.849029+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.848399+0000","last_change":"2024-03-06T07:26:15.848399+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.848399+0000","last_undegraded":"2024-03-06T07:26:15.848399+0000","last_fullsized":"2024-03-06T07:26:15.848399+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850394+0000","last_change":"2024-03-06T07:26:15.850394+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850394+0000","last_undegraded":"2024-03-06T07:26:15.850394+0000","last_fullsized":"2024-03-06T07:26:15.850394+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847905+0000","last_change":"2024-03-06T07:26:15.847905+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847905+0000","last_undegraded":"2024-03-06T07:26:15.847905+0000","last_fullsized":"2024-03-06T07:26:15.847905+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850379+0000","last_change":"2024-03-06T07:26:15.850379+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850379+0000","last_undegraded":"2024-03-06T07:26:15.850379+0000","last_fullsized":"2024-03-06T07:26:15.850379+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.845422+0000","last_change":"2024-03-06T07:26:15.845422+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.845422+0000","last_undegraded":"2024-03-06T07:26:15.845422+0000","last_fullsized":"2024-03-06T07:26:15.845422+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.847954+0000","last_change":"2024-03-06T07:26:15.847954+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.847954+0000","last_undegraded":"2024-03-06T07:26:15.847954+0000","last_fullsized":"2024-03-06T07:26:15.847954+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.849909+0000","last_change":"2024-03-06T07:26:15.849909+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.849909+0000","last_undegraded":"2024-03-06T07:26:15.849909+0000","last_fullsized":"2024-03-06T07:26:15.849909+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":295,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-06T07:26:15.850906+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:26:15.850906+0000","last_peered":"2024-03-06T07:26:15.850906+0000","last_clean":"2024-03-06T07:26:15.850906+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:26:15.850906+0000","last_undegraded":"2024-03-06T07:26:15.850906+0000","last_fullsized":"2024-03-06T07:26:15.850906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32824,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339016,"statfs":{"total":96636764160,"available":96603152384,"internally_reserved":0,"allocated":1683456,"data_stored":1050739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1101824,"data_stored":466188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1101824,"data_stored":466188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":1089536,"data_stored":460371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":460371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":460371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1101824,"data_stored":466188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1695744,"data_stored":1056556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:21.264 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.400+0000 7f839a732640 1 Processor -- start 2024-03-06T07:26:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.400+0000 7f839a732640 1 -- start start 2024-03-06T07:26:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.401+0000 7f839a732640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839415bb20 0x7f83940d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.401+0000 7f839a732640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8394161980 con 0x7f83940cd8c0 2024-03-06T07:26:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.401+0000 7f839a732640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8394161c60 con 0x7f83940cfff0 2024-03-06T07:26:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.401+0000 7f839a732640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8394161f40 con 0x7f839415bb20 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.401+0000 7f8398ccf640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f83940cd8c0 0x7f83940cebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34398/0 (socket says 172.21.15.59:34398) 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.401+0000 7f8398ccf640 1 -- 172.21.15.59:0/1946677791 learned_addr learned my addr 172.21.15.59:0/1946677791 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.401+0000 7f8393fff640 1 --2- 172.21.15.59:0/1946677791 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839415bb20 0x7f83940d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.402+0000 7f8392ffd640 1 -- 172.21.15.59:0/1946677791 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233872984 0 0) 0x7f8394161980 con 0x7f83940cd8c0 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.402+0000 7f8392ffd640 1 -- 172.21.15.59:0/1946677791 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f837c003520 con 0x7f83940cd8c0 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.402+0000 7f8392ffd640 1 -- 172.21.15.59:0/1946677791 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898209398 0 0) 0x7f8394161c60 con 0x7f83940cfff0 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.402+0000 7f8392ffd640 1 -- 172.21.15.59:0/1946677791 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8394161980 con 0x7f83940cfff0 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.402+0000 7f8393fff640 1 -- 172.21.15.59:0/1946677791 >> v1:172.21.15.164:6789/0 conn(0x7f83940cfff0 legacy=0x7f83940ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.402+0000 7f8393fff640 1 -- 172.21.15.59:0/1946677791 >> v1:172.21.15.59:6789/0 conn(0x7f83940cd8c0 legacy=0x7f83940cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.402+0000 7f8393fff640 1 -- 172.21.15.59:0/1946677791 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8394162220 con 0x7f839415bb20 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.403+0000 7f8393fff640 1 --2- 172.21.15.59:0/1946677791 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839415bb20 0x7f83940d1df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f83840097d0 tx=0x7f838402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aab97fdec6108e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.403+0000 7f8392ffd640 1 -- 172.21.15.59:0/1946677791 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384035020 con 0x7f839415bb20 2024-03-06T07:26:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.403+0000 7f8392ffd640 1 -- 172.21.15.59:0/1946677791 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8384031800 con 0x7f839415bb20 2024-03-06T07:26:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.403+0000 7f8392ffd640 1 -- 172.21.15.59:0/1946677791 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384031bd0 con 0x7f839415bb20 2024-03-06T07:26:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.403+0000 7f839a732640 1 -- 172.21.15.59:0/1946677791 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839415bb20 msgr2=0x7f83940d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.403+0000 7f839a732640 1 --2- 172.21.15.59:0/1946677791 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839415bb20 0x7f83940d1df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f83840097d0 tx=0x7f838402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.403+0000 7f839a732640 1 -- 172.21.15.59:0/1946677791 shutdown_connections 2024-03-06T07:26:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.403+0000 7f839a732640 1 --2- 172.21.15.59:0/1946677791 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839415bb20 0x7f83940d1df0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.403+0000 7f839a732640 1 -- 172.21.15.59:0/1946677791 >> 172.21.15.59:0/1946677791 conn(0x7f839409a9f0 msgr2=0x7f839409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.403+0000 7f839a732640 1 -- 172.21.15.59:0/1946677791 shutdown_connections 2024-03-06T07:26:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.403+0000 7f839a732640 1 -- 172.21.15.59:0/1946677791 wait complete. 2024-03-06T07:26:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.404+0000 7f839a732640 1 Processor -- start 2024-03-06T07:26:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.405+0000 7f839a732640 1 -- start start 2024-03-06T07:26:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.405+0000 7f839a732640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83940cd8c0 0x7f83940b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.405+0000 7f839a732640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83940cfff0 0x7f8394150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.405+0000 7f839a732640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839415bb20 0x7f83940b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.405+0000 7f839a732640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83940d2330 con 0x7f83940cfff0 2024-03-06T07:26:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.405+0000 7f839a732640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83940cf2d0 con 0x7f83940cd8c0 2024-03-06T07:26:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.405+0000 7f839a732640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83940d24b0 con 0x7f839415bb20 2024-03-06T07:26:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.405+0000 7f83937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83940cfff0 0x7f8394150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.405+0000 7f83937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83940cfff0 0x7f8394150fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47470/0 (socket says 172.21.15.59:47470) 2024-03-06T07:26:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.405+0000 7f83937fe640 1 -- 172.21.15.59:0/1106000823 learned_addr learned my addr 172.21.15.59:0/1106000823 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.405+0000 7f8393fff640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83940cd8c0 0x7f83940b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f8398ccf640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839415bb20 0x7f83940b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f83937fe640 1 -- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839415bb20 msgr2=0x7f83940b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f83937fe640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839415bb20 0x7f83940b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f83937fe640 1 -- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83940cd8c0 msgr2=0x7f83940b6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f83937fe640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83940cd8c0 0x7f83940b6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f83937fe640 1 -- 172.21.15.59:0/1106000823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839415c500 con 0x7f83940cfff0 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f83937fe640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83940cfff0 0x7f8394150fe0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f838800c460 tx=0x7f838800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f8390ff9640 1 -- 172.21.15.59:0/1106000823 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388017070 con 0x7f83940cfff0 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f8390ff9640 1 -- 172.21.15.59:0/1106000823 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83880052f0 con 0x7f83940cfff0 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f839a732640 1 -- 172.21.15.59:0/1106000823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839415c7e0 con 0x7f83940cfff0 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f8390ff9640 1 -- 172.21.15.59:0/1106000823 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388003830 con 0x7f83940cfff0 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.406+0000 7f839a732640 1 -- 172.21.15.59:0/1106000823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839415cda0 con 0x7f83940cfff0 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.407+0000 7f839a732640 1 -- 172.21.15.59:0/1106000823 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839404f330 con 0x7f83940cfff0 2024-03-06T07:26:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.408+0000 7f8390ff9640 1 -- 172.21.15.59:0/1106000823 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f838801a070 con 0x7f83940cfff0 2024-03-06T07:26:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.408+0000 7f8390ff9640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f838006b4d0 0x7f838006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.408+0000 7f8390ff9640 1 -- 172.21.15.59:0/1106000823 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83880894a0 con 0x7f83940cfff0 2024-03-06T07:26:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.408+0000 7f8393fff640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f838006b4d0 0x7f838006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.409+0000 7f8393fff640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f838006b4d0 0x7f838006d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8384036000 tx=0x7f838403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.412+0000 7f8390ff9640 1 -- 172.21.15.59:0/1106000823 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8388057d60 con 0x7f83940cfff0 2024-03-06T07:26:21.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.593+0000 7f839a732640 1 -- 172.21.15.59:0/1106000823 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83940b1820 con 0x7f838006b4d0 2024-03-06T07:26:21.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.600+0000 7f8390ff9640 1 -- 172.21.15.59:0/1106000823 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+250478 (secure 0 0 0) 0x7f83940b1820 con 0x7f838006b4d0 2024-03-06T07:26:21.604 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:26:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.602+0000 7f839a732640 1 -- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f838006b4d0 msgr2=0x7f838006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.602+0000 7f839a732640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f838006b4d0 0x7f838006d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8384036000 tx=0x7f838403c040 comp rx=0 tx=0).stop 2024-03-06T07:26:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.602+0000 7f839a732640 1 -- 172.21.15.59:0/1106000823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83940cfff0 msgr2=0x7f8394150fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.602+0000 7f839a732640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83940cfff0 0x7f8394150fe0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f838800c460 tx=0x7f838800c920 comp rx=0 tx=0).stop 2024-03-06T07:26:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.602+0000 7f839a732640 1 -- 172.21.15.59:0/1106000823 shutdown_connections 2024-03-06T07:26:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.602+0000 7f839a732640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f838006b4d0 0x7f838006d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.602+0000 7f839a732640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839415bb20 0x7f83940b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.602+0000 7f839a732640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83940cfff0 0x7f8394150fe0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.602+0000 7f839a732640 1 --2- 172.21.15.59:0/1106000823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83940cd8c0 0x7f83940b6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.603+0000 7f839a732640 1 -- 172.21.15.59:0/1106000823 >> 172.21.15.59:0/1106000823 conn(0x7f839409a9f0 msgr2=0x7f83940ce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.603+0000 7f839a732640 1 -- 172.21.15.59:0/1106000823 shutdown_connections 2024-03-06T07:26:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.603+0000 7f839a732640 1 -- 172.21.15.59:0/1106000823 wait complete. 2024-03-06T07:26:21.615 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-03-06T07:26:21.359172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592708,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":176,"acting":176,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259552,"kb_used_data":10076,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715168,"statfs":{"total":773094113280,"available":772828332032,"internally_reserved":0,"allocated":10317824,"data_stored":5049858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2340,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.514806"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854775+0000","last_change":"2024-03-06T07:26:17.877929+0000","last_active":"2024-03-06T07:26:18.854775+0000","last_peered":"2024-03-06T07:26:18.854775+0000","last_clean":"2024-03-06T07:26:18.854775+0000","last_became_active":"2024-03-06T07:26:17.877642+0000","last_became_peered":"2024-03-06T07:26:17.877642+0000","last_unstale":"2024-03-06T07:26:18.854775+0000","last_undegraded":"2024-03-06T07:26:18.854775+0000","last_fullsized":"2024-03-06T07:26:18.854775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:53.393641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875012+0000","last_change":"2024-03-06T07:26:17.868113+0000","last_active":"2024-03-06T07:26:18.875012+0000","last_peered":"2024-03-06T07:26:18.875012+0000","last_clean":"2024-03-06T07:26:18.875012+0000","last_became_active":"2024-03-06T07:26:17.867743+0000","last_became_peered":"2024-03-06T07:26:17.867743+0000","last_unstale":"2024-03-06T07:26:18.875012+0000","last_undegraded":"2024-03-06T07:26:18.875012+0000","last_fullsized":"2024-03-06T07:26:18.875012+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:57.329605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854687+0000","last_change":"2024-03-06T07:26:17.878201+0000","last_active":"2024-03-06T07:26:18.854687+0000","last_peered":"2024-03-06T07:26:18.854687+0000","last_clean":"2024-03-06T07:26:18.854687+0000","last_became_active":"2024-03-06T07:26:17.877378+0000","last_became_peered":"2024-03-06T07:26:17.877378+0000","last_unstale":"2024-03-06T07:26:18.854687+0000","last_undegraded":"2024-03-06T07:26:18.854687+0000","last_fullsized":"2024-03-06T07:26:18.854687+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:41.440891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855757+0000","last_change":"2024-03-06T07:26:17.871063+0000","last_active":"2024-03-06T07:26:18.855757+0000","last_peered":"2024-03-06T07:26:18.855757+0000","last_clean":"2024-03-06T07:26:18.855757+0000","last_became_active":"2024-03-06T07:26:17.870049+0000","last_became_peered":"2024-03-06T07:26:17.870049+0000","last_unstale":"2024-03-06T07:26:18.855757+0000","last_undegraded":"2024-03-06T07:26:18.855757+0000","last_fullsized":"2024-03-06T07:26:18.855757+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:43.505953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631500+0000","last_change":"2024-03-06T07:26:17.864205+0000","last_active":"2024-03-06T07:26:19.631500+0000","last_peered":"2024-03-06T07:26:19.631500+0000","last_clean":"2024-03-06T07:26:19.631500+0000","last_became_active":"2024-03-06T07:26:17.863941+0000","last_became_peered":"2024-03-06T07:26:17.863941+0000","last_unstale":"2024-03-06T07:26:19.631500+0000","last_undegraded":"2024-03-06T07:26:19.631500+0000","last_fullsized":"2024-03-06T07:26:19.631500+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:03.507471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631356+0000","last_change":"2024-03-06T07:26:17.861310+0000","last_active":"2024-03-06T07:26:19.631356+0000","last_peered":"2024-03-06T07:26:19.631356+0000","last_clean":"2024-03-06T07:26:19.631356+0000","last_became_active":"2024-03-06T07:26:17.860944+0000","last_became_peered":"2024-03-06T07:26:17.860944+0000","last_unstale":"2024-03-06T07:26:19.631356+0000","last_undegraded":"2024-03-06T07:26:19.631356+0000","last_fullsized":"2024-03-06T07:26:19.631356+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:55.503575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874629+0000","last_change":"2024-03-06T07:26:17.861435+0000","last_active":"2024-03-06T07:26:18.874629+0000","last_peered":"2024-03-06T07:26:18.874629+0000","last_clean":"2024-03-06T07:26:18.874629+0000","last_became_active":"2024-03-06T07:26:17.861014+0000","last_became_peered":"2024-03-06T07:26:17.861014+0000","last_unstale":"2024-03-06T07:26:18.874629+0000","last_undegraded":"2024-03-06T07:26:18.874629+0000","last_fullsized":"2024-03-06T07:26:18.874629+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:54.987454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632809+0000","last_change":"2024-03-06T07:26:17.873920+0000","last_active":"2024-03-06T07:26:19.632809+0000","last_peered":"2024-03-06T07:26:19.632809+0000","last_clean":"2024-03-06T07:26:19.632809+0000","last_became_active":"2024-03-06T07:26:17.872835+0000","last_became_peered":"2024-03-06T07:26:17.872835+0000","last_unstale":"2024-03-06T07:26:19.632809+0000","last_undegraded":"2024-03-06T07:26:19.632809+0000","last_fullsized":"2024-03-06T07:26:19.632809+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:58.609019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631919+0000","last_change":"2024-03-06T07:26:17.874590+0000","last_active":"2024-03-06T07:26:19.631919+0000","last_peered":"2024-03-06T07:26:19.631919+0000","last_clean":"2024-03-06T07:26:19.631919+0000","last_became_active":"2024-03-06T07:26:17.874324+0000","last_became_peered":"2024-03-06T07:26:17.874324+0000","last_unstale":"2024-03-06T07:26:19.631919+0000","last_undegraded":"2024-03-06T07:26:19.631919+0000","last_fullsized":"2024-03-06T07:26:19.631919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:26.884759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632664+0000","last_change":"2024-03-06T07:26:17.875263+0000","last_active":"2024-03-06T07:26:19.632664+0000","last_peered":"2024-03-06T07:26:19.632664+0000","last_clean":"2024-03-06T07:26:19.632664+0000","last_became_active":"2024-03-06T07:26:17.874730+0000","last_became_peered":"2024-03-06T07:26:17.874730+0000","last_unstale":"2024-03-06T07:26:19.632664+0000","last_undegraded":"2024-03-06T07:26:19.632664+0000","last_fullsized":"2024-03-06T07:26:19.632664+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:55.503575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855537+0000","last_change":"2024-03-06T07:26:17.876108+0000","last_active":"2024-03-06T07:26:18.855537+0000","last_peered":"2024-03-06T07:26:18.855537+0000","last_clean":"2024-03-06T07:26:18.855537+0000","last_became_active":"2024-03-06T07:26:17.875830+0000","last_became_peered":"2024-03-06T07:26:17.875830+0000","last_unstale":"2024-03-06T07:26:18.855537+0000","last_undegraded":"2024-03-06T07:26:18.855537+0000","last_fullsized":"2024-03-06T07:26:18.855537+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:57.406246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876753+0000","last_change":"2024-03-06T07:26:17.868274+0000","last_active":"2024-03-06T07:26:18.876753+0000","last_peered":"2024-03-06T07:26:18.876753+0000","last_clean":"2024-03-06T07:26:18.876753+0000","last_became_active":"2024-03-06T07:26:17.867626+0000","last_became_peered":"2024-03-06T07:26:17.867626+0000","last_unstale":"2024-03-06T07:26:18.876753+0000","last_undegraded":"2024-03-06T07:26:18.876753+0000","last_fullsized":"2024-03-06T07:26:18.876753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:16.118134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855430+0000","last_change":"2024-03-06T07:26:17.870824+0000","last_active":"2024-03-06T07:26:18.855430+0000","last_peered":"2024-03-06T07:26:18.855430+0000","last_clean":"2024-03-06T07:26:18.855430+0000","last_became_active":"2024-03-06T07:26:17.870545+0000","last_became_peered":"2024-03-06T07:26:17.870545+0000","last_unstale":"2024-03-06T07:26:18.855430+0000","last_undegraded":"2024-03-06T07:26:18.855430+0000","last_fullsized":"2024-03-06T07:26:18.855430+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:22.063664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631403+0000","last_change":"2024-03-06T07:26:17.861351+0000","last_active":"2024-03-06T07:26:19.631403+0000","last_peered":"2024-03-06T07:26:19.631403+0000","last_clean":"2024-03-06T07:26:19.631403+0000","last_became_active":"2024-03-06T07:26:17.861004+0000","last_became_peered":"2024-03-06T07:26:17.861004+0000","last_unstale":"2024-03-06T07:26:19.631403+0000","last_undegraded":"2024-03-06T07:26:19.631403+0000","last_fullsized":"2024-03-06T07:26:19.631403+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:58.609019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874995+0000","last_change":"2024-03-06T07:26:17.869794+0000","last_active":"2024-03-06T07:26:18.874995+0000","last_peered":"2024-03-06T07:26:18.874995+0000","last_clean":"2024-03-06T07:26:18.874995+0000","last_became_active":"2024-03-06T07:26:17.869531+0000","last_became_peered":"2024-03-06T07:26:17.869531+0000","last_unstale":"2024-03-06T07:26:18.874995+0000","last_undegraded":"2024-03-06T07:26:18.874995+0000","last_fullsized":"2024-03-06T07:26:18.874995+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:43.505953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855708+0000","last_change":"2024-03-06T07:26:17.871014+0000","last_active":"2024-03-06T07:26:18.855708+0000","last_peered":"2024-03-06T07:26:18.855708+0000","last_clean":"2024-03-06T07:26:18.855708+0000","last_became_active":"2024-03-06T07:26:17.870687+0000","last_became_peered":"2024-03-06T07:26:17.870687+0000","last_unstale":"2024-03-06T07:26:18.855708+0000","last_undegraded":"2024-03-06T07:26:18.855708+0000","last_fullsized":"2024-03-06T07:26:18.855708+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:54.987454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877113+0000","last_change":"2024-03-06T07:26:17.873014+0000","last_active":"2024-03-06T07:26:18.877113+0000","last_peered":"2024-03-06T07:26:18.877113+0000","last_clean":"2024-03-06T07:26:18.877113+0000","last_became_active":"2024-03-06T07:26:17.872701+0000","last_became_peered":"2024-03-06T07:26:17.872701+0000","last_unstale":"2024-03-06T07:26:18.877113+0000","last_undegraded":"2024-03-06T07:26:18.877113+0000","last_fullsized":"2024-03-06T07:26:18.877113+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:38.541605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632622+0000","last_change":"2024-03-06T07:26:17.877086+0000","last_active":"2024-03-06T07:26:19.632622+0000","last_peered":"2024-03-06T07:26:19.632622+0000","last_clean":"2024-03-06T07:26:19.632622+0000","last_became_active":"2024-03-06T07:26:17.875124+0000","last_became_peered":"2024-03-06T07:26:17.875124+0000","last_unstale":"2024-03-06T07:26:19.632622+0000","last_undegraded":"2024-03-06T07:26:19.632622+0000","last_fullsized":"2024-03-06T07:26:19.632622+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:36.314409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876998+0000","last_change":"2024-03-06T07:26:17.862241+0000","last_active":"2024-03-06T07:26:18.876998+0000","last_peered":"2024-03-06T07:26:18.876998+0000","last_clean":"2024-03-06T07:26:18.876998+0000","last_became_active":"2024-03-06T07:26:17.861340+0000","last_became_peered":"2024-03-06T07:26:17.861340+0000","last_unstale":"2024-03-06T07:26:18.876998+0000","last_undegraded":"2024-03-06T07:26:18.876998+0000","last_fullsized":"2024-03-06T07:26:18.876998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:22.664538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874576+0000","last_change":"2024-03-06T07:26:17.861898+0000","last_active":"2024-03-06T07:26:18.874576+0000","last_peered":"2024-03-06T07:26:18.874576+0000","last_clean":"2024-03-06T07:26:18.874576+0000","last_became_active":"2024-03-06T07:26:17.861634+0000","last_became_peered":"2024-03-06T07:26:17.861634+0000","last_unstale":"2024-03-06T07:26:18.874576+0000","last_undegraded":"2024-03-06T07:26:18.874576+0000","last_fullsized":"2024-03-06T07:26:18.874576+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:57.406246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874586+0000","last_change":"2024-03-06T07:26:17.865417+0000","last_active":"2024-03-06T07:26:18.874586+0000","last_peered":"2024-03-06T07:26:18.874586+0000","last_clean":"2024-03-06T07:26:18.874586+0000","last_became_active":"2024-03-06T07:26:17.865139+0000","last_became_peered":"2024-03-06T07:26:17.865139+0000","last_unstale":"2024-03-06T07:26:18.874586+0000","last_undegraded":"2024-03-06T07:26:18.874586+0000","last_fullsized":"2024-03-06T07:26:18.874586+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.528335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631549+0000","last_change":"2024-03-06T07:26:17.868477+0000","last_active":"2024-03-06T07:26:19.631549+0000","last_peered":"2024-03-06T07:26:19.631549+0000","last_clean":"2024-03-06T07:26:19.631549+0000","last_became_active":"2024-03-06T07:26:17.867943+0000","last_became_peered":"2024-03-06T07:26:17.867943+0000","last_unstale":"2024-03-06T07:26:19.631549+0000","last_undegraded":"2024-03-06T07:26:19.631549+0000","last_fullsized":"2024-03-06T07:26:19.631549+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:02.151317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875070+0000","last_change":"2024-03-06T07:26:17.873693+0000","last_active":"2024-03-06T07:26:18.875070+0000","last_peered":"2024-03-06T07:26:18.875070+0000","last_clean":"2024-03-06T07:26:18.875070+0000","last_became_active":"2024-03-06T07:26:17.873363+0000","last_became_peered":"2024-03-06T07:26:17.873363+0000","last_unstale":"2024-03-06T07:26:18.875070+0000","last_undegraded":"2024-03-06T07:26:18.875070+0000","last_fullsized":"2024-03-06T07:26:18.875070+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:06.936077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632055+0000","last_change":"2024-03-06T07:26:17.864951+0000","last_active":"2024-03-06T07:26:19.632055+0000","last_peered":"2024-03-06T07:26:19.632055+0000","last_clean":"2024-03-06T07:26:19.632055+0000","last_became_active":"2024-03-06T07:26:17.864683+0000","last_became_peered":"2024-03-06T07:26:17.864683+0000","last_unstale":"2024-03-06T07:26:19.632055+0000","last_undegraded":"2024-03-06T07:26:19.632055+0000","last_fullsized":"2024-03-06T07:26:19.632055+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:46.261854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874766+0000","last_change":"2024-03-06T07:26:17.873122+0000","last_active":"2024-03-06T07:26:18.874766+0000","last_peered":"2024-03-06T07:26:18.874766+0000","last_clean":"2024-03-06T07:26:18.874766+0000","last_became_active":"2024-03-06T07:26:17.872806+0000","last_became_peered":"2024-03-06T07:26:17.872806+0000","last_unstale":"2024-03-06T07:26:18.874766+0000","last_undegraded":"2024-03-06T07:26:18.874766+0000","last_fullsized":"2024-03-06T07:26:18.874766+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:08.056201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876785+0000","last_change":"2024-03-06T07:26:17.868507+0000","last_active":"2024-03-06T07:26:18.876785+0000","last_peered":"2024-03-06T07:26:18.876785+0000","last_clean":"2024-03-06T07:26:18.876785+0000","last_became_active":"2024-03-06T07:26:17.868006+0000","last_became_peered":"2024-03-06T07:26:17.868006+0000","last_unstale":"2024-03-06T07:26:18.876785+0000","last_undegraded":"2024-03-06T07:26:18.876785+0000","last_fullsized":"2024-03-06T07:26:18.876785+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:10.435503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855724+0000","last_change":"2024-03-06T07:26:17.879833+0000","last_active":"2024-03-06T07:26:18.855724+0000","last_peered":"2024-03-06T07:26:18.855724+0000","last_clean":"2024-03-06T07:26:18.855724+0000","last_became_active":"2024-03-06T07:26:17.878938+0000","last_became_peered":"2024-03-06T07:26:17.878938+0000","last_unstale":"2024-03-06T07:26:18.855724+0000","last_undegraded":"2024-03-06T07:26:18.855724+0000","last_fullsized":"2024-03-06T07:26:18.855724+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:08.056201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854529+0000","last_change":"2024-03-06T07:26:17.866962+0000","last_active":"2024-03-06T07:26:18.854529+0000","last_peered":"2024-03-06T07:26:18.854529+0000","last_clean":"2024-03-06T07:26:18.854529+0000","last_became_active":"2024-03-06T07:26:17.866658+0000","last_became_peered":"2024-03-06T07:26:17.866658+0000","last_unstale":"2024-03-06T07:26:18.854529+0000","last_undegraded":"2024-03-06T07:26:18.854529+0000","last_fullsized":"2024-03-06T07:26:18.854529+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:25.582364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877006+0000","last_change":"2024-03-06T07:26:17.860951+0000","last_active":"2024-03-06T07:26:18.877006+0000","last_peered":"2024-03-06T07:26:18.877006+0000","last_clean":"2024-03-06T07:26:18.877006+0000","last_became_active":"2024-03-06T07:26:17.860689+0000","last_became_peered":"2024-03-06T07:26:17.860689+0000","last_unstale":"2024-03-06T07:26:18.877006+0000","last_undegraded":"2024-03-06T07:26:18.877006+0000","last_fullsized":"2024-03-06T07:26:18.877006+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:50.018552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"91'4","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874923+0000","last_change":"2024-03-06T07:26:16.861346+0000","last_active":"2024-03-06T07:26:18.874923+0000","last_peered":"2024-03-06T07:26:18.874923+0000","last_clean":"2024-03-06T07:26:18.874923+0000","last_became_active":"2024-03-06T07:26:16.860585+0000","last_became_peered":"2024-03-06T07:26:16.860585+0000","last_unstale":"2024-03-06T07:26:18.874923+0000","last_undegraded":"2024-03-06T07:26:18.874923+0000","last_fullsized":"2024-03-06T07:26:18.874923+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:06.710593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855827+0000","last_change":"2024-03-06T07:26:17.879842+0000","last_active":"2024-03-06T07:26:18.855827+0000","last_peered":"2024-03-06T07:26:18.855827+0000","last_clean":"2024-03-06T07:26:18.855827+0000","last_became_active":"2024-03-06T07:26:17.879291+0000","last_became_peered":"2024-03-06T07:26:17.879291+0000","last_unstale":"2024-03-06T07:26:18.855827+0000","last_undegraded":"2024-03-06T07:26:18.855827+0000","last_fullsized":"2024-03-06T07:26:18.855827+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:57.329605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631480+0000","last_change":"2024-03-06T07:26:16.868319+0000","last_active":"2024-03-06T07:26:19.631480+0000","last_peered":"2024-03-06T07:26:19.631480+0000","last_clean":"2024-03-06T07:26:19.631480+0000","last_became_active":"2024-03-06T07:26:16.868043+0000","last_became_peered":"2024-03-06T07:26:16.868043+0000","last_unstale":"2024-03-06T07:26:19.631480+0000","last_undegraded":"2024-03-06T07:26:19.631480+0000","last_fullsized":"2024-03-06T07:26:19.631480+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:25.302742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854455+0000","last_change":"2024-03-06T07:26:17.875745+0000","last_active":"2024-03-06T07:26:18.854455+0000","last_peered":"2024-03-06T07:26:18.854455+0000","last_clean":"2024-03-06T07:26:18.854455+0000","last_became_active":"2024-03-06T07:26:17.875471+0000","last_became_peered":"2024-03-06T07:26:17.875471+0000","last_unstale":"2024-03-06T07:26:18.854455+0000","last_undegraded":"2024-03-06T07:26:18.854455+0000","last_fullsized":"2024-03-06T07:26:18.854455+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:47.848728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874838+0000","last_change":"2024-03-06T07:26:16.870929+0000","last_active":"2024-03-06T07:26:18.874838+0000","last_peered":"2024-03-06T07:26:18.874838+0000","last_clean":"2024-03-06T07:26:18.874838+0000","last_became_active":"2024-03-06T07:26:16.870661+0000","last_became_peered":"2024-03-06T07:26:16.870661+0000","last_unstale":"2024-03-06T07:26:18.874838+0000","last_undegraded":"2024-03-06T07:26:18.874838+0000","last_fullsized":"2024-03-06T07:26:18.874838+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:13.611316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632637+0000","last_change":"2024-03-06T07:26:17.872067+0000","last_active":"2024-03-06T07:26:19.632637+0000","last_peered":"2024-03-06T07:26:19.632637+0000","last_clean":"2024-03-06T07:26:19.632637+0000","last_became_active":"2024-03-06T07:26:17.871800+0000","last_became_peered":"2024-03-06T07:26:17.871800+0000","last_unstale":"2024-03-06T07:26:19.632637+0000","last_undegraded":"2024-03-06T07:26:19.632637+0000","last_fullsized":"2024-03-06T07:26:19.632637+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:27.404185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874719+0000","last_change":"2024-03-06T07:26:16.864464+0000","last_active":"2024-03-06T07:26:18.874719+0000","last_peered":"2024-03-06T07:26:18.874719+0000","last_clean":"2024-03-06T07:26:18.874719+0000","last_became_active":"2024-03-06T07:26:16.863949+0000","last_became_peered":"2024-03-06T07:26:16.863949+0000","last_unstale":"2024-03-06T07:26:18.874719+0000","last_undegraded":"2024-03-06T07:26:18.874719+0000","last_fullsized":"2024-03-06T07:26:18.874719+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:52.388832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856105+0000","last_change":"2024-03-06T07:26:17.872420+0000","last_active":"2024-03-06T07:26:18.856105+0000","last_peered":"2024-03-06T07:26:18.856105+0000","last_clean":"2024-03-06T07:26:18.856105+0000","last_became_active":"2024-03-06T07:26:17.872122+0000","last_became_peered":"2024-03-06T07:26:17.872122+0000","last_unstale":"2024-03-06T07:26:18.856105+0000","last_undegraded":"2024-03-06T07:26:18.856105+0000","last_fullsized":"2024-03-06T07:26:18.856105+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:15.723196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854281+0000","last_change":"2024-03-06T07:26:16.863634+0000","last_active":"2024-03-06T07:26:18.854281+0000","last_peered":"2024-03-06T07:26:18.854281+0000","last_clean":"2024-03-06T07:26:18.854281+0000","last_became_active":"2024-03-06T07:26:16.863372+0000","last_became_peered":"2024-03-06T07:26:16.863372+0000","last_unstale":"2024-03-06T07:26:18.854281+0000","last_undegraded":"2024-03-06T07:26:18.854281+0000","last_fullsized":"2024-03-06T07:26:18.854281+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:29.981611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631644+0000","last_change":"2024-03-06T07:26:17.871415+0000","last_active":"2024-03-06T07:26:19.631644+0000","last_peered":"2024-03-06T07:26:19.631644+0000","last_clean":"2024-03-06T07:26:19.631644+0000","last_became_active":"2024-03-06T07:26:17.871135+0000","last_became_peered":"2024-03-06T07:26:17.871135+0000","last_unstale":"2024-03-06T07:26:19.631644+0000","last_undegraded":"2024-03-06T07:26:19.631644+0000","last_fullsized":"2024-03-06T07:26:19.631644+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:36.291779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874580+0000","last_change":"2024-03-06T07:26:16.866013+0000","last_active":"2024-03-06T07:26:18.874580+0000","last_peered":"2024-03-06T07:26:18.874580+0000","last_clean":"2024-03-06T07:26:18.874580+0000","last_became_active":"2024-03-06T07:26:16.865737+0000","last_became_peered":"2024-03-06T07:26:16.865737+0000","last_unstale":"2024-03-06T07:26:18.874580+0000","last_undegraded":"2024-03-06T07:26:18.874580+0000","last_fullsized":"2024-03-06T07:26:18.874580+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:24.577556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855424+0000","last_change":"2024-03-06T07:26:17.871627+0000","last_active":"2024-03-06T07:26:18.855424+0000","last_peered":"2024-03-06T07:26:18.855424+0000","last_clean":"2024-03-06T07:26:18.855424+0000","last_became_active":"2024-03-06T07:26:17.871289+0000","last_became_peered":"2024-03-06T07:26:17.871289+0000","last_unstale":"2024-03-06T07:26:18.855424+0000","last_undegraded":"2024-03-06T07:26:18.855424+0000","last_fullsized":"2024-03-06T07:26:18.855424+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:52.299295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850220+0000","last_change":"2024-03-06T07:26:15.850220+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850220+0000","last_undegraded":"2024-03-06T07:26:15.850220+0000","last_fullsized":"2024-03-06T07:26:15.850220+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632457+0000","last_change":"2024-03-06T07:26:17.873691+0000","last_active":"2024-03-06T07:26:19.632457+0000","last_peered":"2024-03-06T07:26:19.632457+0000","last_clean":"2024-03-06T07:26:19.632457+0000","last_became_active":"2024-03-06T07:26:17.873417+0000","last_became_peered":"2024-03-06T07:26:17.873417+0000","last_unstale":"2024-03-06T07:26:19.632457+0000","last_undegraded":"2024-03-06T07:26:19.632457+0000","last_fullsized":"2024-03-06T07:26:19.632457+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:59.170419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854190+0000","last_change":"2024-03-06T07:26:16.869647+0000","last_active":"2024-03-06T07:26:18.854190+0000","last_peered":"2024-03-06T07:26:18.854190+0000","last_clean":"2024-03-06T07:26:18.854190+0000","last_became_active":"2024-03-06T07:26:16.869365+0000","last_became_peered":"2024-03-06T07:26:16.869365+0000","last_unstale":"2024-03-06T07:26:18.854190+0000","last_undegraded":"2024-03-06T07:26:18.854190+0000","last_fullsized":"2024-03-06T07:26:18.854190+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:21.875952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875098+0000","last_change":"2024-03-06T07:26:17.870669+0000","last_active":"2024-03-06T07:26:18.875098+0000","last_peered":"2024-03-06T07:26:18.875098+0000","last_clean":"2024-03-06T07:26:18.875098+0000","last_became_active":"2024-03-06T07:26:17.870363+0000","last_became_peered":"2024-03-06T07:26:17.870363+0000","last_unstale":"2024-03-06T07:26:18.875098+0000","last_undegraded":"2024-03-06T07:26:18.875098+0000","last_fullsized":"2024-03-06T07:26:18.875098+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:47.379775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854820+0000","last_change":"2024-03-06T07:26:16.865155+0000","last_active":"2024-03-06T07:26:18.854820+0000","last_peered":"2024-03-06T07:26:18.854820+0000","last_clean":"2024-03-06T07:26:18.854820+0000","last_became_active":"2024-03-06T07:26:16.864873+0000","last_became_peered":"2024-03-06T07:26:16.864873+0000","last_unstale":"2024-03-06T07:26:18.854820+0000","last_undegraded":"2024-03-06T07:26:18.854820+0000","last_fullsized":"2024-03-06T07:26:18.854820+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:57.303182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632939+0000","last_change":"2024-03-06T07:26:17.874753+0000","last_active":"2024-03-06T07:26:19.632939+0000","last_peered":"2024-03-06T07:26:19.632939+0000","last_clean":"2024-03-06T07:26:19.632939+0000","last_became_active":"2024-03-06T07:26:17.874488+0000","last_became_peered":"2024-03-06T07:26:17.874488+0000","last_unstale":"2024-03-06T07:26:19.632939+0000","last_undegraded":"2024-03-06T07:26:19.632939+0000","last_fullsized":"2024-03-06T07:26:19.632939+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:03.507471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855695+0000","last_change":"2024-03-06T07:26:16.872968+0000","last_active":"2024-03-06T07:26:18.855695+0000","last_peered":"2024-03-06T07:26:18.855695+0000","last_clean":"2024-03-06T07:26:18.855695+0000","last_became_active":"2024-03-06T07:26:16.872657+0000","last_became_peered":"2024-03-06T07:26:16.872657+0000","last_unstale":"2024-03-06T07:26:18.855695+0000","last_undegraded":"2024-03-06T07:26:18.855695+0000","last_fullsized":"2024-03-06T07:26:18.855695+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:35.309600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"91'3","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632226+0000","last_change":"2024-03-06T07:26:16.866799+0000","last_active":"2024-03-06T07:26:19.632226+0000","last_peered":"2024-03-06T07:26:19.632226+0000","last_clean":"2024-03-06T07:26:19.632226+0000","last_became_active":"2024-03-06T07:26:16.866520+0000","last_became_peered":"2024-03-06T07:26:16.866520+0000","last_unstale":"2024-03-06T07:26:19.632226+0000","last_undegraded":"2024-03-06T07:26:19.632226+0000","last_fullsized":"2024-03-06T07:26:19.632226+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:21.270064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632180+0000","last_change":"2024-03-06T07:26:17.863780+0000","last_active":"2024-03-06T07:26:19.632180+0000","last_peered":"2024-03-06T07:26:19.632180+0000","last_clean":"2024-03-06T07:26:19.632180+0000","last_became_active":"2024-03-06T07:26:17.863503+0000","last_became_peered":"2024-03-06T07:26:17.863503+0000","last_unstale":"2024-03-06T07:26:19.632180+0000","last_undegraded":"2024-03-06T07:26:19.632180+0000","last_fullsized":"2024-03-06T07:26:19.632180+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:43.263166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"91'6","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632748+0000","last_change":"2024-03-06T07:26:16.868702+0000","last_active":"2024-03-06T07:26:19.632748+0000","last_peered":"2024-03-06T07:26:19.632748+0000","last_clean":"2024-03-06T07:26:19.632748+0000","last_became_active":"2024-03-06T07:26:16.868364+0000","last_became_peered":"2024-03-06T07:26:16.868364+0000","last_unstale":"2024-03-06T07:26:19.632748+0000","last_undegraded":"2024-03-06T07:26:19.632748+0000","last_fullsized":"2024-03-06T07:26:19.632748+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:19.712353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855547+0000","last_change":"2024-03-06T07:26:17.871667+0000","last_active":"2024-03-06T07:26:18.855547+0000","last_peered":"2024-03-06T07:26:18.855547+0000","last_clean":"2024-03-06T07:26:18.855547+0000","last_became_active":"2024-03-06T07:26:17.871317+0000","last_became_peered":"2024-03-06T07:26:17.871317+0000","last_unstale":"2024-03-06T07:26:18.855547+0000","last_undegraded":"2024-03-06T07:26:18.855547+0000","last_fullsized":"2024-03-06T07:26:18.855547+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.528335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877108+0000","last_change":"2024-03-06T07:26:16.865944+0000","last_active":"2024-03-06T07:26:18.877108+0000","last_peered":"2024-03-06T07:26:18.877108+0000","last_clean":"2024-03-06T07:26:18.877108+0000","last_became_active":"2024-03-06T07:26:16.865672+0000","last_became_peered":"2024-03-06T07:26:16.865672+0000","last_unstale":"2024-03-06T07:26:18.877108+0000","last_undegraded":"2024-03-06T07:26:18.877108+0000","last_fullsized":"2024-03-06T07:26:18.877108+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:03.112514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854825+0000","last_change":"2024-03-06T07:26:17.867658+0000","last_active":"2024-03-06T07:26:18.854825+0000","last_peered":"2024-03-06T07:26:18.854825+0000","last_clean":"2024-03-06T07:26:18.854825+0000","last_became_active":"2024-03-06T07:26:17.867387+0000","last_became_peered":"2024-03-06T07:26:17.867387+0000","last_unstale":"2024-03-06T07:26:18.854825+0000","last_undegraded":"2024-03-06T07:26:18.854825+0000","last_fullsized":"2024-03-06T07:26:18.854825+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:14.616124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877004+0000","last_change":"2024-03-06T07:26:16.869461+0000","last_active":"2024-03-06T07:26:18.877004+0000","last_peered":"2024-03-06T07:26:18.877004+0000","last_clean":"2024-03-06T07:26:18.877004+0000","last_became_active":"2024-03-06T07:26:16.869142+0000","last_became_peered":"2024-03-06T07:26:16.869142+0000","last_unstale":"2024-03-06T07:26:18.877004+0000","last_undegraded":"2024-03-06T07:26:18.877004+0000","last_fullsized":"2024-03-06T07:26:18.877004+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:53.301305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631353+0000","last_change":"2024-03-06T07:26:17.875559+0000","last_active":"2024-03-06T07:26:19.631353+0000","last_peered":"2024-03-06T07:26:19.631353+0000","last_clean":"2024-03-06T07:26:19.631353+0000","last_became_active":"2024-03-06T07:26:17.875196+0000","last_became_peered":"2024-03-06T07:26:17.875196+0000","last_unstale":"2024-03-06T07:26:19.631353+0000","last_undegraded":"2024-03-06T07:26:19.631353+0000","last_fullsized":"2024-03-06T07:26:19.631353+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:14.616124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632305+0000","last_change":"2024-03-06T07:26:16.865532+0000","last_active":"2024-03-06T07:26:19.632305+0000","last_peered":"2024-03-06T07:26:19.632305+0000","last_clean":"2024-03-06T07:26:19.632305+0000","last_became_active":"2024-03-06T07:26:16.865038+0000","last_became_peered":"2024-03-06T07:26:16.865038+0000","last_unstale":"2024-03-06T07:26:19.632305+0000","last_undegraded":"2024-03-06T07:26:19.632305+0000","last_fullsized":"2024-03-06T07:26:19.632305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:18.180662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876751+0000","last_change":"2024-03-06T07:26:17.875784+0000","last_active":"2024-03-06T07:26:18.876751+0000","last_peered":"2024-03-06T07:26:18.876751+0000","last_clean":"2024-03-06T07:26:18.876751+0000","last_became_active":"2024-03-06T07:26:17.875509+0000","last_became_peered":"2024-03-06T07:26:17.875509+0000","last_unstale":"2024-03-06T07:26:18.876751+0000","last_undegraded":"2024-03-06T07:26:18.876751+0000","last_fullsized":"2024-03-06T07:26:18.876751+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:39.164839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"91'8","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854323+0000","last_change":"2024-03-06T07:26:16.866435+0000","last_active":"2024-03-06T07:26:18.854323+0000","last_peered":"2024-03-06T07:26:18.854323+0000","last_clean":"2024-03-06T07:26:18.854323+0000","last_became_active":"2024-03-06T07:26:16.866136+0000","last_became_peered":"2024-03-06T07:26:16.866136+0000","last_unstale":"2024-03-06T07:26:18.854323+0000","last_undegraded":"2024-03-06T07:26:18.854323+0000","last_fullsized":"2024-03-06T07:26:18.854323+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:18.526671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632569+0000","last_change":"2024-03-06T07:26:16.865144+0000","last_active":"2024-03-06T07:26:19.632569+0000","last_peered":"2024-03-06T07:26:19.632569+0000","last_clean":"2024-03-06T07:26:19.632569+0000","last_became_active":"2024-03-06T07:26:16.864840+0000","last_became_peered":"2024-03-06T07:26:16.864840+0000","last_unstale":"2024-03-06T07:26:19.632569+0000","last_undegraded":"2024-03-06T07:26:19.632569+0000","last_fullsized":"2024-03-06T07:26:19.632569+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:14.718388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856150+0000","last_change":"2024-03-06T07:26:17.867621+0000","last_active":"2024-03-06T07:26:18.856150+0000","last_peered":"2024-03-06T07:26:18.856150+0000","last_clean":"2024-03-06T07:26:18.856150+0000","last_became_active":"2024-03-06T07:26:17.867337+0000","last_became_peered":"2024-03-06T07:26:17.867337+0000","last_unstale":"2024-03-06T07:26:18.856150+0000","last_undegraded":"2024-03-06T07:26:18.856150+0000","last_fullsized":"2024-03-06T07:26:18.856150+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:20.717162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850317+0000","last_change":"2024-03-06T07:26:15.850317+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850317+0000","last_undegraded":"2024-03-06T07:26:15.850317+0000","last_fullsized":"2024-03-06T07:26:15.850317+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632234+0000","last_change":"2024-03-06T07:26:17.873920+0000","last_active":"2024-03-06T07:26:19.632234+0000","last_peered":"2024-03-06T07:26:19.632234+0000","last_clean":"2024-03-06T07:26:19.632234+0000","last_became_active":"2024-03-06T07:26:17.873419+0000","last_became_peered":"2024-03-06T07:26:17.873419+0000","last_unstale":"2024-03-06T07:26:19.632234+0000","last_undegraded":"2024-03-06T07:26:19.632234+0000","last_fullsized":"2024-03-06T07:26:19.632234+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:22.063664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874737+0000","last_change":"2024-03-06T07:26:16.864486+0000","last_active":"2024-03-06T07:26:18.874737+0000","last_peered":"2024-03-06T07:26:18.874737+0000","last_clean":"2024-03-06T07:26:18.874737+0000","last_became_active":"2024-03-06T07:26:16.862774+0000","last_became_peered":"2024-03-06T07:26:16.862774+0000","last_unstale":"2024-03-06T07:26:18.874737+0000","last_undegraded":"2024-03-06T07:26:18.874737+0000","last_fullsized":"2024-03-06T07:26:18.874737+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:58.165611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876950+0000","last_change":"2024-03-06T07:26:17.873966+0000","last_active":"2024-03-06T07:26:18.876950+0000","last_peered":"2024-03-06T07:26:18.876950+0000","last_clean":"2024-03-06T07:26:18.876950+0000","last_became_active":"2024-03-06T07:26:17.873702+0000","last_became_peered":"2024-03-06T07:26:17.873702+0000","last_unstale":"2024-03-06T07:26:18.876950+0000","last_undegraded":"2024-03-06T07:26:18.876950+0000","last_fullsized":"2024-03-06T07:26:18.876950+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:47.962109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874691+0000","last_change":"2024-03-06T07:26:16.866599+0000","last_active":"2024-03-06T07:26:18.874691+0000","last_peered":"2024-03-06T07:26:18.874691+0000","last_clean":"2024-03-06T07:26:18.874691+0000","last_became_active":"2024-03-06T07:26:16.866342+0000","last_became_peered":"2024-03-06T07:26:16.866342+0000","last_unstale":"2024-03-06T07:26:18.874691+0000","last_undegraded":"2024-03-06T07:26:18.874691+0000","last_fullsized":"2024-03-06T07:26:18.874691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:30.398774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877178+0000","last_change":"2024-03-06T07:26:17.860602+0000","last_active":"2024-03-06T07:26:18.877178+0000","last_peered":"2024-03-06T07:26:18.877178+0000","last_clean":"2024-03-06T07:26:18.877178+0000","last_became_active":"2024-03-06T07:26:17.860333+0000","last_became_peered":"2024-03-06T07:26:17.860333+0000","last_unstale":"2024-03-06T07:26:18.877178+0000","last_undegraded":"2024-03-06T07:26:18.877178+0000","last_fullsized":"2024-03-06T07:26:18.877178+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:45.077065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856177+0000","last_change":"2024-03-06T07:26:16.868618+0000","last_active":"2024-03-06T07:26:18.856177+0000","last_peered":"2024-03-06T07:26:18.856177+0000","last_clean":"2024-03-06T07:26:18.856177+0000","last_became_active":"2024-03-06T07:26:16.868180+0000","last_became_peered":"2024-03-06T07:26:16.868180+0000","last_unstale":"2024-03-06T07:26:18.856177+0000","last_undegraded":"2024-03-06T07:26:18.856177+0000","last_fullsized":"2024-03-06T07:26:18.856177+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:26.488466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876708+0000","last_change":"2024-03-06T07:26:17.868580+0000","last_active":"2024-03-06T07:26:18.876708+0000","last_peered":"2024-03-06T07:26:18.876708+0000","last_clean":"2024-03-06T07:26:18.876708+0000","last_became_active":"2024-03-06T07:26:17.868280+0000","last_became_peered":"2024-03-06T07:26:17.868280+0000","last_unstale":"2024-03-06T07:26:18.876708+0000","last_undegraded":"2024-03-06T07:26:18.876708+0000","last_fullsized":"2024-03-06T07:26:18.876708+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:43.974866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855956+0000","last_change":"2024-03-06T07:26:16.868344+0000","last_active":"2024-03-06T07:26:18.855956+0000","last_peered":"2024-03-06T07:26:18.855956+0000","last_clean":"2024-03-06T07:26:18.855956+0000","last_became_active":"2024-03-06T07:26:16.868005+0000","last_became_peered":"2024-03-06T07:26:16.868005+0000","last_unstale":"2024-03-06T07:26:18.855956+0000","last_undegraded":"2024-03-06T07:26:18.855956+0000","last_fullsized":"2024-03-06T07:26:18.855956+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:57.303182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874532+0000","last_change":"2024-03-06T07:26:16.860891+0000","last_active":"2024-03-06T07:26:18.874532+0000","last_peered":"2024-03-06T07:26:18.874532+0000","last_clean":"2024-03-06T07:26:18.874532+0000","last_became_active":"2024-03-06T07:26:16.860523+0000","last_became_peered":"2024-03-06T07:26:16.860523+0000","last_unstale":"2024-03-06T07:26:18.874532+0000","last_undegraded":"2024-03-06T07:26:18.874532+0000","last_fullsized":"2024-03-06T07:26:18.874532+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:21.270064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875146+0000","last_change":"2024-03-06T07:26:17.868058+0000","last_active":"2024-03-06T07:26:18.875146+0000","last_peered":"2024-03-06T07:26:18.875146+0000","last_clean":"2024-03-06T07:26:18.875146+0000","last_became_active":"2024-03-06T07:26:17.867378+0000","last_became_peered":"2024-03-06T07:26:17.867378+0000","last_unstale":"2024-03-06T07:26:18.875146+0000","last_undegraded":"2024-03-06T07:26:18.875146+0000","last_fullsized":"2024-03-06T07:26:18.875146+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:40.439617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856296+0000","last_change":"2024-03-06T07:26:16.863938+0000","last_active":"2024-03-06T07:26:18.856296+0000","last_peered":"2024-03-06T07:26:18.856296+0000","last_clean":"2024-03-06T07:26:18.856296+0000","last_became_active":"2024-03-06T07:26:16.863664+0000","last_became_peered":"2024-03-06T07:26:16.863664+0000","last_unstale":"2024-03-06T07:26:18.856296+0000","last_undegraded":"2024-03-06T07:26:18.856296+0000","last_fullsized":"2024-03-06T07:26:18.856296+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:18.526671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854824+0000","last_change":"2024-03-06T07:26:17.872137+0000","last_active":"2024-03-06T07:26:18.854824+0000","last_peered":"2024-03-06T07:26:18.854824+0000","last_clean":"2024-03-06T07:26:18.854824+0000","last_became_active":"2024-03-06T07:26:17.871867+0000","last_became_peered":"2024-03-06T07:26:17.871867+0000","last_unstale":"2024-03-06T07:26:18.854824+0000","last_undegraded":"2024-03-06T07:26:18.854824+0000","last_fullsized":"2024-03-06T07:26:18.854824+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:31.403582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"92'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875144+0000","last_change":"2024-03-06T07:26:16.870823+0000","last_active":"2024-03-06T07:26:18.875144+0000","last_peered":"2024-03-06T07:26:18.875144+0000","last_clean":"2024-03-06T07:26:18.875144+0000","last_became_active":"2024-03-06T07:26:16.870507+0000","last_became_peered":"2024-03-06T07:26:16.870507+0000","last_unstale":"2024-03-06T07:26:18.875144+0000","last_undegraded":"2024-03-06T07:26:18.875144+0000","last_fullsized":"2024-03-06T07:26:18.875144+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:35.309600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854500+0000","last_change":"2024-03-06T07:26:17.880876+0000","last_active":"2024-03-06T07:26:18.854500+0000","last_peered":"2024-03-06T07:26:18.854500+0000","last_clean":"2024-03-06T07:26:18.854500+0000","last_became_active":"2024-03-06T07:26:17.880610+0000","last_became_peered":"2024-03-06T07:26:17.880610+0000","last_unstale":"2024-03-06T07:26:18.854500+0000","last_undegraded":"2024-03-06T07:26:18.854500+0000","last_fullsized":"2024-03-06T07:26:18.854500+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:22.274872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874842+0000","last_change":"2024-03-06T07:26:16.864709+0000","last_active":"2024-03-06T07:26:18.874842+0000","last_peered":"2024-03-06T07:26:18.874842+0000","last_clean":"2024-03-06T07:26:18.874842+0000","last_became_active":"2024-03-06T07:26:16.863053+0000","last_became_peered":"2024-03-06T07:26:16.863053+0000","last_unstale":"2024-03-06T07:26:18.874842+0000","last_undegraded":"2024-03-06T07:26:18.874842+0000","last_fullsized":"2024-03-06T07:26:18.874842+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:26.399377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850394+0000","last_change":"2024-03-06T07:26:15.850394+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850394+0000","last_undegraded":"2024-03-06T07:26:15.850394+0000","last_fullsized":"2024-03-06T07:26:15.850394+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877061+0000","last_change":"2024-03-06T07:26:17.876580+0000","last_active":"2024-03-06T07:26:18.877061+0000","last_peered":"2024-03-06T07:26:18.877061+0000","last_clean":"2024-03-06T07:26:18.877061+0000","last_became_active":"2024-03-06T07:26:17.876316+0000","last_became_peered":"2024-03-06T07:26:17.876316+0000","last_unstale":"2024-03-06T07:26:18.877061+0000","last_undegraded":"2024-03-06T07:26:18.877061+0000","last_fullsized":"2024-03-06T07:26:18.877061+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:53.155139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874794+0000","last_change":"2024-03-06T07:26:16.870873+0000","last_active":"2024-03-06T07:26:18.874794+0000","last_peered":"2024-03-06T07:26:18.874794+0000","last_clean":"2024-03-06T07:26:18.874794+0000","last_became_active":"2024-03-06T07:26:16.870581+0000","last_became_peered":"2024-03-06T07:26:16.870581+0000","last_unstale":"2024-03-06T07:26:18.874794+0000","last_undegraded":"2024-03-06T07:26:18.874794+0000","last_fullsized":"2024-03-06T07:26:18.874794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:40.436083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632601+0000","last_change":"2024-03-06T07:26:17.880256+0000","last_active":"2024-03-06T07:26:19.632601+0000","last_peered":"2024-03-06T07:26:19.632601+0000","last_clean":"2024-03-06T07:26:19.632601+0000","last_became_active":"2024-03-06T07:26:17.879582+0000","last_became_peered":"2024-03-06T07:26:17.879582+0000","last_unstale":"2024-03-06T07:26:19.632601+0000","last_undegraded":"2024-03-06T07:26:19.632601+0000","last_fullsized":"2024-03-06T07:26:19.632601+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:07.715401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850379+0000","last_change":"2024-03-06T07:26:15.850379+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850379+0000","last_undegraded":"2024-03-06T07:26:15.850379+0000","last_fullsized":"2024-03-06T07:26:15.850379+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876708+0000","last_change":"2024-03-06T07:26:17.875415+0000","last_active":"2024-03-06T07:26:18.876708+0000","last_peered":"2024-03-06T07:26:18.876708+0000","last_clean":"2024-03-06T07:26:18.876708+0000","last_became_active":"2024-03-06T07:26:17.875138+0000","last_became_peered":"2024-03-06T07:26:17.875138+0000","last_unstale":"2024-03-06T07:26:18.876708+0000","last_undegraded":"2024-03-06T07:26:18.876708+0000","last_fullsized":"2024-03-06T07:26:18.876708+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:27.427674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855626+0000","last_change":"2024-03-06T07:26:16.871839+0000","last_active":"2024-03-06T07:26:18.855626+0000","last_peered":"2024-03-06T07:26:18.855626+0000","last_clean":"2024-03-06T07:26:18.855626+0000","last_became_active":"2024-03-06T07:26:16.871570+0000","last_became_peered":"2024-03-06T07:26:16.871570+0000","last_unstale":"2024-03-06T07:26:18.855626+0000","last_undegraded":"2024-03-06T07:26:18.855626+0000","last_fullsized":"2024-03-06T07:26:18.855626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:06.710593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876761+0000","last_change":"2024-03-06T07:26:16.866300+0000","last_active":"2024-03-06T07:26:18.876761+0000","last_peered":"2024-03-06T07:26:18.876761+0000","last_clean":"2024-03-06T07:26:18.876761+0000","last_became_active":"2024-03-06T07:26:16.866014+0000","last_became_peered":"2024-03-06T07:26:16.866014+0000","last_unstale":"2024-03-06T07:26:18.876761+0000","last_undegraded":"2024-03-06T07:26:18.876761+0000","last_fullsized":"2024-03-06T07:26:18.876761+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:31.226660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877031+0000","last_change":"2024-03-06T07:26:17.874843+0000","last_active":"2024-03-06T07:26:18.877031+0000","last_peered":"2024-03-06T07:26:18.877031+0000","last_clean":"2024-03-06T07:26:18.877031+0000","last_became_active":"2024-03-06T07:26:17.874530+0000","last_became_peered":"2024-03-06T07:26:17.874530+0000","last_unstale":"2024-03-06T07:26:18.877031+0000","last_undegraded":"2024-03-06T07:26:18.877031+0000","last_fullsized":"2024-03-06T07:26:18.877031+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:22.620099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631513+0000","last_change":"2024-03-06T07:26:16.870244+0000","last_active":"2024-03-06T07:26:19.631513+0000","last_peered":"2024-03-06T07:26:19.631513+0000","last_clean":"2024-03-06T07:26:19.631513+0000","last_became_active":"2024-03-06T07:26:16.869996+0000","last_became_peered":"2024-03-06T07:26:16.869996+0000","last_unstale":"2024-03-06T07:26:19.631513+0000","last_undegraded":"2024-03-06T07:26:19.631513+0000","last_fullsized":"2024-03-06T07:26:19.631513+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:46.843920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":301,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631596+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:26:19.631596+0000","last_peered":"2024-03-06T07:26:19.631596+0000","last_clean":"2024-03-06T07:26:19.631596+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:26:19.631596+0000","last_undegraded":"2024-03-06T07:26:19.631596+0000","last_fullsized":"2024-03-06T07:26:19.631596+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":110,"acting":110,"num_store_stats":7},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2340,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":1076018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1101824,"data_stored":466188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1142784,"data_stored":485608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1159168,"data_stored":487274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1155072,"data_stored":487172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1142784,"data_stored":485572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1138688,"data_stored":485538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1736704,"data_stored":1076488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:21.619 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3b is not active+clean 2024-03-06T07:26:21.619 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:16.843320+0000', 'last_change': '2024-03-06T07:26:16.843320+0000', 'last_active': '2024-03-06T07:26:16.843320+0000', 'last_peered': '2024-03-06T07:26:16.843320+0000', 'last_clean': '2024-03-06T07:26:16.843320+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:16.843320+0000', 'last_undegraded': '2024-03-06T07:26:16.843320+0000', 'last_fullsized': '2024-03-06T07:26:16.843320+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:21.619 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.36 is not active+clean 2024-03-06T07:26:21.619 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:16.843320+0000', 'last_change': '2024-03-06T07:26:16.843320+0000', 'last_active': '2024-03-06T07:26:16.843320+0000', 'last_peered': '2024-03-06T07:26:16.843320+0000', 'last_clean': '2024-03-06T07:26:16.843320+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:16.843320+0000', 'last_undegraded': '2024-03-06T07:26:16.843320+0000', 'last_fullsized': '2024-03-06T07:26:16.843320+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:21.619 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.27 is not active+clean 2024-03-06T07:26:21.619 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:16.843320+0000', 'last_change': '2024-03-06T07:26:16.843320+0000', 'last_active': '2024-03-06T07:26:16.843320+0000', 'last_peered': '2024-03-06T07:26:16.843320+0000', 'last_clean': '2024-03-06T07:26:16.843320+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:16.843320+0000', 'last_undegraded': '2024-03-06T07:26:16.843320+0000', 'last_fullsized': '2024-03-06T07:26:16.843320+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:21.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.24 is not active+clean 2024-03-06T07:26:21.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:16.843320+0000', 'last_change': '2024-03-06T07:26:16.843320+0000', 'last_active': '2024-03-06T07:26:16.843320+0000', 'last_peered': '2024-03-06T07:26:16.843320+0000', 'last_clean': '2024-03-06T07:26:16.843320+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:16.843320+0000', 'last_undegraded': '2024-03-06T07:26:16.843320+0000', 'last_fullsized': '2024-03-06T07:26:16.843320+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:21.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.19 is not active+clean 2024-03-06T07:26:21.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 89, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:15.850220+0000', 'last_change': '2024-03-06T07:26:15.850220+0000', 'last_active': '2024-03-06T07:26:15.838511+0000', 'last_peered': '2024-03-06T07:26:15.838511+0000', 'last_clean': '2024-03-06T07:26:15.838511+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:15.850220+0000', 'last_undegraded': '2024-03-06T07:26:15.850220+0000', 'last_fullsized': '2024-03-06T07:26:15.850220+0000', 'mapping_epoch': 89, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 89, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:21.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.14 is not active+clean 2024-03-06T07:26:21.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:16.843320+0000', 'last_change': '2024-03-06T07:26:16.843320+0000', 'last_active': '2024-03-06T07:26:16.843320+0000', 'last_peered': '2024-03-06T07:26:16.843320+0000', 'last_clean': '2024-03-06T07:26:16.843320+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:16.843320+0000', 'last_undegraded': '2024-03-06T07:26:16.843320+0000', 'last_fullsized': '2024-03-06T07:26:16.843320+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:21.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.f is not active+clean 2024-03-06T07:26:21.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:16.843320+0000', 'last_change': '2024-03-06T07:26:16.843320+0000', 'last_active': '2024-03-06T07:26:16.843320+0000', 'last_peered': '2024-03-06T07:26:16.843320+0000', 'last_clean': '2024-03-06T07:26:16.843320+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:16.843320+0000', 'last_undegraded': '2024-03-06T07:26:16.843320+0000', 'last_fullsized': '2024-03-06T07:26:16.843320+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:21.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.d is not active+clean 2024-03-06T07:26:21.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 89, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:15.850317+0000', 'last_change': '2024-03-06T07:26:15.850317+0000', 'last_active': '2024-03-06T07:26:15.838511+0000', 'last_peered': '2024-03-06T07:26:15.838511+0000', 'last_clean': '2024-03-06T07:26:15.838511+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:15.850317+0000', 'last_undegraded': '2024-03-06T07:26:15.850317+0000', 'last_fullsized': '2024-03-06T07:26:15.850317+0000', 'mapping_epoch': 89, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 89, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:21.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.9 is not active+clean 2024-03-06T07:26:21.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:16.843320+0000', 'last_change': '2024-03-06T07:26:16.843320+0000', 'last_active': '2024-03-06T07:26:16.843320+0000', 'last_peered': '2024-03-06T07:26:16.843320+0000', 'last_clean': '2024-03-06T07:26:16.843320+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:16.843320+0000', 'last_undegraded': '2024-03-06T07:26:16.843320+0000', 'last_fullsized': '2024-03-06T07:26:16.843320+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:21.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.5 is not active+clean 2024-03-06T07:26:21.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:16.843320+0000', 'last_change': '2024-03-06T07:26:16.843320+0000', 'last_active': '2024-03-06T07:26:16.843320+0000', 'last_peered': '2024-03-06T07:26:16.843320+0000', 'last_clean': '2024-03-06T07:26:16.843320+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:16.843320+0000', 'last_undegraded': '2024-03-06T07:26:16.843320+0000', 'last_fullsized': '2024-03-06T07:26:16.843320+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:21.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.4 is not active+clean 2024-03-06T07:26:21.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 89, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:15.850394+0000', 'last_change': '2024-03-06T07:26:15.850394+0000', 'last_active': '2024-03-06T07:26:15.838511+0000', 'last_peered': '2024-03-06T07:26:15.838511+0000', 'last_clean': '2024-03-06T07:26:15.838511+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:15.850394+0000', 'last_undegraded': '2024-03-06T07:26:15.850394+0000', 'last_fullsized': '2024-03-06T07:26:15.850394+0000', 'mapping_epoch': 89, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 89, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:21.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2 is not active+clean 2024-03-06T07:26:21.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 89, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:26:15.850379+0000', 'last_change': '2024-03-06T07:26:15.850379+0000', 'last_active': '2024-03-06T07:26:15.838511+0000', 'last_peered': '2024-03-06T07:26:15.838511+0000', 'last_clean': '2024-03-06T07:26:15.838511+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:15.850379+0000', 'last_undegraded': '2024-03-06T07:26:15.850379+0000', 'last_fullsized': '2024-03-06T07:26:15.850379+0000', 'mapping_epoch': 89, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 89, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:15.838511+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:26:21.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1 is not active+clean 2024-03-06T07:26:21.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:26:16.843320+0000', 'last_change': '2024-03-06T07:26:16.843320+0000', 'last_active': '2024-03-06T07:26:16.843320+0000', 'last_peered': '2024-03-06T07:26:16.843320+0000', 'last_clean': '2024-03-06T07:26:16.843320+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:26:16.843320+0000', 'last_undegraded': '2024-03-06T07:26:16.843320+0000', 'last_fullsized': '2024-03-06T07:26:16.843320+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'last_clean_scrub_stamp': '2024-03-06T07:26:16.843320+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:26:21.622 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:26:21.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.754+0000 7f96cb4b5640 1 Processor -- start 2024-03-06T07:26:21.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.754+0000 7f96cb4b5640 1 -- start start 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96cb4b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 0x7f96c406c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96cb4b5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f96c415b380 0x7f96c40708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96cb4b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f96c406cb90 0x7f96c4070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96cb4b5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c406d030 con 0x7f96c415b380 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96cb4b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c406d310 con 0x7f96c406cb90 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96cb4b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c406d5f0 con 0x7f96c4158880 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c8a50640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f96c415b380 0x7f96c40708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c9251640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 0x7f96c406c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c9251640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 0x7f96c406c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:54446/0 (socket says 172.21.15.59:54446) 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c8a50640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f96c415b380 0x7f96c40708d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47482/0 (socket says 172.21.15.59:47482) 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c9251640 1 -- 172.21.15.59:0/2648267082 learned_addr learned my addr 172.21.15.59:0/2648267082 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c9a52640 1 --2- 172.21.15.59:0/2648267082 >> v2:172.21.15.164:3300/0 conn(0x7f96c406cb90 0x7f96c4070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c9251640 1 -- 172.21.15.59:0/2648267082 >> v2:172.21.15.164:3300/0 conn(0x7f96c406cb90 msgr2=0x7f96c4070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c9251640 1 --2- 172.21.15.59:0/2648267082 >> v2:172.21.15.164:3300/0 conn(0x7f96c406cb90 0x7f96c4070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c9251640 1 -- 172.21.15.59:0/2648267082 >> v2:172.21.15.59:3300/0 conn(0x7f96c415b380 msgr2=0x7f96c40708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c9251640 1 --2- 172.21.15.59:0/2648267082 >> v2:172.21.15.59:3300/0 conn(0x7f96c415b380 0x7f96c40708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c9251640 1 -- 172.21.15.59:0/2648267082 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c406d8d0 con 0x7f96c4158880 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c8a50640 1 --2- 172.21.15.59:0/2648267082 >> v2:172.21.15.59:3300/0 conn(0x7f96c415b380 0x7f96c40708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.755+0000 7f96c9a52640 1 --2- 172.21.15.59:0/2648267082 >> v2:172.21.15.164:3300/0 conn(0x7f96c406cb90 0x7f96c4070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.756+0000 7f96c9251640 1 --2- 172.21.15.59:0/2648267082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 0x7f96c406c650 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f96c4071b80 tx=0x7f96c002e0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bac4548974059903 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.756+0000 7f96bbfff640 1 -- 172.21.15.59:0/2648267082 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c003b070 con 0x7f96c4158880 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.756+0000 7f96bbfff640 1 -- 172.21.15.59:0/2648267082 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c0036440 con 0x7f96c4158880 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.756+0000 7f96bbfff640 1 -- 172.21.15.59:0/2648267082 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0036810 con 0x7f96c4158880 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.757+0000 7f96cb4b5640 1 -- 172.21.15.59:0/2648267082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 msgr2=0x7f96c406c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.757+0000 7f96cb4b5640 1 --2- 172.21.15.59:0/2648267082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 0x7f96c406c650 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f96c4071b80 tx=0x7f96c002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.757+0000 7f96cb4b5640 1 -- 172.21.15.59:0/2648267082 shutdown_connections 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.757+0000 7f96cb4b5640 1 --2- 172.21.15.59:0/2648267082 >> v2:172.21.15.164:3300/0 conn(0x7f96c406cb90 0x7f96c4070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.757+0000 7f96cb4b5640 1 --2- 172.21.15.59:0/2648267082 >> v2:172.21.15.59:3300/0 conn(0x7f96c415b380 0x7f96c40708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.757+0000 7f96cb4b5640 1 --2- 172.21.15.59:0/2648267082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 0x7f96c406c650 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.757+0000 7f96cb4b5640 1 -- 172.21.15.59:0/2648267082 >> 172.21.15.59:0/2648267082 conn(0x7f96c40b1c80 msgr2=0x7f96c40cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.757+0000 7f96cb4b5640 1 -- 172.21.15.59:0/2648267082 shutdown_connections 2024-03-06T07:26:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.757+0000 7f96cb4b5640 1 -- 172.21.15.59:0/2648267082 wait complete. 2024-03-06T07:26:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.758+0000 7f96cb4b5640 1 Processor -- start 2024-03-06T07:26:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.758+0000 7f96cb4b5640 1 -- start start 2024-03-06T07:26:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96cb4b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c406cb90 0x7f96c40ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96cb4b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 0x7f96c40c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96cb4b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96c415b380 0x7f96c40c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96cb4b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c415e980 con 0x7f96c415b380 2024-03-06T07:26:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96cb4b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c415e800 con 0x7f96c406cb90 2024-03-06T07:26:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96cb4b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c415eb00 con 0x7f96c4158880 2024-03-06T07:26:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96c9a52640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96c415b380 0x7f96c40c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96c9a52640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96c415b380 0x7f96c40c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47494/0 (socket says 172.21.15.59:47494) 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96c9251640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c406cb90 0x7f96c40ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96c9a52640 1 -- 172.21.15.59:0/375027892 learned_addr learned my addr 172.21.15.59:0/375027892 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96c8a50640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 0x7f96c40c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96c9a52640 1 -- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 msgr2=0x7f96c40c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96c9a52640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 0x7f96c40c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96c9a52640 1 -- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c406cb90 msgr2=0x7f96c40ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96c9a52640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c406cb90 0x7f96c40ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96c9a52640 1 -- 172.21.15.59:0/375027892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c40c4c00 con 0x7f96c415b380 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96c9a52640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96c415b380 0x7f96c40c6620 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00eed0 tx=0x7f96bc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96b9ffb640 1 -- 172.21.15.59:0/375027892 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc018030 con 0x7f96c415b380 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96b9ffb640 1 -- 172.21.15.59:0/375027892 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96bc00bd20 con 0x7f96c415b380 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96cb4b5640 1 -- 172.21.15.59:0/375027892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c40c4ee0 con 0x7f96c415b380 2024-03-06T07:26:21.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.759+0000 7f96b9ffb640 1 -- 172.21.15.59:0/375027892 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc003bb0 con 0x7f96c415b380 2024-03-06T07:26:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.760+0000 7f96cb4b5640 1 -- 172.21.15.59:0/375027892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c40c6e60 con 0x7f96c415b380 2024-03-06T07:26:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.760+0000 7f96cb4b5640 1 -- 172.21.15.59:0/375027892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c40c50b0 con 0x7f96c415b380 2024-03-06T07:26:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.761+0000 7f96b9ffb640 1 -- 172.21.15.59:0/375027892 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96bc01b070 con 0x7f96c415b380 2024-03-06T07:26:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.761+0000 7f96b9ffb640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f96a806b4d0 0x7f96a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.761+0000 7f96b9ffb640 1 -- 172.21.15.59:0/375027892 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96bc08a570 con 0x7f96c415b380 2024-03-06T07:26:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.761+0000 7f96c9251640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f96a806b4d0 0x7f96a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.762+0000 7f96c9251640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f96a806b4d0 0x7f96a806d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f96c0038000 tx=0x7f96c0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:21.766+0000 7f96b9ffb640 1 -- 172.21.15.59:0/375027892 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96bc058e30 con 0x7f96c415b380 2024-03-06T07:26:22.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.021+0000 7f96cb4b5640 1 -- 172.21.15.59:0/375027892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f96c404f330 con 0x7f96c415b380 2024-03-06T07:26:22.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.021+0000 7f96b9ffb640 1 -- 172.21.15.59:0/375027892 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1594 (secure 0 0 0) 0x7f96bc05ced0 con 0x7f96c415b380 2024-03-06T07:26:22.023 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:22.023 INFO:teuthology.orchestra.run.smithi059.stdout:{"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":291,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":92,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709701,"num_in_osds":8,"osd_in_since":1709709692,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":84},{"state_name":"unknown","count":9},{"state_name":"creating+peering","count":4}],"num_pgs":97,"num_pools":3,"num_objects":23,"data_bytes":592708,"bytes_used":265781248,"bytes_avail":772828332032,"bytes_total":773094113280,"unknown_pgs_ratio":0.092783503234386444,"inactive_pgs_ratio":0.041237112134695053,"write_bytes_sec":2413,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":49,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"d","status":"up:active","gid":7422}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-03-06T07:25:35.348800+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:26:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.023+0000 7f96cb4b5640 1 -- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f96a806b4d0 msgr2=0x7f96a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.023+0000 7f96cb4b5640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f96a806b4d0 0x7f96a806d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f96c0038000 tx=0x7f96c0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.023+0000 7f96cb4b5640 1 -- 172.21.15.59:0/375027892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96c415b380 msgr2=0x7f96c40c6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.023+0000 7f96cb4b5640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96c415b380 0x7f96c40c6620 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00eed0 tx=0x7f96bc00b480 comp rx=0 tx=0).stop 2024-03-06T07:26:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.023+0000 7f96cb4b5640 1 -- 172.21.15.59:0/375027892 shutdown_connections 2024-03-06T07:26:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.023+0000 7f96cb4b5640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f96a806b4d0 0x7f96a806d990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.023+0000 7f96cb4b5640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96c415b380 0x7f96c40c6620 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.023+0000 7f96cb4b5640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96c4158880 0x7f96c40c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.023+0000 7f96cb4b5640 1 --2- 172.21.15.59:0/375027892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96c406cb90 0x7f96c40ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.023+0000 7f96cb4b5640 1 -- 172.21.15.59:0/375027892 >> 172.21.15.59:0/375027892 conn(0x7f96c40b1c80 msgr2=0x7f96c4157480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.024+0000 7f96cb4b5640 1 -- 172.21.15.59:0/375027892 shutdown_connections 2024-03-06T07:26:22.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.024+0000 7f96cb4b5640 1 -- 172.21.15.59:0/375027892 wait complete. 2024-03-06T07:26:22.036 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:26:22.036 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:22.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.136+0000 7f6a7d82a640 1 Processor -- start 2024-03-06T07:26:22.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.136+0000 7f6a7d82a640 1 -- start start 2024-03-06T07:26:22.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a7d82a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a78166af0 0x7f6a78168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a7d82a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7815e380 con 0x7f6a780d00c0 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a7d82a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a780cdc10 con 0x7f6a780cf850 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a7d82a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7806ca90 con 0x7f6a78166af0 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a76ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6a780d00c0 0x7f6a780d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34408/0 (socket says 172.21.15.59:34408) 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a76ffd640 1 -- 172.21.15.59:0/343550863 learned_addr learned my addr 172.21.15.59:0/343550863 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a777fe640 1 --2- 172.21.15.59:0/343550863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a78166af0 0x7f6a78168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a75ffb640 1 -- 172.21.15.59:0/343550863 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060120847 0 0) 0x7f6a7815e380 con 0x7f6a780d00c0 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a75ffb640 1 -- 172.21.15.59:0/343550863 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a60003520 con 0x7f6a780d00c0 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a75ffb640 1 -- 172.21.15.59:0/343550863 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995360636 0 0) 0x7f6a780cdc10 con 0x7f6a780cf850 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a75ffb640 1 -- 172.21.15.59:0/343550863 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a7815e380 con 0x7f6a780cf850 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a777fe640 1 -- 172.21.15.59:0/343550863 >> v1:172.21.15.164:6789/0 conn(0x7f6a780cf850 legacy=0x7f6a780cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a777fe640 1 -- 172.21.15.59:0/343550863 >> v1:172.21.15.59:6789/0 conn(0x7f6a780d00c0 legacy=0x7f6a780d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.137+0000 7f6a777fe640 1 -- 172.21.15.59:0/343550863 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7806c160 con 0x7f6a78166af0 2024-03-06T07:26:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.138+0000 7f6a777fe640 1 --2- 172.21.15.59:0/343550863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a78166af0 0x7f6a78168ee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009dc0 tx=0x7f6a7002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4ff81fa3161dad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.138+0000 7f6a75ffb640 1 -- 172.21.15.59:0/343550863 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a70039070 con 0x7f6a78166af0 2024-03-06T07:26:22.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.138+0000 7f6a75ffb640 1 -- 172.21.15.59:0/343550863 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7002ccd0 con 0x7f6a78166af0 2024-03-06T07:26:22.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.138+0000 7f6a75ffb640 1 -- 172.21.15.59:0/343550863 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a70034660 con 0x7f6a78166af0 2024-03-06T07:26:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.138+0000 7f6a7d82a640 1 -- 172.21.15.59:0/343550863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a78166af0 msgr2=0x7f6a78168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.138+0000 7f6a7d82a640 1 --2- 172.21.15.59:0/343550863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a78166af0 0x7f6a78168ee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009dc0 tx=0x7f6a7002bf20 comp rx=0 tx=0).stop 2024-03-06T07:26:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.139+0000 7f6a7d82a640 1 -- 172.21.15.59:0/343550863 shutdown_connections 2024-03-06T07:26:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.139+0000 7f6a7d82a640 1 --2- 172.21.15.59:0/343550863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a78166af0 0x7f6a78168ee0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.139+0000 7f6a7d82a640 1 -- 172.21.15.59:0/343550863 >> 172.21.15.59:0/343550863 conn(0x7f6a780b35a0 msgr2=0x7f6a780b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.139+0000 7f6a7d82a640 1 -- 172.21.15.59:0/343550863 shutdown_connections 2024-03-06T07:26:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.139+0000 7f6a7d82a640 1 -- 172.21.15.59:0/343550863 wait complete. 2024-03-06T07:26:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.140+0000 7f6a7d82a640 1 Processor -- start 2024-03-06T07:26:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.140+0000 7f6a7d82a640 1 -- start start 2024-03-06T07:26:22.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a7d82a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a780cf850 0x7f6a7814f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a7d82a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a780d00c0 0x7f6a78159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a7d82a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a78166af0 0x7f6a7814f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a7d82a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7806c720 con 0x7f6a780cf850 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a7d82a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7806df30 con 0x7f6a780d00c0 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a7d82a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a780b1030 con 0x7f6a78166af0 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a76ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a780cf850 0x7f6a7814f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a76ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a780cf850 0x7f6a7814f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:47508/0 (socket says 172.21.15.59:47508) 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a76ffd640 1 -- 172.21.15.59:0/2182457734 learned_addr learned my addr 172.21.15.59:0/2182457734 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a767fc640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a780d00c0 0x7f6a78159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a777fe640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a78166af0 0x7f6a7814f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a76ffd640 1 -- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a78166af0 msgr2=0x7f6a7814f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a76ffd640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a78166af0 0x7f6a7814f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a76ffd640 1 -- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a780d00c0 msgr2=0x7f6a78159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a76ffd640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a780d00c0 0x7f6a78159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a76ffd640 1 -- 172.21.15.59:0/2182457734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a780c7730 con 0x7f6a780cf850 2024-03-06T07:26:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a76ffd640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a780cf850 0x7f6a7814f6e0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800c530 tx=0x7f6a6800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a5ffff640 1 -- 172.21.15.59:0/2182457734 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68017070 con 0x7f6a780cf850 2024-03-06T07:26:22.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a5ffff640 1 -- 172.21.15.59:0/2182457734 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a68005d00 con 0x7f6a780cf850 2024-03-06T07:26:22.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a7d82a640 1 -- 172.21.15.59:0/2182457734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a780cb750 con 0x7f6a780cf850 2024-03-06T07:26:22.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a5ffff640 1 -- 172.21.15.59:0/2182457734 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a680072b0 con 0x7f6a780cf850 2024-03-06T07:26:22.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.141+0000 7f6a7d82a640 1 -- 172.21.15.59:0/2182457734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a780cba30 con 0x7f6a780cf850 2024-03-06T07:26:22.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.143+0000 7f6a7d82a640 1 -- 172.21.15.59:0/2182457734 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7804f330 con 0x7f6a780cf850 2024-03-06T07:26:22.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.143+0000 7f6a5ffff640 1 -- 172.21.15.59:0/2182457734 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a6801a070 con 0x7f6a780cf850 2024-03-06T07:26:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.143+0000 7f6a5ffff640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a6406b4d0 0x7f6a6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.143+0000 7f6a5ffff640 1 -- 172.21.15.59:0/2182457734 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a68089530 con 0x7f6a780cf850 2024-03-06T07:26:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.143+0000 7f6a767fc640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a6406b4d0 0x7f6a6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.144+0000 7f6a767fc640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a6406b4d0 0x7f6a6406d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c007fd0 tx=0x7f6a6c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:22.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.148+0000 7f6a5ffff640 1 -- 172.21.15.59:0/2182457734 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a68057e20 con 0x7f6a780cf850 2024-03-06T07:26:22.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.324+0000 7f6a7d82a640 1 -- 172.21.15.59:0/2182457734 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a780d06f0 con 0x7f6a6406b4d0 2024-03-06T07:26:22.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.331+0000 7f6a5ffff640 1 -- 172.21.15.59:0/2182457734 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+250478 (secure 0 0 0) 0x7f6a780d06f0 con 0x7f6a6406b4d0 2024-03-06T07:26:22.335 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:22.336 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 -- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a6406b4d0 msgr2=0x7f6a6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a6406b4d0 0x7f6a6406d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c007fd0 tx=0x7f6a6c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 -- 172.21.15.59:0/2182457734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a780cf850 msgr2=0x7f6a7814f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a780cf850 0x7f6a7814f6e0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800c530 tx=0x7f6a6800c9f0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 -- 172.21.15.59:0/2182457734 shutdown_connections 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6a6406b4d0 0x7f6a6406d990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a78166af0 0x7f6a7814f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a780d00c0 0x7f6a78159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 --2- 172.21.15.59:0/2182457734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a780cf850 0x7f6a7814f6e0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 -- 172.21.15.59:0/2182457734 >> 172.21.15.59:0/2182457734 conn(0x7f6a780b35a0 msgr2=0x7f6a780b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 -- 172.21.15.59:0/2182457734 shutdown_connections 2024-03-06T07:26:22.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:22.334+0000 7f6a7d82a640 1 -- 172.21.15.59:0/2182457734 wait complete. 2024-03-06T07:26:22.346 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-03-06T07:26:21.359172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592708,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":176,"acting":176,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259552,"kb_used_data":10076,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715168,"statfs":{"total":773094113280,"available":772828332032,"internally_reserved":0,"allocated":10317824,"data_stored":5049858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2340,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.514806"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854775+0000","last_change":"2024-03-06T07:26:17.877929+0000","last_active":"2024-03-06T07:26:18.854775+0000","last_peered":"2024-03-06T07:26:18.854775+0000","last_clean":"2024-03-06T07:26:18.854775+0000","last_became_active":"2024-03-06T07:26:17.877642+0000","last_became_peered":"2024-03-06T07:26:17.877642+0000","last_unstale":"2024-03-06T07:26:18.854775+0000","last_undegraded":"2024-03-06T07:26:18.854775+0000","last_fullsized":"2024-03-06T07:26:18.854775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:53.393641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875012+0000","last_change":"2024-03-06T07:26:17.868113+0000","last_active":"2024-03-06T07:26:18.875012+0000","last_peered":"2024-03-06T07:26:18.875012+0000","last_clean":"2024-03-06T07:26:18.875012+0000","last_became_active":"2024-03-06T07:26:17.867743+0000","last_became_peered":"2024-03-06T07:26:17.867743+0000","last_unstale":"2024-03-06T07:26:18.875012+0000","last_undegraded":"2024-03-06T07:26:18.875012+0000","last_fullsized":"2024-03-06T07:26:18.875012+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:57.329605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854687+0000","last_change":"2024-03-06T07:26:17.878201+0000","last_active":"2024-03-06T07:26:18.854687+0000","last_peered":"2024-03-06T07:26:18.854687+0000","last_clean":"2024-03-06T07:26:18.854687+0000","last_became_active":"2024-03-06T07:26:17.877378+0000","last_became_peered":"2024-03-06T07:26:17.877378+0000","last_unstale":"2024-03-06T07:26:18.854687+0000","last_undegraded":"2024-03-06T07:26:18.854687+0000","last_fullsized":"2024-03-06T07:26:18.854687+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:41.440891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855757+0000","last_change":"2024-03-06T07:26:17.871063+0000","last_active":"2024-03-06T07:26:18.855757+0000","last_peered":"2024-03-06T07:26:18.855757+0000","last_clean":"2024-03-06T07:26:18.855757+0000","last_became_active":"2024-03-06T07:26:17.870049+0000","last_became_peered":"2024-03-06T07:26:17.870049+0000","last_unstale":"2024-03-06T07:26:18.855757+0000","last_undegraded":"2024-03-06T07:26:18.855757+0000","last_fullsized":"2024-03-06T07:26:18.855757+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:43.505953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631500+0000","last_change":"2024-03-06T07:26:17.864205+0000","last_active":"2024-03-06T07:26:19.631500+0000","last_peered":"2024-03-06T07:26:19.631500+0000","last_clean":"2024-03-06T07:26:19.631500+0000","last_became_active":"2024-03-06T07:26:17.863941+0000","last_became_peered":"2024-03-06T07:26:17.863941+0000","last_unstale":"2024-03-06T07:26:19.631500+0000","last_undegraded":"2024-03-06T07:26:19.631500+0000","last_fullsized":"2024-03-06T07:26:19.631500+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:03.507471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631356+0000","last_change":"2024-03-06T07:26:17.861310+0000","last_active":"2024-03-06T07:26:19.631356+0000","last_peered":"2024-03-06T07:26:19.631356+0000","last_clean":"2024-03-06T07:26:19.631356+0000","last_became_active":"2024-03-06T07:26:17.860944+0000","last_became_peered":"2024-03-06T07:26:17.860944+0000","last_unstale":"2024-03-06T07:26:19.631356+0000","last_undegraded":"2024-03-06T07:26:19.631356+0000","last_fullsized":"2024-03-06T07:26:19.631356+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:55.503575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874629+0000","last_change":"2024-03-06T07:26:17.861435+0000","last_active":"2024-03-06T07:26:18.874629+0000","last_peered":"2024-03-06T07:26:18.874629+0000","last_clean":"2024-03-06T07:26:18.874629+0000","last_became_active":"2024-03-06T07:26:17.861014+0000","last_became_peered":"2024-03-06T07:26:17.861014+0000","last_unstale":"2024-03-06T07:26:18.874629+0000","last_undegraded":"2024-03-06T07:26:18.874629+0000","last_fullsized":"2024-03-06T07:26:18.874629+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:54.987454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632809+0000","last_change":"2024-03-06T07:26:17.873920+0000","last_active":"2024-03-06T07:26:19.632809+0000","last_peered":"2024-03-06T07:26:19.632809+0000","last_clean":"2024-03-06T07:26:19.632809+0000","last_became_active":"2024-03-06T07:26:17.872835+0000","last_became_peered":"2024-03-06T07:26:17.872835+0000","last_unstale":"2024-03-06T07:26:19.632809+0000","last_undegraded":"2024-03-06T07:26:19.632809+0000","last_fullsized":"2024-03-06T07:26:19.632809+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:58.609019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631919+0000","last_change":"2024-03-06T07:26:17.874590+0000","last_active":"2024-03-06T07:26:19.631919+0000","last_peered":"2024-03-06T07:26:19.631919+0000","last_clean":"2024-03-06T07:26:19.631919+0000","last_became_active":"2024-03-06T07:26:17.874324+0000","last_became_peered":"2024-03-06T07:26:17.874324+0000","last_unstale":"2024-03-06T07:26:19.631919+0000","last_undegraded":"2024-03-06T07:26:19.631919+0000","last_fullsized":"2024-03-06T07:26:19.631919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:26.884759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632664+0000","last_change":"2024-03-06T07:26:17.875263+0000","last_active":"2024-03-06T07:26:19.632664+0000","last_peered":"2024-03-06T07:26:19.632664+0000","last_clean":"2024-03-06T07:26:19.632664+0000","last_became_active":"2024-03-06T07:26:17.874730+0000","last_became_peered":"2024-03-06T07:26:17.874730+0000","last_unstale":"2024-03-06T07:26:19.632664+0000","last_undegraded":"2024-03-06T07:26:19.632664+0000","last_fullsized":"2024-03-06T07:26:19.632664+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:55.503575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855537+0000","last_change":"2024-03-06T07:26:17.876108+0000","last_active":"2024-03-06T07:26:18.855537+0000","last_peered":"2024-03-06T07:26:18.855537+0000","last_clean":"2024-03-06T07:26:18.855537+0000","last_became_active":"2024-03-06T07:26:17.875830+0000","last_became_peered":"2024-03-06T07:26:17.875830+0000","last_unstale":"2024-03-06T07:26:18.855537+0000","last_undegraded":"2024-03-06T07:26:18.855537+0000","last_fullsized":"2024-03-06T07:26:18.855537+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:57.406246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876753+0000","last_change":"2024-03-06T07:26:17.868274+0000","last_active":"2024-03-06T07:26:18.876753+0000","last_peered":"2024-03-06T07:26:18.876753+0000","last_clean":"2024-03-06T07:26:18.876753+0000","last_became_active":"2024-03-06T07:26:17.867626+0000","last_became_peered":"2024-03-06T07:26:17.867626+0000","last_unstale":"2024-03-06T07:26:18.876753+0000","last_undegraded":"2024-03-06T07:26:18.876753+0000","last_fullsized":"2024-03-06T07:26:18.876753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:16.118134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855430+0000","last_change":"2024-03-06T07:26:17.870824+0000","last_active":"2024-03-06T07:26:18.855430+0000","last_peered":"2024-03-06T07:26:18.855430+0000","last_clean":"2024-03-06T07:26:18.855430+0000","last_became_active":"2024-03-06T07:26:17.870545+0000","last_became_peered":"2024-03-06T07:26:17.870545+0000","last_unstale":"2024-03-06T07:26:18.855430+0000","last_undegraded":"2024-03-06T07:26:18.855430+0000","last_fullsized":"2024-03-06T07:26:18.855430+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:22.063664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631403+0000","last_change":"2024-03-06T07:26:17.861351+0000","last_active":"2024-03-06T07:26:19.631403+0000","last_peered":"2024-03-06T07:26:19.631403+0000","last_clean":"2024-03-06T07:26:19.631403+0000","last_became_active":"2024-03-06T07:26:17.861004+0000","last_became_peered":"2024-03-06T07:26:17.861004+0000","last_unstale":"2024-03-06T07:26:19.631403+0000","last_undegraded":"2024-03-06T07:26:19.631403+0000","last_fullsized":"2024-03-06T07:26:19.631403+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:58.609019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874995+0000","last_change":"2024-03-06T07:26:17.869794+0000","last_active":"2024-03-06T07:26:18.874995+0000","last_peered":"2024-03-06T07:26:18.874995+0000","last_clean":"2024-03-06T07:26:18.874995+0000","last_became_active":"2024-03-06T07:26:17.869531+0000","last_became_peered":"2024-03-06T07:26:17.869531+0000","last_unstale":"2024-03-06T07:26:18.874995+0000","last_undegraded":"2024-03-06T07:26:18.874995+0000","last_fullsized":"2024-03-06T07:26:18.874995+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:43.505953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855708+0000","last_change":"2024-03-06T07:26:17.871014+0000","last_active":"2024-03-06T07:26:18.855708+0000","last_peered":"2024-03-06T07:26:18.855708+0000","last_clean":"2024-03-06T07:26:18.855708+0000","last_became_active":"2024-03-06T07:26:17.870687+0000","last_became_peered":"2024-03-06T07:26:17.870687+0000","last_unstale":"2024-03-06T07:26:18.855708+0000","last_undegraded":"2024-03-06T07:26:18.855708+0000","last_fullsized":"2024-03-06T07:26:18.855708+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:54.987454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877113+0000","last_change":"2024-03-06T07:26:17.873014+0000","last_active":"2024-03-06T07:26:18.877113+0000","last_peered":"2024-03-06T07:26:18.877113+0000","last_clean":"2024-03-06T07:26:18.877113+0000","last_became_active":"2024-03-06T07:26:17.872701+0000","last_became_peered":"2024-03-06T07:26:17.872701+0000","last_unstale":"2024-03-06T07:26:18.877113+0000","last_undegraded":"2024-03-06T07:26:18.877113+0000","last_fullsized":"2024-03-06T07:26:18.877113+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:38.541605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632622+0000","last_change":"2024-03-06T07:26:17.877086+0000","last_active":"2024-03-06T07:26:19.632622+0000","last_peered":"2024-03-06T07:26:19.632622+0000","last_clean":"2024-03-06T07:26:19.632622+0000","last_became_active":"2024-03-06T07:26:17.875124+0000","last_became_peered":"2024-03-06T07:26:17.875124+0000","last_unstale":"2024-03-06T07:26:19.632622+0000","last_undegraded":"2024-03-06T07:26:19.632622+0000","last_fullsized":"2024-03-06T07:26:19.632622+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:36.314409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876998+0000","last_change":"2024-03-06T07:26:17.862241+0000","last_active":"2024-03-06T07:26:18.876998+0000","last_peered":"2024-03-06T07:26:18.876998+0000","last_clean":"2024-03-06T07:26:18.876998+0000","last_became_active":"2024-03-06T07:26:17.861340+0000","last_became_peered":"2024-03-06T07:26:17.861340+0000","last_unstale":"2024-03-06T07:26:18.876998+0000","last_undegraded":"2024-03-06T07:26:18.876998+0000","last_fullsized":"2024-03-06T07:26:18.876998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:22.664538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874576+0000","last_change":"2024-03-06T07:26:17.861898+0000","last_active":"2024-03-06T07:26:18.874576+0000","last_peered":"2024-03-06T07:26:18.874576+0000","last_clean":"2024-03-06T07:26:18.874576+0000","last_became_active":"2024-03-06T07:26:17.861634+0000","last_became_peered":"2024-03-06T07:26:17.861634+0000","last_unstale":"2024-03-06T07:26:18.874576+0000","last_undegraded":"2024-03-06T07:26:18.874576+0000","last_fullsized":"2024-03-06T07:26:18.874576+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:57.406246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874586+0000","last_change":"2024-03-06T07:26:17.865417+0000","last_active":"2024-03-06T07:26:18.874586+0000","last_peered":"2024-03-06T07:26:18.874586+0000","last_clean":"2024-03-06T07:26:18.874586+0000","last_became_active":"2024-03-06T07:26:17.865139+0000","last_became_peered":"2024-03-06T07:26:17.865139+0000","last_unstale":"2024-03-06T07:26:18.874586+0000","last_undegraded":"2024-03-06T07:26:18.874586+0000","last_fullsized":"2024-03-06T07:26:18.874586+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.528335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631549+0000","last_change":"2024-03-06T07:26:17.868477+0000","last_active":"2024-03-06T07:26:19.631549+0000","last_peered":"2024-03-06T07:26:19.631549+0000","last_clean":"2024-03-06T07:26:19.631549+0000","last_became_active":"2024-03-06T07:26:17.867943+0000","last_became_peered":"2024-03-06T07:26:17.867943+0000","last_unstale":"2024-03-06T07:26:19.631549+0000","last_undegraded":"2024-03-06T07:26:19.631549+0000","last_fullsized":"2024-03-06T07:26:19.631549+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:02.151317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875070+0000","last_change":"2024-03-06T07:26:17.873693+0000","last_active":"2024-03-06T07:26:18.875070+0000","last_peered":"2024-03-06T07:26:18.875070+0000","last_clean":"2024-03-06T07:26:18.875070+0000","last_became_active":"2024-03-06T07:26:17.873363+0000","last_became_peered":"2024-03-06T07:26:17.873363+0000","last_unstale":"2024-03-06T07:26:18.875070+0000","last_undegraded":"2024-03-06T07:26:18.875070+0000","last_fullsized":"2024-03-06T07:26:18.875070+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:06.936077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632055+0000","last_change":"2024-03-06T07:26:17.864951+0000","last_active":"2024-03-06T07:26:19.632055+0000","last_peered":"2024-03-06T07:26:19.632055+0000","last_clean":"2024-03-06T07:26:19.632055+0000","last_became_active":"2024-03-06T07:26:17.864683+0000","last_became_peered":"2024-03-06T07:26:17.864683+0000","last_unstale":"2024-03-06T07:26:19.632055+0000","last_undegraded":"2024-03-06T07:26:19.632055+0000","last_fullsized":"2024-03-06T07:26:19.632055+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:46.261854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874766+0000","last_change":"2024-03-06T07:26:17.873122+0000","last_active":"2024-03-06T07:26:18.874766+0000","last_peered":"2024-03-06T07:26:18.874766+0000","last_clean":"2024-03-06T07:26:18.874766+0000","last_became_active":"2024-03-06T07:26:17.872806+0000","last_became_peered":"2024-03-06T07:26:17.872806+0000","last_unstale":"2024-03-06T07:26:18.874766+0000","last_undegraded":"2024-03-06T07:26:18.874766+0000","last_fullsized":"2024-03-06T07:26:18.874766+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:08.056201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876785+0000","last_change":"2024-03-06T07:26:17.868507+0000","last_active":"2024-03-06T07:26:18.876785+0000","last_peered":"2024-03-06T07:26:18.876785+0000","last_clean":"2024-03-06T07:26:18.876785+0000","last_became_active":"2024-03-06T07:26:17.868006+0000","last_became_peered":"2024-03-06T07:26:17.868006+0000","last_unstale":"2024-03-06T07:26:18.876785+0000","last_undegraded":"2024-03-06T07:26:18.876785+0000","last_fullsized":"2024-03-06T07:26:18.876785+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:10.435503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855724+0000","last_change":"2024-03-06T07:26:17.879833+0000","last_active":"2024-03-06T07:26:18.855724+0000","last_peered":"2024-03-06T07:26:18.855724+0000","last_clean":"2024-03-06T07:26:18.855724+0000","last_became_active":"2024-03-06T07:26:17.878938+0000","last_became_peered":"2024-03-06T07:26:17.878938+0000","last_unstale":"2024-03-06T07:26:18.855724+0000","last_undegraded":"2024-03-06T07:26:18.855724+0000","last_fullsized":"2024-03-06T07:26:18.855724+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:08.056201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854529+0000","last_change":"2024-03-06T07:26:17.866962+0000","last_active":"2024-03-06T07:26:18.854529+0000","last_peered":"2024-03-06T07:26:18.854529+0000","last_clean":"2024-03-06T07:26:18.854529+0000","last_became_active":"2024-03-06T07:26:17.866658+0000","last_became_peered":"2024-03-06T07:26:17.866658+0000","last_unstale":"2024-03-06T07:26:18.854529+0000","last_undegraded":"2024-03-06T07:26:18.854529+0000","last_fullsized":"2024-03-06T07:26:18.854529+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:25.582364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877006+0000","last_change":"2024-03-06T07:26:17.860951+0000","last_active":"2024-03-06T07:26:18.877006+0000","last_peered":"2024-03-06T07:26:18.877006+0000","last_clean":"2024-03-06T07:26:18.877006+0000","last_became_active":"2024-03-06T07:26:17.860689+0000","last_became_peered":"2024-03-06T07:26:17.860689+0000","last_unstale":"2024-03-06T07:26:18.877006+0000","last_undegraded":"2024-03-06T07:26:18.877006+0000","last_fullsized":"2024-03-06T07:26:18.877006+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:50.018552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"91'4","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874923+0000","last_change":"2024-03-06T07:26:16.861346+0000","last_active":"2024-03-06T07:26:18.874923+0000","last_peered":"2024-03-06T07:26:18.874923+0000","last_clean":"2024-03-06T07:26:18.874923+0000","last_became_active":"2024-03-06T07:26:16.860585+0000","last_became_peered":"2024-03-06T07:26:16.860585+0000","last_unstale":"2024-03-06T07:26:18.874923+0000","last_undegraded":"2024-03-06T07:26:18.874923+0000","last_fullsized":"2024-03-06T07:26:18.874923+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:06.710593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855827+0000","last_change":"2024-03-06T07:26:17.879842+0000","last_active":"2024-03-06T07:26:18.855827+0000","last_peered":"2024-03-06T07:26:18.855827+0000","last_clean":"2024-03-06T07:26:18.855827+0000","last_became_active":"2024-03-06T07:26:17.879291+0000","last_became_peered":"2024-03-06T07:26:17.879291+0000","last_unstale":"2024-03-06T07:26:18.855827+0000","last_undegraded":"2024-03-06T07:26:18.855827+0000","last_fullsized":"2024-03-06T07:26:18.855827+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:57.329605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631480+0000","last_change":"2024-03-06T07:26:16.868319+0000","last_active":"2024-03-06T07:26:19.631480+0000","last_peered":"2024-03-06T07:26:19.631480+0000","last_clean":"2024-03-06T07:26:19.631480+0000","last_became_active":"2024-03-06T07:26:16.868043+0000","last_became_peered":"2024-03-06T07:26:16.868043+0000","last_unstale":"2024-03-06T07:26:19.631480+0000","last_undegraded":"2024-03-06T07:26:19.631480+0000","last_fullsized":"2024-03-06T07:26:19.631480+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:25.302742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854455+0000","last_change":"2024-03-06T07:26:17.875745+0000","last_active":"2024-03-06T07:26:18.854455+0000","last_peered":"2024-03-06T07:26:18.854455+0000","last_clean":"2024-03-06T07:26:18.854455+0000","last_became_active":"2024-03-06T07:26:17.875471+0000","last_became_peered":"2024-03-06T07:26:17.875471+0000","last_unstale":"2024-03-06T07:26:18.854455+0000","last_undegraded":"2024-03-06T07:26:18.854455+0000","last_fullsized":"2024-03-06T07:26:18.854455+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:47.848728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874838+0000","last_change":"2024-03-06T07:26:16.870929+0000","last_active":"2024-03-06T07:26:18.874838+0000","last_peered":"2024-03-06T07:26:18.874838+0000","last_clean":"2024-03-06T07:26:18.874838+0000","last_became_active":"2024-03-06T07:26:16.870661+0000","last_became_peered":"2024-03-06T07:26:16.870661+0000","last_unstale":"2024-03-06T07:26:18.874838+0000","last_undegraded":"2024-03-06T07:26:18.874838+0000","last_fullsized":"2024-03-06T07:26:18.874838+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:13.611316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632637+0000","last_change":"2024-03-06T07:26:17.872067+0000","last_active":"2024-03-06T07:26:19.632637+0000","last_peered":"2024-03-06T07:26:19.632637+0000","last_clean":"2024-03-06T07:26:19.632637+0000","last_became_active":"2024-03-06T07:26:17.871800+0000","last_became_peered":"2024-03-06T07:26:17.871800+0000","last_unstale":"2024-03-06T07:26:19.632637+0000","last_undegraded":"2024-03-06T07:26:19.632637+0000","last_fullsized":"2024-03-06T07:26:19.632637+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:27.404185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874719+0000","last_change":"2024-03-06T07:26:16.864464+0000","last_active":"2024-03-06T07:26:18.874719+0000","last_peered":"2024-03-06T07:26:18.874719+0000","last_clean":"2024-03-06T07:26:18.874719+0000","last_became_active":"2024-03-06T07:26:16.863949+0000","last_became_peered":"2024-03-06T07:26:16.863949+0000","last_unstale":"2024-03-06T07:26:18.874719+0000","last_undegraded":"2024-03-06T07:26:18.874719+0000","last_fullsized":"2024-03-06T07:26:18.874719+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:52.388832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856105+0000","last_change":"2024-03-06T07:26:17.872420+0000","last_active":"2024-03-06T07:26:18.856105+0000","last_peered":"2024-03-06T07:26:18.856105+0000","last_clean":"2024-03-06T07:26:18.856105+0000","last_became_active":"2024-03-06T07:26:17.872122+0000","last_became_peered":"2024-03-06T07:26:17.872122+0000","last_unstale":"2024-03-06T07:26:18.856105+0000","last_undegraded":"2024-03-06T07:26:18.856105+0000","last_fullsized":"2024-03-06T07:26:18.856105+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:15.723196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854281+0000","last_change":"2024-03-06T07:26:16.863634+0000","last_active":"2024-03-06T07:26:18.854281+0000","last_peered":"2024-03-06T07:26:18.854281+0000","last_clean":"2024-03-06T07:26:18.854281+0000","last_became_active":"2024-03-06T07:26:16.863372+0000","last_became_peered":"2024-03-06T07:26:16.863372+0000","last_unstale":"2024-03-06T07:26:18.854281+0000","last_undegraded":"2024-03-06T07:26:18.854281+0000","last_fullsized":"2024-03-06T07:26:18.854281+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:29.981611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631644+0000","last_change":"2024-03-06T07:26:17.871415+0000","last_active":"2024-03-06T07:26:19.631644+0000","last_peered":"2024-03-06T07:26:19.631644+0000","last_clean":"2024-03-06T07:26:19.631644+0000","last_became_active":"2024-03-06T07:26:17.871135+0000","last_became_peered":"2024-03-06T07:26:17.871135+0000","last_unstale":"2024-03-06T07:26:19.631644+0000","last_undegraded":"2024-03-06T07:26:19.631644+0000","last_fullsized":"2024-03-06T07:26:19.631644+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:36.291779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874580+0000","last_change":"2024-03-06T07:26:16.866013+0000","last_active":"2024-03-06T07:26:18.874580+0000","last_peered":"2024-03-06T07:26:18.874580+0000","last_clean":"2024-03-06T07:26:18.874580+0000","last_became_active":"2024-03-06T07:26:16.865737+0000","last_became_peered":"2024-03-06T07:26:16.865737+0000","last_unstale":"2024-03-06T07:26:18.874580+0000","last_undegraded":"2024-03-06T07:26:18.874580+0000","last_fullsized":"2024-03-06T07:26:18.874580+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:24.577556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855424+0000","last_change":"2024-03-06T07:26:17.871627+0000","last_active":"2024-03-06T07:26:18.855424+0000","last_peered":"2024-03-06T07:26:18.855424+0000","last_clean":"2024-03-06T07:26:18.855424+0000","last_became_active":"2024-03-06T07:26:17.871289+0000","last_became_peered":"2024-03-06T07:26:17.871289+0000","last_unstale":"2024-03-06T07:26:18.855424+0000","last_undegraded":"2024-03-06T07:26:18.855424+0000","last_fullsized":"2024-03-06T07:26:18.855424+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:52.299295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850220+0000","last_change":"2024-03-06T07:26:15.850220+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850220+0000","last_undegraded":"2024-03-06T07:26:15.850220+0000","last_fullsized":"2024-03-06T07:26:15.850220+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632457+0000","last_change":"2024-03-06T07:26:17.873691+0000","last_active":"2024-03-06T07:26:19.632457+0000","last_peered":"2024-03-06T07:26:19.632457+0000","last_clean":"2024-03-06T07:26:19.632457+0000","last_became_active":"2024-03-06T07:26:17.873417+0000","last_became_peered":"2024-03-06T07:26:17.873417+0000","last_unstale":"2024-03-06T07:26:19.632457+0000","last_undegraded":"2024-03-06T07:26:19.632457+0000","last_fullsized":"2024-03-06T07:26:19.632457+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:59.170419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854190+0000","last_change":"2024-03-06T07:26:16.869647+0000","last_active":"2024-03-06T07:26:18.854190+0000","last_peered":"2024-03-06T07:26:18.854190+0000","last_clean":"2024-03-06T07:26:18.854190+0000","last_became_active":"2024-03-06T07:26:16.869365+0000","last_became_peered":"2024-03-06T07:26:16.869365+0000","last_unstale":"2024-03-06T07:26:18.854190+0000","last_undegraded":"2024-03-06T07:26:18.854190+0000","last_fullsized":"2024-03-06T07:26:18.854190+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:21.875952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875098+0000","last_change":"2024-03-06T07:26:17.870669+0000","last_active":"2024-03-06T07:26:18.875098+0000","last_peered":"2024-03-06T07:26:18.875098+0000","last_clean":"2024-03-06T07:26:18.875098+0000","last_became_active":"2024-03-06T07:26:17.870363+0000","last_became_peered":"2024-03-06T07:26:17.870363+0000","last_unstale":"2024-03-06T07:26:18.875098+0000","last_undegraded":"2024-03-06T07:26:18.875098+0000","last_fullsized":"2024-03-06T07:26:18.875098+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:47.379775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854820+0000","last_change":"2024-03-06T07:26:16.865155+0000","last_active":"2024-03-06T07:26:18.854820+0000","last_peered":"2024-03-06T07:26:18.854820+0000","last_clean":"2024-03-06T07:26:18.854820+0000","last_became_active":"2024-03-06T07:26:16.864873+0000","last_became_peered":"2024-03-06T07:26:16.864873+0000","last_unstale":"2024-03-06T07:26:18.854820+0000","last_undegraded":"2024-03-06T07:26:18.854820+0000","last_fullsized":"2024-03-06T07:26:18.854820+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:57.303182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632939+0000","last_change":"2024-03-06T07:26:17.874753+0000","last_active":"2024-03-06T07:26:19.632939+0000","last_peered":"2024-03-06T07:26:19.632939+0000","last_clean":"2024-03-06T07:26:19.632939+0000","last_became_active":"2024-03-06T07:26:17.874488+0000","last_became_peered":"2024-03-06T07:26:17.874488+0000","last_unstale":"2024-03-06T07:26:19.632939+0000","last_undegraded":"2024-03-06T07:26:19.632939+0000","last_fullsized":"2024-03-06T07:26:19.632939+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:03.507471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855695+0000","last_change":"2024-03-06T07:26:16.872968+0000","last_active":"2024-03-06T07:26:18.855695+0000","last_peered":"2024-03-06T07:26:18.855695+0000","last_clean":"2024-03-06T07:26:18.855695+0000","last_became_active":"2024-03-06T07:26:16.872657+0000","last_became_peered":"2024-03-06T07:26:16.872657+0000","last_unstale":"2024-03-06T07:26:18.855695+0000","last_undegraded":"2024-03-06T07:26:18.855695+0000","last_fullsized":"2024-03-06T07:26:18.855695+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:35.309600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"91'3","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632226+0000","last_change":"2024-03-06T07:26:16.866799+0000","last_active":"2024-03-06T07:26:19.632226+0000","last_peered":"2024-03-06T07:26:19.632226+0000","last_clean":"2024-03-06T07:26:19.632226+0000","last_became_active":"2024-03-06T07:26:16.866520+0000","last_became_peered":"2024-03-06T07:26:16.866520+0000","last_unstale":"2024-03-06T07:26:19.632226+0000","last_undegraded":"2024-03-06T07:26:19.632226+0000","last_fullsized":"2024-03-06T07:26:19.632226+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:21.270064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632180+0000","last_change":"2024-03-06T07:26:17.863780+0000","last_active":"2024-03-06T07:26:19.632180+0000","last_peered":"2024-03-06T07:26:19.632180+0000","last_clean":"2024-03-06T07:26:19.632180+0000","last_became_active":"2024-03-06T07:26:17.863503+0000","last_became_peered":"2024-03-06T07:26:17.863503+0000","last_unstale":"2024-03-06T07:26:19.632180+0000","last_undegraded":"2024-03-06T07:26:19.632180+0000","last_fullsized":"2024-03-06T07:26:19.632180+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:43.263166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"91'6","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632748+0000","last_change":"2024-03-06T07:26:16.868702+0000","last_active":"2024-03-06T07:26:19.632748+0000","last_peered":"2024-03-06T07:26:19.632748+0000","last_clean":"2024-03-06T07:26:19.632748+0000","last_became_active":"2024-03-06T07:26:16.868364+0000","last_became_peered":"2024-03-06T07:26:16.868364+0000","last_unstale":"2024-03-06T07:26:19.632748+0000","last_undegraded":"2024-03-06T07:26:19.632748+0000","last_fullsized":"2024-03-06T07:26:19.632748+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:19.712353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855547+0000","last_change":"2024-03-06T07:26:17.871667+0000","last_active":"2024-03-06T07:26:18.855547+0000","last_peered":"2024-03-06T07:26:18.855547+0000","last_clean":"2024-03-06T07:26:18.855547+0000","last_became_active":"2024-03-06T07:26:17.871317+0000","last_became_peered":"2024-03-06T07:26:17.871317+0000","last_unstale":"2024-03-06T07:26:18.855547+0000","last_undegraded":"2024-03-06T07:26:18.855547+0000","last_fullsized":"2024-03-06T07:26:18.855547+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.528335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877108+0000","last_change":"2024-03-06T07:26:16.865944+0000","last_active":"2024-03-06T07:26:18.877108+0000","last_peered":"2024-03-06T07:26:18.877108+0000","last_clean":"2024-03-06T07:26:18.877108+0000","last_became_active":"2024-03-06T07:26:16.865672+0000","last_became_peered":"2024-03-06T07:26:16.865672+0000","last_unstale":"2024-03-06T07:26:18.877108+0000","last_undegraded":"2024-03-06T07:26:18.877108+0000","last_fullsized":"2024-03-06T07:26:18.877108+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:03.112514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854825+0000","last_change":"2024-03-06T07:26:17.867658+0000","last_active":"2024-03-06T07:26:18.854825+0000","last_peered":"2024-03-06T07:26:18.854825+0000","last_clean":"2024-03-06T07:26:18.854825+0000","last_became_active":"2024-03-06T07:26:17.867387+0000","last_became_peered":"2024-03-06T07:26:17.867387+0000","last_unstale":"2024-03-06T07:26:18.854825+0000","last_undegraded":"2024-03-06T07:26:18.854825+0000","last_fullsized":"2024-03-06T07:26:18.854825+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:14.616124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877004+0000","last_change":"2024-03-06T07:26:16.869461+0000","last_active":"2024-03-06T07:26:18.877004+0000","last_peered":"2024-03-06T07:26:18.877004+0000","last_clean":"2024-03-06T07:26:18.877004+0000","last_became_active":"2024-03-06T07:26:16.869142+0000","last_became_peered":"2024-03-06T07:26:16.869142+0000","last_unstale":"2024-03-06T07:26:18.877004+0000","last_undegraded":"2024-03-06T07:26:18.877004+0000","last_fullsized":"2024-03-06T07:26:18.877004+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:53.301305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631353+0000","last_change":"2024-03-06T07:26:17.875559+0000","last_active":"2024-03-06T07:26:19.631353+0000","last_peered":"2024-03-06T07:26:19.631353+0000","last_clean":"2024-03-06T07:26:19.631353+0000","last_became_active":"2024-03-06T07:26:17.875196+0000","last_became_peered":"2024-03-06T07:26:17.875196+0000","last_unstale":"2024-03-06T07:26:19.631353+0000","last_undegraded":"2024-03-06T07:26:19.631353+0000","last_fullsized":"2024-03-06T07:26:19.631353+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:14.616124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632305+0000","last_change":"2024-03-06T07:26:16.865532+0000","last_active":"2024-03-06T07:26:19.632305+0000","last_peered":"2024-03-06T07:26:19.632305+0000","last_clean":"2024-03-06T07:26:19.632305+0000","last_became_active":"2024-03-06T07:26:16.865038+0000","last_became_peered":"2024-03-06T07:26:16.865038+0000","last_unstale":"2024-03-06T07:26:19.632305+0000","last_undegraded":"2024-03-06T07:26:19.632305+0000","last_fullsized":"2024-03-06T07:26:19.632305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:18.180662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876751+0000","last_change":"2024-03-06T07:26:17.875784+0000","last_active":"2024-03-06T07:26:18.876751+0000","last_peered":"2024-03-06T07:26:18.876751+0000","last_clean":"2024-03-06T07:26:18.876751+0000","last_became_active":"2024-03-06T07:26:17.875509+0000","last_became_peered":"2024-03-06T07:26:17.875509+0000","last_unstale":"2024-03-06T07:26:18.876751+0000","last_undegraded":"2024-03-06T07:26:18.876751+0000","last_fullsized":"2024-03-06T07:26:18.876751+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:39.164839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"91'8","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854323+0000","last_change":"2024-03-06T07:26:16.866435+0000","last_active":"2024-03-06T07:26:18.854323+0000","last_peered":"2024-03-06T07:26:18.854323+0000","last_clean":"2024-03-06T07:26:18.854323+0000","last_became_active":"2024-03-06T07:26:16.866136+0000","last_became_peered":"2024-03-06T07:26:16.866136+0000","last_unstale":"2024-03-06T07:26:18.854323+0000","last_undegraded":"2024-03-06T07:26:18.854323+0000","last_fullsized":"2024-03-06T07:26:18.854323+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:18.526671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632569+0000","last_change":"2024-03-06T07:26:16.865144+0000","last_active":"2024-03-06T07:26:19.632569+0000","last_peered":"2024-03-06T07:26:19.632569+0000","last_clean":"2024-03-06T07:26:19.632569+0000","last_became_active":"2024-03-06T07:26:16.864840+0000","last_became_peered":"2024-03-06T07:26:16.864840+0000","last_unstale":"2024-03-06T07:26:19.632569+0000","last_undegraded":"2024-03-06T07:26:19.632569+0000","last_fullsized":"2024-03-06T07:26:19.632569+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:14.718388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856150+0000","last_change":"2024-03-06T07:26:17.867621+0000","last_active":"2024-03-06T07:26:18.856150+0000","last_peered":"2024-03-06T07:26:18.856150+0000","last_clean":"2024-03-06T07:26:18.856150+0000","last_became_active":"2024-03-06T07:26:17.867337+0000","last_became_peered":"2024-03-06T07:26:17.867337+0000","last_unstale":"2024-03-06T07:26:18.856150+0000","last_undegraded":"2024-03-06T07:26:18.856150+0000","last_fullsized":"2024-03-06T07:26:18.856150+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:20.717162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850317+0000","last_change":"2024-03-06T07:26:15.850317+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850317+0000","last_undegraded":"2024-03-06T07:26:15.850317+0000","last_fullsized":"2024-03-06T07:26:15.850317+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632234+0000","last_change":"2024-03-06T07:26:17.873920+0000","last_active":"2024-03-06T07:26:19.632234+0000","last_peered":"2024-03-06T07:26:19.632234+0000","last_clean":"2024-03-06T07:26:19.632234+0000","last_became_active":"2024-03-06T07:26:17.873419+0000","last_became_peered":"2024-03-06T07:26:17.873419+0000","last_unstale":"2024-03-06T07:26:19.632234+0000","last_undegraded":"2024-03-06T07:26:19.632234+0000","last_fullsized":"2024-03-06T07:26:19.632234+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:22.063664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874737+0000","last_change":"2024-03-06T07:26:16.864486+0000","last_active":"2024-03-06T07:26:18.874737+0000","last_peered":"2024-03-06T07:26:18.874737+0000","last_clean":"2024-03-06T07:26:18.874737+0000","last_became_active":"2024-03-06T07:26:16.862774+0000","last_became_peered":"2024-03-06T07:26:16.862774+0000","last_unstale":"2024-03-06T07:26:18.874737+0000","last_undegraded":"2024-03-06T07:26:18.874737+0000","last_fullsized":"2024-03-06T07:26:18.874737+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:58.165611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876950+0000","last_change":"2024-03-06T07:26:17.873966+0000","last_active":"2024-03-06T07:26:18.876950+0000","last_peered":"2024-03-06T07:26:18.876950+0000","last_clean":"2024-03-06T07:26:18.876950+0000","last_became_active":"2024-03-06T07:26:17.873702+0000","last_became_peered":"2024-03-06T07:26:17.873702+0000","last_unstale":"2024-03-06T07:26:18.876950+0000","last_undegraded":"2024-03-06T07:26:18.876950+0000","last_fullsized":"2024-03-06T07:26:18.876950+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:47.962109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874691+0000","last_change":"2024-03-06T07:26:16.866599+0000","last_active":"2024-03-06T07:26:18.874691+0000","last_peered":"2024-03-06T07:26:18.874691+0000","last_clean":"2024-03-06T07:26:18.874691+0000","last_became_active":"2024-03-06T07:26:16.866342+0000","last_became_peered":"2024-03-06T07:26:16.866342+0000","last_unstale":"2024-03-06T07:26:18.874691+0000","last_undegraded":"2024-03-06T07:26:18.874691+0000","last_fullsized":"2024-03-06T07:26:18.874691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:30.398774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877178+0000","last_change":"2024-03-06T07:26:17.860602+0000","last_active":"2024-03-06T07:26:18.877178+0000","last_peered":"2024-03-06T07:26:18.877178+0000","last_clean":"2024-03-06T07:26:18.877178+0000","last_became_active":"2024-03-06T07:26:17.860333+0000","last_became_peered":"2024-03-06T07:26:17.860333+0000","last_unstale":"2024-03-06T07:26:18.877178+0000","last_undegraded":"2024-03-06T07:26:18.877178+0000","last_fullsized":"2024-03-06T07:26:18.877178+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:45.077065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856177+0000","last_change":"2024-03-06T07:26:16.868618+0000","last_active":"2024-03-06T07:26:18.856177+0000","last_peered":"2024-03-06T07:26:18.856177+0000","last_clean":"2024-03-06T07:26:18.856177+0000","last_became_active":"2024-03-06T07:26:16.868180+0000","last_became_peered":"2024-03-06T07:26:16.868180+0000","last_unstale":"2024-03-06T07:26:18.856177+0000","last_undegraded":"2024-03-06T07:26:18.856177+0000","last_fullsized":"2024-03-06T07:26:18.856177+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:26.488466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876708+0000","last_change":"2024-03-06T07:26:17.868580+0000","last_active":"2024-03-06T07:26:18.876708+0000","last_peered":"2024-03-06T07:26:18.876708+0000","last_clean":"2024-03-06T07:26:18.876708+0000","last_became_active":"2024-03-06T07:26:17.868280+0000","last_became_peered":"2024-03-06T07:26:17.868280+0000","last_unstale":"2024-03-06T07:26:18.876708+0000","last_undegraded":"2024-03-06T07:26:18.876708+0000","last_fullsized":"2024-03-06T07:26:18.876708+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:43.974866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855956+0000","last_change":"2024-03-06T07:26:16.868344+0000","last_active":"2024-03-06T07:26:18.855956+0000","last_peered":"2024-03-06T07:26:18.855956+0000","last_clean":"2024-03-06T07:26:18.855956+0000","last_became_active":"2024-03-06T07:26:16.868005+0000","last_became_peered":"2024-03-06T07:26:16.868005+0000","last_unstale":"2024-03-06T07:26:18.855956+0000","last_undegraded":"2024-03-06T07:26:18.855956+0000","last_fullsized":"2024-03-06T07:26:18.855956+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:57.303182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874532+0000","last_change":"2024-03-06T07:26:16.860891+0000","last_active":"2024-03-06T07:26:18.874532+0000","last_peered":"2024-03-06T07:26:18.874532+0000","last_clean":"2024-03-06T07:26:18.874532+0000","last_became_active":"2024-03-06T07:26:16.860523+0000","last_became_peered":"2024-03-06T07:26:16.860523+0000","last_unstale":"2024-03-06T07:26:18.874532+0000","last_undegraded":"2024-03-06T07:26:18.874532+0000","last_fullsized":"2024-03-06T07:26:18.874532+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:21.270064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875146+0000","last_change":"2024-03-06T07:26:17.868058+0000","last_active":"2024-03-06T07:26:18.875146+0000","last_peered":"2024-03-06T07:26:18.875146+0000","last_clean":"2024-03-06T07:26:18.875146+0000","last_became_active":"2024-03-06T07:26:17.867378+0000","last_became_peered":"2024-03-06T07:26:17.867378+0000","last_unstale":"2024-03-06T07:26:18.875146+0000","last_undegraded":"2024-03-06T07:26:18.875146+0000","last_fullsized":"2024-03-06T07:26:18.875146+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:40.439617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856296+0000","last_change":"2024-03-06T07:26:16.863938+0000","last_active":"2024-03-06T07:26:18.856296+0000","last_peered":"2024-03-06T07:26:18.856296+0000","last_clean":"2024-03-06T07:26:18.856296+0000","last_became_active":"2024-03-06T07:26:16.863664+0000","last_became_peered":"2024-03-06T07:26:16.863664+0000","last_unstale":"2024-03-06T07:26:18.856296+0000","last_undegraded":"2024-03-06T07:26:18.856296+0000","last_fullsized":"2024-03-06T07:26:18.856296+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:18.526671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854824+0000","last_change":"2024-03-06T07:26:17.872137+0000","last_active":"2024-03-06T07:26:18.854824+0000","last_peered":"2024-03-06T07:26:18.854824+0000","last_clean":"2024-03-06T07:26:18.854824+0000","last_became_active":"2024-03-06T07:26:17.871867+0000","last_became_peered":"2024-03-06T07:26:17.871867+0000","last_unstale":"2024-03-06T07:26:18.854824+0000","last_undegraded":"2024-03-06T07:26:18.854824+0000","last_fullsized":"2024-03-06T07:26:18.854824+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:31.403582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"92'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875144+0000","last_change":"2024-03-06T07:26:16.870823+0000","last_active":"2024-03-06T07:26:18.875144+0000","last_peered":"2024-03-06T07:26:18.875144+0000","last_clean":"2024-03-06T07:26:18.875144+0000","last_became_active":"2024-03-06T07:26:16.870507+0000","last_became_peered":"2024-03-06T07:26:16.870507+0000","last_unstale":"2024-03-06T07:26:18.875144+0000","last_undegraded":"2024-03-06T07:26:18.875144+0000","last_fullsized":"2024-03-06T07:26:18.875144+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:35.309600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854500+0000","last_change":"2024-03-06T07:26:17.880876+0000","last_active":"2024-03-06T07:26:18.854500+0000","last_peered":"2024-03-06T07:26:18.854500+0000","last_clean":"2024-03-06T07:26:18.854500+0000","last_became_active":"2024-03-06T07:26:17.880610+0000","last_became_peered":"2024-03-06T07:26:17.880610+0000","last_unstale":"2024-03-06T07:26:18.854500+0000","last_undegraded":"2024-03-06T07:26:18.854500+0000","last_fullsized":"2024-03-06T07:26:18.854500+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:22.274872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874842+0000","last_change":"2024-03-06T07:26:16.864709+0000","last_active":"2024-03-06T07:26:18.874842+0000","last_peered":"2024-03-06T07:26:18.874842+0000","last_clean":"2024-03-06T07:26:18.874842+0000","last_became_active":"2024-03-06T07:26:16.863053+0000","last_became_peered":"2024-03-06T07:26:16.863053+0000","last_unstale":"2024-03-06T07:26:18.874842+0000","last_undegraded":"2024-03-06T07:26:18.874842+0000","last_fullsized":"2024-03-06T07:26:18.874842+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:26.399377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850394+0000","last_change":"2024-03-06T07:26:15.850394+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850394+0000","last_undegraded":"2024-03-06T07:26:15.850394+0000","last_fullsized":"2024-03-06T07:26:15.850394+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877061+0000","last_change":"2024-03-06T07:26:17.876580+0000","last_active":"2024-03-06T07:26:18.877061+0000","last_peered":"2024-03-06T07:26:18.877061+0000","last_clean":"2024-03-06T07:26:18.877061+0000","last_became_active":"2024-03-06T07:26:17.876316+0000","last_became_peered":"2024-03-06T07:26:17.876316+0000","last_unstale":"2024-03-06T07:26:18.877061+0000","last_undegraded":"2024-03-06T07:26:18.877061+0000","last_fullsized":"2024-03-06T07:26:18.877061+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:53.155139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874794+0000","last_change":"2024-03-06T07:26:16.870873+0000","last_active":"2024-03-06T07:26:18.874794+0000","last_peered":"2024-03-06T07:26:18.874794+0000","last_clean":"2024-03-06T07:26:18.874794+0000","last_became_active":"2024-03-06T07:26:16.870581+0000","last_became_peered":"2024-03-06T07:26:16.870581+0000","last_unstale":"2024-03-06T07:26:18.874794+0000","last_undegraded":"2024-03-06T07:26:18.874794+0000","last_fullsized":"2024-03-06T07:26:18.874794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:40.436083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632601+0000","last_change":"2024-03-06T07:26:17.880256+0000","last_active":"2024-03-06T07:26:19.632601+0000","last_peered":"2024-03-06T07:26:19.632601+0000","last_clean":"2024-03-06T07:26:19.632601+0000","last_became_active":"2024-03-06T07:26:17.879582+0000","last_became_peered":"2024-03-06T07:26:17.879582+0000","last_unstale":"2024-03-06T07:26:19.632601+0000","last_undegraded":"2024-03-06T07:26:19.632601+0000","last_fullsized":"2024-03-06T07:26:19.632601+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:07.715401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":89,"state":"creating+peering","last_fresh":"2024-03-06T07:26:15.850379+0000","last_change":"2024-03-06T07:26:15.850379+0000","last_active":"2024-03-06T07:26:15.838511+0000","last_peered":"2024-03-06T07:26:15.838511+0000","last_clean":"2024-03-06T07:26:15.838511+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:15.850379+0000","last_undegraded":"2024-03-06T07:26:15.850379+0000","last_fullsized":"2024-03-06T07:26:15.850379+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876708+0000","last_change":"2024-03-06T07:26:17.875415+0000","last_active":"2024-03-06T07:26:18.876708+0000","last_peered":"2024-03-06T07:26:18.876708+0000","last_clean":"2024-03-06T07:26:18.876708+0000","last_became_active":"2024-03-06T07:26:17.875138+0000","last_became_peered":"2024-03-06T07:26:17.875138+0000","last_unstale":"2024-03-06T07:26:18.876708+0000","last_undegraded":"2024-03-06T07:26:18.876708+0000","last_fullsized":"2024-03-06T07:26:18.876708+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:27.427674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855626+0000","last_change":"2024-03-06T07:26:16.871839+0000","last_active":"2024-03-06T07:26:18.855626+0000","last_peered":"2024-03-06T07:26:18.855626+0000","last_clean":"2024-03-06T07:26:18.855626+0000","last_became_active":"2024-03-06T07:26:16.871570+0000","last_became_peered":"2024-03-06T07:26:16.871570+0000","last_unstale":"2024-03-06T07:26:18.855626+0000","last_undegraded":"2024-03-06T07:26:18.855626+0000","last_fullsized":"2024-03-06T07:26:18.855626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:06.710593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:26:16.843320+0000","last_change":"2024-03-06T07:26:16.843320+0000","last_active":"2024-03-06T07:26:16.843320+0000","last_peered":"2024-03-06T07:26:16.843320+0000","last_clean":"2024-03-06T07:26:16.843320+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:26:16.843320+0000","last_undegraded":"2024-03-06T07:26:16.843320+0000","last_fullsized":"2024-03-06T07:26:16.843320+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876761+0000","last_change":"2024-03-06T07:26:16.866300+0000","last_active":"2024-03-06T07:26:18.876761+0000","last_peered":"2024-03-06T07:26:18.876761+0000","last_clean":"2024-03-06T07:26:18.876761+0000","last_became_active":"2024-03-06T07:26:16.866014+0000","last_became_peered":"2024-03-06T07:26:16.866014+0000","last_unstale":"2024-03-06T07:26:18.876761+0000","last_undegraded":"2024-03-06T07:26:18.876761+0000","last_fullsized":"2024-03-06T07:26:18.876761+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:31.226660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877031+0000","last_change":"2024-03-06T07:26:17.874843+0000","last_active":"2024-03-06T07:26:18.877031+0000","last_peered":"2024-03-06T07:26:18.877031+0000","last_clean":"2024-03-06T07:26:18.877031+0000","last_became_active":"2024-03-06T07:26:17.874530+0000","last_became_peered":"2024-03-06T07:26:17.874530+0000","last_unstale":"2024-03-06T07:26:18.877031+0000","last_undegraded":"2024-03-06T07:26:18.877031+0000","last_fullsized":"2024-03-06T07:26:18.877031+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:22.620099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631513+0000","last_change":"2024-03-06T07:26:16.870244+0000","last_active":"2024-03-06T07:26:19.631513+0000","last_peered":"2024-03-06T07:26:19.631513+0000","last_clean":"2024-03-06T07:26:19.631513+0000","last_became_active":"2024-03-06T07:26:16.869996+0000","last_became_peered":"2024-03-06T07:26:16.869996+0000","last_unstale":"2024-03-06T07:26:19.631513+0000","last_undegraded":"2024-03-06T07:26:19.631513+0000","last_fullsized":"2024-03-06T07:26:19.631513+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:46.843920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":301,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631596+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:26:19.631596+0000","last_peered":"2024-03-06T07:26:19.631596+0000","last_clean":"2024-03-06T07:26:19.631596+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:26:19.631596+0000","last_undegraded":"2024-03-06T07:26:19.631596+0000","last_fullsized":"2024-03-06T07:26:19.631596+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":110,"acting":110,"num_store_stats":7},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2340,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":1076018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1101824,"data_stored":466188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1142784,"data_stored":485608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1159168,"data_stored":487274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1155072,"data_stored":487172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1142784,"data_stored":485572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1138688,"data_stored":485538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1736704,"data_stored":1076488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:25.351 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:25.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.463+0000 7f7ef6a42640 1 Processor -- start 2024-03-06T07:26:25.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.463+0000 7f7ef6a42640 1 -- start start 2024-03-06T07:26:25.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7ef6a42640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef00cdeb0 0x7f7ef015b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7ef6a42640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef0071f40 con 0x7f7ef00cce40 2024-03-06T07:26:25.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7ef6a42640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef0070a70 con 0x7f7ef01515e0 2024-03-06T07:26:25.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7ef6a42640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef00cd2f0 con 0x7f7ef00cdeb0 2024-03-06T07:26:25.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7eef7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7ef00cce40 0x7f7ef00cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49904/0 (socket says 172.21.15.59:49904) 2024-03-06T07:26:25.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7eef7fe640 1 -- 172.21.15.59:0/2157734397 learned_addr learned my addr 172.21.15.59:0/2157734397 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:25.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7ef4fdf640 1 --2- 172.21.15.59:0/2157734397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef00cdeb0 0x7f7ef015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7eeeffd640 1 -- 172.21.15.59:0/2157734397 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234096255 0 0) 0x7f7ef0071f40 con 0x7f7ef00cce40 2024-03-06T07:26:25.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7eeeffd640 1 -- 172.21.15.59:0/2157734397 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed8003520 con 0x7f7ef00cce40 2024-03-06T07:26:25.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7eeeffd640 1 -- 172.21.15.59:0/2157734397 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619459766 0 0) 0x7f7ef0070a70 con 0x7f7ef01515e0 2024-03-06T07:26:25.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7eeeffd640 1 -- 172.21.15.59:0/2157734397 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef0071f40 con 0x7f7ef01515e0 2024-03-06T07:26:25.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7ef4fdf640 1 -- 172.21.15.59:0/2157734397 >> v1:172.21.15.164:6789/0 conn(0x7f7ef01515e0 legacy=0x7f7ef0071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7ef4fdf640 1 -- 172.21.15.59:0/2157734397 >> v1:172.21.15.59:6789/0 conn(0x7f7ef00cce40 legacy=0x7f7ef00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.464+0000 7f7ef4fdf640 1 -- 172.21.15.59:0/2157734397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef0070d50 con 0x7f7ef00cdeb0 2024-03-06T07:26:25.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.465+0000 7f7ef4fdf640 1 --2- 172.21.15.59:0/2157734397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef00cdeb0 0x7f7ef015b430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800aed0 tx=0x7f7ee802e120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66dce65dc5be2148 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.466+0000 7f7eeeffd640 1 -- 172.21.15.59:0/2157734397 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee802ecc0 con 0x7f7ef00cdeb0 2024-03-06T07:26:25.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.466+0000 7f7eeeffd640 1 -- 172.21.15.59:0/2157734397 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee8008440 con 0x7f7ef00cdeb0 2024-03-06T07:26:25.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.466+0000 7f7eeeffd640 1 -- 172.21.15.59:0/2157734397 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee802ee60 con 0x7f7ef00cdeb0 2024-03-06T07:26:25.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.466+0000 7f7ef6a42640 1 -- 172.21.15.59:0/2157734397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef00cdeb0 msgr2=0x7f7ef015b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.466+0000 7f7ef6a42640 1 --2- 172.21.15.59:0/2157734397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef00cdeb0 0x7f7ef015b430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800aed0 tx=0x7f7ee802e120 comp rx=0 tx=0).stop 2024-03-06T07:26:25.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.466+0000 7f7ef6a42640 1 -- 172.21.15.59:0/2157734397 shutdown_connections 2024-03-06T07:26:25.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.466+0000 7f7ef6a42640 1 --2- 172.21.15.59:0/2157734397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef00cdeb0 0x7f7ef015b430 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.466+0000 7f7ef6a42640 1 -- 172.21.15.59:0/2157734397 >> 172.21.15.59:0/2157734397 conn(0x7f7ef00cf8a0 msgr2=0x7f7ef00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.466+0000 7f7ef6a42640 1 -- 172.21.15.59:0/2157734397 shutdown_connections 2024-03-06T07:26:25.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.466+0000 7f7ef6a42640 1 -- 172.21.15.59:0/2157734397 wait complete. 2024-03-06T07:26:25.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.467+0000 7f7ef6a42640 1 Processor -- start 2024-03-06T07:26:25.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7ef6a42640 1 -- start start 2024-03-06T07:26:25.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7ef6a42640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef00cce40 0x7f7ef01f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7ef6a42640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ef00cdeb0 0x7f7ef0159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7ef6a42640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef01515e0 0x7f7ef014dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7ef6a42640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef00cc3a0 con 0x7f7ef00cce40 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7ef6a42640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef0071310 con 0x7f7ef00cdeb0 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7ef6a42640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef00cc520 con 0x7f7ef01515e0 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7eeffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef00cce40 0x7f7ef01f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7eeffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef00cce40 0x7f7ef01f4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60530/0 (socket says 172.21.15.59:60530) 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7eeffff640 1 -- 172.21.15.59:0/4176495337 learned_addr learned my addr 172.21.15.59:0/4176495337 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7ef4fdf640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef01515e0 0x7f7ef014dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.468+0000 7f7eef7fe640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ef00cdeb0 0x7f7ef0159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.469+0000 7f7eeffff640 1 -- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef01515e0 msgr2=0x7f7ef014dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.469+0000 7f7eeffff640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef01515e0 0x7f7ef014dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.469+0000 7f7eeffff640 1 -- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ef00cdeb0 msgr2=0x7f7ef0159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.469+0000 7f7eeffff640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ef00cdeb0 0x7f7ef0159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.469+0000 7f7eeffff640 1 -- 172.21.15.59:0/4176495337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef01598a0 con 0x7f7ef00cce40 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.469+0000 7f7eeffff640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef00cce40 0x7f7ef01f4b10 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c430 tx=0x7f7ee400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.469+0000 7f7eecff9640 1 -- 172.21.15.59:0/4176495337 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4017070 con 0x7f7ef00cce40 2024-03-06T07:26:25.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.469+0000 7f7eecff9640 1 -- 172.21.15.59:0/4176495337 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee40052f0 con 0x7f7ef00cce40 2024-03-06T07:26:25.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.469+0000 7f7eecff9640 1 -- 172.21.15.59:0/4176495337 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4003830 con 0x7f7ef00cce40 2024-03-06T07:26:25.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.469+0000 7f7ef6a42640 1 -- 172.21.15.59:0/4176495337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef01f7a60 con 0x7f7ef00cce40 2024-03-06T07:26:25.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.469+0000 7f7ef6a42640 1 -- 172.21.15.59:0/4176495337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef01f8020 con 0x7f7ef00cce40 2024-03-06T07:26:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.470+0000 7f7ef6a42640 1 -- 172.21.15.59:0/4176495337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef0068840 con 0x7f7ef00cce40 2024-03-06T07:26:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.470+0000 7f7eecff9640 1 -- 172.21.15.59:0/4176495337 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ee401a070 con 0x7f7ef00cce40 2024-03-06T07:26:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.471+0000 7f7eecff9640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7edc06b4d0 0x7f7edc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.471+0000 7f7eecff9640 1 -- 172.21.15.59:0/4176495337 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ee40894f0 con 0x7f7ef00cce40 2024-03-06T07:26:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.471+0000 7f7eef7fe640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7edc06b4d0 0x7f7edc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.472+0000 7f7eef7fe640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7edc06b4d0 0x7f7edc06d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7ef006a2f0 tx=0x7f7ee0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.475+0000 7f7eecff9640 1 -- 172.21.15.59:0/4176495337 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ee4057d20 con 0x7f7ef00cce40 2024-03-06T07:26:25.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.651+0000 7f7ef6a42640 1 -- 172.21.15.59:0/4176495337 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7ef004f330 con 0x7f7edc06b4d0 2024-03-06T07:26:25.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.658+0000 7f7eecff9640 1 -- 172.21.15.59:0/4176495337 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252086 (secure 0 0 0) 0x7f7ef004f330 con 0x7f7edc06b4d0 2024-03-06T07:26:25.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:25.663 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:26:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 -- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7edc06b4d0 msgr2=0x7f7edc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7edc06b4d0 0x7f7edc06d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7ef006a2f0 tx=0x7f7ee0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 -- 172.21.15.59:0/4176495337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef00cce40 msgr2=0x7f7ef01f4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef00cce40 0x7f7ef01f4b10 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f7ee400c430 tx=0x7f7ee400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 -- 172.21.15.59:0/4176495337 shutdown_connections 2024-03-06T07:26:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7edc06b4d0 0x7f7edc06d990 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef01515e0 0x7f7ef014dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ef00cdeb0 0x7f7ef0159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 --2- 172.21.15.59:0/4176495337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef00cce40 0x7f7ef01f4b10 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 -- 172.21.15.59:0/4176495337 >> 172.21.15.59:0/4176495337 conn(0x7f7ef00cf8a0 msgr2=0x7f7ef014d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 -- 172.21.15.59:0/4176495337 shutdown_connections 2024-03-06T07:26:25.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.660+0000 7f7ef6a42640 1 -- 172.21.15.59:0/4176495337 wait complete. 2024-03-06T07:26:25.673 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-03-06T07:26:25.359867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":179,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259592,"kb_used_data":10120,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715128,"statfs":{"total":773094113280,"available":772828291072,"internally_reserved":0,"allocated":10362880,"data_stored":5069860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001303"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854775+0000","last_change":"2024-03-06T07:26:17.877929+0000","last_active":"2024-03-06T07:26:18.854775+0000","last_peered":"2024-03-06T07:26:18.854775+0000","last_clean":"2024-03-06T07:26:18.854775+0000","last_became_active":"2024-03-06T07:26:17.877642+0000","last_became_peered":"2024-03-06T07:26:17.877642+0000","last_unstale":"2024-03-06T07:26:18.854775+0000","last_undegraded":"2024-03-06T07:26:18.854775+0000","last_fullsized":"2024-03-06T07:26:18.854775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:53.393641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875012+0000","last_change":"2024-03-06T07:26:17.868113+0000","last_active":"2024-03-06T07:26:18.875012+0000","last_peered":"2024-03-06T07:26:18.875012+0000","last_clean":"2024-03-06T07:26:18.875012+0000","last_became_active":"2024-03-06T07:26:17.867743+0000","last_became_peered":"2024-03-06T07:26:17.867743+0000","last_unstale":"2024-03-06T07:26:18.875012+0000","last_undegraded":"2024-03-06T07:26:18.875012+0000","last_fullsized":"2024-03-06T07:26:18.875012+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:57.329605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854687+0000","last_change":"2024-03-06T07:26:17.878201+0000","last_active":"2024-03-06T07:26:18.854687+0000","last_peered":"2024-03-06T07:26:18.854687+0000","last_clean":"2024-03-06T07:26:18.854687+0000","last_became_active":"2024-03-06T07:26:17.877378+0000","last_became_peered":"2024-03-06T07:26:17.877378+0000","last_unstale":"2024-03-06T07:26:18.854687+0000","last_undegraded":"2024-03-06T07:26:18.854687+0000","last_fullsized":"2024-03-06T07:26:18.854687+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:41.440891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855757+0000","last_change":"2024-03-06T07:26:17.871063+0000","last_active":"2024-03-06T07:26:18.855757+0000","last_peered":"2024-03-06T07:26:18.855757+0000","last_clean":"2024-03-06T07:26:18.855757+0000","last_became_active":"2024-03-06T07:26:17.870049+0000","last_became_peered":"2024-03-06T07:26:17.870049+0000","last_unstale":"2024-03-06T07:26:18.855757+0000","last_undegraded":"2024-03-06T07:26:18.855757+0000","last_fullsized":"2024-03-06T07:26:18.855757+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:43.505953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631941+0000","last_change":"2024-03-06T07:26:17.872458+0000","last_active":"2024-03-06T07:26:19.631941+0000","last_peered":"2024-03-06T07:26:19.631941+0000","last_clean":"2024-03-06T07:26:19.631941+0000","last_became_active":"2024-03-06T07:26:17.872187+0000","last_became_peered":"2024-03-06T07:26:17.872187+0000","last_unstale":"2024-03-06T07:26:19.631941+0000","last_undegraded":"2024-03-06T07:26:19.631941+0000","last_fullsized":"2024-03-06T07:26:19.631941+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:41.440891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631500+0000","last_change":"2024-03-06T07:26:17.864205+0000","last_active":"2024-03-06T07:26:19.631500+0000","last_peered":"2024-03-06T07:26:19.631500+0000","last_clean":"2024-03-06T07:26:19.631500+0000","last_became_active":"2024-03-06T07:26:17.863941+0000","last_became_peered":"2024-03-06T07:26:17.863941+0000","last_unstale":"2024-03-06T07:26:19.631500+0000","last_undegraded":"2024-03-06T07:26:19.631500+0000","last_fullsized":"2024-03-06T07:26:19.631500+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:03.507471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631356+0000","last_change":"2024-03-06T07:26:17.861310+0000","last_active":"2024-03-06T07:26:19.631356+0000","last_peered":"2024-03-06T07:26:19.631356+0000","last_clean":"2024-03-06T07:26:19.631356+0000","last_became_active":"2024-03-06T07:26:17.860944+0000","last_became_peered":"2024-03-06T07:26:17.860944+0000","last_unstale":"2024-03-06T07:26:19.631356+0000","last_undegraded":"2024-03-06T07:26:19.631356+0000","last_fullsized":"2024-03-06T07:26:19.631356+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:55.503575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874629+0000","last_change":"2024-03-06T07:26:17.861435+0000","last_active":"2024-03-06T07:26:18.874629+0000","last_peered":"2024-03-06T07:26:18.874629+0000","last_clean":"2024-03-06T07:26:18.874629+0000","last_became_active":"2024-03-06T07:26:17.861014+0000","last_became_peered":"2024-03-06T07:26:17.861014+0000","last_unstale":"2024-03-06T07:26:18.874629+0000","last_undegraded":"2024-03-06T07:26:18.874629+0000","last_fullsized":"2024-03-06T07:26:18.874629+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:54.987454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632809+0000","last_change":"2024-03-06T07:26:17.873920+0000","last_active":"2024-03-06T07:26:19.632809+0000","last_peered":"2024-03-06T07:26:19.632809+0000","last_clean":"2024-03-06T07:26:19.632809+0000","last_became_active":"2024-03-06T07:26:17.872835+0000","last_became_peered":"2024-03-06T07:26:17.872835+0000","last_unstale":"2024-03-06T07:26:19.632809+0000","last_undegraded":"2024-03-06T07:26:19.632809+0000","last_fullsized":"2024-03-06T07:26:19.632809+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:58.609019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631947+0000","last_change":"2024-03-06T07:26:17.876670+0000","last_active":"2024-03-06T07:26:19.631947+0000","last_peered":"2024-03-06T07:26:19.631947+0000","last_clean":"2024-03-06T07:26:19.631947+0000","last_became_active":"2024-03-06T07:26:17.876399+0000","last_became_peered":"2024-03-06T07:26:17.876399+0000","last_unstale":"2024-03-06T07:26:19.631947+0000","last_undegraded":"2024-03-06T07:26:19.631947+0000","last_fullsized":"2024-03-06T07:26:19.631947+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:31.403582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631919+0000","last_change":"2024-03-06T07:26:17.874590+0000","last_active":"2024-03-06T07:26:19.631919+0000","last_peered":"2024-03-06T07:26:19.631919+0000","last_clean":"2024-03-06T07:26:19.631919+0000","last_became_active":"2024-03-06T07:26:17.874324+0000","last_became_peered":"2024-03-06T07:26:17.874324+0000","last_unstale":"2024-03-06T07:26:19.631919+0000","last_undegraded":"2024-03-06T07:26:19.631919+0000","last_fullsized":"2024-03-06T07:26:19.631919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:26.884759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632664+0000","last_change":"2024-03-06T07:26:17.875263+0000","last_active":"2024-03-06T07:26:19.632664+0000","last_peered":"2024-03-06T07:26:19.632664+0000","last_clean":"2024-03-06T07:26:19.632664+0000","last_became_active":"2024-03-06T07:26:17.874730+0000","last_became_peered":"2024-03-06T07:26:17.874730+0000","last_unstale":"2024-03-06T07:26:19.632664+0000","last_undegraded":"2024-03-06T07:26:19.632664+0000","last_fullsized":"2024-03-06T07:26:19.632664+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:56:55.503575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855537+0000","last_change":"2024-03-06T07:26:17.876108+0000","last_active":"2024-03-06T07:26:18.855537+0000","last_peered":"2024-03-06T07:26:18.855537+0000","last_clean":"2024-03-06T07:26:18.855537+0000","last_became_active":"2024-03-06T07:26:17.875830+0000","last_became_peered":"2024-03-06T07:26:17.875830+0000","last_unstale":"2024-03-06T07:26:18.855537+0000","last_undegraded":"2024-03-06T07:26:18.855537+0000","last_fullsized":"2024-03-06T07:26:18.855537+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:57.406246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876753+0000","last_change":"2024-03-06T07:26:17.868274+0000","last_active":"2024-03-06T07:26:18.876753+0000","last_peered":"2024-03-06T07:26:18.876753+0000","last_clean":"2024-03-06T07:26:18.876753+0000","last_became_active":"2024-03-06T07:26:17.867626+0000","last_became_peered":"2024-03-06T07:26:17.867626+0000","last_unstale":"2024-03-06T07:26:18.876753+0000","last_undegraded":"2024-03-06T07:26:18.876753+0000","last_fullsized":"2024-03-06T07:26:18.876753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:16.118134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855430+0000","last_change":"2024-03-06T07:26:17.870824+0000","last_active":"2024-03-06T07:26:18.855430+0000","last_peered":"2024-03-06T07:26:18.855430+0000","last_clean":"2024-03-06T07:26:18.855430+0000","last_became_active":"2024-03-06T07:26:17.870545+0000","last_became_peered":"2024-03-06T07:26:17.870545+0000","last_unstale":"2024-03-06T07:26:18.855430+0000","last_undegraded":"2024-03-06T07:26:18.855430+0000","last_fullsized":"2024-03-06T07:26:18.855430+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:22.063664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631403+0000","last_change":"2024-03-06T07:26:17.861351+0000","last_active":"2024-03-06T07:26:19.631403+0000","last_peered":"2024-03-06T07:26:19.631403+0000","last_clean":"2024-03-06T07:26:19.631403+0000","last_became_active":"2024-03-06T07:26:17.861004+0000","last_became_peered":"2024-03-06T07:26:17.861004+0000","last_unstale":"2024-03-06T07:26:19.631403+0000","last_undegraded":"2024-03-06T07:26:19.631403+0000","last_fullsized":"2024-03-06T07:26:19.631403+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:58.609019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874995+0000","last_change":"2024-03-06T07:26:17.869794+0000","last_active":"2024-03-06T07:26:18.874995+0000","last_peered":"2024-03-06T07:26:18.874995+0000","last_clean":"2024-03-06T07:26:18.874995+0000","last_became_active":"2024-03-06T07:26:17.869531+0000","last_became_peered":"2024-03-06T07:26:17.869531+0000","last_unstale":"2024-03-06T07:26:18.874995+0000","last_undegraded":"2024-03-06T07:26:18.874995+0000","last_fullsized":"2024-03-06T07:26:18.874995+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:40:43.505953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855708+0000","last_change":"2024-03-06T07:26:17.871014+0000","last_active":"2024-03-06T07:26:18.855708+0000","last_peered":"2024-03-06T07:26:18.855708+0000","last_clean":"2024-03-06T07:26:18.855708+0000","last_became_active":"2024-03-06T07:26:17.870687+0000","last_became_peered":"2024-03-06T07:26:17.870687+0000","last_unstale":"2024-03-06T07:26:18.855708+0000","last_undegraded":"2024-03-06T07:26:18.855708+0000","last_fullsized":"2024-03-06T07:26:18.855708+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:54.987454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877113+0000","last_change":"2024-03-06T07:26:17.873014+0000","last_active":"2024-03-06T07:26:18.877113+0000","last_peered":"2024-03-06T07:26:18.877113+0000","last_clean":"2024-03-06T07:26:18.877113+0000","last_became_active":"2024-03-06T07:26:17.872701+0000","last_became_peered":"2024-03-06T07:26:17.872701+0000","last_unstale":"2024-03-06T07:26:18.877113+0000","last_undegraded":"2024-03-06T07:26:18.877113+0000","last_fullsized":"2024-03-06T07:26:18.877113+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:38.541605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632622+0000","last_change":"2024-03-06T07:26:17.877086+0000","last_active":"2024-03-06T07:26:19.632622+0000","last_peered":"2024-03-06T07:26:19.632622+0000","last_clean":"2024-03-06T07:26:19.632622+0000","last_became_active":"2024-03-06T07:26:17.875124+0000","last_became_peered":"2024-03-06T07:26:17.875124+0000","last_unstale":"2024-03-06T07:26:19.632622+0000","last_undegraded":"2024-03-06T07:26:19.632622+0000","last_fullsized":"2024-03-06T07:26:19.632622+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:36.314409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876998+0000","last_change":"2024-03-06T07:26:17.862241+0000","last_active":"2024-03-06T07:26:18.876998+0000","last_peered":"2024-03-06T07:26:18.876998+0000","last_clean":"2024-03-06T07:26:18.876998+0000","last_became_active":"2024-03-06T07:26:17.861340+0000","last_became_peered":"2024-03-06T07:26:17.861340+0000","last_unstale":"2024-03-06T07:26:18.876998+0000","last_undegraded":"2024-03-06T07:26:18.876998+0000","last_fullsized":"2024-03-06T07:26:18.876998+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:22.664538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874576+0000","last_change":"2024-03-06T07:26:17.861898+0000","last_active":"2024-03-06T07:26:18.874576+0000","last_peered":"2024-03-06T07:26:18.874576+0000","last_clean":"2024-03-06T07:26:18.874576+0000","last_became_active":"2024-03-06T07:26:17.861634+0000","last_became_peered":"2024-03-06T07:26:17.861634+0000","last_unstale":"2024-03-06T07:26:18.874576+0000","last_undegraded":"2024-03-06T07:26:18.874576+0000","last_fullsized":"2024-03-06T07:26:18.874576+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:57.406246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874586+0000","last_change":"2024-03-06T07:26:17.865417+0000","last_active":"2024-03-06T07:26:18.874586+0000","last_peered":"2024-03-06T07:26:18.874586+0000","last_clean":"2024-03-06T07:26:18.874586+0000","last_became_active":"2024-03-06T07:26:17.865139+0000","last_became_peered":"2024-03-06T07:26:17.865139+0000","last_unstale":"2024-03-06T07:26:18.874586+0000","last_undegraded":"2024-03-06T07:26:18.874586+0000","last_fullsized":"2024-03-06T07:26:18.874586+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.528335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631549+0000","last_change":"2024-03-06T07:26:17.868477+0000","last_active":"2024-03-06T07:26:19.631549+0000","last_peered":"2024-03-06T07:26:19.631549+0000","last_clean":"2024-03-06T07:26:19.631549+0000","last_became_active":"2024-03-06T07:26:17.867943+0000","last_became_peered":"2024-03-06T07:26:17.867943+0000","last_unstale":"2024-03-06T07:26:19.631549+0000","last_undegraded":"2024-03-06T07:26:19.631549+0000","last_fullsized":"2024-03-06T07:26:19.631549+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:02.151317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632145+0000","last_change":"2024-03-06T07:26:17.871658+0000","last_active":"2024-03-06T07:26:19.632145+0000","last_peered":"2024-03-06T07:26:19.632145+0000","last_clean":"2024-03-06T07:26:19.632145+0000","last_became_active":"2024-03-06T07:26:17.871402+0000","last_became_peered":"2024-03-06T07:26:17.871402+0000","last_unstale":"2024-03-06T07:26:19.632145+0000","last_undegraded":"2024-03-06T07:26:19.632145+0000","last_fullsized":"2024-03-06T07:26:19.632145+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:59.170419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875070+0000","last_change":"2024-03-06T07:26:17.873693+0000","last_active":"2024-03-06T07:26:18.875070+0000","last_peered":"2024-03-06T07:26:18.875070+0000","last_clean":"2024-03-06T07:26:18.875070+0000","last_became_active":"2024-03-06T07:26:17.873363+0000","last_became_peered":"2024-03-06T07:26:17.873363+0000","last_unstale":"2024-03-06T07:26:18.875070+0000","last_undegraded":"2024-03-06T07:26:18.875070+0000","last_fullsized":"2024-03-06T07:26:18.875070+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:05:06.936077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632055+0000","last_change":"2024-03-06T07:26:17.864951+0000","last_active":"2024-03-06T07:26:19.632055+0000","last_peered":"2024-03-06T07:26:19.632055+0000","last_clean":"2024-03-06T07:26:19.632055+0000","last_became_active":"2024-03-06T07:26:17.864683+0000","last_became_peered":"2024-03-06T07:26:17.864683+0000","last_unstale":"2024-03-06T07:26:19.632055+0000","last_undegraded":"2024-03-06T07:26:19.632055+0000","last_fullsized":"2024-03-06T07:26:19.632055+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:46.261854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631949+0000","last_change":"2024-03-06T07:26:17.873096+0000","last_active":"2024-03-06T07:26:19.631949+0000","last_peered":"2024-03-06T07:26:19.631949+0000","last_clean":"2024-03-06T07:26:19.631949+0000","last_became_active":"2024-03-06T07:26:17.872843+0000","last_became_peered":"2024-03-06T07:26:17.872843+0000","last_unstale":"2024-03-06T07:26:19.631949+0000","last_undegraded":"2024-03-06T07:26:19.631949+0000","last_fullsized":"2024-03-06T07:26:19.631949+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:53.393641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874766+0000","last_change":"2024-03-06T07:26:17.873122+0000","last_active":"2024-03-06T07:26:18.874766+0000","last_peered":"2024-03-06T07:26:18.874766+0000","last_clean":"2024-03-06T07:26:18.874766+0000","last_became_active":"2024-03-06T07:26:17.872806+0000","last_became_peered":"2024-03-06T07:26:17.872806+0000","last_unstale":"2024-03-06T07:26:18.874766+0000","last_undegraded":"2024-03-06T07:26:18.874766+0000","last_fullsized":"2024-03-06T07:26:18.874766+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:08.056201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876785+0000","last_change":"2024-03-06T07:26:17.868507+0000","last_active":"2024-03-06T07:26:18.876785+0000","last_peered":"2024-03-06T07:26:18.876785+0000","last_clean":"2024-03-06T07:26:18.876785+0000","last_became_active":"2024-03-06T07:26:17.868006+0000","last_became_peered":"2024-03-06T07:26:17.868006+0000","last_unstale":"2024-03-06T07:26:18.876785+0000","last_undegraded":"2024-03-06T07:26:18.876785+0000","last_fullsized":"2024-03-06T07:26:18.876785+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:10.435503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855724+0000","last_change":"2024-03-06T07:26:17.879833+0000","last_active":"2024-03-06T07:26:18.855724+0000","last_peered":"2024-03-06T07:26:18.855724+0000","last_clean":"2024-03-06T07:26:18.855724+0000","last_became_active":"2024-03-06T07:26:17.878938+0000","last_became_peered":"2024-03-06T07:26:17.878938+0000","last_unstale":"2024-03-06T07:26:18.855724+0000","last_undegraded":"2024-03-06T07:26:18.855724+0000","last_fullsized":"2024-03-06T07:26:18.855724+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:08.056201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854529+0000","last_change":"2024-03-06T07:26:17.866962+0000","last_active":"2024-03-06T07:26:18.854529+0000","last_peered":"2024-03-06T07:26:18.854529+0000","last_clean":"2024-03-06T07:26:18.854529+0000","last_became_active":"2024-03-06T07:26:17.866658+0000","last_became_peered":"2024-03-06T07:26:17.866658+0000","last_unstale":"2024-03-06T07:26:18.854529+0000","last_undegraded":"2024-03-06T07:26:18.854529+0000","last_fullsized":"2024-03-06T07:26:18.854529+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:25.582364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877006+0000","last_change":"2024-03-06T07:26:17.860951+0000","last_active":"2024-03-06T07:26:18.877006+0000","last_peered":"2024-03-06T07:26:18.877006+0000","last_clean":"2024-03-06T07:26:18.877006+0000","last_became_active":"2024-03-06T07:26:17.860689+0000","last_became_peered":"2024-03-06T07:26:17.860689+0000","last_unstale":"2024-03-06T07:26:18.877006+0000","last_undegraded":"2024-03-06T07:26:18.877006+0000","last_fullsized":"2024-03-06T07:26:18.877006+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:50.018552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"91'4","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874923+0000","last_change":"2024-03-06T07:26:16.861346+0000","last_active":"2024-03-06T07:26:18.874923+0000","last_peered":"2024-03-06T07:26:18.874923+0000","last_clean":"2024-03-06T07:26:18.874923+0000","last_became_active":"2024-03-06T07:26:16.860585+0000","last_became_peered":"2024-03-06T07:26:16.860585+0000","last_unstale":"2024-03-06T07:26:18.874923+0000","last_undegraded":"2024-03-06T07:26:18.874923+0000","last_fullsized":"2024-03-06T07:26:18.874923+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:06.710593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855827+0000","last_change":"2024-03-06T07:26:17.879842+0000","last_active":"2024-03-06T07:26:18.855827+0000","last_peered":"2024-03-06T07:26:18.855827+0000","last_clean":"2024-03-06T07:26:18.855827+0000","last_became_active":"2024-03-06T07:26:17.879291+0000","last_became_peered":"2024-03-06T07:26:17.879291+0000","last_unstale":"2024-03-06T07:26:18.855827+0000","last_undegraded":"2024-03-06T07:26:18.855827+0000","last_fullsized":"2024-03-06T07:26:18.855827+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:57.329605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631480+0000","last_change":"2024-03-06T07:26:16.868319+0000","last_active":"2024-03-06T07:26:19.631480+0000","last_peered":"2024-03-06T07:26:19.631480+0000","last_clean":"2024-03-06T07:26:19.631480+0000","last_became_active":"2024-03-06T07:26:16.868043+0000","last_became_peered":"2024-03-06T07:26:16.868043+0000","last_unstale":"2024-03-06T07:26:19.631480+0000","last_undegraded":"2024-03-06T07:26:19.631480+0000","last_fullsized":"2024-03-06T07:26:19.631480+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:25.302742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854455+0000","last_change":"2024-03-06T07:26:17.875745+0000","last_active":"2024-03-06T07:26:18.854455+0000","last_peered":"2024-03-06T07:26:18.854455+0000","last_clean":"2024-03-06T07:26:18.854455+0000","last_became_active":"2024-03-06T07:26:17.875471+0000","last_became_peered":"2024-03-06T07:26:17.875471+0000","last_unstale":"2024-03-06T07:26:18.854455+0000","last_undegraded":"2024-03-06T07:26:18.854455+0000","last_fullsized":"2024-03-06T07:26:18.854455+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:47.848728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874838+0000","last_change":"2024-03-06T07:26:16.870929+0000","last_active":"2024-03-06T07:26:18.874838+0000","last_peered":"2024-03-06T07:26:18.874838+0000","last_clean":"2024-03-06T07:26:18.874838+0000","last_became_active":"2024-03-06T07:26:16.870661+0000","last_became_peered":"2024-03-06T07:26:16.870661+0000","last_unstale":"2024-03-06T07:26:18.874838+0000","last_undegraded":"2024-03-06T07:26:18.874838+0000","last_fullsized":"2024-03-06T07:26:18.874838+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:13.611316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632637+0000","last_change":"2024-03-06T07:26:17.872067+0000","last_active":"2024-03-06T07:26:19.632637+0000","last_peered":"2024-03-06T07:26:19.632637+0000","last_clean":"2024-03-06T07:26:19.632637+0000","last_became_active":"2024-03-06T07:26:17.871800+0000","last_became_peered":"2024-03-06T07:26:17.871800+0000","last_unstale":"2024-03-06T07:26:19.632637+0000","last_undegraded":"2024-03-06T07:26:19.632637+0000","last_fullsized":"2024-03-06T07:26:19.632637+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:27.404185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874719+0000","last_change":"2024-03-06T07:26:16.864464+0000","last_active":"2024-03-06T07:26:18.874719+0000","last_peered":"2024-03-06T07:26:18.874719+0000","last_clean":"2024-03-06T07:26:18.874719+0000","last_became_active":"2024-03-06T07:26:16.863949+0000","last_became_peered":"2024-03-06T07:26:16.863949+0000","last_unstale":"2024-03-06T07:26:18.874719+0000","last_undegraded":"2024-03-06T07:26:18.874719+0000","last_fullsized":"2024-03-06T07:26:18.874719+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:52.388832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856105+0000","last_change":"2024-03-06T07:26:17.872420+0000","last_active":"2024-03-06T07:26:18.856105+0000","last_peered":"2024-03-06T07:26:18.856105+0000","last_clean":"2024-03-06T07:26:18.856105+0000","last_became_active":"2024-03-06T07:26:17.872122+0000","last_became_peered":"2024-03-06T07:26:17.872122+0000","last_unstale":"2024-03-06T07:26:18.856105+0000","last_undegraded":"2024-03-06T07:26:18.856105+0000","last_fullsized":"2024-03-06T07:26:18.856105+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:15.723196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854281+0000","last_change":"2024-03-06T07:26:16.863634+0000","last_active":"2024-03-06T07:26:18.854281+0000","last_peered":"2024-03-06T07:26:18.854281+0000","last_clean":"2024-03-06T07:26:18.854281+0000","last_became_active":"2024-03-06T07:26:16.863372+0000","last_became_peered":"2024-03-06T07:26:16.863372+0000","last_unstale":"2024-03-06T07:26:18.854281+0000","last_undegraded":"2024-03-06T07:26:18.854281+0000","last_fullsized":"2024-03-06T07:26:18.854281+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:29.981611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631644+0000","last_change":"2024-03-06T07:26:17.871415+0000","last_active":"2024-03-06T07:26:19.631644+0000","last_peered":"2024-03-06T07:26:19.631644+0000","last_clean":"2024-03-06T07:26:19.631644+0000","last_became_active":"2024-03-06T07:26:17.871135+0000","last_became_peered":"2024-03-06T07:26:17.871135+0000","last_unstale":"2024-03-06T07:26:19.631644+0000","last_undegraded":"2024-03-06T07:26:19.631644+0000","last_fullsized":"2024-03-06T07:26:19.631644+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:36.291779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874580+0000","last_change":"2024-03-06T07:26:16.866013+0000","last_active":"2024-03-06T07:26:18.874580+0000","last_peered":"2024-03-06T07:26:18.874580+0000","last_clean":"2024-03-06T07:26:18.874580+0000","last_became_active":"2024-03-06T07:26:16.865737+0000","last_became_peered":"2024-03-06T07:26:16.865737+0000","last_unstale":"2024-03-06T07:26:18.874580+0000","last_undegraded":"2024-03-06T07:26:18.874580+0000","last_fullsized":"2024-03-06T07:26:18.874580+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:24.577556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855424+0000","last_change":"2024-03-06T07:26:17.871627+0000","last_active":"2024-03-06T07:26:18.855424+0000","last_peered":"2024-03-06T07:26:18.855424+0000","last_clean":"2024-03-06T07:26:18.855424+0000","last_became_active":"2024-03-06T07:26:17.871289+0000","last_became_peered":"2024-03-06T07:26:17.871289+0000","last_unstale":"2024-03-06T07:26:18.855424+0000","last_undegraded":"2024-03-06T07:26:18.855424+0000","last_fullsized":"2024-03-06T07:26:18.855424+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:52.299295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631598+0000","last_change":"2024-03-06T07:26:16.858748+0000","last_active":"2024-03-06T07:26:19.631598+0000","last_peered":"2024-03-06T07:26:19.631598+0000","last_clean":"2024-03-06T07:26:19.631598+0000","last_became_active":"2024-03-06T07:26:16.858490+0000","last_became_peered":"2024-03-06T07:26:16.858490+0000","last_unstale":"2024-03-06T07:26:19.631598+0000","last_undegraded":"2024-03-06T07:26:19.631598+0000","last_fullsized":"2024-03-06T07:26:19.631598+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:11.017436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632457+0000","last_change":"2024-03-06T07:26:17.873691+0000","last_active":"2024-03-06T07:26:19.632457+0000","last_peered":"2024-03-06T07:26:19.632457+0000","last_clean":"2024-03-06T07:26:19.632457+0000","last_became_active":"2024-03-06T07:26:17.873417+0000","last_became_peered":"2024-03-06T07:26:17.873417+0000","last_unstale":"2024-03-06T07:26:19.632457+0000","last_undegraded":"2024-03-06T07:26:19.632457+0000","last_fullsized":"2024-03-06T07:26:19.632457+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:59.170419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854190+0000","last_change":"2024-03-06T07:26:16.869647+0000","last_active":"2024-03-06T07:26:18.854190+0000","last_peered":"2024-03-06T07:26:18.854190+0000","last_clean":"2024-03-06T07:26:18.854190+0000","last_became_active":"2024-03-06T07:26:16.869365+0000","last_became_peered":"2024-03-06T07:26:16.869365+0000","last_unstale":"2024-03-06T07:26:18.854190+0000","last_undegraded":"2024-03-06T07:26:18.854190+0000","last_fullsized":"2024-03-06T07:26:18.854190+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:21.875952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875098+0000","last_change":"2024-03-06T07:26:17.870669+0000","last_active":"2024-03-06T07:26:18.875098+0000","last_peered":"2024-03-06T07:26:18.875098+0000","last_clean":"2024-03-06T07:26:18.875098+0000","last_became_active":"2024-03-06T07:26:17.870363+0000","last_became_peered":"2024-03-06T07:26:17.870363+0000","last_unstale":"2024-03-06T07:26:18.875098+0000","last_undegraded":"2024-03-06T07:26:18.875098+0000","last_fullsized":"2024-03-06T07:26:18.875098+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:47.379775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854820+0000","last_change":"2024-03-06T07:26:16.865155+0000","last_active":"2024-03-06T07:26:18.854820+0000","last_peered":"2024-03-06T07:26:18.854820+0000","last_clean":"2024-03-06T07:26:18.854820+0000","last_became_active":"2024-03-06T07:26:16.864873+0000","last_became_peered":"2024-03-06T07:26:16.864873+0000","last_unstale":"2024-03-06T07:26:18.854820+0000","last_undegraded":"2024-03-06T07:26:18.854820+0000","last_fullsized":"2024-03-06T07:26:18.854820+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:57.303182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632939+0000","last_change":"2024-03-06T07:26:17.874753+0000","last_active":"2024-03-06T07:26:19.632939+0000","last_peered":"2024-03-06T07:26:19.632939+0000","last_clean":"2024-03-06T07:26:19.632939+0000","last_became_active":"2024-03-06T07:26:17.874488+0000","last_became_peered":"2024-03-06T07:26:17.874488+0000","last_unstale":"2024-03-06T07:26:19.632939+0000","last_undegraded":"2024-03-06T07:26:19.632939+0000","last_fullsized":"2024-03-06T07:26:19.632939+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:03.507471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855695+0000","last_change":"2024-03-06T07:26:16.872968+0000","last_active":"2024-03-06T07:26:18.855695+0000","last_peered":"2024-03-06T07:26:18.855695+0000","last_clean":"2024-03-06T07:26:18.855695+0000","last_became_active":"2024-03-06T07:26:16.872657+0000","last_became_peered":"2024-03-06T07:26:16.872657+0000","last_unstale":"2024-03-06T07:26:18.855695+0000","last_undegraded":"2024-03-06T07:26:18.855695+0000","last_fullsized":"2024-03-06T07:26:18.855695+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:35.309600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632154+0000","last_change":"2024-03-06T07:26:17.866225+0000","last_active":"2024-03-06T07:26:19.632154+0000","last_peered":"2024-03-06T07:26:19.632154+0000","last_clean":"2024-03-06T07:26:19.632154+0000","last_became_active":"2024-03-06T07:26:17.865899+0000","last_became_peered":"2024-03-06T07:26:17.865899+0000","last_unstale":"2024-03-06T07:26:19.632154+0000","last_undegraded":"2024-03-06T07:26:19.632154+0000","last_fullsized":"2024-03-06T07:26:19.632154+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:07.715401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"91'3","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632226+0000","last_change":"2024-03-06T07:26:16.866799+0000","last_active":"2024-03-06T07:26:19.632226+0000","last_peered":"2024-03-06T07:26:19.632226+0000","last_clean":"2024-03-06T07:26:19.632226+0000","last_became_active":"2024-03-06T07:26:16.866520+0000","last_became_peered":"2024-03-06T07:26:16.866520+0000","last_unstale":"2024-03-06T07:26:19.632226+0000","last_undegraded":"2024-03-06T07:26:19.632226+0000","last_fullsized":"2024-03-06T07:26:19.632226+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:21.270064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632180+0000","last_change":"2024-03-06T07:26:17.863780+0000","last_active":"2024-03-06T07:26:19.632180+0000","last_peered":"2024-03-06T07:26:19.632180+0000","last_clean":"2024-03-06T07:26:19.632180+0000","last_became_active":"2024-03-06T07:26:17.863503+0000","last_became_peered":"2024-03-06T07:26:17.863503+0000","last_unstale":"2024-03-06T07:26:19.632180+0000","last_undegraded":"2024-03-06T07:26:19.632180+0000","last_fullsized":"2024-03-06T07:26:19.632180+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:43.263166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"91'6","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632748+0000","last_change":"2024-03-06T07:26:16.868702+0000","last_active":"2024-03-06T07:26:19.632748+0000","last_peered":"2024-03-06T07:26:19.632748+0000","last_clean":"2024-03-06T07:26:19.632748+0000","last_became_active":"2024-03-06T07:26:16.868364+0000","last_became_peered":"2024-03-06T07:26:16.868364+0000","last_unstale":"2024-03-06T07:26:19.632748+0000","last_undegraded":"2024-03-06T07:26:19.632748+0000","last_fullsized":"2024-03-06T07:26:19.632748+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:19.712353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855547+0000","last_change":"2024-03-06T07:26:17.871667+0000","last_active":"2024-03-06T07:26:18.855547+0000","last_peered":"2024-03-06T07:26:18.855547+0000","last_clean":"2024-03-06T07:26:18.855547+0000","last_became_active":"2024-03-06T07:26:17.871317+0000","last_became_peered":"2024-03-06T07:26:17.871317+0000","last_unstale":"2024-03-06T07:26:18.855547+0000","last_undegraded":"2024-03-06T07:26:18.855547+0000","last_fullsized":"2024-03-06T07:26:18.855547+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:03.528335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877108+0000","last_change":"2024-03-06T07:26:16.865944+0000","last_active":"2024-03-06T07:26:18.877108+0000","last_peered":"2024-03-06T07:26:18.877108+0000","last_clean":"2024-03-06T07:26:18.877108+0000","last_became_active":"2024-03-06T07:26:16.865672+0000","last_became_peered":"2024-03-06T07:26:16.865672+0000","last_unstale":"2024-03-06T07:26:18.877108+0000","last_undegraded":"2024-03-06T07:26:18.877108+0000","last_fullsized":"2024-03-06T07:26:18.877108+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:10:03.112514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854825+0000","last_change":"2024-03-06T07:26:17.867658+0000","last_active":"2024-03-06T07:26:18.854825+0000","last_peered":"2024-03-06T07:26:18.854825+0000","last_clean":"2024-03-06T07:26:18.854825+0000","last_became_active":"2024-03-06T07:26:17.867387+0000","last_became_peered":"2024-03-06T07:26:17.867387+0000","last_unstale":"2024-03-06T07:26:18.854825+0000","last_undegraded":"2024-03-06T07:26:18.854825+0000","last_fullsized":"2024-03-06T07:26:18.854825+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:14.616124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877004+0000","last_change":"2024-03-06T07:26:16.869461+0000","last_active":"2024-03-06T07:26:18.877004+0000","last_peered":"2024-03-06T07:26:18.877004+0000","last_clean":"2024-03-06T07:26:18.877004+0000","last_became_active":"2024-03-06T07:26:16.869142+0000","last_became_peered":"2024-03-06T07:26:16.869142+0000","last_unstale":"2024-03-06T07:26:18.877004+0000","last_undegraded":"2024-03-06T07:26:18.877004+0000","last_fullsized":"2024-03-06T07:26:18.877004+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:53.301305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631353+0000","last_change":"2024-03-06T07:26:17.875559+0000","last_active":"2024-03-06T07:26:19.631353+0000","last_peered":"2024-03-06T07:26:19.631353+0000","last_clean":"2024-03-06T07:26:19.631353+0000","last_became_active":"2024-03-06T07:26:17.875196+0000","last_became_peered":"2024-03-06T07:26:17.875196+0000","last_unstale":"2024-03-06T07:26:19.631353+0000","last_undegraded":"2024-03-06T07:26:19.631353+0000","last_fullsized":"2024-03-06T07:26:19.631353+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:14.616124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632305+0000","last_change":"2024-03-06T07:26:16.865532+0000","last_active":"2024-03-06T07:26:19.632305+0000","last_peered":"2024-03-06T07:26:19.632305+0000","last_clean":"2024-03-06T07:26:19.632305+0000","last_became_active":"2024-03-06T07:26:16.865038+0000","last_became_peered":"2024-03-06T07:26:16.865038+0000","last_unstale":"2024-03-06T07:26:19.632305+0000","last_undegraded":"2024-03-06T07:26:19.632305+0000","last_fullsized":"2024-03-06T07:26:19.632305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:18.180662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876751+0000","last_change":"2024-03-06T07:26:17.875784+0000","last_active":"2024-03-06T07:26:18.876751+0000","last_peered":"2024-03-06T07:26:18.876751+0000","last_clean":"2024-03-06T07:26:18.876751+0000","last_became_active":"2024-03-06T07:26:17.875509+0000","last_became_peered":"2024-03-06T07:26:17.875509+0000","last_unstale":"2024-03-06T07:26:18.876751+0000","last_undegraded":"2024-03-06T07:26:18.876751+0000","last_fullsized":"2024-03-06T07:26:18.876751+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:39.164839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"91'8","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854323+0000","last_change":"2024-03-06T07:26:16.866435+0000","last_active":"2024-03-06T07:26:18.854323+0000","last_peered":"2024-03-06T07:26:18.854323+0000","last_clean":"2024-03-06T07:26:18.854323+0000","last_became_active":"2024-03-06T07:26:16.866136+0000","last_became_peered":"2024-03-06T07:26:16.866136+0000","last_unstale":"2024-03-06T07:26:18.854323+0000","last_undegraded":"2024-03-06T07:26:18.854323+0000","last_fullsized":"2024-03-06T07:26:18.854323+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:18.526671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632212+0000","last_change":"2024-03-06T07:26:17.873578+0000","last_active":"2024-03-06T07:26:19.632212+0000","last_peered":"2024-03-06T07:26:19.632212+0000","last_clean":"2024-03-06T07:26:19.632212+0000","last_became_active":"2024-03-06T07:26:17.873302+0000","last_became_peered":"2024-03-06T07:26:17.873302+0000","last_unstale":"2024-03-06T07:26:19.632212+0000","last_undegraded":"2024-03-06T07:26:19.632212+0000","last_fullsized":"2024-03-06T07:26:19.632212+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:36.314409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632569+0000","last_change":"2024-03-06T07:26:16.865144+0000","last_active":"2024-03-06T07:26:19.632569+0000","last_peered":"2024-03-06T07:26:19.632569+0000","last_clean":"2024-03-06T07:26:19.632569+0000","last_became_active":"2024-03-06T07:26:16.864840+0000","last_became_peered":"2024-03-06T07:26:16.864840+0000","last_unstale":"2024-03-06T07:26:19.632569+0000","last_undegraded":"2024-03-06T07:26:19.632569+0000","last_fullsized":"2024-03-06T07:26:19.632569+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:14.718388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856150+0000","last_change":"2024-03-06T07:26:17.867621+0000","last_active":"2024-03-06T07:26:18.856150+0000","last_peered":"2024-03-06T07:26:18.856150+0000","last_clean":"2024-03-06T07:26:18.856150+0000","last_became_active":"2024-03-06T07:26:17.867337+0000","last_became_peered":"2024-03-06T07:26:17.867337+0000","last_unstale":"2024-03-06T07:26:18.856150+0000","last_undegraded":"2024-03-06T07:26:18.856150+0000","last_fullsized":"2024-03-06T07:26:18.856150+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:20.717162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632000+0000","last_change":"2024-03-06T07:26:16.856146+0000","last_active":"2024-03-06T07:26:19.632000+0000","last_peered":"2024-03-06T07:26:19.632000+0000","last_clean":"2024-03-06T07:26:19.632000+0000","last_became_active":"2024-03-06T07:26:16.855842+0000","last_became_peered":"2024-03-06T07:26:16.855842+0000","last_unstale":"2024-03-06T07:26:19.632000+0000","last_undegraded":"2024-03-06T07:26:19.632000+0000","last_fullsized":"2024-03-06T07:26:19.632000+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:21.875952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632234+0000","last_change":"2024-03-06T07:26:17.873920+0000","last_active":"2024-03-06T07:26:19.632234+0000","last_peered":"2024-03-06T07:26:19.632234+0000","last_clean":"2024-03-06T07:26:19.632234+0000","last_became_active":"2024-03-06T07:26:17.873419+0000","last_became_peered":"2024-03-06T07:26:17.873419+0000","last_unstale":"2024-03-06T07:26:19.632234+0000","last_undegraded":"2024-03-06T07:26:19.632234+0000","last_fullsized":"2024-03-06T07:26:19.632234+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:22.063664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874737+0000","last_change":"2024-03-06T07:26:16.864486+0000","last_active":"2024-03-06T07:26:18.874737+0000","last_peered":"2024-03-06T07:26:18.874737+0000","last_clean":"2024-03-06T07:26:18.874737+0000","last_became_active":"2024-03-06T07:26:16.862774+0000","last_became_peered":"2024-03-06T07:26:16.862774+0000","last_unstale":"2024-03-06T07:26:18.874737+0000","last_undegraded":"2024-03-06T07:26:18.874737+0000","last_fullsized":"2024-03-06T07:26:18.874737+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:58.165611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876950+0000","last_change":"2024-03-06T07:26:17.873966+0000","last_active":"2024-03-06T07:26:18.876950+0000","last_peered":"2024-03-06T07:26:18.876950+0000","last_clean":"2024-03-06T07:26:18.876950+0000","last_became_active":"2024-03-06T07:26:17.873702+0000","last_became_peered":"2024-03-06T07:26:17.873702+0000","last_unstale":"2024-03-06T07:26:18.876950+0000","last_undegraded":"2024-03-06T07:26:18.876950+0000","last_fullsized":"2024-03-06T07:26:18.876950+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:47.962109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874691+0000","last_change":"2024-03-06T07:26:16.866599+0000","last_active":"2024-03-06T07:26:18.874691+0000","last_peered":"2024-03-06T07:26:18.874691+0000","last_clean":"2024-03-06T07:26:18.874691+0000","last_became_active":"2024-03-06T07:26:16.866342+0000","last_became_peered":"2024-03-06T07:26:16.866342+0000","last_unstale":"2024-03-06T07:26:18.874691+0000","last_undegraded":"2024-03-06T07:26:18.874691+0000","last_fullsized":"2024-03-06T07:26:18.874691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:30.398774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877178+0000","last_change":"2024-03-06T07:26:17.860602+0000","last_active":"2024-03-06T07:26:18.877178+0000","last_peered":"2024-03-06T07:26:18.877178+0000","last_clean":"2024-03-06T07:26:18.877178+0000","last_became_active":"2024-03-06T07:26:17.860333+0000","last_became_peered":"2024-03-06T07:26:17.860333+0000","last_unstale":"2024-03-06T07:26:18.877178+0000","last_undegraded":"2024-03-06T07:26:18.877178+0000","last_fullsized":"2024-03-06T07:26:18.877178+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:45.077065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856177+0000","last_change":"2024-03-06T07:26:16.868618+0000","last_active":"2024-03-06T07:26:18.856177+0000","last_peered":"2024-03-06T07:26:18.856177+0000","last_clean":"2024-03-06T07:26:18.856177+0000","last_became_active":"2024-03-06T07:26:16.868180+0000","last_became_peered":"2024-03-06T07:26:16.868180+0000","last_unstale":"2024-03-06T07:26:18.856177+0000","last_undegraded":"2024-03-06T07:26:18.856177+0000","last_fullsized":"2024-03-06T07:26:18.856177+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:26.488466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876708+0000","last_change":"2024-03-06T07:26:17.868580+0000","last_active":"2024-03-06T07:26:18.876708+0000","last_peered":"2024-03-06T07:26:18.876708+0000","last_clean":"2024-03-06T07:26:18.876708+0000","last_became_active":"2024-03-06T07:26:17.868280+0000","last_became_peered":"2024-03-06T07:26:17.868280+0000","last_unstale":"2024-03-06T07:26:18.876708+0000","last_undegraded":"2024-03-06T07:26:18.876708+0000","last_fullsized":"2024-03-06T07:26:18.876708+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:43.974866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855956+0000","last_change":"2024-03-06T07:26:16.868344+0000","last_active":"2024-03-06T07:26:18.855956+0000","last_peered":"2024-03-06T07:26:18.855956+0000","last_clean":"2024-03-06T07:26:18.855956+0000","last_became_active":"2024-03-06T07:26:16.868005+0000","last_became_peered":"2024-03-06T07:26:16.868005+0000","last_unstale":"2024-03-06T07:26:18.855956+0000","last_undegraded":"2024-03-06T07:26:18.855956+0000","last_fullsized":"2024-03-06T07:26:18.855956+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:57.303182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631553+0000","last_change":"2024-03-06T07:26:17.875016+0000","last_active":"2024-03-06T07:26:19.631553+0000","last_peered":"2024-03-06T07:26:19.631553+0000","last_clean":"2024-03-06T07:26:19.631553+0000","last_became_active":"2024-03-06T07:26:17.874746+0000","last_became_peered":"2024-03-06T07:26:17.874746+0000","last_unstale":"2024-03-06T07:26:19.631553+0000","last_undegraded":"2024-03-06T07:26:19.631553+0000","last_fullsized":"2024-03-06T07:26:19.631553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:22.274872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874532+0000","last_change":"2024-03-06T07:26:16.860891+0000","last_active":"2024-03-06T07:26:18.874532+0000","last_peered":"2024-03-06T07:26:18.874532+0000","last_clean":"2024-03-06T07:26:18.874532+0000","last_became_active":"2024-03-06T07:26:16.860523+0000","last_became_peered":"2024-03-06T07:26:16.860523+0000","last_unstale":"2024-03-06T07:26:18.874532+0000","last_undegraded":"2024-03-06T07:26:18.874532+0000","last_fullsized":"2024-03-06T07:26:18.874532+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:21.270064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875146+0000","last_change":"2024-03-06T07:26:17.868058+0000","last_active":"2024-03-06T07:26:18.875146+0000","last_peered":"2024-03-06T07:26:18.875146+0000","last_clean":"2024-03-06T07:26:18.875146+0000","last_became_active":"2024-03-06T07:26:17.867378+0000","last_became_peered":"2024-03-06T07:26:17.867378+0000","last_unstale":"2024-03-06T07:26:18.875146+0000","last_undegraded":"2024-03-06T07:26:18.875146+0000","last_fullsized":"2024-03-06T07:26:18.875146+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:40.439617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.856296+0000","last_change":"2024-03-06T07:26:16.863938+0000","last_active":"2024-03-06T07:26:18.856296+0000","last_peered":"2024-03-06T07:26:18.856296+0000","last_clean":"2024-03-06T07:26:18.856296+0000","last_became_active":"2024-03-06T07:26:16.863664+0000","last_became_peered":"2024-03-06T07:26:16.863664+0000","last_unstale":"2024-03-06T07:26:18.856296+0000","last_undegraded":"2024-03-06T07:26:18.856296+0000","last_fullsized":"2024-03-06T07:26:18.856296+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:18.526671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854824+0000","last_change":"2024-03-06T07:26:17.872137+0000","last_active":"2024-03-06T07:26:18.854824+0000","last_peered":"2024-03-06T07:26:18.854824+0000","last_clean":"2024-03-06T07:26:18.854824+0000","last_became_active":"2024-03-06T07:26:17.871867+0000","last_became_peered":"2024-03-06T07:26:17.871867+0000","last_unstale":"2024-03-06T07:26:18.854824+0000","last_undegraded":"2024-03-06T07:26:18.854824+0000","last_fullsized":"2024-03-06T07:26:18.854824+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:31.403582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"92'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.875144+0000","last_change":"2024-03-06T07:26:16.870823+0000","last_active":"2024-03-06T07:26:18.875144+0000","last_peered":"2024-03-06T07:26:18.875144+0000","last_clean":"2024-03-06T07:26:18.875144+0000","last_became_active":"2024-03-06T07:26:16.870507+0000","last_became_peered":"2024-03-06T07:26:16.870507+0000","last_unstale":"2024-03-06T07:26:18.875144+0000","last_undegraded":"2024-03-06T07:26:18.875144+0000","last_fullsized":"2024-03-06T07:26:18.875144+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:35.309600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.854500+0000","last_change":"2024-03-06T07:26:17.880876+0000","last_active":"2024-03-06T07:26:18.854500+0000","last_peered":"2024-03-06T07:26:18.854500+0000","last_clean":"2024-03-06T07:26:18.854500+0000","last_became_active":"2024-03-06T07:26:17.880610+0000","last_became_peered":"2024-03-06T07:26:17.880610+0000","last_unstale":"2024-03-06T07:26:18.854500+0000","last_undegraded":"2024-03-06T07:26:18.854500+0000","last_fullsized":"2024-03-06T07:26:18.854500+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:22.274872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874842+0000","last_change":"2024-03-06T07:26:16.864709+0000","last_active":"2024-03-06T07:26:18.874842+0000","last_peered":"2024-03-06T07:26:18.874842+0000","last_clean":"2024-03-06T07:26:18.874842+0000","last_became_active":"2024-03-06T07:26:16.863053+0000","last_became_peered":"2024-03-06T07:26:16.863053+0000","last_unstale":"2024-03-06T07:26:18.874842+0000","last_undegraded":"2024-03-06T07:26:18.874842+0000","last_fullsized":"2024-03-06T07:26:18.874842+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:26.399377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631995+0000","last_change":"2024-03-06T07:26:17.872652+0000","last_active":"2024-03-06T07:26:19.631995+0000","last_peered":"2024-03-06T07:26:19.631995+0000","last_clean":"2024-03-06T07:26:19.631995+0000","last_became_active":"2024-03-06T07:26:17.872005+0000","last_became_peered":"2024-03-06T07:26:17.872005+0000","last_unstale":"2024-03-06T07:26:19.631995+0000","last_undegraded":"2024-03-06T07:26:19.631995+0000","last_fullsized":"2024-03-06T07:26:19.631995+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:14.616124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632055+0000","last_change":"2024-03-06T07:26:16.861475+0000","last_active":"2024-03-06T07:26:19.632055+0000","last_peered":"2024-03-06T07:26:19.632055+0000","last_clean":"2024-03-06T07:26:19.632055+0000","last_became_active":"2024-03-06T07:26:16.861209+0000","last_became_peered":"2024-03-06T07:26:16.861209+0000","last_unstale":"2024-03-06T07:26:19.632055+0000","last_undegraded":"2024-03-06T07:26:19.632055+0000","last_fullsized":"2024-03-06T07:26:19.632055+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:57.303182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877061+0000","last_change":"2024-03-06T07:26:17.876580+0000","last_active":"2024-03-06T07:26:18.877061+0000","last_peered":"2024-03-06T07:26:18.877061+0000","last_clean":"2024-03-06T07:26:18.877061+0000","last_became_active":"2024-03-06T07:26:17.876316+0000","last_became_peered":"2024-03-06T07:26:17.876316+0000","last_unstale":"2024-03-06T07:26:18.877061+0000","last_undegraded":"2024-03-06T07:26:18.877061+0000","last_fullsized":"2024-03-06T07:26:18.877061+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:53.155139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.874794+0000","last_change":"2024-03-06T07:26:16.870873+0000","last_active":"2024-03-06T07:26:18.874794+0000","last_peered":"2024-03-06T07:26:18.874794+0000","last_clean":"2024-03-06T07:26:18.874794+0000","last_became_active":"2024-03-06T07:26:16.870581+0000","last_became_peered":"2024-03-06T07:26:16.870581+0000","last_unstale":"2024-03-06T07:26:18.874794+0000","last_undegraded":"2024-03-06T07:26:18.874794+0000","last_fullsized":"2024-03-06T07:26:18.874794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:40.436083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.632601+0000","last_change":"2024-03-06T07:26:17.880256+0000","last_active":"2024-03-06T07:26:19.632601+0000","last_peered":"2024-03-06T07:26:19.632601+0000","last_clean":"2024-03-06T07:26:19.632601+0000","last_became_active":"2024-03-06T07:26:17.879582+0000","last_became_peered":"2024-03-06T07:26:17.879582+0000","last_unstale":"2024-03-06T07:26:19.632601+0000","last_undegraded":"2024-03-06T07:26:19.632601+0000","last_fullsized":"2024-03-06T07:26:19.632601+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:07.715401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631668+0000","last_change":"2024-03-06T07:26:16.864936+0000","last_active":"2024-03-06T07:26:19.631668+0000","last_peered":"2024-03-06T07:26:19.631668+0000","last_clean":"2024-03-06T07:26:19.631668+0000","last_became_active":"2024-03-06T07:26:16.864637+0000","last_became_peered":"2024-03-06T07:26:16.864637+0000","last_unstale":"2024-03-06T07:26:19.631668+0000","last_undegraded":"2024-03-06T07:26:19.631668+0000","last_fullsized":"2024-03-06T07:26:19.631668+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:29.981611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876708+0000","last_change":"2024-03-06T07:26:17.875415+0000","last_active":"2024-03-06T07:26:18.876708+0000","last_peered":"2024-03-06T07:26:18.876708+0000","last_clean":"2024-03-06T07:26:18.876708+0000","last_became_active":"2024-03-06T07:26:17.875138+0000","last_became_peered":"2024-03-06T07:26:17.875138+0000","last_unstale":"2024-03-06T07:26:18.876708+0000","last_undegraded":"2024-03-06T07:26:18.876708+0000","last_fullsized":"2024-03-06T07:26:18.876708+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:43:27.427674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.855626+0000","last_change":"2024-03-06T07:26:16.871839+0000","last_active":"2024-03-06T07:26:18.855626+0000","last_peered":"2024-03-06T07:26:18.855626+0000","last_clean":"2024-03-06T07:26:18.855626+0000","last_became_active":"2024-03-06T07:26:16.871570+0000","last_became_peered":"2024-03-06T07:26:16.871570+0000","last_unstale":"2024-03-06T07:26:18.855626+0000","last_undegraded":"2024-03-06T07:26:18.855626+0000","last_fullsized":"2024-03-06T07:26:18.855626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:06.710593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631562+0000","last_change":"2024-03-06T07:26:17.873738+0000","last_active":"2024-03-06T07:26:19.631562+0000","last_peered":"2024-03-06T07:26:19.631562+0000","last_clean":"2024-03-06T07:26:19.631562+0000","last_became_active":"2024-03-06T07:26:17.873467+0000","last_became_peered":"2024-03-06T07:26:17.873467+0000","last_unstale":"2024-03-06T07:26:19.631562+0000","last_undegraded":"2024-03-06T07:26:19.631562+0000","last_fullsized":"2024-03-06T07:26:19.631562+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:25.582364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.876761+0000","last_change":"2024-03-06T07:26:16.866300+0000","last_active":"2024-03-06T07:26:18.876761+0000","last_peered":"2024-03-06T07:26:18.876761+0000","last_clean":"2024-03-06T07:26:18.876761+0000","last_became_active":"2024-03-06T07:26:16.866014+0000","last_became_peered":"2024-03-06T07:26:16.866014+0000","last_unstale":"2024-03-06T07:26:18.876761+0000","last_undegraded":"2024-03-06T07:26:18.876761+0000","last_fullsized":"2024-03-06T07:26:18.876761+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:31.226660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:18.877031+0000","last_change":"2024-03-06T07:26:17.874843+0000","last_active":"2024-03-06T07:26:18.877031+0000","last_peered":"2024-03-06T07:26:18.877031+0000","last_clean":"2024-03-06T07:26:18.877031+0000","last_became_active":"2024-03-06T07:26:17.874530+0000","last_became_peered":"2024-03-06T07:26:17.874530+0000","last_unstale":"2024-03-06T07:26:18.877031+0000","last_undegraded":"2024-03-06T07:26:18.877031+0000","last_fullsized":"2024-03-06T07:26:18.877031+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:16.843320+0000","last_clean_scrub_stamp":"2024-03-06T07:26:16.843320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:22.620099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631513+0000","last_change":"2024-03-06T07:26:16.870244+0000","last_active":"2024-03-06T07:26:19.631513+0000","last_peered":"2024-03-06T07:26:19.631513+0000","last_clean":"2024-03-06T07:26:19.631513+0000","last_became_active":"2024-03-06T07:26:16.869996+0000","last_became_peered":"2024-03-06T07:26:16.869996+0000","last_unstale":"2024-03-06T07:26:19.631513+0000","last_undegraded":"2024-03-06T07:26:19.631513+0000","last_fullsized":"2024-03-06T07:26:19.631513+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:15.838511+0000","last_clean_scrub_stamp":"2024-03-06T07:26:15.838511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:39:46.843920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":301,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-06T07:26:19.631596+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:26:19.631596+0000","last_peered":"2024-03-06T07:26:19.631596+0000","last_clean":"2024-03-06T07:26:19.631596+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:26:19.631596+0000","last_undegraded":"2024-03-06T07:26:19.631596+0000","last_fullsized":"2024-03-06T07:26:19.631596+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574907,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32880,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338960,"statfs":{"total":96636764160,"available":96603095040,"internally_reserved":0,"allocated":1740800,"data_stored":1076018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1146880,"data_stored":486190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574907,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1142784,"data_stored":485608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574907,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1159168,"data_stored":487274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1155072,"data_stored":487172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1142784,"data_stored":485572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1138688,"data_stored":485538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1736704,"data_stored":1076488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:25.677 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:26:25.677 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.820+0000 7f7780d32640 1 Processor -- start 2024-03-06T07:26:25.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.820+0000 7f7780d32640 1 -- start start 2024-03-06T07:26:25.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.821+0000 7f7780d32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f777c0d4860 0x7f777c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.821+0000 7f7780d32640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c1618b0 con 0x7f777c0b0e30 2024-03-06T07:26:25.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.821+0000 7f7780d32640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c0cf400 con 0x7f777c1614d0 2024-03-06T07:26:25.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.821+0000 7f7780d32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c0d4cc0 con 0x7f777c0d4860 2024-03-06T07:26:25.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.821+0000 7f777ad9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f777c0b0e30 0x7f777c0b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49918/0 (socket says 172.21.15.59:49918) 2024-03-06T07:26:25.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.821+0000 7f777ad9d640 1 -- 172.21.15.59:0/158104674 learned_addr learned my addr 172.21.15.59:0/158104674 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:25.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.821+0000 7f7779d9b640 1 --2- 172.21.15.59:0/158104674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f777c0d4860 0x7f777c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.822+0000 7f777959a640 1 -- 172.21.15.59:0/158104674 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392511720 0 0) 0x7f777c1618b0 con 0x7f777c0b0e30 2024-03-06T07:26:25.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.822+0000 7f777959a640 1 -- 172.21.15.59:0/158104674 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7768003520 con 0x7f777c0b0e30 2024-03-06T07:26:25.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.822+0000 7f7779d9b640 1 -- 172.21.15.59:0/158104674 >> v1:172.21.15.164:6789/0 conn(0x7f777c1614d0 legacy=0x7f777c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.822+0000 7f777959a640 1 -- 172.21.15.59:0/158104674 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138648879 0 0) 0x7f777c0cf400 con 0x7f777c1614d0 2024-03-06T07:26:25.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.822+0000 7f7779d9b640 1 -- 172.21.15.59:0/158104674 >> v1:172.21.15.59:6789/0 conn(0x7f777c0b0e30 legacy=0x7f777c0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.822+0000 7f7779d9b640 1 -- 172.21.15.59:0/158104674 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c0cf6a0 con 0x7f777c0d4860 2024-03-06T07:26:25.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.823+0000 7f7779d9b640 1 --2- 172.21.15.59:0/158104674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f777c0d4860 0x7f777c0d2020 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f77640097d0 tx=0x7f776402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da4e53b7af7ecb9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.823+0000 7f777959a640 1 -- 172.21.15.59:0/158104674 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764035020 con 0x7f777c0d4860 2024-03-06T07:26:25.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.823+0000 7f777959a640 1 -- 172.21.15.59:0/158104674 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7764031890 con 0x7f777c0d4860 2024-03-06T07:26:25.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.823+0000 7f777959a640 1 -- 172.21.15.59:0/158104674 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764031c60 con 0x7f777c0d4860 2024-03-06T07:26:25.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.823+0000 7f7780d32640 1 -- 172.21.15.59:0/158104674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f777c0d4860 msgr2=0x7f777c0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.823+0000 7f7780d32640 1 --2- 172.21.15.59:0/158104674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f777c0d4860 0x7f777c0d2020 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f77640097d0 tx=0x7f776402b930 comp rx=0 tx=0).stop 2024-03-06T07:26:25.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.823+0000 7f7780d32640 1 -- 172.21.15.59:0/158104674 shutdown_connections 2024-03-06T07:26:25.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.823+0000 7f7780d32640 1 --2- 172.21.15.59:0/158104674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f777c0d4860 0x7f777c0d2020 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.823+0000 7f7780d32640 1 -- 172.21.15.59:0/158104674 >> 172.21.15.59:0/158104674 conn(0x7f777c0d3d40 msgr2=0x7f777c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.823+0000 7f7780d32640 1 -- 172.21.15.59:0/158104674 shutdown_connections 2024-03-06T07:26:25.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.823+0000 7f7780d32640 1 -- 172.21.15.59:0/158104674 wait complete. 2024-03-06T07:26:25.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.825+0000 7f7780d32640 1 Processor -- start 2024-03-06T07:26:25.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.825+0000 7f7780d32640 1 -- start start 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.825+0000 7f7780d32640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f777c0b0e30 0x7f777c0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.825+0000 7f7780d32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f777c0d4860 0x7f777c0b95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.825+0000 7f7780d32640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f777c1614d0 0x7f777c0ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.825+0000 7f7780d32640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c0b0910 con 0x7f777c0b0e30 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.825+0000 7f7780d32640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c0cfc60 con 0x7f777c1614d0 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.825+0000 7f7780d32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777c0d2560 con 0x7f777c0d4860 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.825+0000 7f777a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f777c0b0e30 0x7f777c0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.825+0000 7f777a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f777c0b0e30 0x7f777c0c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60532/0 (socket says 172.21.15.59:60532) 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.825+0000 7f777a59c640 1 -- 172.21.15.59:0/3423145805 learned_addr learned my addr 172.21.15.59:0/3423145805 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f7779d9b640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f777c0d4860 0x7f777c0b95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f777ad9d640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f777c1614d0 0x7f777c0ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f777a59c640 1 -- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f777c0d4860 msgr2=0x7f777c0b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f777a59c640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f777c0d4860 0x7f777c0b95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f777a59c640 1 -- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f777c1614d0 msgr2=0x7f777c0ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f777a59c640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f777c1614d0 0x7f777c0ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f777a59c640 1 -- 172.21.15.59:0/3423145805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c0c9230 con 0x7f777c0b0e30 2024-03-06T07:26:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f777a59c640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f777c0b0e30 0x7f777c0c8c10 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f777400c410 tx=0x7f777400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f7772ffd640 1 -- 172.21.15.59:0/3423145805 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774017070 con 0x7f777c0b0e30 2024-03-06T07:26:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f7772ffd640 1 -- 172.21.15.59:0/3423145805 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77740035f0 con 0x7f777c0b0e30 2024-03-06T07:26:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f7780d32640 1 -- 172.21.15.59:0/3423145805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777c0c9510 con 0x7f777c0b0e30 2024-03-06T07:26:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f7772ffd640 1 -- 172.21.15.59:0/3423145805 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77740039e0 con 0x7f777c0b0e30 2024-03-06T07:26:25.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.826+0000 7f7780d32640 1 -- 172.21.15.59:0/3423145805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777c0c9ad0 con 0x7f777c0b0e30 2024-03-06T07:26:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.827+0000 7f7780d32640 1 -- 172.21.15.59:0/3423145805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777c06b9d0 con 0x7f777c0b0e30 2024-03-06T07:26:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.827+0000 7f7772ffd640 1 -- 172.21.15.59:0/3423145805 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f777401a070 con 0x7f777c0b0e30 2024-03-06T07:26:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.828+0000 7f7772ffd640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f775c06b4d0 0x7f775c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.828+0000 7f7772ffd640 1 -- 172.21.15.59:0/3423145805 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7774089650 con 0x7f777c0b0e30 2024-03-06T07:26:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.828+0000 7f7779d9b640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f775c06b4d0 0x7f775c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.829+0000 7f7779d9b640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f775c06b4d0 0x7f775c06d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7764036000 tx=0x7f7764008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:25.832+0000 7f7772ffd640 1 -- 172.21.15.59:0/3423145805 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f777c06b9d0 con 0x7f777c0b0e30 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.086+0000 7f7780d32640 1 -- 172.21.15.59:0/3423145805 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f777c15e7f0 con 0x7f777c0b0e30 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.088+0000 7f7772ffd640 1 -- 172.21.15.59:0/3423145805 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f7774057df0 con 0x7f777c0b0e30 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA== 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.089+0000 7f7780d32640 1 -- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f775c06b4d0 msgr2=0x7f775c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.089+0000 7f7780d32640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f775c06b4d0 0x7f775c06d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7764036000 tx=0x7f7764008030 comp rx=0 tx=0).stop 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.090+0000 7f7780d32640 1 -- 172.21.15.59:0/3423145805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f777c0b0e30 msgr2=0x7f777c0c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.090+0000 7f7780d32640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f777c0b0e30 0x7f777c0c8c10 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f777400c410 tx=0x7f777400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.090+0000 7f7780d32640 1 -- 172.21.15.59:0/3423145805 shutdown_connections 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.090+0000 7f7780d32640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f775c06b4d0 0x7f775c06d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.090+0000 7f7780d32640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f777c1614d0 0x7f777c0ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.090+0000 7f7780d32640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f777c0d4860 0x7f777c0b95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.090+0000 7f7780d32640 1 --2- 172.21.15.59:0/3423145805 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f777c0b0e30 0x7f777c0c8c10 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.090+0000 7f7780d32640 1 -- 172.21.15.59:0/3423145805 >> 172.21.15.59:0/3423145805 conn(0x7f777c0d3d40 msgr2=0x7f777c15fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.090+0000 7f7780d32640 1 -- 172.21.15.59:0/3423145805 shutdown_connections 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.090+0000 7f7780d32640 1 -- 172.21.15.59:0/3423145805 wait complete. 2024-03-06T07:26:26.102 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:26.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.202+0000 7f9bf4c4e640 1 Processor -- start 2024-03-06T07:26:26.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.202+0000 7f9bf4c4e640 1 -- start start 2024-03-06T07:26:26.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bf4c4e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf006afb0 0x7f9bf006b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bf4c4e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9bf015c490 0x7f9bf015c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bf4c4e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf00d0a70 con 0x7f9bf00b4420 2024-03-06T07:26:26.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bf4c4e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf00d0850 con 0x7f9bf015c490 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bf4c4e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf006b8d0 con 0x7f9bf006afb0 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bedd9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9bf00b4420 0x7f9bf00cdb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49934/0 (socket says 172.21.15.59:49934) 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bedd9b640 1 -- 172.21.15.59:0/548781880 learned_addr learned my addr 172.21.15.59:0/548781880 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bee59c640 1 --2- 172.21.15.59:0/548781880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf006afb0 0x7f9bf006b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9beed9d640 1 --2- 172.21.15.59:0/548781880 >> v2:172.21.15.164:3300/0 conn(0x7f9bf015c490 0x7f9bf015c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bed59a640 1 -- 172.21.15.59:0/548781880 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124544764 0 0) 0x7f9bf00d0a70 con 0x7f9bf00b4420 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bed59a640 1 -- 172.21.15.59:0/548781880 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8003520 con 0x7f9bf00b4420 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bee59c640 1 -- 172.21.15.59:0/548781880 >> v2:172.21.15.164:3300/0 conn(0x7f9bf015c490 msgr2=0x7f9bf015c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bee59c640 1 --2- 172.21.15.59:0/548781880 >> v2:172.21.15.164:3300/0 conn(0x7f9bf015c490 0x7f9bf015c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bee59c640 1 -- 172.21.15.59:0/548781880 >> v1:172.21.15.59:6789/0 conn(0x7f9bf00b4420 legacy=0x7f9bf00cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9bee59c640 1 -- 172.21.15.59:0/548781880 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00d0d10 con 0x7f9bf006afb0 2024-03-06T07:26:26.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.203+0000 7f9beed9d640 1 --2- 172.21.15.59:0/548781880 >> v2:172.21.15.164:3300/0 conn(0x7f9bf015c490 0x7f9bf015c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:26.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.204+0000 7f9bee59c640 1 --2- 172.21.15.59:0/548781880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf006afb0 0x7f9bf006b390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00d0c40 tx=0x7f9be002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e7da95d9dff76c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.204+0000 7f9bed59a640 1 -- 172.21.15.59:0/548781880 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be0031770 con 0x7f9bf006afb0 2024-03-06T07:26:26.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.204+0000 7f9bed59a640 1 -- 172.21.15.59:0/548781880 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be0037070 con 0x7f9bf006afb0 2024-03-06T07:26:26.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.204+0000 7f9bed59a640 1 -- 172.21.15.59:0/548781880 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be0031b30 con 0x7f9bf006afb0 2024-03-06T07:26:26.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.205+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/548781880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf006afb0 msgr2=0x7f9bf006b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.205+0000 7f9bf4c4e640 1 --2- 172.21.15.59:0/548781880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf006afb0 0x7f9bf006b390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9bf00d0c40 tx=0x7f9be002b810 comp rx=0 tx=0).stop 2024-03-06T07:26:26.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.205+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/548781880 shutdown_connections 2024-03-06T07:26:26.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.205+0000 7f9bf4c4e640 1 --2- 172.21.15.59:0/548781880 >> v2:172.21.15.164:3300/0 conn(0x7f9bf015c490 0x7f9bf015c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.205+0000 7f9bf4c4e640 1 --2- 172.21.15.59:0/548781880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf006afb0 0x7f9bf006b390 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.205+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/548781880 >> 172.21.15.59:0/548781880 conn(0x7f9bf006c0c0 msgr2=0x7f9bf006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.205+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/548781880 shutdown_connections 2024-03-06T07:26:26.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.205+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/548781880 wait complete. 2024-03-06T07:26:26.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.206+0000 7f9bf4c4e640 1 Processor -- start 2024-03-06T07:26:26.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.206+0000 7f9bf4c4e640 1 -- start start 2024-03-06T07:26:26.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bf4c4e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf006afb0 0x7f9bf015db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bf4c4e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf00b4420 0x7f9bf00ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bf4c4e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf015c490 0x7f9bf013af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bf4c4e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf0145840 con 0x7f9bf006afb0 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bf4c4e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf00d12d0 con 0x7f9bf00b4420 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bf4c4e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf00ceb90 con 0x7f9bf015c490 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bee59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf006afb0 0x7f9bf015db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bee59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf006afb0 0x7f9bf015db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60542/0 (socket says 172.21.15.59:60542) 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bee59c640 1 -- 172.21.15.59:0/1018271154 learned_addr learned my addr 172.21.15.59:0/1018271154 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9beed9d640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf015c490 0x7f9bf013af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bedd9b640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf00b4420 0x7f9bf00ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bee59c640 1 -- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf015c490 msgr2=0x7f9bf013af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bee59c640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf015c490 0x7f9bf013af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bee59c640 1 -- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf00b4420 msgr2=0x7f9bf00ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bee59c640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf00b4420 0x7f9bf00ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.207+0000 7f9bee59c640 1 -- 172.21.15.59:0/1018271154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00ae2e0 con 0x7f9bf006afb0 2024-03-06T07:26:26.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.208+0000 7f9bee59c640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf006afb0 0x7f9bf015db80 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f9be0000c00 tx=0x7f9be00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.208+0000 7f9bdeffd640 1 -- 172.21.15.59:0/1018271154 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be0032600 con 0x7f9bf006afb0 2024-03-06T07:26:26.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.208+0000 7f9bdeffd640 1 -- 172.21.15.59:0/1018271154 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be00327c0 con 0x7f9bf006afb0 2024-03-06T07:26:26.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.208+0000 7f9bdeffd640 1 -- 172.21.15.59:0/1018271154 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be0032bb0 con 0x7f9bf006afb0 2024-03-06T07:26:26.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.208+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/1018271154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf00ae5c0 con 0x7f9bf006afb0 2024-03-06T07:26:26.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.208+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/1018271154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf00aee60 con 0x7f9bf006afb0 2024-03-06T07:26:26.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.209+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/1018271154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf004f330 con 0x7f9bf006afb0 2024-03-06T07:26:26.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.209+0000 7f9bdeffd640 1 -- 172.21.15.59:0/1018271154 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9be0031070 con 0x7f9bf006afb0 2024-03-06T07:26:26.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.209+0000 7f9bdeffd640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9bd006b4d0 0x7f9bd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.210+0000 7f9bdeffd640 1 -- 172.21.15.59:0/1018271154 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9be00ab910 con 0x7f9bf006afb0 2024-03-06T07:26:26.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.210+0000 7f9bedd9b640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9bd006b4d0 0x7f9bd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.210+0000 7f9bedd9b640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9bd006b4d0 0x7f9bd006d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9bf006c770 tx=0x7f9be4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.214+0000 7f9bdeffd640 1 -- 172.21.15.59:0/1018271154 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9be0040070 con 0x7f9bf006afb0 2024-03-06T07:26:26.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.433+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/1018271154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bf00b5cb0 con 0x7f9bf006afb0 2024-03-06T07:26:26.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.434+0000 7f9bdeffd640 1 -- 172.21.15.59:0/1018271154 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f9be007a140 con 0x7f9bf006afb0 2024-03-06T07:26:26.436 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:26.436 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":49,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6173,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/386662975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":386662975},{"type":"v1","addr":"172.21.15.164:6836","nonce":386662975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":7407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/2799249020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2799249020},{"type":"v1","addr":"172.21.15.59:6834","nonce":2799249020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":7410,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1784120157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1784120157},{"type":"v1","addr":"172.21.15.59:6835","nonce":1784120157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:19.214384+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7422":{"gid":7422,"name":"d","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6837/4292771408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4292771408},{"type":"v1","addr":"172.21.15.164:6837","nonce":4292771408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7422,"qdb_cluster":[7422]},"id":7}]} 2024-03-06T07:26:26.436 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 49 2024-03-06T07:26:26.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9bd006b4d0 msgr2=0x7f9bd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9bd006b4d0 0x7f9bd006d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9bf006c770 tx=0x7f9be4009040 comp rx=0 tx=0).stop 2024-03-06T07:26:26.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/1018271154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf006afb0 msgr2=0x7f9bf015db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf006afb0 0x7f9bf015db80 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f9be0000c00 tx=0x7f9be00046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/1018271154 shutdown_connections 2024-03-06T07:26:26.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9bd006b4d0 0x7f9bd006d990 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf015c490 0x7f9bf013af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf00b4420 0x7f9bf00ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 --2- 172.21.15.59:0/1018271154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bf006afb0 0x7f9bf015db80 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/1018271154 >> 172.21.15.59:0/1018271154 conn(0x7f9bf006c0c0 msgr2=0x7f9bf015af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/1018271154 shutdown_connections 2024-03-06T07:26:26.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.436+0000 7f9bf4c4e640 1 -- 172.21.15.59:0/1018271154 wait complete. 2024-03-06T07:26:26.851 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:17.219713+0000', 'modified': '2024-03-06T07:26:19.214384+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7422}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7422': {'gid': 7422, 'name': 'd', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.164:6837/4292771408', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 4292771408}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 4292771408}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7422, 'qdb_cluster': [7422]} 2024-03-06T07:26:26.851 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:26.851 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-06T07:26:26.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.954+0000 7fc66fe39640 1 Processor -- start 2024-03-06T07:26:26.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.954+0000 7fc66fe39640 1 -- start start 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.955+0000 7fc66fe39640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc668158880 0x7fc66806c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.955+0000 7fc66fe39640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc66815b380 0x7fc6680708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.955+0000 7fc66fe39640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66806d030 con 0x7fc66806cb90 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.955+0000 7fc66fe39640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66806d310 con 0x7fc668158880 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.955+0000 7fc66fe39640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66806d5f0 con 0x7fc66815b380 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.955+0000 7fc66dbd5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc668158880 0x7fc66806c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.955+0000 7fc66d3d4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc66815b380 0x7fc6680708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.955+0000 7fc66e3d6640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc66806cb90 0x7fc66815e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:54552/0 (socket says 172.21.15.59:54552) 2024-03-06T07:26:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.955+0000 7fc66e3d6640 1 -- 172.21.15.59:0/139742850 learned_addr learned my addr 172.21.15.59:0/139742850 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.956+0000 7fc66cbd3640 1 -- 172.21.15.59:0/139742850 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409776301 0 0) 0x7fc66806d030 con 0x7fc66806cb90 2024-03-06T07:26:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.956+0000 7fc66d3d4640 1 -- 172.21.15.59:0/139742850 >> v2:172.21.15.164:3300/0 conn(0x7fc668158880 msgr2=0x7fc66806c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.956+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/139742850 >> v2:172.21.15.164:3300/0 conn(0x7fc668158880 0x7fc66806c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.956+0000 7fc66d3d4640 1 -- 172.21.15.59:0/139742850 >> v1:172.21.15.164:6789/0 conn(0x7fc66806cb90 legacy=0x7fc66815e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.956+0000 7fc66d3d4640 1 -- 172.21.15.59:0/139742850 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66806d8d0 con 0x7fc66815b380 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.956+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/139742850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc66815b380 0x7fc6680708d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0097a0 tx=0x7fc65c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7586f01b85e13b3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.957+0000 7fc66cbd3640 1 -- 172.21.15.59:0/139742850 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c035020 con 0x7fc66815b380 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.957+0000 7fc66cbd3640 1 -- 172.21.15.59:0/139742850 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c031830 con 0x7fc66815b380 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.957+0000 7fc66cbd3640 1 -- 172.21.15.59:0/139742850 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c031c00 con 0x7fc66815b380 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.957+0000 7fc66fe39640 1 -- 172.21.15.59:0/139742850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc66815b380 msgr2=0x7fc6680708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.957+0000 7fc66fe39640 1 --2- 172.21.15.59:0/139742850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc66815b380 0x7fc6680708d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0097a0 tx=0x7fc65c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.957+0000 7fc66fe39640 1 -- 172.21.15.59:0/139742850 shutdown_connections 2024-03-06T07:26:26.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.957+0000 7fc66fe39640 1 --2- 172.21.15.59:0/139742850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc66815b380 0x7fc6680708d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.957+0000 7fc66fe39640 1 --2- 172.21.15.59:0/139742850 >> v2:172.21.15.164:3300/0 conn(0x7fc668158880 0x7fc66806c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.957+0000 7fc66fe39640 1 -- 172.21.15.59:0/139742850 >> 172.21.15.59:0/139742850 conn(0x7fc6680b1c80 msgr2=0x7fc6680cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.957+0000 7fc66fe39640 1 -- 172.21.15.59:0/139742850 shutdown_connections 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.957+0000 7fc66fe39640 1 -- 172.21.15.59:0/139742850 wait complete. 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.958+0000 7fc66fe39640 1 Processor -- start 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.958+0000 7fc66fe39640 1 -- start start 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.959+0000 7fc66fe39640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc66806cb90 0x7fc668138750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.959+0000 7fc66fe39640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc668158880 0x7fc66814fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.959+0000 7fc66fe39640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc66815b380 0x7fc6681398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.959+0000 7fc66fe39640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6680ce950 con 0x7fc66806cb90 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.959+0000 7fc66fe39640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6680ce7d0 con 0x7fc66815b380 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.959+0000 7fc66fe39640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc668070e10 con 0x7fc668158880 2024-03-06T07:26:26.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.959+0000 7fc66dbd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc66806cb90 0x7fc668138750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.959+0000 7fc66dbd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc66806cb90 0x7fc668138750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60548/0 (socket says 172.21.15.59:60548) 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.959+0000 7fc66dbd5640 1 -- 172.21.15.59:0/3623860521 learned_addr learned my addr 172.21.15.59:0/3623860521 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.959+0000 7fc66e3d6640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc66815b380 0x7fc6681398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.959+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc668158880 0x7fc66814fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.960+0000 7fc66dbd5640 1 -- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc668158880 msgr2=0x7fc66814fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.960+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc668158880 0x7fc66814fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.960+0000 7fc66dbd5640 1 -- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc66815b380 msgr2=0x7fc6681398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.960+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc66815b380 0x7fc6681398d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.960+0000 7fc66dbd5640 1 -- 172.21.15.59:0/3623860521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc668138c90 con 0x7fc66806cb90 2024-03-06T07:26:26.965 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.960+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc66806cb90 0x7fc668138750 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fc65800efc0 tx=0x7fc65800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.960+0000 7fc6667fc640 1 -- 172.21.15.59:0/3623860521 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658018020 con 0x7fc66806cb90 2024-03-06T07:26:26.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.960+0000 7fc6667fc640 1 -- 172.21.15.59:0/3623860521 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc658014480 con 0x7fc66806cb90 2024-03-06T07:26:26.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.960+0000 7fc66fe39640 1 -- 172.21.15.59:0/3623860521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66814ee80 con 0x7fc66806cb90 2024-03-06T07:26:26.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.960+0000 7fc6667fc640 1 -- 172.21.15.59:0/3623860521 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658014850 con 0x7fc66806cb90 2024-03-06T07:26:26.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.960+0000 7fc66fe39640 1 -- 172.21.15.59:0/3623860521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66814f440 con 0x7fc66806cb90 2024-03-06T07:26:26.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.961+0000 7fc6667fc640 1 -- 172.21.15.59:0/3623860521 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc658005d10 con 0x7fc66806cb90 2024-03-06T07:26:26.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.962+0000 7fc6667fc640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc65406b4d0 0x7fc65406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.962+0000 7fc6667fc640 1 -- 172.21.15.59:0/3623860521 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc65808ab20 con 0x7fc66806cb90 2024-03-06T07:26:26.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.962+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc65406b4d0 0x7fc65406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.962+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc65406b4d0 0x7fc65406d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc65c036000 tx=0x7fc65c03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.963+0000 7fc672a29740 1 Processor -- start 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.963+0000 7fc672a29740 1 -- start start 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc672a29740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f7ccc0 0x556004f7d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc672a29740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9d090 0x556004f9d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc672a29740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9c7b0 0x556004f4f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc672a29740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556004eb96b0 con 0x556004f7ccc0 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc672a29740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x556004e62a60 con 0x556004f9d090 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc672a29740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x556004eb5750 con 0x556004f9c7b0 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc66dbd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f7ccc0 0x556004f7d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc66dbd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f7ccc0 0x556004f7d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60560/0 (socket says 172.21.15.59:60560) 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc66dbd5640 1 -- 172.21.15.59:0/1819018248 learned_addr learned my addr 172.21.15.59:0/1819018248 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:26.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/1819018248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9d090 0x556004f9d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc66e3d6640 1 --2- 172.21.15.59:0/1819018248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9c7b0 0x556004f4f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc66dbd5640 1 -- 172.21.15.59:0/1819018248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9c7b0 msgr2=0x556004f4f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/1819018248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9c7b0 0x556004f4f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc66dbd5640 1 -- 172.21.15.59:0/1819018248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9d090 msgr2=0x556004f9d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/1819018248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9d090 0x556004f9d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc66dbd5640 1 -- 172.21.15.59:0/1819018248 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556004e80f30 con 0x556004f7ccc0 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/1819018248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f7ccc0 0x556004f7d0a0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc658061510 tx=0x7fc658059560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7333a7733ba0404d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc64f7fe640 1 -- 172.21.15.59:0/1819018248 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658087070 con 0x556004f7ccc0 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc64f7fe640 1 -- 172.21.15.59:0/1819018248 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65805d160 con 0x556004f7ccc0 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.964+0000 7fc64f7fe640 1 -- 172.21.15.59:0/1819018248 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6580615b0 con 0x556004f7ccc0 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.965+0000 7fc672a29740 1 -- 172.21.15.59:0/1819018248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f7ccc0 msgr2=0x556004f7d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.965+0000 7fc672a29740 1 --2- 172.21.15.59:0/1819018248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f7ccc0 0x556004f7d0a0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fc658061510 tx=0x7fc658059560 comp rx=0 tx=0).stop 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.965+0000 7fc672a29740 1 -- 172.21.15.59:0/1819018248 shutdown_connections 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.965+0000 7fc672a29740 1 --2- 172.21.15.59:0/1819018248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9c7b0 0x556004f4f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.965+0000 7fc672a29740 1 --2- 172.21.15.59:0/1819018248 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9d090 0x556004f9d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.965+0000 7fc672a29740 1 --2- 172.21.15.59:0/1819018248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f7ccc0 0x556004f7d0a0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.965+0000 7fc672a29740 1 -- 172.21.15.59:0/1819018248 >> 172.21.15.59:0/1819018248 conn(0x556004f518f0 msgr2=0x556004f531b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.965+0000 7fc672a29740 1 -- 172.21.15.59:0/1819018248 shutdown_connections 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.965+0000 7fc672a29740 1 -- 172.21.15.59:0/1819018248 wait complete. 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.966+0000 7fc672a29740 1 -- start start 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.966+0000 7fc672a29740 1 Processor -- start 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc672a29740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9c7b0 0x556004f78c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc672a29740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9d090 0x556004fa5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc672a29740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f807a0 0x556004faf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc672a29740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556004f80010 con 0x556004f9d090 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc672a29740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x556004e75ea0 con 0x556004f9c7b0 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc672a29740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x556004f9ba30 con 0x556004f807a0 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc672a29740 10 client.0 _ll_register_callbacks cb 0x556004e75870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:26.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc66d3d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9d090 0x556004fa5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc66d3d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9d090 0x556004fa5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60564/0 (socket says 172.21.15.59:60564) 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc66d3d4640 1 -- 172.21.15.59:0/1150655384 learned_addr learned my addr 172.21.15.59:0/1150655384 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9c7b0 0x556004f78c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc66e3d6640 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f807a0 0x556004faf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc66d3d4640 1 -- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f807a0 msgr2=0x556004faf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f807a0 0x556004faf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc66d3d4640 1 -- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9c7b0 msgr2=0x556004f78c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9c7b0 0x556004f78c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc66d3d4640 1 -- 172.21.15.59:0/1150655384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556004e76270 con 0x556004f9d090 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9d090 0x556004fa5e90 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc65c02bee0 tx=0x7fc65c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3941cfba3a533971 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc64cff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc64cff9640 1 -- 172.21.15.59:0/1150655384 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c035040 con 0x556004f9d090 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc64cff9640 1 -- 172.21.15.59:0/1150655384 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc65c044810 con 0x556004f9d090 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.967+0000 7fc672a29740 1 -- 172.21.15.59:0/1150655384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc65805d160 con 0x556004f9d090 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc64cff9640 1 -- 172.21.15.59:0/1150655384 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc65c05b050 con 0x556004f9d090 2024-03-06T07:26:26.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc672a29740 10 client.7569 fetch_fsmap learned FSMap version 49 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc672a29740 1 -- 172.21.15.59:0/1150655384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x556004bce2d0 con 0x556004f9d090 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc64cff9640 1 -- 172.21.15.59:0/1150655384 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65805d160 con 0x556004f9d090 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc64cff9640 1 -- 172.21.15.59:0/1150655384 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 49) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc65c05e020 con 0x556004f9d090 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc672a29740 10 client.7569 fetch_fsmap finished waiting for FSMap version 49 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc672a29740 10 client.7569 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc672a29740 20 client.7569 populate_metadata read hostname 'smithi059' 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc672a29740 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:6835/4292771408,v1:172.21.15.164:6837/4292771408] conn(0x556004fa4210 0x556004fa45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc672a29740 4 client.7569 mds_command: new command op to 7422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc672a29740 1 -- 172.21.15.59:0/1150655384 --> [v2:172.21.15.164:6835/4292771408,v1:172.21.15.164:6837/4292771408] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556004ec6bf0 con 0x556004fa4210 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:6835/4292771408,v1:172.21.15.164:6837/4292771408] conn(0x556004fa4210 0x556004fa45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc64cff9640 10 client.7569 ms_handle_connect on v2:172.21.15.164:6835/4292771408 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.968+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:6835/4292771408,v1:172.21.15.164:6837/4292771408] conn(0x556004fa4210 0x556004fa45f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6af2c822a699376b server_cookie=90c881f6f4795674 in_seq=0 out_seq=0 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc64cff9640 1 -- 172.21.15.59:0/1150655384 <== mds.0 v2:172.21.15.164:6835/4292771408 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556004ec6bf0 con 0x556004fa4210 2024-03-06T07:26:26.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc64cff9640 10 client.7569 handle_command_reply: tid=0 2024-03-06T07:26:26.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 1 client.7569 shutdown 2024-03-06T07:26:26.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 1 -- 172.21.15.59:0/1150655384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9d090 msgr2=0x556004fa5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9d090 0x556004fa5e90 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fc65c02bee0 tx=0x7fc65c004730 comp rx=0 tx=0).stop 2024-03-06T07:26:26.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 1 -- 172.21.15.59:0/1150655384 shutdown_connections 2024-03-06T07:26:26.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:6835/4292771408,v1:172.21.15.164:6837/4292771408] conn(0x556004fa4210 0x556004fa45f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f807a0 0x556004faf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9d090 0x556004fa5e90 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc64cff9640 0 client.7569 ms_handle_reset on v2:172.21.15.164:6835/4292771408 2024-03-06T07:26:26.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 1 --2- 172.21.15.59:0/1150655384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9c7b0 0x556004f78c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 1 -- 172.21.15.59:0/1150655384 >> 172.21.15.59:0/1150655384 conn(0x556004f7ccc0 msgr2=0x556004f9bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 1 -- 172.21.15.59:0/1150655384 shutdown_connections 2024-03-06T07:26:26.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 1 -- 172.21.15.59:0/1150655384 wait complete. 2024-03-06T07:26:26.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.969+0000 7fc672a29740 20 client.7569 trim_cache size 0 max 16384 2024-03-06T07:26:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.986+0000 7fc672a29740 1 Processor -- start 2024-03-06T07:26:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.986+0000 7fc672a29740 1 -- start start 2024-03-06T07:26:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc672a29740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f807a0 0x556004f53820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc672a29740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9c7b0 0x556004f827d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc672a29740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9d090 0x556004f81300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc672a29740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc668168bf0 con 0x556004f807a0 2024-03-06T07:26:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc672a29740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc668167ff0 con 0x556004f9d090 2024-03-06T07:26:26.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc672a29740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6681685f0 con 0x556004f9c7b0 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66dbd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f807a0 0x556004f53820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66dbd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f807a0 0x556004f53820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60570/0 (socket says 172.21.15.59:60570) 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66dbd5640 1 -- 172.21.15.59:0/1806727230 learned_addr learned my addr 172.21.15.59:0/1806727230 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66e3d6640 1 --2- 172.21.15.59:0/1806727230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9d090 0x556004f81300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/1806727230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9c7b0 0x556004f827d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66dbd5640 1 -- 172.21.15.59:0/1806727230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9c7b0 msgr2=0x556004f827d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/1806727230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9c7b0 0x556004f827d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66dbd5640 1 -- 172.21.15.59:0/1806727230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9d090 msgr2=0x556004f81300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/1806727230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9d090 0x556004f81300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66dbd5640 1 -- 172.21.15.59:0/1806727230 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556004c986b0 con 0x556004f807a0 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66e3d6640 1 --2- 172.21.15.59:0/1806727230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9d090 0x556004f81300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.987+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/1806727230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f807a0 0x556004f53820 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc658057690 tx=0x7fc65805c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d187dc1d172030 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc64effd640 1 -- 172.21.15.59:0/1806727230 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658087040 con 0x556004f807a0 2024-03-06T07:26:26.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc64effd640 1 -- 172.21.15.59:0/1806727230 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65805d160 con 0x556004f807a0 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc64effd640 1 -- 172.21.15.59:0/1806727230 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6580615b0 con 0x556004f807a0 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc672a29740 1 -- 172.21.15.59:0/1806727230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f807a0 msgr2=0x556004f53820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc672a29740 1 --2- 172.21.15.59:0/1806727230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f807a0 0x556004f53820 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc658057690 tx=0x7fc65805c000 comp rx=0 tx=0).stop 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc672a29740 1 -- 172.21.15.59:0/1806727230 shutdown_connections 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc672a29740 1 --2- 172.21.15.59:0/1806727230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f9d090 0x556004f81300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc672a29740 1 --2- 172.21.15.59:0/1806727230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9c7b0 0x556004f827d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc672a29740 1 --2- 172.21.15.59:0/1806727230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f807a0 0x556004f53820 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc672a29740 1 -- 172.21.15.59:0/1806727230 >> 172.21.15.59:0/1806727230 conn(0x556004f7ccc0 msgr2=0x556004f7f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc672a29740 1 -- 172.21.15.59:0/1806727230 shutdown_connections 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.988+0000 7fc672a29740 1 -- 172.21.15.59:0/1806727230 wait complete. 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.989+0000 7fc672a29740 1 -- start start 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.989+0000 7fc672a29740 1 Processor -- start 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc672a29740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f807a0 0x556004fb8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc672a29740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9c7b0 0x556004fb91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc672a29740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9d090 0x556004f4f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc672a29740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556004fafbb0 con 0x556004f9c7b0 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc672a29740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x556004fa53b0 con 0x556004f807a0 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc672a29740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x556004f54950 con 0x556004f9d090 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc672a29740 10 client.0 _ll_register_callbacks cb 0x7fc65c031a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc66d3d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9c7b0 0x556004fb91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc66d3d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9c7b0 0x556004fb91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60578/0 (socket says 172.21.15.59:60578) 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc66d3d4640 1 -- 172.21.15.59:0/2644581571 learned_addr learned my addr 172.21.15.59:0/2644581571 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f807a0 0x556004fb8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc66e3d6640 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9d090 0x556004f4f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc66d3d4640 1 -- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9d090 msgr2=0x556004f4f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9d090 0x556004f4f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc66d3d4640 1 -- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f807a0 msgr2=0x556004fb8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f807a0 0x556004fb8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc66d3d4640 1 -- 172.21.15.59:0/2644581571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556004f4f580 con 0x556004f9c7b0 2024-03-06T07:26:26.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc66d3d4640 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9c7b0 0x556004fb91d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc65c02bee0 tx=0x7fc65c031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1631df506fb11730 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc64cff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc64cff9640 1 -- 172.21.15.59:0/2644581571 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c04e810 con 0x556004f9c7b0 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc64cff9640 1 -- 172.21.15.59:0/2644581571 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc65c04e9b0 con 0x556004f9c7b0 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.990+0000 7fc672a29740 1 -- 172.21.15.59:0/2644581571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc65805d160 con 0x556004f9c7b0 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc64cff9640 1 -- 172.21.15.59:0/2644581571 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc65c070050 con 0x556004f9c7b0 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc672a29740 10 client.7575 fetch_fsmap learned FSMap version 49 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc672a29740 1 -- 172.21.15.59:0/2644581571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x556004f4f300 con 0x556004f9c7b0 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc64cff9640 1 -- 172.21.15.59:0/2644581571 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65805d160 con 0x556004f9c7b0 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc64cff9640 1 -- 172.21.15.59:0/2644581571 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 49) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc65c073020 con 0x556004f9c7b0 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc672a29740 10 client.7575 fetch_fsmap finished waiting for FSMap version 49 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc672a29740 10 client.7575 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-06T07:26:27.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc672a29740 20 client.7575 populate_metadata read hostname 'smithi059' 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc672a29740 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:6835/4292771408,v1:172.21.15.164:6837/4292771408] conn(0x556004f4ff80 0x556004f76fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc672a29740 4 client.7575 mds_command: new command op to 7422 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc672a29740 1 -- 172.21.15.59:0/2644581571 --> [v2:172.21.15.164:6835/4292771408,v1:172.21.15.164:6837/4292771408] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556004f52050 con 0x556004f4ff80 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:6835/4292771408,v1:172.21.15.164:6837/4292771408] conn(0x556004f4ff80 0x556004f76fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc64cff9640 10 client.7575 ms_handle_connect on v2:172.21.15.164:6835/4292771408 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.991+0000 7fc66dbd5640 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:6835/4292771408,v1:172.21.15.164:6837/4292771408] conn(0x556004f4ff80 0x556004f76fc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f9d01073aeaedd0 server_cookie=b8c21efa3df9ee9e in_seq=0 out_seq=0 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc64cff9640 1 -- 172.21.15.59:0/2644581571 <== mds.0 v2:172.21.15.164:6835/4292771408 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x556004f52050 con 0x556004f4ff80 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc64cff9640 10 client.7575 handle_command_reply: tid=0 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 1 client.7575 shutdown 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 1 -- 172.21.15.59:0/2644581571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9c7b0 msgr2=0x556004fb91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9c7b0 0x556004fb91d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc65c02bee0 tx=0x7fc65c031000 comp rx=0 tx=0).stop 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 1 -- 172.21.15.59:0/2644581571 shutdown_connections 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:6835/4292771408,v1:172.21.15.164:6837/4292771408] conn(0x556004f4ff80 0x556004f76fc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556004f9d090 0x556004f4f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556004f9c7b0 0x556004fb91d0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 1 --2- 172.21.15.59:0/2644581571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556004f807a0 0x556004fb8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 1 -- 172.21.15.59:0/2644581571 >> 172.21.15.59:0/2644581571 conn(0x556004f7ccc0 msgr2=0x556004f9bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc64cff9640 0 client.7575 ms_handle_reset on v2:172.21.15.164:6835/4292771408 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 1 -- 172.21.15.59:0/2644581571 shutdown_connections 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 1 -- 172.21.15.59:0/2644581571 wait complete. 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc672a29740 20 client.7575 trim_cache size 0 max 16384 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:26:27.002 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 7422, 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 9.1291352670000006, 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 49, 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 92, 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 92, 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 14.140947154999999 2024-03-06T07:26:27.003 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:26:27.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc66fe39640 1 -- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc65406b4d0 msgr2=0x7fc65406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc66fe39640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc65406b4d0 0x7fc65406d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc65c036000 tx=0x7fc65c03c040 comp rx=0 tx=0).stop 2024-03-06T07:26:27.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc66fe39640 1 -- 172.21.15.59:0/3623860521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc66806cb90 msgr2=0x7fc668138750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc66fe39640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc66806cb90 0x7fc668138750 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fc65800efc0 tx=0x7fc65800c490 comp rx=0 tx=0).stop 2024-03-06T07:26:27.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc66fe39640 1 -- 172.21.15.59:0/3623860521 shutdown_connections 2024-03-06T07:26:27.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc66fe39640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc65406b4d0 0x7fc65406d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc66fe39640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc66815b380 0x7fc6681398d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc66fe39640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc668158880 0x7fc66814fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc66fe39640 1 --2- 172.21.15.59:0/3623860521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc66806cb90 0x7fc668138750 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.992+0000 7fc66fe39640 1 -- 172.21.15.59:0/3623860521 >> 172.21.15.59:0/3623860521 conn(0x7fc6680b1c80 msgr2=0x7fc66813b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.993+0000 7fc66fe39640 1 -- 172.21.15.59:0/3623860521 shutdown_connections 2024-03-06T07:26:27.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:26.993+0000 7fc66fe39640 1 -- 172.21.15.59:0/3623860521 wait complete. 2024-03-06T07:26:27.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:26:27.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.105+0000 7fef391e2640 1 Processor -- start 2024-03-06T07:26:27.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.105+0000 7fef391e2640 1 -- start start 2024-03-06T07:26:27.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef391e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef340d4920 0x7fef340cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef391e2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fef341625a0 0x7fef3415dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef391e2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef340ceac0 con 0x7fef341621c0 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef391e2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef340cf690 con 0x7fef341625a0 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef391e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef340d2dc0 con 0x7fef340d4920 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef3359e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fef341625a0 0x7fef3415dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef32d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fef341621c0 0x7fef340d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:54568/0 (socket says 172.21.15.59:54568) 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef3259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef340d4920 0x7fef340cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef32d9d640 1 -- 172.21.15.59:0/1891874156 learned_addr learned my addr 172.21.15.59:0/1891874156 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef3359e640 1 -- 172.21.15.59:0/1891874156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef340d4920 msgr2=0x7fef340cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef3359e640 1 --2- 172.21.15.59:0/1891874156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef340d4920 0x7fef340cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef3359e640 1 -- 172.21.15.59:0/1891874156 >> v1:172.21.15.164:6789/0 conn(0x7fef341621c0 legacy=0x7fef340d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef3359e640 1 -- 172.21.15.59:0/1891874156 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef340ced60 con 0x7fef341625a0 2024-03-06T07:26:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef31d9b640 1 -- 172.21.15.59:0/1891874156 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262523838 0 0) 0x7fef340ceac0 con 0x7fef341621c0 2024-03-06T07:26:27.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.106+0000 7fef3259c640 1 --2- 172.21.15.59:0/1891874156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef340d4920 0x7fef340cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:27.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.107+0000 7fef3359e640 1 --2- 172.21.15.59:0/1891874156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef341625a0 0x7fef3415dd40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fef24009d90 tx=0x7fef2402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ff8ffd488e61026 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.107+0000 7fef31d9b640 1 -- 172.21.15.59:0/1891874156 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24039070 con 0x7fef341625a0 2024-03-06T07:26:27.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.107+0000 7fef31d9b640 1 -- 172.21.15.59:0/1891874156 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2402cc50 con 0x7fef341625a0 2024-03-06T07:26:27.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.107+0000 7fef31d9b640 1 -- 172.21.15.59:0/1891874156 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24034440 con 0x7fef341625a0 2024-03-06T07:26:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.108+0000 7fef391e2640 1 -- 172.21.15.59:0/1891874156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef341625a0 msgr2=0x7fef3415dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.108+0000 7fef391e2640 1 --2- 172.21.15.59:0/1891874156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef341625a0 0x7fef3415dd40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fef24009d90 tx=0x7fef2402bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.108+0000 7fef391e2640 1 -- 172.21.15.59:0/1891874156 shutdown_connections 2024-03-06T07:26:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.108+0000 7fef391e2640 1 --2- 172.21.15.59:0/1891874156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef341625a0 0x7fef3415dd40 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.108+0000 7fef391e2640 1 --2- 172.21.15.59:0/1891874156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef340d4920 0x7fef340cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.108+0000 7fef391e2640 1 -- 172.21.15.59:0/1891874156 >> 172.21.15.59:0/1891874156 conn(0x7fef340d3d10 msgr2=0x7fef340d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.108+0000 7fef391e2640 1 -- 172.21.15.59:0/1891874156 shutdown_connections 2024-03-06T07:26:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.108+0000 7fef391e2640 1 -- 172.21.15.59:0/1891874156 wait complete. 2024-03-06T07:26:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.109+0000 7fef391e2640 1 Processor -- start 2024-03-06T07:26:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.109+0000 7fef391e2640 1 -- start start 2024-03-06T07:26:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.110+0000 7fef391e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef340d4920 0x7fef34152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.110+0000 7fef391e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef341621c0 0x7fef3409e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.110+0000 7fef391e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef341625a0 0x7fef340cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.110+0000 7fef391e2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3415e280 con 0x7fef341625a0 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.110+0000 7fef391e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef340cf320 con 0x7fef340d4920 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.110+0000 7fef391e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3415e400 con 0x7fef341621c0 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.110+0000 7fef3359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef341625a0 0x7fef340cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.110+0000 7fef3359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef341625a0 0x7fef340cd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60594/0 (socket says 172.21.15.59:60594) 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.110+0000 7fef3359e640 1 -- 172.21.15.59:0/4096538314 learned_addr learned my addr 172.21.15.59:0/4096538314 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.110+0000 7fef32d9d640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef340d4920 0x7fef34152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.110+0000 7fef3259c640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef341621c0 0x7fef3409e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.111+0000 7fef3359e640 1 -- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef341621c0 msgr2=0x7fef3409e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.111+0000 7fef3359e640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef341621c0 0x7fef3409e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.111+0000 7fef3359e640 1 -- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef340d4920 msgr2=0x7fef34152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.111+0000 7fef3359e640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef340d4920 0x7fef34152100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.111+0000 7fef3359e640 1 -- 172.21.15.59:0/4096538314 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3409edf0 con 0x7fef341625a0 2024-03-06T07:26:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.111+0000 7fef3359e640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef341625a0 0x7fef340cd5d0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fef24037040 tx=0x7fef24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.111+0000 7fef2b7fe640 1 -- 172.21.15.59:0/4096538314 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24044020 con 0x7fef341625a0 2024-03-06T07:26:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.111+0000 7fef2b7fe640 1 -- 172.21.15.59:0/4096538314 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2402ca50 con 0x7fef341625a0 2024-03-06T07:26:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.111+0000 7fef391e2640 1 -- 172.21.15.59:0/4096538314 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef340cb030 con 0x7fef341625a0 2024-03-06T07:26:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.111+0000 7fef2b7fe640 1 -- 172.21.15.59:0/4096538314 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24039040 con 0x7fef341625a0 2024-03-06T07:26:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.111+0000 7fef391e2640 1 -- 172.21.15.59:0/4096538314 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef340cb5f0 con 0x7fef341625a0 2024-03-06T07:26:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.112+0000 7fef391e2640 1 -- 172.21.15.59:0/4096538314 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3404f330 con 0x7fef341625a0 2024-03-06T07:26:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.112+0000 7fef2b7fe640 1 -- 172.21.15.59:0/4096538314 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef24047020 con 0x7fef341625a0 2024-03-06T07:26:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.113+0000 7fef2b7fe640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef1406b4d0 0x7fef1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.113+0000 7fef2b7fe640 1 -- 172.21.15.59:0/4096538314 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef240ac8b0 con 0x7fef341625a0 2024-03-06T07:26:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.113+0000 7fef32d9d640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef1406b4d0 0x7fef1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.114+0000 7fef32d9d640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef1406b4d0 0x7fef1406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fef2c000f60 tx=0x7fef2c007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:27.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.117+0000 7fef2b7fe640 1 -- 172.21.15.59:0/4096538314 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef2404e030 con 0x7fef341625a0 2024-03-06T07:26:27.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.339+0000 7fef391e2640 1 -- 172.21.15.59:0/4096538314 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fef3415e830 con 0x7fef341625a0 2024-03-06T07:26:27.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.340+0000 7fef2b7fe640 1 -- 172.21.15.59:0/4096538314 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v49) v1 ==== 53+0+73 (secure 0 0 0) 0x7fef2407b170 con 0x7fef341625a0 2024-03-06T07:26:27.342 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:26:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 -- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef1406b4d0 msgr2=0x7fef1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef1406b4d0 0x7fef1406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fef2c000f60 tx=0x7fef2c007d50 comp rx=0 tx=0).stop 2024-03-06T07:26:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 -- 172.21.15.59:0/4096538314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef341625a0 msgr2=0x7fef340cd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef341625a0 0x7fef340cd5d0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fef24037040 tx=0x7fef24004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 -- 172.21.15.59:0/4096538314 shutdown_connections 2024-03-06T07:26:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fef1406b4d0 0x7fef1406d990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef341625a0 0x7fef340cd5d0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef341621c0 0x7fef3409e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 --2- 172.21.15.59:0/4096538314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef340d4920 0x7fef34152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 -- 172.21.15.59:0/4096538314 >> 172.21.15.59:0/4096538314 conn(0x7fef340d3d10 msgr2=0x7fef340d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 -- 172.21.15.59:0/4096538314 shutdown_connections 2024-03-06T07:26:27.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:27.342+0000 7fef391e2640 1 -- 172.21.15.59:0/4096538314 wait complete. 2024-03-06T07:26:27.354 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:26:27.355 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:26:27.355 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:26:27.355 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:26:27.355 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:27.355 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:26:27.355 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:26:27.355 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:26:27.355 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:27.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:27.355 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:26:27.372 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:26:27.372 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:26:27.372 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:26:27.372 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:27.372 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:26:27.372 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:27.372 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:27.372 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:27.372 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42391sec preferred_lft 42391sec 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:26:27.373 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:26:27.374 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:27.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:27.375 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:26:27.438 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:26:27.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:27.439 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:26:27.495 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:26:27.495 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:26:27.495 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:26:27.495 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:26:27.495 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:26:27.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:27.496 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:26:27.496 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:26:27.496 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:26:27.496 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:26:27.496 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:26:27.496 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:26:27.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:27.589 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:26:27.604 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:26:27.604 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:26:27.604 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:27.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:27.604 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:26:27.671 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:26:27.671 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:27.672 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:27.726 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:27.726 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:27.726 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:27.781 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:26:27.847 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:26:27.903 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:26:27.904 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:26:27.905 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:27.905 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:27.905 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:27.905 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:27.905 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:26:27.905 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:27.905 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:27.905 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:27.905 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:27.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:27.905 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:26:27.956 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:26:27.957 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:26:27.999 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:26:28.025 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:28.081 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:26:28.081 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:26:28.081 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:26:28.081 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:26:28.081 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:26:28.081 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:26:28.081 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:26:28.081 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:28.082 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:28.083 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:28.084 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:28.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:28.084 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:26:28.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:26:28.082+0000 7f79884f8480 -1 init, newargv = 0x555ed2a535b0 newargc=15 2024-03-06T07:26:28.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[68746]: starting ceph client 2024-03-06T07:26:28.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[68746]: starting fuse 2024-03-06T07:26:28.099 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:26:28.099 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:26:28.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:26:28.099 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:26:28.099 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:26:28.099 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:26:28.099 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:26:28.099 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:26:28.099 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:26:28.099 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:26:28.100 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:26:28.101 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:26:28.101 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:26:28.101 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:26:28.220 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.68746.asok 2024-03-06T07:26:28.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:28.230 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.68746.asok 2024-03-06T07:26:28.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:28.230 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.68746.asok status 2024-03-06T07:26:28.367 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:26:28.367 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:26:28.367 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:26:28.367 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:26:28.367 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:26:28.367 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:26:28.367 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:26:28.367 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "68746", 2024-03-06T07:26:28.367 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:26:28.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:28.367 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 7584, 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 7584 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 931259349 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:28.368 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 931259349 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.7584 192.168.0.1:0/931259349", 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/931259349", 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 49, 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 92, 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:26:28.369 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:26:28.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:28.375 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:26:28.375 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:28.390 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:26:28.390 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:28.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:28.391 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:28.459 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:28.500 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:28.547 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-03-06T07:26:28.547 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-03-06T07:26:28.547 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:28.548 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:28.549 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-03-06T07:26:28.550 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-03-06T07:26:28.551 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: bdev async discard = True 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: bdev enable discard = True 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore allocator = bitmap 2024-03-06T07:26:28.552 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 1/20 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 1/20 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 4/10 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-03-06T07:26:28.553 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: mon op complaint time = 120 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: client mount timeout = 600 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: debug client = 20 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = "" 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:26:28.556 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout:[mds] 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds = 20 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds balancer = 20 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug frag = True 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug scatterstat = True 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout: mds op complaint time = 180 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout: mds verify scatter = True 2024-03-06T07:26:28.557 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:26:28.558 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:26:28.558 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:26:28.558 INFO:teuthology.orchestra.run.smithi059.stdout: mds_max_purge_files = 6 2024-03-06T07:26:28.558 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-03-06T07:26:28.558 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-03-06T07:26:28.558 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:28.558 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-03-06T07:26:28.558 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.558 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-03-06T07:26:28.558 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.558 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.559 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.560 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-03-06T07:26:28.561 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-03-06T07:26:28.562 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = bitmap 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2024-03-06T07:26:28.563 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:28.564 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:28.565 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: mon op complaint time = 120 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-03-06T07:26:28.566 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout: client mount timeout = 600 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout: debug client = 20 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = "" 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2024-03-06T07:26:28.567 INFO:teuthology.orchestra.run.smithi164.stdout:[mds] 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds = 20 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds balancer = 20 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug frag = True 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug scatterstat = True 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi164.stdout: mds op complaint time = 180 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi164.stdout: mds verify scatter = True 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:26:28.568 INFO:teuthology.orchestra.run.smithi164.stdout: mds_max_purge_files = 6 2024-03-06T07:26:28.570 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:28.612 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:28.659 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-03-06T07:26:28.659 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:28.660 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:28.661 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-03-06T07:26:28.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:28.663 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: bdev async discard = True 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: bdev enable discard = True 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore allocator = bitmap 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-03-06T07:26:28.664 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-03-06T07:26:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 1/20 2024-03-06T07:26:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 1/20 2024-03-06T07:26:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-03-06T07:26:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 4/10 2024-03-06T07:26:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:28.665 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-03-06T07:26:28.666 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:28.667 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-03-06T07:26:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: mon op complaint time = 120 2024-03-06T07:26:28.668 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: client mount timeout = 600 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: debug client = 20 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = "" 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-03-06T07:26:28.669 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout:[mds] 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds = 20 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds balancer = 20 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug frag = True 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug scatterstat = True 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: mds op complaint time = 180 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: mds verify scatter = True 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:26:28.670 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:26:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:26:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: mds_max_purge_files = 6 2024-03-06T07:26:28.671 INFO:teuthology.orchestra.run.smithi059.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:28.671 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-03-06T07:26:28.671 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-03-06T07:26:28.671 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:28.671 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-03-06T07:26:28.671 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.671 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-03-06T07:26:28.671 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.672 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.673 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:28.674 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.675 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2024-03-06T07:26:28.676 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = bitmap 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.677 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-03-06T07:26:28.678 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-03-06T07:26:28.679 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: mon op complaint time = 120 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: client mount timeout = 600 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: debug client = 20 2024-03-06T07:26:28.680 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = "" 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout:[mds] 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds = 20 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds balancer = 20 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug frag = True 2024-03-06T07:26:28.681 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug scatterstat = True 2024-03-06T07:26:28.682 INFO:teuthology.orchestra.run.smithi164.stdout: mds op complaint time = 180 2024-03-06T07:26:28.682 INFO:teuthology.orchestra.run.smithi164.stdout: mds verify scatter = True 2024-03-06T07:26:28.682 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:26:28.682 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:26:28.682 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:26:28.682 INFO:teuthology.orchestra.run.smithi164.stdout: mds_max_purge_files = 6 2024-03-06T07:26:28.682 INFO:teuthology.orchestra.run.smithi164.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:28.684 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:28.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.826+0000 7fcf7453a640 1 Processor -- start 2024-03-06T07:26:28.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.826+0000 7fcf7453a640 1 -- start start 2024-03-06T07:26:28.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.826+0000 7fcf7453a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcf6c161e60 0x7fcf6c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.826+0000 7fcf7453a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c162b70 con 0x7fcf6c06deb0 2024-03-06T07:26:28.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.826+0000 7fcf7453a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c06d2c0 con 0x7fcf6c0b1540 2024-03-06T07:26:28.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.827+0000 7fcf7453a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6c162950 con 0x7fcf6c161e60 2024-03-06T07:26:28.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.827+0000 7fcf71ad5640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcf6c06deb0 0x7fcf6c162240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49946/0 (socket says 172.21.15.59:49946) 2024-03-06T07:26:28.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.827+0000 7fcf71ad5640 1 -- 172.21.15.59:0/324323851 learned_addr learned my addr 172.21.15.59:0/324323851 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:28.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.827+0000 7fcf722d6640 1 --2- 172.21.15.59:0/324323851 >> v2:172.21.15.164:3300/0 conn(0x7fcf6c161e60 0x7fcf6c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.827+0000 7fcf712d4640 1 -- 172.21.15.59:0/324323851 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623237283 0 0) 0x7fcf6c162b70 con 0x7fcf6c06deb0 2024-03-06T07:26:28.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.827+0000 7fcf712d4640 1 -- 172.21.15.59:0/324323851 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf58003520 con 0x7fcf6c06deb0 2024-03-06T07:26:28.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.827+0000 7fcf712d4640 1 -- 172.21.15.59:0/324323851 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210596217 0 0) 0x7fcf6c06d2c0 con 0x7fcf6c0b1540 2024-03-06T07:26:28.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.827+0000 7fcf712d4640 1 -- 172.21.15.59:0/324323851 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf6c162b70 con 0x7fcf6c0b1540 2024-03-06T07:26:28.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.827+0000 7fcf722d6640 1 -- 172.21.15.59:0/324323851 >> v1:172.21.15.164:6789/0 conn(0x7fcf6c0b1540 legacy=0x7fcf6c0b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.827+0000 7fcf722d6640 1 -- 172.21.15.59:0/324323851 >> v1:172.21.15.59:6789/0 conn(0x7fcf6c06deb0 legacy=0x7fcf6c162240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.827+0000 7fcf722d6640 1 -- 172.21.15.59:0/324323851 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c06d5a0 con 0x7fcf6c161e60 2024-03-06T07:26:28.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.828+0000 7fcf722d6640 1 --2- 172.21.15.59:0/324323851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6c161e60 0x7fcf6c0b0c30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcf680097a0 tx=0x7fcf6802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e878ff961dc763f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.828+0000 7fcf712d4640 1 -- 172.21.15.59:0/324323851 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf68034070 con 0x7fcf6c161e60 2024-03-06T07:26:28.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.828+0000 7fcf712d4640 1 -- 172.21.15.59:0/324323851 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf68030610 con 0x7fcf6c161e60 2024-03-06T07:26:28.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.828+0000 7fcf712d4640 1 -- 172.21.15.59:0/324323851 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf680309c0 con 0x7fcf6c161e60 2024-03-06T07:26:28.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.829+0000 7fcf7453a640 1 -- 172.21.15.59:0/324323851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6c161e60 msgr2=0x7fcf6c0b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.829+0000 7fcf7453a640 1 --2- 172.21.15.59:0/324323851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6c161e60 0x7fcf6c0b0c30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcf680097a0 tx=0x7fcf6802b960 comp rx=0 tx=0).stop 2024-03-06T07:26:28.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.829+0000 7fcf7453a640 1 -- 172.21.15.59:0/324323851 shutdown_connections 2024-03-06T07:26:28.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.829+0000 7fcf7453a640 1 --2- 172.21.15.59:0/324323851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6c161e60 0x7fcf6c0b0c30 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.829+0000 7fcf7453a640 1 -- 172.21.15.59:0/324323851 >> 172.21.15.59:0/324323851 conn(0x7fcf6c0d1c00 msgr2=0x7fcf6c0d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:28.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.829+0000 7fcf7453a640 1 -- 172.21.15.59:0/324323851 shutdown_connections 2024-03-06T07:26:28.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.829+0000 7fcf7453a640 1 -- 172.21.15.59:0/324323851 wait complete. 2024-03-06T07:26:28.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.830+0000 7fcf7453a640 1 Processor -- start 2024-03-06T07:26:28.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.830+0000 7fcf7453a640 1 -- start start 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.830+0000 7fcf7453a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6c06deb0 0x7fcf6c159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.830+0000 7fcf7453a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf6c0b1540 0x7fcf6c159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.830+0000 7fcf7453a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c161e60 0x7fcf6c1e7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.830+0000 7fcf7453a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c0b3210 con 0x7fcf6c161e60 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.830+0000 7fcf7453a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c0b3090 con 0x7fcf6c06deb0 2024-03-06T07:26:28.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.830+0000 7fcf7453a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c0b2100 con 0x7fcf6c0b1540 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf72ad7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c161e60 0x7fcf6c1e7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf72ad7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c161e60 0x7fcf6c1e7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60604/0 (socket says 172.21.15.59:60604) 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf72ad7640 1 -- 172.21.15.59:0/707039362 learned_addr learned my addr 172.21.15.59:0/707039362 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf722d6640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6c06deb0 0x7fcf6c159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf71ad5640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf6c0b1540 0x7fcf6c159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf72ad7640 1 -- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf6c0b1540 msgr2=0x7fcf6c159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf72ad7640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf6c0b1540 0x7fcf6c159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf72ad7640 1 -- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6c06deb0 msgr2=0x7fcf6c159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf72ad7640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6c06deb0 0x7fcf6c159830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf72ad7640 1 -- 172.21.15.59:0/707039362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c15cfd0 con 0x7fcf6c161e60 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf72ad7640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c161e60 0x7fcf6c1e7b60 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fcf6400c790 tx=0x7fcf6400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf62ffd640 1 -- 172.21.15.59:0/707039362 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64003b70 con 0x7fcf6c161e60 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf62ffd640 1 -- 172.21.15.59:0/707039362 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf64003d10 con 0x7fcf6c161e60 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf7453a640 1 -- 172.21.15.59:0/707039362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6c15e8f0 con 0x7fcf6c161e60 2024-03-06T07:26:28.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf62ffd640 1 -- 172.21.15.59:0/707039362 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6400aa50 con 0x7fcf6c161e60 2024-03-06T07:26:28.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.831+0000 7fcf7453a640 1 -- 172.21.15.59:0/707039362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c15eeb0 con 0x7fcf6c161e60 2024-03-06T07:26:28.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.832+0000 7fcf7453a640 1 -- 172.21.15.59:0/707039362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6c068fb0 con 0x7fcf6c161e60 2024-03-06T07:26:28.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.833+0000 7fcf62ffd640 1 -- 172.21.15.59:0/707039362 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf6400abf0 con 0x7fcf6c161e60 2024-03-06T07:26:28.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.833+0000 7fcf62ffd640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcf5006b4d0 0x7fcf5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:28.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.833+0000 7fcf62ffd640 1 -- 172.21.15.59:0/707039362 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf6408a080 con 0x7fcf6c161e60 2024-03-06T07:26:28.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.833+0000 7fcf722d6640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcf5006b4d0 0x7fcf5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:28.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.834+0000 7fcf722d6640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcf5006b4d0 0x7fcf5006d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcf68036000 tx=0x7fcf6803b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:28.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:28.837+0000 7fcf62ffd640 1 -- 172.21.15.59:0/707039362 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf640588c0 con 0x7fcf6c161e60 2024-03-06T07:26:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.018+0000 7fcf7453a640 1 -- 172.21.15.59:0/707039362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcf6c06d2b0 con 0x7fcf6c161e60 2024-03-06T07:26:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.018+0000 7fcf62ffd640 1 -- 172.21.15.59:0/707039362 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcf6405c960 con 0x7fcf6c161e60 2024-03-06T07:26:29.020 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 -- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcf5006b4d0 msgr2=0x7fcf5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcf5006b4d0 0x7fcf5006d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcf68036000 tx=0x7fcf6803b040 comp rx=0 tx=0).stop 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 -- 172.21.15.59:0/707039362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c161e60 msgr2=0x7fcf6c1e7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c161e60 0x7fcf6c1e7b60 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fcf6400c790 tx=0x7fcf6400cc50 comp rx=0 tx=0).stop 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 -- 172.21.15.59:0/707039362 shutdown_connections 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcf5006b4d0 0x7fcf5006d990 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcf6c161e60 0x7fcf6c1e7b60 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf6c0b1540 0x7fcf6c159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 --2- 172.21.15.59:0/707039362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6c06deb0 0x7fcf6c159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 -- 172.21.15.59:0/707039362 >> 172.21.15.59:0/707039362 conn(0x7fcf6c0d1c00 msgr2=0x7fcf6c06f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 -- 172.21.15.59:0/707039362 shutdown_connections 2024-03-06T07:26:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.020+0000 7fcf7453a640 1 -- 172.21.15.59:0/707039362 wait complete. 2024-03-06T07:26:29.033 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:29.036 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:29.084 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-03-06T07:26:29.084 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-03-06T07:26:29.084 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:29.084 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-03-06T07:26:29.084 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.084 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-03-06T07:26:29.084 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.084 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:29.084 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.084 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:29.085 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-03-06T07:26:29.086 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:29.087 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.088 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: bdev async discard = True 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: bdev enable discard = True 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore allocator = bitmap 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 1/20 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 1/20 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 4/10 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:29.089 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.090 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.091 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: mon op complaint time = 120 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:29.092 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout: client mount timeout = 600 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout: debug client = 20 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = "" 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout:[mds] 2024-03-06T07:26:29.093 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds = 20 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds balancer = 20 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug frag = True 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug scatterstat = True 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: mds op complaint time = 180 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: mds verify scatter = True 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: mds_max_purge_files = 6 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:29.094 INFO:teuthology.orchestra.run.smithi059.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-03-06T07:26:29.095 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-03-06T07:26:29.096 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-03-06T07:26:29.097 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:29.098 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.099 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = bitmap 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:29.100 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-03-06T07:26:29.101 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.102 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: mon op complaint time = 120 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:29.103 INFO:teuthology.orchestra.run.smithi164.stdout: client mount timeout = 600 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout: debug client = 20 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = "" 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout:[mds] 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds = 20 2024-03-06T07:26:29.104 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds balancer = 20 2024-03-06T07:26:29.105 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:26:29.105 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug frag = True 2024-03-06T07:26:29.105 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug scatterstat = True 2024-03-06T07:26:29.105 INFO:teuthology.orchestra.run.smithi164.stdout: mds op complaint time = 180 2024-03-06T07:26:29.105 INFO:teuthology.orchestra.run.smithi164.stdout: mds verify scatter = True 2024-03-06T07:26:29.105 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:26:29.105 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:26:29.105 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:26:29.105 INFO:teuthology.orchestra.run.smithi164.stdout: mds_max_purge_files = 6 2024-03-06T07:26:29.105 INFO:teuthology.orchestra.run.smithi164.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:29.105 INFO:teuthology.orchestra.run.smithi164.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:29.107 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:29.107 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:29.107 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:29.107 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:29.108 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:29.108 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:29.108 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:29.108 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:29.158 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:29.158 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:26:29.158 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:29.158 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:26:29.159 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:29.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:26:29.160 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:29.160 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:26:29.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.303+0000 7f381c325640 1 Processor -- start 2024-03-06T07:26:29.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.303+0000 7f381c325640 1 -- start start 2024-03-06T07:26:29.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.303+0000 7f683f1bc640 1 Processor -- start 2024-03-06T07:26:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.303+0000 7f64225a1640 1 Processor -- start 2024-03-06T07:26:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.303+0000 7f64225a1640 1 -- start start 2024-03-06T07:26:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.303+0000 7f683f1bc640 1 -- start start 2024-03-06T07:26:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.303+0000 7f869f017640 1 Processor -- start 2024-03-06T07:26:29.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.303+0000 7f869f017640 1 -- start start 2024-03-06T07:26:29.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f64225a1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 0x7f641c0b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f683f1bc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f68380b3690 0x7f68380b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f683f1bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3fb0 0x7f68380cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f64225a1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641c0ccbd0 con 0x7f641c0ce980 2024-03-06T07:26:29.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f683f1bc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68381440a0 con 0x7f68380ccd70 2024-03-06T07:26:29.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f683f1bc640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68380cf8a0 con 0x7f68380b3690 2024-03-06T07:26:29.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f683f1bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68380b5300 con 0x7f68380b3fb0 2024-03-06T07:26:29.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f683cf58640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f68380b3690 0x7f68380b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f6837fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3fb0 0x7f68380cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f6837fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3fb0 0x7f68380cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:46318/0 (socket says 172.21.15.59:46318) 2024-03-06T07:26:29.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f683cf58640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f68380b3690 0x7f68380b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60612/0 (socket says 172.21.15.59:60612) 2024-03-06T07:26:29.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f6837fff640 1 -- 172.21.15.59:0/1015471208 learned_addr learned my addr 172.21.15.59:0/1015471208 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:29.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f381c325640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f381415e150 0x7f38140cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f64225a1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641c0b3b00 con 0x7f641c162940 2024-03-06T07:26:29.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f64225a1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641c0b3de0 con 0x7f641c0ce5a0 2024-03-06T07:26:29.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f381c325640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38140d1060 0x7f38140b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f6420b3e640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f641c162940 0x7f641c164d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:54580/0 (socket says 172.21.15.59:54580) 2024-03-06T07:26:29.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f6420b3e640 1 -- 172.21.15.59:0/127772407 learned_addr learned my addr 172.21.15.59:0/127772407 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:29.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f381a0c1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f381415e150 0x7f38140cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f641b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f641c0ce980 0x7f641c0cede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49960/0 (socket says 172.21.15.59:49960) 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.304+0000 7f641bfff640 1 --2- 172.21.15.59:0/127772407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 0x7f641c0b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f641affd640 1 -- 172.21.15.59:0/127772407 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801007623 0 0) 0x7f641c0ccbd0 con 0x7f641c0ce980 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f6837fff640 1 -- 172.21.15.59:0/1015471208 >> v2:172.21.15.59:3300/0 conn(0x7f68380b3690 msgr2=0x7f68380b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f6837fff640 1 --2- 172.21.15.59:0/1015471208 >> v2:172.21.15.59:3300/0 conn(0x7f68380b3690 0x7f68380b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f6837fff640 1 -- 172.21.15.59:0/1015471208 >> v1:172.21.15.59:6789/0 conn(0x7f68380ccd70 legacy=0x7f6838070d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f6837fff640 1 -- 172.21.15.59:0/1015471208 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68380ce820 con 0x7f68380b3fb0 2024-03-06T07:26:29.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f641affd640 1 -- 172.21.15.59:0/127772407 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6404003520 con 0x7f641c0ce980 2024-03-06T07:26:29.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f381a0c1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f381415e150 0x7f38140cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:42920/0 (socket says 172.21.15.59:42920) 2024-03-06T07:26:29.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f381a0c1640 1 -- 172.21.15.59:0/2130314819 learned_addr learned my addr 172.21.15.59:0/2130314819 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f641affd640 1 -- 172.21.15.59:0/127772407 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953564816 0 0) 0x7f641c0b3b00 con 0x7f641c162940 2024-03-06T07:26:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f641affd640 1 -- 172.21.15.59:0/127772407 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f641c0ccbd0 con 0x7f641c162940 2024-03-06T07:26:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f381a8c2640 1 --2- 172.21.15.59:0/2130314819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38140d1060 0x7f38140b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f641bfff640 1 -- 172.21.15.59:0/127772407 >> v1:172.21.15.164:6789/0 conn(0x7f641c162940 legacy=0x7f641c164d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f641bfff640 1 -- 172.21.15.59:0/127772407 >> v1:172.21.15.59:6789/0 conn(0x7f641c0ce980 legacy=0x7f641c0cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f381c325640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38140d1ea0 con 0x7f38140cfba0 2024-03-06T07:26:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f641bfff640 1 -- 172.21.15.59:0/127772407 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c0b40c0 con 0x7f641c0ce5a0 2024-03-06T07:26:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869f017640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f869815cfb0 0x7f869815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869f017640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f869806bf90 0x7f869806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869f017640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869806c8e0 con 0x7f869815f490 2024-03-06T07:26:29.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f381c325640 1 -- 172.21.15.59:0/2130314819 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38140d2ab0 con 0x7f381415e150 2024-03-06T07:26:29.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f381c325640 1 -- 172.21.15.59:0/2130314819 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38140d2180 con 0x7f38140d1060 2024-03-06T07:26:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869f017640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869815d420 con 0x7f869815cfb0 2024-03-06T07:26:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869f017640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86980cee30 con 0x7f869806bf90 2024-03-06T07:26:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869d5b4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f869806bf90 0x7f869806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869d5b4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f869806bf90 0x7f869806c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:42934/0 (socket says 172.21.15.59:42934) 2024-03-06T07:26:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869d5b4640 1 -- 172.21.15.59:0/3936247905 learned_addr learned my addr 172.21.15.59:0/3936247905 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f8697fff640 1 --2- 172.21.15.59:0/3936247905 >> v2:172.21.15.59:3300/0 conn(0x7f869815cfb0 0x7f869815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869d5b4640 1 -- 172.21.15.59:0/3936247905 >> v2:172.21.15.59:3300/0 conn(0x7f869815cfb0 msgr2=0x7f869815d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f38190bf640 1 -- 172.21.15.59:0/2130314819 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733335718 0 0) 0x7f38140d1ea0 con 0x7f38140cfba0 2024-03-06T07:26:29.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f381a0c1640 1 -- 172.21.15.59:0/2130314819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38140d1060 msgr2=0x7f38140b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:29.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f381a0c1640 1 --2- 172.21.15.59:0/2130314819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38140d1060 0x7f38140b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f381a0c1640 1 -- 172.21.15.59:0/2130314819 >> v1:172.21.15.164:6789/0 conn(0x7f38140cfba0 legacy=0x7f38140d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f381a0c1640 1 -- 172.21.15.59:0/2130314819 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38140d2460 con 0x7f381415e150 2024-03-06T07:26:29.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869d5b4640 1 --2- 172.21.15.59:0/3936247905 >> v2:172.21.15.59:3300/0 conn(0x7f869815cfb0 0x7f869815d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869d5b4640 1 -- 172.21.15.59:0/3936247905 >> v1:172.21.15.59:6789/0 conn(0x7f869815f490 legacy=0x7f86980722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.305+0000 7f869d5b4640 1 -- 172.21.15.59:0/3936247905 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86980cce10 con 0x7f869806bf90 2024-03-06T07:26:29.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f6837fff640 1 --2- 172.21.15.59:0/1015471208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3fb0 0x7f68380cc830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6824009870 tx=0x7f682402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3577a257ace3aac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f641bfff640 1 --2- 172.21.15.59:0/127772407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 0x7f641c0b4bf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f640c0097d0 tx=0x7f640c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad1c9244a91e140f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f68377fe640 1 -- 172.21.15.59:0/1015471208 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6824034070 con 0x7f68380b3fb0 2024-03-06T07:26:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f68377fe640 1 -- 172.21.15.59:0/1015471208 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6824030840 con 0x7f68380b3fb0 2024-03-06T07:26:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f68377fe640 1 -- 172.21.15.59:0/1015471208 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6824030c10 con 0x7f68380b3fb0 2024-03-06T07:26:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f641affd640 1 -- 172.21.15.59:0/127772407 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c035020 con 0x7f641c0ce5a0 2024-03-06T07:26:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f869d5b4640 1 --2- 172.21.15.59:0/3936247905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f869806bf90 0x7f869806c3a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f86980b5f30 tx=0x7f869002bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef28478a3e441357 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f381a0c1640 1 --2- 172.21.15.59:0/2130314819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381415e150 0x7f38140cf660 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3810009870 tx=0x7f381002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4c1228e9486b5e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f641affd640 1 -- 172.21.15.59:0/127772407 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c031890 con 0x7f641c0ce5a0 2024-03-06T07:26:29.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f641affd640 1 -- 172.21.15.59:0/127772407 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c035020 con 0x7f641c0ce5a0 2024-03-06T07:26:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.306+0000 7f86977fe640 1 -- 172.21.15.59:0/3936247905 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869002cb10 con 0x7f869806bf90 2024-03-06T07:26:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f38190bf640 1 -- 172.21.15.59:0/2130314819 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810035020 con 0x7f381415e150 2024-03-06T07:26:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f86977fe640 1 -- 172.21.15.59:0/3936247905 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869002ccb0 con 0x7f869806bf90 2024-03-06T07:26:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f86977fe640 1 -- 172.21.15.59:0/3936247905 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869002ce60 con 0x7f869806bf90 2024-03-06T07:26:29.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f38190bf640 1 -- 172.21.15.59:0/2130314819 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3810031640 con 0x7f381415e150 2024-03-06T07:26:29.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f38190bf640 1 -- 172.21.15.59:0/2130314819 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381002d040 con 0x7f381415e150 2024-03-06T07:26:29.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f683f1bc640 1 -- 172.21.15.59:0/1015471208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3fb0 msgr2=0x7f68380cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f683f1bc640 1 --2- 172.21.15.59:0/1015471208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3fb0 0x7f68380cc830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6824009870 tx=0x7f682402b980 comp rx=0 tx=0).stop 2024-03-06T07:26:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f683f1bc640 1 -- 172.21.15.59:0/1015471208 shutdown_connections 2024-03-06T07:26:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f683f1bc640 1 --2- 172.21.15.59:0/1015471208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3fb0 0x7f68380cc830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f683f1bc640 1 --2- 172.21.15.59:0/1015471208 >> v2:172.21.15.59:3300/0 conn(0x7f68380b3690 0x7f68380b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f683f1bc640 1 -- 172.21.15.59:0/1015471208 >> 172.21.15.59:0/1015471208 conn(0x7f683806b160 msgr2=0x7f683806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f64225a1640 1 -- 172.21.15.59:0/127772407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 msgr2=0x7f641c0b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f64225a1640 1 --2- 172.21.15.59:0/127772407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 0x7f641c0b4bf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f640c0097d0 tx=0x7f640c02b930 comp rx=0 tx=0).stop 2024-03-06T07:26:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f683f1bc640 1 -- 172.21.15.59:0/1015471208 shutdown_connections 2024-03-06T07:26:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f683f1bc640 1 -- 172.21.15.59:0/1015471208 wait complete. 2024-03-06T07:26:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f64225a1640 1 -- 172.21.15.59:0/127772407 shutdown_connections 2024-03-06T07:26:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f64225a1640 1 --2- 172.21.15.59:0/127772407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 0x7f641c0b4bf0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.307+0000 7f64225a1640 1 -- 172.21.15.59:0/127772407 >> 172.21.15.59:0/127772407 conn(0x7f641c06b160 msgr2=0x7f641c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f64225a1640 1 -- 172.21.15.59:0/127772407 shutdown_connections 2024-03-06T07:26:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f64225a1640 1 -- 172.21.15.59:0/127772407 wait complete. 2024-03-06T07:26:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f869f017640 1 -- 172.21.15.59:0/3936247905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f869806bf90 msgr2=0x7f869806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f869f017640 1 --2- 172.21.15.59:0/3936247905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f869806bf90 0x7f869806c3a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f86980b5f30 tx=0x7f869002bf70 comp rx=0 tx=0).stop 2024-03-06T07:26:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f869f017640 1 -- 172.21.15.59:0/3936247905 shutdown_connections 2024-03-06T07:26:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f869f017640 1 --2- 172.21.15.59:0/3936247905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f869806bf90 0x7f869806c3a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f869f017640 1 --2- 172.21.15.59:0/3936247905 >> v2:172.21.15.59:3300/0 conn(0x7f869815cfb0 0x7f869815d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f869f017640 1 -- 172.21.15.59:0/3936247905 >> 172.21.15.59:0/3936247905 conn(0x7f8698071510 msgr2=0x7f8698071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f869f017640 1 -- 172.21.15.59:0/3936247905 shutdown_connections 2024-03-06T07:26:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f381c325640 1 -- 172.21.15.59:0/2130314819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381415e150 msgr2=0x7f38140cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f381c325640 1 --2- 172.21.15.59:0/2130314819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381415e150 0x7f38140cf660 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3810009870 tx=0x7f381002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f869f017640 1 -- 172.21.15.59:0/3936247905 wait complete. 2024-03-06T07:26:29.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f381c325640 1 -- 172.21.15.59:0/2130314819 shutdown_connections 2024-03-06T07:26:29.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f381c325640 1 --2- 172.21.15.59:0/2130314819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38140d1060 0x7f38140b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f381c325640 1 --2- 172.21.15.59:0/2130314819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381415e150 0x7f38140cf660 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f381c325640 1 -- 172.21.15.59:0/2130314819 >> 172.21.15.59:0/2130314819 conn(0x7f381409d230 msgr2=0x7f38140d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f381c325640 1 -- 172.21.15.59:0/2130314819 shutdown_connections 2024-03-06T07:26:29.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.308+0000 7f381c325640 1 -- 172.21.15.59:0/2130314819 wait complete. 2024-03-06T07:26:29.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.309+0000 7f683f1bc640 1 Processor -- start 2024-03-06T07:26:29.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.309+0000 7f683f1bc640 1 -- start start 2024-03-06T07:26:29.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.309+0000 7f64225a1640 1 Processor -- start 2024-03-06T07:26:29.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.309+0000 7f64225a1640 1 -- start start 2024-03-06T07:26:29.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.309+0000 7f683f1bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3690 0x7f6838141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.309+0000 7f683f1bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68380b3fb0 0x7f68381f3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f869f017640 1 Processor -- start 2024-03-06T07:26:29.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f683f1bc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68380ccd70 0x7f68381f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f869f017640 1 -- start start 2024-03-06T07:26:29.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f683f1bc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68380cf0c0 con 0x7f68380b3fb0 2024-03-06T07:26:29.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f683f1bc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68380cede0 con 0x7f68380ccd70 2024-03-06T07:26:29.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f683f1bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68380cf3a0 con 0x7f68380b3690 2024-03-06T07:26:29.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f683cf58640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3690 0x7f6838141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f6837fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68380b3fb0 0x7f68381f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f683cf58640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3690 0x7f6838141020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:46338/0 (socket says 172.21.15.59:46338) 2024-03-06T07:26:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f683cf58640 1 -- 172.21.15.59:0/72592770 learned_addr learned my addr 172.21.15.59:0/72592770 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f683d759640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68380ccd70 0x7f68381f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f64225a1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 0x7f641c132940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f64225a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f641c0ce980 0x7f641c135a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f64225a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f641c162940 0x7f641c131ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f6837fff640 1 -- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3690 msgr2=0x7f6838141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f6837fff640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3690 0x7f6838141020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f6837fff640 1 -- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68380ccd70 msgr2=0x7f68381f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f6837fff640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68380ccd70 0x7f68381f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f6837fff640 1 -- 172.21.15.59:0/72592770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68381f3fc0 con 0x7f68380b3fb0 2024-03-06T07:26:29.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f683cf58640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3690 0x7f6838141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:29.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f64225a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c0cd030 con 0x7f641c0ce980 2024-03-06T07:26:29.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f64225a1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c0cceb0 con 0x7f641c162940 2024-03-06T07:26:29.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f64225a1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641c071d80 con 0x7f641c0ce5a0 2024-03-06T07:26:29.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f641b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f641c0ce980 0x7f641c135a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f641b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f641c0ce980 0x7f641c135a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60632/0 (socket says 172.21.15.59:60632) 2024-03-06T07:26:29.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f641b7fe640 1 -- 172.21.15.59:0/4157529921 learned_addr learned my addr 172.21.15.59:0/4157529921 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:29.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f641bfff640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 0x7f641c132940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f6420b3e640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f641c162940 0x7f641c131ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f6837fff640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68380b3fb0 0x7f68381f3450 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f6824033000 tx=0x7f6824004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.310+0000 7f381c325640 1 Processor -- start 2024-03-06T07:26:29.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f381c325640 1 -- start start 2024-03-06T07:26:29.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f641b7fe640 1 -- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 msgr2=0x7f641c132940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f641b7fe640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 0x7f641c132940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f641b7fe640 1 -- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f641c162940 msgr2=0x7f641c131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f641b7fe640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f641c162940 0x7f641c131ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f641b7fe640 1 -- 172.21.15.59:0/4157529921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c130d10 con 0x7f641c0ce980 2024-03-06T07:26:29.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f68357fa640 1 -- 172.21.15.59:0/72592770 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6824050070 con 0x7f68380b3fb0 2024-03-06T07:26:29.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f68357fa640 1 -- 172.21.15.59:0/72592770 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6824030640 con 0x7f68380b3fb0 2024-03-06T07:26:29.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f641bfff640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 0x7f641c132940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:29.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f68357fa640 1 -- 172.21.15.59:0/72592770 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6824034040 con 0x7f68380b3fb0 2024-03-06T07:26:29.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f6420b3e640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f641c162940 0x7f641c131ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:29.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f683f1bc640 1 -- 172.21.15.59:0/72592770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68381f42a0 con 0x7f68380b3fb0 2024-03-06T07:26:29.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f683f1bc640 1 -- 172.21.15.59:0/72592770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68380aaec0 con 0x7f68380b3fb0 2024-03-06T07:26:29.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f641b7fe640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f641c0ce980 0x7f641c135a70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f641000c4d0 tx=0x7f641000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.311+0000 7f869f017640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f869806bf90 0x7f86980c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f869f017640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f869815cfb0 0x7f86980cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f869f017640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f869815f490 0x7f86980cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f6418ff9640 1 -- 172.21.15.59:0/4157529921 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410018070 con 0x7f641c0ce980 2024-03-06T07:26:29.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f6418ff9640 1 -- 172.21.15.59:0/4157529921 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6410007480 con 0x7f641c0ce980 2024-03-06T07:26:29.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f64225a1640 1 -- 172.21.15.59:0/4157529921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641c130ff0 con 0x7f641c0ce980 2024-03-06T07:26:29.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f869cdb3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f869806bf90 0x7f86980c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f6418ff9640 1 -- 172.21.15.59:0/4157529921 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64100047e0 con 0x7f641c0ce980 2024-03-06T07:26:29.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f64225a1640 1 -- 172.21.15.59:0/4157529921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641c1315b0 con 0x7f641c0ce980 2024-03-06T07:26:29.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f8697fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f869815cfb0 0x7f86980cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f8697fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f869815cfb0 0x7f86980cf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:46354/0 (socket says 172.21.15.59:46354) 2024-03-06T07:26:29.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f869cdb3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f869806bf90 0x7f86980c7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:42946/0 (socket says 172.21.15.59:42946) 2024-03-06T07:26:29.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f381c325640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38140cfba0 0x7f381409f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f381c325640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38140d1060 0x7f38140a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f8697fff640 1 -- 172.21.15.59:0/3118590722 learned_addr learned my addr 172.21.15.59:0/3118590722 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869815db10 con 0x7f869815f490 2024-03-06T07:26:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f381c325640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381415e150 0x7f381409dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86980cd3d0 con 0x7f869806bf90 2024-03-06T07:26:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86980cea50 con 0x7f869815cfb0 2024-03-06T07:26:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f38198c0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38140d1060 0x7f38140a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f38198c0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38140d1060 0x7f38140a1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:46366/0 (socket says 172.21.15.59:46366) 2024-03-06T07:26:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f38198c0640 1 -- 172.21.15.59:0/4076163708 learned_addr learned my addr 172.21.15.59:0/4076163708 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38140d1b30 con 0x7f38140cfba0 2024-03-06T07:26:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f381a8c2640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381415e150 0x7f381409dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f683f1bc640 1 -- 172.21.15.59:0/72592770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6838069ee0 con 0x7f68380b3fb0 2024-03-06T07:26:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38140d2f40 con 0x7f381415e150 2024-03-06T07:26:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381415e530 con 0x7f38140d1060 2024-03-06T07:26:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.312+0000 7f381a0c1640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38140cfba0 0x7f381409f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f869d5b4640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f869815f490 0x7f86980cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f8697fff640 1 -- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f869806bf90 msgr2=0x7f86980c7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f8697fff640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f869806bf90 0x7f86980c7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f8697fff640 1 -- 172.21.15.59:0/3118590722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f869815f490 msgr2=0x7f86980cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f8697fff640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f869815f490 0x7f86980cf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f381a8c2640 1 -- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38140d1060 msgr2=0x7f38140a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f381a8c2640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38140d1060 0x7f38140a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f381a8c2640 1 -- 172.21.15.59:0/4076163708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38140cfba0 msgr2=0x7f381409f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f381a8c2640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38140cfba0 0x7f381409f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f381a8c2640 1 -- 172.21.15.59:0/4076163708 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381409fdc0 con 0x7f381415e150 2024-03-06T07:26:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f8697fff640 1 -- 172.21.15.59:0/3118590722 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86980c7060 con 0x7f869815cfb0 2024-03-06T07:26:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f869d5b4640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f869815f490 0x7f86980cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:29.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f64225a1640 1 -- 172.21.15.59:0/4157529921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641c069ee0 con 0x7f641c0ce980 2024-03-06T07:26:29.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f869cdb3640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f869806bf90 0x7f86980c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:29.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f8697fff640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f869815cfb0 0x7f86980cf1c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f868400e870 tx=0x7f868400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f381a8c2640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381415e150 0x7f381409dba0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f380c00ef90 tx=0x7f380c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f86957fa640 1 -- 172.21.15.59:0/3118590722 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684017070 con 0x7f869815cfb0 2024-03-06T07:26:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f68357fa640 1 -- 172.21.15.59:0/72592770 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6824053070 con 0x7f68380b3fb0 2024-03-06T07:26:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f380affd640 1 -- 172.21.15.59:0/4076163708 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c018020 con 0x7f381415e150 2024-03-06T07:26:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f380affd640 1 -- 172.21.15.59:0/4076163708 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380c014480 con 0x7f381415e150 2024-03-06T07:26:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f380affd640 1 -- 172.21.15.59:0/4076163708 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c014870 con 0x7f381415e150 2024-03-06T07:26:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.314+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38140cb5d0 con 0x7f381415e150 2024-03-06T07:26:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f86957fa640 1 -- 172.21.15.59:0/3118590722 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868400c940 con 0x7f869815cfb0 2024-03-06T07:26:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f86957fa640 1 -- 172.21.15.59:0/3118590722 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868400cd30 con 0x7f869815cfb0 2024-03-06T07:26:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.313+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86980c5fc0 con 0x7f869815cfb0 2024-03-06T07:26:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.314+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86980c6580 con 0x7f869815cfb0 2024-03-06T07:26:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.314+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38140cbe10 con 0x7f381415e150 2024-03-06T07:26:29.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.314+0000 7f6418ff9640 1 -- 172.21.15.59:0/4157529921 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f641001b070 con 0x7f641c0ce980 2024-03-06T07:26:29.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.314+0000 7f68357fa640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f681c06b520 0x7f681c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.314+0000 7f68357fa640 1 -- 172.21.15.59:0/72592770 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68240b4c50 con 0x7f68380b3fb0 2024-03-06T07:26:29.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.314+0000 7f683cf58640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f681c06b520 0x7f681c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.314+0000 7f6418ff9640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f640806b4d0 0x7f640806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.315+0000 7f6418ff9640 1 -- 172.21.15.59:0/4157529921 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6410089770 con 0x7f641c0ce980 2024-03-06T07:26:29.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.315+0000 7f641bfff640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f640806b4d0 0x7f640806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.315+0000 7f683cf58640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f681c06b520 0x7f681c06d9e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f682c00da40 tx=0x7f682c00f040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.315+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38140cb940 con 0x7f381415e150 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.315+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8698068050 con 0x7f869815cfb0 2024-03-06T07:26:29.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.315+0000 7f641bfff640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f640806b4d0 0x7f640806d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f640c036000 tx=0x7f640c03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.315+0000 7f86957fa640 1 -- 172.21.15.59:0/3118590722 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f868401a070 con 0x7f869815cfb0 2024-03-06T07:26:29.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.316+0000 7f86957fa640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867c06f8e0 0x7f867c071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.316+0000 7f86957fa640 1 -- 172.21.15.59:0/3118590722 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f868408a3d0 con 0x7f869815cfb0 2024-03-06T07:26:29.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.316+0000 7f869cdb3640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867c06f8e0 0x7f867c071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.316+0000 7f380affd640 1 -- 172.21.15.59:0/4076163708 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f380c005d10 con 0x7f381415e150 2024-03-06T07:26:29.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.317+0000 7f380affd640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f37f806b520 0x7f37f806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:29.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.317+0000 7f869cdb3640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867c06f8e0 0x7f867c071da0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f86981499f0 tx=0x7f868c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.317+0000 7f380affd640 1 -- 172.21.15.59:0/4076163708 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f380c08aeb0 con 0x7f381415e150 2024-03-06T07:26:29.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.317+0000 7f381a0c1640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f37f806b520 0x7f37f806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:29.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.317+0000 7f381a0c1640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f37f806b520 0x7f37f806d9e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3810036000 tx=0x7f381003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.319+0000 7f6418ff9640 1 -- 172.21.15.59:0/4157529921 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6410058080 con 0x7f641c0ce980 2024-03-06T07:26:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.321+0000 7f86957fa640 1 -- 172.21.15.59:0/3118590722 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8684058c00 con 0x7f869815cfb0 2024-03-06T07:26:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.321+0000 7f380affd640 1 -- 172.21.15.59:0/4076163708 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f380c022030 con 0x7f381415e150 2024-03-06T07:26:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.324+0000 7f68357fa640 1 -- 172.21.15.59:0/72592770 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6824083460 con 0x7f68380b3fb0 2024-03-06T07:26:29.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.576+0000 7f64225a1640 1 -- 172.21.15.59:0/4157529921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f641c139b10 con 0x7f641c0ce980 2024-03-06T07:26:29.579 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:26:29.577+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:29.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.578+0000 7f6418ff9640 1 -- 172.21.15.59:0/4157529921 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 7422 v50) v1 ==== 95+0+0 (secure 0 0 0) 0x7f641005c120 con 0x7f641c0ce980 2024-03-06T07:26:29.580 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 7422 2024-03-06T07:26:29.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.581+0000 7f64225a1640 1 -- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f640806b4d0 msgr2=0x7f640806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.581+0000 7f64225a1640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f640806b4d0 0x7f640806d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f640c036000 tx=0x7f640c03c040 comp rx=0 tx=0).stop 2024-03-06T07:26:29.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.581+0000 7f64225a1640 1 -- 172.21.15.59:0/4157529921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f641c0ce980 msgr2=0x7f641c135a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.581+0000 7f64225a1640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f641c0ce980 0x7f641c135a70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f641000c4d0 tx=0x7f641000c990 comp rx=0 tx=0).stop 2024-03-06T07:26:29.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.582+0000 7f64225a1640 1 -- 172.21.15.59:0/4157529921 shutdown_connections 2024-03-06T07:26:29.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.582+0000 7f64225a1640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f640806b4d0 0x7f640806d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.582+0000 7f64225a1640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f641c162940 0x7f641c131ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.582+0000 7f64225a1640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f641c0ce980 0x7f641c135a70 secure :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f641000c4d0 tx=0x7f641000c990 comp rx=0 tx=0).stop 2024-03-06T07:26:29.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.582+0000 7f64225a1640 1 --2- 172.21.15.59:0/4157529921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f641c0ce5a0 0x7f641c132940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.582+0000 7f64225a1640 1 -- 172.21.15.59:0/4157529921 >> 172.21.15.59:0/4157529921 conn(0x7f641c06b160 msgr2=0x7f641c15b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.581+0000 7f683f1bc640 1 -- 172.21.15.59:0/72592770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f68380b3200 con 0x7f68380b3fb0 2024-03-06T07:26:29.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.582+0000 7f64225a1640 1 -- 172.21.15.59:0/4157529921 shutdown_connections 2024-03-06T07:26:29.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.582+0000 7f64225a1640 1 -- 172.21.15.59:0/4157529921 wait complete. 2024-03-06T07:26:29.586 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:26:29.585+0000 7fed61d70640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:29.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.586+0000 7f68357fa640 1 -- 172.21.15.59:0/72592770 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7410 v52) v1 ==== 95+0+0 (secure 0 0 0) 0x7f6824087500 con 0x7f68380b3fb0 2024-03-06T07:26:29.588 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 7410 2024-03-06T07:26:29.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.589+0000 7f683f1bc640 1 -- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f681c06b520 msgr2=0x7f681c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.589+0000 7f683f1bc640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f681c06b520 0x7f681c06d9e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f682c00da40 tx=0x7f682c00f040 comp rx=0 tx=0).stop 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.589+0000 7f683f1bc640 1 -- 172.21.15.59:0/72592770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68380b3fb0 msgr2=0x7f68381f3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.589+0000 7f683f1bc640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68380b3fb0 0x7f68381f3450 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f6824033000 tx=0x7f6824004730 comp rx=0 tx=0).stop 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.590+0000 7f683f1bc640 1 -- 172.21.15.59:0/72592770 shutdown_connections 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.590+0000 7f683f1bc640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f681c06b520 0x7f681c06d9e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.590+0000 7f683f1bc640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68380ccd70 0x7f68381f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.590+0000 7f683f1bc640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f68380b3fb0 0x7f68381f3450 secure :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f6824033000 tx=0x7f6824004730 comp rx=0 tx=0).stop 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.590+0000 7f683f1bc640 1 --2- 172.21.15.59:0/72592770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68380b3690 0x7f6838141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.590+0000 7f683f1bc640 1 -- 172.21.15.59:0/72592770 >> 172.21.15.59:0/72592770 conn(0x7f683806b160 msgr2=0x7f683815d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.591+0000 7f683f1bc640 1 -- 172.21.15.59:0/72592770 shutdown_connections 2024-03-06T07:26:29.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.592+0000 7f683f1bc640 1 -- 172.21.15.59:0/72592770 wait complete. 2024-03-06T07:26:29.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.595+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f3814069ee0 con 0x7f381415e150 2024-03-06T07:26:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.597+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f8698067200 con 0x7f869815cfb0 2024-03-06T07:26:29.599 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:26:29.597+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:29.600 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:29.601 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:29.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.601+0000 7f380affd640 1 -- 172.21.15.59:0/4076163708 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7407 v54) v1 ==== 95+0+0 (secure 0 0 0) 0x7f38140cb5d0 con 0x7f381415e150 2024-03-06T07:26:29.602 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 7407 2024-03-06T07:26:29.603 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:29.603 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:29.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.604+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f37f806b520 msgr2=0x7f37f806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.604+0000 7f381c325640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f37f806b520 0x7f37f806d9e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3810036000 tx=0x7f381003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:29.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.604+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381415e150 msgr2=0x7f381409dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:29.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.604+0000 7f381c325640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381415e150 0x7f381409dba0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f380c00ef90 tx=0x7f380c00c490 comp rx=0 tx=0).stop 2024-03-06T07:26:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.605+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 shutdown_connections 2024-03-06T07:26:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.605+0000 7f381c325640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f37f806b520 0x7f37f806d9e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.605+0000 7f381c325640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f381415e150 0x7f381409dba0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.605+0000 7f381c325640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38140d1060 0x7f38140a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.605+0000 7f381c325640 1 --2- 172.21.15.59:0/4076163708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38140cfba0 0x7f381409f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.605+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 >> 172.21.15.59:0/4076163708 conn(0x7f381409d230 msgr2=0x7f38140b8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.605+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 shutdown_connections 2024-03-06T07:26:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:29.605+0000 7f381c325640 1 -- 172.21.15.59:0/4076163708 wait complete. 2024-03-06T07:26:29.608 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:29.608 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:29.620 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:29.620 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:29.661 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:29.662 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:29.663 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:29.663 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:29.677 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:26:29.736 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:26:29.737 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:26:30.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.062+0000 7f86957fa640 1 -- 172.21.15.59:0/3118590722 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6173 v55) v1 ==== 95+0+0 (secure 0 0 0) 0x7f868405cca0 con 0x7f869815cfb0 2024-03-06T07:26:30.065 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 6173 2024-03-06T07:26:30.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867c06f8e0 msgr2=0x7f867c071da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867c06f8e0 0x7f867c071da0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f86981499f0 tx=0x7f868c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:30.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f869815cfb0 msgr2=0x7f86980cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f869815cfb0 0x7f86980cf1c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f868400e870 tx=0x7f868400ed30 comp rx=0 tx=0).stop 2024-03-06T07:26:30.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 shutdown_connections 2024-03-06T07:26:30.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867c06f8e0 0x7f867c071da0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f869815f490 0x7f86980cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f869815cfb0 0x7f86980cf1c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 --2- 172.21.15.59:0/3118590722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f869806bf90 0x7f86980c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 >> 172.21.15.59:0/3118590722 conn(0x7f8698071510 msgr2=0x7f869815c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 shutdown_connections 2024-03-06T07:26:30.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.065+0000 7f869f017640 1 -- 172.21.15.59:0/3118590722 wait complete. 2024-03-06T07:26:30.081 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:30.081 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:30.083 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:30.084 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:30.084 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:30.157 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:26:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.222+0000 7f91fdb02640 1 Processor -- start 2024-03-06T07:26:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.222+0000 7f91fdb02640 1 -- start start 2024-03-06T07:26:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.223+0000 7f91fdb02640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f91f80b65e0 0x7f91f81624a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.223+0000 7f91fdb02640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f80d12d0 0x7f91f80d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.223+0000 7f91fdb02640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f80cf2f0 con 0x7f91f815e310 2024-03-06T07:26:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.223+0000 7f91fdb02640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f80b5140 con 0x7f91f80b65e0 2024-03-06T07:26:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.223+0000 7f91fdb02640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f80cf8b0 con 0x7f91f80d12d0 2024-03-06T07:26:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.223+0000 7f91f77fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f91f815e310 0x7f91f8161d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49980/0 (socket says 172.21.15.59:49980) 2024-03-06T07:26:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.223+0000 7f91f77fe640 1 -- 172.21.15.59:0/1442912169 learned_addr learned my addr 172.21.15.59:0/1442912169 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.223+0000 7f91f7fff640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f80d12d0 0x7f91f80d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.223+0000 7f91f6ffd640 1 --2- 172.21.15.59:0/1442912169 >> v2:172.21.15.164:3300/0 conn(0x7f91f80b65e0 0x7f91f81624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.223+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985891721 0 0) 0x7f91f80cf2f0 con 0x7f91f815e310 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e0003520 con 0x7f91f815e310 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 571161799 0 0) 0x7f91e0003520 con 0x7f91f815e310 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f80d12d0 msgr2=0x7f91f80d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f80d12d0 0x7f91f80d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 >> v2:172.21.15.164:3300/0 conn(0x7f91f80b65e0 msgr2=0x7f91f81624a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 --2- 172.21.15.59:0/1442912169 >> v2:172.21.15.164:3300/0 conn(0x7f91f80b65e0 0x7f91f81624a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f80cf5d0 con 0x7f91f815e310 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f91e8001670 con 0x7f91f815e310 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 >> v1:172.21.15.59:6789/0 conn(0x7f91f815e310 legacy=0x7f91f8161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e00039b0 0x7f91e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f80d12d0 0x7f91e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91e0004d40 0x7f91e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f80b5140 con 0x7f91e00039b0 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f80cf8b0 con 0x7f91e0004d40 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e000b9c0 con 0x7f91f80d12d0 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f77fe640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e00039b0 0x7f91e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f6ffd640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f80d12d0 0x7f91e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f7fff640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91e0004d40 0x7f91e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f6ffd640 1 -- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91e0004d40 msgr2=0x7f91e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f6ffd640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91e0004d40 0x7f91e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f6ffd640 1 -- 172.21.15.59:0/1442912169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e00039b0 msgr2=0x7f91e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f6ffd640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e00039b0 0x7f91e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f6ffd640 1 -- 172.21.15.59:0/1442912169 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e0003520 con 0x7f91f80d12d0 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.224+0000 7f91f77fe640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91e00039b0 0x7f91e0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.225+0000 7f91f6ffd640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f80d12d0 0x7f91e0004800 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009a90 tx=0x7f91e0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=828092b0043cc848 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.225+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec039070 con 0x7f91f80d12d0 2024-03-06T07:26:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.225+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ec02c6b0 con 0x7f91f80d12d0 2024-03-06T07:26:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.225+0000 7f91f67fc640 1 -- 172.21.15.59:0/1442912169 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec009690 con 0x7f91f80d12d0 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.226+0000 7f91fdb02640 1 -- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f80d12d0 msgr2=0x7f91e0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.226+0000 7f91fdb02640 1 --2- 172.21.15.59:0/1442912169 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f80d12d0 0x7f91e0004800 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009a90 tx=0x7f91e0005830 comp rx=0 tx=0).stop 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.226+0000 7f91f7fff640 1 -- 172.21.15.59:0/1442912169 reap_dead start 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.226+0000 7f91fdb02640 1 -- 172.21.15.59:0/1442912169 shutdown_connections 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.226+0000 7f91fdb02640 1 -- 172.21.15.59:0/1442912169 >> 172.21.15.59:0/1442912169 conn(0x7f91f809aa40 msgr2=0x7f91f809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.226+0000 7f91fdb02640 1 -- 172.21.15.59:0/1442912169 shutdown_connections 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.226+0000 7f91fdb02640 1 -- 172.21.15.59:0/1442912169 wait complete. 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.227+0000 7f91fdb02640 1 Processor -- start 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.227+0000 7f91fdb02640 1 -- start start 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91fdb02640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91f80d12d0 0x7f91f815cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91fdb02640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91f8157d50 0x7f91f815d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91fdb02640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f81f3570 0x7f91f81f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91fdb02640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f809b7c0 con 0x7f91f80d12d0 2024-03-06T07:26:30.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91fdb02640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f809b640 con 0x7f91f8157d50 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91fdb02640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f81629e0 con 0x7f91f81f3570 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91f77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91f80d12d0 0x7f91f815cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91f77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91f80d12d0 0x7f91f815cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60720/0 (socket says 172.21.15.59:60720) 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91f77fe640 1 -- 172.21.15.59:0/1670537689 learned_addr learned my addr 172.21.15.59:0/1670537689 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91f7fff640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f81f3570 0x7f91f81f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91f6ffd640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91f8157d50 0x7f91f815d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91f77fe640 1 -- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f81f3570 msgr2=0x7f91f81f3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91f77fe640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f81f3570 0x7f91f81f3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91f77fe640 1 -- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91f8157d50 msgr2=0x7f91f815d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91f77fe640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91f8157d50 0x7f91f815d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91f77fe640 1 -- 172.21.15.59:0/1670537689 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f815b0b0 con 0x7f91f80d12d0 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91f77fe640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91f80d12d0 0x7f91f815cd30 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f91e8007310 tx=0x7f91e80077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.228+0000 7f91dffff640 1 -- 172.21.15.59:0/1670537689 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e800f960 con 0x7f91f80d12d0 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.229+0000 7f91dffff640 1 -- 172.21.15.59:0/1670537689 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e8018070 con 0x7f91f80d12d0 2024-03-06T07:26:30.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.229+0000 7f91fdb02640 1 -- 172.21.15.59:0/1670537689 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f815b390 con 0x7f91f80d12d0 2024-03-06T07:26:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.229+0000 7f91dffff640 1 -- 172.21.15.59:0/1670537689 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e800fd00 con 0x7f91f80d12d0 2024-03-06T07:26:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.229+0000 7f91fdb02640 1 -- 172.21.15.59:0/1670537689 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f815b950 con 0x7f91f80d12d0 2024-03-06T07:26:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.230+0000 7f91fdb02640 1 -- 172.21.15.59:0/1670537689 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f815bc80 con 0x7f91f80d12d0 2024-03-06T07:26:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.230+0000 7f91dffff640 1 -- 172.21.15.59:0/1670537689 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91e801d460 con 0x7f91f80d12d0 2024-03-06T07:26:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.230+0000 7f91dffff640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f91e406b500 0x7f91e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.231+0000 7f91dffff640 1 -- 172.21.15.59:0/1670537689 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f91e808a350 con 0x7f91f80d12d0 2024-03-06T07:26:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.231+0000 7f91f6ffd640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f91e406b500 0x7f91e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:30.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.231+0000 7f91f6ffd640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f91e406b500 0x7f91e406d9c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f91ec036000 tx=0x7f91ec030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:30.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.235+0000 7f91dffff640 1 -- 172.21.15.59:0/1670537689 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91e8058a00 con 0x7f91f80d12d0 2024-03-06T07:26:30.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.455+0000 7f91fdb02640 1 -- 172.21.15.59:0/1670537689 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91f8069ee0 con 0x7f91f80d12d0 2024-03-06T07:26:30.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.456+0000 7f91dffff640 1 -- 172.21.15.59:0/1670537689 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4185 (secure 0 0 0) 0x7f91e805caa0 con 0x7f91f80d12d0 2024-03-06T07:26:30.458 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:30.458 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":55,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:30.060545+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:replay","state_seq":1,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:30.458 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 55 2024-03-06T07:26:30.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 -- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f91e406b500 msgr2=0x7f91e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f91e406b500 0x7f91e406d9c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f91ec036000 tx=0x7f91ec030000 comp rx=0 tx=0).stop 2024-03-06T07:26:30.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 -- 172.21.15.59:0/1670537689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91f80d12d0 msgr2=0x7f91f815cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:30.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91f80d12d0 0x7f91f815cd30 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f91e8007310 tx=0x7f91e80077d0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 -- 172.21.15.59:0/1670537689 shutdown_connections 2024-03-06T07:26:30.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f91e406b500 0x7f91e406d9c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f91f81f3570 0x7f91f81f3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91f8157d50 0x7f91f815d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 --2- 172.21.15.59:0/1670537689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91f80d12d0 0x7f91f815cd30 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:30.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 -- 172.21.15.59:0/1670537689 >> 172.21.15.59:0/1670537689 conn(0x7f91f809aa40 msgr2=0x7f91f806efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:30.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 -- 172.21.15.59:0/1670537689 shutdown_connections 2024-03-06T07:26:30.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:30.458+0000 7f91fdb02640 1 -- 172.21.15.59:0/1670537689 wait complete. 2024-03-06T07:26:30.873 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:17.219713+0000', 'modified': '2024-03-06T07:26:30.060545+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 95, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7635}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7635': {'gid': 7635, 'name': 'a', 'rank': 0, 'incarnation': 55, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.59:6835/4004944688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6833', 'nonce': 4004944688}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 4004944688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:30.873 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7635:up:replay 2024-03-06T07:26:31.875 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:31.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.986+0000 7f49b83e4640 1 Processor -- start 2024-03-06T07:26:31.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.986+0000 7f49b83e4640 1 -- start start 2024-03-06T07:26:31.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.987+0000 7f49b83e4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f49b006c750 0x7f49b006cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:31.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.987+0000 7f49b83e4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b006d070 con 0x7f49b00ce560 2024-03-06T07:26:31.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.987+0000 7f49b83e4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b00cc580 con 0x7f49b00b12a0 2024-03-06T07:26:31.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.987+0000 7f49b83e4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49b00cc330 con 0x7f49b006c750 2024-03-06T07:26:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.987+0000 7f49b6180640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f49b006c750 0x7f49b006cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.987+0000 7f49b6981640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f49b00ce560 0x7f49b00ce970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49988/0 (socket says 172.21.15.59:49988) 2024-03-06T07:26:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.987+0000 7f49b6180640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f49b006c750 0x7f49b006cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60730/0 (socket says 172.21.15.59:60730) 2024-03-06T07:26:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.987+0000 7f49b6981640 1 -- 172.21.15.59:0/1724128524 learned_addr learned my addr 172.21.15.59:0/1724128524 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.988+0000 7f49b6180640 1 -- 172.21.15.59:0/1724128524 >> v1:172.21.15.164:6789/0 conn(0x7f49b00b12a0 legacy=0x7f49b00d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.988+0000 7f49b6180640 1 -- 172.21.15.59:0/1724128524 >> v1:172.21.15.59:6789/0 conn(0x7f49b00ce560 legacy=0x7f49b00ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.988+0000 7f49b6180640 1 -- 172.21.15.59:0/1724128524 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b00cc860 con 0x7f49b006c750 2024-03-06T07:26:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.988+0000 7f49b517e640 1 -- 172.21.15.59:0/1724128524 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452691963 0 0) 0x7f49b006d070 con 0x7f49b00ce560 2024-03-06T07:26:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.988+0000 7f49b6180640 1 --2- 172.21.15.59:0/1724128524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b006c750 0x7f49b006cb30 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f49a80097a0 tx=0x7f49a802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421457620299b2b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.989+0000 7f49b517e640 1 -- 172.21.15.59:0/1724128524 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a80315a0 con 0x7f49b006c750 2024-03-06T07:26:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.989+0000 7f49b517e640 1 -- 172.21.15.59:0/1724128524 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a8037070 con 0x7f49b006c750 2024-03-06T07:26:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.989+0000 7f49b517e640 1 -- 172.21.15.59:0/1724128524 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a8031960 con 0x7f49b006c750 2024-03-06T07:26:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.989+0000 7f49b83e4640 1 -- 172.21.15.59:0/1724128524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b006c750 msgr2=0x7f49b006cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.989+0000 7f49b83e4640 1 --2- 172.21.15.59:0/1724128524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b006c750 0x7f49b006cb30 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f49a80097a0 tx=0x7f49a802b840 comp rx=0 tx=0).stop 2024-03-06T07:26:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.989+0000 7f49b83e4640 1 -- 172.21.15.59:0/1724128524 shutdown_connections 2024-03-06T07:26:31.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.989+0000 7f49b83e4640 1 --2- 172.21.15.59:0/1724128524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b006c750 0x7f49b006cb30 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.989+0000 7f49b83e4640 1 -- 172.21.15.59:0/1724128524 >> 172.21.15.59:0/1724128524 conn(0x7f49b00b3850 msgr2=0x7f49b00b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:31.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.989+0000 7f49b83e4640 1 -- 172.21.15.59:0/1724128524 shutdown_connections 2024-03-06T07:26:31.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.989+0000 7f49b83e4640 1 -- 172.21.15.59:0/1724128524 wait complete. 2024-03-06T07:26:31.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.990+0000 7f49b83e4640 1 Processor -- start 2024-03-06T07:26:31.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.990+0000 7f49b83e4640 1 -- start start 2024-03-06T07:26:31.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.991+0000 7f49b83e4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49b006c750 0x7f49b01f7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:31.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.991+0000 7f49b83e4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49b00b12a0 0x7f49b014cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:31.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.991+0000 7f49b83e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b00ce560 0x7f49b00c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:31.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.991+0000 7f49b83e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00d1140 con 0x7f49b00ce560 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.991+0000 7f49b83e4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00cce20 con 0x7f49b006c750 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.991+0000 7f49b83e4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b00b44e0 con 0x7f49b00b12a0 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.991+0000 7f49b6981640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b00ce560 0x7f49b00c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.991+0000 7f49b6981640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b00ce560 0x7f49b00c8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60746/0 (socket says 172.21.15.59:60746) 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.991+0000 7f49b6981640 1 -- 172.21.15.59:0/3456302835 learned_addr learned my addr 172.21.15.59:0/3456302835 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49b6981640 1 -- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49b00b12a0 msgr2=0x7f49b014cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49b6981640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49b00b12a0 0x7f49b014cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49b6981640 1 -- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49b006c750 msgr2=0x7f49b01f7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49b6180640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49b006c750 0x7f49b01f7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49b6981640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49b006c750 0x7f49b01f7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49b6981640 1 -- 172.21.15.59:0/3456302835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b00cada0 con 0x7f49b00ce560 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49b6981640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b00ce560 0x7f49b00c8db0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f49a000c700 tx=0x7f49a000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49aeffd640 1 -- 172.21.15.59:0/3456302835 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a0018070 con 0x7f49b00ce560 2024-03-06T07:26:31.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49aeffd640 1 -- 172.21.15.59:0/3456302835 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a000ce10 con 0x7f49b00ce560 2024-03-06T07:26:31.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49b83e4640 1 -- 172.21.15.59:0/3456302835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b01f8010 con 0x7f49b00ce560 2024-03-06T07:26:31.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49aeffd640 1 -- 172.21.15.59:0/3456302835 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a000a860 con 0x7f49b00ce560 2024-03-06T07:26:31.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.992+0000 7f49b83e4640 1 -- 172.21.15.59:0/3456302835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b00cb600 con 0x7f49b00ce560 2024-03-06T07:26:31.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.993+0000 7f49b83e4640 1 -- 172.21.15.59:0/3456302835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b0069ee0 con 0x7f49b00ce560 2024-03-06T07:26:31.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.993+0000 7f49aeffd640 1 -- 172.21.15.59:0/3456302835 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49a001b070 con 0x7f49b00ce560 2024-03-06T07:26:31.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.994+0000 7f49aeffd640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f499c06b4d0 0x7f499c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:31.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.994+0000 7f49aeffd640 1 -- 172.21.15.59:0/3456302835 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f49a0089900 con 0x7f49b00ce560 2024-03-06T07:26:31.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.994+0000 7f49b6180640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f499c06b4d0 0x7f499c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:31.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.995+0000 7f49b6180640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f499c06b4d0 0x7f499c06d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f49a8036000 tx=0x7f49a8008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:32.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:31.998+0000 7f49aeffd640 1 -- 172.21.15.59:0/3456302835 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49a0058680 con 0x7f49b00ce560 2024-03-06T07:26:32.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.217+0000 7f49b83e4640 1 -- 172.21.15.59:0/3456302835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49b00d0bf0 con 0x7f49b00ce560 2024-03-06T07:26:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.218+0000 7f49aeffd640 1 -- 172.21.15.59:0/3456302835 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f49a0057f80 con 0x7f49b00ce560 2024-03-06T07:26:32.220 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:32.220 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:31.065053+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:32.220 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 57 2024-03-06T07:26:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 -- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f499c06b4d0 msgr2=0x7f499c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f499c06b4d0 0x7f499c06d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f49a8036000 tx=0x7f49a8008030 comp rx=0 tx=0).stop 2024-03-06T07:26:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 -- 172.21.15.59:0/3456302835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b00ce560 msgr2=0x7f49b00c8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b00ce560 0x7f49b00c8db0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f49a000c700 tx=0x7f49a000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 -- 172.21.15.59:0/3456302835 shutdown_connections 2024-03-06T07:26:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f499c06b4d0 0x7f499c06d990 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49b00ce560 0x7f49b00c8db0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49b00b12a0 0x7f49b014cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 --2- 172.21.15.59:0/3456302835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49b006c750 0x7f49b01f7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 -- 172.21.15.59:0/3456302835 >> 172.21.15.59:0/3456302835 conn(0x7f49b00b3850 msgr2=0x7f49b00d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 -- 172.21.15.59:0/3456302835 shutdown_connections 2024-03-06T07:26:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:32.220+0000 7f49b83e4640 1 -- 172.21.15.59:0/3456302835 wait complete. 2024-03-06T07:26:32.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:17.219713+0000', 'modified': '2024-03-06T07:26:31.065053+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 95, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7635}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7635': {'gid': 7635, 'name': 'a', 'rank': 0, 'incarnation': 55, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.59:6835/4004944688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6833', 'nonce': 4004944688}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 4004944688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:32.634 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7635:up:rejoin 2024-03-06T07:26:33.636 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:33.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.747+0000 7efdc7481640 1 Processor -- start 2024-03-06T07:26:33.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.747+0000 7efdc7481640 1 -- start start 2024-03-06T07:26:33.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc7481640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdc0161760 0x7efdc00b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc7481640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efdc00d2240 0x7efdc00c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc7481640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc0161b40 con 0x7efdc00b5240 2024-03-06T07:26:33.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc7481640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc00d27e0 con 0x7efdc00d2240 2024-03-06T07:26:33.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc7481640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc00cf580 con 0x7efdc0161760 2024-03-06T07:26:33.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc5a1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efdc00d2240 0x7efdc00c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc4a1c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7efdc00b5240 0x7efdc00d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35094/0 (socket says 172.21.15.59:35094) 2024-03-06T07:26:33.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc4a1c640 1 -- 172.21.15.59:0/3283120075 learned_addr learned my addr 172.21.15.59:0/3283120075 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:33.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc521d640 1 --2- 172.21.15.59:0/3283120075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdc0161760 0x7efdc00b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc5a1e640 1 -- 172.21.15.59:0/3283120075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdc0161760 msgr2=0x7efdc00b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:33.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc5a1e640 1 --2- 172.21.15.59:0/3283120075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdc0161760 0x7efdc00b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc5a1e640 1 -- 172.21.15.59:0/3283120075 >> v1:172.21.15.59:6789/0 conn(0x7efdc00b5240 legacy=0x7efdc00d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdb7fff640 1 -- 172.21.15.59:0/3283120075 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547983863 0 0) 0x7efdc0161b40 con 0x7efdc00b5240 2024-03-06T07:26:33.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.748+0000 7efdc5a1e640 1 -- 172.21.15.59:0/3283120075 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc00cf860 con 0x7efdc00d2240 2024-03-06T07:26:33.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.749+0000 7efdc5a1e640 1 --2- 172.21.15.59:0/3283120075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdc00d2240 0x7efdc00c7ab0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009d90 tx=0x7efdb802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b247f3165a46c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.749+0000 7efdb7fff640 1 -- 172.21.15.59:0/3283120075 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb8039070 con 0x7efdc00d2240 2024-03-06T07:26:33.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.749+0000 7efdb7fff640 1 -- 172.21.15.59:0/3283120075 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb802cc50 con 0x7efdc00d2240 2024-03-06T07:26:33.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.749+0000 7efdb7fff640 1 -- 172.21.15.59:0/3283120075 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb8034440 con 0x7efdc00d2240 2024-03-06T07:26:33.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.750+0000 7efdc7481640 1 -- 172.21.15.59:0/3283120075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdc00d2240 msgr2=0x7efdc00c7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.750+0000 7efdc7481640 1 --2- 172.21.15.59:0/3283120075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdc00d2240 0x7efdc00c7ab0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009d90 tx=0x7efdb802bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.750+0000 7efdc7481640 1 -- 172.21.15.59:0/3283120075 shutdown_connections 2024-03-06T07:26:33.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.750+0000 7efdc7481640 1 --2- 172.21.15.59:0/3283120075 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdc00d2240 0x7efdc00c7ab0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.750+0000 7efdc7481640 1 --2- 172.21.15.59:0/3283120075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdc0161760 0x7efdc00b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.750+0000 7efdc7481640 1 -- 172.21.15.59:0/3283120075 >> 172.21.15.59:0/3283120075 conn(0x7efdc00d3f60 msgr2=0x7efdc00d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:33.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.750+0000 7efdc7481640 1 -- 172.21.15.59:0/3283120075 shutdown_connections 2024-03-06T07:26:33.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.750+0000 7efdc7481640 1 -- 172.21.15.59:0/3283120075 wait complete. 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.751+0000 7efdc7481640 1 Processor -- start 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.751+0000 7efdc7481640 1 -- start start 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc7481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdc00b5240 0x7efdc01f4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc7481640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdc00d2240 0x7efdc0158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc7481640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdc0161760 0x7efdc00b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc7481640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc00d2c40 con 0x7efdc00b5240 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc7481640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc00d2ac0 con 0x7efdc0161760 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc7481640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc00d4a80 con 0x7efdc00d2240 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc521d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdc00b5240 0x7efdc01f4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc521d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdc00b5240 0x7efdc01f4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44076/0 (socket says 172.21.15.59:44076) 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc521d640 1 -- 172.21.15.59:0/2410574975 learned_addr learned my addr 172.21.15.59:0/2410574975 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:33.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc4a1c640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdc00d2240 0x7efdc0158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc5a1e640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdc0161760 0x7efdc00b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc521d640 1 -- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdc00d2240 msgr2=0x7efdc0158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.752+0000 7efdc521d640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdc00d2240 0x7efdc0158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.753+0000 7efdc521d640 1 -- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdc0161760 msgr2=0x7efdc00b8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.753+0000 7efdc521d640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdc0161760 0x7efdc00b8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.753+0000 7efdc521d640 1 -- 172.21.15.59:0/2410574975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc01f4c90 con 0x7efdc00b5240 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.753+0000 7efdc5a1e640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdc0161760 0x7efdc00b8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.753+0000 7efdc521d640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdc00b5240 0x7efdc01f4750 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00d830 tx=0x7efdbc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.753+0000 7efdb5ffb640 1 -- 172.21.15.59:0/2410574975 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc00b6c0 con 0x7efdc00b5240 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.753+0000 7efdb5ffb640 1 -- 172.21.15.59:0/2410574975 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdbc019070 con 0x7efdc00b5240 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.753+0000 7efdc7481640 1 -- 172.21.15.59:0/2410574975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc00b6f10 con 0x7efdc00b5240 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.753+0000 7efdb5ffb640 1 -- 172.21.15.59:0/2410574975 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc00ba80 con 0x7efdc00b5240 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.753+0000 7efdc7481640 1 -- 172.21.15.59:0/2410574975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc00b77b0 con 0x7efdc00b5240 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.754+0000 7efdc7481640 1 -- 172.21.15.59:0/2410574975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc0069ee0 con 0x7efdc00b5240 2024-03-06T07:26:33.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.754+0000 7efdb5ffb640 1 -- 172.21.15.59:0/2410574975 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdbc01e460 con 0x7efdc00b5240 2024-03-06T07:26:33.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.755+0000 7efdb5ffb640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efda406b4d0 0x7efda406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:33.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.755+0000 7efdb5ffb640 1 -- 172.21.15.59:0/2410574975 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7efdbc08aa50 con 0x7efdc00b5240 2024-03-06T07:26:33.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.755+0000 7efdc4a1c640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efda406b4d0 0x7efda406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:33.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.755+0000 7efdc4a1c640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efda406b4d0 0x7efda406d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7efdc006d850 tx=0x7efdb0007c00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:33.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.759+0000 7efdb5ffb640 1 -- 172.21.15.59:0/2410574975 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdc01f4c90 con 0x7efdc00b5240 2024-03-06T07:26:33.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.980+0000 7efdc7481640 1 -- 172.21.15.59:0/2410574975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdc00d36d0 con 0x7efdc00b5240 2024-03-06T07:26:33.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.981+0000 7efdb5ffb640 1 -- 172.21.15.59:0/2410574975 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4982 (secure 0 0 0) 0x7efdbc059730 con 0x7efdc00b5240 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":58,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:32.070394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:clientreplay","state_seq":4,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:33.983 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 58 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 -- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efda406b4d0 msgr2=0x7efda406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efda406b4d0 0x7efda406d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7efdc006d850 tx=0x7efdb0007c00 comp rx=0 tx=0).stop 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 -- 172.21.15.59:0/2410574975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdc00b5240 msgr2=0x7efdc01f4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdc00b5240 0x7efdc01f4750 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00d830 tx=0x7efdbc00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 -- 172.21.15.59:0/2410574975 shutdown_connections 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efda406b4d0 0x7efda406d990 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdc0161760 0x7efdc00b8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdc00d2240 0x7efdc0158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 --2- 172.21.15.59:0/2410574975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdc00b5240 0x7efdc01f4750 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 -- 172.21.15.59:0/2410574975 >> 172.21.15.59:0/2410574975 conn(0x7efdc00d3f60 msgr2=0x7efdc015f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 -- 172.21.15.59:0/2410574975 shutdown_connections 2024-03-06T07:26:33.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:33.983+0000 7efdc7481640 1 -- 172.21.15.59:0/2410574975 wait complete. 2024-03-06T07:26:34.398 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 58, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:17.219713+0000', 'modified': '2024-03-06T07:26:32.070394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 95, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7635}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7635': {'gid': 7635, 'name': 'a', 'rank': 0, 'incarnation': 55, 'state': 'up:clientreplay', 'state_seq': 4, 'addr': '172.21.15.59:6835/4004944688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6833', 'nonce': 4004944688}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 4004944688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:34.398 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7635:up:clientreplay 2024-03-06T07:26:35.068 INFO:tasks.ceph.mds.a.smithi059.stderr:2024-03-06T07:26:35.066+0000 7f7d47301640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:35.400 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:35.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.499+0000 7f3d67f93640 1 Processor -- start 2024-03-06T07:26:35.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.499+0000 7f3d67f93640 1 -- start start 2024-03-06T07:26:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d67f93640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d60158570 0x7f3d6006f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d67f93640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3d600b0170 0x7f3d6015e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d67f93640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6006b190 con 0x7f3d600b3cc0 2024-03-06T07:26:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d67f93640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d6006b470 con 0x7f3d600b0170 2024-03-06T07:26:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d67f93640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6006b750 con 0x7f3d60158570 2024-03-06T07:26:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d66530640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3d600b0170 0x7f3d6015e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d66530640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3d600b0170 0x7f3d6015e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44084/0 (socket says 172.21.15.59:44084) 2024-03-06T07:26:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d66530640 1 -- 172.21.15.59:0/313299710 learned_addr learned my addr 172.21.15.59:0/313299710 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d66530640 1 -- 172.21.15.59:0/313299710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d60158570 msgr2=0x7f3d6006f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d66530640 1 --2- 172.21.15.59:0/313299710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d60158570 0x7f3d6006f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d66530640 1 -- 172.21.15.59:0/313299710 >> v1:172.21.15.164:6789/0 conn(0x7f3d600b3cc0 legacy=0x7f3d600afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.500+0000 7f3d66530640 1 -- 172.21.15.59:0/313299710 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6006ba30 con 0x7f3d600b0170 2024-03-06T07:26:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.501+0000 7f3d66530640 1 --2- 172.21.15.59:0/313299710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d600b0170 0x7f3d6015e620 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009870 tx=0x7f3d5c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e72d10cae4ce30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.502+0000 7f3d64d2d640 1 -- 172.21.15.59:0/313299710 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c035020 con 0x7f3d600b0170 2024-03-06T07:26:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.502+0000 7f3d64d2d640 1 -- 172.21.15.59:0/313299710 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5c031640 con 0x7f3d600b0170 2024-03-06T07:26:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.502+0000 7f3d64d2d640 1 -- 172.21.15.59:0/313299710 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c02d040 con 0x7f3d600b0170 2024-03-06T07:26:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.502+0000 7f3d67f93640 1 -- 172.21.15.59:0/313299710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d600b0170 msgr2=0x7f3d6015e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.502+0000 7f3d67f93640 1 --2- 172.21.15.59:0/313299710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d600b0170 0x7f3d6015e620 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009870 tx=0x7f3d5c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.502+0000 7f3d67f93640 1 -- 172.21.15.59:0/313299710 shutdown_connections 2024-03-06T07:26:35.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.502+0000 7f3d67f93640 1 --2- 172.21.15.59:0/313299710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d600b0170 0x7f3d6015e620 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.502+0000 7f3d67f93640 1 --2- 172.21.15.59:0/313299710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d60158570 0x7f3d6006f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.502+0000 7f3d67f93640 1 -- 172.21.15.59:0/313299710 >> 172.21.15.59:0/313299710 conn(0x7f3d600b29c0 msgr2=0x7f3d600b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.502+0000 7f3d67f93640 1 -- 172.21.15.59:0/313299710 shutdown_connections 2024-03-06T07:26:35.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.502+0000 7f3d67f93640 1 -- 172.21.15.59:0/313299710 wait complete. 2024-03-06T07:26:35.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.504+0000 7f3d67f93640 1 Processor -- start 2024-03-06T07:26:35.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.504+0000 7f3d67f93640 1 -- start start 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d67f93640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d600b0170 0x7f3d601541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d67f93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d600b3cc0 0x7f3d60160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d67f93640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d60158570 0x7f3d60160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d67f93640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6015eb60 con 0x7f3d600b3cc0 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d67f93640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d60158950 con 0x7f3d60158570 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d67f93640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6015ece0 con 0x7f3d600b0170 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d6552e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d600b3cc0 0x7f3d60160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d6552e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d600b3cc0 0x7f3d60160140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44100/0 (socket says 172.21.15.59:44100) 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d6552e640 1 -- 172.21.15.59:0/13962789 learned_addr learned my addr 172.21.15.59:0/13962789 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d65d2f640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d600b0170 0x7f3d601541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d66530640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d60158570 0x7f3d60160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d6552e640 1 -- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d600b0170 msgr2=0x7f3d601541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d6552e640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d600b0170 0x7f3d601541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d6552e640 1 -- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d60158570 msgr2=0x7f3d60160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d6552e640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d60158570 0x7f3d60160680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.505+0000 7f3d6552e640 1 -- 172.21.15.59:0/13962789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d601e80f0 con 0x7f3d600b3cc0 2024-03-06T07:26:35.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.506+0000 7f3d6552e640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d600b3cc0 0x7f3d60160140 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000c430 tx=0x7f3d5000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.506+0000 7f3d567fc640 1 -- 172.21.15.59:0/13962789 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50011790 con 0x7f3d600b3cc0 2024-03-06T07:26:35.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.506+0000 7f3d567fc640 1 -- 172.21.15.59:0/13962789 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d50017070 con 0x7f3d600b3cc0 2024-03-06T07:26:35.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.506+0000 7f3d567fc640 1 -- 172.21.15.59:0/13962789 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50011bb0 con 0x7f3d600b3cc0 2024-03-06T07:26:35.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.506+0000 7f3d67f93640 1 -- 172.21.15.59:0/13962789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d601e83d0 con 0x7f3d600b3cc0 2024-03-06T07:26:35.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.506+0000 7f3d67f93640 1 -- 172.21.15.59:0/13962789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d601e8990 con 0x7f3d600b3cc0 2024-03-06T07:26:35.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.507+0000 7f3d67f93640 1 -- 172.21.15.59:0/13962789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d60069ee0 con 0x7f3d600b3cc0 2024-03-06T07:26:35.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.507+0000 7f3d567fc640 1 -- 172.21.15.59:0/13962789 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d5001c460 con 0x7f3d600b3cc0 2024-03-06T07:26:35.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.507+0000 7f3d567fc640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d4406b500 0x7f3d4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.508+0000 7f3d567fc640 1 -- 172.21.15.59:0/13962789 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f3d50089f70 con 0x7f3d600b3cc0 2024-03-06T07:26:35.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.508+0000 7f3d65d2f640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d4406b500 0x7f3d4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.508+0000 7f3d65d2f640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d4406b500 0x7f3d4406d9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3d601f7420 tx=0x7f3d5800d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.512+0000 7f3d567fc640 1 -- 172.21.15.59:0/13962789 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d50058730 con 0x7f3d600b3cc0 2024-03-06T07:26:35.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.736+0000 7f3d67f93640 1 -- 172.21.15.59:0/13962789 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d600d07e0 con 0x7f3d600b3cc0 2024-03-06T07:26:35.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.736+0000 7f3d567fc640 1 -- 172.21.15.59:0/13962789 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f3d5005c7d0 con 0x7f3d600b3cc0 2024-03-06T07:26:35.739 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:35.739 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:35.739 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:35.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 -- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d4406b500 msgr2=0x7f3d4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d4406b500 0x7f3d4406d9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3d601f7420 tx=0x7f3d5800d040 comp rx=0 tx=0).stop 2024-03-06T07:26:35.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 -- 172.21.15.59:0/13962789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d600b3cc0 msgr2=0x7f3d60160140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d600b3cc0 0x7f3d60160140 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000c430 tx=0x7f3d5000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 -- 172.21.15.59:0/13962789 shutdown_connections 2024-03-06T07:26:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3d4406b500 0x7f3d4406d9c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d60158570 0x7f3d60160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d600b3cc0 0x7f3d60160140 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 --2- 172.21.15.59:0/13962789 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d600b0170 0x7f3d601541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 -- 172.21.15.59:0/13962789 >> 172.21.15.59:0/13962789 conn(0x7f3d600b29c0 msgr2=0x7f3d600af500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 -- 172.21.15.59:0/13962789 shutdown_connections 2024-03-06T07:26:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:35.738+0000 7f3d67f93640 1 -- 172.21.15.59:0/13962789 wait complete. 2024-03-06T07:26:36.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:17.219713+0000', 'modified': '2024-03-06T07:26:34.073810+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 95, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7635}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7635': {'gid': 7635, 'name': 'a', 'rank': 0, 'incarnation': 55, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.59:6835/4004944688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6833', 'nonce': 4004944688}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 4004944688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7635, 'qdb_cluster': [7635]} 2024-03-06T07:26:36.157 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:36.157 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:26:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.257+0000 7f5b0b49d640 1 Processor -- start 2024-03-06T07:26:36.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.257+0000 7f5b0b49d640 1 -- start start 2024-03-06T07:26:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.258+0000 7f5b0b49d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5b0406bf90 0x7f5b0406c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.258+0000 7f5b0b49d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b040aeb80 0x7f5b040af010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.258+0000 7f5b0b49d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0406ded0 con 0x7f5b04155380 2024-03-06T07:26:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.258+0000 7f5b0b49d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0406c940 con 0x7f5b0406bf90 2024-03-06T07:26:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.258+0000 7f5b0b49d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040ae430 con 0x7f5b040aeb80 2024-03-06T07:26:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.258+0000 7f5b09239640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f5b04155380 0x7f5b0406d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35110/0 (socket says 172.21.15.59:35110) 2024-03-06T07:26:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.258+0000 7f5b09239640 1 -- 172.21.15.59:0/3667123979 learned_addr learned my addr 172.21.15.59:0/3667123979 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.258+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487514732 0 0) 0x7f5b0406ded0 con 0x7f5b04155380 2024-03-06T07:26:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.258+0000 7f5b08a38640 1 --2- 172.21.15.59:0/3667123979 >> v2:172.21.15.164:3300/0 conn(0x7f5b0406bf90 0x7f5b0406c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.258+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5af0003520 con 0x7f5b04155380 2024-03-06T07:26:36.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.258+0000 7f5b09a3a640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b040aeb80 0x7f5b040af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290912640 0 0) 0x7f5af0003520 con 0x7f5b04155380 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b040aeb80 msgr2=0x7f5b040af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b040aeb80 0x7f5b040af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 >> v2:172.21.15.164:3300/0 conn(0x7f5b0406bf90 msgr2=0x7f5b0406c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 --2- 172.21.15.59:0/3667123979 >> v2:172.21.15.164:3300/0 conn(0x7f5b0406bf90 0x7f5b0406c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0406e1b0 con 0x7f5b04155380 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f5af8001670 con 0x7f5b04155380 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 >> v1:172.21.15.59:6789/0 conn(0x7f5b04155380 legacy=0x7f5b0406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5af00039b0 0x7f5af0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af0004800 0x7f5af0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b040aeb80 0x7f5af00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0406c940 con 0x7f5af0004800 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af000b9c0 con 0x7f5af00039b0 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af000bb40 con 0x7f5b040aeb80 2024-03-06T07:26:36.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5b08a38640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af0004800 0x7f5af0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5b09239640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5af00039b0 0x7f5af0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5b09a3a640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b040aeb80 0x7f5af00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5b08a38640 1 -- 172.21.15.59:0/3667123979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b040aeb80 msgr2=0x7f5af00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5b08a38640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b040aeb80 0x7f5af00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5b08a38640 1 -- 172.21.15.59:0/3667123979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5af00039b0 msgr2=0x7f5af0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5b08a38640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5af00039b0 0x7f5af0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.259+0000 7f5b08a38640 1 -- 172.21.15.59:0/3667123979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af0003520 con 0x7f5af0004800 2024-03-06T07:26:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.260+0000 7f5b08a38640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af0004800 0x7f5af0004c80 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009ac0 tx=0x7f5af0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ea4b7148563ce8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.260+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00039070 con 0x7f5af0004800 2024-03-06T07:26:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.260+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0002c710 con 0x7f5af0004800 2024-03-06T07:26:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.260+0000 7f5af7fff640 1 -- 172.21.15.59:0/3667123979 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00009690 con 0x7f5af0004800 2024-03-06T07:26:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.261+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3667123979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af0004800 msgr2=0x7f5af0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.261+0000 7f5b0b49d640 1 --2- 172.21.15.59:0/3667123979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af0004800 0x7f5af0004c80 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009ac0 tx=0x7f5af0005830 comp rx=0 tx=0).stop 2024-03-06T07:26:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.261+0000 7f5b09a3a640 1 -- 172.21.15.59:0/3667123979 reap_dead start 2024-03-06T07:26:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.261+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3667123979 shutdown_connections 2024-03-06T07:26:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.261+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3667123979 >> 172.21.15.59:0/3667123979 conn(0x7f5b040b2980 msgr2=0x7f5b04071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.261+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3667123979 shutdown_connections 2024-03-06T07:26:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.261+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3667123979 wait complete. 2024-03-06T07:26:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.262+0000 7f5b0b49d640 1 Processor -- start 2024-03-06T07:26:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.262+0000 7f5b0b49d640 1 -- start start 2024-03-06T07:26:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b0b49d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5af0004800 0x7f5b041f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b0b49d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b0415aee0 0x7f5b0414fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b0b49d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b04150090 0x7f5b040b12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b0b49d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040ae660 con 0x7f5b04150090 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b0b49d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0406e770 con 0x7f5af0004800 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b0b49d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0406f450 con 0x7f5b0415aee0 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b09a3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b04150090 0x7f5b040b12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b09a3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b04150090 0x7f5b040b12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44116/0 (socket says 172.21.15.59:44116) 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b09a3a640 1 -- 172.21.15.59:0/3514538854 learned_addr learned my addr 172.21.15.59:0/3514538854 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b09239640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5af0004800 0x7f5b041f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b08a38640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b0415aee0 0x7f5b0414fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b09a3a640 1 -- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b0415aee0 msgr2=0x7f5b0414fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b09a3a640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b0415aee0 0x7f5b0414fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b09a3a640 1 -- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5af0004800 msgr2=0x7f5b041f4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b09a3a640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5af0004800 0x7f5b041f4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.263+0000 7f5b09a3a640 1 -- 172.21.15.59:0/3514538854 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b041f4a40 con 0x7f5b04150090 2024-03-06T07:26:36.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.264+0000 7f5b09a3a640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b04150090 0x7f5b040b12b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5b040ae1a0 tx=0x7f5afc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.264+0000 7f5af5ffb640 1 -- 172.21.15.59:0/3514538854 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc015430 con 0x7f5b04150090 2024-03-06T07:26:36.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.264+0000 7f5af5ffb640 1 -- 172.21.15.59:0/3514538854 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5afc01d070 con 0x7f5b04150090 2024-03-06T07:26:36.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.264+0000 7f5af5ffb640 1 -- 172.21.15.59:0/3514538854 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc015810 con 0x7f5b04150090 2024-03-06T07:26:36.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.264+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3514538854 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b04151a10 con 0x7f5b04150090 2024-03-06T07:26:36.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.264+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3514538854 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b04151fd0 con 0x7f5b04150090 2024-03-06T07:26:36.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.266+0000 7f5af5ffb640 1 -- 172.21.15.59:0/3514538854 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5afc0159b0 con 0x7f5b04150090 2024-03-06T07:26:36.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.266+0000 7f5af5ffb640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5ae806b4d0 0x7f5ae806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.266+0000 7f5af5ffb640 1 -- 172.21.15.59:0/3514538854 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5afc08aac0 con 0x7f5b04150090 2024-03-06T07:26:36.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.267+0000 7f5b09239640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5ae806b4d0 0x7f5ae806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.267+0000 7f5b0e167740 1 Processor -- start 2024-03-06T07:26:36.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.267+0000 7f5b0e167740 1 -- start start 2024-03-06T07:26:36.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.267+0000 7f5b09239640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5ae806b4d0 0x7f5ae806d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5af8001050 tx=0x7f5af80078f0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598313ee870 0x5598313eec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5598313f28c0 0x5598313f4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55983141e600 0x55983141ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b0e167740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598313586b0 con 0x5598313ee870 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b0e167740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x559831301a60 con 0x55983141e600 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b0e167740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x559831354750 con 0x5598313f28c0 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b09239640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598313ee870 0x5598313eec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b09239640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598313ee870 0x5598313eec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44130/0 (socket says 172.21.15.59:44130) 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b09239640 1 -- 172.21.15.59:0/928143604 learned_addr learned my addr 172.21.15.59:0/928143604 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b08a38640 1 --2- 172.21.15.59:0/928143604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5598313f28c0 0x5598313f4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b09a3a640 1 --2- 172.21.15.59:0/928143604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55983141e600 0x55983141ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b09239640 1 -- 172.21.15.59:0/928143604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5598313f28c0 msgr2=0x5598313f4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b09239640 1 --2- 172.21.15.59:0/928143604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5598313f28c0 0x5598313f4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b09239640 1 -- 172.21.15.59:0/928143604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55983141e600 msgr2=0x55983141ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b09239640 1 --2- 172.21.15.59:0/928143604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55983141e600 0x55983141ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b09239640 1 -- 172.21.15.59:0/928143604 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55983131ff30 con 0x5598313ee870 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5b09239640 1 --2- 172.21.15.59:0/928143604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598313ee870 0x5598313eec50 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5af8007820 tx=0x7f5af800c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf66937c3153138 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5ae6ffd640 1 -- 172.21.15.59:0/928143604 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af8028070 con 0x5598313ee870 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5ae6ffd640 1 -- 172.21.15.59:0/928143604 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af8015bd0 con 0x5598313ee870 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.268+0000 7f5ae6ffd640 1 -- 172.21.15.59:0/928143604 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af80213c0 con 0x5598313ee870 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.269+0000 7f5b0e167740 1 -- 172.21.15.59:0/928143604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598313ee870 msgr2=0x5598313eec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.269+0000 7f5b0e167740 1 --2- 172.21.15.59:0/928143604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598313ee870 0x5598313eec50 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5af8007820 tx=0x7f5af800c000 comp rx=0 tx=0).stop 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.269+0000 7f5b0e167740 1 -- 172.21.15.59:0/928143604 shutdown_connections 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.269+0000 7f5b0e167740 1 --2- 172.21.15.59:0/928143604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55983141e600 0x55983141ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.269+0000 7f5b0e167740 1 --2- 172.21.15.59:0/928143604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5598313f28c0 0x5598313f4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.269+0000 7f5b0e167740 1 --2- 172.21.15.59:0/928143604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5598313ee870 0x5598313eec50 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.269+0000 7f5b0e167740 1 -- 172.21.15.59:0/928143604 >> 172.21.15.59:0/928143604 conn(0x5598313f08f0 msgr2=0x5598313f21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.269+0000 7f5b0e167740 1 -- 172.21.15.59:0/928143604 shutdown_connections 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.269+0000 7f5b0e167740 1 -- 172.21.15.59:0/928143604 wait complete. 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.270+0000 7f5b0e167740 1 -- start start 2024-03-06T07:26:36.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.270+0000 7f5b0e167740 1 Processor -- start 2024-03-06T07:26:36.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.271+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 0x559831410bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x5598313f3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559831411750 0x5598313f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b0e167740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598313f2d50 con 0x559831413230 2024-03-06T07:26:36.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b0e167740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x559831314ea0 con 0x559831411750 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b0e167740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5598313f3030 con 0x55983141e600 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b0e167740 10 client.0 _ll_register_callbacks cb 0x559831314870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b08a38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x5598313f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b08a38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x5598313f3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44134/0 (socket says 172.21.15.59:44134) 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b08a38640 1 -- 172.21.15.59:0/1453553015 learned_addr learned my addr 172.21.15.59:0/1453553015 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b09239640 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 0x559831410bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b09a3a640 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559831411750 0x5598313f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b08a38640 1 -- 172.21.15.59:0/1453553015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 msgr2=0x559831410bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b08a38640 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 0x559831410bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b08a38640 1 -- 172.21.15.59:0/1453553015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559831411750 msgr2=0x5598313f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b08a38640 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559831411750 0x5598313f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b08a38640 1 -- 172.21.15.59:0/1453553015 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559831315270 con 0x559831413230 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b09239640 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 0x559831410bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b09a3a640 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559831411750 0x5598313f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:36.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b08a38640 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x5598313f3840 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f5b0002c380 tx=0x7f5b000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c5991754759d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5adffff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5adffff640 1 -- 172.21.15.59:0/1453553015 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00045070 con 0x559831413230 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5adffff640 1 -- 172.21.15.59:0/1453553015 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5b0002cb20 con 0x559831413230 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.272+0000 7f5b0e167740 1 -- 172.21.15.59:0/1453553015 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5af8015bd0 con 0x559831413230 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5adffff640 1 -- 172.21.15.59:0/1453553015 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b00050050 con 0x559831413230 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5b0e167740 10 client.7683 fetch_fsmap learned FSMap version 59 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5b0e167740 1 -- 172.21.15.59:0/1453553015 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x55983106d2d0 con 0x559831413230 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5adffff640 1 -- 172.21.15.59:0/1453553015 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af8015bd0 con 0x559831413230 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5adffff640 1 -- 172.21.15.59:0/1453553015 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 59) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5b00053020 con 0x559831413230 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5b0e167740 10 client.7683 fetch_fsmap finished waiting for FSMap version 59 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5b0e167740 10 client.7683 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5b0e167740 20 client.7683 populate_metadata read hostname 'smithi059' 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5b0e167740 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.59:6833/4004944688,v1:172.21.15.59:6835/4004944688] conn(0x55983141a9c0 0x559831458810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5b0e167740 4 client.7683 mds_command: new command op to 7635 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5b0e167740 1 -- 172.21.15.59:0/1453553015 --> [v2:172.21.15.59:6833/4004944688,v1:172.21.15.59:6835/4004944688] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559831365bf0 con 0x55983141a9c0 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5b09239640 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.59:6833/4004944688,v1:172.21.15.59:6835/4004944688] conn(0x55983141a9c0 0x559831458810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5adffff640 10 client.7683 ms_handle_connect on v2:172.21.15.59:6833/4004944688 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.273+0000 7f5b09239640 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.59:6833/4004944688,v1:172.21.15.59:6835/4004944688] conn(0x55983141a9c0 0x559831458810 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d2f6793b1b50c5f server_cookie=ab65900e8415ccb1 in_seq=0 out_seq=0 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5adffff640 1 -- 172.21.15.59:0/1453553015 <== mds.0 v2:172.21.15.59:6833/4004944688 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559831365bf0 con 0x55983141a9c0 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5adffff640 10 client.7683 handle_command_reply: tid=0 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 1 client.7683 shutdown 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 1 -- 172.21.15.59:0/1453553015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 msgr2=0x5598313f3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x5598313f3840 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f5b0002c380 tx=0x7f5b000046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 1 -- 172.21.15.59:0/1453553015 shutdown_connections 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.59:6833/4004944688,v1:172.21.15.59:6835/4004944688] conn(0x55983141a9c0 0x559831458810 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559831411750 0x5598313f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x5598313f3840 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 1 --2- 172.21.15.59:0/1453553015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 0x559831410bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5adffff640 0 client.7683 ms_handle_reset on v2:172.21.15.59:6833/4004944688 2024-03-06T07:26:36.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 1 -- 172.21.15.59:0/1453553015 >> 172.21.15.59:0/1453553015 conn(0x5598313ee870 msgr2=0x55983141c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 1 -- 172.21.15.59:0/1453553015 shutdown_connections 2024-03-06T07:26:36.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 1 -- 172.21.15.59:0/1453553015 wait complete. 2024-03-06T07:26:36.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.274+0000 7f5b0e167740 20 client.7683 trim_cache size 0 max 16384 2024-03-06T07:26:36.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.291+0000 7f5b0e167740 1 Processor -- start 2024-03-06T07:26:36.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.291+0000 7f5b0e167740 1 -- start start 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559831411750 0x5598314152e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x559831416fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 0x5598314a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b0e167740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b04200ae0 con 0x559831413230 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b0e167740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b041ffc10 con 0x559831411750 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b0e167740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b04160d80 con 0x55983141e600 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b08a38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x559831416fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b08a38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x559831416fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44136/0 (socket says 172.21.15.59:44136) 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b08a38640 1 -- 172.21.15.59:0/1440067625 learned_addr learned my addr 172.21.15.59:0/1440067625 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b09239640 1 --2- 172.21.15.59:0/1440067625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559831411750 0x5598314152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b09a3a640 1 --2- 172.21.15.59:0/1440067625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 0x5598314a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b08a38640 1 -- 172.21.15.59:0/1440067625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 msgr2=0x5598314a31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b08a38640 1 --2- 172.21.15.59:0/1440067625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 0x5598314a31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b08a38640 1 -- 172.21.15.59:0/1440067625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559831411750 msgr2=0x5598314152e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b08a38640 1 --2- 172.21.15.59:0/1440067625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559831411750 0x5598314152e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b08a38640 1 -- 172.21.15.59:0/1440067625 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598311376b0 con 0x559831413230 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.292+0000 7f5b09a3a640 1 --2- 172.21.15.59:0/1440067625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 0x5598314a31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5b08a38640 1 --2- 172.21.15.59:0/1440067625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x559831416fe0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f5b0002c350 tx=0x7f5b00041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a2534a698be93f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5ae67fc640 1 -- 172.21.15.59:0/1440067625 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00045040 con 0x559831413230 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5ae67fc640 1 -- 172.21.15.59:0/1440067625 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0003a040 con 0x559831413230 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5ae67fc640 1 -- 172.21.15.59:0/1440067625 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00039040 con 0x559831413230 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5b0e167740 1 -- 172.21.15.59:0/1440067625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 msgr2=0x559831416fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5b0e167740 1 --2- 172.21.15.59:0/1440067625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x559831416fe0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f5b0002c350 tx=0x7f5b00041000 comp rx=0 tx=0).stop 2024-03-06T07:26:36.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5b0e167740 1 -- 172.21.15.59:0/1440067625 shutdown_connections 2024-03-06T07:26:36.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5b0e167740 1 --2- 172.21.15.59:0/1440067625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55983141e600 0x5598314a31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5b0e167740 1 --2- 172.21.15.59:0/1440067625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x559831416fe0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5b0e167740 1 --2- 172.21.15.59:0/1440067625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559831411750 0x5598314152e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5b0e167740 1 -- 172.21.15.59:0/1440067625 >> 172.21.15.59:0/1440067625 conn(0x5598313ee870 msgr2=0x559831422200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5b0e167740 1 -- 172.21.15.59:0/1440067625 shutdown_connections 2024-03-06T07:26:36.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.293+0000 7f5b0e167740 1 -- 172.21.15.59:0/1440067625 wait complete. 2024-03-06T07:26:36.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.294+0000 7f5b0e167740 1 -- start start 2024-03-06T07:26:36.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.294+0000 7f5b0e167740 1 Processor -- start 2024-03-06T07:26:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559831411750 0x559831416fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x55983141e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b0e167740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55983141e600 0x55983141ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b0e167740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x559831416450 con 0x559831413230 2024-03-06T07:26:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b0e167740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b041ff610 con 0x55983141e600 2024-03-06T07:26:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b0e167740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x559831413910 con 0x559831411750 2024-03-06T07:26:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b0e167740 10 client.0 _ll_register_callbacks cb 0x7f5af0004580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b08a38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x55983141e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b08a38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x55983141e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44152/0 (socket says 172.21.15.59:44152) 2024-03-06T07:26:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b08a38640 1 -- 172.21.15.59:0/2704864413 learned_addr learned my addr 172.21.15.59:0/2704864413 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:36.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b09a3a640 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55983141e600 0x55983141ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b09239640 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559831411750 0x559831416fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b08a38640 1 -- 172.21.15.59:0/2704864413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559831411750 msgr2=0x559831416fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b08a38640 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559831411750 0x559831416fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b08a38640 1 -- 172.21.15.59:0/2704864413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55983141e600 msgr2=0x55983141ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b08a38640 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55983141e600 0x55983141ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b08a38640 1 -- 172.21.15.59:0/2704864413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559831453ba0 con 0x559831413230 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b09239640 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559831411750 0x559831416fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5b08a38640 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x55983141e9e0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f5b000098e0 tx=0x7f5b00009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=426288a41c68b619 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.295+0000 7f5adffff640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5adffff640 1 -- 172.21.15.59:0/2704864413 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00039040 con 0x559831413230 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5adffff640 1 -- 172.21.15.59:0/2704864413 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5b0005dba0 con 0x559831413230 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5b0e167740 1 -- 172.21.15.59:0/2704864413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b0003a040 con 0x559831413230 2024-03-06T07:26:36.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5adffff640 1 -- 172.21.15.59:0/2704864413 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b00045040 con 0x559831413230 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5b0e167740 10 client.7689 fetch_fsmap learned FSMap version 59 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5b0e167740 1 -- 172.21.15.59:0/2704864413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) v3 -- 0x559831414e50 con 0x559831413230 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5adffff640 1 -- 172.21.15.59:0/2704864413 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0003a040 con 0x559831413230 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5adffff640 1 -- 172.21.15.59:0/2704864413 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 59) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5b00053070 con 0x559831413230 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5b0e167740 10 client.7689 fetch_fsmap finished waiting for FSMap version 59 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5b0e167740 10 client.7689 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5b0e167740 20 client.7689 populate_metadata read hostname 'smithi059' 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5b0e167740 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.59:6833/4004944688,v1:172.21.15.59:6835/4004944688] conn(0x559831414880 0x55983141b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5b0e167740 4 client.7689 mds_command: new command op to 7635 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5b0e167740 1 -- 172.21.15.59:0/2704864413 --> [v2:172.21.15.59:6833/4004944688,v1:172.21.15.59:6835/4004944688] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55983135ff40 con 0x559831414880 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.296+0000 7f5b09239640 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.59:6833/4004944688,v1:172.21.15.59:6835/4004944688] conn(0x559831414880 0x55983141b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b09239640 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.59:6833/4004944688,v1:172.21.15.59:6835/4004944688] conn(0x559831414880 0x55983141b200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67fb81bc4218673f server_cookie=349f2c6329ba36e4 in_seq=0 out_seq=0 2024-03-06T07:26:36.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5adffff640 10 client.7689 ms_handle_connect on v2:172.21.15.59:6833/4004944688 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5adffff640 1 -- 172.21.15.59:0/2704864413 <== mds.0 v2:172.21.15.59:6833/4004944688 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55983135ff40 con 0x559831414880 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5adffff640 10 client.7689 handle_command_reply: tid=0 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 1 client.7689 shutdown 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 1 -- 172.21.15.59:0/2704864413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 msgr2=0x55983141e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x55983141e9e0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f5b000098e0 tx=0x7f5b00009190 comp rx=0 tx=0).stop 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 1 -- 172.21.15.59:0/2704864413 shutdown_connections 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.59:6833/4004944688,v1:172.21.15.59:6835/4004944688] conn(0x559831414880 0x55983141b200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55983141e600 0x55983141ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x559831413230 0x55983141e9e0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5adffff640 0 client.7689 ms_handle_reset on v2:172.21.15.59:6833/4004944688 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 1 --2- 172.21.15.59:0/2704864413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559831411750 0x559831416fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 1 -- 172.21.15.59:0/2704864413 >> 172.21.15.59:0/2704864413 conn(0x5598313ee870 msgr2=0x55983143d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 1 -- 172.21.15.59:0/2704864413 shutdown_connections 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 1 -- 172.21.15.59:0/2704864413 wait complete. 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.297+0000 7f5b0e167740 20 client.7689 trim_cache size 0 max 16384 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 7635, 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 6.2357468730000001, 2024-03-06T07:26:36.306 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 59, 2024-03-06T07:26:36.307 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 96, 2024-03-06T07:26:36.307 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 96, 2024-03-06T07:26:36.307 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 6.5607106179999999 2024-03-06T07:26:36.307 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5ae806b4d0 msgr2=0x7f5ae806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5ae806b4d0 0x7f5ae806d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5af8001050 tx=0x7f5af80078f0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3514538854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b04150090 msgr2=0x7f5b040b12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b04150090 0x7f5b040b12b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5b040ae1a0 tx=0x7f5afc00f580 comp rx=0 tx=0).stop 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3514538854 shutdown_connections 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5ae806b4d0 0x7f5ae806d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b04150090 0x7f5b040b12b0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b0415aee0 0x7f5b0414fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 --2- 172.21.15.59:0/3514538854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5af0004800 0x7f5b041f4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3514538854 >> 172.21.15.59:0/3514538854 conn(0x7f5b040b2980 msgr2=0x7f5b040b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3514538854 shutdown_connections 2024-03-06T07:26:36.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:36.298+0000 7f5b0b49d640 1 -- 172.21.15.59:0/3514538854 wait complete. 2024-03-06T07:26:36.310 DEBUG:teuthology.orchestra.run.smithi059:> stdin-killer -- python3 -c ' 2024-03-06T07:26:36.311 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:26:36.311 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:26:36.311 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:36.311 DEBUG:teuthology.orchestra.run.smithi059:> subdir = "delete_me" 2024-03-06T07:26:36.311 DEBUG:teuthology.orchestra.run.smithi059:> size_unit = 1024 2024-03-06T07:26:36.311 DEBUG:teuthology.orchestra.run.smithi059:> file_multiplier = 200 2024-03-06T07:26:36.311 DEBUG:teuthology.orchestra.run.smithi059:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:26:36.311 DEBUG:teuthology.orchestra.run.smithi059:> for i in range(0, file_multiplier): 2024-03-06T07:26:36.311 DEBUG:teuthology.orchestra.run.smithi059:> for size in range(0, 16*size_unit, size_unit): 2024-03-06T07:26:36.311 DEBUG:teuthology.orchestra.run.smithi059:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-03-06T07:26:36.311 DEBUG:teuthology.orchestra.run.smithi059:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:26:36.312 DEBUG:teuthology.orchestra.run.smithi059:> f.write(size * '"'"'x'"'"') 2024-03-06T07:26:36.312 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:26:42.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:42.890 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_max_purge_ops 2024-03-06T07:26:42.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:42.890 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:26:43.068 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_max_purge_ops":"100000000"} 2024-03-06T07:26:43.068 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-03-06T07:26:43.069 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json config get mds_max_purge_files 2024-03-06T07:26:43.182 INFO:teuthology.orchestra.run.smithi164.stdout:{"mds_max_purge_files":"6"} 2024-03-06T07:26:43.182 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-03-06T07:26:43.182 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:43.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.299+0000 7fb55b0ec640 1 Processor -- start 2024-03-06T07:26:43.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.300+0000 7fb55b0ec640 1 -- start start 2024-03-06T07:26:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.300+0000 7fb55b0ec640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb5540d0db0 0x7fb5540b9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.300+0000 7fb55b0ec640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5540d1ea0 con 0x7fb5540d0950 2024-03-06T07:26:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.300+0000 7fb55b0ec640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5540d2180 con 0x7fb55415e150 2024-03-06T07:26:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.300+0000 7fb55b0ec640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5540d2ab0 con 0x7fb5540d0db0 2024-03-06T07:26:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.300+0000 7fb553fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb5540d0950 0x7fb5540b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35114/0 (socket says 172.21.15.59:35114) 2024-03-06T07:26:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.300+0000 7fb553fff640 1 -- 172.21.15.59:0/3617785144 learned_addr learned my addr 172.21.15.59:0/3617785144 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.301+0000 7fb559689640 1 --2- 172.21.15.59:0/3617785144 >> v2:172.21.15.164:3300/0 conn(0x7fb5540d0db0 0x7fb5540b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.301+0000 7fb5537fe640 1 -- 172.21.15.59:0/3617785144 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215437742 0 0) 0x7fb5540d1ea0 con 0x7fb5540d0950 2024-03-06T07:26:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.301+0000 7fb5537fe640 1 -- 172.21.15.59:0/3617785144 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb544003520 con 0x7fb5540d0950 2024-03-06T07:26:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.301+0000 7fb5537fe640 1 -- 172.21.15.59:0/3617785144 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033149247 0 0) 0x7fb5540d2180 con 0x7fb55415e150 2024-03-06T07:26:43.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.301+0000 7fb5537fe640 1 -- 172.21.15.59:0/3617785144 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5540d1ea0 con 0x7fb55415e150 2024-03-06T07:26:43.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.301+0000 7fb559689640 1 -- 172.21.15.59:0/3617785144 >> v1:172.21.15.164:6789/0 conn(0x7fb55415e150 legacy=0x7fb5540cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.301+0000 7fb559689640 1 -- 172.21.15.59:0/3617785144 >> v1:172.21.15.59:6789/0 conn(0x7fb5540d0950 legacy=0x7fb5540b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.301+0000 7fb559689640 1 -- 172.21.15.59:0/3617785144 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5540d2460 con 0x7fb5540d0db0 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.302+0000 7fb559689640 1 --2- 172.21.15.59:0/3617785144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5540d0db0 0x7fb5540b9770 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009d90 tx=0x7fb54c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f574e50fc505348a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.302+0000 7fb5537fe640 1 -- 172.21.15.59:0/3617785144 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c039070 con 0x7fb5540d0db0 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.302+0000 7fb5537fe640 1 -- 172.21.15.59:0/3617785144 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c02cc60 con 0x7fb5540d0db0 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.302+0000 7fb5537fe640 1 -- 172.21.15.59:0/3617785144 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c034660 con 0x7fb5540d0db0 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.303+0000 7fb55b0ec640 1 -- 172.21.15.59:0/3617785144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5540d0db0 msgr2=0x7fb5540b9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.303+0000 7fb55b0ec640 1 --2- 172.21.15.59:0/3617785144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5540d0db0 0x7fb5540b9770 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009d90 tx=0x7fb54c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.303+0000 7fb55b0ec640 1 -- 172.21.15.59:0/3617785144 shutdown_connections 2024-03-06T07:26:43.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.303+0000 7fb55b0ec640 1 --2- 172.21.15.59:0/3617785144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5540d0db0 0x7fb5540b9770 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.303+0000 7fb55b0ec640 1 -- 172.21.15.59:0/3617785144 >> 172.21.15.59:0/3617785144 conn(0x7fb55409d230 msgr2=0x7fb5540d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.303+0000 7fb55b0ec640 1 -- 172.21.15.59:0/3617785144 shutdown_connections 2024-03-06T07:26:43.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.303+0000 7fb55b0ec640 1 -- 172.21.15.59:0/3617785144 wait complete. 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.304+0000 7fb55b0ec640 1 Processor -- start 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.304+0000 7fb55b0ec640 1 -- start start 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb55b0ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5540d0950 0x7fb554151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb55b0ec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5540d0db0 0x7fb5540cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb55b0ec640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb55415e150 0x7fb55415f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb55b0ec640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540d2f40 con 0x7fb55415e150 2024-03-06T07:26:43.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb55b0ec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540d1220 con 0x7fb5540d0db0 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb55b0ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540b7dc0 con 0x7fb5540d0950 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb559689640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb55415e150 0x7fb55415f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb559689640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb55415e150 0x7fb55415f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44160/0 (socket says 172.21.15.59:44160) 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb559689640 1 -- 172.21.15.59:0/2882145091 learned_addr learned my addr 172.21.15.59:0/2882145091 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb553fff640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5540d0db0 0x7fb5540cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb558e88640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5540d0950 0x7fb554151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb559689640 1 -- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5540d0950 msgr2=0x7fb554151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb559689640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5540d0950 0x7fb554151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb559689640 1 -- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5540d0db0 msgr2=0x7fb5540cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb559689640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5540d0db0 0x7fb5540cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.305+0000 7fb559689640 1 -- 172.21.15.59:0/2882145091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5540cb0d0 con 0x7fb55415e150 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.306+0000 7fb559689640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb55415e150 0x7fb55415f580 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009eb0 tx=0x7fb54c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.306+0000 7fb5517fa640 1 -- 172.21.15.59:0/2882145091 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c039050 con 0x7fb55415e150 2024-03-06T07:26:43.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.306+0000 7fb5517fa640 1 -- 172.21.15.59:0/2882145091 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c045070 con 0x7fb55415e150 2024-03-06T07:26:43.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.306+0000 7fb55b0ec640 1 -- 172.21.15.59:0/2882145091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5540c8220 con 0x7fb55415e150 2024-03-06T07:26:43.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.306+0000 7fb5517fa640 1 -- 172.21.15.59:0/2882145091 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c009450 con 0x7fb55415e150 2024-03-06T07:26:43.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.306+0000 7fb55b0ec640 1 -- 172.21.15.59:0/2882145091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb554151880 con 0x7fb55415e150 2024-03-06T07:26:43.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.307+0000 7fb55b0ec640 1 -- 172.21.15.59:0/2882145091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5540d1b30 con 0x7fb55415e150 2024-03-06T07:26:43.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.308+0000 7fb5517fa640 1 -- 172.21.15.59:0/2882145091 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb54c03e850 con 0x7fb55415e150 2024-03-06T07:26:43.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.308+0000 7fb5517fa640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb53806b4d0 0x7fb53806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:43.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.308+0000 7fb5517fa640 1 -- 172.21.15.59:0/2882145091 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb54c0ad180 con 0x7fb55415e150 2024-03-06T07:26:43.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.308+0000 7fb558e88640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb53806b4d0 0x7fb53806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:43.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.309+0000 7fb558e88640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb53806b4d0 0x7fb53806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb548007fd0 tx=0x7fb548009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:43.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.313+0000 7fb5517fa640 1 -- 172.21.15.59:0/2882145091 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb5540d1b30 con 0x7fb55415e150 2024-03-06T07:26:43.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.592+0000 7fb55b0ec640 1 -- 172.21.15.59:0/2882145091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb554069ee0 con 0x7fb55415e150 2024-03-06T07:26:43.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.593+0000 7fb5517fa640 1 -- 172.21.15.59:0/2882145091 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fb54c07b9d0 con 0x7fb55415e150 2024-03-06T07:26:43.595 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:43.595 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:43.595 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:43.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 -- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb53806b4d0 msgr2=0x7fb53806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb53806b4d0 0x7fb53806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb548007fd0 tx=0x7fb548009040 comp rx=0 tx=0).stop 2024-03-06T07:26:43.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 -- 172.21.15.59:0/2882145091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb55415e150 msgr2=0x7fb55415f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:43.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb55415e150 0x7fb55415f580 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009eb0 tx=0x7fb54c004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:43.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 -- 172.21.15.59:0/2882145091 shutdown_connections 2024-03-06T07:26:43.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb53806b4d0 0x7fb53806d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb55415e150 0x7fb55415f580 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5540d0db0 0x7fb5540cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 --2- 172.21.15.59:0/2882145091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5540d0950 0x7fb554151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:43.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 -- 172.21.15.59:0/2882145091 >> 172.21.15.59:0/2882145091 conn(0x7fb55409d230 msgr2=0x7fb5540b8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:43.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 -- 172.21.15.59:0/2882145091 shutdown_connections 2024-03-06T07:26:43.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:43.596+0000 7fb55b0ec640 1 -- 172.21.15.59:0/2882145091 wait complete. 2024-03-06T07:26:43.610 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:43.766 INFO:teuthology.orchestra.run.smithi059.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":18,"msgr_send_messages":20,"msgr_recv_bytes":13277,"msgr_send_bytes":4222420,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.013963996,"msgr_running_send_time":0.007862648,"msgr_running_recv_time":0.003088138,"msgr_running_fast_dispatch_time":0.001914178,"msgr_send_messages_queue_lat":{"avgcount":18,"sum":0.004276267,"avgtime":0.000237570},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000225,"avgtime":0.000000112},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"AsyncMessenger::Worker-1":{"msgr_recv_messages":4406,"msgr_send_messages":10402,"msgr_recv_bytes":1523201,"msgr_send_bytes":6918617,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.468531283,"msgr_running_send_time":0.251266126,"msgr_running_recv_time":0.255595871,"msgr_running_fast_dispatch_time":0.002604930,"msgr_send_messages_queue_lat":{"avgcount":10400,"sum":0.120335125,"avgtime":0.000011570},"msgr_handle_ack_lat":{"avgcount":14678,"sum":0.002606954,"avgtime":0.000000177},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":92224},"AsyncMessenger::Worker-2":{"msgr_recv_messages":50,"msgr_send_messages":44,"msgr_recv_bytes":120035,"msgr_send_bytes":3257251,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.013017894,"msgr_running_send_time":0.005795492,"msgr_running_recv_time":0.006991814,"msgr_running_fast_dispatch_time":0.002305619,"msgr_send_messages_queue_lat":{"avgcount":44,"sum":0.008736314,"avgtime":0.000198552},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108304,"msgr_send_encrypted_bytes":7120},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":592,"complete_latency":{"avgcount":74,"sum":1.193160409,"avgtime":0.016123789}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000150226,"avgtime":0.000037556}},"mds":{"request":3308,"reply":3306,"reply_latency":{"avgcount":3306,"sum":3.712325203,"avgtime":0.001122905},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3314,"traverse_hit":3313,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3010,"root_rbytes":2595840,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":87,"ceph_cap_op_grant":3256,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":1063,"handle_client_caps_dirty":973,"handle_client_cap_release":0,"process_request_cap_release":3393},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":96,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":96,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":4348,"evex":0,"evtrm":0,"ev":4351,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":8,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":12826887,"rdpos":4195244,"jlat":{"avgcount":11,"sum":0.053246504,"avgtime":0.004840591},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3307,"dn+":3307,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":80540,"heap":248164},"mds_server":{"dispatch_client_request":3311,"dispatch_server_request":0,"handle_client_request":3309,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.113519323,"avgtime":0.000972974},"req_getattr_latency":{"avgcount":6,"sum":0.001174100,"avgtime":0.000195683},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002442159,"avgtime":0.002442159},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.451304597,"avgtime":0.150434865},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":96,"sum":0.143885024,"avgtime":0.001498802},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3150,"average_load":3150,"avg_session_uptime":839,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":3315234,"buffer_anon_items":1089,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":5977632,"mds_co_items":29045,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":64,"op_send_bytes":8723403,"op_resend":0,"op_reply":64,"op_latency":{"avgcount":64,"sum":0.147998133,"avgtime":0.002312470},"op_inflight":0,"oplen_avg":{"avgcount":64,"sum":118},"op":64,"op_r":36,"op_w":28,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":11,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":7,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":48,"omap_del":1},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":296,"max":104857600,"get_started":0,"get":4470,"get_sum":1251598,"get_or_fail_fail":0,"get_or_fail_success":4470,"take":0,"take_sum":0,"put":4469,"put_sum":1251302,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":64,"take_sum":8724520,"put":64,"put_sum":8724520,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":64,"take_sum":64,"put":64,"put_sum":64,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a5a2aa2820":{"val":243975,"max":3758096384,"get_started":0,"get":4348,"get_sum":8631202,"get_or_fail_fail":0,"get_or_fail_success":4348,"take":0,"take_sum":0,"put":11,"put_sum":8387227,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:43.768 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.12e-07, "sum": 2.25e-07 }, "msgr_recv_bytes": 13277, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 18, "msgr_running_fast_dispatch_time": 0.001914178, "msgr_running_recv_time": 0.003088138, "msgr_running_send_time": 0.007862648, "msgr_running_total_time": 0.013963996, "msgr_send_bytes": 4222420, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 20, "msgr_send_messages_queue_lat": { "avgcount": 18, "avgtime": 0.00023757, "sum": 0.004276267 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 14678, "avgtime": 1.77e-07, "sum": 0.002606954 }, "msgr_recv_bytes": 1523201, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 4406, "msgr_running_fast_dispatch_time": 0.00260493, "msgr_running_recv_time": 0.255595871, "msgr_running_send_time": 0.251266126, "msgr_running_total_time": 0.468531283, "msgr_send_bytes": 6918617, "msgr_send_encrypted_bytes": 92224, "msgr_send_messages": 10402, "msgr_send_messages_queue_lat": { "avgcount": 10400, "avgtime": 1.157e-05, "sum": 0.120335125 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 120035, "msgr_recv_encrypted_bytes": 108304, "msgr_recv_messages": 50, "msgr_running_fast_dispatch_time": 0.002305619, "msgr_running_recv_time": 0.006991814, "msgr_running_send_time": 0.005795492, "msgr_running_total_time": 0.013017894, "msgr_send_bytes": 3257251, "msgr_send_encrypted_bytes": 7120, "msgr_send_messages": 44, "msgr_send_messages_queue_lat": { "avgcount": 44, "avgtime": 0.000198552, "sum": 0.008736314 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 74, "avgtime": 0.016123789, "sum": 1.193160409 }, "queue_len": 592 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.7556e-05, "sum": 0.000150226 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3256, "ceph_cap_op_revoke": 87, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 1063, "handle_client_caps_dirty": 973, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3393, "q": 0, "reply": 3306, "reply_latency": { "avgcount": 3306, "avgtime": 0.001122905, "sum": 3.712325203 }, "request": 3308, "root_rbytes": 2595840, "root_rfiles": 3010, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3314, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3313, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 96, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 96, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 4351, "evadd": 4348, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 11, "avgtime": 0.004840591, "sum": 0.053246504 }, "rdpos": 4195244, "replayed": 3, "seg": 8, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 12826887 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3307, "dn+": 3307, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 80540 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3311, "dispatch_server_request": 0, "handle_client_request": 3309, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000972974, "sum": 3.113519323 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195683, "sum": 0.0011741 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002442159, "sum": 0.002442159 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.150434865, "sum": 0.451304597 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 96, "avgtime": 0.001498802, "sum": 0.143885024 } }, "mds_sessions": { "average_load": 3150, "avg_session_uptime": 839, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3150 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 3315234, "buffer_anon_items": 1089, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 5977632, "mds_co_items": 29045, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 48, "omap_wr": 15, "op": 64, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 64, "avgtime": 0.00231247, "sum": 0.147998133 }, "op_pg": 0, "op_r": 36, "op_reply": 64, "op_resend": 0, "op_rmw": 0, "op_send": 64, "op_send_bytes": 8723403, "op_w": 28, "oplen_avg": { "avgcount": 64, "sum": 118 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 7, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 11, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 4470, "get_or_fail_fail": 0, "get_or_fail_success": 4470, "get_started": 0, "get_sum": 1251598, "max": 104857600, "put": 4469, "put_sum": 1251302, "take": 0, "take_sum": 0, "val": 296, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 64, "put_sum": 8724520, "take": 64, "take_sum": 8724520, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 64, "put_sum": 64, "take": 64, "take_sum": 64, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a5a2aa2820": { "get": 4348, "get_or_fail_fail": 0, "get_or_fail_success": 4348, "get_started": 0, "get_sum": 8631202, "max": 3758096384, "put": 11, "put_sum": 8387227, "take": 0, "take_sum": 0, "val": 243975, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:43.769 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/96, 0/96 2024-03-06T07:26:44.770 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:44.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.892+0000 7f099e1d0640 1 Processor -- start 2024-03-06T07:26:44.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.892+0000 7f099e1d0640 1 -- start start 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.893+0000 7f099e1d0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f099815f5e0 0x7f09980b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.893+0000 7f099e1d0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f09980b16b0 0x7f09980ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.893+0000 7f099e1d0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09980d2870 0x7f09980cea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.893+0000 7f099e1d0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09980cf920 con 0x7f099815f5e0 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.893+0000 7f099e1d0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09980cd4d0 con 0x7f09980b16b0 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.893+0000 7f099e1d0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09980cd7b0 con 0x7f09980d2870 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.893+0000 7f09977fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f099815f5e0 0x7f09980b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.893+0000 7f09977fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f099815f5e0 0x7f09980b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60764/0 (socket says 172.21.15.59:60764) 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.893+0000 7f09977fe640 1 -- 172.21.15.59:0/1918413249 learned_addr learned my addr 172.21.15.59:0/1918413249 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.894+0000 7f0996ffd640 1 --2- 172.21.15.59:0/1918413249 >> v2:172.21.15.164:3300/0 conn(0x7f09980b16b0 0x7f09980ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.894+0000 7f0997fff640 1 --2- 172.21.15.59:0/1918413249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09980d2870 0x7f09980cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.894+0000 7f09977fe640 1 -- 172.21.15.59:0/1918413249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09980d2870 msgr2=0x7f09980cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:44.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.894+0000 7f09977fe640 1 --2- 172.21.15.59:0/1918413249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09980d2870 0x7f09980cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.894+0000 7f09977fe640 1 -- 172.21.15.59:0/1918413249 >> v2:172.21.15.164:3300/0 conn(0x7f09980b16b0 msgr2=0x7f09980ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:44.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.894+0000 7f09977fe640 1 --2- 172.21.15.59:0/1918413249 >> v2:172.21.15.164:3300/0 conn(0x7f09980b16b0 0x7f09980ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.894+0000 7f09977fe640 1 -- 172.21.15.59:0/1918413249 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09980cda90 con 0x7f099815f5e0 2024-03-06T07:26:44.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.894+0000 7f0997fff640 1 --2- 172.21.15.59:0/1918413249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09980d2870 0x7f09980cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:44.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.894+0000 7f0996ffd640 1 --2- 172.21.15.59:0/1918413249 >> v2:172.21.15.164:3300/0 conn(0x7f09980b16b0 0x7f09980ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:44.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.895+0000 7f09977fe640 1 --2- 172.21.15.59:0/1918413249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f099815f5e0 0x7f09980b2bf0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f0998071dc0 tx=0x7f098802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d9edbd23a7250a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.895+0000 7f09967fc640 1 -- 172.21.15.59:0/1918413249 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098802e070 con 0x7f099815f5e0 2024-03-06T07:26:44.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.895+0000 7f09967fc640 1 -- 172.21.15.59:0/1918413249 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0988037070 con 0x7f099815f5e0 2024-03-06T07:26:44.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.895+0000 7f09967fc640 1 -- 172.21.15.59:0/1918413249 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988031a00 con 0x7f099815f5e0 2024-03-06T07:26:44.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.895+0000 7f099e1d0640 1 -- 172.21.15.59:0/1918413249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f099815f5e0 msgr2=0x7f09980b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.895+0000 7f099e1d0640 1 --2- 172.21.15.59:0/1918413249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f099815f5e0 0x7f09980b2bf0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f0998071dc0 tx=0x7f098802b880 comp rx=0 tx=0).stop 2024-03-06T07:26:44.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.896+0000 7f099e1d0640 1 -- 172.21.15.59:0/1918413249 shutdown_connections 2024-03-06T07:26:44.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.896+0000 7f099e1d0640 1 --2- 172.21.15.59:0/1918413249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09980d2870 0x7f09980cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.896+0000 7f099e1d0640 1 --2- 172.21.15.59:0/1918413249 >> v2:172.21.15.164:3300/0 conn(0x7f09980b16b0 0x7f09980ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.896+0000 7f099e1d0640 1 --2- 172.21.15.59:0/1918413249 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f099815f5e0 0x7f09980b2bf0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.896+0000 7f099e1d0640 1 -- 172.21.15.59:0/1918413249 >> 172.21.15.59:0/1918413249 conn(0x7f09980d1d50 msgr2=0x7f09980d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.896+0000 7f099e1d0640 1 -- 172.21.15.59:0/1918413249 shutdown_connections 2024-03-06T07:26:44.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.896+0000 7f099e1d0640 1 -- 172.21.15.59:0/1918413249 wait complete. 2024-03-06T07:26:44.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.897+0000 7f099e1d0640 1 Processor -- start 2024-03-06T07:26:44.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.897+0000 7f099e1d0640 1 -- start start 2024-03-06T07:26:44.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.898+0000 7f099e1d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980b16b0 0x7f099814e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.898+0000 7f099e1d0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09980d2870 0x7f0998159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.898+0000 7f099e1d0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f099815f5e0 0x7f0998159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.898+0000 7f099e1d0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09980cf270 con 0x7f09980b16b0 2024-03-06T07:26:44.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.898+0000 7f099e1d0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09980cf0f0 con 0x7f09980d2870 2024-03-06T07:26:44.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.898+0000 7f099e1d0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099815fcd0 con 0x7f099815f5e0 2024-03-06T07:26:44.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.898+0000 7f09977fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980b16b0 0x7f099814e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.898+0000 7f09977fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980b16b0 0x7f099814e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60778/0 (socket says 172.21.15.59:60778) 2024-03-06T07:26:44.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.898+0000 7f09977fe640 1 -- 172.21.15.59:0/3441675120 learned_addr learned my addr 172.21.15.59:0/3441675120 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f0996ffd640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09980d2870 0x7f0998159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f0997fff640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f099815f5e0 0x7f0998159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f09977fe640 1 -- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f099815f5e0 msgr2=0x7f0998159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f09977fe640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f099815f5e0 0x7f0998159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f09977fe640 1 -- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09980d2870 msgr2=0x7f0998159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f09977fe640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09980d2870 0x7f0998159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f09977fe640 1 -- 172.21.15.59:0/3441675120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0998158e60 con 0x7f09980b16b0 2024-03-06T07:26:44.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f0996ffd640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09980d2870 0x7f0998159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:44.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f09977fe640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980b16b0 0x7f099814e920 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0988000c00 tx=0x7f09880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f097ffff640 1 -- 172.21.15.59:0/3441675120 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988032670 con 0x7f09980b16b0 2024-03-06T07:26:44.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f097ffff640 1 -- 172.21.15.59:0/3441675120 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0988032810 con 0x7f09980b16b0 2024-03-06T07:26:44.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f097ffff640 1 -- 172.21.15.59:0/3441675120 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988032c00 con 0x7f09980b16b0 2024-03-06T07:26:44.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f099e1d0640 1 -- 172.21.15.59:0/3441675120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0998159140 con 0x7f09980b16b0 2024-03-06T07:26:44.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.899+0000 7f099e1d0640 1 -- 172.21.15.59:0/3441675120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099814ee60 con 0x7f09980b16b0 2024-03-06T07:26:44.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.900+0000 7f097dffb640 1 -- 172.21.15.59:0/3441675120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0970005290 con 0x7f09980b16b0 2024-03-06T07:26:44.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.901+0000 7f097ffff640 1 -- 172.21.15.59:0/3441675120 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f098804c060 con 0x7f09980b16b0 2024-03-06T07:26:44.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.902+0000 7f097ffff640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f098406b4d0 0x7f098406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.902+0000 7f097ffff640 1 -- 172.21.15.59:0/3441675120 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f09880abce0 con 0x7f09980b16b0 2024-03-06T07:26:44.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.902+0000 7f0996ffd640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f098406b4d0 0x7f098406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.903+0000 7f0996ffd640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f098406b4d0 0x7f098406d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f09980cefa0 tx=0x7f098c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:44.908+0000 7f097ffff640 1 -- 172.21.15.59:0/3441675120 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0988040070 con 0x7f09980b16b0 2024-03-06T07:26:45.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.229+0000 7f097dffb640 1 -- 172.21.15.59:0/3441675120 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0970005850 con 0x7f09980b16b0 2024-03-06T07:26:45.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.230+0000 7f097ffff640 1 -- 172.21.15.59:0/3441675120 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f098807aa60 con 0x7f09980b16b0 2024-03-06T07:26:45.232 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:45.232 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:45.233 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 -- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f098406b4d0 msgr2=0x7f098406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f098406b4d0 0x7f098406d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f09980cefa0 tx=0x7f098c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 -- 172.21.15.59:0/3441675120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980b16b0 msgr2=0x7f099814e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980b16b0 0x7f099814e920 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0988000c00 tx=0x7f09880046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 -- 172.21.15.59:0/3441675120 shutdown_connections 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f098406b4d0 0x7f098406d990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f099815f5e0 0x7f0998159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09980d2870 0x7f0998159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 --2- 172.21.15.59:0/3441675120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09980b16b0 0x7f099814e920 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 -- 172.21.15.59:0/3441675120 >> 172.21.15.59:0/3441675120 conn(0x7f09980d1d50 msgr2=0x7f09980b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 -- 172.21.15.59:0/3441675120 shutdown_connections 2024-03-06T07:26:45.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:45.233+0000 7f097dffb640 1 -- 172.21.15.59:0/3441675120 wait complete. 2024-03-06T07:26:45.249 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:45.401 INFO:teuthology.orchestra.run.smithi059.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":19,"msgr_send_messages":21,"msgr_recv_bytes":13523,"msgr_send_bytes":4222738,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.014050710,"msgr_running_send_time":0.007891027,"msgr_running_recv_time":0.003109998,"msgr_running_fast_dispatch_time":0.001945973,"msgr_send_messages_queue_lat":{"avgcount":19,"sum":0.004296726,"avgtime":0.000226143},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000225,"avgtime":0.000000112},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6359,"msgr_send_messages":13213,"msgr_recv_bytes":2229143,"msgr_send_bytes":12181798,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.610888412,"msgr_running_send_time":0.330565694,"msgr_running_recv_time":0.332387150,"msgr_running_fast_dispatch_time":0.004592802,"msgr_send_messages_queue_lat":{"avgcount":13211,"sum":0.167433003,"avgtime":0.000012673},"msgr_handle_ack_lat":{"avgcount":19091,"sum":0.003357315,"avgtime":0.000000175},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94016},"AsyncMessenger::Worker-2":{"msgr_recv_messages":51,"msgr_send_messages":45,"msgr_recv_bytes":120307,"msgr_send_bytes":3257875,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.013106821,"msgr_running_send_time":0.005840415,"msgr_running_recv_time":0.007012074,"msgr_running_fast_dispatch_time":0.002323910,"msgr_send_messages_queue_lat":{"avgcount":45,"sum":0.008752884,"avgtime":0.000194508},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108576,"msgr_send_encrypted_bytes":7744},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1310,"complete_latency":{"avgcount":77,"sum":1.787332146,"avgtime":0.023212105}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000150226,"avgtime":0.000037556}},"mds":{"request":3922,"reply":3920,"reply_latency":{"avgcount":3920,"sum":4.797010728,"avgtime":0.001223727},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3921,"inodes_top":0,"inodes_bottom":3921,"inodes_pin_tail":0,"inodes_pinned":3921,"inodes_expired":0,"inodes_with_caps":3194,"caps":3194,"subtrees":2,"traverse":3928,"traverse_hit":3927,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3200,"root_rbytes":6392832,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":542,"ceph_cap_op_grant":4025,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2186,"handle_client_caps_dirty":1939,"handle_client_cap_release":1,"process_request_cap_release":4621},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":710,"num_strays_delayed":8,"num_strays_enqueuing":0,"strays_created":710,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":6086,"evex":0,"evtrm":0,"ev":6089,"evexg":0,"evexd":0,"segadd":8,"segex":0,"segtrm":0,"seg":10,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":16974079,"rdpos":4195244,"jlat":{"avgcount":12,"sum":0.067710890,"avgtime":0.005642574},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3921,"dn+":3921,"dn-":0,"cap":3194,"cap+":3202,"cap-":8,"rss":96412,"heap":248164},"mds_server":{"dispatch_client_request":3925,"dispatch_server_request":0,"handle_client_request":3922,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.113519323,"avgtime":0.000972974},"req_getattr_latency":{"avgcount":6,"sum":0.001174100,"avgtime":0.000195683},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002442159,"avgtime":0.002442159},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.451304597,"avgtime":0.150434865},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":710,"sum":1.228570549,"avgtime":0.001730381},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3706,"average_load":3706,"avg_session_uptime":844,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":19823732,"buffer_anon_items":5402,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13985751,"mds_co_items":100248,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":69,"op_send_bytes":12936767,"op_resend":0,"op_reply":69,"op_latency":{"avgcount":69,"sum":0.173997805,"avgtime":0.002521707},"op_inflight":0,"oplen_avg":{"avgcount":69,"sum":126},"op":69,"op_r":36,"op_w":33,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":13,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":8,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":18,"omap_rd":48,"omap_del":3},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":54636,"max":104857600,"get_started":0,"get":6425,"get_sum":1782056,"get_or_fail_fail":0,"get_or_fail_success":6425,"take":0,"take_sum":0,"put":6215,"put_sum":1727420,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":69,"take_sum":12937884,"put":69,"put_sum":12937884,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":69,"take_sum":69,"put":69,"put_sum":69,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a5a2aa2820":{"val":198796,"max":3758096384,"get_started":0,"get":6087,"get_sum":12780327,"get_or_fail_fail":0,"get_or_fail_success":6087,"take":0,"take_sum":0,"put":13,"put_sum":12581531,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:45.403 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.12e-07, "sum": 2.25e-07 }, "msgr_recv_bytes": 13523, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 19, "msgr_running_fast_dispatch_time": 0.001945973, "msgr_running_recv_time": 0.003109998, "msgr_running_send_time": 0.007891027, "msgr_running_total_time": 0.01405071, "msgr_send_bytes": 4222738, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 21, "msgr_send_messages_queue_lat": { "avgcount": 19, "avgtime": 0.000226143, "sum": 0.004296726 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 19091, "avgtime": 1.75e-07, "sum": 0.003357315 }, "msgr_recv_bytes": 2229143, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 6359, "msgr_running_fast_dispatch_time": 0.004592802, "msgr_running_recv_time": 0.33238715, "msgr_running_send_time": 0.330565694, "msgr_running_total_time": 0.610888412, "msgr_send_bytes": 12181798, "msgr_send_encrypted_bytes": 94016, "msgr_send_messages": 13213, "msgr_send_messages_queue_lat": { "avgcount": 13211, "avgtime": 1.2673e-05, "sum": 0.167433003 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 120307, "msgr_recv_encrypted_bytes": 108576, "msgr_recv_messages": 51, "msgr_running_fast_dispatch_time": 0.00232391, "msgr_running_recv_time": 0.007012074, "msgr_running_send_time": 0.005840415, "msgr_running_total_time": 0.013106821, "msgr_send_bytes": 3257875, "msgr_send_encrypted_bytes": 7744, "msgr_send_messages": 45, "msgr_send_messages_queue_lat": { "avgcount": 45, "avgtime": 0.000194508, "sum": 0.008752884 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 77, "avgtime": 0.023212105, "sum": 1.787332146 }, "queue_len": 1310 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.7556e-05, "sum": 0.000150226 }, "queue_len": 0 }, "mds": { "caps": 3194, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4025, "ceph_cap_op_revoke": 542, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 1, "handle_client_caps": 2186, "handle_client_caps_dirty": 1939, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3921, "inodes_bottom": 3921, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3921, "inodes_top": 0, "inodes_with_caps": 3194, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 4621, "q": 0, "reply": 3920, "reply_latency": { "avgcount": 3920, "avgtime": 0.001223727, "sum": 4.797010728 }, "request": 3922, "root_rbytes": 6392832, "root_rfiles": 3200, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3928, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3927, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 710, "num_strays_delayed": 8, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 710, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6089, "evadd": 6086, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 12, "avgtime": 0.005642574, "sum": 0.06771089 }, "rdpos": 4195244, "replayed": 3, "seg": 10, "segadd": 8, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 16974079 }, "mds_mem": { "cap": 3194, "cap+": 3202, "cap-": 8, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3921, "dn+": 3921, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 96412 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3925, "dispatch_server_request": 0, "handle_client_request": 3922, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000972974, "sum": 3.113519323 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195683, "sum": 0.0011741 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002442159, "sum": 0.002442159 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.150434865, "sum": 0.451304597 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 710, "avgtime": 0.001730381, "sum": 1.228570549 } }, "mds_sessions": { "average_load": 3706, "avg_session_uptime": 844, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3706 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 19823732, "buffer_anon_items": 5402, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 13985751, "mds_co_items": 100248, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 3, "omap_rd": 48, "omap_wr": 18, "op": 69, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 69, "avgtime": 0.002521707, "sum": 0.173997805 }, "op_pg": 0, "op_r": 36, "op_reply": 69, "op_resend": 0, "op_rmw": 0, "op_send": 69, "op_send_bytes": 12936767, "op_w": 33, "oplen_avg": { "avgcount": 69, "sum": 126 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 8, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 13, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6425, "get_or_fail_fail": 0, "get_or_fail_success": 6425, "get_started": 0, "get_sum": 1782056, "max": 104857600, "put": 6215, "put_sum": 1727420, "take": 0, "take_sum": 0, "val": 54636, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 69, "put_sum": 12937884, "take": 69, "take_sum": 12937884, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 69, "put_sum": 69, "take": 69, "take_sum": 69, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a5a2aa2820": { "get": 6087, "get_or_fail_fail": 0, "get_or_fail_success": 6087, "get_started": 0, "get_sum": 12780327, "max": 3758096384, "put": 13, "put_sum": 12581531, "take": 0, "take_sum": 0, "val": 198796, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:45.403 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/710, 0/710 2024-03-06T07:26:46.404 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:46.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.517+0000 7f38a759e640 1 Processor -- start 2024-03-06T07:26:46.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.517+0000 7f38a759e640 1 -- start start 2024-03-06T07:26:46.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.518+0000 7f38a759e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f38a8162320 0x7f38a8168f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.518+0000 7f38a759e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a815e520 con 0x7f38a80d2b20 2024-03-06T07:26:46.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.518+0000 7f38a759e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a80cec50 con 0x7f38a815e140 2024-03-06T07:26:46.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.518+0000 7f38a759e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a80cf860 con 0x7f38a8162320 2024-03-06T07:26:46.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f389dd9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f38a80d2b20 0x7f38a8161c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39786/0 (socket says 172.21.15.59:39786) 2024-03-06T07:26:46.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f389dd9b640 1 -- 172.21.15.59:0/2736726436 learned_addr learned my addr 172.21.15.59:0/2736726436 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:46.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a6d9d640 1 --2- 172.21.15.59:0/2736726436 >> v2:172.21.15.164:3300/0 conn(0x7f38a8162320 0x7f38a8168f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473591979 0 0) 0x7f38a815e520 con 0x7f38a80d2b20 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3894003520 con 0x7f38a80d2b20 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3216198275 0 0) 0x7f3894003520 con 0x7f38a80d2b20 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 >> v2:172.21.15.164:3300/0 conn(0x7f38a8162320 msgr2=0x7f38a8168f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 --2- 172.21.15.59:0/2736726436 >> v2:172.21.15.164:3300/0 conn(0x7f38a8162320 0x7f38a8168f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 >> v1:172.21.15.164:6789/0 conn(0x7f38a815e140 legacy=0x7f38a80d2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a80cef30 con 0x7f38a80d2b20 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f3890001670 con 0x7f38a80d2b20 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 >> v1:172.21.15.59:6789/0 conn(0x7f38a80d2b20 legacy=0x7f38a8161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 --2- 172.21.15.59:0/2736726436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 0x7f3894003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 --2- 172.21.15.59:0/2736726436 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3894004800 0x7f3894004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 --2- 172.21.15.59:0/2736726436 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3894005210 0x7f389400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80cf860 con 0x7f3894003a90 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389400c140 con 0x7f3894005210 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389400c340 con 0x7f3894004800 2024-03-06T07:26:46.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.519+0000 7f38a659c640 1 --2- 172.21.15.59:0/2736726436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 0x7f3894003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.520+0000 7f38a6d9d640 1 --2- 172.21.15.59:0/2736726436 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3894005210 0x7f389400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.520+0000 7f389dd9b640 1 --2- 172.21.15.59:0/2736726436 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3894004800 0x7f3894004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.520+0000 7f38a659c640 1 -- 172.21.15.59:0/2736726436 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3894004800 msgr2=0x7f3894004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.520+0000 7f38a659c640 1 --2- 172.21.15.59:0/2736726436 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3894004800 0x7f3894004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.520+0000 7f38a659c640 1 -- 172.21.15.59:0/2736726436 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3894005210 msgr2=0x7f389400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.520+0000 7f38a659c640 1 --2- 172.21.15.59:0/2736726436 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3894005210 0x7f389400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.520+0000 7f38a659c640 1 -- 172.21.15.59:0/2736726436 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3894003520 con 0x7f3894003a90 2024-03-06T07:26:46.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.520+0000 7f38a6d9d640 1 -- 172.21.15.59:0/2736726436 reap_dead start 2024-03-06T07:26:46.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.521+0000 7f38a659c640 1 --2- 172.21.15.59:0/2736726436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 0x7f3894003eb0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f3898001720 tx=0x7f3894005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b73e4f5bdab7584 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.521+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3898001370 con 0x7f3894003a90 2024-03-06T07:26:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.521+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38980052d0 con 0x7f3894003a90 2024-03-06T07:26:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.521+0000 7f38a5d9b640 1 -- 172.21.15.59:0/2736726436 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3898003830 con 0x7f3894003a90 2024-03-06T07:26:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.521+0000 7f38a759e640 1 -- 172.21.15.59:0/2736726436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 msgr2=0x7f3894003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.521+0000 7f38a759e640 1 --2- 172.21.15.59:0/2736726436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 0x7f3894003eb0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f3898001720 tx=0x7f3894005830 comp rx=0 tx=0).stop 2024-03-06T07:26:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.521+0000 7f38a759e640 1 -- 172.21.15.59:0/2736726436 shutdown_connections 2024-03-06T07:26:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.522+0000 7f38a759e640 1 --2- 172.21.15.59:0/2736726436 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 0x7f3894003eb0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.522+0000 7f38a759e640 1 -- 172.21.15.59:0/2736726436 >> 172.21.15.59:0/2736726436 conn(0x7f38a809d110 msgr2=0x7f38a80d3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.522+0000 7f38a759e640 1 -- 172.21.15.59:0/2736726436 shutdown_connections 2024-03-06T07:26:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.522+0000 7f38a759e640 1 -- 172.21.15.59:0/2736726436 wait complete. 2024-03-06T07:26:46.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.523+0000 7f38a759e640 1 Processor -- start 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.523+0000 7f38a759e640 1 -- start start 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 0x7f38a815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38a80c7890 0x7f38a8146de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a759e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38a80c7d10 0x7f38a8147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a759e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80ce920 con 0x7f3894003a90 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a759e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80cf4f0 con 0x7f38a80c7d10 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a759e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80cfcf0 con 0x7f38a80c7890 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 0x7f38a815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 0x7f38a815c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60796/0 (socket says 172.21.15.59:60796) 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a659c640 1 -- 172.21.15.59:0/4127966815 learned_addr learned my addr 172.21.15.59:0/4127966815 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a6d9d640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38a80c7d10 0x7f38a8147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f389dd9b640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38a80c7890 0x7f38a8146de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a659c640 1 -- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38a80c7890 msgr2=0x7f38a8146de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a659c640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38a80c7890 0x7f38a8146de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a659c640 1 -- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38a80c7d10 msgr2=0x7f38a8147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a659c640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38a80c7d10 0x7f38a8147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a659c640 1 -- 172.21.15.59:0/4127966815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a80c9930 con 0x7f3894003a90 2024-03-06T07:26:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.524+0000 7f38a6d9d640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38a80c7d10 0x7f38a8147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.525+0000 7f38a659c640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 0x7f38a815c8e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f3898000c00 tx=0x7f3898001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.525+0000 7f389f7fe640 1 -- 172.21.15.59:0/4127966815 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3898041020 con 0x7f3894003a90 2024-03-06T07:26:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.525+0000 7f389f7fe640 1 -- 172.21.15.59:0/4127966815 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3898005480 con 0x7f3894003a90 2024-03-06T07:26:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.525+0000 7f38a759e640 1 -- 172.21.15.59:0/4127966815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a80c9c10 con 0x7f3894003a90 2024-03-06T07:26:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.525+0000 7f389f7fe640 1 -- 172.21.15.59:0/4127966815 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3898007a30 con 0x7f3894003a90 2024-03-06T07:26:46.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.525+0000 7f38a759e640 1 -- 172.21.15.59:0/4127966815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a80ca6f0 con 0x7f3894003a90 2024-03-06T07:26:46.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.526+0000 7f38a759e640 1 -- 172.21.15.59:0/4127966815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a804f330 con 0x7f3894003a90 2024-03-06T07:26:46.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.527+0000 7f389f7fe640 1 -- 172.21.15.59:0/4127966815 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3898044020 con 0x7f3894003a90 2024-03-06T07:26:46.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.527+0000 7f389f7fe640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388806b4d0 0x7f388806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.527+0000 7f389f7fe640 1 -- 172.21.15.59:0/4127966815 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f38980abef0 con 0x7f3894003a90 2024-03-06T07:26:46.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.527+0000 7f389dd9b640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388806b4d0 0x7f388806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.528+0000 7f389dd9b640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388806b4d0 0x7f388806d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3890001920 tx=0x7f3890000ed0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.532+0000 7f389f7fe640 1 -- 172.21.15.59:0/4127966815 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f389804b030 con 0x7f3894003a90 2024-03-06T07:26:46.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.836+0000 7f38a759e640 1 -- 172.21.15.59:0/4127966815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38a806c6c0 con 0x7f3894003a90 2024-03-06T07:26:46.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.837+0000 7f389f7fe640 1 -- 172.21.15.59:0/4127966815 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f38980303c0 con 0x7f3894003a90 2024-03-06T07:26:46.839 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:46.839 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:46.839 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:46.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 -- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388806b4d0 msgr2=0x7f388806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388806b4d0 0x7f388806d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3890001920 tx=0x7f3890000ed0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 -- 172.21.15.59:0/4127966815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 msgr2=0x7f38a815c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 0x7f38a815c8e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f3898000c00 tx=0x7f3898001120 comp rx=0 tx=0).stop 2024-03-06T07:26:46.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 -- 172.21.15.59:0/4127966815 shutdown_connections 2024-03-06T07:26:46.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388806b4d0 0x7f388806d990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38a80c7d10 0x7f38a8147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38a80c7890 0x7f38a8146de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 --2- 172.21.15.59:0/4127966815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3894003a90 0x7f38a815c8e0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 -- 172.21.15.59:0/4127966815 >> 172.21.15.59:0/4127966815 conn(0x7f38a809d110 msgr2=0x7f38a80b23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 -- 172.21.15.59:0/4127966815 shutdown_connections 2024-03-06T07:26:46.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:46.840+0000 7f38a759e640 1 -- 172.21.15.59:0/4127966815 wait complete. 2024-03-06T07:26:46.855 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:47.002 INFO:teuthology.orchestra.run.smithi059.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":27,"msgr_send_messages":29,"msgr_recv_bytes":15547,"msgr_send_bytes":4225338,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.015023316,"msgr_running_send_time":0.008567010,"msgr_running_recv_time":0.003206429,"msgr_running_fast_dispatch_time":0.002126433,"msgr_send_messages_queue_lat":{"avgcount":27,"sum":0.007011903,"avgtime":0.000259700},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000225,"avgtime":0.000000112},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7518,"msgr_send_messages":16390,"msgr_recv_bytes":2660033,"msgr_send_bytes":15016423,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.737091956,"msgr_running_send_time":0.403919503,"msgr_running_recv_time":0.391504184,"msgr_running_fast_dispatch_time":0.005716421,"msgr_send_messages_queue_lat":{"avgcount":16388,"sum":0.225196204,"avgtime":0.000013741},"msgr_handle_ack_lat":{"avgcount":22814,"sum":0.004070428,"avgtime":0.000000178},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94016},"AsyncMessenger::Worker-2":{"msgr_recv_messages":55,"msgr_send_messages":49,"msgr_recv_bytes":121299,"msgr_send_bytes":3259335,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.013363122,"msgr_running_send_time":0.005923087,"msgr_running_recv_time":0.007073606,"msgr_running_fast_dispatch_time":0.002418441,"msgr_send_messages_queue_lat":{"avgcount":49,"sum":0.008814128,"avgtime":0.000179880},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108576,"msgr_send_encrypted_bytes":7744},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1142,"complete_latency":{"avgcount":80,"sum":3.133209113,"avgtime":0.039165113}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":24,"sum":0.006203276,"avgtime":0.000258469}},"mds":{"request":4460,"reply":4458,"reply_latency":{"avgcount":4458,"sum":6.213598580,"avgtime":0.001393808},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4458,"inodes_top":0,"inodes_bottom":4458,"inodes_pin_tail":0,"inodes_pinned":4458,"inodes_expired":0,"inodes_with_caps":2962,"caps":2962,"subtrees":2,"traverse":4466,"traverse_hit":4465,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2903,"root_rbytes":9842688,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":788,"ceph_cap_op_grant":4903,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3008,"handle_client_caps_dirty":2627,"handle_client_cap_release":2,"process_request_cap_release":5697},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1240,"num_strays_delayed":232,"num_strays_enqueuing":0,"strays_created":1248,"strays_enqueued":8,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7457,"evex":0,"evtrm":0,"ev":7460,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":11,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":20296215,"rdpos":4195244,"jlat":{"avgcount":14,"sum":0.871786791,"avgtime":0.062270485},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4459,"dn+":4459,"dn-":0,"cap":2962,"cap+":3202,"cap-":240,"rss":96412,"heap":248164},"mds_server":{"dispatch_client_request":4463,"dispatch_server_request":0,"handle_client_request":4460,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.113519323,"avgtime":0.000972974},"req_getattr_latency":{"avgcount":6,"sum":0.001174100,"avgtime":0.000195683},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002442159,"avgtime":0.002442159},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.451304597,"avgtime":0.150434865},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1248,"sum":2.645158401,"avgtime":0.002119517},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4164,"average_load":4164,"avg_session_uptime":844,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":19823732,"buffer_anon_items":5402,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13985751,"mds_co_items":100248,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":88,"op_send_bytes":14667777,"op_resend":0,"op_reply":88,"op_latency":{"avgcount":88,"sum":0.223997177,"avgtime":0.002545422},"op_inflight":0,"oplen_avg":{"avgcount":88,"sum":145},"op":88,"op_r":37,"op_w":51,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":9,"osdop_write":15,"osdop_writefull":3,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":22,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":18,"omap_rd":48,"omap_del":3},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":8,"pq_executed":8,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":297,"max":104857600,"get_started":0,"get":7597,"get_sum":2110869,"get_or_fail_fail":0,"get_or_fail_success":7597,"take":0,"take_sum":0,"put":7596,"put_sum":2110572,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":88,"take_sum":14669702,"put":88,"put_sum":14669702,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":88,"take_sum":88,"put":88,"put_sum":88,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":8,"get_sum":808,"get_or_fail_fail":0,"get_or_fail_success":8,"take":0,"take_sum":0,"put":1,"put_sum":808,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a5a2aa2820":{"val":1788977,"max":3758096384,"get_started":0,"get":7457,"get_sum":16100530,"get_or_fail_fail":0,"get_or_fail_success":7457,"take":0,"take_sum":0,"put":14,"put_sum":14311553,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:47.004 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.12e-07, "sum": 2.25e-07 }, "msgr_recv_bytes": 15547, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 27, "msgr_running_fast_dispatch_time": 0.002126433, "msgr_running_recv_time": 0.003206429, "msgr_running_send_time": 0.00856701, "msgr_running_total_time": 0.015023316, "msgr_send_bytes": 4225338, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 29, "msgr_send_messages_queue_lat": { "avgcount": 27, "avgtime": 0.0002597, "sum": 0.007011903 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 22814, "avgtime": 1.78e-07, "sum": 0.004070428 }, "msgr_recv_bytes": 2660033, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 7518, "msgr_running_fast_dispatch_time": 0.005716421, "msgr_running_recv_time": 0.391504184, "msgr_running_send_time": 0.403919503, "msgr_running_total_time": 0.737091956, "msgr_send_bytes": 15016423, "msgr_send_encrypted_bytes": 94016, "msgr_send_messages": 16390, "msgr_send_messages_queue_lat": { "avgcount": 16388, "avgtime": 1.3741e-05, "sum": 0.225196204 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 121299, "msgr_recv_encrypted_bytes": 108576, "msgr_recv_messages": 55, "msgr_running_fast_dispatch_time": 0.002418441, "msgr_running_recv_time": 0.007073606, "msgr_running_send_time": 0.005923087, "msgr_running_total_time": 0.013363122, "msgr_send_bytes": 3259335, "msgr_send_encrypted_bytes": 7744, "msgr_send_messages": 49, "msgr_send_messages_queue_lat": { "avgcount": 49, "avgtime": 0.00017988, "sum": 0.008814128 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 80, "avgtime": 0.039165113, "sum": 3.133209113 }, "queue_len": 1142 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 24, "avgtime": 0.000258469, "sum": 0.006203276 }, "queue_len": 0 }, "mds": { "caps": 2962, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4903, "ceph_cap_op_revoke": 788, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 2, "handle_client_caps": 3008, "handle_client_caps_dirty": 2627, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4458, "inodes_bottom": 4458, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4458, "inodes_top": 0, "inodes_with_caps": 2962, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 5697, "q": 0, "reply": 4458, "reply_latency": { "avgcount": 4458, "avgtime": 0.001393808, "sum": 6.21359858 }, "request": 4460, "root_rbytes": 9842688, "root_rfiles": 2903, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4466, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4465, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1240, "num_strays_delayed": 232, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1248, "strays_enqueued": 8, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7460, "evadd": 7457, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 14, "avgtime": 0.062270485, "sum": 0.871786791 }, "rdpos": 4195244, "replayed": 3, "seg": 11, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 20296215 }, "mds_mem": { "cap": 2962, "cap+": 3202, "cap-": 240, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4459, "dn+": 4459, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 96412 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4463, "dispatch_server_request": 0, "handle_client_request": 4460, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000972974, "sum": 3.113519323 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195683, "sum": 0.0011741 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002442159, "sum": 0.002442159 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.150434865, "sum": 0.451304597 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1248, "avgtime": 0.002119517, "sum": 2.645158401 } }, "mds_sessions": { "average_load": 4164, "avg_session_uptime": 844, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4164 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 19823732, "buffer_anon_items": 5402, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 13985751, "mds_co_items": 100248, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 3, "omap_rd": 48, "omap_wr": 18, "op": 88, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 88, "avgtime": 0.002545422, "sum": 0.223997177 }, "op_pg": 0, "op_r": 37, "op_reply": 88, "op_resend": 0, "op_rmw": 0, "op_send": 88, "op_send_bytes": 14667777, "op_w": 51, "oplen_avg": { "avgcount": 88, "sum": 145 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 22, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 9, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 15, "osdop_writefull": 3, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 8, "pq_executed_ops": 8, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7597, "get_or_fail_fail": 0, "get_or_fail_success": 7597, "get_started": 0, "get_sum": 2110869, "max": 104857600, "put": 7596, "put_sum": 2110572, "take": 0, "take_sum": 0, "val": 297, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 88, "put_sum": 14669702, "take": 88, "take_sum": 14669702, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 88, "put_sum": 88, "take": 88, "take_sum": 88, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 8, "get_or_fail_fail": 0, "get_or_fail_success": 8, "get_started": 0, "get_sum": 808, "max": 3758096384, "put": 1, "put_sum": 808, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a5a2aa2820": { "get": 7457, "get_or_fail_fail": 0, "get_or_fail_success": 7457, "get_started": 0, "get_sum": 16100530, "max": 3758096384, "put": 14, "put_sum": 14311553, "take": 0, "take_sum": 0, "val": 1788977, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:47.004 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1240, 8/1248 2024-03-06T07:26:48.005 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:48.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.134+0000 7fdbf7df7640 1 Processor -- start 2024-03-06T07:26:48.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.134+0000 7fdbf7df7640 1 -- start start 2024-03-06T07:26:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.135+0000 7fdbf7df7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdbf00ad8b0 0x7fdbf006c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.135+0000 7fdbf7df7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf006e240 con 0x7fdbf00b3bf0 2024-03-06T07:26:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.135+0000 7fdbf7df7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf006e520 con 0x7fdbf015e150 2024-03-06T07:26:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.135+0000 7fdbf7df7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf006b670 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.135+0000 7fdbf6394640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdbf00ad8b0 0x7fdbf006c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.135+0000 7fdbf5392640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdbf00b3bf0 0x7fdbf006bb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39790/0 (socket says 172.21.15.59:39790) 2024-03-06T07:26:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.135+0000 7fdbf5392640 1 -- 172.21.15.59:0/3659009202 learned_addr learned my addr 172.21.15.59:0/3659009202 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:48.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.136+0000 7fdbf6394640 1 -- 172.21.15.59:0/3659009202 >> v1:172.21.15.164:6789/0 conn(0x7fdbf015e150 legacy=0x7fdbf00b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.136+0000 7fdbf6394640 1 -- 172.21.15.59:0/3659009202 >> v1:172.21.15.59:6789/0 conn(0x7fdbf00b3bf0 legacy=0x7fdbf006bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.136+0000 7fdbf4b91640 1 -- 172.21.15.59:0/3659009202 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044039431 0 0) 0x7fdbf006e240 con 0x7fdbf00b3bf0 2024-03-06T07:26:48.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.136+0000 7fdbf6394640 1 -- 172.21.15.59:0/3659009202 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf006b020 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.137+0000 7fdbf6394640 1 --2- 172.21.15.59:0/3659009202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbf00ad8b0 0x7fdbf006c210 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009d90 tx=0x7fdbe802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9ae418c7808249 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.137+0000 7fdbf4b91640 1 -- 172.21.15.59:0/3659009202 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8039070 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.137+0000 7fdbf4b91640 1 -- 172.21.15.59:0/3659009202 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe802cd00 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.137+0000 7fdbf4b91640 1 -- 172.21.15.59:0/3659009202 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8034660 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.138+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3659009202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbf00ad8b0 msgr2=0x7fdbf006c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.138+0000 7fdbf7df7640 1 --2- 172.21.15.59:0/3659009202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbf00ad8b0 0x7fdbf006c210 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009d90 tx=0x7fdbe802bf50 comp rx=0 tx=0).stop 2024-03-06T07:26:48.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.138+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3659009202 shutdown_connections 2024-03-06T07:26:48.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.138+0000 7fdbf7df7640 1 --2- 172.21.15.59:0/3659009202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbf00ad8b0 0x7fdbf006c210 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.138+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3659009202 >> 172.21.15.59:0/3659009202 conn(0x7fdbf00d0470 msgr2=0x7fdbf00d0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.138+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3659009202 shutdown_connections 2024-03-06T07:26:48.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.138+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3659009202 wait complete. 2024-03-06T07:26:48.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.139+0000 7fdbf7df7640 1 Processor -- start 2024-03-06T07:26:48.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.139+0000 7fdbf7df7640 1 -- start start 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.140+0000 7fdbf7df7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbf00ad8b0 0x7fdbf014af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.140+0000 7fdbf7df7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbf00b3bf0 0x7fdbf015ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.140+0000 7fdbf7df7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbf015e150 0x7fdbf00cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.140+0000 7fdbf7df7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf006c750 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.140+0000 7fdbf7df7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf006df10 con 0x7fdbf00b3bf0 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.140+0000 7fdbf7df7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf006c8d0 con 0x7fdbf015e150 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.140+0000 7fdbf5b93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbf00ad8b0 0x7fdbf014af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbf5b93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbf00ad8b0 0x7fdbf014af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60814/0 (socket says 172.21.15.59:60814) 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbf5b93640 1 -- 172.21.15.59:0/3608079348 learned_addr learned my addr 172.21.15.59:0/3608079348 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbf5392640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbf00b3bf0 0x7fdbf015ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbf6394640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbf015e150 0x7fdbf00cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbf5b93640 1 -- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbf015e150 msgr2=0x7fdbf00cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbf5b93640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbf015e150 0x7fdbf00cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbf5b93640 1 -- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbf00b3bf0 msgr2=0x7fdbf015ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbf5b93640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbf00b3bf0 0x7fdbf015ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbf5b93640 1 -- 172.21.15.59:0/3608079348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf015e780 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbf5b93640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbf00ad8b0 0x7fdbf014af70 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00c430 tx=0x7fdbec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbe67fc640 1 -- 172.21.15.59:0/3608079348 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec017070 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbe67fc640 1 -- 172.21.15.59:0/3608079348 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbec0052d0 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbe67fc640 1 -- 172.21.15.59:0/3608079348 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec003830 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.141+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3608079348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf015ea60 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.142+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3608079348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf015f020 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.143+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3608079348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf0068050 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.143+0000 7fdbe67fc640 1 -- 172.21.15.59:0/3608079348 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbec01a070 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.144+0000 7fdbe67fc640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd406b520 0x7fdbd406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.144+0000 7fdbe67fc640 1 -- 172.21.15.59:0/3608079348 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fdbec089630 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.144+0000 7fdbf5392640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd406b520 0x7fdbd406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.145+0000 7fdbf5392640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd406b520 0x7fdbd406d9e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0007fd0 tx=0x7fdbe0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.149+0000 7fdbe67fc640 1 -- 172.21.15.59:0/3608079348 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbec058420 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.520+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3608079348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbf00b3570 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.520+0000 7fdbe67fc640 1 -- 172.21.15.59:0/3608079348 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fdbec057d20 con 0x7fdbf00ad8b0 2024-03-06T07:26:48.522 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:48.523 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:48.523 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:48.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd406b520 msgr2=0x7fdbd406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd406b520 0x7fdbd406d9e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0007fd0 tx=0x7fdbe0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3608079348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbf00ad8b0 msgr2=0x7fdbf014af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbf00ad8b0 0x7fdbf014af70 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00c430 tx=0x7fdbec00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3608079348 shutdown_connections 2024-03-06T07:26:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdbd406b520 0x7fdbd406d9e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbf015e150 0x7fdbf00cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbf00b3bf0 0x7fdbf015ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 --2- 172.21.15.59:0/3608079348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdbf00ad8b0 0x7fdbf014af70 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3608079348 >> 172.21.15.59:0/3608079348 conn(0x7fdbf00d0470 msgr2=0x7fdbf00b1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3608079348 shutdown_connections 2024-03-06T07:26:48.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:48.523+0000 7fdbf7df7640 1 -- 172.21.15.59:0/3608079348 wait complete. 2024-03-06T07:26:48.539 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:48.665 INFO:teuthology.orchestra.run.smithi059.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":298,"msgr_send_messages":300,"msgr_recv_bytes":84373,"msgr_send_bytes":4313676,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.055331116,"msgr_running_send_time":0.033092222,"msgr_running_recv_time":0.009497631,"msgr_running_fast_dispatch_time":0.009488727,"msgr_send_messages_queue_lat":{"avgcount":298,"sum":0.025135416,"avgtime":0.000084347},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000225,"avgtime":0.000000112},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8791,"msgr_send_messages":19120,"msgr_recv_bytes":3191600,"msgr_send_bytes":18531738,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.870102845,"msgr_running_send_time":0.479154626,"msgr_running_recv_time":0.470778023,"msgr_running_fast_dispatch_time":0.011715103,"msgr_send_messages_queue_lat":{"avgcount":19118,"sum":0.278154147,"avgtime":0.000014549},"msgr_handle_ack_lat":{"avgcount":25936,"sum":0.004654456,"avgtime":0.000000179},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94016},"AsyncMessenger::Worker-2":{"msgr_recv_messages":231,"msgr_send_messages":225,"msgr_recv_bytes":165979,"msgr_send_bytes":3316957,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.030683502,"msgr_running_send_time":0.010194605,"msgr_running_recv_time":0.011502580,"msgr_running_fast_dispatch_time":0.008458743,"msgr_send_messages_queue_lat":{"avgcount":225,"sum":0.024914429,"avgtime":0.000110730},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108576,"msgr_send_encrypted_bytes":7744},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":87,"sum":5.336962368,"avgtime":0.061344395}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":951,"sum":0.432660606,"avgtime":0.000454953}},"mds":{"request":5132,"reply":5129,"reply_latency":{"avgcount":5129,"sum":7.466032346,"avgtime":0.001455650},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4738,"inodes_top":0,"inodes_bottom":4738,"inodes_pin_tail":0,"inodes_pinned":4738,"inodes_expired":0,"inodes_with_caps":2225,"caps":2225,"subtrees":2,"traverse":5138,"traverse_hit":5137,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1829,"root_rbytes":11228160,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":886,"ceph_cap_op_grant":5674,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3456,"handle_client_caps_dirty":3000,"handle_client_cap_release":4,"process_request_cap_release":7041},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1335,"num_strays_delayed":393,"num_strays_enqueuing":0,"strays_created":1919,"strays_enqueued":584,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":9155,"evex":0,"evtrm":0,"ev":9158,"evexg":0,"evexd":0,"segadd":11,"segex":0,"segtrm":0,"seg":13,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":23651013,"rdpos":4195244,"jlat":{"avgcount":16,"sum":1.837150956,"avgtime":0.114821934},"replayed":3},"mds_mem":{"ino":2974,"ino+":3214,"ino-":240,"dir":13,"dir+":13,"dir-":0,"dn":4890,"dn+":5130,"dn-":240,"cap":2225,"cap+":3202,"cap-":977,"rss":96412,"heap":248164},"mds_server":{"dispatch_client_request":5135,"dispatch_server_request":0,"handle_client_request":5132,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.113519323,"avgtime":0.000972974},"req_getattr_latency":{"avgcount":6,"sum":0.001174100,"avgtime":0.000195683},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002442159,"avgtime":0.002442159},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.451304597,"avgtime":0.150434865},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1919,"sum":3.897592167,"avgtime":0.002031053},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4750,"average_load":4750,"avg_session_uptime":844,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":19823732,"buffer_anon_items":5402,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13985751,"mds_co_items":100248,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":684,"op_send_bytes":17211734,"op_resend":0,"op_reply":684,"op_latency":{"avgcount":684,"sum":1.283983846,"avgtime":0.001877169},"op_inflight":0,"oplen_avg":{"avgcount":684,"sum":745},"op":684,"op_r":43,"op_w":641,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":15,"osdop_write":23,"osdop_writefull":6,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":599,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":22,"omap_rd":48,"omap_del":5},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":584,"pq_executed":584,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":297,"max":104857600,"get_started":0,"get":9316,"get_sum":2600845,"get_or_fail_fail":0,"get_or_fail_success":9316,"take":0,"take_sum":0,"put":9315,"put_sum":2600548,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":684,"take_sum":17271835,"put":684,"put_sum":17271835,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":684,"take_sum":684,"put":684,"put_sum":684,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":584,"get_sum":58984,"get_or_fail_fail":0,"get_or_fail_success":584,"take":0,"take_sum":0,"put":7,"put_sum":58984,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a5a2aa2820":{"val":2679493,"max":3758096384,"get_started":0,"get":9155,"get_sum":19455328,"get_or_fail_fail":0,"get_or_fail_success":9155,"take":0,"take_sum":0,"put":16,"put_sum":16775835,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:48.667 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.12e-07, "sum": 2.25e-07 }, "msgr_recv_bytes": 84373, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 298, "msgr_running_fast_dispatch_time": 0.009488727, "msgr_running_recv_time": 0.009497631, "msgr_running_send_time": 0.033092222, "msgr_running_total_time": 0.055331116, "msgr_send_bytes": 4313676, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 300, "msgr_send_messages_queue_lat": { "avgcount": 298, "avgtime": 8.4347e-05, "sum": 0.025135416 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 25936, "avgtime": 1.79e-07, "sum": 0.004654456 }, "msgr_recv_bytes": 3191600, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 8791, "msgr_running_fast_dispatch_time": 0.011715103, "msgr_running_recv_time": 0.470778023, "msgr_running_send_time": 0.479154626, "msgr_running_total_time": 0.870102845, "msgr_send_bytes": 18531738, "msgr_send_encrypted_bytes": 94016, "msgr_send_messages": 19120, "msgr_send_messages_queue_lat": { "avgcount": 19118, "avgtime": 1.4549e-05, "sum": 0.278154147 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 165979, "msgr_recv_encrypted_bytes": 108576, "msgr_recv_messages": 231, "msgr_running_fast_dispatch_time": 0.008458743, "msgr_running_recv_time": 0.01150258, "msgr_running_send_time": 0.010194605, "msgr_running_total_time": 0.030683502, "msgr_send_bytes": 3316957, "msgr_send_encrypted_bytes": 7744, "msgr_send_messages": 225, "msgr_send_messages_queue_lat": { "avgcount": 225, "avgtime": 0.00011073, "sum": 0.024914429 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 87, "avgtime": 0.061344395, "sum": 5.336962368 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 951, "avgtime": 0.000454953, "sum": 0.432660606 }, "queue_len": 0 }, "mds": { "caps": 2225, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5674, "ceph_cap_op_revoke": 886, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 4, "handle_client_caps": 3456, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4738, "inodes_bottom": 4738, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4738, "inodes_top": 0, "inodes_with_caps": 2225, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 7041, "q": 0, "reply": 5129, "reply_latency": { "avgcount": 5129, "avgtime": 0.00145565, "sum": 7.466032346 }, "request": 5132, "root_rbytes": 11228160, "root_rfiles": 1829, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5138, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5137, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1335, "num_strays_delayed": 393, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1919, "strays_enqueued": 584, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9158, "evadd": 9155, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 16, "avgtime": 0.114821934, "sum": 1.837150956 }, "rdpos": 4195244, "replayed": 3, "seg": 13, "segadd": 11, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 23651013 }, "mds_mem": { "cap": 2225, "cap+": 3202, "cap-": 977, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4890, "dn+": 5130, "dn-": 240, "heap": 248164, "ino": 2974, "ino+": 3214, "ino-": 240, "rss": 96412 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5135, "dispatch_server_request": 0, "handle_client_request": 5132, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000972974, "sum": 3.113519323 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195683, "sum": 0.0011741 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002442159, "sum": 0.002442159 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.150434865, "sum": 0.451304597 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1919, "avgtime": 0.002031053, "sum": 3.897592167 } }, "mds_sessions": { "average_load": 4750, "avg_session_uptime": 844, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4750 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 19823732, "buffer_anon_items": 5402, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 13985751, "mds_co_items": 100248, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 5, "omap_rd": 48, "omap_wr": 22, "op": 684, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 684, "avgtime": 0.001877169, "sum": 1.283983846 }, "op_pg": 0, "op_r": 43, "op_reply": 684, "op_resend": 0, "op_rmw": 0, "op_send": 684, "op_send_bytes": 17211734, "op_w": 641, "oplen_avg": { "avgcount": 684, "sum": 745 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 599, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 15, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 23, "osdop_writefull": 6, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 584, "pq_executed_ops": 584, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 9316, "get_or_fail_fail": 0, "get_or_fail_success": 9316, "get_started": 0, "get_sum": 2600845, "max": 104857600, "put": 9315, "put_sum": 2600548, "take": 0, "take_sum": 0, "val": 297, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 684, "put_sum": 17271835, "take": 684, "take_sum": 17271835, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 684, "put_sum": 684, "take": 684, "take_sum": 684, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 584, "get_or_fail_fail": 0, "get_or_fail_success": 584, "get_started": 0, "get_sum": 58984, "max": 3758096384, "put": 7, "put_sum": 58984, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a5a2aa2820": { "get": 9155, "get_or_fail_fail": 0, "get_or_fail_success": 9155, "get_started": 0, "get_sum": 19455328, "max": 3758096384, "put": 16, "put_sum": 16775835, "take": 0, "take_sum": 0, "val": 2679493, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:48.667 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1335, 584/1919 2024-03-06T07:26:49.668 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:49.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.790+0000 7f2becfb5640 1 Processor -- start 2024-03-06T07:26:49.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.790+0000 7f2becfb5640 1 -- start start 2024-03-06T07:26:49.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.791+0000 7f2becfb5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2be80d0950 0x7f2be80d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.791+0000 7f2becfb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b9060 0x7f2be80b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.791+0000 7f2becfb5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be80d1ea0 con 0x7f2be815e150 2024-03-06T07:26:49.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.791+0000 7f2becfb5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be80d12f0 con 0x7f2be80d0950 2024-03-06T07:26:49.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.791+0000 7f2becfb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80d2ab0 con 0x7f2be80b9060 2024-03-06T07:26:49.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.791+0000 7f2be659c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2be815e150 0x7f2be80cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39792/0 (socket says 172.21.15.59:39792) 2024-03-06T07:26:49.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.791+0000 7f2be659c640 1 -- 172.21.15.59:0/1665203335 learned_addr learned my addr 172.21.15.59:0/1665203335 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:49.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.792+0000 7f2be5d9b640 1 --2- 172.21.15.59:0/1665203335 >> v2:172.21.15.164:3300/0 conn(0x7f2be80d0950 0x7f2be80d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.792+0000 7f2be6d9d640 1 --2- 172.21.15.59:0/1665203335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b9060 0x7f2be80b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.792+0000 7f2be559a640 1 -- 172.21.15.59:0/1665203335 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310293716 0 0) 0x7f2be80d1ea0 con 0x7f2be815e150 2024-03-06T07:26:49.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.792+0000 7f2be559a640 1 -- 172.21.15.59:0/1665203335 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd0003520 con 0x7f2be815e150 2024-03-06T07:26:49.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.792+0000 7f2be6d9d640 1 -- 172.21.15.59:0/1665203335 >> v2:172.21.15.164:3300/0 conn(0x7f2be80d0950 msgr2=0x7f2be80d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.792+0000 7f2be6d9d640 1 --2- 172.21.15.59:0/1665203335 >> v2:172.21.15.164:3300/0 conn(0x7f2be80d0950 0x7f2be80d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.792+0000 7f2be6d9d640 1 -- 172.21.15.59:0/1665203335 >> v1:172.21.15.59:6789/0 conn(0x7f2be815e150 legacy=0x7f2be80cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.792+0000 7f2be559a640 1 -- 172.21.15.59:0/1665203335 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 924206415 0 0) 0x7f2bd0003520 con 0x7f2be815e150 2024-03-06T07:26:49.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.792+0000 7f2be6d9d640 1 -- 172.21.15.59:0/1665203335 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80d2180 con 0x7f2be80b9060 2024-03-06T07:26:49.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.792+0000 7f2be5d9b640 1 --2- 172.21.15.59:0/1665203335 >> v2:172.21.15.164:3300/0 conn(0x7f2be80d0950 0x7f2be80d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:49.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.793+0000 7f2be6d9d640 1 --2- 172.21.15.59:0/1665203335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b9060 0x7f2be80b94f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009d90 tx=0x7f2be002bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6110dfe1317285d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.793+0000 7f2be559a640 1 -- 172.21.15.59:0/1665203335 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0039070 con 0x7f2be80b9060 2024-03-06T07:26:49.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.793+0000 7f2be559a640 1 -- 172.21.15.59:0/1665203335 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be002c810 con 0x7f2be80b9060 2024-03-06T07:26:49.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.793+0000 7f2be559a640 1 -- 172.21.15.59:0/1665203335 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8001670 con 0x7f2be80b9060 2024-03-06T07:26:49.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.794+0000 7f2becfb5640 1 -- 172.21.15.59:0/1665203335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b9060 msgr2=0x7f2be80b94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.794+0000 7f2becfb5640 1 --2- 172.21.15.59:0/1665203335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b9060 0x7f2be80b94f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009d90 tx=0x7f2be002bce0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.794+0000 7f2becfb5640 1 -- 172.21.15.59:0/1665203335 shutdown_connections 2024-03-06T07:26:49.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.794+0000 7f2becfb5640 1 --2- 172.21.15.59:0/1665203335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80b9060 0x7f2be80b94f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.794+0000 7f2becfb5640 1 --2- 172.21.15.59:0/1665203335 >> v2:172.21.15.164:3300/0 conn(0x7f2be80d0950 0x7f2be80d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.794+0000 7f2becfb5640 1 -- 172.21.15.59:0/1665203335 >> 172.21.15.59:0/1665203335 conn(0x7f2be809d230 msgr2=0x7f2be80d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.794+0000 7f2becfb5640 1 -- 172.21.15.59:0/1665203335 shutdown_connections 2024-03-06T07:26:49.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.794+0000 7f2becfb5640 1 -- 172.21.15.59:0/1665203335 wait complete. 2024-03-06T07:26:49.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.796+0000 7f2becfb5640 1 Processor -- start 2024-03-06T07:26:49.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.796+0000 7f2becfb5640 1 -- start start 2024-03-06T07:26:49.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.796+0000 7f2becfb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2be80b9060 0x7f2be8151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.796+0000 7f2becfb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80d0950 0x7f2be81f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.796+0000 7f2becfb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be815e150 0x7f2be81f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.796+0000 7f2becfb5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80d2f40 con 0x7f2be815e150 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.796+0000 7f2becfb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80d2740 con 0x7f2be80b9060 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.796+0000 7f2becfb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be80d1b30 con 0x7f2be80d0950 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.796+0000 7f2be6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be815e150 0x7f2be81f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2be6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be815e150 0x7f2be81f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60820/0 (socket says 172.21.15.59:60820) 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2be6d9d640 1 -- 172.21.15.59:0/1995628461 learned_addr learned my addr 172.21.15.59:0/1995628461 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2be659c640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2be80b9060 0x7f2be8151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2be5d9b640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80d0950 0x7f2be81f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2be6d9d640 1 -- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80d0950 msgr2=0x7f2be81f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2be6d9d640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80d0950 0x7f2be81f3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2be6d9d640 1 -- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2be80b9060 msgr2=0x7f2be8151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2be6d9d640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2be80b9060 0x7f2be8151560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2be6d9d640 1 -- 172.21.15.59:0/1995628461 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80ce390 con 0x7f2be815e150 2024-03-06T07:26:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2be659c640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2be80b9060 0x7f2be8151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2be6d9d640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be815e150 0x7f2be81f4470 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009d90 tx=0x7f2be0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2bd6ffd640 1 -- 172.21.15.59:0/1995628461 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0039050 con 0x7f2be815e150 2024-03-06T07:26:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2becfb5640 1 -- 172.21.15.59:0/1995628461 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be80c8f80 con 0x7f2be815e150 2024-03-06T07:26:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.798+0000 7f2becfb5640 1 -- 172.21.15.59:0/1995628461 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be80c9500 con 0x7f2be815e150 2024-03-06T07:26:49.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.797+0000 7f2bd6ffd640 1 -- 172.21.15.59:0/1995628461 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be003a040 con 0x7f2be815e150 2024-03-06T07:26:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.798+0000 7f2bd6ffd640 1 -- 172.21.15.59:0/1995628461 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0004490 con 0x7f2be815e150 2024-03-06T07:26:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.799+0000 7f2becfb5640 1 -- 172.21.15.59:0/1995628461 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be8069ee0 con 0x7f2be815e150 2024-03-06T07:26:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.800+0000 7f2bd6ffd640 1 -- 172.21.15.59:0/1995628461 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2be0047020 con 0x7f2be815e150 2024-03-06T07:26:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.800+0000 7f2bd6ffd640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bc806b4d0 0x7f2bc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.800+0000 7f2bd6ffd640 1 -- 172.21.15.59:0/1995628461 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f2be00abf90 con 0x7f2be815e150 2024-03-06T07:26:49.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.801+0000 7f2be659c640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bc806b4d0 0x7f2bc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.801+0000 7f2be659c640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bc806b4d0 0x7f2bc806d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2bd80017e0 tx=0x7f2bd8001290 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:49.805+0000 7f2bd6ffd640 1 -- 172.21.15.59:0/1995628461 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2be004e020 con 0x7f2be815e150 2024-03-06T07:26:49.880 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:49.882 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:50.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.150+0000 7f2becfb5640 1 -- 172.21.15.59:0/1995628461 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2be80d4670 con 0x7f2be815e150 2024-03-06T07:26:50.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.151+0000 7f2bd6ffd640 1 -- 172.21.15.59:0/1995628461 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f2be007a870 con 0x7f2be815e150 2024-03-06T07:26:50.153 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:50.153 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:50.154 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:50.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 -- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bc806b4d0 msgr2=0x7f2bc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bc806b4d0 0x7f2bc806d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2bd80017e0 tx=0x7f2bd8001290 comp rx=0 tx=0).stop 2024-03-06T07:26:50.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 -- 172.21.15.59:0/1995628461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be815e150 msgr2=0x7f2be81f4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be815e150 0x7f2be81f4470 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f2be0009d90 tx=0x7f2be0004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 -- 172.21.15.59:0/1995628461 shutdown_connections 2024-03-06T07:26:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bc806b4d0 0x7f2bc806d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be815e150 0x7f2be81f4470 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2be80d0950 0x7f2be81f3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 --2- 172.21.15.59:0/1995628461 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2be80b9060 0x7f2be8151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 -- 172.21.15.59:0/1995628461 >> 172.21.15.59:0/1995628461 conn(0x7f2be809d230 msgr2=0x7f2be80ceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 -- 172.21.15.59:0/1995628461 shutdown_connections 2024-03-06T07:26:50.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:50.154+0000 7f2becfb5640 1 -- 172.21.15.59:0/1995628461 wait complete. 2024-03-06T07:26:50.169 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:50.333 INFO:teuthology.orchestra.run.smithi059.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":469,"msgr_send_messages":471,"msgr_recv_bytes":127799,"msgr_send_bytes":4369414,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.072722913,"msgr_running_send_time":0.041465895,"msgr_running_recv_time":0.013137033,"msgr_running_fast_dispatch_time":0.013697855,"msgr_send_messages_queue_lat":{"avgcount":469,"sum":0.030955216,"avgtime":0.000066002},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000225,"avgtime":0.000000112},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9828,"msgr_send_messages":21424,"msgr_recv_bytes":3665076,"msgr_send_bytes":23708090,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.988506651,"msgr_running_send_time":0.546419226,"msgr_running_recv_time":0.526486010,"msgr_running_fast_dispatch_time":0.016503371,"msgr_send_messages_queue_lat":{"avgcount":21422,"sum":0.326017318,"avgtime":0.000015218},"msgr_handle_ack_lat":{"avgcount":28556,"sum":0.005125409,"avgtime":0.000000179},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":352,"msgr_send_messages":346,"msgr_recv_bytes":196707,"msgr_send_bytes":3507517,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.042623125,"msgr_running_send_time":0.013976445,"msgr_running_recv_time":0.015162926,"msgr_running_fast_dispatch_time":0.011640403,"msgr_send_messages_queue_lat":{"avgcount":346,"sum":0.032200440,"avgtime":0.000093064},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108848,"msgr_send_encrypted_bytes":8368},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":2071,"complete_latency":{"avgcount":90,"sum":5.337464848,"avgtime":0.059305164}},"finisher-PurgeQueue":{"queue_len":359,"complete_latency":{"avgcount":1600,"sum":0.550695451,"avgtime":0.000344184}},"mds":{"request":6043,"reply":6041,"reply_latency":{"avgcount":6041,"sum":8.638799556,"avgtime":0.001430028},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5458,"inodes_top":0,"inodes_bottom":5458,"inodes_pin_tail":0,"inodes_pinned":5458,"inodes_expired":0,"inodes_with_caps":1830,"caps":1830,"subtrees":2,"traverse":6049,"traverse_hit":6048,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1458,"root_rbytes":10496000,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":886,"ceph_cap_op_grant":6167,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3456,"handle_client_caps_dirty":3000,"handle_client_cap_release":5,"process_request_cap_release":8863},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1665,"num_strays_delayed":0,"num_strays_enqueuing":206,"strays_created":2831,"strays_enqueued":1166,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":10652,"evex":0,"evtrm":0,"ev":10655,"evexg":0,"evexd":0,"segadd":13,"segex":0,"segtrm":0,"seg":15,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":26912455,"rdpos":4195244,"jlat":{"avgcount":17,"sum":1.859504168,"avgtime":0.109382598},"replayed":3},"mds_mem":{"ino":2630,"ino+":3214,"ino-":584,"dir":13,"dir+":13,"dir-":0,"dn":5458,"dn+":6042,"dn-":584,"cap":1830,"cap+":3202,"cap-":1372,"rss":114104,"heap":248164},"mds_server":{"dispatch_client_request":6046,"dispatch_server_request":0,"handle_client_request":6043,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.113519323,"avgtime":0.000972974},"req_getattr_latency":{"avgcount":6,"sum":0.001174100,"avgtime":0.000195683},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002442159,"avgtime":0.002442159},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.451304597,"avgtime":0.150434865},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2831,"sum":5.070359377,"avgtime":0.001791013},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5562,"average_load":5562,"avg_session_uptime":849,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":5185252,"buffer_anon_items":2253,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14899416,"mds_co_items":161817,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1097,"op_send_bytes":22231398,"op_resend":0,"op_reply":1097,"op_latency":{"avgcount":1097,"sum":2.000974829,"avgtime":0.001824042},"op_inflight":0,"oplen_avg":{"avgcount":1097,"sum":1161},"op":1097,"op_r":49,"op_w":1048,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":21,"osdop_write":32,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":994,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":25,"omap_rd":48,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":1,"pq_executing_ops_high_water":6,"pq_executing":1,"pq_executing_high_water":6,"pq_executed_ops":977,"pq_executed":977,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":296,"max":104857600,"get_started":0,"get":10645,"get_sum":3028813,"get_or_fail_fail":0,"get_or_fail_success":10645,"take":0,"take_sum":0,"put":10644,"put_sum":3028517,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1097,"take_sum":22371087,"put":1097,"put_sum":22371087,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1097,"take_sum":1097,"put":1097,"put_sum":1097,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1372,"get_sum":138572,"get_or_fail_fail":0,"get_or_fail_success":1372,"take":0,"take_sum":0,"put":13,"put_sum":138572,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a5a2aa2820":{"val":1002600,"max":3758096384,"get_started":0,"get":10652,"get_sum":22716770,"get_or_fail_fail":0,"get_or_fail_success":10652,"take":0,"take_sum":0,"put":19,"put_sum":21714170,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:50.335 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.12e-07, "sum": 2.25e-07 }, "msgr_recv_bytes": 127799, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 469, "msgr_running_fast_dispatch_time": 0.013697855, "msgr_running_recv_time": 0.013137033, "msgr_running_send_time": 0.041465895, "msgr_running_total_time": 0.072722913, "msgr_send_bytes": 4369414, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 471, "msgr_send_messages_queue_lat": { "avgcount": 469, "avgtime": 6.6002e-05, "sum": 0.030955216 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 28556, "avgtime": 1.79e-07, "sum": 0.005125409 }, "msgr_recv_bytes": 3665076, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9828, "msgr_running_fast_dispatch_time": 0.016503371, "msgr_running_recv_time": 0.52648601, "msgr_running_send_time": 0.546419226, "msgr_running_total_time": 0.988506651, "msgr_send_bytes": 23708090, "msgr_send_encrypted_bytes": 95808, "msgr_send_messages": 21424, "msgr_send_messages_queue_lat": { "avgcount": 21422, "avgtime": 1.5218e-05, "sum": 0.326017318 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 196707, "msgr_recv_encrypted_bytes": 108848, "msgr_recv_messages": 352, "msgr_running_fast_dispatch_time": 0.011640403, "msgr_running_recv_time": 0.015162926, "msgr_running_send_time": 0.013976445, "msgr_running_total_time": 0.042623125, "msgr_send_bytes": 3507517, "msgr_send_encrypted_bytes": 8368, "msgr_send_messages": 346, "msgr_send_messages_queue_lat": { "avgcount": 346, "avgtime": 9.3064e-05, "sum": 0.03220044 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 90, "avgtime": 0.059305164, "sum": 5.337464848 }, "queue_len": 2071 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1600, "avgtime": 0.000344184, "sum": 0.550695451 }, "queue_len": 359 }, "mds": { "caps": 1830, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6167, "ceph_cap_op_revoke": 886, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 5, "handle_client_caps": 3456, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5458, "inodes_bottom": 5458, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5458, "inodes_top": 0, "inodes_with_caps": 1830, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 8863, "q": 0, "reply": 6041, "reply_latency": { "avgcount": 6041, "avgtime": 0.001430028, "sum": 8.638799556 }, "request": 6043, "root_rbytes": 10496000, "root_rfiles": 1458, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6049, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6048, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1665, "num_strays_delayed": 0, "num_strays_enqueuing": 206, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2831, "strays_enqueued": 1166, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 10655, "evadd": 10652, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 17, "avgtime": 0.109382598, "sum": 1.859504168 }, "rdpos": 4195244, "replayed": 3, "seg": 15, "segadd": 13, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 26912455 }, "mds_mem": { "cap": 1830, "cap+": 3202, "cap-": 1372, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5458, "dn+": 6042, "dn-": 584, "heap": 248164, "ino": 2630, "ino+": 3214, "ino-": 584, "rss": 114104 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6046, "dispatch_server_request": 0, "handle_client_request": 6043, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000972974, "sum": 3.113519323 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195683, "sum": 0.0011741 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002442159, "sum": 0.002442159 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.150434865, "sum": 0.451304597 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2831, "avgtime": 0.001791013, "sum": 5.070359377 } }, "mds_sessions": { "average_load": 5562, "avg_session_uptime": 849, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5562 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 5185252, "buffer_anon_items": 2253, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 14899416, "mds_co_items": 161817, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 48, "omap_wr": 25, "op": 1097, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1097, "avgtime": 0.001824042, "sum": 2.000974829 }, "op_pg": 0, "op_r": 49, "op_reply": 1097, "op_resend": 0, "op_rmw": 0, "op_send": 1097, "op_send_bytes": 22231398, "op_w": 1048, "oplen_avg": { "avgcount": 1097, "sum": 1161 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 994, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 21, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 32, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 977, "pq_executed_ops": 977, "pq_executing": 1, "pq_executing_high_water": 6, "pq_executing_ops": 1, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10645, "get_or_fail_fail": 0, "get_or_fail_success": 10645, "get_started": 0, "get_sum": 3028813, "max": 104857600, "put": 10644, "put_sum": 3028517, "take": 0, "take_sum": 0, "val": 296, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1097, "put_sum": 22371087, "take": 1097, "take_sum": 22371087, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1097, "put_sum": 1097, "take": 1097, "take_sum": 1097, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1372, "get_or_fail_fail": 0, "get_or_fail_success": 1372, "get_started": 0, "get_sum": 138572, "max": 3758096384, "put": 13, "put_sum": 138572, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a5a2aa2820": { "get": 10652, "get_or_fail_fail": 0, "get_or_fail_success": 10652, "get_started": 0, "get_sum": 22716770, "max": 3758096384, "put": 19, "put_sum": 21714170, "take": 0, "take_sum": 0, "val": 1002600, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:50.335 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 1/1665, 977/2831 2024-03-06T07:26:51.335 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:51.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.462+0000 7f31267ac640 1 Processor -- start 2024-03-06T07:26:51.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.463+0000 7f31267ac640 1 -- start start 2024-03-06T07:26:51.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.463+0000 7f31267ac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31201405e0 0x7f31201409c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.463+0000 7f31267ac640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312015ebd0 con 0x7f3120140f00 2024-03-06T07:26:51.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.463+0000 7f31267ac640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312015eeb0 con 0x7f3120072610 2024-03-06T07:26:51.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.463+0000 7f31267ac640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31200cd6a0 con 0x7f31201405e0 2024-03-06T07:26:51.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.464+0000 7f311ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31201405e0 0x7f31201409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.464+0000 7f311f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3120140f00 0x7f3120071f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39794/0 (socket says 172.21.15.59:39794) 2024-03-06T07:26:51.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.464+0000 7f311f7fe640 1 -- 172.21.15.59:0/547144762 learned_addr learned my addr 172.21.15.59:0/547144762 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:51.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.464+0000 7f311ffff640 1 -- 172.21.15.59:0/547144762 >> v1:172.21.15.164:6789/0 conn(0x7f3120072610 legacy=0x7f312015b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.464+0000 7f311ffff640 1 -- 172.21.15.59:0/547144762 >> v1:172.21.15.59:6789/0 conn(0x7f3120140f00 legacy=0x7f3120071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.464+0000 7f311ffff640 1 -- 172.21.15.59:0/547144762 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312015f190 con 0x7f31201405e0 2024-03-06T07:26:51.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.464+0000 7f311effd640 1 -- 172.21.15.59:0/547144762 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854394799 0 0) 0x7f312015ebd0 con 0x7f3120140f00 2024-03-06T07:26:51.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.465+0000 7f311ffff640 1 --2- 172.21.15.59:0/547144762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31201405e0 0x7f31201409c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f31100097a0 tx=0x7f311002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ab232cde7b915c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.465+0000 7f311effd640 1 -- 172.21.15.59:0/547144762 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110035020 con 0x7f31201405e0 2024-03-06T07:26:51.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.465+0000 7f311effd640 1 -- 172.21.15.59:0/547144762 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110031830 con 0x7f31201405e0 2024-03-06T07:26:51.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.465+0000 7f311effd640 1 -- 172.21.15.59:0/547144762 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110031c00 con 0x7f31201405e0 2024-03-06T07:26:51.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.466+0000 7f31267ac640 1 -- 172.21.15.59:0/547144762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31201405e0 msgr2=0x7f31201409c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.466+0000 7f31267ac640 1 --2- 172.21.15.59:0/547144762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31201405e0 0x7f31201409c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f31100097a0 tx=0x7f311002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.466+0000 7f31267ac640 1 -- 172.21.15.59:0/547144762 shutdown_connections 2024-03-06T07:26:51.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.466+0000 7f31267ac640 1 --2- 172.21.15.59:0/547144762 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31201405e0 0x7f31201409c0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.466+0000 7f31267ac640 1 -- 172.21.15.59:0/547144762 >> 172.21.15.59:0/547144762 conn(0x7f31200d0e20 msgr2=0x7f31200d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.466+0000 7f31267ac640 1 -- 172.21.15.59:0/547144762 shutdown_connections 2024-03-06T07:26:51.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.466+0000 7f31267ac640 1 -- 172.21.15.59:0/547144762 wait complete. 2024-03-06T07:26:51.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.467+0000 7f31267ac640 1 Processor -- start 2024-03-06T07:26:51.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.468+0000 7f31267ac640 1 -- start start 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.468+0000 7f31267ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3120072610 0x7f312013d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.468+0000 7f31267ac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31201405e0 0x7f31201f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.468+0000 7f31267ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3120140f00 0x7f312015b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.468+0000 7f31267ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312006f5c0 con 0x7f3120072610 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.468+0000 7f31267ac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31200b5ee0 con 0x7f31201405e0 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.468+0000 7f31267ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312015e7f0 con 0x7f3120140f00 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3120072610 0x7f312013d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3120072610 0x7f312013d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60842/0 (socket says 172.21.15.59:60842) 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311ffff640 1 -- 172.21.15.59:0/4104955388 learned_addr learned my addr 172.21.15.59:0/4104955388 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311f7fe640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31201405e0 0x7f31201f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f3124d49640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3120140f00 0x7f312015b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311ffff640 1 -- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3120140f00 msgr2=0x7f312015b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311ffff640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3120140f00 0x7f312015b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311ffff640 1 -- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31201405e0 msgr2=0x7f31201f4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311ffff640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31201405e0 0x7f31201f4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311ffff640 1 -- 172.21.15.59:0/4104955388 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31200ae640 con 0x7f3120072610 2024-03-06T07:26:51.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311f7fe640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31201405e0 0x7f31201f4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:51.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311ffff640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3120072610 0x7f312013d8d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f311002bee0 tx=0x7f31100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311cff9640 1 -- 172.21.15.59:0/4104955388 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110041070 con 0x7f3120072610 2024-03-06T07:26:51.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f311cff9640 1 -- 172.21.15.59:0/4104955388 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311003aa30 con 0x7f3120072610 2024-03-06T07:26:51.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.469+0000 7f31267ac640 1 -- 172.21.15.59:0/4104955388 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31200ae920 con 0x7f3120072610 2024-03-06T07:26:51.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.470+0000 7f31267ac640 1 -- 172.21.15.59:0/4104955388 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31200aeee0 con 0x7f3120072610 2024-03-06T07:26:51.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.470+0000 7f311cff9640 1 -- 172.21.15.59:0/4104955388 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110035040 con 0x7f3120072610 2024-03-06T07:26:51.475 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:51.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.471+0000 7f31267ac640 1 -- 172.21.15.59:0/4104955388 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3120068050 con 0x7f3120072610 2024-03-06T07:26:51.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.472+0000 7f311cff9640 1 -- 172.21.15.59:0/4104955388 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3110038020 con 0x7f3120072610 2024-03-06T07:26:51.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.472+0000 7f311cff9640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310c06b4d0 0x7f310c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.472+0000 7f311cff9640 1 -- 172.21.15.59:0/4104955388 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f31100abd00 con 0x7f3120072610 2024-03-06T07:26:51.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.472+0000 7f311f7fe640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310c06b4d0 0x7f310c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.473+0000 7f311f7fe640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310c06b4d0 0x7f310c06d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3114007fd0 tx=0x7f3114007dc0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.481+0000 7f311cff9640 1 -- 172.21.15.59:0/4104955388 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f311007ab10 con 0x7f3120072610 2024-03-06T07:26:51.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.575+0000 7ff29b713640 1 Processor -- start 2024-03-06T07:26:51.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.576+0000 7ff29b713640 1 -- start start 2024-03-06T07:26:51.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.576+0000 7ff29b713640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff29415e150 0x7ff2940cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.576+0000 7ff29b713640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff2940d1060 0x7ff2940b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.576+0000 7ff29b713640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2940d1ea0 con 0x7ff2940cfba0 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.576+0000 7ff29b713640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2940d2ab0 con 0x7ff2940d1060 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.576+0000 7ff29b713640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2940d2180 con 0x7ff29415e150 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.576+0000 7ff299cb0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff2940d1060 0x7ff2940b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.576+0000 7ff299cb0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff2940d1060 0x7ff2940b9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60856/0 (socket says 172.21.15.59:60856) 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.576+0000 7ff299cb0640 1 -- 172.21.15.59:0/999426576 learned_addr learned my addr 172.21.15.59:0/999426576 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.577+0000 7ff2994af640 1 --2- 172.21.15.59:0/999426576 >> v2:172.21.15.164:3300/0 conn(0x7ff29415e150 0x7ff2940cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.577+0000 7ff299cb0640 1 -- 172.21.15.59:0/999426576 >> v2:172.21.15.164:3300/0 conn(0x7ff29415e150 msgr2=0x7ff2940cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.577+0000 7ff299cb0640 1 --2- 172.21.15.59:0/999426576 >> v2:172.21.15.164:3300/0 conn(0x7ff29415e150 0x7ff2940cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.577+0000 7ff299cb0640 1 -- 172.21.15.59:0/999426576 >> v1:172.21.15.164:6789/0 conn(0x7ff2940cfba0 legacy=0x7ff2940d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.577+0000 7ff299cb0640 1 -- 172.21.15.59:0/999426576 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2940d2460 con 0x7ff2940d1060 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff299cb0640 1 --2- 172.21.15.59:0/999426576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2940d1060 0x7ff2940b9060 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7ff2940b6af0 tx=0x7ff28c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8014ddd4b38e2bd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff28bfff640 1 -- 172.21.15.59:0/999426576 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c02ca80 con 0x7ff2940d1060 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff28bfff640 1 -- 172.21.15.59:0/999426576 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28c02cc20 con 0x7ff2940d1060 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff28bfff640 1 -- 172.21.15.59:0/999426576 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c0343f0 con 0x7ff2940d1060 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff29b713640 1 -- 172.21.15.59:0/999426576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2940d1060 msgr2=0x7ff2940b9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff29b713640 1 --2- 172.21.15.59:0/999426576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2940d1060 0x7ff2940b9060 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7ff2940b6af0 tx=0x7ff28c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff29b713640 1 -- 172.21.15.59:0/999426576 shutdown_connections 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff29b713640 1 --2- 172.21.15.59:0/999426576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2940d1060 0x7ff2940b9060 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff29b713640 1 --2- 172.21.15.59:0/999426576 >> v2:172.21.15.164:3300/0 conn(0x7ff29415e150 0x7ff2940cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff29b713640 1 -- 172.21.15.59:0/999426576 >> 172.21.15.59:0/999426576 conn(0x7ff29409d230 msgr2=0x7ff2940d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff29b713640 1 -- 172.21.15.59:0/999426576 shutdown_connections 2024-03-06T07:26:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.578+0000 7ff29b713640 1 -- 172.21.15.59:0/999426576 wait complete. 2024-03-06T07:26:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.580+0000 7ff29b713640 1 Processor -- start 2024-03-06T07:26:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.580+0000 7ff29b713640 1 -- start start 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.580+0000 7ff29b713640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2940cfba0 0x7ff294158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.580+0000 7ff29b713640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2940d1060 0x7ff2940b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.580+0000 7ff29b713640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff29415e150 0x7ff2940b8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.580+0000 7ff29b713640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2940d1b30 con 0x7ff29415e150 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.580+0000 7ff29b713640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2940d2f40 con 0x7ff2940d1060 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.580+0000 7ff29b713640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29415e530 con 0x7ff2940cfba0 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.580+0000 7ff299cb0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff29415e150 0x7ff2940b8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.580+0000 7ff299cb0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff29415e150 0x7ff2940b8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60858/0 (socket says 172.21.15.59:60858) 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.580+0000 7ff299cb0640 1 -- 172.21.15.59:0/2985612926 learned_addr learned my addr 172.21.15.59:0/2985612926 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff298cae640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2940d1060 0x7ff2940b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff2994af640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2940cfba0 0x7ff294158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff299cb0640 1 -- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2940cfba0 msgr2=0x7ff294158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff299cb0640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2940cfba0 0x7ff294158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff299cb0640 1 -- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2940d1060 msgr2=0x7ff2940b8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff299cb0640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2940d1060 0x7ff2940b8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff299cb0640 1 -- 172.21.15.59:0/2985612926 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2940b6c20 con 0x7ff29415e150 2024-03-06T07:26:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff2994af640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2940cfba0 0x7ff294158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff299cb0640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff29415e150 0x7ff2940b8fa0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7ff28c009f40 tx=0x7ff28c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff289ffb640 1 -- 172.21.15.59:0/2985612926 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c043020 con 0x7ff29415e150 2024-03-06T07:26:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff289ffb640 1 -- 172.21.15.59:0/2985612926 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28c03e450 con 0x7ff29415e150 2024-03-06T07:26:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff29b713640 1 -- 172.21.15.59:0/2985612926 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2940b6f00 con 0x7ff29415e150 2024-03-06T07:26:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff289ffb640 1 -- 172.21.15.59:0/2985612926 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c009510 con 0x7ff29415e150 2024-03-06T07:26:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.581+0000 7ff29b713640 1 -- 172.21.15.59:0/2985612926 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff294158ba0 con 0x7ff29415e150 2024-03-06T07:26:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.583+0000 7ff29b713640 1 -- 172.21.15.59:0/2985612926 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2940d30c0 con 0x7ff29415e150 2024-03-06T07:26:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.583+0000 7ff289ffb640 1 -- 172.21.15.59:0/2985612926 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff28c046020 con 0x7ff29415e150 2024-03-06T07:26:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.583+0000 7ff289ffb640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff27806b4d0 0x7ff27806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.583+0000 7ff289ffb640 1 -- 172.21.15.59:0/2985612926 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff28c0acad0 con 0x7ff29415e150 2024-03-06T07:26:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.584+0000 7ff2994af640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff27806b4d0 0x7ff27806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.584+0000 7ff2994af640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff27806b4d0 0x7ff27806d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff290009af0 tx=0x7ff290008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.592+0000 7ff289ffb640 1 -- 172.21.15.59:0/2985612926 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2940d30c0 con 0x7ff29415e150 2024-03-06T07:26:51.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.714+0000 7f31267ac640 1 -- 172.21.15.59:0/4104955388 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31200d1cf0 con 0x7f3120072610 2024-03-06T07:26:51.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.715+0000 7f311cff9640 1 -- 172.21.15.59:0/4104955388 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f311007a410 con 0x7f3120072610 2024-03-06T07:26:51.718 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:51.718 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:51.718 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:51.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 -- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310c06b4d0 msgr2=0x7f310c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310c06b4d0 0x7f310c06d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3114007fd0 tx=0x7f3114007dc0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 -- 172.21.15.59:0/4104955388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3120072610 msgr2=0x7f312013d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3120072610 0x7f312013d8d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f311002bee0 tx=0x7f31100046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 -- 172.21.15.59:0/4104955388 shutdown_connections 2024-03-06T07:26:51.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310c06b4d0 0x7f310c06d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3120140f00 0x7f312015b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31201405e0 0x7f31201f4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 --2- 172.21.15.59:0/4104955388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3120072610 0x7f312013d8d0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 -- 172.21.15.59:0/4104955388 >> 172.21.15.59:0/4104955388 conn(0x7f31200d0e20 msgr2=0x7f3120159400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 -- 172.21.15.59:0/4104955388 shutdown_connections 2024-03-06T07:26:51.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.717+0000 7f31267ac640 1 -- 172.21.15.59:0/4104955388 wait complete. 2024-03-06T07:26:51.734 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:51.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.858+0000 7ff29b713640 1 -- 172.21.15.59:0/2985612926 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff294069ee0 con 0x7ff29415e150 2024-03-06T07:26:51.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.859+0000 7ff289ffb640 1 -- 172.21.15.59:0/2985612926 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7ff28c04d030 con 0x7ff29415e150 2024-03-06T07:26:51.863 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:51.864 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:51.865 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:51.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 -- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff27806b4d0 msgr2=0x7ff27806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff27806b4d0 0x7ff27806d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff290009af0 tx=0x7ff290008040 comp rx=0 tx=0).stop 2024-03-06T07:26:51.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 -- 172.21.15.59:0/2985612926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff29415e150 msgr2=0x7ff2940b8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff29415e150 0x7ff2940b8fa0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7ff28c009f40 tx=0x7ff28c004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:51.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 -- 172.21.15.59:0/2985612926 shutdown_connections 2024-03-06T07:26:51.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff27806b4d0 0x7ff27806d990 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff29415e150 0x7ff2940b8fa0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2940d1060 0x7ff2940b8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 --2- 172.21.15.59:0/2985612926 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2940cfba0 0x7ff294158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 -- 172.21.15.59:0/2985612926 >> 172.21.15.59:0/2985612926 conn(0x7ff29409d230 msgr2=0x7ff29406d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 -- 172.21.15.59:0/2985612926 shutdown_connections 2024-03-06T07:26:51.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:51.861+0000 7ff29b713640 1 -- 172.21.15.59:0/2985612926 wait complete. 2024-03-06T07:26:51.877 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:26:51.896 INFO:teuthology.orchestra.run.smithi059.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":639,"msgr_send_messages":641,"msgr_recv_bytes":170979,"msgr_send_bytes":4424834,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.092489798,"msgr_running_send_time":0.051809650,"msgr_running_recv_time":0.016125451,"msgr_running_fast_dispatch_time":0.018208070,"msgr_send_messages_queue_lat":{"avgcount":639,"sum":0.038804069,"avgtime":0.000060726},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000225,"avgtime":0.000000112},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10323,"msgr_send_messages":24684,"msgr_recv_bytes":3851266,"msgr_send_bytes":24605333,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.086608574,"msgr_running_send_time":0.608969858,"msgr_running_recv_time":0.558850136,"msgr_running_fast_dispatch_time":0.019542139,"msgr_send_messages_queue_lat":{"avgcount":24682,"sum":0.401336288,"avgtime":0.000016260},"msgr_handle_ack_lat":{"avgcount":30661,"sum":0.005566453,"avgtime":0.000000181},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":469,"msgr_send_messages":463,"msgr_recv_bytes":226401,"msgr_send_bytes":5123817,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.058270775,"msgr_running_send_time":0.021710253,"msgr_running_recv_time":0.017945233,"msgr_running_fast_dispatch_time":0.016117731,"msgr_send_messages_queue_lat":{"avgcount":463,"sum":0.041192387,"avgtime":0.000088968},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108848,"msgr_send_encrypted_bytes":8368},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":94,"sum":7.492916157,"avgtime":0.079711874}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2237,"sum":1.000449257,"avgtime":0.000447228}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.587841570,"avgtime":0.001495529},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5040,"inodes_top":0,"inodes_bottom":5040,"inodes_pin_tail":0,"inodes_pinned":5040,"inodes_expired":0,"inodes_with_caps":1385,"caps":1385,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":398,"root_rbytes":3115008,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":888,"ceph_cap_op_grant":7552,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3458,"handle_client_caps_dirty":3000,"handle_client_cap_release":7,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1824,"num_strays_delayed":440,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":1377,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":11234,"evex":0,"evtrm":0,"ev":11237,"evexg":0,"evexd":0,"segadd":14,"segex":0,"segtrm":0,"seg":16,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":28217888,"rdpos":4195244,"jlat":{"avgcount":20,"sum":4.502955396,"avgtime":0.225147769},"replayed":3},"mds_mem":{"ino":1842,"ino+":3214,"ino-":1372,"dir":13,"dir+":13,"dir-":0,"dn":5040,"dn+":6412,"dn-":1372,"cap":1385,"cap+":3202,"cap-":1817,"rss":114104,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.113519323,"avgtime":0.000972974},"req_getattr_latency":{"avgcount":6,"sum":0.001174100,"avgtime":0.000195683},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002442159,"avgtime":0.002442159},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.451304597,"avgtime":0.150434865},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.455689955,"avgtime":0.455689955},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.563711436,"avgtime":0.001738659},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5855,"average_load":5855,"avg_session_uptime":849,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":5185252,"buffer_anon_items":2253,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14899416,"mds_co_items":161817,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1504,"op_send_bytes":24533848,"op_resend":0,"op_reply":1504,"op_latency":{"avgcount":1504,"sum":2.685966212,"avgtime":0.001785881},"op_inflight":0,"oplen_avg":{"avgcount":1504,"sum":1570},"op":1504,"op_r":51,"op_w":1453,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":23,"osdop_write":35,"osdop_writefull":9,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1393,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":27,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1377,"pq_executed":1377,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":11427,"get_sum":3217497,"get_or_fail_fail":0,"get_or_fail_success":11427,"take":0,"take_sum":0,"put":11427,"put_sum":3217497,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1504,"take_sum":24674042,"put":1504,"put_sum":24674042,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1504,"take_sum":1504,"put":1504,"put_sum":1504,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1377,"get_sum":139077,"get_or_fail_fail":0,"get_or_fail_success":1377,"take":0,"take_sum":0,"put":15,"put_sum":139077,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a5a2aa2820":{"val":6493,"max":3758096384,"get_started":0,"get":11234,"get_sum":24022203,"get_or_fail_fail":0,"get_or_fail_success":11234,"take":0,"take_sum":0,"put":20,"put_sum":24015710,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:51.897 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.12e-07, "sum": 2.25e-07 }, "msgr_recv_bytes": 170979, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 639, "msgr_running_fast_dispatch_time": 0.01820807, "msgr_running_recv_time": 0.016125451, "msgr_running_send_time": 0.05180965, "msgr_running_total_time": 0.092489798, "msgr_send_bytes": 4424834, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 641, "msgr_send_messages_queue_lat": { "avgcount": 639, "avgtime": 6.0726e-05, "sum": 0.038804069 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 30661, "avgtime": 1.81e-07, "sum": 0.005566453 }, "msgr_recv_bytes": 3851266, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 10323, "msgr_running_fast_dispatch_time": 0.019542139, "msgr_running_recv_time": 0.558850136, "msgr_running_send_time": 0.608969858, "msgr_running_total_time": 1.086608574, "msgr_send_bytes": 24605333, "msgr_send_encrypted_bytes": 95808, "msgr_send_messages": 24684, "msgr_send_messages_queue_lat": { "avgcount": 24682, "avgtime": 1.626e-05, "sum": 0.401336288 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 226401, "msgr_recv_encrypted_bytes": 108848, "msgr_recv_messages": 469, "msgr_running_fast_dispatch_time": 0.016117731, "msgr_running_recv_time": 0.017945233, "msgr_running_send_time": 0.021710253, "msgr_running_total_time": 0.058270775, "msgr_send_bytes": 5123817, "msgr_send_encrypted_bytes": 8368, "msgr_send_messages": 463, "msgr_send_messages_queue_lat": { "avgcount": 463, "avgtime": 8.8968e-05, "sum": 0.041192387 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 94, "avgtime": 0.079711874, "sum": 7.492916157 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2237, "avgtime": 0.000447228, "sum": 1.000449257 }, "queue_len": 0 }, "mds": { "caps": 1385, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7552, "ceph_cap_op_revoke": 888, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 7, "handle_client_caps": 3458, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5040, "inodes_bottom": 5040, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5040, "inodes_top": 0, "inodes_with_caps": 1385, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001495529, "sum": 9.58784157 }, "request": 6413, "root_rbytes": 3115008, "root_rfiles": 398, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1824, "num_strays_delayed": 440, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 1377, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 11237, "evadd": 11234, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 20, "avgtime": 0.225147769, "sum": 4.502955396 }, "rdpos": 4195244, "replayed": 3, "seg": 16, "segadd": 14, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 28217888 }, "mds_mem": { "cap": 1385, "cap+": 3202, "cap-": 1817, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5040, "dn+": 6412, "dn-": 1372, "heap": 248164, "ino": 1842, "ino+": 3214, "ino-": 1372, "rss": 114104 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000972974, "sum": 3.113519323 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195683, "sum": 0.0011741 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002442159, "sum": 0.002442159 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.150434865, "sum": 0.451304597 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.455689955, "sum": 0.455689955 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001738659, "sum": 5.563711436 } }, "mds_sessions": { "average_load": 5855, "avg_session_uptime": 849, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5855 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 5185252, "buffer_anon_items": 2253, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 14899416, "mds_co_items": 161817, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 27, "op": 1504, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1504, "avgtime": 0.001785881, "sum": 2.685966212 }, "op_pg": 0, "op_r": 51, "op_reply": 1504, "op_resend": 0, "op_rmw": 0, "op_send": 1504, "op_send_bytes": 24533848, "op_w": 1453, "oplen_avg": { "avgcount": 1504, "sum": 1570 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1393, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 23, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 35, "osdop_writefull": 9, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1377, "pq_executed_ops": 1377, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11427, "get_or_fail_fail": 0, "get_or_fail_success": 11427, "get_started": 0, "get_sum": 3217497, "max": 104857600, "put": 11427, "put_sum": 3217497, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1504, "put_sum": 24674042, "take": 1504, "take_sum": 24674042, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1504, "put_sum": 1504, "take": 1504, "take_sum": 1504, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1377, "get_or_fail_fail": 0, "get_or_fail_success": 1377, "get_started": 0, "get_sum": 139077, "max": 3758096384, "put": 15, "put_sum": 139077, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a5a2aa2820": { "get": 11234, "get_or_fail_fail": 0, "get_or_fail_success": 11234, "get_started": 0, "get_sum": 24022203, "max": 3758096384, "put": 20, "put_sum": 24015710, "take": 0, "take_sum": 0, "val": 6493, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:51.897 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1824, 1377/3201 2024-03-06T07:26:52.306 INFO:teuthology.orchestra.run.smithi059.stdout:{"message":"","return_code":0} 2024-03-06T07:26:52.306 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:26:52.898 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:53.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.017+0000 7f4aabb2b640 1 Processor -- start 2024-03-06T07:26:53.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.017+0000 7f4aabb2b640 1 -- start start 2024-03-06T07:26:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aabb2b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4aa415e150 0x7f4aa40cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aabb2b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa40d1060 0x7f4aa40b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aabb2b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa40d1ea0 con 0x7f4aa40cfba0 2024-03-06T07:26:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aabb2b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa40d2ab0 con 0x7f4aa415e150 2024-03-06T07:26:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aabb2b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40d2180 con 0x7f4aa40d1060 2024-03-06T07:26:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aa98c7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4aa415e150 0x7f4aa40cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aa90c6640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4aa40cfba0 0x7f4aa40d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39806/0 (socket says 172.21.15.59:39806) 2024-03-06T07:26:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aa90c6640 1 -- 172.21.15.59:0/3434203712 learned_addr learned my addr 172.21.15.59:0/3434203712 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:53.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aaa0c8640 1 --2- 172.21.15.59:0/3434203712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa40d1060 0x7f4aa40b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aa98c7640 1 -- 172.21.15.59:0/3434203712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa40d1060 msgr2=0x7f4aa40b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aa98c7640 1 --2- 172.21.15.59:0/3434203712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa40d1060 0x7f4aa40b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aa98c7640 1 -- 172.21.15.59:0/3434203712 >> v1:172.21.15.59:6789/0 conn(0x7f4aa40cfba0 legacy=0x7f4aa40d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aa88c5640 1 -- 172.21.15.59:0/3434203712 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114736917 0 0) 0x7f4aa40d1ea0 con 0x7f4aa40cfba0 2024-03-06T07:26:53.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.018+0000 7f4aa98c7640 1 -- 172.21.15.59:0/3434203712 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa40d2460 con 0x7f4aa415e150 2024-03-06T07:26:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.019+0000 7f4aa98c7640 1 --2- 172.21.15.59:0/3434203712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa415e150 0x7f4aa40cf660 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00097a0 tx=0x7f4aa002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4f64c30386aff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.019+0000 7f4aa88c5640 1 -- 172.21.15.59:0/3434203712 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00315a0 con 0x7f4aa415e150 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.019+0000 7f4aa88c5640 1 -- 172.21.15.59:0/3434203712 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa0037070 con 0x7f4aa415e150 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.019+0000 7f4aa88c5640 1 -- 172.21.15.59:0/3434203712 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0031960 con 0x7f4aa415e150 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.020+0000 7f4aabb2b640 1 -- 172.21.15.59:0/3434203712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa415e150 msgr2=0x7f4aa40cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.020+0000 7f4aabb2b640 1 --2- 172.21.15.59:0/3434203712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa415e150 0x7f4aa40cf660 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00097a0 tx=0x7f4aa002b840 comp rx=0 tx=0).stop 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.020+0000 7f4aabb2b640 1 -- 172.21.15.59:0/3434203712 shutdown_connections 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.020+0000 7f4aabb2b640 1 --2- 172.21.15.59:0/3434203712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa40d1060 0x7f4aa40b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.020+0000 7f4aabb2b640 1 --2- 172.21.15.59:0/3434203712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa415e150 0x7f4aa40cf660 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.020+0000 7f4aabb2b640 1 -- 172.21.15.59:0/3434203712 >> 172.21.15.59:0/3434203712 conn(0x7f4aa409d230 msgr2=0x7f4aa40d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.020+0000 7f4aabb2b640 1 -- 172.21.15.59:0/3434203712 shutdown_connections 2024-03-06T07:26:53.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.020+0000 7f4aabb2b640 1 -- 172.21.15.59:0/3434203712 wait complete. 2024-03-06T07:26:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.021+0000 7f4aabb2b640 1 Processor -- start 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.021+0000 7f4aabb2b640 1 -- start start 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aabb2b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40cfba0 0x7f4aa40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aabb2b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa40d1060 0x7f4aa415fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aabb2b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa415e150 0x7f4aa415b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aabb2b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40d1b30 con 0x7f4aa40cfba0 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aabb2b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40d2f40 con 0x7f4aa40d1060 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aabb2b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa415e530 con 0x7f4aa415e150 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aa98c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40cfba0 0x7f4aa40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aa98c7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40cfba0 0x7f4aa40c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60878/0 (socket says 172.21.15.59:60878) 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aa98c7640 1 -- 172.21.15.59:0/572772144 learned_addr learned my addr 172.21.15.59:0/572772144 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aaa0c8640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa415e150 0x7f4aa415b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aa90c6640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa40d1060 0x7f4aa415fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aa98c7640 1 -- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa415e150 msgr2=0x7f4aa415b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aa98c7640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa415e150 0x7f4aa415b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aa98c7640 1 -- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa40d1060 msgr2=0x7f4aa415fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aa98c7640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa40d1060 0x7f4aa415fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aa98c7640 1 -- 172.21.15.59:0/572772144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa409fd30 con 0x7f4aa40cfba0 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aa90c6640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa40d1060 0x7f4aa415fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aa98c7640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40cfba0 0x7f4aa40c9f90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0000c00 tx=0x7f4aa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4a9a7fc640 1 -- 172.21.15.59:0/572772144 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0032840 con 0x7f4aa40cfba0 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4a9a7fc640 1 -- 172.21.15.59:0/572772144 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa00329e0 con 0x7f4aa40cfba0 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4aabb2b640 1 -- 172.21.15.59:0/572772144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa40cd890 con 0x7f4aa40cfba0 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.022+0000 7f4a9a7fc640 1 -- 172.21.15.59:0/572772144 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0032dd0 con 0x7f4aa40cfba0 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.023+0000 7f4aabb2b640 1 -- 172.21.15.59:0/572772144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa40cde50 con 0x7f4aa40cfba0 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.023+0000 7f4aabb2b640 1 -- 172.21.15.59:0/572772144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa4069ee0 con 0x7f4aa40cfba0 2024-03-06T07:26:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.024+0000 7f4a9a7fc640 1 -- 172.21.15.59:0/572772144 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4aa004d3f0 con 0x7f4aa40cfba0 2024-03-06T07:26:53.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.024+0000 7f4a9a7fc640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8806b4d0 0x7f4a8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.025+0000 7f4a9a7fc640 1 -- 172.21.15.59:0/572772144 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f4aa002f030 con 0x7f4aa40cfba0 2024-03-06T07:26:53.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.025+0000 7f4aa90c6640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8806b4d0 0x7f4a8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.025+0000 7f4aa90c6640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8806b4d0 0x7f4a8806d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40b55b0 tx=0x7f4a94007c10 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.029+0000 7f4a9a7fc640 1 -- 172.21.15.59:0/572772144 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4aa0040070 con 0x7f4aa40cfba0 2024-03-06T07:26:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.352+0000 7f4aabb2b640 1 -- 172.21.15.59:0/572772144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4aa40d4670 con 0x7f4aa40cfba0 2024-03-06T07:26:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.352+0000 7f4a9a7fc640 1 -- 172.21.15.59:0/572772144 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f4aa007aab0 con 0x7f4aa40cfba0 2024-03-06T07:26:53.354 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:53.355 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:53.355 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:53.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.355+0000 7f4aabb2b640 1 -- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8806b4d0 msgr2=0x7f4a8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.355+0000 7f4aabb2b640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8806b4d0 0x7f4a8806d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40b55b0 tx=0x7f4a94007c10 comp rx=0 tx=0).stop 2024-03-06T07:26:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.355+0000 7f4aabb2b640 1 -- 172.21.15.59:0/572772144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40cfba0 msgr2=0x7f4aa40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.355+0000 7f4aabb2b640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40cfba0 0x7f4aa40c9f90 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0000c00 tx=0x7f4aa00046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.356+0000 7f4aabb2b640 1 -- 172.21.15.59:0/572772144 shutdown_connections 2024-03-06T07:26:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.356+0000 7f4aabb2b640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8806b4d0 0x7f4a8806d990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.356+0000 7f4aabb2b640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa415e150 0x7f4aa415b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.356+0000 7f4aabb2b640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa40d1060 0x7f4aa415fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.356+0000 7f4aabb2b640 1 --2- 172.21.15.59:0/572772144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa40cfba0 0x7f4aa40c9f90 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.356+0000 7f4aabb2b640 1 -- 172.21.15.59:0/572772144 >> 172.21.15.59:0/572772144 conn(0x7f4aa409d230 msgr2=0x7f4aa4161020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.356+0000 7f4aabb2b640 1 -- 172.21.15.59:0/572772144 shutdown_connections 2024-03-06T07:26:53.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:53.356+0000 7f4aabb2b640 1 -- 172.21.15.59:0/572772144 wait complete. 2024-03-06T07:26:53.374 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:53.561 INFO:teuthology.orchestra.run.smithi059.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1937,"msgr_send_messages":1941,"msgr_recv_bytes":565970,"msgr_send_bytes":5884671,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.170576753,"msgr_running_send_time":0.084827000,"msgr_running_recv_time":0.029534195,"msgr_running_fast_dispatch_time":0.042584078,"msgr_send_messages_queue_lat":{"avgcount":1939,"sum":0.420104647,"avgtime":0.000216660},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000225,"avgtime":0.000000112},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11183,"msgr_send_messages":25542,"msgr_recv_bytes":4305241,"msgr_send_bytes":25176867,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.146647995,"msgr_running_send_time":0.636732096,"msgr_running_recv_time":0.568580864,"msgr_running_fast_dispatch_time":0.036459609,"msgr_send_messages_queue_lat":{"avgcount":25540,"sum":0.471664844,"avgtime":0.000018467},"msgr_handle_ack_lat":{"avgcount":30666,"sum":0.005566984,"avgtime":0.000000181},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95808},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1342,"msgr_send_messages":1338,"msgr_recv_bytes":493275,"msgr_send_bytes":5667649,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.117641782,"msgr_running_send_time":0.042434091,"msgr_running_recv_time":0.028507183,"msgr_running_fast_dispatch_time":0.040485880,"msgr_send_messages_queue_lat":{"avgcount":1338,"sum":0.791775611,"avgtime":0.000591760},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":109120,"msgr_send_encrypted_bytes":8992},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":120,"sum":7.771173080,"avgtime":0.064759775}},"finisher-PurgeQueue":{"queue_len":1,"complete_latency":{"avgcount":4183,"sum":1.131809058,"avgtime":0.000270573}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.587841570,"avgtime":0.001495529},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":5035,"inodes_top":0,"inodes_bottom":5035,"inodes_pin_tail":0,"inodes_pinned":5035,"inodes_expired":0,"inodes_with_caps":1385,"caps":1385,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":888,"ceph_cap_op_grant":7552,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3458,"handle_client_caps_dirty":3000,"handle_client_cap_release":9,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":260,"num_strays_delayed":260,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":2941,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":12802,"evex":11237,"evtrm":11237,"ev":1568,"evexg":0,"evexd":11236,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":28217888,"wrpos":29161000,"rdpos":4195244,"jlat":{"avgcount":23,"sum":4.602978670,"avgtime":0.200129507},"replayed":3},"mds_mem":{"ino":1837,"ino+":3214,"ino-":1377,"dir":13,"dir+":13,"dir-":0,"dn":1835,"dn+":6412,"dn-":4577,"cap":1,"cap+":3202,"cap-":3201,"rss":117944,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.113519323,"avgtime":0.000972974},"req_getattr_latency":{"avgcount":6,"sum":0.001174100,"avgtime":0.000195683},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002442159,"avgtime":0.002442159},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.451304597,"avgtime":0.150434865},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.455689955,"avgtime":0.455689955},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.563711436,"avgtime":0.001738659},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5855,"average_load":5855,"avg_session_uptime":849,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":5185252,"buffer_anon_items":2253,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14899416,"mds_co_items":161817,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":6,"op_laggy":0,"op_send":4537,"op_send_bytes":25980224,"op_resend":0,"op_reply":4531,"op_latency":{"avgcount":4531,"sum":211.450341796,"avgtime":0.046667477},"op_inflight":5,"oplen_avg":{"avgcount":4537,"sum":8296},"op":4537,"op_r":54,"op_w":4483,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1836,"osdop_read":26,"osdop_write":41,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2561,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3672,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":50,"omap_rd":48,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":6,"pq_executing_ops_high_water":6,"pq_executing":6,"pq_executing_high_water":6,"pq_executed_ops":2534,"pq_executed":2534,"pq_item_in_journal":407},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14459,"get_sum":4060659,"get_or_fail_fail":0,"get_or_fail_success":14459,"take":0,"take_sum":0,"put":14459,"put_sum":4060659,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4537,"take_sum":26278382,"put":4531,"put_sum":26278382,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":6,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4537,"take_sum":4537,"put":4531,"put_sum":4531,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2941,"get_sum":297041,"get_or_fail_fail":0,"get_or_fail_success":2941,"take":0,"take_sum":0,"put":18,"put_sum":297041,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a5a2aa2820":{"val":939431,"max":3758096384,"get_started":0,"get":12802,"get_sum":24965315,"get_or_fail_fail":0,"get_or_fail_success":12802,"take":0,"take_sum":0,"put":23,"put_sum":24025884,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:53.562 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.12e-07, "sum": 2.25e-07 }, "msgr_recv_bytes": 565970, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1937, "msgr_running_fast_dispatch_time": 0.042584078, "msgr_running_recv_time": 0.029534195, "msgr_running_send_time": 0.084827, "msgr_running_total_time": 0.170576753, "msgr_send_bytes": 5884671, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1941, "msgr_send_messages_queue_lat": { "avgcount": 1939, "avgtime": 0.00021666, "sum": 0.420104647 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 30666, "avgtime": 1.81e-07, "sum": 0.005566984 }, "msgr_recv_bytes": 4305241, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 11183, "msgr_running_fast_dispatch_time": 0.036459609, "msgr_running_recv_time": 0.568580864, "msgr_running_send_time": 0.636732096, "msgr_running_total_time": 1.146647995, "msgr_send_bytes": 25176867, "msgr_send_encrypted_bytes": 95808, "msgr_send_messages": 25542, "msgr_send_messages_queue_lat": { "avgcount": 25540, "avgtime": 1.8467e-05, "sum": 0.471664844 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 493275, "msgr_recv_encrypted_bytes": 109120, "msgr_recv_messages": 1342, "msgr_running_fast_dispatch_time": 0.04048588, "msgr_running_recv_time": 0.028507183, "msgr_running_send_time": 0.042434091, "msgr_running_total_time": 0.117641782, "msgr_send_bytes": 5667649, "msgr_send_encrypted_bytes": 8992, "msgr_send_messages": 1338, "msgr_send_messages_queue_lat": { "avgcount": 1338, "avgtime": 0.00059176, "sum": 0.791775611 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 120, "avgtime": 0.064759775, "sum": 7.77117308 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4183, "avgtime": 0.000270573, "sum": 1.131809058 }, "queue_len": 1 }, "mds": { "caps": 1385, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7552, "ceph_cap_op_revoke": 888, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 9, "handle_client_caps": 3458, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5035, "inodes_bottom": 5035, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5035, "inodes_top": 0, "inodes_with_caps": 1385, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001495529, "sum": 9.58784157 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 260, "num_strays_delayed": 260, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 2941, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1568, "evadd": 12802, "evex": 11237, "evexd": 11236, "evexg": 0, "evlrg": 0, "evtrm": 11237, "expos": 28217888, "jlat": { "avgcount": 23, "avgtime": 0.200129507, "sum": 4.60297867 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 29161000 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1835, "dn+": 6412, "dn-": 4577, "heap": 248164, "ino": 1837, "ino+": 3214, "ino-": 1377, "rss": 117944 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000972974, "sum": 3.113519323 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195683, "sum": 0.0011741 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002442159, "sum": 0.002442159 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.150434865, "sum": 0.451304597 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.455689955, "sum": 0.455689955 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001738659, "sum": 5.563711436 } }, "mds_sessions": { "average_load": 5855, "avg_session_uptime": 849, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5855 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 5185252, "buffer_anon_items": 2253, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 14899416, "mds_co_items": 161817, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 48, "omap_wr": 50, "op": 4537, "op_Infinitylight": 5, "op_active": 6, "op_laggy": 0, "op_latency": { "avgcount": 4531, "avgtime": 0.046667477, "sum": 211.450341796 }, "op_pg": 0, "op_r": 54, "op_reply": 4531, "op_resend": 0, "op_rmw": 0, "op_send": 4537, "op_send_bytes": 25980224, "op_w": 4483, "oplen_avg": { "avgcount": 4537, "sum": 8296 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1836, "osdop_delete": 2561, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 26, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3672, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 41, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 2534, "pq_executed_ops": 2534, "pq_executing": 6, "pq_executing_high_water": 6, "pq_executing_ops": 6, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 407 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14459, "get_or_fail_fail": 0, "get_or_fail_success": 14459, "get_started": 0, "get_sum": 4060659, "max": 104857600, "put": 14459, "put_sum": 4060659, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4531, "put_sum": 26278382, "take": 4537, "take_sum": 26278382, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4531, "put_sum": 4531, "take": 4537, "take_sum": 4537, "val": 6, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2941, "get_or_fail_fail": 0, "get_or_fail_success": 2941, "get_started": 0, "get_sum": 297041, "max": 3758096384, "put": 18, "put_sum": 297041, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a5a2aa2820": { "get": 12802, "get_or_fail_fail": 0, "get_or_fail_success": 12802, "get_started": 0, "get_sum": 24965315, "max": 3758096384, "put": 23, "put_sum": 24025884, "take": 0, "take_sum": 0, "val": 939431, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:53.563 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/260, 2534/3201 2024-03-06T07:26:54.563 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:54.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.663+0000 7fbb7acaa640 1 Processor -- start 2024-03-06T07:26:54.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.663+0000 7fbb7acaa640 1 -- start start 2024-03-06T07:26:54.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb7acaa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb74155380 0x7fbb7406d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb7acaa640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7406ded0 con 0x7fbb7406c6a0 2024-03-06T07:26:54.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb7acaa640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7406e1b0 con 0x7fbb7406d950 2024-03-06T07:26:54.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb7acaa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740ae430 con 0x7fbb74155380 2024-03-06T07:26:54.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb79247640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbb7406c6a0 0x7fbb740aeb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35960/0 (socket says 172.21.15.59:35960) 2024-03-06T07:26:54.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb79247640 1 -- 172.21.15.59:0/3025396023 learned_addr learned my addr 172.21.15.59:0/3025396023 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:54.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb78a46640 1 --2- 172.21.15.59:0/3025396023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb74155380 0x7fbb7406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb737fe640 1 -- 172.21.15.59:0/3025396023 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412743506 0 0) 0x7fbb7406ded0 con 0x7fbb7406c6a0 2024-03-06T07:26:54.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb737fe640 1 -- 172.21.15.59:0/3025396023 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb64003520 con 0x7fbb7406c6a0 2024-03-06T07:26:54.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb737fe640 1 -- 172.21.15.59:0/3025396023 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377601837 0 0) 0x7fbb7406e1b0 con 0x7fbb7406d950 2024-03-06T07:26:54.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb737fe640 1 -- 172.21.15.59:0/3025396023 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb7406ded0 con 0x7fbb7406d950 2024-03-06T07:26:54.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb78a46640 1 -- 172.21.15.59:0/3025396023 >> v1:172.21.15.164:6789/0 conn(0x7fbb7406d950 legacy=0x7fbb7406bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb78a46640 1 -- 172.21.15.59:0/3025396023 >> v1:172.21.15.59:6789/0 conn(0x7fbb7406c6a0 legacy=0x7fbb740aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb737fe640 1 -- 172.21.15.59:0/3025396023 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1960254203 0 0) 0x7fbb64003520 con 0x7fbb7406c6a0 2024-03-06T07:26:54.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.664+0000 7fbb78a46640 1 -- 172.21.15.59:0/3025396023 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7406e490 con 0x7fbb74155380 2024-03-06T07:26:54.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.665+0000 7fbb78a46640 1 --2- 172.21.15.59:0/3025396023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb74155380 0x7fbb7406d410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbb680097d0 tx=0x7fbb6802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b444c5214ce0e09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.665+0000 7fbb737fe640 1 -- 172.21.15.59:0/3025396023 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68037070 con 0x7fbb74155380 2024-03-06T07:26:54.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.665+0000 7fbb737fe640 1 -- 172.21.15.59:0/3025396023 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb68030510 con 0x7fbb74155380 2024-03-06T07:26:54.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.665+0000 7fbb737fe640 1 -- 172.21.15.59:0/3025396023 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0019f0 con 0x7fbb74155380 2024-03-06T07:26:54.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.666+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3025396023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb74155380 msgr2=0x7fbb7406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.666+0000 7fbb7acaa640 1 --2- 172.21.15.59:0/3025396023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb74155380 0x7fbb7406d410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbb680097d0 tx=0x7fbb6802b800 comp rx=0 tx=0).stop 2024-03-06T07:26:54.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.666+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3025396023 shutdown_connections 2024-03-06T07:26:54.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.666+0000 7fbb7acaa640 1 --2- 172.21.15.59:0/3025396023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb74155380 0x7fbb7406d410 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.666+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3025396023 >> 172.21.15.59:0/3025396023 conn(0x7fbb740b2980 msgr2=0x7fbb74071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.666+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3025396023 shutdown_connections 2024-03-06T07:26:54.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.666+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3025396023 wait complete. 2024-03-06T07:26:54.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.668+0000 7fbb7acaa640 1 Processor -- start 2024-03-06T07:26:54.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.668+0000 7fbb7acaa640 1 -- start start 2024-03-06T07:26:54.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.669+0000 7fbb7acaa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb7406c6a0 0x7fbb74143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.669+0000 7fbb7acaa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb7406d950 0x7fbb741e9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.669+0000 7fbb7acaa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb74155380 0x7fbb74151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.669+0000 7fbb7acaa640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7406f450 con 0x7fbb74155380 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.669+0000 7fbb7acaa640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7406e770 con 0x7fbb7406c6a0 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.669+0000 7fbb7acaa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7406f5d0 con 0x7fbb7406d950 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.669+0000 7fbb79247640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb74155380 0x7fbb74151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.669+0000 7fbb79247640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb74155380 0x7fbb74151f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48966/0 (socket says 172.21.15.59:48966) 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.669+0000 7fbb79247640 1 -- 172.21.15.59:0/3134620013 learned_addr learned my addr 172.21.15.59:0/3134620013 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb73fff640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb7406d950 0x7fbb741e9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb78a46640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb7406c6a0 0x7fbb74143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb79247640 1 -- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb7406d950 msgr2=0x7fbb741e9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb79247640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb7406d950 0x7fbb741e9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb79247640 1 -- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb7406c6a0 msgr2=0x7fbb74143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb79247640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb7406c6a0 0x7fbb74143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb79247640 1 -- 172.21.15.59:0/3134620013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb74144490 con 0x7fbb74155380 2024-03-06T07:26:54.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb79247640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb74155380 0x7fbb74151f70 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c001f30 tx=0x7fbb6c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb717fa640 1 -- 172.21.15.59:0/3134620013 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c017070 con 0x7fbb74155380 2024-03-06T07:26:54.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb717fa640 1 -- 172.21.15.59:0/3134620013 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c012400 con 0x7fbb74155380 2024-03-06T07:26:54.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3134620013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb741e9e90 con 0x7fbb74155380 2024-03-06T07:26:54.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb717fa640 1 -- 172.21.15.59:0/3134620013 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0127f0 con 0x7fbb74155380 2024-03-06T07:26:54.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.670+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3134620013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb741f8300 con 0x7fbb74155380 2024-03-06T07:26:54.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.671+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3134620013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb74069ee0 con 0x7fbb74155380 2024-03-06T07:26:54.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.672+0000 7fbb717fa640 1 -- 172.21.15.59:0/3134620013 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb6c01a070 con 0x7fbb74155380 2024-03-06T07:26:54.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.672+0000 7fbb717fa640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbb5806b4d0 0x7fbb5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.672+0000 7fbb717fa640 1 -- 172.21.15.59:0/3134620013 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fbb6c08a1f0 con 0x7fbb74155380 2024-03-06T07:26:54.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.672+0000 7fbb78a46640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbb5806b4d0 0x7fbb5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.673+0000 7fbb78a46640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbb5806b4d0 0x7fbb5806d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbb68034000 tx=0x7fbb68008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:54.676+0000 7fbb717fa640 1 -- 172.21.15.59:0/3134620013 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb74144490 con 0x7fbb74155380 2024-03-06T07:26:55.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.050+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3134620013 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb74072780 con 0x7fbb74155380 2024-03-06T07:26:55.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.051+0000 7fbb717fa640 1 -- 172.21.15.59:0/3134620013 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fbb6c058930 con 0x7fbb74155380 2024-03-06T07:26:55.053 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:55.053 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbb5806b4d0 msgr2=0x7fbb5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbb5806b4d0 0x7fbb5806d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbb68034000 tx=0x7fbb68008030 comp rx=0 tx=0).stop 2024-03-06T07:26:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3134620013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb74155380 msgr2=0x7fbb74151f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb74155380 0x7fbb74151f70 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c001f30 tx=0x7fbb6c00a430 comp rx=0 tx=0).stop 2024-03-06T07:26:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3134620013 shutdown_connections 2024-03-06T07:26:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbb5806b4d0 0x7fbb5806d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb74155380 0x7fbb74151f70 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb7406d950 0x7fbb741e9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 --2- 172.21.15.59:0/3134620013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb7406c6a0 0x7fbb74143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3134620013 >> 172.21.15.59:0/3134620013 conn(0x7fbb740b2980 msgr2=0x7fbb7415c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3134620013 shutdown_connections 2024-03-06T07:26:55.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.053+0000 7fbb7acaa640 1 -- 172.21.15.59:0/3134620013 wait complete. 2024-03-06T07:26:55.066 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:55.184 INFO:teuthology.orchestra.run.smithi059.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2189,"msgr_send_messages":2191,"msgr_recv_bytes":629978,"msgr_send_bytes":5966171,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.187069992,"msgr_running_send_time":0.090766149,"msgr_running_recv_time":0.033659478,"msgr_running_fast_dispatch_time":0.047533968,"msgr_send_messages_queue_lat":{"avgcount":2189,"sum":0.427509999,"avgtime":0.000195299},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000225,"avgtime":0.000000112},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11400,"msgr_send_messages":25758,"msgr_recv_bytes":4386773,"msgr_send_bytes":25275077,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.163378039,"msgr_running_send_time":0.643938374,"msgr_running_recv_time":0.572188614,"msgr_running_fast_dispatch_time":0.041014266,"msgr_send_messages_queue_lat":{"avgcount":25756,"sum":0.478889108,"avgtime":0.000018593},"msgr_handle_ack_lat":{"avgcount":30667,"sum":0.005567113,"avgtime":0.000000181},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":97600},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1552,"msgr_send_messages":1546,"msgr_recv_bytes":546591,"msgr_send_bytes":6831633,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.133980480,"msgr_running_send_time":0.049180921,"msgr_running_recv_time":0.031740332,"msgr_running_fast_dispatch_time":0.045647937,"msgr_send_messages_queue_lat":{"avgcount":1546,"sum":0.797361912,"avgtime":0.000515758},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":109120,"msgr_send_encrypted_bytes":8992},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":123,"sum":7.890998986,"avgtime":0.064154463}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5367,"sum":1.179663529,"avgtime":0.000219799}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.587841570,"avgtime":0.001495529},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":1835,"inodes_top":0,"inodes_bottom":1835,"inodes_pin_tail":0,"inodes_pinned":1835,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":888,"ceph_cap_op_grant":7552,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3458,"handle_client_caps_dirty":3000,"handle_client_cap_release":9,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":13062,"evex":11237,"evtrm":11237,"ev":1828,"evexg":0,"evexd":11236,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":28217888,"wrpos":29317589,"rdpos":4195244,"jlat":{"avgcount":24,"sum":4.612036350,"avgtime":0.192168181},"replayed":3},"mds_mem":{"ino":1837,"ino+":3214,"ino-":1377,"dir":13,"dir+":13,"dir-":0,"dn":1835,"dn+":6412,"dn-":4577,"cap":1,"cap+":3202,"cap-":3201,"rss":117944,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.113519323,"avgtime":0.000972974},"req_getattr_latency":{"avgcount":6,"sum":0.001174100,"avgtime":0.000195683},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002442159,"avgtime":0.002442159},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.451304597,"avgtime":0.150434865},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.455689955,"avgtime":0.455689955},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.563711436,"avgtime":0.001738659},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5855,"average_load":5855,"avg_session_uptime":854,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":14924733,"buffer_anon_items":3748,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":5829296,"mds_co_items":96399,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5208,"op_send_bytes":27102766,"op_resend":0,"op_reply":5208,"op_latency":{"avgcount":5208,"sum":212.463329057,"avgtime":0.040795570},"op_inflight":0,"oplen_avg":{"avgcount":5208,"sum":8968},"op":5208,"op_r":57,"op_w":5151,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1836,"osdop_read":29,"osdop_write":45,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3672,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":51,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15137,"get_sum":4198241,"get_or_fail_fail":0,"get_or_fail_success":15137,"take":0,"take_sum":0,"put":15137,"put_sum":4198241,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5208,"take_sum":27427184,"put":5208,"put_sum":27427184,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5208,"take_sum":5208,"put":5208,"put_sum":5208,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":21,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a5a2aa2820":{"val":0,"max":3758096384,"get_started":0,"get":13062,"get_sum":25121904,"get_or_fail_fail":0,"get_or_fail_success":13062,"take":0,"take_sum":0,"put":24,"put_sum":25121904,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:55.186 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.12e-07, "sum": 2.25e-07 }, "msgr_recv_bytes": 629978, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2189, "msgr_running_fast_dispatch_time": 0.047533968, "msgr_running_recv_time": 0.033659478, "msgr_running_send_time": 0.090766149, "msgr_running_total_time": 0.187069992, "msgr_send_bytes": 5966171, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2191, "msgr_send_messages_queue_lat": { "avgcount": 2189, "avgtime": 0.000195299, "sum": 0.427509999 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 30667, "avgtime": 1.81e-07, "sum": 0.005567113 }, "msgr_recv_bytes": 4386773, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 11400, "msgr_running_fast_dispatch_time": 0.041014266, "msgr_running_recv_time": 0.572188614, "msgr_running_send_time": 0.643938374, "msgr_running_total_time": 1.163378039, "msgr_send_bytes": 25275077, "msgr_send_encrypted_bytes": 97600, "msgr_send_messages": 25758, "msgr_send_messages_queue_lat": { "avgcount": 25756, "avgtime": 1.8593e-05, "sum": 0.478889108 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 546591, "msgr_recv_encrypted_bytes": 109120, "msgr_recv_messages": 1552, "msgr_running_fast_dispatch_time": 0.045647937, "msgr_running_recv_time": 0.031740332, "msgr_running_send_time": 0.049180921, "msgr_running_total_time": 0.13398048, "msgr_send_bytes": 6831633, "msgr_send_encrypted_bytes": 8992, "msgr_send_messages": 1546, "msgr_send_messages_queue_lat": { "avgcount": 1546, "avgtime": 0.000515758, "sum": 0.797361912 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 123, "avgtime": 0.064154463, "sum": 7.890998986 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5367, "avgtime": 0.000219799, "sum": 1.179663529 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7552, "ceph_cap_op_revoke": 888, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 9, "handle_client_caps": 3458, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1835, "inodes_bottom": 1835, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 1835, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001495529, "sum": 9.58784157 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1828, "evadd": 13062, "evex": 11237, "evexd": 11236, "evexg": 0, "evlrg": 0, "evtrm": 11237, "expos": 28217888, "jlat": { "avgcount": 24, "avgtime": 0.192168181, "sum": 4.61203635 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 29317589 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1835, "dn+": 6412, "dn-": 4577, "heap": 248164, "ino": 1837, "ino+": 3214, "ino-": 1377, "rss": 117944 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000972974, "sum": 3.113519323 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195683, "sum": 0.0011741 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002442159, "sum": 0.002442159 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.150434865, "sum": 0.451304597 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.455689955, "sum": 0.455689955 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001738659, "sum": 5.563711436 } }, "mds_sessions": { "average_load": 5855, "avg_session_uptime": 854, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5855 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 14924733, "buffer_anon_items": 3748, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 5829296, "mds_co_items": 96399, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 51, "op": 5208, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5208, "avgtime": 0.04079557, "sum": 212.463329057 }, "op_pg": 0, "op_r": 57, "op_reply": 5208, "op_resend": 0, "op_rmw": 0, "op_send": 5208, "op_send_bytes": 27102766, "op_w": 5151, "oplen_avg": { "avgcount": 5208, "sum": 8968 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1836, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 29, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3672, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 45, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15137, "get_or_fail_fail": 0, "get_or_fail_success": 15137, "get_started": 0, "get_sum": 4198241, "max": 104857600, "put": 15137, "put_sum": 4198241, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5208, "put_sum": 27427184, "take": 5208, "take_sum": 27427184, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5208, "put_sum": 5208, "take": 5208, "take_sum": 5208, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 21, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a5a2aa2820": { "get": 13062, "get_or_fail_fail": 0, "get_or_fail_success": 13062, "get_started": 0, "get_sum": 25121904, "max": 3758096384, "put": 24, "put_sum": 25121904, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:55.186 INFO:tasks.cephfs.test_strays:Complete purge in 7 seconds 2024-03-06T07:26:55.186 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:55.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.286+0000 7f7eb4593640 1 Processor -- start 2024-03-06T07:26:55.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.286+0000 7f7eb4593640 1 -- start start 2024-03-06T07:26:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.287+0000 7f7eb4593640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7eac0d0db0 0x7f7eac0b9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.287+0000 7f7eb4593640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac0d1ea0 con 0x7f7eac15e150 2024-03-06T07:26:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.287+0000 7f7eb4593640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac0d2180 con 0x7f7eac0d0950 2024-03-06T07:26:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.287+0000 7f7eb4593640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac0d2ab0 con 0x7f7eac0d0db0 2024-03-06T07:26:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.287+0000 7f7eb232f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7eac15e150 0x7f7eac0cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35970/0 (socket says 172.21.15.59:35970) 2024-03-06T07:26:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.287+0000 7f7eb232f640 1 -- 172.21.15.59:0/3328232937 learned_addr learned my addr 172.21.15.59:0/3328232937 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.287+0000 7f7eb2b30640 1 --2- 172.21.15.59:0/3328232937 >> v2:172.21.15.164:3300/0 conn(0x7f7eac0d0db0 0x7f7eac0b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.287+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584263402 0 0) 0x7f7eac0d1ea0 con 0x7f7eac15e150 2024-03-06T07:26:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e98003520 con 0x7f7eac15e150 2024-03-06T07:26:55.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 398774143 0 0) 0x7f7e98003520 con 0x7f7eac15e150 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 >> v2:172.21.15.164:3300/0 conn(0x7f7eac0d0db0 msgr2=0x7f7eac0b9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 --2- 172.21.15.59:0/3328232937 >> v2:172.21.15.164:3300/0 conn(0x7f7eac0d0db0 0x7f7eac0b9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 >> v1:172.21.15.164:6789/0 conn(0x7f7eac0d0950 legacy=0x7f7eac0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac0d2460 con 0x7f7eac15e150 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f7e9c001670 con 0x7f7eac15e150 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 >> v1:172.21.15.59:6789/0 conn(0x7f7eac15e150 legacy=0x7f7eac0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 --2- 172.21.15.59:0/3328232937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e980039b0 0x7f7e98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 --2- 172.21.15.59:0/3328232937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 0x7f7e98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 --2- 172.21.15.59:0/3328232937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e980051c0 0x7f7e9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac0d2ab0 con 0x7f7e98004800 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9800c160 con 0x7f7e980039b0 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9800c360 con 0x7f7e980051c0 2024-03-06T07:26:55.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb1b2e640 1 --2- 172.21.15.59:0/3328232937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 0x7f7e98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb232f640 1 --2- 172.21.15.59:0/3328232937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e980039b0 0x7f7e98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb2b30640 1 --2- 172.21.15.59:0/3328232937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e980051c0 0x7f7e9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb1b2e640 1 -- 172.21.15.59:0/3328232937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e980051c0 msgr2=0x7f7e9800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb1b2e640 1 --2- 172.21.15.59:0/3328232937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e980051c0 0x7f7e9800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb1b2e640 1 -- 172.21.15.59:0/3328232937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e980039b0 msgr2=0x7f7e98003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb1b2e640 1 --2- 172.21.15.59:0/3328232937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e980039b0 0x7f7e98003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb1b2e640 1 -- 172.21.15.59:0/3328232937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e98003520 con 0x7f7e98004800 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.288+0000 7f7eb2b30640 1 -- 172.21.15.59:0/3328232937 reap_dead start 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.289+0000 7f7eb1b2e640 1 --2- 172.21.15.59:0/3328232937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 0x7f7e98004c80 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8001750 tx=0x7f7e98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391e5f770da23b49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.289+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea8001370 con 0x7f7e98004800 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.289+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea80052d0 con 0x7f7e98004800 2024-03-06T07:26:55.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.289+0000 7f7eb132d640 1 -- 172.21.15.59:0/3328232937 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea8003830 con 0x7f7e98004800 2024-03-06T07:26:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.290+0000 7f7eb4593640 1 -- 172.21.15.59:0/3328232937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 msgr2=0x7f7e98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.290+0000 7f7eb4593640 1 --2- 172.21.15.59:0/3328232937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 0x7f7e98004c80 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8001750 tx=0x7f7e98005830 comp rx=0 tx=0).stop 2024-03-06T07:26:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.290+0000 7f7eb4593640 1 -- 172.21.15.59:0/3328232937 shutdown_connections 2024-03-06T07:26:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.290+0000 7f7eb4593640 1 --2- 172.21.15.59:0/3328232937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 0x7f7e98004c80 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.290+0000 7f7eb4593640 1 -- 172.21.15.59:0/3328232937 >> 172.21.15.59:0/3328232937 conn(0x7f7eac09d230 msgr2=0x7f7eac0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.290+0000 7f7eb4593640 1 -- 172.21.15.59:0/3328232937 shutdown_connections 2024-03-06T07:26:55.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.290+0000 7f7eb4593640 1 -- 172.21.15.59:0/3328232937 wait complete. 2024-03-06T07:26:55.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.291+0000 7f7eb4593640 1 Processor -- start 2024-03-06T07:26:55.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.291+0000 7f7eb4593640 1 -- start start 2024-03-06T07:26:55.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb4593640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 0x7f7eac15be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb4593640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eac0b64a0 0x7f7eac1f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb4593640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7eac1f4720 0x7f7eac0c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb4593640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac0d2f40 con 0x7f7e98004800 2024-03-06T07:26:55.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb4593640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac0d1220 con 0x7f7eac0b64a0 2024-03-06T07:26:55.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb4593640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac0d1b30 con 0x7f7eac1f4720 2024-03-06T07:26:55.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb232f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 0x7f7eac15be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb232f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 0x7f7eac15be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48988/0 (socket says 172.21.15.59:48988) 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb232f640 1 -- 172.21.15.59:0/886415698 learned_addr learned my addr 172.21.15.59:0/886415698 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb2b30640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7eac1f4720 0x7f7eac0c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb1b2e640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eac0b64a0 0x7f7eac1f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb232f640 1 -- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7eac1f4720 msgr2=0x7f7eac0c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb232f640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7eac1f4720 0x7f7eac0c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb232f640 1 -- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eac0b64a0 msgr2=0x7f7eac1f41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb232f640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eac0b64a0 0x7f7eac1f41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb232f640 1 -- 172.21.15.59:0/886415698 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac15c7a0 con 0x7f7e98004800 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7eb232f640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 0x7f7eac15be00 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0072f0 tx=0x7f7e9c0077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.292+0000 7f7ea2ffd640 1 -- 172.21.15.59:0/886415698 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c01a070 con 0x7f7e98004800 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.293+0000 7f7ea2ffd640 1 -- 172.21.15.59:0/886415698 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9c00f960 con 0x7f7e98004800 2024-03-06T07:26:55.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.293+0000 7f7eb4593640 1 -- 172.21.15.59:0/886415698 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac1f4d50 con 0x7f7e98004800 2024-03-06T07:26:55.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.293+0000 7f7ea2ffd640 1 -- 172.21.15.59:0/886415698 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c00fd50 con 0x7f7e98004800 2024-03-06T07:26:55.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.293+0000 7f7eb4593640 1 -- 172.21.15.59:0/886415698 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eac0b59c0 con 0x7f7e98004800 2024-03-06T07:26:55.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.294+0000 7f7eb4593640 1 -- 172.21.15.59:0/886415698 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eac069ee0 con 0x7f7e98004800 2024-03-06T07:26:55.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.294+0000 7f7ea2ffd640 1 -- 172.21.15.59:0/886415698 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e9c01d070 con 0x7f7e98004800 2024-03-06T07:26:55.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.295+0000 7f7ea2ffd640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e9006b500 0x7f7e9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.295+0000 7f7ea2ffd640 1 -- 172.21.15.59:0/886415698 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f7e9c089ac0 con 0x7f7e98004800 2024-03-06T07:26:55.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.295+0000 7f7eb1b2e640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e9006b500 0x7f7e9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.295+0000 7f7eb1b2e640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e9006b500 0x7f7e9006d9c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8033000 tx=0x7f7ea8009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.299+0000 7f7ea2ffd640 1 -- 172.21.15.59:0/886415698 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e9c0581f0 con 0x7f7e98004800 2024-03-06T07:26:55.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.518+0000 7f7eb4593640 1 -- 172.21.15.59:0/886415698 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7eac0d4670 con 0x7f7e98004800 2024-03-06T07:26:55.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.519+0000 7f7ea2ffd640 1 -- 172.21.15.59:0/886415698 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f7e9c05e030 con 0x7f7e98004800 2024-03-06T07:26:55.521 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:55.521 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:55.521 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.520+0000 7f7eb4593640 1 -- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e9006b500 msgr2=0x7f7e9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.521+0000 7f7eb4593640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e9006b500 0x7f7e9006d9c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8033000 tx=0x7f7ea8009030 comp rx=0 tx=0).stop 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.521+0000 7f7eb4593640 1 -- 172.21.15.59:0/886415698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 msgr2=0x7f7eac15be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.521+0000 7f7eb4593640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 0x7f7eac15be00 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0072f0 tx=0x7f7e9c0077b0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.521+0000 7f7eb4593640 1 -- 172.21.15.59:0/886415698 shutdown_connections 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.521+0000 7f7eb4593640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7e9006b500 0x7f7e9006d9c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.521+0000 7f7eb4593640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7eac1f4720 0x7f7eac0c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.521+0000 7f7eb4593640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7eac0b64a0 0x7f7eac1f41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.521+0000 7f7eb4593640 1 --2- 172.21.15.59:0/886415698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e98004800 0x7f7eac15be00 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.521+0000 7f7eb4593640 1 -- 172.21.15.59:0/886415698 >> 172.21.15.59:0/886415698 conn(0x7f7eac09d230 msgr2=0x7f7eac06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.521+0000 7f7eb4593640 1 -- 172.21.15.59:0/886415698 shutdown_connections 2024-03-06T07:26:55.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.521+0000 7f7eb4593640 1 -- 172.21.15.59:0/886415698 wait complete. 2024-03-06T07:26:55.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-03-06T07:26:55.650 INFO:teuthology.orchestra.run.smithi059.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2189,"msgr_send_messages":2191,"msgr_recv_bytes":629978,"msgr_send_bytes":5966171,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.187069992,"msgr_running_send_time":0.090766149,"msgr_running_recv_time":0.033659478,"msgr_running_fast_dispatch_time":0.047533968,"msgr_send_messages_queue_lat":{"avgcount":2189,"sum":0.427509999,"avgtime":0.000195299},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000225,"avgtime":0.000000112},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11401,"msgr_send_messages":25758,"msgr_recv_bytes":4387191,"msgr_send_bytes":25275077,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.163472629,"msgr_running_send_time":0.643969024,"msgr_running_recv_time":0.572209922,"msgr_running_fast_dispatch_time":0.041054198,"msgr_send_messages_queue_lat":{"avgcount":25756,"sum":0.478889108,"avgtime":0.000018593},"msgr_handle_ack_lat":{"avgcount":30668,"sum":0.005567253,"avgtime":0.000000181},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":97600},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1552,"msgr_send_messages":1546,"msgr_recv_bytes":546591,"msgr_send_bytes":6831633,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.133980480,"msgr_running_send_time":0.049180921,"msgr_running_recv_time":0.031740332,"msgr_running_fast_dispatch_time":0.045647937,"msgr_send_messages_queue_lat":{"avgcount":1546,"sum":0.797361912,"avgtime":0.000515758},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":109120,"msgr_send_encrypted_bytes":8992},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":123,"sum":7.890998986,"avgtime":0.064154463}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5367,"sum":1.179663529,"avgtime":0.000219799}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.587841570,"avgtime":0.001495529},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":1835,"inodes_top":0,"inodes_bottom":1835,"inodes_pin_tail":0,"inodes_pinned":1835,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":888,"ceph_cap_op_grant":7552,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3458,"handle_client_caps_dirty":3000,"handle_client_cap_release":9,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":13062,"evex":11237,"evtrm":11237,"ev":1828,"evexg":0,"evexd":11236,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":28217888,"wrpos":29317589,"rdpos":4195244,"jlat":{"avgcount":24,"sum":4.612036350,"avgtime":0.192168181},"replayed":3},"mds_mem":{"ino":13,"ino+":3214,"ino-":3201,"dir":12,"dir+":13,"dir-":1,"dn":1835,"dn+":6412,"dn-":4577,"cap":1,"cap+":3202,"cap-":3201,"rss":117944,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.113519323,"avgtime":0.000972974},"req_getattr_latency":{"avgcount":6,"sum":0.001174100,"avgtime":0.000195683},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.002442159,"avgtime":0.002442159},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.451304597,"avgtime":0.150434865},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.455689955,"avgtime":0.455689955},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.563711436,"avgtime":0.001738659},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5855,"average_load":5855,"avg_session_uptime":854,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":14924733,"buffer_anon_items":3748,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":5829296,"mds_co_items":96399,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5208,"op_send_bytes":27102766,"op_resend":0,"op_reply":5208,"op_latency":{"avgcount":5208,"sum":212.463329057,"avgtime":0.040795570},"op_inflight":0,"oplen_avg":{"avgcount":5208,"sum":8968},"op":5208,"op_r":57,"op_w":5151,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1836,"osdop_read":29,"osdop_write":45,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3672,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":51,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15138,"get_sum":4198569,"get_or_fail_fail":0,"get_or_fail_success":15138,"take":0,"take_sum":0,"put":15138,"put_sum":4198569,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5208,"take_sum":27427184,"put":5208,"put_sum":27427184,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5208,"take_sum":5208,"put":5208,"put_sum":5208,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":21,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55a5a2aa2820":{"val":0,"max":3758096384,"get_started":0,"get":13062,"get_sum":25121904,"get_or_fail_fail":0,"get_or_fail_success":13062,"take":0,"take_sum":0,"put":24,"put_sum":25121904,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:55.651 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.12e-07, "sum": 2.25e-07 }, "msgr_recv_bytes": 629978, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2189, "msgr_running_fast_dispatch_time": 0.047533968, "msgr_running_recv_time": 0.033659478, "msgr_running_send_time": 0.090766149, "msgr_running_total_time": 0.187069992, "msgr_send_bytes": 5966171, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2191, "msgr_send_messages_queue_lat": { "avgcount": 2189, "avgtime": 0.000195299, "sum": 0.427509999 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 30668, "avgtime": 1.81e-07, "sum": 0.005567253 }, "msgr_recv_bytes": 4387191, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 11401, "msgr_running_fast_dispatch_time": 0.041054198, "msgr_running_recv_time": 0.572209922, "msgr_running_send_time": 0.643969024, "msgr_running_total_time": 1.163472629, "msgr_send_bytes": 25275077, "msgr_send_encrypted_bytes": 97600, "msgr_send_messages": 25758, "msgr_send_messages_queue_lat": { "avgcount": 25756, "avgtime": 1.8593e-05, "sum": 0.478889108 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 546591, "msgr_recv_encrypted_bytes": 109120, "msgr_recv_messages": 1552, "msgr_running_fast_dispatch_time": 0.045647937, "msgr_running_recv_time": 0.031740332, "msgr_running_send_time": 0.049180921, "msgr_running_total_time": 0.13398048, "msgr_send_bytes": 6831633, "msgr_send_encrypted_bytes": 8992, "msgr_send_messages": 1546, "msgr_send_messages_queue_lat": { "avgcount": 1546, "avgtime": 0.000515758, "sum": 0.797361912 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 123, "avgtime": 0.064154463, "sum": 7.890998986 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5367, "avgtime": 0.000219799, "sum": 1.179663529 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7552, "ceph_cap_op_revoke": 888, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 9, "handle_client_caps": 3458, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1835, "inodes_bottom": 1835, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 1835, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001495529, "sum": 9.58784157 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1828, "evadd": 13062, "evex": 11237, "evexd": 11236, "evexg": 0, "evlrg": 0, "evtrm": 11237, "expos": 28217888, "jlat": { "avgcount": 24, "avgtime": 0.192168181, "sum": 4.61203635 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 29317589 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 1835, "dn+": 6412, "dn-": 4577, "heap": 248164, "ino": 13, "ino+": 3214, "ino-": 3201, "rss": 117944 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000972974, "sum": 3.113519323 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.000195683, "sum": 0.0011741 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.002442159, "sum": 0.002442159 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.150434865, "sum": 0.451304597 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.455689955, "sum": 0.455689955 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001738659, "sum": 5.563711436 } }, "mds_sessions": { "average_load": 5855, "avg_session_uptime": 854, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5855 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 14924733, "buffer_anon_items": 3748, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 5829296, "mds_co_items": 96399, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 51, "op": 5208, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5208, "avgtime": 0.04079557, "sum": 212.463329057 }, "op_pg": 0, "op_r": 57, "op_reply": 5208, "op_resend": 0, "op_rmw": 0, "op_send": 5208, "op_send_bytes": 27102766, "op_w": 5151, "oplen_avg": { "avgcount": 5208, "sum": 8968 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1836, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 29, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3672, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 45, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15138, "get_or_fail_fail": 0, "get_or_fail_success": 15138, "get_started": 0, "get_sum": 4198569, "max": 104857600, "put": 15138, "put_sum": 4198569, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5208, "put_sum": 27427184, "take": 5208, "take_sum": 27427184, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5208, "put_sum": 5208, "take": 5208, "take_sum": 5208, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 21, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55a5a2aa2820": { "get": 13062, "get_or_fail_fail": 0, "get_or_fail_success": 13062, "get_started": 0, "get_sum": 25121904, "max": 3758096384, "put": 24, "put_sum": 25121904, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:55.651 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:55.651 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:55.695 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:55.745 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:55.745 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:26:55.746 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:55.760 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:55.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.902+0000 7fa603c0e640 1 Processor -- start 2024-03-06T07:26:55.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.903+0000 7fa603c0e640 1 -- start start 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.903+0000 7fa603c0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5fc15e150 0x7fa5fc0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.903+0000 7fa603c0e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa5fc0cfba0 0x7fa5fc0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.903+0000 7fa603c0e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0d1ea0 con 0x7fa5fc0d0e90 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.903+0000 7fa603c0e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5fc0d2ab0 con 0x7fa5fc0cfba0 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.903+0000 7fa603c0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0d2180 con 0x7fa5fc15e150 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.904+0000 7fa6021ab640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa5fc0d0e90 0x7fa5fc0b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35974/0 (socket says 172.21.15.59:35974) 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.904+0000 7fa6011a9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa5fc0cfba0 0x7fa5fc0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.904+0000 7fa6021ab640 1 -- 172.21.15.59:0/1093430868 learned_addr learned my addr 172.21.15.59:0/1093430868 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.904+0000 7fa6019aa640 1 --2- 172.21.15.59:0/1093430868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5fc15e150 0x7fa5fc0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.904+0000 7fa6011a9640 1 -- 172.21.15.59:0/1093430868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5fc15e150 msgr2=0x7fa5fc0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.904+0000 7fa6011a9640 1 --2- 172.21.15.59:0/1093430868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5fc15e150 0x7fa5fc0cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.904+0000 7fa6011a9640 1 -- 172.21.15.59:0/1093430868 >> v1:172.21.15.59:6789/0 conn(0x7fa5fc0d0e90 legacy=0x7fa5fc0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.904+0000 7fa6011a9640 1 -- 172.21.15.59:0/1093430868 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc0d2460 con 0x7fa5fc0cfba0 2024-03-06T07:26:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.904+0000 7fa6009a8640 1 -- 172.21.15.59:0/1093430868 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624630264 0 0) 0x7fa5fc0d1ea0 con 0x7fa5fc0d0e90 2024-03-06T07:26:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa6011a9640 1 --2- 172.21.15.59:0/1093430868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc0cfba0 0x7fa5fc0d0950 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec0097a0 tx=0x7fa5ec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a76dd62d9b92fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa6009a8640 1 -- 172.21.15.59:0/1093430868 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0315a0 con 0x7fa5fc0cfba0 2024-03-06T07:26:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa6009a8640 1 -- 172.21.15.59:0/1093430868 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec037070 con 0x7fa5fc0cfba0 2024-03-06T07:26:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa6009a8640 1 -- 172.21.15.59:0/1093430868 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec031960 con 0x7fa5fc0cfba0 2024-03-06T07:26:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa603c0e640 1 -- 172.21.15.59:0/1093430868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc0cfba0 msgr2=0x7fa5fc0d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa603c0e640 1 --2- 172.21.15.59:0/1093430868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc0cfba0 0x7fa5fc0d0950 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec0097a0 tx=0x7fa5ec02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:55.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa603c0e640 1 -- 172.21.15.59:0/1093430868 shutdown_connections 2024-03-06T07:26:55.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa603c0e640 1 --2- 172.21.15.59:0/1093430868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc0cfba0 0x7fa5fc0d0950 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa603c0e640 1 --2- 172.21.15.59:0/1093430868 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5fc15e150 0x7fa5fc0cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa603c0e640 1 -- 172.21.15.59:0/1093430868 >> 172.21.15.59:0/1093430868 conn(0x7fa5fc09d230 msgr2=0x7fa5fc0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:55.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa603c0e640 1 -- 172.21.15.59:0/1093430868 shutdown_connections 2024-03-06T07:26:55.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.905+0000 7fa603c0e640 1 -- 172.21.15.59:0/1093430868 wait complete. 2024-03-06T07:26:55.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.907+0000 7fa603c0e640 1 Processor -- start 2024-03-06T07:26:55.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.907+0000 7fa603c0e640 1 -- start start 2024-03-06T07:26:55.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.907+0000 7fa603c0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5fc0cfba0 0x7fa5fc0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.907+0000 7fa603c0e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5fc0d0e90 0x7fa5fc1f5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.907+0000 7fa603c0e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc15e150 0x7fa5fc151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.907+0000 7fa603c0e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0d1b30 con 0x7fa5fc15e150 2024-03-06T07:26:55.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.907+0000 7fa603c0e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0d2f40 con 0x7fa5fc0d0e90 2024-03-06T07:26:55.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.907+0000 7fa603c0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc15e530 con 0x7fa5fc0cfba0 2024-03-06T07:26:55.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.907+0000 7fa6021ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc15e150 0x7fa5fc151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.907+0000 7fa6021ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc15e150 0x7fa5fc151d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48992/0 (socket says 172.21.15.59:48992) 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.907+0000 7fa6021ab640 1 -- 172.21.15.59:0/3725154720 learned_addr learned my addr 172.21.15.59:0/3725154720 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa6019aa640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5fc0cfba0 0x7fa5fc0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa6011a9640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5fc0d0e90 0x7fa5fc1f5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa6021ab640 1 -- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5fc0cfba0 msgr2=0x7fa5fc0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa6021ab640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5fc0cfba0 0x7fa5fc0ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa6021ab640 1 -- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5fc0d0e90 msgr2=0x7fa5fc1f5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa6021ab640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5fc0d0e90 0x7fa5fc1f5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa6021ab640 1 -- 172.21.15.59:0/3725154720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc0cd260 con 0x7fa5fc15e150 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa6021ab640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc15e150 0x7fa5fc151d60 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400c7a0 tx=0x7fa5f400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa5f27fc640 1 -- 172.21.15.59:0/3725154720 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f4018070 con 0x7fa5fc15e150 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa5f27fc640 1 -- 172.21.15.59:0/3725154720 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f400a640 con 0x7fa5fc15e150 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa603c0e640 1 -- 172.21.15.59:0/3725154720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc1f4a20 con 0x7fa5fc15e150 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa5f27fc640 1 -- 172.21.15.59:0/3725154720 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f400aab0 con 0x7fa5fc15e150 2024-03-06T07:26:55.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.908+0000 7fa603c0e640 1 -- 172.21.15.59:0/3725154720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc1f46f0 con 0x7fa5fc15e150 2024-03-06T07:26:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.909+0000 7fa603c0e640 1 -- 172.21.15.59:0/3725154720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc069ee0 con 0x7fa5fc15e150 2024-03-06T07:26:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.910+0000 7fa5f27fc640 1 -- 172.21.15.59:0/3725154720 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5f401b070 con 0x7fa5fc15e150 2024-03-06T07:26:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.910+0000 7fa5f27fc640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa5e006b500 0x7fa5e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.910+0000 7fa5f27fc640 1 -- 172.21.15.59:0/3725154720 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa5f4089b00 con 0x7fa5fc15e150 2024-03-06T07:26:55.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.910+0000 7fa6019aa640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa5e006b500 0x7fa5e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:55.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.911+0000 7fa6019aa640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa5e006b500 0x7fa5e006d9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8006fd0 tx=0x7fa5f8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:55.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:55.914+0000 7fa5f27fc640 1 -- 172.21.15.59:0/3725154720 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5f40589f0 con 0x7fa5fc15e150 2024-03-06T07:26:56.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.095+0000 7fa603c0e640 1 -- 172.21.15.59:0/3725154720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5fc06c1b0 con 0x7fa5fc15e150 2024-03-06T07:26:56.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.096+0000 7fa5f27fc640 1 -- 172.21.15.59:0/3725154720 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16987 (secure 0 0 0) 0x7fa5f40582f0 con 0x7fa5fc15e150 2024-03-06T07:26:56.098 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:56.099 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":96,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:26:29.601683+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.489280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:16.183614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6834/2799249020":"2024-03-07T07:26:29.597825+0000","172.21.15.59:6835/1784120157":"2024-03-07T07:26:29.585744+0000","172.21.15.59:6832/2799249020":"2024-03-07T07:26:29.597825+0000","172.21.15.59:6833/1784120157":"2024-03-07T07:26:29.585744+0000","172.21.15.164:6837/4292771408":"2024-03-07T07:26:29.578123+0000","172.21.15.164:6835/4292771408":"2024-03-07T07:26:29.578123+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:56.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 -- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa5e006b500 msgr2=0x7fa5e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa5e006b500 0x7fa5e006d9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8006fd0 tx=0x7fa5f8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:56.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 -- 172.21.15.59:0/3725154720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc15e150 msgr2=0x7fa5fc151d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc15e150 0x7fa5fc151d60 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400c7a0 tx=0x7fa5f400cc60 comp rx=0 tx=0).stop 2024-03-06T07:26:56.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 -- 172.21.15.59:0/3725154720 shutdown_connections 2024-03-06T07:26:56.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa5e006b500 0x7fa5e006d9c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5fc15e150 0x7fa5fc151d60 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5fc0d0e90 0x7fa5fc1f5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 --2- 172.21.15.59:0/3725154720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5fc0cfba0 0x7fa5fc0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 -- 172.21.15.59:0/3725154720 >> 172.21.15.59:0/3725154720 conn(0x7fa5fc09d230 msgr2=0x7fa5fc06b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 -- 172.21.15.59:0/3725154720 shutdown_connections 2024-03-06T07:26:56.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.099+0000 7fa603c0e640 1 -- 172.21.15.59:0/3725154720 wait complete. 2024-03-06T07:26:56.111 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:15.489280+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:16.183614+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:56.112 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:56.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.214+0000 7f4c0ce6b640 1 Processor -- start 2024-03-06T07:26:56.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.214+0000 7f4c0ce6b640 1 -- start start 2024-03-06T07:26:56.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.215+0000 7f4c0ce6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c08155380 0x7f4c0806d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.215+0000 7f4c0ce6b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c0806c6a0 0x7f4c080aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.215+0000 7f4c0ce6b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0806ded0 con 0x7f4c0806d950 2024-03-06T07:26:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.215+0000 7f4c0ce6b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c080ae430 con 0x7f4c0806c6a0 2024-03-06T07:26:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.215+0000 7f4c0ce6b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0806e770 con 0x7f4c08155380 2024-03-06T07:26:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.216+0000 7f4c06d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c0806c6a0 0x7f4c080aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.216+0000 7f4c05d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4c0806d950 0x7f4c0806bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35980/0 (socket says 172.21.15.59:35980) 2024-03-06T07:26:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.216+0000 7f4c05d9b640 1 -- 172.21.15.59:0/3042746111 learned_addr learned my addr 172.21.15.59:0/3042746111 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.216+0000 7f4c0659c640 1 --2- 172.21.15.59:0/3042746111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c08155380 0x7f4c0806d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.216+0000 7f4c06d9d640 1 -- 172.21.15.59:0/3042746111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c08155380 msgr2=0x7f4c0806d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.216+0000 7f4c06d9d640 1 --2- 172.21.15.59:0/3042746111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c08155380 0x7f4c0806d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.216+0000 7f4c06d9d640 1 -- 172.21.15.59:0/3042746111 >> v1:172.21.15.59:6789/0 conn(0x7f4c0806d950 legacy=0x7f4c0806bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.216+0000 7f4c0559a640 1 -- 172.21.15.59:0/3042746111 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036387162 0 0) 0x7f4c0806ded0 con 0x7f4c0806d950 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.216+0000 7f4c06d9d640 1 -- 172.21.15.59:0/3042746111 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0806e1b0 con 0x7f4c0806c6a0 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.217+0000 7f4c06d9d640 1 --2- 172.21.15.59:0/3042746111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0806c6a0 0x7f4c080aeb80 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4c00009dc0 tx=0x7f4c0002be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb63621a0622bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.217+0000 7f4c0559a640 1 -- 172.21.15.59:0/3042746111 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c00039070 con 0x7f4c0806c6a0 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.217+0000 7f4c0559a640 1 -- 172.21.15.59:0/3042746111 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0002cc30 con 0x7f4c0806c6a0 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.217+0000 7f4c0559a640 1 -- 172.21.15.59:0/3042746111 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c00034440 con 0x7f4c0806c6a0 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.217+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3042746111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0806c6a0 msgr2=0x7f4c080aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.217+0000 7f4c0ce6b640 1 --2- 172.21.15.59:0/3042746111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0806c6a0 0x7f4c080aeb80 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4c00009dc0 tx=0x7f4c0002be80 comp rx=0 tx=0).stop 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.218+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3042746111 shutdown_connections 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.218+0000 7f4c0ce6b640 1 --2- 172.21.15.59:0/3042746111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0806c6a0 0x7f4c080aeb80 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.218+0000 7f4c0ce6b640 1 --2- 172.21.15.59:0/3042746111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c08155380 0x7f4c0806d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.218+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3042746111 >> 172.21.15.59:0/3042746111 conn(0x7f4c080b2980 msgr2=0x7f4c08071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.218+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3042746111 shutdown_connections 2024-03-06T07:26:56.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.218+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3042746111 wait complete. 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.219+0000 7f4c0ce6b640 1 Processor -- start 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.219+0000 7f4c0ce6b640 1 -- start start 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c0ce6b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c0806c6a0 0x7f4c081f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c0ce6b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0806d950 0x7f4c08157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c0ce6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c08155380 0x7f4c080b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c0ce6b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0806f450 con 0x7f4c0806d950 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c0ce6b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c080ae660 con 0x7f4c0806c6a0 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c0ce6b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0806f5d0 con 0x7f4c08155380 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c05d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0806d950 0x7f4c08157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c05d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0806d950 0x7f4c08157210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48996/0 (socket says 172.21.15.59:48996) 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c05d9b640 1 -- 172.21.15.59:0/3157132076 learned_addr learned my addr 172.21.15.59:0/3157132076 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c06d9d640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c08155380 0x7f4c080b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c0659c640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c0806c6a0 0x7f4c081f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c05d9b640 1 -- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c08155380 msgr2=0x7f4c080b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c05d9b640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c08155380 0x7f4c080b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c05d9b640 1 -- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c0806c6a0 msgr2=0x7f4c081f4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c05d9b640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c0806c6a0 0x7f4c081f4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c05d9b640 1 -- 172.21.15.59:0/3157132076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c081f4b60 con 0x7f4c0806d950 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c05d9b640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0806d950 0x7f4c08157210 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000c4d0 tx=0x7f4bf000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4bfeffd640 1 -- 172.21.15.59:0/3157132076 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0018070 con 0x7f4c0806d950 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4bfeffd640 1 -- 172.21.15.59:0/3157132076 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf0007500 con 0x7f4c0806d950 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3157132076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c08158fd0 con 0x7f4c0806d950 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.220+0000 7f4bfeffd640 1 -- 172.21.15.59:0/3157132076 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf0004a20 con 0x7f4c0806d950 2024-03-06T07:26:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.221+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3157132076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0815b000 con 0x7f4c0806d950 2024-03-06T07:26:56.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.222+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3157132076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0815baa0 con 0x7f4c0806d950 2024-03-06T07:26:56.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.222+0000 7f4bfeffd640 1 -- 172.21.15.59:0/3157132076 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bf001b070 con 0x7f4c0806d950 2024-03-06T07:26:56.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.222+0000 7f4bfeffd640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be806b4d0 0x7f4be806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.223+0000 7f4bfeffd640 1 -- 172.21.15.59:0/3157132076 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f4bf00895c0 con 0x7f4c0806d950 2024-03-06T07:26:56.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.223+0000 7f4c0659c640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be806b4d0 0x7f4be806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.223+0000 7f4c0659c640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be806b4d0 0x7f4be806d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8006fd0 tx=0x7f4bf8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.231+0000 7f4bfeffd640 1 -- 172.21.15.59:0/3157132076 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c0815baa0 con 0x7f4c0806d950 2024-03-06T07:26:56.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.411+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3157132076 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4c08072780 con 0x7f4c0806d950 2024-03-06T07:26:56.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.411+0000 7f4bfeffd640 1 -- 172.21.15.59:0/3157132076 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4c081f4b60 con 0x7f4c0806d950 2024-03-06T07:26:56.413 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:26:56.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be806b4d0 msgr2=0x7f4be806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be806b4d0 0x7f4be806d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8006fd0 tx=0x7f4bf8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:56.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3157132076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0806d950 msgr2=0x7f4c08157210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0806d950 0x7f4c08157210 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000c4d0 tx=0x7f4bf000c990 comp rx=0 tx=0).stop 2024-03-06T07:26:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3157132076 shutdown_connections 2024-03-06T07:26:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be806b4d0 0x7f4be806d990 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c08155380 0x7f4c080b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0806d950 0x7f4c08157210 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 --2- 172.21.15.59:0/3157132076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c0806c6a0 0x7f4c081f4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3157132076 >> 172.21.15.59:0/3157132076 conn(0x7f4c080b2980 msgr2=0x7f4c08071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3157132076 shutdown_connections 2024-03-06T07:26:56.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.413+0000 7f4c0ce6b640 1 -- 172.21.15.59:0/3157132076 wait complete. 2024-03-06T07:26:56.426 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:56.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.528+0000 7ff1b6ac9640 1 Processor -- start 2024-03-06T07:26:56.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.528+0000 7ff1b6ac9640 1 -- start start 2024-03-06T07:26:56.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1b6ac9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1b0161670 0x7ff1b0161e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1b6ac9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff1b00b2da0 0x7ff1b01623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1b6ac9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1b0162c70 con 0x7ff1b00cc680 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1b6ac9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b00b3590 con 0x7ff1b00b2da0 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1b6ac9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b006cd80 con 0x7ff1b0161670 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1b5066640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff1b00cc680 0x7ff1b00ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35996/0 (socket says 172.21.15.59:35996) 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1b5066640 1 -- 172.21.15.59:0/2854573746 learned_addr learned my addr 172.21.15.59:0/2854573746 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1affff640 1 --2- 172.21.15.59:0/2854573746 >> v2:172.21.15.164:3300/0 conn(0x7ff1b00b2da0 0x7ff1b01623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306903020 0 0) 0x7ff1b0162c70 con 0x7ff1b00cc680 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1a0003520 con 0x7ff1b00cc680 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1b4865640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1b0161670 0x7ff1b0161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2693929397 0 0) 0x7ff1a0003520 con 0x7ff1b00cc680 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1b0161670 msgr2=0x7ff1b0161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.529+0000 7ff1af7fe640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1b0161670 0x7ff1b0161e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 >> v2:172.21.15.164:3300/0 conn(0x7ff1b00b2da0 msgr2=0x7ff1b01623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1af7fe640 1 --2- 172.21.15.59:0/2854573746 >> v2:172.21.15.164:3300/0 conn(0x7ff1b00b2da0 0x7ff1b01623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b006d060 con 0x7ff1b00cc680 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7ff1a80019f0 con 0x7ff1b00cc680 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 >> v1:172.21.15.59:6789/0 conn(0x7ff1b00cc680 legacy=0x7ff1b00ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1af7fe640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a00039b0 0x7ff1a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1af7fe640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1b0161670 0x7ff1a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1af7fe640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1a0004d40 0x7ff1a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b006cd80 con 0x7ff1a00039b0 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a000b9c0 con 0x7ff1a0004d40 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a000bb40 con 0x7ff1b0161670 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1b4865640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a00039b0 0x7ff1a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1affff640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1b0161670 0x7ff1a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1b5066640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1a0004d40 0x7ff1a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1b4865640 1 -- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1b0161670 msgr2=0x7ff1a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1b4865640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1b0161670 0x7ff1a0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1b4865640 1 -- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1a0004d40 msgr2=0x7ff1a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1b4865640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1a0004d40 0x7ff1a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1b4865640 1 -- 172.21.15.59:0/2854573746 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a0003520 con 0x7ff1a00039b0 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.530+0000 7ff1b5066640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1a0004d40 0x7ff1a00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.531+0000 7ff1b4865640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a00039b0 0x7ff1a0003dd0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009ac0 tx=0x7ff1a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1198b30c39c37ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.531+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4039070 con 0x7ff1a00039b0 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.531+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a402c6b0 con 0x7ff1a00039b0 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.531+0000 7ff1af7fe640 1 -- 172.21.15.59:0/2854573746 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4009690 con 0x7ff1a00039b0 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.531+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/2854573746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a00039b0 msgr2=0x7ff1a0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.531+0000 7ff1b6ac9640 1 --2- 172.21.15.59:0/2854573746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1a00039b0 0x7ff1a0003dd0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009ac0 tx=0x7ff1a0005830 comp rx=0 tx=0).stop 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.531+0000 7ff1b5066640 1 -- 172.21.15.59:0/2854573746 reap_dead start 2024-03-06T07:26:56.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.532+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/2854573746 shutdown_connections 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.532+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/2854573746 >> 172.21.15.59:0/2854573746 conn(0x7ff1b00d1d50 msgr2=0x7ff1b00d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.532+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/2854573746 shutdown_connections 2024-03-06T07:26:56.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.532+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/2854573746 wait complete. 2024-03-06T07:26:56.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.533+0000 7ff1b6ac9640 1 Processor -- start 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.533+0000 7ff1b6ac9640 1 -- start start 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b6ac9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1a00039b0 0x7ff1b014d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b6ac9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1b015aa80 0x7ff1b00c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b6ac9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b00ca860 0x7ff1b00cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b6ac9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b006d340 con 0x7ff1b00ca860 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b6ac9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b01628e0 con 0x7ff1b015aa80 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b6ac9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b006d4c0 con 0x7ff1a00039b0 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b5066640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b00ca860 0x7ff1b00cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b5066640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b00ca860 0x7ff1b00cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49028/0 (socket says 172.21.15.59:49028) 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b5066640 1 -- 172.21.15.59:0/1689158438 learned_addr learned my addr 172.21.15.59:0/1689158438 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1affff640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1b015aa80 0x7ff1b00c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b4865640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1a00039b0 0x7ff1b014d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b5066640 1 -- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1a00039b0 msgr2=0x7ff1b014d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b5066640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1a00039b0 0x7ff1b014d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b5066640 1 -- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1b015aa80 msgr2=0x7ff1b00c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b5066640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1b015aa80 0x7ff1b00c8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b5066640 1 -- 172.21.15.59:0/1689158438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b00c9670 con 0x7ff1b00ca860 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b4865640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1a00039b0 0x7ff1b014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1b5066640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b00ca860 0x7ff1b00cace0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8000f80 tx=0x7ff1a800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.534+0000 7ff1ad7fa640 1 -- 172.21.15.59:0/1689158438 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a800eb90 con 0x7ff1b00ca860 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.535+0000 7ff1ad7fa640 1 -- 172.21.15.59:0/1689158438 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a800ed30 con 0x7ff1b00ca860 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.535+0000 7ff1ad7fa640 1 -- 172.21.15.59:0/1689158438 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8014650 con 0x7ff1b00ca860 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.535+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/1689158438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b00c9950 con 0x7ff1b00ca860 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.535+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/1689158438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b00c9f10 con 0x7ff1b00ca860 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.535+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/1689158438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b0068050 con 0x7ff1b00ca860 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.536+0000 7ff1ad7fa640 1 -- 172.21.15.59:0/1689158438 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1a80147f0 con 0x7ff1b00ca860 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.536+0000 7ff1ad7fa640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff19406b4d0 0x7ff19406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.537+0000 7ff1ad7fa640 1 -- 172.21.15.59:0/1689158438 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff1a808a0e0 con 0x7ff1b00ca860 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.537+0000 7ff1b4865640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff19406b4d0 0x7ff19406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.537+0000 7ff1b4865640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff19406b4d0 0x7ff19406d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4036000 tx=0x7ff1a4030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.541+0000 7ff1ad7fa640 1 -- 172.21.15.59:0/1689158438 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1a8058ef0 con 0x7ff1b00ca860 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.719+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/1689158438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff1b00b7cd0 con 0x7ff1b00ca860 2024-03-06T07:26:56.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.720+0000 7ff1ad7fa640 1 -- 172.21.15.59:0/1689158438 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff1a80587f0 con 0x7ff1b00ca860 2024-03-06T07:26:56.722 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:26:56.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.721+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff19406b4d0 msgr2=0x7ff19406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.721+0000 7ff1b6ac9640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff19406b4d0 0x7ff19406d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4036000 tx=0x7ff1a4030000 comp rx=0 tx=0).stop 2024-03-06T07:26:56.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.721+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/1689158438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b00ca860 msgr2=0x7ff1b00cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.721+0000 7ff1b6ac9640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b00ca860 0x7ff1b00cace0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8000f80 tx=0x7ff1a800e530 comp rx=0 tx=0).stop 2024-03-06T07:26:56.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.721+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/1689158438 shutdown_connections 2024-03-06T07:26:56.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.721+0000 7ff1b6ac9640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff19406b4d0 0x7ff19406d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.721+0000 7ff1b6ac9640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1b00ca860 0x7ff1b00cace0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.721+0000 7ff1b6ac9640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1b015aa80 0x7ff1b00c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.722+0000 7ff1b6ac9640 1 --2- 172.21.15.59:0/1689158438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1a00039b0 0x7ff1b014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.722+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/1689158438 >> 172.21.15.59:0/1689158438 conn(0x7ff1b00d1d50 msgr2=0x7ff1b00ccde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.722+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/1689158438 shutdown_connections 2024-03-06T07:26:56.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.722+0000 7ff1b6ac9640 1 -- 172.21.15.59:0/1689158438 wait complete. 2024-03-06T07:26:56.734 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:56.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.836+0000 7f62fc447640 1 Processor -- start 2024-03-06T07:26:56.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.836+0000 7f62fc447640 1 -- start start 2024-03-06T07:26:56.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.837+0000 7f62fc447640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f62f40cec60 0x7f62f40b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.837+0000 7f62fc447640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62f40b16b0 0x7f62f40cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.837+0000 7f62fc447640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f40cd4d0 con 0x7f62f415f5e0 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.837+0000 7f62fc447640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f40cf920 con 0x7f62f40cec60 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.837+0000 7f62fc447640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f40cd7b0 con 0x7f62f40b16b0 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.837+0000 7f62f99e2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f62f40cec60 0x7f62f40b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.837+0000 7f62f99e2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f62f40cec60 0x7f62f40b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49034/0 (socket says 172.21.15.59:49034) 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.837+0000 7f62f99e2640 1 -- 172.21.15.59:0/2255581364 learned_addr learned my addr 172.21.15.59:0/2255581364 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.837+0000 7f62fa9e4640 1 --2- 172.21.15.59:0/2255581364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62f40b16b0 0x7f62f40cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.838+0000 7f62f99e2640 1 -- 172.21.15.59:0/2255581364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62f40b16b0 msgr2=0x7f62f40cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.838+0000 7f62f99e2640 1 --2- 172.21.15.59:0/2255581364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62f40b16b0 0x7f62f40cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.838+0000 7f62f99e2640 1 -- 172.21.15.59:0/2255581364 >> v1:172.21.15.164:6789/0 conn(0x7f62f415f5e0 legacy=0x7f62f40ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.838+0000 7f62f99e2640 1 -- 172.21.15.59:0/2255581364 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f40cda90 con 0x7f62f40cec60 2024-03-06T07:26:56.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.838+0000 7f62fa9e4640 1 --2- 172.21.15.59:0/2255581364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62f40b16b0 0x7f62f40cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.838+0000 7f62f99e2640 1 --2- 172.21.15.59:0/2255581364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f40cec60 0x7f62f40b2bf0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009870 tx=0x7f62e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6852bb087dd4ea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.838+0000 7f62f91e1640 1 -- 172.21.15.59:0/2255581364 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4035020 con 0x7f62f40cec60 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.839+0000 7f62f91e1640 1 -- 172.21.15.59:0/2255581364 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e4031640 con 0x7f62f40cec60 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.839+0000 7f62f91e1640 1 -- 172.21.15.59:0/2255581364 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e402d040 con 0x7f62f40cec60 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.839+0000 7f62fc447640 1 -- 172.21.15.59:0/2255581364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f40cec60 msgr2=0x7f62f40b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.839+0000 7f62fc447640 1 --2- 172.21.15.59:0/2255581364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f40cec60 0x7f62f40b2bf0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009870 tx=0x7f62e402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.839+0000 7f62fc447640 1 -- 172.21.15.59:0/2255581364 shutdown_connections 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.839+0000 7f62fc447640 1 --2- 172.21.15.59:0/2255581364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62f40b16b0 0x7f62f40cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.839+0000 7f62fc447640 1 --2- 172.21.15.59:0/2255581364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f40cec60 0x7f62f40b2bf0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.839+0000 7f62fc447640 1 -- 172.21.15.59:0/2255581364 >> 172.21.15.59:0/2255581364 conn(0x7f62f40d1d50 msgr2=0x7f62f40d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.839+0000 7f62fc447640 1 -- 172.21.15.59:0/2255581364 shutdown_connections 2024-03-06T07:26:56.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.839+0000 7f62fc447640 1 -- 172.21.15.59:0/2255581364 wait complete. 2024-03-06T07:26:56.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.841+0000 7f62fc447640 1 Processor -- start 2024-03-06T07:26:56.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.841+0000 7f62fc447640 1 -- start start 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.842+0000 7f62fc447640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62f40b16b0 0x7f62f406c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.842+0000 7f62fc447640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62f40cec60 0x7f62f41572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.842+0000 7f62fc447640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f415f5e0 0x7f62f41589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.842+0000 7f62fc447640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f40cf270 con 0x7f62f415f5e0 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.842+0000 7f62fc447640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f40cf0f0 con 0x7f62f40cec60 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.842+0000 7f62fc447640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f40d2870 con 0x7f62f40b16b0 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fa9e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f415f5e0 0x7f62f41589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fa9e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f415f5e0 0x7f62f41589c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49036/0 (socket says 172.21.15.59:49036) 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fa9e4640 1 -- 172.21.15.59:0/3755305919 learned_addr learned my addr 172.21.15.59:0/3755305919 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fa1e3640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62f40b16b0 0x7f62f406c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fa9e4640 1 -- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62f40b16b0 msgr2=0x7f62f406c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fa9e4640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62f40b16b0 0x7f62f406c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fa9e4640 1 -- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62f40cec60 msgr2=0x7f62f41572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62f99e2640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62f40cec60 0x7f62f41572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fa9e4640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62f40cec60 0x7f62f41572b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fa9e4640 1 -- 172.21.15.59:0/3755305919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f40c73d0 con 0x7f62f415f5e0 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fa9e4640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f415f5e0 0x7f62f41589c0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f62ec007ec0 tx=0x7f62ec00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62eaffd640 1 -- 172.21.15.59:0/3755305919 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec01a070 con 0x7f62f415f5e0 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62eaffd640 1 -- 172.21.15.59:0/3755305919 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62ec003b90 con 0x7f62f415f5e0 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fc447640 1 -- 172.21.15.59:0/3755305919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f406c700 con 0x7f62f415f5e0 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62eaffd640 1 -- 172.21.15.59:0/3755305919 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec004030 con 0x7f62f415f5e0 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.843+0000 7f62fc447640 1 -- 172.21.15.59:0/3755305919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f406cfa0 con 0x7f62f415f5e0 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.844+0000 7f62fc447640 1 -- 172.21.15.59:0/3755305919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f4068050 con 0x7f62f415f5e0 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.845+0000 7f62eaffd640 1 -- 172.21.15.59:0/3755305919 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62ec01d070 con 0x7f62f415f5e0 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.845+0000 7f62eaffd640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d806b4d0 0x7f62d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.846+0000 7f62eaffd640 1 -- 172.21.15.59:0/3755305919 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f62ec08a5e0 con 0x7f62f415f5e0 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.846+0000 7f62fa1e3640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d806b4d0 0x7f62d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.846+0000 7f62fa1e3640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d806b4d0 0x7f62d806d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f62f0001500 tx=0x7f62f0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:56.850+0000 7f62eaffd640 1 -- 172.21.15.59:0/3755305919 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62ec010070 con 0x7f62f415f5e0 2024-03-06T07:26:57.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.146+0000 7f62fc447640 1 -- 172.21.15.59:0/3755305919 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f62f40b7cd0 con 0x7f62f415f5e0 2024-03-06T07:26:57.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.146+0000 7f62eaffd640 1 -- 172.21.15.59:0/3755305919 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f62ec059350 con 0x7f62f415f5e0 2024-03-06T07:26:57.148 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:26:57.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 -- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d806b4d0 msgr2=0x7f62d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d806b4d0 0x7f62d806d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f62f0001500 tx=0x7f62f0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 -- 172.21.15.59:0/3755305919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f415f5e0 msgr2=0x7f62f41589c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f415f5e0 0x7f62f41589c0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f62ec007ec0 tx=0x7f62ec00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 -- 172.21.15.59:0/3755305919 shutdown_connections 2024-03-06T07:26:57.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d806b4d0 0x7f62d806d990 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62f415f5e0 0x7f62f41589c0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62f40cec60 0x7f62f41572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 --2- 172.21.15.59:0/3755305919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62f40b16b0 0x7f62f406c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 -- 172.21.15.59:0/3755305919 >> 172.21.15.59:0/3755305919 conn(0x7f62f40d1d50 msgr2=0x7f62f40b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 -- 172.21.15.59:0/3755305919 shutdown_connections 2024-03-06T07:26:57.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.149+0000 7f62fc447640 1 -- 172.21.15.59:0/3755305919 wait complete. 2024-03-06T07:26:57.162 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:26:57.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.262+0000 7f2acca64640 1 Processor -- start 2024-03-06T07:26:57.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.262+0000 7f2acca64640 1 -- start start 2024-03-06T07:26:57.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.263+0000 7f2acca64640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2ac815e150 0x7f2ac80cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.263+0000 7f2acca64640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80d1ea0 con 0x7f2ac80d1060 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.263+0000 7f2acca64640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac80d2180 con 0x7f2ac80cfba0 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.263+0000 7f2acca64640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac80d2ab0 con 0x7f2ac815e150 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.263+0000 7f2ac6d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2ac80d1060 0x7f2ac80b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36000/0 (socket says 172.21.15.59:36000) 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.263+0000 7f2ac6d9d640 1 -- 172.21.15.59:0/636913251 learned_addr learned my addr 172.21.15.59:0/636913251 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac659c640 1 --2- 172.21.15.59:0/636913251 >> v2:172.21.15.164:3300/0 conn(0x7f2ac815e150 0x7f2ac80cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83916482 0 0) 0x7f2ac80d1ea0 con 0x7f2ac80d1060 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab4003520 con 0x7f2ac80d1060 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1621494535 0 0) 0x7f2ab4003520 con 0x7f2ac80d1060 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 >> v2:172.21.15.164:3300/0 conn(0x7f2ac815e150 msgr2=0x7f2ac80cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 --2- 172.21.15.59:0/636913251 >> v2:172.21.15.164:3300/0 conn(0x7f2ac815e150 0x7f2ac80cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 >> v1:172.21.15.164:6789/0 conn(0x7f2ac80cfba0 legacy=0x7f2ac80d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80d2460 con 0x7f2ac80d1060 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f2ac00019f0 con 0x7f2ac80d1060 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 >> v1:172.21.15.59:6789/0 conn(0x7f2ac80d1060 legacy=0x7f2ac80b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 --2- 172.21.15.59:0/636913251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ab40039b0 0x7f2ab4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 --2- 172.21.15.59:0/636913251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ab4004800 0x7f2ab4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 --2- 172.21.15.59:0/636913251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab40051c0 0x7f2ab400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80d2ab0 con 0x7f2ab40051c0 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab400c160 con 0x7f2ab40039b0 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab400c360 con 0x7f2ab4004800 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac6d9d640 1 --2- 172.21.15.59:0/636913251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab40051c0 0x7f2ab400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2ac659c640 1 --2- 172.21.15.59:0/636913251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ab40039b0 0x7f2ab4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.264+0000 7f2abdd9b640 1 --2- 172.21.15.59:0/636913251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ab4004800 0x7f2ab4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.265+0000 7f2ac6d9d640 1 -- 172.21.15.59:0/636913251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ab4004800 msgr2=0x7f2ab4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.265+0000 7f2ac6d9d640 1 --2- 172.21.15.59:0/636913251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ab4004800 0x7f2ab4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.265+0000 7f2ac6d9d640 1 -- 172.21.15.59:0/636913251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ab40039b0 msgr2=0x7f2ab4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.265+0000 7f2ac6d9d640 1 --2- 172.21.15.59:0/636913251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ab40039b0 0x7f2ab4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.265+0000 7f2ac6d9d640 1 -- 172.21.15.59:0/636913251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4003520 con 0x7f2ab40051c0 2024-03-06T07:26:57.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2ac6d9d640 1 -- 172.21.15.59:0/636913251 reap_dead start 2024-03-06T07:26:57.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2ac6d9d640 1 --2- 172.21.15.59:0/636913251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab40051c0 0x7f2ab400b9c0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f2ac808dfe0 tx=0x7f2ab4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc81838be3c40cbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000a460 con 0x7f2ab40051c0 2024-03-06T07:26:57.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac0034650 con 0x7f2ab40051c0 2024-03-06T07:26:57.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2ac5d9b640 1 -- 172.21.15.59:0/636913251 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0034a40 con 0x7f2ab40051c0 2024-03-06T07:26:57.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2acca64640 1 -- 172.21.15.59:0/636913251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab40051c0 msgr2=0x7f2ab400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2acca64640 1 --2- 172.21.15.59:0/636913251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab40051c0 0x7f2ab400b9c0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f2ac808dfe0 tx=0x7f2ab4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:57.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2acca64640 1 -- 172.21.15.59:0/636913251 shutdown_connections 2024-03-06T07:26:57.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2acca64640 1 --2- 172.21.15.59:0/636913251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ab40051c0 0x7f2ab400b9c0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2acca64640 1 -- 172.21.15.59:0/636913251 >> 172.21.15.59:0/636913251 conn(0x7f2ac809d230 msgr2=0x7f2ac80d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2acca64640 1 -- 172.21.15.59:0/636913251 shutdown_connections 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.266+0000 7f2acca64640 1 -- 172.21.15.59:0/636913251 wait complete. 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.268+0000 7f2acca64640 1 Processor -- start 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.268+0000 7f2acca64640 1 -- start start 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.268+0000 7f2acca64640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ab40051c0 0x7f2ac8151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.268+0000 7f2acca64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ac80b77a0 0x7f2ac80c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.268+0000 7f2acca64640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac80c8fb0 0x7f2ac815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2acca64640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80d1b30 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2acca64640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac80d2f40 con 0x7f2ab40051c0 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2acca64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac815e530 con 0x7f2ac80b77a0 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2ac6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac80c8fb0 0x7f2ac815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2ac6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac80c8fb0 0x7f2ac815bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49062/0 (socket says 172.21.15.59:49062) 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2ac6d9d640 1 -- 172.21.15.59:0/2405921247 learned_addr learned my addr 172.21.15.59:0/2405921247 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2ac659c640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ab40051c0 0x7f2ac8151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2abdd9b640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ac80b77a0 0x7f2ac80c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2ac6d9d640 1 -- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ac80b77a0 msgr2=0x7f2ac80c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2ac6d9d640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ac80b77a0 0x7f2ac80c7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2ac6d9d640 1 -- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ab40051c0 msgr2=0x7f2ac8151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2ac6d9d640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ab40051c0 0x7f2ac8151560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2ac6d9d640 1 -- 172.21.15.59:0/2405921247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80c7f10 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2abdd9b640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ac80b77a0 0x7f2ac80c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2ac6d9d640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac80c8fb0 0x7f2ac815bc60 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000a900 tx=0x7f2ac80d2070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2abf7fe640 1 -- 172.21.15.59:0/2405921247 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0034a00 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2abf7fe640 1 -- 172.21.15.59:0/2405921247 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac0045070 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2abf7fe640 1 -- 172.21.15.59:0/2405921247 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0034dc0 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2acca64640 1 -- 172.21.15.59:0/2405921247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac80c81f0 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.269+0000 7f2acca64640 1 -- 172.21.15.59:0/2405921247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac80c87b0 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.271+0000 7f2abf7fe640 1 -- 172.21.15.59:0/2405921247 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ac004a460 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.271+0000 7f2acca64640 1 -- 172.21.15.59:0/2405921247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac8069ee0 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.271+0000 7f2abf7fe640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2aa806b500 0x7f2aa806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.271+0000 7f2abf7fe640 1 -- 172.21.15.59:0/2405921247 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f2ac00ac970 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.271+0000 7f2ac659c640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2aa806b500 0x7f2aa806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.272+0000 7f2ac659c640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2aa806b500 0x7f2aa806d9c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80d3670 tx=0x7f2ab8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.275+0000 7f2abf7fe640 1 -- 172.21.15.59:0/2405921247 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ac007b770 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.471+0000 7f2acca64640 1 -- 172.21.15.59:0/2405921247 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2ac815d870 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.471+0000 7f2abf7fe640 1 -- 172.21.15.59:0/2405921247 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v96) v1 ==== 80+0+368 (secure 0 0 0) 0x7f2ac007b070 con 0x7f2ac80c8fb0 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.59:6834/2799249020 2024-03-07T07:26:29.597825+0000 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.59:6835/1784120157 2024-03-07T07:26:29.585744+0000 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.59:6832/2799249020 2024-03-07T07:26:29.597825+0000 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.59:6833/1784120157 2024-03-07T07:26:29.585744+0000 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.164:6837/4292771408 2024-03-07T07:26:29.578123+0000 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.164:6835/4292771408 2024-03-07T07:26:29.578123+0000 2024-03-06T07:26:57.474 INFO:teuthology.orchestra.run.smithi059.stderr:listed 6 entries 2024-03-06T07:26:57.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 -- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2aa806b500 msgr2=0x7f2aa806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2aa806b500 0x7f2aa806d9c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80d3670 tx=0x7f2ab8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 -- 172.21.15.59:0/2405921247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac80c8fb0 msgr2=0x7f2ac815bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac80c8fb0 0x7f2ac815bc60 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000a900 tx=0x7f2ac80d2070 comp rx=0 tx=0).stop 2024-03-06T07:26:57.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 -- 172.21.15.59:0/2405921247 shutdown_connections 2024-03-06T07:26:57.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2aa806b500 0x7f2aa806d9c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ac80c8fb0 0x7f2ac815bc60 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ac80b77a0 0x7f2ac80c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 --2- 172.21.15.59:0/2405921247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ab40051c0 0x7f2ac8151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 -- 172.21.15.59:0/2405921247 >> 172.21.15.59:0/2405921247 conn(0x7f2ac809d230 msgr2=0x7f2ac80ceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 -- 172.21.15.59:0/2405921247 shutdown_connections 2024-03-06T07:26:57.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:57.473+0000 7f2acca64640 1 -- 172.21.15.59:0/2405921247 wait complete. 2024-03-06T07:26:57.490 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:26:57.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:57.490 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:57.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[68746]: fuse finished with error 0 and tester_r 0 2024-03-06T07:26:57.517 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:26:57.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:57.517 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:26:57.688 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:26:57.688 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:57.705 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:57.705 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:26:57.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:57.705 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:26:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:57.762 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:26:57.762 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:26:57.762 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:57.780 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:57.780 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:26:57.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:57.781 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:26:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:57.834 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:26:57.835 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:26:57.835 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:57.851 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:57.852 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:26:57.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:57.852 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:26:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:57.908 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:26:57.908 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:58.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.008+0000 7f1fee53e640 1 Processor -- start 2024-03-06T07:26:58.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.008+0000 7f1fee53e640 1 -- start start 2024-03-06T07:26:58.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.009+0000 7f1fee53e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1fe806e160 0x7f1fe806e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.009+0000 7f1fee53e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe80cdf90 con 0x7f1fe8145900 2024-03-06T07:26:58.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.009+0000 7f1fee53e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe806d3a0 con 0x7f1fe80cdb30 2024-03-06T07:26:58.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.009+0000 7f1fee53e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe806d150 con 0x7f1fe806e160 2024-03-06T07:26:58.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.009+0000 7f1fecadb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1fe806e160 0x7f1fe806e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.009+0000 7f1fecadb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1fe806e160 0x7f1fe806e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49074/0 (socket says 172.21.15.59:49074) 2024-03-06T07:26:58.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.009+0000 7f1fecadb640 1 -- 172.21.15.59:0/4121257672 learned_addr learned my addr 172.21.15.59:0/4121257672 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:58.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.009+0000 7f1fecadb640 1 -- 172.21.15.59:0/4121257672 >> v1:172.21.15.164:6789/0 conn(0x7f1fe80cdb30 legacy=0x7f1fe815c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.009+0000 7f1fecadb640 1 -- 172.21.15.59:0/4121257672 >> v1:172.21.15.59:6789/0 conn(0x7f1fe8145900 legacy=0x7f1fe80cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.009+0000 7f1fecadb640 1 -- 172.21.15.59:0/4121257672 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe806d680 con 0x7f1fe806e160 2024-03-06T07:26:58.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.010+0000 7f1fecadb640 1 --2- 172.21.15.59:0/4121257672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe806e160 0x7f1fe806e5d0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f1fe806d2d0 tx=0x7f1fe002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe6400fe356e5c25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.010+0000 7f1fe6ffd640 1 -- 172.21.15.59:0/4121257672 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe002c9e0 con 0x7f1fe806e160 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.010+0000 7f1fe6ffd640 1 -- 172.21.15.59:0/4121257672 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe002cb80 con 0x7f1fe806e160 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.010+0000 7f1fe6ffd640 1 -- 172.21.15.59:0/4121257672 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe00343f0 con 0x7f1fe806e160 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.011+0000 7f1fee53e640 1 -- 172.21.15.59:0/4121257672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe806e160 msgr2=0x7f1fe806e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.011+0000 7f1fee53e640 1 --2- 172.21.15.59:0/4121257672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe806e160 0x7f1fe806e5d0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f1fe806d2d0 tx=0x7f1fe002be40 comp rx=0 tx=0).stop 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.011+0000 7f1fee53e640 1 -- 172.21.15.59:0/4121257672 shutdown_connections 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.011+0000 7f1fee53e640 1 --2- 172.21.15.59:0/4121257672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe806e160 0x7f1fe806e5d0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.011+0000 7f1fee53e640 1 -- 172.21.15.59:0/4121257672 >> 172.21.15.59:0/4121257672 conn(0x7f1fe806b160 msgr2=0x7f1fe806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.011+0000 7f1fee53e640 1 -- 172.21.15.59:0/4121257672 shutdown_connections 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.011+0000 7f1fee53e640 1 -- 172.21.15.59:0/4121257672 wait complete. 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.012+0000 7f1fee53e640 1 Processor -- start 2024-03-06T07:26:58.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.012+0000 7f1fee53e640 1 -- start start 2024-03-06T07:26:58.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fee53e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe806e160 0x7f1fe815b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fee53e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe80cdb30 0x7f1fe8071d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fee53e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe8145900 0x7f1fe8144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fee53e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe815cb40 con 0x7f1fe8145900 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fee53e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe806dc40 con 0x7f1fe80cdb30 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fee53e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe815fa20 con 0x7f1fe806e160 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fecadb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe8145900 0x7f1fe8144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fecadb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe8145900 0x7f1fe8144710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49088/0 (socket says 172.21.15.59:49088) 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fecadb640 1 -- 172.21.15.59:0/1253355787 learned_addr learned my addr 172.21.15.59:0/1253355787 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fe77fe640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe80cdb30 0x7f1fe8071d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fecadb640 1 -- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe806e160 msgr2=0x7f1fe815b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fecadb640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe806e160 0x7f1fe815b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fecadb640 1 -- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe80cdb30 msgr2=0x7f1fe8071d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fecadb640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe80cdb30 0x7f1fe8071d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fecadb640 1 -- 172.21.15.59:0/1253355787 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe815bbf0 con 0x7f1fe8145900 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fe77fe640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe80cdb30 0x7f1fe8071d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fecadb640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe8145900 0x7f1fe8144710 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009f40 tx=0x7f1fe0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.013+0000 7f1fe4ff9640 1 -- 172.21.15.59:0/1253355787 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0043020 con 0x7f1fe8145900 2024-03-06T07:26:58.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.014+0000 7f1fe4ff9640 1 -- 172.21.15.59:0/1253355787 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe003e450 con 0x7f1fe8145900 2024-03-06T07:26:58.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.014+0000 7f1fee53e640 1 -- 172.21.15.59:0/1253355787 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe8143b10 con 0x7f1fe8145900 2024-03-06T07:26:58.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.014+0000 7f1fe4ff9640 1 -- 172.21.15.59:0/1253355787 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe003e860 con 0x7f1fe8145900 2024-03-06T07:26:58.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.014+0000 7f1fee53e640 1 -- 172.21.15.59:0/1253355787 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe81440d0 con 0x7f1fe8145900 2024-03-06T07:26:58.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.015+0000 7f1fee53e640 1 -- 172.21.15.59:0/1253355787 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe8069ee0 con 0x7f1fe8145900 2024-03-06T07:26:58.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.015+0000 7f1fe4ff9640 1 -- 172.21.15.59:0/1253355787 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fe0046020 con 0x7f1fe8145900 2024-03-06T07:26:58.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.016+0000 7f1fe4ff9640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fd406b4d0 0x7f1fd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.016+0000 7f1fe4ff9640 1 -- 172.21.15.59:0/1253355787 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f1fe00acc30 con 0x7f1fe8145900 2024-03-06T07:26:58.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.016+0000 7f1fe7fff640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fd406b4d0 0x7f1fd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.016+0000 7f1fe7fff640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fd406b4d0 0x7f1fd406d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc000f60 tx=0x7f1fdc0079d0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.020+0000 7f1fe4ff9640 1 -- 172.21.15.59:0/1253355787 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fe004d030 con 0x7f1fe8145900 2024-03-06T07:26:58.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.238+0000 7f1fee53e640 1 -- 172.21.15.59:0/1253355787 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fe815f170 con 0x7f1fe8145900 2024-03-06T07:26:58.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.239+0000 7f1fe4ff9640 1 -- 172.21.15.59:0/1253355787 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f1fe007ba40 con 0x7f1fe8145900 2024-03-06T07:26:58.241 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:58.241 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:58.241 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:58.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.240+0000 7f1fee53e640 1 -- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fd406b4d0 msgr2=0x7f1fd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.240+0000 7f1fee53e640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fd406b4d0 0x7f1fd406d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc000f60 tx=0x7f1fdc0079d0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.240+0000 7f1fee53e640 1 -- 172.21.15.59:0/1253355787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe8145900 msgr2=0x7f1fe8144710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.240+0000 7f1fee53e640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe8145900 0x7f1fe8144710 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009f40 tx=0x7f1fe0004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:58.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.240+0000 7f1fee53e640 1 -- 172.21.15.59:0/1253355787 shutdown_connections 2024-03-06T07:26:58.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.240+0000 7f1fee53e640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1fd406b4d0 0x7f1fd406d990 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.240+0000 7f1fee53e640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1fe8145900 0x7f1fe8144710 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.240+0000 7f1fee53e640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1fe80cdb30 0x7f1fe8071d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.240+0000 7f1fee53e640 1 --2- 172.21.15.59:0/1253355787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1fe806e160 0x7f1fe815b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.240+0000 7f1fee53e640 1 -- 172.21.15.59:0/1253355787 >> 172.21.15.59:0/1253355787 conn(0x7f1fe806b160 msgr2=0x7f1fe80afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.241+0000 7f1fee53e640 1 -- 172.21.15.59:0/1253355787 shutdown_connections 2024-03-06T07:26:58.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.241+0000 7f1fee53e640 1 -- 172.21.15.59:0/1253355787 wait complete. 2024-03-06T07:26:58.254 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:58.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.353+0000 7fa8790e1640 1 Processor -- start 2024-03-06T07:26:58.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.353+0000 7fa8790e1640 1 -- start start 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.354+0000 7fa8790e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8740d0950 0x7fa8740d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.354+0000 7fa8790e1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8740b9060 0x7fa8740b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.354+0000 7fa8790e1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8740d1ea0 con 0x7fa87415e150 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.354+0000 7fa8790e1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8740d12f0 con 0x7fa8740b9060 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.354+0000 7fa8790e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740d2ab0 con 0x7fa8740d0950 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.354+0000 7fa87359e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8740b9060 0x7fa8740b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.354+0000 7fa87359e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8740b9060 0x7fa8740b94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49100/0 (socket says 172.21.15.59:49100) 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.354+0000 7fa87359e640 1 -- 172.21.15.59:0/3142702144 learned_addr learned my addr 172.21.15.59:0/3142702144 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.355+0000 7fa87359e640 1 -- 172.21.15.59:0/3142702144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8740d0950 msgr2=0x7fa8740d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.355+0000 7fa87359e640 1 --2- 172.21.15.59:0/3142702144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8740d0950 0x7fa8740d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.355+0000 7fa87359e640 1 -- 172.21.15.59:0/3142702144 >> v1:172.21.15.164:6789/0 conn(0x7fa87415e150 legacy=0x7fa8740cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.355+0000 7fa87359e640 1 -- 172.21.15.59:0/3142702144 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8740d2180 con 0x7fa8740b9060 2024-03-06T07:26:58.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.355+0000 7fa87359e640 1 --2- 172.21.15.59:0/3142702144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740b9060 0x7fa8740b94f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009e50 tx=0x7fa86c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7e59fc9092f7c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.356+0000 7fa871d9b640 1 -- 172.21.15.59:0/3142702144 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c02c9e0 con 0x7fa8740b9060 2024-03-06T07:26:58.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.356+0000 7fa871d9b640 1 -- 172.21.15.59:0/3142702144 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86c02cb80 con 0x7fa8740b9060 2024-03-06T07:26:58.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.356+0000 7fa871d9b640 1 -- 172.21.15.59:0/3142702144 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c0343f0 con 0x7fa8740b9060 2024-03-06T07:26:58.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.356+0000 7fa8790e1640 1 -- 172.21.15.59:0/3142702144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740b9060 msgr2=0x7fa8740b94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.356+0000 7fa8790e1640 1 --2- 172.21.15.59:0/3142702144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740b9060 0x7fa8740b94f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009e50 tx=0x7fa86c02be40 comp rx=0 tx=0).stop 2024-03-06T07:26:58.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.356+0000 7fa8790e1640 1 -- 172.21.15.59:0/3142702144 shutdown_connections 2024-03-06T07:26:58.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.356+0000 7fa8790e1640 1 --2- 172.21.15.59:0/3142702144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740b9060 0x7fa8740b94f0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.356+0000 7fa8790e1640 1 --2- 172.21.15.59:0/3142702144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8740d0950 0x7fa8740d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.356+0000 7fa8790e1640 1 -- 172.21.15.59:0/3142702144 >> 172.21.15.59:0/3142702144 conn(0x7fa87409d230 msgr2=0x7fa8740d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.356+0000 7fa8790e1640 1 -- 172.21.15.59:0/3142702144 shutdown_connections 2024-03-06T07:26:58.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.356+0000 7fa8790e1640 1 -- 172.21.15.59:0/3142702144 wait complete. 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.358+0000 7fa8790e1640 1 Processor -- start 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.358+0000 7fa8790e1640 1 -- start start 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.358+0000 7fa8790e1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740b9060 0x7fa87414d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.358+0000 7fa8790e1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8740d0950 0x7fa87415e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.358+0000 7fa8790e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87415e150 0x7fa87415bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.358+0000 7fa8790e1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740d2f40 con 0x7fa8740b9060 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.358+0000 7fa8790e1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740d2740 con 0x7fa8740d0950 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.358+0000 7fa8790e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740d1b30 con 0x7fa87415e150 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.358+0000 7fa872d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740b9060 0x7fa87414d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.358+0000 7fa872d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740b9060 0x7fa87414d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49108/0 (socket says 172.21.15.59:49108) 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.358+0000 7fa872d9d640 1 -- 172.21.15.59:0/1602155685 learned_addr learned my addr 172.21.15.59:0/1602155685 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:58.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa87359e640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87415e150 0x7fa87415bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa87259c640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8740d0950 0x7fa87415e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa872d9d640 1 -- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87415e150 msgr2=0x7fa87415bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa872d9d640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87415e150 0x7fa87415bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa872d9d640 1 -- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8740d0950 msgr2=0x7fa87415e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa872d9d640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8740d0950 0x7fa87415e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa872d9d640 1 -- 172.21.15.59:0/1602155685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87415eea0 con 0x7fa8740b9060 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa87259c640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8740d0950 0x7fa87415e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa872d9d640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740b9060 0x7fa87414d490 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fa86800b3f0 tx=0x7fa86800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa85b7fe640 1 -- 172.21.15.59:0/1602155685 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868015020 con 0x7fa8740b9060 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa85b7fe640 1 -- 172.21.15.59:0/1602155685 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa868011830 con 0x7fa8740b9060 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa8790e1640 1 -- 172.21.15.59:0/1602155685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87415f180 con 0x7fa8740b9060 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa85b7fe640 1 -- 172.21.15.59:0/1602155685 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868011c20 con 0x7fa8740b9060 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.359+0000 7fa8790e1640 1 -- 172.21.15.59:0/1602155685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87415f740 con 0x7fa8740b9060 2024-03-06T07:26:58.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.360+0000 7fa8790e1640 1 -- 172.21.15.59:0/1602155685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa874069ee0 con 0x7fa8740b9060 2024-03-06T07:26:58.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.361+0000 7fa85b7fe640 1 -- 172.21.15.59:0/1602155685 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa868018020 con 0x7fa8740b9060 2024-03-06T07:26:58.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.362+0000 7fa85b7fe640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa86006b4d0 0x7fa86006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.362+0000 7fa85b7fe640 1 -- 172.21.15.59:0/1602155685 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa868089fb0 con 0x7fa8740b9060 2024-03-06T07:26:58.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.362+0000 7fa87259c640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa86006b4d0 0x7fa86006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.362+0000 7fa87259c640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa86006b4d0 0x7fa86006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa8740b72f0 tx=0x7fa864008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.366+0000 7fa85b7fe640 1 -- 172.21.15.59:0/1602155685 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa86801f030 con 0x7fa8740b9060 2024-03-06T07:26:58.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.547+0000 7fa8790e1640 1 -- 172.21.15.59:0/1602155685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8740d4670 con 0x7fa8740b9060 2024-03-06T07:26:58.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.548+0000 7fa85b7fe640 1 -- 172.21.15.59:0/1602155685 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16987 (secure 0 0 0) 0x7fa8680172c0 con 0x7fa8740b9060 2024-03-06T07:26:58.550 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:58.551 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":96,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:26:29.601683+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.489280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:16.183614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6834/2799249020":"2024-03-07T07:26:29.597825+0000","172.21.15.59:6835/1784120157":"2024-03-07T07:26:29.585744+0000","172.21.15.59:6832/2799249020":"2024-03-07T07:26:29.597825+0000","172.21.15.59:6833/1784120157":"2024-03-07T07:26:29.585744+0000","172.21.15.164:6837/4292771408":"2024-03-07T07:26:29.578123+0000","172.21.15.164:6835/4292771408":"2024-03-07T07:26:29.578123+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:58.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 -- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa86006b4d0 msgr2=0x7fa86006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa86006b4d0 0x7fa86006d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa8740b72f0 tx=0x7fa864008040 comp rx=0 tx=0).stop 2024-03-06T07:26:58.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 -- 172.21.15.59:0/1602155685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740b9060 msgr2=0x7fa87414d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740b9060 0x7fa87414d490 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fa86800b3f0 tx=0x7fa86800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 -- 172.21.15.59:0/1602155685 shutdown_connections 2024-03-06T07:26:58.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa86006b4d0 0x7fa86006d990 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87415e150 0x7fa87415bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8740d0950 0x7fa87415e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 --2- 172.21.15.59:0/1602155685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8740b9060 0x7fa87414d490 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 -- 172.21.15.59:0/1602155685 >> 172.21.15.59:0/1602155685 conn(0x7fa87409d230 msgr2=0x7fa8740b4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 -- 172.21.15.59:0/1602155685 shutdown_connections 2024-03-06T07:26:58.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.550+0000 7fa8790e1640 1 -- 172.21.15.59:0/1602155685 wait complete. 2024-03-06T07:26:58.567 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:15.489280+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:16.183614+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:58.567 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:58.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.673+0000 7f6f16888640 1 Processor -- start 2024-03-06T07:26:58.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.673+0000 7f6f16888640 1 -- start start 2024-03-06T07:26:58.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.675+0000 7f6f16888640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6f100723a0 0x7f6f10165c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.675+0000 7f6f16888640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f1015f0d0 con 0x7f6f1015b300 2024-03-06T07:26:58.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.675+0000 7f6f16888640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f1015f3b0 con 0x7f6f10155560 2024-03-06T07:26:58.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.675+0000 7f6f16888640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f1015f690 con 0x7f6f100723a0 2024-03-06T07:26:58.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.675+0000 7f6f0f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6f1015b300 0x7f6f10071c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36028/0 (socket says 172.21.15.59:36028) 2024-03-06T07:26:58.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.675+0000 7f6f0f7fe640 1 -- 172.21.15.59:0/2993785942 learned_addr learned my addr 172.21.15.59:0/2993785942 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:26:58.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.675+0000 7f6f14e25640 1 --2- 172.21.15.59:0/2993785942 >> v2:172.21.15.59:3300/0 conn(0x7f6f100723a0 0x7f6f10165c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.675+0000 7f6f14e25640 1 -- 172.21.15.59:0/2993785942 >> v1:172.21.15.164:6789/0 conn(0x7f6f10155560 legacy=0x7f6f1015e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.675+0000 7f6f14e25640 1 -- 172.21.15.59:0/2993785942 >> v1:172.21.15.59:6789/0 conn(0x7f6f1015b300 legacy=0x7f6f10071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.675+0000 7f6f14e25640 1 -- 172.21.15.59:0/2993785942 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f100b3da0 con 0x7f6f100723a0 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.676+0000 7f6f14e25640 1 --2- 172.21.15.59:0/2993785942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f100723a0 0x7f6f10165c80 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009d90 tx=0x7f6f0802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea4cd7c78ca538a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.676+0000 7f6f0effd640 1 -- 172.21.15.59:0/2993785942 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08039070 con 0x7f6f100723a0 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.676+0000 7f6f0effd640 1 -- 172.21.15.59:0/2993785942 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0802cc60 con 0x7f6f100723a0 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.676+0000 7f6f0effd640 1 -- 172.21.15.59:0/2993785942 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08034660 con 0x7f6f100723a0 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.677+0000 7f6f16888640 1 -- 172.21.15.59:0/2993785942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f100723a0 msgr2=0x7f6f10165c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.677+0000 7f6f16888640 1 --2- 172.21.15.59:0/2993785942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f100723a0 0x7f6f10165c80 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009d90 tx=0x7f6f0802beb0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.677+0000 7f6f16888640 1 -- 172.21.15.59:0/2993785942 shutdown_connections 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.677+0000 7f6f16888640 1 --2- 172.21.15.59:0/2993785942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f100723a0 0x7f6f10165c80 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.677+0000 7f6f16888640 1 -- 172.21.15.59:0/2993785942 >> 172.21.15.59:0/2993785942 conn(0x7f6f100b2890 msgr2=0x7f6f100b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.677+0000 7f6f16888640 1 -- 172.21.15.59:0/2993785942 shutdown_connections 2024-03-06T07:26:58.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.677+0000 7f6f16888640 1 -- 172.21.15.59:0/2993785942 wait complete. 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.678+0000 7f6f16888640 1 Processor -- start 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.678+0000 7f6f16888640 1 -- start start 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f16888640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f100723a0 0x7f6f10145fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f16888640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f10155560 0x7f6f1014b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f16888640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f1015b300 0x7f6f1014ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f16888640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f101661c0 con 0x7f6f100723a0 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f16888640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f100b54e0 con 0x7f6f10155560 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f16888640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f10166340 con 0x7f6f1015b300 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f0ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f100723a0 0x7f6f10145fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f0ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f100723a0 0x7f6f10145fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49122/0 (socket says 172.21.15.59:49122) 2024-03-06T07:26:58.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f0ffff640 1 -- 172.21.15.59:0/4243884060 learned_addr learned my addr 172.21.15.59:0/4243884060 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f0f7fe640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f10155560 0x7f6f1014b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f14e25640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f1015b300 0x7f6f1014ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f0ffff640 1 -- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f1015b300 msgr2=0x7f6f1014ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f0ffff640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f1015b300 0x7f6f1014ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f0ffff640 1 -- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f10155560 msgr2=0x7f6f1014b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f0ffff640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f10155560 0x7f6f1014b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f0ffff640 1 -- 172.21.15.59:0/4243884060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1014c200 con 0x7f6f100723a0 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f0f7fe640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f10155560 0x7f6f1014b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.679+0000 7f6f0ffff640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f100723a0 0x7f6f10145fe0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000c460 tx=0x7f6f0000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.680+0000 7f6f0cff9640 1 -- 172.21.15.59:0/4243884060 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00016020 con 0x7f6f100723a0 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.680+0000 7f6f0cff9640 1 -- 172.21.15.59:0/4243884060 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f00012870 con 0x7f6f100723a0 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.680+0000 7f6f16888640 1 -- 172.21.15.59:0/4243884060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f1015cb30 con 0x7f6f100723a0 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.680+0000 7f6f0cff9640 1 -- 172.21.15.59:0/4243884060 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00012c60 con 0x7f6f100723a0 2024-03-06T07:26:58.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.680+0000 7f6f16888640 1 -- 172.21.15.59:0/4243884060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f10146c40 con 0x7f6f100723a0 2024-03-06T07:26:58.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.681+0000 7f6f16888640 1 -- 172.21.15.59:0/4243884060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f10068050 con 0x7f6f100723a0 2024-03-06T07:26:58.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.681+0000 7f6f0cff9640 1 -- 172.21.15.59:0/4243884060 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f00019020 con 0x7f6f100723a0 2024-03-06T07:26:58.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.681+0000 7f6f0cff9640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6efc06b4d0 0x7f6efc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.682+0000 7f6f0cff9640 1 -- 172.21.15.59:0/4243884060 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f6f00089cf0 con 0x7f6f100723a0 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.682+0000 7f6f0f7fe640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6efc06b4d0 0x7f6efc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.682+0000 7f6f0f7fe640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6efc06b4d0 0x7f6efc06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6f1014eb30 tx=0x7f6f04009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.686+0000 7f6f0cff9640 1 -- 172.21.15.59:0/4243884060 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f0008e070 con 0x7f6f100723a0 2024-03-06T07:26:58.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.867+0000 7f6f16888640 1 -- 172.21.15.59:0/4243884060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f100b2600 con 0x7f6f100723a0 2024-03-06T07:26:58.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.867+0000 7f6f0cff9640 1 -- 172.21.15.59:0/4243884060 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f00020070 con 0x7f6f100723a0 2024-03-06T07:26:58.869 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:26:58.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 -- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6efc06b4d0 msgr2=0x7f6efc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6efc06b4d0 0x7f6efc06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6f1014eb30 tx=0x7f6f04009040 comp rx=0 tx=0).stop 2024-03-06T07:26:58.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 -- 172.21.15.59:0/4243884060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f100723a0 msgr2=0x7f6f10145fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f100723a0 0x7f6f10145fe0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000c460 tx=0x7f6f0000c920 comp rx=0 tx=0).stop 2024-03-06T07:26:58.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 -- 172.21.15.59:0/4243884060 shutdown_connections 2024-03-06T07:26:58.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6efc06b4d0 0x7f6efc06d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f1015b300 0x7f6f1014ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f10155560 0x7f6f1014b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 --2- 172.21.15.59:0/4243884060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6f100723a0 0x7f6f10145fe0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 -- 172.21.15.59:0/4243884060 >> 172.21.15.59:0/4243884060 conn(0x7f6f100b2890 msgr2=0x7f6f100d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 -- 172.21.15.59:0/4243884060 shutdown_connections 2024-03-06T07:26:58.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.869+0000 7f6f16888640 1 -- 172.21.15.59:0/4243884060 wait complete. 2024-03-06T07:26:58.882 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.984+0000 7ff0a650c640 1 Processor -- start 2024-03-06T07:26:58.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.984+0000 7ff0a650c640 1 -- start start 2024-03-06T07:26:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff0a650c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff0a0161760 0x7ff0a00b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff0a650c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff0a00b5240 0x7ff0a00d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff0a650c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a0161b40 con 0x7ff0a00d2070 2024-03-06T07:26:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff0a650c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a00c8ac0 con 0x7ff0a00b5240 2024-03-06T07:26:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff0a650c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a00d27e0 con 0x7ff0a0161760 2024-03-06T07:26:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff09f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff0a00b5240 0x7ff0a00d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff09f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff0a00b5240 0x7ff0a00d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49128/0 (socket says 172.21.15.59:49128) 2024-03-06T07:26:58.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff09f7fe640 1 -- 172.21.15.59:0/2081553283 learned_addr learned my addr 172.21.15.59:0/2081553283 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:58.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff09ffff640 1 --2- 172.21.15.59:0/2081553283 >> v2:172.21.15.164:3300/0 conn(0x7ff0a0161760 0x7ff0a00b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff09f7fe640 1 -- 172.21.15.59:0/2081553283 >> v2:172.21.15.164:3300/0 conn(0x7ff0a0161760 msgr2=0x7ff0a00b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff09f7fe640 1 --2- 172.21.15.59:0/2081553283 >> v2:172.21.15.164:3300/0 conn(0x7ff0a0161760 0x7ff0a00b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff09f7fe640 1 -- 172.21.15.59:0/2081553283 >> v1:172.21.15.59:6789/0 conn(0x7ff0a00d2070 legacy=0x7ff0a00d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.985+0000 7ff09f7fe640 1 -- 172.21.15.59:0/2081553283 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a00cf580 con 0x7ff0a00b5240 2024-03-06T07:26:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.986+0000 7ff09f7fe640 1 --2- 172.21.15.59:0/2081553283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0a00b5240 0x7ff0a00d1b30 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7ff0a008e670 tx=0x7ff09002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c93a045adb17b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.986+0000 7ff09effd640 1 -- 172.21.15.59:0/2081553283 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff090034070 con 0x7ff0a00b5240 2024-03-06T07:26:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.986+0000 7ff09effd640 1 -- 172.21.15.59:0/2081553283 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff090030860 con 0x7ff0a00b5240 2024-03-06T07:26:58.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.986+0000 7ff09effd640 1 -- 172.21.15.59:0/2081553283 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff090030c30 con 0x7ff0a00b5240 2024-03-06T07:26:58.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.987+0000 7ff0a650c640 1 -- 172.21.15.59:0/2081553283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0a00b5240 msgr2=0x7ff0a00d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.987+0000 7ff0a650c640 1 --2- 172.21.15.59:0/2081553283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0a00b5240 0x7ff0a00d1b30 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7ff0a008e670 tx=0x7ff09002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.987+0000 7ff0a650c640 1 -- 172.21.15.59:0/2081553283 shutdown_connections 2024-03-06T07:26:58.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.987+0000 7ff0a650c640 1 --2- 172.21.15.59:0/2081553283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0a00b5240 0x7ff0a00d1b30 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.987+0000 7ff0a650c640 1 --2- 172.21.15.59:0/2081553283 >> v2:172.21.15.164:3300/0 conn(0x7ff0a0161760 0x7ff0a00b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.987+0000 7ff0a650c640 1 -- 172.21.15.59:0/2081553283 >> 172.21.15.59:0/2081553283 conn(0x7ff0a00d3f60 msgr2=0x7ff0a00d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.987+0000 7ff0a650c640 1 -- 172.21.15.59:0/2081553283 shutdown_connections 2024-03-06T07:26:58.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.987+0000 7ff0a650c640 1 -- 172.21.15.59:0/2081553283 wait complete. 2024-03-06T07:26:58.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.988+0000 7ff0a650c640 1 Processor -- start 2024-03-06T07:26:58.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.988+0000 7ff0a650c640 1 -- start start 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff0a650c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0a00b5240 0x7ff0a0155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff0a650c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0a00d2070 0x7ff0a00b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff0a650c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0a0161760 0x7ff0a009f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff0a650c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00cfd80 con 0x7ff0a00d2070 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff0a650c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00d24e0 con 0x7ff0a00b5240 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff0a650c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00d2ac0 con 0x7ff0a0161760 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0a00d2070 0x7ff0a00b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0a00d2070 0x7ff0a00b5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49132/0 (socket says 172.21.15.59:49132) 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09f7fe640 1 -- 172.21.15.59:0/95792358 learned_addr learned my addr 172.21.15.59:0/95792358 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09ffff640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0a00b5240 0x7ff0a0155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff0a4aa9640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0a0161760 0x7ff0a009f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09f7fe640 1 -- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0a0161760 msgr2=0x7ff0a009f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09f7fe640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0a0161760 0x7ff0a009f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09f7fe640 1 -- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0a00b5240 msgr2=0x7ff0a0155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09f7fe640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0a00b5240 0x7ff0a0155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09f7fe640 1 -- 172.21.15.59:0/95792358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a00b6910 con 0x7ff0a00d2070 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09ffff640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0a00b5240 0x7ff0a0155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09f7fe640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0a00d2070 0x7ff0a00b5e90 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7ff09002b9e0 tx=0x7ff090004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.989+0000 7ff09cff9640 1 -- 172.21.15.59:0/95792358 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff090048070 con 0x7ff0a00d2070 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.990+0000 7ff09cff9640 1 -- 172.21.15.59:0/95792358 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff090030660 con 0x7ff0a00d2070 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.990+0000 7ff0a650c640 1 -- 172.21.15.59:0/95792358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a00b6bf0 con 0x7ff0a00d2070 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.990+0000 7ff09cff9640 1 -- 172.21.15.59:0/95792358 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff090034040 con 0x7ff0a00d2070 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.990+0000 7ff0a650c640 1 -- 172.21.15.59:0/95792358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a00b71b0 con 0x7ff0a00d2070 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.991+0000 7ff0a650c640 1 -- 172.21.15.59:0/95792358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a0069ee0 con 0x7ff0a00d2070 2024-03-06T07:26:58.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.991+0000 7ff09cff9640 1 -- 172.21.15.59:0/95792358 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff09004b070 con 0x7ff0a00d2070 2024-03-06T07:26:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.991+0000 7ff09cff9640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff08c06b4d0 0x7ff08c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.991+0000 7ff09cff9640 1 -- 172.21.15.59:0/95792358 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff0900ac250 con 0x7ff0a00d2070 2024-03-06T07:26:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.992+0000 7ff09ffff640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff08c06b4d0 0x7ff08c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.992+0000 7ff09ffff640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff08c06b4d0 0x7ff08c06d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff094006fd0 tx=0x7ff094008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:58.996+0000 7ff09cff9640 1 -- 172.21.15.59:0/95792358 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff09007b060 con 0x7ff0a00d2070 2024-03-06T07:26:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.253+0000 7ff0a650c640 1 -- 172.21.15.59:0/95792358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff0a00c9100 con 0x7ff0a00d2070 2024-03-06T07:26:59.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.253+0000 7ff09cff9640 1 -- 172.21.15.59:0/95792358 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff09007a960 con 0x7ff0a00d2070 2024-03-06T07:26:59.255 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:26:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.255+0000 7ff0a650c640 1 -- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff08c06b4d0 msgr2=0x7ff08c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.255+0000 7ff0a650c640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff08c06b4d0 0x7ff08c06d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff094006fd0 tx=0x7ff094008040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.256+0000 7ff0a650c640 1 -- 172.21.15.59:0/95792358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0a00d2070 msgr2=0x7ff0a00b5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.256+0000 7ff0a650c640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0a00d2070 0x7ff0a00b5e90 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7ff09002b9e0 tx=0x7ff090004730 comp rx=0 tx=0).stop 2024-03-06T07:26:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.256+0000 7ff0a650c640 1 -- 172.21.15.59:0/95792358 shutdown_connections 2024-03-06T07:26:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.256+0000 7ff0a650c640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff08c06b4d0 0x7ff08c06d990 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.256+0000 7ff0a650c640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0a0161760 0x7ff0a009f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.256+0000 7ff0a650c640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0a00d2070 0x7ff0a00b5e90 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.256+0000 7ff0a650c640 1 --2- 172.21.15.59:0/95792358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0a00b5240 0x7ff0a0155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.256+0000 7ff0a650c640 1 -- 172.21.15.59:0/95792358 >> 172.21.15.59:0/95792358 conn(0x7ff0a00d3f60 msgr2=0x7ff0a006c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.256+0000 7ff0a650c640 1 -- 172.21.15.59:0/95792358 shutdown_connections 2024-03-06T07:26:59.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.256+0000 7ff0a650c640 1 -- 172.21.15.59:0/95792358 wait complete. 2024-03-06T07:26:59.269 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:59.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.372+0000 7f5b890c3640 1 Processor -- start 2024-03-06T07:26:59.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.372+0000 7f5b890c3640 1 -- start start 2024-03-06T07:26:59.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.373+0000 7f5b890c3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5b840cd690 0x7f5b84162570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.373+0000 7f5b890c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 0x7f5b840cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.373+0000 7f5b890c3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b84162c70 con 0x7f5b84161670 2024-03-06T07:26:59.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.373+0000 7f5b890c3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b840b3590 con 0x7f5b840cd690 2024-03-06T07:26:59.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.373+0000 7f5b890c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8406cd80 con 0x7f5b840b2da0 2024-03-06T07:26:59.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.374+0000 7f5b8359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 0x7f5b840cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.374+0000 7f5b8359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 0x7f5b840cff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:45772/0 (socket says 172.21.15.59:45772) 2024-03-06T07:26:59.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.374+0000 7f5b8359e640 1 -- 172.21.15.59:0/3534782317 learned_addr learned my addr 172.21.15.59:0/3534782317 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:59.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.374+0000 7f5b8259c640 1 --2- 172.21.15.59:0/3534782317 >> v2:172.21.15.164:3300/0 conn(0x7f5b840cd690 0x7f5b84162570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.374+0000 7f5b8359e640 1 -- 172.21.15.59:0/3534782317 >> v2:172.21.15.164:3300/0 conn(0x7f5b840cd690 msgr2=0x7f5b84162570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.374+0000 7f5b8359e640 1 --2- 172.21.15.59:0/3534782317 >> v2:172.21.15.164:3300/0 conn(0x7f5b840cd690 0x7f5b84162570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.374+0000 7f5b8359e640 1 -- 172.21.15.59:0/3534782317 >> v1:172.21.15.164:6789/0 conn(0x7f5b84161670 legacy=0x7f5b84161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.374+0000 7f5b8359e640 1 -- 172.21.15.59:0/3534782317 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8406d060 con 0x7f5b840b2da0 2024-03-06T07:26:59.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.375+0000 7f5b8359e640 1 --2- 172.21.15.59:0/3534782317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 0x7f5b840cff50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009e80 tx=0x7f5b7c02bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ff41fdde8e19baa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.375+0000 7f5b81d9b640 1 -- 172.21.15.59:0/3534782317 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c02caf0 con 0x7f5b840b2da0 2024-03-06T07:26:59.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.375+0000 7f5b81d9b640 1 -- 172.21.15.59:0/3534782317 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c02cc90 con 0x7f5b840b2da0 2024-03-06T07:26:59.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.375+0000 7f5b81d9b640 1 -- 172.21.15.59:0/3534782317 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c034610 con 0x7f5b840b2da0 2024-03-06T07:26:59.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.376+0000 7f5b890c3640 1 -- 172.21.15.59:0/3534782317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 msgr2=0x7f5b840cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.376+0000 7f5b890c3640 1 --2- 172.21.15.59:0/3534782317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 0x7f5b840cff50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009e80 tx=0x7f5b7c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:26:59.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.376+0000 7f5b890c3640 1 -- 172.21.15.59:0/3534782317 shutdown_connections 2024-03-06T07:26:59.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.376+0000 7f5b890c3640 1 --2- 172.21.15.59:0/3534782317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 0x7f5b840cff50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.376+0000 7f5b890c3640 1 --2- 172.21.15.59:0/3534782317 >> v2:172.21.15.164:3300/0 conn(0x7f5b840cd690 0x7f5b84162570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.376+0000 7f5b890c3640 1 -- 172.21.15.59:0/3534782317 >> 172.21.15.59:0/3534782317 conn(0x7f5b840d1d50 msgr2=0x7f5b840d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.376+0000 7f5b890c3640 1 -- 172.21.15.59:0/3534782317 shutdown_connections 2024-03-06T07:26:59.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.376+0000 7f5b890c3640 1 -- 172.21.15.59:0/3534782317 wait complete. 2024-03-06T07:26:59.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.378+0000 7f5b890c3640 1 Processor -- start 2024-03-06T07:26:59.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.378+0000 7f5b890c3640 1 -- start start 2024-03-06T07:26:59.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.379+0000 7f5b890c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 0x7f5b840cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.379+0000 7f5b890c3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b840cd690 0x7f5b840ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.379+0000 7f5b890c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b84161670 0x7f5b84151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.379+0000 7f5b890c3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8406d4c0 con 0x7f5b84161670 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.379+0000 7f5b890c3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8406d340 con 0x7f5b840cd690 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.379+0000 7f5b890c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b84161a50 con 0x7f5b840b2da0 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.379+0000 7f5b8359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b84161670 0x7f5b84151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.379+0000 7f5b8359e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b84161670 0x7f5b84151f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49142/0 (socket says 172.21.15.59:49142) 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.379+0000 7f5b8359e640 1 -- 172.21.15.59:0/2192660246 learned_addr learned my addr 172.21.15.59:0/2192660246 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b8259c640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b840cd690 0x7f5b840ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b82d9d640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 0x7f5b840cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b8359e640 1 -- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 msgr2=0x7f5b840cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b8359e640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 0x7f5b840cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b8359e640 1 -- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b840cd690 msgr2=0x7f5b840ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b8359e640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b840cd690 0x7f5b840ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b8359e640 1 -- 172.21.15.59:0/2192660246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b840c7510 con 0x7f5b84161670 2024-03-06T07:26:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b82d9d640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 0x7f5b840cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b8359e640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b84161670 0x7f5b84151f50 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c036040 tx=0x7f5b7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2192660246 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c043020 con 0x7f5b84161670 2024-03-06T07:26:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2192660246 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c03e450 con 0x7f5b84161670 2024-03-06T07:26:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b890c3640 1 -- 172.21.15.59:0/2192660246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b840c7780 con 0x7f5b84161670 2024-03-06T07:26:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2192660246 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c009510 con 0x7f5b84161670 2024-03-06T07:26:59.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.380+0000 7f5b890c3640 1 -- 172.21.15.59:0/2192660246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b840c7d40 con 0x7f5b84161670 2024-03-06T07:26:59.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.381+0000 7f5b890c3640 1 -- 172.21.15.59:0/2192660246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b84068050 con 0x7f5b84161670 2024-03-06T07:26:59.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.382+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2192660246 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b7c046020 con 0x7f5b84161670 2024-03-06T07:26:59.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.382+0000 7f5b7b7fe640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b6406b4d0 0x7f5b6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.382+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2192660246 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5b7c0ac970 con 0x7f5b84161670 2024-03-06T07:26:59.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.382+0000 7f5b82d9d640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b6406b4d0 0x7f5b6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.383+0000 7f5b82d9d640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b6406b4d0 0x7f5b6406d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5b840cc230 tx=0x7f5b74009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.387+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2192660246 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b7c04d030 con 0x7f5b84161670 2024-03-06T07:26:59.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.566+0000 7f5b890c3640 1 -- 172.21.15.59:0/2192660246 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b840c8b90 con 0x7f5b84161670 2024-03-06T07:26:59.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.567+0000 7f5b7b7fe640 1 -- 172.21.15.59:0/2192660246 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b7c07b810 con 0x7f5b84161670 2024-03-06T07:26:59.569 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:26:59.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 -- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b6406b4d0 msgr2=0x7f5b6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b6406b4d0 0x7f5b6406d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5b840cc230 tx=0x7f5b74009040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 -- 172.21.15.59:0/2192660246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b84161670 msgr2=0x7f5b84151f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b84161670 0x7f5b84151f50 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c036040 tx=0x7f5b7c004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:59.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 -- 172.21.15.59:0/2192660246 shutdown_connections 2024-03-06T07:26:59.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b6406b4d0 0x7f5b6406d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b84161670 0x7f5b84151f50 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b840cd690 0x7f5b840ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 --2- 172.21.15.59:0/2192660246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b840b2da0 0x7f5b840cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 -- 172.21.15.59:0/2192660246 >> 172.21.15.59:0/2192660246 conn(0x7f5b840d1d50 msgr2=0x7f5b840d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 -- 172.21.15.59:0/2192660246 shutdown_connections 2024-03-06T07:26:59.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.569+0000 7f5b890c3640 1 -- 172.21.15.59:0/2192660246 wait complete. 2024-03-06T07:26:59.582 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:59.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.685+0000 7f0f1698a640 1 Processor -- start 2024-03-06T07:26:59.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.685+0000 7f0f1698a640 1 -- start start 2024-03-06T07:26:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.686+0000 7f0f1698a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0f1015e150 0x7f0f100cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.686+0000 7f0f1698a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0f100d1060 0x7f0f100b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.686+0000 7f0f1698a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f100d1ea0 con 0x7f0f100cfba0 2024-03-06T07:26:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.686+0000 7f0f1698a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f100d2ab0 con 0x7f0f1015e150 2024-03-06T07:26:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.686+0000 7f0f1698a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f100d2180 con 0x7f0f100d1060 2024-03-06T07:26:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.686+0000 7f0f0ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0f1015e150 0x7f0f100cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.686+0000 7f0f0ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0f1015e150 0x7f0f100cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49144/0 (socket says 172.21.15.59:49144) 2024-03-06T07:26:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.686+0000 7f0f0ffff640 1 -- 172.21.15.59:0/693123283 learned_addr learned my addr 172.21.15.59:0/693123283 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.686+0000 7f0f14f27640 1 --2- 172.21.15.59:0/693123283 >> v2:172.21.15.164:3300/0 conn(0x7f0f100d1060 0x7f0f100b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.687+0000 7f0f0ffff640 1 -- 172.21.15.59:0/693123283 >> v2:172.21.15.164:3300/0 conn(0x7f0f100d1060 msgr2=0x7f0f100b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.687+0000 7f0f0ffff640 1 --2- 172.21.15.59:0/693123283 >> v2:172.21.15.164:3300/0 conn(0x7f0f100d1060 0x7f0f100b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.687+0000 7f0f0ffff640 1 -- 172.21.15.59:0/693123283 >> v1:172.21.15.59:6789/0 conn(0x7f0f100cfba0 legacy=0x7f0f100d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.687+0000 7f0f0ffff640 1 -- 172.21.15.59:0/693123283 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f100d2460 con 0x7f0f1015e150 2024-03-06T07:26:59.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.687+0000 7f0f0ffff640 1 --2- 172.21.15.59:0/693123283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f1015e150 0x7f0f100cf660 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f0f1008c8d0 tx=0x7f0f0002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5937bd6cd00cea9f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.688+0000 7f0f0effd640 1 -- 172.21.15.59:0/693123283 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f00035020 con 0x7f0f1015e150 2024-03-06T07:26:59.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.688+0000 7f0f0effd640 1 -- 172.21.15.59:0/693123283 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f00031870 con 0x7f0f1015e150 2024-03-06T07:26:59.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.688+0000 7f0f0effd640 1 -- 172.21.15.59:0/693123283 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f00031c40 con 0x7f0f1015e150 2024-03-06T07:26:59.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.688+0000 7f0f1698a640 1 -- 172.21.15.59:0/693123283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f1015e150 msgr2=0x7f0f100cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.688+0000 7f0f1698a640 1 --2- 172.21.15.59:0/693123283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f1015e150 0x7f0f100cf660 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f0f1008c8d0 tx=0x7f0f0002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:59.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.688+0000 7f0f1698a640 1 -- 172.21.15.59:0/693123283 shutdown_connections 2024-03-06T07:26:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.688+0000 7f0f1698a640 1 --2- 172.21.15.59:0/693123283 >> v2:172.21.15.164:3300/0 conn(0x7f0f100d1060 0x7f0f100b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.688+0000 7f0f1698a640 1 --2- 172.21.15.59:0/693123283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f1015e150 0x7f0f100cf660 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.688+0000 7f0f1698a640 1 -- 172.21.15.59:0/693123283 >> 172.21.15.59:0/693123283 conn(0x7f0f1009d230 msgr2=0x7f0f100d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.688+0000 7f0f1698a640 1 -- 172.21.15.59:0/693123283 shutdown_connections 2024-03-06T07:26:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.688+0000 7f0f1698a640 1 -- 172.21.15.59:0/693123283 wait complete. 2024-03-06T07:26:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.689+0000 7f0f1698a640 1 Processor -- start 2024-03-06T07:26:59.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.690+0000 7f0f1698a640 1 -- start start 2024-03-06T07:26:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.690+0000 7f0f1698a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f100cfba0 0x7f0f1015cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.690+0000 7f0f1698a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f100d1060 0x7f0f1015fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.690+0000 7f0f1698a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f1015e150 0x7f0f10160000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.690+0000 7f0f1698a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f100d1b30 con 0x7f0f100d1060 2024-03-06T07:26:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.690+0000 7f0f1698a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f100d2f40 con 0x7f0f1015e150 2024-03-06T07:26:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.690+0000 7f0f1698a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1015e530 con 0x7f0f100cfba0 2024-03-06T07:26:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.690+0000 7f0f0f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f100d1060 0x7f0f1015fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.690+0000 7f0f0f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f100d1060 0x7f0f1015fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49154/0 (socket says 172.21.15.59:49154) 2024-03-06T07:26:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.690+0000 7f0f0f7fe640 1 -- 172.21.15.59:0/3756432102 learned_addr learned my addr 172.21.15.59:0/3756432102 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:26:59.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.690+0000 7f0f14f27640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f1015e150 0x7f0f10160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f0ffff640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f100cfba0 0x7f0f1015cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f0f7fe640 1 -- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f100cfba0 msgr2=0x7f0f1015cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f0f7fe640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f100cfba0 0x7f0f1015cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f0f7fe640 1 -- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f1015e150 msgr2=0x7f0f10160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f0f7fe640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f1015e150 0x7f0f10160000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f0f7fe640 1 -- 172.21.15.59:0/3756432102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1015d910 con 0x7f0f100d1060 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f0ffff640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f100cfba0 0x7f0f1015cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f0f7fe640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f100d1060 0x7f0f1015fac0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400c400 tx=0x7f0f0400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f0cff9640 1 -- 172.21.15.59:0/3756432102 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f040052d0 con 0x7f0f100d1060 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f0cff9640 1 -- 172.21.15.59:0/3756432102 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f04005490 con 0x7f0f100d1060 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f1698a640 1 -- 172.21.15.59:0/3756432102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f100cced0 con 0x7f0f100d1060 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f0cff9640 1 -- 172.21.15.59:0/3756432102 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04007260 con 0x7f0f100d1060 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.691+0000 7f0f1698a640 1 -- 172.21.15.59:0/3756432102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f100cd490 con 0x7f0f100d1060 2024-03-06T07:26:59.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.692+0000 7f0f1698a640 1 -- 172.21.15.59:0/3756432102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f10069ee0 con 0x7f0f100d1060 2024-03-06T07:26:59.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.692+0000 7f0f0cff9640 1 -- 172.21.15.59:0/3756432102 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f04007400 con 0x7f0f100d1060 2024-03-06T07:26:59.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.693+0000 7f0f0cff9640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0efc06b4d0 0x7f0efc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.693+0000 7f0f0cff9640 1 -- 172.21.15.59:0/3756432102 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f0f04089860 con 0x7f0f100d1060 2024-03-06T07:26:59.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.693+0000 7f0f0ffff640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0efc06b4d0 0x7f0efc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.694+0000 7f0f0ffff640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0efc06b4d0 0x7f0efc06d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0f00036000 tx=0x7f0f0003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.697+0000 7f0f0cff9640 1 -- 172.21.15.59:0/3756432102 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f1015d910 con 0x7f0f100d1060 2024-03-06T07:26:59.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.917+0000 7f0f1698a640 1 -- 172.21.15.59:0/3756432102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f100d4670 con 0x7f0f100d1060 2024-03-06T07:26:59.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.917+0000 7f0f0cff9640 1 -- 172.21.15.59:0/3756432102 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f0f04057f00 con 0x7f0f100d1060 2024-03-06T07:26:59.919 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:26:59.920 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:26:59.920 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:26:59.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.919+0000 7f0f1698a640 1 -- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0efc06b4d0 msgr2=0x7f0efc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.919+0000 7f0f1698a640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0efc06b4d0 0x7f0efc06d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0f00036000 tx=0x7f0f0003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.919+0000 7f0f1698a640 1 -- 172.21.15.59:0/3756432102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f100d1060 msgr2=0x7f0f1015fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.919+0000 7f0f1698a640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f100d1060 0x7f0f1015fac0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400c400 tx=0x7f0f0400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.919+0000 7f0f1698a640 1 -- 172.21.15.59:0/3756432102 shutdown_connections 2024-03-06T07:26:59.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.920+0000 7f0f1698a640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0efc06b4d0 0x7f0efc06d990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.920+0000 7f0f1698a640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f1015e150 0x7f0f10160000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.920+0000 7f0f1698a640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f100d1060 0x7f0f1015fac0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.920+0000 7f0f1698a640 1 --2- 172.21.15.59:0/3756432102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f100cfba0 0x7f0f1015cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.920+0000 7f0f1698a640 1 -- 172.21.15.59:0/3756432102 >> 172.21.15.59:0/3756432102 conn(0x7f0f1009d230 msgr2=0x7f0f101612a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.920+0000 7f0f1698a640 1 -- 172.21.15.59:0/3756432102 shutdown_connections 2024-03-06T07:26:59.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:26:59.920+0000 7f0f1698a640 1 -- 172.21.15.59:0/3756432102 wait complete. 2024-03-06T07:26:59.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.033+0000 7f7f31676640 1 Processor -- start 2024-03-06T07:27:00.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.033+0000 7f7f31676640 1 -- start start 2024-03-06T07:27:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.034+0000 7f7f31676640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f2c15f5e0 0x7f7f2c0b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.034+0000 7f7f31676640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c0cec60 0x7f7f2c0cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.034+0000 7f7f31676640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c0cd4d0 con 0x7f7f2c0b16b0 2024-03-06T07:27:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.034+0000 7f7f31676640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2c0cf920 con 0x7f7f2c15f5e0 2024-03-06T07:27:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.034+0000 7f7f31676640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0cd7b0 con 0x7f7f2c0cec60 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.034+0000 7f7f23fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7f2c0b16b0 0x7f7f2c0ce550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36064/0 (socket says 172.21.15.59:36064) 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.034+0000 7f7f23fff640 1 -- 172.21.15.59:0/4118224668 learned_addr learned my addr 172.21.15.59:0/4118224668 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2affd640 1 --2- 172.21.15.59:0/4118224668 >> v2:172.21.15.164:3300/0 conn(0x7f7f2c15f5e0 0x7f7f2c0b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584916277 0 0) 0x7f7f2c0cd4d0 con 0x7f7f2c0b16b0 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f18003520 con 0x7f7f2c0b16b0 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2b7fe640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c0cec60 0x7f7f2c0cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754661990 0 0) 0x7f7f18003520 con 0x7f7f2c0b16b0 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c0cec60 msgr2=0x7f7f2c0cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c0cec60 0x7f7f2c0cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 >> v2:172.21.15.164:3300/0 conn(0x7f7f2c15f5e0 msgr2=0x7f7f2c0b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 --2- 172.21.15.59:0/4118224668 >> v2:172.21.15.164:3300/0 conn(0x7f7f2c15f5e0 0x7f7f2c0b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c0cda90 con 0x7f7f2c0b16b0 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f7f14001670 con 0x7f7f2c0b16b0 2024-03-06T07:27:00.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 >> v1:172.21.15.59:6789/0 conn(0x7f7f2c0b16b0 legacy=0x7f7f2c0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c0cec60 0x7f7f180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f18003f30 0x7f7f18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f18004d40 0x7f7f180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0cf920 con 0x7f7f18003f30 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0cd7b0 con 0x7f7f18004d40 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1800ba10 con 0x7f7f2c0cec60 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f23fff640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f18003f30 0x7f7f18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2affd640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c0cec60 0x7f7f180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f2b7fe640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f18004d40 0x7f7f180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f23fff640 1 -- 172.21.15.59:0/4118224668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c0cec60 msgr2=0x7f7f180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f23fff640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c0cec60 0x7f7f180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f23fff640 1 -- 172.21.15.59:0/4118224668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f18004d40 msgr2=0x7f7f180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f23fff640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f18004d40 0x7f7f180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.035+0000 7f7f23fff640 1 -- 172.21.15.59:0/4118224668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f18003520 con 0x7f7f18003f30 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.036+0000 7f7f23fff640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f18003f30 0x7f7f18004800 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0b1ad0 tx=0x7f7f18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d393bd8dc2c096a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.036+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14007d70 con 0x7f7f18003f30 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.036+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f14030070 con 0x7f7f18003f30 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.036+0000 7f7f2a7fc640 1 -- 172.21.15.59:0/4118224668 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14005d30 con 0x7f7f18003f30 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.037+0000 7f7f31676640 1 -- 172.21.15.59:0/4118224668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f18003f30 msgr2=0x7f7f18004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.037+0000 7f7f31676640 1 --2- 172.21.15.59:0/4118224668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f18003f30 0x7f7f18004800 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0b1ad0 tx=0x7f7f18005830 comp rx=0 tx=0).stop 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.037+0000 7f7f2b7fe640 1 -- 172.21.15.59:0/4118224668 reap_dead start 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.037+0000 7f7f31676640 1 -- 172.21.15.59:0/4118224668 shutdown_connections 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.037+0000 7f7f31676640 1 -- 172.21.15.59:0/4118224668 >> 172.21.15.59:0/4118224668 conn(0x7f7f2c0d1d50 msgr2=0x7f7f2c0d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.037+0000 7f7f31676640 1 -- 172.21.15.59:0/4118224668 shutdown_connections 2024-03-06T07:27:00.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.037+0000 7f7f31676640 1 -- 172.21.15.59:0/4118224668 wait complete. 2024-03-06T07:27:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.039+0000 7f7f31676640 1 Processor -- start 2024-03-06T07:27:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.039+0000 7f7f31676640 1 -- start start 2024-03-06T07:27:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f31676640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f18003f30 0x7f7f2c0b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f31676640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c15e420 0x7f7f2c1f52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f31676640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c1f57f0 0x7f7f2c0cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f31676640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0cf270 con 0x7f7f2c15e420 2024-03-06T07:27:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f31676640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0cf0f0 con 0x7f7f18003f30 2024-03-06T07:27:00.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f31676640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0d2870 con 0x7f7f2c1f57f0 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f23fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c15e420 0x7f7f2c1f52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f23fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c15e420 0x7f7f2c1f52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49180/0 (socket says 172.21.15.59:49180) 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f23fff640 1 -- 172.21.15.59:0/98759754 learned_addr learned my addr 172.21.15.59:0/98759754 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f2affd640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f18003f30 0x7f7f2c0b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f2b7fe640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c1f57f0 0x7f7f2c0cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f23fff640 1 -- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c1f57f0 msgr2=0x7f7f2c0cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f23fff640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c1f57f0 0x7f7f2c0cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.040+0000 7f7f23fff640 1 -- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f18003f30 msgr2=0x7f7f2c0b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.041+0000 7f7f23fff640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f18003f30 0x7f7f2c0b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.041+0000 7f7f23fff640 1 -- 172.21.15.59:0/98759754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c0b73a0 con 0x7f7f2c15e420 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.041+0000 7f7f2affd640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f18003f30 0x7f7f2c0b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.041+0000 7f7f23fff640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c15e420 0x7f7f2c1f52b0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400c300 tx=0x7f7f14000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.041+0000 7f7f237fe640 1 -- 172.21.15.59:0/98759754 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14004bf0 con 0x7f7f2c15e420 2024-03-06T07:27:00.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.041+0000 7f7f237fe640 1 -- 172.21.15.59:0/98759754 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f14004d90 con 0x7f7f2c15e420 2024-03-06T07:27:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.041+0000 7f7f31676640 1 -- 172.21.15.59:0/98759754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c0b59e0 con 0x7f7f2c15e420 2024-03-06T07:27:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.041+0000 7f7f237fe640 1 -- 172.21.15.59:0/98759754 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14007270 con 0x7f7f2c15e420 2024-03-06T07:27:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.041+0000 7f7f31676640 1 -- 172.21.15.59:0/98759754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c1f6100 con 0x7f7f2c15e420 2024-03-06T07:27:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.042+0000 7f7f31676640 1 -- 172.21.15.59:0/98759754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2c0b7a20 con 0x7f7f2c15e420 2024-03-06T07:27:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.042+0000 7f7f237fe640 1 -- 172.21.15.59:0/98759754 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f14007410 con 0x7f7f2c15e420 2024-03-06T07:27:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.043+0000 7f7f237fe640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0c06b500 0x7f7f0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.043+0000 7f7f237fe640 1 -- 172.21.15.59:0/98759754 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f7f140ab3a0 con 0x7f7f2c15e420 2024-03-06T07:27:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.043+0000 7f7f2affd640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0c06b500 0x7f7f0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.044+0000 7f7f2affd640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0c06b500 0x7f7f0c06d9c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009bc0 tx=0x7f7f1c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.047+0000 7f7f237fe640 1 -- 172.21.15.59:0/98759754 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f2c0b7a20 con 0x7f7f2c15e420 2024-03-06T07:27:00.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.234+0000 7f7f31676640 1 -- 172.21.15.59:0/98759754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f2c068050 con 0x7f7f2c15e420 2024-03-06T07:27:00.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.235+0000 7f7f237fe640 1 -- 172.21.15.59:0/98759754 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16987 (secure 0 0 0) 0x7f7f14079ad0 con 0x7f7f2c15e420 2024-03-06T07:27:00.237 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:00.237 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":96,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:26:29.601683+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.489280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:16.183614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6834/2799249020":"2024-03-07T07:26:29.597825+0000","172.21.15.59:6835/1784120157":"2024-03-07T07:26:29.585744+0000","172.21.15.59:6832/2799249020":"2024-03-07T07:26:29.597825+0000","172.21.15.59:6833/1784120157":"2024-03-07T07:26:29.585744+0000","172.21.15.164:6837/4292771408":"2024-03-07T07:26:29.578123+0000","172.21.15.164:6835/4292771408":"2024-03-07T07:26:29.578123+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:00.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 -- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0c06b500 msgr2=0x7f7f0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0c06b500 0x7f7f0c06d9c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009bc0 tx=0x7f7f1c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:00.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 -- 172.21.15.59:0/98759754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c15e420 msgr2=0x7f7f2c1f52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c15e420 0x7f7f2c1f52b0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400c300 tx=0x7f7f14000fe0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 -- 172.21.15.59:0/98759754 shutdown_connections 2024-03-06T07:27:00.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f0c06b500 0x7f7f0c06d9c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f2c1f57f0 0x7f7f2c0cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c15e420 0x7f7f2c1f52b0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 --2- 172.21.15.59:0/98759754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f18003f30 0x7f7f2c0b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 -- 172.21.15.59:0/98759754 >> 172.21.15.59:0/98759754 conn(0x7f7f2c0d1d50 msgr2=0x7f7f2c15a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 -- 172.21.15.59:0/98759754 shutdown_connections 2024-03-06T07:27:00.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.237+0000 7f7f31676640 1 -- 172.21.15.59:0/98759754 wait complete. 2024-03-06T07:27:00.251 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:27:00.251 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.351+0000 7fc6079dd640 1 Processor -- start 2024-03-06T07:27:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.351+0000 7fc6079dd640 1 -- start start 2024-03-06T07:27:00.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc6079dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc600161a80 0x7fc6000d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc6079dd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc6000b5010 0x7fc6000b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc6079dd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc6000d25a0 0x7fc6000d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc6079dd640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6000b3e30 con 0x7fc6000b5010 2024-03-06T07:27:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc6079dd640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc600161e60 con 0x7fc6000d25a0 2024-03-06T07:27:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc6079dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6000d1470 con 0x7fc600161a80 2024-03-06T07:27:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc604f78640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc6000b5010 0x7fc6000b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc604f78640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc6000b5010 0x7fc6000b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49182/0 (socket says 172.21.15.59:49182) 2024-03-06T07:27:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc604f78640 1 -- 172.21.15.59:0/360132594 learned_addr learned my addr 172.21.15.59:0/360132594 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc605779640 1 --2- 172.21.15.59:0/360132594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc600161a80 0x7fc6000d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc605f7a640 1 --2- 172.21.15.59:0/360132594 >> v2:172.21.15.164:3300/0 conn(0x7fc6000d25a0 0x7fc6000d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc604f78640 1 -- 172.21.15.59:0/360132594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc600161a80 msgr2=0x7fc6000d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc604f78640 1 --2- 172.21.15.59:0/360132594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc600161a80 0x7fc6000d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc604f78640 1 -- 172.21.15.59:0/360132594 >> v2:172.21.15.164:3300/0 conn(0x7fc6000d25a0 msgr2=0x7fc6000d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc604f78640 1 --2- 172.21.15.59:0/360132594 >> v2:172.21.15.164:3300/0 conn(0x7fc6000d25a0 0x7fc6000d2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.352+0000 7fc604f78640 1 -- 172.21.15.59:0/360132594 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6000d1750 con 0x7fc6000b5010 2024-03-06T07:27:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.353+0000 7fc604f78640 1 --2- 172.21.15.59:0/360132594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6000b5010 0x7fc6000b0b40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fc6000d5010 tx=0x7fc5f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f673af1b700f7f62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.353+0000 7fc5f7fff640 1 -- 172.21.15.59:0/360132594 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0035020 con 0x7fc6000b5010 2024-03-06T07:27:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.353+0000 7fc5f7fff640 1 -- 172.21.15.59:0/360132594 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f0031870 con 0x7fc6000b5010 2024-03-06T07:27:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.353+0000 7fc5f7fff640 1 -- 172.21.15.59:0/360132594 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0031c40 con 0x7fc6000b5010 2024-03-06T07:27:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.354+0000 7fc6079dd640 1 -- 172.21.15.59:0/360132594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6000b5010 msgr2=0x7fc6000b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.354+0000 7fc6079dd640 1 --2- 172.21.15.59:0/360132594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6000b5010 0x7fc6000b0b40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fc6000d5010 tx=0x7fc5f002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.354+0000 7fc6079dd640 1 -- 172.21.15.59:0/360132594 shutdown_connections 2024-03-06T07:27:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.354+0000 7fc6079dd640 1 --2- 172.21.15.59:0/360132594 >> v2:172.21.15.164:3300/0 conn(0x7fc6000d25a0 0x7fc6000d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.354+0000 7fc6079dd640 1 --2- 172.21.15.59:0/360132594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6000b5010 0x7fc6000b0b40 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.354+0000 7fc6079dd640 1 --2- 172.21.15.59:0/360132594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc600161a80 0x7fc6000d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.354+0000 7fc6079dd640 1 -- 172.21.15.59:0/360132594 >> 172.21.15.59:0/360132594 conn(0x7fc6000d3fc0 msgr2=0x7fc6000d43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.354+0000 7fc6079dd640 1 -- 172.21.15.59:0/360132594 shutdown_connections 2024-03-06T07:27:00.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.354+0000 7fc6079dd640 1 -- 172.21.15.59:0/360132594 wait complete. 2024-03-06T07:27:00.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.355+0000 7fc6079dd640 1 Processor -- start 2024-03-06T07:27:00.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.355+0000 7fc6079dd640 1 -- start start 2024-03-06T07:27:00.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc6079dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6000b5010 0x7fc6001f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc6079dd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6000d25a0 0x7fc6000b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc6079dd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc600161a80 0x7fc6000b8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc6079dd640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6000d1210 con 0x7fc6000d25a0 2024-03-06T07:27:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc6079dd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6000d1090 con 0x7fc600161a80 2024-03-06T07:27:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc6079dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6000b1080 con 0x7fc6000b5010 2024-03-06T07:27:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc604f78640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6000d25a0 0x7fc6000b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc604f78640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6000d25a0 0x7fc6000b8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49188/0 (socket says 172.21.15.59:49188) 2024-03-06T07:27:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc604f78640 1 -- 172.21.15.59:0/357018883 learned_addr learned my addr 172.21.15.59:0/357018883 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc605779640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6000b5010 0x7fc6001f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc605f7a640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc600161a80 0x7fc6000b8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc604f78640 1 -- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6000b5010 msgr2=0x7fc6001f49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc604f78640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6000b5010 0x7fc6001f49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc604f78640 1 -- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc600161a80 msgr2=0x7fc6000b8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc604f78640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc600161a80 0x7fc6000b8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc604f78640 1 -- 172.21.15.59:0/357018883 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6000b6fc0 con 0x7fc6000d25a0 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.356+0000 7fc604f78640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6000d25a0 0x7fc6000b8a70 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0000c00 tx=0x7fc5f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.357+0000 7fc5f5ffb640 1 -- 172.21.15.59:0/357018883 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0040070 con 0x7fc6000d25a0 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.357+0000 7fc5f5ffb640 1 -- 172.21.15.59:0/357018883 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f0030d00 con 0x7fc6000d25a0 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.357+0000 7fc6079dd640 1 -- 172.21.15.59:0/357018883 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6000b72a0 con 0x7fc6000d25a0 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.357+0000 7fc5f5ffb640 1 -- 172.21.15.59:0/357018883 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0035040 con 0x7fc6000d25a0 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.357+0000 7fc6079dd640 1 -- 172.21.15.59:0/357018883 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6000b7860 con 0x7fc6000d25a0 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.358+0000 7fc6079dd640 1 -- 172.21.15.59:0/357018883 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc600068050 con 0x7fc6000d25a0 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.358+0000 7fc5f5ffb640 1 -- 172.21.15.59:0/357018883 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5f004b060 con 0x7fc6000d25a0 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.359+0000 7fc5f5ffb640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc5e406b4d0 0x7fc5e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.359+0000 7fc5f5ffb640 1 -- 172.21.15.59:0/357018883 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc5f00ac180 con 0x7fc6000d25a0 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.359+0000 7fc605779640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc5e406b4d0 0x7fc5e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.360+0000 7fc605779640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc5e406b4d0 0x7fc5e406d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc006fd0 tx=0x7fc5fc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.363+0000 7fc5f5ffb640 1 -- 172.21.15.59:0/357018883 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5f007ae60 con 0x7fc6000d25a0 2024-03-06T07:27:00.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.583+0000 7fc6079dd640 1 -- 172.21.15.59:0/357018883 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6000b8850 con 0x7fc6000d25a0 2024-03-06T07:27:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.584+0000 7fc5f5ffb640 1 -- 172.21.15.59:0/357018883 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fc5f007a760 con 0x7fc6000d25a0 2024-03-06T07:27:00.586 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:00.586 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:27:00.586 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:27:00.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 -- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc5e406b4d0 msgr2=0x7fc5e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc5e406b4d0 0x7fc5e406d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc006fd0 tx=0x7fc5fc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:00.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 -- 172.21.15.59:0/357018883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6000d25a0 msgr2=0x7fc6000b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6000d25a0 0x7fc6000b8a70 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0000c00 tx=0x7fc5f00046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 -- 172.21.15.59:0/357018883 shutdown_connections 2024-03-06T07:27:00.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc5e406b4d0 0x7fc5e406d990 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc600161a80 0x7fc6000b8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6000d25a0 0x7fc6000b8a70 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 --2- 172.21.15.59:0/357018883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6000b5010 0x7fc6001f49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 -- 172.21.15.59:0/357018883 >> 172.21.15.59:0/357018883 conn(0x7fc6000d3fc0 msgr2=0x7fc6000b46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 -- 172.21.15.59:0/357018883 shutdown_connections 2024-03-06T07:27:00.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.586+0000 7fc6079dd640 1 -- 172.21.15.59:0/357018883 wait complete. 2024-03-06T07:27:00.599 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:00.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.700+0000 7f5234cc2640 1 Processor -- start 2024-03-06T07:27:00.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.700+0000 7f5234cc2640 1 -- start start 2024-03-06T07:27:00.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f5234cc2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5230161670 0x7f5230161e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f5234cc2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52300b2da0 0x7f52301623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f5234cc2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5230162c70 con 0x7f52300cc680 2024-03-06T07:27:00.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f5234cc2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52300b3590 con 0x7f5230161670 2024-03-06T07:27:00.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f5234cc2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523006cd80 con 0x7f52300b2da0 2024-03-06T07:27:00.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f522e59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5230161670 0x7f5230161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f522e59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5230161670 0x7f5230161e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49202/0 (socket says 172.21.15.59:49202) 2024-03-06T07:27:00.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f522e59c640 1 -- 172.21.15.59:0/1872815310 learned_addr learned my addr 172.21.15.59:0/1872815310 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:00.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f522e59c640 1 -- 172.21.15.59:0/1872815310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52300b2da0 msgr2=0x7f52301623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f5227fff640 1 --2- 172.21.15.59:0/1872815310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52300b2da0 0x7f52301623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f522e59c640 1 --2- 172.21.15.59:0/1872815310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52300b2da0 0x7f52301623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f522e59c640 1 -- 172.21.15.59:0/1872815310 >> v1:172.21.15.164:6789/0 conn(0x7f52300cc680 legacy=0x7f52300ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.701+0000 7f522e59c640 1 -- 172.21.15.59:0/1872815310 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523006d060 con 0x7f5230161670 2024-03-06T07:27:00.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.702+0000 7f522e59c640 1 --2- 172.21.15.59:0/1872815310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5230161670 0x7f5230161e60 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f5220009870 tx=0x7f522002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc5efd638f2dc38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.702+0000 7f522dd9b640 1 -- 172.21.15.59:0/1872815310 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220035020 con 0x7f5230161670 2024-03-06T07:27:00.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.702+0000 7f522dd9b640 1 -- 172.21.15.59:0/1872815310 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5220031640 con 0x7f5230161670 2024-03-06T07:27:00.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.702+0000 7f522dd9b640 1 -- 172.21.15.59:0/1872815310 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522002d040 con 0x7f5230161670 2024-03-06T07:27:00.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.703+0000 7f5234cc2640 1 -- 172.21.15.59:0/1872815310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5230161670 msgr2=0x7f5230161e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.703+0000 7f5234cc2640 1 --2- 172.21.15.59:0/1872815310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5230161670 0x7f5230161e60 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f5220009870 tx=0x7f522002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.703+0000 7f5234cc2640 1 -- 172.21.15.59:0/1872815310 shutdown_connections 2024-03-06T07:27:00.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.703+0000 7f5234cc2640 1 --2- 172.21.15.59:0/1872815310 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52300b2da0 0x7f52301623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.703+0000 7f5234cc2640 1 --2- 172.21.15.59:0/1872815310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5230161670 0x7f5230161e60 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.703+0000 7f5234cc2640 1 -- 172.21.15.59:0/1872815310 >> 172.21.15.59:0/1872815310 conn(0x7f52300d1d50 msgr2=0x7f52300d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.703+0000 7f5234cc2640 1 -- 172.21.15.59:0/1872815310 shutdown_connections 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.703+0000 7f5234cc2640 1 -- 172.21.15.59:0/1872815310 wait complete. 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.704+0000 7f5234cc2640 1 Processor -- start 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.704+0000 7f5234cc2640 1 -- start start 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f5234cc2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52300b2da0 0x7f5230156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f5234cc2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52300cc680 0x7f52301f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f5234cc2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5230161670 0x7f52301f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f5234cc2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523006d340 con 0x7f52300b2da0 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f5234cc2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52301628e0 con 0x7f52300cc680 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f5234cc2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523006d4c0 con 0x7f5230161670 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f522e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52300b2da0 0x7f5230156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f522e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52300b2da0 0x7f5230156870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49216/0 (socket says 172.21.15.59:49216) 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f522e59c640 1 -- 172.21.15.59:0/3922122956 learned_addr learned my addr 172.21.15.59:0/3922122956 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f5227fff640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52300cc680 0x7f52301f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f522ed9d640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5230161670 0x7f52301f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f522e59c640 1 -- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5230161670 msgr2=0x7f52301f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f522e59c640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5230161670 0x7f52301f41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f522e59c640 1 -- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52300cc680 msgr2=0x7f52301f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f522e59c640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52300cc680 0x7f52301f3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f522e59c640 1 -- 172.21.15.59:0/3922122956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52300b6b20 con 0x7f52300b2da0 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f522e59c640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52300b2da0 0x7f5230156870 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f522002b930 tx=0x7f52200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f5226ffd640 1 -- 172.21.15.59:0/3922122956 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522003ac60 con 0x7f52300b2da0 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f5226ffd640 1 -- 172.21.15.59:0/3922122956 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522003ae20 con 0x7f52300b2da0 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.705+0000 7f5226ffd640 1 -- 172.21.15.59:0/3922122956 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220044630 con 0x7f52300b2da0 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.706+0000 7f5234cc2640 1 -- 172.21.15.59:0/3922122956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52300b6e00 con 0x7f52300b2da0 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.706+0000 7f5234cc2640 1 -- 172.21.15.59:0/3922122956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52300b73c0 con 0x7f52300b2da0 2024-03-06T07:27:00.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.706+0000 7f5234cc2640 1 -- 172.21.15.59:0/3922122956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5230068050 con 0x7f52300b2da0 2024-03-06T07:27:00.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.707+0000 7f5226ffd640 1 -- 172.21.15.59:0/3922122956 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f522004b060 con 0x7f52300b2da0 2024-03-06T07:27:00.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.707+0000 7f5226ffd640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f521006b4d0 0x7f521006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.708+0000 7f5226ffd640 1 -- 172.21.15.59:0/3922122956 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f52200abe20 con 0x7f52300b2da0 2024-03-06T07:27:00.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.708+0000 7f5227fff640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f521006b4d0 0x7f521006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.708+0000 7f5227fff640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f521006b4d0 0x7f521006d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f523006b520 tx=0x7f5218008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.712+0000 7f5226ffd640 1 -- 172.21.15.59:0/3922122956 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f522003f070 con 0x7f52300b2da0 2024-03-06T07:27:00.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.930+0000 7f5234cc2640 1 -- 172.21.15.59:0/3922122956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52300b7cd0 con 0x7f52300b2da0 2024-03-06T07:27:00.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.931+0000 7f5226ffd640 1 -- 172.21.15.59:0/3922122956 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f522007a5e0 con 0x7f52300b2da0 2024-03-06T07:27:00.934 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:00.934 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":59,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:17.219713+0000","modified":"2024-03-06T07:26:34.073810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7635},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7635":{"gid":7635,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6835/4004944688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":4004944688},{"type":"v1","addr":"172.21.15.59:6835","nonce":4004944688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7635,"qdb_cluster":[7635]},"id":7}]} 2024-03-06T07:27:00.936 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 59 2024-03-06T07:27:00.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 -- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f521006b4d0 msgr2=0x7f521006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f521006b4d0 0x7f521006d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f523006b520 tx=0x7f5218008040 comp rx=0 tx=0).stop 2024-03-06T07:27:00.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 -- 172.21.15.59:0/3922122956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52300b2da0 msgr2=0x7f5230156870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52300b2da0 0x7f5230156870 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f522002b930 tx=0x7f52200046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 -- 172.21.15.59:0/3922122956 shutdown_connections 2024-03-06T07:27:00.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f521006b4d0 0x7f521006d990 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5230161670 0x7f52301f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52300cc680 0x7f52301f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 --2- 172.21.15.59:0/3922122956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52300b2da0 0x7f5230156870 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 -- 172.21.15.59:0/3922122956 >> 172.21.15.59:0/3922122956 conn(0x7f52300d1d50 msgr2=0x7f52300d2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 -- 172.21.15.59:0/3922122956 shutdown_connections 2024-03-06T07:27:00.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:00.933+0000 7f5234cc2640 1 -- 172.21.15.59:0/3922122956 wait complete. 2024-03-06T07:27:00.947 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:01.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.046+0000 7fc381b81640 1 Processor -- start 2024-03-06T07:27:01.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.046+0000 7fc381b81640 1 -- start start 2024-03-06T07:27:01.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.047+0000 7fc381b81640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc37c15e150 0x7fc37c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.047+0000 7fc381b81640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc37c0cfba0 0x7fc37c0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.047+0000 7fc381b81640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37c0d0e90 0x7fc37c0b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.047+0000 7fc381b81640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37c0d2ab0 con 0x7fc37c15e150 2024-03-06T07:27:01.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.047+0000 7fc381b81640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37c0d1ea0 con 0x7fc37c0cfba0 2024-03-06T07:27:01.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.047+0000 7fc381b81640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c0d2180 con 0x7fc37c0d0e90 2024-03-06T07:27:01.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.048+0000 7fc37b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc37c15e150 0x7fc37c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.048+0000 7fc37b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc37c15e150 0x7fc37c0cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49232/0 (socket says 172.21.15.59:49232) 2024-03-06T07:27:01.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.048+0000 7fc37b7fe640 1 -- 172.21.15.59:0/739507820 learned_addr learned my addr 172.21.15.59:0/739507820 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:01.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.048+0000 7fc37affd640 1 --2- 172.21.15.59:0/739507820 >> v2:172.21.15.164:3300/0 conn(0x7fc37c0cfba0 0x7fc37c0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.048+0000 7fc37bfff640 1 --2- 172.21.15.59:0/739507820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37c0d0e90 0x7fc37c0b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.048+0000 7fc37b7fe640 1 -- 172.21.15.59:0/739507820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37c0d0e90 msgr2=0x7fc37c0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.048+0000 7fc37b7fe640 1 --2- 172.21.15.59:0/739507820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37c0d0e90 0x7fc37c0b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.048+0000 7fc37b7fe640 1 -- 172.21.15.59:0/739507820 >> v2:172.21.15.164:3300/0 conn(0x7fc37c0cfba0 msgr2=0x7fc37c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.048+0000 7fc37b7fe640 1 --2- 172.21.15.59:0/739507820 >> v2:172.21.15.164:3300/0 conn(0x7fc37c0cfba0 0x7fc37c0d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.048+0000 7fc37b7fe640 1 -- 172.21.15.59:0/739507820 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c0d2460 con 0x7fc37c15e150 2024-03-06T07:27:01.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.049+0000 7fc37b7fe640 1 --2- 172.21.15.59:0/739507820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc37c15e150 0x7fc37c0cf660 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0b6b50 tx=0x7fc36c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99d9d878f2f4f60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.049+0000 7fc37a7fc640 1 -- 172.21.15.59:0/739507820 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c02e070 con 0x7fc37c15e150 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.049+0000 7fc37a7fc640 1 -- 172.21.15.59:0/739507820 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c037070 con 0x7fc37c15e150 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.049+0000 7fc37a7fc640 1 -- 172.21.15.59:0/739507820 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c031a00 con 0x7fc37c15e150 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.049+0000 7fc381b81640 1 -- 172.21.15.59:0/739507820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc37c15e150 msgr2=0x7fc37c0cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.049+0000 7fc381b81640 1 --2- 172.21.15.59:0/739507820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc37c15e150 0x7fc37c0cf660 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0b6b50 tx=0x7fc36c02b880 comp rx=0 tx=0).stop 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.049+0000 7fc381b81640 1 -- 172.21.15.59:0/739507820 shutdown_connections 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.049+0000 7fc381b81640 1 --2- 172.21.15.59:0/739507820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37c0d0e90 0x7fc37c0b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.049+0000 7fc381b81640 1 --2- 172.21.15.59:0/739507820 >> v2:172.21.15.164:3300/0 conn(0x7fc37c0cfba0 0x7fc37c0d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.049+0000 7fc381b81640 1 --2- 172.21.15.59:0/739507820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc37c15e150 0x7fc37c0cf660 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.049+0000 7fc381b81640 1 -- 172.21.15.59:0/739507820 >> 172.21.15.59:0/739507820 conn(0x7fc37c09d230 msgr2=0x7fc37c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.050+0000 7fc381b81640 1 -- 172.21.15.59:0/739507820 shutdown_connections 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.050+0000 7fc381b81640 1 -- 172.21.15.59:0/739507820 wait complete. 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.051+0000 7fc381b81640 1 Processor -- start 2024-03-06T07:27:01.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.051+0000 7fc381b81640 1 -- start start 2024-03-06T07:27:01.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.051+0000 7fc381b81640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc37c0cfba0 0x7fc37c15d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.051+0000 7fc381b81640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37c0d0e90 0x7fc37c09ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc381b81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc37c15e150 0x7fc37c0cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc381b81640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c0d1b30 con 0x7fc37c15e150 2024-03-06T07:27:01.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc381b81640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c0d2f40 con 0x7fc37c0cfba0 2024-03-06T07:27:01.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc381b81640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37c15e530 con 0x7fc37c0d0e90 2024-03-06T07:27:01.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc37bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc37c15e150 0x7fc37c0cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc37bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc37c15e150 0x7fc37c0cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49240/0 (socket says 172.21.15.59:49240) 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc37bfff640 1 -- 172.21.15.59:0/660586117 learned_addr learned my addr 172.21.15.59:0/660586117 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc37b7fe640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc37c0cfba0 0x7fc37c15d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc37affd640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37c0d0e90 0x7fc37c09ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc37bfff640 1 -- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37c0d0e90 msgr2=0x7fc37c09ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc37bfff640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37c0d0e90 0x7fc37c09ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc37bfff640 1 -- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc37c0cfba0 msgr2=0x7fc37c15d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc37bfff640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc37c0cfba0 0x7fc37c15d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc37bfff640 1 -- 172.21.15.59:0/660586117 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c0a0a90 con 0x7fc37c15e150 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc37bfff640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc37c15e150 0x7fc37c0cc1b0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fc37400efc0 tx=0x7fc37400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc373fff640 1 -- 172.21.15.59:0/660586117 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc374018020 con 0x7fc37c15e150 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc373fff640 1 -- 172.21.15.59:0/660586117 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc374014480 con 0x7fc37c15e150 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc381b81640 1 -- 172.21.15.59:0/660586117 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37c0a0d70 con 0x7fc37c15e150 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc373fff640 1 -- 172.21.15.59:0/660586117 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc374014870 con 0x7fc37c15e150 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.052+0000 7fc381b81640 1 -- 172.21.15.59:0/660586117 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37c0a1330 con 0x7fc37c15e150 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.053+0000 7fc381b81640 1 -- 172.21.15.59:0/660586117 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37c069ee0 con 0x7fc37c15e150 2024-03-06T07:27:01.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.054+0000 7fc373fff640 1 -- 172.21.15.59:0/660586117 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc374005d10 con 0x7fc37c15e150 2024-03-06T07:27:01.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.054+0000 7fc373fff640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc35c06b4d0 0x7fc35c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.054+0000 7fc373fff640 1 -- 172.21.15.59:0/660586117 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc37408aee0 con 0x7fc37c15e150 2024-03-06T07:27:01.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.054+0000 7fc37b7fe640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc35c06b4d0 0x7fc35c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.055+0000 7fc37b7fe640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc35c06b4d0 0x7fc35c06d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc36c036000 tx=0x7fc36c03d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.058+0000 7fc373fff640 1 -- 172.21.15.59:0/660586117 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc374022030 con 0x7fc37c15e150 2024-03-06T07:27:01.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.241+0000 7fc381b81640 1 -- 172.21.15.59:0/660586117 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc37c0d4670 con 0x7fc37c15e150 2024-03-06T07:27:01.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.242+0000 7fc373fff640 1 -- 172.21.15.59:0/660586117 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16987 (secure 0 0 0) 0x7fc374010040 con 0x7fc37c15e150 2024-03-06T07:27:01.244 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:01.245 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":96,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:26:29.601683+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:15.489280+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:16.183614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6834/2799249020":"2024-03-07T07:26:29.597825+0000","172.21.15.59:6835/1784120157":"2024-03-07T07:26:29.585744+0000","172.21.15.59:6832/2799249020":"2024-03-07T07:26:29.597825+0000","172.21.15.59:6833/1784120157":"2024-03-07T07:26:29.585744+0000","172.21.15.164:6837/4292771408":"2024-03-07T07:26:29.578123+0000","172.21.15.164:6835/4292771408":"2024-03-07T07:26:29.578123+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:01.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.244+0000 7fc381b81640 1 -- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc35c06b4d0 msgr2=0x7fc35c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.244+0000 7fc381b81640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc35c06b4d0 0x7fc35c06d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc36c036000 tx=0x7fc36c03d040 comp rx=0 tx=0).stop 2024-03-06T07:27:01.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.244+0000 7fc381b81640 1 -- 172.21.15.59:0/660586117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc37c15e150 msgr2=0x7fc37c0cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.244+0000 7fc381b81640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc37c15e150 0x7fc37c0cc1b0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fc37400efc0 tx=0x7fc37400c490 comp rx=0 tx=0).stop 2024-03-06T07:27:01.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.244+0000 7fc381b81640 1 -- 172.21.15.59:0/660586117 shutdown_connections 2024-03-06T07:27:01.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.244+0000 7fc381b81640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc35c06b4d0 0x7fc35c06d990 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.244+0000 7fc381b81640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc37c15e150 0x7fc37c0cc1b0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.244+0000 7fc381b81640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37c0d0e90 0x7fc37c09ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.244+0000 7fc381b81640 1 --2- 172.21.15.59:0/660586117 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc37c0cfba0 0x7fc37c15d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.244+0000 7fc381b81640 1 -- 172.21.15.59:0/660586117 >> 172.21.15.59:0/660586117 conn(0x7fc37c09d230 msgr2=0x7fc37c06d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.245+0000 7fc381b81640 1 -- 172.21.15.59:0/660586117 shutdown_connections 2024-03-06T07:27:01.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.245+0000 7fc381b81640 1 -- 172.21.15.59:0/660586117 wait complete. 2024-03-06T07:27:01.258 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:27:01.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.362+0000 7fa2571d7640 1 Processor -- start 2024-03-06T07:27:01.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.362+0000 7fa2571d7640 1 -- start start 2024-03-06T07:27:01.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.363+0000 7fa2571d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa250145700 0x7fa2500cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.363+0000 7fa2571d7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa2500cf880 0x7fa25006af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.363+0000 7fa2571d7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2500cfec0 con 0x7fa25006b4c0 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.363+0000 7fa2571d7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25006b930 con 0x7fa2500cf880 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.363+0000 7fa2571d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2500d01a0 con 0x7fa250145700 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.363+0000 7fa255774640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa25006b4c0 0x7fa250071c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36066/0 (socket says 172.21.15.59:36066) 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.363+0000 7fa255774640 1 -- 172.21.15.59:0/3013911633 learned_addr learned my addr 172.21.15.59:0/3013911633 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.363+0000 7fa247fff640 1 --2- 172.21.15.59:0/3013911633 >> v2:172.21.15.164:3300/0 conn(0x7fa2500cf880 0x7fa25006af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.363+0000 7fa254f73640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa250145700 0x7fa2500cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.363+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676752184 0 0) 0x7fa2500cfec0 con 0x7fa25006b4c0 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.363+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa238003520 con 0x7fa25006b4c0 2024-03-06T07:27:01.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2573288246 0 0) 0x7fa238003520 con 0x7fa25006b4c0 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa250145700 msgr2=0x7fa2500cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa250145700 0x7fa2500cf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 >> v2:172.21.15.164:3300/0 conn(0x7fa2500cf880 msgr2=0x7fa25006af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 --2- 172.21.15.59:0/3013911633 >> v2:172.21.15.164:3300/0 conn(0x7fa2500cf880 0x7fa25006af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2500d0480 con 0x7fa25006b4c0 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fa2480019f0 con 0x7fa25006b4c0 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 >> v1:172.21.15.59:6789/0 conn(0x7fa25006b4c0 legacy=0x7fa250071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2380039b0 0x7fa238003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa238004800 0x7fa238004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2380051c0 0x7fa23800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25006b930 con 0x7fa2380051c0 2024-03-06T07:27:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2500d01a0 con 0x7fa2380039b0 2024-03-06T07:27:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23800bf00 con 0x7fa238004800 2024-03-06T07:27:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa254f73640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa250145700 0x7fa2500cf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa247fff640 1 --2- 172.21.15.59:0/3013911633 >> v2:172.21.15.164:3300/0 conn(0x7fa2500cf880 0x7fa25006af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa255774640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2380051c0 0x7fa23800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa254f73640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2380039b0 0x7fa238003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa247fff640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa238004800 0x7fa238004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa255774640 1 -- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa238004800 msgr2=0x7fa238004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa255774640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa238004800 0x7fa238004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa255774640 1 -- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2380039b0 msgr2=0x7fa238003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa255774640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2380039b0 0x7fa238003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.364+0000 7fa255774640 1 -- 172.21.15.59:0/3013911633 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa238003520 con 0x7fa2380051c0 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.365+0000 7fa255774640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2380051c0 0x7fa23800b9c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fa24800a8b0 tx=0x7fa238005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd4c78bed9eb0ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.365+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24800a4a0 con 0x7fa2380051c0 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.365+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa248034650 con 0x7fa2380051c0 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.365+0000 7fa2477fe640 1 -- 172.21.15.59:0/3013911633 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248034a40 con 0x7fa2380051c0 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.366+0000 7fa2571d7640 1 -- 172.21.15.59:0/3013911633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2380051c0 msgr2=0x7fa23800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.366+0000 7fa2571d7640 1 --2- 172.21.15.59:0/3013911633 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2380051c0 0x7fa23800b9c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fa24800a8b0 tx=0x7fa238005830 comp rx=0 tx=0).stop 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.366+0000 7fa255774640 1 -- 172.21.15.59:0/3013911633 reap_dead start 2024-03-06T07:27:01.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.366+0000 7fa2571d7640 1 -- 172.21.15.59:0/3013911633 shutdown_connections 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.366+0000 7fa2571d7640 1 -- 172.21.15.59:0/3013911633 >> 172.21.15.59:0/3013911633 conn(0x7fa2500b2890 msgr2=0x7fa2500b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.366+0000 7fa2571d7640 1 -- 172.21.15.59:0/3013911633 shutdown_connections 2024-03-06T07:27:01.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.366+0000 7fa2571d7640 1 -- 172.21.15.59:0/3013911633 wait complete. 2024-03-06T07:27:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.368+0000 7fa2571d7640 1 Processor -- start 2024-03-06T07:27:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.368+0000 7fa2571d7640 1 -- start start 2024-03-06T07:27:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.369+0000 7fa2571d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2380051c0 0x7fa25015b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.369+0000 7fa2571d7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2500aaf70 0x7fa2500ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.369+0000 7fa2571d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500ae7b0 0x7fa2500aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.369+0000 7fa2571d7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25015fc40 con 0x7fa2500ae7b0 2024-03-06T07:27:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.369+0000 7fa2571d7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa250145ae0 con 0x7fa2500aaf70 2024-03-06T07:27:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.369+0000 7fa2571d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2500b5280 con 0x7fa2380051c0 2024-03-06T07:27:01.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa255774640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500ae7b0 0x7fa2500aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa255774640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500ae7b0 0x7fa2500aec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49258/0 (socket says 172.21.15.59:49258) 2024-03-06T07:27:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa255774640 1 -- 172.21.15.59:0/972938680 learned_addr learned my addr 172.21.15.59:0/972938680 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa254f73640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2380051c0 0x7fa25015b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa247fff640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2500aaf70 0x7fa2500ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa255774640 1 -- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2380051c0 msgr2=0x7fa25015b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa255774640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2380051c0 0x7fa25015b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa255774640 1 -- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2500aaf70 msgr2=0x7fa2500ab3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa255774640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2500aaf70 0x7fa2500ab3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa255774640 1 -- 172.21.15.59:0/972938680 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2500a9ed0 con 0x7fa2500ae7b0 2024-03-06T07:27:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa255774640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500ae7b0 0x7fa2500aec30 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fa24800a7d0 tx=0x7fa2500d0090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa2457fa640 1 -- 172.21.15.59:0/972938680 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248001ff0 con 0x7fa2500ae7b0 2024-03-06T07:27:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa2457fa640 1 -- 172.21.15.59:0/972938680 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa248044070 con 0x7fa2500ae7b0 2024-03-06T07:27:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa2571d7640 1 -- 172.21.15.59:0/972938680 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2500aa1b0 con 0x7fa2500ae7b0 2024-03-06T07:27:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa2457fa640 1 -- 172.21.15.59:0/972938680 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248034c00 con 0x7fa2500ae7b0 2024-03-06T07:27:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.370+0000 7fa2571d7640 1 -- 172.21.15.59:0/972938680 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2500aa770 con 0x7fa2500ae7b0 2024-03-06T07:27:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.372+0000 7fa2571d7640 1 -- 172.21.15.59:0/972938680 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa250068050 con 0x7fa2500ae7b0 2024-03-06T07:27:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.372+0000 7fa2457fa640 1 -- 172.21.15.59:0/972938680 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa248049460 con 0x7fa2500ae7b0 2024-03-06T07:27:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.372+0000 7fa2457fa640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b500 0x7fa23c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.373+0000 7fa2457fa640 1 -- 172.21.15.59:0/972938680 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa2480ac700 con 0x7fa2500ae7b0 2024-03-06T07:27:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.373+0000 7fa254f73640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b500 0x7fa23c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.373+0000 7fa254f73640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b500 0x7fa23c06d9c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa250067d40 tx=0x7fa24c009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.377+0000 7fa2457fa640 1 -- 172.21.15.59:0/972938680 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa24807b580 con 0x7fa2500ae7b0 2024-03-06T07:27:01.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.596+0000 7fa2571d7640 1 -- 172.21.15.59:0/972938680 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa2500b3c90 con 0x7fa2500ae7b0 2024-03-06T07:27:01.598 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:27:01.596+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:01.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.597+0000 7fa2457fa640 1 -- 172.21.15.59:0/972938680 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v60) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa24807ae80 con 0x7fa2500ae7b0 2024-03-06T07:27:01.599 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:27:01.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 -- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b500 msgr2=0x7fa23c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b500 0x7fa23c06d9c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa250067d40 tx=0x7fa24c009190 comp rx=0 tx=0).stop 2024-03-06T07:27:01.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 -- 172.21.15.59:0/972938680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500ae7b0 msgr2=0x7fa2500aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500ae7b0 0x7fa2500aec30 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fa24800a7d0 tx=0x7fa2500d0090 comp rx=0 tx=0).stop 2024-03-06T07:27:01.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 -- 172.21.15.59:0/972938680 shutdown_connections 2024-03-06T07:27:01.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa23c06b500 0x7fa23c06d9c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2500ae7b0 0x7fa2500aec30 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2500aaf70 0x7fa2500ab3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 --2- 172.21.15.59:0/972938680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2380051c0 0x7fa25015b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 -- 172.21.15.59:0/972938680 >> 172.21.15.59:0/972938680 conn(0x7fa2500b2890 msgr2=0x7fa250071c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 -- 172.21.15.59:0/972938680 shutdown_connections 2024-03-06T07:27:01.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.600+0000 7fa2571d7640 1 -- 172.21.15.59:0/972938680 wait complete. 2024-03-06T07:27:01.616 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:27:01.650 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:27:01.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.719+0000 7f95114df640 1 Processor -- start 2024-03-06T07:27:01.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.719+0000 7f95114df640 1 -- start start 2024-03-06T07:27:01.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f95114df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f950c06d990 0x7f950c0b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f95114df640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f950c0b4040 0x7f950c15ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f95114df640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f950c0b4830 con 0x7f950c1456e0 2024-03-06T07:27:01.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f95114df640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950c0b4b10 con 0x7f950c0b4040 2024-03-06T07:27:01.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f95114df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c15c640 con 0x7f950c06d990 2024-03-06T07:27:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f950b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f950c0b4040 0x7f950c15ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f950affd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f950c1456e0 0x7f950c06d280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36078/0 (socket says 172.21.15.59:36078) 2024-03-06T07:27:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f950affd640 1 -- 172.21.15.59:0/155286300 learned_addr learned my addr 172.21.15.59:0/155286300 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f950b7fe640 1 -- 172.21.15.59:0/155286300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f950c06d990 msgr2=0x7f950c0b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f950b7fe640 1 --2- 172.21.15.59:0/155286300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f950c06d990 0x7f950c0b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f950b7fe640 1 -- 172.21.15.59:0/155286300 >> v1:172.21.15.59:6789/0 conn(0x7f950c1456e0 legacy=0x7f950c06d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f950b7fe640 1 -- 172.21.15.59:0/155286300 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c15c920 con 0x7f950c0b4040 2024-03-06T07:27:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.720+0000 7f950a7fc640 1 -- 172.21.15.59:0/155286300 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950417883 0 0) 0x7f950c0b4830 con 0x7f950c1456e0 2024-03-06T07:27:01.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.721+0000 7f950b7fe640 1 --2- 172.21.15.59:0/155286300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f950c0b4040 0x7f950c15ce40 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f9504009d90 tx=0x7f950402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20afdb65f01b00aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.721+0000 7f950a7fc640 1 -- 172.21.15.59:0/155286300 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504039070 con 0x7f950c0b4040 2024-03-06T07:27:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.721+0000 7f950a7fc640 1 -- 172.21.15.59:0/155286300 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950402cbc0 con 0x7f950c0b4040 2024-03-06T07:27:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.721+0000 7f950a7fc640 1 -- 172.21.15.59:0/155286300 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504034440 con 0x7f950c0b4040 2024-03-06T07:27:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.722+0000 7f95114df640 1 -- 172.21.15.59:0/155286300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f950c0b4040 msgr2=0x7f950c15ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.722+0000 7f95114df640 1 --2- 172.21.15.59:0/155286300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f950c0b4040 0x7f950c15ce40 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f9504009d90 tx=0x7f950402be10 comp rx=0 tx=0).stop 2024-03-06T07:27:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.722+0000 7f95114df640 1 -- 172.21.15.59:0/155286300 shutdown_connections 2024-03-06T07:27:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.722+0000 7f95114df640 1 --2- 172.21.15.59:0/155286300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f950c0b4040 0x7f950c15ce40 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.722+0000 7f95114df640 1 --2- 172.21.15.59:0/155286300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f950c06d990 0x7f950c0b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.722+0000 7f95114df640 1 -- 172.21.15.59:0/155286300 >> 172.21.15.59:0/155286300 conn(0x7f950c06b160 msgr2=0x7f950c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.722+0000 7f95114df640 1 -- 172.21.15.59:0/155286300 shutdown_connections 2024-03-06T07:27:01.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.722+0000 7f95114df640 1 -- 172.21.15.59:0/155286300 wait complete. 2024-03-06T07:27:01.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.723+0000 7f95114df640 1 Processor -- start 2024-03-06T07:27:01.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.723+0000 7f95114df640 1 -- start start 2024-03-06T07:27:01.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f95114df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f950c06d990 0x7f950c15b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f95114df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f950c0b4040 0x7f950c1f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f95114df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f950c1456e0 0x7f950c0ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f95114df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0b44b0 con 0x7f950c1456e0 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f95114df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c0b4df0 con 0x7f950c0b4040 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f95114df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950c15d380 con 0x7f950c06d990 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f950b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f950c1456e0 0x7f950c0ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f950b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f950c1456e0 0x7f950c0ccb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49286/0 (socket says 172.21.15.59:49286) 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f950b7fe640 1 -- 172.21.15.59:0/499856769 learned_addr learned my addr 172.21.15.59:0/499856769 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f950affd640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f950c06d990 0x7f950c15b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f9503fff640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f950c0b4040 0x7f950c1f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f950b7fe640 1 -- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f950c06d990 msgr2=0x7f950c15b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f950b7fe640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f950c06d990 0x7f950c15b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f950b7fe640 1 -- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f950c0b4040 msgr2=0x7f950c1f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f950b7fe640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f950c0b4040 0x7f950c1f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f950b7fe640 1 -- 172.21.15.59:0/499856769 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c1f1a90 con 0x7f950c1456e0 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f950b7fe640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f950c1456e0 0x7f950c0ccb80 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f9504009d90 tx=0x7f9504004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f95037fe640 1 -- 172.21.15.59:0/499856769 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504044020 con 0x7f950c1456e0 2024-03-06T07:27:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f95037fe640 1 -- 172.21.15.59:0/499856769 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950403f470 con 0x7f950c1456e0 2024-03-06T07:27:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f95114df640 1 -- 172.21.15.59:0/499856769 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c1f5f80 con 0x7f950c1456e0 2024-03-06T07:27:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f95037fe640 1 -- 172.21.15.59:0/499856769 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504039040 con 0x7f950c1456e0 2024-03-06T07:27:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.724+0000 7f95114df640 1 -- 172.21.15.59:0/499856769 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c1f1ff0 con 0x7f950c1456e0 2024-03-06T07:27:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.725+0000 7f95114df640 1 -- 172.21.15.59:0/499856769 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c069ee0 con 0x7f950c1456e0 2024-03-06T07:27:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.726+0000 7f95037fe640 1 -- 172.21.15.59:0/499856769 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9504005d10 con 0x7f950c1456e0 2024-03-06T07:27:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.726+0000 7f95037fe640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f94ec06b4d0 0x7f94ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.726+0000 7f95037fe640 1 -- 172.21.15.59:0/499856769 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f95040acde0 con 0x7f950c1456e0 2024-03-06T07:27:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.727+0000 7f950affd640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f94ec06b4d0 0x7f94ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.727+0000 7f950affd640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f94ec06b4d0 0x7f94ec06d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f94fc007fd0 tx=0x7f94fc007de0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.731+0000 7f95037fe640 1 -- 172.21.15.59:0/499856769 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f950404e070 con 0x7f950c1456e0 2024-03-06T07:27:01.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:01.949+0000 7f95114df640 1 -- 172.21.15.59:0/499856769 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f950c1444b0 con 0x7f950c1456e0 2024-03-06T07:27:02.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.599+0000 7f95037fe640 1 -- 172.21.15.59:0/499856769 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v61) v1 ==== 104+0+0 (secure 0 0 0) 0x7f950c1f1a90 con 0x7f950c1456e0 2024-03-06T07:27:02.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 -- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f94ec06b4d0 msgr2=0x7f94ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f94ec06b4d0 0x7f94ec06d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f94fc007fd0 tx=0x7f94fc007de0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 -- 172.21.15.59:0/499856769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f950c1456e0 msgr2=0x7f950c0ccb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f950c1456e0 0x7f950c0ccb80 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f9504009d90 tx=0x7f9504004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:02.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 -- 172.21.15.59:0/499856769 shutdown_connections 2024-03-06T07:27:02.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f94ec06b4d0 0x7f94ec06d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f950c1456e0 0x7f950c0ccb80 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f950c0b4040 0x7f950c1f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 --2- 172.21.15.59:0/499856769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f950c06d990 0x7f950c15b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 -- 172.21.15.59:0/499856769 >> 172.21.15.59:0/499856769 conn(0x7f950c06b160 msgr2=0x7f950c144a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 -- 172.21.15.59:0/499856769 shutdown_connections 2024-03-06T07:27:02.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.602+0000 7f95114df640 1 -- 172.21.15.59:0/499856769 wait complete. 2024-03-06T07:27:02.617 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:27:02.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.768+0000 7fdded95c640 1 Processor -- start 2024-03-06T07:27:02.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.768+0000 7fdded95c640 1 -- start start 2024-03-06T07:27:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdded95c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdde815f5e0 0x7fdde80b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdded95c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdde80cec60 0x7fdde80cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdded95c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde80cd4d0 con 0x7fdde80b16b0 2024-03-06T07:27:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdded95c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde80cf920 con 0x7fdde815f5e0 2024-03-06T07:27:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdded95c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde80cd7b0 con 0x7fdde80cec60 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdde6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdde815f5e0 0x7fdde80b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdde6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdde815f5e0 0x7fdde80b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49292/0 (socket says 172.21.15.59:49292) 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdde6ffd640 1 -- 172.21.15.59:0/3053758850 learned_addr learned my addr 172.21.15.59:0/3053758850 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdde77fe640 1 --2- 172.21.15.59:0/3053758850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdde80cec60 0x7fdde80cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdde6ffd640 1 -- 172.21.15.59:0/3053758850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdde80cec60 msgr2=0x7fdde80cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdde6ffd640 1 --2- 172.21.15.59:0/3053758850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdde80cec60 0x7fdde80cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.769+0000 7fdde6ffd640 1 -- 172.21.15.59:0/3053758850 >> v1:172.21.15.164:6789/0 conn(0x7fdde80b16b0 legacy=0x7fdde80ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.770+0000 7fdde6ffd640 1 -- 172.21.15.59:0/3053758850 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde80cda90 con 0x7fdde815f5e0 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.770+0000 7fdde6ffd640 1 --2- 172.21.15.59:0/3053758850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdde815f5e0 0x7fdde80b2bf0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009870 tx=0x7fdde002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b409bdbf075ebc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.770+0000 7fdde5ffb640 1 -- 172.21.15.59:0/3053758850 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0035020 con 0x7fdde815f5e0 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.770+0000 7fdde5ffb640 1 -- 172.21.15.59:0/3053758850 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde0031640 con 0x7fdde815f5e0 2024-03-06T07:27:02.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.770+0000 7fdde5ffb640 1 -- 172.21.15.59:0/3053758850 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde002d040 con 0x7fdde815f5e0 2024-03-06T07:27:02.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.771+0000 7fdded95c640 1 -- 172.21.15.59:0/3053758850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdde815f5e0 msgr2=0x7fdde80b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.771+0000 7fdded95c640 1 --2- 172.21.15.59:0/3053758850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdde815f5e0 0x7fdde80b2bf0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009870 tx=0x7fdde002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.771+0000 7fdded95c640 1 -- 172.21.15.59:0/3053758850 shutdown_connections 2024-03-06T07:27:02.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.771+0000 7fdded95c640 1 --2- 172.21.15.59:0/3053758850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdde80cec60 0x7fdde80cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.771+0000 7fdded95c640 1 --2- 172.21.15.59:0/3053758850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdde815f5e0 0x7fdde80b2bf0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.771+0000 7fdded95c640 1 -- 172.21.15.59:0/3053758850 >> 172.21.15.59:0/3053758850 conn(0x7fdde80d1d50 msgr2=0x7fdde80d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.771+0000 7fdded95c640 1 -- 172.21.15.59:0/3053758850 shutdown_connections 2024-03-06T07:27:02.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.771+0000 7fdded95c640 1 -- 172.21.15.59:0/3053758850 wait complete. 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.772+0000 7fdded95c640 1 Processor -- start 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.772+0000 7fdded95c640 1 -- start start 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdded95c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdde80b16b0 0x7fdde80c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdded95c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdde80cec60 0x7fdde8157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdded95c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdde815f5e0 0x7fdde80c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdded95c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde80cf270 con 0x7fdde815f5e0 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdded95c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde80cf0f0 con 0x7fdde80cec60 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdded95c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde80d2870 con 0x7fdde80b16b0 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdde77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdde815f5e0 0x7fdde80c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdde77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdde815f5e0 0x7fdde80c8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49296/0 (socket says 172.21.15.59:49296) 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdde77fe640 1 -- 172.21.15.59:0/4101243834 learned_addr learned my addr 172.21.15.59:0/4101243834 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdde6ffd640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdde80b16b0 0x7fdde80c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdde67fc640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdde80cec60 0x7fdde8157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdde77fe640 1 -- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdde80b16b0 msgr2=0x7fdde80c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdde77fe640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdde80b16b0 0x7fdde80c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdde77fe640 1 -- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdde80cec60 msgr2=0x7fdde8157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdde77fe640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdde80cec60 0x7fdde8157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.773+0000 7fdde77fe640 1 -- 172.21.15.59:0/4101243834 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde8158890 con 0x7fdde815f5e0 2024-03-06T07:27:02.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.774+0000 7fdde77fe640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdde815f5e0 0x7fdde80c8e20 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fddd8007de0 tx=0x7fddd800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.774+0000 7fdddffff640 1 -- 172.21.15.59:0/4101243834 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd8014620 con 0x7fdde815f5e0 2024-03-06T07:27:02.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.774+0000 7fdddffff640 1 -- 172.21.15.59:0/4101243834 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd8019070 con 0x7fdde815f5e0 2024-03-06T07:27:02.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.774+0000 7fdded95c640 1 -- 172.21.15.59:0/4101243834 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde8158340 con 0x7fdde815f5e0 2024-03-06T07:27:02.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.774+0000 7fdddffff640 1 -- 172.21.15.59:0/4101243834 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd8014a20 con 0x7fdde815f5e0 2024-03-06T07:27:02.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.774+0000 7fdded95c640 1 -- 172.21.15.59:0/4101243834 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde80c7620 con 0x7fdde815f5e0 2024-03-06T07:27:02.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.775+0000 7fdded95c640 1 -- 172.21.15.59:0/4101243834 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde8068050 con 0x7fdde815f5e0 2024-03-06T07:27:02.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.775+0000 7fdddffff640 1 -- 172.21.15.59:0/4101243834 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddd8014bc0 con 0x7fdde815f5e0 2024-03-06T07:27:02.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.776+0000 7fdddffff640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddc806b4d0 0x7fddc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.776+0000 7fdddffff640 1 -- 172.21.15.59:0/4101243834 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fddd808aab0 con 0x7fdde815f5e0 2024-03-06T07:27:02.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.776+0000 7fdde6ffd640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddc806b4d0 0x7fddc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.776+0000 7fdde6ffd640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddc806b4d0 0x7fddc806d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fdde0036000 tx=0x7fdde003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.780+0000 7fdddffff640 1 -- 172.21.15.59:0/4101243834 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddd8059210 con 0x7fdde815f5e0 2024-03-06T07:27:02.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:02.961+0000 7fdded95c640 1 -- 172.21.15.59:0/4101243834 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdde80cea40 con 0x7fdde815f5e0 2024-03-06T07:27:03.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.600+0000 7fdddffff640 1 -- 172.21.15.59:0/4101243834 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v98) v1 ==== 181+0+0 (secure 0 0 0) 0x7fddd805d2b0 con 0x7fdde815f5e0 2024-03-06T07:27:03.603 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:27:03.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.604+0000 7fdded95c640 1 -- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddc806b4d0 msgr2=0x7fddc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.604+0000 7fdded95c640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddc806b4d0 0x7fddc806d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fdde0036000 tx=0x7fdde003c040 comp rx=0 tx=0).stop 2024-03-06T07:27:03.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.604+0000 7fdded95c640 1 -- 172.21.15.59:0/4101243834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdde815f5e0 msgr2=0x7fdde80c8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.604+0000 7fdded95c640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdde815f5e0 0x7fdde80c8e20 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fddd8007de0 tx=0x7fddd800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.605+0000 7fdded95c640 1 -- 172.21.15.59:0/4101243834 shutdown_connections 2024-03-06T07:27:03.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.605+0000 7fdded95c640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fddc806b4d0 0x7fddc806d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.605+0000 7fdded95c640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdde815f5e0 0x7fdde80c8e20 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.605+0000 7fdded95c640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdde80cec60 0x7fdde8157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.605+0000 7fdded95c640 1 --2- 172.21.15.59:0/4101243834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdde80b16b0 0x7fdde80c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.605+0000 7fdded95c640 1 -- 172.21.15.59:0/4101243834 >> 172.21.15.59:0/4101243834 conn(0x7fdde80d1d50 msgr2=0x7fdde80b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.605+0000 7fdded95c640 1 -- 172.21.15.59:0/4101243834 shutdown_connections 2024-03-06T07:27:03.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.605+0000 7fdded95c640 1 -- 172.21.15.59:0/4101243834 wait complete. 2024-03-06T07:27:03.622 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:27:03.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.731+0000 7f7612fa2640 1 Processor -- start 2024-03-06T07:27:03.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.731+0000 7f7612fa2640 1 -- start start 2024-03-06T07:27:03.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.732+0000 7f7612fa2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f760c0cf790 0x7f760c0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.732+0000 7f7612fa2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f760c0b34a0 con 0x7f760c0cf330 2024-03-06T07:27:03.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.732+0000 7f7612fa2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f760c0b3780 con 0x7f760c0d2320 2024-03-06T07:27:03.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.732+0000 7f7612fa2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f760c0b3250 con 0x7f760c0cf790 2024-03-06T07:27:03.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.732+0000 7f76127a1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f760c0cf790 0x7f760c0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.732+0000 7f761179f640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f760c0cf330 0x7f760c0b4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58726/0 (socket says 172.21.15.59:58726) 2024-03-06T07:27:03.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.732+0000 7f761179f640 1 -- 172.21.15.59:0/3028718498 learned_addr learned my addr 172.21.15.59:0/3028718498 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:03.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.732+0000 7f76127a1640 1 -- 172.21.15.59:0/3028718498 >> v1:172.21.15.164:6789/0 conn(0x7f760c0d2320 legacy=0x7f760c0d2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.732+0000 7f76127a1640 1 -- 172.21.15.59:0/3028718498 >> v1:172.21.15.59:6789/0 conn(0x7f760c0cf330 legacy=0x7f760c0b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.732+0000 7f7610f9e640 1 -- 172.21.15.59:0/3028718498 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065771754 0 0) 0x7f760c0b34a0 con 0x7f760c0cf330 2024-03-06T07:27:03.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.732+0000 7f76127a1640 1 -- 172.21.15.59:0/3028718498 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c0b3a60 con 0x7f760c0cf790 2024-03-06T07:27:03.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.733+0000 7f76127a1640 1 --2- 172.21.15.59:0/3028718498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760c0cf790 0x7f760c0b4970 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f7604009d90 tx=0x7f760402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2456470ea774da8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.733+0000 7f7610f9e640 1 -- 172.21.15.59:0/3028718498 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604039070 con 0x7f760c0cf790 2024-03-06T07:27:03.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.733+0000 7f7610f9e640 1 -- 172.21.15.59:0/3028718498 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760402cc60 con 0x7f760c0cf790 2024-03-06T07:27:03.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.733+0000 7f7610f9e640 1 -- 172.21.15.59:0/3028718498 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604034660 con 0x7f760c0cf790 2024-03-06T07:27:03.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.734+0000 7f7612fa2640 1 -- 172.21.15.59:0/3028718498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760c0cf790 msgr2=0x7f760c0b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.734+0000 7f7612fa2640 1 --2- 172.21.15.59:0/3028718498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760c0cf790 0x7f760c0b4970 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f7604009d90 tx=0x7f760402beb0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.734+0000 7f7612fa2640 1 -- 172.21.15.59:0/3028718498 shutdown_connections 2024-03-06T07:27:03.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.734+0000 7f7612fa2640 1 --2- 172.21.15.59:0/3028718498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760c0cf790 0x7f760c0b4970 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.734+0000 7f7612fa2640 1 -- 172.21.15.59:0/3028718498 >> 172.21.15.59:0/3028718498 conn(0x7f760c0d3d40 msgr2=0x7f760c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.734+0000 7f7612fa2640 1 -- 172.21.15.59:0/3028718498 shutdown_connections 2024-03-06T07:27:03.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.734+0000 7f7612fa2640 1 -- 172.21.15.59:0/3028718498 wait complete. 2024-03-06T07:27:03.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.735+0000 7f7612fa2640 1 Processor -- start 2024-03-06T07:27:03.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.735+0000 7f7612fa2640 1 -- start start 2024-03-06T07:27:03.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f7612fa2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f760c0cf330 0x7f760c0b83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f7612fa2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760c0cf790 0x7f760c0cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f7612fa2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f760c0d2320 0x7f760c0c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f7612fa2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c0b4eb0 con 0x7f760c0cf790 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f7612fa2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c0b3dd0 con 0x7f760c0cf330 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f7612fa2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760c0b5030 con 0x7f760c0d2320 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f761179f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760c0cf790 0x7f760c0cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f761179f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760c0cf790 0x7f760c0cb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57054/0 (socket says 172.21.15.59:57054) 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f761179f640 1 -- 172.21.15.59:0/2593491438 learned_addr learned my addr 172.21.15.59:0/2593491438 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f7611fa0640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f760c0cf330 0x7f760c0b83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f76127a1640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f760c0d2320 0x7f760c0c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f761179f640 1 -- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f760c0d2320 msgr2=0x7f760c0c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f761179f640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f760c0d2320 0x7f760c0c92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f761179f640 1 -- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f760c0cf330 msgr2=0x7f760c0b83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f761179f640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f760c0cf330 0x7f760c0b83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f761179f640 1 -- 172.21.15.59:0/2593491438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c15d1e0 con 0x7f760c0cf790 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f761179f640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760c0cf790 0x7f760c0cb300 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00c4a0 tx=0x7f75fc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f76027fc640 1 -- 172.21.15.59:0/2593491438 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc018070 con 0x7f760c0cf790 2024-03-06T07:27:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f76027fc640 1 -- 172.21.15.59:0/2593491438 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75fc0074e0 con 0x7f760c0cf790 2024-03-06T07:27:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f7612fa2640 1 -- 172.21.15.59:0/2593491438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760c1511f0 con 0x7f760c0cf790 2024-03-06T07:27:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f76027fc640 1 -- 172.21.15.59:0/2593491438 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc004a20 con 0x7f760c0cf790 2024-03-06T07:27:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.736+0000 7f7612fa2640 1 -- 172.21.15.59:0/2593491438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760c1517b0 con 0x7f760c0cf790 2024-03-06T07:27:03.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.737+0000 7f7612fa2640 1 -- 172.21.15.59:0/2593491438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760c04f330 con 0x7f760c0cf790 2024-03-06T07:27:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.738+0000 7f76027fc640 1 -- 172.21.15.59:0/2593491438 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75fc01b070 con 0x7f760c0cf790 2024-03-06T07:27:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.738+0000 7f76027fc640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75f006b4d0 0x7f75f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.738+0000 7f76027fc640 1 -- 172.21.15.59:0/2593491438 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5615+0+0 (secure 0 0 0) 0x7f75fc089430 con 0x7f760c0cf790 2024-03-06T07:27:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.739+0000 7f7611fa0640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75f006b4d0 0x7f75f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.739+0000 7f7611fa0640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75f006b4d0 0x7f75f006d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7608000f60 tx=0x7f7608009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.742+0000 7f76027fc640 1 -- 172.21.15.59:0/2593491438 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75fc058310 con 0x7f760c0cf790 2024-03-06T07:27:03.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:03.923+0000 7f7612fa2640 1 -- 172.21.15.59:0/2593491438 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f760c0cb0e0 con 0x7f760c0cf790 2024-03-06T07:27:04.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.605+0000 7f76027fc640 1 -- 172.21.15.59:0/2593491438 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v99) v1 ==== 169+0+0 (secure 0 0 0) 0x7f75fc057c10 con 0x7f760c0cf790 2024-03-06T07:27:04.607 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-03-06T07:27:04.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.608+0000 7f7612fa2640 1 -- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75f006b4d0 msgr2=0x7f75f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.608+0000 7f7612fa2640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75f006b4d0 0x7f75f006d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7608000f60 tx=0x7f7608009040 comp rx=0 tx=0).stop 2024-03-06T07:27:04.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.608+0000 7f7612fa2640 1 -- 172.21.15.59:0/2593491438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760c0cf790 msgr2=0x7f760c0cb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.608+0000 7f7612fa2640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760c0cf790 0x7f760c0cb300 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00c4a0 tx=0x7f75fc00c960 comp rx=0 tx=0).stop 2024-03-06T07:27:04.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.610+0000 7f7612fa2640 1 -- 172.21.15.59:0/2593491438 shutdown_connections 2024-03-06T07:27:04.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.610+0000 7f7612fa2640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75f006b4d0 0x7f75f006d990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.610+0000 7f7612fa2640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f760c0d2320 0x7f760c0c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.610+0000 7f7612fa2640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f760c0cf790 0x7f760c0cb300 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.610+0000 7f7612fa2640 1 --2- 172.21.15.59:0/2593491438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f760c0cf330 0x7f760c0b83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.610+0000 7f7612fa2640 1 -- 172.21.15.59:0/2593491438 >> 172.21.15.59:0/2593491438 conn(0x7f760c0d3d40 msgr2=0x7f760c0cfdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.612+0000 7f7612fa2640 1 -- 172.21.15.59:0/2593491438 shutdown_connections 2024-03-06T07:27:04.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:04.612+0000 7f7612fa2640 1 -- 172.21.15.59:0/2593491438 wait complete. 2024-03-06T07:27:04.632 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:04.634 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:04.684 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-03-06T07:27:04.684 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-03-06T07:27:04.684 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:04.684 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-03-06T07:27:04.684 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.684 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:04.685 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:04.687 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:04.688 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: bdev async discard = True 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: bdev enable discard = True 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore allocator = bitmap 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 1/20 2024-03-06T07:27:04.689 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 1/20 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 4/10 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-03-06T07:27:04.690 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-03-06T07:27:04.691 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: mon op complaint time = 120 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.692 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout: client mount timeout = 600 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout: debug client = 20 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = "" 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-03-06T07:27:04.693 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout:[mds] 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds = 20 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds balancer = 20 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug frag = True 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug scatterstat = True 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout: mds op complaint time = 180 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout: mds verify scatter = True 2024-03-06T07:27:04.694 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:27:04.695 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:27:04.695 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:27:04.695 INFO:teuthology.orchestra.run.smithi059.stdout: mds_max_purge_files = 6 2024-03-06T07:27:04.695 INFO:teuthology.orchestra.run.smithi059.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:04.695 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-03-06T07:27:04.695 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.696 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:04.697 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.698 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-03-06T07:27:04.699 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = bitmap 2024-03-06T07:27:04.700 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:04.701 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-03-06T07:27:04.702 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:04.703 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: mon op complaint time = 120 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: client mount timeout = 600 2024-03-06T07:27:04.704 INFO:teuthology.orchestra.run.smithi164.stdout: debug client = 20 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = "" 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout:[mds] 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds = 20 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds balancer = 20 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:04.705 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug frag = True 2024-03-06T07:27:04.706 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug scatterstat = True 2024-03-06T07:27:04.706 INFO:teuthology.orchestra.run.smithi164.stdout: mds op complaint time = 180 2024-03-06T07:27:04.706 INFO:teuthology.orchestra.run.smithi164.stdout: mds verify scatter = True 2024-03-06T07:27:04.706 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:27:04.706 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:27:04.706 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:27:04.706 INFO:teuthology.orchestra.run.smithi164.stdout: mds_max_purge_files = 6 2024-03-06T07:27:04.706 INFO:teuthology.orchestra.run.smithi164.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:04.708 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:04.750 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:04.798 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-03-06T07:27:04.798 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-03-06T07:27:04.798 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:04.798 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-03-06T07:27:04.798 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.798 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-03-06T07:27:04.798 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-03-06T07:27:04.799 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.800 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:04.801 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-03-06T07:27:04.802 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: bdev async discard = True 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: bdev enable discard = True 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore allocator = bitmap 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 1/20 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 1/20 2024-03-06T07:27:04.803 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 4/10 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:04.804 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.805 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: mon op complaint time = 120 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-03-06T07:27:04.806 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout: client mount timeout = 600 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout: debug client = 20 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = "" 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-03-06T07:27:04.807 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout:[mds] 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds = 20 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds balancer = 20 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug frag = True 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug scatterstat = True 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout: mds op complaint time = 180 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout: mds verify scatter = True 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:27:04.808 INFO:teuthology.orchestra.run.smithi059.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.809 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:04.810 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-03-06T07:27:04.811 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-03-06T07:27:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-03-06T07:27:04.813 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = bitmap 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-03-06T07:27:04.814 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:04.815 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.816 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: mon op complaint time = 120 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-03-06T07:27:04.817 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout: client mount timeout = 600 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout: debug client = 20 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = "" 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2024-03-06T07:27:04.818 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout:[mds] 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds = 20 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds balancer = 20 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug frag = True 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug scatterstat = True 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout: mds op complaint time = 180 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout: mds verify scatter = True 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:27:04.819 INFO:teuthology.orchestra.run.smithi164.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:27:04.822 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:04.867 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:27:04.998 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2024-03-06T07:27:04.998 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2024-03-06T07:27:04.998 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:04.999 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug trim objects = true 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.002 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: bdev async discard = True 2024-03-06T07:27:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: bdev enable discard = True 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore allocator = bitmap 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 1/20 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 1/20 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 4/10 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2024-03-06T07:27:05.004 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2024-03-06T07:27:05.005 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.006 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: mon op complaint time = 120 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: client mount timeout = 600 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: debug client = 20 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = "" 2024-03-06T07:27:05.007 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.a] 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.c] 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.b] 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.d] 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout:[mds] 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds = 20 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout: debug mds balancer = 20 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug frag = True 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout: mds debug scatterstat = True 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout: mds op complaint time = 180 2024-03-06T07:27:05.008 INFO:teuthology.orchestra.run.smithi059.stdout: mds verify scatter = True 2024-03-06T07:27:05.009 INFO:teuthology.orchestra.run.smithi059.stdout: osd op complaint time = 180 2024-03-06T07:27:05.009 INFO:teuthology.orchestra.run.smithi059.stdout: rados mon op timeout = 900 2024-03-06T07:27:05.009 INFO:teuthology.orchestra.run.smithi059.stdout: rados osd op timeout = 900 2024-03-06T07:27:05.046 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-03-06T07:27:05.046 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-03-06T07:27:05.046 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:27:05.046 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-03-06T07:27:05.046 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.046 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-03-06T07:27:05.046 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.046 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-03-06T07:27:05.046 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-03-06T07:27:05.047 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.048 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-03-06T07:27:05.048 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-03-06T07:27:05.048 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:27:05.048 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:27:05.048 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-03-06T07:27:05.048 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:27:05.048 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-03-06T07:27:05.049 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:27:05.049 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:27:05.049 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-03-06T07:27:05.049 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.049 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:27:05.049 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.050 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-03-06T07:27:05.050 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.050 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-03-06T07:27:05.050 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.050 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-03-06T07:27:05.050 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-03-06T07:27:05.051 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-03-06T07:27:05.051 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.59,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2024-03-06T07:27:05.051 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 43c1ee46-a208-44bd-a1f3-57a84ef7ed13 2024-03-06T07:27:05.051 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.051 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-03-06T07:27:05.051 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-03-06T07:27:05.052 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.052 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-03-06T07:27:05.052 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-03-06T07:27:05.052 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:27:05.052 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.053 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-03-06T07:27:05.053 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-03-06T07:27:05.053 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.053 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-03-06T07:27:05.053 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-03-06T07:27:05.053 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-03-06T07:27:05.054 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-03-06T07:27:05.054 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.054 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-03-06T07:27:05.054 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-03-06T07:27:05.054 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.055 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:27:05.055 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.055 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-03-06T07:27:05.055 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.055 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-03-06T07:27:05.055 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.056 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:27:05.056 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-03-06T07:27:05.056 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.056 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-03-06T07:27:05.056 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-03-06T07:27:05.057 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2024-03-06T07:27:05.057 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2024-03-06T07:27:05.057 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = bitmap 2024-03-06T07:27:05.057 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-03-06T07:27:05.057 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-03-06T07:27:05.057 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2024-03-06T07:27:05.058 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2024-03-06T07:27:05.058 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:05.058 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-03-06T07:27:05.058 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2024-03-06T07:27:05.058 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:27:05.058 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-03-06T07:27:05.059 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:27:05.059 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:27:05.059 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-03-06T07:27:05.059 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:27:05.059 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.059 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-03-06T07:27:05.060 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:05.060 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-03-06T07:27:05.060 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:27:05.060 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:27:05.060 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:05.061 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:05.061 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-03-06T07:27:05.061 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.061 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-03-06T07:27:05.061 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:05.062 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-03-06T07:27:05.062 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-03-06T07:27:05.062 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-03-06T07:27:05.062 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-03-06T07:27:05.062 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-03-06T07:27:05.062 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:27:05.063 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-03-06T07:27:05.063 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-03-06T07:27:05.063 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:27:05.063 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.063 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:27:05.063 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:27:05.064 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:27:05.064 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.064 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:27:05.064 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:27:05.064 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:27:05.064 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.065 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-03-06T07:27:05.065 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:27:05.065 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.065 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:27:05.065 INFO:teuthology.orchestra.run.smithi164.stdout: mon op complaint time = 120 2024-03-06T07:27:05.065 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-06T07:27:05.066 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-03-06T07:27:05.066 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-03-06T07:27:05.066 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-03-06T07:27:05.066 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-03-06T07:27:05.066 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:27:05.067 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:27:05.067 INFO:teuthology.orchestra.run.smithi164.stdout: client mount timeout = 600 2024-03-06T07:27:05.067 INFO:teuthology.orchestra.run.smithi164.stdout: debug client = 20 2024-03-06T07:27:05.067 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:05.067 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = "" 2024-03-06T07:27:05.067 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:27:05.068 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:27:05.068 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-03-06T07:27:05.068 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2024-03-06T07:27:05.068 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2024-03-06T07:27:05.068 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2024-03-06T07:27:05.068 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2024-03-06T07:27:05.069 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2024-03-06T07:27:05.069 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2024-03-06T07:27:05.069 INFO:teuthology.orchestra.run.smithi164.stdout:[mds] 2024-03-06T07:27:05.069 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds = 20 2024-03-06T07:27:05.069 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds balancer = 20 2024-03-06T07:27:05.069 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-03-06T07:27:05.070 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug frag = True 2024-03-06T07:27:05.070 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug scatterstat = True 2024-03-06T07:27:05.070 INFO:teuthology.orchestra.run.smithi164.stdout: mds op complaint time = 180 2024-03-06T07:27:05.070 INFO:teuthology.orchestra.run.smithi164.stdout: mds verify scatter = True 2024-03-06T07:27:05.070 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2024-03-06T07:27:05.070 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2024-03-06T07:27:05.071 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2024-03-06T07:27:05.072 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:27:05.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.174+0000 7f0c5e302640 1 Processor -- start 2024-03-06T07:27:05.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.175+0000 7f0c5e302640 1 -- start start 2024-03-06T07:27:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.175+0000 7f0c5e302640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c580d4860 0x7f0c580d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.175+0000 7f0c5e302640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0c580b0e30 0x7f0c580b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.175+0000 7f0c5e302640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c581618b0 con 0x7f0c581614d0 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.175+0000 7f0c5e302640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c580d4cc0 con 0x7f0c580b0e30 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.175+0000 7f0c5e302640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580cf400 con 0x7f0c580d4860 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.175+0000 7f0c5c89f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0c580b0e30 0x7f0c580b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.175+0000 7f0c57fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0c581614d0 0x7f0c580d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58738/0 (socket says 172.21.15.59:58738) 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.175+0000 7f0c57fff640 1 -- 172.21.15.59:0/2862470024 learned_addr learned my addr 172.21.15.59:0/2862470024 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.176+0000 7f0c577fe640 1 --2- 172.21.15.59:0/2862470024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c580d4860 0x7f0c580d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.176+0000 7f0c5c89f640 1 -- 172.21.15.59:0/2862470024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c580d4860 msgr2=0x7f0c580d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.176+0000 7f0c5c89f640 1 --2- 172.21.15.59:0/2862470024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c580d4860 0x7f0c580d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.176+0000 7f0c56ffd640 1 -- 172.21.15.59:0/2862470024 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277257840 0 0) 0x7f0c581618b0 con 0x7f0c581614d0 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.176+0000 7f0c5c89f640 1 -- 172.21.15.59:0/2862470024 >> v1:172.21.15.59:6789/0 conn(0x7f0c581614d0 legacy=0x7f0c580d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.176+0000 7f0c5c89f640 1 -- 172.21.15.59:0/2862470024 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c580cf6a0 con 0x7f0c580b0e30 2024-03-06T07:27:05.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.176+0000 7f0c5c89f640 1 --2- 172.21.15.59:0/2862470024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c580b0e30 0x7f0c580b12c0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009d90 tx=0x7f0c5002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f11e1d9e1bfa19a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.177+0000 7f0c56ffd640 1 -- 172.21.15.59:0/2862470024 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50039070 con 0x7f0c580b0e30 2024-03-06T07:27:05.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.177+0000 7f0c56ffd640 1 -- 172.21.15.59:0/2862470024 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5002cc50 con 0x7f0c580b0e30 2024-03-06T07:27:05.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.177+0000 7f0c56ffd640 1 -- 172.21.15.59:0/2862470024 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50034440 con 0x7f0c580b0e30 2024-03-06T07:27:05.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.177+0000 7f0c5e302640 1 -- 172.21.15.59:0/2862470024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c580b0e30 msgr2=0x7f0c580b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.177+0000 7f0c5e302640 1 --2- 172.21.15.59:0/2862470024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c580b0e30 0x7f0c580b12c0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009d90 tx=0x7f0c5002bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.177+0000 7f0c5e302640 1 -- 172.21.15.59:0/2862470024 shutdown_connections 2024-03-06T07:27:05.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.177+0000 7f0c5e302640 1 --2- 172.21.15.59:0/2862470024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c580b0e30 0x7f0c580b12c0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.177+0000 7f0c5e302640 1 --2- 172.21.15.59:0/2862470024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c580d4860 0x7f0c580d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.177+0000 7f0c5e302640 1 -- 172.21.15.59:0/2862470024 >> 172.21.15.59:0/2862470024 conn(0x7f0c580d3d40 msgr2=0x7f0c580d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.177+0000 7f0c5e302640 1 -- 172.21.15.59:0/2862470024 shutdown_connections 2024-03-06T07:27:05.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.177+0000 7f0c5e302640 1 -- 172.21.15.59:0/2862470024 wait complete. 2024-03-06T07:27:05.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.178+0000 7f0c5e302640 1 Processor -- start 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.179+0000 7f0c5e302640 1 -- start start 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.179+0000 7f0c5e302640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c580b0e30 0x7f0c580cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.179+0000 7f0c5e302640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c580d4860 0x7f0c580a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.179+0000 7f0c5e302640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c581614d0 0x7f0c580cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.179+0000 7f0c5e302640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580b0910 con 0x7f0c580b0e30 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.179+0000 7f0c5e302640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580cfc60 con 0x7f0c581614d0 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.179+0000 7f0c5e302640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580d2560 con 0x7f0c580d4860 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.179+0000 7f0c57fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c580b0e30 0x7f0c580cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.179+0000 7f0c57fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c580b0e30 0x7f0c580cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57068/0 (socket says 172.21.15.59:57068) 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.179+0000 7f0c57fff640 1 -- 172.21.15.59:0/1568858747 learned_addr learned my addr 172.21.15.59:0/1568858747 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c577fe640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c580d4860 0x7f0c580a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c5c89f640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c581614d0 0x7f0c580cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c57fff640 1 -- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c580d4860 msgr2=0x7f0c580a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c57fff640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c580d4860 0x7f0c580a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c57fff640 1 -- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c581614d0 msgr2=0x7f0c580cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c57fff640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c581614d0 0x7f0c580cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c57fff640 1 -- 172.21.15.59:0/1568858747 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c580ccf00 con 0x7f0c580b0e30 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c5c89f640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c581614d0 0x7f0c580cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c57fff640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c580b0e30 0x7f0c580cae10 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800c430 tx=0x7f0c4800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c54ff9640 1 -- 172.21.15.59:0/1568858747 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c480074e0 con 0x7f0c580b0e30 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c54ff9640 1 -- 172.21.15.59:0/1568858747 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c480047e0 con 0x7f0c580b0e30 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c5e302640 1 -- 172.21.15.59:0/1568858747 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c580cbd70 con 0x7f0c580b0e30 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c54ff9640 1 -- 172.21.15.59:0/1568858747 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48004bd0 con 0x7f0c580b0e30 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.180+0000 7f0c5e302640 1 -- 172.21.15.59:0/1568858747 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c580cb7b0 con 0x7f0c580b0e30 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.181+0000 7f0c5e302640 1 -- 172.21.15.59:0/1568858747 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c580cbb50 con 0x7f0c580b0e30 2024-03-06T07:27:05.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.181+0000 7f0c54ff9640 1 -- 172.21.15.59:0/1568858747 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c48011070 con 0x7f0c580b0e30 2024-03-06T07:27:05.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.182+0000 7f0c54ff9640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4406b4d0 0x7f0c4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.182+0000 7f0c54ff9640 1 -- 172.21.15.59:0/1568858747 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f0c48089d10 con 0x7f0c580b0e30 2024-03-06T07:27:05.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.182+0000 7f0c577fe640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4406b4d0 0x7f0c4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.182+0000 7f0c577fe640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4406b4d0 0x7f0c4406d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0c580d4720 tx=0x7f0c4c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.186+0000 7f0c54ff9640 1 -- 172.21.15.59:0/1568858747 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c48058cf0 con 0x7f0c580b0e30 2024-03-06T07:27:05.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.368+0000 7f0c5e302640 1 -- 172.21.15.59:0/1568858747 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0c580caad0 con 0x7f0c580b0e30 2024-03-06T07:27:05.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.371+0000 7f0c54ff9640 1 -- 172.21.15.59:0/1568858747 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0c480585f0 con 0x7f0c580b0e30 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 -- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4406b4d0 msgr2=0x7f0c4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4406b4d0 0x7f0c4406d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0c580d4720 tx=0x7f0c4c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 -- 172.21.15.59:0/1568858747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c580b0e30 msgr2=0x7f0c580cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c580b0e30 0x7f0c580cae10 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800c430 tx=0x7f0c4800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 -- 172.21.15.59:0/1568858747 shutdown_connections 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c4406b4d0 0x7f0c4406d990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c581614d0 0x7f0c580cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c580d4860 0x7f0c580a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 --2- 172.21.15.59:0/1568858747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0c580b0e30 0x7f0c580cae10 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 -- 172.21.15.59:0/1568858747 >> 172.21.15.59:0/1568858747 conn(0x7f0c580d3d40 msgr2=0x7f0c580b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 -- 172.21.15.59:0/1568858747 shutdown_connections 2024-03-06T07:27:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.373+0000 7f0c5e302640 1 -- 172.21.15.59:0/1568858747 wait complete. 2024-03-06T07:27:05.386 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:27:05.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.485+0000 7f1d2fff7640 1 Processor -- start 2024-03-06T07:27:05.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.485+0000 7f1d2fff7640 1 -- start start 2024-03-06T07:27:05.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.486+0000 7f1d2fff7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1d2806b6c0 0x7f1d2806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.486+0000 7f1d2fff7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d280d16e0 con 0x7f1d2815e420 2024-03-06T07:27:05.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.486+0000 7f1d2fff7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2815eea0 con 0x7f1d281584d0 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.486+0000 7f1d2fff7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d2815f180 con 0x7f1d2806b6c0 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.486+0000 7f1d2d592640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1d2815e420 0x7f1d2806afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58740/0 (socket says 172.21.15.59:58740) 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.486+0000 7f1d2d592640 1 -- 172.21.15.59:0/3280442662 learned_addr learned my addr 172.21.15.59:0/3280442662 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.486+0000 7f1d2e594640 1 --2- 172.21.15.59:0/3280442662 >> v2:172.21.15.164:3300/0 conn(0x7f1d2806b6c0 0x7f1d2806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.486+0000 7f1d2cd91640 1 -- 172.21.15.59:0/3280442662 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552591375 0 0) 0x7f1d280d16e0 con 0x7f1d2815e420 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.486+0000 7f1d2cd91640 1 -- 172.21.15.59:0/3280442662 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d10003520 con 0x7f1d2815e420 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.486+0000 7f1d2cd91640 1 -- 172.21.15.59:0/3280442662 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014682926 0 0) 0x7f1d2815eea0 con 0x7f1d281584d0 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.486+0000 7f1d2cd91640 1 -- 172.21.15.59:0/3280442662 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d280d16e0 con 0x7f1d281584d0 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.487+0000 7f1d2cd91640 1 -- 172.21.15.59:0/3280442662 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3217981490 0 0) 0x7f1d10003520 con 0x7f1d2815e420 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.487+0000 7f1d2e594640 1 -- 172.21.15.59:0/3280442662 >> v1:172.21.15.164:6789/0 conn(0x7f1d281584d0 legacy=0x7f1d280d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.487+0000 7f1d2e594640 1 -- 172.21.15.59:0/3280442662 >> v1:172.21.15.59:6789/0 conn(0x7f1d2815e420 legacy=0x7f1d2806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.487+0000 7f1d2e594640 1 -- 172.21.15.59:0/3280442662 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d280d07d0 con 0x7f1d2806b6c0 2024-03-06T07:27:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.487+0000 7f1d2e594640 1 --2- 172.21.15.59:0/3280442662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d2806b6c0 0x7f1d2806da20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009d90 tx=0x7f1d1802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54bc10de3b001955 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.487+0000 7f1d2cd91640 1 -- 172.21.15.59:0/3280442662 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18039070 con 0x7f1d2806b6c0 2024-03-06T07:27:05.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.488+0000 7f1d2cd91640 1 -- 172.21.15.59:0/3280442662 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1802c940 con 0x7f1d2806b6c0 2024-03-06T07:27:05.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.488+0000 7f1d2cd91640 1 -- 172.21.15.59:0/3280442662 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c001670 con 0x7f1d2806b6c0 2024-03-06T07:27:05.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.488+0000 7f1d2fff7640 1 -- 172.21.15.59:0/3280442662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d2806b6c0 msgr2=0x7f1d2806da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.488+0000 7f1d2fff7640 1 --2- 172.21.15.59:0/3280442662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d2806b6c0 0x7f1d2806da20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009d90 tx=0x7f1d1802be10 comp rx=0 tx=0).stop 2024-03-06T07:27:05.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.488+0000 7f1d2fff7640 1 -- 172.21.15.59:0/3280442662 shutdown_connections 2024-03-06T07:27:05.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.488+0000 7f1d2fff7640 1 --2- 172.21.15.59:0/3280442662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d2806b6c0 0x7f1d2806da20 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.488+0000 7f1d2fff7640 1 -- 172.21.15.59:0/3280442662 >> 172.21.15.59:0/3280442662 conn(0x7f1d2806c0c0 msgr2=0x7f1d2806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.488+0000 7f1d2fff7640 1 -- 172.21.15.59:0/3280442662 shutdown_connections 2024-03-06T07:27:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.488+0000 7f1d2fff7640 1 -- 172.21.15.59:0/3280442662 wait complete. 2024-03-06T07:27:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.489+0000 7f1d2fff7640 1 Processor -- start 2024-03-06T07:27:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.489+0000 7f1d2fff7640 1 -- start start 2024-03-06T07:27:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2fff7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d2806b6c0 0x7f1d28157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2fff7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d281584d0 0x7f1d281e7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2fff7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2815e420 0x7f1d28145d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2fff7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d280b59b0 con 0x7f1d2815e420 2024-03-06T07:27:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2fff7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d281588b0 con 0x7f1d2806b6c0 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2fff7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d280b5b30 con 0x7f1d281584d0 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2e594640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2815e420 0x7f1d28145d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2e594640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2815e420 0x7f1d28145d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57070/0 (socket says 172.21.15.59:57070) 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2e594640 1 -- 172.21.15.59:0/539413411 learned_addr learned my addr 172.21.15.59:0/539413411 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2e594640 1 -- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d281584d0 msgr2=0x7f1d281e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2e594640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d281584d0 0x7f1d281e7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2e594640 1 -- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d2806b6c0 msgr2=0x7f1d28157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2e594640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d2806b6c0 0x7f1d28157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.490+0000 7f1d2e594640 1 -- 172.21.15.59:0/539413411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d281f4090 con 0x7f1d2815e420 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.491+0000 7f1d2e594640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2815e420 0x7f1d28145d90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f1d18037040 tx=0x7f1d18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.491+0000 7f1d267fc640 1 -- 172.21.15.59:0/539413411 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18039050 con 0x7f1d2815e420 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.491+0000 7f1d267fc640 1 -- 172.21.15.59:0/539413411 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d18045070 con 0x7f1d2815e420 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.491+0000 7f1d2fff7640 1 -- 172.21.15.59:0/539413411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d281f4370 con 0x7f1d2815e420 2024-03-06T07:27:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.491+0000 7f1d267fc640 1 -- 172.21.15.59:0/539413411 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18040690 con 0x7f1d2815e420 2024-03-06T07:27:05.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.491+0000 7f1d2fff7640 1 -- 172.21.15.59:0/539413411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d281f4930 con 0x7f1d2815e420 2024-03-06T07:27:05.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.492+0000 7f1d2fff7640 1 -- 172.21.15.59:0/539413411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2806f8d0 con 0x7f1d2815e420 2024-03-06T07:27:05.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.492+0000 7f1d267fc640 1 -- 172.21.15.59:0/539413411 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d18040830 con 0x7f1d2815e420 2024-03-06T07:27:05.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.493+0000 7f1d267fc640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d1406b4d0 0x7f1d1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.493+0000 7f1d267fc640 1 -- 172.21.15.59:0/539413411 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f1d180ac740 con 0x7f1d2815e420 2024-03-06T07:27:05.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.493+0000 7f1d2dd93640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d1406b4d0 0x7f1d1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.493+0000 7f1d2dd93640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d1406b4d0 0x7f1d1406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1d200010e0 tx=0x7f1d20007950 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.497+0000 7f1d267fc640 1 -- 172.21.15.59:0/539413411 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d2806f8d0 con 0x7f1d2815e420 2024-03-06T07:27:05.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:05.752+0000 7f1d2fff7640 1 -- 172.21.15.59:0/539413411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7f1d2804f330 con 0x7f1d2815e420 2024-03-06T07:27:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.608+0000 7f1d267fc640 1 -- 172.21.15.59:0/539413411 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v299) v1 ==== 134+0+0 (secure 0 0 0) 0x7f1d1804f020 con 0x7f1d2815e420 2024-03-06T07:27:06.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 -- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d1406b4d0 msgr2=0x7f1d1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d1406b4d0 0x7f1d1406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1d200010e0 tx=0x7f1d20007950 comp rx=0 tx=0).stop 2024-03-06T07:27:06.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 -- 172.21.15.59:0/539413411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2815e420 msgr2=0x7f1d28145d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2815e420 0x7f1d28145d90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f1d18037040 tx=0x7f1d18004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:06.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 -- 172.21.15.59:0/539413411 shutdown_connections 2024-03-06T07:27:06.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1d1406b4d0 0x7f1d1406d990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d2815e420 0x7f1d28145d90 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d281584d0 0x7f1d281e7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 --2- 172.21.15.59:0/539413411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d2806b6c0 0x7f1d28157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 -- 172.21.15.59:0/539413411 >> 172.21.15.59:0/539413411 conn(0x7f1d2806c0c0 msgr2=0x7f1d2806e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 -- 172.21.15.59:0/539413411 shutdown_connections 2024-03-06T07:27:06.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.611+0000 7f1d2fff7640 1 -- 172.21.15.59:0/539413411 wait complete. 2024-03-06T07:27:06.627 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-03-06T07:27:06.627 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-03-06T07:27:06.627 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:27:06.627 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:06.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.733+0000 7fcc735e9640 1 Processor -- start 2024-03-06T07:27:06.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.733+0000 7fcc735e9640 1 -- start start 2024-03-06T07:27:06.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.734+0000 7fcc735e9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcc6c0d04a0 0x7fcc6c0ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.734+0000 7fcc735e9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6c161980 con 0x7fcc6c15bb20 2024-03-06T07:27:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.734+0000 7fcc735e9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6c161c60 con 0x7fcc6c0cfff0 2024-03-06T07:27:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.734+0000 7fcc735e9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc6c0d2500 con 0x7fcc6c0d04a0 2024-03-06T07:27:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.734+0000 7fcc71b86640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcc6c0d04a0 0x7fcc6c0ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.734+0000 7fcc71385640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcc6c15bb20 0x7fcc6c0d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58750/0 (socket says 172.21.15.59:58750) 2024-03-06T07:27:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.734+0000 7fcc71385640 1 -- 172.21.15.59:0/19059626 learned_addr learned my addr 172.21.15.59:0/19059626 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.735+0000 7fcc71b86640 1 -- 172.21.15.59:0/19059626 >> v1:172.21.15.164:6789/0 conn(0x7fcc6c0cfff0 legacy=0x7fcc6c0cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.735+0000 7fcc71b86640 1 -- 172.21.15.59:0/19059626 >> v1:172.21.15.59:6789/0 conn(0x7fcc6c15bb20 legacy=0x7fcc6c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.735+0000 7fcc71b86640 1 -- 172.21.15.59:0/19059626 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c161f40 con 0x7fcc6c0d04a0 2024-03-06T07:27:06.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.735+0000 7fcc6bfff640 1 -- 172.21.15.59:0/19059626 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133791481 0 0) 0x7fcc6c161980 con 0x7fcc6c15bb20 2024-03-06T07:27:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.735+0000 7fcc71b86640 1 --2- 172.21.15.59:0/19059626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6c0d04a0 0x7fcc6c0ccfc0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009d90 tx=0x7fcc5c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73bbf582a2fa082 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.736+0000 7fcc6bfff640 1 -- 172.21.15.59:0/19059626 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c039070 con 0x7fcc6c0d04a0 2024-03-06T07:27:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.736+0000 7fcc6bfff640 1 -- 172.21.15.59:0/19059626 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c02ccf0 con 0x7fcc6c0d04a0 2024-03-06T07:27:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.736+0000 7fcc6bfff640 1 -- 172.21.15.59:0/19059626 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c034660 con 0x7fcc6c0d04a0 2024-03-06T07:27:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.736+0000 7fcc735e9640 1 -- 172.21.15.59:0/19059626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6c0d04a0 msgr2=0x7fcc6c0ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.736+0000 7fcc735e9640 1 --2- 172.21.15.59:0/19059626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6c0d04a0 0x7fcc6c0ccfc0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009d90 tx=0x7fcc5c02bf40 comp rx=0 tx=0).stop 2024-03-06T07:27:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.736+0000 7fcc735e9640 1 -- 172.21.15.59:0/19059626 shutdown_connections 2024-03-06T07:27:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.736+0000 7fcc735e9640 1 --2- 172.21.15.59:0/19059626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6c0d04a0 0x7fcc6c0ccfc0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.736+0000 7fcc735e9640 1 -- 172.21.15.59:0/19059626 >> 172.21.15.59:0/19059626 conn(0x7fcc6c09a9f0 msgr2=0x7fcc6c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.736+0000 7fcc735e9640 1 -- 172.21.15.59:0/19059626 shutdown_connections 2024-03-06T07:27:06.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.736+0000 7fcc735e9640 1 -- 172.21.15.59:0/19059626 wait complete. 2024-03-06T07:27:06.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.737+0000 7fcc735e9640 1 Processor -- start 2024-03-06T07:27:06.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.738+0000 7fcc735e9640 1 -- start start 2024-03-06T07:27:06.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.738+0000 7fcc735e9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc6c0cfff0 0x7fcc6c1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.738+0000 7fcc735e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc6c0d04a0 0x7fcc6c0c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.738+0000 7fcc735e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6c15bb20 0x7fcc6c0c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.738+0000 7fcc735e9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c0d08d0 con 0x7fcc6c15bb20 2024-03-06T07:27:06.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.738+0000 7fcc735e9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c162500 con 0x7fcc6c0cfff0 2024-03-06T07:27:06.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.738+0000 7fcc735e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c0d1980 con 0x7fcc6c0d04a0 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.738+0000 7fcc71b86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6c15bb20 0x7fcc6c0c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.738+0000 7fcc71b86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6c15bb20 0x7fcc6c0c6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57094/0 (socket says 172.21.15.59:57094) 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.738+0000 7fcc71b86640 1 -- 172.21.15.59:0/143065837 learned_addr learned my addr 172.21.15.59:0/143065837 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc71385640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc6c0cfff0 0x7fcc6c1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc71b86640 1 -- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc6c0d04a0 msgr2=0x7fcc6c0c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc70b84640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc6c0d04a0 0x7fcc6c0c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc71b86640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc6c0d04a0 0x7fcc6c0c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc71b86640 1 -- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc6c0cfff0 msgr2=0x7fcc6c1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc71b86640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc6c0cfff0 0x7fcc6c1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc71b86640 1 -- 172.21.15.59:0/143065837 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c0cad40 con 0x7fcc6c15bb20 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc71385640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc6c0cfff0 0x7fcc6c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc70b84640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc6c0d04a0 0x7fcc6c0c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc71b86640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6c15bb20 0x7fcc6c0c6c70 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c037040 tx=0x7fcc5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc69ffb640 1 -- 172.21.15.59:0/143065837 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c039050 con 0x7fcc6c15bb20 2024-03-06T07:27:06.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc69ffb640 1 -- 172.21.15.59:0/143065837 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c03a040 con 0x7fcc6c15bb20 2024-03-06T07:27:06.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc69ffb640 1 -- 172.21.15.59:0/143065837 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c03e610 con 0x7fcc6c15bb20 2024-03-06T07:27:06.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc735e9640 1 -- 172.21.15.59:0/143065837 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6c0cb020 con 0x7fcc6c15bb20 2024-03-06T07:27:06.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.739+0000 7fcc735e9640 1 -- 172.21.15.59:0/143065837 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6c0cb5e0 con 0x7fcc6c15bb20 2024-03-06T07:27:06.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.740+0000 7fcc735e9640 1 -- 172.21.15.59:0/143065837 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6c04f330 con 0x7fcc6c15bb20 2024-03-06T07:27:06.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.741+0000 7fcc69ffb640 1 -- 172.21.15.59:0/143065837 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc5c047020 con 0x7fcc6c15bb20 2024-03-06T07:27:06.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.741+0000 7fcc69ffb640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc5806b4d0 0x7fcc5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.741+0000 7fcc69ffb640 1 -- 172.21.15.59:0/143065837 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fcc5c0ac3c0 con 0x7fcc6c15bb20 2024-03-06T07:27:06.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.741+0000 7fcc71385640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc5806b4d0 0x7fcc5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.742+0000 7fcc71385640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc5806b4d0 0x7fcc5806d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fcc64000f60 tx=0x7fcc64009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.745+0000 7fcc69ffb640 1 -- 172.21.15.59:0/143065837 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc5c04e020 con 0x7fcc6c15bb20 2024-03-06T07:27:06.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.927+0000 7fcc735e9640 1 -- 172.21.15.59:0/143065837 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc6c0cdf40 con 0x7fcc6c15bb20 2024-03-06T07:27:06.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.928+0000 7fcc69ffb640 1 -- 172.21.15.59:0/143065837 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+13401 (secure 0 0 0) 0x7fcc5c07b4f0 con 0x7fcc6c15bb20 2024-03-06T07:27:06.930 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:06.930 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":99,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:27:04.602022+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6833/4004944688":"2024-03-07T07:27:01.597332+0000","172.21.15.59:6834/2799249020":"2024-03-07T07:26:29.597825+0000","172.21.15.59:6835/4004944688":"2024-03-07T07:27:01.597332+0000","172.21.15.59:6835/1784120157":"2024-03-07T07:26:29.585744+0000","172.21.15.59:6832/2799249020":"2024-03-07T07:26:29.597825+0000","172.21.15.59:6833/1784120157":"2024-03-07T07:26:29.585744+0000","172.21.15.164:6837/4292771408":"2024-03-07T07:26:29.578123+0000","172.21.15.164:6835/4292771408":"2024-03-07T07:26:29.578123+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:06.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.930+0000 7fcc735e9640 1 -- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc5806b4d0 msgr2=0x7fcc5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.930+0000 7fcc735e9640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc5806b4d0 0x7fcc5806d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fcc64000f60 tx=0x7fcc64009040 comp rx=0 tx=0).stop 2024-03-06T07:27:06.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.930+0000 7fcc735e9640 1 -- 172.21.15.59:0/143065837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6c15bb20 msgr2=0x7fcc6c0c6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.930+0000 7fcc735e9640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6c15bb20 0x7fcc6c0c6c70 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c037040 tx=0x7fcc5c004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:06.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.930+0000 7fcc735e9640 1 -- 172.21.15.59:0/143065837 shutdown_connections 2024-03-06T07:27:06.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.930+0000 7fcc735e9640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcc5806b4d0 0x7fcc5806d990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.930+0000 7fcc735e9640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc6c15bb20 0x7fcc6c0c6c70 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.930+0000 7fcc735e9640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc6c0d04a0 0x7fcc6c0c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.930+0000 7fcc735e9640 1 --2- 172.21.15.59:0/143065837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc6c0cfff0 0x7fcc6c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.930+0000 7fcc735e9640 1 -- 172.21.15.59:0/143065837 >> 172.21.15.59:0/143065837 conn(0x7fcc6c09a9f0 msgr2=0x7fcc6c09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.930+0000 7fcc735e9640 1 -- 172.21.15.59:0/143065837 shutdown_connections 2024-03-06T07:27:06.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:06.931+0000 7fcc735e9640 1 -- 172.21.15.59:0/143065837 wait complete. 2024-03-06T07:27:06.944 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:06.944 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:07.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.046+0000 7f0409c9a640 1 Processor -- start 2024-03-06T07:27:07.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.046+0000 7f0409c9a640 1 -- start start 2024-03-06T07:27:07.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.046+0000 7f0409c9a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f04041614d0 0x7f04040d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.046+0000 7f0409c9a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f04040d1910 0x7f04040d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.046+0000 7f0409c9a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04041618b0 con 0x7f04040d22e0 2024-03-06T07:27:07.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.046+0000 7f0409c9a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04040cf400 con 0x7f04040d1910 2024-03-06T07:27:07.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.046+0000 7f0409c9a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04040d2710 con 0x7f04041614d0 2024-03-06T07:27:07.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.046+0000 7f0402ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f04040d1910 0x7f04040d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.047+0000 7f0402ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f04040d1910 0x7f04040d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57096/0 (socket says 172.21.15.59:57096) 2024-03-06T07:27:07.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.047+0000 7f0402ffd640 1 -- 172.21.15.59:0/784624694 learned_addr learned my addr 172.21.15.59:0/784624694 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:07.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.047+0000 7f0402ffd640 1 -- 172.21.15.59:0/784624694 >> v2:172.21.15.164:3300/0 conn(0x7f04041614d0 msgr2=0x7f04040d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.047+0000 7f04037fe640 1 --2- 172.21.15.59:0/784624694 >> v2:172.21.15.164:3300/0 conn(0x7f04041614d0 0x7f04040d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.047+0000 7f0402ffd640 1 --2- 172.21.15.59:0/784624694 >> v2:172.21.15.164:3300/0 conn(0x7f04041614d0 0x7f04040d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.047+0000 7f0402ffd640 1 -- 172.21.15.59:0/784624694 >> v1:172.21.15.164:6789/0 conn(0x7f04040d22e0 legacy=0x7f04040d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.047+0000 7f0402ffd640 1 -- 172.21.15.59:0/784624694 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04040cf6a0 con 0x7f04040d1910 2024-03-06T07:27:07.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.047+0000 7f04037fe640 1 --2- 172.21.15.59:0/784624694 >> v2:172.21.15.164:3300/0 conn(0x7f04041614d0 0x7f04040d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:07.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f0402ffd640 1 --2- 172.21.15.59:0/784624694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04040d1910 0x7f04040d1da0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f04040ce550 tx=0x7f03f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c723b8e2d818eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f04027fc640 1 -- 172.21.15.59:0/784624694 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8035020 con 0x7f04040d1910 2024-03-06T07:27:07.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f04027fc640 1 -- 172.21.15.59:0/784624694 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f8031640 con 0x7f04040d1910 2024-03-06T07:27:07.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f04027fc640 1 -- 172.21.15.59:0/784624694 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f802d040 con 0x7f04040d1910 2024-03-06T07:27:07.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f0409c9a640 1 -- 172.21.15.59:0/784624694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04040d1910 msgr2=0x7f04040d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f0409c9a640 1 --2- 172.21.15.59:0/784624694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04040d1910 0x7f04040d1da0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f04040ce550 tx=0x7f03f802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f0409c9a640 1 -- 172.21.15.59:0/784624694 shutdown_connections 2024-03-06T07:27:07.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f0409c9a640 1 --2- 172.21.15.59:0/784624694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04040d1910 0x7f04040d1da0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f0409c9a640 1 --2- 172.21.15.59:0/784624694 >> v2:172.21.15.164:3300/0 conn(0x7f04041614d0 0x7f04040d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f0409c9a640 1 -- 172.21.15.59:0/784624694 >> 172.21.15.59:0/784624694 conn(0x7f04040d3d40 msgr2=0x7f04040d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f0409c9a640 1 -- 172.21.15.59:0/784624694 shutdown_connections 2024-03-06T07:27:07.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.048+0000 7f0409c9a640 1 -- 172.21.15.59:0/784624694 wait complete. 2024-03-06T07:27:07.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.049+0000 7f0409c9a640 1 Processor -- start 2024-03-06T07:27:07.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.050+0000 7f0409c9a640 1 -- start start 2024-03-06T07:27:07.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.050+0000 7f0409c9a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04040d1910 0x7f040415daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.050+0000 7f0409c9a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04040d22e0 0x7f040415c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.050+0000 7f0409c9a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04041614d0 0x7f04041e7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.050+0000 7f0409c9a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04040b0910 con 0x7f04041614d0 2024-03-06T07:27:07.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.050+0000 7f0409c9a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04040cfc60 con 0x7f04040d22e0 2024-03-06T07:27:07.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.050+0000 7f0409c9a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04040b1e40 con 0x7f04040d1910 2024-03-06T07:27:07.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.050+0000 7f0403fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04041614d0 0x7f04041e7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.050+0000 7f0403fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04041614d0 0x7f04041e7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57106/0 (socket says 172.21.15.59:57106) 2024-03-06T07:27:07.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.050+0000 7f0403fff640 1 -- 172.21.15.59:0/786871377 learned_addr learned my addr 172.21.15.59:0/786871377 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:07.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f0402ffd640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04040d22e0 0x7f040415c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f0403fff640 1 -- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04040d1910 msgr2=0x7f040415daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f04037fe640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04040d1910 0x7f040415daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f0403fff640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04040d1910 0x7f040415daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f0403fff640 1 -- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04040d22e0 msgr2=0x7f040415c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f0403fff640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04040d22e0 0x7f040415c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f0403fff640 1 -- 172.21.15.59:0/786871377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040415c810 con 0x7f04041614d0 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f0402ffd640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04040d22e0 0x7f040415c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f0403fff640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04041614d0 0x7f04041e7af0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00c780 tx=0x7f03fc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f03ebfff640 1 -- 172.21.15.59:0/786871377 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc004b60 con 0x7f04041614d0 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f03ebfff640 1 -- 172.21.15.59:0/786871377 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03fc004d00 con 0x7f04041614d0 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f0409c9a640 1 -- 172.21.15.59:0/786871377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040415e090 con 0x7f04041614d0 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f03ebfff640 1 -- 172.21.15.59:0/786871377 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc00aa90 con 0x7f04041614d0 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.051+0000 7f0409c9a640 1 -- 172.21.15.59:0/786871377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f040415e650 con 0x7f04041614d0 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.052+0000 7f0409c9a640 1 -- 172.21.15.59:0/786871377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04040ce170 con 0x7f04041614d0 2024-03-06T07:27:07.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.053+0000 7f03ebfff640 1 -- 172.21.15.59:0/786871377 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03fc00ac30 con 0x7f04041614d0 2024-03-06T07:27:07.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.053+0000 7f03ebfff640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03f006b4d0 0x7f03f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.053+0000 7f03ebfff640 1 -- 172.21.15.59:0/786871377 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f03fc08a260 con 0x7f04041614d0 2024-03-06T07:27:07.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.053+0000 7f04037fe640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03f006b4d0 0x7f03f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.054+0000 7f04037fe640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03f006b4d0 0x7f03f006d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f03f400b840 tx=0x7f03f4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.058+0000 7f03ebfff640 1 -- 172.21.15.59:0/786871377 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03fc020070 con 0x7f04041614d0 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.243+0000 7f0409c9a640 1 -- 172.21.15.59:0/786871377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0404068ff0 con 0x7f04041614d0 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.243+0000 7f03ebfff640 1 -- 172.21.15.59:0/786871377 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f040415c810 con 0x7f04041614d0 2024-03-06T07:27:07.245 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:27:07.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 -- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03f006b4d0 msgr2=0x7f03f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03f006b4d0 0x7f03f006d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f03f400b840 tx=0x7f03f4008040 comp rx=0 tx=0).stop 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 -- 172.21.15.59:0/786871377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04041614d0 msgr2=0x7f04041e7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04041614d0 0x7f04041e7af0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00c780 tx=0x7f03fc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 -- 172.21.15.59:0/786871377 shutdown_connections 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03f006b4d0 0x7f03f006d990 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04041614d0 0x7f04041e7af0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04040d22e0 0x7f040415c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 --2- 172.21.15.59:0/786871377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04040d1910 0x7f040415daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 -- 172.21.15.59:0/786871377 >> 172.21.15.59:0/786871377 conn(0x7f04040d3d40 msgr2=0x7f040406d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 -- 172.21.15.59:0/786871377 shutdown_connections 2024-03-06T07:27:07.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.245+0000 7f0409c9a640 1 -- 172.21.15.59:0/786871377 wait complete. 2024-03-06T07:27:07.258 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:27:07.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.359+0000 7f9951f3e640 1 Processor -- start 2024-03-06T07:27:07.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.359+0000 7f9951f3e640 1 -- start start 2024-03-06T07:27:07.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f9951f3e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f994c0d2920 0x7f994c0d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f9951f3e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f994c0d0650 0x7f994c0d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f9951f3e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c15e470 con 0x7f994c15e090 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f9951f3e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0d0fe0 con 0x7f994c0d2920 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f9951f3e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0d18f0 con 0x7f994c0d0650 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f994affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f994c0d2920 0x7f994c0d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f994b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f994c15e090 0x7f994c0b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58752/0 (socket says 172.21.15.59:58752) 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f994b7fe640 1 -- 172.21.15.59:0/1172369745 learned_addr learned my addr 172.21.15.59:0/1172369745 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f994affd640 1 -- 172.21.15.59:0/1172369745 >> v2:172.21.15.164:3300/0 conn(0x7f994c0d0650 msgr2=0x7f994c0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f994bfff640 1 --2- 172.21.15.59:0/1172369745 >> v2:172.21.15.164:3300/0 conn(0x7f994c0d0650 0x7f994c0d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f994affd640 1 --2- 172.21.15.59:0/1172369745 >> v2:172.21.15.164:3300/0 conn(0x7f994c0d0650 0x7f994c0d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f994affd640 1 -- 172.21.15.59:0/1172369745 >> v1:172.21.15.59:6789/0 conn(0x7f994c15e090 legacy=0x7f994c0b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f994a7fc640 1 -- 172.21.15.59:0/1172369745 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407151717 0 0) 0x7f994c15e470 con 0x7f994c15e090 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f994affd640 1 -- 172.21.15.59:0/1172369745 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0b3cf0 con 0x7f994c0d2920 2024-03-06T07:27:07.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.360+0000 7f994bfff640 1 --2- 172.21.15.59:0/1172369745 >> v2:172.21.15.164:3300/0 conn(0x7f994c0d0650 0x7f994c0d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.361+0000 7f994affd640 1 --2- 172.21.15.59:0/1172369745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994c0d2920 0x7f994c0d47a0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f994c08c550 tx=0x7f993c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2553734bfe1fe18e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.361+0000 7f994a7fc640 1 -- 172.21.15.59:0/1172369745 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c035020 con 0x7f994c0d2920 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.361+0000 7f994a7fc640 1 -- 172.21.15.59:0/1172369745 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c031830 con 0x7f994c0d2920 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.361+0000 7f994a7fc640 1 -- 172.21.15.59:0/1172369745 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c031c00 con 0x7f994c0d2920 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.361+0000 7f9951f3e640 1 -- 172.21.15.59:0/1172369745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994c0d2920 msgr2=0x7f994c0d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.361+0000 7f9951f3e640 1 --2- 172.21.15.59:0/1172369745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994c0d2920 0x7f994c0d47a0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f994c08c550 tx=0x7f993c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.362+0000 7f9951f3e640 1 -- 172.21.15.59:0/1172369745 shutdown_connections 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.362+0000 7f9951f3e640 1 --2- 172.21.15.59:0/1172369745 >> v2:172.21.15.164:3300/0 conn(0x7f994c0d0650 0x7f994c0d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.362+0000 7f9951f3e640 1 --2- 172.21.15.59:0/1172369745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994c0d2920 0x7f994c0d47a0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.362+0000 7f9951f3e640 1 -- 172.21.15.59:0/1172369745 >> 172.21.15.59:0/1172369745 conn(0x7f994c0d3c80 msgr2=0x7f994c0d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.362+0000 7f9951f3e640 1 -- 172.21.15.59:0/1172369745 shutdown_connections 2024-03-06T07:27:07.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.362+0000 7f9951f3e640 1 -- 172.21.15.59:0/1172369745 wait complete. 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.363+0000 7f9951f3e640 1 Processor -- start 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.363+0000 7f9951f3e640 1 -- start start 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f9951f3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f994c0d0650 0x7f994c150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f9951f3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994c0d2920 0x7f994c0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f9951f3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f994c15e090 0x7f994c0b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f9951f3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0cee30 con 0x7f994c0d2920 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f9951f3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0ceb50 con 0x7f994c0d0650 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f9951f3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0b2df0 con 0x7f994c15e090 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f994affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994c0d2920 0x7f994c0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f994affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994c0d2920 0x7f994c0cb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57136/0 (socket says 172.21.15.59:57136) 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f994affd640 1 -- 172.21.15.59:0/219013370 learned_addr learned my addr 172.21.15.59:0/219013370 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:07.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f994b7fe640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f994c0d0650 0x7f994c150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f994bfff640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f994c15e090 0x7f994c0b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f994affd640 1 -- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f994c15e090 msgr2=0x7f994c0b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f994affd640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f994c15e090 0x7f994c0b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f994affd640 1 -- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f994c0d0650 msgr2=0x7f994c150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f994affd640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f994c0d0650 0x7f994c150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f994affd640 1 -- 172.21.15.59:0/219013370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0cb620 con 0x7f994c0d2920 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f994affd640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994c0d2920 0x7f994c0cb0e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f993c009890 tx=0x7f993c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f9933fff640 1 -- 172.21.15.59:0/219013370 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c040070 con 0x7f994c0d2920 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f9933fff640 1 -- 172.21.15.59:0/219013370 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c039a20 con 0x7f994c0d2920 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f9951f3e640 1 -- 172.21.15.59:0/219013370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c0b6a40 con 0x7f994c0d2920 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f9933fff640 1 -- 172.21.15.59:0/219013370 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c035040 con 0x7f994c0d2920 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.364+0000 7f9951f3e640 1 -- 172.21.15.59:0/219013370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c0b7000 con 0x7f994c0d2920 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.365+0000 7f9951f3e640 1 -- 172.21.15.59:0/219013370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c04f330 con 0x7f994c0d2920 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.366+0000 7f9933fff640 1 -- 172.21.15.59:0/219013370 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f993c038020 con 0x7f994c0d2920 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.366+0000 7f9933fff640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993806b4d0 0x7f993806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.366+0000 7f9933fff640 1 -- 172.21.15.59:0/219013370 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f993c0aba70 con 0x7f994c0d2920 2024-03-06T07:27:07.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.366+0000 7f994b7fe640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993806b4d0 0x7f993806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.367+0000 7f994b7fe640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993806b4d0 0x7f993806d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9940000f60 tx=0x7f9940007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.370+0000 7f9933fff640 1 -- 172.21.15.59:0/219013370 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f993c07aae0 con 0x7f994c0d2920 2024-03-06T07:27:07.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:07.622+0000 7f9951f3e640 1 -- 172.21.15.59:0/219013370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f994c0d1230 con 0x7f994c0d2920 2024-03-06T07:27:08.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.616+0000 7f9933fff640 1 -- 172.21.15.59:0/219013370 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v301) v1 ==== 145+0+0 (secure 0 0 0) 0x7f993c07a3e0 con 0x7f994c0d2920 2024-03-06T07:27:08.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 -- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993806b4d0 msgr2=0x7f993806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993806b4d0 0x7f993806d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9940000f60 tx=0x7f9940007d50 comp rx=0 tx=0).stop 2024-03-06T07:27:08.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 -- 172.21.15.59:0/219013370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994c0d2920 msgr2=0x7f994c0cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994c0d2920 0x7f994c0cb0e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f993c009890 tx=0x7f993c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 -- 172.21.15.59:0/219013370 shutdown_connections 2024-03-06T07:27:08.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f993806b4d0 0x7f993806d990 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f994c15e090 0x7f994c0b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f994c0d2920 0x7f994c0cb0e0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 --2- 172.21.15.59:0/219013370 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f994c0d0650 0x7f994c150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 -- 172.21.15.59:0/219013370 >> 172.21.15.59:0/219013370 conn(0x7f994c0d3c80 msgr2=0x7f994c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 -- 172.21.15.59:0/219013370 shutdown_connections 2024-03-06T07:27:08.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.619+0000 7f9951f3e640 1 -- 172.21.15.59:0/219013370 wait complete. 2024-03-06T07:27:08.634 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:27:08.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.743+0000 7fcb6bd6b640 1 Processor -- start 2024-03-06T07:27:08.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.743+0000 7fcb6bd6b640 1 -- start start 2024-03-06T07:27:08.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.744+0000 7fcb6bd6b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcb640d4860 0x7fcb640d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.744+0000 7fcb6bd6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb640b0e30 0x7fcb640b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.744+0000 7fcb6bd6b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb641618b0 con 0x7fcb641614d0 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.744+0000 7fcb6bd6b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb640d4cf0 con 0x7fcb640d4860 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.744+0000 7fcb6bd6b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640cf400 con 0x7fcb640b0e30 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.744+0000 7fcb69306640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcb640d4860 0x7fcb640d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.744+0000 7fcb69306640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcb640d4860 0x7fcb640d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57140/0 (socket says 172.21.15.59:57140) 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.744+0000 7fcb69306640 1 -- 172.21.15.59:0/257191732 learned_addr learned my addr 172.21.15.59:0/257191732 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.745+0000 7fcb6a308640 1 --2- 172.21.15.59:0/257191732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb640b0e30 0x7fcb640b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.745+0000 7fcb69306640 1 -- 172.21.15.59:0/257191732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb640b0e30 msgr2=0x7fcb640b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.745+0000 7fcb69306640 1 --2- 172.21.15.59:0/257191732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb640b0e30 0x7fcb640b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.745+0000 7fcb69306640 1 -- 172.21.15.59:0/257191732 >> v1:172.21.15.164:6789/0 conn(0x7fcb641614d0 legacy=0x7fcb640d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.745+0000 7fcb69306640 1 -- 172.21.15.59:0/257191732 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb640cf6a0 con 0x7fcb640d4860 2024-03-06T07:27:08.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.745+0000 7fcb69306640 1 --2- 172.21.15.59:0/257191732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb640d4860 0x7fcb640d2020 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009870 tx=0x7fcb5802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5001ea161e5a9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.746+0000 7fcb68b05640 1 -- 172.21.15.59:0/257191732 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58034070 con 0x7fcb640d4860 2024-03-06T07:27:08.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.746+0000 7fcb68b05640 1 -- 172.21.15.59:0/257191732 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb58030840 con 0x7fcb640d4860 2024-03-06T07:27:08.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.746+0000 7fcb68b05640 1 -- 172.21.15.59:0/257191732 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58030c10 con 0x7fcb640d4860 2024-03-06T07:27:08.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.746+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/257191732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb640d4860 msgr2=0x7fcb640d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.746+0000 7fcb6bd6b640 1 --2- 172.21.15.59:0/257191732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb640d4860 0x7fcb640d2020 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009870 tx=0x7fcb5802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:08.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.746+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/257191732 shutdown_connections 2024-03-06T07:27:08.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.746+0000 7fcb6bd6b640 1 --2- 172.21.15.59:0/257191732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb640b0e30 0x7fcb640b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.746+0000 7fcb6bd6b640 1 --2- 172.21.15.59:0/257191732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb640d4860 0x7fcb640d2020 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.746+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/257191732 >> 172.21.15.59:0/257191732 conn(0x7fcb640d3d40 msgr2=0x7fcb640d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.746+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/257191732 shutdown_connections 2024-03-06T07:27:08.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.746+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/257191732 wait complete. 2024-03-06T07:27:08.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.747+0000 7fcb6bd6b640 1 Processor -- start 2024-03-06T07:27:08.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.747+0000 7fcb6bd6b640 1 -- start start 2024-03-06T07:27:08.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.748+0000 7fcb6bd6b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb640b0e30 0x7fcb6414aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.748+0000 7fcb6bd6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb640d4860 0x7fcb6414dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.748+0000 7fcb6bd6b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb641614d0 0x7fcb640cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.748+0000 7fcb6bd6b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640b0910 con 0x7fcb641614d0 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.748+0000 7fcb6bd6b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640cfc60 con 0x7fcb640b0e30 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.748+0000 7fcb6bd6b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640d2560 con 0x7fcb640d4860 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.748+0000 7fcb6a308640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb641614d0 0x7fcb640cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.748+0000 7fcb6a308640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb641614d0 0x7fcb640cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57156/0 (socket says 172.21.15.59:57156) 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.748+0000 7fcb6a308640 1 -- 172.21.15.59:0/1505148644 learned_addr learned my addr 172.21.15.59:0/1505148644 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.748+0000 7fcb69b07640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb640b0e30 0x7fcb6414aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.748+0000 7fcb69306640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb640d4860 0x7fcb6414dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb6a308640 1 -- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb640d4860 msgr2=0x7fcb6414dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb6a308640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb640d4860 0x7fcb6414dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb6a308640 1 -- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb640b0e30 msgr2=0x7fcb6414aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb6a308640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb640b0e30 0x7fcb6414aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb6a308640 1 -- 172.21.15.59:0/1505148644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6414e270 con 0x7fcb641614d0 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb69b07640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb640b0e30 0x7fcb6414aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb69306640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb640d4860 0x7fcb6414dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb6a308640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb641614d0 0x7fcb640cfea0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00d800 tx=0x7fcb5c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb627fc640 1 -- 172.21.15.59:0/1505148644 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c00b6c0 con 0x7fcb641614d0 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb627fc640 1 -- 172.21.15.59:0/1505148644 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c019070 con 0x7fcb641614d0 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/1505148644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb640b8570 con 0x7fcb641614d0 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb627fc640 1 -- 172.21.15.59:0/1505148644 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c00bac0 con 0x7fcb641614d0 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.749+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/1505148644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb640b8b30 con 0x7fcb641614d0 2024-03-06T07:27:08.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.750+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/1505148644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb64068ff0 con 0x7fcb641614d0 2024-03-06T07:27:08.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.750+0000 7fcb627fc640 1 -- 172.21.15.59:0/1505148644 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb5c00bc60 con 0x7fcb641614d0 2024-03-06T07:27:08.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.751+0000 7fcb627fc640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb5006b4d0 0x7fcb5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.751+0000 7fcb627fc640 1 -- 172.21.15.59:0/1505148644 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fcb5c08a580 con 0x7fcb641614d0 2024-03-06T07:27:08.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.751+0000 7fcb69b07640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb5006b4d0 0x7fcb5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.752+0000 7fcb69b07640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb5006b4d0 0x7fcb5006d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcb54001cd0 tx=0x7fcb54009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.755+0000 7fcb627fc640 1 -- 172.21.15.59:0/1505148644 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb6414e270 con 0x7fcb641614d0 2024-03-06T07:27:08.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.936+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/1505148644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcb640d4250 con 0x7fcb641614d0 2024-03-06T07:27:08.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.941+0000 7fcb627fc640 1 -- 172.21.15.59:0/1505148644 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcb5c059560 con 0x7fcb641614d0 2024-03-06T07:27:08.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb5006b4d0 msgr2=0x7fcb5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb5006b4d0 0x7fcb5006d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcb54001cd0 tx=0x7fcb54009040 comp rx=0 tx=0).stop 2024-03-06T07:27:08.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/1505148644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb641614d0 msgr2=0x7fcb640cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb641614d0 0x7fcb640cfea0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00d800 tx=0x7fcb5c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/1505148644 shutdown_connections 2024-03-06T07:27:08.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb5006b4d0 0x7fcb5006d990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb641614d0 0x7fcb640cfea0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb640d4860 0x7fcb6414dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 --2- 172.21.15.59:0/1505148644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb640b0e30 0x7fcb6414aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/1505148644 >> 172.21.15.59:0/1505148644 conn(0x7fcb640d3d40 msgr2=0x7fcb640b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/1505148644 shutdown_connections 2024-03-06T07:27:08.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:08.943+0000 7fcb6bd6b640 1 -- 172.21.15.59:0/1505148644 wait complete. 2024-03-06T07:27:08.956 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:27:08.956 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:08.996 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:27:09.039 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:27:09.039 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:27:09.040 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:09.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:27:09.054 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:27:09.054 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:09.097 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:09.097 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:27:09.098 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:09.112 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:27:09.112 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:27:09.112 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:09.154 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:27:09.155 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:27:09.155 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:09.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.254+0000 7f28e3bf7640 1 Processor -- start 2024-03-06T07:27:09.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.254+0000 7f28e3bf7640 1 -- start start 2024-03-06T07:27:09.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e3bf7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f28dc162020 0x7f28dc0d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e3bf7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f28dc162400 0x7f28dc0cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e3bf7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28dc0b1070 con 0x7f28dc162a00 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e3bf7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28dc0ce9d0 con 0x7f28dc162400 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e3bf7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28dc0d2da0 con 0x7f28dc162020 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e2194640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f28dc162400 0x7f28dc0cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e1192640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f28dc162a00 0x7f28dc0b0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58766/0 (socket says 172.21.15.59:58766) 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e1192640 1 -- 172.21.15.59:0/224058033 learned_addr learned my addr 172.21.15.59:0/224058033 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e1993640 1 --2- 172.21.15.59:0/224058033 >> v2:172.21.15.164:3300/0 conn(0x7f28dc162020 0x7f28dc0d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e2194640 1 -- 172.21.15.59:0/224058033 >> v2:172.21.15.164:3300/0 conn(0x7f28dc162020 msgr2=0x7f28dc0d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e2194640 1 --2- 172.21.15.59:0/224058033 >> v2:172.21.15.164:3300/0 conn(0x7f28dc162020 0x7f28dc0d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.255+0000 7f28e2194640 1 -- 172.21.15.59:0/224058033 >> v1:172.21.15.59:6789/0 conn(0x7f28dc162a00 legacy=0x7f28dc0b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.256+0000 7f28e2194640 1 -- 172.21.15.59:0/224058033 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc0d2500 con 0x7f28dc162400 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.256+0000 7f28e2194640 1 --2- 172.21.15.59:0/224058033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28dc162400 0x7f28dc0cf400 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f28dc0736c0 tx=0x7f28d402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12bf46ae4c760ed3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.256+0000 7f28e0991640 1 -- 172.21.15.59:0/224058033 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4034070 con 0x7f28dc162400 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.256+0000 7f28e0991640 1 -- 172.21.15.59:0/224058033 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d4030610 con 0x7f28dc162400 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.256+0000 7f28e0991640 1 -- 172.21.15.59:0/224058033 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d40309e0 con 0x7f28dc162400 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.257+0000 7f28e3bf7640 1 -- 172.21.15.59:0/224058033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28dc162400 msgr2=0x7f28dc0cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.257+0000 7f28e3bf7640 1 --2- 172.21.15.59:0/224058033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28dc162400 0x7f28dc0cf400 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f28dc0736c0 tx=0x7f28d402b960 comp rx=0 tx=0).stop 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.257+0000 7f28e3bf7640 1 -- 172.21.15.59:0/224058033 shutdown_connections 2024-03-06T07:27:09.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.257+0000 7f28e3bf7640 1 --2- 172.21.15.59:0/224058033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28dc162400 0x7f28dc0cf400 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.257+0000 7f28e3bf7640 1 --2- 172.21.15.59:0/224058033 >> v2:172.21.15.164:3300/0 conn(0x7f28dc162020 0x7f28dc0d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.257+0000 7f28e3bf7640 1 -- 172.21.15.59:0/224058033 >> 172.21.15.59:0/224058033 conn(0x7f28dc0d3d10 msgr2=0x7f28dc0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.257+0000 7f28e3bf7640 1 -- 172.21.15.59:0/224058033 shutdown_connections 2024-03-06T07:27:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.257+0000 7f28e3bf7640 1 -- 172.21.15.59:0/224058033 wait complete. 2024-03-06T07:27:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.258+0000 7f28e3bf7640 1 Processor -- start 2024-03-06T07:27:09.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.258+0000 7f28e3bf7640 1 -- start start 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e3bf7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28dc162020 0x7f28dc0b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e3bf7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28dc162400 0x7f28dc153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e3bf7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28dc162a00 0x7f28dc0cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e3bf7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc0cec00 con 0x7f28dc162400 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e3bf7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc0d2ac0 con 0x7f28dc162a00 2024-03-06T07:27:09.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e3bf7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc0cf940 con 0x7f28dc162020 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e1192640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28dc162400 0x7f28dc153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e1192640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28dc162400 0x7f28dc153070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57168/0 (socket says 172.21.15.59:57168) 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e1192640 1 -- 172.21.15.59:0/1911238914 learned_addr learned my addr 172.21.15.59:0/1911238914 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e1993640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28dc162020 0x7f28dc0b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e2194640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28dc162a00 0x7f28dc0cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e1192640 1 -- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28dc162020 msgr2=0x7f28dc0b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e1192640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28dc162020 0x7f28dc0b4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e1192640 1 -- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28dc162a00 msgr2=0x7f28dc0cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e1192640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28dc162a00 0x7f28dc0cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e1192640 1 -- 172.21.15.59:0/1911238914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc0b45c0 con 0x7f28dc162400 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e1993640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28dc162020 0x7f28dc0b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e2194640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28dc162a00 0x7f28dc0cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.259+0000 7f28e1192640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28dc162400 0x7f28dc153070 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f28d000c440 tx=0x7f28d000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.260+0000 7f28da7fc640 1 -- 172.21.15.59:0/1911238914 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d0017070 con 0x7f28dc162400 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.260+0000 7f28da7fc640 1 -- 172.21.15.59:0/1911238914 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d0005ce0 con 0x7f28dc162400 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.260+0000 7f28e3bf7640 1 -- 172.21.15.59:0/1911238914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28dc0b2fe0 con 0x7f28dc162400 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.260+0000 7f28da7fc640 1 -- 172.21.15.59:0/1911238914 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d00072f0 con 0x7f28dc162400 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.260+0000 7f28e3bf7640 1 -- 172.21.15.59:0/1911238914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28dc0b3880 con 0x7f28dc162400 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.261+0000 7f28e3bf7640 1 -- 172.21.15.59:0/1911238914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28dc0b33d0 con 0x7f28dc162400 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.261+0000 7f28da7fc640 1 -- 172.21.15.59:0/1911238914 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f28d001a070 con 0x7f28dc162400 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.261+0000 7f28da7fc640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f28c806b500 0x7f28c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.261+0000 7f28da7fc640 1 -- 172.21.15.59:0/1911238914 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f28d00892b0 con 0x7f28dc162400 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.262+0000 7f28e1993640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f28c806b500 0x7f28c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.262+0000 7f28e1993640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f28c806b500 0x7f28c806d9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f28dc06c360 tx=0x7f28cc0039f0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.266+0000 7f28da7fc640 1 -- 172.21.15.59:0/1911238914 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28d00583b0 con 0x7f28dc162400 2024-03-06T07:27:09.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.485+0000 7f28e3bf7640 1 -- 172.21.15.59:0/1911238914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28dc04f330 con 0x7f28dc162400 2024-03-06T07:27:09.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.485+0000 7f28da7fc640 1 -- 172.21.15.59:0/1911238914 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f28d00890d0 con 0x7f28dc162400 2024-03-06T07:27:09.487 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:09.487 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":61,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/670844356","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":670844356},{"type":"v1","addr":"172.21.15.164:6837","nonce":670844356}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56},{"gid":6994,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/1573527474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1573527474},{"type":"v1","addr":"172.21.15.164:6835","nonce":1573527474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7632,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/1541508518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1541508518},{"type":"v1","addr":"172.21.15.59:6834","nonce":1541508518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7869,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/1260499663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":1260499663},{"type":"v1","addr":"172.21.15.59:6835","nonce":1260499663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[]} 2024-03-06T07:27:09.488 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 61 2024-03-06T07:27:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 -- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f28c806b500 msgr2=0x7f28c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f28c806b500 0x7f28c806d9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f28dc06c360 tx=0x7f28cc0039f0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 -- 172.21.15.59:0/1911238914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28dc162400 msgr2=0x7f28dc153070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28dc162400 0x7f28dc153070 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f28d000c440 tx=0x7f28d000c900 comp rx=0 tx=0).stop 2024-03-06T07:27:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 -- 172.21.15.59:0/1911238914 shutdown_connections 2024-03-06T07:27:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f28c806b500 0x7f28c806d9c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28dc162a00 0x7f28dc0cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28dc162400 0x7f28dc153070 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 --2- 172.21.15.59:0/1911238914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28dc162020 0x7f28dc0b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 -- 172.21.15.59:0/1911238914 >> 172.21.15.59:0/1911238914 conn(0x7f28dc0d3d10 msgr2=0x7f28dc0c7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 -- 172.21.15.59:0/1911238914 shutdown_connections 2024-03-06T07:27:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.487+0000 7f28e3bf7640 1 -- 172.21.15.59:0/1911238914 wait complete. 2024-03-06T07:27:09.500 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:09.500 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:27:09.500 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:09.501 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:09.501 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:09.501 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:27:09.501 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:09.501 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:09.501 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:09.501 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:27:09.502 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:09.502 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:09.502 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:09.502 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:27:09.502 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:09.502 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:09.509 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:09.510 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:09.510 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:09.510 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:09.513 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:09.513 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:09.514 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:09.514 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:09.552 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:09.552 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:09.553 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:09.553 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:09.556 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:09.556 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:09.557 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:09.557 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:09.557 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:27:09.628 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:27:09.628 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:27:09.641 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:27:09.641 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:27:09.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.662+0000 7f891e58c640 1 Processor -- start 2024-03-06T07:27:09.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.662+0000 7f891e58c640 1 -- start start 2024-03-06T07:27:09.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f891e58c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f89180b4c00 0x7f89180b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f891e58c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f891815ba50 0x7f891815bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f891e58c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f891815d510 con 0x7f89180b28a0 2024-03-06T07:27:09.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f891e58c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89180b3420 con 0x7f89180b4c00 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f891e58c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f891815e0b0 con 0x7f891815ba50 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f8917fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f89180b4c00 0x7f89180b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f8917fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f89180b4c00 0x7f89180b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57212/0 (socket says 172.21.15.59:57212) 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f8917fff640 1 -- 172.21.15.59:0/4132568393 learned_addr learned my addr 172.21.15.59:0/4132568393 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f891cb29640 1 --2- 172.21.15.59:0/4132568393 >> v2:172.21.15.164:3300/0 conn(0x7f891815ba50 0x7f891815bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f8916ffd640 1 -- 172.21.15.59:0/4132568393 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173521403 0 0) 0x7f891815d510 con 0x7f89180b28a0 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f8917fff640 1 -- 172.21.15.59:0/4132568393 >> v2:172.21.15.164:3300/0 conn(0x7f891815ba50 msgr2=0x7f891815bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f8917fff640 1 --2- 172.21.15.59:0/4132568393 >> v2:172.21.15.164:3300/0 conn(0x7f891815ba50 0x7f891815bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f8917fff640 1 -- 172.21.15.59:0/4132568393 >> v1:172.21.15.164:6789/0 conn(0x7f89180b28a0 legacy=0x7f89180b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f8917fff640 1 -- 172.21.15.59:0/4132568393 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891815d780 con 0x7f89180b4c00 2024-03-06T07:27:09.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.663+0000 7f891cb29640 1 --2- 172.21.15.59:0/4132568393 >> v2:172.21.15.164:3300/0 conn(0x7f891815ba50 0x7f891815bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.664+0000 7f8917fff640 1 --2- 172.21.15.59:0/4132568393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89180b4c00 0x7f89180b4fe0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f891806b930 tx=0x7f890c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c323a15ba3e1cf9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.664+0000 7f8916ffd640 1 -- 172.21.15.59:0/4132568393 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c02f850 con 0x7f89180b4c00 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.664+0000 7f8916ffd640 1 -- 172.21.15.59:0/4132568393 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890c02f9f0 con 0x7f89180b4c00 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.664+0000 7f8916ffd640 1 -- 172.21.15.59:0/4132568393 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c02fdc0 con 0x7f89180b4c00 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.665+0000 7f891e58c640 1 -- 172.21.15.59:0/4132568393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89180b4c00 msgr2=0x7f89180b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.665+0000 7f891e58c640 1 --2- 172.21.15.59:0/4132568393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89180b4c00 0x7f89180b4fe0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f891806b930 tx=0x7f890c02ba30 comp rx=0 tx=0).stop 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.665+0000 7f891e58c640 1 -- 172.21.15.59:0/4132568393 shutdown_connections 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.665+0000 7f891e58c640 1 --2- 172.21.15.59:0/4132568393 >> v2:172.21.15.164:3300/0 conn(0x7f891815ba50 0x7f891815bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.665+0000 7f891e58c640 1 --2- 172.21.15.59:0/4132568393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89180b4c00 0x7f89180b4fe0 secure :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f891806b930 tx=0x7f890c02ba30 comp rx=0 tx=0).stop 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.665+0000 7f891e58c640 1 -- 172.21.15.59:0/4132568393 >> 172.21.15.59:0/4132568393 conn(0x7f891809a9f0 msgr2=0x7f891809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.665+0000 7f891e58c640 1 -- 172.21.15.59:0/4132568393 shutdown_connections 2024-03-06T07:27:09.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.665+0000 7f891e58c640 1 -- 172.21.15.59:0/4132568393 wait complete. 2024-03-06T07:27:09.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.666+0000 7f891e58c640 1 Processor -- start 2024-03-06T07:27:09.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.666+0000 7f891e58c640 1 -- start start 2024-03-06T07:27:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891e58c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89180b28a0 0x7f8918151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891e58c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f891815ba50 0x7f89180b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891e58c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89180b1fd0 0x7f8918149620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891e58c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891815f550 con 0x7f89180b1fd0 2024-03-06T07:27:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891e58c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891815dd40 con 0x7f891815ba50 2024-03-06T07:27:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891e58c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891815f6d0 con 0x7f89180b28a0 2024-03-06T07:27:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891cb29640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89180b1fd0 0x7f8918149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891cb29640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89180b1fd0 0x7f8918149620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57220/0 (socket says 172.21.15.59:57220) 2024-03-06T07:27:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891cb29640 1 -- 172.21.15.59:0/136052560 learned_addr learned my addr 172.21.15.59:0/136052560 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f8917fff640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89180b28a0 0x7f8918151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f89177fe640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f891815ba50 0x7f89180b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891cb29640 1 -- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89180b28a0 msgr2=0x7f8918151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891cb29640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89180b28a0 0x7f8918151280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891cb29640 1 -- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f891815ba50 msgr2=0x7f89180b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891cb29640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f891815ba50 0x7f89180b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f891cb29640 1 -- 172.21.15.59:0/136052560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89180b0190 con 0x7f89180b1fd0 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.667+0000 7f89177fe640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f891815ba50 0x7f89180b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.668+0000 7f891cb29640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89180b1fd0 0x7f8918149620 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f891000efc0 tx=0x7f891000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.668+0000 7f8914ff9640 1 -- 172.21.15.59:0/136052560 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8910018030 con 0x7f89180b1fd0 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.668+0000 7f8914ff9640 1 -- 172.21.15.59:0/136052560 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891000cdf0 con 0x7f89180b1fd0 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.668+0000 7f891e58c640 1 -- 172.21.15.59:0/136052560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89180b0470 con 0x7f89180b1fd0 2024-03-06T07:27:09.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.668+0000 7f8914ff9640 1 -- 172.21.15.59:0/136052560 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8910003bb0 con 0x7f89180b1fd0 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.668+0000 7f891e58c640 1 -- 172.21.15.59:0/136052560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89180b0a30 con 0x7f89180b1fd0 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.669+0000 7f891e58c640 1 -- 172.21.15.59:0/136052560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f891804f330 con 0x7f89180b1fd0 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.669+0000 7f8914ff9640 1 -- 172.21.15.59:0/136052560 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f891001b070 con 0x7f89180b1fd0 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.670+0000 7f8914ff9640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f890406b4d0 0x7f890406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.670+0000 7f8914ff9640 1 -- 172.21.15.59:0/136052560 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f891008a9d0 con 0x7f89180b1fd0 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.670+0000 7f8917fff640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f890406b4d0 0x7f890406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.670+0000 7f8917fff640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f890406b4d0 0x7f890406d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f890c036000 tx=0x7f890c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.674+0000 7f8914ff9640 1 -- 172.21.15.59:0/136052560 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8910010040 con 0x7f89180b1fd0 2024-03-06T07:27:09.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:09.866+0000 7f891e58c640 1 -- 172.21.15.59:0/136052560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8918158bd0 con 0x7f89180b1fd0 2024-03-06T07:27:10.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.964+0000 7f8914ff9640 1 -- 172.21.15.59:0/136052560 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v104) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8910059a40 con 0x7f89180b1fd0 2024-03-06T07:27:10.966 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-03-06T07:27:10.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 -- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f890406b4d0 msgr2=0x7f890406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f890406b4d0 0x7f890406d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f890c036000 tx=0x7f890c008030 comp rx=0 tx=0).stop 2024-03-06T07:27:10.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 -- 172.21.15.59:0/136052560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89180b1fd0 msgr2=0x7f8918149620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89180b1fd0 0x7f8918149620 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f891000efc0 tx=0x7f891000c490 comp rx=0 tx=0).stop 2024-03-06T07:27:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 -- 172.21.15.59:0/136052560 shutdown_connections 2024-03-06T07:27:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f890406b4d0 0x7f890406d990 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89180b1fd0 0x7f8918149620 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f891815ba50 0x7f89180b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 --2- 172.21.15.59:0/136052560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89180b28a0 0x7f8918151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 -- 172.21.15.59:0/136052560 >> 172.21.15.59:0/136052560 conn(0x7f891809a9f0 msgr2=0x7f891806ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 -- 172.21.15.59:0/136052560 shutdown_connections 2024-03-06T07:27:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:10.967+0000 7f891e58c640 1 -- 172.21.15.59:0/136052560 wait complete. 2024-03-06T07:27:10.983 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:27:11.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.090+0000 7f30fe239640 1 Processor -- start 2024-03-06T07:27:11.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.090+0000 7f30fe239640 1 -- start start 2024-03-06T07:27:11.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.090+0000 7f30fe239640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30f80cce40 0x7f30f815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.090+0000 7f30fe239640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f30f80cd7a0 0x7f30f80cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.090+0000 7f30fe239640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f8071f40 con 0x7f30f81515e0 2024-03-06T07:27:11.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.090+0000 7f30fe239640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f80cd2f0 con 0x7f30f80cce40 2024-03-06T07:27:11.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.090+0000 7f30fe239640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f8070a70 con 0x7f30f80cd7a0 2024-03-06T07:27:11.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.091+0000 7f30f6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30f80cce40 0x7f30f815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.091+0000 7f30f6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30f80cce40 0x7f30f815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57236/0 (socket says 172.21.15.59:57236) 2024-03-06T07:27:11.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.091+0000 7f30f6ffd640 1 -- 172.21.15.59:0/710926895 learned_addr learned my addr 172.21.15.59:0/710926895 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:11.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.091+0000 7f30f7fff640 1 --2- 172.21.15.59:0/710926895 >> v2:172.21.15.164:3300/0 conn(0x7f30f80cd7a0 0x7f30f80cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.091+0000 7f30f6ffd640 1 -- 172.21.15.59:0/710926895 >> v2:172.21.15.164:3300/0 conn(0x7f30f80cd7a0 msgr2=0x7f30f80cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.091+0000 7f30f6ffd640 1 --2- 172.21.15.59:0/710926895 >> v2:172.21.15.164:3300/0 conn(0x7f30f80cd7a0 0x7f30f80cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.091+0000 7f30f6ffd640 1 -- 172.21.15.59:0/710926895 >> v1:172.21.15.164:6789/0 conn(0x7f30f81515e0 legacy=0x7f30f8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.091+0000 7f30f6ffd640 1 -- 172.21.15.59:0/710926895 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f8070d50 con 0x7f30f80cce40 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30f6ffd640 1 --2- 172.21.15.59:0/710926895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30f80cce40 0x7f30f815b430 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f30f80cc110 tx=0x7f30ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e677dbd3f42b21e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30f67fc640 1 -- 172.21.15.59:0/710926895 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec035020 con 0x7f30f80cce40 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30f67fc640 1 -- 172.21.15.59:0/710926895 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ec031640 con 0x7f30f80cce40 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30f67fc640 1 -- 172.21.15.59:0/710926895 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec02d040 con 0x7f30f80cce40 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30fe239640 1 -- 172.21.15.59:0/710926895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30f80cce40 msgr2=0x7f30f815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30fe239640 1 --2- 172.21.15.59:0/710926895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30f80cce40 0x7f30f815b430 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f30f80cc110 tx=0x7f30ec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30fe239640 1 -- 172.21.15.59:0/710926895 shutdown_connections 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30fe239640 1 --2- 172.21.15.59:0/710926895 >> v2:172.21.15.164:3300/0 conn(0x7f30f80cd7a0 0x7f30f80cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30fe239640 1 --2- 172.21.15.59:0/710926895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30f80cce40 0x7f30f815b430 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30fe239640 1 -- 172.21.15.59:0/710926895 >> 172.21.15.59:0/710926895 conn(0x7f30f80cf8a0 msgr2=0x7f30f80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30fe239640 1 -- 172.21.15.59:0/710926895 shutdown_connections 2024-03-06T07:27:11.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.092+0000 7f30fe239640 1 -- 172.21.15.59:0/710926895 wait complete. 2024-03-06T07:27:11.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.094+0000 7f30fe239640 1 Processor -- start 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.094+0000 7f30fe239640 1 -- start start 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.094+0000 7f30fe239640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30f80cce40 0x7f30f813a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.094+0000 7f30fe239640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30f80cd7a0 0x7f30f813df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.094+0000 7f30fe239640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30f81515e0 0x7f30f813f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.094+0000 7f30fe239640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f80ce0c0 con 0x7f30f81515e0 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.094+0000 7f30fe239640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f8071310 con 0x7f30f80cce40 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.094+0000 7f30fe239640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f80cc3a0 con 0x7f30f80cd7a0 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30f81515e0 0x7f30f813f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30f81515e0 0x7f30f813f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57244/0 (socket says 172.21.15.59:57244) 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f7fff640 1 -- 172.21.15.59:0/567692253 learned_addr learned my addr 172.21.15.59:0/567692253 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f6ffd640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30f80cd7a0 0x7f30f813df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f77fe640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30f80cce40 0x7f30f813a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f7fff640 1 -- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30f80cd7a0 msgr2=0x7f30f813df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f7fff640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30f80cd7a0 0x7f30f813df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f7fff640 1 -- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30f80cce40 msgr2=0x7f30f813a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f7fff640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30f80cce40 0x7f30f813a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f7fff640 1 -- 172.21.15.59:0/567692253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f813bf60 con 0x7f30f81515e0 2024-03-06T07:27:11.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f77fe640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30f80cce40 0x7f30f813a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:11.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30f7fff640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30f81515e0 0x7f30f813f870 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f30f0007e40 tx=0x7f30f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30dffff640 1 -- 172.21.15.59:0/567692253 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0014620 con 0x7f30f81515e0 2024-03-06T07:27:11.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30dffff640 1 -- 172.21.15.59:0/567692253 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f0019070 con 0x7f30f81515e0 2024-03-06T07:27:11.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30dffff640 1 -- 172.21.15.59:0/567692253 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0014a20 con 0x7f30f81515e0 2024-03-06T07:27:11.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30fe239640 1 -- 172.21.15.59:0/567692253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f813c240 con 0x7f30f81515e0 2024-03-06T07:27:11.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.095+0000 7f30fe239640 1 -- 172.21.15.59:0/567692253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f8151bc0 con 0x7f30f81515e0 2024-03-06T07:27:11.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.096+0000 7f30fe239640 1 -- 172.21.15.59:0/567692253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f8068840 con 0x7f30f81515e0 2024-03-06T07:27:11.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.097+0000 7f30dffff640 1 -- 172.21.15.59:0/567692253 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30f0014bc0 con 0x7f30f81515e0 2024-03-06T07:27:11.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.097+0000 7f30dffff640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30e406b4d0 0x7f30e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.097+0000 7f30dffff640 1 -- 172.21.15.59:0/567692253 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f30f008a710 con 0x7f30f81515e0 2024-03-06T07:27:11.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.098+0000 7f30f77fe640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30e406b4d0 0x7f30e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.098+0000 7f30f77fe640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30e406b4d0 0x7f30e406d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f30e8006fd0 tx=0x7f30e8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.102+0000 7f30dffff640 1 -- 172.21.15.59:0/567692253 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30f0059970 con 0x7f30f81515e0 2024-03-06T07:27:11.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.353+0000 7f30fe239640 1 -- 172.21.15.59:0/567692253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f30f813c760 con 0x7f30f81515e0 2024-03-06T07:27:11.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.354+0000 7f30dffff640 1 -- 172.21.15.59:0/567692253 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f30f813c760 con 0x7f30f81515e0 2024-03-06T07:27:11.356 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:11.356 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:27:11.356 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-03-06T07:27:11.356 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.356 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-03-06T07:27:11.356 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlODBZGBAAqV+TH2t2ZKyV9l13QEn6ew==", 2024-03-06T07:27:11.356 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.356 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:27:11.357 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:27:11.357 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:27:11.357 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.357 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.357 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.357 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.357 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-03-06T07:27:11.357 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlR1MuJRAAp6OAQ9gZemWtwI1dh3H2SA==", 2024-03-06T07:27:11.357 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlDXW2HxAAnKyuS1GW0wLO98/F9wdOAg==", 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehl5MO4LBAAKmt2dvKk7BtcCM4o7cSwgg==", 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.360 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.360 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.360 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-03-06T07:27:11.360 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBXGehlzdinFhAA0ACDAs9dtoY6EhUIveeIvA==", 2024-03-06T07:27:11.360 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.360 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:11.360 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:27:11.360 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.360 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.360 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.361 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.361 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-03-06T07:27:11.361 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGehlH92eFRAANQceHGuQyrbI6gP1m/Gakw==", 2024-03-06T07:27:11.361 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.361 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:11.361 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:27:11.361 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.361 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.361 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGehlMWpVFBAAbOG9nScbFlvKHXZMQJoQug==", 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGehlLCyWEhAAyfoW/FmU37QD2IohknBwWw==", 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBmGehldqv5GhAAWEloqqKgdpKPHjt7VKFodw==", 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGehlc0ENGRAA5V9Az+vxAR3OUwWpDP7w1A==", 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBsGehl5SYEFxAAPLyrHQV9SGguCV9f5VIkYg==", 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBvGehle3v0FBAA4xUPwX5couppHgi9VwSi2g==", 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA==", 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlqhlAOhAAQ4l3uI8MlcCH+NAIWo/UuA==", 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehltL/GNhAAZqSQ+UtwjI/PA95O1RGBNg==", 2024-03-06T07:27:11.367 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.368 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:27:11.370 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGehlltx5BBAAFa/+/w8Hd8lZegfIWtpJOA==", 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:27:11.371 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBSGehlMExaNhAAQ2FD7afktMhrTbMMAK1epQ==", 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-03-06T07:27:11.372 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlLSTzFBAACeLdHsCX9KJGEsq44aqTyw==", 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlAgqXEBAACCHZ9Lp2lEyF93xD4Ns9Nw==", 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:27:11.373 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.374 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:11.374 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-06T07:27:11.374 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:27:11.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 -- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30e406b4d0 msgr2=0x7f30e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30e406b4d0 0x7f30e406d990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f30e8006fd0 tx=0x7f30e8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:11.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 -- 172.21.15.59:0/567692253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30f81515e0 msgr2=0x7f30f813f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30f81515e0 0x7f30f813f870 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f30f0007e40 tx=0x7f30f000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 -- 172.21.15.59:0/567692253 shutdown_connections 2024-03-06T07:27:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30e406b4d0 0x7f30e406d990 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30f81515e0 0x7f30f813f870 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30f80cd7a0 0x7f30f813df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 --2- 172.21.15.59:0/567692253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30f80cce40 0x7f30f813a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 -- 172.21.15.59:0/567692253 >> 172.21.15.59:0/567692253 conn(0x7f30f80cf8a0 msgr2=0x7f30f814fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 -- 172.21.15.59:0/567692253 shutdown_connections 2024-03-06T07:27:11.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.356+0000 7f30fe239640 1 -- 172.21.15.59:0/567692253 wait complete. 2024-03-06T07:27:11.377 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:11.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.476+0000 7f24602f4640 1 Processor -- start 2024-03-06T07:27:11.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.476+0000 7f24602f4640 1 -- start start 2024-03-06T07:27:11.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f24602f4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f245813dc10 0x7f24580b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f24602f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 0x7f24580b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f24602f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f245815f580 con 0x7f245815e850 2024-03-06T07:27:11.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f24602f4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24580cf3f0 con 0x7f245813dc10 2024-03-06T07:27:11.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f24602f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245815b210 con 0x7f245815bf30 2024-03-06T07:27:11.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f245e090640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f245813dc10 0x7f24580b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f245e891640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f245815e850 0x7f245815eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58778/0 (socket says 172.21.15.59:58778) 2024-03-06T07:27:11.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f245e891640 1 -- 172.21.15.59:0/447348579 learned_addr learned my addr 172.21.15.59:0/447348579 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:11.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f245d88f640 1 --2- 172.21.15.59:0/447348579 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 0x7f24580b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f245e090640 1 -- 172.21.15.59:0/447348579 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 msgr2=0x7f24580b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f245e090640 1 --2- 172.21.15.59:0/447348579 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 0x7f24580b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f245e090640 1 -- 172.21.15.59:0/447348579 >> v1:172.21.15.59:6789/0 conn(0x7f245815e850 legacy=0x7f245815eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f245d08e640 1 -- 172.21.15.59:0/447348579 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264693617 0 0) 0x7f245815f580 con 0x7f245815e850 2024-03-06T07:27:11.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f245e090640 1 -- 172.21.15.59:0/447348579 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245815b450 con 0x7f245813dc10 2024-03-06T07:27:11.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.477+0000 7f245d88f640 1 --2- 172.21.15.59:0/447348579 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 0x7f24580b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:11.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.478+0000 7f245e090640 1 --2- 172.21.15.59:0/447348579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f245813dc10 0x7f24580b2950 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f24540097a0 tx=0x7f245402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a10c5585dd613f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.478+0000 7f245d08e640 1 -- 172.21.15.59:0/447348579 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454031780 con 0x7f245813dc10 2024-03-06T07:27:11.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.478+0000 7f245d08e640 1 -- 172.21.15.59:0/447348579 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2454037070 con 0x7f245813dc10 2024-03-06T07:27:11.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.478+0000 7f245d08e640 1 -- 172.21.15.59:0/447348579 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454031b40 con 0x7f245813dc10 2024-03-06T07:27:11.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.479+0000 7f24602f4640 1 -- 172.21.15.59:0/447348579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f245813dc10 msgr2=0x7f24580b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.479+0000 7f24602f4640 1 --2- 172.21.15.59:0/447348579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f245813dc10 0x7f24580b2950 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f24540097a0 tx=0x7f245402b820 comp rx=0 tx=0).stop 2024-03-06T07:27:11.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.479+0000 7f24602f4640 1 -- 172.21.15.59:0/447348579 shutdown_connections 2024-03-06T07:27:11.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.479+0000 7f24602f4640 1 --2- 172.21.15.59:0/447348579 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 0x7f24580b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.479+0000 7f24602f4640 1 --2- 172.21.15.59:0/447348579 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f245813dc10 0x7f24580b2950 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.479+0000 7f24602f4640 1 -- 172.21.15.59:0/447348579 >> 172.21.15.59:0/447348579 conn(0x7f24580cf8a0 msgr2=0x7f24580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.479+0000 7f24602f4640 1 -- 172.21.15.59:0/447348579 shutdown_connections 2024-03-06T07:27:11.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.479+0000 7f24602f4640 1 -- 172.21.15.59:0/447348579 wait complete. 2024-03-06T07:27:11.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.480+0000 7f24602f4640 1 Processor -- start 2024-03-06T07:27:11.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.480+0000 7f24602f4640 1 -- start start 2024-03-06T07:27:11.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f24602f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f245813dc10 0x7f2458154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f24602f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 0x7f2458157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f24602f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f245815e850 0x7f245806df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f24602f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24580cd7a0 con 0x7f245813dc10 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f24602f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245813dff0 con 0x7f245815e850 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f24602f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24580cd920 con 0x7f245815bf30 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245e090640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f245813dc10 0x7f2458154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245e090640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f245813dc10 0x7f2458154830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57262/0 (socket says 172.21.15.59:57262) 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245e090640 1 -- 172.21.15.59:0/1135063361 learned_addr learned my addr 172.21.15.59:0/1135063361 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245d88f640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 0x7f2458157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245e891640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f245815e850 0x7f245806df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245e090640 1 -- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 msgr2=0x7f2458157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245e090640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 0x7f2458157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245e090640 1 -- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f245815e850 msgr2=0x7f245806df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245e090640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f245815e850 0x7f245806df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245e090640 1 -- 172.21.15.59:0/1135063361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2458158830 con 0x7f245813dc10 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245d88f640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 0x7f2458157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f245e090640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f245813dc10 0x7f2458154830 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f2454034040 tx=0x7f24540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f244effd640 1 -- 172.21.15.59:0/1135063361 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454032820 con 0x7f245813dc10 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f244effd640 1 -- 172.21.15.59:0/1135063361 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24540329e0 con 0x7f245813dc10 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f244effd640 1 -- 172.21.15.59:0/1135063361 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454032dd0 con 0x7f245813dc10 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.481+0000 7f24602f4640 1 -- 172.21.15.59:0/1135063361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2458158b10 con 0x7f245813dc10 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.482+0000 7f24602f4640 1 -- 172.21.15.59:0/1135063361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24581f1e10 con 0x7f245813dc10 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.482+0000 7f24602f4640 1 -- 172.21.15.59:0/1135063361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f245804f330 con 0x7f245813dc10 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.483+0000 7f244effd640 1 -- 172.21.15.59:0/1135063361 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f245404e440 con 0x7f245813dc10 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.483+0000 7f244effd640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f243c06b4d0 0x7f243c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.484+0000 7f244effd640 1 -- 172.21.15.59:0/1135063361 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f24540abe50 con 0x7f245813dc10 2024-03-06T07:27:11.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.484+0000 7f245d88f640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f243c06b4d0 0x7f243c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.484+0000 7f245d88f640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f243c06b4d0 0x7f243c06d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2448006fd0 tx=0x7f2448008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.488+0000 7f244effd640 1 -- 172.21.15.59:0/1135063361 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2454040070 con 0x7f245813dc10 2024-03-06T07:27:11.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.672+0000 7f24602f4640 1 -- 172.21.15.59:0/1135063361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2458157c30 con 0x7f245813dc10 2024-03-06T07:27:11.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.672+0000 7f244effd640 1 -- 172.21.15.59:0/1135063361 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+12881 (secure 0 0 0) 0x7f245402f050 con 0x7f245813dc10 2024-03-06T07:27:11.674 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:11.674 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":104,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:27:10.963683+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:11.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 -- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f243c06b4d0 msgr2=0x7f243c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f243c06b4d0 0x7f243c06d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2448006fd0 tx=0x7f2448008040 comp rx=0 tx=0).stop 2024-03-06T07:27:11.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 -- 172.21.15.59:0/1135063361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f245813dc10 msgr2=0x7f2458154830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f245813dc10 0x7f2458154830 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f2454034040 tx=0x7f24540046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 -- 172.21.15.59:0/1135063361 shutdown_connections 2024-03-06T07:27:11.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f243c06b4d0 0x7f243c06d990 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f245815e850 0x7f245806df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f245815bf30 0x7f2458157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 --2- 172.21.15.59:0/1135063361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f245813dc10 0x7f2458154830 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 -- 172.21.15.59:0/1135063361 >> 172.21.15.59:0/1135063361 conn(0x7f24580cf8a0 msgr2=0x7f24581369c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 -- 172.21.15.59:0/1135063361 shutdown_connections 2024-03-06T07:27:11.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.674+0000 7f24602f4640 1 -- 172.21.15.59:0/1135063361 wait complete. 2024-03-06T07:27:11.687 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:11.688 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.790+0000 7ff04693e640 1 Processor -- start 2024-03-06T07:27:11.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.790+0000 7ff04693e640 1 -- start start 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.791+0000 7ff04693e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff04006fd10 0x7ff0400cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.791+0000 7ff04693e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0400cd4b0 con 0x7ff0400d0af0 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.791+0000 7ff04693e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0400cd790 con 0x7ff04015c050 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.791+0000 7ff04693e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0400cda70 con 0x7ff04006fd10 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.791+0000 7ff044edb640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff0400d0af0 0x7ff0400ccca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58792/0 (socket says 172.21.15.59:58792) 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.791+0000 7ff044edb640 1 -- 172.21.15.59:0/1629903198 learned_addr learned my addr 172.21.15.59:0/1629903198 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03f7fe640 1 --2- 172.21.15.59:0/1629903198 >> v2:172.21.15.164:3300/0 conn(0x7ff04006fd10 0x7ff0400cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250724617 0 0) 0x7ff0400cd4b0 con 0x7ff0400d0af0 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff028003520 con 0x7ff0400d0af0 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 28300483 0 0) 0x7ff028003520 con 0x7ff0400d0af0 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 >> v2:172.21.15.164:3300/0 conn(0x7ff04006fd10 msgr2=0x7ff0400cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 --2- 172.21.15.59:0/1629903198 >> v2:172.21.15.164:3300/0 conn(0x7ff04006fd10 0x7ff0400cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 >> v1:172.21.15.164:6789/0 conn(0x7ff04015c050 legacy=0x7ff04006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0400cdd50 con 0x7ff0400d0af0 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7ff0380019f0 con 0x7ff0400d0af0 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 >> v1:172.21.15.59:6789/0 conn(0x7ff0400d0af0 legacy=0x7ff0400ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0280039b0 0x7ff028003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff028004800 0x7ff028004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0280051c0 0x7ff02800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400cda70 con 0x7ff0280039b0 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02800c160 con 0x7ff0280051c0 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff02800c360 con 0x7ff028004800 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03ffff640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0280039b0 0x7ff028003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff044edb640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0280051c0 0x7ff02800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03f7fe640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff028004800 0x7ff028004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03ffff640 1 -- 172.21.15.59:0/1629903198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff028004800 msgr2=0x7ff028004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03ffff640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff028004800 0x7ff028004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03ffff640 1 -- 172.21.15.59:0/1629903198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0280051c0 msgr2=0x7ff02800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03ffff640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0280051c0 0x7ff02800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff03ffff640 1 -- 172.21.15.59:0/1629903198 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff028003520 con 0x7ff0280039b0 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff044edb640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0280051c0 0x7ff02800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.792+0000 7ff044edb640 1 -- 172.21.15.59:0/1629903198 reap_dead start 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.793+0000 7ff03ffff640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0280039b0 0x7ff028003dd0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7ff030001750 tx=0x7ff028005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45825a77cc30338 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.793+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030001370 con 0x7ff0280039b0 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.793+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0300052d0 con 0x7ff0280039b0 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.793+0000 7ff03effd640 1 -- 172.21.15.59:0/1629903198 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030003830 con 0x7ff0280039b0 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.794+0000 7ff04693e640 1 -- 172.21.15.59:0/1629903198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0280039b0 msgr2=0x7ff028003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.794+0000 7ff04693e640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0280039b0 0x7ff028003dd0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7ff030001750 tx=0x7ff028005830 comp rx=0 tx=0).stop 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.794+0000 7ff04693e640 1 -- 172.21.15.59:0/1629903198 shutdown_connections 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.794+0000 7ff04693e640 1 --2- 172.21.15.59:0/1629903198 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0280039b0 0x7ff028003dd0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.794+0000 7ff04693e640 1 -- 172.21.15.59:0/1629903198 >> 172.21.15.59:0/1629903198 conn(0x7ff0400b3800 msgr2=0x7ff0400b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.794+0000 7ff04693e640 1 -- 172.21.15.59:0/1629903198 shutdown_connections 2024-03-06T07:27:11.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.794+0000 7ff04693e640 1 -- 172.21.15.59:0/1629903198 wait complete. 2024-03-06T07:27:11.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.795+0000 7ff04693e640 1 Processor -- start 2024-03-06T07:27:11.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.795+0000 7ff04693e640 1 -- start start 2024-03-06T07:27:11.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff04693e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0280039b0 0x7ff0400b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff04693e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0401f3000 0x7ff0400afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff04693e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0401f3480 0x7ff0400aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff04693e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04006f990 con 0x7ff0401f3000 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff04693e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04006f810 con 0x7ff0280039b0 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff04693e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04006fb10 con 0x7ff0401f3480 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff03f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0401f3000 0x7ff0400afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff03f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0401f3000 0x7ff0400afa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57280/0 (socket says 172.21.15.59:57280) 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff03f7fe640 1 -- 172.21.15.59:0/1105852331 learned_addr learned my addr 172.21.15.59:0/1105852331 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff03ffff640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0280039b0 0x7ff0400b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff044edb640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0401f3480 0x7ff0400aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff03f7fe640 1 -- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0401f3480 msgr2=0x7ff0400aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff03f7fe640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0401f3480 0x7ff0400aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff03f7fe640 1 -- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0280039b0 msgr2=0x7ff0400b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff03f7fe640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0280039b0 0x7ff0400b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff03f7fe640 1 -- 172.21.15.59:0/1105852331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0400aec60 con 0x7ff0401f3000 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff03ffff640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0280039b0 0x7ff0400b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.796+0000 7ff03f7fe640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0401f3000 0x7ff0400afa20 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7ff0340116d0 tx=0x7ff034011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.797+0000 7ff03cff9640 1 -- 172.21.15.59:0/1105852331 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03400f9c0 con 0x7ff0401f3000 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.797+0000 7ff03cff9640 1 -- 172.21.15.59:0/1105852331 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03400fb60 con 0x7ff0401f3000 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.797+0000 7ff04693e640 1 -- 172.21.15.59:0/1105852331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0400aef40 con 0x7ff0401f3000 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.797+0000 7ff03cff9640 1 -- 172.21.15.59:0/1105852331 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034015460 con 0x7ff0401f3000 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.797+0000 7ff04693e640 1 -- 172.21.15.59:0/1105852331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04014f1f0 con 0x7ff0401f3000 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.798+0000 7ff04693e640 1 -- 172.21.15.59:0/1105852331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04009fe60 con 0x7ff0401f3000 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.798+0000 7ff03cff9640 1 -- 172.21.15.59:0/1105852331 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff034015600 con 0x7ff0401f3000 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.798+0000 7ff03cff9640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff02c06b4d0 0x7ff02c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.798+0000 7ff03cff9640 1 -- 172.21.15.59:0/1105852331 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff03408a720 con 0x7ff0401f3000 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.799+0000 7ff03ffff640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff02c06b4d0 0x7ff02c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.799+0000 7ff03ffff640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff02c06b4d0 0x7ff02c06d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff030033000 tx=0x7ff030009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.803+0000 7ff03cff9640 1 -- 172.21.15.59:0/1105852331 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff034019030 con 0x7ff0401f3000 2024-03-06T07:27:11.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.981+0000 7ff04693e640 1 -- 172.21.15.59:0/1105852331 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff040068ff0 con 0x7ff0401f3000 2024-03-06T07:27:11.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.982+0000 7ff03cff9640 1 -- 172.21.15.59:0/1105852331 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff034059980 con 0x7ff0401f3000 2024-03-06T07:27:11.984 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:27:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 -- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff02c06b4d0 msgr2=0x7ff02c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff02c06b4d0 0x7ff02c06d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff030033000 tx=0x7ff030009030 comp rx=0 tx=0).stop 2024-03-06T07:27:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 -- 172.21.15.59:0/1105852331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0401f3000 msgr2=0x7ff0400afa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0401f3000 0x7ff0400afa20 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7ff0340116d0 tx=0x7ff034011b90 comp rx=0 tx=0).stop 2024-03-06T07:27:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 -- 172.21.15.59:0/1105852331 shutdown_connections 2024-03-06T07:27:11.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff02c06b4d0 0x7ff02c06d990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0401f3480 0x7ff0400aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0401f3000 0x7ff0400afa20 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 --2- 172.21.15.59:0/1105852331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0280039b0 0x7ff0400b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 -- 172.21.15.59:0/1105852331 >> 172.21.15.59:0/1105852331 conn(0x7ff0400b3800 msgr2=0x7ff0400b52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 -- 172.21.15.59:0/1105852331 shutdown_connections 2024-03-06T07:27:11.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:11.984+0000 7ff04693e640 1 -- 172.21.15.59:0/1105852331 wait complete. 2024-03-06T07:27:11.997 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:12.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.097+0000 7f479020e640 1 Processor -- start 2024-03-06T07:27:12.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.098+0000 7f479020e640 1 -- start start 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.098+0000 7f479020e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f478815dfe0 0x7f47880cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.098+0000 7f479020e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47880d29c0 0x7f47880b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.098+0000 7f479020e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47880b3430 con 0x7f47880cf830 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.098+0000 7f479020e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47880b31e0 con 0x7f478815dfe0 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.098+0000 7f479020e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47880b3d60 con 0x7f47880d29c0 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.098+0000 7f478dfaa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f478815dfe0 0x7f47880cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.098+0000 7f478dfaa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f478815dfe0 0x7f47880cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57292/0 (socket says 172.21.15.59:57292) 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.098+0000 7f478dfaa640 1 -- 172.21.15.59:0/1035516374 learned_addr learned my addr 172.21.15.59:0/1035516374 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.099+0000 7f478e7ab640 1 --2- 172.21.15.59:0/1035516374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47880d29c0 0x7f47880b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.099+0000 7f478dfaa640 1 -- 172.21.15.59:0/1035516374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47880d29c0 msgr2=0x7f47880b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.099+0000 7f478dfaa640 1 --2- 172.21.15.59:0/1035516374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47880d29c0 0x7f47880b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.099+0000 7f478dfaa640 1 -- 172.21.15.59:0/1035516374 >> v1:172.21.15.164:6789/0 conn(0x7f47880cf830 legacy=0x7f47880d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.099+0000 7f478dfaa640 1 -- 172.21.15.59:0/1035516374 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47880b3710 con 0x7f478815dfe0 2024-03-06T07:27:12.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.099+0000 7f478dfaa640 1 --2- 172.21.15.59:0/1035516374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f478815dfe0 0x7f47880cf2f0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f4784009870 tx=0x7f478402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6efa21ba1393737 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.100+0000 7f478cfa8640 1 -- 172.21.15.59:0/1035516374 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4784035020 con 0x7f478815dfe0 2024-03-06T07:27:12.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.100+0000 7f478cfa8640 1 -- 172.21.15.59:0/1035516374 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4784031640 con 0x7f478815dfe0 2024-03-06T07:27:12.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.100+0000 7f478cfa8640 1 -- 172.21.15.59:0/1035516374 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478402d040 con 0x7f478815dfe0 2024-03-06T07:27:12.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.100+0000 7f479020e640 1 -- 172.21.15.59:0/1035516374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f478815dfe0 msgr2=0x7f47880cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.100+0000 7f479020e640 1 --2- 172.21.15.59:0/1035516374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f478815dfe0 0x7f47880cf2f0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f4784009870 tx=0x7f478402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.100+0000 7f479020e640 1 -- 172.21.15.59:0/1035516374 shutdown_connections 2024-03-06T07:27:12.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.100+0000 7f479020e640 1 --2- 172.21.15.59:0/1035516374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47880d29c0 0x7f47880b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.100+0000 7f479020e640 1 --2- 172.21.15.59:0/1035516374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f478815dfe0 0x7f47880cf2f0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.100+0000 7f479020e640 1 -- 172.21.15.59:0/1035516374 >> 172.21.15.59:0/1035516374 conn(0x7f47880d3cd0 msgr2=0x7f47880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.100+0000 7f479020e640 1 -- 172.21.15.59:0/1035516374 shutdown_connections 2024-03-06T07:27:12.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.100+0000 7f479020e640 1 -- 172.21.15.59:0/1035516374 wait complete. 2024-03-06T07:27:12.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.101+0000 7f479020e640 1 Processor -- start 2024-03-06T07:27:12.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.102+0000 7f479020e640 1 -- start start 2024-03-06T07:27:12.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.102+0000 7f479020e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880cf830 0x7f47880cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.102+0000 7f479020e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47880d29c0 0x7f47881520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.102+0000 7f479020e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f478815dfe0 0x7f478815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.102+0000 7f479020e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880d1df0 con 0x7f47880cf830 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.102+0000 7f479020e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47880d1c70 con 0x7f47880d29c0 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.102+0000 7f479020e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478815e3c0 con 0x7f478815dfe0 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.102+0000 7f478dfaa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880cf830 0x7f47880cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.102+0000 7f478dfaa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880cf830 0x7f47880cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57308/0 (socket says 172.21.15.59:57308) 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.102+0000 7f478dfaa640 1 -- 172.21.15.59:0/1366821627 learned_addr learned my addr 172.21.15.59:0/1366821627 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.102+0000 7f478e7ab640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f478815dfe0 0x7f478815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f478d7a9640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47880d29c0 0x7f47881520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f478dfaa640 1 -- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f478815dfe0 msgr2=0x7f478815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f478dfaa640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f478815dfe0 0x7f478815bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f478dfaa640 1 -- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47880d29c0 msgr2=0x7f47881520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f478dfaa640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47880d29c0 0x7f47881520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f478dfaa640 1 -- 172.21.15.59:0/1366821627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478815ba70 con 0x7f47880cf830 2024-03-06T07:27:12.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f478e7ab640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f478815dfe0 0x7f478815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:12.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f478dfaa640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880cf830 0x7f47880cd880 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f478402b930 tx=0x7f47840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f477e7fc640 1 -- 172.21.15.59:0/1366821627 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4784040070 con 0x7f47880cf830 2024-03-06T07:27:12.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f477e7fc640 1 -- 172.21.15.59:0/1366821627 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47840399f0 con 0x7f47880cf830 2024-03-06T07:27:12.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f479020e640 1 -- 172.21.15.59:0/1366821627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47880ce010 con 0x7f47880cf830 2024-03-06T07:27:12.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f477e7fc640 1 -- 172.21.15.59:0/1366821627 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4784035040 con 0x7f47880cf830 2024-03-06T07:27:12.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.103+0000 7f479020e640 1 -- 172.21.15.59:0/1366821627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47881f7580 con 0x7f47880cf830 2024-03-06T07:27:12.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.104+0000 7f479020e640 1 -- 172.21.15.59:0/1366821627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478804f330 con 0x7f47880cf830 2024-03-06T07:27:12.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.105+0000 7f477e7fc640 1 -- 172.21.15.59:0/1366821627 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f478404b060 con 0x7f47880cf830 2024-03-06T07:27:12.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.105+0000 7f477e7fc640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f476c06b4d0 0x7f476c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.105+0000 7f477e7fc640 1 -- 172.21.15.59:0/1366821627 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f47840ab750 con 0x7f47880cf830 2024-03-06T07:27:12.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.105+0000 7f478d7a9640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f476c06b4d0 0x7f476c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.106+0000 7f478d7a9640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f476c06b4d0 0x7f476c06d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f47880cb0c0 tx=0x7f4778009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.109+0000 7f477e7fc640 1 -- 172.21.15.59:0/1366821627 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f478407a9b0 con 0x7f47880cf830 2024-03-06T07:27:12.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.293+0000 7f479020e640 1 -- 172.21.15.59:0/1366821627 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f47880ca6b0 con 0x7f47880cf830 2024-03-06T07:27:12.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.294+0000 7f477e7fc640 1 -- 172.21.15.59:0/1366821627 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+65 (secure 0 0 0) 0x7f478407a2b0 con 0x7f47880cf830 2024-03-06T07:27:12.296 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:12.296 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-03-06T07:27:12.296 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:12.296 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-03-06T07:27:12.296 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-03-06T07:27:12.296 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:12.296 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-03-06T07:27:12.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 -- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f476c06b4d0 msgr2=0x7f476c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f476c06b4d0 0x7f476c06d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f47880cb0c0 tx=0x7f4778009040 comp rx=0 tx=0).stop 2024-03-06T07:27:12.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 -- 172.21.15.59:0/1366821627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880cf830 msgr2=0x7f47880cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880cf830 0x7f47880cd880 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f478402b930 tx=0x7f47840046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 -- 172.21.15.59:0/1366821627 shutdown_connections 2024-03-06T07:27:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f476c06b4d0 0x7f476c06d990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f478815dfe0 0x7f478815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47880d29c0 0x7f47881520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 --2- 172.21.15.59:0/1366821627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47880cf830 0x7f47880cd880 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 -- 172.21.15.59:0/1366821627 >> 172.21.15.59:0/1366821627 conn(0x7f47880d3cd0 msgr2=0x7f478806b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 -- 172.21.15.59:0/1366821627 shutdown_connections 2024-03-06T07:27:12.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.296+0000 7f479020e640 1 -- 172.21.15.59:0/1366821627 wait complete. 2024-03-06T07:27:12.309 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:12.309 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.413+0000 7f64efdf4640 1 Processor -- start 2024-03-06T07:27:12.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.413+0000 7f64efdf4640 1 -- start start 2024-03-06T07:27:12.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64efdf4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64e815bd60 0x7f64e806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64efdf4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f64e80cc6e0 0x7f64e81601a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64efdf4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e806f270 con 0x7f64e80b50a0 2024-03-06T07:27:12.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64efdf4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e815f750 con 0x7f64e80cc6e0 2024-03-06T07:27:12.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64efdf4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e815fd10 con 0x7f64e815bd60 2024-03-06T07:27:12.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64ed38f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f64e80cc6e0 0x7f64e81601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64ed38f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f64e80cc6e0 0x7f64e81601a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57320/0 (socket says 172.21.15.59:57320) 2024-03-06T07:27:12.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64ed38f640 1 -- 172.21.15.59:0/2143242248 learned_addr learned my addr 172.21.15.59:0/2143242248 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:12.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64edb90640 1 --2- 172.21.15.59:0/2143242248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64e815bd60 0x7f64e806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64ed38f640 1 -- 172.21.15.59:0/2143242248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64e815bd60 msgr2=0x7f64e806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64ed38f640 1 --2- 172.21.15.59:0/2143242248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64e815bd60 0x7f64e806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64ed38f640 1 -- 172.21.15.59:0/2143242248 >> v1:172.21.15.164:6789/0 conn(0x7f64e80b50a0 legacy=0x7f64e80ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.414+0000 7f64ed38f640 1 -- 172.21.15.59:0/2143242248 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e806f550 con 0x7f64e80cc6e0 2024-03-06T07:27:12.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.415+0000 7f64ed38f640 1 --2- 172.21.15.59:0/2143242248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64e80cc6e0 0x7f64e81601a0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f64d8009870 tx=0x7f64d802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73846d35282b6f0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.415+0000 7f64ecb8e640 1 -- 172.21.15.59:0/2143242248 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d80317c0 con 0x7f64e80cc6e0 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.415+0000 7f64ecb8e640 1 -- 172.21.15.59:0/2143242248 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d8037070 con 0x7f64e80cc6e0 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.415+0000 7f64ecb8e640 1 -- 172.21.15.59:0/2143242248 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8031b80 con 0x7f64e80cc6e0 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.416+0000 7f64efdf4640 1 -- 172.21.15.59:0/2143242248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64e80cc6e0 msgr2=0x7f64e81601a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.416+0000 7f64efdf4640 1 --2- 172.21.15.59:0/2143242248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64e80cc6e0 0x7f64e81601a0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f64d8009870 tx=0x7f64d802b860 comp rx=0 tx=0).stop 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.416+0000 7f64efdf4640 1 -- 172.21.15.59:0/2143242248 shutdown_connections 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.416+0000 7f64efdf4640 1 --2- 172.21.15.59:0/2143242248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64e80cc6e0 0x7f64e81601a0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.416+0000 7f64efdf4640 1 --2- 172.21.15.59:0/2143242248 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64e815bd60 0x7f64e806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.416+0000 7f64efdf4640 1 -- 172.21.15.59:0/2143242248 >> 172.21.15.59:0/2143242248 conn(0x7f64e80b1a90 msgr2=0x7f64e80b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.416+0000 7f64efdf4640 1 -- 172.21.15.59:0/2143242248 shutdown_connections 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.416+0000 7f64efdf4640 1 -- 172.21.15.59:0/2143242248 wait complete. 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.417+0000 7f64efdf4640 1 Processor -- start 2024-03-06T07:27:12.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.417+0000 7f64efdf4640 1 -- start start 2024-03-06T07:27:12.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64efdf4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64e80b50a0 0x7f64e815e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64efdf4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64e80cc6e0 0x7f64e815d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64efdf4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64e815bd60 0x7f64e815d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64efdf4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80b5530 con 0x7f64e80b50a0 2024-03-06T07:27:12.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64efdf4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e81606e0 con 0x7f64e80cc6e0 2024-03-06T07:27:12.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64efdf4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80b56b0 con 0x7f64e815bd60 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64edb90640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64e80b50a0 0x7f64e815e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64edb90640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64e80b50a0 0x7f64e815e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57330/0 (socket says 172.21.15.59:57330) 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64edb90640 1 -- 172.21.15.59:0/65259383 learned_addr learned my addr 172.21.15.59:0/65259383 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64ee391640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64e815bd60 0x7f64e815d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64ed38f640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64e80cc6e0 0x7f64e815d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64edb90640 1 -- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64e815bd60 msgr2=0x7f64e815d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64edb90640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64e815bd60 0x7f64e815d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64edb90640 1 -- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64e80cc6e0 msgr2=0x7f64e815d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64edb90640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64e80cc6e0 0x7f64e815d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64edb90640 1 -- 172.21.15.59:0/65259383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e8159eb0 con 0x7f64e80b50a0 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64ed38f640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64e80cc6e0 0x7f64e815d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64edb90640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64e80b50a0 0x7f64e815e1f0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f64e400d5f0 tx=0x7f64e400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64de7fc640 1 -- 172.21.15.59:0/65259383 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e40125f0 con 0x7f64e80b50a0 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64de7fc640 1 -- 172.21.15.59:0/65259383 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e4019070 con 0x7f64e80b50a0 2024-03-06T07:27:12.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64efdf4640 1 -- 172.21.15.59:0/65259383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e815a190 con 0x7f64e80b50a0 2024-03-06T07:27:12.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64de7fc640 1 -- 172.21.15.59:0/65259383 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e40129d0 con 0x7f64e80b50a0 2024-03-06T07:27:12.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.418+0000 7f64efdf4640 1 -- 172.21.15.59:0/65259383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e815a750 con 0x7f64e80b50a0 2024-03-06T07:27:12.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.419+0000 7f64efdf4640 1 -- 172.21.15.59:0/65259383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e804f330 con 0x7f64e80b50a0 2024-03-06T07:27:12.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.420+0000 7f64de7fc640 1 -- 172.21.15.59:0/65259383 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64e4012b70 con 0x7f64e80b50a0 2024-03-06T07:27:12.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.420+0000 7f64de7fc640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f64cc06b4d0 0x7f64cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.420+0000 7f64de7fc640 1 -- 172.21.15.59:0/65259383 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f64e408a250 con 0x7f64e80b50a0 2024-03-06T07:27:12.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.421+0000 7f64ed38f640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f64cc06b4d0 0x7f64cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.421+0000 7f64ed38f640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f64cc06b4d0 0x7f64cc06d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f64d8036000 tx=0x7f64d803d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.424+0000 7f64de7fc640 1 -- 172.21.15.59:0/65259383 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64e4059420 con 0x7f64e80b50a0 2024-03-06T07:27:12.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.605+0000 7f64efdf4640 1 -- 172.21.15.59:0/65259383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f64e806eeb0 con 0x7f64e80b50a0 2024-03-06T07:27:12.969 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.967+0000 7f64de7fc640 1 -- 172.21.15.59:0/65259383 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v105) v1 ==== 138+0+0 (secure 0 0 0) 0x7f64e4058d20 con 0x7f64e80b50a0 2024-03-06T07:27:12.969 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:12.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 -- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f64cc06b4d0 msgr2=0x7f64cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f64cc06b4d0 0x7f64cc06d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f64d8036000 tx=0x7f64d803d040 comp rx=0 tx=0).stop 2024-03-06T07:27:12.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 -- 172.21.15.59:0/65259383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64e80b50a0 msgr2=0x7f64e815e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64e80b50a0 0x7f64e815e1f0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f64e400d5f0 tx=0x7f64e400dab0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 -- 172.21.15.59:0/65259383 shutdown_connections 2024-03-06T07:27:12.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f64cc06b4d0 0x7f64cc06d990 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64e815bd60 0x7f64e815d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64e80cc6e0 0x7f64e815d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 --2- 172.21.15.59:0/65259383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64e80b50a0 0x7f64e815e1f0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 -- 172.21.15.59:0/65259383 >> 172.21.15.59:0/65259383 conn(0x7f64e80b1a90 msgr2=0x7f64e8071260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 -- 172.21.15.59:0/65259383 shutdown_connections 2024-03-06T07:27:12.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:12.969+0000 7f64efdf4640 1 -- 172.21.15.59:0/65259383 wait complete. 2024-03-06T07:27:12.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.093+0000 7f97c21a1640 1 Processor -- start 2024-03-06T07:27:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.093+0000 7f97c21a1640 1 -- start start 2024-03-06T07:27:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.094+0000 7f97c21a1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f97bc0d4860 0x7f97bc0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.094+0000 7f97c21a1640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97bc1618b0 con 0x7f97bc0b0e30 2024-03-06T07:27:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.094+0000 7f97c21a1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97bc0cf400 con 0x7f97bc1614d0 2024-03-06T07:27:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.094+0000 7f97c21a1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97bc0d4cf0 con 0x7f97bc0d4860 2024-03-06T07:27:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.094+0000 7f97baffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f97bc0d4860 0x7f97bc0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.094+0000 7f97bbfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f97bc0b0e30 0x7f97bc0b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58802/0 (socket says 172.21.15.59:58802) 2024-03-06T07:27:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.094+0000 7f97bbfff640 1 -- 172.21.15.59:0/956265891 learned_addr learned my addr 172.21.15.59:0/956265891 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.094+0000 7f97baffd640 1 -- 172.21.15.59:0/956265891 >> v1:172.21.15.164:6789/0 conn(0x7f97bc1614d0 legacy=0x7f97bc0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.094+0000 7f97baffd640 1 -- 172.21.15.59:0/956265891 >> v1:172.21.15.59:6789/0 conn(0x7f97bc0b0e30 legacy=0x7f97bc0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.094+0000 7f97baffd640 1 -- 172.21.15.59:0/956265891 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc0cf6a0 con 0x7f97bc0d4860 2024-03-06T07:27:13.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.094+0000 7f97ba7fc640 1 -- 172.21.15.59:0/956265891 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359357419 0 0) 0x7f97bc1618b0 con 0x7f97bc0b0e30 2024-03-06T07:27:13.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.095+0000 7f97baffd640 1 --2- 172.21.15.59:0/956265891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97bc0d4860 0x7f97bc0d2020 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f97b00097a0 tx=0x7f97b002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7973e20ce82dc395 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.095+0000 7f97ba7fc640 1 -- 172.21.15.59:0/956265891 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0034070 con 0x7f97bc0d4860 2024-03-06T07:27:13.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.095+0000 7f97ba7fc640 1 -- 172.21.15.59:0/956265891 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b0030610 con 0x7f97bc0d4860 2024-03-06T07:27:13.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.095+0000 7f97ba7fc640 1 -- 172.21.15.59:0/956265891 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b00309e0 con 0x7f97bc0d4860 2024-03-06T07:27:13.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.096+0000 7f97c21a1640 1 -- 172.21.15.59:0/956265891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97bc0d4860 msgr2=0x7f97bc0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.096+0000 7f97c21a1640 1 --2- 172.21.15.59:0/956265891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97bc0d4860 0x7f97bc0d2020 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f97b00097a0 tx=0x7f97b002b960 comp rx=0 tx=0).stop 2024-03-06T07:27:13.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.096+0000 7f97c21a1640 1 -- 172.21.15.59:0/956265891 shutdown_connections 2024-03-06T07:27:13.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.096+0000 7f97c21a1640 1 --2- 172.21.15.59:0/956265891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97bc0d4860 0x7f97bc0d2020 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.096+0000 7f97c21a1640 1 -- 172.21.15.59:0/956265891 >> 172.21.15.59:0/956265891 conn(0x7f97bc0d3d40 msgr2=0x7f97bc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.096+0000 7f97c21a1640 1 -- 172.21.15.59:0/956265891 shutdown_connections 2024-03-06T07:27:13.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.096+0000 7f97c21a1640 1 -- 172.21.15.59:0/956265891 wait complete. 2024-03-06T07:27:13.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.097+0000 7f97c21a1640 1 Processor -- start 2024-03-06T07:27:13.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.097+0000 7f97c21a1640 1 -- start start 2024-03-06T07:27:13.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97c21a1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97bc0b0e30 0x7f97bc0cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97c21a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97bc0d4860 0x7f97bc156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97c21a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97bc1614d0 0x7f97bc1f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97c21a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc0b0910 con 0x7f97bc0d4860 2024-03-06T07:27:13.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97c21a1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc0cfc60 con 0x7f97bc1614d0 2024-03-06T07:27:13.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97c21a1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc0d2560 con 0x7f97bc0b0e30 2024-03-06T07:27:13.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97baffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97bc0d4860 0x7f97bc156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97baffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97bc0d4860 0x7f97bc156270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57346/0 (socket says 172.21.15.59:57346) 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97baffd640 1 -- 172.21.15.59:0/2350852529 learned_addr learned my addr 172.21.15.59:0/2350852529 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97bb7fe640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97bc0b0e30 0x7f97bc0cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97bbfff640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97bc1614d0 0x7f97bc1f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97baffd640 1 -- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97bc0b0e30 msgr2=0x7f97bc0cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97baffd640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97bc0b0e30 0x7f97bc0cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97baffd640 1 -- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97bc1614d0 msgr2=0x7f97bc1f7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97baffd640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97bc1614d0 0x7f97bc1f7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97baffd640 1 -- 172.21.15.59:0/2350852529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc0ca020 con 0x7f97bc0d4860 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97bbfff640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97bc1614d0 0x7f97bc1f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.098+0000 7f97baffd640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97bc0d4860 0x7f97bc156270 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f97b002bf70 tx=0x7f97b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.099+0000 7f97a3fff640 1 -- 172.21.15.59:0/2350852529 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0034040 con 0x7f97bc0d4860 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.099+0000 7f97a3fff640 1 -- 172.21.15.59:0/2350852529 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b0039a10 con 0x7f97bc0d4860 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.099+0000 7f97c21a1640 1 -- 172.21.15.59:0/2350852529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97bc0cbee0 con 0x7f97bc0d4860 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.099+0000 7f97a3fff640 1 -- 172.21.15.59:0/2350852529 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0039de0 con 0x7f97bc0d4860 2024-03-06T07:27:13.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.099+0000 7f97c21a1640 1 -- 172.21.15.59:0/2350852529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97bc0cc4a0 con 0x7f97bc0d4860 2024-03-06T07:27:13.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.100+0000 7f97c21a1640 1 -- 172.21.15.59:0/2350852529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97bc068ff0 con 0x7f97bc0d4860 2024-03-06T07:27:13.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.100+0000 7f97a3fff640 1 -- 172.21.15.59:0/2350852529 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97b0037070 con 0x7f97bc0d4860 2024-03-06T07:27:13.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.101+0000 7f97a3fff640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a806b4d0 0x7f97a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:13.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.101+0000 7f97a3fff640 1 -- 172.21.15.59:0/2350852529 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f97b00ac0a0 con 0x7f97bc0d4860 2024-03-06T07:27:13.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.101+0000 7f97bb7fe640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a806b4d0 0x7f97a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:13.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.101+0000 7f97bb7fe640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a806b4d0 0x7f97a806d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f97ac000f60 tx=0x7f97ac009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:13.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.105+0000 7f97a3fff640 1 -- 172.21.15.59:0/2350852529 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97b007b190 con 0x7f97bc0d4860 2024-03-06T07:27:13.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.286+0000 7f97c21a1640 1 -- 172.21.15.59:0/2350852529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f97bc0d0af0 con 0x7f97bc0d4860 2024-03-06T07:27:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.973+0000 7f97a3fff640 1 -- 172.21.15.59:0/2350852529 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v106) v1 ==== 170+0+0 (secure 0 0 0) 0x7f97b007aa90 con 0x7f97bc0d4860 2024-03-06T07:27:13.975 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-03-06T07:27:13.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 -- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a806b4d0 msgr2=0x7f97a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a806b4d0 0x7f97a806d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f97ac000f60 tx=0x7f97ac009040 comp rx=0 tx=0).stop 2024-03-06T07:27:13.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 -- 172.21.15.59:0/2350852529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97bc0d4860 msgr2=0x7f97bc156270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:13.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97bc0d4860 0x7f97bc156270 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f97b002bf70 tx=0x7f97b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 -- 172.21.15.59:0/2350852529 shutdown_connections 2024-03-06T07:27:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a806b4d0 0x7f97a806d990 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97bc1614d0 0x7f97bc1f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97bc0d4860 0x7f97bc156270 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 --2- 172.21.15.59:0/2350852529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97bc0b0e30 0x7f97bc0cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 -- 172.21.15.59:0/2350852529 >> 172.21.15.59:0/2350852529 conn(0x7f97bc0d3d40 msgr2=0x7f97bc0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 -- 172.21.15.59:0/2350852529 shutdown_connections 2024-03-06T07:27:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:13.976+0000 7f97c21a1640 1 -- 172.21.15.59:0/2350852529 wait complete. 2024-03-06T07:27:14.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:14.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.113+0000 7f87afa64640 1 Processor -- start 2024-03-06T07:27:14.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.113+0000 7f87afa64640 1 -- start start 2024-03-06T07:27:14.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87afa64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87a80b5ee0 0x7f87a8165a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87afa64640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f87a8070650 0x7f87a8167f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87afa64640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a80b6430 con 0x7f87a806cc90 2024-03-06T07:27:14.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87afa64640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a806ebf0 con 0x7f87a8070650 2024-03-06T07:27:14.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87afa64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a80b6710 con 0x7f87a80b5ee0 2024-03-06T07:27:14.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87ad800640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f87a806cc90 0x7f87a806d070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60798/0 (socket says 172.21.15.59:60798) 2024-03-06T07:27:14.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87ad800640 1 -- 172.21.15.59:0/4240851606 learned_addr learned my addr 172.21.15.59:0/4240851606 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:14.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87ae001640 1 --2- 172.21.15.59:0/4240851606 >> v2:172.21.15.59:3300/0 conn(0x7f87a8070650 0x7f87a8167f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87acfff640 1 --2- 172.21.15.59:0/4240851606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87a80b5ee0 0x7f87a8165a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f879ffff640 1 -- 172.21.15.59:0/4240851606 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632005095 0 0) 0x7f87a80b6430 con 0x7f87a806cc90 2024-03-06T07:27:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f879ffff640 1 -- 172.21.15.59:0/4240851606 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8794003520 con 0x7f87a806cc90 2024-03-06T07:27:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87ae001640 1 -- 172.21.15.59:0/4240851606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87a80b5ee0 msgr2=0x7f87a8165a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87ae001640 1 --2- 172.21.15.59:0/4240851606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87a80b5ee0 0x7f87a8165a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87ae001640 1 -- 172.21.15.59:0/4240851606 >> v1:172.21.15.59:6789/0 conn(0x7f87a806cc90 legacy=0x7f87a806d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.114+0000 7f87ae001640 1 -- 172.21.15.59:0/4240851606 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a806ee40 con 0x7f87a8070650 2024-03-06T07:27:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.115+0000 7f87ae001640 1 --2- 172.21.15.59:0/4240851606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87a8070650 0x7f87a8167f90 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f87a0009d90 tx=0x7f87a002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7558a091c5cae0d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.115+0000 7f879ffff640 1 -- 172.21.15.59:0/4240851606 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a0039070 con 0x7f87a8070650 2024-03-06T07:27:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.115+0000 7f879ffff640 1 -- 172.21.15.59:0/4240851606 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a002cbc0 con 0x7f87a8070650 2024-03-06T07:27:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.115+0000 7f879ffff640 1 -- 172.21.15.59:0/4240851606 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a0034440 con 0x7f87a8070650 2024-03-06T07:27:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.116+0000 7f87afa64640 1 -- 172.21.15.59:0/4240851606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87a8070650 msgr2=0x7f87a8167f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.116+0000 7f87afa64640 1 --2- 172.21.15.59:0/4240851606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87a8070650 0x7f87a8167f90 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f87a0009d90 tx=0x7f87a002be10 comp rx=0 tx=0).stop 2024-03-06T07:27:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.116+0000 7f87afa64640 1 -- 172.21.15.59:0/4240851606 shutdown_connections 2024-03-06T07:27:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.116+0000 7f87afa64640 1 --2- 172.21.15.59:0/4240851606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87a8070650 0x7f87a8167f90 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.116+0000 7f87afa64640 1 --2- 172.21.15.59:0/4240851606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87a80b5ee0 0x7f87a8165a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.116+0000 7f87afa64640 1 -- 172.21.15.59:0/4240851606 >> 172.21.15.59:0/4240851606 conn(0x7f87a80b6ea0 msgr2=0x7f87a80b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.116+0000 7f87afa64640 1 -- 172.21.15.59:0/4240851606 shutdown_connections 2024-03-06T07:27:14.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.116+0000 7f87afa64640 1 -- 172.21.15.59:0/4240851606 wait complete. 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.117+0000 7f87afa64640 1 Processor -- start 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.117+0000 7f87afa64640 1 -- start start 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87afa64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87a806cc90 0x7f87a809c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87afa64640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87a8070650 0x7f87a814e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87afa64640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87a80b5ee0 0x7f87a8158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87afa64640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a806c9d0 con 0x7f87a8070650 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87afa64640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a806c850 con 0x7f87a80b5ee0 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87afa64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a815e1b0 con 0x7f87a806cc90 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87acfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87a8070650 0x7f87a814e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87acfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87a8070650 0x7f87a814e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37188/0 (socket says 172.21.15.59:37188) 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87acfff640 1 -- 172.21.15.59:0/2288552375 learned_addr learned my addr 172.21.15.59:0/2288552375 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87ad800640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87a806cc90 0x7f87a809c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87ae001640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87a80b5ee0 0x7f87a8158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87acfff640 1 -- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87a806cc90 msgr2=0x7f87a809c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87acfff640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87a806cc90 0x7f87a809c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87acfff640 1 -- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87a80b5ee0 msgr2=0x7f87a8158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87acfff640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87a80b5ee0 0x7f87a8158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87acfff640 1 -- 172.21.15.59:0/2288552375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a809bb40 con 0x7f87a8070650 2024-03-06T07:27:14.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f87acfff640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87a8070650 0x7f87a814e540 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f879800e9b0 tx=0x7f879800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.118+0000 7f879dffb640 1 -- 172.21.15.59:0/2288552375 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798017070 con 0x7f87a8070650 2024-03-06T07:27:14.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.119+0000 7f879dffb640 1 -- 172.21.15.59:0/2288552375 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879800c9a0 con 0x7f87a8070650 2024-03-06T07:27:14.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.119+0000 7f87afa64640 1 -- 172.21.15.59:0/2288552375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a809be20 con 0x7f87a8070650 2024-03-06T07:27:14.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.119+0000 7f879dffb640 1 -- 172.21.15.59:0/2288552375 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879800cd90 con 0x7f87a8070650 2024-03-06T07:27:14.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.119+0000 7f87afa64640 1 -- 172.21.15.59:0/2288552375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a809c3e0 con 0x7f87a8070650 2024-03-06T07:27:14.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.120+0000 7f87afa64640 1 -- 172.21.15.59:0/2288552375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a814eae0 con 0x7f87a8070650 2024-03-06T07:27:14.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.120+0000 7f879dffb640 1 -- 172.21.15.59:0/2288552375 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f879801a070 con 0x7f87a8070650 2024-03-06T07:27:14.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.121+0000 7f879dffb640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f878c06b4d0 0x7f878c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.121+0000 7f879dffb640 1 -- 172.21.15.59:0/2288552375 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f879808a4d0 con 0x7f87a8070650 2024-03-06T07:27:14.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.121+0000 7f87ad800640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f878c06b4d0 0x7f878c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.121+0000 7f87ad800640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f878c06b4d0 0x7f878c06d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f87a4005880 tx=0x7f87a4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.125+0000 7f879dffb640 1 -- 172.21.15.59:0/2288552375 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8798058d60 con 0x7f87a8070650 2024-03-06T07:27:14.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.343+0000 7f87afa64640 1 -- 172.21.15.59:0/2288552375 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f87a804f330 con 0x7f87a8070650 2024-03-06T07:27:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.974+0000 7f879dffb640 1 -- 172.21.15.59:0/2288552375 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v63) v1 ==== 174+0+0 (secure 0 0 0) 0x7f879805ce00 con 0x7f87a8070650 2024-03-06T07:27:14.976 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 16 and data pool 17 2024-03-06T07:27:14.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.978+0000 7f87afa64640 1 -- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f878c06b4d0 msgr2=0x7f878c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.978+0000 7f87afa64640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f878c06b4d0 0x7f878c06d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f87a4005880 tx=0x7f87a4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:14.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.978+0000 7f87afa64640 1 -- 172.21.15.59:0/2288552375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87a8070650 msgr2=0x7f87a814e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.978+0000 7f87afa64640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87a8070650 0x7f87a814e540 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f879800e9b0 tx=0x7f879800ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:14.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.978+0000 7f87afa64640 1 -- 172.21.15.59:0/2288552375 shutdown_connections 2024-03-06T07:27:14.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.978+0000 7f87afa64640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f878c06b4d0 0x7f878c06d990 secure :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f87a4005880 tx=0x7f87a4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:14.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.978+0000 7f87afa64640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87a80b5ee0 0x7f87a8158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.978+0000 7f87afa64640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f87a8070650 0x7f87a814e540 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.978+0000 7f87afa64640 1 --2- 172.21.15.59:0/2288552375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87a806cc90 0x7f87a809c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.978+0000 7f87afa64640 1 -- 172.21.15.59:0/2288552375 >> 172.21.15.59:0/2288552375 conn(0x7f87a80b6ea0 msgr2=0x7f87a815c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.979+0000 7f87afa64640 1 -- 172.21.15.59:0/2288552375 shutdown_connections 2024-03-06T07:27:14.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:14.981+0000 7f87afa64640 1 -- 172.21.15.59:0/2288552375 wait complete. 2024-03-06T07:27:15.014 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:15.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.116+0000 7fe8720d4640 1 Processor -- start 2024-03-06T07:27:15.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.116+0000 7fe8720d4640 1 -- start start 2024-03-06T07:27:15.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.116+0000 7fe8720d4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c06afb0 0x7fe86c06b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.116+0000 7fe8720d4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe86c0b4420 0x7fe86c0cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.116+0000 7fe8720d4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86c0d0a70 con 0x7fe86c0ce0d0 2024-03-06T07:27:15.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.116+0000 7fe8720d4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86c0d0850 con 0x7fe86c0b4420 2024-03-06T07:27:15.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.116+0000 7fe8720d4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c06b8d0 con 0x7fe86c06afb0 2024-03-06T07:27:15.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.117+0000 7fe86affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe86c0b4420 0x7fe86c0cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.117+0000 7fe86bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe86c0ce0d0 0x7fe86c071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60804/0 (socket says 172.21.15.59:60804) 2024-03-06T07:27:15.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.117+0000 7fe86bfff640 1 -- 172.21.15.59:0/326691757 learned_addr learned my addr 172.21.15.59:0/326691757 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:15.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.117+0000 7fe86b7fe640 1 --2- 172.21.15.59:0/326691757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c06afb0 0x7fe86c06b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.117+0000 7fe86affd640 1 -- 172.21.15.59:0/326691757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c06afb0 msgr2=0x7fe86c06b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.117+0000 7fe86affd640 1 --2- 172.21.15.59:0/326691757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c06afb0 0x7fe86c06b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.117+0000 7fe86affd640 1 -- 172.21.15.59:0/326691757 >> v1:172.21.15.59:6789/0 conn(0x7fe86c0ce0d0 legacy=0x7fe86c071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.117+0000 7fe86affd640 1 -- 172.21.15.59:0/326691757 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c0d0d10 con 0x7fe86c0b4420 2024-03-06T07:27:15.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe86affd640 1 --2- 172.21.15.59:0/326691757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0b4420 0x7fe86c0cdb90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe8600097a0 tx=0x7fe86002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5f7708f784c2a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe86a7fc640 1 -- 172.21.15.59:0/326691757 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe860038070 con 0x7fe86c0b4420 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe86a7fc640 1 -- 172.21.15.59:0/326691757 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe860031700 con 0x7fe86c0b4420 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe86a7fc640 1 -- 172.21.15.59:0/326691757 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe860031ad0 con 0x7fe86c0b4420 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe8720d4640 1 -- 172.21.15.59:0/326691757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0b4420 msgr2=0x7fe86c0cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe8720d4640 1 --2- 172.21.15.59:0/326691757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0b4420 0x7fe86c0cdb90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe8600097a0 tx=0x7fe86002b7a0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe8720d4640 1 -- 172.21.15.59:0/326691757 shutdown_connections 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe8720d4640 1 --2- 172.21.15.59:0/326691757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0b4420 0x7fe86c0cdb90 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe8720d4640 1 --2- 172.21.15.59:0/326691757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c06afb0 0x7fe86c06b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe8720d4640 1 -- 172.21.15.59:0/326691757 >> 172.21.15.59:0/326691757 conn(0x7fe86c06c0c0 msgr2=0x7fe86c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe8720d4640 1 -- 172.21.15.59:0/326691757 shutdown_connections 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.118+0000 7fe8720d4640 1 -- 172.21.15.59:0/326691757 wait complete. 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.119+0000 7fe8720d4640 1 Processor -- start 2024-03-06T07:27:15.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.120+0000 7fe8720d4640 1 -- start start 2024-03-06T07:27:15.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.120+0000 7fe8720d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c06afb0 0x7fe86c1562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.120+0000 7fe8720d4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c0b4420 0x7fe86c144600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.120+0000 7fe8720d4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe86c0ce0d0 0x7fe86c144b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.120+0000 7fe8720d4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c145840 con 0x7fe86c06afb0 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.120+0000 7fe8720d4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c0d12d0 con 0x7fe86c0ce0d0 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.120+0000 7fe8720d4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c0ceb90 con 0x7fe86c0b4420 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe86b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c06afb0 0x7fe86c1562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe86b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c06afb0 0x7fe86c1562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37212/0 (socket says 172.21.15.59:37212) 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe86b7fe640 1 -- 172.21.15.59:0/1038474382 learned_addr learned my addr 172.21.15.59:0/1038474382 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe86affd640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c0b4420 0x7fe86c144600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe86bfff640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe86c0ce0d0 0x7fe86c144b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe86b7fe640 1 -- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c0b4420 msgr2=0x7fe86c144600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe86b7fe640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c0b4420 0x7fe86c144600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe86b7fe640 1 -- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe86c0ce0d0 msgr2=0x7fe86c144b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe86b7fe640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe86c0ce0d0 0x7fe86c144b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe86b7fe640 1 -- 172.21.15.59:0/1038474382 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c0c9d50 con 0x7fe86c06afb0 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe86b7fe640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c06afb0 0x7fe86c1562a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00e9e0 tx=0x7fe85c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe853fff640 1 -- 172.21.15.59:0/1038474382 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c017070 con 0x7fe86c06afb0 2024-03-06T07:27:15.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe853fff640 1 -- 172.21.15.59:0/1038474382 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85c00c9d0 con 0x7fe86c06afb0 2024-03-06T07:27:15.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe8720d4640 1 -- 172.21.15.59:0/1038474382 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86c0ca030 con 0x7fe86c06afb0 2024-03-06T07:27:15.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe853fff640 1 -- 172.21.15.59:0/1038474382 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c00cdc0 con 0x7fe86c06afb0 2024-03-06T07:27:15.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.121+0000 7fe8720d4640 1 -- 172.21.15.59:0/1038474382 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86c0ca860 con 0x7fe86c06afb0 2024-03-06T07:27:15.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.122+0000 7fe8720d4640 1 -- 172.21.15.59:0/1038474382 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86c0ce540 con 0x7fe86c06afb0 2024-03-06T07:27:15.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.123+0000 7fe853fff640 1 -- 172.21.15.59:0/1038474382 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe85c01a070 con 0x7fe86c06afb0 2024-03-06T07:27:15.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.123+0000 7fe853fff640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85806b4d0 0x7fe85806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.123+0000 7fe853fff640 1 -- 172.21.15.59:0/1038474382 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe85c08a590 con 0x7fe86c06afb0 2024-03-06T07:27:15.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.123+0000 7fe86affd640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85806b4d0 0x7fe85806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.124+0000 7fe86affd640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85806b4d0 0x7fe85806d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe860035000 tx=0x7fe860008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.128+0000 7fe853fff640 1 -- 172.21.15.59:0/1038474382 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe85c058dc0 con 0x7fe86c06afb0 2024-03-06T07:27:15.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.308+0000 7fe8720d4640 1 -- 172.21.15.59:0/1038474382 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe86c04f330 con 0x7fe86c06afb0 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.309+0000 7fe853fff640 1 -- 172.21.15.59:0/1038474382 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16628 (secure 0 0 0) 0x7fe85c05ce60 con 0x7fe86c06afb0 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:15.311 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:27:14.344872+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:12.625799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:13.306376+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.311+0000 7fe8720d4640 1 -- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85806b4d0 msgr2=0x7fe85806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.312+0000 7fe8720d4640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85806b4d0 0x7fe85806d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe860035000 tx=0x7fe860008030 comp rx=0 tx=0).stop 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.312+0000 7fe8720d4640 1 -- 172.21.15.59:0/1038474382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c06afb0 msgr2=0x7fe86c1562a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.312+0000 7fe8720d4640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c06afb0 0x7fe86c1562a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00e9e0 tx=0x7fe85c00eea0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.312+0000 7fe8720d4640 1 -- 172.21.15.59:0/1038474382 shutdown_connections 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.312+0000 7fe8720d4640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85806b4d0 0x7fe85806d990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.312+0000 7fe8720d4640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe86c0ce0d0 0x7fe86c144b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.312+0000 7fe8720d4640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c0b4420 0x7fe86c144600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.312+0000 7fe8720d4640 1 --2- 172.21.15.59:0/1038474382 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c06afb0 0x7fe86c1562a0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.312+0000 7fe8720d4640 1 -- 172.21.15.59:0/1038474382 >> 172.21.15.59:0/1038474382 conn(0x7fe86c06c0c0 msgr2=0x7fe86c142c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.312+0000 7fe8720d4640 1 -- 172.21.15.59:0/1038474382 shutdown_connections 2024-03-06T07:27:15.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.312+0000 7fe8720d4640 1 -- 172.21.15.59:0/1038474382 wait complete. 2024-03-06T07:27:15.325 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:15.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.424+0000 7fbf28eb5640 1 Processor -- start 2024-03-06T07:27:15.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.424+0000 7fbf28eb5640 1 -- start start 2024-03-06T07:27:15.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.425+0000 7fbf28eb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf2415dfe0 0x7fbf240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.425+0000 7fbf28eb5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbf240cf830 0x7fbf240d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.425+0000 7fbf28eb5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf240d27f0 0x7fbf240d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.425+0000 7fbf28eb5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf240d2c50 con 0x7fbf240d27f0 2024-03-06T07:27:15.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.425+0000 7fbf28eb5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf240b31e0 con 0x7fbf240cf830 2024-03-06T07:27:15.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.425+0000 7fbf28eb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240b3d60 con 0x7fbf2415dfe0 2024-03-06T07:27:15.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.425+0000 7fbf22d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf240d27f0 0x7fbf240d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.425+0000 7fbf22d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf240d27f0 0x7fbf240d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37216/0 (socket says 172.21.15.59:37216) 2024-03-06T07:27:15.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.425+0000 7fbf22d9d640 1 -- 172.21.15.59:0/4186706063 learned_addr learned my addr 172.21.15.59:0/4186706063 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:15.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.425+0000 7fbf2259c640 1 --2- 172.21.15.59:0/4186706063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf2415dfe0 0x7fbf240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.426+0000 7fbf1bfff640 1 --2- 172.21.15.59:0/4186706063 >> v2:172.21.15.164:3300/0 conn(0x7fbf240cf830 0x7fbf240d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.426+0000 7fbf22d9d640 1 -- 172.21.15.59:0/4186706063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf2415dfe0 msgr2=0x7fbf240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.426+0000 7fbf22d9d640 1 --2- 172.21.15.59:0/4186706063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf2415dfe0 0x7fbf240cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.426+0000 7fbf22d9d640 1 -- 172.21.15.59:0/4186706063 >> v2:172.21.15.164:3300/0 conn(0x7fbf240cf830 msgr2=0x7fbf240d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.426+0000 7fbf22d9d640 1 --2- 172.21.15.59:0/4186706063 >> v2:172.21.15.164:3300/0 conn(0x7fbf240cf830 0x7fbf240d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.426+0000 7fbf22d9d640 1 -- 172.21.15.59:0/4186706063 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240b3430 con 0x7fbf240d27f0 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.426+0000 7fbf22d9d640 1 --2- 172.21.15.59:0/4186706063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf240d27f0 0x7fbf240d0300 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fbf2408cba0 tx=0x7fbf1c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99644532266740fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf21d9b640 1 -- 172.21.15.59:0/4186706063 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c02ca70 con 0x7fbf240d27f0 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf21d9b640 1 -- 172.21.15.59:0/4186706063 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1c02cc10 con 0x7fbf240d27f0 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf21d9b640 1 -- 172.21.15.59:0/4186706063 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c0343f0 con 0x7fbf240d27f0 2024-03-06T07:27:15.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf28eb5640 1 -- 172.21.15.59:0/4186706063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf240d27f0 msgr2=0x7fbf240d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf28eb5640 1 --2- 172.21.15.59:0/4186706063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf240d27f0 0x7fbf240d0300 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fbf2408cba0 tx=0x7fbf1c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf28eb5640 1 -- 172.21.15.59:0/4186706063 shutdown_connections 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf28eb5640 1 --2- 172.21.15.59:0/4186706063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf240d27f0 0x7fbf240d0300 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf28eb5640 1 --2- 172.21.15.59:0/4186706063 >> v2:172.21.15.164:3300/0 conn(0x7fbf240cf830 0x7fbf240d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf28eb5640 1 --2- 172.21.15.59:0/4186706063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf2415dfe0 0x7fbf240cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf28eb5640 1 -- 172.21.15.59:0/4186706063 >> 172.21.15.59:0/4186706063 conn(0x7fbf240d3cd0 msgr2=0x7fbf240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf28eb5640 1 -- 172.21.15.59:0/4186706063 shutdown_connections 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.427+0000 7fbf28eb5640 1 -- 172.21.15.59:0/4186706063 wait complete. 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf28eb5640 1 Processor -- start 2024-03-06T07:27:15.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf28eb5640 1 -- start start 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf28eb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf240cf830 0x7fbf241f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf28eb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf240d27f0 0x7fbf240cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf28eb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf2415dfe0 0x7fbf240b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf28eb5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240d1c70 con 0x7fbf240d27f0 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf28eb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240b39f0 con 0x7fbf240cf830 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf28eb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240d1df0 con 0x7fbf2415dfe0 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf1bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf240d27f0 0x7fbf240cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf1bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf240d27f0 0x7fbf240cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37232/0 (socket says 172.21.15.59:37232) 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf1bfff640 1 -- 172.21.15.59:0/3011197534 learned_addr learned my addr 172.21.15.59:0/3011197534 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf22d9d640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf2415dfe0 0x7fbf240b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.429+0000 7fbf2259c640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf240cf830 0x7fbf241f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.430+0000 7fbf1bfff640 1 -- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf2415dfe0 msgr2=0x7fbf240b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.430+0000 7fbf1bfff640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf2415dfe0 0x7fbf240b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.430+0000 7fbf1bfff640 1 -- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf240cf830 msgr2=0x7fbf241f71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.430+0000 7fbf1bfff640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf240cf830 0x7fbf241f71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.430+0000 7fbf1bfff640 1 -- 172.21.15.59:0/3011197534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240b8970 con 0x7fbf240d27f0 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.430+0000 7fbf1bfff640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf240d27f0 0x7fbf240cbdf0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c00e860 tx=0x7fbf0c00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.430+0000 7fbf1affd640 1 -- 172.21.15.59:0/3011197534 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c017070 con 0x7fbf240d27f0 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.430+0000 7fbf1affd640 1 -- 172.21.15.59:0/3011197534 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0c00c940 con 0x7fbf240d27f0 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.430+0000 7fbf28eb5640 1 -- 172.21.15.59:0/3011197534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf241f7910 con 0x7fbf240d27f0 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.430+0000 7fbf1affd640 1 -- 172.21.15.59:0/3011197534 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c00cd10 con 0x7fbf240d27f0 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.430+0000 7fbf28eb5640 1 -- 172.21.15.59:0/3011197534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf241f7ed0 con 0x7fbf240d27f0 2024-03-06T07:27:15.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.431+0000 7fbf28eb5640 1 -- 172.21.15.59:0/3011197534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2404f330 con 0x7fbf240d27f0 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.431+0000 7fbf1affd640 1 -- 172.21.15.59:0/3011197534 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf0c01a070 con 0x7fbf240d27f0 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.432+0000 7fbf1affd640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf0406b500 0x7fbf0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.432+0000 7fbf1affd640 1 -- 172.21.15.59:0/3011197534 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf0c08a0e0 con 0x7fbf240d27f0 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.432+0000 7fbf2259c640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf0406b500 0x7fbf0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.432+0000 7fbf2259c640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf0406b500 0x7fbf0406d9c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbf24157b90 tx=0x7fbf14009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.436+0000 7fbf1affd640 1 -- 172.21.15.59:0/3011197534 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf240b8970 con 0x7fbf240d27f0 2024-03-06T07:27:15.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.616+0000 7fbf28eb5640 1 -- 172.21.15.59:0/3011197534 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf240ce870 con 0x7fbf240d27f0 2024-03-06T07:27:15.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.617+0000 7fbf1affd640 1 -- 172.21.15.59:0/3011197534 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16628 (secure 0 0 0) 0x7fbf0c058a60 con 0x7fbf240d27f0 2024-03-06T07:27:15.619 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:15.620 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:27:14.344872+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:12.625799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:13.306376+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.619+0000 7fbf28eb5640 1 -- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf0406b500 msgr2=0x7fbf0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.619+0000 7fbf28eb5640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf0406b500 0x7fbf0406d9c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbf24157b90 tx=0x7fbf14009190 comp rx=0 tx=0).stop 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.619+0000 7fbf28eb5640 1 -- 172.21.15.59:0/3011197534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf240d27f0 msgr2=0x7fbf240cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.619+0000 7fbf28eb5640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf240d27f0 0x7fbf240cbdf0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c00e860 tx=0x7fbf0c00ed20 comp rx=0 tx=0).stop 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.619+0000 7fbf28eb5640 1 -- 172.21.15.59:0/3011197534 shutdown_connections 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.619+0000 7fbf28eb5640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf0406b500 0x7fbf0406d9c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.619+0000 7fbf28eb5640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf2415dfe0 0x7fbf240b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.619+0000 7fbf28eb5640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf240d27f0 0x7fbf240cbdf0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.619+0000 7fbf28eb5640 1 --2- 172.21.15.59:0/3011197534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf240cf830 0x7fbf241f71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.619+0000 7fbf28eb5640 1 -- 172.21.15.59:0/3011197534 >> 172.21.15.59:0/3011197534 conn(0x7fbf240d3cd0 msgr2=0x7fbf240cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.620+0000 7fbf28eb5640 1 -- 172.21.15.59:0/3011197534 shutdown_connections 2024-03-06T07:27:15.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.620+0000 7fbf28eb5640 1 -- 172.21.15.59:0/3011197534 wait complete. 2024-03-06T07:27:15.631 INFO:tasks.ceph.mds.b.smithi164.stderr:2024-03-06T07:27:15.629+0000 7fa3dd3a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:15.633 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:15.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.735+0000 7f3090193640 1 Processor -- start 2024-03-06T07:27:15.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.735+0000 7f3090193640 1 -- start start 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.736+0000 7f3090193640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30880cf9a0 0x7f30880b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.736+0000 7f3090193640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f30880b55e0 0x7f30880ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.736+0000 7f3090193640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308806b710 con 0x7f3088066d90 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.736+0000 7f3090193640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308806b150 con 0x7f30880b55e0 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.736+0000 7f3090193640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308806b430 con 0x7f30880cf9a0 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.736+0000 7f308d72e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3088066d90 0x7f30880b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60810/0 (socket says 172.21.15.59:60810) 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.736+0000 7f308d72e640 1 -- 172.21.15.59:0/2650707930 learned_addr learned my addr 172.21.15.59:0/2650707930 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.736+0000 7f308df2f640 1 --2- 172.21.15.59:0/2650707930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30880cf9a0 0x7f30880b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.736+0000 7f308e730640 1 --2- 172.21.15.59:0/2650707930 >> v2:172.21.15.164:3300/0 conn(0x7f30880b55e0 0x7f30880ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.737+0000 7f308cf2d640 1 -- 172.21.15.59:0/2650707930 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579263803 0 0) 0x7f308806b710 con 0x7f3088066d90 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.737+0000 7f308cf2d640 1 -- 172.21.15.59:0/2650707930 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3070003520 con 0x7f3088066d90 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.737+0000 7f308e730640 1 -- 172.21.15.59:0/2650707930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30880cf9a0 msgr2=0x7f30880b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.737+0000 7f308e730640 1 --2- 172.21.15.59:0/2650707930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30880cf9a0 0x7f30880b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.737+0000 7f308e730640 1 -- 172.21.15.59:0/2650707930 >> v1:172.21.15.59:6789/0 conn(0x7f3088066d90 legacy=0x7f30880b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.737+0000 7f308e730640 1 -- 172.21.15.59:0/2650707930 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30880b0970 con 0x7f30880b55e0 2024-03-06T07:27:15.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.737+0000 7f308df2f640 1 --2- 172.21.15.59:0/2650707930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30880cf9a0 0x7f30880b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f308e730640 1 --2- 172.21.15.59:0/2650707930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30880b55e0 0x7f30880ce040 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f30780097a0 tx=0x7f307802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86e385536b009a8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f308cf2d640 1 -- 172.21.15.59:0/2650707930 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078035020 con 0x7f30880b55e0 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f308cf2d640 1 -- 172.21.15.59:0/2650707930 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3078031830 con 0x7f30880b55e0 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f308cf2d640 1 -- 172.21.15.59:0/2650707930 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078031be0 con 0x7f30880b55e0 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f3090193640 1 -- 172.21.15.59:0/2650707930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30880b55e0 msgr2=0x7f30880ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f3090193640 1 --2- 172.21.15.59:0/2650707930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30880b55e0 0x7f30880ce040 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f30780097a0 tx=0x7f307802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f3090193640 1 -- 172.21.15.59:0/2650707930 shutdown_connections 2024-03-06T07:27:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f3090193640 1 --2- 172.21.15.59:0/2650707930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30880b55e0 0x7f30880ce040 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f3090193640 1 --2- 172.21.15.59:0/2650707930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30880cf9a0 0x7f30880b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f3090193640 1 -- 172.21.15.59:0/2650707930 >> 172.21.15.59:0/2650707930 conn(0x7f30880b1d70 msgr2=0x7f30880b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f3090193640 1 -- 172.21.15.59:0/2650707930 shutdown_connections 2024-03-06T07:27:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.738+0000 7f3090193640 1 -- 172.21.15.59:0/2650707930 wait complete. 2024-03-06T07:27:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.739+0000 7f3090193640 1 Processor -- start 2024-03-06T07:27:15.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.739+0000 7f3090193640 1 -- start start 2024-03-06T07:27:15.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.740+0000 7f3090193640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3088066d90 0x7f30881f72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.740+0000 7f3090193640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30880b55e0 0x7f30881f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.740+0000 7f3090193640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30880cf9a0 0x7f308815fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.740+0000 7f3090193640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30880b6170 con 0x7f30880b55e0 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.740+0000 7f3090193640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30880b5ff0 con 0x7f30880cf9a0 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.740+0000 7f3090193640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30880cfd80 con 0x7f3088066d90 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.740+0000 7f308d72e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30880b55e0 0x7f30881f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.740+0000 7f308d72e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30880b55e0 0x7f30881f4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37248/0 (socket says 172.21.15.59:37248) 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.740+0000 7f308d72e640 1 -- 172.21.15.59:0/3446540170 learned_addr learned my addr 172.21.15.59:0/3446540170 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.740+0000 7f308df2f640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3088066d90 0x7f30881f72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.740+0000 7f308e730640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30880cf9a0 0x7f308815fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.741+0000 7f308d72e640 1 -- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3088066d90 msgr2=0x7f30881f72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.741+0000 7f308d72e640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3088066d90 0x7f30881f72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.741+0000 7f308d72e640 1 -- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30880cf9a0 msgr2=0x7f308815fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.741+0000 7f308d72e640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30880cf9a0 0x7f308815fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.741+0000 7f308d72e640 1 -- 172.21.15.59:0/3446540170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30881f81a0 con 0x7f30880b55e0 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.741+0000 7f308d72e640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30880b55e0 0x7f30881f4720 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f307c00c400 tx=0x7f307c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.741+0000 7f30867fc640 1 -- 172.21.15.59:0/3446540170 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c017070 con 0x7f30880b55e0 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.741+0000 7f30867fc640 1 -- 172.21.15.59:0/3446540170 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307c005d00 con 0x7f30880b55e0 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.741+0000 7f3090193640 1 -- 172.21.15.59:0/3446540170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30881f7900 con 0x7f30880b55e0 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.741+0000 7f30867fc640 1 -- 172.21.15.59:0/3446540170 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c007310 con 0x7f30880b55e0 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.741+0000 7f3090193640 1 -- 172.21.15.59:0/3446540170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30881f7ec0 con 0x7f30880b55e0 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.742+0000 7f3090193640 1 -- 172.21.15.59:0/3446540170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308809e3d0 con 0x7f30880b55e0 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.742+0000 7f30867fc640 1 -- 172.21.15.59:0/3446540170 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f307c01a070 con 0x7f30880b55e0 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.743+0000 7f30867fc640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f307406b4d0 0x7f307406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.743+0000 7f30867fc640 1 -- 172.21.15.59:0/3446540170 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f307c089550 con 0x7f30880b55e0 2024-03-06T07:27:15.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.743+0000 7f308df2f640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f307406b4d0 0x7f307406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.743+0000 7f308df2f640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f307406b4d0 0x7f307406d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3080009bd0 tx=0x7f308000b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.747+0000 7f30867fc640 1 -- 172.21.15.59:0/3446540170 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f307c057e10 con 0x7f30880b55e0 2024-03-06T07:27:15.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:15.965+0000 7f3090193640 1 -- 172.21.15.59:0/3446540170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3088068ff0 con 0x7f30880b55e0 2024-03-06T07:27:16.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.076+0000 7f30867fc640 1 -- 172.21.15.59:0/3446540170 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v65) v1 ==== 118+0+0 (secure 0 0 0) 0x7f307c05beb0 con 0x7f30880b55e0 2024-03-06T07:27:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 -- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f307406b4d0 msgr2=0x7f307406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f307406b4d0 0x7f307406d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3080009bd0 tx=0x7f308000b040 comp rx=0 tx=0).stop 2024-03-06T07:27:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 -- 172.21.15.59:0/3446540170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30880b55e0 msgr2=0x7f30881f4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30880b55e0 0x7f30881f4720 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f307c00c400 tx=0x7f307c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 -- 172.21.15.59:0/3446540170 shutdown_connections 2024-03-06T07:27:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f307406b4d0 0x7f307406d990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30880cf9a0 0x7f308815fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30880b55e0 0x7f30881f4720 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 --2- 172.21.15.59:0/3446540170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3088066d90 0x7f30881f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 -- 172.21.15.59:0/3446540170 >> 172.21.15.59:0/3446540170 conn(0x7f30880b1d70 msgr2=0x7f308806de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 -- 172.21.15.59:0/3446540170 shutdown_connections 2024-03-06T07:27:16.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.078+0000 7f3090193640 1 -- 172.21.15.59:0/3446540170 wait complete. 2024-03-06T07:27:16.091 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:16.091 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:16.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.193+0000 7f33ed8f4640 1 Processor -- start 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.193+0000 7f33ed8f4640 1 -- start start 2024-03-06T07:27:16.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33ed8f4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f33e80cea50 0x7f33e80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33ed8f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33e806cab0 0x7f33e80ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33ed8f4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e806d030 con 0x7f33e815def0 2024-03-06T07:27:16.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33ed8f4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e80cf1c0 con 0x7f33e80cea50 2024-03-06T07:27:16.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33ed8f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e80cc760 con 0x7f33e806cab0 2024-03-06T07:27:16.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33dffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f33e80cea50 0x7f33e80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33dffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f33e80cea50 0x7f33e80b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37260/0 (socket says 172.21.15.59:37260) 2024-03-06T07:27:16.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33dffff640 1 -- 172.21.15.59:0/3976200038 learned_addr learned my addr 172.21.15.59:0/3976200038 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:16.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33e77fe640 1 --2- 172.21.15.59:0/3976200038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33e806cab0 0x7f33e80ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33dffff640 1 -- 172.21.15.59:0/3976200038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33e806cab0 msgr2=0x7f33e80ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33dffff640 1 --2- 172.21.15.59:0/3976200038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33e806cab0 0x7f33e80ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33dffff640 1 -- 172.21.15.59:0/3976200038 >> v1:172.21.15.164:6789/0 conn(0x7f33e815def0 legacy=0x7f33e80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.194+0000 7f33dffff640 1 -- 172.21.15.59:0/3976200038 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e806d310 con 0x7f33e80cea50 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.195+0000 7f33dffff640 1 --2- 172.21.15.59:0/3976200038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33e80cea50 0x7f33e80b41b0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009870 tx=0x7f33d002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29e4dcb239528ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.195+0000 7f33e67fc640 1 -- 172.21.15.59:0/3976200038 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d002f840 con 0x7f33e80cea50 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.195+0000 7f33e67fc640 1 -- 172.21.15.59:0/3976200038 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d002f9e0 con 0x7f33e80cea50 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.195+0000 7f33e67fc640 1 -- 172.21.15.59:0/3976200038 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d002fdb0 con 0x7f33e80cea50 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.196+0000 7f33ed8f4640 1 -- 172.21.15.59:0/3976200038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33e80cea50 msgr2=0x7f33e80b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.196+0000 7f33ed8f4640 1 --2- 172.21.15.59:0/3976200038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33e80cea50 0x7f33e80b41b0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009870 tx=0x7f33d002ba20 comp rx=0 tx=0).stop 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.196+0000 7f33ed8f4640 1 -- 172.21.15.59:0/3976200038 shutdown_connections 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.196+0000 7f33ed8f4640 1 --2- 172.21.15.59:0/3976200038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33e806cab0 0x7f33e80ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.196+0000 7f33ed8f4640 1 --2- 172.21.15.59:0/3976200038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33e80cea50 0x7f33e80b41b0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.196+0000 7f33ed8f4640 1 -- 172.21.15.59:0/3976200038 >> 172.21.15.59:0/3976200038 conn(0x7f33e80b3690 msgr2=0x7f33e80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.196+0000 7f33ed8f4640 1 -- 172.21.15.59:0/3976200038 shutdown_connections 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.196+0000 7f33ed8f4640 1 -- 172.21.15.59:0/3976200038 wait complete. 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.197+0000 7f33ed8f4640 1 Processor -- start 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.197+0000 7f33ed8f4640 1 -- start start 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33ed8f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33e806cab0 0x7f33e815bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33ed8f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33e80cea50 0x7f33e80c7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33ed8f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33e815def0 0x7f33e80cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33ed8f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e815e2d0 con 0x7f33e80cea50 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33ed8f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e80cc3f0 con 0x7f33e815def0 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33ed8f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e815e450 con 0x7f33e806cab0 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33dffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33e80cea50 0x7f33e80c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33dffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33e80cea50 0x7f33e80c7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37276/0 (socket says 172.21.15.59:37276) 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33dffff640 1 -- 172.21.15.59:0/1150312743 learned_addr learned my addr 172.21.15.59:0/1150312743 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:16.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33e6ffd640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33e806cab0 0x7f33e815bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33e77fe640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33e815def0 0x7f33e80cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33dffff640 1 -- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33e806cab0 msgr2=0x7f33e815bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33dffff640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33e806cab0 0x7f33e815bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33dffff640 1 -- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33e815def0 msgr2=0x7f33e80cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33dffff640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33e815def0 0x7f33e80cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33dffff640 1 -- 172.21.15.59:0/1150312743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e80c5540 con 0x7f33e80cea50 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33dffff640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33e80cea50 0x7f33e80c7090 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f33d0000c00 tx=0x7f33d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33e4ff9640 1 -- 172.21.15.59:0/1150312743 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0037bc0 con 0x7f33e80cea50 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33e4ff9640 1 -- 172.21.15.59:0/1150312743 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d0046070 con 0x7f33e80cea50 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33ed8f4640 1 -- 172.21.15.59:0/1150312743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e80c5820 con 0x7f33e80cea50 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33e4ff9640 1 -- 172.21.15.59:0/1150312743 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0041400 con 0x7f33e80cea50 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.198+0000 7f33ed8f4640 1 -- 172.21.15.59:0/1150312743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e80c5de0 con 0x7f33e80cea50 2024-03-06T07:27:16.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.199+0000 7f33ed8f4640 1 -- 172.21.15.59:0/1150312743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e809fcf0 con 0x7f33e80cea50 2024-03-06T07:27:16.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.200+0000 7f33e4ff9640 1 -- 172.21.15.59:0/1150312743 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33d0040070 con 0x7f33e80cea50 2024-03-06T07:27:16.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.200+0000 7f33e4ff9640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f33c806b500 0x7f33c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.201+0000 7f33e4ff9640 1 -- 172.21.15.59:0/1150312743 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33d00ac770 con 0x7f33e80cea50 2024-03-06T07:27:16.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.201+0000 7f33e6ffd640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f33c806b500 0x7f33c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.201+0000 7f33e6ffd640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f33c806b500 0x7f33c806d9c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f33e80cb950 tx=0x7f33d8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.205+0000 7f33e4ff9640 1 -- 172.21.15.59:0/1150312743 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33d0033030 con 0x7f33e80cea50 2024-03-06T07:27:16.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:16.423+0000 7f33ed8f4640 1 -- 172.21.15.59:0/1150312743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f33e8068ff0 con 0x7f33e80cea50 2024-03-06T07:27:17.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.077+0000 7f33e4ff9640 1 -- 172.21.15.59:0/1150312743 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v66) v1 ==== 122+0+0 (secure 0 0 0) 0x7f33d007af10 con 0x7f33e80cea50 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.079+0000 7f33ed8f4640 1 -- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f33c806b500 msgr2=0x7f33c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.079+0000 7f33ed8f4640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f33c806b500 0x7f33c806d9c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f33e80cb950 tx=0x7f33d8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.079+0000 7f33ed8f4640 1 -- 172.21.15.59:0/1150312743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33e80cea50 msgr2=0x7f33e80c7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.079+0000 7f33ed8f4640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33e80cea50 0x7f33e80c7090 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f33d0000c00 tx=0x7f33d0004730 comp rx=0 tx=0).stop 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.080+0000 7f33ed8f4640 1 -- 172.21.15.59:0/1150312743 shutdown_connections 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.080+0000 7f33ed8f4640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f33c806b500 0x7f33c806d9c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.080+0000 7f33ed8f4640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33e815def0 0x7f33e80cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.080+0000 7f33ed8f4640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33e80cea50 0x7f33e80c7090 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.080+0000 7f33ed8f4640 1 --2- 172.21.15.59:0/1150312743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33e806cab0 0x7f33e815bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.080+0000 7f33ed8f4640 1 -- 172.21.15.59:0/1150312743 >> 172.21.15.59:0/1150312743 conn(0x7f33e80b3690 msgr2=0x7f33e815e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.080+0000 7f33ed8f4640 1 -- 172.21.15.59:0/1150312743 shutdown_connections 2024-03-06T07:27:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.080+0000 7f33ed8f4640 1 -- 172.21.15.59:0/1150312743 wait complete. 2024-03-06T07:27:17.095 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:17.095 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:17.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.201+0000 7efcff593640 1 Processor -- start 2024-03-06T07:27:17.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.201+0000 7efcff593640 1 -- start start 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.202+0000 7efcff593640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efcf80d0320 0x7efcf80b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.202+0000 7efcff593640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcf80d0700 0x7efcf80b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.202+0000 7efcff593640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efcf80ccae0 0x7efcf80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.202+0000 7efcff593640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf80b04e0 con 0x7efcf80ccae0 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.202+0000 7efcff593640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf815c270 con 0x7efcf80d0320 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.202+0000 7efcff593640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf80d0e90 con 0x7efcf80d0700 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.202+0000 7efcfdb30640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efcf80ccae0 0x7efcf80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.202+0000 7efcfdb30640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efcf80ccae0 0x7efcf80ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37286/0 (socket says 172.21.15.59:37286) 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.202+0000 7efcfdb30640 1 -- 172.21.15.59:0/2480537975 learned_addr learned my addr 172.21.15.59:0/2480537975 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.202+0000 7efcfcb2e640 1 --2- 172.21.15.59:0/2480537975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcf80d0700 0x7efcf80b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.202+0000 7efcfd32f640 1 --2- 172.21.15.59:0/2480537975 >> v2:172.21.15.164:3300/0 conn(0x7efcf80d0320 0x7efcf80b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.203+0000 7efcfdb30640 1 -- 172.21.15.59:0/2480537975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcf80d0700 msgr2=0x7efcf80b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.203+0000 7efcfdb30640 1 --2- 172.21.15.59:0/2480537975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcf80d0700 0x7efcf80b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.203+0000 7efcfdb30640 1 -- 172.21.15.59:0/2480537975 >> v2:172.21.15.164:3300/0 conn(0x7efcf80d0320 msgr2=0x7efcf80b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.203+0000 7efcfdb30640 1 --2- 172.21.15.59:0/2480537975 >> v2:172.21.15.164:3300/0 conn(0x7efcf80d0320 0x7efcf80b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.203+0000 7efcfdb30640 1 -- 172.21.15.59:0/2480537975 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf80b5ae0 con 0x7efcf80ccae0 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.203+0000 7efcfdb30640 1 --2- 172.21.15.59:0/2480537975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf80ccae0 0x7efcf80ccf60 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7efcf80715b0 tx=0x7efcf402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d130bfbe6a53362 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.203+0000 7efceffff640 1 -- 172.21.15.59:0/2480537975 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf402cb00 con 0x7efcf80ccae0 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.204+0000 7efceffff640 1 -- 172.21.15.59:0/2480537975 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf402cca0 con 0x7efcf80ccae0 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.204+0000 7efceffff640 1 -- 172.21.15.59:0/2480537975 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4034610 con 0x7efcf80ccae0 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.204+0000 7efcff593640 1 -- 172.21.15.59:0/2480537975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf80ccae0 msgr2=0x7efcf80ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.204+0000 7efcff593640 1 --2- 172.21.15.59:0/2480537975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf80ccae0 0x7efcf80ccf60 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7efcf80715b0 tx=0x7efcf402bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.204+0000 7efcff593640 1 -- 172.21.15.59:0/2480537975 shutdown_connections 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.204+0000 7efcff593640 1 --2- 172.21.15.59:0/2480537975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf80ccae0 0x7efcf80ccf60 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.204+0000 7efcff593640 1 --2- 172.21.15.59:0/2480537975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcf80d0700 0x7efcf80b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.204+0000 7efcff593640 1 --2- 172.21.15.59:0/2480537975 >> v2:172.21.15.164:3300/0 conn(0x7efcf80d0320 0x7efcf80b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.204+0000 7efcff593640 1 -- 172.21.15.59:0/2480537975 >> 172.21.15.59:0/2480537975 conn(0x7efcf80724d0 msgr2=0x7efcf80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.204+0000 7efcff593640 1 -- 172.21.15.59:0/2480537975 shutdown_connections 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.204+0000 7efcff593640 1 -- 172.21.15.59:0/2480537975 wait complete. 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.205+0000 7efcff593640 1 Processor -- start 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.205+0000 7efcff593640 1 -- start start 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcff593640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcf80ccae0 0x7efcf809a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcff593640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcf80d0320 0x7efcf809aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcff593640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf80d0700 0x7efcf81f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcff593640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf80b60a0 con 0x7efcf80d0700 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcff593640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf80b0170 con 0x7efcf80d0320 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcff593640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf80ca790 con 0x7efcf80ccae0 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcfdb30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf80d0700 0x7efcf81f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcfdb30640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf80d0700 0x7efcf81f7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37288/0 (socket says 172.21.15.59:37288) 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcfdb30640 1 -- 172.21.15.59:0/808069899 learned_addr learned my addr 172.21.15.59:0/808069899 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcfd32f640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcf80ccae0 0x7efcf809a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcfcb2e640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcf80d0320 0x7efcf809aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcfdb30640 1 -- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcf80ccae0 msgr2=0x7efcf809a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcfdb30640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcf80ccae0 0x7efcf809a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcfdb30640 1 -- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcf80d0320 msgr2=0x7efcf809aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcfdb30640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcf80d0320 0x7efcf809aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.206+0000 7efcfdb30640 1 -- 172.21.15.59:0/808069899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf806e380 con 0x7efcf80d0700 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.207+0000 7efcfdb30640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf80d0700 0x7efcf81f7750 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7efcf4036040 tx=0x7efcf4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.207+0000 7efcedffb640 1 -- 172.21.15.59:0/808069899 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4043020 con 0x7efcf80d0700 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.207+0000 7efcedffb640 1 -- 172.21.15.59:0/808069899 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf402ce50 con 0x7efcf80d0700 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.207+0000 7efcff593640 1 -- 172.21.15.59:0/808069899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf81f41c0 con 0x7efcf80d0700 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.207+0000 7efcedffb640 1 -- 172.21.15.59:0/808069899 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf4009510 con 0x7efcf80d0700 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.207+0000 7efcff593640 1 -- 172.21.15.59:0/808069899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf81f7250 con 0x7efcf80d0700 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.208+0000 7efcff593640 1 -- 172.21.15.59:0/808069899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf804f330 con 0x7efcf80d0700 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.209+0000 7efcedffb640 1 -- 172.21.15.59:0/808069899 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcf4046020 con 0x7efcf80d0700 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.209+0000 7efcedffb640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efcdc06b4d0 0x7efcdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.209+0000 7efcedffb640 1 -- 172.21.15.59:0/808069899 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcf40ac600 con 0x7efcf80d0700 2024-03-06T07:27:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.209+0000 7efcfd32f640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efcdc06b4d0 0x7efcdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.210+0000 7efcfd32f640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efcdc06b4d0 0x7efcdc06d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efcf80ca000 tx=0x7efcf0009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.213+0000 7efcedffb640 1 -- 172.21.15.59:0/808069899 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcf404d030 con 0x7efcf80d0700 2024-03-06T07:27:17.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.439+0000 7efcff593640 1 -- 172.21.15.59:0/808069899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcf80aedd0 con 0x7efcf80d0700 2024-03-06T07:27:17.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.439+0000 7efcedffb640 1 -- 172.21.15.59:0/808069899 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4979 (secure 0 0 0) 0x7efcf81f41c0 con 0x7efcf80d0700 2024-03-06T07:27:17.441 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:17.441 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3012766532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3012766532},{"type":"v1","addr":"172.21.15.164:6837","nonce":3012766532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/930220384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":930220384},{"type":"v1","addr":"172.21.15.59:6834","nonce":930220384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/557371287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":557371287},{"type":"v1","addr":"172.21.15.59:6835","nonce":557371287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:16.424559+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6836/2139041464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2139041464},{"type":"v1","addr":"172.21.15.164:6836","nonce":2139041464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":8}]} 2024-03-06T07:27:17.442 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 66 2024-03-06T07:27:17.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 -- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efcdc06b4d0 msgr2=0x7efcdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efcdc06b4d0 0x7efcdc06d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efcf80ca000 tx=0x7efcf0009190 comp rx=0 tx=0).stop 2024-03-06T07:27:17.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 -- 172.21.15.59:0/808069899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf80d0700 msgr2=0x7efcf81f7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf80d0700 0x7efcf81f7750 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7efcf4036040 tx=0x7efcf4004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:17.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 -- 172.21.15.59:0/808069899 shutdown_connections 2024-03-06T07:27:17.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efcdc06b4d0 0x7efcdc06d990 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efcf80d0700 0x7efcf81f7750 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcf80d0320 0x7efcf809aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 --2- 172.21.15.59:0/808069899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcf80ccae0 0x7efcf809a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 -- 172.21.15.59:0/808069899 >> 172.21.15.59:0/808069899 conn(0x7efcf80724d0 msgr2=0x7efcf80728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 -- 172.21.15.59:0/808069899 shutdown_connections 2024-03-06T07:27:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.442+0000 7efcff593640 1 -- 172.21.15.59:0/808069899 wait complete. 2024-03-06T07:27:17.455 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:17.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.554+0000 7ff8875a0640 1 Processor -- start 2024-03-06T07:27:17.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.554+0000 7ff8875a0640 1 -- start start 2024-03-06T07:27:17.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.555+0000 7ff8875a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff8800cfff0 0x7ff8800d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.555+0000 7ff8875a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8800cc8b0 0x7ff8800ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.555+0000 7ff8875a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff880161980 con 0x7ff88015bb20 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.555+0000 7ff8875a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8800d2500 con 0x7ff8800cfff0 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.555+0000 7ff8875a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff880161c60 con 0x7ff8800cc8b0 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.555+0000 7ff884b3b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff8800cfff0 0x7ff8800d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.555+0000 7ff88533c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff88015bb20 0x7ff8800d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:55784/0 (socket says 172.21.15.59:55784) 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.555+0000 7ff88533c640 1 -- 172.21.15.59:0/9390746 learned_addr learned my addr 172.21.15.59:0/9390746 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.555+0000 7ff884b3b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff8800cfff0 0x7ff8800d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:50800/0 (socket says 172.21.15.59:50800) 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.555+0000 7ff885b3d640 1 --2- 172.21.15.59:0/9390746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8800cc8b0 0x7ff8800ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.556+0000 7ff884b3b640 1 -- 172.21.15.59:0/9390746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8800cc8b0 msgr2=0x7ff8800ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.556+0000 7ff884b3b640 1 --2- 172.21.15.59:0/9390746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8800cc8b0 0x7ff8800ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.556+0000 7ff884b3b640 1 -- 172.21.15.59:0/9390746 >> v1:172.21.15.164:6789/0 conn(0x7ff88015bb20 legacy=0x7ff8800d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.556+0000 7ff884b3b640 1 -- 172.21.15.59:0/9390746 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff880161f40 con 0x7ff8800cfff0 2024-03-06T07:27:17.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.556+0000 7ff87ffff640 1 -- 172.21.15.59:0/9390746 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118720259 0 0) 0x7ff880161980 con 0x7ff88015bb20 2024-03-06T07:27:17.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.556+0000 7ff884b3b640 1 --2- 172.21.15.59:0/9390746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8800cfff0 0x7ff8800d04a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff8740097a0 tx=0x7ff87402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c48f7258b8fdb131 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.557+0000 7ff87ffff640 1 -- 172.21.15.59:0/9390746 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874035020 con 0x7ff8800cfff0 2024-03-06T07:27:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.557+0000 7ff87ffff640 1 -- 172.21.15.59:0/9390746 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff874031830 con 0x7ff8800cfff0 2024-03-06T07:27:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.557+0000 7ff87ffff640 1 -- 172.21.15.59:0/9390746 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874031c00 con 0x7ff8800cfff0 2024-03-06T07:27:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.557+0000 7ff8875a0640 1 -- 172.21.15.59:0/9390746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8800cfff0 msgr2=0x7ff8800d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.557+0000 7ff8875a0640 1 --2- 172.21.15.59:0/9390746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8800cfff0 0x7ff8800d04a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff8740097a0 tx=0x7ff87402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.557+0000 7ff8875a0640 1 -- 172.21.15.59:0/9390746 shutdown_connections 2024-03-06T07:27:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.557+0000 7ff8875a0640 1 --2- 172.21.15.59:0/9390746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8800cc8b0 0x7ff8800ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.557+0000 7ff8875a0640 1 --2- 172.21.15.59:0/9390746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8800cfff0 0x7ff8800d04a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.557+0000 7ff8875a0640 1 -- 172.21.15.59:0/9390746 >> 172.21.15.59:0/9390746 conn(0x7ff88009a9f0 msgr2=0x7ff88009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.557+0000 7ff8875a0640 1 -- 172.21.15.59:0/9390746 shutdown_connections 2024-03-06T07:27:17.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.557+0000 7ff8875a0640 1 -- 172.21.15.59:0/9390746 wait complete. 2024-03-06T07:27:17.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.558+0000 7ff8875a0640 1 Processor -- start 2024-03-06T07:27:17.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.558+0000 7ff8875a0640 1 -- start start 2024-03-06T07:27:17.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.559+0000 7ff8875a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8800cc8b0 0x7ff88015a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.559+0000 7ff8875a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8800cfff0 0x7ff8800ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.559+0000 7ff8875a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff88015bb20 0x7ff88015d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.559+0000 7ff8875a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800d1980 con 0x7ff88015bb20 2024-03-06T07:27:17.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.559+0000 7ff8875a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff880162500 con 0x7ff8800cc8b0 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.559+0000 7ff8875a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800d1b00 con 0x7ff8800cfff0 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.559+0000 7ff885b3d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff88015bb20 0x7ff88015d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.559+0000 7ff885b3d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff88015bb20 0x7ff88015d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37300/0 (socket says 172.21.15.59:37300) 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.559+0000 7ff885b3d640 1 -- 172.21.15.59:0/3938726670 learned_addr learned my addr 172.21.15.59:0/3938726670 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.559+0000 7ff88533c640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8800cc8b0 0x7ff88015a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.559+0000 7ff884b3b640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8800cfff0 0x7ff8800ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.560+0000 7ff885b3d640 1 -- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8800cfff0 msgr2=0x7ff8800ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.560+0000 7ff885b3d640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8800cfff0 0x7ff8800ca9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.560+0000 7ff885b3d640 1 -- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8800cc8b0 msgr2=0x7ff88015a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.560+0000 7ff885b3d640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8800cc8b0 0x7ff88015a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.560+0000 7ff885b3d640 1 -- 172.21.15.59:0/3938726670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8800c68e0 con 0x7ff88015bb20 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.560+0000 7ff885b3d640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff88015bb20 0x7ff88015d440 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7ff87800e7f0 tx=0x7ff87800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.560+0000 7ff87dffb640 1 -- 172.21.15.59:0/3938726670 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878017070 con 0x7ff88015bb20 2024-03-06T07:27:17.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.560+0000 7ff87dffb640 1 -- 172.21.15.59:0/3938726670 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87800ee40 con 0x7ff88015bb20 2024-03-06T07:27:17.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.560+0000 7ff8875a0640 1 -- 172.21.15.59:0/3938726670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8800c5840 con 0x7ff88015bb20 2024-03-06T07:27:17.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.560+0000 7ff87dffb640 1 -- 172.21.15.59:0/3938726670 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87800cb80 con 0x7ff88015bb20 2024-03-06T07:27:17.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.560+0000 7ff8875a0640 1 -- 172.21.15.59:0/3938726670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8800c5e00 con 0x7ff88015bb20 2024-03-06T07:27:17.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.561+0000 7ff8875a0640 1 -- 172.21.15.59:0/3938726670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88004f330 con 0x7ff88015bb20 2024-03-06T07:27:17.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.562+0000 7ff87dffb640 1 -- 172.21.15.59:0/3938726670 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff87801a070 con 0x7ff88015bb20 2024-03-06T07:27:17.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.562+0000 7ff87dffb640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff86c06b4d0 0x7ff86c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.562+0000 7ff87dffb640 1 -- 172.21.15.59:0/3938726670 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff878089ff0 con 0x7ff88015bb20 2024-03-06T07:27:17.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.562+0000 7ff88533c640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff86c06b4d0 0x7ff86c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.563+0000 7ff88533c640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff86c06b4d0 0x7ff86c06d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff8700012e0 tx=0x7ff870001630 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.566+0000 7ff87dffb640 1 -- 172.21.15.59:0/3938726670 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8780587e0 con 0x7ff88015bb20 2024-03-06T07:27:17.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.748+0000 7ff8875a0640 1 -- 172.21.15.59:0/3938726670 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8800b1820 con 0x7ff88015bb20 2024-03-06T07:27:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.749+0000 7ff87dffb640 1 -- 172.21.15.59:0/3938726670 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16615 (secure 0 0 0) 0x7ff87805c880 con 0x7ff88015bb20 2024-03-06T07:27:17.751 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:17.752 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:27:15.985080+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:12.625799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:13.306376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.751+0000 7ff8875a0640 1 -- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff86c06b4d0 msgr2=0x7ff86c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.751+0000 7ff8875a0640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff86c06b4d0 0x7ff86c06d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff8700012e0 tx=0x7ff870001630 comp rx=0 tx=0).stop 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.751+0000 7ff8875a0640 1 -- 172.21.15.59:0/3938726670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff88015bb20 msgr2=0x7ff88015d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.751+0000 7ff8875a0640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff88015bb20 0x7ff88015d440 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7ff87800e7f0 tx=0x7ff87800ebf0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.751+0000 7ff8875a0640 1 -- 172.21.15.59:0/3938726670 shutdown_connections 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.751+0000 7ff8875a0640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff86c06b4d0 0x7ff86c06d990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.751+0000 7ff8875a0640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff88015bb20 0x7ff88015d440 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.751+0000 7ff8875a0640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8800cfff0 0x7ff8800ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.751+0000 7ff8875a0640 1 --2- 172.21.15.59:0/3938726670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8800cc8b0 0x7ff88015a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.751+0000 7ff8875a0640 1 -- 172.21.15.59:0/3938726670 >> 172.21.15.59:0/3938726670 conn(0x7ff88009a9f0 msgr2=0x7ff88006fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.752+0000 7ff8875a0640 1 -- 172.21.15.59:0/3938726670 shutdown_connections 2024-03-06T07:27:17.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.752+0000 7ff8875a0640 1 -- 172.21.15.59:0/3938726670 wait complete. 2024-03-06T07:27:17.765 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:17.765 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:17.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.864+0000 7f97c8b2c640 1 Processor -- start 2024-03-06T07:27:17.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.864+0000 7f97c8b2c640 1 -- start start 2024-03-06T07:27:17.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.865+0000 7f97c8b2c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f97c415fa10 0x7f97c40ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.865+0000 7f97c8b2c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f97c40d16e0 0x7f97c40d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.865+0000 7f97c8b2c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c40cebe0 con 0x7f97c415bdd0 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.865+0000 7f97c8b2c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c40ceec0 con 0x7f97c40d16e0 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.865+0000 7f97c8b2c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c40cf1a0 con 0x7f97c415fa10 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.865+0000 7f97c2d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f97c40d16e0 0x7f97c40d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.865+0000 7f97c2d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f97c40d16e0 0x7f97c40d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37304/0 (socket says 172.21.15.59:37304) 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.865+0000 7f97c2d9d640 1 -- 172.21.15.59:0/1001251754 learned_addr learned my addr 172.21.15.59:0/1001251754 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.865+0000 7f97c1d9b640 1 --2- 172.21.15.59:0/1001251754 >> v2:172.21.15.164:3300/0 conn(0x7f97c415fa10 0x7f97c40ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.866+0000 7f97c2d9d640 1 -- 172.21.15.59:0/1001251754 >> v2:172.21.15.164:3300/0 conn(0x7f97c415fa10 msgr2=0x7f97c40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.866+0000 7f97c2d9d640 1 --2- 172.21.15.59:0/1001251754 >> v2:172.21.15.164:3300/0 conn(0x7f97c415fa10 0x7f97c40ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.866+0000 7f97c2d9d640 1 -- 172.21.15.59:0/1001251754 >> v1:172.21.15.164:6789/0 conn(0x7f97c415bdd0 legacy=0x7f97c40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.866+0000 7f97c2d9d640 1 -- 172.21.15.59:0/1001251754 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c40cc8e0 con 0x7f97c40d16e0 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.866+0000 7f97c2d9d640 1 --2- 172.21.15.59:0/1001251754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c40d16e0 0x7f97c40d1af0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f97c40cdfd0 tx=0x7f97b402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4434d4c6c05f83e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.867+0000 7f97c159a640 1 -- 172.21.15.59:0/1001251754 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b402ca80 con 0x7f97c40d16e0 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.867+0000 7f97c159a640 1 -- 172.21.15.59:0/1001251754 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b402cc20 con 0x7f97c40d16e0 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.867+0000 7f97c159a640 1 -- 172.21.15.59:0/1001251754 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b40343f0 con 0x7f97c40d16e0 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.867+0000 7f97c8b2c640 1 -- 172.21.15.59:0/1001251754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c40d16e0 msgr2=0x7f97c40d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.867+0000 7f97c8b2c640 1 --2- 172.21.15.59:0/1001251754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c40d16e0 0x7f97c40d1af0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f97c40cdfd0 tx=0x7f97b402bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.867+0000 7f97c8b2c640 1 -- 172.21.15.59:0/1001251754 shutdown_connections 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.867+0000 7f97c8b2c640 1 --2- 172.21.15.59:0/1001251754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c40d16e0 0x7f97c40d1af0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.867+0000 7f97c8b2c640 1 --2- 172.21.15.59:0/1001251754 >> v2:172.21.15.164:3300/0 conn(0x7f97c415fa10 0x7f97c40ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.867+0000 7f97c8b2c640 1 -- 172.21.15.59:0/1001251754 >> 172.21.15.59:0/1001251754 conn(0x7f97c409a9f0 msgr2=0x7f97c409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.867+0000 7f97c8b2c640 1 -- 172.21.15.59:0/1001251754 shutdown_connections 2024-03-06T07:27:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.867+0000 7f97c8b2c640 1 -- 172.21.15.59:0/1001251754 wait complete. 2024-03-06T07:27:17.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.868+0000 7f97c8b2c640 1 Processor -- start 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.868+0000 7f97c8b2c640 1 -- start start 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.869+0000 7f97c8b2c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c40d16e0 0x7f97c4154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.869+0000 7f97c8b2c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c415bdd0 0x7f97c40cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.869+0000 7f97c8b2c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97c415fa10 0x7f97c415c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.869+0000 7f97c8b2c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40d2030 con 0x7f97c415bdd0 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.869+0000 7f97c8b2c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c415c1b0 con 0x7f97c40d16e0 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.869+0000 7f97c8b2c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40ce7b0 con 0x7f97c415fa10 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.869+0000 7f97c1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c415bdd0 0x7f97c40cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.869+0000 7f97c259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c40d16e0 0x7f97c4154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.869+0000 7f97c1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c415bdd0 0x7f97c40cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37320/0 (socket says 172.21.15.59:37320) 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.869+0000 7f97c1d9b640 1 -- 172.21.15.59:0/3534500859 learned_addr learned my addr 172.21.15.59:0/3534500859 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.869+0000 7f97c2d9d640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97c415fa10 0x7f97c415c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97c1d9b640 1 -- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97c415fa10 msgr2=0x7f97c415c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97c1d9b640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97c415fa10 0x7f97c415c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97c1d9b640 1 -- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c40d16e0 msgr2=0x7f97c4154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97c1d9b640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c40d16e0 0x7f97c4154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97c1d9b640 1 -- 172.21.15.59:0/3534500859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c40c9db0 con 0x7f97c415bdd0 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97c259c640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c40d16e0 0x7f97c4154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97c1d9b640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c415bdd0 0x7f97c40cae50 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00e870 tx=0x7f97ac00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97baffd640 1 -- 172.21.15.59:0/3534500859 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac017070 con 0x7f97c415bdd0 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97baffd640 1 -- 172.21.15.59:0/3534500859 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ac00c940 con 0x7f97c415bdd0 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97c8b2c640 1 -- 172.21.15.59:0/3534500859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c40ca090 con 0x7f97c415bdd0 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97baffd640 1 -- 172.21.15.59:0/3534500859 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac00cd10 con 0x7f97c415bdd0 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.870+0000 7f97c8b2c640 1 -- 172.21.15.59:0/3534500859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c40ca650 con 0x7f97c415bdd0 2024-03-06T07:27:17.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.871+0000 7f97c8b2c640 1 -- 172.21.15.59:0/3534500859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c404f330 con 0x7f97c415bdd0 2024-03-06T07:27:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.871+0000 7f97baffd640 1 -- 172.21.15.59:0/3534500859 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97ac01a070 con 0x7f97c415bdd0 2024-03-06T07:27:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.872+0000 7f97baffd640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a406b4d0 0x7f97a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.872+0000 7f97baffd640 1 -- 172.21.15.59:0/3534500859 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97ac08a3b0 con 0x7f97c415bdd0 2024-03-06T07:27:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.872+0000 7f97c259c640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a406b4d0 0x7f97a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.872+0000 7f97c259c640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a406b4d0 0x7f97a406d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f97c40cb890 tx=0x7f97bc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:17.876+0000 7f97baffd640 1 -- 172.21.15.59:0/3534500859 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97ac058be0 con 0x7f97c415bdd0 2024-03-06T07:27:18.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.053+0000 7f97c8b2c640 1 -- 172.21.15.59:0/3534500859 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97c406f9a0 con 0x7f97a406b4d0 2024-03-06T07:27:18.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.060+0000 7f97baffd640 1 -- 172.21.15.59:0/3534500859 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252434 (secure 0 0 0) 0x7f97c406f9a0 con 0x7f97a406b4d0 2024-03-06T07:27:18.064 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:18.066 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 -- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a406b4d0 msgr2=0x7f97a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a406b4d0 0x7f97a406d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f97c40cb890 tx=0x7f97bc009040 comp rx=0 tx=0).stop 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 -- 172.21.15.59:0/3534500859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c415bdd0 msgr2=0x7f97c40cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c415bdd0 0x7f97c40cae50 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00e870 tx=0x7f97ac00ed30 comp rx=0 tx=0).stop 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 -- 172.21.15.59:0/3534500859 shutdown_connections 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f97a406b4d0 0x7f97a406d990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97c415fa10 0x7f97c415c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97c415bdd0 0x7f97c40cae50 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 --2- 172.21.15.59:0/3534500859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c40d16e0 0x7f97c4154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 -- 172.21.15.59:0/3534500859 >> 172.21.15.59:0/3534500859 conn(0x7f97c409a9f0 msgr2=0x7f97c40c63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 -- 172.21.15.59:0/3534500859 shutdown_connections 2024-03-06T07:27:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.063+0000 7f97c8b2c640 1 -- 172.21.15.59:0/3534500859 wait complete. 2024-03-06T07:27:18.076 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-03-06T07:27:17.371614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319372,"kb_used_data":11636,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754655348,"statfs":{"total":773094113280,"available":772767076352,"internally_reserved":0,"allocated":11915264,"data_stored":5787352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":314953265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.401813"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.002249+0000","last_change":"2024-03-06T07:27:14.999220+0000","last_active":"2024-03-06T07:27:15.002249+0000","last_peered":"2024-03-06T07:27:15.002249+0000","last_clean":"2024-03-06T07:27:15.002249+0000","last_became_active":"2024-03-06T07:27:14.998932+0000","last_became_peered":"2024-03-06T07:27:14.998932+0000","last_unstale":"2024-03-06T07:27:15.002249+0000","last_undegraded":"2024-03-06T07:27:15.002249+0000","last_fullsized":"2024-03-06T07:27:15.002249+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:25.925986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006793+0000","last_change":"2024-03-06T07:27:14.998126+0000","last_active":"2024-03-06T07:27:15.006793+0000","last_peered":"2024-03-06T07:27:15.006793+0000","last_clean":"2024-03-06T07:27:15.006793+0000","last_became_active":"2024-03-06T07:27:14.997828+0000","last_became_peered":"2024-03-06T07:27:14.997828+0000","last_unstale":"2024-03-06T07:27:15.006793+0000","last_undegraded":"2024-03-06T07:27:15.006793+0000","last_fullsized":"2024-03-06T07:27:15.006793+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:37.880813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.008700+0000","last_change":"2024-03-06T07:27:15.007394+0000","last_active":"2024-03-06T07:27:15.008700+0000","last_peered":"2024-03-06T07:27:15.008700+0000","last_clean":"2024-03-06T07:27:15.008700+0000","last_became_active":"2024-03-06T07:27:15.007120+0000","last_became_peered":"2024-03-06T07:27:15.007120+0000","last_unstale":"2024-03-06T07:27:15.008700+0000","last_undegraded":"2024-03-06T07:27:15.008700+0000","last_fullsized":"2024-03-06T07:27:15.008700+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:09.140300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.009551+0000","last_change":"2024-03-06T07:27:15.004996+0000","last_active":"2024-03-06T07:27:15.009551+0000","last_peered":"2024-03-06T07:27:15.009551+0000","last_clean":"2024-03-06T07:27:15.009551+0000","last_became_active":"2024-03-06T07:27:15.003336+0000","last_became_peered":"2024-03-06T07:27:15.003336+0000","last_unstale":"2024-03-06T07:27:15.009551+0000","last_undegraded":"2024-03-06T07:27:15.009551+0000","last_fullsized":"2024-03-06T07:27:15.009551+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:59.743756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.001632+0000","last_change":"2024-03-06T07:27:14.993845+0000","last_active":"2024-03-06T07:27:15.001632+0000","last_peered":"2024-03-06T07:27:15.001632+0000","last_clean":"2024-03-06T07:27:15.001632+0000","last_became_active":"2024-03-06T07:27:14.990629+0000","last_became_peered":"2024-03-06T07:27:14.990629+0000","last_unstale":"2024-03-06T07:27:15.001632+0000","last_undegraded":"2024-03-06T07:27:15.001632+0000","last_fullsized":"2024-03-06T07:27:15.001632+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:04.840675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987801+0000","last_change":"2024-03-06T07:27:14.987595+0000","last_active":"2024-03-06T07:27:15.987801+0000","last_peered":"2024-03-06T07:27:15.987801+0000","last_clean":"2024-03-06T07:27:15.987801+0000","last_became_active":"2024-03-06T07:27:14.987334+0000","last_became_peered":"2024-03-06T07:27:14.987334+0000","last_unstale":"2024-03-06T07:27:15.987801+0000","last_undegraded":"2024-03-06T07:27:15.987801+0000","last_fullsized":"2024-03-06T07:27:15.987801+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:36.824848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990367+0000","last_change":"2024-03-06T07:27:14.999083+0000","last_active":"2024-03-06T07:27:15.990367+0000","last_peered":"2024-03-06T07:27:15.990367+0000","last_clean":"2024-03-06T07:27:15.990367+0000","last_became_active":"2024-03-06T07:27:14.998768+0000","last_became_peered":"2024-03-06T07:27:14.998768+0000","last_unstale":"2024-03-06T07:27:15.990367+0000","last_undegraded":"2024-03-06T07:27:15.990367+0000","last_fullsized":"2024-03-06T07:27:15.990367+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:13.525561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003166+0000","last_change":"2024-03-06T07:27:14.997221+0000","last_active":"2024-03-06T07:27:15.003166+0000","last_peered":"2024-03-06T07:27:15.003166+0000","last_clean":"2024-03-06T07:27:15.003166+0000","last_became_active":"2024-03-06T07:27:14.996626+0000","last_became_peered":"2024-03-06T07:27:14.996626+0000","last_unstale":"2024-03-06T07:27:15.003166+0000","last_undegraded":"2024-03-06T07:27:15.003166+0000","last_fullsized":"2024-03-06T07:27:15.003166+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:37.880813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.999641+0000","last_change":"2024-03-06T07:27:14.998232+0000","last_active":"2024-03-06T07:27:14.999641+0000","last_peered":"2024-03-06T07:27:14.999641+0000","last_clean":"2024-03-06T07:27:14.999641+0000","last_became_active":"2024-03-06T07:27:14.997790+0000","last_became_peered":"2024-03-06T07:27:14.997790+0000","last_unstale":"2024-03-06T07:27:14.999641+0000","last_undegraded":"2024-03-06T07:27:14.999641+0000","last_fullsized":"2024-03-06T07:27:14.999641+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:07.794199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.009543+0000","last_change":"2024-03-06T07:27:15.004383+0000","last_active":"2024-03-06T07:27:15.009543+0000","last_peered":"2024-03-06T07:27:15.009543+0000","last_clean":"2024-03-06T07:27:15.009543+0000","last_became_active":"2024-03-06T07:27:15.002877+0000","last_became_peered":"2024-03-06T07:27:15.002877+0000","last_unstale":"2024-03-06T07:27:15.009543+0000","last_undegraded":"2024-03-06T07:27:15.009543+0000","last_fullsized":"2024-03-06T07:27:15.009543+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:33.221493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006997+0000","last_change":"2024-03-06T07:27:15.000651+0000","last_active":"2024-03-06T07:27:15.006997+0000","last_peered":"2024-03-06T07:27:15.006997+0000","last_clean":"2024-03-06T07:27:15.006997+0000","last_became_active":"2024-03-06T07:27:15.000353+0000","last_became_peered":"2024-03-06T07:27:15.000353+0000","last_unstale":"2024-03-06T07:27:15.006997+0000","last_undegraded":"2024-03-06T07:27:15.006997+0000","last_fullsized":"2024-03-06T07:27:15.006997+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:12.434709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007029+0000","last_change":"2024-03-06T07:27:15.002602+0000","last_active":"2024-03-06T07:27:15.007029+0000","last_peered":"2024-03-06T07:27:15.007029+0000","last_clean":"2024-03-06T07:27:15.007029+0000","last_became_active":"2024-03-06T07:27:15.002340+0000","last_became_peered":"2024-03-06T07:27:15.002340+0000","last_unstale":"2024-03-06T07:27:15.007029+0000","last_undegraded":"2024-03-06T07:27:15.007029+0000","last_fullsized":"2024-03-06T07:27:15.007029+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:49.838205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006581+0000","last_change":"2024-03-06T07:27:15.005196+0000","last_active":"2024-03-06T07:27:15.006581+0000","last_peered":"2024-03-06T07:27:15.006581+0000","last_clean":"2024-03-06T07:27:15.006581+0000","last_became_active":"2024-03-06T07:27:15.004920+0000","last_became_peered":"2024-03-06T07:27:15.004920+0000","last_unstale":"2024-03-06T07:27:15.006581+0000","last_undegraded":"2024-03-06T07:27:15.006581+0000","last_fullsized":"2024-03-06T07:27:15.006581+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:51.665162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006041+0000","last_change":"2024-03-06T07:27:14.999595+0000","last_active":"2024-03-06T07:27:15.006041+0000","last_peered":"2024-03-06T07:27:15.006041+0000","last_clean":"2024-03-06T07:27:15.006041+0000","last_became_active":"2024-03-06T07:27:14.999266+0000","last_became_peered":"2024-03-06T07:27:14.999266+0000","last_unstale":"2024-03-06T07:27:15.006041+0000","last_undegraded":"2024-03-06T07:27:15.006041+0000","last_fullsized":"2024-03-06T07:27:15.006041+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:27.696930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987764+0000","last_change":"2024-03-06T07:27:14.988732+0000","last_active":"2024-03-06T07:27:15.987764+0000","last_peered":"2024-03-06T07:27:15.987764+0000","last_clean":"2024-03-06T07:27:15.987764+0000","last_became_active":"2024-03-06T07:27:14.988467+0000","last_became_peered":"2024-03-06T07:27:14.988467+0000","last_unstale":"2024-03-06T07:27:15.987764+0000","last_undegraded":"2024-03-06T07:27:15.987764+0000","last_fullsized":"2024-03-06T07:27:15.987764+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:19.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006991+0000","last_change":"2024-03-06T07:27:15.002743+0000","last_active":"2024-03-06T07:27:15.006991+0000","last_peered":"2024-03-06T07:27:15.006991+0000","last_clean":"2024-03-06T07:27:15.006991+0000","last_became_active":"2024-03-06T07:27:15.002292+0000","last_became_peered":"2024-03-06T07:27:15.002292+0000","last_unstale":"2024-03-06T07:27:15.006991+0000","last_undegraded":"2024-03-06T07:27:15.006991+0000","last_fullsized":"2024-03-06T07:27:15.006991+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:13:01.277640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007096+0000","last_change":"2024-03-06T07:27:15.001098+0000","last_active":"2024-03-06T07:27:15.007096+0000","last_peered":"2024-03-06T07:27:15.007096+0000","last_clean":"2024-03-06T07:27:15.007096+0000","last_became_active":"2024-03-06T07:27:15.000815+0000","last_became_peered":"2024-03-06T07:27:15.000815+0000","last_unstale":"2024-03-06T07:27:15.007096+0000","last_undegraded":"2024-03-06T07:27:15.007096+0000","last_fullsized":"2024-03-06T07:27:15.007096+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:06.357185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.011651+0000","last_change":"2024-03-06T07:27:15.007615+0000","last_active":"2024-03-06T07:27:15.011651+0000","last_peered":"2024-03-06T07:27:15.011651+0000","last_clean":"2024-03-06T07:27:15.011651+0000","last_became_active":"2024-03-06T07:27:15.007251+0000","last_became_peered":"2024-03-06T07:27:15.007251+0000","last_unstale":"2024-03-06T07:27:15.011651+0000","last_undegraded":"2024-03-06T07:27:15.011651+0000","last_fullsized":"2024-03-06T07:27:15.011651+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:19.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006637+0000","last_change":"2024-03-06T07:27:15.002925+0000","last_active":"2024-03-06T07:27:15.006637+0000","last_peered":"2024-03-06T07:27:15.006637+0000","last_clean":"2024-03-06T07:27:15.006637+0000","last_became_active":"2024-03-06T07:27:15.002643+0000","last_became_peered":"2024-03-06T07:27:15.002643+0000","last_unstale":"2024-03-06T07:27:15.006637+0000","last_undegraded":"2024-03-06T07:27:15.006637+0000","last_fullsized":"2024-03-06T07:27:15.006637+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:39.117581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007190+0000","last_change":"2024-03-06T07:27:15.002720+0000","last_active":"2024-03-06T07:27:15.007190+0000","last_peered":"2024-03-06T07:27:15.007190+0000","last_clean":"2024-03-06T07:27:15.007190+0000","last_became_active":"2024-03-06T07:27:15.002378+0000","last_became_peered":"2024-03-06T07:27:15.002378+0000","last_unstale":"2024-03-06T07:27:15.007190+0000","last_undegraded":"2024-03-06T07:27:15.007190+0000","last_fullsized":"2024-03-06T07:27:15.007190+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:59.098860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007783+0000","last_change":"2024-03-06T07:27:14.999312+0000","last_active":"2024-03-06T07:27:15.007783+0000","last_peered":"2024-03-06T07:27:15.007783+0000","last_clean":"2024-03-06T07:27:15.007783+0000","last_became_active":"2024-03-06T07:27:14.998844+0000","last_became_peered":"2024-03-06T07:27:14.998844+0000","last_unstale":"2024-03-06T07:27:15.007783+0000","last_undegraded":"2024-03-06T07:27:15.007783+0000","last_fullsized":"2024-03-06T07:27:15.007783+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:57.351882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990524+0000","last_change":"2024-03-06T07:27:15.002714+0000","last_active":"2024-03-06T07:27:15.990524+0000","last_peered":"2024-03-06T07:27:15.990524+0000","last_clean":"2024-03-06T07:27:15.990524+0000","last_became_active":"2024-03-06T07:27:15.002345+0000","last_became_peered":"2024-03-06T07:27:15.002345+0000","last_unstale":"2024-03-06T07:27:15.990524+0000","last_undegraded":"2024-03-06T07:27:15.990524+0000","last_fullsized":"2024-03-06T07:27:15.990524+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:54.707369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006772+0000","last_change":"2024-03-06T07:27:14.996096+0000","last_active":"2024-03-06T07:27:15.006772+0000","last_peered":"2024-03-06T07:27:15.006772+0000","last_clean":"2024-03-06T07:27:15.006772+0000","last_became_active":"2024-03-06T07:27:14.995823+0000","last_became_peered":"2024-03-06T07:27:14.995823+0000","last_unstale":"2024-03-06T07:27:15.006772+0000","last_undegraded":"2024-03-06T07:27:15.006772+0000","last_fullsized":"2024-03-06T07:27:15.006772+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:36.824848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990397+0000","last_change":"2024-03-06T07:27:14.999164+0000","last_active":"2024-03-06T07:27:15.990397+0000","last_peered":"2024-03-06T07:27:15.990397+0000","last_clean":"2024-03-06T07:27:15.990397+0000","last_became_active":"2024-03-06T07:27:14.998896+0000","last_became_peered":"2024-03-06T07:27:14.998896+0000","last_unstale":"2024-03-06T07:27:15.990397+0000","last_undegraded":"2024-03-06T07:27:15.990397+0000","last_fullsized":"2024-03-06T07:27:15.990397+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:46.411357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007118+0000","last_change":"2024-03-06T07:27:15.001662+0000","last_active":"2024-03-06T07:27:15.007118+0000","last_peered":"2024-03-06T07:27:15.007118+0000","last_clean":"2024-03-06T07:27:15.007118+0000","last_became_active":"2024-03-06T07:27:15.001377+0000","last_became_peered":"2024-03-06T07:27:15.001377+0000","last_unstale":"2024-03-06T07:27:15.007118+0000","last_undegraded":"2024-03-06T07:27:15.007118+0000","last_fullsized":"2024-03-06T07:27:15.007118+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:39.117581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.004520+0000","last_change":"2024-03-06T07:27:15.000099+0000","last_active":"2024-03-06T07:27:15.004520+0000","last_peered":"2024-03-06T07:27:15.004520+0000","last_clean":"2024-03-06T07:27:15.004520+0000","last_became_active":"2024-03-06T07:27:14.999838+0000","last_became_peered":"2024-03-06T07:27:14.999838+0000","last_unstale":"2024-03-06T07:27:15.004520+0000","last_undegraded":"2024-03-06T07:27:15.004520+0000","last_fullsized":"2024-03-06T07:27:15.004520+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:13.372370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987905+0000","last_change":"2024-03-06T07:27:14.992914+0000","last_active":"2024-03-06T07:27:15.987905+0000","last_peered":"2024-03-06T07:27:15.987905+0000","last_clean":"2024-03-06T07:27:15.987905+0000","last_became_active":"2024-03-06T07:27:14.992630+0000","last_became_peered":"2024-03-06T07:27:14.992630+0000","last_unstale":"2024-03-06T07:27:15.987905+0000","last_undegraded":"2024-03-06T07:27:15.987905+0000","last_fullsized":"2024-03-06T07:27:15.987905+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:37.880813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990732+0000","last_change":"2024-03-06T07:27:14.999901+0000","last_active":"2024-03-06T07:27:15.990732+0000","last_peered":"2024-03-06T07:27:15.990732+0000","last_clean":"2024-03-06T07:27:15.990732+0000","last_became_active":"2024-03-06T07:27:14.999412+0000","last_became_peered":"2024-03-06T07:27:14.999412+0000","last_unstale":"2024-03-06T07:27:15.990732+0000","last_undegraded":"2024-03-06T07:27:15.990732+0000","last_fullsized":"2024-03-06T07:27:15.990732+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:17.563603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006786+0000","last_change":"2024-03-06T07:27:14.998165+0000","last_active":"2024-03-06T07:27:15.006786+0000","last_peered":"2024-03-06T07:27:15.006786+0000","last_clean":"2024-03-06T07:27:15.006786+0000","last_became_active":"2024-03-06T07:27:14.997874+0000","last_became_peered":"2024-03-06T07:27:14.997874+0000","last_unstale":"2024-03-06T07:27:15.006786+0000","last_undegraded":"2024-03-06T07:27:15.006786+0000","last_fullsized":"2024-03-06T07:27:15.006786+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:01.873864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990446+0000","last_change":"2024-03-06T07:27:14.988198+0000","last_active":"2024-03-06T07:27:15.990446+0000","last_peered":"2024-03-06T07:27:15.990446+0000","last_clean":"2024-03-06T07:27:15.990446+0000","last_became_active":"2024-03-06T07:27:14.986976+0000","last_became_peered":"2024-03-06T07:27:14.986976+0000","last_unstale":"2024-03-06T07:27:15.990446+0000","last_undegraded":"2024-03-06T07:27:15.990446+0000","last_fullsized":"2024-03-06T07:27:15.990446+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:15.510457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007037+0000","last_change":"2024-03-06T07:27:15.003601+0000","last_active":"2024-03-06T07:27:15.007037+0000","last_peered":"2024-03-06T07:27:15.007037+0000","last_clean":"2024-03-06T07:27:15.007037+0000","last_became_active":"2024-03-06T07:27:15.003349+0000","last_became_peered":"2024-03-06T07:27:15.003349+0000","last_unstale":"2024-03-06T07:27:15.007037+0000","last_undegraded":"2024-03-06T07:27:15.007037+0000","last_fullsized":"2024-03-06T07:27:15.007037+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:31.291940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987769+0000","last_change":"2024-03-06T07:27:14.994279+0000","last_active":"2024-03-06T07:27:15.987769+0000","last_peered":"2024-03-06T07:27:15.987769+0000","last_clean":"2024-03-06T07:27:15.987769+0000","last_became_active":"2024-03-06T07:27:14.992622+0000","last_became_peered":"2024-03-06T07:27:14.992622+0000","last_unstale":"2024-03-06T07:27:15.987769+0000","last_undegraded":"2024-03-06T07:27:15.987769+0000","last_fullsized":"2024-03-06T07:27:15.987769+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:40.108009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987939+0000","last_change":"2024-03-06T07:27:14.988071+0000","last_active":"2024-03-06T07:27:15.987939+0000","last_peered":"2024-03-06T07:27:15.987939+0000","last_clean":"2024-03-06T07:27:15.987939+0000","last_became_active":"2024-03-06T07:27:14.987324+0000","last_became_peered":"2024-03-06T07:27:14.987324+0000","last_unstale":"2024-03-06T07:27:15.987939+0000","last_undegraded":"2024-03-06T07:27:15.987939+0000","last_fullsized":"2024-03-06T07:27:15.987939+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:01.873864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990732+0000","last_change":"2024-03-06T07:27:13.985966+0000","last_active":"2024-03-06T07:27:15.990732+0000","last_peered":"2024-03-06T07:27:15.990732+0000","last_clean":"2024-03-06T07:27:15.990732+0000","last_became_active":"2024-03-06T07:27:13.985623+0000","last_became_peered":"2024-03-06T07:27:13.985623+0000","last_unstale":"2024-03-06T07:27:15.990732+0000","last_undegraded":"2024-03-06T07:27:15.990732+0000","last_fullsized":"2024-03-06T07:27:15.990732+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:48.422576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003444+0000","last_change":"2024-03-06T07:27:15.000616+0000","last_active":"2024-03-06T07:27:15.003444+0000","last_peered":"2024-03-06T07:27:15.003444+0000","last_clean":"2024-03-06T07:27:15.003444+0000","last_became_active":"2024-03-06T07:27:15.000347+0000","last_became_peered":"2024-03-06T07:27:15.000347+0000","last_unstale":"2024-03-06T07:27:15.003444+0000","last_undegraded":"2024-03-06T07:27:15.003444+0000","last_fullsized":"2024-03-06T07:27:15.003444+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:42.996423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.996810+0000","last_change":"2024-03-06T07:27:13.988934+0000","last_active":"2024-03-06T07:27:14.996810+0000","last_peered":"2024-03-06T07:27:14.996810+0000","last_clean":"2024-03-06T07:27:14.996810+0000","last_became_active":"2024-03-06T07:27:13.988646+0000","last_became_peered":"2024-03-06T07:27:13.988646+0000","last_unstale":"2024-03-06T07:27:14.996810+0000","last_undegraded":"2024-03-06T07:27:14.996810+0000","last_fullsized":"2024-03-06T07:27:14.996810+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:35.822855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.988068+0000","last_change":"2024-03-06T07:27:14.991900+0000","last_active":"2024-03-06T07:27:15.988068+0000","last_peered":"2024-03-06T07:27:15.988068+0000","last_clean":"2024-03-06T07:27:15.988068+0000","last_became_active":"2024-03-06T07:27:14.991622+0000","last_became_peered":"2024-03-06T07:27:14.991622+0000","last_unstale":"2024-03-06T07:27:15.988068+0000","last_undegraded":"2024-03-06T07:27:15.988068+0000","last_fullsized":"2024-03-06T07:27:15.988068+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:57.351882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.001779+0000","last_change":"2024-03-06T07:27:13.994045+0000","last_active":"2024-03-06T07:27:15.001779+0000","last_peered":"2024-03-06T07:27:15.001779+0000","last_clean":"2024-03-06T07:27:15.001779+0000","last_became_active":"2024-03-06T07:27:13.993753+0000","last_became_peered":"2024-03-06T07:27:13.993753+0000","last_unstale":"2024-03-06T07:27:15.001779+0000","last_undegraded":"2024-03-06T07:27:15.001779+0000","last_fullsized":"2024-03-06T07:27:15.001779+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:55.293700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006655+0000","last_change":"2024-03-06T07:27:15.003936+0000","last_active":"2024-03-06T07:27:15.006655+0000","last_peered":"2024-03-06T07:27:15.006655+0000","last_clean":"2024-03-06T07:27:15.006655+0000","last_became_active":"2024-03-06T07:27:15.002848+0000","last_became_peered":"2024-03-06T07:27:15.002848+0000","last_unstale":"2024-03-06T07:27:15.006655+0000","last_undegraded":"2024-03-06T07:27:15.006655+0000","last_fullsized":"2024-03-06T07:27:15.006655+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:59.098860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.999486+0000","last_change":"2024-03-06T07:27:13.993657+0000","last_active":"2024-03-06T07:27:14.999486+0000","last_peered":"2024-03-06T07:27:14.999486+0000","last_clean":"2024-03-06T07:27:14.999486+0000","last_became_active":"2024-03-06T07:27:13.993371+0000","last_became_peered":"2024-03-06T07:27:13.993371+0000","last_unstale":"2024-03-06T07:27:14.999486+0000","last_undegraded":"2024-03-06T07:27:14.999486+0000","last_fullsized":"2024-03-06T07:27:14.999486+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:49.516922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.998332+0000","last_change":"2024-03-06T07:27:14.995941+0000","last_active":"2024-03-06T07:27:14.998332+0000","last_peered":"2024-03-06T07:27:14.998332+0000","last_clean":"2024-03-06T07:27:14.998332+0000","last_became_active":"2024-03-06T07:27:14.995634+0000","last_became_peered":"2024-03-06T07:27:14.995634+0000","last_unstale":"2024-03-06T07:27:14.998332+0000","last_undegraded":"2024-03-06T07:27:14.998332+0000","last_fullsized":"2024-03-06T07:27:14.998332+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:46.411357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990447+0000","last_change":"2024-03-06T07:27:13.990461+0000","last_active":"2024-03-06T07:27:15.990447+0000","last_peered":"2024-03-06T07:27:15.990447+0000","last_clean":"2024-03-06T07:27:15.990447+0000","last_became_active":"2024-03-06T07:27:13.990177+0000","last_became_peered":"2024-03-06T07:27:13.990177+0000","last_unstale":"2024-03-06T07:27:15.990447+0000","last_undegraded":"2024-03-06T07:27:15.990447+0000","last_fullsized":"2024-03-06T07:27:15.990447+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:59.651616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007761+0000","last_change":"2024-03-06T07:27:14.999102+0000","last_active":"2024-03-06T07:27:15.007761+0000","last_peered":"2024-03-06T07:27:15.007761+0000","last_clean":"2024-03-06T07:27:15.007761+0000","last_became_active":"2024-03-06T07:27:14.998476+0000","last_became_peered":"2024-03-06T07:27:14.998476+0000","last_unstale":"2024-03-06T07:27:15.007761+0000","last_undegraded":"2024-03-06T07:27:15.007761+0000","last_fullsized":"2024-03-06T07:27:15.007761+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:27.696930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990638+0000","last_change":"2024-03-06T07:27:13.993648+0000","last_active":"2024-03-06T07:27:15.990638+0000","last_peered":"2024-03-06T07:27:15.990638+0000","last_clean":"2024-03-06T07:27:15.990638+0000","last_became_active":"2024-03-06T07:27:13.991583+0000","last_became_peered":"2024-03-06T07:27:13.991583+0000","last_unstale":"2024-03-06T07:27:15.990638+0000","last_undegraded":"2024-03-06T07:27:15.990638+0000","last_fullsized":"2024-03-06T07:27:15.990638+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:08.138307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003176+0000","last_change":"2024-03-06T07:27:14.992051+0000","last_active":"2024-03-06T07:27:15.003176+0000","last_peered":"2024-03-06T07:27:15.003176+0000","last_clean":"2024-03-06T07:27:15.003176+0000","last_became_active":"2024-03-06T07:27:14.991772+0000","last_became_peered":"2024-03-06T07:27:14.991772+0000","last_unstale":"2024-03-06T07:27:15.003176+0000","last_undegraded":"2024-03-06T07:27:15.003176+0000","last_fullsized":"2024-03-06T07:27:15.003176+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:38.766369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.988166+0000","last_change":"2024-03-06T07:27:13.991976+0000","last_active":"2024-03-06T07:27:14.988166+0000","last_peered":"2024-03-06T07:27:14.988166+0000","last_clean":"2024-03-06T07:27:14.988166+0000","last_became_active":"2024-03-06T07:27:13.991674+0000","last_became_peered":"2024-03-06T07:27:13.991674+0000","last_unstale":"2024-03-06T07:27:14.988166+0000","last_undegraded":"2024-03-06T07:27:14.988166+0000","last_fullsized":"2024-03-06T07:27:14.988166+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:39.106016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990302+0000","last_change":"2024-03-06T07:27:14.999744+0000","last_active":"2024-03-06T07:27:15.990302+0000","last_peered":"2024-03-06T07:27:15.990302+0000","last_clean":"2024-03-06T07:27:15.990302+0000","last_became_active":"2024-03-06T07:27:14.999478+0000","last_became_peered":"2024-03-06T07:27:14.999478+0000","last_unstale":"2024-03-06T07:27:15.990302+0000","last_undegraded":"2024-03-06T07:27:15.990302+0000","last_fullsized":"2024-03-06T07:27:15.990302+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:37.564891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.982354+0000","last_change":"2024-03-06T07:27:13.994282+0000","last_active":"2024-03-06T07:27:14.982354+0000","last_peered":"2024-03-06T07:27:14.982354+0000","last_clean":"2024-03-06T07:27:14.982354+0000","last_became_active":"2024-03-06T07:27:13.993998+0000","last_became_peered":"2024-03-06T07:27:13.993998+0000","last_unstale":"2024-03-06T07:27:14.982354+0000","last_undegraded":"2024-03-06T07:27:14.982354+0000","last_fullsized":"2024-03-06T07:27:14.982354+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:33.681814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990724+0000","last_change":"2024-03-06T07:27:14.991025+0000","last_active":"2024-03-06T07:27:15.990724+0000","last_peered":"2024-03-06T07:27:15.990724+0000","last_clean":"2024-03-06T07:27:15.990724+0000","last_became_active":"2024-03-06T07:27:14.990748+0000","last_became_peered":"2024-03-06T07:27:14.990748+0000","last_unstale":"2024-03-06T07:27:15.990724+0000","last_undegraded":"2024-03-06T07:27:15.990724+0000","last_fullsized":"2024-03-06T07:27:15.990724+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:22.264605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.981178+0000","last_change":"2024-03-06T07:27:13.996786+0000","last_active":"2024-03-06T07:27:14.981178+0000","last_peered":"2024-03-06T07:27:14.981178+0000","last_clean":"2024-03-06T07:27:14.981178+0000","last_became_active":"2024-03-06T07:27:13.996505+0000","last_became_peered":"2024-03-06T07:27:13.996505+0000","last_unstale":"2024-03-06T07:27:14.981178+0000","last_undegraded":"2024-03-06T07:27:14.981178+0000","last_fullsized":"2024-03-06T07:27:14.981178+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:21.705645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006542+0000","last_change":"2024-03-06T07:27:15.002887+0000","last_active":"2024-03-06T07:27:15.006542+0000","last_peered":"2024-03-06T07:27:15.006542+0000","last_clean":"2024-03-06T07:27:15.006542+0000","last_became_active":"2024-03-06T07:27:15.001744+0000","last_became_peered":"2024-03-06T07:27:15.001744+0000","last_unstale":"2024-03-06T07:27:15.006542+0000","last_undegraded":"2024-03-06T07:27:15.006542+0000","last_fullsized":"2024-03-06T07:27:15.006542+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:40.108009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990597+0000","last_change":"2024-03-06T07:27:13.993489+0000","last_active":"2024-03-06T07:27:15.990597+0000","last_peered":"2024-03-06T07:27:15.990597+0000","last_clean":"2024-03-06T07:27:15.990597+0000","last_became_active":"2024-03-06T07:27:13.990858+0000","last_became_peered":"2024-03-06T07:27:13.990858+0000","last_unstale":"2024-03-06T07:27:15.990597+0000","last_undegraded":"2024-03-06T07:27:15.990597+0000","last_fullsized":"2024-03-06T07:27:15.990597+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:23.983874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990752+0000","last_change":"2024-03-06T07:27:15.005424+0000","last_active":"2024-03-06T07:27:15.990752+0000","last_peered":"2024-03-06T07:27:15.990752+0000","last_clean":"2024-03-06T07:27:15.990752+0000","last_became_active":"2024-03-06T07:27:15.005134+0000","last_became_peered":"2024-03-06T07:27:15.005134+0000","last_unstale":"2024-03-06T07:27:15.990752+0000","last_undegraded":"2024-03-06T07:27:15.990752+0000","last_fullsized":"2024-03-06T07:27:15.990752+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:49.838205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.996569+0000","last_change":"2024-03-06T07:27:13.995644+0000","last_active":"2024-03-06T07:27:14.996569+0000","last_peered":"2024-03-06T07:27:14.996569+0000","last_clean":"2024-03-06T07:27:14.996569+0000","last_became_active":"2024-03-06T07:27:13.995368+0000","last_became_peered":"2024-03-06T07:27:13.995368+0000","last_unstale":"2024-03-06T07:27:14.996569+0000","last_undegraded":"2024-03-06T07:27:14.996569+0000","last_fullsized":"2024-03-06T07:27:14.996569+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:26.706259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990623+0000","last_change":"2024-03-06T07:27:14.997011+0000","last_active":"2024-03-06T07:27:15.990623+0000","last_peered":"2024-03-06T07:27:15.990623+0000","last_clean":"2024-03-06T07:27:15.990623+0000","last_became_active":"2024-03-06T07:27:14.996708+0000","last_became_peered":"2024-03-06T07:27:14.996708+0000","last_unstale":"2024-03-06T07:27:15.990623+0000","last_undegraded":"2024-03-06T07:27:15.990623+0000","last_fullsized":"2024-03-06T07:27:15.990623+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:42.996423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.012330+0000","last_change":"2024-03-06T07:27:13.995213+0000","last_active":"2024-03-06T07:27:15.012330+0000","last_peered":"2024-03-06T07:27:15.012330+0000","last_clean":"2024-03-06T07:27:15.012330+0000","last_became_active":"2024-03-06T07:27:13.994753+0000","last_became_peered":"2024-03-06T07:27:13.994753+0000","last_unstale":"2024-03-06T07:27:15.012330+0000","last_undegraded":"2024-03-06T07:27:15.012330+0000","last_fullsized":"2024-03-06T07:27:15.012330+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:55.759680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990590+0000","last_change":"2024-03-06T07:27:14.990689+0000","last_active":"2024-03-06T07:27:15.990590+0000","last_peered":"2024-03-06T07:27:15.990590+0000","last_clean":"2024-03-06T07:27:15.990590+0000","last_became_active":"2024-03-06T07:27:14.990384+0000","last_became_peered":"2024-03-06T07:27:14.990384+0000","last_unstale":"2024-03-06T07:27:15.990590+0000","last_undegraded":"2024-03-06T07:27:15.990590+0000","last_fullsized":"2024-03-06T07:27:15.990590+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:25.925986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987961+0000","last_change":"2024-03-06T07:27:13.985607+0000","last_active":"2024-03-06T07:27:15.987961+0000","last_peered":"2024-03-06T07:27:15.987961+0000","last_clean":"2024-03-06T07:27:15.987961+0000","last_became_active":"2024-03-06T07:27:13.985298+0000","last_became_peered":"2024-03-06T07:27:13.985298+0000","last_unstale":"2024-03-06T07:27:15.987961+0000","last_undegraded":"2024-03-06T07:27:15.987961+0000","last_fullsized":"2024-03-06T07:27:15.987961+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:04.179482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990640+0000","last_change":"2024-03-06T07:27:14.998801+0000","last_active":"2024-03-06T07:27:15.990640+0000","last_peered":"2024-03-06T07:27:15.990640+0000","last_clean":"2024-03-06T07:27:15.990640+0000","last_became_active":"2024-03-06T07:27:14.997099+0000","last_became_peered":"2024-03-06T07:27:14.997099+0000","last_unstale":"2024-03-06T07:27:15.990640+0000","last_undegraded":"2024-03-06T07:27:15.990640+0000","last_fullsized":"2024-03-06T07:27:15.990640+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:13.372370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.996705+0000","last_change":"2024-03-06T07:27:13.990670+0000","last_active":"2024-03-06T07:27:14.996705+0000","last_peered":"2024-03-06T07:27:14.996705+0000","last_clean":"2024-03-06T07:27:14.996705+0000","last_became_active":"2024-03-06T07:27:13.990368+0000","last_became_peered":"2024-03-06T07:27:13.990368+0000","last_unstale":"2024-03-06T07:27:14.996705+0000","last_undegraded":"2024-03-06T07:27:14.996705+0000","last_fullsized":"2024-03-06T07:27:14.996705+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:12.523568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990383+0000","last_change":"2024-03-06T07:27:15.003310+0000","last_active":"2024-03-06T07:27:15.990383+0000","last_peered":"2024-03-06T07:27:15.990383+0000","last_clean":"2024-03-06T07:27:15.990383+0000","last_became_active":"2024-03-06T07:27:15.002619+0000","last_became_peered":"2024-03-06T07:27:15.002619+0000","last_unstale":"2024-03-06T07:27:15.990383+0000","last_undegraded":"2024-03-06T07:27:15.990383+0000","last_fullsized":"2024-03-06T07:27:15.990383+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:38.766369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":20,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.005387+0000","last_change":"2024-03-06T07:27:13.991882+0000","last_active":"2024-03-06T07:27:15.005387+0000","last_peered":"2024-03-06T07:27:15.005387+0000","last_clean":"2024-03-06T07:27:15.005387+0000","last_became_active":"2024-03-06T07:27:13.991620+0000","last_became_peered":"2024-03-06T07:27:13.991620+0000","last_unstale":"2024-03-06T07:27:15.005387+0000","last_undegraded":"2024-03-06T07:27:15.005387+0000","last_fullsized":"2024-03-06T07:27:15.005387+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:37.564172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.001000+0000","last_change":"2024-03-06T07:27:14.996573+0000","last_active":"2024-03-06T07:27:15.001000+0000","last_peered":"2024-03-06T07:27:15.001000+0000","last_clean":"2024-03-06T07:27:15.001000+0000","last_became_active":"2024-03-06T07:27:14.994777+0000","last_became_peered":"2024-03-06T07:27:14.994777+0000","last_unstale":"2024-03-06T07:27:15.001000+0000","last_undegraded":"2024-03-06T07:27:15.001000+0000","last_fullsized":"2024-03-06T07:27:15.001000+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:33.439683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003849+0000","last_change":"2024-03-06T07:27:13.989262+0000","last_active":"2024-03-06T07:27:15.003849+0000","last_peered":"2024-03-06T07:27:15.003849+0000","last_clean":"2024-03-06T07:27:15.003849+0000","last_became_active":"2024-03-06T07:27:13.988626+0000","last_became_peered":"2024-03-06T07:27:13.988626+0000","last_unstale":"2024-03-06T07:27:15.003849+0000","last_undegraded":"2024-03-06T07:27:15.003849+0000","last_fullsized":"2024-03-06T07:27:15.003849+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:45.852417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987981+0000","last_change":"2024-03-06T07:27:15.002326+0000","last_active":"2024-03-06T07:27:15.987981+0000","last_peered":"2024-03-06T07:27:15.987981+0000","last_clean":"2024-03-06T07:27:15.987981+0000","last_became_active":"2024-03-06T07:27:15.001552+0000","last_became_peered":"2024-03-06T07:27:15.001552+0000","last_unstale":"2024-03-06T07:27:15.987981+0000","last_undegraded":"2024-03-06T07:27:15.987981+0000","last_fullsized":"2024-03-06T07:27:15.987981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:27.696930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007322+0000","last_change":"2024-03-06T07:27:13.997890+0000","last_active":"2024-03-06T07:27:15.007322+0000","last_peered":"2024-03-06T07:27:15.007322+0000","last_clean":"2024-03-06T07:27:15.007322+0000","last_became_active":"2024-03-06T07:27:13.997543+0000","last_became_peered":"2024-03-06T07:27:13.997543+0000","last_unstale":"2024-03-06T07:27:15.007322+0000","last_undegraded":"2024-03-06T07:27:15.007322+0000","last_fullsized":"2024-03-06T07:27:15.007322+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:10.739405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006396+0000","last_change":"2024-03-06T07:27:15.001305+0000","last_active":"2024-03-06T07:27:15.006396+0000","last_peered":"2024-03-06T07:27:15.006396+0000","last_clean":"2024-03-06T07:27:15.006396+0000","last_became_active":"2024-03-06T07:27:15.001041+0000","last_became_peered":"2024-03-06T07:27:15.001041+0000","last_unstale":"2024-03-06T07:27:15.006396+0000","last_undegraded":"2024-03-06T07:27:15.006396+0000","last_fullsized":"2024-03-06T07:27:15.006396+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:12.434709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.005120+0000","last_change":"2024-03-06T07:27:13.992257+0000","last_active":"2024-03-06T07:27:15.005120+0000","last_peered":"2024-03-06T07:27:15.005120+0000","last_clean":"2024-03-06T07:27:15.005120+0000","last_became_active":"2024-03-06T07:27:13.991972+0000","last_became_peered":"2024-03-06T07:27:13.991972+0000","last_unstale":"2024-03-06T07:27:15.005120+0000","last_undegraded":"2024-03-06T07:27:15.005120+0000","last_fullsized":"2024-03-06T07:27:15.005120+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:36.562898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.005931+0000","last_change":"2024-03-06T07:27:14.999417+0000","last_active":"2024-03-06T07:27:15.005931+0000","last_peered":"2024-03-06T07:27:15.005931+0000","last_clean":"2024-03-06T07:27:15.005931+0000","last_became_active":"2024-03-06T07:27:14.999111+0000","last_became_peered":"2024-03-06T07:27:14.999111+0000","last_unstale":"2024-03-06T07:27:15.005931+0000","last_undegraded":"2024-03-06T07:27:15.005931+0000","last_fullsized":"2024-03-06T07:27:15.005931+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:01.873864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990489+0000","last_change":"2024-03-06T07:27:13.992361+0000","last_active":"2024-03-06T07:27:15.990489+0000","last_peered":"2024-03-06T07:27:15.990489+0000","last_clean":"2024-03-06T07:27:15.990489+0000","last_became_active":"2024-03-06T07:27:13.992004+0000","last_became_peered":"2024-03-06T07:27:13.992004+0000","last_unstale":"2024-03-06T07:27:15.990489+0000","last_undegraded":"2024-03-06T07:27:15.990489+0000","last_fullsized":"2024-03-06T07:27:15.990489+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:18.186945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.008698+0000","last_change":"2024-03-06T07:27:15.007358+0000","last_active":"2024-03-06T07:27:15.008698+0000","last_peered":"2024-03-06T07:27:15.008698+0000","last_clean":"2024-03-06T07:27:15.008698+0000","last_became_active":"2024-03-06T07:27:15.007096+0000","last_became_peered":"2024-03-06T07:27:15.007096+0000","last_unstale":"2024-03-06T07:27:15.008698+0000","last_undegraded":"2024-03-06T07:27:15.008698+0000","last_fullsized":"2024-03-06T07:27:15.008698+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:58.566040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.988051+0000","last_change":"2024-03-06T07:27:13.989196+0000","last_active":"2024-03-06T07:27:15.988051+0000","last_peered":"2024-03-06T07:27:15.988051+0000","last_clean":"2024-03-06T07:27:15.988051+0000","last_became_active":"2024-03-06T07:27:13.988905+0000","last_became_peered":"2024-03-06T07:27:13.988905+0000","last_unstale":"2024-03-06T07:27:15.988051+0000","last_undegraded":"2024-03-06T07:27:15.988051+0000","last_fullsized":"2024-03-06T07:27:15.988051+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:39.629234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.002588+0000","last_change":"2024-03-06T07:27:14.996929+0000","last_active":"2024-03-06T07:27:15.002588+0000","last_peered":"2024-03-06T07:27:15.002588+0000","last_clean":"2024-03-06T07:27:15.002588+0000","last_became_active":"2024-03-06T07:27:14.996191+0000","last_became_peered":"2024-03-06T07:27:14.996191+0000","last_unstale":"2024-03-06T07:27:15.002588+0000","last_undegraded":"2024-03-06T07:27:15.002588+0000","last_fullsized":"2024-03-06T07:27:15.002588+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:59.786571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.981347+0000","last_change":"2024-03-06T07:27:13.994067+0000","last_active":"2024-03-06T07:27:14.981347+0000","last_peered":"2024-03-06T07:27:14.981347+0000","last_clean":"2024-03-06T07:27:14.981347+0000","last_became_active":"2024-03-06T07:27:13.993531+0000","last_became_peered":"2024-03-06T07:27:13.993531+0000","last_unstale":"2024-03-06T07:27:14.981347+0000","last_undegraded":"2024-03-06T07:27:14.981347+0000","last_fullsized":"2024-03-06T07:27:14.981347+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:27.526863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006585+0000","last_change":"2024-03-06T07:27:15.002276+0000","last_active":"2024-03-06T07:27:15.006585+0000","last_peered":"2024-03-06T07:27:15.006585+0000","last_clean":"2024-03-06T07:27:15.006585+0000","last_became_active":"2024-03-06T07:27:15.002008+0000","last_became_peered":"2024-03-06T07:27:15.002008+0000","last_unstale":"2024-03-06T07:27:15.006585+0000","last_undegraded":"2024-03-06T07:27:15.006585+0000","last_fullsized":"2024-03-06T07:27:15.006585+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:06.357185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987720+0000","last_change":"2024-03-06T07:27:13.988664+0000","last_active":"2024-03-06T07:27:15.987720+0000","last_peered":"2024-03-06T07:27:15.987720+0000","last_clean":"2024-03-06T07:27:15.987720+0000","last_became_active":"2024-03-06T07:27:13.988373+0000","last_became_peered":"2024-03-06T07:27:13.988373+0000","last_unstale":"2024-03-06T07:27:15.987720+0000","last_undegraded":"2024-03-06T07:27:15.987720+0000","last_fullsized":"2024-03-06T07:27:15.987720+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:51.110735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990435+0000","last_change":"2024-03-06T07:27:15.003040+0000","last_active":"2024-03-06T07:27:15.990435+0000","last_peered":"2024-03-06T07:27:15.990435+0000","last_clean":"2024-03-06T07:27:15.990435+0000","last_became_active":"2024-03-06T07:27:15.002349+0000","last_became_peered":"2024-03-06T07:27:15.002349+0000","last_unstale":"2024-03-06T07:27:15.990435+0000","last_undegraded":"2024-03-06T07:27:15.990435+0000","last_fullsized":"2024-03-06T07:27:15.990435+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:07.794199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.981761+0000","last_change":"2024-03-06T07:27:13.990768+0000","last_active":"2024-03-06T07:27:14.981761+0000","last_peered":"2024-03-06T07:27:14.981761+0000","last_clean":"2024-03-06T07:27:14.981761+0000","last_became_active":"2024-03-06T07:27:13.990303+0000","last_became_peered":"2024-03-06T07:27:13.990303+0000","last_unstale":"2024-03-06T07:27:14.981761+0000","last_undegraded":"2024-03-06T07:27:14.981761+0000","last_fullsized":"2024-03-06T07:27:14.981761+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:04.546779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006392+0000","last_change":"2024-03-06T07:27:15.002408+0000","last_active":"2024-03-06T07:27:15.006392+0000","last_peered":"2024-03-06T07:27:15.006392+0000","last_clean":"2024-03-06T07:27:15.006392+0000","last_became_active":"2024-03-06T07:27:15.002122+0000","last_became_peered":"2024-03-06T07:27:15.002122+0000","last_unstale":"2024-03-06T07:27:15.006392+0000","last_undegraded":"2024-03-06T07:27:15.006392+0000","last_fullsized":"2024-03-06T07:27:15.006392+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:22.264605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990698+0000","last_change":"2024-03-06T07:27:13.992258+0000","last_active":"2024-03-06T07:27:15.990698+0000","last_peered":"2024-03-06T07:27:15.990698+0000","last_clean":"2024-03-06T07:27:15.990698+0000","last_became_active":"2024-03-06T07:27:13.991991+0000","last_became_peered":"2024-03-06T07:27:13.991991+0000","last_unstale":"2024-03-06T07:27:15.990698+0000","last_undegraded":"2024-03-06T07:27:15.990698+0000","last_fullsized":"2024-03-06T07:27:15.990698+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:20.081724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003095+0000","last_change":"2024-03-06T07:27:15.000980+0000","last_active":"2024-03-06T07:27:15.003095+0000","last_peered":"2024-03-06T07:27:15.003095+0000","last_clean":"2024-03-06T07:27:15.003095+0000","last_became_active":"2024-03-06T07:27:15.000702+0000","last_became_peered":"2024-03-06T07:27:15.000702+0000","last_unstale":"2024-03-06T07:27:15.003095+0000","last_undegraded":"2024-03-06T07:27:15.003095+0000","last_fullsized":"2024-03-06T07:27:15.003095+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:17.563603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.979590+0000","last_change":"2024-03-06T07:27:13.987019+0000","last_active":"2024-03-06T07:27:15.979590+0000","last_peered":"2024-03-06T07:27:15.979590+0000","last_clean":"2024-03-06T07:27:15.979590+0000","last_became_active":"2024-03-06T07:27:13.986708+0000","last_became_peered":"2024-03-06T07:27:13.986708+0000","last_unstale":"2024-03-06T07:27:15.979590+0000","last_undegraded":"2024-03-06T07:27:15.979590+0000","last_fullsized":"2024-03-06T07:27:15.979590+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:43.503056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.002737+0000","last_change":"2024-03-06T07:27:14.999202+0000","last_active":"2024-03-06T07:27:15.002737+0000","last_peered":"2024-03-06T07:27:15.002737+0000","last_clean":"2024-03-06T07:27:15.002737+0000","last_became_active":"2024-03-06T07:27:14.998732+0000","last_became_peered":"2024-03-06T07:27:14.998732+0000","last_unstale":"2024-03-06T07:27:15.002737+0000","last_undegraded":"2024-03-06T07:27:15.002737+0000","last_fullsized":"2024-03-06T07:27:15.002737+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:54.707369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.988828+0000","last_change":"2024-03-06T07:27:13.990648+0000","last_active":"2024-03-06T07:27:14.988828+0000","last_peered":"2024-03-06T07:27:14.988828+0000","last_clean":"2024-03-06T07:27:14.988828+0000","last_became_active":"2024-03-06T07:27:13.990278+0000","last_became_peered":"2024-03-06T07:27:13.990278+0000","last_unstale":"2024-03-06T07:27:14.988828+0000","last_undegraded":"2024-03-06T07:27:14.988828+0000","last_fullsized":"2024-03-06T07:27:14.988828+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:14.168079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003361+0000","last_change":"2024-03-06T07:27:14.999908+0000","last_active":"2024-03-06T07:27:15.003361+0000","last_peered":"2024-03-06T07:27:15.003361+0000","last_clean":"2024-03-06T07:27:15.003361+0000","last_became_active":"2024-03-06T07:27:14.999282+0000","last_became_peered":"2024-03-06T07:27:14.999282+0000","last_unstale":"2024-03-06T07:27:15.003361+0000","last_undegraded":"2024-03-06T07:27:15.003361+0000","last_fullsized":"2024-03-06T07:27:15.003361+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:15.510457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990783+0000","last_change":"2024-03-06T07:27:13.993319+0000","last_active":"2024-03-06T07:27:15.990783+0000","last_peered":"2024-03-06T07:27:15.990783+0000","last_clean":"2024-03-06T07:27:15.990783+0000","last_became_active":"2024-03-06T07:27:13.993046+0000","last_became_peered":"2024-03-06T07:27:13.993046+0000","last_unstale":"2024-03-06T07:27:15.990783+0000","last_undegraded":"2024-03-06T07:27:15.990783+0000","last_fullsized":"2024-03-06T07:27:15.990783+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:13.358651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006726+0000","last_change":"2024-03-06T07:27:14.992939+0000","last_active":"2024-03-06T07:27:15.006726+0000","last_peered":"2024-03-06T07:27:15.006726+0000","last_clean":"2024-03-06T07:27:15.006726+0000","last_became_active":"2024-03-06T07:27:14.992622+0000","last_became_peered":"2024-03-06T07:27:14.992622+0000","last_unstale":"2024-03-06T07:27:15.006726+0000","last_undegraded":"2024-03-06T07:27:15.006726+0000","last_fullsized":"2024-03-06T07:27:15.006726+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:19.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990541+0000","last_change":"2024-03-06T07:27:13.986130+0000","last_active":"2024-03-06T07:27:15.990541+0000","last_peered":"2024-03-06T07:27:15.990541+0000","last_clean":"2024-03-06T07:27:15.990541+0000","last_became_active":"2024-03-06T07:27:13.985828+0000","last_became_peered":"2024-03-06T07:27:13.985828+0000","last_unstale":"2024-03-06T07:27:15.990541+0000","last_undegraded":"2024-03-06T07:27:15.990541+0000","last_fullsized":"2024-03-06T07:27:15.990541+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:53.529526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.008902+0000","last_change":"2024-03-06T07:27:15.005319+0000","last_active":"2024-03-06T07:27:15.008902+0000","last_peered":"2024-03-06T07:27:15.008902+0000","last_clean":"2024-03-06T07:27:15.008902+0000","last_became_active":"2024-03-06T07:27:15.005010+0000","last_became_peered":"2024-03-06T07:27:15.005010+0000","last_unstale":"2024-03-06T07:27:15.008902+0000","last_undegraded":"2024-03-06T07:27:15.008902+0000","last_fullsized":"2024-03-06T07:27:15.008902+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:51.665162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.987508+0000","last_change":"2024-03-06T07:27:13.986231+0000","last_active":"2024-03-06T07:27:14.987508+0000","last_peered":"2024-03-06T07:27:14.987508+0000","last_clean":"2024-03-06T07:27:14.987508+0000","last_became_active":"2024-03-06T07:27:13.985948+0000","last_became_peered":"2024-03-06T07:27:13.985948+0000","last_unstale":"2024-03-06T07:27:14.987508+0000","last_undegraded":"2024-03-06T07:27:14.987508+0000","last_fullsized":"2024-03-06T07:27:14.987508+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:21.262613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006231+0000","last_change":"2024-03-06T07:27:15.000818+0000","last_active":"2024-03-06T07:27:15.006231+0000","last_peered":"2024-03-06T07:27:15.006231+0000","last_clean":"2024-03-06T07:27:15.006231+0000","last_became_active":"2024-03-06T07:27:14.999909+0000","last_became_peered":"2024-03-06T07:27:14.999909+0000","last_unstale":"2024-03-06T07:27:15.006231+0000","last_undegraded":"2024-03-06T07:27:15.006231+0000","last_fullsized":"2024-03-06T07:27:15.006231+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:57.351882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.981360+0000","last_change":"2024-03-06T07:27:13.996405+0000","last_active":"2024-03-06T07:27:14.981360+0000","last_peered":"2024-03-06T07:27:14.981360+0000","last_clean":"2024-03-06T07:27:14.981360+0000","last_became_active":"2024-03-06T07:27:13.996115+0000","last_became_peered":"2024-03-06T07:27:13.996115+0000","last_unstale":"2024-03-06T07:27:14.981360+0000","last_undegraded":"2024-03-06T07:27:14.981360+0000","last_fullsized":"2024-03-06T07:27:14.981360+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:18.398153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.995747+0000","last_change":"2024-03-06T07:27:14.991513+0000","last_active":"2024-03-06T07:27:14.995747+0000","last_peered":"2024-03-06T07:27:14.995747+0000","last_clean":"2024-03-06T07:27:14.995747+0000","last_became_active":"2024-03-06T07:27:14.991247+0000","last_became_peered":"2024-03-06T07:27:14.991247+0000","last_unstale":"2024-03-06T07:27:14.995747+0000","last_undegraded":"2024-03-06T07:27:14.995747+0000","last_fullsized":"2024-03-06T07:27:14.995747+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:58.244707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.981911+0000","last_change":"2024-03-06T07:27:13.991042+0000","last_active":"2024-03-06T07:27:14.981911+0000","last_peered":"2024-03-06T07:27:14.981911+0000","last_clean":"2024-03-06T07:27:14.981911+0000","last_became_active":"2024-03-06T07:27:13.990421+0000","last_became_peered":"2024-03-06T07:27:13.990421+0000","last_unstale":"2024-03-06T07:27:14.981911+0000","last_undegraded":"2024-03-06T07:27:14.981911+0000","last_fullsized":"2024-03-06T07:27:14.981911+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:00.043646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990434+0000","last_change":"2024-03-06T07:27:14.998464+0000","last_active":"2024-03-06T07:27:15.990434+0000","last_peered":"2024-03-06T07:27:15.990434+0000","last_clean":"2024-03-06T07:27:15.990434+0000","last_became_active":"2024-03-06T07:27:14.997264+0000","last_became_peered":"2024-03-06T07:27:14.997264+0000","last_unstale":"2024-03-06T07:27:15.990434+0000","last_undegraded":"2024-03-06T07:27:15.990434+0000","last_fullsized":"2024-03-06T07:27:15.990434+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:59.786571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990402+0000","last_change":"2024-03-06T07:27:13.988793+0000","last_active":"2024-03-06T07:27:15.990402+0000","last_peered":"2024-03-06T07:27:15.990402+0000","last_clean":"2024-03-06T07:27:15.990402+0000","last_became_active":"2024-03-06T07:27:13.988528+0000","last_became_peered":"2024-03-06T07:27:13.988528+0000","last_unstale":"2024-03-06T07:27:15.990402+0000","last_undegraded":"2024-03-06T07:27:15.990402+0000","last_fullsized":"2024-03-06T07:27:15.990402+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:51.110735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":326,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.990838+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:27:14.990838+0000","last_peered":"2024-03-06T07:27:14.990838+0000","last_clean":"2024-03-06T07:27:14.990838+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:27:14.990838+0000","last_undegraded":"2024-03-06T07:27:14.990838+0000","last_fullsized":"2024-03-06T07:27:14.990838+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37236,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334604,"statfs":{"total":96636764160,"available":96598634496,"internally_reserved":0,"allocated":1933312,"data_stored":1165421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574919,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1335296,"data_stored":574507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40828,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331012,"statfs":{"total":96636764160,"available":96594956288,"internally_reserved":0,"allocated":1351680,"data_stored":581341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1331200,"data_stored":574437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574918,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40824,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331016,"statfs":{"total":96636764160,"available":96594960384,"internally_reserved":0,"allocated":1347584,"data_stored":576119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36644,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335196,"statfs":{"total":96636764160,"available":96599240704,"internally_reserved":0,"allocated":1331200,"data_stored":574437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40812,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331028,"statfs":{"total":96636764160,"available":96594972672,"internally_reserved":0,"allocated":1339392,"data_stored":574561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41404,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330436,"statfs":{"total":96636764160,"available":96594366464,"internally_reserved":0,"allocated":1945600,"data_stored":1166529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:18.080 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:18.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.215+0000 7f8c74c83640 1 Processor -- start 2024-03-06T07:27:18.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.215+0000 7f8c74c83640 1 -- start start 2024-03-06T07:27:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.216+0000 7f8c74c83640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8c7015e090 0x7f8c700d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.216+0000 7f8c74c83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c700d2920 0x7f8c700b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.216+0000 7f8c74c83640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7015e470 con 0x7f8c700b3b20 2024-03-06T07:27:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.216+0000 7f8c74c83640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c700d18f0 con 0x7f8c7015e090 2024-03-06T07:27:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.216+0000 7f8c74c83640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c700d1b90 con 0x7f8c700d2920 2024-03-06T07:27:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.216+0000 7f8c6ed9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8c700b3b20 0x7f8c700d0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60824/0 (socket says 172.21.15.59:60824) 2024-03-06T07:27:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.216+0000 7f8c6ed9d640 1 -- 172.21.15.59:0/1241502462 learned_addr learned my addr 172.21.15.59:0/1241502462 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.216+0000 7f8c6dd9b640 1 --2- 172.21.15.59:0/1241502462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c700d2920 0x7f8c700b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.216+0000 7f8c6e59c640 1 --2- 172.21.15.59:0/1241502462 >> v2:172.21.15.164:3300/0 conn(0x7f8c7015e090 0x7f8c700d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.216+0000 7f8c6d59a640 1 -- 172.21.15.59:0/1241502462 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329919895 0 0) 0x7f8c7015e470 con 0x7f8c700b3b20 2024-03-06T07:27:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.217+0000 7f8c6d59a640 1 -- 172.21.15.59:0/1241502462 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c5c003520 con 0x7f8c700b3b20 2024-03-06T07:27:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.217+0000 7f8c6e59c640 1 -- 172.21.15.59:0/1241502462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c700d2920 msgr2=0x7f8c700b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.217+0000 7f8c6e59c640 1 --2- 172.21.15.59:0/1241502462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c700d2920 0x7f8c700b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.217+0000 7f8c6e59c640 1 -- 172.21.15.59:0/1241502462 >> v1:172.21.15.59:6789/0 conn(0x7f8c700b3b20 legacy=0x7f8c700d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.217+0000 7f8c6e59c640 1 -- 172.21.15.59:0/1241502462 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c700ceb50 con 0x7f8c7015e090 2024-03-06T07:27:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.217+0000 7f8c6dd9b640 1 --2- 172.21.15.59:0/1241502462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c700d2920 0x7f8c700b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.217+0000 7f8c6e59c640 1 --2- 172.21.15.59:0/1241502462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8c7015e090 0x7f8c700d47a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8c680097a0 tx=0x7f8c6802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=114bf00ddbcae171 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.218+0000 7f8c6d59a640 1 -- 172.21.15.59:0/1241502462 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68035020 con 0x7f8c7015e090 2024-03-06T07:27:18.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.218+0000 7f8c6d59a640 1 -- 172.21.15.59:0/1241502462 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c68031830 con 0x7f8c7015e090 2024-03-06T07:27:18.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.218+0000 7f8c6d59a640 1 -- 172.21.15.59:0/1241502462 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68031c00 con 0x7f8c7015e090 2024-03-06T07:27:18.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.218+0000 7f8c74c83640 1 -- 172.21.15.59:0/1241502462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8c7015e090 msgr2=0x7f8c700d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.218+0000 7f8c74c83640 1 --2- 172.21.15.59:0/1241502462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8c7015e090 0x7f8c700d47a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8c680097a0 tx=0x7f8c6802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.218+0000 7f8c74c83640 1 -- 172.21.15.59:0/1241502462 shutdown_connections 2024-03-06T07:27:18.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.218+0000 7f8c74c83640 1 --2- 172.21.15.59:0/1241502462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c700d2920 0x7f8c700b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.218+0000 7f8c74c83640 1 --2- 172.21.15.59:0/1241502462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8c7015e090 0x7f8c700d47a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.218+0000 7f8c74c83640 1 -- 172.21.15.59:0/1241502462 >> 172.21.15.59:0/1241502462 conn(0x7f8c700d3c80 msgr2=0x7f8c700d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.218+0000 7f8c74c83640 1 -- 172.21.15.59:0/1241502462 shutdown_connections 2024-03-06T07:27:18.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.218+0000 7f8c74c83640 1 -- 172.21.15.59:0/1241502462 wait complete. 2024-03-06T07:27:18.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c74c83640 1 Processor -- start 2024-03-06T07:27:18.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c74c83640 1 -- start start 2024-03-06T07:27:18.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c74c83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c700d2920 0x7f8c70151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c74c83640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7015e090 0x7f8c700b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c74c83640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8c7015f1a0 0x7f8c700b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c74c83640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c700ce800 con 0x7f8c7015e090 2024-03-06T07:27:18.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c74c83640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c700b2df0 con 0x7f8c7015f1a0 2024-03-06T07:27:18.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c74c83640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c700d0d60 con 0x7f8c700d2920 2024-03-06T07:27:18.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c6dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7015e090 0x7f8c700b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c6dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7015e090 0x7f8c700b0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37328/0 (socket says 172.21.15.59:37328) 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c6dd9b640 1 -- 172.21.15.59:0/831890851 learned_addr learned my addr 172.21.15.59:0/831890851 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c6e59c640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c700d2920 0x7f8c70151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.220+0000 7f8c6ed9d640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8c7015f1a0 0x7f8c700b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.221+0000 7f8c6dd9b640 1 -- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c700d2920 msgr2=0x7f8c70151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.221+0000 7f8c6dd9b640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c700d2920 0x7f8c70151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.221+0000 7f8c6dd9b640 1 -- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8c7015f1a0 msgr2=0x7f8c700b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.221+0000 7f8c6dd9b640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8c7015f1a0 0x7f8c700b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.221+0000 7f8c6dd9b640 1 -- 172.21.15.59:0/831890851 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c700b19e0 con 0x7f8c7015e090 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.221+0000 7f8c6dd9b640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7015e090 0x7f8c700b0930 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f8c5800ebe0 tx=0x7f8c5800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.221+0000 7f8c66ffd640 1 -- 172.21.15.59:0/831890851 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c58017070 con 0x7f8c7015e090 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.221+0000 7f8c66ffd640 1 -- 172.21.15.59:0/831890851 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5800cc30 con 0x7f8c7015e090 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.221+0000 7f8c74c83640 1 -- 172.21.15.59:0/831890851 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c700b1cc0 con 0x7f8c7015e090 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.221+0000 7f8c66ffd640 1 -- 172.21.15.59:0/831890851 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c58013470 con 0x7f8c7015e090 2024-03-06T07:27:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.221+0000 7f8c74c83640 1 -- 172.21.15.59:0/831890851 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c70151d50 con 0x7f8c7015e090 2024-03-06T07:27:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.222+0000 7f8c74c83640 1 -- 172.21.15.59:0/831890851 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7004f330 con 0x7f8c7015e090 2024-03-06T07:27:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.223+0000 7f8c66ffd640 1 -- 172.21.15.59:0/831890851 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c5801a070 con 0x7f8c7015e090 2024-03-06T07:27:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.223+0000 7f8c66ffd640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8c5006b4d0 0x7f8c5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.223+0000 7f8c66ffd640 1 -- 172.21.15.59:0/831890851 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c5808a110 con 0x7f8c7015e090 2024-03-06T07:27:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.223+0000 7f8c6e59c640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8c5006b4d0 0x7f8c5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.224+0000 7f8c6e59c640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8c5006b4d0 0x7f8c5006d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8c68036000 tx=0x7f8c6803c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.227+0000 7f8c66ffd640 1 -- 172.21.15.59:0/831890851 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c58058940 con 0x7f8c7015e090 2024-03-06T07:27:18.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.404+0000 7f8c74c83640 1 -- 172.21.15.59:0/831890851 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c70162270 con 0x7f8c5006b4d0 2024-03-06T07:27:18.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.410+0000 7f8c66ffd640 1 -- 172.21.15.59:0/831890851 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252434 (secure 0 0 0) 0x7f8c70162270 con 0x7f8c5006b4d0 2024-03-06T07:27:18.415 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:27:18.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 -- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8c5006b4d0 msgr2=0x7f8c5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8c5006b4d0 0x7f8c5006d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8c68036000 tx=0x7f8c6803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:18.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 -- 172.21.15.59:0/831890851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7015e090 msgr2=0x7f8c700b0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7015e090 0x7f8c700b0930 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f8c5800ebe0 tx=0x7f8c5800c520 comp rx=0 tx=0).stop 2024-03-06T07:27:18.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 -- 172.21.15.59:0/831890851 shutdown_connections 2024-03-06T07:27:18.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8c5006b4d0 0x7f8c5006d990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8c7015f1a0 0x7f8c700b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8c7015e090 0x7f8c700b0930 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 --2- 172.21.15.59:0/831890851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c700d2920 0x7f8c70151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 -- 172.21.15.59:0/831890851 >> 172.21.15.59:0/831890851 conn(0x7f8c700d3c80 msgr2=0x7f8c7006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 -- 172.21.15.59:0/831890851 shutdown_connections 2024-03-06T07:27:18.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.413+0000 7f8c74c83640 1 -- 172.21.15.59:0/831890851 wait complete. 2024-03-06T07:27:18.426 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-03-06T07:27:17.371614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319372,"kb_used_data":11636,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754655348,"statfs":{"total":773094113280,"available":772767076352,"internally_reserved":0,"allocated":11915264,"data_stored":5787352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":314953265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.401813"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.002249+0000","last_change":"2024-03-06T07:27:14.999220+0000","last_active":"2024-03-06T07:27:15.002249+0000","last_peered":"2024-03-06T07:27:15.002249+0000","last_clean":"2024-03-06T07:27:15.002249+0000","last_became_active":"2024-03-06T07:27:14.998932+0000","last_became_peered":"2024-03-06T07:27:14.998932+0000","last_unstale":"2024-03-06T07:27:15.002249+0000","last_undegraded":"2024-03-06T07:27:15.002249+0000","last_fullsized":"2024-03-06T07:27:15.002249+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:25.925986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006793+0000","last_change":"2024-03-06T07:27:14.998126+0000","last_active":"2024-03-06T07:27:15.006793+0000","last_peered":"2024-03-06T07:27:15.006793+0000","last_clean":"2024-03-06T07:27:15.006793+0000","last_became_active":"2024-03-06T07:27:14.997828+0000","last_became_peered":"2024-03-06T07:27:14.997828+0000","last_unstale":"2024-03-06T07:27:15.006793+0000","last_undegraded":"2024-03-06T07:27:15.006793+0000","last_fullsized":"2024-03-06T07:27:15.006793+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:37.880813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.008700+0000","last_change":"2024-03-06T07:27:15.007394+0000","last_active":"2024-03-06T07:27:15.008700+0000","last_peered":"2024-03-06T07:27:15.008700+0000","last_clean":"2024-03-06T07:27:15.008700+0000","last_became_active":"2024-03-06T07:27:15.007120+0000","last_became_peered":"2024-03-06T07:27:15.007120+0000","last_unstale":"2024-03-06T07:27:15.008700+0000","last_undegraded":"2024-03-06T07:27:15.008700+0000","last_fullsized":"2024-03-06T07:27:15.008700+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:09.140300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.009551+0000","last_change":"2024-03-06T07:27:15.004996+0000","last_active":"2024-03-06T07:27:15.009551+0000","last_peered":"2024-03-06T07:27:15.009551+0000","last_clean":"2024-03-06T07:27:15.009551+0000","last_became_active":"2024-03-06T07:27:15.003336+0000","last_became_peered":"2024-03-06T07:27:15.003336+0000","last_unstale":"2024-03-06T07:27:15.009551+0000","last_undegraded":"2024-03-06T07:27:15.009551+0000","last_fullsized":"2024-03-06T07:27:15.009551+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:59.743756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.001632+0000","last_change":"2024-03-06T07:27:14.993845+0000","last_active":"2024-03-06T07:27:15.001632+0000","last_peered":"2024-03-06T07:27:15.001632+0000","last_clean":"2024-03-06T07:27:15.001632+0000","last_became_active":"2024-03-06T07:27:14.990629+0000","last_became_peered":"2024-03-06T07:27:14.990629+0000","last_unstale":"2024-03-06T07:27:15.001632+0000","last_undegraded":"2024-03-06T07:27:15.001632+0000","last_fullsized":"2024-03-06T07:27:15.001632+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:04.840675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987801+0000","last_change":"2024-03-06T07:27:14.987595+0000","last_active":"2024-03-06T07:27:15.987801+0000","last_peered":"2024-03-06T07:27:15.987801+0000","last_clean":"2024-03-06T07:27:15.987801+0000","last_became_active":"2024-03-06T07:27:14.987334+0000","last_became_peered":"2024-03-06T07:27:14.987334+0000","last_unstale":"2024-03-06T07:27:15.987801+0000","last_undegraded":"2024-03-06T07:27:15.987801+0000","last_fullsized":"2024-03-06T07:27:15.987801+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:36.824848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990367+0000","last_change":"2024-03-06T07:27:14.999083+0000","last_active":"2024-03-06T07:27:15.990367+0000","last_peered":"2024-03-06T07:27:15.990367+0000","last_clean":"2024-03-06T07:27:15.990367+0000","last_became_active":"2024-03-06T07:27:14.998768+0000","last_became_peered":"2024-03-06T07:27:14.998768+0000","last_unstale":"2024-03-06T07:27:15.990367+0000","last_undegraded":"2024-03-06T07:27:15.990367+0000","last_fullsized":"2024-03-06T07:27:15.990367+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:13.525561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003166+0000","last_change":"2024-03-06T07:27:14.997221+0000","last_active":"2024-03-06T07:27:15.003166+0000","last_peered":"2024-03-06T07:27:15.003166+0000","last_clean":"2024-03-06T07:27:15.003166+0000","last_became_active":"2024-03-06T07:27:14.996626+0000","last_became_peered":"2024-03-06T07:27:14.996626+0000","last_unstale":"2024-03-06T07:27:15.003166+0000","last_undegraded":"2024-03-06T07:27:15.003166+0000","last_fullsized":"2024-03-06T07:27:15.003166+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:37.880813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.999641+0000","last_change":"2024-03-06T07:27:14.998232+0000","last_active":"2024-03-06T07:27:14.999641+0000","last_peered":"2024-03-06T07:27:14.999641+0000","last_clean":"2024-03-06T07:27:14.999641+0000","last_became_active":"2024-03-06T07:27:14.997790+0000","last_became_peered":"2024-03-06T07:27:14.997790+0000","last_unstale":"2024-03-06T07:27:14.999641+0000","last_undegraded":"2024-03-06T07:27:14.999641+0000","last_fullsized":"2024-03-06T07:27:14.999641+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:07.794199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.009543+0000","last_change":"2024-03-06T07:27:15.004383+0000","last_active":"2024-03-06T07:27:15.009543+0000","last_peered":"2024-03-06T07:27:15.009543+0000","last_clean":"2024-03-06T07:27:15.009543+0000","last_became_active":"2024-03-06T07:27:15.002877+0000","last_became_peered":"2024-03-06T07:27:15.002877+0000","last_unstale":"2024-03-06T07:27:15.009543+0000","last_undegraded":"2024-03-06T07:27:15.009543+0000","last_fullsized":"2024-03-06T07:27:15.009543+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:18:33.221493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006997+0000","last_change":"2024-03-06T07:27:15.000651+0000","last_active":"2024-03-06T07:27:15.006997+0000","last_peered":"2024-03-06T07:27:15.006997+0000","last_clean":"2024-03-06T07:27:15.006997+0000","last_became_active":"2024-03-06T07:27:15.000353+0000","last_became_peered":"2024-03-06T07:27:15.000353+0000","last_unstale":"2024-03-06T07:27:15.006997+0000","last_undegraded":"2024-03-06T07:27:15.006997+0000","last_fullsized":"2024-03-06T07:27:15.006997+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:12.434709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007029+0000","last_change":"2024-03-06T07:27:15.002602+0000","last_active":"2024-03-06T07:27:15.007029+0000","last_peered":"2024-03-06T07:27:15.007029+0000","last_clean":"2024-03-06T07:27:15.007029+0000","last_became_active":"2024-03-06T07:27:15.002340+0000","last_became_peered":"2024-03-06T07:27:15.002340+0000","last_unstale":"2024-03-06T07:27:15.007029+0000","last_undegraded":"2024-03-06T07:27:15.007029+0000","last_fullsized":"2024-03-06T07:27:15.007029+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:49.838205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006581+0000","last_change":"2024-03-06T07:27:15.005196+0000","last_active":"2024-03-06T07:27:15.006581+0000","last_peered":"2024-03-06T07:27:15.006581+0000","last_clean":"2024-03-06T07:27:15.006581+0000","last_became_active":"2024-03-06T07:27:15.004920+0000","last_became_peered":"2024-03-06T07:27:15.004920+0000","last_unstale":"2024-03-06T07:27:15.006581+0000","last_undegraded":"2024-03-06T07:27:15.006581+0000","last_fullsized":"2024-03-06T07:27:15.006581+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:51.665162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006041+0000","last_change":"2024-03-06T07:27:14.999595+0000","last_active":"2024-03-06T07:27:15.006041+0000","last_peered":"2024-03-06T07:27:15.006041+0000","last_clean":"2024-03-06T07:27:15.006041+0000","last_became_active":"2024-03-06T07:27:14.999266+0000","last_became_peered":"2024-03-06T07:27:14.999266+0000","last_unstale":"2024-03-06T07:27:15.006041+0000","last_undegraded":"2024-03-06T07:27:15.006041+0000","last_fullsized":"2024-03-06T07:27:15.006041+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:27.696930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987764+0000","last_change":"2024-03-06T07:27:14.988732+0000","last_active":"2024-03-06T07:27:15.987764+0000","last_peered":"2024-03-06T07:27:15.987764+0000","last_clean":"2024-03-06T07:27:15.987764+0000","last_became_active":"2024-03-06T07:27:14.988467+0000","last_became_peered":"2024-03-06T07:27:14.988467+0000","last_unstale":"2024-03-06T07:27:15.987764+0000","last_undegraded":"2024-03-06T07:27:15.987764+0000","last_fullsized":"2024-03-06T07:27:15.987764+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:19.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006991+0000","last_change":"2024-03-06T07:27:15.002743+0000","last_active":"2024-03-06T07:27:15.006991+0000","last_peered":"2024-03-06T07:27:15.006991+0000","last_clean":"2024-03-06T07:27:15.006991+0000","last_became_active":"2024-03-06T07:27:15.002292+0000","last_became_peered":"2024-03-06T07:27:15.002292+0000","last_unstale":"2024-03-06T07:27:15.006991+0000","last_undegraded":"2024-03-06T07:27:15.006991+0000","last_fullsized":"2024-03-06T07:27:15.006991+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:13:01.277640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007096+0000","last_change":"2024-03-06T07:27:15.001098+0000","last_active":"2024-03-06T07:27:15.007096+0000","last_peered":"2024-03-06T07:27:15.007096+0000","last_clean":"2024-03-06T07:27:15.007096+0000","last_became_active":"2024-03-06T07:27:15.000815+0000","last_became_peered":"2024-03-06T07:27:15.000815+0000","last_unstale":"2024-03-06T07:27:15.007096+0000","last_undegraded":"2024-03-06T07:27:15.007096+0000","last_fullsized":"2024-03-06T07:27:15.007096+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:06.357185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.011651+0000","last_change":"2024-03-06T07:27:15.007615+0000","last_active":"2024-03-06T07:27:15.011651+0000","last_peered":"2024-03-06T07:27:15.011651+0000","last_clean":"2024-03-06T07:27:15.011651+0000","last_became_active":"2024-03-06T07:27:15.007251+0000","last_became_peered":"2024-03-06T07:27:15.007251+0000","last_unstale":"2024-03-06T07:27:15.011651+0000","last_undegraded":"2024-03-06T07:27:15.011651+0000","last_fullsized":"2024-03-06T07:27:15.011651+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:19.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006637+0000","last_change":"2024-03-06T07:27:15.002925+0000","last_active":"2024-03-06T07:27:15.006637+0000","last_peered":"2024-03-06T07:27:15.006637+0000","last_clean":"2024-03-06T07:27:15.006637+0000","last_became_active":"2024-03-06T07:27:15.002643+0000","last_became_peered":"2024-03-06T07:27:15.002643+0000","last_unstale":"2024-03-06T07:27:15.006637+0000","last_undegraded":"2024-03-06T07:27:15.006637+0000","last_fullsized":"2024-03-06T07:27:15.006637+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:39.117581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007190+0000","last_change":"2024-03-06T07:27:15.002720+0000","last_active":"2024-03-06T07:27:15.007190+0000","last_peered":"2024-03-06T07:27:15.007190+0000","last_clean":"2024-03-06T07:27:15.007190+0000","last_became_active":"2024-03-06T07:27:15.002378+0000","last_became_peered":"2024-03-06T07:27:15.002378+0000","last_unstale":"2024-03-06T07:27:15.007190+0000","last_undegraded":"2024-03-06T07:27:15.007190+0000","last_fullsized":"2024-03-06T07:27:15.007190+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:59.098860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007783+0000","last_change":"2024-03-06T07:27:14.999312+0000","last_active":"2024-03-06T07:27:15.007783+0000","last_peered":"2024-03-06T07:27:15.007783+0000","last_clean":"2024-03-06T07:27:15.007783+0000","last_became_active":"2024-03-06T07:27:14.998844+0000","last_became_peered":"2024-03-06T07:27:14.998844+0000","last_unstale":"2024-03-06T07:27:15.007783+0000","last_undegraded":"2024-03-06T07:27:15.007783+0000","last_fullsized":"2024-03-06T07:27:15.007783+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:57.351882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990524+0000","last_change":"2024-03-06T07:27:15.002714+0000","last_active":"2024-03-06T07:27:15.990524+0000","last_peered":"2024-03-06T07:27:15.990524+0000","last_clean":"2024-03-06T07:27:15.990524+0000","last_became_active":"2024-03-06T07:27:15.002345+0000","last_became_peered":"2024-03-06T07:27:15.002345+0000","last_unstale":"2024-03-06T07:27:15.990524+0000","last_undegraded":"2024-03-06T07:27:15.990524+0000","last_fullsized":"2024-03-06T07:27:15.990524+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:54.707369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006772+0000","last_change":"2024-03-06T07:27:14.996096+0000","last_active":"2024-03-06T07:27:15.006772+0000","last_peered":"2024-03-06T07:27:15.006772+0000","last_clean":"2024-03-06T07:27:15.006772+0000","last_became_active":"2024-03-06T07:27:14.995823+0000","last_became_peered":"2024-03-06T07:27:14.995823+0000","last_unstale":"2024-03-06T07:27:15.006772+0000","last_undegraded":"2024-03-06T07:27:15.006772+0000","last_fullsized":"2024-03-06T07:27:15.006772+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:36.824848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990397+0000","last_change":"2024-03-06T07:27:14.999164+0000","last_active":"2024-03-06T07:27:15.990397+0000","last_peered":"2024-03-06T07:27:15.990397+0000","last_clean":"2024-03-06T07:27:15.990397+0000","last_became_active":"2024-03-06T07:27:14.998896+0000","last_became_peered":"2024-03-06T07:27:14.998896+0000","last_unstale":"2024-03-06T07:27:15.990397+0000","last_undegraded":"2024-03-06T07:27:15.990397+0000","last_fullsized":"2024-03-06T07:27:15.990397+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:46.411357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007118+0000","last_change":"2024-03-06T07:27:15.001662+0000","last_active":"2024-03-06T07:27:15.007118+0000","last_peered":"2024-03-06T07:27:15.007118+0000","last_clean":"2024-03-06T07:27:15.007118+0000","last_became_active":"2024-03-06T07:27:15.001377+0000","last_became_peered":"2024-03-06T07:27:15.001377+0000","last_unstale":"2024-03-06T07:27:15.007118+0000","last_undegraded":"2024-03-06T07:27:15.007118+0000","last_fullsized":"2024-03-06T07:27:15.007118+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:39.117581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.004520+0000","last_change":"2024-03-06T07:27:15.000099+0000","last_active":"2024-03-06T07:27:15.004520+0000","last_peered":"2024-03-06T07:27:15.004520+0000","last_clean":"2024-03-06T07:27:15.004520+0000","last_became_active":"2024-03-06T07:27:14.999838+0000","last_became_peered":"2024-03-06T07:27:14.999838+0000","last_unstale":"2024-03-06T07:27:15.004520+0000","last_undegraded":"2024-03-06T07:27:15.004520+0000","last_fullsized":"2024-03-06T07:27:15.004520+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:13.372370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987905+0000","last_change":"2024-03-06T07:27:14.992914+0000","last_active":"2024-03-06T07:27:15.987905+0000","last_peered":"2024-03-06T07:27:15.987905+0000","last_clean":"2024-03-06T07:27:15.987905+0000","last_became_active":"2024-03-06T07:27:14.992630+0000","last_became_peered":"2024-03-06T07:27:14.992630+0000","last_unstale":"2024-03-06T07:27:15.987905+0000","last_undegraded":"2024-03-06T07:27:15.987905+0000","last_fullsized":"2024-03-06T07:27:15.987905+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:41:37.880813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990732+0000","last_change":"2024-03-06T07:27:14.999901+0000","last_active":"2024-03-06T07:27:15.990732+0000","last_peered":"2024-03-06T07:27:15.990732+0000","last_clean":"2024-03-06T07:27:15.990732+0000","last_became_active":"2024-03-06T07:27:14.999412+0000","last_became_peered":"2024-03-06T07:27:14.999412+0000","last_unstale":"2024-03-06T07:27:15.990732+0000","last_undegraded":"2024-03-06T07:27:15.990732+0000","last_fullsized":"2024-03-06T07:27:15.990732+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:17.563603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006786+0000","last_change":"2024-03-06T07:27:14.998165+0000","last_active":"2024-03-06T07:27:15.006786+0000","last_peered":"2024-03-06T07:27:15.006786+0000","last_clean":"2024-03-06T07:27:15.006786+0000","last_became_active":"2024-03-06T07:27:14.997874+0000","last_became_peered":"2024-03-06T07:27:14.997874+0000","last_unstale":"2024-03-06T07:27:15.006786+0000","last_undegraded":"2024-03-06T07:27:15.006786+0000","last_fullsized":"2024-03-06T07:27:15.006786+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:01.873864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990446+0000","last_change":"2024-03-06T07:27:14.988198+0000","last_active":"2024-03-06T07:27:15.990446+0000","last_peered":"2024-03-06T07:27:15.990446+0000","last_clean":"2024-03-06T07:27:15.990446+0000","last_became_active":"2024-03-06T07:27:14.986976+0000","last_became_peered":"2024-03-06T07:27:14.986976+0000","last_unstale":"2024-03-06T07:27:15.990446+0000","last_undegraded":"2024-03-06T07:27:15.990446+0000","last_fullsized":"2024-03-06T07:27:15.990446+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:15.510457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007037+0000","last_change":"2024-03-06T07:27:15.003601+0000","last_active":"2024-03-06T07:27:15.007037+0000","last_peered":"2024-03-06T07:27:15.007037+0000","last_clean":"2024-03-06T07:27:15.007037+0000","last_became_active":"2024-03-06T07:27:15.003349+0000","last_became_peered":"2024-03-06T07:27:15.003349+0000","last_unstale":"2024-03-06T07:27:15.007037+0000","last_undegraded":"2024-03-06T07:27:15.007037+0000","last_fullsized":"2024-03-06T07:27:15.007037+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:31.291940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987769+0000","last_change":"2024-03-06T07:27:14.994279+0000","last_active":"2024-03-06T07:27:15.987769+0000","last_peered":"2024-03-06T07:27:15.987769+0000","last_clean":"2024-03-06T07:27:15.987769+0000","last_became_active":"2024-03-06T07:27:14.992622+0000","last_became_peered":"2024-03-06T07:27:14.992622+0000","last_unstale":"2024-03-06T07:27:15.987769+0000","last_undegraded":"2024-03-06T07:27:15.987769+0000","last_fullsized":"2024-03-06T07:27:15.987769+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:40.108009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987939+0000","last_change":"2024-03-06T07:27:14.988071+0000","last_active":"2024-03-06T07:27:15.987939+0000","last_peered":"2024-03-06T07:27:15.987939+0000","last_clean":"2024-03-06T07:27:15.987939+0000","last_became_active":"2024-03-06T07:27:14.987324+0000","last_became_peered":"2024-03-06T07:27:14.987324+0000","last_unstale":"2024-03-06T07:27:15.987939+0000","last_undegraded":"2024-03-06T07:27:15.987939+0000","last_fullsized":"2024-03-06T07:27:15.987939+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:01.873864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990732+0000","last_change":"2024-03-06T07:27:13.985966+0000","last_active":"2024-03-06T07:27:15.990732+0000","last_peered":"2024-03-06T07:27:15.990732+0000","last_clean":"2024-03-06T07:27:15.990732+0000","last_became_active":"2024-03-06T07:27:13.985623+0000","last_became_peered":"2024-03-06T07:27:13.985623+0000","last_unstale":"2024-03-06T07:27:15.990732+0000","last_undegraded":"2024-03-06T07:27:15.990732+0000","last_fullsized":"2024-03-06T07:27:15.990732+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:48.422576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003444+0000","last_change":"2024-03-06T07:27:15.000616+0000","last_active":"2024-03-06T07:27:15.003444+0000","last_peered":"2024-03-06T07:27:15.003444+0000","last_clean":"2024-03-06T07:27:15.003444+0000","last_became_active":"2024-03-06T07:27:15.000347+0000","last_became_peered":"2024-03-06T07:27:15.000347+0000","last_unstale":"2024-03-06T07:27:15.003444+0000","last_undegraded":"2024-03-06T07:27:15.003444+0000","last_fullsized":"2024-03-06T07:27:15.003444+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:42.996423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.996810+0000","last_change":"2024-03-06T07:27:13.988934+0000","last_active":"2024-03-06T07:27:14.996810+0000","last_peered":"2024-03-06T07:27:14.996810+0000","last_clean":"2024-03-06T07:27:14.996810+0000","last_became_active":"2024-03-06T07:27:13.988646+0000","last_became_peered":"2024-03-06T07:27:13.988646+0000","last_unstale":"2024-03-06T07:27:14.996810+0000","last_undegraded":"2024-03-06T07:27:14.996810+0000","last_fullsized":"2024-03-06T07:27:14.996810+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:03:35.822855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.988068+0000","last_change":"2024-03-06T07:27:14.991900+0000","last_active":"2024-03-06T07:27:15.988068+0000","last_peered":"2024-03-06T07:27:15.988068+0000","last_clean":"2024-03-06T07:27:15.988068+0000","last_became_active":"2024-03-06T07:27:14.991622+0000","last_became_peered":"2024-03-06T07:27:14.991622+0000","last_unstale":"2024-03-06T07:27:15.988068+0000","last_undegraded":"2024-03-06T07:27:15.988068+0000","last_fullsized":"2024-03-06T07:27:15.988068+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:57.351882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.001779+0000","last_change":"2024-03-06T07:27:13.994045+0000","last_active":"2024-03-06T07:27:15.001779+0000","last_peered":"2024-03-06T07:27:15.001779+0000","last_clean":"2024-03-06T07:27:15.001779+0000","last_became_active":"2024-03-06T07:27:13.993753+0000","last_became_peered":"2024-03-06T07:27:13.993753+0000","last_unstale":"2024-03-06T07:27:15.001779+0000","last_undegraded":"2024-03-06T07:27:15.001779+0000","last_fullsized":"2024-03-06T07:27:15.001779+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:55.293700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006655+0000","last_change":"2024-03-06T07:27:15.003936+0000","last_active":"2024-03-06T07:27:15.006655+0000","last_peered":"2024-03-06T07:27:15.006655+0000","last_clean":"2024-03-06T07:27:15.006655+0000","last_became_active":"2024-03-06T07:27:15.002848+0000","last_became_peered":"2024-03-06T07:27:15.002848+0000","last_unstale":"2024-03-06T07:27:15.006655+0000","last_undegraded":"2024-03-06T07:27:15.006655+0000","last_fullsized":"2024-03-06T07:27:15.006655+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:59.098860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.999486+0000","last_change":"2024-03-06T07:27:13.993657+0000","last_active":"2024-03-06T07:27:14.999486+0000","last_peered":"2024-03-06T07:27:14.999486+0000","last_clean":"2024-03-06T07:27:14.999486+0000","last_became_active":"2024-03-06T07:27:13.993371+0000","last_became_peered":"2024-03-06T07:27:13.993371+0000","last_unstale":"2024-03-06T07:27:14.999486+0000","last_undegraded":"2024-03-06T07:27:14.999486+0000","last_fullsized":"2024-03-06T07:27:14.999486+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:49.516922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.998332+0000","last_change":"2024-03-06T07:27:14.995941+0000","last_active":"2024-03-06T07:27:14.998332+0000","last_peered":"2024-03-06T07:27:14.998332+0000","last_clean":"2024-03-06T07:27:14.998332+0000","last_became_active":"2024-03-06T07:27:14.995634+0000","last_became_peered":"2024-03-06T07:27:14.995634+0000","last_unstale":"2024-03-06T07:27:14.998332+0000","last_undegraded":"2024-03-06T07:27:14.998332+0000","last_fullsized":"2024-03-06T07:27:14.998332+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:46.411357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990447+0000","last_change":"2024-03-06T07:27:13.990461+0000","last_active":"2024-03-06T07:27:15.990447+0000","last_peered":"2024-03-06T07:27:15.990447+0000","last_clean":"2024-03-06T07:27:15.990447+0000","last_became_active":"2024-03-06T07:27:13.990177+0000","last_became_peered":"2024-03-06T07:27:13.990177+0000","last_unstale":"2024-03-06T07:27:15.990447+0000","last_undegraded":"2024-03-06T07:27:15.990447+0000","last_fullsized":"2024-03-06T07:27:15.990447+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:59.651616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007761+0000","last_change":"2024-03-06T07:27:14.999102+0000","last_active":"2024-03-06T07:27:15.007761+0000","last_peered":"2024-03-06T07:27:15.007761+0000","last_clean":"2024-03-06T07:27:15.007761+0000","last_became_active":"2024-03-06T07:27:14.998476+0000","last_became_peered":"2024-03-06T07:27:14.998476+0000","last_unstale":"2024-03-06T07:27:15.007761+0000","last_undegraded":"2024-03-06T07:27:15.007761+0000","last_fullsized":"2024-03-06T07:27:15.007761+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:27.696930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990638+0000","last_change":"2024-03-06T07:27:13.993648+0000","last_active":"2024-03-06T07:27:15.990638+0000","last_peered":"2024-03-06T07:27:15.990638+0000","last_clean":"2024-03-06T07:27:15.990638+0000","last_became_active":"2024-03-06T07:27:13.991583+0000","last_became_peered":"2024-03-06T07:27:13.991583+0000","last_unstale":"2024-03-06T07:27:15.990638+0000","last_undegraded":"2024-03-06T07:27:15.990638+0000","last_fullsized":"2024-03-06T07:27:15.990638+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:08.138307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003176+0000","last_change":"2024-03-06T07:27:14.992051+0000","last_active":"2024-03-06T07:27:15.003176+0000","last_peered":"2024-03-06T07:27:15.003176+0000","last_clean":"2024-03-06T07:27:15.003176+0000","last_became_active":"2024-03-06T07:27:14.991772+0000","last_became_peered":"2024-03-06T07:27:14.991772+0000","last_unstale":"2024-03-06T07:27:15.003176+0000","last_undegraded":"2024-03-06T07:27:15.003176+0000","last_fullsized":"2024-03-06T07:27:15.003176+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:38.766369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.988166+0000","last_change":"2024-03-06T07:27:13.991976+0000","last_active":"2024-03-06T07:27:14.988166+0000","last_peered":"2024-03-06T07:27:14.988166+0000","last_clean":"2024-03-06T07:27:14.988166+0000","last_became_active":"2024-03-06T07:27:13.991674+0000","last_became_peered":"2024-03-06T07:27:13.991674+0000","last_unstale":"2024-03-06T07:27:14.988166+0000","last_undegraded":"2024-03-06T07:27:14.988166+0000","last_fullsized":"2024-03-06T07:27:14.988166+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:39.106016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990302+0000","last_change":"2024-03-06T07:27:14.999744+0000","last_active":"2024-03-06T07:27:15.990302+0000","last_peered":"2024-03-06T07:27:15.990302+0000","last_clean":"2024-03-06T07:27:15.990302+0000","last_became_active":"2024-03-06T07:27:14.999478+0000","last_became_peered":"2024-03-06T07:27:14.999478+0000","last_unstale":"2024-03-06T07:27:15.990302+0000","last_undegraded":"2024-03-06T07:27:15.990302+0000","last_fullsized":"2024-03-06T07:27:15.990302+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:37.564891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.982354+0000","last_change":"2024-03-06T07:27:13.994282+0000","last_active":"2024-03-06T07:27:14.982354+0000","last_peered":"2024-03-06T07:27:14.982354+0000","last_clean":"2024-03-06T07:27:14.982354+0000","last_became_active":"2024-03-06T07:27:13.993998+0000","last_became_peered":"2024-03-06T07:27:13.993998+0000","last_unstale":"2024-03-06T07:27:14.982354+0000","last_undegraded":"2024-03-06T07:27:14.982354+0000","last_fullsized":"2024-03-06T07:27:14.982354+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:33.681814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990724+0000","last_change":"2024-03-06T07:27:14.991025+0000","last_active":"2024-03-06T07:27:15.990724+0000","last_peered":"2024-03-06T07:27:15.990724+0000","last_clean":"2024-03-06T07:27:15.990724+0000","last_became_active":"2024-03-06T07:27:14.990748+0000","last_became_peered":"2024-03-06T07:27:14.990748+0000","last_unstale":"2024-03-06T07:27:15.990724+0000","last_undegraded":"2024-03-06T07:27:15.990724+0000","last_fullsized":"2024-03-06T07:27:15.990724+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:22.264605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.981178+0000","last_change":"2024-03-06T07:27:13.996786+0000","last_active":"2024-03-06T07:27:14.981178+0000","last_peered":"2024-03-06T07:27:14.981178+0000","last_clean":"2024-03-06T07:27:14.981178+0000","last_became_active":"2024-03-06T07:27:13.996505+0000","last_became_peered":"2024-03-06T07:27:13.996505+0000","last_unstale":"2024-03-06T07:27:14.981178+0000","last_undegraded":"2024-03-06T07:27:14.981178+0000","last_fullsized":"2024-03-06T07:27:14.981178+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:21.705645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006542+0000","last_change":"2024-03-06T07:27:15.002887+0000","last_active":"2024-03-06T07:27:15.006542+0000","last_peered":"2024-03-06T07:27:15.006542+0000","last_clean":"2024-03-06T07:27:15.006542+0000","last_became_active":"2024-03-06T07:27:15.001744+0000","last_became_peered":"2024-03-06T07:27:15.001744+0000","last_unstale":"2024-03-06T07:27:15.006542+0000","last_undegraded":"2024-03-06T07:27:15.006542+0000","last_fullsized":"2024-03-06T07:27:15.006542+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:40.108009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990597+0000","last_change":"2024-03-06T07:27:13.993489+0000","last_active":"2024-03-06T07:27:15.990597+0000","last_peered":"2024-03-06T07:27:15.990597+0000","last_clean":"2024-03-06T07:27:15.990597+0000","last_became_active":"2024-03-06T07:27:13.990858+0000","last_became_peered":"2024-03-06T07:27:13.990858+0000","last_unstale":"2024-03-06T07:27:15.990597+0000","last_undegraded":"2024-03-06T07:27:15.990597+0000","last_fullsized":"2024-03-06T07:27:15.990597+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:23.983874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990752+0000","last_change":"2024-03-06T07:27:15.005424+0000","last_active":"2024-03-06T07:27:15.990752+0000","last_peered":"2024-03-06T07:27:15.990752+0000","last_clean":"2024-03-06T07:27:15.990752+0000","last_became_active":"2024-03-06T07:27:15.005134+0000","last_became_peered":"2024-03-06T07:27:15.005134+0000","last_unstale":"2024-03-06T07:27:15.990752+0000","last_undegraded":"2024-03-06T07:27:15.990752+0000","last_fullsized":"2024-03-06T07:27:15.990752+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:29:49.838205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.996569+0000","last_change":"2024-03-06T07:27:13.995644+0000","last_active":"2024-03-06T07:27:14.996569+0000","last_peered":"2024-03-06T07:27:14.996569+0000","last_clean":"2024-03-06T07:27:14.996569+0000","last_became_active":"2024-03-06T07:27:13.995368+0000","last_became_peered":"2024-03-06T07:27:13.995368+0000","last_unstale":"2024-03-06T07:27:14.996569+0000","last_undegraded":"2024-03-06T07:27:14.996569+0000","last_fullsized":"2024-03-06T07:27:14.996569+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:26.706259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990623+0000","last_change":"2024-03-06T07:27:14.997011+0000","last_active":"2024-03-06T07:27:15.990623+0000","last_peered":"2024-03-06T07:27:15.990623+0000","last_clean":"2024-03-06T07:27:15.990623+0000","last_became_active":"2024-03-06T07:27:14.996708+0000","last_became_peered":"2024-03-06T07:27:14.996708+0000","last_unstale":"2024-03-06T07:27:15.990623+0000","last_undegraded":"2024-03-06T07:27:15.990623+0000","last_fullsized":"2024-03-06T07:27:15.990623+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:42.996423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.012330+0000","last_change":"2024-03-06T07:27:13.995213+0000","last_active":"2024-03-06T07:27:15.012330+0000","last_peered":"2024-03-06T07:27:15.012330+0000","last_clean":"2024-03-06T07:27:15.012330+0000","last_became_active":"2024-03-06T07:27:13.994753+0000","last_became_peered":"2024-03-06T07:27:13.994753+0000","last_unstale":"2024-03-06T07:27:15.012330+0000","last_undegraded":"2024-03-06T07:27:15.012330+0000","last_fullsized":"2024-03-06T07:27:15.012330+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:55.759680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990590+0000","last_change":"2024-03-06T07:27:14.990689+0000","last_active":"2024-03-06T07:27:15.990590+0000","last_peered":"2024-03-06T07:27:15.990590+0000","last_clean":"2024-03-06T07:27:15.990590+0000","last_became_active":"2024-03-06T07:27:14.990384+0000","last_became_peered":"2024-03-06T07:27:14.990384+0000","last_unstale":"2024-03-06T07:27:15.990590+0000","last_undegraded":"2024-03-06T07:27:15.990590+0000","last_fullsized":"2024-03-06T07:27:15.990590+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:25.925986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987961+0000","last_change":"2024-03-06T07:27:13.985607+0000","last_active":"2024-03-06T07:27:15.987961+0000","last_peered":"2024-03-06T07:27:15.987961+0000","last_clean":"2024-03-06T07:27:15.987961+0000","last_became_active":"2024-03-06T07:27:13.985298+0000","last_became_peered":"2024-03-06T07:27:13.985298+0000","last_unstale":"2024-03-06T07:27:15.987961+0000","last_undegraded":"2024-03-06T07:27:15.987961+0000","last_fullsized":"2024-03-06T07:27:15.987961+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:04.179482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990640+0000","last_change":"2024-03-06T07:27:14.998801+0000","last_active":"2024-03-06T07:27:15.990640+0000","last_peered":"2024-03-06T07:27:15.990640+0000","last_clean":"2024-03-06T07:27:15.990640+0000","last_became_active":"2024-03-06T07:27:14.997099+0000","last_became_peered":"2024-03-06T07:27:14.997099+0000","last_unstale":"2024-03-06T07:27:15.990640+0000","last_undegraded":"2024-03-06T07:27:15.990640+0000","last_fullsized":"2024-03-06T07:27:15.990640+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:13.372370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.996705+0000","last_change":"2024-03-06T07:27:13.990670+0000","last_active":"2024-03-06T07:27:14.996705+0000","last_peered":"2024-03-06T07:27:14.996705+0000","last_clean":"2024-03-06T07:27:14.996705+0000","last_became_active":"2024-03-06T07:27:13.990368+0000","last_became_peered":"2024-03-06T07:27:13.990368+0000","last_unstale":"2024-03-06T07:27:14.996705+0000","last_undegraded":"2024-03-06T07:27:14.996705+0000","last_fullsized":"2024-03-06T07:27:14.996705+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:12.523568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990383+0000","last_change":"2024-03-06T07:27:15.003310+0000","last_active":"2024-03-06T07:27:15.990383+0000","last_peered":"2024-03-06T07:27:15.990383+0000","last_clean":"2024-03-06T07:27:15.990383+0000","last_became_active":"2024-03-06T07:27:15.002619+0000","last_became_peered":"2024-03-06T07:27:15.002619+0000","last_unstale":"2024-03-06T07:27:15.990383+0000","last_undegraded":"2024-03-06T07:27:15.990383+0000","last_fullsized":"2024-03-06T07:27:15.990383+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:38.766369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":20,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.005387+0000","last_change":"2024-03-06T07:27:13.991882+0000","last_active":"2024-03-06T07:27:15.005387+0000","last_peered":"2024-03-06T07:27:15.005387+0000","last_clean":"2024-03-06T07:27:15.005387+0000","last_became_active":"2024-03-06T07:27:13.991620+0000","last_became_peered":"2024-03-06T07:27:13.991620+0000","last_unstale":"2024-03-06T07:27:15.005387+0000","last_undegraded":"2024-03-06T07:27:15.005387+0000","last_fullsized":"2024-03-06T07:27:15.005387+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:37.564172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.001000+0000","last_change":"2024-03-06T07:27:14.996573+0000","last_active":"2024-03-06T07:27:15.001000+0000","last_peered":"2024-03-06T07:27:15.001000+0000","last_clean":"2024-03-06T07:27:15.001000+0000","last_became_active":"2024-03-06T07:27:14.994777+0000","last_became_peered":"2024-03-06T07:27:14.994777+0000","last_unstale":"2024-03-06T07:27:15.001000+0000","last_undegraded":"2024-03-06T07:27:15.001000+0000","last_fullsized":"2024-03-06T07:27:15.001000+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:33.439683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003849+0000","last_change":"2024-03-06T07:27:13.989262+0000","last_active":"2024-03-06T07:27:15.003849+0000","last_peered":"2024-03-06T07:27:15.003849+0000","last_clean":"2024-03-06T07:27:15.003849+0000","last_became_active":"2024-03-06T07:27:13.988626+0000","last_became_peered":"2024-03-06T07:27:13.988626+0000","last_unstale":"2024-03-06T07:27:15.003849+0000","last_undegraded":"2024-03-06T07:27:15.003849+0000","last_fullsized":"2024-03-06T07:27:15.003849+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:45.852417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987981+0000","last_change":"2024-03-06T07:27:15.002326+0000","last_active":"2024-03-06T07:27:15.987981+0000","last_peered":"2024-03-06T07:27:15.987981+0000","last_clean":"2024-03-06T07:27:15.987981+0000","last_became_active":"2024-03-06T07:27:15.001552+0000","last_became_peered":"2024-03-06T07:27:15.001552+0000","last_unstale":"2024-03-06T07:27:15.987981+0000","last_undegraded":"2024-03-06T07:27:15.987981+0000","last_fullsized":"2024-03-06T07:27:15.987981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:40:27.696930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.007322+0000","last_change":"2024-03-06T07:27:13.997890+0000","last_active":"2024-03-06T07:27:15.007322+0000","last_peered":"2024-03-06T07:27:15.007322+0000","last_clean":"2024-03-06T07:27:15.007322+0000","last_became_active":"2024-03-06T07:27:13.997543+0000","last_became_peered":"2024-03-06T07:27:13.997543+0000","last_unstale":"2024-03-06T07:27:15.007322+0000","last_undegraded":"2024-03-06T07:27:15.007322+0000","last_fullsized":"2024-03-06T07:27:15.007322+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:10.739405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006396+0000","last_change":"2024-03-06T07:27:15.001305+0000","last_active":"2024-03-06T07:27:15.006396+0000","last_peered":"2024-03-06T07:27:15.006396+0000","last_clean":"2024-03-06T07:27:15.006396+0000","last_became_active":"2024-03-06T07:27:15.001041+0000","last_became_peered":"2024-03-06T07:27:15.001041+0000","last_unstale":"2024-03-06T07:27:15.006396+0000","last_undegraded":"2024-03-06T07:27:15.006396+0000","last_fullsized":"2024-03-06T07:27:15.006396+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:12.434709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.005120+0000","last_change":"2024-03-06T07:27:13.992257+0000","last_active":"2024-03-06T07:27:15.005120+0000","last_peered":"2024-03-06T07:27:15.005120+0000","last_clean":"2024-03-06T07:27:15.005120+0000","last_became_active":"2024-03-06T07:27:13.991972+0000","last_became_peered":"2024-03-06T07:27:13.991972+0000","last_unstale":"2024-03-06T07:27:15.005120+0000","last_undegraded":"2024-03-06T07:27:15.005120+0000","last_fullsized":"2024-03-06T07:27:15.005120+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:36.562898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.005931+0000","last_change":"2024-03-06T07:27:14.999417+0000","last_active":"2024-03-06T07:27:15.005931+0000","last_peered":"2024-03-06T07:27:15.005931+0000","last_clean":"2024-03-06T07:27:15.005931+0000","last_became_active":"2024-03-06T07:27:14.999111+0000","last_became_peered":"2024-03-06T07:27:14.999111+0000","last_unstale":"2024-03-06T07:27:15.005931+0000","last_undegraded":"2024-03-06T07:27:15.005931+0000","last_fullsized":"2024-03-06T07:27:15.005931+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:01.873864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990489+0000","last_change":"2024-03-06T07:27:13.992361+0000","last_active":"2024-03-06T07:27:15.990489+0000","last_peered":"2024-03-06T07:27:15.990489+0000","last_clean":"2024-03-06T07:27:15.990489+0000","last_became_active":"2024-03-06T07:27:13.992004+0000","last_became_peered":"2024-03-06T07:27:13.992004+0000","last_unstale":"2024-03-06T07:27:15.990489+0000","last_undegraded":"2024-03-06T07:27:15.990489+0000","last_fullsized":"2024-03-06T07:27:15.990489+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:18.186945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.008698+0000","last_change":"2024-03-06T07:27:15.007358+0000","last_active":"2024-03-06T07:27:15.008698+0000","last_peered":"2024-03-06T07:27:15.008698+0000","last_clean":"2024-03-06T07:27:15.008698+0000","last_became_active":"2024-03-06T07:27:15.007096+0000","last_became_peered":"2024-03-06T07:27:15.007096+0000","last_unstale":"2024-03-06T07:27:15.008698+0000","last_undegraded":"2024-03-06T07:27:15.008698+0000","last_fullsized":"2024-03-06T07:27:15.008698+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:58.566040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.988051+0000","last_change":"2024-03-06T07:27:13.989196+0000","last_active":"2024-03-06T07:27:15.988051+0000","last_peered":"2024-03-06T07:27:15.988051+0000","last_clean":"2024-03-06T07:27:15.988051+0000","last_became_active":"2024-03-06T07:27:13.988905+0000","last_became_peered":"2024-03-06T07:27:13.988905+0000","last_unstale":"2024-03-06T07:27:15.988051+0000","last_undegraded":"2024-03-06T07:27:15.988051+0000","last_fullsized":"2024-03-06T07:27:15.988051+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:39.629234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.002588+0000","last_change":"2024-03-06T07:27:14.996929+0000","last_active":"2024-03-06T07:27:15.002588+0000","last_peered":"2024-03-06T07:27:15.002588+0000","last_clean":"2024-03-06T07:27:15.002588+0000","last_became_active":"2024-03-06T07:27:14.996191+0000","last_became_peered":"2024-03-06T07:27:14.996191+0000","last_unstale":"2024-03-06T07:27:15.002588+0000","last_undegraded":"2024-03-06T07:27:15.002588+0000","last_fullsized":"2024-03-06T07:27:15.002588+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:59.786571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.981347+0000","last_change":"2024-03-06T07:27:13.994067+0000","last_active":"2024-03-06T07:27:14.981347+0000","last_peered":"2024-03-06T07:27:14.981347+0000","last_clean":"2024-03-06T07:27:14.981347+0000","last_became_active":"2024-03-06T07:27:13.993531+0000","last_became_peered":"2024-03-06T07:27:13.993531+0000","last_unstale":"2024-03-06T07:27:14.981347+0000","last_undegraded":"2024-03-06T07:27:14.981347+0000","last_fullsized":"2024-03-06T07:27:14.981347+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:27.526863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006585+0000","last_change":"2024-03-06T07:27:15.002276+0000","last_active":"2024-03-06T07:27:15.006585+0000","last_peered":"2024-03-06T07:27:15.006585+0000","last_clean":"2024-03-06T07:27:15.006585+0000","last_became_active":"2024-03-06T07:27:15.002008+0000","last_became_peered":"2024-03-06T07:27:15.002008+0000","last_unstale":"2024-03-06T07:27:15.006585+0000","last_undegraded":"2024-03-06T07:27:15.006585+0000","last_fullsized":"2024-03-06T07:27:15.006585+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:06.357185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.987720+0000","last_change":"2024-03-06T07:27:13.988664+0000","last_active":"2024-03-06T07:27:15.987720+0000","last_peered":"2024-03-06T07:27:15.987720+0000","last_clean":"2024-03-06T07:27:15.987720+0000","last_became_active":"2024-03-06T07:27:13.988373+0000","last_became_peered":"2024-03-06T07:27:13.988373+0000","last_unstale":"2024-03-06T07:27:15.987720+0000","last_undegraded":"2024-03-06T07:27:15.987720+0000","last_fullsized":"2024-03-06T07:27:15.987720+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:51.110735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990435+0000","last_change":"2024-03-06T07:27:15.003040+0000","last_active":"2024-03-06T07:27:15.990435+0000","last_peered":"2024-03-06T07:27:15.990435+0000","last_clean":"2024-03-06T07:27:15.990435+0000","last_became_active":"2024-03-06T07:27:15.002349+0000","last_became_peered":"2024-03-06T07:27:15.002349+0000","last_unstale":"2024-03-06T07:27:15.990435+0000","last_undegraded":"2024-03-06T07:27:15.990435+0000","last_fullsized":"2024-03-06T07:27:15.990435+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:07.794199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.981761+0000","last_change":"2024-03-06T07:27:13.990768+0000","last_active":"2024-03-06T07:27:14.981761+0000","last_peered":"2024-03-06T07:27:14.981761+0000","last_clean":"2024-03-06T07:27:14.981761+0000","last_became_active":"2024-03-06T07:27:13.990303+0000","last_became_peered":"2024-03-06T07:27:13.990303+0000","last_unstale":"2024-03-06T07:27:14.981761+0000","last_undegraded":"2024-03-06T07:27:14.981761+0000","last_fullsized":"2024-03-06T07:27:14.981761+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:04.546779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006392+0000","last_change":"2024-03-06T07:27:15.002408+0000","last_active":"2024-03-06T07:27:15.006392+0000","last_peered":"2024-03-06T07:27:15.006392+0000","last_clean":"2024-03-06T07:27:15.006392+0000","last_became_active":"2024-03-06T07:27:15.002122+0000","last_became_peered":"2024-03-06T07:27:15.002122+0000","last_unstale":"2024-03-06T07:27:15.006392+0000","last_undegraded":"2024-03-06T07:27:15.006392+0000","last_fullsized":"2024-03-06T07:27:15.006392+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:22.264605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990698+0000","last_change":"2024-03-06T07:27:13.992258+0000","last_active":"2024-03-06T07:27:15.990698+0000","last_peered":"2024-03-06T07:27:15.990698+0000","last_clean":"2024-03-06T07:27:15.990698+0000","last_became_active":"2024-03-06T07:27:13.991991+0000","last_became_peered":"2024-03-06T07:27:13.991991+0000","last_unstale":"2024-03-06T07:27:15.990698+0000","last_undegraded":"2024-03-06T07:27:15.990698+0000","last_fullsized":"2024-03-06T07:27:15.990698+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:20.081724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003095+0000","last_change":"2024-03-06T07:27:15.000980+0000","last_active":"2024-03-06T07:27:15.003095+0000","last_peered":"2024-03-06T07:27:15.003095+0000","last_clean":"2024-03-06T07:27:15.003095+0000","last_became_active":"2024-03-06T07:27:15.000702+0000","last_became_peered":"2024-03-06T07:27:15.000702+0000","last_unstale":"2024-03-06T07:27:15.003095+0000","last_undegraded":"2024-03-06T07:27:15.003095+0000","last_fullsized":"2024-03-06T07:27:15.003095+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:17.563603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"107'1","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.979590+0000","last_change":"2024-03-06T07:27:13.987019+0000","last_active":"2024-03-06T07:27:15.979590+0000","last_peered":"2024-03-06T07:27:15.979590+0000","last_clean":"2024-03-06T07:27:15.979590+0000","last_became_active":"2024-03-06T07:27:13.986708+0000","last_became_peered":"2024-03-06T07:27:13.986708+0000","last_unstale":"2024-03-06T07:27:15.979590+0000","last_undegraded":"2024-03-06T07:27:15.979590+0000","last_fullsized":"2024-03-06T07:27:15.979590+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:43.503056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.002737+0000","last_change":"2024-03-06T07:27:14.999202+0000","last_active":"2024-03-06T07:27:15.002737+0000","last_peered":"2024-03-06T07:27:15.002737+0000","last_clean":"2024-03-06T07:27:15.002737+0000","last_became_active":"2024-03-06T07:27:14.998732+0000","last_became_peered":"2024-03-06T07:27:14.998732+0000","last_unstale":"2024-03-06T07:27:15.002737+0000","last_undegraded":"2024-03-06T07:27:15.002737+0000","last_fullsized":"2024-03-06T07:27:15.002737+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:16:54.707369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.988828+0000","last_change":"2024-03-06T07:27:13.990648+0000","last_active":"2024-03-06T07:27:14.988828+0000","last_peered":"2024-03-06T07:27:14.988828+0000","last_clean":"2024-03-06T07:27:14.988828+0000","last_became_active":"2024-03-06T07:27:13.990278+0000","last_became_peered":"2024-03-06T07:27:13.990278+0000","last_unstale":"2024-03-06T07:27:14.988828+0000","last_undegraded":"2024-03-06T07:27:14.988828+0000","last_fullsized":"2024-03-06T07:27:14.988828+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:14.168079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.003361+0000","last_change":"2024-03-06T07:27:14.999908+0000","last_active":"2024-03-06T07:27:15.003361+0000","last_peered":"2024-03-06T07:27:15.003361+0000","last_clean":"2024-03-06T07:27:15.003361+0000","last_became_active":"2024-03-06T07:27:14.999282+0000","last_became_peered":"2024-03-06T07:27:14.999282+0000","last_unstale":"2024-03-06T07:27:15.003361+0000","last_undegraded":"2024-03-06T07:27:15.003361+0000","last_fullsized":"2024-03-06T07:27:15.003361+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:15.510457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990783+0000","last_change":"2024-03-06T07:27:13.993319+0000","last_active":"2024-03-06T07:27:15.990783+0000","last_peered":"2024-03-06T07:27:15.990783+0000","last_clean":"2024-03-06T07:27:15.990783+0000","last_became_active":"2024-03-06T07:27:13.993046+0000","last_became_peered":"2024-03-06T07:27:13.993046+0000","last_unstale":"2024-03-06T07:27:15.990783+0000","last_undegraded":"2024-03-06T07:27:15.990783+0000","last_fullsized":"2024-03-06T07:27:15.990783+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:13.358651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006726+0000","last_change":"2024-03-06T07:27:14.992939+0000","last_active":"2024-03-06T07:27:15.006726+0000","last_peered":"2024-03-06T07:27:15.006726+0000","last_clean":"2024-03-06T07:27:15.006726+0000","last_became_active":"2024-03-06T07:27:14.992622+0000","last_became_peered":"2024-03-06T07:27:14.992622+0000","last_unstale":"2024-03-06T07:27:15.006726+0000","last_undegraded":"2024-03-06T07:27:15.006726+0000","last_fullsized":"2024-03-06T07:27:15.006726+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:19.546764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990541+0000","last_change":"2024-03-06T07:27:13.986130+0000","last_active":"2024-03-06T07:27:15.990541+0000","last_peered":"2024-03-06T07:27:15.990541+0000","last_clean":"2024-03-06T07:27:15.990541+0000","last_became_active":"2024-03-06T07:27:13.985828+0000","last_became_peered":"2024-03-06T07:27:13.985828+0000","last_unstale":"2024-03-06T07:27:15.990541+0000","last_undegraded":"2024-03-06T07:27:15.990541+0000","last_fullsized":"2024-03-06T07:27:15.990541+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:53.529526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.008902+0000","last_change":"2024-03-06T07:27:15.005319+0000","last_active":"2024-03-06T07:27:15.008902+0000","last_peered":"2024-03-06T07:27:15.008902+0000","last_clean":"2024-03-06T07:27:15.008902+0000","last_became_active":"2024-03-06T07:27:15.005010+0000","last_became_peered":"2024-03-06T07:27:15.005010+0000","last_unstale":"2024-03-06T07:27:15.008902+0000","last_undegraded":"2024-03-06T07:27:15.008902+0000","last_fullsized":"2024-03-06T07:27:15.008902+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:51.665162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.987508+0000","last_change":"2024-03-06T07:27:13.986231+0000","last_active":"2024-03-06T07:27:14.987508+0000","last_peered":"2024-03-06T07:27:14.987508+0000","last_clean":"2024-03-06T07:27:14.987508+0000","last_became_active":"2024-03-06T07:27:13.985948+0000","last_became_peered":"2024-03-06T07:27:13.985948+0000","last_unstale":"2024-03-06T07:27:14.987508+0000","last_undegraded":"2024-03-06T07:27:14.987508+0000","last_fullsized":"2024-03-06T07:27:14.987508+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:21.262613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.006231+0000","last_change":"2024-03-06T07:27:15.000818+0000","last_active":"2024-03-06T07:27:15.006231+0000","last_peered":"2024-03-06T07:27:15.006231+0000","last_clean":"2024-03-06T07:27:15.006231+0000","last_became_active":"2024-03-06T07:27:14.999909+0000","last_became_peered":"2024-03-06T07:27:14.999909+0000","last_unstale":"2024-03-06T07:27:15.006231+0000","last_undegraded":"2024-03-06T07:27:15.006231+0000","last_fullsized":"2024-03-06T07:27:15.006231+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:57.351882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.981360+0000","last_change":"2024-03-06T07:27:13.996405+0000","last_active":"2024-03-06T07:27:14.981360+0000","last_peered":"2024-03-06T07:27:14.981360+0000","last_clean":"2024-03-06T07:27:14.981360+0000","last_became_active":"2024-03-06T07:27:13.996115+0000","last_became_peered":"2024-03-06T07:27:13.996115+0000","last_unstale":"2024-03-06T07:27:14.981360+0000","last_undegraded":"2024-03-06T07:27:14.981360+0000","last_fullsized":"2024-03-06T07:27:14.981360+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:18.398153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.995747+0000","last_change":"2024-03-06T07:27:14.991513+0000","last_active":"2024-03-06T07:27:14.995747+0000","last_peered":"2024-03-06T07:27:14.995747+0000","last_clean":"2024-03-06T07:27:14.995747+0000","last_became_active":"2024-03-06T07:27:14.991247+0000","last_became_peered":"2024-03-06T07:27:14.991247+0000","last_unstale":"2024-03-06T07:27:14.995747+0000","last_undegraded":"2024-03-06T07:27:14.995747+0000","last_fullsized":"2024-03-06T07:27:14.995747+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:58.244707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.981911+0000","last_change":"2024-03-06T07:27:13.991042+0000","last_active":"2024-03-06T07:27:14.981911+0000","last_peered":"2024-03-06T07:27:14.981911+0000","last_clean":"2024-03-06T07:27:14.981911+0000","last_became_active":"2024-03-06T07:27:13.990421+0000","last_became_peered":"2024-03-06T07:27:13.990421+0000","last_unstale":"2024-03-06T07:27:14.981911+0000","last_undegraded":"2024-03-06T07:27:14.981911+0000","last_fullsized":"2024-03-06T07:27:14.981911+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:00.043646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990434+0000","last_change":"2024-03-06T07:27:14.998464+0000","last_active":"2024-03-06T07:27:15.990434+0000","last_peered":"2024-03-06T07:27:15.990434+0000","last_clean":"2024-03-06T07:27:15.990434+0000","last_became_active":"2024-03-06T07:27:14.997264+0000","last_became_peered":"2024-03-06T07:27:14.997264+0000","last_unstale":"2024-03-06T07:27:15.990434+0000","last_undegraded":"2024-03-06T07:27:15.990434+0000","last_fullsized":"2024-03-06T07:27:15.990434+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:13.968594+0000","last_clean_scrub_stamp":"2024-03-06T07:27:13.968594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:59.786571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:27:15.990402+0000","last_change":"2024-03-06T07:27:13.988793+0000","last_active":"2024-03-06T07:27:15.990402+0000","last_peered":"2024-03-06T07:27:15.990402+0000","last_clean":"2024-03-06T07:27:15.990402+0000","last_became_active":"2024-03-06T07:27:13.988528+0000","last_became_peered":"2024-03-06T07:27:13.988528+0000","last_unstale":"2024-03-06T07:27:15.990402+0000","last_undegraded":"2024-03-06T07:27:15.990402+0000","last_fullsized":"2024-03-06T07:27:15.990402+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:12.966601+0000","last_clean_scrub_stamp":"2024-03-06T07:27:12.966601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:51.110735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":326,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:14.990838+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:27:14.990838+0000","last_peered":"2024-03-06T07:27:14.990838+0000","last_clean":"2024-03-06T07:27:14.990838+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:27:14.990838+0000","last_undegraded":"2024-03-06T07:27:14.990838+0000","last_fullsized":"2024-03-06T07:27:14.990838+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37236,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334604,"statfs":{"total":96636764160,"available":96598634496,"internally_reserved":0,"allocated":1933312,"data_stored":1165421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574919,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1335296,"data_stored":574507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40828,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331012,"statfs":{"total":96636764160,"available":96594956288,"internally_reserved":0,"allocated":1351680,"data_stored":581341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1331200,"data_stored":574437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574918,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40824,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331016,"statfs":{"total":96636764160,"available":96594960384,"internally_reserved":0,"allocated":1347584,"data_stored":576119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36644,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335196,"statfs":{"total":96636764160,"available":96599240704,"internally_reserved":0,"allocated":1331200,"data_stored":574437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40812,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331028,"statfs":{"total":96636764160,"available":96594972672,"internally_reserved":0,"allocated":1339392,"data_stored":574561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41404,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330436,"statfs":{"total":96636764160,"available":96594366464,"internally_reserved":0,"allocated":1945600,"data_stored":1166529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:18.461 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:18.461 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:27:18.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.573+0000 7f04c37eb640 1 Processor -- start 2024-03-06T07:27:18.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.573+0000 7f04c37eb640 1 -- start start 2024-03-06T07:27:18.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.574+0000 7f04c37eb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc15bdd0 0x7f04bc0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.574+0000 7f04c37eb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04bc0d0240 con 0x7f04bc0ce300 2024-03-06T07:27:18.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.574+0000 7f04c37eb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04bc0cf210 con 0x7f04bc0b15c0 2024-03-06T07:27:18.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.574+0000 7f04c37eb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0cf4f0 con 0x7f04bc15bdd0 2024-03-06T07:27:18.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.574+0000 7f04c0d86640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f04bc0ce300 0x7f04bc0ce760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60832/0 (socket says 172.21.15.59:60832) 2024-03-06T07:27:18.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.574+0000 7f04c0d86640 1 -- 172.21.15.59:0/4151355870 learned_addr learned my addr 172.21.15.59:0/4151355870 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:18.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.574+0000 7f04c1587640 1 --2- 172.21.15.59:0/4151355870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc15bdd0 0x7f04bc0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.574+0000 7f04b3fff640 1 -- 172.21.15.59:0/4151355870 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672764622 0 0) 0x7f04bc0d0240 con 0x7f04bc0ce300 2024-03-06T07:27:18.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.575+0000 7f04b3fff640 1 -- 172.21.15.59:0/4151355870 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a8003520 con 0x7f04bc0ce300 2024-03-06T07:27:18.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.575+0000 7f04b3fff640 1 -- 172.21.15.59:0/4151355870 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304814591 0 0) 0x7f04bc0cf210 con 0x7f04bc0b15c0 2024-03-06T07:27:18.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.575+0000 7f04c1587640 1 -- 172.21.15.59:0/4151355870 >> v1:172.21.15.164:6789/0 conn(0x7f04bc0b15c0 legacy=0x7f04bc0b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.575+0000 7f04c1587640 1 -- 172.21.15.59:0/4151355870 >> v1:172.21.15.59:6789/0 conn(0x7f04bc0ce300 legacy=0x7f04bc0ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.575+0000 7f04c1587640 1 -- 172.21.15.59:0/4151355870 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc0d04e0 con 0x7f04bc15bdd0 2024-03-06T07:27:18.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.575+0000 7f04c1587640 1 --2- 172.21.15.59:0/4151355870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc15bdd0 0x7f04bc0b41b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f04b40097d0 tx=0x7f04b402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4022d6925e0a12f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.575+0000 7f04b3fff640 1 -- 172.21.15.59:0/4151355870 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4035020 con 0x7f04bc15bdd0 2024-03-06T07:27:18.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.576+0000 7f04b3fff640 1 -- 172.21.15.59:0/4151355870 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b4031890 con 0x7f04bc15bdd0 2024-03-06T07:27:18.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.576+0000 7f04b3fff640 1 -- 172.21.15.59:0/4151355870 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4031c40 con 0x7f04bc15bdd0 2024-03-06T07:27:18.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.576+0000 7f04c37eb640 1 -- 172.21.15.59:0/4151355870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc15bdd0 msgr2=0x7f04bc0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.576+0000 7f04c37eb640 1 --2- 172.21.15.59:0/4151355870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc15bdd0 0x7f04bc0b41b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f04b40097d0 tx=0x7f04b402b930 comp rx=0 tx=0).stop 2024-03-06T07:27:18.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.576+0000 7f04c37eb640 1 -- 172.21.15.59:0/4151355870 shutdown_connections 2024-03-06T07:27:18.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.576+0000 7f04c37eb640 1 --2- 172.21.15.59:0/4151355870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc15bdd0 0x7f04bc0b41b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.576+0000 7f04c37eb640 1 -- 172.21.15.59:0/4151355870 >> 172.21.15.59:0/4151355870 conn(0x7f04bc0b3690 msgr2=0x7f04bc0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.576+0000 7f04c37eb640 1 -- 172.21.15.59:0/4151355870 shutdown_connections 2024-03-06T07:27:18.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.576+0000 7f04c37eb640 1 -- 172.21.15.59:0/4151355870 wait complete. 2024-03-06T07:27:18.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.577+0000 7f04c37eb640 1 Processor -- start 2024-03-06T07:27:18.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.577+0000 7f04c37eb640 1 -- start start 2024-03-06T07:27:18.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c37eb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04bc0b15c0 0x7f04bc0c5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c37eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04bc0ce300 0x7f04bc151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c37eb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc15bdd0 0x7f04bc15e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c37eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0cee70 con 0x7f04bc0ce300 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c37eb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0d0850 con 0x7f04bc0b15c0 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c37eb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0ceff0 con 0x7f04bc15bdd0 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c0d86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04bc0ce300 0x7f04bc151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c0d86640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04bc0ce300 0x7f04bc151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37340/0 (socket says 172.21.15.59:37340) 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c0d86640 1 -- 172.21.15.59:0/2976270458 learned_addr learned my addr 172.21.15.59:0/2976270458 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c1d88640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc15bdd0 0x7f04bc15e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c1587640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04bc0b15c0 0x7f04bc0c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c0d86640 1 -- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc15bdd0 msgr2=0x7f04bc15e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c0d86640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc15bdd0 0x7f04bc15e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c0d86640 1 -- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04bc0b15c0 msgr2=0x7f04bc0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c0d86640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04bc0b15c0 0x7f04bc0c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c0d86640 1 -- 172.21.15.59:0/2976270458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc151590 con 0x7f04bc0ce300 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.578+0000 7f04c1587640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04bc0b15c0 0x7f04bc0c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:18.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.579+0000 7f04c0d86640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04bc0ce300 0x7f04bc151050 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c390 tx=0x7f04ac00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.579+0000 7f04b1ffb640 1 -- 172.21.15.59:0/2976270458 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac017070 con 0x7f04bc0ce300 2024-03-06T07:27:18.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.579+0000 7f04b1ffb640 1 -- 172.21.15.59:0/2976270458 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac005ce0 con 0x7f04bc0ce300 2024-03-06T07:27:18.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.579+0000 7f04b1ffb640 1 -- 172.21.15.59:0/2976270458 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac0072f0 con 0x7f04bc0ce300 2024-03-06T07:27:18.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.579+0000 7f04c37eb640 1 -- 172.21.15.59:0/2976270458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04bc15eda0 con 0x7f04bc0ce300 2024-03-06T07:27:18.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.579+0000 7f04c37eb640 1 -- 172.21.15.59:0/2976270458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04bc0b4c70 con 0x7f04bc0ce300 2024-03-06T07:27:18.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.580+0000 7f04c37eb640 1 -- 172.21.15.59:0/2976270458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04bc0b4780 con 0x7f04bc0ce300 2024-03-06T07:27:18.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.580+0000 7f04b1ffb640 1 -- 172.21.15.59:0/2976270458 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04ac01a070 con 0x7f04bc0ce300 2024-03-06T07:27:18.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.581+0000 7f04b1ffb640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a006b4d0 0x7f04a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.581+0000 7f04b1ffb640 1 -- 172.21.15.59:0/2976270458 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04ac089560 con 0x7f04bc0ce300 2024-03-06T07:27:18.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.581+0000 7f04c1587640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a006b4d0 0x7f04a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.581+0000 7f04c1587640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a006b4d0 0x7f04a006d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f04b4036000 tx=0x7f04b4008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.585+0000 7f04b1ffb640 1 -- 172.21.15.59:0/2976270458 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04bc151590 con 0x7f04bc0ce300 2024-03-06T07:27:18.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.834+0000 7f04c37eb640 1 -- 172.21.15.59:0/2976270458 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f04bc09fcf0 con 0x7f04bc0ce300 2024-03-06T07:27:18.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.835+0000 7f04b1ffb640 1 -- 172.21.15.59:0/2976270458 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f04ac057da0 con 0x7f04bc0ce300 2024-03-06T07:27:18.837 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-06T07:27:18.837 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA== 2024-03-06T07:27:18.838 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-03-06T07:27:18.838 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-03-06T07:27:18.838 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:18.838 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 -- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a006b4d0 msgr2=0x7f04a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a006b4d0 0x7f04a006d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f04b4036000 tx=0x7f04b4008030 comp rx=0 tx=0).stop 2024-03-06T07:27:18.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 -- 172.21.15.59:0/2976270458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04bc0ce300 msgr2=0x7f04bc151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04bc0ce300 0x7f04bc151050 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c390 tx=0x7f04ac00c850 comp rx=0 tx=0).stop 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 -- 172.21.15.59:0/2976270458 shutdown_connections 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04a006b4d0 0x7f04a006d990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04bc15bdd0 0x7f04bc15e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04bc0ce300 0x7f04bc151050 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 --2- 172.21.15.59:0/2976270458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04bc0b15c0 0x7f04bc0c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 -- 172.21.15.59:0/2976270458 >> 172.21.15.59:0/2976270458 conn(0x7f04bc0b3690 msgr2=0x7f04bc0cc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 -- 172.21.15.59:0/2976270458 shutdown_connections 2024-03-06T07:27:18.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.838+0000 7f04c37eb640 1 -- 172.21.15.59:0/2976270458 wait complete. 2024-03-06T07:27:18.851 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:18.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.951+0000 7f2c33c2a640 1 Processor -- start 2024-03-06T07:27:18.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.951+0000 7f2c33c2a640 1 -- start start 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c33c2a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2c2c1621c0 0x7f2c2c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c33c2a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2c2c0d4920 0x7f2c2c0d26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c33c2a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2c0ceac0 con 0x7f2c2c0d2bf0 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c33c2a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c2c0cf690 con 0x7f2c2c1621c0 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c33c2a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c2c0ced60 con 0x7f2c2c0d4920 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c319c6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2c2c1621c0 0x7f2c2c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c321c7640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2c2c0d2bf0 0x7f2c2c15dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60848/0 (socket says 172.21.15.59:60848) 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c321c7640 1 -- 172.21.15.59:0/1799400932 learned_addr learned my addr 172.21.15.59:0/1799400932 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c311c5640 1 --2- 172.21.15.59:0/1799400932 >> v2:172.21.15.164:3300/0 conn(0x7f2c2c0d4920 0x7f2c2c0d26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c319c6640 1 -- 172.21.15.59:0/1799400932 >> v2:172.21.15.164:3300/0 conn(0x7f2c2c0d4920 msgr2=0x7f2c2c0d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:18.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c319c6640 1 --2- 172.21.15.59:0/1799400932 >> v2:172.21.15.164:3300/0 conn(0x7f2c2c0d4920 0x7f2c2c0d26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c319c6640 1 -- 172.21.15.59:0/1799400932 >> v1:172.21.15.59:6789/0 conn(0x7f2c2c0d2bf0 legacy=0x7f2c2c15dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.952+0000 7f2c319c6640 1 -- 172.21.15.59:0/1799400932 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c0cf040 con 0x7f2c2c1621c0 2024-03-06T07:27:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.953+0000 7f2c319c6640 1 --2- 172.21.15.59:0/1799400932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c2c1621c0 0x7f2c2c0cfb20 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0736c0 tx=0x7f2c2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b0756f2119d5554 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.953+0000 7f2c309c4640 1 -- 172.21.15.59:0/1799400932 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c28035020 con 0x7f2c2c1621c0 2024-03-06T07:27:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.953+0000 7f2c309c4640 1 -- 172.21.15.59:0/1799400932 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c28031830 con 0x7f2c2c1621c0 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.953+0000 7f2c309c4640 1 -- 172.21.15.59:0/1799400932 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c28031c00 con 0x7f2c2c1621c0 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.953+0000 7f2c33c2a640 1 -- 172.21.15.59:0/1799400932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c2c1621c0 msgr2=0x7f2c2c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.953+0000 7f2c33c2a640 1 --2- 172.21.15.59:0/1799400932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c2c1621c0 0x7f2c2c0cfb20 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0736c0 tx=0x7f2c2802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.953+0000 7f2c33c2a640 1 -- 172.21.15.59:0/1799400932 shutdown_connections 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.954+0000 7f2c33c2a640 1 --2- 172.21.15.59:0/1799400932 >> v2:172.21.15.164:3300/0 conn(0x7f2c2c0d4920 0x7f2c2c0d26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.954+0000 7f2c33c2a640 1 --2- 172.21.15.59:0/1799400932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c2c1621c0 0x7f2c2c0cfb20 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.954+0000 7f2c33c2a640 1 -- 172.21.15.59:0/1799400932 >> 172.21.15.59:0/1799400932 conn(0x7f2c2c0d3d10 msgr2=0x7f2c2c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.954+0000 7f2c33c2a640 1 -- 172.21.15.59:0/1799400932 shutdown_connections 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.954+0000 7f2c33c2a640 1 -- 172.21.15.59:0/1799400932 wait complete. 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.955+0000 7f2c33c2a640 1 Processor -- start 2024-03-06T07:27:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.955+0000 7f2c33c2a640 1 -- start start 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.955+0000 7f2c33c2a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2c0d2bf0 0x7f2c2c1f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.955+0000 7f2c33c2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c2c0d4920 0x7f2c2c1602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.955+0000 7f2c33c2a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c2c1621c0 0x7f2c2c0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.955+0000 7f2c33c2a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c162720 con 0x7f2c2c0d4920 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.955+0000 7f2c33c2a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c1625a0 con 0x7f2c2c1621c0 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.955+0000 7f2c33c2a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c1628a0 con 0x7f2c2c0d2bf0 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c311c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c2c0d4920 0x7f2c2c1602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c311c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c2c0d4920 0x7f2c2c1602d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37354/0 (socket says 172.21.15.59:37354) 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c311c5640 1 -- 172.21.15.59:0/866356245 learned_addr learned my addr 172.21.15.59:0/866356245 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c321c7640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c2c1621c0 0x7f2c2c0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c319c6640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2c0d2bf0 0x7f2c2c1f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c311c5640 1 -- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2c0d2bf0 msgr2=0x7f2c2c1f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c311c5640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2c0d2bf0 0x7f2c2c1f6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c311c5640 1 -- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c2c1621c0 msgr2=0x7f2c2c0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c311c5640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c2c1621c0 0x7f2c2c0cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c311c5640 1 -- 172.21.15.59:0/866356245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c15ea50 con 0x7f2c2c0d4920 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c311c5640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c2c0d4920 0x7f2c2c1602d0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00e9b0 tx=0x7f2c1c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c227fc640 1 -- 172.21.15.59:0/866356245 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c017070 con 0x7f2c2c0d4920 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c227fc640 1 -- 172.21.15.59:0/866356245 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1c00c9a0 con 0x7f2c2c0d4920 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c33c2a640 1 -- 172.21.15.59:0/866356245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2c15ed30 con 0x7f2c2c0d4920 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c227fc640 1 -- 172.21.15.59:0/866356245 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c00cd90 con 0x7f2c2c0d4920 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.956+0000 7f2c33c2a640 1 -- 172.21.15.59:0/866356245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2c15f2f0 con 0x7f2c2c0d4920 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.957+0000 7f2c33c2a640 1 -- 172.21.15.59:0/866356245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2c04f330 con 0x7f2c2c0d4920 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.958+0000 7f2c227fc640 1 -- 172.21.15.59:0/866356245 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c1c01a070 con 0x7f2c2c0d4920 2024-03-06T07:27:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.958+0000 7f2c227fc640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b4d0 0x7f2c1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.958+0000 7f2c227fc640 1 -- 172.21.15.59:0/866356245 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c1c08a4e0 con 0x7f2c2c0d4920 2024-03-06T07:27:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.958+0000 7f2c319c6640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b4d0 0x7f2c1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.959+0000 7f2c319c6640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b4d0 0x7f2c1006d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2c28036000 tx=0x7f2c2803c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:18.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:18.962+0000 7f2c227fc640 1 -- 172.21.15.59:0/866356245 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c1c058d10 con 0x7f2c2c0d4920 2024-03-06T07:27:19.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.183+0000 7f2c33c2a640 1 -- 172.21.15.59:0/866356245 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c2c1f71f0 con 0x7f2c2c0d4920 2024-03-06T07:27:19.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.183+0000 7f2c227fc640 1 -- 172.21.15.59:0/866356245 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f2c1c05cdb0 con 0x7f2c2c0d4920 2024-03-06T07:27:19.185 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:19.185 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3012766532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3012766532},{"type":"v1","addr":"172.21.15.164:6837","nonce":3012766532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/930220384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":930220384},{"type":"v1","addr":"172.21.15.59:6834","nonce":930220384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/557371287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":557371287},{"type":"v1","addr":"172.21.15.59:6835","nonce":557371287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:16.424559+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6836/2139041464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2139041464},{"type":"v1","addr":"172.21.15.164:6836","nonce":2139041464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":8}]} 2024-03-06T07:27:19.186 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 66 2024-03-06T07:27:19.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.185+0000 7f2c33c2a640 1 -- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b4d0 msgr2=0x7f2c1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.185+0000 7f2c33c2a640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b4d0 0x7f2c1006d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2c28036000 tx=0x7f2c2803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.185+0000 7f2c33c2a640 1 -- 172.21.15.59:0/866356245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c2c0d4920 msgr2=0x7f2c2c1602d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.185+0000 7f2c33c2a640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c2c0d4920 0x7f2c2c1602d0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00e9b0 tx=0x7f2c1c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:19.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.186+0000 7f2c33c2a640 1 -- 172.21.15.59:0/866356245 shutdown_connections 2024-03-06T07:27:19.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.186+0000 7f2c33c2a640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b4d0 0x7f2c1006d990 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.186+0000 7f2c33c2a640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c2c1621c0 0x7f2c2c0cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.186+0000 7f2c33c2a640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c2c0d4920 0x7f2c2c1602d0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.186+0000 7f2c33c2a640 1 --2- 172.21.15.59:0/866356245 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2c0d2bf0 0x7f2c2c1f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.186+0000 7f2c33c2a640 1 -- 172.21.15.59:0/866356245 >> 172.21.15.59:0/866356245 conn(0x7f2c2c0d3d10 msgr2=0x7f2c2c0b7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.186+0000 7f2c33c2a640 1 -- 172.21.15.59:0/866356245 shutdown_connections 2024-03-06T07:27:19.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.186+0000 7f2c33c2a640 1 -- 172.21.15.59:0/866356245 wait complete. 2024-03-06T07:27:19.601 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:14.345180+0000', 'modified': '2024-03-06T07:27:16.424559+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7968}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7968': {'gid': 7968, 'name': 'b', 'rank': 0, 'incarnation': 63, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.164:6836/2139041464', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6834', 'nonce': 2139041464}, {'type': 'v1', 'addr': '172.21.15.164:6836', 'nonce': 2139041464}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7968, 'qdb_cluster': [7968]} 2024-03-06T07:27:19.601 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:19.601 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:27:19.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.714+0000 7f51222ec640 1 Processor -- start 2024-03-06T07:27:19.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.714+0000 7f51222ec640 1 -- start start 2024-03-06T07:27:19.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.715+0000 7f51222ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 0x7f511c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.715+0000 7f51222ec640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511c0af4b0 con 0x7f511c0b0270 2024-03-06T07:27:19.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.715+0000 7f51222ec640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511c0af790 con 0x7f511c0b1a20 2024-03-06T07:27:19.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.715+0000 7f51222ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511c0afa70 con 0x7f511c1614a0 2024-03-06T07:27:19.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.715+0000 7f511b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f511c0b0270 0x7f511c0b1310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60860/0 (socket says 172.21.15.59:60860) 2024-03-06T07:27:19.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.715+0000 7f511b7fe640 1 -- 172.21.15.59:0/4185908255 learned_addr learned my addr 172.21.15.59:0/4185908255 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:19.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.716+0000 7f511bfff640 1 --2- 172.21.15.59:0/4185908255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 0x7f511c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.716+0000 7f511affd640 1 -- 172.21.15.59:0/4185908255 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831800560 0 0) 0x7f511c0af4b0 con 0x7f511c0b0270 2024-03-06T07:27:19.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.716+0000 7f511affd640 1 -- 172.21.15.59:0/4185908255 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5108003520 con 0x7f511c0b0270 2024-03-06T07:27:19.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.716+0000 7f511affd640 1 -- 172.21.15.59:0/4185908255 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860536517 0 0) 0x7f511c0af790 con 0x7f511c0b1a20 2024-03-06T07:27:19.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.716+0000 7f511affd640 1 -- 172.21.15.59:0/4185908255 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f511c0af4b0 con 0x7f511c0b1a20 2024-03-06T07:27:19.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.716+0000 7f511bfff640 1 -- 172.21.15.59:0/4185908255 >> v1:172.21.15.164:6789/0 conn(0x7f511c0b1a20 legacy=0x7f511c0b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.716+0000 7f511bfff640 1 -- 172.21.15.59:0/4185908255 >> v1:172.21.15.59:6789/0 conn(0x7f511c0b0270 legacy=0x7f511c0b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.716+0000 7f511bfff640 1 -- 172.21.15.59:0/4185908255 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c0afd50 con 0x7f511c1614a0 2024-03-06T07:27:19.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.717+0000 7f511bfff640 1 --2- 172.21.15.59:0/4185908255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 0x7f511c06d3c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f51040097d0 tx=0x7f510402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f73919212e6137e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.717+0000 7f511affd640 1 -- 172.21.15.59:0/4185908255 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104035020 con 0x7f511c1614a0 2024-03-06T07:27:19.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.717+0000 7f511affd640 1 -- 172.21.15.59:0/4185908255 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5104031800 con 0x7f511c1614a0 2024-03-06T07:27:19.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.717+0000 7f511affd640 1 -- 172.21.15.59:0/4185908255 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104031bd0 con 0x7f511c1614a0 2024-03-06T07:27:19.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.717+0000 7f51222ec640 1 -- 172.21.15.59:0/4185908255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 msgr2=0x7f511c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.717+0000 7f51222ec640 1 --2- 172.21.15.59:0/4185908255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 0x7f511c06d3c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f51040097d0 tx=0x7f510402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.717+0000 7f51222ec640 1 -- 172.21.15.59:0/4185908255 shutdown_connections 2024-03-06T07:27:19.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.717+0000 7f51222ec640 1 --2- 172.21.15.59:0/4185908255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 0x7f511c06d3c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.717+0000 7f51222ec640 1 -- 172.21.15.59:0/4185908255 >> 172.21.15.59:0/4185908255 conn(0x7f511c09a9f0 msgr2=0x7f511c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.718+0000 7f51222ec640 1 -- 172.21.15.59:0/4185908255 shutdown_connections 2024-03-06T07:27:19.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.718+0000 7f51222ec640 1 -- 172.21.15.59:0/4185908255 wait complete. 2024-03-06T07:27:19.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.719+0000 7f51222ec640 1 Processor -- start 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.719+0000 7f51222ec640 1 -- start start 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f51222ec640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f511c0b0270 0x7f511c1e68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f51222ec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f511c0b1a20 0x7f511c0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f51222ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 0x7f511c1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f51222ec640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c0b06b0 con 0x7f511c0b0270 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f51222ec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c0b1e50 con 0x7f511c0b1a20 2024-03-06T07:27:19.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f51222ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511c0b5c20 con 0x7f511c1614a0 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f511bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f511c0b0270 0x7f511c1e68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f511bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f511c0b0270 0x7f511c1e68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37368/0 (socket says 172.21.15.59:37368) 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f511bfff640 1 -- 172.21.15.59:0/2500728337 learned_addr learned my addr 172.21.15.59:0/2500728337 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f5120889640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 0x7f511c1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f511b7fe640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f511c0b1a20 0x7f511c0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f511bfff640 1 -- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 msgr2=0x7f511c1527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f511bfff640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 0x7f511c1527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f511bfff640 1 -- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f511c0b1a20 msgr2=0x7f511c0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f511bfff640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f511c0b1a20 0x7f511c0b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f511bfff640 1 -- 172.21.15.59:0/2500728337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c1e6e20 con 0x7f511c0b0270 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f5120889640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 0x7f511c1527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f511bfff640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f511c0b0270 0x7f511c1e68e0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f5104034040 tx=0x7f51040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f5118ff9640 1 -- 172.21.15.59:0/2500728337 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104040070 con 0x7f511c0b0270 2024-03-06T07:27:19.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f5118ff9640 1 -- 172.21.15.59:0/2500728337 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5104031e30 con 0x7f511c0b0270 2024-03-06T07:27:19.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f51222ec640 1 -- 172.21.15.59:0/2500728337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511c1e8140 con 0x7f511c0b0270 2024-03-06T07:27:19.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.720+0000 7f5118ff9640 1 -- 172.21.15.59:0/2500728337 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104035040 con 0x7f511c0b0270 2024-03-06T07:27:19.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.721+0000 7f51222ec640 1 -- 172.21.15.59:0/2500728337 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511c1e7380 con 0x7f511c0b0270 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.722+0000 7f5118ff9640 1 -- 172.21.15.59:0/2500728337 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f510404b060 con 0x7f511c0b0270 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.723+0000 7f5118ff9640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f50fc06b4d0 0x7f50fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.723+0000 7f5118ff9640 1 -- 172.21.15.59:0/2500728337 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51040aba10 con 0x7f511c0b0270 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.723+0000 7f511b7fe640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f50fc06b4d0 0x7f50fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.723+0000 7f511b7fe640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f50fc06b4d0 0x7f50fc06d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f511c09fcc0 tx=0x7f510c007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.724+0000 7f5124fa8740 1 Processor -- start 2024-03-06T07:27:19.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.724+0000 7f5124fa8740 1 -- start start 2024-03-06T07:27:19.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.724+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136ccbd870 0x56136ccbdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.724+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136ccc18c0 0x56136ccc3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.724+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cced600 0x56136cceda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.724+0000 7f5124fa8740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136cc276b0 con 0x56136ccbd870 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.724+0000 7f5124fa8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136cbd0a60 con 0x56136ccc18c0 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.724+0000 7f5124fa8740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x56136cc23750 con 0x56136cced600 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f511bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136ccbd870 0x56136ccbdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f511bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136ccbd870 0x56136ccbdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37382/0 (socket says 172.21.15.59:37382) 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f511bfff640 1 -- 172.21.15.59:0/2388400365 learned_addr learned my addr 172.21.15.59:0/2388400365 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f511b7fe640 1 --2- 172.21.15.59:0/2388400365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136ccc18c0 0x56136ccc3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f5120889640 1 --2- 172.21.15.59:0/2388400365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cced600 0x56136cceda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f511bfff640 1 -- 172.21.15.59:0/2388400365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cced600 msgr2=0x56136cceda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f511bfff640 1 --2- 172.21.15.59:0/2388400365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cced600 0x56136cceda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f511bfff640 1 -- 172.21.15.59:0/2388400365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136ccc18c0 msgr2=0x56136ccc3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f511bfff640 1 --2- 172.21.15.59:0/2388400365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136ccc18c0 0x56136ccc3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f511bfff640 1 -- 172.21.15.59:0/2388400365 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56136cbeef30 con 0x56136ccbd870 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f511bfff640 1 --2- 172.21.15.59:0/2388400365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136ccbd870 0x56136ccbdc50 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f51040a9ff0 tx=0x7f5104079cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5184ccc8052ee407 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f5111ffb640 1 -- 172.21.15.59:0/2388400365 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510407e370 con 0x56136ccbd870 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f5111ffb640 1 -- 172.21.15.59:0/2388400365 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5104082930 con 0x56136ccbd870 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.725+0000 7f5111ffb640 1 -- 172.21.15.59:0/2388400365 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104078370 con 0x56136ccbd870 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.726+0000 7f5124fa8740 1 -- 172.21.15.59:0/2388400365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136ccbd870 msgr2=0x56136ccbdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.726+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2388400365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136ccbd870 0x56136ccbdc50 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f51040a9ff0 tx=0x7f5104079cf0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.726+0000 7f5124fa8740 1 -- 172.21.15.59:0/2388400365 shutdown_connections 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.726+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2388400365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cced600 0x56136cceda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.726+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2388400365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136ccc18c0 0x56136ccc3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.726+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2388400365 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136ccbd870 0x56136ccbdc50 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.726+0000 7f5124fa8740 1 -- 172.21.15.59:0/2388400365 >> 172.21.15.59:0/2388400365 conn(0x56136ccbf8f0 msgr2=0x56136ccc11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.726+0000 7f5124fa8740 1 -- 172.21.15.59:0/2388400365 shutdown_connections 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.726+0000 7f5124fa8740 1 -- 172.21.15.59:0/2388400365 wait complete. 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.726+0000 7f5124fa8740 1 -- start start 2024-03-06T07:27:19.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.727+0000 7f5124fa8740 1 Processor -- start 2024-03-06T07:27:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.727+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136cce8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.727+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd08b30 0x56136cd06160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.727+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cd066a0 0x56136cd04760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.727+0000 7f5124fa8740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136ccc1d50 con 0x56136cced600 2024-03-06T07:27:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.727+0000 7f5124fa8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136cbe3ea0 con 0x56136cd08b30 2024-03-06T07:27:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.727+0000 7f5124fa8740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x56136ccc2030 con 0x56136cd066a0 2024-03-06T07:27:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.727+0000 7f5124fa8740 10 client.0 _ll_register_callbacks cb 0x56136cbe3870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f511bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136cce8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f511bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136cce8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37398/0 (socket says 172.21.15.59:37398) 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f511bfff640 1 -- 172.21.15.59:0/1750949330 learned_addr learned my addr 172.21.15.59:0/1750949330 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f5120889640 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cd066a0 0x56136cd04760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f511b7fe640 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd08b30 0x56136cd06160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f511bfff640 1 -- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cd066a0 msgr2=0x56136cd04760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f511bfff640 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cd066a0 0x56136cd04760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f511bfff640 1 -- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd08b30 msgr2=0x56136cd06160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f511bfff640 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd08b30 0x56136cd06160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f511bfff640 1 -- 172.21.15.59:0/1750949330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56136cbe4270 con 0x56136cced600 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f511bfff640 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136cce8990 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f510402beb0 tx=0x7f510403c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb96c25eff383c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f50faffd640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f50faffd640 1 -- 172.21.15.59:0/1750949330 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510407e370 con 0x56136cced600 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f50faffd640 1 -- 172.21.15.59:0/1750949330 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51040a9fa0 con 0x56136cced600 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f5124fa8740 1 -- 172.21.15.59:0/1750949330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5104082930 con 0x56136cced600 2024-03-06T07:27:19.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f50faffd640 1 -- 172.21.15.59:0/1750949330 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5104079720 con 0x56136cced600 2024-03-06T07:27:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f5124fa8740 10 client.8103 fetch_fsmap learned FSMap version 66 2024-03-06T07:27:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f5124fa8740 1 -- 172.21.15.59:0/1750949330 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x56136c93c2d0 con 0x56136cced600 2024-03-06T07:27:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.728+0000 7f50faffd640 1 -- 172.21.15.59:0/1750949330 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5104082930 con 0x56136cced600 2024-03-06T07:27:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.729+0000 7f50faffd640 1 -- 172.21.15.59:0/1750949330 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f51040798c0 con 0x56136cced600 2024-03-06T07:27:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.729+0000 7f5124fa8740 10 client.8103 fetch_fsmap finished waiting for FSMap version 66 2024-03-06T07:27:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.729+0000 7f5124fa8740 10 client.8103 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:27:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.729+0000 7f5124fa8740 20 client.8103 populate_metadata read hostname 'smithi059' 2024-03-06T07:27:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.729+0000 7f5124fa8740 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:6834/2139041464,v1:172.21.15.164:6836/2139041464] conn(0x56136ccd6870 0x56136cd278f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.729+0000 7f5124fa8740 4 client.8103 mds_command: new command op to 7968 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:19.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.729+0000 7f5124fa8740 1 -- 172.21.15.59:0/1750949330 --> [v2:172.21.15.164:6834/2139041464,v1:172.21.15.164:6836/2139041464] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56136cc34bf0 con 0x56136ccd6870 2024-03-06T07:27:19.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.729+0000 7f511b7fe640 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:6834/2139041464,v1:172.21.15.164:6836/2139041464] conn(0x56136ccd6870 0x56136cd278f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.729+0000 7f511b7fe640 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:6834/2139041464,v1:172.21.15.164:6836/2139041464] conn(0x56136ccd6870 0x56136cd278f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b387c5535f3b89fb server_cookie=27bbdf571f98c034 in_seq=0 out_seq=0 2024-03-06T07:27:19.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.729+0000 7f50faffd640 10 client.8103 ms_handle_connect on v2:172.21.15.164:6834/2139041464 2024-03-06T07:27:19.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f50faffd640 1 -- 172.21.15.59:0/1750949330 <== mds.0 v2:172.21.15.164:6834/2139041464 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56136cc34bf0 con 0x56136ccd6870 2024-03-06T07:27:19.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f50faffd640 10 client.8103 handle_command_reply: tid=0 2024-03-06T07:27:19.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 1 client.8103 shutdown 2024-03-06T07:27:19.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 1 -- 172.21.15.59:0/1750949330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 msgr2=0x56136cce8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136cce8990 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f510402beb0 tx=0x7f510403c040 comp rx=0 tx=0).stop 2024-03-06T07:27:19.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 1 -- 172.21.15.59:0/1750949330 shutdown_connections 2024-03-06T07:27:19.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:6834/2139041464,v1:172.21.15.164:6836/2139041464] conn(0x56136ccd6870 0x56136cd278f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cd066a0 0x56136cd04760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd08b30 0x56136cd06160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 1 --2- 172.21.15.59:0/1750949330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136cce8990 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f50faffd640 0 client.8103 ms_handle_reset on v2:172.21.15.164:6834/2139041464 2024-03-06T07:27:19.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 1 -- 172.21.15.59:0/1750949330 >> 172.21.15.59:0/1750949330 conn(0x56136ccbd870 msgr2=0x56136cceb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 1 -- 172.21.15.59:0/1750949330 shutdown_connections 2024-03-06T07:27:19.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 1 -- 172.21.15.59:0/1750949330 wait complete. 2024-03-06T07:27:19.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.730+0000 7f5124fa8740 20 client.8103 trim_cache size 0 max 16384 2024-03-06T07:27:19.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.747+0000 7f5124fa8740 1 Processor -- start 2024-03-06T07:27:19.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.747+0000 7f5124fa8740 1 -- start start 2024-03-06T07:27:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136ccec590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cd066a0 0x56136ccecad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd08b30 0x56136cceb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f5124fa8740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c1f4c90 con 0x56136cced600 2024-03-06T07:27:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f5124fa8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c15f2e0 con 0x56136cd08b30 2024-03-06T07:27:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f5124fa8740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511c1f4690 con 0x56136cd066a0 2024-03-06T07:27:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f511bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136ccec590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f511bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136ccec590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37414/0 (socket says 172.21.15.59:37414) 2024-03-06T07:27:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f511bfff640 1 -- 172.21.15.59:0/2379831990 learned_addr learned my addr 172.21.15.59:0/2379831990 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f511b7fe640 1 --2- 172.21.15.59:0/2379831990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cd066a0 0x56136ccecad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f5120889640 1 --2- 172.21.15.59:0/2379831990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd08b30 0x56136cceb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f511bfff640 1 -- 172.21.15.59:0/2379831990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cd066a0 msgr2=0x56136ccecad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f511bfff640 1 --2- 172.21.15.59:0/2379831990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cd066a0 0x56136ccecad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f511bfff640 1 -- 172.21.15.59:0/2379831990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd08b30 msgr2=0x56136cceb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f511bfff640 1 --2- 172.21.15.59:0/2379831990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd08b30 0x56136cceb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f511bfff640 1 -- 172.21.15.59:0/2379831990 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56136ca066b0 con 0x56136cced600 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.748+0000 7f511b7fe640 1 --2- 172.21.15.59:0/2379831990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cd066a0 0x56136ccecad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f511bfff640 1 --2- 172.21.15.59:0/2379831990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136ccec590 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f5104076730 tx=0x7f5104093070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a146efb86b2f25ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f51117fa640 1 -- 172.21.15.59:0/2379831990 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104086030 con 0x56136cced600 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f51117fa640 1 -- 172.21.15.59:0/2379831990 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5104095070 con 0x56136cced600 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f51117fa640 1 -- 172.21.15.59:0/2379831990 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510407e370 con 0x56136cced600 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f5124fa8740 1 -- 172.21.15.59:0/2379831990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 msgr2=0x56136ccec590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2379831990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136ccec590 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f5104076730 tx=0x7f5104093070 comp rx=0 tx=0).stop 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f5124fa8740 1 -- 172.21.15.59:0/2379831990 shutdown_connections 2024-03-06T07:27:19.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2379831990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd08b30 0x56136cceb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2379831990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cd066a0 0x56136ccecad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2379831990 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cced600 0x56136ccec590 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f5124fa8740 1 -- 172.21.15.59:0/2379831990 >> 172.21.15.59:0/2379831990 conn(0x56136ccbd870 msgr2=0x56136cd04520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f5124fa8740 1 -- 172.21.15.59:0/2379831990 shutdown_connections 2024-03-06T07:27:19.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.749+0000 7f5124fa8740 1 -- 172.21.15.59:0/2379831990 wait complete. 2024-03-06T07:27:19.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.750+0000 7f5124fa8740 1 -- start start 2024-03-06T07:27:19.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.750+0000 7f5124fa8740 1 Processor -- start 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cced600 0x56136ccd4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd066a0 0x56136cd77140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5124fa8740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cd08b30 0x56136ccbc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5124fa8740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136ccc1d50 con 0x56136cd08b30 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5124fa8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56136cbe3ea0 con 0x56136cd066a0 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5124fa8740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x56136cd03c80 con 0x56136cced600 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5124fa8740 10 client.0 _ll_register_callbacks cb 0x7f5104079e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5120889640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cd08b30 0x56136ccbc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5120889640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cd08b30 0x56136ccbc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37424/0 (socket says 172.21.15.59:37424) 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5120889640 1 -- 172.21.15.59:0/2759000750 learned_addr learned my addr 172.21.15.59:0/2759000750 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f511bfff640 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cced600 0x56136ccd4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f511b7fe640 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd066a0 0x56136cd77140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5120889640 1 -- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cced600 msgr2=0x56136ccd4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5120889640 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cced600 0x56136ccd4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5120889640 1 -- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd066a0 msgr2=0x56136cd77140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5120889640 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd066a0 0x56136cd77140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.751+0000 7f5120889640 1 -- 172.21.15.59:0/2759000750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56136ccee340 con 0x56136cd08b30 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f5120889640 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cd08b30 0x56136ccbc510 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f5114005320 tx=0x7f51140057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51f2056a88f59a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f50faffd640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f50faffd640 1 -- 172.21.15.59:0/2759000750 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5114017c40 con 0x56136cd08b30 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f50faffd640 1 -- 172.21.15.59:0/2759000750 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5114017de0 con 0x56136cd08b30 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f5124fa8740 1 -- 172.21.15.59:0/2759000750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5104095070 con 0x56136cd08b30 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f50faffd640 1 -- 172.21.15.59:0/2759000750 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5114027050 con 0x56136cd08b30 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f5124fa8740 10 client.8109 fetch_fsmap learned FSMap version 66 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f5124fa8740 1 -- 172.21.15.59:0/2759000750 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x56136ccebbc0 con 0x56136cd08b30 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f50faffd640 1 -- 172.21.15.59:0/2759000750 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5104095070 con 0x56136cd08b30 2024-03-06T07:27:19.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f50faffd640 1 -- 172.21.15.59:0/2759000750 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 66) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f511402a020 con 0x56136cd08b30 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f5124fa8740 10 client.8109 fetch_fsmap finished waiting for FSMap version 66 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f5124fa8740 10 client.8109 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f5124fa8740 20 client.8109 populate_metadata read hostname 'smithi059' 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:6834/2139041464,v1:172.21.15.164:6836/2139041464] conn(0x56136ccbcb90 0x56136cd7ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f5124fa8740 4 client.8109 mds_command: new command op to 7968 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f5124fa8740 1 -- 172.21.15.59:0/2759000750 --> [v2:172.21.15.164:6834/2139041464,v1:172.21.15.164:6836/2139041464] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56136cc2ef40 con 0x56136ccbcb90 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.752+0000 7f511bfff640 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:6834/2139041464,v1:172.21.15.164:6836/2139041464] conn(0x56136ccbcb90 0x56136cd7ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f50faffd640 10 client.8109 ms_handle_connect on v2:172.21.15.164:6834/2139041464 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f511bfff640 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:6834/2139041464,v1:172.21.15.164:6836/2139041464] conn(0x56136ccbcb90 0x56136cd7ad20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39498a4bb1079c08 server_cookie=ee3d5d544482bcca in_seq=0 out_seq=0 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f50faffd640 1 -- 172.21.15.59:0/2759000750 <== mds.0 v2:172.21.15.164:6834/2139041464 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x56136cc2ef40 con 0x56136ccbcb90 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f50faffd640 10 client.8109 handle_command_reply: tid=0 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 1 client.8109 shutdown 2024-03-06T07:27:19.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 1 -- 172.21.15.59:0/2759000750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cd08b30 msgr2=0x56136ccbc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cd08b30 0x56136ccbc510 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f5114005320 tx=0x7f51140057e0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 1 -- 172.21.15.59:0/2759000750 shutdown_connections 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:6834/2139041464,v1:172.21.15.164:6836/2139041464] conn(0x56136ccbcb90 0x56136cd7ad20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56136cd08b30 0x56136ccbc510 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56136cd066a0 0x56136cd77140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f50faffd640 0 client.8109 ms_handle_reset on v2:172.21.15.164:6834/2139041464 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 1 --2- 172.21.15.59:0/2759000750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56136cced600 0x56136ccd4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 1 -- 172.21.15.59:0/2759000750 >> 172.21.15.59:0/2759000750 conn(0x56136ccbd870 msgr2=0x56136cced9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 1 -- 172.21.15.59:0/2759000750 shutdown_connections 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 1 -- 172.21.15.59:0/2759000750 wait complete. 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.753+0000 7f5124fa8740 20 client.8109 trim_cache size 0 max 16384 2024-03-06T07:27:19.761 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 7968, 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 4.7781992109999996, 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 66, 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 108, 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 108, 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 10.112360709000001 2024-03-06T07:27:19.762 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:27:19.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 -- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f50fc06b4d0 msgr2=0x7f50fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f50fc06b4d0 0x7f50fc06d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f511c09fcc0 tx=0x7f510c007d50 comp rx=0 tx=0).stop 2024-03-06T07:27:19.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 -- 172.21.15.59:0/2500728337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f511c0b0270 msgr2=0x7f511c1e68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f511c0b0270 0x7f511c1e68e0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f5104034040 tx=0x7f51040046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 -- 172.21.15.59:0/2500728337 shutdown_connections 2024-03-06T07:27:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f50fc06b4d0 0x7f50fc06d990 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f511c1614a0 0x7f511c1527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f511c0b1a20 0x7f511c0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 --2- 172.21.15.59:0/2500728337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f511c0b0270 0x7f511c1e68e0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 -- 172.21.15.59:0/2500728337 >> 172.21.15.59:0/2500728337 conn(0x7f511c09a9f0 msgr2=0x7f511c0b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 -- 172.21.15.59:0/2500728337 shutdown_connections 2024-03-06T07:27:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.754+0000 7f51222ec640 1 -- 172.21.15.59:0/2500728337 wait complete. 2024-03-06T07:27:19.767 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:19.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.866+0000 7f56edceb640 1 Processor -- start 2024-03-06T07:27:19.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.866+0000 7f56edceb640 1 -- start start 2024-03-06T07:27:19.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.867+0000 7f56edceb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56e80b4170 0x7f56e80b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.867+0000 7f56edceb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f56e80d4760 0x7f56e80d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.867+0000 7f56edceb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e80d25d0 con 0x7f56e80b4dd0 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.867+0000 7f56edceb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e80d4bf0 con 0x7f56e80d4760 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.867+0000 7f56edceb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e80ce9e0 con 0x7f56e80b4170 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.867+0000 7f56e6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f56e80d4760 0x7f56e80d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.867+0000 7f56e6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f56e80d4760 0x7f56e80d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37440/0 (socket says 172.21.15.59:37440) 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.867+0000 7f56e6ffd640 1 -- 172.21.15.59:0/2115270546 learned_addr learned my addr 172.21.15.59:0/2115270546 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.867+0000 7f56e77fe640 1 --2- 172.21.15.59:0/2115270546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56e80b4170 0x7f56e80b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.868+0000 7f56e6ffd640 1 -- 172.21.15.59:0/2115270546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56e80b4170 msgr2=0x7f56e80b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.868+0000 7f56e6ffd640 1 --2- 172.21.15.59:0/2115270546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56e80b4170 0x7f56e80b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.868+0000 7f56e6ffd640 1 -- 172.21.15.59:0/2115270546 >> v1:172.21.15.164:6789/0 conn(0x7f56e80b4dd0 legacy=0x7f56e80cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.868+0000 7f56e67fc640 1 -- 172.21.15.59:0/2115270546 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427616261 0 0) 0x7f56e80d25d0 con 0x7f56e80b4dd0 2024-03-06T07:27:19.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.868+0000 7f56e6ffd640 1 -- 172.21.15.59:0/2115270546 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e80d2870 con 0x7f56e80d4760 2024-03-06T07:27:19.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.868+0000 7f56e6ffd640 1 --2- 172.21.15.59:0/2115270546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e80d4760 0x7f56e80d0370 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009870 tx=0x7f56dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c697ad45858602c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.869+0000 7f56e67fc640 1 -- 172.21.15.59:0/2115270546 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc034070 con 0x7f56e80d4760 2024-03-06T07:27:19.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.869+0000 7f56e67fc640 1 -- 172.21.15.59:0/2115270546 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc030840 con 0x7f56e80d4760 2024-03-06T07:27:19.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.869+0000 7f56e67fc640 1 -- 172.21.15.59:0/2115270546 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc030c10 con 0x7f56e80d4760 2024-03-06T07:27:19.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.869+0000 7f56edceb640 1 -- 172.21.15.59:0/2115270546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e80d4760 msgr2=0x7f56e80d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.869+0000 7f56edceb640 1 --2- 172.21.15.59:0/2115270546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e80d4760 0x7f56e80d0370 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009870 tx=0x7f56dc02b980 comp rx=0 tx=0).stop 2024-03-06T07:27:19.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.869+0000 7f56edceb640 1 -- 172.21.15.59:0/2115270546 shutdown_connections 2024-03-06T07:27:19.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.869+0000 7f56edceb640 1 --2- 172.21.15.59:0/2115270546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e80d4760 0x7f56e80d0370 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.869+0000 7f56edceb640 1 --2- 172.21.15.59:0/2115270546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56e80b4170 0x7f56e80b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.869+0000 7f56edceb640 1 -- 172.21.15.59:0/2115270546 >> 172.21.15.59:0/2115270546 conn(0x7f56e80d3c40 msgr2=0x7f56e80d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:19.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.869+0000 7f56edceb640 1 -- 172.21.15.59:0/2115270546 shutdown_connections 2024-03-06T07:27:19.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.869+0000 7f56edceb640 1 -- 172.21.15.59:0/2115270546 wait complete. 2024-03-06T07:27:19.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.870+0000 7f56edceb640 1 Processor -- start 2024-03-06T07:27:19.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.870+0000 7f56edceb640 1 -- start start 2024-03-06T07:27:19.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.871+0000 7f56edceb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56e80b4170 0x7f56e815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.871+0000 7f56edceb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e80b4dd0 0x7f56e80cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.871+0000 7f56edceb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56e80cc2a0 0x7f56e8157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.871+0000 7f56edceb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e80b47d0 con 0x7f56e80b4dd0 2024-03-06T07:27:19.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.871+0000 7f56edceb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e80b4550 con 0x7f56e80cc2a0 2024-03-06T07:27:19.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.871+0000 7f56edceb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e80b4950 con 0x7f56e80b4170 2024-03-06T07:27:19.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.871+0000 7f56e6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e80b4dd0 0x7f56e80cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.871+0000 7f56e6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e80b4dd0 0x7f56e80cbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37452/0 (socket says 172.21.15.59:37452) 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.871+0000 7f56e6ffd640 1 -- 172.21.15.59:0/227448632 learned_addr learned my addr 172.21.15.59:0/227448632 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.871+0000 7f56e77fe640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56e80b4170 0x7f56e815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.871+0000 7f56e7fff640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56e80cc2a0 0x7f56e8157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.872+0000 7f56e6ffd640 1 -- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56e80b4170 msgr2=0x7f56e815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.872+0000 7f56e6ffd640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56e80b4170 0x7f56e815aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.872+0000 7f56e6ffd640 1 -- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56e80cc2a0 msgr2=0x7f56e8157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.872+0000 7f56e6ffd640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56e80cc2a0 0x7f56e8157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.872+0000 7f56e6ffd640 1 -- 172.21.15.59:0/227448632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e80cc7e0 con 0x7f56e80b4dd0 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.872+0000 7f56e6ffd640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e80b4dd0 0x7f56e80cbd60 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f56dc000c00 tx=0x7f56dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.872+0000 7f56cffff640 1 -- 172.21.15.59:0/227448632 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc034040 con 0x7f56e80b4dd0 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.872+0000 7f56cffff640 1 -- 172.21.15.59:0/227448632 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc03ab90 con 0x7f56e80b4dd0 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.872+0000 7f56edceb640 1 -- 172.21.15.59:0/227448632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e80ccac0 con 0x7f56e80b4dd0 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.872+0000 7f56cffff640 1 -- 172.21.15.59:0/227448632 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc004030 con 0x7f56e80b4dd0 2024-03-06T07:27:19.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.872+0000 7f56edceb640 1 -- 172.21.15.59:0/227448632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e80cd080 con 0x7f56e80b4dd0 2024-03-06T07:27:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.873+0000 7f56edceb640 1 -- 172.21.15.59:0/227448632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e804f330 con 0x7f56e80b4dd0 2024-03-06T07:27:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.874+0000 7f56cffff640 1 -- 172.21.15.59:0/227448632 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56dc04b060 con 0x7f56e80b4dd0 2024-03-06T07:27:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.874+0000 7f56cffff640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56d406b4d0 0x7f56d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.874+0000 7f56cffff640 1 -- 172.21.15.59:0/227448632 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56dc0abe70 con 0x7f56e80b4dd0 2024-03-06T07:27:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.874+0000 7f56e77fe640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56d406b4d0 0x7f56d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:19.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.875+0000 7f56e77fe640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56d406b4d0 0x7f56d406d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009b40 tx=0x7f56d8009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:19.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:19.878+0000 7f56cffff640 1 -- 172.21.15.59:0/227448632 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56dc03f070 con 0x7f56e80b4dd0 2024-03-06T07:27:19.944 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:19.947 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:20.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.099+0000 7f56edceb640 1 -- 172.21.15.59:0/227448632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f56e80733b0 con 0x7f56e80b4dd0 2024-03-06T07:27:20.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.099+0000 7f56cffff640 1 -- 172.21.15.59:0/227448632 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) v1 ==== 53+0+73 (secure 0 0 0) 0x7f56dc07a850 con 0x7f56e80b4dd0 2024-03-06T07:27:20.101 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:20.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 -- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56d406b4d0 msgr2=0x7f56d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56d406b4d0 0x7f56d406d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009b40 tx=0x7f56d8009190 comp rx=0 tx=0).stop 2024-03-06T07:27:20.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 -- 172.21.15.59:0/227448632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e80b4dd0 msgr2=0x7f56e80cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:20.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e80b4dd0 0x7f56e80cbd60 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f56dc000c00 tx=0x7f56dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 -- 172.21.15.59:0/227448632 shutdown_connections 2024-03-06T07:27:20.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56d406b4d0 0x7f56d406d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56e80cc2a0 0x7f56e8157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e80b4dd0 0x7f56e80cbd60 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 --2- 172.21.15.59:0/227448632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56e80b4170 0x7f56e815aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:20.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 -- 172.21.15.59:0/227448632 >> 172.21.15.59:0/227448632 conn(0x7f56e80d3c40 msgr2=0x7f56e80b69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:20.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 -- 172.21.15.59:0/227448632 shutdown_connections 2024-03-06T07:27:20.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:20.102+0000 7f56edceb640 1 -- 172.21.15.59:0/227448632 wait complete. 2024-03-06T07:27:20.114 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:20.115 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:20.115 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:27:20.115 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:20.115 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:20.115 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:20.115 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:20.115 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:20.115 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:20.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.115 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:27:20.130 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:20.130 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.130 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:27:20.131 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:27:20.131 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:27:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:27:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42339sec preferred_lft 42339sec 2024-03-06T07:27:20.131 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:20.132 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:27:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.133 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:27:20.133 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:20.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.133 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:20.195 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:27:20.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.197 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:27:20.252 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:27:20.253 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:20.253 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:27:20.253 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:27:20.253 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:20.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.253 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:27:20.253 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:27:20.253 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:27:20.253 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:27:20.253 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:27:20.254 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:27:20.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.348 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:20.403 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:20.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.403 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:27:20.470 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:20.470 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:20.470 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:20.525 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:20.525 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:20.525 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:20.580 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:27:20.646 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:27:20.702 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:20.703 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:20.704 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:20.704 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:20.704 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:27:20.704 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:20.704 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:20.704 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:20.704 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:20.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.704 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:27:20.755 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:20.756 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:20.798 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:27:20.824 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:27:20.879 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:20.880 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:20.881 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:20.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:20.882 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:27:20.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:27:20.881+0000 7f3ff3241480 -1 init, newargv = 0x5597e2d35650 newargc=15 2024-03-06T07:27:20.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[72756]: starting ceph client 2024-03-06T07:27:20.887 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[72756]: starting fuse 2024-03-06T07:27:20.897 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:27:20.897 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:27:20.898 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:20.899 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:27:21.018 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.72756.asok 2024-03-06T07:27:21.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:21.028 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.72756.asok 2024-03-06T07:27:21.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.028 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.72756.asok status 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "72756", 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:27:21.166 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 8118, 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 8118 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 133369749 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 133369749 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.8118 192.168.0.1:0/133369749", 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/133369749", 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 66, 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 108, 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:27:21.168 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:27:21.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.174 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:27:21.174 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.189 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:27:21.189 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.189 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:21.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.256 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-03-06T07:27:21.316 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:27:21.316 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:21.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.371 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-03-06T07:27:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:8+0 records in 2024-03-06T07:27:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:8+0 records out 2024-03-06T07:27:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0564225 s, 149 MB/s 2024-03-06T07:27:21.486 DEBUG:teuthology.orchestra.run.smithi059:> stdin-killer -- python3 -c ' 2024-03-06T07:27:21.486 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:27:21.486 DEBUG:teuthology.orchestra.run.smithi059:> import stat 2024-03-06T07:27:21.486 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:27:21.486 DEBUG:teuthology.orchestra.run.smithi059:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-03-06T07:27:21.486 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:27:21.551 INFO:teuthology.orchestra.run.smithi059.stdout:1099511627777 2024-03-06T07:27:21.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:21.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.560 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-03-06T07:27:21.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:21.617 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-03-06T07:27:21.678 DEBUG:teuthology.orchestra.run.smithi059:> stdin-killer -- python3 -c ' 2024-03-06T07:27:21.678 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:27:21.678 DEBUG:teuthology.orchestra.run.smithi059:> import stat 2024-03-06T07:27:21.678 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:27:21.678 DEBUG:teuthology.orchestra.run.smithi059:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-03-06T07:27:21.679 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:27:21.783 INFO:teuthology.orchestra.run.smithi059.stdout:1099511627777 2024-03-06T07:27:21.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:21.792 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:21.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.932+0000 7f2c29b13640 1 Processor -- start 2024-03-06T07:27:21.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.932+0000 7f2c29b13640 1 -- start start 2024-03-06T07:27:21.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c29b13640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2c240b3060 0x7f2c2415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c29b13640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2415bf30 0x7f2c2415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c29b13640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2415f2d0 con 0x7f2c2413dc10 2024-03-06T07:27:21.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c29b13640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c240cf3f0 con 0x7f2c240b3060 2024-03-06T07:27:21.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c29b13640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2415b210 con 0x7f2c2415bf30 2024-03-06T07:27:21.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c22ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2c240b3060 0x7f2c2415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c237fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2c2413dc10 0x7f2c240b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60872/0 (socket says 172.21.15.59:60872) 2024-03-06T07:27:21.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c237fe640 1 -- 172.21.15.59:0/4094176144 learned_addr learned my addr 172.21.15.59:0/4094176144 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:21.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c22ffd640 1 --2- 172.21.15.59:0/4094176144 >> v2:172.21.15.59:3300/0 conn(0x7f2c240b3060 0x7f2c2415e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37462/0 (socket says 172.21.15.59:37462) 2024-03-06T07:27:21.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c23fff640 1 --2- 172.21.15.59:0/4094176144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2415bf30 0x7f2c2415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c22ffd640 1 -- 172.21.15.59:0/4094176144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2415bf30 msgr2=0x7f2c2415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:21.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c22ffd640 1 --2- 172.21.15.59:0/4094176144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2415bf30 0x7f2c2415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c22ffd640 1 -- 172.21.15.59:0/4094176144 >> v1:172.21.15.59:6789/0 conn(0x7f2c2413dc10 legacy=0x7f2c240b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c22ffd640 1 -- 172.21.15.59:0/4094176144 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2415f580 con 0x7f2c240b3060 2024-03-06T07:27:21.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.933+0000 7f2c227fc640 1 -- 172.21.15.59:0/4094176144 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898901914 0 0) 0x7f2c2415f2d0 con 0x7f2c2413dc10 2024-03-06T07:27:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.934+0000 7f2c22ffd640 1 --2- 172.21.15.59:0/4094176144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c240b3060 0x7f2c2415e850 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2c180097a0 tx=0x7f2c1802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a6c0c794da7d2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.934+0000 7f2c227fc640 1 -- 172.21.15.59:0/4094176144 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c180315a0 con 0x7f2c240b3060 2024-03-06T07:27:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.934+0000 7f2c227fc640 1 -- 172.21.15.59:0/4094176144 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c18037070 con 0x7f2c240b3060 2024-03-06T07:27:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.934+0000 7f2c227fc640 1 -- 172.21.15.59:0/4094176144 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c18031960 con 0x7f2c240b3060 2024-03-06T07:27:21.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.935+0000 7f2c29b13640 1 -- 172.21.15.59:0/4094176144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c240b3060 msgr2=0x7f2c2415e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.935+0000 7f2c29b13640 1 --2- 172.21.15.59:0/4094176144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c240b3060 0x7f2c2415e850 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2c180097a0 tx=0x7f2c1802b840 comp rx=0 tx=0).stop 2024-03-06T07:27:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.935+0000 7f2c29b13640 1 -- 172.21.15.59:0/4094176144 shutdown_connections 2024-03-06T07:27:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.935+0000 7f2c29b13640 1 --2- 172.21.15.59:0/4094176144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2415bf30 0x7f2c2415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.935+0000 7f2c29b13640 1 --2- 172.21.15.59:0/4094176144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c240b3060 0x7f2c2415e850 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.935+0000 7f2c29b13640 1 -- 172.21.15.59:0/4094176144 >> 172.21.15.59:0/4094176144 conn(0x7f2c240cf8a0 msgr2=0x7f2c240cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.935+0000 7f2c29b13640 1 -- 172.21.15.59:0/4094176144 shutdown_connections 2024-03-06T07:27:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.935+0000 7f2c29b13640 1 -- 172.21.15.59:0/4094176144 wait complete. 2024-03-06T07:27:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.936+0000 7f2c29b13640 1 Processor -- start 2024-03-06T07:27:21.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.936+0000 7f2c29b13640 1 -- start start 2024-03-06T07:27:21.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c29b13640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c240b3060 0x7f2c241f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c29b13640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2413dc10 0x7f2c2415a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c29b13640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c2415bf30 0x7f2c241f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c29b13640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2413dff0 con 0x7f2c240b3060 2024-03-06T07:27:21.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c29b13640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2415b730 con 0x7f2c2415bf30 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c29b13640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c240708d0 con 0x7f2c2413dc10 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c237fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c240b3060 0x7f2c241f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c237fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c240b3060 0x7f2c241f6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37478/0 (socket says 172.21.15.59:37478) 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c237fe640 1 -- 172.21.15.59:0/1840925871 learned_addr learned my addr 172.21.15.59:0/1840925871 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c22ffd640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2413dc10 0x7f2c2415a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c23fff640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c2415bf30 0x7f2c241f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c237fe640 1 -- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2413dc10 msgr2=0x7f2c2415a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c237fe640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2413dc10 0x7f2c2415a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c237fe640 1 -- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c2415bf30 msgr2=0x7f2c241f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c237fe640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c2415bf30 0x7f2c241f4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c237fe640 1 -- 172.21.15.59:0/1840925871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c241f7600 con 0x7f2c240b3060 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c237fe640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c240b3060 0x7f2c241f6fb0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c500 tx=0x7f2c1400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c0bfff640 1 -- 172.21.15.59:0/1840925871 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c14018070 con 0x7f2c240b3060 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c0bfff640 1 -- 172.21.15.59:0/1840925871 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c14007580 con 0x7f2c240b3060 2024-03-06T07:27:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c29b13640 1 -- 172.21.15.59:0/1840925871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c241f78e0 con 0x7f2c240b3060 2024-03-06T07:27:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.937+0000 7f2c0bfff640 1 -- 172.21.15.59:0/1840925871 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c14004a20 con 0x7f2c240b3060 2024-03-06T07:27:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.938+0000 7f2c29b13640 1 -- 172.21.15.59:0/1840925871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c241f3b60 con 0x7f2c240b3060 2024-03-06T07:27:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.939+0000 7f2c29b13640 1 -- 172.21.15.59:0/1840925871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2404f330 con 0x7f2c240b3060 2024-03-06T07:27:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.939+0000 7f2c0bfff640 1 -- 172.21.15.59:0/1840925871 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c1401b070 con 0x7f2c240b3060 2024-03-06T07:27:21.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.939+0000 7f2c0bfff640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b500 0x7f2c1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:21.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.940+0000 7f2c0bfff640 1 -- 172.21.15.59:0/1840925871 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c14089440 con 0x7f2c240b3060 2024-03-06T07:27:21.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.940+0000 7f2c22ffd640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b500 0x7f2c1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:21.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.940+0000 7f2c22ffd640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b500 0x7f2c1006d9c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2c18036000 tx=0x7f2c18008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:21.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:21.944+0000 7f2c0bfff640 1 -- 172.21.15.59:0/1840925871 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c14057c60 con 0x7f2c240b3060 2024-03-06T07:27:22.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.166+0000 7f2c29b13640 1 -- 172.21.15.59:0/1840925871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c2406eb00 con 0x7f2c240b3060 2024-03-06T07:27:22.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.166+0000 7f2c0bfff640 1 -- 172.21.15.59:0/1840925871 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f2c1405bd00 con 0x7f2c240b3060 2024-03-06T07:27:22.168 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:22.169 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3012766532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3012766532},{"type":"v1","addr":"172.21.15.164:6837","nonce":3012766532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/930220384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":930220384},{"type":"v1","addr":"172.21.15.59:6834","nonce":930220384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/557371287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":557371287},{"type":"v1","addr":"172.21.15.59:6835","nonce":557371287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:16.424559+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6836/2139041464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2139041464},{"type":"v1","addr":"172.21.15.164:6836","nonce":2139041464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":8}]} 2024-03-06T07:27:22.169 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 66 2024-03-06T07:27:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 -- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b500 msgr2=0x7f2c1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b500 0x7f2c1006d9c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2c18036000 tx=0x7f2c18008030 comp rx=0 tx=0).stop 2024-03-06T07:27:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 -- 172.21.15.59:0/1840925871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c240b3060 msgr2=0x7f2c241f6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c240b3060 0x7f2c241f6fb0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c500 tx=0x7f2c1400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 -- 172.21.15.59:0/1840925871 shutdown_connections 2024-03-06T07:27:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c1006b500 0x7f2c1006d9c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c2415bf30 0x7f2c241f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c2413dc10 0x7f2c2415a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 --2- 172.21.15.59:0/1840925871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c240b3060 0x7f2c241f6fb0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 -- 172.21.15.59:0/1840925871 >> 172.21.15.59:0/1840925871 conn(0x7f2c240cf8a0 msgr2=0x7f2c240b0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 -- 172.21.15.59:0/1840925871 shutdown_connections 2024-03-06T07:27:22.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.169+0000 7f2c29b13640 1 -- 172.21.15.59:0/1840925871 wait complete. 2024-03-06T07:27:22.182 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:27:22.309 INFO:teuthology.orchestra.run.smithi164.stdout:{"message":"","return_code":0} 2024-03-06T07:27:22.310 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:22.310 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:27:22.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.350+0000 7f0c17849f40 1 Processor -- start 2024-03-06T07:27:22.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c17849f40 1 -- start start 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c17849f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563729c566c0 0x563729d2b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c17849f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c58f50 0x563729c9f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c17849f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563729d15570 0x563729c9fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c17849f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x563729c57c80 con 0x563729c58f50 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c17849f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563729bfeb70 con 0x563729c566c0 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c17849f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x563729c57f20 con 0x563729d15570 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c14313640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c58f50 0x563729c9f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c14313640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c58f50 0x563729c9f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37490/0 (socket says 172.21.15.59:37490) 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c14313640 1 -- 172.21.15.59:0/526998358 learned_addr learned my addr 172.21.15.59:0/526998358 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c14b14640 1 --2- 172.21.15.59:0/526998358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563729c566c0 0x563729d2b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.351+0000 7f0c15315640 1 --2- 172.21.15.59:0/526998358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563729d15570 0x563729c9fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c14313640 1 -- 172.21.15.59:0/526998358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563729d15570 msgr2=0x563729c9fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c14313640 1 --2- 172.21.15.59:0/526998358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563729d15570 0x563729c9fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c14313640 1 -- 172.21.15.59:0/526998358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563729c566c0 msgr2=0x563729d2b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c14313640 1 --2- 172.21.15.59:0/526998358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563729c566c0 0x563729d2b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c14313640 1 -- 172.21.15.59:0/526998358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563729d2e190 con 0x563729c58f50 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c14313640 1 --2- 172.21.15.59:0/526998358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c58f50 0x563729c9f650 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00b310 tx=0x7f0bfc00b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b22d89e8d29506 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c13311640 1 -- 172.21.15.59:0/526998358 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc008070 con 0x563729c58f50 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c13311640 1 -- 172.21.15.59:0/526998358 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc00d500 con 0x563729c58f50 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c13311640 1 -- 172.21.15.59:0/526998358 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc00d8b0 con 0x563729c58f50 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c17849f40 1 -- 172.21.15.59:0/526998358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c58f50 msgr2=0x563729c9f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c17849f40 1 --2- 172.21.15.59:0/526998358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c58f50 0x563729c9f650 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00b310 tx=0x7f0bfc00b7d0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c17849f40 1 -- 172.21.15.59:0/526998358 shutdown_connections 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c17849f40 1 --2- 172.21.15.59:0/526998358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563729d15570 0x563729c9fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c17849f40 1 --2- 172.21.15.59:0/526998358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c58f50 0x563729c9f650 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c17849f40 1 --2- 172.21.15.59:0/526998358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563729c566c0 0x563729d2b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c17849f40 1 -- 172.21.15.59:0/526998358 >> 172.21.15.59:0/526998358 conn(0x563729c83350 msgr2=0x563729c55b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c17849f40 1 -- 172.21.15.59:0/526998358 shutdown_connections 2024-03-06T07:27:22.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.352+0000 7f0c17849f40 1 -- 172.21.15.59:0/526998358 wait complete. 2024-03-06T07:27:22.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.353+0000 7f0c17849f40 1 Processor -- start 2024-03-06T07:27:22.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.353+0000 7f0c17849f40 1 -- start start 2024-03-06T07:27:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c17849f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c566c0 0x563729d0f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c17849f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563729c58f50 0x563729dc6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c17849f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563729d15570 0x563729dc7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c17849f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x563729c9dce0 con 0x563729c566c0 2024-03-06T07:27:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c17849f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563729d14d40 con 0x563729c58f50 2024-03-06T07:27:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c17849f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x563729d0cf10 con 0x563729d15570 2024-03-06T07:27:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c14b14640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c566c0 0x563729d0f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c14b14640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c566c0 0x563729d0f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37504/0 (socket says 172.21.15.59:37504) 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c14b14640 1 -- 172.21.15.59:0/1111331834 learned_addr learned my addr 172.21.15.59:0/1111331834 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c15315640 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563729d15570 0x563729dc7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c14313640 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563729c58f50 0x563729dc6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c14b14640 1 -- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563729d15570 msgr2=0x563729dc7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c14b14640 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563729d15570 0x563729dc7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c14b14640 1 -- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563729c58f50 msgr2=0x563729dc6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c14b14640 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563729c58f50 0x563729dc6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c14b14640 1 -- 172.21.15.59:0/1111331834 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563729d14140 con 0x563729c566c0 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c15315640 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563729d15570 0x563729dc7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c14b14640 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c566c0 0x563729d0f7f0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0086b0 tx=0x7f0c0c0087a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c1230f640 1 -- 172.21.15.59:0/1111331834 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c019020 con 0x563729c566c0 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c1230f640 1 -- 172.21.15.59:0/1111331834 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c015880 con 0x563729c566c0 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c17849f40 1 -- 172.21.15.59:0/1111331834 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563729c9d390 con 0x563729c566c0 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.354+0000 7f0c1230f640 1 -- 172.21.15.59:0/1111331834 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c015c30 con 0x563729c566c0 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.355+0000 7f0c17849f40 1 -- 172.21.15.59:0/1111331834 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563729c85b60 con 0x563729c566c0 2024-03-06T07:27:22.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.356+0000 7f0c1230f640 1 -- 172.21.15.59:0/1111331834 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c0c01c020 con 0x563729c566c0 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.356+0000 7f0c1230f640 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0bf4069ee0 0x7f0bf406c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.356+0000 7f0c1230f640 1 -- 172.21.15.59:0/1111331834 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c0c089780 con 0x563729c566c0 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.356+0000 7f0c14313640 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0bf4069ee0 0x7f0bf406c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.356+0000 7f0c17849f40 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x563729d0fd30 0x563729dcefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.356+0000 7f0c17849f40 1 -- 172.21.15.59:0/1111331834 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x563729d288d0 con 0x563729d0fd30 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c15315640 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x563729d0fd30 0x563729dcefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c14313640 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0bf4069ee0 0x7f0bf406c3a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00b810 tx=0x7f0bfc008000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c15315640 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x563729d0fd30 0x563729dcefd0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c15315640 1 -- 172.21.15.59:0/1111331834 <== osd.0 v2:172.21.15.59:6824/841983189 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f0c0405e040 con 0x563729d0fd30 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c17849f40 1 -- 172.21.15.59:0/1111331834 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x563729d0fd30 msgr2=0x563729dcefd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c17849f40 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x563729d0fd30 0x563729dcefd0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c17849f40 1 -- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0bf4069ee0 msgr2=0x7f0bf406c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c17849f40 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0bf4069ee0 0x7f0bf406c3a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00b810 tx=0x7f0bfc008000 comp rx=0 tx=0).stop 2024-03-06T07:27:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c17849f40 1 -- 172.21.15.59:0/1111331834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c566c0 msgr2=0x563729d0f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c17849f40 1 --2- 172.21.15.59:0/1111331834 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x563729c566c0 0x563729d0f7f0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0086b0 tx=0x7f0c0c0087a0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c15315640 1 -- 172.21.15.59:0/1111331834 reap_dead start 2024-03-06T07:27:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c17849f40 1 -- 172.21.15.59:0/1111331834 shutdown_connections 2024-03-06T07:27:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c17849f40 1 -- 172.21.15.59:0/1111331834 >> 172.21.15.59:0/1111331834 conn(0x563729c83350 msgr2=0x563729c9f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c17849f40 1 -- 172.21.15.59:0/1111331834 shutdown_connections 2024-03-06T07:27:22.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.357+0000 7f0c17849f40 1 -- 172.21.15.59:0/1111331834 wait complete. 2024-03-06T07:27:22.366 INFO:teuthology.orchestra.run.smithi059.stdout:Wfile_a 2024-03-06T07:27:22.366 INFO:teuthology.orchestra.run.smithi059.stdout:dir_1 2024-03-06T07:27:22.366 DEBUG:teuthology.orchestra.run.smithi059:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:27:22.487 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:27:22.487 INFO:teuthology.orchestra.run.smithi059.stdout: "ino": 1099511627777, 2024-03-06T07:27:22.487 INFO:teuthology.orchestra.run.smithi059.stdout: "ancestors": [ 2024-03-06T07:27:22.487 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: "dirino": 1099511627776, 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: "dname": "file_a", 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: "version": 10 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: "dirino": 1, 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: "dname": "dir_1", 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: "version": 17 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: "pool": 17, 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: "old_pools": [] 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:27:22.488 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:22.498 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:27:22.498 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:22.554 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:22.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.693+0000 7f56dfb5c640 1 Processor -- start 2024-03-06T07:27:22.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.693+0000 7f56dfb5c640 1 -- start start 2024-03-06T07:27:22.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dfb5c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f56d80d00c0 0x7f56d80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dfb5c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f56d806cf20 0x7f56d806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dfb5c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d80d09e0 con 0x7f56d80cf850 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dfb5c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d806ca90 con 0x7f56d80d00c0 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dfb5c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d815e380 con 0x7f56d806cf20 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dd8f8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f56d80d00c0 0x7f56d80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dd8f8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f56d80d00c0 0x7f56d80d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37518/0 (socket says 172.21.15.59:37518) 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dd8f8640 1 -- 172.21.15.59:0/240050682 learned_addr learned my addr 172.21.15.59:0/240050682 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56de0f9640 1 --2- 172.21.15.59:0/240050682 >> v2:172.21.15.164:3300/0 conn(0x7f56d806cf20 0x7f56d806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dd8f8640 1 -- 172.21.15.59:0/240050682 >> v2:172.21.15.164:3300/0 conn(0x7f56d806cf20 msgr2=0x7f56d806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dd8f8640 1 --2- 172.21.15.59:0/240050682 >> v2:172.21.15.164:3300/0 conn(0x7f56d806cf20 0x7f56d806d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dd8f8640 1 -- 172.21.15.59:0/240050682 >> v1:172.21.15.164:6789/0 conn(0x7f56d80cf850 legacy=0x7f56d80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.694+0000 7f56dd8f8640 1 -- 172.21.15.59:0/240050682 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d80cdc10 con 0x7f56d80d00c0 2024-03-06T07:27:22.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.695+0000 7f56dd8f8640 1 --2- 172.21.15.59:0/240050682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d80d00c0 0x7f56d80d04a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f56d80cfda0 tx=0x7f56d002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d3c84618b66e8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.695+0000 7f56dc8f6640 1 -- 172.21.15.59:0/240050682 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0034070 con 0x7f56d80d00c0 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.695+0000 7f56dc8f6640 1 -- 172.21.15.59:0/240050682 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d0030860 con 0x7f56d80d00c0 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.695+0000 7f56dc8f6640 1 -- 172.21.15.59:0/240050682 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0030c30 con 0x7f56d80d00c0 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.696+0000 7f56dfb5c640 1 -- 172.21.15.59:0/240050682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d80d00c0 msgr2=0x7f56d80d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.696+0000 7f56dfb5c640 1 --2- 172.21.15.59:0/240050682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d80d00c0 0x7f56d80d04a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f56d80cfda0 tx=0x7f56d002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.696+0000 7f56dfb5c640 1 -- 172.21.15.59:0/240050682 shutdown_connections 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.696+0000 7f56dfb5c640 1 --2- 172.21.15.59:0/240050682 >> v2:172.21.15.164:3300/0 conn(0x7f56d806cf20 0x7f56d806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.696+0000 7f56dfb5c640 1 --2- 172.21.15.59:0/240050682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d80d00c0 0x7f56d80d04a0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.696+0000 7f56dfb5c640 1 -- 172.21.15.59:0/240050682 >> 172.21.15.59:0/240050682 conn(0x7f56d80b35a0 msgr2=0x7f56d80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.696+0000 7f56dfb5c640 1 -- 172.21.15.59:0/240050682 shutdown_connections 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.696+0000 7f56dfb5c640 1 -- 172.21.15.59:0/240050682 wait complete. 2024-03-06T07:27:22.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.697+0000 7f56dfb5c640 1 Processor -- start 2024-03-06T07:27:22.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.697+0000 7f56dfb5c640 1 -- start start 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dfb5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56d806cf20 0x7f56d814a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dfb5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d80cf850 0x7f56d815a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dfb5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56d80d00c0 0x7f56d815aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dfb5c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d806c720 con 0x7f56d80cf850 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dfb5c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d806c440 con 0x7f56d806cf20 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dfb5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d80b1030 con 0x7f56d80d00c0 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dd0f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d80cf850 0x7f56d815a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dd0f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d80cf850 0x7f56d815a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37534/0 (socket says 172.21.15.59:37534) 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dd0f7640 1 -- 172.21.15.59:0/2871771809 learned_addr learned my addr 172.21.15.59:0/2871771809 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dd8f8640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56d806cf20 0x7f56d814a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56de0f9640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56d80d00c0 0x7f56d815aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dd0f7640 1 -- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56d80d00c0 msgr2=0x7f56d815aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dd0f7640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56d80d00c0 0x7f56d815aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dd0f7640 1 -- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56d806cf20 msgr2=0x7f56d814a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dd0f7640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56d806cf20 0x7f56d814a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dd0f7640 1 -- 172.21.15.59:0/2871771809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d80c9420 con 0x7f56d80cf850 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56dd0f7640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d80cf850 0x7f56d815a5b0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f56c800c400 tx=0x7f56c800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56ce7fc640 1 -- 172.21.15.59:0/2871771809 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c80052d0 con 0x7f56d80cf850 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56ce7fc640 1 -- 172.21.15.59:0/2871771809 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c8005490 con 0x7f56d80cf850 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.698+0000 7f56ce7fc640 1 -- 172.21.15.59:0/2871771809 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c8007260 con 0x7f56d80cf850 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.699+0000 7f56dfb5c640 1 -- 172.21.15.59:0/2871771809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d80c9700 con 0x7f56d80cf850 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.699+0000 7f56dfb5c640 1 -- 172.21.15.59:0/2871771809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d80c9cc0 con 0x7f56d80cf850 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.700+0000 7f56dfb5c640 1 -- 172.21.15.59:0/2871771809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d804f330 con 0x7f56d80cf850 2024-03-06T07:27:22.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.700+0000 7f56ce7fc640 1 -- 172.21.15.59:0/2871771809 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56c8007400 con 0x7f56d80cf850 2024-03-06T07:27:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.700+0000 7f56ce7fc640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56bc06b500 0x7f56bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.701+0000 7f56ce7fc640 1 -- 172.21.15.59:0/2871771809 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56c8089630 con 0x7f56d80cf850 2024-03-06T07:27:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.701+0000 7f56dd8f8640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56bc06b500 0x7f56bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:22.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.701+0000 7f56dd8f8640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56bc06b500 0x7f56bc06d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f56d0036000 tx=0x7f56d0008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:22.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.705+0000 7f56ce7fc640 1 -- 172.21.15.59:0/2871771809 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56c8057e60 con 0x7f56d80cf850 2024-03-06T07:27:22.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.887+0000 7f56dfb5c640 1 -- 172.21.15.59:0/2871771809 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56d806ff60 con 0x7f56d80cf850 2024-03-06T07:27:22.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.888+0000 7f56ce7fc640 1 -- 172.21.15.59:0/2871771809 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16615 (secure 0 0 0) 0x7f56c805ce20 con 0x7f56d80cf850 2024-03-06T07:27:22.890 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:22.890 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:27:15.985080+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:12.625799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:13.306376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:22.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 -- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56bc06b500 msgr2=0x7f56bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56bc06b500 0x7f56bc06d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f56d0036000 tx=0x7f56d0008030 comp rx=0 tx=0).stop 2024-03-06T07:27:22.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 -- 172.21.15.59:0/2871771809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d80cf850 msgr2=0x7f56d815a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:22.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d80cf850 0x7f56d815a5b0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f56c800c400 tx=0x7f56c800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 -- 172.21.15.59:0/2871771809 shutdown_connections 2024-03-06T07:27:22.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56bc06b500 0x7f56bc06d9c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56d80d00c0 0x7f56d815aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56d80cf850 0x7f56d815a5b0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 --2- 172.21.15.59:0/2871771809 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56d806cf20 0x7f56d814a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:22.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 -- 172.21.15.59:0/2871771809 >> 172.21.15.59:0/2871771809 conn(0x7f56d80b35a0 msgr2=0x7f56d806f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:22.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 -- 172.21.15.59:0/2871771809 shutdown_connections 2024-03-06T07:27:22.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:22.890+0000 7f56dfb5c640 1 -- 172.21.15.59:0/2871771809 wait complete. 2024-03-06T07:27:22.903 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:12.625799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:13.306376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:22.903 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:23.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.007+0000 7f47b23f7640 1 Processor -- start 2024-03-06T07:27:23.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.007+0000 7f47b23f7640 1 -- start start 2024-03-06T07:27:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.008+0000 7f47b23f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47ac0cea50 0x7f47ac0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.008+0000 7f47b23f7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f47ac06cab0 0x7f47ac0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.008+0000 7f47b23f7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac06d030 con 0x7f47ac15def0 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.008+0000 7f47b23f7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ac0cf1c0 con 0x7f47ac06cab0 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.008+0000 7f47b23f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac0cc760 con 0x7f47ac0cea50 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.008+0000 7f47abfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f47ac15def0 0x7f47ac0ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60884/0 (socket says 172.21.15.59:60884) 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.008+0000 7f47abfff640 1 -- 172.21.15.59:0/2547856748 learned_addr learned my addr 172.21.15.59:0/2547856748 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.008+0000 7f47b0994640 1 --2- 172.21.15.59:0/2547856748 >> v2:172.21.15.164:3300/0 conn(0x7f47ac06cab0 0x7f47ac0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.008+0000 7f47ab7fe640 1 --2- 172.21.15.59:0/2547856748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47ac0cea50 0x7f47ac0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.008+0000 7f47aaffd640 1 -- 172.21.15.59:0/2547856748 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204275499 0 0) 0x7f47ac06d030 con 0x7f47ac15def0 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.008+0000 7f47aaffd640 1 -- 172.21.15.59:0/2547856748 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4798003520 con 0x7f47ac15def0 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.009+0000 7f47b0994640 1 -- 172.21.15.59:0/2547856748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47ac0cea50 msgr2=0x7f47ac0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.009+0000 7f47b0994640 1 --2- 172.21.15.59:0/2547856748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47ac0cea50 0x7f47ac0b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.009+0000 7f47b0994640 1 -- 172.21.15.59:0/2547856748 >> v1:172.21.15.59:6789/0 conn(0x7f47ac15def0 legacy=0x7f47ac0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.009+0000 7f47b0994640 1 -- 172.21.15.59:0/2547856748 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac06d310 con 0x7f47ac06cab0 2024-03-06T07:27:23.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.009+0000 7f47ab7fe640 1 --2- 172.21.15.59:0/2547856748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47ac0cea50 0x7f47ac0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.009+0000 7f47b0994640 1 --2- 172.21.15.59:0/2547856748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47ac06cab0 0x7f47ac0ccbf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f47a4009d90 tx=0x7f47a402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d2f774939c58486 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.009+0000 7f47aaffd640 1 -- 172.21.15.59:0/2547856748 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a4039070 con 0x7f47ac06cab0 2024-03-06T07:27:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.010+0000 7f47aaffd640 1 -- 172.21.15.59:0/2547856748 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a402cce0 con 0x7f47ac06cab0 2024-03-06T07:27:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.010+0000 7f47aaffd640 1 -- 172.21.15.59:0/2547856748 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a4034660 con 0x7f47ac06cab0 2024-03-06T07:27:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.010+0000 7f47b23f7640 1 -- 172.21.15.59:0/2547856748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47ac06cab0 msgr2=0x7f47ac0ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.010+0000 7f47b23f7640 1 --2- 172.21.15.59:0/2547856748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47ac06cab0 0x7f47ac0ccbf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f47a4009d90 tx=0x7f47a402bf30 comp rx=0 tx=0).stop 2024-03-06T07:27:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.010+0000 7f47b23f7640 1 -- 172.21.15.59:0/2547856748 shutdown_connections 2024-03-06T07:27:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.010+0000 7f47b23f7640 1 --2- 172.21.15.59:0/2547856748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47ac06cab0 0x7f47ac0ccbf0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.010+0000 7f47b23f7640 1 --2- 172.21.15.59:0/2547856748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47ac0cea50 0x7f47ac0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.010+0000 7f47b23f7640 1 -- 172.21.15.59:0/2547856748 >> 172.21.15.59:0/2547856748 conn(0x7f47ac0b3690 msgr2=0x7f47ac0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.010+0000 7f47b23f7640 1 -- 172.21.15.59:0/2547856748 shutdown_connections 2024-03-06T07:27:23.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.010+0000 7f47b23f7640 1 -- 172.21.15.59:0/2547856748 wait complete. 2024-03-06T07:27:23.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.011+0000 7f47b23f7640 1 Processor -- start 2024-03-06T07:27:23.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.011+0000 7f47b23f7640 1 -- start start 2024-03-06T07:27:23.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47b23f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47ac06cab0 0x7f47ac158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47b23f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47ac0cea50 0x7f47ac14b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47b23f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47ac15def0 0x7f47ac0b57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47b23f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac15e2d0 con 0x7f47ac06cab0 2024-03-06T07:27:23.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47b23f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac0cc3f0 con 0x7f47ac0cea50 2024-03-06T07:27:23.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47b23f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac15e450 con 0x7f47ac15def0 2024-03-06T07:27:23.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47abfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47ac06cab0 0x7f47ac158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47abfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47ac06cab0 0x7f47ac158290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37536/0 (socket says 172.21.15.59:37536) 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47abfff640 1 -- 172.21.15.59:0/1433812632 learned_addr learned my addr 172.21.15.59:0/1433812632 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47ab7fe640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47ac0cea50 0x7f47ac14b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47b0994640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47ac15def0 0x7f47ac0b57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47abfff640 1 -- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47ac15def0 msgr2=0x7f47ac0b57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.012+0000 7f47abfff640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47ac15def0 0x7f47ac0b57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.013+0000 7f47abfff640 1 -- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47ac0cea50 msgr2=0x7f47ac14b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.013+0000 7f47abfff640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47ac0cea50 0x7f47ac14b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.013+0000 7f47abfff640 1 -- 172.21.15.59:0/1433812632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac15af20 con 0x7f47ac06cab0 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.013+0000 7f47abfff640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47ac06cab0 0x7f47ac158290 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f479c00c390 tx=0x7f479c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.013+0000 7f47a8ff9640 1 -- 172.21.15.59:0/1433812632 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479c017070 con 0x7f47ac06cab0 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.013+0000 7f47a8ff9640 1 -- 172.21.15.59:0/1433812632 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479c005ce0 con 0x7f47ac06cab0 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.013+0000 7f47b23f7640 1 -- 172.21.15.59:0/1433812632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ac15b200 con 0x7f47ac06cab0 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.013+0000 7f47a8ff9640 1 -- 172.21.15.59:0/1433812632 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479c0072f0 con 0x7f47ac06cab0 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.013+0000 7f47b23f7640 1 -- 172.21.15.59:0/1433812632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ac1490e0 con 0x7f47ac06cab0 2024-03-06T07:27:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.014+0000 7f47b23f7640 1 -- 172.21.15.59:0/1433812632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ac09fcf0 con 0x7f47ac06cab0 2024-03-06T07:27:23.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.014+0000 7f47a8ff9640 1 -- 172.21.15.59:0/1433812632 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f479c01a070 con 0x7f47ac06cab0 2024-03-06T07:27:23.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.015+0000 7f47a8ff9640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f478c06b500 0x7f478c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.015+0000 7f47a8ff9640 1 -- 172.21.15.59:0/1433812632 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f479c0895c0 con 0x7f47ac06cab0 2024-03-06T07:27:23.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.015+0000 7f47ab7fe640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f478c06b500 0x7f478c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.015+0000 7f47ab7fe640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f478c06b500 0x7f478c06d9c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4794009560 tx=0x7f4794009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.019+0000 7f47a8ff9640 1 -- 172.21.15.59:0/1433812632 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f479c057cc0 con 0x7f47ac06cab0 2024-03-06T07:27:23.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.199+0000 7f47b23f7640 1 -- 172.21.15.59:0/1433812632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f47ac068ff0 con 0x7f47ac06cab0 2024-03-06T07:27:23.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.199+0000 7f47a8ff9640 1 -- 172.21.15.59:0/1433812632 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7f479c05bd60 con 0x7f47ac06cab0 2024-03-06T07:27:23.201 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:27:23.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 -- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f478c06b500 msgr2=0x7f478c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f478c06b500 0x7f478c06d9c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4794009560 tx=0x7f4794009190 comp rx=0 tx=0).stop 2024-03-06T07:27:23.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 -- 172.21.15.59:0/1433812632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47ac06cab0 msgr2=0x7f47ac158290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47ac06cab0 0x7f47ac158290 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f479c00c390 tx=0x7f479c00c850 comp rx=0 tx=0).stop 2024-03-06T07:27:23.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 -- 172.21.15.59:0/1433812632 shutdown_connections 2024-03-06T07:27:23.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f478c06b500 0x7f478c06d9c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47ac15def0 0x7f47ac0b57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47ac0cea50 0x7f47ac14b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 --2- 172.21.15.59:0/1433812632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47ac06cab0 0x7f47ac158290 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 -- 172.21.15.59:0/1433812632 >> 172.21.15.59:0/1433812632 conn(0x7f47ac0b3690 msgr2=0x7f47ac0d2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 -- 172.21.15.59:0/1433812632 shutdown_connections 2024-03-06T07:27:23.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.201+0000 7f47b23f7640 1 -- 172.21.15.59:0/1433812632 wait complete. 2024-03-06T07:27:23.214 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:23.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.316+0000 7f88d0720640 1 Processor -- start 2024-03-06T07:27:23.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.316+0000 7f88d0720640 1 -- start start 2024-03-06T07:27:23.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.317+0000 7f88d0720640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f88c81614d0 0x7f88c80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.317+0000 7f88d0720640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c81618b0 con 0x7f88c80d1910 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.317+0000 7f88d0720640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c80cf400 con 0x7f88c80d24b0 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.317+0000 7f88d0720640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c80cf6a0 con 0x7f88c81614d0 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.317+0000 7f88cdcbb640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f88c80d1910 0x7f88c80d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60892/0 (socket says 172.21.15.59:60892) 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.317+0000 7f88cdcbb640 1 -- 172.21.15.59:0/799372728 learned_addr learned my addr 172.21.15.59:0/799372728 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.317+0000 7f88ce4bc640 1 --2- 172.21.15.59:0/799372728 >> v2:172.21.15.164:3300/0 conn(0x7f88c81614d0 0x7f88c80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.318+0000 7f88cd4ba640 1 -- 172.21.15.59:0/799372728 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475890519 0 0) 0x7f88c81618b0 con 0x7f88c80d1910 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.318+0000 7f88cd4ba640 1 -- 172.21.15.59:0/799372728 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b0003520 con 0x7f88c80d1910 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.318+0000 7f88cd4ba640 1 -- 172.21.15.59:0/799372728 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308164807 0 0) 0x7f88c80cf400 con 0x7f88c80d24b0 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.318+0000 7f88cd4ba640 1 -- 172.21.15.59:0/799372728 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c81618b0 con 0x7f88c80d24b0 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.318+0000 7f88ce4bc640 1 -- 172.21.15.59:0/799372728 >> v1:172.21.15.164:6789/0 conn(0x7f88c80d24b0 legacy=0x7f88c80b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.318+0000 7f88ce4bc640 1 -- 172.21.15.59:0/799372728 >> v1:172.21.15.59:6789/0 conn(0x7f88c80d1910 legacy=0x7f88c80d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.318+0000 7f88cd4ba640 1 -- 172.21.15.59:0/799372728 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 925313707 0 0) 0x7f88b0003520 con 0x7f88c80d1910 2024-03-06T07:27:23.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.318+0000 7f88ce4bc640 1 -- 172.21.15.59:0/799372728 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c80cf980 con 0x7f88c81614d0 2024-03-06T07:27:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.319+0000 7f88ce4bc640 1 --2- 172.21.15.59:0/799372728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c81614d0 0x7f88c80d4860 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009d90 tx=0x7f88b802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bc30ddd767c114f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.319+0000 7f88cd4ba640 1 -- 172.21.15.59:0/799372728 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8039070 con 0x7f88c81614d0 2024-03-06T07:27:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.319+0000 7f88cd4ba640 1 -- 172.21.15.59:0/799372728 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b802c940 con 0x7f88c81614d0 2024-03-06T07:27:23.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.319+0000 7f88cd4ba640 1 -- 172.21.15.59:0/799372728 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc001670 con 0x7f88c81614d0 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.319+0000 7f88d0720640 1 -- 172.21.15.59:0/799372728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c81614d0 msgr2=0x7f88c80d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.319+0000 7f88d0720640 1 --2- 172.21.15.59:0/799372728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c81614d0 0x7f88c80d4860 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009d90 tx=0x7f88b802be10 comp rx=0 tx=0).stop 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.319+0000 7f88d0720640 1 -- 172.21.15.59:0/799372728 shutdown_connections 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.319+0000 7f88d0720640 1 --2- 172.21.15.59:0/799372728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c81614d0 0x7f88c80d4860 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.319+0000 7f88d0720640 1 -- 172.21.15.59:0/799372728 >> 172.21.15.59:0/799372728 conn(0x7f88c80d3d40 msgr2=0x7f88c80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.319+0000 7f88d0720640 1 -- 172.21.15.59:0/799372728 shutdown_connections 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.319+0000 7f88d0720640 1 -- 172.21.15.59:0/799372728 wait complete. 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.320+0000 7f88d0720640 1 Processor -- start 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88d0720640 1 -- start start 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88d0720640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c80d1910 0x7f88c81f4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88d0720640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88c80d24b0 0x7f88c81530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88d0720640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c81614d0 0x7f88c80c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88d0720640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80b1540 con 0x7f88c80d24b0 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88d0720640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80b0910 con 0x7f88c80d1910 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88d0720640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80b16c0 con 0x7f88c81614d0 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88cdcbb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88c80d24b0 0x7f88c81530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88cdcbb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88c80d24b0 0x7f88c81530a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37550/0 (socket says 172.21.15.59:37550) 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88cdcbb640 1 -- 172.21.15.59:0/2969334487 learned_addr learned my addr 172.21.15.59:0/2969334487 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88ce4bc640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c80d1910 0x7f88c81f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.321+0000 7f88cecbd640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c81614d0 0x7f88c80c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.322+0000 7f88cdcbb640 1 -- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c81614d0 msgr2=0x7f88c80c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.322+0000 7f88cdcbb640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c81614d0 0x7f88c80c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.322+0000 7f88cdcbb640 1 -- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c80d1910 msgr2=0x7f88c81f4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.322+0000 7f88cdcbb640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c80d1910 0x7f88c81f4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.322+0000 7f88cdcbb640 1 -- 172.21.15.59:0/2969334487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c81f4850 con 0x7f88c80d24b0 2024-03-06T07:27:23.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.322+0000 7f88cdcbb640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88c80d24b0 0x7f88c81530a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f88bc00c780 tx=0x7f88bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.322+0000 7f88c6ffd640 1 -- 172.21.15.59:0/2969334487 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc017070 con 0x7f88c80d24b0 2024-03-06T07:27:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.322+0000 7f88c6ffd640 1 -- 172.21.15.59:0/2969334487 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88bc005af0 con 0x7f88c80d24b0 2024-03-06T07:27:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.322+0000 7f88d0720640 1 -- 172.21.15.59:0/2969334487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c80c7be0 con 0x7f88c80d24b0 2024-03-06T07:27:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.322+0000 7f88c6ffd640 1 -- 172.21.15.59:0/2969334487 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc007070 con 0x7f88c80d24b0 2024-03-06T07:27:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.322+0000 7f88d0720640 1 -- 172.21.15.59:0/2969334487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c80c8480 con 0x7f88c80d24b0 2024-03-06T07:27:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.323+0000 7f88d0720640 1 -- 172.21.15.59:0/2969334487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c8068ff0 con 0x7f88c80d24b0 2024-03-06T07:27:23.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.323+0000 7f88c6ffd640 1 -- 172.21.15.59:0/2969334487 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88bc01a070 con 0x7f88c80d24b0 2024-03-06T07:27:23.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.324+0000 7f88c6ffd640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88b406b4d0 0x7f88b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.324+0000 7f88c6ffd640 1 -- 172.21.15.59:0/2969334487 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88bc089730 con 0x7f88c80d24b0 2024-03-06T07:27:23.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.324+0000 7f88ce4bc640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88b406b4d0 0x7f88b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.324+0000 7f88ce4bc640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88b406b4d0 0x7f88b406d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f88b8036000 tx=0x7f88b8030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.328+0000 7f88c6ffd640 1 -- 172.21.15.59:0/2969334487 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88bc057f60 con 0x7f88c80d24b0 2024-03-06T07:27:23.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.507+0000 7f88d0720640 1 -- 172.21.15.59:0/2969334487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88c80a1c10 con 0x7f88c80d24b0 2024-03-06T07:27:23.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.507+0000 7f88c6ffd640 1 -- 172.21.15.59:0/2969334487 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) v1 ==== 104+0+11 (secure 0 0 0) 0x7f88bc05c000 con 0x7f88c80d24b0 2024-03-06T07:27:23.509 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:27:23.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 -- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88b406b4d0 msgr2=0x7f88b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88b406b4d0 0x7f88b406d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f88b8036000 tx=0x7f88b8030000 comp rx=0 tx=0).stop 2024-03-06T07:27:23.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 -- 172.21.15.59:0/2969334487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88c80d24b0 msgr2=0x7f88c81530a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88c80d24b0 0x7f88c81530a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f88bc00c780 tx=0x7f88bc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:27:23.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 -- 172.21.15.59:0/2969334487 shutdown_connections 2024-03-06T07:27:23.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f88b406b4d0 0x7f88b406d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c81614d0 0x7f88c80c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88c80d24b0 0x7f88c81530a0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 --2- 172.21.15.59:0/2969334487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c80d1910 0x7f88c81f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 -- 172.21.15.59:0/2969334487 >> 172.21.15.59:0/2969334487 conn(0x7f88c80d3d40 msgr2=0x7f88c80b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 -- 172.21.15.59:0/2969334487 shutdown_connections 2024-03-06T07:27:23.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.509+0000 7f88d0720640 1 -- 172.21.15.59:0/2969334487 wait complete. 2024-03-06T07:27:23.522 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:23.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.624+0000 7fe9a0a0e640 1 Processor -- start 2024-03-06T07:27:23.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.624+0000 7fe9a0a0e640 1 -- start start 2024-03-06T07:27:23.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe9a0a0e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe99c15bb40 0x7fe99c0b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe9a0a0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe99c0cfa90 0x7fe99c0cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe9a0a0e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c15ff90 con 0x7fe99c15f7d0 2024-03-06T07:27:23.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe9a0a0e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe99c06c910 con 0x7fe99c15bb40 2024-03-06T07:27:23.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe9a0a0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c160270 con 0x7fe99c0cfa90 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe99a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe99c15bb40 0x7fe99c0b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe99a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe99c15bb40 0x7fe99c0b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38394/0 (socket says 172.21.15.59:38394) 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe99a59c640 1 -- 172.21.15.59:0/3616317991 learned_addr learned my addr 172.21.15.59:0/3616317991 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe99ad9d640 1 --2- 172.21.15.59:0/3616317991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe99c0cfa90 0x7fe99c0cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe99a59c640 1 -- 172.21.15.59:0/3616317991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe99c0cfa90 msgr2=0x7fe99c0cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe99a59c640 1 --2- 172.21.15.59:0/3616317991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe99c0cfa90 0x7fe99c0cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe99a59c640 1 -- 172.21.15.59:0/3616317991 >> v1:172.21.15.164:6789/0 conn(0x7fe99c15f7d0 legacy=0x7fe99c06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.625+0000 7fe99a59c640 1 -- 172.21.15.59:0/3616317991 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c160550 con 0x7fe99c15bb40 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.626+0000 7fe99a59c640 1 --2- 172.21.15.59:0/3616317991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c15bb40 0x7fe99c0b4970 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fe994009870 tx=0x7fe99402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29dbe9a622222a38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.626+0000 7fe99959a640 1 -- 172.21.15.59:0/3616317991 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994034070 con 0x7fe99c15bb40 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.626+0000 7fe99959a640 1 -- 172.21.15.59:0/3616317991 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe994030840 con 0x7fe99c15bb40 2024-03-06T07:27:23.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.626+0000 7fe99959a640 1 -- 172.21.15.59:0/3616317991 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994030c10 con 0x7fe99c15bb40 2024-03-06T07:27:23.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.627+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3616317991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c15bb40 msgr2=0x7fe99c0b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.627+0000 7fe9a0a0e640 1 --2- 172.21.15.59:0/3616317991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c15bb40 0x7fe99c0b4970 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fe994009870 tx=0x7fe99402b980 comp rx=0 tx=0).stop 2024-03-06T07:27:23.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.627+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3616317991 shutdown_connections 2024-03-06T07:27:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.627+0000 7fe9a0a0e640 1 --2- 172.21.15.59:0/3616317991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe99c0cfa90 0x7fe99c0cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.627+0000 7fe9a0a0e640 1 --2- 172.21.15.59:0/3616317991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c15bb40 0x7fe99c0b4970 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.627+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3616317991 >> 172.21.15.59:0/3616317991 conn(0x7fe99c06b190 msgr2=0x7fe99c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.627+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3616317991 shutdown_connections 2024-03-06T07:27:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.627+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3616317991 wait complete. 2024-03-06T07:27:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.628+0000 7fe9a0a0e640 1 Processor -- start 2024-03-06T07:27:23.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.628+0000 7fe9a0a0e640 1 -- start start 2024-03-06T07:27:23.633 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.629+0000 7fe9a0a0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe99c0cfa90 0x7fe99c147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.629+0000 7fe9a0a0e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe99c15bb40 0x7fe99c0ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.629+0000 7fe9a0a0e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c15f7d0 0x7fe99c1f2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.629+0000 7fe9a0a0e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c06d790 con 0x7fe99c15f7d0 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.629+0000 7fe9a0a0e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c15fc30 con 0x7fe99c15bb40 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.629+0000 7fe9a0a0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c15bf20 con 0x7fe99c0cfa90 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.629+0000 7fe99ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c15f7d0 0x7fe99c1f2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.629+0000 7fe99ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c15f7d0 0x7fe99c1f2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38400/0 (socket says 172.21.15.59:38400) 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.629+0000 7fe99ad9d640 1 -- 172.21.15.59:0/3302672848 learned_addr learned my addr 172.21.15.59:0/3302672848 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.629+0000 7fe99a59c640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe99c0cfa90 0x7fe99c147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.629+0000 7fe999d9b640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe99c15bb40 0x7fe99c0ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.630+0000 7fe99ad9d640 1 -- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe99c0cfa90 msgr2=0x7fe99c147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.630+0000 7fe99ad9d640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe99c0cfa90 0x7fe99c147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.630+0000 7fe99ad9d640 1 -- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe99c15bb40 msgr2=0x7fe99c0ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.630+0000 7fe99ad9d640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe99c15bb40 0x7fe99c0ae380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.630+0000 7fe99ad9d640 1 -- 172.21.15.59:0/3302672848 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c0ad2e0 con 0x7fe99c15f7d0 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.630+0000 7fe99ad9d640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c15f7d0 0x7fe99c1f2e70 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fe98c00efc0 tx=0x7fe98c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.630+0000 7fe992ffd640 1 -- 172.21.15.59:0/3302672848 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c018020 con 0x7fe99c15f7d0 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.630+0000 7fe992ffd640 1 -- 172.21.15.59:0/3302672848 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98c014480 con 0x7fe99c15f7d0 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.630+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3302672848 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99c0ad5c0 con 0x7fe99c15f7d0 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.630+0000 7fe992ffd640 1 -- 172.21.15.59:0/3302672848 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c014870 con 0x7fe99c15f7d0 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.630+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3302672848 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99c0adb80 con 0x7fe99c15f7d0 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.631+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3302672848 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe99c0732b0 con 0x7fe99c15f7d0 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.631+0000 7fe992ffd640 1 -- 172.21.15.59:0/3302672848 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe98c005d10 con 0x7fe99c15f7d0 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.632+0000 7fe992ffd640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe97c06b4d0 0x7fe97c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.632+0000 7fe992ffd640 1 -- 172.21.15.59:0/3302672848 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe98c08ab80 con 0x7fe99c15f7d0 2024-03-06T07:27:23.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.632+0000 7fe99a59c640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe97c06b4d0 0x7fe97c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.632+0000 7fe99a59c640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe97c06b4d0 0x7fe97c06d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe994036000 tx=0x7fe994008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.636+0000 7fe992ffd640 1 -- 172.21.15.59:0/3302672848 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe98c022070 con 0x7fe99c15f7d0 2024-03-06T07:27:23.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.819+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3302672848 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe99c068ff0 con 0x7fe99c15f7d0 2024-03-06T07:27:23.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.819+0000 7fe992ffd640 1 -- 172.21.15.59:0/3302672848 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe98c010040 con 0x7fe99c15f7d0 2024-03-06T07:27:23.821 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:27:23.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.821+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe97c06b4d0 msgr2=0x7fe97c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.821+0000 7fe9a0a0e640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe97c06b4d0 0x7fe97c06d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe994036000 tx=0x7fe994008030 comp rx=0 tx=0).stop 2024-03-06T07:27:23.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.822+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3302672848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c15f7d0 msgr2=0x7fe99c1f2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.822+0000 7fe9a0a0e640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c15f7d0 0x7fe99c1f2e70 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fe98c00efc0 tx=0x7fe98c00c490 comp rx=0 tx=0).stop 2024-03-06T07:27:23.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.822+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3302672848 shutdown_connections 2024-03-06T07:27:23.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.822+0000 7fe9a0a0e640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe97c06b4d0 0x7fe97c06d990 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.822+0000 7fe9a0a0e640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c15f7d0 0x7fe99c1f2e70 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.822+0000 7fe9a0a0e640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe99c15bb40 0x7fe99c0ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.822+0000 7fe9a0a0e640 1 --2- 172.21.15.59:0/3302672848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe99c0cfa90 0x7fe99c147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.822+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3302672848 >> 172.21.15.59:0/3302672848 conn(0x7fe99c06b190 msgr2=0x7fe99c0b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.822+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3302672848 shutdown_connections 2024-03-06T07:27:23.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.822+0000 7fe9a0a0e640 1 -- 172.21.15.59:0/3302672848 wait complete. 2024-03-06T07:27:23.834 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:27:23.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.934+0000 7fd828cdf640 1 Processor -- start 2024-03-06T07:27:23.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.934+0000 7fd828cdf640 1 -- start start 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.935+0000 7fd828cdf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd8240b4460 0x7fd8240d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.935+0000 7fd828cdf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd824161cf0 0x7fd824162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.935+0000 7fd828cdf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8240d22c0 con 0x7fd824162970 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.935+0000 7fd828cdf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd824161b00 con 0x7fd8240b4460 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.935+0000 7fd828cdf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240cecc0 con 0x7fd824161cf0 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.935+0000 7fd821d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd824162970 0x7fd8240b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:53618/0 (socket says 172.21.15.59:53618) 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.935+0000 7fd821d9b640 1 -- 172.21.15.59:0/293958286 learned_addr learned my addr 172.21.15.59:0/293958286 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.935+0000 7fd822d9d640 1 --2- 172.21.15.59:0/293958286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd824161cf0 0x7fd824162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.935+0000 7fd82259c640 1 --2- 172.21.15.59:0/293958286 >> v2:172.21.15.164:3300/0 conn(0x7fd8240b4460 0x7fd8240d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.936+0000 7fd82159a640 1 -- 172.21.15.59:0/293958286 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836811782 0 0) 0x7fd8240d22c0 con 0x7fd824162970 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.936+0000 7fd82159a640 1 -- 172.21.15.59:0/293958286 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd810003520 con 0x7fd824162970 2024-03-06T07:27:23.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.936+0000 7fd822d9d640 1 -- 172.21.15.59:0/293958286 >> v2:172.21.15.164:3300/0 conn(0x7fd8240b4460 msgr2=0x7fd8240d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.936+0000 7fd822d9d640 1 --2- 172.21.15.59:0/293958286 >> v2:172.21.15.164:3300/0 conn(0x7fd8240b4460 0x7fd8240d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.936+0000 7fd822d9d640 1 -- 172.21.15.59:0/293958286 >> v1:172.21.15.59:6789/0 conn(0x7fd824162970 legacy=0x7fd8240b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.936+0000 7fd822d9d640 1 -- 172.21.15.59:0/293958286 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8240d25a0 con 0x7fd824161cf0 2024-03-06T07:27:23.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.936+0000 7fd82259c640 1 --2- 172.21.15.59:0/293958286 >> v2:172.21.15.164:3300/0 conn(0x7fd8240b4460 0x7fd8240d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:23.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.936+0000 7fd822d9d640 1 --2- 172.21.15.59:0/293958286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd824161cf0 0x7fd824162180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd814009d90 tx=0x7fd81402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=409d9b9cd431599d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.937+0000 7fd82159a640 1 -- 172.21.15.59:0/293958286 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814039070 con 0x7fd824161cf0 2024-03-06T07:27:23.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.937+0000 7fd82159a640 1 -- 172.21.15.59:0/293958286 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81402cc50 con 0x7fd824161cf0 2024-03-06T07:27:23.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.937+0000 7fd82159a640 1 -- 172.21.15.59:0/293958286 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814034440 con 0x7fd824161cf0 2024-03-06T07:27:23.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.937+0000 7fd828cdf640 1 -- 172.21.15.59:0/293958286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd824161cf0 msgr2=0x7fd824162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.937+0000 7fd828cdf640 1 --2- 172.21.15.59:0/293958286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd824161cf0 0x7fd824162180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd814009d90 tx=0x7fd81402bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.937+0000 7fd828cdf640 1 -- 172.21.15.59:0/293958286 shutdown_connections 2024-03-06T07:27:23.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.937+0000 7fd828cdf640 1 --2- 172.21.15.59:0/293958286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd824161cf0 0x7fd824162180 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.937+0000 7fd828cdf640 1 --2- 172.21.15.59:0/293958286 >> v2:172.21.15.164:3300/0 conn(0x7fd8240b4460 0x7fd8240d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.937+0000 7fd828cdf640 1 -- 172.21.15.59:0/293958286 >> 172.21.15.59:0/293958286 conn(0x7fd8240d3ce0 msgr2=0x7fd8240d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.937+0000 7fd828cdf640 1 -- 172.21.15.59:0/293958286 shutdown_connections 2024-03-06T07:27:23.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.937+0000 7fd828cdf640 1 -- 172.21.15.59:0/293958286 wait complete. 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd828cdf640 1 Processor -- start 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd828cdf640 1 -- start start 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd828cdf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8240b4460 0x7fd824155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd828cdf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd824161cf0 0x7fd824153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd828cdf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd824162970 0x7fd8240cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd828cdf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240ce950 con 0x7fd8240b4460 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd828cdf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240d2b60 con 0x7fd824161cf0 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd828cdf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240b4840 con 0x7fd824162970 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd82259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8240b4460 0x7fd824155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd82259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8240b4460 0x7fd824155980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38402/0 (socket says 172.21.15.59:38402) 2024-03-06T07:27:23.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd82259c640 1 -- 172.21.15.59:0/1892151849 learned_addr learned my addr 172.21.15.59:0/1892151849 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd822d9d640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd824162970 0x7fd8240cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.939+0000 7fd821d9b640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd824161cf0 0x7fd824153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.940+0000 7fd82259c640 1 -- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd824162970 msgr2=0x7fd8240cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.940+0000 7fd82259c640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd824162970 0x7fd8240cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.940+0000 7fd82259c640 1 -- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd824161cf0 msgr2=0x7fd824153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.940+0000 7fd82259c640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd824161cf0 0x7fd824153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.940+0000 7fd82259c640 1 -- 172.21.15.59:0/1892151849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8240cc900 con 0x7fd8240b4460 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.940+0000 7fd82259c640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8240b4460 0x7fd824155980 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00e990 tx=0x7fd81c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.940+0000 7fd81affd640 1 -- 172.21.15.59:0/1892151849 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c017070 con 0x7fd8240b4460 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.940+0000 7fd81affd640 1 -- 172.21.15.59:0/1892151849 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81c00c9e0 con 0x7fd8240b4460 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.940+0000 7fd81affd640 1 -- 172.21.15.59:0/1892151849 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c00cdb0 con 0x7fd8240b4460 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.940+0000 7fd828cdf640 1 -- 172.21.15.59:0/1892151849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8240ccbe0 con 0x7fd8240b4460 2024-03-06T07:27:23.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.940+0000 7fd828cdf640 1 -- 172.21.15.59:0/1892151849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8240b6aa0 con 0x7fd8240b4460 2024-03-06T07:27:23.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.941+0000 7fd828cdf640 1 -- 172.21.15.59:0/1892151849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82404f330 con 0x7fd8240b4460 2024-03-06T07:27:23.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.942+0000 7fd81affd640 1 -- 172.21.15.59:0/1892151849 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd81c01a070 con 0x7fd8240b4460 2024-03-06T07:27:23.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.942+0000 7fd81affd640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd80406b4d0 0x7fd80406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.942+0000 7fd81affd640 1 -- 172.21.15.59:0/1892151849 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd81c08a3b0 con 0x7fd8240b4460 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.942+0000 7fd821d9b640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd80406b4d0 0x7fd80406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.943+0000 7fd821d9b640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd80406b4d0 0x7fd80406d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd8240ca460 tx=0x7fd80c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:23.946+0000 7fd81affd640 1 -- 172.21.15.59:0/1892151849 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd81c058be0 con 0x7fd8240b4460 2024-03-06T07:27:24.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.143+0000 7fd828cdf640 1 -- 172.21.15.59:0/1892151849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd82406b180 con 0x7fd8240b4460 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.144+0000 7fd81affd640 1 -- 172.21.15.59:0/1892151849 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v108) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd81c05cc80 con 0x7fd8240b4460 2024-03-06T07:27:24.146 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:27:24.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 -- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd80406b4d0 msgr2=0x7fd80406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd80406b4d0 0x7fd80406d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd8240ca460 tx=0x7fd80c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:24.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 -- 172.21.15.59:0/1892151849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8240b4460 msgr2=0x7fd824155980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8240b4460 0x7fd824155980 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00e990 tx=0x7fd81c00ee50 comp rx=0 tx=0).stop 2024-03-06T07:27:24.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 -- 172.21.15.59:0/1892151849 shutdown_connections 2024-03-06T07:27:24.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd80406b4d0 0x7fd80406d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd824162970 0x7fd8240cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd824161cf0 0x7fd824153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 --2- 172.21.15.59:0/1892151849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8240b4460 0x7fd824155980 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 -- 172.21.15.59:0/1892151849 >> 172.21.15.59:0/1892151849 conn(0x7fd8240d3ce0 msgr2=0x7fd8240d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 -- 172.21.15.59:0/1892151849 shutdown_connections 2024-03-06T07:27:24.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.146+0000 7fd828cdf640 1 -- 172.21.15.59:0/1892151849 wait complete. 2024-03-06T07:27:24.158 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:27:24.158 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:24.173 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.312+0000 7f5847ced640 1 Processor -- start 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.312+0000 7f5847ced640 1 -- start start 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.313+0000 7f5847ced640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58400d22b0 0x7f58400d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.313+0000 7f5847ced640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58400b3430 con 0x7f584015dfe0 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.313+0000 7f5847ced640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58400b3710 con 0x7f58400b41f0 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.313+0000 7f5847ced640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58400d2c50 con 0x7f58400d22b0 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.313+0000 7f5845288640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58400d22b0 0x7f58400d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.313+0000 7f5845a89640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f584015dfe0 0x7f58400cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:53634/0 (socket says 172.21.15.59:53634) 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.313+0000 7f5845a89640 1 -- 172.21.15.59:0/2930158179 learned_addr learned my addr 172.21.15.59:0/2930158179 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.314+0000 7f5845288640 1 -- 172.21.15.59:0/2930158179 >> v1:172.21.15.164:6789/0 conn(0x7f58400b41f0 legacy=0x7f58400b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.314+0000 7f5845288640 1 -- 172.21.15.59:0/2930158179 >> v1:172.21.15.59:6789/0 conn(0x7f584015dfe0 legacy=0x7f58400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.314+0000 7f5845288640 1 -- 172.21.15.59:0/2930158179 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58400b39f0 con 0x7f58400d22b0 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.314+0000 7f5844a87640 1 -- 172.21.15.59:0/2930158179 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132418240 0 0) 0x7f58400b3430 con 0x7f584015dfe0 2024-03-06T07:27:24.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.314+0000 7f5845288640 1 --2- 172.21.15.59:0/2930158179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400d22b0 0x7f58400d2710 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f58340097d0 tx=0x7f583402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3305adef27a64bb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.315+0000 7f5844a87640 1 -- 172.21.15.59:0/2930158179 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834034070 con 0x7f58400d22b0 2024-03-06T07:27:24.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.315+0000 7f5844a87640 1 -- 172.21.15.59:0/2930158179 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5834030610 con 0x7f58400d22b0 2024-03-06T07:27:24.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.315+0000 7f5844a87640 1 -- 172.21.15.59:0/2930158179 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58340309e0 con 0x7f58400d22b0 2024-03-06T07:27:24.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.315+0000 7f5847ced640 1 -- 172.21.15.59:0/2930158179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400d22b0 msgr2=0x7f58400d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.315+0000 7f5847ced640 1 --2- 172.21.15.59:0/2930158179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400d22b0 0x7f58400d2710 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f58340097d0 tx=0x7f583402b960 comp rx=0 tx=0).stop 2024-03-06T07:27:24.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.315+0000 7f5847ced640 1 -- 172.21.15.59:0/2930158179 shutdown_connections 2024-03-06T07:27:24.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.315+0000 7f5847ced640 1 --2- 172.21.15.59:0/2930158179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400d22b0 0x7f58400d2710 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.315+0000 7f5847ced640 1 -- 172.21.15.59:0/2930158179 >> 172.21.15.59:0/2930158179 conn(0x7f58400d3cd0 msgr2=0x7f58400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.315+0000 7f5847ced640 1 -- 172.21.15.59:0/2930158179 shutdown_connections 2024-03-06T07:27:24.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.315+0000 7f5847ced640 1 -- 172.21.15.59:0/2930158179 wait complete. 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.316+0000 7f5847ced640 1 Processor -- start 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.316+0000 7f5847ced640 1 -- start start 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f5847ced640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58400b41f0 0x7f5840160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f5847ced640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58400d22b0 0x7f58400b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f5847ced640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f584015dfe0 0x7f584015fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f5847ced640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58400b3d60 con 0x7f584015dfe0 2024-03-06T07:27:24.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f5847ced640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58400b31e0 con 0x7f58400b41f0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f5847ced640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58400d1c70 con 0x7f58400d22b0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f584628a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f584015dfe0 0x7f584015fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f584628a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f584015dfe0 0x7f584015fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38420/0 (socket says 172.21.15.59:38420) 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f584628a640 1 -- 172.21.15.59:0/4087274613 learned_addr learned my addr 172.21.15.59:0/4087274613 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f5845a89640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58400b41f0 0x7f5840160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f5845288640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58400d22b0 0x7f58400b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f584628a640 1 -- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58400d22b0 msgr2=0x7f58400b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f584628a640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58400d22b0 0x7f58400b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f584628a640 1 -- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58400b41f0 msgr2=0x7f5840160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f584628a640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58400b41f0 0x7f5840160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.317+0000 7f584628a640 1 -- 172.21.15.59:0/4087274613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584015e910 con 0x7f584015dfe0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.318+0000 7f584628a640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f584015dfe0 0x7f584015fdb0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f5830001eb0 tx=0x7f583000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.318+0000 7f583e7fc640 1 -- 172.21.15.59:0/4087274613 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830017070 con 0x7f584015dfe0 2024-03-06T07:27:24.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.318+0000 7f583e7fc640 1 -- 172.21.15.59:0/4087274613 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58300052d0 con 0x7f584015dfe0 2024-03-06T07:27:24.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.318+0000 7f583e7fc640 1 -- 172.21.15.59:0/4087274613 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830003810 con 0x7f584015dfe0 2024-03-06T07:27:24.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.318+0000 7f5847ced640 1 -- 172.21.15.59:0/4087274613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584015ebf0 con 0x7f584015dfe0 2024-03-06T07:27:24.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.318+0000 7f5847ced640 1 -- 172.21.15.59:0/4087274613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584015f1b0 con 0x7f584015dfe0 2024-03-06T07:27:24.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.319+0000 7f5847ced640 1 -- 172.21.15.59:0/4087274613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f584004f330 con 0x7f584015dfe0 2024-03-06T07:27:24.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.319+0000 7f583e7fc640 1 -- 172.21.15.59:0/4087274613 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f583001a070 con 0x7f584015dfe0 2024-03-06T07:27:24.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.320+0000 7f583e7fc640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f582c06b4d0 0x7f582c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.320+0000 7f583e7fc640 1 -- 172.21.15.59:0/4087274613 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58300894d0 con 0x7f584015dfe0 2024-03-06T07:27:24.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.320+0000 7f5845a89640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f582c06b4d0 0x7f582c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.320+0000 7f5845a89640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f582c06b4d0 0x7f582c06d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f58380012e0 tx=0x7f5838004d40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.324+0000 7f583e7fc640 1 -- 172.21.15.59:0/4087274613 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5830057d00 con 0x7f584015dfe0 2024-03-06T07:27:24.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.508+0000 7f5847ced640 1 -- 172.21.15.59:0/4087274613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58400cea90 con 0x7f584015dfe0 2024-03-06T07:27:24.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.509+0000 7f583e7fc640 1 -- 172.21.15.59:0/4087274613 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16615 (secure 0 0 0) 0x7f5830087e20 con 0x7f584015dfe0 2024-03-06T07:27:24.511 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:24.511 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:27:15.985080+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:12.625799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:13.306376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 -- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f582c06b4d0 msgr2=0x7f582c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f582c06b4d0 0x7f582c06d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f58380012e0 tx=0x7f5838004d40 comp rx=0 tx=0).stop 2024-03-06T07:27:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 -- 172.21.15.59:0/4087274613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f584015dfe0 msgr2=0x7f584015fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f584015dfe0 0x7f584015fdb0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f5830001eb0 tx=0x7f583000c890 comp rx=0 tx=0).stop 2024-03-06T07:27:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 -- 172.21.15.59:0/4087274613 shutdown_connections 2024-03-06T07:27:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f582c06b4d0 0x7f582c06d990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f584015dfe0 0x7f584015fdb0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58400d22b0 0x7f58400b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 --2- 172.21.15.59:0/4087274613 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58400b41f0 0x7f5840160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 -- 172.21.15.59:0/4087274613 >> 172.21.15.59:0/4087274613 conn(0x7f58400d3cd0 msgr2=0x7f584015d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 -- 172.21.15.59:0/4087274613 shutdown_connections 2024-03-06T07:27:24.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.511+0000 7f5847ced640 1 -- 172.21.15.59:0/4087274613 wait complete. 2024-03-06T07:27:24.524 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:12.625799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:13.306376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:24.524 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:24.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.626+0000 7fb365d1e640 1 Processor -- start 2024-03-06T07:27:24.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.627+0000 7fb365d1e640 1 -- start start 2024-03-06T07:27:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.627+0000 7fb365d1e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb3600d06e0 0x7fb36015fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.627+0000 7fb365d1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb3600d0ac0 0x7fb3600b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.627+0000 7fb365d1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3600b4d10 0x7fb3600cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.627+0000 7fb365d1e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3600ced70 con 0x7fb3600d0ac0 2024-03-06T07:27:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.627+0000 7fb365d1e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3600d47f0 con 0x7fb3600d06e0 2024-03-06T07:27:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.627+0000 7fb365d1e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3600d1ab0 con 0x7fb3600b4d10 2024-03-06T07:27:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.627+0000 7fb35effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb3600d0ac0 0x7fb3600b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.627+0000 7fb35effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb3600d0ac0 0x7fb3600b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38430/0 (socket says 172.21.15.59:38430) 2024-03-06T07:27:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.627+0000 7fb35effd640 1 -- 172.21.15.59:0/899833898 learned_addr learned my addr 172.21.15.59:0/899833898 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.627+0000 7fb35ffff640 1 --2- 172.21.15.59:0/899833898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3600b4d10 0x7fb3600cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.628+0000 7fb35f7fe640 1 --2- 172.21.15.59:0/899833898 >> v2:172.21.15.164:3300/0 conn(0x7fb3600d06e0 0x7fb36015fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.628+0000 7fb35effd640 1 -- 172.21.15.59:0/899833898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3600b4d10 msgr2=0x7fb3600cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.628+0000 7fb35effd640 1 --2- 172.21.15.59:0/899833898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3600b4d10 0x7fb3600cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.628+0000 7fb35effd640 1 -- 172.21.15.59:0/899833898 >> v2:172.21.15.164:3300/0 conn(0x7fb3600d06e0 msgr2=0x7fb36015fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.628+0000 7fb35effd640 1 --2- 172.21.15.59:0/899833898 >> v2:172.21.15.164:3300/0 conn(0x7fb3600d06e0 0x7fb36015fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.628+0000 7fb35effd640 1 -- 172.21.15.59:0/899833898 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3600d1d90 con 0x7fb3600d0ac0 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.628+0000 7fb35effd640 1 --2- 172.21.15.59:0/899833898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3600d0ac0 0x7fb3600b47d0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fb3600b6850 tx=0x7fb35002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6245373f5348e241 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb35e7fc640 1 -- 172.21.15.59:0/899833898 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb350035020 con 0x7fb3600d0ac0 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb35e7fc640 1 -- 172.21.15.59:0/899833898 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb350031870 con 0x7fb3600d0ac0 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb35e7fc640 1 -- 172.21.15.59:0/899833898 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb350031c40 con 0x7fb3600d0ac0 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb365d1e640 1 -- 172.21.15.59:0/899833898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3600d0ac0 msgr2=0x7fb3600b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb365d1e640 1 --2- 172.21.15.59:0/899833898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3600d0ac0 0x7fb3600b47d0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fb3600b6850 tx=0x7fb35002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb365d1e640 1 -- 172.21.15.59:0/899833898 shutdown_connections 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb365d1e640 1 --2- 172.21.15.59:0/899833898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3600b4d10 0x7fb3600cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb365d1e640 1 --2- 172.21.15.59:0/899833898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3600d0ac0 0x7fb3600b47d0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb365d1e640 1 --2- 172.21.15.59:0/899833898 >> v2:172.21.15.164:3300/0 conn(0x7fb3600d06e0 0x7fb36015fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb365d1e640 1 -- 172.21.15.59:0/899833898 >> 172.21.15.59:0/899833898 conn(0x7fb3600d3cd0 msgr2=0x7fb3600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb365d1e640 1 -- 172.21.15.59:0/899833898 shutdown_connections 2024-03-06T07:27:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.629+0000 7fb365d1e640 1 -- 172.21.15.59:0/899833898 wait complete. 2024-03-06T07:27:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.630+0000 7fb365d1e640 1 Processor -- start 2024-03-06T07:27:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb365d1e640 1 -- start start 2024-03-06T07:27:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb365d1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3600b4d10 0x7fb3600c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb365d1e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3600d06e0 0x7fb3600b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb365d1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3600d0ac0 0x7fb3600b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb365d1e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3600b44d0 con 0x7fb3600b4d10 2024-03-06T07:27:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb365d1e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3600ce990 con 0x7fb3600d06e0 2024-03-06T07:27:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb365d1e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3600d0f30 con 0x7fb3600d0ac0 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb35f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3600b4d10 0x7fb3600c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb35f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3600b4d10 0x7fb3600c9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38438/0 (socket says 172.21.15.59:38438) 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb35f7fe640 1 -- 172.21.15.59:0/4051594556 learned_addr learned my addr 172.21.15.59:0/4051594556 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb35effd640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3600d06e0 0x7fb3600b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.631+0000 7fb35ffff640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3600d0ac0 0x7fb3600b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.632+0000 7fb35f7fe640 1 -- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3600d0ac0 msgr2=0x7fb3600b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.632+0000 7fb35f7fe640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3600d0ac0 0x7fb3600b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.632+0000 7fb35f7fe640 1 -- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3600d06e0 msgr2=0x7fb3600b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.632+0000 7fb35f7fe640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3600d06e0 0x7fb3600b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.632+0000 7fb35f7fe640 1 -- 172.21.15.59:0/4051594556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3600c85f0 con 0x7fb3600b4d10 2024-03-06T07:27:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.632+0000 7fb35f7fe640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3600b4d10 0x7fb3600c9690 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fb35400e870 tx=0x7fb35400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.632+0000 7fb347fff640 1 -- 172.21.15.59:0/4051594556 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb354017070 con 0x7fb3600b4d10 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.632+0000 7fb347fff640 1 -- 172.21.15.59:0/4051594556 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35400c940 con 0x7fb3600b4d10 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.632+0000 7fb365d1e640 1 -- 172.21.15.59:0/4051594556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3600c88d0 con 0x7fb3600b4d10 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.632+0000 7fb347fff640 1 -- 172.21.15.59:0/4051594556 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35400cd30 con 0x7fb3600b4d10 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.632+0000 7fb365d1e640 1 -- 172.21.15.59:0/4051594556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3600c9170 con 0x7fb3600b4d10 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.633+0000 7fb365d1e640 1 -- 172.21.15.59:0/4051594556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3600a1c80 con 0x7fb3600b4d10 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.634+0000 7fb347fff640 1 -- 172.21.15.59:0/4051594556 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb35401a070 con 0x7fb3600b4d10 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.634+0000 7fb347fff640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb34c06b4d0 0x7fb34c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.634+0000 7fb347fff640 1 -- 172.21.15.59:0/4051594556 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb35408a3d0 con 0x7fb3600b4d10 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.634+0000 7fb35effd640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb34c06b4d0 0x7fb34c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.635+0000 7fb35effd640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb34c06b4d0 0x7fb34c06d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb350036000 tx=0x7fb350008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.638+0000 7fb347fff640 1 -- 172.21.15.59:0/4051594556 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb354058c00 con 0x7fb3600b4d10 2024-03-06T07:27:24.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.818+0000 7fb365d1e640 1 -- 172.21.15.59:0/4051594556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb360161b40 con 0x7fb3600b4d10 2024-03-06T07:27:24.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.818+0000 7fb347fff640 1 -- 172.21.15.59:0/4051594556 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb35405cca0 con 0x7fb3600b4d10 2024-03-06T07:27:24.820 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:27:24.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 -- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb34c06b4d0 msgr2=0x7fb34c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb34c06b4d0 0x7fb34c06d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb350036000 tx=0x7fb350008030 comp rx=0 tx=0).stop 2024-03-06T07:27:24.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 -- 172.21.15.59:0/4051594556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3600b4d10 msgr2=0x7fb3600c9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3600b4d10 0x7fb3600c9690 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fb35400e870 tx=0x7fb35400ed30 comp rx=0 tx=0).stop 2024-03-06T07:27:24.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 -- 172.21.15.59:0/4051594556 shutdown_connections 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb34c06b4d0 0x7fb34c06d990 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3600d0ac0 0x7fb3600b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3600d06e0 0x7fb3600b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 --2- 172.21.15.59:0/4051594556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3600b4d10 0x7fb3600c9690 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 -- 172.21.15.59:0/4051594556 >> 172.21.15.59:0/4051594556 conn(0x7fb3600d3cd0 msgr2=0x7fb3600d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 -- 172.21.15.59:0/4051594556 shutdown_connections 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.820+0000 7fb365d1e640 1 -- 172.21.15.59:0/4051594556 wait complete. 2024-03-06T07:27:24.833 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:24.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.936+0000 7fadbf2af640 1 Processor -- start 2024-03-06T07:27:24.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.936+0000 7fadbf2af640 1 -- start start 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadbf2af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadb815def0 0x7fadb80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadbf2af640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb806d030 con 0x7fadb80cea50 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadbf2af640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb806d310 con 0x7fadb806cab0 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadbf2af640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb80cf1c0 con 0x7fadb815def0 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadbd84c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fadb80cea50 0x7fadb80ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:53648/0 (socket says 172.21.15.59:53648) 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadbd84c640 1 -- 172.21.15.59:0/1381062488 learned_addr learned my addr 172.21.15.59:0/1381062488 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadbd04b640 1 --2- 172.21.15.59:0/1381062488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadb815def0 0x7fadb80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadaffff640 1 -- 172.21.15.59:0/1381062488 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150275199 0 0) 0x7fadb806d030 con 0x7fadb80cea50 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadaffff640 1 -- 172.21.15.59:0/1381062488 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada4003520 con 0x7fadb80cea50 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadaffff640 1 -- 172.21.15.59:0/1381062488 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204357443 0 0) 0x7fadb806d310 con 0x7fadb806cab0 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadaffff640 1 -- 172.21.15.59:0/1381062488 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb806d030 con 0x7fadb806cab0 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadbd04b640 1 -- 172.21.15.59:0/1381062488 >> v1:172.21.15.164:6789/0 conn(0x7fadb806cab0 legacy=0x7fadb80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadbd04b640 1 -- 172.21.15.59:0/1381062488 >> v1:172.21.15.59:6789/0 conn(0x7fadb80cea50 legacy=0x7fadb80ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.937+0000 7fadbd04b640 1 -- 172.21.15.59:0/1381062488 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb80cc110 con 0x7fadb815def0 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.938+0000 7fadbd04b640 1 --2- 172.21.15.59:0/1381062488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadb815def0 0x7fadb80b41b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fadb40097d0 tx=0x7fadb402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66d0f79da71eddbc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.938+0000 7fadaffff640 1 -- 172.21.15.59:0/1381062488 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb4034070 con 0x7fadb815def0 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.938+0000 7fadaffff640 1 -- 172.21.15.59:0/1381062488 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb4030880 con 0x7fadb815def0 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.938+0000 7fadaffff640 1 -- 172.21.15.59:0/1381062488 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb4030c30 con 0x7fadb815def0 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.939+0000 7fadbf2af640 1 -- 172.21.15.59:0/1381062488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadb815def0 msgr2=0x7fadb80b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.939+0000 7fadbf2af640 1 --2- 172.21.15.59:0/1381062488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadb815def0 0x7fadb80b41b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fadb40097d0 tx=0x7fadb402b9c0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.939+0000 7fadbf2af640 1 -- 172.21.15.59:0/1381062488 shutdown_connections 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.939+0000 7fadbf2af640 1 --2- 172.21.15.59:0/1381062488 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadb815def0 0x7fadb80b41b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.939+0000 7fadbf2af640 1 -- 172.21.15.59:0/1381062488 >> 172.21.15.59:0/1381062488 conn(0x7fadb80b3690 msgr2=0x7fadb80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.939+0000 7fadbf2af640 1 -- 172.21.15.59:0/1381062488 shutdown_connections 2024-03-06T07:27:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.939+0000 7fadbf2af640 1 -- 172.21.15.59:0/1381062488 wait complete. 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.940+0000 7fadbf2af640 1 Processor -- start 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.940+0000 7fadbf2af640 1 -- start start 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbf2af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb806cab0 0x7fadb815d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbf2af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadb80cea50 0x7fadb80b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbf2af640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadb815def0 0x7fadb80b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbf2af640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb815e2d0 con 0x7fadb806cab0 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbf2af640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb80cc760 con 0x7fadb815def0 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbf2af640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb815e450 con 0x7fadb80cea50 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbd04b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb806cab0 0x7fadb815d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbd04b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb806cab0 0x7fadb815d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38452/0 (socket says 172.21.15.59:38452) 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbd04b640 1 -- 172.21.15.59:0/4139225256 learned_addr learned my addr 172.21.15.59:0/4139225256 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbc84a640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadb80cea50 0x7fadb80b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbd84c640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadb815def0 0x7fadb80b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbd04b640 1 -- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadb80cea50 msgr2=0x7fadb80b5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbd04b640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadb80cea50 0x7fadb80b5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbd04b640 1 -- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadb815def0 msgr2=0x7fadb80b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbd04b640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadb815def0 0x7fadb80b5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbd04b640 1 -- 172.21.15.59:0/4139225256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb80b6330 con 0x7fadb806cab0 2024-03-06T07:27:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbd04b640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb806cab0 0x7fadb815d130 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fadb4000c00 tx=0x7fadb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadadffb640 1 -- 172.21.15.59:0/4139225256 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb4047070 con 0x7fadb806cab0 2024-03-06T07:27:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadadffb640 1 -- 172.21.15.59:0/4139225256 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb4030680 con 0x7fadb806cab0 2024-03-06T07:27:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadbf2af640 1 -- 172.21.15.59:0/4139225256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb80b6610 con 0x7fadb806cab0 2024-03-06T07:27:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.941+0000 7fadadffb640 1 -- 172.21.15.59:0/4139225256 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb4034040 con 0x7fadb806cab0 2024-03-06T07:27:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.942+0000 7fadbf2af640 1 -- 172.21.15.59:0/4139225256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb80b6bd0 con 0x7fadb806cab0 2024-03-06T07:27:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.942+0000 7fadbf2af640 1 -- 172.21.15.59:0/4139225256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb809fcf0 con 0x7fadb806cab0 2024-03-06T07:27:24.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.943+0000 7fadadffb640 1 -- 172.21.15.59:0/4139225256 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fadb4037070 con 0x7fadb806cab0 2024-03-06T07:27:24.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.943+0000 7fadadffb640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9c06b500 0x7fad9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.943+0000 7fadadffb640 1 -- 172.21.15.59:0/4139225256 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadb40abac0 con 0x7fadb806cab0 2024-03-06T07:27:24.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.944+0000 7fadbc84a640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9c06b500 0x7fad9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.944+0000 7fadbc84a640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9c06b500 0x7fad9c06d9c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fada8007fd0 tx=0x7fada8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:24.948+0000 7fadadffb640 1 -- 172.21.15.59:0/4139225256 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fadb407a260 con 0x7fadb806cab0 2024-03-06T07:27:25.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.128+0000 7fadbf2af640 1 -- 172.21.15.59:0/4139225256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fadb80cbdf0 con 0x7fadb806cab0 2024-03-06T07:27:25.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.128+0000 7fadadffb640 1 -- 172.21.15.59:0/4139225256 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) v1 ==== 104+0+11 (secure 0 0 0) 0x7fadb407e300 con 0x7fadb806cab0 2024-03-06T07:27:25.131 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:27:25.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 -- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9c06b500 msgr2=0x7fad9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9c06b500 0x7fad9c06d9c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fada8007fd0 tx=0x7fada8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:25.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 -- 172.21.15.59:0/4139225256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb806cab0 msgr2=0x7fadb815d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb806cab0 0x7fadb815d130 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fadb4000c00 tx=0x7fadb40046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 -- 172.21.15.59:0/4139225256 shutdown_connections 2024-03-06T07:27:25.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9c06b500 0x7fad9c06d9c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadb815def0 0x7fadb80b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadb80cea50 0x7fadb80b5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 --2- 172.21.15.59:0/4139225256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb806cab0 0x7fadb815d130 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 -- 172.21.15.59:0/4139225256 >> 172.21.15.59:0/4139225256 conn(0x7fadb80b3690 msgr2=0x7fadb806f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 -- 172.21.15.59:0/4139225256 shutdown_connections 2024-03-06T07:27:25.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.131+0000 7fadbf2af640 1 -- 172.21.15.59:0/4139225256 wait complete. 2024-03-06T07:27:25.143 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:25.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.246+0000 7fca60972640 1 Processor -- start 2024-03-06T07:27:25.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.246+0000 7fca60972640 1 -- start start 2024-03-06T07:27:25.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca60972640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fca5c15def0 0x7fca5c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca60972640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca5c06cab0 0x7fca5c0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca60972640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fca5c0ce880 0x7fca5c0cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca60972640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca5c0cecf0 con 0x7fca5c15def0 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca60972640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca5c0cf1c0 con 0x7fca5c0ce880 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca60972640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c0cc760 con 0x7fca5c06cab0 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca5a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fca5c15def0 0x7fca5c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca5a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fca5c15def0 0x7fca5c0b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38466/0 (socket says 172.21.15.59:38466) 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca5a59c640 1 -- 172.21.15.59:0/3788195225 learned_addr learned my addr 172.21.15.59:0/3788195225 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca59d9b640 1 --2- 172.21.15.59:0/3788195225 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca5c06cab0 0x7fca5c0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca5ad9d640 1 --2- 172.21.15.59:0/3788195225 >> v2:172.21.15.164:3300/0 conn(0x7fca5c0ce880 0x7fca5c0cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.247+0000 7fca5a59c640 1 -- 172.21.15.59:0/3788195225 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca5c06cab0 msgr2=0x7fca5c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.248+0000 7fca5a59c640 1 --2- 172.21.15.59:0/3788195225 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca5c06cab0 0x7fca5c0ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.248+0000 7fca5a59c640 1 -- 172.21.15.59:0/3788195225 >> v2:172.21.15.164:3300/0 conn(0x7fca5c0ce880 msgr2=0x7fca5c0cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.248+0000 7fca5a59c640 1 --2- 172.21.15.59:0/3788195225 >> v2:172.21.15.164:3300/0 conn(0x7fca5c0ce880 0x7fca5c0cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.248+0000 7fca5a59c640 1 -- 172.21.15.59:0/3788195225 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c06d030 con 0x7fca5c15def0 2024-03-06T07:27:25.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.248+0000 7fca5a59c640 1 --2- 172.21.15.59:0/3788195225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5c15def0 0x7fca5c0b41b0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fca5c071e70 tx=0x7fca5402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6698c3c5d48995a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.248+0000 7fca5959a640 1 -- 172.21.15.59:0/3788195225 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54034070 con 0x7fca5c15def0 2024-03-06T07:27:25.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.248+0000 7fca5959a640 1 -- 172.21.15.59:0/3788195225 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca54030860 con 0x7fca5c15def0 2024-03-06T07:27:25.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.248+0000 7fca5959a640 1 -- 172.21.15.59:0/3788195225 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54030c30 con 0x7fca5c15def0 2024-03-06T07:27:25.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.249+0000 7fca60972640 1 -- 172.21.15.59:0/3788195225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5c15def0 msgr2=0x7fca5c0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.249+0000 7fca60972640 1 --2- 172.21.15.59:0/3788195225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5c15def0 0x7fca5c0b41b0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fca5c071e70 tx=0x7fca5402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.249+0000 7fca60972640 1 -- 172.21.15.59:0/3788195225 shutdown_connections 2024-03-06T07:27:25.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.249+0000 7fca60972640 1 --2- 172.21.15.59:0/3788195225 >> v2:172.21.15.164:3300/0 conn(0x7fca5c0ce880 0x7fca5c0cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.249+0000 7fca60972640 1 --2- 172.21.15.59:0/3788195225 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca5c06cab0 0x7fca5c0ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.249+0000 7fca60972640 1 --2- 172.21.15.59:0/3788195225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5c15def0 0x7fca5c0b41b0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.249+0000 7fca60972640 1 -- 172.21.15.59:0/3788195225 >> 172.21.15.59:0/3788195225 conn(0x7fca5c0b3690 msgr2=0x7fca5c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.249+0000 7fca60972640 1 -- 172.21.15.59:0/3788195225 shutdown_connections 2024-03-06T07:27:25.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.249+0000 7fca60972640 1 -- 172.21.15.59:0/3788195225 wait complete. 2024-03-06T07:27:25.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.250+0000 7fca60972640 1 Processor -- start 2024-03-06T07:27:25.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.250+0000 7fca60972640 1 -- start start 2024-03-06T07:27:25.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca60972640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca5c06cab0 0x7fca5c0b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca60972640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca5c0ce880 0x7fca5c14bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca60972640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5c15def0 0x7fca5c14c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca60972640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c0cc3f0 con 0x7fca5c15def0 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca60972640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c0cc110 con 0x7fca5c0ce880 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca60972640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c15e2d0 con 0x7fca5c06cab0 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca5ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5c15def0 0x7fca5c14c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca5ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5c15def0 0x7fca5c14c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38468/0 (socket says 172.21.15.59:38468) 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca5ad9d640 1 -- 172.21.15.59:0/980428343 learned_addr learned my addr 172.21.15.59:0/980428343 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca59d9b640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca5c0ce880 0x7fca5c14bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca5a59c640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca5c06cab0 0x7fca5c0b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca5ad9d640 1 -- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca5c06cab0 msgr2=0x7fca5c0b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca5ad9d640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca5c06cab0 0x7fca5c0b3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca5ad9d640 1 -- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca5c0ce880 msgr2=0x7fca5c14bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.251+0000 7fca5ad9d640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca5c0ce880 0x7fca5c14bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.252+0000 7fca5ad9d640 1 -- 172.21.15.59:0/980428343 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c0b7620 con 0x7fca5c15def0 2024-03-06T07:27:25.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.252+0000 7fca5ad9d640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5c15def0 0x7fca5c14c010 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00efc0 tx=0x7fca4c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.252+0000 7fca52ffd640 1 -- 172.21.15.59:0/980428343 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c018020 con 0x7fca5c15def0 2024-03-06T07:27:25.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.252+0000 7fca52ffd640 1 -- 172.21.15.59:0/980428343 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4c014480 con 0x7fca5c15def0 2024-03-06T07:27:25.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.252+0000 7fca60972640 1 -- 172.21.15.59:0/980428343 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca5c0ca780 con 0x7fca5c15def0 2024-03-06T07:27:25.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.252+0000 7fca52ffd640 1 -- 172.21.15.59:0/980428343 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c014870 con 0x7fca5c15def0 2024-03-06T07:27:25.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.252+0000 7fca60972640 1 -- 172.21.15.59:0/980428343 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca5c0cad40 con 0x7fca5c15def0 2024-03-06T07:27:25.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.253+0000 7fca60972640 1 -- 172.21.15.59:0/980428343 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5c0d19c0 con 0x7fca5c15def0 2024-03-06T07:27:25.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.253+0000 7fca52ffd640 1 -- 172.21.15.59:0/980428343 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca4c005d10 con 0x7fca5c15def0 2024-03-06T07:27:25.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.254+0000 7fca52ffd640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3c06b4d0 0x7fca3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.254+0000 7fca52ffd640 1 -- 172.21.15.59:0/980428343 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca4c08aa40 con 0x7fca5c15def0 2024-03-06T07:27:25.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.254+0000 7fca5a59c640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3c06b4d0 0x7fca3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.255+0000 7fca5a59c640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3c06b4d0 0x7fca3c06d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fca54036000 tx=0x7fca5403b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.258+0000 7fca52ffd640 1 -- 172.21.15.59:0/980428343 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca5c0d19c0 con 0x7fca5c15def0 2024-03-06T07:27:25.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.439+0000 7fca60972640 1 -- 172.21.15.59:0/980428343 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fca5c15d710 con 0x7fca5c15def0 2024-03-06T07:27:25.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.439+0000 7fca52ffd640 1 -- 172.21.15.59:0/980428343 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) v1 ==== 100+0+11 (secure 0 0 0) 0x7fca4c08ae20 con 0x7fca5c15def0 2024-03-06T07:27:25.441 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 -- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3c06b4d0 msgr2=0x7fca3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3c06b4d0 0x7fca3c06d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fca54036000 tx=0x7fca5403b040 comp rx=0 tx=0).stop 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 -- 172.21.15.59:0/980428343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5c15def0 msgr2=0x7fca5c14c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5c15def0 0x7fca5c14c010 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00efc0 tx=0x7fca4c00c490 comp rx=0 tx=0).stop 2024-03-06T07:27:25.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 -- 172.21.15.59:0/980428343 shutdown_connections 2024-03-06T07:27:25.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca3c06b4d0 0x7fca3c06d990 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca5c15def0 0x7fca5c14c010 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca5c0ce880 0x7fca5c14bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 --2- 172.21.15.59:0/980428343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca5c06cab0 0x7fca5c0b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 -- 172.21.15.59:0/980428343 >> 172.21.15.59:0/980428343 conn(0x7fca5c0b3690 msgr2=0x7fca5c0d12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 -- 172.21.15.59:0/980428343 shutdown_connections 2024-03-06T07:27:25.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.441+0000 7fca60972640 1 -- 172.21.15.59:0/980428343 wait complete. 2024-03-06T07:27:25.454 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:27:25.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.554+0000 7f65ce53e640 1 Processor -- start 2024-03-06T07:27:25.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.554+0000 7f65ce53e640 1 -- start start 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.555+0000 7f65ce53e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65c806d3c0 0x7f65c80b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.555+0000 7f65ce53e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f65c80b1850 0x7f65c80b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.555+0000 7f65ce53e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c80af4b0 con 0x7f65c81614a0 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.555+0000 7f65ce53e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c806d800 con 0x7f65c80b1850 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.555+0000 7f65ce53e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80af790 con 0x7f65c806d3c0 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.555+0000 7f65ccadb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f65c80b1850 0x7f65c80b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.555+0000 7f65ccadb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f65c80b1850 0x7f65c80b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38470/0 (socket says 172.21.15.59:38470) 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.555+0000 7f65ccadb640 1 -- 172.21.15.59:0/2772225969 learned_addr learned my addr 172.21.15.59:0/2772225969 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.555+0000 7f65c77fe640 1 --2- 172.21.15.59:0/2772225969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65c806d3c0 0x7f65c80b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.556+0000 7f65ccadb640 1 -- 172.21.15.59:0/2772225969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65c806d3c0 msgr2=0x7f65c80b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.556+0000 7f65ccadb640 1 --2- 172.21.15.59:0/2772225969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65c806d3c0 0x7f65c80b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.556+0000 7f65ccadb640 1 -- 172.21.15.59:0/2772225969 >> v1:172.21.15.164:6789/0 conn(0x7f65c81614a0 legacy=0x7f65c80b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.556+0000 7f65c6ffd640 1 -- 172.21.15.59:0/2772225969 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088548989 0 0) 0x7f65c80af4b0 con 0x7f65c81614a0 2024-03-06T07:27:25.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.556+0000 7f65ccadb640 1 -- 172.21.15.59:0/2772225969 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c80afa70 con 0x7f65c80b1850 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.556+0000 7f65ccadb640 1 --2- 172.21.15.59:0/2772225969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65c80b1850 0x7f65c80b5510 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f65c0009e50 tx=0x7f65c002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52beed899ef3f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.557+0000 7f65c6ffd640 1 -- 172.21.15.59:0/2772225969 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c002cb00 con 0x7f65c80b1850 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.557+0000 7f65c6ffd640 1 -- 172.21.15.59:0/2772225969 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c002cca0 con 0x7f65c80b1850 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.557+0000 7f65c6ffd640 1 -- 172.21.15.59:0/2772225969 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0034610 con 0x7f65c80b1850 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.557+0000 7f65ce53e640 1 -- 172.21.15.59:0/2772225969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65c80b1850 msgr2=0x7f65c80b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.557+0000 7f65ce53e640 1 --2- 172.21.15.59:0/2772225969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65c80b1850 0x7f65c80b5510 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f65c0009e50 tx=0x7f65c002bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.557+0000 7f65ce53e640 1 -- 172.21.15.59:0/2772225969 shutdown_connections 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.557+0000 7f65ce53e640 1 --2- 172.21.15.59:0/2772225969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65c80b1850 0x7f65c80b5510 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.557+0000 7f65ce53e640 1 --2- 172.21.15.59:0/2772225969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65c806d3c0 0x7f65c80b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.557+0000 7f65ce53e640 1 -- 172.21.15.59:0/2772225969 >> 172.21.15.59:0/2772225969 conn(0x7f65c809a9f0 msgr2=0x7f65c809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.557+0000 7f65ce53e640 1 -- 172.21.15.59:0/2772225969 shutdown_connections 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.557+0000 7f65ce53e640 1 -- 172.21.15.59:0/2772225969 wait complete. 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.558+0000 7f65ce53e640 1 Processor -- start 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65ce53e640 1 -- start start 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65ce53e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65c806d3c0 0x7f65c80d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65ce53e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65c80b1850 0x7f65c80d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65ce53e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65c81614a0 0x7f65c814cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65ce53e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80b1c80 con 0x7f65c81614a0 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65ce53e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c8161880 con 0x7f65c806d3c0 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65ce53e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80b1e00 con 0x7f65c80b1850 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65ccadb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65c81614a0 0x7f65c814cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65ccadb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65c81614a0 0x7f65c814cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38474/0 (socket says 172.21.15.59:38474) 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65ccadb640 1 -- 172.21.15.59:0/3180985735 learned_addr learned my addr 172.21.15.59:0/3180985735 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65c77fe640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65c80b1850 0x7f65c80d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.559+0000 7f65c7fff640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65c806d3c0 0x7f65c80d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.560+0000 7f65ccadb640 1 -- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65c80b1850 msgr2=0x7f65c80d10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.560+0000 7f65ccadb640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65c80b1850 0x7f65c80d10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.560+0000 7f65ccadb640 1 -- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65c806d3c0 msgr2=0x7f65c80d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.560+0000 7f65ccadb640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65c806d3c0 0x7f65c80d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.560+0000 7f65ccadb640 1 -- 172.21.15.59:0/3180985735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c80cf040 con 0x7f65c81614a0 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.560+0000 7f65ccadb640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65c81614a0 0x7f65c814cef0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f65c0036040 tx=0x7f65c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.560+0000 7f65c4ff9640 1 -- 172.21.15.59:0/3180985735 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0043020 con 0x7f65c81614a0 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.560+0000 7f65c4ff9640 1 -- 172.21.15.59:0/3180985735 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c002ce50 con 0x7f65c81614a0 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.560+0000 7f65ce53e640 1 -- 172.21.15.59:0/3180985735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c80cf320 con 0x7f65c81614a0 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.560+0000 7f65c4ff9640 1 -- 172.21.15.59:0/3180985735 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0009510 con 0x7f65c81614a0 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.560+0000 7f65ce53e640 1 -- 172.21.15.59:0/3180985735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c80cf8e0 con 0x7f65c81614a0 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.561+0000 7f65ce53e640 1 -- 172.21.15.59:0/3180985735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c804f330 con 0x7f65c81614a0 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.561+0000 7f65c4ff9640 1 -- 172.21.15.59:0/3180985735 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65c0005d10 con 0x7f65c81614a0 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.562+0000 7f65c4ff9640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65b406b4d0 0x7f65b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.562+0000 7f65c4ff9640 1 -- 172.21.15.59:0/3180985735 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65c00ac960 con 0x7f65c81614a0 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.562+0000 7f65c7fff640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65b406b4d0 0x7f65b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.562+0000 7f65c7fff640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65b406b4d0 0x7f65b406d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f65b8000f60 tx=0x7f65b8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.566+0000 7f65c4ff9640 1 -- 172.21.15.59:0/3180985735 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65c007b190 con 0x7f65c81614a0 2024-03-06T07:27:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.762+0000 7f65ce53e640 1 -- 172.21.15.59:0/3180985735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f65c806e8e0 con 0x7f65c81614a0 2024-03-06T07:27:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.762+0000 7f65c4ff9640 1 -- 172.21.15.59:0/3180985735 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v108) v1 ==== 80+0+0 (secure 0 0 0) 0x7f65c007f230 con 0x7f65c81614a0 2024-03-06T07:27:25.764 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:27:25.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 -- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65b406b4d0 msgr2=0x7f65b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65b406b4d0 0x7f65b406d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f65b8000f60 tx=0x7f65b8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 -- 172.21.15.59:0/3180985735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65c81614a0 msgr2=0x7f65c814cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65c81614a0 0x7f65c814cef0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f65c0036040 tx=0x7f65c0004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 -- 172.21.15.59:0/3180985735 shutdown_connections 2024-03-06T07:27:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f65b406b4d0 0x7f65b406d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65c81614a0 0x7f65c814cef0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65c80b1850 0x7f65c80d10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 --2- 172.21.15.59:0/3180985735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65c806d3c0 0x7f65c80d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 -- 172.21.15.59:0/3180985735 >> 172.21.15.59:0/3180985735 conn(0x7f65c809a9f0 msgr2=0x7f65c8160540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 -- 172.21.15.59:0/3180985735 shutdown_connections 2024-03-06T07:27:25.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:25.764+0000 7f65ce53e640 1 -- 172.21.15.59:0/3180985735 wait complete. 2024-03-06T07:27:25.777 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:27:25.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.778 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:25.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[72756]: fuse finished with error 0 and tester_r 0 2024-03-06T07:27:25.806 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:25.877 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:27:25.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:25.877 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:27:25.892 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:26.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.031+0000 7ff50ad5e640 1 Processor -- start 2024-03-06T07:27:26.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.031+0000 7ff50ad5e640 1 -- start start 2024-03-06T07:27:26.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff50ad5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50406c6d0 0x7ff50415ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff50ad5e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff504066d90 0x7ff5040ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff50ad5e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5040cd450 con 0x7ff5040b4f70 2024-03-06T07:27:26.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff50ad5e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50415e9f0 con 0x7ff504066d90 2024-03-06T07:27:26.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff50ad5e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040cda10 con 0x7ff50406c6d0 2024-03-06T07:27:26.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff4fbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50406c6d0 0x7ff50415ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff5092fb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff504066d90 0x7ff5040ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff5092fb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff504066d90 0x7ff5040ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:58154/0 (socket says 172.21.15.59:58154) 2024-03-06T07:27:26.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff4fbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50406c6d0 0x7ff50415ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:40708/0 (socket says 172.21.15.59:40708) 2024-03-06T07:27:26.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff5092fb640 1 -- 172.21.15.59:0/920660183 learned_addr learned my addr 172.21.15.59:0/920660183 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:26.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff508afa640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff5040b4f70 0x7ff5040b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:43210/0 (socket says 172.21.15.59:43210) 2024-03-06T07:27:26.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff5092fb640 1 -- 172.21.15.59:0/920660183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50406c6d0 msgr2=0x7ff50415ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff5092fb640 1 --2- 172.21.15.59:0/920660183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50406c6d0 0x7ff50415ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff5092fb640 1 -- 172.21.15.59:0/920660183 >> v1:172.21.15.164:6789/0 conn(0x7ff5040b4f70 legacy=0x7ff5040b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff5092fb640 1 -- 172.21.15.59:0/920660183 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5040cd730 con 0x7ff504066d90 2024-03-06T07:27:26.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.032+0000 7ff4fbfff640 1 --2- 172.21.15.59:0/920660183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50406c6d0 0x7ff50415ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:26.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.033+0000 7ff5092fb640 1 --2- 172.21.15.59:0/920660183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff504066d90 0x7ff5040ccb50 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009e50 tx=0x7ff4fc02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adee690a46ad3a7d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.033+0000 7ff4fb7fe640 1 -- 172.21.15.59:0/920660183 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc02cb00 con 0x7ff504066d90 2024-03-06T07:27:26.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.033+0000 7ff4fb7fe640 1 -- 172.21.15.59:0/920660183 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc02cca0 con 0x7ff504066d90 2024-03-06T07:27:26.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.033+0000 7ff4fb7fe640 1 -- 172.21.15.59:0/920660183 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc034610 con 0x7ff504066d90 2024-03-06T07:27:26.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.034+0000 7ff50ad5e640 1 -- 172.21.15.59:0/920660183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff504066d90 msgr2=0x7ff5040ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.034+0000 7ff50ad5e640 1 --2- 172.21.15.59:0/920660183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff504066d90 0x7ff5040ccb50 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009e50 tx=0x7ff4fc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:26.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.034+0000 7ff50ad5e640 1 -- 172.21.15.59:0/920660183 shutdown_connections 2024-03-06T07:27:26.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.034+0000 7ff50ad5e640 1 --2- 172.21.15.59:0/920660183 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff504066d90 0x7ff5040ccb50 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.034+0000 7ff50ad5e640 1 --2- 172.21.15.59:0/920660183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50406c6d0 0x7ff50415ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.034+0000 7ff50ad5e640 1 -- 172.21.15.59:0/920660183 >> 172.21.15.59:0/920660183 conn(0x7ff5040724d0 msgr2=0x7ff5040728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.034+0000 7ff50ad5e640 1 -- 172.21.15.59:0/920660183 shutdown_connections 2024-03-06T07:27:26.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.034+0000 7ff50ad5e640 1 -- 172.21.15.59:0/920660183 wait complete. 2024-03-06T07:27:26.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.035+0000 7ff50ad5e640 1 Processor -- start 2024-03-06T07:27:26.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.035+0000 7ff50ad5e640 1 -- start start 2024-03-06T07:27:26.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff50ad5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff504066d90 0x7ff50415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff50ad5e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5040b4f70 0x7ff50415da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff50ad5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50415df80 0x7ff5040ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff50ad5e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040cd090 con 0x7ff504066d90 2024-03-06T07:27:26.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff50ad5e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50415f3c0 con 0x7ff5040b4f70 2024-03-06T07:27:26.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff50ad5e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040cd210 con 0x7ff50415df80 2024-03-06T07:27:26.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff508afa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff504066d90 0x7ff50415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff508afa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff504066d90 0x7ff50415a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38482/0 (socket says 172.21.15.59:38482) 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff508afa640 1 -- 172.21.15.59:0/4120196495 learned_addr learned my addr 172.21.15.59:0/4120196495 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff4fbfff640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5040b4f70 0x7ff50415da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff5092fb640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50415df80 0x7ff5040ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff508afa640 1 -- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50415df80 msgr2=0x7ff5040ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff508afa640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50415df80 0x7ff5040ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff508afa640 1 -- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5040b4f70 msgr2=0x7ff50415da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff508afa640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5040b4f70 0x7ff50415da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff508afa640 1 -- 172.21.15.59:0/4120196495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5040cf5a0 con 0x7ff504066d90 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff4fbfff640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5040b4f70 0x7ff50415da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff508afa640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff504066d90 0x7ff50415a7a0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7ff50000c400 tx=0x7ff50000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff4f97fa640 1 -- 172.21.15.59:0/4120196495 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5000074e0 con 0x7ff504066d90 2024-03-06T07:27:26.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff4f97fa640 1 -- 172.21.15.59:0/4120196495 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff500007680 con 0x7ff504066d90 2024-03-06T07:27:26.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff50ad5e640 1 -- 172.21.15.59:0/4120196495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5040cf880 con 0x7ff504066d90 2024-03-06T07:27:26.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff4f97fa640 1 -- 172.21.15.59:0/4120196495 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500004a00 con 0x7ff504066d90 2024-03-06T07:27:26.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.036+0000 7ff50ad5e640 1 -- 172.21.15.59:0/4120196495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5040cfe40 con 0x7ff504066d90 2024-03-06T07:27:26.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.037+0000 7ff50ad5e640 1 -- 172.21.15.59:0/4120196495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50404f330 con 0x7ff504066d90 2024-03-06T07:27:26.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.038+0000 7ff4f97fa640 1 -- 172.21.15.59:0/4120196495 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff50001a020 con 0x7ff504066d90 2024-03-06T07:27:26.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.038+0000 7ff4f97fa640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff4f006b4d0 0x7ff4f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.038+0000 7ff4f97fa640 1 -- 172.21.15.59:0/4120196495 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff500089860 con 0x7ff504066d90 2024-03-06T07:27:26.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.038+0000 7ff4fbfff640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff4f006b4d0 0x7ff4f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.039+0000 7ff4fbfff640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff4f006b4d0 0x7ff4f006d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff5040ca990 tx=0x7ff4f4009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.042+0000 7ff4f97fa640 1 -- 172.21.15.59:0/4120196495 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff500058120 con 0x7ff504066d90 2024-03-06T07:27:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.263+0000 7ff50ad5e640 1 -- 172.21.15.59:0/4120196495 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5040b1860 con 0x7ff504066d90 2024-03-06T07:27:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.263+0000 7ff4f97fa640 1 -- 172.21.15.59:0/4120196495 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4979 (secure 0 0 0) 0x7ff500089020 con 0x7ff504066d90 2024-03-06T07:27:26.266 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:26.266 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3012766532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3012766532},{"type":"v1","addr":"172.21.15.164:6837","nonce":3012766532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7956,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6834/930220384","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":930220384},{"type":"v1","addr":"172.21.15.59:6834","nonce":930220384}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":7959,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/557371287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6833","nonce":557371287},{"type":"v1","addr":"172.21.15.59:6835","nonce":557371287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:16.424559+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7968},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7968":{"gid":7968,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.164:6836/2139041464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2139041464},{"type":"v1","addr":"172.21.15.164:6836","nonce":2139041464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7968,"qdb_cluster":[7968]},"id":8}]} 2024-03-06T07:27:26.266 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 66 2024-03-06T07:27:26.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.265+0000 7ff50ad5e640 1 -- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff4f006b4d0 msgr2=0x7ff4f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.265+0000 7ff50ad5e640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff4f006b4d0 0x7ff4f006d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff5040ca990 tx=0x7ff4f4009190 comp rx=0 tx=0).stop 2024-03-06T07:27:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.265+0000 7ff50ad5e640 1 -- 172.21.15.59:0/4120196495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff504066d90 msgr2=0x7ff50415a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.265+0000 7ff50ad5e640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff504066d90 0x7ff50415a7a0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7ff50000c400 tx=0x7ff50000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.265+0000 7ff50ad5e640 1 -- 172.21.15.59:0/4120196495 shutdown_connections 2024-03-06T07:27:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.265+0000 7ff50ad5e640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff4f006b4d0 0x7ff4f006d990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.265+0000 7ff50ad5e640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff50415df80 0x7ff5040ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.265+0000 7ff50ad5e640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5040b4f70 0x7ff50415da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.265+0000 7ff50ad5e640 1 --2- 172.21.15.59:0/4120196495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff504066d90 0x7ff50415a7a0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.265+0000 7ff50ad5e640 1 -- 172.21.15.59:0/4120196495 >> 172.21.15.59:0/4120196495 conn(0x7ff5040724d0 msgr2=0x7ff50406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.266+0000 7ff50ad5e640 1 -- 172.21.15.59:0/4120196495 shutdown_connections 2024-03-06T07:27:26.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.266+0000 7ff50ad5e640 1 -- 172.21.15.59:0/4120196495 wait complete. 2024-03-06T07:27:26.278 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:27:26.400 INFO:teuthology.orchestra.run.smithi164.stdout:{"message":"","return_code":0} 2024-03-06T07:27:26.400 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:26.401 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:26.401 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:26.401 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:26.401 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:26.402 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:26.402 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:26.402 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:26.402 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:26.410 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:26.410 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:27:26.411 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:26.411 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:27:26.412 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:26.412 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:27:26.413 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:26.413 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:27:26.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.556+0000 7fb429239640 1 Processor -- start 2024-03-06T07:27:26.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.556+0000 7f45f417a640 1 Processor -- start 2024-03-06T07:27:26.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.556+0000 7fd4c358a640 1 Processor -- start 2024-03-06T07:27:26.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.556+0000 7fb429239640 1 -- start start 2024-03-06T07:27:26.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.557+0000 7fd4c358a640 1 -- start start 2024-03-06T07:27:26.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.557+0000 7f45f417a640 1 -- start start 2024-03-06T07:27:26.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.557+0000 7f0718fa0640 1 Processor -- start 2024-03-06T07:27:26.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.557+0000 7f0718fa0640 1 -- start start 2024-03-06T07:27:26.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.557+0000 7fb429239640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb4241444c0 0x7fb42406ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.557+0000 7fb429239640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb42415b2d0 0x7fb4240708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.557+0000 7fb429239640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42406d370 con 0x7fb424070e10 2024-03-06T07:27:26.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.557+0000 7fb429239640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4241448a0 con 0x7fb4241444c0 2024-03-06T07:27:26.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.557+0000 7fb429239640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb424071260 con 0x7fb42415b2d0 2024-03-06T07:27:26.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f417a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec06d000 0x7f45ec06d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f417a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f45ec06da60 0x7f45ec162a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.557+0000 7fd4c358a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd4bc06d3c0 0x7fd4bc0b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c358a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd4bc0b1850 0x7fd4bc0b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c358a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4bc0af4b0 con 0x7fd4bc1614a0 2024-03-06T07:27:26.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c358a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4bc0af790 con 0x7fd4bc0b1850 2024-03-06T07:27:26.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c358a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4bc0afa70 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fb422d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb4241444c0 0x7fb42406ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fb42259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb42415b2d0 0x7fb4240708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fb422d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb4241444c0 0x7fb42406ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:58172/0 (socket says 172.21.15.59:58172) 2024-03-06T07:27:26.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fb422d9d640 1 -- 172.21.15.59:0/1307718581 learned_addr learned my addr 172.21.15.59:0/1307718581 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:26.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f0718fa0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07140b0080 0x7f0714166900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f417a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ec06dea0 con 0x7f45ec15ff70 2024-03-06T07:27:26.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f417a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ec0717e0 con 0x7f45ec06da60 2024-03-06T07:27:26.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f417a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec06e170 con 0x7f45ec06d000 2024-03-06T07:27:26.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c1b27640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd4bc0b1850 0x7fd4bc0b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c1b27640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd4bc0b1850 0x7fd4bc0b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38496/0 (socket says 172.21.15.59:38496) 2024-03-06T07:27:26.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c1b27640 1 -- 172.21.15.59:0/2698861198 learned_addr learned my addr 172.21.15.59:0/2698861198 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:26.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f0718fa0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07140b32b0 con 0x7f07140d1810 2024-03-06T07:27:26.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f0718fa0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07140ae8b0 con 0x7f07140b2030 2024-03-06T07:27:26.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f0718fa0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07140aebe0 con 0x7f07140b0080 2024-03-06T07:27:26.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c0b25640 1 --2- 172.21.15.59:0/2698861198 >> v2:172.21.15.164:3300/0 conn(0x7fd4bc06d3c0 0x7fd4bc0b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f07137fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f07140d1810 0x7f07140d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:53662/0 (socket says 172.21.15.59:53662) 2024-03-06T07:27:26.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f07137fe640 1 -- 172.21.15.59:0/424705687 learned_addr learned my addr 172.21.15.59:0/424705687 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:26.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f1715640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec06d000 0x7f45ec06d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f2717640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f45ec06da60 0x7f45ec162a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f1715640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec06d000 0x7f45ec06d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:40724/0 (socket says 172.21.15.59:40724) 2024-03-06T07:27:26.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f1715640 1 -- 172.21.15.59:0/3767928148 learned_addr learned my addr 172.21.15.59:0/3767928148 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:26.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f0713fff640 1 --2- 172.21.15.59:0/424705687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07140b0080 0x7f0714166900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fb42259c640 1 -- 172.21.15.59:0/1307718581 >> v2:172.21.15.164:3300/0 conn(0x7fb4241444c0 msgr2=0x7fb42406ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fb42259c640 1 --2- 172.21.15.59:0/1307718581 >> v2:172.21.15.164:3300/0 conn(0x7fb4241444c0 0x7fb42406ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fb42259c640 1 -- 172.21.15.59:0/1307718581 >> v1:172.21.15.164:6789/0 conn(0x7fb424070e10 legacy=0x7fb42415e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f1715640 1 -- 172.21.15.59:0/3767928148 >> v2:172.21.15.164:3300/0 conn(0x7f45ec06da60 msgr2=0x7f45ec162a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f1715640 1 --2- 172.21.15.59:0/3767928148 >> v2:172.21.15.164:3300/0 conn(0x7f45ec06da60 0x7f45ec162a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f1715640 1 -- 172.21.15.59:0/3767928148 >> v1:172.21.15.164:6789/0 conn(0x7f45ec15ff70 legacy=0x7f45ec06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fb42259c640 1 -- 172.21.15.59:0/1307718581 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42406d650 con 0x7fb42415b2d0 2024-03-06T07:27:26.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f45f1715640 1 -- 172.21.15.59:0/3767928148 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec165ff0 con 0x7f45ec06d000 2024-03-06T07:27:26.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c0b25640 1 -- 172.21.15.59:0/2698861198 >> v2:172.21.15.59:3300/0 conn(0x7fd4bc0b1850 msgr2=0x7fd4bc0b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c0b25640 1 --2- 172.21.15.59:0/2698861198 >> v2:172.21.15.59:3300/0 conn(0x7fd4bc0b1850 0x7fd4bc0b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c0b25640 1 -- 172.21.15.59:0/2698861198 >> v1:172.21.15.164:6789/0 conn(0x7fd4bc1614a0 legacy=0x7fd4bc0b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fd4c0b25640 1 -- 172.21.15.59:0/2698861198 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4bc0afd50 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7fb422d9d640 1 --2- 172.21.15.59:0/1307718581 >> v2:172.21.15.164:3300/0 conn(0x7fb4241444c0 0x7fb42406ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:26.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f07127fc640 1 -- 172.21.15.59:0/424705687 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026089059 0 0) 0x7f07140b32b0 con 0x7f07140d1810 2024-03-06T07:27:26.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.558+0000 7f0713fff640 1 -- 172.21.15.59:0/424705687 >> v1:172.21.15.164:6789/0 conn(0x7f07140b2030 legacy=0x7f07140b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.559+0000 7f0713fff640 1 -- 172.21.15.59:0/424705687 >> v1:172.21.15.59:6789/0 conn(0x7f07140d1810 legacy=0x7f07140d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.559+0000 7f0713fff640 1 -- 172.21.15.59:0/424705687 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07140b2cf0 con 0x7f07140b0080 2024-03-06T07:27:26.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.559+0000 7fd4c0b25640 1 --2- 172.21.15.59:0/2698861198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4bc06d3c0 0x7fd4bc0b1310 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc06bd30 tx=0x7fd4ac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b9458b82249d5d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.559+0000 7f45f1715640 1 --2- 172.21.15.59:0/3767928148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec06d000 0x7f45ec06d520 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f45dc009870 tx=0x7f45dc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b687dbd448d6394 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.559+0000 7fb42259c640 1 --2- 172.21.15.59:0/1307718581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb42415b2d0 0x7fb4240708d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb4180097a0 tx=0x7fb41802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb159d08439ddd1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.559+0000 7f0713fff640 1 --2- 172.21.15.59:0/424705687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07140b0080 0x7f0714166900 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f070c009dc0 tx=0x7f070c02bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62944df862c61f0c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.559+0000 7f45f0f14640 1 -- 172.21.15.59:0/3767928148 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc035020 con 0x7f45ec06d000 2024-03-06T07:27:26.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.559+0000 7fd4b3fff640 1 -- 172.21.15.59:0/2698861198 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac035020 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7fd4b3fff640 1 -- 172.21.15.59:0/2698861198 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac031830 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7fd4b3fff640 1 -- 172.21.15.59:0/2698861198 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac031c00 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f45f0f14640 1 -- 172.21.15.59:0/3767928148 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45dc031640 con 0x7f45ec06d000 2024-03-06T07:27:26.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f45f0f14640 1 -- 172.21.15.59:0/3767928148 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc02d040 con 0x7f45ec06d000 2024-03-06T07:27:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f07127fc640 1 -- 172.21.15.59:0/424705687 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c039070 con 0x7f07140b0080 2024-03-06T07:27:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7fb421d9b640 1 -- 172.21.15.59:0/1307718581 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418035020 con 0x7fb42415b2d0 2024-03-06T07:27:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7fb421d9b640 1 -- 172.21.15.59:0/1307718581 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb418031830 con 0x7fb42415b2d0 2024-03-06T07:27:26.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7fb421d9b640 1 -- 172.21.15.59:0/1307718581 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418031c00 con 0x7fb42415b2d0 2024-03-06T07:27:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f45f417a640 1 -- 172.21.15.59:0/3767928148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec06d000 msgr2=0x7f45ec06d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f45f417a640 1 --2- 172.21.15.59:0/3767928148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec06d000 0x7f45ec06d520 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f45dc009870 tx=0x7f45dc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f07127fc640 1 -- 172.21.15.59:0/424705687 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070c02ca10 con 0x7f07140b0080 2024-03-06T07:27:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f07127fc640 1 -- 172.21.15.59:0/424705687 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c02cde0 con 0x7f07140b0080 2024-03-06T07:27:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f0718fa0640 1 -- 172.21.15.59:0/424705687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07140b0080 msgr2=0x7f0714166900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f0718fa0640 1 --2- 172.21.15.59:0/424705687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07140b0080 0x7f0714166900 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f070c009dc0 tx=0x7f070c02bce0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f45f417a640 1 -- 172.21.15.59:0/3767928148 shutdown_connections 2024-03-06T07:27:26.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f0718fa0640 1 -- 172.21.15.59:0/424705687 shutdown_connections 2024-03-06T07:27:26.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f45f417a640 1 --2- 172.21.15.59:0/3767928148 >> v2:172.21.15.164:3300/0 conn(0x7f45ec06da60 0x7f45ec162a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f45f417a640 1 --2- 172.21.15.59:0/3767928148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec06d000 0x7f45ec06d520 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f45f417a640 1 -- 172.21.15.59:0/3767928148 >> 172.21.15.59:0/3767928148 conn(0x7f45ec0b1c80 msgr2=0x7f45ec0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f0718fa0640 1 --2- 172.21.15.59:0/424705687 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07140b0080 0x7f0714166900 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.560+0000 7f0718fa0640 1 -- 172.21.15.59:0/424705687 >> 172.21.15.59:0/424705687 conn(0x7f071409a9f0 msgr2=0x7f071409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7f0718fa0640 1 -- 172.21.15.59:0/424705687 shutdown_connections 2024-03-06T07:27:26.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7f45f417a640 1 -- 172.21.15.59:0/3767928148 shutdown_connections 2024-03-06T07:27:26.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fd4c358a640 1 -- 172.21.15.59:0/2698861198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4bc06d3c0 msgr2=0x7fd4bc0b1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fb429239640 1 -- 172.21.15.59:0/1307718581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb42415b2d0 msgr2=0x7fb4240708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7f45f417a640 1 -- 172.21.15.59:0/3767928148 wait complete. 2024-03-06T07:27:26.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fd4c358a640 1 --2- 172.21.15.59:0/2698861198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4bc06d3c0 0x7fd4bc0b1310 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc06bd30 tx=0x7fd4ac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7f0718fa0640 1 -- 172.21.15.59:0/424705687 wait complete. 2024-03-06T07:27:26.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fb429239640 1 --2- 172.21.15.59:0/1307718581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb42415b2d0 0x7fb4240708d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb4180097a0 tx=0x7fb41802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fd4c358a640 1 -- 172.21.15.59:0/2698861198 shutdown_connections 2024-03-06T07:27:26.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fd4c358a640 1 --2- 172.21.15.59:0/2698861198 >> v2:172.21.15.59:3300/0 conn(0x7fd4bc0b1850 0x7fd4bc0b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fd4c358a640 1 --2- 172.21.15.59:0/2698861198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4bc06d3c0 0x7fd4bc0b1310 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fd4c358a640 1 -- 172.21.15.59:0/2698861198 >> 172.21.15.59:0/2698861198 conn(0x7fd4bc09a9f0 msgr2=0x7fd4bc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fb429239640 1 -- 172.21.15.59:0/1307718581 shutdown_connections 2024-03-06T07:27:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fb429239640 1 --2- 172.21.15.59:0/1307718581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb42415b2d0 0x7fb4240708d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fb429239640 1 --2- 172.21.15.59:0/1307718581 >> v2:172.21.15.164:3300/0 conn(0x7fb4241444c0 0x7fb42406ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fb429239640 1 -- 172.21.15.59:0/1307718581 >> 172.21.15.59:0/1307718581 conn(0x7fb4240b1c80 msgr2=0x7fb4240cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fd4c358a640 1 -- 172.21.15.59:0/2698861198 shutdown_connections 2024-03-06T07:27:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fb429239640 1 -- 172.21.15.59:0/1307718581 shutdown_connections 2024-03-06T07:27:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fd4c358a640 1 -- 172.21.15.59:0/2698861198 wait complete. 2024-03-06T07:27:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.561+0000 7fb429239640 1 -- 172.21.15.59:0/1307718581 wait complete. 2024-03-06T07:27:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.562+0000 7f45f417a640 1 Processor -- start 2024-03-06T07:27:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.562+0000 7f0718fa0640 1 Processor -- start 2024-03-06T07:27:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.562+0000 7f45f417a640 1 -- start start 2024-03-06T07:27:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.562+0000 7f0718fa0640 1 -- start start 2024-03-06T07:27:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.562+0000 7fd4c358a640 1 Processor -- start 2024-03-06T07:27:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.562+0000 7fd4c358a640 1 -- start start 2024-03-06T07:27:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f417a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45ec06d000 0x7f45ec15adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fb429239640 1 Processor -- start 2024-03-06T07:27:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f417a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45ec06da60 0x7f45ec0c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fb429239640 1 -- start start 2024-03-06T07:27:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f417a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec15ff70 0x7f45ec1f7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f417a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec160350 con 0x7f45ec06da60 2024-03-06T07:27:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f417a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec1665b0 con 0x7f45ec06d000 2024-03-06T07:27:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f417a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec166db0 con 0x7f45ec15ff70 2024-03-06T07:27:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f1715640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45ec06da60 0x7f45ec0c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f0718fa0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07140b0080 0x7f07140c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f1715640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45ec06da60 0x7f45ec0c52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38510/0 (socket says 172.21.15.59:38510) 2024-03-06T07:27:26.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f1715640 1 -- 172.21.15.59:0/499144495 learned_addr learned my addr 172.21.15.59:0/499144495 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:26.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f0718fa0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07140b2030 0x7f0714152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f2717640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec15ff70 0x7f45ec1f7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f1f16640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45ec06d000 0x7f45ec15adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f0718fa0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07140d1810 0x7f071415cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fd4c358a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4bc06d3c0 0x7fd4bc15bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f0718fa0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071415bee0 con 0x7f07140b2030 2024-03-06T07:27:26.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f0718fa0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071415bd60 con 0x7f07140b0080 2024-03-06T07:27:26.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f0718fa0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0714166e40 con 0x7f07140d1810 2024-03-06T07:27:26.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f07137fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07140b0080 0x7f07140c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f0712ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07140b2030 0x7f0714152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f07137fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07140b0080 0x7f07140c9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:58210/0 (socket says 172.21.15.59:58210) 2024-03-06T07:27:26.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f07137fe640 1 -- 172.21.15.59:0/3523652903 learned_addr learned my addr 172.21.15.59:0/3523652903 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:26.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fd4c358a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4bc0b1850 0x7fd4bc158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fd4c358a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4bc1614a0 0x7fd4bc14a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fd4c358a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4bc0b1db0 con 0x7fd4bc0b1850 2024-03-06T07:27:26.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fd4c358a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4bc0b1c30 con 0x7fd4bc1614a0 2024-03-06T07:27:26.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fd4c358a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4bc0b1f30 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f2717640 1 -- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45ec06d000 msgr2=0x7f45ec15adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f2717640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45ec06d000 0x7f45ec15adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f2717640 1 -- 172.21.15.59:0/499144495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45ec06da60 msgr2=0x7f45ec0c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f2717640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45ec06da60 0x7f45ec0c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f0713fff640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07140d1810 0x7f071415cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fd4c1326640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4bc06d3c0 0x7fd4bc15bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fd4c1326640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4bc06d3c0 0x7fd4bc15bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:40742/0 (socket says 172.21.15.59:40742) 2024-03-06T07:27:26.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fd4c1326640 1 -- 172.21.15.59:0/2769577596 learned_addr learned my addr 172.21.15.59:0/2769577596 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:26.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7f45f2717640 1 -- 172.21.15.59:0/499144495 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec1f8860 con 0x7f45ec15ff70 2024-03-06T07:27:26.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.563+0000 7fd4c0b25640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4bc0b1850 0x7fd4bc158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4c1b27640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4bc1614a0 0x7fd4bc14a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f07137fe640 1 -- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07140d1810 msgr2=0x7f071415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f07137fe640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07140d1810 0x7f071415cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f07137fe640 1 -- 172.21.15.59:0/3523652903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07140b2030 msgr2=0x7f0714152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f07137fe640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07140b2030 0x7f0714152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f07137fe640 1 -- 172.21.15.59:0/3523652903 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07140c9fb0 con 0x7f07140b0080 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f45f2717640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec15ff70 0x7f45ec1f7d20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f45e400efc0 tx=0x7f45e400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4c1326640 1 -- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4bc1614a0 msgr2=0x7fd4bc14a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4c1326640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4bc1614a0 0x7fd4bc14a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4c1326640 1 -- 172.21.15.59:0/2769577596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4bc0b1850 msgr2=0x7fd4bc158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4c1326640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4bc0b1850 0x7fd4bc158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f07137fe640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07140b0080 0x7f07140c9a70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0704001a70 tx=0x7f0704014420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fb429239640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb424070e10 0x7fb42415a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4c1326640 1 -- 172.21.15.59:0/2769577596 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4bc0cc560 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fb429239640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4241444c0 0x7fb4241408b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4c0b25640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4bc0b1850 0x7fd4bc158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f45e27fc640 1 -- 172.21.15.59:0/499144495 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4018020 con 0x7f45ec15ff70 2024-03-06T07:27:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fb429239640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb42415b2d0 0x7fb424159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f45e27fc640 1 -- 172.21.15.59:0/499144495 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e4014480 con 0x7f45ec15ff70 2024-03-06T07:27:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f45e27fc640 1 -- 172.21.15.59:0/499144495 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4014870 con 0x7f45ec15ff70 2024-03-06T07:27:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fb429239640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42406def0 con 0x7fb4241444c0 2024-03-06T07:27:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f06fbfff640 1 -- 172.21.15.59:0/3523652903 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0704014ad0 con 0x7f07140b0080 2024-03-06T07:27:26.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fb429239640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42406dc10 con 0x7fb42415b2d0 2024-03-06T07:27:26.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f06fbfff640 1 -- 172.21.15.59:0/3523652903 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0704014c70 con 0x7f07140b0080 2024-03-06T07:27:26.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f06fbfff640 1 -- 172.21.15.59:0/3523652903 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0704014e60 con 0x7f07140b0080 2024-03-06T07:27:26.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fb429239640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb42406cf40 con 0x7fb424070e10 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4c1326640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4bc06d3c0 0x7fd4bc15bd60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800c450 tx=0x7fd4b800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fb422d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb424070e10 0x7fb42415a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fb422d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb424070e10 0x7fb42415a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:40746/0 (socket says 172.21.15.59:40746) 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fb422d9d640 1 -- 172.21.15.59:0/340294879 learned_addr learned my addr 172.21.15.59:0/340294879 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f0718fa0640 1 -- 172.21.15.59:0/3523652903 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0714152f80 con 0x7f07140b0080 2024-03-06T07:27:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f0718fa0640 1 -- 172.21.15.59:0/3523652903 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07140ca540 con 0x7f07140b0080 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fb42259c640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4241444c0 0x7fb4241408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fb42359e640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb42415b2d0 0x7fb424159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4b1ffb640 1 -- 172.21.15.59:0/2769577596 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80052d0 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4b1ffb640 1 -- 172.21.15.59:0/2769577596 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b8005470 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4b1ffb640 1 -- 172.21.15.59:0/2769577596 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8012870 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7fd4c358a640 1 -- 172.21.15.59:0/2769577596 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4bc0cc840 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.565+0000 7fb422d9d640 1 -- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb42415b2d0 msgr2=0x7fb424159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.565+0000 7fb422d9d640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb42415b2d0 0x7fb424159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.565+0000 7fb422d9d640 1 -- 172.21.15.59:0/340294879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4241444c0 msgr2=0x7fb4241408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.565+0000 7fb422d9d640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4241444c0 0x7fb4241408b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.565+0000 7fb422d9d640 1 -- 172.21.15.59:0/340294879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb424140df0 con 0x7fb424070e10 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.565+0000 7fb42259c640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4241444c0 0x7fb4241408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f45f417a640 1 -- 172.21.15.59:0/499144495 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ec1f8b40 con 0x7f45ec15ff70 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.564+0000 7f45f417a640 1 -- 172.21.15.59:0/499144495 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ec0c4db0 con 0x7f45ec15ff70 2024-03-06T07:27:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.565+0000 7fd4c358a640 1 -- 172.21.15.59:0/2769577596 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4bc0cce00 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.565+0000 7fb422d9d640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb424070e10 0x7fb42415a560 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb414010f10 tx=0x7fb41400e730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.566+0000 7f45f417a640 1 -- 172.21.15.59:0/499144495 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ec0c48c0 con 0x7f45ec15ff70 2024-03-06T07:27:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.566+0000 7fb40b7fe640 1 -- 172.21.15.59:0/340294879 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41401e070 con 0x7fb424070e10 2024-03-06T07:27:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.566+0000 7fb40b7fe640 1 -- 172.21.15.59:0/340294879 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb414011860 con 0x7fb424070e10 2024-03-06T07:27:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.566+0000 7fb40b7fe640 1 -- 172.21.15.59:0/340294879 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb414011c10 con 0x7fb424070e10 2024-03-06T07:27:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.566+0000 7fb429239640 1 -- 172.21.15.59:0/340294879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb424159e50 con 0x7fb424070e10 2024-03-06T07:27:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.566+0000 7fb429239640 1 -- 172.21.15.59:0/340294879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4240a9940 con 0x7fb424070e10 2024-03-06T07:27:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.566+0000 7fd4c358a640 1 -- 172.21.15.59:0/2769577596 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4bc04f330 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.566+0000 7f0718fa0640 1 -- 172.21.15.59:0/3523652903 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0714068f80 con 0x7f07140b0080 2024-03-06T07:27:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.566+0000 7f06fbfff640 1 -- 172.21.15.59:0/3523652903 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0704007310 con 0x7f07140b0080 2024-03-06T07:27:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.567+0000 7f45e27fc640 1 -- 172.21.15.59:0/499144495 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45e4005d10 con 0x7f45ec15ff70 2024-03-06T07:27:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.567+0000 7f06fbfff640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f070006b4d0 0x7f070006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.566+0000 7fd4b1ffb640 1 -- 172.21.15.59:0/2769577596 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4b8012a10 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.567+0000 7fd4b1ffb640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4a006b4d0 0x7fd4a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.567+0000 7f06fbfff640 1 -- 172.21.15.59:0/3523652903 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0704091e40 con 0x7f07140b0080 2024-03-06T07:27:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.567+0000 7f45e27fc640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45d006b520 0x7f45d006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.567+0000 7f0712ffd640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f070006b4d0 0x7f070006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.567+0000 7fd4b1ffb640 1 -- 172.21.15.59:0/2769577596 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4b8089c20 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.567+0000 7fd4c0b25640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4a006b4d0 0x7fd4a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.567+0000 7fb429239640 1 -- 172.21.15.59:0/340294879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42404f330 con 0x7fb424070e10 2024-03-06T07:27:26.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.567+0000 7f45e27fc640 1 -- 172.21.15.59:0/499144495 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45e408b420 con 0x7f45ec15ff70 2024-03-06T07:27:26.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.567+0000 7f45f1f16640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45d006b520 0x7f45d006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.568+0000 7f0712ffd640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f070006b4d0 0x7f070006d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0708007fd0 tx=0x7f0708009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.568+0000 7fd4c0b25640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4a006b4d0 0x7fd4a006d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac036000 tx=0x7fd4ac03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.568+0000 7f45f1f16640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45d006b520 0x7f45d006d9e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f45e80016e0 tx=0x7f45e8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.571+0000 7f45e27fc640 1 -- 172.21.15.59:0/499144495 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45e4022030 con 0x7f45ec15ff70 2024-03-06T07:27:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.571+0000 7f06fbfff640 1 -- 172.21.15.59:0/3523652903 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0704060c30 con 0x7f07140b0080 2024-03-06T07:27:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.572+0000 7fb40b7fe640 1 -- 172.21.15.59:0/340294879 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb414021070 con 0x7fb424070e10 2024-03-06T07:27:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.572+0000 7fb40b7fe640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb41006b500 0x7fb41006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.573+0000 7fb42259c640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb41006b500 0x7fb41006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.573+0000 7fb42259c640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb41006b500 0x7fb41006d9c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb418036000 tx=0x7fb41803c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.576+0000 7fb40b7fe640 1 -- 172.21.15.59:0/340294879 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb414059a00 con 0x7fb424070e10 2024-03-06T07:27:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.576+0000 7fd4b1ffb640 1 -- 172.21.15.59:0/2769577596 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4b80584e0 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.581+0000 7fb40b7fe640 1 -- 172.21.15.59:0/340294879 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb414028020 con 0x7fb424070e10 2024-03-06T07:27:26.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.825+0000 7f45f417a640 1 -- 172.21.15.59:0/499144495 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f45ec04f330 con 0x7f45ec15ff70 2024-03-06T07:27:26.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.830+0000 7fd4c358a640 1 -- 172.21.15.59:0/2769577596 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7fd4bc06e8e0 con 0x7fd4bc06d3c0 2024-03-06T07:27:26.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.831+0000 7f0718fa0640 1 -- 172.21.15.59:0/3523652903 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f07140b2940 con 0x7f07140b0080 2024-03-06T07:27:26.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:26.836+0000 7fb429239640 1 -- 172.21.15.59:0/340294879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fb4240748c0 con 0x7fb424070e10 2024-03-06T07:27:27.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.113+0000 7f45e27fc640 1 -- 172.21.15.59:0/499144495 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7956 v67) v1 ==== 95+0+0 (secure 0 0 0) 0x7f45e4010040 con 0x7f45ec15ff70 2024-03-06T07:27:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 7956 2024-03-06T07:27:27.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 -- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45d006b520 msgr2=0x7f45d006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45d006b520 0x7f45d006d9e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f45e80016e0 tx=0x7f45e8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:27.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 -- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec15ff70 msgr2=0x7f45ec1f7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec15ff70 0x7f45ec1f7d20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f45e400efc0 tx=0x7f45e400c490 comp rx=0 tx=0).stop 2024-03-06T07:27:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 -- 172.21.15.59:0/499144495 shutdown_connections 2024-03-06T07:27:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45d006b520 0x7f45d006d9e0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45ec15ff70 0x7f45ec1f7d20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45ec06da60 0x7f45ec0c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 --2- 172.21.15.59:0/499144495 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45ec06d000 0x7f45ec15adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 -- 172.21.15.59:0/499144495 >> 172.21.15.59:0/499144495 conn(0x7f45ec0b1c80 msgr2=0x7f45ec0b4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 -- 172.21.15.59:0/499144495 shutdown_connections 2024-03-06T07:27:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.117+0000 7f45f417a640 1 -- 172.21.15.59:0/499144495 wait complete. 2024-03-06T07:27:27.122 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:27:27.121+0000 7fed61d70640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:27.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.125+0000 7f06fbfff640 1 -- 172.21.15.59:0/3523652903 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 7231 v68) v1 ==== 95+0+0 (secure 0 0 0) 0x7f0704060530 con 0x7f07140b0080 2024-03-06T07:27:27.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.126+0000 7fd4b1ffb640 1 -- 172.21.15.59:0/2769577596 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7959 v68) v1 ==== 95+0+0 (secure 0 0 0) 0x7fd4b805c580 con 0x7fd4bc06d3c0 2024-03-06T07:27:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 7231 2024-03-06T07:27:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.126+0000 7fb40b7fe640 1 -- 172.21.15.59:0/340294879 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7968 v68) v1 ==== 95+0+0 (secure 0 0 0) 0x7fb41405b960 con 0x7fb424070e10 2024-03-06T07:27:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 7959 2024-03-06T07:27:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 7968 2024-03-06T07:27:27.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 -- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f070006b4d0 msgr2=0x7f070006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f070006b4d0 0x7f070006d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0708007fd0 tx=0x7f0708009040 comp rx=0 tx=0).stop 2024-03-06T07:27:27.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 -- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07140b0080 msgr2=0x7f07140c9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07140b0080 0x7f07140c9a70 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0704001a70 tx=0x7f0704014420 comp rx=0 tx=0).stop 2024-03-06T07:27:27.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 -- 172.21.15.59:0/3523652903 shutdown_connections 2024-03-06T07:27:27.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f070006b4d0 0x7f070006d990 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07140d1810 0x7f071415cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f07140b2030 0x7f0714152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 --2- 172.21.15.59:0/3523652903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07140b0080 0x7f07140c9a70 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 -- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4a006b4d0 msgr2=0x7fd4a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4a006b4d0 0x7fd4a006d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac036000 tx=0x7fd4ac03c040 comp rx=0 tx=0).stop 2024-03-06T07:27:27.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 -- 172.21.15.59:0/3523652903 >> 172.21.15.59:0/3523652903 conn(0x7f071409a9f0 msgr2=0x7f07140d1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 -- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4bc06d3c0 msgr2=0x7fd4bc15bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4bc06d3c0 0x7fd4bc15bd60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800c450 tx=0x7fd4b800c910 comp rx=0 tx=0).stop 2024-03-06T07:27:27.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 -- 172.21.15.59:0/2769577596 shutdown_connections 2024-03-06T07:27:27.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd4a006b4d0 0x7fd4a006d990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4bc1614a0 0x7fd4bc14a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd4bc0b1850 0x7fd4bc158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 --2- 172.21.15.59:0/2769577596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4bc06d3c0 0x7fd4bc15bd60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 -- 172.21.15.59:0/2769577596 >> 172.21.15.59:0/2769577596 conn(0x7fd4bc09a9f0 msgr2=0x7fd4bc0b5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 -- 172.21.15.59:0/3523652903 shutdown_connections 2024-03-06T07:27:27.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7f0718fa0640 1 -- 172.21.15.59:0/3523652903 wait complete. 2024-03-06T07:27:27.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 -- 172.21.15.59:0/2769577596 shutdown_connections 2024-03-06T07:27:27.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.129+0000 7fd4c358a640 1 -- 172.21.15.59:0/2769577596 wait complete. 2024-03-06T07:27:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 -- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb41006b500 msgr2=0x7fb41006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb41006b500 0x7fb41006d9c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb418036000 tx=0x7fb41803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 -- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb424070e10 msgr2=0x7fb42415a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb424070e10 0x7fb42415a560 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb414010f10 tx=0x7fb41400e730 comp rx=0 tx=0).stop 2024-03-06T07:27:27.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 -- 172.21.15.59:0/340294879 shutdown_connections 2024-03-06T07:27:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb41006b500 0x7fb41006d9c0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb42415b2d0 0x7fb424159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4241444c0 0x7fb4241408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 --2- 172.21.15.59:0/340294879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb424070e10 0x7fb42415a560 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 -- 172.21.15.59:0/340294879 >> 172.21.15.59:0/340294879 conn(0x7fb4240b1c80 msgr2=0x7fb42415de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 -- 172.21.15.59:0/340294879 shutdown_connections 2024-03-06T07:27:27.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.130+0000 7fb429239640 1 -- 172.21.15.59:0/340294879 wait complete. 2024-03-06T07:27:27.137 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:27.137 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:27.139 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:27.140 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:27.145 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:27.145 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:27.150 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:27.151 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:27.152 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:27.152 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:27.154 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:27.154 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:27.154 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:27.155 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:27.192 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:27.192 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:27.193 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:27.215 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:27:27.230 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:27:27.231 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:27:27.269 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:27:27.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.294+0000 7ff354dee640 1 Processor -- start 2024-03-06T07:27:27.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.294+0000 7ff354dee640 1 -- start start 2024-03-06T07:27:27.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff354dee640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff3500b3060 0x7ff35015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff354dee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35015bf30 0x7ff35015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff354dee640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35015f2d0 con 0x7ff35013dc10 2024-03-06T07:27:27.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff354dee640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3500cf3f0 con 0x7ff3500b3060 2024-03-06T07:27:27.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff354dee640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35015b210 con 0x7ff35015bf30 2024-03-06T07:27:27.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff34dd9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff3500b3060 0x7ff35015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff34dd9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff3500b3060 0x7ff35015e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38570/0 (socket says 172.21.15.59:38570) 2024-03-06T07:27:27.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff34dd9b640 1 -- 172.21.15.59:0/1329554737 learned_addr learned my addr 172.21.15.59:0/1329554737 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:27.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff34ed9d640 1 --2- 172.21.15.59:0/1329554737 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35015bf30 0x7ff35015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff34dd9b640 1 -- 172.21.15.59:0/1329554737 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35015bf30 msgr2=0x7ff35015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:27.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff34dd9b640 1 --2- 172.21.15.59:0/1329554737 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35015bf30 0x7ff35015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff34dd9b640 1 -- 172.21.15.59:0/1329554737 >> v1:172.21.15.164:6789/0 conn(0x7ff35013dc10 legacy=0x7ff3500b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.295+0000 7ff34dd9b640 1 -- 172.21.15.59:0/1329554737 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35015f580 con 0x7ff3500b3060 2024-03-06T07:27:27.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.296+0000 7ff34dd9b640 1 --2- 172.21.15.59:0/1329554737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3500b3060 0x7ff35015e850 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ff338009870 tx=0x7ff33802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a937a1ba63073c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.296+0000 7ff34d59a640 1 -- 172.21.15.59:0/1329554737 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338035020 con 0x7ff3500b3060 2024-03-06T07:27:27.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.296+0000 7ff34d59a640 1 -- 172.21.15.59:0/1329554737 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff338031640 con 0x7ff3500b3060 2024-03-06T07:27:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.296+0000 7ff34d59a640 1 -- 172.21.15.59:0/1329554737 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33802d040 con 0x7ff3500b3060 2024-03-06T07:27:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.297+0000 7ff354dee640 1 -- 172.21.15.59:0/1329554737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3500b3060 msgr2=0x7ff35015e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.297+0000 7ff354dee640 1 --2- 172.21.15.59:0/1329554737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3500b3060 0x7ff35015e850 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ff338009870 tx=0x7ff33802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.297+0000 7ff354dee640 1 -- 172.21.15.59:0/1329554737 shutdown_connections 2024-03-06T07:27:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.297+0000 7ff354dee640 1 --2- 172.21.15.59:0/1329554737 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35015bf30 0x7ff35015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.297+0000 7ff354dee640 1 --2- 172.21.15.59:0/1329554737 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3500b3060 0x7ff35015e850 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.297+0000 7ff354dee640 1 -- 172.21.15.59:0/1329554737 >> 172.21.15.59:0/1329554737 conn(0x7ff3500cf8a0 msgr2=0x7ff3500cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.297+0000 7ff354dee640 1 -- 172.21.15.59:0/1329554737 shutdown_connections 2024-03-06T07:27:27.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.297+0000 7ff354dee640 1 -- 172.21.15.59:0/1329554737 wait complete. 2024-03-06T07:27:27.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.298+0000 7ff354dee640 1 Processor -- start 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.298+0000 7ff354dee640 1 -- start start 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff354dee640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3500b3060 0x7ff350138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff354dee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff35013dc10 0x7ff350154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff354dee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35015bf30 0x7ff350153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff354dee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35013dff0 con 0x7ff35013dc10 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff354dee640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35015b730 con 0x7ff3500b3060 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff354dee640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500708d0 con 0x7ff35015bf30 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff34dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff35013dc10 0x7ff350154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff34dd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff35013dc10 0x7ff350154e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38584/0 (socket says 172.21.15.59:38584) 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff34dd9b640 1 -- 172.21.15.59:0/1869952403 learned_addr learned my addr 172.21.15.59:0/1869952403 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff34dd9b640 1 -- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35015bf30 msgr2=0x7ff350153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff34dd9b640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35015bf30 0x7ff350153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff34dd9b640 1 -- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3500b3060 msgr2=0x7ff350138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff34e59c640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3500b3060 0x7ff350138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff34dd9b640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3500b3060 0x7ff350138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff34dd9b640 1 -- 172.21.15.59:0/1869952403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff350154af0 con 0x7ff35013dc10 2024-03-06T07:27:27.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.299+0000 7ff34e59c640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3500b3060 0x7ff350138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:27.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.300+0000 7ff34dd9b640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff35013dc10 0x7ff350154e20 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7ff338000c00 tx=0x7ff3380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.300+0000 7ff346ffd640 1 -- 172.21.15.59:0/1869952403 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338040070 con 0x7ff35013dc10 2024-03-06T07:27:27.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.300+0000 7ff346ffd640 1 -- 172.21.15.59:0/1869952403 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff338039970 con 0x7ff35013dc10 2024-03-06T07:27:27.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.300+0000 7ff354dee640 1 -- 172.21.15.59:0/1869952403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff350154530 con 0x7ff35013dc10 2024-03-06T07:27:27.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.300+0000 7ff346ffd640 1 -- 172.21.15.59:0/1869952403 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338035040 con 0x7ff35013dc10 2024-03-06T07:27:27.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.300+0000 7ff354dee640 1 -- 172.21.15.59:0/1869952403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35013f430 con 0x7ff35013dc10 2024-03-06T07:27:27.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.301+0000 7ff354dee640 1 -- 172.21.15.59:0/1869952403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35004f330 con 0x7ff35013dc10 2024-03-06T07:27:27.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.301+0000 7ff346ffd640 1 -- 172.21.15.59:0/1869952403 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff33804b060 con 0x7ff35013dc10 2024-03-06T07:27:27.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.302+0000 7ff346ffd640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff33006b4d0 0x7ff33006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.302+0000 7ff346ffd640 1 -- 172.21.15.59:0/1869952403 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff3380abaa0 con 0x7ff35013dc10 2024-03-06T07:27:27.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.302+0000 7ff34e59c640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff33006b4d0 0x7ff33006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.303+0000 7ff34e59c640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff33006b4d0 0x7ff33006d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff340001680 tx=0x7ff340009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.306+0000 7ff346ffd640 1 -- 172.21.15.59:0/1869952403 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff33807a8c0 con 0x7ff35013dc10 2024-03-06T07:27:27.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.531+0000 7ff354dee640 1 -- 172.21.15.59:0/1869952403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff35006eb00 con 0x7ff35013dc10 2024-03-06T07:27:27.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.531+0000 7ff346ffd640 1 -- 172.21.15.59:0/1869952403 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+1802 (secure 0 0 0) 0x7ff33807a1c0 con 0x7ff35013dc10 2024-03-06T07:27:27.533 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:27.533 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":68,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:27.122164+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:27:27.533 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 68 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 -- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff33006b4d0 msgr2=0x7ff33006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff33006b4d0 0x7ff33006d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff340001680 tx=0x7ff340009040 comp rx=0 tx=0).stop 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 -- 172.21.15.59:0/1869952403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff35013dc10 msgr2=0x7ff350154e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff35013dc10 0x7ff350154e20 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7ff338000c00 tx=0x7ff3380046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 -- 172.21.15.59:0/1869952403 shutdown_connections 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff33006b4d0 0x7ff33006d990 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35015bf30 0x7ff350153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff35013dc10 0x7ff350154e20 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 --2- 172.21.15.59:0/1869952403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3500b3060 0x7ff350138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 -- 172.21.15.59:0/1869952403 >> 172.21.15.59:0/1869952403 conn(0x7ff3500cf8a0 msgr2=0x7ff35013a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 -- 172.21.15.59:0/1869952403 shutdown_connections 2024-03-06T07:27:27.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:27.533+0000 7ff354dee640 1 -- 172.21.15.59:0/1869952403 wait complete. 2024-03-06T07:27:27.949 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:14.345180+0000', 'modified': '2024-03-06T07:27:27.122164+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:27.949 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-03-06T07:27:28.950 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.062+0000 7f3425a47640 1 Processor -- start 2024-03-06T07:27:29.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.062+0000 7f3425a47640 1 -- start start 2024-03-06T07:27:29.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f3425a47640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200d00c0 0x7f34200d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f3425a47640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f34200cf850 0x7f34200cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f3425a47640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f34200ce1b0 0x7f34200ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f3425a47640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f342006ca90 con 0x7f34200ce1b0 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f3425a47640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34200d09e0 con 0x7f34200cf850 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f3425a47640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342015e380 con 0x7f34200d00c0 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f341f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f34200ce1b0 0x7f34200ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f341f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f34200ce1b0 0x7f34200ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38598/0 (socket says 172.21.15.59:38598) 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f341f7fe640 1 -- 172.21.15.59:0/724108227 learned_addr learned my addr 172.21.15.59:0/724108227 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f341effd640 1 --2- 172.21.15.59:0/724108227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200d00c0 0x7f34200d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f341e7fc640 1 --2- 172.21.15.59:0/724108227 >> v2:172.21.15.164:3300/0 conn(0x7f34200cf850 0x7f34200cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f341f7fe640 1 -- 172.21.15.59:0/724108227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200d00c0 msgr2=0x7f34200d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f341f7fe640 1 --2- 172.21.15.59:0/724108227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200d00c0 0x7f34200d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f341f7fe640 1 -- 172.21.15.59:0/724108227 >> v2:172.21.15.164:3300/0 conn(0x7f34200cf850 msgr2=0x7f34200cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f341f7fe640 1 --2- 172.21.15.59:0/724108227 >> v2:172.21.15.164:3300/0 conn(0x7f34200cf850 0x7f34200cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.063+0000 7f341f7fe640 1 -- 172.21.15.59:0/724108227 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34200cdc10 con 0x7f34200ce1b0 2024-03-06T07:27:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.064+0000 7f341f7fe640 1 --2- 172.21.15.59:0/724108227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200ce1b0 0x7f34200ce640 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f34200722b0 tx=0x7f341002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d591c9b15673dcd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.064+0000 7f341dffb640 1 -- 172.21.15.59:0/724108227 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341002ca70 con 0x7f34200ce1b0 2024-03-06T07:27:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.064+0000 7f341dffb640 1 -- 172.21.15.59:0/724108227 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341002cc10 con 0x7f34200ce1b0 2024-03-06T07:27:29.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.064+0000 7f341dffb640 1 -- 172.21.15.59:0/724108227 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34100343f0 con 0x7f34200ce1b0 2024-03-06T07:27:29.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.065+0000 7f3425a47640 1 -- 172.21.15.59:0/724108227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200ce1b0 msgr2=0x7f34200ce640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.065+0000 7f3425a47640 1 --2- 172.21.15.59:0/724108227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200ce1b0 0x7f34200ce640 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f34200722b0 tx=0x7f341002bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.065+0000 7f3425a47640 1 -- 172.21.15.59:0/724108227 shutdown_connections 2024-03-06T07:27:29.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.065+0000 7f3425a47640 1 --2- 172.21.15.59:0/724108227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200ce1b0 0x7f34200ce640 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.065+0000 7f3425a47640 1 --2- 172.21.15.59:0/724108227 >> v2:172.21.15.164:3300/0 conn(0x7f34200cf850 0x7f34200cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.065+0000 7f3425a47640 1 --2- 172.21.15.59:0/724108227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200d00c0 0x7f34200d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.065+0000 7f3425a47640 1 -- 172.21.15.59:0/724108227 >> 172.21.15.59:0/724108227 conn(0x7f34200b35a0 msgr2=0x7f34200b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.065+0000 7f3425a47640 1 -- 172.21.15.59:0/724108227 shutdown_connections 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.065+0000 7f3425a47640 1 -- 172.21.15.59:0/724108227 wait complete. 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.066+0000 7f3425a47640 1 Processor -- start 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.066+0000 7f3425a47640 1 -- start start 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f3425a47640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200ce1b0 0x7f34200b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f3425a47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f3425a47640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34200d00c0 0x7f34200c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f3425a47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342006c720 con 0x7f34200cf850 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f3425a47640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342006c440 con 0x7f34200d00c0 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f3425a47640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34200cd900 con 0x7f34200ce1b0 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f341e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f341e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38614/0 (socket says 172.21.15.59:38614) 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f341e7fc640 1 -- 172.21.15.59:0/1632602619 learned_addr learned my addr 172.21.15.59:0/1632602619 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f341f7fe640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34200d00c0 0x7f34200c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f341effd640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200ce1b0 0x7f34200b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f341e7fc640 1 -- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200ce1b0 msgr2=0x7f34200b5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f341e7fc640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200ce1b0 0x7f34200b5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f341e7fc640 1 -- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34200d00c0 msgr2=0x7f34200c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.067+0000 7f341e7fc640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34200d00c0 0x7f34200c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.068+0000 7f341e7fc640 1 -- 172.21.15.59:0/1632602619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3420153be0 con 0x7f34200cf850 2024-03-06T07:27:29.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.068+0000 7f341e7fc640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200ca590 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f340800e9e0 tx=0x7f340800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.068+0000 7f3417fff640 1 -- 172.21.15.59:0/1632602619 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408017070 con 0x7f34200cf850 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.068+0000 7f3417fff640 1 -- 172.21.15.59:0/1632602619 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340800c9d0 con 0x7f34200cf850 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.068+0000 7f3425a47640 1 -- 172.21.15.59:0/1632602619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3420153ec0 con 0x7f34200cf850 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.068+0000 7f3417fff640 1 -- 172.21.15.59:0/1632602619 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340800cdc0 con 0x7f34200cf850 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.068+0000 7f3425a47640 1 -- 172.21.15.59:0/1632602619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3420154480 con 0x7f34200cf850 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.069+0000 7f3425a47640 1 -- 172.21.15.59:0/1632602619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342004f330 con 0x7f34200cf850 2024-03-06T07:27:29.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.069+0000 7f3417fff640 1 -- 172.21.15.59:0/1632602619 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f340801a070 con 0x7f34200cf850 2024-03-06T07:27:29.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.070+0000 7f3417fff640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340006b4d0 0x7f340006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.070+0000 7f3417fff640 1 -- 172.21.15.59:0/1632602619 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f340808a6a0 con 0x7f34200cf850 2024-03-06T07:27:29.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.070+0000 7f341effd640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340006b4d0 0x7f340006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.071+0000 7f341effd640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340006b4d0 0x7f340006d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f34180098f0 tx=0x7f3418008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.074+0000 7f3417fff640 1 -- 172.21.15.59:0/1632602619 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34080594c0 con 0x7f34200cf850 2024-03-06T07:27:29.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.297+0000 7f3425a47640 1 -- 172.21.15.59:0/1632602619 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34200b4740 con 0x7f34200cf850 2024-03-06T07:27:29.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.298+0000 7f3417fff640 1 -- 172.21.15.59:0/1632602619 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f3408058dc0 con 0x7f34200cf850 2024-03-06T07:27:29.300 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:29.300 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":71,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:28.138261+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:27:29.301 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 71 2024-03-06T07:27:29.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 -- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340006b4d0 msgr2=0x7f340006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340006b4d0 0x7f340006d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f34180098f0 tx=0x7f3418008040 comp rx=0 tx=0).stop 2024-03-06T07:27:29.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 -- 172.21.15.59:0/1632602619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 msgr2=0x7f34200ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200ca590 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f340800e9e0 tx=0x7f340800eea0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 -- 172.21.15.59:0/1632602619 shutdown_connections 2024-03-06T07:27:29.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f340006b4d0 0x7f340006d990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34200d00c0 0x7f34200c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34200cf850 0x7f34200ca590 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 --2- 172.21.15.59:0/1632602619 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34200ce1b0 0x7f34200b5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 -- 172.21.15.59:0/1632602619 >> 172.21.15.59:0/1632602619 conn(0x7f34200b35a0 msgr2=0x7f34200cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 -- 172.21.15.59:0/1632602619 shutdown_connections 2024-03-06T07:27:29.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:29.300+0000 7f3425a47640 1 -- 172.21.15.59:0/1632602619 wait complete. 2024-03-06T07:27:29.715 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:14.345180+0000', 'modified': '2024-03-06T07:27:28.138261+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8253}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8253': {'gid': 8253, 'name': 'c', 'rank': 0, 'incarnation': 70, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.59:6835/3442407979', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 3442407979}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 3442407979}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:29.715 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8253:up:reconnect 2024-03-06T07:27:30.716 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:30.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.828+0000 7fa22860e640 1 Processor -- start 2024-03-06T07:27:30.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.828+0000 7fa22860e640 1 -- start start 2024-03-06T07:27:30.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.829+0000 7fa22860e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2200b6550 0x7fa2200cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.829+0000 7fa22860e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2200cef50 con 0x7fa2200d06a0 2024-03-06T07:27:30.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.829+0000 7fa22860e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa22015c620 con 0x7fa2200d02c0 2024-03-06T07:27:30.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.829+0000 7fa22860e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200b50b0 con 0x7fa2200b6550 2024-03-06T07:27:30.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.829+0000 7fa226bab640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa2200d06a0 0x7fa2200ce840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:53690/0 (socket says 172.21.15.59:53690) 2024-03-06T07:27:30.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.829+0000 7fa226bab640 1 -- 172.21.15.59:0/3064345600 learned_addr learned my addr 172.21.15.59:0/3064345600 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:30.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.829+0000 7fa225ba9640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2200b6550 0x7fa2200cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.829+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315950353 0 0) 0x7fa2200cef50 con 0x7fa2200d06a0 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.829+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa20c003520 con 0x7fa2200d06a0 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076656686 0 0) 0x7fa22015c620 con 0x7fa2200d02c0 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2200cef50 con 0x7fa2200d02c0 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2839884223 0 0) 0x7fa20c003520 con 0x7fa2200d06a0 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2200b6550 msgr2=0x7fa2200cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2200b6550 0x7fa2200cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 >> v1:172.21.15.164:6789/0 conn(0x7fa2200d02c0 legacy=0x7fa2200ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2200cd800 con 0x7fa2200d06a0 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fa218001670 con 0x7fa2200d06a0 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 >> v1:172.21.15.59:6789/0 conn(0x7fa2200d06a0 legacy=0x7fa2200ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2200b6550 0x7fa20c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa20c0040d0 0x7fa20c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 0x7fa20c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200b50b0 con 0x7fa20c004d40 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20c00baa0 con 0x7fa20c0040d0 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20c00bca0 con 0x7fa2200b6550 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa226bab640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 0x7fa20c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa2263aa640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2200b6550 0x7fa20c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa225ba9640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa20c0040d0 0x7fa20c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa226bab640 1 -- 172.21.15.59:0/3064345600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2200b6550 msgr2=0x7fa20c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa226bab640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2200b6550 0x7fa20c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa226bab640 1 -- 172.21.15.59:0/3064345600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa20c0040d0 msgr2=0x7fa20c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa226bab640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa20c0040d0 0x7fa20c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.830+0000 7fa226bab640 1 -- 172.21.15.59:0/3064345600 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20c003520 con 0x7fa20c004d40 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.831+0000 7fa226bab640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 0x7fa20c0051c0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c1c0 tx=0x7fa20c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a6e66cb42a2476 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.831+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218007cc0 con 0x7fa20c004d40 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.831+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa218030070 con 0x7fa20c004d40 2024-03-06T07:27:30.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.831+0000 7fa2253a8640 1 -- 172.21.15.59:0/3064345600 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218005d50 con 0x7fa20c004d40 2024-03-06T07:27:30.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.832+0000 7fa22860e640 1 -- 172.21.15.59:0/3064345600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 msgr2=0x7fa20c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.832+0000 7fa22860e640 1 --2- 172.21.15.59:0/3064345600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 0x7fa20c0051c0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c1c0 tx=0x7fa20c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:30.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.832+0000 7fa226bab640 1 -- 172.21.15.59:0/3064345600 reap_dead start 2024-03-06T07:27:30.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.832+0000 7fa22860e640 1 -- 172.21.15.59:0/3064345600 shutdown_connections 2024-03-06T07:27:30.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.832+0000 7fa22860e640 1 -- 172.21.15.59:0/3064345600 >> 172.21.15.59:0/3064345600 conn(0x7fa2200b35a0 msgr2=0x7fa2200b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.832+0000 7fa22860e640 1 -- 172.21.15.59:0/3064345600 shutdown_connections 2024-03-06T07:27:30.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.832+0000 7fa22860e640 1 -- 172.21.15.59:0/3064345600 wait complete. 2024-03-06T07:27:30.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.833+0000 7fa22860e640 1 Processor -- start 2024-03-06T07:27:30.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.833+0000 7fa22860e640 1 -- start start 2024-03-06T07:27:30.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa22860e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 0x7fa22014dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa22860e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2201572e0 0x7fa22015a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa22860e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa22015abf0 0x7fa2200c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa22860e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200cf850 con 0x7fa20c004d40 2024-03-06T07:27:30.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa22860e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200b4d40 con 0x7fa22015abf0 2024-03-06T07:27:30.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa22860e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200cf9d0 con 0x7fa2201572e0 2024-03-06T07:27:30.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa2263aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 0x7fa22014dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa2263aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 0x7fa22014dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38632/0 (socket says 172.21.15.59:38632) 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa2263aa640 1 -- 172.21.15.59:0/1992213130 learned_addr learned my addr 172.21.15.59:0/1992213130 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa2263aa640 1 -- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2201572e0 msgr2=0x7fa22015a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa2263aa640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2201572e0 0x7fa22015a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa2263aa640 1 -- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa22015abf0 msgr2=0x7fa2200c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa226bab640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa22015abf0 0x7fa2200c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa2263aa640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa22015abf0 0x7fa2200c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa2263aa640 1 -- 172.21.15.59:0/1992213130 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22015b070 con 0x7fa20c004d40 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa2263aa640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 0x7fa22014dda0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00fae0 tx=0x7fa21c00d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa216ffd640 1 -- 172.21.15.59:0/1992213130 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c003b80 con 0x7fa20c004d40 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa216ffd640 1 -- 172.21.15.59:0/1992213130 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c018070 con 0x7fa20c004d40 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.834+0000 7fa22860e640 1 -- 172.21.15.59:0/1992213130 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22015b350 con 0x7fa20c004d40 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.835+0000 7fa216ffd640 1 -- 172.21.15.59:0/1992213130 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c00d920 con 0x7fa20c004d40 2024-03-06T07:27:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.835+0000 7fa22860e640 1 -- 172.21.15.59:0/1992213130 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2200c5d40 con 0x7fa20c004d40 2024-03-06T07:27:30.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.836+0000 7fa22860e640 1 -- 172.21.15.59:0/1992213130 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2200c5f40 con 0x7fa20c004d40 2024-03-06T07:27:30.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.836+0000 7fa216ffd640 1 -- 172.21.15.59:0/1992213130 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa21c00dac0 con 0x7fa20c004d40 2024-03-06T07:27:30.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.836+0000 7fa216ffd640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa20406b500 0x7fa20406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.836+0000 7fa216ffd640 1 -- 172.21.15.59:0/1992213130 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa21c08a510 con 0x7fa20c004d40 2024-03-06T07:27:30.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.837+0000 7fa225ba9640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa20406b500 0x7fa20406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.837+0000 7fa225ba9640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa20406b500 0x7fa20406d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa2200b28b0 tx=0x7fa2100095d0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:30.841+0000 7fa216ffd640 1 -- 172.21.15.59:0/1992213130 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2200c5f40 con 0x7fa20c004d40 2024-03-06T07:27:31.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.064+0000 7fa22860e640 1 -- 172.21.15.59:0/1992213130 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa22004f330 con 0x7fa20c004d40 2024-03-06T07:27:31.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.065+0000 7fa216ffd640 1 -- 172.21.15.59:0/1992213130 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fa21c0593c0 con 0x7fa20c004d40 2024-03-06T07:27:31.067 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:31.067 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:29.134044+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:27:31.067 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 72 2024-03-06T07:27:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 -- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa20406b500 msgr2=0x7fa20406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa20406b500 0x7fa20406d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa2200b28b0 tx=0x7fa2100095d0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 -- 172.21.15.59:0/1992213130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 msgr2=0x7fa22014dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 0x7fa22014dda0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00fae0 tx=0x7fa21c00d150 comp rx=0 tx=0).stop 2024-03-06T07:27:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 -- 172.21.15.59:0/1992213130 shutdown_connections 2024-03-06T07:27:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa20406b500 0x7fa20406d9c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa22015abf0 0x7fa2200c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2201572e0 0x7fa22015a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 --2- 172.21.15.59:0/1992213130 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa20c004d40 0x7fa22014dda0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 -- 172.21.15.59:0/1992213130 >> 172.21.15.59:0/1992213130 conn(0x7fa2200b35a0 msgr2=0x7fa2200ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 -- 172.21.15.59:0/1992213130 shutdown_connections 2024-03-06T07:27:31.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:31.067+0000 7fa22860e640 1 -- 172.21.15.59:0/1992213130 wait complete. 2024-03-06T07:27:31.482 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:14.345180+0000', 'modified': '2024-03-06T07:27:29.134044+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8253}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8253': {'gid': 8253, 'name': 'c', 'rank': 0, 'incarnation': 70, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.59:6835/3442407979', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 3442407979}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 3442407979}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:31.482 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8253:up:rejoin 2024-03-06T07:27:32.483 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:32.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.582+0000 7f23e7a6f640 1 Processor -- start 2024-03-06T07:27:32.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.582+0000 7f23e7a6f640 1 -- start start 2024-03-06T07:27:32.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.583+0000 7f23e7a6f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f23e00b3060 0x7f23e015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.583+0000 7f23e7a6f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f23e015bf30 0x7f23e015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.583+0000 7f23e7a6f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e015f2d0 con 0x7f23e013dc10 2024-03-06T07:27:32.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.583+0000 7f23e7a6f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e00cf3f0 con 0x7f23e00b3060 2024-03-06T07:27:32.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.583+0000 7f23e7a6f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e015b210 con 0x7f23e015bf30 2024-03-06T07:27:32.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.583+0000 7f23e500a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f23e00b3060 0x7f23e015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.583+0000 7f23e500a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f23e00b3060 0x7f23e015e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38636/0 (socket says 172.21.15.59:38636) 2024-03-06T07:27:32.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.583+0000 7f23e500a640 1 -- 172.21.15.59:0/4176911050 learned_addr learned my addr 172.21.15.59:0/4176911050 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:32.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.583+0000 7f23e600c640 1 --2- 172.21.15.59:0/4176911050 >> v2:172.21.15.164:3300/0 conn(0x7f23e015bf30 0x7f23e015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.584+0000 7f23e500a640 1 -- 172.21.15.59:0/4176911050 >> v2:172.21.15.164:3300/0 conn(0x7f23e015bf30 msgr2=0x7f23e015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.584+0000 7f23e500a640 1 --2- 172.21.15.59:0/4176911050 >> v2:172.21.15.164:3300/0 conn(0x7f23e015bf30 0x7f23e015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.584+0000 7f23e500a640 1 -- 172.21.15.59:0/4176911050 >> v1:172.21.15.164:6789/0 conn(0x7f23e013dc10 legacy=0x7f23e00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.584+0000 7f23e500a640 1 -- 172.21.15.59:0/4176911050 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e015f580 con 0x7f23e00b3060 2024-03-06T07:27:32.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.584+0000 7f23e500a640 1 --2- 172.21.15.59:0/4176911050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e00b3060 0x7f23e015e850 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f23e00b4e00 tx=0x7f23d002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7420d6b7df5ff88d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.585+0000 7f23e4809640 1 -- 172.21.15.59:0/4176911050 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d003b070 con 0x7f23e00b3060 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.585+0000 7f23e4809640 1 -- 172.21.15.59:0/4176911050 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d0036440 con 0x7f23e00b3060 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.585+0000 7f23e4809640 1 -- 172.21.15.59:0/4176911050 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0036810 con 0x7f23e00b3060 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.585+0000 7f23e7a6f640 1 -- 172.21.15.59:0/4176911050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e00b3060 msgr2=0x7f23e015e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.585+0000 7f23e7a6f640 1 --2- 172.21.15.59:0/4176911050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e00b3060 0x7f23e015e850 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f23e00b4e00 tx=0x7f23d002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.585+0000 7f23e7a6f640 1 -- 172.21.15.59:0/4176911050 shutdown_connections 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.585+0000 7f23e7a6f640 1 --2- 172.21.15.59:0/4176911050 >> v2:172.21.15.164:3300/0 conn(0x7f23e015bf30 0x7f23e015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.585+0000 7f23e7a6f640 1 --2- 172.21.15.59:0/4176911050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e00b3060 0x7f23e015e850 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.585+0000 7f23e7a6f640 1 -- 172.21.15.59:0/4176911050 >> 172.21.15.59:0/4176911050 conn(0x7f23e00cf8a0 msgr2=0x7f23e00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.585+0000 7f23e7a6f640 1 -- 172.21.15.59:0/4176911050 shutdown_connections 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.585+0000 7f23e7a6f640 1 -- 172.21.15.59:0/4176911050 wait complete. 2024-03-06T07:27:32.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.586+0000 7f23e7a6f640 1 Processor -- start 2024-03-06T07:27:32.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.586+0000 7f23e7a6f640 1 -- start start 2024-03-06T07:27:32.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.587+0000 7f23e7a6f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e00b3060 0x7f23e01f6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.587+0000 7f23e7a6f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23e013dc10 0x7f23e013ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.587+0000 7f23e7a6f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23e015bf30 0x7f23e0159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.587+0000 7f23e7a6f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e013dff0 con 0x7f23e00b3060 2024-03-06T07:27:32.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.587+0000 7f23e7a6f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e015b730 con 0x7f23e013dc10 2024-03-06T07:27:32.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.587+0000 7f23e7a6f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e00cd7a0 con 0x7f23e015bf30 2024-03-06T07:27:32.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.587+0000 7f23e580b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e00b3060 0x7f23e01f6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.587+0000 7f23e580b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e00b3060 0x7f23e01f6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38642/0 (socket says 172.21.15.59:38642) 2024-03-06T07:27:32.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.587+0000 7f23e580b640 1 -- 172.21.15.59:0/3062327335 learned_addr learned my addr 172.21.15.59:0/3062327335 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:32.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.587+0000 7f23e500a640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23e013dc10 0x7f23e013ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.587+0000 7f23e600c640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23e015bf30 0x7f23e0159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23e580b640 1 -- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23e015bf30 msgr2=0x7f23e0159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23e580b640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23e015bf30 0x7f23e0159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23e580b640 1 -- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23e013dc10 msgr2=0x7f23e013ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23e580b640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23e013dc10 0x7f23e013ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23e580b640 1 -- 172.21.15.59:0/3062327335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e0152380 con 0x7f23e00b3060 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23e600c640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23e015bf30 0x7f23e0159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23e580b640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e00b3060 0x7f23e01f6e70 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00c430 tx=0x7f23dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23d67fc640 1 -- 172.21.15.59:0/3062327335 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc0052d0 con 0x7f23e00b3060 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23d67fc640 1 -- 172.21.15.59:0/3062327335 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23dc005490 con 0x7f23e00b3060 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23e7a6f640 1 -- 172.21.15.59:0/3062327335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e01409e0 con 0x7f23e00b3060 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23d67fc640 1 -- 172.21.15.59:0/3062327335 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc007260 con 0x7f23e00b3060 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.588+0000 7f23e7a6f640 1 -- 172.21.15.59:0/3062327335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e0152640 con 0x7f23e00b3060 2024-03-06T07:27:32.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.589+0000 7f23e7a6f640 1 -- 172.21.15.59:0/3062327335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e004f330 con 0x7f23e00b3060 2024-03-06T07:27:32.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.590+0000 7f23d67fc640 1 -- 172.21.15.59:0/3062327335 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23dc007400 con 0x7f23e00b3060 2024-03-06T07:27:32.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.590+0000 7f23d67fc640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23c406b4d0 0x7f23c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.590+0000 7f23d67fc640 1 -- 172.21.15.59:0/3062327335 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f23dc089650 con 0x7f23e00b3060 2024-03-06T07:27:32.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.590+0000 7f23e500a640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23c406b4d0 0x7f23c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.591+0000 7f23e500a640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23c406b4d0 0x7f23c406d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f23d0038000 tx=0x7f23d0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.594+0000 7f23d67fc640 1 -- 172.21.15.59:0/3062327335 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23dc057eb0 con 0x7f23e00b3060 2024-03-06T07:27:32.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.810+0000 7f23e7a6f640 1 -- 172.21.15.59:0/3062327335 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23e0140d70 con 0x7f23e00b3060 2024-03-06T07:27:32.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.811+0000 7f23d67fc640 1 -- 172.21.15.59:0/3062327335 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f23dc05bf50 con 0x7f23e00b3060 2024-03-06T07:27:32.813 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:32.813 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:27:32.813 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:27:32.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.812+0000 7f23e7a6f640 1 -- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23c406b4d0 msgr2=0x7f23c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.812+0000 7f23e7a6f640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23c406b4d0 0x7f23c406d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f23d0038000 tx=0x7f23d0008040 comp rx=0 tx=0).stop 2024-03-06T07:27:32.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.812+0000 7f23e7a6f640 1 -- 172.21.15.59:0/3062327335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e00b3060 msgr2=0x7f23e01f6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.812+0000 7f23e7a6f640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e00b3060 0x7f23e01f6e70 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00c430 tx=0x7f23dc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.813+0000 7f23e7a6f640 1 -- 172.21.15.59:0/3062327335 shutdown_connections 2024-03-06T07:27:32.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.813+0000 7f23e7a6f640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23c406b4d0 0x7f23c406d990 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.813+0000 7f23e7a6f640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23e015bf30 0x7f23e0159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.813+0000 7f23e7a6f640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23e013dc10 0x7f23e013ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.813+0000 7f23e7a6f640 1 --2- 172.21.15.59:0/3062327335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e00b3060 0x7f23e01f6e70 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.813+0000 7f23e7a6f640 1 -- 172.21.15.59:0/3062327335 >> 172.21.15.59:0/3062327335 conn(0x7f23e00cf8a0 msgr2=0x7f23e0071040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.813+0000 7f23e7a6f640 1 -- 172.21.15.59:0/3062327335 shutdown_connections 2024-03-06T07:27:32.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:32.813+0000 7f23e7a6f640 1 -- 172.21.15.59:0/3062327335 wait complete. 2024-03-06T07:27:33.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:14.345180+0000', 'modified': '2024-03-06T07:27:31.136752+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8253}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8253': {'gid': 8253, 'name': 'c', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.59:6835/3442407979', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 3442407979}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 3442407979}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8253, 'qdb_cluster': [8253]} 2024-03-06T07:27:33.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:33.228 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:27:33.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.336+0000 7f39d0ac3640 1 Processor -- start 2024-03-06T07:27:33.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.336+0000 7f39d0ac3640 1 -- start start 2024-03-06T07:27:33.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.337+0000 7f39d0ac3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39cc0b4460 0x7f39cc0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.337+0000 7f39d0ac3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc0d22c0 con 0x7f39cc161cf0 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.337+0000 7f39d0ac3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc0d25a0 con 0x7f39cc162970 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.337+0000 7f39d0ac3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc161b00 con 0x7f39cc0b4460 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.337+0000 7f39cad9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f39cc161cf0 0x7f39cc0cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:53706/0 (socket says 172.21.15.59:53706) 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.337+0000 7f39cad9d640 1 -- 172.21.15.59:0/307599170 learned_addr learned my addr 172.21.15.59:0/307599170 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.337+0000 7f39ca59c640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39cc0b4460 0x7f39cc0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.337+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856350749 0 0) 0x7f39cc0d22c0 con 0x7f39cc161cf0 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.337+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b4003520 con 0x7f39cc161cf0 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3051652000 0 0) 0x7f39b4003520 con 0x7f39cc161cf0 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39cc0b4460 msgr2=0x7f39cc0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39cc0b4460 0x7f39cc0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 >> v1:172.21.15.164:6789/0 conn(0x7f39cc162970 legacy=0x7f39cc0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc0d2880 con 0x7f39cc161cf0 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f39c40019f0 con 0x7f39cc161cf0 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 >> v1:172.21.15.59:6789/0 conn(0x7f39cc161cf0 legacy=0x7f39cc0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39cc0b4460 0x7f39b40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b4003f20 0x7f39b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39b4004d40 0x7f39b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc161b00 con 0x7f39b4003f20 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b400ba20 con 0x7f39b4004d40 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b400bc20 con 0x7f39cc0b4460 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b4003f20 0x7f39b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39ca59c640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39cc0b4460 0x7f39b40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39cad9d640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39b4004d40 0x7f39b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c9d9b640 1 -- 172.21.15.59:0/307599170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39cc0b4460 msgr2=0x7f39b40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39cc0b4460 0x7f39b40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c9d9b640 1 -- 172.21.15.59:0/307599170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39b4004d40 msgr2=0x7f39b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39b4004d40 0x7f39b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.338+0000 7f39c9d9b640 1 -- 172.21.15.59:0/307599170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b4003520 con 0x7f39b4003f20 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.339+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b4003f20 0x7f39b4004800 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f39c0001830 tx=0x7f39b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfc824c01ab4070 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.339+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c0001430 con 0x7f39b4003f20 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.339+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c0032620 con 0x7f39b4003f20 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.339+0000 7f39c959a640 1 -- 172.21.15.59:0/307599170 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c0032a10 con 0x7f39b4003f20 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.339+0000 7f39d0ac3640 1 -- 172.21.15.59:0/307599170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b4003f20 msgr2=0x7f39b4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.339+0000 7f39d0ac3640 1 --2- 172.21.15.59:0/307599170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b4003f20 0x7f39b4004800 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f39c0001830 tx=0x7f39b4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.339+0000 7f39cad9d640 1 -- 172.21.15.59:0/307599170 reap_dead start 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.340+0000 7f39d0ac3640 1 -- 172.21.15.59:0/307599170 shutdown_connections 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.340+0000 7f39d0ac3640 1 -- 172.21.15.59:0/307599170 >> 172.21.15.59:0/307599170 conn(0x7f39cc0d3ce0 msgr2=0x7f39cc0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.340+0000 7f39d0ac3640 1 -- 172.21.15.59:0/307599170 shutdown_connections 2024-03-06T07:27:33.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.340+0000 7f39d0ac3640 1 -- 172.21.15.59:0/307599170 wait complete. 2024-03-06T07:27:33.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.341+0000 7f39d0ac3640 1 Processor -- start 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.341+0000 7f39d0ac3640 1 -- start start 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39d0ac3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39b4003f20 0x7f39cc1f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39d0ac3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39cc0b26b0 0x7f39cc0b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39d0ac3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39cc0c9ef0 0x7f39cc0b6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39d0ac3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc0ce950 con 0x7f39cc0b26b0 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39d0ac3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc0cecc0 con 0x7f39cc0c9ef0 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39d0ac3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc0b4840 con 0x7f39b4003f20 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39c9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39cc0b26b0 0x7f39cc0b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39c9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39cc0b26b0 0x7f39cc0b0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38666/0 (socket says 172.21.15.59:38666) 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39c9d9b640 1 -- 172.21.15.59:0/2430748794 learned_addr learned my addr 172.21.15.59:0/2430748794 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39ca59c640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39b4003f20 0x7f39cc1f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39cad9d640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39cc0c9ef0 0x7f39cc0b6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39c9d9b640 1 -- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39b4003f20 msgr2=0x7f39cc1f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39b4003f20 0x7f39cc1f4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39c9d9b640 1 -- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39cc0c9ef0 msgr2=0x7f39cc0b6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39cc0c9ef0 0x7f39cc0b6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39c9d9b640 1 -- 172.21.15.59:0/2430748794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc0b1610 con 0x7f39cc0b26b0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39cad9d640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39cc0c9ef0 0x7f39cc0b6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39cc0b26b0 0x7f39cc0b0ee0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f39c0001890 tx=0x7f39c0001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39baffd640 1 -- 172.21.15.59:0/2430748794 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c00035f0 con 0x7f39cc0b26b0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39baffd640 1 -- 172.21.15.59:0/2430748794 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c0003790 con 0x7f39cc0b26b0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39d0ac3640 1 -- 172.21.15.59:0/2430748794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39cc0b18f0 con 0x7f39cc0b26b0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.342+0000 7f39baffd640 1 -- 172.21.15.59:0/2430748794 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c0003b60 con 0x7f39cc0b26b0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.343+0000 7f39d0ac3640 1 -- 172.21.15.59:0/2430748794 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39cc0b1eb0 con 0x7f39cc0b26b0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.344+0000 7f39baffd640 1 -- 172.21.15.59:0/2430748794 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39c0003d00 con 0x7f39cc0b26b0 2024-03-06T07:27:33.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.344+0000 7f39baffd640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39ac06b500 0x7f39ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.344+0000 7f39baffd640 1 -- 172.21.15.59:0/2430748794 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f39c00ac360 con 0x7f39cc0b26b0 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.344+0000 7f39ca59c640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39ac06b500 0x7f39ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.345+0000 7f39ca59c640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39ac06b500 0x7f39ac06d9c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0cb290 tx=0x7f39bc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.346+0000 7f39d3784740 1 Processor -- start 2024-03-06T07:27:33.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.346+0000 7f39d3784740 1 -- start start 2024-03-06T07:27:33.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39d3784740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f976870 0x56448f976c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39d3784740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f97a8c0 0x56448f97c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39d3784740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f9a6600 0x56448f9a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39d3784740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56448f8e06b0 con 0x56448f976870 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39d3784740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56448f889a60 con 0x56448f9a6600 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39d3784740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x56448f8dc750 con 0x56448f97a8c0 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39ca59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f976870 0x56448f976c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39ca59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f976870 0x56448f976c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38672/0 (socket says 172.21.15.59:38672) 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39ca59c640 1 -- 172.21.15.59:0/473557980 learned_addr learned my addr 172.21.15.59:0/473557980 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/473557980 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f97a8c0 0x56448f97c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39cad9d640 1 --2- 172.21.15.59:0/473557980 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f9a6600 0x56448f9a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39ca59c640 1 -- 172.21.15.59:0/473557980 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f97a8c0 msgr2=0x56448f97c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39ca59c640 1 --2- 172.21.15.59:0/473557980 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f97a8c0 0x56448f97c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39ca59c640 1 -- 172.21.15.59:0/473557980 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f9a6600 msgr2=0x56448f9a6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39ca59c640 1 --2- 172.21.15.59:0/473557980 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f9a6600 0x56448f9a6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39ca59c640 1 -- 172.21.15.59:0/473557980 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56448f8a7f30 con 0x56448f976870 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39ca59c640 1 --2- 172.21.15.59:0/473557980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f976870 0x56448f976c50 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f39bc013d80 tx=0x7f39bc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189f46cf1f470164 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39abfff640 1 -- 172.21.15.59:0/473557980 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc004030 con 0x56448f976870 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39abfff640 1 -- 172.21.15.59:0/473557980 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc028070 con 0x56448f976870 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.347+0000 7f39abfff640 1 -- 172.21.15.59:0/473557980 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc0043d0 con 0x56448f976870 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.348+0000 7f39d3784740 1 -- 172.21.15.59:0/473557980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f976870 msgr2=0x56448f976c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.348+0000 7f39d3784740 1 --2- 172.21.15.59:0/473557980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f976870 0x56448f976c50 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f39bc013d80 tx=0x7f39bc016000 comp rx=0 tx=0).stop 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.348+0000 7f39d3784740 1 -- 172.21.15.59:0/473557980 shutdown_connections 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.348+0000 7f39d3784740 1 --2- 172.21.15.59:0/473557980 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f9a6600 0x56448f9a6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.348+0000 7f39d3784740 1 --2- 172.21.15.59:0/473557980 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f97a8c0 0x56448f97c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.348+0000 7f39d3784740 1 --2- 172.21.15.59:0/473557980 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f976870 0x56448f976c50 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.348+0000 7f39d3784740 1 -- 172.21.15.59:0/473557980 >> 172.21.15.59:0/473557980 conn(0x56448f9788f0 msgr2=0x56448f97a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.348+0000 7f39d3784740 1 -- 172.21.15.59:0/473557980 shutdown_connections 2024-03-06T07:27:33.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.348+0000 7f39d3784740 1 -- 172.21.15.59:0/473557980 wait complete. 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.349+0000 7f39d3784740 1 -- start start 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.349+0000 7f39d3784740 1 Processor -- start 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39d3784740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f9a6600 0x56448f99f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39d3784740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99f880 0x56448f9a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39d3784740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 0x56448f9a1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39d3784740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56448f97ad50 con 0x56448f99f880 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39d3784740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56448f89cea0 con 0x56448f9a6600 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39d3784740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x56448f97b030 con 0x56448f99c1d0 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39d3784740 10 client.0 _ll_register_callbacks cb 0x56448f89c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39c9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99f880 0x56448f9a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39c9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99f880 0x56448f9a0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38676/0 (socket says 172.21.15.59:38676) 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39c9d9b640 1 -- 172.21.15.59:0/3220958938 learned_addr learned my addr 172.21.15.59:0/3220958938 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39ca59c640 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f9a6600 0x56448f99f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39cad9d640 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 0x56448f9a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39c9d9b640 1 -- 172.21.15.59:0/3220958938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 msgr2=0x56448f9a1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 0x56448f9a1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39c9d9b640 1 -- 172.21.15.59:0/3220958938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f9a6600 msgr2=0x56448f99f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f9a6600 0x56448f99f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39c9d9b640 1 -- 172.21.15.59:0/3220958938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56448f89d270 con 0x56448f99f880 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39cad9d640 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 0x56448f9a1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99f880 0x56448f9a0e10 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f39c0008fd0 tx=0x7f39c007a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5156430aa73f1a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.350+0000 7f39a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39a97fa640 1 -- 172.21.15.59:0/3220958938 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c0092020 con 0x56448f99f880 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39a97fa640 1 -- 172.21.15.59:0/3220958938 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f39c007ad70 con 0x56448f99f880 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39d3784740 1 -- 172.21.15.59:0/3220958938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39bc028070 con 0x56448f99f880 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39a97fa640 1 -- 172.21.15.59:0/3220958938 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39c007ab10 con 0x56448f99f880 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39d3784740 10 client.8289 fetch_fsmap learned FSMap version 73 2024-03-06T07:27:33.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39d3784740 1 -- 172.21.15.59:0/3220958938 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x56448f5f52d0 con 0x56448f99f880 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39a97fa640 1 -- 172.21.15.59:0/3220958938 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc028070 con 0x56448f99f880 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39a97fa640 1 -- 172.21.15.59:0/3220958938 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 73) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f39c0082d90 con 0x56448f99f880 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39d3784740 10 client.8289 fetch_fsmap finished waiting for FSMap version 73 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39d3784740 10 client.8289 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39d3784740 20 client.8289 populate_metadata read hostname 'smithi059' 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39d3784740 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.59:6834/3442407979,v1:172.21.15.59:6835/3442407979] conn(0x56448f9a7de0 0x56448f9a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39d3784740 4 client.8289 mds_command: new command op to 8253 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39d3784740 1 -- 172.21.15.59:0/3220958938 --> [v2:172.21.15.59:6834/3442407979,v1:172.21.15.59:6835/3442407979] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56448f8edbf0 con 0x56448f9a7de0 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.351+0000 7f39ca59c640 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.59:6834/3442407979,v1:172.21.15.59:6835/3442407979] conn(0x56448f9a7de0 0x56448f9a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39a97fa640 10 client.8289 ms_handle_connect on v2:172.21.15.59:6834/3442407979 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39ca59c640 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.59:6834/3442407979,v1:172.21.15.59:6835/3442407979] conn(0x56448f9a7de0 0x56448f9a81c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62e3a881950aa79d server_cookie=ef143f3c783d183 in_seq=0 out_seq=0 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39a97fa640 1 -- 172.21.15.59:0/3220958938 <== mds.0 v2:172.21.15.59:6834/3442407979 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56448f8edbf0 con 0x56448f9a7de0 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39a97fa640 10 client.8289 handle_command_reply: tid=0 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39d3784740 1 client.8289 shutdown 2024-03-06T07:27:33.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39d3784740 1 -- 172.21.15.59:0/3220958938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99f880 msgr2=0x56448f9a0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39d3784740 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99f880 0x56448f9a0e10 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f39c0008fd0 tx=0x7f39c007a710 comp rx=0 tx=0).stop 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39d3784740 1 -- 172.21.15.59:0/3220958938 shutdown_connections 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39d3784740 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.59:6834/3442407979,v1:172.21.15.59:6835/3442407979] conn(0x56448f9a7de0 0x56448f9a81c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39d3784740 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 0x56448f9a1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39d3784740 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99f880 0x56448f9a0e10 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39a97fa640 0 client.8289 ms_handle_reset on v2:172.21.15.59:6834/3442407979 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39d3784740 1 --2- 172.21.15.59:0/3220958938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f9a6600 0x56448f99f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39d3784740 1 -- 172.21.15.59:0/3220958938 >> 172.21.15.59:0/3220958938 conn(0x56448f976870 msgr2=0x56448f9a45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.352+0000 7f39d3784740 1 -- 172.21.15.59:0/3220958938 shutdown_connections 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.353+0000 7f39d3784740 1 -- 172.21.15.59:0/3220958938 wait complete. 2024-03-06T07:27:33.362 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.353+0000 7f39d3784740 20 client.8289 trim_cache size 0 max 16384 2024-03-06T07:27:33.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.369+0000 7f39d3784740 1 Processor -- start 2024-03-06T07:27:33.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.369+0000 7f39d3784740 1 -- start start 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39d3784740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99c1d0 0x56448f9a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39d3784740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f99f880 0x56448f97a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39d3784740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f9a6600 0x56448f9c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39d3784740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc1ffa40 con 0x56448f99c1d0 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39d3784740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc1f5700 con 0x56448f99f880 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39d3784740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc1ff560 con 0x56448f9a6600 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39ca59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99c1d0 0x56448f9a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39ca59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99c1d0 0x56448f9a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38686/0 (socket says 172.21.15.59:38686) 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39ca59c640 1 -- 172.21.15.59:0/1697332898 learned_addr learned my addr 172.21.15.59:0/1697332898 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39ca59c640 1 -- 172.21.15.59:0/1697332898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f9a6600 msgr2=0x56448f9c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39ca59c640 1 --2- 172.21.15.59:0/1697332898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f9a6600 0x56448f9c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39ca59c640 1 -- 172.21.15.59:0/1697332898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f99f880 msgr2=0x56448f97a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39ca59c640 1 --2- 172.21.15.59:0/1697332898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f99f880 0x56448f97a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.370+0000 7f39ca59c640 1 -- 172.21.15.59:0/1697332898 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56448f99f6b0 con 0x56448f99c1d0 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39ca59c640 1 --2- 172.21.15.59:0/1697332898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99c1d0 0x56448f9a4d00 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0097e0 tx=0x7f39bc01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b803bf2f9bed5c2c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39ab7fe640 1 -- 172.21.15.59:0/1697332898 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc037260 con 0x56448f99c1d0 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39ab7fe640 1 -- 172.21.15.59:0/1697332898 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc028040 con 0x56448f99c1d0 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39ab7fe640 1 -- 172.21.15.59:0/1697332898 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc004230 con 0x56448f99c1d0 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39d3784740 1 -- 172.21.15.59:0/1697332898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99c1d0 msgr2=0x56448f9a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39d3784740 1 --2- 172.21.15.59:0/1697332898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99c1d0 0x56448f9a4d00 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0097e0 tx=0x7f39bc01a000 comp rx=0 tx=0).stop 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39d3784740 1 -- 172.21.15.59:0/1697332898 shutdown_connections 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39d3784740 1 --2- 172.21.15.59:0/1697332898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f9a6600 0x56448f9c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39d3784740 1 --2- 172.21.15.59:0/1697332898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f99f880 0x56448f97a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39d3784740 1 --2- 172.21.15.59:0/1697332898 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f99c1d0 0x56448f9a4d00 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39d3784740 1 -- 172.21.15.59:0/1697332898 >> 172.21.15.59:0/1697332898 conn(0x56448f976870 msgr2=0x56448f99b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39d3784740 1 -- 172.21.15.59:0/1697332898 shutdown_connections 2024-03-06T07:27:33.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.371+0000 7f39d3784740 1 -- 172.21.15.59:0/1697332898 wait complete. 2024-03-06T07:27:33.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.372+0000 7f39d3784740 1 -- start start 2024-03-06T07:27:33.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.372+0000 7f39d3784740 1 Processor -- start 2024-03-06T07:27:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.373+0000 7f39d3784740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 0x56448fa30df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.373+0000 7f39d3784740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f99f880 0x56448fa31330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.373+0000 7f39d3784740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f9a6600 0x56448f9e49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.373+0000 7f39d3784740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x56448f97ad50 con 0x56448f9a6600 2024-03-06T07:27:33.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.373+0000 7f39d3784740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56448f89cea0 con 0x56448f99f880 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.373+0000 7f39d3784740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x56448f9a9070 con 0x56448f99c1d0 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.373+0000 7f39d3784740 10 client.0 _ll_register_callbacks cb 0x7f39c0076e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.373+0000 7f39cad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f9a6600 0x56448f9e49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39cad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f9a6600 0x56448f9e49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38702/0 (socket says 172.21.15.59:38702) 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39cad9d640 1 -- 172.21.15.59:0/659020947 learned_addr learned my addr 172.21.15.59:0/659020947 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f99f880 0x56448fa31330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39ca59c640 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 0x56448fa30df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39cad9d640 1 -- 172.21.15.59:0/659020947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 msgr2=0x56448fa30df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39cad9d640 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 0x56448fa30df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39cad9d640 1 -- 172.21.15.59:0/659020947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f99f880 msgr2=0x56448fa31330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39cad9d640 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f99f880 0x56448fa31330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39cad9d640 1 -- 172.21.15.59:0/659020947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56448f6bf6b0 con 0x56448f9a6600 2024-03-06T07:27:33.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39c9d9b640 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f99f880 0x56448fa31330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39ca59c640 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 0x56448fa30df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39cad9d640 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f9a6600 0x56448f9e49d0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0d17e0 tx=0x7f39c4007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d837b2bf80951821 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39a97fa640 1 -- 172.21.15.59:0/659020947 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4010040 con 0x56448f9a6600 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39a97fa640 1 -- 172.21.15.59:0/659020947 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f39c4011b70 con 0x56448f9a6600 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.374+0000 7f39d3784740 1 -- 172.21.15.59:0/659020947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39bc028040 con 0x56448f9a6600 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39a97fa640 1 -- 172.21.15.59:0/659020947 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39c4022050 con 0x56448f9a6600 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39d3784740 10 client.8295 fetch_fsmap learned FSMap version 73 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39d3784740 1 -- 172.21.15.59:0/659020947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x56448f9a0f00 con 0x56448f9a6600 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39a97fa640 1 -- 172.21.15.59:0/659020947 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc028040 con 0x56448f9a6600 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39a97fa640 1 -- 172.21.15.59:0/659020947 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 73) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f39c4025020 con 0x56448f9a6600 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39d3784740 10 client.8295 fetch_fsmap finished waiting for FSMap version 73 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39d3784740 10 client.8295 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:33.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39d3784740 20 client.8295 populate_metadata read hostname 'smithi059' 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39d3784740 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.59:6834/3442407979,v1:172.21.15.59:6835/3442407979] conn(0x56448f9d1610 0x56448f9d3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39d3784740 4 client.8295 mds_command: new command op to 8253 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39d3784740 1 -- 172.21.15.59:0/659020947 --> [v2:172.21.15.59:6834/3442407979,v1:172.21.15.59:6835/3442407979] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56448f99e8f0 con 0x56448f9d1610 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39ca59c640 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.59:6834/3442407979,v1:172.21.15.59:6835/3442407979] conn(0x56448f9d1610 0x56448f9d3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39a97fa640 10 client.8295 ms_handle_connect on v2:172.21.15.59:6834/3442407979 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.375+0000 7f39ca59c640 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.59:6834/3442407979,v1:172.21.15.59:6835/3442407979] conn(0x56448f9d1610 0x56448f9d3ae0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=879df7a5096519c server_cookie=533cc452280811ae in_seq=0 out_seq=0 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39a97fa640 1 -- 172.21.15.59:0/659020947 <== mds.0 v2:172.21.15.59:6834/3442407979 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x56448f99e8f0 con 0x56448f9d1610 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39a97fa640 10 client.8295 handle_command_reply: tid=0 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 1 client.8295 shutdown 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 1 -- 172.21.15.59:0/659020947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f9a6600 msgr2=0x56448f9e49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f9a6600 0x56448f9e49d0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0d17e0 tx=0x7f39c4007a90 comp rx=0 tx=0).stop 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 1 -- 172.21.15.59:0/659020947 shutdown_connections 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.59:6834/3442407979,v1:172.21.15.59:6835/3442407979] conn(0x56448f9d1610 0x56448f9d3ae0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x56448f9a6600 0x56448f9e49d0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39a97fa640 0 client.8295 ms_handle_reset on v2:172.21.15.59:6834/3442407979 2024-03-06T07:27:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56448f99f880 0x56448fa31330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 1 --2- 172.21.15.59:0/659020947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56448f99c1d0 0x56448fa30df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 1 -- 172.21.15.59:0/659020947 >> 172.21.15.59:0/659020947 conn(0x56448f976870 msgr2=0x56448f971e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 1 -- 172.21.15.59:0/659020947 shutdown_connections 2024-03-06T07:27:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 1 -- 172.21.15.59:0/659020947 wait complete. 2024-03-06T07:27:33.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.376+0000 7f39d3784740 20 client.8295 trim_cache size 0 max 16384 2024-03-06T07:27:33.384 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:27:33.384 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:27:33.384 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:27:33.385 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 8253, 2024-03-06T07:27:33.385 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:27:33.385 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:27:33.385 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:27:33.385 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 5.2441894769999999, 2024-03-06T07:27:33.385 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 73, 2024-03-06T07:27:33.385 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 112, 2024-03-06T07:27:33.385 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 112, 2024-03-06T07:27:33.385 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 6.1073022540000004 2024-03-06T07:27:33.385 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:27:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 -- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39ac06b500 msgr2=0x7f39ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39ac06b500 0x7f39ac06d9c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0cb290 tx=0x7f39bc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 -- 172.21.15.59:0/2430748794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39cc0b26b0 msgr2=0x7f39cc0b0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39cc0b26b0 0x7f39cc0b0ee0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f39c0001890 tx=0x7f39c0001230 comp rx=0 tx=0).stop 2024-03-06T07:27:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 -- 172.21.15.59:0/2430748794 shutdown_connections 2024-03-06T07:27:33.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39ac06b500 0x7f39ac06d9c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39cc0c9ef0 0x7f39cc0b6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39cc0b26b0 0x7f39cc0b0ee0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 --2- 172.21.15.59:0/2430748794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39b4003f20 0x7f39cc1f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 -- 172.21.15.59:0/2430748794 >> 172.21.15.59:0/2430748794 conn(0x7f39cc0d3ce0 msgr2=0x7f39cc15f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 -- 172.21.15.59:0/2430748794 shutdown_connections 2024-03-06T07:27:33.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.377+0000 7f39d0ac3640 1 -- 172.21.15.59:0/2430748794 wait complete. 2024-03-06T07:27:33.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:33.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.517+0000 7f34aeafb640 1 Processor -- start 2024-03-06T07:27:33.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.517+0000 7f34aeafb640 1 -- start start 2024-03-06T07:27:33.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34aeafb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34a80ce920 0x7f34a815e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34aeafb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f34a806b600 0x7f34a8162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34aeafb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a815ea30 con 0x7f34a806b220 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34aeafb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a8161b70 con 0x7f34a806b600 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34aeafb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a815ed10 con 0x7f34a80ce920 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34ad098640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f34a806b600 0x7f34a8162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34ac897640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f34a806b220 0x7f34a8071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:53908/0 (socket says 172.21.15.59:53908) 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34ac897640 1 -- 172.21.15.59:0/4131054607 learned_addr learned my addr 172.21.15.59:0/4131054607 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34ad098640 1 --2- 172.21.15.59:0/4131054607 >> v2:172.21.15.59:3300/0 conn(0x7f34a806b600 0x7f34a8162000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33184/0 (socket says 172.21.15.59:33184) 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34ad098640 1 -- 172.21.15.59:0/4131054607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34a80ce920 msgr2=0x7f34a815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34a7fff640 1 --2- 172.21.15.59:0/4131054607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34a80ce920 0x7f34a815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34ad098640 1 --2- 172.21.15.59:0/4131054607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34a80ce920 0x7f34a815e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34ad098640 1 -- 172.21.15.59:0/4131054607 >> v1:172.21.15.59:6789/0 conn(0x7f34a806b220 legacy=0x7f34a8071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34a77fe640 1 -- 172.21.15.59:0/4131054607 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900799931 0 0) 0x7f34a815ea30 con 0x7f34a806b220 2024-03-06T07:27:33.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.518+0000 7f34ad098640 1 -- 172.21.15.59:0/4131054607 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a815eff0 con 0x7f34a806b600 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.519+0000 7f34ad098640 1 --2- 172.21.15.59:0/4131054607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a806b600 0x7f34a8162000 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f34980097a0 tx=0x7f349802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9c04d47195e6bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.519+0000 7f34a77fe640 1 -- 172.21.15.59:0/4131054607 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3498035020 con 0x7f34a806b600 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.519+0000 7f34a77fe640 1 -- 172.21.15.59:0/4131054607 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3498031830 con 0x7f34a806b600 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.519+0000 7f34a77fe640 1 -- 172.21.15.59:0/4131054607 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3498031c00 con 0x7f34a806b600 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.520+0000 7f34aeafb640 1 -- 172.21.15.59:0/4131054607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a806b600 msgr2=0x7f34a8162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.520+0000 7f34aeafb640 1 --2- 172.21.15.59:0/4131054607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a806b600 0x7f34a8162000 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f34980097a0 tx=0x7f349802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.520+0000 7f34aeafb640 1 -- 172.21.15.59:0/4131054607 shutdown_connections 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.520+0000 7f34aeafb640 1 --2- 172.21.15.59:0/4131054607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a806b600 0x7f34a8162000 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.520+0000 7f34aeafb640 1 --2- 172.21.15.59:0/4131054607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34a80ce920 0x7f34a815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.520+0000 7f34aeafb640 1 -- 172.21.15.59:0/4131054607 >> 172.21.15.59:0/4131054607 conn(0x7f34a806c0c0 msgr2=0x7f34a806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.520+0000 7f34aeafb640 1 -- 172.21.15.59:0/4131054607 shutdown_connections 2024-03-06T07:27:33.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.520+0000 7f34aeafb640 1 -- 172.21.15.59:0/4131054607 wait complete. 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.521+0000 7f34aeafb640 1 Processor -- start 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.521+0000 7f34aeafb640 1 -- start start 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.522+0000 7f34aeafb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a806b220 0x7f34a81f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.522+0000 7f34aeafb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34a806b600 0x7f34a81609c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.522+0000 7f34aeafb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34a80ce920 0x7f34a8160f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.522+0000 7f34aeafb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a8072090 con 0x7f34a806b220 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.522+0000 7f34aeafb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a80ce200 con 0x7f34a80ce920 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.522+0000 7f34aeafb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a81f7630 con 0x7f34a806b600 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.522+0000 7f34ac897640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a806b220 0x7f34a81f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.522+0000 7f34ac897640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a806b220 0x7f34a81f70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33190/0 (socket says 172.21.15.59:33190) 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.522+0000 7f34ac897640 1 -- 172.21.15.59:0/1242739889 learned_addr learned my addr 172.21.15.59:0/1242739889 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:33.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34a7fff640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34a806b600 0x7f34a81609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34ad098640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34a80ce920 0x7f34a8160f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34ac897640 1 -- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34a806b600 msgr2=0x7f34a81609c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34ac897640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34a806b600 0x7f34a81609c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34ac897640 1 -- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34a80ce920 msgr2=0x7f34a8160f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34ac897640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34a80ce920 0x7f34a8160f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34ac897640 1 -- 172.21.15.59:0/1242739889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a80af5e0 con 0x7f34a806b220 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34a7fff640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34a806b600 0x7f34a81609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34ad098640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34a80ce920 0x7f34a8160f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34ac897640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a806b220 0x7f34a81f70f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f34a000c7c0 tx=0x7f34a000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34a57fa640 1 -- 172.21.15.59:0/1242739889 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0018070 con 0x7f34a806b220 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34a57fa640 1 -- 172.21.15.59:0/1242739889 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a000a640 con 0x7f34a806b220 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34aeafb640 1 -- 172.21.15.59:0/1242739889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a80af8c0 con 0x7f34a806b220 2024-03-06T07:27:33.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34a57fa640 1 -- 172.21.15.59:0/1242739889 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a000aa30 con 0x7f34a806b220 2024-03-06T07:27:33.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.523+0000 7f34aeafb640 1 -- 172.21.15.59:0/1242739889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a80afe80 con 0x7f34a806b220 2024-03-06T07:27:33.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.524+0000 7f34aeafb640 1 -- 172.21.15.59:0/1242739889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a804f330 con 0x7f34a806b220 2024-03-06T07:27:33.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.525+0000 7f34a57fa640 1 -- 172.21.15.59:0/1242739889 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34a001b070 con 0x7f34a806b220 2024-03-06T07:27:33.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.525+0000 7f34a57fa640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f349406b4d0 0x7f349406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.525+0000 7f34a57fa640 1 -- 172.21.15.59:0/1242739889 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f34a00898b0 con 0x7f34a806b220 2024-03-06T07:27:33.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.525+0000 7f34a7fff640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f349406b4d0 0x7f349406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.526+0000 7f34a7fff640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f349406b4d0 0x7f349406d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f34a80b08d0 tx=0x7f349c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.529+0000 7f34a57fa640 1 -- 172.21.15.59:0/1242739889 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34a00586d0 con 0x7f34a806b220 2024-03-06T07:27:33.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.855+0000 7f34aeafb640 1 -- 172.21.15.59:0/1242739889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f34a80cdf00 con 0x7f34a806b220 2024-03-06T07:27:33.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.855+0000 7f34a57fa640 1 -- 172.21.15.59:0/1242739889 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7f34a0057fd0 con 0x7f34a806b220 2024-03-06T07:27:33.857 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 -- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f349406b4d0 msgr2=0x7f349406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f349406b4d0 0x7f349406d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f34a80b08d0 tx=0x7f349c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 -- 172.21.15.59:0/1242739889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a806b220 msgr2=0x7f34a81f70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a806b220 0x7f34a81f70f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f34a000c7c0 tx=0x7f34a000cc80 comp rx=0 tx=0).stop 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 -- 172.21.15.59:0/1242739889 shutdown_connections 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f349406b4d0 0x7f349406d990 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34a80ce920 0x7f34a8160f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34a806b600 0x7f34a81609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 --2- 172.21.15.59:0/1242739889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34a806b220 0x7f34a81f70f0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 -- 172.21.15.59:0/1242739889 >> 172.21.15.59:0/1242739889 conn(0x7f34a806c0c0 msgr2=0x7f34a806f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 -- 172.21.15.59:0/1242739889 shutdown_connections 2024-03-06T07:27:33.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:33.858+0000 7f34aeafb640 1 -- 172.21.15.59:0/1242739889 wait complete. 2024-03-06T07:27:33.871 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:33.871 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:33.871 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:27:33.871 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:33.871 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:33.871 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:33.871 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:33.871 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:33.871 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:33.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:33.871 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:27:33.886 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:33.886 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:33.886 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:33.886 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.886 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:33.886 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.886 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:33.886 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:33.886 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42325sec preferred_lft 42325sec 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:33.887 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:33.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:33.888 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:27:33.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:33.954 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:27:34.009 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:27:34.009 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:34.009 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:27:34.009 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:27:34.009 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:34.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.009 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:27:34.009 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:27:34.010 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:27:34.010 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:27:34.010 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:27:34.010 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:27:34.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.103 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:27:34.117 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:34.118 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:34.118 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:34.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.118 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:27:34.133 INFO:tasks.ceph.mds.c.smithi059.stderr:2024-03-06T07:27:34.131+0000 7f0d99d7e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:34.160 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:34.160 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:34.161 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:34.215 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:34.215 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:34.215 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:34.270 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:27:34.337 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:34.392 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:27:34.392 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:27:34.392 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:34.393 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:34.394 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:34.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.395 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:27:34.446 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:34.447 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:34.489 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:27:34.515 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:34.571 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:27:34.571 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:27:34.571 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:27:34.571 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:27:34.571 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:27:34.571 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:27:34.571 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:27:34.571 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:27:34.571 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:27:34.571 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:27:34.572 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:27:34.573 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:34.573 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:34.573 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:34.573 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:34.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:27:34.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:34.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:34.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:34.573 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:34.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.573 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:27:34.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:27:34.573+0000 7f8d1b775480 -1 init, newargv = 0x565432f265e0 newargc=15 2024-03-06T07:27:34.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[74412]: starting ceph client 2024-03-06T07:27:34.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[74412]: starting fuse 2024-03-06T07:27:34.590 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:27:34.590 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:27:34.590 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:34.590 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:27:34.590 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:27:34.590 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:27:34.590 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:27:34.590 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:27:34.590 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:27:34.590 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:34.590 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:27:34.591 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:34.592 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:27:34.709 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.74412.asok 2024-03-06T07:27:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:34.720 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.74412.asok 2024-03-06T07:27:34.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.720 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.74412.asok status 2024-03-06T07:27:34.857 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:27:34.857 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:27:34.857 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:34.857 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:34.857 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:27:34.857 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:27:34.857 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "74412", 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 8313, 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 8313 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1641055272 2024-03-06T07:27:34.858 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1641055272 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.8313 192.168.0.1:0/1641055272", 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/1641055272", 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 73, 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 112, 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:27:34.859 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:27:34.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.866 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:27:34.866 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:34.880 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:27:34.880 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:34.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.881 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:34.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:34.946 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-03-06T07:27:35.005 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:35.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.143+0000 7fe0ef481640 1 Processor -- start 2024-03-06T07:27:35.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.144+0000 7fe0ef481640 1 -- start start 2024-03-06T07:27:35.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.144+0000 7fe0ef481640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0e80b4f70 0x7fe0e80b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.144+0000 7fe0ef481640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e8066d90 0x7fe0e80ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.144+0000 7fe0ef481640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e80b5890 con 0x7fe0e806c6d0 2024-03-06T07:27:35.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.144+0000 7fe0ef481640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e815e9f0 con 0x7fe0e80b4f70 2024-03-06T07:27:35.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.144+0000 7fe0ef481640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e80cda10 con 0x7fe0e8066d90 2024-03-06T07:27:35.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.145+0000 7fe0ed21d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0e80b4f70 0x7fe0e80b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.145+0000 7fe0ed21d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0e80b4f70 0x7fe0e80b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33202/0 (socket says 172.21.15.59:33202) 2024-03-06T07:27:35.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.145+0000 7fe0ed21d640 1 -- 172.21.15.59:0/3976250833 learned_addr learned my addr 172.21.15.59:0/3976250833 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:35.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.145+0000 7fe0ed21d640 1 -- 172.21.15.59:0/3976250833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e8066d90 msgr2=0x7fe0e80ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.145+0000 7fe0eda1e640 1 --2- 172.21.15.59:0/3976250833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e8066d90 0x7fe0e80ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.145+0000 7fe0ed21d640 1 --2- 172.21.15.59:0/3976250833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e8066d90 0x7fe0e80ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.145+0000 7fe0ed21d640 1 -- 172.21.15.59:0/3976250833 >> v1:172.21.15.164:6789/0 conn(0x7fe0e806c6d0 legacy=0x7fe0e815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.145+0000 7fe0ed21d640 1 -- 172.21.15.59:0/3976250833 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e80cd450 con 0x7fe0e80b4f70 2024-03-06T07:27:35.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0ed21d640 1 --2- 172.21.15.59:0/3976250833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e80b4f70 0x7fe0e80b5350 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc009870 tx=0x7fe0dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1827758d2c8fbdd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0dbfff640 1 -- 172.21.15.59:0/3976250833 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc035020 con 0x7fe0e80b4f70 2024-03-06T07:27:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0dbfff640 1 -- 172.21.15.59:0/3976250833 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0dc031640 con 0x7fe0e80b4f70 2024-03-06T07:27:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0dbfff640 1 -- 172.21.15.59:0/3976250833 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc02d040 con 0x7fe0e80b4f70 2024-03-06T07:27:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0ef481640 1 -- 172.21.15.59:0/3976250833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e80b4f70 msgr2=0x7fe0e80b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0ef481640 1 --2- 172.21.15.59:0/3976250833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e80b4f70 0x7fe0e80b5350 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc009870 tx=0x7fe0dc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0ef481640 1 -- 172.21.15.59:0/3976250833 shutdown_connections 2024-03-06T07:27:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0ef481640 1 --2- 172.21.15.59:0/3976250833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e8066d90 0x7fe0e80ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0ef481640 1 --2- 172.21.15.59:0/3976250833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e80b4f70 0x7fe0e80b5350 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0ef481640 1 -- 172.21.15.59:0/3976250833 >> 172.21.15.59:0/3976250833 conn(0x7fe0e80724d0 msgr2=0x7fe0e80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0ef481640 1 -- 172.21.15.59:0/3976250833 shutdown_connections 2024-03-06T07:27:35.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.146+0000 7fe0ef481640 1 -- 172.21.15.59:0/3976250833 wait complete. 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.148+0000 7fe0ef481640 1 Processor -- start 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.148+0000 7fe0ef481640 1 -- start start 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.148+0000 7fe0ef481640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e8066d90 0x7fe0e8148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.148+0000 7fe0ef481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e80b4f70 0x7fe0e8157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.148+0000 7fe0ef481640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0e8158e60 0x7fe0e81577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.148+0000 7fe0ef481640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e80cd090 con 0x7fe0e80b4f70 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.148+0000 7fe0ef481640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e815e680 con 0x7fe0e8158e60 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.148+0000 7fe0ef481640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e80cd210 con 0x7fe0e8066d90 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.148+0000 7fe0eca1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e80b4f70 0x7fe0e8157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.148+0000 7fe0eca1c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e80b4f70 0x7fe0e8157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33206/0 (socket says 172.21.15.59:33206) 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.148+0000 7fe0eca1c640 1 -- 172.21.15.59:0/1519450143 learned_addr learned my addr 172.21.15.59:0/1519450143 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0ed21d640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e8066d90 0x7fe0e8148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.154 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0eda1e640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0e8158e60 0x7fe0e81577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0eca1c640 1 -- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e8066d90 msgr2=0x7fe0e8148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0eca1c640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e8066d90 0x7fe0e8148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0eca1c640 1 -- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0e8158e60 msgr2=0x7fe0e81577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0eca1c640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0e8158e60 0x7fe0e81577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0eca1c640 1 -- 172.21.15.59:0/1519450143 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e8148aa0 con 0x7fe0e80b4f70 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0eda1e640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0e8158e60 0x7fe0e81577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0ed21d640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e8066d90 0x7fe0e8148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0eca1c640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e80b4f70 0x7fe0e8157270 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400c430 tx=0x7fe0e400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0d9ffb640 1 -- 172.21.15.59:0/1519450143 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4011790 con 0x7fe0e80b4f70 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0d9ffb640 1 -- 172.21.15.59:0/1519450143 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e4017070 con 0x7fe0e80b4f70 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0ef481640 1 -- 172.21.15.59:0/1519450143 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e8157dc0 con 0x7fe0e80b4f70 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0d9ffb640 1 -- 172.21.15.59:0/1519450143 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4011bb0 con 0x7fe0e80b4f70 2024-03-06T07:27:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.149+0000 7fe0ef481640 1 -- 172.21.15.59:0/1519450143 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e8158380 con 0x7fe0e80b4f70 2024-03-06T07:27:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.151+0000 7fe0d9ffb640 1 -- 172.21.15.59:0/1519450143 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0e401c460 con 0x7fe0e80b4f70 2024-03-06T07:27:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.151+0000 7fe0ef481640 1 -- 172.21.15.59:0/1519450143 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e806b4e0 con 0x7fe0e80b4f70 2024-03-06T07:27:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.151+0000 7fe0d9ffb640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0cc06b500 0x7fe0cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.151+0000 7fe0d9ffb640 1 -- 172.21.15.59:0/1519450143 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe0e408a0f0 con 0x7fe0e80b4f70 2024-03-06T07:27:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.151+0000 7fe0ed21d640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0cc06b500 0x7fe0cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.152+0000 7fe0ed21d640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0cc06b500 0x7fe0cc06d9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc036000 tx=0x7fe0dc03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.155+0000 7fe0d9ffb640 1 -- 172.21.15.59:0/1519450143 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0e806b4e0 con 0x7fe0e80b4f70 2024-03-06T07:27:35.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.383+0000 7fe0ef481640 1 -- 172.21.15.59:0/1519450143 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0e80b1860 con 0x7fe0e80b4f70 2024-03-06T07:27:35.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.384+0000 7fe0d9ffb640 1 -- 172.21.15.59:0/1519450143 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fe0e8148aa0 con 0x7fe0e80b4f70 2024-03-06T07:27:35.386 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:35.386 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:27:35.386 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:27:35.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 -- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0cc06b500 msgr2=0x7fe0cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0cc06b500 0x7fe0cc06d9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc036000 tx=0x7fe0dc03c040 comp rx=0 tx=0).stop 2024-03-06T07:27:35.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 -- 172.21.15.59:0/1519450143 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e80b4f70 msgr2=0x7fe0e8157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e80b4f70 0x7fe0e8157270 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400c430 tx=0x7fe0e400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 -- 172.21.15.59:0/1519450143 shutdown_connections 2024-03-06T07:27:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0cc06b500 0x7fe0cc06d9c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0e8158e60 0x7fe0e81577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0e80b4f70 0x7fe0e8157270 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 --2- 172.21.15.59:0/1519450143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0e8066d90 0x7fe0e8148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 -- 172.21.15.59:0/1519450143 >> 172.21.15.59:0/1519450143 conn(0x7fe0e80724d0 msgr2=0x7fe0e806bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 -- 172.21.15.59:0/1519450143 shutdown_connections 2024-03-06T07:27:35.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.386+0000 7fe0ef481640 1 -- 172.21.15.59:0/1519450143 wait complete. 2024-03-06T07:27:35.399 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:27:35.516 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:27:35.517 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:27:35.517 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:35.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.617+0000 7f9001ab5640 1 Processor -- start 2024-03-06T07:27:35.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.617+0000 7f9001ab5640 1 -- start start 2024-03-06T07:27:35.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.617+0000 7f9001ab5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8ffc0b4170 0x7f8ffc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.617+0000 7f9001ab5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8ffc0d0a80 0x7f8ffc0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.617+0000 7f9001ab5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc0d25d0 con 0x7f8ffc0d4760 2024-03-06T07:27:35.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.617+0000 7f9001ab5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ffc0d4bf0 con 0x7f8ffc0d0a80 2024-03-06T07:27:35.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.617+0000 7f9001ab5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ffc0ce9e0 con 0x7f8ffc0b4170 2024-03-06T07:27:35.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.617+0000 7f8ffbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8ffc0d0a80 0x7f8ffc0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.617+0000 7f8ffbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8ffc0d0a80 0x7f8ffc0cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33214/0 (socket says 172.21.15.59:33214) 2024-03-06T07:27:35.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.618+0000 7f8ffbfff640 1 -- 172.21.15.59:0/4151632312 learned_addr learned my addr 172.21.15.59:0/4151632312 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:35.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.618+0000 7f8ffb7fe640 1 --2- 172.21.15.59:0/4151632312 >> v2:172.21.15.164:3300/0 conn(0x7f8ffc0b4170 0x7f8ffc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.618+0000 7f8ffbfff640 1 -- 172.21.15.59:0/4151632312 >> v2:172.21.15.164:3300/0 conn(0x7f8ffc0b4170 msgr2=0x7f8ffc0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.618+0000 7f8ffbfff640 1 --2- 172.21.15.59:0/4151632312 >> v2:172.21.15.164:3300/0 conn(0x7f8ffc0b4170 0x7f8ffc0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.618+0000 7f8ffbfff640 1 -- 172.21.15.59:0/4151632312 >> v1:172.21.15.164:6789/0 conn(0x7f8ffc0d4760 legacy=0x7f8ffc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.618+0000 7f8ffbfff640 1 -- 172.21.15.59:0/4151632312 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc0d2870 con 0x7f8ffc0d0a80 2024-03-06T07:27:35.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.618+0000 7f8ffb7fe640 1 --2- 172.21.15.59:0/4151632312 >> v2:172.21.15.164:3300/0 conn(0x7f8ffc0b4170 0x7f8ffc0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:35.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.618+0000 7f8ffbfff640 1 --2- 172.21.15.59:0/4151632312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ffc0d0a80 0x7f8ffc0cee70 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc06e030 tx=0x7f8ff402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=585ee112989ae817 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.619+0000 7f8ffa7fc640 1 -- 172.21.15.59:0/4151632312 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff402ca80 con 0x7f8ffc0d0a80 2024-03-06T07:27:35.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.619+0000 7f8ffa7fc640 1 -- 172.21.15.59:0/4151632312 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff402cc20 con 0x7f8ffc0d0a80 2024-03-06T07:27:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.619+0000 7f8ffa7fc640 1 -- 172.21.15.59:0/4151632312 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff40343f0 con 0x7f8ffc0d0a80 2024-03-06T07:27:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.619+0000 7f9001ab5640 1 -- 172.21.15.59:0/4151632312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ffc0d0a80 msgr2=0x7f8ffc0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.619+0000 7f9001ab5640 1 --2- 172.21.15.59:0/4151632312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ffc0d0a80 0x7f8ffc0cee70 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc06e030 tx=0x7f8ff402bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.619+0000 7f9001ab5640 1 -- 172.21.15.59:0/4151632312 shutdown_connections 2024-03-06T07:27:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.619+0000 7f9001ab5640 1 --2- 172.21.15.59:0/4151632312 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ffc0d0a80 0x7f8ffc0cee70 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.619+0000 7f9001ab5640 1 --2- 172.21.15.59:0/4151632312 >> v2:172.21.15.164:3300/0 conn(0x7f8ffc0b4170 0x7f8ffc0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.619+0000 7f9001ab5640 1 -- 172.21.15.59:0/4151632312 >> 172.21.15.59:0/4151632312 conn(0x7f8ffc0d3c40 msgr2=0x7f8ffc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.619+0000 7f9001ab5640 1 -- 172.21.15.59:0/4151632312 shutdown_connections 2024-03-06T07:27:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.619+0000 7f9001ab5640 1 -- 172.21.15.59:0/4151632312 wait complete. 2024-03-06T07:27:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.620+0000 7f9001ab5640 1 Processor -- start 2024-03-06T07:27:35.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.621+0000 7f9001ab5640 1 -- start start 2024-03-06T07:27:35.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.621+0000 7f9001ab5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ffc0b4170 0x7f8ffc0b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.621+0000 7f9001ab5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ffc0d0a80 0x7f8ffc06c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.621+0000 7f9001ab5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ffc15af00 0x7f8ffc06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.621+0000 7f9001ab5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0b4dd0 con 0x7f8ffc0b4170 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.621+0000 7f9001ab5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0b4550 con 0x7f8ffc15af00 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.621+0000 7f9001ab5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc0b4f50 con 0x7f8ffc0d0a80 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.621+0000 7f8ffb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ffc0b4170 0x7f8ffc0b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.621+0000 7f8ffb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ffc0b4170 0x7f8ffc0b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33216/0 (socket says 172.21.15.59:33216) 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.621+0000 7f8ffb7fe640 1 -- 172.21.15.59:0/3124231073 learned_addr learned my addr 172.21.15.59:0/3124231073 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f8ffbfff640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ffc15af00 0x7f8ffc06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f8ffaffd640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ffc0d0a80 0x7f8ffc06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f8ffb7fe640 1 -- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ffc0d0a80 msgr2=0x7f8ffc06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f8ffb7fe640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ffc0d0a80 0x7f8ffc06c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f8ffb7fe640 1 -- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ffc15af00 msgr2=0x7f8ffc06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f8ffb7fe640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ffc15af00 0x7f8ffc06c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f8ffb7fe640 1 -- 172.21.15.59:0/3124231073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc0b1fb0 con 0x7f8ffc0b4170 2024-03-06T07:27:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f8ffb7fe640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ffc0b4170 0x7f8ffc0b1840 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00c820 tx=0x7f8fec00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f8ff3fff640 1 -- 172.21.15.59:0/3124231073 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec015070 con 0x7f8ffc0b4170 2024-03-06T07:27:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f8ff3fff640 1 -- 172.21.15.59:0/3124231073 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec007a30 con 0x7f8ffc0b4170 2024-03-06T07:27:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f9001ab5640 1 -- 172.21.15.59:0/3124231073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc0b2290 con 0x7f8ffc0b4170 2024-03-06T07:27:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f8ff3fff640 1 -- 172.21.15.59:0/3124231073 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec007e00 con 0x7f8ffc0b4170 2024-03-06T07:27:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.622+0000 7f9001ab5640 1 -- 172.21.15.59:0/3124231073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc159810 con 0x7f8ffc0b4170 2024-03-06T07:27:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.623+0000 7f9001ab5640 1 -- 172.21.15.59:0/3124231073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc04f330 con 0x7f8ffc0b4170 2024-03-06T07:27:35.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.623+0000 7f8ff3fff640 1 -- 172.21.15.59:0/3124231073 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fec018070 con 0x7f8ffc0b4170 2024-03-06T07:27:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.624+0000 7f8ff3fff640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 0x7f8fdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.624+0000 7f8ff3fff640 1 -- 172.21.15.59:0/3124231073 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8fec08a110 con 0x7f8ffc0b4170 2024-03-06T07:27:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.624+0000 7f8ffaffd640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 0x7f8fdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.625+0000 7f8ffaffd640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 0x7f8fdc06d9c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4006fd0 tx=0x7f8fe4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.634 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.632+0000 7f8ff3fff640 1 -- 172.21.15.59:0/3124231073 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fec058fc0 con 0x7f8ffc0b4170 2024-03-06T07:27:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.853+0000 7f9001ab5640 1 -- 172.21.15.59:0/3124231073 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ffc159ad0 con 0x7f8ffc0b4170 2024-03-06T07:27:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.853+0000 7f8ff3fff640 1 -- 172.21.15.59:0/3124231073 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f8fec0588c0 con 0x7f8ffc0b4170 2024-03-06T07:27:35.855 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:27:35.856 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:27:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:27:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.855+0000 7f9001ab5640 1 -- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 msgr2=0x7f8fdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.856+0000 7f9001ab5640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 0x7f8fdc06d9c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4006fd0 tx=0x7f8fe4008040 comp rx=0 tx=0).stop 2024-03-06T07:27:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.856+0000 7f9001ab5640 1 -- 172.21.15.59:0/3124231073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ffc0b4170 msgr2=0x7f8ffc0b1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.856+0000 7f9001ab5640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ffc0b4170 0x7f8ffc0b1840 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00c820 tx=0x7f8fec00cc20 comp rx=0 tx=0).stop 2024-03-06T07:27:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.856+0000 7f9001ab5640 1 -- 172.21.15.59:0/3124231073 shutdown_connections 2024-03-06T07:27:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.856+0000 7f9001ab5640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 0x7f8fdc06d9c0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.856+0000 7f9001ab5640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ffc15af00 0x7f8ffc06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.856+0000 7f9001ab5640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ffc0d0a80 0x7f8ffc06c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.856+0000 7f9001ab5640 1 --2- 172.21.15.59:0/3124231073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ffc0b4170 0x7f8ffc0b1840 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.856+0000 7f9001ab5640 1 -- 172.21.15.59:0/3124231073 >> 172.21.15.59:0/3124231073 conn(0x7f8ffc0d3c40 msgr2=0x7f8ffc0cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.856+0000 7f9001ab5640 1 -- 172.21.15.59:0/3124231073 shutdown_connections 2024-03-06T07:27:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:27:35.856+0000 7f9001ab5640 1 -- 172.21.15.59:0/3124231073 wait complete. 2024-03-06T07:27:35.869 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:27:35.986 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:27:35.986 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:27:50.330 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:50.332 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:05.991 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:28:06.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.030+0000 7f0c8e5d4f40 1 Processor -- start 2024-03-06T07:28:06.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.030+0000 7f0c8e5d4f40 1 -- start start 2024-03-06T07:28:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.031+0000 7f0c8e5d4f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4eb43980 0x55df4ebfef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.031+0000 7f0c8e5d4f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4eb8ad70 0x55df4eb6f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.031+0000 7f0c8e5d4f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 0x55df4eb6f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.031+0000 7f0c8e5d4f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df4eb886f0 con 0x55df4ec14f20 2024-03-06T07:28:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.031+0000 7f0c8e5d4f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df4eae6b70 con 0x55df4eb43980 2024-03-06T07:28:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.031+0000 7f0c8e5d4f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55df4eb43d90 con 0x55df4eb8ad70 2024-03-06T07:28:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.031+0000 7f0c8c8d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 0x55df4eb6f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.031+0000 7f0c8c8d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 0x55df4eb6f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49946/0 (socket says 172.21.15.59:49946) 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.031+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/1730080626 learned_addr learned my addr 172.21.15.59:0/1730080626 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.031+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/1730080626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4eb43980 0x55df4ebfef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.031+0000 7f0c8b8d4640 1 --2- 172.21.15.59:0/1730080626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4eb8ad70 0x55df4eb6f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/1730080626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4eb8ad70 msgr2=0x55df4eb6f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8c8d6640 1 --2- 172.21.15.59:0/1730080626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4eb8ad70 0x55df4eb6f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/1730080626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4eb43980 msgr2=0x55df4ebfef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8c8d6640 1 --2- 172.21.15.59:0/1730080626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4eb43980 0x55df4ebfef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/1730080626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df4ec0fd60 con 0x55df4ec14f20 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/1730080626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4eb43980 0x55df4ebfef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8c8d6640 1 --2- 172.21.15.59:0/1730080626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 0x55df4eb6f830 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f0c84001650 tx=0x7f0c84001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4675b0f89795a034 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8a8d2640 1 -- 172.21.15.59:0/1730080626 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8400f710 con 0x55df4ec14f20 2024-03-06T07:28:06.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8a8d2640 1 -- 172.21.15.59:0/1730080626 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8400f8b0 con 0x55df4ec14f20 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8a8d2640 1 -- 172.21.15.59:0/1730080626 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8400fca0 con 0x55df4ec14f20 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/1730080626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 msgr2=0x55df4eb6f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/1730080626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 0x55df4eb6f830 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f0c84001650 tx=0x7f0c84001b10 comp rx=0 tx=0).stop 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/1730080626 shutdown_connections 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/1730080626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 0x55df4eb6f830 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/1730080626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4eb8ad70 0x55df4eb6f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/1730080626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4eb43980 0x55df4ebfef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/1730080626 >> 172.21.15.59:0/1730080626 conn(0x55df4eb3e8a0 msgr2=0x55df4eb6ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/1730080626 shutdown_connections 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.032+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/1730080626 wait complete. 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.033+0000 7f0c8e5d4f40 1 Processor -- start 2024-03-06T07:28:06.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.033+0000 7f0c8e5d4f40 1 -- start start 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8e5d4f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4eb43980 0x55df4ec11ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8e5d4f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4eb8ad70 0x55df4ec0df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8e5d4f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 0x55df4ec0e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8e5d4f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df4ec0fac0 con 0x55df4ec14f20 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8e5d4f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df4ec11840 con 0x55df4eb8ad70 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8e5d4f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55df4ec114d0 con 0x55df4eb43980 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8c8d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 0x55df4ec0e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8c8d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 0x55df4ec0e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49954/0 (socket says 172.21.15.59:49954) 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8b8d4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4eb8ad70 0x55df4ec0df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 learned_addr learned my addr 172.21.15.59:0/2692171984 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4eb43980 0x55df4ec11ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4eb43980 msgr2=0x55df4ec11ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8c8d6640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df4eb43980 0x55df4ec11ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4eb8ad70 msgr2=0x55df4ec0df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8c8d6640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df4eb8ad70 0x55df4ec0df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df4ebfdd90 con 0x55df4ec14f20 2024-03-06T07:28:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.034+0000 7f0c8c8d6640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 0x55df4ec0e4d0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400f740 tx=0x7f0c84001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.035+0000 7f0c898d0640 1 -- 172.21.15.59:0/2692171984 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c84022020 con 0x55df4ec14f20 2024-03-06T07:28:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.035+0000 7f0c898d0640 1 -- 172.21.15.59:0/2692171984 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c84012890 con 0x55df4ec14f20 2024-03-06T07:28:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.035+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55df4ec0c630 con 0x55df4ec14f20 2024-03-06T07:28:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.035+0000 7f0c898d0640 1 -- 172.21.15.59:0/2692171984 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c84012c80 con 0x55df4ec14f20 2024-03-06T07:28:06.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.035+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df4ec0f550 con 0x55df4ec14f20 2024-03-06T07:28:06.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.036+0000 7f0c898d0640 1 -- 172.21.15.59:0/2692171984 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c84025020 con 0x55df4ec14f20 2024-03-06T07:28:06.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.037+0000 7f0c898d0640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c78069eb0 0x7f0c7806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.037+0000 7f0c898d0640 1 -- 172.21.15.59:0/2692171984 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0c8408b160 con 0x55df4ec14f20 2024-03-06T07:28:06.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.037+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x55df4ec12400 0x55df4eb816d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.037+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x55df4ec12400 2024-03-06T07:28:06.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.037+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c78069eb0 0x7f0c7806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.037+0000 7f0c8b8d4640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x55df4ec12400 0x55df4eb816d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.038+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c78069eb0 0x7f0c7806c370 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c057000 tx=0x7f0c7c051000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.038+0000 7f0c8b8d4640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x55df4ec12400 0x55df4eb816d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.038+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.7 v2:172.21.15.164:6808/3612340295 1 ==== osd_op_reply(1 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x55df4ec12400 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.038+0000 7f0c8b8d4640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7f0c80009be0 0x7f0c80006380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.038+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80009be0 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.038+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7f0c80009be0 0x7f0c80006380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.039+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7f0c80009be0 0x7f0c80006380 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.039+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.1 v2:172.21.15.59:6816/1773892168 1 ==== osd_op_reply(2 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80009be0 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.039+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f0c7c0603f0 0x7f0c7c060830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.039+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c0603f0 2024-03-06T07:28:06.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.039+0000 7f0c8b8d4640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f0c7c0603f0 0x7f0c7c060830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.039+0000 7f0c8b8d4640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f0c7c0603f0 0x7f0c7c060830 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.040+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.3 v2:172.21.15.59:6808/3602142643 1 ==== osd_op_reply(3 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c0603f0 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.040+0000 7f0c8b8d4640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f0c8001b940 0x7f0c8001a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.040+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.040+0000 7f0c8c8d6640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f0c8001b940 0x7f0c8001a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.040+0000 7f0c8c8d6640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f0c8001b940 0x7f0c8001a3c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.041+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 1 ==== osd_op_reply(4 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.041+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x55df4ec12400 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.041+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.7 v2:172.21.15.164:6808/3612340295 2 ==== osd_op_reply(5 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x55df4ec12400 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.041+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c0603f0 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.042+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.3 v2:172.21.15.59:6808/3602142643 2 ==== osd_op_reply(6 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c0603f0 2024-03-06T07:28:06.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.042+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.042+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 2 ==== osd_op_reply(7 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.042+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x55df4ec12400 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.042+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.7 v2:172.21.15.164:6808/3612340295 3 ==== osd_op_reply(8 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x55df4ec12400 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.042+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.043+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 3 ==== osd_op_reply(9 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.043+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c0603f0 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.043+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.3 v2:172.21.15.59:6808/3602142643 3 ==== osd_op_reply(10 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c0603f0 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.043+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c0603f0 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.043+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.3 v2:172.21.15.59:6808/3602142643 4 ==== osd_op_reply(11 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c0603f0 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.043+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.043+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 4 ==== osd_op_reply(12 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.043+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x55df4ec12400 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.044+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.7 v2:172.21.15.164:6808/3612340295 4 ==== osd_op_reply(13 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x55df4ec12400 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.044+0000 7f0c8b8d4640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f0c80027440 0x7f0c800278c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.044+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80027440 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.044+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f0c80027440 0x7f0c800278c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.044+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f0c80027440 0x7f0c800278c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.045+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.4 v2:172.21.15.164:6818/102497754 1 ==== osd_op_reply(14 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80027440 2024-03-06T07:28:06.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.045+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80009be0 2024-03-06T07:28:06.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.045+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.1 v2:172.21.15.59:6816/1773892168 2 ==== osd_op_reply(15 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80009be0 2024-03-06T07:28:06.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.045+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.045+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 5 ==== osd_op_reply(16 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.045+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80027440 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.046+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.4 v2:172.21.15.164:6818/102497754 2 ==== osd_op_reply(17 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80027440 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.046+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f0c7c06f3a0 0x7f0c7c06f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.046+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c06f3a0 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.046+0000 7f0c8b8d4640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f0c7c06f3a0 0x7f0c7c06f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.047+0000 7f0c8b8d4640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f0c7c06f3a0 0x7f0c7c06f800 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.047+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.5 v2:172.21.15.164:6800/2889679248 1 ==== osd_op_reply(18 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c06f3a0 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.047+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c0603f0 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.047+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.3 v2:172.21.15.59:6808/3602142643 5 ==== osd_op_reply(19 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c0603f0 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.047+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x55df4ec12400 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.048+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.7 v2:172.21.15.164:6808/3612340295 5 ==== osd_op_reply(20 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x55df4ec12400 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.048+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x55df4ec12400 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.048+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.7 v2:172.21.15.164:6808/3612340295 6 ==== osd_op_reply(21 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x55df4ec12400 2024-03-06T07:28:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.048+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c0603f0 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.048+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.3 v2:172.21.15.59:6808/3602142643 6 ==== osd_op_reply(22 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c0603f0 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.048+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.048+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 6 ==== osd_op_reply(23 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.048+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80009be0 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.049+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.1 v2:172.21.15.59:6816/1773892168 3 ==== osd_op_reply(24 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80009be0 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.049+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f0c7c05f860 0x7f0c7c006380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.049+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c05f860 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.049+0000 7f0c8c8d6640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f0c7c05f860 0x7f0c7c006380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.049+0000 7f0c8c8d6640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f0c7c05f860 0x7f0c7c006380 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.050+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.6 v2:172.21.15.164:6816/1594998822 1 ==== osd_op_reply(25 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c7c05f860 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.050+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.050+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 7 ==== osd_op_reply(26 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.050+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c06f3a0 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.051+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.5 v2:172.21.15.164:6800/2889679248 2 ==== osd_op_reply(27 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c06f3a0 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.051+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.051+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 8 ==== osd_op_reply(28 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.051+0000 7f0c8c8d6640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f0c84027020 0x7f0c84058970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.051+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c84027020 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.051+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f0c84027020 0x7f0c84058970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.051+0000 7f0c8c0d5640 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f0c84027020 0x7f0c84058970 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.051+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.2 v2:172.21.15.59:6800/1020996386 1 ==== osd_op_reply(29 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c84027020 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.052+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c05f860 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.052+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.6 v2:172.21.15.164:6816/1594998822 2 ==== osd_op_reply(30 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c7c05f860 2024-03-06T07:28:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.052+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80009be0 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.052+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.1 v2:172.21.15.59:6816/1773892168 4 ==== osd_op_reply(31 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80009be0 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.052+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80027440 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.052+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.4 v2:172.21.15.164:6818/102497754 3 ==== osd_op_reply(32 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80027440 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.052+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.053+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 9 ==== osd_op_reply(33 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.053+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.053+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 10 ==== osd_op_reply(34 [pgnls start_epoch 114 out=49b] v108'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.053+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c0603f0 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.053+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.3 v2:172.21.15.59:6808/3602142643 7 ==== osd_op_reply(35 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c0603f0 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.053+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80009be0 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.054+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.1 v2:172.21.15.59:6816/1773892168 5 ==== osd_op_reply(36 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80009be0 2024-03-06T07:28:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.054+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c0603f0 2024-03-06T07:28:06.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.054+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.3 v2:172.21.15.59:6808/3602142643 8 ==== osd_op_reply(37 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c0603f0 2024-03-06T07:28:06.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.054+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80027440 2024-03-06T07:28:06.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.054+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.4 v2:172.21.15.164:6818/102497754 4 ==== osd_op_reply(38 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80027440 2024-03-06T07:28:06.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.054+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c06f3a0 2024-03-06T07:28:06.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.055+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.5 v2:172.21.15.164:6800/2889679248 3 ==== osd_op_reply(39 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c06f3a0 2024-03-06T07:28:06.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.055+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c06f3a0 2024-03-06T07:28:06.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.055+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.5 v2:172.21.15.164:6800/2889679248 4 ==== osd_op_reply(40 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c06f3a0 2024-03-06T07:28:06.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.055+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c05f860 2024-03-06T07:28:06.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.056+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.6 v2:172.21.15.164:6816/1594998822 3 ==== osd_op_reply(41 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c7c05f860 2024-03-06T07:28:06.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.056+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80009be0 2024-03-06T07:28:06.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.056+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.1 v2:172.21.15.59:6816/1773892168 6 ==== osd_op_reply(42 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80009be0 2024-03-06T07:28:06.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.056+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c06f3a0 2024-03-06T07:28:06.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.056+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.5 v2:172.21.15.164:6800/2889679248 5 ==== osd_op_reply(43 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c06f3a0 2024-03-06T07:28:06.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.056+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80027440 2024-03-06T07:28:06.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.057+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.4 v2:172.21.15.164:6818/102497754 5 ==== osd_op_reply(44 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80027440 2024-03-06T07:28:06.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.057+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x55df4ec12400 2024-03-06T07:28:06.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.057+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.7 v2:172.21.15.164:6808/3612340295 7 ==== osd_op_reply(45 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x55df4ec12400 2024-03-06T07:28:06.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.057+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x55df4ec12400 2024-03-06T07:28:06.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.057+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.7 v2:172.21.15.164:6808/3612340295 8 ==== osd_op_reply(46 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x55df4ec12400 2024-03-06T07:28:06.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.057+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x55df4ec12400 2024-03-06T07:28:06.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.058+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.7 v2:172.21.15.164:6808/3612340295 9 ==== osd_op_reply(47 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x55df4ec12400 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.058+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x55df4ec12400 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.058+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.7 v2:172.21.15.164:6808/3612340295 10 ==== osd_op_reply(48 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x55df4ec12400 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.058+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80027440 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.058+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.4 v2:172.21.15.164:6818/102497754 6 ==== osd_op_reply(49 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80027440 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.058+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80027440 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.059+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.4 v2:172.21.15.164:6818/102497754 7 ==== osd_op_reply(50 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80027440 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.059+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c0603f0 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.059+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.3 v2:172.21.15.59:6808/3602142643 9 ==== osd_op_reply(51 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x7f0c7c0603f0 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.059+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x55df4ec12400 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.059+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 <== osd.7 v2:172.21.15.164:6808/3612340295 11 ==== osd_op_reply(52 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c80009950 con 0x55df4ec12400 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.059+0000 7f0c8b8d4640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.060+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 11 ==== osd_op_reply(53 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.060+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80027440 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.060+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.4 v2:172.21.15.164:6818/102497754 8 ==== osd_op_reply(54 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80027440 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.060+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80027440 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.060+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.4 v2:172.21.15.164:6818/102497754 9 ==== osd_op_reply(55 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80027440 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.060+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c84027020 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.061+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.2 v2:172.21.15.59:6800/1020996386 2 ==== osd_op_reply(56 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c84027020 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.061+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.061+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 12 ==== osd_op_reply(57 [pgnls start_epoch 114 out=81b] v108'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.061+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80027440 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.061+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.4 v2:172.21.15.164:6818/102497754 10 ==== osd_op_reply(58 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80027440 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.061+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c05f860 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.062+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.6 v2:172.21.15.164:6816/1594998822 4 ==== osd_op_reply(59 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c7c05f860 2024-03-06T07:28:06.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.062+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.062+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 13 ==== osd_op_reply(60 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.062+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80009be0 2024-03-06T07:28:06.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.062+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.1 v2:172.21.15.59:6816/1773892168 7 ==== osd_op_reply(61 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80009be0 2024-03-06T07:28:06.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.062+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c80027440 2024-03-06T07:28:06.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 <== osd.4 v2:172.21.15.164:6818/102497754 11 ==== osd_op_reply(62 [pgnls start_epoch 114 out=81b] v108'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f0c7c04dcb0 con 0x7f0c80027440 2024-03-06T07:28:06.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8c0d5640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c7c05f860 2024-03-06T07:28:06.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.6 v2:172.21.15.164:6816/1594998822 5 ==== osd_op_reply(63 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c7c05f860 2024-03-06T07:28:06.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55df4eb87bb0 con 0x7f0c8001b940 2024-03-06T07:28:06.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 <== osd.0 v2:172.21.15.59:6824/841983189 14 ==== osd_op_reply(64 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0c8400f070 con 0x7f0c8001b940 2024-03-06T07:28:06.070 INFO:teuthology.orchestra.run.smithi059.stdout:10000000001.00000000 2024-03-06T07:28:06.071 INFO:teuthology.orchestra.run.smithi059.stdout:10000000001.00000001 2024-03-06T07:28:06.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f0c8001b940 msgr2=0x7f0c8001a3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f0c8001b940 0x7f0c8001a3c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7f0c80009be0 msgr2=0x7f0c80006380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7f0c80009be0 0x7f0c80006380 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f0c84027020 msgr2=0x7f0c84058970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f0c84027020 0x7f0c84058970 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f0c7c0603f0 msgr2=0x7f0c7c060830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f0c7c0603f0 0x7f0c7c060830 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 reap_dead start 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f0c80027440 msgr2=0x7f0c800278c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.063+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f0c80027440 0x7f0c800278c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f0c7c06f3a0 msgr2=0x7f0c7c06f800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f0c7c06f3a0 0x7f0c7c06f800 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f0c7c05f860 msgr2=0x7f0c7c006380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f0c7c05f860 0x7f0c7c006380 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x55df4ec12400 msgr2=0x55df4eb816d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x55df4ec12400 0x55df4eb816d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c78069eb0 msgr2=0x7f0c7806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0c78069eb0 0x7f0c7806c370 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c057000 tx=0x7f0c7c051000 comp rx=0 tx=0).stop 2024-03-06T07:28:06.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 reap_dead start 2024-03-06T07:28:06.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 msgr2=0x55df4ec0e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 --2- 172.21.15.59:0/2692171984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55df4ec14f20 0x55df4ec0e4d0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400f740 tx=0x7f0c84001250 comp rx=0 tx=0).stop 2024-03-06T07:28:06.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8c8d6640 1 -- 172.21.15.59:0/2692171984 reap_dead start 2024-03-06T07:28:06.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 shutdown_connections 2024-03-06T07:28:06.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 >> 172.21.15.59:0/2692171984 conn(0x55df4eb3e8a0 msgr2=0x55df4eb6ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 shutdown_connections 2024-03-06T07:28:06.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.064+0000 7f0c8e5d4f40 1 -- 172.21.15.59:0/2692171984 wait complete. 2024-03-06T07:28:06.074 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-03-06T07:28:06.075 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:06.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.213+0000 7f5a988fb640 1 Processor -- start 2024-03-06T07:28:06.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.213+0000 7f5a988fb640 1 -- start start 2024-03-06T07:28:06.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.213+0000 7f5a988fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9415dfe0 0x7f5a940cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a988fb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a940d29c0 0x7f5a940b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a988fb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a940b3430 con 0x7f5a940cf830 2024-03-06T07:28:06.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a988fb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a940b31e0 con 0x7f5a940d29c0 2024-03-06T07:28:06.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a988fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a940b3d60 con 0x7f5a9415dfe0 2024-03-06T07:28:06.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a92d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a940d29c0 0x7f5a940b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a92d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a940d29c0 0x7f5a940b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49960/0 (socket says 172.21.15.59:49960) 2024-03-06T07:28:06.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a92d9d640 1 -- 172.21.15.59:0/852678298 learned_addr learned my addr 172.21.15.59:0/852678298 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a9259c640 1 --2- 172.21.15.59:0/852678298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9415dfe0 0x7f5a940cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a92d9d640 1 -- 172.21.15.59:0/852678298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9415dfe0 msgr2=0x7f5a940cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a92d9d640 1 --2- 172.21.15.59:0/852678298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9415dfe0 0x7f5a940cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a92d9d640 1 -- 172.21.15.59:0/852678298 >> v1:172.21.15.164:6789/0 conn(0x7f5a940cf830 legacy=0x7f5a940d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a92d9d640 1 -- 172.21.15.59:0/852678298 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a940b3710 con 0x7f5a940d29c0 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.214+0000 7f5a9259c640 1 --2- 172.21.15.59:0/852678298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9415dfe0 0x7f5a940cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.215+0000 7f5a92d9d640 1 --2- 172.21.15.59:0/852678298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a940d29c0 0x7f5a940b41f0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c009e50 tx=0x7f5a8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a298cf9232666a2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.215+0000 7f5a91d9b640 1 -- 172.21.15.59:0/852678298 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c02ca70 con 0x7f5a940d29c0 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.215+0000 7f5a91d9b640 1 -- 172.21.15.59:0/852678298 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a8c02cc10 con 0x7f5a940d29c0 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.215+0000 7f5a91d9b640 1 -- 172.21.15.59:0/852678298 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c0343f0 con 0x7f5a940d29c0 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.215+0000 7f5a988fb640 1 -- 172.21.15.59:0/852678298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a940d29c0 msgr2=0x7f5a940b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.215+0000 7f5a988fb640 1 --2- 172.21.15.59:0/852678298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a940d29c0 0x7f5a940b41f0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c009e50 tx=0x7f5a8c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.215+0000 7f5a988fb640 1 -- 172.21.15.59:0/852678298 shutdown_connections 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.215+0000 7f5a988fb640 1 --2- 172.21.15.59:0/852678298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a940d29c0 0x7f5a940b41f0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.215+0000 7f5a988fb640 1 --2- 172.21.15.59:0/852678298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a9415dfe0 0x7f5a940cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.215+0000 7f5a988fb640 1 -- 172.21.15.59:0/852678298 >> 172.21.15.59:0/852678298 conn(0x7f5a940d3cd0 msgr2=0x7f5a940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.216+0000 7f5a988fb640 1 -- 172.21.15.59:0/852678298 shutdown_connections 2024-03-06T07:28:06.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.216+0000 7f5a988fb640 1 -- 172.21.15.59:0/852678298 wait complete. 2024-03-06T07:28:06.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.217+0000 7f5a988fb640 1 Processor -- start 2024-03-06T07:28:06.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.217+0000 7f5a988fb640 1 -- start start 2024-03-06T07:28:06.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a988fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a940cf830 0x7f5a9414bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a988fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a940d29c0 0x7f5a9415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a988fb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a9415dfe0 0x7f5a9415c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a988fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a940d1df0 con 0x7f5a940cf830 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a988fb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a940d1c70 con 0x7f5a9415dfe0 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a988fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9415e3c0 con 0x7f5a940d29c0 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a9259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a940cf830 0x7f5a9414bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a9259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a940cf830 0x7f5a9414bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49968/0 (socket says 172.21.15.59:49968) 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a9259c640 1 -- 172.21.15.59:0/1240798696 learned_addr learned my addr 172.21.15.59:0/1240798696 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a89d9b640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a940d29c0 0x7f5a9415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a92d9d640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a9415dfe0 0x7f5a9415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a9259c640 1 -- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a940d29c0 msgr2=0x7f5a9415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a9259c640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a940d29c0 0x7f5a9415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a9259c640 1 -- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a9415dfe0 msgr2=0x7f5a9415c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a9259c640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a9415dfe0 0x7f5a9415c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a9259c640 1 -- 172.21.15.59:0/1240798696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a941f78c0 con 0x7f5a940cf830 2024-03-06T07:28:06.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a92d9d640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a9415dfe0 0x7f5a9415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:06.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a9259c640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a940cf830 0x7f5a9414bea0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400c820 tx=0x7f5a8400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.218+0000 7f5a8b7fe640 1 -- 172.21.15.59:0/1240798696 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84015070 con 0x7f5a940cf830 2024-03-06T07:28:06.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.219+0000 7f5a8b7fe640 1 -- 172.21.15.59:0/1240798696 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a84007a30 con 0x7f5a940cf830 2024-03-06T07:28:06.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.219+0000 7f5a988fb640 1 -- 172.21.15.59:0/1240798696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a941f7ba0 con 0x7f5a940cf830 2024-03-06T07:28:06.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.219+0000 7f5a8b7fe640 1 -- 172.21.15.59:0/1240798696 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84007de0 con 0x7f5a940cf830 2024-03-06T07:28:06.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.219+0000 7f5a988fb640 1 -- 172.21.15.59:0/1240798696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a941f8160 con 0x7f5a940cf830 2024-03-06T07:28:06.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.220+0000 7f5a988fb640 1 -- 172.21.15.59:0/1240798696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a940d1a90 con 0x7f5a940cf830 2024-03-06T07:28:06.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.220+0000 7f5a8b7fe640 1 -- 172.21.15.59:0/1240798696 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a84018070 con 0x7f5a940cf830 2024-03-06T07:28:06.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.220+0000 7f5a8b7fe640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7406b4d0 0x7f5a7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.221+0000 7f5a8b7fe640 1 -- 172.21.15.59:0/1240798696 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5a8408a120 con 0x7f5a940cf830 2024-03-06T07:28:06.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.221+0000 7f5a89d9b640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7406b4d0 0x7f5a7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.221+0000 7f5a89d9b640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7406b4d0 0x7f5a7406d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5a940a1d10 tx=0x7f5a7c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.225+0000 7f5a8b7fe640 1 -- 172.21.15.59:0/1240798696 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a940d1a90 con 0x7f5a940cf830 2024-03-06T07:28:06.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.545+0000 7f5a988fb640 1 -- 172.21.15.59:0/1240798696 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a940ce7e0 con 0x7f5a940cf830 2024-03-06T07:28:06.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.546+0000 7f5a8b7fe640 1 -- 172.21.15.59:0/1240798696 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f5a940ce7e0 con 0x7f5a940cf830 2024-03-06T07:28:06.548 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:06.548 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:06.548 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:06.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 -- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7406b4d0 msgr2=0x7f5a7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7406b4d0 0x7f5a7406d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5a940a1d10 tx=0x7f5a7c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:06.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 -- 172.21.15.59:0/1240798696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a940cf830 msgr2=0x7f5a9414bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a940cf830 0x7f5a9414bea0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400c820 tx=0x7f5a8400cce0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 -- 172.21.15.59:0/1240798696 shutdown_connections 2024-03-06T07:28:06.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7406b4d0 0x7f5a7406d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a9415dfe0 0x7f5a9415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a940d29c0 0x7f5a9415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 --2- 172.21.15.59:0/1240798696 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a940cf830 0x7f5a9414bea0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 -- 172.21.15.59:0/1240798696 >> 172.21.15.59:0/1240798696 conn(0x7f5a940d3cd0 msgr2=0x7f5a9406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 -- 172.21.15.59:0/1240798696 shutdown_connections 2024-03-06T07:28:06.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.548+0000 7f5a988fb640 1 -- 172.21.15.59:0/1240798696 wait complete. 2024-03-06T07:28:06.561 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:06.682 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:28:06.682 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:28:06.682 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:06.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.819+0000 7fe9dc777640 1 Processor -- start 2024-03-06T07:28:06.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.819+0000 7fe9dc777640 1 -- start start 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.820+0000 7fe9dc777640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe9d415ddb0 0x7fe9d4072ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.820+0000 7fe9dc777640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe9d406b6c0 0x7fe9d406d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.820+0000 7fe9dc777640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d4066fe0 con 0x7fe9d415e190 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.820+0000 7fe9dc777640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d4073400 con 0x7fe9d415ddb0 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.820+0000 7fe9dc777640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d40b5200 con 0x7fe9d406b6c0 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.820+0000 7fe9da513640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe9d415ddb0 0x7fe9d4072ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.821+0000 7fe9da513640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe9d415ddb0 0x7fe9d4072ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49970/0 (socket says 172.21.15.59:49970) 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.821+0000 7fe9da513640 1 -- 172.21.15.59:0/3976132129 learned_addr learned my addr 172.21.15.59:0/3976132129 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.821+0000 7fe9dad14640 1 --2- 172.21.15.59:0/3976132129 >> v2:172.21.15.164:3300/0 conn(0x7fe9d406b6c0 0x7fe9d406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.821+0000 7fe9da513640 1 -- 172.21.15.59:0/3976132129 >> v2:172.21.15.164:3300/0 conn(0x7fe9d406b6c0 msgr2=0x7fe9d406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.821+0000 7fe9da513640 1 --2- 172.21.15.59:0/3976132129 >> v2:172.21.15.164:3300/0 conn(0x7fe9d406b6c0 0x7fe9d406d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.821+0000 7fe9da513640 1 -- 172.21.15.59:0/3976132129 >> v1:172.21.15.164:6789/0 conn(0x7fe9d415e190 legacy=0x7fe9d406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.821+0000 7fe9da513640 1 -- 172.21.15.59:0/3976132129 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d40b54e0 con 0x7fe9d415ddb0 2024-03-06T07:28:06.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.822+0000 7fe9da513640 1 --2- 172.21.15.59:0/3976132129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9d415ddb0 0x7fe9d4072ec0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fe9d406f9c0 tx=0x7fe9d002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d781258b63dc39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.822+0000 7fe9d9511640 1 -- 172.21.15.59:0/3976132129 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0034070 con 0x7fe9d415ddb0 2024-03-06T07:28:06.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.822+0000 7fe9d9511640 1 -- 172.21.15.59:0/3976132129 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d0030840 con 0x7fe9d415ddb0 2024-03-06T07:28:06.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.822+0000 7fe9d9511640 1 -- 172.21.15.59:0/3976132129 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0030c10 con 0x7fe9d415ddb0 2024-03-06T07:28:06.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.822+0000 7fe9dc777640 1 -- 172.21.15.59:0/3976132129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9d415ddb0 msgr2=0x7fe9d4072ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.822+0000 7fe9dc777640 1 --2- 172.21.15.59:0/3976132129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9d415ddb0 0x7fe9d4072ec0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fe9d406f9c0 tx=0x7fe9d002b980 comp rx=0 tx=0).stop 2024-03-06T07:28:06.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.822+0000 7fe9dc777640 1 -- 172.21.15.59:0/3976132129 shutdown_connections 2024-03-06T07:28:06.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.822+0000 7fe9dc777640 1 --2- 172.21.15.59:0/3976132129 >> v2:172.21.15.164:3300/0 conn(0x7fe9d406b6c0 0x7fe9d406d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.822+0000 7fe9dc777640 1 --2- 172.21.15.59:0/3976132129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9d415ddb0 0x7fe9d4072ec0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.822+0000 7fe9dc777640 1 -- 172.21.15.59:0/3976132129 >> 172.21.15.59:0/3976132129 conn(0x7fe9d40b1d00 msgr2=0x7fe9d40b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.823+0000 7fe9dc777640 1 -- 172.21.15.59:0/3976132129 shutdown_connections 2024-03-06T07:28:06.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.823+0000 7fe9dc777640 1 -- 172.21.15.59:0/3976132129 wait complete. 2024-03-06T07:28:06.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.824+0000 7fe9dc777640 1 Processor -- start 2024-03-06T07:28:06.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.824+0000 7fe9dc777640 1 -- start start 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.825+0000 7fe9dc777640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9d406b6c0 0x7fe9d40b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.825+0000 7fe9dc777640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9d41f3e60 0x7fe9d409b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.825+0000 7fe9dc777640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9d409d230 0x7fe9d409d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.825+0000 7fe9dc777640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d40b5d80 con 0x7fe9d409d230 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.825+0000 7fe9dc777640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d40b5aa0 con 0x7fe9d406b6c0 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.825+0000 7fe9dc777640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d40b4e60 con 0x7fe9d41f3e60 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9dad14640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9d409d230 0x7fe9d409d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9dad14640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9d409d230 0x7fe9d409d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49980/0 (socket says 172.21.15.59:49980) 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9dad14640 1 -- 172.21.15.59:0/1765291487 learned_addr learned my addr 172.21.15.59:0/1765291487 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9da513640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9d406b6c0 0x7fe9d40b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9d9d12640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9d41f3e60 0x7fe9d409b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9dad14640 1 -- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9d41f3e60 msgr2=0x7fe9d409b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9dad14640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9d41f3e60 0x7fe9d409b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9dad14640 1 -- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9d406b6c0 msgr2=0x7fe9d40b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9dad14640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9d406b6c0 0x7fe9d40b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9dad14640 1 -- 172.21.15.59:0/1765291487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d41f42c0 con 0x7fe9d409d230 2024-03-06T07:28:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9d9d12640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9d41f3e60 0x7fe9d409b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9da513640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9d406b6c0 0x7fe9d40b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9dad14640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9d409d230 0x7fe9d409d6b0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc007e90 tx=0x7fe9cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9caffd640 1 -- 172.21.15.59:0/1765291487 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc014420 con 0x7fe9d409d230 2024-03-06T07:28:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9caffd640 1 -- 172.21.15.59:0/1765291487 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9cc019070 con 0x7fe9d409d230 2024-03-06T07:28:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9dc777640 1 -- 172.21.15.59:0/1765291487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d40cf880 con 0x7fe9d409d230 2024-03-06T07:28:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9caffd640 1 -- 172.21.15.59:0/1765291487 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc014800 con 0x7fe9d409d230 2024-03-06T07:28:06.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.826+0000 7fe9dc777640 1 -- 172.21.15.59:0/1765291487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d409aff0 con 0x7fe9d409d230 2024-03-06T07:28:06.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.828+0000 7fe9dc777640 1 -- 172.21.15.59:0/1765291487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d406f0f0 con 0x7fe9d409d230 2024-03-06T07:28:06.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.828+0000 7fe9caffd640 1 -- 172.21.15.59:0/1765291487 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9cc0149a0 con 0x7fe9d409d230 2024-03-06T07:28:06.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.829+0000 7fe9caffd640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe9b806b4d0 0x7fe9b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.829+0000 7fe9caffd640 1 -- 172.21.15.59:0/1765291487 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe9cc08aa30 con 0x7fe9d409d230 2024-03-06T07:28:06.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.829+0000 7fe9da513640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe9b806b4d0 0x7fe9b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.830+0000 7fe9da513640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe9b806b4d0 0x7fe9b806d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0036000 tx=0x7fe9d003b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:06.833+0000 7fe9caffd640 1 -- 172.21.15.59:0/1765291487 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9d406f0f0 con 0x7fe9d409d230 2024-03-06T07:28:07.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.124+0000 7fe9dc777640 1 -- 172.21.15.59:0/1765291487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9d404f330 con 0x7fe9d409d230 2024-03-06T07:28:07.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.125+0000 7fe9caffd640 1 -- 172.21.15.59:0/1765291487 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fe9d41f42c0 con 0x7fe9d409d230 2024-03-06T07:28:07.127 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:07.127 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:07.127 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:07.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 -- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe9b806b4d0 msgr2=0x7fe9b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe9b806b4d0 0x7fe9b806d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0036000 tx=0x7fe9d003b040 comp rx=0 tx=0).stop 2024-03-06T07:28:07.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 -- 172.21.15.59:0/1765291487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9d409d230 msgr2=0x7fe9d409d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9d409d230 0x7fe9d409d6b0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc007e90 tx=0x7fe9cc00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 -- 172.21.15.59:0/1765291487 shutdown_connections 2024-03-06T07:28:07.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe9b806b4d0 0x7fe9b806d990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9d409d230 0x7fe9d409d6b0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9d41f3e60 0x7fe9d409b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 --2- 172.21.15.59:0/1765291487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9d406b6c0 0x7fe9d40b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 -- 172.21.15.59:0/1765291487 >> 172.21.15.59:0/1765291487 conn(0x7fe9d40b1d00 msgr2=0x7fe9d40b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 -- 172.21.15.59:0/1765291487 shutdown_connections 2024-03-06T07:28:07.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.128+0000 7fe9dc777640 1 -- 172.21.15.59:0/1765291487 wait complete. 2024-03-06T07:28:07.144 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:28:07.276 INFO:teuthology.orchestra.run.smithi059.stdout:{"message":"","return_code":0} 2024-03-06T07:28:07.277 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:07.277 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:07.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.317+0000 7f77b0447f40 1 Processor -- start 2024-03-06T07:28:07.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.317+0000 7f77b0447f40 1 -- start start 2024-03-06T07:28:07.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77b0447f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a666af2a90 0x55a666b8b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77b0447f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a666b9dd10 0x55a666ba0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77b0447f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666b9e0f0 0x55a666ba12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77b0447f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a666a6fb70 con 0x55a666b9e0f0 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77b0447f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a666b9f8b0 con 0x55a666b9dd10 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77b0447f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a666b99fd0 con 0x55a666af2a90 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77adf89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666b9e0f0 0x55a666ba12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77adf89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666b9e0f0 0x55a666ba12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49988/0 (socket says 172.21.15.59:49988) 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77adf89640 1 -- 172.21.15.59:0/2560120918 learned_addr learned my addr 172.21.15.59:0/2560120918 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77ad788640 1 --2- 172.21.15.59:0/2560120918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a666af2a90 0x55a666b8b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77acf87640 1 --2- 172.21.15.59:0/2560120918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a666b9dd10 0x55a666ba0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77adf89640 1 -- 172.21.15.59:0/2560120918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a666af2a90 msgr2=0x55a666b8b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77adf89640 1 --2- 172.21.15.59:0/2560120918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a666af2a90 0x55a666b8b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77adf89640 1 -- 172.21.15.59:0/2560120918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a666b9dd10 msgr2=0x55a666ba0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77adf89640 1 --2- 172.21.15.59:0/2560120918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a666b9dd10 0x55a666ba0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77adf89640 1 -- 172.21.15.59:0/2560120918 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a666b9b470 con 0x55a666b9e0f0 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.318+0000 7f77ad788640 1 --2- 172.21.15.59:0/2560120918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a666af2a90 0x55a666b8b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.319+0000 7f77adf89640 1 --2- 172.21.15.59:0/2560120918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666b9e0f0 0x55a666ba12f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f77a8046f00 tx=0x7f77a8051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc021166078d59e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.319+0000 7f779f7fe640 1 -- 172.21.15.59:0/2560120918 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a804e5d0 con 0x55a666b9e0f0 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.319+0000 7f779f7fe640 1 -- 172.21.15.59:0/2560120918 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a8065070 con 0x55a666b9e0f0 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.319+0000 7f779f7fe640 1 -- 172.21.15.59:0/2560120918 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a804ec40 con 0x55a666b9e0f0 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.319+0000 7f77b0447f40 1 -- 172.21.15.59:0/2560120918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666b9e0f0 msgr2=0x55a666ba12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.319+0000 7f77b0447f40 1 --2- 172.21.15.59:0/2560120918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666b9e0f0 0x55a666ba12f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f77a8046f00 tx=0x7f77a8051000 comp rx=0 tx=0).stop 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.320+0000 7f77b0447f40 1 -- 172.21.15.59:0/2560120918 shutdown_connections 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.320+0000 7f77b0447f40 1 --2- 172.21.15.59:0/2560120918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666b9e0f0 0x55a666ba12f0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.320+0000 7f77b0447f40 1 --2- 172.21.15.59:0/2560120918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a666b9dd10 0x55a666ba0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.320+0000 7f77b0447f40 1 --2- 172.21.15.59:0/2560120918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a666af2a90 0x55a666b8b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.320+0000 7f77b0447f40 1 -- 172.21.15.59:0/2560120918 >> 172.21.15.59:0/2560120918 conn(0x55a666af4bd0 msgr2=0x55a666ac8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.320+0000 7f77b0447f40 1 -- 172.21.15.59:0/2560120918 shutdown_connections 2024-03-06T07:28:07.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.320+0000 7f77b0447f40 1 -- 172.21.15.59:0/2560120918 wait complete. 2024-03-06T07:28:07.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.320+0000 7f77b0447f40 1 Processor -- start 2024-03-06T07:28:07.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.321+0000 7f77b0447f40 1 -- start start 2024-03-06T07:28:07.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77b0447f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666af2a90 0x55a666b977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77b0447f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a666b9dd10 0x55a666ac8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77b0447f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a666b9e0f0 0x55a666b899e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77b0447f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a666b9b710 con 0x55a666af2a90 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77b0447f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a666b9b9b0 con 0x55a666b9e0f0 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77b0447f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a666b957d0 con 0x55a666b9dd10 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77ad788640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666af2a90 0x55a666b977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77ad788640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666af2a90 0x55a666b977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49992/0 (socket says 172.21.15.59:49992) 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77ad788640 1 -- 172.21.15.59:0/3848238515 learned_addr learned my addr 172.21.15.59:0/3848238515 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77acf87640 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a666b9dd10 0x55a666ac8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77adf89640 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a666b9e0f0 0x55a666b899e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77ad788640 1 -- 172.21.15.59:0/3848238515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a666b9dd10 msgr2=0x55a666ac8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77ad788640 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a666b9dd10 0x55a666ac8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77ad788640 1 -- 172.21.15.59:0/3848238515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a666b9e0f0 msgr2=0x55a666b899e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77ad788640 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a666b9e0f0 0x55a666b899e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77ad788640 1 -- 172.21.15.59:0/3848238515 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a666c2c4d0 con 0x55a666af2a90 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77ad788640 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666af2a90 0x55a666b977c0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009f20 tx=0x7f77a00084c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f779e7fc640 1 -- 172.21.15.59:0/3848238515 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a0008e40 con 0x55a666af2a90 2024-03-06T07:28:07.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f779e7fc640 1 -- 172.21.15.59:0/3848238515 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a00143f0 con 0x55a666af2a90 2024-03-06T07:28:07.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77b0447f40 1 -- 172.21.15.59:0/3848238515 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a666b95180 con 0x55a666af2a90 2024-03-06T07:28:07.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f779e7fc640 1 -- 172.21.15.59:0/3848238515 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a00147c0 con 0x55a666af2a90 2024-03-06T07:28:07.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.322+0000 7f77b0447f40 1 -- 172.21.15.59:0/3848238515 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a666ac92e0 con 0x55a666af2a90 2024-03-06T07:28:07.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.324+0000 7f779e7fc640 1 -- 172.21.15.59:0/3848238515 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77a001b020 con 0x55a666af2a90 2024-03-06T07:28:07.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.324+0000 7f779e7fc640 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7790069eb0 0x7f779006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.325+0000 7f779e7fc640 1 -- 172.21.15.59:0/3848238515 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f77a00888a0 con 0x55a666af2a90 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.325+0000 7f77acf87640 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7790069eb0 0x7f779006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.325+0000 7f77b0447f40 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x55a666c2bfc0 0x55a666af69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.325+0000 7f77b0447f40 1 -- 172.21.15.59:0/3848238515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x55a666af6f00 con 0x55a666c2bfc0 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.325+0000 7f77adf89640 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x55a666c2bfc0 0x55a666af69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.325+0000 7f77adf89640 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x55a666c2bfc0 0x55a666af69c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.325+0000 7f77acf87640 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7790069eb0 0x7f779006c370 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f77a4000c00 tx=0x7f77a40015f0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77adf89640 1 -- 172.21.15.59:0/3848238515 <== osd.0 v2:172.21.15.59:6824/841983189 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7f77a804ec00 con 0x55a666c2bfc0 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77b0447f40 1 -- 172.21.15.59:0/3848238515 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x55a666c2bfc0 msgr2=0x55a666af69c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77b0447f40 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x55a666c2bfc0 0x55a666af69c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77b0447f40 1 -- 172.21.15.59:0/3848238515 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7790069eb0 msgr2=0x7f779006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77b0447f40 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7790069eb0 0x7f779006c370 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f77a4000c00 tx=0x7f77a40015f0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77b0447f40 1 -- 172.21.15.59:0/3848238515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666af2a90 msgr2=0x55a666b977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77b0447f40 1 --2- 172.21.15.59:0/3848238515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a666af2a90 0x55a666b977c0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009f20 tx=0x7f77a00084c0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77adf89640 1 -- 172.21.15.59:0/3848238515 reap_dead start 2024-03-06T07:28:07.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77b0447f40 1 -- 172.21.15.59:0/3848238515 shutdown_connections 2024-03-06T07:28:07.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77b0447f40 1 -- 172.21.15.59:0/3848238515 >> 172.21.15.59:0/3848238515 conn(0x55a666af4bd0 msgr2=0x55a666ba0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77b0447f40 1 -- 172.21.15.59:0/3848238515 shutdown_connections 2024-03-06T07:28:07.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.326+0000 7f77b0447f40 1 -- 172.21.15.59:0/3848238515 wait complete. 2024-03-06T07:28:07.332 INFO:teuthology.orchestra.run.smithi059.stdout:] 10000000001 stray0 2024-03-06T07:28:07.332 DEBUG:teuthology.orchestra.run.smithi059:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:07.419 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:28:07.419 INFO:teuthology.orchestra.run.smithi059.stdout: "ino": 1099511627777, 2024-03-06T07:28:07.419 INFO:teuthology.orchestra.run.smithi059.stdout: "ancestors": [ 2024-03-06T07:28:07.419 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:07.420 INFO:teuthology.orchestra.run.smithi059.stdout: "dirino": 1536, 2024-03-06T07:28:07.420 INFO:teuthology.orchestra.run.smithi059.stdout: "dname": "10000000001", 2024-03-06T07:28:07.420 INFO:teuthology.orchestra.run.smithi059.stdout: "version": 11 2024-03-06T07:28:07.420 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:07.420 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:07.420 INFO:teuthology.orchestra.run.smithi059.stdout: "dirino": 256, 2024-03-06T07:28:07.420 INFO:teuthology.orchestra.run.smithi059.stdout: "dname": "stray0", 2024-03-06T07:28:07.420 INFO:teuthology.orchestra.run.smithi059.stdout: "version": 5 2024-03-06T07:28:07.420 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:07.420 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-03-06T07:28:07.420 INFO:teuthology.orchestra.run.smithi059.stdout: "pool": 17, 2024-03-06T07:28:07.421 INFO:teuthology.orchestra.run.smithi059.stdout: "old_pools": [] 2024-03-06T07:28:07.421 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:28:07.421 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:07.422 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:07.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.563+0000 7f85e513e640 1 Processor -- start 2024-03-06T07:28:07.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.563+0000 7f85e513e640 1 -- start start 2024-03-06T07:28:07.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.563+0000 7f85e513e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f85e00d2710 0x7f85e00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85e513e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e00b3430 con 0x7f85e015dfe0 2024-03-06T07:28:07.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85e513e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e00b3710 con 0x7f85e00d22b0 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85e513e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e00b31e0 con 0x7f85e00d2710 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ded9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f85e015dfe0 0x7f85e00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34742/0 (socket says 172.21.15.59:34742) 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ded9d640 1 -- 172.21.15.59:0/3380066400 learned_addr learned my addr 172.21.15.59:0/3380066400 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85df59e640 1 --2- 172.21.15.59:0/3380066400 >> v2:172.21.15.164:3300/0 conn(0x7f85e00d2710 0x7f85e00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113625837 0 0) 0x7f85e00b3430 con 0x7f85e015dfe0 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c8003520 con 0x7f85e015dfe0 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3913416479 0 0) 0x7f85c8003520 con 0x7f85e015dfe0 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 >> v2:172.21.15.164:3300/0 conn(0x7f85e00d2710 msgr2=0x7f85e00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 --2- 172.21.15.59:0/3380066400 >> v2:172.21.15.164:3300/0 conn(0x7f85e00d2710 0x7f85e00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 >> v1:172.21.15.164:6789/0 conn(0x7f85e00d22b0 legacy=0x7f85e00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e00b39f0 con 0x7f85e015dfe0 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f85d0001670 con 0x7f85e015dfe0 2024-03-06T07:28:07.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85df59e640 1 --2- 172.21.15.59:0/3380066400 >> v2:172.21.15.164:3300/0 conn(0x7f85e00d2710 0x7f85e00b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 >> v1:172.21.15.59:6789/0 conn(0x7f85e015dfe0 legacy=0x7f85e00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 --2- 172.21.15.59:0/3380066400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85c80039e0 0x7f85c8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 --2- 172.21.15.59:0/3380066400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85c8004800 0x7f85c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 --2- 172.21.15.59:0/3380066400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85c80051c0 0x7f85c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e00b31e0 con 0x7f85c8004800 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c800c090 con 0x7f85c80051c0 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.564+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c800c290 con 0x7f85c80039e0 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.565+0000 7f85de59c640 1 --2- 172.21.15.59:0/3380066400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85c8004800 0x7f85c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.565+0000 7f85ded9d640 1 --2- 172.21.15.59:0/3380066400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85c80039e0 0x7f85c8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.565+0000 7f85df59e640 1 --2- 172.21.15.59:0/3380066400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85c80051c0 0x7f85c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.565+0000 7f85de59c640 1 -- 172.21.15.59:0/3380066400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85c80039e0 msgr2=0x7f85c8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.565+0000 7f85de59c640 1 --2- 172.21.15.59:0/3380066400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85c80039e0 0x7f85c8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.565+0000 7f85de59c640 1 -- 172.21.15.59:0/3380066400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85c80051c0 msgr2=0x7f85c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.565+0000 7f85de59c640 1 --2- 172.21.15.59:0/3380066400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85c80051c0 0x7f85c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.565+0000 7f85de59c640 1 -- 172.21.15.59:0/3380066400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c8003520 con 0x7f85c8004800 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.565+0000 7f85df59e640 1 -- 172.21.15.59:0/3380066400 reap_dead start 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.566+0000 7f85de59c640 1 --2- 172.21.15.59:0/3380066400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85c8004800 0x7f85c8004c80 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f85d4001750 tx=0x7f85c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d821d83dcb8dd24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.566+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d4001370 con 0x7f85c8004800 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.566+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d40052d0 con 0x7f85c8004800 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.566+0000 7f85ddd9b640 1 -- 172.21.15.59:0/3380066400 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d4003830 con 0x7f85c8004800 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.566+0000 7f85e513e640 1 -- 172.21.15.59:0/3380066400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85c8004800 msgr2=0x7f85c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.566+0000 7f85e513e640 1 --2- 172.21.15.59:0/3380066400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85c8004800 0x7f85c8004c80 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f85d4001750 tx=0x7f85c8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.566+0000 7f85e513e640 1 -- 172.21.15.59:0/3380066400 shutdown_connections 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.566+0000 7f85e513e640 1 --2- 172.21.15.59:0/3380066400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85c8004800 0x7f85c8004c80 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.566+0000 7f85e513e640 1 -- 172.21.15.59:0/3380066400 >> 172.21.15.59:0/3380066400 conn(0x7f85e00d3cd0 msgr2=0x7f85e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.566+0000 7f85e513e640 1 -- 172.21.15.59:0/3380066400 shutdown_connections 2024-03-06T07:28:07.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.566+0000 7f85e513e640 1 -- 172.21.15.59:0/3380066400 wait complete. 2024-03-06T07:28:07.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.568+0000 7f85e513e640 1 Processor -- start 2024-03-06T07:28:07.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.568+0000 7f85e513e640 1 -- start start 2024-03-06T07:28:07.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.568+0000 7f85e513e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85c8004800 0x7f85e01572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85e513e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85e01f41a0 0x7f85e01f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85e513e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85e01f72c0 0x7f85e006c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85e513e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e00d1c70 con 0x7f85e01f72c0 2024-03-06T07:28:07.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85e513e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e00b3d60 con 0x7f85e01f41a0 2024-03-06T07:28:07.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85e513e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e00d1df0 con 0x7f85c8004800 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85df59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85e01f72c0 0x7f85e006c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85df59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85e01f72c0 0x7f85e006c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50016/0 (socket says 172.21.15.59:50016) 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85ded9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85c8004800 0x7f85e01572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85de59c640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85e01f41a0 0x7f85e01f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85df59e640 1 -- 172.21.15.59:0/4034104352 learned_addr learned my addr 172.21.15.59:0/4034104352 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85df59e640 1 -- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85c8004800 msgr2=0x7f85e01572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85df59e640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85c8004800 0x7f85e01572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85df59e640 1 -- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85e01f41a0 msgr2=0x7f85e01f6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85df59e640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85e01f41a0 0x7f85e01f6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85df59e640 1 -- 172.21.15.59:0/4034104352 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e01f7760 con 0x7f85e01f72c0 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85de59c640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85e01f41a0 0x7f85e01f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85ded9d640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85c8004800 0x7f85e01572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.569+0000 7f85df59e640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85e01f72c0 0x7f85e006c840 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f85e0161bf0 tx=0x7f85d800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.570+0000 7f85c77fe640 1 -- 172.21.15.59:0/4034104352 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d8003990 con 0x7f85e01f72c0 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.570+0000 7f85c77fe640 1 -- 172.21.15.59:0/4034104352 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d8003b30 con 0x7f85e01f72c0 2024-03-06T07:28:07.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.570+0000 7f85c77fe640 1 -- 172.21.15.59:0/4034104352 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d8004020 con 0x7f85e01f72c0 2024-03-06T07:28:07.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.570+0000 7f85e513e640 1 -- 172.21.15.59:0/4034104352 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e01f7a40 con 0x7f85e01f72c0 2024-03-06T07:28:07.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.570+0000 7f85e513e640 1 -- 172.21.15.59:0/4034104352 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e01f4620 con 0x7f85e01f72c0 2024-03-06T07:28:07.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.571+0000 7f85e513e640 1 -- 172.21.15.59:0/4034104352 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e01f37b0 con 0x7f85e01f72c0 2024-03-06T07:28:07.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.571+0000 7f85c77fe640 1 -- 172.21.15.59:0/4034104352 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85d80041c0 con 0x7f85e01f72c0 2024-03-06T07:28:07.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.572+0000 7f85c77fe640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f85cc06b4d0 0x7f85cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.572+0000 7f85c77fe640 1 -- 172.21.15.59:0/4034104352 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f85d808af40 con 0x7f85e01f72c0 2024-03-06T07:28:07.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.572+0000 7f85ded9d640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f85cc06b4d0 0x7f85cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.572+0000 7f85ded9d640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f85cc06b4d0 0x7f85cc06d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f85e01f7e80 tx=0x7f85d0000ed0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.576+0000 7f85c77fe640 1 -- 172.21.15.59:0/4034104352 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85d8019030 con 0x7f85e01f72c0 2024-03-06T07:28:07.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.818+0000 7f85e513e640 1 -- 172.21.15.59:0/4034104352 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85e004f330 con 0x7f85e01f72c0 2024-03-06T07:28:07.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.818+0000 7f85c77fe640 1 -- 172.21.15.59:0/4034104352 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f85d8010040 con 0x7f85e01f72c0 2024-03-06T07:28:07.820 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:07.821 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:07.821 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:07.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.821+0000 7f85e513e640 1 -- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f85cc06b4d0 msgr2=0x7f85cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.821+0000 7f85e513e640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f85cc06b4d0 0x7f85cc06d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f85e01f7e80 tx=0x7f85d0000ed0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.821+0000 7f85e513e640 1 -- 172.21.15.59:0/4034104352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85e01f72c0 msgr2=0x7f85e006c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.821+0000 7f85e513e640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85e01f72c0 0x7f85e006c840 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f85e0161bf0 tx=0x7f85d800f580 comp rx=0 tx=0).stop 2024-03-06T07:28:07.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.821+0000 7f85e513e640 1 -- 172.21.15.59:0/4034104352 shutdown_connections 2024-03-06T07:28:07.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.821+0000 7f85e513e640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f85cc06b4d0 0x7f85cc06d990 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.821+0000 7f85e513e640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85e01f72c0 0x7f85e006c840 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.821+0000 7f85e513e640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85e01f41a0 0x7f85e01f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.821+0000 7f85e513e640 1 --2- 172.21.15.59:0/4034104352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85c8004800 0x7f85e01572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.821+0000 7f85e513e640 1 -- 172.21.15.59:0/4034104352 >> 172.21.15.59:0/4034104352 conn(0x7f85e00d3cd0 msgr2=0x7f85e0160a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.821+0000 7f85e513e640 1 -- 172.21.15.59:0/4034104352 shutdown_connections 2024-03-06T07:28:07.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:07.822+0000 7f85e513e640 1 -- 172.21.15.59:0/4034104352 wait complete. 2024-03-06T07:28:07.835 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:07.952 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-03-06T07:28:07.952 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-03-06T07:28:07.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:07.952 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-03-06T07:28:07.978 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:08.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.117+0000 7fa4a185c640 1 Processor -- start 2024-03-06T07:28:08.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.118+0000 7fa4a185c640 1 -- start start 2024-03-06T07:28:08.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.118+0000 7fa4a185c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa49c0b4170 0x7fa49c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.118+0000 7fa4a185c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa49c0d0a80 0x7fa49c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.118+0000 7fa4a185c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49c0d25d0 con 0x7fa49c0d4760 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.118+0000 7fa4a185c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa49c0d4bf0 con 0x7fa49c0d0a80 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.118+0000 7fa4a185c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c0ce9e0 con 0x7fa49c0b4170 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.119+0000 7fa49affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa49c0b4170 0x7fa49c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.119+0000 7fa49b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa49c0d0a80 0x7fa49c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.119+0000 7fa49b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa49c0d0a80 0x7fa49c0cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:40414/0 (socket says 172.21.15.59:40414) 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.119+0000 7fa49b7fe640 1 -- 172.21.15.59:0/3747286342 learned_addr learned my addr 172.21.15.59:0/3747286342 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.119+0000 7fa49a7fc640 1 --1- 172.21.15.59:0/3747286342 >> v1:172.21.15.164:6789/0 conn(0x7fa49c0d4760 0x7fa49c0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:38458/0 (socket says 172.21.15.59:38458) 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.119+0000 7fa49b7fe640 1 -- 172.21.15.59:0/3747286342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa49c0b4170 msgr2=0x7fa49c0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.119+0000 7fa49b7fe640 1 --2- 172.21.15.59:0/3747286342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa49c0b4170 0x7fa49c0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.119+0000 7fa49b7fe640 1 -- 172.21.15.59:0/3747286342 >> v1:172.21.15.164:6789/0 conn(0x7fa49c0d4760 legacy=0x7fa49c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.119+0000 7fa49b7fe640 1 -- 172.21.15.59:0/3747286342 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c0d2870 con 0x7fa49c0d0a80 2024-03-06T07:28:08.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.119+0000 7fa499ffb640 1 -- 172.21.15.59:0/3747286342 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875922417 0 0) 0x7fa49c0d25d0 con 0x7fa49c0d4760 2024-03-06T07:28:08.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.120+0000 7fa49b7fe640 1 --2- 172.21.15.59:0/3747286342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa49c0d0a80 0x7fa49c0cee70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa494009e50 tx=0x7fa49402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6744eae91e22d486 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.120+0000 7fa499ffb640 1 -- 172.21.15.59:0/3747286342 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49402cb00 con 0x7fa49c0d0a80 2024-03-06T07:28:08.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.120+0000 7fa499ffb640 1 -- 172.21.15.59:0/3747286342 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49402cca0 con 0x7fa49c0d0a80 2024-03-06T07:28:08.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.120+0000 7fa499ffb640 1 -- 172.21.15.59:0/3747286342 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494034610 con 0x7fa49c0d0a80 2024-03-06T07:28:08.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.121+0000 7fa4a185c640 1 -- 172.21.15.59:0/3747286342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa49c0d0a80 msgr2=0x7fa49c0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.121+0000 7fa4a185c640 1 --2- 172.21.15.59:0/3747286342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa49c0d0a80 0x7fa49c0cee70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa494009e50 tx=0x7fa49402bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.121+0000 7fa4a185c640 1 -- 172.21.15.59:0/3747286342 shutdown_connections 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.121+0000 7fa4a185c640 1 --2- 172.21.15.59:0/3747286342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa49c0d0a80 0x7fa49c0cee70 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.121+0000 7fa4a185c640 1 --2- 172.21.15.59:0/3747286342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa49c0b4170 0x7fa49c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.121+0000 7fa4a185c640 1 -- 172.21.15.59:0/3747286342 >> 172.21.15.59:0/3747286342 conn(0x7fa49c0d3c40 msgr2=0x7fa49c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.121+0000 7fa4a185c640 1 -- 172.21.15.59:0/3747286342 shutdown_connections 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.121+0000 7fa4a185c640 1 -- 172.21.15.59:0/3747286342 wait complete. 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.122+0000 7fa4a185c640 1 Processor -- start 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.122+0000 7fa4a185c640 1 -- start start 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa4a185c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa49c0b4170 0x7fa49c15af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa4a185c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa49c0d0a80 0x7fa49c0b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa4a185c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa49c15b470 0x7fa49c0b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa4a185c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c0b4dd0 con 0x7fa49c0b4170 2024-03-06T07:28:08.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa4a185c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c0b4550 con 0x7fa49c0d0a80 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa4a185c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c0b4f50 con 0x7fa49c15b470 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa49c0b4170 0x7fa49c15af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa49c0b4170 0x7fa49c15af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50026/0 (socket says 172.21.15.59:50026) 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49affd640 1 -- 172.21.15.59:0/3825819832 learned_addr learned my addr 172.21.15.59:0/3825819832 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49a7fc640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa49c0d0a80 0x7fa49c0b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49b7fe640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa49c15b470 0x7fa49c0b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49affd640 1 -- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa49c15b470 msgr2=0x7fa49c0b6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49affd640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa49c15b470 0x7fa49c0b6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49affd640 1 -- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa49c0d0a80 msgr2=0x7fa49c0b6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49affd640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa49c0d0a80 0x7fa49c0b6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49affd640 1 -- 172.21.15.59:0/3825819832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c0b2a20 con 0x7fa49c0b4170 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49b7fe640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa49c15b470 0x7fa49c0b6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa49affd640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa49c0b4170 0x7fa49c15af30 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00e9b0 tx=0x7fa48c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.128 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa483fff640 1 -- 172.21.15.59:0/3825819832 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c017070 con 0x7fa49c0b4170 2024-03-06T07:28:08.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa483fff640 1 -- 172.21.15.59:0/3825819832 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48c00c9a0 con 0x7fa49c0b4170 2024-03-06T07:28:08.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa4a185c640 1 -- 172.21.15.59:0/3825819832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa49c0b2d00 con 0x7fa49c0b4170 2024-03-06T07:28:08.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.123+0000 7fa483fff640 1 -- 172.21.15.59:0/3825819832 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c00cd90 con 0x7fa49c0b4170 2024-03-06T07:28:08.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.124+0000 7fa4a185c640 1 -- 172.21.15.59:0/3825819832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa49c0b32c0 con 0x7fa49c0b4170 2024-03-06T07:28:08.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.125+0000 7fa4a185c640 1 -- 172.21.15.59:0/3825819832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa49c04f330 con 0x7fa49c0b4170 2024-03-06T07:28:08.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.125+0000 7fa483fff640 1 -- 172.21.15.59:0/3825819832 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa48c01a070 con 0x7fa49c0b4170 2024-03-06T07:28:08.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.125+0000 7fa483fff640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa48806b4d0 0x7fa48806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.125+0000 7fa483fff640 1 -- 172.21.15.59:0/3825819832 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa48c08a6d0 con 0x7fa49c0b4170 2024-03-06T07:28:08.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.126+0000 7fa49a7fc640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa48806b4d0 0x7fa48806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.126+0000 7fa49a7fc640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa48806b4d0 0x7fa48806d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa490000f60 tx=0x7fa490007c00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.134+0000 7fa483fff640 1 -- 172.21.15.59:0/3825819832 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa48c059440 con 0x7fa49c0b4170 2024-03-06T07:28:08.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.365+0000 7fa4a185c640 1 -- 172.21.15.59:0/3825819832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa49c0d1020 con 0x7fa49c0b4170 2024-03-06T07:28:08.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.366+0000 7fa483fff640 1 -- 172.21.15.59:0/3825819832 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fa48c058d40 con 0x7fa49c0b4170 2024-03-06T07:28:08.368 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:08.368 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:08.368 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:08.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 -- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa48806b4d0 msgr2=0x7fa48806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa48806b4d0 0x7fa48806d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa490000f60 tx=0x7fa490007c00 comp rx=0 tx=0).stop 2024-03-06T07:28:08.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 -- 172.21.15.59:0/3825819832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa49c0b4170 msgr2=0x7fa49c15af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa49c0b4170 0x7fa49c15af30 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00e9b0 tx=0x7fa48c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:28:08.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 -- 172.21.15.59:0/3825819832 shutdown_connections 2024-03-06T07:28:08.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa48806b4d0 0x7fa48806d990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa49c15b470 0x7fa49c0b6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa49c0d0a80 0x7fa49c0b6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 --2- 172.21.15.59:0/3825819832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa49c0b4170 0x7fa49c15af30 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 -- 172.21.15.59:0/3825819832 >> 172.21.15.59:0/3825819832 conn(0x7fa49c0d3c40 msgr2=0x7fa49c0d02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 -- 172.21.15.59:0/3825819832 shutdown_connections 2024-03-06T07:28:08.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.368+0000 7fa4a185c640 1 -- 172.21.15.59:0/3825819832 wait complete. 2024-03-06T07:28:08.381 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:08.498 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:28:08.499 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:28:08.499 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:08.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.599+0000 7f3f9d120640 1 Processor -- start 2024-03-06T07:28:08.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.599+0000 7f3f9d120640 1 -- start start 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.599+0000 7f3f9d120640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3f9815fa10 0x7f3f980ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.599+0000 7f3f9d120640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3f980d16e0 0x7f3f980d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.599+0000 7f3f9d120640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f980cebe0 con 0x7f3f9815bdd0 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.599+0000 7f3f9d120640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f980ceec0 con 0x7f3f980d16e0 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.599+0000 7f3f9d120640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f980cf1a0 con 0x7f3f9815fa10 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.599+0000 7f3f9759e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3f980d16e0 0x7f3f980d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.599+0000 7f3f96d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3f9815bdd0 0x7f3f980c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34758/0 (socket says 172.21.15.59:34758) 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.599+0000 7f3f96d9d640 1 -- 172.21.15.59:0/432393731 learned_addr learned my addr 172.21.15.59:0/432393731 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.600+0000 7f3f9659c640 1 --2- 172.21.15.59:0/432393731 >> v2:172.21.15.164:3300/0 conn(0x7f3f9815fa10 0x7f3f980ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.600+0000 7f3f9759e640 1 -- 172.21.15.59:0/432393731 >> v2:172.21.15.164:3300/0 conn(0x7f3f9815fa10 msgr2=0x7f3f980ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.600+0000 7f3f9759e640 1 --2- 172.21.15.59:0/432393731 >> v2:172.21.15.164:3300/0 conn(0x7f3f9815fa10 0x7f3f980ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.600+0000 7f3f9759e640 1 -- 172.21.15.59:0/432393731 >> v1:172.21.15.59:6789/0 conn(0x7f3f9815bdd0 legacy=0x7f3f980c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.600+0000 7f3f9759e640 1 -- 172.21.15.59:0/432393731 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f980cc8e0 con 0x7f3f980d16e0 2024-03-06T07:28:08.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.600+0000 7f3f95d9b640 1 -- 172.21.15.59:0/432393731 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211723180 0 0) 0x7f3f980cebe0 con 0x7f3f9815bdd0 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f9759e640 1 --2- 172.21.15.59:0/432393731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f980d16e0 0x7f3f980d1af0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f3f980cdfd0 tx=0x7f3f8802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a91d9ca30a4da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f95d9b640 1 -- 172.21.15.59:0/432393731 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88039070 con 0x7f3f980d16e0 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f95d9b640 1 -- 172.21.15.59:0/432393731 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8802cbf0 con 0x7f3f980d16e0 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f95d9b640 1 -- 172.21.15.59:0/432393731 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88034440 con 0x7f3f980d16e0 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f9d120640 1 -- 172.21.15.59:0/432393731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f980d16e0 msgr2=0x7f3f980d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f9d120640 1 --2- 172.21.15.59:0/432393731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f980d16e0 0x7f3f980d1af0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f3f980cdfd0 tx=0x7f3f8802be40 comp rx=0 tx=0).stop 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f9d120640 1 -- 172.21.15.59:0/432393731 shutdown_connections 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f9d120640 1 --2- 172.21.15.59:0/432393731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f980d16e0 0x7f3f980d1af0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f9d120640 1 --2- 172.21.15.59:0/432393731 >> v2:172.21.15.164:3300/0 conn(0x7f3f9815fa10 0x7f3f980ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f9d120640 1 -- 172.21.15.59:0/432393731 >> 172.21.15.59:0/432393731 conn(0x7f3f9809a9f0 msgr2=0x7f3f9809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f9d120640 1 -- 172.21.15.59:0/432393731 shutdown_connections 2024-03-06T07:28:08.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.601+0000 7f3f9d120640 1 -- 172.21.15.59:0/432393731 wait complete. 2024-03-06T07:28:08.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.603+0000 7f3f9d120640 1 Processor -- start 2024-03-06T07:28:08.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.603+0000 7f3f9d120640 1 -- start start 2024-03-06T07:28:08.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.604+0000 7f3f9d120640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f980d16e0 0x7f3f98154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.604+0000 7f3f9d120640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9815bdd0 0x7f3f9815cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.604+0000 7f3f9d120640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9815fa10 0x7f3f980b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.604+0000 7f3f9d120640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f980d2030 con 0x7f3f980d16e0 2024-03-06T07:28:08.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.604+0000 7f3f9d120640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9815c1b0 con 0x7f3f9815fa10 2024-03-06T07:28:08.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.604+0000 7f3f9d120640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f980ce7b0 con 0x7f3f9815bdd0 2024-03-06T07:28:08.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.604+0000 7f3f96d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f980d16e0 0x7f3f98154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.604+0000 7f3f96d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f980d16e0 0x7f3f98154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50038/0 (socket says 172.21.15.59:50038) 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.604+0000 7f3f96d9d640 1 -- 172.21.15.59:0/2347817455 learned_addr learned my addr 172.21.15.59:0/2347817455 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f9759e640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9815fa10 0x7f3f980b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f96d9d640 1 -- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9815bdd0 msgr2=0x7f3f9815cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f9659c640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9815bdd0 0x7f3f9815cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f96d9d640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9815bdd0 0x7f3f9815cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f96d9d640 1 -- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9815fa10 msgr2=0x7f3f980b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f96d9d640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9815fa10 0x7f3f980b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f96d9d640 1 -- 172.21.15.59:0/2347817455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9815d040 con 0x7f3f980d16e0 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f9759e640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9815fa10 0x7f3f980b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f96d9d640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f980d16e0 0x7f3f98154b40 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f3f9000c4a0 tx=0x7f3f9000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f7f7fe640 1 -- 172.21.15.59:0/2347817455 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f90017070 con 0x7f3f980d16e0 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f7f7fe640 1 -- 172.21.15.59:0/2347817455 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f90005ce0 con 0x7f3f980d16e0 2024-03-06T07:28:08.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f9d120640 1 -- 172.21.15.59:0/2347817455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f980b5040 con 0x7f3f980d16e0 2024-03-06T07:28:08.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f7f7fe640 1 -- 172.21.15.59:0/2347817455 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f900072b0 con 0x7f3f980d16e0 2024-03-06T07:28:08.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.605+0000 7f3f9d120640 1 -- 172.21.15.59:0/2347817455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f980b5600 con 0x7f3f980d16e0 2024-03-06T07:28:08.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.606+0000 7f3f9d120640 1 -- 172.21.15.59:0/2347817455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9804f330 con 0x7f3f980d16e0 2024-03-06T07:28:08.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.607+0000 7f3f7f7fe640 1 -- 172.21.15.59:0/2347817455 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f9001a070 con 0x7f3f980d16e0 2024-03-06T07:28:08.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.607+0000 7f3f7f7fe640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f8406b4d0 0x7f3f8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.607+0000 7f3f7f7fe640 1 -- 172.21.15.59:0/2347817455 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3f900896b0 con 0x7f3f980d16e0 2024-03-06T07:28:08.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.607+0000 7f3f9659c640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f8406b4d0 0x7f3f8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.608+0000 7f3f9659c640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f8406b4d0 0x7f3f8406d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3f980b7970 tx=0x7f3f8c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.611+0000 7f3f7f7fe640 1 -- 172.21.15.59:0/2347817455 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f900584a0 con 0x7f3f980d16e0 2024-03-06T07:28:08.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.829+0000 7f3f9d120640 1 -- 172.21.15.59:0/2347817455 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f980b7a40 con 0x7f3f980d16e0 2024-03-06T07:28:08.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.830+0000 7f3f7f7fe640 1 -- 172.21.15.59:0/2347817455 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f3f90057da0 con 0x7f3f980d16e0 2024-03-06T07:28:08.832 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:08.832 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:08.832 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:08.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.831+0000 7f3f9d120640 1 -- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f8406b4d0 msgr2=0x7f3f8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.831+0000 7f3f9d120640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f8406b4d0 0x7f3f8406d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3f980b7970 tx=0x7f3f8c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:08.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.831+0000 7f3f9d120640 1 -- 172.21.15.59:0/2347817455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f980d16e0 msgr2=0x7f3f98154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.831+0000 7f3f9d120640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f980d16e0 0x7f3f98154b40 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f3f9000c4a0 tx=0x7f3f9000c960 comp rx=0 tx=0).stop 2024-03-06T07:28:08.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.831+0000 7f3f9d120640 1 -- 172.21.15.59:0/2347817455 shutdown_connections 2024-03-06T07:28:08.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.831+0000 7f3f9d120640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f8406b4d0 0x7f3f8406d990 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.831+0000 7f3f9d120640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f9815fa10 0x7f3f980b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.831+0000 7f3f9d120640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9815bdd0 0x7f3f9815cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.831+0000 7f3f9d120640 1 --2- 172.21.15.59:0/2347817455 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f980d16e0 0x7f3f98154b40 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.831+0000 7f3f9d120640 1 -- 172.21.15.59:0/2347817455 >> 172.21.15.59:0/2347817455 conn(0x7f3f9809a9f0 msgr2=0x7f3f980cd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.832+0000 7f3f9d120640 1 -- 172.21.15.59:0/2347817455 shutdown_connections 2024-03-06T07:28:08.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:08.832+0000 7f3f9d120640 1 -- 172.21.15.59:0/2347817455 wait complete. 2024-03-06T07:28:08.848 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:08.964 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-03-06T07:28:08.965 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-03-06T07:28:08.965 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:09.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.064+0000 7f059842a640 1 Processor -- start 2024-03-06T07:28:09.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.064+0000 7f059842a640 1 -- start start 2024-03-06T07:28:09.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.065+0000 7f059842a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f059015dfe0 0x7f05900cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.065+0000 7f059842a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05900b3430 con 0x7f05900cf830 2024-03-06T07:28:09.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.065+0000 7f059842a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05900b3710 con 0x7f05900d29c0 2024-03-06T07:28:09.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.065+0000 7f059842a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05900b31e0 con 0x7f059015dfe0 2024-03-06T07:28:09.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.065+0000 7f05959c5640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f05900cf830 0x7f05900d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34768/0 (socket says 172.21.15.59:34768) 2024-03-06T07:28:09.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.065+0000 7f05959c5640 1 -- 172.21.15.59:0/976901946 learned_addr learned my addr 172.21.15.59:0/976901946 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:09.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.065+0000 7f05961c6640 1 --2- 172.21.15.59:0/976901946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f059015dfe0 0x7f05900cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.066+0000 7f05951c4640 1 -- 172.21.15.59:0/976901946 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172278901 0 0) 0x7f05900b3430 con 0x7f05900cf830 2024-03-06T07:28:09.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.066+0000 7f05951c4640 1 -- 172.21.15.59:0/976901946 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f057c003520 con 0x7f05900cf830 2024-03-06T07:28:09.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.066+0000 7f05951c4640 1 -- 172.21.15.59:0/976901946 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053363888 0 0) 0x7f05900b3710 con 0x7f05900d29c0 2024-03-06T07:28:09.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.066+0000 7f05951c4640 1 -- 172.21.15.59:0/976901946 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05900b3430 con 0x7f05900d29c0 2024-03-06T07:28:09.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.066+0000 7f05961c6640 1 -- 172.21.15.59:0/976901946 >> v1:172.21.15.164:6789/0 conn(0x7f05900d29c0 legacy=0x7f05900b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.066+0000 7f05961c6640 1 -- 172.21.15.59:0/976901946 >> v1:172.21.15.59:6789/0 conn(0x7f05900cf830 legacy=0x7f05900d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.066+0000 7f05961c6640 1 -- 172.21.15.59:0/976901946 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05900b39f0 con 0x7f059015dfe0 2024-03-06T07:28:09.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.067+0000 7f05961c6640 1 --2- 172.21.15.59:0/976901946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f059015dfe0 0x7f05900cf2f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f058c0097d0 tx=0x7f058c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ada6794ce2343692 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.067+0000 7f05951c4640 1 -- 172.21.15.59:0/976901946 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c035020 con 0x7f059015dfe0 2024-03-06T07:28:09.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.067+0000 7f05951c4640 1 -- 172.21.15.59:0/976901946 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c031890 con 0x7f059015dfe0 2024-03-06T07:28:09.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.067+0000 7f05951c4640 1 -- 172.21.15.59:0/976901946 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c031c40 con 0x7f059015dfe0 2024-03-06T07:28:09.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.068+0000 7f059842a640 1 -- 172.21.15.59:0/976901946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f059015dfe0 msgr2=0x7f05900cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.068+0000 7f059842a640 1 --2- 172.21.15.59:0/976901946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f059015dfe0 0x7f05900cf2f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f058c0097d0 tx=0x7f058c02b930 comp rx=0 tx=0).stop 2024-03-06T07:28:09.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.068+0000 7f059842a640 1 -- 172.21.15.59:0/976901946 shutdown_connections 2024-03-06T07:28:09.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.068+0000 7f059842a640 1 --2- 172.21.15.59:0/976901946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f059015dfe0 0x7f05900cf2f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.068+0000 7f059842a640 1 -- 172.21.15.59:0/976901946 >> 172.21.15.59:0/976901946 conn(0x7f05900d3cd0 msgr2=0x7f05900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.068+0000 7f059842a640 1 -- 172.21.15.59:0/976901946 shutdown_connections 2024-03-06T07:28:09.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.068+0000 7f059842a640 1 -- 172.21.15.59:0/976901946 wait complete. 2024-03-06T07:28:09.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.069+0000 7f059842a640 1 Processor -- start 2024-03-06T07:28:09.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.070+0000 7f059842a640 1 -- start start 2024-03-06T07:28:09.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f059842a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05900cf830 0x7f05901f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f059842a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900d29c0 0x7f05900cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f059842a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f059015dfe0 0x7f05900b8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f059842a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059015e3c0 con 0x7f05900d29c0 2024-03-06T07:28:09.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f059842a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05900b3d60 con 0x7f059015dfe0 2024-03-06T07:28:09.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f059842a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059015e540 con 0x7f05900cf830 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05959c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900d29c0 0x7f05900cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05959c5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900d29c0 0x7f05900cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50054/0 (socket says 172.21.15.59:50054) 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05959c5640 1 -- 172.21.15.59:0/2896153159 learned_addr learned my addr 172.21.15.59:0/2896153159 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05969c7640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f059015dfe0 0x7f05900b8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05961c6640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05900cf830 0x7f05901f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05959c5640 1 -- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05900cf830 msgr2=0x7f05901f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05959c5640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05900cf830 0x7f05901f2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05959c5640 1 -- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f059015dfe0 msgr2=0x7f05900b8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05959c5640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f059015dfe0 0x7f05900b8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05959c5640 1 -- 172.21.15.59:0/2896153159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05900cc300 con 0x7f05900d29c0 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05969c7640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f059015dfe0 0x7f05900b8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:09.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f05959c5640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900d29c0 0x7f05900cada0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f058000c4d0 tx=0x7f058000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.071+0000 7f0586ffd640 1 -- 172.21.15.59:0/2896153159 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580017070 con 0x7f05900d29c0 2024-03-06T07:28:09.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.072+0000 7f0586ffd640 1 -- 172.21.15.59:0/2896153159 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0580005ce0 con 0x7f05900d29c0 2024-03-06T07:28:09.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.072+0000 7f059842a640 1 -- 172.21.15.59:0/2896153159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05900cb780 con 0x7f05900d29c0 2024-03-06T07:28:09.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.072+0000 7f0586ffd640 1 -- 172.21.15.59:0/2896153159 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580007270 con 0x7f05900d29c0 2024-03-06T07:28:09.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.072+0000 7f059842a640 1 -- 172.21.15.59:0/2896153159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05900cbd40 con 0x7f05900d29c0 2024-03-06T07:28:09.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.072+0000 7f059842a640 1 -- 172.21.15.59:0/2896153159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059004f330 con 0x7f05900d29c0 2024-03-06T07:28:09.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.073+0000 7f0586ffd640 1 -- 172.21.15.59:0/2896153159 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f058001a070 con 0x7f05900d29c0 2024-03-06T07:28:09.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.073+0000 7f0586ffd640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057406b4d0 0x7f057406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.073+0000 7f0586ffd640 1 -- 172.21.15.59:0/2896153159 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0580089460 con 0x7f05900d29c0 2024-03-06T07:28:09.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.074+0000 7f05961c6640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057406b4d0 0x7f057406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.074+0000 7f05961c6640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057406b4d0 0x7f057406d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f058c036000 tx=0x7f058c03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.082+0000 7f0586ffd640 1 -- 172.21.15.59:0/2896153159 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0580057c90 con 0x7f05900d29c0 2024-03-06T07:28:09.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.300+0000 7f059842a640 1 -- 172.21.15.59:0/2896153159 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05900caa20 con 0x7f05900d29c0 2024-03-06T07:28:09.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.301+0000 7f0586ffd640 1 -- 172.21.15.59:0/2896153159 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f058005bd30 con 0x7f05900d29c0 2024-03-06T07:28:09.303 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:09.303 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:09.303 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.302+0000 7f059842a640 1 -- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057406b4d0 msgr2=0x7f057406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.302+0000 7f059842a640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057406b4d0 0x7f057406d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f058c036000 tx=0x7f058c03c040 comp rx=0 tx=0).stop 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.302+0000 7f059842a640 1 -- 172.21.15.59:0/2896153159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900d29c0 msgr2=0x7f05900cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.302+0000 7f059842a640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900d29c0 0x7f05900cada0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f058000c4d0 tx=0x7f058000c990 comp rx=0 tx=0).stop 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.302+0000 7f059842a640 1 -- 172.21.15.59:0/2896153159 shutdown_connections 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.302+0000 7f059842a640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f057406b4d0 0x7f057406d990 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.302+0000 7f059842a640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f059015dfe0 0x7f05900b8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.302+0000 7f059842a640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05900d29c0 0x7f05900cada0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.302+0000 7f059842a640 1 --2- 172.21.15.59:0/2896153159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05900cf830 0x7f05901f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.302+0000 7f059842a640 1 -- 172.21.15.59:0/2896153159 >> 172.21.15.59:0/2896153159 conn(0x7f05900d3cd0 msgr2=0x7f05900cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.303+0000 7f059842a640 1 -- 172.21.15.59:0/2896153159 shutdown_connections 2024-03-06T07:28:09.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.303+0000 7f059842a640 1 -- 172.21.15.59:0/2896153159 wait complete. 2024-03-06T07:28:09.316 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:28:09.444 INFO:teuthology.orchestra.run.smithi059.stdout:{"message":"","return_code":0} 2024-03-06T07:28:09.444 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:09.445 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:09.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.484+0000 7f754be44f40 1 Processor -- start 2024-03-06T07:28:09.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.484+0000 7f754be44f40 1 -- start start 2024-03-06T07:28:09.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.484+0000 7f754be44f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de7623d80 0x556de761fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.484+0000 7f754be44f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556de76adf30 0x556de7697cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.484+0000 7f754be44f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556de75d87d0 0x556de75d8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.484+0000 7f754be44f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556de757fb70 con 0x556de7623d80 2024-03-06T07:28:09.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.484+0000 7f754be44f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x556de7583ad0 con 0x556de75d87d0 2024-03-06T07:28:09.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.484+0000 7f754be44f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x556de7621700 con 0x556de76adf30 2024-03-06T07:28:09.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.484+0000 7f75490aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de7623d80 0x556de761fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75490aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de7623d80 0x556de761fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50070/0 (socket says 172.21.15.59:50070) 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75490aa640 1 -- 172.21.15.59:0/2008345174 learned_addr learned my addr 172.21.15.59:0/2008345174 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75498ab640 1 --2- 172.21.15.59:0/2008345174 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556de75d87d0 0x556de75d8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f7543fff640 1 --2- 172.21.15.59:0/2008345174 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556de76adf30 0x556de7697cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75490aa640 1 -- 172.21.15.59:0/2008345174 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556de76adf30 msgr2=0x556de7697cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75490aa640 1 --2- 172.21.15.59:0/2008345174 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556de76adf30 0x556de7697cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75490aa640 1 -- 172.21.15.59:0/2008345174 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556de75d87d0 msgr2=0x556de75d8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75490aa640 1 --2- 172.21.15.59:0/2008345174 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556de75d87d0 0x556de75d8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75490aa640 1 -- 172.21.15.59:0/2008345174 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556de76ad430 con 0x556de7623d80 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75490aa640 1 --2- 172.21.15.59:0/2008345174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de7623d80 0x556de761fdf0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f753c00a4f0 tx=0x7f753c051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c46f8eadde6905d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75437fe640 1 -- 172.21.15.59:0/2008345174 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c056040 con 0x556de7623d80 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75437fe640 1 -- 172.21.15.59:0/2008345174 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c052bc0 con 0x556de7623d80 2024-03-06T07:28:09.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.485+0000 7f75437fe640 1 -- 172.21.15.59:0/2008345174 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c05b420 con 0x556de7623d80 2024-03-06T07:28:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.486+0000 7f754be44f40 1 -- 172.21.15.59:0/2008345174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de7623d80 msgr2=0x556de761fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.486+0000 7f754be44f40 1 --2- 172.21.15.59:0/2008345174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de7623d80 0x556de761fdf0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f753c00a4f0 tx=0x7f753c051000 comp rx=0 tx=0).stop 2024-03-06T07:28:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.486+0000 7f754be44f40 1 -- 172.21.15.59:0/2008345174 shutdown_connections 2024-03-06T07:28:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.486+0000 7f754be44f40 1 --2- 172.21.15.59:0/2008345174 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556de75d87d0 0x556de75d8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.486+0000 7f754be44f40 1 --2- 172.21.15.59:0/2008345174 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556de76adf30 0x556de7697cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.486+0000 7f754be44f40 1 --2- 172.21.15.59:0/2008345174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de7623d80 0x556de761fdf0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.486+0000 7f754be44f40 1 -- 172.21.15.59:0/2008345174 >> 172.21.15.59:0/2008345174 conn(0x556de75d78b0 msgr2=0x556de76b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.486+0000 7f754be44f40 1 -- 172.21.15.59:0/2008345174 shutdown_connections 2024-03-06T07:28:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.486+0000 7f754be44f40 1 -- 172.21.15.59:0/2008345174 wait complete. 2024-03-06T07:28:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.486+0000 7f754be44f40 1 Processor -- start 2024-03-06T07:28:09.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.486+0000 7f754be44f40 1 -- start start 2024-03-06T07:28:09.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.487+0000 7f754be44f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556de75d87d0 0x556de7696b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.487+0000 7f754be44f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556de7623d80 0x556de7692e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.487+0000 7f754be44f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de76adf30 0x556de75d6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.487+0000 7f754be44f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x556de76b0690 con 0x556de76adf30 2024-03-06T07:28:09.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.487+0000 7f754be44f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x556de76b03f0 con 0x556de7623d80 2024-03-06T07:28:09.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.487+0000 7f754be44f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x556de7620330 con 0x556de75d87d0 2024-03-06T07:28:09.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.487+0000 7f75498ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de76adf30 0x556de75d6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.487+0000 7f75498ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de76adf30 0x556de75d6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50074/0 (socket says 172.21.15.59:50074) 2024-03-06T07:28:09.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.487+0000 7f75498ab640 1 -- 172.21.15.59:0/1600902119 learned_addr learned my addr 172.21.15.59:0/1600902119 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:09.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.487+0000 7f75490aa640 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556de75d87d0 0x556de7696b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.487+0000 7f7543fff640 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556de7623d80 0x556de7692e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.488+0000 7f75498ab640 1 -- 172.21.15.59:0/1600902119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556de75d87d0 msgr2=0x556de7696b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.488+0000 7f75498ab640 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x556de75d87d0 0x556de7696b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.488+0000 7f75498ab640 1 -- 172.21.15.59:0/1600902119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556de7623d80 msgr2=0x556de7692e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.488+0000 7f75498ab640 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556de7623d80 0x556de7692e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.488+0000 7f75498ab640 1 -- 172.21.15.59:0/1600902119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556de76933c0 con 0x556de76adf30 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.488+0000 7f75498ab640 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de76adf30 0x556de75d6b90 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f754400e7f0 tx=0x7f754400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.488+0000 7f75427fc640 1 -- 172.21.15.59:0/1600902119 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544016020 con 0x556de76adf30 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.488+0000 7f75427fc640 1 -- 172.21.15.59:0/1600902119 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7544001870 con 0x556de76adf30 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.488+0000 7f75427fc640 1 -- 172.21.15.59:0/1600902119 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754400c8d0 con 0x556de76adf30 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.488+0000 7f754be44f40 1 -- 172.21.15.59:0/1600902119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556de76ac680 con 0x556de76adf30 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.488+0000 7f754be44f40 1 -- 172.21.15.59:0/1600902119 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556de76aeee0 con 0x556de76adf30 2024-03-06T07:28:09.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.489+0000 7f75427fc640 1 -- 172.21.15.59:0/1600902119 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7544019020 con 0x556de76adf30 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.490+0000 7f75427fc640 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f752c069ee0 0x7f752c06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.490+0000 7f75427fc640 1 -- 172.21.15.59:0/1600902119 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7544088890 con 0x556de76adf30 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.490+0000 7f754be44f40 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x556de774fff0 0x556de77523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.490+0000 7f754be44f40 1 -- 172.21.15.59:0/1600902119 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x556de7752920 con 0x556de774fff0 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.490+0000 7f75490aa640 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f752c069ee0 0x7f752c06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.490+0000 7f7543fff640 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x556de774fff0 0x556de77523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.490+0000 7f7543fff640 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x556de774fff0 0x556de77523e0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f7543fff640 1 -- 172.21.15.59:0/1600902119 <== osd.0 v2:172.21.15.59:6824/841983189 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f753400e040 con 0x556de774fff0 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f75490aa640 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f752c069ee0 0x7f752c06c3a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f753c055040 tx=0x7f753c053000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f754be44f40 1 -- 172.21.15.59:0/1600902119 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x556de774fff0 msgr2=0x556de77523e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f754be44f40 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x556de774fff0 0x556de77523e0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f754be44f40 1 -- 172.21.15.59:0/1600902119 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f752c069ee0 msgr2=0x7f752c06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f754be44f40 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f752c069ee0 0x7f752c06c3a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f753c055040 tx=0x7f753c053000 comp rx=0 tx=0).stop 2024-03-06T07:28:09.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f754be44f40 1 -- 172.21.15.59:0/1600902119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de76adf30 msgr2=0x556de75d6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f754be44f40 1 --2- 172.21.15.59:0/1600902119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x556de76adf30 0x556de75d6b90 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f754400e7f0 tx=0x7f754400ecb0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f75498ab640 1 -- 172.21.15.59:0/1600902119 reap_dead start 2024-03-06T07:28:09.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f754be44f40 1 -- 172.21.15.59:0/1600902119 shutdown_connections 2024-03-06T07:28:09.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f754be44f40 1 -- 172.21.15.59:0/1600902119 >> 172.21.15.59:0/1600902119 conn(0x556de75d78b0 msgr2=0x556de76b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f754be44f40 1 -- 172.21.15.59:0/1600902119 shutdown_connections 2024-03-06T07:28:09.497 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.491+0000 7f754be44f40 1 -- 172.21.15.59:0/1600902119 wait complete. 2024-03-06T07:28:09.497 INFO:teuthology.orchestra.run.smithi059.stdout:W�file_cdir_2 2024-03-06T07:28:09.498 DEBUG:teuthology.orchestra.run.smithi059:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:09.596 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:28:09.596 INFO:teuthology.orchestra.run.smithi059.stdout: "ino": 1099511627777, 2024-03-06T07:28:09.596 INFO:teuthology.orchestra.run.smithi059.stdout: "ancestors": [ 2024-03-06T07:28:09.596 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:09.596 INFO:teuthology.orchestra.run.smithi059.stdout: "dirino": 1099511628278, 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: "dname": "file_c", 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: "version": 15 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: "dirino": 1, 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: "dname": "dir_2", 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: "version": 31 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: "pool": 17, 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: "old_pools": [] 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:28:09.597 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:09.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:09.600 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:09.659 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-03-06T07:28:09.718 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:09.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.858+0000 7ffbae437640 1 Processor -- start 2024-03-06T07:28:09.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.858+0000 7ffbae437640 1 -- start start 2024-03-06T07:28:09.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.859+0000 7ffbae437640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffba815dfe0 0x7ffba80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffbae437640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffba80cf830 0x7ffba80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffbae437640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba80b3430 con 0x7ffba80d27f0 2024-03-06T07:28:09.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffbae437640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba80b31e0 con 0x7ffba80cf830 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffbae437640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba80b3d60 con 0x7ffba815dfe0 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffba77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffba80cf830 0x7ffba80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffbac9d4640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ffba80d27f0 0x7ffba80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34786/0 (socket says 172.21.15.59:34786) 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffbac9d4640 1 -- 172.21.15.59:0/2588563086 learned_addr learned my addr 172.21.15.59:0/2588563086 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffba7fff640 1 --2- 172.21.15.59:0/2588563086 >> v2:172.21.15.164:3300/0 conn(0x7ffba815dfe0 0x7ffba80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffba77fe640 1 -- 172.21.15.59:0/2588563086 >> v2:172.21.15.164:3300/0 conn(0x7ffba815dfe0 msgr2=0x7ffba80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffba77fe640 1 --2- 172.21.15.59:0/2588563086 >> v2:172.21.15.164:3300/0 conn(0x7ffba815dfe0 0x7ffba80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffba77fe640 1 -- 172.21.15.59:0/2588563086 >> v1:172.21.15.59:6789/0 conn(0x7ffba80d27f0 legacy=0x7ffba80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffba77fe640 1 -- 172.21.15.59:0/2588563086 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba80b3710 con 0x7ffba80cf830 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffba6ffd640 1 -- 172.21.15.59:0/2588563086 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000431868 0 0) 0x7ffba80b3430 con 0x7ffba80d27f0 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.860+0000 7ffba7fff640 1 --2- 172.21.15.59:0/2588563086 >> v2:172.21.15.164:3300/0 conn(0x7ffba815dfe0 0x7ffba80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:09.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.861+0000 7ffba77fe640 1 --2- 172.21.15.59:0/2588563086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba80cf830 0x7ffba80d22b0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ffb900097a0 tx=0x7ffb9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5686e38c0d1e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.861+0000 7ffba6ffd640 1 -- 172.21.15.59:0/2588563086 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90035020 con 0x7ffba80cf830 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.861+0000 7ffba6ffd640 1 -- 172.21.15.59:0/2588563086 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb90031830 con 0x7ffba80cf830 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.861+0000 7ffba6ffd640 1 -- 172.21.15.59:0/2588563086 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90031c00 con 0x7ffba80cf830 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.861+0000 7ffbae437640 1 -- 172.21.15.59:0/2588563086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba80cf830 msgr2=0x7ffba80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.861+0000 7ffbae437640 1 --2- 172.21.15.59:0/2588563086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba80cf830 0x7ffba80d22b0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ffb900097a0 tx=0x7ffb9002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.861+0000 7ffbae437640 1 -- 172.21.15.59:0/2588563086 shutdown_connections 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.861+0000 7ffbae437640 1 --2- 172.21.15.59:0/2588563086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba80cf830 0x7ffba80d22b0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.861+0000 7ffbae437640 1 --2- 172.21.15.59:0/2588563086 >> v2:172.21.15.164:3300/0 conn(0x7ffba815dfe0 0x7ffba80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.862+0000 7ffbae437640 1 -- 172.21.15.59:0/2588563086 >> 172.21.15.59:0/2588563086 conn(0x7ffba80d3cd0 msgr2=0x7ffba80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.862+0000 7ffbae437640 1 -- 172.21.15.59:0/2588563086 shutdown_connections 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.862+0000 7ffbae437640 1 -- 172.21.15.59:0/2588563086 wait complete. 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.863+0000 7ffbae437640 1 Processor -- start 2024-03-06T07:28:09.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.864+0000 7ffbae437640 1 -- start start 2024-03-06T07:28:09.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffbae437640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba80cf830 0x7ffba815c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffbae437640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba80d27f0 0x7ffba80b8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffbae437640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffba815dfe0 0x7ffba80b8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffbae437640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba80d1df0 con 0x7ffba80d27f0 2024-03-06T07:28:09.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffbae437640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba80d1c70 con 0x7ffba815dfe0 2024-03-06T07:28:09.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffbae437640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba815e3c0 con 0x7ffba80cf830 2024-03-06T07:28:09.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffba77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba80d27f0 0x7ffba80b8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffba77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba80d27f0 0x7ffba80b8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50100/0 (socket says 172.21.15.59:50100) 2024-03-06T07:28:09.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffba77fe640 1 -- 172.21.15.59:0/3284607057 learned_addr learned my addr 172.21.15.59:0/3284607057 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:09.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffbac9d4640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffba815dfe0 0x7ffba80b8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffba77fe640 1 -- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba80cf830 msgr2=0x7ffba815c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffba77fe640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba80cf830 0x7ffba815c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffba77fe640 1 -- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffba815dfe0 msgr2=0x7ffba80b8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffba77fe640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffba815dfe0 0x7ffba80b8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffba77fe640 1 -- 172.21.15.59:0/3284607057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba815f730 con 0x7ffba80d27f0 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffbac9d4640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffba815dfe0 0x7ffba80b8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffba77fe640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba80d27f0 0x7ffba80b8230 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7ffb900098c0 tx=0x7ffb900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.865+0000 7ffba4ff9640 1 -- 172.21.15.59:0/3284607057 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90040070 con 0x7ffba80d27f0 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.866+0000 7ffba4ff9640 1 -- 172.21.15.59:0/3284607057 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9003e070 con 0x7ffba80d27f0 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.866+0000 7ffbae437640 1 -- 172.21.15.59:0/3284607057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba815fa10 con 0x7ffba80d27f0 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.866+0000 7ffba4ff9640 1 -- 172.21.15.59:0/3284607057 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90035040 con 0x7ffba80d27f0 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.866+0000 7ffbae437640 1 -- 172.21.15.59:0/3284607057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba815ffd0 con 0x7ffba80d27f0 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.867+0000 7ffbae437640 1 -- 172.21.15.59:0/3284607057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba804f330 con 0x7ffba80d27f0 2024-03-06T07:28:09.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.867+0000 7ffba4ff9640 1 -- 172.21.15.59:0/3284607057 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb90038020 con 0x7ffba80d27f0 2024-03-06T07:28:09.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.867+0000 7ffba4ff9640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8806b4d0 0x7ffb8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.868+0000 7ffba4ff9640 1 -- 172.21.15.59:0/3284607057 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ffb900ab900 con 0x7ffba80d27f0 2024-03-06T07:28:09.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.868+0000 7ffba7fff640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8806b4d0 0x7ffb8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.868+0000 7ffba7fff640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8806b4d0 0x7ffb8806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ffb98006fd0 tx=0x7ffb98008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:09.876+0000 7ffba4ff9640 1 -- 172.21.15.59:0/3284607057 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb9007a700 con 0x7ffba80d27f0 2024-03-06T07:28:10.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.096+0000 7ffbae437640 1 -- 172.21.15.59:0/3284607057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffba815d240 con 0x7ffba80d27f0 2024-03-06T07:28:10.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.097+0000 7ffba4ff9640 1 -- 172.21.15.59:0/3284607057 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7ffb9007a000 con 0x7ffba80d27f0 2024-03-06T07:28:10.099 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:10.099 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:10.099 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:10.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 -- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8806b4d0 msgr2=0x7ffb8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8806b4d0 0x7ffb8806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ffb98006fd0 tx=0x7ffb98008040 comp rx=0 tx=0).stop 2024-03-06T07:28:10.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 -- 172.21.15.59:0/3284607057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba80d27f0 msgr2=0x7ffba80b8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba80d27f0 0x7ffba80b8230 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7ffb900098c0 tx=0x7ffb900046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 -- 172.21.15.59:0/3284607057 shutdown_connections 2024-03-06T07:28:10.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8806b4d0 0x7ffb8806d990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffba815dfe0 0x7ffba80b8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba80d27f0 0x7ffba80b8230 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 --2- 172.21.15.59:0/3284607057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba80cf830 0x7ffba815c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 -- 172.21.15.59:0/3284607057 >> 172.21.15.59:0/3284607057 conn(0x7ffba80d3cd0 msgr2=0x7ffba806ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 -- 172.21.15.59:0/3284607057 shutdown_connections 2024-03-06T07:28:10.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:10.099+0000 7ffbae437640 1 -- 172.21.15.59:0/3284607057 wait complete. 2024-03-06T07:28:10.112 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:10.231 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:28:10.231 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:28:10.232 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-03-06T07:28:15.233 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:15.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.333+0000 7fc6c373a640 1 Processor -- start 2024-03-06T07:28:15.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.333+0000 7fc6c373a640 1 -- start start 2024-03-06T07:28:15.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c373a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc6bc15ab00 0x7fc6bc0b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c373a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc0b48f0 0x7fc6bc0af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c373a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc0b4e30 con 0x7fc6bc0afc00 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c373a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6bc0b5110 con 0x7fc6bc15ab00 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c373a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc15b310 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c0cd5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc0b48f0 0x7fc6bc0af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c14d6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc6bc15ab00 0x7fc6bc0b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c1cd7640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc6bc0afc00 0x7fc6bc06e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:41752/0 (socket says 172.21.15.59:41752) 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c1cd7640 1 -- 172.21.15.59:0/3165853244 learned_addr learned my addr 172.21.15.59:0/3165853244 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6b3fff640 1 -- 172.21.15.59:0/3165853244 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151761173 0 0) 0x7fc6bc0b4e30 con 0x7fc6bc0afc00 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6b3fff640 1 -- 172.21.15.59:0/3165853244 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a8003520 con 0x7fc6bc0afc00 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c0cd5640 1 -- 172.21.15.59:0/3165853244 >> v2:172.21.15.164:3300/0 conn(0x7fc6bc15ab00 msgr2=0x7fc6bc0b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c0cd5640 1 --2- 172.21.15.59:0/3165853244 >> v2:172.21.15.164:3300/0 conn(0x7fc6bc15ab00 0x7fc6bc0b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c0cd5640 1 -- 172.21.15.59:0/3165853244 >> v1:172.21.15.164:6789/0 conn(0x7fc6bc0afc00 legacy=0x7fc6bc06e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c0cd5640 1 -- 172.21.15.59:0/3165853244 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc15b5f0 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.334+0000 7fc6c14d6640 1 --2- 172.21.15.59:0/3165853244 >> v2:172.21.15.164:3300/0 conn(0x7fc6bc15ab00 0x7fc6bc0b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:15.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.335+0000 7fc6c0cd5640 1 --2- 172.21.15.59:0/3165853244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc0b48f0 0x7fc6bc0af6c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0097a0 tx=0x7fc6ac02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b812bb214b2f0630 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.335+0000 7fc6b3fff640 1 -- 172.21.15.59:0/3165853244 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac035020 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.335+0000 7fc6b3fff640 1 -- 172.21.15.59:0/3165853244 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ac031830 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.335+0000 7fc6b3fff640 1 -- 172.21.15.59:0/3165853244 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac031be0 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.336+0000 7fc6c373a640 1 -- 172.21.15.59:0/3165853244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc0b48f0 msgr2=0x7fc6bc0af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.336+0000 7fc6c373a640 1 --2- 172.21.15.59:0/3165853244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc0b48f0 0x7fc6bc0af6c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0097a0 tx=0x7fc6ac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.336+0000 7fc6c373a640 1 -- 172.21.15.59:0/3165853244 shutdown_connections 2024-03-06T07:28:15.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.336+0000 7fc6c373a640 1 --2- 172.21.15.59:0/3165853244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc0b48f0 0x7fc6bc0af6c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.336+0000 7fc6c373a640 1 --2- 172.21.15.59:0/3165853244 >> v2:172.21.15.164:3300/0 conn(0x7fc6bc15ab00 0x7fc6bc0b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.336+0000 7fc6c373a640 1 -- 172.21.15.59:0/3165853244 >> 172.21.15.59:0/3165853244 conn(0x7fc6bc06c0c0 msgr2=0x7fc6bc06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.336+0000 7fc6c373a640 1 -- 172.21.15.59:0/3165853244 shutdown_connections 2024-03-06T07:28:15.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.336+0000 7fc6c373a640 1 -- 172.21.15.59:0/3165853244 wait complete. 2024-03-06T07:28:15.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.337+0000 7fc6c373a640 1 Processor -- start 2024-03-06T07:28:15.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.337+0000 7fc6c373a640 1 -- start start 2024-03-06T07:28:15.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c373a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6bc0afc00 0x7fc6bc143e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c373a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6bc0b48f0 0x7fc6bc0d0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c373a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc15ab00 0x7fc6bc0b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c373a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc15b060 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c373a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc15aee0 con 0x7fc6bc0afc00 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c373a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc15cb20 con 0x7fc6bc15ab00 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c0cd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6bc0b48f0 0x7fc6bc0d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c0cd5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6bc0b48f0 0x7fc6bc0d0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33444/0 (socket says 172.21.15.59:33444) 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c0cd5640 1 -- 172.21.15.59:0/3277175129 learned_addr learned my addr 172.21.15.59:0/3277175129 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c14d6640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6bc0afc00 0x7fc6bc143e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c1cd7640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc15ab00 0x7fc6bc0b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c0cd5640 1 -- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc15ab00 msgr2=0x7fc6bc0b3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c0cd5640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc15ab00 0x7fc6bc0b3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c0cd5640 1 -- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6bc0afc00 msgr2=0x7fc6bc143e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c0cd5640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6bc0afc00 0x7fc6bc143e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c0cd5640 1 -- 172.21.15.59:0/3277175129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc0cf120 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c1cd7640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc15ab00 0x7fc6bc0b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.338+0000 7fc6c14d6640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6bc0afc00 0x7fc6bc143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.339+0000 7fc6c0cd5640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6bc0b48f0 0x7fc6bc0d0580 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0098c0 tx=0x7fc6ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.339+0000 7fc6b1ffb640 1 -- 172.21.15.59:0/3277175129 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac035040 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.339+0000 7fc6b1ffb640 1 -- 172.21.15.59:0/3277175129 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ac03b900 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.339+0000 7fc6b1ffb640 1 -- 172.21.15.59:0/3277175129 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac03bcf0 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.339+0000 7fc6c373a640 1 -- 172.21.15.59:0/3277175129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6bc0ce880 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.339+0000 7fc6c373a640 1 -- 172.21.15.59:0/3277175129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6bc0cee40 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.340+0000 7fc6c373a640 1 -- 172.21.15.59:0/3277175129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6bc04f330 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.340+0000 7fc6b1ffb640 1 -- 172.21.15.59:0/3277175129 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6ac038020 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.341+0000 7fc6b1ffb640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc6a006b4d0 0x7fc6a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.341+0000 7fc6b1ffb640 1 -- 172.21.15.59:0/3277175129 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc6ac0ac2e0 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.341+0000 7fc6c14d6640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc6a006b4d0 0x7fc6a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.342+0000 7fc6c14d6640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc6a006b4d0 0x7fc6a006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc14bc20 tx=0x7fc6b8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.345+0000 7fc6b1ffb640 1 -- 172.21.15.59:0/3277175129 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6bc0cf120 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.668+0000 7fc6c373a640 1 -- 172.21.15.59:0/3277175129 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6bc152130 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.669+0000 7fc6b1ffb640 1 -- 172.21.15.59:0/3277175129 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fc6ac07b0d0 con 0x7fc6bc0b48f0 2024-03-06T07:28:15.671 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:15.671 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:15.671 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:15.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.670+0000 7fc6c373a640 1 -- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc6a006b4d0 msgr2=0x7fc6a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.670+0000 7fc6c373a640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc6a006b4d0 0x7fc6a006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc14bc20 tx=0x7fc6b8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.670+0000 7fc6c373a640 1 -- 172.21.15.59:0/3277175129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6bc0b48f0 msgr2=0x7fc6bc0d0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.670+0000 7fc6c373a640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6bc0b48f0 0x7fc6bc0d0580 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0098c0 tx=0x7fc6ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.670+0000 7fc6c373a640 1 -- 172.21.15.59:0/3277175129 shutdown_connections 2024-03-06T07:28:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.670+0000 7fc6c373a640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc6a006b4d0 0x7fc6a006d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.670+0000 7fc6c373a640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6bc15ab00 0x7fc6bc0b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.670+0000 7fc6c373a640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6bc0b48f0 0x7fc6bc0d0580 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.670+0000 7fc6c373a640 1 --2- 172.21.15.59:0/3277175129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6bc0afc00 0x7fc6bc143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.671+0000 7fc6c373a640 1 -- 172.21.15.59:0/3277175129 >> 172.21.15.59:0/3277175129 conn(0x7fc6bc06c0c0 msgr2=0x7fc6bc159100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.671+0000 7fc6c373a640 1 -- 172.21.15.59:0/3277175129 shutdown_connections 2024-03-06T07:28:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.671+0000 7fc6c373a640 1 -- 172.21.15.59:0/3277175129 wait complete. 2024-03-06T07:28:15.683 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:15.801 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:28:15.802 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:28:15.802 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:15.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.902+0000 7f802f1ac640 1 Processor -- start 2024-03-06T07:28:15.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.902+0000 7f802f1ac640 1 -- start start 2024-03-06T07:28:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.903+0000 7f802f1ac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f80280d22b0 0x7f80280d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.903+0000 7f802f1ac640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f80280b41f0 0x7f80280b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.903+0000 7f802f1ac640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80280b3430 con 0x7f802815dfe0 2024-03-06T07:28:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.903+0000 7f802f1ac640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80280d2c50 con 0x7f80280d22b0 2024-03-06T07:28:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.903+0000 7f802f1ac640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80280b31e0 con 0x7f80280b41f0 2024-03-06T07:28:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.903+0000 7f8027fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f80280d22b0 0x7f80280d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.903+0000 7f8027fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f80280d22b0 0x7f80280d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33456/0 (socket says 172.21.15.59:33456) 2024-03-06T07:28:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.903+0000 7f8027fff640 1 -- 172.21.15.59:0/4212932920 learned_addr learned my addr 172.21.15.59:0/4212932920 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:15.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.904+0000 7f802d749640 1 --2- 172.21.15.59:0/4212932920 >> v2:172.21.15.164:3300/0 conn(0x7f80280b41f0 0x7f80280b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.904+0000 7f8027fff640 1 -- 172.21.15.59:0/4212932920 >> v2:172.21.15.164:3300/0 conn(0x7f80280b41f0 msgr2=0x7f80280b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.904+0000 7f8027fff640 1 --2- 172.21.15.59:0/4212932920 >> v2:172.21.15.164:3300/0 conn(0x7f80280b41f0 0x7f80280b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.904+0000 7f8027fff640 1 -- 172.21.15.59:0/4212932920 >> v1:172.21.15.164:6789/0 conn(0x7f802815dfe0 legacy=0x7f80280cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.904+0000 7f8027fff640 1 -- 172.21.15.59:0/4212932920 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80280b3710 con 0x7f80280d22b0 2024-03-06T07:28:15.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.904+0000 7f802d749640 1 --2- 172.21.15.59:0/4212932920 >> v2:172.21.15.164:3300/0 conn(0x7f80280b41f0 0x7f80280b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:15.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f8027fff640 1 --2- 172.21.15.59:0/4212932920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80280d22b0 0x7f80280d2710 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f8014009870 tx=0x7f801402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702b333d87b70394 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f80277fe640 1 -- 172.21.15.59:0/4212932920 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014034070 con 0x7f80280d22b0 2024-03-06T07:28:15.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f80277fe640 1 -- 172.21.15.59:0/4212932920 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8014030860 con 0x7f80280d22b0 2024-03-06T07:28:15.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f80277fe640 1 -- 172.21.15.59:0/4212932920 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014030c30 con 0x7f80280d22b0 2024-03-06T07:28:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f802f1ac640 1 -- 172.21.15.59:0/4212932920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80280d22b0 msgr2=0x7f80280d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f802f1ac640 1 --2- 172.21.15.59:0/4212932920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80280d22b0 0x7f80280d2710 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f8014009870 tx=0x7f801402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f802f1ac640 1 -- 172.21.15.59:0/4212932920 shutdown_connections 2024-03-06T07:28:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f802f1ac640 1 --2- 172.21.15.59:0/4212932920 >> v2:172.21.15.164:3300/0 conn(0x7f80280b41f0 0x7f80280b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f802f1ac640 1 --2- 172.21.15.59:0/4212932920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80280d22b0 0x7f80280d2710 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f802f1ac640 1 -- 172.21.15.59:0/4212932920 >> 172.21.15.59:0/4212932920 conn(0x7f80280d3cd0 msgr2=0x7f80280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f802f1ac640 1 -- 172.21.15.59:0/4212932920 shutdown_connections 2024-03-06T07:28:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.905+0000 7f802f1ac640 1 -- 172.21.15.59:0/4212932920 wait complete. 2024-03-06T07:28:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.907+0000 7f802f1ac640 1 Processor -- start 2024-03-06T07:28:15.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.907+0000 7f802f1ac640 1 -- start start 2024-03-06T07:28:15.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.907+0000 7f802f1ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80280b41f0 0x7f802815aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.907+0000 7f802f1ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80280d22b0 0x7f8028157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.907+0000 7f802f1ac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f802815dfe0 0x7f80280b9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.907+0000 7f802f1ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80280cfa00 con 0x7f80280d22b0 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.907+0000 7f802f1ac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80280b3d60 con 0x7f802815dfe0 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.907+0000 7f802f1ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80280d1c70 con 0x7f80280b41f0 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f8027fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80280d22b0 0x7f8028157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f8027fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80280d22b0 0x7f8028157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33472/0 (socket says 172.21.15.59:33472) 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f8027fff640 1 -- 172.21.15.59:0/2342912394 learned_addr learned my addr 172.21.15.59:0/2342912394 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f802d749640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f802815dfe0 0x7f80280b9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f802cf48640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80280b41f0 0x7f802815aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f8027fff640 1 -- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80280b41f0 msgr2=0x7f802815aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f8027fff640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80280b41f0 0x7f802815aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f8027fff640 1 -- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f802815dfe0 msgr2=0x7f80280b9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f8027fff640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f802815dfe0 0x7f80280b9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f8027fff640 1 -- 172.21.15.59:0/2342912394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8028157f90 con 0x7f80280d22b0 2024-03-06T07:28:15.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f8027fff640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80280d22b0 0x7f8028157a50 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f801402b9e0 tx=0x7f8014004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f80257fa640 1 -- 172.21.15.59:0/2342912394 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014048070 con 0x7f80280d22b0 2024-03-06T07:28:15.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f80257fa640 1 -- 172.21.15.59:0/2342912394 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8014030660 con 0x7f80280d22b0 2024-03-06T07:28:15.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f802f1ac640 1 -- 172.21.15.59:0/2342912394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802815ffc0 con 0x7f80280d22b0 2024-03-06T07:28:15.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f80257fa640 1 -- 172.21.15.59:0/2342912394 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014034040 con 0x7f80280d22b0 2024-03-06T07:28:15.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.908+0000 7f802f1ac640 1 -- 172.21.15.59:0/2342912394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8028160580 con 0x7f80280d22b0 2024-03-06T07:28:15.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.909+0000 7f802f1ac640 1 -- 172.21.15.59:0/2342912394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802804f330 con 0x7f80280d22b0 2024-03-06T07:28:15.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.910+0000 7f80257fa640 1 -- 172.21.15.59:0/2342912394 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f801404b070 con 0x7f80280d22b0 2024-03-06T07:28:15.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.910+0000 7f80257fa640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f800c06b4d0 0x7f800c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.910+0000 7f80257fa640 1 -- 172.21.15.59:0/2342912394 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f80140ac260 con 0x7f80280d22b0 2024-03-06T07:28:15.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.910+0000 7f802cf48640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f800c06b4d0 0x7f800c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.911+0000 7f802cf48640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f800c06b4d0 0x7f800c06d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f801c000f60 tx=0x7f801c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:15.919+0000 7f80257fa640 1 -- 172.21.15.59:0/2342912394 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8028157f90 con 0x7f80280d22b0 2024-03-06T07:28:16.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.140+0000 7f802f1ac640 1 -- 172.21.15.59:0/2342912394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f802815d690 con 0x7f80280d22b0 2024-03-06T07:28:16.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.141+0000 7f80257fa640 1 -- 172.21.15.59:0/2342912394 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f801407b170 con 0x7f80280d22b0 2024-03-06T07:28:16.143 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:16.145 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:16.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 -- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f800c06b4d0 msgr2=0x7f800c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f800c06b4d0 0x7f800c06d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f801c000f60 tx=0x7f801c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:16.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 -- 172.21.15.59:0/2342912394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80280d22b0 msgr2=0x7f8028157a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80280d22b0 0x7f8028157a50 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f801402b9e0 tx=0x7f8014004730 comp rx=0 tx=0).stop 2024-03-06T07:28:16.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 -- 172.21.15.59:0/2342912394 shutdown_connections 2024-03-06T07:28:16.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f800c06b4d0 0x7f800c06d990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f802815dfe0 0x7f80280b9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80280d22b0 0x7f8028157a50 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 --2- 172.21.15.59:0/2342912394 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80280b41f0 0x7f802815aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 -- 172.21.15.59:0/2342912394 >> 172.21.15.59:0/2342912394 conn(0x7f80280d3cd0 msgr2=0x7f802806ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 -- 172.21.15.59:0/2342912394 shutdown_connections 2024-03-06T07:28:16.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.143+0000 7f802f1ac640 1 -- 172.21.15.59:0/2342912394 wait complete. 2024-03-06T07:28:16.159 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:16.278 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-03-06T07:28:16.279 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-03-06T07:28:16.279 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:16.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.382+0000 7f5f9b62a640 1 Processor -- start 2024-03-06T07:28:16.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.382+0000 7f5f9b62a640 1 -- start start 2024-03-06T07:28:16.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f9b62a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f9415dfe0 0x7f5f940cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f9b62a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5f940cf830 0x7f5f940d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f9b62a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f940b3430 con 0x7f5f940d27f0 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f9b62a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f940d2c50 con 0x7f5f940cf830 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f9b62a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f940b31e0 con 0x7f5f9415dfe0 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f993c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f9415dfe0 0x7f5f940cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f98bc5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5f940cf830 0x7f5f940d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f993c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f9415dfe0 0x7f5f940cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:47624/0 (socket says 172.21.15.59:47624) 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f993c6640 1 -- 172.21.15.59:0/4143412858 learned_addr learned my addr 172.21.15.59:0/4143412858 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f993c6640 1 -- 172.21.15.59:0/4143412858 >> v2:172.21.15.164:3300/0 conn(0x7f5f940cf830 msgr2=0x7f5f940d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f993c6640 1 --2- 172.21.15.59:0/4143412858 >> v2:172.21.15.164:3300/0 conn(0x7f5f940cf830 0x7f5f940d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f993c6640 1 -- 172.21.15.59:0/4143412858 >> v1:172.21.15.164:6789/0 conn(0x7f5f940d27f0 legacy=0x7f5f940d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.384+0000 7f5f993c6640 1 -- 172.21.15.59:0/4143412858 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f940b3710 con 0x7f5f9415dfe0 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.383+0000 7f5f8bfff640 1 -- 172.21.15.59:0/4143412858 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077042443 0 0) 0x7f5f940b3430 con 0x7f5f940d27f0 2024-03-06T07:28:16.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.384+0000 7f5f98bc5640 1 --2- 172.21.15.59:0/4143412858 >> v2:172.21.15.164:3300/0 conn(0x7f5f940cf830 0x7f5f940d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:16.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.384+0000 7f5f993c6640 1 --2- 172.21.15.59:0/4143412858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f9415dfe0 0x7f5f940cf2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009870 tx=0x7f5f9002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5c2b1d02b47df67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.385+0000 7f5f8bfff640 1 -- 172.21.15.59:0/4143412858 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90034070 con 0x7f5f9415dfe0 2024-03-06T07:28:16.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.385+0000 7f5f8bfff640 1 -- 172.21.15.59:0/4143412858 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f90030840 con 0x7f5f9415dfe0 2024-03-06T07:28:16.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.385+0000 7f5f8bfff640 1 -- 172.21.15.59:0/4143412858 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90030c10 con 0x7f5f9415dfe0 2024-03-06T07:28:16.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.385+0000 7f5f9b62a640 1 -- 172.21.15.59:0/4143412858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f9415dfe0 msgr2=0x7f5f940cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.385+0000 7f5f9b62a640 1 --2- 172.21.15.59:0/4143412858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f9415dfe0 0x7f5f940cf2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009870 tx=0x7f5f9002b980 comp rx=0 tx=0).stop 2024-03-06T07:28:16.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.385+0000 7f5f9b62a640 1 -- 172.21.15.59:0/4143412858 shutdown_connections 2024-03-06T07:28:16.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.385+0000 7f5f9b62a640 1 --2- 172.21.15.59:0/4143412858 >> v2:172.21.15.164:3300/0 conn(0x7f5f940cf830 0x7f5f940d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.385+0000 7f5f9b62a640 1 --2- 172.21.15.59:0/4143412858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f9415dfe0 0x7f5f940cf2f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.385+0000 7f5f9b62a640 1 -- 172.21.15.59:0/4143412858 >> 172.21.15.59:0/4143412858 conn(0x7f5f940d3cd0 msgr2=0x7f5f940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.385+0000 7f5f9b62a640 1 -- 172.21.15.59:0/4143412858 shutdown_connections 2024-03-06T07:28:16.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.385+0000 7f5f9b62a640 1 -- 172.21.15.59:0/4143412858 wait complete. 2024-03-06T07:28:16.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.386+0000 7f5f9b62a640 1 Processor -- start 2024-03-06T07:28:16.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.386+0000 7f5f9b62a640 1 -- start start 2024-03-06T07:28:16.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.387+0000 7f5f9b62a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f940cf830 0x7f5f94160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.387+0000 7f5f9b62a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f940d27f0 0x7f5f940b9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.387+0000 7f5f9b62a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f9415dfe0 0x7f5f9414dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.387+0000 7f5f9b62a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f940d1c70 con 0x7f5f940cf830 2024-03-06T07:28:16.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.387+0000 7f5f9b62a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f940b3d60 con 0x7f5f9415dfe0 2024-03-06T07:28:16.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.387+0000 7f5f9b62a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f940d1df0 con 0x7f5f940d27f0 2024-03-06T07:28:16.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.387+0000 7f5f993c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f940cf830 0x7f5f94160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.387+0000 7f5f993c6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f940cf830 0x7f5f94160470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33478/0 (socket says 172.21.15.59:33478) 2024-03-06T07:28:16.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.387+0000 7f5f993c6640 1 -- 172.21.15.59:0/1001690103 learned_addr learned my addr 172.21.15.59:0/1001690103 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:16.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.387+0000 7f5f98bc5640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f940d27f0 0x7f5f940b9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.387+0000 7f5f99bc7640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f9415dfe0 0x7f5f9414dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f993c6640 1 -- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f940d27f0 msgr2=0x7f5f940b9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f993c6640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f940d27f0 0x7f5f940b9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f993c6640 1 -- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f9415dfe0 msgr2=0x7f5f9414dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f993c6640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f9415dfe0 0x7f5f9414dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f993c6640 1 -- 172.21.15.59:0/1001690103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9415f3d0 con 0x7f5f940cf830 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f99bc7640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f9415dfe0 0x7f5f9414dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f993c6640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f940cf830 0x7f5f94160470 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009870 tx=0x7f5f900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f89ffb640 1 -- 172.21.15.59:0/1001690103 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90034040 con 0x7f5f940cf830 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f89ffb640 1 -- 172.21.15.59:0/1001690103 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f90039d90 con 0x7f5f940cf830 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f9b62a640 1 -- 172.21.15.59:0/1001690103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9415f6b0 con 0x7f5f940cf830 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f89ffb640 1 -- 172.21.15.59:0/1001690103 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90043630 con 0x7f5f940cf830 2024-03-06T07:28:16.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.388+0000 7f5f9b62a640 1 -- 172.21.15.59:0/1001690103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9415fc70 con 0x7f5f940cf830 2024-03-06T07:28:16.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.389+0000 7f5f9b62a640 1 -- 172.21.15.59:0/1001690103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f940cf5f0 con 0x7f5f940cf830 2024-03-06T07:28:16.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.390+0000 7f5f89ffb640 1 -- 172.21.15.59:0/1001690103 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f9004a060 con 0x7f5f940cf830 2024-03-06T07:28:16.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.390+0000 7f5f89ffb640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f7806b4d0 0x7f5f7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.391+0000 7f5f89ffb640 1 -- 172.21.15.59:0/1001690103 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5f900ac140 con 0x7f5f940cf830 2024-03-06T07:28:16.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.391+0000 7f5f98bc5640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f7806b4d0 0x7f5f7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.391+0000 7f5f98bc5640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f7806b4d0 0x7f5f7806d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009560 tx=0x7f5f84009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.395+0000 7f5f89ffb640 1 -- 172.21.15.59:0/1001690103 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f940cf5f0 con 0x7f5f940cf830 2024-03-06T07:28:16.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.616+0000 7f5f9b62a640 1 -- 172.21.15.59:0/1001690103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f940cdfe0 con 0x7f5f940cf830 2024-03-06T07:28:16.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.617+0000 7f5f89ffb640 1 -- 172.21.15.59:0/1001690103 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f5f9003e070 con 0x7f5f940cf830 2024-03-06T07:28:16.619 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:16.619 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:16.619 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:16.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 -- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f7806b4d0 msgr2=0x7f5f7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f7806b4d0 0x7f5f7806d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009560 tx=0x7f5f84009190 comp rx=0 tx=0).stop 2024-03-06T07:28:16.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 -- 172.21.15.59:0/1001690103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f940cf830 msgr2=0x7f5f94160470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f940cf830 0x7f5f94160470 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009870 tx=0x7f5f900046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 -- 172.21.15.59:0/1001690103 shutdown_connections 2024-03-06T07:28:16.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f7806b4d0 0x7f5f7806d990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f9415dfe0 0x7f5f9414dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f940d27f0 0x7f5f940b9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 --2- 172.21.15.59:0/1001690103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f940cf830 0x7f5f94160470 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 -- 172.21.15.59:0/1001690103 >> 172.21.15.59:0/1001690103 conn(0x7f5f940d3cd0 msgr2=0x7f5f940cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 -- 172.21.15.59:0/1001690103 shutdown_connections 2024-03-06T07:28:16.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.619+0000 7f5f9b62a640 1 -- 172.21.15.59:0/1001690103 wait complete. 2024-03-06T07:28:16.631 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:28:16.765 INFO:teuthology.orchestra.run.smithi059.stdout:{"message":"","return_code":0} 2024-03-06T07:28:16.765 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:16.765 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.805+0000 7f3825a37f40 1 Processor -- start 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.805+0000 7f3825a37f40 1 -- start start 2024-03-06T07:28:16.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3825a37f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100b81f0 0x55991014d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.809 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3825a37f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5599100b85d0 0x5599101de830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3825a37f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5599100bc650 0x5599101ded70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3825a37f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599100bb450 con 0x5599100b81f0 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3825a37f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55991001bb70 con 0x5599100bc650 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3825a37f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5599100bccf0 con 0x5599100b85d0 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3822c48640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100b81f0 0x55991014d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3822c48640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100b81f0 0x55991014d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33498/0 (socket says 172.21.15.59:33498) 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3822c48640 1 -- 172.21.15.59:0/2263073545 learned_addr learned my addr 172.21.15.59:0/2263073545 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3823449640 1 --2- 172.21.15.59:0/2263073545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5599100bc650 0x5599101ded70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3822447640 1 --2- 172.21.15.59:0/2263073545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5599100b85d0 0x5599101de830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3822c48640 1 -- 172.21.15.59:0/2263073545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5599100b85d0 msgr2=0x5599101de830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3822c48640 1 --2- 172.21.15.59:0/2263073545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5599100b85d0 0x5599101de830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3822c48640 1 -- 172.21.15.59:0/2263073545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5599100bc650 msgr2=0x5599101ded70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3822c48640 1 --2- 172.21.15.59:0/2263073545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5599100bc650 0x5599101ded70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3822c48640 1 -- 172.21.15.59:0/2263073545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55991014cab0 con 0x5599100b81f0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3823449640 1 --2- 172.21.15.59:0/2263073545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5599100bc650 0x5599101ded70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3822447640 1 --2- 172.21.15.59:0/2263073545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5599100b85d0 0x5599101de830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3822c48640 1 --2- 172.21.15.59:0/2263073545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100b81f0 0x55991014d720 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f381400b2e0 tx=0x7f381400b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ef52c79a8d5da3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3821445640 1 -- 172.21.15.59:0/2263073545 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3814008070 con 0x5599100b81f0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3821445640 1 -- 172.21.15.59:0/2263073545 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381400d500 con 0x5599100b81f0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.806+0000 7f3821445640 1 -- 172.21.15.59:0/2263073545 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381400d8f0 con 0x5599100b81f0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.807+0000 7f3825a37f40 1 -- 172.21.15.59:0/2263073545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100b81f0 msgr2=0x55991014d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.807+0000 7f3825a37f40 1 --2- 172.21.15.59:0/2263073545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100b81f0 0x55991014d720 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f381400b2e0 tx=0x7f381400b7a0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.807+0000 7f3825a37f40 1 -- 172.21.15.59:0/2263073545 shutdown_connections 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.807+0000 7f3825a37f40 1 --2- 172.21.15.59:0/2263073545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5599100bc650 0x5599101ded70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.807+0000 7f3825a37f40 1 --2- 172.21.15.59:0/2263073545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5599100b85d0 0x5599101de830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.807+0000 7f3825a37f40 1 --2- 172.21.15.59:0/2263073545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100b81f0 0x55991014d720 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.807+0000 7f3825a37f40 1 -- 172.21.15.59:0/2263073545 >> 172.21.15.59:0/2263073545 conn(0x5599100a0350 msgr2=0x5599100be2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.807+0000 7f3825a37f40 1 -- 172.21.15.59:0/2263073545 shutdown_connections 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.807+0000 7f3825a37f40 1 -- 172.21.15.59:0/2263073545 wait complete. 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.807+0000 7f3825a37f40 1 Processor -- start 2024-03-06T07:28:16.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.808+0000 7f3825a37f40 1 -- start start 2024-03-06T07:28:16.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.808+0000 7f3825a37f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5599100b81f0 0x55991014bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.808+0000 7f3825a37f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5599100b85d0 0x559910075c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.808+0000 7f3825a37f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100bc650 0x559910076150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.808+0000 7f3825a37f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55991014c810 con 0x5599100bc650 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.808+0000 7f3825a37f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599100ba480 con 0x5599100b85d0 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.808+0000 7f3825a37f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x559910133360 con 0x5599100b81f0 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.808+0000 7f3823449640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100bc650 0x559910076150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.808+0000 7f3823449640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100bc650 0x559910076150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33508/0 (socket says 172.21.15.59:33508) 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.808+0000 7f3823449640 1 -- 172.21.15.59:0/1212640971 learned_addr learned my addr 172.21.15.59:0/1212640971 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f3822c48640 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5599100b81f0 0x55991014bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f3822447640 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5599100b85d0 0x559910075c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f3823449640 1 -- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5599100b81f0 msgr2=0x55991014bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f3823449640 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5599100b81f0 0x55991014bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f3823449640 1 -- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5599100b85d0 msgr2=0x559910075c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f3823449640 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5599100b85d0 0x559910075c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f3823449640 1 -- 172.21.15.59:0/1212640971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559910074260 con 0x5599100bc650 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f3822447640 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5599100b85d0 0x559910075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f3823449640 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100bc650 0x559910076150 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f381c00a4d0 tx=0x7f381c00a990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f380bfff640 1 -- 172.21.15.59:0/1212640971 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c0107f0 con 0x5599100bc650 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f380bfff640 1 -- 172.21.15.59:0/1212640971 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381c016070 con 0x5599100bc650 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f3825a37f40 1 -- 172.21.15.59:0/1212640971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559910128950 con 0x5599100bc650 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f380bfff640 1 -- 172.21.15.59:0/1212640971 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c010bf0 con 0x5599100bc650 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.809+0000 7f3825a37f40 1 -- 172.21.15.59:0/1212640971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55991014c0a0 con 0x5599100bc650 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.810+0000 7f380bfff640 1 -- 172.21.15.59:0/1212640971 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f381c011480 con 0x5599100bc650 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.811+0000 7f380bfff640 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3810069ee0 0x7f381006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.811+0000 7f380bfff640 1 -- 172.21.15.59:0/1212640971 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f381c0895b0 con 0x5599100bc650 2024-03-06T07:28:16.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.811+0000 7f3825a37f40 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x559910129430 0x559910129810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.811+0000 7f3825a37f40 1 -- 172.21.15.59:0/1212640971 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x559910129d50 con 0x559910129430 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.811+0000 7f3822c48640 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3810069ee0 0x7f381006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.811+0000 7f3822447640 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x559910129430 0x559910129810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3822447640 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x559910129430 0x559910129810 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3822c48640 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3810069ee0 0x7f381006c3a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f381400b2e0 tx=0x7f3814008000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3822447640 1 -- 172.21.15.59:0/1212640971 <== osd.0 v2:172.21.15.59:6824/841983189 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f381804ea90 con 0x559910129430 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3825a37f40 1 -- 172.21.15.59:0/1212640971 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x559910129430 msgr2=0x559910129810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3825a37f40 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x559910129430 0x559910129810 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3825a37f40 1 -- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3810069ee0 msgr2=0x7f381006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3825a37f40 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3810069ee0 0x7f381006c3a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f381400b2e0 tx=0x7f3814008000 comp rx=0 tx=0).stop 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3825a37f40 1 -- 172.21.15.59:0/1212640971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100bc650 msgr2=0x559910076150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3825a37f40 1 --2- 172.21.15.59:0/1212640971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5599100bc650 0x559910076150 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f381c00a4d0 tx=0x7f381c00a990 comp rx=0 tx=0).stop 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3823449640 1 -- 172.21.15.59:0/1212640971 reap_dead start 2024-03-06T07:28:16.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3825a37f40 1 -- 172.21.15.59:0/1212640971 shutdown_connections 2024-03-06T07:28:16.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3825a37f40 1 -- 172.21.15.59:0/1212640971 >> 172.21.15.59:0/1212640971 conn(0x5599100a0350 msgr2=0x5599100be2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3825a37f40 1 -- 172.21.15.59:0/1212640971 shutdown_connections 2024-03-06T07:28:16.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:16.812+0000 7f3825a37f40 1 -- 172.21.15.59:0/1212640971 wait complete. 2024-03-06T07:28:16.820 INFO:teuthology.orchestra.run.smithi059.stdout:W�file_ddir_2* 2024-03-06T07:28:16.821 DEBUG:teuthology.orchestra.run.smithi059:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:16.905 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:28:16.905 INFO:teuthology.orchestra.run.smithi059.stdout: "ino": 1099511627777, 2024-03-06T07:28:16.905 INFO:teuthology.orchestra.run.smithi059.stdout: "ancestors": [ 2024-03-06T07:28:16.905 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: "dirino": 1099511628278, 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: "dname": "file_d", 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: "version": 25 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: "dirino": 1, 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: "dname": "dir_2", 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: "version": 42 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: ], 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: "pool": 17, 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: "old_pools": [] 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:28:16.906 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:16.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:16.908 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-03-06T07:28:16.965 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:17.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.105+0000 7f88addc7640 1 Processor -- start 2024-03-06T07:28:17.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.105+0000 7f88addc7640 1 -- start start 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f88addc7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88a80d22b0 0x7f88a80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f88addc7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f88a80b41f0 0x7f88a80b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f88addc7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a80b3430 con 0x7f88a815dfe0 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f88addc7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a80d2c50 con 0x7f88a80b41f0 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f88addc7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80b31e0 con 0x7f88a80d22b0 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f88a7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f88a80b41f0 0x7f88a80b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f88a7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f88a80b41f0 0x7f88a80b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33524/0 (socket says 172.21.15.59:33524) 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f88a7fff640 1 -- 172.21.15.59:0/3978549283 learned_addr learned my addr 172.21.15.59:0/3978549283 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f889effd640 1 --2- 172.21.15.59:0/3978549283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88a80d22b0 0x7f88a80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f88a7fff640 1 -- 172.21.15.59:0/3978549283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88a80d22b0 msgr2=0x7f88a80d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f88a7fff640 1 --2- 172.21.15.59:0/3978549283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88a80d22b0 0x7f88a80d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.106+0000 7f88a7fff640 1 -- 172.21.15.59:0/3978549283 >> v1:172.21.15.164:6789/0 conn(0x7f88a815dfe0 legacy=0x7f88a80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.107+0000 7f88a7fff640 1 -- 172.21.15.59:0/3978549283 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a80b3710 con 0x7f88a80b41f0 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88a7fff640 1 --2- 172.21.15.59:0/3978549283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88a80b41f0 0x7f88a80b4650 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009e50 tx=0x7f88a002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142e663913b90dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88a6ffd640 1 -- 172.21.15.59:0/3978549283 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a002cb00 con 0x7f88a80b41f0 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88a6ffd640 1 -- 172.21.15.59:0/3978549283 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a002cca0 con 0x7f88a80b41f0 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88a6ffd640 1 -- 172.21.15.59:0/3978549283 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0034610 con 0x7f88a80b41f0 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88addc7640 1 -- 172.21.15.59:0/3978549283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88a80b41f0 msgr2=0x7f88a80b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88addc7640 1 --2- 172.21.15.59:0/3978549283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88a80b41f0 0x7f88a80b4650 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009e50 tx=0x7f88a002bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88addc7640 1 -- 172.21.15.59:0/3978549283 shutdown_connections 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88addc7640 1 --2- 172.21.15.59:0/3978549283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88a80b41f0 0x7f88a80b4650 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88addc7640 1 --2- 172.21.15.59:0/3978549283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88a80d22b0 0x7f88a80d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88addc7640 1 -- 172.21.15.59:0/3978549283 >> 172.21.15.59:0/3978549283 conn(0x7f88a80d3cd0 msgr2=0x7f88a80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88addc7640 1 -- 172.21.15.59:0/3978549283 shutdown_connections 2024-03-06T07:28:17.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.108+0000 7f88addc7640 1 -- 172.21.15.59:0/3978549283 wait complete. 2024-03-06T07:28:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.110+0000 7f88addc7640 1 Processor -- start 2024-03-06T07:28:17.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.110+0000 7f88addc7640 1 -- start start 2024-03-06T07:28:17.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88addc7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88a80b41f0 0x7f88a80b9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88addc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88a80d22b0 0x7f88a80b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88addc7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88a815dfe0 0x7f88a81610d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88addc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80cfa00 con 0x7f88a80d22b0 2024-03-06T07:28:17.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88addc7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80b3d60 con 0x7f88a80b41f0 2024-03-06T07:28:17.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88addc7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80d1c70 con 0x7f88a815dfe0 2024-03-06T07:28:17.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f889effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88a80d22b0 0x7f88a80b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f889effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88a80d22b0 0x7f88a80b8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33540/0 (socket says 172.21.15.59:33540) 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f889effd640 1 -- 172.21.15.59:0/3876617847 learned_addr learned my addr 172.21.15.59:0/3876617847 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88a77fe640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88a80b41f0 0x7f88a80b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88a7fff640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88a815dfe0 0x7f88a81610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f889effd640 1 -- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88a815dfe0 msgr2=0x7f88a81610d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f889effd640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88a815dfe0 0x7f88a81610d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f889effd640 1 -- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88a80b41f0 msgr2=0x7f88a80b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f889effd640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88a80b41f0 0x7f88a80b9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f889effd640 1 -- 172.21.15.59:0/3876617847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a815a700 con 0x7f88a80d22b0 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f889effd640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88a80d22b0 0x7f88a80b8010 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f889000d830 tx=0x7f889000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88a4ff9640 1 -- 172.21.15.59:0/3876617847 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889000b660 con 0x7f88a80d22b0 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88a4ff9640 1 -- 172.21.15.59:0/3876617847 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8890019070 con 0x7f88a80d22b0 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88addc7640 1 -- 172.21.15.59:0/3876617847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a80b86f0 con 0x7f88a80d22b0 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88a4ff9640 1 -- 172.21.15.59:0/3876617847 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889000ba20 con 0x7f88a80d22b0 2024-03-06T07:28:17.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.112+0000 7f88addc7640 1 -- 172.21.15.59:0/3876617847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a80b8cb0 con 0x7f88a80d22b0 2024-03-06T07:28:17.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.113+0000 7f88addc7640 1 -- 172.21.15.59:0/3876617847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a804f330 con 0x7f88a80d22b0 2024-03-06T07:28:17.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.114+0000 7f88a4ff9640 1 -- 172.21.15.59:0/3876617847 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f889001e460 con 0x7f88a80d22b0 2024-03-06T07:28:17.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.114+0000 7f88a4ff9640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 0x7f888806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.114+0000 7f88a4ff9640 1 -- 172.21.15.59:0/3876617847 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f889008a790 con 0x7f88a80d22b0 2024-03-06T07:28:17.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.115+0000 7f88a77fe640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 0x7f888806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.115+0000 7f88a77fe640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 0x7f888806d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f88980016e0 tx=0x7f8898009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.122+0000 7f88a4ff9640 1 -- 172.21.15.59:0/3876617847 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8890058f30 con 0x7f88a80d22b0 2024-03-06T07:28:17.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.351+0000 7f88addc7640 1 -- 172.21.15.59:0/3876617847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88a80ce910 con 0x7f88a80d22b0 2024-03-06T07:28:17.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.352+0000 7f88a4ff9640 1 -- 172.21.15.59:0/3876617847 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f889005cfd0 con 0x7f88a80d22b0 2024-03-06T07:28:17.354 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:17.354 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:17.354 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:17.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 -- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 msgr2=0x7f888806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 0x7f888806d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f88980016e0 tx=0x7f8898009040 comp rx=0 tx=0).stop 2024-03-06T07:28:17.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 -- 172.21.15.59:0/3876617847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88a80d22b0 msgr2=0x7f88a80b8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88a80d22b0 0x7f88a80b8010 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f889000d830 tx=0x7f889000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 -- 172.21.15.59:0/3876617847 shutdown_connections 2024-03-06T07:28:17.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f888806b4d0 0x7f888806d990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88a815dfe0 0x7f88a81610d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88a80d22b0 0x7f88a80b8010 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 --2- 172.21.15.59:0/3876617847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88a80b41f0 0x7f88a80b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 -- 172.21.15.59:0/3876617847 >> 172.21.15.59:0/3876617847 conn(0x7f88a80d3cd0 msgr2=0x7f88a806d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 -- 172.21.15.59:0/3876617847 shutdown_connections 2024-03-06T07:28:17.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:17.354+0000 7f88addc7640 1 -- 172.21.15.59:0/3876617847 wait complete. 2024-03-06T07:28:17.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:17.485 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:28:17.485 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:28:17.485 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-03-06T07:28:20.399 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:20.402 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:22.486 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:22.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.587+0000 7f4b68b5c640 1 Processor -- start 2024-03-06T07:28:22.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.587+0000 7f4b68b5c640 1 -- start start 2024-03-06T07:28:22.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b68b5c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4b641515e0 0x7f4b640cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b68b5c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4b6415b430 0x7f4b64071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b68b5c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6415b8e0 con 0x7f4b640cd7a0 2024-03-06T07:28:22.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b68b5c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b64070a70 con 0x7f4b641515e0 2024-03-06T07:28:22.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b68b5c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b64070d50 con 0x7f4b6415b430 2024-03-06T07:28:22.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b6259c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4b641515e0 0x7f4b640cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b62d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4b640cd7a0 0x7f4b640cdb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60034/0 (socket says 172.21.15.59:60034) 2024-03-06T07:28:22.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b62d9d640 1 -- 172.21.15.59:0/2173160178 learned_addr learned my addr 172.21.15.59:0/2173160178 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:22.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b61d9b640 1 --2- 172.21.15.59:0/2173160178 >> v2:172.21.15.164:3300/0 conn(0x7f4b6415b430 0x7f4b64071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b6259c640 1 -- 172.21.15.59:0/2173160178 >> v2:172.21.15.164:3300/0 conn(0x7f4b6415b430 msgr2=0x7f4b64071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b6259c640 1 --2- 172.21.15.59:0/2173160178 >> v2:172.21.15.164:3300/0 conn(0x7f4b6415b430 0x7f4b64071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b6259c640 1 -- 172.21.15.59:0/2173160178 >> v1:172.21.15.59:6789/0 conn(0x7f4b640cd7a0 legacy=0x7f4b640cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b6159a640 1 -- 172.21.15.59:0/2173160178 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935932262 0 0) 0x7f4b6415b8e0 con 0x7f4b640cd7a0 2024-03-06T07:28:22.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.588+0000 7f4b6259c640 1 -- 172.21.15.59:0/2173160178 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b64071030 con 0x7f4b641515e0 2024-03-06T07:28:22.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.589+0000 7f4b6259c640 1 --2- 172.21.15.59:0/2173160178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b641515e0 0x7f4b640cce40 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f4b640cc110 tx=0x7f4b5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ee410aee292d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.589+0000 7f4b6159a640 1 -- 172.21.15.59:0/2173160178 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b54035020 con 0x7f4b641515e0 2024-03-06T07:28:22.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.589+0000 7f4b6159a640 1 -- 172.21.15.59:0/2173160178 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b54031830 con 0x7f4b641515e0 2024-03-06T07:28:22.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.589+0000 7f4b6159a640 1 -- 172.21.15.59:0/2173160178 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b54031c00 con 0x7f4b641515e0 2024-03-06T07:28:22.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.590+0000 7f4b68b5c640 1 -- 172.21.15.59:0/2173160178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b641515e0 msgr2=0x7f4b640cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.590+0000 7f4b68b5c640 1 --2- 172.21.15.59:0/2173160178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b641515e0 0x7f4b640cce40 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f4b640cc110 tx=0x7f4b5402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.590+0000 7f4b68b5c640 1 -- 172.21.15.59:0/2173160178 shutdown_connections 2024-03-06T07:28:22.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.590+0000 7f4b68b5c640 1 --2- 172.21.15.59:0/2173160178 >> v2:172.21.15.164:3300/0 conn(0x7f4b6415b430 0x7f4b64071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.590+0000 7f4b68b5c640 1 --2- 172.21.15.59:0/2173160178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b641515e0 0x7f4b640cce40 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.590+0000 7f4b68b5c640 1 -- 172.21.15.59:0/2173160178 >> 172.21.15.59:0/2173160178 conn(0x7f4b640cf8a0 msgr2=0x7f4b640cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.590+0000 7f4b68b5c640 1 -- 172.21.15.59:0/2173160178 shutdown_connections 2024-03-06T07:28:22.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.590+0000 7f4b68b5c640 1 -- 172.21.15.59:0/2173160178 wait complete. 2024-03-06T07:28:22.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.592+0000 7f4b68b5c640 1 Processor -- start 2024-03-06T07:28:22.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.592+0000 7f4b68b5c640 1 -- start start 2024-03-06T07:28:22.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b68b5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b640cd7a0 0x7f4b641e87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b68b5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b641515e0 0x7f4b640cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b68b5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b6415b430 0x7f4b6414b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b68b5c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b64071ef0 con 0x7f4b641515e0 2024-03-06T07:28:22.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b68b5c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b64071d70 con 0x7f4b640cd7a0 2024-03-06T07:28:22.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b68b5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b640cc3a0 con 0x7f4b6415b430 2024-03-06T07:28:22.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b61d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b641515e0 0x7f4b640cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b61d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b641515e0 0x7f4b640cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33560/0 (socket says 172.21.15.59:33560) 2024-03-06T07:28:22.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b61d9b640 1 -- 172.21.15.59:0/4177873237 learned_addr learned my addr 172.21.15.59:0/4177873237 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b6259c640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b640cd7a0 0x7f4b641e87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b62d9d640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b6415b430 0x7f4b6414b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b61d9b640 1 -- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b6415b430 msgr2=0x7f4b6414b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b61d9b640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b6415b430 0x7f4b6414b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b61d9b640 1 -- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b640cd7a0 msgr2=0x7f4b641e87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b61d9b640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b640cd7a0 0x7f4b641e87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b61d9b640 1 -- 172.21.15.59:0/4177873237 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b640cbf20 con 0x7f4b641515e0 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b61d9b640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b641515e0 0x7f4b640cb9e0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800d5f0 tx=0x7f4b5800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.593+0000 7f4b52ffd640 1 -- 172.21.15.59:0/4177873237 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b580125f0 con 0x7f4b641515e0 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.594+0000 7f4b52ffd640 1 -- 172.21.15.59:0/4177873237 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b58019070 con 0x7f4b641515e0 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.594+0000 7f4b52ffd640 1 -- 172.21.15.59:0/4177873237 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b580129d0 con 0x7f4b641515e0 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.594+0000 7f4b68b5c640 1 -- 172.21.15.59:0/4177873237 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6414c0a0 con 0x7f4b641515e0 2024-03-06T07:28:22.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.594+0000 7f4b68b5c640 1 -- 172.21.15.59:0/4177873237 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b640cb1e0 con 0x7f4b641515e0 2024-03-06T07:28:22.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.595+0000 7f4b68b5c640 1 -- 172.21.15.59:0/4177873237 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b64068840 con 0x7f4b641515e0 2024-03-06T07:28:22.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.595+0000 7f4b52ffd640 1 -- 172.21.15.59:0/4177873237 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b58012b70 con 0x7f4b641515e0 2024-03-06T07:28:22.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.595+0000 7f4b52ffd640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b4406b4d0 0x7f4b4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.596+0000 7f4b52ffd640 1 -- 172.21.15.59:0/4177873237 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4b5808a5c0 con 0x7f4b641515e0 2024-03-06T07:28:22.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.596+0000 7f4b6259c640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b4406b4d0 0x7f4b4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.596+0000 7f4b6259c640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b4406b4d0 0x7f4b4406d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4b54036000 tx=0x7f4b54008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.604+0000 7f4b52ffd640 1 -- 172.21.15.59:0/4177873237 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b640cbf20 con 0x7f4b641515e0 2024-03-06T07:28:22.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.823+0000 7f4b68b5c640 1 -- 172.21.15.59:0/4177873237 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b6404f330 con 0x7f4b641515e0 2024-03-06T07:28:22.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.823+0000 7f4b52ffd640 1 -- 172.21.15.59:0/4177873237 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f4b58059440 con 0x7f4b641515e0 2024-03-06T07:28:22.825 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:22.826 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:22.826 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 -- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b4406b4d0 msgr2=0x7f4b4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b4406b4d0 0x7f4b4406d990 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4b54036000 tx=0x7f4b54008030 comp rx=0 tx=0).stop 2024-03-06T07:28:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 -- 172.21.15.59:0/4177873237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b641515e0 msgr2=0x7f4b640cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b641515e0 0x7f4b640cb9e0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800d5f0 tx=0x7f4b5800dab0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 -- 172.21.15.59:0/4177873237 shutdown_connections 2024-03-06T07:28:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b4406b4d0 0x7f4b4406d990 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b6415b430 0x7f4b6414b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b641515e0 0x7f4b640cb9e0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 --2- 172.21.15.59:0/4177873237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b640cd7a0 0x7f4b641e87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 -- 172.21.15.59:0/4177873237 >> 172.21.15.59:0/4177873237 conn(0x7f4b640cf8a0 msgr2=0x7f4b6414e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 -- 172.21.15.59:0/4177873237 shutdown_connections 2024-03-06T07:28:22.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:22.826+0000 7f4b68b5c640 1 -- 172.21.15.59:0/4177873237 wait complete. 2024-03-06T07:28:22.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:22.956 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:28:22.957 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:28:22.957 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:23.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.056+0000 7f82f10f4640 1 Processor -- start 2024-03-06T07:28:23.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.056+0000 7f82f10f4640 1 -- start start 2024-03-06T07:28:23.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.057+0000 7f82f10f4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f82ec0d2710 0x7f82ec0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.057+0000 7f82f10f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ec0b3430 con 0x7f82ec0d22b0 2024-03-06T07:28:23.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.057+0000 7f82f10f4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ec0b3710 con 0x7f82ec15dfe0 2024-03-06T07:28:23.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.057+0000 7f82f10f4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ec0b31e0 con 0x7f82ec0d2710 2024-03-06T07:28:23.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.057+0000 7f82eb59e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f82ec0d2710 0x7f82ec0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.057+0000 7f82ea59c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f82ec0d22b0 0x7f82ec0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60042/0 (socket says 172.21.15.59:60042) 2024-03-06T07:28:23.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.057+0000 7f82ea59c640 1 -- 172.21.15.59:0/2317621114 learned_addr learned my addr 172.21.15.59:0/2317621114 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:23.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.058+0000 7f82e9d9b640 1 -- 172.21.15.59:0/2317621114 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228597473 0 0) 0x7f82ec0b3430 con 0x7f82ec0d22b0 2024-03-06T07:28:23.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.058+0000 7f82e9d9b640 1 -- 172.21.15.59:0/2317621114 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d8003520 con 0x7f82ec0d22b0 2024-03-06T07:28:23.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.058+0000 7f82eb59e640 1 -- 172.21.15.59:0/2317621114 >> v1:172.21.15.164:6789/0 conn(0x7f82ec15dfe0 legacy=0x7f82ec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.058+0000 7f82eb59e640 1 -- 172.21.15.59:0/2317621114 >> v1:172.21.15.59:6789/0 conn(0x7f82ec0d22b0 legacy=0x7f82ec0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.058+0000 7f82eb59e640 1 -- 172.21.15.59:0/2317621114 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec0b39f0 con 0x7f82ec0d2710 2024-03-06T07:28:23.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.058+0000 7f82eb59e640 1 --2- 172.21.15.59:0/2317621114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82ec0d2710 0x7f82ec0b4900 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009dc0 tx=0x7f82dc02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1091731837f0df95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.059+0000 7f82e9d9b640 1 -- 172.21.15.59:0/2317621114 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc039070 con 0x7f82ec0d2710 2024-03-06T07:28:23.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.059+0000 7f82e9d9b640 1 -- 172.21.15.59:0/2317621114 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc02cc40 con 0x7f82ec0d2710 2024-03-06T07:28:23.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.059+0000 7f82e9d9b640 1 -- 172.21.15.59:0/2317621114 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc034440 con 0x7f82ec0d2710 2024-03-06T07:28:23.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.059+0000 7f82f10f4640 1 -- 172.21.15.59:0/2317621114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82ec0d2710 msgr2=0x7f82ec0b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.059+0000 7f82f10f4640 1 --2- 172.21.15.59:0/2317621114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82ec0d2710 0x7f82ec0b4900 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009dc0 tx=0x7f82dc02be90 comp rx=0 tx=0).stop 2024-03-06T07:28:23.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.059+0000 7f82f10f4640 1 -- 172.21.15.59:0/2317621114 shutdown_connections 2024-03-06T07:28:23.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.059+0000 7f82f10f4640 1 --2- 172.21.15.59:0/2317621114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82ec0d2710 0x7f82ec0b4900 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.059+0000 7f82f10f4640 1 -- 172.21.15.59:0/2317621114 >> 172.21.15.59:0/2317621114 conn(0x7f82ec0d3cd0 msgr2=0x7f82ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.059+0000 7f82f10f4640 1 -- 172.21.15.59:0/2317621114 shutdown_connections 2024-03-06T07:28:23.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.059+0000 7f82f10f4640 1 -- 172.21.15.59:0/2317621114 wait complete. 2024-03-06T07:28:23.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.061+0000 7f82f10f4640 1 Processor -- start 2024-03-06T07:28:23.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.061+0000 7f82f10f4640 1 -- start start 2024-03-06T07:28:23.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.061+0000 7f82f10f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82ec0d22b0 0x7f82ec15e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.061+0000 7f82f10f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82ec0d2710 0x7f82ec0b9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.061+0000 7f82f10f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82ec15dfe0 0x7f82ec160030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.061+0000 7f82f10f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0cfa00 con 0x7f82ec0d2710 2024-03-06T07:28:23.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.061+0000 7f82f10f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0b3d60 con 0x7f82ec0d22b0 2024-03-06T07:28:23.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.061+0000 7f82f10f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0d1c70 con 0x7f82ec15dfe0 2024-03-06T07:28:23.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.061+0000 7f82ea59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82ec0d2710 0x7f82ec0b9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.061+0000 7f82ea59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82ec0d2710 0x7f82ec0b9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33570/0 (socket says 172.21.15.59:33570) 2024-03-06T07:28:23.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.061+0000 7f82ea59c640 1 -- 172.21.15.59:0/2702228693 learned_addr learned my addr 172.21.15.59:0/2702228693 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82eb59e640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82ec15dfe0 0x7f82ec160030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82ead9d640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82ec0d22b0 0x7f82ec15e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82ea59c640 1 -- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82ec15dfe0 msgr2=0x7f82ec160030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82ea59c640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82ec15dfe0 0x7f82ec160030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82ea59c640 1 -- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82ec0d22b0 msgr2=0x7f82ec15e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82ea59c640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82ec0d22b0 0x7f82ec15e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82ea59c640 1 -- 172.21.15.59:0/2702228693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec1609d0 con 0x7f82ec0d2710 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82eb59e640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82ec15dfe0 0x7f82ec160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82ead9d640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82ec0d22b0 0x7f82ec15e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82ea59c640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82ec0d2710 0x7f82ec0b9180 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f82d400c430 tx=0x7f82d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82e37fe640 1 -- 172.21.15.59:0/2702228693 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4017070 con 0x7f82ec0d2710 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82e37fe640 1 -- 172.21.15.59:0/2702228693 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d4005ce0 con 0x7f82ec0d2710 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82f10f4640 1 -- 172.21.15.59:0/2702228693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec160cb0 con 0x7f82ec0d2710 2024-03-06T07:28:23.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82e37fe640 1 -- 172.21.15.59:0/2702228693 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d40072f0 con 0x7f82ec0d2710 2024-03-06T07:28:23.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.062+0000 7f82f10f4640 1 -- 172.21.15.59:0/2702228693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec161550 con 0x7f82ec0d2710 2024-03-06T07:28:23.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.063+0000 7f82f10f4640 1 -- 172.21.15.59:0/2702228693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec160e80 con 0x7f82ec0d2710 2024-03-06T07:28:23.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.064+0000 7f82e37fe640 1 -- 172.21.15.59:0/2702228693 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82d401a070 con 0x7f82ec0d2710 2024-03-06T07:28:23.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.064+0000 7f82e37fe640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82cc06b4d0 0x7f82cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.064+0000 7f82e37fe640 1 -- 172.21.15.59:0/2702228693 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f82d4089750 con 0x7f82ec0d2710 2024-03-06T07:28:23.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.065+0000 7f82ead9d640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82cc06b4d0 0x7f82cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.065+0000 7f82ead9d640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82cc06b4d0 0x7f82cc06d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f82e400a830 tx=0x7f82e4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.069+0000 7f82e37fe640 1 -- 172.21.15.59:0/2702228693 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82d4058540 con 0x7f82ec0d2710 2024-03-06T07:28:23.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.287+0000 7f82f10f4640 1 -- 172.21.15.59:0/2702228693 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82ec04f330 con 0x7f82ec0d2710 2024-03-06T07:28:23.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.288+0000 7f82e37fe640 1 -- 172.21.15.59:0/2702228693 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f82d4057e40 con 0x7f82ec0d2710 2024-03-06T07:28:23.290 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:23.290 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:23.290 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:23.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 -- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82cc06b4d0 msgr2=0x7f82cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82cc06b4d0 0x7f82cc06d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f82e400a830 tx=0x7f82e4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:23.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 -- 172.21.15.59:0/2702228693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82ec0d2710 msgr2=0x7f82ec0b9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82ec0d2710 0x7f82ec0b9180 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f82d400c430 tx=0x7f82d400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 -- 172.21.15.59:0/2702228693 shutdown_connections 2024-03-06T07:28:23.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f82cc06b4d0 0x7f82cc06d990 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82ec15dfe0 0x7f82ec160030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82ec0d2710 0x7f82ec0b9180 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 --2- 172.21.15.59:0/2702228693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82ec0d22b0 0x7f82ec15e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 -- 172.21.15.59:0/2702228693 >> 172.21.15.59:0/2702228693 conn(0x7f82ec0d3cd0 msgr2=0x7f82ec0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 -- 172.21.15.59:0/2702228693 shutdown_connections 2024-03-06T07:28:23.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.290+0000 7f82f10f4640 1 -- 172.21.15.59:0/2702228693 wait complete. 2024-03-06T07:28:23.303 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:28:23.443 INFO:teuthology.orchestra.run.smithi059.stdout:{"purge_queue":{"pq_executed":1}} 2024-03-06T07:28:23.443 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-03-06T07:28:23.443 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:23.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.545+0000 7f4a9cd20640 1 Processor -- start 2024-03-06T07:28:23.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.545+0000 7f4a9cd20640 1 -- start start 2024-03-06T07:28:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.545+0000 7f4a9cd20640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a9815bdd0 0x7f4a980c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a9cd20640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4a9815fa10 0x7f4a980ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a9cd20640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a980cebe0 con 0x7f4a980d16e0 2024-03-06T07:28:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a9cd20640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a980ceec0 con 0x7f4a9815fa10 2024-03-06T07:28:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a9cd20640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980cf1a0 con 0x7f4a9815bdd0 2024-03-06T07:28:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a95d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4a9815fa10 0x7f4a980ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a96d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4a980d16e0 0x7f4a980d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58974/0 (socket says 172.21.15.59:58974) 2024-03-06T07:28:23.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a96d9d640 1 -- 172.21.15.59:0/4156567712 learned_addr learned my addr 172.21.15.59:0/4156567712 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a95d9b640 1 -- 172.21.15.59:0/4156567712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a9815bdd0 msgr2=0x7f4a980c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a9659c640 1 --2- 172.21.15.59:0/4156567712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a9815bdd0 0x7f4a980c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a9559a640 1 -- 172.21.15.59:0/4156567712 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383774595 0 0) 0x7f4a980cebe0 con 0x7f4a980d16e0 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a95d9b640 1 --2- 172.21.15.59:0/4156567712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a9815bdd0 0x7f4a980c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a95d9b640 1 -- 172.21.15.59:0/4156567712 >> v1:172.21.15.59:6789/0 conn(0x7f4a980d16e0 legacy=0x7f4a980d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a95d9b640 1 -- 172.21.15.59:0/4156567712 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a980cc8e0 con 0x7f4a9815fa10 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.546+0000 7f4a9659c640 1 --2- 172.21.15.59:0/4156567712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a9815bdd0 0x7f4a980c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.547+0000 7f4a95d9b640 1 --2- 172.21.15.59:0/4156567712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a9815fa10 0x7f4a980ce270 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f4a880097a0 tx=0x7f4a8802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71c8361e57cdf87 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.547+0000 7f4a9559a640 1 -- 172.21.15.59:0/4156567712 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a880315a0 con 0x7f4a9815fa10 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.547+0000 7f4a9559a640 1 -- 172.21.15.59:0/4156567712 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88037070 con 0x7f4a9815fa10 2024-03-06T07:28:23.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.547+0000 7f4a9559a640 1 -- 172.21.15.59:0/4156567712 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88031960 con 0x7f4a9815fa10 2024-03-06T07:28:23.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.548+0000 7f4a9cd20640 1 -- 172.21.15.59:0/4156567712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a9815fa10 msgr2=0x7f4a980ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.548+0000 7f4a9cd20640 1 --2- 172.21.15.59:0/4156567712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a9815fa10 0x7f4a980ce270 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f4a880097a0 tx=0x7f4a8802b840 comp rx=0 tx=0).stop 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.548+0000 7f4a9cd20640 1 -- 172.21.15.59:0/4156567712 shutdown_connections 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.548+0000 7f4a9cd20640 1 --2- 172.21.15.59:0/4156567712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a9815fa10 0x7f4a980ce270 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.548+0000 7f4a9cd20640 1 --2- 172.21.15.59:0/4156567712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a9815bdd0 0x7f4a980c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.548+0000 7f4a9cd20640 1 -- 172.21.15.59:0/4156567712 >> 172.21.15.59:0/4156567712 conn(0x7f4a9809a9f0 msgr2=0x7f4a9809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.548+0000 7f4a9cd20640 1 -- 172.21.15.59:0/4156567712 shutdown_connections 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.548+0000 7f4a9cd20640 1 -- 172.21.15.59:0/4156567712 wait complete. 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.550+0000 7f4a9cd20640 1 Processor -- start 2024-03-06T07:28:23.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.550+0000 7f4a9cd20640 1 -- start start 2024-03-06T07:28:23.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9cd20640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a980d16e0 0x7f4a9815a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9cd20640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a9815bdd0 0x7f4a9815d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9cd20640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a9815fa10 0x7f4a9815e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9cd20640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9815c1b0 con 0x7f4a980d16e0 2024-03-06T07:28:23.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9cd20640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980cc5d0 con 0x7f4a9815fa10 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9cd20640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980ce7b0 con 0x7f4a9815bdd0 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a980d16e0 0x7f4a9815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a980d16e0 0x7f4a9815a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51896/0 (socket says 172.21.15.59:51896) 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9659c640 1 -- 172.21.15.59:0/1668000199 learned_addr learned my addr 172.21.15.59:0/1668000199 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a96d9d640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a9815fa10 0x7f4a9815e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a95d9b640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a9815bdd0 0x7f4a9815d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9659c640 1 -- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a9815bdd0 msgr2=0x7f4a9815d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9659c640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a9815bdd0 0x7f4a9815d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9659c640 1 -- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a9815fa10 msgr2=0x7f4a9815e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.551+0000 7f4a9659c640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a9815fa10 0x7f4a9815e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.552+0000 7f4a9659c640 1 -- 172.21.15.59:0/1668000199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9815aeb0 con 0x7f4a980d16e0 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.552+0000 7f4a9659c640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a980d16e0 0x7f4a9815a970 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00c7f0 tx=0x7f4a8c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.552+0000 7f4a86ffd640 1 -- 172.21.15.59:0/1668000199 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c015070 con 0x7f4a980d16e0 2024-03-06T07:28:23.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.552+0000 7f4a86ffd640 1 -- 172.21.15.59:0/1668000199 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8c007a30 con 0x7f4a980d16e0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.552+0000 7f4a9cd20640 1 -- 172.21.15.59:0/1668000199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a981f1640 con 0x7f4a980d16e0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.552+0000 7f4a86ffd640 1 -- 172.21.15.59:0/1668000199 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c007e00 con 0x7f4a980d16e0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.552+0000 7f4a9cd20640 1 -- 172.21.15.59:0/1668000199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9815c580 con 0x7f4a980d16e0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.553+0000 7f4a9cd20640 1 -- 172.21.15.59:0/1668000199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9804f330 con 0x7f4a980d16e0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.553+0000 7f4a86ffd640 1 -- 172.21.15.59:0/1668000199 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a8c018070 con 0x7f4a980d16e0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.554+0000 7f4a86ffd640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a7806b4d0 0x7f4a7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.554+0000 7f4a86ffd640 1 -- 172.21.15.59:0/1668000199 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4a8c08a280 con 0x7f4a980d16e0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.554+0000 7f4a95d9b640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a7806b4d0 0x7f4a7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.555+0000 7f4a95d9b640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a7806b4d0 0x7f4a7806d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4a88036000 tx=0x7f4a88008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.558+0000 7f4a86ffd640 1 -- 172.21.15.59:0/1668000199 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a8c058fd0 con 0x7f4a980d16e0 2024-03-06T07:28:23.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.775+0000 7f4a9cd20640 1 -- 172.21.15.59:0/1668000199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a980c5800 con 0x7f4a980d16e0 2024-03-06T07:28:23.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.776+0000 7f4a86ffd640 1 -- 172.21.15.59:0/1668000199 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f4a8c0588d0 con 0x7f4a980d16e0 2024-03-06T07:28:23.778 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:23.778 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:23.778 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:23.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 -- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a7806b4d0 msgr2=0x7f4a7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a7806b4d0 0x7f4a7806d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4a88036000 tx=0x7f4a88008030 comp rx=0 tx=0).stop 2024-03-06T07:28:23.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 -- 172.21.15.59:0/1668000199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a980d16e0 msgr2=0x7f4a9815a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a980d16e0 0x7f4a9815a970 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00c7f0 tx=0x7f4a8c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 -- 172.21.15.59:0/1668000199 shutdown_connections 2024-03-06T07:28:23.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a7806b4d0 0x7f4a7806d990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a9815fa10 0x7f4a9815e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a9815bdd0 0x7f4a9815d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 --2- 172.21.15.59:0/1668000199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a980d16e0 0x7f4a9815a970 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 -- 172.21.15.59:0/1668000199 >> 172.21.15.59:0/1668000199 conn(0x7f4a9809a9f0 msgr2=0x7f4a9809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 -- 172.21.15.59:0/1668000199 shutdown_connections 2024-03-06T07:28:23.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:23.778+0000 7f4a9cd20640 1 -- 172.21.15.59:0/1668000199 wait complete. 2024-03-06T07:28:23.791 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache 2024-03-06T07:28:23.933 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-03-06T07:28:23.933 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-03-06T07:28:23.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:24.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.033+0000 7f0b59b3a640 1 Processor -- start 2024-03-06T07:28:24.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.033+0000 7f0b59b3a640 1 -- start start 2024-03-06T07:28:24.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.034+0000 7f0b59b3a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0b5415bdd0 0x7f0b540c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.034+0000 7f0b59b3a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0b5415fa10 0x7f0b540ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.034+0000 7f0b59b3a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b540cebe0 con 0x7f0b540d16e0 2024-03-06T07:28:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.034+0000 7f0b59b3a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b540ceec0 con 0x7f0b5415fa10 2024-03-06T07:28:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.034+0000 7f0b59b3a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b540cf1a0 con 0x7f0b5415bdd0 2024-03-06T07:28:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.034+0000 7f0b52ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0b5415fa10 0x7f0b540ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.034+0000 7f0b53fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0b540d16e0 0x7f0b540d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58976/0 (socket says 172.21.15.59:58976) 2024-03-06T07:28:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.034+0000 7f0b52ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0b5415fa10 0x7f0b540ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51908/0 (socket says 172.21.15.59:51908) 2024-03-06T07:28:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.034+0000 7f0b53fff640 1 -- 172.21.15.59:0/4017805496 learned_addr learned my addr 172.21.15.59:0/4017805496 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.034+0000 7f0b537fe640 1 --2- 172.21.15.59:0/4017805496 >> v2:172.21.15.164:3300/0 conn(0x7f0b5415bdd0 0x7f0b540c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.035+0000 7f0b52ffd640 1 -- 172.21.15.59:0/4017805496 >> v2:172.21.15.164:3300/0 conn(0x7f0b5415bdd0 msgr2=0x7f0b540c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.035+0000 7f0b52ffd640 1 --2- 172.21.15.59:0/4017805496 >> v2:172.21.15.164:3300/0 conn(0x7f0b5415bdd0 0x7f0b540c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.035+0000 7f0b52ffd640 1 -- 172.21.15.59:0/4017805496 >> v1:172.21.15.59:6789/0 conn(0x7f0b540d16e0 legacy=0x7f0b540d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.035+0000 7f0b52ffd640 1 -- 172.21.15.59:0/4017805496 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b540cc8e0 con 0x7f0b5415fa10 2024-03-06T07:28:24.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.035+0000 7f0b52ffd640 1 --2- 172.21.15.59:0/4017805496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b5415fa10 0x7f0b540ce270 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f0b5406e3c0 tx=0x7f0b4402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9e36632f91b8db server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.035+0000 7f0b527fc640 1 -- 172.21.15.59:0/4017805496 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b440315a0 con 0x7f0b5415fa10 2024-03-06T07:28:24.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.035+0000 7f0b527fc640 1 -- 172.21.15.59:0/4017805496 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b44037070 con 0x7f0b5415fa10 2024-03-06T07:28:24.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.035+0000 7f0b527fc640 1 -- 172.21.15.59:0/4017805496 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b44031960 con 0x7f0b5415fa10 2024-03-06T07:28:24.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.036+0000 7f0b59b3a640 1 -- 172.21.15.59:0/4017805496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b5415fa10 msgr2=0x7f0b540ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.036+0000 7f0b59b3a640 1 --2- 172.21.15.59:0/4017805496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b5415fa10 0x7f0b540ce270 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f0b5406e3c0 tx=0x7f0b4402b840 comp rx=0 tx=0).stop 2024-03-06T07:28:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.036+0000 7f0b59b3a640 1 -- 172.21.15.59:0/4017805496 shutdown_connections 2024-03-06T07:28:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.036+0000 7f0b59b3a640 1 --2- 172.21.15.59:0/4017805496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b5415fa10 0x7f0b540ce270 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.036+0000 7f0b59b3a640 1 --2- 172.21.15.59:0/4017805496 >> v2:172.21.15.164:3300/0 conn(0x7f0b5415bdd0 0x7f0b540c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.036+0000 7f0b59b3a640 1 -- 172.21.15.59:0/4017805496 >> 172.21.15.59:0/4017805496 conn(0x7f0b5409a9f0 msgr2=0x7f0b5409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.036+0000 7f0b59b3a640 1 -- 172.21.15.59:0/4017805496 shutdown_connections 2024-03-06T07:28:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.036+0000 7f0b59b3a640 1 -- 172.21.15.59:0/4017805496 wait complete. 2024-03-06T07:28:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.037+0000 7f0b59b3a640 1 Processor -- start 2024-03-06T07:28:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.037+0000 7f0b59b3a640 1 -- start start 2024-03-06T07:28:24.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.038+0000 7f0b59b3a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b540d16e0 0x7f0b540c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.038+0000 7f0b59b3a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b5415bdd0 0x7f0b541f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.038+0000 7f0b59b3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b5415fa10 0x7f0b541f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.038+0000 7f0b59b3a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5415c1b0 con 0x7f0b5415fa10 2024-03-06T07:28:24.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.038+0000 7f0b59b3a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b540cc5d0 con 0x7f0b540d16e0 2024-03-06T07:28:24.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.038+0000 7f0b59b3a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b540ce7b0 con 0x7f0b5415bdd0 2024-03-06T07:28:24.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.038+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b5415fa10 0x7f0b541f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.038+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b5415fa10 0x7f0b541f7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51916/0 (socket says 172.21.15.59:51916) 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.038+0000 7f0b53fff640 1 -- 172.21.15.59:0/2742793411 learned_addr learned my addr 172.21.15.59:0/2742793411 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.038+0000 7f0b52ffd640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b5415bdd0 0x7f0b541f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.038+0000 7f0b537fe640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b540d16e0 0x7f0b540c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.039+0000 7f0b53fff640 1 -- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b5415bdd0 msgr2=0x7f0b541f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.039+0000 7f0b53fff640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b5415bdd0 0x7f0b541f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.039+0000 7f0b53fff640 1 -- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b540d16e0 msgr2=0x7f0b540c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.039+0000 7f0b53fff640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b540d16e0 0x7f0b540c9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.039+0000 7f0b53fff640 1 -- 172.21.15.59:0/2742793411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5406cf30 con 0x7f0b5415fa10 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.039+0000 7f0b53fff640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b5415fa10 0x7f0b541f7a80 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00ca70 tx=0x7f0b4c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.039+0000 7f0b3bfff640 1 -- 172.21.15.59:0/2742793411 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c017070 con 0x7f0b5415fa10 2024-03-06T07:28:24.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.039+0000 7f0b3bfff640 1 -- 172.21.15.59:0/2742793411 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4c00abd0 con 0x7f0b5415fa10 2024-03-06T07:28:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.039+0000 7f0b59b3a640 1 -- 172.21.15.59:0/2742793411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5406d210 con 0x7f0b5415fa10 2024-03-06T07:28:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.039+0000 7f0b3bfff640 1 -- 172.21.15.59:0/2742793411 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c012400 con 0x7f0b5415fa10 2024-03-06T07:28:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.039+0000 7f0b59b3a640 1 -- 172.21.15.59:0/2742793411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5406d7d0 con 0x7f0b5415fa10 2024-03-06T07:28:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.040+0000 7f0b59b3a640 1 -- 172.21.15.59:0/2742793411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5404f330 con 0x7f0b5415fa10 2024-03-06T07:28:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.041+0000 7f0b3bfff640 1 -- 172.21.15.59:0/2742793411 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b4c01a070 con 0x7f0b5415fa10 2024-03-06T07:28:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.041+0000 7f0b3bfff640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b4006b4d0 0x7f0b4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.041+0000 7f0b3bfff640 1 -- 172.21.15.59:0/2742793411 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0b4c089f40 con 0x7f0b5415fa10 2024-03-06T07:28:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.041+0000 7f0b537fe640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b4006b4d0 0x7f0b4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.042+0000 7f0b537fe640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b4006b4d0 0x7f0b4006d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f0b48006fd0 tx=0x7f0b48008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.046+0000 7f0b3bfff640 1 -- 172.21.15.59:0/2742793411 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b4c058660 con 0x7f0b5415fa10 2024-03-06T07:28:24.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.265+0000 7f0b59b3a640 1 -- 172.21.15.59:0/2742793411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b540b7a40 con 0x7f0b5415fa10 2024-03-06T07:28:24.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.266+0000 7f0b3bfff640 1 -- 172.21.15.59:0/2742793411 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f0b4c05c700 con 0x7f0b5415fa10 2024-03-06T07:28:24.268 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:24.268 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:24.268 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.267+0000 7f0b59b3a640 1 -- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b4006b4d0 msgr2=0x7f0b4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.267+0000 7f0b59b3a640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b4006b4d0 0x7f0b4006d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f0b48006fd0 tx=0x7f0b48008040 comp rx=0 tx=0).stop 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.267+0000 7f0b59b3a640 1 -- 172.21.15.59:0/2742793411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b5415fa10 msgr2=0x7f0b541f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.267+0000 7f0b59b3a640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b5415fa10 0x7f0b541f7a80 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00ca70 tx=0x7f0b4c00cf30 comp rx=0 tx=0).stop 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.267+0000 7f0b59b3a640 1 -- 172.21.15.59:0/2742793411 shutdown_connections 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.267+0000 7f0b59b3a640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0b4006b4d0 0x7f0b4006d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.268+0000 7f0b59b3a640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b5415fa10 0x7f0b541f7a80 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.268+0000 7f0b59b3a640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b5415bdd0 0x7f0b541f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.268+0000 7f0b59b3a640 1 --2- 172.21.15.59:0/2742793411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b540d16e0 0x7f0b540c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.268+0000 7f0b59b3a640 1 -- 172.21.15.59:0/2742793411 >> 172.21.15.59:0/2742793411 conn(0x7f0b5409a9f0 msgr2=0x7f0b540cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.268+0000 7f0b59b3a640 1 -- 172.21.15.59:0/2742793411 shutdown_connections 2024-03-06T07:28:24.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.268+0000 7f0b59b3a640 1 -- 172.21.15.59:0/2742793411 wait complete. 2024-03-06T07:28:24.281 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue 2024-03-06T07:28:24.398 INFO:teuthology.orchestra.run.smithi059.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-03-06T07:28:24.398 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-03-06T07:28:24.398 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:28:24.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.437+0000 7f70e7c48f40 1 Processor -- start 2024-03-06T07:28:24.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.437+0000 7f70e7c48f40 1 -- start start 2024-03-06T07:28:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e7c48f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a0a6d72980 0x55a0a6e47380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e7c48f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a0a6db9d70 0x55a0a6e2de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e7c48f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6e42070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e7c48f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a6db76f0 con 0x55a0a6e43f20 2024-03-06T07:28:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e7c48f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a6d15b70 con 0x55a0a6d72980 2024-03-06T07:28:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e7c48f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0a6d72d90 con 0x55a0a6db9d70 2024-03-06T07:28:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e56ce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6e42070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e56ce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6e42070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51922/0 (socket says 172.21.15.59:51922) 2024-03-06T07:28:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e56ce640 1 -- 172.21.15.59:0/4191487316 learned_addr learned my addr 172.21.15.59:0/4191487316 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/4191487316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a0a6d72980 0x55a0a6e47380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70dffff640 1 --2- 172.21.15.59:0/4191487316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a0a6db9d70 0x55a0a6e2de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e56ce640 1 -- 172.21.15.59:0/4191487316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a0a6db9d70 msgr2=0x55a0a6e2de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e56ce640 1 --2- 172.21.15.59:0/4191487316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a0a6db9d70 0x55a0a6e2de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e56ce640 1 -- 172.21.15.59:0/4191487316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a0a6d72980 msgr2=0x55a0a6e47380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e56ce640 1 --2- 172.21.15.59:0/4191487316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a0a6d72980 0x55a0a6e47380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.438+0000 7f70e56ce640 1 -- 172.21.15.59:0/4191487316 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0a6d9ab60 con 0x55a0a6e43f20 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70e56ce640 1 --2- 172.21.15.59:0/4191487316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6e42070 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f70e0001600 tx=0x7f70e0001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3bbb23ba09f035 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70deffd640 1 -- 172.21.15.59:0/4191487316 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0013620 con 0x55a0a6e43f20 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70deffd640 1 -- 172.21.15.59:0/4191487316 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e00137c0 con 0x55a0a6e43f20 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70deffd640 1 -- 172.21.15.59:0/4191487316 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0013bb0 con 0x55a0a6e43f20 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70e7c48f40 1 -- 172.21.15.59:0/4191487316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 msgr2=0x55a0a6e42070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/4191487316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6e42070 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f70e0001600 tx=0x7f70e0001ac0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70e7c48f40 1 -- 172.21.15.59:0/4191487316 shutdown_connections 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/4191487316 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6e42070 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/4191487316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a0a6db9d70 0x55a0a6e2de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/4191487316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a0a6d72980 0x55a0a6e47380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70e7c48f40 1 -- 172.21.15.59:0/4191487316 >> 172.21.15.59:0/4191487316 conn(0x55a0a6d6d8a0 msgr2=0x55a0a6e44c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70e7c48f40 1 -- 172.21.15.59:0/4191487316 shutdown_connections 2024-03-06T07:28:24.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.439+0000 7f70e7c48f40 1 -- 172.21.15.59:0/4191487316 wait complete. 2024-03-06T07:28:24.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.440+0000 7f70e7c48f40 1 Processor -- start 2024-03-06T07:28:24.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.440+0000 7f70e7c48f40 1 -- start start 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e7c48f40 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a0a6d72980 0x55a0a6e425d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e7c48f40 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a0a6db9d70 0x55a0a6d9b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e7c48f40 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6d9b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e7c48f40 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a6d9a880 con 0x55a0a6e43f20 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e7c48f40 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0a6d6dc80 con 0x55a0a6db9d70 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e7c48f40 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0a6e444a0 con 0x55a0a6d72980 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e56ce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6d9b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e56ce640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6d9b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51932/0 (socket says 172.21.15.59:51932) 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 learned_addr learned my addr 172.21.15.59:0/1177884515 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a0a6d72980 0x55a0a6e425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70dffff640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a0a6db9d70 0x55a0a6d9b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a0a6d72980 msgr2=0x55a0a6e425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e56ce640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55a0a6d72980 0x55a0a6e425d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a0a6db9d70 msgr2=0x55a0a6d9b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e56ce640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a0a6db9d70 0x55a0a6d9b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0a6e2b560 con 0x55a0a6e43f20 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70e56ce640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6d9b800 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f70e0001b00 tx=0x7f70e0000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70ddffb640 1 -- 172.21.15.59:0/1177884515 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0003450 con 0x55a0a6e43f20 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.441+0000 7f70ddffb640 1 -- 172.21.15.59:0/1177884515 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e0021070 con 0x55a0a6e43f20 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.442+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a0a6e2b840 con 0x55a0a6e43f20 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.442+0000 7f70ddffb640 1 -- 172.21.15.59:0/1177884515 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0003810 con 0x55a0a6e43f20 2024-03-06T07:28:24.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.442+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a0a6e2be00 con 0x55a0a6e43f20 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.443+0000 7f70ddffb640 1 -- 172.21.15.59:0/1177884515 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70e001b070 con 0x55a0a6e43f20 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.443+0000 7f70ddffb640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f70c8069eb0 0x7f70c806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.443+0000 7f70ddffb640 1 -- 172.21.15.59:0/1177884515 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f70e008a150 con 0x55a0a6e43f20 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.443+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x55a0a6e28360 0x55a0a6e3dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.443+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55a0a6e3e1c0 con 0x55a0a6e28360 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.443+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f70c8069eb0 0x7f70c806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.444+0000 7f70dffff640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x55a0a6e28360 0x55a0a6e3dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.444+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f70c8069eb0 0x7f70c806c370 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f70d8057000 tx=0x7f70d8051000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.444+0000 7f70dffff640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x55a0a6e28360 0x55a0a6e3dc80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.445+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.7 v2:172.21.15.164:6808/3612340295 1 ==== osd_op_reply(1 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x55a0a6e28360 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.445+0000 7f70dffff640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7f70d0006380 0x7f70d0006800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.445+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55a0a6e3e1c0 con 0x7f70d0006380 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.445+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7f70d0006380 0x7f70d0006800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.445+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7f70d0006380 0x7f70d0006800 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.445+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.1 v2:172.21.15.59:6816/1773892168 1 ==== osd_op_reply(2 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0006380 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.445+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f70d80603f0 0x7f70d8060830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.445+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55a0a6e3e1c0 con 0x7f70d80603f0 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.445+0000 7f70dffff640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f70d80603f0 0x7f70d8060830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.446+0000 7f70dffff640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f70d80603f0 0x7f70d8060830 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.446+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.3 v2:172.21.15.59:6808/3602142643 1 ==== osd_op_reply(3 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d80603f0 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.446+0000 7f70dffff640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f70d001c950 0x7f70d00193b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.446+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55a0a6e3e1c0 con 0x7f70d001c950 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.446+0000 7f70e56ce640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f70d001c950 0x7f70d00193b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.446+0000 7f70e56ce640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f70d001c950 0x7f70d00193b0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.447+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 1 ==== osd_op_reply(4 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.447+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55a0a6e3e1c0 con 0x55a0a6e28360 2024-03-06T07:28:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.447+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.7 v2:172.21.15.164:6808/3612340295 2 ==== osd_op_reply(5 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x55a0a6e28360 2024-03-06T07:28:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.447+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55a0a6e3e1c0 con 0x7f70d80603f0 2024-03-06T07:28:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.448+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.3 v2:172.21.15.59:6808/3602142643 2 ==== osd_op_reply(6 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d80603f0 2024-03-06T07:28:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.448+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55a0a6e3e1c0 con 0x7f70d001c950 2024-03-06T07:28:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.448+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 2 ==== osd_op_reply(7 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.448+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55a0a6e3e1c0 con 0x55a0a6e28360 2024-03-06T07:28:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.448+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.7 v2:172.21.15.164:6808/3612340295 3 ==== osd_op_reply(8 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x55a0a6e28360 2024-03-06T07:28:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.448+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55a0a6e3e1c0 con 0x7f70d001c950 2024-03-06T07:28:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.448+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 3 ==== osd_op_reply(9 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.449+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55a0a6e3e1c0 con 0x7f70d80603f0 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.449+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.3 v2:172.21.15.59:6808/3602142643 3 ==== osd_op_reply(10 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d80603f0 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.449+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d80603f0 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.449+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.3 v2:172.21.15.59:6808/3602142643 4 ==== osd_op_reply(11 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d80603f0 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.449+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d001c950 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.449+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 4 ==== osd_op_reply(12 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.449+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x55a0a6e28360 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.450+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.7 v2:172.21.15.164:6808/3612340295 4 ==== osd_op_reply(13 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x55a0a6e28360 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.450+0000 7f70dffff640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f70d0029450 0x7f70d00298d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.450+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0029450 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.450+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f70d0029450 0x7f70d00298d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.450+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f70d0029450 0x7f70d00298d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.451+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.4 v2:172.21.15.164:6818/102497754 1 ==== osd_op_reply(14 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0029450 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.451+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0006380 2024-03-06T07:28:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.451+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.1 v2:172.21.15.59:6816/1773892168 2 ==== osd_op_reply(15 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0006380 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.451+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d001c950 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.451+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 5 ==== osd_op_reply(16 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.451+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0029450 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.452+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.4 v2:172.21.15.164:6818/102497754 2 ==== osd_op_reply(17 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0029450 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.452+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f70d806f3a0 0x7f70d806f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.452+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d806f3a0 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.452+0000 7f70dffff640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f70d806f3a0 0x7f70d806f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.452+0000 7f70dffff640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f70d806f3a0 0x7f70d806f7e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.453+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.5 v2:172.21.15.164:6800/2889679248 1 ==== osd_op_reply(18 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d806f3a0 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.453+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d80603f0 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.453+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.3 v2:172.21.15.59:6808/3602142643 5 ==== osd_op_reply(19 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d80603f0 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.453+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x55a0a6e28360 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.453+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.7 v2:172.21.15.164:6808/3612340295 5 ==== osd_op_reply(20 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x55a0a6e28360 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.453+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x55a0a6e28360 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.453+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.7 v2:172.21.15.164:6808/3612340295 6 ==== osd_op_reply(21 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x55a0a6e28360 2024-03-06T07:28:24.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.453+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d80603f0 2024-03-06T07:28:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.454+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.3 v2:172.21.15.59:6808/3602142643 6 ==== osd_op_reply(22 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d80603f0 2024-03-06T07:28:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.454+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d001c950 2024-03-06T07:28:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.454+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 6 ==== osd_op_reply(23 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.454+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0006380 2024-03-06T07:28:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.454+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.1 v2:172.21.15.59:6816/1773892168 3 ==== osd_op_reply(24 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0006380 2024-03-06T07:28:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.454+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f70d805f860 0x7f70d8006380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.454+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d805f860 2024-03-06T07:28:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.454+0000 7f70e56ce640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f70d805f860 0x7f70d8006380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.455+0000 7f70e56ce640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f70d805f860 0x7f70d8006380 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.455+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.6 v2:172.21.15.164:6816/1594998822 1 ==== osd_op_reply(25 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d805f860 2024-03-06T07:28:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.455+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d001c950 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.456+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 7 ==== osd_op_reply(26 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.456+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d806f3a0 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.456+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.5 v2:172.21.15.164:6800/2889679248 2 ==== osd_op_reply(27 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d806f3a0 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.456+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d001c950 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.456+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 8 ==== osd_op_reply(28 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.456+0000 7f70e56ce640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f70e0058be0 0x7f70e0057840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.456+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70e0058be0 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.456+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f70e0058be0 0x7f70e0057840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.457+0000 7f70e4ecd640 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f70e0058be0 0x7f70e0057840 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.457+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.2 v2:172.21.15.59:6800/1020996386 1 ==== osd_op_reply(29 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70e0058be0 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.457+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d805f860 2024-03-06T07:28:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.457+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.6 v2:172.21.15.164:6816/1594998822 2 ==== osd_op_reply(30 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d805f860 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.457+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0006380 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.457+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.1 v2:172.21.15.59:6816/1773892168 4 ==== osd_op_reply(31 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0006380 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.458+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0029450 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.458+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.4 v2:172.21.15.164:6818/102497754 3 ==== osd_op_reply(32 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0029450 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.458+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d001c950 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.458+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 9 ==== osd_op_reply(33 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.458+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d001c950 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.458+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 10 ==== osd_op_reply(34 [pgnls start_epoch 114 out=49b] v108'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.458+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d80603f0 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.459+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.3 v2:172.21.15.59:6808/3602142643 7 ==== osd_op_reply(35 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d80603f0 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.459+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0006380 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.459+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.1 v2:172.21.15.59:6816/1773892168 5 ==== osd_op_reply(36 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0006380 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.459+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d80603f0 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.459+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.3 v2:172.21.15.59:6808/3602142643 8 ==== osd_op_reply(37 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d80603f0 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.459+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0029450 2024-03-06T07:28:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.460+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.4 v2:172.21.15.164:6818/102497754 4 ==== osd_op_reply(38 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0029450 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.460+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d806f3a0 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.460+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.5 v2:172.21.15.164:6800/2889679248 3 ==== osd_op_reply(39 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d806f3a0 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.460+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d806f3a0 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.460+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.5 v2:172.21.15.164:6800/2889679248 4 ==== osd_op_reply(40 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d806f3a0 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.460+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d805f860 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.461+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.6 v2:172.21.15.164:6816/1594998822 3 ==== osd_op_reply(41 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d805f860 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.461+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0006380 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.461+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.1 v2:172.21.15.59:6816/1773892168 6 ==== osd_op_reply(42 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0006380 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.461+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d806f3a0 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.461+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.5 v2:172.21.15.164:6800/2889679248 5 ==== osd_op_reply(43 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d806f3a0 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.461+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0029450 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.462+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.4 v2:172.21.15.164:6818/102497754 5 ==== osd_op_reply(44 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0029450 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.462+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x55a0a6e28360 2024-03-06T07:28:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.462+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.7 v2:172.21.15.164:6808/3612340295 7 ==== osd_op_reply(45 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x55a0a6e28360 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.462+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x55a0a6e28360 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.462+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.7 v2:172.21.15.164:6808/3612340295 8 ==== osd_op_reply(46 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x55a0a6e28360 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.462+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x55a0a6e28360 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.463+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.7 v2:172.21.15.164:6808/3612340295 9 ==== osd_op_reply(47 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x55a0a6e28360 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.463+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x55a0a6e28360 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.463+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.7 v2:172.21.15.164:6808/3612340295 10 ==== osd_op_reply(48 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x55a0a6e28360 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.463+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0029450 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.463+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.4 v2:172.21.15.164:6818/102497754 6 ==== osd_op_reply(49 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0029450 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.463+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0029450 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.464+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.4 v2:172.21.15.164:6818/102497754 7 ==== osd_op_reply(50 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0029450 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.464+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d80603f0 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.464+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.3 v2:172.21.15.59:6808/3602142643 9 ==== osd_op_reply(51 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x7f70d80603f0 2024-03-06T07:28:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.464+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x55a0a6e28360 2024-03-06T07:28:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.464+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 <== osd.7 v2:172.21.15.164:6808/3612340295 11 ==== osd_op_reply(52 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d00099b0 con 0x55a0a6e28360 2024-03-06T07:28:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.464+0000 7f70dffff640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d001c950 2024-03-06T07:28:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.465+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 11 ==== osd_op_reply(53 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.465+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0029450 2024-03-06T07:28:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.465+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.4 v2:172.21.15.164:6818/102497754 8 ==== osd_op_reply(54 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0029450 2024-03-06T07:28:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.465+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0029450 2024-03-06T07:28:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.465+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.4 v2:172.21.15.164:6818/102497754 9 ==== osd_op_reply(55 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0029450 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.465+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70e0058be0 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.465+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.2 v2:172.21.15.59:6800/1020996386 2 ==== osd_op_reply(56 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70e0058be0 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.465+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d001c950 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.466+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 12 ==== osd_op_reply(57 [pgnls start_epoch 114 out=49b] v114'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.466+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0029450 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.466+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.4 v2:172.21.15.164:6818/102497754 10 ==== osd_op_reply(58 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0029450 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.466+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d805f860 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.466+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.6 v2:172.21.15.164:6816/1594998822 4 ==== osd_op_reply(59 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d805f860 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.466+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d001c950 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.467+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 13 ==== osd_op_reply(60 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.467+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0006380 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.467+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.1 v2:172.21.15.59:6816/1773892168 7 ==== osd_op_reply(61 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0006380 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.467+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d0029450 2024-03-06T07:28:24.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.467+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 <== osd.4 v2:172.21.15.164:6818/102497754 11 ==== osd_op_reply(62 [pgnls start_epoch 114 out=49b] v114'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70d804dc80 con 0x7f70d0029450 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.467+0000 7f70e4ecd640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d805f860 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.6 v2:172.21.15.164:6816/1594998822 5 ==== osd_op_reply(63 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d805f860 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7f70d0028040 con 0x7f70d001c950 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 <== osd.0 v2:172.21.15.59:6824/841983189 14 ==== osd_op_reply(64 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f70e000f030 con 0x7f70d001c950 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f70d001c950 msgr2=0x7f70d00193b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f70d001c950 0x7f70d00193b0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7f70d0006380 msgr2=0x7f70d0006800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7f70d0006380 0x7f70d0006800 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f70e0058be0 msgr2=0x7f70e0057840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f70e0058be0 0x7f70e0057840 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f70d80603f0 msgr2=0x7f70d8060830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 reap_dead start 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f70d80603f0 0x7f70d8060830 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f70d0029450 msgr2=0x7f70d00298d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7f70d0029450 0x7f70d00298d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f70d806f3a0 msgr2=0x7f70d806f7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f70d806f3a0 0x7f70d806f7e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f70d805f860 msgr2=0x7f70d8006380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f70d805f860 0x7f70d8006380 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x55a0a6e28360 msgr2=0x55a0a6e3dc80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x55a0a6e28360 0x55a0a6e3dc80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e56ce640 1 -- 172.21.15.59:0/1177884515 reap_dead start 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f70c8069eb0 msgr2=0x7f70c806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f70c8069eb0 0x7f70c806c370 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f70d8057000 tx=0x7f70d8051000 comp rx=0 tx=0).stop 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 msgr2=0x55a0a6d9b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6d9b800 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f70e0001b00 tx=0x7f70e0000f40 comp rx=0 tx=0).stop 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 shutdown_connections 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f70c8069eb0 0x7f70c806c370 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 --2- 172.21.15.59:0/1177884515 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55a0a6e43f20 0x55a0a6d9b800 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 >> 172.21.15.59:0/1177884515 conn(0x55a0a6d6d8a0 msgr2=0x55a0a6e467c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 shutdown_connections 2024-03-06T07:28:24.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.468+0000 7f70e7c48f40 1 -- 172.21.15.59:0/1177884515 wait complete. 2024-03-06T07:28:24.478 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-03-06T07:28:24.478 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:24.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.614+0000 7fbef0a64640 1 Processor -- start 2024-03-06T07:28:24.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.614+0000 7fbef0a64640 1 -- start start 2024-03-06T07:28:24.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbef0a64640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbeec15dfe0 0x7fbeec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbef0a64640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbeec0cf830 0x7fbeec0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbef0a64640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeec0b3430 con 0x7fbeec0d27f0 2024-03-06T07:28:24.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbef0a64640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeec0b31e0 con 0x7fbeec0cf830 2024-03-06T07:28:24.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbef0a64640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeec0b3d60 con 0x7fbeec15dfe0 2024-03-06T07:28:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbee9d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbeec0cf830 0x7fbeec0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbee9d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbeec0cf830 0x7fbeec0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51936/0 (socket says 172.21.15.59:51936) 2024-03-06T07:28:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbee9d9b640 1 -- 172.21.15.59:0/1328539004 learned_addr learned my addr 172.21.15.59:0/1328539004 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbeea59c640 1 --2- 172.21.15.59:0/1328539004 >> v2:172.21.15.164:3300/0 conn(0x7fbeec15dfe0 0x7fbeec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbee9d9b640 1 -- 172.21.15.59:0/1328539004 >> v2:172.21.15.164:3300/0 conn(0x7fbeec15dfe0 msgr2=0x7fbeec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbee9d9b640 1 --2- 172.21.15.59:0/1328539004 >> v2:172.21.15.164:3300/0 conn(0x7fbeec15dfe0 0x7fbeec0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbee9d9b640 1 -- 172.21.15.59:0/1328539004 >> v1:172.21.15.164:6789/0 conn(0x7fbeec0d27f0 legacy=0x7fbeec0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbee9d9b640 1 -- 172.21.15.59:0/1328539004 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec0b3710 con 0x7fbeec0cf830 2024-03-06T07:28:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.615+0000 7fbeea59c640 1 --2- 172.21.15.59:0/1328539004 >> v2:172.21.15.164:3300/0 conn(0x7fbeec15dfe0 0x7fbeec0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.616+0000 7fbee9d9b640 1 --2- 172.21.15.59:0/1328539004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec0cf830 0x7fbeec0d22b0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009870 tx=0x7fbee002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcaad5cdf4cd0643 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.616+0000 7fbee959a640 1 -- 172.21.15.59:0/1328539004 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee0035020 con 0x7fbeec0cf830 2024-03-06T07:28:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.616+0000 7fbee959a640 1 -- 172.21.15.59:0/1328539004 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee002d040 con 0x7fbeec0cf830 2024-03-06T07:28:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.616+0000 7fbee959a640 1 -- 172.21.15.59:0/1328539004 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee0031a80 con 0x7fbeec0cf830 2024-03-06T07:28:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.617+0000 7fbef0a64640 1 -- 172.21.15.59:0/1328539004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec0cf830 msgr2=0x7fbeec0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.617+0000 7fbef0a64640 1 --2- 172.21.15.59:0/1328539004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec0cf830 0x7fbeec0d22b0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009870 tx=0x7fbee002b900 comp rx=0 tx=0).stop 2024-03-06T07:28:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.617+0000 7fbef0a64640 1 -- 172.21.15.59:0/1328539004 shutdown_connections 2024-03-06T07:28:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.617+0000 7fbef0a64640 1 --2- 172.21.15.59:0/1328539004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec0cf830 0x7fbeec0d22b0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.617+0000 7fbef0a64640 1 --2- 172.21.15.59:0/1328539004 >> v2:172.21.15.164:3300/0 conn(0x7fbeec15dfe0 0x7fbeec0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.617+0000 7fbef0a64640 1 -- 172.21.15.59:0/1328539004 >> 172.21.15.59:0/1328539004 conn(0x7fbeec0d3cd0 msgr2=0x7fbeec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.617+0000 7fbef0a64640 1 -- 172.21.15.59:0/1328539004 shutdown_connections 2024-03-06T07:28:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.617+0000 7fbef0a64640 1 -- 172.21.15.59:0/1328539004 wait complete. 2024-03-06T07:28:24.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.618+0000 7fbef0a64640 1 Processor -- start 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.618+0000 7fbef0a64640 1 -- start start 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbef0a64640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec0cf830 0x7fbeec15bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbef0a64640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbeec0d27f0 0x7fbeec160230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbef0a64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbeec15dfe0 0x7fbeec0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbef0a64640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec0d1df0 con 0x7fbeec0cf830 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbef0a64640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec0d1c70 con 0x7fbeec0d27f0 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbef0a64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec15e3c0 con 0x7fbeec15dfe0 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbeea59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec0cf830 0x7fbeec15bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbeea59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec0cf830 0x7fbeec15bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51948/0 (socket says 172.21.15.59:51948) 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbeea59c640 1 -- 172.21.15.59:0/3457524522 learned_addr learned my addr 172.21.15.59:0/3457524522 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbee9d9b640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbeec0d27f0 0x7fbeec160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbeead9d640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbeec15dfe0 0x7fbeec0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbeea59c640 1 -- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbeec15dfe0 msgr2=0x7fbeec0ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbeea59c640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbeec15dfe0 0x7fbeec0ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbeea59c640 1 -- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbeec0d27f0 msgr2=0x7fbeec160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbeea59c640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbeec0d27f0 0x7fbeec160230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbeea59c640 1 -- 172.21.15.59:0/3457524522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec1f3e90 con 0x7fbeec0cf830 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbeead9d640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbeec15dfe0 0x7fbeec0ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbeea59c640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec0cf830 0x7fbeec15bd80 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00c7f0 tx=0x7fbedc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbedaffd640 1 -- 172.21.15.59:0/3457524522 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc015070 con 0x7fbeec0cf830 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.619+0000 7fbedaffd640 1 -- 172.21.15.59:0/3457524522 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbedc0079d0 con 0x7fbeec0cf830 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.620+0000 7fbef0a64640 1 -- 172.21.15.59:0/3457524522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeec1f4130 con 0x7fbeec0cf830 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.620+0000 7fbedaffd640 1 -- 172.21.15.59:0/3457524522 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc007da0 con 0x7fbeec0cf830 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.620+0000 7fbef0a64640 1 -- 172.21.15.59:0/3457524522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeec1f46f0 con 0x7fbeec0cf830 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.620+0000 7fbef0a64640 1 -- 172.21.15.59:0/3457524522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeec04f330 con 0x7fbeec0cf830 2024-03-06T07:28:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.621+0000 7fbedaffd640 1 -- 172.21.15.59:0/3457524522 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbedc018070 con 0x7fbeec0cf830 2024-03-06T07:28:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.622+0000 7fbedaffd640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbecc06b4d0 0x7fbecc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.622+0000 7fbedaffd640 1 -- 172.21.15.59:0/3457524522 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fbedc08a090 con 0x7fbeec0cf830 2024-03-06T07:28:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.622+0000 7fbee9d9b640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbecc06b4d0 0x7fbecc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.622+0000 7fbee9d9b640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbecc06b4d0 0x7fbecc06d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbee0036000 tx=0x7fbee003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.626+0000 7fbedaffd640 1 -- 172.21.15.59:0/3457524522 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbedc0588c0 con 0x7fbeec0cf830 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.847+0000 7fbef0a64640 1 -- 172.21.15.59:0/3457524522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbeec0b09c0 con 0x7fbeec0cf830 2024-03-06T07:28:24.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.848+0000 7fbedaffd640 1 -- 172.21.15.59:0/3457524522 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fbedc05c960 con 0x7fbeec0cf830 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:24.850 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 -- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbecc06b4d0 msgr2=0x7fbecc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbecc06b4d0 0x7fbecc06d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbee0036000 tx=0x7fbee003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 -- 172.21.15.59:0/3457524522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec0cf830 msgr2=0x7fbeec15bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec0cf830 0x7fbeec15bd80 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00c7f0 tx=0x7fbedc00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 -- 172.21.15.59:0/3457524522 shutdown_connections 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbecc06b4d0 0x7fbecc06d990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbeec15dfe0 0x7fbeec0ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbeec0d27f0 0x7fbeec160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 --2- 172.21.15.59:0/3457524522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbeec0cf830 0x7fbeec15bd80 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 -- 172.21.15.59:0/3457524522 >> 172.21.15.59:0/3457524522 conn(0x7fbeec0d3cd0 msgr2=0x7fbeec0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 -- 172.21.15.59:0/3457524522 shutdown_connections 2024-03-06T07:28:24.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:24.850+0000 7fbef0a64640 1 -- 172.21.15.59:0/3457524522 wait complete. 2024-03-06T07:28:24.863 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:28:24.979 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_created":3}} 2024-03-06T07:28:24.979 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-03-06T07:28:24.979 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:25.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.079+0000 7faf28bd0640 1 Processor -- start 2024-03-06T07:28:25.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.079+0000 7faf28bd0640 1 -- start start 2024-03-06T07:28:25.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.080+0000 7faf28bd0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faf241597c0 0x7faf2415cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.080+0000 7faf28bd0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf2406f290 0x7faf2406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.080+0000 7faf28bd0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf240cd440 con 0x7faf2406b4f0 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.080+0000 7faf28bd0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf24159ba0 con 0x7faf241597c0 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.080+0000 7faf28bd0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2415c7d0 con 0x7faf2406f290 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.080+0000 7faf22d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7faf2406b4f0 0x7faf240d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58986/0 (socket says 172.21.15.59:58986) 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.080+0000 7faf22d9d640 1 -- 172.21.15.59:0/1586464446 learned_addr learned my addr 172.21.15.59:0/1586464446 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.080+0000 7faf2259c640 1 --2- 172.21.15.59:0/1586464446 >> v2:172.21.15.164:3300/0 conn(0x7faf241597c0 0x7faf2415cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.080+0000 7faf1bfff640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf2406f290 0x7faf2406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.080+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703362786 0 0) 0x7faf240cd440 con 0x7faf2406b4f0 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.080+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf10003520 con 0x7faf2406b4f0 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3152564496 0 0) 0x7faf10003520 con 0x7faf2406b4f0 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf2406f290 msgr2=0x7faf2406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf2406f290 0x7faf2406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 >> v2:172.21.15.164:3300/0 conn(0x7faf241597c0 msgr2=0x7faf2415cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 --2- 172.21.15.59:0/1586464446 >> v2:172.21.15.164:3300/0 conn(0x7faf241597c0 0x7faf2415cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf240cd720 con 0x7faf2406b4f0 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7faf1c0019f0 con 0x7faf2406b4f0 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 >> v1:172.21.15.59:6789/0 conn(0x7faf2406b4f0 legacy=0x7faf240d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf100039b0 0x7faf10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf10004800 0x7faf10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf2406f290 0x7faf100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2415c7d0 con 0x7faf10004800 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1000b9c0 con 0x7faf100039b0 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1000bb40 con 0x7faf2406f290 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf2259c640 1 --2- 172.21.15.59:0/1586464446 >> v2:172.21.15.164:3300/0 conn(0x7faf241597c0 0x7faf2415cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf1bfff640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf10004800 0x7faf10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf22d9d640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf2406f290 0x7faf100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf2259c640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf100039b0 0x7faf10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf1bfff640 1 -- 172.21.15.59:0/1586464446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf2406f290 msgr2=0x7faf100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf1bfff640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf2406f290 0x7faf100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf1bfff640 1 -- 172.21.15.59:0/1586464446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf100039b0 msgr2=0x7faf10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf1bfff640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf100039b0 0x7faf10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.081+0000 7faf1bfff640 1 -- 172.21.15.59:0/1586464446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf10003520 con 0x7faf10004800 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.082+0000 7faf1bfff640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf10004800 0x7faf10004c80 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7faf0c009ac0 tx=0x7faf10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d32dab6b6b6311 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.082+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c039070 con 0x7faf10004800 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.082+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c02c6b0 con 0x7faf10004800 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.082+0000 7faf21d9b640 1 -- 172.21.15.59:0/1586464446 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c009690 con 0x7faf10004800 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.083+0000 7faf28bd0640 1 -- 172.21.15.59:0/1586464446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf10004800 msgr2=0x7faf10004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.083+0000 7faf28bd0640 1 --2- 172.21.15.59:0/1586464446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf10004800 0x7faf10004c80 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7faf0c009ac0 tx=0x7faf10005830 comp rx=0 tx=0).stop 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.083+0000 7faf22d9d640 1 -- 172.21.15.59:0/1586464446 reap_dead start 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.083+0000 7faf28bd0640 1 -- 172.21.15.59:0/1586464446 shutdown_connections 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.083+0000 7faf28bd0640 1 -- 172.21.15.59:0/1586464446 >> 172.21.15.59:0/1586464446 conn(0x7faf2406c0c0 msgr2=0x7faf2406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.083+0000 7faf28bd0640 1 -- 172.21.15.59:0/1586464446 shutdown_connections 2024-03-06T07:28:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.083+0000 7faf28bd0640 1 -- 172.21.15.59:0/1586464446 wait complete. 2024-03-06T07:28:25.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.085+0000 7faf28bd0640 1 Processor -- start 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.085+0000 7faf28bd0640 1 -- start start 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf28bd0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf10004800 0x7faf2414f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf28bd0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2415dbb0 0x7faf2415e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf28bd0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf24150b80 0x7faf24151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf28bd0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf240cd0c0 con 0x7faf2415dbb0 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf28bd0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2415c150 con 0x7faf10004800 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf28bd0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf240728d0 con 0x7faf24150b80 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf1bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2415dbb0 0x7faf2415e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf1bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2415dbb0 0x7faf2415e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51964/0 (socket says 172.21.15.59:51964) 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf1bfff640 1 -- 172.21.15.59:0/3444518521 learned_addr learned my addr 172.21.15.59:0/3444518521 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf2259c640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf10004800 0x7faf2414f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf1bfff640 1 -- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf24150b80 msgr2=0x7faf24151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf1bfff640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf24150b80 0x7faf24151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf1bfff640 1 -- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf10004800 msgr2=0x7faf2414f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf1bfff640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf10004800 0x7faf2414f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf1bfff640 1 -- 172.21.15.59:0/3444518521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf241f16c0 con 0x7faf2415dbb0 2024-03-06T07:28:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.086+0000 7faf1bfff640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2415dbb0 0x7faf2415e030 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7faf0c037040 tx=0x7faf0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.087+0000 7faf1affd640 1 -- 172.21.15.59:0/3444518521 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c039050 con 0x7faf2415dbb0 2024-03-06T07:28:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.087+0000 7faf1affd640 1 -- 172.21.15.59:0/3444518521 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c045070 con 0x7faf2415dbb0 2024-03-06T07:28:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.087+0000 7faf28bd0640 1 -- 172.21.15.59:0/3444518521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf241f19a0 con 0x7faf2415dbb0 2024-03-06T07:28:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.087+0000 7faf1affd640 1 -- 172.21.15.59:0/3444518521 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c003850 con 0x7faf2415dbb0 2024-03-06T07:28:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.087+0000 7faf28bd0640 1 -- 172.21.15.59:0/3444518521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf241f1f60 con 0x7faf2415dbb0 2024-03-06T07:28:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.088+0000 7faf28bd0640 1 -- 172.21.15.59:0/3444518521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf2404f330 con 0x7faf2415dbb0 2024-03-06T07:28:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.088+0000 7faf1affd640 1 -- 172.21.15.59:0/3444518521 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf0c0039f0 con 0x7faf2415dbb0 2024-03-06T07:28:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.089+0000 7faf1affd640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf0406b4d0 0x7faf0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.089+0000 7faf1affd640 1 -- 172.21.15.59:0/3444518521 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7faf0c0abd30 con 0x7faf2415dbb0 2024-03-06T07:28:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.089+0000 7faf2259c640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf0406b4d0 0x7faf0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.090+0000 7faf2259c640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf0406b4d0 0x7faf0406d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7faf241f5a60 tx=0x7faf14008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.093+0000 7faf1affd640 1 -- 172.21.15.59:0/3444518521 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf0c04e020 con 0x7faf2415dbb0 2024-03-06T07:28:25.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.312+0000 7faf28bd0640 1 -- 172.21.15.59:0/3444518521 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf2406c8e0 con 0x7faf2415dbb0 2024-03-06T07:28:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.313+0000 7faf1affd640 1 -- 172.21.15.59:0/3444518521 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7faf0c07abe0 con 0x7faf2415dbb0 2024-03-06T07:28:25.315 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:25.315 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 -- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf0406b4d0 msgr2=0x7faf0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf0406b4d0 0x7faf0406d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7faf241f5a60 tx=0x7faf14008040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 -- 172.21.15.59:0/3444518521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2415dbb0 msgr2=0x7faf2415e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2415dbb0 0x7faf2415e030 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7faf0c037040 tx=0x7faf0c004730 comp rx=0 tx=0).stop 2024-03-06T07:28:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 -- 172.21.15.59:0/3444518521 shutdown_connections 2024-03-06T07:28:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf0406b4d0 0x7faf0406d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf24150b80 0x7faf24151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf2415dbb0 0x7faf2415e030 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 --2- 172.21.15.59:0/3444518521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf10004800 0x7faf2414f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 -- 172.21.15.59:0/3444518521 >> 172.21.15.59:0/3444518521 conn(0x7faf2406c0c0 msgr2=0x7faf24159e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 -- 172.21.15.59:0/3444518521 shutdown_connections 2024-03-06T07:28:25.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.315+0000 7faf28bd0640 1 -- 172.21.15.59:0/3444518521 wait complete. 2024-03-06T07:28:25.329 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:25.447 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:28:25.448 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:28:25.448 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:25.448 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:25.495 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:25.545 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:25.545 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:28:25.545 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:25.561 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.699+0000 7f6130b0c640 1 Processor -- start 2024-03-06T07:28:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.700+0000 7f6130b0c640 1 -- start start 2024-03-06T07:28:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.700+0000 7f6130b0c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f612c0d22b0 0x7f612c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.700+0000 7f6130b0c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c0b41f0 0x7f612c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.700+0000 7f6130b0c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c0b3430 con 0x7f612c15dfe0 2024-03-06T07:28:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.700+0000 7f6130b0c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f612c0d2c50 con 0x7f612c0d22b0 2024-03-06T07:28:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.700+0000 7f6130b0c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0b31e0 con 0x7f612c0b41f0 2024-03-06T07:28:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.700+0000 7f612a59c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f612c15dfe0 0x7f612c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58994/0 (socket says 172.21.15.59:58994) 2024-03-06T07:28:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.700+0000 7f612a59c640 1 -- 172.21.15.59:0/3516863813 learned_addr learned my addr 172.21.15.59:0/3516863813 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635270568 0 0) 0x7f612c0b3430 con 0x7f612c15dfe0 2024-03-06T07:28:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6121d9b640 1 --2- 172.21.15.59:0/3516863813 >> v2:172.21.15.164:3300/0 conn(0x7f612c0d22b0 0x7f612c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6118003520 con 0x7f612c15dfe0 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f612ad9d640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c0b41f0 0x7f612c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 737077023 0 0) 0x7f6118003520 con 0x7f612c15dfe0 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c0b41f0 msgr2=0x7f612c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c0b41f0 0x7f612c0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 >> v2:172.21.15.164:3300/0 conn(0x7f612c0d22b0 msgr2=0x7f612c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 --2- 172.21.15.59:0/3516863813 >> v2:172.21.15.164:3300/0 conn(0x7f612c0d22b0 0x7f612c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c0b3710 con 0x7f612c15dfe0 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f611c001670 con 0x7f612c15dfe0 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 >> v1:172.21.15.59:6789/0 conn(0x7f612c15dfe0 legacy=0x7f612c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61180039b0 0x7f6118003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6118004800 0x7f6118004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c0b41f0 0x7f61180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0b31e0 con 0x7f61180039b0 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611800b9c0 con 0x7f6118004800 2024-03-06T07:28:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611800bb40 con 0x7f612c0b41f0 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f612a59c640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61180039b0 0x7f6118003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f6121d9b640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6118004800 0x7f6118004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f612ad9d640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c0b41f0 0x7f61180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f612a59c640 1 -- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c0b41f0 msgr2=0x7f61180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f612a59c640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c0b41f0 0x7f61180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f612a59c640 1 -- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6118004800 msgr2=0x7f6118004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f612a59c640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6118004800 0x7f6118004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f612a59c640 1 -- 172.21.15.59:0/3516863813 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6118003520 con 0x7f61180039b0 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.701+0000 7f612ad9d640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c0b41f0 0x7f61180051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.702+0000 7f612a59c640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61180039b0 0x7f6118003dd0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f611c007fd0 tx=0x7f6118005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b112c312c941df51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.702+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c007d50 con 0x7f61180039b0 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.702+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611c030070 con 0x7f61180039b0 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.702+0000 7f6129d9b640 1 -- 172.21.15.59:0/3516863813 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c02cd40 con 0x7f61180039b0 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.703+0000 7f6130b0c640 1 -- 172.21.15.59:0/3516863813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61180039b0 msgr2=0x7f6118003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.703+0000 7f6130b0c640 1 --2- 172.21.15.59:0/3516863813 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61180039b0 0x7f6118003dd0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f611c007fd0 tx=0x7f6118005830 comp rx=0 tx=0).stop 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.703+0000 7f612ad9d640 1 -- 172.21.15.59:0/3516863813 reap_dead start 2024-03-06T07:28:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.703+0000 7f6130b0c640 1 -- 172.21.15.59:0/3516863813 shutdown_connections 2024-03-06T07:28:25.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.703+0000 7f6130b0c640 1 -- 172.21.15.59:0/3516863813 >> 172.21.15.59:0/3516863813 conn(0x7f612c0d3cd0 msgr2=0x7f612c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.703+0000 7f6130b0c640 1 -- 172.21.15.59:0/3516863813 shutdown_connections 2024-03-06T07:28:25.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.703+0000 7f6130b0c640 1 -- 172.21.15.59:0/3516863813 wait complete. 2024-03-06T07:28:25.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.705+0000 7f6130b0c640 1 Processor -- start 2024-03-06T07:28:25.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.705+0000 7f6130b0c640 1 -- start start 2024-03-06T07:28:25.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6130b0c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61180039b0 0x7f612c15edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6130b0c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f612c160490 0x7f612c0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6130b0c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c15c5f0 0x7f612c06bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6130b0c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0cfa00 con 0x7f612c160490 2024-03-06T07:28:25.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6130b0c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0b3d60 con 0x7f61180039b0 2024-03-06T07:28:25.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6130b0c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f612c0d1c70 con 0x7f612c15c5f0 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6121d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f612c160490 0x7f612c0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6121d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f612c160490 0x7f612c0ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51990/0 (socket says 172.21.15.59:51990) 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6121d9b640 1 -- 172.21.15.59:0/2585222963 learned_addr learned my addr 172.21.15.59:0/2585222963 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f612ad9d640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c15c5f0 0x7f612c06bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f612a59c640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61180039b0 0x7f612c15edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6121d9b640 1 -- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c15c5f0 msgr2=0x7f612c06bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6121d9b640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c15c5f0 0x7f612c06bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6121d9b640 1 -- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61180039b0 msgr2=0x7f612c15edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6121d9b640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61180039b0 0x7f612c15edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6121d9b640 1 -- 172.21.15.59:0/2585222963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c0cd450 con 0x7f612c160490 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6121d9b640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f612c160490 0x7f612c0ccf10 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6114011990 tx=0x7f6114011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f61237fe640 1 -- 172.21.15.59:0/2585222963 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611400fa30 con 0x7f612c160490 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f61237fe640 1 -- 172.21.15.59:0/2585222963 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611400fbd0 con 0x7f612c160490 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f6130b0c640 1 -- 172.21.15.59:0/2585222963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c15f3f0 con 0x7f612c160490 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.706+0000 7f61237fe640 1 -- 172.21.15.59:0/2585222963 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6114014440 con 0x7f612c160490 2024-03-06T07:28:25.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.707+0000 7f6130b0c640 1 -- 172.21.15.59:0/2585222963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c15fc90 con 0x7f612c160490 2024-03-06T07:28:25.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.708+0000 7f6130b0c640 1 -- 172.21.15.59:0/2585222963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c04f330 con 0x7f612c160490 2024-03-06T07:28:25.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.708+0000 7f61237fe640 1 -- 172.21.15.59:0/2585222963 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6114012040 con 0x7f612c160490 2024-03-06T07:28:25.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.708+0000 7f61237fe640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f610c06b500 0x7f610c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.708+0000 7f61237fe640 1 -- 172.21.15.59:0/2585222963 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f611408a740 con 0x7f612c160490 2024-03-06T07:28:25.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.709+0000 7f612a59c640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f610c06b500 0x7f610c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.709+0000 7f612a59c640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f610c06b500 0x7f610c06d9c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f611c034000 tx=0x7f611c007000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.713+0000 7f61237fe640 1 -- 172.21.15.59:0/2585222963 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6114019030 con 0x7f612c160490 2024-03-06T07:28:25.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.896+0000 7f6130b0c640 1 -- 172.21.15.59:0/2585222963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f612c0d0e30 con 0x7f612c160490 2024-03-06T07:28:25.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.897+0000 7f61237fe640 1 -- 172.21.15.59:0/2585222963 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16843 (secure 0 0 0) 0x7f61140594a0 con 0x7f612c160490 2024-03-06T07:28:25.900 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:25.900 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":114,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:27:38.442474+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:12.625799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:13.306376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.25","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:6836/2139041464":"2024-03-07T07:27:27.121928+0000","172.21.15.164:6834/2139041464":"2024-03-07T07:27:27.121928+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.899+0000 7f6130b0c640 1 -- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f610c06b500 msgr2=0x7f610c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.900+0000 7f6130b0c640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f610c06b500 0x7f610c06d9c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f611c034000 tx=0x7f611c007000 comp rx=0 tx=0).stop 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.900+0000 7f6130b0c640 1 -- 172.21.15.59:0/2585222963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f612c160490 msgr2=0x7f612c0ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.900+0000 7f6130b0c640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f612c160490 0x7f612c0ccf10 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6114011990 tx=0x7f6114011d90 comp rx=0 tx=0).stop 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.900+0000 7f6130b0c640 1 -- 172.21.15.59:0/2585222963 shutdown_connections 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.900+0000 7f6130b0c640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f610c06b500 0x7f610c06d9c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.900+0000 7f6130b0c640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f612c15c5f0 0x7f612c06bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.900+0000 7f6130b0c640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f612c160490 0x7f612c0ccf10 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.900+0000 7f6130b0c640 1 --2- 172.21.15.59:0/2585222963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61180039b0 0x7f612c15edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.900+0000 7f6130b0c640 1 -- 172.21.15.59:0/2585222963 >> 172.21.15.59:0/2585222963 conn(0x7f612c0d3cd0 msgr2=0x7f612c0cf060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.900+0000 7f6130b0c640 1 -- 172.21.15.59:0/2585222963 shutdown_connections 2024-03-06T07:28:25.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:25.900+0000 7f6130b0c640 1 -- 172.21.15.59:0/2585222963 wait complete. 2024-03-06T07:28:25.913 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:12.625799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:13.306376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:25.913 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:26.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.018+0000 7fe704555640 1 Processor -- start 2024-03-06T07:28:26.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.018+0000 7fe704555640 1 -- start start 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.019+0000 7fe704555640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe6fc1614d0 0x7fe6fc0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.019+0000 7fe704555640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d1910 0x7fe6fc0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.019+0000 7fe704555640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6fc0d2740 con 0x7fe6fc0d22b0 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.019+0000 7fe704555640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6fc1618b0 con 0x7fe6fc1614d0 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.019+0000 7fe704555640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0cf400 con 0x7fe6fc0d1910 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.019+0000 7fe7022f1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe6fc1614d0 0x7fe6fc0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.019+0000 7fe7022f1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe6fc1614d0 0x7fe6fc0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52006/0 (socket says 172.21.15.59:52006) 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.019+0000 7fe7022f1640 1 -- 172.21.15.59:0/790648810 learned_addr learned my addr 172.21.15.59:0/790648810 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.020+0000 7fe7022f1640 1 -- 172.21.15.59:0/790648810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d1910 msgr2=0x7fe6fc0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.020+0000 7fe701af0640 1 --2- 172.21.15.59:0/790648810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d1910 0x7fe6fc0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.020+0000 7fe7022f1640 1 --2- 172.21.15.59:0/790648810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d1910 0x7fe6fc0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.020+0000 7fe7022f1640 1 -- 172.21.15.59:0/790648810 >> v1:172.21.15.164:6789/0 conn(0x7fe6fc0d22b0 legacy=0x7fe6fc0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.020+0000 7fe7022f1640 1 -- 172.21.15.59:0/790648810 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc0cf6a0 con 0x7fe6fc1614d0 2024-03-06T07:28:26.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.020+0000 7fe701af0640 1 --2- 172.21.15.59:0/790648810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d1910 0x7fe6fc0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.020+0000 7fe7022f1640 1 --2- 172.21.15.59:0/790648810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6fc1614d0 0x7fe6fc0d4860 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009870 tx=0x7fe6f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44cb10374c00220f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.021+0000 7fe7012ef640 1 -- 172.21.15.59:0/790648810 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8035020 con 0x7fe6fc1614d0 2024-03-06T07:28:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.021+0000 7fe7012ef640 1 -- 172.21.15.59:0/790648810 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f8031640 con 0x7fe6fc1614d0 2024-03-06T07:28:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.021+0000 7fe7012ef640 1 -- 172.21.15.59:0/790648810 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f802d040 con 0x7fe6fc1614d0 2024-03-06T07:28:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.021+0000 7fe704555640 1 -- 172.21.15.59:0/790648810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6fc1614d0 msgr2=0x7fe6fc0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.021+0000 7fe704555640 1 --2- 172.21.15.59:0/790648810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6fc1614d0 0x7fe6fc0d4860 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009870 tx=0x7fe6f802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.021+0000 7fe704555640 1 -- 172.21.15.59:0/790648810 shutdown_connections 2024-03-06T07:28:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.021+0000 7fe704555640 1 --2- 172.21.15.59:0/790648810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d1910 0x7fe6fc0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.021+0000 7fe704555640 1 --2- 172.21.15.59:0/790648810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6fc1614d0 0x7fe6fc0d4860 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.021+0000 7fe704555640 1 -- 172.21.15.59:0/790648810 >> 172.21.15.59:0/790648810 conn(0x7fe6fc0d3d40 msgr2=0x7fe6fc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.021+0000 7fe704555640 1 -- 172.21.15.59:0/790648810 shutdown_connections 2024-03-06T07:28:26.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.021+0000 7fe704555640 1 -- 172.21.15.59:0/790648810 wait complete. 2024-03-06T07:28:26.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.023+0000 7fe704555640 1 Processor -- start 2024-03-06T07:28:26.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.023+0000 7fe704555640 1 -- start start 2024-03-06T07:28:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe704555640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6fc0d1910 0x7fe6fc0cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe704555640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d22b0 0x7fe6fc14e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe704555640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6fc1614d0 0x7fe6fc15ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe704555640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0b0910 con 0x7fe6fc1614d0 2024-03-06T07:28:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe704555640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0cfc60 con 0x7fe6fc0d1910 2024-03-06T07:28:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe704555640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0b1e40 con 0x7fe6fc0d22b0 2024-03-06T07:28:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe702af2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6fc1614d0 0x7fe6fc15ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe702af2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6fc1614d0 0x7fe6fc15ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52016/0 (socket says 172.21.15.59:52016) 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe702af2640 1 -- 172.21.15.59:0/2721111735 learned_addr learned my addr 172.21.15.59:0/2721111735 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe7022f1640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6fc0d1910 0x7fe6fc0cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe701af0640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d22b0 0x7fe6fc14e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe702af2640 1 -- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d22b0 msgr2=0x7fe6fc14e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe702af2640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d22b0 0x7fe6fc14e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe702af2640 1 -- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6fc0d1910 msgr2=0x7fe6fc0cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe702af2640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6fc0d1910 0x7fe6fc0cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe702af2640 1 -- 172.21.15.59:0/2721111735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc0cd710 con 0x7fe6fc1614d0 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe701af0640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d22b0 0x7fe6fc14e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.024+0000 7fe702af2640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6fc1614d0 0x7fe6fc15ea30 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400c780 tx=0x7fe6f400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.025+0000 7fe6f2ffd640 1 -- 172.21.15.59:0/2721111735 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4004b60 con 0x7fe6fc1614d0 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.025+0000 7fe6f2ffd640 1 -- 172.21.15.59:0/2721111735 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f4004d00 con 0x7fe6fc1614d0 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.025+0000 7fe704555640 1 -- 172.21.15.59:0/2721111735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6fc0cd9f0 con 0x7fe6fc1614d0 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.025+0000 7fe6f2ffd640 1 -- 172.21.15.59:0/2721111735 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f400aa90 con 0x7fe6fc1614d0 2024-03-06T07:28:26.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.025+0000 7fe704555640 1 -- 172.21.15.59:0/2721111735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6fc151f50 con 0x7fe6fc1614d0 2024-03-06T07:28:26.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.026+0000 7fe704555640 1 -- 172.21.15.59:0/2721111735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6fc068ff0 con 0x7fe6fc1614d0 2024-03-06T07:28:26.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.026+0000 7fe6f2ffd640 1 -- 172.21.15.59:0/2721111735 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6f400ac30 con 0x7fe6fc1614d0 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.027+0000 7fe6f2ffd640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe6e006b500 0x7fe6e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.027+0000 7fe6f2ffd640 1 -- 172.21.15.59:0/2721111735 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe6f408a420 con 0x7fe6fc1614d0 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.027+0000 7fe7022f1640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe6e006b500 0x7fe6e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.027+0000 7fe7022f1640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe6e006b500 0x7fe6e006d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8036000 tx=0x7fe6f803c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.031+0000 7fe6f2ffd640 1 -- 172.21.15.59:0/2721111735 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6f4020070 con 0x7fe6fc1614d0 2024-03-06T07:28:26.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.210+0000 7fe704555640 1 -- 172.21.15.59:0/2721111735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6fc0c7260 con 0x7fe6fc1614d0 2024-03-06T07:28:26.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.210+0000 7fe6f2ffd640 1 -- 172.21.15.59:0/2721111735 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe6f4059200 con 0x7fe6fc1614d0 2024-03-06T07:28:26.212 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:28:26.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 -- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe6e006b500 msgr2=0x7fe6e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe6e006b500 0x7fe6e006d9c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8036000 tx=0x7fe6f803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 -- 172.21.15.59:0/2721111735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6fc1614d0 msgr2=0x7fe6fc15ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6fc1614d0 0x7fe6fc15ea30 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400c780 tx=0x7fe6f400cc40 comp rx=0 tx=0).stop 2024-03-06T07:28:26.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 -- 172.21.15.59:0/2721111735 shutdown_connections 2024-03-06T07:28:26.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe6e006b500 0x7fe6e006d9c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe6fc1614d0 0x7fe6fc15ea30 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6fc0d22b0 0x7fe6fc14e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 --2- 172.21.15.59:0/2721111735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6fc0d1910 0x7fe6fc0cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 -- 172.21.15.59:0/2721111735 >> 172.21.15.59:0/2721111735 conn(0x7fe6fc0d3d40 msgr2=0x7fe6fc06cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 -- 172.21.15.59:0/2721111735 shutdown_connections 2024-03-06T07:28:26.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.212+0000 7fe704555640 1 -- 172.21.15.59:0/2721111735 wait complete. 2024-03-06T07:28:26.225 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:26.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.327+0000 7f9d7c5a1640 1 Processor -- start 2024-03-06T07:28:26.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.327+0000 7f9d7c5a1640 1 -- start start 2024-03-06T07:28:26.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.328+0000 7f9d7c5a1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9d741614d0 0x7f9d740d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.328+0000 7f9d7c5a1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9d740d1910 0x7f9d740d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.328+0000 7f9d7c5a1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d741618b0 con 0x7f9d740d22e0 2024-03-06T07:28:26.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.328+0000 7f9d7c5a1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d740cf400 con 0x7f9d741614d0 2024-03-06T07:28:26.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.328+0000 7f9d7c5a1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d740d2710 con 0x7f9d740d1910 2024-03-06T07:28:26.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.328+0000 7f9d7a33d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9d741614d0 0x7f9d740d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.328+0000 7f9d7a33d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9d741614d0 0x7f9d740d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52032/0 (socket says 172.21.15.59:52032) 2024-03-06T07:28:26.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.328+0000 7f9d7a33d640 1 -- 172.21.15.59:0/3463197620 learned_addr learned my addr 172.21.15.59:0/3463197620 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:26.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.329+0000 7f9d79b3c640 1 --2- 172.21.15.59:0/3463197620 >> v2:172.21.15.164:3300/0 conn(0x7f9d740d1910 0x7f9d740d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.329+0000 7f9d7a33d640 1 -- 172.21.15.59:0/3463197620 >> v2:172.21.15.164:3300/0 conn(0x7f9d740d1910 msgr2=0x7f9d740d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.329+0000 7f9d7a33d640 1 --2- 172.21.15.59:0/3463197620 >> v2:172.21.15.164:3300/0 conn(0x7f9d740d1910 0x7f9d740d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.329+0000 7f9d7a33d640 1 -- 172.21.15.59:0/3463197620 >> v1:172.21.15.164:6789/0 conn(0x7f9d740d22e0 legacy=0x7f9d740d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.329+0000 7f9d7a33d640 1 -- 172.21.15.59:0/3463197620 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d740cf6a0 con 0x7f9d741614d0 2024-03-06T07:28:26.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.330+0000 7f9d7a33d640 1 --2- 172.21.15.59:0/3463197620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d741614d0 0x7f9d740d4860 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f9d7408c470 tx=0x7f9d7002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0fe70473d1cdf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.330+0000 7f9d7933b640 1 -- 172.21.15.59:0/3463197620 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70035020 con 0x7f9d741614d0 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.330+0000 7f9d7933b640 1 -- 172.21.15.59:0/3463197620 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d70031640 con 0x7f9d741614d0 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.330+0000 7f9d7933b640 1 -- 172.21.15.59:0/3463197620 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7002d040 con 0x7f9d741614d0 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.330+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/3463197620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d741614d0 msgr2=0x7f9d740d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.330+0000 7f9d7c5a1640 1 --2- 172.21.15.59:0/3463197620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d741614d0 0x7f9d740d4860 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f9d7408c470 tx=0x7f9d7002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.331+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/3463197620 shutdown_connections 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.331+0000 7f9d7c5a1640 1 --2- 172.21.15.59:0/3463197620 >> v2:172.21.15.164:3300/0 conn(0x7f9d740d1910 0x7f9d740d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.331+0000 7f9d7c5a1640 1 --2- 172.21.15.59:0/3463197620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d741614d0 0x7f9d740d4860 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.331+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/3463197620 >> 172.21.15.59:0/3463197620 conn(0x7f9d740d3d40 msgr2=0x7f9d740d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.331+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/3463197620 shutdown_connections 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.331+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/3463197620 wait complete. 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.332+0000 7f9d7c5a1640 1 Processor -- start 2024-03-06T07:28:26.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.332+0000 7f9d7c5a1640 1 -- start start 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7c5a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d740d1910 0x7f9d7415f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7c5a1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d740d22e0 0x7f9d741f6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7c5a1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d741614d0 0x7f9d741f7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7c5a1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740b0910 con 0x7f9d741614d0 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7c5a1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740cfc60 con 0x7f9d740d1910 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7c5a1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d740b1e40 con 0x7f9d740d22e0 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7ab3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d741614d0 0x7f9d741f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7ab3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d741614d0 0x7f9d741f7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52048/0 (socket says 172.21.15.59:52048) 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7ab3e640 1 -- 172.21.15.59:0/486603816 learned_addr learned my addr 172.21.15.59:0/486603816 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:26.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7a33d640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d740d1910 0x7f9d7415f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d79b3c640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d740d22e0 0x7f9d741f6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7ab3e640 1 -- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d740d22e0 msgr2=0x7f9d741f6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7ab3e640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d740d22e0 0x7f9d741f6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7ab3e640 1 -- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d740d1910 msgr2=0x7f9d7415f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7ab3e640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d740d1910 0x7f9d7415f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7ab3e640 1 -- 172.21.15.59:0/486603816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d740cc130 con 0x7f9d741614d0 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7ab3e640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d741614d0 0x7f9d741f7180 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c780 tx=0x7f9d6c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d6affd640 1 -- 172.21.15.59:0/486603816 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c004b60 con 0x7f9d741614d0 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d6affd640 1 -- 172.21.15.59:0/486603816 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c004d00 con 0x7f9d741614d0 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d6affd640 1 -- 172.21.15.59:0/486603816 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00aa90 con 0x7f9d741614d0 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/486603816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d740c8680 con 0x7f9d741614d0 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.333+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/486603816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d740cc3d0 con 0x7f9d741614d0 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.335+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/486603816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d740d05b0 con 0x7f9d741614d0 2024-03-06T07:28:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.335+0000 7f9d6affd640 1 -- 172.21.15.59:0/486603816 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d6c00ac30 con 0x7f9d741614d0 2024-03-06T07:28:26.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.335+0000 7f9d6affd640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d5806b4d0 0x7f9d5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.335+0000 7f9d6affd640 1 -- 172.21.15.59:0/486603816 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9d6c08a2f0 con 0x7f9d741614d0 2024-03-06T07:28:26.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.335+0000 7f9d7a33d640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d5806b4d0 0x7f9d5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.336+0000 7f9d7a33d640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d5806b4d0 0x7f9d5806d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9d70036000 tx=0x7f9d7003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.339+0000 7f9d6affd640 1 -- 172.21.15.59:0/486603816 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d740d05b0 con 0x7f9d741614d0 2024-03-06T07:28:26.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.521+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/486603816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9d740c8ba0 con 0x7f9d741614d0 2024-03-06T07:28:26.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.522+0000 7f9d6affd640 1 -- 172.21.15.59:0/486603816 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9d6c020070 con 0x7f9d741614d0 2024-03-06T07:28:26.524 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:28:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d5806b4d0 msgr2=0x7f9d5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d5806b4d0 0x7f9d5806d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9d70036000 tx=0x7f9d7003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/486603816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d741614d0 msgr2=0x7f9d741f7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d741614d0 0x7f9d741f7180 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c780 tx=0x7f9d6c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:28:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/486603816 shutdown_connections 2024-03-06T07:28:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d5806b4d0 0x7f9d5806d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d741614d0 0x7f9d741f7180 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d740d22e0 0x7f9d741f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 --2- 172.21.15.59:0/486603816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d740d1910 0x7f9d7415f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/486603816 >> 172.21.15.59:0/486603816 conn(0x7f9d740d3d40 msgr2=0x7f9d740cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/486603816 shutdown_connections 2024-03-06T07:28:26.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.524+0000 7f9d7c5a1640 1 -- 172.21.15.59:0/486603816 wait complete. 2024-03-06T07:28:26.537 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:26.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.639+0000 7f7cd417a640 1 Processor -- start 2024-03-06T07:28:26.641 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.639+0000 7f7cd417a640 1 -- start start 2024-03-06T07:28:26.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.641+0000 7f7cd417a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ccc1564f0 0x7f7ccc06b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.643 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.641+0000 7f7cd417a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ccc15faa0 0x7f7ccc15ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.641+0000 7f7cd417a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc1568d0 con 0x7f7ccc06c8a0 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.641+0000 7f7cd417a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc0673f0 con 0x7f7ccc15faa0 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.641+0000 7f7cd417a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc06cd20 con 0x7f7ccc1564f0 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.641+0000 7f7cd2717640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ccc15faa0 0x7f7ccc15ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.641+0000 7f7cd1f16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ccc1564f0 0x7f7ccc06b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.641+0000 7f7cd1715640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7ccc06c8a0 0x7f7ccc0cd630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:44526/0 (socket says 172.21.15.59:44526) 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.641+0000 7f7cd2717640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ccc15faa0 0x7f7ccc15ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:45358/0 (socket says 172.21.15.59:45358) 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.641+0000 7f7cd1715640 1 -- 172.21.15.59:0/1911385568 learned_addr learned my addr 172.21.15.59:0/1911385568 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.642+0000 7f7cd1f16640 1 -- 172.21.15.59:0/1911385568 >> v2:172.21.15.164:3300/0 conn(0x7f7ccc15faa0 msgr2=0x7f7ccc15ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.642+0000 7f7cd1f16640 1 --2- 172.21.15.59:0/1911385568 >> v2:172.21.15.164:3300/0 conn(0x7f7ccc15faa0 0x7f7ccc15ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.642+0000 7f7cd1f16640 1 -- 172.21.15.59:0/1911385568 >> v1:172.21.15.164:6789/0 conn(0x7f7ccc06c8a0 legacy=0x7f7ccc0cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.642+0000 7f7cd0f14640 1 -- 172.21.15.59:0/1911385568 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224150938 0 0) 0x7f7ccc1568d0 con 0x7f7ccc06c8a0 2024-03-06T07:28:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.642+0000 7f7cd1f16640 1 -- 172.21.15.59:0/1911385568 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc0af260 con 0x7f7ccc1564f0 2024-03-06T07:28:26.645 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.642+0000 7f7cd2717640 1 --2- 172.21.15.59:0/1911385568 >> v2:172.21.15.164:3300/0 conn(0x7f7ccc15faa0 0x7f7ccc15ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.643+0000 7f7cd1f16640 1 --2- 172.21.15.59:0/1911385568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ccc1564f0 0x7f7ccc06b090 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80097a0 tx=0x7f7cc802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3883fd0260a0277e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.643+0000 7f7cd0f14640 1 -- 172.21.15.59:0/1911385568 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8035020 con 0x7f7ccc1564f0 2024-03-06T07:28:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.643+0000 7f7cd0f14640 1 -- 172.21.15.59:0/1911385568 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc8031830 con 0x7f7ccc1564f0 2024-03-06T07:28:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.643+0000 7f7cd0f14640 1 -- 172.21.15.59:0/1911385568 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8031c00 con 0x7f7ccc1564f0 2024-03-06T07:28:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.643+0000 7f7cd417a640 1 -- 172.21.15.59:0/1911385568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ccc1564f0 msgr2=0x7f7ccc06b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.643+0000 7f7cd417a640 1 --2- 172.21.15.59:0/1911385568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ccc1564f0 0x7f7ccc06b090 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7cc80097a0 tx=0x7f7cc802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.644+0000 7f7cd417a640 1 -- 172.21.15.59:0/1911385568 shutdown_connections 2024-03-06T07:28:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.644+0000 7f7cd417a640 1 --2- 172.21.15.59:0/1911385568 >> v2:172.21.15.164:3300/0 conn(0x7f7ccc15faa0 0x7f7ccc15ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.644+0000 7f7cd417a640 1 --2- 172.21.15.59:0/1911385568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ccc1564f0 0x7f7ccc06b090 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.644+0000 7f7cd417a640 1 -- 172.21.15.59:0/1911385568 >> 172.21.15.59:0/1911385568 conn(0x7f7ccc0b3c70 msgr2=0x7f7ccc0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.644+0000 7f7cd417a640 1 -- 172.21.15.59:0/1911385568 shutdown_connections 2024-03-06T07:28:26.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.644+0000 7f7cd417a640 1 -- 172.21.15.59:0/1911385568 wait complete. 2024-03-06T07:28:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.645+0000 7f7cd417a640 1 Processor -- start 2024-03-06T07:28:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.645+0000 7f7cd417a640 1 -- start start 2024-03-06T07:28:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd417a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ccc06c8a0 0x7f7ccc0d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd417a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ccc1564f0 0x7f7ccc1f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd417a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ccc15faa0 0x7f7ccc14a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd417a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc066da0 con 0x7f7ccc15faa0 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd417a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0af820 con 0x7f7ccc06c8a0 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd417a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc067080 con 0x7f7ccc1564f0 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd2717640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ccc15faa0 0x7f7ccc14a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd2717640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ccc15faa0 0x7f7ccc14a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52052/0 (socket says 172.21.15.59:52052) 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd2717640 1 -- 172.21.15.59:0/3609405965 learned_addr learned my addr 172.21.15.59:0/3609405965 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd1715640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ccc1564f0 0x7f7ccc1f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd1f16640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ccc06c8a0 0x7f7ccc0d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd2717640 1 -- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ccc1564f0 msgr2=0x7f7ccc1f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd2717640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ccc1564f0 0x7f7ccc1f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd2717640 1 -- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ccc06c8a0 msgr2=0x7f7ccc0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd2717640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ccc06c8a0 0x7f7ccc0d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd2717640 1 -- 172.21.15.59:0/3609405965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc14a850 con 0x7f7ccc15faa0 2024-03-06T07:28:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd2717640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ccc15faa0 0x7f7ccc14a310 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400efc0 tx=0x7f7cc400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cc27fc640 1 -- 172.21.15.59:0/3609405965 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4018030 con 0x7f7ccc15faa0 2024-03-06T07:28:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cc27fc640 1 -- 172.21.15.59:0/3609405965 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc400cdf0 con 0x7f7ccc15faa0 2024-03-06T07:28:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cd417a640 1 -- 172.21.15.59:0/3609405965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc1f44d0 con 0x7f7ccc15faa0 2024-03-06T07:28:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.646+0000 7f7cc27fc640 1 -- 172.21.15.59:0/3609405965 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4003bb0 con 0x7f7ccc15faa0 2024-03-06T07:28:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.647+0000 7f7cd417a640 1 -- 172.21.15.59:0/3609405965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc1f4a90 con 0x7f7ccc15faa0 2024-03-06T07:28:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.647+0000 7f7cd417a640 1 -- 172.21.15.59:0/3609405965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc09e3d0 con 0x7f7ccc15faa0 2024-03-06T07:28:26.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.648+0000 7f7cc27fc640 1 -- 172.21.15.59:0/3609405965 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7cc401b070 con 0x7f7ccc15faa0 2024-03-06T07:28:26.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.648+0000 7f7cc27fc640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7cb006b4d0 0x7f7cb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.648+0000 7f7cc27fc640 1 -- 172.21.15.59:0/3609405965 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7cc408aaa0 con 0x7f7ccc15faa0 2024-03-06T07:28:26.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.649+0000 7f7cd1f16640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7cb006b4d0 0x7f7cb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.649+0000 7f7cd1f16640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7cb006b4d0 0x7f7cb006d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8036000 tx=0x7f7cc803e040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.653+0000 7f7cc27fc640 1 -- 172.21.15.59:0/3609405965 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ccc14a850 con 0x7f7ccc15faa0 2024-03-06T07:28:26.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.831+0000 7f7cd417a640 1 -- 172.21.15.59:0/3609405965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7ccc068ff0 con 0x7f7ccc15faa0 2024-03-06T07:28:26.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.832+0000 7f7cc27fc640 1 -- 172.21.15.59:0/3609405965 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7cc4010040 con 0x7f7ccc15faa0 2024-03-06T07:28:26.834 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 -- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7cb006b4d0 msgr2=0x7f7cb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7cb006b4d0 0x7f7cb006d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8036000 tx=0x7f7cc803e040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 -- 172.21.15.59:0/3609405965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ccc15faa0 msgr2=0x7f7ccc14a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ccc15faa0 0x7f7ccc14a310 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400efc0 tx=0x7f7cc400c490 comp rx=0 tx=0).stop 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 -- 172.21.15.59:0/3609405965 shutdown_connections 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7cb006b4d0 0x7f7cb006d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ccc15faa0 0x7f7ccc14a310 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ccc1564f0 0x7f7ccc1f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 --2- 172.21.15.59:0/3609405965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ccc06c8a0 0x7f7ccc0d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 -- 172.21.15.59:0/3609405965 >> 172.21.15.59:0/3609405965 conn(0x7f7ccc0b3c70 msgr2=0x7f7ccc0b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 -- 172.21.15.59:0/3609405965 shutdown_connections 2024-03-06T07:28:26.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.834+0000 7f7cd417a640 1 -- 172.21.15.59:0/3609405965 wait complete. 2024-03-06T07:28:26.847 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:26.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.947+0000 7f9d44b7a640 1 Processor -- start 2024-03-06T07:28:26.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.947+0000 7f9d44b7a640 1 -- start start 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.948+0000 7f9d44b7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d400b4460 0x7f9d400d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.948+0000 7f9d44b7a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9d40162970 0x7f9d400b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.948+0000 7f9d44b7a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d400d22c0 con 0x7f9d40161cf0 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.948+0000 7f9d44b7a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d40161b00 con 0x7f9d40162970 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.948+0000 7f9d44b7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d400cecc0 con 0x7f9d400b4460 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.948+0000 7f9d3dd9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9d40162970 0x7f9d400b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.948+0000 7f9d3ed9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9d40161cf0 0x7f9d400cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58998/0 (socket says 172.21.15.59:58998) 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.948+0000 7f9d3dd9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9d40162970 0x7f9d400b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52056/0 (socket says 172.21.15.59:52056) 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.948+0000 7f9d3ed9d640 1 -- 172.21.15.59:0/894390081 learned_addr learned my addr 172.21.15.59:0/894390081 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.948+0000 7f9d3e59c640 1 --2- 172.21.15.59:0/894390081 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d400b4460 0x7f9d400d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.949+0000 7f9d3dd9b640 1 -- 172.21.15.59:0/894390081 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d400b4460 msgr2=0x7f9d400d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.949+0000 7f9d3dd9b640 1 --2- 172.21.15.59:0/894390081 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d400b4460 0x7f9d400d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.949+0000 7f9d3dd9b640 1 -- 172.21.15.59:0/894390081 >> v1:172.21.15.59:6789/0 conn(0x7f9d40161cf0 legacy=0x7f9d400cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.949+0000 7f9d3d59a640 1 -- 172.21.15.59:0/894390081 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302708309 0 0) 0x7f9d400d22c0 con 0x7f9d40161cf0 2024-03-06T07:28:26.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.949+0000 7f9d3dd9b640 1 -- 172.21.15.59:0/894390081 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d400d25a0 con 0x7f9d40162970 2024-03-06T07:28:26.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.949+0000 7f9d3dd9b640 1 --2- 172.21.15.59:0/894390081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d40162970 0x7f9d400b3580 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f9d340097a0 tx=0x7f9d3402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9c80e371350d9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.950+0000 7f9d3d59a640 1 -- 172.21.15.59:0/894390081 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34035020 con 0x7f9d40162970 2024-03-06T07:28:26.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.950+0000 7f9d3d59a640 1 -- 172.21.15.59:0/894390081 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d34031830 con 0x7f9d40162970 2024-03-06T07:28:26.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.950+0000 7f9d3d59a640 1 -- 172.21.15.59:0/894390081 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34031c00 con 0x7f9d40162970 2024-03-06T07:28:26.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.950+0000 7f9d44b7a640 1 -- 172.21.15.59:0/894390081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d40162970 msgr2=0x7f9d400b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.950+0000 7f9d44b7a640 1 --2- 172.21.15.59:0/894390081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d40162970 0x7f9d400b3580 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f9d340097a0 tx=0x7f9d3402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.950+0000 7f9d44b7a640 1 -- 172.21.15.59:0/894390081 shutdown_connections 2024-03-06T07:28:26.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.950+0000 7f9d44b7a640 1 --2- 172.21.15.59:0/894390081 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d40162970 0x7f9d400b3580 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.950+0000 7f9d44b7a640 1 --2- 172.21.15.59:0/894390081 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d400b4460 0x7f9d400d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.950+0000 7f9d44b7a640 1 -- 172.21.15.59:0/894390081 >> 172.21.15.59:0/894390081 conn(0x7f9d400d3ce0 msgr2=0x7f9d400d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.950+0000 7f9d44b7a640 1 -- 172.21.15.59:0/894390081 shutdown_connections 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.950+0000 7f9d44b7a640 1 -- 172.21.15.59:0/894390081 wait complete. 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.951+0000 7f9d44b7a640 1 Processor -- start 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.952+0000 7f9d44b7a640 1 -- start start 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.952+0000 7f9d44b7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d400b4460 0x7f9d40156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.952+0000 7f9d44b7a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d40161cf0 0x7f9d400cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.952+0000 7f9d44b7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d40162970 0x7f9d400b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.952+0000 7f9d44b7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d400ce950 con 0x7f9d40162970 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.952+0000 7f9d44b7a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d400d2b60 con 0x7f9d40161cf0 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.952+0000 7f9d44b7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d400b3ac0 con 0x7f9d400b4460 2024-03-06T07:28:26.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.952+0000 7f9d3ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d40162970 0x7f9d400b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.952+0000 7f9d3ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d40162970 0x7f9d400b20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52058/0 (socket says 172.21.15.59:52058) 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.952+0000 7f9d3ed9d640 1 -- 172.21.15.59:0/3644799819 learned_addr learned my addr 172.21.15.59:0/3644799819 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d3e59c640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d400b4460 0x7f9d40156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d3dd9b640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d40161cf0 0x7f9d400cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d3ed9d640 1 -- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d400b4460 msgr2=0x7f9d40156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d3ed9d640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d400b4460 0x7f9d40156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d3ed9d640 1 -- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d40161cf0 msgr2=0x7f9d400cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d3ed9d640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d40161cf0 0x7f9d400cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d3ed9d640 1 -- 172.21.15.59:0/3644799819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d400cb870 con 0x7f9d40162970 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d3dd9b640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d40161cf0 0x7f9d400cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d3e59c640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d400b4460 0x7f9d40156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d3ed9d640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d40162970 0x7f9d400b20b0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800c730 tx=0x7f9d3800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d2effd640 1 -- 172.21.15.59:0/3644799819 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d38018070 con 0x7f9d40162970 2024-03-06T07:28:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d2effd640 1 -- 172.21.15.59:0/3644799819 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3800ce40 con 0x7f9d40162970 2024-03-06T07:28:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d44b7a640 1 -- 172.21.15.59:0/3644799819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d400cbb50 con 0x7f9d40162970 2024-03-06T07:28:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d2effd640 1 -- 172.21.15.59:0/3644799819 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3800a860 con 0x7f9d40162970 2024-03-06T07:28:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.953+0000 7f9d44b7a640 1 -- 172.21.15.59:0/3644799819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4015a780 con 0x7f9d40162970 2024-03-06T07:28:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.954+0000 7f9d44b7a640 1 -- 172.21.15.59:0/3644799819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d400cc070 con 0x7f9d40162970 2024-03-06T07:28:26.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.955+0000 7f9d2effd640 1 -- 172.21.15.59:0/3644799819 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d3801b070 con 0x7f9d40162970 2024-03-06T07:28:26.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.955+0000 7f9d2effd640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d2006b4d0 0x7f9d2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.955+0000 7f9d2effd640 1 -- 172.21.15.59:0/3644799819 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9d38089780 con 0x7f9d40162970 2024-03-06T07:28:26.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.956+0000 7f9d3e59c640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d2006b4d0 0x7f9d2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.956+0000 7f9d3e59c640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d2006b4d0 0x7f9d2006d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9d400b9900 tx=0x7f9d30008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:26.960+0000 7f9d2effd640 1 -- 172.21.15.59:0/3644799819 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d400cc070 con 0x7f9d40162970 2024-03-06T07:28:27.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.156+0000 7f9d44b7a640 1 -- 172.21.15.59:0/3644799819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9d4004f330 con 0x7f9d40162970 2024-03-06T07:28:27.158 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.157+0000 7f9d2effd640 1 -- 172.21.15.59:0/3644799819 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v114) v1 ==== 80+0+124 (secure 0 0 0) 0x7f9d38058600 con 0x7f9d40162970 2024-03-06T07:28:27.159 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.164:6836/2139041464 2024-03-07T07:27:27.121928+0000 2024-03-06T07:28:27.159 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.164:6834/2139041464 2024-03-07T07:27:27.121928+0000 2024-03-06T07:28:27.159 INFO:teuthology.orchestra.run.smithi059.stderr:listed 2 entries 2024-03-06T07:28:27.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.158+0000 7f9d44b7a640 1 -- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d2006b4d0 msgr2=0x7f9d2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.158+0000 7f9d44b7a640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d2006b4d0 0x7f9d2006d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9d400b9900 tx=0x7f9d30008040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.158+0000 7f9d44b7a640 1 -- 172.21.15.59:0/3644799819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d40162970 msgr2=0x7f9d400b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.158+0000 7f9d44b7a640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d40162970 0x7f9d400b20b0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800c730 tx=0x7f9d3800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.158+0000 7f9d44b7a640 1 -- 172.21.15.59:0/3644799819 shutdown_connections 2024-03-06T07:28:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.158+0000 7f9d44b7a640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9d2006b4d0 0x7f9d2006d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.158+0000 7f9d44b7a640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9d40162970 0x7f9d400b20b0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.158+0000 7f9d44b7a640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d40161cf0 0x7f9d400cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.158+0000 7f9d44b7a640 1 --2- 172.21.15.59:0/3644799819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d400b4460 0x7f9d40156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.158+0000 7f9d44b7a640 1 -- 172.21.15.59:0/3644799819 >> 172.21.15.59:0/3644799819 conn(0x7f9d400d3ce0 msgr2=0x7f9d4015fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.159+0000 7f9d44b7a640 1 -- 172.21.15.59:0/3644799819 shutdown_connections 2024-03-06T07:28:27.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.159+0000 7f9d44b7a640 1 -- 172.21.15.59:0/3644799819 wait complete. 2024-03-06T07:28:27.171 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:28:27.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:27.171 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:27.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[74412]: fuse finished with error 0 and tester_r 0 2024-03-06T07:28:27.198 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:28:27.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:27.198 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:28:27.274 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:28:27.274 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:27.293 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:27.293 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:28:27.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:27.293 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:28:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:27.351 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:28:27.351 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:28:27.351 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:27.365 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:27.366 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:28:27.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:27.366 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:28:27.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:28:27.420 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:28:27.421 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:27.438 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:27.438 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:28:27.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:27.438 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:28:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:27.494 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:28:27.495 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:27.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.595+0000 7fcbf6bf6640 1 Processor -- start 2024-03-06T07:28:27.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.595+0000 7fcbf6bf6640 1 -- start start 2024-03-06T07:28:27.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf6bf6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcbf015dfe0 0x7fcbf00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf6bf6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcbf00cf830 0x7fcbf00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf6bf6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf00b3430 con 0x7fcbf00d27f0 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf6bf6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf00b31e0 con 0x7fcbf015dfe0 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf6bf6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf00b3d60 con 0x7fcbf00cf830 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf4992640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcbf015dfe0 0x7fcbf00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf4992640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcbf015dfe0 0x7fcbf00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52066/0 (socket says 172.21.15.59:52066) 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf4992640 1 -- 172.21.15.59:0/3022522028 learned_addr learned my addr 172.21.15.59:0/3022522028 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf5193640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcbf00d27f0 0x7fcbf00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59006/0 (socket says 172.21.15.59:59006) 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbe7fff640 1 --2- 172.21.15.59:0/3022522028 >> v2:172.21.15.164:3300/0 conn(0x7fcbf00cf830 0x7fcbf00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf4992640 1 -- 172.21.15.59:0/3022522028 >> v2:172.21.15.164:3300/0 conn(0x7fcbf00cf830 msgr2=0x7fcbf00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf4992640 1 --2- 172.21.15.59:0/3022522028 >> v2:172.21.15.164:3300/0 conn(0x7fcbf00cf830 0x7fcbf00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf4992640 1 -- 172.21.15.59:0/3022522028 >> v1:172.21.15.59:6789/0 conn(0x7fcbf00d27f0 legacy=0x7fcbf00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbf4992640 1 -- 172.21.15.59:0/3022522028 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf00b3710 con 0x7fcbf015dfe0 2024-03-06T07:28:27.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.596+0000 7fcbe77fe640 1 -- 172.21.15.59:0/3022522028 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221014027 0 0) 0x7fcbf00b3430 con 0x7fcbf00d27f0 2024-03-06T07:28:27.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.597+0000 7fcbf4992640 1 --2- 172.21.15.59:0/3022522028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf015dfe0 0x7fcbf00cf2f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009870 tx=0x7fcbec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4b504aeddf081c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.597+0000 7fcbe77fe640 1 -- 172.21.15.59:0/3022522028 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec034070 con 0x7fcbf015dfe0 2024-03-06T07:28:27.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.597+0000 7fcbe77fe640 1 -- 172.21.15.59:0/3022522028 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec030860 con 0x7fcbf015dfe0 2024-03-06T07:28:27.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.597+0000 7fcbe77fe640 1 -- 172.21.15.59:0/3022522028 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec030c30 con 0x7fcbf015dfe0 2024-03-06T07:28:27.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.598+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/3022522028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf015dfe0 msgr2=0x7fcbf00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.598+0000 7fcbf6bf6640 1 --2- 172.21.15.59:0/3022522028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf015dfe0 0x7fcbf00cf2f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009870 tx=0x7fcbec02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.598+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/3022522028 shutdown_connections 2024-03-06T07:28:27.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.598+0000 7fcbf6bf6640 1 --2- 172.21.15.59:0/3022522028 >> v2:172.21.15.164:3300/0 conn(0x7fcbf00cf830 0x7fcbf00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.598+0000 7fcbf6bf6640 1 --2- 172.21.15.59:0/3022522028 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf015dfe0 0x7fcbf00cf2f0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.598+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/3022522028 >> 172.21.15.59:0/3022522028 conn(0x7fcbf00d3cd0 msgr2=0x7fcbf00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.598+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/3022522028 shutdown_connections 2024-03-06T07:28:27.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.598+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/3022522028 wait complete. 2024-03-06T07:28:27.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.599+0000 7fcbf6bf6640 1 Processor -- start 2024-03-06T07:28:27.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.599+0000 7fcbf6bf6640 1 -- start start 2024-03-06T07:28:27.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf6bf6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf00cf830 0x7fcbf00b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf6bf6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbf00d27f0 0x7fcbf01f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf6bf6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbf015dfe0 0x7fcbf015b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf6bf6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf00d1df0 con 0x7fcbf00cf830 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf6bf6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf00d1c70 con 0x7fcbf00d27f0 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf6bf6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf015e3c0 con 0x7fcbf015dfe0 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf4992640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf00cf830 0x7fcbf00b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf4992640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf00cf830 0x7fcbf00b8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52076/0 (socket says 172.21.15.59:52076) 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf4992640 1 -- 172.21.15.59:0/1639996042 learned_addr learned my addr 172.21.15.59:0/1639996042 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbe7fff640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbf00d27f0 0x7fcbf01f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf5193640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbf015dfe0 0x7fcbf015b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf4992640 1 -- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbf015dfe0 msgr2=0x7fcbf015b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf4992640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbf015dfe0 0x7fcbf015b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf4992640 1 -- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbf00d27f0 msgr2=0x7fcbf01f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf4992640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbf00d27f0 0x7fcbf01f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf4992640 1 -- 172.21.15.59:0/1639996042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf00b95a0 con 0x7fcbf00cf830 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf5193640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbf015dfe0 0x7fcbf015b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbf4992640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf00cf830 0x7fcbf00b8bc0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fcbec000c00 tx=0x7fcbec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.600+0000 7fcbe57fa640 1 -- 172.21.15.59:0/1639996042 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec048070 con 0x7fcbf00cf830 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.601+0000 7fcbe57fa640 1 -- 172.21.15.59:0/1639996042 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbec030660 con 0x7fcbf00cf830 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.601+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/1639996042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf01f6d60 con 0x7fcbf00cf830 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.601+0000 7fcbe57fa640 1 -- 172.21.15.59:0/1639996042 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbec034040 con 0x7fcbf00cf830 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.601+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/1639996042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf01f7320 con 0x7fcbf00cf830 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.602+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/1639996042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf004f330 con 0x7fcbf00cf830 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.602+0000 7fcbe57fa640 1 -- 172.21.15.59:0/1639996042 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbec04b070 con 0x7fcbf00cf830 2024-03-06T07:28:27.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.603+0000 7fcbe57fa640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 0x7fcbdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.603+0000 7fcbe57fa640 1 -- 172.21.15.59:0/1639996042 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fcbec0ac200 con 0x7fcbf00cf830 2024-03-06T07:28:27.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.603+0000 7fcbe7fff640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 0x7fcbdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.604+0000 7fcbe7fff640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 0x7fcbdc06d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0006fd0 tx=0x7fcbe0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.607+0000 7fcbe57fa640 1 -- 172.21.15.59:0/1639996042 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbec07b1a0 con 0x7fcbf00cf830 2024-03-06T07:28:27.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.826+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/1639996042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbf01f7560 con 0x7fcbf00cf830 2024-03-06T07:28:27.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.826+0000 7fcbe57fa640 1 -- 172.21.15.59:0/1639996042 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fcbec030830 con 0x7fcbf00cf830 2024-03-06T07:28:27.828 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:27.828 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:27.829 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:27.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 msgr2=0x7fcbdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 0x7fcbdc06d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0006fd0 tx=0x7fcbe0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/1639996042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf00cf830 msgr2=0x7fcbf00b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf00cf830 0x7fcbf00b8bc0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fcbec000c00 tx=0x7fcbec004730 comp rx=0 tx=0).stop 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/1639996042 shutdown_connections 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbdc06b4d0 0x7fcbdc06d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbf015dfe0 0x7fcbf015b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbf00d27f0 0x7fcbf01f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 --2- 172.21.15.59:0/1639996042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbf00cf830 0x7fcbf00b8bc0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/1639996042 >> 172.21.15.59:0/1639996042 conn(0x7fcbf00d3cd0 msgr2=0x7fcbf00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/1639996042 shutdown_connections 2024-03-06T07:28:27.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.828+0000 7fcbf6bf6640 1 -- 172.21.15.59:0/1639996042 wait complete. 2024-03-06T07:28:27.841 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:27.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.941+0000 7f71d82df640 1 Processor -- start 2024-03-06T07:28:27.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.941+0000 7f71d82df640 1 -- start start 2024-03-06T07:28:27.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d82df640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f71d015dfe0 0x7f71d00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d82df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71d00d29c0 0x7f71d00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d82df640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d00b3430 con 0x7f71d00cf830 2024-03-06T07:28:27.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d82df640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d00b31e0 con 0x7f71d015dfe0 2024-03-06T07:28:27.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d82df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d00b3d60 con 0x7f71d00d29c0 2024-03-06T07:28:27.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d607b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f71d015dfe0 0x7f71d00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d607b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f71d015dfe0 0x7f71d00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52092/0 (socket says 172.21.15.59:52092) 2024-03-06T07:28:27.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d607b640 1 -- 172.21.15.59:0/2032008073 learned_addr learned my addr 172.21.15.59:0/2032008073 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:27.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d687c640 1 --2- 172.21.15.59:0/2032008073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71d00d29c0 0x7f71d00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d607b640 1 -- 172.21.15.59:0/2032008073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71d00d29c0 msgr2=0x7f71d00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d607b640 1 --2- 172.21.15.59:0/2032008073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71d00d29c0 0x7f71d00b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d607b640 1 -- 172.21.15.59:0/2032008073 >> v1:172.21.15.164:6789/0 conn(0x7f71d00cf830 legacy=0x7f71d00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.942+0000 7f71d607b640 1 -- 172.21.15.59:0/2032008073 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d00b3710 con 0x7f71d015dfe0 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.943+0000 7f71d607b640 1 --2- 172.21.15.59:0/2032008073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71d015dfe0 0x7f71d00cf2f0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f71cc009870 tx=0x7f71cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4d64061c910dc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.943+0000 7f71d5079640 1 -- 172.21.15.59:0/2032008073 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc034070 con 0x7f71d015dfe0 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.943+0000 7f71d5079640 1 -- 172.21.15.59:0/2032008073 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc030840 con 0x7f71d015dfe0 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.943+0000 7f71d5079640 1 -- 172.21.15.59:0/2032008073 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc030c10 con 0x7f71d015dfe0 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.944+0000 7f71d82df640 1 -- 172.21.15.59:0/2032008073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71d015dfe0 msgr2=0x7f71d00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.944+0000 7f71d82df640 1 --2- 172.21.15.59:0/2032008073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71d015dfe0 0x7f71d00cf2f0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f71cc009870 tx=0x7f71cc02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.944+0000 7f71d82df640 1 -- 172.21.15.59:0/2032008073 shutdown_connections 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.944+0000 7f71d82df640 1 --2- 172.21.15.59:0/2032008073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71d00d29c0 0x7f71d00b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.944+0000 7f71d82df640 1 --2- 172.21.15.59:0/2032008073 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71d015dfe0 0x7f71d00cf2f0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.944+0000 7f71d82df640 1 -- 172.21.15.59:0/2032008073 >> 172.21.15.59:0/2032008073 conn(0x7f71d00d3cd0 msgr2=0x7f71d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.944+0000 7f71d82df640 1 -- 172.21.15.59:0/2032008073 shutdown_connections 2024-03-06T07:28:27.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.944+0000 7f71d82df640 1 -- 172.21.15.59:0/2032008073 wait complete. 2024-03-06T07:28:27.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.945+0000 7f71d82df640 1 Processor -- start 2024-03-06T07:28:27.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.945+0000 7f71d82df640 1 -- start start 2024-03-06T07:28:27.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d82df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71d00cf830 0x7f71d015ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d82df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71d00d29c0 0x7f71d00ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d82df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71d015dfe0 0x7f71d00d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d82df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d00d1df0 con 0x7f71d015dfe0 2024-03-06T07:28:27.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d82df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d00d1c70 con 0x7f71d00d29c0 2024-03-06T07:28:27.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d82df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d015e3c0 con 0x7f71d00cf830 2024-03-06T07:28:27.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d687c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71d015dfe0 0x7f71d00d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d687c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71d015dfe0 0x7f71d00d02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52098/0 (socket says 172.21.15.59:52098) 2024-03-06T07:28:27.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d687c640 1 -- 172.21.15.59:0/265427034 learned_addr learned my addr 172.21.15.59:0/265427034 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:27.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d607b640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71d00cf830 0x7f71d015ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d587a640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71d00d29c0 0x7f71d00ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d687c640 1 -- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71d00cf830 msgr2=0x7f71d015ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d687c640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71d00cf830 0x7f71d015ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d687c640 1 -- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71d00d29c0 msgr2=0x7f71d00ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d687c640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71d00d29c0 0x7f71d00ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d687c640 1 -- 172.21.15.59:0/265427034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d015b3d0 con 0x7f71d015dfe0 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d587a640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71d00d29c0 0x7f71d00ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.946+0000 7f71d687c640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71d015dfe0 0x7f71d00d02c0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f71c800efc0 tx=0x7f71c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.947+0000 7f71c6ffd640 1 -- 172.21.15.59:0/265427034 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8018020 con 0x7f71d015dfe0 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.947+0000 7f71c6ffd640 1 -- 172.21.15.59:0/265427034 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c8014480 con 0x7f71d015dfe0 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.947+0000 7f71d82df640 1 -- 172.21.15.59:0/265427034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d00caf80 con 0x7f71d015dfe0 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.947+0000 7f71c6ffd640 1 -- 172.21.15.59:0/265427034 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c8014870 con 0x7f71d015dfe0 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.947+0000 7f71d82df640 1 -- 172.21.15.59:0/265427034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d00cb540 con 0x7f71d015dfe0 2024-03-06T07:28:27.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.948+0000 7f71d82df640 1 -- 172.21.15.59:0/265427034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d0068090 con 0x7f71d015dfe0 2024-03-06T07:28:27.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.949+0000 7f71c6ffd640 1 -- 172.21.15.59:0/265427034 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71c8005d10 con 0x7f71d015dfe0 2024-03-06T07:28:27.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.949+0000 7f71c6ffd640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71b406b4d0 0x7f71b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.949+0000 7f71c6ffd640 1 -- 172.21.15.59:0/265427034 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f71c808aff0 con 0x7f71d015dfe0 2024-03-06T07:28:27.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.949+0000 7f71d607b640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71b406b4d0 0x7f71b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.950+0000 7f71d607b640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71b406b4d0 0x7f71b406d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f71cc036000 tx=0x7f71cc03b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:27.954+0000 7f71c6ffd640 1 -- 172.21.15.59:0/265427034 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71d0068090 con 0x7f71d015dfe0 2024-03-06T07:28:28.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.135+0000 7f71d82df640 1 -- 172.21.15.59:0/265427034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71d004f330 con 0x7f71d015dfe0 2024-03-06T07:28:28.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.137+0000 7f71c6ffd640 1 -- 172.21.15.59:0/265427034 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16843 (secure 0 0 0) 0x7f71c8022030 con 0x7f71d015dfe0 2024-03-06T07:28:28.139 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:28.139 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":114,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:27:38.442474+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:12.625799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:13.306376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.25","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:6836/2139041464":"2024-03-07T07:27:27.121928+0000","172.21.15.164:6834/2139041464":"2024-03-07T07:27:27.121928+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:28.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 -- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71b406b4d0 msgr2=0x7f71b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71b406b4d0 0x7f71b406d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f71cc036000 tx=0x7f71cc03b040 comp rx=0 tx=0).stop 2024-03-06T07:28:28.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 -- 172.21.15.59:0/265427034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71d015dfe0 msgr2=0x7f71d00d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71d015dfe0 0x7f71d00d02c0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f71c800efc0 tx=0x7f71c800c490 comp rx=0 tx=0).stop 2024-03-06T07:28:28.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 -- 172.21.15.59:0/265427034 shutdown_connections 2024-03-06T07:28:28.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71b406b4d0 0x7f71b406d990 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71d015dfe0 0x7f71d00d02c0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71d00d29c0 0x7f71d00ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 --2- 172.21.15.59:0/265427034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71d00cf830 0x7f71d015ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 -- 172.21.15.59:0/265427034 >> 172.21.15.59:0/265427034 conn(0x7f71d00d3cd0 msgr2=0x7f71d006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 -- 172.21.15.59:0/265427034 shutdown_connections 2024-03-06T07:28:28.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.139+0000 7f71d82df640 1 -- 172.21.15.59:0/265427034 wait complete. 2024-03-06T07:28:28.154 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:12.625799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:13.306376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:28.155 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:28.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.257+0000 7f39d8772640 1 Processor -- start 2024-03-06T07:28:28.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.257+0000 7f39d8772640 1 -- start start 2024-03-06T07:28:28.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.258+0000 7f39d8772640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f39d00b1e40 0x7f39d0168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.258+0000 7f39d8772640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d00d2730 con 0x7f39d00d4860 2024-03-06T07:28:28.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.259+0000 7f39d8772640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d01618b0 con 0x7f39d01614d0 2024-03-06T07:28:28.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.259+0000 7f39d8772640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39d00d4cf0 con 0x7f39d00b1e40 2024-03-06T07:28:28.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.259+0000 7f39d6d0f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f39d00b1e40 0x7f39d0168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.259+0000 7f39d6d0f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f39d00b1e40 0x7f39d0168b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52104/0 (socket says 172.21.15.59:52104) 2024-03-06T07:28:28.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.259+0000 7f39d5d0d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f39d00d4860 0x7f39d00d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59014/0 (socket says 172.21.15.59:59014) 2024-03-06T07:28:28.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.259+0000 7f39d6d0f640 1 -- 172.21.15.59:0/1429257895 learned_addr learned my addr 172.21.15.59:0/1429257895 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:28.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.259+0000 7f39d6d0f640 1 -- 172.21.15.59:0/1429257895 >> v1:172.21.15.164:6789/0 conn(0x7f39d01614d0 legacy=0x7f39d00d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.259+0000 7f39d6d0f640 1 -- 172.21.15.59:0/1429257895 >> v1:172.21.15.59:6789/0 conn(0x7f39d00d4860 legacy=0x7f39d00d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.259+0000 7f39d6d0f640 1 -- 172.21.15.59:0/1429257895 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d00cf400 con 0x7f39d00b1e40 2024-03-06T07:28:28.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.260+0000 7f39d6d0f640 1 --2- 172.21.15.59:0/1429257895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39d00b1e40 0x7f39d0168b60 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f39d00cf5d0 tx=0x7f39c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20a64ac2888d7b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.260+0000 7f39d550c640 1 -- 172.21.15.59:0/1429257895 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c802ca70 con 0x7f39d00b1e40 2024-03-06T07:28:28.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.260+0000 7f39d550c640 1 -- 172.21.15.59:0/1429257895 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c802cc10 con 0x7f39d00b1e40 2024-03-06T07:28:28.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.260+0000 7f39d550c640 1 -- 172.21.15.59:0/1429257895 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c80343f0 con 0x7f39d00b1e40 2024-03-06T07:28:28.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.260+0000 7f39d8772640 1 -- 172.21.15.59:0/1429257895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39d00b1e40 msgr2=0x7f39d0168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.260+0000 7f39d8772640 1 --2- 172.21.15.59:0/1429257895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39d00b1e40 0x7f39d0168b60 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f39d00cf5d0 tx=0x7f39c802bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.261+0000 7f39d8772640 1 -- 172.21.15.59:0/1429257895 shutdown_connections 2024-03-06T07:28:28.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.261+0000 7f39d8772640 1 --2- 172.21.15.59:0/1429257895 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39d00b1e40 0x7f39d0168b60 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.261+0000 7f39d8772640 1 -- 172.21.15.59:0/1429257895 >> 172.21.15.59:0/1429257895 conn(0x7f39d00d3d40 msgr2=0x7f39d00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.261+0000 7f39d8772640 1 -- 172.21.15.59:0/1429257895 shutdown_connections 2024-03-06T07:28:28.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.261+0000 7f39d8772640 1 -- 172.21.15.59:0/1429257895 wait complete. 2024-03-06T07:28:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.262+0000 7f39d8772640 1 Processor -- start 2024-03-06T07:28:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.262+0000 7f39d8772640 1 -- start start 2024-03-06T07:28:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.262+0000 7f39d8772640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39d00b1e40 0x7f39d00b9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.262+0000 7f39d8772640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39d00d4860 0x7f39d00cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.262+0000 7f39d8772640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39d01614d0 0x7f39d01f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.262+0000 7f39d8772640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d00cfc60 con 0x7f39d00b1e40 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.262+0000 7f39d8772640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d00cf980 con 0x7f39d00d4860 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.262+0000 7f39d8772640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d00b0910 con 0x7f39d01614d0 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.262+0000 7f39d650e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39d00b1e40 0x7f39d00b9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39d650e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39d00b1e40 0x7f39d00b9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52120/0 (socket says 172.21.15.59:52120) 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39d650e640 1 -- 172.21.15.59:0/968133034 learned_addr learned my addr 172.21.15.59:0/968133034 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39d650e640 1 -- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39d01614d0 msgr2=0x7f39d01f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39d650e640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39d01614d0 0x7f39d01f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39d5d0d640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39d00d4860 0x7f39d00cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39d650e640 1 -- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39d00d4860 msgr2=0x7f39d00cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39d650e640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39d00d4860 0x7f39d00cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39d650e640 1 -- 172.21.15.59:0/968133034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d00cb1d0 con 0x7f39d00b1e40 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39d650e640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39d00b1e40 0x7f39d00b9630 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00c460 tx=0x7f39cc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39c2ffd640 1 -- 172.21.15.59:0/968133034 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc016020 con 0x7f39d00b1e40 2024-03-06T07:28:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39c2ffd640 1 -- 172.21.15.59:0/968133034 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39cc012870 con 0x7f39d00b1e40 2024-03-06T07:28:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39d8772640 1 -- 172.21.15.59:0/968133034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d00cb4b0 con 0x7f39d00b1e40 2024-03-06T07:28:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39c2ffd640 1 -- 172.21.15.59:0/968133034 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc012c60 con 0x7f39d00b1e40 2024-03-06T07:28:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.263+0000 7f39d8772640 1 -- 172.21.15.59:0/968133034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d00cba70 con 0x7f39d00b1e40 2024-03-06T07:28:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.264+0000 7f39d8772640 1 -- 172.21.15.59:0/968133034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d0068ff0 con 0x7f39d00b1e40 2024-03-06T07:28:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.265+0000 7f39c2ffd640 1 -- 172.21.15.59:0/968133034 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39cc019020 con 0x7f39d00b1e40 2024-03-06T07:28:28.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.265+0000 7f39c2ffd640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39b406b4d0 0x7f39b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.265+0000 7f39c2ffd640 1 -- 172.21.15.59:0/968133034 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f39cc089f50 con 0x7f39d00b1e40 2024-03-06T07:28:28.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.265+0000 7f39d5d0d640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39b406b4d0 0x7f39b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.266+0000 7f39d5d0d640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39b406b4d0 0x7f39b406d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f39d00ce030 tx=0x7f39c4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.273+0000 7f39c2ffd640 1 -- 172.21.15.59:0/968133034 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39cc020030 con 0x7f39d00b1e40 2024-03-06T07:28:28.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.452+0000 7f39d8772640 1 -- 172.21.15.59:0/968133034 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f39d00670d0 con 0x7f39d00b1e40 2024-03-06T07:28:28.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.453+0000 7f39c2ffd640 1 -- 172.21.15.59:0/968133034 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f39cc058d40 con 0x7f39d00b1e40 2024-03-06T07:28:28.455 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:28:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 -- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39b406b4d0 msgr2=0x7f39b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39b406b4d0 0x7f39b406d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f39d00ce030 tx=0x7f39c4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 -- 172.21.15.59:0/968133034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39d00b1e40 msgr2=0x7f39d00b9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39d00b1e40 0x7f39d00b9630 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00c460 tx=0x7f39cc00c920 comp rx=0 tx=0).stop 2024-03-06T07:28:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 -- 172.21.15.59:0/968133034 shutdown_connections 2024-03-06T07:28:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39b406b4d0 0x7f39b406d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39d01614d0 0x7f39d01f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39d00d4860 0x7f39d00cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 --2- 172.21.15.59:0/968133034 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39d00b1e40 0x7f39d00b9630 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 -- 172.21.15.59:0/968133034 >> 172.21.15.59:0/968133034 conn(0x7f39d00d3d40 msgr2=0x7f39d006c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 -- 172.21.15.59:0/968133034 shutdown_connections 2024-03-06T07:28:28.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.455+0000 7f39d8772640 1 -- 172.21.15.59:0/968133034 wait complete. 2024-03-06T07:28:28.468 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:28.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.574+0000 7f4a8cc81640 1 Processor -- start 2024-03-06T07:28:28.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.574+0000 7f4a8cc81640 1 -- start start 2024-03-06T07:28:28.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.575+0000 7f4a8cc81640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4a880b1e40 0x7f4a88168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.575+0000 7f4a8cc81640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a880d2730 con 0x7f4a880d4860 2024-03-06T07:28:28.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.575+0000 7f4a8cc81640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a881618b0 con 0x7f4a881614d0 2024-03-06T07:28:28.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.575+0000 7f4a8cc81640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a880d4cf0 con 0x7f4a880b1e40 2024-03-06T07:28:28.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a7dd9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4a880d4860 0x7f4a880d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59016/0 (socket says 172.21.15.59:59016) 2024-03-06T07:28:28.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a7dd9b640 1 -- 172.21.15.59:0/11626512 learned_addr learned my addr 172.21.15.59:0/11626512 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:28.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111866800 0 0) 0x7f4a880d2730 con 0x7f4a880d4860 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a74003520 con 0x7f4a880d4860 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a86d9d640 1 --2- 172.21.15.59:0/11626512 >> v2:172.21.15.164:3300/0 conn(0x7f4a880b1e40 0x7f4a88168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1022505415 0 0) 0x7f4a74003520 con 0x7f4a880d4860 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 >> v2:172.21.15.164:3300/0 conn(0x7f4a880b1e40 msgr2=0x7f4a88168b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 --2- 172.21.15.59:0/11626512 >> v2:172.21.15.164:3300/0 conn(0x7f4a880b1e40 0x7f4a88168b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 >> v1:172.21.15.164:6789/0 conn(0x7f4a881614d0 legacy=0x7f4a880d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a880cf400 con 0x7f4a880d4860 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f4a70001670 con 0x7f4a880d4860 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 >> v1:172.21.15.59:6789/0 conn(0x7f4a880d4860 legacy=0x7f4a880d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 --2- 172.21.15.59:0/11626512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a740039b0 0x7f4a74003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 --2- 172.21.15.59:0/11626512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a74004800 0x7f4a74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 --2- 172.21.15.59:0/11626512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a740052e0 0x7f4a7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a880d4cf0 con 0x7f4a740052e0 2024-03-06T07:28:28.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7400c1e0 con 0x7f4a74004800 2024-03-06T07:28:28.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7400c3e0 con 0x7f4a740039b0 2024-03-06T07:28:28.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a86d9d640 1 --2- 172.21.15.59:0/11626512 >> v2:172.21.15.164:3300/0 conn(0x7f4a880b1e40 0x7f4a88168b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:28.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a86d9d640 1 --2- 172.21.15.59:0/11626512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a740052e0 0x7f4a7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a8659c640 1 --2- 172.21.15.59:0/11626512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a740039b0 0x7f4a74003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.576+0000 7f4a7dd9b640 1 --2- 172.21.15.59:0/11626512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a74004800 0x7f4a74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.577+0000 7f4a86d9d640 1 -- 172.21.15.59:0/11626512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a740039b0 msgr2=0x7f4a74003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.577+0000 7f4a86d9d640 1 --2- 172.21.15.59:0/11626512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a740039b0 0x7f4a74003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.577+0000 7f4a86d9d640 1 -- 172.21.15.59:0/11626512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a74004800 msgr2=0x7f4a74004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.577+0000 7f4a86d9d640 1 --2- 172.21.15.59:0/11626512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a74004800 0x7f4a74004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.577+0000 7f4a86d9d640 1 -- 172.21.15.59:0/11626512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a74003520 con 0x7f4a740052e0 2024-03-06T07:28:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.578+0000 7f4a86d9d640 1 -- 172.21.15.59:0/11626512 reap_dead start 2024-03-06T07:28:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.578+0000 7f4a86d9d640 1 --2- 172.21.15.59:0/11626512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a740052e0 0x7f4a7400b9c0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f4a800073b0 tx=0x7f4a74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=468317aaf702a75b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.578+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80039070 con 0x7f4a740052e0 2024-03-06T07:28:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.578+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8002cdb0 con 0x7f4a740052e0 2024-03-06T07:28:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.578+0000 7f4a85d9b640 1 -- 172.21.15.59:0/11626512 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80009c70 con 0x7f4a740052e0 2024-03-06T07:28:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.579+0000 7f4a8cc81640 1 -- 172.21.15.59:0/11626512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a740052e0 msgr2=0x7f4a7400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.579+0000 7f4a8cc81640 1 --2- 172.21.15.59:0/11626512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a740052e0 0x7f4a7400b9c0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f4a800073b0 tx=0x7f4a74005830 comp rx=0 tx=0).stop 2024-03-06T07:28:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.579+0000 7f4a8cc81640 1 -- 172.21.15.59:0/11626512 shutdown_connections 2024-03-06T07:28:28.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.579+0000 7f4a8cc81640 1 --2- 172.21.15.59:0/11626512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a740052e0 0x7f4a7400b9c0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.579+0000 7f4a8cc81640 1 -- 172.21.15.59:0/11626512 >> 172.21.15.59:0/11626512 conn(0x7f4a880d3d40 msgr2=0x7f4a880d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.579+0000 7f4a8cc81640 1 -- 172.21.15.59:0/11626512 shutdown_connections 2024-03-06T07:28:28.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.579+0000 7f4a8cc81640 1 -- 172.21.15.59:0/11626512 wait complete. 2024-03-06T07:28:28.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.581+0000 7f4a8cc81640 1 Processor -- start 2024-03-06T07:28:28.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.581+0000 7f4a8cc81640 1 -- start start 2024-03-06T07:28:28.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a8cc81640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a740052e0 0x7f4a880c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a8cc81640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a88157d90 0x7f4a880c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a8cc81640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a88147930 0x7f4a88147db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a8cc81640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a880cfc60 con 0x7f4a88147930 2024-03-06T07:28:28.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a8cc81640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a880cf980 con 0x7f4a740052e0 2024-03-06T07:28:28.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a8cc81640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a880b0910 con 0x7f4a88157d90 2024-03-06T07:28:28.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a86d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a88147930 0x7f4a88147db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a86d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a88147930 0x7f4a88147db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52140/0 (socket says 172.21.15.59:52140) 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a86d9d640 1 -- 172.21.15.59:0/1647742539 learned_addr learned my addr 172.21.15.59:0/1647742539 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a8659c640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a740052e0 0x7f4a880c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a7dd9b640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a88157d90 0x7f4a880c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a86d9d640 1 -- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a88157d90 msgr2=0x7f4a880c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a86d9d640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a88157d90 0x7f4a880c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a86d9d640 1 -- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a740052e0 msgr2=0x7f4a880c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a86d9d640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a740052e0 0x7f4a880c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.582+0000 7f4a86d9d640 1 -- 172.21.15.59:0/1647742539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a880c84d0 con 0x7f4a88147930 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.583+0000 7f4a86d9d640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a88147930 0x7f4a88147db0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f4a800074d0 tx=0x7f4a80004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.583+0000 7f4a7f7fe640 1 -- 172.21.15.59:0/1647742539 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80039040 con 0x7f4a88147930 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.583+0000 7f4a7f7fe640 1 -- 172.21.15.59:0/1647742539 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a80045070 con 0x7f4a88147930 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.583+0000 7f4a7f7fe640 1 -- 172.21.15.59:0/1647742539 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80009c70 con 0x7f4a88147930 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.583+0000 7f4a8cc81640 1 -- 172.21.15.59:0/1647742539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a880c87b0 con 0x7f4a88147930 2024-03-06T07:28:28.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.583+0000 7f4a8cc81640 1 -- 172.21.15.59:0/1647742539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a880c8d70 con 0x7f4a88147930 2024-03-06T07:28:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.584+0000 7f4a8cc81640 1 -- 172.21.15.59:0/1647742539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8806bfe0 con 0x7f4a88147930 2024-03-06T07:28:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.584+0000 7f4a7f7fe640 1 -- 172.21.15.59:0/1647742539 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a8003c070 con 0x7f4a88147930 2024-03-06T07:28:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.585+0000 7f4a7f7fe640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a6806b4d0 0x7f4a6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.585+0000 7f4a7f7fe640 1 -- 172.21.15.59:0/1647742539 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4a800ac060 con 0x7f4a88147930 2024-03-06T07:28:28.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.585+0000 7f4a8659c640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a6806b4d0 0x7f4a6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.586+0000 7f4a8659c640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a6806b4d0 0x7f4a6806d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f4a78005590 tx=0x7f4a78009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.589+0000 7f4a7f7fe640 1 -- 172.21.15.59:0/1647742539 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a8806bfe0 con 0x7f4a88147930 2024-03-06T07:28:28.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.798+0000 7f4a8cc81640 1 -- 172.21.15.59:0/1647742539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4a880d2520 con 0x7f4a88147930 2024-03-06T07:28:28.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.798+0000 7f4a7f7fe640 1 -- 172.21.15.59:0/1647742539 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4a8004f020 con 0x7f4a88147930 2024-03-06T07:28:28.800 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 -- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a6806b4d0 msgr2=0x7f4a6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a6806b4d0 0x7f4a6806d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f4a78005590 tx=0x7f4a78009040 comp rx=0 tx=0).stop 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 -- 172.21.15.59:0/1647742539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a88147930 msgr2=0x7f4a88147db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a88147930 0x7f4a88147db0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f4a800074d0 tx=0x7f4a80004ab0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 -- 172.21.15.59:0/1647742539 shutdown_connections 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a6806b4d0 0x7f4a6806d990 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a88147930 0x7f4a88147db0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a88157d90 0x7f4a880c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 --2- 172.21.15.59:0/1647742539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a740052e0 0x7f4a880c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 -- 172.21.15.59:0/1647742539 >> 172.21.15.59:0/1647742539 conn(0x7f4a880d3d40 msgr2=0x7f4a8806b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 -- 172.21.15.59:0/1647742539 shutdown_connections 2024-03-06T07:28:28.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.800+0000 7f4a8cc81640 1 -- 172.21.15.59:0/1647742539 wait complete. 2024-03-06T07:28:28.813 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:28.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.916+0000 7f7a7260e640 1 Processor -- start 2024-03-06T07:28:28.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.916+0000 7f7a7260e640 1 -- start start 2024-03-06T07:28:28.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.917+0000 7f7a7260e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7a6c0d10c0 0x7f7a6c0d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.917+0000 7f7a7260e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a6c15ecd0 0x7f7a6c0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.917+0000 7f7a7260e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6c0cfa50 con 0x7f7a6c0d14a0 2024-03-06T07:28:28.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.917+0000 7f7a7260e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a6c15c050 con 0x7f7a6c0d10c0 2024-03-06T07:28:28.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.917+0000 7f7a7260e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0d0060 con 0x7f7a6c15ecd0 2024-03-06T07:28:28.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.917+0000 7f7a6b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7a6c0d14a0 0x7f7a6c15e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59030/0 (socket says 172.21.15.59:59030) 2024-03-06T07:28:28.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.917+0000 7f7a6b7fe640 1 -- 172.21.15.59:0/3171103798 learned_addr learned my addr 172.21.15.59:0/3171103798 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:28.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.917+0000 7f7a6bfff640 1 --2- 172.21.15.59:0/3171103798 >> v2:172.21.15.164:3300/0 conn(0x7f7a6c0d10c0 0x7f7a6c0d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.917+0000 7f7a70bab640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a6c15ecd0 0x7f7a6c0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.917+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922472087 0 0) 0x7f7a6c0cfa50 con 0x7f7a6c0d14a0 2024-03-06T07:28:28.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a54003520 con 0x7f7a6c0d14a0 2024-03-06T07:28:28.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3988311342 0 0) 0x7f7a54003520 con 0x7f7a6c0d14a0 2024-03-06T07:28:28.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a6c15ecd0 msgr2=0x7f7a6c0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a6c15ecd0 0x7f7a6c0cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 >> v2:172.21.15.164:3300/0 conn(0x7f7a6c0d10c0 msgr2=0x7f7a6c0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 --2- 172.21.15.59:0/3171103798 >> v2:172.21.15.164:3300/0 conn(0x7f7a6c0d10c0 0x7f7a6c0d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c0d0a30 con 0x7f7a6c0d14a0 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f7a60001670 con 0x7f7a6c0d14a0 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 >> v1:172.21.15.59:6789/0 conn(0x7f7a6c0d14a0 legacy=0x7f7a6c15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a540039b0 0x7f7a54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a54004800 0x7f7a54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a54005180 0x7f7a5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0d0060 con 0x7f7a540039b0 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a540045b0 con 0x7f7a54005180 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5400bf00 con 0x7f7a54004800 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6bfff640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a540039b0 0x7f7a54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6b7fe640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a54004800 0x7f7a54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a70bab640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a54005180 0x7f7a5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6bfff640 1 -- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a54004800 msgr2=0x7f7a54004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6bfff640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a54004800 0x7f7a54004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6bfff640 1 -- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a54005180 msgr2=0x7f7a5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6bfff640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a54005180 0x7f7a5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a6bfff640 1 -- 172.21.15.59:0/3171103798 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a54003520 con 0x7f7a540039b0 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.918+0000 7f7a70bab640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a54005180 0x7f7a5400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.919+0000 7f7a6bfff640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a540039b0 0x7f7a54003dd0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009ac0 tx=0x7f7a54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a849477524c240a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.919+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c039070 con 0x7f7a540039b0 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.919+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5c02c710 con 0x7f7a540039b0 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.919+0000 7f7a6affd640 1 -- 172.21.15.59:0/3171103798 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c009690 con 0x7f7a540039b0 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.920+0000 7f7a7260e640 1 -- 172.21.15.59:0/3171103798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a540039b0 msgr2=0x7f7a54003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.920+0000 7f7a7260e640 1 --2- 172.21.15.59:0/3171103798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a540039b0 0x7f7a54003dd0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009ac0 tx=0x7f7a54005830 comp rx=0 tx=0).stop 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.920+0000 7f7a70bab640 1 -- 172.21.15.59:0/3171103798 reap_dead start 2024-03-06T07:28:28.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.920+0000 7f7a7260e640 1 -- 172.21.15.59:0/3171103798 shutdown_connections 2024-03-06T07:28:28.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.920+0000 7f7a7260e640 1 -- 172.21.15.59:0/3171103798 >> 172.21.15.59:0/3171103798 conn(0x7f7a6c0b3800 msgr2=0x7f7a6c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.920+0000 7f7a7260e640 1 -- 172.21.15.59:0/3171103798 shutdown_connections 2024-03-06T07:28:28.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.920+0000 7f7a7260e640 1 -- 172.21.15.59:0/3171103798 wait complete. 2024-03-06T07:28:28.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.921+0000 7f7a7260e640 1 Processor -- start 2024-03-06T07:28:28.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.921+0000 7f7a7260e640 1 -- start start 2024-03-06T07:28:28.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a7260e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a540039b0 0x7f7a6c1f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a7260e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a6c0b5de0 0x7f7a6c0b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a7260e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a6c0b7450 0x7f7a6c1f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a7260e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c15f440 con 0x7f7a6c0b7450 2024-03-06T07:28:28.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a7260e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c15f160 con 0x7f7a540039b0 2024-03-06T07:28:28.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a7260e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0cfcf0 con 0x7f7a6c0b5de0 2024-03-06T07:28:28.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a70bab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a6c0b7450 0x7f7a6c1f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a70bab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a6c0b7450 0x7f7a6c1f70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52158/0 (socket says 172.21.15.59:52158) 2024-03-06T07:28:28.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a70bab640 1 -- 172.21.15.59:0/1578116149 learned_addr learned my addr 172.21.15.59:0/1578116149 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a6bfff640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a540039b0 0x7f7a6c1f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a6b7fe640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a6c0b5de0 0x7f7a6c0b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a70bab640 1 -- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a6c0b5de0 msgr2=0x7f7a6c0b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a70bab640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a6c0b5de0 0x7f7a6c0b6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a70bab640 1 -- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a540039b0 msgr2=0x7f7a6c1f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a70bab640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a540039b0 0x7f7a6c1f3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.922+0000 7f7a70bab640 1 -- 172.21.15.59:0/1578116149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c0b4d90 con 0x7f7a6c0b7450 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.923+0000 7f7a70bab640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a6c0b7450 0x7f7a6c1f70a0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c0d1ab0 tx=0x7f7a64010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.923+0000 7f7a68ff9640 1 -- 172.21.15.59:0/1578116149 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64010e00 con 0x7f7a6c0b7450 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.923+0000 7f7a68ff9640 1 -- 172.21.15.59:0/1578116149 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a64003990 con 0x7f7a6c0b7450 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.923+0000 7f7a68ff9640 1 -- 172.21.15.59:0/1578116149 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64003dc0 con 0x7f7a6c0b7450 2024-03-06T07:28:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.923+0000 7f7a7260e640 1 -- 172.21.15.59:0/1578116149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c0b5320 con 0x7f7a6c0b7450 2024-03-06T07:28:28.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.923+0000 7f7a7260e640 1 -- 172.21.15.59:0/1578116149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c0b58e0 con 0x7f7a6c0b7450 2024-03-06T07:28:28.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.924+0000 7f7a7260e640 1 -- 172.21.15.59:0/1578116149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c09fe60 con 0x7f7a6c0b7450 2024-03-06T07:28:28.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.924+0000 7f7a68ff9640 1 -- 172.21.15.59:0/1578116149 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a6401e030 con 0x7f7a6c0b7450 2024-03-06T07:28:28.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.925+0000 7f7a68ff9640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a5806b4d0 0x7f7a5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.925+0000 7f7a68ff9640 1 -- 172.21.15.59:0/1578116149 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7a6408a8c0 con 0x7f7a6c0b7450 2024-03-06T07:28:28.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.925+0000 7f7a6bfff640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a5806b4d0 0x7f7a5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.925+0000 7f7a6bfff640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a5806b4d0 0x7f7a5806d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c036000 tx=0x7f7a5c030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:28.929+0000 7f7a68ff9640 1 -- 172.21.15.59:0/1578116149 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a64025020 con 0x7f7a6c0b7450 2024-03-06T07:28:29.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.111+0000 7f7a7260e640 1 -- 172.21.15.59:0/1578116149 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7a6c068ff0 con 0x7f7a6c0b7450 2024-03-06T07:28:29.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.111+0000 7f7a68ff9640 1 -- 172.21.15.59:0/1578116149 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7a64059740 con 0x7f7a6c0b7450 2024-03-06T07:28:29.113 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:28:29.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 -- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a5806b4d0 msgr2=0x7f7a5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a5806b4d0 0x7f7a5806d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c036000 tx=0x7f7a5c030000 comp rx=0 tx=0).stop 2024-03-06T07:28:29.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 -- 172.21.15.59:0/1578116149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a6c0b7450 msgr2=0x7f7a6c1f70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a6c0b7450 0x7f7a6c1f70a0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c0d1ab0 tx=0x7f7a64010590 comp rx=0 tx=0).stop 2024-03-06T07:28:29.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 -- 172.21.15.59:0/1578116149 shutdown_connections 2024-03-06T07:28:29.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7a5806b4d0 0x7f7a5806d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a6c0b7450 0x7f7a6c1f70a0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a6c0b5de0 0x7f7a6c0b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 --2- 172.21.15.59:0/1578116149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a540039b0 0x7f7a6c1f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 -- 172.21.15.59:0/1578116149 >> 172.21.15.59:0/1578116149 conn(0x7f7a6c0b3800 msgr2=0x7f7a6c0b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 -- 172.21.15.59:0/1578116149 shutdown_connections 2024-03-06T07:28:29.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.113+0000 7f7a7260e640 1 -- 172.21.15.59:0/1578116149 wait complete. 2024-03-06T07:28:29.129 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.228+0000 7ff477b3a640 1 Processor -- start 2024-03-06T07:28:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.229+0000 7ff477b3a640 1 -- start start 2024-03-06T07:28:29.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.230+0000 7ff477b3a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4700d2710 0x7ff4700b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.230+0000 7ff477b3a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4700b3430 con 0x7ff47015dfe0 2024-03-06T07:28:29.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.230+0000 7ff477b3a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4700b3710 con 0x7ff4700d22b0 2024-03-06T07:28:29.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.230+0000 7ff477b3a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4700b31e0 con 0x7ff4700d2710 2024-03-06T07:28:29.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.230+0000 7ff4760d7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4700d2710 0x7ff4700b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.230+0000 7ff4758d6640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff47015dfe0 0x7ff4700cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59044/0 (socket says 172.21.15.59:59044) 2024-03-06T07:28:29.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.230+0000 7ff4758d6640 1 -- 172.21.15.59:0/4147229770 learned_addr learned my addr 172.21.15.59:0/4147229770 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:29.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.230+0000 7ff4760d7640 1 -- 172.21.15.59:0/4147229770 >> v1:172.21.15.164:6789/0 conn(0x7ff4700d22b0 legacy=0x7ff4700b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.230+0000 7ff4760d7640 1 -- 172.21.15.59:0/4147229770 >> v1:172.21.15.59:6789/0 conn(0x7ff47015dfe0 legacy=0x7ff4700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.230+0000 7ff4748d4640 1 -- 172.21.15.59:0/4147229770 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319325663 0 0) 0x7ff4700b3430 con 0x7ff47015dfe0 2024-03-06T07:28:29.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.230+0000 7ff4760d7640 1 -- 172.21.15.59:0/4147229770 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4700b39f0 con 0x7ff4700d2710 2024-03-06T07:28:29.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.231+0000 7ff4760d7640 1 --2- 172.21.15.59:0/4147229770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4700d2710 0x7ff4700b4900 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff468009dc0 tx=0x7ff46802bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299416ec4c8a122a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.231+0000 7ff4748d4640 1 -- 172.21.15.59:0/4147229770 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468039070 con 0x7ff4700d2710 2024-03-06T07:28:29.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.231+0000 7ff4748d4640 1 -- 172.21.15.59:0/4147229770 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46802ccd0 con 0x7ff4700d2710 2024-03-06T07:28:29.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.231+0000 7ff4748d4640 1 -- 172.21.15.59:0/4147229770 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468034660 con 0x7ff4700d2710 2024-03-06T07:28:29.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.231+0000 7ff477b3a640 1 -- 172.21.15.59:0/4147229770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4700d2710 msgr2=0x7ff4700b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.231+0000 7ff477b3a640 1 --2- 172.21.15.59:0/4147229770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4700d2710 0x7ff4700b4900 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff468009dc0 tx=0x7ff46802bf20 comp rx=0 tx=0).stop 2024-03-06T07:28:29.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.232+0000 7ff477b3a640 1 -- 172.21.15.59:0/4147229770 shutdown_connections 2024-03-06T07:28:29.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.232+0000 7ff477b3a640 1 --2- 172.21.15.59:0/4147229770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4700d2710 0x7ff4700b4900 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.232+0000 7ff477b3a640 1 -- 172.21.15.59:0/4147229770 >> 172.21.15.59:0/4147229770 conn(0x7ff4700d3cd0 msgr2=0x7ff4700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.232+0000 7ff477b3a640 1 -- 172.21.15.59:0/4147229770 shutdown_connections 2024-03-06T07:28:29.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.232+0000 7ff477b3a640 1 -- 172.21.15.59:0/4147229770 wait complete. 2024-03-06T07:28:29.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.233+0000 7ff477b3a640 1 Processor -- start 2024-03-06T07:28:29.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.233+0000 7ff477b3a640 1 -- start start 2024-03-06T07:28:29.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.233+0000 7ff477b3a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4700d22b0 0x7ff470157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff477b3a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4700d2710 0x7ff4700b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff477b3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47015dfe0 0x7ff4700b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff477b3a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700cfa00 con 0x7ff47015dfe0 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff477b3a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700b3d60 con 0x7ff4700d22b0 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff477b3a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700d1c70 con 0x7ff4700d2710 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4760d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47015dfe0 0x7ff4700b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4760d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47015dfe0 0x7ff4700b1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52186/0 (socket says 172.21.15.59:52186) 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4760d7640 1 -- 172.21.15.59:0/523134993 learned_addr learned my addr 172.21.15.59:0/523134993 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4758d6640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4700d22b0 0x7ff470157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4750d5640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4700d2710 0x7ff4700b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4760d7640 1 -- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4700d2710 msgr2=0x7ff4700b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4760d7640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4700d2710 0x7ff4700b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4760d7640 1 -- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4700d22b0 msgr2=0x7ff470157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4760d7640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4700d22b0 0x7ff470157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4760d7640 1 -- 172.21.15.59:0/523134993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4700b8bb0 con 0x7ff47015dfe0 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4760d7640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47015dfe0 0x7ff4700b1550 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7ff468009ee0 tx=0x7ff468004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4667fc640 1 -- 172.21.15.59:0/523134993 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468044020 con 0x7ff47015dfe0 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4667fc640 1 -- 172.21.15.59:0/523134993 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46803a040 con 0x7ff47015dfe0 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff4667fc640 1 -- 172.21.15.59:0/523134993 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468039050 con 0x7ff47015dfe0 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff477b3a640 1 -- 172.21.15.59:0/523134993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4700b8e90 con 0x7ff47015dfe0 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.234+0000 7ff477b3a640 1 -- 172.21.15.59:0/523134993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4700b9450 con 0x7ff47015dfe0 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.236+0000 7ff477b3a640 1 -- 172.21.15.59:0/523134993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47004f330 con 0x7ff47015dfe0 2024-03-06T07:28:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.236+0000 7ff4667fc640 1 -- 172.21.15.59:0/523134993 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff46802cb90 con 0x7ff47015dfe0 2024-03-06T07:28:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.236+0000 7ff4667fc640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 0x7ff45406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.236+0000 7ff4667fc640 1 -- 172.21.15.59:0/523134993 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff4680ac810 con 0x7ff47015dfe0 2024-03-06T07:28:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.237+0000 7ff4758d6640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 0x7ff45406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.237+0000 7ff4758d6640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 0x7ff45406d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff46c000f60 tx=0x7ff46c007e40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.241+0000 7ff4667fc640 1 -- 172.21.15.59:0/523134993 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff46804e060 con 0x7ff47015dfe0 2024-03-06T07:28:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.593+0000 7ff477b3a640 1 -- 172.21.15.59:0/523134993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4700ce960 con 0x7ff47015dfe0 2024-03-06T07:28:29.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.594+0000 7ff4667fc640 1 -- 172.21.15.59:0/523134993 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7ff46807afc0 con 0x7ff47015dfe0 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 -- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 msgr2=0x7ff45406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 0x7ff45406d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff46c000f60 tx=0x7ff46c007e40 comp rx=0 tx=0).stop 2024-03-06T07:28:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 -- 172.21.15.59:0/523134993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47015dfe0 msgr2=0x7ff4700b1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47015dfe0 0x7ff4700b1550 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7ff468009ee0 tx=0x7ff468004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 -- 172.21.15.59:0/523134993 shutdown_connections 2024-03-06T07:28:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff45406b4d0 0x7ff45406d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff47015dfe0 0x7ff4700b1550 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4700d2710 0x7ff4700b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 --2- 172.21.15.59:0/523134993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4700d22b0 0x7ff470157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 -- 172.21.15.59:0/523134993 >> 172.21.15.59:0/523134993 conn(0x7ff4700d3cd0 msgr2=0x7ff4700ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 -- 172.21.15.59:0/523134993 shutdown_connections 2024-03-06T07:28:29.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.596+0000 7ff477b3a640 1 -- 172.21.15.59:0/523134993 wait complete. 2024-03-06T07:28:29.609 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:29.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.709+0000 7fe194e88640 1 Processor -- start 2024-03-06T07:28:29.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.709+0000 7fe194e88640 1 -- start start 2024-03-06T07:28:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.710+0000 7fe194e88640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe19015dfe0 0x7fe1900cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.710+0000 7fe194e88640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1900b3430 con 0x7fe1900d29c0 2024-03-06T07:28:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.710+0000 7fe194e88640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1900b3710 con 0x7fe1900cf830 2024-03-06T07:28:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.710+0000 7fe194e88640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1900b31e0 con 0x7fe19015dfe0 2024-03-06T07:28:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.710+0000 7fe18ed9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe1900d29c0 0x7fe1900b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59052/0 (socket says 172.21.15.59:59052) 2024-03-06T07:28:29.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.710+0000 7fe18ed9d640 1 -- 172.21.15.59:0/97461232 learned_addr learned my addr 172.21.15.59:0/97461232 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.710+0000 7fe18e59c640 1 --2- 172.21.15.59:0/97461232 >> v2:172.21.15.164:3300/0 conn(0x7fe19015dfe0 0x7fe1900cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.710+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284937215 0 0) 0x7fe1900b3430 con 0x7fe1900d29c0 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe17c003520 con 0x7fe1900d29c0 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2603740595 0 0) 0x7fe17c003520 con 0x7fe1900d29c0 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 >> v2:172.21.15.164:3300/0 conn(0x7fe19015dfe0 msgr2=0x7fe1900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 --2- 172.21.15.59:0/97461232 >> v2:172.21.15.164:3300/0 conn(0x7fe19015dfe0 0x7fe1900cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 >> v1:172.21.15.164:6789/0 conn(0x7fe1900cf830 legacy=0x7fe1900d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1900b39f0 con 0x7fe1900d29c0 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fe1800019f0 con 0x7fe1900d29c0 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 >> v1:172.21.15.59:6789/0 conn(0x7fe1900d29c0 legacy=0x7fe1900b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 --2- 172.21.15.59:0/97461232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0039b0 0x7fe17c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 --2- 172.21.15.59:0/97461232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe17c004800 0x7fe17c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 --2- 172.21.15.59:0/97461232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c0051c0 0x7fe17c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1900b31e0 con 0x7fe17c0051c0 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c00c090 con 0x7fe17c004800 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c00c290 con 0x7fe17c0039b0 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18e59c640 1 --2- 172.21.15.59:0/97461232 >> v2:172.21.15.164:3300/0 conn(0x7fe19015dfe0 0x7fe1900cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18ed9d640 1 --2- 172.21.15.59:0/97461232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c0051c0 0x7fe17c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18dd9b640 1 --2- 172.21.15.59:0/97461232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe17c004800 0x7fe17c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18e59c640 1 --2- 172.21.15.59:0/97461232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0039b0 0x7fe17c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18ed9d640 1 -- 172.21.15.59:0/97461232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0039b0 msgr2=0x7fe17c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18ed9d640 1 --2- 172.21.15.59:0/97461232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0039b0 0x7fe17c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18ed9d640 1 -- 172.21.15.59:0/97461232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe17c004800 msgr2=0x7fe17c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18ed9d640 1 --2- 172.21.15.59:0/97461232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe17c004800 0x7fe17c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.711+0000 7fe18ed9d640 1 -- 172.21.15.59:0/97461232 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c003520 con 0x7fe17c0051c0 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.712+0000 7fe18ed9d640 1 -- 172.21.15.59:0/97461232 reap_dead start 2024-03-06T07:28:29.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.712+0000 7fe18ed9d640 1 --2- 172.21.15.59:0/97461232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c0051c0 0x7fe17c00b9c0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fe18000a830 tx=0x7fe17c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ee00b8212bbbed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.713+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18000a480 con 0x7fe17c0051c0 2024-03-06T07:28:29.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.713+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe180034630 con 0x7fe17c0051c0 2024-03-06T07:28:29.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.713+0000 7fe18d59a640 1 -- 172.21.15.59:0/97461232 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180034a00 con 0x7fe17c0051c0 2024-03-06T07:28:29.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.713+0000 7fe194e88640 1 -- 172.21.15.59:0/97461232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c0051c0 msgr2=0x7fe17c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.713+0000 7fe194e88640 1 --2- 172.21.15.59:0/97461232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c0051c0 0x7fe17c00b9c0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fe18000a830 tx=0x7fe17c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:29.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.713+0000 7fe194e88640 1 -- 172.21.15.59:0/97461232 shutdown_connections 2024-03-06T07:28:29.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.713+0000 7fe194e88640 1 --2- 172.21.15.59:0/97461232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe17c0051c0 0x7fe17c00b9c0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.713+0000 7fe194e88640 1 -- 172.21.15.59:0/97461232 >> 172.21.15.59:0/97461232 conn(0x7fe1900d3cd0 msgr2=0x7fe1900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.713+0000 7fe194e88640 1 -- 172.21.15.59:0/97461232 shutdown_connections 2024-03-06T07:28:29.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.713+0000 7fe194e88640 1 -- 172.21.15.59:0/97461232 wait complete. 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.715+0000 7fe194e88640 1 Processor -- start 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.715+0000 7fe194e88640 1 -- start start 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.716+0000 7fe194e88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0051c0 0x7fe1900b8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe194e88640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe19014c0a0 0x7fe19014d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe194e88640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19014dbf0 0x7fe19009dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe194e88640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19015e3c0 con 0x7fe19014dbf0 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe194e88640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1900b3d60 con 0x7fe19014c0a0 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe194e88640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19015e540 con 0x7fe17c0051c0 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe18ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19014dbf0 0x7fe19009dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe18ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19014dbf0 0x7fe19009dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52214/0 (socket says 172.21.15.59:52214) 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe18ed9d640 1 -- 172.21.15.59:0/61010810 learned_addr learned my addr 172.21.15.59:0/61010810 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe18e59c640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0051c0 0x7fe1900b8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe18dd9b640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe19014c0a0 0x7fe19014d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe18ed9d640 1 -- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0051c0 msgr2=0x7fe1900b8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe18ed9d640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0051c0 0x7fe1900b8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe18ed9d640 1 -- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe19014c0a0 msgr2=0x7fe19014d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe18ed9d640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe19014c0a0 0x7fe19014d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe18ed9d640 1 -- 172.21.15.59:0/61010810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19015b9b0 con 0x7fe19014dbf0 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe18ed9d640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19014dbf0 0x7fe19009dc60 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe18000a950 tx=0x7fe1900b38e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe186ffd640 1 -- 172.21.15.59:0/61010810 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180039040 con 0x7fe19014dbf0 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe186ffd640 1 -- 172.21.15.59:0/61010810 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe180043070 con 0x7fe19014dbf0 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.717+0000 7fe194e88640 1 -- 172.21.15.59:0/61010810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19014c610 con 0x7fe19014dbf0 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.718+0000 7fe186ffd640 1 -- 172.21.15.59:0/61010810 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18000a480 con 0x7fe19014dbf0 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.718+0000 7fe194e88640 1 -- 172.21.15.59:0/61010810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19014cbd0 con 0x7fe19014dbf0 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.719+0000 7fe194e88640 1 -- 172.21.15.59:0/61010810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19004f330 con 0x7fe19014dbf0 2024-03-06T07:28:29.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.719+0000 7fe186ffd640 1 -- 172.21.15.59:0/61010810 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe180034630 con 0x7fe19014dbf0 2024-03-06T07:28:29.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.719+0000 7fe186ffd640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe17006b4d0 0x7fe17006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.719+0000 7fe186ffd640 1 -- 172.21.15.59:0/61010810 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe1800ac1a0 con 0x7fe19014dbf0 2024-03-06T07:28:29.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.720+0000 7fe18e59c640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe17006b4d0 0x7fe17006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.720+0000 7fe18e59c640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe17006b4d0 0x7fe17006d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe19006d000 tx=0x7fe188008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.724+0000 7fe186ffd640 1 -- 172.21.15.59:0/61010810 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe18007aa60 con 0x7fe19014dbf0 2024-03-06T07:28:29.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.906+0000 7fe194e88640 1 -- 172.21.15.59:0/61010810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1900b0b60 con 0x7fe19014dbf0 2024-03-06T07:28:29.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.908+0000 7fe186ffd640 1 -- 172.21.15.59:0/61010810 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16843 (secure 0 0 0) 0x7fe18007eb00 con 0x7fe19014dbf0 2024-03-06T07:28:29.910 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:29.910 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":114,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:27:38.442474+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:12.625799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:13.306376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.25","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:6836/2139041464":"2024-03-07T07:27:27.121928+0000","172.21.15.164:6834/2139041464":"2024-03-07T07:27:27.121928+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:29.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 -- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe17006b4d0 msgr2=0x7fe17006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe17006b4d0 0x7fe17006d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe19006d000 tx=0x7fe188008040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 -- 172.21.15.59:0/61010810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19014dbf0 msgr2=0x7fe19009dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19014dbf0 0x7fe19009dc60 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe18000a950 tx=0x7fe1900b38e0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 -- 172.21.15.59:0/61010810 shutdown_connections 2024-03-06T07:28:29.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe17006b4d0 0x7fe17006d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe19014dbf0 0x7fe19009dc60 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe19014c0a0 0x7fe19014d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 --2- 172.21.15.59:0/61010810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe17c0051c0 0x7fe1900b8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 -- 172.21.15.59:0/61010810 >> 172.21.15.59:0/61010810 conn(0x7fe1900d3cd0 msgr2=0x7fe190160fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 -- 172.21.15.59:0/61010810 shutdown_connections 2024-03-06T07:28:29.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:29.910+0000 7fe194e88640 1 -- 172.21.15.59:0/61010810 wait complete. 2024-03-06T07:28:29.923 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:28:29.924 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:30.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.023+0000 7f7db57ae640 1 Processor -- start 2024-03-06T07:28:30.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.023+0000 7f7db57ae640 1 -- start start 2024-03-06T07:28:30.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7db57ae640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7db015dfe0 0x7f7db00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7db57ae640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7db00d29c0 0x7f7db00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7db57ae640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db00b3430 con 0x7f7db00cf830 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7db57ae640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db00b31e0 con 0x7f7db00d29c0 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7db57ae640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00b3d60 con 0x7f7db015dfe0 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7daf7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7db00d29c0 0x7f7db00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7da7fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7db00cf830 0x7f7db00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59056/0 (socket says 172.21.15.59:59056) 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7da7fff640 1 -- 172.21.15.59:0/4168889595 learned_addr learned my addr 172.21.15.59:0/4168889595 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7daeffd640 1 --2- 172.21.15.59:0/4168889595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7db015dfe0 0x7f7db00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7daf7fe640 1 -- 172.21.15.59:0/4168889595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7db015dfe0 msgr2=0x7f7db00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7daf7fe640 1 --2- 172.21.15.59:0/4168889595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7db015dfe0 0x7f7db00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7daf7fe640 1 -- 172.21.15.59:0/4168889595 >> v1:172.21.15.59:6789/0 conn(0x7f7db00cf830 legacy=0x7f7db00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7daf7fe640 1 -- 172.21.15.59:0/4168889595 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db00b3710 con 0x7f7db00d29c0 2024-03-06T07:28:30.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.024+0000 7f7dae7fc640 1 -- 172.21.15.59:0/4168889595 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021093238 0 0) 0x7f7db00b3430 con 0x7f7db00cf830 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.025+0000 7f7daf7fe640 1 --2- 172.21.15.59:0/4168889595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7db00d29c0 0x7f7db00b41f0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f7da8009d90 tx=0x7f7da802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465eec786a73531d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.025+0000 7f7dae7fc640 1 -- 172.21.15.59:0/4168889595 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da8039070 con 0x7f7db00d29c0 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.025+0000 7f7dae7fc640 1 -- 172.21.15.59:0/4168889595 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da802cc50 con 0x7f7db00d29c0 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.025+0000 7f7dae7fc640 1 -- 172.21.15.59:0/4168889595 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da8034440 con 0x7f7db00d29c0 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.026+0000 7f7db57ae640 1 -- 172.21.15.59:0/4168889595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7db00d29c0 msgr2=0x7f7db00b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.026+0000 7f7db57ae640 1 --2- 172.21.15.59:0/4168889595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7db00d29c0 0x7f7db00b41f0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f7da8009d90 tx=0x7f7da802bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.026+0000 7f7db57ae640 1 -- 172.21.15.59:0/4168889595 shutdown_connections 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.026+0000 7f7db57ae640 1 --2- 172.21.15.59:0/4168889595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7db00d29c0 0x7f7db00b41f0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.026+0000 7f7db57ae640 1 --2- 172.21.15.59:0/4168889595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7db015dfe0 0x7f7db00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.026+0000 7f7db57ae640 1 -- 172.21.15.59:0/4168889595 >> 172.21.15.59:0/4168889595 conn(0x7f7db00d3cd0 msgr2=0x7f7db00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.026+0000 7f7db57ae640 1 -- 172.21.15.59:0/4168889595 shutdown_connections 2024-03-06T07:28:30.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.026+0000 7f7db57ae640 1 -- 172.21.15.59:0/4168889595 wait complete. 2024-03-06T07:28:30.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.028+0000 7f7db57ae640 1 Processor -- start 2024-03-06T07:28:30.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.028+0000 7f7db57ae640 1 -- start start 2024-03-06T07:28:30.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7db57ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7db00cf830 0x7f7db0154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7db57ae640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7db00d29c0 0x7f7db006caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7db57ae640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7db015dfe0 0x7f7db006d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7db57ae640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00d1df0 con 0x7f7db00cf830 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7db57ae640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00d1c70 con 0x7f7db015dfe0 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7db57ae640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db015e3c0 con 0x7f7db00d29c0 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7daeffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7db00cf830 0x7f7db0154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7daeffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7db00cf830 0x7f7db0154b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52222/0 (socket says 172.21.15.59:52222) 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7daeffd640 1 -- 172.21.15.59:0/3298129933 learned_addr learned my addr 172.21.15.59:0/3298129933 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7daeffd640 1 -- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7db00d29c0 msgr2=0x7f7db006caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7daeffd640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7db00d29c0 0x7f7db006caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7daeffd640 1 -- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7db015dfe0 msgr2=0x7f7db006d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7daf7fe640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7db015dfe0 0x7f7db006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7daeffd640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7db015dfe0 0x7f7db006d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.029+0000 7f7daeffd640 1 -- 172.21.15.59:0/3298129933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db01f7130 con 0x7f7db00cf830 2024-03-06T07:28:30.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.030+0000 7f7daeffd640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7db00cf830 0x7f7db0154b00 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f7da000d620 tx=0x7f7da000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.030+0000 7f7da77fe640 1 -- 172.21.15.59:0/3298129933 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da000b620 con 0x7f7db00cf830 2024-03-06T07:28:30.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.030+0000 7f7da77fe640 1 -- 172.21.15.59:0/3298129933 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da0019070 con 0x7f7db00cf830 2024-03-06T07:28:30.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.030+0000 7f7db57ae640 1 -- 172.21.15.59:0/3298129933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db01f7410 con 0x7f7db00cf830 2024-03-06T07:28:30.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.030+0000 7f7da77fe640 1 -- 172.21.15.59:0/3298129933 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da000ba00 con 0x7f7db00cf830 2024-03-06T07:28:30.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.030+0000 7f7db57ae640 1 -- 172.21.15.59:0/3298129933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db01f79d0 con 0x7f7db00cf830 2024-03-06T07:28:30.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.031+0000 7f7db57ae640 1 -- 172.21.15.59:0/3298129933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db004f330 con 0x7f7db00cf830 2024-03-06T07:28:30.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.031+0000 7f7da77fe640 1 -- 172.21.15.59:0/3298129933 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7da000bba0 con 0x7f7db00cf830 2024-03-06T07:28:30.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.032+0000 7f7da77fe640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 0x7f7d9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.032+0000 7f7da77fe640 1 -- 172.21.15.59:0/3298129933 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7da008a6e0 con 0x7f7db00cf830 2024-03-06T07:28:30.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.032+0000 7f7da7fff640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 0x7f7d9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.033+0000 7f7da7fff640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 0x7f7d9006d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7d98007fd0 tx=0x7f7d98007920 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.036+0000 7f7da77fe640 1 -- 172.21.15.59:0/3298129933 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7da00594c0 con 0x7f7db00cf830 2024-03-06T07:28:30.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.254+0000 7f7db57ae640 1 -- 172.21.15.59:0/3298129933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7db00b2df0 con 0x7f7db00cf830 2024-03-06T07:28:30.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.254+0000 7f7da77fe640 1 -- 172.21.15.59:0/3298129933 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f7da0058dc0 con 0x7f7db00cf830 2024-03-06T07:28:30.256 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:30.256 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:30.257 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:30.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 -- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 msgr2=0x7f7d9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 0x7f7d9006d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7d98007fd0 tx=0x7f7d98007920 comp rx=0 tx=0).stop 2024-03-06T07:28:30.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 -- 172.21.15.59:0/3298129933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7db00cf830 msgr2=0x7f7db0154b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7db00cf830 0x7f7db0154b00 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f7da000d620 tx=0x7f7da000dae0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 -- 172.21.15.59:0/3298129933 shutdown_connections 2024-03-06T07:28:30.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d9006b4d0 0x7f7d9006d990 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7db015dfe0 0x7f7db006d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7db00d29c0 0x7f7db006caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 --2- 172.21.15.59:0/3298129933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7db00cf830 0x7f7db0154b00 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 -- 172.21.15.59:0/3298129933 >> 172.21.15.59:0/3298129933 conn(0x7f7db00d3cd0 msgr2=0x7f7db00d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 -- 172.21.15.59:0/3298129933 shutdown_connections 2024-03-06T07:28:30.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.256+0000 7f7db57ae640 1 -- 172.21.15.59:0/3298129933 wait complete. 2024-03-06T07:28:30.269 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.368+0000 7fac3220e640 1 Processor -- start 2024-03-06T07:28:30.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.368+0000 7fac3220e640 1 -- start start 2024-03-06T07:28:30.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.369+0000 7fac3220e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac2c0cfff0 0x7fac2c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.369+0000 7fac3220e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fac2c0cc8b0 0x7fac2c0ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.369+0000 7fac3220e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2c161980 con 0x7fac2c15bb20 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.369+0000 7fac3220e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac2c0d2500 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.369+0000 7fac3220e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c161c60 con 0x7fac2c0cfff0 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.369+0000 7fac2bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fac2c0cc8b0 0x7fac2c0ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.369+0000 7fac2bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fac2c0cc8b0 0x7fac2c0ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52236/0 (socket says 172.21.15.59:52236) 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.369+0000 7fac2bfff640 1 -- 172.21.15.59:0/2381456459 learned_addr learned my addr 172.21.15.59:0/2381456459 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.369+0000 7fac2affd640 1 --2- 172.21.15.59:0/2381456459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac2c0cfff0 0x7fac2c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.370+0000 7fac2bfff640 1 -- 172.21.15.59:0/2381456459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac2c0cfff0 msgr2=0x7fac2c0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.370+0000 7fac2bfff640 1 --2- 172.21.15.59:0/2381456459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac2c0cfff0 0x7fac2c0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.370+0000 7fac2bfff640 1 -- 172.21.15.59:0/2381456459 >> v1:172.21.15.164:6789/0 conn(0x7fac2c15bb20 legacy=0x7fac2c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.370+0000 7fac2bfff640 1 -- 172.21.15.59:0/2381456459 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c161f40 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac2bfff640 1 --2- 172.21.15.59:0/2381456459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac2c0cc8b0 0x7fac2c0ccd40 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fac24009e50 tx=0x7fac2402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efbeb9a7e62dda7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac2a7fc640 1 -- 172.21.15.59:0/2381456459 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2402cb00 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac2a7fc640 1 -- 172.21.15.59:0/2381456459 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2402cca0 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac2a7fc640 1 -- 172.21.15.59:0/2381456459 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac24034610 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac3220e640 1 -- 172.21.15.59:0/2381456459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac2c0cc8b0 msgr2=0x7fac2c0ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac3220e640 1 --2- 172.21.15.59:0/2381456459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac2c0cc8b0 0x7fac2c0ccd40 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fac24009e50 tx=0x7fac2402bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac3220e640 1 -- 172.21.15.59:0/2381456459 shutdown_connections 2024-03-06T07:28:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac3220e640 1 --2- 172.21.15.59:0/2381456459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac2c0cc8b0 0x7fac2c0ccd40 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac3220e640 1 --2- 172.21.15.59:0/2381456459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac2c0cfff0 0x7fac2c0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac3220e640 1 -- 172.21.15.59:0/2381456459 >> 172.21.15.59:0/2381456459 conn(0x7fac2c09a9f0 msgr2=0x7fac2c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac3220e640 1 -- 172.21.15.59:0/2381456459 shutdown_connections 2024-03-06T07:28:30.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.371+0000 7fac3220e640 1 -- 172.21.15.59:0/2381456459 wait complete. 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.373+0000 7fac3220e640 1 Processor -- start 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.373+0000 7fac3220e640 1 -- start start 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.373+0000 7fac3220e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac2c0cc8b0 0x7fac2c1554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.373+0000 7fac3220e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac2c0cfff0 0x7fac2c0c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.373+0000 7fac3220e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac2c15bb20 0x7fac2c15a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.373+0000 7fac3220e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c0d1980 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.373+0000 7fac3220e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c162500 con 0x7fac2c15bb20 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.373+0000 7fac3220e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c0d1b00 con 0x7fac2c0cfff0 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac2b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac2c0cc8b0 0x7fac2c1554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac2b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac2c0cc8b0 0x7fac2c1554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52242/0 (socket says 172.21.15.59:52242) 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac2b7fe640 1 -- 172.21.15.59:0/1421758965 learned_addr learned my addr 172.21.15.59:0/1421758965 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:30.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac2affd640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac2c0cfff0 0x7fac2c0c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac2bfff640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac2c15bb20 0x7fac2c15a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac2b7fe640 1 -- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac2c0cfff0 msgr2=0x7fac2c0c55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac2b7fe640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac2c0cfff0 0x7fac2c0c55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac2b7fe640 1 -- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac2c15bb20 msgr2=0x7fac2c15a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac2b7fe640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac2c15bb20 0x7fac2c15a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac2b7fe640 1 -- 172.21.15.59:0/1421758965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c0c6020 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac2b7fe640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac2c0cc8b0 0x7fac2c1554e0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00c3c0 tx=0x7fac1c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac13fff640 1 -- 172.21.15.59:0/1421758965 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1c0052d0 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac13fff640 1 -- 172.21.15.59:0/1421758965 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1c005490 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac3220e640 1 -- 172.21.15.59:0/1421758965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2c0c6300 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac13fff640 1 -- 172.21.15.59:0/1421758965 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1c007260 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.374+0000 7fac3220e640 1 -- 172.21.15.59:0/1421758965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2c0c68c0 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.375+0000 7fac3220e640 1 -- 172.21.15.59:0/1421758965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2c04f330 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.376+0000 7fac13fff640 1 -- 172.21.15.59:0/1421758965 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac1c007400 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.376+0000 7fac13fff640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac1806b4d0 0x7fac1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.376+0000 7fac13fff640 1 -- 172.21.15.59:0/1421758965 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fac1c0896c0 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.376+0000 7fac2affd640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac1806b4d0 0x7fac1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.377+0000 7fac2affd640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac1806b4d0 0x7fac1806d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fac20006fd0 tx=0x7fac20008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.380+0000 7fac13fff640 1 -- 172.21.15.59:0/1421758965 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac1c058520 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.603+0000 7fac3220e640 1 -- 172.21.15.59:0/1421758965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac2c09b0b0 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.603+0000 7fac13fff640 1 -- 172.21.15.59:0/1421758965 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fac1c057e20 con 0x7fac2c0cc8b0 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":73,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:14.345180+0000","modified":"2024-03-06T07:27:31.136752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8253":{"gid":8253,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":4,"addr":"172.21.15.59:6835/3442407979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3442407979},{"type":"v1","addr":"172.21.15.59:6835","nonce":3442407979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8253,"qdb_cluster":[8253]},"id":8}]} 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 73 2024-03-06T07:28:30.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 -- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac1806b4d0 msgr2=0x7fac1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac1806b4d0 0x7fac1806d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fac20006fd0 tx=0x7fac20008040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 -- 172.21.15.59:0/1421758965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac2c0cc8b0 msgr2=0x7fac2c1554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac2c0cc8b0 0x7fac2c1554e0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00c3c0 tx=0x7fac1c00c880 comp rx=0 tx=0).stop 2024-03-06T07:28:30.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 -- 172.21.15.59:0/1421758965 shutdown_connections 2024-03-06T07:28:30.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fac1806b4d0 0x7fac1806d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac2c15bb20 0x7fac2c15a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac2c0cfff0 0x7fac2c0c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 --2- 172.21.15.59:0/1421758965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac2c0cc8b0 0x7fac2c1554e0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 -- 172.21.15.59:0/1421758965 >> 172.21.15.59:0/1421758965 conn(0x7fac2c09a9f0 msgr2=0x7fac2c0b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 -- 172.21.15.59:0/1421758965 shutdown_connections 2024-03-06T07:28:30.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.606+0000 7fac3220e640 1 -- 172.21.15.59:0/1421758965 wait complete. 2024-03-06T07:28:30.619 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:30.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.719+0000 7ff6fd69a640 1 Processor -- start 2024-03-06T07:28:30.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.719+0000 7ff6fd69a640 1 -- start start 2024-03-06T07:28:30.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.720+0000 7ff6fd69a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff6f815e090 0x7ff6f80d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.720+0000 7ff6fd69a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f815e470 con 0x7ff6f80d0650 2024-03-06T07:28:30.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.720+0000 7ff6fd69a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f80d18f0 con 0x7ff6f80d2920 2024-03-06T07:28:30.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.720+0000 7ff6fd69a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f80b3cf0 con 0x7ff6f815e090 2024-03-06T07:28:30.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.720+0000 7ff6f6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff6f815e090 0x7ff6f80d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.720+0000 7ff6f77fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff6f80d0650 0x7ff6f80cf350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59070/0 (socket says 172.21.15.59:59070) 2024-03-06T07:28:30.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.720+0000 7ff6f77fe640 1 -- 172.21.15.59:0/3039384604 learned_addr learned my addr 172.21.15.59:0/3039384604 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:30.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.720+0000 7ff6f6ffd640 1 -- 172.21.15.59:0/3039384604 >> v1:172.21.15.164:6789/0 conn(0x7ff6f80d2920 legacy=0x7ff6f80b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.720+0000 7ff6f6ffd640 1 -- 172.21.15.59:0/3039384604 >> v1:172.21.15.59:6789/0 conn(0x7ff6f80d0650 legacy=0x7ff6f80cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.720+0000 7ff6f6ffd640 1 -- 172.21.15.59:0/3039384604 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f80d1b90 con 0x7ff6f815e090 2024-03-06T07:28:30.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.721+0000 7ff6f6ffd640 1 --2- 172.21.15.59:0/3039384604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6f815e090 0x7ff6f80d47a0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80097a0 tx=0x7ff6e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b4945d1620c25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.721+0000 7ff6f67fc640 1 -- 172.21.15.59:0/3039384604 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8034070 con 0x7ff6f815e090 2024-03-06T07:28:30.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.721+0000 7ff6f67fc640 1 -- 172.21.15.59:0/3039384604 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e8030610 con 0x7ff6f815e090 2024-03-06T07:28:30.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.721+0000 7ff6f67fc640 1 -- 172.21.15.59:0/3039384604 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80309e0 con 0x7ff6f815e090 2024-03-06T07:28:30.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.722+0000 7ff6fd69a640 1 -- 172.21.15.59:0/3039384604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6f815e090 msgr2=0x7ff6f80d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.722+0000 7ff6fd69a640 1 --2- 172.21.15.59:0/3039384604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6f815e090 0x7ff6f80d47a0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80097a0 tx=0x7ff6e802b960 comp rx=0 tx=0).stop 2024-03-06T07:28:30.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.722+0000 7ff6fd69a640 1 -- 172.21.15.59:0/3039384604 shutdown_connections 2024-03-06T07:28:30.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.722+0000 7ff6fd69a640 1 --2- 172.21.15.59:0/3039384604 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6f815e090 0x7ff6f80d47a0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.722+0000 7ff6fd69a640 1 -- 172.21.15.59:0/3039384604 >> 172.21.15.59:0/3039384604 conn(0x7ff6f80d3c80 msgr2=0x7ff6f80d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.722+0000 7ff6fd69a640 1 -- 172.21.15.59:0/3039384604 shutdown_connections 2024-03-06T07:28:30.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.722+0000 7ff6fd69a640 1 -- 172.21.15.59:0/3039384604 wait complete. 2024-03-06T07:28:30.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.723+0000 7ff6fd69a640 1 Processor -- start 2024-03-06T07:28:30.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.723+0000 7ff6fd69a640 1 -- start start 2024-03-06T07:28:30.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6fd69a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff6f80d0650 0x7ff6f81600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6fd69a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6f80d2920 0x7ff6f8161b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6fd69a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6f815e090 0x7ff6f8157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6fd69a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80b2df0 con 0x7ff6f815e090 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6fd69a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80cee30 con 0x7ff6f80d2920 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6fd69a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f80ce800 con 0x7ff6f80d0650 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6f77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6f815e090 0x7ff6f8157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6f77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6f815e090 0x7ff6f8157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52254/0 (socket says 172.21.15.59:52254) 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6f77fe640 1 -- 172.21.15.59:0/4155812299 learned_addr learned my addr 172.21.15.59:0/4155812299 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6f6ffd640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff6f80d0650 0x7ff6f81600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6ee7fc640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6f80d2920 0x7ff6f8161b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6f77fe640 1 -- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff6f80d0650 msgr2=0x7ff6f81600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6f77fe640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff6f80d0650 0x7ff6f81600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6f77fe640 1 -- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6f80d2920 msgr2=0x7ff6f8161b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6f77fe640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6f80d2920 0x7ff6f8161b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6f77fe640 1 -- 172.21.15.59:0/4155812299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f80b6c60 con 0x7ff6f815e090 2024-03-06T07:28:30.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.724+0000 7ff6ee7fc640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6f80d2920 0x7ff6f8161b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:30.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.725+0000 7ff6f77fe640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6f815e090 0x7ff6f8157200 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7ff6f000ca80 tx=0x7ff6f000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.725+0000 7ff6effff640 1 -- 172.21.15.59:0/4155812299 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0017070 con 0x7ff6f815e090 2024-03-06T07:28:30.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.725+0000 7ff6effff640 1 -- 172.21.15.59:0/4155812299 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f000ac30 con 0x7ff6f815e090 2024-03-06T07:28:30.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.725+0000 7ff6fd69a640 1 -- 172.21.15.59:0/4155812299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f80b6f40 con 0x7ff6f815e090 2024-03-06T07:28:30.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.725+0000 7ff6effff640 1 -- 172.21.15.59:0/4155812299 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0012600 con 0x7ff6f815e090 2024-03-06T07:28:30.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.725+0000 7ff6fd69a640 1 -- 172.21.15.59:0/4155812299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f80b7500 con 0x7ff6f815e090 2024-03-06T07:28:30.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.726+0000 7ff6fd69a640 1 -- 172.21.15.59:0/4155812299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f804f330 con 0x7ff6f815e090 2024-03-06T07:28:30.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.726+0000 7ff6effff640 1 -- 172.21.15.59:0/4155812299 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6f001a070 con 0x7ff6f815e090 2024-03-06T07:28:30.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.727+0000 7ff6effff640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6d806b500 0x7ff6d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.727+0000 7ff6effff640 1 -- 172.21.15.59:0/4155812299 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff6f0089df0 con 0x7ff6f815e090 2024-03-06T07:28:30.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.727+0000 7ff6f6ffd640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6d806b500 0x7ff6d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.727+0000 7ff6f6ffd640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6d806b500 0x7ff6d806d9c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8036000 tx=0x7ff6e803b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.731+0000 7ff6effff640 1 -- 172.21.15.59:0/4155812299 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6f000ae20 con 0x7ff6f815e090 2024-03-06T07:28:30.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.931+0000 7ff6fd69a640 1 -- 172.21.15.59:0/4155812299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6f80caac0 con 0x7ff6f815e090 2024-03-06T07:28:30.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.933+0000 7ff6effff640 1 -- 172.21.15.59:0/4155812299 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16843 (secure 0 0 0) 0x7ff6f0058620 con 0x7ff6f815e090 2024-03-06T07:28:30.935 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:30.935 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":114,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:27:38.442474+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:12.625799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:13.306376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.25","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:6836/2139041464":"2024-03-07T07:27:27.121928+0000","172.21.15.164:6834/2139041464":"2024-03-07T07:27:27.121928+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.934+0000 7ff6fd69a640 1 -- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6d806b500 msgr2=0x7ff6d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.934+0000 7ff6fd69a640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6d806b500 0x7ff6d806d9c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8036000 tx=0x7ff6e803b040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.934+0000 7ff6fd69a640 1 -- 172.21.15.59:0/4155812299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6f815e090 msgr2=0x7ff6f8157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.934+0000 7ff6fd69a640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6f815e090 0x7ff6f8157200 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7ff6f000ca80 tx=0x7ff6f000cf40 comp rx=0 tx=0).stop 2024-03-06T07:28:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.935+0000 7ff6fd69a640 1 -- 172.21.15.59:0/4155812299 shutdown_connections 2024-03-06T07:28:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.935+0000 7ff6fd69a640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff6d806b500 0x7ff6d806d9c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.935+0000 7ff6fd69a640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6f815e090 0x7ff6f8157200 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.935+0000 7ff6fd69a640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff6f80d2920 0x7ff6f8161b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.935+0000 7ff6fd69a640 1 --2- 172.21.15.59:0/4155812299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff6f80d0650 0x7ff6f81600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.935+0000 7ff6fd69a640 1 -- 172.21.15.59:0/4155812299 >> 172.21.15.59:0/4155812299 conn(0x7ff6f80d3c80 msgr2=0x7ff6f806bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.935+0000 7ff6fd69a640 1 -- 172.21.15.59:0/4155812299 shutdown_connections 2024-03-06T07:28:30.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:30.935+0000 7ff6fd69a640 1 -- 172.21.15.59:0/4155812299 wait complete. 2024-03-06T07:28:30.948 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:28:31.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.077+0000 7f475f35e640 1 Processor -- start 2024-03-06T07:28:31.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.078+0000 7f475f35e640 1 -- start start 2024-03-06T07:28:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.078+0000 7f475f35e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f475815e010 0x7f47580cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.078+0000 7f475f35e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f47580b3240 0x7f47580b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.078+0000 7f475f35e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475815e3f0 con 0x7f47580b2910 2024-03-06T07:28:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.078+0000 7f475f35e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47580b1f40 con 0x7f47580b3240 2024-03-06T07:28:31.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.078+0000 7f475f35e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47580b1c10 con 0x7f475815e010 2024-03-06T07:28:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.079+0000 7f475d0fa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f475815e010 0x7f47580cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.079+0000 7f475d8fb640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f47580b2910 0x7f47580d1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:44620/0 (socket says 172.21.15.59:44620) 2024-03-06T07:28:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.079+0000 7f475c8f9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f47580b3240 0x7f47580b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.079+0000 7f475d8fb640 1 -- 172.21.15.59:0/4238193315 learned_addr learned my addr 172.21.15.59:0/4238193315 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.079+0000 7f4757fff640 1 -- 172.21.15.59:0/4238193315 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207463882 0 0) 0x7f475815e3f0 con 0x7f47580b2910 2024-03-06T07:28:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.079+0000 7f4757fff640 1 -- 172.21.15.59:0/4238193315 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4740003520 con 0x7f47580b2910 2024-03-06T07:28:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.079+0000 7f475d0fa640 1 -- 172.21.15.59:0/4238193315 >> v2:172.21.15.164:3300/0 conn(0x7f47580b3240 msgr2=0x7f47580b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.079+0000 7f475d0fa640 1 --2- 172.21.15.59:0/4238193315 >> v2:172.21.15.164:3300/0 conn(0x7f47580b3240 0x7f47580b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.079+0000 7f475d0fa640 1 -- 172.21.15.59:0/4238193315 >> v1:172.21.15.164:6789/0 conn(0x7f47580b2910 legacy=0x7f47580d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.079+0000 7f475d0fa640 1 -- 172.21.15.59:0/4238193315 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475806d1b0 con 0x7f475815e010 2024-03-06T07:28:31.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.079+0000 7f475c8f9640 1 --2- 172.21.15.59:0/4238193315 >> v2:172.21.15.164:3300/0 conn(0x7f47580b3240 0x7f47580b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.080+0000 7f475d0fa640 1 --2- 172.21.15.59:0/4238193315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f475815e010 0x7f47580cedd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f47500097d0 tx=0x7f475002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa47e3582221d3f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.080+0000 7f4757fff640 1 -- 172.21.15.59:0/4238193315 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750031770 con 0x7f475815e010 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.080+0000 7f4757fff640 1 -- 172.21.15.59:0/4238193315 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4750037070 con 0x7f475815e010 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.080+0000 7f4757fff640 1 -- 172.21.15.59:0/4238193315 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750031b30 con 0x7f475815e010 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.081+0000 7f475f35e640 1 -- 172.21.15.59:0/4238193315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f475815e010 msgr2=0x7f47580cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.081+0000 7f475f35e640 1 --2- 172.21.15.59:0/4238193315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f475815e010 0x7f47580cedd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f47500097d0 tx=0x7f475002b810 comp rx=0 tx=0).stop 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.081+0000 7f475f35e640 1 -- 172.21.15.59:0/4238193315 shutdown_connections 2024-03-06T07:28:31.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.081+0000 7f475f35e640 1 --2- 172.21.15.59:0/4238193315 >> v2:172.21.15.164:3300/0 conn(0x7f47580b3240 0x7f47580b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.081+0000 7f475f35e640 1 --2- 172.21.15.59:0/4238193315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f475815e010 0x7f47580cedd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.081+0000 7f475f35e640 1 -- 172.21.15.59:0/4238193315 >> 172.21.15.59:0/4238193315 conn(0x7f475809a9f0 msgr2=0x7f475809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.081+0000 7f475f35e640 1 -- 172.21.15.59:0/4238193315 shutdown_connections 2024-03-06T07:28:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.081+0000 7f475f35e640 1 -- 172.21.15.59:0/4238193315 wait complete. 2024-03-06T07:28:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.082+0000 7f475f35e640 1 Processor -- start 2024-03-06T07:28:31.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.082+0000 7f475f35e640 1 -- start start 2024-03-06T07:28:31.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475f35e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f475815e010 0x7f47580c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475f35e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47581611e0 0x7f4758159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475f35e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47581f44b0 0x7f475815ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475f35e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47580cf310 con 0x7f47581f44b0 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475f35e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475806ced0 con 0x7f47581611e0 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475f35e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47580d21e0 con 0x7f475815e010 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475d8fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47581f44b0 0x7f475815ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475d8fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47581f44b0 0x7f475815ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52264/0 (socket says 172.21.15.59:52264) 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475d8fb640 1 -- 172.21.15.59:0/2610751273 learned_addr learned my addr 172.21.15.59:0/2610751273 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475d0fa640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f475815e010 0x7f47580c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475c8f9640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47581611e0 0x7f4758159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475d8fb640 1 -- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f475815e010 msgr2=0x7f47580c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475d8fb640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f475815e010 0x7f47580c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475d8fb640 1 -- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47581611e0 msgr2=0x7f4758159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475d8fb640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47581611e0 0x7f4758159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475d8fb640 1 -- 172.21.15.59:0/2610751273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475815a130 con 0x7f47581f44b0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.083+0000 7f475d8fb640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47581f44b0 0x7f475815ea80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f474800cae0 tx=0x7f474800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.084+0000 7f4755ffb640 1 -- 172.21.15.59:0/2610751273 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748017070 con 0x7f47581f44b0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.084+0000 7f4755ffb640 1 -- 172.21.15.59:0/2610751273 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474800abe0 con 0x7f47581f44b0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.084+0000 7f475f35e640 1 -- 172.21.15.59:0/2610751273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4758160420 con 0x7f47581f44b0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.084+0000 7f4755ffb640 1 -- 172.21.15.59:0/2610751273 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748012400 con 0x7f47581f44b0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.084+0000 7f475f35e640 1 -- 172.21.15.59:0/2610751273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47581609e0 con 0x7f47581f44b0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.085+0000 7f4755ffb640 1 -- 172.21.15.59:0/2610751273 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f474801a070 con 0x7f47581f44b0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.085+0000 7f475f35e640 1 -- 172.21.15.59:0/2610751273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475804f330 con 0x7f47581f44b0 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.085+0000 7f4755ffb640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f474406b500 0x7f474406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.086+0000 7f4755ffb640 1 -- 172.21.15.59:0/2610751273 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4748089d70 con 0x7f47581f44b0 2024-03-06T07:28:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.086+0000 7f475d0fa640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f474406b500 0x7f474406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.086+0000 7f475d0fa640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f474406b500 0x7f474406d9c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4750036000 tx=0x7f475003d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.090+0000 7f4755ffb640 1 -- 172.21.15.59:0/2610751273 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f475815a130 con 0x7f47581f44b0 2024-03-06T07:28:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.422+0000 7f475f35e640 1 -- 172.21.15.59:0/2610751273 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f47580d2980 con 0x7f47581f44b0 2024-03-06T07:28:31.424 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:28:31.422+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:31.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.424+0000 7f4755ffb640 1 -- 172.21.15.59:0/2610751273 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) v1 ==== 161+0+0 (secure 0 0 0) 0x7f47480585a0 con 0x7f47581f44b0 2024-03-06T07:28:31.426 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 -- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f474406b500 msgr2=0x7f474406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f474406b500 0x7f474406d9c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4750036000 tx=0x7f475003d040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 -- 172.21.15.59:0/2610751273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47581f44b0 msgr2=0x7f475815ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47581f44b0 0x7f475815ea80 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f474800cae0 tx=0x7f474800a060 comp rx=0 tx=0).stop 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 -- 172.21.15.59:0/2610751273 shutdown_connections 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f474406b500 0x7f474406d9c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47581f44b0 0x7f475815ea80 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47581611e0 0x7f4758159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 --2- 172.21.15.59:0/2610751273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f475815e010 0x7f47580c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 -- 172.21.15.59:0/2610751273 >> 172.21.15.59:0/2610751273 conn(0x7f475809a9f0 msgr2=0x7f47580ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 -- 172.21.15.59:0/2610751273 shutdown_connections 2024-03-06T07:28:31.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.427+0000 7f475f35e640 1 -- 172.21.15.59:0/2610751273 wait complete. 2024-03-06T07:28:31.433 INFO:tasks.ceph.mds.c.smithi059.stderr: -8790> 2024-03-06T07:27:34.131+0000 7f0d99d7e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:31.442 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:28:31.467 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.544+0000 7f92c1c64640 1 Processor -- start 2024-03-06T07:28:31.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.544+0000 7f92c1c64640 1 -- start start 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92c1c64640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f92bc0d4860 0x7f92bc0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92c1c64640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f92bc0b0e30 0x7f92bc0b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92c1c64640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92bc1618b0 con 0x7f92bc1614d0 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92c1c64640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92bc0d4cf0 con 0x7f92bc0d4860 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92c1c64640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92bc0cf400 con 0x7f92bc0b0e30 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92bb7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f92bc1614d0 0x7f92bc0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59080/0 (socket says 172.21.15.59:59080) 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92bb7fe640 1 -- 172.21.15.59:0/374897680 learned_addr learned my addr 172.21.15.59:0/374897680 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92baffd640 1 --2- 172.21.15.59:0/374897680 >> v2:172.21.15.59:3300/0 conn(0x7f92bc0d4860 0x7f92bc0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92bbfff640 1 --2- 172.21.15.59:0/374897680 >> v2:172.21.15.164:3300/0 conn(0x7f92bc0b0e30 0x7f92bc0b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92baffd640 1 -- 172.21.15.59:0/374897680 >> v2:172.21.15.164:3300/0 conn(0x7f92bc0b0e30 msgr2=0x7f92bc0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92baffd640 1 --2- 172.21.15.59:0/374897680 >> v2:172.21.15.164:3300/0 conn(0x7f92bc0b0e30 0x7f92bc0b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92baffd640 1 -- 172.21.15.59:0/374897680 >> v1:172.21.15.59:6789/0 conn(0x7f92bc1614d0 legacy=0x7f92bc0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92baffd640 1 -- 172.21.15.59:0/374897680 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc0cf6a0 con 0x7f92bc0d4860 2024-03-06T07:28:31.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.546+0000 7f92bbfff640 1 --2- 172.21.15.59:0/374897680 >> v2:172.21.15.164:3300/0 conn(0x7f92bc0b0e30 0x7f92bc0b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.547+0000 7f92baffd640 1 --2- 172.21.15.59:0/374897680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92bc0d4860 0x7f92bc0d2020 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0b2a70 tx=0x7f92b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48310d2bf82a2f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.547+0000 7f92ba7fc640 1 -- 172.21.15.59:0/374897680 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b0035020 con 0x7f92bc0d4860 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.547+0000 7f92ba7fc640 1 -- 172.21.15.59:0/374897680 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b0031830 con 0x7f92bc0d4860 2024-03-06T07:28:31.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.547+0000 7f92ba7fc640 1 -- 172.21.15.59:0/374897680 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b0031c00 con 0x7f92bc0d4860 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.548+0000 7f92c1c64640 1 -- 172.21.15.59:0/374897680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92bc0d4860 msgr2=0x7f92bc0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.548+0000 7f92c1c64640 1 --2- 172.21.15.59:0/374897680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92bc0d4860 0x7f92bc0d2020 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0b2a70 tx=0x7f92b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.548+0000 7f92c1c64640 1 -- 172.21.15.59:0/374897680 shutdown_connections 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.548+0000 7f92c1c64640 1 --2- 172.21.15.59:0/374897680 >> v2:172.21.15.164:3300/0 conn(0x7f92bc0b0e30 0x7f92bc0b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.548+0000 7f92c1c64640 1 --2- 172.21.15.59:0/374897680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92bc0d4860 0x7f92bc0d2020 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.548+0000 7f92c1c64640 1 -- 172.21.15.59:0/374897680 >> 172.21.15.59:0/374897680 conn(0x7f92bc0d3d40 msgr2=0x7f92bc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.548+0000 7f92c1c64640 1 -- 172.21.15.59:0/374897680 shutdown_connections 2024-03-06T07:28:31.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.548+0000 7f92c1c64640 1 -- 172.21.15.59:0/374897680 wait complete. 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.550+0000 7f92c1c64640 1 Processor -- start 2024-03-06T07:28:31.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.550+0000 7f92c1c64640 1 -- start start 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92c1c64640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92bc0b0e30 0x7f92bc14f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92c1c64640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92bc0d4860 0x7f92bc0cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92c1c64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc1614d0 0x7f92bc0c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92c1c64640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc0b0910 con 0x7f92bc0d4860 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92c1c64640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc0cfc60 con 0x7f92bc0b0e30 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92c1c64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc0d2560 con 0x7f92bc1614d0 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92baffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92bc0d4860 0x7f92bc0cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92baffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92bc0d4860 0x7f92bc0cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52292/0 (socket says 172.21.15.59:52292) 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92baffd640 1 -- 172.21.15.59:0/1140238962 learned_addr learned my addr 172.21.15.59:0/1140238962 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:31.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92bbfff640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc1614d0 0x7f92bc0c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92bb7fe640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92bc0b0e30 0x7f92bc14f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92baffd640 1 -- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc1614d0 msgr2=0x7f92bc0c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92baffd640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc1614d0 0x7f92bc0c8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92baffd640 1 -- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92bc0b0e30 msgr2=0x7f92bc14f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92baffd640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92bc0b0e30 0x7f92bc14f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92baffd640 1 -- 172.21.15.59:0/1140238962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc0c7cc0 con 0x7f92bc0d4860 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92bbfff640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc1614d0 0x7f92bc0c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92bb7fe640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92bc0b0e30 0x7f92bc14f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92baffd640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92bc0d4860 0x7f92bc0cd240 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009890 tx=0x7f92b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92a3fff640 1 -- 172.21.15.59:0/1140238962 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b0041070 con 0x7f92bc0d4860 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92a3fff640 1 -- 172.21.15.59:0/1140238962 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b003aa30 con 0x7f92bc0d4860 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92a3fff640 1 -- 172.21.15.59:0/1140238962 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b0035040 con 0x7f92bc0d4860 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.551+0000 7f92c1c64640 1 -- 172.21.15.59:0/1140238962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92bc0c7fa0 con 0x7f92bc0d4860 2024-03-06T07:28:31.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.552+0000 7f92c1c64640 1 -- 172.21.15.59:0/1140238962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92bc0c8560 con 0x7f92bc0d4860 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.552+0000 7f92c1c64640 1 -- 172.21.15.59:0/1140238962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92bc068ff0 con 0x7f92bc0d4860 2024-03-06T07:28:31.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.553+0000 7f92a3fff640 1 -- 172.21.15.59:0/1140238962 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92b0038020 con 0x7f92bc0d4860 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.553+0000 7f92a3fff640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92a806b4d0 0x7f92a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.553+0000 7f92a3fff640 1 -- 172.21.15.59:0/1140238962 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f92b00abd80 con 0x7f92bc0d4860 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.554+0000 7f92bb7fe640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92a806b4d0 0x7f92a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.554+0000 7f92bb7fe640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92a806b4d0 0x7f92a806d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0012e0 tx=0x7f92ac009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.558+0000 7f92a3fff640 1 -- 172.21.15.59:0/1140238962 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92b007ab10 con 0x7f92bc0d4860 2024-03-06T07:28:31.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:31.776+0000 7f92c1c64640 1 -- 172.21.15.59:0/1140238962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f92bc069fc0 con 0x7f92bc0d4860 2024-03-06T07:28:32.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.425+0000 7f92a3fff640 1 -- 172.21.15.59:0/1140238962 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7f92b007a410 con 0x7f92bc0d4860 2024-03-06T07:28:32.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 -- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92a806b4d0 msgr2=0x7f92a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92a806b4d0 0x7f92a806d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0012e0 tx=0x7f92ac009040 comp rx=0 tx=0).stop 2024-03-06T07:28:32.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 -- 172.21.15.59:0/1140238962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92bc0d4860 msgr2=0x7f92bc0cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92bc0d4860 0x7f92bc0cd240 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009890 tx=0x7f92b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 -- 172.21.15.59:0/1140238962 shutdown_connections 2024-03-06T07:28:32.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f92a806b4d0 0x7f92a806d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92bc1614d0 0x7f92bc0c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92bc0d4860 0x7f92bc0cd240 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 --2- 172.21.15.59:0/1140238962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92bc0b0e30 0x7f92bc14f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 -- 172.21.15.59:0/1140238962 >> 172.21.15.59:0/1140238962 conn(0x7f92bc0d3d40 msgr2=0x7f92bc0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 -- 172.21.15.59:0/1140238962 shutdown_connections 2024-03-06T07:28:32.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.429+0000 7f92c1c64640 1 -- 172.21.15.59:0/1140238962 wait complete. 2024-03-06T07:28:32.445 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.596+0000 7f1c3ce9d640 1 Processor -- start 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.596+0000 7f1c3ce9d640 1 -- start start 2024-03-06T07:28:32.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.597+0000 7f1c3ce9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1c3806be80 0x7f1c380b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.597+0000 7f1c3ce9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1c380b5ca0 0x7f1c3806c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.597+0000 7f1c3ce9d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3806c7a0 con 0x7f1c38072590 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.597+0000 7f1c3ce9d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c3815edf0 con 0x7f1c3806be80 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.597+0000 7f1c3ce9d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c381618a0 con 0x7f1c380b5ca0 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.597+0000 7f1c3659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1c3806be80 0x7f1c380b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.597+0000 7f1c36d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1c38072590 0x7f1c380729a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59086/0 (socket says 172.21.15.59:59086) 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.597+0000 7f1c36d9d640 1 -- 172.21.15.59:0/759409233 learned_addr learned my addr 172.21.15.59:0/759409233 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.598+0000 7f1c35d9b640 1 --2- 172.21.15.59:0/759409233 >> v2:172.21.15.164:3300/0 conn(0x7f1c380b5ca0 0x7f1c3806c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.598+0000 7f1c3659c640 1 -- 172.21.15.59:0/759409233 >> v2:172.21.15.164:3300/0 conn(0x7f1c380b5ca0 msgr2=0x7f1c3806c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.598+0000 7f1c3659c640 1 --2- 172.21.15.59:0/759409233 >> v2:172.21.15.164:3300/0 conn(0x7f1c380b5ca0 0x7f1c3806c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.598+0000 7f1c3659c640 1 -- 172.21.15.59:0/759409233 >> v1:172.21.15.59:6789/0 conn(0x7f1c38072590 legacy=0x7f1c380729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.598+0000 7f1c3659c640 1 -- 172.21.15.59:0/759409233 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c38068b00 con 0x7f1c3806be80 2024-03-06T07:28:32.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.598+0000 7f1c3559a640 1 -- 172.21.15.59:0/759409233 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096820209 0 0) 0x7f1c3806c7a0 con 0x7f1c38072590 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3659c640 1 --2- 172.21.15.59:0/759409233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c3806be80 0x7f1c380b11b0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f1c38071130 tx=0x7f1c3002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8313f95b1f4234 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3559a640 1 -- 172.21.15.59:0/759409233 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c300315a0 con 0x7f1c3806be80 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3559a640 1 -- 172.21.15.59:0/759409233 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c30037070 con 0x7f1c3806be80 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3559a640 1 -- 172.21.15.59:0/759409233 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30031960 con 0x7f1c3806be80 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/759409233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c3806be80 msgr2=0x7f1c380b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3ce9d640 1 --2- 172.21.15.59:0/759409233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c3806be80 0x7f1c380b11b0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f1c38071130 tx=0x7f1c3002b840 comp rx=0 tx=0).stop 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/759409233 shutdown_connections 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3ce9d640 1 --2- 172.21.15.59:0/759409233 >> v2:172.21.15.164:3300/0 conn(0x7f1c380b5ca0 0x7f1c3806c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3ce9d640 1 --2- 172.21.15.59:0/759409233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c3806be80 0x7f1c380b11b0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/759409233 >> 172.21.15.59:0/759409233 conn(0x7f1c380b4ed0 msgr2=0x7f1c380b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/759409233 shutdown_connections 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.599+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/759409233 wait complete. 2024-03-06T07:28:32.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.601+0000 7f1c3ce9d640 1 Processor -- start 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.601+0000 7f1c3ce9d640 1 -- start start 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.601+0000 7f1c3ce9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c3806be80 0x7f1c3814b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.601+0000 7f1c3ce9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c38072590 0x7f1c380cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.601+0000 7f1c3ce9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c380b5ca0 0x7f1c3814b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.601+0000 7f1c3ce9d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3815e7a0 con 0x7f1c38072590 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.601+0000 7f1c3ce9d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c380b1900 con 0x7f1c380b5ca0 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.601+0000 7f1c3ce9d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3815ea80 con 0x7f1c3806be80 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c35d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c38072590 0x7f1c380cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c35d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c38072590 0x7f1c380cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52302/0 (socket says 172.21.15.59:52302) 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c35d9b640 1 -- 172.21.15.59:0/4153450083 learned_addr learned my addr 172.21.15.59:0/4153450083 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c3659c640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c3806be80 0x7f1c3814b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c36d9d640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c380b5ca0 0x7f1c3814b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c35d9b640 1 -- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c3806be80 msgr2=0x7f1c3814b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c35d9b640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c3806be80 0x7f1c3814b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c35d9b640 1 -- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c380b5ca0 msgr2=0x7f1c3814b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c35d9b640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c380b5ca0 0x7f1c3814b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c35d9b640 1 -- 172.21.15.59:0/4153450083 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c381f3460 con 0x7f1c38072590 2024-03-06T07:28:32.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c36d9d640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c380b5ca0 0x7f1c3814b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c35d9b640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c38072590 0x7f1c380cbaa0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000d5f0 tx=0x7f1c2000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c2effd640 1 -- 172.21.15.59:0/4153450083 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c200125f0 con 0x7f1c38072590 2024-03-06T07:28:32.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c2effd640 1 -- 172.21.15.59:0/4153450083 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c20019070 con 0x7f1c38072590 2024-03-06T07:28:32.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/4153450083 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3814b640 con 0x7f1c38072590 2024-03-06T07:28:32.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c2effd640 1 -- 172.21.15.59:0/4153450083 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c200129d0 con 0x7f1c38072590 2024-03-06T07:28:32.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.602+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/4153450083 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c380cace0 con 0x7f1c38072590 2024-03-06T07:28:32.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.603+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/4153450083 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3804f330 con 0x7f1c38072590 2024-03-06T07:28:32.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.604+0000 7f1c2effd640 1 -- 172.21.15.59:0/4153450083 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c20012b70 con 0x7f1c38072590 2024-03-06T07:28:32.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.604+0000 7f1c2effd640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c1806b4d0 0x7f1c1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.604+0000 7f1c2effd640 1 -- 172.21.15.59:0/4153450083 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f1c2008a6d0 con 0x7f1c38072590 2024-03-06T07:28:32.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.604+0000 7f1c3659c640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c1806b4d0 0x7f1c1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.605+0000 7f1c3659c640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c1806b4d0 0x7f1c1806d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1c30036000 tx=0x7f1c30008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.608+0000 7f1c2effd640 1 -- 172.21.15.59:0/4153450083 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c20059470 con 0x7f1c38072590 2024-03-06T07:28:32.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:32.787+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/4153450083 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1c380cbfe0 con 0x7f1c38072590 2024-03-06T07:28:33.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.559+0000 7f1c2effd640 1 -- 172.21.15.59:0/4153450083 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v116) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1c20058d70 con 0x7f1c38072590 2024-03-06T07:28:33.561 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.562+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c1806b4d0 msgr2=0x7f1c1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.562+0000 7f1c3ce9d640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c1806b4d0 0x7f1c1806d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1c30036000 tx=0x7f1c30008030 comp rx=0 tx=0).stop 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.562+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/4153450083 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c38072590 msgr2=0x7f1c380cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.562+0000 7f1c3ce9d640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c38072590 0x7f1c380cbaa0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000d5f0 tx=0x7f1c2000dab0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.562+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/4153450083 shutdown_connections 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.562+0000 7f1c3ce9d640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1c1806b4d0 0x7f1c1806d990 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.562+0000 7f1c3ce9d640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c380b5ca0 0x7f1c3814b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.562+0000 7f1c3ce9d640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1c38072590 0x7f1c380cbaa0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.562+0000 7f1c3ce9d640 1 --2- 172.21.15.59:0/4153450083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c3806be80 0x7f1c3814b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.562+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/4153450083 >> 172.21.15.59:0/4153450083 conn(0x7f1c380b4ed0 msgr2=0x7f1c3806de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.563+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/4153450083 shutdown_connections 2024-03-06T07:28:33.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.563+0000 7f1c3ce9d640 1 -- 172.21.15.59:0/4153450083 wait complete. 2024-03-06T07:28:33.582 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:28:33.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.691+0000 7fe7fa313640 1 Processor -- start 2024-03-06T07:28:33.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.691+0000 7fe7fa313640 1 -- start start 2024-03-06T07:28:33.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7fa313640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe7f406be80 0x7fe7f40b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7fa313640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f40b5ca0 0x7fe7f406c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7fa313640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f406c7a0 con 0x7fe7f4072590 2024-03-06T07:28:33.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7fa313640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f415edf0 con 0x7fe7f406be80 2024-03-06T07:28:33.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7fa313640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f41618a0 con 0x7fe7f40b5ca0 2024-03-06T07:28:33.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7f3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe7f406be80 0x7fe7f40b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7f88b0640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe7f4072590 0x7fe7f4072a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40170/0 (socket says 172.21.15.59:40170) 2024-03-06T07:28:33.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7f88b0640 1 -- 172.21.15.59:0/619652570 learned_addr learned my addr 172.21.15.59:0/619652570 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:33.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7f37fe640 1 --2- 172.21.15.59:0/619652570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f40b5ca0 0x7fe7f406c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7f3fff640 1 -- 172.21.15.59:0/619652570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f40b5ca0 msgr2=0x7fe7f406c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7f3fff640 1 --2- 172.21.15.59:0/619652570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f40b5ca0 0x7fe7f406c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7f3fff640 1 -- 172.21.15.59:0/619652570 >> v1:172.21.15.59:6789/0 conn(0x7fe7f4072590 legacy=0x7fe7f4072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.692+0000 7fe7f2ffd640 1 -- 172.21.15.59:0/619652570 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082446102 0 0) 0x7fe7f406c7a0 con 0x7fe7f4072590 2024-03-06T07:28:33.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.693+0000 7fe7f3fff640 1 -- 172.21.15.59:0/619652570 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f4068b00 con 0x7fe7f406be80 2024-03-06T07:28:33.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.693+0000 7fe7f3fff640 1 --2- 172.21.15.59:0/619652570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7f406be80 0x7fe7f40b11b0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80097a0 tx=0x7fe7e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9358b9eeff08ab9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.693+0000 7fe7f2ffd640 1 -- 172.21.15.59:0/619652570 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e80315a0 con 0x7fe7f406be80 2024-03-06T07:28:33.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.693+0000 7fe7f2ffd640 1 -- 172.21.15.59:0/619652570 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e8037070 con 0x7fe7f406be80 2024-03-06T07:28:33.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.693+0000 7fe7f2ffd640 1 -- 172.21.15.59:0/619652570 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8031960 con 0x7fe7f406be80 2024-03-06T07:28:33.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.694+0000 7fe7fa313640 1 -- 172.21.15.59:0/619652570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7f406be80 msgr2=0x7fe7f40b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.694+0000 7fe7fa313640 1 --2- 172.21.15.59:0/619652570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7f406be80 0x7fe7f40b11b0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80097a0 tx=0x7fe7e802b840 comp rx=0 tx=0).stop 2024-03-06T07:28:33.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.694+0000 7fe7fa313640 1 -- 172.21.15.59:0/619652570 shutdown_connections 2024-03-06T07:28:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.694+0000 7fe7fa313640 1 --2- 172.21.15.59:0/619652570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f40b5ca0 0x7fe7f406c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.694+0000 7fe7fa313640 1 --2- 172.21.15.59:0/619652570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7f406be80 0x7fe7f40b11b0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.694+0000 7fe7fa313640 1 -- 172.21.15.59:0/619652570 >> 172.21.15.59:0/619652570 conn(0x7fe7f40b4ed0 msgr2=0x7fe7f40b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.694+0000 7fe7fa313640 1 -- 172.21.15.59:0/619652570 shutdown_connections 2024-03-06T07:28:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.694+0000 7fe7fa313640 1 -- 172.21.15.59:0/619652570 wait complete. 2024-03-06T07:28:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.695+0000 7fe7fa313640 1 Processor -- start 2024-03-06T07:28:33.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.695+0000 7fe7fa313640 1 -- start start 2024-03-06T07:28:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7fa313640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7f406be80 0x7fe7f406dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7fa313640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f4072590 0x7fe7f414d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7fa313640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f40b5ca0 0x7fe7f414d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7fa313640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f415e7a0 con 0x7fe7f406be80 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7fa313640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f40b1900 con 0x7fe7f40b5ca0 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7fa313640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f415ea80 con 0x7fe7f4072590 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7f3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7f406be80 0x7fe7f406dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7f3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7f406be80 0x7fe7f406dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60038/0 (socket says 172.21.15.59:60038) 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7f3fff640 1 -- 172.21.15.59:0/1736705104 learned_addr learned my addr 172.21.15.59:0/1736705104 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7f3fff640 1 -- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f4072590 msgr2=0x7fe7f414d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7f3fff640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f4072590 0x7fe7f414d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7f3fff640 1 -- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f40b5ca0 msgr2=0x7fe7f414d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7f3fff640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f40b5ca0 0x7fe7f414d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.696+0000 7fe7f3fff640 1 -- 172.21.15.59:0/1736705104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f40cbe70 con 0x7fe7f406be80 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.697+0000 7fe7f3fff640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7f406be80 0x7fe7f406dd40 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8034040 tx=0x7fe7e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.697+0000 7fe7f0ff9640 1 -- 172.21.15.59:0/1736705104 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8032840 con 0x7fe7f406be80 2024-03-06T07:28:33.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.697+0000 7fe7f0ff9640 1 -- 172.21.15.59:0/1736705104 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e8032a00 con 0x7fe7f406be80 2024-03-06T07:28:33.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.697+0000 7fe7fa313640 1 -- 172.21.15.59:0/1736705104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f41570e0 con 0x7fe7f406be80 2024-03-06T07:28:33.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.697+0000 7fe7f0ff9640 1 -- 172.21.15.59:0/1736705104 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8032df0 con 0x7fe7f406be80 2024-03-06T07:28:33.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.697+0000 7fe7fa313640 1 -- 172.21.15.59:0/1736705104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f41576a0 con 0x7fe7f406be80 2024-03-06T07:28:33.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.698+0000 7fe7fa313640 1 -- 172.21.15.59:0/1736705104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f4067330 con 0x7fe7f406be80 2024-03-06T07:28:33.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.698+0000 7fe7f0ff9640 1 -- 172.21.15.59:0/1736705104 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7e803b3f0 con 0x7fe7f406be80 2024-03-06T07:28:33.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.699+0000 7fe7f0ff9640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7e006b520 0x7fe7e006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.699+0000 7fe7f0ff9640 1 -- 172.21.15.59:0/1736705104 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5501+0+0 (secure 0 0 0) 0x7fe7e802f070 con 0x7fe7f406be80 2024-03-06T07:28:33.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.699+0000 7fe7f37fe640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7e006b520 0x7fe7e006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.700+0000 7fe7f37fe640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7e006b520 0x7fe7e006d9e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4006fd0 tx=0x7fe7e4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.703+0000 7fe7f0ff9640 1 -- 172.21.15.59:0/1736705104 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7f4067330 con 0x7fe7f406be80 2024-03-06T07:28:33.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:33.885+0000 7fe7fa313640 1 -- 172.21.15.59:0/1736705104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe7f404f330 con 0x7fe7f406be80 2024-03-06T07:28:34.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.568+0000 7fe7f0ff9640 1 -- 172.21.15.59:0/1736705104 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v117) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe7e8040070 con 0x7fe7f406be80 2024-03-06T07:28:34.570 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-03-06T07:28:34.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.571+0000 7fe7fa313640 1 -- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7e006b520 msgr2=0x7fe7e006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.571+0000 7fe7fa313640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7e006b520 0x7fe7e006d9e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4006fd0 tx=0x7fe7e4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.571+0000 7fe7fa313640 1 -- 172.21.15.59:0/1736705104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7f406be80 msgr2=0x7fe7f406dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.571+0000 7fe7fa313640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7f406be80 0x7fe7f406dd40 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8034040 tx=0x7fe7e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.573+0000 7fe7fa313640 1 -- 172.21.15.59:0/1736705104 shutdown_connections 2024-03-06T07:28:34.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.573+0000 7fe7fa313640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe7e006b520 0x7fe7e006d9e0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.573+0000 7fe7fa313640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f40b5ca0 0x7fe7f414d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.573+0000 7fe7fa313640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f4072590 0x7fe7f414d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.573+0000 7fe7fa313640 1 --2- 172.21.15.59:0/1736705104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7f406be80 0x7fe7f406dd40 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.573+0000 7fe7fa313640 1 -- 172.21.15.59:0/1736705104 >> 172.21.15.59:0/1736705104 conn(0x7fe7f40b4ed0 msgr2=0x7fe7f40b5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.574+0000 7fe7fa313640 1 -- 172.21.15.59:0/1736705104 shutdown_connections 2024-03-06T07:28:34.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.574+0000 7fe7fa313640 1 -- 172.21.15.59:0/1736705104 wait complete. 2024-03-06T07:28:34.598 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:28:34.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.706+0000 7fc51b088640 1 Processor -- start 2024-03-06T07:28:34.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.706+0000 7fc51b088640 1 -- start start 2024-03-06T07:28:34.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.706+0000 7fc51b088640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5141614d0 0x7fc5140d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.706+0000 7fc51b088640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc5140d2480 0x7fc5140b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.706+0000 7fc51b088640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5141618b0 con 0x7fc5140d1910 2024-03-06T07:28:34.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.706+0000 7fc51b088640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5140cf400 con 0x7fc5140d2480 2024-03-06T07:28:34.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.706+0000 7fc51b088640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5140cf6a0 con 0x7fc5141614d0 2024-03-06T07:28:34.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.707+0000 7fc519625640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc5140d2480 0x7fc5140b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.707+0000 7fc519625640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc5140d2480 0x7fc5140b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60048/0 (socket says 172.21.15.59:60048) 2024-03-06T07:28:34.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.707+0000 7fc519625640 1 -- 172.21.15.59:0/1474011655 learned_addr learned my addr 172.21.15.59:0/1474011655 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:34.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.707+0000 7fc518e24640 1 --2- 172.21.15.59:0/1474011655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5141614d0 0x7fc5140d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.707+0000 7fc519625640 1 -- 172.21.15.59:0/1474011655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5141614d0 msgr2=0x7fc5140d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.707+0000 7fc519625640 1 --2- 172.21.15.59:0/1474011655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5141614d0 0x7fc5140d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.707+0000 7fc519625640 1 -- 172.21.15.59:0/1474011655 >> v1:172.21.15.164:6789/0 conn(0x7fc5140d1910 legacy=0x7fc5140d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.707+0000 7fc519625640 1 -- 172.21.15.59:0/1474011655 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5140cf980 con 0x7fc5140d2480 2024-03-06T07:28:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc519625640 1 --2- 172.21.15.59:0/1474011655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5140d2480 0x7fc5140b0e30 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fc504009870 tx=0x7fc50402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e347dc1c786bf102 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc5137fe640 1 -- 172.21.15.59:0/1474011655 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504034070 con 0x7fc5140d2480 2024-03-06T07:28:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc5137fe640 1 -- 172.21.15.59:0/1474011655 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc504030840 con 0x7fc5140d2480 2024-03-06T07:28:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc5137fe640 1 -- 172.21.15.59:0/1474011655 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504030c10 con 0x7fc5140d2480 2024-03-06T07:28:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc51b088640 1 -- 172.21.15.59:0/1474011655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5140d2480 msgr2=0x7fc5140b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc51b088640 1 --2- 172.21.15.59:0/1474011655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5140d2480 0x7fc5140b0e30 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fc504009870 tx=0x7fc50402b980 comp rx=0 tx=0).stop 2024-03-06T07:28:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc51b088640 1 -- 172.21.15.59:0/1474011655 shutdown_connections 2024-03-06T07:28:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc51b088640 1 --2- 172.21.15.59:0/1474011655 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5140d2480 0x7fc5140b0e30 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc51b088640 1 --2- 172.21.15.59:0/1474011655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5141614d0 0x7fc5140d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc51b088640 1 -- 172.21.15.59:0/1474011655 >> 172.21.15.59:0/1474011655 conn(0x7fc5140d3d40 msgr2=0x7fc5140d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc51b088640 1 -- 172.21.15.59:0/1474011655 shutdown_connections 2024-03-06T07:28:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.708+0000 7fc51b088640 1 -- 172.21.15.59:0/1474011655 wait complete. 2024-03-06T07:28:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.710+0000 7fc51b088640 1 Processor -- start 2024-03-06T07:28:34.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.710+0000 7fc51b088640 1 -- start start 2024-03-06T07:28:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.710+0000 7fc51b088640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5140d1910 0x7fc51415bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.710+0000 7fc51b088640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5140d2480 0x7fc5140b8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.710+0000 7fc51b088640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5141614d0 0x7fc51415dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.710+0000 7fc51b088640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5140b1370 con 0x7fc5140d1910 2024-03-06T07:28:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.710+0000 7fc51b088640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5140b0910 con 0x7fc5140d2480 2024-03-06T07:28:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.710+0000 7fc51b088640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5140b14f0 con 0x7fc5141614d0 2024-03-06T07:28:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.710+0000 7fc518e24640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5140d1910 0x7fc51415bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc518e24640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5140d1910 0x7fc51415bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60060/0 (socket says 172.21.15.59:60060) 2024-03-06T07:28:34.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc518e24640 1 -- 172.21.15.59:0/395641287 learned_addr learned my addr 172.21.15.59:0/395641287 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc519625640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5141614d0 0x7fc51415dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc513fff640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5140d2480 0x7fc5140b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc518e24640 1 -- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5141614d0 msgr2=0x7fc51415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc518e24640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5141614d0 0x7fc51415dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc518e24640 1 -- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5140d2480 msgr2=0x7fc5140b8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc518e24640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5140d2480 0x7fc5140b8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc518e24640 1 -- 172.21.15.59:0/395641287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5140b91a0 con 0x7fc5140d1910 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc513fff640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5140d2480 0x7fc5140b8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc518e24640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5140d1910 0x7fc51415bbe0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fc50c007e40 tx=0x7fc50c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc5117fa640 1 -- 172.21.15.59:0/395641287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c014620 con 0x7fc5140d1910 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc5117fa640 1 -- 172.21.15.59:0/395641287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50c019070 con 0x7fc5140d1910 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc51b088640 1 -- 172.21.15.59:0/395641287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5140b6ae0 con 0x7fc5140d1910 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc5117fa640 1 -- 172.21.15.59:0/395641287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c014a00 con 0x7fc5140d1910 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.711+0000 7fc51b088640 1 -- 172.21.15.59:0/395641287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5140b70a0 con 0x7fc5140d1910 2024-03-06T07:28:34.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.712+0000 7fc51b088640 1 -- 172.21.15.59:0/395641287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc51406c620 con 0x7fc5140d1910 2024-03-06T07:28:34.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.713+0000 7fc5117fa640 1 -- 172.21.15.59:0/395641287 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc50c014ba0 con 0x7fc5140d1910 2024-03-06T07:28:34.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.713+0000 7fc5117fa640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4f406b4d0 0x7fc4f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.713+0000 7fc5117fa640 1 -- 172.21.15.59:0/395641287 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc50c08a6b0 con 0x7fc5140d1910 2024-03-06T07:28:34.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.713+0000 7fc513fff640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4f406b4d0 0x7fc4f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.714+0000 7fc513fff640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4f406b4d0 0x7fc4f406d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc5140ca610 tx=0x7fc4fc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.717+0000 7fc5117fa640 1 -- 172.21.15.59:0/395641287 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc50c059860 con 0x7fc5140d1910 2024-03-06T07:28:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.897+0000 7fc51b088640 1 -- 172.21.15.59:0/395641287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc514068ff0 con 0x7fc5140d1910 2024-03-06T07:28:34.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.901+0000 7fc5117fa640 1 -- 172.21.15.59:0/395641287 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc50c059160 con 0x7fc5140d1910 2024-03-06T07:28:34.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 -- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4f406b4d0 msgr2=0x7fc4f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4f406b4d0 0x7fc4f406d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc5140ca610 tx=0x7fc4fc009040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 -- 172.21.15.59:0/395641287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5140d1910 msgr2=0x7fc51415bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5140d1910 0x7fc51415bbe0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fc50c007e40 tx=0x7fc50c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 -- 172.21.15.59:0/395641287 shutdown_connections 2024-03-06T07:28:34.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc4f406b4d0 0x7fc4f406d990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5141614d0 0x7fc51415dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5140d2480 0x7fc5140b8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 --2- 172.21.15.59:0/395641287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5140d1910 0x7fc51415bbe0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 -- 172.21.15.59:0/395641287 >> 172.21.15.59:0/395641287 conn(0x7fc5140d3d40 msgr2=0x7fc5140b55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 -- 172.21.15.59:0/395641287 shutdown_connections 2024-03-06T07:28:34.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:34.903+0000 7fc51b088640 1 -- 172.21.15.59:0/395641287 wait complete. 2024-03-06T07:28:34.916 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:28:35.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.016+0000 7f72fd8c2640 1 Processor -- start 2024-03-06T07:28:35.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.016+0000 7f72fd8c2640 1 -- start start 2024-03-06T07:28:35.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.017+0000 7f72fd8c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8158600 0x7f72f8072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.017+0000 7f72fd8c2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f72f8162030 0x7f72f81624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.017+0000 7f72fd8c2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f815eb40 con 0x7f72f815e3a0 2024-03-06T07:28:35.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.017+0000 7f72fd8c2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f8161ba0 con 0x7f72f8162030 2024-03-06T07:28:35.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.017+0000 7f72fd8c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f815ee20 con 0x7f72f8158600 2024-03-06T07:28:35.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.017+0000 7f72f67fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f72f815e3a0 0x7f72f806d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40184/0 (socket says 172.21.15.59:40184) 2024-03-06T07:28:35.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.017+0000 7f72f67fc640 1 -- 172.21.15.59:0/712865398 learned_addr learned my addr 172.21.15.59:0/712865398 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f6ffd640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8158600 0x7f72f8072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024173691 0 0) 0x7f72f815eb40 con 0x7f72f815e3a0 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f77fe640 1 --2- 172.21.15.59:0/712865398 >> v2:172.21.15.164:3300/0 conn(0x7f72f8162030 0x7f72f81624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e4003520 con 0x7f72f815e3a0 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3072917866 0 0) 0x7f72e4003520 con 0x7f72f815e3a0 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8158600 msgr2=0x7f72f8072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8158600 0x7f72f8072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 >> v2:172.21.15.164:3300/0 conn(0x7f72f8162030 msgr2=0x7f72f81624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 --2- 172.21.15.59:0/712865398 >> v2:172.21.15.164:3300/0 conn(0x7f72f8162030 0x7f72f81624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f815f100 con 0x7f72f815e3a0 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f72e0001670 con 0x7f72f815e3a0 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 >> v1:172.21.15.59:6789/0 conn(0x7f72f815e3a0 legacy=0x7f72f806d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8158600 0x7f72e40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 0x7f72e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72e4004d40 0x7f72e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8161ba0 con 0x7f72e4003f30 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f815ee20 con 0x7f72e4004d40 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e400ba10 con 0x7f72f8158600 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f67fc640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 0x7f72e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f6ffd640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8158600 0x7f72e40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f77fe640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72e4004d40 0x7f72e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f67fc640 1 -- 172.21.15.59:0/712865398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8158600 msgr2=0x7f72e40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f67fc640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8158600 0x7f72e40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f67fc640 1 -- 172.21.15.59:0/712865398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72e4004d40 msgr2=0x7f72e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f67fc640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72e4004d40 0x7f72e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.018+0000 7f72f67fc640 1 -- 172.21.15.59:0/712865398 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e4003520 con 0x7f72e4003f30 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.019+0000 7f72f67fc640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 0x7f72e4004800 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f72e000c270 tx=0x7f72e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0369c2d81bb9c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.019+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0007d30 con 0x7f72e4003f30 2024-03-06T07:28:35.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.019+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e0030070 con 0x7f72e4003f30 2024-03-06T07:28:35.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.019+0000 7f72f5ffb640 1 -- 172.21.15.59:0/712865398 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e002cd60 con 0x7f72e4003f30 2024-03-06T07:28:35.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.020+0000 7f72fd8c2640 1 -- 172.21.15.59:0/712865398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 msgr2=0x7f72e4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.020+0000 7f72fd8c2640 1 --2- 172.21.15.59:0/712865398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 0x7f72e4004800 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f72e000c270 tx=0x7f72e4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:35.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.020+0000 7f72f77fe640 1 -- 172.21.15.59:0/712865398 reap_dead start 2024-03-06T07:28:35.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.020+0000 7f72fd8c2640 1 -- 172.21.15.59:0/712865398 shutdown_connections 2024-03-06T07:28:35.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.020+0000 7f72fd8c2640 1 -- 172.21.15.59:0/712865398 >> 172.21.15.59:0/712865398 conn(0x7f72f806c0c0 msgr2=0x7f72f806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.020+0000 7f72fd8c2640 1 -- 172.21.15.59:0/712865398 shutdown_connections 2024-03-06T07:28:35.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.020+0000 7f72fd8c2640 1 -- 172.21.15.59:0/712865398 wait complete. 2024-03-06T07:28:35.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.021+0000 7f72fd8c2640 1 Processor -- start 2024-03-06T07:28:35.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.021+0000 7f72fd8c2640 1 -- start start 2024-03-06T07:28:35.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72fd8c2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 0x7f72f8161130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72fd8c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8150550 0x7f72f80b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72fd8c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72f8142f30 0x7f72f81433c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72fd8c2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f806e080 con 0x7f72e4003f30 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72fd8c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8072730 con 0x7f72f8142f30 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72fd8c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f814d770 con 0x7f72f8150550 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72f6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 0x7f72f8161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72f6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 0x7f72f8161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60072/0 (socket says 172.21.15.59:60072) 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72f6ffd640 1 -- 172.21.15.59:0/866296545 learned_addr learned my addr 172.21.15.59:0/866296545 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72f6ffd640 1 -- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8150550 msgr2=0x7f72f80b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72f6ffd640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8150550 0x7f72f80b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72f6ffd640 1 -- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72f8142f30 msgr2=0x7f72f81433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72f6ffd640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72f8142f30 0x7f72f81433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72f6ffd640 1 -- 172.21.15.59:0/866296545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f8143900 con 0x7f72e4003f30 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72f6ffd640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 0x7f72f8161130 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f72f00118e0 tx=0x7f72f0011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72effff640 1 -- 172.21.15.59:0/866296545 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f000fa70 con 0x7f72e4003f30 2024-03-06T07:28:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72effff640 1 -- 172.21.15.59:0/866296545 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f000fc10 con 0x7f72e4003f30 2024-03-06T07:28:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.022+0000 7f72fd8c2640 1 -- 172.21.15.59:0/866296545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f8150a50 con 0x7f72e4003f30 2024-03-06T07:28:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.023+0000 7f72effff640 1 -- 172.21.15.59:0/866296545 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0014440 con 0x7f72e4003f30 2024-03-06T07:28:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.023+0000 7f72fd8c2640 1 -- 172.21.15.59:0/866296545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f8151010 con 0x7f72e4003f30 2024-03-06T07:28:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.023+0000 7f72fd8c2640 1 -- 172.21.15.59:0/866296545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f804f330 con 0x7f72e4003f30 2024-03-06T07:28:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.024+0000 7f72effff640 1 -- 172.21.15.59:0/866296545 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72f0012040 con 0x7f72e4003f30 2024-03-06T07:28:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.024+0000 7f72effff640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 0x7f72d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.024+0000 7f72effff640 1 -- 172.21.15.59:0/866296545 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f72f008a4b0 con 0x7f72e4003f30 2024-03-06T07:28:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.024+0000 7f72f67fc640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 0x7f72d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.025+0000 7f72f67fc640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 0x7f72d806d9c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f72e0034000 tx=0x7f72e0039040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.029+0000 7f72effff640 1 -- 172.21.15.59:0/866296545 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72f0019030 con 0x7f72e4003f30 2024-03-06T07:28:35.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.285+0000 7f72fd8c2640 1 -- 172.21.15.59:0/866296545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f72f80d06f0 con 0x7f72e4003f30 2024-03-06T07:28:35.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.569+0000 7f72effff640 1 -- 172.21.15.59:0/866296545 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v368) v1 ==== 142+0+0 (secure 0 0 0) 0x7f72f00595d0 con 0x7f72e4003f30 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 -- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 msgr2=0x7f72d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 0x7f72d806d9c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f72e0034000 tx=0x7f72e0039040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 -- 172.21.15.59:0/866296545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 msgr2=0x7f72f8161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 0x7f72f8161130 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f72f00118e0 tx=0x7f72f0011da0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 -- 172.21.15.59:0/866296545 shutdown_connections 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f72d806b500 0x7f72d806d9c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72f8142f30 0x7f72f81433c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72f8150550 0x7f72f80b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 --2- 172.21.15.59:0/866296545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72e4003f30 0x7f72f8161130 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 -- 172.21.15.59:0/866296545 >> 172.21.15.59:0/866296545 conn(0x7f72f806c0c0 msgr2=0x7f72f81575a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 -- 172.21.15.59:0/866296545 shutdown_connections 2024-03-06T07:28:35.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.572+0000 7f72fd8c2640 1 -- 172.21.15.59:0/866296545 wait complete. 2024-03-06T07:28:35.586 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-03-06T07:28:35.586 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:28:35.586 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:28:35.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:35.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.687+0000 7f14945bc640 1 Processor -- start 2024-03-06T07:28:35.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.687+0000 7f14945bc640 1 -- start start 2024-03-06T07:28:35.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f14945bc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f148c13dc10 0x7f148c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f14945bc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f148c15bf30 0x7f148c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f14945bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148c15e850 0x7f148c15eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f14945bc640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148c0cf3f0 con 0x7f148c13dc10 2024-03-06T07:28:35.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f14945bc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148c15b210 con 0x7f148c15bf30 2024-03-06T07:28:35.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f14945bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c15f580 con 0x7f148c15e850 2024-03-06T07:28:35.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1492358640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f148c13dc10 0x7f148c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1492358640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f148c13dc10 0x7f148c0b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60082/0 (socket says 172.21.15.59:60082) 2024-03-06T07:28:35.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1492358640 1 -- 172.21.15.59:0/4137337969 learned_addr learned my addr 172.21.15.59:0/4137337969 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:35.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1491b57640 1 --2- 172.21.15.59:0/4137337969 >> v2:172.21.15.164:3300/0 conn(0x7f148c15bf30 0x7f148c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1492b59640 1 --2- 172.21.15.59:0/4137337969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148c15e850 0x7f148c15eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1492358640 1 -- 172.21.15.59:0/4137337969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148c15e850 msgr2=0x7f148c15eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1492358640 1 --2- 172.21.15.59:0/4137337969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148c15e850 0x7f148c15eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1492358640 1 -- 172.21.15.59:0/4137337969 >> v2:172.21.15.164:3300/0 conn(0x7f148c15bf30 msgr2=0x7f148c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1492358640 1 --2- 172.21.15.59:0/4137337969 >> v2:172.21.15.164:3300/0 conn(0x7f148c15bf30 0x7f148c0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1492358640 1 -- 172.21.15.59:0/4137337969 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c15b450 con 0x7f148c13dc10 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1491b57640 1 --2- 172.21.15.59:0/4137337969 >> v2:172.21.15.164:3300/0 conn(0x7f148c15bf30 0x7f148c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.688+0000 7f1492b59640 1 --2- 172.21.15.59:0/4137337969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148c15e850 0x7f148c15eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.689+0000 7f1492358640 1 --2- 172.21.15.59:0/4137337969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148c13dc10 0x7f148c0b2950 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f148c06c790 tx=0x7f148802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2611417b557f88e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.689+0000 7f1491356640 1 -- 172.21.15.59:0/4137337969 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148802e070 con 0x7f148c13dc10 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.689+0000 7f1491356640 1 -- 172.21.15.59:0/4137337969 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1488037070 con 0x7f148c13dc10 2024-03-06T07:28:35.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.689+0000 7f1491356640 1 -- 172.21.15.59:0/4137337969 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488031a00 con 0x7f148c13dc10 2024-03-06T07:28:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.690+0000 7f14945bc640 1 -- 172.21.15.59:0/4137337969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148c13dc10 msgr2=0x7f148c0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.690+0000 7f14945bc640 1 --2- 172.21.15.59:0/4137337969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148c13dc10 0x7f148c0b2950 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f148c06c790 tx=0x7f148802b880 comp rx=0 tx=0).stop 2024-03-06T07:28:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.690+0000 7f14945bc640 1 -- 172.21.15.59:0/4137337969 shutdown_connections 2024-03-06T07:28:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.690+0000 7f14945bc640 1 --2- 172.21.15.59:0/4137337969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148c15e850 0x7f148c15eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.690+0000 7f14945bc640 1 --2- 172.21.15.59:0/4137337969 >> v2:172.21.15.164:3300/0 conn(0x7f148c15bf30 0x7f148c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.690+0000 7f14945bc640 1 --2- 172.21.15.59:0/4137337969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148c13dc10 0x7f148c0b2950 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.690+0000 7f14945bc640 1 -- 172.21.15.59:0/4137337969 >> 172.21.15.59:0/4137337969 conn(0x7f148c0cf8a0 msgr2=0x7f148c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.690+0000 7f14945bc640 1 -- 172.21.15.59:0/4137337969 shutdown_connections 2024-03-06T07:28:35.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.690+0000 7f14945bc640 1 -- 172.21.15.59:0/4137337969 wait complete. 2024-03-06T07:28:35.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.691+0000 7f14945bc640 1 Processor -- start 2024-03-06T07:28:35.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.691+0000 7f14945bc640 1 -- start start 2024-03-06T07:28:35.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f14945bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148c13dc10 0x7f148c1437f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f14945bc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f148c15bf30 0x7f148c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f14945bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148c15e850 0x7f148c1f18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f14945bc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c15f1e0 con 0x7f148c15e850 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f14945bc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c13dff0 con 0x7f148c15bf30 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f14945bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148c15f360 con 0x7f148c13dc10 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1492b59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148c15e850 0x7f148c1f18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1492b59640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148c15e850 0x7f148c1f18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60092/0 (socket says 172.21.15.59:60092) 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1492b59640 1 -- 172.21.15.59:0/61687606 learned_addr learned my addr 172.21.15.59:0/61687606 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1492358640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148c13dc10 0x7f148c1437f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1491b57640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f148c15bf30 0x7f148c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1492b59640 1 -- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148c13dc10 msgr2=0x7f148c1437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1492b59640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148c13dc10 0x7f148c1437f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1492b59640 1 -- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f148c15bf30 msgr2=0x7f148c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1492b59640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f148c15bf30 0x7f148c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1492b59640 1 -- 172.21.15.59:0/61687606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c1418d0 con 0x7f148c15e850 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1492b59640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148c15e850 0x7f148c1f18b0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f148400cf20 tx=0x7f148400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1482ffd640 1 -- 172.21.15.59:0/61687606 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148401a070 con 0x7f148c15e850 2024-03-06T07:28:35.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1482ffd640 1 -- 172.21.15.59:0/61687606 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148400d840 con 0x7f148c15e850 2024-03-06T07:28:35.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f14945bc640 1 -- 172.21.15.59:0/61687606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148c141bb0 con 0x7f148c15e850 2024-03-06T07:28:35.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f1482ffd640 1 -- 172.21.15.59:0/61687606 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148400dc30 con 0x7f148c15e850 2024-03-06T07:28:35.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.692+0000 7f14945bc640 1 -- 172.21.15.59:0/61687606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148c142170 con 0x7f148c15e850 2024-03-06T07:28:35.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.693+0000 7f14945bc640 1 -- 172.21.15.59:0/61687606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148c04f330 con 0x7f148c15e850 2024-03-06T07:28:35.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.694+0000 7f1482ffd640 1 -- 172.21.15.59:0/61687606 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f148401d070 con 0x7f148c15e850 2024-03-06T07:28:35.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.694+0000 7f1482ffd640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f147006b4d0 0x7f147006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.694+0000 7f1482ffd640 1 -- 172.21.15.59:0/61687606 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f148408a6e0 con 0x7f148c15e850 2024-03-06T07:28:35.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.695+0000 7f1492358640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f147006b4d0 0x7f147006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.695+0000 7f1492358640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f147006b4d0 0x7f147006d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1488036000 tx=0x7f148803d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.699+0000 7f1482ffd640 1 -- 172.21.15.59:0/61687606 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1484059240 con 0x7f148c15e850 2024-03-06T07:28:35.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.881+0000 7f14945bc640 1 -- 172.21.15.59:0/61687606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f148c0748f0 con 0x7f148c15e850 2024-03-06T07:28:35.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.882+0000 7f1482ffd640 1 -- 172.21.15.59:0/61687606 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+13150 (secure 0 0 0) 0x7f148405d2e0 con 0x7f148c15e850 2024-03-06T07:28:35.884 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:35.884 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":117,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:28:34.560415+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6835/3442407979":"2024-03-07T07:28:31.423517+0000","172.21.15.59:6834/3442407979":"2024-03-07T07:28:31.423517+0000","172.21.15.164:6836/2139041464":"2024-03-07T07:27:27.121928+0000","172.21.15.164:6834/2139041464":"2024-03-07T07:27:27.121928+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 -- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f147006b4d0 msgr2=0x7f147006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f147006b4d0 0x7f147006d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1488036000 tx=0x7f148803d040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 -- 172.21.15.59:0/61687606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148c15e850 msgr2=0x7f148c1f18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148c15e850 0x7f148c1f18b0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f148400cf20 tx=0x7f148400a7d0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 -- 172.21.15.59:0/61687606 shutdown_connections 2024-03-06T07:28:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f147006b4d0 0x7f147006d990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148c15e850 0x7f148c1f18b0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f148c15bf30 0x7f148c143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 --2- 172.21.15.59:0/61687606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148c13dc10 0x7f148c1437f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 -- 172.21.15.59:0/61687606 >> 172.21.15.59:0/61687606 conn(0x7f148c0cf8a0 msgr2=0x7f148c06c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 -- 172.21.15.59:0/61687606 shutdown_connections 2024-03-06T07:28:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:35.884+0000 7f14945bc640 1 -- 172.21.15.59:0/61687606 wait complete. 2024-03-06T07:28:35.897 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:35.897 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.000+0000 7fb09dcb5640 1 Processor -- start 2024-03-06T07:28:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.000+0000 7fb09dcb5640 1 -- start start 2024-03-06T07:28:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb09dcb5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb098145590 0x7fb09806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb09dcb5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb09806b4f0 0x7fb09806efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb09dcb5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09806b970 con 0x7fb09806f510 2024-03-06T07:28:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb09dcb5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb09806f980 con 0x7fb09806b4f0 2024-03-06T07:28:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb09dcb5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb098145970 con 0x7fb098145590 2024-03-06T07:28:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb096ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb09806b4f0 0x7fb09806efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb097fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb09806f510 0x7fb09806daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40188/0 (socket says 172.21.15.59:40188) 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb097fff640 1 -- 172.21.15.59:0/740207695 learned_addr learned my addr 172.21.15.59:0/740207695 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb0977fe640 1 --2- 172.21.15.59:0/740207695 >> v2:172.21.15.164:3300/0 conn(0x7fb098145590 0x7fb09806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb096ffd640 1 -- 172.21.15.59:0/740207695 >> v2:172.21.15.164:3300/0 conn(0x7fb098145590 msgr2=0x7fb09806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb096ffd640 1 --2- 172.21.15.59:0/740207695 >> v2:172.21.15.164:3300/0 conn(0x7fb098145590 0x7fb09806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb096ffd640 1 -- 172.21.15.59:0/740207695 >> v1:172.21.15.59:6789/0 conn(0x7fb09806f510 legacy=0x7fb09806daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb096ffd640 1 -- 172.21.15.59:0/740207695 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09815f7e0 con 0x7fb09806b4f0 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.001+0000 7fb0967fc640 1 -- 172.21.15.59:0/740207695 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775439859 0 0) 0x7fb09806b970 con 0x7fb09806f510 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.002+0000 7fb096ffd640 1 --2- 172.21.15.59:0/740207695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb09806b4f0 0x7fb09806efd0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fb0980d0830 tx=0x7fb08802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41b850de50e0a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.002+0000 7fb0967fc640 1 -- 172.21.15.59:0/740207695 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088035020 con 0x7fb09806b4f0 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.002+0000 7fb0967fc640 1 -- 172.21.15.59:0/740207695 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb088031830 con 0x7fb09806b4f0 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.002+0000 7fb0967fc640 1 -- 172.21.15.59:0/740207695 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088031c00 con 0x7fb09806b4f0 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.002+0000 7fb09dcb5640 1 -- 172.21.15.59:0/740207695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb09806b4f0 msgr2=0x7fb09806efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.002+0000 7fb09dcb5640 1 --2- 172.21.15.59:0/740207695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb09806b4f0 0x7fb09806efd0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fb0980d0830 tx=0x7fb08802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.003+0000 7fb09dcb5640 1 -- 172.21.15.59:0/740207695 shutdown_connections 2024-03-06T07:28:36.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.003+0000 7fb09dcb5640 1 --2- 172.21.15.59:0/740207695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb09806b4f0 0x7fb09806efd0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.003+0000 7fb09dcb5640 1 --2- 172.21.15.59:0/740207695 >> v2:172.21.15.164:3300/0 conn(0x7fb098145590 0x7fb09806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.003+0000 7fb09dcb5640 1 -- 172.21.15.59:0/740207695 >> 172.21.15.59:0/740207695 conn(0x7fb0980b2660 msgr2=0x7fb0980b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.003+0000 7fb09dcb5640 1 -- 172.21.15.59:0/740207695 shutdown_connections 2024-03-06T07:28:36.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.003+0000 7fb09dcb5640 1 -- 172.21.15.59:0/740207695 wait complete. 2024-03-06T07:28:36.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.004+0000 7fb09dcb5640 1 Processor -- start 2024-03-06T07:28:36.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.004+0000 7fb09dcb5640 1 -- start start 2024-03-06T07:28:36.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb09dcb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb09806b4f0 0x7fb0980b4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb09dcb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb09806f510 0x7fb09814f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb09dcb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb098145590 0x7fb09813fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb09dcb5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09815fda0 con 0x7fb098145590 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb09dcb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb098160110 con 0x7fb09806f510 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb09dcb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0980cd5d0 con 0x7fb09806b4f0 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb097fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb098145590 0x7fb09813fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb097fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb098145590 0x7fb09813fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60106/0 (socket says 172.21.15.59:60106) 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb097fff640 1 -- 172.21.15.59:0/1826231827 learned_addr learned my addr 172.21.15.59:0/1826231827 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb096ffd640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb09806f510 0x7fb09814f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb0977fe640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb09806b4f0 0x7fb0980b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb097fff640 1 -- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb09806b4f0 msgr2=0x7fb0980b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb097fff640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb09806b4f0 0x7fb0980b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb097fff640 1 -- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb09806f510 msgr2=0x7fb09814f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb097fff640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb09806f510 0x7fb09814f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb097fff640 1 -- 172.21.15.59:0/1826231827 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09813aa40 con 0x7fb098145590 2024-03-06T07:28:36.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb097fff640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb098145590 0x7fb09813fd70 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fb09000c7a0 tx=0x7fb09000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb07ffff640 1 -- 172.21.15.59:0/1826231827 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090018070 con 0x7fb098145590 2024-03-06T07:28:36.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb07ffff640 1 -- 172.21.15.59:0/1826231827 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09000a640 con 0x7fb098145590 2024-03-06T07:28:36.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb09dcb5640 1 -- 172.21.15.59:0/1826231827 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0980b5310 con 0x7fb098145590 2024-03-06T07:28:36.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb07ffff640 1 -- 172.21.15.59:0/1826231827 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09000aab0 con 0x7fb098145590 2024-03-06T07:28:36.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.005+0000 7fb09dcb5640 1 -- 172.21.15.59:0/1826231827 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09813afa0 con 0x7fb098145590 2024-03-06T07:28:36.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.006+0000 7fb09dcb5640 1 -- 172.21.15.59:0/1826231827 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0980733b0 con 0x7fb098145590 2024-03-06T07:28:36.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.007+0000 7fb07ffff640 1 -- 172.21.15.59:0/1826231827 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb09000ac50 con 0x7fb098145590 2024-03-06T07:28:36.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.007+0000 7fb07ffff640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b500 0x7fb08406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.007+0000 7fb07ffff640 1 -- 172.21.15.59:0/1826231827 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb0900894b0 con 0x7fb098145590 2024-03-06T07:28:36.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.008+0000 7fb0977fe640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b500 0x7fb08406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.008+0000 7fb0977fe640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b500 0x7fb08406d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0079a0 tx=0x7fb08c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.011+0000 7fb07ffff640 1 -- 172.21.15.59:0/1826231827 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb09813aa40 con 0x7fb098145590 2024-03-06T07:28:36.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.191+0000 7fb09dcb5640 1 -- 172.21.15.59:0/1826231827 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb098068ff0 con 0x7fb098145590 2024-03-06T07:28:36.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.192+0000 7fb07ffff640 1 -- 172.21.15.59:0/1826231827 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb090057f00 con 0x7fb098145590 2024-03-06T07:28:36.194 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 -- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b500 msgr2=0x7fb08406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b500 0x7fb08406d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0079a0 tx=0x7fb08c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 -- 172.21.15.59:0/1826231827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb098145590 msgr2=0x7fb09813fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb098145590 0x7fb09813fd70 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fb09000c7a0 tx=0x7fb09000cc60 comp rx=0 tx=0).stop 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 -- 172.21.15.59:0/1826231827 shutdown_connections 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb08406b500 0x7fb08406d9c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb098145590 0x7fb09813fd70 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb09806f510 0x7fb09814f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 --2- 172.21.15.59:0/1826231827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb09806b4f0 0x7fb0980b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 -- 172.21.15.59:0/1826231827 >> 172.21.15.59:0/1826231827 conn(0x7fb0980b2660 msgr2=0x7fb0980b16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 -- 172.21.15.59:0/1826231827 shutdown_connections 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.194+0000 7fb09dcb5640 1 -- 172.21.15.59:0/1826231827 wait complete. 2024-03-06T07:28:36.207 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:28:36.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.307+0000 7f898260e640 1 Processor -- start 2024-03-06T07:28:36.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.307+0000 7f898260e640 1 -- start start 2024-03-06T07:28:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f898260e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c15dfe0 0x7f897c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f898260e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f897c0d29c0 0x7f897c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f898260e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897c0b3430 con 0x7f897c0cf830 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f898260e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f897c0b31e0 con 0x7f897c0d29c0 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f898260e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f897c0b3d60 con 0x7f897c15dfe0 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f897c0cf830 0x7f897c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40196/0 (socket says 172.21.15.59:40196) 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897b7fe640 1 -- 172.21.15.59:0/1786261843 learned_addr learned my addr 172.21.15.59:0/1786261843 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f8980bab640 1 --2- 172.21.15.59:0/1786261843 >> v2:172.21.15.164:3300/0 conn(0x7f897c0d29c0 0x7f897c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897bfff640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c15dfe0 0x7f897c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839205600 0 0) 0x7f897c0b3430 con 0x7f897c0cf830 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8964003520 con 0x7f897c0cf830 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 89547805 0 0) 0x7f8964003520 con 0x7f897c0cf830 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c15dfe0 msgr2=0x7f897c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c15dfe0 0x7f897c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 >> v2:172.21.15.164:3300/0 conn(0x7f897c0d29c0 msgr2=0x7f897c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 --2- 172.21.15.59:0/1786261843 >> v2:172.21.15.164:3300/0 conn(0x7f897c0d29c0 0x7f897c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c0b3710 con 0x7f897c0cf830 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f8970001670 con 0x7f897c0cf830 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 >> v1:172.21.15.59:6789/0 conn(0x7f897c0cf830 legacy=0x7f897c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c15dfe0 0x7f89640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8964003f30 0x7f8964004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8964004d40 0x7f89640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897c0b31e0 con 0x7f8964004d40 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897c0b3d60 con 0x7f8964003f30 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896400b9c0 con 0x7f897c15dfe0 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.308+0000 7f8980bab640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8964004d40 0x7f89640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.309+0000 7f897bfff640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c15dfe0 0x7f89640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.309+0000 7f897b7fe640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8964003f30 0x7f8964004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.309+0000 7f8980bab640 1 -- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c15dfe0 msgr2=0x7f89640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.309+0000 7f8980bab640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c15dfe0 0x7f89640039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.309+0000 7f8980bab640 1 -- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8964003f30 msgr2=0x7f8964004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.309+0000 7f8980bab640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8964003f30 0x7f8964004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.309+0000 7f8980bab640 1 -- 172.21.15.59:0/1786261843 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8964003520 con 0x7f8964004d40 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.309+0000 7f897b7fe640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8964003f30 0x7f8964004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.309+0000 7f8980bab640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8964004d40 0x7f89640051c0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f8974009fc0 tx=0x7f8964005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6b2762ec236732 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.310+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8974039070 con 0x7f8964004d40 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.310+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f897402cc40 con 0x7f8964004d40 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.310+0000 7f897affd640 1 -- 172.21.15.59:0/1786261843 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8974009c10 con 0x7f8964004d40 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.310+0000 7f898260e640 1 -- 172.21.15.59:0/1786261843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8964004d40 msgr2=0x7f89640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.310+0000 7f898260e640 1 --2- 172.21.15.59:0/1786261843 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8964004d40 0x7f89640051c0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f8974009fc0 tx=0x7f8964005830 comp rx=0 tx=0).stop 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.310+0000 7f8980bab640 1 -- 172.21.15.59:0/1786261843 reap_dead start 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.310+0000 7f898260e640 1 -- 172.21.15.59:0/1786261843 shutdown_connections 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.310+0000 7f898260e640 1 -- 172.21.15.59:0/1786261843 >> 172.21.15.59:0/1786261843 conn(0x7f897c0d3cd0 msgr2=0x7f897c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.310+0000 7f898260e640 1 -- 172.21.15.59:0/1786261843 shutdown_connections 2024-03-06T07:28:36.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.310+0000 7f898260e640 1 -- 172.21.15.59:0/1786261843 wait complete. 2024-03-06T07:28:36.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.311+0000 7f898260e640 1 Processor -- start 2024-03-06T07:28:36.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.311+0000 7f898260e640 1 -- start start 2024-03-06T07:28:36.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.312+0000 7f898260e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c1f6cb0 0x7f897c161110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.312+0000 7f898260e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c0cea80 0x7f897c0cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.312+0000 7f898260e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f897c0ccdb0 0x7f897c0cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.312+0000 7f898260e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897c0d1df0 con 0x7f897c1f6cb0 2024-03-06T07:28:36.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.312+0000 7f898260e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897c0d1c70 con 0x7f897c0ccdb0 2024-03-06T07:28:36.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.312+0000 7f898260e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f897c15e3c0 con 0x7f897c0cea80 2024-03-06T07:28:36.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.312+0000 7f897bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c1f6cb0 0x7f897c161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.312+0000 7f897bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c1f6cb0 0x7f897c161110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60124/0 (socket says 172.21.15.59:60124) 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.312+0000 7f897bfff640 1 -- 172.21.15.59:0/151674686 learned_addr learned my addr 172.21.15.59:0/151674686 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.312+0000 7f897b7fe640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c0cea80 0x7f897c0cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.312+0000 7f8980bab640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f897c0ccdb0 0x7f897c0cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.313+0000 7f897bfff640 1 -- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c0cea80 msgr2=0x7f897c0cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.313+0000 7f897bfff640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c0cea80 0x7f897c0cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.313+0000 7f897bfff640 1 -- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f897c0ccdb0 msgr2=0x7f897c0cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.313+0000 7f897bfff640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f897c0ccdb0 0x7f897c0cd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.313+0000 7f897bfff640 1 -- 172.21.15.59:0/151674686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c0b68d0 con 0x7f897c1f6cb0 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.313+0000 7f897bfff640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c1f6cb0 0x7f897c161110 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f896c011a00 tx=0x7f896c011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.313+0000 7f8978ff9640 1 -- 172.21.15.59:0/151674686 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c00fb50 con 0x7f897c1f6cb0 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.313+0000 7f8978ff9640 1 -- 172.21.15.59:0/151674686 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f896c00fcf0 con 0x7f897c1f6cb0 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.313+0000 7f898260e640 1 -- 172.21.15.59:0/151674686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f897c161650 con 0x7f897c1f6cb0 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.313+0000 7f8978ff9640 1 -- 172.21.15.59:0/151674686 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c00fea0 con 0x7f897c1f6cb0 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.313+0000 7f898260e640 1 -- 172.21.15.59:0/151674686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f897c0b6e50 con 0x7f897c1f6cb0 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.314+0000 7f898260e640 1 -- 172.21.15.59:0/151674686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f897c04f330 con 0x7f897c1f6cb0 2024-03-06T07:28:36.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.314+0000 7f8978ff9640 1 -- 172.21.15.59:0/151674686 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f896c012040 con 0x7f897c1f6cb0 2024-03-06T07:28:36.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.315+0000 7f8978ff9640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f896806b4d0 0x7f896806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.315+0000 7f8978ff9640 1 -- 172.21.15.59:0/151674686 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f896c08a580 con 0x7f897c1f6cb0 2024-03-06T07:28:36.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.315+0000 7f897b7fe640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f896806b4d0 0x7f896806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.315+0000 7f897b7fe640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f896806b4d0 0x7f896806d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8970001860 tx=0x7f8970007040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.319+0000 7f8978ff9640 1 -- 172.21.15.59:0/151674686 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f896c019030 con 0x7f897c1f6cb0 2024-03-06T07:28:36.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.568+0000 7f898260e640 1 -- 172.21.15.59:0/151674686 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f897c0b4d70 con 0x7f897c1f6cb0 2024-03-06T07:28:36.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.571+0000 7f8978ff9640 1 -- 172.21.15.59:0/151674686 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v369) v1 ==== 143+0+0 (secure 0 0 0) 0x7f896c058fb0 con 0x7f897c1f6cb0 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 -- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f896806b4d0 msgr2=0x7f896806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f896806b4d0 0x7f896806d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8970001860 tx=0x7f8970007040 comp rx=0 tx=0).stop 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 -- 172.21.15.59:0/151674686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c1f6cb0 msgr2=0x7f897c161110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c1f6cb0 0x7f897c161110 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f896c011a00 tx=0x7f896c011ec0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 -- 172.21.15.59:0/151674686 shutdown_connections 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f896806b4d0 0x7f896806d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f897c0ccdb0 0x7f897c0cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f897c0cea80 0x7f897c0cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 --2- 172.21.15.59:0/151674686 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f897c1f6cb0 0x7f897c161110 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 -- 172.21.15.59:0/151674686 >> 172.21.15.59:0/151674686 conn(0x7f897c0d3cd0 msgr2=0x7f897c06bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 -- 172.21.15.59:0/151674686 shutdown_connections 2024-03-06T07:28:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.574+0000 7f898260e640 1 -- 172.21.15.59:0/151674686 wait complete. 2024-03-06T07:28:36.587 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:28:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.689+0000 7f02a1fb6640 1 Processor -- start 2024-03-06T07:28:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.690+0000 7f02a1fb6640 1 -- start start 2024-03-06T07:28:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.690+0000 7f02a1fb6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 0x7f029c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.690+0000 7f02a1fb6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f029c0d1910 0x7f029c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.690+0000 7f02a1fb6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029c0d2740 con 0x7f029c0d22b0 2024-03-06T07:28:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.690+0000 7f02a1fb6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029c1618b0 con 0x7f029c0d1910 2024-03-06T07:28:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.690+0000 7f02a1fb6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0cf400 con 0x7f029c1614d0 2024-03-06T07:28:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 0x7f029c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029affd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f029c0d1910 0x7f029c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029bfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f029c0d22b0 0x7f029c0d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:54316/0 (socket says 172.21.15.59:54316) 2024-03-06T07:28:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029bfff640 1 -- 172.21.15.59:0/3253211393 learned_addr learned my addr 172.21.15.59:0/3253211393 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 0x7f029c0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:34106/0 (socket says 172.21.15.59:34106) 2024-03-06T07:28:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029a7fc640 1 -- 172.21.15.59:0/3253211393 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121334827 0 0) 0x7f0294001490 con 0x7f029c0d22b0 2024-03-06T07:28:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029a7fc640 1 -- 172.21.15.59:0/3253211393 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0284003520 con 0x7f029c0d22b0 2024-03-06T07:28:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029b7fe640 1 -- 172.21.15.59:0/3253211393 >> v2:172.21.15.164:3300/0 conn(0x7f029c0d1910 msgr2=0x7f029c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029b7fe640 1 --2- 172.21.15.59:0/3253211393 >> v2:172.21.15.164:3300/0 conn(0x7f029c0d1910 0x7f029c0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029b7fe640 1 -- 172.21.15.59:0/3253211393 >> v1:172.21.15.164:6789/0 conn(0x7f029c0d22b0 legacy=0x7f029c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029b7fe640 1 -- 172.21.15.59:0/3253211393 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c0cf6a0 con 0x7f029c1614d0 2024-03-06T07:28:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.691+0000 7f029affd640 1 --2- 172.21.15.59:0/3253211393 >> v2:172.21.15.164:3300/0 conn(0x7f029c0d1910 0x7f029c0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:36.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.692+0000 7f029b7fe640 1 --2- 172.21.15.59:0/3253211393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 0x7f029c0d4860 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f028c0097a0 tx=0x7f028c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28c477cfd21a7d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.692+0000 7f029a7fc640 1 -- 172.21.15.59:0/3253211393 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c035020 con 0x7f029c1614d0 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.692+0000 7f029a7fc640 1 -- 172.21.15.59:0/3253211393 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c031830 con 0x7f029c1614d0 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.692+0000 7f029a7fc640 1 -- 172.21.15.59:0/3253211393 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c031be0 con 0x7f029c1614d0 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.692+0000 7f02a1fb6640 1 -- 172.21.15.59:0/3253211393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 msgr2=0x7f029c0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.692+0000 7f02a1fb6640 1 --2- 172.21.15.59:0/3253211393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 0x7f029c0d4860 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f028c0097a0 tx=0x7f028c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.692+0000 7f02a1fb6640 1 -- 172.21.15.59:0/3253211393 shutdown_connections 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.693+0000 7f02a1fb6640 1 --2- 172.21.15.59:0/3253211393 >> v2:172.21.15.164:3300/0 conn(0x7f029c0d1910 0x7f029c0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.693+0000 7f02a1fb6640 1 --2- 172.21.15.59:0/3253211393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 0x7f029c0d4860 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.693+0000 7f02a1fb6640 1 -- 172.21.15.59:0/3253211393 >> 172.21.15.59:0/3253211393 conn(0x7f029c0d3d40 msgr2=0x7f029c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.693+0000 7f02a1fb6640 1 -- 172.21.15.59:0/3253211393 shutdown_connections 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.693+0000 7f02a1fb6640 1 -- 172.21.15.59:0/3253211393 wait complete. 2024-03-06T07:28:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.694+0000 7f02a1fb6640 1 Processor -- start 2024-03-06T07:28:36.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.694+0000 7f02a1fb6640 1 -- start start 2024-03-06T07:28:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.694+0000 7f02a1fb6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f029c0d1910 0x7f029c0b9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.694+0000 7f02a1fb6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f029c0d22b0 0x7f029c0cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.694+0000 7f02a1fb6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 0x7f029c0ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.694+0000 7f02a1fb6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0b0910 con 0x7f029c0d1910 2024-03-06T07:28:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f02a1fb6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0cfc60 con 0x7f029c0d22b0 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f02a1fb6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0b1e40 con 0x7f029c1614d0 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f029c0d1910 0x7f029c0b9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f029c0d1910 0x7f029c0b9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60132/0 (socket says 172.21.15.59:60132) 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029b7fe640 1 -- 172.21.15.59:0/2572349022 learned_addr learned my addr 172.21.15.59:0/2572349022 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029bfff640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 0x7f029c0ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029affd640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f029c0d22b0 0x7f029c0cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029b7fe640 1 -- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 msgr2=0x7f029c0ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029b7fe640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 0x7f029c0ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029b7fe640 1 -- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f029c0d22b0 msgr2=0x7f029c0cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029b7fe640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f029c0d22b0 0x7f029c0cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029b7fe640 1 -- 172.21.15.59:0/2572349022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c0b9960 con 0x7f029c0d1910 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029affd640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f029c0d22b0 0x7f029c0cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f029b7fe640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f029c0d1910 0x7f029c0b9420 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f028c000c00 tx=0x7f028c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f0283fff640 1 -- 172.21.15.59:0/2572349022 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c040070 con 0x7f029c0d1910 2024-03-06T07:28:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f0283fff640 1 -- 172.21.15.59:0/2572349022 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c03a830 con 0x7f029c0d1910 2024-03-06T07:28:36.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f02a1fb6640 1 -- 172.21.15.59:0/2572349022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029c0cc070 con 0x7f029c0d1910 2024-03-06T07:28:36.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f0283fff640 1 -- 172.21.15.59:0/2572349022 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c035040 con 0x7f029c0d1910 2024-03-06T07:28:36.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.695+0000 7f02a1fb6640 1 -- 172.21.15.59:0/2572349022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029c0cc630 con 0x7f029c0d1910 2024-03-06T07:28:36.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.697+0000 7f02a1fb6640 1 -- 172.21.15.59:0/2572349022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029c068ff0 con 0x7f029c0d1910 2024-03-06T07:28:36.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.697+0000 7f0283fff640 1 -- 172.21.15.59:0/2572349022 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f028c04b060 con 0x7f029c0d1910 2024-03-06T07:28:36.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.697+0000 7f0283fff640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f028806b4d0 0x7f028806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.697+0000 7f0283fff640 1 -- 172.21.15.59:0/2572349022 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f028c0ab790 con 0x7f029c0d1910 2024-03-06T07:28:36.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.698+0000 7f029affd640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f028806b4d0 0x7f028806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.698+0000 7f029affd640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f028806b4d0 0x7f028806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f029c06b940 tx=0x7f0290008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.702+0000 7f0283fff640 1 -- 172.21.15.59:0/2572349022 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f028c07a7a0 con 0x7f029c0d1910 2024-03-06T07:28:36.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.882+0000 7f02a1fb6640 1 -- 172.21.15.59:0/2572349022 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f029c06b970 con 0x7f029c0d1910 2024-03-06T07:28:36.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.888+0000 7f0283fff640 1 -- 172.21.15.59:0/2572349022 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f028c07a0a0 con 0x7f029c0d1910 2024-03-06T07:28:36.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 -- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f028806b4d0 msgr2=0x7f028806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f028806b4d0 0x7f028806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f029c06b940 tx=0x7f0290008040 comp rx=0 tx=0).stop 2024-03-06T07:28:36.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 -- 172.21.15.59:0/2572349022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f029c0d1910 msgr2=0x7f029c0b9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f029c0d1910 0x7f029c0b9420 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f028c000c00 tx=0x7f028c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 -- 172.21.15.59:0/2572349022 shutdown_connections 2024-03-06T07:28:36.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f028806b4d0 0x7f028806d990 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f029c1614d0 0x7f029c0ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f029c0d22b0 0x7f029c0cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 --2- 172.21.15.59:0/2572349022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f029c0d1910 0x7f029c0b9420 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 -- 172.21.15.59:0/2572349022 >> 172.21.15.59:0/2572349022 conn(0x7f029c0d3d40 msgr2=0x7f029c06cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 -- 172.21.15.59:0/2572349022 shutdown_connections 2024-03-06T07:28:36.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:36.890+0000 7f02a1fb6640 1 -- 172.21.15.59:0/2572349022 wait complete. 2024-03-06T07:28:36.903 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:36.903 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:36.948 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:37.003 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:37.003 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:28:37.003 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:37.018 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:37.018 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:28:37.018 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:37.035 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:37.036 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:28:37.036 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:37.073 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:37.073 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:28:37.073 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:37.092 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:37.093 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:28:37.093 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:37.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.213+0000 7fe8743eb640 1 Processor -- start 2024-03-06T07:28:37.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.213+0000 7fe8743eb640 1 -- start start 2024-03-06T07:28:37.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe8743eb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe86c06b220 0x7fe86c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe8743eb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe86c0ce920 0x7fe86c071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe8743eb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86c15ea30 con 0x7fe86c071ec0 2024-03-06T07:28:37.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe8743eb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86c161b70 con 0x7fe86c0ce920 2024-03-06T07:28:37.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe8743eb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86c15ed10 con 0x7fe86c06b220 2024-03-06T07:28:37.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe871986640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe86c0ce920 0x7fe86c071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe871986640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe86c0ce920 0x7fe86c071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60140/0 (socket says 172.21.15.59:60140) 2024-03-06T07:28:37.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe871986640 1 -- 172.21.15.59:0/3893125345 learned_addr learned my addr 172.21.15.59:0/3893125345 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:37.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe871986640 1 -- 172.21.15.59:0/3893125345 >> v2:172.21.15.164:3300/0 conn(0x7fe86c06b220 msgr2=0x7fe86c15e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe872187640 1 --2- 172.21.15.59:0/3893125345 >> v2:172.21.15.164:3300/0 conn(0x7fe86c06b220 0x7fe86c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe871986640 1 --2- 172.21.15.59:0/3893125345 >> v2:172.21.15.164:3300/0 conn(0x7fe86c06b220 0x7fe86c15e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe871986640 1 -- 172.21.15.59:0/3893125345 >> v1:172.21.15.164:6789/0 conn(0x7fe86c071ec0 legacy=0x7fe86c162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.214+0000 7fe871986640 1 -- 172.21.15.59:0/3893125345 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c15eff0 con 0x7fe86c0ce920 2024-03-06T07:28:37.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.215+0000 7fe871986640 1 --2- 172.21.15.59:0/3893125345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0ce920 0x7fe86c071980 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0d0980 tx=0x7fe85c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90fb47ad7e52e0e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.215+0000 7fe871185640 1 -- 172.21.15.59:0/3893125345 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c035020 con 0x7fe86c0ce920 2024-03-06T07:28:37.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.215+0000 7fe871185640 1 -- 172.21.15.59:0/3893125345 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85c031870 con 0x7fe86c0ce920 2024-03-06T07:28:37.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.215+0000 7fe871185640 1 -- 172.21.15.59:0/3893125345 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c031c40 con 0x7fe86c0ce920 2024-03-06T07:28:37.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.216+0000 7fe8743eb640 1 -- 172.21.15.59:0/3893125345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0ce920 msgr2=0x7fe86c071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.216+0000 7fe8743eb640 1 --2- 172.21.15.59:0/3893125345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0ce920 0x7fe86c071980 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0d0980 tx=0x7fe85c02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:37.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.216+0000 7fe8743eb640 1 -- 172.21.15.59:0/3893125345 shutdown_connections 2024-03-06T07:28:37.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.216+0000 7fe8743eb640 1 --2- 172.21.15.59:0/3893125345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0ce920 0x7fe86c071980 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.216+0000 7fe8743eb640 1 --2- 172.21.15.59:0/3893125345 >> v2:172.21.15.164:3300/0 conn(0x7fe86c06b220 0x7fe86c15e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.216+0000 7fe8743eb640 1 -- 172.21.15.59:0/3893125345 >> 172.21.15.59:0/3893125345 conn(0x7fe86c06c0c0 msgr2=0x7fe86c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.216+0000 7fe8743eb640 1 -- 172.21.15.59:0/3893125345 shutdown_connections 2024-03-06T07:28:37.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.216+0000 7fe8743eb640 1 -- 172.21.15.59:0/3893125345 wait complete. 2024-03-06T07:28:37.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.217+0000 7fe8743eb640 1 Processor -- start 2024-03-06T07:28:37.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.217+0000 7fe8743eb640 1 -- start start 2024-03-06T07:28:37.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe8743eb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe86c06b220 0x7fe86c14a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe8743eb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c071ec0 0x7fe86c15b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe8743eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0ce920 0x7fe86c0ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe8743eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c0ce380 con 0x7fe86c0ce920 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe8743eb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c0ce200 con 0x7fe86c06b220 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe8743eb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c162710 con 0x7fe86c071ec0 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe872988640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0ce920 0x7fe86c0ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe872988640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0ce920 0x7fe86c0ae270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60156/0 (socket says 172.21.15.59:60156) 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe872988640 1 -- 172.21.15.59:0/3456444327 learned_addr learned my addr 172.21.15.59:0/3456444327 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe871986640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c071ec0 0x7fe86c15b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe872187640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe86c06b220 0x7fe86c14a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe872988640 1 -- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c071ec0 msgr2=0x7fe86c15b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe872988640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c071ec0 0x7fe86c15b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe872988640 1 -- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe86c06b220 msgr2=0x7fe86c14a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe872988640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe86c06b220 0x7fe86c14a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe872988640 1 -- 172.21.15.59:0/3456444327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c14ab20 con 0x7fe86c0ce920 2024-03-06T07:28:37.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe872988640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0ce920 0x7fe86c0ae270 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fe86400c7b0 tx=0x7fe86400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe862ffd640 1 -- 172.21.15.59:0/3456444327 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe864004b60 con 0x7fe86c0ce920 2024-03-06T07:28:37.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe862ffd640 1 -- 172.21.15.59:0/3456444327 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe864004d00 con 0x7fe86c0ce920 2024-03-06T07:28:37.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe8743eb640 1 -- 172.21.15.59:0/3456444327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86c15bcc0 con 0x7fe86c0ce920 2024-03-06T07:28:37.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe862ffd640 1 -- 172.21.15.59:0/3456444327 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86400aa90 con 0x7fe86c0ce920 2024-03-06T07:28:37.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.218+0000 7fe8743eb640 1 -- 172.21.15.59:0/3456444327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86c0b0320 con 0x7fe86c0ce920 2024-03-06T07:28:37.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.219+0000 7fe8743eb640 1 -- 172.21.15.59:0/3456444327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86c04f330 con 0x7fe86c0ce920 2024-03-06T07:28:37.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.220+0000 7fe862ffd640 1 -- 172.21.15.59:0/3456444327 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe86400ac30 con 0x7fe86c0ce920 2024-03-06T07:28:37.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.220+0000 7fe862ffd640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85006b4d0 0x7fe85006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.220+0000 7fe862ffd640 1 -- 172.21.15.59:0/3456444327 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe86408a1e0 con 0x7fe86c0ce920 2024-03-06T07:28:37.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.221+0000 7fe872187640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85006b4d0 0x7fe85006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.221+0000 7fe872187640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85006b4d0 0x7fe85006d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe868006fd0 tx=0x7fe868008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.225+0000 7fe862ffd640 1 -- 172.21.15.59:0/3456444327 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe864020070 con 0x7fe86c0ce920 2024-03-06T07:28:37.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.444+0000 7fe8743eb640 1 -- 172.21.15.59:0/3456444327 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe86c0cdf00 con 0x7fe86c0ce920 2024-03-06T07:28:37.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.445+0000 7fe862ffd640 1 -- 172.21.15.59:0/3456444327 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fe864059300 con 0x7fe86c0ce920 2024-03-06T07:28:37.447 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:37.447 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6713,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/4023680293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":4023680293},{"type":"v1","addr":"172.21.15.164:6837","nonce":4023680293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7495,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/416613252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":416613252},{"type":"v1","addr":"172.21.15.164:6836","nonce":416613252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/525736351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":525736351},{"type":"v1","addr":"172.21.15.59:6833","nonce":525736351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":8592,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/3115514499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":3115514499},{"type":"v1","addr":"172.21.15.59:6835","nonce":3115514499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[]} 2024-03-06T07:28:37.447 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 75 2024-03-06T07:28:37.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 -- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85006b4d0 msgr2=0x7fe85006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85006b4d0 0x7fe85006d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe868006fd0 tx=0x7fe868008040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 -- 172.21.15.59:0/3456444327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0ce920 msgr2=0x7fe86c0ae270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0ce920 0x7fe86c0ae270 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fe86400c7b0 tx=0x7fe86400cc70 comp rx=0 tx=0).stop 2024-03-06T07:28:37.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 -- 172.21.15.59:0/3456444327 shutdown_connections 2024-03-06T07:28:37.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe85006b4d0 0x7fe85006d990 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe86c0ce920 0x7fe86c0ae270 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86c071ec0 0x7fe86c15b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 --2- 172.21.15.59:0/3456444327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe86c06b220 0x7fe86c14a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 -- 172.21.15.59:0/3456444327 >> 172.21.15.59:0/3456444327 conn(0x7fe86c06c0c0 msgr2=0x7fe86c15c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 -- 172.21.15.59:0/3456444327 shutdown_connections 2024-03-06T07:28:37.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.447+0000 7fe8743eb640 1 -- 172.21.15.59:0/3456444327 wait complete. 2024-03-06T07:28:37.460 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:37.460 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:28:37.461 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:37.461 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:37.461 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:28:37.461 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:28:37.461 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:28:37.461 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:37.461 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:37.461 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:28:37.462 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:37.462 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:37.462 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:37.462 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:28:37.462 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:37.462 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:37.470 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:37.470 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:37.471 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:28:37.471 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:28:37.474 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:37.474 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:37.475 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:37.475 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:37.512 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:37.512 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:37.513 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:28:37.513 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:37.516 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:37.516 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:37.516 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:37.517 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:37.517 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:28:37.588 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:28:37.588 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:28:37.601 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:28:37.601 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:28:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.623+0000 7f46c182a640 1 Processor -- start 2024-03-06T07:28:37.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.623+0000 7f46c182a640 1 -- start start 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.624+0000 7f46c182a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f46bc0b4460 0x7f46bc0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.624+0000 7f46c182a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f46bc162970 0x7f46bc0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.624+0000 7f46c182a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46bc161cf0 0x7f46bc162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.624+0000 7f46c182a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46bc161b00 con 0x7f46bc0b4460 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.624+0000 7f46c182a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46bc0cecc0 con 0x7f46bc162970 2024-03-06T07:28:37.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.624+0000 7f46c182a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0d22c0 con 0x7f46bc161cf0 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.624+0000 7f46baffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f46bc0b4460 0x7f46bc0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.624+0000 7f46baffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f46bc0b4460 0x7f46bc0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60214/0 (socket says 172.21.15.59:60214) 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.624+0000 7f46baffd640 1 -- 172.21.15.59:0/1193722769 learned_addr learned my addr 172.21.15.59:0/1193722769 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.624+0000 7f46bb7fe640 1 --2- 172.21.15.59:0/1193722769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46bc161cf0 0x7f46bc162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.624+0000 7f46ba7fc640 1 --2- 172.21.15.59:0/1193722769 >> v2:172.21.15.164:3300/0 conn(0x7f46bc162970 0x7f46bc0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.625+0000 7f46baffd640 1 -- 172.21.15.59:0/1193722769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46bc161cf0 msgr2=0x7f46bc162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.625+0000 7f46baffd640 1 --2- 172.21.15.59:0/1193722769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46bc161cf0 0x7f46bc162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.625+0000 7f46baffd640 1 -- 172.21.15.59:0/1193722769 >> v2:172.21.15.164:3300/0 conn(0x7f46bc162970 msgr2=0x7f46bc0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.625+0000 7f46baffd640 1 --2- 172.21.15.59:0/1193722769 >> v2:172.21.15.164:3300/0 conn(0x7f46bc162970 0x7f46bc0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.625+0000 7f46baffd640 1 -- 172.21.15.59:0/1193722769 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc0d25a0 con 0x7f46bc0b4460 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.625+0000 7f46bb7fe640 1 --2- 172.21.15.59:0/1193722769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46bc161cf0 0x7f46bc162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.625+0000 7f46ba7fc640 1 --2- 172.21.15.59:0/1193722769 >> v2:172.21.15.164:3300/0 conn(0x7f46bc162970 0x7f46bc0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:37.628 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.625+0000 7f46baffd640 1 --2- 172.21.15.59:0/1193722769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc0b4460 0x7f46bc0d4800 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0d17e0 tx=0x7f46ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ac697049ce2393 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.625+0000 7f46b9ffb640 1 -- 172.21.15.59:0/1193722769 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac035020 con 0x7f46bc0b4460 2024-03-06T07:28:37.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.625+0000 7f46b9ffb640 1 -- 172.21.15.59:0/1193722769 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ac031870 con 0x7f46bc0b4460 2024-03-06T07:28:37.629 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.626+0000 7f46b9ffb640 1 -- 172.21.15.59:0/1193722769 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac031c40 con 0x7f46bc0b4460 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.626+0000 7f46c182a640 1 -- 172.21.15.59:0/1193722769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc0b4460 msgr2=0x7f46bc0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.626+0000 7f46c182a640 1 --2- 172.21.15.59:0/1193722769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc0b4460 0x7f46bc0d4800 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0d17e0 tx=0x7f46ac02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.626+0000 7f46c182a640 1 -- 172.21.15.59:0/1193722769 shutdown_connections 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.626+0000 7f46c182a640 1 --2- 172.21.15.59:0/1193722769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46bc161cf0 0x7f46bc162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.626+0000 7f46c182a640 1 --2- 172.21.15.59:0/1193722769 >> v2:172.21.15.164:3300/0 conn(0x7f46bc162970 0x7f46bc0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.626+0000 7f46c182a640 1 --2- 172.21.15.59:0/1193722769 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc0b4460 0x7f46bc0d4800 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.626+0000 7f46c182a640 1 -- 172.21.15.59:0/1193722769 >> 172.21.15.59:0/1193722769 conn(0x7f46bc0d3ce0 msgr2=0x7f46bc0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.626+0000 7f46c182a640 1 -- 172.21.15.59:0/1193722769 shutdown_connections 2024-03-06T07:28:37.631 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.626+0000 7f46c182a640 1 -- 172.21.15.59:0/1193722769 wait complete. 2024-03-06T07:28:37.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.627+0000 7f46c182a640 1 Processor -- start 2024-03-06T07:28:37.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.627+0000 7f46c182a640 1 -- start start 2024-03-06T07:28:37.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.628+0000 7f46c182a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46bc0b4460 0x7f46bc155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.628+0000 7f46c182a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc161cf0 0x7f46bc0b2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.628+0000 7f46c182a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46bc162970 0x7f46bc0b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.628+0000 7f46c182a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0ce950 con 0x7f46bc161cf0 2024-03-06T07:28:37.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.628+0000 7f46c182a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0d2b60 con 0x7f46bc0b4460 2024-03-06T07:28:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.628+0000 7f46c182a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc0b3ac0 con 0x7f46bc162970 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.628+0000 7f46ba7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc161cf0 0x7f46bc0b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.628+0000 7f46ba7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc161cf0 0x7f46bc0b2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60230/0 (socket says 172.21.15.59:60230) 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.628+0000 7f46ba7fc640 1 -- 172.21.15.59:0/4183642366 learned_addr learned my addr 172.21.15.59:0/4183642366 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.628+0000 7f46baffd640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46bc0b4460 0x7f46bc155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.628+0000 7f46ba7fc640 1 -- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46bc162970 msgr2=0x7f46bc0b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.629+0000 7f46ba7fc640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46bc162970 0x7f46bc0b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.629+0000 7f46ba7fc640 1 -- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46bc0b4460 msgr2=0x7f46bc155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.629+0000 7f46ba7fc640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46bc0b4460 0x7f46bc155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.629+0000 7f46ba7fc640 1 -- 172.21.15.59:0/4183642366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc0b69a0 con 0x7f46bc161cf0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.629+0000 7f46ba7fc640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc161cf0 0x7f46bc0b2560 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f46b000e810 tx=0x7f46b000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.629+0000 7f46a3fff640 1 -- 172.21.15.59:0/4183642366 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b000f040 con 0x7f46bc161cf0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.629+0000 7f46a3fff640 1 -- 172.21.15.59:0/4183642366 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b0004230 con 0x7f46bc161cf0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.629+0000 7f46c182a640 1 -- 172.21.15.59:0/4183642366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46bc0b6c80 con 0x7f46bc161cf0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.629+0000 7f46a3fff640 1 -- 172.21.15.59:0/4183642366 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b000c6d0 con 0x7f46bc161cf0 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.629+0000 7f46c182a640 1 -- 172.21.15.59:0/4183642366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46bc0b7240 con 0x7f46bc161cf0 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.630+0000 7f46c182a640 1 -- 172.21.15.59:0/4183642366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46bc04f330 con 0x7f46bc161cf0 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.630+0000 7f46a3fff640 1 -- 172.21.15.59:0/4183642366 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46b000c870 con 0x7f46bc161cf0 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.631+0000 7f46a3fff640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f46a806b4d0 0x7f46a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.631+0000 7f46a3fff640 1 -- 172.21.15.59:0/4183642366 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f46b0089c10 con 0x7f46bc161cf0 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.631+0000 7f46baffd640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f46a806b4d0 0x7f46a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.632+0000 7f46baffd640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f46a806b4d0 0x7f46a806d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f46ac036000 tx=0x7f46ac008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.635+0000 7f46a3fff640 1 -- 172.21.15.59:0/4183642366 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46b0017020 con 0x7f46bc161cf0 2024-03-06T07:28:37.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:37.829+0000 7f46c182a640 1 -- 172.21.15.59:0/4183642366 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f46bc06d6d0 con 0x7f46bc161cf0 2024-03-06T07:28:38.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.909+0000 7f46a3fff640 1 -- 172.21.15.59:0/4183642366 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v122) v1 ==== 97+0+0 (secure 0 0 0) 0x7f46b0058dc0 con 0x7f46bc161cf0 2024-03-06T07:28:38.911 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-03-06T07:28:38.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 -- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f46a806b4d0 msgr2=0x7f46a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f46a806b4d0 0x7f46a806d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f46ac036000 tx=0x7f46ac008030 comp rx=0 tx=0).stop 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 -- 172.21.15.59:0/4183642366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc161cf0 msgr2=0x7f46bc0b2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc161cf0 0x7f46bc0b2560 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f46b000e810 tx=0x7f46b000ecd0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 -- 172.21.15.59:0/4183642366 shutdown_connections 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f46a806b4d0 0x7f46a806d990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46bc162970 0x7f46bc0b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46bc161cf0 0x7f46bc0b2560 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 --2- 172.21.15.59:0/4183642366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46bc0b4460 0x7f46bc155e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 -- 172.21.15.59:0/4183642366 >> 172.21.15.59:0/4183642366 conn(0x7f46bc0d3ce0 msgr2=0x7f46bc0cf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 -- 172.21.15.59:0/4183642366 shutdown_connections 2024-03-06T07:28:38.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:38.912+0000 7f46c182a640 1 -- 172.21.15.59:0/4183642366 wait complete. 2024-03-06T07:28:38.928 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:28:39.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.034+0000 7f5743481640 1 Processor -- start 2024-03-06T07:28:39.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.034+0000 7f5743481640 1 -- start start 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5743481640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f573c0cce40 0x7f573c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5743481640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c071f40 con 0x7f573c0cd7a0 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5743481640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c070a70 con 0x7f573c1515e0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5743481640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c0cd2f0 con 0x7f573c0cce40 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5741a1e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f573c0cd7a0 0x7f573c0cdb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40222/0 (socket says 172.21.15.59:40222) 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5741a1e640 1 -- 172.21.15.59:0/2421895467 learned_addr learned my addr 172.21.15.59:0/2421895467 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520868825 0 0) 0x7f573c071f40 con 0x7f573c0cd7a0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5728003520 con 0x7f573c0cd7a0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5740a1c640 1 --2- 172.21.15.59:0/2421895467 >> v2:172.21.15.164:3300/0 conn(0x7f573c0cce40 0x7f573c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1123965907 0 0) 0x7f5728003520 con 0x7f573c0cd7a0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 >> v2:172.21.15.164:3300/0 conn(0x7f573c0cce40 msgr2=0x7f573c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 --2- 172.21.15.59:0/2421895467 >> v2:172.21.15.164:3300/0 conn(0x7f573c0cce40 0x7f573c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 >> v1:172.21.15.164:6789/0 conn(0x7f573c1515e0 legacy=0x7f573c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c070d50 con 0x7f573c0cd7a0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f5734001a10 con 0x7f573c0cd7a0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 >> v1:172.21.15.59:6789/0 conn(0x7f573c0cd7a0 legacy=0x7f573c0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57280039b0 0x7f5728003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 0x7f5728004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57280051c0 0x7f572800cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0cd2f0 con 0x7f5728004800 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572800d170 con 0x7f57280051c0 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.035+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572800d370 con 0x7f57280039b0 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.036+0000 7f5740a1c640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 0x7f5728004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.036+0000 7f5741a1e640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57280051c0 0x7f572800cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.036+0000 7f574121d640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57280039b0 0x7f5728003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.036+0000 7f5740a1c640 1 -- 172.21.15.59:0/2421895467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57280039b0 msgr2=0x7f5728003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.036+0000 7f5740a1c640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57280039b0 0x7f5728003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.036+0000 7f5740a1c640 1 -- 172.21.15.59:0/2421895467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57280051c0 msgr2=0x7f572800cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.036+0000 7f5740a1c640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57280051c0 0x7f572800cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.036+0000 7f5740a1c640 1 -- 172.21.15.59:0/2421895467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5728003520 con 0x7f5728004800 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.036+0000 7f5741a1e640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57280051c0 0x7f572800cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.036+0000 7f574121d640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57280039b0 0x7f5728003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.036+0000 7f5741a1e640 1 -- 172.21.15.59:0/2421895467 reap_dead start 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.037+0000 7f5740a1c640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 0x7f5728004c80 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f572c009650 tx=0x7f5728005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f681bee3a632e40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.037+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c03b070 con 0x7f5728004800 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.037+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c02e8c0 con 0x7f5728004800 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.037+0000 7f5733fff640 1 -- 172.21.15.59:0/2421895467 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c009690 con 0x7f5728004800 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.037+0000 7f5743481640 1 -- 172.21.15.59:0/2421895467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 msgr2=0x7f5728004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.037+0000 7f5743481640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 0x7f5728004c80 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f572c009650 tx=0x7f5728005830 comp rx=0 tx=0).stop 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.037+0000 7f5743481640 1 -- 172.21.15.59:0/2421895467 shutdown_connections 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.037+0000 7f5743481640 1 --2- 172.21.15.59:0/2421895467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 0x7f5728004c80 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.037+0000 7f5743481640 1 -- 172.21.15.59:0/2421895467 >> 172.21.15.59:0/2421895467 conn(0x7f573c0cf8a0 msgr2=0x7f573c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.037+0000 7f5743481640 1 -- 172.21.15.59:0/2421895467 shutdown_connections 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.037+0000 7f5743481640 1 -- 172.21.15.59:0/2421895467 wait complete. 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.038+0000 7f5743481640 1 Processor -- start 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.039+0000 7f5743481640 1 -- start start 2024-03-06T07:28:39.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.039+0000 7f5743481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 0x7f573c1525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.039+0000 7f5743481640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f573c1f4210 0x7f573c13d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.039+0000 7f5743481640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f573c15c5a0 0x7f573c071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.039+0000 7f5743481640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c15b430 con 0x7f5728004800 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.039+0000 7f5743481640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c071310 con 0x7f573c1f4210 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.039+0000 7f5743481640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c15b5b0 con 0x7f573c15c5a0 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.039+0000 7f574121d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 0x7f573c1525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.039+0000 7f574121d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 0x7f573c1525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60240/0 (socket says 172.21.15.59:60240) 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.039+0000 7f574121d640 1 -- 172.21.15.59:0/210440568 learned_addr learned my addr 172.21.15.59:0/210440568 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f574121d640 1 -- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f573c15c5a0 msgr2=0x7f573c071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f5740a1c640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f573c1f4210 0x7f573c13d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f5741a1e640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f573c15c5a0 0x7f573c071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f574121d640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f573c15c5a0 0x7f573c071cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f574121d640 1 -- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f573c1f4210 msgr2=0x7f573c13d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f574121d640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f573c1f4210 0x7f573c13d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f574121d640 1 -- 172.21.15.59:0/210440568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c15ce20 con 0x7f5728004800 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f5740a1c640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f573c1f4210 0x7f573c13d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f5741a1e640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f573c15c5a0 0x7f573c071cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f574121d640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 0x7f573c1525c0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f573800e480 tx=0x7f573800e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f5731ffb640 1 -- 172.21.15.59:0/210440568 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57380052e0 con 0x7f5728004800 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f5731ffb640 1 -- 172.21.15.59:0/210440568 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5738018070 con 0x7f5728004800 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f5731ffb640 1 -- 172.21.15.59:0/210440568 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5738003810 con 0x7f5728004800 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f5743481640 1 -- 172.21.15.59:0/210440568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c152e30 con 0x7f5728004800 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.040+0000 7f5743481640 1 -- 172.21.15.59:0/210440568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c1533f0 con 0x7f5728004800 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.041+0000 7f5743481640 1 -- 172.21.15.59:0/210440568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c068840 con 0x7f5728004800 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.041+0000 7f5731ffb640 1 -- 172.21.15.59:0/210440568 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57380039b0 con 0x7f5728004800 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.042+0000 7f5731ffb640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572006b500 0x7f572006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.042+0000 7f5731ffb640 1 -- 172.21.15.59:0/210440568 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5738089ae0 con 0x7f5728004800 2024-03-06T07:28:39.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.042+0000 7f5740a1c640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572006b500 0x7f572006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.043+0000 7f5740a1c640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572006b500 0x7f572006d9c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f572c038000 tx=0x7f572c032000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.046+0000 7f5731ffb640 1 -- 172.21.15.59:0/210440568 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5738058d40 con 0x7f5728004800 2024-03-06T07:28:39.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.295+0000 7f5743481640 1 -- 172.21.15.59:0/210440568 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f573c154c80 con 0x7f5728004800 2024-03-06T07:28:39.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.295+0000 7f5731ffb640 1 -- 172.21.15.59:0/210440568 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f5738058640 con 0x7f5728004800 2024-03-06T07:28:39.297 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:39.297 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:28:39.297 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-03-06T07:28:39.297 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.297 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlODBZGBAAqV+TH2t2ZKyV9l13QEn6ew==", 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlR1MuJRAAp6OAQ9gZemWtwI1dh3H2SA==", 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:28:39.298 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlDXW2HxAAnKyuS1GW0wLO98/F9wdOAg==", 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.299 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehl5MO4LBAAKmt2dvKk7BtcCM4o7cSwgg==", 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBXGehlzdinFhAA0ACDAs9dtoY6EhUIveeIvA==", 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.300 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGehlH92eFRAANQceHGuQyrbI6gP1m/Gakw==", 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.301 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGehlMWpVFBAAbOG9nScbFlvKHXZMQJoQug==", 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGehlLCyWEhAAyfoW/FmU37QD2IohknBwWw==", 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:39.302 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBmGehldqv5GhAAWEloqqKgdpKPHjt7VKFodw==", 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-03-06T07:28:39.303 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGehlc0ENGRAA5V9Az+vxAR3OUwWpDP7w1A==", 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBsGehl5SYEFxAAPLyrHQV9SGguCV9f5VIkYg==", 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:28:39.304 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBvGehle3v0FBAA4xUPwX5couppHgi9VwSi2g==", 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA==", 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.305 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlqhlAOhAAQ4l3uI8MlcCH+NAIWo/UuA==", 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.306 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehltL/GNhAAZqSQ+UtwjI/PA95O1RGBNg==", 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGehlltx5BBAAFa/+/w8Hd8lZegfIWtpJOA==", 2024-03-06T07:28:39.307 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBSGehlMExaNhAAQ2FD7afktMhrTbMMAK1epQ==", 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:28:39.308 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlLSTzFBAACeLdHsCX9KJGEsq44aqTyw==", 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlAgqXEBAACCHZ9Lp2lEyF93xD4Ns9Nw==", 2024-03-06T07:28:39.309 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:28:39.310 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:28:39.310 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:39.310 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:28:39.310 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.310 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:39.310 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-06T07:28:39.310 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:28:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 -- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572006b500 msgr2=0x7f572006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572006b500 0x7f572006d9c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f572c038000 tx=0x7f572c032000 comp rx=0 tx=0).stop 2024-03-06T07:28:39.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 -- 172.21.15.59:0/210440568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 msgr2=0x7f573c1525c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 0x7f573c1525c0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f573800e480 tx=0x7f573800e940 comp rx=0 tx=0).stop 2024-03-06T07:28:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 -- 172.21.15.59:0/210440568 shutdown_connections 2024-03-06T07:28:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572006b500 0x7f572006d9c0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f573c15c5a0 0x7f573c071cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f573c1f4210 0x7f573c13d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 --2- 172.21.15.59:0/210440568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5728004800 0x7f573c1525c0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 -- 172.21.15.59:0/210440568 >> 172.21.15.59:0/210440568 conn(0x7f573c0cf8a0 msgr2=0x7f573c150700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 -- 172.21.15.59:0/210440568 shutdown_connections 2024-03-06T07:28:39.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.297+0000 7f5743481640 1 -- 172.21.15.59:0/210440568 wait complete. 2024-03-06T07:28:39.313 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:39.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.412+0000 7f0a04f8a640 1 Processor -- start 2024-03-06T07:28:39.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.412+0000 7f0a04f8a640 1 -- start start 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.413+0000 7f0a04f8a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0a001649b0 0x7f0a00166da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.413+0000 7f0a04f8a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000d0a70 con 0x7f0a0006afb0 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.413+0000 7f0a04f8a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000d0d10 con 0x7f0a000b4420 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.413+0000 7f0a04f8a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a000d0850 con 0x7f0a001649b0 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.413+0000 7f09fe59c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0a0006afb0 0x7f0a0006b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40234/0 (socket says 172.21.15.59:40234) 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.413+0000 7f09fe59c640 1 -- 172.21.15.59:0/4160439245 learned_addr learned my addr 172.21.15.59:0/4160439245 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.413+0000 7f09fed9d640 1 --2- 172.21.15.59:0/4160439245 >> v2:172.21.15.164:3300/0 conn(0x7f0a001649b0 0x7f0a00166da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.413+0000 7f09fd59a640 1 -- 172.21.15.59:0/4160439245 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595719682 0 0) 0x7f0a000d0a70 con 0x7f0a0006afb0 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.413+0000 7f09fd59a640 1 -- 172.21.15.59:0/4160439245 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ec003520 con 0x7f0a0006afb0 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.414+0000 7f09fd59a640 1 -- 172.21.15.59:0/4160439245 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3776445372 0 0) 0x7f09ec003520 con 0x7f0a0006afb0 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.414+0000 7f09fed9d640 1 -- 172.21.15.59:0/4160439245 >> v1:172.21.15.164:6789/0 conn(0x7f0a000b4420 legacy=0x7f0a000cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.414+0000 7f09fed9d640 1 -- 172.21.15.59:0/4160439245 >> v1:172.21.15.59:6789/0 conn(0x7f0a0006afb0 legacy=0x7f0a0006b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.414+0000 7f09fed9d640 1 -- 172.21.15.59:0/4160439245 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000d0ff0 con 0x7f0a001649b0 2024-03-06T07:28:39.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.415+0000 7f09fed9d640 1 --2- 172.21.15.59:0/4160439245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a001649b0 0x7f0a00166da0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009d90 tx=0x7f09f002bb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78d4c57cce91d152 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.415+0000 7f09fd59a640 1 -- 172.21.15.59:0/4160439245 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8001670 con 0x7f0a001649b0 2024-03-06T07:28:39.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.415+0000 7f09fd59a640 1 -- 172.21.15.59:0/4160439245 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f002d690 con 0x7f0a001649b0 2024-03-06T07:28:39.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.415+0000 7f09fd59a640 1 -- 172.21.15.59:0/4160439245 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f002da40 con 0x7f0a001649b0 2024-03-06T07:28:39.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.415+0000 7f0a04f8a640 1 -- 172.21.15.59:0/4160439245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a001649b0 msgr2=0x7f0a00166da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.415+0000 7f0a04f8a640 1 --2- 172.21.15.59:0/4160439245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a001649b0 0x7f0a00166da0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009d90 tx=0x7f09f002bb50 comp rx=0 tx=0).stop 2024-03-06T07:28:39.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.415+0000 7f0a04f8a640 1 -- 172.21.15.59:0/4160439245 shutdown_connections 2024-03-06T07:28:39.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.415+0000 7f0a04f8a640 1 --2- 172.21.15.59:0/4160439245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a001649b0 0x7f0a00166da0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.415+0000 7f0a04f8a640 1 -- 172.21.15.59:0/4160439245 >> 172.21.15.59:0/4160439245 conn(0x7f0a0006c0c0 msgr2=0x7f0a0006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.415+0000 7f0a04f8a640 1 -- 172.21.15.59:0/4160439245 shutdown_connections 2024-03-06T07:28:39.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.415+0000 7f0a04f8a640 1 -- 172.21.15.59:0/4160439245 wait complete. 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.417+0000 7f0a04f8a640 1 Processor -- start 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.417+0000 7f0a04f8a640 1 -- start start 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.417+0000 7f0a04f8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0006afb0 0x7f0a000b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.417+0000 7f0a04f8a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a000b4420 0x7f0a000b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.417+0000 7f0a04f8a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a001649b0 0x7f0a0012ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.417+0000 7f0a04f8a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000ceb90 con 0x7f0a000b4420 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.417+0000 7f0a04f8a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a00145840 con 0x7f0a001649b0 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.417+0000 7f0a04f8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000ce2a0 con 0x7f0a0006afb0 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fdd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a000b4420 0x7f0a000b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fdd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a000b4420 0x7f0a000b2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60252/0 (socket says 172.21.15.59:60252) 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fdd9b640 1 -- 172.21.15.59:0/3125759662 learned_addr learned my addr 172.21.15.59:0/3125759662 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fe59c640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0006afb0 0x7f0a000b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fed9d640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a001649b0 0x7f0a0012ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fdd9b640 1 -- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0006afb0 msgr2=0x7f0a000b1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fdd9b640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0006afb0 0x7f0a000b1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fdd9b640 1 -- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a001649b0 msgr2=0x7f0a0012ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fdd9b640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a001649b0 0x7f0a0012ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fdd9b640 1 -- 172.21.15.59:0/3125759662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a00156c60 con 0x7f0a000b4420 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fed9d640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a001649b0 0x7f0a0012ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fe59c640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0006afb0 0x7f0a000b1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09fdd9b640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a000b4420 0x7f0a000b2920 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c7b0 tx=0x7f09e800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09f6ffd640 1 -- 172.21.15.59:0/3125759662 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8017070 con 0x7f0a000b4420 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09f6ffd640 1 -- 172.21.15.59:0/3125759662 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e8005290 con 0x7f0a000b4420 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f0a04f8a640 1 -- 172.21.15.59:0/3125759662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a00156f40 con 0x7f0a000b4420 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f09f6ffd640 1 -- 172.21.15.59:0/3125759662 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e800a880 con 0x7f0a000b4420 2024-03-06T07:28:39.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.418+0000 7f0a04f8a640 1 -- 172.21.15.59:0/3125759662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a00157500 con 0x7f0a000b4420 2024-03-06T07:28:39.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.419+0000 7f0a04f8a640 1 -- 172.21.15.59:0/3125759662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0004f330 con 0x7f0a000b4420 2024-03-06T07:28:39.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.420+0000 7f09f6ffd640 1 -- 172.21.15.59:0/3125759662 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09e801a070 con 0x7f0a000b4420 2024-03-06T07:28:39.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.420+0000 7f09f6ffd640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09e006b4d0 0x7f09e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.420+0000 7f09f6ffd640 1 -- 172.21.15.59:0/3125759662 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f09e8089370 con 0x7f0a000b4420 2024-03-06T07:28:39.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.420+0000 7f09fe59c640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09e006b4d0 0x7f09e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.421+0000 7f09fe59c640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09e006b4d0 0x7f09e006d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f09f8001eb0 tx=0x7f09f8001d40 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.425+0000 7f09f6ffd640 1 -- 172.21.15.59:0/3125759662 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09e80585d0 con 0x7f0a000b4420 2024-03-06T07:28:39.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.606+0000 7f0a04f8a640 1 -- 172.21.15.59:0/3125759662 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a001581e0 con 0x7f0a000b4420 2024-03-06T07:28:39.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.607+0000 7f09f6ffd640 1 -- 172.21.15.59:0/3125759662 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+12889 (secure 0 0 0) 0x7f09e8057ed0 con 0x7f0a000b4420 2024-03-06T07:28:39.609 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:39.609 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":122,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:28:38.909136+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:39.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 -- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09e006b4d0 msgr2=0x7f09e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09e006b4d0 0x7f09e006d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f09f8001eb0 tx=0x7f09f8001d40 comp rx=0 tx=0).stop 2024-03-06T07:28:39.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 -- 172.21.15.59:0/3125759662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a000b4420 msgr2=0x7f0a000b2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a000b4420 0x7f0a000b2920 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c7b0 tx=0x7f09e800cc70 comp rx=0 tx=0).stop 2024-03-06T07:28:39.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 -- 172.21.15.59:0/3125759662 shutdown_connections 2024-03-06T07:28:39.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09e006b4d0 0x7f09e006d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a001649b0 0x7f0a0012ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a000b4420 0x7f0a000b2920 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 --2- 172.21.15.59:0/3125759662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a0006afb0 0x7f0a000b1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 -- 172.21.15.59:0/3125759662 >> 172.21.15.59:0/3125759662 conn(0x7f0a0006c0c0 msgr2=0x7f0a00144280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 -- 172.21.15.59:0/3125759662 shutdown_connections 2024-03-06T07:28:39.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.609+0000 7f0a04f8a640 1 -- 172.21.15.59:0/3125759662 wait complete. 2024-03-06T07:28:39.622 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:39.622 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:39.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.725+0000 7f77aaa78640 1 Processor -- start 2024-03-06T07:28:39.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.725+0000 7f77aaa78640 1 -- start start 2024-03-06T07:28:39.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.725+0000 7f77aaa78640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f77a41614d0 0x7f77a40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.725+0000 7f77aaa78640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f77a40d24b0 0x7f77a40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.725+0000 7f77aaa78640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a41618b0 con 0x7f77a40d1910 2024-03-06T07:28:39.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.725+0000 7f77aaa78640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a40cf400 con 0x7f77a40d24b0 2024-03-06T07:28:39.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.725+0000 7f77aaa78640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a40cf6a0 con 0x7f77a41614d0 2024-03-06T07:28:39.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.726+0000 7f77a9015640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f77a40d24b0 0x7f77a40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.726+0000 7f77a9015640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f77a40d24b0 0x7f77a40b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60260/0 (socket says 172.21.15.59:60260) 2024-03-06T07:28:39.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.726+0000 7f77a9015640 1 -- 172.21.15.59:0/1489552224 learned_addr learned my addr 172.21.15.59:0/1489552224 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:39.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.726+0000 7f77a8814640 1 --2- 172.21.15.59:0/1489552224 >> v2:172.21.15.164:3300/0 conn(0x7f77a41614d0 0x7f77a40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.726+0000 7f77a9015640 1 -- 172.21.15.59:0/1489552224 >> v2:172.21.15.164:3300/0 conn(0x7f77a41614d0 msgr2=0x7f77a40d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.726+0000 7f77a9015640 1 --2- 172.21.15.59:0/1489552224 >> v2:172.21.15.164:3300/0 conn(0x7f77a41614d0 0x7f77a40d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.726+0000 7f77a9015640 1 -- 172.21.15.59:0/1489552224 >> v1:172.21.15.164:6789/0 conn(0x7f77a40d1910 legacy=0x7f77a40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.726+0000 7f77a9015640 1 -- 172.21.15.59:0/1489552224 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a40cf980 con 0x7f77a40d24b0 2024-03-06T07:28:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77a9015640 1 --2- 172.21.15.59:0/1489552224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40d24b0 0x7f77a40b0e30 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f77a408c470 tx=0x7f779402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1241b90468ad1d2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77a37fe640 1 -- 172.21.15.59:0/1489552224 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794035020 con 0x7f77a40d24b0 2024-03-06T07:28:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77a37fe640 1 -- 172.21.15.59:0/1489552224 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7794031870 con 0x7f77a40d24b0 2024-03-06T07:28:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77a37fe640 1 -- 172.21.15.59:0/1489552224 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794031c40 con 0x7f77a40d24b0 2024-03-06T07:28:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77aaa78640 1 -- 172.21.15.59:0/1489552224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40d24b0 msgr2=0x7f77a40b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77aaa78640 1 --2- 172.21.15.59:0/1489552224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40d24b0 0x7f77a40b0e30 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f77a408c470 tx=0x7f779402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77aaa78640 1 -- 172.21.15.59:0/1489552224 shutdown_connections 2024-03-06T07:28:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77aaa78640 1 --2- 172.21.15.59:0/1489552224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40d24b0 0x7f77a40b0e30 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77aaa78640 1 --2- 172.21.15.59:0/1489552224 >> v2:172.21.15.164:3300/0 conn(0x7f77a41614d0 0x7f77a40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77aaa78640 1 -- 172.21.15.59:0/1489552224 >> 172.21.15.59:0/1489552224 conn(0x7f77a40d3d40 msgr2=0x7f77a40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77aaa78640 1 -- 172.21.15.59:0/1489552224 shutdown_connections 2024-03-06T07:28:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.727+0000 7f77aaa78640 1 -- 172.21.15.59:0/1489552224 wait complete. 2024-03-06T07:28:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.728+0000 7f77aaa78640 1 Processor -- start 2024-03-06T07:28:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.729+0000 7f77aaa78640 1 -- start start 2024-03-06T07:28:39.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.729+0000 7f77aaa78640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a40d1910 0x7f77a414ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.729+0000 7f77aaa78640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77a40d24b0 0x7f77a40c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.729+0000 7f77aaa78640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a41614d0 0x7f77a40b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.729+0000 7f77aaa78640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40b1370 con 0x7f77a41614d0 2024-03-06T07:28:39.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.729+0000 7f77aaa78640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40b0910 con 0x7f77a40d24b0 2024-03-06T07:28:39.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.729+0000 7f77aaa78640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40b14f0 con 0x7f77a40d1910 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.729+0000 7f77a9015640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a41614d0 0x7f77a40b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a9015640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a41614d0 0x7f77a40b6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60274/0 (socket says 172.21.15.59:60274) 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a9015640 1 -- 172.21.15.59:0/3930302594 learned_addr learned my addr 172.21.15.59:0/3930302594 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a8814640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a40d1910 0x7f77a414ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a9015640 1 -- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a40d1910 msgr2=0x7f77a414ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a3fff640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77a40d24b0 0x7f77a40c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a9015640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a40d1910 0x7f77a414ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a9015640 1 -- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77a40d24b0 msgr2=0x7f77a40c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a9015640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77a40d24b0 0x7f77a40c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a9015640 1 -- 172.21.15.59:0/3930302594 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a40b8ba0 con 0x7f77a41614d0 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a9015640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a41614d0 0x7f77a40b6930 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f7794009960 tx=0x7f77940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a17fa640 1 -- 172.21.15.59:0/3930302594 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794040070 con 0x7f77a41614d0 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a17fa640 1 -- 172.21.15.59:0/3930302594 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7794030d00 con 0x7f77a41614d0 2024-03-06T07:28:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77a17fa640 1 -- 172.21.15.59:0/3930302594 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794035040 con 0x7f77a41614d0 2024-03-06T07:28:39.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77aaa78640 1 -- 172.21.15.59:0/3930302594 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a40b8e80 con 0x7f77a41614d0 2024-03-06T07:28:39.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.730+0000 7f77aaa78640 1 -- 172.21.15.59:0/3930302594 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a40b9440 con 0x7f77a41614d0 2024-03-06T07:28:39.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.731+0000 7f77aaa78640 1 -- 172.21.15.59:0/3930302594 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a4068ff0 con 0x7f77a41614d0 2024-03-06T07:28:39.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.732+0000 7f77a17fa640 1 -- 172.21.15.59:0/3930302594 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7794038020 con 0x7f77a41614d0 2024-03-06T07:28:39.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.732+0000 7f77a17fa640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f779006b4d0 0x7f779006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.732+0000 7f77a17fa640 1 -- 172.21.15.59:0/3930302594 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f77940abce0 con 0x7f77a41614d0 2024-03-06T07:28:39.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.732+0000 7f77a8814640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f779006b4d0 0x7f779006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.733+0000 7f77a8814640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f779006b4d0 0x7f779006d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f77a40cacc0 tx=0x7f779c00b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.736+0000 7f77a17fa640 1 -- 172.21.15.59:0/3930302594 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f779407aeb0 con 0x7f77a41614d0 2024-03-06T07:28:39.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.917+0000 7f77aaa78640 1 -- 172.21.15.59:0/3930302594 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f77a40d4880 con 0x7f77a41614d0 2024-03-06T07:28:39.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.917+0000 7f77a17fa640 1 -- 172.21.15.59:0/3930302594 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7f779407a7b0 con 0x7f77a41614d0 2024-03-06T07:28:39.919 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:28:39.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 -- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f779006b4d0 msgr2=0x7f779006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f779006b4d0 0x7f779006d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f77a40cacc0 tx=0x7f779c00b040 comp rx=0 tx=0).stop 2024-03-06T07:28:39.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 -- 172.21.15.59:0/3930302594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a41614d0 msgr2=0x7f77a40b6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a41614d0 0x7f77a40b6930 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f7794009960 tx=0x7f77940046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 -- 172.21.15.59:0/3930302594 shutdown_connections 2024-03-06T07:28:39.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f779006b4d0 0x7f779006d990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a41614d0 0x7f77a40b6930 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77a40d24b0 0x7f77a40c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 --2- 172.21.15.59:0/3930302594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77a40d1910 0x7f77a414ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 -- 172.21.15.59:0/3930302594 >> 172.21.15.59:0/3930302594 conn(0x7f77a40d3d40 msgr2=0x7f77a40d1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 -- 172.21.15.59:0/3930302594 shutdown_connections 2024-03-06T07:28:39.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:39.919+0000 7f77aaa78640 1 -- 172.21.15.59:0/3930302594 wait complete. 2024-03-06T07:28:39.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:28:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.031+0000 7f01564c3640 1 Processor -- start 2024-03-06T07:28:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.031+0000 7f01564c3640 1 -- start start 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f01564c3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01501515e0 0x7f01500cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f01564c3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f01500cd7a0 0x7f01500cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f01564c3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0150071f40 con 0x7f015015b430 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f01564c3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015015b8e0 con 0x7f01501515e0 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f01564c3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0150070a70 con 0x7f01500cd7a0 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f014ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01501515e0 0x7f01500cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f014f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f015015b430 0x7f0150071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40250/0 (socket says 172.21.15.59:40250) 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f014f7fe640 1 -- 172.21.15.59:0/297375518 learned_addr learned my addr 172.21.15.59:0/297375518 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f0154a60640 1 --2- 172.21.15.59:0/297375518 >> v2:172.21.15.164:3300/0 conn(0x7f01500cd7a0 0x7f01500cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f014ffff640 1 -- 172.21.15.59:0/297375518 >> v2:172.21.15.164:3300/0 conn(0x7f01500cd7a0 msgr2=0x7f01500cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f014ffff640 1 --2- 172.21.15.59:0/297375518 >> v2:172.21.15.164:3300/0 conn(0x7f01500cd7a0 0x7f01500cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.032+0000 7f014ffff640 1 -- 172.21.15.59:0/297375518 >> v1:172.21.15.59:6789/0 conn(0x7f015015b430 legacy=0x7f0150071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.033+0000 7f014ffff640 1 -- 172.21.15.59:0/297375518 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0150070d50 con 0x7f01501515e0 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.033+0000 7f014effd640 1 -- 172.21.15.59:0/297375518 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794314764 0 0) 0x7f0150071f40 con 0x7f015015b430 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.033+0000 7f014ffff640 1 --2- 172.21.15.59:0/297375518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01501515e0 0x7f01500cce40 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f01500cc340 tx=0x7f014002cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94730ca3c8f2966 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.033+0000 7f014effd640 1 -- 172.21.15.59:0/297375518 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014003a070 con 0x7f01501515e0 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.033+0000 7f014effd640 1 -- 172.21.15.59:0/297375518 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014002db50 con 0x7f01501515e0 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.033+0000 7f014effd640 1 -- 172.21.15.59:0/297375518 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014000b040 con 0x7f01501515e0 2024-03-06T07:28:40.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.034+0000 7f01564c3640 1 -- 172.21.15.59:0/297375518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01501515e0 msgr2=0x7f01500cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.034+0000 7f01564c3640 1 --2- 172.21.15.59:0/297375518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01501515e0 0x7f01500cce40 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f01500cc340 tx=0x7f014002cda0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.034+0000 7f01564c3640 1 -- 172.21.15.59:0/297375518 shutdown_connections 2024-03-06T07:28:40.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.034+0000 7f01564c3640 1 --2- 172.21.15.59:0/297375518 >> v2:172.21.15.164:3300/0 conn(0x7f01500cd7a0 0x7f01500cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.034+0000 7f01564c3640 1 --2- 172.21.15.59:0/297375518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01501515e0 0x7f01500cce40 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.034+0000 7f01564c3640 1 -- 172.21.15.59:0/297375518 >> 172.21.15.59:0/297375518 conn(0x7f01500cf8a0 msgr2=0x7f01500cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.034+0000 7f01564c3640 1 -- 172.21.15.59:0/297375518 shutdown_connections 2024-03-06T07:28:40.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.034+0000 7f01564c3640 1 -- 172.21.15.59:0/297375518 wait complete. 2024-03-06T07:28:40.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.035+0000 7f01564c3640 1 Processor -- start 2024-03-06T07:28:40.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.035+0000 7f01564c3640 1 -- start start 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f01564c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500cd7a0 0x7f015015e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f01564c3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01501515e0 0x7f015014d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f01564c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015015b430 0x7f015013b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f01564c3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01500ce0c0 con 0x7f01500cd7a0 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f01564c3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0150071310 con 0x7f01501515e0 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f01564c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01500cc3a0 con 0x7f015015b430 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f014ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500cd7a0 0x7f015015e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f014ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500cd7a0 0x7f015015e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60286/0 (socket says 172.21.15.59:60286) 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f014ffff640 1 -- 172.21.15.59:0/2868393014 learned_addr learned my addr 172.21.15.59:0/2868393014 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f014ffff640 1 -- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015015b430 msgr2=0x7f015013b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f014ffff640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015015b430 0x7f015013b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f014ffff640 1 -- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01501515e0 msgr2=0x7f015014d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f014ffff640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01501515e0 0x7f015014d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.036+0000 7f014ffff640 1 -- 172.21.15.59:0/2868393014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015015e870 con 0x7f01500cd7a0 2024-03-06T07:28:40.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.037+0000 7f014ffff640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500cd7a0 0x7f015015e290 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f0140000c00 tx=0x7f01400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.037+0000 7f014cff9640 1 -- 172.21.15.59:0/2868393014 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140045070 con 0x7f01500cd7a0 2024-03-06T07:28:40.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.037+0000 7f014cff9640 1 -- 172.21.15.59:0/2868393014 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014003e550 con 0x7f01500cd7a0 2024-03-06T07:28:40.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.037+0000 7f01564c3640 1 -- 172.21.15.59:0/2868393014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015014a950 con 0x7f01500cd7a0 2024-03-06T07:28:40.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.037+0000 7f014cff9640 1 -- 172.21.15.59:0/2868393014 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014003a070 con 0x7f01500cd7a0 2024-03-06T07:28:40.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.037+0000 7f01564c3640 1 -- 172.21.15.59:0/2868393014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0150147d90 con 0x7f01500cd7a0 2024-03-06T07:28:40.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.038+0000 7f01564c3640 1 -- 172.21.15.59:0/2868393014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015014ab20 con 0x7f01500cd7a0 2024-03-06T07:28:40.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.038+0000 7f014cff9640 1 -- 172.21.15.59:0/2868393014 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f014003e940 con 0x7f01500cd7a0 2024-03-06T07:28:40.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.039+0000 7f014cff9640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f013006b500 0x7f013006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.039+0000 7f014cff9640 1 -- 172.21.15.59:0/2868393014 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f01400aba20 con 0x7f01500cd7a0 2024-03-06T07:28:40.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.039+0000 7f014f7fe640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f013006b500 0x7f013006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.039+0000 7f014f7fe640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f013006b500 0x7f013006d9c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0138005880 tx=0x7f0138007a00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.043+0000 7f014cff9640 1 -- 172.21.15.59:0/2868393014 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f014007ab60 con 0x7f01500cd7a0 2024-03-06T07:28:40.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.226+0000 7f01564c3640 1 -- 172.21.15.59:0/2868393014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0150068840 con 0x7f01500cd7a0 2024-03-06T07:28:40.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.226+0000 7f014cff9640 1 -- 172.21.15.59:0/2868393014 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v122) v1 ==== 84+0+65 (secure 0 0 0) 0x7f014007a460 con 0x7f01500cd7a0 2024-03-06T07:28:40.228 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:40.228 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-03-06T07:28:40.228 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:28:40.228 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-03-06T07:28:40.228 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-03-06T07:28:40.228 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:40.228 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-03-06T07:28:40.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 -- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f013006b500 msgr2=0x7f013006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f013006b500 0x7f013006d9c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0138005880 tx=0x7f0138007a00 comp rx=0 tx=0).stop 2024-03-06T07:28:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 -- 172.21.15.59:0/2868393014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500cd7a0 msgr2=0x7f015015e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500cd7a0 0x7f015015e290 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f0140000c00 tx=0x7f01400046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 -- 172.21.15.59:0/2868393014 shutdown_connections 2024-03-06T07:28:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f013006b500 0x7f013006d9c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015015b430 0x7f015013b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01501515e0 0x7f015014d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 --2- 172.21.15.59:0/2868393014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01500cd7a0 0x7f015015e290 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 -- 172.21.15.59:0/2868393014 >> 172.21.15.59:0/2868393014 conn(0x7f01500cf8a0 msgr2=0x7f015014e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 -- 172.21.15.59:0/2868393014 shutdown_connections 2024-03-06T07:28:40.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.228+0000 7f01564c3640 1 -- 172.21.15.59:0/2868393014 wait complete. 2024-03-06T07:28:40.241 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:28:40.242 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:28:40.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.345+0000 7f539710c640 1 Processor -- start 2024-03-06T07:28:40.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.346+0000 7f539710c640 1 -- start start 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.346+0000 7f539710c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f539006be80 0x7f53900b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.346+0000 7f539710c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5390068b00 con 0x7f539015f9d0 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.346+0000 7f539710c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539015e4e0 con 0x7f53900b5ca0 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.346+0000 7f539710c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f539015edf0 con 0x7f539006be80 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.346+0000 7f53956a9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f539015f9d0 0x7f539015fe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40252/0 (socket says 172.21.15.59:40252) 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.346+0000 7f53956a9640 1 -- 172.21.15.59:0/299948829 learned_addr learned my addr 172.21.15.59:0/299948829 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013272452 0 0) 0x7f5390068b00 con 0x7f539015f9d0 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f5394ea8640 1 --2- 172.21.15.59:0/299948829 >> v2:172.21.15.164:3300/0 conn(0x7f539006be80 0x7f53900b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5378003520 con 0x7f539015f9d0 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3897536153 0 0) 0x7f5378003520 con 0x7f539015f9d0 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 >> v2:172.21.15.164:3300/0 conn(0x7f539006be80 msgr2=0x7f53900b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 --2- 172.21.15.59:0/299948829 >> v2:172.21.15.164:3300/0 conn(0x7f539006be80 0x7f53900b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 >> v1:172.21.15.164:6789/0 conn(0x7f53900b5ca0 legacy=0x7f539006c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53900b1900 con 0x7f539015f9d0 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f53880019f0 con 0x7f539015f9d0 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 >> v1:172.21.15.59:6789/0 conn(0x7f539015f9d0 legacy=0x7f539015fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 --2- 172.21.15.59:0/299948829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53780039b0 0x7f5378003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 --2- 172.21.15.59:0/299948829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5378004800 0x7f5378004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 --2- 172.21.15.59:0/299948829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 0x7f537800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539015edf0 con 0x7f5378005210 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537800c140 con 0x7f5378004800 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537800c340 con 0x7f53780039b0 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53956a9640 1 --2- 172.21.15.59:0/299948829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 0x7f537800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f5394ea8640 1 --2- 172.21.15.59:0/299948829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53780039b0 0x7f5378003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f5387fff640 1 --2- 172.21.15.59:0/299948829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5378004800 0x7f5378004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53956a9640 1 -- 172.21.15.59:0/299948829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53780039b0 msgr2=0x7f5378003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53956a9640 1 --2- 172.21.15.59:0/299948829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53780039b0 0x7f5378003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53956a9640 1 -- 172.21.15.59:0/299948829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5378004800 msgr2=0x7f5378004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53956a9640 1 --2- 172.21.15.59:0/299948829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5378004800 0x7f5378004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.347+0000 7f53956a9640 1 -- 172.21.15.59:0/299948829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5378003520 con 0x7f5378005210 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.348+0000 7f53956a9640 1 -- 172.21.15.59:0/299948829 reap_dead start 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.348+0000 7f53956a9640 1 --2- 172.21.15.59:0/299948829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 0x7f537800b9c0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f538800a860 tx=0x7f5378005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78cd220763b770d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.348+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538800a480 con 0x7f5378005210 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.348+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5388034630 con 0x7f5378005210 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.348+0000 7f53877fe640 1 -- 172.21.15.59:0/299948829 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388034a00 con 0x7f5378005210 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.349+0000 7f539710c640 1 -- 172.21.15.59:0/299948829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 msgr2=0x7f537800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.349+0000 7f539710c640 1 --2- 172.21.15.59:0/299948829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 0x7f537800b9c0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f538800a860 tx=0x7f5378005830 comp rx=0 tx=0).stop 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.349+0000 7f539710c640 1 -- 172.21.15.59:0/299948829 shutdown_connections 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.349+0000 7f539710c640 1 --2- 172.21.15.59:0/299948829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 0x7f537800b9c0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.349+0000 7f539710c640 1 -- 172.21.15.59:0/299948829 >> 172.21.15.59:0/299948829 conn(0x7f53900b4ed0 msgr2=0x7f53900b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.349+0000 7f539710c640 1 -- 172.21.15.59:0/299948829 shutdown_connections 2024-03-06T07:28:40.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.349+0000 7f539710c640 1 -- 172.21.15.59:0/299948829 wait complete. 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.350+0000 7f539710c640 1 Processor -- start 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.350+0000 7f539710c640 1 -- start start 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f539710c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 0x7f539015a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f539710c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53900ca8d0 0x7f53900cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f539710c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53900cec70 0x7f53900cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f539710c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539015ea80 con 0x7f5378005210 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f539710c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53901618a0 con 0x7f53900ca8d0 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f539710c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539015f280 con 0x7f53900cec70 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f5394ea8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 0x7f539015a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f5394ea8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 0x7f539015a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60308/0 (socket says 172.21.15.59:60308) 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f5394ea8640 1 -- 172.21.15.59:0/2147684266 learned_addr learned my addr 172.21.15.59:0/2147684266 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f5394ea8640 1 -- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53900cec70 msgr2=0x7f53900cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f5387fff640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53900ca8d0 0x7f53900cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f53956a9640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53900cec70 0x7f53900cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f5394ea8640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53900cec70 0x7f53900cf0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f5394ea8640 1 -- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53900ca8d0 msgr2=0x7f53900cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f5394ea8640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53900ca8d0 0x7f53900cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.351+0000 7f5394ea8640 1 -- 172.21.15.59:0/2147684266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539015ac80 con 0x7f5378005210 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.352+0000 7f5394ea8640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 0x7f539015a740 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f538c011910 tx=0x7f538c011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.352+0000 7f53857fa640 1 -- 172.21.15.59:0/2147684266 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c00faf0 con 0x7f5378005210 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.352+0000 7f53857fa640 1 -- 172.21.15.59:0/2147684266 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c00fc90 con 0x7f5378005210 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.352+0000 7f539710c640 1 -- 172.21.15.59:0/2147684266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53900c9830 con 0x7f5378005210 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.352+0000 7f53857fa640 1 -- 172.21.15.59:0/2147684266 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c014440 con 0x7f5378005210 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.352+0000 7f539710c640 1 -- 172.21.15.59:0/2147684266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53900c9df0 con 0x7f5378005210 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.353+0000 7f539710c640 1 -- 172.21.15.59:0/2147684266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539004f330 con 0x7f5378005210 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.353+0000 7f53857fa640 1 -- 172.21.15.59:0/2147684266 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f538c012040 con 0x7f5378005210 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.354+0000 7f53857fa640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537c06b4d0 0x7f537c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.354+0000 7f53857fa640 1 -- 172.21.15.59:0/2147684266 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f538c08a570 con 0x7f5378005210 2024-03-06T07:28:40.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.354+0000 7f5387fff640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537c06b4d0 0x7f537c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.354+0000 7f5387fff640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537c06b4d0 0x7f537c06d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5380000c00 tx=0x7f5380009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.358+0000 7f53857fa640 1 -- 172.21.15.59:0/2147684266 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f539015ac80 con 0x7f5378005210 2024-03-06T07:28:40.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.538+0000 7f539710c640 1 -- 172.21.15.59:0/2147684266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f539015b690 con 0x7f5378005210 2024-03-06T07:28:40.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.915+0000 7f53857fa640 1 -- 172.21.15.59:0/2147684266 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v123) v1 ==== 138+0+0 (secure 0 0 0) 0x7f538c019030 con 0x7f5378005210 2024-03-06T07:28:40.917 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-03-06T07:28:40.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 -- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537c06b4d0 msgr2=0x7f537c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537c06b4d0 0x7f537c06d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5380000c00 tx=0x7f5380009040 comp rx=0 tx=0).stop 2024-03-06T07:28:40.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 -- 172.21.15.59:0/2147684266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 msgr2=0x7f539015a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 0x7f539015a740 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f538c011910 tx=0x7f538c011dd0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 -- 172.21.15.59:0/2147684266 shutdown_connections 2024-03-06T07:28:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f537c06b4d0 0x7f537c06d990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53900cec70 0x7f53900cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53900ca8d0 0x7f53900cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 --2- 172.21.15.59:0/2147684266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5378005210 0x7f539015a740 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 -- 172.21.15.59:0/2147684266 >> 172.21.15.59:0/2147684266 conn(0x7f53900b4ed0 msgr2=0x7f539006e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 -- 172.21.15.59:0/2147684266 shutdown_connections 2024-03-06T07:28:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:40.918+0000 7f539710c640 1 -- 172.21.15.59:0/2147684266 wait complete. 2024-03-06T07:28:40.935 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:28:41.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.042+0000 7f8055d3e640 1 Processor -- start 2024-03-06T07:28:41.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.042+0000 7f8055d3e640 1 -- start start 2024-03-06T07:28:41.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.043+0000 7f8055d3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015def0 0x7f80500b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.043+0000 7f8055d3e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f805006cab0 0x7f80500ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.043+0000 7f8055d3e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805006d030 con 0x7f80500ce880 2024-03-06T07:28:41.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.043+0000 7f8055d3e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80500cecf0 con 0x7f805006cab0 2024-03-06T07:28:41.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.043+0000 7f8055d3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80500cf1c0 con 0x7f805015def0 2024-03-06T07:28:41.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.043+0000 7f804ffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f80500ce880 0x7f80500cfd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40266/0 (socket says 172.21.15.59:40266) 2024-03-06T07:28:41.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.043+0000 7f804ffff640 1 -- 172.21.15.59:0/1604223918 learned_addr learned my addr 172.21.15.59:0/1604223918 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:41.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550434451 0 0) 0x7f805006d030 con 0x7f80500ce880 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8038003520 con 0x7f80500ce880 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804f7fe640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015def0 0x7f80500b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804effd640 1 --2- 172.21.15.59:0/1604223918 >> v2:172.21.15.164:3300/0 conn(0x7f805006cab0 0x7f80500ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 425834038 0 0) 0x7f8038003520 con 0x7f80500ce880 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015def0 msgr2=0x7f80500b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015def0 0x7f80500b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 >> v2:172.21.15.164:3300/0 conn(0x7f805006cab0 msgr2=0x7f80500ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 --2- 172.21.15.59:0/1604223918 >> v2:172.21.15.164:3300/0 conn(0x7f805006cab0 0x7f80500ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805006d310 con 0x7f80500ce880 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f80480019f0 con 0x7f80500ce880 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 >> v1:172.21.15.59:6789/0 conn(0x7f80500ce880 legacy=0x7f80500cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80380039b0 0x7f8038003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8038004800 0x7f8038004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015def0 0x7f80380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80500cecf0 con 0x7f80380039b0 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803800b9c0 con 0x7f8038004800 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803800bb40 con 0x7f805015def0 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804f7fe640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80380039b0 0x7f8038003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804ffff640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015def0 0x7f80380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804effd640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8038004800 0x7f8038004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804f7fe640 1 -- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015def0 msgr2=0x7f80380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804f7fe640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015def0 0x7f80380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804f7fe640 1 -- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8038004800 msgr2=0x7f8038004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804f7fe640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8038004800 0x7f8038004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804f7fe640 1 -- 172.21.15.59:0/1604223918 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8038003520 con 0x7f80380039b0 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.044+0000 7f804effd640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8038004800 0x7f8038004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.045+0000 7f804f7fe640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80380039b0 0x7f8038003dd0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f8040009ac0 tx=0x7f8038005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baae4996260c15af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.045+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8040039070 con 0x7f80380039b0 2024-03-06T07:28:41.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.045+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804002c6b0 con 0x7f80380039b0 2024-03-06T07:28:41.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.045+0000 7f804e7fc640 1 -- 172.21.15.59:0/1604223918 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8040009690 con 0x7f80380039b0 2024-03-06T07:28:41.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.046+0000 7f8055d3e640 1 -- 172.21.15.59:0/1604223918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80380039b0 msgr2=0x7f8038003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.046+0000 7f8055d3e640 1 --2- 172.21.15.59:0/1604223918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80380039b0 0x7f8038003dd0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f8040009ac0 tx=0x7f8038005830 comp rx=0 tx=0).stop 2024-03-06T07:28:41.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.046+0000 7f804ffff640 1 -- 172.21.15.59:0/1604223918 reap_dead start 2024-03-06T07:28:41.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.046+0000 7f8055d3e640 1 -- 172.21.15.59:0/1604223918 shutdown_connections 2024-03-06T07:28:41.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.046+0000 7f8055d3e640 1 -- 172.21.15.59:0/1604223918 >> 172.21.15.59:0/1604223918 conn(0x7f80500b3690 msgr2=0x7f80500b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.046+0000 7f8055d3e640 1 -- 172.21.15.59:0/1604223918 shutdown_connections 2024-03-06T07:28:41.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.046+0000 7f8055d3e640 1 -- 172.21.15.59:0/1604223918 wait complete. 2024-03-06T07:28:41.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.047+0000 7f8055d3e640 1 Processor -- start 2024-03-06T07:28:41.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.047+0000 7f8055d3e640 1 -- start start 2024-03-06T07:28:41.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f8055d3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80380039b0 0x7f80500c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f8055d3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015c080 0x7f8050152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f8055d3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80500cad40 0x7f80500cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f8055d3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80500cc3f0 con 0x7f80500cad40 2024-03-06T07:28:41.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f8055d3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80500cc760 con 0x7f80380039b0 2024-03-06T07:28:41.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f8055d3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805015e2d0 con 0x7f805015c080 2024-03-06T07:28:41.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80500cad40 0x7f80500cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80500cad40 0x7f80500cb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60328/0 (socket says 172.21.15.59:60328) 2024-03-06T07:28:41.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804ffff640 1 -- 172.21.15.59:0/3983354658 learned_addr learned my addr 172.21.15.59:0/3983354658 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:41.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804effd640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015c080 0x7f8050152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804f7fe640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80380039b0 0x7f80500c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804ffff640 1 -- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015c080 msgr2=0x7f8050152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804ffff640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015c080 0x7f8050152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804ffff640 1 -- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80380039b0 msgr2=0x7f80500c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804ffff640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80380039b0 0x7f80500c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804ffff640 1 -- 172.21.15.59:0/3983354658 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805015eee0 con 0x7f80500cad40 2024-03-06T07:28:41.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804effd640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015c080 0x7f8050152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:41.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804f7fe640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80380039b0 0x7f80500c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:41.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.048+0000 7f804ffff640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80500cad40 0x7f80500cb1c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f8048001330 tx=0x7f8048007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.049+0000 7f8037fff640 1 -- 172.21.15.59:0/3983354658 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804800ace0 con 0x7f80500cad40 2024-03-06T07:28:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.049+0000 7f8037fff640 1 -- 172.21.15.59:0/3983354658 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8048019070 con 0x7f80500cad40 2024-03-06T07:28:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.049+0000 7f8055d3e640 1 -- 172.21.15.59:0/3983354658 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80500c9490 con 0x7f80500cad40 2024-03-06T07:28:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.049+0000 7f8037fff640 1 -- 172.21.15.59:0/3983354658 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048013630 con 0x7f80500cad40 2024-03-06T07:28:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.049+0000 7f8055d3e640 1 -- 172.21.15.59:0/3983354658 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805015f430 con 0x7f80500cad40 2024-03-06T07:28:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.050+0000 7f8055d3e640 1 -- 172.21.15.59:0/3983354658 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805015f630 con 0x7f80500cad40 2024-03-06T07:28:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.050+0000 7f8037fff640 1 -- 172.21.15.59:0/3983354658 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80480137d0 con 0x7f80500cad40 2024-03-06T07:28:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.050+0000 7f8037fff640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f803c06b4d0 0x7f803c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.051+0000 7f8037fff640 1 -- 172.21.15.59:0/3983354658 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f804808a670 con 0x7f80500cad40 2024-03-06T07:28:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.051+0000 7f804f7fe640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f803c06b4d0 0x7f803c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.051+0000 7f804f7fe640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f803c06b4d0 0x7f803c06d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8040036000 tx=0x7f8040030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.055+0000 7f8037fff640 1 -- 172.21.15.59:0/3983354658 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f805015f630 con 0x7f80500cad40 2024-03-06T07:28:41.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.235+0000 7f8055d3e640 1 -- 172.21.15.59:0/3983354658 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f80500cf9a0 con 0x7f80500cad40 2024-03-06T07:28:41.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.921+0000 7f8037fff640 1 -- 172.21.15.59:0/3983354658 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v124) v1 ==== 170+0+0 (secure 0 0 0) 0x7f80480596d0 con 0x7f80500cad40 2024-03-06T07:28:41.923 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-03-06T07:28:41.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.923+0000 7f8055d3e640 1 -- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f803c06b4d0 msgr2=0x7f803c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.923+0000 7f8055d3e640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f803c06b4d0 0x7f803c06d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8040036000 tx=0x7f8040030000 comp rx=0 tx=0).stop 2024-03-06T07:28:41.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.923+0000 7f8055d3e640 1 -- 172.21.15.59:0/3983354658 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80500cad40 msgr2=0x7f80500cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.923+0000 7f8055d3e640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80500cad40 0x7f80500cb1c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f8048001330 tx=0x7f8048007b20 comp rx=0 tx=0).stop 2024-03-06T07:28:41.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.923+0000 7f8055d3e640 1 -- 172.21.15.59:0/3983354658 shutdown_connections 2024-03-06T07:28:41.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.923+0000 7f8055d3e640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f803c06b4d0 0x7f803c06d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.923+0000 7f8055d3e640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f80500cad40 0x7f80500cb1c0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.923+0000 7f8055d3e640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f805015c080 0x7f8050152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.923+0000 7f8055d3e640 1 --2- 172.21.15.59:0/3983354658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80380039b0 0x7f80500c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.923+0000 7f8055d3e640 1 -- 172.21.15.59:0/3983354658 >> 172.21.15.59:0/3983354658 conn(0x7f80500b3690 msgr2=0x7f80500b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.924+0000 7f8055d3e640 1 -- 172.21.15.59:0/3983354658 shutdown_connections 2024-03-06T07:28:41.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:41.924+0000 7f8055d3e640 1 -- 172.21.15.59:0/3983354658 wait complete. 2024-03-06T07:28:41.950 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:28:42.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.058+0000 7fc3a0c83640 1 Processor -- start 2024-03-06T07:28:42.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.058+0000 7fc3a0c83640 1 -- start start 2024-03-06T07:28:42.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc3a0c83640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc39c06cc90 0x7fc39c06d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc3a0c83640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc39c0b5ee0 0x7fc39c06f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc3a0c83640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39c06d5b0 con 0x7fc39c06c290 2024-03-06T07:28:42.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc3a0c83640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc39c06ebf0 con 0x7fc39c06cc90 2024-03-06T07:28:42.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc3a0c83640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc39c0b6430 con 0x7fc39c0b5ee0 2024-03-06T07:28:42.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc39a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc39c06cc90 0x7fc39c06d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc39ad9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc39c06c290 0x7fc39c06fb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40268/0 (socket says 172.21.15.59:40268) 2024-03-06T07:28:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc39ad9d640 1 -- 172.21.15.59:0/4010773915 learned_addr learned my addr 172.21.15.59:0/4010773915 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc393fff640 1 --2- 172.21.15.59:0/4010773915 >> v2:172.21.15.164:3300/0 conn(0x7fc39c0b5ee0 0x7fc39c06f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc39a59c640 1 -- 172.21.15.59:0/4010773915 >> v2:172.21.15.164:3300/0 conn(0x7fc39c0b5ee0 msgr2=0x7fc39c06f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc39a59c640 1 --2- 172.21.15.59:0/4010773915 >> v2:172.21.15.164:3300/0 conn(0x7fc39c0b5ee0 0x7fc39c06f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc39a59c640 1 -- 172.21.15.59:0/4010773915 >> v1:172.21.15.59:6789/0 conn(0x7fc39c06c290 legacy=0x7fc39c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc399d9b640 1 -- 172.21.15.59:0/4010773915 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18758272 0 0) 0x7fc39c06d5b0 con 0x7fc39c06c290 2024-03-06T07:28:42.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.059+0000 7fc39a59c640 1 -- 172.21.15.59:0/4010773915 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c0b6710 con 0x7fc39c06cc90 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.060+0000 7fc39a59c640 1 --2- 172.21.15.59:0/4010773915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c06cc90 0x7fc39c06d070 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fc39c0723e0 tx=0x7fc38c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b16a02da893841 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.060+0000 7fc399d9b640 1 -- 172.21.15.59:0/4010773915 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c035020 con 0x7fc39c06cc90 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.060+0000 7fc399d9b640 1 -- 172.21.15.59:0/4010773915 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38c031830 con 0x7fc39c06cc90 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.060+0000 7fc399d9b640 1 -- 172.21.15.59:0/4010773915 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c031c00 con 0x7fc39c06cc90 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.060+0000 7fc3a0c83640 1 -- 172.21.15.59:0/4010773915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c06cc90 msgr2=0x7fc39c06d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.060+0000 7fc3a0c83640 1 --2- 172.21.15.59:0/4010773915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c06cc90 0x7fc39c06d070 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fc39c0723e0 tx=0x7fc38c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.061+0000 7fc3a0c83640 1 -- 172.21.15.59:0/4010773915 shutdown_connections 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.061+0000 7fc3a0c83640 1 --2- 172.21.15.59:0/4010773915 >> v2:172.21.15.164:3300/0 conn(0x7fc39c0b5ee0 0x7fc39c06f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.061+0000 7fc3a0c83640 1 --2- 172.21.15.59:0/4010773915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c06cc90 0x7fc39c06d070 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.061+0000 7fc3a0c83640 1 -- 172.21.15.59:0/4010773915 >> 172.21.15.59:0/4010773915 conn(0x7fc39c0b6ea0 msgr2=0x7fc39c0b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.061+0000 7fc3a0c83640 1 -- 172.21.15.59:0/4010773915 shutdown_connections 2024-03-06T07:28:42.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.061+0000 7fc3a0c83640 1 -- 172.21.15.59:0/4010773915 wait complete. 2024-03-06T07:28:42.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.062+0000 7fc3a0c83640 1 Processor -- start 2024-03-06T07:28:42.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.062+0000 7fc3a0c83640 1 -- start start 2024-03-06T07:28:42.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc3a0c83640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c06c290 0x7fc39c09e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc3a0c83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc39c06cc90 0x7fc39c0cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc3a0c83640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc39c0b5ee0 0x7fc39c0cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc3a0c83640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c06c850 con 0x7fc39c06c290 2024-03-06T07:28:42.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc3a0c83640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c06f120 con 0x7fc39c0b5ee0 2024-03-06T07:28:42.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc3a0c83640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc39c06c9d0 con 0x7fc39c06cc90 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc39a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c06c290 0x7fc39c09e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc39a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c06c290 0x7fc39c09e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60342/0 (socket says 172.21.15.59:60342) 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc39a59c640 1 -- 172.21.15.59:0/440931344 learned_addr learned my addr 172.21.15.59:0/440931344 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc393fff640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc39c06cc90 0x7fc39c0cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc39a59c640 1 -- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc39c06cc90 msgr2=0x7fc39c0cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc39a59c640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc39c06cc90 0x7fc39c0cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc39a59c640 1 -- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc39c0b5ee0 msgr2=0x7fc39c0cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc39ad9d640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc39c0b5ee0 0x7fc39c0cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc39a59c640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc39c0b5ee0 0x7fc39c0cd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc39a59c640 1 -- 172.21.15.59:0/440931344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c09ea30 con 0x7fc39c06c290 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc39ad9d640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc39c0b5ee0 0x7fc39c0cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc39a59c640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c06c290 0x7fc39c09e320 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fc38c009890 tx=0x7fc38c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc392ffd640 1 -- 172.21.15.59:0/440931344 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c041070 con 0x7fc39c06c290 2024-03-06T07:28:42.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc392ffd640 1 -- 172.21.15.59:0/440931344 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38c03aa30 con 0x7fc39c06c290 2024-03-06T07:28:42.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc3a0c83640 1 -- 172.21.15.59:0/440931344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc39c09ed10 con 0x7fc39c06c290 2024-03-06T07:28:42.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.063+0000 7fc392ffd640 1 -- 172.21.15.59:0/440931344 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38c035040 con 0x7fc39c06c290 2024-03-06T07:28:42.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.064+0000 7fc3a0c83640 1 -- 172.21.15.59:0/440931344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc39c09f2d0 con 0x7fc39c06c290 2024-03-06T07:28:42.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.065+0000 7fc3a0c83640 1 -- 172.21.15.59:0/440931344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39c04f330 con 0x7fc39c06c290 2024-03-06T07:28:42.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.065+0000 7fc392ffd640 1 -- 172.21.15.59:0/440931344 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc38c038020 con 0x7fc39c06c290 2024-03-06T07:28:42.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.065+0000 7fc392ffd640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc37c06b4d0 0x7fc37c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.065+0000 7fc392ffd640 1 -- 172.21.15.59:0/440931344 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fc38c0abc00 con 0x7fc39c06c290 2024-03-06T07:28:42.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.066+0000 7fc393fff640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc37c06b4d0 0x7fc37c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.066+0000 7fc393fff640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc37c06b4d0 0x7fc37c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc384006fd0 tx=0x7fc384008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.070+0000 7fc392ffd640 1 -- 172.21.15.59:0/440931344 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc38c0b2080 con 0x7fc39c06c290 2024-03-06T07:28:42.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.287+0000 7fc3a0c83640 1 -- 172.21.15.59:0/440931344 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc39c0af580 con 0x7fc39c06c290 2024-03-06T07:28:42.925 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.923+0000 7fc392ffd640 1 -- 172.21.15.59:0/440931344 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v77) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc38c07aae0 con 0x7fc39c06c290 2024-03-06T07:28:42.925 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 18 and data pool 19 2024-03-06T07:28:42.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 -- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc37c06b4d0 msgr2=0x7fc37c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc37c06b4d0 0x7fc37c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc384006fd0 tx=0x7fc384008040 comp rx=0 tx=0).stop 2024-03-06T07:28:42.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 -- 172.21.15.59:0/440931344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c06c290 msgr2=0x7fc39c09e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c06c290 0x7fc39c09e320 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fc38c009890 tx=0x7fc38c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 -- 172.21.15.59:0/440931344 shutdown_connections 2024-03-06T07:28:42.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc37c06b4d0 0x7fc37c06d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc39c0b5ee0 0x7fc39c0cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc39c06cc90 0x7fc39c0cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 --2- 172.21.15.59:0/440931344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc39c06c290 0x7fc39c09e320 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 -- 172.21.15.59:0/440931344 >> 172.21.15.59:0/440931344 conn(0x7fc39c0b6ea0 msgr2=0x7fc39c15c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 -- 172.21.15.59:0/440931344 shutdown_connections 2024-03-06T07:28:42.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:42.926+0000 7fc3a0c83640 1 -- 172.21.15.59:0/440931344 wait complete. 2024-03-06T07:28:42.953 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:43.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.056+0000 7f49417a0640 1 Processor -- start 2024-03-06T07:28:43.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.056+0000 7f49417a0640 1 -- start start 2024-03-06T07:28:43.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f49417a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f493c0cf590 0x7f493c0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f49417a0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c0b2fa0 con 0x7f493c0d20c0 2024-03-06T07:28:43.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f49417a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f493c0b3260 con 0x7f493c0cf190 2024-03-06T07:28:43.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f49417a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f493c0b3540 con 0x7f493c0cf590 2024-03-06T07:28:43.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f493bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f493c0d20c0 0x7f493c0d24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40280/0 (socket says 172.21.15.59:40280) 2024-03-06T07:28:43.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f493bfff640 1 -- 172.21.15.59:0/1593673031 learned_addr learned my addr 172.21.15.59:0/1593673031 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:43.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f4940f9f640 1 --2- 172.21.15.59:0/1593673031 >> v2:172.21.15.164:3300/0 conn(0x7f493c0cf590 0x7f493c0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f493affd640 1 -- 172.21.15.59:0/1593673031 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955347214 0 0) 0x7f493c0b2fa0 con 0x7f493c0d20c0 2024-03-06T07:28:43.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f493affd640 1 -- 172.21.15.59:0/1593673031 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4924003520 con 0x7f493c0d20c0 2024-03-06T07:28:43.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f493affd640 1 -- 172.21.15.59:0/1593673031 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2154098578 0 0) 0x7f4924003520 con 0x7f493c0d20c0 2024-03-06T07:28:43.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f4940f9f640 1 -- 172.21.15.59:0/1593673031 >> v1:172.21.15.164:6789/0 conn(0x7f493c0cf190 legacy=0x7f493c0b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f4940f9f640 1 -- 172.21.15.59:0/1593673031 >> v1:172.21.15.59:6789/0 conn(0x7f493c0d20c0 legacy=0x7f493c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.057+0000 7f4940f9f640 1 -- 172.21.15.59:0/1593673031 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c0b3820 con 0x7f493c0cf590 2024-03-06T07:28:43.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.058+0000 7f4940f9f640 1 --2- 172.21.15.59:0/1593673031 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c0cf590 0x7f493c0b4730 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4934009d90 tx=0x7f493402bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d26279bcb4c60e47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.058+0000 7f493affd640 1 -- 172.21.15.59:0/1593673031 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4934039070 con 0x7f493c0cf590 2024-03-06T07:28:43.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.058+0000 7f493affd640 1 -- 172.21.15.59:0/1593673031 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493402c830 con 0x7f493c0cf590 2024-03-06T07:28:43.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.058+0000 7f493affd640 1 -- 172.21.15.59:0/1593673031 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c001670 con 0x7f493c0cf590 2024-03-06T07:28:43.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.059+0000 7f49417a0640 1 -- 172.21.15.59:0/1593673031 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c0cf590 msgr2=0x7f493c0b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.059+0000 7f49417a0640 1 --2- 172.21.15.59:0/1593673031 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c0cf590 0x7f493c0b4730 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f4934009d90 tx=0x7f493402bd00 comp rx=0 tx=0).stop 2024-03-06T07:28:43.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.059+0000 7f49417a0640 1 -- 172.21.15.59:0/1593673031 shutdown_connections 2024-03-06T07:28:43.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.059+0000 7f49417a0640 1 --2- 172.21.15.59:0/1593673031 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c0cf590 0x7f493c0b4730 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.059+0000 7f49417a0640 1 -- 172.21.15.59:0/1593673031 >> 172.21.15.59:0/1593673031 conn(0x7f493c09ce20 msgr2=0x7f493c0d3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.059+0000 7f49417a0640 1 -- 172.21.15.59:0/1593673031 shutdown_connections 2024-03-06T07:28:43.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.059+0000 7f49417a0640 1 -- 172.21.15.59:0/1593673031 wait complete. 2024-03-06T07:28:43.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.060+0000 7f49417a0640 1 Processor -- start 2024-03-06T07:28:43.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.060+0000 7f49417a0640 1 -- start start 2024-03-06T07:28:43.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f49417a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493c0cf190 0x7f493c151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f49417a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c0cf590 0x7f493c0b7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f49417a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f493c0d20c0 0x7f493c09da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f49417a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c161630 con 0x7f493c0cf190 2024-03-06T07:28:43.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f49417a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f493c0d1af0 con 0x7f493c0cf590 2024-03-06T07:28:43.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f49417a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493c1617b0 con 0x7f493c0d20c0 2024-03-06T07:28:43.064 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f493bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493c0cf190 0x7f493c151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f493bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493c0cf190 0x7f493c151530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60358/0 (socket says 172.21.15.59:60358) 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f493bfff640 1 -- 172.21.15.59:0/3711303463 learned_addr learned my addr 172.21.15.59:0/3711303463 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f493b7fe640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c0cf590 0x7f493c0b7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f4940f9f640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f493c0d20c0 0x7f493c09da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f493bfff640 1 -- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f493c0d20c0 msgr2=0x7f493c09da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f493bfff640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f493c0d20c0 0x7f493c09da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f493bfff640 1 -- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c0cf590 msgr2=0x7f493c0b7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f493bfff640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c0cf590 0x7f493c0b7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.061+0000 7f493bfff640 1 -- 172.21.15.59:0/3711303463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c0b80e0 con 0x7f493c0cf190 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.062+0000 7f493bfff640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493c0cf190 0x7f493c151530 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f492c00c530 tx=0x7f492c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.062+0000 7f4938ff9640 1 -- 172.21.15.59:0/3711303463 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c018070 con 0x7f493c0cf190 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.062+0000 7f4938ff9640 1 -- 172.21.15.59:0/3711303463 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c007500 con 0x7f493c0cf190 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.062+0000 7f49417a0640 1 -- 172.21.15.59:0/3711303463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f493c0b83c0 con 0x7f493c0cf190 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.062+0000 7f4938ff9640 1 -- 172.21.15.59:0/3711303463 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c004a20 con 0x7f493c0cf190 2024-03-06T07:28:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.062+0000 7f49417a0640 1 -- 172.21.15.59:0/3711303463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f493c0b8980 con 0x7f493c0cf190 2024-03-06T07:28:43.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.063+0000 7f49417a0640 1 -- 172.21.15.59:0/3711303463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f493c068f80 con 0x7f493c0cf190 2024-03-06T07:28:43.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.063+0000 7f4938ff9640 1 -- 172.21.15.59:0/3711303463 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f492c01b070 con 0x7f493c0cf190 2024-03-06T07:28:43.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.064+0000 7f4938ff9640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f492806b4d0 0x7f492806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.064+0000 7f4938ff9640 1 -- 172.21.15.59:0/3711303463 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f492c0895e0 con 0x7f493c0cf190 2024-03-06T07:28:43.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.064+0000 7f493b7fe640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f492806b4d0 0x7f492806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.064+0000 7f493b7fe640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f492806b4d0 0x7f492806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f493c1f70d0 tx=0x7f4930009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.068+0000 7f4938ff9640 1 -- 172.21.15.59:0/3711303463 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f492c058460 con 0x7f493c0cf190 2024-03-06T07:28:43.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.249+0000 7f49417a0640 1 -- 172.21.15.59:0/3711303463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f493c0b2fa0 con 0x7f493c0cf190 2024-03-06T07:28:43.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.250+0000 7f4938ff9640 1 -- 172.21.15.59:0/3711303463 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f492c057d60 con 0x7f493c0cf190 2024-03-06T07:28:43.252 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:43.252 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":125,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:28:42.288503+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:40.558382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:41.255127+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:43.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 -- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f492806b4d0 msgr2=0x7f492806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f492806b4d0 0x7f492806d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f493c1f70d0 tx=0x7f4930009040 comp rx=0 tx=0).stop 2024-03-06T07:28:43.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 -- 172.21.15.59:0/3711303463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493c0cf190 msgr2=0x7f493c151530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493c0cf190 0x7f493c151530 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f492c00c530 tx=0x7f492c00c9f0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 -- 172.21.15.59:0/3711303463 shutdown_connections 2024-03-06T07:28:43.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f492806b4d0 0x7f492806d990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f493c0d20c0 0x7f493c09da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c0cf590 0x7f493c0b7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 --2- 172.21.15.59:0/3711303463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f493c0cf190 0x7f493c151530 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 -- 172.21.15.59:0/3711303463 >> 172.21.15.59:0/3711303463 conn(0x7f493c09ce20 msgr2=0x7f493c0d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 -- 172.21.15.59:0/3711303463 shutdown_connections 2024-03-06T07:28:43.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.252+0000 7f49417a0640 1 -- 172.21.15.59:0/3711303463 wait complete. 2024-03-06T07:28:43.266 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:43.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.366+0000 7ff44b75e640 1 Processor -- start 2024-03-06T07:28:43.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.366+0000 7ff44b75e640 1 -- start start 2024-03-06T07:28:43.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff44b75e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440d00c0 0x7ff4440d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff44b75e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff4440cf850 0x7ff4440cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff44b75e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4440d09e0 con 0x7ff4440ce1b0 2024-03-06T07:28:43.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff44b75e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff44406ca90 con 0x7ff4440cf850 2024-03-06T07:28:43.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff44b75e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44406df30 con 0x7ff4440d00c0 2024-03-06T07:28:43.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff4494fa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440d00c0 0x7ff4440d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff448cf9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff4440cf850 0x7ff4440cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff4494fa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440d00c0 0x7ff4440d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:34292/0 (socket says 172.21.15.59:34292) 2024-03-06T07:28:43.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff448cf9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff4440cf850 0x7ff4440cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:37706/0 (socket says 172.21.15.59:37706) 2024-03-06T07:28:43.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff4494fa640 1 -- 172.21.15.59:0/1993812234 learned_addr learned my addr 172.21.15.59:0/1993812234 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:43.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff448cf9640 1 -- 172.21.15.59:0/1993812234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440d00c0 msgr2=0x7ff4440d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff448cf9640 1 --2- 172.21.15.59:0/1993812234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440d00c0 0x7ff4440d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff448cf9640 1 -- 172.21.15.59:0/1993812234 >> v1:172.21.15.164:6789/0 conn(0x7ff4440ce1b0 legacy=0x7ff4440ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff448cf9640 1 -- 172.21.15.59:0/1993812234 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44415e380 con 0x7ff4440cf850 2024-03-06T07:28:43.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.367+0000 7ff4494fa640 1 --2- 172.21.15.59:0/1993812234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440d00c0 0x7ff4440d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.368+0000 7ff448cf9640 1 --2- 172.21.15.59:0/1993812234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4440cf850 0x7ff4440cd3c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff4340097a0 tx=0x7ff43402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4938dea7446122fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.368+0000 7ff43bfff640 1 -- 172.21.15.59:0/1993812234 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434035020 con 0x7ff4440cf850 2024-03-06T07:28:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.368+0000 7ff43bfff640 1 -- 172.21.15.59:0/1993812234 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff434031830 con 0x7ff4440cf850 2024-03-06T07:28:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.368+0000 7ff43bfff640 1 -- 172.21.15.59:0/1993812234 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434031c00 con 0x7ff4440cf850 2024-03-06T07:28:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.369+0000 7ff44b75e640 1 -- 172.21.15.59:0/1993812234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4440cf850 msgr2=0x7ff4440cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.369+0000 7ff44b75e640 1 --2- 172.21.15.59:0/1993812234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4440cf850 0x7ff4440cd3c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff4340097a0 tx=0x7ff43402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.369+0000 7ff44b75e640 1 -- 172.21.15.59:0/1993812234 shutdown_connections 2024-03-06T07:28:43.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.369+0000 7ff44b75e640 1 --2- 172.21.15.59:0/1993812234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4440cf850 0x7ff4440cd3c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.369+0000 7ff44b75e640 1 --2- 172.21.15.59:0/1993812234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440d00c0 0x7ff4440d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.369+0000 7ff44b75e640 1 -- 172.21.15.59:0/1993812234 >> 172.21.15.59:0/1993812234 conn(0x7ff4440b35a0 msgr2=0x7ff4440b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.369+0000 7ff44b75e640 1 -- 172.21.15.59:0/1993812234 shutdown_connections 2024-03-06T07:28:43.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.369+0000 7ff44b75e640 1 -- 172.21.15.59:0/1993812234 wait complete. 2024-03-06T07:28:43.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.370+0000 7ff44b75e640 1 Processor -- start 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.370+0000 7ff44b75e640 1 -- start start 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff44b75e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4440ce1b0 0x7ff4441589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff44b75e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440cf850 0x7ff4440c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff44b75e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4440d00c0 0x7ff4441f7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff44b75e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44406c440 con 0x7ff4440d00c0 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff44b75e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44406c160 con 0x7ff4440ce1b0 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff44b75e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44406c720 con 0x7ff4440cf850 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff449cfb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4440d00c0 0x7ff4441f7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff449cfb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4440d00c0 0x7ff4441f7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60372/0 (socket says 172.21.15.59:60372) 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff449cfb640 1 -- 172.21.15.59:0/431972815 learned_addr learned my addr 172.21.15.59:0/431972815 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff448cf9640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440cf850 0x7ff4440c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff449cfb640 1 -- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440cf850 msgr2=0x7ff4440c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff4494fa640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4440ce1b0 0x7ff4441589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff449cfb640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440cf850 0x7ff4440c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff449cfb640 1 -- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4440ce1b0 msgr2=0x7ff4441589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff449cfb640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4440ce1b0 0x7ff4441589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.371+0000 7ff449cfb640 1 -- 172.21.15.59:0/431972815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44415b0a0 con 0x7ff4440d00c0 2024-03-06T07:28:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.372+0000 7ff449cfb640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4440d00c0 0x7ff4441f7910 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00c7d0 tx=0x7ff43c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.372+0000 7ff439ffb640 1 -- 172.21.15.59:0/431972815 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c003b70 con 0x7ff4440d00c0 2024-03-06T07:28:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.372+0000 7ff439ffb640 1 -- 172.21.15.59:0/431972815 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43c003d30 con 0x7ff4440d00c0 2024-03-06T07:28:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.372+0000 7ff44b75e640 1 -- 172.21.15.59:0/431972815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44415b380 con 0x7ff4440d00c0 2024-03-06T07:28:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.372+0000 7ff439ffb640 1 -- 172.21.15.59:0/431972815 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c00aa30 con 0x7ff4440d00c0 2024-03-06T07:28:43.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.372+0000 7ff44b75e640 1 -- 172.21.15.59:0/431972815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44415b940 con 0x7ff4440d00c0 2024-03-06T07:28:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.373+0000 7ff44b75e640 1 -- 172.21.15.59:0/431972815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4440b0da0 con 0x7ff4440d00c0 2024-03-06T07:28:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.373+0000 7ff439ffb640 1 -- 172.21.15.59:0/431972815 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff43c00abd0 con 0x7ff4440d00c0 2024-03-06T07:28:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.374+0000 7ff439ffb640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff42806b4d0 0x7ff42806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.374+0000 7ff439ffb640 1 -- 172.21.15.59:0/431972815 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff43c08a0d0 con 0x7ff4440d00c0 2024-03-06T07:28:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.374+0000 7ff4494fa640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff42806b4d0 0x7ff42806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.374+0000 7ff4494fa640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff42806b4d0 0x7ff42806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff4400099e0 tx=0x7ff440008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.378+0000 7ff439ffb640 1 -- 172.21.15.59:0/431972815 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4440b0da0 con 0x7ff4440d00c0 2024-03-06T07:28:43.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.559+0000 7ff44b75e640 1 -- 172.21.15.59:0/431972815 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff444071290 con 0x7ff4440d00c0 2024-03-06T07:28:43.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.560+0000 7ff439ffb640 1 -- 172.21.15.59:0/431972815 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16680 (secure 0 0 0) 0x7ff43c058f50 con 0x7ff4440d00c0 2024-03-06T07:28:43.562 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:43.563 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":125,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:28:42.288503+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:40.558382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:41.255127+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:43.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.562+0000 7ff44b75e640 1 -- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff42806b4d0 msgr2=0x7ff42806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.562+0000 7ff44b75e640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff42806b4d0 0x7ff42806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff4400099e0 tx=0x7ff440008040 comp rx=0 tx=0).stop 2024-03-06T07:28:43.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.562+0000 7ff44b75e640 1 -- 172.21.15.59:0/431972815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4440d00c0 msgr2=0x7ff4441f7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.562+0000 7ff44b75e640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4440d00c0 0x7ff4441f7910 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00c7d0 tx=0x7ff43c00cc90 comp rx=0 tx=0).stop 2024-03-06T07:28:43.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.563+0000 7ff44b75e640 1 -- 172.21.15.59:0/431972815 shutdown_connections 2024-03-06T07:28:43.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.563+0000 7ff44b75e640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff42806b4d0 0x7ff42806d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.563+0000 7ff44b75e640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4440d00c0 0x7ff4441f7910 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.563+0000 7ff44b75e640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4440cf850 0x7ff4440c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.563+0000 7ff44b75e640 1 --2- 172.21.15.59:0/431972815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4440ce1b0 0x7ff4441589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.563+0000 7ff44b75e640 1 -- 172.21.15.59:0/431972815 >> 172.21.15.59:0/431972815 conn(0x7ff4440b35a0 msgr2=0x7ff44406fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.563+0000 7ff44b75e640 1 -- 172.21.15.59:0/431972815 shutdown_connections 2024-03-06T07:28:43.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.563+0000 7ff44b75e640 1 -- 172.21.15.59:0/431972815 wait complete. 2024-03-06T07:28:43.576 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:28:43.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.678+0000 7fc782dac640 1 Processor -- start 2024-03-06T07:28:43.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.678+0000 7fc782dac640 1 -- start start 2024-03-06T07:28:43.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc782dac640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc77c15def0 0x7fc77c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc782dac640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77c06d030 con 0x7fc77c0cea50 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc782dac640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77c06d310 con 0x7fc77c06cab0 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc782dac640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc77c0cf1c0 con 0x7fc77c15def0 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc781349640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc77c0cea50 0x7fc77c0ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44422/0 (socket says 172.21.15.59:44422) 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc781349640 1 -- 172.21.15.59:0/2642208354 learned_addr learned my addr 172.21.15.59:0/2642208354 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018775398 0 0) 0x7fc77c06d030 con 0x7fc77c0cea50 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc780b48640 1 --2- 172.21.15.59:0/2642208354 >> v2:172.21.15.164:3300/0 conn(0x7fc77c15def0 0x7fc77c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc764003520 con 0x7fc77c0cea50 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3262101547 0 0) 0x7fc764003520 con 0x7fc77c0cea50 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 >> v2:172.21.15.164:3300/0 conn(0x7fc77c15def0 msgr2=0x7fc77c0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc77b7fe640 1 --2- 172.21.15.59:0/2642208354 >> v2:172.21.15.164:3300/0 conn(0x7fc77c15def0 0x7fc77c0b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 >> v1:172.21.15.164:6789/0 conn(0x7fc77c06cab0 legacy=0x7fc77c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c0cc110 con 0x7fc77c0cea50 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fc7740019f0 con 0x7fc77c0cea50 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.679+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 >> v1:172.21.15.59:6789/0 conn(0x7fc77c0cea50 legacy=0x7fc77c0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc77b7fe640 1 --2- 172.21.15.59:0/2642208354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7640039b0 0x7fc764003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc77b7fe640 1 --2- 172.21.15.59:0/2642208354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc764004800 0x7fc764004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc77b7fe640 1 --2- 172.21.15.59:0/2642208354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7640051c0 0x7fc76400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76400c040 con 0x7fc7640051c0 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76400c1f0 con 0x7fc7640039b0 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76400c3f0 con 0x7fc764004800 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc781349640 1 --2- 172.21.15.59:0/2642208354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7640051c0 0x7fc76400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc780b48640 1 --2- 172.21.15.59:0/2642208354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7640039b0 0x7fc764003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc77bfff640 1 --2- 172.21.15.59:0/2642208354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc764004800 0x7fc764004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc781349640 1 -- 172.21.15.59:0/2642208354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc764004800 msgr2=0x7fc764004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc781349640 1 --2- 172.21.15.59:0/2642208354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc764004800 0x7fc764004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc781349640 1 -- 172.21.15.59:0/2642208354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7640039b0 msgr2=0x7fc764003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc781349640 1 --2- 172.21.15.59:0/2642208354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7640039b0 0x7fc764003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.680+0000 7fc781349640 1 -- 172.21.15.59:0/2642208354 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc764003520 con 0x7fc7640051c0 2024-03-06T07:28:43.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc781349640 1 -- 172.21.15.59:0/2642208354 reap_dead start 2024-03-06T07:28:43.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc781349640 1 --2- 172.21.15.59:0/2642208354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7640051c0 0x7fc76400b9c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fc77c0b7c50 tx=0x7fc764005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb74067b6eded5da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77400a4a0 con 0x7fc7640051c0 2024-03-06T07:28:43.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc774034650 con 0x7fc7640051c0 2024-03-06T07:28:43.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc77b7fe640 1 -- 172.21.15.59:0/2642208354 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc774034a40 con 0x7fc7640051c0 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc782dac640 1 -- 172.21.15.59:0/2642208354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7640051c0 msgr2=0x7fc76400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc782dac640 1 --2- 172.21.15.59:0/2642208354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7640051c0 0x7fc76400b9c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fc77c0b7c50 tx=0x7fc764005830 comp rx=0 tx=0).stop 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc782dac640 1 -- 172.21.15.59:0/2642208354 shutdown_connections 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc782dac640 1 --2- 172.21.15.59:0/2642208354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7640051c0 0x7fc76400b9c0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc782dac640 1 -- 172.21.15.59:0/2642208354 >> 172.21.15.59:0/2642208354 conn(0x7fc77c0b3690 msgr2=0x7fc77c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc782dac640 1 -- 172.21.15.59:0/2642208354 shutdown_connections 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.681+0000 7fc782dac640 1 -- 172.21.15.59:0/2642208354 wait complete. 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.683+0000 7fc782dac640 1 Processor -- start 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.683+0000 7fc782dac640 1 -- start start 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.683+0000 7fc782dac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7640051c0 0x7fc77c0b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.683+0000 7fc782dac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77c0b6f70 0x7fc77c150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.683+0000 7fc782dac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c0c75b0 0x7fc77c0c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.683+0000 7fc782dac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c15e2d0 con 0x7fc77c0c75b0 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.683+0000 7fc782dac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c0cc760 con 0x7fc77c0b6f70 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.683+0000 7fc782dac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c15e450 con 0x7fc7640051c0 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc781349640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c0c75b0 0x7fc77c0c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc781349640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c0c75b0 0x7fc77c0c7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48142/0 (socket says 172.21.15.59:48142) 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc781349640 1 -- 172.21.15.59:0/2259375933 learned_addr learned my addr 172.21.15.59:0/2259375933 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc780b48640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7640051c0 0x7fc77c0b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc77bfff640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77c0b6f70 0x7fc77c150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc781349640 1 -- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7640051c0 msgr2=0x7fc77c0b5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc781349640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7640051c0 0x7fc77c0b5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc781349640 1 -- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77c0b6f70 msgr2=0x7fc77c150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc781349640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77c0b6f70 0x7fc77c150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc781349640 1 -- 172.21.15.59:0/2259375933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c0b76e0 con 0x7fc77c0c75b0 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc781349640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c0c75b0 0x7fc77c0c7a40 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fc77400a940 tx=0x7fc774001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc7797fa640 1 -- 172.21.15.59:0/2259375933 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc774043070 con 0x7fc77c0c75b0 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc7797fa640 1 -- 172.21.15.59:0/2259375933 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc774004d60 con 0x7fc77c0c75b0 2024-03-06T07:28:43.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc782dac640 1 -- 172.21.15.59:0/2259375933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77c0b7400 con 0x7fc77c0c75b0 2024-03-06T07:28:43.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc7797fa640 1 -- 172.21.15.59:0/2259375933 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc774039040 con 0x7fc77c0c75b0 2024-03-06T07:28:43.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.684+0000 7fc782dac640 1 -- 172.21.15.59:0/2259375933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77c0b5580 con 0x7fc77c0c75b0 2024-03-06T07:28:43.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.685+0000 7fc782dac640 1 -- 172.21.15.59:0/2259375933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc77c09fcf0 con 0x7fc77c0c75b0 2024-03-06T07:28:43.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.686+0000 7fc7797fa640 1 -- 172.21.15.59:0/2259375933 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc774046070 con 0x7fc77c0c75b0 2024-03-06T07:28:43.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.686+0000 7fc7797fa640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc76806b4d0 0x7fc76806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.686+0000 7fc7797fa640 1 -- 172.21.15.59:0/2259375933 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7740ab560 con 0x7fc77c0c75b0 2024-03-06T07:28:43.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.686+0000 7fc780b48640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc76806b4d0 0x7fc76806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.687+0000 7fc780b48640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc76806b4d0 0x7fc76806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009a80 tx=0x7fc76c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.690+0000 7fc7797fa640 1 -- 172.21.15.59:0/2259375933 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc774079d90 con 0x7fc77c0c75b0 2024-03-06T07:28:43.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.909+0000 7fc782dac640 1 -- 172.21.15.59:0/2259375933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc77c068ff0 con 0x7fc77c0c75b0 2024-03-06T07:28:43.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.939+0000 7fc7797fa640 1 -- 172.21.15.59:0/2259375933 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v78) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc77407de30 con 0x7fc77c0c75b0 2024-03-06T07:28:43.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 -- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc76806b4d0 msgr2=0x7fc76806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc76806b4d0 0x7fc76806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009a80 tx=0x7fc76c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:43.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 -- 172.21.15.59:0/2259375933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c0c75b0 msgr2=0x7fc77c0c7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c0c75b0 0x7fc77c0c7a40 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fc77400a940 tx=0x7fc774001dc0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 -- 172.21.15.59:0/2259375933 shutdown_connections 2024-03-06T07:28:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc76806b4d0 0x7fc76806d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc77c0c75b0 0x7fc77c0c7a40 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc77c0b6f70 0x7fc77c150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 --2- 172.21.15.59:0/2259375933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7640051c0 0x7fc77c0b5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 -- 172.21.15.59:0/2259375933 >> 172.21.15.59:0/2259375933 conn(0x7fc77c0b3690 msgr2=0x7fc77c06f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 -- 172.21.15.59:0/2259375933 shutdown_connections 2024-03-06T07:28:43.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:43.942+0000 7fc782dac640 1 -- 172.21.15.59:0/2259375933 wait complete. 2024-03-06T07:28:43.959 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:28:43.959 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:28:44.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.064+0000 7fbf42716640 1 Processor -- start 2024-03-06T07:28:44.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.064+0000 7fbf42716640 1 -- start start 2024-03-06T07:28:44.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.065+0000 7fbf42716640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf3c1614d0 0x7fbf3c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.065+0000 7fbf42716640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbf3c0d1910 0x7fbf3c0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.065+0000 7fbf42716640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3c1618b0 con 0x7fbf3c0d22e0 2024-03-06T07:28:44.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.065+0000 7fbf42716640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3c0cf400 con 0x7fbf3c1614d0 2024-03-06T07:28:44.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.065+0000 7fbf42716640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3c0d2710 con 0x7fbf3c0d1910 2024-03-06T07:28:44.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.065+0000 7fbf3bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf3c1614d0 0x7fbf3c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.065+0000 7fbf3bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf3c1614d0 0x7fbf3c0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48148/0 (socket says 172.21.15.59:48148) 2024-03-06T07:28:44.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.065+0000 7fbf3bfff640 1 -- 172.21.15.59:0/2360066097 learned_addr learned my addr 172.21.15.59:0/2360066097 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:44.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.066+0000 7fbf3b7fe640 1 --2- 172.21.15.59:0/2360066097 >> v2:172.21.15.164:3300/0 conn(0x7fbf3c0d1910 0x7fbf3c0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.066+0000 7fbf3bfff640 1 -- 172.21.15.59:0/2360066097 >> v2:172.21.15.164:3300/0 conn(0x7fbf3c0d1910 msgr2=0x7fbf3c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.066+0000 7fbf3bfff640 1 --2- 172.21.15.59:0/2360066097 >> v2:172.21.15.164:3300/0 conn(0x7fbf3c0d1910 0x7fbf3c0d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.066+0000 7fbf3bfff640 1 -- 172.21.15.59:0/2360066097 >> v1:172.21.15.164:6789/0 conn(0x7fbf3c0d22e0 legacy=0x7fbf3c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.066+0000 7fbf3bfff640 1 -- 172.21.15.59:0/2360066097 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c0cf6a0 con 0x7fbf3c1614d0 2024-03-06T07:28:44.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.066+0000 7fbf3b7fe640 1 --2- 172.21.15.59:0/2360066097 >> v2:172.21.15.164:3300/0 conn(0x7fbf3c0d1910 0x7fbf3c0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:44.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.066+0000 7fbf3bfff640 1 --2- 172.21.15.59:0/2360066097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf3c1614d0 0x7fbf3c0d4860 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c08c470 tx=0x7fbf2c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45762cb60d0b5c75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.067+0000 7fbf3affd640 1 -- 172.21.15.59:0/2360066097 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c035020 con 0x7fbf3c1614d0 2024-03-06T07:28:44.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.067+0000 7fbf3affd640 1 -- 172.21.15.59:0/2360066097 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c031640 con 0x7fbf3c1614d0 2024-03-06T07:28:44.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.067+0000 7fbf3affd640 1 -- 172.21.15.59:0/2360066097 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c02d040 con 0x7fbf3c1614d0 2024-03-06T07:28:44.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.067+0000 7fbf42716640 1 -- 172.21.15.59:0/2360066097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf3c1614d0 msgr2=0x7fbf3c0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.067+0000 7fbf42716640 1 --2- 172.21.15.59:0/2360066097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf3c1614d0 0x7fbf3c0d4860 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c08c470 tx=0x7fbf2c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.067+0000 7fbf42716640 1 -- 172.21.15.59:0/2360066097 shutdown_connections 2024-03-06T07:28:44.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.067+0000 7fbf42716640 1 --2- 172.21.15.59:0/2360066097 >> v2:172.21.15.164:3300/0 conn(0x7fbf3c0d1910 0x7fbf3c0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.067+0000 7fbf42716640 1 --2- 172.21.15.59:0/2360066097 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf3c1614d0 0x7fbf3c0d4860 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.067+0000 7fbf42716640 1 -- 172.21.15.59:0/2360066097 >> 172.21.15.59:0/2360066097 conn(0x7fbf3c0d3d40 msgr2=0x7fbf3c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.067+0000 7fbf42716640 1 -- 172.21.15.59:0/2360066097 shutdown_connections 2024-03-06T07:28:44.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.067+0000 7fbf42716640 1 -- 172.21.15.59:0/2360066097 wait complete. 2024-03-06T07:28:44.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.068+0000 7fbf42716640 1 Processor -- start 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf42716640 1 -- start start 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf42716640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf3c0d1910 0x7fbf3c0c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf42716640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf3c0d22e0 0x7fbf3c0b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf42716640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf3c1614d0 0x7fbf3c0b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf42716640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0b0910 con 0x7fbf3c0d1910 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf42716640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0cfc60 con 0x7fbf3c1614d0 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf42716640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0b1e40 con 0x7fbf3c0d22e0 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf3bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf3c0d1910 0x7fbf3c0c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf3bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf3c0d1910 0x7fbf3c0c92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48152/0 (socket says 172.21.15.59:48152) 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf3bfff640 1 -- 172.21.15.59:0/3173426027 learned_addr learned my addr 172.21.15.59:0/3173426027 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf40cb3640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf3c1614d0 0x7fbf3c0b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.069+0000 7fbf3b7fe640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf3c0d22e0 0x7fbf3c0b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf3bfff640 1 -- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf3c0d22e0 msgr2=0x7fbf3c0b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf3bfff640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf3c0d22e0 0x7fbf3c0b6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf3bfff640 1 -- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf3c1614d0 msgr2=0x7fbf3c0b6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf3bfff640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf3c1614d0 0x7fbf3c0b6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf3bfff640 1 -- 172.21.15.59:0/3173426027 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c0ca200 con 0x7fbf3c0d1910 2024-03-06T07:28:44.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf3b7fe640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf3c0d22e0 0x7fbf3c0b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:44.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf3bfff640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf3c0d1910 0x7fbf3c0c92d0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c02b930 tx=0x7fbf2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf38ff9640 1 -- 172.21.15.59:0/3173426027 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c040070 con 0x7fbf3c0d1910 2024-03-06T07:28:44.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf38ff9640 1 -- 172.21.15.59:0/3173426027 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c039c10 con 0x7fbf3c0d1910 2024-03-06T07:28:44.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf42716640 1 -- 172.21.15.59:0/3173426027 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3c0c9c40 con 0x7fbf3c0d1910 2024-03-06T07:28:44.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf38ff9640 1 -- 172.21.15.59:0/3173426027 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c035040 con 0x7fbf3c0d1910 2024-03-06T07:28:44.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.070+0000 7fbf42716640 1 -- 172.21.15.59:0/3173426027 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3c0cb780 con 0x7fbf3c0d1910 2024-03-06T07:28:44.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.071+0000 7fbf42716640 1 -- 172.21.15.59:0/3173426027 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3c0d05b0 con 0x7fbf3c0d1910 2024-03-06T07:28:44.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.072+0000 7fbf38ff9640 1 -- 172.21.15.59:0/3173426027 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf2c038020 con 0x7fbf3c0d1910 2024-03-06T07:28:44.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.072+0000 7fbf38ff9640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf2806b4d0 0x7fbf2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.072+0000 7fbf38ff9640 1 -- 172.21.15.59:0/3173426027 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf2c0abc90 con 0x7fbf3c0d1910 2024-03-06T07:28:44.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.073+0000 7fbf3b7fe640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf2806b4d0 0x7fbf2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.073+0000 7fbf3b7fe640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf2806b4d0 0x7fbf2806d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c15db60 tx=0x7fbf30008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.077+0000 7fbf38ff9640 1 -- 172.21.15.59:0/3173426027 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf3c0d05b0 con 0x7fbf3c0d1910 2024-03-06T07:28:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.362+0000 7fbf42716640 1 -- 172.21.15.59:0/3173426027 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbf3c068ff0 con 0x7fbf3c0d1910 2024-03-06T07:28:44.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.944+0000 7fbf38ff9640 1 -- 172.21.15.59:0/3173426027 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v79) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbf2c031610 con 0x7fbf3c0d1910 2024-03-06T07:28:44.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.946+0000 7fbf42716640 1 -- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf2806b4d0 msgr2=0x7fbf2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.946+0000 7fbf42716640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf2806b4d0 0x7fbf2806d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c15db60 tx=0x7fbf30008040 comp rx=0 tx=0).stop 2024-03-06T07:28:44.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.946+0000 7fbf42716640 1 -- 172.21.15.59:0/3173426027 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf3c0d1910 msgr2=0x7fbf3c0c92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.946+0000 7fbf42716640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf3c0d1910 0x7fbf3c0c92d0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c02b930 tx=0x7fbf2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.946+0000 7fbf42716640 1 -- 172.21.15.59:0/3173426027 shutdown_connections 2024-03-06T07:28:44.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.946+0000 7fbf42716640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf2806b4d0 0x7fbf2806d990 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.946+0000 7fbf42716640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf3c1614d0 0x7fbf3c0b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.946+0000 7fbf42716640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf3c0d22e0 0x7fbf3c0b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.946+0000 7fbf42716640 1 --2- 172.21.15.59:0/3173426027 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf3c0d1910 0x7fbf3c0c92d0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.946+0000 7fbf42716640 1 -- 172.21.15.59:0/3173426027 >> 172.21.15.59:0/3173426027 conn(0x7fbf3c0d3d40 msgr2=0x7fbf3c0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.947+0000 7fbf42716640 1 -- 172.21.15.59:0/3173426027 shutdown_connections 2024-03-06T07:28:44.949 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:44.947+0000 7fbf42716640 1 -- 172.21.15.59:0/3173426027 wait complete. 2024-03-06T07:28:44.959 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:28:44.959 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.063+0000 7f5132116640 1 Processor -- start 2024-03-06T07:28:45.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.063+0000 7f5132116640 1 -- start start 2024-03-06T07:28:45.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f5132116640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c15dfe0 0x7f512c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f5132116640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f512c0d29c0 0x7f512c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f5132116640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f512c0b3430 con 0x7f512c0cf830 2024-03-06T07:28:45.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f5132116640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512c0b31e0 con 0x7f512c0d29c0 2024-03-06T07:28:45.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f5132116640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512c0b3d60 con 0x7f512c15dfe0 2024-03-06T07:28:45.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f512bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f512c0d29c0 0x7f512c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f512affd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f512c0cf830 0x7f512c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44432/0 (socket says 172.21.15.59:44432) 2024-03-06T07:28:45.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f512affd640 1 -- 172.21.15.59:0/1517834979 learned_addr learned my addr 172.21.15.59:0/1517834979 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:45.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f512bfff640 1 -- 172.21.15.59:0/1517834979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c15dfe0 msgr2=0x7f512c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f512b7fe640 1 --2- 172.21.15.59:0/1517834979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c15dfe0 0x7f512c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f512bfff640 1 --2- 172.21.15.59:0/1517834979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c15dfe0 0x7f512c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f512a7fc640 1 -- 172.21.15.59:0/1517834979 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472732676 0 0) 0x7f512c0b3430 con 0x7f512c0cf830 2024-03-06T07:28:45.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f512bfff640 1 -- 172.21.15.59:0/1517834979 >> v1:172.21.15.59:6789/0 conn(0x7f512c0cf830 legacy=0x7f512c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f512bfff640 1 -- 172.21.15.59:0/1517834979 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c0b3710 con 0x7f512c0d29c0 2024-03-06T07:28:45.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.064+0000 7f512b7fe640 1 --2- 172.21.15.59:0/1517834979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c15dfe0 0x7f512c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:45.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.065+0000 7f512bfff640 1 --2- 172.21.15.59:0/1517834979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f512c0d29c0 0x7f512c0b41f0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f5124009d90 tx=0x7f512402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9bd7aa6e6b560e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.065+0000 7f512a7fc640 1 -- 172.21.15.59:0/1517834979 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124039070 con 0x7f512c0d29c0 2024-03-06T07:28:45.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.065+0000 7f512a7fc640 1 -- 172.21.15.59:0/1517834979 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512402cc50 con 0x7f512c0d29c0 2024-03-06T07:28:45.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.065+0000 7f512a7fc640 1 -- 172.21.15.59:0/1517834979 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124034440 con 0x7f512c0d29c0 2024-03-06T07:28:45.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.066+0000 7f5132116640 1 -- 172.21.15.59:0/1517834979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f512c0d29c0 msgr2=0x7f512c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.066+0000 7f5132116640 1 --2- 172.21.15.59:0/1517834979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f512c0d29c0 0x7f512c0b41f0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f5124009d90 tx=0x7f512402bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.066+0000 7f5132116640 1 -- 172.21.15.59:0/1517834979 shutdown_connections 2024-03-06T07:28:45.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.066+0000 7f5132116640 1 --2- 172.21.15.59:0/1517834979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f512c0d29c0 0x7f512c0b41f0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.066+0000 7f5132116640 1 --2- 172.21.15.59:0/1517834979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c15dfe0 0x7f512c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.066+0000 7f5132116640 1 -- 172.21.15.59:0/1517834979 >> 172.21.15.59:0/1517834979 conn(0x7f512c0d3cd0 msgr2=0x7f512c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.066+0000 7f5132116640 1 -- 172.21.15.59:0/1517834979 shutdown_connections 2024-03-06T07:28:45.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.066+0000 7f5132116640 1 -- 172.21.15.59:0/1517834979 wait complete. 2024-03-06T07:28:45.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.067+0000 7f5132116640 1 Processor -- start 2024-03-06T07:28:45.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.067+0000 7f5132116640 1 -- start start 2024-03-06T07:28:45.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.068+0000 7f5132116640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f512c0cf830 0x7f512c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.068+0000 7f5132116640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c0d29c0 0x7f512c15f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.068+0000 7f5132116640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f512c15dfe0 0x7f512c0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.068+0000 7f5132116640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c0d1df0 con 0x7f512c0cf830 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.068+0000 7f5132116640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512c0d1c70 con 0x7f512c15dfe0 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.068+0000 7f5132116640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512c15e3c0 con 0x7f512c0d29c0 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.068+0000 7f512b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f512c0cf830 0x7f512c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.068+0000 7f512b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f512c0cf830 0x7f512c0ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48172/0 (socket says 172.21.15.59:48172) 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.068+0000 7f512b7fe640 1 -- 172.21.15.59:0/170446913 learned_addr learned my addr 172.21.15.59:0/170446913 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f512affd640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c0d29c0 0x7f512c15f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f512bfff640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f512c15dfe0 0x7f512c0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f512b7fe640 1 -- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c0d29c0 msgr2=0x7f512c15f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f512b7fe640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c0d29c0 0x7f512c15f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f512b7fe640 1 -- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f512c15dfe0 msgr2=0x7f512c0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f512b7fe640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f512c15dfe0 0x7f512c0cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f512b7fe640 1 -- 172.21.15.59:0/170446913 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c15faa0 con 0x7f512c0cf830 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f512affd640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c0d29c0 0x7f512c15f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f512bfff640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f512c15dfe0 0x7f512c0cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f512b7fe640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f512c0cf830 0x7f512c0ccf30 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f511c00c7f0 tx=0x7f511c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f5113fff640 1 -- 172.21.15.59:0/170446913 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c015070 con 0x7f512c0cf830 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f5113fff640 1 -- 172.21.15.59:0/170446913 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511c007820 con 0x7f512c0cf830 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f5132116640 1 -- 172.21.15.59:0/170446913 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f512c15fd80 con 0x7f512c0cf830 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f5113fff640 1 -- 172.21.15.59:0/170446913 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c007bd0 con 0x7f512c0cf830 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.069+0000 7f5132116640 1 -- 172.21.15.59:0/170446913 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512c0caf20 con 0x7f512c0cf830 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.070+0000 7f5132116640 1 -- 172.21.15.59:0/170446913 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f512c0b3360 con 0x7f512c0cf830 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.071+0000 7f5113fff640 1 -- 172.21.15.59:0/170446913 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f511c018070 con 0x7f512c0cf830 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.071+0000 7f5113fff640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f511806b4d0 0x7f511806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.071+0000 7f5113fff640 1 -- 172.21.15.59:0/170446913 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f511c08a370 con 0x7f512c0cf830 2024-03-06T07:28:45.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.071+0000 7f512affd640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f511806b4d0 0x7f511806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.072+0000 7f512affd640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f511806b4d0 0x7f511806d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f512c0ca540 tx=0x7f5120009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.075+0000 7f5113fff640 1 -- 172.21.15.59:0/170446913 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f512c0b3360 con 0x7f512c0cf830 2024-03-06T07:28:45.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.294+0000 7f5132116640 1 -- 172.21.15.59:0/170446913 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f512c04f330 con 0x7f512c0cf830 2024-03-06T07:28:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.295+0000 7f5113fff640 1 -- 172.21.15.59:0/170446913 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f511c059e20 con 0x7f512c0cf830 2024-03-06T07:28:45.297 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:45.297 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1772616514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1772616514},{"type":"v1","addr":"172.21.15.164:6836","nonce":1772616514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":7816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3891024511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3891024511},{"type":"v1","addr":"172.21.15.164:6837","nonce":3891024511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":8673,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:28:44.364208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:28:45.297 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 79 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 -- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f511806b4d0 msgr2=0x7f511806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f511806b4d0 0x7f511806d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f512c0ca540 tx=0x7f5120009040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 -- 172.21.15.59:0/170446913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f512c0cf830 msgr2=0x7f512c0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f512c0cf830 0x7f512c0ccf30 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f511c00c7f0 tx=0x7f511c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 -- 172.21.15.59:0/170446913 shutdown_connections 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f511806b4d0 0x7f511806d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f512c15dfe0 0x7f512c0cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f512c0d29c0 0x7f512c15f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 --2- 172.21.15.59:0/170446913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f512c0cf830 0x7f512c0ccf30 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 -- 172.21.15.59:0/170446913 >> 172.21.15.59:0/170446913 conn(0x7f512c0d3cd0 msgr2=0x7f512c0b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 -- 172.21.15.59:0/170446913 shutdown_connections 2024-03-06T07:28:45.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.297+0000 7f5132116640 1 -- 172.21.15.59:0/170446913 wait complete. 2024-03-06T07:28:45.310 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:45.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.410+0000 7f1f0f53a640 1 Processor -- start 2024-03-06T07:28:45.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.410+0000 7f1f0f53a640 1 -- start start 2024-03-06T07:28:45.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.411+0000 7f1f0f53a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080b4170 0x7f1f080b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.411+0000 7f1f0f53a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1f080d0a80 0x7f1f080cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.411+0000 7f1f0f53a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f080d25d0 con 0x7f1f080d4760 2024-03-06T07:28:45.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.411+0000 7f1f0f53a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f080d4bf0 con 0x7f1f080d0a80 2024-03-06T07:28:45.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.411+0000 7f1f0f53a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080ce9e0 con 0x7f1f080b4170 2024-03-06T07:28:45.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.411+0000 7f1f0cad5640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1f080d4760 0x7f1f080d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44446/0 (socket says 172.21.15.59:44446) 2024-03-06T07:28:45.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.411+0000 7f1f0cad5640 1 -- 172.21.15.59:0/2942189001 learned_addr learned my addr 172.21.15.59:0/2942189001 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:45.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.411+0000 7f1f0dad7640 1 --2- 172.21.15.59:0/2942189001 >> v2:172.21.15.164:3300/0 conn(0x7f1f080d0a80 0x7f1f080cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.411+0000 7f1f0d2d6640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080b4170 0x7f1f080b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.411+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642116478 0 0) 0x7f1f080d25d0 con 0x7f1f080d4760 2024-03-06T07:28:45.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.411+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef4003520 con 0x7f1f080d4760 2024-03-06T07:28:45.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3651129 0 0) 0x7f1ef4003520 con 0x7f1f080d4760 2024-03-06T07:28:45.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080b4170 msgr2=0x7f1f080b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080b4170 0x7f1f080b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 >> v2:172.21.15.164:3300/0 conn(0x7f1f080d0a80 msgr2=0x7f1f080cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 --2- 172.21.15.59:0/2942189001 >> v2:172.21.15.164:3300/0 conn(0x7f1f080d0a80 0x7f1f080cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f080d2870 con 0x7f1f080d4760 2024-03-06T07:28:45.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f1ef8001670 con 0x7f1f080d4760 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 >> v1:172.21.15.59:6789/0 conn(0x7f1f080d4760 legacy=0x7f1f080d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080b4170 0x7f1ef40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ef4003f30 0x7f1ef4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ef4004d40 0x7f1ef40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080d4bf0 con 0x7f1ef4003f30 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080ce9e0 con 0x7f1ef4004d40 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef400b9c0 con 0x7f1f080b4170 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1f0dad7640 1 --2- 172.21.15.59:0/2942189001 >> v2:172.21.15.164:3300/0 conn(0x7f1f080d0a80 0x7f1f080cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1f0cad5640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ef4003f30 0x7f1ef4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1f0d2d6640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080b4170 0x7f1ef40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1f0dad7640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ef4004d40 0x7f1ef40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1f0cad5640 1 -- 172.21.15.59:0/2942189001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080b4170 msgr2=0x7f1ef40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1f0cad5640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080b4170 0x7f1ef40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1f0cad5640 1 -- 172.21.15.59:0/2942189001 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ef4004d40 msgr2=0x7f1ef40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1f0cad5640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ef4004d40 0x7f1ef40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.412+0000 7f1f0cad5640 1 -- 172.21.15.59:0/2942189001 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef4003520 con 0x7f1ef4003f30 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.413+0000 7f1f0cad5640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ef4003f30 0x7f1ef4004800 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800c210 tx=0x7f1ef4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26039cbcd6406e47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.413+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8007d50 con 0x7f1ef4003f30 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.413+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef8030070 con 0x7f1ef4003f30 2024-03-06T07:28:45.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.413+0000 7f1efffff640 1 -- 172.21.15.59:0/2942189001 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef802cdd0 con 0x7f1ef4003f30 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.414+0000 7f1f0f53a640 1 -- 172.21.15.59:0/2942189001 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ef4003f30 msgr2=0x7f1ef4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.414+0000 7f1f0f53a640 1 --2- 172.21.15.59:0/2942189001 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ef4003f30 0x7f1ef4004800 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800c210 tx=0x7f1ef4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.414+0000 7f1f0dad7640 1 -- 172.21.15.59:0/2942189001 reap_dead start 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.414+0000 7f1f0f53a640 1 -- 172.21.15.59:0/2942189001 shutdown_connections 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.414+0000 7f1f0f53a640 1 -- 172.21.15.59:0/2942189001 >> 172.21.15.59:0/2942189001 conn(0x7f1f080d3c40 msgr2=0x7f1f080d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.414+0000 7f1f0f53a640 1 -- 172.21.15.59:0/2942189001 shutdown_connections 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.414+0000 7f1f0f53a640 1 -- 172.21.15.59:0/2942189001 wait complete. 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.416+0000 7f1f0f53a640 1 Processor -- start 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.416+0000 7f1f0f53a640 1 -- start start 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0f53a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ef4003f30 0x7f1f081f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0f53a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080ccc30 0x7f1f080cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0f53a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f08158200 0x7f1f081586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0f53a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080b4dd0 con 0x7f1f08158200 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0f53a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080b4550 con 0x7f1ef4003f30 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0f53a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f080b4f50 con 0x7f1f080ccc30 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0dad7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f08158200 0x7f1f081586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0dad7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f08158200 0x7f1f081586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48192/0 (socket says 172.21.15.59:48192) 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0dad7640 1 -- 172.21.15.59:0/1701820613 learned_addr learned my addr 172.21.15.59:0/1701820613 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0d2d6640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ef4003f30 0x7f1f081f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0cad5640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080ccc30 0x7f1f080cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0dad7640 1 -- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080ccc30 msgr2=0x7f1f080cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0dad7640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080ccc30 0x7f1f080cd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0dad7640 1 -- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ef4003f30 msgr2=0x7f1f081f6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0dad7640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ef4003f30 0x7f1f081f6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0dad7640 1 -- 172.21.15.59:0/1701820613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f080c9bb0 con 0x7f1f08158200 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.417+0000 7f1f0dad7640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f08158200 0x7f1f081586b0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1f0808df00 tx=0x7f1f04009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.418+0000 7f1efdffb640 1 -- 172.21.15.59:0/1701820613 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f04011e00 con 0x7f1f08158200 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.418+0000 7f1efdffb640 1 -- 172.21.15.59:0/1701820613 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0401b070 con 0x7f1f08158200 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.418+0000 7f1f0f53a640 1 -- 172.21.15.59:0/1701820613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f080cd7f0 con 0x7f1f08158200 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.418+0000 7f1efdffb640 1 -- 172.21.15.59:0/1701820613 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f04003990 con 0x7f1f08158200 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.418+0000 7f1f0f53a640 1 -- 172.21.15.59:0/1701820613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f080b7360 con 0x7f1f08158200 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.419+0000 7f1f0f53a640 1 -- 172.21.15.59:0/1701820613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f0804f330 con 0x7f1f08158200 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.419+0000 7f1efdffb640 1 -- 172.21.15.59:0/1701820613 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f04005d10 con 0x7f1f08158200 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.420+0000 7f1efdffb640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b500 0x7f1eec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.420+0000 7f1efdffb640 1 -- 172.21.15.59:0/1701820613 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f0408a8f0 con 0x7f1f08158200 2024-03-06T07:28:45.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.420+0000 7f1f0d2d6640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b500 0x7f1eec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.420+0000 7f1f0d2d6640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b500 0x7f1eec06d9c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f1f080cdea0 tx=0x7f1f00008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.424+0000 7f1efdffb640 1 -- 172.21.15.59:0/1701820613 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f04025020 con 0x7f1f08158200 2024-03-06T07:28:45.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.610+0000 7f1f0f53a640 1 -- 172.21.15.59:0/1701820613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f080733b0 con 0x7f1f08158200 2024-03-06T07:28:45.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.611+0000 7f1efdffb640 1 -- 172.21.15.59:0/1701820613 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16667 (secure 0 0 0) 0x7f1f040597e0 con 0x7f1f08158200 2024-03-06T07:28:45.613 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:45.613 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":126,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:28:43.933880+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:40.558382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:41.255127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:45.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 -- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b500 msgr2=0x7f1eec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b500 0x7f1eec06d9c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f1f080cdea0 tx=0x7f1f00008040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 -- 172.21.15.59:0/1701820613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f08158200 msgr2=0x7f1f081586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f08158200 0x7f1f081586b0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1f0808df00 tx=0x7f1f04009f90 comp rx=0 tx=0).stop 2024-03-06T07:28:45.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 -- 172.21.15.59:0/1701820613 shutdown_connections 2024-03-06T07:28:45.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1eec06b500 0x7f1eec06d9c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f08158200 0x7f1f081586b0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f080ccc30 0x7f1f080cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 --2- 172.21.15.59:0/1701820613 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ef4003f30 0x7f1f081f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 -- 172.21.15.59:0/1701820613 >> 172.21.15.59:0/1701820613 conn(0x7f1f080d3c40 msgr2=0x7f1f080ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 -- 172.21.15.59:0/1701820613 shutdown_connections 2024-03-06T07:28:45.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.613+0000 7f1f0f53a640 1 -- 172.21.15.59:0/1701820613 wait complete. 2024-03-06T07:28:45.626 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:28:45.627 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.727+0000 7f67a3d7a640 1 Processor -- start 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.727+0000 7f67a3d7a640 1 -- start start 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.728+0000 7f67a3d7a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f679c15dfe0 0x7f679c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.728+0000 7f67a3d7a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f679c0d29c0 0x7f679c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.728+0000 7f67a3d7a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c0b3430 con 0x7f679c0cf830 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.728+0000 7f67a3d7a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679c0b31e0 con 0x7f679c0d29c0 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.728+0000 7f67a3d7a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679c0b3d60 con 0x7f679c15dfe0 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.729+0000 7f67a2317640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f679c0d29c0 0x7f679c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.729+0000 7f67a2317640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f679c0d29c0 0x7f679c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48202/0 (socket says 172.21.15.59:48202) 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.729+0000 7f67a2317640 1 -- 172.21.15.59:0/4269608250 learned_addr learned my addr 172.21.15.59:0/4269608250 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.729+0000 7f67a1b16640 1 --2- 172.21.15.59:0/4269608250 >> v2:172.21.15.164:3300/0 conn(0x7f679c15dfe0 0x7f679c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.729+0000 7f67a2317640 1 -- 172.21.15.59:0/4269608250 >> v2:172.21.15.164:3300/0 conn(0x7f679c15dfe0 msgr2=0x7f679c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.729+0000 7f67a2317640 1 --2- 172.21.15.59:0/4269608250 >> v2:172.21.15.164:3300/0 conn(0x7f679c15dfe0 0x7f679c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.729+0000 7f67a2317640 1 -- 172.21.15.59:0/4269608250 >> v1:172.21.15.164:6789/0 conn(0x7f679c0cf830 legacy=0x7f679c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.729+0000 7f67a2317640 1 -- 172.21.15.59:0/4269608250 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c0b3710 con 0x7f679c0d29c0 2024-03-06T07:28:45.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.729+0000 7f67a1b16640 1 --2- 172.21.15.59:0/4269608250 >> v2:172.21.15.164:3300/0 conn(0x7f679c15dfe0 0x7f679c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:45.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a2317640 1 --2- 172.21.15.59:0/4269608250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f679c0d29c0 0x7f679c0b41f0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f679c067990 tx=0x7f679402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ee7cdb3ed4c52d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a0b14640 1 -- 172.21.15.59:0/4269608250 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679402ca80 con 0x7f679c0d29c0 2024-03-06T07:28:45.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a0b14640 1 -- 172.21.15.59:0/4269608250 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679402cc20 con 0x7f679c0d29c0 2024-03-06T07:28:45.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a0b14640 1 -- 172.21.15.59:0/4269608250 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67940343f0 con 0x7f679c0d29c0 2024-03-06T07:28:45.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a3d7a640 1 -- 172.21.15.59:0/4269608250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f679c0d29c0 msgr2=0x7f679c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a3d7a640 1 --2- 172.21.15.59:0/4269608250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f679c0d29c0 0x7f679c0b41f0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f679c067990 tx=0x7f679402bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a3d7a640 1 -- 172.21.15.59:0/4269608250 shutdown_connections 2024-03-06T07:28:45.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a3d7a640 1 --2- 172.21.15.59:0/4269608250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f679c0d29c0 0x7f679c0b41f0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a3d7a640 1 --2- 172.21.15.59:0/4269608250 >> v2:172.21.15.164:3300/0 conn(0x7f679c15dfe0 0x7f679c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a3d7a640 1 -- 172.21.15.59:0/4269608250 >> 172.21.15.59:0/4269608250 conn(0x7f679c0d3cd0 msgr2=0x7f679c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a3d7a640 1 -- 172.21.15.59:0/4269608250 shutdown_connections 2024-03-06T07:28:45.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.730+0000 7f67a3d7a640 1 -- 172.21.15.59:0/4269608250 wait complete. 2024-03-06T07:28:45.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.732+0000 7f67a3d7a640 1 Processor -- start 2024-03-06T07:28:45.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.732+0000 7f67a3d7a640 1 -- start start 2024-03-06T07:28:45.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.733+0000 7f67a3d7a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f679c0cf830 0x7f679c15fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.733+0000 7f67a3d7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f679c0d29c0 0x7f679c160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.733+0000 7f67a3d7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f679c15dfe0 0x7f679c14fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.733+0000 7f67a3d7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c0d1df0 con 0x7f679c0d29c0 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.733+0000 7f67a3d7a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c0d1c70 con 0x7f679c0cf830 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.733+0000 7f67a3d7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679c15e3c0 con 0x7f679c15dfe0 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.733+0000 7f67a1315640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f679c0d29c0 0x7f679c160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.733+0000 7f67a1315640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f679c0d29c0 0x7f679c160d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48216/0 (socket says 172.21.15.59:48216) 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.733+0000 7f67a1315640 1 -- 172.21.15.59:0/2843841190 learned_addr learned my addr 172.21.15.59:0/2843841190 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67a2317640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f679c15dfe0 0x7f679c14fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67a1b16640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f679c0cf830 0x7f679c15fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67a1315640 1 -- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f679c15dfe0 msgr2=0x7f679c14fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67a1315640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f679c15dfe0 0x7f679c14fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67a1315640 1 -- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f679c0cf830 msgr2=0x7f679c15fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67a1315640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f679c0cf830 0x7f679c15fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67a1315640 1 -- 172.21.15.59:0/2843841190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c15c9e0 con 0x7f679c0d29c0 2024-03-06T07:28:45.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67a1315640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f679c0d29c0 0x7f679c160d60 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f678c00c430 tx=0x7f678c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67927fc640 1 -- 172.21.15.59:0/2843841190 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c011790 con 0x7f679c0d29c0 2024-03-06T07:28:45.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67927fc640 1 -- 172.21.15.59:0/2843841190 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678c017070 con 0x7f679c0d29c0 2024-03-06T07:28:45.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67a3d7a640 1 -- 172.21.15.59:0/2843841190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c15e750 con 0x7f679c0d29c0 2024-03-06T07:28:45.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67927fc640 1 -- 172.21.15.59:0/2843841190 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c011b70 con 0x7f679c0d29c0 2024-03-06T07:28:45.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.734+0000 7f67a3d7a640 1 -- 172.21.15.59:0/2843841190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c15ed10 con 0x7f679c0d29c0 2024-03-06T07:28:45.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.735+0000 7f67a3d7a640 1 -- 172.21.15.59:0/2843841190 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c04f330 con 0x7f679c0d29c0 2024-03-06T07:28:45.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.735+0000 7f67927fc640 1 -- 172.21.15.59:0/2843841190 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f678c011d10 con 0x7f679c0d29c0 2024-03-06T07:28:45.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.736+0000 7f67927fc640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678006b4d0 0x7f678006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.736+0000 7f67927fc640 1 -- 172.21.15.59:0/2843841190 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f678c08a0c0 con 0x7f679c0d29c0 2024-03-06T07:28:45.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.736+0000 7f67a1b16640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678006b4d0 0x7f678006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.737+0000 7f67a1b16640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678006b4d0 0x7f678006d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f679c161820 tx=0x7f6798008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.740+0000 7f67927fc640 1 -- 172.21.15.59:0/2843841190 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f678c058f40 con 0x7f679c0d29c0 2024-03-06T07:28:45.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.918+0000 7f67a3d7a640 1 -- 172.21.15.59:0/2843841190 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f679c06c580 con 0x7f678006b4d0 2024-03-06T07:28:45.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.924+0000 7f67927fc640 1 -- 172.21.15.59:0/2843841190 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240269 (secure 0 0 0) 0x7f679c06c580 con 0x7f678006b4d0 2024-03-06T07:28:45.929 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:45.930 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:28:45.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 -- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678006b4d0 msgr2=0x7f678006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678006b4d0 0x7f678006d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f679c161820 tx=0x7f6798008040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 -- 172.21.15.59:0/2843841190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f679c0d29c0 msgr2=0x7f679c160d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f679c0d29c0 0x7f679c160d60 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f678c00c430 tx=0x7f678c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 -- 172.21.15.59:0/2843841190 shutdown_connections 2024-03-06T07:28:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678006b4d0 0x7f678006d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f679c15dfe0 0x7f679c14fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f679c0d29c0 0x7f679c160d60 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 --2- 172.21.15.59:0/2843841190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f679c0cf830 0x7f679c15fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 -- 172.21.15.59:0/2843841190 >> 172.21.15.59:0/2843841190 conn(0x7f679c0d3cd0 msgr2=0x7f679c0b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 -- 172.21.15.59:0/2843841190 shutdown_connections 2024-03-06T07:28:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:45.927+0000 7f67a3d7a640 1 -- 172.21.15.59:0/2843841190 wait complete. 2024-03-06T07:28:45.943 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-03-06T07:28:45.392211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329140,"kb_used_data":13112,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645580,"statfs":{"total":773094113280,"available":772757073920,"internally_reserved":0,"allocated":13426688,"data_stored":6491164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.471326"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924544+0000","last_change":"2024-03-06T07:28:40.924544+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924544+0000","last_undegraded":"2024-03-06T07:28:40.924544+0000","last_fullsized":"2024-03-06T07:28:40.924544+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922334+0000","last_change":"2024-03-06T07:28:40.922334+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922334+0000","last_undegraded":"2024-03-06T07:28:40.922334+0000","last_fullsized":"2024-03-06T07:28:40.922334+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927923+0000","last_change":"2024-03-06T07:28:40.927923+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927923+0000","last_undegraded":"2024-03-06T07:28:40.927923+0000","last_fullsized":"2024-03-06T07:28:40.927923+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.929980+0000","last_change":"2024-03-06T07:28:40.929980+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.929980+0000","last_undegraded":"2024-03-06T07:28:40.929980+0000","last_fullsized":"2024-03-06T07:28:40.929980+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.929949+0000","last_change":"2024-03-06T07:28:40.929949+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.929949+0000","last_undegraded":"2024-03-06T07:28:40.929949+0000","last_fullsized":"2024-03-06T07:28:40.929949+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924277+0000","last_change":"2024-03-06T07:28:40.924277+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924277+0000","last_undegraded":"2024-03-06T07:28:40.924277+0000","last_fullsized":"2024-03-06T07:28:40.924277+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922654+0000","last_change":"2024-03-06T07:28:40.922654+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922654+0000","last_undegraded":"2024-03-06T07:28:40.922654+0000","last_fullsized":"2024-03-06T07:28:40.922654+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927409+0000","last_change":"2024-03-06T07:28:40.927409+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927409+0000","last_undegraded":"2024-03-06T07:28:40.927409+0000","last_fullsized":"2024-03-06T07:28:40.927409+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922559+0000","last_change":"2024-03-06T07:28:40.922559+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922559+0000","last_undegraded":"2024-03-06T07:28:40.922559+0000","last_fullsized":"2024-03-06T07:28:40.922559+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923890+0000","last_change":"2024-03-06T07:28:40.923890+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923890+0000","last_undegraded":"2024-03-06T07:28:40.923890+0000","last_fullsized":"2024-03-06T07:28:40.923890+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.921998+0000","last_change":"2024-03-06T07:28:40.921998+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.921998+0000","last_undegraded":"2024-03-06T07:28:40.921998+0000","last_fullsized":"2024-03-06T07:28:40.921998+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923046+0000","last_change":"2024-03-06T07:28:40.923046+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923046+0000","last_undegraded":"2024-03-06T07:28:40.923046+0000","last_fullsized":"2024-03-06T07:28:40.923046+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.919747+0000","last_change":"2024-03-06T07:28:40.919747+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.919747+0000","last_undegraded":"2024-03-06T07:28:40.919747+0000","last_fullsized":"2024-03-06T07:28:40.919747+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927342+0000","last_change":"2024-03-06T07:28:40.927342+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927342+0000","last_undegraded":"2024-03-06T07:28:40.927342+0000","last_fullsized":"2024-03-06T07:28:40.927342+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922365+0000","last_change":"2024-03-06T07:28:40.922365+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922365+0000","last_undegraded":"2024-03-06T07:28:40.922365+0000","last_fullsized":"2024-03-06T07:28:40.922365+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922081+0000","last_change":"2024-03-06T07:28:40.922081+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922081+0000","last_undegraded":"2024-03-06T07:28:40.922081+0000","last_fullsized":"2024-03-06T07:28:40.922081+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927877+0000","last_change":"2024-03-06T07:28:40.927877+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927877+0000","last_undegraded":"2024-03-06T07:28:40.927877+0000","last_fullsized":"2024-03-06T07:28:40.927877+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922113+0000","last_change":"2024-03-06T07:28:40.922113+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922113+0000","last_undegraded":"2024-03-06T07:28:40.922113+0000","last_fullsized":"2024-03-06T07:28:40.922113+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922430+0000","last_change":"2024-03-06T07:28:40.922430+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922430+0000","last_undegraded":"2024-03-06T07:28:40.922430+0000","last_fullsized":"2024-03-06T07:28:40.922430+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922580+0000","last_change":"2024-03-06T07:28:40.922580+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922580+0000","last_undegraded":"2024-03-06T07:28:40.922580+0000","last_fullsized":"2024-03-06T07:28:40.922580+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923848+0000","last_change":"2024-03-06T07:28:40.923848+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923848+0000","last_undegraded":"2024-03-06T07:28:40.923848+0000","last_fullsized":"2024-03-06T07:28:40.923848+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924394+0000","last_change":"2024-03-06T07:28:40.924394+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924394+0000","last_undegraded":"2024-03-06T07:28:40.924394+0000","last_fullsized":"2024-03-06T07:28:40.924394+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.928144+0000","last_change":"2024-03-06T07:28:40.928144+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.928144+0000","last_undegraded":"2024-03-06T07:28:40.928144+0000","last_fullsized":"2024-03-06T07:28:40.928144+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.919708+0000","last_change":"2024-03-06T07:28:40.919708+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.919708+0000","last_undegraded":"2024-03-06T07:28:40.919708+0000","last_fullsized":"2024-03-06T07:28:40.919708+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924586+0000","last_change":"2024-03-06T07:28:40.924586+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924586+0000","last_undegraded":"2024-03-06T07:28:40.924586+0000","last_fullsized":"2024-03-06T07:28:40.924586+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.930002+0000","last_change":"2024-03-06T07:28:40.930002+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.930002+0000","last_undegraded":"2024-03-06T07:28:40.930002+0000","last_fullsized":"2024-03-06T07:28:40.930002+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927300+0000","last_change":"2024-03-06T07:28:40.927300+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927300+0000","last_undegraded":"2024-03-06T07:28:40.927300+0000","last_fullsized":"2024-03-06T07:28:40.927300+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927340+0000","last_change":"2024-03-06T07:28:40.927340+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927340+0000","last_undegraded":"2024-03-06T07:28:40.927340+0000","last_fullsized":"2024-03-06T07:28:40.927340+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.925257+0000","last_change":"2024-03-06T07:28:40.925257+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.925257+0000","last_undegraded":"2024-03-06T07:28:40.925257+0000","last_fullsized":"2024-03-06T07:28:40.925257+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923956+0000","last_change":"2024-03-06T07:28:40.923956+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923956+0000","last_undegraded":"2024-03-06T07:28:40.923956+0000","last_fullsized":"2024-03-06T07:28:40.923956+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927584+0000","last_change":"2024-03-06T07:28:40.927584+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927584+0000","last_undegraded":"2024-03-06T07:28:40.927584+0000","last_fullsized":"2024-03-06T07:28:40.927584+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.925204+0000","last_change":"2024-03-06T07:28:40.925204+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.925204+0000","last_undegraded":"2024-03-06T07:28:40.925204+0000","last_fullsized":"2024-03-06T07:28:40.925204+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":353,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.925605+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:28:40.925605+0000","last_peered":"2024-03-06T07:28:40.925605+0000","last_clean":"2024-03-06T07:28:40.925605+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:28:40.925605+0000","last_undegraded":"2024-03-06T07:28:40.925605+0000","last_fullsized":"2024-03-06T07:28:40.925605+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41572,"kb_used_data":2068,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330268,"statfs":{"total":96636764160,"available":96594194432,"internally_reserved":0,"allocated":2117632,"data_stored":1251263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40996,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330844,"statfs":{"total":96636764160,"available":96594784256,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40992,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330848,"statfs":{"total":96636764160,"available":96594788352,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41580,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330260,"statfs":{"total":96636764160,"available":96594186240,"internally_reserved":0,"allocated":2129920,"data_stored":1257080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:45.948 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:46.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.083+0000 7f312b8c9640 1 Processor -- start 2024-03-06T07:28:46.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.083+0000 7f312b8c9640 1 -- start start 2024-03-06T07:28:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.084+0000 7f312b8c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31240d04a0 0x7f31240ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.084+0000 7f312b8c9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31240d0910 con 0x7f31240cfff0 2024-03-06T07:28:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.084+0000 7f312b8c9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3124161980 con 0x7f312415bb20 2024-03-06T07:28:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.084+0000 7f312b8c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31240d2500 con 0x7f31240d04a0 2024-03-06T07:28:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.084+0000 7f3128e64640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f31240cfff0 0x7f31240cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44454/0 (socket says 172.21.15.59:44454) 2024-03-06T07:28:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.084+0000 7f3128e64640 1 -- 172.21.15.59:0/3325375060 learned_addr learned my addr 172.21.15.59:0/3325375060 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.084+0000 7f3129e66640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31240d04a0 0x7f31240ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.084+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233698342 0 0) 0x7f31240d0910 con 0x7f31240cfff0 2024-03-06T07:28:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3110003520 con 0x7f31240cfff0 2024-03-06T07:28:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794559279 0 0) 0x7f3124161980 con 0x7f312415bb20 2024-03-06T07:28:46.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31240d0910 con 0x7f312415bb20 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2050651252 0 0) 0x7f3110003520 con 0x7f31240cfff0 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31240d04a0 msgr2=0x7f31240ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31240d04a0 0x7f31240ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 >> v1:172.21.15.164:6789/0 conn(0x7f312415bb20 legacy=0x7f31240d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3124161c60 con 0x7f31240cfff0 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f3114001670 con 0x7f31240cfff0 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 >> v1:172.21.15.59:6789/0 conn(0x7f31240cfff0 legacy=0x7f31240cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3110003b70 0x7f3110003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 0x7f3110004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31240d04a0 0x7f31100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31240d2500 con 0x7f3110004800 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311000ba70 con 0x7f3110003b70 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311000bc40 con 0x7f31240d04a0 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f3128e64640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 0x7f3110004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f3129665640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3110003b70 0x7f3110003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f3129e66640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31240d04a0 0x7f31100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f3128e64640 1 -- 172.21.15.59:0/3325375060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31240d04a0 msgr2=0x7f31100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f3128e64640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31240d04a0 0x7f31100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f3128e64640 1 -- 172.21.15.59:0/3325375060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3110003b70 msgr2=0x7f3110003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f3128e64640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3110003b70 0x7f3110003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.085+0000 7f3128e64640 1 -- 172.21.15.59:0/3325375060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3110003520 con 0x7f3110004800 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.086+0000 7f3128e64640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 0x7f3110004c80 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f311400c190 tx=0x7f3110005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad330a472f0ec47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.086+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114007ce0 con 0x7f3110004800 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.086+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3114030070 con 0x7f3110004800 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.086+0000 7f311bfff640 1 -- 172.21.15.59:0/3325375060 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114005d50 con 0x7f3110004800 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.087+0000 7f312b8c9640 1 -- 172.21.15.59:0/3325375060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 msgr2=0x7f3110004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.087+0000 7f312b8c9640 1 --2- 172.21.15.59:0/3325375060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 0x7f3110004c80 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f311400c190 tx=0x7f3110005830 comp rx=0 tx=0).stop 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.087+0000 7f3129e66640 1 -- 172.21.15.59:0/3325375060 reap_dead start 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.087+0000 7f312b8c9640 1 -- 172.21.15.59:0/3325375060 shutdown_connections 2024-03-06T07:28:46.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.087+0000 7f312b8c9640 1 -- 172.21.15.59:0/3325375060 >> 172.21.15.59:0/3325375060 conn(0x7f312409a9f0 msgr2=0x7f312409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.087+0000 7f312b8c9640 1 -- 172.21.15.59:0/3325375060 shutdown_connections 2024-03-06T07:28:46.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.087+0000 7f312b8c9640 1 -- 172.21.15.59:0/3325375060 wait complete. 2024-03-06T07:28:46.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.088+0000 7f312b8c9640 1 Processor -- start 2024-03-06T07:28:46.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.088+0000 7f312b8c9640 1 -- start start 2024-03-06T07:28:46.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f312b8c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 0x7f3124154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f312b8c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31241f4b70 0x7f31241f7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f312b8c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31241f7ab0 0x7f312406fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f312b8c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124162500 con 0x7f3110004800 2024-03-06T07:28:46.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f312b8c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124162220 con 0x7f31241f7ab0 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f312b8c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31240d1980 con 0x7f31241f4b70 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3129665640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 0x7f3124154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3129665640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 0x7f3124154bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48230/0 (socket says 172.21.15.59:48230) 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3129665640 1 -- 172.21.15.59:0/2928763945 learned_addr learned my addr 172.21.15.59:0/2928763945 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3129e66640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31241f7ab0 0x7f312406fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3129665640 1 -- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31241f4b70 msgr2=0x7f31241f7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3128e64640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31241f4b70 0x7f31241f7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3129665640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31241f4b70 0x7f31241f7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3129665640 1 -- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31241f7ab0 msgr2=0x7f312406fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3129665640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31241f7ab0 0x7f312406fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3129665640 1 -- 172.21.15.59:0/2928763945 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3124155110 con 0x7f3110004800 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3129e66640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31241f7ab0 0x7f312406fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3128e64640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31241f4b70 0x7f31241f7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.089+0000 7f3129665640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 0x7f3124154bd0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f312000f7d0 tx=0x7f312000fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.090+0000 7f3119ffb640 1 -- 172.21.15.59:0/2928763945 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312001a070 con 0x7f3110004800 2024-03-06T07:28:46.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.090+0000 7f3119ffb640 1 -- 172.21.15.59:0/2928763945 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312000fe80 con 0x7f3110004800 2024-03-06T07:28:46.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.090+0000 7f3119ffb640 1 -- 172.21.15.59:0/2928763945 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120007050 con 0x7f3110004800 2024-03-06T07:28:46.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.090+0000 7f312b8c9640 1 -- 172.21.15.59:0/2928763945 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31241f8070 con 0x7f3110004800 2024-03-06T07:28:46.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.090+0000 7f312b8c9640 1 -- 172.21.15.59:0/2928763945 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31241f4370 con 0x7f3110004800 2024-03-06T07:28:46.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.091+0000 7f312b8c9640 1 -- 172.21.15.59:0/2928763945 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312404f330 con 0x7f3110004800 2024-03-06T07:28:46.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.091+0000 7f3119ffb640 1 -- 172.21.15.59:0/2928763945 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f312001d070 con 0x7f3110004800 2024-03-06T07:28:46.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.091+0000 7f3119ffb640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310806b4d0 0x7f310806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.092+0000 7f3119ffb640 1 -- 172.21.15.59:0/2928763945 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3120089930 con 0x7f3110004800 2024-03-06T07:28:46.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.092+0000 7f3128e64640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310806b4d0 0x7f310806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.092+0000 7f3128e64640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310806b4d0 0x7f310806d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3114034000 tx=0x7f3114007000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.096+0000 7f3119ffb640 1 -- 172.21.15.59:0/2928763945 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3120058840 con 0x7f3110004800 2024-03-06T07:28:46.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.272+0000 7f312b8c9640 1 -- 172.21.15.59:0/2928763945 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f31240b1820 con 0x7f310806b4d0 2024-03-06T07:28:46.279 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.278+0000 7f3119ffb640 1 -- 172.21.15.59:0/2928763945 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240269 (secure 0 0 0) 0x7f31240b1820 con 0x7f310806b4d0 2024-03-06T07:28:46.282 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:46.283 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:28:46.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 -- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310806b4d0 msgr2=0x7f310806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310806b4d0 0x7f310806d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3114034000 tx=0x7f3114007000 comp rx=0 tx=0).stop 2024-03-06T07:28:46.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 -- 172.21.15.59:0/2928763945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 msgr2=0x7f3124154bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 0x7f3124154bd0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f312000f7d0 tx=0x7f312000fc90 comp rx=0 tx=0).stop 2024-03-06T07:28:46.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 -- 172.21.15.59:0/2928763945 shutdown_connections 2024-03-06T07:28:46.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f310806b4d0 0x7f310806d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31241f7ab0 0x7f312406fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31241f4b70 0x7f31241f7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 --2- 172.21.15.59:0/2928763945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3110004800 0x7f3124154bd0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 -- 172.21.15.59:0/2928763945 >> 172.21.15.59:0/2928763945 conn(0x7f312409a9f0 msgr2=0x7f31240cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 -- 172.21.15.59:0/2928763945 shutdown_connections 2024-03-06T07:28:46.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.281+0000 7f312b8c9640 1 -- 172.21.15.59:0/2928763945 wait complete. 2024-03-06T07:28:46.294 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-03-06T07:28:45.392211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329140,"kb_used_data":13112,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645580,"statfs":{"total":773094113280,"available":772757073920,"internally_reserved":0,"allocated":13426688,"data_stored":6491164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.471326"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924544+0000","last_change":"2024-03-06T07:28:40.924544+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924544+0000","last_undegraded":"2024-03-06T07:28:40.924544+0000","last_fullsized":"2024-03-06T07:28:40.924544+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922334+0000","last_change":"2024-03-06T07:28:40.922334+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922334+0000","last_undegraded":"2024-03-06T07:28:40.922334+0000","last_fullsized":"2024-03-06T07:28:40.922334+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927923+0000","last_change":"2024-03-06T07:28:40.927923+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927923+0000","last_undegraded":"2024-03-06T07:28:40.927923+0000","last_fullsized":"2024-03-06T07:28:40.927923+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.929980+0000","last_change":"2024-03-06T07:28:40.929980+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.929980+0000","last_undegraded":"2024-03-06T07:28:40.929980+0000","last_fullsized":"2024-03-06T07:28:40.929980+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.929949+0000","last_change":"2024-03-06T07:28:40.929949+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.929949+0000","last_undegraded":"2024-03-06T07:28:40.929949+0000","last_fullsized":"2024-03-06T07:28:40.929949+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924277+0000","last_change":"2024-03-06T07:28:40.924277+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924277+0000","last_undegraded":"2024-03-06T07:28:40.924277+0000","last_fullsized":"2024-03-06T07:28:40.924277+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922654+0000","last_change":"2024-03-06T07:28:40.922654+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922654+0000","last_undegraded":"2024-03-06T07:28:40.922654+0000","last_fullsized":"2024-03-06T07:28:40.922654+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927409+0000","last_change":"2024-03-06T07:28:40.927409+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927409+0000","last_undegraded":"2024-03-06T07:28:40.927409+0000","last_fullsized":"2024-03-06T07:28:40.927409+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922559+0000","last_change":"2024-03-06T07:28:40.922559+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922559+0000","last_undegraded":"2024-03-06T07:28:40.922559+0000","last_fullsized":"2024-03-06T07:28:40.922559+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923890+0000","last_change":"2024-03-06T07:28:40.923890+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923890+0000","last_undegraded":"2024-03-06T07:28:40.923890+0000","last_fullsized":"2024-03-06T07:28:40.923890+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.921998+0000","last_change":"2024-03-06T07:28:40.921998+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.921998+0000","last_undegraded":"2024-03-06T07:28:40.921998+0000","last_fullsized":"2024-03-06T07:28:40.921998+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923046+0000","last_change":"2024-03-06T07:28:40.923046+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923046+0000","last_undegraded":"2024-03-06T07:28:40.923046+0000","last_fullsized":"2024-03-06T07:28:40.923046+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.919747+0000","last_change":"2024-03-06T07:28:40.919747+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.919747+0000","last_undegraded":"2024-03-06T07:28:40.919747+0000","last_fullsized":"2024-03-06T07:28:40.919747+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927342+0000","last_change":"2024-03-06T07:28:40.927342+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927342+0000","last_undegraded":"2024-03-06T07:28:40.927342+0000","last_fullsized":"2024-03-06T07:28:40.927342+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922365+0000","last_change":"2024-03-06T07:28:40.922365+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922365+0000","last_undegraded":"2024-03-06T07:28:40.922365+0000","last_fullsized":"2024-03-06T07:28:40.922365+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922081+0000","last_change":"2024-03-06T07:28:40.922081+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922081+0000","last_undegraded":"2024-03-06T07:28:40.922081+0000","last_fullsized":"2024-03-06T07:28:40.922081+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927877+0000","last_change":"2024-03-06T07:28:40.927877+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927877+0000","last_undegraded":"2024-03-06T07:28:40.927877+0000","last_fullsized":"2024-03-06T07:28:40.927877+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922113+0000","last_change":"2024-03-06T07:28:40.922113+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922113+0000","last_undegraded":"2024-03-06T07:28:40.922113+0000","last_fullsized":"2024-03-06T07:28:40.922113+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922430+0000","last_change":"2024-03-06T07:28:40.922430+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922430+0000","last_undegraded":"2024-03-06T07:28:40.922430+0000","last_fullsized":"2024-03-06T07:28:40.922430+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922580+0000","last_change":"2024-03-06T07:28:40.922580+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922580+0000","last_undegraded":"2024-03-06T07:28:40.922580+0000","last_fullsized":"2024-03-06T07:28:40.922580+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923848+0000","last_change":"2024-03-06T07:28:40.923848+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923848+0000","last_undegraded":"2024-03-06T07:28:40.923848+0000","last_fullsized":"2024-03-06T07:28:40.923848+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924394+0000","last_change":"2024-03-06T07:28:40.924394+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924394+0000","last_undegraded":"2024-03-06T07:28:40.924394+0000","last_fullsized":"2024-03-06T07:28:40.924394+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.928144+0000","last_change":"2024-03-06T07:28:40.928144+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.928144+0000","last_undegraded":"2024-03-06T07:28:40.928144+0000","last_fullsized":"2024-03-06T07:28:40.928144+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.919708+0000","last_change":"2024-03-06T07:28:40.919708+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.919708+0000","last_undegraded":"2024-03-06T07:28:40.919708+0000","last_fullsized":"2024-03-06T07:28:40.919708+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924586+0000","last_change":"2024-03-06T07:28:40.924586+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924586+0000","last_undegraded":"2024-03-06T07:28:40.924586+0000","last_fullsized":"2024-03-06T07:28:40.924586+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.930002+0000","last_change":"2024-03-06T07:28:40.930002+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.930002+0000","last_undegraded":"2024-03-06T07:28:40.930002+0000","last_fullsized":"2024-03-06T07:28:40.930002+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927300+0000","last_change":"2024-03-06T07:28:40.927300+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927300+0000","last_undegraded":"2024-03-06T07:28:40.927300+0000","last_fullsized":"2024-03-06T07:28:40.927300+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927340+0000","last_change":"2024-03-06T07:28:40.927340+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927340+0000","last_undegraded":"2024-03-06T07:28:40.927340+0000","last_fullsized":"2024-03-06T07:28:40.927340+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.925257+0000","last_change":"2024-03-06T07:28:40.925257+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.925257+0000","last_undegraded":"2024-03-06T07:28:40.925257+0000","last_fullsized":"2024-03-06T07:28:40.925257+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923956+0000","last_change":"2024-03-06T07:28:40.923956+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923956+0000","last_undegraded":"2024-03-06T07:28:40.923956+0000","last_fullsized":"2024-03-06T07:28:40.923956+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927584+0000","last_change":"2024-03-06T07:28:40.927584+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927584+0000","last_undegraded":"2024-03-06T07:28:40.927584+0000","last_fullsized":"2024-03-06T07:28:40.927584+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.925204+0000","last_change":"2024-03-06T07:28:40.925204+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.925204+0000","last_undegraded":"2024-03-06T07:28:40.925204+0000","last_fullsized":"2024-03-06T07:28:40.925204+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":353,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.925605+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:28:40.925605+0000","last_peered":"2024-03-06T07:28:40.925605+0000","last_clean":"2024-03-06T07:28:40.925605+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:28:40.925605+0000","last_undegraded":"2024-03-06T07:28:40.925605+0000","last_fullsized":"2024-03-06T07:28:40.925605+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41572,"kb_used_data":2068,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330268,"statfs":{"total":96636764160,"available":96594194432,"internally_reserved":0,"allocated":2117632,"data_stored":1251263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40996,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330844,"statfs":{"total":96636764160,"available":96594784256,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40992,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330848,"statfs":{"total":96636764160,"available":96594788352,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41580,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330260,"statfs":{"total":96636764160,"available":96594186240,"internally_reserved":0,"allocated":2129920,"data_stored":1257080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:46.297 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.433+0000 7fa5b74fb640 1 Processor -- start 2024-03-06T07:28:46.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.433+0000 7fa5b74fb640 1 -- start start 2024-03-06T07:28:46.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.434+0000 7fa5b74fb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa5b015dfe0 0x7fa5b00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.434+0000 7fa5b74fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b00d29c0 0x7fa5b00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.434+0000 7fa5b74fb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b00b3430 con 0x7fa5b00cf830 2024-03-06T07:28:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.434+0000 7fa5b74fb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b00b31e0 con 0x7fa5b015dfe0 2024-03-06T07:28:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.434+0000 7fa5b74fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b00b3d60 con 0x7fa5b00d29c0 2024-03-06T07:28:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.434+0000 7fa5b5297640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa5b015dfe0 0x7fa5b00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.434+0000 7fa5b5297640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa5b015dfe0 0x7fa5b00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48232/0 (socket says 172.21.15.59:48232) 2024-03-06T07:28:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.434+0000 7fa5b5297640 1 -- 172.21.15.59:0/283468772 learned_addr learned my addr 172.21.15.59:0/283468772 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.435+0000 7fa5b5297640 1 -- 172.21.15.59:0/283468772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b00d29c0 msgr2=0x7fa5b00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.435+0000 7fa5b5297640 1 --2- 172.21.15.59:0/283468772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b00d29c0 0x7fa5b00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.435+0000 7fa5b5297640 1 -- 172.21.15.59:0/283468772 >> v1:172.21.15.164:6789/0 conn(0x7fa5b00cf830 legacy=0x7fa5b00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.435+0000 7fa5b5297640 1 -- 172.21.15.59:0/283468772 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b00b3710 con 0x7fa5b015dfe0 2024-03-06T07:28:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.435+0000 7fa5b5297640 1 --2- 172.21.15.59:0/283468772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b015dfe0 0x7fa5b00cf2f0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009870 tx=0x7fa5ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7404eb968dfbdc41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.435+0000 7fa5a7fff640 1 -- 172.21.15.59:0/283468772 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac035020 con 0x7fa5b015dfe0 2024-03-06T07:28:46.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.435+0000 7fa5a7fff640 1 -- 172.21.15.59:0/283468772 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac031640 con 0x7fa5b015dfe0 2024-03-06T07:28:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.435+0000 7fa5a7fff640 1 -- 172.21.15.59:0/283468772 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac02d040 con 0x7fa5b015dfe0 2024-03-06T07:28:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.436+0000 7fa5b74fb640 1 -- 172.21.15.59:0/283468772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b015dfe0 msgr2=0x7fa5b00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.436+0000 7fa5b74fb640 1 --2- 172.21.15.59:0/283468772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b015dfe0 0x7fa5b00cf2f0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009870 tx=0x7fa5ac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.436+0000 7fa5b74fb640 1 -- 172.21.15.59:0/283468772 shutdown_connections 2024-03-06T07:28:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.436+0000 7fa5b74fb640 1 --2- 172.21.15.59:0/283468772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b00d29c0 0x7fa5b00b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.436+0000 7fa5b74fb640 1 --2- 172.21.15.59:0/283468772 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b015dfe0 0x7fa5b00cf2f0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.436+0000 7fa5b74fb640 1 -- 172.21.15.59:0/283468772 >> 172.21.15.59:0/283468772 conn(0x7fa5b00d3cd0 msgr2=0x7fa5b00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.436+0000 7fa5b74fb640 1 -- 172.21.15.59:0/283468772 shutdown_connections 2024-03-06T07:28:46.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.436+0000 7fa5b74fb640 1 -- 172.21.15.59:0/283468772 wait complete. 2024-03-06T07:28:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.437+0000 7fa5b74fb640 1 Processor -- start 2024-03-06T07:28:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.437+0000 7fa5b74fb640 1 -- start start 2024-03-06T07:28:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b74fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b00cf830 0x7fa5b01f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b74fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b00d29c0 0x7fa5b00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b74fb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5b015dfe0 0x7fa5b0160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b74fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b00d1df0 con 0x7fa5b00d29c0 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b74fb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b00d1c70 con 0x7fa5b015dfe0 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b74fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b015e3c0 con 0x7fa5b00cf830 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b4a96640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b00d29c0 0x7fa5b00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b4a96640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b00d29c0 0x7fa5b00cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48234/0 (socket says 172.21.15.59:48234) 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b4a96640 1 -- 172.21.15.59:0/1042658284 learned_addr learned my addr 172.21.15.59:0/1042658284 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b5297640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b00cf830 0x7fa5b01f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b5a98640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5b015dfe0 0x7fa5b0160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b4a96640 1 -- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b00cf830 msgr2=0x7fa5b01f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b4a96640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b00cf830 0x7fa5b01f3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b4a96640 1 -- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5b015dfe0 msgr2=0x7fa5b0160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b4a96640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5b015dfe0 0x7fa5b0160ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.438+0000 7fa5b4a96640 1 -- 172.21.15.59:0/1042658284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b015fc80 con 0x7fa5b00d29c0 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.439+0000 7fa5b4a96640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b00d29c0 0x7fa5b00cada0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000c400 tx=0x7fa5a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.439+0000 7fa5a5ffb640 1 -- 172.21.15.59:0/1042658284 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0017070 con 0x7fa5b00d29c0 2024-03-06T07:28:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.439+0000 7fa5a5ffb640 1 -- 172.21.15.59:0/1042658284 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a00035f0 con 0x7fa5b00d29c0 2024-03-06T07:28:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.439+0000 7fa5b74fb640 1 -- 172.21.15.59:0/1042658284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b00cb780 con 0x7fa5b00d29c0 2024-03-06T07:28:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.439+0000 7fa5a5ffb640 1 -- 172.21.15.59:0/1042658284 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0003a20 con 0x7fa5b00d29c0 2024-03-06T07:28:46.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.439+0000 7fa5b74fb640 1 -- 172.21.15.59:0/1042658284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b00cbd40 con 0x7fa5b00d29c0 2024-03-06T07:28:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.440+0000 7fa5b74fb640 1 -- 172.21.15.59:0/1042658284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b004f330 con 0x7fa5b00d29c0 2024-03-06T07:28:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.440+0000 7fa5a5ffb640 1 -- 172.21.15.59:0/1042658284 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5a001a070 con 0x7fa5b00d29c0 2024-03-06T07:28:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.441+0000 7fa5a5ffb640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b4d0 0x7fa59406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.441+0000 7fa5a5ffb640 1 -- 172.21.15.59:0/1042658284 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5a0089510 con 0x7fa5b00d29c0 2024-03-06T07:28:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.441+0000 7fa5b5297640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b4d0 0x7fa59406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.442+0000 7fa5b5297640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b4d0 0x7fa59406d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac036000 tx=0x7fa5ac03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.445+0000 7fa5a5ffb640 1 -- 172.21.15.59:0/1042658284 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5a0057d40 con 0x7fa5b00d29c0 2024-03-06T07:28:46.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.625+0000 7fa5b74fb640 1 -- 172.21.15.59:0/1042658284 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa5b00747a0 con 0x7fa59406b4d0 2024-03-06T07:28:46.632 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.631+0000 7fa5a5ffb640 1 -- 172.21.15.59:0/1042658284 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240269 (secure 0 0 0) 0x7fa5b00747a0 con 0x7fa59406b4d0 2024-03-06T07:28:46.635 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:46.636 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:28:46.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 -- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b4d0 msgr2=0x7fa59406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b4d0 0x7fa59406d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac036000 tx=0x7fa5ac03c040 comp rx=0 tx=0).stop 2024-03-06T07:28:46.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 -- 172.21.15.59:0/1042658284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b00d29c0 msgr2=0x7fa5b00cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b00d29c0 0x7fa5b00cada0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000c400 tx=0x7fa5a000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.637 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 -- 172.21.15.59:0/1042658284 shutdown_connections 2024-03-06T07:28:46.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa59406b4d0 0x7fa59406d990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5b015dfe0 0x7fa5b0160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5b00d29c0 0x7fa5b00cada0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 --2- 172.21.15.59:0/1042658284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5b00cf830 0x7fa5b01f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 -- 172.21.15.59:0/1042658284 >> 172.21.15.59:0/1042658284 conn(0x7fa5b00d3cd0 msgr2=0x7fa5b00d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 -- 172.21.15.59:0/1042658284 shutdown_connections 2024-03-06T07:28:46.638 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.634+0000 7fa5b74fb640 1 -- 172.21.15.59:0/1042658284 wait complete. 2024-03-06T07:28:46.647 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-03-06T07:28:45.392211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329140,"kb_used_data":13112,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645580,"statfs":{"total":773094113280,"available":772757073920,"internally_reserved":0,"allocated":13426688,"data_stored":6491164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.471326"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924544+0000","last_change":"2024-03-06T07:28:40.924544+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924544+0000","last_undegraded":"2024-03-06T07:28:40.924544+0000","last_fullsized":"2024-03-06T07:28:40.924544+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922334+0000","last_change":"2024-03-06T07:28:40.922334+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922334+0000","last_undegraded":"2024-03-06T07:28:40.922334+0000","last_fullsized":"2024-03-06T07:28:40.922334+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927923+0000","last_change":"2024-03-06T07:28:40.927923+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927923+0000","last_undegraded":"2024-03-06T07:28:40.927923+0000","last_fullsized":"2024-03-06T07:28:40.927923+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.929980+0000","last_change":"2024-03-06T07:28:40.929980+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.929980+0000","last_undegraded":"2024-03-06T07:28:40.929980+0000","last_fullsized":"2024-03-06T07:28:40.929980+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.929949+0000","last_change":"2024-03-06T07:28:40.929949+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.929949+0000","last_undegraded":"2024-03-06T07:28:40.929949+0000","last_fullsized":"2024-03-06T07:28:40.929949+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924277+0000","last_change":"2024-03-06T07:28:40.924277+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924277+0000","last_undegraded":"2024-03-06T07:28:40.924277+0000","last_fullsized":"2024-03-06T07:28:40.924277+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922654+0000","last_change":"2024-03-06T07:28:40.922654+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922654+0000","last_undegraded":"2024-03-06T07:28:40.922654+0000","last_fullsized":"2024-03-06T07:28:40.922654+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927409+0000","last_change":"2024-03-06T07:28:40.927409+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927409+0000","last_undegraded":"2024-03-06T07:28:40.927409+0000","last_fullsized":"2024-03-06T07:28:40.927409+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922559+0000","last_change":"2024-03-06T07:28:40.922559+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922559+0000","last_undegraded":"2024-03-06T07:28:40.922559+0000","last_fullsized":"2024-03-06T07:28:40.922559+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923890+0000","last_change":"2024-03-06T07:28:40.923890+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923890+0000","last_undegraded":"2024-03-06T07:28:40.923890+0000","last_fullsized":"2024-03-06T07:28:40.923890+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.921998+0000","last_change":"2024-03-06T07:28:40.921998+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.921998+0000","last_undegraded":"2024-03-06T07:28:40.921998+0000","last_fullsized":"2024-03-06T07:28:40.921998+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923046+0000","last_change":"2024-03-06T07:28:40.923046+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923046+0000","last_undegraded":"2024-03-06T07:28:40.923046+0000","last_fullsized":"2024-03-06T07:28:40.923046+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.919747+0000","last_change":"2024-03-06T07:28:40.919747+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.919747+0000","last_undegraded":"2024-03-06T07:28:40.919747+0000","last_fullsized":"2024-03-06T07:28:40.919747+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927342+0000","last_change":"2024-03-06T07:28:40.927342+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927342+0000","last_undegraded":"2024-03-06T07:28:40.927342+0000","last_fullsized":"2024-03-06T07:28:40.927342+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922365+0000","last_change":"2024-03-06T07:28:40.922365+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922365+0000","last_undegraded":"2024-03-06T07:28:40.922365+0000","last_fullsized":"2024-03-06T07:28:40.922365+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922081+0000","last_change":"2024-03-06T07:28:40.922081+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922081+0000","last_undegraded":"2024-03-06T07:28:40.922081+0000","last_fullsized":"2024-03-06T07:28:40.922081+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927877+0000","last_change":"2024-03-06T07:28:40.927877+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927877+0000","last_undegraded":"2024-03-06T07:28:40.927877+0000","last_fullsized":"2024-03-06T07:28:40.927877+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922113+0000","last_change":"2024-03-06T07:28:40.922113+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922113+0000","last_undegraded":"2024-03-06T07:28:40.922113+0000","last_fullsized":"2024-03-06T07:28:40.922113+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922430+0000","last_change":"2024-03-06T07:28:40.922430+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922430+0000","last_undegraded":"2024-03-06T07:28:40.922430+0000","last_fullsized":"2024-03-06T07:28:40.922430+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922580+0000","last_change":"2024-03-06T07:28:40.922580+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922580+0000","last_undegraded":"2024-03-06T07:28:40.922580+0000","last_fullsized":"2024-03-06T07:28:40.922580+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923848+0000","last_change":"2024-03-06T07:28:40.923848+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923848+0000","last_undegraded":"2024-03-06T07:28:40.923848+0000","last_fullsized":"2024-03-06T07:28:40.923848+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924394+0000","last_change":"2024-03-06T07:28:40.924394+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924394+0000","last_undegraded":"2024-03-06T07:28:40.924394+0000","last_fullsized":"2024-03-06T07:28:40.924394+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.928144+0000","last_change":"2024-03-06T07:28:40.928144+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.928144+0000","last_undegraded":"2024-03-06T07:28:40.928144+0000","last_fullsized":"2024-03-06T07:28:40.928144+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.919708+0000","last_change":"2024-03-06T07:28:40.919708+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.919708+0000","last_undegraded":"2024-03-06T07:28:40.919708+0000","last_fullsized":"2024-03-06T07:28:40.919708+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924586+0000","last_change":"2024-03-06T07:28:40.924586+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924586+0000","last_undegraded":"2024-03-06T07:28:40.924586+0000","last_fullsized":"2024-03-06T07:28:40.924586+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.930002+0000","last_change":"2024-03-06T07:28:40.930002+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.930002+0000","last_undegraded":"2024-03-06T07:28:40.930002+0000","last_fullsized":"2024-03-06T07:28:40.930002+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927300+0000","last_change":"2024-03-06T07:28:40.927300+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927300+0000","last_undegraded":"2024-03-06T07:28:40.927300+0000","last_fullsized":"2024-03-06T07:28:40.927300+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927340+0000","last_change":"2024-03-06T07:28:40.927340+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927340+0000","last_undegraded":"2024-03-06T07:28:40.927340+0000","last_fullsized":"2024-03-06T07:28:40.927340+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.925257+0000","last_change":"2024-03-06T07:28:40.925257+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.925257+0000","last_undegraded":"2024-03-06T07:28:40.925257+0000","last_fullsized":"2024-03-06T07:28:40.925257+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923956+0000","last_change":"2024-03-06T07:28:40.923956+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923956+0000","last_undegraded":"2024-03-06T07:28:40.923956+0000","last_fullsized":"2024-03-06T07:28:40.923956+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927584+0000","last_change":"2024-03-06T07:28:40.927584+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927584+0000","last_undegraded":"2024-03-06T07:28:40.927584+0000","last_fullsized":"2024-03-06T07:28:40.927584+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.925204+0000","last_change":"2024-03-06T07:28:40.925204+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.925204+0000","last_undegraded":"2024-03-06T07:28:40.925204+0000","last_fullsized":"2024-03-06T07:28:40.925204+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":353,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.925605+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:28:40.925605+0000","last_peered":"2024-03-06T07:28:40.925605+0000","last_clean":"2024-03-06T07:28:40.925605+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:28:40.925605+0000","last_undegraded":"2024-03-06T07:28:40.925605+0000","last_fullsized":"2024-03-06T07:28:40.925605+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41572,"kb_used_data":2068,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330268,"statfs":{"total":96636764160,"available":96594194432,"internally_reserved":0,"allocated":2117632,"data_stored":1251263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40996,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330844,"statfs":{"total":96636764160,"available":96594784256,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40992,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330848,"statfs":{"total":96636764160,"available":96594788352,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41580,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330260,"statfs":{"total":96636764160,"available":96594186240,"internally_reserved":0,"allocated":2129920,"data_stored":1257080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:46.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3f is not active+clean 2024-03-06T07:28:46.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3e is not active+clean 2024-03-06T07:28:46.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.650 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3d is not active+clean 2024-03-06T07:28:46.650 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3c is not active+clean 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3b is not active+clean 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3a is not active+clean 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.39 is not active+clean 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.38 is not active+clean 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.37 is not active+clean 2024-03-06T07:28:46.651 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.36 is not active+clean 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.35 is not active+clean 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.34 is not active+clean 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.33 is not active+clean 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.32 is not active+clean 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.31 is not active+clean 2024-03-06T07:28:46.652 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.30 is not active+clean 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2f is not active+clean 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2e is not active+clean 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2d is not active+clean 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2c is not active+clean 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2b is not active+clean 2024-03-06T07:28:46.653 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2a is not active+clean 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.29 is not active+clean 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.28 is not active+clean 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.27 is not active+clean 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.26 is not active+clean 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.25 is not active+clean 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.24 is not active+clean 2024-03-06T07:28:46.654 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.23 is not active+clean 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.22 is not active+clean 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.21 is not active+clean 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.20 is not active+clean 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1e is not active+clean 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1f is not active+clean 2024-03-06T07:28:46.655 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.924544+0000', 'last_change': '2024-03-06T07:28:40.924544+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.924544+0000', 'last_undegraded': '2024-03-06T07:28:40.924544+0000', 'last_fullsized': '2024-03-06T07:28:40.924544+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1f is not active+clean 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1e is not active+clean 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.922334+0000', 'last_change': '2024-03-06T07:28:40.922334+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.922334+0000', 'last_undegraded': '2024-03-06T07:28:40.922334+0000', 'last_fullsized': '2024-03-06T07:28:40.922334+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1c is not active+clean 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1d is not active+clean 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.927923+0000', 'last_change': '2024-03-06T07:28:40.927923+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.927923+0000', 'last_undegraded': '2024-03-06T07:28:40.927923+0000', 'last_fullsized': '2024-03-06T07:28:40.927923+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1d is not active+clean 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1c is not active+clean 2024-03-06T07:28:46.656 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.929980+0000', 'last_change': '2024-03-06T07:28:40.929980+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.929980+0000', 'last_undegraded': '2024-03-06T07:28:40.929980+0000', 'last_fullsized': '2024-03-06T07:28:40.929980+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1a is not active+clean 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1b is not active+clean 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.929949+0000', 'last_change': '2024-03-06T07:28:40.929949+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.929949+0000', 'last_undegraded': '2024-03-06T07:28:40.929949+0000', 'last_fullsized': '2024-03-06T07:28:40.929949+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1b is not active+clean 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1a is not active+clean 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.924277+0000', 'last_change': '2024-03-06T07:28:40.924277+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.924277+0000', 'last_undegraded': '2024-03-06T07:28:40.924277+0000', 'last_fullsized': '2024-03-06T07:28:40.924277+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.18 is not active+clean 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.19 is not active+clean 2024-03-06T07:28:46.657 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.922654+0000', 'last_change': '2024-03-06T07:28:40.922654+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.922654+0000', 'last_undegraded': '2024-03-06T07:28:40.922654+0000', 'last_fullsized': '2024-03-06T07:28:40.922654+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.19 is not active+clean 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.18 is not active+clean 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.927409+0000', 'last_change': '2024-03-06T07:28:40.927409+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.927409+0000', 'last_undegraded': '2024-03-06T07:28:40.927409+0000', 'last_fullsized': '2024-03-06T07:28:40.927409+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.16 is not active+clean 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.17 is not active+clean 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.922559+0000', 'last_change': '2024-03-06T07:28:40.922559+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.922559+0000', 'last_undegraded': '2024-03-06T07:28:40.922559+0000', 'last_fullsized': '2024-03-06T07:28:40.922559+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.17 is not active+clean 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.16 is not active+clean 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.923890+0000', 'last_change': '2024-03-06T07:28:40.923890+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.923890+0000', 'last_undegraded': '2024-03-06T07:28:40.923890+0000', 'last_fullsized': '2024-03-06T07:28:40.923890+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.14 is not active+clean 2024-03-06T07:28:46.658 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.15 is not active+clean 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.921998+0000', 'last_change': '2024-03-06T07:28:40.921998+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.921998+0000', 'last_undegraded': '2024-03-06T07:28:40.921998+0000', 'last_fullsized': '2024-03-06T07:28:40.921998+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.15 is not active+clean 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.14 is not active+clean 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.923046+0000', 'last_change': '2024-03-06T07:28:40.923046+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.923046+0000', 'last_undegraded': '2024-03-06T07:28:40.923046+0000', 'last_fullsized': '2024-03-06T07:28:40.923046+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.13 is not active+clean 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.12 is not active+clean 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.919747+0000', 'last_change': '2024-03-06T07:28:40.919747+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.919747+0000', 'last_undegraded': '2024-03-06T07:28:40.919747+0000', 'last_fullsized': '2024-03-06T07:28:40.919747+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.10 is not active+clean 2024-03-06T07:28:46.659 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.11 is not active+clean 2024-03-06T07:28:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.927342+0000', 'last_change': '2024-03-06T07:28:40.927342+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.927342+0000', 'last_undegraded': '2024-03-06T07:28:40.927342+0000', 'last_fullsized': '2024-03-06T07:28:40.927342+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.11 is not active+clean 2024-03-06T07:28:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.10 is not active+clean 2024-03-06T07:28:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.922365+0000', 'last_change': '2024-03-06T07:28:40.922365+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.922365+0000', 'last_undegraded': '2024-03-06T07:28:40.922365+0000', 'last_fullsized': '2024-03-06T07:28:40.922365+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.e is not active+clean 2024-03-06T07:28:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.f is not active+clean 2024-03-06T07:28:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.922081+0000', 'last_change': '2024-03-06T07:28:40.922081+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.922081+0000', 'last_undegraded': '2024-03-06T07:28:40.922081+0000', 'last_fullsized': '2024-03-06T07:28:40.922081+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:46.660 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.f is not active+clean 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.e is not active+clean 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.927877+0000', 'last_change': '2024-03-06T07:28:40.927877+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.927877+0000', 'last_undegraded': '2024-03-06T07:28:40.927877+0000', 'last_fullsized': '2024-03-06T07:28:40.927877+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.c is not active+clean 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.d is not active+clean 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.922113+0000', 'last_change': '2024-03-06T07:28:40.922113+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.922113+0000', 'last_undegraded': '2024-03-06T07:28:40.922113+0000', 'last_fullsized': '2024-03-06T07:28:40.922113+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.d is not active+clean 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.c is not active+clean 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.922430+0000', 'last_change': '2024-03-06T07:28:40.922430+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.922430+0000', 'last_undegraded': '2024-03-06T07:28:40.922430+0000', 'last_fullsized': '2024-03-06T07:28:40.922430+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.a is not active+clean 2024-03-06T07:28:46.661 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.b is not active+clean 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.922580+0000', 'last_change': '2024-03-06T07:28:40.922580+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.922580+0000', 'last_undegraded': '2024-03-06T07:28:40.922580+0000', 'last_fullsized': '2024-03-06T07:28:40.922580+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.b is not active+clean 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.a is not active+clean 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.923848+0000', 'last_change': '2024-03-06T07:28:40.923848+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.923848+0000', 'last_undegraded': '2024-03-06T07:28:40.923848+0000', 'last_fullsized': '2024-03-06T07:28:40.923848+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.8 is not active+clean 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.9 is not active+clean 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.924394+0000', 'last_change': '2024-03-06T07:28:40.924394+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.924394+0000', 'last_undegraded': '2024-03-06T07:28:40.924394+0000', 'last_fullsized': '2024-03-06T07:28:40.924394+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.9 is not active+clean 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.662 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.8 is not active+clean 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.928144+0000', 'last_change': '2024-03-06T07:28:40.928144+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.928144+0000', 'last_undegraded': '2024-03-06T07:28:40.928144+0000', 'last_fullsized': '2024-03-06T07:28:40.928144+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.6 is not active+clean 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.7 is not active+clean 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.919708+0000', 'last_change': '2024-03-06T07:28:40.919708+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.919708+0000', 'last_undegraded': '2024-03-06T07:28:40.919708+0000', 'last_fullsized': '2024-03-06T07:28:40.919708+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.7 is not active+clean 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.6 is not active+clean 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.924586+0000', 'last_change': '2024-03-06T07:28:40.924586+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.924586+0000', 'last_undegraded': '2024-03-06T07:28:40.924586+0000', 'last_fullsized': '2024-03-06T07:28:40.924586+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.4 is not active+clean 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.5 is not active+clean 2024-03-06T07:28:46.663 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.930002+0000', 'last_change': '2024-03-06T07:28:40.930002+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.930002+0000', 'last_undegraded': '2024-03-06T07:28:40.930002+0000', 'last_fullsized': '2024-03-06T07:28:40.930002+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:46.664 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.5 is not active+clean 2024-03-06T07:28:46.664 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.664 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.4 is not active+clean 2024-03-06T07:28:46.664 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.927300+0000', 'last_change': '2024-03-06T07:28:40.927300+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.927300+0000', 'last_undegraded': '2024-03-06T07:28:40.927300+0000', 'last_fullsized': '2024-03-06T07:28:40.927300+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:46.664 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2 is not active+clean 2024-03-06T07:28:46.664 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.664 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.3 is not active+clean 2024-03-06T07:28:46.664 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.927340+0000', 'last_change': '2024-03-06T07:28:40.927340+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.927340+0000', 'last_undegraded': '2024-03-06T07:28:40.927340+0000', 'last_fullsized': '2024-03-06T07:28:40.927340+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:46.665 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3 is not active+clean 2024-03-06T07:28:46.665 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.665 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.2 is not active+clean 2024-03-06T07:28:46.665 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.925257+0000', 'last_change': '2024-03-06T07:28:40.925257+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.925257+0000', 'last_undegraded': '2024-03-06T07:28:40.925257+0000', 'last_fullsized': '2024-03-06T07:28:40.925257+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:46.665 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.0 is not active+clean 2024-03-06T07:28:46.665 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.1 is not active+clean 2024-03-06T07:28:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.923956+0000', 'last_change': '2024-03-06T07:28:40.923956+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.923956+0000', 'last_undegraded': '2024-03-06T07:28:40.923956+0000', 'last_fullsized': '2024-03-06T07:28:40.923956+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1 is not active+clean 2024-03-06T07:28:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.0 is not active+clean 2024-03-06T07:28:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.927584+0000', 'last_change': '2024-03-06T07:28:40.927584+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.927584+0000', 'last_undegraded': '2024-03-06T07:28:40.927584+0000', 'last_fullsized': '2024-03-06T07:28:40.927584+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.12 is not active+clean 2024-03-06T07:28:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:28:41.916963+0000', 'last_change': '2024-03-06T07:28:41.916963+0000', 'last_active': '2024-03-06T07:28:41.916963+0000', 'last_peered': '2024-03-06T07:28:41.916963+0000', 'last_clean': '2024-03-06T07:28:41.916963+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:41.916963+0000', 'last_undegraded': '2024-03-06T07:28:41.916963+0000', 'last_fullsized': '2024-03-06T07:28:41.916963+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:41.916963+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:28:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:PG 18.13 is not active+clean 2024-03-06T07:28:46.666 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '18.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 123, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:40.925204+0000', 'last_change': '2024-03-06T07:28:40.925204+0000', 'last_active': '2024-03-06T07:28:40.911886+0000', 'last_peered': '2024-03-06T07:28:40.911886+0000', 'last_clean': '2024-03-06T07:28:40.911886+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:40.925204+0000', 'last_undegraded': '2024-03-06T07:28:40.925204+0000', 'last_fullsized': '2024-03-06T07:28:40.925204+0000', 'mapping_epoch': 123, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 123, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:40.911886+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:46.667 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:28:46.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.785+0000 7f2b4cdbc640 1 Processor -- start 2024-03-06T07:28:46.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.785+0000 7f2b4cdbc640 1 -- start start 2024-03-06T07:28:46.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b4cdbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b48162970 0x7f2b480d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b4cdbc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2b48161cf0 0x7f2b48162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b4cdbc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b480d22c0 con 0x7f2b480b4460 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b4cdbc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b48161b00 con 0x7f2b48161cf0 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b4cdbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480cecc0 con 0x7f2b48162970 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b4659c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2b480b4460 0x7f2b480b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44462/0 (socket says 172.21.15.59:44462) 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b4659c640 1 -- 172.21.15.59:0/2980438298 learned_addr learned my addr 172.21.15.59:0/2980438298 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b3ffff640 1 --2- 172.21.15.59:0/2980438298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b48162970 0x7f2b480d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b46d9d640 1 --2- 172.21.15.59:0/2980438298 >> v2:172.21.15.164:3300/0 conn(0x7f2b48161cf0 0x7f2b48162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b45d9b640 1 -- 172.21.15.59:0/2980438298 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534746832 0 0) 0x7f2b480d22c0 con 0x7f2b480b4460 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b45d9b640 1 -- 172.21.15.59:0/2980438298 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b34003520 con 0x7f2b480b4460 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b46d9d640 1 -- 172.21.15.59:0/2980438298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b48162970 msgr2=0x7f2b480d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b46d9d640 1 --2- 172.21.15.59:0/2980438298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b48162970 0x7f2b480d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b46d9d640 1 -- 172.21.15.59:0/2980438298 >> v1:172.21.15.59:6789/0 conn(0x7f2b480b4460 legacy=0x7f2b480b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b46d9d640 1 -- 172.21.15.59:0/2980438298 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b480d25a0 con 0x7f2b48161cf0 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.787+0000 7f2b3ffff640 1 --2- 172.21.15.59:0/2980438298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b48162970 0x7f2b480d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.788+0000 7f2b46d9d640 1 --2- 172.21.15.59:0/2980438298 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b48161cf0 0x7f2b48162180 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009d90 tx=0x7f2b4002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=938087c7e8a4faa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.788+0000 7f2b45d9b640 1 -- 172.21.15.59:0/2980438298 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b40039070 con 0x7f2b48161cf0 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.788+0000 7f2b45d9b640 1 -- 172.21.15.59:0/2980438298 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4002cc50 con 0x7f2b48161cf0 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.788+0000 7f2b45d9b640 1 -- 172.21.15.59:0/2980438298 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b40034440 con 0x7f2b48161cf0 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.789+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2980438298 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b48161cf0 msgr2=0x7f2b48162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.789+0000 7f2b4cdbc640 1 --2- 172.21.15.59:0/2980438298 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b48161cf0 0x7f2b48162180 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009d90 tx=0x7f2b4002bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.789+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2980438298 shutdown_connections 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.789+0000 7f2b4cdbc640 1 --2- 172.21.15.59:0/2980438298 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b48161cf0 0x7f2b48162180 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.789+0000 7f2b4cdbc640 1 --2- 172.21.15.59:0/2980438298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b48162970 0x7f2b480d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.789+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2980438298 >> 172.21.15.59:0/2980438298 conn(0x7f2b480d3ce0 msgr2=0x7f2b480d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.789+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2980438298 shutdown_connections 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.789+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2980438298 wait complete. 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.791+0000 7f2b4cdbc640 1 Processor -- start 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.791+0000 7f2b4cdbc640 1 -- start start 2024-03-06T07:28:46.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.792+0000 7f2b4cdbc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b480b4460 0x7f2b48156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.792+0000 7f2b4cdbc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b48161cf0 0x7f2b481f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.792+0000 7f2b4cdbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b48162970 0x7f2b480c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.792+0000 7f2b4cdbc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480ce950 con 0x7f2b480b4460 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.792+0000 7f2b4cdbc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480d2b60 con 0x7f2b48161cf0 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.792+0000 7f2b4cdbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480b4840 con 0x7f2b48162970 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b4659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b480b4460 0x7f2b48156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b4659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b480b4460 0x7f2b48156210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48240/0 (socket says 172.21.15.59:48240) 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b4659c640 1 -- 172.21.15.59:0/2380757554 learned_addr learned my addr 172.21.15.59:0/2380757554 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b3ffff640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b48161cf0 0x7f2b481f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b46d9d640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b48162970 0x7f2b480c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b4659c640 1 -- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b48162970 msgr2=0x7f2b480c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b4659c640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b48162970 0x7f2b480c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b4659c640 1 -- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b48161cf0 msgr2=0x7f2b481f7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b4659c640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b48161cf0 0x7f2b481f7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b4659c640 1 -- 172.21.15.59:0/2380757554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b48156750 con 0x7f2b480b4460 2024-03-06T07:28:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b4659c640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b480b4460 0x7f2b48156210 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800c390 tx=0x7f2b3800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b3effd640 1 -- 172.21.15.59:0/2380757554 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b38017070 con 0x7f2b480b4460 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b3effd640 1 -- 172.21.15.59:0/2380757554 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b38005ce0 con 0x7f2b480b4460 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2380757554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b481f81a0 con 0x7f2b480b4460 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b3effd640 1 -- 172.21.15.59:0/2380757554 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b380072f0 con 0x7f2b480b4460 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.793+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2380757554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b480b6e30 con 0x7f2b480b4460 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.794+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2380757554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4804f330 con 0x7f2b480b4460 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.795+0000 7f2b3effd640 1 -- 172.21.15.59:0/2380757554 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b3801a070 con 0x7f2b480b4460 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.795+0000 7f2b3effd640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b2806b4d0 0x7f2b2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.795+0000 7f2b3effd640 1 -- 172.21.15.59:0/2380757554 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b38089620 con 0x7f2b480b4460 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.795+0000 7f2b3ffff640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b2806b4d0 0x7f2b2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.796+0000 7f2b3ffff640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b2806b4d0 0x7f2b2806d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2b30009560 tx=0x7f2b30008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:46.800+0000 7f2b3effd640 1 -- 172.21.15.59:0/2380757554 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b48156750 con 0x7f2b480b4460 2024-03-06T07:28:47.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.052+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2380757554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2b4806b180 con 0x7f2b480b4460 2024-03-06T07:28:47.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.052+0000 7f2b3effd640 1 -- 172.21.15.59:0/2380757554 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1272 (secure 0 0 0) 0x7f2b380584a0 con 0x7f2b480b4460 2024-03-06T07:28:47.054 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:47.055 INFO:teuthology.orchestra.run.smithi059.stdout:{"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":436,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":126,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709701,"num_in_osds":8,"osd_in_since":1709709692,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":337039360,"bytes_avail":772757073920,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":79,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"c","status":"up:active","gid":8676}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-03-06T07:28:29.389218+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.054+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b2806b4d0 msgr2=0x7f2b2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.054+0000 7f2b4cdbc640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b2806b4d0 0x7f2b2806d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2b30009560 tx=0x7f2b30008040 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.054+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2380757554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b480b4460 msgr2=0x7f2b48156210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.054+0000 7f2b4cdbc640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b480b4460 0x7f2b48156210 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800c390 tx=0x7f2b3800c850 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.054+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2380757554 shutdown_connections 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.054+0000 7f2b4cdbc640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2b2806b4d0 0x7f2b2806d990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.054+0000 7f2b4cdbc640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b48162970 0x7f2b480c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.054+0000 7f2b4cdbc640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b48161cf0 0x7f2b481f7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.055+0000 7f2b4cdbc640 1 --2- 172.21.15.59:0/2380757554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2b480b4460 0x7f2b48156210 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.055+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2380757554 >> 172.21.15.59:0/2380757554 conn(0x7f2b480d3ce0 msgr2=0x7f2b480d0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.055+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2380757554 shutdown_connections 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.055+0000 7f2b4cdbc640 1 -- 172.21.15.59:0/2380757554 wait complete. 2024-03-06T07:28:47.067 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:28:47.068 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:47.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.168+0000 7f63a59c9640 1 Processor -- start 2024-03-06T07:28:47.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.168+0000 7f63a59c9640 1 -- start start 2024-03-06T07:28:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.169+0000 7f63a59c9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f63a00d4760 0x7f63a00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.169+0000 7f63a59c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63a00d0a80 0x7f63a00cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.169+0000 7f63a59c9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a00d25d0 con 0x7f63a00b4170 2024-03-06T07:28:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.169+0000 7f63a59c9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63a00d4bf0 con 0x7f63a00d4760 2024-03-06T07:28:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.169+0000 7f63a59c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a00ce9e0 con 0x7f63a00d0a80 2024-03-06T07:28:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.169+0000 7f639e7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f63a00d4760 0x7f63a00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.169+0000 7f639e7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f63a00d4760 0x7f63a00b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48252/0 (socket says 172.21.15.59:48252) 2024-03-06T07:28:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.169+0000 7f639e7fc640 1 -- 172.21.15.59:0/1248903023 learned_addr learned my addr 172.21.15.59:0/1248903023 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.170+0000 7f639f7fe640 1 --2- 172.21.15.59:0/1248903023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63a00d0a80 0x7f63a00cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.170+0000 7f639e7fc640 1 -- 172.21.15.59:0/1248903023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63a00d0a80 msgr2=0x7f63a00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.170+0000 7f639e7fc640 1 --2- 172.21.15.59:0/1248903023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63a00d0a80 0x7f63a00cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.170+0000 7f639e7fc640 1 -- 172.21.15.59:0/1248903023 >> v1:172.21.15.164:6789/0 conn(0x7f63a00b4170 legacy=0x7f63a00d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.170+0000 7f639e7fc640 1 -- 172.21.15.59:0/1248903023 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a00d2870 con 0x7f63a00d4760 2024-03-06T07:28:47.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.170+0000 7f639f7fe640 1 --2- 172.21.15.59:0/1248903023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63a00d0a80 0x7f63a00cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:47.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f639e7fc640 1 --2- 172.21.15.59:0/1248903023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63a00d4760 0x7f63a00b6a90 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f6388009870 tx=0x7f638802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d6aa36692ae58f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f639dffb640 1 -- 172.21.15.59:0/1248903023 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6388034070 con 0x7f63a00d4760 2024-03-06T07:28:47.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f639dffb640 1 -- 172.21.15.59:0/1248903023 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6388030840 con 0x7f63a00d4760 2024-03-06T07:28:47.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f639dffb640 1 -- 172.21.15.59:0/1248903023 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6388030c10 con 0x7f63a00d4760 2024-03-06T07:28:47.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f63a59c9640 1 -- 172.21.15.59:0/1248903023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63a00d4760 msgr2=0x7f63a00b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f63a59c9640 1 --2- 172.21.15.59:0/1248903023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63a00d4760 0x7f63a00b6a90 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f6388009870 tx=0x7f638802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:47.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f63a59c9640 1 -- 172.21.15.59:0/1248903023 shutdown_connections 2024-03-06T07:28:47.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f63a59c9640 1 --2- 172.21.15.59:0/1248903023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63a00d0a80 0x7f63a00cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f63a59c9640 1 --2- 172.21.15.59:0/1248903023 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63a00d4760 0x7f63a00b6a90 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f63a59c9640 1 -- 172.21.15.59:0/1248903023 >> 172.21.15.59:0/1248903023 conn(0x7f63a00d3c40 msgr2=0x7f63a00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f63a59c9640 1 -- 172.21.15.59:0/1248903023 shutdown_connections 2024-03-06T07:28:47.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.171+0000 7f63a59c9640 1 -- 172.21.15.59:0/1248903023 wait complete. 2024-03-06T07:28:47.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.173+0000 7f63a59c9640 1 Processor -- start 2024-03-06T07:28:47.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.173+0000 7f63a59c9640 1 -- start start 2024-03-06T07:28:47.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.173+0000 7f63a59c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63a00b4170 0x7f63a015dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.173+0000 7f63a59c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63a00d0a80 0x7f63a0156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.173+0000 7f63a59c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a00cbeb0 0x7f63a00cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.173+0000 7f63a59c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a00b4dd0 con 0x7f63a00b4170 2024-03-06T07:28:47.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.173+0000 7f63a59c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a00b4550 con 0x7f63a00cbeb0 2024-03-06T07:28:47.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.173+0000 7f63a59c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a00b4f50 con 0x7f63a00d0a80 2024-03-06T07:28:47.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f639effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63a00b4170 0x7f63a015dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f639effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63a00b4170 0x7f63a015dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48256/0 (socket says 172.21.15.59:48256) 2024-03-06T07:28:47.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f639effd640 1 -- 172.21.15.59:0/3854794871 learned_addr learned my addr 172.21.15.59:0/3854794871 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:47.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f639f7fe640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a00cbeb0 0x7f63a00cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f639e7fc640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63a00d0a80 0x7f63a0156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f639effd640 1 -- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63a00d0a80 msgr2=0x7f63a0156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f639effd640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63a00d0a80 0x7f63a0156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f639effd640 1 -- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a00cbeb0 msgr2=0x7f63a00cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f639effd640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a00cbeb0 0x7f63a00cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f639effd640 1 -- 172.21.15.59:0/3854794871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a00cc3b0 con 0x7f63a00b4170 2024-03-06T07:28:47.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f639effd640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63a00b4170 0x7f63a015dd00 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f639000c340 tx=0x7f639000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f6397fff640 1 -- 172.21.15.59:0/3854794871 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6390017070 con 0x7f63a00b4170 2024-03-06T07:28:47.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f6397fff640 1 -- 172.21.15.59:0/3854794871 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63900052d0 con 0x7f63a00b4170 2024-03-06T07:28:47.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f63a59c9640 1 -- 172.21.15.59:0/3854794871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a00cc690 con 0x7f63a00b4170 2024-03-06T07:28:47.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f6397fff640 1 -- 172.21.15.59:0/3854794871 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6390003850 con 0x7f63a00b4170 2024-03-06T07:28:47.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.174+0000 7f63a59c9640 1 -- 172.21.15.59:0/3854794871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a00ccc50 con 0x7f63a00b4170 2024-03-06T07:28:47.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.175+0000 7f63a59c9640 1 -- 172.21.15.59:0/3854794871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a004f330 con 0x7f63a00b4170 2024-03-06T07:28:47.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.176+0000 7f6397fff640 1 -- 172.21.15.59:0/3854794871 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f639001a070 con 0x7f63a00b4170 2024-03-06T07:28:47.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.176+0000 7f6397fff640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f638006b4d0 0x7f638006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.177+0000 7f6397fff640 1 -- 172.21.15.59:0/3854794871 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63900893d0 con 0x7f63a00b4170 2024-03-06T07:28:47.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.177+0000 7f639e7fc640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f638006b4d0 0x7f638006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.177+0000 7f639e7fc640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f638006b4d0 0x7f638006d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6388036000 tx=0x7f6388008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.181+0000 7f6397fff640 1 -- 172.21.15.59:0/3854794871 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6390057c90 con 0x7f63a00b4170 2024-03-06T07:28:47.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.375+0000 7f63a59c9640 1 -- 172.21.15.59:0/3854794871 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63a00733b0 con 0x7f638006b4d0 2024-03-06T07:28:47.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.381+0000 7f6397fff640 1 -- 172.21.15.59:0/3854794871 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240269 (secure 0 0 0) 0x7f63a00733b0 con 0x7f638006b4d0 2024-03-06T07:28:47.385 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:47.386 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:28:47.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 -- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f638006b4d0 msgr2=0x7f638006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f638006b4d0 0x7f638006d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6388036000 tx=0x7f6388008030 comp rx=0 tx=0).stop 2024-03-06T07:28:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 -- 172.21.15.59:0/3854794871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63a00b4170 msgr2=0x7f63a015dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63a00b4170 0x7f63a015dd00 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f639000c340 tx=0x7f639000c800 comp rx=0 tx=0).stop 2024-03-06T07:28:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 -- 172.21.15.59:0/3854794871 shutdown_connections 2024-03-06T07:28:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f638006b4d0 0x7f638006d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63a00cbeb0 0x7f63a00cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63a00d0a80 0x7f63a0156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 --2- 172.21.15.59:0/3854794871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63a00b4170 0x7f63a015dd00 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 -- 172.21.15.59:0/3854794871 >> 172.21.15.59:0/3854794871 conn(0x7f63a00d3c40 msgr2=0x7f63a00b76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 -- 172.21.15.59:0/3854794871 shutdown_connections 2024-03-06T07:28:47.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:47.384+0000 7f63a59c9640 1 -- 172.21.15.59:0/3854794871 wait complete. 2024-03-06T07:28:47.397 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-03-06T07:28:45.392211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329140,"kb_used_data":13112,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645580,"statfs":{"total":773094113280,"available":772757073920,"internally_reserved":0,"allocated":13426688,"data_stored":6491164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.471326"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924544+0000","last_change":"2024-03-06T07:28:40.924544+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924544+0000","last_undegraded":"2024-03-06T07:28:40.924544+0000","last_fullsized":"2024-03-06T07:28:40.924544+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922334+0000","last_change":"2024-03-06T07:28:40.922334+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922334+0000","last_undegraded":"2024-03-06T07:28:40.922334+0000","last_fullsized":"2024-03-06T07:28:40.922334+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927923+0000","last_change":"2024-03-06T07:28:40.927923+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927923+0000","last_undegraded":"2024-03-06T07:28:40.927923+0000","last_fullsized":"2024-03-06T07:28:40.927923+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.929980+0000","last_change":"2024-03-06T07:28:40.929980+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.929980+0000","last_undegraded":"2024-03-06T07:28:40.929980+0000","last_fullsized":"2024-03-06T07:28:40.929980+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.929949+0000","last_change":"2024-03-06T07:28:40.929949+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.929949+0000","last_undegraded":"2024-03-06T07:28:40.929949+0000","last_fullsized":"2024-03-06T07:28:40.929949+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924277+0000","last_change":"2024-03-06T07:28:40.924277+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924277+0000","last_undegraded":"2024-03-06T07:28:40.924277+0000","last_fullsized":"2024-03-06T07:28:40.924277+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922654+0000","last_change":"2024-03-06T07:28:40.922654+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922654+0000","last_undegraded":"2024-03-06T07:28:40.922654+0000","last_fullsized":"2024-03-06T07:28:40.922654+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927409+0000","last_change":"2024-03-06T07:28:40.927409+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927409+0000","last_undegraded":"2024-03-06T07:28:40.927409+0000","last_fullsized":"2024-03-06T07:28:40.927409+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922559+0000","last_change":"2024-03-06T07:28:40.922559+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922559+0000","last_undegraded":"2024-03-06T07:28:40.922559+0000","last_fullsized":"2024-03-06T07:28:40.922559+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923890+0000","last_change":"2024-03-06T07:28:40.923890+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923890+0000","last_undegraded":"2024-03-06T07:28:40.923890+0000","last_fullsized":"2024-03-06T07:28:40.923890+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.921998+0000","last_change":"2024-03-06T07:28:40.921998+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.921998+0000","last_undegraded":"2024-03-06T07:28:40.921998+0000","last_fullsized":"2024-03-06T07:28:40.921998+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923046+0000","last_change":"2024-03-06T07:28:40.923046+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923046+0000","last_undegraded":"2024-03-06T07:28:40.923046+0000","last_fullsized":"2024-03-06T07:28:40.923046+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.919747+0000","last_change":"2024-03-06T07:28:40.919747+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.919747+0000","last_undegraded":"2024-03-06T07:28:40.919747+0000","last_fullsized":"2024-03-06T07:28:40.919747+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927342+0000","last_change":"2024-03-06T07:28:40.927342+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927342+0000","last_undegraded":"2024-03-06T07:28:40.927342+0000","last_fullsized":"2024-03-06T07:28:40.927342+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922365+0000","last_change":"2024-03-06T07:28:40.922365+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922365+0000","last_undegraded":"2024-03-06T07:28:40.922365+0000","last_fullsized":"2024-03-06T07:28:40.922365+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922081+0000","last_change":"2024-03-06T07:28:40.922081+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922081+0000","last_undegraded":"2024-03-06T07:28:40.922081+0000","last_fullsized":"2024-03-06T07:28:40.922081+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927877+0000","last_change":"2024-03-06T07:28:40.927877+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927877+0000","last_undegraded":"2024-03-06T07:28:40.927877+0000","last_fullsized":"2024-03-06T07:28:40.927877+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922113+0000","last_change":"2024-03-06T07:28:40.922113+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922113+0000","last_undegraded":"2024-03-06T07:28:40.922113+0000","last_fullsized":"2024-03-06T07:28:40.922113+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922430+0000","last_change":"2024-03-06T07:28:40.922430+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922430+0000","last_undegraded":"2024-03-06T07:28:40.922430+0000","last_fullsized":"2024-03-06T07:28:40.922430+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.922580+0000","last_change":"2024-03-06T07:28:40.922580+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.922580+0000","last_undegraded":"2024-03-06T07:28:40.922580+0000","last_fullsized":"2024-03-06T07:28:40.922580+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923848+0000","last_change":"2024-03-06T07:28:40.923848+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923848+0000","last_undegraded":"2024-03-06T07:28:40.923848+0000","last_fullsized":"2024-03-06T07:28:40.923848+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924394+0000","last_change":"2024-03-06T07:28:40.924394+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924394+0000","last_undegraded":"2024-03-06T07:28:40.924394+0000","last_fullsized":"2024-03-06T07:28:40.924394+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.928144+0000","last_change":"2024-03-06T07:28:40.928144+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.928144+0000","last_undegraded":"2024-03-06T07:28:40.928144+0000","last_fullsized":"2024-03-06T07:28:40.928144+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.919708+0000","last_change":"2024-03-06T07:28:40.919708+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.919708+0000","last_undegraded":"2024-03-06T07:28:40.919708+0000","last_fullsized":"2024-03-06T07:28:40.919708+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.924586+0000","last_change":"2024-03-06T07:28:40.924586+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.924586+0000","last_undegraded":"2024-03-06T07:28:40.924586+0000","last_fullsized":"2024-03-06T07:28:40.924586+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.930002+0000","last_change":"2024-03-06T07:28:40.930002+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.930002+0000","last_undegraded":"2024-03-06T07:28:40.930002+0000","last_fullsized":"2024-03-06T07:28:40.930002+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927300+0000","last_change":"2024-03-06T07:28:40.927300+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927300+0000","last_undegraded":"2024-03-06T07:28:40.927300+0000","last_fullsized":"2024-03-06T07:28:40.927300+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927340+0000","last_change":"2024-03-06T07:28:40.927340+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927340+0000","last_undegraded":"2024-03-06T07:28:40.927340+0000","last_fullsized":"2024-03-06T07:28:40.927340+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.925257+0000","last_change":"2024-03-06T07:28:40.925257+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.925257+0000","last_undegraded":"2024-03-06T07:28:40.925257+0000","last_fullsized":"2024-03-06T07:28:40.925257+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.923956+0000","last_change":"2024-03-06T07:28:40.923956+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.923956+0000","last_undegraded":"2024-03-06T07:28:40.923956+0000","last_fullsized":"2024-03-06T07:28:40.923956+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.927584+0000","last_change":"2024-03-06T07:28:40.927584+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.927584+0000","last_undegraded":"2024-03-06T07:28:40.927584+0000","last_fullsized":"2024-03-06T07:28:40.927584+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:28:41.916963+0000","last_change":"2024-03-06T07:28:41.916963+0000","last_active":"2024-03-06T07:28:41.916963+0000","last_peered":"2024-03-06T07:28:41.916963+0000","last_clean":"2024-03-06T07:28:41.916963+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:41.916963+0000","last_undegraded":"2024-03-06T07:28:41.916963+0000","last_fullsized":"2024-03-06T07:28:41.916963+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":1,"reported_epoch":123,"state":"creating+peering","last_fresh":"2024-03-06T07:28:40.925204+0000","last_change":"2024-03-06T07:28:40.925204+0000","last_active":"2024-03-06T07:28:40.911886+0000","last_peered":"2024-03-06T07:28:40.911886+0000","last_clean":"2024-03-06T07:28:40.911886+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:40.925204+0000","last_undegraded":"2024-03-06T07:28:40.925204+0000","last_fullsized":"2024-03-06T07:28:40.925204+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":353,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.925605+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:28:40.925605+0000","last_peered":"2024-03-06T07:28:40.925605+0000","last_clean":"2024-03-06T07:28:40.925605+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:28:40.925605+0000","last_undegraded":"2024-03-06T07:28:40.925605+0000","last_fullsized":"2024-03-06T07:28:40.925605+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41572,"kb_used_data":2068,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330268,"statfs":{"total":96636764160,"available":96594194432,"internally_reserved":0,"allocated":2117632,"data_stored":1251263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40996,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330844,"statfs":{"total":96636764160,"available":96594784256,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40992,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330848,"statfs":{"total":96636764160,"available":96594788352,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1523712,"data_stored":660895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41000,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330840,"statfs":{"total":96636764160,"available":96594780160,"internally_reserved":0,"allocated":1536000,"data_stored":666712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41580,"kb_used_data":2080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330260,"statfs":{"total":96636764160,"available":96594186240,"internally_reserved":0,"allocated":2129920,"data_stored":1257080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:48.587 INFO:tasks.ceph.mds.c.smithi059.stderr:2024-03-06T07:28:48.585+0000 7f2a20acc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:50.401 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:50.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.501+0000 7f17acba2640 1 Processor -- start 2024-03-06T07:28:50.503 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.501+0000 7f17acba2640 1 -- start start 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17acba2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f17a8152bc0 0x7f17a80b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17acba2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17a80b7370 0x7f17a815c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17acba2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a80b2960 con 0x7f17a815c990 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17acba2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a80b77d0 con 0x7f17a8152bc0 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17acba2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a8152fa0 con 0x7f17a80b7370 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17a77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f17a8152bc0 0x7f17a80b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17a77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f17a8152bc0 0x7f17a80b6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48266/0 (socket says 172.21.15.59:48266) 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17a77fe640 1 -- 172.21.15.59:0/3243658783 learned_addr learned my addr 172.21.15.59:0/3243658783 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17a6ffd640 1 --2- 172.21.15.59:0/3243658783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17a80b7370 0x7f17a815c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17a77fe640 1 -- 172.21.15.59:0/3243658783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17a80b7370 msgr2=0x7f17a815c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17a77fe640 1 --2- 172.21.15.59:0/3243658783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17a80b7370 0x7f17a815c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17a77fe640 1 -- 172.21.15.59:0/3243658783 >> v1:172.21.15.164:6789/0 conn(0x7f17a815c990 legacy=0x7f17a815f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.502+0000 7f17a77fe640 1 -- 172.21.15.59:0/3243658783 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a80d2780 con 0x7f17a8152bc0 2024-03-06T07:28:50.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.503+0000 7f17a77fe640 1 --2- 172.21.15.59:0/3243658783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17a8152bc0 0x7f17a80b6e30 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f179c009870 tx=0x7f179c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94be5db4e8fd1970 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.503+0000 7f17a67fc640 1 -- 172.21.15.59:0/3243658783 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c034070 con 0x7f17a8152bc0 2024-03-06T07:28:50.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.503+0000 7f17a67fc640 1 -- 172.21.15.59:0/3243658783 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179c030840 con 0x7f17a8152bc0 2024-03-06T07:28:50.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.503+0000 7f17a67fc640 1 -- 172.21.15.59:0/3243658783 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c030c10 con 0x7f17a8152bc0 2024-03-06T07:28:50.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.504+0000 7f17acba2640 1 -- 172.21.15.59:0/3243658783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17a8152bc0 msgr2=0x7f17a80b6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.504+0000 7f17acba2640 1 --2- 172.21.15.59:0/3243658783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17a8152bc0 0x7f17a80b6e30 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f179c009870 tx=0x7f179c02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:50.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.504+0000 7f17acba2640 1 -- 172.21.15.59:0/3243658783 shutdown_connections 2024-03-06T07:28:50.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.504+0000 7f17acba2640 1 --2- 172.21.15.59:0/3243658783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17a80b7370 0x7f17a815c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.504+0000 7f17acba2640 1 --2- 172.21.15.59:0/3243658783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17a8152bc0 0x7f17a80b6e30 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.504+0000 7f17acba2640 1 -- 172.21.15.59:0/3243658783 >> 172.21.15.59:0/3243658783 conn(0x7f17a809a9f0 msgr2=0x7f17a809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.504+0000 7f17acba2640 1 -- 172.21.15.59:0/3243658783 shutdown_connections 2024-03-06T07:28:50.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.504+0000 7f17acba2640 1 -- 172.21.15.59:0/3243658783 wait complete. 2024-03-06T07:28:50.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.505+0000 7f17acba2640 1 Processor -- start 2024-03-06T07:28:50.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.505+0000 7f17acba2640 1 -- start start 2024-03-06T07:28:50.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17acba2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17a80b7370 0x7f17a815d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17acba2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17a8152bc0 0x7f17a813bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17acba2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17a815c990 0x7f17a813c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.509 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17acba2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80d1ca0 con 0x7f17a80b7370 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17acba2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80d19c0 con 0x7f17a8152bc0 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17acba2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a80d1f80 con 0x7f17a815c990 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17a80b7370 0x7f17a815d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17a80b7370 0x7f17a815d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48278/0 (socket says 172.21.15.59:48278) 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17a77fe640 1 -- 172.21.15.59:0/1090504701 learned_addr learned my addr 172.21.15.59:0/1090504701 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17a6ffd640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17a8152bc0 0x7f17a813bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17a7fff640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17a815c990 0x7f17a813c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17a77fe640 1 -- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17a815c990 msgr2=0x7f17a813c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17a77fe640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17a815c990 0x7f17a813c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17a77fe640 1 -- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17a8152bc0 msgr2=0x7f17a813bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17a77fe640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17a8152bc0 0x7f17a813bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17a77fe640 1 -- 172.21.15.59:0/1090504701 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a81605a0 con 0x7f17a80b7370 2024-03-06T07:28:50.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17a77fe640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17a80b7370 0x7f17a815d070 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f179c000c00 tx=0x7f179c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f178ffff640 1 -- 172.21.15.59:0/1090504701 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c034040 con 0x7f17a80b7370 2024-03-06T07:28:50.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f178ffff640 1 -- 172.21.15.59:0/1090504701 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179c039d90 con 0x7f17a80b7370 2024-03-06T07:28:50.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.506+0000 7f17acba2640 1 -- 172.21.15.59:0/1090504701 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a815ffe0 con 0x7f17a80b7370 2024-03-06T07:28:50.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.507+0000 7f178ffff640 1 -- 172.21.15.59:0/1090504701 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c03f700 con 0x7f17a80b7370 2024-03-06T07:28:50.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.507+0000 7f17acba2640 1 -- 172.21.15.59:0/1090504701 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a80cf4f0 con 0x7f17a80b7370 2024-03-06T07:28:50.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.508+0000 7f17acba2640 1 -- 172.21.15.59:0/1090504701 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a8068f80 con 0x7f17a80b7370 2024-03-06T07:28:50.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.508+0000 7f178ffff640 1 -- 172.21.15.59:0/1090504701 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f179c037070 con 0x7f17a80b7370 2024-03-06T07:28:50.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.508+0000 7f178ffff640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f179406b4d0 0x7f179406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.508+0000 7f178ffff640 1 -- 172.21.15.59:0/1090504701 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f179c0b06a0 con 0x7f17a80b7370 2024-03-06T07:28:50.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.509+0000 7f17a6ffd640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f179406b4d0 0x7f179406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.509+0000 7f17a6ffd640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f179406b4d0 0x7f179406d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f17a81f5cf0 tx=0x7f1798008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.513+0000 7f178ffff640 1 -- 172.21.15.59:0/1090504701 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f179c07f520 con 0x7f17a80b7370 2024-03-06T07:28:50.537 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:50.539 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.690+0000 7f17acba2640 1 -- 172.21.15.59:0/1090504701 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17a80b16f0 con 0x7f179406b4d0 2024-03-06T07:28:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.696+0000 7f178ffff640 1 -- 172.21.15.59:0/1090504701 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252481 (secure 0 0 0) 0x7f17a80b16f0 con 0x7f179406b4d0 2024-03-06T07:28:50.701 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:50.702 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:28:50.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 -- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f179406b4d0 msgr2=0x7f179406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f179406b4d0 0x7f179406d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f17a81f5cf0 tx=0x7f1798008040 comp rx=0 tx=0).stop 2024-03-06T07:28:50.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 -- 172.21.15.59:0/1090504701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17a80b7370 msgr2=0x7f17a815d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17a80b7370 0x7f17a815d070 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f179c000c00 tx=0x7f179c004730 comp rx=0 tx=0).stop 2024-03-06T07:28:50.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 -- 172.21.15.59:0/1090504701 shutdown_connections 2024-03-06T07:28:50.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f179406b4d0 0x7f179406d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17a815c990 0x7f17a813c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17a8152bc0 0x7f17a813bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 --2- 172.21.15.59:0/1090504701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17a80b7370 0x7f17a815d070 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 -- 172.21.15.59:0/1090504701 >> 172.21.15.59:0/1090504701 conn(0x7f17a809a9f0 msgr2=0x7f17a80af180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 -- 172.21.15.59:0/1090504701 shutdown_connections 2024-03-06T07:28:50.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.700+0000 7f17acba2640 1 -- 172.21.15.59:0/1090504701 wait complete. 2024-03-06T07:28:50.713 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-03-06T07:28:49.393039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329528,"kb_used_data":13512,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645192,"statfs":{"total":773094113280,"available":772756676608,"internally_reserved":0,"allocated":13836288,"data_stored":6674052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.465338"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.943027+0000","last_change":"2024-03-06T07:28:42.946046+0000","last_active":"2024-03-06T07:28:43.943027+0000","last_peered":"2024-03-06T07:28:43.943027+0000","last_clean":"2024-03-06T07:28:43.943027+0000","last_became_active":"2024-03-06T07:28:42.945806+0000","last_became_peered":"2024-03-06T07:28:42.945806+0000","last_unstale":"2024-03-06T07:28:43.943027+0000","last_undegraded":"2024-03-06T07:28:43.943027+0000","last_fullsized":"2024-03-06T07:28:43.943027+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:07:55.656621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610111+0000","last_change":"2024-03-06T07:28:42.942656+0000","last_active":"2024-03-06T07:28:45.610111+0000","last_peered":"2024-03-06T07:28:45.610111+0000","last_clean":"2024-03-06T07:28:45.610111+0000","last_became_active":"2024-03-06T07:28:42.942209+0000","last_became_peered":"2024-03-06T07:28:42.942209+0000","last_unstale":"2024-03-06T07:28:45.610111+0000","last_undegraded":"2024-03-06T07:28:45.610111+0000","last_fullsized":"2024-03-06T07:28:45.610111+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:53.478946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610644+0000","last_change":"2024-03-06T07:28:42.941733+0000","last_active":"2024-03-06T07:28:45.610644+0000","last_peered":"2024-03-06T07:28:45.610644+0000","last_clean":"2024-03-06T07:28:45.610644+0000","last_became_active":"2024-03-06T07:28:42.941451+0000","last_became_peered":"2024-03-06T07:28:42.941451+0000","last_unstale":"2024-03-06T07:28:45.610644+0000","last_undegraded":"2024-03-06T07:28:45.610644+0000","last_fullsized":"2024-03-06T07:28:45.610644+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:13.513769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.942915+0000","last_change":"2024-03-06T07:28:42.944683+0000","last_active":"2024-03-06T07:28:43.942915+0000","last_peered":"2024-03-06T07:28:43.942915+0000","last_clean":"2024-03-06T07:28:43.942915+0000","last_became_active":"2024-03-06T07:28:42.944417+0000","last_became_peered":"2024-03-06T07:28:42.944417+0000","last_unstale":"2024-03-06T07:28:43.942915+0000","last_undegraded":"2024-03-06T07:28:43.942915+0000","last_fullsized":"2024-03-06T07:28:43.942915+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:24.710043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199696+0000","last_change":"2024-03-06T07:28:42.946386+0000","last_active":"2024-03-06T07:28:44.199696+0000","last_peered":"2024-03-06T07:28:44.199696+0000","last_clean":"2024-03-06T07:28:44.199696+0000","last_became_active":"2024-03-06T07:28:42.946119+0000","last_became_peered":"2024-03-06T07:28:42.946119+0000","last_unstale":"2024-03-06T07:28:44.199696+0000","last_undegraded":"2024-03-06T07:28:44.199696+0000","last_fullsized":"2024-03-06T07:28:44.199696+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:55.491399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.930565+0000","last_change":"2024-03-06T07:28:42.954935+0000","last_active":"2024-03-06T07:28:44.930565+0000","last_peered":"2024-03-06T07:28:44.930565+0000","last_clean":"2024-03-06T07:28:44.930565+0000","last_became_active":"2024-03-06T07:28:42.954664+0000","last_became_peered":"2024-03-06T07:28:42.954664+0000","last_unstale":"2024-03-06T07:28:44.930565+0000","last_undegraded":"2024-03-06T07:28:44.930565+0000","last_fullsized":"2024-03-06T07:28:44.930565+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:42:46.264526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.930375+0000","last_change":"2024-03-06T07:28:42.948783+0000","last_active":"2024-03-06T07:28:44.930375+0000","last_peered":"2024-03-06T07:28:44.930375+0000","last_clean":"2024-03-06T07:28:44.930375+0000","last_became_active":"2024-03-06T07:28:42.948510+0000","last_became_peered":"2024-03-06T07:28:42.948510+0000","last_unstale":"2024-03-06T07:28:44.930375+0000","last_undegraded":"2024-03-06T07:28:44.930375+0000","last_fullsized":"2024-03-06T07:28:44.930375+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:27.123907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610611+0000","last_change":"2024-03-06T07:28:42.942685+0000","last_active":"2024-03-06T07:28:45.610611+0000","last_peered":"2024-03-06T07:28:45.610611+0000","last_clean":"2024-03-06T07:28:45.610611+0000","last_became_active":"2024-03-06T07:28:42.942403+0000","last_became_peered":"2024-03-06T07:28:42.942403+0000","last_unstale":"2024-03-06T07:28:45.610611+0000","last_undegraded":"2024-03-06T07:28:45.610611+0000","last_fullsized":"2024-03-06T07:28:45.610611+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:19.011651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610771+0000","last_change":"2024-03-06T07:28:42.938264+0000","last_active":"2024-03-06T07:28:45.610771+0000","last_peered":"2024-03-06T07:28:45.610771+0000","last_clean":"2024-03-06T07:28:45.610771+0000","last_became_active":"2024-03-06T07:28:42.937935+0000","last_became_peered":"2024-03-06T07:28:42.937935+0000","last_unstale":"2024-03-06T07:28:45.610771+0000","last_undegraded":"2024-03-06T07:28:45.610771+0000","last_fullsized":"2024-03-06T07:28:45.610771+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:55.463154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610158+0000","last_change":"2024-03-06T07:28:42.942454+0000","last_active":"2024-03-06T07:28:45.610158+0000","last_peered":"2024-03-06T07:28:45.610158+0000","last_clean":"2024-03-06T07:28:45.610158+0000","last_became_active":"2024-03-06T07:28:42.941860+0000","last_became_peered":"2024-03-06T07:28:42.941860+0000","last_unstale":"2024-03-06T07:28:45.610158+0000","last_undegraded":"2024-03-06T07:28:45.610158+0000","last_fullsized":"2024-03-06T07:28:45.610158+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:05.726097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610268+0000","last_change":"2024-03-06T07:28:42.954436+0000","last_active":"2024-03-06T07:28:45.610268+0000","last_peered":"2024-03-06T07:28:45.610268+0000","last_clean":"2024-03-06T07:28:45.610268+0000","last_became_active":"2024-03-06T07:28:42.954136+0000","last_became_peered":"2024-03-06T07:28:42.954136+0000","last_unstale":"2024-03-06T07:28:45.610268+0000","last_undegraded":"2024-03-06T07:28:45.610268+0000","last_fullsized":"2024-03-06T07:28:45.610268+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:31.682373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610529+0000","last_change":"2024-03-06T07:28:42.934871+0000","last_active":"2024-03-06T07:28:45.610529+0000","last_peered":"2024-03-06T07:28:45.610529+0000","last_clean":"2024-03-06T07:28:45.610529+0000","last_became_active":"2024-03-06T07:28:42.934566+0000","last_became_peered":"2024-03-06T07:28:42.934566+0000","last_unstale":"2024-03-06T07:28:45.610529+0000","last_undegraded":"2024-03-06T07:28:45.610529+0000","last_fullsized":"2024-03-06T07:28:45.610529+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:21.933590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.943147+0000","last_change":"2024-03-06T07:28:42.945441+0000","last_active":"2024-03-06T07:28:43.943147+0000","last_peered":"2024-03-06T07:28:43.943147+0000","last_clean":"2024-03-06T07:28:43.943147+0000","last_became_active":"2024-03-06T07:28:42.945167+0000","last_became_peered":"2024-03-06T07:28:42.945167+0000","last_unstale":"2024-03-06T07:28:43.943147+0000","last_undegraded":"2024-03-06T07:28:43.943147+0000","last_fullsized":"2024-03-06T07:28:43.943147+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:06.714739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610766+0000","last_change":"2024-03-06T07:28:42.942897+0000","last_active":"2024-03-06T07:28:45.610766+0000","last_peered":"2024-03-06T07:28:45.610766+0000","last_clean":"2024-03-06T07:28:45.610766+0000","last_became_active":"2024-03-06T07:28:42.938308+0000","last_became_peered":"2024-03-06T07:28:42.938308+0000","last_unstale":"2024-03-06T07:28:45.610766+0000","last_undegraded":"2024-03-06T07:28:45.610766+0000","last_fullsized":"2024-03-06T07:28:45.610766+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:42.374144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.942528+0000","last_change":"2024-03-06T07:28:42.950389+0000","last_active":"2024-03-06T07:28:43.942528+0000","last_peered":"2024-03-06T07:28:43.942528+0000","last_clean":"2024-03-06T07:28:43.942528+0000","last_became_active":"2024-03-06T07:28:42.949856+0000","last_became_peered":"2024-03-06T07:28:42.949856+0000","last_unstale":"2024-03-06T07:28:43.942528+0000","last_undegraded":"2024-03-06T07:28:43.942528+0000","last_fullsized":"2024-03-06T07:28:43.942528+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:33.497142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.945619+0000","last_change":"2024-03-06T07:28:42.954807+0000","last_active":"2024-03-06T07:28:43.945619+0000","last_peered":"2024-03-06T07:28:43.945619+0000","last_clean":"2024-03-06T07:28:43.945619+0000","last_became_active":"2024-03-06T07:28:42.954533+0000","last_became_peered":"2024-03-06T07:28:42.954533+0000","last_unstale":"2024-03-06T07:28:43.945619+0000","last_undegraded":"2024-03-06T07:28:43.945619+0000","last_fullsized":"2024-03-06T07:28:43.945619+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:25.781346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.930735+0000","last_change":"2024-03-06T07:28:42.951097+0000","last_active":"2024-03-06T07:28:44.930735+0000","last_peered":"2024-03-06T07:28:44.930735+0000","last_clean":"2024-03-06T07:28:44.930735+0000","last_became_active":"2024-03-06T07:28:42.950765+0000","last_became_peered":"2024-03-06T07:28:42.950765+0000","last_unstale":"2024-03-06T07:28:44.930735+0000","last_undegraded":"2024-03-06T07:28:44.930735+0000","last_fullsized":"2024-03-06T07:28:44.930735+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:13.004418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.945881+0000","last_change":"2024-03-06T07:28:42.942198+0000","last_active":"2024-03-06T07:28:43.945881+0000","last_peered":"2024-03-06T07:28:43.945881+0000","last_clean":"2024-03-06T07:28:43.945881+0000","last_became_active":"2024-03-06T07:28:42.941895+0000","last_became_peered":"2024-03-06T07:28:42.941895+0000","last_unstale":"2024-03-06T07:28:43.945881+0000","last_undegraded":"2024-03-06T07:28:43.945881+0000","last_fullsized":"2024-03-06T07:28:43.945881+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:47.885401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610412+0000","last_change":"2024-03-06T07:28:42.955562+0000","last_active":"2024-03-06T07:28:45.610412+0000","last_peered":"2024-03-06T07:28:45.610412+0000","last_clean":"2024-03-06T07:28:45.610412+0000","last_became_active":"2024-03-06T07:28:42.955299+0000","last_became_peered":"2024-03-06T07:28:42.955299+0000","last_unstale":"2024-03-06T07:28:45.610412+0000","last_undegraded":"2024-03-06T07:28:45.610412+0000","last_fullsized":"2024-03-06T07:28:45.610412+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:58.660364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.945692+0000","last_change":"2024-03-06T07:28:42.941907+0000","last_active":"2024-03-06T07:28:43.945692+0000","last_peered":"2024-03-06T07:28:43.945692+0000","last_clean":"2024-03-06T07:28:43.945692+0000","last_became_active":"2024-03-06T07:28:42.941640+0000","last_became_peered":"2024-03-06T07:28:42.941640+0000","last_unstale":"2024-03-06T07:28:43.945692+0000","last_undegraded":"2024-03-06T07:28:43.945692+0000","last_fullsized":"2024-03-06T07:28:43.945692+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:36.323438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199548+0000","last_change":"2024-03-06T07:28:42.947070+0000","last_active":"2024-03-06T07:28:44.199548+0000","last_peered":"2024-03-06T07:28:44.199548+0000","last_clean":"2024-03-06T07:28:44.199548+0000","last_became_active":"2024-03-06T07:28:42.946764+0000","last_became_peered":"2024-03-06T07:28:42.946764+0000","last_unstale":"2024-03-06T07:28:44.199548+0000","last_undegraded":"2024-03-06T07:28:44.199548+0000","last_fullsized":"2024-03-06T07:28:44.199548+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:54.936513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.609923+0000","last_change":"2024-03-06T07:28:42.954861+0000","last_active":"2024-03-06T07:28:45.609923+0000","last_peered":"2024-03-06T07:28:45.609923+0000","last_clean":"2024-03-06T07:28:45.609923+0000","last_became_active":"2024-03-06T07:28:42.954474+0000","last_became_peered":"2024-03-06T07:28:42.954474+0000","last_unstale":"2024-03-06T07:28:45.609923+0000","last_undegraded":"2024-03-06T07:28:45.609923+0000","last_fullsized":"2024-03-06T07:28:45.609923+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:39.809637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610548+0000","last_change":"2024-03-06T07:28:42.935342+0000","last_active":"2024-03-06T07:28:45.610548+0000","last_peered":"2024-03-06T07:28:45.610548+0000","last_clean":"2024-03-06T07:28:45.610548+0000","last_became_active":"2024-03-06T07:28:42.935060+0000","last_became_peered":"2024-03-06T07:28:42.935060+0000","last_unstale":"2024-03-06T07:28:45.610548+0000","last_undegraded":"2024-03-06T07:28:45.610548+0000","last_fullsized":"2024-03-06T07:28:45.610548+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:35:08.386109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.609919+0000","last_change":"2024-03-06T07:28:42.948267+0000","last_active":"2024-03-06T07:28:45.609919+0000","last_peered":"2024-03-06T07:28:45.609919+0000","last_clean":"2024-03-06T07:28:45.609919+0000","last_became_active":"2024-03-06T07:28:42.947028+0000","last_became_peered":"2024-03-06T07:28:42.947028+0000","last_unstale":"2024-03-06T07:28:45.609919+0000","last_undegraded":"2024-03-06T07:28:45.609919+0000","last_fullsized":"2024-03-06T07:28:45.609919+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:37.284827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610267+0000","last_change":"2024-03-06T07:28:42.946091+0000","last_active":"2024-03-06T07:28:45.610267+0000","last_peered":"2024-03-06T07:28:45.610267+0000","last_clean":"2024-03-06T07:28:45.610267+0000","last_became_active":"2024-03-06T07:28:42.945562+0000","last_became_peered":"2024-03-06T07:28:42.945562+0000","last_unstale":"2024-03-06T07:28:45.610267+0000","last_undegraded":"2024-03-06T07:28:45.610267+0000","last_fullsized":"2024-03-06T07:28:45.610267+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:01.635389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.946031+0000","last_change":"2024-03-06T07:28:42.940977+0000","last_active":"2024-03-06T07:28:43.946031+0000","last_peered":"2024-03-06T07:28:43.946031+0000","last_clean":"2024-03-06T07:28:43.946031+0000","last_became_active":"2024-03-06T07:28:42.940703+0000","last_became_peered":"2024-03-06T07:28:42.940703+0000","last_unstale":"2024-03-06T07:28:43.946031+0000","last_undegraded":"2024-03-06T07:28:43.946031+0000","last_fullsized":"2024-03-06T07:28:43.946031+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:32.775434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.943076+0000","last_change":"2024-03-06T07:28:42.944716+0000","last_active":"2024-03-06T07:28:43.943076+0000","last_peered":"2024-03-06T07:28:43.943076+0000","last_clean":"2024-03-06T07:28:43.943076+0000","last_became_active":"2024-03-06T07:28:42.944465+0000","last_became_peered":"2024-03-06T07:28:42.944465+0000","last_unstale":"2024-03-06T07:28:43.943076+0000","last_undegraded":"2024-03-06T07:28:43.943076+0000","last_fullsized":"2024-03-06T07:28:43.943076+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:49:43.118441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.948869+0000","last_change":"2024-03-06T07:28:42.943175+0000","last_active":"2024-03-06T07:28:43.948869+0000","last_peered":"2024-03-06T07:28:43.948869+0000","last_clean":"2024-03-06T07:28:43.948869+0000","last_became_active":"2024-03-06T07:28:42.942881+0000","last_became_peered":"2024-03-06T07:28:42.942881+0000","last_unstale":"2024-03-06T07:28:43.948869+0000","last_undegraded":"2024-03-06T07:28:43.948869+0000","last_fullsized":"2024-03-06T07:28:43.948869+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:26.318871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.943139+0000","last_change":"2024-03-06T07:28:42.935821+0000","last_active":"2024-03-06T07:28:43.943139+0000","last_peered":"2024-03-06T07:28:43.943139+0000","last_clean":"2024-03-06T07:28:43.943139+0000","last_became_active":"2024-03-06T07:28:42.935462+0000","last_became_peered":"2024-03-06T07:28:42.935462+0000","last_unstale":"2024-03-06T07:28:43.943139+0000","last_undegraded":"2024-03-06T07:28:43.943139+0000","last_fullsized":"2024-03-06T07:28:43.943139+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:14.802780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.942652+0000","last_change":"2024-03-06T07:28:42.936875+0000","last_active":"2024-03-06T07:28:43.942652+0000","last_peered":"2024-03-06T07:28:43.942652+0000","last_clean":"2024-03-06T07:28:43.942652+0000","last_became_active":"2024-03-06T07:28:42.936613+0000","last_became_peered":"2024-03-06T07:28:42.936613+0000","last_unstale":"2024-03-06T07:28:43.942652+0000","last_undegraded":"2024-03-06T07:28:43.942652+0000","last_fullsized":"2024-03-06T07:28:43.942652+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:28.994008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199623+0000","last_change":"2024-03-06T07:28:42.952638+0000","last_active":"2024-03-06T07:28:44.199623+0000","last_peered":"2024-03-06T07:28:44.199623+0000","last_clean":"2024-03-06T07:28:44.199623+0000","last_became_active":"2024-03-06T07:28:42.951965+0000","last_became_peered":"2024-03-06T07:28:42.951965+0000","last_unstale":"2024-03-06T07:28:44.199623+0000","last_undegraded":"2024-03-06T07:28:44.199623+0000","last_fullsized":"2024-03-06T07:28:44.199623+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:39.794675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.930336+0000","last_change":"2024-03-06T07:28:42.943687+0000","last_active":"2024-03-06T07:28:44.930336+0000","last_peered":"2024-03-06T07:28:44.930336+0000","last_clean":"2024-03-06T07:28:44.930336+0000","last_became_active":"2024-03-06T07:28:42.943428+0000","last_became_peered":"2024-03-06T07:28:42.943428+0000","last_unstale":"2024-03-06T07:28:44.930336+0000","last_undegraded":"2024-03-06T07:28:44.930336+0000","last_fullsized":"2024-03-06T07:28:44.930336+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:45.969174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.946018+0000","last_change":"2024-03-06T07:28:42.939561+0000","last_active":"2024-03-06T07:28:43.946018+0000","last_peered":"2024-03-06T07:28:43.946018+0000","last_clean":"2024-03-06T07:28:43.946018+0000","last_became_active":"2024-03-06T07:28:42.939270+0000","last_became_peered":"2024-03-06T07:28:42.939270+0000","last_unstale":"2024-03-06T07:28:43.946018+0000","last_undegraded":"2024-03-06T07:28:43.946018+0000","last_fullsized":"2024-03-06T07:28:43.946018+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:52.661554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"125'4","reported_seq":18,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199907+0000","last_change":"2024-03-06T07:28:41.944607+0000","last_active":"2024-03-06T07:28:44.199907+0000","last_peered":"2024-03-06T07:28:44.199907+0000","last_clean":"2024-03-06T07:28:44.199907+0000","last_became_active":"2024-03-06T07:28:41.943834+0000","last_became_peered":"2024-03-06T07:28:41.943834+0000","last_unstale":"2024-03-06T07:28:44.199907+0000","last_undegraded":"2024-03-06T07:28:44.199907+0000","last_fullsized":"2024-03-06T07:28:44.199907+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:58.546625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610809+0000","last_change":"2024-03-06T07:28:42.932968+0000","last_active":"2024-03-06T07:28:45.610809+0000","last_peered":"2024-03-06T07:28:45.610809+0000","last_clean":"2024-03-06T07:28:45.610809+0000","last_became_active":"2024-03-06T07:28:42.932765+0000","last_became_peered":"2024-03-06T07:28:42.932765+0000","last_unstale":"2024-03-06T07:28:45.610809+0000","last_undegraded":"2024-03-06T07:28:45.610809+0000","last_fullsized":"2024-03-06T07:28:45.610809+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:28:56.306828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"125'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.948806+0000","last_change":"2024-03-06T07:28:41.944778+0000","last_active":"2024-03-06T07:28:43.948806+0000","last_peered":"2024-03-06T07:28:43.948806+0000","last_clean":"2024-03-06T07:28:43.948806+0000","last_became_active":"2024-03-06T07:28:41.944281+0000","last_became_peered":"2024-03-06T07:28:41.944281+0000","last_unstale":"2024-03-06T07:28:43.948806+0000","last_undegraded":"2024-03-06T07:28:43.948806+0000","last_fullsized":"2024-03-06T07:28:43.948806+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:18:21.650661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199796+0000","last_change":"2024-03-06T07:28:42.947656+0000","last_active":"2024-03-06T07:28:44.199796+0000","last_peered":"2024-03-06T07:28:44.199796+0000","last_clean":"2024-03-06T07:28:44.199796+0000","last_became_active":"2024-03-06T07:28:42.947393+0000","last_became_peered":"2024-03-06T07:28:42.947393+0000","last_unstale":"2024-03-06T07:28:44.199796+0000","last_undegraded":"2024-03-06T07:28:44.199796+0000","last_fullsized":"2024-03-06T07:28:44.199796+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:11.391481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610464+0000","last_change":"2024-03-06T07:28:41.946726+0000","last_active":"2024-03-06T07:28:45.610464+0000","last_peered":"2024-03-06T07:28:45.610464+0000","last_clean":"2024-03-06T07:28:45.610464+0000","last_became_active":"2024-03-06T07:28:41.946432+0000","last_became_peered":"2024-03-06T07:28:41.946432+0000","last_unstale":"2024-03-06T07:28:45.610464+0000","last_undegraded":"2024-03-06T07:28:45.610464+0000","last_fullsized":"2024-03-06T07:28:45.610464+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:20:51.656476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199930+0000","last_change":"2024-03-06T07:28:42.946394+0000","last_active":"2024-03-06T07:28:44.199930+0000","last_peered":"2024-03-06T07:28:44.199930+0000","last_clean":"2024-03-06T07:28:44.199930+0000","last_became_active":"2024-03-06T07:28:42.946140+0000","last_became_peered":"2024-03-06T07:28:42.946140+0000","last_unstale":"2024-03-06T07:28:44.199930+0000","last_undegraded":"2024-03-06T07:28:44.199930+0000","last_fullsized":"2024-03-06T07:28:44.199930+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:09.037569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610589+0000","last_change":"2024-03-06T07:28:41.946871+0000","last_active":"2024-03-06T07:28:45.610589+0000","last_peered":"2024-03-06T07:28:45.610589+0000","last_clean":"2024-03-06T07:28:45.610589+0000","last_became_active":"2024-03-06T07:28:41.946379+0000","last_became_peered":"2024-03-06T07:28:41.946379+0000","last_unstale":"2024-03-06T07:28:45.610589+0000","last_undegraded":"2024-03-06T07:28:45.610589+0000","last_fullsized":"2024-03-06T07:28:45.610589+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:11.817315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.945486+0000","last_change":"2024-03-06T07:28:42.942931+0000","last_active":"2024-03-06T07:28:43.945486+0000","last_peered":"2024-03-06T07:28:43.945486+0000","last_clean":"2024-03-06T07:28:43.945486+0000","last_became_active":"2024-03-06T07:28:42.942666+0000","last_became_peered":"2024-03-06T07:28:42.942666+0000","last_unstale":"2024-03-06T07:28:43.945486+0000","last_undegraded":"2024-03-06T07:28:43.945486+0000","last_fullsized":"2024-03-06T07:28:43.945486+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:29:29.841061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"125'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610241+0000","last_change":"2024-03-06T07:28:41.932502+0000","last_active":"2024-03-06T07:28:45.610241+0000","last_peered":"2024-03-06T07:28:45.610241+0000","last_clean":"2024-03-06T07:28:45.610241+0000","last_became_active":"2024-03-06T07:28:41.932152+0000","last_became_peered":"2024-03-06T07:28:41.932152+0000","last_unstale":"2024-03-06T07:28:45.610241+0000","last_undegraded":"2024-03-06T07:28:45.610241+0000","last_fullsized":"2024-03-06T07:28:45.610241+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:58.235232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.930642+0000","last_change":"2024-03-06T07:28:42.950759+0000","last_active":"2024-03-06T07:28:44.930642+0000","last_peered":"2024-03-06T07:28:44.930642+0000","last_clean":"2024-03-06T07:28:44.930642+0000","last_became_active":"2024-03-06T07:28:42.950419+0000","last_became_peered":"2024-03-06T07:28:42.950419+0000","last_unstale":"2024-03-06T07:28:44.930642+0000","last_undegraded":"2024-03-06T07:28:44.930642+0000","last_fullsized":"2024-03-06T07:28:44.930642+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:06:03.292500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"125'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.945367+0000","last_change":"2024-03-06T07:28:41.932348+0000","last_active":"2024-03-06T07:28:43.945367+0000","last_peered":"2024-03-06T07:28:43.945367+0000","last_clean":"2024-03-06T07:28:43.945367+0000","last_became_active":"2024-03-06T07:28:41.932044+0000","last_became_peered":"2024-03-06T07:28:41.932044+0000","last_unstale":"2024-03-06T07:28:43.945367+0000","last_undegraded":"2024-03-06T07:28:43.945367+0000","last_fullsized":"2024-03-06T07:28:43.945367+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:29.773637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610509+0000","last_change":"2024-03-06T07:28:42.939132+0000","last_active":"2024-03-06T07:28:45.610509+0000","last_peered":"2024-03-06T07:28:45.610509+0000","last_clean":"2024-03-06T07:28:45.610509+0000","last_became_active":"2024-03-06T07:28:42.938656+0000","last_became_peered":"2024-03-06T07:28:42.938656+0000","last_unstale":"2024-03-06T07:28:45.610509+0000","last_undegraded":"2024-03-06T07:28:45.610509+0000","last_fullsized":"2024-03-06T07:28:45.610509+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:36.440572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.930182+0000","last_change":"2024-03-06T07:28:41.939877+0000","last_active":"2024-03-06T07:28:44.930182+0000","last_peered":"2024-03-06T07:28:44.930182+0000","last_clean":"2024-03-06T07:28:44.930182+0000","last_became_active":"2024-03-06T07:28:41.939614+0000","last_became_peered":"2024-03-06T07:28:41.939614+0000","last_unstale":"2024-03-06T07:28:44.930182+0000","last_undegraded":"2024-03-06T07:28:44.930182+0000","last_fullsized":"2024-03-06T07:28:44.930182+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:08.197735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610050+0000","last_change":"2024-03-06T07:28:42.947772+0000","last_active":"2024-03-06T07:28:45.610050+0000","last_peered":"2024-03-06T07:28:45.610050+0000","last_clean":"2024-03-06T07:28:45.610050+0000","last_became_active":"2024-03-06T07:28:42.947310+0000","last_became_peered":"2024-03-06T07:28:42.947310+0000","last_unstale":"2024-03-06T07:28:45.610050+0000","last_undegraded":"2024-03-06T07:28:45.610050+0000","last_fullsized":"2024-03-06T07:28:45.610050+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:29.192625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.609945+0000","last_change":"2024-03-06T07:28:41.939704+0000","last_active":"2024-03-06T07:28:45.609945+0000","last_peered":"2024-03-06T07:28:45.609945+0000","last_clean":"2024-03-06T07:28:45.609945+0000","last_became_active":"2024-03-06T07:28:41.939399+0000","last_became_peered":"2024-03-06T07:28:41.939399+0000","last_unstale":"2024-03-06T07:28:45.609945+0000","last_undegraded":"2024-03-06T07:28:45.609945+0000","last_fullsized":"2024-03-06T07:28:45.609945+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:00.164785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.943146+0000","last_change":"2024-03-06T07:28:42.952268+0000","last_active":"2024-03-06T07:28:43.943146+0000","last_peered":"2024-03-06T07:28:43.943146+0000","last_clean":"2024-03-06T07:28:43.943146+0000","last_became_active":"2024-03-06T07:28:42.951306+0000","last_became_peered":"2024-03-06T07:28:42.951306+0000","last_unstale":"2024-03-06T07:28:43.943146+0000","last_undegraded":"2024-03-06T07:28:43.943146+0000","last_fullsized":"2024-03-06T07:28:43.943146+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:26.193077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.949110+0000","last_change":"2024-03-06T07:28:41.937340+0000","last_active":"2024-03-06T07:28:43.949110+0000","last_peered":"2024-03-06T07:28:43.949110+0000","last_clean":"2024-03-06T07:28:43.949110+0000","last_became_active":"2024-03-06T07:28:41.937014+0000","last_became_peered":"2024-03-06T07:28:41.937014+0000","last_unstale":"2024-03-06T07:28:43.949110+0000","last_undegraded":"2024-03-06T07:28:43.949110+0000","last_fullsized":"2024-03-06T07:28:43.949110+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:16.781497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.949048+0000","last_change":"2024-03-06T07:28:42.939387+0000","last_active":"2024-03-06T07:28:43.949048+0000","last_peered":"2024-03-06T07:28:43.949048+0000","last_clean":"2024-03-06T07:28:43.949048+0000","last_became_active":"2024-03-06T07:28:42.938412+0000","last_became_peered":"2024-03-06T07:28:42.938412+0000","last_unstale":"2024-03-06T07:28:43.949048+0000","last_undegraded":"2024-03-06T07:28:43.949048+0000","last_fullsized":"2024-03-06T07:28:43.949048+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:50:29.325469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.946047+0000","last_change":"2024-03-06T07:28:41.935901+0000","last_active":"2024-03-06T07:28:43.946047+0000","last_peered":"2024-03-06T07:28:43.946047+0000","last_clean":"2024-03-06T07:28:43.946047+0000","last_became_active":"2024-03-06T07:28:41.935614+0000","last_became_peered":"2024-03-06T07:28:41.935614+0000","last_unstale":"2024-03-06T07:28:43.946047+0000","last_undegraded":"2024-03-06T07:28:43.946047+0000","last_fullsized":"2024-03-06T07:28:43.946047+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:59.407826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199774+0000","last_change":"2024-03-06T07:28:42.947840+0000","last_active":"2024-03-06T07:28:44.199774+0000","last_peered":"2024-03-06T07:28:44.199774+0000","last_clean":"2024-03-06T07:28:44.199774+0000","last_became_active":"2024-03-06T07:28:42.947576+0000","last_became_peered":"2024-03-06T07:28:42.947576+0000","last_unstale":"2024-03-06T07:28:44.199774+0000","last_undegraded":"2024-03-06T07:28:44.199774+0000","last_fullsized":"2024-03-06T07:28:44.199774+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:32.031181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"125'3","reported_seq":17,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.948626+0000","last_change":"2024-03-06T07:28:41.938256+0000","last_active":"2024-03-06T07:28:43.948626+0000","last_peered":"2024-03-06T07:28:43.948626+0000","last_clean":"2024-03-06T07:28:43.948626+0000","last_became_active":"2024-03-06T07:28:41.937965+0000","last_became_peered":"2024-03-06T07:28:41.937965+0000","last_unstale":"2024-03-06T07:28:43.948626+0000","last_undegraded":"2024-03-06T07:28:43.948626+0000","last_fullsized":"2024-03-06T07:28:43.948626+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:52.869278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199974+0000","last_change":"2024-03-06T07:28:42.945762+0000","last_active":"2024-03-06T07:28:44.199974+0000","last_peered":"2024-03-06T07:28:44.199974+0000","last_clean":"2024-03-06T07:28:44.199974+0000","last_became_active":"2024-03-06T07:28:42.945508+0000","last_became_peered":"2024-03-06T07:28:42.945508+0000","last_unstale":"2024-03-06T07:28:44.199974+0000","last_undegraded":"2024-03-06T07:28:44.199974+0000","last_fullsized":"2024-03-06T07:28:44.199974+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:29:29.841061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"125'6","reported_seq":20,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610380+0000","last_change":"2024-03-06T07:28:41.941916+0000","last_active":"2024-03-06T07:28:45.610380+0000","last_peered":"2024-03-06T07:28:45.610380+0000","last_clean":"2024-03-06T07:28:45.610380+0000","last_became_active":"2024-03-06T07:28:41.941625+0000","last_became_peered":"2024-03-06T07:28:41.941625+0000","last_unstale":"2024-03-06T07:28:45.610380+0000","last_undegraded":"2024-03-06T07:28:45.610380+0000","last_fullsized":"2024-03-06T07:28:45.610380+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:44.506896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.943275+0000","last_change":"2024-03-06T07:28:42.953210+0000","last_active":"2024-03-06T07:28:43.943275+0000","last_peered":"2024-03-06T07:28:43.943275+0000","last_clean":"2024-03-06T07:28:43.943275+0000","last_became_active":"2024-03-06T07:28:42.952713+0000","last_became_peered":"2024-03-06T07:28:42.952713+0000","last_unstale":"2024-03-06T07:28:43.943275+0000","last_undegraded":"2024-03-06T07:28:43.943275+0000","last_fullsized":"2024-03-06T07:28:43.943275+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:27.734941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.942700+0000","last_change":"2024-03-06T07:28:41.941538+0000","last_active":"2024-03-06T07:28:43.942700+0000","last_peered":"2024-03-06T07:28:43.942700+0000","last_clean":"2024-03-06T07:28:43.942700+0000","last_became_active":"2024-03-06T07:28:41.941272+0000","last_became_peered":"2024-03-06T07:28:41.941272+0000","last_unstale":"2024-03-06T07:28:43.942700+0000","last_undegraded":"2024-03-06T07:28:43.942700+0000","last_fullsized":"2024-03-06T07:28:43.942700+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:54.640222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.609889+0000","last_change":"2024-03-06T07:28:42.948559+0000","last_active":"2024-03-06T07:28:45.609889+0000","last_peered":"2024-03-06T07:28:45.609889+0000","last_clean":"2024-03-06T07:28:45.609889+0000","last_became_active":"2024-03-06T07:28:42.948307+0000","last_became_peered":"2024-03-06T07:28:42.948307+0000","last_unstale":"2024-03-06T07:28:45.609889+0000","last_undegraded":"2024-03-06T07:28:45.609889+0000","last_fullsized":"2024-03-06T07:28:45.609889+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:09.202812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610430+0000","last_change":"2024-03-06T07:28:41.944004+0000","last_active":"2024-03-06T07:28:45.610430+0000","last_peered":"2024-03-06T07:28:45.610430+0000","last_clean":"2024-03-06T07:28:45.610430+0000","last_became_active":"2024-03-06T07:28:41.943707+0000","last_became_peered":"2024-03-06T07:28:41.943707+0000","last_unstale":"2024-03-06T07:28:45.610430+0000","last_undegraded":"2024-03-06T07:28:45.610430+0000","last_fullsized":"2024-03-06T07:28:45.610430+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:50.623803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610440+0000","last_change":"2024-03-06T07:28:42.933332+0000","last_active":"2024-03-06T07:28:45.610440+0000","last_peered":"2024-03-06T07:28:45.610440+0000","last_clean":"2024-03-06T07:28:45.610440+0000","last_became_active":"2024-03-06T07:28:42.933060+0000","last_became_peered":"2024-03-06T07:28:42.933060+0000","last_unstale":"2024-03-06T07:28:45.610440+0000","last_undegraded":"2024-03-06T07:28:45.610440+0000","last_fullsized":"2024-03-06T07:28:45.610440+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:07.994078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"125'8","reported_seq":22,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610723+0000","last_change":"2024-03-06T07:28:41.939557+0000","last_active":"2024-03-06T07:28:45.610723+0000","last_peered":"2024-03-06T07:28:45.610723+0000","last_clean":"2024-03-06T07:28:45.610723+0000","last_became_active":"2024-03-06T07:28:41.939288+0000","last_became_peered":"2024-03-06T07:28:41.939288+0000","last_unstale":"2024-03-06T07:28:45.610723+0000","last_undegraded":"2024-03-06T07:28:45.610723+0000","last_fullsized":"2024-03-06T07:28:45.610723+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:40.315662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610783+0000","last_change":"2024-03-06T07:28:42.932884+0000","last_active":"2024-03-06T07:28:45.610783+0000","last_peered":"2024-03-06T07:28:45.610783+0000","last_clean":"2024-03-06T07:28:45.610783+0000","last_became_active":"2024-03-06T07:28:42.932629+0000","last_became_peered":"2024-03-06T07:28:42.932629+0000","last_unstale":"2024-03-06T07:28:45.610783+0000","last_undegraded":"2024-03-06T07:28:45.610783+0000","last_fullsized":"2024-03-06T07:28:45.610783+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:39.726844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"125'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.949147+0000","last_change":"2024-03-06T07:28:41.939490+0000","last_active":"2024-03-06T07:28:43.949147+0000","last_peered":"2024-03-06T07:28:43.949147+0000","last_clean":"2024-03-06T07:28:43.949147+0000","last_became_active":"2024-03-06T07:28:41.939200+0000","last_became_peered":"2024-03-06T07:28:41.939200+0000","last_unstale":"2024-03-06T07:28:43.949147+0000","last_undegraded":"2024-03-06T07:28:43.949147+0000","last_fullsized":"2024-03-06T07:28:43.949147+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:28.220933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.949094+0000","last_change":"2024-03-06T07:28:42.941054+0000","last_active":"2024-03-06T07:28:43.949094+0000","last_peered":"2024-03-06T07:28:43.949094+0000","last_clean":"2024-03-06T07:28:43.949094+0000","last_became_active":"2024-03-06T07:28:42.940741+0000","last_became_peered":"2024-03-06T07:28:42.940741+0000","last_unstale":"2024-03-06T07:28:43.949094+0000","last_undegraded":"2024-03-06T07:28:43.949094+0000","last_fullsized":"2024-03-06T07:28:43.949094+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:10.916297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610208+0000","last_change":"2024-03-06T07:28:41.941854+0000","last_active":"2024-03-06T07:28:45.610208+0000","last_peered":"2024-03-06T07:28:45.610208+0000","last_clean":"2024-03-06T07:28:45.610208+0000","last_became_active":"2024-03-06T07:28:41.941567+0000","last_became_peered":"2024-03-06T07:28:41.941567+0000","last_unstale":"2024-03-06T07:28:45.610208+0000","last_undegraded":"2024-03-06T07:28:45.610208+0000","last_fullsized":"2024-03-06T07:28:45.610208+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:31.770357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.945713+0000","last_change":"2024-03-06T07:28:42.939493+0000","last_active":"2024-03-06T07:28:43.945713+0000","last_peered":"2024-03-06T07:28:43.945713+0000","last_clean":"2024-03-06T07:28:43.945713+0000","last_became_active":"2024-03-06T07:28:42.939181+0000","last_became_peered":"2024-03-06T07:28:42.939181+0000","last_unstale":"2024-03-06T07:28:43.945713+0000","last_undegraded":"2024-03-06T07:28:43.945713+0000","last_fullsized":"2024-03-06T07:28:43.945713+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:28.852420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"125'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.930687+0000","last_change":"2024-03-06T07:28:41.940546+0000","last_active":"2024-03-06T07:28:44.930687+0000","last_peered":"2024-03-06T07:28:44.930687+0000","last_clean":"2024-03-06T07:28:44.930687+0000","last_became_active":"2024-03-06T07:28:41.940263+0000","last_became_peered":"2024-03-06T07:28:41.940263+0000","last_unstale":"2024-03-06T07:28:44.930687+0000","last_undegraded":"2024-03-06T07:28:44.930687+0000","last_fullsized":"2024-03-06T07:28:44.930687+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:39.923244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610913+0000","last_change":"2024-03-06T07:28:42.949628+0000","last_active":"2024-03-06T07:28:45.610913+0000","last_peered":"2024-03-06T07:28:45.610913+0000","last_clean":"2024-03-06T07:28:45.610913+0000","last_became_active":"2024-03-06T07:28:42.949127+0000","last_became_peered":"2024-03-06T07:28:42.949127+0000","last_unstale":"2024-03-06T07:28:45.610913+0000","last_undegraded":"2024-03-06T07:28:45.610913+0000","last_fullsized":"2024-03-06T07:28:45.610913+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:39:51.254295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.948800+0000","last_change":"2024-03-06T07:28:41.945850+0000","last_active":"2024-03-06T07:28:43.948800+0000","last_peered":"2024-03-06T07:28:43.948800+0000","last_clean":"2024-03-06T07:28:43.948800+0000","last_became_active":"2024-03-06T07:28:41.945270+0000","last_became_peered":"2024-03-06T07:28:41.945270+0000","last_unstale":"2024-03-06T07:28:43.948800+0000","last_undegraded":"2024-03-06T07:28:43.948800+0000","last_fullsized":"2024-03-06T07:28:43.948800+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:44.506896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.942695+0000","last_change":"2024-03-06T07:28:42.950709+0000","last_active":"2024-03-06T07:28:43.942695+0000","last_peered":"2024-03-06T07:28:43.942695+0000","last_clean":"2024-03-06T07:28:43.942695+0000","last_became_active":"2024-03-06T07:28:42.946427+0000","last_became_peered":"2024-03-06T07:28:42.946427+0000","last_unstale":"2024-03-06T07:28:43.942695+0000","last_undegraded":"2024-03-06T07:28:43.942695+0000","last_fullsized":"2024-03-06T07:28:43.942695+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:02.632176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.948668+0000","last_change":"2024-03-06T07:28:41.942791+0000","last_active":"2024-03-06T07:28:43.948668+0000","last_peered":"2024-03-06T07:28:43.948668+0000","last_clean":"2024-03-06T07:28:43.948668+0000","last_became_active":"2024-03-06T07:28:41.942238+0000","last_became_peered":"2024-03-06T07:28:41.942238+0000","last_unstale":"2024-03-06T07:28:43.948668+0000","last_undegraded":"2024-03-06T07:28:43.948668+0000","last_fullsized":"2024-03-06T07:28:43.948668+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:40.315662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.930668+0000","last_change":"2024-03-06T07:28:42.947011+0000","last_active":"2024-03-06T07:28:44.930668+0000","last_peered":"2024-03-06T07:28:44.930668+0000","last_clean":"2024-03-06T07:28:44.930668+0000","last_became_active":"2024-03-06T07:28:42.946725+0000","last_became_peered":"2024-03-06T07:28:42.946725+0000","last_unstale":"2024-03-06T07:28:44.930668+0000","last_undegraded":"2024-03-06T07:28:44.930668+0000","last_fullsized":"2024-03-06T07:28:44.930668+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:32.039689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.945482+0000","last_change":"2024-03-06T07:28:41.936580+0000","last_active":"2024-03-06T07:28:43.945482+0000","last_peered":"2024-03-06T07:28:43.945482+0000","last_clean":"2024-03-06T07:28:43.945482+0000","last_became_active":"2024-03-06T07:28:41.936171+0000","last_became_peered":"2024-03-06T07:28:41.936171+0000","last_unstale":"2024-03-06T07:28:43.945482+0000","last_undegraded":"2024-03-06T07:28:43.945482+0000","last_fullsized":"2024-03-06T07:28:43.945482+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:01.940041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.609932+0000","last_change":"2024-03-06T07:28:42.951862+0000","last_active":"2024-03-06T07:28:45.609932+0000","last_peered":"2024-03-06T07:28:45.609932+0000","last_clean":"2024-03-06T07:28:45.609932+0000","last_became_active":"2024-03-06T07:28:42.951611+0000","last_became_peered":"2024-03-06T07:28:42.951611+0000","last_unstale":"2024-03-06T07:28:45.609932+0000","last_undegraded":"2024-03-06T07:28:45.609932+0000","last_fullsized":"2024-03-06T07:28:45.609932+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:40.928322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.945345+0000","last_change":"2024-03-06T07:28:41.937569+0000","last_active":"2024-03-06T07:28:43.945345+0000","last_peered":"2024-03-06T07:28:43.945345+0000","last_clean":"2024-03-06T07:28:43.945345+0000","last_became_active":"2024-03-06T07:28:41.937296+0000","last_became_peered":"2024-03-06T07:28:41.937296+0000","last_unstale":"2024-03-06T07:28:43.945345+0000","last_undegraded":"2024-03-06T07:28:43.945345+0000","last_fullsized":"2024-03-06T07:28:43.945345+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:31.034611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199436+0000","last_change":"2024-03-06T07:28:42.952249+0000","last_active":"2024-03-06T07:28:44.199436+0000","last_peered":"2024-03-06T07:28:44.199436+0000","last_clean":"2024-03-06T07:28:44.199436+0000","last_became_active":"2024-03-06T07:28:42.951944+0000","last_became_peered":"2024-03-06T07:28:42.951944+0000","last_unstale":"2024-03-06T07:28:44.199436+0000","last_undegraded":"2024-03-06T07:28:44.199436+0000","last_fullsized":"2024-03-06T07:28:44.199436+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:51:00.412904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.609722+0000","last_change":"2024-03-06T07:28:41.945475+0000","last_active":"2024-03-06T07:28:45.609722+0000","last_peered":"2024-03-06T07:28:45.609722+0000","last_clean":"2024-03-06T07:28:45.609722+0000","last_became_active":"2024-03-06T07:28:41.945185+0000","last_became_peered":"2024-03-06T07:28:41.945185+0000","last_unstale":"2024-03-06T07:28:45.609722+0000","last_undegraded":"2024-03-06T07:28:45.609722+0000","last_fullsized":"2024-03-06T07:28:45.609722+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:12.051951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.930738+0000","last_change":"2024-03-06T07:28:42.950096+0000","last_active":"2024-03-06T07:28:44.930738+0000","last_peered":"2024-03-06T07:28:44.930738+0000","last_clean":"2024-03-06T07:28:44.930738+0000","last_became_active":"2024-03-06T07:28:42.949812+0000","last_became_peered":"2024-03-06T07:28:42.949812+0000","last_unstale":"2024-03-06T07:28:44.930738+0000","last_undegraded":"2024-03-06T07:28:44.930738+0000","last_fullsized":"2024-03-06T07:28:44.930738+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:30.778714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.943106+0000","last_change":"2024-03-06T07:28:41.937110+0000","last_active":"2024-03-06T07:28:43.943106+0000","last_peered":"2024-03-06T07:28:43.943106+0000","last_clean":"2024-03-06T07:28:43.943106+0000","last_became_active":"2024-03-06T07:28:41.936705+0000","last_became_peered":"2024-03-06T07:28:41.936705+0000","last_unstale":"2024-03-06T07:28:43.943106+0000","last_undegraded":"2024-03-06T07:28:43.943106+0000","last_fullsized":"2024-03-06T07:28:43.943106+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:24.295174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.943184+0000","last_change":"2024-03-06T07:28:42.941197+0000","last_active":"2024-03-06T07:28:43.943184+0000","last_peered":"2024-03-06T07:28:43.943184+0000","last_clean":"2024-03-06T07:28:43.943184+0000","last_became_active":"2024-03-06T07:28:42.940879+0000","last_became_peered":"2024-03-06T07:28:42.940879+0000","last_unstale":"2024-03-06T07:28:43.943184+0000","last_undegraded":"2024-03-06T07:28:43.943184+0000","last_fullsized":"2024-03-06T07:28:43.943184+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:14.359727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"126'1","reported_seq":15,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199869+0000","last_change":"2024-03-06T07:28:41.944308+0000","last_active":"2024-03-06T07:28:44.199869+0000","last_peered":"2024-03-06T07:28:44.199869+0000","last_clean":"2024-03-06T07:28:44.199869+0000","last_became_active":"2024-03-06T07:28:41.944025+0000","last_became_peered":"2024-03-06T07:28:41.944025+0000","last_unstale":"2024-03-06T07:28:44.199869+0000","last_undegraded":"2024-03-06T07:28:44.199869+0000","last_fullsized":"2024-03-06T07:28:44.199869+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:00:41.369067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610542+0000","last_change":"2024-03-06T07:28:42.945861+0000","last_active":"2024-03-06T07:28:45.610542+0000","last_peered":"2024-03-06T07:28:45.610542+0000","last_clean":"2024-03-06T07:28:45.610542+0000","last_became_active":"2024-03-06T07:28:42.945568+0000","last_became_peered":"2024-03-06T07:28:42.945568+0000","last_unstale":"2024-03-06T07:28:45.610542+0000","last_undegraded":"2024-03-06T07:28:45.610542+0000","last_fullsized":"2024-03-06T07:28:45.610542+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:59.551703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610846+0000","last_change":"2024-03-06T07:28:41.941827+0000","last_active":"2024-03-06T07:28:45.610846+0000","last_peered":"2024-03-06T07:28:45.610846+0000","last_clean":"2024-03-06T07:28:45.610846+0000","last_became_active":"2024-03-06T07:28:41.941566+0000","last_became_peered":"2024-03-06T07:28:41.941566+0000","last_unstale":"2024-03-06T07:28:45.610846+0000","last_undegraded":"2024-03-06T07:28:45.610846+0000","last_fullsized":"2024-03-06T07:28:45.610846+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:50:25.313793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.930655+0000","last_change":"2024-03-06T07:28:42.953988+0000","last_active":"2024-03-06T07:28:44.930655+0000","last_peered":"2024-03-06T07:28:44.930655+0000","last_clean":"2024-03-06T07:28:44.930655+0000","last_became_active":"2024-03-06T07:28:42.953582+0000","last_became_peered":"2024-03-06T07:28:42.953582+0000","last_unstale":"2024-03-06T07:28:44.930655+0000","last_undegraded":"2024-03-06T07:28:44.930655+0000","last_fullsized":"2024-03-06T07:28:44.930655+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:46.376827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610313+0000","last_change":"2024-03-06T07:28:41.949188+0000","last_active":"2024-03-06T07:28:45.610313+0000","last_peered":"2024-03-06T07:28:45.610313+0000","last_clean":"2024-03-06T07:28:45.610313+0000","last_became_active":"2024-03-06T07:28:41.948848+0000","last_became_peered":"2024-03-06T07:28:41.948848+0000","last_unstale":"2024-03-06T07:28:45.610313+0000","last_undegraded":"2024-03-06T07:28:45.610313+0000","last_fullsized":"2024-03-06T07:28:45.610313+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:44.260284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199527+0000","last_change":"2024-03-06T07:28:42.948566+0000","last_active":"2024-03-06T07:28:44.199527+0000","last_peered":"2024-03-06T07:28:44.199527+0000","last_clean":"2024-03-06T07:28:44.199527+0000","last_became_active":"2024-03-06T07:28:42.948305+0000","last_became_peered":"2024-03-06T07:28:42.948305+0000","last_unstale":"2024-03-06T07:28:44.199527+0000","last_undegraded":"2024-03-06T07:28:44.199527+0000","last_fullsized":"2024-03-06T07:28:44.199527+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:29.022233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610343+0000","last_change":"2024-03-06T07:28:41.941100+0000","last_active":"2024-03-06T07:28:45.610343+0000","last_peered":"2024-03-06T07:28:45.610343+0000","last_clean":"2024-03-06T07:28:45.610343+0000","last_became_active":"2024-03-06T07:28:41.940801+0000","last_became_peered":"2024-03-06T07:28:41.940801+0000","last_unstale":"2024-03-06T07:28:45.610343+0000","last_undegraded":"2024-03-06T07:28:45.610343+0000","last_fullsized":"2024-03-06T07:28:45.610343+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:26.687048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.948944+0000","last_change":"2024-03-06T07:28:42.931591+0000","last_active":"2024-03-06T07:28:43.948944+0000","last_peered":"2024-03-06T07:28:43.948944+0000","last_clean":"2024-03-06T07:28:43.948944+0000","last_became_active":"2024-03-06T07:28:42.931004+0000","last_became_peered":"2024-03-06T07:28:42.931004+0000","last_unstale":"2024-03-06T07:28:43.948944+0000","last_undegraded":"2024-03-06T07:28:43.948944+0000","last_fullsized":"2024-03-06T07:28:43.948944+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:32:12.822392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199600+0000","last_change":"2024-03-06T07:28:41.943502+0000","last_active":"2024-03-06T07:28:44.199600+0000","last_peered":"2024-03-06T07:28:44.199600+0000","last_clean":"2024-03-06T07:28:44.199600+0000","last_became_active":"2024-03-06T07:28:41.943189+0000","last_became_peered":"2024-03-06T07:28:41.943189+0000","last_unstale":"2024-03-06T07:28:44.199600+0000","last_undegraded":"2024-03-06T07:28:44.199600+0000","last_fullsized":"2024-03-06T07:28:44.199600+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:00.630311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.945736+0000","last_change":"2024-03-06T07:28:42.945847+0000","last_active":"2024-03-06T07:28:43.945736+0000","last_peered":"2024-03-06T07:28:43.945736+0000","last_clean":"2024-03-06T07:28:43.945736+0000","last_became_active":"2024-03-06T07:28:42.945567+0000","last_became_peered":"2024-03-06T07:28:42.945567+0000","last_unstale":"2024-03-06T07:28:43.945736+0000","last_undegraded":"2024-03-06T07:28:43.945736+0000","last_fullsized":"2024-03-06T07:28:43.945736+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:13.057028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:43.945422+0000","last_change":"2024-03-06T07:28:41.936622+0000","last_active":"2024-03-06T07:28:43.945422+0000","last_peered":"2024-03-06T07:28:43.945422+0000","last_clean":"2024-03-06T07:28:43.945422+0000","last_became_active":"2024-03-06T07:28:41.936347+0000","last_became_peered":"2024-03-06T07:28:41.936347+0000","last_unstale":"2024-03-06T07:28:43.945422+0000","last_undegraded":"2024-03-06T07:28:43.945422+0000","last_fullsized":"2024-03-06T07:28:43.945422+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:11.999340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.610375+0000","last_change":"2024-03-06T07:28:42.945061+0000","last_active":"2024-03-06T07:28:45.610375+0000","last_peered":"2024-03-06T07:28:45.610375+0000","last_clean":"2024-03-06T07:28:45.610375+0000","last_became_active":"2024-03-06T07:28:42.944770+0000","last_became_peered":"2024-03-06T07:28:42.944770+0000","last_unstale":"2024-03-06T07:28:45.610375+0000","last_undegraded":"2024-03-06T07:28:45.610375+0000","last_fullsized":"2024-03-06T07:28:45.610375+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:39.296923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:45.609838+0000","last_change":"2024-03-06T07:28:41.945255+0000","last_active":"2024-03-06T07:28:45.609838+0000","last_peered":"2024-03-06T07:28:45.609838+0000","last_clean":"2024-03-06T07:28:45.609838+0000","last_became_active":"2024-03-06T07:28:41.945018+0000","last_became_peered":"2024-03-06T07:28:41.945018+0000","last_unstale":"2024-03-06T07:28:45.609838+0000","last_undegraded":"2024-03-06T07:28:45.609838+0000","last_fullsized":"2024-03-06T07:28:45.609838+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:46.880323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199536+0000","last_change":"2024-03-06T07:28:42.952489+0000","last_active":"2024-03-06T07:28:44.199536+0000","last_peered":"2024-03-06T07:28:44.199536+0000","last_clean":"2024-03-06T07:28:44.199536+0000","last_became_active":"2024-03-06T07:28:42.951812+0000","last_became_peered":"2024-03-06T07:28:42.951812+0000","last_unstale":"2024-03-06T07:28:44.199536+0000","last_undegraded":"2024-03-06T07:28:44.199536+0000","last_fullsized":"2024-03-06T07:28:44.199536+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:41.916963+0000","last_clean_scrub_stamp":"2024-03-06T07:28:41.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:59.778048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199638+0000","last_change":"2024-03-06T07:28:41.944250+0000","last_active":"2024-03-06T07:28:44.199638+0000","last_peered":"2024-03-06T07:28:44.199638+0000","last_clean":"2024-03-06T07:28:44.199638+0000","last_became_active":"2024-03-06T07:28:41.943677+0000","last_became_peered":"2024-03-06T07:28:41.943677+0000","last_unstale":"2024-03-06T07:28:44.199638+0000","last_undegraded":"2024-03-06T07:28:44.199638+0000","last_fullsized":"2024-03-06T07:28:44.199638+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:40.911886+0000","last_clean_scrub_stamp":"2024-03-06T07:28:40.911886+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:38.291846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":359,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:28:44.199268+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:28:44.199268+0000","last_peered":"2024-03-06T07:28:44.199268+0000","last_clean":"2024-03-06T07:28:44.199268+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:28:44.199268+0000","last_undegraded":"2024-03-06T07:28:44.199268+0000","last_fullsized":"2024-03-06T07:28:44.199268+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41644,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330196,"statfs":{"total":96636764160,"available":96594120704,"internally_reserved":0,"allocated":2191360,"data_stored":1278680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574937,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41052,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330788,"statfs":{"total":96636764160,"available":96594726912,"internally_reserved":0,"allocated":1585152,"data_stored":687674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574936,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41044,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330796,"statfs":{"total":96636764160,"available":96594735104,"internally_reserved":0,"allocated":1581056,"data_stored":686174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41044,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330796,"statfs":{"total":96636764160,"available":96594735104,"internally_reserved":0,"allocated":1576960,"data_stored":686132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41048,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330792,"statfs":{"total":96636764160,"available":96594731008,"internally_reserved":0,"allocated":1581056,"data_stored":686222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574935,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41044,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330796,"statfs":{"total":96636764160,"available":96594735104,"internally_reserved":0,"allocated":1581056,"data_stored":686678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41036,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330804,"statfs":{"total":96636764160,"available":96594743296,"internally_reserved":0,"allocated":1572864,"data_stored":686062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574937,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41616,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330224,"statfs":{"total":96636764160,"available":96594149376,"internally_reserved":0,"allocated":2166784,"data_stored":1276430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:50.717 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:28:50.717 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:28:50.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.861+0000 7f8f383d7640 1 Processor -- start 2024-03-06T07:28:50.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.861+0000 7f8f383d7640 1 -- start start 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f383d7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8f301614d0 0x7f8f300d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f383d7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8f300d24b0 0x7f8f300b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f383d7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f301618b0 con 0x7f8f300d1910 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f383d7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f300cf400 con 0x7f8f301614d0 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f383d7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f300cf6a0 con 0x7f8f300d24b0 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f36173640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8f301614d0 0x7f8f300d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f36173640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8f301614d0 0x7f8f300d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48292/0 (socket says 172.21.15.59:48292) 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f36173640 1 -- 172.21.15.59:0/1201585474 learned_addr learned my addr 172.21.15.59:0/1201585474 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f36974640 1 --2- 172.21.15.59:0/1201585474 >> v2:172.21.15.164:3300/0 conn(0x7f8f300d24b0 0x7f8f300b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f36173640 1 -- 172.21.15.59:0/1201585474 >> v2:172.21.15.164:3300/0 conn(0x7f8f300d24b0 msgr2=0x7f8f300b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f36173640 1 --2- 172.21.15.59:0/1201585474 >> v2:172.21.15.164:3300/0 conn(0x7f8f300d24b0 0x7f8f300b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.862+0000 7f8f36173640 1 -- 172.21.15.59:0/1201585474 >> v1:172.21.15.164:6789/0 conn(0x7f8f300d1910 legacy=0x7f8f300d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.863+0000 7f8f36173640 1 -- 172.21.15.59:0/1201585474 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f300cf980 con 0x7f8f301614d0 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.863+0000 7f8f36173640 1 --2- 172.21.15.59:0/1201585474 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f301614d0 0x7f8f300d4860 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f8f300b2a70 tx=0x7f8f2c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5743a442569ce2d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.863+0000 7f8f35171640 1 -- 172.21.15.59:0/1201585474 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c034070 con 0x7f8f301614d0 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.863+0000 7f8f35171640 1 -- 172.21.15.59:0/1201585474 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c030860 con 0x7f8f301614d0 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.863+0000 7f8f35171640 1 -- 172.21.15.59:0/1201585474 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c030c30 con 0x7f8f301614d0 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.864+0000 7f8f383d7640 1 -- 172.21.15.59:0/1201585474 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f301614d0 msgr2=0x7f8f300d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.864+0000 7f8f383d7640 1 --2- 172.21.15.59:0/1201585474 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f301614d0 0x7f8f300d4860 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f8f300b2a70 tx=0x7f8f2c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.864+0000 7f8f383d7640 1 -- 172.21.15.59:0/1201585474 shutdown_connections 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.864+0000 7f8f383d7640 1 --2- 172.21.15.59:0/1201585474 >> v2:172.21.15.164:3300/0 conn(0x7f8f300d24b0 0x7f8f300b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.864+0000 7f8f383d7640 1 --2- 172.21.15.59:0/1201585474 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f301614d0 0x7f8f300d4860 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.864+0000 7f8f383d7640 1 -- 172.21.15.59:0/1201585474 >> 172.21.15.59:0/1201585474 conn(0x7f8f300d3d40 msgr2=0x7f8f300d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.864+0000 7f8f383d7640 1 -- 172.21.15.59:0/1201585474 shutdown_connections 2024-03-06T07:28:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.864+0000 7f8f383d7640 1 -- 172.21.15.59:0/1201585474 wait complete. 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.865+0000 7f8f383d7640 1 Processor -- start 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.865+0000 7f8f383d7640 1 -- start start 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f383d7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f300d1910 0x7f8f30154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f383d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f300d24b0 0x7f8f301f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f383d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f301614d0 0x7f8f300ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f383d7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f300b1370 con 0x7f8f301614d0 2024-03-06T07:28:50.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f383d7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f300b0910 con 0x7f8f300d1910 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f383d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f300b14f0 con 0x7f8f300d24b0 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f36974640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f301614d0 0x7f8f300ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f36974640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f301614d0 0x7f8f300ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48294/0 (socket says 172.21.15.59:48294) 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f36974640 1 -- 172.21.15.59:0/1391928490 learned_addr learned my addr 172.21.15.59:0/1391928490 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f36173640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f300d1910 0x7f8f30154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f36974640 1 -- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f300d24b0 msgr2=0x7f8f301f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f36974640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f300d24b0 0x7f8f301f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f36974640 1 -- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f300d1910 msgr2=0x7f8f30154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f36974640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f300d1910 0x7f8f30154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f36974640 1 -- 172.21.15.59:0/1391928490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f300cc970 con 0x7f8f301614d0 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.866+0000 7f8f36173640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f300d1910 0x7f8f30154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.867+0000 7f8f36974640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f301614d0 0x7f8f300ca570 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f8f28007e70 tx=0x7f8f2800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.867+0000 7f8f26ffd640 1 -- 172.21.15.59:0/1391928490 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28014620 con 0x7f8f301614d0 2024-03-06T07:28:50.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.867+0000 7f8f26ffd640 1 -- 172.21.15.59:0/1391928490 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f28019070 con 0x7f8f301614d0 2024-03-06T07:28:50.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.867+0000 7f8f383d7640 1 -- 172.21.15.59:0/1391928490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f300ccc50 con 0x7f8f301614d0 2024-03-06T07:28:50.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.867+0000 7f8f26ffd640 1 -- 172.21.15.59:0/1391928490 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28014a20 con 0x7f8f301614d0 2024-03-06T07:28:50.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.867+0000 7f8f383d7640 1 -- 172.21.15.59:0/1391928490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f301f77c0 con 0x7f8f301614d0 2024-03-06T07:28:50.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.868+0000 7f8f383d7640 1 -- 172.21.15.59:0/1391928490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f301f79c0 con 0x7f8f301614d0 2024-03-06T07:28:50.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.868+0000 7f8f26ffd640 1 -- 172.21.15.59:0/1391928490 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f28014bc0 con 0x7f8f301614d0 2024-03-06T07:28:50.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.869+0000 7f8f26ffd640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f1406b4d0 0x7f8f1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:50.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.869+0000 7f8f26ffd640 1 -- 172.21.15.59:0/1391928490 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f2808ab60 con 0x7f8f301614d0 2024-03-06T07:28:50.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.869+0000 7f8f36173640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f1406b4d0 0x7f8f1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:50.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.870+0000 7f8f36173640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f1406b4d0 0x7f8f1406d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c036000 tx=0x7f8f2c03b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:50.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:50.877+0000 7f8f26ffd640 1 -- 172.21.15.59:0/1391928490 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f280599e0 con 0x7f8f301614d0 2024-03-06T07:28:51.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.133+0000 7f8f383d7640 1 -- 172.21.15.59:0/1391928490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8f3015a8b0 con 0x7f8f301614d0 2024-03-06T07:28:51.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.137+0000 7f8f26ffd640 1 -- 172.21.15.59:0/1391928490 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f8f280592e0 con 0x7f8f301614d0 2024-03-06T07:28:51.139 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-06T07:28:51.140 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA== 2024-03-06T07:28:51.140 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-03-06T07:28:51.140 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-03-06T07:28:51.140 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:51.140 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-03-06T07:28:51.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 -- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f1406b4d0 msgr2=0x7f8f1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f1406b4d0 0x7f8f1406d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c036000 tx=0x7f8f2c03b040 comp rx=0 tx=0).stop 2024-03-06T07:28:51.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 -- 172.21.15.59:0/1391928490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f301614d0 msgr2=0x7f8f300ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f301614d0 0x7f8f300ca570 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f8f28007e70 tx=0x7f8f2800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 -- 172.21.15.59:0/1391928490 shutdown_connections 2024-03-06T07:28:51.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f1406b4d0 0x7f8f1406d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f301614d0 0x7f8f300ca570 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f300d24b0 0x7f8f301f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 --2- 172.21.15.59:0/1391928490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f300d1910 0x7f8f30154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 -- 172.21.15.59:0/1391928490 >> 172.21.15.59:0/1391928490 conn(0x7f8f300d3d40 msgr2=0x7f8f300b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 -- 172.21.15.59:0/1391928490 shutdown_connections 2024-03-06T07:28:51.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.139+0000 7f8f383d7640 1 -- 172.21.15.59:0/1391928490 wait complete. 2024-03-06T07:28:51.152 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.252+0000 7f3d1212c640 1 Processor -- start 2024-03-06T07:28:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.252+0000 7f3d1212c640 1 -- start start 2024-03-06T07:28:51.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.253+0000 7f3d1212c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3d0c0d22b0 0x7f3d0c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.253+0000 7f3d1212c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3d0c0b41f0 0x7f3d0c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.253+0000 7f3d1212c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0c0b3430 con 0x7f3d0c15dfe0 2024-03-06T07:28:51.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.253+0000 7f3d1212c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0c0d2c50 con 0x7f3d0c0b41f0 2024-03-06T07:28:51.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.253+0000 7f3d1212c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0c0b31e0 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.253+0000 7f3d0bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3d0c0b41f0 0x7f3d0c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.253+0000 7f3d0b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3d0c15dfe0 0x7f3d0c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44478/0 (socket says 172.21.15.59:44478) 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.253+0000 7f3d0b7fe640 1 -- 172.21.15.59:0/803485795 learned_addr learned my addr 172.21.15.59:0/803485795 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.254+0000 7f3d0affd640 1 --2- 172.21.15.59:0/803485795 >> v2:172.21.15.164:3300/0 conn(0x7f3d0c0d22b0 0x7f3d0c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.254+0000 7f3d0bfff640 1 -- 172.21.15.59:0/803485795 >> v2:172.21.15.164:3300/0 conn(0x7f3d0c0d22b0 msgr2=0x7f3d0c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.254+0000 7f3d0bfff640 1 --2- 172.21.15.59:0/803485795 >> v2:172.21.15.164:3300/0 conn(0x7f3d0c0d22b0 0x7f3d0c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.254+0000 7f3d0bfff640 1 -- 172.21.15.59:0/803485795 >> v1:172.21.15.59:6789/0 conn(0x7f3d0c15dfe0 legacy=0x7f3d0c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.254+0000 7f3d0bfff640 1 -- 172.21.15.59:0/803485795 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c0b3710 con 0x7f3d0c0b41f0 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.254+0000 7f3d0bfff640 1 --2- 172.21.15.59:0/803485795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c0b41f0 0x7f3d0c0b4680 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009d90 tx=0x7f3d0402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68553ed2c7cb939c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.254+0000 7f3d0a7fc640 1 -- 172.21.15.59:0/803485795 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d04039070 con 0x7f3d0c0b41f0 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.255+0000 7f3d0a7fc640 1 -- 172.21.15.59:0/803485795 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0402cc60 con 0x7f3d0c0b41f0 2024-03-06T07:28:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.255+0000 7f3d0a7fc640 1 -- 172.21.15.59:0/803485795 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d04034660 con 0x7f3d0c0b41f0 2024-03-06T07:28:51.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.255+0000 7f3d1212c640 1 -- 172.21.15.59:0/803485795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c0b41f0 msgr2=0x7f3d0c0b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.255+0000 7f3d1212c640 1 --2- 172.21.15.59:0/803485795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c0b41f0 0x7f3d0c0b4680 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009d90 tx=0x7f3d0402beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.255+0000 7f3d1212c640 1 -- 172.21.15.59:0/803485795 shutdown_connections 2024-03-06T07:28:51.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.255+0000 7f3d1212c640 1 --2- 172.21.15.59:0/803485795 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c0b41f0 0x7f3d0c0b4680 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.255+0000 7f3d1212c640 1 --2- 172.21.15.59:0/803485795 >> v2:172.21.15.164:3300/0 conn(0x7f3d0c0d22b0 0x7f3d0c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.255+0000 7f3d1212c640 1 -- 172.21.15.59:0/803485795 >> 172.21.15.59:0/803485795 conn(0x7f3d0c0d3cd0 msgr2=0x7f3d0c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.255+0000 7f3d1212c640 1 -- 172.21.15.59:0/803485795 shutdown_connections 2024-03-06T07:28:51.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.255+0000 7f3d1212c640 1 -- 172.21.15.59:0/803485795 wait complete. 2024-03-06T07:28:51.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.256+0000 7f3d1212c640 1 Processor -- start 2024-03-06T07:28:51.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.256+0000 7f3d1212c640 1 -- start start 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.257+0000 7f3d1212c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d0c0b41f0 0x7f3d0c1f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.257+0000 7f3d1212c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c0d22b0 0x7f3d0c0cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.257+0000 7f3d1212c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d0c15dfe0 0x7f3d0c149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.257+0000 7f3d1212c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c0cfa00 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.257+0000 7f3d1212c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c0b3d60 con 0x7f3d0c15dfe0 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.257+0000 7f3d1212c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0c0d1c70 con 0x7f3d0c0b41f0 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.257+0000 7f3d0affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c0d22b0 0x7f3d0c0cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.257+0000 7f3d0affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c0d22b0 0x7f3d0c0cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48308/0 (socket says 172.21.15.59:48308) 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.257+0000 7f3d0affd640 1 -- 172.21.15.59:0/431518915 learned_addr learned my addr 172.21.15.59:0/431518915 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.257+0000 7f3d0bfff640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d0c15dfe0 0x7f3d0c149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.257+0000 7f3d0b7fe640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d0c0b41f0 0x7f3d0c1f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.258+0000 7f3d0affd640 1 -- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d0c0b41f0 msgr2=0x7f3d0c1f6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.258+0000 7f3d0affd640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d0c0b41f0 0x7f3d0c1f6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.258+0000 7f3d0affd640 1 -- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d0c15dfe0 msgr2=0x7f3d0c149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.258+0000 7f3d0affd640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d0c15dfe0 0x7f3d0c149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.258+0000 7f3d0affd640 1 -- 172.21.15.59:0/431518915 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c0b70b0 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.258+0000 7f3d0affd640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c0d22b0 0x7f3d0c0cd1a0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00d830 tx=0x7f3cfc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.258+0000 7f3d03fff640 1 -- 172.21.15.59:0/431518915 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc00b6c0 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.258+0000 7f3d03fff640 1 -- 172.21.15.59:0/431518915 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc019070 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.258+0000 7f3d1212c640 1 -- 172.21.15.59:0/431518915 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0c0b6af0 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.258+0000 7f3d03fff640 1 -- 172.21.15.59:0/431518915 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc00ba80 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.258+0000 7f3d1212c640 1 -- 172.21.15.59:0/431518915 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0c0cc9a0 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.259+0000 7f3d1212c640 1 -- 172.21.15.59:0/431518915 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0c0cc7d0 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.259+0000 7f3d03fff640 1 -- 172.21.15.59:0/431518915 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cfc01e460 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.260+0000 7f3d03fff640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3cec06b4d0 0x7f3cec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.260+0000 7f3d03fff640 1 -- 172.21.15.59:0/431518915 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cfc08a7d0 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.260+0000 7f3d0b7fe640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3cec06b4d0 0x7f3cec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.261+0000 7f3d0b7fe640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3cec06b4d0 0x7f3cec06d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4007fd0 tx=0x7f3cf4007e80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.264+0000 7f3d03fff640 1 -- 172.21.15.59:0/431518915 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d0c0b70b0 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.508+0000 7f3d1212c640 1 -- 172.21.15.59:0/431518915 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d0c0d0860 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.509+0000 7f3d03fff640 1 -- 172.21.15.59:0/431518915 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f3cfc058f70 con 0x7f3d0c0d22b0 2024-03-06T07:28:51.511 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:51.511 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1772616514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1772616514},{"type":"v1","addr":"172.21.15.164:6836","nonce":1772616514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":7816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3891024511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3891024511},{"type":"v1","addr":"172.21.15.164:6837","nonce":3891024511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":8673,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:28:44.364208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:28:51.511 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 79 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 -- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3cec06b4d0 msgr2=0x7f3cec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3cec06b4d0 0x7f3cec06d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4007fd0 tx=0x7f3cf4007e80 comp rx=0 tx=0).stop 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 -- 172.21.15.59:0/431518915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c0d22b0 msgr2=0x7f3d0c0cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c0d22b0 0x7f3d0c0cd1a0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00d830 tx=0x7f3cfc00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 -- 172.21.15.59:0/431518915 shutdown_connections 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3cec06b4d0 0x7f3cec06d990 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d0c15dfe0 0x7f3d0c149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d0c0d22b0 0x7f3d0c0cd1a0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 --2- 172.21.15.59:0/431518915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d0c0b41f0 0x7f3d0c1f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 -- 172.21.15.59:0/431518915 >> 172.21.15.59:0/431518915 conn(0x7f3d0c0d3cd0 msgr2=0x7f3d0c0b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 -- 172.21.15.59:0/431518915 shutdown_connections 2024-03-06T07:28:51.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:51.511+0000 7f3d1212c640 1 -- 172.21.15.59:0/431518915 wait complete. 2024-03-06T07:28:51.926 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:28:44.364208+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8676}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:28:51.926 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:28:51.926 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:28:52.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.059+0000 7fc9d5c22640 1 Processor -- start 2024-03-06T07:28:52.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.059+0000 7fc9d5c22640 1 -- start start 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.060+0000 7fc9d5c22640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc9d00b4c00 0x7fc9d00b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.060+0000 7fc9d5c22640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d015d510 con 0x7fc9d00b28a0 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.060+0000 7fc9d5c22640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d015d780 con 0x7fc9d015ba50 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.060+0000 7fc9d5c22640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d00b3420 con 0x7fc9d00b4c00 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.060+0000 7fc9ceffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc9d00b28a0 0x7fc9d00b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44494/0 (socket says 172.21.15.59:44494) 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.060+0000 7fc9ceffd640 1 -- 172.21.15.59:0/2764397535 learned_addr learned my addr 172.21.15.59:0/2764397535 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143897366 0 0) 0x7fc9d015d510 con 0x7fc9d00b28a0 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/2764397535 >> v2:172.21.15.164:3300/0 conn(0x7fc9d00b4c00 0x7fc9d00b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9bc003520 con 0x7fc9d00b28a0 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1481390017 0 0) 0x7fc9bc003520 con 0x7fc9d00b28a0 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 >> v2:172.21.15.164:3300/0 conn(0x7fc9d00b4c00 msgr2=0x7fc9d00b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 --2- 172.21.15.59:0/2764397535 >> v2:172.21.15.164:3300/0 conn(0x7fc9d00b4c00 0x7fc9d00b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 >> v1:172.21.15.164:6789/0 conn(0x7fc9d015ba50 legacy=0x7fc9d015bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d015da60 con 0x7fc9d00b28a0 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fc9b8001670 con 0x7fc9d00b28a0 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 >> v1:172.21.15.59:6789/0 conn(0x7fc9d00b28a0 legacy=0x7fc9d00b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 --2- 172.21.15.59:0/2764397535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc0039b0 0x7fc9bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 --2- 172.21.15.59:0/2764397535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9bc004800 0x7fc9bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 --2- 172.21.15.59:0/2764397535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9bc0051c0 0x7fc9bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00b3420 con 0x7fc9bc004800 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc00c070 con 0x7fc9bc0039b0 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc00c270 con 0x7fc9bc0051c0 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/2764397535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9bc004800 0x7fc9bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/2764397535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc0039b0 0x7fc9bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9cffff640 1 --2- 172.21.15.59:0/2764397535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9bc0051c0 0x7fc9bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ceffd640 1 -- 172.21.15.59:0/2764397535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9bc0051c0 msgr2=0x7fc9bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/2764397535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9bc0051c0 0x7fc9bc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.066 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ceffd640 1 -- 172.21.15.59:0/2764397535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc0039b0 msgr2=0x7fc9bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/2764397535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc0039b0 0x7fc9bc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9ceffd640 1 -- 172.21.15.59:0/2764397535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc003520 con 0x7fc9bc004800 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.061+0000 7fc9cffff640 1 -- 172.21.15.59:0/2764397535 reap_dead start 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.062+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/2764397535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9bc004800 0x7fc9bc004c80 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc9b800c210 tx=0x7fc9bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b970563296978d80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.062+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8007d30 con 0x7fc9bc004800 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.062+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b8030070 con 0x7fc9bc004800 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.062+0000 7fc9ce7fc640 1 -- 172.21.15.59:0/2764397535 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8005d70 con 0x7fc9bc004800 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.063+0000 7fc9d5c22640 1 -- 172.21.15.59:0/2764397535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9bc004800 msgr2=0x7fc9bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.063+0000 7fc9d5c22640 1 --2- 172.21.15.59:0/2764397535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9bc004800 0x7fc9bc004c80 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc9b800c210 tx=0x7fc9bc005830 comp rx=0 tx=0).stop 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.063+0000 7fc9d5c22640 1 -- 172.21.15.59:0/2764397535 shutdown_connections 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.063+0000 7fc9d5c22640 1 --2- 172.21.15.59:0/2764397535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9bc004800 0x7fc9bc004c80 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.063+0000 7fc9d5c22640 1 -- 172.21.15.59:0/2764397535 >> 172.21.15.59:0/2764397535 conn(0x7fc9d009a9f0 msgr2=0x7fc9d009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.063+0000 7fc9d5c22640 1 -- 172.21.15.59:0/2764397535 shutdown_connections 2024-03-06T07:28:52.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.063+0000 7fc9d5c22640 1 -- 172.21.15.59:0/2764397535 wait complete. 2024-03-06T07:28:52.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.064+0000 7fc9d5c22640 1 Processor -- start 2024-03-06T07:28:52.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.064+0000 7fc9d5c22640 1 -- start start 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9d5c22640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc004800 0x7fc9d0142500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9d5c22640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d01554e0 0x7fc9d01f7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9d5c22640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d01f7850 0x7fc9d01e75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9d5c22640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00b4fe0 con 0x7fc9d01554e0 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9d5c22640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d015e0b0 con 0x7fc9bc004800 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9d5c22640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00b5160 con 0x7fc9d01f7850 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9ceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d01554e0 0x7fc9d01f7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9ceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d01554e0 0x7fc9d01f7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48312/0 (socket says 172.21.15.59:48312) 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9ceffd640 1 -- 172.21.15.59:0/1711914051 learned_addr learned my addr 172.21.15.59:0/1711914051 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc004800 0x7fc9d0142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9cffff640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d01f7850 0x7fc9d01e75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9ceffd640 1 -- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d01f7850 msgr2=0x7fc9d01e75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d01f7850 0x7fc9d01e75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9ceffd640 1 -- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc004800 msgr2=0x7fc9d0142500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc004800 0x7fc9d0142500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9ceffd640 1 -- 172.21.15.59:0/1711914051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d01f7cd0 con 0x7fc9d01554e0 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.065+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc004800 0x7fc9d0142500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.066+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d01554e0 0x7fc9d01f7310 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fc9b800c330 tx=0x7fc9b802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.066+0000 7fc9c7fff640 1 -- 172.21.15.59:0/1711914051 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8046070 con 0x7fc9d01554e0 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.066+0000 7fc9c7fff640 1 -- 172.21.15.59:0/1711914051 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b8004bf0 con 0x7fc9d01554e0 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.066+0000 7fc9d5c22640 1 -- 172.21.15.59:0/1711914051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d01f7fb0 con 0x7fc9d01554e0 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.066+0000 7fc9c7fff640 1 -- 172.21.15.59:0/1711914051 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8007d10 con 0x7fc9d01554e0 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.066+0000 7fc9d5c22640 1 -- 172.21.15.59:0/1711914051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d01510a0 con 0x7fc9d01554e0 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.068+0000 7fc9c7fff640 1 -- 172.21.15.59:0/1711914051 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9b8007070 con 0x7fc9d01554e0 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.068+0000 7fc9c7fff640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc9b006b4d0 0x7fc9b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.068+0000 7fc9c7fff640 1 -- 172.21.15.59:0/1711914051 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9b80ab5c0 con 0x7fc9d01554e0 2024-03-06T07:28:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.069+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc9b006b4d0 0x7fc9b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.069+0000 7fc9d8805740 1 Processor -- start 2024-03-06T07:28:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.069+0000 7fc9d8805740 1 -- start start 2024-03-06T07:28:52.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.069+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc9b006b4d0 0x7fc9b006d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0009bc0 tx=0x7fc9c0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5a1870 0x5625fe5a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5a58c0 0x5625fe5a7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5d1600 0x5625fe5d1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9d8805740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625fe50b6b0 con 0x5625fe5a58c0 2024-03-06T07:28:52.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9d8805740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625fe4b4a60 con 0x5625fe5a1870 2024-03-06T07:28:52.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9d8805740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625fe507750 con 0x5625fe5d1600 2024-03-06T07:28:52.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9ceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5a58c0 0x5625fe5a7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9ceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5a58c0 0x5625fe5a7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48328/0 (socket says 172.21.15.59:48328) 2024-03-06T07:28:52.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9ceffd640 1 -- 172.21.15.59:0/750907033 learned_addr learned my addr 172.21.15.59:0/750907033 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/750907033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5a1870 0x5625fe5a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9cffff640 1 --2- 172.21.15.59:0/750907033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5d1600 0x5625fe5d1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9ceffd640 1 -- 172.21.15.59:0/750907033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5d1600 msgr2=0x5625fe5d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/750907033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5d1600 0x5625fe5d1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9ceffd640 1 -- 172.21.15.59:0/750907033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5a1870 msgr2=0x5625fe5a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/750907033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5a1870 0x5625fe5a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9ceffd640 1 -- 172.21.15.59:0/750907033 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625fe4d2f30 con 0x5625fe5a58c0 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/750907033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5a58c0 0x5625fe5a7360 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8081f20 tx=0x7fc9b802df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aeddb68600a3fc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9c57fa640 1 -- 172.21.15.59:0/750907033 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b80800c0 con 0x5625fe5a58c0 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9c57fa640 1 -- 172.21.15.59:0/750907033 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b8030040 con 0x5625fe5a58c0 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.070+0000 7fc9c57fa640 1 -- 172.21.15.59:0/750907033 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8076460 con 0x5625fe5a58c0 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.071+0000 7fc9d8805740 1 -- 172.21.15.59:0/750907033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5a58c0 msgr2=0x5625fe5a7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.071+0000 7fc9d8805740 1 --2- 172.21.15.59:0/750907033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5a58c0 0x5625fe5a7360 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8081f20 tx=0x7fc9b802df90 comp rx=0 tx=0).stop 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.071+0000 7fc9d8805740 1 -- 172.21.15.59:0/750907033 shutdown_connections 2024-03-06T07:28:52.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.071+0000 7fc9d8805740 1 --2- 172.21.15.59:0/750907033 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5d1600 0x5625fe5d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.071+0000 7fc9d8805740 1 --2- 172.21.15.59:0/750907033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5a58c0 0x5625fe5a7360 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.071+0000 7fc9d8805740 1 --2- 172.21.15.59:0/750907033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5a1870 0x5625fe5a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.071+0000 7fc9d8805740 1 -- 172.21.15.59:0/750907033 >> 172.21.15.59:0/750907033 conn(0x5625fe5a38f0 msgr2=0x5625fe5a51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.071+0000 7fc9d8805740 1 -- 172.21.15.59:0/750907033 shutdown_connections 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.071+0000 7fc9d8805740 1 -- 172.21.15.59:0/750907033 wait complete. 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.072+0000 7fc9d8805740 1 -- start start 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.072+0000 7fc9d8805740 1 Processor -- start 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5d1600 0x5625fe5ea320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe5ebd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5e8870 0x5625fe5d4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9d8805740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625fe4b4a60 con 0x5625fe5cc690 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9d8805740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625fe4c7ea0 con 0x5625fe5d1600 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9d8805740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625fe50b6b0 con 0x5625fe5e8870 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9d8805740 10 client.0 _ll_register_callbacks cb 0x5625fe4c7870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe5ebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe5ebd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48336/0 (socket says 172.21.15.59:48336) 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ceffd640 1 -- 172.21.15.59:0/2260072408 learned_addr learned my addr 172.21.15.59:0/2260072408 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5d1600 0x5625fe5ea320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9cffff640 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5e8870 0x5625fe5d4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ceffd640 1 -- 172.21.15.59:0/2260072408 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5e8870 msgr2=0x5625fe5d4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5e8870 0x5625fe5d4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ceffd640 1 -- 172.21.15.59:0/2260072408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5d1600 msgr2=0x5625fe5ea320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5d1600 0x5625fe5ea320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ceffd640 1 -- 172.21.15.59:0/2260072408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625fe4c8270 con 0x5625fe5cc690 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe5ebd50 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8081f20 tx=0x7fc9b80017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a0dcf8d994ff92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2260072408 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b80a7020 con 0x5625fe5cc690 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2260072408 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9b8076460 con 0x5625fe5cc690 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.073+0000 7fc9d8805740 1 -- 172.21.15.59:0/2260072408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc9b8030040 con 0x5625fe5cc690 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2260072408 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9b8079240 con 0x5625fe5cc690 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9d8805740 10 client.8844 fetch_fsmap learned FSMap version 79 2024-03-06T07:28:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9d8805740 1 -- 172.21.15.59:0/2260072408 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) v3 -- 0x5625fe2202d0 con 0x5625fe5cc690 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2260072408 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b8030040 con 0x5625fe5cc690 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2260072408 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 79) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc9b80793e0 con 0x5625fe5cc690 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9d8805740 10 client.8844 fetch_fsmap finished waiting for FSMap version 79 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9d8805740 10 client.8844 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9d8805740 20 client.8844 populate_metadata read hostname 'smithi059' 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x5625fe600000 0x5625fe6003e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9d8805740 4 client.8844 mds_command: new command op to 8676 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9d8805740 1 -- 172.21.15.59:0/2260072408 --> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625fe518bf0 con 0x5625fe600000 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x5625fe600000 0x5625fe6003e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9ae7fc640 10 client.8844 ms_handle_connect on v2:172.21.15.59:6834/1740714701 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.074+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x5625fe600000 0x5625fe6003e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c54f0bd17be8f9f0 server_cookie=fa1cde7b923554e3 in_seq=0 out_seq=0 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2260072408 <== mds.0 v2:172.21.15.59:6834/1740714701 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5625fe518bf0 con 0x5625fe600000 2024-03-06T07:28:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9ae7fc640 10 client.8844 handle_command_reply: tid=0 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9d8805740 1 client.8844 shutdown 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9d8805740 1 -- 172.21.15.59:0/2260072408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 msgr2=0x5625fe5ebd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe5ebd50 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8081f20 tx=0x7fc9b80017a0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9d8805740 1 -- 172.21.15.59:0/2260072408 shutdown_connections 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5e8870 0x5625fe5d4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x5625fe600000 0x5625fe6003e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe5ebd50 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2260072408 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5d1600 0x5625fe5ea320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9d8805740 1 -- 172.21.15.59:0/2260072408 >> 172.21.15.59:0/2260072408 conn(0x5625fe5a1870 msgr2=0x5625fe5eda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.075+0000 7fc9ae7fc640 0 client.8844 ms_handle_reset on v2:172.21.15.59:6834/1740714701 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.076+0000 7fc9d8805740 1 -- 172.21.15.59:0/2260072408 shutdown_connections 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.076+0000 7fc9d8805740 1 -- 172.21.15.59:0/2260072408 wait complete. 2024-03-06T07:28:52.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.076+0000 7fc9d8805740 20 client.8844 trim_cache size 0 max 16384 2024-03-06T07:28:52.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.092+0000 7fc9d8805740 1 Processor -- start 2024-03-06T07:28:52.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.092+0000 7fc9d8805740 1 -- start start 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.093+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5cc690 0x5625fe5d4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.093+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5d1600 0x5625fe5ff260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.093+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5e8870 0x5625fe5d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.093+0000 7fc9d8805740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d01ff7d0 con 0x5625fe5d1600 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.093+0000 7fc9d8805740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d0152150 con 0x5625fe5e8870 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.093+0000 7fc9d8805740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d006cc30 con 0x5625fe5cc690 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.093+0000 7fc9ceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5d1600 0x5625fe5ff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.093+0000 7fc9ceffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5d1600 0x5625fe5ff260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48346/0 (socket says 172.21.15.59:48346) 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.093+0000 7fc9ceffd640 1 -- 172.21.15.59:0/1209736107 learned_addr learned my addr 172.21.15.59:0/1209736107 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.093+0000 7fc9cffff640 1 --2- 172.21.15.59:0/1209736107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5e8870 0x5625fe5d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.093+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/1209736107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5cc690 0x5625fe5d4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.094+0000 7fc9ceffd640 1 -- 172.21.15.59:0/1209736107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5cc690 msgr2=0x5625fe5d4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.094+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/1209736107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5cc690 0x5625fe5d4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.094+0000 7fc9ceffd640 1 -- 172.21.15.59:0/1209736107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5e8870 msgr2=0x5625fe5d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.094+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/1209736107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5e8870 0x5625fe5d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.094+0000 7fc9ceffd640 1 -- 172.21.15.59:0/1209736107 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625fe5cf590 con 0x5625fe5d1600 2024-03-06T07:28:52.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.094+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/1209736107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5d1600 0x5625fe5ff260 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8078340 tx=0x7fc9b808f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670eaf89ce13cc33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.094+0000 7fc9c4ff9640 1 -- 172.21.15.59:0/1209736107 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b80a7070 con 0x5625fe5d1600 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.094+0000 7fc9c4ff9640 1 -- 172.21.15.59:0/1209736107 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b8030070 con 0x5625fe5d1600 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.094+0000 7fc9c4ff9640 1 -- 172.21.15.59:0/1209736107 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b80ab1d0 con 0x5625fe5d1600 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.094+0000 7fc9d8805740 1 -- 172.21.15.59:0/1209736107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5d1600 msgr2=0x5625fe5ff260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.094+0000 7fc9d8805740 1 --2- 172.21.15.59:0/1209736107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5d1600 0x5625fe5ff260 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8078340 tx=0x7fc9b808f070 comp rx=0 tx=0).stop 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.095+0000 7fc9d8805740 1 -- 172.21.15.59:0/1209736107 shutdown_connections 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.095+0000 7fc9d8805740 1 --2- 172.21.15.59:0/1209736107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5e8870 0x5625fe5d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.095+0000 7fc9d8805740 1 --2- 172.21.15.59:0/1209736107 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5d1600 0x5625fe5ff260 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.095+0000 7fc9d8805740 1 --2- 172.21.15.59:0/1209736107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5cc690 0x5625fe5d4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.095+0000 7fc9d8805740 1 -- 172.21.15.59:0/1209736107 >> 172.21.15.59:0/1209736107 conn(0x5625fe5a1870 msgr2=0x5625fe5c7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.095+0000 7fc9d8805740 1 -- 172.21.15.59:0/1209736107 shutdown_connections 2024-03-06T07:28:52.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.095+0000 7fc9d8805740 1 -- 172.21.15.59:0/1209736107 wait complete. 2024-03-06T07:28:52.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.095+0000 7fc9d8805740 1 -- start start 2024-03-06T07:28:52.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.095+0000 7fc9d8805740 1 Processor -- start 2024-03-06T07:28:52.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe65bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5d1600 0x5625fe5e6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9d8805740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5e8870 0x5625fe5e7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9d8805740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625fe4b4a60 con 0x5625fe5cc690 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9d8805740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625fe4c7ea0 con 0x5625fe5e8870 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9d8805740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5625fe5d30e0 con 0x5625fe5d1600 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9d8805740 10 client.0 _ll_register_callbacks cb 0x7fc9b8076120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9cf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe65bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9cf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe65bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48356/0 (socket says 172.21.15.59:48356) 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9cf7fe640 1 -- 172.21.15.59:0/2717428019 learned_addr learned my addr 172.21.15.59:0/2717428019 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9cffff640 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5e8870 0x5625fe5e7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5d1600 0x5625fe5e6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9cf7fe640 1 -- 172.21.15.59:0/2717428019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5d1600 msgr2=0x5625fe5e6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5d1600 0x5625fe5e6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9cf7fe640 1 -- 172.21.15.59:0/2717428019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5e8870 msgr2=0x5625fe5e7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5e8870 0x5625fe5e7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9cf7fe640 1 -- 172.21.15.59:0/2717428019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625fe2ea6b0 con 0x5625fe5cc690 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.096+0000 7fc9cffff640 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5e8870 0x5625fe5e7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9cf7fe640 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe65bc70 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fc9c002d570 tx=0x7fc9c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e10c419c5346b32 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2717428019 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c00256d0 con 0x5625fe5cc690 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2717428019 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9c0025870 con 0x5625fe5cc690 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9d8805740 1 -- 172.21.15.59:0/2717428019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc9b8030070 con 0x5625fe5cc690 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2717428019 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9c003d050 con 0x5625fe5cc690 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9d8805740 10 client.8850 fetch_fsmap learned FSMap version 79 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9d8805740 1 -- 172.21.15.59:0/2717428019 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) v3 -- 0x5625fe5d2180 con 0x5625fe5cc690 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2717428019 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b8030070 con 0x5625fe5cc690 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2717428019 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 79) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fc9c0040020 con 0x5625fe5cc690 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9d8805740 10 client.8850 fetch_fsmap finished waiting for FSMap version 79 2024-03-06T07:28:52.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9d8805740 10 client.8850 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9d8805740 20 client.8850 populate_metadata read hostname 'smithi059' 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x5625fe5d0ec0 0x5625fe5a0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9d8805740 4 client.8850 mds_command: new command op to 8676 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9d8805740 1 -- 172.21.15.59:0/2717428019 --> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5625fe512f40 con 0x5625fe5d0ec0 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.097+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x5625fe5d0ec0 0x5625fe5a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9ae7fc640 10 client.8850 ms_handle_connect on v2:172.21.15.59:6834/1740714701 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9ceffd640 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x5625fe5d0ec0 0x5625fe5a0510 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8457c37aaa688434 server_cookie=d32ad53975817837 in_seq=0 out_seq=0 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9ae7fc640 1 -- 172.21.15.59:0/2717428019 <== mds.0 v2:172.21.15.59:6834/1740714701 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5625fe512f40 con 0x5625fe5d0ec0 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9ae7fc640 10 client.8850 handle_command_reply: tid=0 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 1 client.8850 shutdown 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 1 -- 172.21.15.59:0/2717428019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 msgr2=0x5625fe65bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe65bc70 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fc9c002d570 tx=0x7fc9c0004730 comp rx=0 tx=0).stop 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 1 -- 172.21.15.59:0/2717428019 shutdown_connections 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5625fe5e8870 0x5625fe5e7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5625fe5d1600 0x5625fe5e6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x5625fe5d0ec0 0x5625fe5a0510 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 1 --2- 172.21.15.59:0/2717428019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x5625fe5cc690 0x5625fe65bc70 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 1 -- 172.21.15.59:0/2717428019 >> 172.21.15.59:0/2717428019 conn(0x5625fe5a1870 msgr2=0x5625fe5d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9ae7fc640 0 client.8850 ms_handle_reset on v2:172.21.15.59:6834/1740714701 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 1 -- 172.21.15.59:0/2717428019 shutdown_connections 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 1 -- 172.21.15.59:0/2717428019 wait complete. 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.098+0000 7fc9d8805740 20 client.8850 trim_cache size 0 max 16384 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 8676, 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:28:52.106 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:28:52.107 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:28:52.107 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 9.1747197620000005, 2024-03-06T07:28:52.107 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 79, 2024-03-06T07:28:52.107 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 126, 2024-03-06T07:28:52.107 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 126, 2024-03-06T07:28:52.107 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 14.508646075 2024-03-06T07:28:52.107 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 -- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc9b006b4d0 msgr2=0x7fc9b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc9b006b4d0 0x7fc9b006d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0009bc0 tx=0x7fc9c0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 -- 172.21.15.59:0/1711914051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d01554e0 msgr2=0x7fc9d01f7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d01554e0 0x7fc9d01f7310 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fc9b800c330 tx=0x7fc9b802e070 comp rx=0 tx=0).stop 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 -- 172.21.15.59:0/1711914051 shutdown_connections 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc9b006b4d0 0x7fc9b006d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9d01f7850 0x7fc9d01e75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9d01554e0 0x7fc9d01f7310 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 --2- 172.21.15.59:0/1711914051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9bc004800 0x7fc9d0142500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 -- 172.21.15.59:0/1711914051 >> 172.21.15.59:0/1711914051 conn(0x7fc9d009a9f0 msgr2=0x7fc9d0158dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 -- 172.21.15.59:0/1711914051 shutdown_connections 2024-03-06T07:28:52.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.099+0000 7fc9d5c22640 1 -- 172.21.15.59:0/1711914051 wait complete. 2024-03-06T07:28:52.111 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:52.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.211+0000 7f67a6bd0640 1 Processor -- start 2024-03-06T07:28:52.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.211+0000 7f67a6bd0640 1 -- start start 2024-03-06T07:28:52.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a6bd0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67a015dfe0 0x7f67a00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a6bd0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67a00d29c0 0x7f67a00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a6bd0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a00b3430 con 0x7f67a00cf830 2024-03-06T07:28:52.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a6bd0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a00b31e0 con 0x7f67a00d29c0 2024-03-06T07:28:52.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a6bd0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00b3d60 con 0x7f67a015dfe0 2024-03-06T07:28:52.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a516d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67a00d29c0 0x7f67a00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f6797fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f67a00cf830 0x7f67a00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44510/0 (socket says 172.21.15.59:44510) 2024-03-06T07:28:52.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a516d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67a00d29c0 0x7f67a00b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48362/0 (socket says 172.21.15.59:48362) 2024-03-06T07:28:52.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f6797fff640 1 -- 172.21.15.59:0/3174866761 learned_addr learned my addr 172.21.15.59:0/3174866761 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:52.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a496c640 1 --2- 172.21.15.59:0/3174866761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67a015dfe0 0x7f67a00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a516d640 1 -- 172.21.15.59:0/3174866761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67a015dfe0 msgr2=0x7f67a00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a516d640 1 --2- 172.21.15.59:0/3174866761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67a015dfe0 0x7f67a00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a516d640 1 -- 172.21.15.59:0/3174866761 >> v1:172.21.15.59:6789/0 conn(0x7f67a00cf830 legacy=0x7f67a00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67977fe640 1 -- 172.21.15.59:0/3174866761 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447821190 0 0) 0x7f67a00b3430 con 0x7f67a00cf830 2024-03-06T07:28:52.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.212+0000 7f67a516d640 1 -- 172.21.15.59:0/3174866761 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a00b3710 con 0x7f67a00d29c0 2024-03-06T07:28:52.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.213+0000 7f67a516d640 1 --2- 172.21.15.59:0/3174866761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67a00d29c0 0x7f67a00b41f0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f67980097a0 tx=0x7f679802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf6e38eb5d053b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.213+0000 7f67977fe640 1 -- 172.21.15.59:0/3174866761 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6798035020 con 0x7f67a00d29c0 2024-03-06T07:28:52.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.213+0000 7f67977fe640 1 -- 172.21.15.59:0/3174866761 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6798031830 con 0x7f67a00d29c0 2024-03-06T07:28:52.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.213+0000 7f67977fe640 1 -- 172.21.15.59:0/3174866761 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6798031c00 con 0x7f67a00d29c0 2024-03-06T07:28:52.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.214+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3174866761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67a00d29c0 msgr2=0x7f67a00b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.214+0000 7f67a6bd0640 1 --2- 172.21.15.59:0/3174866761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67a00d29c0 0x7f67a00b41f0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f67980097a0 tx=0x7f679802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.214+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3174866761 shutdown_connections 2024-03-06T07:28:52.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.214+0000 7f67a6bd0640 1 --2- 172.21.15.59:0/3174866761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67a00d29c0 0x7f67a00b41f0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.214+0000 7f67a6bd0640 1 --2- 172.21.15.59:0/3174866761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67a015dfe0 0x7f67a00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.214+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3174866761 >> 172.21.15.59:0/3174866761 conn(0x7f67a00d3cd0 msgr2=0x7f67a00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.214+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3174866761 shutdown_connections 2024-03-06T07:28:52.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.214+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3174866761 wait complete. 2024-03-06T07:28:52.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.216+0000 7f67a6bd0640 1 Processor -- start 2024-03-06T07:28:52.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.216+0000 7f67a6bd0640 1 -- start start 2024-03-06T07:28:52.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.217+0000 7f67a6bd0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67a00cf830 0x7f67a00b9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.217+0000 7f67a6bd0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67a00d29c0 0x7f67a00b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.217+0000 7f67a6bd0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67a015dfe0 0x7f67a00c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.217+0000 7f67a6bd0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00d1df0 con 0x7f67a00cf830 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.217+0000 7f67a6bd0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00d1c70 con 0x7f67a00d29c0 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.217+0000 7f67a6bd0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a015e3c0 con 0x7f67a015dfe0 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.217+0000 7f67a496c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67a00cf830 0x7f67a00b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.217+0000 7f67a496c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67a00cf830 0x7f67a00b9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48374/0 (socket says 172.21.15.59:48374) 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.217+0000 7f67a496c640 1 -- 172.21.15.59:0/3826861820 learned_addr learned my addr 172.21.15.59:0/3826861820 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f6797fff640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67a00d29c0 0x7f67a00b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67a516d640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67a015dfe0 0x7f67a00c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67a496c640 1 -- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67a015dfe0 msgr2=0x7f67a00c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67a496c640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67a015dfe0 0x7f67a00c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67a496c640 1 -- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67a00d29c0 msgr2=0x7f67a00b8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67a496c640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67a00d29c0 0x7f67a00b8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67a496c640 1 -- 172.21.15.59:0/3826861820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a00b7380 con 0x7f67a00cf830 2024-03-06T07:28:52.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67a496c640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67a00cf830 0x7f67a00b9220 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f679c009d10 tx=0x7f679c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67957fa640 1 -- 172.21.15.59:0/3826861820 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c018020 con 0x7f67a00cf830 2024-03-06T07:28:52.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67957fa640 1 -- 172.21.15.59:0/3826861820 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679c014480 con 0x7f67a00cf830 2024-03-06T07:28:52.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67957fa640 1 -- 172.21.15.59:0/3826861820 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c014830 con 0x7f67a00cf830 2024-03-06T07:28:52.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3826861820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a00b75f0 con 0x7f67a00cf830 2024-03-06T07:28:52.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.218+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3826861820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a00b7bb0 con 0x7f67a00cf830 2024-03-06T07:28:52.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.219+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3826861820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a004f330 con 0x7f67a00cf830 2024-03-06T07:28:52.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.220+0000 7f67957fa640 1 -- 172.21.15.59:0/3826861820 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f679c005d10 con 0x7f67a00cf830 2024-03-06T07:28:52.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.220+0000 7f67957fa640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678c06b4d0 0x7f678c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.220+0000 7f67957fa640 1 -- 172.21.15.59:0/3826861820 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f679c08ac30 con 0x7f67a00cf830 2024-03-06T07:28:52.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.220+0000 7f6797fff640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678c06b4d0 0x7f678c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.221+0000 7f6797fff640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678c06b4d0 0x7f678c06d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f67a00cde70 tx=0x7f6790007b50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.228+0000 7f67957fa640 1 -- 172.21.15.59:0/3826861820 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f679c022070 con 0x7f67a00cf830 2024-03-06T07:28:52.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.449+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3826861820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f67a00b9000 con 0x7f67a00cf830 2024-03-06T07:28:52.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.449+0000 7f67957fa640 1 -- 172.21.15.59:0/3826861820 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v79) v1 ==== 53+0+73 (secure 0 0 0) 0x7f679c010040 con 0x7f67a00cf830 2024-03-06T07:28:52.451 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:52.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678c06b4d0 msgr2=0x7f678c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678c06b4d0 0x7f678c06d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f67a00cde70 tx=0x7f6790007b50 comp rx=0 tx=0).stop 2024-03-06T07:28:52.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3826861820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67a00cf830 msgr2=0x7f67a00b9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67a00cf830 0x7f67a00b9220 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f679c009d10 tx=0x7f679c00c490 comp rx=0 tx=0).stop 2024-03-06T07:28:52.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3826861820 shutdown_connections 2024-03-06T07:28:52.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f678c06b4d0 0x7f678c06d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67a015dfe0 0x7f67a00c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67a00d29c0 0x7f67a00b8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 --2- 172.21.15.59:0/3826861820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67a00cf830 0x7f67a00b9220 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3826861820 >> 172.21.15.59:0/3826861820 conn(0x7f67a00d3cd0 msgr2=0x7f67a006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3826861820 shutdown_connections 2024-03-06T07:28:52.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:52.451+0000 7f67a6bd0640 1 -- 172.21.15.59:0/3826861820 wait complete. 2024-03-06T07:28:52.464 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:52.464 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:52.464 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:28:52.464 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:28:52.464 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:52.464 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:52.465 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:52.465 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:52.465 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:52.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.465 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.481 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42246sec preferred_lft 42246sec 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:28:52.482 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:52.483 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:52.484 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:28:52.484 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:52.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.484 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:52.546 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:28:52.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.548 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:28:52.602 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:28:52.602 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:52.602 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:28:52.602 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:28:52.602 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:52.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.603 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:28:52.603 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:28:52.603 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:28:52.603 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:28:52.603 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:28:52.603 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:28:52.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.698 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:28:52.712 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:28:52.713 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:28:52.713 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:52.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.713 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:28:52.779 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:28:52.779 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:52.779 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:52.833 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:52.833 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:52.834 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:52.888 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:28:52.954 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:28:53.008 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:28:53.009 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:28:53.010 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:53.011 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:53.011 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:53.011 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:53.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.012 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:28:53.065 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:28:53.065 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:28:53.108 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:28:53.135 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:28:53.190 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:28:53.190 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:28:53.190 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:28:53.191 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:53.192 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:53.193 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:53.193 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:53.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.193 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:28:53.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:28:53.199+0000 7f2d8540f480 -1 init, newargv = 0x5590c06a9400 newargc=15 2024-03-06T07:28:53.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[78753]: starting ceph client 2024-03-06T07:28:53.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[78753]: starting fuse 2024-03-06T07:28:53.216 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:28:53.217 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:28:53.217 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:28:53.218 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:28:53.349 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.78753.asok 2024-03-06T07:28:53.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:28:53.361 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.78753.asok 2024-03-06T07:28:53.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.361 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.78753.asok status 2024-03-06T07:28:53.500 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "78753", 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 8868, 2024-03-06T07:28:53.501 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 8868 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3499039453 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:28:53.502 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 3499039453 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.8868 192.168.0.1:0/3499039453", 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/3499039453", 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 79, 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 126, 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:28:53.503 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:28:53.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.509 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:28:53.509 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:53.524 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:28:53.525 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:53.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:53.525 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:53.595 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:28:53.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.737+0000 7f389d555640 1 Processor -- start 2024-03-06T07:28:53.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.738+0000 7f389d555640 1 -- start start 2024-03-06T07:28:53.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.738+0000 7f389d555640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f38980cff40 0x7f38980b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.738+0000 7f389d555640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3898158ac0 con 0x7f38981586e0 2024-03-06T07:28:53.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.738+0000 7f389d555640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389815b2a0 con 0x7f38980cfa90 2024-03-06T07:28:53.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.738+0000 7f389d555640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38980d0390 con 0x7f38980cff40 2024-03-06T07:28:53.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.738+0000 7f3896ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f38981586e0 0x7f38980aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46320/0 (socket says 172.21.15.59:46320) 2024-03-06T07:28:53.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.738+0000 7f3896ffd640 1 -- 172.21.15.59:0/755032860 learned_addr learned my addr 172.21.15.59:0/755032860 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:28:53.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f38977fe640 1 --2- 172.21.15.59:0/755032860 >> v2:172.21.15.164:3300/0 conn(0x7f38980cff40 0x7f38980b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184805192 0 0) 0x7f3898158ac0 con 0x7f38981586e0 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3880003520 con 0x7f38981586e0 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297024302 0 0) 0x7f389815b2a0 con 0x7f38980cfa90 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3898158ac0 con 0x7f38980cfa90 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926927798 0 0) 0x7f3880003520 con 0x7f38981586e0 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 >> v2:172.21.15.164:3300/0 conn(0x7f38980cff40 msgr2=0x7f38980b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 --2- 172.21.15.59:0/755032860 >> v2:172.21.15.164:3300/0 conn(0x7f38980cff40 0x7f38980b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 >> v1:172.21.15.164:6789/0 conn(0x7f38980cfa90 legacy=0x7f38980b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389815b540 con 0x7f38981586e0 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f38977fe640 1 --2- 172.21.15.59:0/755032860 >> v2:172.21.15.164:3300/0 conn(0x7f38980cff40 0x7f38980b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f388c001670 con 0x7f38981586e0 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 >> v1:172.21.15.59:6789/0 conn(0x7f38981586e0 legacy=0x7f38980aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3880003b40 0x7f3880003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3880004800 0x7f3880004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800051c0 0x7f388000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38980d0390 con 0x7f3880004800 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388000c040 con 0x7f38800051c0 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388000c240 con 0x7f3880003b40 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f38967fc640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3880004800 0x7f3880004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f3896ffd640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3880003b40 0x7f3880003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.739+0000 7f38977fe640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800051c0 0x7f388000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.740+0000 7f38967fc640 1 -- 172.21.15.59:0/755032860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3880003b40 msgr2=0x7f3880003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.740+0000 7f38967fc640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3880003b40 0x7f3880003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.740+0000 7f38967fc640 1 -- 172.21.15.59:0/755032860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800051c0 msgr2=0x7f388000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.740+0000 7f38967fc640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38800051c0 0x7f388000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.740+0000 7f38967fc640 1 -- 172.21.15.59:0/755032860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3880003520 con 0x7f3880004800 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.740+0000 7f38977fe640 1 -- 172.21.15.59:0/755032860 reap_dead start 2024-03-06T07:28:53.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.740+0000 7f3896ffd640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3880003b40 0x7f3880003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:53.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.741+0000 7f38967fc640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3880004800 0x7f3880004c80 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f38880018d0 tx=0x7f3880005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178cf0b1ede90343 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.741+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888037070 con 0x7f3880004800 2024-03-06T07:28:53.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.741+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38880052d0 con 0x7f3880004800 2024-03-06T07:28:53.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.741+0000 7f3895ffb640 1 -- 172.21.15.59:0/755032860 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38880014f0 con 0x7f3880004800 2024-03-06T07:28:53.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.741+0000 7f389d555640 1 -- 172.21.15.59:0/755032860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3880004800 msgr2=0x7f3880004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.741+0000 7f389d555640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3880004800 0x7f3880004c80 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f38880018d0 tx=0x7f3880005830 comp rx=0 tx=0).stop 2024-03-06T07:28:53.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.741+0000 7f389d555640 1 -- 172.21.15.59:0/755032860 shutdown_connections 2024-03-06T07:28:53.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.741+0000 7f389d555640 1 --2- 172.21.15.59:0/755032860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3880004800 0x7f3880004c80 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.741+0000 7f389d555640 1 -- 172.21.15.59:0/755032860 >> 172.21.15.59:0/755032860 conn(0x7f389806b190 msgr2=0x7f389806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:53.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.741+0000 7f389d555640 1 -- 172.21.15.59:0/755032860 shutdown_connections 2024-03-06T07:28:53.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.741+0000 7f389d555640 1 -- 172.21.15.59:0/755032860 wait complete. 2024-03-06T07:28:53.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.743+0000 7f389d555640 1 Processor -- start 2024-03-06T07:28:53.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.743+0000 7f389d555640 1 -- start start 2024-03-06T07:28:53.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f389d555640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3880004800 0x7f38980b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f389d555640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f389814a450 0x7f38980b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f389d555640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f389806c910 0x7f389806cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f389d555640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389815bde0 con 0x7f389806c910 2024-03-06T07:28:53.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f389d555640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389815bb00 con 0x7f389814a450 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f389d555640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3898071350 con 0x7f3880004800 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f38977fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f389806c910 0x7f389806cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f38977fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f389806c910 0x7f389806cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40774/0 (socket says 172.21.15.59:40774) 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f38977fe640 1 -- 172.21.15.59:0/1395728560 learned_addr learned my addr 172.21.15.59:0/1395728560 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f38967fc640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f389814a450 0x7f38980b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f3896ffd640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3880004800 0x7f38980b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f38977fe640 1 -- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3880004800 msgr2=0x7f38980b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f38977fe640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3880004800 0x7f38980b0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f38977fe640 1 -- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f389814a450 msgr2=0x7f38980b0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f38977fe640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f389814a450 0x7f38980b0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f38977fe640 1 -- 172.21.15.59:0/1395728560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38980b0870 con 0x7f389806c910 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.744+0000 7f3896ffd640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3880004800 0x7f38980b0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:53.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.745+0000 7f38977fe640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f389806c910 0x7f389806cd90 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f38980d0910 tx=0x7f389000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.745+0000 7f387f7fe640 1 -- 172.21.15.59:0/1395728560 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389000fdf0 con 0x7f389806c910 2024-03-06T07:28:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.745+0000 7f387f7fe640 1 -- 172.21.15.59:0/1395728560 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3890003990 con 0x7f389806c910 2024-03-06T07:28:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.745+0000 7f389d555640 1 -- 172.21.15.59:0/1395728560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38980a9fb0 con 0x7f389806c910 2024-03-06T07:28:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.745+0000 7f387f7fe640 1 -- 172.21.15.59:0/1395728560 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3890003d80 con 0x7f389806c910 2024-03-06T07:28:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.745+0000 7f389d555640 1 -- 172.21.15.59:0/1395728560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38980aa570 con 0x7f389806c910 2024-03-06T07:28:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.746+0000 7f389d555640 1 -- 172.21.15.59:0/1395728560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38980aba40 con 0x7f389806c910 2024-03-06T07:28:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.746+0000 7f387f7fe640 1 -- 172.21.15.59:0/1395728560 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f389001f020 con 0x7f389806c910 2024-03-06T07:28:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.747+0000 7f387f7fe640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388406b4d0 0x7f388406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.747+0000 7f387f7fe640 1 -- 172.21.15.59:0/1395728560 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f389008a750 con 0x7f389806c910 2024-03-06T07:28:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.747+0000 7f3896ffd640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388406b4d0 0x7f388406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:53.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.748+0000 7f3896ffd640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388406b4d0 0x7f388406d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f388c000f60 tx=0x7f388c000ed0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:53.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.752+0000 7f387f7fe640 1 -- 172.21.15.59:0/1395728560 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3890010040 con 0x7f389806c910 2024-03-06T07:28:53.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:53.972+0000 7f389d555640 1 -- 172.21.15.59:0/1395728560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f38980732b0 con 0x7f389806c910 2024-03-06T07:28:54.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.969+0000 7f387f7fe640 1 -- 172.21.15.59:0/1395728560 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v80) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3890059530 con 0x7f389806c910 2024-03-06T07:28:54.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.971+0000 7f389d555640 1 -- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388406b4d0 msgr2=0x7f388406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.971+0000 7f389d555640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388406b4d0 0x7f388406d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f388c000f60 tx=0x7f388c000ed0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.971+0000 7f389d555640 1 -- 172.21.15.59:0/1395728560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f389806c910 msgr2=0x7f389806cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.971+0000 7f389d555640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f389806c910 0x7f389806cd90 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f38980d0910 tx=0x7f389000f580 comp rx=0 tx=0).stop 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.971+0000 7f389d555640 1 -- 172.21.15.59:0/1395728560 shutdown_connections 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.971+0000 7f389d555640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f388406b4d0 0x7f388406d990 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.971+0000 7f389d555640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f389806c910 0x7f389806cd90 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.971+0000 7f389d555640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f389814a450 0x7f38980b0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.971+0000 7f389d555640 1 --2- 172.21.15.59:0/1395728560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3880004800 0x7f38980b0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.971+0000 7f389d555640 1 -- 172.21.15.59:0/1395728560 >> 172.21.15.59:0/1395728560 conn(0x7f389806b190 msgr2=0x7f3898157610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.971+0000 7f389d555640 1 -- 172.21.15.59:0/1395728560 shutdown_connections 2024-03-06T07:28:54.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:54.972+0000 7f389d555640 1 -- 172.21.15.59:0/1395728560 wait complete. 2024-03-06T07:28:54.988 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:55.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.087+0000 7f626473a640 1 Processor -- start 2024-03-06T07:28:55.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.087+0000 7f626473a640 1 -- start start 2024-03-06T07:28:55.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.088+0000 7f626473a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f625c0d22b0 0x7f625c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.088+0000 7f626473a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f625c0b41f0 0x7f625c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.088+0000 7f626473a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625c0b3430 con 0x7f625c15dfe0 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.088+0000 7f626473a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f625c0d2c50 con 0x7f625c0d22b0 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.088+0000 7f626473a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0b31e0 con 0x7f625c0b41f0 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.088+0000 7f6261cd5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f625c0d22b0 0x7f625c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.088+0000 7f6261cd5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f625c0d22b0 0x7f625c0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40788/0 (socket says 172.21.15.59:40788) 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.088+0000 7f6261cd5640 1 -- 172.21.15.59:0/2352489861 learned_addr learned my addr 172.21.15.59:0/2352489861 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.088+0000 7f6262cd7640 1 --2- 172.21.15.59:0/2352489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f625c0b41f0 0x7f625c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.089+0000 7f6261cd5640 1 -- 172.21.15.59:0/2352489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f625c0b41f0 msgr2=0x7f625c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.089+0000 7f6261cd5640 1 --2- 172.21.15.59:0/2352489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f625c0b41f0 0x7f625c0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.089+0000 7f6261cd5640 1 -- 172.21.15.59:0/2352489861 >> v1:172.21.15.164:6789/0 conn(0x7f625c15dfe0 legacy=0x7f625c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.089+0000 7f62614d4640 1 -- 172.21.15.59:0/2352489861 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754734984 0 0) 0x7f625c0b3430 con 0x7f625c15dfe0 2024-03-06T07:28:55.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.089+0000 7f6261cd5640 1 -- 172.21.15.59:0/2352489861 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c0b3710 con 0x7f625c0d22b0 2024-03-06T07:28:55.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.089+0000 7f6261cd5640 1 --2- 172.21.15.59:0/2352489861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f625c0d22b0 0x7f625c0d2710 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f624c009870 tx=0x7f624c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff88fae2f619f74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.090+0000 7f62614d4640 1 -- 172.21.15.59:0/2352489861 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c034070 con 0x7f625c0d22b0 2024-03-06T07:28:55.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.090+0000 7f62614d4640 1 -- 172.21.15.59:0/2352489861 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c030840 con 0x7f625c0d22b0 2024-03-06T07:28:55.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.090+0000 7f62614d4640 1 -- 172.21.15.59:0/2352489861 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c030c10 con 0x7f625c0d22b0 2024-03-06T07:28:55.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.090+0000 7f626473a640 1 -- 172.21.15.59:0/2352489861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f625c0d22b0 msgr2=0x7f625c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.090+0000 7f626473a640 1 --2- 172.21.15.59:0/2352489861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f625c0d22b0 0x7f625c0d2710 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f624c009870 tx=0x7f624c02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:55.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.090+0000 7f626473a640 1 -- 172.21.15.59:0/2352489861 shutdown_connections 2024-03-06T07:28:55.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.090+0000 7f626473a640 1 --2- 172.21.15.59:0/2352489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f625c0b41f0 0x7f625c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.090+0000 7f626473a640 1 --2- 172.21.15.59:0/2352489861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f625c0d22b0 0x7f625c0d2710 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.090+0000 7f626473a640 1 -- 172.21.15.59:0/2352489861 >> 172.21.15.59:0/2352489861 conn(0x7f625c0d3cd0 msgr2=0x7f625c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.090+0000 7f626473a640 1 -- 172.21.15.59:0/2352489861 shutdown_connections 2024-03-06T07:28:55.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.090+0000 7f626473a640 1 -- 172.21.15.59:0/2352489861 wait complete. 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.091+0000 7f626473a640 1 Processor -- start 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.091+0000 7f626473a640 1 -- start start 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.092+0000 7f626473a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f625c0b41f0 0x7f625c09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.092+0000 7f626473a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f625c0d22b0 0x7f625c09f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.092+0000 7f626473a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f625c15dfe0 0x7f625c0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.092+0000 7f626473a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0cfa00 con 0x7f625c15dfe0 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.092+0000 7f626473a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0b3d60 con 0x7f625c0d22b0 2024-03-06T07:28:55.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.092+0000 7f626473a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625c0d1c70 con 0x7f625c0b41f0 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.092+0000 7f6262cd7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f625c15dfe0 0x7f625c0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.092+0000 7f6262cd7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f625c15dfe0 0x7f625c0cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40800/0 (socket says 172.21.15.59:40800) 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.092+0000 7f6262cd7640 1 -- 172.21.15.59:0/198033810 learned_addr learned my addr 172.21.15.59:0/198033810 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.092+0000 7f6261cd5640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f625c0d22b0 0x7f625c09f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.092+0000 7f62624d6640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f625c0b41f0 0x7f625c09ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.093+0000 7f6262cd7640 1 -- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f625c0b41f0 msgr2=0x7f625c09ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.093+0000 7f6262cd7640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f625c0b41f0 0x7f625c09ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.093+0000 7f6262cd7640 1 -- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f625c0d22b0 msgr2=0x7f625c09f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.093+0000 7f6262cd7640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f625c0d22b0 0x7f625c09f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.093+0000 7f6262cd7640 1 -- 172.21.15.59:0/198033810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625c09fa30 con 0x7f625c15dfe0 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.093+0000 7f6262cd7640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f625c15dfe0 0x7f625c0cbdf0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f625400efc0 tx=0x7f625400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.093+0000 7f6252ffd640 1 -- 172.21.15.59:0/198033810 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254018020 con 0x7f625c15dfe0 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.093+0000 7f6252ffd640 1 -- 172.21.15.59:0/198033810 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6254014480 con 0x7f625c15dfe0 2024-03-06T07:28:55.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.093+0000 7f626473a640 1 -- 172.21.15.59:0/198033810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625c09fd10 con 0x7f625c15dfe0 2024-03-06T07:28:55.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.093+0000 7f6252ffd640 1 -- 172.21.15.59:0/198033810 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254014870 con 0x7f625c15dfe0 2024-03-06T07:28:55.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.093+0000 7f626473a640 1 -- 172.21.15.59:0/198033810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625c0a02d0 con 0x7f625c15dfe0 2024-03-06T07:28:55.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.094+0000 7f626473a640 1 -- 172.21.15.59:0/198033810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625c04f330 con 0x7f625c15dfe0 2024-03-06T07:28:55.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.095+0000 7f6252ffd640 1 -- 172.21.15.59:0/198033810 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6254005d10 con 0x7f625c15dfe0 2024-03-06T07:28:55.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.095+0000 7f6252ffd640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f624006b4d0 0x7f624006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.095+0000 7f6252ffd640 1 -- 172.21.15.59:0/198033810 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f625408abe0 con 0x7f625c15dfe0 2024-03-06T07:28:55.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.095+0000 7f62624d6640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f624006b4d0 0x7f624006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.096+0000 7f62624d6640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f624006b4d0 0x7f624006d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6258000f60 tx=0x7f6258009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.099+0000 7f6252ffd640 1 -- 172.21.15.59:0/198033810 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6254022070 con 0x7f625c15dfe0 2024-03-06T07:28:55.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.321+0000 7f626473a640 1 -- 172.21.15.59:0/198033810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f625c0b2c70 con 0x7f625c15dfe0 2024-03-06T07:28:55.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.321+0000 7f6252ffd640 1 -- 172.21.15.59:0/198033810 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5001 (secure 0 0 0) 0x7f6254010040 con 0x7f625c15dfe0 2024-03-06T07:28:55.323 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:28:55.323 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1772616514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1772616514},{"type":"v1","addr":"172.21.15.164:6836","nonce":1772616514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":7816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3891024511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3891024511},{"type":"v1","addr":"172.21.15.164:6837","nonce":3891024511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:28:54.971838+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:creating","state_seq":1,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:28:55.323 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 81 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 -- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f624006b4d0 msgr2=0x7f624006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f624006b4d0 0x7f624006d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6258000f60 tx=0x7f6258009040 comp rx=0 tx=0).stop 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 -- 172.21.15.59:0/198033810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f625c15dfe0 msgr2=0x7f625c0cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f625c15dfe0 0x7f625c0cbdf0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f625400efc0 tx=0x7f625400c490 comp rx=0 tx=0).stop 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 -- 172.21.15.59:0/198033810 shutdown_connections 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f624006b4d0 0x7f624006d990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f625c15dfe0 0x7f625c0cbdf0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f625c0d22b0 0x7f625c09f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 --2- 172.21.15.59:0/198033810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f625c0b41f0 0x7f625c09ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 -- 172.21.15.59:0/198033810 >> 172.21.15.59:0/198033810 conn(0x7f625c0d3cd0 msgr2=0x7f625c06c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 -- 172.21.15.59:0/198033810 shutdown_connections 2024-03-06T07:28:55.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:28:55.323+0000 7f626473a640 1 -- 172.21.15.59:0/198033810 wait complete. 2024-03-06T07:28:55.337 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:28:58.588 INFO:tasks.ceph.mds.c.smithi059.stderr:2024-03-06T07:28:58.586+0000 7f2a20acc640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:29:00.338 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.451+0000 7f88b89f7640 1 Processor -- start 2024-03-06T07:29:00.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.451+0000 7f88b89f7640 1 -- start start 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.452+0000 7f88b89f7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f88b41424f0 0x7f88b40cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.452+0000 7f88b89f7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f88b406da40 0x7f88b415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.452+0000 7f88b89f7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b40cc980 con 0x7f88b40cd7a0 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.452+0000 7f88b89f7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b40cd220 con 0x7f88b406da40 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.452+0000 7f88b89f7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b40ccc60 con 0x7f88b41424f0 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.452+0000 7f88b2d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f88b406da40 0x7f88b415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.452+0000 7f88b2d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f88b406da40 0x7f88b415cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40812/0 (socket says 172.21.15.59:40812) 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.452+0000 7f88b2d9d640 1 -- 172.21.15.59:0/1870671783 learned_addr learned my addr 172.21.15.59:0/1870671783 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.452+0000 7f88b259c640 1 --2- 172.21.15.59:0/1870671783 >> v2:172.21.15.164:3300/0 conn(0x7f88b41424f0 0x7f88b40cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.453+0000 7f88b2d9d640 1 -- 172.21.15.59:0/1870671783 >> v2:172.21.15.164:3300/0 conn(0x7f88b41424f0 msgr2=0x7f88b40cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.453+0000 7f88b2d9d640 1 --2- 172.21.15.59:0/1870671783 >> v2:172.21.15.164:3300/0 conn(0x7f88b41424f0 0x7f88b40cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.453+0000 7f88b2d9d640 1 -- 172.21.15.59:0/1870671783 >> v1:172.21.15.164:6789/0 conn(0x7f88b40cd7a0 legacy=0x7f88b406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.453+0000 7f88b2d9d640 1 -- 172.21.15.59:0/1870671783 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40ccf40 con 0x7f88b406da40 2024-03-06T07:29:00.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.453+0000 7f88b2d9d640 1 --2- 172.21.15.59:0/1870671783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b406da40 0x7f88b415cef0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f88b406c6d0 tx=0x7f88ac02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6aaa1d61e5a62c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.454+0000 7f88b1d9b640 1 -- 172.21.15.59:0/1870671783 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac02cb20 con 0x7f88b406da40 2024-03-06T07:29:00.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.454+0000 7f88b1d9b640 1 -- 172.21.15.59:0/1870671783 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ac02ccc0 con 0x7f88b406da40 2024-03-06T07:29:00.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.454+0000 7f88b1d9b640 1 -- 172.21.15.59:0/1870671783 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac034610 con 0x7f88b406da40 2024-03-06T07:29:00.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.454+0000 7f88b89f7640 1 -- 172.21.15.59:0/1870671783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b406da40 msgr2=0x7f88b415cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.454+0000 7f88b89f7640 1 --2- 172.21.15.59:0/1870671783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b406da40 0x7f88b415cef0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f88b406c6d0 tx=0x7f88ac02bf80 comp rx=0 tx=0).stop 2024-03-06T07:29:00.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.454+0000 7f88b89f7640 1 -- 172.21.15.59:0/1870671783 shutdown_connections 2024-03-06T07:29:00.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.454+0000 7f88b89f7640 1 --2- 172.21.15.59:0/1870671783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b406da40 0x7f88b415cef0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.454+0000 7f88b89f7640 1 --2- 172.21.15.59:0/1870671783 >> v2:172.21.15.164:3300/0 conn(0x7f88b41424f0 0x7f88b40cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.454+0000 7f88b89f7640 1 -- 172.21.15.59:0/1870671783 >> 172.21.15.59:0/1870671783 conn(0x7f88b40cf8a0 msgr2=0x7f88b40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.454+0000 7f88b89f7640 1 -- 172.21.15.59:0/1870671783 shutdown_connections 2024-03-06T07:29:00.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.454+0000 7f88b89f7640 1 -- 172.21.15.59:0/1870671783 wait complete. 2024-03-06T07:29:00.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.455+0000 7f88b89f7640 1 Processor -- start 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.455+0000 7f88b89f7640 1 -- start start 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.456+0000 7f88b89f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88b406da40 0x7f88b4154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.456+0000 7f88b89f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88b40cd7a0 0x7f88b40ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.456+0000 7f88b89f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b41424f0 0x7f88b4159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.456+0000 7f88b89f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4071ce0 con 0x7f88b41424f0 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.456+0000 7f88b89f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4071b60 con 0x7f88b406da40 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.456+0000 7f88b89f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4071e60 con 0x7f88b40cd7a0 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.456+0000 7f88b2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b41424f0 0x7f88b4159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.456+0000 7f88b2d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b41424f0 0x7f88b4159aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40828/0 (socket says 172.21.15.59:40828) 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.456+0000 7f88b2d9d640 1 -- 172.21.15.59:0/1152867306 learned_addr learned my addr 172.21.15.59:0/1152867306 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88b259c640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88b406da40 0x7f88b4154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88abfff640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88b40cd7a0 0x7f88b40ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88b2d9d640 1 -- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88b40cd7a0 msgr2=0x7f88b40ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88b2d9d640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88b40cd7a0 0x7f88b40ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88b2d9d640 1 -- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88b406da40 msgr2=0x7f88b4154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88b2d9d640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88b406da40 0x7f88b4154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88b2d9d640 1 -- 172.21.15.59:0/1152867306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b413e550 con 0x7f88b41424f0 2024-03-06T07:29:00.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88b259c640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88b406da40 0x7f88b4154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:00.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88b2d9d640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b41424f0 0x7f88b4159aa0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f88ac02c590 tx=0x7f88ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88aaffd640 1 -- 172.21.15.59:0/1152867306 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac043020 con 0x7f88b41424f0 2024-03-06T07:29:00.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88aaffd640 1 -- 172.21.15.59:0/1152867306 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ac03e450 con 0x7f88b41424f0 2024-03-06T07:29:00.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88b89f7640 1 -- 172.21.15.59:0/1152867306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b40b3570 con 0x7f88b41424f0 2024-03-06T07:29:00.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88aaffd640 1 -- 172.21.15.59:0/1152867306 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac009510 con 0x7f88b41424f0 2024-03-06T07:29:00.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.457+0000 7f88b89f7640 1 -- 172.21.15.59:0/1152867306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b40adda0 con 0x7f88b41424f0 2024-03-06T07:29:00.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.458+0000 7f88b89f7640 1 -- 172.21.15.59:0/1152867306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b404f330 con 0x7f88b41424f0 2024-03-06T07:29:00.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.459+0000 7f88aaffd640 1 -- 172.21.15.59:0/1152867306 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88ac046020 con 0x7f88b41424f0 2024-03-06T07:29:00.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.459+0000 7f88aaffd640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f889406b4d0 0x7f889406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.459+0000 7f88aaffd640 1 -- 172.21.15.59:0/1152867306 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88ac0acd70 con 0x7f88b41424f0 2024-03-06T07:29:00.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.459+0000 7f88b259c640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f889406b4d0 0x7f889406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.460+0000 7f88b259c640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f889406b4d0 0x7f889406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f88a4006fd0 tx=0x7f88a4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.463+0000 7f88aaffd640 1 -- 172.21.15.59:0/1152867306 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88ac04d020 con 0x7f88b41424f0 2024-03-06T07:29:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.682+0000 7f88b89f7640 1 -- 172.21.15.59:0/1152867306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88b40d0620 con 0x7f88b41424f0 2024-03-06T07:29:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.682+0000 7f88aaffd640 1 -- 172.21.15.59:0/1152867306 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5004 (secure 0 0 0) 0x7f88ac07bbf0 con 0x7f88b41424f0 2024-03-06T07:29:00.684 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:00.684 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1772616514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1772616514},{"type":"v1","addr":"172.21.15.164:6836","nonce":1772616514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":7816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3891024511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3891024511},{"type":"v1","addr":"172.21.15.164:6837","nonce":3891024511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:28:55.978187+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8673,8676]},"id":9}]} 2024-03-06T07:29:00.685 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 82 2024-03-06T07:29:00.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 -- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f889406b4d0 msgr2=0x7f889406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f889406b4d0 0x7f889406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f88a4006fd0 tx=0x7f88a4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:00.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 -- 172.21.15.59:0/1152867306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b41424f0 msgr2=0x7f88b4159aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b41424f0 0x7f88b4159aa0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f88ac02c590 tx=0x7f88ac004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:00.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 -- 172.21.15.59:0/1152867306 shutdown_connections 2024-03-06T07:29:00.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f889406b4d0 0x7f889406d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f88b41424f0 0x7f88b4159aa0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88b40cd7a0 0x7f88b40ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 --2- 172.21.15.59:0/1152867306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88b406da40 0x7f88b4154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 -- 172.21.15.59:0/1152867306 >> 172.21.15.59:0/1152867306 conn(0x7f88b40cf8a0 msgr2=0x7f88b406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 -- 172.21.15.59:0/1152867306 shutdown_connections 2024-03-06T07:29:00.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.684+0000 7f88b89f7640 1 -- 172.21.15.59:0/1152867306 wait complete. 2024-03-06T07:29:00.699 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:00.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.799+0000 7f3f96034640 1 Processor -- start 2024-03-06T07:29:00.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.799+0000 7f3f96034640 1 -- start start 2024-03-06T07:29:00.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.800+0000 7f3f96034640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015dfe0 0x7f3f900cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.800+0000 7f3f96034640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f900b3430 con 0x7f3f900d29c0 2024-03-06T07:29:00.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.800+0000 7f3f96034640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f900b3710 con 0x7f3f900cf830 2024-03-06T07:29:00.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.800+0000 7f3f96034640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f900b31e0 con 0x7f3f9015dfe0 2024-03-06T07:29:00.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8ffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3f900d29c0 0x7f3f900b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46322/0 (socket says 172.21.15.59:46322) 2024-03-06T07:29:00.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8ffff640 1 -- 172.21.15.59:0/1360561219 learned_addr learned my addr 172.21.15.59:0/1360561219 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:00.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8f7fe640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015dfe0 0x7f3f900cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279881105 0 0) 0x7f3f900b3430 con 0x7f3f900d29c0 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f78003520 con 0x7f3f900d29c0 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184193088 0 0) 0x7f3f900b3710 con 0x7f3f900cf830 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f900b3430 con 0x7f3f900cf830 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4275044917 0 0) 0x7f3f78003520 con 0x7f3f900d29c0 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015dfe0 msgr2=0x7f3f900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015dfe0 0x7f3f900cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 >> v1:172.21.15.164:6789/0 conn(0x7f3f900cf830 legacy=0x7f3f900d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f900b39f0 con 0x7f3f900d29c0 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f3f88001670 con 0x7f3f900d29c0 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 >> v1:172.21.15.59:6789/0 conn(0x7f3f900d29c0 legacy=0x7f3f900b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f78003b70 0x7f3f78003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015dfe0 0x7f3f78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 0x7f3f780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f900b31e0 con 0x7f3f78004d40 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7800ba70 con 0x7f3f78003b70 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7800bc20 con 0x7f3f9015dfe0 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8ffff640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 0x7f3f780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8f7fe640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f78003b70 0x7f3f78003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.801+0000 7f3f8effd640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015dfe0 0x7f3f78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.802+0000 7f3f8ffff640 1 -- 172.21.15.59:0/1360561219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015dfe0 msgr2=0x7f3f78004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.802+0000 7f3f8ffff640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015dfe0 0x7f3f78004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.802+0000 7f3f8ffff640 1 -- 172.21.15.59:0/1360561219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f78003b70 msgr2=0x7f3f78003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.802+0000 7f3f8ffff640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f78003b70 0x7f3f78003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.802+0000 7f3f8ffff640 1 -- 172.21.15.59:0/1360561219 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f78003520 con 0x7f3f78004d40 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.802+0000 7f3f8ffff640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 0x7f3f780051c0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f3f8800c190 tx=0x7f3f78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8cc6a97c5a606cf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.803+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88007cc0 con 0x7f3f78004d40 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.803+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f88030070 con 0x7f3f78004d40 2024-03-06T07:29:00.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.803+0000 7f3f8e7fc640 1 -- 172.21.15.59:0/1360561219 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88005d30 con 0x7f3f78004d40 2024-03-06T07:29:00.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.803+0000 7f3f96034640 1 -- 172.21.15.59:0/1360561219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 msgr2=0x7f3f780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.803+0000 7f3f96034640 1 --2- 172.21.15.59:0/1360561219 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 0x7f3f780051c0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f3f8800c190 tx=0x7f3f78005830 comp rx=0 tx=0).stop 2024-03-06T07:29:00.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.803+0000 7f3f8ffff640 1 -- 172.21.15.59:0/1360561219 reap_dead start 2024-03-06T07:29:00.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.803+0000 7f3f96034640 1 -- 172.21.15.59:0/1360561219 shutdown_connections 2024-03-06T07:29:00.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.803+0000 7f3f96034640 1 -- 172.21.15.59:0/1360561219 >> 172.21.15.59:0/1360561219 conn(0x7f3f900d3cd0 msgr2=0x7f3f900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.803+0000 7f3f96034640 1 -- 172.21.15.59:0/1360561219 shutdown_connections 2024-03-06T07:29:00.808 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.803+0000 7f3f96034640 1 -- 172.21.15.59:0/1360561219 wait complete. 2024-03-06T07:29:00.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.804+0000 7f3f96034640 1 Processor -- start 2024-03-06T07:29:00.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.805+0000 7f3f96034640 1 -- start start 2024-03-06T07:29:00.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.805+0000 7f3f96034640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 0x7f3f901f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.805+0000 7f3f96034640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f901584c0 0x7f3f90158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.805+0000 7f3f96034640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015e770 0x7f3f9015ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.805+0000 7f3f96034640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9015e3c0 con 0x7f3f78004d40 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.805+0000 7f3f96034640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f900b3d60 con 0x7f3f901584c0 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.805+0000 7f3f96034640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9015e540 con 0x7f3f9015e770 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.805+0000 7f3f8f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 0x7f3f901f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.805+0000 7f3f8f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 0x7f3f901f3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40848/0 (socket says 172.21.15.59:40848) 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.805+0000 7f3f8f7fe640 1 -- 172.21.15.59:0/249803361 learned_addr learned my addr 172.21.15.59:0/249803361 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.806+0000 7f3f8f7fe640 1 -- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015e770 msgr2=0x7f3f9015ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.806+0000 7f3f8f7fe640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015e770 0x7f3f9015ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.806+0000 7f3f8f7fe640 1 -- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f901584c0 msgr2=0x7f3f90158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.806+0000 7f3f8f7fe640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f901584c0 0x7f3f90158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.806+0000 7f3f8f7fe640 1 -- 172.21.15.59:0/249803361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9015f120 con 0x7f3f78004d40 2024-03-06T07:29:00.811 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.806+0000 7f3f8f7fe640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 0x7f3f901f3e50 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f3f80011e60 tx=0x7f3f8000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.806+0000 7f3f77fff640 1 -- 172.21.15.59:0/249803361 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f80015630 con 0x7f3f78004d40 2024-03-06T07:29:00.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.806+0000 7f3f77fff640 1 -- 172.21.15.59:0/249803361 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f800157d0 con 0x7f3f78004d40 2024-03-06T07:29:00.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.806+0000 7f3f96034640 1 -- 172.21.15.59:0/249803361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f90157440 con 0x7f3f78004d40 2024-03-06T07:29:00.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.806+0000 7f3f77fff640 1 -- 172.21.15.59:0/249803361 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f80015ba0 con 0x7f3f78004d40 2024-03-06T07:29:00.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.806+0000 7f3f96034640 1 -- 172.21.15.59:0/249803361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f90157ce0 con 0x7f3f78004d40 2024-03-06T07:29:00.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.807+0000 7f3f96034640 1 -- 172.21.15.59:0/249803361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9004f330 con 0x7f3f78004d40 2024-03-06T07:29:00.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.808+0000 7f3f77fff640 1 -- 172.21.15.59:0/249803361 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f8001d400 con 0x7f3f78004d40 2024-03-06T07:29:00.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.808+0000 7f3f77fff640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f7c06b4d0 0x7f3f7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:00.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.808+0000 7f3f77fff640 1 -- 172.21.15.59:0/249803361 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f8008b310 con 0x7f3f78004d40 2024-03-06T07:29:00.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.808+0000 7f3f8effd640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f7c06b4d0 0x7f3f7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:00.812 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.809+0000 7f3f8effd640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f7c06b4d0 0x7f3f7c06d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3f901604b0 tx=0x7f3f84009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:00.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:00.817+0000 7f3f77fff640 1 -- 172.21.15.59:0/249803361 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f80019030 con 0x7f3f78004d40 2024-03-06T07:29:01.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.041+0000 7f3f96034640 1 -- 172.21.15.59:0/249803361 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f900cea80 con 0x7f3f78004d40 2024-03-06T07:29:01.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.041+0000 7f3f77fff640 1 -- 172.21.15.59:0/249803361 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5004 (secure 0 0 0) 0x7f3f80010040 con 0x7f3f78004d40 2024-03-06T07:29:01.043 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:01.043 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6938,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/1772616514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1772616514},{"type":"v1","addr":"172.21.15.164:6836","nonce":1772616514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":7816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3891024511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":3891024511},{"type":"v1","addr":"172.21.15.164:6837","nonce":3891024511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:28:55.978187+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8673,8676]},"id":9}]} 2024-03-06T07:29:01.044 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 82 2024-03-06T07:29:01.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 -- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f7c06b4d0 msgr2=0x7f3f7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f7c06b4d0 0x7f3f7c06d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3f901604b0 tx=0x7f3f84009040 comp rx=0 tx=0).stop 2024-03-06T07:29:01.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 -- 172.21.15.59:0/249803361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 msgr2=0x7f3f901f3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 0x7f3f901f3e50 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f3f80011e60 tx=0x7f3f8000f580 comp rx=0 tx=0).stop 2024-03-06T07:29:01.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 -- 172.21.15.59:0/249803361 shutdown_connections 2024-03-06T07:29:01.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3f7c06b4d0 0x7f3f7c06d990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f9015e770 0x7f3f9015ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f901584c0 0x7f3f90158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 --2- 172.21.15.59:0/249803361 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f78004d40 0x7f3f901f3e50 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 -- 172.21.15.59:0/249803361 >> 172.21.15.59:0/249803361 conn(0x7f3f900d3cd0 msgr2=0x7f3f9006caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 -- 172.21.15.59:0/249803361 shutdown_connections 2024-03-06T07:29:01.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.044+0000 7f3f96034640 1 -- 172.21.15.59:0/249803361 wait complete. 2024-03-06T07:29:01.057 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are c and a 2024-03-06T07:29:01.057 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:29:01.057 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:01.069 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:29:01.069 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:29:01.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.192+0000 7fd86c237640 1 Processor -- start 2024-03-06T07:29:01.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.192+0000 7fd86c237640 1 -- start start 2024-03-06T07:29:01.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.193+0000 7fd86c237640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd864162970 0x7fd8640d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.193+0000 7fd86c237640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8640d22c0 con 0x7fd864161cf0 2024-03-06T07:29:01.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.193+0000 7fd86c237640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8640d25a0 con 0x7fd8640b4460 2024-03-06T07:29:01.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.193+0000 7fd86c237640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd864161b00 con 0x7fd864162970 2024-03-06T07:29:01.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.194+0000 7fd86a7d4640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd864161cf0 0x7fd8640cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46338/0 (socket says 172.21.15.59:46338) 2024-03-06T07:29:01.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.194+0000 7fd86a7d4640 1 -- 172.21.15.59:0/1848560962 learned_addr learned my addr 172.21.15.59:0/1848560962 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:01.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.194+0000 7fd8697d2640 1 --2- 172.21.15.59:0/1848560962 >> v2:172.21.15.164:3300/0 conn(0x7fd864162970 0x7fd8640d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.194+0000 7fd868fd1640 1 -- 172.21.15.59:0/1848560962 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910480398 0 0) 0x7fd8640d22c0 con 0x7fd864161cf0 2024-03-06T07:29:01.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.194+0000 7fd868fd1640 1 -- 172.21.15.59:0/1848560962 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd850003520 con 0x7fd864161cf0 2024-03-06T07:29:01.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.194+0000 7fd868fd1640 1 -- 172.21.15.59:0/1848560962 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 956649043 0 0) 0x7fd850003520 con 0x7fd864161cf0 2024-03-06T07:29:01.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.194+0000 7fd8697d2640 1 -- 172.21.15.59:0/1848560962 >> v1:172.21.15.164:6789/0 conn(0x7fd8640b4460 legacy=0x7fd8640b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.194+0000 7fd8697d2640 1 -- 172.21.15.59:0/1848560962 >> v1:172.21.15.59:6789/0 conn(0x7fd864161cf0 legacy=0x7fd8640cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.194+0000 7fd8697d2640 1 -- 172.21.15.59:0/1848560962 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8640d2880 con 0x7fd864162970 2024-03-06T07:29:01.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.195+0000 7fd8697d2640 1 --2- 172.21.15.59:0/1848560962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd864162970 0x7fd8640d4800 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd8540097a0 tx=0x7fd85402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a99320544e150f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.195+0000 7fd868fd1640 1 -- 172.21.15.59:0/1848560962 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd854038070 con 0x7fd864162970 2024-03-06T07:29:01.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.195+0000 7fd868fd1640 1 -- 172.21.15.59:0/1848560962 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd854031500 con 0x7fd864162970 2024-03-06T07:29:01.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.195+0000 7fd868fd1640 1 -- 172.21.15.59:0/1848560962 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0019f0 con 0x7fd864162970 2024-03-06T07:29:01.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.196+0000 7fd86c237640 1 -- 172.21.15.59:0/1848560962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd864162970 msgr2=0x7fd8640d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.196+0000 7fd86c237640 1 --2- 172.21.15.59:0/1848560962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd864162970 0x7fd8640d4800 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd8540097a0 tx=0x7fd85402b820 comp rx=0 tx=0).stop 2024-03-06T07:29:01.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.196+0000 7fd86c237640 1 -- 172.21.15.59:0/1848560962 shutdown_connections 2024-03-06T07:29:01.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.196+0000 7fd86c237640 1 --2- 172.21.15.59:0/1848560962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd864162970 0x7fd8640d4800 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.196+0000 7fd86c237640 1 -- 172.21.15.59:0/1848560962 >> 172.21.15.59:0/1848560962 conn(0x7fd8640d3ce0 msgr2=0x7fd8640d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:01.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.196+0000 7fd86c237640 1 -- 172.21.15.59:0/1848560962 shutdown_connections 2024-03-06T07:29:01.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.196+0000 7fd86c237640 1 -- 172.21.15.59:0/1848560962 wait complete. 2024-03-06T07:29:01.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.197+0000 7fd86c237640 1 Processor -- start 2024-03-06T07:29:01.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.197+0000 7fd86c237640 1 -- start start 2024-03-06T07:29:01.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd86c237640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8640b4460 0x7fd8641511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd86c237640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd864161cf0 0x7fd8641f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd86c237640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd864162970 0x7fd86415fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd86c237640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8640ce950 con 0x7fd8640b4460 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd86c237640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8640cecc0 con 0x7fd864162970 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd86c237640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8640b4840 con 0x7fd864161cf0 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd869fd3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8640b4460 0x7fd8641511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd869fd3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8640b4460 0x7fd8641511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40852/0 (socket says 172.21.15.59:40852) 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd869fd3640 1 -- 172.21.15.59:0/3966804850 learned_addr learned my addr 172.21.15.59:0/3966804850 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd8697d2640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd864161cf0 0x7fd8641f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd86a7d4640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd864162970 0x7fd86415fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd869fd3640 1 -- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd864161cf0 msgr2=0x7fd8641f44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd869fd3640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd864161cf0 0x7fd8641f44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd869fd3640 1 -- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd864162970 msgr2=0x7fd86415fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd869fd3640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd864162970 0x7fd86415fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:01.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd869fd3640 1 -- 172.21.15.59:0/3966804850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8641f4a30 con 0x7fd8640b4460 2024-03-06T07:29:01.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd8697d2640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd864161cf0 0x7fd8641f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:01.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.198+0000 7fd869fd3640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8640b4460 0x7fd8641511c0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fd86000c400 tx=0x7fd86000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.199+0000 7fd85a7fc640 1 -- 172.21.15.59:0/3966804850 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860017070 con 0x7fd8640b4460 2024-03-06T07:29:01.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.199+0000 7fd85a7fc640 1 -- 172.21.15.59:0/3966804850 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8600052d0 con 0x7fd8640b4460 2024-03-06T07:29:01.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.199+0000 7fd85a7fc640 1 -- 172.21.15.59:0/3966804850 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860003830 con 0x7fd8640b4460 2024-03-06T07:29:01.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.199+0000 7fd86c237640 1 -- 172.21.15.59:0/3966804850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8640ccb70 con 0x7fd8640b4460 2024-03-06T07:29:01.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.199+0000 7fd86c237640 1 -- 172.21.15.59:0/3966804850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8640cd130 con 0x7fd8640b4460 2024-03-06T07:29:01.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.200+0000 7fd86c237640 1 -- 172.21.15.59:0/3966804850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86404f330 con 0x7fd8640b4460 2024-03-06T07:29:01.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.201+0000 7fd85a7fc640 1 -- 172.21.15.59:0/3966804850 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd86001a070 con 0x7fd8640b4460 2024-03-06T07:29:01.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.201+0000 7fd85a7fc640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84806b4d0 0x7fd84806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:01.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.201+0000 7fd85a7fc640 1 -- 172.21.15.59:0/3966804850 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd860089560 con 0x7fd8640b4460 2024-03-06T07:29:01.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.201+0000 7fd8697d2640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84806b4d0 0x7fd84806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:01.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.202+0000 7fd8697d2640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84806b4d0 0x7fd84806d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd854035000 tx=0x7fd854008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:01.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.206+0000 7fd85a7fc640 1 -- 172.21.15.59:0/3966804850 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8641f4a30 con 0x7fd8640b4460 2024-03-06T07:29:01.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.474+0000 7fd86c237640 1 -- 172.21.15.59:0/3966804850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fd86406b180 con 0x7fd8640b4460 2024-03-06T07:29:01.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.994+0000 7fd85a7fc640 1 -- 172.21.15.59:0/3966804850 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7816 v83) v1 ==== 95+0+0 (secure 0 0 0) 0x7fd8600583e0 con 0x7fd8640b4460 2024-03-06T07:29:01.996 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 7816 2024-03-06T07:29:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 -- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84806b4d0 msgr2=0x7fd84806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84806b4d0 0x7fd84806d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd854035000 tx=0x7fd854008030 comp rx=0 tx=0).stop 2024-03-06T07:29:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 -- 172.21.15.59:0/3966804850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8640b4460 msgr2=0x7fd8641511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8640b4460 0x7fd8641511c0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fd86000c400 tx=0x7fd86000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 -- 172.21.15.59:0/3966804850 shutdown_connections 2024-03-06T07:29:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd84806b4d0 0x7fd84806d990 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd864162970 0x7fd86415fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd864161cf0 0x7fd8641f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 --2- 172.21.15.59:0/3966804850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8640b4460 0x7fd8641511c0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 -- 172.21.15.59:0/3966804850 >> 172.21.15.59:0/3966804850 conn(0x7fd8640d3ce0 msgr2=0x7fd8640d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 -- 172.21.15.59:0/3966804850 shutdown_connections 2024-03-06T07:29:02.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:01.997+0000 7fd86c237640 1 -- 172.21.15.59:0/3966804850 wait complete. 2024-03-06T07:29:02.010 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:29:02.010 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:02.022 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:29:02.022 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:29:02.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.129+0000 7fbe0ab5e640 1 Processor -- start 2024-03-06T07:29:02.131 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.129+0000 7fbe0ab5e640 1 -- start start 2024-03-06T07:29:02.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.130+0000 7fbe0ab5e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbe04162970 0x7fbe040d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.130+0000 7fbe0ab5e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe040d22c0 con 0x7fbe04161cf0 2024-03-06T07:29:02.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.130+0000 7fbe0ab5e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe040d25a0 con 0x7fbe040b4460 2024-03-06T07:29:02.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.130+0000 7fbe0ab5e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe04161b00 con 0x7fbe04162970 2024-03-06T07:29:02.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.130+0000 7fbe03fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbe04162970 0x7fbe040d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.130+0000 7fbe090fb640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbe04161cf0 0x7fbe040cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46340/0 (socket says 172.21.15.59:46340) 2024-03-06T07:29:02.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.130+0000 7fbe090fb640 1 -- 172.21.15.59:0/879755212 learned_addr learned my addr 172.21.15.59:0/879755212 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:02.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.131+0000 7fbe03fff640 1 -- 172.21.15.59:0/879755212 >> v1:172.21.15.164:6789/0 conn(0x7fbe040b4460 legacy=0x7fbe040b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.131+0000 7fbe03fff640 1 -- 172.21.15.59:0/879755212 >> v1:172.21.15.59:6789/0 conn(0x7fbe04161cf0 legacy=0x7fbe040cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.131+0000 7fbe03fff640 1 -- 172.21.15.59:0/879755212 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe040d2880 con 0x7fbe04162970 2024-03-06T07:29:02.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.131+0000 7fbe037fe640 1 -- 172.21.15.59:0/879755212 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177301962 0 0) 0x7fbe040d22c0 con 0x7fbe04161cf0 2024-03-06T07:29:02.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.131+0000 7fbe03fff640 1 --2- 172.21.15.59:0/879755212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe04162970 0x7fbe040d4800 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00097a0 tx=0x7fbdf002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e30b6ae87927756 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.132+0000 7fbe037fe640 1 -- 172.21.15.59:0/879755212 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0035020 con 0x7fbe04162970 2024-03-06T07:29:02.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.132+0000 7fbe037fe640 1 -- 172.21.15.59:0/879755212 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf0031830 con 0x7fbe04162970 2024-03-06T07:29:02.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.132+0000 7fbe037fe640 1 -- 172.21.15.59:0/879755212 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0031c00 con 0x7fbe04162970 2024-03-06T07:29:02.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.132+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/879755212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe04162970 msgr2=0x7fbe040d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.132+0000 7fbe0ab5e640 1 --2- 172.21.15.59:0/879755212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe04162970 0x7fbe040d4800 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00097a0 tx=0x7fbdf002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.132+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/879755212 shutdown_connections 2024-03-06T07:29:02.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.132+0000 7fbe0ab5e640 1 --2- 172.21.15.59:0/879755212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe04162970 0x7fbe040d4800 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.132+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/879755212 >> 172.21.15.59:0/879755212 conn(0x7fbe040d3ce0 msgr2=0x7fbe040d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.132+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/879755212 shutdown_connections 2024-03-06T07:29:02.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.132+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/879755212 wait complete. 2024-03-06T07:29:02.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.133+0000 7fbe0ab5e640 1 Processor -- start 2024-03-06T07:29:02.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.133+0000 7fbe0ab5e640 1 -- start start 2024-03-06T07:29:02.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.134+0000 7fbe0ab5e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe040b4460 0x7fbe04155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.134+0000 7fbe0ab5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe04161cf0 0x7fbe041f7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.134+0000 7fbe0ab5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe04162970 0x7fbe040b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.134+0000 7fbe0ab5e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040ce950 con 0x7fbe04161cf0 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.134+0000 7fbe0ab5e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040cecc0 con 0x7fbe040b4460 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.134+0000 7fbe0ab5e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe040b4840 con 0x7fbe04162970 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.134+0000 7fbe03fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe04161cf0 0x7fbe041f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.134+0000 7fbe03fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe04161cf0 0x7fbe041f7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40880/0 (socket says 172.21.15.59:40880) 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.134+0000 7fbe03fff640 1 -- 172.21.15.59:0/3565712873 learned_addr learned my addr 172.21.15.59:0/3565712873 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.134+0000 7fbe088fa640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe040b4460 0x7fbe04155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe090fb640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe04162970 0x7fbe040b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe03fff640 1 -- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe04162970 msgr2=0x7fbe040b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe03fff640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe04162970 0x7fbe040b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe03fff640 1 -- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe040b4460 msgr2=0x7fbe04155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe03fff640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe040b4460 0x7fbe04155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe03fff640 1 -- 172.21.15.59:0/3565712873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe04156060 con 0x7fbe04161cf0 2024-03-06T07:29:02.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe088fa640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe040b4460 0x7fbe04155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:02.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe03fff640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe04161cf0 0x7fbe041f7800 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00098c0 tx=0x7fbdf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe017fa640 1 -- 172.21.15.59:0/3565712873 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0004230 con 0x7fbe04161cf0 2024-03-06T07:29:02.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe017fa640 1 -- 172.21.15.59:0/3565712873 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf00043d0 con 0x7fbe04161cf0 2024-03-06T07:29:02.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/3565712873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe041f8250 con 0x7fbe04161cf0 2024-03-06T07:29:02.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe017fa640 1 -- 172.21.15.59:0/3565712873 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf003cad0 con 0x7fbe04161cf0 2024-03-06T07:29:02.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.135+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/3565712873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe040cb000 con 0x7fbe04161cf0 2024-03-06T07:29:02.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.136+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/3565712873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0404f330 con 0x7fbe04161cf0 2024-03-06T07:29:02.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.137+0000 7fbe017fa640 1 -- 172.21.15.59:0/3565712873 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbdf0038020 con 0x7fbe04161cf0 2024-03-06T07:29:02.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.137+0000 7fbe017fa640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbde806b4d0 0x7fbde806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.137+0000 7fbe017fa640 1 -- 172.21.15.59:0/3565712873 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbdf00ac3d0 con 0x7fbe04161cf0 2024-03-06T07:29:02.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.137+0000 7fbe088fa640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbde806b4d0 0x7fbde806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.138+0000 7fbe088fa640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbde806b4d0 0x7fbde806d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8006fd0 tx=0x7fbdf8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.142+0000 7fbe017fa640 1 -- 172.21.15.59:0/3565712873 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbdf0041070 con 0x7fbe04161cf0 2024-03-06T07:29:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.460+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/3565712873 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7fbe0406b180 con 0x7fbe04161cf0 2024-03-06T07:29:02.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.995+0000 7fbe017fa640 1 -- 172.21.15.59:0/3565712873 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6938 v84) v1 ==== 95+0+0 (secure 0 0 0) 0x7fbdf007b400 con 0x7fbe04161cf0 2024-03-06T07:29:02.998 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 6938 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.998+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbde806b4d0 msgr2=0x7fbde806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.998+0000 7fbe0ab5e640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbde806b4d0 0x7fbde806d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8006fd0 tx=0x7fbdf8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.998+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/3565712873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe04161cf0 msgr2=0x7fbe041f7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.998+0000 7fbe0ab5e640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe04161cf0 0x7fbe041f7800 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00098c0 tx=0x7fbdf00046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.998+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/3565712873 shutdown_connections 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.998+0000 7fbe0ab5e640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbde806b4d0 0x7fbde806d990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.998+0000 7fbe0ab5e640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe04162970 0x7fbe040b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.998+0000 7fbe0ab5e640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe04161cf0 0x7fbe041f7800 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.999+0000 7fbe0ab5e640 1 --2- 172.21.15.59:0/3565712873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe040b4460 0x7fbe04155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.999+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/3565712873 >> 172.21.15.59:0/3565712873 conn(0x7fbe040d3ce0 msgr2=0x7fbe040d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.999+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/3565712873 shutdown_connections 2024-03-06T07:29:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:02.999+0000 7fbe0ab5e640 1 -- 172.21.15.59:0/3565712873 wait complete. 2024-03-06T07:29:03.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:03.012 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:29:03.012 DEBUG:teuthology.orchestra.run.smithi059:> mkdir pin 2024-03-06T07:29:03.012 DEBUG:teuthology.orchestra.run.smithi059:> touch pin/placeholder 2024-03-06T07:29:03.012 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:29:03.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:03.040 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-03-06T07:29:03.098 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.254+0000 7f58ef71e640 1 Processor -- start 2024-03-06T07:29:03.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.255+0000 7f58ef71e640 1 -- start start 2024-03-06T07:29:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.255+0000 7f58ef71e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f58e815dfe0 0x7f58e80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.255+0000 7f58ef71e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e80cf830 0x7f58e80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.255+0000 7f58ef71e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58e80d27f0 0x7f58e80d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.255+0000 7f58ef71e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e80d2c50 con 0x7f58e80d27f0 2024-03-06T07:29:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.255+0000 7f58ef71e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e80b31e0 con 0x7f58e815dfe0 2024-03-06T07:29:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.255+0000 7f58ef71e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e80b3d60 con 0x7f58e80cf830 2024-03-06T07:29:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.255+0000 7f58edcbb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58e80d27f0 0x7f58e80d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.255+0000 7f58edcbb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58e80d27f0 0x7f58e80d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40890/0 (socket says 172.21.15.59:40890) 2024-03-06T07:29:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.256+0000 7f58edcbb640 1 -- 172.21.15.59:0/3976022159 learned_addr learned my addr 172.21.15.59:0/3976022159 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.256+0000 7f58edcbb640 1 -- 172.21.15.59:0/3976022159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e80cf830 msgr2=0x7f58e80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.256+0000 7f58edcbb640 1 --2- 172.21.15.59:0/3976022159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e80cf830 0x7f58e80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.256+0000 7f58edcbb640 1 -- 172.21.15.59:0/3976022159 >> v2:172.21.15.164:3300/0 conn(0x7f58e815dfe0 msgr2=0x7f58e80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.256+0000 7f58edcbb640 1 --2- 172.21.15.59:0/3976022159 >> v2:172.21.15.164:3300/0 conn(0x7f58e815dfe0 0x7f58e80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.256+0000 7f58edcbb640 1 -- 172.21.15.59:0/3976022159 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e80b3430 con 0x7f58e80d27f0 2024-03-06T07:29:03.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.257+0000 7f58edcbb640 1 --2- 172.21.15.59:0/3976022159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e80d27f0 0x7f58e80d0300 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f58e8067990 tx=0x7f58e002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b402341fa951f570 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.257+0000 7f58dffff640 1 -- 172.21.15.59:0/3976022159 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e002ca70 con 0x7f58e80d27f0 2024-03-06T07:29:03.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.257+0000 7f58dffff640 1 -- 172.21.15.59:0/3976022159 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e002cc10 con 0x7f58e80d27f0 2024-03-06T07:29:03.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.257+0000 7f58dffff640 1 -- 172.21.15.59:0/3976022159 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e00343f0 con 0x7f58e80d27f0 2024-03-06T07:29:03.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.258+0000 7f58ef71e640 1 -- 172.21.15.59:0/3976022159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e80d27f0 msgr2=0x7f58e80d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.258+0000 7f58ef71e640 1 --2- 172.21.15.59:0/3976022159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e80d27f0 0x7f58e80d0300 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f58e8067990 tx=0x7f58e002bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.258+0000 7f58ef71e640 1 -- 172.21.15.59:0/3976022159 shutdown_connections 2024-03-06T07:29:03.260 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.258+0000 7f58ef71e640 1 --2- 172.21.15.59:0/3976022159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e80d27f0 0x7f58e80d0300 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.258+0000 7f58ef71e640 1 --2- 172.21.15.59:0/3976022159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e80cf830 0x7f58e80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.258+0000 7f58ef71e640 1 --2- 172.21.15.59:0/3976022159 >> v2:172.21.15.164:3300/0 conn(0x7f58e815dfe0 0x7f58e80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.258+0000 7f58ef71e640 1 -- 172.21.15.59:0/3976022159 >> 172.21.15.59:0/3976022159 conn(0x7f58e80d3cd0 msgr2=0x7f58e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.258+0000 7f58ef71e640 1 -- 172.21.15.59:0/3976022159 shutdown_connections 2024-03-06T07:29:03.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.258+0000 7f58ef71e640 1 -- 172.21.15.59:0/3976022159 wait complete. 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.259+0000 7f58ef71e640 1 Processor -- start 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.259+0000 7f58ef71e640 1 -- start start 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58ef71e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e80cf830 0x7f58e8158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58ef71e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e80d27f0 0x7f58e815fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58ef71e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e815dfe0 0x7f58e815b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58ef71e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e80d1c70 con 0x7f58e80d27f0 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58ef71e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e80b39f0 con 0x7f58e815dfe0 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58ef71e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e80d1df0 con 0x7f58e80cf830 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58eccb9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e80d27f0 0x7f58e815fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58eccb9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e80d27f0 0x7f58e815fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40896/0 (socket says 172.21.15.59:40896) 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58eccb9640 1 -- 172.21.15.59:0/416886903 learned_addr learned my addr 172.21.15.59:0/416886903 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58edcbb640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e815dfe0 0x7f58e815b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58ed4ba640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e80cf830 0x7f58e8158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58eccb9640 1 -- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e80cf830 msgr2=0x7f58e8158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58eccb9640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e80cf830 0x7f58e8158140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58eccb9640 1 -- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e815dfe0 msgr2=0x7f58e815b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58eccb9640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e815dfe0 0x7f58e815b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.260+0000 7f58eccb9640 1 -- 172.21.15.59:0/416886903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e80ccbf0 con 0x7f58e80d27f0 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.261+0000 7f58eccb9640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e80d27f0 0x7f58e815fea0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f58d800b450 tx=0x7f58d800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.261+0000 7f58ddffb640 1 -- 172.21.15.59:0/416886903 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8015020 con 0x7f58e80d27f0 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.261+0000 7f58ddffb640 1 -- 172.21.15.59:0/416886903 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d8011890 con 0x7f58e80d27f0 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.261+0000 7f58ef71e640 1 -- 172.21.15.59:0/416886903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e80cbb50 con 0x7f58e80d27f0 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.261+0000 7f58ddffb640 1 -- 172.21.15.59:0/416886903 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8011c80 con 0x7f58e80d27f0 2024-03-06T07:29:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.261+0000 7f58ef71e640 1 -- 172.21.15.59:0/416886903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e80cc3f0 con 0x7f58e80d27f0 2024-03-06T07:29:03.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.262+0000 7f58ef71e640 1 -- 172.21.15.59:0/416886903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e804f330 con 0x7f58e80d27f0 2024-03-06T07:29:03.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.262+0000 7f58ddffb640 1 -- 172.21.15.59:0/416886903 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58d8018020 con 0x7f58e80d27f0 2024-03-06T07:29:03.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.263+0000 7f58ddffb640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58cc06b4d0 0x7f58cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.263+0000 7f58ddffb640 1 -- 172.21.15.59:0/416886903 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58d808a360 con 0x7f58e80d27f0 2024-03-06T07:29:03.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.263+0000 7f58ed4ba640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58cc06b4d0 0x7f58cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.264+0000 7f58ed4ba640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58cc06b4d0 0x7f58cc06d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f58e80ca2e0 tx=0x7f58e4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.267+0000 7f58ddffb640 1 -- 172.21.15.59:0/416886903 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58d801f020 con 0x7f58e80d27f0 2024-03-06T07:29:03.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.583+0000 7f58ef71e640 1 -- 172.21.15.59:0/416886903 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58e80ceb30 con 0x7f58e80d27f0 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.583+0000 7f58ddffb640 1 -- 172.21.15.59:0/416886903 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3417 (secure 0 0 0) 0x7f58d800d040 con 0x7f58e80d27f0 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:03.585 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":84,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:28:55.978187+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8673,8676]},"id":9}]} 2024-03-06T07:29:03.586 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 84 2024-03-06T07:29:03.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 -- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58cc06b4d0 msgr2=0x7f58cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58cc06b4d0 0x7f58cc06d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f58e80ca2e0 tx=0x7f58e4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:03.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 -- 172.21.15.59:0/416886903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e80d27f0 msgr2=0x7f58e815fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e80d27f0 0x7f58e815fea0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f58d800b450 tx=0x7f58d800b910 comp rx=0 tx=0).stop 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 -- 172.21.15.59:0/416886903 shutdown_connections 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f58cc06b4d0 0x7f58cc06d990 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e815dfe0 0x7f58e815b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58e80d27f0 0x7f58e815fea0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 --2- 172.21.15.59:0/416886903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e80cf830 0x7f58e8158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 -- 172.21.15.59:0/416886903 >> 172.21.15.59:0/416886903 conn(0x7f58e80d3cd0 msgr2=0x7f58e80b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 -- 172.21.15.59:0/416886903 shutdown_connections 2024-03-06T07:29:03.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:03.585+0000 7f58ef71e640 1 -- 172.21.15.59:0/416886903 wait complete. 2024-03-06T07:29:03.598 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:29:03.715 INFO:teuthology.orchestra.run.smithi059.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:03.715 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:03.715 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:05.717 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:05.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.816+0000 7f39e1bbc640 1 Processor -- start 2024-03-06T07:29:05.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.817+0000 7f39e1bbc640 1 -- start start 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.817+0000 7f39e1bbc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f39dc15bb20 0x7f39dc0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.817+0000 7f39e1bbc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc161980 con 0x7f39dc0cd8c0 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.817+0000 7f39e1bbc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39dc161c60 con 0x7f39dc0cfff0 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.817+0000 7f39e1bbc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39dc161f40 con 0x7f39dc15bb20 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.817+0000 7f39dbfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f39dc0cd8c0 0x7f39dc0cebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49666/0 (socket says 172.21.15.59:49666) 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.817+0000 7f39dbfff640 1 -- 172.21.15.59:0/971828401 learned_addr learned my addr 172.21.15.59:0/971828401 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.818+0000 7f39db7fe640 1 --2- 172.21.15.59:0/971828401 >> v2:172.21.15.164:3300/0 conn(0x7f39dc15bb20 0x7f39dc0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.818+0000 7f39da7fc640 1 -- 172.21.15.59:0/971828401 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757661634 0 0) 0x7f39dc161980 con 0x7f39dc0cd8c0 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.818+0000 7f39da7fc640 1 -- 172.21.15.59:0/971828401 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c4003520 con 0x7f39dc0cd8c0 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.818+0000 7f39da7fc640 1 -- 172.21.15.59:0/971828401 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48458360 0 0) 0x7f39dc161c60 con 0x7f39dc0cfff0 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.818+0000 7f39da7fc640 1 -- 172.21.15.59:0/971828401 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39dc161980 con 0x7f39dc0cfff0 2024-03-06T07:29:05.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.818+0000 7f39da7fc640 1 -- 172.21.15.59:0/971828401 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2455628331 0 0) 0x7f39c4003520 con 0x7f39dc0cd8c0 2024-03-06T07:29:05.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.818+0000 7f39db7fe640 1 -- 172.21.15.59:0/971828401 >> v1:172.21.15.164:6789/0 conn(0x7f39dc0cfff0 legacy=0x7f39dc0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.818+0000 7f39db7fe640 1 -- 172.21.15.59:0/971828401 >> v1:172.21.15.59:6789/0 conn(0x7f39dc0cd8c0 legacy=0x7f39dc0cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.818+0000 7f39db7fe640 1 -- 172.21.15.59:0/971828401 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc162220 con 0x7f39dc15bb20 2024-03-06T07:29:05.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.819+0000 7f39db7fe640 1 --2- 172.21.15.59:0/971828401 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39dc15bb20 0x7f39dc0d1df0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0097a0 tx=0x7f39cc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b94aac26c5b3141 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.819+0000 7f39da7fc640 1 -- 172.21.15.59:0/971828401 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc038070 con 0x7f39dc15bb20 2024-03-06T07:29:05.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.819+0000 7f39da7fc640 1 -- 172.21.15.59:0/971828401 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39cc031500 con 0x7f39dc15bb20 2024-03-06T07:29:05.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.819+0000 7f39da7fc640 1 -- 172.21.15.59:0/971828401 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d40019f0 con 0x7f39dc15bb20 2024-03-06T07:29:05.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.820+0000 7f39e1bbc640 1 -- 172.21.15.59:0/971828401 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39dc15bb20 msgr2=0x7f39dc0d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.820+0000 7f39e1bbc640 1 --2- 172.21.15.59:0/971828401 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39dc15bb20 0x7f39dc0d1df0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0097a0 tx=0x7f39cc02b820 comp rx=0 tx=0).stop 2024-03-06T07:29:05.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.820+0000 7f39e1bbc640 1 -- 172.21.15.59:0/971828401 shutdown_connections 2024-03-06T07:29:05.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.820+0000 7f39e1bbc640 1 --2- 172.21.15.59:0/971828401 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39dc15bb20 0x7f39dc0d1df0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.820+0000 7f39e1bbc640 1 -- 172.21.15.59:0/971828401 >> 172.21.15.59:0/971828401 conn(0x7f39dc09a9f0 msgr2=0x7f39dc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.820+0000 7f39e1bbc640 1 -- 172.21.15.59:0/971828401 shutdown_connections 2024-03-06T07:29:05.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.820+0000 7f39e1bbc640 1 -- 172.21.15.59:0/971828401 wait complete. 2024-03-06T07:29:05.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.821+0000 7f39e1bbc640 1 Processor -- start 2024-03-06T07:29:05.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.821+0000 7f39e1bbc640 1 -- start start 2024-03-06T07:29:05.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39e1bbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39dc0cd8c0 0x7f39dc0b5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39e1bbc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39dc0cfff0 0x7f39dc0b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39e1bbc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39dc15bb20 0x7f39dc150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39e1bbc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0d2330 con 0x7f39dc15bb20 2024-03-06T07:29:05.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39e1bbc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0cf2d0 con 0x7f39dc0cfff0 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39e1bbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0d24b0 con 0x7f39dc0cd8c0 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39dc15bb20 0x7f39dc150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39dc15bb20 0x7f39dc150d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60142/0 (socket says 172.21.15.59:60142) 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39dbfff640 1 -- 172.21.15.59:0/596704842 learned_addr learned my addr 172.21.15.59:0/596704842 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39daffd640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39dc0cfff0 0x7f39dc0b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39db7fe640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39dc0cd8c0 0x7f39dc0b5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39dbfff640 1 -- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39dc0cd8c0 msgr2=0x7f39dc0b5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39dbfff640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39dc0cd8c0 0x7f39dc0b5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39dbfff640 1 -- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39dc0cfff0 msgr2=0x7f39dc0b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39dbfff640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39dc0cfff0 0x7f39dc0b6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39dbfff640 1 -- 172.21.15.59:0/596704842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc15b5b0 con 0x7f39dc15bb20 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39dbfff640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39dc15bb20 0x7f39dc150d00 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f39d400cc20 tx=0x7f39d400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.822+0000 7f39c3fff640 1 -- 172.21.15.59:0/596704842 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d400ce30 con 0x7f39dc15bb20 2024-03-06T07:29:05.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.823+0000 7f39c3fff640 1 -- 172.21.15.59:0/596704842 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d4012400 con 0x7f39dc15bb20 2024-03-06T07:29:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.823+0000 7f39e1bbc640 1 -- 172.21.15.59:0/596704842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc0b5e10 con 0x7f39dc15bb20 2024-03-06T07:29:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.823+0000 7f39c3fff640 1 -- 172.21.15.59:0/596704842 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d4012830 con 0x7f39dc15bb20 2024-03-06T07:29:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.823+0000 7f39e1bbc640 1 -- 172.21.15.59:0/596704842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc0b63d0 con 0x7f39dc15bb20 2024-03-06T07:29:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.824+0000 7f39e1bbc640 1 -- 172.21.15.59:0/596704842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39dc04f330 con 0x7f39dc15bb20 2024-03-06T07:29:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.824+0000 7f39c3fff640 1 -- 172.21.15.59:0/596704842 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39d40129d0 con 0x7f39dc15bb20 2024-03-06T07:29:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.824+0000 7f39c3fff640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39c806b500 0x7f39c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.825+0000 7f39c3fff640 1 -- 172.21.15.59:0/596704842 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39d408a8a0 con 0x7f39dc15bb20 2024-03-06T07:29:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.825+0000 7f39db7fe640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39c806b500 0x7f39c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.828 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.825+0000 7f39db7fe640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39c806b500 0x7f39c806d9c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f39cc035000 tx=0x7f39cc008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:05.829+0000 7f39c3fff640 1 -- 172.21.15.59:0/596704842 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39d40590d0 con 0x7f39dc15bb20 2024-03-06T07:29:06.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.055+0000 7f39e1bbc640 1 -- 172.21.15.59:0/596704842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39dc0b2e60 con 0x7f39dc15bb20 2024-03-06T07:29:06.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.055+0000 7f39c3fff640 1 -- 172.21.15.59:0/596704842 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3417 (secure 0 0 0) 0x7f39d405d170 con 0x7f39dc15bb20 2024-03-06T07:29:06.057 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:06.057 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":84,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:28:55.978187+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8673,8676]},"id":9}]} 2024-03-06T07:29:06.058 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 84 2024-03-06T07:29:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 -- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39c806b500 msgr2=0x7f39c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39c806b500 0x7f39c806d9c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f39cc035000 tx=0x7f39cc008030 comp rx=0 tx=0).stop 2024-03-06T07:29:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 -- 172.21.15.59:0/596704842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39dc15bb20 msgr2=0x7f39dc150d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39dc15bb20 0x7f39dc150d00 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f39d400cc20 tx=0x7f39d400a430 comp rx=0 tx=0).stop 2024-03-06T07:29:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 -- 172.21.15.59:0/596704842 shutdown_connections 2024-03-06T07:29:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f39c806b500 0x7f39c806d9c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39dc15bb20 0x7f39dc150d00 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39dc0cfff0 0x7f39dc0b6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 --2- 172.21.15.59:0/596704842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39dc0cd8c0 0x7f39dc0b5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 -- 172.21.15.59:0/596704842 >> 172.21.15.59:0/596704842 conn(0x7f39dc09a9f0 msgr2=0x7f39dc0ce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 -- 172.21.15.59:0/596704842 shutdown_connections 2024-03-06T07:29:06.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:06.058+0000 7f39e1bbc640 1 -- 172.21.15.59:0/596704842 wait complete. 2024-03-06T07:29:06.071 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:29:06.244 INFO:teuthology.orchestra.run.smithi059.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:06.244 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:06.244 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:08.246 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:08.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.346+0000 7f712549d640 1 Processor -- start 2024-03-06T07:29:08.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.346+0000 7f712549d640 1 -- start start 2024-03-06T07:29:08.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.347+0000 7f712549d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f712015dfe0 0x7f71200cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f712549d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71200cf830 0x7f71200d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f712549d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f71200d27f0 0x7f71200d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f712549d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71200d2c50 con 0x7f712015dfe0 2024-03-06T07:29:08.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f712549d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71200b31e0 con 0x7f71200d27f0 2024-03-06T07:29:08.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f712549d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71200b3d60 con 0x7f71200cf830 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f712015dfe0 0x7f71200cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f712015dfe0 0x7f71200cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60146/0 (socket says 172.21.15.59:60146) 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711effd640 1 -- 172.21.15.59:0/1584900731 learned_addr learned my addr 172.21.15.59:0/1584900731 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711effd640 1 -- 172.21.15.59:0/1584900731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71200cf830 msgr2=0x7f71200d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711f7fe640 1 --2- 172.21.15.59:0/1584900731 >> v2:172.21.15.164:3300/0 conn(0x7f71200d27f0 0x7f71200d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711e7fc640 1 --2- 172.21.15.59:0/1584900731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71200cf830 0x7f71200d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711effd640 1 --2- 172.21.15.59:0/1584900731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71200cf830 0x7f71200d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711effd640 1 -- 172.21.15.59:0/1584900731 >> v2:172.21.15.164:3300/0 conn(0x7f71200d27f0 msgr2=0x7f71200d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711effd640 1 --2- 172.21.15.59:0/1584900731 >> v2:172.21.15.164:3300/0 conn(0x7f71200d27f0 0x7f71200d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711effd640 1 -- 172.21.15.59:0/1584900731 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71200b3430 con 0x7f712015dfe0 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711f7fe640 1 --2- 172.21.15.59:0/1584900731 >> v2:172.21.15.164:3300/0 conn(0x7f71200d27f0 0x7f71200d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:08.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.348+0000 7f711e7fc640 1 --2- 172.21.15.59:0/1584900731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71200cf830 0x7f71200d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.349+0000 7f711effd640 1 --2- 172.21.15.59:0/1584900731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f712015dfe0 0x7f71200cf2f0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f7120067ad0 tx=0x7f711002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf42fa632f90d84 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.349+0000 7f711dffb640 1 -- 172.21.15.59:0/1584900731 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7110035020 con 0x7f712015dfe0 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.349+0000 7f711dffb640 1 -- 172.21.15.59:0/1584900731 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7110031870 con 0x7f712015dfe0 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.349+0000 7f711dffb640 1 -- 172.21.15.59:0/1584900731 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7110031c40 con 0x7f712015dfe0 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.350+0000 7f712549d640 1 -- 172.21.15.59:0/1584900731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f712015dfe0 msgr2=0x7f71200cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.350+0000 7f712549d640 1 --2- 172.21.15.59:0/1584900731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f712015dfe0 0x7f71200cf2f0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f7120067ad0 tx=0x7f711002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.350+0000 7f712549d640 1 -- 172.21.15.59:0/1584900731 shutdown_connections 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.350+0000 7f712549d640 1 --2- 172.21.15.59:0/1584900731 >> v2:172.21.15.164:3300/0 conn(0x7f71200d27f0 0x7f71200d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.350+0000 7f712549d640 1 --2- 172.21.15.59:0/1584900731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71200cf830 0x7f71200d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.350+0000 7f712549d640 1 --2- 172.21.15.59:0/1584900731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f712015dfe0 0x7f71200cf2f0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.350+0000 7f712549d640 1 -- 172.21.15.59:0/1584900731 >> 172.21.15.59:0/1584900731 conn(0x7f71200d3cd0 msgr2=0x7f71200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.350+0000 7f712549d640 1 -- 172.21.15.59:0/1584900731 shutdown_connections 2024-03-06T07:29:08.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.350+0000 7f712549d640 1 -- 172.21.15.59:0/1584900731 wait complete. 2024-03-06T07:29:08.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.351+0000 7f712549d640 1 Processor -- start 2024-03-06T07:29:08.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.351+0000 7f712549d640 1 -- start start 2024-03-06T07:29:08.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.352+0000 7f712549d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71200cf830 0x7f712015a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.352+0000 7f712549d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71200d27f0 0x7f71200cc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.352+0000 7f712549d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f712015dfe0 0x7f71201605e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.352+0000 7f712549d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71200d1c70 con 0x7f71200d27f0 2024-03-06T07:29:08.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.352+0000 7f712549d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71200b39f0 con 0x7f712015dfe0 2024-03-06T07:29:08.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.352+0000 7f712549d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71200d1df0 con 0x7f71200cf830 2024-03-06T07:29:08.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.352+0000 7f711e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71200d27f0 0x7f71200cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.352+0000 7f711e7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71200d27f0 0x7f71200cc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60154/0 (socket says 172.21.15.59:60154) 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.352+0000 7f711e7fc640 1 -- 172.21.15.59:0/4210714541 learned_addr learned my addr 172.21.15.59:0/4210714541 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.352+0000 7f711effd640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71200cf830 0x7f712015a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.352+0000 7f711f7fe640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f712015dfe0 0x7f71201605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.353+0000 7f711e7fc640 1 -- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71200cf830 msgr2=0x7f712015a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.353+0000 7f711e7fc640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71200cf830 0x7f712015a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.353+0000 7f711e7fc640 1 -- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f712015dfe0 msgr2=0x7f71201605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.353+0000 7f711e7fc640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f712015dfe0 0x7f71201605e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.353+0000 7f711e7fc640 1 -- 172.21.15.59:0/4210714541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712015f540 con 0x7f71200d27f0 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.353+0000 7f711e7fc640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71200d27f0 0x7f71200cc510 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f711400c7f0 tx=0x7f711400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.353+0000 7f71077fe640 1 -- 172.21.15.59:0/4210714541 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7114015070 con 0x7f71200d27f0 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.353+0000 7f71077fe640 1 -- 172.21.15.59:0/4210714541 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71140077c0 con 0x7f71200d27f0 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.353+0000 7f712549d640 1 -- 172.21.15.59:0/4210714541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712015f820 con 0x7f71200d27f0 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.353+0000 7f712549d640 1 -- 172.21.15.59:0/4210714541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712015fde0 con 0x7f71200d27f0 2024-03-06T07:29:08.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.353+0000 7f71077fe640 1 -- 172.21.15.59:0/4210714541 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7114007b90 con 0x7f71200d27f0 2024-03-06T07:29:08.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.354+0000 7f712549d640 1 -- 172.21.15.59:0/4210714541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712004f330 con 0x7f71200d27f0 2024-03-06T07:29:08.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.355+0000 7f71077fe640 1 -- 172.21.15.59:0/4210714541 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7114018070 con 0x7f71200d27f0 2024-03-06T07:29:08.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.355+0000 7f71077fe640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f710c06b4d0 0x7f710c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.355+0000 7f71077fe640 1 -- 172.21.15.59:0/4210714541 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f711408a230 con 0x7f71200d27f0 2024-03-06T07:29:08.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.356+0000 7f711effd640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f710c06b4d0 0x7f710c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.356+0000 7f711effd640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f710c06b4d0 0x7f710c06d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7110036000 tx=0x7f711003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.360+0000 7f71077fe640 1 -- 172.21.15.59:0/4210714541 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7114058a60 con 0x7f71200d27f0 2024-03-06T07:29:08.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.578+0000 7f712549d640 1 -- 172.21.15.59:0/4210714541 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71200747a0 con 0x7f71200d27f0 2024-03-06T07:29:08.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.579+0000 7f71077fe640 1 -- 172.21.15.59:0/4210714541 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3417 (secure 0 0 0) 0x7f711405cb00 con 0x7f71200d27f0 2024-03-06T07:29:08.581 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:08.581 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":84,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:28:55.978187+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8673,8676]},"id":9}]} 2024-03-06T07:29:08.582 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 84 2024-03-06T07:29:08.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 -- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f710c06b4d0 msgr2=0x7f710c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f710c06b4d0 0x7f710c06d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7110036000 tx=0x7f711003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:08.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 -- 172.21.15.59:0/4210714541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71200d27f0 msgr2=0x7f71200cc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71200d27f0 0x7f71200cc510 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f711400c7f0 tx=0x7f711400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 -- 172.21.15.59:0/4210714541 shutdown_connections 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f710c06b4d0 0x7f710c06d990 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f712015dfe0 0x7f71201605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71200d27f0 0x7f71200cc510 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 --2- 172.21.15.59:0/4210714541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71200cf830 0x7f712015a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 -- 172.21.15.59:0/4210714541 >> 172.21.15.59:0/4210714541 conn(0x7f71200d3cd0 msgr2=0x7f712006bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 -- 172.21.15.59:0/4210714541 shutdown_connections 2024-03-06T07:29:08.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:08.581+0000 7f712549d640 1 -- 172.21.15.59:0/4210714541 wait complete. 2024-03-06T07:29:08.594 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:29:08.711 INFO:teuthology.orchestra.run.smithi059.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:08.711 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:08.711 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:10.712 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:10.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.847+0000 7fb391fa1640 1 Processor -- start 2024-03-06T07:29:10.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.847+0000 7fb391fa1640 1 -- start start 2024-03-06T07:29:10.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.848+0000 7fb391fa1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 0x7fb38c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.848+0000 7fb391fa1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb38c0cf830 0x7fb38c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.848+0000 7fb391fa1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38c0b3430 con 0x7fb38c0d27f0 2024-03-06T07:29:10.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.848+0000 7fb391fa1640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38c0d2c50 con 0x7fb38c0cf830 2024-03-06T07:29:10.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.848+0000 7fb391fa1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0b31e0 con 0x7fb38c15dfe0 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.848+0000 7fb383fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb38c0cf830 0x7fb38c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.848+0000 7fb383fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb38c0cf830 0x7fb38c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60158/0 (socket says 172.21.15.59:60158) 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.848+0000 7fb383fff640 1 -- 172.21.15.59:0/2844521879 learned_addr learned my addr 172.21.15.59:0/2844521879 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.849+0000 7fb383fff640 1 -- 172.21.15.59:0/2844521879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 msgr2=0x7fb38c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.849+0000 7fb38b7fe640 1 --2- 172.21.15.59:0/2844521879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 0x7fb38c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.849+0000 7fb383fff640 1 --2- 172.21.15.59:0/2844521879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 0x7fb38c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.849+0000 7fb383fff640 1 -- 172.21.15.59:0/2844521879 >> v1:172.21.15.164:6789/0 conn(0x7fb38c0d27f0 legacy=0x7fb38c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.849+0000 7fb383fff640 1 -- 172.21.15.59:0/2844521879 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c0b3710 con 0x7fb38c0cf830 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.849+0000 7fb38b7fe640 1 --2- 172.21.15.59:0/2844521879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 0x7fb38c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.849+0000 7fb383fff640 1 --2- 172.21.15.59:0/2844521879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0cf830 0x7fb38c0d22b0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fb374009870 tx=0x7fb37402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224642155c207f86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.850+0000 7fb38affd640 1 -- 172.21.15.59:0/2844521879 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb374035020 con 0x7fb38c0cf830 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.850+0000 7fb38affd640 1 -- 172.21.15.59:0/2844521879 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb374031640 con 0x7fb38c0cf830 2024-03-06T07:29:10.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.850+0000 7fb38affd640 1 -- 172.21.15.59:0/2844521879 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37402d040 con 0x7fb38c0cf830 2024-03-06T07:29:10.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.850+0000 7fb391fa1640 1 -- 172.21.15.59:0/2844521879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0cf830 msgr2=0x7fb38c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.850+0000 7fb391fa1640 1 --2- 172.21.15.59:0/2844521879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0cf830 0x7fb38c0d22b0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fb374009870 tx=0x7fb37402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.850+0000 7fb391fa1640 1 -- 172.21.15.59:0/2844521879 shutdown_connections 2024-03-06T07:29:10.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.850+0000 7fb391fa1640 1 --2- 172.21.15.59:0/2844521879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0cf830 0x7fb38c0d22b0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.850+0000 7fb391fa1640 1 --2- 172.21.15.59:0/2844521879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 0x7fb38c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.850+0000 7fb391fa1640 1 -- 172.21.15.59:0/2844521879 >> 172.21.15.59:0/2844521879 conn(0x7fb38c0d3cd0 msgr2=0x7fb38c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.850+0000 7fb391fa1640 1 -- 172.21.15.59:0/2844521879 shutdown_connections 2024-03-06T07:29:10.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.850+0000 7fb391fa1640 1 -- 172.21.15.59:0/2844521879 wait complete. 2024-03-06T07:29:10.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.852+0000 7fb391fa1640 1 Processor -- start 2024-03-06T07:29:10.854 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.852+0000 7fb391fa1640 1 -- start start 2024-03-06T07:29:10.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.853+0000 7fb391fa1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0cf830 0x7fb38c1f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.853+0000 7fb391fa1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb38c0d27f0 0x7fb38c15bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.853+0000 7fb391fa1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 0x7fb38c0cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.853+0000 7fb391fa1640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0d1c70 con 0x7fb38c0cf830 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.853+0000 7fb391fa1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0b3d60 con 0x7fb38c0d27f0 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.853+0000 7fb391fa1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0d1df0 con 0x7fb38c15dfe0 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.853+0000 7fb38b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0cf830 0x7fb38c1f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.853+0000 7fb38b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0cf830 0x7fb38c1f4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60170/0 (socket says 172.21.15.59:60170) 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.853+0000 7fb38b7fe640 1 -- 172.21.15.59:0/3352131301 learned_addr learned my addr 172.21.15.59:0/3352131301 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.853+0000 7fb38bfff640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 0x7fb38c0cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.853+0000 7fb383fff640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb38c0d27f0 0x7fb38c15bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb38b7fe640 1 -- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 msgr2=0x7fb38c0cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb38b7fe640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 0x7fb38c0cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb38b7fe640 1 -- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb38c0d27f0 msgr2=0x7fb38c15bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb38b7fe640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb38c0d27f0 0x7fb38c15bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb38b7fe640 1 -- 172.21.15.59:0/3352131301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c0cc560 con 0x7fb38c0cf830 2024-03-06T07:29:10.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb38bfff640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 0x7fb38c0cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:10.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb383fff640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb38c0d27f0 0x7fb38c15bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:10.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb38b7fe640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0cf830 0x7fb38c1f4c20 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00c7f0 tx=0x7fb37c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb388ff9640 1 -- 172.21.15.59:0/3352131301 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c015070 con 0x7fb38c0cf830 2024-03-06T07:29:10.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb388ff9640 1 -- 172.21.15.59:0/3352131301 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37c007820 con 0x7fb38c0cf830 2024-03-06T07:29:10.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb391fa1640 1 -- 172.21.15.59:0/3352131301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38c15fba0 con 0x7fb38c0cf830 2024-03-06T07:29:10.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb388ff9640 1 -- 172.21.15.59:0/3352131301 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c007bd0 con 0x7fb38c0cf830 2024-03-06T07:29:10.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.854+0000 7fb391fa1640 1 -- 172.21.15.59:0/3352131301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38c160160 con 0x7fb38c0cf830 2024-03-06T07:29:10.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.855+0000 7fb391fa1640 1 -- 172.21.15.59:0/3352131301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38c04f330 con 0x7fb38c0cf830 2024-03-06T07:29:10.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.856+0000 7fb388ff9640 1 -- 172.21.15.59:0/3352131301 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb37c018070 con 0x7fb38c0cf830 2024-03-06T07:29:10.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.856+0000 7fb388ff9640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb36c06b4d0 0x7fb36c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.856+0000 7fb388ff9640 1 -- 172.21.15.59:0/3352131301 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb37c08a0d0 con 0x7fb38c0cf830 2024-03-06T07:29:10.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.856+0000 7fb383fff640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb36c06b4d0 0x7fb36c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.857+0000 7fb383fff640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb36c06b4d0 0x7fb36c06d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb374036000 tx=0x7fb37403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:10.865+0000 7fb388ff9640 1 -- 172.21.15.59:0/3352131301 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb37c058fe0 con 0x7fb38c0cf830 2024-03-06T07:29:11.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.085+0000 7fb391fa1640 1 -- 172.21.15.59:0/3352131301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb38c06c6d0 con 0x7fb38c0cf830 2024-03-06T07:29:11.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.086+0000 7fb388ff9640 1 -- 172.21.15.59:0/3352131301 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3417 (secure 0 0 0) 0x7fb37c0588e0 con 0x7fb38c0cf830 2024-03-06T07:29:11.088 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:11.088 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":84,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:28:55.978187+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8673,8676]},"id":9}]} 2024-03-06T07:29:11.088 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 84 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 -- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb36c06b4d0 msgr2=0x7fb36c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb36c06b4d0 0x7fb36c06d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb374036000 tx=0x7fb37403c040 comp rx=0 tx=0).stop 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 -- 172.21.15.59:0/3352131301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0cf830 msgr2=0x7fb38c1f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0cf830 0x7fb38c1f4c20 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fb37c00c7f0 tx=0x7fb37c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 -- 172.21.15.59:0/3352131301 shutdown_connections 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb36c06b4d0 0x7fb36c06d990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb38c15dfe0 0x7fb38c0cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb38c0d27f0 0x7fb38c15bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 --2- 172.21.15.59:0/3352131301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb38c0cf830 0x7fb38c1f4c20 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 -- 172.21.15.59:0/3352131301 >> 172.21.15.59:0/3352131301 conn(0x7fb38c0d3cd0 msgr2=0x7fb38c06cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 -- 172.21.15.59:0/3352131301 shutdown_connections 2024-03-06T07:29:11.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:11.088+0000 7fb391fa1640 1 -- 172.21.15.59:0/3352131301 wait complete. 2024-03-06T07:29:11.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:29:11.220 INFO:teuthology.orchestra.run.smithi059.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:11.221 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:11.221 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:13.222 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:13.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.322+0000 7ffba89e2640 1 Processor -- start 2024-03-06T07:29:13.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.322+0000 7ffba89e2640 1 -- start start 2024-03-06T07:29:13.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba89e2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffba40d22b0 0x7ffba40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba89e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba40b41f0 0x7ffba40b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba89e2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba40b3430 con 0x7ffba415dfe0 2024-03-06T07:29:13.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba89e2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba40d2c50 con 0x7ffba40d22b0 2024-03-06T07:29:13.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba89e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba40b31e0 con 0x7ffba40b41f0 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba259c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ffba415dfe0 0x7ffba40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49682/0 (socket says 172.21.15.59:49682) 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba259c640 1 -- 172.21.15.59:0/2067685566 learned_addr learned my addr 172.21.15.59:0/2067685566 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba2d9d640 1 --2- 172.21.15.59:0/2067685566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba40b41f0 0x7ffba40b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba1d9b640 1 --2- 172.21.15.59:0/2067685566 >> v2:172.21.15.164:3300/0 conn(0x7ffba40d22b0 0x7ffba40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba159a640 1 -- 172.21.15.59:0/2067685566 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646595314 0 0) 0x7ffba40b3430 con 0x7ffba415dfe0 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba159a640 1 -- 172.21.15.59:0/2067685566 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb8c003520 con 0x7ffba415dfe0 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba159a640 1 -- 172.21.15.59:0/2067685566 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351457760 0 0) 0x7ffb8c003520 con 0x7ffba415dfe0 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba2d9d640 1 -- 172.21.15.59:0/2067685566 >> v2:172.21.15.164:3300/0 conn(0x7ffba40d22b0 msgr2=0x7ffba40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba2d9d640 1 --2- 172.21.15.59:0/2067685566 >> v2:172.21.15.164:3300/0 conn(0x7ffba40d22b0 0x7ffba40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba2d9d640 1 -- 172.21.15.59:0/2067685566 >> v1:172.21.15.59:6789/0 conn(0x7ffba415dfe0 legacy=0x7ffba40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba2d9d640 1 -- 172.21.15.59:0/2067685566 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba40b3710 con 0x7ffba40b41f0 2024-03-06T07:29:13.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.324+0000 7ffba1d9b640 1 --2- 172.21.15.59:0/2067685566 >> v2:172.21.15.164:3300/0 conn(0x7ffba40d22b0 0x7ffba40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:13.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.325+0000 7ffba2d9d640 1 --2- 172.21.15.59:0/2067685566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba40b41f0 0x7ffba40b4680 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c009d90 tx=0x7ffb9c02bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76d736887b1f5f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.325+0000 7ffba159a640 1 -- 172.21.15.59:0/2067685566 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c039070 con 0x7ffba40b41f0 2024-03-06T07:29:13.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.325+0000 7ffba159a640 1 -- 172.21.15.59:0/2067685566 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9c02c8a0 con 0x7ffba40b41f0 2024-03-06T07:29:13.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.325+0000 7ffba159a640 1 -- 172.21.15.59:0/2067685566 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb94001670 con 0x7ffba40b41f0 2024-03-06T07:29:13.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.326+0000 7ffba89e2640 1 -- 172.21.15.59:0/2067685566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba40b41f0 msgr2=0x7ffba40b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.326+0000 7ffba89e2640 1 --2- 172.21.15.59:0/2067685566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba40b41f0 0x7ffba40b4680 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c009d90 tx=0x7ffb9c02bd70 comp rx=0 tx=0).stop 2024-03-06T07:29:13.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.326+0000 7ffba89e2640 1 -- 172.21.15.59:0/2067685566 shutdown_connections 2024-03-06T07:29:13.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.326+0000 7ffba89e2640 1 --2- 172.21.15.59:0/2067685566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba40b41f0 0x7ffba40b4680 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.326+0000 7ffba89e2640 1 --2- 172.21.15.59:0/2067685566 >> v2:172.21.15.164:3300/0 conn(0x7ffba40d22b0 0x7ffba40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.326+0000 7ffba89e2640 1 -- 172.21.15.59:0/2067685566 >> 172.21.15.59:0/2067685566 conn(0x7ffba40d3cd0 msgr2=0x7ffba40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.326+0000 7ffba89e2640 1 -- 172.21.15.59:0/2067685566 shutdown_connections 2024-03-06T07:29:13.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.326+0000 7ffba89e2640 1 -- 172.21.15.59:0/2067685566 wait complete. 2024-03-06T07:29:13.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.327+0000 7ffba89e2640 1 Processor -- start 2024-03-06T07:29:13.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.327+0000 7ffba89e2640 1 -- start start 2024-03-06T07:29:13.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba89e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba40b41f0 0x7ffba415c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba89e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba40d22b0 0x7ffba40cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba89e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffba415dfe0 0x7ffba4160590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba89e2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba40cfa00 con 0x7ffba40b41f0 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba89e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba40b3d60 con 0x7ffba415dfe0 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba89e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba40d1c70 con 0x7ffba40d22b0 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba40b41f0 0x7ffba415c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba40b41f0 0x7ffba415c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60176/0 (socket says 172.21.15.59:60176) 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba259c640 1 -- 172.21.15.59:0/1684140430 learned_addr learned my addr 172.21.15.59:0/1684140430 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba2d9d640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffba415dfe0 0x7ffba4160590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba1d9b640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba40d22b0 0x7ffba40cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba259c640 1 -- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba40d22b0 msgr2=0x7ffba40cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba259c640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba40d22b0 0x7ffba40cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba259c640 1 -- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffba415dfe0 msgr2=0x7ffba4160590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba259c640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffba415dfe0 0x7ffba4160590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.328+0000 7ffba259c640 1 -- 172.21.15.59:0/1684140430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba40cc820 con 0x7ffba40b41f0 2024-03-06T07:29:13.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.329+0000 7ffba259c640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba40b41f0 0x7ffba415c510 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7ffb940017e0 tx=0x7ffb9400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.329+0000 7ffb92ffd640 1 -- 172.21.15.59:0/1684140430 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb94005ad0 con 0x7ffba40b41f0 2024-03-06T07:29:13.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.329+0000 7ffb92ffd640 1 -- 172.21.15.59:0/1684140430 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb94005c90 con 0x7ffba40b41f0 2024-03-06T07:29:13.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.329+0000 7ffba89e2640 1 -- 172.21.15.59:0/1684140430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba40ccb00 con 0x7ffba40b41f0 2024-03-06T07:29:13.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.329+0000 7ffb92ffd640 1 -- 172.21.15.59:0/1684140430 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb940072b0 con 0x7ffba40b41f0 2024-03-06T07:29:13.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.329+0000 7ffba89e2640 1 -- 172.21.15.59:0/1684140430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba40cd0c0 con 0x7ffba40b41f0 2024-03-06T07:29:13.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.330+0000 7ffba89e2640 1 -- 172.21.15.59:0/1684140430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba40d1a90 con 0x7ffba40b41f0 2024-03-06T07:29:13.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.330+0000 7ffb92ffd640 1 -- 172.21.15.59:0/1684140430 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb94007450 con 0x7ffba40b41f0 2024-03-06T07:29:13.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.331+0000 7ffb92ffd640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8406b4d0 0x7ffb8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.331+0000 7ffb92ffd640 1 -- 172.21.15.59:0/1684140430 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb94089ee0 con 0x7ffba40b41f0 2024-03-06T07:29:13.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.331+0000 7ffba1d9b640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8406b4d0 0x7ffb8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.331+0000 7ffba1d9b640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8406b4d0 0x7ffb8406d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ffb98009560 tx=0x7ffb98009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.339+0000 7ffb92ffd640 1 -- 172.21.15.59:0/1684140430 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffba40d1a90 con 0x7ffba40b41f0 2024-03-06T07:29:13.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.584+0000 7ffba89e2640 1 -- 172.21.15.59:0/1684140430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffba406bfe0 con 0x7ffba40b41f0 2024-03-06T07:29:13.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.584+0000 7ffb92ffd640 1 -- 172.21.15.59:0/1684140430 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+3418 (secure 0 0 0) 0x7ffb94058710 con 0x7ffba40b41f0 2024-03-06T07:29:13.586 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:13.586 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":85,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:12.593143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:active","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676,8673]},"id":9}]} 2024-03-06T07:29:13.587 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 85 2024-03-06T07:29:13.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 -- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8406b4d0 msgr2=0x7ffb8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8406b4d0 0x7ffb8406d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ffb98009560 tx=0x7ffb98009190 comp rx=0 tx=0).stop 2024-03-06T07:29:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 -- 172.21.15.59:0/1684140430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba40b41f0 msgr2=0x7ffba415c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba40b41f0 0x7ffba415c510 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7ffb940017e0 tx=0x7ffb9400cbe0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 -- 172.21.15.59:0/1684140430 shutdown_connections 2024-03-06T07:29:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ffb8406b4d0 0x7ffb8406d990 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffba415dfe0 0x7ffba4160590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffba40d22b0 0x7ffba40cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 --2- 172.21.15.59:0/1684140430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffba40b41f0 0x7ffba415c510 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 -- 172.21.15.59:0/1684140430 >> 172.21.15.59:0/1684140430 conn(0x7ffba40d3cd0 msgr2=0x7ffba40d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 -- 172.21.15.59:0/1684140430 shutdown_connections 2024-03-06T07:29:13.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:13.586+0000 7ffba89e2640 1 -- 172.21.15.59:0/1684140430 wait complete. 2024-03-06T07:29:13.599 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-06T07:29:13.768 INFO:teuthology.orchestra.run.smithi059.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"10","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:29:13.768 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "10" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:13.769 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-03-06T07:29:13.769 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:29:13.769 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> stdin-killer -- python3 -c ' 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> import fcntl 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> import sys 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> import time 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> while True: 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-03-06T07:29:13.785 DEBUG:teuthology.orchestra.run.smithi059:> try: 2024-03-06T07:29:13.786 DEBUG:teuthology.orchestra.run.smithi059:> if os.read(0, 4096) == b"": 2024-03-06T07:29:13.786 DEBUG:teuthology.orchestra.run.smithi059:> break 2024-03-06T07:29:13.786 DEBUG:teuthology.orchestra.run.smithi059:> except BlockingIOError: 2024-03-06T07:29:13.786 DEBUG:teuthology.orchestra.run.smithi059:> pass 2024-03-06T07:29:13.786 DEBUG:teuthology.orchestra.run.smithi059:> time.sleep(2) 2024-03-06T07:29:13.786 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:29:13.827 DEBUG:teuthology.orchestra.run.smithi059:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:13.843 INFO:teuthology.orchestra.run.smithi059.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-03-06T07:29:13.901 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:14.844 DEBUG:teuthology.orchestra.run.smithi059:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:14.860 INFO:teuthology.orchestra.run.smithi059.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:14.861 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-03-06T07:29:14.861 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 2ah/42d Inode: 2199023255552 Links: 2 2024-03-06T07:29:14.861 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-06T07:29:14.861 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-06T07:29:14.861 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2024-03-06 07:29:13.892644751 +0000 2024-03-06T07:29:14.861 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2024-03-06 07:29:13.892644751 +0000 2024-03-06T07:29:14.861 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2024-03-06 07:29:13.892644751 +0000 2024-03-06T07:29:14.861 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2024-03-06T07:29:14.861 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-03-06T07:29:14.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:14.861 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-03-06T07:29:14.919 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:29:15.078 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:29:15.078 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:29:15.078 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:29:15.198 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:29:15.199 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:29:15.199 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:15.317 INFO:teuthology.orchestra.run.smithi059.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:29:15.318 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:29:15.318 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:29:15.469 INFO:teuthology.orchestra.run.smithi059.stdout:{"message":"","return_code":0} 2024-03-06T07:29:15.469 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:15.469 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json cache drop 2024-03-06T07:29:15.588 INFO:teuthology.orchestra.run.smithi059.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":328,"bytes":65087}},"duration":0.0057247640000000002} 2024-03-06T07:29:15.589 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 65087, "items": 328 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.005724764, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:29:15.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:29:15.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.691+0000 7fb1e8815640 1 Processor -- start 2024-03-06T07:29:15.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.691+0000 7fb1e8815640 1 -- start start 2024-03-06T07:29:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e8815640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb1e40d4860 0x7fb1e40d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e8815640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb1e40b0e30 0x7fb1e40b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e8815640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e41618b0 con 0x7fb1e41614d0 2024-03-06T07:29:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e8815640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e40d4cf0 con 0x7fb1e40b0e30 2024-03-06T07:29:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e8815640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e40cf400 con 0x7fb1e40d4860 2024-03-06T07:29:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e2d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb1e40b0e30 0x7fb1e40b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e259c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb1e41614d0 0x7fb1e40d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44956/0 (socket says 172.21.15.59:44956) 2024-03-06T07:29:15.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e259c640 1 -- 172.21.15.59:0/1401201466 learned_addr learned my addr 172.21.15.59:0/1401201466 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e2d9d640 1 -- 172.21.15.59:0/1401201466 >> v2:172.21.15.164:3300/0 conn(0x7fb1e40d4860 msgr2=0x7fb1e40d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e2d9d640 1 --2- 172.21.15.59:0/1401201466 >> v2:172.21.15.164:3300/0 conn(0x7fb1e40d4860 0x7fb1e40d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e2d9d640 1 -- 172.21.15.59:0/1401201466 >> v1:172.21.15.59:6789/0 conn(0x7fb1e41614d0 legacy=0x7fb1e40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e159a640 1 -- 172.21.15.59:0/1401201466 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081349231 0 0) 0x7fb1e41618b0 con 0x7fb1e41614d0 2024-03-06T07:29:15.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.692+0000 7fb1e2d9d640 1 -- 172.21.15.59:0/1401201466 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e40cf6a0 con 0x7fb1e40b0e30 2024-03-06T07:29:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.693+0000 7fb1e2d9d640 1 --2- 172.21.15.59:0/1401201466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1e40b0e30 0x7fb1e40b1260 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40ce550 tx=0x7fb1dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb761aa8bfcb4f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.693+0000 7fb1e159a640 1 -- 172.21.15.59:0/1401201466 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc039070 con 0x7fb1e40b0e30 2024-03-06T07:29:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.694+0000 7fb1e159a640 1 -- 172.21.15.59:0/1401201466 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1dc02cc80 con 0x7fb1e40b0e30 2024-03-06T07:29:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.694+0000 7fb1e159a640 1 -- 172.21.15.59:0/1401201466 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc034660 con 0x7fb1e40b0e30 2024-03-06T07:29:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.694+0000 7fb1e8815640 1 -- 172.21.15.59:0/1401201466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1e40b0e30 msgr2=0x7fb1e40b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.694+0000 7fb1e8815640 1 --2- 172.21.15.59:0/1401201466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1e40b0e30 0x7fb1e40b1260 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40ce550 tx=0x7fb1dc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.694+0000 7fb1e8815640 1 -- 172.21.15.59:0/1401201466 shutdown_connections 2024-03-06T07:29:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.694+0000 7fb1e8815640 1 --2- 172.21.15.59:0/1401201466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1e40b0e30 0x7fb1e40b1260 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.694+0000 7fb1e8815640 1 --2- 172.21.15.59:0/1401201466 >> v2:172.21.15.164:3300/0 conn(0x7fb1e40d4860 0x7fb1e40d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.694+0000 7fb1e8815640 1 -- 172.21.15.59:0/1401201466 >> 172.21.15.59:0/1401201466 conn(0x7fb1e40d3d40 msgr2=0x7fb1e40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.694+0000 7fb1e8815640 1 -- 172.21.15.59:0/1401201466 shutdown_connections 2024-03-06T07:29:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.694+0000 7fb1e8815640 1 -- 172.21.15.59:0/1401201466 wait complete. 2024-03-06T07:29:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.696+0000 7fb1e8815640 1 Processor -- start 2024-03-06T07:29:15.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.696+0000 7fb1e8815640 1 -- start start 2024-03-06T07:29:15.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.696+0000 7fb1e8815640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1e40b0e30 0x7fb1e40ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.696+0000 7fb1e8815640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1e40d4860 0x7fb1e4155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.696+0000 7fb1e8815640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1e41614d0 0x7fb1e40b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.696+0000 7fb1e8815640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e40b0910 con 0x7fb1e40d4860 2024-03-06T07:29:15.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.696+0000 7fb1e8815640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e40cfc60 con 0x7fb1e41614d0 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.696+0000 7fb1e8815640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e40d2560 con 0x7fb1e40b0e30 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.696+0000 7fb1e1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1e40d4860 0x7fb1e4155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e1d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1e40d4860 0x7fb1e4155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53094/0 (socket says 172.21.15.59:53094) 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e1d9b640 1 -- 172.21.15.59:0/3583941308 learned_addr learned my addr 172.21.15.59:0/3583941308 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e259c640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1e40b0e30 0x7fb1e40ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e2d9d640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1e41614d0 0x7fb1e40b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e1d9b640 1 -- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1e40b0e30 msgr2=0x7fb1e40ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e1d9b640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1e40b0e30 0x7fb1e40ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e1d9b640 1 -- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1e41614d0 msgr2=0x7fb1e40b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e1d9b640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1e41614d0 0x7fb1e40b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e1d9b640 1 -- 172.21.15.59:0/3583941308 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e41558c0 con 0x7fb1e40d4860 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e1d9b640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1e40d4860 0x7fb1e4155380 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800b4a0 tx=0x7fb1d800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1d2ffd640 1 -- 172.21.15.59:0/3583941308 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8014070 con 0x7fb1e40d4860 2024-03-06T07:29:15.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1d2ffd640 1 -- 172.21.15.59:0/3583941308 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d8010610 con 0x7fb1e40d4860 2024-03-06T07:29:15.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e8815640 1 -- 172.21.15.59:0/3583941308 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e415e740 con 0x7fb1e40d4860 2024-03-06T07:29:15.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1d2ffd640 1 -- 172.21.15.59:0/3583941308 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8010a00 con 0x7fb1e40d4860 2024-03-06T07:29:15.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.697+0000 7fb1e8815640 1 -- 172.21.15.59:0/3583941308 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e40c95e0 con 0x7fb1e40d4860 2024-03-06T07:29:15.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.699+0000 7fb1e8815640 1 -- 172.21.15.59:0/3583941308 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e415ecc0 con 0x7fb1e40d4860 2024-03-06T07:29:15.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.699+0000 7fb1d2ffd640 1 -- 172.21.15.59:0/3583941308 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1d8017070 con 0x7fb1e40d4860 2024-03-06T07:29:15.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.699+0000 7fb1d2ffd640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1c406b4d0 0x7fb1c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.700+0000 7fb1d2ffd640 1 -- 172.21.15.59:0/3583941308 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1d8089c60 con 0x7fb1e40d4860 2024-03-06T07:29:15.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.700+0000 7fb1e259c640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1c406b4d0 0x7fb1c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.700+0000 7fb1e259c640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1c406b4d0 0x7fb1c406d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4000f60 tx=0x7fb1d4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.704+0000 7fb1d2ffd640 1 -- 172.21.15.59:0/3583941308 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1d8058ae0 con 0x7fb1e40d4860 2024-03-06T07:29:15.902 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:15.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:15.960+0000 7fb1e8815640 1 -- 172.21.15.59:0/3583941308 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fb1e40d0aa0 con 0x7fb1e40d4860 2024-03-06T07:29:16.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.041+0000 7fb1d2ffd640 1 -- 172.21.15.59:0/3583941308 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v86) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb1d80583e0 con 0x7fb1e40d4860 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 -- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1c406b4d0 msgr2=0x7fb1c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1c406b4d0 0x7fb1c406d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4000f60 tx=0x7fb1d4009040 comp rx=0 tx=0).stop 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 -- 172.21.15.59:0/3583941308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1e40d4860 msgr2=0x7fb1e4155380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1e40d4860 0x7fb1e4155380 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800b4a0 tx=0x7fb1d800b960 comp rx=0 tx=0).stop 2024-03-06T07:29:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 -- 172.21.15.59:0/3583941308 shutdown_connections 2024-03-06T07:29:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1c406b4d0 0x7fb1c406d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1e41614d0 0x7fb1e40b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1e40d4860 0x7fb1e4155380 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 --2- 172.21.15.59:0/3583941308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1e40b0e30 0x7fb1e40ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 -- 172.21.15.59:0/3583941308 >> 172.21.15.59:0/3583941308 conn(0x7fb1e40d3d40 msgr2=0x7fb1e406b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 -- 172.21.15.59:0/3583941308 shutdown_connections 2024-03-06T07:29:16.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.044+0000 7fb1e8815640 1 -- 172.21.15.59:0/3583941308 wait complete. 2024-03-06T07:29:16.058 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:16.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.157+0000 7f54b450c640 1 Processor -- start 2024-03-06T07:29:16.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.158+0000 7f54b450c640 1 -- start start 2024-03-06T07:29:16.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.158+0000 7f54b450c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f54ac0d22b0 0x7f54ac0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.158+0000 7f54b450c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac0b3430 con 0x7f54ac0b41f0 2024-03-06T07:29:16.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.158+0000 7f54b450c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ac0b3710 con 0x7f54ac15dfe0 2024-03-06T07:29:16.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.158+0000 7f54b450c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54ac0d2c50 con 0x7f54ac0d22b0 2024-03-06T07:29:16.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.159+0000 7f54b1aa7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f54ac0d22b0 0x7f54ac0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.159+0000 7f54b2aa9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f54ac0b41f0 0x7f54ac0b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44964/0 (socket says 172.21.15.59:44964) 2024-03-06T07:29:16.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.159+0000 7f54b2aa9640 1 -- 172.21.15.59:0/487046756 learned_addr learned my addr 172.21.15.59:0/487046756 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:16.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.159+0000 7f54b1aa7640 1 -- 172.21.15.59:0/487046756 >> v1:172.21.15.164:6789/0 conn(0x7f54ac15dfe0 legacy=0x7f54ac0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:16.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.159+0000 7f54b1aa7640 1 -- 172.21.15.59:0/487046756 >> v1:172.21.15.59:6789/0 conn(0x7f54ac0b41f0 legacy=0x7f54ac0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.159+0000 7f54b1aa7640 1 -- 172.21.15.59:0/487046756 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac0b39f0 con 0x7f54ac0d22b0 2024-03-06T07:29:16.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.159+0000 7f54b12a6640 1 -- 172.21.15.59:0/487046756 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838538129 0 0) 0x7f54ac0b3430 con 0x7f54ac0b41f0 2024-03-06T07:29:16.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.160+0000 7f54b1aa7640 1 --2- 172.21.15.59:0/487046756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54ac0d22b0 0x7f54ac0d2710 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f54a00097d0 tx=0x7f54a002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3691e21c6ce0ad03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.160+0000 7f54b12a6640 1 -- 172.21.15.59:0/487046756 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0034070 con 0x7f54ac0d22b0 2024-03-06T07:29:16.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.160+0000 7f54b12a6640 1 -- 172.21.15.59:0/487046756 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a0030610 con 0x7f54ac0d22b0 2024-03-06T07:29:16.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.160+0000 7f54b12a6640 1 -- 172.21.15.59:0/487046756 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a00309e0 con 0x7f54ac0d22b0 2024-03-06T07:29:16.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.161+0000 7f54b450c640 1 -- 172.21.15.59:0/487046756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54ac0d22b0 msgr2=0x7f54ac0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.161+0000 7f54b450c640 1 --2- 172.21.15.59:0/487046756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54ac0d22b0 0x7f54ac0d2710 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f54a00097d0 tx=0x7f54a002b960 comp rx=0 tx=0).stop 2024-03-06T07:29:16.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.161+0000 7f54b450c640 1 -- 172.21.15.59:0/487046756 shutdown_connections 2024-03-06T07:29:16.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.161+0000 7f54b450c640 1 --2- 172.21.15.59:0/487046756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54ac0d22b0 0x7f54ac0d2710 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.161+0000 7f54b450c640 1 -- 172.21.15.59:0/487046756 >> 172.21.15.59:0/487046756 conn(0x7f54ac0d3cd0 msgr2=0x7f54ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.161+0000 7f54b450c640 1 -- 172.21.15.59:0/487046756 shutdown_connections 2024-03-06T07:29:16.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.161+0000 7f54b450c640 1 -- 172.21.15.59:0/487046756 wait complete. 2024-03-06T07:29:16.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.163+0000 7f54b450c640 1 Processor -- start 2024-03-06T07:29:16.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.163+0000 7f54b450c640 1 -- start start 2024-03-06T07:29:16.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b450c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54ac0b41f0 0x7f54ac0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b450c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54ac0d22b0 0x7f54ac1510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b450c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54ac15dfe0 0x7f54ac0b8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b450c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac0b3d60 con 0x7f54ac0d22b0 2024-03-06T07:29:16.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b450c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac0b31e0 con 0x7f54ac15dfe0 2024-03-06T07:29:16.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b450c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac0d1c70 con 0x7f54ac0b41f0 2024-03-06T07:29:16.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b1aa7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54ac0d22b0 0x7f54ac1510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b1aa7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54ac0d22b0 0x7f54ac1510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53110/0 (socket says 172.21.15.59:53110) 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b1aa7640 1 -- 172.21.15.59:0/1031254050 learned_addr learned my addr 172.21.15.59:0/1031254050 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b22a8640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54ac0b41f0 0x7f54ac0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b2aa9640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54ac15dfe0 0x7f54ac0b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b1aa7640 1 -- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54ac0b41f0 msgr2=0x7f54ac0b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b1aa7640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54ac0b41f0 0x7f54ac0b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b1aa7640 1 -- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54ac15dfe0 msgr2=0x7f54ac0b8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b1aa7640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54ac15dfe0 0x7f54ac0b8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b1aa7640 1 -- 172.21.15.59:0/1031254050 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac0b9380 con 0x7f54ac0d22b0 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.164+0000 7f54b1aa7640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54ac0d22b0 0x7f54ac1510b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f54a00098f0 tx=0x7f54a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.165+0000 7f54aaffd640 1 -- 172.21.15.59:0/1031254050 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0034040 con 0x7f54ac0d22b0 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.165+0000 7f54aaffd640 1 -- 172.21.15.59:0/1031254050 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a0039a10 con 0x7f54ac0d22b0 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.165+0000 7f54b450c640 1 -- 172.21.15.59:0/1031254050 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ac0b9660 con 0x7f54ac0d22b0 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.165+0000 7f54aaffd640 1 -- 172.21.15.59:0/1031254050 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0039de0 con 0x7f54ac0d22b0 2024-03-06T07:29:16.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.165+0000 7f54b450c640 1 -- 172.21.15.59:0/1031254050 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ac0b6e10 con 0x7f54ac0d22b0 2024-03-06T07:29:16.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.166+0000 7f54b450c640 1 -- 172.21.15.59:0/1031254050 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ac04f330 con 0x7f54ac0d22b0 2024-03-06T07:29:16.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.166+0000 7f54aaffd640 1 -- 172.21.15.59:0/1031254050 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54a0037070 con 0x7f54ac0d22b0 2024-03-06T07:29:16.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.166+0000 7f54aaffd640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f549806b4d0 0x7f549806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.167+0000 7f54aaffd640 1 -- 172.21.15.59:0/1031254050 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54a00ac1d0 con 0x7f54ac0d22b0 2024-03-06T07:29:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.167+0000 7f54b22a8640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f549806b4d0 0x7f549806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.167+0000 7f54b22a8640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f549806b4d0 0x7f549806d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f54a40012e0 tx=0x7f54a4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.171+0000 7f54aaffd640 1 -- 172.21.15.59:0/1031254050 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54a007aa00 con 0x7f54ac0d22b0 2024-03-06T07:29:16.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.421+0000 7f54b450c640 1 -- 172.21.15.59:0/1031254050 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54ac0d0920 con 0x7f54ac0d22b0 2024-03-06T07:29:16.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.421+0000 7f54aaffd640 1 -- 172.21.15.59:0/1031254050 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f54a007eaa0 con 0x7f54ac0d22b0 2024-03-06T07:29:16.423 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:16.423 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:16.044432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:16.424 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 87 2024-03-06T07:29:16.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 -- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f549806b4d0 msgr2=0x7f549806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f549806b4d0 0x7f549806d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f54a40012e0 tx=0x7f54a4009040 comp rx=0 tx=0).stop 2024-03-06T07:29:16.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 -- 172.21.15.59:0/1031254050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54ac0d22b0 msgr2=0x7f54ac1510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54ac0d22b0 0x7f54ac1510b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f54a00098f0 tx=0x7f54a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 -- 172.21.15.59:0/1031254050 shutdown_connections 2024-03-06T07:29:16.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f549806b4d0 0x7f549806d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54ac15dfe0 0x7f54ac0b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54ac0d22b0 0x7f54ac1510b0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 --2- 172.21.15.59:0/1031254050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54ac0b41f0 0x7f54ac0b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 -- 172.21.15.59:0/1031254050 >> 172.21.15.59:0/1031254050 conn(0x7f54ac0d3cd0 msgr2=0x7f54ac0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 -- 172.21.15.59:0/1031254050 shutdown_connections 2024-03-06T07:29:16.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:16.423+0000 7f54b450c640 1 -- 172.21.15.59:0/1031254050 wait complete. 2024-03-06T07:29:16.640 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:16.044432+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:16.640 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:17.641 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:17.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.742+0000 7fa03bb5c640 1 Processor -- start 2024-03-06T07:29:17.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.742+0000 7fa03bb5c640 1 -- start start 2024-03-06T07:29:17.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.743+0000 7fa03bb5c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa0340d22b0 0x7fa0340d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.743+0000 7fa03bb5c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa0340b41f0 0x7fa0340b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.743+0000 7fa03bb5c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0340b3430 con 0x7fa03415dfe0 2024-03-06T07:29:17.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.743+0000 7fa03bb5c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0340d2c50 con 0x7fa0340b41f0 2024-03-06T07:29:17.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.743+0000 7fa03bb5c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0340b31e0 con 0x7fa0340d22b0 2024-03-06T07:29:17.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.743+0000 7fa03a0f9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa0340b41f0 0x7fa0340b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.743+0000 7fa03a0f9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa0340b41f0 0x7fa0340b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53112/0 (socket says 172.21.15.59:53112) 2024-03-06T07:29:17.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.743+0000 7fa03a0f9640 1 -- 172.21.15.59:0/538799576 learned_addr learned my addr 172.21.15.59:0/538799576 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:17.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.743+0000 7fa0390f7640 1 --2- 172.21.15.59:0/538799576 >> v2:172.21.15.164:3300/0 conn(0x7fa0340d22b0 0x7fa0340d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.744+0000 7fa03a0f9640 1 -- 172.21.15.59:0/538799576 >> v2:172.21.15.164:3300/0 conn(0x7fa0340d22b0 msgr2=0x7fa0340d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.744+0000 7fa03a0f9640 1 --2- 172.21.15.59:0/538799576 >> v2:172.21.15.164:3300/0 conn(0x7fa0340d22b0 0x7fa0340d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.744+0000 7fa03a0f9640 1 -- 172.21.15.59:0/538799576 >> v1:172.21.15.164:6789/0 conn(0x7fa03415dfe0 legacy=0x7fa0340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.744+0000 7fa03a0f9640 1 -- 172.21.15.59:0/538799576 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0340b3710 con 0x7fa0340b41f0 2024-03-06T07:29:17.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.744+0000 7fa03a0f9640 1 --2- 172.21.15.59:0/538799576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0340b41f0 0x7fa0340b4680 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009e50 tx=0x7fa02c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e359f83ec59945 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.744+0000 7fa0388f6640 1 -- 172.21.15.59:0/538799576 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c02cb20 con 0x7fa0340b41f0 2024-03-06T07:29:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.745+0000 7fa0388f6640 1 -- 172.21.15.59:0/538799576 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02c02ccc0 con 0x7fa0340b41f0 2024-03-06T07:29:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.745+0000 7fa0388f6640 1 -- 172.21.15.59:0/538799576 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c034610 con 0x7fa0340b41f0 2024-03-06T07:29:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.745+0000 7fa03bb5c640 1 -- 172.21.15.59:0/538799576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0340b41f0 msgr2=0x7fa0340b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.745+0000 7fa03bb5c640 1 --2- 172.21.15.59:0/538799576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0340b41f0 0x7fa0340b4680 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009e50 tx=0x7fa02c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:29:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.745+0000 7fa03bb5c640 1 -- 172.21.15.59:0/538799576 shutdown_connections 2024-03-06T07:29:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.745+0000 7fa03bb5c640 1 --2- 172.21.15.59:0/538799576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0340b41f0 0x7fa0340b4680 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.745+0000 7fa03bb5c640 1 --2- 172.21.15.59:0/538799576 >> v2:172.21.15.164:3300/0 conn(0x7fa0340d22b0 0x7fa0340d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.745+0000 7fa03bb5c640 1 -- 172.21.15.59:0/538799576 >> 172.21.15.59:0/538799576 conn(0x7fa0340d3cd0 msgr2=0x7fa0340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.745+0000 7fa03bb5c640 1 -- 172.21.15.59:0/538799576 shutdown_connections 2024-03-06T07:29:17.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.745+0000 7fa03bb5c640 1 -- 172.21.15.59:0/538799576 wait complete. 2024-03-06T07:29:17.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.746+0000 7fa03bb5c640 1 Processor -- start 2024-03-06T07:29:17.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.746+0000 7fa03bb5c640 1 -- start start 2024-03-06T07:29:17.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa03bb5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0340b41f0 0x7fa03414f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa03bb5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0340d22b0 0x7fa0340b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa03bb5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa03415dfe0 0x7fa0340b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa03bb5c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0340cfa00 con 0x7fa0340b41f0 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa03bb5c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0340b3d60 con 0x7fa03415dfe0 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa03bb5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0340d1c70 con 0x7fa0340d22b0 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa0398f8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0340b41f0 0x7fa03414f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa0398f8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0340b41f0 0x7fa03414f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53120/0 (socket says 172.21.15.59:53120) 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa0398f8640 1 -- 172.21.15.59:0/1382019088 learned_addr learned my addr 172.21.15.59:0/1382019088 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa03a0f9640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa03415dfe0 0x7fa0340b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa0390f7640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0340d22b0 0x7fa0340b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa0398f8640 1 -- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0340d22b0 msgr2=0x7fa0340b82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa0398f8640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0340d22b0 0x7fa0340b82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa0398f8640 1 -- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa03415dfe0 msgr2=0x7fa0340b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa0398f8640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa03415dfe0 0x7fa0340b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.747+0000 7fa0398f8640 1 -- 172.21.15.59:0/1382019088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0341f73c0 con 0x7fa0340b41f0 2024-03-06T07:29:17.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.748+0000 7fa0398f8640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0340b41f0 0x7fa03414f090 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c430 tx=0x7fa03000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.748+0000 7fa02a7fc640 1 -- 172.21.15.59:0/1382019088 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030017070 con 0x7fa0340b41f0 2024-03-06T07:29:17.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.748+0000 7fa02a7fc640 1 -- 172.21.15.59:0/1382019088 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0300052d0 con 0x7fa0340b41f0 2024-03-06T07:29:17.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.748+0000 7fa03bb5c640 1 -- 172.21.15.59:0/1382019088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0341f76a0 con 0x7fa0340b41f0 2024-03-06T07:29:17.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.748+0000 7fa02a7fc640 1 -- 172.21.15.59:0/1382019088 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030003830 con 0x7fa0340b41f0 2024-03-06T07:29:17.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.748+0000 7fa03bb5c640 1 -- 172.21.15.59:0/1382019088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03414f5d0 con 0x7fa0340b41f0 2024-03-06T07:29:17.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.749+0000 7fa03bb5c640 1 -- 172.21.15.59:0/1382019088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa03404f330 con 0x7fa0340b41f0 2024-03-06T07:29:17.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.749+0000 7fa02a7fc640 1 -- 172.21.15.59:0/1382019088 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa03001a070 con 0x7fa0340b41f0 2024-03-06T07:29:17.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.750+0000 7fa02a7fc640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa01806b500 0x7fa01806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.750+0000 7fa02a7fc640 1 -- 172.21.15.59:0/1382019088 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa030089500 con 0x7fa0340b41f0 2024-03-06T07:29:17.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.750+0000 7fa0390f7640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa01806b500 0x7fa01806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.751+0000 7fa0390f7640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa01806b500 0x7fa01806d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa03415c2e0 tx=0x7fa024008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.754+0000 7fa02a7fc640 1 -- 172.21.15.59:0/1382019088 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa030058410 con 0x7fa0340b41f0 2024-03-06T07:29:17.904 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:17.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.972+0000 7fa03bb5c640 1 -- 172.21.15.59:0/1382019088 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0340b0af0 con 0x7fa0340b41f0 2024-03-06T07:29:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.972+0000 7fa02a7fc640 1 -- 172.21.15.59:0/1382019088 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fa030057d10 con 0x7fa0340b41f0 2024-03-06T07:29:17.974 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:17.974 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:16.044432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 87 2024-03-06T07:29:17.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 -- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa01806b500 msgr2=0x7fa01806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa01806b500 0x7fa01806d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa03415c2e0 tx=0x7fa024008040 comp rx=0 tx=0).stop 2024-03-06T07:29:17.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 -- 172.21.15.59:0/1382019088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0340b41f0 msgr2=0x7fa03414f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0340b41f0 0x7fa03414f090 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c430 tx=0x7fa03000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 -- 172.21.15.59:0/1382019088 shutdown_connections 2024-03-06T07:29:17.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa01806b500 0x7fa01806d9c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa03415dfe0 0x7fa0340b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0340d22b0 0x7fa0340b82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 --2- 172.21.15.59:0/1382019088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0340b41f0 0x7fa03414f090 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 -- 172.21.15.59:0/1382019088 >> 172.21.15.59:0/1382019088 conn(0x7fa0340d3cd0 msgr2=0x7fa03415d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 -- 172.21.15.59:0/1382019088 shutdown_connections 2024-03-06T07:29:17.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:17.975+0000 7fa03bb5c640 1 -- 172.21.15.59:0/1382019088 wait complete. 2024-03-06T07:29:18.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:16.044432+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:18.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:19.190 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.290+0000 7fecd4fb5640 1 Processor -- start 2024-03-06T07:29:19.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.290+0000 7fecd4fb5640 1 -- start start 2024-03-06T07:29:19.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.290+0000 7fecd4fb5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fecd00afdd0 0x7fecd015c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecd4fb5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fecd00d14f0 0x7fecd006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecd4fb5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd015e740 con 0x7fecd0158920 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecd4fb5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd006fae0 con 0x7fecd00afdd0 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecd4fb5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd015f5f0 con 0x7fecd00d14f0 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecc7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fecd00afdd0 0x7fecd015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecc7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fecd00afdd0 0x7fecd015c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53132/0 (socket says 172.21.15.59:53132) 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecc7fff640 1 -- 172.21.15.59:0/3012220501 learned_addr learned my addr 172.21.15.59:0/3012220501 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecced9d640 1 --2- 172.21.15.59:0/3012220501 >> v2:172.21.15.164:3300/0 conn(0x7fecd00d14f0 0x7fecd006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecc7fff640 1 -- 172.21.15.59:0/3012220501 >> v2:172.21.15.164:3300/0 conn(0x7fecd00d14f0 msgr2=0x7fecd006f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecc7fff640 1 --2- 172.21.15.59:0/3012220501 >> v2:172.21.15.164:3300/0 conn(0x7fecd00d14f0 0x7fecd006f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecc7fff640 1 -- 172.21.15.59:0/3012220501 >> v1:172.21.15.164:6789/0 conn(0x7fecd0158920 legacy=0x7fecd00af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.291+0000 7fecc7fff640 1 -- 172.21.15.59:0/3012220501 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd015e9e0 con 0x7fecd00afdd0 2024-03-06T07:29:19.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7fecc7fff640 1 --2- 172.21.15.59:0/3012220501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd00afdd0 0x7fecd015c310 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fecd00b6b10 tx=0x7fecb802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad474afc11fd5fc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7feccdd9b640 1 -- 172.21.15.59:0/3012220501 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8034070 con 0x7fecd00afdd0 2024-03-06T07:29:19.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7feccdd9b640 1 -- 172.21.15.59:0/3012220501 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb8030860 con 0x7fecd00afdd0 2024-03-06T07:29:19.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7feccdd9b640 1 -- 172.21.15.59:0/3012220501 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb8030c30 con 0x7fecd00afdd0 2024-03-06T07:29:19.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3012220501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd00afdd0 msgr2=0x7fecd015c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7fecd4fb5640 1 --2- 172.21.15.59:0/3012220501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd00afdd0 0x7fecd015c310 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fecd00b6b10 tx=0x7fecb802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3012220501 shutdown_connections 2024-03-06T07:29:19.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7fecd4fb5640 1 --2- 172.21.15.59:0/3012220501 >> v2:172.21.15.164:3300/0 conn(0x7fecd00d14f0 0x7fecd006f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7fecd4fb5640 1 --2- 172.21.15.59:0/3012220501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd00afdd0 0x7fecd015c310 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3012220501 >> 172.21.15.59:0/3012220501 conn(0x7fecd00d0960 msgr2=0x7fecd00d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3012220501 shutdown_connections 2024-03-06T07:29:19.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.292+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3012220501 wait complete. 2024-03-06T07:29:19.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.294+0000 7fecd4fb5640 1 Processor -- start 2024-03-06T07:29:19.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.294+0000 7fecd4fb5640 1 -- start start 2024-03-06T07:29:19.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.294+0000 7fecd4fb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecd00afdd0 0x7fecd0150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.294+0000 7fecd4fb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecd00d14f0 0x7fecd0157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecd4fb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd0158920 0x7fecd014b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecd4fb5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd015f280 con 0x7fecd0158920 2024-03-06T07:29:19.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecd4fb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd015efa0 con 0x7fecd00afdd0 2024-03-06T07:29:19.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecd4fb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd015b400 con 0x7fecd00d14f0 2024-03-06T07:29:19.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecced9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd0158920 0x7fecd014b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecced9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd0158920 0x7fecd014b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53142/0 (socket says 172.21.15.59:53142) 2024-03-06T07:29:19.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecced9d640 1 -- 172.21.15.59:0/3536814251 learned_addr learned my addr 172.21.15.59:0/3536814251 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:19.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecc7fff640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecd00d14f0 0x7fecd0157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecced9d640 1 -- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecd00d14f0 msgr2=0x7fecd0157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecced9d640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecd00d14f0 0x7fecd0157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecced9d640 1 -- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecd00afdd0 msgr2=0x7fecd0150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecce59c640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecd00afdd0 0x7fecd0150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecced9d640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecd00afdd0 0x7fecd0150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecced9d640 1 -- 172.21.15.59:0/3536814251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd01526c0 con 0x7fecd0158920 2024-03-06T07:29:19.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecc7fff640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecd00d14f0 0x7fecd0157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:19.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecced9d640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd0158920 0x7fecd014b3a0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fecc8007e70 tx=0x7fecc800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecc6ffd640 1 -- 172.21.15.59:0/3536814251 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc8014620 con 0x7fecd0158920 2024-03-06T07:29:19.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecc6ffd640 1 -- 172.21.15.59:0/3536814251 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc8019070 con 0x7fecd0158920 2024-03-06T07:29:19.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3536814251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd0152960 con 0x7fecd0158920 2024-03-06T07:29:19.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecc6ffd640 1 -- 172.21.15.59:0/3536814251 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc8014a20 con 0x7fecd0158920 2024-03-06T07:29:19.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.295+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3536814251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd014b8e0 con 0x7fecd0158920 2024-03-06T07:29:19.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.296+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3536814251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd004f330 con 0x7fecd0158920 2024-03-06T07:29:19.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.297+0000 7fecc6ffd640 1 -- 172.21.15.59:0/3536814251 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fecc8014bc0 con 0x7fecd0158920 2024-03-06T07:29:19.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.297+0000 7fecc6ffd640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecb006b4d0 0x7fecb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.297+0000 7fecc6ffd640 1 -- 172.21.15.59:0/3536814251 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fecc808aa70 con 0x7fecd0158920 2024-03-06T07:29:19.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.297+0000 7fecce59c640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecb006b4d0 0x7fecb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.298+0000 7fecce59c640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecb006b4d0 0x7fecb006d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fecc0001500 tx=0x7fecc0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.302+0000 7fecc6ffd640 1 -- 172.21.15.59:0/3536814251 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fecc8059980 con 0x7fecd0158920 2024-03-06T07:29:19.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.519+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3536814251 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecd0072b80 con 0x7fecd0158920 2024-03-06T07:29:19.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.520+0000 7fecc6ffd640 1 -- 172.21.15.59:0/3536814251 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fecc8059280 con 0x7fecd0158920 2024-03-06T07:29:19.523 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:19.524 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:16.044432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 87 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecb006b4d0 msgr2=0x7fecb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecb006b4d0 0x7fecb006d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fecc0001500 tx=0x7fecc0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3536814251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd0158920 msgr2=0x7fecd014b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd0158920 0x7fecd014b3a0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fecc8007e70 tx=0x7fecc800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3536814251 shutdown_connections 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fecb006b4d0 0x7fecb006d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecd0158920 0x7fecd014b3a0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecd00d14f0 0x7fecd0157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 --2- 172.21.15.59:0/3536814251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecd00afdd0 0x7fecd0150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3536814251 >> 172.21.15.59:0/3536814251 conn(0x7fecd00d0960 msgr2=0x7fecd0159ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3536814251 shutdown_connections 2024-03-06T07:29:19.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:19.522+0000 7fecd4fb5640 1 -- 172.21.15.59:0/3536814251 wait complete. 2024-03-06T07:29:19.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:16.044432+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:19.737 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:19.906 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:20.600 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:20.603 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:20.738 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:20.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.870+0000 7fc3617e9640 1 Processor -- start 2024-03-06T07:29:20.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.870+0000 7fc3617e9640 1 -- start start 2024-03-06T07:29:20.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.871+0000 7fc3617e9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc35c0d00c0 0x7fc35c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.871+0000 7fc3617e9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c0d09e0 con 0x7fc35c06cf20 2024-03-06T07:29:20.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.871+0000 7fc3617e9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c15e380 con 0x7fc35c0cf850 2024-03-06T07:29:20.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.871+0000 7fc3617e9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35c06ca90 con 0x7fc35c0d00c0 2024-03-06T07:29:20.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.871+0000 7fc35b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc35c06cf20 0x7fc35c06d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44968/0 (socket says 172.21.15.59:44968) 2024-03-06T07:29:20.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.871+0000 7fc35b7fe640 1 -- 172.21.15.59:0/511560140 learned_addr learned my addr 172.21.15.59:0/511560140 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:20.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc35affd640 1 --2- 172.21.15.59:0/511560140 >> v2:172.21.15.164:3300/0 conn(0x7fc35c0d00c0 0x7fc35c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723415372 0 0) 0x7fc35c0d09e0 con 0x7fc35c06cf20 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc344003520 con 0x7fc35c06cf20 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2235445082 0 0) 0x7fc344003520 con 0x7fc35c06cf20 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 >> v2:172.21.15.164:3300/0 conn(0x7fc35c0d00c0 msgr2=0x7fc35c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 --2- 172.21.15.59:0/511560140 >> v2:172.21.15.164:3300/0 conn(0x7fc35c0d00c0 0x7fc35c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 >> v1:172.21.15.164:6789/0 conn(0x7fc35c0cf850 legacy=0x7fc35c0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c0cdc10 con 0x7fc35c06cf20 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fc3540019f0 con 0x7fc35c06cf20 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 >> v1:172.21.15.59:6789/0 conn(0x7fc35c06cf20 legacy=0x7fc35c06d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 --2- 172.21.15.59:0/511560140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3440039b0 0x7fc344003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 --2- 172.21.15.59:0/511560140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc344004800 0x7fc344004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 --2- 172.21.15.59:0/511560140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3440051c0 0x7fc34400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c06ca90 con 0x7fc3440051c0 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34400c060 con 0x7fc3440039b0 2024-03-06T07:29:20.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34400c260 con 0x7fc344004800 2024-03-06T07:29:20.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc35b7fe640 1 --2- 172.21.15.59:0/511560140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3440051c0 0x7fc34400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc35affd640 1 --2- 172.21.15.59:0/511560140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3440039b0 0x7fc344003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc35a7fc640 1 --2- 172.21.15.59:0/511560140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc344004800 0x7fc344004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc35b7fe640 1 -- 172.21.15.59:0/511560140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc344004800 msgr2=0x7fc344004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc35b7fe640 1 --2- 172.21.15.59:0/511560140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc344004800 0x7fc344004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc35b7fe640 1 -- 172.21.15.59:0/511560140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3440039b0 msgr2=0x7fc344003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc35b7fe640 1 --2- 172.21.15.59:0/511560140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3440039b0 0x7fc344003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.872+0000 7fc35b7fe640 1 -- 172.21.15.59:0/511560140 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc344003520 con 0x7fc3440051c0 2024-03-06T07:29:20.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.873+0000 7fc35b7fe640 1 -- 172.21.15.59:0/511560140 reap_dead start 2024-03-06T07:29:20.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.874+0000 7fc35b7fe640 1 --2- 172.21.15.59:0/511560140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3440051c0 0x7fc34400b9c0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fc35400a850 tx=0x7fc344005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2ef143d509e605 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.874+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35400a4a0 con 0x7fc3440051c0 2024-03-06T07:29:20.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.874+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc354034650 con 0x7fc3440051c0 2024-03-06T07:29:20.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.874+0000 7fc359ffb640 1 -- 172.21.15.59:0/511560140 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354034a40 con 0x7fc3440051c0 2024-03-06T07:29:20.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.874+0000 7fc3617e9640 1 -- 172.21.15.59:0/511560140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3440051c0 msgr2=0x7fc34400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.874+0000 7fc3617e9640 1 --2- 172.21.15.59:0/511560140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3440051c0 0x7fc34400b9c0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fc35400a850 tx=0x7fc344005830 comp rx=0 tx=0).stop 2024-03-06T07:29:20.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.874+0000 7fc3617e9640 1 -- 172.21.15.59:0/511560140 shutdown_connections 2024-03-06T07:29:20.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.874+0000 7fc3617e9640 1 --2- 172.21.15.59:0/511560140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3440051c0 0x7fc34400b9c0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.874+0000 7fc3617e9640 1 -- 172.21.15.59:0/511560140 >> 172.21.15.59:0/511560140 conn(0x7fc35c0b35a0 msgr2=0x7fc35c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.874+0000 7fc3617e9640 1 -- 172.21.15.59:0/511560140 shutdown_connections 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.874+0000 7fc3617e9640 1 -- 172.21.15.59:0/511560140 wait complete. 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.875+0000 7fc3617e9640 1 Processor -- start 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc3617e9640 1 -- start start 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc3617e9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3440051c0 0x7fc35c0c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc3617e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc35c0c7b50 0x7fc35c1f76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc3617e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc35c1525a0 0x7fc35c152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc3617e9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c06c720 con 0x7fc35c0c7b50 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc3617e9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c06c440 con 0x7fc3440051c0 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc3617e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0b1030 con 0x7fc35c1525a0 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc35a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc35c0c7b50 0x7fc35c1f76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc35a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc35c0c7b50 0x7fc35c1f76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53156/0 (socket says 172.21.15.59:53156) 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc35a7fc640 1 -- 172.21.15.59:0/3848242893 learned_addr learned my addr 172.21.15.59:0/3848242893 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc35affd640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3440051c0 0x7fc35c0c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.876+0000 7fc35b7fe640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc35c1525a0 0x7fc35c152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc35a7fc640 1 -- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc35c1525a0 msgr2=0x7fc35c152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc35a7fc640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc35c1525a0 0x7fc35c152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc35a7fc640 1 -- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3440051c0 msgr2=0x7fc35c0c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc35a7fc640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3440051c0 0x7fc35c0c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc35a7fc640 1 -- 172.21.15.59:0/3848242893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c0c80a0 con 0x7fc35c0c7b50 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc35b7fe640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc35c1525a0 0x7fc35c152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc35affd640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3440051c0 0x7fc35c0c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc35a7fc640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc35c0c7b50 0x7fc35c1f76f0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fc35000e4b0 tx=0x7fc35000e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc3437fe640 1 -- 172.21.15.59:0/3848242893 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3500052e0 con 0x7fc35c0c7b50 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc3437fe640 1 -- 172.21.15.59:0/3848242893 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc350018070 con 0x7fc35c0c7b50 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc3617e9640 1 -- 172.21.15.59:0/3848242893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c0c8380 con 0x7fc35c0c7b50 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc3437fe640 1 -- 172.21.15.59:0/3848242893 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350003830 con 0x7fc35c0c7b50 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.877+0000 7fc3617e9640 1 -- 172.21.15.59:0/3848242893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c0c9290 con 0x7fc35c0c7b50 2024-03-06T07:29:20.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.878+0000 7fc3617e9640 1 -- 172.21.15.59:0/3848242893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35c0c88a0 con 0x7fc35c0c7b50 2024-03-06T07:29:20.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.879+0000 7fc3437fe640 1 -- 172.21.15.59:0/3848242893 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3500039d0 con 0x7fc35c0c7b50 2024-03-06T07:29:20.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.879+0000 7fc3437fe640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc34806b500 0x7fc34806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.879+0000 7fc3437fe640 1 -- 172.21.15.59:0/3848242893 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc350089df0 con 0x7fc35c0c7b50 2024-03-06T07:29:20.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.879+0000 7fc35affd640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc34806b500 0x7fc34806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.880+0000 7fc35affd640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc34806b500 0x7fc34806d9c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc34c009b40 tx=0x7fc34c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:20.883+0000 7fc3437fe640 1 -- 172.21.15.59:0/3848242893 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc35c0c88a0 con 0x7fc35c0c7b50 2024-03-06T07:29:21.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.101+0000 7fc3617e9640 1 -- 172.21.15.59:0/3848242893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc35c04f330 con 0x7fc35c0c7b50 2024-03-06T07:29:21.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.101+0000 7fc3437fe640 1 -- 172.21.15.59:0/3848242893 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fc350058c70 con 0x7fc35c0c7b50 2024-03-06T07:29:21.103 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:21.103 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:16.044432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:21.104 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 87 2024-03-06T07:29:21.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 -- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc34806b500 msgr2=0x7fc34806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc34806b500 0x7fc34806d9c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc34c009b40 tx=0x7fc34c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:21.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 -- 172.21.15.59:0/3848242893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc35c0c7b50 msgr2=0x7fc35c1f76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc35c0c7b50 0x7fc35c1f76f0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fc35000e4b0 tx=0x7fc35000e970 comp rx=0 tx=0).stop 2024-03-06T07:29:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 -- 172.21.15.59:0/3848242893 shutdown_connections 2024-03-06T07:29:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc34806b500 0x7fc34806d9c0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc35c1525a0 0x7fc35c152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc35c0c7b50 0x7fc35c1f76f0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 --2- 172.21.15.59:0/3848242893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3440051c0 0x7fc35c0c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 -- 172.21.15.59:0/3848242893 >> 172.21.15.59:0/3848242893 conn(0x7fc35c0b35a0 msgr2=0x7fc35c0d0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 -- 172.21.15.59:0/3848242893 shutdown_connections 2024-03-06T07:29:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:21.103+0000 7fc3617e9640 1 -- 172.21.15.59:0/3848242893 wait complete. 2024-03-06T07:29:21.317 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:16.044432+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:21.318 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:21.908 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:22.318 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:22.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.419+0000 7fa0c9847640 1 Processor -- start 2024-03-06T07:29:22.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.419+0000 7fa0c9847640 1 -- start start 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.420+0000 7fa0c9847640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa0c40d00c0 0x7fa0c40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.420+0000 7fa0c9847640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa0c40cf850 0x7fa0c40cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.420+0000 7fa0c9847640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c40d09e0 con 0x7fa0c40ce1b0 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.420+0000 7fa0c9847640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c406ca90 con 0x7fa0c40d00c0 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.420+0000 7fa0c9847640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c406df30 con 0x7fa0c40cf850 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.420+0000 7fa0c2ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa0c40d00c0 0x7fa0c40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.420+0000 7fa0c2ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa0c40d00c0 0x7fa0c40d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53160/0 (socket says 172.21.15.59:53160) 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.420+0000 7fa0c2ffd640 1 -- 172.21.15.59:0/459408024 learned_addr learned my addr 172.21.15.59:0/459408024 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.421+0000 7fa0ba7fc640 1 --2- 172.21.15.59:0/459408024 >> v2:172.21.15.164:3300/0 conn(0x7fa0c40cf850 0x7fa0c40cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.421+0000 7fa0c2ffd640 1 -- 172.21.15.59:0/459408024 >> v2:172.21.15.164:3300/0 conn(0x7fa0c40cf850 msgr2=0x7fa0c40cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.421+0000 7fa0c2ffd640 1 --2- 172.21.15.59:0/459408024 >> v2:172.21.15.164:3300/0 conn(0x7fa0c40cf850 0x7fa0c40cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.421+0000 7fa0c2ffd640 1 -- 172.21.15.59:0/459408024 >> v1:172.21.15.164:6789/0 conn(0x7fa0c40ce1b0 legacy=0x7fa0c40ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.421+0000 7fa0c2ffd640 1 -- 172.21.15.59:0/459408024 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c415e380 con 0x7fa0c40d00c0 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.421+0000 7fa0c2ffd640 1 --2- 172.21.15.59:0/459408024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c40d00c0 0x7fa0c40d04a0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4071de0 tx=0x7fa0b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a170e4c46fb51f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.422+0000 7fa0c27fc640 1 -- 172.21.15.59:0/459408024 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4035020 con 0x7fa0c40d00c0 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.422+0000 7fa0c27fc640 1 -- 172.21.15.59:0/459408024 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b4031870 con 0x7fa0c40d00c0 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.422+0000 7fa0c27fc640 1 -- 172.21.15.59:0/459408024 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4031c40 con 0x7fa0c40d00c0 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.422+0000 7fa0c9847640 1 -- 172.21.15.59:0/459408024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c40d00c0 msgr2=0x7fa0c40d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.422+0000 7fa0c9847640 1 --2- 172.21.15.59:0/459408024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c40d00c0 0x7fa0c40d04a0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4071de0 tx=0x7fa0b402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.422+0000 7fa0c9847640 1 -- 172.21.15.59:0/459408024 shutdown_connections 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.422+0000 7fa0c9847640 1 --2- 172.21.15.59:0/459408024 >> v2:172.21.15.164:3300/0 conn(0x7fa0c40cf850 0x7fa0c40cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.422+0000 7fa0c9847640 1 --2- 172.21.15.59:0/459408024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c40d00c0 0x7fa0c40d04a0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.422+0000 7fa0c9847640 1 -- 172.21.15.59:0/459408024 >> 172.21.15.59:0/459408024 conn(0x7fa0c40b35a0 msgr2=0x7fa0c40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.422+0000 7fa0c9847640 1 -- 172.21.15.59:0/459408024 shutdown_connections 2024-03-06T07:29:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.422+0000 7fa0c9847640 1 -- 172.21.15.59:0/459408024 wait complete. 2024-03-06T07:29:22.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.424+0000 7fa0c9847640 1 Processor -- start 2024-03-06T07:29:22.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.424+0000 7fa0c9847640 1 -- start start 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.425+0000 7fa0c9847640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0c40ce1b0 0x7fa0c4155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c9847640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c40cf850 0x7fa0c40affd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c9847640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c40d00c0 0x7fa0c40b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c9847640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c406c440 con 0x7fa0c40d00c0 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c9847640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c406c160 con 0x7fa0c40cf850 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c9847640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c406c720 con 0x7fa0c40ce1b0 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c40d00c0 0x7fa0c40b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c40d00c0 0x7fa0c40b0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:53170/0 (socket says 172.21.15.59:53170) 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c37fe640 1 -- 172.21.15.59:0/3135050447 learned_addr learned my addr 172.21.15.59:0/3135050447 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0ba7fc640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c40cf850 0x7fa0c40affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c2ffd640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0c40ce1b0 0x7fa0c4155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c37fe640 1 -- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0c40ce1b0 msgr2=0x7fa0c4155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c37fe640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0c40ce1b0 0x7fa0c4155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c37fe640 1 -- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c40cf850 msgr2=0x7fa0c40affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c37fe640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c40cf850 0x7fa0c40affd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c37fe640 1 -- 172.21.15.59:0/3135050447 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c40c56b0 con 0x7fa0c40d00c0 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c37fe640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c40d00c0 0x7fa0c40b0510 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00c780 tx=0x7fa0bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c0ff9640 1 -- 172.21.15.59:0/3135050447 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc004b60 con 0x7fa0c40d00c0 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c0ff9640 1 -- 172.21.15.59:0/3135050447 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0bc004d00 con 0x7fa0c40d00c0 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c9847640 1 -- 172.21.15.59:0/3135050447 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c40c5990 con 0x7fa0c40d00c0 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c0ff9640 1 -- 172.21.15.59:0/3135050447 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc00aa90 con 0x7fa0c40d00c0 2024-03-06T07:29:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.426+0000 7fa0c9847640 1 -- 172.21.15.59:0/3135050447 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c40c5f50 con 0x7fa0c40d00c0 2024-03-06T07:29:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.427+0000 7fa0c9847640 1 -- 172.21.15.59:0/3135050447 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c406a090 con 0x7fa0c40d00c0 2024-03-06T07:29:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.428+0000 7fa0c0ff9640 1 -- 172.21.15.59:0/3135050447 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0bc00ac30 con 0x7fa0c40d00c0 2024-03-06T07:29:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.428+0000 7fa0c0ff9640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0a406b4d0 0x7fa0a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.428+0000 7fa0c0ff9640 1 -- 172.21.15.59:0/3135050447 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0bc08a440 con 0x7fa0c40d00c0 2024-03-06T07:29:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.429+0000 7fa0c2ffd640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0a406b4d0 0x7fa0a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.429+0000 7fa0c2ffd640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0a406b4d0 0x7fa0a406d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4036000 tx=0x7fa0b403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.433+0000 7fa0c0ff9640 1 -- 172.21.15.59:0/3135050447 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0c406a090 con 0x7fa0c40d00c0 2024-03-06T07:29:22.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.653+0000 7fa0c9847640 1 -- 172.21.15.59:0/3135050447 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0c404f330 con 0x7fa0c40d00c0 2024-03-06T07:29:22.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.654+0000 7fa0c0ff9640 1 -- 172.21.15.59:0/3135050447 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fa0bc020070 con 0x7fa0c40d00c0 2024-03-06T07:29:22.656 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:22.656 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:16.044432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:22.656 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 87 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 -- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0a406b4d0 msgr2=0x7fa0a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0a406b4d0 0x7fa0a406d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4036000 tx=0x7fa0b403c040 comp rx=0 tx=0).stop 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 -- 172.21.15.59:0/3135050447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c40d00c0 msgr2=0x7fa0c40b0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c40d00c0 0x7fa0c40b0510 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00c780 tx=0x7fa0bc00cc40 comp rx=0 tx=0).stop 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 -- 172.21.15.59:0/3135050447 shutdown_connections 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa0a406b4d0 0x7fa0a406d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0c40d00c0 0x7fa0c40b0510 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0c40cf850 0x7fa0c40affd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 --2- 172.21.15.59:0/3135050447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0c40ce1b0 0x7fa0c4155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 -- 172.21.15.59:0/3135050447 >> 172.21.15.59:0/3135050447 conn(0x7fa0c40b35a0 msgr2=0x7fa0c4070070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 -- 172.21.15.59:0/3135050447 shutdown_connections 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:22.656+0000 7fa0c9847640 1 -- 172.21.15.59:0/3135050447 wait complete. 2024-03-06T07:29:22.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:16.044432+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:22.870 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:23.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:23.910 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:24.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.008+0000 7f32dd01c640 1 Processor -- start 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.008+0000 7f32dd01c640 1 -- start start 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.009+0000 7f32dd01c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f32d80d02c0 0x7f32d80cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.009+0000 7f32dd01c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f32d80d06a0 0x7f32d80cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.009+0000 7f32dd01c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d815c620 con 0x7f32d80ce130 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.009+0000 7f32dd01c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d80b50b0 con 0x7f32d80d06a0 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.009+0000 7f32dd01c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d80cd800 con 0x7f32d80d02c0 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.009+0000 7f32cffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f32d80d06a0 0x7f32d80cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.009+0000 7f32d759e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f32d80ce130 0x7f32d80ce540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:56002/0 (socket says 172.21.15.59:56002) 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.009+0000 7f32d759e640 1 -- 172.21.15.59:0/847175522 learned_addr learned my addr 172.21.15.59:0/847175522 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.010+0000 7f32d6d9d640 1 --2- 172.21.15.59:0/847175522 >> v2:172.21.15.164:3300/0 conn(0x7f32d80d02c0 0x7f32d80cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.010+0000 7f32cffff640 1 -- 172.21.15.59:0/847175522 >> v2:172.21.15.164:3300/0 conn(0x7f32d80d02c0 msgr2=0x7f32d80cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.010+0000 7f32cffff640 1 --2- 172.21.15.59:0/847175522 >> v2:172.21.15.164:3300/0 conn(0x7f32d80d02c0 0x7f32d80cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.010+0000 7f32cffff640 1 -- 172.21.15.59:0/847175522 >> v1:172.21.15.59:6789/0 conn(0x7f32d80ce130 legacy=0x7f32d80ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.010+0000 7f32cffff640 1 -- 172.21.15.59:0/847175522 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d80b4a60 con 0x7f32d80d06a0 2024-03-06T07:29:24.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.010+0000 7f32cffff640 1 --2- 172.21.15.59:0/847175522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32d80d06a0 0x7f32d80cd0d0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f32d8071cf0 tx=0x7f32c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5043ffa7ffb7b5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.011+0000 7f32d659c640 1 -- 172.21.15.59:0/847175522 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c0035020 con 0x7f32d80d06a0 2024-03-06T07:29:24.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.011+0000 7f32d659c640 1 -- 172.21.15.59:0/847175522 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c0031830 con 0x7f32d80d06a0 2024-03-06T07:29:24.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.011+0000 7f32d659c640 1 -- 172.21.15.59:0/847175522 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c0031c00 con 0x7f32d80d06a0 2024-03-06T07:29:24.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.011+0000 7f32dd01c640 1 -- 172.21.15.59:0/847175522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32d80d06a0 msgr2=0x7f32d80cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.011+0000 7f32dd01c640 1 --2- 172.21.15.59:0/847175522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32d80d06a0 0x7f32d80cd0d0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f32d8071cf0 tx=0x7f32c002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.011+0000 7f32dd01c640 1 -- 172.21.15.59:0/847175522 shutdown_connections 2024-03-06T07:29:24.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.011+0000 7f32dd01c640 1 --2- 172.21.15.59:0/847175522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32d80d06a0 0x7f32d80cd0d0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.011+0000 7f32dd01c640 1 --2- 172.21.15.59:0/847175522 >> v2:172.21.15.164:3300/0 conn(0x7f32d80d02c0 0x7f32d80cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.011+0000 7f32dd01c640 1 -- 172.21.15.59:0/847175522 >> 172.21.15.59:0/847175522 conn(0x7f32d80b35a0 msgr2=0x7f32d80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.011+0000 7f32dd01c640 1 -- 172.21.15.59:0/847175522 shutdown_connections 2024-03-06T07:29:24.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.011+0000 7f32dd01c640 1 -- 172.21.15.59:0/847175522 wait complete. 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.012+0000 7f32dd01c640 1 Processor -- start 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.013+0000 7f32dd01c640 1 -- start start 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.013+0000 7f32dd01c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32d80ce130 0x7f32d81e5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.013+0000 7f32dd01c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32d80d02c0 0x7f32d80cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.013+0000 7f32dd01c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32d80d06a0 0x7f32d8146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.013+0000 7f32dd01c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80b66d0 con 0x7f32d80d06a0 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.013+0000 7f32dd01c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80b6550 con 0x7f32d80ce130 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.013+0000 7f32dd01c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d80b6850 con 0x7f32d80d02c0 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.013+0000 7f32d759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32d80d06a0 0x7f32d8146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.013+0000 7f32d759e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32d80d06a0 0x7f32d8146d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58606/0 (socket says 172.21.15.59:58606) 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.013+0000 7f32d759e640 1 -- 172.21.15.59:0/3292475346 learned_addr learned my addr 172.21.15.59:0/3292475346 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.014+0000 7f32d759e640 1 -- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32d80d02c0 msgr2=0x7f32d80cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.014+0000 7f32d759e640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32d80d02c0 0x7f32d80cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.014+0000 7f32d759e640 1 -- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32d80ce130 msgr2=0x7f32d81e5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.014+0000 7f32d759e640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32d80ce130 0x7f32d81e5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.014+0000 7f32d759e640 1 -- 172.21.15.59:0/3292475346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d81472a0 con 0x7f32d80d06a0 2024-03-06T07:29:24.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.014+0000 7f32d759e640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32d80d06a0 0x7f32d8146d60 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f32d000ca80 tx=0x7f32d000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.014+0000 7f32cf7fe640 1 -- 172.21.15.59:0/3292475346 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d0017070 con 0x7f32d80d06a0 2024-03-06T07:29:24.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.014+0000 7f32cf7fe640 1 -- 172.21.15.59:0/3292475346 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d000ac30 con 0x7f32d80d06a0 2024-03-06T07:29:24.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.014+0000 7f32dd01c640 1 -- 172.21.15.59:0/3292475346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d81e8160 con 0x7f32d80d06a0 2024-03-06T07:29:24.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.014+0000 7f32cf7fe640 1 -- 172.21.15.59:0/3292475346 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d0012600 con 0x7f32d80d06a0 2024-03-06T07:29:24.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.014+0000 7f32dd01c640 1 -- 172.21.15.59:0/3292475346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d81e8720 con 0x7f32d80d06a0 2024-03-06T07:29:24.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.015+0000 7f32dd01c640 1 -- 172.21.15.59:0/3292475346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d804f330 con 0x7f32d80d06a0 2024-03-06T07:29:24.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.015+0000 7f32cf7fe640 1 -- 172.21.15.59:0/3292475346 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32d001a070 con 0x7f32d80d06a0 2024-03-06T07:29:24.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.016+0000 7f32cf7fe640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32b806b4d0 0x7f32b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.016+0000 7f32cf7fe640 1 -- 172.21.15.59:0/3292475346 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32d0089dc0 con 0x7f32d80d06a0 2024-03-06T07:29:24.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.016+0000 7f32d6d9d640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32b806b4d0 0x7f32b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.017+0000 7f32d6d9d640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32b806b4d0 0x7f32b806d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f32c8006fd0 tx=0x7f32c8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.020+0000 7f32cf7fe640 1 -- 172.21.15.59:0/3292475346 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32d000ae20 con 0x7f32d80d06a0 2024-03-06T07:29:24.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.237+0000 7f32dd01c640 1 -- 172.21.15.59:0/3292475346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32d80cf3d0 con 0x7f32d80d06a0 2024-03-06T07:29:24.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.238+0000 7f32cf7fe640 1 -- 172.21.15.59:0/3292475346 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f32d00585f0 con 0x7f32d80d06a0 2024-03-06T07:29:24.240 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:24.240 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.239+0000 7f32dd01c640 1 -- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32b806b4d0 msgr2=0x7f32b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.239+0000 7f32dd01c640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32b806b4d0 0x7f32b806d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f32c8006fd0 tx=0x7f32c8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.239+0000 7f32dd01c640 1 -- 172.21.15.59:0/3292475346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32d80d06a0 msgr2=0x7f32d8146d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.239+0000 7f32dd01c640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32d80d06a0 0x7f32d8146d60 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f32d000ca80 tx=0x7f32d000cf40 comp rx=0 tx=0).stop 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.239+0000 7f32dd01c640 1 -- 172.21.15.59:0/3292475346 shutdown_connections 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.239+0000 7f32dd01c640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f32b806b4d0 0x7f32b806d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.239+0000 7f32dd01c640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32d80d06a0 0x7f32d8146d60 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.239+0000 7f32dd01c640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32d80d02c0 0x7f32d80cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.239+0000 7f32dd01c640 1 --2- 172.21.15.59:0/3292475346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32d80ce130 0x7f32d81e5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.239+0000 7f32dd01c640 1 -- 172.21.15.59:0/3292475346 >> 172.21.15.59:0/3292475346 conn(0x7f32d80b35a0 msgr2=0x7f32d806f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.239+0000 7f32dd01c640 1 -- 172.21.15.59:0/3292475346 shutdown_connections 2024-03-06T07:29:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:24.240+0000 7f32dd01c640 1 -- 172.21.15.59:0/3292475346 wait complete. 2024-03-06T07:29:24.454 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:24.454 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:25.455 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:25.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.557+0000 7f6c90e2a640 1 Processor -- start 2024-03-06T07:29:25.559 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.557+0000 7f6c90e2a640 1 -- start start 2024-03-06T07:29:25.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c90e2a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6c8c13dc10 0x7f6c8c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c90e2a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6c8c15bf30 0x7f6c8c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c90e2a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8c15f580 con 0x7f6c8c15e850 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c90e2a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c8c0cf3f0 con 0x7f6c8c15bf30 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c90e2a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c8c15b210 con 0x7f6c8c13dc10 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c89d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6c8c15bf30 0x7f6c8c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c89d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6c8c15bf30 0x7f6c8c0b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58618/0 (socket says 172.21.15.59:58618) 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c89d9b640 1 -- 172.21.15.59:0/1977107248 learned_addr learned my addr 172.21.15.59:0/1977107248 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c8a59c640 1 --2- 172.21.15.59:0/1977107248 >> v2:172.21.15.164:3300/0 conn(0x7f6c8c13dc10 0x7f6c8c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c89d9b640 1 -- 172.21.15.59:0/1977107248 >> v2:172.21.15.164:3300/0 conn(0x7f6c8c13dc10 msgr2=0x7f6c8c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c89d9b640 1 --2- 172.21.15.59:0/1977107248 >> v2:172.21.15.164:3300/0 conn(0x7f6c8c13dc10 0x7f6c8c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c89d9b640 1 -- 172.21.15.59:0/1977107248 >> v1:172.21.15.164:6789/0 conn(0x7f6c8c15e850 legacy=0x7f6c8c15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:25.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.559+0000 7f6c89d9b640 1 -- 172.21.15.59:0/1977107248 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c15b450 con 0x7f6c8c15bf30 2024-03-06T07:29:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.560+0000 7f6c89d9b640 1 --2- 172.21.15.59:0/1977107248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c0b2e90 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0731a0 tx=0x7f6c7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791e4a7c02e0c053 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.560+0000 7f6c8959a640 1 -- 172.21.15.59:0/1977107248 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7402ca80 con 0x7f6c8c15bf30 2024-03-06T07:29:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.560+0000 7f6c8959a640 1 -- 172.21.15.59:0/1977107248 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7402cc20 con 0x7f6c8c15bf30 2024-03-06T07:29:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.560+0000 7f6c8959a640 1 -- 172.21.15.59:0/1977107248 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c740343f0 con 0x7f6c8c15bf30 2024-03-06T07:29:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.560+0000 7f6c90e2a640 1 -- 172.21.15.59:0/1977107248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c8c15bf30 msgr2=0x7f6c8c0b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.560+0000 7f6c90e2a640 1 --2- 172.21.15.59:0/1977107248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c0b2e90 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0731a0 tx=0x7f6c7402bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.561+0000 7f6c90e2a640 1 -- 172.21.15.59:0/1977107248 shutdown_connections 2024-03-06T07:29:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.561+0000 7f6c90e2a640 1 --2- 172.21.15.59:0/1977107248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c0b2e90 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.561+0000 7f6c90e2a640 1 --2- 172.21.15.59:0/1977107248 >> v2:172.21.15.164:3300/0 conn(0x7f6c8c13dc10 0x7f6c8c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.561+0000 7f6c90e2a640 1 -- 172.21.15.59:0/1977107248 >> 172.21.15.59:0/1977107248 conn(0x7f6c8c0cf8a0 msgr2=0x7f6c8c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.561+0000 7f6c90e2a640 1 -- 172.21.15.59:0/1977107248 shutdown_connections 2024-03-06T07:29:25.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.561+0000 7f6c90e2a640 1 -- 172.21.15.59:0/1977107248 wait complete. 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.562+0000 7f6c90e2a640 1 Processor -- start 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.562+0000 7f6c90e2a640 1 -- start start 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c90e2a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c8c13dc10 0x7f6c8c159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c90e2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c90e2a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c8c15e850 0x7f6c8c06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c90e2a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0708d0 con 0x7f6c8c15bf30 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c90e2a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c13dff0 con 0x7f6c8c13dc10 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c90e2a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c070a50 con 0x7f6c8c15e850 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c89d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c89d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c06de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58634/0 (socket says 172.21.15.59:58634) 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c89d9b640 1 -- 172.21.15.59:0/3924562706 learned_addr learned my addr 172.21.15.59:0/3924562706 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:25.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c8ad9d640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c8c15e850 0x7f6c8c06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c89d9b640 1 -- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c8c15e850 msgr2=0x7f6c8c06e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c8a59c640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c8c13dc10 0x7f6c8c159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c89d9b640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c8c15e850 0x7f6c8c06e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c89d9b640 1 -- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c8c13dc10 msgr2=0x7f6c8c159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c89d9b640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c8c13dc10 0x7f6c8c159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c89d9b640 1 -- 172.21.15.59:0/3924562706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c13f410 con 0x7f6c8c15bf30 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c89d9b640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c06de80 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009f40 tx=0x7f6c74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c82ffd640 1 -- 172.21.15.59:0/3924562706 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74043020 con 0x7f6c8c15bf30 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.563+0000 7f6c82ffd640 1 -- 172.21.15.59:0/3924562706 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7403e450 con 0x7f6c8c15bf30 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.564+0000 7f6c90e2a640 1 -- 172.21.15.59:0/3924562706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c13f6f0 con 0x7f6c8c15bf30 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.564+0000 7f6c82ffd640 1 -- 172.21.15.59:0/3924562706 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74009510 con 0x7f6c8c15bf30 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.564+0000 7f6c90e2a640 1 -- 172.21.15.59:0/3924562706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c13fcb0 con 0x7f6c8c15bf30 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.565+0000 7f6c90e2a640 1 -- 172.21.15.59:0/3924562706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c0682b0 con 0x7f6c8c15bf30 2024-03-06T07:29:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.565+0000 7f6c82ffd640 1 -- 172.21.15.59:0/3924562706 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c74046020 con 0x7f6c8c15bf30 2024-03-06T07:29:25.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.566+0000 7f6c82ffd640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6c6c06b4d0 0x7f6c6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.566+0000 7f6c82ffd640 1 -- 172.21.15.59:0/3924562706 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c740ac9d0 con 0x7f6c8c15bf30 2024-03-06T07:29:25.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.566+0000 7f6c8a59c640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6c6c06b4d0 0x7f6c6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.567+0000 7f6c8a59c640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6c6c06b4d0 0x7f6c6c06d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6c84006fd0 tx=0x7f6c84008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.570+0000 7f6c82ffd640 1 -- 172.21.15.59:0/3924562706 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c8c0682b0 con 0x7f6c8c15bf30 2024-03-06T07:29:25.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.790+0000 7f6c90e2a640 1 -- 172.21.15.59:0/3924562706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c8c04f330 con 0x7f6c8c15bf30 2024-03-06T07:29:25.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.791+0000 7f6c82ffd640 1 -- 172.21.15.59:0/3924562706 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f6c7404d030 con 0x7f6c8c15bf30 2024-03-06T07:29:25.793 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:25.793 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:25.793 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:25.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.792+0000 7f6c90e2a640 1 -- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6c6c06b4d0 msgr2=0x7f6c6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.792+0000 7f6c90e2a640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6c6c06b4d0 0x7f6c6c06d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6c84006fd0 tx=0x7f6c84008040 comp rx=0 tx=0).stop 2024-03-06T07:29:25.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.792+0000 7f6c90e2a640 1 -- 172.21.15.59:0/3924562706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c8c15bf30 msgr2=0x7f6c8c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.792+0000 7f6c90e2a640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c06de80 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009f40 tx=0x7f6c74004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:25.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.793+0000 7f6c90e2a640 1 -- 172.21.15.59:0/3924562706 shutdown_connections 2024-03-06T07:29:25.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.793+0000 7f6c90e2a640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6c6c06b4d0 0x7f6c6c06d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.793+0000 7f6c90e2a640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c8c15e850 0x7f6c8c06e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.793+0000 7f6c90e2a640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c8c15bf30 0x7f6c8c06de80 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.793+0000 7f6c90e2a640 1 --2- 172.21.15.59:0/3924562706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c8c13dc10 0x7f6c8c159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.793+0000 7f6c90e2a640 1 -- 172.21.15.59:0/3924562706 >> 172.21.15.59:0/3924562706 conn(0x7f6c8c0cf8a0 msgr2=0x7f6c8c0b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.793+0000 7f6c90e2a640 1 -- 172.21.15.59:0/3924562706 shutdown_connections 2024-03-06T07:29:25.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:25.793+0000 7f6c90e2a640 1 -- 172.21.15.59:0/3924562706 wait complete. 2024-03-06T07:29:25.912 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:26.006 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:26.006 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:27.007 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.107+0000 7fb317f7a640 1 Processor -- start 2024-03-06T07:29:27.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.107+0000 7fb317f7a640 1 -- start start 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.108+0000 7fb317f7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100d2710 0x7fb3100b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.108+0000 7fb317f7a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3100b3430 con 0x7fb31015dfe0 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.108+0000 7fb317f7a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3100b3710 con 0x7fb3100d22b0 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.108+0000 7fb317f7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100b31e0 con 0x7fb3100d2710 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.108+0000 7fb315d16640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb31015dfe0 0x7fb3100cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:56008/0 (socket says 172.21.15.59:56008) 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.108+0000 7fb315d16640 1 -- 172.21.15.59:0/3738085641 learned_addr learned my addr 172.21.15.59:0/3738085641 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.108+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571253722 0 0) 0x7fb3100b3430 con 0x7fb31015dfe0 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.108+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc003520 con 0x7fb31015dfe0 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.108+0000 7fb316517640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100d2710 0x7fb3100b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1246639889 0 0) 0x7fb2fc003520 con 0x7fb31015dfe0 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100d2710 msgr2=0x7fb3100b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100d2710 0x7fb3100b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 >> v1:172.21.15.164:6789/0 conn(0x7fb3100d22b0 legacy=0x7fb3100b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3100b39f0 con 0x7fb31015dfe0 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fb30c001670 con 0x7fb31015dfe0 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 >> v1:172.21.15.59:6789/0 conn(0x7fb31015dfe0 legacy=0x7fb3100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 0x7fb2fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2fc004800 0x7fb2fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100d2710 0x7fb2fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100b31e0 con 0x7fb2fc0039e0 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc00baa0 con 0x7fb2fc004800 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc00bca0 con 0x7fb3100d2710 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb315d16640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 0x7fb2fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb316517640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100d2710 0x7fb2fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb315515640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2fc004800 0x7fb2fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb315d16640 1 -- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100d2710 msgr2=0x7fb2fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb315d16640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100d2710 0x7fb2fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb315d16640 1 -- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2fc004800 msgr2=0x7fb2fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb315d16640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2fc004800 0x7fb2fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb315d16640 1 -- 172.21.15.59:0/3738085641 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc003520 con 0x7fb2fc0039e0 2024-03-06T07:29:27.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.109+0000 7fb315515640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2fc004800 0x7fb2fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:27.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.110+0000 7fb315d16640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 0x7fb2fc003e20 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c210 tx=0x7fb2fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e6830093ac7f61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.111+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c007d10 con 0x7fb2fc0039e0 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.111+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30c030070 con 0x7fb2fc0039e0 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.111+0000 7fb314d14640 1 -- 172.21.15.59:0/3738085641 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c005d10 con 0x7fb2fc0039e0 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.111+0000 7fb317f7a640 1 -- 172.21.15.59:0/3738085641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 msgr2=0x7fb2fc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.111+0000 7fb317f7a640 1 --2- 172.21.15.59:0/3738085641 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 0x7fb2fc003e20 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c210 tx=0x7fb2fc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.111+0000 7fb316517640 1 -- 172.21.15.59:0/3738085641 reap_dead start 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.111+0000 7fb317f7a640 1 -- 172.21.15.59:0/3738085641 shutdown_connections 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.111+0000 7fb317f7a640 1 -- 172.21.15.59:0/3738085641 >> 172.21.15.59:0/3738085641 conn(0x7fb3100d3cd0 msgr2=0x7fb3100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.111+0000 7fb317f7a640 1 -- 172.21.15.59:0/3738085641 shutdown_connections 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.111+0000 7fb317f7a640 1 -- 172.21.15.59:0/3738085641 wait complete. 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.112+0000 7fb317f7a640 1 Processor -- start 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.112+0000 7fb317f7a640 1 -- start start 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb317f7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 0x7fb3101f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb317f7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100cbdf0 0x7fb3101f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb317f7a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3101510a0 0x7fb3101f7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb317f7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100cfa00 con 0x7fb2fc0039e0 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb317f7a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100b3d60 con 0x7fb3101510a0 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb317f7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100d1c70 con 0x7fb3100cbdf0 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb315d16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 0x7fb3101f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb315d16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 0x7fb3101f3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58662/0 (socket says 172.21.15.59:58662) 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb315d16640 1 -- 172.21.15.59:0/2443860064 learned_addr learned my addr 172.21.15.59:0/2443860064 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb315d16640 1 -- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100cbdf0 msgr2=0x7fb3101f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb315d16640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100cbdf0 0x7fb3101f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb315d16640 1 -- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3101510a0 msgr2=0x7fb3101f7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb315d16640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3101510a0 0x7fb3101f7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.113+0000 7fb315d16640 1 -- 172.21.15.59:0/2443860064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3101612b0 con 0x7fb2fc0039e0 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.114+0000 7fb315d16640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 0x7fb3101f3470 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c330 tx=0x7fb3100cfd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.114+0000 7fb3067fc640 1 -- 172.21.15.59:0/2443860064 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c045070 con 0x7fb2fc0039e0 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.114+0000 7fb3067fc640 1 -- 172.21.15.59:0/2443860064 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30c004bf0 con 0x7fb2fc0039e0 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.114+0000 7fb317f7a640 1 -- 172.21.15.59:0/2443860064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb310161590 con 0x7fb2fc0039e0 2024-03-06T07:29:27.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.114+0000 7fb3067fc640 1 -- 172.21.15.59:0/2443860064 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c007d10 con 0x7fb2fc0039e0 2024-03-06T07:29:27.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.114+0000 7fb317f7a640 1 -- 172.21.15.59:0/2443860064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb310150df0 con 0x7fb2fc0039e0 2024-03-06T07:29:27.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.115+0000 7fb317f7a640 1 -- 172.21.15.59:0/2443860064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31004f330 con 0x7fb2fc0039e0 2024-03-06T07:29:27.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.115+0000 7fb3067fc640 1 -- 172.21.15.59:0/2443860064 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb30c007070 con 0x7fb2fc0039e0 2024-03-06T07:29:27.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.115+0000 7fb3067fc640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2f406b500 0x7fb2f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.116+0000 7fb3067fc640 1 -- 172.21.15.59:0/2443860064 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb30c0aaf90 con 0x7fb2fc0039e0 2024-03-06T07:29:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.116+0000 7fb315515640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2f406b500 0x7fb2f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.116+0000 7fb315515640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2f406b500 0x7fb2f406d9c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb300007fd0 tx=0x7fb300009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.124+0000 7fb3067fc640 1 -- 172.21.15.59:0/2443860064 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb30c079930 con 0x7fb2fc0039e0 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.346+0000 7fb317f7a640 1 -- 172.21.15.59:0/2443860064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb31015d3e0 con 0x7fb2fc0039e0 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.346+0000 7fb3067fc640 1 -- 172.21.15.59:0/2443860064 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7fb30c07d9d0 con 0x7fb2fc0039e0 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:27.349 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:27.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 -- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2f406b500 msgr2=0x7fb2f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2f406b500 0x7fb2f406d9c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb300007fd0 tx=0x7fb300009040 comp rx=0 tx=0).stop 2024-03-06T07:29:27.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 -- 172.21.15.59:0/2443860064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 msgr2=0x7fb3101f3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 0x7fb3101f3470 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c330 tx=0x7fb3100cfd40 comp rx=0 tx=0).stop 2024-03-06T07:29:27.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 -- 172.21.15.59:0/2443860064 shutdown_connections 2024-03-06T07:29:27.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2f406b500 0x7fb2f406d9c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3101510a0 0x7fb3101f7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3100cbdf0 0x7fb3101f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 --2- 172.21.15.59:0/2443860064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2fc0039e0 0x7fb3101f3470 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 -- 172.21.15.59:0/2443860064 >> 172.21.15.59:0/2443860064 conn(0x7fb3100d3cd0 msgr2=0x7fb3100d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 -- 172.21.15.59:0/2443860064 shutdown_connections 2024-03-06T07:29:27.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:27.348+0000 7fb317f7a640 1 -- 172.21.15.59:0/2443860064 wait complete. 2024-03-06T07:29:27.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:27.562 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:27.914 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:28.563 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.664+0000 7f2bf5d1e640 1 Processor -- start 2024-03-06T07:29:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.664+0000 7f2bf5d1e640 1 -- start start 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bf5d1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf00d2710 0x7f2bf00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bf5d1e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf00b3430 con 0x7f2bf015dfe0 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bf5d1e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf00b3710 con 0x7f2bf00d22b0 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bf5d1e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf00b31e0 con 0x7f2bf00d2710 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bef7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2bf015dfe0 0x7f2bf00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:56020/0 (socket says 172.21.15.59:56020) 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bef7fe640 1 -- 172.21.15.59:0/38322965 learned_addr learned my addr 172.21.15.59:0/38322965 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2beffff640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf00d2710 0x7f2bf00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685246791 0 0) 0x7f2bf00b3430 con 0x7f2bf015dfe0 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bdc003520 con 0x7f2bf015dfe0 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2550845139 0 0) 0x7f2bdc003520 con 0x7f2bf015dfe0 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf00d2710 msgr2=0x7f2bf00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bee7fc640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf00d2710 0x7f2bf00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 >> v1:172.21.15.164:6789/0 conn(0x7f2bf00d22b0 legacy=0x7f2bf00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.665+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf00b39f0 con 0x7f2bf015dfe0 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f2be00019f0 con 0x7f2bf015dfe0 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 >> v1:172.21.15.59:6789/0 conn(0x7f2bf015dfe0 legacy=0x7f2bf00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2bee7fc640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bdc0039e0 0x7f2bdc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2bee7fc640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 0x7f2bdc004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2bee7fc640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf00d2710 0x7f2bdc0050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf00b31e0 con 0x7f2bdc0043b0 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc00ba20 con 0x7f2bdc0039e0 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc00bc20 con 0x7f2bf00d2710 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2beeffd640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 0x7f2bdc004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2bef7fe640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bdc0039e0 0x7f2bdc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2beffff640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf00d2710 0x7f2bdc0050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2beeffd640 1 -- 172.21.15.59:0/38322965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf00d2710 msgr2=0x7f2bdc0050e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2beeffd640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf00d2710 0x7f2bdc0050e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2beeffd640 1 -- 172.21.15.59:0/38322965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bdc0039e0 msgr2=0x7f2bdc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2beeffd640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bdc0039e0 0x7f2bdc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.666+0000 7f2beeffd640 1 -- 172.21.15.59:0/38322965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc003520 con 0x7f2bdc0043b0 2024-03-06T07:29:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.667+0000 7f2beeffd640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 0x7f2bdc004880 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8001860 tx=0x7f2bdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1aa03509ada8e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.667+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8038070 con 0x7f2bdc0043b0 2024-03-06T07:29:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.667+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd8033410 con 0x7f2bdc0043b0 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.667+0000 7f2bee7fc640 1 -- 172.21.15.59:0/38322965 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8001430 con 0x7f2bdc0043b0 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.668+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/38322965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 msgr2=0x7f2bdc004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.668+0000 7f2bf5d1e640 1 --2- 172.21.15.59:0/38322965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 0x7f2bdc004880 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8001860 tx=0x7f2bdc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.668+0000 7f2beffff640 1 -- 172.21.15.59:0/38322965 reap_dead start 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.668+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/38322965 shutdown_connections 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.668+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/38322965 >> 172.21.15.59:0/38322965 conn(0x7f2bf00d3cd0 msgr2=0x7f2bf00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.668+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/38322965 shutdown_connections 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.668+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/38322965 wait complete. 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.669+0000 7f2bf5d1e640 1 Processor -- start 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.669+0000 7f2bf5d1e640 1 -- start start 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bf5d1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 0x7f2bf015c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bf5d1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf01f3b20 0x7f2bf01f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bf5d1e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bf006bd90 0x7f2bf006c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bf5d1e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf00cfa00 con 0x7f2bdc0043b0 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bf5d1e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf00b3d60 con 0x7f2bf006bd90 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bf5d1e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf00d1c70 con 0x7f2bf01f3b20 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bef7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 0x7f2bf015c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bef7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 0x7f2bf015c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58682/0 (socket says 172.21.15.59:58682) 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bef7fe640 1 -- 172.21.15.59:0/3876386955 learned_addr learned my addr 172.21.15.59:0/3876386955 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bef7fe640 1 -- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf01f3b20 msgr2=0x7f2bf01f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bef7fe640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf01f3b20 0x7f2bf01f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bef7fe640 1 -- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bf006bd90 msgr2=0x7f2bf006c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bef7fe640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bf006bd90 0x7f2bf006c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bef7fe640 1 -- 172.21.15.59:0/3876386955 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf01f45a0 con 0x7f2bdc0043b0 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.670+0000 7f2bef7fe640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 0x7f2bf015c360 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f2be0007970 tx=0x7f2be000a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.671+0000 7f2be7fff640 1 -- 172.21.15.59:0/3876386955 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be00133f0 con 0x7f2bdc0043b0 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.671+0000 7f2be7fff640 1 -- 172.21.15.59:0/3876386955 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be0018070 con 0x7f2bdc0043b0 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.671+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/3876386955 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf01f4880 con 0x7f2bdc0043b0 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.671+0000 7f2be7fff640 1 -- 172.21.15.59:0/3876386955 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be00137d0 con 0x7f2bdc0043b0 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.671+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/3876386955 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf00b9730 con 0x7f2bdc0043b0 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.672+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/3876386955 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf00b3430 con 0x7f2bdc0043b0 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.672+0000 7f2be7fff640 1 -- 172.21.15.59:0/3876386955 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2be0013b80 con 0x7f2bdc0043b0 2024-03-06T07:29:28.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.673+0000 7f2be7fff640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bd006b4d0 0x7f2bd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.673+0000 7f2be7fff640 1 -- 172.21.15.59:0/3876386955 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2be008ae40 con 0x7f2bdc0043b0 2024-03-06T07:29:28.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.673+0000 7f2beeffd640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bd006b4d0 0x7f2bd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.673+0000 7f2beeffd640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bd006b4d0 0x7f2bd006d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8035000 tx=0x7f2bd8030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.677+0000 7f2be7fff640 1 -- 172.21.15.59:0/3876386955 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2bf00b3430 con 0x7f2bdc0043b0 2024-03-06T07:29:28.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.894+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/3876386955 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bf004f330 con 0x7f2bdc0043b0 2024-03-06T07:29:28.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.895+0000 7f2be7fff640 1 -- 172.21.15.59:0/3876386955 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f2be0059c20 con 0x7f2bdc0043b0 2024-03-06T07:29:28.897 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:28.897 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:28.897 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:28.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bd006b4d0 msgr2=0x7f2bd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bd006b4d0 0x7f2bd006d990 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8035000 tx=0x7f2bd8030000 comp rx=0 tx=0).stop 2024-03-06T07:29:28.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/3876386955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 msgr2=0x7f2bf015c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 0x7f2bf015c360 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f2be0007970 tx=0x7f2be000a730 comp rx=0 tx=0).stop 2024-03-06T07:29:28.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/3876386955 shutdown_connections 2024-03-06T07:29:28.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2bd006b4d0 0x7f2bd006d990 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bf006bd90 0x7f2bf006c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bf01f3b20 0x7f2bf01f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 --2- 172.21.15.59:0/3876386955 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bdc0043b0 0x7f2bf015c360 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/3876386955 >> 172.21.15.59:0/3876386955 conn(0x7f2bf00d3cd0 msgr2=0x7f2bf00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/3876386955 shutdown_connections 2024-03-06T07:29:28.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:28.897+0000 7f2bf5d1e640 1 -- 172.21.15.59:0/3876386955 wait complete. 2024-03-06T07:29:29.112 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:29.112 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:29.916 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:30.113 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.216+0000 7fcb28047640 1 Processor -- start 2024-03-06T07:29:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.216+0000 7fcb28047640 1 -- start start 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb28047640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb2015fa10 0x7fcb200ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb28047640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcb200d16e0 0x7fcb200d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb28047640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb200cebe0 con 0x7fcb2015bdd0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb28047640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb200ceec0 con 0x7fcb200d16e0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb28047640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200cf1a0 con 0x7fcb2015fa10 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb25de3640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcb2015bdd0 0x7fcb200c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:56032/0 (socket says 172.21.15.59:56032) 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb25de3640 1 -- 172.21.15.59:0/2285440058 learned_addr learned my addr 172.21.15.59:0/2285440058 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb255e2640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb2015fa10 0x7fcb200ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb265e4640 1 --2- 172.21.15.59:0/2285440058 >> v2:172.21.15.164:3300/0 conn(0x7fcb200d16e0 0x7fcb200d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990508501 0 0) 0x7fcb200cebe0 con 0x7fcb2015bdd0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb0c003520 con 0x7fcb2015bdd0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2677263797 0 0) 0x7fcb0c003520 con 0x7fcb2015bdd0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb2015fa10 msgr2=0x7fcb200ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb2015fa10 0x7fcb200ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 >> v2:172.21.15.164:3300/0 conn(0x7fcb200d16e0 msgr2=0x7fcb200d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 --2- 172.21.15.59:0/2285440058 >> v2:172.21.15.164:3300/0 conn(0x7fcb200d16e0 0x7fcb200d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200cc8e0 con 0x7fcb2015bdd0 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fcb1c001670 con 0x7fcb2015bdd0 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 >> v1:172.21.15.59:6789/0 conn(0x7fcb2015bdd0 legacy=0x7fcb200c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb0c0039b0 0x7fcb0c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb2015fa10 0x7fcb0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.218+0000 7fcb24de1640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c004d40 0x7fcb0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200ceec0 con 0x7fcb0c0039b0 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200cf1a0 con 0x7fcb0c004d40 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c00b9f0 con 0x7fcb2015fa10 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb25de3640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb0c0039b0 0x7fcb0c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb255e2640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb2015fa10 0x7fcb0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb265e4640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c004d40 0x7fcb0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb25de3640 1 -- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb2015fa10 msgr2=0x7fcb0c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb25de3640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb2015fa10 0x7fcb0c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb25de3640 1 -- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c004d40 msgr2=0x7fcb0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb25de3640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c004d40 0x7fcb0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb25de3640 1 -- 172.21.15.59:0/2285440058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c003520 con 0x7fcb0c0039b0 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.219+0000 7fcb265e4640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb0c004d40 0x7fcb0c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.220+0000 7fcb25de3640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb0c0039b0 0x7fcb0c003dd0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c001050 tx=0x7fcb0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655ecfb751b4601f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.220+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c007d70 con 0x7fcb0c0039b0 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.220+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1c030070 con 0x7fcb0c0039b0 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.220+0000 7fcb24de1640 1 -- 172.21.15.59:0/2285440058 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c02cd80 con 0x7fcb0c0039b0 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.220+0000 7fcb28047640 1 -- 172.21.15.59:0/2285440058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb0c0039b0 msgr2=0x7fcb0c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.220+0000 7fcb28047640 1 --2- 172.21.15.59:0/2285440058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb0c0039b0 0x7fcb0c003dd0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c001050 tx=0x7fcb0c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.220+0000 7fcb265e4640 1 -- 172.21.15.59:0/2285440058 reap_dead start 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.220+0000 7fcb28047640 1 -- 172.21.15.59:0/2285440058 shutdown_connections 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.220+0000 7fcb28047640 1 -- 172.21.15.59:0/2285440058 >> 172.21.15.59:0/2285440058 conn(0x7fcb2009a9f0 msgr2=0x7fcb2009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.221+0000 7fcb28047640 1 -- 172.21.15.59:0/2285440058 shutdown_connections 2024-03-06T07:29:30.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.221+0000 7fcb28047640 1 -- 172.21.15.59:0/2285440058 wait complete. 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.222+0000 7fcb28047640 1 Processor -- start 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.222+0000 7fcb28047640 1 -- start start 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.222+0000 7fcb28047640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb0c0039b0 0x7fcb20157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.222+0000 7fcb28047640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb201f7b10 0x7fcb2014ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.222+0000 7fcb28047640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb200b68f0 0x7fcb200c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.222+0000 7fcb28047640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2015c1b0 con 0x7fcb201f7b10 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.222+0000 7fcb28047640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200cc5d0 con 0x7fcb200b68f0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.222+0000 7fcb28047640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200ce270 con 0x7fcb0c0039b0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb255e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb201f7b10 0x7fcb2014ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb255e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb201f7b10 0x7fcb2014ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58702/0 (socket says 172.21.15.59:58702) 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb255e2640 1 -- 172.21.15.59:0/3922319197 learned_addr learned my addr 172.21.15.59:0/3922319197 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb25de3640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb0c0039b0 0x7fcb20157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb265e4640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb200b68f0 0x7fcb200c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb255e2640 1 -- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb0c0039b0 msgr2=0x7fcb20157610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb255e2640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb0c0039b0 0x7fcb20157610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb255e2640 1 -- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb200b68f0 msgr2=0x7fcb200c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb255e2640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb200b68f0 0x7fcb200c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb255e2640 1 -- 172.21.15.59:0/3922319197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb201f7f70 con 0x7fcb201f7b10 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb255e2640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb201f7b10 0x7fcb2014ee00 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fcb10011630 tx=0x7fcb10011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb167fc640 1 -- 172.21.15.59:0/3922319197 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1000f8c0 con 0x7fcb201f7b10 2024-03-06T07:29:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb167fc640 1 -- 172.21.15.59:0/3922319197 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1000fa60 con 0x7fcb201f7b10 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb28047640 1 -- 172.21.15.59:0/3922319197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb200b5850 con 0x7fcb201f7b10 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb167fc640 1 -- 172.21.15.59:0/3922319197 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1000fe10 con 0x7fcb201f7b10 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.223+0000 7fcb28047640 1 -- 172.21.15.59:0/3922319197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb200b5e10 con 0x7fcb201f7b10 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.224+0000 7fcb28047640 1 -- 172.21.15.59:0/3922319197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb200ce6a0 con 0x7fcb201f7b10 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.225+0000 7fcb167fc640 1 -- 172.21.15.59:0/3922319197 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb10015460 con 0x7fcb201f7b10 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.226+0000 7fcb167fc640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb0406b4d0 0x7fcb0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.226+0000 7fcb167fc640 1 -- 172.21.15.59:0/3922319197 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb1008a8f0 con 0x7fcb201f7b10 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.226+0000 7fcb25de3640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb0406b4d0 0x7fcb0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.226+0000 7fcb25de3640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb0406b4d0 0x7fcb0406d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c034000 tx=0x7fcb1c007000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.230+0000 7fcb167fc640 1 -- 172.21.15.59:0/3922319197 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb200ce6a0 con 0x7fcb201f7b10 2024-03-06T07:29:30.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.659+0000 7fcb28047640 1 -- 172.21.15.59:0/3922319197 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb200b7a40 con 0x7fcb201f7b10 2024-03-06T07:29:30.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.660+0000 7fcb167fc640 1 -- 172.21.15.59:0/3922319197 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7fcb10019030 con 0x7fcb201f7b10 2024-03-06T07:29:30.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:30.662 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:30.662 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 -- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb0406b4d0 msgr2=0x7fcb0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb0406b4d0 0x7fcb0406d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c034000 tx=0x7fcb1c007000 comp rx=0 tx=0).stop 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 -- 172.21.15.59:0/3922319197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb201f7b10 msgr2=0x7fcb2014ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb201f7b10 0x7fcb2014ee00 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fcb10011630 tx=0x7fcb10011af0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 -- 172.21.15.59:0/3922319197 shutdown_connections 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcb0406b4d0 0x7fcb0406d990 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb200b68f0 0x7fcb200c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb201f7b10 0x7fcb2014ee00 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 --2- 172.21.15.59:0/3922319197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb0c0039b0 0x7fcb20157610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 -- 172.21.15.59:0/3922319197 >> 172.21.15.59:0/3922319197 conn(0x7fcb2009a9f0 msgr2=0x7fcb200c55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 -- 172.21.15.59:0/3922319197 shutdown_connections 2024-03-06T07:29:30.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:30.662+0000 7fcb28047640 1 -- 172.21.15.59:0/3922319197 wait complete. 2024-03-06T07:29:30.880 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:30.880 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:31.881 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:31.917 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:31.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.980+0000 7f30d31d7640 1 Processor -- start 2024-03-06T07:29:31.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.980+0000 7f30d31d7640 1 -- start start 2024-03-06T07:29:31.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30d31d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30cc15dfe0 0x7f30cc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30d31d7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30cc0cf830 0x7f30cc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30d31d7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30cc0b3430 con 0x7f30cc0d27f0 2024-03-06T07:29:31.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30d31d7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30cc0d2c50 con 0x7f30cc0cf830 2024-03-06T07:29:31.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30d31d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0b31e0 con 0x7f30cc15dfe0 2024-03-06T07:29:31.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30cbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30cc0cf830 0x7f30cc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30cbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30cc0cf830 0x7f30cc0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58704/0 (socket says 172.21.15.59:58704) 2024-03-06T07:29:31.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30cbfff640 1 -- 172.21.15.59:0/575243140 learned_addr learned my addr 172.21.15.59:0/575243140 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:31.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30d0f73640 1 --2- 172.21.15.59:0/575243140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30cc15dfe0 0x7f30cc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30cbfff640 1 -- 172.21.15.59:0/575243140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30cc15dfe0 msgr2=0x7f30cc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30cbfff640 1 --2- 172.21.15.59:0/575243140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30cc15dfe0 0x7f30cc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30cbfff640 1 -- 172.21.15.59:0/575243140 >> v1:172.21.15.164:6789/0 conn(0x7f30cc0d27f0 legacy=0x7f30cc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.982+0000 7f30cbfff640 1 -- 172.21.15.59:0/575243140 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc0b3710 con 0x7f30cc0cf830 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.983+0000 7f30cbfff640 1 --2- 172.21.15.59:0/575243140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30cc0cf830 0x7f30cc0d22b0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009870 tx=0x7f30b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199180fd59115da8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.983+0000 7f30cb7fe640 1 -- 172.21.15.59:0/575243140 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8035020 con 0x7f30cc0cf830 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.983+0000 7f30cb7fe640 1 -- 172.21.15.59:0/575243140 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b8031640 con 0x7f30cc0cf830 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.983+0000 7f30cb7fe640 1 -- 172.21.15.59:0/575243140 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b802d040 con 0x7f30cc0cf830 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.984+0000 7f30d31d7640 1 -- 172.21.15.59:0/575243140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30cc0cf830 msgr2=0x7f30cc0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.984+0000 7f30d31d7640 1 --2- 172.21.15.59:0/575243140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30cc0cf830 0x7f30cc0d22b0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009870 tx=0x7f30b802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.984+0000 7f30d31d7640 1 -- 172.21.15.59:0/575243140 shutdown_connections 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.984+0000 7f30d31d7640 1 --2- 172.21.15.59:0/575243140 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30cc0cf830 0x7f30cc0d22b0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.984+0000 7f30d31d7640 1 --2- 172.21.15.59:0/575243140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30cc15dfe0 0x7f30cc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.984+0000 7f30d31d7640 1 -- 172.21.15.59:0/575243140 >> 172.21.15.59:0/575243140 conn(0x7f30cc0d3cd0 msgr2=0x7f30cc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.984+0000 7f30d31d7640 1 -- 172.21.15.59:0/575243140 shutdown_connections 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.984+0000 7f30d31d7640 1 -- 172.21.15.59:0/575243140 wait complete. 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.985+0000 7f30d31d7640 1 Processor -- start 2024-03-06T07:29:31.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.985+0000 7f30d31d7640 1 -- start start 2024-03-06T07:29:31.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d31d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30cc0cf830 0x7f30cc160e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d31d7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30cc0d27f0 0x7f30cc0b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d31d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30cc15dfe0 0x7f30cc15b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d31d7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0d1c70 con 0x7f30cc15dfe0 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d31d7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0b3d60 con 0x7f30cc0d27f0 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d31d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc0d1df0 con 0x7f30cc0cf830 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d1774640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30cc15dfe0 0x7f30cc15b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d1774640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30cc15dfe0 0x7f30cc15b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58716/0 (socket says 172.21.15.59:58716) 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d1774640 1 -- 172.21.15.59:0/3401891957 learned_addr learned my addr 172.21.15.59:0/3401891957 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d0f73640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30cc0cf830 0x7f30cc160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d1774640 1 -- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30cc0cf830 msgr2=0x7f30cc160e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d1774640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30cc0cf830 0x7f30cc160e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d1774640 1 -- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30cc0d27f0 msgr2=0x7f30cc0b8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30cbfff640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30cc0d27f0 0x7f30cc0b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d1774640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30cc0d27f0 0x7f30cc0b8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d1774640 1 -- 172.21.15.59:0/3401891957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc1613b0 con 0x7f30cc15dfe0 2024-03-06T07:29:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d1774640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30cc15dfe0 0x7f30cc15b8e0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f30c400c7b0 tx=0x7f30c400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30c97fa640 1 -- 172.21.15.59:0/3401891957 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4004b60 con 0x7f30cc15dfe0 2024-03-06T07:29:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30c97fa640 1 -- 172.21.15.59:0/3401891957 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c4004d00 con 0x7f30cc15dfe0 2024-03-06T07:29:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30c97fa640 1 -- 172.21.15.59:0/3401891957 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c400aa90 con 0x7f30cc15dfe0 2024-03-06T07:29:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.986+0000 7f30d31d7640 1 -- 172.21.15.59:0/3401891957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30cc15fdd0 con 0x7f30cc15dfe0 2024-03-06T07:29:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.987+0000 7f30d31d7640 1 -- 172.21.15.59:0/3401891957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30cc160390 con 0x7f30cc15dfe0 2024-03-06T07:29:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.988+0000 7f30d31d7640 1 -- 172.21.15.59:0/3401891957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30cc0cf5f0 con 0x7f30cc15dfe0 2024-03-06T07:29:31.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.988+0000 7f30c97fa640 1 -- 172.21.15.59:0/3401891957 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30c400ac30 con 0x7f30cc15dfe0 2024-03-06T07:29:31.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.988+0000 7f30c97fa640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30b006b500 0x7f30b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.989+0000 7f30c97fa640 1 -- 172.21.15.59:0/3401891957 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30c408a380 con 0x7f30cc15dfe0 2024-03-06T07:29:31.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.989+0000 7f30d0f73640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30b006b500 0x7f30b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.989+0000 7f30d0f73640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30b006b500 0x7f30b006d9c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f30c0006fd0 tx=0x7f30c0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:31.992+0000 7f30c97fa640 1 -- 172.21.15.59:0/3401891957 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30cc0cf5f0 con 0x7f30cc15dfe0 2024-03-06T07:29:32.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.379+0000 7f30d31d7640 1 -- 172.21.15.59:0/3401891957 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30cc04f330 con 0x7f30cc15dfe0 2024-03-06T07:29:32.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.379+0000 7f30c97fa640 1 -- 172.21.15.59:0/3401891957 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f30c4020070 con 0x7f30cc15dfe0 2024-03-06T07:29:32.382 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:32.382 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:32.382 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:32.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 -- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30b006b500 msgr2=0x7f30b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30b006b500 0x7f30b006d9c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f30c0006fd0 tx=0x7f30c0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:32.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 -- 172.21.15.59:0/3401891957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30cc15dfe0 msgr2=0x7f30cc15b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30cc15dfe0 0x7f30cc15b8e0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f30c400c7b0 tx=0x7f30c400cc70 comp rx=0 tx=0).stop 2024-03-06T07:29:32.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 -- 172.21.15.59:0/3401891957 shutdown_connections 2024-03-06T07:29:32.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f30b006b500 0x7f30b006d9c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30cc15dfe0 0x7f30cc15b8e0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30cc0d27f0 0x7f30cc0b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 --2- 172.21.15.59:0/3401891957 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30cc0cf830 0x7f30cc160e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 -- 172.21.15.59:0/3401891957 >> 172.21.15.59:0/3401891957 conn(0x7f30cc0d3cd0 msgr2=0x7f30cc0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 -- 172.21.15.59:0/3401891957 shutdown_connections 2024-03-06T07:29:32.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:32.382+0000 7f30d31d7640 1 -- 172.21.15.59:0/3401891957 wait complete. 2024-03-06T07:29:32.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:32.596 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:33.597 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:33.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.720+0000 7f71f02a4640 1 Processor -- start 2024-03-06T07:29:33.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.720+0000 7f71f02a4640 1 -- start start 2024-03-06T07:29:33.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71f02a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e815dfe0 0x7f71e80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71f02a4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f71e80cf830 0x7f71e80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71f02a4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f71e80d27f0 0x7f71e80d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71f02a4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e80d2c50 con 0x7f71e80d27f0 2024-03-06T07:29:33.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71f02a4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e80b31e0 con 0x7f71e80cf830 2024-03-06T07:29:33.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71f02a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80b3d60 con 0x7f71e815dfe0 2024-03-06T07:29:33.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71ee841640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f71e80d27f0 0x7f71e80d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71ee841640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f71e80d27f0 0x7f71e80d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55332/0 (socket says 172.21.15.59:55332) 2024-03-06T07:29:33.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71ee841640 1 -- 172.21.15.59:0/4205115003 learned_addr learned my addr 172.21.15.59:0/4205115003 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:33.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71ee841640 1 -- 172.21.15.59:0/4205115003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e815dfe0 msgr2=0x7f71e80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71ed83f640 1 --2- 172.21.15.59:0/4205115003 >> v2:172.21.15.164:3300/0 conn(0x7f71e80cf830 0x7f71e80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71ee040640 1 --2- 172.21.15.59:0/4205115003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e815dfe0 0x7f71e80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71ee841640 1 --2- 172.21.15.59:0/4205115003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e815dfe0 0x7f71e80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71ee841640 1 -- 172.21.15.59:0/4205115003 >> v2:172.21.15.164:3300/0 conn(0x7f71e80cf830 msgr2=0x7f71e80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71ee841640 1 --2- 172.21.15.59:0/4205115003 >> v2:172.21.15.164:3300/0 conn(0x7f71e80cf830 0x7f71e80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.721+0000 7f71ee841640 1 -- 172.21.15.59:0/4205115003 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e80b3430 con 0x7f71e80d27f0 2024-03-06T07:29:33.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.722+0000 7f71ee841640 1 --2- 172.21.15.59:0/4205115003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e80d27f0 0x7f71e80d0300 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f71e808cba0 tx=0x7f71e002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09fc040cd919827 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.722+0000 7f71ed03e640 1 -- 172.21.15.59:0/4205115003 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e002ca70 con 0x7f71e80d27f0 2024-03-06T07:29:33.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.722+0000 7f71ed03e640 1 -- 172.21.15.59:0/4205115003 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e002cc10 con 0x7f71e80d27f0 2024-03-06T07:29:33.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.722+0000 7f71ed03e640 1 -- 172.21.15.59:0/4205115003 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e00343f0 con 0x7f71e80d27f0 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.723+0000 7f71f02a4640 1 -- 172.21.15.59:0/4205115003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e80d27f0 msgr2=0x7f71e80d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.723+0000 7f71f02a4640 1 --2- 172.21.15.59:0/4205115003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e80d27f0 0x7f71e80d0300 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f71e808cba0 tx=0x7f71e002bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.723+0000 7f71f02a4640 1 -- 172.21.15.59:0/4205115003 shutdown_connections 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.723+0000 7f71f02a4640 1 --2- 172.21.15.59:0/4205115003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e80d27f0 0x7f71e80d0300 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.723+0000 7f71f02a4640 1 --2- 172.21.15.59:0/4205115003 >> v2:172.21.15.164:3300/0 conn(0x7f71e80cf830 0x7f71e80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.723+0000 7f71f02a4640 1 --2- 172.21.15.59:0/4205115003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e815dfe0 0x7f71e80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.723+0000 7f71f02a4640 1 -- 172.21.15.59:0/4205115003 >> 172.21.15.59:0/4205115003 conn(0x7f71e80d3cd0 msgr2=0x7f71e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.723+0000 7f71f02a4640 1 -- 172.21.15.59:0/4205115003 shutdown_connections 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.723+0000 7f71f02a4640 1 -- 172.21.15.59:0/4205115003 wait complete. 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.724+0000 7f71f02a4640 1 Processor -- start 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.724+0000 7f71f02a4640 1 -- start start 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71f02a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e80cf830 0x7f71e81f0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71f02a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e80d27f0 0x7f71e80b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71f02a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e815dfe0 0x7f71e81f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71f02a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80d1c70 con 0x7f71e80d27f0 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71f02a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80b39f0 con 0x7f71e80cf830 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71f02a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e80d1df0 con 0x7f71e815dfe0 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ed83f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e80d27f0 0x7f71e80b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ed83f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e80d27f0 0x7f71e80b95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55346/0 (socket says 172.21.15.59:55346) 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ed83f640 1 -- 172.21.15.59:0/853248021 learned_addr learned my addr 172.21.15.59:0/853248021 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ee040640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e80cf830 0x7f71e81f0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ee841640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e815dfe0 0x7f71e81f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ed83f640 1 -- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e815dfe0 msgr2=0x7f71e81f6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ed83f640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e815dfe0 0x7f71e81f6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ed83f640 1 -- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e80cf830 msgr2=0x7f71e81f0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ed83f640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e80cf830 0x7f71e81f0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ed83f640 1 -- 172.21.15.59:0/853248021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e81605f0 con 0x7f71e80d27f0 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ee040640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e80cf830 0x7f71e81f0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71ed83f640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e80d27f0 0x7f71e80b95d0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f71d800c7f0 tx=0x7f71d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71deffd640 1 -- 172.21.15.59:0/853248021 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d8015070 con 0x7f71e80d27f0 2024-03-06T07:29:33.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71deffd640 1 -- 172.21.15.59:0/853248021 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d8004220 con 0x7f71e80d27f0 2024-03-06T07:29:33.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71f02a4640 1 -- 172.21.15.59:0/853248021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e81608d0 con 0x7f71e80d27f0 2024-03-06T07:29:33.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71deffd640 1 -- 172.21.15.59:0/853248021 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d8007620 con 0x7f71e80d27f0 2024-03-06T07:29:33.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.725+0000 7f71f02a4640 1 -- 172.21.15.59:0/853248021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e8160e90 con 0x7f71e80d27f0 2024-03-06T07:29:33.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.727+0000 7f71f02a4640 1 -- 172.21.15.59:0/853248021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e804f330 con 0x7f71e80d27f0 2024-03-06T07:29:33.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.727+0000 7f71deffd640 1 -- 172.21.15.59:0/853248021 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71d80077c0 con 0x7f71e80d27f0 2024-03-06T07:29:33.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.727+0000 7f71deffd640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71cc06b4d0 0x7f71cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.727+0000 7f71deffd640 1 -- 172.21.15.59:0/853248021 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71d808a0c0 con 0x7f71e80d27f0 2024-03-06T07:29:33.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.727+0000 7f71ee040640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71cc06b4d0 0x7f71cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.728+0000 7f71ee040640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71cc06b4d0 0x7f71cc06d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f71e400b820 tx=0x7f71e4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.731+0000 7f71deffd640 1 -- 172.21.15.59:0/853248021 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71d8058eb0 con 0x7f71e80d27f0 2024-03-06T07:29:33.919 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:33.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.953+0000 7f71f02a4640 1 -- 172.21.15.59:0/853248021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71e81f1310 con 0x7f71e80d27f0 2024-03-06T07:29:33.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.954+0000 7f71deffd640 1 -- 172.21.15.59:0/853248021 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f71d80587b0 con 0x7f71e80d27f0 2024-03-06T07:29:33.956 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:33.956 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:33.956 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 -- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71cc06b4d0 msgr2=0x7f71cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71cc06b4d0 0x7f71cc06d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f71e400b820 tx=0x7f71e4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 -- 172.21.15.59:0/853248021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e80d27f0 msgr2=0x7f71e80b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e80d27f0 0x7f71e80b95d0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f71d800c7f0 tx=0x7f71d800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 -- 172.21.15.59:0/853248021 shutdown_connections 2024-03-06T07:29:33.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71cc06b4d0 0x7f71cc06d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e815dfe0 0x7f71e81f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e80d27f0 0x7f71e80b95d0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 --2- 172.21.15.59:0/853248021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e80cf830 0x7f71e81f0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 -- 172.21.15.59:0/853248021 >> 172.21.15.59:0/853248021 conn(0x7f71e80d3cd0 msgr2=0x7f71e806cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 -- 172.21.15.59:0/853248021 shutdown_connections 2024-03-06T07:29:33.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:33.956+0000 7f71f02a4640 1 -- 172.21.15.59:0/853248021 wait complete. 2024-03-06T07:29:34.170 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:34.170 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:35.170 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:35.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.271+0000 7f5376066640 1 Processor -- start 2024-03-06T07:29:35.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.271+0000 7f5376066640 1 -- start start 2024-03-06T07:29:35.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.271+0000 7f5376066640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f537015bb20 0x7f53700d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.271+0000 7f5376066640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5370161980 con 0x7f53700cfff0 2024-03-06T07:29:35.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.271+0000 7f5376066640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5370161c60 con 0x7f53700cd8c0 2024-03-06T07:29:35.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.271+0000 7f5376066640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5370161f40 con 0x7f537015bb20 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.271+0000 7f536effd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f53700cfff0 0x7f53700ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51724/0 (socket says 172.21.15.59:51724) 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536effd640 1 -- 172.21.15.59:0/3881612207 learned_addr learned my addr 172.21.15.59:0/3881612207 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536f7fe640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f537015bb20 0x7f53700d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419971468 0 0) 0x7f5370161980 con 0x7f53700cfff0 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f535c003520 con 0x7f53700cfff0 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1068130460 0 0) 0x7f535c003520 con 0x7f53700cfff0 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f537015bb20 msgr2=0x7f53700d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f537015bb20 0x7f53700d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 >> v1:172.21.15.164:6789/0 conn(0x7f53700cd8c0 legacy=0x7f53700cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5370162220 con 0x7f53700cfff0 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536f7fe640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f537015bb20 0x7f53700d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f5358001670 con 0x7f53700cfff0 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 >> v1:172.21.15.59:6789/0 conn(0x7f53700cfff0 legacy=0x7f53700ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 0x7f535c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f537015bb20 0x7f535c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f535c004d40 0x7f535c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5370161f40 con 0x7f535c0039e0 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535c00ba10 con 0x7f535c004d40 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f535c00bc10 con 0x7f537015bb20 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536f7fe640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 0x7f535c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536effd640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f537015bb20 0x7f535c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.272+0000 7f536ffff640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f535c004d40 0x7f535c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.273+0000 7f536f7fe640 1 -- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f537015bb20 msgr2=0x7f535c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.273+0000 7f536f7fe640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f537015bb20 0x7f535c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.273+0000 7f536f7fe640 1 -- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f535c004d40 msgr2=0x7f535c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.273+0000 7f536f7fe640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f535c004d40 0x7f535c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.273+0000 7f536f7fe640 1 -- 172.21.15.59:0/3881612207 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535c003520 con 0x7f535c0039e0 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.273+0000 7f536ffff640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f535c004d40 0x7f535c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.273+0000 7f536f7fe640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 0x7f535c003dc0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f5360009c90 tx=0x7f535c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71209a9462ba289 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.274+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360039070 con 0x7f535c0039e0 2024-03-06T07:29:35.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.274+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536002c8f0 con 0x7f535c0039e0 2024-03-06T07:29:35.278 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.274+0000 7f536e7fc640 1 -- 172.21.15.59:0/3881612207 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360009860 con 0x7f535c0039e0 2024-03-06T07:29:35.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.274+0000 7f5376066640 1 -- 172.21.15.59:0/3881612207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 msgr2=0x7f535c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.274+0000 7f5376066640 1 --2- 172.21.15.59:0/3881612207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 0x7f535c003dc0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f5360009c90 tx=0x7f535c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.274+0000 7f536ffff640 1 -- 172.21.15.59:0/3881612207 reap_dead start 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.274+0000 7f5376066640 1 -- 172.21.15.59:0/3881612207 shutdown_connections 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.274+0000 7f5376066640 1 -- 172.21.15.59:0/3881612207 >> 172.21.15.59:0/3881612207 conn(0x7f537009a9f0 msgr2=0x7f537009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.274+0000 7f5376066640 1 -- 172.21.15.59:0/3881612207 shutdown_connections 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.274+0000 7f5376066640 1 -- 172.21.15.59:0/3881612207 wait complete. 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.275+0000 7f5376066640 1 Processor -- start 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.276+0000 7f5376066640 1 -- start start 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.276+0000 7f5376066640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 0x7f5370160e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.276+0000 7f5376066640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53700c6720 0x7f537015cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.276+0000 7f5376066640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53700c6ba0 0x7f537015d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.276+0000 7f5376066640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53700d1980 con 0x7f535c0039e0 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.276+0000 7f5376066640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53700cf2d0 con 0x7f53700c6ba0 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.276+0000 7f5376066640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53700d1b00 con 0x7f53700c6720 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.276+0000 7f536f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 0x7f5370160e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.276+0000 7f536f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 0x7f5370160e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55368/0 (socket says 172.21.15.59:55368) 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.276+0000 7f536f7fe640 1 -- 172.21.15.59:0/561709644 learned_addr learned my addr 172.21.15.59:0/561709644 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:35.281 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f536effd640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53700c6720 0x7f537015cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f536ffff640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53700c6ba0 0x7f537015d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f536f7fe640 1 -- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53700c6720 msgr2=0x7f537015cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f536f7fe640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53700c6720 0x7f537015cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f536f7fe640 1 -- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53700c6ba0 msgr2=0x7f537015d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f536f7fe640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53700c6ba0 0x7f537015d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f536f7fe640 1 -- 172.21.15.59:0/561709644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53701f7440 con 0x7f535c0039e0 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f536effd640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53700c6720 0x7f537015cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f536ffff640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53700c6ba0 0x7f537015d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f536f7fe640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 0x7f5370160e50 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f5360009c90 tx=0x7f5360004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f5367fff640 1 -- 172.21.15.59:0/561709644 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360039050 con 0x7f535c0039e0 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f5367fff640 1 -- 172.21.15.59:0/561709644 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5360045070 con 0x7f535c0039e0 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f5376066640 1 -- 172.21.15.59:0/561709644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53701f7720 con 0x7f535c0039e0 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f5367fff640 1 -- 172.21.15.59:0/561709644 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360009860 con 0x7f535c0039e0 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.277+0000 7f5376066640 1 -- 172.21.15.59:0/561709644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53701f7ce0 con 0x7f535c0039e0 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.278+0000 7f5376066640 1 -- 172.21.15.59:0/561709644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f537004f330 con 0x7f535c0039e0 2024-03-06T07:29:35.282 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.279+0000 7f5367fff640 1 -- 172.21.15.59:0/561709644 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f536004a460 con 0x7f535c0039e0 2024-03-06T07:29:35.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.279+0000 7f5367fff640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f535006b4d0 0x7f535006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:35.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.279+0000 7f5367fff640 1 -- 172.21.15.59:0/561709644 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53600ac0a0 con 0x7f535c0039e0 2024-03-06T07:29:35.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.279+0000 7f536effd640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f535006b4d0 0x7f535006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:35.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.280+0000 7f536effd640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f535006b4d0 0x7f535006d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f53700ce140 tx=0x7f5358009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:35.285 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.284+0000 7f5367fff640 1 -- 172.21.15.59:0/561709644 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f536004f020 con 0x7f535c0039e0 2024-03-06T07:29:35.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.503+0000 7f5376066640 1 -- 172.21.15.59:0/561709644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53700b1820 con 0x7f535c0039e0 2024-03-06T07:29:35.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.503+0000 7f5367fff640 1 -- 172.21.15.59:0/561709644 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f536007af20 con 0x7f535c0039e0 2024-03-06T07:29:35.505 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:35.505 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:35.506 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:35.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 -- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f535006b4d0 msgr2=0x7f535006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f535006b4d0 0x7f535006d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f53700ce140 tx=0x7f5358009040 comp rx=0 tx=0).stop 2024-03-06T07:29:35.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 -- 172.21.15.59:0/561709644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 msgr2=0x7f5370160e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 0x7f5370160e50 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f5360009c90 tx=0x7f5360004730 comp rx=0 tx=0).stop 2024-03-06T07:29:35.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 -- 172.21.15.59:0/561709644 shutdown_connections 2024-03-06T07:29:35.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f535006b4d0 0x7f535006d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53700c6ba0 0x7f537015d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53700c6720 0x7f537015cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 --2- 172.21.15.59:0/561709644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f535c0039e0 0x7f5370160e50 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 -- 172.21.15.59:0/561709644 >> 172.21.15.59:0/561709644 conn(0x7f537009a9f0 msgr2=0x7f537006eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 -- 172.21.15.59:0/561709644 shutdown_connections 2024-03-06T07:29:35.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:35.505+0000 7f5376066640 1 -- 172.21.15.59:0/561709644 wait complete. 2024-03-06T07:29:35.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:35.719 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:35.921 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:36.720 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:36.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.821+0000 7f3035381640 1 Processor -- start 2024-03-06T07:29:36.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.821+0000 7f3035381640 1 -- start start 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.821+0000 7f3035381640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30300b3060 0x7f303015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.821+0000 7f3035381640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f303015bf30 0x7f303015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.821+0000 7f3035381640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f303015f2d0 con 0x7f303013dc10 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.821+0000 7f3035381640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30300cf3f0 con 0x7f30300b3060 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.821+0000 7f3035381640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303015b210 con 0x7f303015bf30 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.821+0000 7f3027fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30300b3060 0x7f303015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.822+0000 7f3027fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f30300b3060 0x7f303015e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55372/0 (socket says 172.21.15.59:55372) 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.822+0000 7f3027fff640 1 -- 172.21.15.59:0/3353749296 learned_addr learned my addr 172.21.15.59:0/3353749296 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.822+0000 7f302f7fe640 1 --2- 172.21.15.59:0/3353749296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f303015bf30 0x7f303015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.822+0000 7f3027fff640 1 -- 172.21.15.59:0/3353749296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f303015bf30 msgr2=0x7f303015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.822+0000 7f3027fff640 1 --2- 172.21.15.59:0/3353749296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f303015bf30 0x7f303015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.822+0000 7f3027fff640 1 -- 172.21.15.59:0/3353749296 >> v1:172.21.15.164:6789/0 conn(0x7f303013dc10 legacy=0x7f30300b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.822+0000 7f3027fff640 1 -- 172.21.15.59:0/3353749296 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303015f580 con 0x7f30300b3060 2024-03-06T07:29:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.823+0000 7f3027fff640 1 --2- 172.21.15.59:0/3353749296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30300b3060 0x7f303015e850 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f3018009870 tx=0x7f301802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c369aff391e577 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.823+0000 7f302e7fc640 1 -- 172.21.15.59:0/3353749296 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30180317c0 con 0x7f30300b3060 2024-03-06T07:29:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.823+0000 7f302e7fc640 1 -- 172.21.15.59:0/3353749296 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3018037070 con 0x7f30300b3060 2024-03-06T07:29:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.823+0000 7f302e7fc640 1 -- 172.21.15.59:0/3353749296 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018031b80 con 0x7f30300b3060 2024-03-06T07:29:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.824+0000 7f3035381640 1 -- 172.21.15.59:0/3353749296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30300b3060 msgr2=0x7f303015e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.824+0000 7f3035381640 1 --2- 172.21.15.59:0/3353749296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30300b3060 0x7f303015e850 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f3018009870 tx=0x7f301802b860 comp rx=0 tx=0).stop 2024-03-06T07:29:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.824+0000 7f3035381640 1 -- 172.21.15.59:0/3353749296 shutdown_connections 2024-03-06T07:29:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.824+0000 7f3035381640 1 --2- 172.21.15.59:0/3353749296 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f303015bf30 0x7f303015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.824+0000 7f3035381640 1 --2- 172.21.15.59:0/3353749296 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30300b3060 0x7f303015e850 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.824+0000 7f3035381640 1 -- 172.21.15.59:0/3353749296 >> 172.21.15.59:0/3353749296 conn(0x7f30300cf8a0 msgr2=0x7f30300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.824+0000 7f3035381640 1 -- 172.21.15.59:0/3353749296 shutdown_connections 2024-03-06T07:29:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.824+0000 7f3035381640 1 -- 172.21.15.59:0/3353749296 wait complete. 2024-03-06T07:29:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.825+0000 7f3035381640 1 Processor -- start 2024-03-06T07:29:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.825+0000 7f3035381640 1 -- start start 2024-03-06T07:29:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3035381640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30300b3060 0x7f30301f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3035381640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303013dc10 0x7f3030140340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3035381640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f303015bf30 0x7f3030140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3035381640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303013dff0 con 0x7f303013dc10 2024-03-06T07:29:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3035381640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303015b730 con 0x7f30300b3060 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3035381640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30300708d0 con 0x7f303015bf30 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3027fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303013dc10 0x7f3030140340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3027fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303013dc10 0x7f3030140340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55374/0 (socket says 172.21.15.59:55374) 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3027fff640 1 -- 172.21.15.59:0/2214841688 learned_addr learned my addr 172.21.15.59:0/2214841688 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f302f7fe640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f303015bf30 0x7f3030140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3027fff640 1 -- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f303015bf30 msgr2=0x7f3030140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f302effd640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30300b3060 0x7f30301f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3027fff640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f303015bf30 0x7f3030140880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3027fff640 1 -- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30300b3060 msgr2=0x7f30301f1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3027fff640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30300b3060 0x7f30301f1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3027fff640 1 -- 172.21.15.59:0/2214841688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3030153850 con 0x7f303013dc10 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.826+0000 7f3027fff640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303013dc10 0x7f3030140340 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f3018009870 tx=0x7f30180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.827+0000 7f30277fe640 1 -- 172.21.15.59:0/2214841688 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018032650 con 0x7f303013dc10 2024-03-06T07:29:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.827+0000 7f30277fe640 1 -- 172.21.15.59:0/2214841688 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30180327f0 con 0x7f303013dc10 2024-03-06T07:29:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.827+0000 7f3035381640 1 -- 172.21.15.59:0/2214841688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30301f2310 con 0x7f303013dc10 2024-03-06T07:29:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.827+0000 7f30277fe640 1 -- 172.21.15.59:0/2214841688 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018032be0 con 0x7f303013dc10 2024-03-06T07:29:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.827+0000 7f3035381640 1 -- 172.21.15.59:0/2214841688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30301f25b0 con 0x7f303013dc10 2024-03-06T07:29:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.828+0000 7f3035381640 1 -- 172.21.15.59:0/2214841688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30300b13f0 con 0x7f303013dc10 2024-03-06T07:29:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.828+0000 7f30277fe640 1 -- 172.21.15.59:0/2214841688 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f301804c060 con 0x7f303013dc10 2024-03-06T07:29:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.829+0000 7f30277fe640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f301006b4d0 0x7f301006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.829+0000 7f30277fe640 1 -- 172.21.15.59:0/2214841688 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30180ab9a0 con 0x7f303013dc10 2024-03-06T07:29:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.829+0000 7f302effd640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f301006b4d0 0x7f301006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.829+0000 7f302effd640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f301006b4d0 0x7f301006d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f30300b2b00 tx=0x7f3020009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:36.833+0000 7f30277fe640 1 -- 172.21.15.59:0/2214841688 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30300b13f0 con 0x7f303013dc10 2024-03-06T07:29:37.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.058+0000 7f3035381640 1 -- 172.21.15.59:0/2214841688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f303004f330 con 0x7f303013dc10 2024-03-06T07:29:37.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.059+0000 7f30277fe640 1 -- 172.21.15.59:0/2214841688 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f3030153850 con 0x7f303013dc10 2024-03-06T07:29:37.061 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:37.061 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:37.061 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.060+0000 7f3035381640 1 -- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f301006b4d0 msgr2=0x7f301006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.060+0000 7f3035381640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f301006b4d0 0x7f301006d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f30300b2b00 tx=0x7f3020009040 comp rx=0 tx=0).stop 2024-03-06T07:29:37.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.061+0000 7f3035381640 1 -- 172.21.15.59:0/2214841688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303013dc10 msgr2=0x7f3030140340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.061+0000 7f3035381640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303013dc10 0x7f3030140340 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f3018009870 tx=0x7f30180046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.061+0000 7f3035381640 1 -- 172.21.15.59:0/2214841688 shutdown_connections 2024-03-06T07:29:37.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.061+0000 7f3035381640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f301006b4d0 0x7f301006d990 secure :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f30300b2b00 tx=0x7f3020009040 comp rx=0 tx=0).stop 2024-03-06T07:29:37.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.061+0000 7f3035381640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f303015bf30 0x7f3030140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.061+0000 7f3035381640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303013dc10 0x7f3030140340 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.061+0000 7f3035381640 1 --2- 172.21.15.59:0/2214841688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30300b3060 0x7f30301f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.061+0000 7f3035381640 1 -- 172.21.15.59:0/2214841688 >> 172.21.15.59:0/2214841688 conn(0x7f30300cf8a0 msgr2=0x7f30300cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.061+0000 7f3035381640 1 -- 172.21.15.59:0/2214841688 shutdown_connections 2024-03-06T07:29:37.063 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:37.061+0000 7f3035381640 1 -- 172.21.15.59:0/2214841688 wait complete. 2024-03-06T07:29:37.275 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:37.275 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:37.923 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:38.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:38.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.376+0000 7f45dfd6c640 1 Processor -- start 2024-03-06T07:29:38.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.377+0000 7f45dfd6c640 1 -- start start 2024-03-06T07:29:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.377+0000 7f45dfd6c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d815dfe0 0x7f45d80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.377+0000 7f45dfd6c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d80b3430 con 0x7f45d80cf830 2024-03-06T07:29:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.377+0000 7f45dfd6c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d80b3710 con 0x7f45d80d29c0 2024-03-06T07:29:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.377+0000 7f45dfd6c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d80b31e0 con 0x7f45d815dfe0 2024-03-06T07:29:38.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.377+0000 7f45dd307640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f45d80cf830 0x7f45d80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51726/0 (socket says 172.21.15.59:51726) 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.377+0000 7f45dd307640 1 -- 172.21.15.59:0/3086809770 learned_addr learned my addr 172.21.15.59:0/3086809770 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45ddb08640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d815dfe0 0x7f45d80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554412314 0 0) 0x7f45d80b3430 con 0x7f45d80cf830 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c4003520 con 0x7f45d80cf830 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1057938917 0 0) 0x7f45c4003520 con 0x7f45d80cf830 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d815dfe0 msgr2=0x7f45d80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d815dfe0 0x7f45d80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 >> v1:172.21.15.164:6789/0 conn(0x7f45d80d29c0 legacy=0x7f45d80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d80b39f0 con 0x7f45d80cf830 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f45d4001670 con 0x7f45d80cf830 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 >> v1:172.21.15.59:6789/0 conn(0x7f45d80cf830 legacy=0x7f45d80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 0x7f45c4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45c4004800 0x7f45c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d815dfe0 0x7f45c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d80b31e0 con 0x7f45c40039e0 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c400baa0 con 0x7f45c4004800 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c400bca0 con 0x7f45d815dfe0 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45ddb08640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 0x7f45c4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45dd307640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45c4004800 0x7f45c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45de309640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d815dfe0 0x7f45c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45ddb08640 1 -- 172.21.15.59:0/3086809770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d815dfe0 msgr2=0x7f45c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45ddb08640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d815dfe0 0x7f45c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45ddb08640 1 -- 172.21.15.59:0/3086809770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45c4004800 msgr2=0x7f45c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45ddb08640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45c4004800 0x7f45c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.378+0000 7f45ddb08640 1 -- 172.21.15.59:0/3086809770 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c4003520 con 0x7f45c40039e0 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.379+0000 7f45ddb08640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 0x7f45c4003e00 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009a90 tx=0x7f45c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8697770469de25f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.379+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc039070 con 0x7f45c40039e0 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.379+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45cc02c680 con 0x7f45c40039e0 2024-03-06T07:29:38.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.379+0000 7f45dcb06640 1 -- 172.21.15.59:0/3086809770 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc009690 con 0x7f45c40039e0 2024-03-06T07:29:38.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.380+0000 7f45dfd6c640 1 -- 172.21.15.59:0/3086809770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 msgr2=0x7f45c4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.380+0000 7f45dfd6c640 1 --2- 172.21.15.59:0/3086809770 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 0x7f45c4003e00 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009a90 tx=0x7f45c4005830 comp rx=0 tx=0).stop 2024-03-06T07:29:38.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.380+0000 7f45de309640 1 -- 172.21.15.59:0/3086809770 reap_dead start 2024-03-06T07:29:38.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.380+0000 7f45dfd6c640 1 -- 172.21.15.59:0/3086809770 shutdown_connections 2024-03-06T07:29:38.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.380+0000 7f45dfd6c640 1 -- 172.21.15.59:0/3086809770 >> 172.21.15.59:0/3086809770 conn(0x7f45d80d3cd0 msgr2=0x7f45d80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.380+0000 7f45dfd6c640 1 -- 172.21.15.59:0/3086809770 shutdown_connections 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.380+0000 7f45dfd6c640 1 -- 172.21.15.59:0/3086809770 wait complete. 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.381+0000 7f45dfd6c640 1 Processor -- start 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.381+0000 7f45dfd6c640 1 -- start start 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45dfd6c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 0x7f45d81f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45dfd6c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d80cd4c0 0x7f45d80b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45dfd6c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d80b40f0 0x7f45d80b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45dfd6c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d815e3c0 con 0x7f45c40039e0 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45dfd6c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d80b3d60 con 0x7f45d80cd4c0 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45dfd6c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d815e540 con 0x7f45d80b40f0 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45ddb08640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 0x7f45d81f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45ddb08640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 0x7f45d81f31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55400/0 (socket says 172.21.15.59:55400) 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45ddb08640 1 -- 172.21.15.59:0/1034113407 learned_addr learned my addr 172.21.15.59:0/1034113407 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45dd307640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d80cd4c0 0x7f45d80b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45de309640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d80b40f0 0x7f45d80b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45ddb08640 1 -- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d80b40f0 msgr2=0x7f45d80b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45ddb08640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d80b40f0 0x7f45d80b45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45ddb08640 1 -- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d80cd4c0 msgr2=0x7f45d80b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45ddb08640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d80cd4c0 0x7f45d80b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45ddb08640 1 -- 172.21.15.59:0/1034113407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d80b71d0 con 0x7f45c40039e0 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45ddb08640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 0x7f45d81f31b0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f45cc000c00 tx=0x7f45cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.382+0000 7f45ca7fc640 1 -- 172.21.15.59:0/1034113407 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc042020 con 0x7f45c40039e0 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.383+0000 7f45ca7fc640 1 -- 172.21.15.59:0/1034113407 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45cc03a040 con 0x7f45c40039e0 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.383+0000 7f45dfd6c640 1 -- 172.21.15.59:0/1034113407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d80b74b0 con 0x7f45c40039e0 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.383+0000 7f45ca7fc640 1 -- 172.21.15.59:0/1034113407 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc039070 con 0x7f45c40039e0 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.383+0000 7f45dfd6c640 1 -- 172.21.15.59:0/1034113407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d80b7a70 con 0x7f45c40039e0 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.384+0000 7f45dfd6c640 1 -- 172.21.15.59:0/1034113407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d804f330 con 0x7f45c40039e0 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.384+0000 7f45ca7fc640 1 -- 172.21.15.59:0/1034113407 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45cc045020 con 0x7f45c40039e0 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.385+0000 7f45ca7fc640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45bc06b500 0x7f45bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.385+0000 7f45ca7fc640 1 -- 172.21.15.59:0/1034113407 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f45cc0ab640 con 0x7f45c40039e0 2024-03-06T07:29:38.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.385+0000 7f45dd307640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45bc06b500 0x7f45bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.386+0000 7f45dd307640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45bc06b500 0x7f45bc06d9c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f45d80cd8c0 tx=0x7f45d4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.389+0000 7f45ca7fc640 1 -- 172.21.15.59:0/1034113407 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45cc04c070 con 0x7f45c40039e0 2024-03-06T07:29:38.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.611+0000 7f45dfd6c640 1 -- 172.21.15.59:0/1034113407 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45d80d49b0 con 0x7f45c40039e0 2024-03-06T07:29:38.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.612+0000 7f45ca7fc640 1 -- 172.21.15.59:0/1034113407 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f45cc07a400 con 0x7f45c40039e0 2024-03-06T07:29:38.614 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:38.614 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:38.614 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 -- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45bc06b500 msgr2=0x7f45bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45bc06b500 0x7f45bc06d9c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f45d80cd8c0 tx=0x7f45d4009040 comp rx=0 tx=0).stop 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 -- 172.21.15.59:0/1034113407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 msgr2=0x7f45d81f31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 0x7f45d81f31b0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f45cc000c00 tx=0x7f45cc004730 comp rx=0 tx=0).stop 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 -- 172.21.15.59:0/1034113407 shutdown_connections 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45bc06b500 0x7f45bc06d9c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45d80b40f0 0x7f45d80b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d80cd4c0 0x7f45d80b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 --2- 172.21.15.59:0/1034113407 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c40039e0 0x7f45d81f31b0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 -- 172.21.15.59:0/1034113407 >> 172.21.15.59:0/1034113407 conn(0x7f45d80d3cd0 msgr2=0x7f45d80b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 -- 172.21.15.59:0/1034113407 shutdown_connections 2024-03-06T07:29:38.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:38.614+0000 7f45dfd6c640 1 -- 172.21.15.59:0/1034113407 wait complete. 2024-03-06T07:29:38.827 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:38.827 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:39.828 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:39.925 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:39.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.927+0000 7f29b437a640 1 Processor -- start 2024-03-06T07:29:39.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.928+0000 7f29b437a640 1 -- start start 2024-03-06T07:29:39.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.928+0000 7f29b437a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f29ac15dfe0 0x7f29ac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.928+0000 7f29b437a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ac0b3430 con 0x7f29ac0d29c0 2024-03-06T07:29:39.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.928+0000 7f29b437a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29ac0b3710 con 0x7f29ac0cf830 2024-03-06T07:29:39.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.928+0000 7f29b437a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ac0b31e0 con 0x7f29ac15dfe0 2024-03-06T07:29:39.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.929+0000 7f29b2116640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f29ac15dfe0 0x7f29ac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.929+0000 7f29b2917640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f29ac0d29c0 0x7f29ac0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51730/0 (socket says 172.21.15.59:51730) 2024-03-06T07:29:39.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.929+0000 7f29b2917640 1 -- 172.21.15.59:0/1738988110 learned_addr learned my addr 172.21.15.59:0/1738988110 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:39.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.929+0000 7f29b2116640 1 -- 172.21.15.59:0/1738988110 >> v1:172.21.15.164:6789/0 conn(0x7f29ac0cf830 legacy=0x7f29ac0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.929+0000 7f29b2116640 1 -- 172.21.15.59:0/1738988110 >> v1:172.21.15.59:6789/0 conn(0x7f29ac0d29c0 legacy=0x7f29ac0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.929+0000 7f29b2116640 1 -- 172.21.15.59:0/1738988110 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac0b39f0 con 0x7f29ac15dfe0 2024-03-06T07:29:39.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.929+0000 7f29b1114640 1 -- 172.21.15.59:0/1738988110 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649347701 0 0) 0x7f29ac0b3430 con 0x7f29ac0d29c0 2024-03-06T07:29:39.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.930+0000 7f29b2116640 1 --2- 172.21.15.59:0/1738988110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29ac15dfe0 0x7f29ac0cf2f0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0b3360 tx=0x7f29a802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39fa2032c0c82683 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.930+0000 7f29b1114640 1 -- 172.21.15.59:0/1738988110 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8034070 con 0x7f29ac15dfe0 2024-03-06T07:29:39.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.930+0000 7f29b1114640 1 -- 172.21.15.59:0/1738988110 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a8030610 con 0x7f29ac15dfe0 2024-03-06T07:29:39.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.930+0000 7f29b1114640 1 -- 172.21.15.59:0/1738988110 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a80309e0 con 0x7f29ac15dfe0 2024-03-06T07:29:39.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.930+0000 7f29b437a640 1 -- 172.21.15.59:0/1738988110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29ac15dfe0 msgr2=0x7f29ac0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.930+0000 7f29b437a640 1 --2- 172.21.15.59:0/1738988110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29ac15dfe0 0x7f29ac0cf2f0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0b3360 tx=0x7f29a802b960 comp rx=0 tx=0).stop 2024-03-06T07:29:39.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.930+0000 7f29b437a640 1 -- 172.21.15.59:0/1738988110 shutdown_connections 2024-03-06T07:29:39.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.930+0000 7f29b437a640 1 --2- 172.21.15.59:0/1738988110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29ac15dfe0 0x7f29ac0cf2f0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.931+0000 7f29b437a640 1 -- 172.21.15.59:0/1738988110 >> 172.21.15.59:0/1738988110 conn(0x7f29ac0d3cd0 msgr2=0x7f29ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.931+0000 7f29b437a640 1 -- 172.21.15.59:0/1738988110 shutdown_connections 2024-03-06T07:29:39.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.931+0000 7f29b437a640 1 -- 172.21.15.59:0/1738988110 wait complete. 2024-03-06T07:29:39.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.932+0000 7f29b437a640 1 Processor -- start 2024-03-06T07:29:39.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.932+0000 7f29b437a640 1 -- start start 2024-03-06T07:29:39.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.932+0000 7f29b437a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29ac0cf830 0x7f29ac0b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.932+0000 7f29b437a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29ac0d29c0 0x7f29ac1f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.932+0000 7f29b437a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29ac15dfe0 0x7f29ac0cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.932+0000 7f29b437a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac15e3c0 con 0x7f29ac0cf830 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.932+0000 7f29b437a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac0b3d60 con 0x7f29ac15dfe0 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.932+0000 7f29b437a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac15e540 con 0x7f29ac0d29c0 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b2116640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29ac0cf830 0x7f29ac0b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b2116640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29ac0cf830 0x7f29ac0b0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55418/0 (socket says 172.21.15.59:55418) 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b2116640 1 -- 172.21.15.59:0/750403779 learned_addr learned my addr 172.21.15.59:0/750403779 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b2917640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29ac15dfe0 0x7f29ac0cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b1915640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29ac0d29c0 0x7f29ac1f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b2116640 1 -- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29ac0d29c0 msgr2=0x7f29ac1f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b2116640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29ac0d29c0 0x7f29ac1f3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b2116640 1 -- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29ac15dfe0 msgr2=0x7f29ac0cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b2116640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29ac15dfe0 0x7f29ac0cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b2116640 1 -- 172.21.15.59:0/750403779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac0b1880 con 0x7f29ac0cf830 2024-03-06T07:29:39.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b2116640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29ac0cf830 0x7f29ac0b0e50 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f29a802bf70 tx=0x7f29a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29a2ffd640 1 -- 172.21.15.59:0/750403779 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8034040 con 0x7f29ac0cf830 2024-03-06T07:29:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29a2ffd640 1 -- 172.21.15.59:0/750403779 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a803ace0 con 0x7f29ac0cf830 2024-03-06T07:29:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b437a640 1 -- 172.21.15.59:0/750403779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ac0b15a0 con 0x7f29ac0cf830 2024-03-06T07:29:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29a2ffd640 1 -- 172.21.15.59:0/750403779 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8044630 con 0x7f29ac0cf830 2024-03-06T07:29:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.933+0000 7f29b437a640 1 -- 172.21.15.59:0/750403779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ac0cb120 con 0x7f29ac0cf830 2024-03-06T07:29:39.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.934+0000 7f29b437a640 1 -- 172.21.15.59:0/750403779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ac04f330 con 0x7f29ac0cf830 2024-03-06T07:29:39.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.935+0000 7f29a2ffd640 1 -- 172.21.15.59:0/750403779 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29a8037070 con 0x7f29ac0cf830 2024-03-06T07:29:39.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.935+0000 7f29a2ffd640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f299006b500 0x7f299006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.936+0000 7f29a2ffd640 1 -- 172.21.15.59:0/750403779 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f29a80ac260 con 0x7f29ac0cf830 2024-03-06T07:29:39.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.936+0000 7f29b1915640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f299006b500 0x7f299006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.936+0000 7f29b1915640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f299006b500 0x7f299006d9c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f29ac15b1c0 tx=0x7f299c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:39.940+0000 7f29a2ffd640 1 -- 172.21.15.59:0/750403779 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29a803f070 con 0x7f29ac0cf830 2024-03-06T07:29:40.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.161+0000 7f29b437a640 1 -- 172.21.15.59:0/750403779 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29ac1f3790 con 0x7f29ac0cf830 2024-03-06T07:29:40.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.161+0000 7f29a2ffd640 1 -- 172.21.15.59:0/750403779 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f29a807b0a0 con 0x7f29ac0cf830 2024-03-06T07:29:40.163 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:40.163 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:40.163 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:40.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 -- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f299006b500 msgr2=0x7f299006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f299006b500 0x7f299006d9c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f29ac15b1c0 tx=0x7f299c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:40.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 -- 172.21.15.59:0/750403779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29ac0cf830 msgr2=0x7f29ac0b0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29ac0cf830 0x7f29ac0b0e50 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f29a802bf70 tx=0x7f29a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 -- 172.21.15.59:0/750403779 shutdown_connections 2024-03-06T07:29:40.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f299006b500 0x7f299006d9c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29ac15dfe0 0x7f29ac0cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29ac0d29c0 0x7f29ac1f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 --2- 172.21.15.59:0/750403779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29ac0cf830 0x7f29ac0b0e50 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 -- 172.21.15.59:0/750403779 >> 172.21.15.59:0/750403779 conn(0x7f29ac0d3cd0 msgr2=0x7f29ac0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 -- 172.21.15.59:0/750403779 shutdown_connections 2024-03-06T07:29:40.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:40.163+0000 7f29b437a640 1 -- 172.21.15.59:0/750403779 wait complete. 2024-03-06T07:29:40.377 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:40.377 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:41.378 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:41.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.478+0000 7fdf8c617640 1 Processor -- start 2024-03-06T07:29:41.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.478+0000 7fdf8c617640 1 -- start start 2024-03-06T07:29:41.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf8c617640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf840d22b0 0x7fdf840d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf8c617640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf840b41f0 0x7fdf840b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf8c617640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf840b3430 con 0x7fdf8415dfe0 2024-03-06T07:29:41.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf8c617640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf840d2c50 con 0x7fdf840d22b0 2024-03-06T07:29:41.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf8c617640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840b31e0 con 0x7fdf840b41f0 2024-03-06T07:29:41.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf89bb2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf840d22b0 0x7fdf840d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf89bb2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf840d22b0 0x7fdf840d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55422/0 (socket says 172.21.15.59:55422) 2024-03-06T07:29:41.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf89bb2640 1 -- 172.21.15.59:0/2263947577 learned_addr learned my addr 172.21.15.59:0/2263947577 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:41.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf8abb4640 1 --2- 172.21.15.59:0/2263947577 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf840b41f0 0x7fdf840b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf89bb2640 1 -- 172.21.15.59:0/2263947577 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf840b41f0 msgr2=0x7fdf840b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf89bb2640 1 --2- 172.21.15.59:0/2263947577 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf840b41f0 0x7fdf840b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf89bb2640 1 -- 172.21.15.59:0/2263947577 >> v1:172.21.15.164:6789/0 conn(0x7fdf8415dfe0 legacy=0x7fdf840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.479+0000 7fdf89bb2640 1 -- 172.21.15.59:0/2263947577 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf840b3710 con 0x7fdf840d22b0 2024-03-06T07:29:41.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.480+0000 7fdf89bb2640 1 --2- 172.21.15.59:0/2263947577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf840d22b0 0x7fdf840d2710 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009870 tx=0x7fdf7802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a690e7b84c81100 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.480+0000 7fdf893b1640 1 -- 172.21.15.59:0/2263947577 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf78035020 con 0x7fdf840d22b0 2024-03-06T07:29:41.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.480+0000 7fdf893b1640 1 -- 172.21.15.59:0/2263947577 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf78031640 con 0x7fdf840d22b0 2024-03-06T07:29:41.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.480+0000 7fdf893b1640 1 -- 172.21.15.59:0/2263947577 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7802d040 con 0x7fdf840d22b0 2024-03-06T07:29:41.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.481+0000 7fdf8c617640 1 -- 172.21.15.59:0/2263947577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf840d22b0 msgr2=0x7fdf840d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.481+0000 7fdf8c617640 1 --2- 172.21.15.59:0/2263947577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf840d22b0 0x7fdf840d2710 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009870 tx=0x7fdf7802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.481+0000 7fdf8c617640 1 -- 172.21.15.59:0/2263947577 shutdown_connections 2024-03-06T07:29:41.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.481+0000 7fdf8c617640 1 --2- 172.21.15.59:0/2263947577 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf840b41f0 0x7fdf840b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.481+0000 7fdf8c617640 1 --2- 172.21.15.59:0/2263947577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf840d22b0 0x7fdf840d2710 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.481+0000 7fdf8c617640 1 -- 172.21.15.59:0/2263947577 >> 172.21.15.59:0/2263947577 conn(0x7fdf840d3cd0 msgr2=0x7fdf840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.481+0000 7fdf8c617640 1 -- 172.21.15.59:0/2263947577 shutdown_connections 2024-03-06T07:29:41.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.481+0000 7fdf8c617640 1 -- 172.21.15.59:0/2263947577 wait complete. 2024-03-06T07:29:41.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.482+0000 7fdf8c617640 1 Processor -- start 2024-03-06T07:29:41.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.482+0000 7fdf8c617640 1 -- start start 2024-03-06T07:29:41.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8c617640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf840b41f0 0x7fdf84152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8c617640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf840d22b0 0x7fdf840b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8c617640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf8415dfe0 0x7fdf84160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8c617640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840cfa00 con 0x7fdf8415dfe0 2024-03-06T07:29:41.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8c617640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840b3d60 con 0x7fdf840d22b0 2024-03-06T07:29:41.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8c617640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840d1c70 con 0x7fdf840b41f0 2024-03-06T07:29:41.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8abb4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf8415dfe0 0x7fdf84160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8abb4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf8415dfe0 0x7fdf84160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55438/0 (socket says 172.21.15.59:55438) 2024-03-06T07:29:41.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8abb4640 1 -- 172.21.15.59:0/134244669 learned_addr learned my addr 172.21.15.59:0/134244669 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8a3b3640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf840b41f0 0x7fdf84152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf89bb2640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf840d22b0 0x7fdf840b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8abb4640 1 -- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf840b41f0 msgr2=0x7fdf84152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8abb4640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf840b41f0 0x7fdf84152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8abb4640 1 -- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf840d22b0 msgr2=0x7fdf840b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8abb4640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf840d22b0 0x7fdf840b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.483+0000 7fdf8abb4640 1 -- 172.21.15.59:0/134244669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf840b7210 con 0x7fdf8415dfe0 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.484+0000 7fdf8abb4640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf8415dfe0 0x7fdf84160cb0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c00cd70 tx=0x7fdf7c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.484+0000 7fdf76ffd640 1 -- 172.21.15.59:0/134244669 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c003b90 con 0x7fdf8415dfe0 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.484+0000 7fdf76ffd640 1 -- 172.21.15.59:0/134244669 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7c003d30 con 0x7fdf8415dfe0 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.484+0000 7fdf8c617640 1 -- 172.21.15.59:0/134244669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf840b74f0 con 0x7fdf8415dfe0 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.484+0000 7fdf76ffd640 1 -- 172.21.15.59:0/134244669 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c004240 con 0x7fdf8415dfe0 2024-03-06T07:29:41.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.484+0000 7fdf8c617640 1 -- 172.21.15.59:0/134244669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf840b7ab0 con 0x7fdf8415dfe0 2024-03-06T07:29:41.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.485+0000 7fdf8c617640 1 -- 172.21.15.59:0/134244669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8404f330 con 0x7fdf8415dfe0 2024-03-06T07:29:41.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.485+0000 7fdf76ffd640 1 -- 172.21.15.59:0/134244669 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf7c0043e0 con 0x7fdf8415dfe0 2024-03-06T07:29:41.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.486+0000 7fdf76ffd640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf6806b4d0 0x7fdf6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.486+0000 7fdf76ffd640 1 -- 172.21.15.59:0/134244669 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdf7c08adc0 con 0x7fdf8415dfe0 2024-03-06T07:29:41.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.486+0000 7fdf8a3b3640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf6806b4d0 0x7fdf6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.486+0000 7fdf8a3b3640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf6806b4d0 0x7fdf6806d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdf80006fd0 tx=0x7fdf80008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.494+0000 7fdf76ffd640 1 -- 172.21.15.59:0/134244669 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf7c01d340 con 0x7fdf8415dfe0 2024-03-06T07:29:41.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.712+0000 7fdf8c617640 1 -- 172.21.15.59:0/134244669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf840ceb20 con 0x7fdf8415dfe0 2024-03-06T07:29:41.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.713+0000 7fdf76ffd640 1 -- 172.21.15.59:0/134244669 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7fdf7c059c00 con 0x7fdf8415dfe0 2024-03-06T07:29:41.715 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:41.715 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 -- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf6806b4d0 msgr2=0x7fdf6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf6806b4d0 0x7fdf6806d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdf80006fd0 tx=0x7fdf80008040 comp rx=0 tx=0).stop 2024-03-06T07:29:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 -- 172.21.15.59:0/134244669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf8415dfe0 msgr2=0x7fdf84160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf8415dfe0 0x7fdf84160cb0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c00cd70 tx=0x7fdf7c00a660 comp rx=0 tx=0).stop 2024-03-06T07:29:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 -- 172.21.15.59:0/134244669 shutdown_connections 2024-03-06T07:29:41.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf6806b4d0 0x7fdf6806d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf8415dfe0 0x7fdf84160cb0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf840d22b0 0x7fdf840b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 --2- 172.21.15.59:0/134244669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf840b41f0 0x7fdf84152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 -- 172.21.15.59:0/134244669 >> 172.21.15.59:0/134244669 conn(0x7fdf840d3cd0 msgr2=0x7fdf8406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 -- 172.21.15.59:0/134244669 shutdown_connections 2024-03-06T07:29:41.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:41.715+0000 7fdf8c617640 1 -- 172.21.15.59:0/134244669 wait complete. 2024-03-06T07:29:41.927 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:41.928 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:41.928 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:42.929 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:43.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.029+0000 7f7f56b3e640 1 Processor -- start 2024-03-06T07:29:43.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.029+0000 7f7f56b3e640 1 -- start start 2024-03-06T07:29:43.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.031+0000 7f7f56b3e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f5015fa10 0x7f7f500ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.031+0000 7f7f56b3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f500d16e0 0x7f7f500d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.031+0000 7f7f56b3e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f500cebe0 con 0x7f7f5015bdd0 2024-03-06T07:29:43.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.031+0000 7f7f56b3e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f500ceec0 con 0x7f7f5015fa10 2024-03-06T07:29:43.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.031+0000 7f7f56b3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500cf1a0 con 0x7f7f500d16e0 2024-03-06T07:29:43.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.031+0000 7f7f550db640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f500d16e0 0x7f7f500d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.031+0000 7f7f4ffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f5015fa10 0x7f7f500ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.031+0000 7f7f550db640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f500d16e0 0x7f7f500d1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:45612/0 (socket says 172.21.15.59:45612) 2024-03-06T07:29:43.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.031+0000 7f7f550db640 1 -- 172.21.15.59:0/2016128968 learned_addr learned my addr 172.21.15.59:0/2016128968 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:43.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.032+0000 7f7f4ffff640 1 -- 172.21.15.59:0/2016128968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f500d16e0 msgr2=0x7f7f500d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:43.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.032+0000 7f7f4ffff640 1 --2- 172.21.15.59:0/2016128968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f500d16e0 0x7f7f500d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.032+0000 7f7f4ffff640 1 -- 172.21.15.59:0/2016128968 >> v1:172.21.15.164:6789/0 conn(0x7f7f5015bdd0 legacy=0x7f7f500c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.032+0000 7f7f4ffff640 1 -- 172.21.15.59:0/2016128968 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f500cc8e0 con 0x7f7f5015fa10 2024-03-06T07:29:43.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.032+0000 7f7f550db640 1 --2- 172.21.15.59:0/2016128968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f500d16e0 0x7f7f500d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:43.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.032+0000 7f7f4ffff640 1 --2- 172.21.15.59:0/2016128968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5015fa10 0x7f7f500ce270 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0097a0 tx=0x7f7f3c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eb7fa9ce66f2652 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.033+0000 7f7f4f7fe640 1 -- 172.21.15.59:0/2016128968 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c0315a0 con 0x7f7f5015fa10 2024-03-06T07:29:43.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.033+0000 7f7f4f7fe640 1 -- 172.21.15.59:0/2016128968 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3c037070 con 0x7f7f5015fa10 2024-03-06T07:29:43.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.033+0000 7f7f4f7fe640 1 -- 172.21.15.59:0/2016128968 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c031960 con 0x7f7f5015fa10 2024-03-06T07:29:43.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.033+0000 7f7f56b3e640 1 -- 172.21.15.59:0/2016128968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5015fa10 msgr2=0x7f7f500ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.033+0000 7f7f56b3e640 1 --2- 172.21.15.59:0/2016128968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5015fa10 0x7f7f500ce270 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0097a0 tx=0x7f7f3c02b840 comp rx=0 tx=0).stop 2024-03-06T07:29:43.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.033+0000 7f7f56b3e640 1 -- 172.21.15.59:0/2016128968 shutdown_connections 2024-03-06T07:29:43.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.033+0000 7f7f56b3e640 1 --2- 172.21.15.59:0/2016128968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f500d16e0 0x7f7f500d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.033+0000 7f7f56b3e640 1 --2- 172.21.15.59:0/2016128968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5015fa10 0x7f7f500ce270 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.033+0000 7f7f56b3e640 1 -- 172.21.15.59:0/2016128968 >> 172.21.15.59:0/2016128968 conn(0x7f7f5009a9f0 msgr2=0x7f7f5009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.033+0000 7f7f56b3e640 1 -- 172.21.15.59:0/2016128968 shutdown_connections 2024-03-06T07:29:43.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.033+0000 7f7f56b3e640 1 -- 172.21.15.59:0/2016128968 wait complete. 2024-03-06T07:29:43.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.034+0000 7f7f56b3e640 1 Processor -- start 2024-03-06T07:29:43.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.034+0000 7f7f56b3e640 1 -- start start 2024-03-06T07:29:43.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.035+0000 7f7f56b3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f500d16e0 0x7f7f500ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.035+0000 7f7f56b3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5015bdd0 0x7f7f500b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.035+0000 7f7f56b3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5015fa10 0x7f7f50158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.035+0000 7f7f56b3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5015c1b0 con 0x7f7f5015fa10 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.035+0000 7f7f56b3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500cc5d0 con 0x7f7f500d16e0 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.035+0000 7f7f56b3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500ce7b0 con 0x7f7f5015bdd0 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.035+0000 7f7f550db640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5015fa10 0x7f7f50158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.035+0000 7f7f550db640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5015fa10 0x7f7f50158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55440/0 (socket says 172.21.15.59:55440) 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.035+0000 7f7f550db640 1 -- 172.21.15.59:0/4072330403 learned_addr learned my addr 172.21.15.59:0/4072330403 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f4ffff640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5015bdd0 0x7f7f500b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f548da640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f500d16e0 0x7f7f500ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f550db640 1 -- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5015bdd0 msgr2=0x7f7f500b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f550db640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5015bdd0 0x7f7f500b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f550db640 1 -- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f500d16e0 msgr2=0x7f7f500ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f550db640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f500d16e0 0x7f7f500ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f550db640 1 -- 172.21.15.59:0/4072330403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f500cac10 con 0x7f7f5015fa10 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f550db640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5015fa10 0x7f7f50158100 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800efc0 tx=0x7f7f4800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f4d7fa640 1 -- 172.21.15.59:0/4072330403 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4800cbe0 con 0x7f7f5015fa10 2024-03-06T07:29:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f4d7fa640 1 -- 172.21.15.59:0/4072330403 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4800cd80 con 0x7f7f5015fa10 2024-03-06T07:29:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f56b3e640 1 -- 172.21.15.59:0/4072330403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f500c8bf0 con 0x7f7f5015fa10 2024-03-06T07:29:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f4d7fa640 1 -- 172.21.15.59:0/4072330403 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48003bb0 con 0x7f7f5015fa10 2024-03-06T07:29:43.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.036+0000 7f7f56b3e640 1 -- 172.21.15.59:0/4072330403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f500c9430 con 0x7f7f5015fa10 2024-03-06T07:29:43.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.037+0000 7f7f56b3e640 1 -- 172.21.15.59:0/4072330403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f500c8ed0 con 0x7f7f5015fa10 2024-03-06T07:29:43.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.038+0000 7f7f4d7fa640 1 -- 172.21.15.59:0/4072330403 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f4801b030 con 0x7f7f5015fa10 2024-03-06T07:29:43.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.038+0000 7f7f4d7fa640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f3406b4d0 0x7f7f3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:43.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.038+0000 7f7f4d7fa640 1 -- 172.21.15.59:0/4072330403 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7f4808a620 con 0x7f7f5015fa10 2024-03-06T07:29:43.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.038+0000 7f7f548da640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f3406b4d0 0x7f7f3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:43.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.039+0000 7f7f548da640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f3406b4d0 0x7f7f3406d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7f440015a0 tx=0x7f7f44009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:43.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.046+0000 7f7f4d7fa640 1 -- 172.21.15.59:0/4072330403 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f500c8ed0 con 0x7f7f5015fa10 2024-03-06T07:29:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.429+0000 7f7f56b3e640 1 -- 172.21.15.59:0/4072330403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f5015c6a0 con 0x7f7f5015fa10 2024-03-06T07:29:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.430+0000 7f7f4d7fa640 1 -- 172.21.15.59:0/4072330403 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f7f5015c6a0 con 0x7f7f5015fa10 2024-03-06T07:29:43.432 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:43.432 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 -- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f3406b4d0 msgr2=0x7f7f3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f3406b4d0 0x7f7f3406d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7f440015a0 tx=0x7f7f44009040 comp rx=0 tx=0).stop 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 -- 172.21.15.59:0/4072330403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5015fa10 msgr2=0x7f7f50158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5015fa10 0x7f7f50158100 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800efc0 tx=0x7f7f4800c490 comp rx=0 tx=0).stop 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 -- 172.21.15.59:0/4072330403 shutdown_connections 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f3406b4d0 0x7f7f3406d990 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5015fa10 0x7f7f50158100 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5015bdd0 0x7f7f500b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 --2- 172.21.15.59:0/4072330403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f500d16e0 0x7f7f500ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 -- 172.21.15.59:0/4072330403 >> 172.21.15.59:0/4072330403 conn(0x7f7f5009a9f0 msgr2=0x7f7f5006f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 -- 172.21.15.59:0/4072330403 shutdown_connections 2024-03-06T07:29:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:43.433+0000 7f7f56b3e640 1 -- 172.21.15.59:0/4072330403 wait complete. 2024-03-06T07:29:43.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:43.646 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:43.929 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:44.647 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:44.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.761+0000 7fdfca066640 1 Processor -- start 2024-03-06T07:29:44.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.761+0000 7fdfca066640 1 -- start start 2024-03-06T07:29:44.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.762+0000 7fdfca066640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdfc40d22b0 0x7fdfc40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.762+0000 7fdfca066640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdfc40b41f0 0x7fdfc40b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.762+0000 7fdfca066640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc40b3430 con 0x7fdfc415dfe0 2024-03-06T07:29:44.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.762+0000 7fdfca066640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc40d2c50 con 0x7fdfc40b41f0 2024-03-06T07:29:44.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.762+0000 7fdfca066640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc40b31e0 con 0x7fdfc40d22b0 2024-03-06T07:29:44.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.762+0000 7fdfc3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdfc40b41f0 0x7fdfc40b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.762+0000 7fdfc3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdfc40b41f0 0x7fdfc40b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39590/0 (socket says 172.21.15.59:39590) 2024-03-06T07:29:44.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.762+0000 7fdfc3fff640 1 -- 172.21.15.59:0/1979295160 learned_addr learned my addr 172.21.15.59:0/1979295160 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:44.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.763+0000 7fdfc3fff640 1 -- 172.21.15.59:0/1979295160 >> v2:172.21.15.164:3300/0 conn(0x7fdfc40d22b0 msgr2=0x7fdfc40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.763+0000 7fdfc3fff640 1 --2- 172.21.15.59:0/1979295160 >> v2:172.21.15.164:3300/0 conn(0x7fdfc40d22b0 0x7fdfc40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.763+0000 7fdfc3fff640 1 -- 172.21.15.59:0/1979295160 >> v1:172.21.15.164:6789/0 conn(0x7fdfc415dfe0 legacy=0x7fdfc40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.763+0000 7fdfc3fff640 1 -- 172.21.15.59:0/1979295160 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc40b3710 con 0x7fdfc40b41f0 2024-03-06T07:29:44.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.763+0000 7fdfc3fff640 1 --2- 172.21.15.59:0/1979295160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfc40b41f0 0x7fdfc40b4680 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009e50 tx=0x7fdfbc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260325d619461c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.763+0000 7fdfc27fc640 1 -- 172.21.15.59:0/1979295160 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc02ca80 con 0x7fdfc40b41f0 2024-03-06T07:29:44.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.763+0000 7fdfc27fc640 1 -- 172.21.15.59:0/1979295160 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfbc02cc20 con 0x7fdfc40b41f0 2024-03-06T07:29:44.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.763+0000 7fdfc27fc640 1 -- 172.21.15.59:0/1979295160 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc0343f0 con 0x7fdfc40b41f0 2024-03-06T07:29:44.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.764+0000 7fdfca066640 1 -- 172.21.15.59:0/1979295160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfc40b41f0 msgr2=0x7fdfc40b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.764+0000 7fdfca066640 1 --2- 172.21.15.59:0/1979295160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfc40b41f0 0x7fdfc40b4680 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009e50 tx=0x7fdfbc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.764+0000 7fdfca066640 1 -- 172.21.15.59:0/1979295160 shutdown_connections 2024-03-06T07:29:44.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.764+0000 7fdfca066640 1 --2- 172.21.15.59:0/1979295160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfc40b41f0 0x7fdfc40b4680 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.764+0000 7fdfca066640 1 --2- 172.21.15.59:0/1979295160 >> v2:172.21.15.164:3300/0 conn(0x7fdfc40d22b0 0x7fdfc40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.764+0000 7fdfca066640 1 -- 172.21.15.59:0/1979295160 >> 172.21.15.59:0/1979295160 conn(0x7fdfc40d3cd0 msgr2=0x7fdfc40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.764+0000 7fdfca066640 1 -- 172.21.15.59:0/1979295160 shutdown_connections 2024-03-06T07:29:44.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.764+0000 7fdfca066640 1 -- 172.21.15.59:0/1979295160 wait complete. 2024-03-06T07:29:44.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.765+0000 7fdfca066640 1 Processor -- start 2024-03-06T07:29:44.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.765+0000 7fdfca066640 1 -- start start 2024-03-06T07:29:44.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.766+0000 7fdfca066640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdfc40b41f0 0x7fdfc415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.766+0000 7fdfca066640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfc40d22b0 0x7fdfc40b7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.766+0000 7fdfca066640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfc415dfe0 0x7fdfc4150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.766+0000 7fdfca066640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40cfa00 con 0x7fdfc40d22b0 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.766+0000 7fdfca066640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40b3d60 con 0x7fdfc40b41f0 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.766+0000 7fdfca066640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40d1c70 con 0x7fdfc415dfe0 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.766+0000 7fdfc2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfc40d22b0 0x7fdfc40b7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.766+0000 7fdfc2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfc40d22b0 0x7fdfc40b7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39592/0 (socket says 172.21.15.59:39592) 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.766+0000 7fdfc2ffd640 1 -- 172.21.15.59:0/1196659299 learned_addr learned my addr 172.21.15.59:0/1196659299 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.766+0000 7fdfc37fe640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdfc40b41f0 0x7fdfc415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.766+0000 7fdfc3fff640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfc415dfe0 0x7fdfc4150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfc2ffd640 1 -- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfc415dfe0 msgr2=0x7fdfc4150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfc2ffd640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfc415dfe0 0x7fdfc4150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfc2ffd640 1 -- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdfc40b41f0 msgr2=0x7fdfc415e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfc2ffd640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdfc40b41f0 0x7fdfc415e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfc2ffd640 1 -- 172.21.15.59:0/1196659299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc415eaa0 con 0x7fdfc40d22b0 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfc37fe640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdfc40b41f0 0x7fdfc415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfc3fff640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfc415dfe0 0x7fdfc4150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:44.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfc2ffd640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfc40d22b0 0x7fdfc40b7a40 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fdfac00b4a0 tx=0x7fdfac00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfbbfff640 1 -- 172.21.15.59:0/1196659299 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac014070 con 0x7fdfc40d22b0 2024-03-06T07:29:44.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfbbfff640 1 -- 172.21.15.59:0/1196659299 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac010610 con 0x7fdfc40d22b0 2024-03-06T07:29:44.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfbbfff640 1 -- 172.21.15.59:0/1196659299 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac010a00 con 0x7fdfc40d22b0 2024-03-06T07:29:44.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfca066640 1 -- 172.21.15.59:0/1196659299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc415ed80 con 0x7fdfc40d22b0 2024-03-06T07:29:44.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.767+0000 7fdfca066640 1 -- 172.21.15.59:0/1196659299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc415f340 con 0x7fdfc40d22b0 2024-03-06T07:29:44.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.768+0000 7fdfca066640 1 -- 172.21.15.59:0/1196659299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc404f330 con 0x7fdfc40d22b0 2024-03-06T07:29:44.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.768+0000 7fdfbbfff640 1 -- 172.21.15.59:0/1196659299 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdfac017070 con 0x7fdfc40d22b0 2024-03-06T07:29:44.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.769+0000 7fdfbbfff640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdfa406b500 0x7fdfa406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.769+0000 7fdfbbfff640 1 -- 172.21.15.59:0/1196659299 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdfac089c60 con 0x7fdfc40d22b0 2024-03-06T07:29:44.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.769+0000 7fdfc37fe640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdfa406b500 0x7fdfa406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.770+0000 7fdfc37fe640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdfa406b500 0x7fdfa406d9c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400a810 tx=0x7fdfb4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:44.773+0000 7fdfbbfff640 1 -- 172.21.15.59:0/1196659299 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdfac058aa0 con 0x7fdfc40d22b0 2024-03-06T07:29:45.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.038+0000 7fdfca066640 1 -- 172.21.15.59:0/1196659299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfc40d4940 con 0x7fdfc40d22b0 2024-03-06T07:29:45.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.038+0000 7fdfbbfff640 1 -- 172.21.15.59:0/1196659299 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7fdfac0583a0 con 0x7fdfc40d22b0 2024-03-06T07:29:45.040 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:45.041 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:45.041 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:45.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 -- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdfa406b500 msgr2=0x7fdfa406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdfa406b500 0x7fdfa406d9c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400a810 tx=0x7fdfb4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:45.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 -- 172.21.15.59:0/1196659299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfc40d22b0 msgr2=0x7fdfc40b7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfc40d22b0 0x7fdfc40b7a40 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fdfac00b4a0 tx=0x7fdfac00b960 comp rx=0 tx=0).stop 2024-03-06T07:29:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 -- 172.21.15.59:0/1196659299 shutdown_connections 2024-03-06T07:29:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdfa406b500 0x7fdfa406d9c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfc415dfe0 0x7fdfc4150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdfc40d22b0 0x7fdfc40b7a40 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 --2- 172.21.15.59:0/1196659299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdfc40b41f0 0x7fdfc415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 -- 172.21.15.59:0/1196659299 >> 172.21.15.59:0/1196659299 conn(0x7fdfc40d3cd0 msgr2=0x7fdfc415d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 -- 172.21.15.59:0/1196659299 shutdown_connections 2024-03-06T07:29:45.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:45.040+0000 7fdfca066640 1 -- 172.21.15.59:0/1196659299 wait complete. 2024-03-06T07:29:45.254 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:45.254 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:45.931 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:46.256 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:46.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.357+0000 7f46a0f16640 1 Processor -- start 2024-03-06T07:29:46.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.357+0000 7f46a0f16640 1 -- start start 2024-03-06T07:29:46.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f46a0f16640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f469c15dfe0 0x7f469c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f46a0f16640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f469c0d29c0 0x7f469c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f46a0f16640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c0b3430 con 0x7f469c0cf830 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f46a0f16640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f469c0b31e0 con 0x7f469c15dfe0 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f46a0f16640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f469c0b3d60 con 0x7f469c0d29c0 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f469a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f469c15dfe0 0x7f469c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f4693fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f469c0cf830 0x7f469c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37540/0 (socket says 172.21.15.59:37540) 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f4693fff640 1 -- 172.21.15.59:0/3109015475 learned_addr learned my addr 172.21.15.59:0/3109015475 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f469a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f469c15dfe0 0x7f469c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39600/0 (socket says 172.21.15.59:39600) 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f469ad9d640 1 --2- 172.21.15.59:0/3109015475 >> v2:172.21.15.164:3300/0 conn(0x7f469c0d29c0 0x7f469c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f469a59c640 1 -- 172.21.15.59:0/3109015475 >> v2:172.21.15.164:3300/0 conn(0x7f469c0d29c0 msgr2=0x7f469c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f469a59c640 1 --2- 172.21.15.59:0/3109015475 >> v2:172.21.15.164:3300/0 conn(0x7f469c0d29c0 0x7f469c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f469a59c640 1 -- 172.21.15.59:0/3109015475 >> v1:172.21.15.59:6789/0 conn(0x7f469c0cf830 legacy=0x7f469c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f469a59c640 1 -- 172.21.15.59:0/3109015475 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c0b3710 con 0x7f469c15dfe0 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f4699d9b640 1 -- 172.21.15.59:0/3109015475 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856075124 0 0) 0x7f469c0b3430 con 0x7f469c0cf830 2024-03-06T07:29:46.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.358+0000 7f469ad9d640 1 --2- 172.21.15.59:0/3109015475 >> v2:172.21.15.164:3300/0 conn(0x7f469c0d29c0 0x7f469c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:46.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.359+0000 7f469a59c640 1 --2- 172.21.15.59:0/3109015475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f469c15dfe0 0x7f469c0cf2f0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f469c0b3cd0 tx=0x7f468c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3316bc87670c8d5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.359+0000 7f4699d9b640 1 -- 172.21.15.59:0/3109015475 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c035020 con 0x7f469c15dfe0 2024-03-06T07:29:46.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.359+0000 7f4699d9b640 1 -- 172.21.15.59:0/3109015475 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c031830 con 0x7f469c15dfe0 2024-03-06T07:29:46.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.359+0000 7f4699d9b640 1 -- 172.21.15.59:0/3109015475 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c031c00 con 0x7f469c15dfe0 2024-03-06T07:29:46.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.360+0000 7f46a0f16640 1 -- 172.21.15.59:0/3109015475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f469c15dfe0 msgr2=0x7f469c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.360+0000 7f46a0f16640 1 --2- 172.21.15.59:0/3109015475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f469c15dfe0 0x7f469c0cf2f0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f469c0b3cd0 tx=0x7f468c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.360+0000 7f46a0f16640 1 -- 172.21.15.59:0/3109015475 shutdown_connections 2024-03-06T07:29:46.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.360+0000 7f46a0f16640 1 --2- 172.21.15.59:0/3109015475 >> v2:172.21.15.164:3300/0 conn(0x7f469c0d29c0 0x7f469c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.360+0000 7f46a0f16640 1 --2- 172.21.15.59:0/3109015475 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f469c15dfe0 0x7f469c0cf2f0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.360+0000 7f46a0f16640 1 -- 172.21.15.59:0/3109015475 >> 172.21.15.59:0/3109015475 conn(0x7f469c0d3cd0 msgr2=0x7f469c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.360+0000 7f46a0f16640 1 -- 172.21.15.59:0/3109015475 shutdown_connections 2024-03-06T07:29:46.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.360+0000 7f46a0f16640 1 -- 172.21.15.59:0/3109015475 wait complete. 2024-03-06T07:29:46.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.361+0000 7f46a0f16640 1 Processor -- start 2024-03-06T07:29:46.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.361+0000 7f46a0f16640 1 -- start start 2024-03-06T07:29:46.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f46a0f16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f469c0cf830 0x7f469c161020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f46a0f16640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f469c0d29c0 0x7f469c0cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f46a0f16640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f469c15dfe0 0x7f469c151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f46a0f16640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0d1df0 con 0x7f469c15dfe0 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f46a0f16640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0d1c70 con 0x7f469c0d29c0 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f46a0f16640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469c15e3c0 con 0x7f469c0cf830 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f469ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f469c15dfe0 0x7f469c151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f469ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f469c15dfe0 0x7f469c151010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39602/0 (socket says 172.21.15.59:39602) 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f469ad9d640 1 -- 172.21.15.59:0/923210322 learned_addr learned my addr 172.21.15.59:0/923210322 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f469a59c640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f469c0cf830 0x7f469c161020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f4693fff640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f469c0d29c0 0x7f469c0cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f469ad9d640 1 -- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f469c0cf830 msgr2=0x7f469c161020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f469ad9d640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f469c0cf830 0x7f469c161020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f469ad9d640 1 -- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f469c0d29c0 msgr2=0x7f469c0cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f469ad9d640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f469c0d29c0 0x7f469c0cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f469ad9d640 1 -- 172.21.15.59:0/923210322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c0cb370 con 0x7f469c15dfe0 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.362+0000 7f469ad9d640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f469c15dfe0 0x7f469c151010 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f4694007ec0 tx=0x7f469400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.363+0000 7f4692ffd640 1 -- 172.21.15.59:0/923210322 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469401a070 con 0x7f469c15dfe0 2024-03-06T07:29:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.363+0000 7f4692ffd640 1 -- 172.21.15.59:0/923210322 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4694003b90 con 0x7f469c15dfe0 2024-03-06T07:29:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.363+0000 7f46a0f16640 1 -- 172.21.15.59:0/923210322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469c151550 con 0x7f469c15dfe0 2024-03-06T07:29:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.363+0000 7f4692ffd640 1 -- 172.21.15.59:0/923210322 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4694004030 con 0x7f469c15dfe0 2024-03-06T07:29:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.363+0000 7f46a0f16640 1 -- 172.21.15.59:0/923210322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469c0cd8d0 con 0x7f469c15dfe0 2024-03-06T07:29:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.364+0000 7f46a0f16640 1 -- 172.21.15.59:0/923210322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469c04f330 con 0x7f469c15dfe0 2024-03-06T07:29:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.364+0000 7f4692ffd640 1 -- 172.21.15.59:0/923210322 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f469401d070 con 0x7f469c15dfe0 2024-03-06T07:29:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.364+0000 7f4692ffd640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f467c06b4d0 0x7f467c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.365+0000 7f4692ffd640 1 -- 172.21.15.59:0/923210322 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f469408a5b0 con 0x7f469c15dfe0 2024-03-06T07:29:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.365+0000 7f469a59c640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f467c06b4d0 0x7f467c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:46.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.365+0000 7f469a59c640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f467c06b4d0 0x7f467c06d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f468c036000 tx=0x7f468c03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:46.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.369+0000 7f4692ffd640 1 -- 172.21.15.59:0/923210322 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4694010070 con 0x7f469c15dfe0 2024-03-06T07:29:46.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.587+0000 7f46a0f16640 1 -- 172.21.15.59:0/923210322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f469c0b5510 con 0x7f469c15dfe0 2024-03-06T07:29:46.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.587+0000 7f4692ffd640 1 -- 172.21.15.59:0/923210322 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f4694059350 con 0x7f469c15dfe0 2024-03-06T07:29:46.589 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:46.590 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:46.590 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:46.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 -- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f467c06b4d0 msgr2=0x7f467c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f467c06b4d0 0x7f467c06d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f468c036000 tx=0x7f468c03c040 comp rx=0 tx=0).stop 2024-03-06T07:29:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 -- 172.21.15.59:0/923210322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f469c15dfe0 msgr2=0x7f469c151010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f469c15dfe0 0x7f469c151010 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f4694007ec0 tx=0x7f469400d4a0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 -- 172.21.15.59:0/923210322 shutdown_connections 2024-03-06T07:29:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f467c06b4d0 0x7f467c06d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f469c15dfe0 0x7f469c151010 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f469c0d29c0 0x7f469c0cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 --2- 172.21.15.59:0/923210322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f469c0cf830 0x7f469c161020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 -- 172.21.15.59:0/923210322 >> 172.21.15.59:0/923210322 conn(0x7f469c0d3cd0 msgr2=0x7f469c0b4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 -- 172.21.15.59:0/923210322 shutdown_connections 2024-03-06T07:29:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:46.589+0000 7f46a0f16640 1 -- 172.21.15.59:0/923210322 wait complete. 2024-03-06T07:29:46.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:46.803 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:47.804 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:47.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.904+0000 7fabc6664640 1 Processor -- start 2024-03-06T07:29:47.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.904+0000 7fabc6664640 1 -- start start 2024-03-06T07:29:47.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.904+0000 7fabc6664640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fabc015dfe0 0x7fabc00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.904+0000 7fabc6664640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fabc00d29c0 0x7fabc00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.904+0000 7fabc6664640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc00b3430 con 0x7fabc00cf830 2024-03-06T07:29:47.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.904+0000 7fabc6664640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc00b31e0 con 0x7fabc00d29c0 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.904+0000 7fabc6664640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc00b3d60 con 0x7fabc015dfe0 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.905+0000 7fabbf7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fabc00cf830 0x7fabc00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37556/0 (socket says 172.21.15.59:37556) 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.905+0000 7fabbf7fe640 1 -- 172.21.15.59:0/2965827446 learned_addr learned my addr 172.21.15.59:0/2965827446 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.905+0000 7fabc4c01640 1 --2- 172.21.15.59:0/2965827446 >> v2:172.21.15.59:3300/0 conn(0x7fabc00d29c0 0x7fabc00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.905+0000 7fabbffff640 1 --2- 172.21.15.59:0/2965827446 >> v2:172.21.15.164:3300/0 conn(0x7fabc015dfe0 0x7fabc00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.905+0000 7fabbeffd640 1 -- 172.21.15.59:0/2965827446 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914786156 0 0) 0x7fabc00b3430 con 0x7fabc00cf830 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.905+0000 7fabbeffd640 1 -- 172.21.15.59:0/2965827446 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faba8003520 con 0x7fabc00cf830 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.905+0000 7fabc4c01640 1 -- 172.21.15.59:0/2965827446 >> v2:172.21.15.164:3300/0 conn(0x7fabc015dfe0 msgr2=0x7fabc00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.905+0000 7fabc4c01640 1 --2- 172.21.15.59:0/2965827446 >> v2:172.21.15.164:3300/0 conn(0x7fabc015dfe0 0x7fabc00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.905+0000 7fabc4c01640 1 -- 172.21.15.59:0/2965827446 >> v1:172.21.15.59:6789/0 conn(0x7fabc00cf830 legacy=0x7fabc00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.905+0000 7fabbeffd640 1 -- 172.21.15.59:0/2965827446 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 478337691 0 0) 0x7faba8003520 con 0x7fabc00cf830 2024-03-06T07:29:47.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.905+0000 7fabc4c01640 1 -- 172.21.15.59:0/2965827446 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc00b3710 con 0x7fabc00d29c0 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.906+0000 7fabc4c01640 1 --2- 172.21.15.59:0/2965827446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabc00d29c0 0x7fabc00b41f0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fabc0067990 tx=0x7fabb802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd841fbd649c44e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.906+0000 7fabbeffd640 1 -- 172.21.15.59:0/2965827446 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb8039070 con 0x7fabc00d29c0 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.906+0000 7fabbeffd640 1 -- 172.21.15.59:0/2965827446 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb802c8c0 con 0x7fabc00d29c0 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.906+0000 7fabbeffd640 1 -- 172.21.15.59:0/2965827446 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4001670 con 0x7fabc00d29c0 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.907+0000 7fabc6664640 1 -- 172.21.15.59:0/2965827446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabc00d29c0 msgr2=0x7fabc00b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.907+0000 7fabc6664640 1 --2- 172.21.15.59:0/2965827446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabc00d29c0 0x7fabc00b41f0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fabc0067990 tx=0x7fabb802bd90 comp rx=0 tx=0).stop 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.907+0000 7fabc6664640 1 -- 172.21.15.59:0/2965827446 shutdown_connections 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.907+0000 7fabc6664640 1 --2- 172.21.15.59:0/2965827446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabc00d29c0 0x7fabc00b41f0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.907+0000 7fabc6664640 1 --2- 172.21.15.59:0/2965827446 >> v2:172.21.15.164:3300/0 conn(0x7fabc015dfe0 0x7fabc00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.907+0000 7fabc6664640 1 -- 172.21.15.59:0/2965827446 >> 172.21.15.59:0/2965827446 conn(0x7fabc00d3cd0 msgr2=0x7fabc00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.907+0000 7fabc6664640 1 -- 172.21.15.59:0/2965827446 shutdown_connections 2024-03-06T07:29:47.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.907+0000 7fabc6664640 1 -- 172.21.15.59:0/2965827446 wait complete. 2024-03-06T07:29:47.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.908+0000 7fabc6664640 1 Processor -- start 2024-03-06T07:29:47.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.908+0000 7fabc6664640 1 -- start start 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc6664640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabc00cf830 0x7fabc015c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc6664640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabc00d29c0 0x7fabc01f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc6664640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabc015dfe0 0x7fabc00cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc6664640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc00d1df0 con 0x7fabc015dfe0 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc6664640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc00d1c70 con 0x7fabc00cf830 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc6664640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc015e3c0 con 0x7fabc00d29c0 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc4c01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabc015dfe0 0x7fabc00cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc4c01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabc015dfe0 0x7fabc00cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39620/0 (socket says 172.21.15.59:39620) 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc4c01640 1 -- 172.21.15.59:0/1760973862 learned_addr learned my addr 172.21.15.59:0/1760973862 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabbffff640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabc00cf830 0x7fabc015c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabbf7fe640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabc00d29c0 0x7fabc01f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc4c01640 1 -- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabc00d29c0 msgr2=0x7fabc01f4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc4c01640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabc00d29c0 0x7fabc01f4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc4c01640 1 -- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabc00cf830 msgr2=0x7fabc015c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc4c01640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabc00cf830 0x7fabc015c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc4c01640 1 -- 172.21.15.59:0/1760973862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc01f5250 con 0x7fabc015dfe0 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabbf7fe640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabc00d29c0 0x7fabc01f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabbffff640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabc00cf830 0x7fabc015c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc4c01640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabc015dfe0 0x7fabc00cca10 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fabb8009e80 tx=0x7fabb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabbcff9640 1 -- 172.21.15.59:0/1760973862 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb8039050 con 0x7fabc015dfe0 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabbcff9640 1 -- 172.21.15.59:0/1760973862 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb8045070 con 0x7fabc015dfe0 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabbcff9640 1 -- 172.21.15.59:0/1760973862 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb8009450 con 0x7fabc015dfe0 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.909+0000 7fabc6664640 1 -- 172.21.15.59:0/1760973862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc015eaf0 con 0x7fabc015dfe0 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.910+0000 7fabc6664640 1 -- 172.21.15.59:0/1760973862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc015f0b0 con 0x7fabc015dfe0 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.911+0000 7fabc6664640 1 -- 172.21.15.59:0/1760973862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc004f330 con 0x7fabc015dfe0 2024-03-06T07:29:47.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.911+0000 7fabbcff9640 1 -- 172.21.15.59:0/1760973862 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabb8040690 con 0x7fabc015dfe0 2024-03-06T07:29:47.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.911+0000 7fabbcff9640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabac06b500 0x7fabac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.912+0000 7fabbcff9640 1 -- 172.21.15.59:0/1760973862 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fabb80ac9e0 con 0x7fabc015dfe0 2024-03-06T07:29:47.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.912+0000 7fabbffff640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabac06b500 0x7fabac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.912+0000 7fabbffff640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabac06b500 0x7fabac06d9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009a20 tx=0x7fabb0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:47.920+0000 7fabbcff9640 1 -- 172.21.15.59:0/1760973862 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabb807b8b0 con 0x7fabc015dfe0 2024-03-06T07:29:47.933 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.137+0000 7fabc6664640 1 -- 172.21.15.59:0/1760973862 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabc00cff50 con 0x7fabc015dfe0 2024-03-06T07:29:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.137+0000 7fabbcff9640 1 -- 172.21.15.59:0/1760973862 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7fabb807b1b0 con 0x7fabc015dfe0 2024-03-06T07:29:48.140 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:48.140 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:48.140 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:48.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 -- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabac06b500 msgr2=0x7fabac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabac06b500 0x7fabac06d9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009a20 tx=0x7fabb0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 -- 172.21.15.59:0/1760973862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabc015dfe0 msgr2=0x7fabc00cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabc015dfe0 0x7fabc00cca10 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fabb8009e80 tx=0x7fabb8004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 -- 172.21.15.59:0/1760973862 shutdown_connections 2024-03-06T07:29:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabac06b500 0x7fabac06d9c0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabc015dfe0 0x7fabc00cca10 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabc00d29c0 0x7fabc01f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 --2- 172.21.15.59:0/1760973862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabc00cf830 0x7fabc015c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 -- 172.21.15.59:0/1760973862 >> 172.21.15.59:0/1760973862 conn(0x7fabc00d3cd0 msgr2=0x7fabc00d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 -- 172.21.15.59:0/1760973862 shutdown_connections 2024-03-06T07:29:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:48.139+0000 7fabc6664640 1 -- 172.21.15.59:0/1760973862 wait complete. 2024-03-06T07:29:48.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:48.354 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:49.354 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:49.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.458+0000 7fe7600f4640 1 Processor -- start 2024-03-06T07:29:49.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.458+0000 7fe7600f4640 1 -- start start 2024-03-06T07:29:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.459+0000 7fe7600f4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe7580d2710 0x7fe7580b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.459+0000 7fe7600f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7580b3430 con 0x7fe75815dfe0 2024-03-06T07:29:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.459+0000 7fe7600f4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7580b3710 con 0x7fe7580d22b0 2024-03-06T07:29:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.459+0000 7fe7600f4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7580b31e0 con 0x7fe7580d2710 2024-03-06T07:29:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.460+0000 7fe75de90640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe75815dfe0 0x7fe7580cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37570/0 (socket says 172.21.15.59:37570) 2024-03-06T07:29:49.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.460+0000 7fe75de90640 1 -- 172.21.15.59:0/3079212885 learned_addr learned my addr 172.21.15.59:0/3079212885 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.460+0000 7fe75e691640 1 --2- 172.21.15.59:0/3079212885 >> v2:172.21.15.59:3300/0 conn(0x7fe7580d2710 0x7fe7580b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.460+0000 7fe75ce8e640 1 -- 172.21.15.59:0/3079212885 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972671335 0 0) 0x7fe7580b3430 con 0x7fe75815dfe0 2024-03-06T07:29:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.460+0000 7fe75ce8e640 1 -- 172.21.15.59:0/3079212885 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe744003520 con 0x7fe75815dfe0 2024-03-06T07:29:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.460+0000 7fe75e691640 1 -- 172.21.15.59:0/3079212885 >> v1:172.21.15.164:6789/0 conn(0x7fe7580d22b0 legacy=0x7fe7580b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.460+0000 7fe75e691640 1 -- 172.21.15.59:0/3079212885 >> v1:172.21.15.59:6789/0 conn(0x7fe75815dfe0 legacy=0x7fe7580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.460+0000 7fe75e691640 1 -- 172.21.15.59:0/3079212885 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7580b39f0 con 0x7fe7580d2710 2024-03-06T07:29:49.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.461+0000 7fe75e691640 1 --2- 172.21.15.59:0/3079212885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7580d2710 0x7fe7580b4900 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fe750009dc0 tx=0x7fe75002bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15841800d0ac5d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.462+0000 7fe75ce8e640 1 -- 172.21.15.59:0/3079212885 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750039070 con 0x7fe7580d2710 2024-03-06T07:29:49.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.462+0000 7fe75ce8e640 1 -- 172.21.15.59:0/3079212885 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75002ccd0 con 0x7fe7580d2710 2024-03-06T07:29:49.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.462+0000 7fe75ce8e640 1 -- 172.21.15.59:0/3079212885 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe750034660 con 0x7fe7580d2710 2024-03-06T07:29:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.463+0000 7fe7600f4640 1 -- 172.21.15.59:0/3079212885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7580d2710 msgr2=0x7fe7580b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.463+0000 7fe7600f4640 1 --2- 172.21.15.59:0/3079212885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7580d2710 0x7fe7580b4900 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fe750009dc0 tx=0x7fe75002bf20 comp rx=0 tx=0).stop 2024-03-06T07:29:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.463+0000 7fe7600f4640 1 -- 172.21.15.59:0/3079212885 shutdown_connections 2024-03-06T07:29:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.463+0000 7fe7600f4640 1 --2- 172.21.15.59:0/3079212885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7580d2710 0x7fe7580b4900 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.463+0000 7fe7600f4640 1 -- 172.21.15.59:0/3079212885 >> 172.21.15.59:0/3079212885 conn(0x7fe7580d3cd0 msgr2=0x7fe7580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.463+0000 7fe7600f4640 1 -- 172.21.15.59:0/3079212885 shutdown_connections 2024-03-06T07:29:49.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.463+0000 7fe7600f4640 1 -- 172.21.15.59:0/3079212885 wait complete. 2024-03-06T07:29:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.464+0000 7fe7600f4640 1 Processor -- start 2024-03-06T07:29:49.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.464+0000 7fe7600f4640 1 -- start start 2024-03-06T07:29:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe7600f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7580d22b0 0x7fe7580b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe7600f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7580d2710 0x7fe7580cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe7600f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe75815dfe0 0x7fe7580b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe7600f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580cfa00 con 0x7fe7580d2710 2024-03-06T07:29:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe7600f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580b3d60 con 0x7fe7580d22b0 2024-03-06T07:29:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe7600f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580d1c70 con 0x7fe75815dfe0 2024-03-06T07:29:49.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75d68f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7580d2710 0x7fe7580cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75d68f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7580d2710 0x7fe7580cdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39638/0 (socket says 172.21.15.59:39638) 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75d68f640 1 -- 172.21.15.59:0/3363629736 learned_addr learned my addr 172.21.15.59:0/3363629736 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75de90640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7580d22b0 0x7fe7580b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75e691640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe75815dfe0 0x7fe7580b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75d68f640 1 -- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe75815dfe0 msgr2=0x7fe7580b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75d68f640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe75815dfe0 0x7fe7580b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75d68f640 1 -- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7580d22b0 msgr2=0x7fe7580b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75d68f640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7580d22b0 0x7fe7580b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75d68f640 1 -- 172.21.15.59:0/3363629736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7580b97f0 con 0x7fe7580d2710 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75e691640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe75815dfe0 0x7fe7580b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.465+0000 7fe75d68f640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7580d2710 0x7fe7580cdb80 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c450 tx=0x7fe74800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.466+0000 7fe74e7fc640 1 -- 172.21.15.59:0/3363629736 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7480052d0 con 0x7fe7580d2710 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.466+0000 7fe74e7fc640 1 -- 172.21.15.59:0/3363629736 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe748005470 con 0x7fe7580d2710 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.466+0000 7fe7600f4640 1 -- 172.21.15.59:0/3363629736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7580b9ad0 con 0x7fe7580d2710 2024-03-06T07:29:49.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.466+0000 7fe74e7fc640 1 -- 172.21.15.59:0/3363629736 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748012870 con 0x7fe7580d2710 2024-03-06T07:29:49.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.466+0000 7fe7600f4640 1 -- 172.21.15.59:0/3363629736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7580b1c70 con 0x7fe7580d2710 2024-03-06T07:29:49.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.467+0000 7fe7600f4640 1 -- 172.21.15.59:0/3363629736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75804f330 con 0x7fe7580d2710 2024-03-06T07:29:49.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.467+0000 7fe74e7fc640 1 -- 172.21.15.59:0/3363629736 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe748012a10 con 0x7fe7580d2710 2024-03-06T07:29:49.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.468+0000 7fe74e7fc640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe73c06b4d0 0x7fe73c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:49.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.468+0000 7fe74e7fc640 1 -- 172.21.15.59:0/3363629736 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe748089d10 con 0x7fe7580d2710 2024-03-06T07:29:49.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.468+0000 7fe75de90640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe73c06b4d0 0x7fe73c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:49.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.468+0000 7fe75de90640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe73c06b4d0 0x7fe73c06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe7540011e0 tx=0x7fe754007d30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:49.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.472+0000 7fe74e7fc640 1 -- 172.21.15.59:0/3363629736 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe748058be0 con 0x7fe7580d2710 2024-03-06T07:29:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.692+0000 7fe7600f4640 1 -- 172.21.15.59:0/3363629736 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7580ce9f0 con 0x7fe7580d2710 2024-03-06T07:29:49.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.692+0000 7fe74e7fc640 1 -- 172.21.15.59:0/3363629736 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7fe7480584e0 con 0x7fe7580d2710 2024-03-06T07:29:49.694 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:49.694 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:49.695 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 -- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe73c06b4d0 msgr2=0x7fe73c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe73c06b4d0 0x7fe73c06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe7540011e0 tx=0x7fe754007d30 comp rx=0 tx=0).stop 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 -- 172.21.15.59:0/3363629736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7580d2710 msgr2=0x7fe7580cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7580d2710 0x7fe7580cdb80 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c450 tx=0x7fe74800c910 comp rx=0 tx=0).stop 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 -- 172.21.15.59:0/3363629736 shutdown_connections 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe73c06b4d0 0x7fe73c06d990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe75815dfe0 0x7fe7580b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7580d2710 0x7fe7580cdb80 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 --2- 172.21.15.59:0/3363629736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7580d22b0 0x7fe7580b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 -- 172.21.15.59:0/3363629736 >> 172.21.15.59:0/3363629736 conn(0x7fe7580d3cd0 msgr2=0x7fe75806ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 -- 172.21.15.59:0/3363629736 shutdown_connections 2024-03-06T07:29:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:49.695+0000 7fe7600f4640 1 -- 172.21.15.59:0/3363629736 wait complete. 2024-03-06T07:29:49.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:49.912 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:49.935 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:50.748 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:50.750 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:50.914 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:51.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.018+0000 7f3c9f466640 1 Processor -- start 2024-03-06T07:29:51.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.018+0000 7f3c9f466640 1 -- start start 2024-03-06T07:29:51.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9f466640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3c9815dfe0 0x7f3c980cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9f466640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c980cf830 0x7f3c980d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9f466640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c980b3430 con 0x7f3c980d27f0 2024-03-06T07:29:51.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9f466640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c980d2c50 con 0x7f3c9815dfe0 2024-03-06T07:29:51.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9f466640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980b31e0 con 0x7f3c980cf830 2024-03-06T07:29:51.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9d202640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3c9815dfe0 0x7f3c980cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9d202640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3c9815dfe0 0x7f3c980cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39646/0 (socket says 172.21.15.59:39646) 2024-03-06T07:29:51.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9d202640 1 -- 172.21.15.59:0/3712298885 learned_addr learned my addr 172.21.15.59:0/3712298885 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:51.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9ca01640 1 --2- 172.21.15.59:0/3712298885 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c980cf830 0x7f3c980d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9d202640 1 -- 172.21.15.59:0/3712298885 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c980cf830 msgr2=0x7f3c980d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:51.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9d202640 1 --2- 172.21.15.59:0/3712298885 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c980cf830 0x7f3c980d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9d202640 1 -- 172.21.15.59:0/3712298885 >> v1:172.21.15.164:6789/0 conn(0x7f3c980d27f0 legacy=0x7f3c980d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.019+0000 7f3c9d202640 1 -- 172.21.15.59:0/3712298885 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c980b3710 con 0x7f3c9815dfe0 2024-03-06T07:29:51.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.020+0000 7f3c9d202640 1 --2- 172.21.15.59:0/3712298885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c9815dfe0 0x7f3c980cf2f0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009870 tx=0x7f3c9402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded8df6ce508d195 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.020+0000 7f3c8ffff640 1 -- 172.21.15.59:0/3712298885 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c94034070 con 0x7f3c9815dfe0 2024-03-06T07:29:51.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.020+0000 7f3c8ffff640 1 -- 172.21.15.59:0/3712298885 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c94030840 con 0x7f3c9815dfe0 2024-03-06T07:29:51.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.020+0000 7f3c8ffff640 1 -- 172.21.15.59:0/3712298885 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c94030c10 con 0x7f3c9815dfe0 2024-03-06T07:29:51.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.021+0000 7f3c9f466640 1 -- 172.21.15.59:0/3712298885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c9815dfe0 msgr2=0x7f3c980cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.021+0000 7f3c9f466640 1 --2- 172.21.15.59:0/3712298885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c9815dfe0 0x7f3c980cf2f0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009870 tx=0x7f3c9402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:51.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.021+0000 7f3c9f466640 1 -- 172.21.15.59:0/3712298885 shutdown_connections 2024-03-06T07:29:51.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.021+0000 7f3c9f466640 1 --2- 172.21.15.59:0/3712298885 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c980cf830 0x7f3c980d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.021+0000 7f3c9f466640 1 --2- 172.21.15.59:0/3712298885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c9815dfe0 0x7f3c980cf2f0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.021+0000 7f3c9f466640 1 -- 172.21.15.59:0/3712298885 >> 172.21.15.59:0/3712298885 conn(0x7f3c980d3cd0 msgr2=0x7f3c980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.021+0000 7f3c9f466640 1 -- 172.21.15.59:0/3712298885 shutdown_connections 2024-03-06T07:29:51.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.021+0000 7f3c9f466640 1 -- 172.21.15.59:0/3712298885 wait complete. 2024-03-06T07:29:51.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.022+0000 7f3c9f466640 1 Processor -- start 2024-03-06T07:29:51.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.022+0000 7f3c9f466640 1 -- start start 2024-03-06T07:29:51.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9f466640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c980cf830 0x7f3c9814af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9f466640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c980d27f0 0x7f3c9815f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9f466640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c9815dfe0 0x7f3c98160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9f466640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980d1c70 con 0x7f3c9815dfe0 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9f466640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980b3d60 con 0x7f3c980d27f0 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9f466640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980d1df0 con 0x7f3c980cf830 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9da03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c9815dfe0 0x7f3c98160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9da03640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c9815dfe0 0x7f3c98160490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39660/0 (socket says 172.21.15.59:39660) 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9da03640 1 -- 172.21.15.59:0/3327067377 learned_addr learned my addr 172.21.15.59:0/3327067377 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9d202640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c980cf830 0x7f3c9814af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9ca01640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c980d27f0 0x7f3c9815f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9da03640 1 -- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c980cf830 msgr2=0x7f3c9814af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9da03640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c980cf830 0x7f3c9814af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9da03640 1 -- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c980d27f0 msgr2=0x7f3c9815f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9da03640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c980d27f0 0x7f3c9815f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9da03640 1 -- 172.21.15.59:0/3327067377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9815a330 con 0x7f3c9815dfe0 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.023+0000 7f3c9ca01640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c980d27f0 0x7f3c9815f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:51.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.024+0000 7f3c9da03640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c9815dfe0 0x7f3c98160490 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000c7a0 tx=0x7f3c9000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.024+0000 7f3c8dffb640 1 -- 172.21.15.59:0/3327067377 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c90003b70 con 0x7f3c9815dfe0 2024-03-06T07:29:51.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.024+0000 7f3c8dffb640 1 -- 172.21.15.59:0/3327067377 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c90003d30 con 0x7f3c9815dfe0 2024-03-06T07:29:51.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.024+0000 7f3c9f466640 1 -- 172.21.15.59:0/3327067377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c980cda60 con 0x7f3c9815dfe0 2024-03-06T07:29:51.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.024+0000 7f3c8dffb640 1 -- 172.21.15.59:0/3327067377 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000aa90 con 0x7f3c9815dfe0 2024-03-06T07:29:51.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.024+0000 7f3c9f466640 1 -- 172.21.15.59:0/3327067377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c980ce020 con 0x7f3c9815dfe0 2024-03-06T07:29:51.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.025+0000 7f3c9f466640 1 -- 172.21.15.59:0/3327067377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c980cde50 con 0x7f3c9815dfe0 2024-03-06T07:29:51.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.025+0000 7f3c8dffb640 1 -- 172.21.15.59:0/3327067377 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c9000ac30 con 0x7f3c9815dfe0 2024-03-06T07:29:51.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.026+0000 7f3c8dffb640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:51.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.026+0000 7f3c8dffb640 1 -- 172.21.15.59:0/3327067377 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3c9008a0f0 con 0x7f3c9815dfe0 2024-03-06T07:29:51.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.026+0000 7f3c9d202640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:51.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.027+0000 7f3c9d202640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3c94036000 tx=0x7f3c9403b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:51.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.030+0000 7f3c8dffb640 1 -- 172.21.15.59:0/3327067377 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c90058f30 con 0x7f3c9815dfe0 2024-03-06T07:29:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.252+0000 7f3c9f466640 1 -- 172.21.15.59:0/3327067377 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c980cdc30 con 0x7f3c9815dfe0 2024-03-06T07:29:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.252+0000 7f3c8dffb640 1 -- 172.21.15.59:0/3327067377 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f3c90058830 con 0x7f3c9815dfe0 2024-03-06T07:29:51.254 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:51.254 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.254+0000 7f3c9f466640 1 -- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c7c06b4d0 msgr2=0x7f3c7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.254+0000 7f3c9f466640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3c94036000 tx=0x7f3c9403b040 comp rx=0 tx=0).stop 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.254+0000 7f3c9f466640 1 -- 172.21.15.59:0/3327067377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c9815dfe0 msgr2=0x7f3c98160490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.254+0000 7f3c9f466640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c9815dfe0 0x7f3c98160490 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000c7a0 tx=0x7f3c9000cc60 comp rx=0 tx=0).stop 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.254+0000 7f3c9f466640 1 -- 172.21.15.59:0/3327067377 shutdown_connections 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.254+0000 7f3c9f466640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.254+0000 7f3c9f466640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c9815dfe0 0x7f3c98160490 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.254+0000 7f3c9f466640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c980d27f0 0x7f3c9815f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.254+0000 7f3c9f466640 1 --2- 172.21.15.59:0/3327067377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c980cf830 0x7f3c9814af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.254+0000 7f3c9f466640 1 -- 172.21.15.59:0/3327067377 >> 172.21.15.59:0/3327067377 conn(0x7f3c980d3cd0 msgr2=0x7f3c9806d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.254+0000 7f3c9f466640 1 -- 172.21.15.59:0/3327067377 shutdown_connections 2024-03-06T07:29:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:51.255+0000 7f3c9f466640 1 -- 172.21.15.59:0/3327067377 wait complete. 2024-03-06T07:29:51.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:51.469 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:51.938 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:52.469 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:52.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.569+0000 7f4a70c6d640 1 Processor -- start 2024-03-06T07:29:52.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.569+0000 7f4a70c6d640 1 -- start start 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.570+0000 7f4a70c6d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4a6c15dfe0 0x7f4a6c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.570+0000 7f4a70c6d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0cf830 0x7f4a6c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.570+0000 7f4a70c6d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6c0b3430 con 0x7f4a6c0d27f0 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.570+0000 7f4a70c6d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6c0d2c50 con 0x7f4a6c15dfe0 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.570+0000 7f4a70c6d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c0b31e0 con 0x7f4a6c0cf830 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.570+0000 7f4a6ad9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4a6c0d27f0 0x7f4a6c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37584/0 (socket says 172.21.15.59:37584) 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.570+0000 7f4a6ad9d640 1 -- 172.21.15.59:0/693182566 learned_addr learned my addr 172.21.15.59:0/693182566 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.570+0000 7f4a69d9b640 1 --2- 172.21.15.59:0/693182566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0cf830 0x7f4a6c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.570+0000 7f4a6a59c640 1 --2- 172.21.15.59:0/693182566 >> v2:172.21.15.164:3300/0 conn(0x7f4a6c15dfe0 0x7f4a6c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.571+0000 7f4a6959a640 1 -- 172.21.15.59:0/693182566 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743232873 0 0) 0x7f4a6c0b3430 con 0x7f4a6c0d27f0 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.571+0000 7f4a6959a640 1 -- 172.21.15.59:0/693182566 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a58003520 con 0x7f4a6c0d27f0 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.571+0000 7f4a69d9b640 1 -- 172.21.15.59:0/693182566 >> v2:172.21.15.164:3300/0 conn(0x7f4a6c15dfe0 msgr2=0x7f4a6c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.571+0000 7f4a69d9b640 1 --2- 172.21.15.59:0/693182566 >> v2:172.21.15.164:3300/0 conn(0x7f4a6c15dfe0 0x7f4a6c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.571+0000 7f4a69d9b640 1 -- 172.21.15.59:0/693182566 >> v1:172.21.15.59:6789/0 conn(0x7f4a6c0d27f0 legacy=0x7f4a6c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.571+0000 7f4a69d9b640 1 -- 172.21.15.59:0/693182566 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c0b3710 con 0x7f4a6c0cf830 2024-03-06T07:29:52.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.571+0000 7f4a6a59c640 1 --2- 172.21.15.59:0/693182566 >> v2:172.21.15.164:3300/0 conn(0x7f4a6c15dfe0 0x7f4a6c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a69d9b640 1 --2- 172.21.15.59:0/693182566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0cf830 0x7f4a6c0d22b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4a540097a0 tx=0x7f4a5402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78ad42bc8b3d8404 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a6959a640 1 -- 172.21.15.59:0/693182566 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54035020 con 0x7f4a6c0cf830 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a6959a640 1 -- 172.21.15.59:0/693182566 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a54031830 con 0x7f4a6c0cf830 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a6959a640 1 -- 172.21.15.59:0/693182566 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54031be0 con 0x7f4a6c0cf830 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a70c6d640 1 -- 172.21.15.59:0/693182566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0cf830 msgr2=0x7f4a6c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a70c6d640 1 --2- 172.21.15.59:0/693182566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0cf830 0x7f4a6c0d22b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4a540097a0 tx=0x7f4a5402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a70c6d640 1 -- 172.21.15.59:0/693182566 shutdown_connections 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a70c6d640 1 --2- 172.21.15.59:0/693182566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0cf830 0x7f4a6c0d22b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a70c6d640 1 --2- 172.21.15.59:0/693182566 >> v2:172.21.15.164:3300/0 conn(0x7f4a6c15dfe0 0x7f4a6c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a70c6d640 1 -- 172.21.15.59:0/693182566 >> 172.21.15.59:0/693182566 conn(0x7f4a6c0d3cd0 msgr2=0x7f4a6c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a70c6d640 1 -- 172.21.15.59:0/693182566 shutdown_connections 2024-03-06T07:29:52.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.572+0000 7f4a70c6d640 1 -- 172.21.15.59:0/693182566 wait complete. 2024-03-06T07:29:52.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.574+0000 7f4a70c6d640 1 Processor -- start 2024-03-06T07:29:52.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.574+0000 7f4a70c6d640 1 -- start start 2024-03-06T07:29:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a70c6d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c0cf830 0x7f4a6c14e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a70c6d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0d27f0 0x7f4a6c0b1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a70c6d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a6c15dfe0 0x7f4a6c157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a70c6d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c0d1c70 con 0x7f4a6c0cf830 2024-03-06T07:29:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a70c6d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c0b3d60 con 0x7f4a6c15dfe0 2024-03-06T07:29:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a70c6d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c0d1df0 con 0x7f4a6c0d27f0 2024-03-06T07:29:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a6a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c0cf830 0x7f4a6c14e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a6a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c0cf830 0x7f4a6c14e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39668/0 (socket says 172.21.15.59:39668) 2024-03-06T07:29:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a6a59c640 1 -- 172.21.15.59:0/1741075256 learned_addr learned my addr 172.21.15.59:0/1741075256 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a69d9b640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0d27f0 0x7f4a6c0b1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a6ad9d640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a6c15dfe0 0x7f4a6c157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a6a59c640 1 -- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0d27f0 msgr2=0x7f4a6c0b1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a6a59c640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0d27f0 0x7f4a6c0b1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a6a59c640 1 -- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a6c15dfe0 msgr2=0x7f4a6c157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a6a59c640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a6c15dfe0 0x7f4a6c157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a6a59c640 1 -- 172.21.15.59:0/1741075256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c0b2370 con 0x7f4a6c0cf830 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a69d9b640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0d27f0 0x7f4a6c0b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.575+0000 7f4a6a59c640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c0cf830 0x7f4a6c14e2a0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400e960 tx=0x7f4a6400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.576+0000 7f4a62ffd640 1 -- 172.21.15.59:0/1741075256 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a64017070 con 0x7f4a6c0cf830 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.576+0000 7f4a62ffd640 1 -- 172.21.15.59:0/1741075256 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6400ca00 con 0x7f4a6c0cf830 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.576+0000 7f4a70c6d640 1 -- 172.21.15.59:0/1741075256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6c0b2650 con 0x7f4a6c0cf830 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.576+0000 7f4a62ffd640 1 -- 172.21.15.59:0/1741075256 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6400cdd0 con 0x7f4a6c0cf830 2024-03-06T07:29:52.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.576+0000 7f4a70c6d640 1 -- 172.21.15.59:0/1741075256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6c157970 con 0x7f4a6c0cf830 2024-03-06T07:29:52.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.577+0000 7f4a70c6d640 1 -- 172.21.15.59:0/1741075256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6c04f330 con 0x7f4a6c0cf830 2024-03-06T07:29:52.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.577+0000 7f4a62ffd640 1 -- 172.21.15.59:0/1741075256 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a6401a070 con 0x7f4a6c0cf830 2024-03-06T07:29:52.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.578+0000 7f4a62ffd640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a4c06b4d0 0x7f4a4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.578+0000 7f4a62ffd640 1 -- 172.21.15.59:0/1741075256 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4a6408a140 con 0x7f4a6c0cf830 2024-03-06T07:29:52.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.578+0000 7f4a69d9b640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a4c06b4d0 0x7f4a4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.578+0000 7f4a69d9b640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a4c06b4d0 0x7f4a4c06d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4a54036000 tx=0x7f4a54008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.582+0000 7f4a62ffd640 1 -- 172.21.15.59:0/1741075256 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a64058f00 con 0x7f4a6c0cf830 2024-03-06T07:29:52.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.820+0000 7f4a70c6d640 1 -- 172.21.15.59:0/1741075256 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a6c0b2f80 con 0x7f4a6c0cf830 2024-03-06T07:29:52.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.820+0000 7f4a62ffd640 1 -- 172.21.15.59:0/1741075256 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3416 (secure 0 0 0) 0x7f4a64058800 con 0x7f4a6c0cf830 2024-03-06T07:29:52.823 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:52.823 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:22.591394+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8676,"mds_1":8673},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8673":{"gid":8673,"name":"a","rank":1,"incarnation":81,"state":"up:stopping","state_seq":6,"addr":"172.21.15.59:6833/1247924565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":1247924565},{"type":"v1","addr":"172.21.15.59:6833","nonce":1247924565}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:52.823 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 88 2024-03-06T07:29:52.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 -- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a4c06b4d0 msgr2=0x7f4a4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a4c06b4d0 0x7f4a4c06d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4a54036000 tx=0x7f4a54008030 comp rx=0 tx=0).stop 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 -- 172.21.15.59:0/1741075256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c0cf830 msgr2=0x7f4a6c14e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c0cf830 0x7f4a6c14e2a0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400e960 tx=0x7f4a6400ee20 comp rx=0 tx=0).stop 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 -- 172.21.15.59:0/1741075256 shutdown_connections 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a4c06b4d0 0x7f4a4c06d990 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a6c15dfe0 0x7f4a6c157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a6c0d27f0 0x7f4a6c0b1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 --2- 172.21.15.59:0/1741075256 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a6c0cf830 0x7f4a6c14e2a0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 -- 172.21.15.59:0/1741075256 >> 172.21.15.59:0/1741075256 conn(0x7f4a6c0d3cd0 msgr2=0x7f4a6c0d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 -- 172.21.15.59:0/1741075256 shutdown_connections 2024-03-06T07:29:52.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:52.822+0000 7f4a70c6d640 1 -- 172.21.15.59:0/1741075256 wait complete. 2024-03-06T07:29:53.036 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:22.591394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8676, 'mds_1': 8673}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8673': {'gid': 8673, 'name': 'a', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 6, 'addr': '172.21.15.59:6833/1247924565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 1247924565}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 1247924565}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:53.036 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8673:up:stopping 2024-03-06T07:29:53.509 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:29:53.939 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:54.037 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:54.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.136+0000 7ff439e88640 1 Processor -- start 2024-03-06T07:29:54.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.136+0000 7ff439e88640 1 -- start start 2024-03-06T07:29:54.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.137+0000 7ff439e88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4340cfff0 0x7ff4340d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.137+0000 7ff439e88640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4340cc8b0 0x7ff4340ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.137+0000 7ff439e88640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff434161980 con 0x7ff43415bb20 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.137+0000 7ff439e88640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4340d2500 con 0x7ff4340cc8b0 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.137+0000 7ff439e88640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff434161c60 con 0x7ff4340cfff0 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.137+0000 7ff433fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4340cc8b0 0x7ff4340ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.137+0000 7ff433fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff4340cc8b0 0x7ff4340ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51134/0 (socket says 172.21.15.59:51134) 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.137+0000 7ff433fff640 1 -- 172.21.15.59:0/806469147 learned_addr learned my addr 172.21.15.59:0/806469147 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.137+0000 7ff432ffd640 1 --2- 172.21.15.59:0/806469147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4340cfff0 0x7ff4340d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.138+0000 7ff433fff640 1 -- 172.21.15.59:0/806469147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4340cfff0 msgr2=0x7ff4340d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.138+0000 7ff433fff640 1 --2- 172.21.15.59:0/806469147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4340cfff0 0x7ff4340d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.138+0000 7ff433fff640 1 -- 172.21.15.59:0/806469147 >> v1:172.21.15.164:6789/0 conn(0x7ff43415bb20 legacy=0x7ff4340d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.138+0000 7ff433fff640 1 -- 172.21.15.59:0/806469147 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff434161f40 con 0x7ff4340cc8b0 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.138+0000 7ff432ffd640 1 --2- 172.21.15.59:0/806469147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4340cfff0 0x7ff4340d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.138+0000 7ff433fff640 1 --2- 172.21.15.59:0/806469147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4340cc8b0 0x7ff4340ccd40 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009e50 tx=0x7ff42c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c19ac58229f34b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.139+0000 7ff4327fc640 1 -- 172.21.15.59:0/806469147 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c02cb00 con 0x7ff4340cc8b0 2024-03-06T07:29:54.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.139+0000 7ff4327fc640 1 -- 172.21.15.59:0/806469147 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42c02cca0 con 0x7ff4340cc8b0 2024-03-06T07:29:54.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.139+0000 7ff4327fc640 1 -- 172.21.15.59:0/806469147 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c034610 con 0x7ff4340cc8b0 2024-03-06T07:29:54.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.139+0000 7ff439e88640 1 -- 172.21.15.59:0/806469147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4340cc8b0 msgr2=0x7ff4340ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.139+0000 7ff439e88640 1 --2- 172.21.15.59:0/806469147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4340cc8b0 0x7ff4340ccd40 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009e50 tx=0x7ff42c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:54.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.139+0000 7ff439e88640 1 -- 172.21.15.59:0/806469147 shutdown_connections 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.139+0000 7ff439e88640 1 --2- 172.21.15.59:0/806469147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4340cc8b0 0x7ff4340ccd40 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.139+0000 7ff439e88640 1 --2- 172.21.15.59:0/806469147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4340cfff0 0x7ff4340d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.139+0000 7ff439e88640 1 -- 172.21.15.59:0/806469147 >> 172.21.15.59:0/806469147 conn(0x7ff43409a9f0 msgr2=0x7ff43409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.139+0000 7ff439e88640 1 -- 172.21.15.59:0/806469147 shutdown_connections 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.139+0000 7ff439e88640 1 -- 172.21.15.59:0/806469147 wait complete. 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.140+0000 7ff439e88640 1 Processor -- start 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.141+0000 7ff439e88640 1 -- start start 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.141+0000 7ff439e88640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4340cc8b0 0x7ff4340c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.141+0000 7ff439e88640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4340cfff0 0x7ff43415aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.141+0000 7ff439e88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff43415bb20 0x7ff43415b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.141+0000 7ff439e88640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4340d1980 con 0x7ff4340cc8b0 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.141+0000 7ff439e88640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff434162500 con 0x7ff4340cfff0 2024-03-06T07:29:54.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.141+0000 7ff439e88640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4340d1b00 con 0x7ff43415bb20 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.141+0000 7ff4337fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4340cc8b0 0x7ff4340c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.141+0000 7ff4337fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4340cc8b0 0x7ff4340c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51142/0 (socket says 172.21.15.59:51142) 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.141+0000 7ff4337fe640 1 -- 172.21.15.59:0/4088128754 learned_addr learned my addr 172.21.15.59:0/4088128754 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff4337fe640 1 -- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff43415bb20 msgr2=0x7ff43415b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff4337fe640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff43415bb20 0x7ff43415b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff4337fe640 1 -- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4340cfff0 msgr2=0x7ff43415aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff432ffd640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4340cfff0 0x7ff43415aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff4337fe640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4340cfff0 0x7ff43415aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff4337fe640 1 -- 172.21.15.59:0/4088128754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4340c7b20 con 0x7ff4340cc8b0 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff432ffd640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4340cfff0 0x7ff43415aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff4337fe640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4340cc8b0 0x7ff4340c8b90 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00c3f0 tx=0x7ff41c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff42bfff640 1 -- 172.21.15.59:0/4088128754 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0052d0 con 0x7ff4340cc8b0 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff42bfff640 1 -- 172.21.15.59:0/4088128754 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41c005490 con 0x7ff4340cc8b0 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff439e88640 1 -- 172.21.15.59:0/4088128754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4340c7e00 con 0x7ff4340cc8b0 2024-03-06T07:29:54.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff42bfff640 1 -- 172.21.15.59:0/4088128754 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c007260 con 0x7ff4340cc8b0 2024-03-06T07:29:54.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.142+0000 7ff439e88640 1 -- 172.21.15.59:0/4088128754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4340ca740 con 0x7ff4340cc8b0 2024-03-06T07:29:54.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.143+0000 7ff439e88640 1 -- 172.21.15.59:0/4088128754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43404f330 con 0x7ff4340cc8b0 2024-03-06T07:29:54.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.144+0000 7ff42bfff640 1 -- 172.21.15.59:0/4088128754 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff41c007400 con 0x7ff4340cc8b0 2024-03-06T07:29:54.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.144+0000 7ff42bfff640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff41406b500 0x7ff41406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.144+0000 7ff42bfff640 1 -- 172.21.15.59:0/4088128754 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff41c0897a0 con 0x7ff4340cc8b0 2024-03-06T07:29:54.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.144+0000 7ff432ffd640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff41406b500 0x7ff41406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.145+0000 7ff432ffd640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff41406b500 0x7ff41406d9c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff424006fd0 tx=0x7ff424008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.148+0000 7ff42bfff640 1 -- 172.21.15.59:0/4088128754 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff41c058560 con 0x7ff4340cc8b0 2024-03-06T07:29:54.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.370+0000 7ff439e88640 1 -- 172.21.15.59:0/4088128754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff43406f200 con 0x7ff4340cc8b0 2024-03-06T07:29:54.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.370+0000 7ff42bfff640 1 -- 172.21.15.59:0/4088128754 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+2609 (secure 0 0 0) 0x7ff41c057e60 con 0x7ff4340cc8b0 2024-03-06T07:29:54.373 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:54.373 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:52.591736+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8676},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:54.373 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 89 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.372+0000 7ff439e88640 1 -- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff41406b500 msgr2=0x7ff41406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.372+0000 7ff439e88640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff41406b500 0x7ff41406d9c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7ff424006fd0 tx=0x7ff424008040 comp rx=0 tx=0).stop 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.372+0000 7ff439e88640 1 -- 172.21.15.59:0/4088128754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4340cc8b0 msgr2=0x7ff4340c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.372+0000 7ff439e88640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4340cc8b0 0x7ff4340c8b90 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00c3f0 tx=0x7ff41c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.372+0000 7ff439e88640 1 -- 172.21.15.59:0/4088128754 shutdown_connections 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.372+0000 7ff439e88640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff41406b500 0x7ff41406d9c0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.372+0000 7ff439e88640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff43415bb20 0x7ff43415b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.372+0000 7ff439e88640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4340cfff0 0x7ff43415aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.372+0000 7ff439e88640 1 --2- 172.21.15.59:0/4088128754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4340cc8b0 0x7ff4340c8b90 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.372+0000 7ff439e88640 1 -- 172.21.15.59:0/4088128754 >> 172.21.15.59:0/4088128754 conn(0x7ff43409a9f0 msgr2=0x7ff43406f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.372+0000 7ff439e88640 1 -- 172.21.15.59:0/4088128754 shutdown_connections 2024-03-06T07:29:54.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.373+0000 7ff439e88640 1 -- 172.21.15.59:0/4088128754 wait complete. 2024-03-06T07:29:54.586 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:42.288807+0000', 'modified': '2024-03-06T07:29:52.591736+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8676}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_8676': {'gid': 8676, 'name': 'c', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6835/1740714701', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6834', 'nonce': 1740714701}, {'type': 'v1', 'addr': '172.21.15.59:6835', 'nonce': 1740714701}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8676, 'qdb_cluster': [8676]} 2024-03-06T07:29:54.586 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:29:54.587 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:29:54.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.687+0000 7fc164bc7640 1 Processor -- start 2024-03-06T07:29:54.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.687+0000 7fc164bc7640 1 -- start start 2024-03-06T07:29:54.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.688+0000 7fc164bc7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc1600b2fb0 0x7fc1601669a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.688+0000 7fc164bc7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16015d510 con 0x7fc1600b4fe0 2024-03-06T07:29:54.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.688+0000 7fc164bc7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16015d780 con 0x7fc1600b4c00 2024-03-06T07:29:54.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.688+0000 7fc164bc7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1600b3420 con 0x7fc1600b2fb0 2024-03-06T07:29:54.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.688+0000 7fc15ed9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc1600b2fb0 0x7fc1601669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.688+0000 7fc157fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc1600b4fe0 0x7fc16015ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60284/0 (socket says 172.21.15.59:60284) 2024-03-06T07:29:54.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.688+0000 7fc157fff640 1 -- 172.21.15.59:0/4053187336 learned_addr learned my addr 172.21.15.59:0/4053187336 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:54.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.688+0000 7fc15ed9d640 1 -- 172.21.15.59:0/4053187336 >> v1:172.21.15.164:6789/0 conn(0x7fc1600b4c00 legacy=0x7fc1600b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.688+0000 7fc15ed9d640 1 -- 172.21.15.59:0/4053187336 >> v1:172.21.15.59:6789/0 conn(0x7fc1600b4fe0 legacy=0x7fc16015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.688+0000 7fc15ed9d640 1 -- 172.21.15.59:0/4053187336 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16015da60 con 0x7fc1600b2fb0 2024-03-06T07:29:54.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.688+0000 7fc15dd9b640 1 -- 172.21.15.59:0/4053187336 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742578816 0 0) 0x7fc16015d510 con 0x7fc1600b4fe0 2024-03-06T07:29:54.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.689+0000 7fc15ed9d640 1 --2- 172.21.15.59:0/4053187336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1600b2fb0 0x7fc1601669a0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fc158009d90 tx=0x7fc15802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a027dba0b23453 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.689+0000 7fc15dd9b640 1 -- 172.21.15.59:0/4053187336 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158039070 con 0x7fc1600b2fb0 2024-03-06T07:29:54.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.689+0000 7fc15dd9b640 1 -- 172.21.15.59:0/4053187336 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15802cd00 con 0x7fc1600b2fb0 2024-03-06T07:29:54.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.689+0000 7fc15dd9b640 1 -- 172.21.15.59:0/4053187336 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158034660 con 0x7fc1600b2fb0 2024-03-06T07:29:54.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.690+0000 7fc164bc7640 1 -- 172.21.15.59:0/4053187336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1600b2fb0 msgr2=0x7fc1601669a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.690+0000 7fc164bc7640 1 --2- 172.21.15.59:0/4053187336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1600b2fb0 0x7fc1601669a0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fc158009d90 tx=0x7fc15802bf50 comp rx=0 tx=0).stop 2024-03-06T07:29:54.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.690+0000 7fc164bc7640 1 -- 172.21.15.59:0/4053187336 shutdown_connections 2024-03-06T07:29:54.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.690+0000 7fc164bc7640 1 --2- 172.21.15.59:0/4053187336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1600b2fb0 0x7fc1601669a0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.690+0000 7fc164bc7640 1 -- 172.21.15.59:0/4053187336 >> 172.21.15.59:0/4053187336 conn(0x7fc16009a9f0 msgr2=0x7fc16009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.690+0000 7fc164bc7640 1 -- 172.21.15.59:0/4053187336 shutdown_connections 2024-03-06T07:29:54.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.690+0000 7fc164bc7640 1 -- 172.21.15.59:0/4053187336 wait complete. 2024-03-06T07:29:54.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.691+0000 7fc164bc7640 1 Processor -- start 2024-03-06T07:29:54.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.691+0000 7fc164bc7640 1 -- start start 2024-03-06T07:29:54.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc164bc7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1600b2fb0 0x7fc16006cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc164bc7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1600b4c00 0x7fc16006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc164bc7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1600b4fe0 0x7fc160148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc164bc7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16015d170 con 0x7fc1600b4c00 2024-03-06T07:29:54.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc164bc7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16015e0b0 con 0x7fc1600b4fe0 2024-03-06T07:29:54.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc164bc7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16015d2f0 con 0x7fc1600b2fb0 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc157fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1600b4c00 0x7fc16006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc157fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1600b4c00 0x7fc16006dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51166/0 (socket says 172.21.15.59:51166) 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc157fff640 1 -- 172.21.15.59:0/3438182674 learned_addr learned my addr 172.21.15.59:0/3438182674 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc15ed9d640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1600b4fe0 0x7fc160148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc15e59c640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1600b2fb0 0x7fc16006cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc157fff640 1 -- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1600b2fb0 msgr2=0x7fc16006cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc157fff640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1600b2fb0 0x7fc16006cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc157fff640 1 -- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1600b4fe0 msgr2=0x7fc160148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc157fff640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1600b4fe0 0x7fc160148410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc157fff640 1 -- 172.21.15.59:0/3438182674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc160154080 con 0x7fc1600b4c00 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc15e59c640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1600b2fb0 0x7fc16006cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc15ed9d640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1600b4fe0 0x7fc160148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.692+0000 7fc157fff640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1600b4c00 0x7fc16006dc60 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fc14800c460 tx=0x7fc14800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.693+0000 7fc156ffd640 1 -- 172.21.15.59:0/3438182674 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1480074e0 con 0x7fc1600b4c00 2024-03-06T07:29:54.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.693+0000 7fc156ffd640 1 -- 172.21.15.59:0/3438182674 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1480047e0 con 0x7fc1600b4c00 2024-03-06T07:29:54.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.693+0000 7fc156ffd640 1 -- 172.21.15.59:0/3438182674 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc148004bd0 con 0x7fc1600b4c00 2024-03-06T07:29:54.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.693+0000 7fc164bc7640 1 -- 172.21.15.59:0/3438182674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc160154360 con 0x7fc1600b4c00 2024-03-06T07:29:54.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.693+0000 7fc164bc7640 1 -- 172.21.15.59:0/3438182674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc160154920 con 0x7fc1600b4c00 2024-03-06T07:29:54.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.694+0000 7fc156ffd640 1 -- 172.21.15.59:0/3438182674 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc148011070 con 0x7fc1600b4c00 2024-03-06T07:29:54.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.695+0000 7fc156ffd640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc14006b4d0 0x7fc14006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.695+0000 7fc156ffd640 1 -- 172.21.15.59:0/3438182674 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc148089dd0 con 0x7fc1600b4c00 2024-03-06T07:29:54.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.695+0000 7fc15e59c640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc14006b4d0 0x7fc14006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.696+0000 7fc1677aa740 1 Processor -- start 2024-03-06T07:29:54.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.696+0000 7fc15e59c640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc14006b4d0 0x7fc14006d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc150007fd0 tx=0x7fc150009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.696+0000 7fc1677aa740 1 -- start start 2024-03-06T07:29:54.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf288d8870 0x55bf288d8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf288dc8c0 0x55bf288de360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf28908a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc1677aa740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf288426b0 con 0x55bf28908600 2024-03-06T07:29:54.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc1677aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf287eba60 con 0x55bf288dc8c0 2024-03-06T07:29:54.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc1677aa740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf2883e750 con 0x55bf288d8870 2024-03-06T07:29:54.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc15ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf28908a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc15ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf28908a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51176/0 (socket says 172.21.15.59:51176) 2024-03-06T07:29:54.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc15ed9d640 1 -- 172.21.15.59:0/3851501291 learned_addr learned my addr 172.21.15.59:0/3851501291 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:54.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc15e59c640 1 --2- 172.21.15.59:0/3851501291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf288d8870 0x55bf288d8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc157fff640 1 --2- 172.21.15.59:0/3851501291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf288dc8c0 0x55bf288de360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc15ed9d640 1 -- 172.21.15.59:0/3851501291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf288d8870 msgr2=0x55bf288d8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc15ed9d640 1 --2- 172.21.15.59:0/3851501291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf288d8870 0x55bf288d8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc15ed9d640 1 -- 172.21.15.59:0/3851501291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf288dc8c0 msgr2=0x55bf288de360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc15ed9d640 1 --2- 172.21.15.59:0/3851501291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf288dc8c0 0x55bf288de360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc15ed9d640 1 -- 172.21.15.59:0/3851501291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf28809f30 con 0x55bf28908600 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc157fff640 1 --2- 172.21.15.59:0/3851501291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf288dc8c0 0x55bf288de360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc15ed9d640 1 --2- 172.21.15.59:0/3851501291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf28908a80 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fc15802c560 tx=0x7fc15802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365b80b1094e8138 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc13ffff640 1 -- 172.21.15.59:0/3851501291 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158039040 con 0x55bf28908600 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc13ffff640 1 -- 172.21.15.59:0/3851501291 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15803f440 con 0x55bf28908600 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.697+0000 7fc13ffff640 1 -- 172.21.15.59:0/3851501291 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15803f7f0 con 0x55bf28908600 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.698+0000 7fc1677aa740 1 -- 172.21.15.59:0/3851501291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 msgr2=0x55bf28908a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.698+0000 7fc1677aa740 1 --2- 172.21.15.59:0/3851501291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf28908a80 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fc15802c560 tx=0x7fc15802e000 comp rx=0 tx=0).stop 2024-03-06T07:29:54.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.698+0000 7fc1677aa740 1 -- 172.21.15.59:0/3851501291 shutdown_connections 2024-03-06T07:29:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.698+0000 7fc1677aa740 1 --2- 172.21.15.59:0/3851501291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf28908a80 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.698+0000 7fc1677aa740 1 --2- 172.21.15.59:0/3851501291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf288dc8c0 0x55bf288de360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.698+0000 7fc1677aa740 1 --2- 172.21.15.59:0/3851501291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf288d8870 0x55bf288d8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.698+0000 7fc1677aa740 1 -- 172.21.15.59:0/3851501291 >> 172.21.15.59:0/3851501291 conn(0x55bf288da8f0 msgr2=0x55bf288dc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.698+0000 7fc1677aa740 1 -- 172.21.15.59:0/3851501291 shutdown_connections 2024-03-06T07:29:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.698+0000 7fc1677aa740 1 -- 172.21.15.59:0/3851501291 wait complete. 2024-03-06T07:29:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.699+0000 7fc1677aa740 1 -- start start 2024-03-06T07:29:54.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.699+0000 7fc1677aa740 1 Processor -- start 2024-03-06T07:29:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf288ff170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf2890aa20 0x55bf28921160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf2890bf70 0x55bf2890c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc1677aa740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf288dcd50 con 0x55bf28908600 2024-03-06T07:29:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc1677aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf287feea0 con 0x55bf2890bf70 2024-03-06T07:29:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc1677aa740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf288dd030 con 0x55bf2890aa20 2024-03-06T07:29:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc1677aa740 10 client.0 _ll_register_callbacks cb 0x55bf287fe870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc15e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf288ff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc15e59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf288ff170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51188/0 (socket says 172.21.15.59:51188) 2024-03-06T07:29:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc15e59c640 1 -- 172.21.15.59:0/789846373 learned_addr learned my addr 172.21.15.59:0/789846373 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:54.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc157fff640 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf2890aa20 0x55bf28921160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc15ed9d640 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf2890bf70 0x55bf2890c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc15e59c640 1 -- 172.21.15.59:0/789846373 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf2890aa20 msgr2=0x55bf28921160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc15e59c640 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf2890aa20 0x55bf28921160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc15e59c640 1 -- 172.21.15.59:0/789846373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf2890bf70 msgr2=0x55bf2890c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc15e59c640 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf2890bf70 0x55bf2890c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.700+0000 7fc15e59c640 1 -- 172.21.15.59:0/789846373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf287ff270 con 0x55bf28908600 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc15e59c640 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf288ff170 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fc15000dd80 tx=0x7fc150001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55075dbd591a6f18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc13d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc13d7fa640 1 -- 172.21.15.59:0/789846373 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150026060 con 0x55bf28908600 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc13d7fa640 1 -- 172.21.15.59:0/789846373 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc150021ae0 con 0x55bf28908600 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc1677aa740 1 -- 172.21.15.59:0/789846373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc15803f440 con 0x55bf28908600 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc13d7fa640 1 -- 172.21.15.59:0/789846373 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc15000de70 con 0x55bf28908600 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc1677aa740 10 client.9135 fetch_fsmap learned FSMap version 90 2024-03-06T07:29:54.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc1677aa740 1 -- 172.21.15.59:0/789846373 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x55bf285572d0 con 0x55bf28908600 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc13d7fa640 1 -- 172.21.15.59:0/789846373 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15803f440 con 0x55bf28908600 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc13d7fa640 1 -- 172.21.15.59:0/789846373 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 90) v1 ==== 2172+0+0 (secure 0 0 0) 0x7fc150021ce0 con 0x55bf28908600 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc1677aa740 10 client.9135 fetch_fsmap finished waiting for FSMap version 90 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc1677aa740 10 client.9135 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc1677aa740 20 client.9135 populate_metadata read hostname 'smithi059' 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc1677aa740 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x55bf289217a0 0x55bf28909c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc1677aa740 4 client.9135 mds_command: new command op to 8676 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc1677aa740 1 -- 172.21.15.59:0/789846373 --> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bf2884fbf0 con 0x55bf289217a0 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.701+0000 7fc157fff640 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x55bf289217a0 0x55bf28909c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.702+0000 7fc157fff640 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x55bf289217a0 0x55bf28909c40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aab5394314b292a9 server_cookie=b1ab618b0e77169d in_seq=0 out_seq=0 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.702+0000 7fc13d7fa640 10 client.9135 ms_handle_connect on v2:172.21.15.59:6834/1740714701 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.702+0000 7fc13d7fa640 1 -- 172.21.15.59:0/789846373 <== mds.0 v2:172.21.15.59:6834/1740714701 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bf2884fbf0 con 0x55bf289217a0 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.702+0000 7fc13d7fa640 10 client.9135 handle_command_reply: tid=0 2024-03-06T07:29:54.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.702+0000 7fc1677aa740 1 client.9135 shutdown 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.702+0000 7fc1677aa740 1 -- 172.21.15.59:0/789846373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 msgr2=0x55bf288ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.702+0000 7fc1677aa740 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf288ff170 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fc15000dd80 tx=0x7fc150001020 comp rx=0 tx=0).stop 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.703+0000 7fc1677aa740 1 -- 172.21.15.59:0/789846373 shutdown_connections 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.703+0000 7fc1677aa740 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf2890bf70 0x55bf2890c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.703+0000 7fc1677aa740 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf2890aa20 0x55bf28921160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.703+0000 7fc1677aa740 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x55bf289217a0 0x55bf28909c40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.703+0000 7fc1677aa740 1 --2- 172.21.15.59:0/789846373 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf28908600 0x55bf288ff170 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.703+0000 7fc1677aa740 1 -- 172.21.15.59:0/789846373 >> 172.21.15.59:0/789846373 conn(0x55bf288d8870 msgr2=0x55bf289065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.703+0000 7fc13d7fa640 0 client.9135 ms_handle_reset on v2:172.21.15.59:6834/1740714701 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.703+0000 7fc1677aa740 1 -- 172.21.15.59:0/789846373 shutdown_connections 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.703+0000 7fc1677aa740 1 -- 172.21.15.59:0/789846373 wait complete. 2024-03-06T07:29:54.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.703+0000 7fc1677aa740 20 client.9135 trim_cache size 0 max 16384 2024-03-06T07:29:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.720+0000 7fc1677aa740 1 Processor -- start 2024-03-06T07:29:54.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.720+0000 7fc1677aa740 1 -- start start 2024-03-06T07:29:54.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf28908600 0x55bf288fea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf2890aa20 0x55bf28903a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890bf70 0x55bf28909c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc1677aa740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1601ff3c0 con 0x55bf2890bf70 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc1677aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc160156920 con 0x55bf2890aa20 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc1677aa740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf2890b490 con 0x55bf28908600 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc15ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890bf70 0x55bf28909c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc15ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890bf70 0x55bf28909c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51200/0 (socket says 172.21.15.59:51200) 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc15ed9d640 1 -- 172.21.15.59:0/3530533449 learned_addr learned my addr 172.21.15.59:0/3530533449 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc15e59c640 1 --2- 172.21.15.59:0/3530533449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf28908600 0x55bf288fea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc15ed9d640 1 -- 172.21.15.59:0/3530533449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf28908600 msgr2=0x55bf288fea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc157fff640 1 --2- 172.21.15.59:0/3530533449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf2890aa20 0x55bf28903a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc15ed9d640 1 --2- 172.21.15.59:0/3530533449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf28908600 0x55bf288fea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc15ed9d640 1 -- 172.21.15.59:0/3530533449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf2890aa20 msgr2=0x55bf28903a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc15ed9d640 1 --2- 172.21.15.59:0/3530533449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf2890aa20 0x55bf28903a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc15ed9d640 1 -- 172.21.15.59:0/3530533449 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf286216b0 con 0x55bf2890bf70 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc15e59c640 1 --2- 172.21.15.59:0/3530533449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf28908600 0x55bf288fea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.721+0000 7fc157fff640 1 --2- 172.21.15.59:0/3530533449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf2890aa20 0x55bf28903a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc15ed9d640 1 --2- 172.21.15.59:0/3530533449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890bf70 0x55bf28909c40 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fc1580095c0 tx=0x7fc158031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e2b974e673e193 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc13f7fe640 1 -- 172.21.15.59:0/3530533449 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158039070 con 0x55bf2890bf70 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc13f7fe640 1 -- 172.21.15.59:0/3530533449 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15803f660 con 0x55bf2890bf70 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc13f7fe640 1 -- 172.21.15.59:0/3530533449 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15803fa50 con 0x55bf2890bf70 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc1677aa740 1 -- 172.21.15.59:0/3530533449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890bf70 msgr2=0x55bf28909c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc1677aa740 1 --2- 172.21.15.59:0/3530533449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890bf70 0x55bf28909c40 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fc1580095c0 tx=0x7fc158031000 comp rx=0 tx=0).stop 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc1677aa740 1 -- 172.21.15.59:0/3530533449 shutdown_connections 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc1677aa740 1 --2- 172.21.15.59:0/3530533449 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890bf70 0x55bf28909c40 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc1677aa740 1 --2- 172.21.15.59:0/3530533449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf2890aa20 0x55bf28903a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc1677aa740 1 --2- 172.21.15.59:0/3530533449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf28908600 0x55bf288fea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc1677aa740 1 -- 172.21.15.59:0/3530533449 >> 172.21.15.59:0/3530533449 conn(0x55bf288d8870 msgr2=0x55bf288dc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc1677aa740 1 -- 172.21.15.59:0/3530533449 shutdown_connections 2024-03-06T07:29:54.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.722+0000 7fc1677aa740 1 -- 172.21.15.59:0/3530533449 wait complete. 2024-03-06T07:29:54.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.723+0000 7fc1677aa740 1 -- start start 2024-03-06T07:29:54.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.723+0000 7fc1677aa740 1 Processor -- start 2024-03-06T07:29:54.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf28908600 0x55bf288dd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890aa20 0x55bf288ddc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc1677aa740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf2890bf70 0x55bf28926ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc1677aa740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf2890a180 con 0x55bf2890aa20 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc1677aa740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf28900430 con 0x55bf28908600 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc1677aa740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf2898a1c0 con 0x55bf2890bf70 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc1677aa740 10 client.0 _ll_register_callbacks cb 0x7fc15000da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc157fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890aa20 0x55bf288ddc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc157fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890aa20 0x55bf288ddc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51216/0 (socket says 172.21.15.59:51216) 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc157fff640 1 -- 172.21.15.59:0/1542655567 learned_addr learned my addr 172.21.15.59:0/1542655567 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc15ed9d640 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf2890bf70 0x55bf28926ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc15e59c640 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf28908600 0x55bf288dd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc157fff640 1 -- 172.21.15.59:0/1542655567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf2890bf70 msgr2=0x55bf28926ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc157fff640 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf2890bf70 0x55bf28926ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc157fff640 1 -- 172.21.15.59:0/1542655567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf28908600 msgr2=0x55bf288dd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc157fff640 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf28908600 0x55bf288dd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc157fff640 1 -- 172.21.15.59:0/1542655567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf28940700 con 0x55bf2890aa20 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc157fff640 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890aa20 0x55bf288ddc30 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fc1480582a0 tx=0x7fc148009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f165fe3e78994351 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc13d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.724+0000 7fc13d7fa640 1 -- 172.21.15.59:0/1542655567 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14801ad90 con 0x55bf2890aa20 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc13d7fa640 1 -- 172.21.15.59:0/1542655567 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc148054c10 con 0x55bf2890aa20 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc1677aa740 1 -- 172.21.15.59:0/1542655567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc15803f660 con 0x55bf2890aa20 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc13d7fa640 1 -- 172.21.15.59:0/1542655567 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1480579f0 con 0x55bf2890aa20 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc1677aa740 10 client.9141 fetch_fsmap learned FSMap version 90 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc1677aa740 1 -- 172.21.15.59:0/1542655567 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x55bf28994010 con 0x55bf2890aa20 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc13d7fa640 1 -- 172.21.15.59:0/1542655567 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15803f660 con 0x55bf2890aa20 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc13d7fa640 1 -- 172.21.15.59:0/1542655567 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 90) v1 ==== 2172+0+0 (secure 0 0 0) 0x7fc148057b90 con 0x55bf2890aa20 2024-03-06T07:29:54.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc1677aa740 10 client.9141 fetch_fsmap finished waiting for FSMap version 90 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc1677aa740 10 client.9141 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc1677aa740 20 client.9141 populate_metadata read hostname 'smithi059' 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc1677aa740 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x55bf28909c40 0x55bf28927010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc1677aa740 4 client.9141 mds_command: new command op to 8676 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc1677aa740 1 -- 172.21.15.59:0/1542655567 --> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bf28849f40 con 0x55bf28909c40 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc15e59c640 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x55bf28909c40 0x55bf28927010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc13d7fa640 10 client.9141 ms_handle_connect on v2:172.21.15.59:6834/1740714701 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.725+0000 7fc15e59c640 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x55bf28909c40 0x55bf28927010 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eab17ac4a923cd2c server_cookie=e34a424d33684c89 in_seq=0 out_seq=0 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc13d7fa640 1 -- 172.21.15.59:0/1542655567 <== mds.0 v2:172.21.15.59:6834/1740714701 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55bf28849f40 con 0x55bf28909c40 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc13d7fa640 10 client.9141 handle_command_reply: tid=0 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 1 client.9141 shutdown 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 1 -- 172.21.15.59:0/1542655567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890aa20 msgr2=0x55bf288ddc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890aa20 0x55bf288ddc30 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fc1480582a0 tx=0x7fc148009040 comp rx=0 tx=0).stop 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 1 -- 172.21.15.59:0/1542655567 shutdown_connections 2024-03-06T07:29:54.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bf2890bf70 0x55bf28926ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.59:6834/1740714701,v1:172.21.15.59:6835/1740714701] conn(0x55bf28909c40 0x55bf28927010 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55bf2890aa20 0x55bf288ddc30 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc13d7fa640 0 client.9141 ms_handle_reset on v2:172.21.15.59:6834/1740714701 2024-03-06T07:29:54.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 1 --2- 172.21.15.59:0/1542655567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bf28908600 0x55bf288dd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 1 -- 172.21.15.59:0/1542655567 >> 172.21.15.59:0/1542655567 conn(0x55bf288d8870 msgr2=0x55bf28909050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 1 -- 172.21.15.59:0/1542655567 shutdown_connections 2024-03-06T07:29:54.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 1 -- 172.21.15.59:0/1542655567 wait complete. 2024-03-06T07:29:54.734 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc1677aa740 20 client.9141 trim_cache size 0 max 16384 2024-03-06T07:29:54.734 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:29:54.734 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:29:54.734 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:29:54.735 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 8676, 2024-03-06T07:29:54.735 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:29:54.735 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:29:54.735 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:29:54.735 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 71.802427850000001, 2024-03-06T07:29:54.735 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 89, 2024-03-06T07:29:54.735 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 130, 2024-03-06T07:29:54.735 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 130, 2024-03-06T07:29:54.735 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 77.136354510999993 2024-03-06T07:29:54.735 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:29:54.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc164bc7640 1 -- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc14006b4d0 msgr2=0x7fc14006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc164bc7640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc14006b4d0 0x7fc14006d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc150007fd0 tx=0x7fc150009040 comp rx=0 tx=0).stop 2024-03-06T07:29:54.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc164bc7640 1 -- 172.21.15.59:0/3438182674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1600b4c00 msgr2=0x7fc16006dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.726+0000 7fc164bc7640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1600b4c00 0x7fc16006dc60 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fc14800c460 tx=0x7fc14800c920 comp rx=0 tx=0).stop 2024-03-06T07:29:54.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.727+0000 7fc164bc7640 1 -- 172.21.15.59:0/3438182674 shutdown_connections 2024-03-06T07:29:54.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.727+0000 7fc164bc7640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc14006b4d0 0x7fc14006d990 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.727+0000 7fc164bc7640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1600b4fe0 0x7fc160148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.727+0000 7fc164bc7640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1600b4c00 0x7fc16006dc60 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.727+0000 7fc164bc7640 1 --2- 172.21.15.59:0/3438182674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1600b2fb0 0x7fc16006cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.727+0000 7fc164bc7640 1 -- 172.21.15.59:0/3438182674 >> 172.21.15.59:0/3438182674 conn(0x7fc16009a9f0 msgr2=0x7fc16006ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.727+0000 7fc164bc7640 1 -- 172.21.15.59:0/3438182674 shutdown_connections 2024-03-06T07:29:54.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54.727+0000 7fc164bc7640 1 -- 172.21.15.59:0/3438182674 wait complete. 2024-03-06T07:29:54.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:54 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-06T07:29:55.941 INFO:teuthology.orchestra.run.smithi059.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:55.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:29:55.951 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:55.951 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:56.035 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:56.084 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:56.085 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:29:56.085 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:56.100 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:56.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.240+0000 7f358e963640 1 Processor -- start 2024-03-06T07:29:56.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.240+0000 7f358e963640 1 -- start start 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358e963640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f35880d22b0 0x7f35880d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358e963640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f35880b41f0 0x7f35880b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358e963640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35880b3430 con 0x7f358815dfe0 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358e963640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35880d2c50 con 0x7f35880b41f0 2024-03-06T07:29:56.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358e963640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35880b31e0 con 0x7f35880d22b0 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358cf00640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f35880b41f0 0x7f35880b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358cf00640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f35880b41f0 0x7f35880b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51220/0 (socket says 172.21.15.59:51220) 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358cf00640 1 -- 172.21.15.59:0/4075577311 learned_addr learned my addr 172.21.15.59:0/4075577311 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f35877fe640 1 --2- 172.21.15.59:0/4075577311 >> v2:172.21.15.164:3300/0 conn(0x7f35880d22b0 0x7f35880d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358cf00640 1 -- 172.21.15.59:0/4075577311 >> v2:172.21.15.164:3300/0 conn(0x7f35880d22b0 msgr2=0x7f35880d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358cf00640 1 --2- 172.21.15.59:0/4075577311 >> v2:172.21.15.164:3300/0 conn(0x7f35880d22b0 0x7f35880d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358cf00640 1 -- 172.21.15.59:0/4075577311 >> v1:172.21.15.164:6789/0 conn(0x7f358815dfe0 legacy=0x7f35880cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.242+0000 7f358cf00640 1 -- 172.21.15.59:0/4075577311 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35880b3710 con 0x7f35880b41f0 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.243+0000 7f358cf00640 1 --2- 172.21.15.59:0/4075577311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35880b41f0 0x7f35880b4680 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f3580009e50 tx=0x7f358002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2306487d56e9925b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.243+0000 7f3586ffd640 1 -- 172.21.15.59:0/4075577311 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358002ca80 con 0x7f35880b41f0 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.243+0000 7f3586ffd640 1 -- 172.21.15.59:0/4075577311 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358002cc20 con 0x7f35880b41f0 2024-03-06T07:29:56.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.243+0000 7f3586ffd640 1 -- 172.21.15.59:0/4075577311 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35800343f0 con 0x7f35880b41f0 2024-03-06T07:29:56.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.243+0000 7f358e963640 1 -- 172.21.15.59:0/4075577311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35880b41f0 msgr2=0x7f35880b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.243+0000 7f358e963640 1 --2- 172.21.15.59:0/4075577311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35880b41f0 0x7f35880b4680 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f3580009e50 tx=0x7f358002bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.244+0000 7f358e963640 1 -- 172.21.15.59:0/4075577311 shutdown_connections 2024-03-06T07:29:56.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.244+0000 7f358e963640 1 --2- 172.21.15.59:0/4075577311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35880b41f0 0x7f35880b4680 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.244+0000 7f358e963640 1 --2- 172.21.15.59:0/4075577311 >> v2:172.21.15.164:3300/0 conn(0x7f35880d22b0 0x7f35880d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.244+0000 7f358e963640 1 -- 172.21.15.59:0/4075577311 >> 172.21.15.59:0/4075577311 conn(0x7f35880d3cd0 msgr2=0x7f35880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.244+0000 7f358e963640 1 -- 172.21.15.59:0/4075577311 shutdown_connections 2024-03-06T07:29:56.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.244+0000 7f358e963640 1 -- 172.21.15.59:0/4075577311 wait complete. 2024-03-06T07:29:56.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.245+0000 7f358e963640 1 Processor -- start 2024-03-06T07:29:56.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.246+0000 7f358e963640 1 -- start start 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f358e963640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35880b41f0 0x7f35880b8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f358e963640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35880d22b0 0x7f35880b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f358e963640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f358815dfe0 0x7f35880b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f358e963640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35880cfa00 con 0x7f35880d22b0 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f358e963640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35880b3d60 con 0x7f358815dfe0 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f358e963640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35880d1c70 con 0x7f35880b41f0 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f35877fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35880d22b0 0x7f35880b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f35877fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35880d22b0 0x7f35880b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51234/0 (socket says 172.21.15.59:51234) 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f35877fe640 1 -- 172.21.15.59:0/1878709554 learned_addr learned my addr 172.21.15.59:0/1878709554 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f358cf00640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f358815dfe0 0x7f35880b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f3587fff640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35880b41f0 0x7f35880b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f35877fe640 1 -- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35880b41f0 msgr2=0x7f35880b8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f35877fe640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35880b41f0 0x7f35880b8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f35877fe640 1 -- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f358815dfe0 msgr2=0x7f35880b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f35877fe640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f358815dfe0 0x7f35880b6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f35877fe640 1 -- 172.21.15.59:0/1878709554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35880b9be0 con 0x7f35880d22b0 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f3587fff640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35880b41f0 0x7f35880b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.247+0000 7f35877fe640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35880d22b0 0x7f35880b68a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f357000d830 tx=0x7f357000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.248+0000 7f3584ff9640 1 -- 172.21.15.59:0/1878709554 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357000b6c0 con 0x7f35880d22b0 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.248+0000 7f3584ff9640 1 -- 172.21.15.59:0/1878709554 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3570019070 con 0x7f35880d22b0 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.248+0000 7f358e963640 1 -- 172.21.15.59:0/1878709554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35880b9060 con 0x7f35880d22b0 2024-03-06T07:29:56.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.248+0000 7f3584ff9640 1 -- 172.21.15.59:0/1878709554 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357000ba80 con 0x7f35880d22b0 2024-03-06T07:29:56.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.248+0000 7f358e963640 1 -- 172.21.15.59:0/1878709554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35880b9620 con 0x7f35880d22b0 2024-03-06T07:29:56.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.249+0000 7f358e963640 1 -- 172.21.15.59:0/1878709554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358804f330 con 0x7f35880d22b0 2024-03-06T07:29:56.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.249+0000 7f3584ff9640 1 -- 172.21.15.59:0/1878709554 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f357001e460 con 0x7f35880d22b0 2024-03-06T07:29:56.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.250+0000 7f3584ff9640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f356806b500 0x7f356806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.250+0000 7f3584ff9640 1 -- 172.21.15.59:0/1878709554 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f357008a840 con 0x7f35880d22b0 2024-03-06T07:29:56.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.250+0000 7f3587fff640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f356806b500 0x7f356806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.251+0000 7f3587fff640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f356806b500 0x7f356806d9c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3578006fd0 tx=0x7f3578008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.254+0000 7f3584ff9640 1 -- 172.21.15.59:0/1878709554 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3570059680 con 0x7f35880d22b0 2024-03-06T07:29:56.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.441+0000 7f358e963640 1 -- 172.21.15.59:0/1878709554 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35880ce7a0 con 0x7f35880d22b0 2024-03-06T07:29:56.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.442+0000 7f3584ff9640 1 -- 172.21.15.59:0/1878709554 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16708 (secure 0 0 0) 0x7f3570058f80 con 0x7f35880d22b0 2024-03-06T07:29:56.444 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:56.444 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":130,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:29:39.117588+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:40.558382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:41.255127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.3","mappings":[{"from":2,"to":0}]},{"pgid":"19.25","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:56.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 -- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f356806b500 msgr2=0x7f356806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f356806b500 0x7f356806d9c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3578006fd0 tx=0x7f3578008040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 -- 172.21.15.59:0/1878709554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35880d22b0 msgr2=0x7f35880b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35880d22b0 0x7f35880b68a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f357000d830 tx=0x7f357000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 -- 172.21.15.59:0/1878709554 shutdown_connections 2024-03-06T07:29:56.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f356806b500 0x7f356806d9c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f358815dfe0 0x7f35880b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35880d22b0 0x7f35880b68a0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 --2- 172.21.15.59:0/1878709554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35880b41f0 0x7f35880b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 -- 172.21.15.59:0/1878709554 >> 172.21.15.59:0/1878709554 conn(0x7f35880d3cd0 msgr2=0x7f35880d0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 -- 172.21.15.59:0/1878709554 shutdown_connections 2024-03-06T07:29:56.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.444+0000 7f358e963640 1 -- 172.21.15.59:0/1878709554 wait complete. 2024-03-06T07:29:56.457 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:40.558382+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:41.255127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:56.458 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:56.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.585+0000 7f4b8e89a640 1 Processor -- start 2024-03-06T07:29:56.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.585+0000 7f4b8e89a640 1 -- start start 2024-03-06T07:29:56.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b8e89a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4b8815bbb0 0x7f4b8806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b8e89a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b8815ee10 0x7f4b88071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b8e89a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b880cea90 con 0x7f4b8806b4f0 2024-03-06T07:29:56.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b8e89a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b88161970 con 0x7f4b8815bbb0 2024-03-06T07:29:56.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b8e89a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b881624f0 con 0x7f4b8815ee10 2024-03-06T07:29:56.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b87fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4b8815bbb0 0x7f4b8806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b87fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4b8815bbb0 0x7f4b8806afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51240/0 (socket says 172.21.15.59:51240) 2024-03-06T07:29:56.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b87fff640 1 -- 172.21.15.59:0/1790570881 learned_addr learned my addr 172.21.15.59:0/1790570881 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:56.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b8ce37640 1 --2- 172.21.15.59:0/1790570881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b8815ee10 0x7f4b88071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b87fff640 1 -- 172.21.15.59:0/1790570881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b8815ee10 msgr2=0x7f4b88071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b87fff640 1 --2- 172.21.15.59:0/1790570881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b8815ee10 0x7f4b88071ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b87fff640 1 -- 172.21.15.59:0/1790570881 >> v1:172.21.15.164:6789/0 conn(0x7f4b8806b4f0 legacy=0x7f4b8815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b87fff640 1 -- 172.21.15.59:0/1790570881 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b88161bc0 con 0x7f4b8815bbb0 2024-03-06T07:29:56.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.586+0000 7f4b8ce37640 1 --2- 172.21.15.59:0/1790570881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b8815ee10 0x7f4b88071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:56.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.587+0000 7f4b87fff640 1 --2- 172.21.15.59:0/1790570881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b8815bbb0 0x7f4b8806afb0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009870 tx=0x7f4b7c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967afe966fd700e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.588+0000 7f4b86ffd640 1 -- 172.21.15.59:0/1790570881 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c035020 con 0x7f4b8815bbb0 2024-03-06T07:29:56.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.588+0000 7f4b86ffd640 1 -- 172.21.15.59:0/1790570881 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c031640 con 0x7f4b8815bbb0 2024-03-06T07:29:56.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.588+0000 7f4b86ffd640 1 -- 172.21.15.59:0/1790570881 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c02d040 con 0x7f4b8815bbb0 2024-03-06T07:29:56.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.588+0000 7f4b8e89a640 1 -- 172.21.15.59:0/1790570881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b8815bbb0 msgr2=0x7f4b8806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.588+0000 7f4b8e89a640 1 --2- 172.21.15.59:0/1790570881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b8815bbb0 0x7f4b8806afb0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009870 tx=0x7f4b7c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.588+0000 7f4b8e89a640 1 -- 172.21.15.59:0/1790570881 shutdown_connections 2024-03-06T07:29:56.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.588+0000 7f4b8e89a640 1 --2- 172.21.15.59:0/1790570881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b8815ee10 0x7f4b88071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.588+0000 7f4b8e89a640 1 --2- 172.21.15.59:0/1790570881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b8815bbb0 0x7f4b8806afb0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.588+0000 7f4b8e89a640 1 -- 172.21.15.59:0/1790570881 >> 172.21.15.59:0/1790570881 conn(0x7f4b880b2660 msgr2=0x7f4b880b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.588+0000 7f4b8e89a640 1 -- 172.21.15.59:0/1790570881 shutdown_connections 2024-03-06T07:29:56.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.588+0000 7f4b8e89a640 1 -- 172.21.15.59:0/1790570881 wait complete. 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.589+0000 7f4b8e89a640 1 Processor -- start 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.590+0000 7f4b8e89a640 1 -- start start 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.590+0000 7f4b8e89a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b8806b4f0 0x7f4b880cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.590+0000 7f4b8e89a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b8815ee10 0x7f4b88160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.590+0000 7f4b8e89a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b881600b0 0x7f4b881f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.590+0000 7f4b8e89a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b880721e0 con 0x7f4b881600b0 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.590+0000 7f4b8e89a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b88162180 con 0x7f4b8806b4f0 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.590+0000 7f4b8e89a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b88072360 con 0x7f4b8815ee10 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.590+0000 7f4b8ce37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b881600b0 0x7f4b881f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.590+0000 7f4b8ce37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b881600b0 0x7f4b881f3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51254/0 (socket says 172.21.15.59:51254) 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.590+0000 7f4b8ce37640 1 -- 172.21.15.59:0/2282221690 learned_addr learned my addr 172.21.15.59:0/2282221690 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:56.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b87fff640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b8806b4f0 0x7f4b880cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b877fe640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b8815ee10 0x7f4b88160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b8ce37640 1 -- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b8815ee10 msgr2=0x7f4b88160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b8ce37640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b8815ee10 0x7f4b88160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b8ce37640 1 -- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b8806b4f0 msgr2=0x7f4b880cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b8ce37640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b8806b4f0 0x7f4b880cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b8ce37640 1 -- 172.21.15.59:0/2282221690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b880cb750 con 0x7f4b881600b0 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b87fff640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b8806b4f0 0x7f4b880cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b8ce37640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b881600b0 0x7f4b881f3d80 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000cd50 tx=0x7f4b8000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b84ff9640 1 -- 172.21.15.59:0/2282221690 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8000f680 con 0x7f4b881600b0 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b84ff9640 1 -- 172.21.15.59:0/2282221690 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8000f840 con 0x7f4b881600b0 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b8e89a640 1 -- 172.21.15.59:0/2282221690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b880cba30 con 0x7f4b881600b0 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b84ff9640 1 -- 172.21.15.59:0/2282221690 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8000fc30 con 0x7f4b881600b0 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.591+0000 7f4b8e89a640 1 -- 172.21.15.59:0/2282221690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b880cbff0 con 0x7f4b881600b0 2024-03-06T07:29:56.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.592+0000 7f4b8e89a640 1 -- 172.21.15.59:0/2282221690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b880733b0 con 0x7f4b881600b0 2024-03-06T07:29:56.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.593+0000 7f4b84ff9640 1 -- 172.21.15.59:0/2282221690 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b800213f0 con 0x7f4b881600b0 2024-03-06T07:29:56.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.593+0000 7f4b84ff9640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b7406b4d0 0x7f4b7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.593+0000 7f4b84ff9640 1 -- 172.21.15.59:0/2282221690 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4b8008af80 con 0x7f4b881600b0 2024-03-06T07:29:56.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.593+0000 7f4b87fff640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b7406b4d0 0x7f4b7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.594+0000 7f4b87fff640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b7406b4d0 0x7f4b7406d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c036000 tx=0x7f4b7c03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.597+0000 7f4b84ff9640 1 -- 172.21.15.59:0/2282221690 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b80014030 con 0x7f4b881600b0 2024-03-06T07:29:56.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.777+0000 7f4b8e89a640 1 -- 172.21.15.59:0/2282221690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4b88068ff0 con 0x7f4b881600b0 2024-03-06T07:29:56.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.778+0000 7f4b84ff9640 1 -- 172.21.15.59:0/2282221690 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4b80059e50 con 0x7f4b881600b0 2024-03-06T07:29:56.780 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:29:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 -- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b7406b4d0 msgr2=0x7f4b7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b7406b4d0 0x7f4b7406d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c036000 tx=0x7f4b7c03c040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 -- 172.21.15.59:0/2282221690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b881600b0 msgr2=0x7f4b881f3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b881600b0 0x7f4b881f3d80 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000cd50 tx=0x7f4b8000a650 comp rx=0 tx=0).stop 2024-03-06T07:29:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 -- 172.21.15.59:0/2282221690 shutdown_connections 2024-03-06T07:29:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b7406b4d0 0x7f4b7406d990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b881600b0 0x7f4b881f3d80 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b8815ee10 0x7f4b88160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 --2- 172.21.15.59:0/2282221690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b8806b4f0 0x7f4b880cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 -- 172.21.15.59:0/2282221690 >> 172.21.15.59:0/2282221690 conn(0x7f4b880b2660 msgr2=0x7f4b880726f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 -- 172.21.15.59:0/2282221690 shutdown_connections 2024-03-06T07:29:56.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.780+0000 7f4b8e89a640 1 -- 172.21.15.59:0/2282221690 wait complete. 2024-03-06T07:29:56.793 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:56.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.895+0000 7f41f2084640 1 Processor -- start 2024-03-06T07:29:56.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.895+0000 7f41f2084640 1 -- start start 2024-03-06T07:29:56.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.896+0000 7f41f2084640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f41ec15c050 0x7f41ec06f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.896+0000 7f41f2084640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec0cd4b0 con 0x7f41ec06fb40 2024-03-06T07:29:56.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.896+0000 7f41f2084640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec0cd790 con 0x7f41ec0d0af0 2024-03-06T07:29:56.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.896+0000 7f41f2084640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41ec06ffb0 con 0x7f41ec15c050 2024-03-06T07:29:56.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.896+0000 7f41eaffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f41ec06fb40 0x7f41ec0cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60286/0 (socket says 172.21.15.59:60286) 2024-03-06T07:29:56.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.896+0000 7f41eaffd640 1 -- 172.21.15.59:0/157040349 learned_addr learned my addr 172.21.15.59:0/157040349 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:56.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.896+0000 7f41eb7fe640 1 --2- 172.21.15.59:0/157040349 >> v2:172.21.15.164:3300/0 conn(0x7f41ec15c050 0x7f41ec06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.896+0000 7f41ea7fc640 1 -- 172.21.15.59:0/157040349 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630790359 0 0) 0x7f41ec0cd4b0 con 0x7f41ec06fb40 2024-03-06T07:29:56.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.896+0000 7f41ea7fc640 1 -- 172.21.15.59:0/157040349 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d8003520 con 0x7f41ec06fb40 2024-03-06T07:29:56.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.897+0000 7f41eb7fe640 1 -- 172.21.15.59:0/157040349 >> v1:172.21.15.164:6789/0 conn(0x7f41ec0d0af0 legacy=0x7f41ec165ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.897+0000 7f41ea7fc640 1 -- 172.21.15.59:0/157040349 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1045952479 0 0) 0x7f41d8003520 con 0x7f41ec06fb40 2024-03-06T07:29:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.897+0000 7f41eb7fe640 1 -- 172.21.15.59:0/157040349 >> v1:172.21.15.59:6789/0 conn(0x7f41ec06fb40 legacy=0x7f41ec0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.897+0000 7f41eb7fe640 1 -- 172.21.15.59:0/157040349 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec0cda70 con 0x7f41ec15c050 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.898+0000 7f41eb7fe640 1 --2- 172.21.15.59:0/157040349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec15c050 0x7f41ec06f600 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f41d40097a0 tx=0x7f41d402b600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb344ae4ff74361f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.898+0000 7f41ea7fc640 1 -- 172.21.15.59:0/157040349 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d4037070 con 0x7f41ec15c050 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.898+0000 7f41ea7fc640 1 -- 172.21.15.59:0/157040349 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d4030510 con 0x7f41ec15c050 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.898+0000 7f41ea7fc640 1 -- 172.21.15.59:0/157040349 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc001670 con 0x7f41ec15c050 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.898+0000 7f41f2084640 1 -- 172.21.15.59:0/157040349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec15c050 msgr2=0x7f41ec06f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.898+0000 7f41f2084640 1 --2- 172.21.15.59:0/157040349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec15c050 0x7f41ec06f600 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f41d40097a0 tx=0x7f41d402b600 comp rx=0 tx=0).stop 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.898+0000 7f41f2084640 1 -- 172.21.15.59:0/157040349 shutdown_connections 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.898+0000 7f41f2084640 1 --2- 172.21.15.59:0/157040349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec15c050 0x7f41ec06f600 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.898+0000 7f41f2084640 1 -- 172.21.15.59:0/157040349 >> 172.21.15.59:0/157040349 conn(0x7f41ec0b3800 msgr2=0x7f41ec0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.898+0000 7f41f2084640 1 -- 172.21.15.59:0/157040349 shutdown_connections 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.898+0000 7f41f2084640 1 -- 172.21.15.59:0/157040349 wait complete. 2024-03-06T07:29:56.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.899+0000 7f41f2084640 1 Processor -- start 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.900+0000 7f41f2084640 1 -- start start 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.900+0000 7f41f2084640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41ec06fb40 0x7f41ec1f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.900+0000 7f41f2084640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41ec0d0af0 0x7f41ec156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.900+0000 7f41f2084640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec15c050 0x7f41ec06db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.900+0000 7f41f2084640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec0ccff0 con 0x7f41ec0d0af0 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.900+0000 7f41f2084640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec0cce70 con 0x7f41ec15c050 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.900+0000 7f41f2084640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec0cd170 con 0x7f41ec06fb40 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41eaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41ec0d0af0 0x7f41ec156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41eaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41ec0d0af0 0x7f41ec156a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51256/0 (socket says 172.21.15.59:51256) 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41eaffd640 1 -- 172.21.15.59:0/4230829547 learned_addr learned my addr 172.21.15.59:0/4230829547 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41eb7fe640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41ec06fb40 0x7f41ec1f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41ebfff640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec15c050 0x7f41ec06db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41eaffd640 1 -- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41ec06fb40 msgr2=0x7f41ec1f3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41eaffd640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41ec06fb40 0x7f41ec1f3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41eaffd640 1 -- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec15c050 msgr2=0x7f41ec06db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41eaffd640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec15c050 0x7f41ec06db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41eaffd640 1 -- 172.21.15.59:0/4230829547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec1f4420 con 0x7f41ec0d0af0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41eaffd640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41ec0d0af0 0x7f41ec156a40 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f41dc00c480 tx=0x7f41dc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41e3fff640 1 -- 172.21.15.59:0/4230829547 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc018070 con 0x7f41ec0d0af0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41e3fff640 1 -- 172.21.15.59:0/4230829547 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41dc0074e0 con 0x7f41ec0d0af0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41e3fff640 1 -- 172.21.15.59:0/4230829547 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41dc004a20 con 0x7f41ec0d0af0 2024-03-06T07:29:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41f2084640 1 -- 172.21.15.59:0/4230829547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ec1f4700 con 0x7f41ec0d0af0 2024-03-06T07:29:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.901+0000 7f41f2084640 1 -- 172.21.15.59:0/4230829547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec1f4cc0 con 0x7f41ec0d0af0 2024-03-06T07:29:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.902+0000 7f41f2084640 1 -- 172.21.15.59:0/4230829547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ec09fe60 con 0x7f41ec0d0af0 2024-03-06T07:29:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.903+0000 7f41e3fff640 1 -- 172.21.15.59:0/4230829547 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f41dc01b070 con 0x7f41ec0d0af0 2024-03-06T07:29:56.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.903+0000 7f41e3fff640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41cc06b4d0 0x7f41cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.903+0000 7f41e3fff640 1 -- 172.21.15.59:0/4230829547 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f41dc089640 con 0x7f41ec0d0af0 2024-03-06T07:29:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.903+0000 7f41eb7fe640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41cc06b4d0 0x7f41cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.904+0000 7f41eb7fe640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41cc06b4d0 0x7f41cc06d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f41d4034000 tx=0x7f41d4008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:56.907+0000 7f41e3fff640 1 -- 172.21.15.59:0/4230829547 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41dc058470 con 0x7f41ec0d0af0 2024-03-06T07:29:57.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.097+0000 7f41f2084640 1 -- 172.21.15.59:0/4230829547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f41ec0b1120 con 0x7f41ec0d0af0 2024-03-06T07:29:57.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.098+0000 7f41e3fff640 1 -- 172.21.15.59:0/4230829547 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) v1 ==== 104+0+11 (secure 0 0 0) 0x7f41dc057d70 con 0x7f41ec0d0af0 2024-03-06T07:29:57.100 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:29:57.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 -- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41cc06b4d0 msgr2=0x7f41cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41cc06b4d0 0x7f41cc06d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f41d4034000 tx=0x7f41d4008030 comp rx=0 tx=0).stop 2024-03-06T07:29:57.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 -- 172.21.15.59:0/4230829547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41ec0d0af0 msgr2=0x7f41ec156a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41ec0d0af0 0x7f41ec156a40 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f41dc00c480 tx=0x7f41dc00c940 comp rx=0 tx=0).stop 2024-03-06T07:29:57.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 -- 172.21.15.59:0/4230829547 shutdown_connections 2024-03-06T07:29:57.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f41cc06b4d0 0x7f41cc06d990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec15c050 0x7f41ec06db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41ec0d0af0 0x7f41ec156a40 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 --2- 172.21.15.59:0/4230829547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41ec06fb40 0x7f41ec1f3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 -- 172.21.15.59:0/4230829547 >> 172.21.15.59:0/4230829547 conn(0x7f41ec0b3800 msgr2=0x7f41ec0d23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 -- 172.21.15.59:0/4230829547 shutdown_connections 2024-03-06T07:29:57.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.100+0000 7f41f2084640 1 -- 172.21.15.59:0/4230829547 wait complete. 2024-03-06T07:29:57.116 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:57.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.219+0000 7fb771ac2640 1 Processor -- start 2024-03-06T07:29:57.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.219+0000 7fb771ac2640 1 -- start start 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.220+0000 7fb771ac2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb76c0cea50 0x7fb76c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.220+0000 7fb771ac2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb76c06cab0 0x7fb76c0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.220+0000 7fb771ac2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c06d030 con 0x7fb76c15def0 2024-03-06T07:29:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.220+0000 7fb771ac2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb76c0cf1c0 con 0x7fb76c0cea50 2024-03-06T07:29:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.220+0000 7fb771ac2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb76c0cc760 con 0x7fb76c06cab0 2024-03-06T07:29:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.220+0000 7fb76affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb76c0cea50 0x7fb76c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.220+0000 7fb76affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb76c0cea50 0x7fb76c0b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51262/0 (socket says 172.21.15.59:51262) 2024-03-06T07:29:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.220+0000 7fb76affd640 1 -- 172.21.15.59:0/248119192 learned_addr learned my addr 172.21.15.59:0/248119192 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.220+0000 7fb76affd640 1 -- 172.21.15.59:0/248119192 >> v2:172.21.15.164:3300/0 conn(0x7fb76c06cab0 msgr2=0x7fb76c0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.220+0000 7fb76affd640 1 --2- 172.21.15.59:0/248119192 >> v2:172.21.15.164:3300/0 conn(0x7fb76c06cab0 0x7fb76c0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.220+0000 7fb76affd640 1 -- 172.21.15.59:0/248119192 >> v1:172.21.15.164:6789/0 conn(0x7fb76c15def0 legacy=0x7fb76c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.221+0000 7fb76affd640 1 -- 172.21.15.59:0/248119192 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c06d310 con 0x7fb76c0cea50 2024-03-06T07:29:57.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.221+0000 7fb76affd640 1 --2- 172.21.15.59:0/248119192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb76c0cea50 0x7fb76c0b41b0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0729a0 tx=0x7fb75402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b21ac6c130ecec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.221+0000 7fb76a7fc640 1 -- 172.21.15.59:0/248119192 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb754035020 con 0x7fb76c0cea50 2024-03-06T07:29:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.221+0000 7fb76a7fc640 1 -- 172.21.15.59:0/248119192 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb754031870 con 0x7fb76c0cea50 2024-03-06T07:29:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.221+0000 7fb76a7fc640 1 -- 172.21.15.59:0/248119192 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb754031c40 con 0x7fb76c0cea50 2024-03-06T07:29:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.222+0000 7fb771ac2640 1 -- 172.21.15.59:0/248119192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb76c0cea50 msgr2=0x7fb76c0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.222+0000 7fb771ac2640 1 --2- 172.21.15.59:0/248119192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb76c0cea50 0x7fb76c0b41b0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0729a0 tx=0x7fb75402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.222+0000 7fb771ac2640 1 -- 172.21.15.59:0/248119192 shutdown_connections 2024-03-06T07:29:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.222+0000 7fb771ac2640 1 --2- 172.21.15.59:0/248119192 >> v2:172.21.15.164:3300/0 conn(0x7fb76c06cab0 0x7fb76c0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.222+0000 7fb771ac2640 1 --2- 172.21.15.59:0/248119192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb76c0cea50 0x7fb76c0b41b0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.222+0000 7fb771ac2640 1 -- 172.21.15.59:0/248119192 >> 172.21.15.59:0/248119192 conn(0x7fb76c0b3690 msgr2=0x7fb76c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.222+0000 7fb771ac2640 1 -- 172.21.15.59:0/248119192 shutdown_connections 2024-03-06T07:29:57.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.222+0000 7fb771ac2640 1 -- 172.21.15.59:0/248119192 wait complete. 2024-03-06T07:29:57.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.224+0000 7fb771ac2640 1 Processor -- start 2024-03-06T07:29:57.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.224+0000 7fb771ac2640 1 -- start start 2024-03-06T07:29:57.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.225+0000 7fb771ac2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb76c06cab0 0x7fb76c0c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.225+0000 7fb771ac2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb76c0cea50 0x7fb76c09f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.225+0000 7fb771ac2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb76c15def0 0x7fb76c15b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.225+0000 7fb771ac2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c15e2d0 con 0x7fb76c0cea50 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.225+0000 7fb771ac2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c0cc3f0 con 0x7fb76c06cab0 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.225+0000 7fb771ac2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c15e450 con 0x7fb76c15def0 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.225+0000 7fb76affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb76c0cea50 0x7fb76c09f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.225+0000 7fb76affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb76c0cea50 0x7fb76c09f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51266/0 (socket says 172.21.15.59:51266) 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.225+0000 7fb76affd640 1 -- 172.21.15.59:0/983285710 learned_addr learned my addr 172.21.15.59:0/983285710 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.225+0000 7fb76b7fe640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb76c06cab0 0x7fb76c0c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.225+0000 7fb76bfff640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb76c15def0 0x7fb76c15b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.226+0000 7fb76affd640 1 -- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb76c15def0 msgr2=0x7fb76c15b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.226+0000 7fb76affd640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb76c15def0 0x7fb76c15b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.226+0000 7fb76affd640 1 -- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb76c06cab0 msgr2=0x7fb76c0c7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.226+0000 7fb76affd640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb76c06cab0 0x7fb76c0c7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.226+0000 7fb76affd640 1 -- 172.21.15.59:0/983285710 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c0c8390 con 0x7fb76c0cea50 2024-03-06T07:29:57.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.226+0000 7fb76affd640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb76c0cea50 0x7fb76c09f3d0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fb754000c00 tx=0x7fb7540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.226+0000 7fb763fff640 1 -- 172.21.15.59:0/983285710 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75403ad60 con 0x7fb76c0cea50 2024-03-06T07:29:57.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.226+0000 7fb771ac2640 1 -- 172.21.15.59:0/983285710 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76c0caf90 con 0x7fb76c0cea50 2024-03-06T07:29:57.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.226+0000 7fb763fff640 1 -- 172.21.15.59:0/983285710 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7540443f0 con 0x7fb76c0cea50 2024-03-06T07:29:57.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.226+0000 7fb763fff640 1 -- 172.21.15.59:0/983285710 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7540447e0 con 0x7fb76c0cea50 2024-03-06T07:29:57.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.226+0000 7fb771ac2640 1 -- 172.21.15.59:0/983285710 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76c0cb550 con 0x7fb76c0cea50 2024-03-06T07:29:57.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.227+0000 7fb771ac2640 1 -- 172.21.15.59:0/983285710 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76c09fcf0 con 0x7fb76c0cea50 2024-03-06T07:29:57.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.227+0000 7fb763fff640 1 -- 172.21.15.59:0/983285710 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb75404b060 con 0x7fb76c0cea50 2024-03-06T07:29:57.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.228+0000 7fb763fff640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb74c06b500 0x7fb74c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.228+0000 7fb763fff640 1 -- 172.21.15.59:0/983285710 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb7540ac310 con 0x7fb76c0cea50 2024-03-06T07:29:57.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.228+0000 7fb76b7fe640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb74c06b500 0x7fb74c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.229+0000 7fb76b7fe640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb74c06b500 0x7fb74c06d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb75c006fd0 tx=0x7fb75c009060 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.232+0000 7fb763fff640 1 -- 172.21.15.59:0/983285710 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb76c0c8390 con 0x7fb76c0cea50 2024-03-06T07:29:57.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.413+0000 7fb771ac2640 1 -- 172.21.15.59:0/983285710 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb76c0ce340 con 0x7fb76c0cea50 2024-03-06T07:29:57.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.413+0000 7fb763fff640 1 -- 172.21.15.59:0/983285710 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb75403f070 con 0x7fb76c0cea50 2024-03-06T07:29:57.415 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:29:57.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 -- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb74c06b500 msgr2=0x7fb74c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb74c06b500 0x7fb74c06d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb75c006fd0 tx=0x7fb75c009060 comp rx=0 tx=0).stop 2024-03-06T07:29:57.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 -- 172.21.15.59:0/983285710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb76c0cea50 msgr2=0x7fb76c09f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb76c0cea50 0x7fb76c09f3d0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fb754000c00 tx=0x7fb7540046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 -- 172.21.15.59:0/983285710 shutdown_connections 2024-03-06T07:29:57.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb74c06b500 0x7fb74c06d9c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb76c15def0 0x7fb76c15b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb76c0cea50 0x7fb76c09f3d0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 --2- 172.21.15.59:0/983285710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb76c06cab0 0x7fb76c0c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 -- 172.21.15.59:0/983285710 >> 172.21.15.59:0/983285710 conn(0x7fb76c0b3690 msgr2=0x7fb76c06f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 -- 172.21.15.59:0/983285710 shutdown_connections 2024-03-06T07:29:57.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.415+0000 7fb771ac2640 1 -- 172.21.15.59:0/983285710 wait complete. 2024-03-06T07:29:57.430 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:57.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.531+0000 7f082e2fb640 1 Processor -- start 2024-03-06T07:29:57.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.531+0000 7f082e2fb640 1 -- start start 2024-03-06T07:29:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f082e2fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 0x7f08280d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f082e2fb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f08280cf7d0 0x7f08280d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f082e2fb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08280d25e0 con 0x7f08280d4810 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f082e2fb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08280c7c50 con 0x7f08280cf7d0 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f082e2fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08280d2880 con 0x7f082815e000 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f082c898640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f08280cf7d0 0x7f08280d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f081ffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f08280d4810 0x7f08280c80e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60292/0 (socket says 172.21.15.59:60292) 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f081ffff640 1 -- 172.21.15.59:0/1111580118 learned_addr learned my addr 172.21.15.59:0/1111580118 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f0827fff640 1 --2- 172.21.15.59:0/1111580118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 0x7f08280d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f082c898640 1 -- 172.21.15.59:0/1111580118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 msgr2=0x7f08280d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f082c898640 1 --2- 172.21.15.59:0/1111580118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 0x7f08280d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f082c898640 1 -- 172.21.15.59:0/1111580118 >> v1:172.21.15.59:6789/0 conn(0x7f08280d4810 legacy=0x7f08280c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f082c898640 1 -- 172.21.15.59:0/1111580118 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08280d2b40 con 0x7f08280cf7d0 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f08277fe640 1 -- 172.21.15.59:0/1111580118 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953847532 0 0) 0x7f08280d25e0 con 0x7f08280d4810 2024-03-06T07:29:57.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.532+0000 7f0827fff640 1 --2- 172.21.15.59:0/1111580118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 0x7f08280d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.533+0000 7f082c898640 1 --2- 172.21.15.59:0/1111580118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08280cf7d0 0x7f08280d0bc0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f0820009d90 tx=0x7f082002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5adadf9a48b10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.533+0000 7f08277fe640 1 -- 172.21.15.59:0/1111580118 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0820039070 con 0x7f08280cf7d0 2024-03-06T07:29:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.533+0000 7f08277fe640 1 -- 172.21.15.59:0/1111580118 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082002cce0 con 0x7f08280cf7d0 2024-03-06T07:29:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.533+0000 7f08277fe640 1 -- 172.21.15.59:0/1111580118 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0820034660 con 0x7f08280cf7d0 2024-03-06T07:29:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.534+0000 7f082e2fb640 1 -- 172.21.15.59:0/1111580118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08280cf7d0 msgr2=0x7f08280d0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.534+0000 7f082e2fb640 1 --2- 172.21.15.59:0/1111580118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08280cf7d0 0x7f08280d0bc0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f0820009d90 tx=0x7f082002bf30 comp rx=0 tx=0).stop 2024-03-06T07:29:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.534+0000 7f082e2fb640 1 -- 172.21.15.59:0/1111580118 shutdown_connections 2024-03-06T07:29:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.534+0000 7f082e2fb640 1 --2- 172.21.15.59:0/1111580118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08280cf7d0 0x7f08280d0bc0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.534+0000 7f082e2fb640 1 --2- 172.21.15.59:0/1111580118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 0x7f08280d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.534+0000 7f082e2fb640 1 -- 172.21.15.59:0/1111580118 >> 172.21.15.59:0/1111580118 conn(0x7f08280d3cf0 msgr2=0x7f08280d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.534+0000 7f082e2fb640 1 -- 172.21.15.59:0/1111580118 shutdown_connections 2024-03-06T07:29:57.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.534+0000 7f082e2fb640 1 -- 172.21.15.59:0/1111580118 wait complete. 2024-03-06T07:29:57.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.535+0000 7f082e2fb640 1 Processor -- start 2024-03-06T07:29:57.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.535+0000 7f082e2fb640 1 -- start start 2024-03-06T07:29:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f082e2fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08280cf7d0 0x7f082814c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f082e2fb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08280d4810 0x7f08280b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f082e2fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 0x7f08280b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f082e2fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082815e3e0 con 0x7f08280cf7d0 2024-03-06T07:29:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f082e2fb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08280d22d0 con 0x7f08280d4810 2024-03-06T07:29:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f082e2fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08280c87f0 con 0x7f082815e000 2024-03-06T07:29:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f0827fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08280cf7d0 0x7f082814c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f0827fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08280cf7d0 0x7f082814c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51288/0 (socket says 172.21.15.59:51288) 2024-03-06T07:29:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f0827fff640 1 -- 172.21.15.59:0/3593494170 learned_addr learned my addr 172.21.15.59:0/3593494170 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f082c898640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 0x7f08280b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f081ffff640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08280d4810 0x7f08280b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f0827fff640 1 -- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 msgr2=0x7f08280b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f0827fff640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 0x7f08280b5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f0827fff640 1 -- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08280d4810 msgr2=0x7f08280b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f0827fff640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08280d4810 0x7f08280b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f0827fff640 1 -- 172.21.15.59:0/3593494170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0828151ed0 con 0x7f08280cf7d0 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f082c898640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 0x7f08280b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.536+0000 7f0827fff640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08280cf7d0 0x7f082814c410 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f081800c820 tx=0x7f081800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.537+0000 7f08257fa640 1 -- 172.21.15.59:0/3593494170 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818015070 con 0x7f08280cf7d0 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.537+0000 7f08257fa640 1 -- 172.21.15.59:0/3593494170 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0818007a30 con 0x7f08280cf7d0 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.537+0000 7f08257fa640 1 -- 172.21.15.59:0/3593494170 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818007de0 con 0x7f08280cf7d0 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.537+0000 7f082e2fb640 1 -- 172.21.15.59:0/3593494170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0828150e30 con 0x7f08280cf7d0 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.537+0000 7f082e2fb640 1 -- 172.21.15.59:0/3593494170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08281513f0 con 0x7f08280cf7d0 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.538+0000 7f082e2fb640 1 -- 172.21.15.59:0/3593494170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082804f330 con 0x7f08280cf7d0 2024-03-06T07:29:57.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.538+0000 7f08257fa640 1 -- 172.21.15.59:0/3593494170 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0818018070 con 0x7f08280cf7d0 2024-03-06T07:29:57.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.539+0000 7f08257fa640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f080806b4d0 0x7f080806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.539+0000 7f08257fa640 1 -- 172.21.15.59:0/3593494170 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f081808a0e0 con 0x7f08280cf7d0 2024-03-06T07:29:57.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.539+0000 7f081ffff640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f080806b4d0 0x7f080806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.539+0000 7f081ffff640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f080806b4d0 0x7f080806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f082815cd30 tx=0x7f0810007b50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.543+0000 7f08257fa640 1 -- 172.21.15.59:0/3593494170 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0818058fb0 con 0x7f08280cf7d0 2024-03-06T07:29:57.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.739+0000 7f082e2fb640 1 -- 172.21.15.59:0/3593494170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f08280747a0 con 0x7f08280cf7d0 2024-03-06T07:29:57.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.739+0000 7f08257fa640 1 -- 172.21.15.59:0/3593494170 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v130) v1 ==== 80+0+0 (secure 0 0 0) 0x7f08180588b0 con 0x7f08280cf7d0 2024-03-06T07:29:57.741 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:29:57.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 -- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f080806b4d0 msgr2=0x7f080806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f080806b4d0 0x7f080806d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f082815cd30 tx=0x7f0810007b50 comp rx=0 tx=0).stop 2024-03-06T07:29:57.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 -- 172.21.15.59:0/3593494170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08280cf7d0 msgr2=0x7f082814c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08280cf7d0 0x7f082814c410 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f081800c820 tx=0x7f081800cce0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 -- 172.21.15.59:0/3593494170 shutdown_connections 2024-03-06T07:29:57.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f080806b4d0 0x7f080806d990 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f082815e000 0x7f08280b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08280d4810 0x7f08280b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 --2- 172.21.15.59:0/3593494170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f08280cf7d0 0x7f082814c410 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 -- 172.21.15.59:0/3593494170 >> 172.21.15.59:0/3593494170 conn(0x7f08280d3cf0 msgr2=0x7f082806be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 -- 172.21.15.59:0/3593494170 shutdown_connections 2024-03-06T07:29:57.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:57.741+0000 7f082e2fb640 1 -- 172.21.15.59:0/3593494170 wait complete. 2024-03-06T07:29:57.754 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:29:57.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:57.754 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:57.779 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[78753]: fuse finished with error 0 and tester_r 0 2024-03-06T07:29:57.781 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:29:57.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:57.781 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:29:57.850 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:29:57.850 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:57.868 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:57.868 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:29:57.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:57.868 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:29:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:57.925 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:29:57.925 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:29:57.925 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:57.940 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:57.940 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:29:57.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:57.941 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:29:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:57.995 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:29:57.995 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:29:57.995 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:58.012 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:58.012 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:29:58.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:58.012 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:29:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:58.069 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:29:58.069 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:58.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.168+0000 7f14900b5640 1 Processor -- start 2024-03-06T07:29:58.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.169+0000 7f14900b5640 1 -- start start 2024-03-06T07:29:58.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.169+0000 7f14900b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815fa10 0x7f14880ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.169+0000 7f14900b5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880cebe0 con 0x7f148815bdd0 2024-03-06T07:29:58.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.169+0000 7f14900b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880ceec0 con 0x7f14880d16e0 2024-03-06T07:29:58.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.169+0000 7f14900b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14880cf1a0 con 0x7f148815fa10 2024-03-06T07:29:58.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.169+0000 7f148de51640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f148815bdd0 0x7f14880c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60304/0 (socket says 172.21.15.59:60304) 2024-03-06T07:29:58.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.169+0000 7f148de51640 1 -- 172.21.15.59:0/434907909 learned_addr learned my addr 172.21.15.59:0/434907909 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:58.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148d650640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815fa10 0x7f14880ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136873194 0 0) 0x7f14880cebe0 con 0x7f148815bdd0 2024-03-06T07:29:58.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1474003520 con 0x7f148815bdd0 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3774522642 0 0) 0x7f1474003520 con 0x7f148815bdd0 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815fa10 msgr2=0x7f14880ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815fa10 0x7f14880ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 >> v1:172.21.15.164:6789/0 conn(0x7f14880d16e0 legacy=0x7f14880d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880cc8e0 con 0x7f148815bdd0 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f1484001670 con 0x7f148815bdd0 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 >> v1:172.21.15.59:6789/0 conn(0x7f148815bdd0 legacy=0x7f14880c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815fa10 0x7f14740039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1474003fb0 0x7f1474004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1474004d40 0x7f14740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880cf1a0 con 0x7f1474003fb0 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147400bbb0 con 0x7f1474004d40 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147400bdb0 con 0x7f148815fa10 2024-03-06T07:29:58.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148d650640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1474003fb0 0x7f1474004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148de51640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815fa10 0x7f14740039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148e652640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1474004d40 0x7f14740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148d650640 1 -- 172.21.15.59:0/434907909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815fa10 msgr2=0x7f14740039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148d650640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815fa10 0x7f14740039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148d650640 1 -- 172.21.15.59:0/434907909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1474004d40 msgr2=0x7f14740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148d650640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1474004d40 0x7f14740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.170+0000 7f148d650640 1 -- 172.21.15.59:0/434907909 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1474003520 con 0x7f1474003fb0 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.171+0000 7f148d650640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1474003fb0 0x7f1474004800 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f1478009ac0 tx=0x7f1474005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8e8f6b729bd65f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.171+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478039070 con 0x7f1474003fb0 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.171+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147802c6b0 con 0x7f1474003fb0 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.171+0000 7f148ce4f640 1 -- 172.21.15.59:0/434907909 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478009690 con 0x7f1474003fb0 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.172+0000 7f14900b5640 1 -- 172.21.15.59:0/434907909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1474003fb0 msgr2=0x7f1474004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.172+0000 7f14900b5640 1 --2- 172.21.15.59:0/434907909 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1474003fb0 0x7f1474004800 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f1478009ac0 tx=0x7f1474005830 comp rx=0 tx=0).stop 2024-03-06T07:29:58.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.172+0000 7f148e652640 1 -- 172.21.15.59:0/434907909 reap_dead start 2024-03-06T07:29:58.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.172+0000 7f14900b5640 1 -- 172.21.15.59:0/434907909 shutdown_connections 2024-03-06T07:29:58.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.172+0000 7f14900b5640 1 -- 172.21.15.59:0/434907909 >> 172.21.15.59:0/434907909 conn(0x7f148809a9f0 msgr2=0x7f148809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.172+0000 7f14900b5640 1 -- 172.21.15.59:0/434907909 shutdown_connections 2024-03-06T07:29:58.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.172+0000 7f14900b5640 1 -- 172.21.15.59:0/434907909 wait complete. 2024-03-06T07:29:58.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.173+0000 7f14900b5640 1 Processor -- start 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.173+0000 7f14900b5640 1 -- start start 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f14900b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148814bd60 0x7f148815c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f14900b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f148814c140 0x7f148815df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f14900b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815e4c0 0x7f148806f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f14900b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880ce270 con 0x7f148814bd60 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f14900b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148815c1b0 con 0x7f148814c140 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f14900b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14880ce3f0 con 0x7f148815e4c0 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f148de51640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148814bd60 0x7f148815c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f148de51640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148814bd60 0x7f148815c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51304/0 (socket says 172.21.15.59:51304) 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f148de51640 1 -- 172.21.15.59:0/416402562 learned_addr learned my addr 172.21.15.59:0/416402562 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f148de51640 1 -- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815e4c0 msgr2=0x7f148806f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f148de51640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815e4c0 0x7f148806f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f148de51640 1 -- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f148814c140 msgr2=0x7f148815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f148d650640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f148814c140 0x7f148815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f148de51640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f148814c140 0x7f148815df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.174+0000 7f148de51640 1 -- 172.21.15.59:0/416402562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14881f77c0 con 0x7f148814bd60 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.175+0000 7f148de51640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148814bd60 0x7f148815c3e0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f1484007290 tx=0x7f1484007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.175+0000 7f147e7fc640 1 -- 172.21.15.59:0/416402562 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148400f860 con 0x7f148814bd60 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.175+0000 7f147e7fc640 1 -- 172.21.15.59:0/416402562 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1484018070 con 0x7f148814bd60 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.175+0000 7f14900b5640 1 -- 172.21.15.59:0/416402562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14881f7aa0 con 0x7f148814bd60 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.175+0000 7f147e7fc640 1 -- 172.21.15.59:0/416402562 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148400fc40 con 0x7f148814bd60 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.175+0000 7f14900b5640 1 -- 172.21.15.59:0/416402562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14881f8060 con 0x7f148814bd60 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.176+0000 7f14900b5640 1 -- 172.21.15.59:0/416402562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148804f330 con 0x7f148814bd60 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.176+0000 7f147e7fc640 1 -- 172.21.15.59:0/416402562 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f148401d460 con 0x7f148814bd60 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.176+0000 7f147e7fc640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f146c06b500 0x7f146c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.177+0000 7f147e7fc640 1 -- 172.21.15.59:0/416402562 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f148408a300 con 0x7f148814bd60 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.177+0000 7f148d650640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f146c06b500 0x7f146c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.177+0000 7f148d650640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f146c06b500 0x7f146c06d9c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f1478036000 tx=0x7f1478030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.181+0000 7f147e7fc640 1 -- 172.21.15.59:0/416402562 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1484059140 con 0x7f148814bd60 2024-03-06T07:29:58.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.400+0000 7f14900b5640 1 -- 172.21.15.59:0/416402562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14880b7a40 con 0x7f148814bd60 2024-03-06T07:29:58.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.400+0000 7f147e7fc640 1 -- 172.21.15.59:0/416402562 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+3399 (secure 0 0 0) 0x7f1484058a40 con 0x7f148814bd60 2024-03-06T07:29:58.402 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:58.402 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/2939413041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2939413041},{"type":"v1","addr":"172.21.15.59:6833","nonce":2939413041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:52.591736+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8676},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:29:58.403 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 90 2024-03-06T07:29:58.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 -- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f146c06b500 msgr2=0x7f146c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f146c06b500 0x7f146c06d9c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f1478036000 tx=0x7f1478030000 comp rx=0 tx=0).stop 2024-03-06T07:29:58.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 -- 172.21.15.59:0/416402562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148814bd60 msgr2=0x7f148815c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148814bd60 0x7f148815c3e0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f1484007290 tx=0x7f1484007750 comp rx=0 tx=0).stop 2024-03-06T07:29:58.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 -- 172.21.15.59:0/416402562 shutdown_connections 2024-03-06T07:29:58.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f146c06b500 0x7f146c06d9c0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f148815e4c0 0x7f148806f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f148814c140 0x7f148815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 --2- 172.21.15.59:0/416402562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f148814bd60 0x7f148815c3e0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 -- 172.21.15.59:0/416402562 >> 172.21.15.59:0/416402562 conn(0x7f148809a9f0 msgr2=0x7f14880d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 -- 172.21.15.59:0/416402562 shutdown_connections 2024-03-06T07:29:58.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.402+0000 7f14900b5640 1 -- 172.21.15.59:0/416402562 wait complete. 2024-03-06T07:29:58.415 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:58.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.514+0000 7f4aa7fbc640 1 Processor -- start 2024-03-06T07:29:58.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.514+0000 7f4aa7fbc640 1 -- start start 2024-03-06T07:29:58.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa7fbc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4aa0158600 0x7f4aa0072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa7fbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa015e3a0 0x7f4aa006d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa7fbc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa015eb40 con 0x7f4aa006deb0 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa7fbc640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa0161ba0 con 0x7f4aa0158600 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa7fbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa015ee20 con 0x7f4aa015e3a0 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa5d58640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4aa0158600 0x7f4aa0072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa5d58640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4aa0158600 0x7f4aa0072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51314/0 (socket says 172.21.15.59:51314) 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa5d58640 1 -- 172.21.15.59:0/2006596816 learned_addr learned my addr 172.21.15.59:0/2006596816 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa5557640 1 --2- 172.21.15.59:0/2006596816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa015e3a0 0x7f4aa006d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa5d58640 1 -- 172.21.15.59:0/2006596816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa015e3a0 msgr2=0x7f4aa006d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa5d58640 1 --2- 172.21.15.59:0/2006596816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa015e3a0 0x7f4aa006d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa5d58640 1 -- 172.21.15.59:0/2006596816 >> v1:172.21.15.164:6789/0 conn(0x7f4aa006deb0 legacy=0x7f4aa0162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.515+0000 7f4aa5d58640 1 -- 172.21.15.59:0/2006596816 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa015f100 con 0x7f4aa0158600 2024-03-06T07:29:58.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.516+0000 7f4aa5d58640 1 --2- 172.21.15.59:0/2006596816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa0158600 0x7f4aa0072730 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009870 tx=0x7f4a9c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e856146d391f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.516+0000 7f4aa4d56640 1 -- 172.21.15.59:0/2006596816 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c0317c0 con 0x7f4aa0158600 2024-03-06T07:29:58.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.516+0000 7f4aa4d56640 1 -- 172.21.15.59:0/2006596816 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9c037070 con 0x7f4aa0158600 2024-03-06T07:29:58.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.516+0000 7f4aa4d56640 1 -- 172.21.15.59:0/2006596816 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c031b80 con 0x7f4aa0158600 2024-03-06T07:29:58.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.517+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/2006596816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa0158600 msgr2=0x7f4aa0072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.517+0000 7f4aa7fbc640 1 --2- 172.21.15.59:0/2006596816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa0158600 0x7f4aa0072730 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009870 tx=0x7f4a9c02b860 comp rx=0 tx=0).stop 2024-03-06T07:29:58.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.517+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/2006596816 shutdown_connections 2024-03-06T07:29:58.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.517+0000 7f4aa7fbc640 1 --2- 172.21.15.59:0/2006596816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa015e3a0 0x7f4aa006d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.520 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.517+0000 7f4aa7fbc640 1 --2- 172.21.15.59:0/2006596816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa0158600 0x7f4aa0072730 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.517+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/2006596816 >> 172.21.15.59:0/2006596816 conn(0x7f4aa006c0c0 msgr2=0x7f4aa006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.517+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/2006596816 shutdown_connections 2024-03-06T07:29:58.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.517+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/2006596816 wait complete. 2024-03-06T07:29:58.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.518+0000 7f4aa7fbc640 1 Processor -- start 2024-03-06T07:29:58.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.518+0000 7f4aa7fbc640 1 -- start start 2024-03-06T07:29:58.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa7fbc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa006deb0 0x7f4aa0141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa7fbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa0158600 0x7f4aa0152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa7fbc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa015e3a0 0x7f4aa0160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa7fbc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa01628c0 con 0x7f4aa006deb0 2024-03-06T07:29:58.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa7fbc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa0162740 con 0x7f4aa015e3a0 2024-03-06T07:29:58.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa7fbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa006afb0 con 0x7f4aa0158600 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa5d58640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa006deb0 0x7f4aa0141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa5d58640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa006deb0 0x7f4aa0141420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51320/0 (socket says 172.21.15.59:51320) 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa5d58640 1 -- 172.21.15.59:0/815980542 learned_addr learned my addr 172.21.15.59:0/815980542 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa5557640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa0158600 0x7f4aa0152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa6559640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa015e3a0 0x7f4aa0160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa5d58640 1 -- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa0158600 msgr2=0x7f4aa0152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa5d58640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa0158600 0x7f4aa0152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa5d58640 1 -- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa015e3a0 msgr2=0x7f4aa0160350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa5d58640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa015e3a0 0x7f4aa0160350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa5d58640 1 -- 172.21.15.59:0/815980542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa0142bd0 con 0x7f4aa006deb0 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa5557640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa0158600 0x7f4aa0152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.519+0000 7f4aa6559640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa015e3a0 0x7f4aa0160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:58.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.520+0000 7f4aa5d58640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa006deb0 0x7f4aa0141420 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c000c00 tx=0x7f4a9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.520+0000 7f4a967fc640 1 -- 172.21.15.59:0/815980542 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c032860 con 0x7f4aa006deb0 2024-03-06T07:29:58.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.520+0000 7f4a967fc640 1 -- 172.21.15.59:0/815980542 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9c032a20 con 0x7f4aa006deb0 2024-03-06T07:29:58.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.520+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/815980542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa0142eb0 con 0x7f4aa006deb0 2024-03-06T07:29:58.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.520+0000 7f4a967fc640 1 -- 172.21.15.59:0/815980542 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c032e10 con 0x7f4aa006deb0 2024-03-06T07:29:58.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.520+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/815980542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa0143470 con 0x7f4aa006deb0 2024-03-06T07:29:58.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.521+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/815980542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa004f330 con 0x7f4aa006deb0 2024-03-06T07:29:58.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.521+0000 7f4a967fc640 1 -- 172.21.15.59:0/815980542 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a9c04c060 con 0x7f4aa006deb0 2024-03-06T07:29:58.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.522+0000 7f4a967fc640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8406b4d0 0x7f4a8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.522+0000 7f4a967fc640 1 -- 172.21.15.59:0/815980542 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4a9c0ab9f0 con 0x7f4aa006deb0 2024-03-06T07:29:58.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.522+0000 7f4aa5557640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8406b4d0 0x7f4a8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.522+0000 7f4aa5557640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8406b4d0 0x7f4a8406d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4a90006fd0 tx=0x7f4a90008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.526+0000 7f4a967fc640 1 -- 172.21.15.59:0/815980542 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a9c040070 con 0x7f4aa006deb0 2024-03-06T07:29:58.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.707+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/815980542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4aa00d06f0 con 0x7f4aa006deb0 2024-03-06T07:29:58.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.708+0000 7f4a967fc640 1 -- 172.21.15.59:0/815980542 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16708 (secure 0 0 0) 0x7f4a9c031e90 con 0x7f4aa006deb0 2024-03-06T07:29:58.710 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:29:58.710 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":130,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:29:39.117588+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:40.558382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:41.255127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.3","mappings":[{"from":2,"to":0}]},{"pgid":"19.25","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:58.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8406b4d0 msgr2=0x7f4a8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8406b4d0 0x7f4a8406d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4a90006fd0 tx=0x7f4a90008040 comp rx=0 tx=0).stop 2024-03-06T07:29:58.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/815980542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa006deb0 msgr2=0x7f4aa0141420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa006deb0 0x7f4aa0141420 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c000c00 tx=0x7f4a9c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/815980542 shutdown_connections 2024-03-06T07:29:58.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4a8406b4d0 0x7f4a8406d990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa015e3a0 0x7f4aa0160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa0158600 0x7f4aa0152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 --2- 172.21.15.59:0/815980542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4aa006deb0 0x7f4aa0141420 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/815980542 >> 172.21.15.59:0/815980542 conn(0x7f4aa006c0c0 msgr2=0x7f4aa006e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/815980542 shutdown_connections 2024-03-06T07:29:58.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.710+0000 7f4aa7fbc640 1 -- 172.21.15.59:0/815980542 wait complete. 2024-03-06T07:29:58.726 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:40.558382+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:41.255127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:58.727 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:58.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.829+0000 7f5501247640 1 Processor -- start 2024-03-06T07:29:58.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.829+0000 7f5501247640 1 -- start start 2024-03-06T07:29:58.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.830+0000 7f5501247640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54fc0d10c0 0x7f54fc0d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.830+0000 7f5501247640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f54fc0d14a0 0x7f54fc15e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.830+0000 7f5501247640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f54fc15eb00 0x7f54fc0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.830+0000 7f5501247640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc15c050 con 0x7f54fc15eb00 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.830+0000 7f5501247640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc0d0060 con 0x7f54fc0d14a0 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.830+0000 7f5501247640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc0cfa50 con 0x7f54fc0d10c0 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.830+0000 7f54fb59e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f54fc15eb00 0x7f54fc0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.830+0000 7f54fb59e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f54fc15eb00 0x7f54fc0cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51330/0 (socket says 172.21.15.59:51330) 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.830+0000 7f54fb59e640 1 -- 172.21.15.59:0/1058014138 learned_addr learned my addr 172.21.15.59:0/1058014138 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.830+0000 7f54f3fff640 1 --2- 172.21.15.59:0/1058014138 >> v2:172.21.15.164:3300/0 conn(0x7f54fc0d14a0 0x7f54fc15e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.830+0000 7f54fad9d640 1 --2- 172.21.15.59:0/1058014138 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54fc0d10c0 0x7f54fc0d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.831+0000 7f54fb59e640 1 -- 172.21.15.59:0/1058014138 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54fc0d10c0 msgr2=0x7f54fc0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.831+0000 7f54fb59e640 1 --2- 172.21.15.59:0/1058014138 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54fc0d10c0 0x7f54fc0d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.831+0000 7f54fb59e640 1 -- 172.21.15.59:0/1058014138 >> v2:172.21.15.164:3300/0 conn(0x7f54fc0d14a0 msgr2=0x7f54fc15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.831+0000 7f54fb59e640 1 --2- 172.21.15.59:0/1058014138 >> v2:172.21.15.164:3300/0 conn(0x7f54fc0d14a0 0x7f54fc15e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.831+0000 7f54fb59e640 1 -- 172.21.15.59:0/1058014138 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc0d0a30 con 0x7f54fc15eb00 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.831+0000 7f54f3fff640 1 --2- 172.21.15.59:0/1058014138 >> v2:172.21.15.164:3300/0 conn(0x7f54fc0d14a0 0x7f54fc15e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.831+0000 7f54fad9d640 1 --2- 172.21.15.59:0/1058014138 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54fc0d10c0 0x7f54fc0d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:58.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.831+0000 7f54fb59e640 1 --2- 172.21.15.59:0/1058014138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54fc15eb00 0x7f54fc0cae80 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009e50 tx=0x7f54f402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228c32183a2969ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f54fa59c640 1 -- 172.21.15.59:0/1058014138 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f402ca70 con 0x7f54fc15eb00 2024-03-06T07:29:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f54fa59c640 1 -- 172.21.15.59:0/1058014138 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f402cc10 con 0x7f54fc15eb00 2024-03-06T07:29:58.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f54fa59c640 1 -- 172.21.15.59:0/1058014138 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f40343f0 con 0x7f54fc15eb00 2024-03-06T07:29:58.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f5501247640 1 -- 172.21.15.59:0/1058014138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54fc15eb00 msgr2=0x7f54fc0cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f5501247640 1 --2- 172.21.15.59:0/1058014138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54fc15eb00 0x7f54fc0cae80 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009e50 tx=0x7f54f402bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f5501247640 1 -- 172.21.15.59:0/1058014138 shutdown_connections 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f5501247640 1 --2- 172.21.15.59:0/1058014138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54fc15eb00 0x7f54fc0cae80 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f5501247640 1 --2- 172.21.15.59:0/1058014138 >> v2:172.21.15.164:3300/0 conn(0x7f54fc0d14a0 0x7f54fc15e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f5501247640 1 --2- 172.21.15.59:0/1058014138 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54fc0d10c0 0x7f54fc0d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f5501247640 1 -- 172.21.15.59:0/1058014138 >> 172.21.15.59:0/1058014138 conn(0x7f54fc0b3800 msgr2=0x7f54fc0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f5501247640 1 -- 172.21.15.59:0/1058014138 shutdown_connections 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.832+0000 7f5501247640 1 -- 172.21.15.59:0/1058014138 wait complete. 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.833+0000 7f5501247640 1 Processor -- start 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.834+0000 7f5501247640 1 -- start start 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.834+0000 7f5501247640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54fc0d10c0 0x7f54fc0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.834+0000 7f5501247640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54fc0d14a0 0x7f54fc147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.834+0000 7f5501247640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54fc15eb00 0x7f54fc0b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.834+0000 7f5501247640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc15f440 con 0x7f54fc15eb00 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.834+0000 7f5501247640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc15f160 con 0x7f54fc0d10c0 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.834+0000 7f5501247640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc0cfcf0 con 0x7f54fc0d14a0 2024-03-06T07:29:58.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.834+0000 7f54fb59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54fc15eb00 0x7f54fc0b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.834+0000 7f54fb59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54fc15eb00 0x7f54fc0b7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51342/0 (socket says 172.21.15.59:51342) 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.834+0000 7f54fb59e640 1 -- 172.21.15.59:0/1878586552 learned_addr learned my addr 172.21.15.59:0/1878586552 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f54f3fff640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54fc0d14a0 0x7f54fc147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f54fb59e640 1 -- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54fc0d14a0 msgr2=0x7f54fc147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f54fb59e640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54fc0d14a0 0x7f54fc147790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f54fad9d640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54fc0d10c0 0x7f54fc0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f54fb59e640 1 -- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54fc0d10c0 msgr2=0x7f54fc0b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f54fb59e640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54fc0d10c0 0x7f54fc0b4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f54fb59e640 1 -- 172.21.15.59:0/1878586552 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc147cd0 con 0x7f54fc15eb00 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f54fb59e640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54fc15eb00 0x7f54fc0b7190 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009f70 tx=0x7f54f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f54f37fe640 1 -- 172.21.15.59:0/1878586552 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4044020 con 0x7f54fc15eb00 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f54f37fe640 1 -- 172.21.15.59:0/1878586552 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f4040690 con 0x7f54fc15eb00 2024-03-06T07:29:58.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f5501247640 1 -- 172.21.15.59:0/1878586552 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54fc0b76d0 con 0x7f54fc15eb00 2024-03-06T07:29:58.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f54f37fe640 1 -- 172.21.15.59:0/1878586552 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4009510 con 0x7f54fc15eb00 2024-03-06T07:29:58.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.835+0000 7f5501247640 1 -- 172.21.15.59:0/1878586552 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54fc0aeae0 con 0x7f54fc15eb00 2024-03-06T07:29:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.836+0000 7f5501247640 1 -- 172.21.15.59:0/1878586552 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54fc09fe60 con 0x7f54fc15eb00 2024-03-06T07:29:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.837+0000 7f54f37fe640 1 -- 172.21.15.59:0/1878586552 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54f4047020 con 0x7f54fc15eb00 2024-03-06T07:29:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.837+0000 7f54f37fe640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f54dc06b4d0 0x7f54dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.837+0000 7f54f37fe640 1 -- 172.21.15.59:0/1878586552 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f54f40acc20 con 0x7f54fc15eb00 2024-03-06T07:29:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.837+0000 7f54fad9d640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f54dc06b4d0 0x7f54dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.838+0000 7f54fad9d640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f54dc06b4d0 0x7f54dc06d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f54ec006fd0 tx=0x7f54ec008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:58.846+0000 7f54f37fe640 1 -- 172.21.15.59:0/1878586552 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54f404e030 con 0x7f54fc15eb00 2024-03-06T07:29:59.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.165+0000 7f5501247640 1 -- 172.21.15.59:0/1878586552 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54fc0d0c30 con 0x7f54fc15eb00 2024-03-06T07:29:59.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.166+0000 7f54f37fe640 1 -- 172.21.15.59:0/1878586552 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) v1 ==== 93+0+10 (secure 0 0 0) 0x7f54f407baf0 con 0x7f54fc15eb00 2024-03-06T07:29:59.168 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:29:59.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 -- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f54dc06b4d0 msgr2=0x7f54dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f54dc06b4d0 0x7f54dc06d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f54ec006fd0 tx=0x7f54ec008040 comp rx=0 tx=0).stop 2024-03-06T07:29:59.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 -- 172.21.15.59:0/1878586552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54fc15eb00 msgr2=0x7f54fc0b7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54fc15eb00 0x7f54fc0b7190 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009f70 tx=0x7f54f4004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:59.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 -- 172.21.15.59:0/1878586552 shutdown_connections 2024-03-06T07:29:59.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f54dc06b4d0 0x7f54dc06d990 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54fc15eb00 0x7f54fc0b7190 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54fc0d14a0 0x7f54fc147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 --2- 172.21.15.59:0/1878586552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54fc0d10c0 0x7f54fc0b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 -- 172.21.15.59:0/1878586552 >> 172.21.15.59:0/1878586552 conn(0x7f54fc0b3800 msgr2=0x7f54fc15c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 -- 172.21.15.59:0/1878586552 shutdown_connections 2024-03-06T07:29:59.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.168+0000 7f5501247640 1 -- 172.21.15.59:0/1878586552 wait complete. 2024-03-06T07:29:59.181 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:59.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.284+0000 7f104e481640 1 Processor -- start 2024-03-06T07:29:59.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.284+0000 7f104e481640 1 -- start start 2024-03-06T07:29:59.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f104e481640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f10481586e0 0x7f10480aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f104e481640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f10480aef80 0x7f10480cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f104e481640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1048158ac0 con 0x7f10480cffd0 2024-03-06T07:29:59.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f104e481640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f104815b2a0 con 0x7f10481586e0 2024-03-06T07:29:59.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f104e481640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f104815b540 con 0x7f10480aef80 2024-03-06T07:29:59.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f1047fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f10481586e0 0x7f10480aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f104ca1e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f10480cffd0 0x7f104806da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60314/0 (socket says 172.21.15.59:60314) 2024-03-06T07:29:59.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f104ca1e640 1 -- 172.21.15.59:0/2205931266 learned_addr learned my addr 172.21.15.59:0/2205931266 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:59.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f1047fff640 1 -- 172.21.15.59:0/2205931266 >> v2:172.21.15.164:3300/0 conn(0x7f10480aef80 msgr2=0x7f10480cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f10477fe640 1 --2- 172.21.15.59:0/2205931266 >> v2:172.21.15.164:3300/0 conn(0x7f10480aef80 0x7f10480cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.288 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f1047fff640 1 --2- 172.21.15.59:0/2205931266 >> v2:172.21.15.164:3300/0 conn(0x7f10480aef80 0x7f10480cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f1047fff640 1 -- 172.21.15.59:0/2205931266 >> v1:172.21.15.59:6789/0 conn(0x7f10480cffd0 legacy=0x7f104806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f1046ffd640 1 -- 172.21.15.59:0/2205931266 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505232438 0 0) 0x7f1048158ac0 con 0x7f10480cffd0 2024-03-06T07:29:59.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f1047fff640 1 -- 172.21.15.59:0/2205931266 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104815b820 con 0x7f10481586e0 2024-03-06T07:29:59.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.285+0000 7f10477fe640 1 --2- 172.21.15.59:0/2205931266 >> v2:172.21.15.164:3300/0 conn(0x7f10480aef80 0x7f10480cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:59.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.286+0000 7f1047fff640 1 --2- 172.21.15.59:0/2205931266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10481586e0 0x7f10480aea40 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f1048072e30 tx=0x7f103c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979fbc9011ac98b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.286+0000 7f1046ffd640 1 -- 172.21.15.59:0/2205931266 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c035020 con 0x7f10481586e0 2024-03-06T07:29:59.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.286+0000 7f1046ffd640 1 -- 172.21.15.59:0/2205931266 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103c031830 con 0x7f10481586e0 2024-03-06T07:29:59.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.286+0000 7f1046ffd640 1 -- 172.21.15.59:0/2205931266 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c031c00 con 0x7f10481586e0 2024-03-06T07:29:59.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.287+0000 7f104e481640 1 -- 172.21.15.59:0/2205931266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10481586e0 msgr2=0x7f10480aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.287+0000 7f104e481640 1 --2- 172.21.15.59:0/2205931266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10481586e0 0x7f10480aea40 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f1048072e30 tx=0x7f103c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.287+0000 7f104e481640 1 -- 172.21.15.59:0/2205931266 shutdown_connections 2024-03-06T07:29:59.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.287+0000 7f104e481640 1 --2- 172.21.15.59:0/2205931266 >> v2:172.21.15.164:3300/0 conn(0x7f10480aef80 0x7f10480cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.287+0000 7f104e481640 1 --2- 172.21.15.59:0/2205931266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10481586e0 0x7f10480aea40 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.287+0000 7f104e481640 1 -- 172.21.15.59:0/2205931266 >> 172.21.15.59:0/2205931266 conn(0x7f104806b190 msgr2=0x7f104806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.287+0000 7f104e481640 1 -- 172.21.15.59:0/2205931266 shutdown_connections 2024-03-06T07:29:59.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.287+0000 7f104e481640 1 -- 172.21.15.59:0/2205931266 wait complete. 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.288+0000 7f104e481640 1 Processor -- start 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.288+0000 7f104e481640 1 -- start start 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f104e481640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10480aef80 0x7f10480ce670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f104e481640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10480cffd0 0x7f1048158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f104e481640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10481586e0 0x7f104806c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f104e481640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1048071350 con 0x7f10480aef80 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f104e481640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104815bde0 con 0x7f10480cffd0 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f104e481640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10480714d0 con 0x7f10481586e0 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1047fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10480aef80 0x7f10480ce670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1047fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10480aef80 0x7f10480ce670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51372/0 (socket says 172.21.15.59:51372) 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1047fff640 1 -- 172.21.15.59:0/3547530703 learned_addr learned my addr 172.21.15.59:0/3547530703 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f104ca1e640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10481586e0 0x7f104806c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1047fff640 1 -- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10481586e0 msgr2=0x7f104806c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f10477fe640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10480cffd0 0x7f1048158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1047fff640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10481586e0 0x7f104806c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1047fff640 1 -- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10480cffd0 msgr2=0x7f1048158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1047fff640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10480cffd0 0x7f1048158f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1047fff640 1 -- 172.21.15.59:0/3547530703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1048146700 con 0x7f10480aef80 2024-03-06T07:29:59.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1047fff640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10480aef80 0x7f10480ce670 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f103c034040 tx=0x7f103c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1044ff9640 1 -- 172.21.15.59:0/3547530703 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c041070 con 0x7f10480aef80 2024-03-06T07:29:59.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1044ff9640 1 -- 172.21.15.59:0/3547530703 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103c004230 con 0x7f10480aef80 2024-03-06T07:29:59.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f104e481640 1 -- 172.21.15.59:0/3547530703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1048147f30 con 0x7f10480aef80 2024-03-06T07:29:59.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f1044ff9640 1 -- 172.21.15.59:0/3547530703 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c035040 con 0x7f10480aef80 2024-03-06T07:29:59.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.289+0000 7f104e481640 1 -- 172.21.15.59:0/3547530703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10481484f0 con 0x7f10480aef80 2024-03-06T07:29:59.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.290+0000 7f104e481640 1 -- 172.21.15.59:0/3547530703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1048068210 con 0x7f10480aef80 2024-03-06T07:29:59.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.291+0000 7f1044ff9640 1 -- 172.21.15.59:0/3547530703 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f103c038020 con 0x7f10480aef80 2024-03-06T07:29:59.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.291+0000 7f1044ff9640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f103406b4d0 0x7f103406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.291+0000 7f1044ff9640 1 -- 172.21.15.59:0/3547530703 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f103c0abdd0 con 0x7f10480aef80 2024-03-06T07:29:59.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.292+0000 7f10477fe640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f103406b4d0 0x7f103406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.292+0000 7f10477fe640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f103406b4d0 0x7f103406d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f10480ce080 tx=0x7f1038008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.296+0000 7f1044ff9640 1 -- 172.21.15.59:0/3547530703 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1048068210 con 0x7f10480aef80 2024-03-06T07:29:59.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.475+0000 7f104e481640 1 -- 172.21.15.59:0/3547530703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f10480732b0 con 0x7f10480aef80 2024-03-06T07:29:59.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.476+0000 7f1044ff9640 1 -- 172.21.15.59:0/3547530703 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) v1 ==== 104+0+11 (secure 0 0 0) 0x7f103c07ab80 con 0x7f10480aef80 2024-03-06T07:29:59.478 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 -- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f103406b4d0 msgr2=0x7f103406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f103406b4d0 0x7f103406d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f10480ce080 tx=0x7f1038008040 comp rx=0 tx=0).stop 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 -- 172.21.15.59:0/3547530703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10480aef80 msgr2=0x7f10480ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10480aef80 0x7f10480ce670 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f103c034040 tx=0x7f103c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 -- 172.21.15.59:0/3547530703 shutdown_connections 2024-03-06T07:29:59.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f103406b4d0 0x7f103406d990 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10481586e0 0x7f104806c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10480cffd0 0x7f1048158f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 --2- 172.21.15.59:0/3547530703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10480aef80 0x7f10480ce670 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 -- 172.21.15.59:0/3547530703 >> 172.21.15.59:0/3547530703 conn(0x7f104806b190 msgr2=0x7f104815a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 -- 172.21.15.59:0/3547530703 shutdown_connections 2024-03-06T07:29:59.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.478+0000 7f104e481640 1 -- 172.21.15.59:0/3547530703 wait complete. 2024-03-06T07:29:59.491 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:59.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.595+0000 7f84feb3e640 1 Processor -- start 2024-03-06T07:29:59.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.595+0000 7f84feb3e640 1 -- start start 2024-03-06T07:29:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84feb3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f80d4860 0x7f84f80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84feb3e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f84f80b0e30 0x7f84f80b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84feb3e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f81618b0 con 0x7f84f81614d0 2024-03-06T07:29:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84feb3e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f80d4cc0 con 0x7f84f80b0e30 2024-03-06T07:29:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84feb3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f80cf400 con 0x7f84f80d4860 2024-03-06T07:29:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84fc8da640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f84f81614d0 0x7f84f80d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60328/0 (socket says 172.21.15.59:60328) 2024-03-06T07:29:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84fc8da640 1 -- 172.21.15.59:0/3300258503 learned_addr learned my addr 172.21.15.59:0/3300258503 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84fd0db640 1 --2- 172.21.15.59:0/3300258503 >> v2:172.21.15.164:3300/0 conn(0x7f84f80b0e30 0x7f84f80b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f7fff640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f80d4860 0x7f84f80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687150712 0 0) 0x7f84f81618b0 con 0x7f84f81614d0 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e8003520 con 0x7f84f81614d0 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 841574521 0 0) 0x7f84e8003520 con 0x7f84f81614d0 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f80d4860 msgr2=0x7f84f80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f80d4860 0x7f84f80d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 >> v2:172.21.15.164:3300/0 conn(0x7f84f80b0e30 msgr2=0x7f84f80b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 --2- 172.21.15.59:0/3300258503 >> v2:172.21.15.164:3300/0 conn(0x7f84f80b0e30 0x7f84f80b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f80cf6a0 con 0x7f84f81614d0 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f84ec001670 con 0x7f84f81614d0 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 >> v1:172.21.15.59:6789/0 conn(0x7f84f81614d0 legacy=0x7f84f80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84e80039b0 0x7f84e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f80d4860 0x7f84e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84e8004d40 0x7f84e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f80d4cc0 con 0x7f84e8004d40 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f80cf400 con 0x7f84e80039b0 2024-03-06T07:29:59.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.596+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e800b9c0 con 0x7f84f80d4860 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.597+0000 7f84fd0db640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84e8004d40 0x7f84e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.597+0000 7f84f7fff640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f80d4860 0x7f84e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.597+0000 7f84fc8da640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84e80039b0 0x7f84e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.597+0000 7f84fd0db640 1 -- 172.21.15.59:0/3300258503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f80d4860 msgr2=0x7f84e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.597+0000 7f84fd0db640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f80d4860 0x7f84e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.597+0000 7f84fd0db640 1 -- 172.21.15.59:0/3300258503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84e80039b0 msgr2=0x7f84e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.597+0000 7f84fd0db640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84e80039b0 0x7f84e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.597+0000 7f84fd0db640 1 -- 172.21.15.59:0/3300258503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e8003520 con 0x7f84e8004d40 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.598+0000 7f84fd0db640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84e8004d40 0x7f84e80051c0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f84f000e170 tx=0x7f84e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80ee9fb560e1f30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.598+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0039070 con 0x7f84e8004d40 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.598+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f002cc20 con 0x7f84e8004d40 2024-03-06T07:29:59.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.598+0000 7f84f77fe640 1 -- 172.21.15.59:0/3300258503 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0009c10 con 0x7f84e8004d40 2024-03-06T07:29:59.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.598+0000 7f84feb3e640 1 -- 172.21.15.59:0/3300258503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84e8004d40 msgr2=0x7f84e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.598+0000 7f84feb3e640 1 --2- 172.21.15.59:0/3300258503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84e8004d40 0x7f84e80051c0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f84f000e170 tx=0x7f84e8005830 comp rx=0 tx=0).stop 2024-03-06T07:29:59.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.598+0000 7f84fd0db640 1 -- 172.21.15.59:0/3300258503 reap_dead start 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.598+0000 7f84feb3e640 1 -- 172.21.15.59:0/3300258503 shutdown_connections 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.598+0000 7f84feb3e640 1 -- 172.21.15.59:0/3300258503 >> 172.21.15.59:0/3300258503 conn(0x7f84f80d3d40 msgr2=0x7f84f80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.599+0000 7f84feb3e640 1 -- 172.21.15.59:0/3300258503 shutdown_connections 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.599+0000 7f84feb3e640 1 -- 172.21.15.59:0/3300258503 wait complete. 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.600+0000 7f84feb3e640 1 Processor -- start 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.600+0000 7f84feb3e640 1 -- start start 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84feb3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84f8154d10 0x7f84f815e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84feb3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f8155630 0x7f84f80c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84feb3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84f81550f0 0x7f84f80ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84feb3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f80b0910 con 0x7f84f8154d10 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84feb3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f80cfc60 con 0x7f84f81550f0 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84feb3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f80d2020 con 0x7f84f8155630 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84fc8da640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84f8154d10 0x7f84f815e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84fc8da640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84f8154d10 0x7f84f815e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51402/0 (socket says 172.21.15.59:51402) 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84fc8da640 1 -- 172.21.15.59:0/2618635261 learned_addr learned my addr 172.21.15.59:0/2618635261 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84f7fff640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f8155630 0x7f84f80c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84fd0db640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84f81550f0 0x7f84f80ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84fc8da640 1 -- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f8155630 msgr2=0x7f84f80c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84fc8da640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f8155630 0x7f84f80c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84fc8da640 1 -- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84f81550f0 msgr2=0x7f84f80ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84fc8da640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84f81550f0 0x7f84f80ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84fc8da640 1 -- 172.21.15.59:0/2618635261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f815b5a0 con 0x7f84f8154d10 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84f7fff640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f8155630 0x7f84f80c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84fc8da640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84f8154d10 0x7f84f815e100 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f84ec0072c0 tx=0x7f84ec007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84f57fa640 1 -- 172.21.15.59:0/2618635261 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec00fa80 con 0x7f84f8154d10 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84f57fa640 1 -- 172.21.15.59:0/2618635261 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ec018070 con 0x7f84f8154d10 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84feb3e640 1 -- 172.21.15.59:0/2618635261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f815ae20 con 0x7f84f8154d10 2024-03-06T07:29:59.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84f57fa640 1 -- 172.21.15.59:0/2618635261 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec00fe80 con 0x7f84f8154d10 2024-03-06T07:29:59.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.601+0000 7f84feb3e640 1 -- 172.21.15.59:0/2618635261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f815bb20 con 0x7f84f8154d10 2024-03-06T07:29:59.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.602+0000 7f84feb3e640 1 -- 172.21.15.59:0/2618635261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f8068ff0 con 0x7f84f8154d10 2024-03-06T07:29:59.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.603+0000 7f84f57fa640 1 -- 172.21.15.59:0/2618635261 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84ec01d460 con 0x7f84f8154d10 2024-03-06T07:29:59.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.603+0000 7f84f57fa640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84dc06b4d0 0x7f84dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.603+0000 7f84f57fa640 1 -- 172.21.15.59:0/2618635261 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f84ec08a490 con 0x7f84f8154d10 2024-03-06T07:29:59.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.603+0000 7f84f7fff640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84dc06b4d0 0x7f84dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.604+0000 7f84f7fff640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84dc06b4d0 0x7f84dc06d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f84f80a1b30 tx=0x7f84e4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.607+0000 7f84f57fa640 1 -- 172.21.15.59:0/2618635261 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84ec059360 con 0x7f84f8154d10 2024-03-06T07:29:59.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.814+0000 7f84feb3e640 1 -- 172.21.15.59:0/2618635261 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84f815bcf0 con 0x7f84f8154d10 2024-03-06T07:29:59.816 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.814+0000 7f84f57fa640 1 -- 172.21.15.59:0/2618635261 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84ec058c60 con 0x7f84f8154d10 2024-03-06T07:29:59.816 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:29:59.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 -- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84dc06b4d0 msgr2=0x7f84dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84dc06b4d0 0x7f84dc06d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f84f80a1b30 tx=0x7f84e4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:59.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 -- 172.21.15.59:0/2618635261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84f8154d10 msgr2=0x7f84f815e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84f8154d10 0x7f84f815e100 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f84ec0072c0 tx=0x7f84ec007780 comp rx=0 tx=0).stop 2024-03-06T07:29:59.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 -- 172.21.15.59:0/2618635261 shutdown_connections 2024-03-06T07:29:59.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84dc06b4d0 0x7f84dc06d990 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84f81550f0 0x7f84f80ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84f8155630 0x7f84f80c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 --2- 172.21.15.59:0/2618635261 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84f8154d10 0x7f84f815e100 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 -- 172.21.15.59:0/2618635261 >> 172.21.15.59:0/2618635261 conn(0x7f84f80d3d40 msgr2=0x7f84f806b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 -- 172.21.15.59:0/2618635261 shutdown_connections 2024-03-06T07:29:59.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.816+0000 7f84feb3e640 1 -- 172.21.15.59:0/2618635261 wait complete. 2024-03-06T07:29:59.830 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:59.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.929+0000 7fa75f35e640 1 Processor -- start 2024-03-06T07:29:59.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.929+0000 7fa75f35e640 1 -- start start 2024-03-06T07:29:59.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa75f35e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa75815dfe0 0x7fa7580cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa75f35e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa7580cf830 0x7fa7580d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa75f35e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7580b3430 con 0x7fa7580d27f0 2024-03-06T07:29:59.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa75f35e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7580d2c50 con 0x7fa7580cf830 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa75f35e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580b31e0 con 0x7fa75815dfe0 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa75d8fb640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa7580d27f0 0x7fa7580d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60336/0 (socket says 172.21.15.59:60336) 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa75d8fb640 1 -- 172.21.15.59:0/3386281913 learned_addr learned my addr 172.21.15.59:0/3386281913 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa75d0fa640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa75815dfe0 0x7fa7580cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa75c8f9640 1 --2- 172.21.15.59:0/3386281913 >> v2:172.21.15.164:3300/0 conn(0x7fa7580cf830 0x7fa7580d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585090687 0 0) 0x7fa7580b3430 con 0x7fa7580d27f0 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa744003520 con 0x7fa7580d27f0 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1351837879 0 0) 0x7fa744003520 con 0x7fa7580d27f0 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa75815dfe0 msgr2=0x7fa7580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa75815dfe0 0x7fa7580cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 >> v2:172.21.15.164:3300/0 conn(0x7fa7580cf830 msgr2=0x7fa7580d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 --2- 172.21.15.59:0/3386281913 >> v2:172.21.15.164:3300/0 conn(0x7fa7580cf830 0x7fa7580d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7580b3710 con 0x7fa7580d27f0 2024-03-06T07:29:59.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fa7500019f0 con 0x7fa7580d27f0 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa75d0fa640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa75815dfe0 0x7fa7580cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 >> v1:172.21.15.59:6789/0 conn(0x7fa7580d27f0 legacy=0x7fa7580d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7440039b0 0x7fa744003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa744004800 0x7fa744004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa75815dfe0 0x7fa7440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580b31e0 con 0x7fa7440039b0 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74400b9c0 con 0x7fa744004800 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.930+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74400bb40 con 0x7fa75815dfe0 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.931+0000 7fa75d0fa640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7440039b0 0x7fa744003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.931+0000 7fa75d8fb640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa75815dfe0 0x7fa7440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.931+0000 7fa75c8f9640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa744004800 0x7fa744004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.931+0000 7fa75d0fa640 1 -- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa75815dfe0 msgr2=0x7fa7440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.931+0000 7fa75d0fa640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa75815dfe0 0x7fa7440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.931+0000 7fa75d0fa640 1 -- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa744004800 msgr2=0x7fa744004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.931+0000 7fa75d0fa640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa744004800 0x7fa744004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.931+0000 7fa75d0fa640 1 -- 172.21.15.59:0/3386281913 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa744003520 con 0x7fa7440039b0 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.931+0000 7fa75d8fb640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa75815dfe0 0x7fa7440051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.931+0000 7fa75c8f9640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa744004800 0x7fa744004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.932+0000 7fa75d0fa640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7440039b0 0x7fa744003dd0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fa754009c90 tx=0x7fa744005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120f5ed515f8bcbf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.932+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754039070 con 0x7fa7440039b0 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.932+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75402c8f0 con 0x7fa7440039b0 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.932+0000 7fa74ffff640 1 -- 172.21.15.59:0/3386281913 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754009860 con 0x7fa7440039b0 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.932+0000 7fa75f35e640 1 -- 172.21.15.59:0/3386281913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7440039b0 msgr2=0x7fa744003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.932+0000 7fa75f35e640 1 --2- 172.21.15.59:0/3386281913 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7440039b0 0x7fa744003dd0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fa754009c90 tx=0x7fa744005830 comp rx=0 tx=0).stop 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.932+0000 7fa75d8fb640 1 -- 172.21.15.59:0/3386281913 reap_dead start 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.932+0000 7fa75f35e640 1 -- 172.21.15.59:0/3386281913 shutdown_connections 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.932+0000 7fa75f35e640 1 -- 172.21.15.59:0/3386281913 >> 172.21.15.59:0/3386281913 conn(0x7fa7580d3cd0 msgr2=0x7fa7580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.932+0000 7fa75f35e640 1 -- 172.21.15.59:0/3386281913 shutdown_connections 2024-03-06T07:29:59.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.932+0000 7fa75f35e640 1 -- 172.21.15.59:0/3386281913 wait complete. 2024-03-06T07:29:59.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.934+0000 7fa75f35e640 1 Processor -- start 2024-03-06T07:29:59.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.934+0000 7fa75f35e640 1 -- start start 2024-03-06T07:29:59.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75f35e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7440039b0 0x7fa7580cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75f35e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7580a1620 0x7fa7581f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75f35e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7580b68d0 0x7fa75806bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75f35e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580d1c70 con 0x7fa7580b68d0 2024-03-06T07:29:59.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75f35e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580b3d60 con 0x7fa7580a1620 2024-03-06T07:29:59.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75f35e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580d1df0 con 0x7fa7440039b0 2024-03-06T07:29:59.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75d8fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7580b68d0 0x7fa75806bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75d8fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7580b68d0 0x7fa75806bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51424/0 (socket says 172.21.15.59:51424) 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75d8fb640 1 -- 172.21.15.59:0/1918040463 learned_addr learned my addr 172.21.15.59:0/1918040463 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75d0fa640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7440039b0 0x7fa7580cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75c8f9640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7580a1620 0x7fa7581f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75d8fb640 1 -- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7440039b0 msgr2=0x7fa7580cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75d8fb640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7440039b0 0x7fa7580cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75d8fb640 1 -- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7580a1620 msgr2=0x7fa7581f4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75d8fb640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7580a1620 0x7fa7581f4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75d8fb640 1 -- 172.21.15.59:0/1918040463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7580a1a80 con 0x7fa7580b68d0 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75d8fb640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7580b68d0 0x7fa75806bb00 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fa75808cb30 tx=0x7fa75000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa74dffb640 1 -- 172.21.15.59:0/1918040463 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75000edd0 con 0x7fa7580b68d0 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa74dffb640 1 -- 172.21.15.59:0/1918040463 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa750014430 con 0x7fa7580b68d0 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.935+0000 7fa75f35e640 1 -- 172.21.15.59:0/1918040463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7581f54c0 con 0x7fa7580b68d0 2024-03-06T07:29:59.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.936+0000 7fa74dffb640 1 -- 172.21.15.59:0/1918040463 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7500147e0 con 0x7fa7580b68d0 2024-03-06T07:29:59.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.936+0000 7fa75f35e640 1 -- 172.21.15.59:0/1918040463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7581f5a80 con 0x7fa7580b68d0 2024-03-06T07:29:59.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.937+0000 7fa74dffb640 1 -- 172.21.15.59:0/1918040463 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa750014980 con 0x7fa7580b68d0 2024-03-06T07:29:59.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.937+0000 7fa75f35e640 1 -- 172.21.15.59:0/1918040463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7580cf5f0 con 0x7fa7580b68d0 2024-03-06T07:29:59.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.937+0000 7fa74dffb640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa73c06b500 0x7fa73c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.938+0000 7fa74dffb640 1 -- 172.21.15.59:0/1918040463 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa750089fd0 con 0x7fa7580b68d0 2024-03-06T07:29:59.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.938+0000 7fa75d0fa640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa73c06b500 0x7fa73c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.938+0000 7fa75d0fa640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa73c06b500 0x7fa73c06d9c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa754036000 tx=0x7fa754030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:29:59.942+0000 7fa74dffb640 1 -- 172.21.15.59:0/1918040463 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7580cf5f0 con 0x7fa7580b68d0 2024-03-06T07:30:00.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.281+0000 7fa75f35e640 1 -- 172.21.15.59:0/1918040463 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa75804f330 con 0x7fa7580b68d0 2024-03-06T07:30:00.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.281+0000 7fa74dffb640 1 -- 172.21.15.59:0/1918040463 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+3399 (secure 0 0 0) 0x7fa7500587c0 con 0x7fa7580b68d0 2024-03-06T07:30:00.283 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:00.283 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/2939413041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2939413041},{"type":"v1","addr":"172.21.15.59:6833","nonce":2939413041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:52.591736+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8676},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:30:00.284 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 90 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.283+0000 7fa75f35e640 1 -- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa73c06b500 msgr2=0x7fa73c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.283+0000 7fa75f35e640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa73c06b500 0x7fa73c06d9c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa754036000 tx=0x7fa754030000 comp rx=0 tx=0).stop 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.283+0000 7fa75f35e640 1 -- 172.21.15.59:0/1918040463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7580b68d0 msgr2=0x7fa75806bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.283+0000 7fa75f35e640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7580b68d0 0x7fa75806bb00 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fa75808cb30 tx=0x7fa75000e530 comp rx=0 tx=0).stop 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.284+0000 7fa75f35e640 1 -- 172.21.15.59:0/1918040463 shutdown_connections 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.284+0000 7fa75f35e640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa73c06b500 0x7fa73c06d9c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.284+0000 7fa75f35e640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7580b68d0 0x7fa75806bb00 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.284+0000 7fa75f35e640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7580a1620 0x7fa7581f4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.284+0000 7fa75f35e640 1 --2- 172.21.15.59:0/1918040463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7440039b0 0x7fa7580cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.284+0000 7fa75f35e640 1 -- 172.21.15.59:0/1918040463 >> 172.21.15.59:0/1918040463 conn(0x7fa7580d3cd0 msgr2=0x7fa7580b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.284+0000 7fa75f35e640 1 -- 172.21.15.59:0/1918040463 shutdown_connections 2024-03-06T07:30:00.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.284+0000 7fa75f35e640 1 -- 172.21.15.59:0/1918040463 wait complete. 2024-03-06T07:30:00.298 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:00.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.403+0000 7fbcd40b5640 1 Processor -- start 2024-03-06T07:30:00.405 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.403+0000 7fbcd40b5640 1 -- start start 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd40b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbccc0d04a0 0x7fbccc0ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd40b5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbccc0d0910 con 0x7fbccc15bb20 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd40b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbccc161980 con 0x7fbccc0cfff0 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd40b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc0d2500 con 0x7fbccc0d04a0 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd1e51640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbccc15bb20 0x7fbccc0d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60350/0 (socket says 172.21.15.59:60350) 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd1e51640 1 -- 172.21.15.59:0/40191759 learned_addr learned my addr 172.21.15.59:0/40191759 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd2652640 1 --2- 172.21.15.59:0/40191759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbccc0d04a0 0x7fbccc0ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd0e4f640 1 -- 172.21.15.59:0/40191759 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306496589 0 0) 0x7fbccc0d0910 con 0x7fbccc15bb20 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd0e4f640 1 -- 172.21.15.59:0/40191759 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcb8003520 con 0x7fbccc15bb20 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd2652640 1 -- 172.21.15.59:0/40191759 >> v1:172.21.15.164:6789/0 conn(0x7fbccc0cfff0 legacy=0x7fbccc0cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd0e4f640 1 -- 172.21.15.59:0/40191759 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513382794 0 0) 0x7fbccc161980 con 0x7fbccc0cfff0 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd2652640 1 -- 172.21.15.59:0/40191759 >> v1:172.21.15.59:6789/0 conn(0x7fbccc15bb20 legacy=0x7fbccc0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.404+0000 7fbcd2652640 1 -- 172.21.15.59:0/40191759 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc161c60 con 0x7fbccc0d04a0 2024-03-06T07:30:00.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.405+0000 7fbcd2652640 1 --2- 172.21.15.59:0/40191759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbccc0d04a0 0x7fbccc0ccfc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009dc0 tx=0x7fbcc402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f854997612ff8ef8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.405+0000 7fbcd0e4f640 1 -- 172.21.15.59:0/40191759 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4039070 con 0x7fbccc0d04a0 2024-03-06T07:30:00.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.405+0000 7fbcd0e4f640 1 -- 172.21.15.59:0/40191759 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc402ccd0 con 0x7fbccc0d04a0 2024-03-06T07:30:00.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.405+0000 7fbcd0e4f640 1 -- 172.21.15.59:0/40191759 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4034660 con 0x7fbccc0d04a0 2024-03-06T07:30:00.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.406+0000 7fbcd40b5640 1 -- 172.21.15.59:0/40191759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbccc0d04a0 msgr2=0x7fbccc0ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.406+0000 7fbcd40b5640 1 --2- 172.21.15.59:0/40191759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbccc0d04a0 0x7fbccc0ccfc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009dc0 tx=0x7fbcc402bf20 comp rx=0 tx=0).stop 2024-03-06T07:30:00.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.406+0000 7fbcd40b5640 1 -- 172.21.15.59:0/40191759 shutdown_connections 2024-03-06T07:30:00.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.406+0000 7fbcd40b5640 1 --2- 172.21.15.59:0/40191759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbccc0d04a0 0x7fbccc0ccfc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.406+0000 7fbcd40b5640 1 -- 172.21.15.59:0/40191759 >> 172.21.15.59:0/40191759 conn(0x7fbccc09a9f0 msgr2=0x7fbccc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.406+0000 7fbcd40b5640 1 -- 172.21.15.59:0/40191759 shutdown_connections 2024-03-06T07:30:00.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.406+0000 7fbcd40b5640 1 -- 172.21.15.59:0/40191759 wait complete. 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.407+0000 7fbcd40b5640 1 Processor -- start 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.407+0000 7fbcd40b5640 1 -- start start 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd40b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbccc0cfff0 0x7fbccc0c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd40b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbccc0d04a0 0x7fbccc1606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd40b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbccc15bb20 0x7fbccc14e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd40b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc162500 con 0x7fbccc15bb20 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd40b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc162220 con 0x7fbccc0d04a0 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd40b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc0d1980 con 0x7fbccc0cfff0 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd2652640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbccc15bb20 0x7fbccc14e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd2652640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbccc15bb20 0x7fbccc14e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51440/0 (socket says 172.21.15.59:51440) 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd2652640 1 -- 172.21.15.59:0/495539416 learned_addr learned my addr 172.21.15.59:0/495539416 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd1e51640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbccc0cfff0 0x7fbccc0c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd1650640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbccc0d04a0 0x7fbccc1606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd2652640 1 -- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbccc0cfff0 msgr2=0x7fbccc0c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd2652640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbccc0cfff0 0x7fbccc0c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd2652640 1 -- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbccc0d04a0 msgr2=0x7fbccc1606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd2652640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbccc0d04a0 0x7fbccc1606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd2652640 1 -- 172.21.15.59:0/495539416 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc0c83a0 con 0x7fbccc15bb20 2024-03-06T07:30:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd2652640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbccc15bb20 0x7fbccc14e990 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009ee0 tx=0x7fbcc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcc27fc640 1 -- 172.21.15.59:0/495539416 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4039050 con 0x7fbccc15bb20 2024-03-06T07:30:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcc27fc640 1 -- 172.21.15.59:0/495539416 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc4045070 con 0x7fbccc15bb20 2024-03-06T07:30:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcd40b5640 1 -- 172.21.15.59:0/495539416 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc0c62e0 con 0x7fbccc15bb20 2024-03-06T07:30:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.408+0000 7fbcc27fc640 1 -- 172.21.15.59:0/495539416 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4009450 con 0x7fbccc15bb20 2024-03-06T07:30:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.409+0000 7fbcd40b5640 1 -- 172.21.15.59:0/495539416 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc0c5fa0 con 0x7fbccc15bb20 2024-03-06T07:30:00.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.410+0000 7fbcd40b5640 1 -- 172.21.15.59:0/495539416 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbccc0ce1d0 con 0x7fbccc15bb20 2024-03-06T07:30:00.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.410+0000 7fbcc27fc640 1 -- 172.21.15.59:0/495539416 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcc403e850 con 0x7fbccc15bb20 2024-03-06T07:30:00.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.410+0000 7fbcc27fc640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcb006b4d0 0x7fbcb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.410+0000 7fbcc27fc640 1 -- 172.21.15.59:0/495539416 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbcc40ad050 con 0x7fbccc15bb20 2024-03-06T07:30:00.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.411+0000 7fbcd1e51640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcb006b4d0 0x7fbcb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.411+0000 7fbcd1e51640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcb006b4d0 0x7fbcb006d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8005880 tx=0x7fbcc8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.415+0000 7fbcc27fc640 1 -- 172.21.15.59:0/495539416 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbccc0ce1d0 con 0x7fbccc15bb20 2024-03-06T07:30:00.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.597+0000 7fbcd40b5640 1 -- 172.21.15.59:0/495539416 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbccc04f330 con 0x7fbccc15bb20 2024-03-06T07:30:00.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.598+0000 7fbcc27fc640 1 -- 172.21.15.59:0/495539416 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16708 (secure 0 0 0) 0x7fbcc407c040 con 0x7fbccc15bb20 2024-03-06T07:30:00.600 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:00.600 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":130,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:29:39.117588+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:40.558382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:41.255127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.3","mappings":[{"from":2,"to":0}]},{"pgid":"19.25","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:00.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 -- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcb006b4d0 msgr2=0x7fbcb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcb006b4d0 0x7fbcb006d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fbcc8005880 tx=0x7fbcc8009040 comp rx=0 tx=0).stop 2024-03-06T07:30:00.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 -- 172.21.15.59:0/495539416 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbccc15bb20 msgr2=0x7fbccc14e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbccc15bb20 0x7fbccc14e990 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009ee0 tx=0x7fbcc4004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:00.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 -- 172.21.15.59:0/495539416 shutdown_connections 2024-03-06T07:30:00.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcb006b4d0 0x7fbcb006d990 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbccc15bb20 0x7fbccc14e990 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbccc0d04a0 0x7fbccc1606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 --2- 172.21.15.59:0/495539416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbccc0cfff0 0x7fbccc0c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 -- 172.21.15.59:0/495539416 >> 172.21.15.59:0/495539416 conn(0x7fbccc09a9f0 msgr2=0x7fbccc06f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 -- 172.21.15.59:0/495539416 shutdown_connections 2024-03-06T07:30:00.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.600+0000 7fbcd40b5640 1 -- 172.21.15.59:0/495539416 wait complete. 2024-03-06T07:30:00.617 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:30:00.617 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:00.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.716+0000 7fbfe5f3a640 1 Processor -- start 2024-03-06T07:30:00.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.716+0000 7fbfe5f3a640 1 -- start start 2024-03-06T07:30:00.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfe5f3a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbfe015dfe0 0x7fbfe00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfe5f3a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbfe00d29c0 0x7fbfe00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfe5f3a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe00b3430 con 0x7fbfe00cf830 2024-03-06T07:30:00.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfe5f3a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe00b31e0 con 0x7fbfe00d29c0 2024-03-06T07:30:00.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfe5f3a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe00b3d60 con 0x7fbfe015dfe0 2024-03-06T07:30:00.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfdffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbfe00d29c0 0x7fbfe00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfdeffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbfe00cf830 0x7fbfe00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60356/0 (socket says 172.21.15.59:60356) 2024-03-06T07:30:00.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfdeffd640 1 -- 172.21.15.59:0/2075864680 learned_addr learned my addr 172.21.15.59:0/2075864680 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:00.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfdf7fe640 1 --2- 172.21.15.59:0/2075864680 >> v2:172.21.15.164:3300/0 conn(0x7fbfe015dfe0 0x7fbfe00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfdffff640 1 -- 172.21.15.59:0/2075864680 >> v2:172.21.15.164:3300/0 conn(0x7fbfe015dfe0 msgr2=0x7fbfe00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfdffff640 1 --2- 172.21.15.59:0/2075864680 >> v2:172.21.15.164:3300/0 conn(0x7fbfe015dfe0 0x7fbfe00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfdffff640 1 -- 172.21.15.59:0/2075864680 >> v1:172.21.15.59:6789/0 conn(0x7fbfe00cf830 legacy=0x7fbfe00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfdffff640 1 -- 172.21.15.59:0/2075864680 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe00b3710 con 0x7fbfe00d29c0 2024-03-06T07:30:00.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.718+0000 7fbfde7fc640 1 -- 172.21.15.59:0/2075864680 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536773774 0 0) 0x7fbfe00b3430 con 0x7fbfe00cf830 2024-03-06T07:30:00.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.719+0000 7fbfdffff640 1 --2- 172.21.15.59:0/2075864680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe00d29c0 0x7fbfe00b41f0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0067990 tx=0x7fbfd802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1a070fb123fb93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.719+0000 7fbfde7fc640 1 -- 172.21.15.59:0/2075864680 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8039070 con 0x7fbfe00d29c0 2024-03-06T07:30:00.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.719+0000 7fbfde7fc640 1 -- 172.21.15.59:0/2075864680 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd802cc80 con 0x7fbfe00d29c0 2024-03-06T07:30:00.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.719+0000 7fbfde7fc640 1 -- 172.21.15.59:0/2075864680 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8034660 con 0x7fbfe00d29c0 2024-03-06T07:30:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.720+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/2075864680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe00d29c0 msgr2=0x7fbfe00b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.720+0000 7fbfe5f3a640 1 --2- 172.21.15.59:0/2075864680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe00d29c0 0x7fbfe00b41f0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0067990 tx=0x7fbfd802bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.720+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/2075864680 shutdown_connections 2024-03-06T07:30:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.720+0000 7fbfe5f3a640 1 --2- 172.21.15.59:0/2075864680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe00d29c0 0x7fbfe00b41f0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.720+0000 7fbfe5f3a640 1 --2- 172.21.15.59:0/2075864680 >> v2:172.21.15.164:3300/0 conn(0x7fbfe015dfe0 0x7fbfe00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.720+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/2075864680 >> 172.21.15.59:0/2075864680 conn(0x7fbfe00d3cd0 msgr2=0x7fbfe00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.720+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/2075864680 shutdown_connections 2024-03-06T07:30:00.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.720+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/2075864680 wait complete. 2024-03-06T07:30:00.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.722+0000 7fbfe5f3a640 1 Processor -- start 2024-03-06T07:30:00.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.722+0000 7fbfe5f3a640 1 -- start start 2024-03-06T07:30:00.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfe5f3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe00cf830 0x7fbfe01f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfe5f3a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe00d29c0 0x7fbfe01f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfe5f3a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe015dfe0 0x7fbfe00b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfe5f3a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00d1df0 con 0x7fbfe00cf830 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfe5f3a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00d1c70 con 0x7fbfe00d29c0 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfe5f3a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe015e3c0 con 0x7fbfe015dfe0 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfdf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe00cf830 0x7fbfe01f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfdf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe00cf830 0x7fbfe01f31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51464/0 (socket says 172.21.15.59:51464) 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfdf7fe640 1 -- 172.21.15.59:0/162032569 learned_addr learned my addr 172.21.15.59:0/162032569 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfdffff640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe015dfe0 0x7fbfe00b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfdeffd640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe00d29c0 0x7fbfe01f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfdf7fe640 1 -- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe015dfe0 msgr2=0x7fbfe00b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfdf7fe640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe015dfe0 0x7fbfe00b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.723+0000 7fbfdf7fe640 1 -- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe00d29c0 msgr2=0x7fbfe01f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.724+0000 7fbfdf7fe640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe00d29c0 0x7fbfe01f42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.724+0000 7fbfdf7fe640 1 -- 172.21.15.59:0/162032569 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe01f3700 con 0x7fbfe00cf830 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.724+0000 7fbfdffff640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe015dfe0 0x7fbfe00b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.724+0000 7fbfdeffd640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe00d29c0 0x7fbfe01f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.724+0000 7fbfdf7fe640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe00cf830 0x7fbfe01f31c0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000d5f0 tx=0x7fbfd000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.724+0000 7fbfc7fff640 1 -- 172.21.15.59:0/162032569 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd00125f0 con 0x7fbfe00cf830 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.724+0000 7fbfc7fff640 1 -- 172.21.15.59:0/162032569 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd0019070 con 0x7fbfe00cf830 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.724+0000 7fbfc7fff640 1 -- 172.21.15.59:0/162032569 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd00129d0 con 0x7fbfe00cf830 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.724+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/162032569 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe01f4820 con 0x7fbfe00cf830 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.724+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/162032569 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe00b9080 con 0x7fbfe00cf830 2024-03-06T07:30:00.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.725+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/162032569 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe004f330 con 0x7fbfe00cf830 2024-03-06T07:30:00.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.725+0000 7fbfc7fff640 1 -- 172.21.15.59:0/162032569 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfd0012b70 con 0x7fbfe00cf830 2024-03-06T07:30:00.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.726+0000 7fbfc7fff640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfcc06b4d0 0x7fbfcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.726+0000 7fbfc7fff640 1 -- 172.21.15.59:0/162032569 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbfd008a630 con 0x7fbfe00cf830 2024-03-06T07:30:00.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.726+0000 7fbfdeffd640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfcc06b4d0 0x7fbfcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.727+0000 7fbfdeffd640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfcc06b4d0 0x7fbfcc06d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fbfe015bf70 tx=0x7fbfd4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.730+0000 7fbfc7fff640 1 -- 172.21.15.59:0/162032569 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfe01f4820 con 0x7fbfe00cf830 2024-03-06T07:30:00.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.950+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/162032569 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfe00b49c0 con 0x7fbfe00cf830 2024-03-06T07:30:00.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.951+0000 7fbfc7fff640 1 -- 172.21.15.59:0/162032569 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+3399 (secure 0 0 0) 0x7fbfd0059470 con 0x7fbfe00cf830 2024-03-06T07:30:00.953 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:00.953 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/2939413041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2939413041},{"type":"v1","addr":"172.21.15.59:6833","nonce":2939413041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:52.591736+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8676},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:30:00.953 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 90 2024-03-06T07:30:00.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.952+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfcc06b4d0 msgr2=0x7fbfcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.952+0000 7fbfe5f3a640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfcc06b4d0 0x7fbfcc06d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fbfe015bf70 tx=0x7fbfd4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.952+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/162032569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe00cf830 msgr2=0x7fbfe01f31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.952+0000 7fbfe5f3a640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe00cf830 0x7fbfe01f31c0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000d5f0 tx=0x7fbfd000dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.952+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/162032569 shutdown_connections 2024-03-06T07:30:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.952+0000 7fbfe5f3a640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfcc06b4d0 0x7fbfcc06d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.952+0000 7fbfe5f3a640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe015dfe0 0x7fbfe00b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.952+0000 7fbfe5f3a640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe00d29c0 0x7fbfe01f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.952+0000 7fbfe5f3a640 1 --2- 172.21.15.59:0/162032569 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfe00cf830 0x7fbfe01f31c0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.952+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/162032569 >> 172.21.15.59:0/162032569 conn(0x7fbfe00d3cd0 msgr2=0x7fbfe006c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.953+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/162032569 shutdown_connections 2024-03-06T07:30:00.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:00.953+0000 7fbfe5f3a640 1 -- 172.21.15.59:0/162032569 wait complete. 2024-03-06T07:30:00.965 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:01.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.089+0000 7f868b93a640 1 Processor -- start 2024-03-06T07:30:01.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.089+0000 7f868b93a640 1 -- start start 2024-03-06T07:30:01.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f868b93a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f868415dfe0 0x7f86840cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f868b93a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86840d29c0 0x7f86840b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f868b93a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86840b3430 con 0x7f86840cf830 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f868b93a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86840b31e0 con 0x7f868415dfe0 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f868b93a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86840b3d60 con 0x7f86840d29c0 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f86896d6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f868415dfe0 0x7f86840cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f8688ed5640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f86840cf830 0x7f86840d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60358/0 (socket says 172.21.15.59:60358) 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f8688ed5640 1 -- 172.21.15.59:0/482955032 learned_addr learned my addr 172.21.15.59:0/482955032 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f86896d6640 1 --2- 172.21.15.59:0/482955032 >> v2:172.21.15.59:3300/0 conn(0x7f868415dfe0 0x7f86840cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51474/0 (socket says 172.21.15.59:51474) 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f8689ed7640 1 --2- 172.21.15.59:0/482955032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86840d29c0 0x7f86840b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f86896d6640 1 -- 172.21.15.59:0/482955032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86840d29c0 msgr2=0x7f86840b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f86896d6640 1 --2- 172.21.15.59:0/482955032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86840d29c0 0x7f86840b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f86896d6640 1 -- 172.21.15.59:0/482955032 >> v1:172.21.15.59:6789/0 conn(0x7f86840cf830 legacy=0x7f86840d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f86896d6640 1 -- 172.21.15.59:0/482955032 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86840b3710 con 0x7f868415dfe0 2024-03-06T07:30:01.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.090+0000 7f8689ed7640 1 --2- 172.21.15.59:0/482955032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86840d29c0 0x7f86840b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.091+0000 7f86896d6640 1 --2- 172.21.15.59:0/482955032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868415dfe0 0x7f86840cf2f0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f867c0097a0 tx=0x7f867c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67caefead9a2d27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.091+0000 7f8683fff640 1 -- 172.21.15.59:0/482955032 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c035020 con 0x7f868415dfe0 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.091+0000 7f8683fff640 1 -- 172.21.15.59:0/482955032 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c031830 con 0x7f868415dfe0 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.091+0000 7f8683fff640 1 -- 172.21.15.59:0/482955032 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c031c00 con 0x7f868415dfe0 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.092+0000 7f868b93a640 1 -- 172.21.15.59:0/482955032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868415dfe0 msgr2=0x7f86840cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.092+0000 7f868b93a640 1 --2- 172.21.15.59:0/482955032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868415dfe0 0x7f86840cf2f0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f867c0097a0 tx=0x7f867c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.092+0000 7f868b93a640 1 -- 172.21.15.59:0/482955032 shutdown_connections 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.092+0000 7f868b93a640 1 --2- 172.21.15.59:0/482955032 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86840d29c0 0x7f86840b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.092+0000 7f868b93a640 1 --2- 172.21.15.59:0/482955032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868415dfe0 0x7f86840cf2f0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.092+0000 7f868b93a640 1 -- 172.21.15.59:0/482955032 >> 172.21.15.59:0/482955032 conn(0x7f86840d3cd0 msgr2=0x7f86840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.092+0000 7f868b93a640 1 -- 172.21.15.59:0/482955032 shutdown_connections 2024-03-06T07:30:01.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.092+0000 7f868b93a640 1 -- 172.21.15.59:0/482955032 wait complete. 2024-03-06T07:30:01.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.093+0000 7f868b93a640 1 Processor -- start 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.093+0000 7f868b93a640 1 -- start start 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.094+0000 7f868b93a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86840cf830 0x7f86841f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.094+0000 7f868b93a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86840d29c0 0x7f8684160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.094+0000 7f868b93a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868415dfe0 0x7f86840b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.094+0000 7f868b93a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86840d1df0 con 0x7f868415dfe0 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.094+0000 7f868b93a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86840d1c70 con 0x7f86840d29c0 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.094+0000 7f868b93a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868415e3c0 con 0x7f86840cf830 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.094+0000 7f8689ed7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868415dfe0 0x7f86840b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.094+0000 7f8689ed7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868415dfe0 0x7f86840b88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51478/0 (socket says 172.21.15.59:51478) 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.094+0000 7f8689ed7640 1 -- 172.21.15.59:0/56780875 learned_addr learned my addr 172.21.15.59:0/56780875 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.094+0000 7f86896d6640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86840cf830 0x7f86841f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.094+0000 7f8688ed5640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86840d29c0 0x7f8684160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.095+0000 7f8689ed7640 1 -- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86840cf830 msgr2=0x7f86841f0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.095+0000 7f8689ed7640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86840cf830 0x7f86841f0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.095+0000 7f8689ed7640 1 -- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86840d29c0 msgr2=0x7f8684160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.095+0000 7f8689ed7640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86840d29c0 0x7f8684160f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.095+0000 7f8689ed7640 1 -- 172.21.15.59:0/56780875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86841f1520 con 0x7f868415dfe0 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.095+0000 7f8689ed7640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868415dfe0 0x7f86840b88b0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f867400cef0 tx=0x7f867400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.095+0000 7f8681ffb640 1 -- 172.21.15.59:0/56780875 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867401a070 con 0x7f868415dfe0 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.095+0000 7f8681ffb640 1 -- 172.21.15.59:0/56780875 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867400d860 con 0x7f868415dfe0 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.095+0000 7f868b93a640 1 -- 172.21.15.59:0/56780875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86841601f0 con 0x7f868415dfe0 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.095+0000 7f8681ffb640 1 -- 172.21.15.59:0/56780875 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867400dc50 con 0x7f868415dfe0 2024-03-06T07:30:01.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.095+0000 7f868b93a640 1 -- 172.21.15.59:0/56780875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8684160790 con 0x7f868415dfe0 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.096+0000 7f868b93a640 1 -- 172.21.15.59:0/56780875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868404f330 con 0x7f868415dfe0 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.097+0000 7f8681ffb640 1 -- 172.21.15.59:0/56780875 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f867401d070 con 0x7f868415dfe0 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.097+0000 7f8681ffb640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867006b4d0 0x7f867006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.097+0000 7f8681ffb640 1 -- 172.21.15.59:0/56780875 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f867408a9d0 con 0x7f868415dfe0 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.097+0000 7f86896d6640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867006b4d0 0x7f867006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.098+0000 7f86896d6640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867006b4d0 0x7f867006d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f867c036000 tx=0x7f867c03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.101+0000 7f8681ffb640 1 -- 172.21.15.59:0/56780875 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8674059310 con 0x7f868415dfe0 2024-03-06T07:30:01.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.319+0000 7f868b93a640 1 -- 172.21.15.59:0/56780875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86840cab60 con 0x7f868415dfe0 2024-03-06T07:30:01.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.320+0000 7f8681ffb640 1 -- 172.21.15.59:0/56780875 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+3399 (secure 0 0 0) 0x7f867405d3b0 con 0x7f868415dfe0 2024-03-06T07:30:01.322 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:01.322 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/2939413041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2939413041},{"type":"v1","addr":"172.21.15.59:6833","nonce":2939413041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:42.288807+0000","modified":"2024-03-06T07:29:52.591736+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8676},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8676":{"gid":8676,"name":"c","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6835/1740714701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":1740714701},{"type":"v1","addr":"172.21.15.59:6835","nonce":1740714701}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8676,"qdb_cluster":[8676]},"id":9}]} 2024-03-06T07:30:01.322 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 90 2024-03-06T07:30:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 -- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867006b4d0 msgr2=0x7f867006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867006b4d0 0x7f867006d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f867c036000 tx=0x7f867c03c040 comp rx=0 tx=0).stop 2024-03-06T07:30:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 -- 172.21.15.59:0/56780875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868415dfe0 msgr2=0x7f86840b88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868415dfe0 0x7f86840b88b0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f867400cef0 tx=0x7f867400a710 comp rx=0 tx=0).stop 2024-03-06T07:30:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 -- 172.21.15.59:0/56780875 shutdown_connections 2024-03-06T07:30:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f867006b4d0 0x7f867006d990 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f868415dfe0 0x7f86840b88b0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86840d29c0 0x7f8684160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 --2- 172.21.15.59:0/56780875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86840cf830 0x7f86841f0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 -- 172.21.15.59:0/56780875 >> 172.21.15.59:0/56780875 conn(0x7f86840d3cd0 msgr2=0x7f868406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 -- 172.21.15.59:0/56780875 shutdown_connections 2024-03-06T07:30:01.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.322+0000 7f868b93a640 1 -- 172.21.15.59:0/56780875 wait complete. 2024-03-06T07:30:01.334 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:01.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.451+0000 7fda56fbc640 1 Processor -- start 2024-03-06T07:30:01.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.452+0000 7fda56fbc640 1 -- start start 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.452+0000 7fda56fbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda500d16e0 0x7fda500d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.452+0000 7fda56fbc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda500cebe0 con 0x7fda5015bdd0 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.452+0000 7fda56fbc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda500ceec0 con 0x7fda5015fa10 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.452+0000 7fda56fbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda500ce980 con 0x7fda500d16e0 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.452+0000 7fda54d58640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fda5015bdd0 0x7fda500c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60366/0 (socket says 172.21.15.59:60366) 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.452+0000 7fda54d58640 1 -- 172.21.15.59:0/717663937 learned_addr learned my addr 172.21.15.59:0/717663937 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.453+0000 7fda55559640 1 --2- 172.21.15.59:0/717663937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda500d16e0 0x7fda500d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.453+0000 7fda4f7fe640 1 -- 172.21.15.59:0/717663937 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166176186 0 0) 0x7fda500cebe0 con 0x7fda5015bdd0 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.453+0000 7fda4f7fe640 1 -- 172.21.15.59:0/717663937 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda38003520 con 0x7fda5015bdd0 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.453+0000 7fda55559640 1 -- 172.21.15.59:0/717663937 >> v1:172.21.15.164:6789/0 conn(0x7fda5015fa10 legacy=0x7fda500ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.453+0000 7fda55559640 1 -- 172.21.15.59:0/717663937 >> v1:172.21.15.59:6789/0 conn(0x7fda5015bdd0 legacy=0x7fda500c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.453+0000 7fda55559640 1 -- 172.21.15.59:0/717663937 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda500cf1a0 con 0x7fda500d16e0 2024-03-06T07:30:01.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.454+0000 7fda55559640 1 --2- 172.21.15.59:0/717663937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda500d16e0 0x7fda500d1b10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fda40009dc0 tx=0x7fda4002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f87312e4b3f64ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.454+0000 7fda4f7fe640 1 -- 172.21.15.59:0/717663937 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40039070 con 0x7fda500d16e0 2024-03-06T07:30:01.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.454+0000 7fda4f7fe640 1 -- 172.21.15.59:0/717663937 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4002cc40 con 0x7fda500d16e0 2024-03-06T07:30:01.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.454+0000 7fda4f7fe640 1 -- 172.21.15.59:0/717663937 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40034440 con 0x7fda500d16e0 2024-03-06T07:30:01.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.455+0000 7fda56fbc640 1 -- 172.21.15.59:0/717663937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda500d16e0 msgr2=0x7fda500d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.455+0000 7fda56fbc640 1 --2- 172.21.15.59:0/717663937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda500d16e0 0x7fda500d1b10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fda40009dc0 tx=0x7fda4002be90 comp rx=0 tx=0).stop 2024-03-06T07:30:01.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.455+0000 7fda56fbc640 1 -- 172.21.15.59:0/717663937 shutdown_connections 2024-03-06T07:30:01.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.455+0000 7fda56fbc640 1 --2- 172.21.15.59:0/717663937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda500d16e0 0x7fda500d1b10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.455+0000 7fda56fbc640 1 -- 172.21.15.59:0/717663937 >> 172.21.15.59:0/717663937 conn(0x7fda5009a9f0 msgr2=0x7fda5009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.455+0000 7fda56fbc640 1 -- 172.21.15.59:0/717663937 shutdown_connections 2024-03-06T07:30:01.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.455+0000 7fda56fbc640 1 -- 172.21.15.59:0/717663937 wait complete. 2024-03-06T07:30:01.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.456+0000 7fda56fbc640 1 Processor -- start 2024-03-06T07:30:01.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.456+0000 7fda56fbc640 1 -- start start 2024-03-06T07:30:01.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda56fbc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda500d16e0 0x7fda500ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda56fbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda5015bdd0 0x7fda5015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda56fbc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5015fa10 0x7fda501e5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda56fbc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda500cc5d0 con 0x7fda5015fa10 2024-03-06T07:30:01.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda56fbc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda500d2050 con 0x7fda500d16e0 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda56fbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5015c1b0 con 0x7fda5015bdd0 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda55559640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5015fa10 0x7fda501e5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda55559640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5015fa10 0x7fda501e5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51488/0 (socket says 172.21.15.59:51488) 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda55559640 1 -- 172.21.15.59:0/3937261778 learned_addr learned my addr 172.21.15.59:0/3937261778 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda4ffff640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda5015bdd0 0x7fda5015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda54d58640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda500d16e0 0x7fda500ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda55559640 1 -- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda5015bdd0 msgr2=0x7fda5015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda55559640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda5015bdd0 0x7fda5015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda55559640 1 -- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda500d16e0 msgr2=0x7fda500ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda55559640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda500d16e0 0x7fda500ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda55559640 1 -- 172.21.15.59:0/3937261778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda500b5c40 con 0x7fda5015fa10 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.458+0000 7fda4ffff640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda5015bdd0 0x7fda5015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.459+0000 7fda55559640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5015fa10 0x7fda501e5730 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fda40009ee0 tx=0x7fda40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.459+0000 7fda4d7fa640 1 -- 172.21.15.59:0/3937261778 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40042020 con 0x7fda5015fa10 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.459+0000 7fda4d7fa640 1 -- 172.21.15.59:0/3937261778 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4003a040 con 0x7fda5015fa10 2024-03-06T07:30:01.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.459+0000 7fda56fbc640 1 -- 172.21.15.59:0/3937261778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda500cad60 con 0x7fda5015fa10 2024-03-06T07:30:01.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.459+0000 7fda4d7fa640 1 -- 172.21.15.59:0/3937261778 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40039070 con 0x7fda5015fa10 2024-03-06T07:30:01.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.459+0000 7fda56fbc640 1 -- 172.21.15.59:0/3937261778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda500cb320 con 0x7fda5015fa10 2024-03-06T07:30:01.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.460+0000 7fda56fbc640 1 -- 172.21.15.59:0/3937261778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda500d11c0 con 0x7fda5015fa10 2024-03-06T07:30:01.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.460+0000 7fda4d7fa640 1 -- 172.21.15.59:0/3937261778 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda40045020 con 0x7fda5015fa10 2024-03-06T07:30:01.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.461+0000 7fda4d7fa640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda3c06b500 0x7fda3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.461+0000 7fda4d7fa640 1 -- 172.21.15.59:0/3937261778 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fda400ac360 con 0x7fda5015fa10 2024-03-06T07:30:01.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.461+0000 7fda54d58640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda3c06b500 0x7fda3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.462+0000 7fda54d58640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda3c06b500 0x7fda3c06d9c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fda480011e0 tx=0x7fda48009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.465+0000 7fda4d7fa640 1 -- 172.21.15.59:0/3937261778 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda500d11c0 con 0x7fda5015fa10 2024-03-06T07:30:01.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.665+0000 7fda56fbc640 1 -- 172.21.15.59:0/3937261778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda500b5e10 con 0x7fda5015fa10 2024-03-06T07:30:01.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.666+0000 7fda4d7fa640 1 -- 172.21.15.59:0/3937261778 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16708 (secure 0 0 0) 0x7fda400a9160 con 0x7fda5015fa10 2024-03-06T07:30:01.668 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:01.669 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":130,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:29:39.117588+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:40.558382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:41.255127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.3","mappings":[{"from":2,"to":0}]},{"pgid":"19.25","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:01.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.668+0000 7fda56fbc640 1 -- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda3c06b500 msgr2=0x7fda3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.668+0000 7fda56fbc640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda3c06b500 0x7fda3c06d9c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fda480011e0 tx=0x7fda48009040 comp rx=0 tx=0).stop 2024-03-06T07:30:01.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.668+0000 7fda56fbc640 1 -- 172.21.15.59:0/3937261778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5015fa10 msgr2=0x7fda501e5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.668+0000 7fda56fbc640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5015fa10 0x7fda501e5730 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fda40009ee0 tx=0x7fda40004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:01.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.669+0000 7fda56fbc640 1 -- 172.21.15.59:0/3937261778 shutdown_connections 2024-03-06T07:30:01.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.669+0000 7fda56fbc640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda3c06b500 0x7fda3c06d9c0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.669+0000 7fda56fbc640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda5015fa10 0x7fda501e5730 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.669+0000 7fda56fbc640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda5015bdd0 0x7fda5015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.669+0000 7fda56fbc640 1 --2- 172.21.15.59:0/3937261778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda500d16e0 0x7fda500ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.669+0000 7fda56fbc640 1 -- 172.21.15.59:0/3937261778 >> 172.21.15.59:0/3937261778 conn(0x7fda5009a9f0 msgr2=0x7fda5009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.669+0000 7fda56fbc640 1 -- 172.21.15.59:0/3937261778 shutdown_connections 2024-03-06T07:30:01.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.669+0000 7fda56fbc640 1 -- 172.21.15.59:0/3937261778 wait complete. 2024-03-06T07:30:01.682 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:30:01.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.785+0000 7fd0833e2640 1 Processor -- start 2024-03-06T07:30:01.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.786+0000 7fd0833e2640 1 -- start start 2024-03-06T07:30:01.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.786+0000 7fd0833e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07c15b380 0x7fd07c0708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.786+0000 7fd0833e2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd07c070e10 0x7fd07c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.786+0000 7fd0833e2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c06cd60 con 0x7fd07c158880 2024-03-06T07:30:01.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.786+0000 7fd0833e2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07c06d030 con 0x7fd07c070e10 2024-03-06T07:30:01.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.786+0000 7fd0833e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c06d310 con 0x7fd07c15b380 2024-03-06T07:30:01.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.787+0000 7fd08097d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07c15b380 0x7fd07c0708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.787+0000 7fd08197f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd07c070e10 0x7fd07c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.787+0000 7fd08197f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd07c070e10 0x7fd07c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:42212/0 (socket says 172.21.15.59:42212) 2024-03-06T07:30:01.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.787+0000 7fd08197f640 1 -- 172.21.15.59:0/1743497387 learned_addr learned my addr 172.21.15.59:0/1743497387 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:01.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.787+0000 7fd08097d640 1 -- 172.21.15.59:0/1743497387 >> v2:172.21.15.164:3300/0 conn(0x7fd07c070e10 msgr2=0x7fd07c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.787+0000 7fd08097d640 1 --2- 172.21.15.59:0/1743497387 >> v2:172.21.15.164:3300/0 conn(0x7fd07c070e10 0x7fd07c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.787+0000 7fd08097d640 1 -- 172.21.15.59:0/1743497387 >> v1:172.21.15.164:6789/0 conn(0x7fd07c158880 legacy=0x7fd07c06c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.787+0000 7fd08097d640 1 -- 172.21.15.59:0/1743497387 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c06d5f0 con 0x7fd07c15b380 2024-03-06T07:30:01.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.787+0000 7fd08197f640 1 --2- 172.21.15.59:0/1743497387 >> v2:172.21.15.164:3300/0 conn(0x7fd07c070e10 0x7fd07c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.788+0000 7fd08097d640 1 --2- 172.21.15.59:0/1743497387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07c15b380 0x7fd07c0708d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd070009d90 tx=0x7fd07002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa8e6164c902a957 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.789+0000 7fd06ffff640 1 -- 172.21.15.59:0/1743497387 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070039070 con 0x7fd07c15b380 2024-03-06T07:30:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.789+0000 7fd06ffff640 1 -- 172.21.15.59:0/1743497387 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07002cc50 con 0x7fd07c15b380 2024-03-06T07:30:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.789+0000 7fd06ffff640 1 -- 172.21.15.59:0/1743497387 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070034440 con 0x7fd07c15b380 2024-03-06T07:30:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.789+0000 7fd0833e2640 1 -- 172.21.15.59:0/1743497387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07c15b380 msgr2=0x7fd07c0708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.789+0000 7fd0833e2640 1 --2- 172.21.15.59:0/1743497387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07c15b380 0x7fd07c0708d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd070009d90 tx=0x7fd07002bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.789+0000 7fd0833e2640 1 -- 172.21.15.59:0/1743497387 shutdown_connections 2024-03-06T07:30:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.789+0000 7fd0833e2640 1 --2- 172.21.15.59:0/1743497387 >> v2:172.21.15.164:3300/0 conn(0x7fd07c070e10 0x7fd07c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.789+0000 7fd0833e2640 1 --2- 172.21.15.59:0/1743497387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07c15b380 0x7fd07c0708d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.789+0000 7fd0833e2640 1 -- 172.21.15.59:0/1743497387 >> 172.21.15.59:0/1743497387 conn(0x7fd07c0b1c80 msgr2=0x7fd07c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.789+0000 7fd0833e2640 1 -- 172.21.15.59:0/1743497387 shutdown_connections 2024-03-06T07:30:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.789+0000 7fd0833e2640 1 -- 172.21.15.59:0/1743497387 wait complete. 2024-03-06T07:30:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.790+0000 7fd0833e2640 1 Processor -- start 2024-03-06T07:30:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.790+0000 7fd0833e2640 1 -- start start 2024-03-06T07:30:01.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.791+0000 7fd0833e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07c070e10 0x7fd07c14a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.791+0000 7fd0833e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07c158880 0x7fd07c0b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.791+0000 7fd0833e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd07c15b380 0x7fd07c1e77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.791+0000 7fd0833e2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c0cdd00 con 0x7fd07c158880 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.791+0000 7fd0833e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c06dbb0 con 0x7fd07c15b380 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.791+0000 7fd0833e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c0cde80 con 0x7fd07c070e10 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.791+0000 7fd08097d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07c158880 0x7fd07c0b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.791+0000 7fd08097d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07c158880 0x7fd07c0b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51496/0 (socket says 172.21.15.59:51496) 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.791+0000 7fd08097d640 1 -- 172.21.15.59:0/107128589 learned_addr learned my addr 172.21.15.59:0/107128589 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.791+0000 7fd08197f640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd07c15b380 0x7fd07c1e77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.791+0000 7fd08117e640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07c070e10 0x7fd07c14a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd08097d640 1 -- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07c070e10 msgr2=0x7fd07c14a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd08097d640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07c070e10 0x7fd07c14a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd08097d640 1 -- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd07c15b380 msgr2=0x7fd07c1e77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd08097d640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd07c15b380 0x7fd07c1e77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd08097d640 1 -- 172.21.15.59:0/107128589 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c0a9580 con 0x7fd07c158880 2024-03-06T07:30:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd08197f640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd07c15b380 0x7fd07c1e77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd08097d640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07c158880 0x7fd07c0b3520 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fd070009d90 tx=0x7fd070004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd06dffb640 1 -- 172.21.15.59:0/107128589 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070039050 con 0x7fd07c158880 2024-03-06T07:30:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd06dffb640 1 -- 172.21.15.59:0/107128589 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070045070 con 0x7fd07c158880 2024-03-06T07:30:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd0833e2640 1 -- 172.21.15.59:0/107128589 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c0a9820 con 0x7fd07c158880 2024-03-06T07:30:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd06dffb640 1 -- 172.21.15.59:0/107128589 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07003e610 con 0x7fd07c158880 2024-03-06T07:30:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.792+0000 7fd0833e2640 1 -- 172.21.15.59:0/107128589 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c0a9de0 con 0x7fd07c158880 2024-03-06T07:30:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.793+0000 7fd0833e2640 1 -- 172.21.15.59:0/107128589 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07c04f330 con 0x7fd07c158880 2024-03-06T07:30:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.794+0000 7fd06dffb640 1 -- 172.21.15.59:0/107128589 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd07003e7b0 con 0x7fd07c158880 2024-03-06T07:30:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.794+0000 7fd06dffb640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06006b4d0 0x7fd06006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.794+0000 7fd06dffb640 1 -- 172.21.15.59:0/107128589 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd0700acd30 con 0x7fd07c158880 2024-03-06T07:30:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.794+0000 7fd08117e640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06006b4d0 0x7fd06006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.795+0000 7fd08117e640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06006b4d0 0x7fd06006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd078001780 tx=0x7fd078009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:01.798+0000 7fd06dffb640 1 -- 172.21.15.59:0/107128589 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd07007bd20 con 0x7fd07c158880 2024-03-06T07:30:02.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.024+0000 7fd0833e2640 1 -- 172.21.15.59:0/107128589 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd07c15b160 con 0x7fd07c158880 2024-03-06T07:30:02.026 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:30:02.024+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:30:02.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.026+0000 7fd06dffb640 1 -- 172.21.15.59:0/107128589 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v91) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd07007b620 con 0x7fd07c158880 2024-03-06T07:30:02.027 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:30:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 -- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06006b4d0 msgr2=0x7fd06006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06006b4d0 0x7fd06006d990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd078001780 tx=0x7fd078009040 comp rx=0 tx=0).stop 2024-03-06T07:30:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 -- 172.21.15.59:0/107128589 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07c158880 msgr2=0x7fd07c0b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07c158880 0x7fd07c0b3520 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fd070009d90 tx=0x7fd070004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 -- 172.21.15.59:0/107128589 shutdown_connections 2024-03-06T07:30:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06006b4d0 0x7fd06006d990 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd07c15b380 0x7fd07c1e77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd07c158880 0x7fd07c0b3520 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 --2- 172.21.15.59:0/107128589 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd07c070e10 0x7fd07c14a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 -- 172.21.15.59:0/107128589 >> 172.21.15.59:0/107128589 conn(0x7fd07c0b1c80 msgr2=0x7fd07c06c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 -- 172.21.15.59:0/107128589 shutdown_connections 2024-03-06T07:30:02.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.029+0000 7fd0833e2640 1 -- 172.21.15.59:0/107128589 wait complete. 2024-03-06T07:30:02.036 INFO:tasks.ceph.mds.c.smithi059.stderr: -8630> 2024-03-06T07:28:48.585+0000 7f2a20acc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:30:02.037 INFO:tasks.ceph.mds.c.smithi059.stderr: -7913> 2024-03-06T07:28:58.586+0000 7f2a20acc640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:30:02.043 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:30:02.071 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:30:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.145+0000 7f100eaed640 1 Processor -- start 2024-03-06T07:30:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.145+0000 7f100eaed640 1 -- start start 2024-03-06T07:30:02.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.146+0000 7f100eaed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10080d4860 0x7f10080d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.146+0000 7f100eaed640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10081618b0 con 0x7f10081614d0 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.146+0000 7f100eaed640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10080cf400 con 0x7f10080b0e30 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.146+0000 7f100eaed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10080d4cc0 con 0x7f10080d4860 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.146+0000 7f100c889640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f10081614d0 0x7f10080d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60382/0 (socket says 172.21.15.59:60382) 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.146+0000 7f100c889640 1 -- 172.21.15.59:0/3691161483 learned_addr learned my addr 172.21.15.59:0/3691161483 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.146+0000 7f1007fff640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10080d4860 0x7f10080d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.146+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856207608 0 0) 0x7f10081618b0 con 0x7f10081614d0 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.146+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff0003520 con 0x7f10081614d0 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.146+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507778750 0 0) 0x7f10080cf400 con 0x7f10080b0e30 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10081618b0 con 0x7f10080b0e30 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3094061411 0 0) 0x7f0ff0003520 con 0x7f10081614d0 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10080d4860 msgr2=0x7f10080d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10080d4860 0x7f10080d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.151 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 >> v1:172.21.15.164:6789/0 conn(0x7f10080b0e30 legacy=0x7f10080b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10080cf6a0 con 0x7f10081614d0 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f10000019f0 con 0x7f10081614d0 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 >> v1:172.21.15.59:6789/0 conn(0x7f10081614d0 legacy=0x7f10080d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10080d4860 0x7f0ff0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ff00040d0 0x7f0ff0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ff0004d40 0x7f0ff00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10080d4cc0 con 0x7f0ff0004d40 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff000baa0 con 0x7f0ff00040d0 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff000bca0 con 0x7f10080d4860 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f100d08a640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ff0004d40 0x7f0ff00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f100c889640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10080d4860 0x7f0ff0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f1007fff640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ff00040d0 0x7f0ff0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f100d08a640 1 -- 172.21.15.59:0/3691161483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10080d4860 msgr2=0x7f0ff0003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f100d08a640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10080d4860 0x7f0ff0003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f100d08a640 1 -- 172.21.15.59:0/3691161483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ff00040d0 msgr2=0x7f0ff0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f100d08a640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ff00040d0 0x7f0ff0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.147+0000 7f100d08a640 1 -- 172.21.15.59:0/3691161483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff0003520 con 0x7f0ff0004d40 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.148+0000 7f100d08a640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ff0004d40 0x7f0ff00051c0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f0ff80018d0 tx=0x7f0ff0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476b63d41921a723 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.148+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8037070 con 0x7f0ff0004d40 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.148+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff80052d0 con 0x7f0ff0004d40 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.148+0000 7f10077fe640 1 -- 172.21.15.59:0/3691161483 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff80014f0 con 0x7f0ff0004d40 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.149+0000 7f100eaed640 1 -- 172.21.15.59:0/3691161483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ff0004d40 msgr2=0x7f0ff00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.149+0000 7f100eaed640 1 --2- 172.21.15.59:0/3691161483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ff0004d40 0x7f0ff00051c0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f0ff80018d0 tx=0x7f0ff0005830 comp rx=0 tx=0).stop 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.149+0000 7f100d08a640 1 -- 172.21.15.59:0/3691161483 reap_dead start 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.149+0000 7f100eaed640 1 -- 172.21.15.59:0/3691161483 shutdown_connections 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.149+0000 7f100eaed640 1 -- 172.21.15.59:0/3691161483 >> 172.21.15.59:0/3691161483 conn(0x7f10080d3d40 msgr2=0x7f10080d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.149+0000 7f100eaed640 1 -- 172.21.15.59:0/3691161483 shutdown_connections 2024-03-06T07:30:02.153 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.149+0000 7f100eaed640 1 -- 172.21.15.59:0/3691161483 wait complete. 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.150+0000 7f100eaed640 1 Processor -- start 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.150+0000 7f100eaed640 1 -- start start 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100eaed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1008153560 0x7f10080ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100eaed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1008153940 0x7f10080c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100eaed640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080c9060 0x7f10080cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100eaed640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10080b0910 con 0x7f10080c9060 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100eaed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10080cfc60 con 0x7f1008153940 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100eaed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10080d2020 con 0x7f1008153560 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100d08a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080c9060 0x7f10080cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100d08a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080c9060 0x7f10080cbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51540/0 (socket says 172.21.15.59:51540) 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100d08a640 1 -- 172.21.15.59:0/3469989610 learned_addr learned my addr 172.21.15.59:0/3469989610 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100c889640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1008153560 0x7f10080ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f1007fff640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1008153940 0x7f10080c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100d08a640 1 -- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1008153560 msgr2=0x7f10080ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100d08a640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1008153560 0x7f10080ca090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100d08a640 1 -- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1008153940 msgr2=0x7f10080c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100d08a640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1008153940 0x7f10080c8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100d08a640 1 -- 172.21.15.59:0/3469989610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100815ba00 con 0x7f10080c9060 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100d08a640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080c9060 0x7f10080cbf30 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8005040 tx=0x7f0ff80012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f10057fa640 1 -- 172.21.15.59:0/3469989610 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8037070 con 0x7f10080c9060 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f10057fa640 1 -- 172.21.15.59:0/3469989610 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff8038040 con 0x7f10080c9060 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.151+0000 7f100eaed640 1 -- 172.21.15.59:0/3469989610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100815bce0 con 0x7f10080c9060 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.152+0000 7f10057fa640 1 -- 172.21.15.59:0/3469989610 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff80014f0 con 0x7f10080c9060 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.152+0000 7f100eaed640 1 -- 172.21.15.59:0/3469989610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10080c8600 con 0x7f10080c9060 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.152+0000 7f100eaed640 1 -- 172.21.15.59:0/3469989610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10080c87d0 con 0x7f10080c9060 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.153+0000 7f10057fa640 1 -- 172.21.15.59:0/3469989610 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ff8047060 con 0x7f10080c9060 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.153+0000 7f10057fa640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ff406b500 0x7f0ff406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.153+0000 7f10057fa640 1 -- 172.21.15.59:0/3469989610 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0ff80aab90 con 0x7f10080c9060 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.153+0000 7f100c889640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ff406b500 0x7f0ff406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.154+0000 7f100c889640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ff406b500 0x7f0ff406d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1000001fe0 tx=0x7f1000006040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.159 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.157+0000 7f10057fa640 1 -- 172.21.15.59:0/3469989610 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ff8079480 con 0x7f10080c9060 2024-03-06T07:30:02.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:02.377+0000 7f100eaed640 1 -- 172.21.15.59:0/3469989610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1008068ff0 con 0x7f10080c9060 2024-03-06T07:30:03.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.027+0000 7f10057fa640 1 -- 172.21.15.59:0/3469989610 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v92) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0ff807d520 con 0x7f10080c9060 2024-03-06T07:30:03.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 -- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ff406b500 msgr2=0x7f0ff406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ff406b500 0x7f0ff406d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1000001fe0 tx=0x7f1000006040 comp rx=0 tx=0).stop 2024-03-06T07:30:03.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 -- 172.21.15.59:0/3469989610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080c9060 msgr2=0x7f10080cbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080c9060 0x7f10080cbf30 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8005040 tx=0x7f0ff80012d0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 -- 172.21.15.59:0/3469989610 shutdown_connections 2024-03-06T07:30:03.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ff406b500 0x7f0ff406d9c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10080c9060 0x7f10080cbf30 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1008153940 0x7f10080c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 --2- 172.21.15.59:0/3469989610 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1008153560 0x7f10080ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 -- 172.21.15.59:0/3469989610 >> 172.21.15.59:0/3469989610 conn(0x7f10080d3d40 msgr2=0x7f10080b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 -- 172.21.15.59:0/3469989610 shutdown_connections 2024-03-06T07:30:03.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.030+0000 7f100eaed640 1 -- 172.21.15.59:0/3469989610 wait complete. 2024-03-06T07:30:03.046 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.196+0000 7f3e820ed640 1 Processor -- start 2024-03-06T07:30:03.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.196+0000 7f3e820ed640 1 -- start start 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e820ed640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3e7c0d4630 0x7f3e7c0d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e820ed640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7c15e380 con 0x7f3e7c0d2820 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e820ed640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7c0cf470 con 0x7f3e7c15dfa0 2024-03-06T07:30:03.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e820ed640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e7c0d2c70 con 0x7f3e7c0d4630 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e7affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3e7c0d4630 0x7f3e7c0d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e7affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3e7c0d4630 0x7f3e7c0d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51552/0 (socket says 172.21.15.59:51552) 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e7bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3e7c0d2820 0x7f3e7c0cb190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60396/0 (socket says 172.21.15.59:60396) 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e7affd640 1 -- 172.21.15.59:0/3962739082 learned_addr learned my addr 172.21.15.59:0/3962739082 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e7affd640 1 -- 172.21.15.59:0/3962739082 >> v1:172.21.15.164:6789/0 conn(0x7f3e7c15dfa0 legacy=0x7f3e7c0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e7affd640 1 -- 172.21.15.59:0/3962739082 >> v1:172.21.15.59:6789/0 conn(0x7f3e7c0d2820 legacy=0x7f3e7c0cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e7affd640 1 -- 172.21.15.59:0/3962739082 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7c0cf750 con 0x7f3e7c0d4630 2024-03-06T07:30:03.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.197+0000 7f3e7a7fc640 1 -- 172.21.15.59:0/3962739082 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368391659 0 0) 0x7f3e7c15e380 con 0x7f3e7c0d2820 2024-03-06T07:30:03.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.198+0000 7f3e7affd640 1 --2- 172.21.15.59:0/3962739082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7c0d4630 0x7f3e7c0d22e0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009870 tx=0x7f3e7002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7859a9a2abbb35d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.198+0000 7f3e7a7fc640 1 -- 172.21.15.59:0/3962739082 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e70034070 con 0x7f3e7c0d4630 2024-03-06T07:30:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.198+0000 7f3e7a7fc640 1 -- 172.21.15.59:0/3962739082 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e70030860 con 0x7f3e7c0d4630 2024-03-06T07:30:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.198+0000 7f3e7a7fc640 1 -- 172.21.15.59:0/3962739082 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e70030c30 con 0x7f3e7c0d4630 2024-03-06T07:30:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.199+0000 7f3e820ed640 1 -- 172.21.15.59:0/3962739082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7c0d4630 msgr2=0x7f3e7c0d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.199+0000 7f3e820ed640 1 --2- 172.21.15.59:0/3962739082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7c0d4630 0x7f3e7c0d22e0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009870 tx=0x7f3e7002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.199+0000 7f3e820ed640 1 -- 172.21.15.59:0/3962739082 shutdown_connections 2024-03-06T07:30:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.199+0000 7f3e820ed640 1 --2- 172.21.15.59:0/3962739082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7c0d4630 0x7f3e7c0d22e0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.199+0000 7f3e820ed640 1 -- 172.21.15.59:0/3962739082 >> 172.21.15.59:0/3962739082 conn(0x7f3e7c09d000 msgr2=0x7f3e7c0d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.199+0000 7f3e820ed640 1 -- 172.21.15.59:0/3962739082 shutdown_connections 2024-03-06T07:30:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.199+0000 7f3e820ed640 1 -- 172.21.15.59:0/3962739082 wait complete. 2024-03-06T07:30:03.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.200+0000 7f3e820ed640 1 Processor -- start 2024-03-06T07:30:03.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.201+0000 7f3e820ed640 1 -- start start 2024-03-06T07:30:03.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e820ed640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7c0d2820 0x7f3e7c15b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e820ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3e7c0d4630 0x7f3e7c0b8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e820ed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3e7c15dfa0 0x7f3e7c0b93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e820ed640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c0cb8a0 con 0x7f3e7c0d2820 2024-03-06T07:30:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e820ed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c0d2100 con 0x7f3e7c15dfa0 2024-03-06T07:30:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e820ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7c0cba20 con 0x7f3e7c0d4630 2024-03-06T07:30:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7c0d2820 0x7f3e7c15b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7c0d2820 0x7f3e7c15b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51558/0 (socket says 172.21.15.59:51558) 2024-03-06T07:30:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7b7fe640 1 -- 172.21.15.59:0/3784287808 learned_addr learned my addr 172.21.15.59:0/3784287808 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7affd640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3e7c0d4630 0x7f3e7c0b8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7bfff640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3e7c15dfa0 0x7f3e7c0b93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7b7fe640 1 -- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3e7c0d4630 msgr2=0x7f3e7c0b8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7b7fe640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3e7c0d4630 0x7f3e7c0b8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7b7fe640 1 -- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3e7c15dfa0 msgr2=0x7f3e7c0b93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7b7fe640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3e7c15dfa0 0x7f3e7c0b93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7b7fe640 1 -- 172.21.15.59:0/3784287808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e7c15e630 con 0x7f3e7c0d2820 2024-03-06T07:30:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7affd640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3e7c0d4630 0x7f3e7c0b8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e7b7fe640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7c0d2820 0x7f3e7c15b530 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00c400 tx=0x7f3e6c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e63fff640 1 -- 172.21.15.59:0/3784287808 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c017070 con 0x7f3e7c0d2820 2024-03-06T07:30:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e63fff640 1 -- 172.21.15.59:0/3784287808 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6c0052d0 con 0x7f3e7c0d2820 2024-03-06T07:30:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.202+0000 7f3e820ed640 1 -- 172.21.15.59:0/3784287808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7c15e910 con 0x7f3e7c0d2820 2024-03-06T07:30:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.203+0000 7f3e63fff640 1 -- 172.21.15.59:0/3784287808 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c003830 con 0x7f3e7c0d2820 2024-03-06T07:30:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.203+0000 7f3e820ed640 1 -- 172.21.15.59:0/3784287808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7c15eed0 con 0x7f3e7c0d2820 2024-03-06T07:30:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.203+0000 7f3e820ed640 1 -- 172.21.15.59:0/3784287808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e7c04f330 con 0x7f3e7c0d2820 2024-03-06T07:30:03.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.204+0000 7f3e63fff640 1 -- 172.21.15.59:0/3784287808 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e6c01a070 con 0x7f3e7c0d2820 2024-03-06T07:30:03.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.204+0000 7f3e63fff640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3e6806b4d0 0x7f3e6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.204+0000 7f3e63fff640 1 -- 172.21.15.59:0/3784287808 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3e6c089610 con 0x7f3e7c0d2820 2024-03-06T07:30:03.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.205+0000 7f3e7affd640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3e6806b4d0 0x7f3e6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.205+0000 7f3e7affd640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3e6806b4d0 0x7f3e6806d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3e70036000 tx=0x7f3e70008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.209+0000 7f3e63fff640 1 -- 172.21.15.59:0/3784287808 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e6c058400 con 0x7f3e7c0d2820 2024-03-06T07:30:03.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:03.389+0000 7f3e820ed640 1 -- 172.21.15.59:0/3784287808 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3e7c06ba00 con 0x7f3e7c0d2820 2024-03-06T07:30:04.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.032+0000 7f3e63fff640 1 -- 172.21.15.59:0/3784287808 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v132) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3e6c057d00 con 0x7f3e7c0d2820 2024-03-06T07:30:04.034 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:30:04.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.037+0000 7f3e820ed640 1 -- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3e6806b4d0 msgr2=0x7f3e6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.037+0000 7f3e820ed640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3e6806b4d0 0x7f3e6806d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3e70036000 tx=0x7f3e70008030 comp rx=0 tx=0).stop 2024-03-06T07:30:04.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.037+0000 7f3e820ed640 1 -- 172.21.15.59:0/3784287808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7c0d2820 msgr2=0x7f3e7c15b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.037+0000 7f3e820ed640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7c0d2820 0x7f3e7c15b530 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00c400 tx=0x7f3e6c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.037+0000 7f3e820ed640 1 -- 172.21.15.59:0/3784287808 shutdown_connections 2024-03-06T07:30:04.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.037+0000 7f3e820ed640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3e6806b4d0 0x7f3e6806d990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.038+0000 7f3e820ed640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3e7c15dfa0 0x7f3e7c0b93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.038+0000 7f3e820ed640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3e7c0d4630 0x7f3e7c0b8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.038+0000 7f3e820ed640 1 --2- 172.21.15.59:0/3784287808 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e7c0d2820 0x7f3e7c15b530 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.038+0000 7f3e820ed640 1 -- 172.21.15.59:0/3784287808 >> 172.21.15.59:0/3784287808 conn(0x7f3e7c09d000 msgr2=0x7f3e7c0cbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.038+0000 7f3e820ed640 1 -- 172.21.15.59:0/3784287808 shutdown_connections 2024-03-06T07:30:04.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.038+0000 7f3e820ed640 1 -- 172.21.15.59:0/3784287808 wait complete. 2024-03-06T07:30:04.055 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:30:04.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.168+0000 7f794d5a0640 1 Processor -- start 2024-03-06T07:30:04.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.168+0000 7f794d5a0640 1 -- start start 2024-03-06T07:30:04.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.169+0000 7f794d5a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480ceb60 0x7f794815efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.169+0000 7f794d5a0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79480ceff0 con 0x7f79480d0db0 2024-03-06T07:30:04.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.169+0000 7f794d5a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79480cf280 con 0x7f794815e0a0 2024-03-06T07:30:04.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.169+0000 7f794d5a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794815f4e0 con 0x7f79480ceb60 2024-03-06T07:30:04.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.169+0000 7f79477fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f79480d0db0 0x7f79480d1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55214/0 (socket says 172.21.15.59:55214) 2024-03-06T07:30:04.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.169+0000 7f79477fe640 1 -- 172.21.15.59:0/283397221 learned_addr learned my addr 172.21.15.59:0/283397221 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:04.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.169+0000 7f794cd9f640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480ceb60 0x7f794815efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.169+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334284706 0 0) 0x7f79480ceff0 con 0x7f79480d0db0 2024-03-06T07:30:04.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.169+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7930003520 con 0x7f79480d0db0 2024-03-06T07:30:04.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191125359 0 0) 0x7f79480cf280 con 0x7f794815e0a0 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79480ceff0 con 0x7f794815e0a0 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2851520533 0 0) 0x7f7930003520 con 0x7f79480d0db0 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480ceb60 msgr2=0x7f794815efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480ceb60 0x7f794815efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 >> v1:172.21.15.164:6789/0 conn(0x7f794815e0a0 legacy=0x7f79480d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79480cf560 con 0x7f79480d0db0 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f793c001670 con 0x7f79480d0db0 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 >> v1:172.21.15.59:6789/0 conn(0x7f79480d0db0 legacy=0x7f79480d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7930003b70 0x7f7930003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7930004800 0x7f7930004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480ceb60 0x7f79300051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794815f4e0 con 0x7f7930004800 2024-03-06T07:30:04.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793000bb00 con 0x7f7930003b70 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f793000bd00 con 0x7f79480ceb60 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f79477fe640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7930004800 0x7f7930004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f7947fff640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7930003b70 0x7f7930003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f794cd9f640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480ceb60 0x7f79300051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f79477fe640 1 -- 172.21.15.59:0/283397221 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480ceb60 msgr2=0x7f79300051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f79477fe640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480ceb60 0x7f79300051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f79477fe640 1 -- 172.21.15.59:0/283397221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7930003b70 msgr2=0x7f7930003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f79477fe640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7930003b70 0x7f7930003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.170+0000 7f79477fe640 1 -- 172.21.15.59:0/283397221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7930003520 con 0x7f7930004800 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.171+0000 7f79477fe640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7930004800 0x7f7930004c80 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f793c007fd0 tx=0x7f7930005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9afd3740646bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.171+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c007ca0 con 0x7f7930004800 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.171+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793c030070 con 0x7f7930004800 2024-03-06T07:30:04.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.171+0000 7f7946ffd640 1 -- 172.21.15.59:0/283397221 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793c005d30 con 0x7f7930004800 2024-03-06T07:30:04.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.172+0000 7f794d5a0640 1 -- 172.21.15.59:0/283397221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7930004800 msgr2=0x7f7930004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.172+0000 7f794d5a0640 1 --2- 172.21.15.59:0/283397221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7930004800 0x7f7930004c80 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f793c007fd0 tx=0x7f7930005830 comp rx=0 tx=0).stop 2024-03-06T07:30:04.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.172+0000 7f794cd9f640 1 -- 172.21.15.59:0/283397221 reap_dead start 2024-03-06T07:30:04.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.172+0000 7f794d5a0640 1 -- 172.21.15.59:0/283397221 shutdown_connections 2024-03-06T07:30:04.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.172+0000 7f794d5a0640 1 -- 172.21.15.59:0/283397221 >> 172.21.15.59:0/283397221 conn(0x7f79480d3c90 msgr2=0x7f79480d40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.172+0000 7f794d5a0640 1 -- 172.21.15.59:0/283397221 shutdown_connections 2024-03-06T07:30:04.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.172+0000 7f794d5a0640 1 -- 172.21.15.59:0/283397221 wait complete. 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.173+0000 7f794d5a0640 1 Processor -- start 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.173+0000 7f794d5a0640 1 -- start start 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794d5a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7930004800 0x7f794814c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794d5a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480b8d50 0x7f79480cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794d5a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79480cc2f0 0x7f79480cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794d5a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794815efa0 con 0x7f79480cc2f0 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794d5a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79480cfb20 con 0x7f7930004800 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794d5a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794815f120 con 0x7f79480b8d50 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794cd9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79480cc2f0 0x7f79480cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794cd9f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79480cc2f0 0x7f79480cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45258/0 (socket says 172.21.15.59:45258) 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794cd9f640 1 -- 172.21.15.59:0/1990349726 learned_addr learned my addr 172.21.15.59:0/1990349726 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:04.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f7947fff640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7930004800 0x7f794814c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f79477fe640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480b8d50 0x7f79480cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794cd9f640 1 -- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480b8d50 msgr2=0x7f79480cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794cd9f640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480b8d50 0x7f79480cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794cd9f640 1 -- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7930004800 msgr2=0x7f794814c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794cd9f640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7930004800 0x7f794814c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794cd9f640 1 -- 172.21.15.59:0/1990349726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7948161b40 con 0x7f79480cc2f0 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.174+0000 7f794cd9f640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79480cc2f0 0x7f79480cc770 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f7930004780 tx=0x7f794000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.175+0000 7f7944ff9640 1 -- 172.21.15.59:0/1990349726 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7940011e60 con 0x7f79480cc2f0 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.175+0000 7f7944ff9640 1 -- 172.21.15.59:0/1990349726 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794001d070 con 0x7f79480cc2f0 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.175+0000 7f794d5a0640 1 -- 172.21.15.59:0/1990349726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7948161e20 con 0x7f79480cc2f0 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.175+0000 7f7944ff9640 1 -- 172.21.15.59:0/1990349726 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7940015670 con 0x7f79480cc2f0 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.175+0000 7f794d5a0640 1 -- 172.21.15.59:0/1990349726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794814c540 con 0x7f79480cc2f0 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.176+0000 7f794d5a0640 1 -- 172.21.15.59:0/1990349726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7948068f10 con 0x7f79480cc2f0 2024-03-06T07:30:04.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.176+0000 7f7944ff9640 1 -- 172.21.15.59:0/1990349726 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7940015810 con 0x7f79480cc2f0 2024-03-06T07:30:04.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.177+0000 7f7944ff9640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f793406b4d0 0x7f793406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.177+0000 7f7944ff9640 1 -- 172.21.15.59:0/1990349726 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f794008a8c0 con 0x7f79480cc2f0 2024-03-06T07:30:04.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.177+0000 7f7947fff640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f793406b4d0 0x7f793406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.177+0000 7f7947fff640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f793406b4d0 0x7f793406d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7938001850 tx=0x7f7938009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.181+0000 7f7944ff9640 1 -- 172.21.15.59:0/1990349726 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7940010040 con 0x7f79480cc2f0 2024-03-06T07:30:04.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:04.363+0000 7f794d5a0640 1 -- 172.21.15.59:0/1990349726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f79480ce0c0 con 0x7f79480cc2f0 2024-03-06T07:30:05.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.039+0000 7f7944ff9640 1 -- 172.21.15.59:0/1990349726 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v133) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7940059860 con 0x7f79480cc2f0 2024-03-06T07:30:05.041 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-03-06T07:30:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.041+0000 7f794d5a0640 1 -- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f793406b4d0 msgr2=0x7f793406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.041+0000 7f794d5a0640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f793406b4d0 0x7f793406d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7938001850 tx=0x7f7938009040 comp rx=0 tx=0).stop 2024-03-06T07:30:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.041+0000 7f794d5a0640 1 -- 172.21.15.59:0/1990349726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79480cc2f0 msgr2=0x7f79480cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.041+0000 7f794d5a0640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79480cc2f0 0x7f79480cc770 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f7930004780 tx=0x7f794000f580 comp rx=0 tx=0).stop 2024-03-06T07:30:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.041+0000 7f794d5a0640 1 -- 172.21.15.59:0/1990349726 shutdown_connections 2024-03-06T07:30:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.041+0000 7f794d5a0640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f793406b4d0 0x7f793406d990 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.041+0000 7f794d5a0640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79480cc2f0 0x7f79480cc770 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.041+0000 7f794d5a0640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79480b8d50 0x7f79480cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.041+0000 7f794d5a0640 1 --2- 172.21.15.59:0/1990349726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7930004800 0x7f794814c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.041+0000 7f794d5a0640 1 -- 172.21.15.59:0/1990349726 >> 172.21.15.59:0/1990349726 conn(0x7f79480d3c90 msgr2=0x7f794815e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.042+0000 7f794d5a0640 1 -- 172.21.15.59:0/1990349726 shutdown_connections 2024-03-06T07:30:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.042+0000 7f794d5a0640 1 -- 172.21.15.59:0/1990349726 wait complete. 2024-03-06T07:30:05.060 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:30:05.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.168+0000 7fd538b6c640 1 Processor -- start 2024-03-06T07:30:05.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.168+0000 7fd538b6c640 1 -- start start 2024-03-06T07:30:05.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd538b6c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd53415bdd0 0x7fd5340b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.171 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd538b6c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd5340b15c0 0x7fd5340b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd538b6c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5340d0240 con 0x7fd5340ce300 2024-03-06T07:30:05.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd538b6c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5340cf4f0 con 0x7fd5340b15c0 2024-03-06T07:30:05.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd538b6c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5340d0850 con 0x7fd53415bdd0 2024-03-06T07:30:05.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd532d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd5340b15c0 0x7fd5340b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd531d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd5340ce300 0x7fd5340ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55230/0 (socket says 172.21.15.59:55230) 2024-03-06T07:30:05.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd531d9b640 1 -- 172.21.15.59:0/497281578 learned_addr learned my addr 172.21.15.59:0/497281578 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:05.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd532d9d640 1 -- 172.21.15.59:0/497281578 >> v2:172.21.15.164:3300/0 conn(0x7fd53415bdd0 msgr2=0x7fd5340b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd532d9d640 1 --2- 172.21.15.59:0/497281578 >> v2:172.21.15.164:3300/0 conn(0x7fd53415bdd0 0x7fd5340b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd532d9d640 1 -- 172.21.15.59:0/497281578 >> v1:172.21.15.59:6789/0 conn(0x7fd5340ce300 legacy=0x7fd5340ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd53159a640 1 -- 172.21.15.59:0/497281578 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820831980 0 0) 0x7fd5340d0240 con 0x7fd5340ce300 2024-03-06T07:30:05.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.169+0000 7fd532d9d640 1 -- 172.21.15.59:0/497281578 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5340cf210 con 0x7fd5340b15c0 2024-03-06T07:30:05.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.170+0000 7fd532d9d640 1 --2- 172.21.15.59:0/497281578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5340b15c0 0x7fd5340b19d0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fd534071cc0 tx=0x7fd52402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8b68f73504c91e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.170+0000 7fd53159a640 1 -- 172.21.15.59:0/497281578 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd524039070 con 0x7fd5340b15c0 2024-03-06T07:30:05.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.170+0000 7fd53159a640 1 -- 172.21.15.59:0/497281578 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52402cc80 con 0x7fd5340b15c0 2024-03-06T07:30:05.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.170+0000 7fd53159a640 1 -- 172.21.15.59:0/497281578 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd524034660 con 0x7fd5340b15c0 2024-03-06T07:30:05.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.171+0000 7fd538b6c640 1 -- 172.21.15.59:0/497281578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5340b15c0 msgr2=0x7fd5340b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.171+0000 7fd538b6c640 1 --2- 172.21.15.59:0/497281578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5340b15c0 0x7fd5340b19d0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fd534071cc0 tx=0x7fd52402bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.171+0000 7fd538b6c640 1 -- 172.21.15.59:0/497281578 shutdown_connections 2024-03-06T07:30:05.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.171+0000 7fd538b6c640 1 --2- 172.21.15.59:0/497281578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5340b15c0 0x7fd5340b19d0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.171+0000 7fd538b6c640 1 --2- 172.21.15.59:0/497281578 >> v2:172.21.15.164:3300/0 conn(0x7fd53415bdd0 0x7fd5340b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.171+0000 7fd538b6c640 1 -- 172.21.15.59:0/497281578 >> 172.21.15.59:0/497281578 conn(0x7fd5340b3690 msgr2=0x7fd5340b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.171+0000 7fd538b6c640 1 -- 172.21.15.59:0/497281578 shutdown_connections 2024-03-06T07:30:05.174 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.171+0000 7fd538b6c640 1 -- 172.21.15.59:0/497281578 wait complete. 2024-03-06T07:30:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.172+0000 7fd538b6c640 1 Processor -- start 2024-03-06T07:30:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.172+0000 7fd538b6c640 1 -- start start 2024-03-06T07:30:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd538b6c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5340b15c0 0x7fd53415c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd538b6c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5340ce300 0x7fd53415c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd538b6c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd53415bdd0 0x7fd53415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd538b6c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53415c1b0 con 0x7fd5340b15c0 2024-03-06T07:30:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd538b6c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5340ceea0 con 0x7fd53415bdd0 2024-03-06T07:30:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd538b6c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5340b10c0 con 0x7fd5340ce300 2024-03-06T07:30:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd53259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5340b15c0 0x7fd53415c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd53259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5340b15c0 0x7fd53415c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45268/0 (socket says 172.21.15.59:45268) 2024-03-06T07:30:05.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd53259c640 1 -- 172.21.15.59:0/832944561 learned_addr learned my addr 172.21.15.59:0/832944561 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd532d9d640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd53415bdd0 0x7fd53415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd531d9b640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5340ce300 0x7fd53415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd53259c640 1 -- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5340ce300 msgr2=0x7fd53415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd53259c640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5340ce300 0x7fd53415c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd53259c640 1 -- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd53415bdd0 msgr2=0x7fd53415e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd53259c640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd53415bdd0 0x7fd53415e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd53259c640 1 -- 172.21.15.59:0/832944561 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5340c5d10 con 0x7fd5340b15c0 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd532d9d640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd53415bdd0 0x7fd53415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.173+0000 7fd531d9b640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5340ce300 0x7fd53415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.174+0000 7fd53259c640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5340b15c0 0x7fd53415c330 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00b4a0 tx=0x7fd52c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.174+0000 7fd522ffd640 1 -- 172.21.15.59:0/832944561 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c014070 con 0x7fd5340b15c0 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.174+0000 7fd522ffd640 1 -- 172.21.15.59:0/832944561 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c010610 con 0x7fd5340b15c0 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.174+0000 7fd538b6c640 1 -- 172.21.15.59:0/832944561 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5340c5ff0 con 0x7fd5340b15c0 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.174+0000 7fd522ffd640 1 -- 172.21.15.59:0/832944561 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c010a00 con 0x7fd5340b15c0 2024-03-06T07:30:05.178 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.174+0000 7fd538b6c640 1 -- 172.21.15.59:0/832944561 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53415e820 con 0x7fd5340b15c0 2024-03-06T07:30:05.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.175+0000 7fd538b6c640 1 -- 172.21.15.59:0/832944561 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53409fcf0 con 0x7fd5340b15c0 2024-03-06T07:30:05.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.175+0000 7fd522ffd640 1 -- 172.21.15.59:0/832944561 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd52c017070 con 0x7fd5340b15c0 2024-03-06T07:30:05.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.176+0000 7fd522ffd640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd51406b500 0x7fd51406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.176+0000 7fd522ffd640 1 -- 172.21.15.59:0/832944561 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd52c0899c0 con 0x7fd5340b15c0 2024-03-06T07:30:05.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.176+0000 7fd531d9b640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd51406b500 0x7fd51406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.176+0000 7fd531d9b640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd51406b500 0x7fd51406d9c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd534155270 tx=0x7fd528009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.180+0000 7fd522ffd640 1 -- 172.21.15.59:0/832944561 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd52c058aa0 con 0x7fd5340b15c0 2024-03-06T07:30:05.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.368+0000 7fd538b6c640 1 -- 172.21.15.59:0/832944561 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd534068ff0 con 0x7fd5340b15c0 2024-03-06T07:30:05.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.371+0000 7fd522ffd640 1 -- 172.21.15.59:0/832944561 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd52c0583a0 con 0x7fd5340b15c0 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.372+0000 7fd538b6c640 1 -- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd51406b500 msgr2=0x7fd51406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.372+0000 7fd538b6c640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd51406b500 0x7fd51406d9c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd534155270 tx=0x7fd528009040 comp rx=0 tx=0).stop 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.372+0000 7fd538b6c640 1 -- 172.21.15.59:0/832944561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5340b15c0 msgr2=0x7fd53415c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.372+0000 7fd538b6c640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5340b15c0 0x7fd53415c330 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00b4a0 tx=0x7fd52c00b960 comp rx=0 tx=0).stop 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.373+0000 7fd538b6c640 1 -- 172.21.15.59:0/832944561 shutdown_connections 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.373+0000 7fd538b6c640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd51406b500 0x7fd51406d9c0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.373+0000 7fd538b6c640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd53415bdd0 0x7fd53415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.373+0000 7fd538b6c640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5340ce300 0x7fd53415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.373+0000 7fd538b6c640 1 --2- 172.21.15.59:0/832944561 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5340b15c0 0x7fd53415c330 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.373+0000 7fd538b6c640 1 -- 172.21.15.59:0/832944561 >> 172.21.15.59:0/832944561 conn(0x7fd5340b3690 msgr2=0x7fd5340cce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.373+0000 7fd538b6c640 1 -- 172.21.15.59:0/832944561 shutdown_connections 2024-03-06T07:30:05.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.373+0000 7fd538b6c640 1 -- 172.21.15.59:0/832944561 wait complete. 2024-03-06T07:30:05.386 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:30:05.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.485+0000 7fb3bca88640 1 Processor -- start 2024-03-06T07:30:05.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.485+0000 7fb3bca88640 1 -- start start 2024-03-06T07:30:05.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3bca88640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb3b80b4f70 0x7fb3b80b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3bca88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b8066d90 0x7fb3b80ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3bca88640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b80b5890 con 0x7fb3b806c6d0 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3bca88640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b815e9f0 con 0x7fb3b80b4f70 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3bca88640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b80cda10 con 0x7fb3b8066d90 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3b659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb3b80b4f70 0x7fb3b80b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3b659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb3b80b4f70 0x7fb3b80b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45282/0 (socket says 172.21.15.59:45282) 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3b659c640 1 -- 172.21.15.59:0/2748950894 learned_addr learned my addr 172.21.15.59:0/2748950894 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3b6d9d640 1 --2- 172.21.15.59:0/2748950894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b8066d90 0x7fb3b80ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3b659c640 1 -- 172.21.15.59:0/2748950894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b8066d90 msgr2=0x7fb3b80ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3b659c640 1 --2- 172.21.15.59:0/2748950894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b8066d90 0x7fb3b80ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.486+0000 7fb3b659c640 1 -- 172.21.15.59:0/2748950894 >> v1:172.21.15.164:6789/0 conn(0x7fb3b806c6d0 legacy=0x7fb3b815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.487+0000 7fb3b659c640 1 -- 172.21.15.59:0/2748950894 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b80cd450 con 0x7fb3b80b4f70 2024-03-06T07:30:05.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.487+0000 7fb3b659c640 1 --2- 172.21.15.59:0/2748950894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b80b4f70 0x7fb3b80b5350 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009870 tx=0x7fb3b002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a834d543a869e9bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.487+0000 7fb3b559a640 1 -- 172.21.15.59:0/2748950894 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0035020 con 0x7fb3b80b4f70 2024-03-06T07:30:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.487+0000 7fb3b559a640 1 -- 172.21.15.59:0/2748950894 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b0031640 con 0x7fb3b80b4f70 2024-03-06T07:30:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.487+0000 7fb3b559a640 1 -- 172.21.15.59:0/2748950894 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b002d040 con 0x7fb3b80b4f70 2024-03-06T07:30:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.488+0000 7fb3bca88640 1 -- 172.21.15.59:0/2748950894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b80b4f70 msgr2=0x7fb3b80b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.488+0000 7fb3bca88640 1 --2- 172.21.15.59:0/2748950894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b80b4f70 0x7fb3b80b5350 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0009870 tx=0x7fb3b002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.488+0000 7fb3bca88640 1 -- 172.21.15.59:0/2748950894 shutdown_connections 2024-03-06T07:30:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.488+0000 7fb3bca88640 1 --2- 172.21.15.59:0/2748950894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b8066d90 0x7fb3b80ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.488+0000 7fb3bca88640 1 --2- 172.21.15.59:0/2748950894 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b80b4f70 0x7fb3b80b5350 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.488+0000 7fb3bca88640 1 -- 172.21.15.59:0/2748950894 >> 172.21.15.59:0/2748950894 conn(0x7fb3b80724d0 msgr2=0x7fb3b80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.488+0000 7fb3bca88640 1 -- 172.21.15.59:0/2748950894 shutdown_connections 2024-03-06T07:30:05.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.488+0000 7fb3bca88640 1 -- 172.21.15.59:0/2748950894 wait complete. 2024-03-06T07:30:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.489+0000 7fb3bca88640 1 Processor -- start 2024-03-06T07:30:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.489+0000 7fb3bca88640 1 -- start start 2024-03-06T07:30:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3bca88640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b8066d90 0x7fb3b815a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3bca88640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b80b4f70 0x7fb3b815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3bca88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b81453e0 0x7fb3b8145860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3bca88640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b80cd090 con 0x7fb3b80b4f70 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3bca88640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b815e680 con 0x7fb3b8066d90 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3bca88640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b80cd210 con 0x7fb3b81453e0 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3b5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b80b4f70 0x7fb3b815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3b5d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b80b4f70 0x7fb3b815aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45294/0 (socket says 172.21.15.59:45294) 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3b5d9b640 1 -- 172.21.15.59:0/2965073893 learned_addr learned my addr 172.21.15.59:0/2965073893 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3b659c640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b8066d90 0x7fb3b815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3b6d9d640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b81453e0 0x7fb3b8145860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3b5d9b640 1 -- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b81453e0 msgr2=0x7fb3b8145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3b5d9b640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b81453e0 0x7fb3b8145860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3b5d9b640 1 -- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b8066d90 msgr2=0x7fb3b815a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3b5d9b640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b8066d90 0x7fb3b815a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.490+0000 7fb3b5d9b640 1 -- 172.21.15.59:0/2965073893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b8144f70 con 0x7fb3b80b4f70 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.491+0000 7fb3b5d9b640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b80b4f70 0x7fb3b815aa60 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000c430 tx=0x7fb3a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.491+0000 7fb3aeffd640 1 -- 172.21.15.59:0/2965073893 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0011790 con 0x7fb3b80b4f70 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.491+0000 7fb3aeffd640 1 -- 172.21.15.59:0/2965073893 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a0017070 con 0x7fb3b80b4f70 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.491+0000 7fb3bca88640 1 -- 172.21.15.59:0/2965073893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b80ad2b0 con 0x7fb3b80b4f70 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.491+0000 7fb3aeffd640 1 -- 172.21.15.59:0/2965073893 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0011bb0 con 0x7fb3b80b4f70 2024-03-06T07:30:05.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.491+0000 7fb3bca88640 1 -- 172.21.15.59:0/2965073893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b80ad870 con 0x7fb3b80b4f70 2024-03-06T07:30:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.492+0000 7fb3bca88640 1 -- 172.21.15.59:0/2965073893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b80ada70 con 0x7fb3b80b4f70 2024-03-06T07:30:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.492+0000 7fb3aeffd640 1 -- 172.21.15.59:0/2965073893 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3a001c460 con 0x7fb3b80b4f70 2024-03-06T07:30:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.493+0000 7fb3aeffd640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb39806b4d0 0x7fb39806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.493+0000 7fb3aeffd640 1 -- 172.21.15.59:0/2965073893 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb3a0089bf0 con 0x7fb3b80b4f70 2024-03-06T07:30:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.493+0000 7fb3b659c640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb39806b4d0 0x7fb39806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.494+0000 7fb3b659c640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb39806b4d0 0x7fb39806d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0036000 tx=0x7fb3b003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.497+0000 7fb3aeffd640 1 -- 172.21.15.59:0/2965073893 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3b80ada70 con 0x7fb3b80b4f70 2024-03-06T07:30:05.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:05.748+0000 7fb3bca88640 1 -- 172.21.15.59:0/2965073893 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7fb3b8074880 con 0x7fb3b80b4f70 2024-03-06T07:30:06.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.040+0000 7fb3aeffd640 1 -- 172.21.15.59:0/2965073893 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v447) v1 ==== 140+0+0 (secure 0 0 0) 0x7fb3a00587b0 con 0x7fb3b80b4f70 2024-03-06T07:30:06.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 -- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb39806b4d0 msgr2=0x7fb39806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb39806b4d0 0x7fb39806d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0036000 tx=0x7fb3b003c040 comp rx=0 tx=0).stop 2024-03-06T07:30:06.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 -- 172.21.15.59:0/2965073893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b80b4f70 msgr2=0x7fb3b815aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b80b4f70 0x7fb3b815aa60 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000c430 tx=0x7fb3a000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 -- 172.21.15.59:0/2965073893 shutdown_connections 2024-03-06T07:30:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb39806b4d0 0x7fb39806d990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b81453e0 0x7fb3b8145860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b80b4f70 0x7fb3b815aa60 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 --2- 172.21.15.59:0/2965073893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b8066d90 0x7fb3b815a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 -- 172.21.15.59:0/2965073893 >> 172.21.15.59:0/2965073893 conn(0x7fb3b80724d0 msgr2=0x7fb3b806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 -- 172.21.15.59:0/2965073893 shutdown_connections 2024-03-06T07:30:06.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.042+0000 7fb3bca88640 1 -- 172.21.15.59:0/2965073893 wait complete. 2024-03-06T07:30:06.056 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 ... ok 2024-03-06T07:30:06.056 INFO:tasks.cephfs_test_runner:Starting test: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:30:06.056 INFO:tasks.cephfs_test_runner:test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:30:06.057 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:06.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.158+0000 7f3c1eb8a640 1 Processor -- start 2024-03-06T07:30:06.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.158+0000 7f3c1eb8a640 1 -- start start 2024-03-06T07:30:06.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c1eb8a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3c181597c0 0x7f3c1815cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c1eb8a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3c1806f290 0x7f3c1806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c1eb8a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c180cd440 con 0x7f3c1806b4f0 2024-03-06T07:30:06.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c1eb8a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1806b960 con 0x7f3c1806f290 2024-03-06T07:30:06.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c1eb8a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c18159ba0 con 0x7f3c181597c0 2024-03-06T07:30:06.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c17fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3c1806f290 0x7f3c1806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c17fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3c1806f290 0x7f3c1806afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45308/0 (socket says 172.21.15.59:45308) 2024-03-06T07:30:06.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c17fff640 1 -- 172.21.15.59:0/733414310 learned_addr learned my addr 172.21.15.59:0/733414310 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:06.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c17fff640 1 -- 172.21.15.59:0/733414310 >> v2:172.21.15.164:3300/0 conn(0x7f3c181597c0 msgr2=0x7f3c1815cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c17fff640 1 --2- 172.21.15.59:0/733414310 >> v2:172.21.15.164:3300/0 conn(0x7f3c181597c0 0x7f3c1815cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c17fff640 1 -- 172.21.15.59:0/733414310 >> v1:172.21.15.164:6789/0 conn(0x7f3c1806b4f0 legacy=0x7f3c180d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.159+0000 7f3c17fff640 1 -- 172.21.15.59:0/733414310 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c180cd720 con 0x7f3c1806f290 2024-03-06T07:30:06.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.160+0000 7f3c17fff640 1 --2- 172.21.15.59:0/733414310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c1806f290 0x7f3c1806afb0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f3c1815f6f0 tx=0x7f3c0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16db2db443beecc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.160+0000 7f3c177fe640 1 -- 172.21.15.59:0/733414310 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04035020 con 0x7f3c1806f290 2024-03-06T07:30:06.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.160+0000 7f3c177fe640 1 -- 172.21.15.59:0/733414310 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c04031870 con 0x7f3c1806f290 2024-03-06T07:30:06.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.160+0000 7f3c177fe640 1 -- 172.21.15.59:0/733414310 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04031c40 con 0x7f3c1806f290 2024-03-06T07:30:06.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.161+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/733414310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c1806f290 msgr2=0x7f3c1806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.161+0000 7f3c1eb8a640 1 --2- 172.21.15.59:0/733414310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c1806f290 0x7f3c1806afb0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f3c1815f6f0 tx=0x7f3c0402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:06.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.161+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/733414310 shutdown_connections 2024-03-06T07:30:06.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.161+0000 7f3c1eb8a640 1 --2- 172.21.15.59:0/733414310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c1806f290 0x7f3c1806afb0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.161+0000 7f3c1eb8a640 1 --2- 172.21.15.59:0/733414310 >> v2:172.21.15.164:3300/0 conn(0x7f3c181597c0 0x7f3c1815cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.161+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/733414310 >> 172.21.15.59:0/733414310 conn(0x7f3c1806c0c0 msgr2=0x7f3c1806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.161+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/733414310 shutdown_connections 2024-03-06T07:30:06.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.161+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/733414310 wait complete. 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.162+0000 7f3c1eb8a640 1 Processor -- start 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.162+0000 7f3c1eb8a640 1 -- start start 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1eb8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c1806b4f0 0x7f3c1814a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1eb8a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c1806f290 0x7f3c180b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1eb8a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c181597c0 0x7f3c181f16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1eb8a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1815c150 con 0x7f3c181597c0 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1eb8a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1815c7d0 con 0x7f3c1806f290 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1eb8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c180cd0c0 con 0x7f3c1806b4f0 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1d127640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c181597c0 0x7f3c181f16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1d127640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c181597c0 0x7f3c181f16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45322/0 (socket says 172.21.15.59:45322) 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1d127640 1 -- 172.21.15.59:0/3122322605 learned_addr learned my addr 172.21.15.59:0/3122322605 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1c926640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c1806b4f0 0x7f3c1814a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c17fff640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c1806f290 0x7f3c180b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1d127640 1 -- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c1806b4f0 msgr2=0x7f3c1814a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1d127640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c1806b4f0 0x7f3c1814a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1d127640 1 -- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c1806f290 msgr2=0x7f3c180b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1d127640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c1806f290 0x7f3c180b2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1d127640 1 -- 172.21.15.59:0/3122322605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1814b040 con 0x7f3c181597c0 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c1c926640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c1806b4f0 0x7f3c1814a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.163+0000 7f3c17fff640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c1806f290 0x7f3c180b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.164+0000 7f3c1d127640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c181597c0 0x7f3c181f16e0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000c780 tx=0x7f3c1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.164+0000 7f3c157fa640 1 -- 172.21.15.59:0/3122322605 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c10017070 con 0x7f3c181597c0 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.164+0000 7f3c157fa640 1 -- 172.21.15.59:0/3122322605 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c10003da0 con 0x7f3c181597c0 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.164+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/3122322605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c181f1f60 con 0x7f3c181597c0 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.164+0000 7f3c157fa640 1 -- 172.21.15.59:0/3122322605 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c100052b0 con 0x7f3c181597c0 2024-03-06T07:30:06.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.164+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/3122322605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c181f2520 con 0x7f3c181597c0 2024-03-06T07:30:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.165+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/3122322605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1804f330 con 0x7f3c181597c0 2024-03-06T07:30:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.165+0000 7f3c157fa640 1 -- 172.21.15.59:0/3122322605 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c1001a070 con 0x7f3c181597c0 2024-03-06T07:30:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.165+0000 7f3c157fa640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3bfc06b4d0 0x7f3bfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.166+0000 7f3c157fa640 1 -- 172.21.15.59:0/3122322605 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3c100894c0 con 0x7f3c181597c0 2024-03-06T07:30:06.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.166+0000 7f3c1c926640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3bfc06b4d0 0x7f3bfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.166+0000 7f3c1c926640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3bfc06b4d0 0x7f3bfc06d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009890 tx=0x7f3c0c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.170+0000 7f3c157fa640 1 -- 172.21.15.59:0/3122322605 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c10058640 con 0x7f3c181597c0 2024-03-06T07:30:06.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.352+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/3122322605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c1806c5d0 con 0x7f3c181597c0 2024-03-06T07:30:06.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.353+0000 7f3c157fa640 1 -- 172.21.15.59:0/3122322605 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) v1 ==== 74+0+13018 (secure 0 0 0) 0x7f3c10057f40 con 0x7f3c181597c0 2024-03-06T07:30:06.355 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:06.355 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":133,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:05.033315+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6835/1740714701":"2024-03-07T07:30:02.025620+0000","172.21.15.59:6834/1740714701":"2024-03-07T07:30:02.025620+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:06.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3bfc06b4d0 msgr2=0x7f3bfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3bfc06b4d0 0x7f3bfc06d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009890 tx=0x7f3c0c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:06.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/3122322605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c181597c0 msgr2=0x7f3c181f16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c181597c0 0x7f3c181f16e0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000c780 tx=0x7f3c1000cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:06.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/3122322605 shutdown_connections 2024-03-06T07:30:06.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3bfc06b4d0 0x7f3bfc06d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c181597c0 0x7f3c181f16e0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c1806f290 0x7f3c180b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 --2- 172.21.15.59:0/3122322605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c1806b4f0 0x7f3c1814a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/3122322605 >> 172.21.15.59:0/3122322605 conn(0x7f3c1806c0c0 msgr2=0x7f3c180cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/3122322605 shutdown_connections 2024-03-06T07:30:06.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.355+0000 7f3c1eb8a640 1 -- 172.21.15.59:0/3122322605 wait complete. 2024-03-06T07:30:06.368 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:06.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:06.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.471+0000 7f692260e640 1 Processor -- start 2024-03-06T07:30:06.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.471+0000 7f692260e640 1 -- start start 2024-03-06T07:30:06.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f692260e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f691c0d4860 0x7f691c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f692260e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c0b0e30 0x7f691c0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f692260e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c1618b0 con 0x7f691c1614d0 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f692260e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691c0d4cf0 con 0x7f691c0d4860 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f692260e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0cf400 con 0x7f691c0b0e30 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f6920bab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c0b0e30 0x7f691c0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f691b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f691c0d4860 0x7f691c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f691b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f691c0d4860 0x7f691c0d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:43832/0 (socket says 172.21.15.59:43832) 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f691b7fe640 1 -- 172.21.15.59:0/3937864295 learned_addr learned my addr 172.21.15.59:0/3937864295 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f6920bab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c0b0e30 0x7f691c0b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:41310/0 (socket says 172.21.15.59:41310) 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f691bfff640 1 --1- 172.21.15.59:0/3937864295 >> v1:172.21.15.164:6789/0 conn(0x7f691c1614d0 0x7f691c0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:46208/0 (socket says 172.21.15.59:46208) 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f691b7fe640 1 -- 172.21.15.59:0/3937864295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c0b0e30 msgr2=0x7f691c0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f691b7fe640 1 --2- 172.21.15.59:0/3937864295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c0b0e30 0x7f691c0b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f691b7fe640 1 -- 172.21.15.59:0/3937864295 >> v1:172.21.15.164:6789/0 conn(0x7f691c1614d0 legacy=0x7f691c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f691b7fe640 1 -- 172.21.15.59:0/3937864295 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c0cf6a0 con 0x7f691c0d4860 2024-03-06T07:30:06.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.472+0000 7f691affd640 1 -- 172.21.15.59:0/3937864295 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541517780 0 0) 0x7f691c1618b0 con 0x7f691c1614d0 2024-03-06T07:30:06.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.473+0000 7f691b7fe640 1 --2- 172.21.15.59:0/3937864295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c0d4860 0x7f691c0d2020 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f690c009870 tx=0x7f690c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5e013dabcf1960e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.473+0000 7f691affd640 1 -- 172.21.15.59:0/3937864295 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c034070 con 0x7f691c0d4860 2024-03-06T07:30:06.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.473+0000 7f691affd640 1 -- 172.21.15.59:0/3937864295 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690c030840 con 0x7f691c0d4860 2024-03-06T07:30:06.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.473+0000 7f691affd640 1 -- 172.21.15.59:0/3937864295 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c030c10 con 0x7f691c0d4860 2024-03-06T07:30:06.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.474+0000 7f692260e640 1 -- 172.21.15.59:0/3937864295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c0d4860 msgr2=0x7f691c0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.474+0000 7f692260e640 1 --2- 172.21.15.59:0/3937864295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c0d4860 0x7f691c0d2020 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f690c009870 tx=0x7f690c02b980 comp rx=0 tx=0).stop 2024-03-06T07:30:06.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.474+0000 7f692260e640 1 -- 172.21.15.59:0/3937864295 shutdown_connections 2024-03-06T07:30:06.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.474+0000 7f692260e640 1 --2- 172.21.15.59:0/3937864295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c0b0e30 0x7f691c0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.474+0000 7f692260e640 1 --2- 172.21.15.59:0/3937864295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c0d4860 0x7f691c0d2020 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.474+0000 7f692260e640 1 -- 172.21.15.59:0/3937864295 >> 172.21.15.59:0/3937864295 conn(0x7f691c0d3d40 msgr2=0x7f691c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.474+0000 7f692260e640 1 -- 172.21.15.59:0/3937864295 shutdown_connections 2024-03-06T07:30:06.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.474+0000 7f692260e640 1 -- 172.21.15.59:0/3937864295 wait complete. 2024-03-06T07:30:06.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.475+0000 7f692260e640 1 Processor -- start 2024-03-06T07:30:06.478 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.475+0000 7f692260e640 1 -- start start 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f692260e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c0b0e30 0x7f691c0c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f692260e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c0d4860 0x7f691c0cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f692260e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c1614d0 0x7f691c1572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f692260e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0b0910 con 0x7f691c0d4860 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f692260e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0cfc60 con 0x7f691c0b0e30 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f692260e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0d2560 con 0x7f691c1614d0 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f691b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c0d4860 0x7f691c0cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f691b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c0d4860 0x7f691c0cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45334/0 (socket says 172.21.15.59:45334) 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f691b7fe640 1 -- 172.21.15.59:0/2779074100 learned_addr learned my addr 172.21.15.59:0/2779074100 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f691bfff640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c0b0e30 0x7f691c0c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f6920bab640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c1614d0 0x7f691c1572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f691b7fe640 1 -- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c1614d0 msgr2=0x7f691c1572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f691b7fe640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c1614d0 0x7f691c1572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f691b7fe640 1 -- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c0b0e30 msgr2=0x7f691c0c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f691b7fe640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c0b0e30 0x7f691c0c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.476+0000 7f691b7fe640 1 -- 172.21.15.59:0/2779074100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c0c8980 con 0x7f691c0d4860 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.477+0000 7f691b7fe640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c0d4860 0x7f691c0cd850 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f690c0090b0 tx=0x7f690c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.477+0000 7f6918ff9640 1 -- 172.21.15.59:0/2779074100 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c034040 con 0x7f691c0d4860 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.477+0000 7f6918ff9640 1 -- 172.21.15.59:0/2779074100 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690c030640 con 0x7f691c0d4860 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.477+0000 7f6918ff9640 1 -- 172.21.15.59:0/2779074100 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c039d40 con 0x7f691c0d4860 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.477+0000 7f692260e640 1 -- 172.21.15.59:0/2779074100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c0c8c60 con 0x7f691c0d4860 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.477+0000 7f692260e640 1 -- 172.21.15.59:0/2779074100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c0c9220 con 0x7f691c0d4860 2024-03-06T07:30:06.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.478+0000 7f692260e640 1 -- 172.21.15.59:0/2779074100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c0caba0 con 0x7f691c0d4860 2024-03-06T07:30:06.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.478+0000 7f6918ff9640 1 -- 172.21.15.59:0/2779074100 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f690c04b060 con 0x7f691c0d4860 2024-03-06T07:30:06.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.479+0000 7f6918ff9640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f690806b4d0 0x7f690806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.479+0000 7f6918ff9640 1 -- 172.21.15.59:0/2779074100 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f690c0aba50 con 0x7f691c0d4860 2024-03-06T07:30:06.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.479+0000 7f691bfff640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f690806b4d0 0x7f690806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.479+0000 7f691bfff640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f690806b4d0 0x7f690806d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6910000f60 tx=0x7f6910007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.483+0000 7f6918ff9640 1 -- 172.21.15.59:0/2779074100 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f690c07a6a0 con 0x7f691c0d4860 2024-03-06T07:30:06.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.664+0000 7f692260e640 1 -- 172.21.15.59:0/2779074100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f691c068ff0 con 0x7f691c0d4860 2024-03-06T07:30:06.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.665+0000 7f6918ff9640 1 -- 172.21.15.59:0/2779074100 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v133) v1 ==== 93+0+10 (secure 0 0 0) 0x7f690c07e740 con 0x7f691c0d4860 2024-03-06T07:30:06.667 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:30:06.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 -- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f690806b4d0 msgr2=0x7f690806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f690806b4d0 0x7f690806d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6910000f60 tx=0x7f6910007d50 comp rx=0 tx=0).stop 2024-03-06T07:30:06.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 -- 172.21.15.59:0/2779074100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c0d4860 msgr2=0x7f691c0cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c0d4860 0x7f691c0cd850 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f690c0090b0 tx=0x7f690c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 -- 172.21.15.59:0/2779074100 shutdown_connections 2024-03-06T07:30:06.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f690806b4d0 0x7f690806d990 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f691c1614d0 0x7f691c1572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f691c0d4860 0x7f691c0cd850 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 --2- 172.21.15.59:0/2779074100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f691c0b0e30 0x7f691c0c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 -- 172.21.15.59:0/2779074100 >> 172.21.15.59:0/2779074100 conn(0x7f691c0d3d40 msgr2=0x7f691c0d0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 -- 172.21.15.59:0/2779074100 shutdown_connections 2024-03-06T07:30:06.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.667+0000 7f692260e640 1 -- 172.21.15.59:0/2779074100 wait complete. 2024-03-06T07:30:06.680 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-03-06T07:30:06.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.781+0000 7fce31602640 1 Processor -- start 2024-03-06T07:30:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.781+0000 7fce31602640 1 -- start start 2024-03-06T07:30:06.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.781+0000 7fce31602640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fce2c158380 0x7fce2c0adf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.781+0000 7fce31602640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fce2c06afb0 0x7fce2c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.781+0000 7fce31602640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c15e420 con 0x7fce2c0af8b0 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.781+0000 7fce31602640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce2c0ae480 con 0x7fce2c158380 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.781+0000 7fce31602640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce2c06b950 con 0x7fce2c06afb0 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.782+0000 7fce2affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fce2c158380 0x7fce2c0adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.782+0000 7fce2affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fce2c158380 0x7fce2c0adf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45348/0 (socket says 172.21.15.59:45348) 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.782+0000 7fce2affd640 1 -- 172.21.15.59:0/3475508798 learned_addr learned my addr 172.21.15.59:0/3475508798 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.782+0000 7fce2a7fc640 1 --2- 172.21.15.59:0/3475508798 >> v2:172.21.15.164:3300/0 conn(0x7fce2c06afb0 0x7fce2c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.782+0000 7fce2affd640 1 -- 172.21.15.59:0/3475508798 >> v2:172.21.15.164:3300/0 conn(0x7fce2c06afb0 msgr2=0x7fce2c06b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.782+0000 7fce2affd640 1 --2- 172.21.15.59:0/3475508798 >> v2:172.21.15.164:3300/0 conn(0x7fce2c06afb0 0x7fce2c06b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.782+0000 7fce2affd640 1 -- 172.21.15.59:0/3475508798 >> v1:172.21.15.164:6789/0 conn(0x7fce2c0af8b0 legacy=0x7fce2c0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.782+0000 7fce2affd640 1 -- 172.21.15.59:0/3475508798 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c15e700 con 0x7fce2c158380 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce2affd640 1 --2- 172.21.15.59:0/3475508798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce2c158380 0x7fce2c0adf40 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fce2c06d930 tx=0x7fce2402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a87604d5170764 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce29ffb640 1 -- 172.21.15.59:0/3475508798 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24035020 con 0x7fce2c158380 2024-03-06T07:30:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce29ffb640 1 -- 172.21.15.59:0/3475508798 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce24031870 con 0x7fce2c158380 2024-03-06T07:30:06.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce29ffb640 1 -- 172.21.15.59:0/3475508798 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24031c40 con 0x7fce2c158380 2024-03-06T07:30:06.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce31602640 1 -- 172.21.15.59:0/3475508798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce2c158380 msgr2=0x7fce2c0adf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce31602640 1 --2- 172.21.15.59:0/3475508798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce2c158380 0x7fce2c0adf40 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fce2c06d930 tx=0x7fce2402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:06.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce31602640 1 -- 172.21.15.59:0/3475508798 shutdown_connections 2024-03-06T07:30:06.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce31602640 1 --2- 172.21.15.59:0/3475508798 >> v2:172.21.15.164:3300/0 conn(0x7fce2c06afb0 0x7fce2c06b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce31602640 1 --2- 172.21.15.59:0/3475508798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce2c158380 0x7fce2c0adf40 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce31602640 1 -- 172.21.15.59:0/3475508798 >> 172.21.15.59:0/3475508798 conn(0x7fce2c06c0c0 msgr2=0x7fce2c06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce31602640 1 -- 172.21.15.59:0/3475508798 shutdown_connections 2024-03-06T07:30:06.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.783+0000 7fce31602640 1 -- 172.21.15.59:0/3475508798 wait complete. 2024-03-06T07:30:06.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.785+0000 7fce31602640 1 Processor -- start 2024-03-06T07:30:06.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.785+0000 7fce31602640 1 -- start start 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.785+0000 7fce31602640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce2c06afb0 0x7fce2c153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.785+0000 7fce31602640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce2c0af8b0 0x7fce2c1f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.785+0000 7fce31602640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce2c158380 0x7fce2c1f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.785+0000 7fce31602640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c15ecc0 con 0x7fce2c158380 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.785+0000 7fce31602640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c15f030 con 0x7fce2c0af8b0 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.785+0000 7fce31602640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c1619b0 con 0x7fce2c06afb0 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.785+0000 7fce2b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce2c158380 0x7fce2c1f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.785+0000 7fce2b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce2c158380 0x7fce2c1f72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45362/0 (socket says 172.21.15.59:45362) 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.785+0000 7fce2b7fe640 1 -- 172.21.15.59:0/1273998993 learned_addr learned my addr 172.21.15.59:0/1273998993 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce2affd640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce2c06afb0 0x7fce2c153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce2a7fc640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce2c0af8b0 0x7fce2c1f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce2b7fe640 1 -- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce2c06afb0 msgr2=0x7fce2c153ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce2b7fe640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce2c06afb0 0x7fce2c153ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce2b7fe640 1 -- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce2c0af8b0 msgr2=0x7fce2c1f6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce2b7fe640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce2c0af8b0 0x7fce2c1f6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce2b7fe640 1 -- 172.21.15.59:0/1273998993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c1547a0 con 0x7fce2c158380 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce2a7fc640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce2c0af8b0 0x7fce2c1f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce2b7fe640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce2c158380 0x7fce2c1f72c0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00c780 tx=0x7fce1c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce137fe640 1 -- 172.21.15.59:0/1273998993 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c004b60 con 0x7fce2c158380 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce137fe640 1 -- 172.21.15.59:0/1273998993 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1c004d00 con 0x7fce2c158380 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce31602640 1 -- 172.21.15.59:0/1273998993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2c154a80 con 0x7fce2c158380 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce137fe640 1 -- 172.21.15.59:0/1273998993 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c00aa90 con 0x7fce2c158380 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.786+0000 7fce31602640 1 -- 172.21.15.59:0/1273998993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2c155040 con 0x7fce2c158380 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.787+0000 7fce31602640 1 -- 172.21.15.59:0/1273998993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2c0733b0 con 0x7fce2c158380 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.788+0000 7fce137fe640 1 -- 172.21.15.59:0/1273998993 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce1c00ac30 con 0x7fce2c158380 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.788+0000 7fce137fe640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1806b4d0 0x7fce1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.788+0000 7fce137fe640 1 -- 172.21.15.59:0/1273998993 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fce1c08a150 con 0x7fce2c158380 2024-03-06T07:30:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.788+0000 7fce2affd640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1806b4d0 0x7fce1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.789+0000 7fce2affd640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1806b4d0 0x7fce1806d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fce24036000 tx=0x7fce2403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:06.792+0000 7fce137fe640 1 -- 172.21.15.59:0/1273998993 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce1c020070 con 0x7fce2c158380 2024-03-06T07:30:07.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:07.043+0000 7fce31602640 1 -- 172.21.15.59:0/1273998993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7fce2c068ff0 con 0x7fce2c158380 2024-03-06T07:30:08.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.047+0000 7fce137fe640 1 -- 172.21.15.59:0/1273998993 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v449) v1 ==== 144+0+0 (secure 0 0 0) 0x7fce1c059360 con 0x7fce2c158380 2024-03-06T07:30:08.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.049+0000 7fce31602640 1 -- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1806b4d0 msgr2=0x7fce1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.049+0000 7fce31602640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1806b4d0 0x7fce1806d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fce24036000 tx=0x7fce2403c040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.049+0000 7fce31602640 1 -- 172.21.15.59:0/1273998993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce2c158380 msgr2=0x7fce2c1f72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.049+0000 7fce31602640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce2c158380 0x7fce2c1f72c0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00c780 tx=0x7fce1c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:08.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.049+0000 7fce31602640 1 -- 172.21.15.59:0/1273998993 shutdown_connections 2024-03-06T07:30:08.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.049+0000 7fce31602640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1806b4d0 0x7fce1806d990 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.049+0000 7fce31602640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce2c158380 0x7fce2c1f72c0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.049+0000 7fce31602640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce2c0af8b0 0x7fce2c1f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.049+0000 7fce31602640 1 --2- 172.21.15.59:0/1273998993 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce2c06afb0 0x7fce2c153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.049+0000 7fce31602640 1 -- 172.21.15.59:0/1273998993 >> 172.21.15.59:0/1273998993 conn(0x7fce2c06c0c0 msgr2=0x7fce2c06e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.050+0000 7fce31602640 1 -- 172.21.15.59:0/1273998993 shutdown_connections 2024-03-06T07:30:08.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.050+0000 7fce31602640 1 -- 172.21.15.59:0/1273998993 wait complete. 2024-03-06T07:30:08.066 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:30:08.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.178+0000 7f6fbf84e640 1 Processor -- start 2024-03-06T07:30:08.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.178+0000 7f6fbf84e640 1 -- start start 2024-03-06T07:30:08.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.179+0000 7f6fbf84e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6fb815def0 0x7f6fb80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.179+0000 7f6fbf84e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb806cab0 0x7f6fb80ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.179+0000 7f6fbf84e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb806d030 con 0x7f6fb80ce880 2024-03-06T07:30:08.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.179+0000 7f6fbf84e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb80cecf0 con 0x7f6fb815def0 2024-03-06T07:30:08.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.179+0000 7f6fbf84e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80cf1c0 con 0x7f6fb806cab0 2024-03-06T07:30:08.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.180+0000 7f6fbcde9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb806cab0 0x7f6fb80ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.180+0000 7f6fbd5ea640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6fb815def0 0x7f6fb80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.180+0000 7f6fbcde9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb806cab0 0x7f6fb80ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:41326/0 (socket says 172.21.15.59:41326) 2024-03-06T07:30:08.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.180+0000 7f6fbcde9640 1 -- 172.21.15.59:0/3289228648 learned_addr learned my addr 172.21.15.59:0/3289228648 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:08.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.180+0000 7f6fbddeb640 1 --1- 172.21.15.59:0/3289228648 >> v1:172.21.15.164:6789/0 conn(0x7f6fb80ce880 0x7f6fb80cfd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:46240/0 (socket says 172.21.15.59:46240) 2024-03-06T07:30:08.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.180+0000 7f6fbcde9640 1 -- 172.21.15.59:0/3289228648 >> v2:172.21.15.164:3300/0 conn(0x7f6fb815def0 msgr2=0x7f6fb80b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.180+0000 7f6fbcde9640 1 --2- 172.21.15.59:0/3289228648 >> v2:172.21.15.164:3300/0 conn(0x7f6fb815def0 0x7f6fb80b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.180+0000 7f6fbcde9640 1 -- 172.21.15.59:0/3289228648 >> v1:172.21.15.164:6789/0 conn(0x7f6fb80ce880 legacy=0x7f6fb80cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.183 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.180+0000 7f6fbcde9640 1 -- 172.21.15.59:0/3289228648 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb806d310 con 0x7f6fb806cab0 2024-03-06T07:30:08.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.180+0000 7f6faffff640 1 -- 172.21.15.59:0/3289228648 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071271984 0 0) 0x7f6fb806d030 con 0x7f6fb80ce880 2024-03-06T07:30:08.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.180+0000 7f6fbd5ea640 1 --2- 172.21.15.59:0/3289228648 >> v2:172.21.15.164:3300/0 conn(0x7f6fb815def0 0x7f6fb80b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:08.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.181+0000 7f6fbcde9640 1 --2- 172.21.15.59:0/3289228648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb806cab0 0x7f6fb80ce340 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8009870 tx=0x7f6fa802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0a4aefcf5292094 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.181+0000 7f6faffff640 1 -- 172.21.15.59:0/3289228648 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa802f840 con 0x7f6fb806cab0 2024-03-06T07:30:08.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.181+0000 7f6faffff640 1 -- 172.21.15.59:0/3289228648 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa802f9e0 con 0x7f6fb806cab0 2024-03-06T07:30:08.184 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.181+0000 7f6faffff640 1 -- 172.21.15.59:0/3289228648 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa802fdb0 con 0x7f6fb806cab0 2024-03-06T07:30:08.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.182+0000 7f6fbf84e640 1 -- 172.21.15.59:0/3289228648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb806cab0 msgr2=0x7f6fb80ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.182+0000 7f6fbf84e640 1 --2- 172.21.15.59:0/3289228648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb806cab0 0x7f6fb80ce340 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8009870 tx=0x7f6fa802ba20 comp rx=0 tx=0).stop 2024-03-06T07:30:08.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.182+0000 7f6fbf84e640 1 -- 172.21.15.59:0/3289228648 shutdown_connections 2024-03-06T07:30:08.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.182+0000 7f6fbf84e640 1 --2- 172.21.15.59:0/3289228648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb806cab0 0x7f6fb80ce340 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.182+0000 7f6fbf84e640 1 --2- 172.21.15.59:0/3289228648 >> v2:172.21.15.164:3300/0 conn(0x7f6fb815def0 0x7f6fb80b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.182+0000 7f6fbf84e640 1 -- 172.21.15.59:0/3289228648 >> 172.21.15.59:0/3289228648 conn(0x7f6fb80b3690 msgr2=0x7f6fb80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.182+0000 7f6fbf84e640 1 -- 172.21.15.59:0/3289228648 shutdown_connections 2024-03-06T07:30:08.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.182+0000 7f6fbf84e640 1 -- 172.21.15.59:0/3289228648 wait complete. 2024-03-06T07:30:08.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.183+0000 7f6fbf84e640 1 Processor -- start 2024-03-06T07:30:08.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.183+0000 7f6fbf84e640 1 -- start start 2024-03-06T07:30:08.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbf84e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb806cab0 0x7f6fb8154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbf84e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb80ce880 0x7f6fb80c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbf84e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb815def0 0x7f6fb80c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbf84e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80cc3f0 con 0x7f6fb80ce880 2024-03-06T07:30:08.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbf84e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80cc760 con 0x7f6fb806cab0 2024-03-06T07:30:08.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbf84e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb815e2d0 con 0x7f6fb815def0 2024-03-06T07:30:08.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbcde9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb80ce880 0x7f6fb80c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbcde9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb80ce880 0x7f6fb80c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45378/0 (socket says 172.21.15.59:45378) 2024-03-06T07:30:08.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbcde9640 1 -- 172.21.15.59:0/168271605 learned_addr learned my addr 172.21.15.59:0/168271605 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbddeb640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb815def0 0x7f6fb80c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbd5ea640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb806cab0 0x7f6fb8154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbcde9640 1 -- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb815def0 msgr2=0x7f6fb80c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbcde9640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb815def0 0x7f6fb80c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbcde9640 1 -- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb806cab0 msgr2=0x7f6fb8154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbcde9640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb806cab0 0x7f6fb8154540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbcde9640 1 -- 172.21.15.59:0/168271605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb80cb8b0 con 0x7f6fb80ce880 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbd5ea640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb806cab0 0x7f6fb8154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.184+0000 7f6fbcde9640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb80ce880 0x7f6fb80c8b90 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8000c00 tx=0x7f6fa8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.185+0000 7f6fadffb640 1 -- 172.21.15.59:0/168271605 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8037bc0 con 0x7f6fb80ce880 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.185+0000 7f6fadffb640 1 -- 172.21.15.59:0/168271605 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa8046070 con 0x7f6fb80ce880 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.185+0000 7f6fbf84e640 1 -- 172.21.15.59:0/168271605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb80c9700 con 0x7f6fb80ce880 2024-03-06T07:30:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.185+0000 7f6fadffb640 1 -- 172.21.15.59:0/168271605 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8041400 con 0x7f6fb80ce880 2024-03-06T07:30:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.185+0000 7f6fbf84e640 1 -- 172.21.15.59:0/168271605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb80c9cc0 con 0x7f6fb80ce880 2024-03-06T07:30:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.186+0000 7f6fbf84e640 1 -- 172.21.15.59:0/168271605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb809fcf0 con 0x7f6fb80ce880 2024-03-06T07:30:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.186+0000 7f6fadffb640 1 -- 172.21.15.59:0/168271605 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fa8040070 con 0x7f6fb80ce880 2024-03-06T07:30:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.187+0000 7f6fadffb640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 0x7f6f9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.187+0000 7f6fadffb640 1 -- 172.21.15.59:0/168271605 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6fa80ac3c0 con 0x7f6fb80ce880 2024-03-06T07:30:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.187+0000 7f6fbd5ea640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 0x7f6f9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.187+0000 7f6fbd5ea640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 0x7f6f9c06d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f6fb815d6e0 tx=0x7f6fb0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.191+0000 7f6fadffb640 1 -- 172.21.15.59:0/168271605 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fa8033030 con 0x7f6fb80ce880 2024-03-06T07:30:08.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.372+0000 7f6fbf84e640 1 -- 172.21.15.59:0/168271605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6fb8068ff0 con 0x7f6fb80ce880 2024-03-06T07:30:08.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.375+0000 7f6fadffb640 1 -- 172.21.15.59:0/168271605 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6fa807b5d0 con 0x7f6fb80ce880 2024-03-06T07:30:08.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 -- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 msgr2=0x7f6f9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 0x7f6f9c06d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f6fb815d6e0 tx=0x7f6fb0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 -- 172.21.15.59:0/168271605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb80ce880 msgr2=0x7f6fb80c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb80ce880 0x7f6fb80c8b90 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8000c00 tx=0x7f6fa8004730 comp rx=0 tx=0).stop 2024-03-06T07:30:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 -- 172.21.15.59:0/168271605 shutdown_connections 2024-03-06T07:30:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6f9c06b4d0 0x7f6f9c06d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fb815def0 0x7f6fb80c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6fb80ce880 0x7f6fb80c8b90 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 --2- 172.21.15.59:0/168271605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb806cab0 0x7f6fb8154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 -- 172.21.15.59:0/168271605 >> 172.21.15.59:0/168271605 conn(0x7f6fb80b3690 msgr2=0x7f6fb81605f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 -- 172.21.15.59:0/168271605 shutdown_connections 2024-03-06T07:30:08.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.377+0000 7f6fbf84e640 1 -- 172.21.15.59:0/168271605 wait complete. 2024-03-06T07:30:08.390 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:08.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:08.436 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:08.487 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:08.487 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:30:08.488 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:08.502 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:08.503 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:30:08.503 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:08.519 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:08.520 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:30:08.520 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:08.557 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:08.557 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:30:08.557 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:08.577 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:08.577 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:30:08.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:08.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.697+0000 7fb4299d0640 1 Processor -- start 2024-03-06T07:30:08.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.697+0000 7fb4299d0640 1 -- start start 2024-03-06T07:30:08.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.698+0000 7fb4299d0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb424166af0 0x7fb424168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.698+0000 7fb4299d0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42415e380 con 0x7fb4240d00c0 2024-03-06T07:30:08.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.698+0000 7fb4299d0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4240cdc10 con 0x7fb4240cf850 2024-03-06T07:30:08.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.698+0000 7fb4299d0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42406ca90 con 0x7fb424166af0 2024-03-06T07:30:08.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.698+0000 7fb4237fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb424166af0 0x7fb424168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.698+0000 7fb422ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb4240d00c0 0x7fb4240d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55244/0 (socket says 172.21.15.59:55244) 2024-03-06T07:30:08.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.698+0000 7fb422ffd640 1 -- 172.21.15.59:0/2890889445 learned_addr learned my addr 172.21.15.59:0/2890889445 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:08.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.698+0000 7fb4237fe640 1 -- 172.21.15.59:0/2890889445 >> v1:172.21.15.164:6789/0 conn(0x7fb4240cf850 legacy=0x7fb4240cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.698+0000 7fb4237fe640 1 -- 172.21.15.59:0/2890889445 >> v1:172.21.15.59:6789/0 conn(0x7fb4240d00c0 legacy=0x7fb4240d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.698+0000 7fb4237fe640 1 -- 172.21.15.59:0/2890889445 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42406c160 con 0x7fb424166af0 2024-03-06T07:30:08.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.698+0000 7fb4227fc640 1 -- 172.21.15.59:0/2890889445 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205512754 0 0) 0x7fb42415e380 con 0x7fb4240d00c0 2024-03-06T07:30:08.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.699+0000 7fb4237fe640 1 --2- 172.21.15.59:0/2890889445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb424166af0 0x7fb424168ee0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009d90 tx=0x7fb41c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e168a8ad3695025b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.699+0000 7fb4227fc640 1 -- 172.21.15.59:0/2890889445 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c039070 con 0x7fb424166af0 2024-03-06T07:30:08.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.699+0000 7fb4227fc640 1 -- 172.21.15.59:0/2890889445 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c02cd00 con 0x7fb424166af0 2024-03-06T07:30:08.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.699+0000 7fb4227fc640 1 -- 172.21.15.59:0/2890889445 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c034660 con 0x7fb424166af0 2024-03-06T07:30:08.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.699+0000 7fb4299d0640 1 -- 172.21.15.59:0/2890889445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb424166af0 msgr2=0x7fb424168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.699+0000 7fb4299d0640 1 --2- 172.21.15.59:0/2890889445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb424166af0 0x7fb424168ee0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009d90 tx=0x7fb41c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:08.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.699+0000 7fb4299d0640 1 -- 172.21.15.59:0/2890889445 shutdown_connections 2024-03-06T07:30:08.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.699+0000 7fb4299d0640 1 --2- 172.21.15.59:0/2890889445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb424166af0 0x7fb424168ee0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.699+0000 7fb4299d0640 1 -- 172.21.15.59:0/2890889445 >> 172.21.15.59:0/2890889445 conn(0x7fb4240b35a0 msgr2=0x7fb4240b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.700+0000 7fb4299d0640 1 -- 172.21.15.59:0/2890889445 shutdown_connections 2024-03-06T07:30:08.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.700+0000 7fb4299d0640 1 -- 172.21.15.59:0/2890889445 wait complete. 2024-03-06T07:30:08.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.701+0000 7fb4299d0640 1 Processor -- start 2024-03-06T07:30:08.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.701+0000 7fb4299d0640 1 -- start start 2024-03-06T07:30:08.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.701+0000 7fb4299d0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4240cf850 0x7fb4240c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.701+0000 7fb4299d0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4240d00c0 0x7fb4241548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.701+0000 7fb4299d0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb424166af0 0x7fb424159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.701+0000 7fb4299d0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42406c720 con 0x7fb4240d00c0 2024-03-06T07:30:08.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.701+0000 7fb4299d0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42406df30 con 0x7fb424166af0 2024-03-06T07:30:08.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.701+0000 7fb4299d0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4240b1030 con 0x7fb4240cf850 2024-03-06T07:30:08.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.701+0000 7fb41bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4240d00c0 0x7fb4241548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.701+0000 7fb41bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4240d00c0 0x7fb4241548c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45390/0 (socket says 172.21.15.59:45390) 2024-03-06T07:30:08.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.701+0000 7fb41bfff640 1 -- 172.21.15.59:0/3604223615 learned_addr learned my addr 172.21.15.59:0/3604223615 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb4237fe640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb424166af0 0x7fb424159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb422ffd640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4240cf850 0x7fb4240c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb41bfff640 1 -- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4240cf850 msgr2=0x7fb4240c6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb41bfff640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4240cf850 0x7fb4240c6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb41bfff640 1 -- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb424166af0 msgr2=0x7fb424159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb41bfff640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb424166af0 0x7fb424159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb41bfff640 1 -- 172.21.15.59:0/3604223615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb424154e00 con 0x7fb4240d00c0 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb41bfff640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4240d00c0 0x7fb4241548c0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00c400 tx=0x7fb40c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb420ff9640 1 -- 172.21.15.59:0/3604223615 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c017070 con 0x7fb4240d00c0 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb420ff9640 1 -- 172.21.15.59:0/3604223615 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40c0052d0 con 0x7fb4240d00c0 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb4299d0640 1 -- 172.21.15.59:0/3604223615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4240c57d0 con 0x7fb4240d00c0 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb420ff9640 1 -- 172.21.15.59:0/3604223615 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c003830 con 0x7fb4240d00c0 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.702+0000 7fb4299d0640 1 -- 172.21.15.59:0/3604223615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4240c5d90 con 0x7fb4240d00c0 2024-03-06T07:30:08.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.703+0000 7fb4299d0640 1 -- 172.21.15.59:0/3604223615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42404f330 con 0x7fb4240d00c0 2024-03-06T07:30:08.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.704+0000 7fb420ff9640 1 -- 172.21.15.59:0/3604223615 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb40c01a070 con 0x7fb4240d00c0 2024-03-06T07:30:08.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.704+0000 7fb420ff9640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb40406b4d0 0x7fb40406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.704+0000 7fb420ff9640 1 -- 172.21.15.59:0/3604223615 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb40c089260 con 0x7fb4240d00c0 2024-03-06T07:30:08.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.704+0000 7fb422ffd640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb40406b4d0 0x7fb40406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.705+0000 7fb422ffd640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb40406b4d0 0x7fb40406d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb414000f60 tx=0x7fb414007b90 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.708+0000 7fb420ff9640 1 -- 172.21.15.59:0/3604223615 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb40c0583e0 con 0x7fb4240d00c0 2024-03-06T07:30:08.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.928+0000 7fb4299d0640 1 -- 172.21.15.59:0/3604223615 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4240b3bb0 con 0x7fb4240d00c0 2024-03-06T07:30:08.930 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.928+0000 7fb420ff9640 1 -- 172.21.15.59:0/3604223615 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2117 (secure 0 0 0) 0x7fb40c089080 con 0x7fb4240d00c0 2024-03-06T07:30:08.931 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:08.931 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":92,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/2939413041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":2939413041},{"type":"v1","addr":"172.21.15.59:6833","nonce":2939413041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":9255,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2734041273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2734041273},{"type":"v1","addr":"172.21.15.59:6835","nonce":2734041273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[]} 2024-03-06T07:30:08.931 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 92 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 -- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb40406b4d0 msgr2=0x7fb40406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb40406b4d0 0x7fb40406d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb414000f60 tx=0x7fb414007b90 comp rx=0 tx=0).stop 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 -- 172.21.15.59:0/3604223615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4240d00c0 msgr2=0x7fb4241548c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4240d00c0 0x7fb4241548c0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00c400 tx=0x7fb40c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 -- 172.21.15.59:0/3604223615 shutdown_connections 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb40406b4d0 0x7fb40406d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb424166af0 0x7fb424159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4240d00c0 0x7fb4241548c0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 --2- 172.21.15.59:0/3604223615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4240cf850 0x7fb4240c6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 -- 172.21.15.59:0/3604223615 >> 172.21.15.59:0/3604223615 conn(0x7fb4240b35a0 msgr2=0x7fb4240d1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 -- 172.21.15.59:0/3604223615 shutdown_connections 2024-03-06T07:30:08.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:08.930+0000 7fb4299d0640 1 -- 172.21.15.59:0/3604223615 wait complete. 2024-03-06T07:30:08.944 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:30:08.944 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:30:08.944 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:30:08.944 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:08.944 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:30:08.944 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:30:08.944 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:30:08.945 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:08.945 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:30:08.945 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:30:08.945 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:30:08.945 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:30:08.947 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:30:08.948 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:08.948 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:30:08.948 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:08.953 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:30:08.953 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:30:08.953 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:30:08.953 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:30:08.995 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:30:08.995 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:08.995 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:30:08.996 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:08.996 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:30:09.033 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2024-03-06T07:30:09.034 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2024-03-06T07:30:09.072 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:30:09.072 INFO:tasks.ceph.mds.a.smithi059.stdout:starting mds.a at 2024-03-06T07:30:09.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.099+0000 7f388bf8c640 1 Processor -- start 2024-03-06T07:30:09.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.100+0000 7f388bf8c640 1 -- start start 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.100+0000 7f388bf8c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38840d0d90 0x7f38840ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.100+0000 7f388bf8c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38840c90f0 0x7f38840ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.100+0000 7f388bf8c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38840d25e0 con 0x7f388415e000 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.100+0000 7f388bf8c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38840c7c50 con 0x7f38840d0d90 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.100+0000 7f388bf8c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38840d2880 con 0x7f38840c90f0 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.100+0000 7f3889527640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38840d0d90 0x7f38840ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.101+0000 7f3889527640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38840d0d90 0x7f38840ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45416/0 (socket says 172.21.15.59:45416) 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.101+0000 7f3889527640 1 -- 172.21.15.59:0/3286060758 learned_addr learned my addr 172.21.15.59:0/3286060758 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.101+0000 7f388a529640 1 --2- 172.21.15.59:0/3286060758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38840c90f0 0x7f38840ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.101+0000 7f3889527640 1 -- 172.21.15.59:0/3286060758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38840c90f0 msgr2=0x7f38840ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.101+0000 7f3889527640 1 --2- 172.21.15.59:0/3286060758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38840c90f0 0x7f38840ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.101+0000 7f3889527640 1 -- 172.21.15.59:0/3286060758 >> v1:172.21.15.164:6789/0 conn(0x7f388415e000 legacy=0x7f38840d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.101+0000 7f3889527640 1 -- 172.21.15.59:0/3286060758 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38840d2b40 con 0x7f38840d0d90 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.102+0000 7f3889527640 1 --2- 172.21.15.59:0/3286060758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38840d0d90 0x7f38840ce7c0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f3880009870 tx=0x7f388002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16297fb5a4592359 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.102+0000 7f3888d26640 1 -- 172.21.15.59:0/3286060758 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880034070 con 0x7f38840d0d90 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.102+0000 7f3888d26640 1 -- 172.21.15.59:0/3286060758 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3880030840 con 0x7f38840d0d90 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.102+0000 7f3888d26640 1 -- 172.21.15.59:0/3286060758 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880030c10 con 0x7f38840d0d90 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.102+0000 7f388bf8c640 1 -- 172.21.15.59:0/3286060758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38840d0d90 msgr2=0x7f38840ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.102+0000 7f388bf8c640 1 --2- 172.21.15.59:0/3286060758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38840d0d90 0x7f38840ce7c0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f3880009870 tx=0x7f388002b980 comp rx=0 tx=0).stop 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.102+0000 7f388bf8c640 1 -- 172.21.15.59:0/3286060758 shutdown_connections 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.102+0000 7f388bf8c640 1 --2- 172.21.15.59:0/3286060758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38840c90f0 0x7f38840ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.102+0000 7f388bf8c640 1 --2- 172.21.15.59:0/3286060758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38840d0d90 0x7f38840ce7c0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.102+0000 7f388bf8c640 1 -- 172.21.15.59:0/3286060758 >> 172.21.15.59:0/3286060758 conn(0x7f38840d3cf0 msgr2=0x7f38840d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.102+0000 7f388bf8c640 1 -- 172.21.15.59:0/3286060758 shutdown_connections 2024-03-06T07:30:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.103+0000 7f388bf8c640 1 -- 172.21.15.59:0/3286060758 wait complete. 2024-03-06T07:30:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.104+0000 7f388bf8c640 1 Processor -- start 2024-03-06T07:30:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.104+0000 7f388bf8c640 1 -- start start 2024-03-06T07:30:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.104+0000 7f388bf8c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38840c90f0 0x7f388415b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.104+0000 7f388bf8c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38840d0d90 0x7f38840cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.104+0000 7f388bf8c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f388415e000 0x7f38840cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.104+0000 7f388bf8c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388415e3e0 con 0x7f38840d0d90 2024-03-06T07:30:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.104+0000 7f388bf8c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840d22d0 con 0x7f388415e000 2024-03-06T07:30:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.104+0000 7f388bf8c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38840d4810 con 0x7f38840c90f0 2024-03-06T07:30:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.104+0000 7f3889527640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38840d0d90 0x7f38840cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.104+0000 7f3889527640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38840d0d90 0x7f38840cd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45426/0 (socket says 172.21.15.59:45426) 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.104+0000 7f3889527640 1 -- 172.21.15.59:0/1672081934 learned_addr learned my addr 172.21.15.59:0/1672081934 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f388a529640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f388415e000 0x7f38840cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f3889d28640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38840c90f0 0x7f388415b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f3889527640 1 -- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38840c90f0 msgr2=0x7f388415b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f3889527640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38840c90f0 0x7f388415b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f3889527640 1 -- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f388415e000 msgr2=0x7f38840cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f3889527640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f388415e000 0x7f38840cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f3889527640 1 -- 172.21.15.59:0/1672081934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388415a930 con 0x7f38840d0d90 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f388a529640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f388415e000 0x7f38840cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f3889527640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38840d0d90 0x7f38840cd230 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f3880009870 tx=0x7f3880004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f387a7fc640 1 -- 172.21.15.59:0/1672081934 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880048070 con 0x7f38840d0d90 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f387a7fc640 1 -- 172.21.15.59:0/1672081934 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3880030640 con 0x7f38840d0d90 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f387a7fc640 1 -- 172.21.15.59:0/1672081934 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880034040 con 0x7f38840d0d90 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f388bf8c640 1 -- 172.21.15.59:0/1672081934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388415ac10 con 0x7f38840d0d90 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.105+0000 7f388bf8c640 1 -- 172.21.15.59:0/1672081934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388415b1d0 con 0x7f38840d0d90 2024-03-06T07:30:09.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.106+0000 7f388bf8c640 1 -- 172.21.15.59:0/1672081934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388404f330 con 0x7f38840d0d90 2024-03-06T07:30:09.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.107+0000 7f387a7fc640 1 -- 172.21.15.59:0/1672081934 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f388004b070 con 0x7f38840d0d90 2024-03-06T07:30:09.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.107+0000 7f387a7fc640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386806b550 0x7f386806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.107+0000 7f387a7fc640 1 -- 172.21.15.59:0/1672081934 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f38800abf90 con 0x7f38840d0d90 2024-03-06T07:30:09.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.108+0000 7f3889d28640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386806b550 0x7f386806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.108+0000 7f3889d28640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386806b550 0x7f386806da10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3874006fd0 tx=0x7f3874008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.112+0000 7f387a7fc640 1 -- 172.21.15.59:0/1672081934 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f388007b210 con 0x7f38840d0d90 2024-03-06T07:30:09.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:09.307+0000 7f388bf8c640 1 -- 172.21.15.59:0/1672081934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f38840b4e10 con 0x7f38840d0d90 2024-03-06T07:30:10.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.060+0000 7f387a7fc640 1 -- 172.21.15.59:0/1672081934 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v135) v1 ==== 97+0+0 (secure 0 0 0) 0x7f388007ab10 con 0x7f38840d0d90 2024-03-06T07:30:10.062 INFO:teuthology.orchestra.run.smithi059.stderr: removed all blocklist entries 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.062+0000 7f388bf8c640 1 -- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386806b550 msgr2=0x7f386806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.062+0000 7f388bf8c640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386806b550 0x7f386806da10 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3874006fd0 tx=0x7f3874008040 comp rx=0 tx=0).stop 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.062+0000 7f388bf8c640 1 -- 172.21.15.59:0/1672081934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38840d0d90 msgr2=0x7f38840cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.062+0000 7f388bf8c640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38840d0d90 0x7f38840cd230 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f3880009870 tx=0x7f3880004730 comp rx=0 tx=0).stop 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.062+0000 7f388bf8c640 1 -- 172.21.15.59:0/1672081934 shutdown_connections 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.062+0000 7f388bf8c640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f386806b550 0x7f386806da10 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.062+0000 7f388bf8c640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f388415e000 0x7f38840cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.062+0000 7f388bf8c640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38840d0d90 0x7f38840cd230 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.062+0000 7f388bf8c640 1 --2- 172.21.15.59:0/1672081934 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38840c90f0 0x7f388415b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.062+0000 7f388bf8c640 1 -- 172.21.15.59:0/1672081934 >> 172.21.15.59:0/1672081934 conn(0x7f38840d3cf0 msgr2=0x7f38840b8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.062+0000 7f388bf8c640 1 -- 172.21.15.59:0/1672081934 shutdown_connections 2024-03-06T07:30:10.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.063+0000 7f388bf8c640 1 -- 172.21.15.59:0/1672081934 wait complete. 2024-03-06T07:30:10.078 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:30:10.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.185+0000 7fc77469d640 1 Processor -- start 2024-03-06T07:30:10.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.185+0000 7fc77469d640 1 -- start start 2024-03-06T07:30:10.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.186+0000 7fc77469d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc76c15dfe0 0x7fc76c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.186+0000 7fc77469d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc76c0cf830 0x7fc76c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.186+0000 7fc77469d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76c0b3430 con 0x7fc76c0d27f0 2024-03-06T07:30:10.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.186+0000 7fc77469d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc76c0d2c50 con 0x7fc76c0cf830 2024-03-06T07:30:10.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.186+0000 7fc77469d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c0b31e0 con 0x7fc76c15dfe0 2024-03-06T07:30:10.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.186+0000 7fc772c3a640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc76c0d27f0 0x7fc76c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55266/0 (socket says 172.21.15.59:55266) 2024-03-06T07:30:10.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.186+0000 7fc772c3a640 1 -- 172.21.15.59:0/2524137322 learned_addr learned my addr 172.21.15.59:0/2524137322 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:10.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737087382 0 0) 0x7fc76c0b3430 con 0x7fc76c0d27f0 2024-03-06T07:30:10.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc772439640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc76c15dfe0 0x7fc76c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc758003520 con 0x7fc76c0d27f0 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771c38640 1 --2- 172.21.15.59:0/2524137322 >> v2:172.21.15.164:3300/0 conn(0x7fc76c0cf830 0x7fc76c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1211900211 0 0) 0x7fc758003520 con 0x7fc76c0d27f0 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc76c15dfe0 msgr2=0x7fc76c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc76c15dfe0 0x7fc76c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 >> v2:172.21.15.164:3300/0 conn(0x7fc76c0cf830 msgr2=0x7fc76c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 --2- 172.21.15.59:0/2524137322 >> v2:172.21.15.164:3300/0 conn(0x7fc76c0cf830 0x7fc76c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c0b3710 con 0x7fc76c0d27f0 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fc7640019f0 con 0x7fc76c0d27f0 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 >> v1:172.21.15.59:6789/0 conn(0x7fc76c0d27f0 legacy=0x7fc76c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7580039b0 0x7fc758003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc76c15dfe0 0x7fc758004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc758004d40 0x7fc7580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c0b31e0 con 0x7fc758004d40 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75800b9c0 con 0x7fc7580039b0 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75800bb40 con 0x7fc76c15dfe0 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc772c3a640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc758004d40 0x7fc7580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc772439640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7580039b0 0x7fc758003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.187+0000 7fc771c38640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc76c15dfe0 0x7fc758004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.188+0000 7fc772c3a640 1 -- 172.21.15.59:0/2524137322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc76c15dfe0 msgr2=0x7fc758004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.188+0000 7fc772c3a640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc76c15dfe0 0x7fc758004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.188+0000 7fc772c3a640 1 -- 172.21.15.59:0/2524137322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7580039b0 msgr2=0x7fc758003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.188+0000 7fc772c3a640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7580039b0 0x7fc758003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.188+0000 7fc772c3a640 1 -- 172.21.15.59:0/2524137322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc758003520 con 0x7fc758004d40 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.188+0000 7fc772c3a640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc758004d40 0x7fc7580051c0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fc76400a870 tx=0x7fc758005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d13406961874c88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.188+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76400a4c0 con 0x7fc758004d40 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.188+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc764034630 con 0x7fc758004d40 2024-03-06T07:30:10.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.188+0000 7fc771437640 1 -- 172.21.15.59:0/2524137322 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc764034a20 con 0x7fc758004d40 2024-03-06T07:30:10.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.189+0000 7fc77469d640 1 -- 172.21.15.59:0/2524137322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc758004d40 msgr2=0x7fc7580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.189+0000 7fc77469d640 1 --2- 172.21.15.59:0/2524137322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc758004d40 0x7fc7580051c0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fc76400a870 tx=0x7fc758005830 comp rx=0 tx=0).stop 2024-03-06T07:30:10.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.189+0000 7fc772c3a640 1 -- 172.21.15.59:0/2524137322 reap_dead start 2024-03-06T07:30:10.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.189+0000 7fc77469d640 1 -- 172.21.15.59:0/2524137322 shutdown_connections 2024-03-06T07:30:10.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.189+0000 7fc77469d640 1 -- 172.21.15.59:0/2524137322 >> 172.21.15.59:0/2524137322 conn(0x7fc76c0d3cd0 msgr2=0x7fc76c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.189+0000 7fc77469d640 1 -- 172.21.15.59:0/2524137322 shutdown_connections 2024-03-06T07:30:10.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.189+0000 7fc77469d640 1 -- 172.21.15.59:0/2524137322 wait complete. 2024-03-06T07:30:10.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.190+0000 7fc77469d640 1 Processor -- start 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.190+0000 7fc77469d640 1 -- start start 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc77469d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc758004d40 0x7fc76c15c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc77469d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc76c1f37b0 0x7fc76c1f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc77469d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc76c06c8c0 0x7fc76c06cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc77469d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c0d1c70 con 0x7fc76c1f37b0 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc77469d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c0b3d60 con 0x7fc76c06c8c0 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc77469d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c0d1df0 con 0x7fc758004d40 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc771c38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc76c1f37b0 0x7fc76c1f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc771c38640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc76c1f37b0 0x7fc76c1f3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45444/0 (socket says 172.21.15.59:45444) 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc771c38640 1 -- 172.21.15.59:0/2453843214 learned_addr learned my addr 172.21.15.59:0/2453843214 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc772439640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc758004d40 0x7fc76c15c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc772c3a640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc76c06c8c0 0x7fc76c06cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc771c38640 1 -- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc758004d40 msgr2=0x7fc76c15c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc771c38640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc758004d40 0x7fc76c15c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc771c38640 1 -- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc76c06c8c0 msgr2=0x7fc76c06cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc771c38640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc76c06c8c0 0x7fc76c06cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.191+0000 7fc771c38640 1 -- 172.21.15.59:0/2453843214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c09db70 con 0x7fc76c1f37b0 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.192+0000 7fc771c38640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc76c1f37b0 0x7fc76c1f3c30 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fc75c011980 tx=0x7fc75c011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.192+0000 7fc762ffd640 1 -- 172.21.15.59:0/2453843214 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c019070 con 0x7fc76c1f37b0 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.192+0000 7fc762ffd640 1 -- 172.21.15.59:0/2453843214 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75c004240 con 0x7fc76c1f37b0 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.192+0000 7fc77469d640 1 -- 172.21.15.59:0/2453843214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76c09dde0 con 0x7fc76c1f37b0 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.192+0000 7fc762ffd640 1 -- 172.21.15.59:0/2453843214 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c00f820 con 0x7fc76c1f37b0 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.192+0000 7fc77469d640 1 -- 172.21.15.59:0/2453843214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76c09e3a0 con 0x7fc76c1f37b0 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.193+0000 7fc77469d640 1 -- 172.21.15.59:0/2453843214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76c0d1a90 con 0x7fc76c1f37b0 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.193+0000 7fc762ffd640 1 -- 172.21.15.59:0/2453843214 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc75c01c070 con 0x7fc76c1f37b0 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.194+0000 7fc762ffd640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc75006b500 0x7fc75006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.194+0000 7fc762ffd640 1 -- 172.21.15.59:0/2453843214 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc75c08a260 con 0x7fc76c1f37b0 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.194+0000 7fc772439640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc75006b500 0x7fc75006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.194+0000 7fc772439640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc75006b500 0x7fc75006d9c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc768009bc0 tx=0x7fc768009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.198+0000 7fc762ffd640 1 -- 172.21.15.59:0/2453843214 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc76c0d1a90 con 0x7fc76c1f37b0 2024-03-06T07:30:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.447+0000 7fc77469d640 1 -- 172.21.15.59:0/2453843214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc76c15d5a0 con 0x7fc76c1f37b0 2024-03-06T07:30:10.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.447+0000 7fc762ffd640 1 -- 172.21.15.59:0/2453843214 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc75c058de0 con 0x7fc76c1f37b0 2024-03-06T07:30:10.449 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:10.449 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:30:10.449 INFO:teuthology.orchestra.run.smithi059.stdout: "auth_dump": [ 2024-03-06T07:30:10.449 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.a", 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlODBZGBAAqV+TH2t2ZKyV9l13QEn6ew==", 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.b", 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlR1MuJRAAp6OAQ9gZemWtwI1dh3H2SA==", 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:30:10.450 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.c", 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlDXW2HxAAnKyuS1GW0wLO98/F9wdOAg==", 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.451 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mds.d", 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehl5MO4LBAAKmt2dvKk7BtcCM4o7cSwgg==", 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.452 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.0", 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBXGehlzdinFhAA0ACDAs9dtoY6EhUIveeIvA==", 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.1", 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBaGehlH92eFRAANQceHGuQyrbI6gP1m/Gakw==", 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.2", 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBdGehlMWpVFBAAbOG9nScbFlvKHXZMQJoQug==", 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.3", 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBgGehlLCyWEhAAyfoW/FmU37QD2IohknBwWw==", 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.454 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.4", 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBmGehldqv5GhAAWEloqqKgdpKPHjt7VKFodw==", 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.455 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.5", 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBpGehlc0ENGRAA5V9Az+vxAR3OUwWpDP7w1A==", 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.6", 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBsGehl5SYEFxAAPLyrHQV9SGguCV9f5VIkYg==", 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.456 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "osd.7", 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBvGehle3v0FBAA4xUPwX5couppHgi9VwSi2g==", 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile osd", 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.0", 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA==", 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow r", 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:10.457 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.1", 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlqhlAOhAAQ4l3uI8MlcCH+NAIWo/UuA==", 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.2", 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehltL/GNhAAZqSQ+UtwjI/PA95O1RGBNg==", 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:30:10.458 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.3", 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBUGehlltx5BBAAFa/+/w8Hd8lZegfIWtpJOA==", 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow", 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow r", 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow rw", 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow rwx" 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "client.admin", 2024-03-06T07:30:10.459 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBSGehlMExaNhAAQ2FD7afktMhrTbMMAK1epQ==", 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": "allow *", 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow *", 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.x", 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlLSTzFBAACeLdHsCX9KJGEsq44aqTyw==", 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.460 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "entity": "mgr.y", 2024-03-06T07:30:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "key": "AQBTGehlAgqXEBAACCHZ9Lp2lEyF93xD4Ns9Nw==", 2024-03-06T07:30:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "caps": { 2024-03-06T07:30:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": "allow *", 2024-03-06T07:30:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": "allow *" 2024-03-06T07:30:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:10.461 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2024-03-06T07:30:10.461 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:30:10.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 -- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc75006b500 msgr2=0x7fc75006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc75006b500 0x7fc75006d9c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc768009bc0 tx=0x7fc768009190 comp rx=0 tx=0).stop 2024-03-06T07:30:10.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 -- 172.21.15.59:0/2453843214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc76c1f37b0 msgr2=0x7fc76c1f3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc76c1f37b0 0x7fc76c1f3c30 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fc75c011980 tx=0x7fc75c011d80 comp rx=0 tx=0).stop 2024-03-06T07:30:10.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 -- 172.21.15.59:0/2453843214 shutdown_connections 2024-03-06T07:30:10.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc75006b500 0x7fc75006d9c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc76c06c8c0 0x7fc76c06cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc76c1f37b0 0x7fc76c1f3c30 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 --2- 172.21.15.59:0/2453843214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc758004d40 0x7fc76c15c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 -- 172.21.15.59:0/2453843214 >> 172.21.15.59:0/2453843214 conn(0x7fc76c0d3cd0 msgr2=0x7fc76c0ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 -- 172.21.15.59:0/2453843214 shutdown_connections 2024-03-06T07:30:10.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.449+0000 7fc77469d640 1 -- 172.21.15.59:0/2453843214 wait complete. 2024-03-06T07:30:10.464 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:10.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.563+0000 7ff80efbc640 1 Processor -- start 2024-03-06T07:30:10.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.563+0000 7ff80efbc640 1 -- start start 2024-03-06T07:30:10.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff80efbc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff80806da40 0x7ff8080cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff80efbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8080cd7a0 0x7ff80815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff80efbc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8080cc980 con 0x7ff8081424f0 2024-03-06T07:30:10.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff80efbc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8080cd220 con 0x7ff80806da40 2024-03-06T07:30:10.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff80efbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8080ccc60 con 0x7ff8080cd7a0 2024-03-06T07:30:10.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff80cd58640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff8081424f0 0x7ff80806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55272/0 (socket says 172.21.15.59:55272) 2024-03-06T07:30:10.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff80cd58640 1 -- 172.21.15.59:0/40811501 learned_addr learned my addr 172.21.15.59:0/40811501 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207818701 0 0) 0x7ff8080cc980 con 0x7ff8081424f0 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f8003520 con 0x7ff8081424f0 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff80d559640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8080cd7a0 0x7ff80815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff807fff640 1 --2- 172.21.15.59:0/40811501 >> v2:172.21.15.164:3300/0 conn(0x7ff80806da40 0x7ff8080cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 106829034 0 0) 0x7ff7f8003520 con 0x7ff8081424f0 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8080cd7a0 msgr2=0x7ff80815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8080cd7a0 0x7ff80815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 >> v2:172.21.15.164:3300/0 conn(0x7ff80806da40 msgr2=0x7ff8080cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 --2- 172.21.15.59:0/40811501 >> v2:172.21.15.164:3300/0 conn(0x7ff80806da40 0x7ff8080cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8080ccf40 con 0x7ff8081424f0 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7ff7fc001670 con 0x7ff8081424f0 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 >> v1:172.21.15.59:6789/0 conn(0x7ff8081424f0 legacy=0x7ff80806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7f80039b0 0x7ff7f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8080cd7a0 0x7ff7f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7f8004d40 0x7ff7f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8080cd220 con 0x7ff7f80039b0 2024-03-06T07:30:10.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8080ccc60 con 0x7ff7f8004d40 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f800ba70 con 0x7ff8080cd7a0 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.564+0000 7ff80cd58640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7f80039b0 0x7ff7f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.565+0000 7ff807fff640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8080cd7a0 0x7ff7f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.565+0000 7ff80d559640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7f8004d40 0x7ff7f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.565+0000 7ff80cd58640 1 -- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8080cd7a0 msgr2=0x7ff7f8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.565+0000 7ff80cd58640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8080cd7a0 0x7ff7f8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.565+0000 7ff80cd58640 1 -- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7f8004d40 msgr2=0x7ff7f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.565+0000 7ff80cd58640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7f8004d40 0x7ff7f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.565+0000 7ff80cd58640 1 -- 172.21.15.59:0/40811501 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f8003520 con 0x7ff7f80039b0 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.565+0000 7ff80d559640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7f8004d40 0x7ff7f80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.565+0000 7ff807fff640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8080cd7a0 0x7ff7f8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.565+0000 7ff80cd58640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7f80039b0 0x7ff7f8003dd0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00c240 tx=0x7ff7f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8406e243b1abea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.566+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc007d50 con 0x7ff7f80039b0 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.566+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc030070 con 0x7ff7f80039b0 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.566+0000 7ff8077fe640 1 -- 172.21.15.59:0/40811501 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc02cd80 con 0x7ff7f80039b0 2024-03-06T07:30:10.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.566+0000 7ff80efbc640 1 -- 172.21.15.59:0/40811501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7f80039b0 msgr2=0x7ff7f8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.566+0000 7ff80efbc640 1 --2- 172.21.15.59:0/40811501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7f80039b0 0x7ff7f8003dd0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00c240 tx=0x7ff7f8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:10.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.566+0000 7ff80d559640 1 -- 172.21.15.59:0/40811501 reap_dead start 2024-03-06T07:30:10.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.566+0000 7ff80efbc640 1 -- 172.21.15.59:0/40811501 shutdown_connections 2024-03-06T07:30:10.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.566+0000 7ff80efbc640 1 -- 172.21.15.59:0/40811501 >> 172.21.15.59:0/40811501 conn(0x7ff8080cf8a0 msgr2=0x7ff8080cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.566+0000 7ff80efbc640 1 -- 172.21.15.59:0/40811501 shutdown_connections 2024-03-06T07:30:10.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.566+0000 7ff80efbc640 1 -- 172.21.15.59:0/40811501 wait complete. 2024-03-06T07:30:10.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.567+0000 7ff80efbc640 1 Processor -- start 2024-03-06T07:30:10.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.567+0000 7ff80efbc640 1 -- start start 2024-03-06T07:30:10.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.568+0000 7ff80efbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7f80039b0 0x7ff808133e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.568+0000 7ff80efbc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff808157260 0x7ff8080b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.568+0000 7ff80efbc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8081313b0 0x7ff8081539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.568+0000 7ff80efbc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80815d070 con 0x7ff8081313b0 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.568+0000 7ff80efbc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80815cef0 con 0x7ff808157260 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.568+0000 7ff80efbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80815d1f0 con 0x7ff7f80039b0 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.568+0000 7ff80d559640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8081313b0 0x7ff8081539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.568+0000 7ff80d559640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8081313b0 0x7ff8081539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45454/0 (socket says 172.21.15.59:45454) 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.568+0000 7ff80d559640 1 -- 172.21.15.59:0/1683398285 learned_addr learned my addr 172.21.15.59:0/1683398285 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff807fff640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff808157260 0x7ff8080b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff80cd58640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7f80039b0 0x7ff808133e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff80d559640 1 -- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7f80039b0 msgr2=0x7ff808133e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff80d559640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7f80039b0 0x7ff808133e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff80d559640 1 -- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff808157260 msgr2=0x7ff8080b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff80d559640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff808157260 0x7ff8080b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff80d559640 1 -- 172.21.15.59:0/1683398285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8081417c0 con 0x7ff8081313b0 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff80d559640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8081313b0 0x7ff8081539b0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ff808073220 tx=0x7ff800004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff8057fa640 1 -- 172.21.15.59:0/1683398285 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800007c50 con 0x7ff8081313b0 2024-03-06T07:30:10.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff8057fa640 1 -- 172.21.15.59:0/1683398285 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff800019070 con 0x7ff8081313b0 2024-03-06T07:30:10.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff80efbc640 1 -- 172.21.15.59:0/1683398285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff808134390 con 0x7ff8081313b0 2024-03-06T07:30:10.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff8057fa640 1 -- 172.21.15.59:0/1683398285 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800014650 con 0x7ff8081313b0 2024-03-06T07:30:10.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.569+0000 7ff80efbc640 1 -- 172.21.15.59:0/1683398285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff808157720 con 0x7ff8081313b0 2024-03-06T07:30:10.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.570+0000 7ff80efbc640 1 -- 172.21.15.59:0/1683398285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80804f330 con 0x7ff8081313b0 2024-03-06T07:30:10.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.570+0000 7ff8057fa640 1 -- 172.21.15.59:0/1683398285 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8000147f0 con 0x7ff8081313b0 2024-03-06T07:30:10.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.571+0000 7ff8057fa640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff7ec06b500 0x7ff7ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.571+0000 7ff8057fa640 1 -- 172.21.15.59:0/1683398285 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff80008a450 con 0x7ff8081313b0 2024-03-06T07:30:10.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.571+0000 7ff80cd58640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff7ec06b500 0x7ff7ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.572+0000 7ff80cd58640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff7ec06b500 0x7ff7ec06d9c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc034000 tx=0x7ff7fc007000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.575+0000 7ff8057fa640 1 -- 172.21.15.59:0/1683398285 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff808134390 con 0x7ff8081313b0 2024-03-06T07:30:10.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.756+0000 7ff80efbc640 1 -- 172.21.15.59:0/1683398285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff80815bed0 con 0x7ff8081313b0 2024-03-06T07:30:10.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.757+0000 7ff8057fa640 1 -- 172.21.15.59:0/1683398285 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+12889 (secure 0 0 0) 0x7ff800059620 con 0x7ff8081313b0 2024-03-06T07:30:10.759 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:10.759 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":135,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:10.059528+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:10.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 -- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff7ec06b500 msgr2=0x7ff7ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff7ec06b500 0x7ff7ec06d9c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc034000 tx=0x7ff7fc007000 comp rx=0 tx=0).stop 2024-03-06T07:30:10.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 -- 172.21.15.59:0/1683398285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8081313b0 msgr2=0x7ff8081539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8081313b0 0x7ff8081539b0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ff808073220 tx=0x7ff800004840 comp rx=0 tx=0).stop 2024-03-06T07:30:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 -- 172.21.15.59:0/1683398285 shutdown_connections 2024-03-06T07:30:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff7ec06b500 0x7ff7ec06d9c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff8081313b0 0x7ff8081539b0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff808157260 0x7ff8080b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 --2- 172.21.15.59:0/1683398285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7f80039b0 0x7ff808133e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 -- 172.21.15.59:0/1683398285 >> 172.21.15.59:0/1683398285 conn(0x7ff8080cf8a0 msgr2=0x7ff808150a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 -- 172.21.15.59:0/1683398285 shutdown_connections 2024-03-06T07:30:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.759+0000 7ff80efbc640 1 -- 172.21.15.59:0/1683398285 wait complete. 2024-03-06T07:30:10.772 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:10.772 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:10.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.874+0000 7f6ec431e640 1 Processor -- start 2024-03-06T07:30:10.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.874+0000 7f6ec431e640 1 -- start start 2024-03-06T07:30:10.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.875+0000 7f6ec431e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6ebc06fd10 0x7f6ebc0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.875+0000 7f6ec431e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ebc0d0af0 0x7f6ebc0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.875+0000 7f6ec431e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc0cd1e0 con 0x7f6ebc15c050 2024-03-06T07:30:10.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.875+0000 7f6ec431e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ebc0cd4b0 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.875+0000 7f6ec431e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ebc0cd790 con 0x7f6ebc06fd10 2024-03-06T07:30:10.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.876+0000 7f6ec28bb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ebc0d0af0 0x7f6ebc0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.876+0000 7f6ec28bb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6ebc0d0af0 0x7f6ebc0ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45464/0 (socket says 172.21.15.59:45464) 2024-03-06T07:30:10.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.876+0000 7f6ec28bb640 1 -- 172.21.15.59:0/2499094664 learned_addr learned my addr 172.21.15.59:0/2499094664 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:10.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.876+0000 7f6ec18b9640 1 --2- 172.21.15.59:0/2499094664 >> v2:172.21.15.164:3300/0 conn(0x7f6ebc06fd10 0x7f6ebc0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.876+0000 7f6ec28bb640 1 -- 172.21.15.59:0/2499094664 >> v2:172.21.15.164:3300/0 conn(0x7f6ebc06fd10 msgr2=0x7f6ebc0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.876+0000 7f6ec28bb640 1 --2- 172.21.15.59:0/2499094664 >> v2:172.21.15.164:3300/0 conn(0x7f6ebc06fd10 0x7f6ebc0cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.876+0000 7f6ec28bb640 1 -- 172.21.15.59:0/2499094664 >> v1:172.21.15.164:6789/0 conn(0x7f6ebc15c050 legacy=0x7f6ebc06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.876+0000 7f6ec28bb640 1 -- 172.21.15.59:0/2499094664 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc0cda70 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.876+0000 7f6ec18b9640 1 --2- 172.21.15.59:0/2499094664 >> v2:172.21.15.164:3300/0 conn(0x7f6ebc06fd10 0x7f6ebc0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:10.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.877+0000 7f6ec28bb640 1 --2- 172.21.15.59:0/2499094664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ebc0d0af0 0x7f6ebc0ccca0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc0b7dc0 tx=0x7f6eac02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b54e2f916b10f60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.877+0000 7f6ec10b8640 1 -- 172.21.15.59:0/2499094664 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac02ca80 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.877+0000 7f6ec10b8640 1 -- 172.21.15.59:0/2499094664 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eac02cc20 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.877+0000 7f6ec10b8640 1 -- 172.21.15.59:0/2499094664 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac0343f0 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.877+0000 7f6ec431e640 1 -- 172.21.15.59:0/2499094664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ebc0d0af0 msgr2=0x7f6ebc0ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.877+0000 7f6ec431e640 1 --2- 172.21.15.59:0/2499094664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ebc0d0af0 0x7f6ebc0ccca0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc0b7dc0 tx=0x7f6eac02bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.877+0000 7f6ec431e640 1 -- 172.21.15.59:0/2499094664 shutdown_connections 2024-03-06T07:30:10.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.877+0000 7f6ec431e640 1 --2- 172.21.15.59:0/2499094664 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ebc0d0af0 0x7f6ebc0ccca0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.877+0000 7f6ec431e640 1 --2- 172.21.15.59:0/2499094664 >> v2:172.21.15.164:3300/0 conn(0x7f6ebc06fd10 0x7f6ebc0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.877+0000 7f6ec431e640 1 -- 172.21.15.59:0/2499094664 >> 172.21.15.59:0/2499094664 conn(0x7f6ebc0b3800 msgr2=0x7f6ebc0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.878+0000 7f6ec431e640 1 -- 172.21.15.59:0/2499094664 shutdown_connections 2024-03-06T07:30:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.878+0000 7f6ec431e640 1 -- 172.21.15.59:0/2499094664 wait complete. 2024-03-06T07:30:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.879+0000 7f6ec431e640 1 Processor -- start 2024-03-06T07:30:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.879+0000 7f6ec431e640 1 -- start start 2024-03-06T07:30:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.879+0000 7f6ec431e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ebc06fd10 0x7f6ebc158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.879+0000 7f6ec431e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ebc0d0af0 0x7f6ebc0b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.879+0000 7f6ec431e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ebc15c050 0x7f6ebc0b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.879+0000 7f6ec431e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc165c30 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.879+0000 7f6ec431e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc165ab0 con 0x7f6ebc06fd10 2024-03-06T07:30:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.879+0000 7f6ec431e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc165db0 con 0x7f6ebc15c050 2024-03-06T07:30:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.879+0000 7f6ec18b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ebc0d0af0 0x7f6ebc0b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.879+0000 7f6ec18b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ebc0d0af0 0x7f6ebc0b5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45472/0 (socket says 172.21.15.59:45472) 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.879+0000 7f6ec18b9640 1 -- 172.21.15.59:0/449111409 learned_addr learned my addr 172.21.15.59:0/449111409 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ec20ba640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ebc06fd10 0x7f6ebc158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ec28bb640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ebc15c050 0x7f6ebc0b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ec18b9640 1 -- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ebc15c050 msgr2=0x7f6ebc0b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ec18b9640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ebc15c050 0x7f6ebc0b6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ec18b9640 1 -- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ebc06fd10 msgr2=0x7f6ebc158d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ec18b9640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ebc06fd10 0x7f6ebc158d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ec18b9640 1 -- 172.21.15.59:0/449111409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc0b7850 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ec20ba640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ebc06fd10 0x7f6ebc158d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ec18b9640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ebc0d0af0 0x7f6ebc0b5ee0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000d7d0 tx=0x7f6eb000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ebaffd640 1 -- 172.21.15.59:0/449111409 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0004030 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ebaffd640 1 -- 172.21.15.59:0/449111409 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb00041d0 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ebaffd640 1 -- 172.21.15.59:0/449111409 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb00096a0 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ec431e640 1 -- 172.21.15.59:0/449111409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc0b6a90 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.880+0000 7f6ec431e640 1 -- 172.21.15.59:0/449111409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc0b7050 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.881+0000 7f6ec431e640 1 -- 172.21.15.59:0/449111409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ebc09fe60 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.882+0000 7f6ebaffd640 1 -- 172.21.15.59:0/449111409 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6eb0009840 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.882+0000 7f6ebaffd640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6ea806b500 0x7f6ea806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.882+0000 7f6ebaffd640 1 -- 172.21.15.59:0/449111409 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6eb008a170 con 0x7f6ebc0d0af0 2024-03-06T07:30:10.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.882+0000 7f6ec20ba640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6ea806b500 0x7f6ea806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.883+0000 7f6ec20ba640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6ea806b500 0x7f6ea806d9c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40015e0 tx=0x7f6eb4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:10.886+0000 7f6ebaffd640 1 -- 172.21.15.59:0/449111409 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6eb00592b0 con 0x7f6ebc0d0af0 2024-03-06T07:30:11.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.069+0000 7f6ec431e640 1 -- 172.21.15.59:0/449111409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ebc068ff0 con 0x7f6ebc0d0af0 2024-03-06T07:30:11.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.070+0000 7f6ebaffd640 1 -- 172.21.15.59:0/449111409 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6eb0058bb0 con 0x7f6ebc0d0af0 2024-03-06T07:30:11.072 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.071+0000 7f6ec431e640 1 -- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6ea806b500 msgr2=0x7f6ea806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.071+0000 7f6ec431e640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6ea806b500 0x7f6ea806d9c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40015e0 tx=0x7f6eb4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.072+0000 7f6ec431e640 1 -- 172.21.15.59:0/449111409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ebc0d0af0 msgr2=0x7f6ebc0b5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.072+0000 7f6ec431e640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ebc0d0af0 0x7f6ebc0b5ee0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000d7d0 tx=0x7f6eb000dc90 comp rx=0 tx=0).stop 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.072+0000 7f6ec431e640 1 -- 172.21.15.59:0/449111409 shutdown_connections 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.072+0000 7f6ec431e640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6ea806b500 0x7f6ea806d9c0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.072+0000 7f6ec431e640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ebc15c050 0x7f6ebc0b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.072+0000 7f6ec431e640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ebc0d0af0 0x7f6ebc0b5ee0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.072+0000 7f6ec431e640 1 --2- 172.21.15.59:0/449111409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ebc06fd10 0x7f6ebc158d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.072+0000 7f6ec431e640 1 -- 172.21.15.59:0/449111409 >> 172.21.15.59:0/449111409 conn(0x7f6ebc0b3800 msgr2=0x7f6ebc154f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.072+0000 7f6ec431e640 1 -- 172.21.15.59:0/449111409 shutdown_connections 2024-03-06T07:30:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.072+0000 7f6ec431e640 1 -- 172.21.15.59:0/449111409 wait complete. 2024-03-06T07:30:11.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:30:11.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.184+0000 7f09604a4640 1 Processor -- start 2024-03-06T07:30:11.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.184+0000 7f09604a4640 1 -- start start 2024-03-06T07:30:11.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.185+0000 7f09604a4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f09580cce40 0x7f095815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.185+0000 7f09604a4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0958071f40 con 0x7f09581515e0 2024-03-06T07:30:11.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.185+0000 7f09604a4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0958070a70 con 0x7f09580cc340 2024-03-06T07:30:11.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.185+0000 7f09604a4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09580cd2f0 con 0x7f09580cce40 2024-03-06T07:30:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.185+0000 7f095da3f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f09580cce40 0x7f095815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.185+0000 7f095e240640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f09581515e0 0x7f0958071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55274/0 (socket says 172.21.15.59:55274) 2024-03-06T07:30:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.185+0000 7f095e240640 1 -- 172.21.15.59:0/102571356 learned_addr learned my addr 172.21.15.59:0/102571356 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.185+0000 7f095da3f640 1 -- 172.21.15.59:0/102571356 >> v1:172.21.15.164:6789/0 conn(0x7f09580cc340 legacy=0x7f09580cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.185+0000 7f095da3f640 1 -- 172.21.15.59:0/102571356 >> v1:172.21.15.59:6789/0 conn(0x7f09581515e0 legacy=0x7f0958071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.185+0000 7f095da3f640 1 -- 172.21.15.59:0/102571356 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0958070d50 con 0x7f09580cce40 2024-03-06T07:30:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.186+0000 7f095da3f640 1 --2- 172.21.15.59:0/102571356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09580cce40 0x7f095815b430 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f09480097a0 tx=0x7f094802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2eb396e9689d79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.186+0000 7f095d23e640 1 -- 172.21.15.59:0/102571356 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948035020 con 0x7f09580cce40 2024-03-06T07:30:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.186+0000 7f095d23e640 1 -- 172.21.15.59:0/102571356 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0948031830 con 0x7f09580cce40 2024-03-06T07:30:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.186+0000 7f095d23e640 1 -- 172.21.15.59:0/102571356 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948031c00 con 0x7f09580cce40 2024-03-06T07:30:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.187+0000 7f09604a4640 1 -- 172.21.15.59:0/102571356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09580cce40 msgr2=0x7f095815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.187+0000 7f09604a4640 1 --2- 172.21.15.59:0/102571356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09580cce40 0x7f095815b430 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f09480097a0 tx=0x7f094802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.187+0000 7f09604a4640 1 -- 172.21.15.59:0/102571356 shutdown_connections 2024-03-06T07:30:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.187+0000 7f09604a4640 1 --2- 172.21.15.59:0/102571356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09580cce40 0x7f095815b430 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.187+0000 7f09604a4640 1 -- 172.21.15.59:0/102571356 >> 172.21.15.59:0/102571356 conn(0x7f09580cf8a0 msgr2=0x7f09580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.187+0000 7f09604a4640 1 -- 172.21.15.59:0/102571356 shutdown_connections 2024-03-06T07:30:11.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.187+0000 7f09604a4640 1 -- 172.21.15.59:0/102571356 wait complete. 2024-03-06T07:30:11.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.188+0000 7f09604a4640 1 Processor -- start 2024-03-06T07:30:11.190 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.188+0000 7f09604a4640 1 -- start start 2024-03-06T07:30:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f09604a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09580cc340 0x7f095806d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f09604a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09580cce40 0x7f095813af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f09604a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09581515e0 0x7f095813b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f09604a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09580cdeb0 con 0x7f09580cce40 2024-03-06T07:30:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f09604a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0958071310 con 0x7f09581515e0 2024-03-06T07:30:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f09604a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09580ce030 con 0x7f09580cc340 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f095da3f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09580cce40 0x7f095813af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f095da3f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09580cce40 0x7f095813af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45488/0 (socket says 172.21.15.59:45488) 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f095da3f640 1 -- 172.21.15.59:0/4219224529 learned_addr learned my addr 172.21.15.59:0/4219224529 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f095da3f640 1 -- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09580cc340 msgr2=0x7f095806d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f095da3f640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09580cc340 0x7f095806d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f095da3f640 1 -- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09581515e0 msgr2=0x7f095813b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f095ea41640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09581515e0 0x7f095813b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f095da3f640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09581515e0 0x7f095813b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f095da3f640 1 -- 172.21.15.59:0/4219224529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0958138f40 con 0x7f09580cce40 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.189+0000 7f095da3f640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09580cce40 0x7f095813af50 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f09480098c0 tx=0x7f09480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.190+0000 7f094effd640 1 -- 172.21.15.59:0/4219224529 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948041070 con 0x7f09580cce40 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.190+0000 7f094effd640 1 -- 172.21.15.59:0/4219224529 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094803aa30 con 0x7f09580cce40 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.190+0000 7f094effd640 1 -- 172.21.15.59:0/4219224529 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948035040 con 0x7f09580cce40 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.190+0000 7f09604a4640 1 -- 172.21.15.59:0/4219224529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095806de00 con 0x7f09580cce40 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.190+0000 7f09604a4640 1 -- 172.21.15.59:0/4219224529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09581394c0 con 0x7f09580cce40 2024-03-06T07:30:11.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.191+0000 7f09604a4640 1 -- 172.21.15.59:0/4219224529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0958068840 con 0x7f09580cce40 2024-03-06T07:30:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.191+0000 7f094effd640 1 -- 172.21.15.59:0/4219224529 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0948038020 con 0x7f09580cce40 2024-03-06T07:30:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.191+0000 7f094effd640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f093c06b4d0 0x7f093c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.191+0000 7f094effd640 1 -- 172.21.15.59:0/4219224529 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f09480ab950 con 0x7f09580cce40 2024-03-06T07:30:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.192+0000 7f095e240640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f093c06b4d0 0x7f093c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.192+0000 7f095e240640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f093c06b4d0 0x7f093c06d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0950000f60 tx=0x7f0950007a80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.196+0000 7f094effd640 1 -- 172.21.15.59:0/4219224529 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f094807ab10 con 0x7f09580cce40 2024-03-06T07:30:11.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.380+0000 7f09604a4640 1 -- 172.21.15.59:0/4219224529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f095804f330 con 0x7f09580cce40 2024-03-06T07:30:11.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.380+0000 7f094effd640 1 -- 172.21.15.59:0/4219224529 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v135) v1 ==== 84+0+65 (secure 0 0 0) 0x7f094807a410 con 0x7f09580cce40 2024-03-06T07:30:11.382 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:11.382 INFO:teuthology.orchestra.run.smithi059.stdout:[ 2024-03-06T07:30:11.383 INFO:teuthology.orchestra.run.smithi059.stdout: { 2024-03-06T07:30:11.383 INFO:teuthology.orchestra.run.smithi059.stdout: "poolnum": 1, 2024-03-06T07:30:11.383 INFO:teuthology.orchestra.run.smithi059.stdout: "poolname": ".mgr" 2024-03-06T07:30:11.383 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:11.383 INFO:teuthology.orchestra.run.smithi059.stdout:] 2024-03-06T07:30:11.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 -- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f093c06b4d0 msgr2=0x7f093c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f093c06b4d0 0x7f093c06d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0950000f60 tx=0x7f0950007a80 comp rx=0 tx=0).stop 2024-03-06T07:30:11.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 -- 172.21.15.59:0/4219224529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09580cce40 msgr2=0x7f095813af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09580cce40 0x7f095813af50 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f09480098c0 tx=0x7f09480046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 -- 172.21.15.59:0/4219224529 shutdown_connections 2024-03-06T07:30:11.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f093c06b4d0 0x7f093c06d990 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09581515e0 0x7f095813b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09580cce40 0x7f095813af50 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 --2- 172.21.15.59:0/4219224529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09580cc340 0x7f095806d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 -- 172.21.15.59:0/4219224529 >> 172.21.15.59:0/4219224529 conn(0x7f09580cf8a0 msgr2=0x7f0958071830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 -- 172.21.15.59:0/4219224529 shutdown_connections 2024-03-06T07:30:11.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.382+0000 7f09604a4640 1 -- 172.21.15.59:0/4219224529 wait complete. 2024-03-06T07:30:11.395 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:30:11.395 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:30:11.500 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.499+0000 7f0eeab4b640 1 Processor -- start 2024-03-06T07:30:11.501 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.499+0000 7f0eeab4b640 1 -- start start 2024-03-06T07:30:11.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.499+0000 7f0eeab4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee415e050 0x7f0ee40cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.504 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.499+0000 7f0eeab4b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0ee406b6c0 0x7f0ee40cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.499+0000 7f0eeab4b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee40d1860 con 0x7f0ee40ce170 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.499+0000 7f0eeab4b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee415e430 con 0x7f0ee406b6c0 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.499+0000 7f0eeab4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee40cd440 con 0x7f0ee415e050 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee3fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0ee40ce170 0x7f0ee406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55280/0 (socket says 172.21.15.59:55280) 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee3fff640 1 -- 172.21.15.59:0/4032639628 learned_addr learned my addr 172.21.15.59:0/4032639628 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee90e8640 1 --2- 172.21.15.59:0/4032639628 >> v2:172.21.15.164:3300/0 conn(0x7f0ee406b6c0 0x7f0ee40cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee88e7640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee415e050 0x7f0ee40cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514671310 0 0) 0x7f0ee40d1860 con 0x7f0ee40ce170 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed4003520 con 0x7f0ee40ce170 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 254969987 0 0) 0x7f0ed4003520 con 0x7f0ee40ce170 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee415e050 msgr2=0x7f0ee40cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee415e050 0x7f0ee40cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 >> v2:172.21.15.164:3300/0 conn(0x7f0ee406b6c0 msgr2=0x7f0ee40cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.505 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 --2- 172.21.15.59:0/4032639628 >> v2:172.21.15.164:3300/0 conn(0x7f0ee406b6c0 0x7f0ee40cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee406f1f0 con 0x7f0ee40ce170 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f0ed0001670 con 0x7f0ee40ce170 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee90e8640 1 --2- 172.21.15.59:0/4032639628 >> v2:172.21.15.164:3300/0 conn(0x7f0ee406b6c0 0x7f0ee40cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee88e7640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee415e050 0x7f0ee40cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 >> v1:172.21.15.59:6789/0 conn(0x7f0ee40ce170 legacy=0x7f0ee406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ed40039b0 0x7f0ed4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee415e050 0x7f0ed4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ed4004d40 0x7f0ed40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee415e430 con 0x7f0ed4004d40 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee40cd440 con 0x7f0ed40039b0 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed400b9c0 con 0x7f0ee415e050 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee90e8640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ed4004d40 0x7f0ed40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee88e7640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ed40039b0 0x7f0ed4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.506 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.500+0000 7f0ee3fff640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee415e050 0x7f0ed4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.501+0000 7f0ee90e8640 1 -- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee415e050 msgr2=0x7f0ed4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.501+0000 7f0ee90e8640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee415e050 0x7f0ed4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.501+0000 7f0ee90e8640 1 -- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ed40039b0 msgr2=0x7f0ed4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.501+0000 7f0ee90e8640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ed40039b0 0x7f0ed4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.501+0000 7f0ee90e8640 1 -- 172.21.15.59:0/4032639628 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed4003520 con 0x7f0ed4004d40 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.501+0000 7f0ee90e8640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ed4004d40 0x7f0ed40051c0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00e320 tx=0x7f0ed4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea56905008f5220 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.501+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc009de0 con 0x7f0ed4004d40 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.502+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0edc02cde0 con 0x7f0ed4004d40 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.502+0000 7f0ee37fe640 1 -- 172.21.15.59:0/4032639628 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc003b90 con 0x7f0ed4004d40 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.502+0000 7f0eeab4b640 1 -- 172.21.15.59:0/4032639628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ed4004d40 msgr2=0x7f0ed40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.502+0000 7f0eeab4b640 1 --2- 172.21.15.59:0/4032639628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ed4004d40 0x7f0ed40051c0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00e320 tx=0x7f0ed4005830 comp rx=0 tx=0).stop 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.502+0000 7f0ee90e8640 1 -- 172.21.15.59:0/4032639628 reap_dead start 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.502+0000 7f0eeab4b640 1 -- 172.21.15.59:0/4032639628 shutdown_connections 2024-03-06T07:30:11.507 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.502+0000 7f0eeab4b640 1 -- 172.21.15.59:0/4032639628 >> 172.21.15.59:0/4032639628 conn(0x7f0ee40d0a30 msgr2=0x7f0ee40d0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.502+0000 7f0eeab4b640 1 -- 172.21.15.59:0/4032639628 shutdown_connections 2024-03-06T07:30:11.508 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.502+0000 7f0eeab4b640 1 -- 172.21.15.59:0/4032639628 wait complete. 2024-03-06T07:30:11.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.503+0000 7f0eeab4b640 1 Processor -- start 2024-03-06T07:30:11.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.503+0000 7f0eeab4b640 1 -- start start 2024-03-06T07:30:11.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0eeab4b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ee40b4f50 0x7f0ee4156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0eeab4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40b5330 0x7f0ee41f4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0eeab4b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee41f53f0 0x7f0ee414f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.510 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0eeab4b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee40cd690 con 0x7f0ee40b4f50 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0eeab4b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee40cdce0 con 0x7f0ee41f53f0 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0eeab4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee40cd970 con 0x7f0ee40b5330 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0ee88e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ee40b4f50 0x7f0ee4156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0ee88e7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ee40b4f50 0x7f0ee4156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45504/0 (socket says 172.21.15.59:45504) 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0ee88e7640 1 -- 172.21.15.59:0/2637020063 learned_addr learned my addr 172.21.15.59:0/2637020063 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0ee3fff640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40b5330 0x7f0ee41f4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0ee90e8640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee41f53f0 0x7f0ee414f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0ee88e7640 1 -- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40b5330 msgr2=0x7f0ee41f4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0ee88e7640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40b5330 0x7f0ee41f4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0ee88e7640 1 -- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee41f53f0 msgr2=0x7f0ee414f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.504+0000 7f0ee88e7640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee41f53f0 0x7f0ee414f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.505+0000 7f0ee88e7640 1 -- 172.21.15.59:0/2637020063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee41f51c0 con 0x7f0ee40b4f50 2024-03-06T07:30:11.511 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.505+0000 7f0ee3fff640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40b5330 0x7f0ee41f4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.505+0000 7f0ee90e8640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee41f53f0 0x7f0ee414f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.505+0000 7f0ee88e7640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ee40b4f50 0x7f0ee4156650 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8011930 tx=0x7f0ed8011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.505+0000 7f0ee17fa640 1 -- 172.21.15.59:0/2637020063 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed800faf0 con 0x7f0ee40b4f50 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.505+0000 7f0ee17fa640 1 -- 172.21.15.59:0/2637020063 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed800fc90 con 0x7f0ee40b4f50 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.505+0000 7f0eeab4b640 1 -- 172.21.15.59:0/2637020063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee4156c30 con 0x7f0ee40b4f50 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.505+0000 7f0ee17fa640 1 -- 172.21.15.59:0/2637020063 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8004030 con 0x7f0ee40b4f50 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.505+0000 7f0eeab4b640 1 -- 172.21.15.59:0/2637020063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee41550e0 con 0x7f0ee40b4f50 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.506+0000 7f0eeab4b640 1 -- 172.21.15.59:0/2637020063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee404f330 con 0x7f0ee40b4f50 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.506+0000 7f0ee17fa640 1 -- 172.21.15.59:0/2637020063 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ed8012040 con 0x7f0ee40b4f50 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.507+0000 7f0ee17fa640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ec806b500 0x7f0ec806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.507+0000 7f0ee17fa640 1 -- 172.21.15.59:0/2637020063 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0ed808a240 con 0x7f0ee40b4f50 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.507+0000 7f0ee3fff640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ec806b500 0x7f0ec806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:11.512 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.507+0000 7f0ee3fff640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ec806b500 0x7f0ec806d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0004840 tx=0x7f0ed0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:11.513 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.511+0000 7f0ee17fa640 1 -- 172.21.15.59:0/2637020063 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ed8019070 con 0x7f0ee40b4f50 2024-03-06T07:30:11.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:11.692+0000 7f0eeab4b640 1 -- 172.21.15.59:0/2637020063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0ee41554e0 con 0x7f0ee40b4f50 2024-03-06T07:30:12.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.067+0000 7f0ee17fa640 1 -- 172.21.15.59:0/2637020063 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v136) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0ed8059410 con 0x7f0ee40b4f50 2024-03-06T07:30:12.069 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' created 2024-03-06T07:30:12.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 -- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ec806b500 msgr2=0x7f0ec806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ec806b500 0x7f0ec806d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0004840 tx=0x7f0ed0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:12.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 -- 172.21.15.59:0/2637020063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ee40b4f50 msgr2=0x7f0ee4156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ee40b4f50 0x7f0ee4156650 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8011930 tx=0x7f0ed8011df0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 -- 172.21.15.59:0/2637020063 shutdown_connections 2024-03-06T07:30:12.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0ec806b500 0x7f0ec806d9c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee41f53f0 0x7f0ee414f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40b5330 0x7f0ee41f4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 --2- 172.21.15.59:0/2637020063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ee40b4f50 0x7f0ee4156650 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 -- 172.21.15.59:0/2637020063 >> 172.21.15.59:0/2637020063 conn(0x7f0ee40d0a30 msgr2=0x7f0ee406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 -- 172.21.15.59:0/2637020063 shutdown_connections 2024-03-06T07:30:12.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.070+0000 7f0eeab4b640 1 -- 172.21.15.59:0/2637020063 wait complete. 2024-03-06T07:30:12.086 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:30:12.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.193+0000 7fbcf414b640 1 Processor -- start 2024-03-06T07:30:12.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.193+0000 7fbcf414b640 1 -- start start 2024-03-06T07:30:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf414b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbcec06cab0 0x7fbcec0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf414b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec06d030 con 0x7fbcec0cea50 2024-03-06T07:30:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf414b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec06d310 con 0x7fbcec15def0 2024-03-06T07:30:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf414b640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcec0cf1c0 con 0x7fbcec06cab0 2024-03-06T07:30:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf26e8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbcec06cab0 0x7fbcec0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf16e6640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbcec0cea50 0x7fbcec0ccbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55292/0 (socket says 172.21.15.59:55292) 2024-03-06T07:30:12.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf26e8640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbcec06cab0 0x7fbcec0b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45518/0 (socket says 172.21.15.59:45518) 2024-03-06T07:30:12.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf16e6640 1 -- 172.21.15.59:0/3688268555 learned_addr learned my addr 172.21.15.59:0/3688268555 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:12.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf26e8640 1 -- 172.21.15.59:0/3688268555 >> v1:172.21.15.164:6789/0 conn(0x7fbcec15def0 legacy=0x7fbcec0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf26e8640 1 -- 172.21.15.59:0/3688268555 >> v1:172.21.15.59:6789/0 conn(0x7fbcec0cea50 legacy=0x7fbcec0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf26e8640 1 -- 172.21.15.59:0/3688268555 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec0cc110 con 0x7fbcec06cab0 2024-03-06T07:30:12.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.194+0000 7fbcf0ee5640 1 -- 172.21.15.59:0/3688268555 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502106715 0 0) 0x7fbcec06d030 con 0x7fbcec0cea50 2024-03-06T07:30:12.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.195+0000 7fbcf26e8640 1 --2- 172.21.15.59:0/3688268555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcec06cab0 0x7fbcec0b41b0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fbce40097a0 tx=0x7fbce402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef1b803a30aa525 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.195+0000 7fbcf0ee5640 1 -- 172.21.15.59:0/3688268555 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce4034070 con 0x7fbcec06cab0 2024-03-06T07:30:12.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.195+0000 7fbcf0ee5640 1 -- 172.21.15.59:0/3688268555 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce4030610 con 0x7fbcec06cab0 2024-03-06T07:30:12.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.195+0000 7fbcf0ee5640 1 -- 172.21.15.59:0/3688268555 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40309e0 con 0x7fbcec06cab0 2024-03-06T07:30:12.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.196+0000 7fbcf414b640 1 -- 172.21.15.59:0/3688268555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcec06cab0 msgr2=0x7fbcec0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.196+0000 7fbcf414b640 1 --2- 172.21.15.59:0/3688268555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcec06cab0 0x7fbcec0b41b0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fbce40097a0 tx=0x7fbce402b960 comp rx=0 tx=0).stop 2024-03-06T07:30:12.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.196+0000 7fbcf414b640 1 -- 172.21.15.59:0/3688268555 shutdown_connections 2024-03-06T07:30:12.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.196+0000 7fbcf414b640 1 --2- 172.21.15.59:0/3688268555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcec06cab0 0x7fbcec0b41b0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.196+0000 7fbcf414b640 1 -- 172.21.15.59:0/3688268555 >> 172.21.15.59:0/3688268555 conn(0x7fbcec0b3690 msgr2=0x7fbcec0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.196+0000 7fbcf414b640 1 -- 172.21.15.59:0/3688268555 shutdown_connections 2024-03-06T07:30:12.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.196+0000 7fbcf414b640 1 -- 172.21.15.59:0/3688268555 wait complete. 2024-03-06T07:30:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.197+0000 7fbcf414b640 1 Processor -- start 2024-03-06T07:30:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.197+0000 7fbcf414b640 1 -- start start 2024-03-06T07:30:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf414b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcec06cab0 0x7fbcec0c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf414b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcec0cea50 0x7fbcec1f4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf414b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcec15def0 0x7fbcec09f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf414b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec15e2d0 con 0x7fbcec0cea50 2024-03-06T07:30:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf414b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec0cc760 con 0x7fbcec15def0 2024-03-06T07:30:12.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf414b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec15e450 con 0x7fbcec06cab0 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf16e6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcec0cea50 0x7fbcec1f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf16e6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcec0cea50 0x7fbcec1f4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45528/0 (socket says 172.21.15.59:45528) 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf16e6640 1 -- 172.21.15.59:0/3860621669 learned_addr learned my addr 172.21.15.59:0/3860621669 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf1ee7640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcec06cab0 0x7fbcec0c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf26e8640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcec15def0 0x7fbcec09f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf16e6640 1 -- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcec06cab0 msgr2=0x7fbcec0c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf16e6640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcec06cab0 0x7fbcec0c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf16e6640 1 -- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcec15def0 msgr2=0x7fbcec09f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf16e6640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcec15def0 0x7fbcec09f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf16e6640 1 -- 172.21.15.59:0/3860621669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec1f4da0 con 0x7fbcec0cea50 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf26e8640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcec15def0 0x7fbcec09f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf16e6640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcec0cea50 0x7fbcec1f4860 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fbce000c4a0 tx=0x7fbce000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcde7fc640 1 -- 172.21.15.59:0/3860621669 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0018070 con 0x7fbcec0cea50 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcde7fc640 1 -- 172.21.15.59:0/3860621669 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce00074e0 con 0x7fbcec0cea50 2024-03-06T07:30:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf414b640 1 -- 172.21.15.59:0/3860621669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcec09f720 con 0x7fbcec0cea50 2024-03-06T07:30:12.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcde7fc640 1 -- 172.21.15.59:0/3860621669 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0004a20 con 0x7fbcec0cea50 2024-03-06T07:30:12.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.198+0000 7fbcf414b640 1 -- 172.21.15.59:0/3860621669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcec0c5e00 con 0x7fbcec0cea50 2024-03-06T07:30:12.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.199+0000 7fbcf414b640 1 -- 172.21.15.59:0/3860621669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcec161260 con 0x7fbcec0cea50 2024-03-06T07:30:12.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.200+0000 7fbcde7fc640 1 -- 172.21.15.59:0/3860621669 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbce001b070 con 0x7fbcec0cea50 2024-03-06T07:30:12.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.200+0000 7fbcde7fc640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcd006b4d0 0x7fbcd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.200+0000 7fbcde7fc640 1 -- 172.21.15.59:0/3860621669 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fbce0089200 con 0x7fbcec0cea50 2024-03-06T07:30:12.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.200+0000 7fbcf1ee7640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcd006b4d0 0x7fbcd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.201+0000 7fbcf1ee7640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcd006b4d0 0x7fbcd006d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fbcec1f19d0 tx=0x7fbce8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.205+0000 7fbcde7fc640 1 -- 172.21.15.59:0/3860621669 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbcec161260 con 0x7fbcec0cea50 2024-03-06T07:30:12.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:12.387+0000 7fbcf414b640 1 -- 172.21.15.59:0/3860621669 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbcec09fcf0 con 0x7fbcec0cea50 2024-03-06T07:30:13.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.073+0000 7fbcde7fc640 1 -- 172.21.15.59:0/3860621669 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v137) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbcec1f4da0 con 0x7fbcec0cea50 2024-03-06T07:30:13.075 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' created 2024-03-06T07:30:13.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 -- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcd006b4d0 msgr2=0x7fbcd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcd006b4d0 0x7fbcd006d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fbcec1f19d0 tx=0x7fbce8009040 comp rx=0 tx=0).stop 2024-03-06T07:30:13.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 -- 172.21.15.59:0/3860621669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcec0cea50 msgr2=0x7fbcec1f4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcec0cea50 0x7fbcec1f4860 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fbce000c4a0 tx=0x7fbce000c960 comp rx=0 tx=0).stop 2024-03-06T07:30:13.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 -- 172.21.15.59:0/3860621669 shutdown_connections 2024-03-06T07:30:13.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbcd006b4d0 0x7fbcd006d990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcec15def0 0x7fbcec09f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcec0cea50 0x7fbcec1f4860 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 --2- 172.21.15.59:0/3860621669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcec06cab0 0x7fbcec0c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 -- 172.21.15.59:0/3860621669 >> 172.21.15.59:0/3860621669 conn(0x7fbcec0b3690 msgr2=0x7fbcec160b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 -- 172.21.15.59:0/3860621669 shutdown_connections 2024-03-06T07:30:13.080 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.076+0000 7fbcf414b640 1 -- 172.21.15.59:0/3860621669 wait complete. 2024-03-06T07:30:13.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:30:13.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.211+0000 7f626a023640 1 Processor -- start 2024-03-06T07:30:13.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.211+0000 7f626a023640 1 -- start start 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.211+0000 7f626a023640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f626415e0b0 0x7f62640b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.211+0000 7f626a023640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62640b4290 con 0x7f62640b3df0 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.211+0000 7f626a023640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62640b4570 con 0x7f62640d2a90 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.211+0000 7f626a023640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62640ce990 con 0x7f626415e0b0 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.212+0000 7f62637fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f626415e0b0 0x7f62640b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.212+0000 7f62637fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f626415e0b0 0x7f62640b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45532/0 (socket says 172.21.15.59:45532) 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.212+0000 7f62637fe640 1 -- 172.21.15.59:0/2091162103 learned_addr learned my addr 172.21.15.59:0/2091162103 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.212+0000 7f6262ffd640 1 --1- 172.21.15.59:0/2091162103 >> v1:172.21.15.59:6789/0 conn(0x7f62640b3df0 0x7f62640d2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55302/0 (socket says 172.21.15.59:55302) 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.212+0000 7f62637fe640 1 -- 172.21.15.59:0/2091162103 >> v1:172.21.15.164:6789/0 conn(0x7f62640d2a90 legacy=0x7f62640cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.212+0000 7f62637fe640 1 -- 172.21.15.59:0/2091162103 >> v1:172.21.15.59:6789/0 conn(0x7f62640b3df0 legacy=0x7f62640d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.212+0000 7f62627fc640 1 -- 172.21.15.59:0/2091162103 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853120727 0 0) 0x7f62640b4290 con 0x7f62640b3df0 2024-03-06T07:30:13.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.212+0000 7f62637fe640 1 -- 172.21.15.59:0/2091162103 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62640b4850 con 0x7f626415e0b0 2024-03-06T07:30:13.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.213+0000 7f62637fe640 1 --2- 172.21.15.59:0/2091162103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f626415e0b0 0x7f62640b38b0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f6254009870 tx=0x7f625402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c702c02c6f25aefd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.213+0000 7f62627fc640 1 -- 172.21.15.59:0/2091162103 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625402f840 con 0x7f626415e0b0 2024-03-06T07:30:13.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.213+0000 7f62627fc640 1 -- 172.21.15.59:0/2091162103 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625402f9e0 con 0x7f626415e0b0 2024-03-06T07:30:13.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.213+0000 7f62627fc640 1 -- 172.21.15.59:0/2091162103 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625402fdb0 con 0x7f626415e0b0 2024-03-06T07:30:13.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.213+0000 7f626a023640 1 -- 172.21.15.59:0/2091162103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f626415e0b0 msgr2=0x7f62640b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.213+0000 7f626a023640 1 --2- 172.21.15.59:0/2091162103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f626415e0b0 0x7f62640b38b0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f6254009870 tx=0x7f625402ba20 comp rx=0 tx=0).stop 2024-03-06T07:30:13.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.213+0000 7f626a023640 1 -- 172.21.15.59:0/2091162103 shutdown_connections 2024-03-06T07:30:13.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.213+0000 7f626a023640 1 --2- 172.21.15.59:0/2091162103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f626415e0b0 0x7f62640b38b0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.213+0000 7f626a023640 1 -- 172.21.15.59:0/2091162103 >> 172.21.15.59:0/2091162103 conn(0x7f626408d6f0 msgr2=0x7f626409d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:13.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.213+0000 7f626a023640 1 -- 172.21.15.59:0/2091162103 shutdown_connections 2024-03-06T07:30:13.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.213+0000 7f626a023640 1 -- 172.21.15.59:0/2091162103 wait complete. 2024-03-06T07:30:13.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.214+0000 7f626a023640 1 Processor -- start 2024-03-06T07:30:13.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.215+0000 7f626a023640 1 -- start start 2024-03-06T07:30:13.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.215+0000 7f626a023640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62640b3df0 0x7f626414e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.215+0000 7f626a023640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62640d2a90 0x7f62640cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.215+0000 7f626a023640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f626415e0b0 0x7f62640b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.215+0000 7f626a023640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62640d1cd0 con 0x7f62640b3df0 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.215+0000 7f626a023640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626415e490 con 0x7f62640d2a90 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.215+0000 7f626a023640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62640d1e50 con 0x7f626415e0b0 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.215+0000 7f62637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62640b3df0 0x7f626414e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f62637fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62640b3df0 0x7f626414e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45534/0 (socket says 172.21.15.59:45534) 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f62637fe640 1 -- 172.21.15.59:0/189295571 learned_addr learned my addr 172.21.15.59:0/189295571 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f6262ffd640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62640d2a90 0x7f62640cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f6263fff640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f626415e0b0 0x7f62640b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f62637fe640 1 -- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f626415e0b0 msgr2=0x7f62640b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f62637fe640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f626415e0b0 0x7f62640b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f62637fe640 1 -- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62640d2a90 msgr2=0x7f62640cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f62637fe640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62640d2a90 0x7f62640cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:13.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f62637fe640 1 -- 172.21.15.59:0/189295571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626415f250 con 0x7f62640b3df0 2024-03-06T07:30:13.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f6263fff640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f626415e0b0 0x7f62640b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:13.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f62637fe640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62640b3df0 0x7f626414e0b0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f6254000c00 tx=0x7f6254004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f624bfff640 1 -- 172.21.15.59:0/189295571 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254041400 con 0x7f62640b3df0 2024-03-06T07:30:13.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f624bfff640 1 -- 172.21.15.59:0/189295571 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6254046070 con 0x7f62640b3df0 2024-03-06T07:30:13.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f624bfff640 1 -- 172.21.15.59:0/189295571 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254041800 con 0x7f62640b3df0 2024-03-06T07:30:13.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f626a023640 1 -- 172.21.15.59:0/189295571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626415f530 con 0x7f62640b3df0 2024-03-06T07:30:13.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.216+0000 7f626a023640 1 -- 172.21.15.59:0/189295571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626415faf0 con 0x7f62640b3df0 2024-03-06T07:30:13.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.217+0000 7f626a023640 1 -- 172.21.15.59:0/189295571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626404f330 con 0x7f62640b3df0 2024-03-06T07:30:13.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.218+0000 7f624bfff640 1 -- 172.21.15.59:0/189295571 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6254040070 con 0x7f62640b3df0 2024-03-06T07:30:13.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.218+0000 7f624bfff640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f625006b4d0 0x7f625006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:13.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.218+0000 7f624bfff640 1 -- 172.21.15.59:0/189295571 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f62540ac860 con 0x7f62640b3df0 2024-03-06T07:30:13.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.218+0000 7f6262ffd640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f625006b4d0 0x7f625006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:13.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.219+0000 7f6262ffd640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f625006b4d0 0x7f625006d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6258007f70 tx=0x7f6258007f00 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:13.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.223+0000 7f624bfff640 1 -- 172.21.15.59:0/189295571 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6254033030 con 0x7f62640b3df0 2024-03-06T07:30:13.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:13.444+0000 7f626a023640 1 -- 172.21.15.59:0/189295571 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f62640d2160 con 0x7f62640b3df0 2024-03-06T07:30:14.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.074+0000 7f624bfff640 1 -- 172.21.15.59:0/189295571 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v95) v1 ==== 174+0+0 (secure 0 0 0) 0x7f625407b8f0 con 0x7f62640b3df0 2024-03-06T07:30:14.077 INFO:teuthology.orchestra.run.smithi059.stderr:new fs with metadata pool 20 and data pool 21 2024-03-06T07:30:14.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.080+0000 7f626a023640 1 -- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f625006b4d0 msgr2=0x7f625006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.080+0000 7f626a023640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f625006b4d0 0x7f625006d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6258007f70 tx=0x7f6258007f00 comp rx=0 tx=0).stop 2024-03-06T07:30:14.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.080+0000 7f626a023640 1 -- 172.21.15.59:0/189295571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62640b3df0 msgr2=0x7f626414e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.080+0000 7f626a023640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62640b3df0 0x7f626414e0b0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f6254000c00 tx=0x7f6254004730 comp rx=0 tx=0).stop 2024-03-06T07:30:14.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.082+0000 7f626a023640 1 -- 172.21.15.59:0/189295571 shutdown_connections 2024-03-06T07:30:14.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.082+0000 7f626a023640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f625006b4d0 0x7f625006d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.082+0000 7f626a023640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f626415e0b0 0x7f62640b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.082+0000 7f626a023640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62640d2a90 0x7f62640cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.082+0000 7f626a023640 1 --2- 172.21.15.59:0/189295571 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62640b3df0 0x7f626414e0b0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.084 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.082+0000 7f626a023640 1 -- 172.21.15.59:0/189295571 >> 172.21.15.59:0/189295571 conn(0x7f626408d6f0 msgr2=0x7f62640b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.083+0000 7f626a023640 1 -- 172.21.15.59:0/189295571 shutdown_connections 2024-03-06T07:30:14.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.086+0000 7f626a023640 1 -- 172.21.15.59:0/189295571 wait complete. 2024-03-06T07:30:14.115 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:14.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.219+0000 7fbeda9d7640 1 Processor -- start 2024-03-06T07:30:14.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.219+0000 7fbeda9d7640 1 -- start start 2024-03-06T07:30:14.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.219+0000 7fbeda9d7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbed415dfe0 0x7fbed40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.219+0000 7fbeda9d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40cf830 0x7fbed40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.219+0000 7fbeda9d7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed40b3430 con 0x7fbed40d27f0 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.219+0000 7fbeda9d7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed40d2c50 con 0x7fbed415dfe0 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.219+0000 7fbeda9d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40b31e0 con 0x7fbed40cf830 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.220+0000 7fbed3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbed415dfe0 0x7fbed40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.220+0000 7fbed8f74640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbed40d27f0 0x7fbed40d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41508/0 (socket says 172.21.15.59:41508) 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.220+0000 7fbed3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbed415dfe0 0x7fbed40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36280/0 (socket says 172.21.15.59:36280) 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.220+0000 7fbed8f74640 1 -- 172.21.15.59:0/1404241829 learned_addr learned my addr 172.21.15.59:0/1404241829 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.220+0000 7fbed37fe640 1 --2- 172.21.15.59:0/1404241829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40cf830 0x7fbed40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.220+0000 7fbed3fff640 1 -- 172.21.15.59:0/1404241829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40cf830 msgr2=0x7fbed40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.220+0000 7fbed3fff640 1 --2- 172.21.15.59:0/1404241829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40cf830 0x7fbed40d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.220+0000 7fbed3fff640 1 -- 172.21.15.59:0/1404241829 >> v1:172.21.15.59:6789/0 conn(0x7fbed40d27f0 legacy=0x7fbed40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.220+0000 7fbed3fff640 1 -- 172.21.15.59:0/1404241829 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed40b3710 con 0x7fbed415dfe0 2024-03-06T07:30:14.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.220+0000 7fbed2ffd640 1 -- 172.21.15.59:0/1404241829 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45230421 0 0) 0x7fbed40b3430 con 0x7fbed40d27f0 2024-03-06T07:30:14.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbed3fff640 1 --2- 172.21.15.59:0/1404241829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed415dfe0 0x7fbed40cf2f0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fbec40097a0 tx=0x7fbec402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81439808f4589c17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbed2ffd640 1 -- 172.21.15.59:0/1404241829 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec4035020 con 0x7fbed415dfe0 2024-03-06T07:30:14.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbed2ffd640 1 -- 172.21.15.59:0/1404241829 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec4031830 con 0x7fbed415dfe0 2024-03-06T07:30:14.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbed2ffd640 1 -- 172.21.15.59:0/1404241829 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec4031c00 con 0x7fbed415dfe0 2024-03-06T07:30:14.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbeda9d7640 1 -- 172.21.15.59:0/1404241829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed415dfe0 msgr2=0x7fbed40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbeda9d7640 1 --2- 172.21.15.59:0/1404241829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed415dfe0 0x7fbed40cf2f0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fbec40097a0 tx=0x7fbec402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbeda9d7640 1 -- 172.21.15.59:0/1404241829 shutdown_connections 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbeda9d7640 1 --2- 172.21.15.59:0/1404241829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40cf830 0x7fbed40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbeda9d7640 1 --2- 172.21.15.59:0/1404241829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed415dfe0 0x7fbed40cf2f0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbeda9d7640 1 -- 172.21.15.59:0/1404241829 >> 172.21.15.59:0/1404241829 conn(0x7fbed40d3cd0 msgr2=0x7fbed40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbeda9d7640 1 -- 172.21.15.59:0/1404241829 shutdown_connections 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.221+0000 7fbeda9d7640 1 -- 172.21.15.59:0/1404241829 wait complete. 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.222+0000 7fbeda9d7640 1 Processor -- start 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.223+0000 7fbeda9d7640 1 -- start start 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.223+0000 7fbeda9d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40cf830 0x7fbed41f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.223+0000 7fbeda9d7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40d27f0 0x7fbed4160820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.223+0000 7fbeda9d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed415dfe0 0x7fbed40b9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.223+0000 7fbeda9d7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40d1c70 con 0x7fbed415dfe0 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.223+0000 7fbeda9d7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40b3d60 con 0x7fbed40d27f0 2024-03-06T07:30:14.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.223+0000 7fbeda9d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed40d1df0 con 0x7fbed40cf830 2024-03-06T07:30:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.223+0000 7fbed8f74640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed415dfe0 0x7fbed40b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed8f74640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed415dfe0 0x7fbed40b9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36296/0 (socket says 172.21.15.59:36296) 2024-03-06T07:30:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed8f74640 1 -- 172.21.15.59:0/721893595 learned_addr learned my addr 172.21.15.59:0/721893595 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed37fe640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40d27f0 0x7fbed4160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed3fff640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40cf830 0x7fbed41f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed8f74640 1 -- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40cf830 msgr2=0x7fbed41f6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed8f74640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40cf830 0x7fbed41f6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed8f74640 1 -- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40d27f0 msgr2=0x7fbed4160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed8f74640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40d27f0 0x7fbed4160820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed8f74640 1 -- 172.21.15.59:0/721893595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed415f4a0 con 0x7fbed415dfe0 2024-03-06T07:30:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed8f74640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed415dfe0 0x7fbed40b9390 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fbecc00c7a0 tx=0x7fbecc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed0ff9640 1 -- 172.21.15.59:0/721893595 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc018070 con 0x7fbed415dfe0 2024-03-06T07:30:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed0ff9640 1 -- 172.21.15.59:0/721893595 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbecc00a640 con 0x7fbed415dfe0 2024-03-06T07:30:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbeda9d7640 1 -- 172.21.15.59:0/721893595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed415f780 con 0x7fbed415dfe0 2024-03-06T07:30:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbed0ff9640 1 -- 172.21.15.59:0/721893595 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc00aab0 con 0x7fbed415dfe0 2024-03-06T07:30:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.224+0000 7fbeda9d7640 1 -- 172.21.15.59:0/721893595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed415fd40 con 0x7fbed415dfe0 2024-03-06T07:30:14.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.225+0000 7fbeda9d7640 1 -- 172.21.15.59:0/721893595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed404f330 con 0x7fbed415dfe0 2024-03-06T07:30:14.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.226+0000 7fbed0ff9640 1 -- 172.21.15.59:0/721893595 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbecc01b070 con 0x7fbed415dfe0 2024-03-06T07:30:14.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.226+0000 7fbed0ff9640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbec006b500 0x7fbec006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.226+0000 7fbed0ff9640 1 -- 172.21.15.59:0/721893595 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbecc0897b0 con 0x7fbed415dfe0 2024-03-06T07:30:14.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.226+0000 7fbed3fff640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbec006b500 0x7fbec006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.227+0000 7fbed3fff640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbec006b500 0x7fbec006d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fbec4036000 tx=0x7fbec4008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.231+0000 7fbed0ff9640 1 -- 172.21.15.59:0/721893595 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbecc058630 con 0x7fbed415dfe0 2024-03-06T07:30:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.411+0000 7fbeda9d7640 1 -- 172.21.15.59:0/721893595 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbed40d2350 con 0x7fbed415dfe0 2024-03-06T07:30:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.412+0000 7fbed0ff9640 1 -- 172.21.15.59:0/721893595 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16680 (secure 0 0 0) 0x7fbecc057f30 con 0x7fbed415dfe0 2024-03-06T07:30:14.414 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:14.414 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":138,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:13.445943+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:11.712614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:12.406781+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 -- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbec006b500 msgr2=0x7fbec006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbec006b500 0x7fbec006d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fbec4036000 tx=0x7fbec4008030 comp rx=0 tx=0).stop 2024-03-06T07:30:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 -- 172.21.15.59:0/721893595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed415dfe0 msgr2=0x7fbed40b9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed415dfe0 0x7fbed40b9390 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fbecc00c7a0 tx=0x7fbecc00cc60 comp rx=0 tx=0).stop 2024-03-06T07:30:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 -- 172.21.15.59:0/721893595 shutdown_connections 2024-03-06T07:30:14.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbec006b500 0x7fbec006d9c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbed415dfe0 0x7fbed40b9390 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40d27f0 0x7fbed4160820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 --2- 172.21.15.59:0/721893595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40cf830 0x7fbed41f6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 -- 172.21.15.59:0/721893595 >> 172.21.15.59:0/721893595 conn(0x7fbed40d3cd0 msgr2=0x7fbed40d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 -- 172.21.15.59:0/721893595 shutdown_connections 2024-03-06T07:30:14.417 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.414+0000 7fbeda9d7640 1 -- 172.21.15.59:0/721893595 wait complete. 2024-03-06T07:30:14.427 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:14.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.527+0000 7f50990fb640 1 Processor -- start 2024-03-06T07:30:14.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.527+0000 7f50990fb640 1 -- start start 2024-03-06T07:30:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f50990fb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f50940d00c0 0x7f50940d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f50990fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f509406cf20 0x7f509406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f50990fb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50940d09e0 con 0x7f50940cf850 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f50990fb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509406ca90 con 0x7f50940d00c0 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f50990fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509415e380 con 0x7f509406cf20 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f5092d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f50940d00c0 0x7f50940d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f509359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f509406cf20 0x7f509406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f5092d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f50940d00c0 0x7f50940d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:47568/0 (socket says 172.21.15.59:47568) 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f5092d9d640 1 -- 172.21.15.59:0/2741042093 learned_addr learned my addr 172.21.15.59:0/2741042093 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f508bfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f50940cf850 0x7f50940cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:39322/0 (socket says 172.21.15.59:39322) 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f5092d9d640 1 -- 172.21.15.59:0/2741042093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f509406cf20 msgr2=0x7f509406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f5092d9d640 1 --2- 172.21.15.59:0/2741042093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f509406cf20 0x7f509406d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f5092d9d640 1 -- 172.21.15.59:0/2741042093 >> v1:172.21.15.164:6789/0 conn(0x7f50940cf850 legacy=0x7f50940cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f5092d9d640 1 -- 172.21.15.59:0/2741042093 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50940cdc10 con 0x7f50940d00c0 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f509259c640 1 -- 172.21.15.59:0/2741042093 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213978284 0 0) 0x7f50940d09e0 con 0x7f50940cf850 2024-03-06T07:30:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.528+0000 7f509359e640 1 --2- 172.21.15.59:0/2741042093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f509406cf20 0x7f509406d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.529+0000 7f5092d9d640 1 --2- 172.21.15.59:0/2741042093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50940d00c0 0x7f50940d04a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5084009870 tx=0x7f508402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba31c1efc8a2ab8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.529+0000 7f509259c640 1 -- 172.21.15.59:0/2741042093 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084034070 con 0x7f50940d00c0 2024-03-06T07:30:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.529+0000 7f509259c640 1 -- 172.21.15.59:0/2741042093 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5084030840 con 0x7f50940d00c0 2024-03-06T07:30:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.529+0000 7f509259c640 1 -- 172.21.15.59:0/2741042093 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084030c10 con 0x7f50940d00c0 2024-03-06T07:30:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.530+0000 7f50990fb640 1 -- 172.21.15.59:0/2741042093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50940d00c0 msgr2=0x7f50940d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.530+0000 7f50990fb640 1 --2- 172.21.15.59:0/2741042093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50940d00c0 0x7f50940d04a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5084009870 tx=0x7f508402b980 comp rx=0 tx=0).stop 2024-03-06T07:30:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.530+0000 7f50990fb640 1 -- 172.21.15.59:0/2741042093 shutdown_connections 2024-03-06T07:30:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.530+0000 7f50990fb640 1 --2- 172.21.15.59:0/2741042093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f509406cf20 0x7f509406d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.530+0000 7f50990fb640 1 --2- 172.21.15.59:0/2741042093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50940d00c0 0x7f50940d04a0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.530+0000 7f50990fb640 1 -- 172.21.15.59:0/2741042093 >> 172.21.15.59:0/2741042093 conn(0x7f50940b35a0 msgr2=0x7f50940b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.530+0000 7f50990fb640 1 -- 172.21.15.59:0/2741042093 shutdown_connections 2024-03-06T07:30:14.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.530+0000 7f50990fb640 1 -- 172.21.15.59:0/2741042093 wait complete. 2024-03-06T07:30:14.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.531+0000 7f50990fb640 1 Processor -- start 2024-03-06T07:30:14.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.531+0000 7f50990fb640 1 -- start start 2024-03-06T07:30:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f50990fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509406cf20 0x7f509415e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f50990fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50940cf850 0x7f50940b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f50990fb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50940d00c0 0x7f50940c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f50990fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509406c720 con 0x7f509406cf20 2024-03-06T07:30:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f50990fb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509406c440 con 0x7f50940d00c0 2024-03-06T07:30:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f50990fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50940b1030 con 0x7f50940cf850 2024-03-06T07:30:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f5092d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509406cf20 0x7f509415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f5092d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509406cf20 0x7f509415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36310/0 (socket says 172.21.15.59:36310) 2024-03-06T07:30:14.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f5092d9d640 1 -- 172.21.15.59:0/3836631274 learned_addr learned my addr 172.21.15.59:0/3836631274 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f508bfff640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50940cf850 0x7f50940b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f509359e640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50940d00c0 0x7f50940c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f5092d9d640 1 -- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50940cf850 msgr2=0x7f50940b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f5092d9d640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50940cf850 0x7f50940b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f5092d9d640 1 -- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50940d00c0 msgr2=0x7f50940c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f5092d9d640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50940d00c0 0x7f50940c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f5092d9d640 1 -- 172.21.15.59:0/3836631274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509415ede0 con 0x7f509406cf20 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.532+0000 7f509359e640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50940d00c0 0x7f50940c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.533+0000 7f5092d9d640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509406cf20 0x7f509415e590 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f5084000c00 tx=0x7f50840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.533+0000 7f508b7fe640 1 -- 172.21.15.59:0/3836631274 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084034040 con 0x7f509406cf20 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.533+0000 7f508b7fe640 1 -- 172.21.15.59:0/3836631274 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5084030640 con 0x7f509406cf20 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.533+0000 7f50990fb640 1 -- 172.21.15.59:0/3836631274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509415f0c0 con 0x7f509406cf20 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.533+0000 7f508b7fe640 1 -- 172.21.15.59:0/3836631274 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084039d40 con 0x7f509406cf20 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.533+0000 7f50990fb640 1 -- 172.21.15.59:0/3836631274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509415f680 con 0x7f509406cf20 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.534+0000 7f50990fb640 1 -- 172.21.15.59:0/3836631274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509404f330 con 0x7f509406cf20 2024-03-06T07:30:14.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.534+0000 7f508b7fe640 1 -- 172.21.15.59:0/3836631274 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f508404b060 con 0x7f509406cf20 2024-03-06T07:30:14.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.535+0000 7f508b7fe640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f507406b500 0x7f507406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.535+0000 7f508b7fe640 1 -- 172.21.15.59:0/3836631274 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50840abed0 con 0x7f509406cf20 2024-03-06T07:30:14.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.535+0000 7f508bfff640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f507406b500 0x7f507406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.535+0000 7f508bfff640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f507406b500 0x7f507406d9c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f50940b3360 tx=0x7f507c007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.539+0000 7f508b7fe640 1 -- 172.21.15.59:0/3836631274 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f508407ade0 con 0x7f509406cf20 2024-03-06T07:30:14.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.721+0000 7f50990fb640 1 -- 172.21.15.59:0/3836631274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50940b1ba0 con 0x7f509406cf20 2024-03-06T07:30:14.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.722+0000 7f508b7fe640 1 -- 172.21.15.59:0/3836631274 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f508407a6e0 con 0x7f509406cf20 2024-03-06T07:30:14.724 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:14.724 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":138,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:13.445943+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:11.712614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:12.406781+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:14.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 -- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f507406b500 msgr2=0x7f507406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f507406b500 0x7f507406d9c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f50940b3360 tx=0x7f507c007d50 comp rx=0 tx=0).stop 2024-03-06T07:30:14.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 -- 172.21.15.59:0/3836631274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509406cf20 msgr2=0x7f509415e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509406cf20 0x7f509415e590 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f5084000c00 tx=0x7f50840046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 -- 172.21.15.59:0/3836631274 shutdown_connections 2024-03-06T07:30:14.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f507406b500 0x7f507406d9c0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50940d00c0 0x7f50940c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50940cf850 0x7f50940b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 --2- 172.21.15.59:0/3836631274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f509406cf20 0x7f509415e590 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 -- 172.21.15.59:0/3836631274 >> 172.21.15.59:0/3836631274 conn(0x7f50940b35a0 msgr2=0x7f50940b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 -- 172.21.15.59:0/3836631274 shutdown_connections 2024-03-06T07:30:14.727 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.724+0000 7f50990fb640 1 -- 172.21.15.59:0/3836631274 wait complete. 2024-03-06T07:30:14.737 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:30:14.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.839+0000 7f36d0179640 1 Processor -- start 2024-03-06T07:30:14.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.839+0000 7f36d0179640 1 -- start start 2024-03-06T07:30:14.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36d0179640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f36c815bdd0 0x7f36c80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36d0179640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f36c80ce300 0x7f36c80ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36d0179640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c80d0240 con 0x7f36c80cecd0 2024-03-06T07:30:14.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36d0179640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c80cf4f0 con 0x7f36c815bdd0 2024-03-06T07:30:14.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36d0179640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c80d0850 con 0x7f36c80ce300 2024-03-06T07:30:14.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36cdf15640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f36c815bdd0 0x7f36c80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36cdf15640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f36c815bdd0 0x7f36c80b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36314/0 (socket says 172.21.15.59:36314) 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36cdf15640 1 -- 172.21.15.59:0/3874949248 learned_addr learned my addr 172.21.15.59:0/3874949248 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36ce716640 1 --1- 172.21.15.59:0/3874949248 >> v1:172.21.15.59:6789/0 conn(0x7f36c80cecd0 0x7f36c80cc5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41514/0 (socket says 172.21.15.59:41514) 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36cdf15640 1 -- 172.21.15.59:0/3874949248 >> v2:172.21.15.164:3300/0 conn(0x7f36c80ce300 msgr2=0x7f36c80ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36cd714640 1 --2- 172.21.15.59:0/3874949248 >> v2:172.21.15.164:3300/0 conn(0x7f36c80ce300 0x7f36c80ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36ccf13640 1 -- 172.21.15.59:0/3874949248 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790331615 0 0) 0x7f36c80d0240 con 0x7f36c80cecd0 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36cdf15640 1 --2- 172.21.15.59:0/3874949248 >> v2:172.21.15.164:3300/0 conn(0x7f36c80ce300 0x7f36c80ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36cdf15640 1 -- 172.21.15.59:0/3874949248 >> v1:172.21.15.59:6789/0 conn(0x7f36c80cecd0 legacy=0x7f36c80cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36cdf15640 1 -- 172.21.15.59:0/3874949248 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c80cf210 con 0x7f36c815bdd0 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.840+0000 7f36cd714640 1 --2- 172.21.15.59:0/3874949248 >> v2:172.21.15.164:3300/0 conn(0x7f36c80ce300 0x7f36c80ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.841+0000 7f36cdf15640 1 --2- 172.21.15.59:0/3874949248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c815bdd0 0x7f36c80b41b0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f36c80b1470 tx=0x7f36c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c725f931ff827e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.841+0000 7f36ccf13640 1 -- 172.21.15.59:0/3874949248 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0035020 con 0x7f36c815bdd0 2024-03-06T07:30:14.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.841+0000 7f36ccf13640 1 -- 172.21.15.59:0/3874949248 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0031870 con 0x7f36c815bdd0 2024-03-06T07:30:14.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.841+0000 7f36ccf13640 1 -- 172.21.15.59:0/3874949248 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0031c40 con 0x7f36c815bdd0 2024-03-06T07:30:14.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.842+0000 7f36d0179640 1 -- 172.21.15.59:0/3874949248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c815bdd0 msgr2=0x7f36c80b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.842+0000 7f36d0179640 1 --2- 172.21.15.59:0/3874949248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c815bdd0 0x7f36c80b41b0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f36c80b1470 tx=0x7f36c002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:14.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.842+0000 7f36d0179640 1 -- 172.21.15.59:0/3874949248 shutdown_connections 2024-03-06T07:30:14.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.842+0000 7f36d0179640 1 --2- 172.21.15.59:0/3874949248 >> v2:172.21.15.164:3300/0 conn(0x7f36c80ce300 0x7f36c80ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.842+0000 7f36d0179640 1 --2- 172.21.15.59:0/3874949248 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c815bdd0 0x7f36c80b41b0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.842+0000 7f36d0179640 1 -- 172.21.15.59:0/3874949248 >> 172.21.15.59:0/3874949248 conn(0x7f36c80b3690 msgr2=0x7f36c80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.842+0000 7f36d0179640 1 -- 172.21.15.59:0/3874949248 shutdown_connections 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.842+0000 7f36d0179640 1 -- 172.21.15.59:0/3874949248 wait complete. 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.843+0000 7f36d0179640 1 Processor -- start 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.843+0000 7f36d0179640 1 -- start start 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36d0179640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36c80ce300 0x7f36c80c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36d0179640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c80cecd0 0x7f36c806c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36d0179640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36c815bdd0 0x7f36c806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36d0179640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80b10c0 con 0x7f36c80cecd0 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36d0179640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c815c1b0 con 0x7f36c815bdd0 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36d0179640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80b1240 con 0x7f36c80ce300 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36cd714640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c80cecd0 0x7f36c806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36cd714640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c80cecd0 0x7f36c806c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36330/0 (socket says 172.21.15.59:36330) 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36cd714640 1 -- 172.21.15.59:0/3678173850 learned_addr learned my addr 172.21.15.59:0/3678173850 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36cdf15640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36c80ce300 0x7f36c80c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36ce716640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36c815bdd0 0x7f36c806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36cd714640 1 -- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36c80ce300 msgr2=0x7f36c80c55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36cd714640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36c80ce300 0x7f36c80c55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36cd714640 1 -- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36c815bdd0 msgr2=0x7f36c806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36cd714640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36c815bdd0 0x7f36c806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36cd714640 1 -- 172.21.15.59:0/3678173850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c806cff0 con 0x7f36c80cecd0 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36cd714640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c80cecd0 0x7f36c806c280 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00c7f0 tx=0x7f36bc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.844+0000 7f36c67fc640 1 -- 172.21.15.59:0/3678173850 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc015070 con 0x7f36c80cecd0 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.845+0000 7f36c67fc640 1 -- 172.21.15.59:0/3678173850 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36bc007820 con 0x7f36c80cecd0 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.845+0000 7f36d0179640 1 -- 172.21.15.59:0/3678173850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c806d2d0 con 0x7f36c80cecd0 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.845+0000 7f36c67fc640 1 -- 172.21.15.59:0/3678173850 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc007bd0 con 0x7f36c80cecd0 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.845+0000 7f36d0179640 1 -- 172.21.15.59:0/3678173850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c806cd00 con 0x7f36c80cecd0 2024-03-06T07:30:14.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.845+0000 7f36d0179640 1 -- 172.21.15.59:0/3678173850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c80d1250 con 0x7f36c80cecd0 2024-03-06T07:30:14.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.846+0000 7f36c67fc640 1 -- 172.21.15.59:0/3678173850 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36bc018070 con 0x7f36c80cecd0 2024-03-06T07:30:14.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.846+0000 7f36c67fc640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36b406b4d0 0x7f36b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.846+0000 7f36c67fc640 1 -- 172.21.15.59:0/3678173850 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36bc08a2e0 con 0x7f36c80cecd0 2024-03-06T07:30:14.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.847+0000 7f36cdf15640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36b406b4d0 0x7f36b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.847+0000 7f36cdf15640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36b406b4d0 0x7f36b406d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f36c0036000 tx=0x7f36c0008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:14.851+0000 7f36c67fc640 1 -- 172.21.15.59:0/3678173850 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f36c80d1250 con 0x7f36c80cecd0 2024-03-06T07:30:15.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.069+0000 7f36d0179640 1 -- 172.21.15.59:0/3678173850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f36c809fcf0 con 0x7f36c80cecd0 2024-03-06T07:30:15.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.094+0000 7f36c67fc640 1 -- 172.21.15.59:0/3678173850 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v96) v1 ==== 118+0+0 (secure 0 0 0) 0x7f36bc0591f0 con 0x7f36c80cecd0 2024-03-06T07:30:15.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.096+0000 7f36d0179640 1 -- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36b406b4d0 msgr2=0x7f36b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.096+0000 7f36d0179640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36b406b4d0 0x7f36b406d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f36c0036000 tx=0x7f36c0008030 comp rx=0 tx=0).stop 2024-03-06T07:30:15.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.096+0000 7f36d0179640 1 -- 172.21.15.59:0/3678173850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c80cecd0 msgr2=0x7f36c806c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.096+0000 7f36d0179640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c80cecd0 0x7f36c806c280 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00c7f0 tx=0x7f36bc00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.096+0000 7f36d0179640 1 -- 172.21.15.59:0/3678173850 shutdown_connections 2024-03-06T07:30:15.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.096+0000 7f36d0179640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f36b406b4d0 0x7f36b406d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.096+0000 7f36d0179640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36c815bdd0 0x7f36c806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.096+0000 7f36d0179640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36c80cecd0 0x7f36c806c280 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.096+0000 7f36d0179640 1 --2- 172.21.15.59:0/3678173850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36c80ce300 0x7f36c80c55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.097+0000 7f36d0179640 1 -- 172.21.15.59:0/3678173850 >> 172.21.15.59:0/3678173850 conn(0x7f36c80b3690 msgr2=0x7f36c806f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.097+0000 7f36d0179640 1 -- 172.21.15.59:0/3678173850 shutdown_connections 2024-03-06T07:30:15.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.097+0000 7f36d0179640 1 -- 172.21.15.59:0/3678173850 wait complete. 2024-03-06T07:30:15.110 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:30:15.110 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.212+0000 7f6b880fb640 1 Processor -- start 2024-03-06T07:30:15.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.213+0000 7f6b880fb640 1 -- start start 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.213+0000 7f6b880fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801614d0 0x7f6b800d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.213+0000 7f6b880fb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6b800d1910 0x7f6b800d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.213+0000 7f6b880fb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b800d2740 con 0x7f6b800d22b0 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.213+0000 7f6b880fb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b801618b0 con 0x7f6b800d1910 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.213+0000 7f6b880fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800cf400 con 0x7f6b801614d0 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.213+0000 7f6b86698640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6b800d22b0 0x7f6b800d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41518/0 (socket says 172.21.15.59:41518) 2024-03-06T07:30:15.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b86698640 1 -- 172.21.15.59:0/1309432857 learned_addr learned my addr 172.21.15.59:0/1309432857 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268320465 0 0) 0x7f6b78001490 con 0x7f6b800d22b0 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b6c003520 con 0x7f6b800d22b0 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b85696640 1 --2- 172.21.15.59:0/1309432857 >> v2:172.21.15.164:3300/0 conn(0x7f6b800d1910 0x7f6b800d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b85e97640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801614d0 0x7f6b800d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2126932069 0 0) 0x7f6b6c003520 con 0x7f6b800d22b0 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801614d0 msgr2=0x7f6b800d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801614d0 0x7f6b800d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 >> v2:172.21.15.164:3300/0 conn(0x7f6b800d1910 msgr2=0x7f6b800d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 --2- 172.21.15.59:0/1309432857 >> v2:172.21.15.164:3300/0 conn(0x7f6b800d1910 0x7f6b800d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b800cf6a0 con 0x7f6b800d22b0 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f6b78001bc0 con 0x7f6b800d22b0 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 >> v1:172.21.15.59:6789/0 conn(0x7f6b800d22b0 legacy=0x7f6b800d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801614d0 0x7f6b6c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b6c003f30 0x7f6b6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b6c004d40 0x7f6b6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b801618b0 con 0x7f6b6c003f30 2024-03-06T07:30:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800cf400 con 0x7f6b6c004d40 2024-03-06T07:30:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c00b9c0 con 0x7f6b801614d0 2024-03-06T07:30:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b85696640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b6c003f30 0x7f6b6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b85e97640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801614d0 0x7f6b6c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.214+0000 7f6b86698640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b6c004d40 0x7f6b6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.215+0000 7f6b85696640 1 -- 172.21.15.59:0/1309432857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801614d0 msgr2=0x7f6b6c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.215+0000 7f6b85696640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801614d0 0x7f6b6c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.215+0000 7f6b85696640 1 -- 172.21.15.59:0/1309432857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b6c004d40 msgr2=0x7f6b6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.215+0000 7f6b85696640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b6c004d40 0x7f6b6c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.215+0000 7f6b85696640 1 -- 172.21.15.59:0/1309432857 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c003520 con 0x7f6b6c003f30 2024-03-06T07:30:15.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.216+0000 7f6b85696640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b6c003f30 0x7f6b6c004800 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009ac0 tx=0x7f6b6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87e8ba3560c7652 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.216+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70039070 con 0x7f6b6c003f30 2024-03-06T07:30:15.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.216+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7002c6d0 con 0x7f6b6c003f30 2024-03-06T07:30:15.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.216+0000 7f6b84e95640 1 -- 172.21.15.59:0/1309432857 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70009690 con 0x7f6b6c003f30 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.216+0000 7f6b880fb640 1 -- 172.21.15.59:0/1309432857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b6c003f30 msgr2=0x7f6b6c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.216+0000 7f6b880fb640 1 --2- 172.21.15.59:0/1309432857 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b6c003f30 0x7f6b6c004800 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009ac0 tx=0x7f6b6c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.216+0000 7f6b86698640 1 -- 172.21.15.59:0/1309432857 reap_dead start 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.216+0000 7f6b880fb640 1 -- 172.21.15.59:0/1309432857 shutdown_connections 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.216+0000 7f6b880fb640 1 -- 172.21.15.59:0/1309432857 >> 172.21.15.59:0/1309432857 conn(0x7f6b800d3d40 msgr2=0x7f6b800d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.217+0000 7f6b880fb640 1 -- 172.21.15.59:0/1309432857 shutdown_connections 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.217+0000 7f6b880fb640 1 -- 172.21.15.59:0/1309432857 wait complete. 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.218+0000 7f6b880fb640 1 Processor -- start 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.218+0000 7f6b880fb640 1 -- start start 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.218+0000 7f6b880fb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b6c003f30 0x7f6b8009da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.218+0000 7f6b880fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801f7f90 0x7f6b801542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.218+0000 7f6b880fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b800cae10 0x7f6b800c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.218+0000 7f6b880fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800b0910 con 0x7f6b800cae10 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.218+0000 7f6b880fb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800cfc60 con 0x7f6b6c003f30 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.218+0000 7f6b880fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800b1e40 con 0x7f6b801f7f90 2024-03-06T07:30:15.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b86698640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b800cae10 0x7f6b800c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b86698640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b800cae10 0x7f6b800c88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36348/0 (socket says 172.21.15.59:36348) 2024-03-06T07:30:15.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b86698640 1 -- 172.21.15.59:0/585438276 learned_addr learned my addr 172.21.15.59:0/585438276 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:15.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b85e97640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b6c003f30 0x7f6b8009da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b85696640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801f7f90 0x7f6b801542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b86698640 1 -- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801f7f90 msgr2=0x7f6b801542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b86698640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801f7f90 0x7f6b801542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b86698640 1 -- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b6c003f30 msgr2=0x7f6b8009da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b86698640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b6c003f30 0x7f6b8009da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b86698640 1 -- 172.21.15.59:0/585438276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b800cb660 con 0x7f6b800cae10 2024-03-06T07:30:15.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b86698640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b800cae10 0x7f6b800c88c0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f6b78002230 tx=0x7f6b78004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b767fc640 1 -- 172.21.15.59:0/585438276 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78017020 con 0x7f6b800cae10 2024-03-06T07:30:15.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b767fc640 1 -- 172.21.15.59:0/585438276 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b780078c0 con 0x7f6b800cae10 2024-03-06T07:30:15.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b767fc640 1 -- 172.21.15.59:0/585438276 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b780133f0 con 0x7f6b800cae10 2024-03-06T07:30:15.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b880fb640 1 -- 172.21.15.59:0/585438276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b800cb940 con 0x7f6b800cae10 2024-03-06T07:30:15.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.219+0000 7f6b880fb640 1 -- 172.21.15.59:0/585438276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b800cb290 con 0x7f6b800cae10 2024-03-06T07:30:15.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.221+0000 7f6b880fb640 1 -- 172.21.15.59:0/585438276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b80068ff0 con 0x7f6b800cae10 2024-03-06T07:30:15.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.221+0000 7f6b767fc640 1 -- 172.21.15.59:0/585438276 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b7801a020 con 0x7f6b800cae10 2024-03-06T07:30:15.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.221+0000 7f6b767fc640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b6406b500 0x7f6b6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.221+0000 7f6b767fc640 1 -- 172.21.15.59:0/585438276 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b7808a980 con 0x7f6b800cae10 2024-03-06T07:30:15.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.221+0000 7f6b85e97640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b6406b500 0x7f6b6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.222+0000 7f6b85e97640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b6406b500 0x7f6b6406d9c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009a20 tx=0x7f6b7c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.226+0000 7f6b767fc640 1 -- 172.21.15.59:0/585438276 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b78021020 con 0x7f6b800cae10 2024-03-06T07:30:15.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:15.474+0000 7f6b880fb640 1 -- 172.21.15.59:0/585438276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6b800d1100 con 0x7f6b800cae10 2024-03-06T07:30:16.065 INFO:tasks.ceph.mds.c.smithi059.stderr:2024-03-06T07:30:16.063+0000 7fa36cef1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:30:16.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.102+0000 7f6b767fc640 1 -- 172.21.15.59:0/585438276 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v97) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6b800d2740 con 0x7f6b800cae10 2024-03-06T07:30:16.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 -- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b6406b500 msgr2=0x7f6b6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b6406b500 0x7f6b6406d9c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009a20 tx=0x7f6b7c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:16.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 -- 172.21.15.59:0/585438276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b800cae10 msgr2=0x7f6b800c88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b800cae10 0x7f6b800c88c0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f6b78002230 tx=0x7f6b78004f20 comp rx=0 tx=0).stop 2024-03-06T07:30:16.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 -- 172.21.15.59:0/585438276 shutdown_connections 2024-03-06T07:30:16.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6b6406b500 0x7f6b6406d9c0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b800cae10 0x7f6b800c88c0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b801f7f90 0x7f6b801542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 --2- 172.21.15.59:0/585438276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b6c003f30 0x7f6b8009da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 -- 172.21.15.59:0/585438276 >> 172.21.15.59:0/585438276 conn(0x7f6b800d3d40 msgr2=0x7f6b800b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 -- 172.21.15.59:0/585438276 shutdown_connections 2024-03-06T07:30:16.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.105+0000 7f6b880fb640 1 -- 172.21.15.59:0/585438276 wait complete. 2024-03-06T07:30:16.117 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:30:16.118 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:16.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.253+0000 7f04d0713640 1 Processor -- start 2024-03-06T07:30:16.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.253+0000 7f04d0713640 1 -- start start 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.254+0000 7f04d0713640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f04c815dfe0 0x7f04c80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.254+0000 7f04d0713640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f04c80d29c0 0x7f04c80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.254+0000 7f04d0713640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c80b3430 con 0x7f04c80cf830 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.254+0000 7f04d0713640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c80b31e0 con 0x7f04c80d29c0 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.254+0000 7f04d0713640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c80b3d60 con 0x7f04c815dfe0 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.254+0000 7f04cecb0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f04c80d29c0 0x7f04c80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.254+0000 7f04cdcae640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f04c80cf830 0x7f04c80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41532/0 (socket says 172.21.15.59:41532) 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.254+0000 7f04cdcae640 1 -- 172.21.15.59:0/3951815749 learned_addr learned my addr 172.21.15.59:0/3951815749 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.254+0000 7f04cecb0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f04c80d29c0 0x7f04c80b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36356/0 (socket says 172.21.15.59:36356) 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.255+0000 7f04ce4af640 1 --2- 172.21.15.59:0/3951815749 >> v2:172.21.15.164:3300/0 conn(0x7f04c815dfe0 0x7f04c80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.255+0000 7f04cecb0640 1 -- 172.21.15.59:0/3951815749 >> v2:172.21.15.164:3300/0 conn(0x7f04c815dfe0 msgr2=0x7f04c80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.255+0000 7f04cecb0640 1 --2- 172.21.15.59:0/3951815749 >> v2:172.21.15.164:3300/0 conn(0x7f04c815dfe0 0x7f04c80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.255+0000 7f04cecb0640 1 -- 172.21.15.59:0/3951815749 >> v1:172.21.15.59:6789/0 conn(0x7f04c80cf830 legacy=0x7f04c80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.255+0000 7f04cecb0640 1 -- 172.21.15.59:0/3951815749 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c80b3710 con 0x7f04c80d29c0 2024-03-06T07:30:16.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.255+0000 7f04cd4ad640 1 -- 172.21.15.59:0/3951815749 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953308965 0 0) 0x7f04c80b3430 con 0x7f04c80cf830 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.255+0000 7f04cecb0640 1 --2- 172.21.15.59:0/3951815749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c80d29c0 0x7f04c80b41f0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f04c8067990 tx=0x7f04c402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed258f9cc55c5e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.256+0000 7f04cd4ad640 1 -- 172.21.15.59:0/3951815749 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4039070 con 0x7f04c80d29c0 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.256+0000 7f04cd4ad640 1 -- 172.21.15.59:0/3951815749 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c402cc80 con 0x7f04c80d29c0 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.256+0000 7f04cd4ad640 1 -- 172.21.15.59:0/3951815749 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4034660 con 0x7f04c80d29c0 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.256+0000 7f04d0713640 1 -- 172.21.15.59:0/3951815749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c80d29c0 msgr2=0x7f04c80b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.256+0000 7f04d0713640 1 --2- 172.21.15.59:0/3951815749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c80d29c0 0x7f04c80b41f0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f04c8067990 tx=0x7f04c402bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.256+0000 7f04d0713640 1 -- 172.21.15.59:0/3951815749 shutdown_connections 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.256+0000 7f04d0713640 1 --2- 172.21.15.59:0/3951815749 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c80d29c0 0x7f04c80b41f0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.256+0000 7f04d0713640 1 --2- 172.21.15.59:0/3951815749 >> v2:172.21.15.164:3300/0 conn(0x7f04c815dfe0 0x7f04c80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.256+0000 7f04d0713640 1 -- 172.21.15.59:0/3951815749 >> 172.21.15.59:0/3951815749 conn(0x7f04c80d3cd0 msgr2=0x7f04c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.256+0000 7f04d0713640 1 -- 172.21.15.59:0/3951815749 shutdown_connections 2024-03-06T07:30:16.259 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.256+0000 7f04d0713640 1 -- 172.21.15.59:0/3951815749 wait complete. 2024-03-06T07:30:16.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.257+0000 7f04d0713640 1 Processor -- start 2024-03-06T07:30:16.261 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.257+0000 7f04d0713640 1 -- start start 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04d0713640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c80cf830 0x7f04c81f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04d0713640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c80d29c0 0x7f04c81f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04d0713640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c815dfe0 0x7f04c80b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04d0713640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80d1df0 con 0x7f04c815dfe0 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04d0713640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80d1c70 con 0x7f04c80d29c0 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04d0713640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c815e3c0 con 0x7f04c80cf830 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04cecb0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c815dfe0 0x7f04c80b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04cecb0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c815dfe0 0x7f04c80b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36360/0 (socket says 172.21.15.59:36360) 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04cecb0640 1 -- 172.21.15.59:0/2649369869 learned_addr learned my addr 172.21.15.59:0/2649369869 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04cecb0640 1 -- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c80cf830 msgr2=0x7f04c81f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04cecb0640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c80cf830 0x7f04c81f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04cecb0640 1 -- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c80d29c0 msgr2=0x7f04c81f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.262 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04cecb0640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c80d29c0 0x7f04c81f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.258+0000 7f04cecb0640 1 -- 172.21.15.59:0/2649369869 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c81f3700 con 0x7f04c815dfe0 2024-03-06T07:30:16.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.259+0000 7f04cecb0640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c815dfe0 0x7f04c80b68d0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f04c4037040 tx=0x7f04c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.259+0000 7f04beffd640 1 -- 172.21.15.59:0/2649369869 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4039050 con 0x7f04c815dfe0 2024-03-06T07:30:16.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.259+0000 7f04beffd640 1 -- 172.21.15.59:0/2649369869 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c4045070 con 0x7f04c815dfe0 2024-03-06T07:30:16.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.259+0000 7f04d0713640 1 -- 172.21.15.59:0/2649369869 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c81f4820 con 0x7f04c815dfe0 2024-03-06T07:30:16.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.259+0000 7f04beffd640 1 -- 172.21.15.59:0/2649369869 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c403e830 con 0x7f04c815dfe0 2024-03-06T07:30:16.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.259+0000 7f04d0713640 1 -- 172.21.15.59:0/2649369869 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c80b9080 con 0x7f04c815dfe0 2024-03-06T07:30:16.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.260+0000 7f04d0713640 1 -- 172.21.15.59:0/2649369869 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c804f330 con 0x7f04c815dfe0 2024-03-06T07:30:16.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.260+0000 7f04beffd640 1 -- 172.21.15.59:0/2649369869 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04c403e9d0 con 0x7f04c815dfe0 2024-03-06T07:30:16.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.260+0000 7f04beffd640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ac06b4d0 0x7f04ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.261+0000 7f04beffd640 1 -- 172.21.15.59:0/2649369869 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04c40ad220 con 0x7f04c815dfe0 2024-03-06T07:30:16.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.261+0000 7f04ce4af640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ac06b4d0 0x7f04ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.261+0000 7f04ce4af640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ac06b4d0 0x7f04ac06d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f04c0006fd0 tx=0x7f04c0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.265+0000 7f04beffd640 1 -- 172.21.15.59:0/2649369869 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04c81f4820 con 0x7f04c815dfe0 2024-03-06T07:30:16.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.599+0000 7f04d0713640 1 -- 172.21.15.59:0/2649369869 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04c80b49c0 con 0x7f04c815dfe0 2024-03-06T07:30:16.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.600+0000 7f04beffd640 1 -- 172.21.15.59:0/2649369869 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f04c407c1c0 con 0x7f04c815dfe0 2024-03-06T07:30:16.602 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:16.602 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":97,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4015228738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4015228738},{"type":"v1","addr":"172.21.15.164:6836","nonce":4015228738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93},{"gid":7247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2855659185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2855659185},{"type":"v1","addr":"172.21.15.59:6835","nonce":2855659185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93},{"gid":8236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:15.475267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:30:16.602 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 97 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 -- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ac06b4d0 msgr2=0x7f04ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ac06b4d0 0x7f04ac06d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f04c0006fd0 tx=0x7f04c0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 -- 172.21.15.59:0/2649369869 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c815dfe0 msgr2=0x7f04c80b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c815dfe0 0x7f04c80b68d0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f04c4037040 tx=0x7f04c4004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 -- 172.21.15.59:0/2649369869 shutdown_connections 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04ac06b4d0 0x7f04ac06d990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c815dfe0 0x7f04c80b68d0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c80d29c0 0x7f04c81f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 --2- 172.21.15.59:0/2649369869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c80cf830 0x7f04c81f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 -- 172.21.15.59:0/2649369869 >> 172.21.15.59:0/2649369869 conn(0x7f04c80d3cd0 msgr2=0x7f04c806c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 -- 172.21.15.59:0/2649369869 shutdown_connections 2024-03-06T07:30:16.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.602+0000 7f04d0713640 1 -- 172.21.15.59:0/2649369869 wait complete. 2024-03-06T07:30:16.615 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:16.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.715+0000 7f393409a640 1 Processor -- start 2024-03-06T07:30:16.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.715+0000 7f393409a640 1 -- start start 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.716+0000 7f393409a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f392c15dfe0 0x7f392c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.716+0000 7f393409a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f392c0d29c0 0x7f392c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.716+0000 7f393409a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f392c0b3430 con 0x7f392c0cf830 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.716+0000 7f393409a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f392c0b31e0 con 0x7f392c0d29c0 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.716+0000 7f393409a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f392c0b3d60 con 0x7f392c15dfe0 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.716+0000 7f3932637640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f392c0d29c0 0x7f392c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.716+0000 7f3932637640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f392c0d29c0 0x7f392c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36362/0 (socket says 172.21.15.59:36362) 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.716+0000 7f3932637640 1 -- 172.21.15.59:0/782050607 learned_addr learned my addr 172.21.15.59:0/782050607 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.717+0000 7f3931e36640 1 --2- 172.21.15.59:0/782050607 >> v2:172.21.15.164:3300/0 conn(0x7f392c15dfe0 0x7f392c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.717+0000 7f3932637640 1 -- 172.21.15.59:0/782050607 >> v2:172.21.15.164:3300/0 conn(0x7f392c15dfe0 msgr2=0x7f392c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.717+0000 7f3932637640 1 --2- 172.21.15.59:0/782050607 >> v2:172.21.15.164:3300/0 conn(0x7f392c15dfe0 0x7f392c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.717+0000 7f3932637640 1 -- 172.21.15.59:0/782050607 >> v1:172.21.15.164:6789/0 conn(0x7f392c0cf830 legacy=0x7f392c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.717+0000 7f3932637640 1 -- 172.21.15.59:0/782050607 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c0b3710 con 0x7f392c0d29c0 2024-03-06T07:30:16.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.717+0000 7f3932637640 1 --2- 172.21.15.59:0/782050607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f392c0d29c0 0x7f392c0b41f0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f392c067990 tx=0x7f392402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fa7eb1d540cd52 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.717+0000 7f3930e34640 1 -- 172.21.15.59:0/782050607 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392402ca80 con 0x7f392c0d29c0 2024-03-06T07:30:16.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.718+0000 7f3930e34640 1 -- 172.21.15.59:0/782050607 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392402cc20 con 0x7f392c0d29c0 2024-03-06T07:30:16.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.718+0000 7f3930e34640 1 -- 172.21.15.59:0/782050607 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39240343f0 con 0x7f392c0d29c0 2024-03-06T07:30:16.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.718+0000 7f393409a640 1 -- 172.21.15.59:0/782050607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f392c0d29c0 msgr2=0x7f392c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.718+0000 7f393409a640 1 --2- 172.21.15.59:0/782050607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f392c0d29c0 0x7f392c0b41f0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f392c067990 tx=0x7f392402bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.718+0000 7f393409a640 1 -- 172.21.15.59:0/782050607 shutdown_connections 2024-03-06T07:30:16.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.718+0000 7f393409a640 1 --2- 172.21.15.59:0/782050607 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f392c0d29c0 0x7f392c0b41f0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.718+0000 7f393409a640 1 --2- 172.21.15.59:0/782050607 >> v2:172.21.15.164:3300/0 conn(0x7f392c15dfe0 0x7f392c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.718+0000 7f393409a640 1 -- 172.21.15.59:0/782050607 >> 172.21.15.59:0/782050607 conn(0x7f392c0d3cd0 msgr2=0x7f392c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.718+0000 7f393409a640 1 -- 172.21.15.59:0/782050607 shutdown_connections 2024-03-06T07:30:16.722 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.718+0000 7f393409a640 1 -- 172.21.15.59:0/782050607 wait complete. 2024-03-06T07:30:16.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.719+0000 7f393409a640 1 Processor -- start 2024-03-06T07:30:16.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.719+0000 7f393409a640 1 -- start start 2024-03-06T07:30:16.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.720+0000 7f393409a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f392c0cf830 0x7f392c0cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.720+0000 7f393409a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f392c0d29c0 0x7f392c0cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.723 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.720+0000 7f393409a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f392c15dfe0 0x7f392c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.720+0000 7f393409a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0d1df0 con 0x7f392c15dfe0 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.720+0000 7f393409a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0d1c70 con 0x7f392c0cf830 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.720+0000 7f393409a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c15e3c0 con 0x7f392c0d29c0 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.720+0000 7f3932637640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f392c15dfe0 0x7f392c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.720+0000 7f3932637640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f392c15dfe0 0x7f392c06ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36374/0 (socket says 172.21.15.59:36374) 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.720+0000 7f3932637640 1 -- 172.21.15.59:0/3405520497 learned_addr learned my addr 172.21.15.59:0/3405520497 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.720+0000 7f3931635640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f392c0d29c0 0x7f392c0cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.720+0000 7f3931e36640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f392c0cf830 0x7f392c0cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.721+0000 7f3932637640 1 -- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f392c0d29c0 msgr2=0x7f392c0cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.721+0000 7f3932637640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f392c0d29c0 0x7f392c0cb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.721+0000 7f3932637640 1 -- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f392c0cf830 msgr2=0x7f392c0cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.721+0000 7f3932637640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f392c0cf830 0x7f392c0cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.724 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.721+0000 7f3932637640 1 -- 172.21.15.59:0/3405520497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c159370 con 0x7f392c15dfe0 2024-03-06T07:30:16.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.721+0000 7f3932637640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f392c15dfe0 0x7f392c06ccb0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f3924036040 tx=0x7f3924004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.721+0000 7f39227fc640 1 -- 172.21.15.59:0/3405520497 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924043020 con 0x7f392c15dfe0 2024-03-06T07:30:16.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.721+0000 7f39227fc640 1 -- 172.21.15.59:0/3405520497 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392403e450 con 0x7f392c15dfe0 2024-03-06T07:30:16.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.721+0000 7f393409a640 1 -- 172.21.15.59:0/3405520497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392c0cbd80 con 0x7f392c15dfe0 2024-03-06T07:30:16.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.721+0000 7f39227fc640 1 -- 172.21.15.59:0/3405520497 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924009510 con 0x7f392c15dfe0 2024-03-06T07:30:16.725 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.721+0000 7f393409a640 1 -- 172.21.15.59:0/3405520497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c0cdd60 con 0x7f392c15dfe0 2024-03-06T07:30:16.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.722+0000 7f393409a640 1 -- 172.21.15.59:0/3405520497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392c04f330 con 0x7f392c15dfe0 2024-03-06T07:30:16.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.722+0000 7f39227fc640 1 -- 172.21.15.59:0/3405520497 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3924046020 con 0x7f392c15dfe0 2024-03-06T07:30:16.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.723+0000 7f39227fc640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f391006b4d0 0x7f391006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.723+0000 7f39227fc640 1 -- 172.21.15.59:0/3405520497 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39240aca10 con 0x7f392c15dfe0 2024-03-06T07:30:16.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.723+0000 7f3931e36640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f391006b4d0 0x7f391006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.726 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.724+0000 7f3931e36640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f391006b4d0 0x7f391006d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f3928006fd0 tx=0x7f3928008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.727+0000 7f39227fc640 1 -- 172.21.15.59:0/3405520497 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f392404d030 con 0x7f392c15dfe0 2024-03-06T07:30:16.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.909+0000 7f393409a640 1 -- 172.21.15.59:0/3405520497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f392c0d2630 con 0x7f392c15dfe0 2024-03-06T07:30:16.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.911+0000 7f39227fc640 1 -- 172.21.15.59:0/3405520497 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16667 (secure 0 0 0) 0x7f392407b920 con 0x7f392c15dfe0 2024-03-06T07:30:16.913 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:16.913 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":139,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:15.075718+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:11.712614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:12.406781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.912+0000 7f393409a640 1 -- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f391006b4d0 msgr2=0x7f391006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.913+0000 7f393409a640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f391006b4d0 0x7f391006d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f3928006fd0 tx=0x7f3928008040 comp rx=0 tx=0).stop 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.913+0000 7f393409a640 1 -- 172.21.15.59:0/3405520497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f392c15dfe0 msgr2=0x7f392c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.913+0000 7f393409a640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f392c15dfe0 0x7f392c06ccb0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f3924036040 tx=0x7f3924004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.913+0000 7f393409a640 1 -- 172.21.15.59:0/3405520497 shutdown_connections 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.913+0000 7f393409a640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f391006b4d0 0x7f391006d990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.913+0000 7f393409a640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f392c15dfe0 0x7f392c06ccb0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.913+0000 7f393409a640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f392c0d29c0 0x7f392c0cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.913+0000 7f393409a640 1 --2- 172.21.15.59:0/3405520497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f392c0cf830 0x7f392c0cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.913+0000 7f393409a640 1 -- 172.21.15.59:0/3405520497 >> 172.21.15.59:0/3405520497 conn(0x7f392c0d3cd0 msgr2=0x7f392c0b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.913+0000 7f393409a640 1 -- 172.21.15.59:0/3405520497 shutdown_connections 2024-03-06T07:30:16.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:16.913+0000 7f393409a640 1 -- 172.21.15.59:0/3405520497 wait complete. 2024-03-06T07:30:16.926 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:30:16.926 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:17.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.025+0000 7f44f86aa640 1 Processor -- start 2024-03-06T07:30:17.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.025+0000 7f44f86aa640 1 -- start start 2024-03-06T07:30:17.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.026+0000 7f44f86aa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f015dfe0 0x7f44f00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.026+0000 7f44f86aa640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f00b3430 con 0x7f44f00d29c0 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.026+0000 7f44f86aa640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f00b3710 con 0x7f44f00cf830 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.026+0000 7f44f86aa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00b31e0 con 0x7f44f015dfe0 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.026+0000 7f44f6c47640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f44f00d29c0 0x7f44f00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41536/0 (socket says 172.21.15.59:41536) 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.026+0000 7f44f6c47640 1 -- 172.21.15.59:0/2468314167 learned_addr learned my addr 172.21.15.59:0/2468314167 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.026+0000 7f44f6446640 1 --2- 172.21.15.59:0/2468314167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f015dfe0 0x7f44f00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.027+0000 7f44f5444640 1 -- 172.21.15.59:0/2468314167 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604390606 0 0) 0x7f44f00b3430 con 0x7f44f00d29c0 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.027+0000 7f44f5444640 1 -- 172.21.15.59:0/2468314167 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44dc003520 con 0x7f44f00d29c0 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.027+0000 7f44f5444640 1 -- 172.21.15.59:0/2468314167 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551070442 0 0) 0x7f44f00b3710 con 0x7f44f00cf830 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.027+0000 7f44f6446640 1 -- 172.21.15.59:0/2468314167 >> v1:172.21.15.164:6789/0 conn(0x7f44f00cf830 legacy=0x7f44f00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.027+0000 7f44f6446640 1 -- 172.21.15.59:0/2468314167 >> v1:172.21.15.59:6789/0 conn(0x7f44f00d29c0 legacy=0x7f44f00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.027+0000 7f44f6446640 1 -- 172.21.15.59:0/2468314167 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00b39f0 con 0x7f44f015dfe0 2024-03-06T07:30:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.028+0000 7f44f6446640 1 --2- 172.21.15.59:0/2468314167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f015dfe0 0x7f44f00cf2f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009dc0 tx=0x7f44ec02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57decc5468f7aea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.028+0000 7f44f5444640 1 -- 172.21.15.59:0/2468314167 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec039070 con 0x7f44f015dfe0 2024-03-06T07:30:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.028+0000 7f44f5444640 1 -- 172.21.15.59:0/2468314167 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec02ccd0 con 0x7f44f015dfe0 2024-03-06T07:30:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.028+0000 7f44f5444640 1 -- 172.21.15.59:0/2468314167 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec034640 con 0x7f44f015dfe0 2024-03-06T07:30:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.029+0000 7f44f86aa640 1 -- 172.21.15.59:0/2468314167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f015dfe0 msgr2=0x7f44f00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.029+0000 7f44f86aa640 1 --2- 172.21.15.59:0/2468314167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f015dfe0 0x7f44f00cf2f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009dc0 tx=0x7f44ec02bf20 comp rx=0 tx=0).stop 2024-03-06T07:30:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.029+0000 7f44f86aa640 1 -- 172.21.15.59:0/2468314167 shutdown_connections 2024-03-06T07:30:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.029+0000 7f44f86aa640 1 --2- 172.21.15.59:0/2468314167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f015dfe0 0x7f44f00cf2f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.029+0000 7f44f86aa640 1 -- 172.21.15.59:0/2468314167 >> 172.21.15.59:0/2468314167 conn(0x7f44f00d3cd0 msgr2=0x7f44f00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.029+0000 7f44f86aa640 1 -- 172.21.15.59:0/2468314167 shutdown_connections 2024-03-06T07:30:17.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.029+0000 7f44f86aa640 1 -- 172.21.15.59:0/2468314167 wait complete. 2024-03-06T07:30:17.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.030+0000 7f44f86aa640 1 Processor -- start 2024-03-06T07:30:17.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.030+0000 7f44f86aa640 1 -- start start 2024-03-06T07:30:17.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f86aa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44f00cf830 0x7f44f00cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f86aa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f00d29c0 0x7f44f015a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f86aa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f015dfe0 0x7f44f00cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f86aa640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f015e3c0 con 0x7f44f00cf830 2024-03-06T07:30:17.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f86aa640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00b3d60 con 0x7f44f00d29c0 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f86aa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f015e540 con 0x7f44f015dfe0 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f6446640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44f00cf830 0x7f44f00cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f6446640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44f00cf830 0x7f44f00cda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36388/0 (socket says 172.21.15.59:36388) 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f6446640 1 -- 172.21.15.59:0/1982963663 learned_addr learned my addr 172.21.15.59:0/1982963663 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f5c45640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f00d29c0 0x7f44f015a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f6c47640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f015dfe0 0x7f44f00cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f6446640 1 -- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f015dfe0 msgr2=0x7f44f00cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f6446640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f015dfe0 0x7f44f00cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f6446640 1 -- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f00d29c0 msgr2=0x7f44f015a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f6446640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f00d29c0 0x7f44f015a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f6446640 1 -- 172.21.15.59:0/1982963663 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00cbf50 con 0x7f44f00cf830 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f5c45640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f00d29c0 0x7f44f015a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f6446640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44f00cf830 0x7f44f00cda30 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000f80 tx=0x7f44ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44e6ffd640 1 -- 172.21.15.59:0/1982963663 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec039050 con 0x7f44f00cf830 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44e6ffd640 1 -- 172.21.15.59:0/1982963663 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec045070 con 0x7f44f00cf830 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44f86aa640 1 -- 172.21.15.59:0/1982963663 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f00cc230 con 0x7f44f00cf830 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.031+0000 7f44e6ffd640 1 -- 172.21.15.59:0/1982963663 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec03e850 con 0x7f44f00cf830 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.032+0000 7f44f86aa640 1 -- 172.21.15.59:0/1982963663 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f00b6a70 con 0x7f44f00cf830 2024-03-06T07:30:17.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.032+0000 7f44f86aa640 1 -- 172.21.15.59:0/1982963663 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f00cc750 con 0x7f44f00cf830 2024-03-06T07:30:17.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.033+0000 7f44e6ffd640 1 -- 172.21.15.59:0/1982963663 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44ec03e9f0 con 0x7f44f00cf830 2024-03-06T07:30:17.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.033+0000 7f44e6ffd640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 0x7f44d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.033+0000 7f44e6ffd640 1 -- 172.21.15.59:0/1982963663 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44ec0ad030 con 0x7f44f00cf830 2024-03-06T07:30:17.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.034+0000 7f44f5c45640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 0x7f44d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.034+0000 7f44f5c45640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 0x7f44d406d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f44f01f70b0 tx=0x7f44e0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.042+0000 7f44e6ffd640 1 -- 172.21.15.59:0/1982963663 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44f00cc750 con 0x7f44f00cf830 2024-03-06T07:30:17.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.220+0000 7f44f86aa640 1 -- 172.21.15.59:0/1982963663 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44f00cbcf0 con 0x7f44d406b4d0 2024-03-06T07:30:17.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.226+0000 7f44e6ffd640 1 -- 172.21.15.59:0/1982963663 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237265 (secure 0 0 0) 0x7f44e00013d0 con 0x7f44d406b4d0 2024-03-06T07:30:17.231 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:17.232 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:30:17.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 -- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 msgr2=0x7f44d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 0x7f44d406d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f44f01f70b0 tx=0x7f44e0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:17.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 -- 172.21.15.59:0/1982963663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44f00cf830 msgr2=0x7f44f00cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44f00cf830 0x7f44f00cda30 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000f80 tx=0x7f44ec004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:17.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 -- 172.21.15.59:0/1982963663 shutdown_connections 2024-03-06T07:30:17.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f44d406b4d0 0x7f44d406d990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f015dfe0 0x7f44f00cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f00d29c0 0x7f44f015a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 --2- 172.21.15.59:0/1982963663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44f00cf830 0x7f44f00cda30 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 -- 172.21.15.59:0/1982963663 >> 172.21.15.59:0/1982963663 conn(0x7f44f00d3cd0 msgr2=0x7f44f00b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 -- 172.21.15.59:0/1982963663 shutdown_connections 2024-03-06T07:30:17.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.229+0000 7f44f86aa640 1 -- 172.21.15.59:0/1982963663 wait complete. 2024-03-06T07:30:17.242 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-03-06T07:30:15.415100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":330392,"kb_used_data":14376,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644328,"statfs":{"total":773094113280,"available":772755791872,"internally_reserved":0,"allocated":14721024,"data_stored":7084392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000770"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:13.068641+0000","last_change":"2024-03-06T07:30:13.068641+0000","last_active":"2024-03-06T07:30:13.068641+0000","last_peered":"2024-03-06T07:30:13.068641+0000","last_clean":"2024-03-06T07:30:13.068641+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:13.068641+0000","last_undegraded":"2024-03-06T07:30:13.068641+0000","last_fullsized":"2024-03-06T07:30:13.068641+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:30:12.063411+0000","last_change":"2024-03-06T07:30:12.063411+0000","last_active":"2024-03-06T07:30:12.063411+0000","last_peered":"2024-03-06T07:30:12.063411+0000","last_clean":"2024-03-06T07:30:12.063411+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:12.063411+0000","last_undegraded":"2024-03-06T07:30:12.063411+0000","last_fullsized":"2024-03-06T07:30:12.063411+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":377,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-03-06T07:30:10.211570+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:30:10.211570+0000","last_peered":"2024-03-06T07:30:10.211570+0000","last_clean":"2024-03-06T07:30:10.211570+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:30:10.211570+0000","last_undegraded":"2024-03-06T07:30:10.211570+0000","last_fullsized":"2024-03-06T07:30:10.211570+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574953,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":2285568,"data_stored":1328325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":737957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":737957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":737957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":737957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41156,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330684,"statfs":{"total":96636764160,"available":96594620416,"internally_reserved":0,"allocated":1691648,"data_stored":737957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41152,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330688,"statfs":{"total":96636764160,"available":96594624512,"internally_reserved":0,"allocated":1691648,"data_stored":737957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574954,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41736,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330104,"statfs":{"total":96636764160,"available":96594026496,"internally_reserved":0,"allocated":2285568,"data_stored":1328325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:17.247 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:17.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.382+0000 7f5f83f55640 1 Processor -- start 2024-03-06T07:30:17.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.382+0000 7f5f83f55640 1 -- start start 2024-03-06T07:30:17.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.383+0000 7f5f83f55640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f7c15bb20 0x7f5f7c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.383+0000 7f5f83f55640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5f7c0cd8c0 0x7f5f7c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.383+0000 7f5f83f55640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f7c161980 con 0x7f5f7c0cfff0 2024-03-06T07:30:17.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.383+0000 7f5f83f55640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7c161c60 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.383+0000 7f5f83f55640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c161f40 con 0x7f5f7c15bb20 2024-03-06T07:30:17.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.383+0000 7f5f824f2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5f7c0cd8c0 0x7f5f7c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.383+0000 7f5f81cf1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f7c15bb20 0x7f5f7c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.383+0000 7f5f814f0640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5f7c0cfff0 0x7f5f7c0ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:39348/0 (socket says 172.21.15.59:39348) 2024-03-06T07:30:17.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.383+0000 7f5f814f0640 1 -- 172.21.15.59:0/2391628319 learned_addr learned my addr 172.21.15.59:0/2391628319 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:17.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.384+0000 7f5f81cf1640 1 -- 172.21.15.59:0/2391628319 >> v2:172.21.15.164:3300/0 conn(0x7f5f7c0cd8c0 msgr2=0x7f5f7c0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.384+0000 7f5f81cf1640 1 --2- 172.21.15.59:0/2391628319 >> v2:172.21.15.164:3300/0 conn(0x7f5f7c0cd8c0 0x7f5f7c0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.384+0000 7f5f81cf1640 1 -- 172.21.15.59:0/2391628319 >> v1:172.21.15.164:6789/0 conn(0x7f5f7c0cfff0 legacy=0x7f5f7c0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.384+0000 7f5f81cf1640 1 -- 172.21.15.59:0/2391628319 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c162220 con 0x7f5f7c15bb20 2024-03-06T07:30:17.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.384+0000 7f5f824f2640 1 --2- 172.21.15.59:0/2391628319 >> v2:172.21.15.164:3300/0 conn(0x7f5f7c0cd8c0 0x7f5f7c0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:17.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.384+0000 7f5f81cf1640 1 --2- 172.21.15.59:0/2391628319 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f7c15bb20 0x7f5f7c0d1df0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5f780097a0 tx=0x7f5f7802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33fdd785ecaa975d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.385+0000 7f5f80cef640 1 -- 172.21.15.59:0/2391628319 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f780315a0 con 0x7f5f7c15bb20 2024-03-06T07:30:17.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.385+0000 7f5f80cef640 1 -- 172.21.15.59:0/2391628319 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f78037070 con 0x7f5f7c15bb20 2024-03-06T07:30:17.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.385+0000 7f5f80cef640 1 -- 172.21.15.59:0/2391628319 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78031960 con 0x7f5f7c15bb20 2024-03-06T07:30:17.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.385+0000 7f5f83f55640 1 -- 172.21.15.59:0/2391628319 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f7c15bb20 msgr2=0x7f5f7c0d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.385+0000 7f5f83f55640 1 --2- 172.21.15.59:0/2391628319 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f7c15bb20 0x7f5f7c0d1df0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5f780097a0 tx=0x7f5f7802b840 comp rx=0 tx=0).stop 2024-03-06T07:30:17.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.385+0000 7f5f83f55640 1 -- 172.21.15.59:0/2391628319 shutdown_connections 2024-03-06T07:30:17.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.385+0000 7f5f83f55640 1 --2- 172.21.15.59:0/2391628319 >> v2:172.21.15.164:3300/0 conn(0x7f5f7c0cd8c0 0x7f5f7c0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.385+0000 7f5f83f55640 1 --2- 172.21.15.59:0/2391628319 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f7c15bb20 0x7f5f7c0d1df0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.385+0000 7f5f83f55640 1 -- 172.21.15.59:0/2391628319 >> 172.21.15.59:0/2391628319 conn(0x7f5f7c09a9f0 msgr2=0x7f5f7c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.385+0000 7f5f83f55640 1 -- 172.21.15.59:0/2391628319 shutdown_connections 2024-03-06T07:30:17.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.385+0000 7f5f83f55640 1 -- 172.21.15.59:0/2391628319 wait complete. 2024-03-06T07:30:17.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.386+0000 7f5f83f55640 1 Processor -- start 2024-03-06T07:30:17.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.386+0000 7f5f83f55640 1 -- start start 2024-03-06T07:30:17.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.387+0000 7f5f83f55640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f7c0cd8c0 0x7f5f7c15af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.387+0000 7f5f83f55640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f7c0cfff0 0x7f5f7c15f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.387+0000 7f5f83f55640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f7c15bb20 0x7f5f7c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.387+0000 7f5f83f55640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0d24b0 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.387+0000 7f5f83f55640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0d2330 con 0x7f5f7c15bb20 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.387+0000 7f5f83f55640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0d1980 con 0x7f5f7c0cfff0 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.387+0000 7f5f81cf1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f7c0cd8c0 0x7f5f7c15af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.387+0000 7f5f81cf1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f7c0cd8c0 0x7f5f7c15af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36394/0 (socket says 172.21.15.59:36394) 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.387+0000 7f5f81cf1640 1 -- 172.21.15.59:0/80638156 learned_addr learned my addr 172.21.15.59:0/80638156 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.387+0000 7f5f814f0640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f7c0cfff0 0x7f5f7c15f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.387+0000 7f5f824f2640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f7c15bb20 0x7f5f7c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.388+0000 7f5f81cf1640 1 -- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f7c0cfff0 msgr2=0x7f5f7c15f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.388+0000 7f5f81cf1640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f7c0cfff0 0x7f5f7c15f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.388+0000 7f5f81cf1640 1 -- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f7c15bb20 msgr2=0x7f5f7c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.388+0000 7f5f81cf1640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f7c15bb20 0x7f5f7c06c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.388+0000 7f5f81cf1640 1 -- 172.21.15.59:0/80638156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c0c7a70 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.388+0000 7f5f81cf1640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f7c0cd8c0 0x7f5f7c15af80 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f5f780098c0 tx=0x7f5f780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.388+0000 7f5f727fc640 1 -- 172.21.15.59:0/80638156 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78032630 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.388+0000 7f5f727fc640 1 -- 172.21.15.59:0/80638156 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f780327f0 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.388+0000 7f5f83f55640 1 -- 172.21.15.59:0/80638156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7c0c7d50 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.388+0000 7f5f727fc640 1 -- 172.21.15.59:0/80638156 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78032be0 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.388+0000 7f5f83f55640 1 -- 172.21.15.59:0/80638156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7c0c8310 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.389+0000 7f5f83f55640 1 -- 172.21.15.59:0/80638156 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7c04f330 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.390+0000 7f5f727fc640 1 -- 172.21.15.59:0/80638156 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f7803b3f0 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.390+0000 7f5f727fc640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f6006b4d0 0x7f5f6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.390+0000 7f5f727fc640 1 -- 172.21.15.59:0/80638156 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f7802f030 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.391+0000 7f5f814f0640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f6006b4d0 0x7f5f6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.391+0000 7f5f814f0640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f6006b4d0 0x7f5f6006d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c06d1c0 tx=0x7f5f6c007e80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.399+0000 7f5f727fc640 1 -- 172.21.15.59:0/80638156 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f78040070 con 0x7f5f7c0cd8c0 2024-03-06T07:30:17.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.582+0000 7f5f83f55640 1 -- 172.21.15.59:0/80638156 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f7c0b1820 con 0x7f5f6006b4d0 2024-03-06T07:30:17.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.588+0000 7f5f727fc640 1 -- 172.21.15.59:0/80638156 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252508 (secure 0 0 0) 0x7f5f7c0b1820 con 0x7f5f6006b4d0 2024-03-06T07:30:17.593 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:17.594 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:30:17.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.591+0000 7f5f83f55640 1 -- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f6006b4d0 msgr2=0x7f5f6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.591+0000 7f5f83f55640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f6006b4d0 0x7f5f6006d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c06d1c0 tx=0x7f5f6c007e80 comp rx=0 tx=0).stop 2024-03-06T07:30:17.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.591+0000 7f5f83f55640 1 -- 172.21.15.59:0/80638156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f7c0cd8c0 msgr2=0x7f5f7c15af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.591+0000 7f5f83f55640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f7c0cd8c0 0x7f5f7c15af80 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f5f780098c0 tx=0x7f5f780046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.592+0000 7f5f83f55640 1 -- 172.21.15.59:0/80638156 shutdown_connections 2024-03-06T07:30:17.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.592+0000 7f5f83f55640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5f6006b4d0 0x7f5f6006d990 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.592+0000 7f5f83f55640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f7c15bb20 0x7f5f7c06c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.592+0000 7f5f83f55640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f7c0cfff0 0x7f5f7c15f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.592+0000 7f5f83f55640 1 --2- 172.21.15.59:0/80638156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5f7c0cd8c0 0x7f5f7c15af80 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.592+0000 7f5f83f55640 1 -- 172.21.15.59:0/80638156 >> 172.21.15.59:0/80638156 conn(0x7f5f7c09a9f0 msgr2=0x7f5f7c0703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.592+0000 7f5f83f55640 1 -- 172.21.15.59:0/80638156 shutdown_connections 2024-03-06T07:30:17.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.592+0000 7f5f83f55640 1 -- 172.21.15.59:0/80638156 wait complete. 2024-03-06T07:30:17.604 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-03-06T07:30:17.415588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330852,"kb_used_data":14812,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643868,"statfs":{"total":773094113280,"available":772755320832,"internally_reserved":0,"allocated":15167488,"data_stored":7284248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.346486"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696422+0000","last_change":"2024-03-06T07:30:14.092526+0000","last_active":"2024-03-06T07:30:15.696422+0000","last_peered":"2024-03-06T07:30:15.696422+0000","last_clean":"2024-03-06T07:30:15.696422+0000","last_became_active":"2024-03-06T07:30:14.092224+0000","last_became_peered":"2024-03-06T07:30:14.092224+0000","last_unstale":"2024-03-06T07:30:15.696422+0000","last_undegraded":"2024-03-06T07:30:15.696422+0000","last_fullsized":"2024-03-06T07:30:15.696422+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:04.500655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695596+0000","last_change":"2024-03-06T07:30:14.087409+0000","last_active":"2024-03-06T07:30:15.695596+0000","last_peered":"2024-03-06T07:30:15.695596+0000","last_clean":"2024-03-06T07:30:15.695596+0000","last_became_active":"2024-03-06T07:30:14.087130+0000","last_became_peered":"2024-03-06T07:30:14.087130+0000","last_unstale":"2024-03-06T07:30:15.695596+0000","last_undegraded":"2024-03-06T07:30:15.695596+0000","last_fullsized":"2024-03-06T07:30:15.695596+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:21.750545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.082825+0000","last_change":"2024-03-06T07:30:14.098045+0000","last_active":"2024-03-06T07:30:15.082825+0000","last_peered":"2024-03-06T07:30:15.082825+0000","last_clean":"2024-03-06T07:30:15.082825+0000","last_became_active":"2024-03-06T07:30:14.097771+0000","last_became_peered":"2024-03-06T07:30:14.097771+0000","last_unstale":"2024-03-06T07:30:15.082825+0000","last_undegraded":"2024-03-06T07:30:15.082825+0000","last_fullsized":"2024-03-06T07:30:15.082825+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:15.421159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696379+0000","last_change":"2024-03-06T07:30:14.096492+0000","last_active":"2024-03-06T07:30:15.696379+0000","last_peered":"2024-03-06T07:30:15.696379+0000","last_clean":"2024-03-06T07:30:15.696379+0000","last_became_active":"2024-03-06T07:30:14.096094+0000","last_became_peered":"2024-03-06T07:30:14.096094+0000","last_unstale":"2024-03-06T07:30:15.696379+0000","last_undegraded":"2024-03-06T07:30:15.696379+0000","last_fullsized":"2024-03-06T07:30:15.696379+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:36.512641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083448+0000","last_change":"2024-03-06T07:30:14.093605+0000","last_active":"2024-03-06T07:30:15.083448+0000","last_peered":"2024-03-06T07:30:15.083448+0000","last_clean":"2024-03-06T07:30:15.083448+0000","last_became_active":"2024-03-06T07:30:14.093330+0000","last_became_peered":"2024-03-06T07:30:14.093330+0000","last_unstale":"2024-03-06T07:30:15.083448+0000","last_undegraded":"2024-03-06T07:30:15.083448+0000","last_fullsized":"2024-03-06T07:30:15.083448+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:41.447972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695488+0000","last_change":"2024-03-06T07:30:14.100441+0000","last_active":"2024-03-06T07:30:15.695488+0000","last_peered":"2024-03-06T07:30:15.695488+0000","last_clean":"2024-03-06T07:30:15.695488+0000","last_became_active":"2024-03-06T07:30:14.100107+0000","last_became_peered":"2024-03-06T07:30:14.100107+0000","last_unstale":"2024-03-06T07:30:15.695488+0000","last_undegraded":"2024-03-06T07:30:15.695488+0000","last_fullsized":"2024-03-06T07:30:15.695488+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:49.496226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695535+0000","last_change":"2024-03-06T07:30:14.095784+0000","last_active":"2024-03-06T07:30:15.695535+0000","last_peered":"2024-03-06T07:30:15.695535+0000","last_clean":"2024-03-06T07:30:15.695535+0000","last_became_active":"2024-03-06T07:30:14.095468+0000","last_became_peered":"2024-03-06T07:30:14.095468+0000","last_unstale":"2024-03-06T07:30:15.695535+0000","last_undegraded":"2024-03-06T07:30:15.695535+0000","last_fullsized":"2024-03-06T07:30:15.695535+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:40.521861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695048+0000","last_change":"2024-03-06T07:30:14.096008+0000","last_active":"2024-03-06T07:30:15.695048+0000","last_peered":"2024-03-06T07:30:15.695048+0000","last_clean":"2024-03-06T07:30:15.695048+0000","last_became_active":"2024-03-06T07:30:14.092073+0000","last_became_peered":"2024-03-06T07:30:14.092073+0000","last_unstale":"2024-03-06T07:30:15.695048+0000","last_undegraded":"2024-03-06T07:30:15.695048+0000","last_fullsized":"2024-03-06T07:30:15.695048+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:14.664617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695346+0000","last_change":"2024-03-06T07:30:14.093394+0000","last_active":"2024-03-06T07:30:15.695346+0000","last_peered":"2024-03-06T07:30:15.695346+0000","last_clean":"2024-03-06T07:30:15.695346+0000","last_became_active":"2024-03-06T07:30:14.093107+0000","last_became_peered":"2024-03-06T07:30:14.093107+0000","last_unstale":"2024-03-06T07:30:15.695346+0000","last_undegraded":"2024-03-06T07:30:15.695346+0000","last_fullsized":"2024-03-06T07:30:15.695346+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:05.682107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696419+0000","last_change":"2024-03-06T07:30:14.100475+0000","last_active":"2024-03-06T07:30:15.696419+0000","last_peered":"2024-03-06T07:30:15.696419+0000","last_clean":"2024-03-06T07:30:15.696419+0000","last_became_active":"2024-03-06T07:30:14.098219+0000","last_became_peered":"2024-03-06T07:30:14.098219+0000","last_unstale":"2024-03-06T07:30:15.696419+0000","last_undegraded":"2024-03-06T07:30:15.696419+0000","last_fullsized":"2024-03-06T07:30:15.696419+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:10.834419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695884+0000","last_change":"2024-03-06T07:30:14.096756+0000","last_active":"2024-03-06T07:30:15.695884+0000","last_peered":"2024-03-06T07:30:15.695884+0000","last_clean":"2024-03-06T07:30:15.695884+0000","last_became_active":"2024-03-06T07:30:14.096511+0000","last_became_peered":"2024-03-06T07:30:14.096511+0000","last_unstale":"2024-03-06T07:30:15.695884+0000","last_undegraded":"2024-03-06T07:30:15.695884+0000","last_fullsized":"2024-03-06T07:30:15.695884+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:10.453283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696313+0000","last_change":"2024-03-06T07:30:14.100463+0000","last_active":"2024-03-06T07:30:15.696313+0000","last_peered":"2024-03-06T07:30:15.696313+0000","last_clean":"2024-03-06T07:30:15.696313+0000","last_became_active":"2024-03-06T07:30:14.100186+0000","last_became_peered":"2024-03-06T07:30:14.100186+0000","last_unstale":"2024-03-06T07:30:15.696313+0000","last_undegraded":"2024-03-06T07:30:15.696313+0000","last_fullsized":"2024-03-06T07:30:15.696313+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:21.750545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083623+0000","last_change":"2024-03-06T07:30:14.092934+0000","last_active":"2024-03-06T07:30:15.083623+0000","last_peered":"2024-03-06T07:30:15.083623+0000","last_clean":"2024-03-06T07:30:15.083623+0000","last_became_active":"2024-03-06T07:30:14.092670+0000","last_became_peered":"2024-03-06T07:30:14.092670+0000","last_unstale":"2024-03-06T07:30:15.083623+0000","last_undegraded":"2024-03-06T07:30:15.083623+0000","last_fullsized":"2024-03-06T07:30:15.083623+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:25.754352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695661+0000","last_change":"2024-03-06T07:30:14.093091+0000","last_active":"2024-03-06T07:30:15.695661+0000","last_peered":"2024-03-06T07:30:15.695661+0000","last_clean":"2024-03-06T07:30:15.695661+0000","last_became_active":"2024-03-06T07:30:14.092834+0000","last_became_peered":"2024-03-06T07:30:14.092834+0000","last_unstale":"2024-03-06T07:30:15.695661+0000","last_undegraded":"2024-03-06T07:30:15.695661+0000","last_fullsized":"2024-03-06T07:30:15.695661+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:04.500655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695665+0000","last_change":"2024-03-06T07:30:14.101101+0000","last_active":"2024-03-06T07:30:15.695665+0000","last_peered":"2024-03-06T07:30:15.695665+0000","last_clean":"2024-03-06T07:30:15.695665+0000","last_became_active":"2024-03-06T07:30:14.100837+0000","last_became_peered":"2024-03-06T07:30:14.100837+0000","last_unstale":"2024-03-06T07:30:15.695665+0000","last_undegraded":"2024-03-06T07:30:15.695665+0000","last_fullsized":"2024-03-06T07:30:15.695665+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:12.290001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.078958+0000","last_change":"2024-03-06T07:30:14.104003+0000","last_active":"2024-03-06T07:30:15.078958+0000","last_peered":"2024-03-06T07:30:15.078958+0000","last_clean":"2024-03-06T07:30:15.078958+0000","last_became_active":"2024-03-06T07:30:14.103455+0000","last_became_peered":"2024-03-06T07:30:14.103455+0000","last_unstale":"2024-03-06T07:30:15.078958+0000","last_undegraded":"2024-03-06T07:30:15.078958+0000","last_fullsized":"2024-03-06T07:30:15.078958+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:42.543160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696507+0000","last_change":"2024-03-06T07:30:14.097815+0000","last_active":"2024-03-06T07:30:15.696507+0000","last_peered":"2024-03-06T07:30:15.696507+0000","last_clean":"2024-03-06T07:30:15.696507+0000","last_became_active":"2024-03-06T07:30:14.097555+0000","last_became_peered":"2024-03-06T07:30:14.097555+0000","last_unstale":"2024-03-06T07:30:15.696507+0000","last_undegraded":"2024-03-06T07:30:15.696507+0000","last_fullsized":"2024-03-06T07:30:15.696507+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:02.401640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083322+0000","last_change":"2024-03-06T07:30:14.100551+0000","last_active":"2024-03-06T07:30:15.083322+0000","last_peered":"2024-03-06T07:30:15.083322+0000","last_clean":"2024-03-06T07:30:15.083322+0000","last_became_active":"2024-03-06T07:30:14.100281+0000","last_became_peered":"2024-03-06T07:30:14.100281+0000","last_unstale":"2024-03-06T07:30:15.083322+0000","last_undegraded":"2024-03-06T07:30:15.083322+0000","last_fullsized":"2024-03-06T07:30:15.083322+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:00.899477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696138+0000","last_change":"2024-03-06T07:30:14.099311+0000","last_active":"2024-03-06T07:30:15.696138+0000","last_peered":"2024-03-06T07:30:15.696138+0000","last_clean":"2024-03-06T07:30:15.696138+0000","last_became_active":"2024-03-06T07:30:14.099057+0000","last_became_peered":"2024-03-06T07:30:14.099057+0000","last_unstale":"2024-03-06T07:30:15.696138+0000","last_undegraded":"2024-03-06T07:30:15.696138+0000","last_fullsized":"2024-03-06T07:30:15.696138+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.191719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.079137+0000","last_change":"2024-03-06T07:30:14.097536+0000","last_active":"2024-03-06T07:30:15.079137+0000","last_peered":"2024-03-06T07:30:15.079137+0000","last_clean":"2024-03-06T07:30:15.079137+0000","last_became_active":"2024-03-06T07:30:14.097218+0000","last_became_peered":"2024-03-06T07:30:14.097218+0000","last_unstale":"2024-03-06T07:30:15.079137+0000","last_undegraded":"2024-03-06T07:30:15.079137+0000","last_fullsized":"2024-03-06T07:30:15.079137+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:00.004098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695991+0000","last_change":"2024-03-06T07:30:14.099292+0000","last_active":"2024-03-06T07:30:15.695991+0000","last_peered":"2024-03-06T07:30:15.695991+0000","last_clean":"2024-03-06T07:30:15.695991+0000","last_became_active":"2024-03-06T07:30:14.098946+0000","last_became_peered":"2024-03-06T07:30:14.098946+0000","last_unstale":"2024-03-06T07:30:15.695991+0000","last_undegraded":"2024-03-06T07:30:15.695991+0000","last_fullsized":"2024-03-06T07:30:15.695991+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:42.117837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695750+0000","last_change":"2024-03-06T07:30:14.092782+0000","last_active":"2024-03-06T07:30:15.695750+0000","last_peered":"2024-03-06T07:30:15.695750+0000","last_clean":"2024-03-06T07:30:15.695750+0000","last_became_active":"2024-03-06T07:30:14.092505+0000","last_became_peered":"2024-03-06T07:30:14.092505+0000","last_unstale":"2024-03-06T07:30:15.695750+0000","last_undegraded":"2024-03-06T07:30:15.695750+0000","last_fullsized":"2024-03-06T07:30:15.695750+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:42.160137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695437+0000","last_change":"2024-03-06T07:30:14.090658+0000","last_active":"2024-03-06T07:30:15.695437+0000","last_peered":"2024-03-06T07:30:15.695437+0000","last_clean":"2024-03-06T07:30:15.695437+0000","last_became_active":"2024-03-06T07:30:14.090379+0000","last_became_peered":"2024-03-06T07:30:14.090379+0000","last_unstale":"2024-03-06T07:30:15.695437+0000","last_undegraded":"2024-03-06T07:30:15.695437+0000","last_fullsized":"2024-03-06T07:30:15.695437+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:57.388740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695549+0000","last_change":"2024-03-06T07:30:14.093235+0000","last_active":"2024-03-06T07:30:15.695549+0000","last_peered":"2024-03-06T07:30:15.695549+0000","last_clean":"2024-03-06T07:30:15.695549+0000","last_became_active":"2024-03-06T07:30:14.092970+0000","last_became_peered":"2024-03-06T07:30:14.092970+0000","last_unstale":"2024-03-06T07:30:15.695549+0000","last_undegraded":"2024-03-06T07:30:15.695549+0000","last_fullsized":"2024-03-06T07:30:15.695549+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:58.729040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.079018+0000","last_change":"2024-03-06T07:30:14.103479+0000","last_active":"2024-03-06T07:30:15.079018+0000","last_peered":"2024-03-06T07:30:15.079018+0000","last_clean":"2024-03-06T07:30:15.079018+0000","last_became_active":"2024-03-06T07:30:14.103200+0000","last_became_peered":"2024-03-06T07:30:14.103200+0000","last_unstale":"2024-03-06T07:30:15.079018+0000","last_undegraded":"2024-03-06T07:30:15.079018+0000","last_fullsized":"2024-03-06T07:30:15.079018+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:40.189248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696275+0000","last_change":"2024-03-06T07:30:14.091159+0000","last_active":"2024-03-06T07:30:15.696275+0000","last_peered":"2024-03-06T07:30:15.696275+0000","last_clean":"2024-03-06T07:30:15.696275+0000","last_became_active":"2024-03-06T07:30:14.090880+0000","last_became_peered":"2024-03-06T07:30:14.090880+0000","last_unstale":"2024-03-06T07:30:15.696275+0000","last_undegraded":"2024-03-06T07:30:15.696275+0000","last_fullsized":"2024-03-06T07:30:15.696275+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:57.388740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.079025+0000","last_change":"2024-03-06T07:30:14.095206+0000","last_active":"2024-03-06T07:30:15.079025+0000","last_peered":"2024-03-06T07:30:15.079025+0000","last_clean":"2024-03-06T07:30:15.079025+0000","last_became_active":"2024-03-06T07:30:14.094830+0000","last_became_peered":"2024-03-06T07:30:14.094830+0000","last_unstale":"2024-03-06T07:30:15.079025+0000","last_undegraded":"2024-03-06T07:30:15.079025+0000","last_fullsized":"2024-03-06T07:30:15.079025+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:00.992740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695788+0000","last_change":"2024-03-06T07:30:14.097378+0000","last_active":"2024-03-06T07:30:15.695788+0000","last_peered":"2024-03-06T07:30:15.695788+0000","last_clean":"2024-03-06T07:30:15.695788+0000","last_became_active":"2024-03-06T07:30:14.096582+0000","last_became_peered":"2024-03-06T07:30:14.096582+0000","last_unstale":"2024-03-06T07:30:15.695788+0000","last_undegraded":"2024-03-06T07:30:15.695788+0000","last_fullsized":"2024-03-06T07:30:15.695788+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:05.682107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695633+0000","last_change":"2024-03-06T07:30:14.101353+0000","last_active":"2024-03-06T07:30:15.695633+0000","last_peered":"2024-03-06T07:30:15.695633+0000","last_clean":"2024-03-06T07:30:15.695633+0000","last_became_active":"2024-03-06T07:30:14.101083+0000","last_became_peered":"2024-03-06T07:30:14.101083+0000","last_unstale":"2024-03-06T07:30:15.695633+0000","last_undegraded":"2024-03-06T07:30:15.695633+0000","last_fullsized":"2024-03-06T07:30:15.695633+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:41.785244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695831+0000","last_change":"2024-03-06T07:30:14.105214+0000","last_active":"2024-03-06T07:30:15.695831+0000","last_peered":"2024-03-06T07:30:15.695831+0000","last_clean":"2024-03-06T07:30:15.695831+0000","last_became_active":"2024-03-06T07:30:14.104931+0000","last_became_peered":"2024-03-06T07:30:14.104931+0000","last_unstale":"2024-03-06T07:30:15.695831+0000","last_undegraded":"2024-03-06T07:30:15.695831+0000","last_fullsized":"2024-03-06T07:30:15.695831+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:08.133310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695798+0000","last_change":"2024-03-06T07:30:14.096206+0000","last_active":"2024-03-06T07:30:15.695798+0000","last_peered":"2024-03-06T07:30:15.695798+0000","last_clean":"2024-03-06T07:30:15.695798+0000","last_became_active":"2024-03-06T07:30:14.095939+0000","last_became_peered":"2024-03-06T07:30:14.095939+0000","last_unstale":"2024-03-06T07:30:15.695798+0000","last_undegraded":"2024-03-06T07:30:15.695798+0000","last_fullsized":"2024-03-06T07:30:15.695798+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:42.109692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.078840+0000","last_change":"2024-03-06T07:30:14.094701+0000","last_active":"2024-03-06T07:30:15.078840+0000","last_peered":"2024-03-06T07:30:15.078840+0000","last_clean":"2024-03-06T07:30:15.078840+0000","last_became_active":"2024-03-06T07:30:14.094347+0000","last_became_peered":"2024-03-06T07:30:14.094347+0000","last_unstale":"2024-03-06T07:30:15.078840+0000","last_undegraded":"2024-03-06T07:30:15.078840+0000","last_fullsized":"2024-03-06T07:30:15.078840+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:03.182859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696499+0000","last_change":"2024-03-06T07:30:14.094403+0000","last_active":"2024-03-06T07:30:15.696499+0000","last_peered":"2024-03-06T07:30:15.696499+0000","last_clean":"2024-03-06T07:30:15.696499+0000","last_became_active":"2024-03-06T07:30:14.093771+0000","last_became_peered":"2024-03-06T07:30:14.093771+0000","last_unstale":"2024-03-06T07:30:15.696499+0000","last_undegraded":"2024-03-06T07:30:15.696499+0000","last_fullsized":"2024-03-06T07:30:15.696499+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:19.135167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"138'4","reported_seq":18,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696312+0000","last_change":"2024-03-06T07:30:13.096317+0000","last_active":"2024-03-06T07:30:15.696312+0000","last_peered":"2024-03-06T07:30:15.696312+0000","last_clean":"2024-03-06T07:30:15.696312+0000","last_became_active":"2024-03-06T07:30:13.095858+0000","last_became_peered":"2024-03-06T07:30:13.095858+0000","last_unstale":"2024-03-06T07:30:15.696312+0000","last_undegraded":"2024-03-06T07:30:15.696312+0000","last_fullsized":"2024-03-06T07:30:15.696312+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:44.503304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695489+0000","last_change":"2024-03-06T07:30:14.086449+0000","last_active":"2024-03-06T07:30:15.695489+0000","last_peered":"2024-03-06T07:30:15.695489+0000","last_clean":"2024-03-06T07:30:15.695489+0000","last_became_active":"2024-03-06T07:30:14.086133+0000","last_became_peered":"2024-03-06T07:30:14.086133+0000","last_unstale":"2024-03-06T07:30:15.695489+0000","last_undegraded":"2024-03-06T07:30:15.695489+0000","last_fullsized":"2024-03-06T07:30:15.695489+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:42.117837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695692+0000","last_change":"2024-03-06T07:30:13.093866+0000","last_active":"2024-03-06T07:30:15.695692+0000","last_peered":"2024-03-06T07:30:15.695692+0000","last_clean":"2024-03-06T07:30:15.695692+0000","last_became_active":"2024-03-06T07:30:13.093556+0000","last_became_peered":"2024-03-06T07:30:13.093556+0000","last_unstale":"2024-03-06T07:30:15.695692+0000","last_undegraded":"2024-03-06T07:30:15.695692+0000","last_fullsized":"2024-03-06T07:30:15.695692+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:41.037432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695351+0000","last_change":"2024-03-06T07:30:14.094690+0000","last_active":"2024-03-06T07:30:15.695351+0000","last_peered":"2024-03-06T07:30:15.695351+0000","last_clean":"2024-03-06T07:30:15.695351+0000","last_became_active":"2024-03-06T07:30:14.093765+0000","last_became_peered":"2024-03-06T07:30:14.093765+0000","last_unstale":"2024-03-06T07:30:15.695351+0000","last_undegraded":"2024-03-06T07:30:15.695351+0000","last_fullsized":"2024-03-06T07:30:15.695351+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:22.529185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696296+0000","last_change":"2024-03-06T07:30:13.095125+0000","last_active":"2024-03-06T07:30:15.696296+0000","last_peered":"2024-03-06T07:30:15.696296+0000","last_clean":"2024-03-06T07:30:15.696296+0000","last_became_active":"2024-03-06T07:30:13.094730+0000","last_became_peered":"2024-03-06T07:30:13.094730+0000","last_unstale":"2024-03-06T07:30:15.696296+0000","last_undegraded":"2024-03-06T07:30:15.696296+0000","last_fullsized":"2024-03-06T07:30:15.696296+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:31.409738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695733+0000","last_change":"2024-03-06T07:30:14.092114+0000","last_active":"2024-03-06T07:30:15.695733+0000","last_peered":"2024-03-06T07:30:15.695733+0000","last_clean":"2024-03-06T07:30:15.695733+0000","last_became_active":"2024-03-06T07:30:14.091766+0000","last_became_peered":"2024-03-06T07:30:14.091766+0000","last_unstale":"2024-03-06T07:30:15.695733+0000","last_undegraded":"2024-03-06T07:30:15.695733+0000","last_fullsized":"2024-03-06T07:30:15.695733+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:42:05.682107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696499+0000","last_change":"2024-03-06T07:30:13.094131+0000","last_active":"2024-03-06T07:30:15.696499+0000","last_peered":"2024-03-06T07:30:15.696499+0000","last_clean":"2024-03-06T07:30:15.696499+0000","last_became_active":"2024-03-06T07:30:13.093833+0000","last_became_peered":"2024-03-06T07:30:13.093833+0000","last_unstale":"2024-03-06T07:30:15.696499+0000","last_undegraded":"2024-03-06T07:30:15.696499+0000","last_fullsized":"2024-03-06T07:30:15.696499+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:21.138766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695214+0000","last_change":"2024-03-06T07:30:14.097724+0000","last_active":"2024-03-06T07:30:15.695214+0000","last_peered":"2024-03-06T07:30:15.695214+0000","last_clean":"2024-03-06T07:30:15.695214+0000","last_became_active":"2024-03-06T07:30:14.097441+0000","last_became_peered":"2024-03-06T07:30:14.097441+0000","last_unstale":"2024-03-06T07:30:15.695214+0000","last_undegraded":"2024-03-06T07:30:15.695214+0000","last_fullsized":"2024-03-06T07:30:15.695214+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:12.290001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.078859+0000","last_change":"2024-03-06T07:30:13.093301+0000","last_active":"2024-03-06T07:30:15.078859+0000","last_peered":"2024-03-06T07:30:15.078859+0000","last_clean":"2024-03-06T07:30:15.078859+0000","last_became_active":"2024-03-06T07:30:13.092932+0000","last_became_peered":"2024-03-06T07:30:13.092932+0000","last_unstale":"2024-03-06T07:30:15.078859+0000","last_undegraded":"2024-03-06T07:30:15.078859+0000","last_fullsized":"2024-03-06T07:30:15.078859+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:57.270355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696013+0000","last_change":"2024-03-06T07:30:14.103126+0000","last_active":"2024-03-06T07:30:15.696013+0000","last_peered":"2024-03-06T07:30:15.696013+0000","last_clean":"2024-03-06T07:30:15.696013+0000","last_became_active":"2024-03-06T07:30:14.102362+0000","last_became_peered":"2024-03-06T07:30:14.102362+0000","last_unstale":"2024-03-06T07:30:15.696013+0000","last_undegraded":"2024-03-06T07:30:15.696013+0000","last_fullsized":"2024-03-06T07:30:15.696013+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:55.359146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083788+0000","last_change":"2024-03-06T07:30:13.094270+0000","last_active":"2024-03-06T07:30:15.083788+0000","last_peered":"2024-03-06T07:30:15.083788+0000","last_clean":"2024-03-06T07:30:15.083788+0000","last_became_active":"2024-03-06T07:30:13.093236+0000","last_became_peered":"2024-03-06T07:30:13.093236+0000","last_unstale":"2024-03-06T07:30:15.083788+0000","last_undegraded":"2024-03-06T07:30:15.083788+0000","last_fullsized":"2024-03-06T07:30:15.083788+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:25.637847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696052+0000","last_change":"2024-03-06T07:30:14.082827+0000","last_active":"2024-03-06T07:30:15.696052+0000","last_peered":"2024-03-06T07:30:15.696052+0000","last_clean":"2024-03-06T07:30:15.696052+0000","last_became_active":"2024-03-06T07:30:14.082529+0000","last_became_peered":"2024-03-06T07:30:14.082529+0000","last_unstale":"2024-03-06T07:30:15.696052+0000","last_undegraded":"2024-03-06T07:30:15.696052+0000","last_fullsized":"2024-03-06T07:30:15.696052+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:19.135167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.078886+0000","last_change":"2024-03-06T07:30:13.084799+0000","last_active":"2024-03-06T07:30:15.078886+0000","last_peered":"2024-03-06T07:30:15.078886+0000","last_clean":"2024-03-06T07:30:15.078886+0000","last_became_active":"2024-03-06T07:30:13.084493+0000","last_became_peered":"2024-03-06T07:30:13.084493+0000","last_unstale":"2024-03-06T07:30:15.078886+0000","last_undegraded":"2024-03-06T07:30:15.078886+0000","last_fullsized":"2024-03-06T07:30:15.078886+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:44:16.410974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.082993+0000","last_change":"2024-03-06T07:30:14.091979+0000","last_active":"2024-03-06T07:30:15.082993+0000","last_peered":"2024-03-06T07:30:15.082993+0000","last_clean":"2024-03-06T07:30:15.082993+0000","last_became_active":"2024-03-06T07:30:14.091707+0000","last_became_peered":"2024-03-06T07:30:14.091707+0000","last_unstale":"2024-03-06T07:30:15.082993+0000","last_undegraded":"2024-03-06T07:30:15.082993+0000","last_fullsized":"2024-03-06T07:30:15.082993+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:23.813232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695321+0000","last_change":"2024-03-06T07:30:13.093874+0000","last_active":"2024-03-06T07:30:15.695321+0000","last_peered":"2024-03-06T07:30:15.695321+0000","last_clean":"2024-03-06T07:30:15.695321+0000","last_became_active":"2024-03-06T07:30:13.093610+0000","last_became_peered":"2024-03-06T07:30:13.093610+0000","last_unstale":"2024-03-06T07:30:15.695321+0000","last_undegraded":"2024-03-06T07:30:15.695321+0000","last_fullsized":"2024-03-06T07:30:15.695321+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:18.390208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083829+0000","last_change":"2024-03-06T07:30:14.088887+0000","last_active":"2024-03-06T07:30:15.083829+0000","last_peered":"2024-03-06T07:30:15.083829+0000","last_clean":"2024-03-06T07:30:15.083829+0000","last_became_active":"2024-03-06T07:30:14.088633+0000","last_became_peered":"2024-03-06T07:30:14.088633+0000","last_unstale":"2024-03-06T07:30:15.083829+0000","last_undegraded":"2024-03-06T07:30:15.083829+0000","last_fullsized":"2024-03-06T07:30:15.083829+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:58.486488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695586+0000","last_change":"2024-03-06T07:30:13.089648+0000","last_active":"2024-03-06T07:30:15.695586+0000","last_peered":"2024-03-06T07:30:15.695586+0000","last_clean":"2024-03-06T07:30:15.695586+0000","last_became_active":"2024-03-06T07:30:13.089344+0000","last_became_peered":"2024-03-06T07:30:13.089344+0000","last_unstale":"2024-03-06T07:30:15.695586+0000","last_undegraded":"2024-03-06T07:30:15.695586+0000","last_fullsized":"2024-03-06T07:30:15.695586+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:45.669588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696390+0000","last_change":"2024-03-06T07:30:14.097445+0000","last_active":"2024-03-06T07:30:15.696390+0000","last_peered":"2024-03-06T07:30:15.696390+0000","last_clean":"2024-03-06T07:30:15.696390+0000","last_became_active":"2024-03-06T07:30:14.096891+0000","last_became_peered":"2024-03-06T07:30:14.096891+0000","last_unstale":"2024-03-06T07:30:15.696390+0000","last_undegraded":"2024-03-06T07:30:15.696390+0000","last_fullsized":"2024-03-06T07:30:15.696390+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:42.109692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696651+0000","last_change":"2024-03-06T07:30:13.094858+0000","last_active":"2024-03-06T07:30:15.696651+0000","last_peered":"2024-03-06T07:30:15.696651+0000","last_clean":"2024-03-06T07:30:15.696651+0000","last_became_active":"2024-03-06T07:30:13.094587+0000","last_became_peered":"2024-03-06T07:30:13.094587+0000","last_unstale":"2024-03-06T07:30:15.696651+0000","last_undegraded":"2024-03-06T07:30:15.696651+0000","last_fullsized":"2024-03-06T07:30:15.696651+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:45.669588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.079113+0000","last_change":"2024-03-06T07:30:14.098169+0000","last_active":"2024-03-06T07:30:15.079113+0000","last_peered":"2024-03-06T07:30:15.079113+0000","last_clean":"2024-03-06T07:30:15.079113+0000","last_became_active":"2024-03-06T07:30:14.097863+0000","last_became_peered":"2024-03-06T07:30:14.097863+0000","last_unstale":"2024-03-06T07:30:15.079113+0000","last_undegraded":"2024-03-06T07:30:15.079113+0000","last_fullsized":"2024-03-06T07:30:15.079113+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:07.475117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"138'6","reported_seq":20,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696027+0000","last_change":"2024-03-06T07:30:13.092492+0000","last_active":"2024-03-06T07:30:15.696027+0000","last_peered":"2024-03-06T07:30:15.696027+0000","last_clean":"2024-03-06T07:30:15.696027+0000","last_became_active":"2024-03-06T07:30:13.092188+0000","last_became_peered":"2024-03-06T07:30:13.092188+0000","last_unstale":"2024-03-06T07:30:15.696027+0000","last_undegraded":"2024-03-06T07:30:15.696027+0000","last_fullsized":"2024-03-06T07:30:15.696027+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:10.690646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083208+0000","last_change":"2024-03-06T07:30:14.097644+0000","last_active":"2024-03-06T07:30:15.083208+0000","last_peered":"2024-03-06T07:30:15.083208+0000","last_clean":"2024-03-06T07:30:15.083208+0000","last_became_active":"2024-03-06T07:30:14.097362+0000","last_became_peered":"2024-03-06T07:30:14.097362+0000","last_unstale":"2024-03-06T07:30:15.083208+0000","last_undegraded":"2024-03-06T07:30:15.083208+0000","last_fullsized":"2024-03-06T07:30:15.083208+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:24:29.812042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083395+0000","last_change":"2024-03-06T07:30:13.089149+0000","last_active":"2024-03-06T07:30:15.083395+0000","last_peered":"2024-03-06T07:30:15.083395+0000","last_clean":"2024-03-06T07:30:15.083395+0000","last_became_active":"2024-03-06T07:30:13.088882+0000","last_became_peered":"2024-03-06T07:30:13.088882+0000","last_unstale":"2024-03-06T07:30:15.083395+0000","last_undegraded":"2024-03-06T07:30:15.083395+0000","last_fullsized":"2024-03-06T07:30:15.083395+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:18.031849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695180+0000","last_change":"2024-03-06T07:30:14.085531+0000","last_active":"2024-03-06T07:30:15.695180+0000","last_peered":"2024-03-06T07:30:15.695180+0000","last_clean":"2024-03-06T07:30:15.695180+0000","last_became_active":"2024-03-06T07:30:14.085213+0000","last_became_peered":"2024-03-06T07:30:14.085213+0000","last_unstale":"2024-03-06T07:30:15.695180+0000","last_undegraded":"2024-03-06T07:30:15.695180+0000","last_fullsized":"2024-03-06T07:30:15.695180+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:40.521861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.082883+0000","last_change":"2024-03-06T07:30:13.090031+0000","last_active":"2024-03-06T07:30:15.082883+0000","last_peered":"2024-03-06T07:30:15.082883+0000","last_clean":"2024-03-06T07:30:15.082883+0000","last_became_active":"2024-03-06T07:30:13.089747+0000","last_became_peered":"2024-03-06T07:30:13.089747+0000","last_unstale":"2024-03-06T07:30:15.082883+0000","last_undegraded":"2024-03-06T07:30:15.082883+0000","last_fullsized":"2024-03-06T07:30:15.082883+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:02.921882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.082998+0000","last_change":"2024-03-06T07:30:14.096878+0000","last_active":"2024-03-06T07:30:15.082998+0000","last_peered":"2024-03-06T07:30:15.082998+0000","last_clean":"2024-03-06T07:30:15.082998+0000","last_became_active":"2024-03-06T07:30:14.096571+0000","last_became_peered":"2024-03-06T07:30:14.096571+0000","last_unstale":"2024-03-06T07:30:15.082998+0000","last_undegraded":"2024-03-06T07:30:15.082998+0000","last_fullsized":"2024-03-06T07:30:15.082998+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:03.927112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695165+0000","last_change":"2024-03-06T07:30:13.087971+0000","last_active":"2024-03-06T07:30:15.695165+0000","last_peered":"2024-03-06T07:30:15.695165+0000","last_clean":"2024-03-06T07:30:15.695165+0000","last_became_active":"2024-03-06T07:30:13.087048+0000","last_became_peered":"2024-03-06T07:30:13.087048+0000","last_unstale":"2024-03-06T07:30:15.695165+0000","last_undegraded":"2024-03-06T07:30:15.695165+0000","last_fullsized":"2024-03-06T07:30:15.695165+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:07.431275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.082528+0000","last_change":"2024-03-06T07:30:14.093128+0000","last_active":"2024-03-06T07:30:15.082528+0000","last_peered":"2024-03-06T07:30:15.082528+0000","last_clean":"2024-03-06T07:30:15.082528+0000","last_became_active":"2024-03-06T07:30:14.092852+0000","last_became_peered":"2024-03-06T07:30:14.092852+0000","last_unstale":"2024-03-06T07:30:15.082528+0000","last_undegraded":"2024-03-06T07:30:15.082528+0000","last_fullsized":"2024-03-06T07:30:15.082528+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:02.834051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"138'8","reported_seq":22,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.078955+0000","last_change":"2024-03-06T07:30:13.089054+0000","last_active":"2024-03-06T07:30:15.078955+0000","last_peered":"2024-03-06T07:30:15.078955+0000","last_clean":"2024-03-06T07:30:15.078955+0000","last_became_active":"2024-03-06T07:30:13.088745+0000","last_became_peered":"2024-03-06T07:30:13.088745+0000","last_unstale":"2024-03-06T07:30:15.078955+0000","last_undegraded":"2024-03-06T07:30:15.078955+0000","last_fullsized":"2024-03-06T07:30:15.078955+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:16.523275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696593+0000","last_change":"2024-03-06T07:30:14.104102+0000","last_active":"2024-03-06T07:30:15.696593+0000","last_peered":"2024-03-06T07:30:15.696593+0000","last_clean":"2024-03-06T07:30:15.696593+0000","last_became_active":"2024-03-06T07:30:14.103830+0000","last_became_peered":"2024-03-06T07:30:14.103830+0000","last_unstale":"2024-03-06T07:30:15.696593+0000","last_undegraded":"2024-03-06T07:30:15.696593+0000","last_fullsized":"2024-03-06T07:30:15.696593+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:33.957477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083167+0000","last_change":"2024-03-06T07:30:13.092367+0000","last_active":"2024-03-06T07:30:15.083167+0000","last_peered":"2024-03-06T07:30:15.083167+0000","last_clean":"2024-03-06T07:30:15.083167+0000","last_became_active":"2024-03-06T07:30:13.091033+0000","last_became_peered":"2024-03-06T07:30:13.091033+0000","last_unstale":"2024-03-06T07:30:15.083167+0000","last_undegraded":"2024-03-06T07:30:15.083167+0000","last_fullsized":"2024-03-06T07:30:15.083167+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:02.177629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.079027+0000","last_change":"2024-03-06T07:30:14.093488+0000","last_active":"2024-03-06T07:30:15.079027+0000","last_peered":"2024-03-06T07:30:15.079027+0000","last_clean":"2024-03-06T07:30:15.079027+0000","last_became_active":"2024-03-06T07:30:14.093225+0000","last_became_peered":"2024-03-06T07:30:14.093225+0000","last_unstale":"2024-03-06T07:30:15.079027+0000","last_undegraded":"2024-03-06T07:30:15.079027+0000","last_fullsized":"2024-03-06T07:30:15.079027+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:56.933025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695797+0000","last_change":"2024-03-06T07:30:13.094981+0000","last_active":"2024-03-06T07:30:15.695797+0000","last_peered":"2024-03-06T07:30:15.695797+0000","last_clean":"2024-03-06T07:30:15.695797+0000","last_became_active":"2024-03-06T07:30:13.094652+0000","last_became_peered":"2024-03-06T07:30:13.094652+0000","last_unstale":"2024-03-06T07:30:15.695797+0000","last_undegraded":"2024-03-06T07:30:15.695797+0000","last_fullsized":"2024-03-06T07:30:15.695797+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:31:55.081668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083076+0000","last_change":"2024-03-06T07:30:14.093702+0000","last_active":"2024-03-06T07:30:15.083076+0000","last_peered":"2024-03-06T07:30:15.083076+0000","last_clean":"2024-03-06T07:30:15.083076+0000","last_became_active":"2024-03-06T07:30:14.093351+0000","last_became_peered":"2024-03-06T07:30:14.093351+0000","last_unstale":"2024-03-06T07:30:15.083076+0000","last_undegraded":"2024-03-06T07:30:15.083076+0000","last_fullsized":"2024-03-06T07:30:15.083076+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:27:52.956777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"138'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.079113+0000","last_change":"2024-03-06T07:30:13.093826+0000","last_active":"2024-03-06T07:30:15.079113+0000","last_peered":"2024-03-06T07:30:15.079113+0000","last_clean":"2024-03-06T07:30:15.079113+0000","last_became_active":"2024-03-06T07:30:13.093514+0000","last_became_peered":"2024-03-06T07:30:13.093514+0000","last_unstale":"2024-03-06T07:30:15.079113+0000","last_undegraded":"2024-03-06T07:30:15.079113+0000","last_fullsized":"2024-03-06T07:30:15.079113+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:33.438949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695987+0000","last_change":"2024-03-06T07:30:14.098713+0000","last_active":"2024-03-06T07:30:15.695987+0000","last_peered":"2024-03-06T07:30:15.695987+0000","last_clean":"2024-03-06T07:30:15.695987+0000","last_became_active":"2024-03-06T07:30:14.098432+0000","last_became_peered":"2024-03-06T07:30:14.098432+0000","last_unstale":"2024-03-06T07:30:15.695987+0000","last_undegraded":"2024-03-06T07:30:15.695987+0000","last_fullsized":"2024-03-06T07:30:15.695987+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:33.360610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695965+0000","last_change":"2024-03-06T07:30:13.090542+0000","last_active":"2024-03-06T07:30:15.695965+0000","last_peered":"2024-03-06T07:30:15.695965+0000","last_clean":"2024-03-06T07:30:15.695965+0000","last_became_active":"2024-03-06T07:30:13.090263+0000","last_became_peered":"2024-03-06T07:30:13.090263+0000","last_unstale":"2024-03-06T07:30:15.695965+0000","last_undegraded":"2024-03-06T07:30:15.695965+0000","last_fullsized":"2024-03-06T07:30:15.695965+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:07:57.188994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696220+0000","last_change":"2024-03-06T07:30:14.095317+0000","last_active":"2024-03-06T07:30:15.696220+0000","last_peered":"2024-03-06T07:30:15.696220+0000","last_clean":"2024-03-06T07:30:15.696220+0000","last_became_active":"2024-03-06T07:30:14.095031+0000","last_became_peered":"2024-03-06T07:30:14.095031+0000","last_unstale":"2024-03-06T07:30:15.696220+0000","last_undegraded":"2024-03-06T07:30:15.696220+0000","last_fullsized":"2024-03-06T07:30:15.696220+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:45.508534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696428+0000","last_change":"2024-03-06T07:30:13.088590+0000","last_active":"2024-03-06T07:30:15.696428+0000","last_peered":"2024-03-06T07:30:15.696428+0000","last_clean":"2024-03-06T07:30:15.696428+0000","last_became_active":"2024-03-06T07:30:13.088319+0000","last_became_peered":"2024-03-06T07:30:13.088319+0000","last_unstale":"2024-03-06T07:30:15.696428+0000","last_undegraded":"2024-03-06T07:30:15.696428+0000","last_fullsized":"2024-03-06T07:30:15.696428+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:57.780833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696327+0000","last_change":"2024-03-06T07:30:14.097199+0000","last_active":"2024-03-06T07:30:15.696327+0000","last_peered":"2024-03-06T07:30:15.696327+0000","last_clean":"2024-03-06T07:30:15.696327+0000","last_became_active":"2024-03-06T07:30:14.096871+0000","last_became_peered":"2024-03-06T07:30:14.096871+0000","last_unstale":"2024-03-06T07:30:15.696327+0000","last_undegraded":"2024-03-06T07:30:15.696327+0000","last_fullsized":"2024-03-06T07:30:15.696327+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:21.750545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695761+0000","last_change":"2024-03-06T07:30:13.096163+0000","last_active":"2024-03-06T07:30:15.695761+0000","last_peered":"2024-03-06T07:30:15.695761+0000","last_clean":"2024-03-06T07:30:15.695761+0000","last_became_active":"2024-03-06T07:30:13.095814+0000","last_became_peered":"2024-03-06T07:30:13.095814+0000","last_unstale":"2024-03-06T07:30:15.695761+0000","last_undegraded":"2024-03-06T07:30:15.695761+0000","last_fullsized":"2024-03-06T07:30:15.695761+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:54.353916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695309+0000","last_change":"2024-03-06T07:30:14.089795+0000","last_active":"2024-03-06T07:30:15.695309+0000","last_peered":"2024-03-06T07:30:15.695309+0000","last_clean":"2024-03-06T07:30:15.695309+0000","last_became_active":"2024-03-06T07:30:14.089492+0000","last_became_peered":"2024-03-06T07:30:14.089492+0000","last_unstale":"2024-03-06T07:30:15.695309+0000","last_undegraded":"2024-03-06T07:30:15.695309+0000","last_fullsized":"2024-03-06T07:30:15.695309+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:36.191719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695773+0000","last_change":"2024-03-06T07:30:13.091318+0000","last_active":"2024-03-06T07:30:15.695773+0000","last_peered":"2024-03-06T07:30:15.695773+0000","last_clean":"2024-03-06T07:30:15.695773+0000","last_became_active":"2024-03-06T07:30:13.091037+0000","last_became_peered":"2024-03-06T07:30:13.091037+0000","last_unstale":"2024-03-06T07:30:15.695773+0000","last_undegraded":"2024-03-06T07:30:15.695773+0000","last_fullsized":"2024-03-06T07:30:15.695773+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:07.128080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695605+0000","last_change":"2024-03-06T07:30:14.093119+0000","last_active":"2024-03-06T07:30:15.695605+0000","last_peered":"2024-03-06T07:30:15.695605+0000","last_clean":"2024-03-06T07:30:15.695605+0000","last_became_active":"2024-03-06T07:30:14.092810+0000","last_became_peered":"2024-03-06T07:30:14.092810+0000","last_unstale":"2024-03-06T07:30:15.695605+0000","last_undegraded":"2024-03-06T07:30:15.695605+0000","last_fullsized":"2024-03-06T07:30:15.695605+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:12.290001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.082421+0000","last_change":"2024-03-06T07:30:13.090104+0000","last_active":"2024-03-06T07:30:15.082421+0000","last_peered":"2024-03-06T07:30:15.082421+0000","last_clean":"2024-03-06T07:30:15.082421+0000","last_became_active":"2024-03-06T07:30:13.089808+0000","last_became_peered":"2024-03-06T07:30:13.089808+0000","last_unstale":"2024-03-06T07:30:15.082421+0000","last_undegraded":"2024-03-06T07:30:15.082421+0000","last_fullsized":"2024-03-06T07:30:15.082421+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:15.411809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083868+0000","last_change":"2024-03-06T07:30:14.095342+0000","last_active":"2024-03-06T07:30:15.083868+0000","last_peered":"2024-03-06T07:30:15.083868+0000","last_clean":"2024-03-06T07:30:15.083868+0000","last_became_active":"2024-03-06T07:30:14.095084+0000","last_became_peered":"2024-03-06T07:30:14.095084+0000","last_unstale":"2024-03-06T07:30:15.083868+0000","last_undegraded":"2024-03-06T07:30:15.083868+0000","last_fullsized":"2024-03-06T07:30:15.083868+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:47:08.436505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"138'2","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696498+0000","last_change":"2024-03-06T07:30:13.097051+0000","last_active":"2024-03-06T07:30:15.696498+0000","last_peered":"2024-03-06T07:30:15.696498+0000","last_clean":"2024-03-06T07:30:15.696498+0000","last_became_active":"2024-03-06T07:30:13.096782+0000","last_became_peered":"2024-03-06T07:30:13.096782+0000","last_unstale":"2024-03-06T07:30:15.696498+0000","last_undegraded":"2024-03-06T07:30:15.696498+0000","last_fullsized":"2024-03-06T07:30:15.696498+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:28:35.507410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083604+0000","last_change":"2024-03-06T07:30:14.102876+0000","last_active":"2024-03-06T07:30:15.083604+0000","last_peered":"2024-03-06T07:30:15.083604+0000","last_clean":"2024-03-06T07:30:15.083604+0000","last_became_active":"2024-03-06T07:30:14.102448+0000","last_became_peered":"2024-03-06T07:30:14.102448+0000","last_unstale":"2024-03-06T07:30:15.083604+0000","last_undegraded":"2024-03-06T07:30:15.083604+0000","last_fullsized":"2024-03-06T07:30:15.083604+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:04.282911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"139'1","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:16.082782+0000","last_change":"2024-03-06T07:30:13.093268+0000","last_active":"2024-03-06T07:30:16.082782+0000","last_peered":"2024-03-06T07:30:16.082782+0000","last_clean":"2024-03-06T07:30:16.082782+0000","last_became_active":"2024-03-06T07:30:13.092584+0000","last_became_peered":"2024-03-06T07:30:13.092584+0000","last_unstale":"2024-03-06T07:30:16.082782+0000","last_undegraded":"2024-03-06T07:30:16.082782+0000","last_fullsized":"2024-03-06T07:30:16.082782+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:01.828821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695873+0000","last_change":"2024-03-06T07:30:14.093186+0000","last_active":"2024-03-06T07:30:15.695873+0000","last_peered":"2024-03-06T07:30:15.695873+0000","last_clean":"2024-03-06T07:30:15.695873+0000","last_became_active":"2024-03-06T07:30:14.092888+0000","last_became_peered":"2024-03-06T07:30:14.092888+0000","last_unstale":"2024-03-06T07:30:15.695873+0000","last_undegraded":"2024-03-06T07:30:15.695873+0000","last_fullsized":"2024-03-06T07:30:15.695873+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:10.834419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696231+0000","last_change":"2024-03-06T07:30:13.092729+0000","last_active":"2024-03-06T07:30:15.696231+0000","last_peered":"2024-03-06T07:30:15.696231+0000","last_clean":"2024-03-06T07:30:15.696231+0000","last_became_active":"2024-03-06T07:30:13.092236+0000","last_became_peered":"2024-03-06T07:30:13.092236+0000","last_unstale":"2024-03-06T07:30:15.696231+0000","last_undegraded":"2024-03-06T07:30:15.696231+0000","last_fullsized":"2024-03-06T07:30:15.696231+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:54:09.956085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696511+0000","last_change":"2024-03-06T07:30:14.093332+0000","last_active":"2024-03-06T07:30:15.696511+0000","last_peered":"2024-03-06T07:30:15.696511+0000","last_clean":"2024-03-06T07:30:15.696511+0000","last_became_active":"2024-03-06T07:30:14.093073+0000","last_became_peered":"2024-03-06T07:30:14.093073+0000","last_unstale":"2024-03-06T07:30:15.696511+0000","last_undegraded":"2024-03-06T07:30:15.696511+0000","last_fullsized":"2024-03-06T07:30:15.696511+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:42.160137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695763+0000","last_change":"2024-03-06T07:30:13.092659+0000","last_active":"2024-03-06T07:30:15.695763+0000","last_peered":"2024-03-06T07:30:15.695763+0000","last_clean":"2024-03-06T07:30:15.695763+0000","last_became_active":"2024-03-06T07:30:13.092355+0000","last_became_peered":"2024-03-06T07:30:13.092355+0000","last_unstale":"2024-03-06T07:30:15.695763+0000","last_undegraded":"2024-03-06T07:30:15.695763+0000","last_fullsized":"2024-03-06T07:30:15.695763+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:33:57.481258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.078832+0000","last_change":"2024-03-06T07:30:14.099152+0000","last_active":"2024-03-06T07:30:15.078832+0000","last_peered":"2024-03-06T07:30:15.078832+0000","last_clean":"2024-03-06T07:30:15.078832+0000","last_became_active":"2024-03-06T07:30:14.098866+0000","last_became_peered":"2024-03-06T07:30:14.098866+0000","last_unstale":"2024-03-06T07:30:15.078832+0000","last_undegraded":"2024-03-06T07:30:15.078832+0000","last_fullsized":"2024-03-06T07:30:15.078832+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:26.643077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696494+0000","last_change":"2024-03-06T07:30:13.092062+0000","last_active":"2024-03-06T07:30:15.696494+0000","last_peered":"2024-03-06T07:30:15.696494+0000","last_clean":"2024-03-06T07:30:15.696494+0000","last_became_active":"2024-03-06T07:30:13.091779+0000","last_became_peered":"2024-03-06T07:30:13.091779+0000","last_unstale":"2024-03-06T07:30:15.696494+0000","last_undegraded":"2024-03-06T07:30:15.696494+0000","last_fullsized":"2024-03-06T07:30:15.696494+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:00:57.434186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083610+0000","last_change":"2024-03-06T07:30:14.094169+0000","last_active":"2024-03-06T07:30:15.083610+0000","last_peered":"2024-03-06T07:30:15.083610+0000","last_clean":"2024-03-06T07:30:15.083610+0000","last_became_active":"2024-03-06T07:30:14.093918+0000","last_became_peered":"2024-03-06T07:30:14.093918+0000","last_unstale":"2024-03-06T07:30:15.083610+0000","last_undegraded":"2024-03-06T07:30:15.083610+0000","last_fullsized":"2024-03-06T07:30:15.083610+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:04.384314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695500+0000","last_change":"2024-03-06T07:30:13.097022+0000","last_active":"2024-03-06T07:30:15.695500+0000","last_peered":"2024-03-06T07:30:15.695500+0000","last_clean":"2024-03-06T07:30:15.695500+0000","last_became_active":"2024-03-06T07:30:13.096747+0000","last_became_peered":"2024-03-06T07:30:13.096747+0000","last_unstale":"2024-03-06T07:30:15.695500+0000","last_undegraded":"2024-03-06T07:30:15.695500+0000","last_fullsized":"2024-03-06T07:30:15.695500+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:41.154906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695649+0000","last_change":"2024-03-06T07:30:14.102541+0000","last_active":"2024-03-06T07:30:15.695649+0000","last_peered":"2024-03-06T07:30:15.695649+0000","last_clean":"2024-03-06T07:30:15.695649+0000","last_became_active":"2024-03-06T07:30:14.101478+0000","last_became_peered":"2024-03-06T07:30:14.101478+0000","last_unstale":"2024-03-06T07:30:15.695649+0000","last_undegraded":"2024-03-06T07:30:15.695649+0000","last_fullsized":"2024-03-06T07:30:15.695649+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:40.521861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.082594+0000","last_change":"2024-03-06T07:30:13.088126+0000","last_active":"2024-03-06T07:30:15.082594+0000","last_peered":"2024-03-06T07:30:15.082594+0000","last_clean":"2024-03-06T07:30:15.082594+0000","last_became_active":"2024-03-06T07:30:13.087838+0000","last_became_peered":"2024-03-06T07:30:13.087838+0000","last_unstale":"2024-03-06T07:30:15.082594+0000","last_undegraded":"2024-03-06T07:30:15.082594+0000","last_fullsized":"2024-03-06T07:30:15.082594+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:22.808002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083117+0000","last_change":"2024-03-06T07:30:14.088248+0000","last_active":"2024-03-06T07:30:15.083117+0000","last_peered":"2024-03-06T07:30:15.083117+0000","last_clean":"2024-03-06T07:30:15.083117+0000","last_became_active":"2024-03-06T07:30:14.087987+0000","last_became_peered":"2024-03-06T07:30:14.087987+0000","last_unstale":"2024-03-06T07:30:15.083117+0000","last_undegraded":"2024-03-06T07:30:15.083117+0000","last_fullsized":"2024-03-06T07:30:15.083117+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:16.417039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.696015+0000","last_change":"2024-03-06T07:30:13.094297+0000","last_active":"2024-03-06T07:30:15.696015+0000","last_peered":"2024-03-06T07:30:15.696015+0000","last_clean":"2024-03-06T07:30:15.696015+0000","last_became_active":"2024-03-06T07:30:13.094021+0000","last_became_peered":"2024-03-06T07:30:13.094021+0000","last_unstale":"2024-03-06T07:30:15.696015+0000","last_undegraded":"2024-03-06T07:30:15.696015+0000","last_fullsized":"2024-03-06T07:30:15.696015+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:20.925257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695926+0000","last_change":"2024-03-06T07:30:14.097713+0000","last_active":"2024-03-06T07:30:15.695926+0000","last_peered":"2024-03-06T07:30:15.695926+0000","last_clean":"2024-03-06T07:30:15.695926+0000","last_became_active":"2024-03-06T07:30:14.097108+0000","last_became_peered":"2024-03-06T07:30:14.097108+0000","last_unstale":"2024-03-06T07:30:15.695926+0000","last_undegraded":"2024-03-06T07:30:15.695926+0000","last_fullsized":"2024-03-06T07:30:15.695926+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:13.068641+0000","last_clean_scrub_stamp":"2024-03-06T07:30:13.068641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:42.117837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"138'3","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.083445+0000","last_change":"2024-03-06T07:30:13.090003+0000","last_active":"2024-03-06T07:30:15.083445+0000","last_peered":"2024-03-06T07:30:15.083445+0000","last_clean":"2024-03-06T07:30:15.083445+0000","last_became_active":"2024-03-06T07:30:13.089626+0000","last_became_peered":"2024-03-06T07:30:13.089626+0000","last_unstale":"2024-03-06T07:30:15.083445+0000","last_undegraded":"2024-03-06T07:30:15.083445+0000","last_fullsized":"2024-03-06T07:30:15.083445+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:12.063411+0000","last_clean_scrub_stamp":"2024-03-06T07:30:12.063411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:43.203477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":385,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-06T07:30:15.695721+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:30:15.695721+0000","last_peered":"2024-03-06T07:30:15.695721+0000","last_clean":"2024-03-06T07:30:15.695721+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:30:15.695721+0000","last_undegraded":"2024-03-06T07:30:15.695721+0000","last_fullsized":"2024-03-06T07:30:15.695721+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574954,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41796,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330044,"statfs":{"total":96636764160,"available":96593965056,"internally_reserved":0,"allocated":2342912,"data_stored":1354144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574955,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41216,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330624,"statfs":{"total":96636764160,"available":96594558976,"internally_reserved":0,"allocated":1753088,"data_stored":763270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574954,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41208,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330632,"statfs":{"total":96636764160,"available":96594567168,"internally_reserved":0,"allocated":1744896,"data_stored":763146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574954,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41208,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330632,"statfs":{"total":96636764160,"available":96594567168,"internally_reserved":0,"allocated":1740800,"data_stored":763124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574954,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41196,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330644,"statfs":{"total":96636764160,"available":96594579456,"internally_reserved":0,"allocated":1732608,"data_stored":756914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574953,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41220,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330620,"statfs":{"total":96636764160,"available":96594554880,"internally_reserved":0,"allocated":1757184,"data_stored":764806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41208,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330632,"statfs":{"total":96636764160,"available":96594567168,"internally_reserved":0,"allocated":1744896,"data_stored":763706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574955,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41800,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330040,"statfs":{"total":96636764160,"available":96593960960,"internally_reserved":0,"allocated":2351104,"data_stored":1355138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:17.609 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:30:17.609 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:30:17.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.751+0000 7f5fc99ac640 1 Processor -- start 2024-03-06T07:30:17.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.751+0000 7f5fc99ac640 1 -- start start 2024-03-06T07:30:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.752+0000 7f5fc99ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fc41614d0 0x7f5fc40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.752+0000 7f5fc99ac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5fc40d2480 0x7f5fc40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.752+0000 7f5fc99ac640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc41618b0 con 0x7f5fc40d1910 2024-03-06T07:30:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.752+0000 7f5fc99ac640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc40cf400 con 0x7f5fc40d2480 2024-03-06T07:30:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.752+0000 7f5fc99ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40cf6a0 con 0x7f5fc41614d0 2024-03-06T07:30:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.753+0000 7f5fc37fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5fc40d2480 0x7f5fc40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.753+0000 7f5fc27fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f5fc40d1910 0x7f5fc40d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41552/0 (socket says 172.21.15.59:41552) 2024-03-06T07:30:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.753+0000 7f5fc27fc640 1 -- 172.21.15.59:0/1941482456 learned_addr learned my addr 172.21.15.59:0/1941482456 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.753+0000 7f5fc2ffd640 1 --2- 172.21.15.59:0/1941482456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fc41614d0 0x7f5fc40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.753+0000 7f5fc37fe640 1 -- 172.21.15.59:0/1941482456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fc41614d0 msgr2=0x7f5fc40d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.753+0000 7f5fc37fe640 1 --2- 172.21.15.59:0/1941482456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fc41614d0 0x7f5fc40d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.753+0000 7f5fc37fe640 1 -- 172.21.15.59:0/1941482456 >> v1:172.21.15.59:6789/0 conn(0x7f5fc40d1910 legacy=0x7f5fc40d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.753+0000 7f5fc37fe640 1 -- 172.21.15.59:0/1941482456 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc40cf980 con 0x7f5fc40d2480 2024-03-06T07:30:17.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.753+0000 7f5fc1ffb640 1 -- 172.21.15.59:0/1941482456 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760879859 0 0) 0x7f5fc41618b0 con 0x7f5fc40d1910 2024-03-06T07:30:17.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.754+0000 7f5fc37fe640 1 --2- 172.21.15.59:0/1941482456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc40d2480 0x7f5fc40b0e30 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009d90 tx=0x7f5fbc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6984866f511b5d90 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.754+0000 7f5fc1ffb640 1 -- 172.21.15.59:0/1941482456 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc039070 con 0x7f5fc40d2480 2024-03-06T07:30:17.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.754+0000 7f5fc1ffb640 1 -- 172.21.15.59:0/1941482456 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc02cc50 con 0x7f5fc40d2480 2024-03-06T07:30:17.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.754+0000 7f5fc1ffb640 1 -- 172.21.15.59:0/1941482456 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc034440 con 0x7f5fc40d2480 2024-03-06T07:30:17.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.754+0000 7f5fc99ac640 1 -- 172.21.15.59:0/1941482456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc40d2480 msgr2=0x7f5fc40b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.754+0000 7f5fc99ac640 1 --2- 172.21.15.59:0/1941482456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc40d2480 0x7f5fc40b0e30 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009d90 tx=0x7f5fbc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.755+0000 7f5fc99ac640 1 -- 172.21.15.59:0/1941482456 shutdown_connections 2024-03-06T07:30:17.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.755+0000 7f5fc99ac640 1 --2- 172.21.15.59:0/1941482456 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc40d2480 0x7f5fc40b0e30 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.755+0000 7f5fc99ac640 1 --2- 172.21.15.59:0/1941482456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fc41614d0 0x7f5fc40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.755+0000 7f5fc99ac640 1 -- 172.21.15.59:0/1941482456 >> 172.21.15.59:0/1941482456 conn(0x7f5fc40d3d40 msgr2=0x7f5fc40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.755+0000 7f5fc99ac640 1 -- 172.21.15.59:0/1941482456 shutdown_connections 2024-03-06T07:30:17.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.755+0000 7f5fc99ac640 1 -- 172.21.15.59:0/1941482456 wait complete. 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.756+0000 7f5fc99ac640 1 Processor -- start 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.756+0000 7f5fc99ac640 1 -- start start 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc99ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc40d1910 0x7f5fc40b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc99ac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fc40d2480 0x7f5fc41f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc99ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fc41614d0 0x7f5fc415cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc99ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40b1370 con 0x7f5fc40d1910 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc99ac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40b0910 con 0x7f5fc40d2480 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc99ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40b14f0 con 0x7f5fc41614d0 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc40d1910 0x7f5fc40b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc40d1910 0x7f5fc40b8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36424/0 (socket says 172.21.15.59:36424) 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc2ffd640 1 -- 172.21.15.59:0/838003699 learned_addr learned my addr 172.21.15.59:0/838003699 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc37fe640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fc41614d0 0x7f5fc415cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc27fc640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fc40d2480 0x7f5fc41f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc2ffd640 1 -- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fc41614d0 msgr2=0x7f5fc415cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc2ffd640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fc41614d0 0x7f5fc415cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc2ffd640 1 -- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fc40d2480 msgr2=0x7f5fc41f3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc2ffd640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fc40d2480 0x7f5fc41f3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc2ffd640 1 -- 172.21.15.59:0/838003699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc40c7d70 con 0x7f5fc40d1910 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fc2ffd640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc40d1910 0x7f5fc40b8c00 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400d5f0 tx=0x7f5fb400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.757+0000 7f5fabfff640 1 -- 172.21.15.59:0/838003699 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb40125f0 con 0x7f5fc40d1910 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.758+0000 7f5fabfff640 1 -- 172.21.15.59:0/838003699 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb4019070 con 0x7f5fc40d1910 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.758+0000 7f5fc99ac640 1 -- 172.21.15.59:0/838003699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc40c8050 con 0x7f5fc40d1910 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.758+0000 7f5fabfff640 1 -- 172.21.15.59:0/838003699 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb40129d0 con 0x7f5fc40d1910 2024-03-06T07:30:17.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.758+0000 7f5fc99ac640 1 -- 172.21.15.59:0/838003699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc40c8610 con 0x7f5fc40d1910 2024-03-06T07:30:17.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.759+0000 7f5fc99ac640 1 -- 172.21.15.59:0/838003699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc4068ff0 con 0x7f5fc40d1910 2024-03-06T07:30:17.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.759+0000 7f5fabfff640 1 -- 172.21.15.59:0/838003699 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fb4012b70 con 0x7f5fc40d1910 2024-03-06T07:30:17.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.759+0000 7f5fabfff640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fb006b4d0 0x7f5fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.760+0000 7f5fabfff640 1 -- 172.21.15.59:0/838003699 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fb408a610 con 0x7f5fc40d1910 2024-03-06T07:30:17.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.760+0000 7f5fc27fc640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fb006b4d0 0x7f5fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.760+0000 7f5fc27fc640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fb006b4d0 0x7f5fb006d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40a1830 tx=0x7f5fb8007b50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:17.763+0000 7f5fabfff640 1 -- 172.21.15.59:0/838003699 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fb4059490 con 0x7f5fc40d1910 2024-03-06T07:30:18.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.020+0000 7f5fc99ac640 1 -- 172.21.15.59:0/838003699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f5fc40a1c10 con 0x7f5fc40d1910 2024-03-06T07:30:18.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.022+0000 7f5fabfff640 1 -- 172.21.15.59:0/838003699 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5fb4058d90 con 0x7f5fc40d1910 2024-03-06T07:30:18.024 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2024-03-06T07:30:18.024 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQBTGehlv1b+MBAAUj2EZYj+x6mYfGG5Fx2JOA== 2024-03-06T07:30:18.024 INFO:teuthology.orchestra.run.smithi059.stdout: caps mds = "allow" 2024-03-06T07:30:18.024 INFO:teuthology.orchestra.run.smithi059.stdout: caps mon = "allow r" 2024-03-06T07:30:18.024 INFO:teuthology.orchestra.run.smithi059.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:18.024 INFO:teuthology.orchestra.run.smithi059.stderr:updated caps for client.0 2024-03-06T07:30:18.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 -- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fb006b4d0 msgr2=0x7f5fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fb006b4d0 0x7f5fb006d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40a1830 tx=0x7f5fb8007b50 comp rx=0 tx=0).stop 2024-03-06T07:30:18.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 -- 172.21.15.59:0/838003699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc40d1910 msgr2=0x7f5fc40b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc40d1910 0x7f5fc40b8c00 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f5fb400d5f0 tx=0x7f5fb400dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 -- 172.21.15.59:0/838003699 shutdown_connections 2024-03-06T07:30:18.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fb006b4d0 0x7f5fb006d990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fc41614d0 0x7f5fc415cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fc40d2480 0x7f5fc41f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 --2- 172.21.15.59:0/838003699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fc40d1910 0x7f5fc40b8c00 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 -- 172.21.15.59:0/838003699 >> 172.21.15.59:0/838003699 conn(0x7f5fc40d3d40 msgr2=0x7f5fc40d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 -- 172.21.15.59:0/838003699 shutdown_connections 2024-03-06T07:30:18.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.024+0000 7f5fc99ac640 1 -- 172.21.15.59:0/838003699 wait complete. 2024-03-06T07:30:18.037 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:18.175 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.173+0000 7fcda313e640 1 Processor -- start 2024-03-06T07:30:18.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.173+0000 7fcda313e640 1 -- start start 2024-03-06T07:30:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.174+0000 7fcda313e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcd9c0d2920 0x7fcd9c0d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.174+0000 7fcda313e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c0d0650 0x7fcd9c0d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.174+0000 7fcda313e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c15e470 con 0x7fcd9c15e090 2024-03-06T07:30:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.174+0000 7fcda313e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c0d18f0 con 0x7fcd9c0d2920 2024-03-06T07:30:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.174+0000 7fcda313e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0b3cf0 con 0x7fcd9c0d0650 2024-03-06T07:30:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.174+0000 7fcda0eda640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcd9c15e090 0x7fcd9c0b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41566/0 (socket says 172.21.15.59:41566) 2024-03-06T07:30:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.174+0000 7fcda0eda640 1 -- 172.21.15.59:0/3156877467 learned_addr learned my addr 172.21.15.59:0/3156877467 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888115914 0 0) 0x7fcd9c15e470 con 0x7fcd9c15e090 2024-03-06T07:30:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd84003520 con 0x7fcd9c15e090 2024-03-06T07:30:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9bfff640 1 --2- 172.21.15.59:0/3156877467 >> v2:172.21.15.164:3300/0 conn(0x7fcd9c0d2920 0x7fcd9c0d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcda16db640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c0d0650 0x7fcd9c0d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3007860219 0 0) 0x7fcd84003520 con 0x7fcd9c15e090 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c0d0650 msgr2=0x7fcd9c0d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c0d0650 0x7fcd9c0d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 >> v2:172.21.15.164:3300/0 conn(0x7fcd9c0d2920 msgr2=0x7fcd9c0d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 --2- 172.21.15.59:0/3156877467 >> v2:172.21.15.164:3300/0 conn(0x7fcd9c0d2920 0x7fcd9c0d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c0d1b90 con 0x7fcd9c15e090 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fcd8c001670 con 0x7fcd9c15e090 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 >> v1:172.21.15.59:6789/0 conn(0x7fcd9c15e090 legacy=0x7fcd9c0b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd840039b0 0x7fcd84003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd84004800 0x7fcd84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c0d0650 0x7fcd840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0d18f0 con 0x7fcd84004800 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0b3cf0 con 0x7fcd840039b0 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8400b9c0 con 0x7fcd9c0d0650 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9bfff640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd84004800 0x7fcd84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcda0eda640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd840039b0 0x7fcd84003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcda16db640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c0d0650 0x7fcd840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9bfff640 1 -- 172.21.15.59:0/3156877467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c0d0650 msgr2=0x7fcd840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9bfff640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c0d0650 0x7fcd840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9bfff640 1 -- 172.21.15.59:0/3156877467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd840039b0 msgr2=0x7fcd84003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9bfff640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd840039b0 0x7fcd84003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.175+0000 7fcd9bfff640 1 -- 172.21.15.59:0/3156877467 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd84003520 con 0x7fcd84004800 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.176+0000 7fcd9bfff640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd84004800 0x7fcd84004c80 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009ac0 tx=0x7fcd84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee55eb85dc5f6d58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.176+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90039070 con 0x7fcd84004800 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.176+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9002c710 con 0x7fcd84004800 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.176+0000 7fcd9b7fe640 1 -- 172.21.15.59:0/3156877467 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90009690 con 0x7fcd84004800 2024-03-06T07:30:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.177+0000 7fcda313e640 1 -- 172.21.15.59:0/3156877467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd84004800 msgr2=0x7fcd84004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.177+0000 7fcda313e640 1 --2- 172.21.15.59:0/3156877467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd84004800 0x7fcd84004c80 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009ac0 tx=0x7fcd84005830 comp rx=0 tx=0).stop 2024-03-06T07:30:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.177+0000 7fcda16db640 1 -- 172.21.15.59:0/3156877467 reap_dead start 2024-03-06T07:30:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.177+0000 7fcda313e640 1 -- 172.21.15.59:0/3156877467 shutdown_connections 2024-03-06T07:30:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.177+0000 7fcda313e640 1 -- 172.21.15.59:0/3156877467 >> 172.21.15.59:0/3156877467 conn(0x7fcd9c0d3c80 msgr2=0x7fcd9c0d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.177+0000 7fcda313e640 1 -- 172.21.15.59:0/3156877467 shutdown_connections 2024-03-06T07:30:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.177+0000 7fcda313e640 1 -- 172.21.15.59:0/3156877467 wait complete. 2024-03-06T07:30:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.178+0000 7fcda313e640 1 Processor -- start 2024-03-06T07:30:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.178+0000 7fcda313e640 1 -- start start 2024-03-06T07:30:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda313e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd84004800 0x7fcd9c1500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda313e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd9c0b85b0 0x7fcd9c15f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda313e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd9c15f5b0 0x7fcd9c0cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda313e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0b2df0 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda313e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cee30 con 0x7fcd9c0b85b0 2024-03-06T07:30:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda313e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0ce800 con 0x7fcd84004800 2024-03-06T07:30:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda16db640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd9c15f5b0 0x7fcd9c0cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.185 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda16db640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd9c15f5b0 0x7fcd9c0cb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36438/0 (socket says 172.21.15.59:36438) 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda16db640 1 -- 172.21.15.59:0/1319938005 learned_addr learned my addr 172.21.15.59:0/1319938005 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcd9bfff640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd9c0b85b0 0x7fcd9c15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda0eda640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd84004800 0x7fcd9c1500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda16db640 1 -- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd84004800 msgr2=0x7fcd9c1500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda16db640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd84004800 0x7fcd9c1500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda16db640 1 -- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd9c0b85b0 msgr2=0x7fcd9c15f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda16db640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd9c0b85b0 0x7fcd9c15f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda16db640 1 -- 172.21.15.59:0/1319938005 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c15fb30 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcd9bfff640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd9c0b85b0 0x7fcd9c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.179+0000 7fcda0eda640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd84004800 0x7fcd9c1500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.180+0000 7fcda16db640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd9c15f5b0 0x7fcd9c0cb0a0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0b3530 tx=0x7fcd9400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.180+0000 7fcd997fa640 1 -- 172.21.15.59:0/1319938005 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9400fe10 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.180+0000 7fcd997fa640 1 -- 172.21.15.59:0/1319938005 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9401d070 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.180+0000 7fcda313e640 1 -- 172.21.15.59:0/1319938005 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c15fe10 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.180+0000 7fcd997fa640 1 -- 172.21.15.59:0/1319938005 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94015670 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.180+0000 7fcda313e640 1 -- 172.21.15.59:0/1319938005 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c1603d0 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.181+0000 7fcda313e640 1 -- 172.21.15.59:0/1319938005 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c04f330 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.181+0000 7fcd997fa640 1 -- 172.21.15.59:0/1319938005 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd94015810 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.182+0000 7fcd997fa640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8806b4d0 0x7fcd8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.182+0000 7fcd997fa640 1 -- 172.21.15.59:0/1319938005 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd9408aa30 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.182+0000 7fcda0eda640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8806b4d0 0x7fcd8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.183+0000 7fcda0eda640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8806b4d0 0x7fcd8806d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c001eb0 tx=0x7fcd8c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.190+0000 7fcd997fa640 1 -- 172.21.15.59:0/1319938005 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd94010040 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.410+0000 7fcda313e640 1 -- 172.21.15.59:0/1319938005 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd9c0d1230 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.411+0000 7fcd997fa640 1 -- 172.21.15.59:0/1319938005 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fcd940598b0 con 0x7fcd9c15f5b0 2024-03-06T07:30:18.413 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:18.413 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":97,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4015228738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4015228738},{"type":"v1","addr":"172.21.15.164:6836","nonce":4015228738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93},{"gid":7247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2855659185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2855659185},{"type":"v1","addr":"172.21.15.59:6835","nonce":2855659185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93},{"gid":8236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:15.475267+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:30:18.413 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 97 2024-03-06T07:30:18.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 -- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8806b4d0 msgr2=0x7fcd8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8806b4d0 0x7fcd8806d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c001eb0 tx=0x7fcd8c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:18.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 -- 172.21.15.59:0/1319938005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd9c15f5b0 msgr2=0x7fcd9c0cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd9c15f5b0 0x7fcd9c0cb0a0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0b3530 tx=0x7fcd9400f580 comp rx=0 tx=0).stop 2024-03-06T07:30:18.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 -- 172.21.15.59:0/1319938005 shutdown_connections 2024-03-06T07:30:18.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcd8806b4d0 0x7fcd8806d990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd9c15f5b0 0x7fcd9c0cb0a0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd9c0b85b0 0x7fcd9c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 --2- 172.21.15.59:0/1319938005 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd84004800 0x7fcd9c1500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 -- 172.21.15.59:0/1319938005 >> 172.21.15.59:0/1319938005 conn(0x7fcd9c0d3c80 msgr2=0x7fcd9c0b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 -- 172.21.15.59:0/1319938005 shutdown_connections 2024-03-06T07:30:18.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.413+0000 7fcda313e640 1 -- 172.21.15.59:0/1319938005 wait complete. 2024-03-06T07:30:18.828 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:30:15.475267+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9336}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:30:18.828 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:30:18.829 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:30:18.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.930+0000 7f8e77dc9640 1 Processor -- start 2024-03-06T07:30:18.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.930+0000 7f8e77dc9640 1 -- start start 2024-03-06T07:30:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.931+0000 7f8e77dc9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e70162970 0x7f8e700d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.931+0000 7f8e77dc9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8e70161cf0 0x7f8e70162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.931+0000 7f8e77dc9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e700d22c0 con 0x7f8e700b4460 2024-03-06T07:30:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.931+0000 7f8e77dc9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e70161b00 con 0x7f8e70161cf0 2024-03-06T07:30:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.931+0000 7f8e77dc9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e700cecc0 con 0x7f8e70162970 2024-03-06T07:30:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.931+0000 7f8e75b65640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8e700b4460 0x7f8e700b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41580/0 (socket says 172.21.15.59:41580) 2024-03-06T07:30:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.931+0000 7f8e75b65640 1 -- 172.21.15.59:0/1491057922 learned_addr learned my addr 172.21.15.59:0/1491057922 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:18.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.931+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912171721 0 0) 0x7f8e700d22c0 con 0x7f8e700b4460 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e76366640 1 --2- 172.21.15.59:0/1491057922 >> v2:172.21.15.164:3300/0 conn(0x7f8e70161cf0 0x7f8e70162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5c003520 con 0x7f8e700b4460 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e75364640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e70162970 0x7f8e700d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1991897357 0 0) 0x7f8e5c003520 con 0x7f8e700b4460 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e70162970 msgr2=0x7f8e700d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e70162970 0x7f8e700d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 >> v2:172.21.15.164:3300/0 conn(0x7f8e70161cf0 msgr2=0x7f8e70162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 --2- 172.21.15.59:0/1491057922 >> v2:172.21.15.164:3300/0 conn(0x7f8e70161cf0 0x7f8e70162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e700d25a0 con 0x7f8e700b4460 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f8e6c001670 con 0x7f8e700b4460 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 >> v1:172.21.15.59:6789/0 conn(0x7f8e700b4460 legacy=0x7f8e700b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e70162970 0x7f8e5c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e5c003f30 0x7f8e5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 0x7f8e5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e70161b00 con 0x7f8e5c004d40 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e700cecc0 con 0x7f8e5c003f30 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c00b9c0 con 0x7f8e70162970 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e76366640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 0x7f8e5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e75b65640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e70162970 0x7f8e5c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e75364640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e5c003f30 0x7f8e5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e76366640 1 -- 172.21.15.59:0/1491057922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e70162970 msgr2=0x7f8e5c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e76366640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e70162970 0x7f8e5c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e76366640 1 -- 172.21.15.59:0/1491057922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e5c003f30 msgr2=0x7f8e5c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e76366640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e5c003f30 0x7f8e5c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.932+0000 7f8e76366640 1 -- 172.21.15.59:0/1491057922 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c003520 con 0x7f8e5c004d40 2024-03-06T07:30:18.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.933+0000 7f8e76366640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 0x7f8e5c0051c0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009fc0 tx=0x7f8e5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35e655ba729e420 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.933+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e68039070 con 0x7f8e5c004d40 2024-03-06T07:30:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.933+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6802cc40 con 0x7f8e5c004d40 2024-03-06T07:30:18.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.933+0000 7f8e74b63640 1 -- 172.21.15.59:0/1491057922 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e68009c10 con 0x7f8e5c004d40 2024-03-06T07:30:18.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.934+0000 7f8e77dc9640 1 -- 172.21.15.59:0/1491057922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 msgr2=0x7f8e5c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.934+0000 7f8e77dc9640 1 --2- 172.21.15.59:0/1491057922 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 0x7f8e5c0051c0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009fc0 tx=0x7f8e5c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:18.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.934+0000 7f8e76366640 1 -- 172.21.15.59:0/1491057922 reap_dead start 2024-03-06T07:30:18.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.934+0000 7f8e77dc9640 1 -- 172.21.15.59:0/1491057922 shutdown_connections 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.934+0000 7f8e77dc9640 1 -- 172.21.15.59:0/1491057922 >> 172.21.15.59:0/1491057922 conn(0x7f8e700d3ce0 msgr2=0x7f8e700d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.934+0000 7f8e77dc9640 1 -- 172.21.15.59:0/1491057922 shutdown_connections 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.934+0000 7f8e77dc9640 1 -- 172.21.15.59:0/1491057922 wait complete. 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.935+0000 7f8e77dc9640 1 Processor -- start 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.935+0000 7f8e77dc9640 1 -- start start 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e77dc9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 0x7f8e700c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e77dc9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e700b8bc0 0x7f8e70155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e77dc9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e700cadb0 0x7f8e700cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e77dc9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e700ce950 con 0x7f8e5c004d40 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e77dc9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e700d2b60 con 0x7f8e700cadb0 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e77dc9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e700b4840 con 0x7f8e700b8bc0 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e75b65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 0x7f8e700c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e75b65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 0x7f8e700c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36466/0 (socket says 172.21.15.59:36466) 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e75b65640 1 -- 172.21.15.59:0/3739622735 learned_addr learned my addr 172.21.15.59:0/3739622735 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:18.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e75364640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e700b8bc0 0x7f8e70155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e76366640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e700cadb0 0x7f8e700cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e75b65640 1 -- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e700b8bc0 msgr2=0x7f8e70155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e75b65640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e700b8bc0 0x7f8e70155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e75b65640 1 -- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e700cadb0 msgr2=0x7f8e700cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e75b65640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e700cadb0 0x7f8e700cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e75b65640 1 -- 172.21.15.59:0/3739622735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e70156650 con 0x7f8e5c004d40 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e76366640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e700cadb0 0x7f8e700cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e75b65640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 0x7f8e700c88e0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c007310 tx=0x7f8e6c0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e667fc640 1 -- 172.21.15.59:0/3739622735 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c00fa00 con 0x7f8e5c004d40 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e667fc640 1 -- 172.21.15.59:0/3739622735 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c018070 con 0x7f8e5c004d40 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e77dc9640 1 -- 172.21.15.59:0/3739622735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e700c8fe0 con 0x7f8e5c004d40 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.936+0000 7f8e667fc640 1 -- 172.21.15.59:0/3739622735 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c00fde0 con 0x7f8e5c004d40 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.937+0000 7f8e77dc9640 1 -- 172.21.15.59:0/3739622735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e700c95a0 con 0x7f8e5c004d40 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.938+0000 7f8e667fc640 1 -- 172.21.15.59:0/3739622735 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e6c01d460 con 0x7f8e5c004d40 2024-03-06T07:30:18.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.938+0000 7f8e667fc640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e5406b4d0 0x7f8e5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.938+0000 7f8e667fc640 1 -- 172.21.15.59:0/3739622735 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e6c08a440 con 0x7f8e5c004d40 2024-03-06T07:30:18.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.939+0000 7f8e75364640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e5406b4d0 0x7f8e5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.939+0000 7f8e75364640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e5406b4d0 0x7f8e5406d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8e700c7530 tx=0x7f8e60008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.939+0000 7f8e7a9b5740 1 Processor -- start 2024-03-06T07:30:18.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.939+0000 7f8e7a9b5740 1 -- start start 2024-03-06T07:30:18.955 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.940+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc0d870 0x55940bc0dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.940+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc118c0 0x55940bc13360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.940+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 0x55940bc3da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.940+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55940bb776b0 con 0x55940bc0d870 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.940+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55940bb20a60 con 0x55940bc3d600 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.940+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55940bb73750 con 0x55940bc118c0 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.940+0000 7f8e75b65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc0d870 0x55940bc0dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.940+0000 7f8e75b65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc0d870 0x55940bc0dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36468/0 (socket says 172.21.15.59:36468) 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.940+0000 7f8e75b65640 1 -- 172.21.15.59:0/2522531871 learned_addr learned my addr 172.21.15.59:0/2522531871 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.940+0000 7f8e75364640 1 --2- 172.21.15.59:0/2522531871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc118c0 0x55940bc13360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.940+0000 7f8e76366640 1 --2- 172.21.15.59:0/2522531871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 0x55940bc3da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.941+0000 7f8e75b65640 1 -- 172.21.15.59:0/2522531871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc118c0 msgr2=0x55940bc13360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.941+0000 7f8e75b65640 1 --2- 172.21.15.59:0/2522531871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc118c0 0x55940bc13360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.941+0000 7f8e75b65640 1 -- 172.21.15.59:0/2522531871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 msgr2=0x55940bc3da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.941+0000 7f8e75b65640 1 --2- 172.21.15.59:0/2522531871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 0x55940bc3da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.941+0000 7f8e75b65640 1 -- 172.21.15.59:0/2522531871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55940bb3ef30 con 0x55940bc0d870 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.941+0000 7f8e75b65640 1 --2- 172.21.15.59:0/2522531871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc0d870 0x55940bc0dc50 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c088990 tx=0x7f8e6c0592c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719368d64343ce40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.941+0000 7f8e537fe640 1 -- 172.21.15.59:0/2522531871 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0612d0 con 0x55940bc0d870 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.941+0000 7f8e537fe640 1 -- 172.21.15.59:0/2522531871 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c056ec0 con 0x55940bc0d870 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.941+0000 7f8e537fe640 1 -- 172.21.15.59:0/2522531871 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c060e40 con 0x55940bc0d870 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.942+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2522531871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc0d870 msgr2=0x55940bc0dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.942+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2522531871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc0d870 0x55940bc0dc50 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c088990 tx=0x7f8e6c0592c0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.942+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2522531871 shutdown_connections 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.942+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2522531871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 0x55940bc3da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.942+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2522531871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc118c0 0x55940bc13360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.942+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2522531871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc0d870 0x55940bc0dc50 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.942+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2522531871 >> 172.21.15.59:0/2522531871 conn(0x55940bc0f8f0 msgr2=0x55940bc111b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.942+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2522531871 shutdown_connections 2024-03-06T07:30:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.942+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2522531871 wait complete. 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.943+0000 7f8e7a9b5740 1 -- start start 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.943+0000 7f8e7a9b5740 1 Processor -- start 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.944+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc3d600 0x55940bc32410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.944+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc308c0 0x55940bc3efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.944+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc33af0 0x55940bc35ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.944+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55940bc11d50 con 0x55940bc3d600 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.944+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55940bb33ea0 con 0x55940bc33af0 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.944+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55940bc12030 con 0x55940bc308c0 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.944+0000 7f8e7a9b5740 10 client.0 _ll_register_callbacks cb 0x55940bb33870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.944+0000 7f8e75b65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc3d600 0x55940bc32410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.944+0000 7f8e75b65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc3d600 0x55940bc32410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36470/0 (socket says 172.21.15.59:36470) 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.944+0000 7f8e75b65640 1 -- 172.21.15.59:0/191882863 learned_addr learned my addr 172.21.15.59:0/191882863 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e75b65640 1 -- 172.21.15.59:0/191882863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc308c0 msgr2=0x55940bc3efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.958 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e75b65640 1 --2- 172.21.15.59:0/191882863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc308c0 0x55940bc3efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e75b65640 1 -- 172.21.15.59:0/191882863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc33af0 msgr2=0x55940bc35ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e75b65640 1 --2- 172.21.15.59:0/191882863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc33af0 0x55940bc35ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e75b65640 1 -- 172.21.15.59:0/191882863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55940bb34270 con 0x55940bc3d600 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e75b65640 1 --2- 172.21.15.59:0/191882863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc3d600 0x55940bc32410 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c007fd0 tx=0x7f8e6c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c28ed5695487eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e50ff9640 1 -- 172.21.15.59:0/191882863 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0612d0 con 0x55940bc3d600 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e50ff9640 1 -- 172.21.15.59:0/191882863 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e6c06cb80 con 0x55940bc3d600 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/191882863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e6c056ec0 con 0x55940bc3d600 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e50ff9640 1 -- 172.21.15.59:0/191882863 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e6c075050 con 0x55940bc3d600 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e7a9b5740 10 client.9474 fetch_fsmap learned FSMap version 97 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/191882863 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x55940b88c2d0 con 0x55940bc3d600 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e50ff9640 1 -- 172.21.15.59:0/191882863 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c056ec0 con 0x55940bc3d600 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e50ff9640 1 -- 172.21.15.59:0/191882863 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 97) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8e6c058010 con 0x55940bc3d600 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e7a9b5740 10 client.9474 fetch_fsmap finished waiting for FSMap version 97 2024-03-06T07:30:18.959 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e7a9b5740 10 client.9474 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e7a9b5740 20 client.9474 populate_metadata read hostname 'smithi059' 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/191882863 >> [v2:172.21.15.59:6832/720863754,v1:172.21.15.59:6833/720863754] conn(0x55940bc37fc0 0x55940bc383c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e7a9b5740 4 client.9474 mds_command: new command op to 9336 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.945+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/191882863 --> [v2:172.21.15.59:6832/720863754,v1:172.21.15.59:6833/720863754] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55940bb84bf0 con 0x55940bc37fc0 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.946+0000 7f8e75364640 1 --2- 172.21.15.59:0/191882863 >> [v2:172.21.15.59:6832/720863754,v1:172.21.15.59:6833/720863754] conn(0x55940bc37fc0 0x55940bc383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.946+0000 7f8e50ff9640 10 client.9474 ms_handle_connect on v2:172.21.15.59:6832/720863754 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.946+0000 7f8e75364640 1 --2- 172.21.15.59:0/191882863 >> [v2:172.21.15.59:6832/720863754,v1:172.21.15.59:6833/720863754] conn(0x55940bc37fc0 0x55940bc383c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c13016f573ad72e server_cookie=61b62aa2f67b26f1 in_seq=0 out_seq=0 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.946+0000 7f8e50ff9640 1 -- 172.21.15.59:0/191882863 <== mds.0 v2:172.21.15.59:6832/720863754 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55940bb84bf0 con 0x55940bc37fc0 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.946+0000 7f8e50ff9640 10 client.9474 handle_command_reply: tid=0 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 1 client.9474 shutdown 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/191882863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc3d600 msgr2=0x55940bc32410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/191882863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc3d600 0x55940bc32410 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c007fd0 tx=0x7f8e6c001a40 comp rx=0 tx=0).stop 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/191882863 shutdown_connections 2024-03-06T07:30:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/191882863 >> [v2:172.21.15.59:6832/720863754,v1:172.21.15.59:6833/720863754] conn(0x55940bc37fc0 0x55940bc383c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/191882863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc33af0 0x55940bc35ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/191882863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc308c0 0x55940bc3efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/191882863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc3d600 0x55940bc32410 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/191882863 >> 172.21.15.59:0/191882863 conn(0x55940bc0d870 msgr2=0x55940bc3b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e50ff9640 0 client.9474 ms_handle_reset on v2:172.21.15.59:6832/720863754 2024-03-06T07:30:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/191882863 shutdown_connections 2024-03-06T07:30:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/191882863 wait complete. 2024-03-06T07:30:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.947+0000 7f8e7a9b5740 20 client.9474 trim_cache size 0 max 16384 2024-03-06T07:30:18.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.975+0000 7f8e7a9b5740 1 Processor -- start 2024-03-06T07:30:18.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.975+0000 7f8e7a9b5740 1 -- start start 2024-03-06T07:30:18.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc72ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc33af0 0x55940bc13f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 0x55940bcc1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e701ff950 con 0x55940bc308c0 2024-03-06T07:30:18.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e701fed50 con 0x55940bc3d600 2024-03-06T07:30:18.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e701ff350 con 0x55940bc33af0 2024-03-06T07:30:18.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e75b65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc72ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e75b65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc72ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36486/0 (socket says 172.21.15.59:36486) 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e75b65640 1 -- 172.21.15.59:0/2596655370 learned_addr learned my addr 172.21.15.59:0/2596655370 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e75b65640 1 -- 172.21.15.59:0/2596655370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc33af0 msgr2=0x55940bc13f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e75b65640 1 --2- 172.21.15.59:0/2596655370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc33af0 0x55940bc13f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e75b65640 1 -- 172.21.15.59:0/2596655370 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 msgr2=0x55940bcc1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e75b65640 1 --2- 172.21.15.59:0/2596655370 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 0x55940bcc1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e75b65640 1 -- 172.21.15.59:0/2596655370 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55940bc12520 con 0x55940bc308c0 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e75b65640 1 --2- 172.21.15.59:0/2596655370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc72ea0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c058540 tx=0x7f8e6c01f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d6553c53948a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e52ffd640 1 -- 172.21.15.59:0/2596655370 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c054e60 con 0x55940bc308c0 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e52ffd640 1 -- 172.21.15.59:0/2596655370 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c056ec0 con 0x55940bc308c0 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.976+0000 7f8e52ffd640 1 -- 172.21.15.59:0/2596655370 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c058990 con 0x55940bc308c0 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.977+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2596655370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 msgr2=0x55940bc72ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.977+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2596655370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc72ea0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c058540 tx=0x7f8e6c01f040 comp rx=0 tx=0).stop 2024-03-06T07:30:18.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.977+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2596655370 shutdown_connections 2024-03-06T07:30:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.977+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2596655370 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 0x55940bcc1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.977+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2596655370 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc33af0 0x55940bc13f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.977+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2596655370 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc72ea0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.977+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2596655370 >> 172.21.15.59:0/2596655370 conn(0x55940bc0d870 msgr2=0x55940bc37f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.977+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2596655370 shutdown_connections 2024-03-06T07:30:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.977+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2596655370 wait complete. 2024-03-06T07:30:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.977+0000 7f8e7a9b5740 1 -- start start 2024-03-06T07:30:18.981 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.977+0000 7f8e7a9b5740 1 Processor -- start 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.978+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc60960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.978+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc33af0 0x55940bc7b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.978+0000 7f8e7a9b5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 0x55940bc3db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.978+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x55940bc11d50 con 0x55940bc308c0 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.978+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55940bb33ea0 con 0x55940bc3d600 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.978+0000 7f8e7a9b5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55940bc33f70 con 0x55940bc33af0 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.978+0000 7f8e7a9b5740 10 client.0 _ll_register_callbacks cb 0x7f8e6001a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.978+0000 7f8e75b65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc60960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.978+0000 7f8e75b65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc60960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36498/0 (socket says 172.21.15.59:36498) 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.978+0000 7f8e75b65640 1 -- 172.21.15.59:0/2457222494 learned_addr learned my addr 172.21.15.59:0/2457222494 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e75364640 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc33af0 0x55940bc7b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e76366640 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 0x55940bc3db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e75b65640 1 -- 172.21.15.59:0/2457222494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc33af0 msgr2=0x55940bc7b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e75b65640 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc33af0 0x55940bc7b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e75b65640 1 -- 172.21.15.59:0/2457222494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 msgr2=0x55940bc3db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e75b65640 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 0x55940bc3db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e75b65640 1 -- 172.21.15.59:0/2457222494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55940b9566b0 con 0x55940bc308c0 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e75b65640 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc60960 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c056e70 tx=0x7f8e6c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0e9c0ef4163170 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.59:3300/0 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e50ff9640 1 -- 172.21.15.59:0/2457222494 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c058990 con 0x55940bc308c0 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e50ff9640 1 -- 172.21.15.59:0/2457222494 <== mon.0 v2:172.21.15.59:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e6c054b10 con 0x55940bc308c0 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2457222494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e6c056ec0 con 0x55940bc308c0 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e50ff9640 1 -- 172.21.15.59:0/2457222494 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e6c054e60 con 0x55940bc308c0 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e7a9b5740 10 client.9480 fetch_fsmap learned FSMap version 97 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2457222494 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x55940b9566b0 con 0x55940bc308c0 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.979+0000 7f8e50ff9640 1 -- 172.21.15.59:0/2457222494 <== mon.0 v2:172.21.15.59:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c056ec0 con 0x55940bc308c0 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e50ff9640 1 -- 172.21.15.59:0/2457222494 <== mon.0 v2:172.21.15.59:3300/0 5 ==== fsmap(e 97) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8e6c057a80 con 0x55940bc308c0 2024-03-06T07:30:18.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 10 client.9480 fetch_fsmap finished waiting for FSMap version 97 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 10 client.9480 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 20 client.9480 populate_metadata read hostname 'smithi059' 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.59:6832/720863754,v1:172.21.15.59:6833/720863754] conn(0x55940bc3c530 0x55940bc3c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 4 client.9480 mds_command: new command op to 9336 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2457222494 --> [v2:172.21.15.59:6832/720863754,v1:172.21.15.59:6833/720863754] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55940bb7ef40 con 0x55940bc3c530 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e75364640 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.59:6832/720863754,v1:172.21.15.59:6833/720863754] conn(0x55940bc3c530 0x55940bc3c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e50ff9640 10 client.9480 ms_handle_connect on v2:172.21.15.59:6832/720863754 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e75364640 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.59:6832/720863754,v1:172.21.15.59:6833/720863754] conn(0x55940bc3c530 0x55940bc3c910 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed12cb22a56524c9 server_cookie=1dd8d83b9755959d in_seq=0 out_seq=0 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e50ff9640 1 -- 172.21.15.59:0/2457222494 <== mds.0 v2:172.21.15.59:6832/720863754 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55940bb7ef40 con 0x55940bc3c530 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e50ff9640 10 client.9480 handle_command_reply: tid=0 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 1 client.9480 shutdown 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2457222494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 msgr2=0x55940bc60960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc60960 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c056e70 tx=0x7f8e6c000fe0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2457222494 shutdown_connections 2024-03-06T07:30:18.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.59:6832/720863754,v1:172.21.15.59:6833/720863754] conn(0x55940bc3c530 0x55940bc3c910 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55940bc3d600 0x55940bc3db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55940bc33af0 0x55940bc7b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 1 --2- 172.21.15.59:0/2457222494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x55940bc308c0 0x55940bc60960 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2457222494 >> 172.21.15.59:0/2457222494 conn(0x55940bc0d870 msgr2=0x55940bc13f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.980+0000 7f8e50ff9640 0 client.9480 ms_handle_reset on v2:172.21.15.59:6832/720863754 2024-03-06T07:30:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2457222494 shutdown_connections 2024-03-06T07:30:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e7a9b5740 1 -- 172.21.15.59:0/2457222494 wait complete. 2024-03-06T07:30:18.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e7a9b5740 20 client.9480 trim_cache size 0 max 16384 2024-03-06T07:30:18.988 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:30:18.989 INFO:teuthology.orchestra.run.smithi059.stdout: "cluster_fsid": "43c1ee46-a208-44bd-a1f3-57a84ef7ed13", 2024-03-06T07:30:18.989 INFO:teuthology.orchestra.run.smithi059.stdout: "whoami": 0, 2024-03-06T07:30:18.989 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 9336, 2024-03-06T07:30:18.989 INFO:teuthology.orchestra.run.smithi059.stdout: "want_state": "up:active", 2024-03-06T07:30:18.989 INFO:teuthology.orchestra.run.smithi059.stdout: "state": "up:active", 2024-03-06T07:30:18.989 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs", 2024-03-06T07:30:18.990 INFO:teuthology.orchestra.run.smithi059.stdout: "rank_uptime": 4.9051802059999998, 2024-03-06T07:30:18.990 INFO:teuthology.orchestra.run.smithi059.stdout: "mdsmap_epoch": 97, 2024-03-06T07:30:18.990 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch": 139, 2024-03-06T07:30:18.990 INFO:teuthology.orchestra.run.smithi059.stdout: "osdmap_epoch_barrier": 139, 2024-03-06T07:30:18.990 INFO:teuthology.orchestra.run.smithi059.stdout: "uptime": 9.9088772459999994 2024-03-06T07:30:18.990 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:30:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 -- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e5406b4d0 msgr2=0x7f8e5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e5406b4d0 0x7f8e5406d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8e700c7530 tx=0x7f8e60008040 comp rx=0 tx=0).stop 2024-03-06T07:30:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 -- 172.21.15.59:0/3739622735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 msgr2=0x7f8e700c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 0x7f8e700c88e0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c007310 tx=0x7f8e6c0077d0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 -- 172.21.15.59:0/3739622735 shutdown_connections 2024-03-06T07:30:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e5406b4d0 0x7f8e5406d990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e700cadb0 0x7f8e700cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e700b8bc0 0x7f8e70155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 --2- 172.21.15.59:0/3739622735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e5c004d40 0x7f8e700c88e0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 -- 172.21.15.59:0/3739622735 >> 172.21.15.59:0/3739622735 conn(0x7f8e700d3ce0 msgr2=0x7f8e700d05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 -- 172.21.15.59:0/3739622735 shutdown_connections 2024-03-06T07:30:18.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:18.981+0000 7f8e77dc9640 1 -- 172.21.15.59:0/3739622735 wait complete. 2024-03-06T07:30:18.994 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:19.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.093+0000 7fe5ba3b6640 1 Processor -- start 2024-03-06T07:30:19.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.093+0000 7fe5ba3b6640 1 -- start start 2024-03-06T07:30:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.094+0000 7fe5ba3b6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe5b415bb20 0x7fe5b40d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.094+0000 7fe5ba3b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cd8c0 0x7fe5b40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.094+0000 7fe5ba3b6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b4161980 con 0x7fe5b40cfff0 2024-03-06T07:30:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.094+0000 7fe5ba3b6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b4161c60 con 0x7fe5b415bb20 2024-03-06T07:30:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.094+0000 7fe5ba3b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b4161f40 con 0x7fe5b40cd8c0 2024-03-06T07:30:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.094+0000 7fe5b37fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe5b40cfff0 0x7fe5b40ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41584/0 (socket says 172.21.15.59:41584) 2024-03-06T07:30:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.094+0000 7fe5b37fe640 1 -- 172.21.15.59:0/1026021162 learned_addr learned my addr 172.21.15.59:0/1026021162 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:19.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b8953640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cd8c0 0x7fe5b40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/1026021162 >> v2:172.21.15.164:3300/0 conn(0x7fe5b415bb20 0x7fe5b40d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305495182 0 0) 0x7fe5b4161980 con 0x7fe5b40cfff0 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe59c003520 con 0x7fe5b40cfff0 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 979207059 0 0) 0x7fe59c003520 con 0x7fe5b40cfff0 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cd8c0 msgr2=0x7fe5b40d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cd8c0 0x7fe5b40d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 >> v2:172.21.15.164:3300/0 conn(0x7fe5b415bb20 msgr2=0x7fe5b40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 --2- 172.21.15.59:0/1026021162 >> v2:172.21.15.164:3300/0 conn(0x7fe5b415bb20 0x7fe5b40d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b4162220 con 0x7fe5b40cfff0 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fe5a4001670 con 0x7fe5b40cfff0 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/1026021162 >> v2:172.21.15.164:3300/0 conn(0x7fe5b415bb20 0x7fe5b40d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 >> v1:172.21.15.59:6789/0 conn(0x7fe5b40cfff0 legacy=0x7fe5b40ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0039b0 0x7fe59c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cd8c0 0x7fe59c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c004d40 0x7fe59c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b4161c60 con 0x7fe59c0039b0 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b4161f40 con 0x7fe59c004d40 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c00b9c0 con 0x7fe5b40cd8c0 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0039b0 0x7fe59c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b37fe640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cd8c0 0x7fe59c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b8953640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c004d40 0x7fe59c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b3fff640 1 -- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cd8c0 msgr2=0x7fe59c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cd8c0 0x7fe59c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b3fff640 1 -- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c004d40 msgr2=0x7fe59c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c004d40 0x7fe59c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b3fff640 1 -- 172.21.15.59:0/1026021162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c003520 con 0x7fe59c0039b0 2024-03-06T07:30:19.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.095+0000 7fe5b8953640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c004d40 0x7fe59c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:19.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.096+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0039b0 0x7fe59c003dd0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8009c90 tx=0x7fe59c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648b5e4e618cdbec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.096+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8039070 con 0x7fe59c0039b0 2024-03-06T07:30:19.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.096+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a802c980 con 0x7fe59c0039b0 2024-03-06T07:30:19.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.096+0000 7fe5b2ffd640 1 -- 172.21.15.59:0/1026021162 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8009860 con 0x7fe59c0039b0 2024-03-06T07:30:19.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.097+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/1026021162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0039b0 msgr2=0x7fe59c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.097+0000 7fe5ba3b6640 1 --2- 172.21.15.59:0/1026021162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0039b0 0x7fe59c003dd0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8009c90 tx=0x7fe59c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:19.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.097+0000 7fe5b8953640 1 -- 172.21.15.59:0/1026021162 reap_dead start 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.097+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/1026021162 shutdown_connections 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.097+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/1026021162 >> 172.21.15.59:0/1026021162 conn(0x7fe5b409a9f0 msgr2=0x7fe5b409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.097+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/1026021162 shutdown_connections 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.097+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/1026021162 wait complete. 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.099+0000 7fe5ba3b6640 1 Processor -- start 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.099+0000 7fe5ba3b6640 1 -- start start 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.100+0000 7fe5ba3b6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5b414df70 0x7fe5b415fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.100+0000 7fe5ba3b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b41f3ca0 0x7fe5b4160270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.100+0000 7fe5ba3b6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b415d6e0 0x7fe5b40caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.100+0000 7fe5ba3b6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b40d24b0 con 0x7fe5b415d6e0 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.100+0000 7fe5ba3b6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b40d2330 con 0x7fe5b414df70 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.100+0000 7fe5ba3b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b40d1980 con 0x7fe5b41f3ca0 2024-03-06T07:30:19.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.100+0000 7fe5b8953640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b415d6e0 0x7fe5b40caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.100+0000 7fe5b8953640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b415d6e0 0x7fe5b40caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36514/0 (socket says 172.21.15.59:36514) 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.100+0000 7fe5b8953640 1 -- 172.21.15.59:0/291798276 learned_addr learned my addr 172.21.15.59:0/291798276 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5b414df70 0x7fe5b415fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5b37fe640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b41f3ca0 0x7fe5b4160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5b8953640 1 -- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b41f3ca0 msgr2=0x7fe5b4160270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5b8953640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b41f3ca0 0x7fe5b4160270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5b8953640 1 -- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5b414df70 msgr2=0x7fe5b415fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5b8953640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5b414df70 0x7fe5b415fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5b8953640 1 -- 172.21.15.59:0/291798276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b415dca0 con 0x7fe5b415d6e0 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5b8953640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b415d6e0 0x7fe5b40caaf0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40cf5e0 tx=0x7fe5ac010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5b0ff9640 1 -- 172.21.15.59:0/291798276 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac010e00 con 0x7fe5b415d6e0 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5b0ff9640 1 -- 172.21.15.59:0/291798276 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ac003990 con 0x7fe5b415d6e0 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/291798276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b40ca010 con 0x7fe5b415d6e0 2024-03-06T07:30:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5b0ff9640 1 -- 172.21.15.59:0/291798276 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac003d80 con 0x7fe5b415d6e0 2024-03-06T07:30:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.101+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/291798276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b414e350 con 0x7fe5b415d6e0 2024-03-06T07:30:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.102+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/291798276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b404f330 con 0x7fe5b415d6e0 2024-03-06T07:30:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.103+0000 7fe5b0ff9640 1 -- 172.21.15.59:0/291798276 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5ac01e020 con 0x7fe5b415d6e0 2024-03-06T07:30:19.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.103+0000 7fe5b0ff9640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 0x7fe5a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.103+0000 7fe5b0ff9640 1 -- 172.21.15.59:0/291798276 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5ac08a710 con 0x7fe5b415d6e0 2024-03-06T07:30:19.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.103+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 0x7fe5a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.104+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 0x7fe5a006d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8036000 tx=0x7fe5a8030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.112+0000 7fe5b0ff9640 1 -- 172.21.15.59:0/291798276 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5ac058e10 con 0x7fe5b415d6e0 2024-03-06T07:30:19.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.332+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/291798276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe5b40b2e60 con 0x7fe5b415d6e0 2024-03-06T07:30:19.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.332+0000 7fe5b0ff9640 1 -- 172.21.15.59:0/291798276 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v97) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe5ac05ceb0 con 0x7fe5b415d6e0 2024-03-06T07:30:19.334 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:19.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 msgr2=0x7fe5a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 0x7fe5a006d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8036000 tx=0x7fe5a8030000 comp rx=0 tx=0).stop 2024-03-06T07:30:19.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/291798276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b415d6e0 msgr2=0x7fe5b40caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b415d6e0 0x7fe5b40caaf0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40cf5e0 tx=0x7fe5ac010590 comp rx=0 tx=0).stop 2024-03-06T07:30:19.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/291798276 shutdown_connections 2024-03-06T07:30:19.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 0x7fe5a006d9c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b415d6e0 0x7fe5b40caaf0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b41f3ca0 0x7fe5b4160270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 --2- 172.21.15.59:0/291798276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5b414df70 0x7fe5b415fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/291798276 >> 172.21.15.59:0/291798276 conn(0x7fe5b409a9f0 msgr2=0x7fe5b415bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/291798276 shutdown_connections 2024-03-06T07:30:19.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:19.334+0000 7fe5ba3b6640 1 -- 172.21.15.59:0/291798276 wait complete. 2024-03-06T07:30:19.347 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:30:19.347 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:19.347 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:30:19.347 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:19.347 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:19.347 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:30:19.347 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:30:19.347 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:19.347 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:19.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.348 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:30:19.363 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:19.363 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:19.363 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:19.363 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.363 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:19.363 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.363 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:19.363 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:19.363 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:30:19.364 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:19.364 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:19.364 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:30:19.364 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:19.364 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:19.364 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:30:19.365 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:30:19.365 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42159sec preferred_lft 42159sec 2024-03-06T07:30:19.365 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:30:19.365 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.365 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:19.366 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:19.366 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:30:19.366 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:19.366 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:19.366 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:19.367 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.367 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:30:19.367 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.367 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:19.367 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:19.367 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:30:19.367 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.367 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:19.367 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:19.367 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:30:19.367 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:19.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.368 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:19.428 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:30:19.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.429 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:30:19.484 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:30:19.484 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:19.484 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:30:19.484 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:30:19.485 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:19.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.485 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:30:19.485 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:30:19.485 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:30:19.485 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:30:19.485 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:30:19.485 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:30:19.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.579 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:30:19.594 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:19.594 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:19.594 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:19.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.594 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:30:19.661 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:30:19.661 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:19.661 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:19.716 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:19.717 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:19.717 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:19.771 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:30:19.842 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:19.896 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:30:19.896 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:30:19.896 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:30:19.896 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:30:19.896 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:30:19.896 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:30:19.896 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:30:19.896 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:19.897 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:19.898 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:19.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:19.899 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:30:19.953 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:30:19.954 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:30:19.996 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:30:20.026 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:30:20.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:30:20.080+0000 7f817944b480 -1 init, newargv = 0x56513806a8e0 newargc=15 2024-03-06T07:30:20.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[83363]: starting ceph client 2024-03-06T07:30:20.083 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:30:20.083 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:30:20.083 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:30:20.083 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:30:20.084 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:20.085 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:20.086 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:20.086 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:20.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:20.087 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:30:20.087 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[83363]: starting fuse 2024-03-06T07:30:20.141 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:30:20.141 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:30:20.141 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:30:20.142 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:30:20.143 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:30:20.143 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:30:20.143 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:30:20.143 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:30:20.143 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:30:20.143 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:30:20.143 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:30:20.143 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:30:20.143 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:30:20.143 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:30:20.266 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.83363.asok 2024-03-06T07:30:20.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:20.276 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.83363.asok 2024-03-06T07:30:20.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:20.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.83363.asok status 2024-03-06T07:30:20.414 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:30:20.414 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:30:20.414 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:30:20.414 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:30:20.414 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:30:20.414 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:30:20.414 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:30:20.414 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "83363", 2024-03-06T07:30:20.414 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 9492, 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 9492 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:20.415 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1872329002 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1872329002 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.9492 192.168.0.1:0/1872329002", 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/1872329002", 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 97, 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 139, 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:30:20.416 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:30:20.417 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:30:20.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:20.423 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:30:20.423 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:20.437 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:30:20.437 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:20.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:20.437 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:20.505 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:30:20.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.647+0000 7fd70f039640 1 Processor -- start 2024-03-06T07:30:20.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.647+0000 7fd70f039640 1 -- start start 2024-03-06T07:30:20.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.648+0000 7fd70f039640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7080cea50 0x7fd7080b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.648+0000 7fd70f039640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd70806cab0 0x7fd7080ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.648+0000 7fd70f039640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70806d030 con 0x7fd70815def0 2024-03-06T07:30:20.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.648+0000 7fd70f039640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7080cf1c0 con 0x7fd70806cab0 2024-03-06T07:30:20.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.648+0000 7fd70f039640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7080cc760 con 0x7fd7080cea50 2024-03-06T07:30:20.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.648+0000 7fd707fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7080cea50 0x7fd7080b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.648+0000 7fd70d5d6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd70806cab0 0x7fd7080ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.648+0000 7fd70cdd5640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd70815def0 0x7fd7080ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:39350/0 (socket says 172.21.15.59:39350) 2024-03-06T07:30:20.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.648+0000 7fd70cdd5640 1 -- 172.21.15.59:0/4063462079 learned_addr learned my addr 172.21.15.59:0/4063462079 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:20.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.649+0000 7fd707fff640 1 -- 172.21.15.59:0/4063462079 >> v2:172.21.15.164:3300/0 conn(0x7fd70806cab0 msgr2=0x7fd7080ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.649+0000 7fd707fff640 1 --2- 172.21.15.59:0/4063462079 >> v2:172.21.15.164:3300/0 conn(0x7fd70806cab0 0x7fd7080ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.649+0000 7fd707fff640 1 -- 172.21.15.59:0/4063462079 >> v1:172.21.15.164:6789/0 conn(0x7fd70815def0 legacy=0x7fd7080ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.649+0000 7fd707fff640 1 -- 172.21.15.59:0/4063462079 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70806d310 con 0x7fd7080cea50 2024-03-06T07:30:20.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.649+0000 7fd70d5d6640 1 --2- 172.21.15.59:0/4063462079 >> v2:172.21.15.164:3300/0 conn(0x7fd70806cab0 0x7fd7080ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.649+0000 7fd707fff640 1 --2- 172.21.15.59:0/4063462079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7080cea50 0x7fd7080b41b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd6f40097a0 tx=0x7fd6f402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e6b8d45b462bda0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.650+0000 7fd7077fe640 1 -- 172.21.15.59:0/4063462079 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4034070 con 0x7fd7080cea50 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.650+0000 7fd7077fe640 1 -- 172.21.15.59:0/4063462079 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f4030610 con 0x7fd7080cea50 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.650+0000 7fd7077fe640 1 -- 172.21.15.59:0/4063462079 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40309e0 con 0x7fd7080cea50 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.650+0000 7fd70f039640 1 -- 172.21.15.59:0/4063462079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7080cea50 msgr2=0x7fd7080b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.650+0000 7fd70f039640 1 --2- 172.21.15.59:0/4063462079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7080cea50 0x7fd7080b41b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd6f40097a0 tx=0x7fd6f402b960 comp rx=0 tx=0).stop 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.650+0000 7fd70f039640 1 -- 172.21.15.59:0/4063462079 shutdown_connections 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.650+0000 7fd70f039640 1 --2- 172.21.15.59:0/4063462079 >> v2:172.21.15.164:3300/0 conn(0x7fd70806cab0 0x7fd7080ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.650+0000 7fd70f039640 1 --2- 172.21.15.59:0/4063462079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7080cea50 0x7fd7080b41b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.650+0000 7fd70f039640 1 -- 172.21.15.59:0/4063462079 >> 172.21.15.59:0/4063462079 conn(0x7fd7080b3690 msgr2=0x7fd7080b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.650+0000 7fd70f039640 1 -- 172.21.15.59:0/4063462079 shutdown_connections 2024-03-06T07:30:20.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.650+0000 7fd70f039640 1 -- 172.21.15.59:0/4063462079 wait complete. 2024-03-06T07:30:20.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.651+0000 7fd70f039640 1 Processor -- start 2024-03-06T07:30:20.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.651+0000 7fd70f039640 1 -- start start 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.652+0000 7fd70f039640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd70806cab0 0x7fd7080cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.652+0000 7fd70f039640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7080cea50 0x7fd70809c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.652+0000 7fd70f039640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd70815def0 0x7fd70815cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.652+0000 7fd70f039640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70815e2d0 con 0x7fd7080cea50 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.652+0000 7fd70f039640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7080cc3f0 con 0x7fd70815def0 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.652+0000 7fd70f039640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70815e450 con 0x7fd70806cab0 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.652+0000 7fd707fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7080cea50 0x7fd70809c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.652+0000 7fd707fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7080cea50 0x7fd70809c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36516/0 (socket says 172.21.15.59:36516) 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.652+0000 7fd707fff640 1 -- 172.21.15.59:0/875471690 learned_addr learned my addr 172.21.15.59:0/875471690 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd70cdd5640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd70806cab0 0x7fd7080cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd70d5d6640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd70815def0 0x7fd70815cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd707fff640 1 -- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd70806cab0 msgr2=0x7fd7080cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd707fff640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd70806cab0 0x7fd7080cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd707fff640 1 -- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd70815def0 msgr2=0x7fd70815cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd707fff640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd70815def0 0x7fd70815cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd707fff640 1 -- 172.21.15.59:0/875471690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70809ca40 con 0x7fd7080cea50 2024-03-06T07:30:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd70d5d6640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd70815def0 0x7fd70815cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd707fff640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7080cea50 0x7fd70809c500 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fd6f40097a0 tx=0x7fd6f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd7057fa640 1 -- 172.21.15.59:0/875471690 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4049070 con 0x7fd7080cea50 2024-03-06T07:30:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd7057fa640 1 -- 172.21.15.59:0/875471690 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f4030410 con 0x7fd7080cea50 2024-03-06T07:30:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd7057fa640 1 -- 172.21.15.59:0/875471690 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4034040 con 0x7fd7080cea50 2024-03-06T07:30:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd70f039640 1 -- 172.21.15.59:0/875471690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7080c7d90 con 0x7fd7080cea50 2024-03-06T07:30:20.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.653+0000 7fd70f039640 1 -- 172.21.15.59:0/875471690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7080c59c0 con 0x7fd7080cea50 2024-03-06T07:30:20.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.654+0000 7fd70f039640 1 -- 172.21.15.59:0/875471690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd70809e910 con 0x7fd7080cea50 2024-03-06T07:30:20.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.655+0000 7fd7057fa640 1 -- 172.21.15.59:0/875471690 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6f404c070 con 0x7fd7080cea50 2024-03-06T07:30:20.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.655+0000 7fd7057fa640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd6ec06b500 0x7fd6ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.655+0000 7fd7057fa640 1 -- 172.21.15.59:0/875471690 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6f40abca0 con 0x7fd7080cea50 2024-03-06T07:30:20.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.655+0000 7fd70cdd5640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd6ec06b500 0x7fd6ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.656+0000 7fd70cdd5640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd6ec06b500 0x7fd6ec06d9c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc000f60 tx=0x7fd6fc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.664+0000 7fd7057fa640 1 -- 172.21.15.59:0/875471690 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6f407aa90 con 0x7fd7080cea50 2024-03-06T07:30:20.836 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:20.838 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:20.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:20.884+0000 7fd70f039640 1 -- 172.21.15.59:0/875471690 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fd708068ff0 con 0x7fd7080cea50 2024-03-06T07:30:21.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.121+0000 7fd7057fa640 1 -- 172.21.15.59:0/875471690 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v98) v1 ==== 105+0+0 (secure 0 0 0) 0x7fd6f407a390 con 0x7fd7080cea50 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.123+0000 7fd70f039640 1 -- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd6ec06b500 msgr2=0x7fd6ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.123+0000 7fd70f039640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd6ec06b500 0x7fd6ec06d9c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc000f60 tx=0x7fd6fc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.123+0000 7fd70f039640 1 -- 172.21.15.59:0/875471690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7080cea50 msgr2=0x7fd70809c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.123+0000 7fd70f039640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7080cea50 0x7fd70809c500 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fd6f40097a0 tx=0x7fd6f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.124+0000 7fd70f039640 1 -- 172.21.15.59:0/875471690 shutdown_connections 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.124+0000 7fd70f039640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd6ec06b500 0x7fd6ec06d9c0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.124+0000 7fd70f039640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd70815def0 0x7fd70815cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.124+0000 7fd70f039640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7080cea50 0x7fd70809c500 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.124+0000 7fd70f039640 1 --2- 172.21.15.59:0/875471690 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd70806cab0 0x7fd7080cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.124+0000 7fd70f039640 1 -- 172.21.15.59:0/875471690 >> 172.21.15.59:0/875471690 conn(0x7fd7080b3690 msgr2=0x7fd7080ccaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.124+0000 7fd70f039640 1 -- 172.21.15.59:0/875471690 shutdown_connections 2024-03-06T07:30:21.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.124+0000 7fd70f039640 1 -- 172.21.15.59:0/875471690 wait complete. 2024-03-06T07:30:21.139 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:21.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.238+0000 7fc02caaf640 1 Processor -- start 2024-03-06T07:30:21.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.238+0000 7fc02caaf640 1 -- start start 2024-03-06T07:30:21.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.238+0000 7fc02caaf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0280d22b0 0x7fc0280d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.238+0000 7fc02caaf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc0280b41f0 0x7fc0280b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc02caaf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0280b3430 con 0x7fc02815dfe0 2024-03-06T07:30:21.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc02caaf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0280d2c50 con 0x7fc0280b41f0 2024-03-06T07:30:21.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc02caaf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280b31e0 con 0x7fc0280d22b0 2024-03-06T07:30:21.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc02659c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc02815dfe0 0x7fc0280cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41592/0 (socket says 172.21.15.59:41592) 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc02659c640 1 -- 172.21.15.59:0/2883516391 learned_addr learned my addr 172.21.15.59:0/2883516391 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc026d9d640 1 --2- 172.21.15.59:0/2883516391 >> v2:172.21.15.164:3300/0 conn(0x7fc0280b41f0 0x7fc0280b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc01dd9b640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0280d22b0 0x7fc0280d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715033144 0 0) 0x7fc0280b3430 con 0x7fc02815dfe0 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc014003520 con 0x7fc02815dfe0 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3833187779 0 0) 0x7fc014003520 con 0x7fc02815dfe0 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0280d22b0 msgr2=0x7fc0280d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0280d22b0 0x7fc0280d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 >> v2:172.21.15.164:3300/0 conn(0x7fc0280b41f0 msgr2=0x7fc0280b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 --2- 172.21.15.59:0/2883516391 >> v2:172.21.15.164:3300/0 conn(0x7fc0280b41f0 0x7fc0280b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0280b3710 con 0x7fc02815dfe0 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fc018001670 con 0x7fc02815dfe0 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 >> v1:172.21.15.59:6789/0 conn(0x7fc02815dfe0 legacy=0x7fc0280cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0280d22b0 0x7fc0140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc014003f30 0x7fc014004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc014004d40 0x7fc0140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280b31e0 con 0x7fc014004d40 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01400b9c0 con 0x7fc014003f30 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01400bb40 con 0x7fc0280d22b0 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc026d9d640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc014004d40 0x7fc0140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc02659c640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0280d22b0 0x7fc0140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.239+0000 7fc01dd9b640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc014003f30 0x7fc014004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.240+0000 7fc026d9d640 1 -- 172.21.15.59:0/2883516391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0280d22b0 msgr2=0x7fc0140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.240+0000 7fc026d9d640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0280d22b0 0x7fc0140039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.240+0000 7fc026d9d640 1 -- 172.21.15.59:0/2883516391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc014003f30 msgr2=0x7fc014004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.240+0000 7fc026d9d640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc014003f30 0x7fc014004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.240+0000 7fc026d9d640 1 -- 172.21.15.59:0/2883516391 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014003520 con 0x7fc014004d40 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.240+0000 7fc026d9d640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc014004d40 0x7fc0140051c0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fc020009fc0 tx=0x7fc014005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ad68fb1edacf3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.241+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020039070 con 0x7fc014004d40 2024-03-06T07:30:21.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.241+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02002cc40 con 0x7fc014004d40 2024-03-06T07:30:21.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.241+0000 7fc025d9b640 1 -- 172.21.15.59:0/2883516391 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020009c10 con 0x7fc014004d40 2024-03-06T07:30:21.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.241+0000 7fc02caaf640 1 -- 172.21.15.59:0/2883516391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc014004d40 msgr2=0x7fc0140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.241+0000 7fc02caaf640 1 --2- 172.21.15.59:0/2883516391 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc014004d40 0x7fc0140051c0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fc020009fc0 tx=0x7fc014005830 comp rx=0 tx=0).stop 2024-03-06T07:30:21.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.241+0000 7fc026d9d640 1 -- 172.21.15.59:0/2883516391 reap_dead start 2024-03-06T07:30:21.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.241+0000 7fc02caaf640 1 -- 172.21.15.59:0/2883516391 shutdown_connections 2024-03-06T07:30:21.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.241+0000 7fc02caaf640 1 -- 172.21.15.59:0/2883516391 >> 172.21.15.59:0/2883516391 conn(0x7fc0280d3cd0 msgr2=0x7fc0280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.241+0000 7fc02caaf640 1 -- 172.21.15.59:0/2883516391 shutdown_connections 2024-03-06T07:30:21.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.241+0000 7fc02caaf640 1 -- 172.21.15.59:0/2883516391 wait complete. 2024-03-06T07:30:21.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.243+0000 7fc02caaf640 1 Processor -- start 2024-03-06T07:30:21.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.243+0000 7fc02caaf640 1 -- start start 2024-03-06T07:30:21.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.244+0000 7fc02caaf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc028152c30 0x7fc0280b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc02caaf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0280b8170 0x7fc0280ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc02caaf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc028153010 0x7fc0280cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc02caaf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280cfa00 con 0x7fc028153010 2024-03-06T07:30:21.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc02caaf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280b3d60 con 0x7fc0280b8170 2024-03-06T07:30:21.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc02caaf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280d1c70 con 0x7fc028152c30 2024-03-06T07:30:21.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc026d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc028153010 0x7fc0280cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc026d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc028153010 0x7fc0280cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36526/0 (socket says 172.21.15.59:36526) 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc026d9d640 1 -- 172.21.15.59:0/831027124 learned_addr learned my addr 172.21.15.59:0/831027124 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc01dd9b640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0280b8170 0x7fc0280ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc02659c640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc028152c30 0x7fc0280b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc026d9d640 1 -- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc028152c30 msgr2=0x7fc0280b7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc026d9d640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc028152c30 0x7fc0280b7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc026d9d640 1 -- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0280b8170 msgr2=0x7fc0280ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc026d9d640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0280b8170 0x7fc0280ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc026d9d640 1 -- 172.21.15.59:0/831027124 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0280b68d0 con 0x7fc028153010 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc02659c640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc028152c30 0x7fc0280b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc026d9d640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc028153010 0x7fc0280cd170 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fc020037040 tx=0x7fc020009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc01f7fe640 1 -- 172.21.15.59:0/831027124 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020039030 con 0x7fc028153010 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc01f7fe640 1 -- 172.21.15.59:0/831027124 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02003a040 con 0x7fc028153010 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc02caaf640 1 -- 172.21.15.59:0/831027124 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0280b6b90 con 0x7fc028153010 2024-03-06T07:30:21.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc01f7fe640 1 -- 172.21.15.59:0/831027124 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020009c10 con 0x7fc028153010 2024-03-06T07:30:21.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.245+0000 7fc02caaf640 1 -- 172.21.15.59:0/831027124 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0280b7150 con 0x7fc028153010 2024-03-06T07:30:21.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.247+0000 7fc02caaf640 1 -- 172.21.15.59:0/831027124 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02804f330 con 0x7fc028153010 2024-03-06T07:30:21.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.247+0000 7fc01f7fe640 1 -- 172.21.15.59:0/831027124 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc020045030 con 0x7fc028153010 2024-03-06T07:30:21.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.247+0000 7fc01f7fe640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc00806b4d0 0x7fc00806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.247+0000 7fc01f7fe640 1 -- 172.21.15.59:0/831027124 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0200abc40 con 0x7fc028153010 2024-03-06T07:30:21.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.248+0000 7fc02659c640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc00806b4d0 0x7fc00806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.248+0000 7fc02659c640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc00806b4d0 0x7fc00806d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc018004840 tx=0x7fc018009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.252+0000 7fc01f7fe640 1 -- 172.21.15.59:0/831027124 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc02004c060 con 0x7fc028153010 2024-03-06T07:30:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.472+0000 7fc02caaf640 1 -- 172.21.15.59:0/831027124 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc02806d530 con 0x7fc028153010 2024-03-06T07:30:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.472+0000 7fc01f7fe640 1 -- 172.21.15.59:0/831027124 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4997 (secure 0 0 0) 0x7fc02007ab80 con 0x7fc028153010 2024-03-06T07:30:21.474 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:21.474 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":99,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4015228738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4015228738},{"type":"v1","addr":"172.21.15.164:6836","nonce":4015228738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93},{"gid":7247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2855659185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2855659185},{"type":"v1","addr":"172.21.15.59:6835","nonce":2855659185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:21.124585+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:creating","state_seq":1,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:30:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 99 2024-03-06T07:30:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 -- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc00806b4d0 msgr2=0x7fc00806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc00806b4d0 0x7fc00806d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc018004840 tx=0x7fc018009040 comp rx=0 tx=0).stop 2024-03-06T07:30:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 -- 172.21.15.59:0/831027124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc028153010 msgr2=0x7fc0280cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc028153010 0x7fc0280cd170 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fc020037040 tx=0x7fc020009750 comp rx=0 tx=0).stop 2024-03-06T07:30:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 -- 172.21.15.59:0/831027124 shutdown_connections 2024-03-06T07:30:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc00806b4d0 0x7fc00806d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc028153010 0x7fc0280cd170 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0280b8170 0x7fc0280ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 --2- 172.21.15.59:0/831027124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc028152c30 0x7fc0280b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 -- 172.21.15.59:0/831027124 >> 172.21.15.59:0/831027124 conn(0x7fc0280d3cd0 msgr2=0x7fc0280b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 -- 172.21.15.59:0/831027124 shutdown_connections 2024-03-06T07:30:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:21.474+0000 7fc02caaf640 1 -- 172.21.15.59:0/831027124 wait complete. 2024-03-06T07:30:21.487 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:30:26.489 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.588+0000 7f936f3ee640 1 Processor -- start 2024-03-06T07:30:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.588+0000 7f936f3ee640 1 -- start start 2024-03-06T07:30:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936f3ee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93680d22b0 0x7f93680d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936f3ee640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f93680b41f0 0x7f93680b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936f3ee640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93680b3430 con 0x7f936815dfe0 2024-03-06T07:30:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936f3ee640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93680d2c50 con 0x7f93680b41f0 2024-03-06T07:30:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936f3ee640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93680b31e0 con 0x7f93680d22b0 2024-03-06T07:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936d18a640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f936815dfe0 0x7f93680cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51128/0 (socket says 172.21.15.59:51128) 2024-03-06T07:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936d18a640 1 -- 172.21.15.59:0/3704877069 learned_addr learned my addr 172.21.15.59:0/3704877069 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936d98b640 1 --2- 172.21.15.59:0/3704877069 >> v2:172.21.15.164:3300/0 conn(0x7f93680b41f0 0x7f93680b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936c989640 1 --2- 172.21.15.59:0/3704877069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93680d22b0 0x7f93680d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f935ffff640 1 -- 172.21.15.59:0/3704877069 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352136455 0 0) 0x7f93680b3430 con 0x7f936815dfe0 2024-03-06T07:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f935ffff640 1 -- 172.21.15.59:0/3704877069 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9354003520 con 0x7f936815dfe0 2024-03-06T07:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936c989640 1 -- 172.21.15.59:0/3704877069 >> v2:172.21.15.164:3300/0 conn(0x7f93680b41f0 msgr2=0x7f93680b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936c989640 1 --2- 172.21.15.59:0/3704877069 >> v2:172.21.15.164:3300/0 conn(0x7f93680b41f0 0x7f93680b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936c989640 1 -- 172.21.15.59:0/3704877069 >> v1:172.21.15.59:6789/0 conn(0x7f936815dfe0 legacy=0x7f93680cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936c989640 1 -- 172.21.15.59:0/3704877069 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93680b3710 con 0x7f93680d22b0 2024-03-06T07:30:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.589+0000 7f936d98b640 1 --2- 172.21.15.59:0/3704877069 >> v2:172.21.15.164:3300/0 conn(0x7f93680b41f0 0x7f93680b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.590+0000 7f936c989640 1 --2- 172.21.15.59:0/3704877069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93680d22b0 0x7f93680d2710 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f93580097a0 tx=0x7f935802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f33d4bbf2d7e6dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.591+0000 7f935ffff640 1 -- 172.21.15.59:0/3704877069 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358035020 con 0x7f93680d22b0 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.591+0000 7f935ffff640 1 -- 172.21.15.59:0/3704877069 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9358031830 con 0x7f93680d22b0 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.591+0000 7f935ffff640 1 -- 172.21.15.59:0/3704877069 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358031c00 con 0x7f93680d22b0 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.591+0000 7f936f3ee640 1 -- 172.21.15.59:0/3704877069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93680d22b0 msgr2=0x7f93680d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.591+0000 7f936f3ee640 1 --2- 172.21.15.59:0/3704877069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93680d22b0 0x7f93680d2710 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f93580097a0 tx=0x7f935802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.591+0000 7f936f3ee640 1 -- 172.21.15.59:0/3704877069 shutdown_connections 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.591+0000 7f936f3ee640 1 --2- 172.21.15.59:0/3704877069 >> v2:172.21.15.164:3300/0 conn(0x7f93680b41f0 0x7f93680b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.591+0000 7f936f3ee640 1 --2- 172.21.15.59:0/3704877069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93680d22b0 0x7f93680d2710 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.591+0000 7f936f3ee640 1 -- 172.21.15.59:0/3704877069 >> 172.21.15.59:0/3704877069 conn(0x7f93680d3cd0 msgr2=0x7f93680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.591+0000 7f936f3ee640 1 -- 172.21.15.59:0/3704877069 shutdown_connections 2024-03-06T07:30:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.591+0000 7f936f3ee640 1 -- 172.21.15.59:0/3704877069 wait complete. 2024-03-06T07:30:26.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.593+0000 7f936f3ee640 1 Processor -- start 2024-03-06T07:30:26.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.593+0000 7f936f3ee640 1 -- start start 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.594+0000 7f936f3ee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93680b41f0 0x7f936814f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.594+0000 7f936f3ee640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93680d22b0 0x7f936806ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.594+0000 7f936f3ee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f936815dfe0 0x7f936806bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.594+0000 7f936f3ee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93680cfa00 con 0x7f93680b41f0 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.594+0000 7f936f3ee640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93680b3d60 con 0x7f93680d22b0 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.594+0000 7f936f3ee640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93680d1c70 con 0x7f936815dfe0 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.594+0000 7f936d18a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93680b41f0 0x7f936814f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.594+0000 7f936d18a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93680b41f0 0x7f936814f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50982/0 (socket says 172.21.15.59:50982) 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.594+0000 7f936d18a640 1 -- 172.21.15.59:0/2320690060 learned_addr learned my addr 172.21.15.59:0/2320690060 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f936d98b640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f936815dfe0 0x7f936806bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f936c989640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93680d22b0 0x7f936806ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f936d18a640 1 -- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f936815dfe0 msgr2=0x7f936806bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f936d18a640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f936815dfe0 0x7f936806bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f936d18a640 1 -- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93680d22b0 msgr2=0x7f936806ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f936d18a640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93680d22b0 0x7f936806ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f936d18a640 1 -- 172.21.15.59:0/2320690060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936815e540 con 0x7f93680b41f0 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f936d18a640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93680b41f0 0x7f936814f120 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f936400c390 tx=0x7f936400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f935dffb640 1 -- 172.21.15.59:0/2320690060 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364017070 con 0x7f93680b41f0 2024-03-06T07:30:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f935dffb640 1 -- 172.21.15.59:0/2320690060 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9364005ce0 con 0x7f93680b41f0 2024-03-06T07:30:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f936f3ee640 1 -- 172.21.15.59:0/2320690060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936815e800 con 0x7f93680b41f0 2024-03-06T07:30:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f935dffb640 1 -- 172.21.15.59:0/2320690060 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93640072f0 con 0x7f93680b41f0 2024-03-06T07:30:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.595+0000 7f936f3ee640 1 -- 172.21.15.59:0/2320690060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936815edc0 con 0x7f93680b41f0 2024-03-06T07:30:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.596+0000 7f936f3ee640 1 -- 172.21.15.59:0/2320690060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93680cf5f0 con 0x7f93680b41f0 2024-03-06T07:30:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.597+0000 7f935dffb640 1 -- 172.21.15.59:0/2320690060 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f936401a070 con 0x7f93680b41f0 2024-03-06T07:30:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.597+0000 7f935dffb640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f934c06b4d0 0x7f934c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.597+0000 7f935dffb640 1 -- 172.21.15.59:0/2320690060 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93640894a0 con 0x7f93680b41f0 2024-03-06T07:30:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.597+0000 7f936c989640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f934c06b4d0 0x7f934c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.598+0000 7f936c989640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f934c06b4d0 0x7f934c06d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9358036000 tx=0x7f9358008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.602+0000 7f935dffb640 1 -- 172.21.15.59:0/2320690060 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93680cf5f0 con 0x7f93680b41f0 2024-03-06T07:30:26.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.820+0000 7f936f3ee640 1 -- 172.21.15.59:0/2320690060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f936804f330 con 0x7f93680b41f0 2024-03-06T07:30:26.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.821+0000 7f935dffb640 1 -- 172.21.15.59:0/2320690060 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5002 (secure 0 0 0) 0x7f9364057cd0 con 0x7f93680b41f0 2024-03-06T07:30:26.823 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:26.823 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4015228738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4015228738},{"type":"v1","addr":"172.21.15.164:6836","nonce":4015228738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93},{"gid":7247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2855659185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2855659185},{"type":"v1","addr":"172.21.15.59:6835","nonce":2855659185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:22.126145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[8236,9336]},"id":10}]} 2024-03-06T07:30:26.823 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 100 2024-03-06T07:30:26.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 -- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f934c06b4d0 msgr2=0x7f934c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f934c06b4d0 0x7f934c06d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9358036000 tx=0x7f9358008030 comp rx=0 tx=0).stop 2024-03-06T07:30:26.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 -- 172.21.15.59:0/2320690060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93680b41f0 msgr2=0x7f936814f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93680b41f0 0x7f936814f120 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f936400c390 tx=0x7f936400c790 comp rx=0 tx=0).stop 2024-03-06T07:30:26.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 -- 172.21.15.59:0/2320690060 shutdown_connections 2024-03-06T07:30:26.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f934c06b4d0 0x7f934c06d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f936815dfe0 0x7f936806bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93680d22b0 0x7f936806ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 --2- 172.21.15.59:0/2320690060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93680b41f0 0x7f936814f120 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 -- 172.21.15.59:0/2320690060 >> 172.21.15.59:0/2320690060 conn(0x7f93680d3cd0 msgr2=0x7f936806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 -- 172.21.15.59:0/2320690060 shutdown_connections 2024-03-06T07:30:26.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.823+0000 7f936f3ee640 1 -- 172.21.15.59:0/2320690060 wait complete. 2024-03-06T07:30:26.836 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:26.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.936+0000 7f291a81c640 1 Processor -- start 2024-03-06T07:30:26.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.936+0000 7f291a81c640 1 -- start start 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.936+0000 7f291a81c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f29140d22b0 0x7f29140d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.936+0000 7f291a81c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f29140b41f0 0x7f29140b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.936+0000 7f291a81c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29140b3430 con 0x7f291415dfe0 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.936+0000 7f291a81c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29140d2c50 con 0x7f29140d22b0 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.936+0000 7f291a81c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29140b31e0 con 0x7f29140b41f0 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.936+0000 7f29137fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f29140d22b0 0x7f29140d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.937+0000 7f29137fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f29140d22b0 0x7f29140d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50988/0 (socket says 172.21.15.59:50988) 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.937+0000 7f29137fe640 1 -- 172.21.15.59:0/3428479649 learned_addr learned my addr 172.21.15.59:0/3428479649 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.937+0000 7f2918db9640 1 --2- 172.21.15.59:0/3428479649 >> v2:172.21.15.164:3300/0 conn(0x7f29140b41f0 0x7f29140b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.937+0000 7f29137fe640 1 -- 172.21.15.59:0/3428479649 >> v2:172.21.15.164:3300/0 conn(0x7f29140b41f0 msgr2=0x7f29140b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.937+0000 7f29137fe640 1 --2- 172.21.15.59:0/3428479649 >> v2:172.21.15.164:3300/0 conn(0x7f29140b41f0 0x7f29140b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.937+0000 7f29137fe640 1 -- 172.21.15.59:0/3428479649 >> v1:172.21.15.164:6789/0 conn(0x7f291415dfe0 legacy=0x7f29140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.937+0000 7f29137fe640 1 -- 172.21.15.59:0/3428479649 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29140b3710 con 0x7f29140d22b0 2024-03-06T07:30:26.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.938+0000 7f29137fe640 1 --2- 172.21.15.59:0/3428479649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29140d22b0 0x7f29140d2710 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f2904009870 tx=0x7f290402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952ce61918bd7e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.938+0000 7f2912ffd640 1 -- 172.21.15.59:0/3428479649 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904034070 con 0x7f29140d22b0 2024-03-06T07:30:26.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.938+0000 7f2912ffd640 1 -- 172.21.15.59:0/3428479649 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2904030860 con 0x7f29140d22b0 2024-03-06T07:30:26.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.938+0000 7f2912ffd640 1 -- 172.21.15.59:0/3428479649 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904030c30 con 0x7f29140d22b0 2024-03-06T07:30:26.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.939+0000 7f291a81c640 1 -- 172.21.15.59:0/3428479649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29140d22b0 msgr2=0x7f29140d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.939+0000 7f291a81c640 1 --2- 172.21.15.59:0/3428479649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29140d22b0 0x7f29140d2710 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f2904009870 tx=0x7f290402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.939+0000 7f291a81c640 1 -- 172.21.15.59:0/3428479649 shutdown_connections 2024-03-06T07:30:26.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.939+0000 7f291a81c640 1 --2- 172.21.15.59:0/3428479649 >> v2:172.21.15.164:3300/0 conn(0x7f29140b41f0 0x7f29140b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.939+0000 7f291a81c640 1 --2- 172.21.15.59:0/3428479649 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29140d22b0 0x7f29140d2710 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.939+0000 7f291a81c640 1 -- 172.21.15.59:0/3428479649 >> 172.21.15.59:0/3428479649 conn(0x7f29140d3cd0 msgr2=0x7f29140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.939+0000 7f291a81c640 1 -- 172.21.15.59:0/3428479649 shutdown_connections 2024-03-06T07:30:26.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.939+0000 7f291a81c640 1 -- 172.21.15.59:0/3428479649 wait complete. 2024-03-06T07:30:26.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.941+0000 7f291a81c640 1 Processor -- start 2024-03-06T07:30:26.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.941+0000 7f291a81c640 1 -- start start 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f291a81c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29140b41f0 0x7f291414b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f291a81c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29140d22b0 0x7f29141f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f291a81c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f291415dfe0 0x7f2914160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f291a81c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29140cfa00 con 0x7f29140d22b0 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f291a81c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29140b3d60 con 0x7f291415dfe0 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f291a81c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29140d1c70 con 0x7f29140b41f0 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f29137fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29140d22b0 0x7f29141f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f2913fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29140b41f0 0x7f291414b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f29137fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29140d22b0 0x7f29141f6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51002/0 (socket says 172.21.15.59:51002) 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f29137fe640 1 -- 172.21.15.59:0/1282687258 learned_addr learned my addr 172.21.15.59:0/1282687258 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f2918db9640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f291415dfe0 0x7f2914160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f29137fe640 1 -- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29140b41f0 msgr2=0x7f291414b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f29137fe640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29140b41f0 0x7f291414b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f29137fe640 1 -- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f291415dfe0 msgr2=0x7f2914160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f29137fe640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f291415dfe0 0x7f2914160320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f29137fe640 1 -- 172.21.15.59:0/1282687258 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291415a640 con 0x7f29140d22b0 2024-03-06T07:30:26.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.942+0000 7f29137fe640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29140d22b0 0x7f29141f6a30 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f2904000c00 tx=0x7f2904004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.943+0000 7f2910ff9640 1 -- 172.21.15.59:0/1282687258 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904048070 con 0x7f29140d22b0 2024-03-06T07:30:26.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.943+0000 7f2910ff9640 1 -- 172.21.15.59:0/1282687258 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2904030660 con 0x7f29140d22b0 2024-03-06T07:30:26.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.943+0000 7f291a81c640 1 -- 172.21.15.59:0/1282687258 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29140cae40 con 0x7f29140d22b0 2024-03-06T07:30:26.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.943+0000 7f2910ff9640 1 -- 172.21.15.59:0/1282687258 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904034040 con 0x7f29140d22b0 2024-03-06T07:30:26.946 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.943+0000 7f291a81c640 1 -- 172.21.15.59:0/1282687258 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291415fdc0 con 0x7f29140d22b0 2024-03-06T07:30:26.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.944+0000 7f291a81c640 1 -- 172.21.15.59:0/1282687258 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291404f330 con 0x7f29140d22b0 2024-03-06T07:30:26.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.944+0000 7f2910ff9640 1 -- 172.21.15.59:0/1282687258 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2904037070 con 0x7f29140d22b0 2024-03-06T07:30:26.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.945+0000 7f2910ff9640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f290006b500 0x7f290006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.945+0000 7f2910ff9640 1 -- 172.21.15.59:0/1282687258 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29040ac210 con 0x7f29140d22b0 2024-03-06T07:30:26.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.945+0000 7f2913fff640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f290006b500 0x7f290006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.947 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.945+0000 7f2913fff640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f290006b500 0x7f290006d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f29140cf6a0 tx=0x7f2908009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:26.949+0000 7f2910ff9640 1 -- 172.21.15.59:0/1282687258 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f290407ab70 con 0x7f29140d22b0 2024-03-06T07:30:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.168+0000 7f291a81c640 1 -- 172.21.15.59:0/1282687258 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29140cb1d0 con 0x7f29140d22b0 2024-03-06T07:30:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.168+0000 7f2910ff9640 1 -- 172.21.15.59:0/1282687258 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5002 (secure 0 0 0) 0x7f290407ec10 con 0x7f29140d22b0 2024-03-06T07:30:27.170 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:27.170 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":100,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6836/4015228738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":4015228738},{"type":"v1","addr":"172.21.15.164:6836","nonce":4015228738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93},{"gid":7247,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6835/2855659185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6834","nonce":2855659185},{"type":"v1","addr":"172.21.15.59:6835","nonce":2855659185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:22.126145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[8236,9336]},"id":10}]} 2024-03-06T07:30:27.171 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 100 2024-03-06T07:30:27.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 -- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f290006b500 msgr2=0x7f290006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f290006b500 0x7f290006d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f29140cf6a0 tx=0x7f2908009040 comp rx=0 tx=0).stop 2024-03-06T07:30:27.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 -- 172.21.15.59:0/1282687258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29140d22b0 msgr2=0x7f29141f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29140d22b0 0x7f29141f6a30 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f2904000c00 tx=0x7f2904004730 comp rx=0 tx=0).stop 2024-03-06T07:30:27.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 -- 172.21.15.59:0/1282687258 shutdown_connections 2024-03-06T07:30:27.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f290006b500 0x7f290006d9c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f291415dfe0 0x7f2914160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29140d22b0 0x7f29141f6a30 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 --2- 172.21.15.59:0/1282687258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29140b41f0 0x7f291414b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 -- 172.21.15.59:0/1282687258 >> 172.21.15.59:0/1282687258 conn(0x7f29140d3cd0 msgr2=0x7f29140d08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 -- 172.21.15.59:0/1282687258 shutdown_connections 2024-03-06T07:30:27.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.170+0000 7f291a81c640 1 -- 172.21.15.59:0/1282687258 wait complete. 2024-03-06T07:30:27.183 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are c and b 2024-03-06T07:30:27.184 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:30:27.184 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:27.192 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:30:27.193 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:30:27.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.335+0000 7f62f4702640 1 Processor -- start 2024-03-06T07:30:27.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.335+0000 7f62f4702640 1 -- start start 2024-03-06T07:30:27.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f4702640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62ec1614a0 0x7f62ec06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f4702640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f62ec0b0270 0x7f62ec0b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f4702640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62ec0af4b0 con 0x7f62ec0b1850 2024-03-06T07:30:27.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f4702640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62ec0af790 con 0x7f62ec0b0270 2024-03-06T07:30:27.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f4702640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec0afa70 con 0x7f62ec1614a0 2024-03-06T07:30:27.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f1c9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f62ec0b0270 0x7f62ec0b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f1c9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f62ec0b0270 0x7f62ec0b1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51016/0 (socket says 172.21.15.59:51016) 2024-03-06T07:30:27.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f1c9d640 1 -- 172.21.15.59:0/820344544 learned_addr learned my addr 172.21.15.59:0/820344544 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:27.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f249e640 1 --2- 172.21.15.59:0/820344544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62ec1614a0 0x7f62ec06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f1c9d640 1 -- 172.21.15.59:0/820344544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62ec1614a0 msgr2=0x7f62ec06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f1c9d640 1 --2- 172.21.15.59:0/820344544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62ec1614a0 0x7f62ec06d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f1c9d640 1 -- 172.21.15.59:0/820344544 >> v1:172.21.15.59:6789/0 conn(0x7f62ec0b1850 legacy=0x7f62ec06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f1c9d640 1 -- 172.21.15.59:0/820344544 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec0afd50 con 0x7f62ec0b0270 2024-03-06T07:30:27.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.336+0000 7f62f149c640 1 -- 172.21.15.59:0/820344544 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267790239 0 0) 0x7f62ec0af4b0 con 0x7f62ec0b1850 2024-03-06T07:30:27.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.337+0000 7f62f1c9d640 1 --2- 172.21.15.59:0/820344544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62ec0b0270 0x7f62ec0b1310 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009870 tx=0x7f62dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b48196b0802e1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.337+0000 7f62f149c640 1 -- 172.21.15.59:0/820344544 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc035020 con 0x7f62ec0b0270 2024-03-06T07:30:27.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.337+0000 7f62f149c640 1 -- 172.21.15.59:0/820344544 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62dc031640 con 0x7f62ec0b0270 2024-03-06T07:30:27.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.337+0000 7f62f149c640 1 -- 172.21.15.59:0/820344544 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc02d040 con 0x7f62ec0b0270 2024-03-06T07:30:27.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.337+0000 7f62f4702640 1 -- 172.21.15.59:0/820344544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62ec0b0270 msgr2=0x7f62ec0b1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.337+0000 7f62f4702640 1 --2- 172.21.15.59:0/820344544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62ec0b0270 0x7f62ec0b1310 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009870 tx=0x7f62dc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.338+0000 7f62f4702640 1 -- 172.21.15.59:0/820344544 shutdown_connections 2024-03-06T07:30:27.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.338+0000 7f62f4702640 1 --2- 172.21.15.59:0/820344544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62ec0b0270 0x7f62ec0b1310 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.338+0000 7f62f4702640 1 --2- 172.21.15.59:0/820344544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62ec1614a0 0x7f62ec06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.338+0000 7f62f4702640 1 -- 172.21.15.59:0/820344544 >> 172.21.15.59:0/820344544 conn(0x7f62ec09a9f0 msgr2=0x7f62ec09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.338+0000 7f62f4702640 1 -- 172.21.15.59:0/820344544 shutdown_connections 2024-03-06T07:30:27.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.338+0000 7f62f4702640 1 -- 172.21.15.59:0/820344544 wait complete. 2024-03-06T07:30:27.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.339+0000 7f62f4702640 1 Processor -- start 2024-03-06T07:30:27.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.339+0000 7f62f4702640 1 -- start start 2024-03-06T07:30:27.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f4702640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62ec0b0270 0x7f62ec14b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f4702640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62ec0b1850 0x7f62ec06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f4702640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62ec1614a0 0x7f62ec06f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f4702640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec0b6520 con 0x7f62ec0b0270 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f4702640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec0b06b0 con 0x7f62ec1614a0 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f4702640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec0b1c80 con 0x7f62ec0b1850 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f249e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62ec0b0270 0x7f62ec14b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f249e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62ec0b0270 0x7f62ec14b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51024/0 (socket says 172.21.15.59:51024) 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f249e640 1 -- 172.21.15.59:0/2557267252 learned_addr learned my addr 172.21.15.59:0/2557267252 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f1c9d640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62ec0b1850 0x7f62ec06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f2c9f640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62ec1614a0 0x7f62ec06f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f249e640 1 -- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62ec0b1850 msgr2=0x7f62ec06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f249e640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62ec0b1850 0x7f62ec06ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f249e640 1 -- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62ec1614a0 msgr2=0x7f62ec06f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f249e640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62ec1614a0 0x7f62ec06f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f249e640 1 -- 172.21.15.59:0/2557267252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec0caeb0 con 0x7f62ec0b0270 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f249e640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62ec0b0270 0x7f62ec14b420 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f62e800e8a0 tx=0x7f62e800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62e2ffd640 1 -- 172.21.15.59:0/2557267252 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e8017070 con 0x7f62ec0b0270 2024-03-06T07:30:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62e2ffd640 1 -- 172.21.15.59:0/2557267252 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e800c940 con 0x7f62ec0b0270 2024-03-06T07:30:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f4702640 1 -- 172.21.15.59:0/2557267252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ec09f2a0 con 0x7f62ec0b0270 2024-03-06T07:30:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62e2ffd640 1 -- 172.21.15.59:0/2557267252 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e800cd30 con 0x7f62ec0b0270 2024-03-06T07:30:27.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.340+0000 7f62f4702640 1 -- 172.21.15.59:0/2557267252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ec09f860 con 0x7f62ec0b0270 2024-03-06T07:30:27.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.341+0000 7f62f4702640 1 -- 172.21.15.59:0/2557267252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ec04f330 con 0x7f62ec0b0270 2024-03-06T07:30:27.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.342+0000 7f62e2ffd640 1 -- 172.21.15.59:0/2557267252 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62e801a070 con 0x7f62ec0b0270 2024-03-06T07:30:27.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.342+0000 7f62e2ffd640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d006b4d0 0x7f62d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.343+0000 7f62e2ffd640 1 -- 172.21.15.59:0/2557267252 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62e808a340 con 0x7f62ec0b0270 2024-03-06T07:30:27.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.343+0000 7f62f1c9d640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d006b4d0 0x7f62d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.343+0000 7f62f1c9d640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d006b4d0 0x7f62d006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f62dc036000 tx=0x7f62dc008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.347+0000 7f62e2ffd640 1 -- 172.21.15.59:0/2557267252 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62e8058c00 con 0x7f62ec0b0270 2024-03-06T07:30:27.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:27.602+0000 7f62f4702640 1 -- 172.21.15.59:0/2557267252 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f62ec06e8e0 con 0x7f62ec0b0270 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.139+0000 7f62e2ffd640 1 -- 172.21.15.59:0/2557267252 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7247 v101) v1 ==== 95+0+0 (secure 0 0 0) 0x7f62e805cca0 con 0x7f62ec0b0270 2024-03-06T07:30:28.141 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 7247 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 -- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d006b4d0 msgr2=0x7f62d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d006b4d0 0x7f62d006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f62dc036000 tx=0x7f62dc008030 comp rx=0 tx=0).stop 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 -- 172.21.15.59:0/2557267252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62ec0b0270 msgr2=0x7f62ec14b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62ec0b0270 0x7f62ec14b420 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f62e800e8a0 tx=0x7f62e800ed60 comp rx=0 tx=0).stop 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 -- 172.21.15.59:0/2557267252 shutdown_connections 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f62d006b4d0 0x7f62d006d990 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62ec1614a0 0x7f62ec06f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62ec0b1850 0x7f62ec06ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 --2- 172.21.15.59:0/2557267252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62ec0b0270 0x7f62ec14b420 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 -- 172.21.15.59:0/2557267252 >> 172.21.15.59:0/2557267252 conn(0x7f62ec09a9f0 msgr2=0x7f62ec0b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 -- 172.21.15.59:0/2557267252 shutdown_connections 2024-03-06T07:30:28.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.141+0000 7f62f4702640 1 -- 172.21.15.59:0/2557267252 wait complete. 2024-03-06T07:30:28.155 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:30:28.155 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:28.163 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:30:28.164 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:30:28.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.265+0000 7fc0cb13e640 1 Processor -- start 2024-03-06T07:30:28.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.265+0000 7fc0cb13e640 1 -- start start 2024-03-06T07:30:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.265+0000 7fc0cb13e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0c40b3240 0x7fc0c40b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.265+0000 7fc0cb13e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc0c40b2910 0x7fc0c40d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.265+0000 7fc0cb13e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c415e3f0 con 0x7fc0c415e010 2024-03-06T07:30:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.265+0000 7fc0cb13e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c40b1f40 con 0x7fc0c40b2910 2024-03-06T07:30:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.265+0000 7fc0cb13e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c40b1c10 con 0x7fc0c40b3240 2024-03-06T07:30:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.265+0000 7fc0c8eda640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc0c415e010 0x7fc0c40cedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51156/0 (socket says 172.21.15.59:51156) 2024-03-06T07:30:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.266+0000 7fc0c8eda640 1 -- 172.21.15.59:0/3371704675 learned_addr learned my addr 172.21.15.59:0/3371704675 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.266+0000 7fc0c3fff640 1 --2- 172.21.15.59:0/3371704675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0c40b3240 0x7fc0c40b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.266+0000 7fc0c96db640 1 --2- 172.21.15.59:0/3371704675 >> v2:172.21.15.164:3300/0 conn(0x7fc0c40b2910 0x7fc0c40d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.266+0000 7fc0c37fe640 1 -- 172.21.15.59:0/3371704675 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492132332 0 0) 0x7fc0c415e3f0 con 0x7fc0c415e010 2024-03-06T07:30:28.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.266+0000 7fc0c37fe640 1 -- 172.21.15.59:0/3371704675 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b0003520 con 0x7fc0c415e010 2024-03-06T07:30:28.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.266+0000 7fc0c3fff640 1 -- 172.21.15.59:0/3371704675 >> v2:172.21.15.164:3300/0 conn(0x7fc0c40b2910 msgr2=0x7fc0c40d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.266+0000 7fc0c3fff640 1 --2- 172.21.15.59:0/3371704675 >> v2:172.21.15.164:3300/0 conn(0x7fc0c40b2910 0x7fc0c40d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.266+0000 7fc0c3fff640 1 -- 172.21.15.59:0/3371704675 >> v1:172.21.15.59:6789/0 conn(0x7fc0c415e010 legacy=0x7fc0c40cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.266+0000 7fc0c3fff640 1 -- 172.21.15.59:0/3371704675 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c406d1b0 con 0x7fc0c40b3240 2024-03-06T07:30:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.267+0000 7fc0c3fff640 1 --2- 172.21.15.59:0/3371704675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0c40b3240 0x7fc0c40b23d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009dc0 tx=0x7fc0b402be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17f03519d69c01b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.267+0000 7fc0c37fe640 1 -- 172.21.15.59:0/3371704675 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4039070 con 0x7fc0c40b3240 2024-03-06T07:30:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.267+0000 7fc0c37fe640 1 -- 172.21.15.59:0/3371704675 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b402cbb0 con 0x7fc0c40b3240 2024-03-06T07:30:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.267+0000 7fc0c37fe640 1 -- 172.21.15.59:0/3371704675 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4034440 con 0x7fc0c40b3240 2024-03-06T07:30:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.268+0000 7fc0cb13e640 1 -- 172.21.15.59:0/3371704675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0c40b3240 msgr2=0x7fc0c40b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.268+0000 7fc0cb13e640 1 --2- 172.21.15.59:0/3371704675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0c40b3240 0x7fc0c40b23d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009dc0 tx=0x7fc0b402be00 comp rx=0 tx=0).stop 2024-03-06T07:30:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.268+0000 7fc0cb13e640 1 -- 172.21.15.59:0/3371704675 shutdown_connections 2024-03-06T07:30:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.268+0000 7fc0cb13e640 1 --2- 172.21.15.59:0/3371704675 >> v2:172.21.15.164:3300/0 conn(0x7fc0c40b2910 0x7fc0c40d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.270 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.268+0000 7fc0cb13e640 1 --2- 172.21.15.59:0/3371704675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0c40b3240 0x7fc0c40b23d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.268+0000 7fc0cb13e640 1 -- 172.21.15.59:0/3371704675 >> 172.21.15.59:0/3371704675 conn(0x7fc0c409a9f0 msgr2=0x7fc0c409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.268+0000 7fc0cb13e640 1 -- 172.21.15.59:0/3371704675 shutdown_connections 2024-03-06T07:30:28.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.268+0000 7fc0cb13e640 1 -- 172.21.15.59:0/3371704675 wait complete. 2024-03-06T07:30:28.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.270+0000 7fc0cb13e640 1 Processor -- start 2024-03-06T07:30:28.271 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.270+0000 7fc0cb13e640 1 -- start start 2024-03-06T07:30:28.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.271+0000 7fc0cb13e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c415e010 0x7fc0c4154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.271+0000 7fc0cb13e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0c40b6a20 0x7fc0c40b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.271+0000 7fc0cb13e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0c409d060 0x7fc0c409d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.271+0000 7fc0cb13e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c40d2010 con 0x7fc0c415e010 2024-03-06T07:30:28.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.271+0000 7fc0cb13e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c406ced0 con 0x7fc0c409d060 2024-03-06T07:30:28.274 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.271+0000 7fc0cb13e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c40d2190 con 0x7fc0c40b6a20 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.271+0000 7fc0c8eda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c415e010 0x7fc0c4154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.271+0000 7fc0c8eda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c415e010 0x7fc0c4154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51028/0 (socket says 172.21.15.59:51028) 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.271+0000 7fc0c8eda640 1 -- 172.21.15.59:0/203262257 learned_addr learned my addr 172.21.15.59:0/203262257 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.271+0000 7fc0c3fff640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0c40b6a20 0x7fc0c40b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.271+0000 7fc0c96db640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0c409d060 0x7fc0c409d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.272+0000 7fc0c8eda640 1 -- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0c40b6a20 msgr2=0x7fc0c40b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.272+0000 7fc0c8eda640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0c40b6a20 0x7fc0c40b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.272+0000 7fc0c8eda640 1 -- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0c409d060 msgr2=0x7fc0c409d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.272+0000 7fc0c8eda640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0c409d060 0x7fc0c409d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.272+0000 7fc0c8eda640 1 -- 172.21.15.59:0/203262257 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c4161300 con 0x7fc0c415e010 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.272+0000 7fc0c8eda640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c415e010 0x7fc0c4154280 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac00c390 tx=0x7fc0ac00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.272+0000 7fc0c17fa640 1 -- 172.21.15.59:0/203262257 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac017070 con 0x7fc0c415e010 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.272+0000 7fc0cb13e640 1 -- 172.21.15.59:0/203262257 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c415f320 con 0x7fc0c415e010 2024-03-06T07:30:28.275 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.272+0000 7fc0c17fa640 1 -- 172.21.15.59:0/203262257 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ac005ce0 con 0x7fc0c415e010 2024-03-06T07:30:28.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.272+0000 7fc0c17fa640 1 -- 172.21.15.59:0/203262257 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac0072f0 con 0x7fc0c415e010 2024-03-06T07:30:28.276 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.272+0000 7fc0cb13e640 1 -- 172.21.15.59:0/203262257 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c415f8e0 con 0x7fc0c415e010 2024-03-06T07:30:28.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.273+0000 7fc0cb13e640 1 -- 172.21.15.59:0/203262257 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c40cf260 con 0x7fc0c415e010 2024-03-06T07:30:28.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.273+0000 7fc0c17fa640 1 -- 172.21.15.59:0/203262257 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0ac01a070 con 0x7fc0c415e010 2024-03-06T07:30:28.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.274+0000 7fc0c17fa640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc0a406b500 0x7fc0a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.274+0000 7fc0c17fa640 1 -- 172.21.15.59:0/203262257 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0ac0895a0 con 0x7fc0c415e010 2024-03-06T07:30:28.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.274+0000 7fc0c3fff640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc0a406b500 0x7fc0a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.277 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.274+0000 7fc0c3fff640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc0a406b500 0x7fc0a406d9c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4036000 tx=0x7fc0b4030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.280 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.278+0000 7fc0c17fa640 1 -- 172.21.15.59:0/203262257 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0c40cf260 con 0x7fc0c415e010 2024-03-06T07:30:28.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:28.573+0000 7fc0cb13e640 1 -- 172.21.15.59:0/203262257 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7fc0c404f330 con 0x7fc0c415e010 2024-03-06T07:30:29.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.140+0000 7fc0c17fa640 1 -- 172.21.15.59:0/203262257 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 7235 v102) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc0ac057d50 con 0x7fc0c415e010 2024-03-06T07:30:29.142 INFO:teuthology.orchestra.run.smithi059.stderr:failed mds gid 7235 2024-03-06T07:30:29.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 -- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc0a406b500 msgr2=0x7fc0a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc0a406b500 0x7fc0a406d9c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4036000 tx=0x7fc0b4030000 comp rx=0 tx=0).stop 2024-03-06T07:30:29.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 -- 172.21.15.59:0/203262257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c415e010 msgr2=0x7fc0c4154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c415e010 0x7fc0c4154280 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac00c390 tx=0x7fc0ac00c850 comp rx=0 tx=0).stop 2024-03-06T07:30:29.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 -- 172.21.15.59:0/203262257 shutdown_connections 2024-03-06T07:30:29.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc0a406b500 0x7fc0a406d9c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0c409d060 0x7fc0c409d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0c40b6a20 0x7fc0c40b6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 --2- 172.21.15.59:0/203262257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0c415e010 0x7fc0c4154280 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 -- 172.21.15.59:0/203262257 >> 172.21.15.59:0/203262257 conn(0x7fc0c409a9f0 msgr2=0x7fc0c40702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 -- 172.21.15.59:0/203262257 shutdown_connections 2024-03-06T07:30:29.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.142+0000 7fc0cb13e640 1 -- 172.21.15.59:0/203262257 wait complete. 2024-03-06T07:30:29.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:29.156 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:30:29.156 DEBUG:teuthology.orchestra.run.smithi059:> mkdir dir_1 dir_2 2024-03-06T07:30:29.156 DEBUG:teuthology.orchestra.run.smithi059:> touch dir_1/original 2024-03-06T07:30:29.156 DEBUG:teuthology.orchestra.run.smithi059:> ln dir_1/original dir_2/linkto 2024-03-06T07:30:29.156 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:30:29.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:29.188 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 dir_1) 2024-03-06T07:30:29.245 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:29.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.384+0000 7efdd20c3640 1 Processor -- start 2024-03-06T07:30:29.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.384+0000 7efdd20c3640 1 -- start start 2024-03-06T07:30:29.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdd20c3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efdcc0b4170 0x7efdcc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdd20c3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efdcc0d0a80 0x7efdcc0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdd20c3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdcc0d25d0 con 0x7efdcc0d4760 2024-03-06T07:30:29.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdd20c3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdcc0d4bf0 con 0x7efdcc0b4170 2024-03-06T07:30:29.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdd20c3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdcc0ce9e0 con 0x7efdcc0d0a80 2024-03-06T07:30:29.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdcb7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efdcc0b4170 0x7efdcc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdcaffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7efdcc0d4760 0x7efdcc0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51164/0 (socket says 172.21.15.59:51164) 2024-03-06T07:30:29.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdcaffd640 1 -- 172.21.15.59:0/2112687949 learned_addr learned my addr 172.21.15.59:0/2112687949 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:29.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdcbfff640 1 --2- 172.21.15.59:0/2112687949 >> v2:172.21.15.164:3300/0 conn(0x7efdcc0d0a80 0x7efdcc0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdcb7fe640 1 -- 172.21.15.59:0/2112687949 >> v2:172.21.15.164:3300/0 conn(0x7efdcc0d0a80 msgr2=0x7efdcc0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdcb7fe640 1 --2- 172.21.15.59:0/2112687949 >> v2:172.21.15.164:3300/0 conn(0x7efdcc0d0a80 0x7efdcc0cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdcb7fe640 1 -- 172.21.15.59:0/2112687949 >> v1:172.21.15.59:6789/0 conn(0x7efdcc0d4760 legacy=0x7efdcc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdcb7fe640 1 -- 172.21.15.59:0/2112687949 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc0d2870 con 0x7efdcc0b4170 2024-03-06T07:30:29.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.385+0000 7efdca7fc640 1 -- 172.21.15.59:0/2112687949 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128814547 0 0) 0x7efdcc0d25d0 con 0x7efdcc0d4760 2024-03-06T07:30:29.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.386+0000 7efdcb7fe640 1 --2- 172.21.15.59:0/2112687949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0b4170 0x7efdcc0b6a90 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7efdcc08df00 tx=0x7efdbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7bcdd07c1b5c5f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.386+0000 7efdca7fc640 1 -- 172.21.15.59:0/2112687949 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc035020 con 0x7efdcc0b4170 2024-03-06T07:30:29.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.386+0000 7efdca7fc640 1 -- 172.21.15.59:0/2112687949 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdbc031830 con 0x7efdcc0b4170 2024-03-06T07:30:29.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.386+0000 7efdca7fc640 1 -- 172.21.15.59:0/2112687949 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdbc031c00 con 0x7efdcc0b4170 2024-03-06T07:30:29.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.387+0000 7efdd20c3640 1 -- 172.21.15.59:0/2112687949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0b4170 msgr2=0x7efdcc0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.387+0000 7efdd20c3640 1 --2- 172.21.15.59:0/2112687949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0b4170 0x7efdcc0b6a90 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7efdcc08df00 tx=0x7efdbc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.387+0000 7efdd20c3640 1 -- 172.21.15.59:0/2112687949 shutdown_connections 2024-03-06T07:30:29.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.387+0000 7efdd20c3640 1 --2- 172.21.15.59:0/2112687949 >> v2:172.21.15.164:3300/0 conn(0x7efdcc0d0a80 0x7efdcc0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.387+0000 7efdd20c3640 1 --2- 172.21.15.59:0/2112687949 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0b4170 0x7efdcc0b6a90 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.387+0000 7efdd20c3640 1 -- 172.21.15.59:0/2112687949 >> 172.21.15.59:0/2112687949 conn(0x7efdcc0d3c40 msgr2=0x7efdcc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.387+0000 7efdd20c3640 1 -- 172.21.15.59:0/2112687949 shutdown_connections 2024-03-06T07:30:29.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.387+0000 7efdd20c3640 1 -- 172.21.15.59:0/2112687949 wait complete. 2024-03-06T07:30:29.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.388+0000 7efdd20c3640 1 Processor -- start 2024-03-06T07:30:29.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.388+0000 7efdd20c3640 1 -- start start 2024-03-06T07:30:29.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.389+0000 7efdd20c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc0b4170 0x7efdcc0ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdd20c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0d0a80 0x7efdcc1f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdd20c3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc1f41a0 0x7efdcc1584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdd20c3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0b4dd0 con 0x7efdcc0d0a80 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdd20c3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0b4550 con 0x7efdcc1f41a0 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdd20c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0b4f50 con 0x7efdcc0b4170 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0d0a80 0x7efdcc1f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc0b4170 0x7efdcc0ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0d0a80 0x7efdcc1f3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51050/0 (socket says 172.21.15.59:51050) 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcaffd640 1 -- 172.21.15.59:0/145232383 learned_addr learned my addr 172.21.15.59:0/145232383 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcbfff640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc1f41a0 0x7efdcc1584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcaffd640 1 -- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc0b4170 msgr2=0x7efdcc0ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcaffd640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc0b4170 0x7efdcc0ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcaffd640 1 -- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc1f41a0 msgr2=0x7efdcc1584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcaffd640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc1f41a0 0x7efdcc1584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcaffd640 1 -- 172.21.15.59:0/145232383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc1f4be0 con 0x7efdcc0d0a80 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcb7fe640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc0b4170 0x7efdcc0ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:29.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdcaffd640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0d0a80 0x7efdcc1f3c60 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c430 tx=0x7efdc000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdb3fff640 1 -- 172.21.15.59:0/145232383 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc00074e0 con 0x7efdcc0d0a80 2024-03-06T07:30:29.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdb3fff640 1 -- 172.21.15.59:0/145232383 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc00047e0 con 0x7efdcc0d0a80 2024-03-06T07:30:29.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdb3fff640 1 -- 172.21.15.59:0/145232383 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0004bd0 con 0x7efdcc0d0a80 2024-03-06T07:30:29.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdd20c3640 1 -- 172.21.15.59:0/145232383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdcc1f4620 con 0x7efdcc0d0a80 2024-03-06T07:30:29.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.390+0000 7efdd20c3640 1 -- 172.21.15.59:0/145232383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdcc0b1d20 con 0x7efdcc0d0a80 2024-03-06T07:30:29.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.391+0000 7efdd20c3640 1 -- 172.21.15.59:0/145232383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdcc0d1490 con 0x7efdcc0d0a80 2024-03-06T07:30:29.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.392+0000 7efdb3fff640 1 -- 172.21.15.59:0/145232383 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdc0011070 con 0x7efdcc0d0a80 2024-03-06T07:30:29.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.392+0000 7efdb3fff640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efdb806b4d0 0x7efdb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.393+0000 7efdb3fff640 1 -- 172.21.15.59:0/145232383 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efdc0089c00 con 0x7efdcc0d0a80 2024-03-06T07:30:29.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.393+0000 7efdcb7fe640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efdb806b4d0 0x7efdb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.393+0000 7efdcb7fe640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efdb806b4d0 0x7efdb806d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7efdbc036000 tx=0x7efdbc008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.397+0000 7efdb3fff640 1 -- 172.21.15.59:0/145232383 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdcc0d1490 con 0x7efdcc0d0a80 2024-03-06T07:30:29.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.617+0000 7efdd20c3640 1 -- 172.21.15.59:0/145232383 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdcc04f330 con 0x7efdcc0d0a80 2024-03-06T07:30:29.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.617+0000 7efdb3fff640 1 -- 172.21.15.59:0/145232383 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+3418 (secure 0 0 0) 0x7efdc0090080 con 0x7efdcc0d0a80 2024-03-06T07:30:29.619 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:29.619 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:22.126145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[8236,9336]},"id":10}]} 2024-03-06T07:30:29.620 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 102 2024-03-06T07:30:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 -- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efdb806b4d0 msgr2=0x7efdb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efdb806b4d0 0x7efdb806d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7efdbc036000 tx=0x7efdbc008030 comp rx=0 tx=0).stop 2024-03-06T07:30:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 -- 172.21.15.59:0/145232383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0d0a80 msgr2=0x7efdcc1f3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0d0a80 0x7efdcc1f3c60 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7efdc000c430 tx=0x7efdc000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 -- 172.21.15.59:0/145232383 shutdown_connections 2024-03-06T07:30:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efdb806b4d0 0x7efdb806d990 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc1f41a0 0x7efdcc1584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efdcc0d0a80 0x7efdcc1f3c60 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 --2- 172.21.15.59:0/145232383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc0b4170 0x7efdcc0ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 -- 172.21.15.59:0/145232383 >> 172.21.15.59:0/145232383 conn(0x7efdcc0d3c40 msgr2=0x7efdcc0b7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.622 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 -- 172.21.15.59:0/145232383 shutdown_connections 2024-03-06T07:30:29.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:29.620+0000 7efdd20c3640 1 -- 172.21.15.59:0/145232383 wait complete. 2024-03-06T07:30:29.636 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-06T07:30:29.753 INFO:teuthology.orchestra.run.smithi164.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:29.753 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:29.753 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:30:31.754 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:31.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.858+0000 7f851609d640 1 Processor -- start 2024-03-06T07:30:31.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.858+0000 7f851609d640 1 -- start start 2024-03-06T07:30:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.859+0000 7f851609d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f85100d0a80 0x7f85100b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.859+0000 7f851609d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85100d25d0 con 0x7f85100b4170 2024-03-06T07:30:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.859+0000 7f851609d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85100d2870 con 0x7f85100d4760 2024-03-06T07:30:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.859+0000 7f851609d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85100d4bf0 con 0x7f85100d0a80 2024-03-06T07:30:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.859+0000 7f850ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f85100d0a80 0x7f85100b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.859+0000 7f850f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f85100b4170 0x7f85100d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51172/0 (socket says 172.21.15.59:51172) 2024-03-06T07:30:31.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.859+0000 7f850f7fe640 1 -- 172.21.15.59:0/2234181735 learned_addr learned my addr 172.21.15.59:0/2234181735 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.859+0000 7f850ffff640 1 -- 172.21.15.59:0/2234181735 >> v1:172.21.15.164:6789/0 conn(0x7f85100d4760 legacy=0x7f85100cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.859+0000 7f850ffff640 1 -- 172.21.15.59:0/2234181735 >> v1:172.21.15.59:6789/0 conn(0x7f85100b4170 legacy=0x7f85100d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.859+0000 7f850ffff640 1 -- 172.21.15.59:0/2234181735 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85100d2b50 con 0x7f85100d0a80 2024-03-06T07:30:31.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.860+0000 7f850ffff640 1 --2- 172.21.15.59:0/2234181735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85100d0a80 0x7f85100b6a90 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f8508009d90 tx=0x7f850802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52d02a2475e5f72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.860+0000 7f850e7fc640 1 -- 172.21.15.59:0/2234181735 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8508039070 con 0x7f85100d0a80 2024-03-06T07:30:31.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.860+0000 7f850e7fc640 1 -- 172.21.15.59:0/2234181735 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850802cd00 con 0x7f85100d0a80 2024-03-06T07:30:31.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.860+0000 7f850e7fc640 1 -- 172.21.15.59:0/2234181735 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8508034660 con 0x7f85100d0a80 2024-03-06T07:30:31.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.861+0000 7f851609d640 1 -- 172.21.15.59:0/2234181735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85100d0a80 msgr2=0x7f85100b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.861+0000 7f851609d640 1 --2- 172.21.15.59:0/2234181735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85100d0a80 0x7f85100b6a90 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f8508009d90 tx=0x7f850802bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:31.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.861+0000 7f851609d640 1 -- 172.21.15.59:0/2234181735 shutdown_connections 2024-03-06T07:30:31.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.861+0000 7f851609d640 1 --2- 172.21.15.59:0/2234181735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85100d0a80 0x7f85100b6a90 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.861+0000 7f851609d640 1 -- 172.21.15.59:0/2234181735 >> 172.21.15.59:0/2234181735 conn(0x7f85100d3c40 msgr2=0x7f85100d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.861+0000 7f851609d640 1 -- 172.21.15.59:0/2234181735 shutdown_connections 2024-03-06T07:30:31.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.861+0000 7f851609d640 1 -- 172.21.15.59:0/2234181735 wait complete. 2024-03-06T07:30:31.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.862+0000 7f851609d640 1 Processor -- start 2024-03-06T07:30:31.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.862+0000 7f851609d640 1 -- start start 2024-03-06T07:30:31.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f851609d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85100b4170 0x7f8510156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f851609d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85100d0a80 0x7f85100ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f851609d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85100cd4d0 0x7f85100c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f851609d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85100b4dd0 con 0x7f85100d0a80 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f851609d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85100ce9e0 con 0x7f85100cd4d0 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f851609d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85100b4f50 con 0x7f85100b4170 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f850effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85100d0a80 0x7f85100ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f850effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85100d0a80 0x7f85100ccf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51066/0 (socket says 172.21.15.59:51066) 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f850effd640 1 -- 172.21.15.59:0/870321014 learned_addr learned my addr 172.21.15.59:0/870321014 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f850f7fe640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85100b4170 0x7f8510156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f850ffff640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85100cd4d0 0x7f85100c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f850effd640 1 -- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85100b4170 msgr2=0x7f8510156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f850effd640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85100b4170 0x7f8510156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f850effd640 1 -- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85100cd4d0 msgr2=0x7f85100c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f850effd640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85100cd4d0 0x7f85100c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.863+0000 7f850effd640 1 -- 172.21.15.59:0/870321014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85100cb970 con 0x7f85100d0a80 2024-03-06T07:30:31.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.864+0000 7f850effd640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85100d0a80 0x7f85100ccf90 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f850000c440 tx=0x7f850000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.864+0000 7f84f7fff640 1 -- 172.21.15.59:0/870321014 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85000052d0 con 0x7f85100d0a80 2024-03-06T07:30:31.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.864+0000 7f84f7fff640 1 -- 172.21.15.59:0/870321014 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8500005470 con 0x7f85100d0a80 2024-03-06T07:30:31.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.864+0000 7f851609d640 1 -- 172.21.15.59:0/870321014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85100cbc50 con 0x7f85100d0a80 2024-03-06T07:30:31.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.864+0000 7f84f7fff640 1 -- 172.21.15.59:0/870321014 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500007260 con 0x7f85100d0a80 2024-03-06T07:30:31.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.864+0000 7f851609d640 1 -- 172.21.15.59:0/870321014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85100cc210 con 0x7f85100d0a80 2024-03-06T07:30:31.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.865+0000 7f851609d640 1 -- 172.21.15.59:0/870321014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85100cb700 con 0x7f85100d0a80 2024-03-06T07:30:31.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.865+0000 7f84f7fff640 1 -- 172.21.15.59:0/870321014 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85000074c0 con 0x7f85100d0a80 2024-03-06T07:30:31.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.866+0000 7f84f7fff640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84fc06b4d0 0x7f84fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.866+0000 7f84f7fff640 1 -- 172.21.15.59:0/870321014 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8500089510 con 0x7f85100d0a80 2024-03-06T07:30:31.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.866+0000 7f850f7fe640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84fc06b4d0 0x7f84fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.866+0000 7f850f7fe640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84fc06b4d0 0x7f84fc06d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f8504005880 tx=0x7f8504009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:31.870+0000 7f84f7fff640 1 -- 172.21.15.59:0/870321014 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8500057dd0 con 0x7f85100d0a80 2024-03-06T07:30:32.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.114+0000 7f851609d640 1 -- 172.21.15.59:0/870321014 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f851004f330 con 0x7f85100d0a80 2024-03-06T07:30:32.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.114+0000 7f84f7fff640 1 -- 172.21.15.59:0/870321014 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+3418 (secure 0 0 0) 0x7f850005be70 con 0x7f85100d0a80 2024-03-06T07:30:32.116 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:32.117 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:22.126145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[8236,9336]},"id":10}]} 2024-03-06T07:30:32.117 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 102 2024-03-06T07:30:32.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 -- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84fc06b4d0 msgr2=0x7f84fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84fc06b4d0 0x7f84fc06d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f8504005880 tx=0x7f8504009040 comp rx=0 tx=0).stop 2024-03-06T07:30:32.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 -- 172.21.15.59:0/870321014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85100d0a80 msgr2=0x7f85100ccf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85100d0a80 0x7f85100ccf90 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f850000c440 tx=0x7f850000c900 comp rx=0 tx=0).stop 2024-03-06T07:30:32.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 -- 172.21.15.59:0/870321014 shutdown_connections 2024-03-06T07:30:32.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f84fc06b4d0 0x7f84fc06d990 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85100cd4d0 0x7f85100c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85100d0a80 0x7f85100ccf90 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 --2- 172.21.15.59:0/870321014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85100b4170 0x7f8510156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 -- 172.21.15.59:0/870321014 >> 172.21.15.59:0/870321014 conn(0x7f85100d3c40 msgr2=0x7f85100cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 -- 172.21.15.59:0/870321014 shutdown_connections 2024-03-06T07:30:32.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:32.116+0000 7f851609d640 1 -- 172.21.15.59:0/870321014 wait complete. 2024-03-06T07:30:32.129 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-06T07:30:32.246 INFO:teuthology.orchestra.run.smithi164.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:32.246 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:32.246 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:30:34.247 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:34.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.347+0000 7fa92b5f7640 1 Processor -- start 2024-03-06T07:30:34.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.347+0000 7fa92b5f7640 1 -- start start 2024-03-06T07:30:34.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa92b5f7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa92415dfe0 0x7fa9240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa92b5f7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa9240d29c0 0x7fa9240b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa92b5f7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9240b3430 con 0x7fa9240cf830 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa92b5f7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9240b31e0 con 0x7fa9240d29c0 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa92b5f7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9240b3d60 con 0x7fa92415dfe0 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa929b94640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa9240d29c0 0x7fa9240b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa928b92640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa9240cf830 0x7fa9240d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35680/0 (socket says 172.21.15.59:35680) 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa928b92640 1 -- 172.21.15.59:0/1103501243 learned_addr learned my addr 172.21.15.59:0/1103501243 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa929393640 1 --2- 172.21.15.59:0/1103501243 >> v2:172.21.15.164:3300/0 conn(0x7fa92415dfe0 0x7fa9240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa929b94640 1 -- 172.21.15.59:0/1103501243 >> v2:172.21.15.164:3300/0 conn(0x7fa92415dfe0 msgr2=0x7fa9240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa929b94640 1 --2- 172.21.15.59:0/1103501243 >> v2:172.21.15.164:3300/0 conn(0x7fa92415dfe0 0x7fa9240cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa929b94640 1 -- 172.21.15.59:0/1103501243 >> v1:172.21.15.59:6789/0 conn(0x7fa9240cf830 legacy=0x7fa9240d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa929b94640 1 -- 172.21.15.59:0/1103501243 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9240b3710 con 0x7fa9240d29c0 2024-03-06T07:30:34.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.348+0000 7fa91bfff640 1 -- 172.21.15.59:0/1103501243 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671373543 0 0) 0x7fa9240b3430 con 0x7fa9240cf830 2024-03-06T07:30:34.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.349+0000 7fa929b94640 1 --2- 172.21.15.59:0/1103501243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9240d29c0 0x7fa9240b41f0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fa924067990 tx=0x7fa91c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cc05691fc0ae7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.349+0000 7fa91bfff640 1 -- 172.21.15.59:0/1103501243 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c039070 con 0x7fa9240d29c0 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.349+0000 7fa91bfff640 1 -- 172.21.15.59:0/1103501243 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c02cc80 con 0x7fa9240d29c0 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.349+0000 7fa91bfff640 1 -- 172.21.15.59:0/1103501243 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c034660 con 0x7fa9240d29c0 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.350+0000 7fa92b5f7640 1 -- 172.21.15.59:0/1103501243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9240d29c0 msgr2=0x7fa9240b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.350+0000 7fa92b5f7640 1 --2- 172.21.15.59:0/1103501243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9240d29c0 0x7fa9240b41f0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fa924067990 tx=0x7fa91c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.350+0000 7fa92b5f7640 1 -- 172.21.15.59:0/1103501243 shutdown_connections 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.350+0000 7fa92b5f7640 1 --2- 172.21.15.59:0/1103501243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9240d29c0 0x7fa9240b41f0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.350+0000 7fa92b5f7640 1 --2- 172.21.15.59:0/1103501243 >> v2:172.21.15.164:3300/0 conn(0x7fa92415dfe0 0x7fa9240cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.350+0000 7fa92b5f7640 1 -- 172.21.15.59:0/1103501243 >> 172.21.15.59:0/1103501243 conn(0x7fa9240d3cd0 msgr2=0x7fa9240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.350+0000 7fa92b5f7640 1 -- 172.21.15.59:0/1103501243 shutdown_connections 2024-03-06T07:30:34.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.350+0000 7fa92b5f7640 1 -- 172.21.15.59:0/1103501243 wait complete. 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.351+0000 7fa92b5f7640 1 Processor -- start 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.351+0000 7fa92b5f7640 1 -- start start 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa92b5f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9240cf830 0x7fa924160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa92b5f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9240d29c0 0x7fa9240ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa92b5f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa92415dfe0 0x7fa9241e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa92b5f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240d1df0 con 0x7fa9240d29c0 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa92b5f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240d1c70 con 0x7fa92415dfe0 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa92b5f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92415e3c0 con 0x7fa9240cf830 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa928b92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9240d29c0 0x7fa9240ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa928b92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9240d29c0 0x7fa9240ccc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57674/0 (socket says 172.21.15.59:57674) 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa928b92640 1 -- 172.21.15.59:0/3181729047 learned_addr learned my addr 172.21.15.59:0/3181729047 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:34.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa929393640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9240cf830 0x7fa924160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa929b94640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa92415dfe0 0x7fa9241e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa928b92640 1 -- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9240cf830 msgr2=0x7fa924160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa928b92640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9240cf830 0x7fa924160d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.352+0000 7fa928b92640 1 -- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa92415dfe0 msgr2=0x7fa9241e8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.353+0000 7fa928b92640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa92415dfe0 0x7fa9241e8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.353+0000 7fa928b92640 1 -- 172.21.15.59:0/3181729047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa924160830 con 0x7fa9240d29c0 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.353+0000 7fa928b92640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9240d29c0 0x7fa9240ccc10 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fa91400c4b0 tx=0x7fa91400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.353+0000 7fa919ffb640 1 -- 172.21.15.59:0/3181729047 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914018070 con 0x7fa9240d29c0 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.353+0000 7fa919ffb640 1 -- 172.21.15.59:0/3181729047 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa914007500 con 0x7fa9240d29c0 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.353+0000 7fa919ffb640 1 -- 172.21.15.59:0/3181729047 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914004a20 con 0x7fa9240d29c0 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.353+0000 7fa92b5f7640 1 -- 172.21.15.59:0/3181729047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa924161300 con 0x7fa9240d29c0 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.353+0000 7fa92b5f7640 1 -- 172.21.15.59:0/3181729047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9241f76c0 con 0x7fa9240d29c0 2024-03-06T07:30:34.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.354+0000 7fa92b5f7640 1 -- 172.21.15.59:0/3181729047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92404f330 con 0x7fa9240d29c0 2024-03-06T07:30:34.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.355+0000 7fa919ffb640 1 -- 172.21.15.59:0/3181729047 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa914004bc0 con 0x7fa9240d29c0 2024-03-06T07:30:34.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.355+0000 7fa919ffb640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa90806b4d0 0x7fa90806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.355+0000 7fa919ffb640 1 -- 172.21.15.59:0/3181729047 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa914089390 con 0x7fa9240d29c0 2024-03-06T07:30:34.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.355+0000 7fa929393640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa90806b4d0 0x7fa90806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.356+0000 7fa929393640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa90806b4d0 0x7fa90806d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa920009aa0 tx=0x7fa920009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.365 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.364+0000 7fa919ffb640 1 -- 172.21.15.59:0/3181729047 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa914057c80 con 0x7fa9240d29c0 2024-03-06T07:30:34.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.586+0000 7fa92b5f7640 1 -- 172.21.15.59:0/3181729047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9240b3390 con 0x7fa9240d29c0 2024-03-06T07:30:34.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.587+0000 7fa919ffb640 1 -- 172.21.15.59:0/3181729047 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+3418 (secure 0 0 0) 0x7fa91405bd20 con 0x7fa9240d29c0 2024-03-06T07:30:34.589 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:34.589 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:22.126145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[8236,9336]},"id":10}]} 2024-03-06T07:30:34.589 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 102 2024-03-06T07:30:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 -- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa90806b4d0 msgr2=0x7fa90806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa90806b4d0 0x7fa90806d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa920009aa0 tx=0x7fa920009190 comp rx=0 tx=0).stop 2024-03-06T07:30:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 -- 172.21.15.59:0/3181729047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9240d29c0 msgr2=0x7fa9240ccc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9240d29c0 0x7fa9240ccc10 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fa91400c4b0 tx=0x7fa91400c970 comp rx=0 tx=0).stop 2024-03-06T07:30:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 -- 172.21.15.59:0/3181729047 shutdown_connections 2024-03-06T07:30:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa90806b4d0 0x7fa90806d990 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa92415dfe0 0x7fa9241e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9240d29c0 0x7fa9240ccc10 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 --2- 172.21.15.59:0/3181729047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9240cf830 0x7fa924160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 -- 172.21.15.59:0/3181729047 >> 172.21.15.59:0/3181729047 conn(0x7fa9240d3cd0 msgr2=0x7fa9240d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 -- 172.21.15.59:0/3181729047 shutdown_connections 2024-03-06T07:30:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:34.589+0000 7fa92b5f7640 1 -- 172.21.15.59:0/3181729047 wait complete. 2024-03-06T07:30:34.602 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-06T07:30:34.718 INFO:teuthology.orchestra.run.smithi164.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:34.718 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:34.719 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:30:36.720 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:36.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.819+0000 7fb7e9b0c640 1 Processor -- start 2024-03-06T07:30:36.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.819+0000 7fb7e9b0c640 1 -- start start 2024-03-06T07:30:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e9b0c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb7e40cfff0 0x7fb7e40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e9b0c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cc8b0 0x7fb7e40ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e9b0c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e4161980 con 0x7fb7e415bb20 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e9b0c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e40d2500 con 0x7fb7e40cfff0 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e9b0c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e4161c60 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cc8b0 0x7fb7e40ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e2ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb7e40cfff0 0x7fb7e40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e2ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb7e40cfff0 0x7fb7e40d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:36216/0 (socket says 172.21.15.59:36216) 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e37fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb7e415bb20 0x7fb7e40d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:45048/0 (socket says 172.21.15.59:45048) 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cc8b0 0x7fb7e40ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:39436/0 (socket says 172.21.15.59:39436) 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e2ffd640 1 -- 172.21.15.59:0/1458340531 learned_addr learned my addr 172.21.15.59:0/1458340531 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e3fff640 1 -- 172.21.15.59:0/1458340531 >> v2:172.21.15.164:3300/0 conn(0x7fb7e40cfff0 msgr2=0x7fb7e40d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e3fff640 1 --2- 172.21.15.59:0/1458340531 >> v2:172.21.15.164:3300/0 conn(0x7fb7e40cfff0 0x7fb7e40d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e3fff640 1 -- 172.21.15.59:0/1458340531 >> v1:172.21.15.164:6789/0 conn(0x7fb7e415bb20 legacy=0x7fb7e40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e3fff640 1 -- 172.21.15.59:0/1458340531 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e4161f40 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e27fc640 1 -- 172.21.15.59:0/1458340531 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045647613 0 0) 0x7fb7e4161980 con 0x7fb7e415bb20 2024-03-06T07:30:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.821+0000 7fb7e2ffd640 1 --2- 172.21.15.59:0/1458340531 >> v2:172.21.15.164:3300/0 conn(0x7fb7e40cfff0 0x7fb7e40d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.822+0000 7fb7e3fff640 1 --2- 172.21.15.59:0/1458340531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cc8b0 0x7fb7e40ccd20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009d90 tx=0x7fb7dc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba31dd6656b83f4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.822+0000 7fb7e27fc640 1 -- 172.21.15.59:0/1458340531 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc039070 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.822+0000 7fb7e27fc640 1 -- 172.21.15.59:0/1458340531 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc02cc50 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.822+0000 7fb7e27fc640 1 -- 172.21.15.59:0/1458340531 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc034440 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.823+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1458340531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cc8b0 msgr2=0x7fb7e40ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.823+0000 7fb7e9b0c640 1 --2- 172.21.15.59:0/1458340531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cc8b0 0x7fb7e40ccd20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009d90 tx=0x7fb7dc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.823+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1458340531 shutdown_connections 2024-03-06T07:30:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.823+0000 7fb7e9b0c640 1 --2- 172.21.15.59:0/1458340531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cc8b0 0x7fb7e40ccd20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.823+0000 7fb7e9b0c640 1 --2- 172.21.15.59:0/1458340531 >> v2:172.21.15.164:3300/0 conn(0x7fb7e40cfff0 0x7fb7e40d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.823+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1458340531 >> 172.21.15.59:0/1458340531 conn(0x7fb7e409a9f0 msgr2=0x7fb7e409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.823+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1458340531 shutdown_connections 2024-03-06T07:30:36.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.823+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1458340531 wait complete. 2024-03-06T07:30:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e9b0c640 1 Processor -- start 2024-03-06T07:30:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e9b0c640 1 -- start start 2024-03-06T07:30:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e9b0c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e40cc8b0 0x7fb7e4157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e9b0c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cfff0 0x7fb7e406be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e9b0c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7e415bb20 0x7fb7e415f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.829 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e9b0c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e40d1980 con 0x7fb7e415bb20 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e9b0c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e4162500 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e9b0c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e40d1b00 con 0x7fb7e40cfff0 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7e415bb20 0x7fb7e415f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e40cc8b0 0x7fb7e4157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7e415bb20 0x7fb7e415f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57686/0 (socket says 172.21.15.59:57686) 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cfff0 0x7fb7e406be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e3fff640 1 -- 172.21.15.59:0/1040825849 learned_addr learned my addr 172.21.15.59:0/1040825849 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e37fe640 1 -- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cfff0 msgr2=0x7fb7e406be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e37fe640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cfff0 0x7fb7e406be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e37fe640 1 -- 172.21.15.59:0/1040825849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7e415bb20 msgr2=0x7fb7e415f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e37fe640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7e415bb20 0x7fb7e415f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.825+0000 7fb7e37fe640 1 -- 172.21.15.59:0/1040825849 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e406c8b0 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.826+0000 7fb7e2ffd640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cfff0 0x7fb7e406be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.826+0000 7fb7e3fff640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7e415bb20 0x7fb7e415f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.826+0000 7fb7e37fe640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e40cc8b0 0x7fb7e4157890 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400c410 tx=0x7fb7d400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.826+0000 7fb7cbfff640 1 -- 172.21.15.59:0/1040825849 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d40074e0 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.826+0000 7fb7cbfff640 1 -- 172.21.15.59:0/1040825849 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d40047e0 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.826+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1040825849 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e406cb90 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.826+0000 7fb7cbfff640 1 -- 172.21.15.59:0/1040825849 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4004bd0 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.826+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1040825849 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e41580d0 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.827+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1040825849 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e404f330 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.829+0000 7fb7cbfff640 1 -- 172.21.15.59:0/1040825849 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7d4011070 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.829+0000 7fb7cbfff640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb7d006b500 0x7fb7d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.829+0000 7fb7cbfff640 1 -- 172.21.15.59:0/1040825849 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7d4089c20 con 0x7fb7e40cc8b0 2024-03-06T07:30:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.829+0000 7fb7e2ffd640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb7d006b500 0x7fb7d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.830+0000 7fb7e2ffd640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb7d006b500 0x7fb7d006d9c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009630 tx=0x7fb7d8009240 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:36.834+0000 7fb7cbfff640 1 -- 172.21.15.59:0/1040825849 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7d4058510 con 0x7fb7e40cc8b0 2024-03-06T07:30:37.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.057+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1040825849 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7e406f880 con 0x7fb7e40cc8b0 2024-03-06T07:30:37.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.057+0000 7fb7cbfff640 1 -- 172.21.15.59:0/1040825849 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+3418 (secure 0 0 0) 0x7fb7d405c5b0 con 0x7fb7e40cc8b0 2024-03-06T07:30:37.059 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:37.059 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":102,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:22.126145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[8236,9336]},"id":10}]} 2024-03-06T07:30:37.060 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 102 2024-03-06T07:30:37.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.059+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb7d006b500 msgr2=0x7fb7d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.059+0000 7fb7e9b0c640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb7d006b500 0x7fb7d006d9c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009630 tx=0x7fb7d8009240 comp rx=0 tx=0).stop 2024-03-06T07:30:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.059+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e40cc8b0 msgr2=0x7fb7e4157890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.059+0000 7fb7e9b0c640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e40cc8b0 0x7fb7e4157890 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400c410 tx=0x7fb7d400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.059+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1040825849 shutdown_connections 2024-03-06T07:30:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.059+0000 7fb7e9b0c640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb7d006b500 0x7fb7d006d9c0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.059+0000 7fb7e9b0c640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7e415bb20 0x7fb7e415f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.059+0000 7fb7e9b0c640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7e40cfff0 0x7fb7e406be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.059+0000 7fb7e9b0c640 1 --2- 172.21.15.59:0/1040825849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e40cc8b0 0x7fb7e4157890 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.059+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1040825849 >> 172.21.15.59:0/1040825849 conn(0x7fb7e409a9f0 msgr2=0x7fb7e40cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.060+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1040825849 shutdown_connections 2024-03-06T07:30:37.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:37.060+0000 7fb7e9b0c640 1 -- 172.21.15.59:0/1040825849 wait complete. 2024-03-06T07:30:37.076 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-06T07:30:37.195 INFO:teuthology.orchestra.run.smithi164.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:37.196 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:37.196 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-03-06T07:30:39.197 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:39.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.296+0000 7fc262ff6640 1 Processor -- start 2024-03-06T07:30:39.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.296+0000 7fc262ff6640 1 -- start start 2024-03-06T07:30:39.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.297+0000 7fc262ff6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc25c0d2710 0x7fc25c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.297+0000 7fc262ff6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25c0b3430 con 0x7fc25c0d22b0 2024-03-06T07:30:39.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.297+0000 7fc262ff6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25c0b3710 con 0x7fc25c15dfe0 2024-03-06T07:30:39.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.297+0000 7fc262ff6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc25c0b31e0 con 0x7fc25c0d2710 2024-03-06T07:30:39.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.297+0000 7fc25bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc25c0d22b0 0x7fc25c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35692/0 (socket says 172.21.15.59:35692) 2024-03-06T07:30:39.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.297+0000 7fc25bfff640 1 -- 172.21.15.59:0/1290270686 learned_addr learned my addr 172.21.15.59:0/1290270686 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:39.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.297+0000 7fc261593640 1 --2- 172.21.15.59:0/1290270686 >> v2:172.21.15.164:3300/0 conn(0x7fc25c0d2710 0x7fc25c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.298+0000 7fc25b7fe640 1 -- 172.21.15.59:0/1290270686 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471753015 0 0) 0x7fc25c0b3430 con 0x7fc25c0d22b0 2024-03-06T07:30:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.298+0000 7fc25b7fe640 1 -- 172.21.15.59:0/1290270686 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc24c003520 con 0x7fc25c0d22b0 2024-03-06T07:30:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.298+0000 7fc25b7fe640 1 -- 172.21.15.59:0/1290270686 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548858199 0 0) 0x7fc25c0b3710 con 0x7fc25c15dfe0 2024-03-06T07:30:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.298+0000 7fc25b7fe640 1 -- 172.21.15.59:0/1290270686 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc25c0b3430 con 0x7fc25c15dfe0 2024-03-06T07:30:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.298+0000 7fc261593640 1 -- 172.21.15.59:0/1290270686 >> v1:172.21.15.164:6789/0 conn(0x7fc25c15dfe0 legacy=0x7fc25c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.298+0000 7fc261593640 1 -- 172.21.15.59:0/1290270686 >> v1:172.21.15.59:6789/0 conn(0x7fc25c0d22b0 legacy=0x7fc25c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.298+0000 7fc261593640 1 -- 172.21.15.59:0/1290270686 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c0b39f0 con 0x7fc25c0d2710 2024-03-06T07:30:39.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.298+0000 7fc261593640 1 --2- 172.21.15.59:0/1290270686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c0d2710 0x7fc25c0b4900 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc254009dc0 tx=0x7fc25402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ae28a0b92211d3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.299+0000 7fc25b7fe640 1 -- 172.21.15.59:0/1290270686 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc254039070 con 0x7fc25c0d2710 2024-03-06T07:30:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.299+0000 7fc25b7fe640 1 -- 172.21.15.59:0/1290270686 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25402cce0 con 0x7fc25c0d2710 2024-03-06T07:30:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.299+0000 7fc25b7fe640 1 -- 172.21.15.59:0/1290270686 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc254034640 con 0x7fc25c0d2710 2024-03-06T07:30:39.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.299+0000 7fc262ff6640 1 -- 172.21.15.59:0/1290270686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c0d2710 msgr2=0x7fc25c0b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.299+0000 7fc262ff6640 1 --2- 172.21.15.59:0/1290270686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c0d2710 0x7fc25c0b4900 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc254009dc0 tx=0x7fc25402bf30 comp rx=0 tx=0).stop 2024-03-06T07:30:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.299+0000 7fc262ff6640 1 -- 172.21.15.59:0/1290270686 shutdown_connections 2024-03-06T07:30:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.299+0000 7fc262ff6640 1 --2- 172.21.15.59:0/1290270686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c0d2710 0x7fc25c0b4900 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.299+0000 7fc262ff6640 1 -- 172.21.15.59:0/1290270686 >> 172.21.15.59:0/1290270686 conn(0x7fc25c0d3cd0 msgr2=0x7fc25c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.299+0000 7fc262ff6640 1 -- 172.21.15.59:0/1290270686 shutdown_connections 2024-03-06T07:30:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.299+0000 7fc262ff6640 1 -- 172.21.15.59:0/1290270686 wait complete. 2024-03-06T07:30:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.300+0000 7fc262ff6640 1 Processor -- start 2024-03-06T07:30:39.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.300+0000 7fc262ff6640 1 -- start start 2024-03-06T07:30:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.301+0000 7fc262ff6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c0d22b0 0x7fc25c0b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.301+0000 7fc262ff6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c0d2710 0x7fc25c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.301+0000 7fc262ff6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc25c15dfe0 0x7fc25c1f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.301+0000 7fc262ff6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c0cfa00 con 0x7fc25c0d22b0 2024-03-06T07:30:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.301+0000 7fc262ff6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c0b3d60 con 0x7fc25c0d2710 2024-03-06T07:30:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.301+0000 7fc262ff6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c0d1c70 con 0x7fc25c15dfe0 2024-03-06T07:30:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.301+0000 7fc260d92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c0d22b0 0x7fc25c0b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.301+0000 7fc260d92640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c0d22b0 0x7fc25c0b8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57700/0 (socket says 172.21.15.59:57700) 2024-03-06T07:30:39.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.301+0000 7fc260d92640 1 -- 172.21.15.59:0/635274482 learned_addr learned my addr 172.21.15.59:0/635274482 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.301+0000 7fc25bfff640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c0d2710 0x7fc25c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.301+0000 7fc261593640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc25c15dfe0 0x7fc25c1f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.302+0000 7fc260d92640 1 -- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc25c15dfe0 msgr2=0x7fc25c1f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.302+0000 7fc260d92640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc25c15dfe0 0x7fc25c1f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.302+0000 7fc260d92640 1 -- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c0d2710 msgr2=0x7fc25c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.302+0000 7fc260d92640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c0d2710 0x7fc25c0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.302+0000 7fc260d92640 1 -- 172.21.15.59:0/635274482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c0cb7f0 con 0x7fc25c0d22b0 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.302+0000 7fc260d92640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c0d22b0 0x7fc25c0b8220 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fc25000c430 tx=0x7fc25000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.302+0000 7fc2597fa640 1 -- 172.21.15.59:0/635274482 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250017070 con 0x7fc25c0d22b0 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.302+0000 7fc2597fa640 1 -- 172.21.15.59:0/635274482 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2500052f0 con 0x7fc25c0d22b0 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.302+0000 7fc262ff6640 1 -- 172.21.15.59:0/635274482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25c0cbad0 con 0x7fc25c0d22b0 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.302+0000 7fc2597fa640 1 -- 172.21.15.59:0/635274482 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250003830 con 0x7fc25c0d22b0 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.302+0000 7fc262ff6640 1 -- 172.21.15.59:0/635274482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25c0cc370 con 0x7fc25c0d22b0 2024-03-06T07:30:39.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.303+0000 7fc262ff6640 1 -- 172.21.15.59:0/635274482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25c04f330 con 0x7fc25c0d22b0 2024-03-06T07:30:39.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.304+0000 7fc2597fa640 1 -- 172.21.15.59:0/635274482 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc25001a070 con 0x7fc25c0d22b0 2024-03-06T07:30:39.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.304+0000 7fc2597fa640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc24006b4d0 0x7fc24006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.304+0000 7fc2597fa640 1 -- 172.21.15.59:0/635274482 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc250089530 con 0x7fc25c0d22b0 2024-03-06T07:30:39.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.304+0000 7fc25bfff640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc24006b4d0 0x7fc24006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.305+0000 7fc25bfff640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc24006b4d0 0x7fc24006d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc25c06cd40 tx=0x7fc248007d30 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.308+0000 7fc2597fa640 1 -- 172.21.15.59:0/635274482 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc250057d40 con 0x7fc25c0d22b0 2024-03-06T07:30:39.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.548+0000 7fc262ff6640 1 -- 172.21.15.59:0/635274482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc25c15d770 con 0x7fc25c0d22b0 2024-03-06T07:30:39.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.549+0000 7fc2597fa640 1 -- 172.21.15.59:0/635274482 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7fc25005bde0 con 0x7fc25c0d22b0 2024-03-06T07:30:39.551 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:39.551 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:39.075108+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336,8236]},"id":10}]} 2024-03-06T07:30:39.551 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 103 2024-03-06T07:30:39.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 -- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc24006b4d0 msgr2=0x7fc24006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc24006b4d0 0x7fc24006d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc25c06cd40 tx=0x7fc248007d30 comp rx=0 tx=0).stop 2024-03-06T07:30:39.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 -- 172.21.15.59:0/635274482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c0d22b0 msgr2=0x7fc25c0b8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c0d22b0 0x7fc25c0b8220 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fc25000c430 tx=0x7fc25000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 -- 172.21.15.59:0/635274482 shutdown_connections 2024-03-06T07:30:39.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc24006b4d0 0x7fc24006d990 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc25c15dfe0 0x7fc25c1f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c0d2710 0x7fc25c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 --2- 172.21.15.59:0/635274482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc25c0d22b0 0x7fc25c0b8220 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 -- 172.21.15.59:0/635274482 >> 172.21.15.59:0/635274482 conn(0x7fc25c0d3cd0 msgr2=0x7fc25c0b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 -- 172.21.15.59:0/635274482 shutdown_connections 2024-03-06T07:30:39.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.551+0000 7fc262ff6640 1 -- 172.21.15.59:0/635274482 wait complete. 2024-03-06T07:30:39.564 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-06T07:30:39.679 INFO:teuthology.orchestra.run.smithi164.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"16","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:30:39.680 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/dir_1", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:39.680 INFO:tasks.cephfs.cephfs_test_case:[('/dir_1', 1)] =?= [('/dir_1', 1)] 2024-03-06T07:30:39.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:39.680 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 dir_2) 2024-03-06T07:30:39.696 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:39.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.838+0000 7fda29525640 1 Processor -- start 2024-03-06T07:30:39.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.838+0000 7fda29525640 1 -- start start 2024-03-06T07:30:39.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda29525640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fda24166af0 0x7fda24168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda29525640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda2415e380 con 0x7fda240cf850 2024-03-06T07:30:39.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda29525640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda240cdc10 con 0x7fda240d00c0 2024-03-06T07:30:39.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda29525640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2406ca90 con 0x7fda24166af0 2024-03-06T07:30:39.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda227fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fda240cf850 0x7fda240cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35694/0 (socket says 172.21.15.59:35694) 2024-03-06T07:30:39.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda227fc640 1 -- 172.21.15.59:0/276123645 learned_addr learned my addr 172.21.15.59:0/276123645 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:39.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda237fe640 1 --2- 172.21.15.59:0/276123645 >> v2:172.21.15.164:3300/0 conn(0x7fda24166af0 0x7fda24168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda21ffb640 1 -- 172.21.15.59:0/276123645 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721583698 0 0) 0x7fda2415e380 con 0x7fda240cf850 2024-03-06T07:30:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda21ffb640 1 -- 172.21.15.59:0/276123645 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda0c003520 con 0x7fda240cf850 2024-03-06T07:30:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda237fe640 1 -- 172.21.15.59:0/276123645 >> v1:172.21.15.164:6789/0 conn(0x7fda240d00c0 legacy=0x7fda240d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda21ffb640 1 -- 172.21.15.59:0/276123645 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144704505 0 0) 0x7fda240cdc10 con 0x7fda240d00c0 2024-03-06T07:30:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda237fe640 1 -- 172.21.15.59:0/276123645 >> v1:172.21.15.59:6789/0 conn(0x7fda240cf850 legacy=0x7fda240cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.839+0000 7fda237fe640 1 -- 172.21.15.59:0/276123645 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2406c160 con 0x7fda24166af0 2024-03-06T07:30:39.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.840+0000 7fda237fe640 1 --2- 172.21.15.59:0/276123645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda24166af0 0x7fda24168ee0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fda1c009d90 tx=0x7fda1c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac98dabc9c206e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.840+0000 7fda21ffb640 1 -- 172.21.15.59:0/276123645 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c039070 con 0x7fda24166af0 2024-03-06T07:30:39.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.840+0000 7fda21ffb640 1 -- 172.21.15.59:0/276123645 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c02cd00 con 0x7fda24166af0 2024-03-06T07:30:39.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.840+0000 7fda21ffb640 1 -- 172.21.15.59:0/276123645 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c034640 con 0x7fda24166af0 2024-03-06T07:30:39.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.841+0000 7fda29525640 1 -- 172.21.15.59:0/276123645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda24166af0 msgr2=0x7fda24168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.841+0000 7fda29525640 1 --2- 172.21.15.59:0/276123645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda24166af0 0x7fda24168ee0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fda1c009d90 tx=0x7fda1c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:30:39.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.841+0000 7fda29525640 1 -- 172.21.15.59:0/276123645 shutdown_connections 2024-03-06T07:30:39.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.841+0000 7fda29525640 1 --2- 172.21.15.59:0/276123645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda24166af0 0x7fda24168ee0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.841+0000 7fda29525640 1 -- 172.21.15.59:0/276123645 >> 172.21.15.59:0/276123645 conn(0x7fda240b35a0 msgr2=0x7fda240b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.841+0000 7fda29525640 1 -- 172.21.15.59:0/276123645 shutdown_connections 2024-03-06T07:30:39.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.841+0000 7fda29525640 1 -- 172.21.15.59:0/276123645 wait complete. 2024-03-06T07:30:39.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.842+0000 7fda29525640 1 Processor -- start 2024-03-06T07:30:39.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.842+0000 7fda29525640 1 -- start start 2024-03-06T07:30:39.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda29525640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda240cf850 0x7fda240c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda29525640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda240d00c0 0x7fda24157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda29525640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda24166af0 0x7fda24157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda29525640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2406c720 con 0x7fda240d00c0 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda29525640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2406df30 con 0x7fda24166af0 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda29525640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda240b1030 con 0x7fda240cf850 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda227fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda240d00c0 0x7fda24157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda227fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda240d00c0 0x7fda24157920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57706/0 (socket says 172.21.15.59:57706) 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda227fc640 1 -- 172.21.15.59:0/455824973 learned_addr learned my addr 172.21.15.59:0/455824973 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda22ffd640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda240cf850 0x7fda240c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda237fe640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda24166af0 0x7fda24157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda227fc640 1 -- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda240cf850 msgr2=0x7fda240c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda227fc640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda240cf850 0x7fda240c7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda227fc640 1 -- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda24166af0 msgr2=0x7fda24157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda227fc640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda24166af0 0x7fda24157e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.843+0000 7fda227fc640 1 -- 172.21.15.59:0/455824973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda240c6050 con 0x7fda240d00c0 2024-03-06T07:30:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.844+0000 7fda227fc640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda240d00c0 0x7fda24157920 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fda1800c3c0 tx=0x7fda1800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.844+0000 7fda0b7fe640 1 -- 172.21.15.59:0/455824973 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18017070 con 0x7fda240d00c0 2024-03-06T07:30:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.844+0000 7fda0b7fe640 1 -- 172.21.15.59:0/455824973 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda18005ce0 con 0x7fda240d00c0 2024-03-06T07:30:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.844+0000 7fda29525640 1 -- 172.21.15.59:0/455824973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda240c62c0 con 0x7fda240d00c0 2024-03-06T07:30:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.844+0000 7fda0b7fe640 1 -- 172.21.15.59:0/455824973 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda180072f0 con 0x7fda240d00c0 2024-03-06T07:30:39.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.844+0000 7fda29525640 1 -- 172.21.15.59:0/455824973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda240c6880 con 0x7fda240d00c0 2024-03-06T07:30:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.845+0000 7fda29525640 1 -- 172.21.15.59:0/455824973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda240c6d70 con 0x7fda240d00c0 2024-03-06T07:30:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.845+0000 7fda0b7fe640 1 -- 172.21.15.59:0/455824973 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda1801a070 con 0x7fda240d00c0 2024-03-06T07:30:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.845+0000 7fda0b7fe640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1006b4d0 0x7fda1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.846+0000 7fda0b7fe640 1 -- 172.21.15.59:0/455824973 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fda18089520 con 0x7fda240d00c0 2024-03-06T07:30:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.846+0000 7fda22ffd640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1006b4d0 0x7fda1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.846+0000 7fda22ffd640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1006b4d0 0x7fda1006d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fda14000f60 tx=0x7fda14009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.856 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:39.854+0000 7fda0b7fe640 1 -- 172.21.15.59:0/455824973 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda18057d30 con 0x7fda240d00c0 2024-03-06T07:30:40.172 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.171+0000 7fda29525640 1 -- 172.21.15.59:0/455824973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda2404f330 con 0x7fda240d00c0 2024-03-06T07:30:40.173 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.171+0000 7fda0b7fe640 1 -- 172.21.15.59:0/455824973 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7fda1805bdd0 con 0x7fda240d00c0 2024-03-06T07:30:40.173 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:40.174 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:39.075108+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336,8236]},"id":10}]} 2024-03-06T07:30:40.174 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 103 2024-03-06T07:30:40.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 -- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1006b4d0 msgr2=0x7fda1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1006b4d0 0x7fda1006d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fda14000f60 tx=0x7fda14009040 comp rx=0 tx=0).stop 2024-03-06T07:30:40.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 -- 172.21.15.59:0/455824973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda240d00c0 msgr2=0x7fda24157920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda240d00c0 0x7fda24157920 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fda1800c3c0 tx=0x7fda1800c880 comp rx=0 tx=0).stop 2024-03-06T07:30:40.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 -- 172.21.15.59:0/455824973 shutdown_connections 2024-03-06T07:30:40.176 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fda1006b4d0 0x7fda1006d990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda24166af0 0x7fda24157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda240d00c0 0x7fda24157920 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 --2- 172.21.15.59:0/455824973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda240cf850 0x7fda240c7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 -- 172.21.15.59:0/455824973 >> 172.21.15.59:0/455824973 conn(0x7fda240b35a0 msgr2=0x7fda2415d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 -- 172.21.15.59:0/455824973 shutdown_connections 2024-03-06T07:30:40.177 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:40.174+0000 7fda29525640 1 -- 172.21.15.59:0/455824973 wait complete. 2024-03-06T07:30:40.187 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:40.303 INFO:teuthology.orchestra.run.smithi059.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:30:40.303 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:30:40.303 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:30:42.304 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:42.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.405+0000 7f61eada0640 1 Processor -- start 2024-03-06T07:30:42.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.405+0000 7f61eada0640 1 -- start start 2024-03-06T07:30:42.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.405+0000 7f61eada0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f61e40d22b0 0x7f61e40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.406+0000 7f61eada0640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e40b3430 con 0x7f61e40b41f0 2024-03-06T07:30:42.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.406+0000 7f61eada0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e40b3710 con 0x7f61e415dfe0 2024-03-06T07:30:42.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.406+0000 7f61eada0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e40d2c50 con 0x7f61e40d22b0 2024-03-06T07:30:42.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.406+0000 7f61e3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f61e40d22b0 0x7f61e40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.406+0000 7f61e933d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f61e40b41f0 0x7f61e40b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35700/0 (socket says 172.21.15.59:35700) 2024-03-06T07:30:42.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.406+0000 7f61e933d640 1 -- 172.21.15.59:0/2119435829 learned_addr learned my addr 172.21.15.59:0/2119435829 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:42.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.406+0000 7f61e3fff640 1 -- 172.21.15.59:0/2119435829 >> v1:172.21.15.164:6789/0 conn(0x7f61e415dfe0 legacy=0x7f61e40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.406+0000 7f61e3fff640 1 -- 172.21.15.59:0/2119435829 >> v1:172.21.15.59:6789/0 conn(0x7f61e40b41f0 legacy=0x7f61e40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.406+0000 7f61e3fff640 1 -- 172.21.15.59:0/2119435829 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e40b39f0 con 0x7f61e40d22b0 2024-03-06T07:30:42.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.406+0000 7f61e37fe640 1 -- 172.21.15.59:0/2119435829 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279846078 0 0) 0x7f61e40b3430 con 0x7f61e40b41f0 2024-03-06T07:30:42.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.407+0000 7f61e3fff640 1 --2- 172.21.15.59:0/2119435829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61e40d22b0 0x7f61e40d2710 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0097d0 tx=0x7f61cc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6447d3c9be4e0ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.407+0000 7f61e37fe640 1 -- 172.21.15.59:0/2119435829 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc034070 con 0x7f61e40d22b0 2024-03-06T07:30:42.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.407+0000 7f61e37fe640 1 -- 172.21.15.59:0/2119435829 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc030610 con 0x7f61e40d22b0 2024-03-06T07:30:42.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.407+0000 7f61e37fe640 1 -- 172.21.15.59:0/2119435829 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc0309e0 con 0x7f61e40d22b0 2024-03-06T07:30:42.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.407+0000 7f61eada0640 1 -- 172.21.15.59:0/2119435829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61e40d22b0 msgr2=0x7f61e40d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.407+0000 7f61eada0640 1 --2- 172.21.15.59:0/2119435829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61e40d22b0 0x7f61e40d2710 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0097d0 tx=0x7f61cc02b960 comp rx=0 tx=0).stop 2024-03-06T07:30:42.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.407+0000 7f61eada0640 1 -- 172.21.15.59:0/2119435829 shutdown_connections 2024-03-06T07:30:42.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.407+0000 7f61eada0640 1 --2- 172.21.15.59:0/2119435829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61e40d22b0 0x7f61e40d2710 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.407+0000 7f61eada0640 1 -- 172.21.15.59:0/2119435829 >> 172.21.15.59:0/2119435829 conn(0x7f61e40d3cd0 msgr2=0x7f61e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.407+0000 7f61eada0640 1 -- 172.21.15.59:0/2119435829 shutdown_connections 2024-03-06T07:30:42.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.407+0000 7f61eada0640 1 -- 172.21.15.59:0/2119435829 wait complete. 2024-03-06T07:30:42.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.409+0000 7f61eada0640 1 Processor -- start 2024-03-06T07:30:42.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.409+0000 7f61eada0640 1 -- start start 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61eada0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61e40b41f0 0x7f61e40b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61eada0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61e40d22b0 0x7f61e41510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61eada0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61e415dfe0 0x7f61e40b8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61eada0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40b3d60 con 0x7f61e40d22b0 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61eada0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40b31e0 con 0x7f61e40b41f0 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61eada0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40d1c70 con 0x7f61e415dfe0 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61e40d22b0 0x7f61e41510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61e40d22b0 0x7f61e41510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57732/0 (socket says 172.21.15.59:57732) 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e3fff640 1 -- 172.21.15.59:0/330723419 learned_addr learned my addr 172.21.15.59:0/330723419 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e8b3c640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61e40b41f0 0x7f61e40b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e933d640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61e415dfe0 0x7f61e40b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e3fff640 1 -- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61e415dfe0 msgr2=0x7f61e40b8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e3fff640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61e415dfe0 0x7f61e40b8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e3fff640 1 -- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61e40b41f0 msgr2=0x7f61e40b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e3fff640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61e40b41f0 0x7f61e40b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e3fff640 1 -- 172.21.15.59:0/330723419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e40b9380 con 0x7f61e40d22b0 2024-03-06T07:30:42.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e3fff640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61e40d22b0 0x7f61e41510b0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0098f0 tx=0x7f61cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e17fa640 1 -- 172.21.15.59:0/330723419 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc034040 con 0x7f61e40d22b0 2024-03-06T07:30:42.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e17fa640 1 -- 172.21.15.59:0/330723419 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc039a10 con 0x7f61e40d22b0 2024-03-06T07:30:42.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61e17fa640 1 -- 172.21.15.59:0/330723419 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc039de0 con 0x7f61e40d22b0 2024-03-06T07:30:42.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.411+0000 7f61eada0640 1 -- 172.21.15.59:0/330723419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e40b9660 con 0x7f61e40d22b0 2024-03-06T07:30:42.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.412+0000 7f61eada0640 1 -- 172.21.15.59:0/330723419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e40b6e10 con 0x7f61e40d22b0 2024-03-06T07:30:42.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.413+0000 7f61eada0640 1 -- 172.21.15.59:0/330723419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e404f330 con 0x7f61e40d22b0 2024-03-06T07:30:42.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.413+0000 7f61e17fa640 1 -- 172.21.15.59:0/330723419 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61cc037070 con 0x7f61e40d22b0 2024-03-06T07:30:42.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.414+0000 7f61e17fa640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f61c406b4d0 0x7f61c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.414+0000 7f61e17fa640 1 -- 172.21.15.59:0/330723419 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f61cc0ac1b0 con 0x7f61e40d22b0 2024-03-06T07:30:42.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.414+0000 7f61e8b3c640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f61c406b4d0 0x7f61c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.414+0000 7f61e8b3c640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f61c406b4d0 0x7f61c406d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f61dc0012e0 tx=0x7f61dc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.418+0000 7f61e17fa640 1 -- 172.21.15.59:0/330723419 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61cc07a9c0 con 0x7f61e40d22b0 2024-03-06T07:30:42.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.645+0000 7f61eada0640 1 -- 172.21.15.59:0/330723419 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61e40d0920 con 0x7f61e40d22b0 2024-03-06T07:30:42.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.645+0000 7f61e17fa640 1 -- 172.21.15.59:0/330723419 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7f61cc07ea60 con 0x7f61e40d22b0 2024-03-06T07:30:42.647 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:42.648 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:39.075108+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336,8236]},"id":10}]} 2024-03-06T07:30:42.648 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 103 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.647+0000 7f61eada0640 1 -- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f61c406b4d0 msgr2=0x7f61c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.647+0000 7f61eada0640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f61c406b4d0 0x7f61c406d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f61dc0012e0 tx=0x7f61dc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.647+0000 7f61eada0640 1 -- 172.21.15.59:0/330723419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61e40d22b0 msgr2=0x7f61e41510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.647+0000 7f61eada0640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61e40d22b0 0x7f61e41510b0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0098f0 tx=0x7f61cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.648+0000 7f61eada0640 1 -- 172.21.15.59:0/330723419 shutdown_connections 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.648+0000 7f61eada0640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f61c406b4d0 0x7f61c406d990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.648+0000 7f61eada0640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61e415dfe0 0x7f61e40b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.648+0000 7f61eada0640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61e40d22b0 0x7f61e41510b0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.648+0000 7f61eada0640 1 --2- 172.21.15.59:0/330723419 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61e40b41f0 0x7f61e40b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.648+0000 7f61eada0640 1 -- 172.21.15.59:0/330723419 >> 172.21.15.59:0/330723419 conn(0x7f61e40d3cd0 msgr2=0x7f61e40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.648+0000 7f61eada0640 1 -- 172.21.15.59:0/330723419 shutdown_connections 2024-03-06T07:30:42.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:42.648+0000 7f61eada0640 1 -- 172.21.15.59:0/330723419 wait complete. 2024-03-06T07:30:42.661 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:42.777 INFO:teuthology.orchestra.run.smithi059.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:30:42.777 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:30:42.777 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:30:44.778 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:44.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.878+0000 7f675d579640 1 Processor -- start 2024-03-06T07:30:44.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.878+0000 7f675d579640 1 -- start start 2024-03-06T07:30:44.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.879+0000 7f675d579640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f67580d22b0 0x7f67580d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.879+0000 7f675d579640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67580b3430 con 0x7f675815dfe0 2024-03-06T07:30:44.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.879+0000 7f675d579640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67580b3710 con 0x7f67580b41f0 2024-03-06T07:30:44.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.879+0000 7f675d579640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67580d2c50 con 0x7f67580d22b0 2024-03-06T07:30:44.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.879+0000 7f6756ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f675815dfe0 0x7f67580cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49128/0 (socket says 172.21.15.59:49128) 2024-03-06T07:30:44.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.879+0000 7f6756ffd640 1 -- 172.21.15.59:0/900168198 learned_addr learned my addr 172.21.15.59:0/900168198 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:44.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.879+0000 7f674ffff640 1 --2- 172.21.15.59:0/900168198 >> v2:172.21.15.164:3300/0 conn(0x7f67580d22b0 0x7f67580d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.880+0000 7f67567fc640 1 -- 172.21.15.59:0/900168198 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496946033 0 0) 0x7f67580b3430 con 0x7f675815dfe0 2024-03-06T07:30:44.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.880+0000 7f67567fc640 1 -- 172.21.15.59:0/900168198 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6744003520 con 0x7f675815dfe0 2024-03-06T07:30:44.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.880+0000 7f674ffff640 1 -- 172.21.15.59:0/900168198 >> v1:172.21.15.164:6789/0 conn(0x7f67580b41f0 legacy=0x7f67580b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.880+0000 7f67567fc640 1 -- 172.21.15.59:0/900168198 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818022559 0 0) 0x7f67580b3710 con 0x7f67580b41f0 2024-03-06T07:30:44.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.880+0000 7f674ffff640 1 -- 172.21.15.59:0/900168198 >> v1:172.21.15.59:6789/0 conn(0x7f675815dfe0 legacy=0x7f67580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.880+0000 7f674ffff640 1 -- 172.21.15.59:0/900168198 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67580b39f0 con 0x7f67580d22b0 2024-03-06T07:30:44.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.881+0000 7f674ffff640 1 --2- 172.21.15.59:0/900168198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67580d22b0 0x7f67580d2710 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f67400097d0 tx=0x7f674002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c33f4a9353782041 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.881+0000 7f67567fc640 1 -- 172.21.15.59:0/900168198 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6740034070 con 0x7f67580d22b0 2024-03-06T07:30:44.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.881+0000 7f67567fc640 1 -- 172.21.15.59:0/900168198 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6740030610 con 0x7f67580d22b0 2024-03-06T07:30:44.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.881+0000 7f67567fc640 1 -- 172.21.15.59:0/900168198 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67400309c0 con 0x7f67580d22b0 2024-03-06T07:30:44.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.881+0000 7f675d579640 1 -- 172.21.15.59:0/900168198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67580d22b0 msgr2=0x7f67580d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.881+0000 7f675d579640 1 --2- 172.21.15.59:0/900168198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67580d22b0 0x7f67580d2710 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f67400097d0 tx=0x7f674002b960 comp rx=0 tx=0).stop 2024-03-06T07:30:44.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.881+0000 7f675d579640 1 -- 172.21.15.59:0/900168198 shutdown_connections 2024-03-06T07:30:44.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.881+0000 7f675d579640 1 --2- 172.21.15.59:0/900168198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67580d22b0 0x7f67580d2710 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.881+0000 7f675d579640 1 -- 172.21.15.59:0/900168198 >> 172.21.15.59:0/900168198 conn(0x7f67580d3cd0 msgr2=0x7f67580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.881+0000 7f675d579640 1 -- 172.21.15.59:0/900168198 shutdown_connections 2024-03-06T07:30:44.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.881+0000 7f675d579640 1 -- 172.21.15.59:0/900168198 wait complete. 2024-03-06T07:30:44.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f675d579640 1 Processor -- start 2024-03-06T07:30:44.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f675d579640 1 -- start start 2024-03-06T07:30:44.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f675d579640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67580b41f0 0x7f67581f7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f675d579640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67580d22b0 0x7f67581602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f675d579640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675815dfe0 0x7f67581f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f675d579640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67580b3d60 con 0x7f675815dfe0 2024-03-06T07:30:44.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f675d579640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67580b31e0 con 0x7f67580d22b0 2024-03-06T07:30:44.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f675d579640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67580d1c70 con 0x7f67580b41f0 2024-03-06T07:30:44.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f67577fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675815dfe0 0x7f67581f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f67577fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675815dfe0 0x7f67581f2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35614/0 (socket says 172.21.15.59:35614) 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f67577fe640 1 -- 172.21.15.59:0/3701100548 learned_addr learned my addr 172.21.15.59:0/3701100548 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.883+0000 7f6756ffd640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67580b41f0 0x7f67581f7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f674ffff640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67580d22b0 0x7f67581602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f67577fe640 1 -- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67580b41f0 msgr2=0x7f67581f7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f67577fe640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67580b41f0 0x7f67581f7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f67577fe640 1 -- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67580d22b0 msgr2=0x7f67581602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f67577fe640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67580d22b0 0x7f67581602d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f67577fe640 1 -- 172.21.15.59:0/3701100548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6758161660 con 0x7f675815dfe0 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f67577fe640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675815dfe0 0x7f67581f2db0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f675000c760 tx=0x7f675000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f674f7fe640 1 -- 172.21.15.59:0/3701100548 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6750003da0 con 0x7f675815dfe0 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f674f7fe640 1 -- 172.21.15.59:0/3701100548 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6750005070 con 0x7f675815dfe0 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f674f7fe640 1 -- 172.21.15.59:0/3701100548 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6750005460 con 0x7f675815dfe0 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f675d579640 1 -- 172.21.15.59:0/3701100548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6758160ae0 con 0x7f675815dfe0 2024-03-06T07:30:44.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.884+0000 7f675d579640 1 -- 172.21.15.59:0/3701100548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67581610a0 con 0x7f675815dfe0 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.885+0000 7f675d579640 1 -- 172.21.15.59:0/3701100548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67580d3770 con 0x7f675815dfe0 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.886+0000 7f674f7fe640 1 -- 172.21.15.59:0/3701100548 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6750005600 con 0x7f675815dfe0 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.886+0000 7f674f7fe640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f673806b4d0 0x7f673806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:44.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.886+0000 7f674f7fe640 1 -- 172.21.15.59:0/3701100548 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6750089f50 con 0x7f675815dfe0 2024-03-06T07:30:44.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.886+0000 7f6756ffd640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f673806b4d0 0x7f673806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:44.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.887+0000 7f6756ffd640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f673806b4d0 0x7f673806d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6748005880 tx=0x7f6748009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:44.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:44.890+0000 7f674f7fe640 1 -- 172.21.15.59:0/3701100548 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f67580d3770 con 0x7f675815dfe0 2024-03-06T07:30:45.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.114+0000 7f675d579640 1 -- 172.21.15.59:0/3701100548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f675804f330 con 0x7f675815dfe0 2024-03-06T07:30:45.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.115+0000 7f674f7fe640 1 -- 172.21.15.59:0/3701100548 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7f67500587f0 con 0x7f675815dfe0 2024-03-06T07:30:45.117 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:45.117 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:39.075108+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336,8236]},"id":10}]} 2024-03-06T07:30:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 103 2024-03-06T07:30:45.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 -- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f673806b4d0 msgr2=0x7f673806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f673806b4d0 0x7f673806d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6748005880 tx=0x7f6748009040 comp rx=0 tx=0).stop 2024-03-06T07:30:45.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 -- 172.21.15.59:0/3701100548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675815dfe0 msgr2=0x7f67581f2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675815dfe0 0x7f67581f2db0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f675000c760 tx=0x7f675000cc20 comp rx=0 tx=0).stop 2024-03-06T07:30:45.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 -- 172.21.15.59:0/3701100548 shutdown_connections 2024-03-06T07:30:45.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f673806b4d0 0x7f673806d990 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f675815dfe0 0x7f67581f2db0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67580d22b0 0x7f67581602d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 --2- 172.21.15.59:0/3701100548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f67580b41f0 0x7f67581f7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 -- 172.21.15.59:0/3701100548 >> 172.21.15.59:0/3701100548 conn(0x7f67580d3cd0 msgr2=0x7f67580d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 -- 172.21.15.59:0/3701100548 shutdown_connections 2024-03-06T07:30:45.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:45.117+0000 7f675d579640 1 -- 172.21.15.59:0/3701100548 wait complete. 2024-03-06T07:30:45.133 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:45.248 INFO:teuthology.orchestra.run.smithi059.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:30:45.249 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:30:45.249 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:30:47.250 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:47.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.350+0000 7f733929a640 1 Processor -- start 2024-03-06T07:30:47.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.350+0000 7f733929a640 1 -- start start 2024-03-06T07:30:47.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f733929a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f73340d00c0 0x7f73340d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f733929a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f73340cf850 0x7f73340cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f733929a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73340d09e0 con 0x7f73340ce1b0 2024-03-06T07:30:47.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f733929a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f733406ca90 con 0x7f73340d00c0 2024-03-06T07:30:47.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f733929a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f733406df30 con 0x7f73340cf850 2024-03-06T07:30:47.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f7332ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f73340d00c0 0x7f73340d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f7332ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f73340d00c0 0x7f73340d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35622/0 (socket says 172.21.15.59:35622) 2024-03-06T07:30:47.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f7332ffd640 1 -- 172.21.15.59:0/2137197354 learned_addr learned my addr 172.21.15.59:0/2137197354 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:47.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f732bfff640 1 --2- 172.21.15.59:0/2137197354 >> v2:172.21.15.164:3300/0 conn(0x7f73340cf850 0x7f73340cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f7332ffd640 1 -- 172.21.15.59:0/2137197354 >> v2:172.21.15.164:3300/0 conn(0x7f73340cf850 msgr2=0x7f73340cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f7332ffd640 1 --2- 172.21.15.59:0/2137197354 >> v2:172.21.15.164:3300/0 conn(0x7f73340cf850 0x7f73340cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f7332ffd640 1 -- 172.21.15.59:0/2137197354 >> v1:172.21.15.164:6789/0 conn(0x7f73340ce1b0 legacy=0x7f73340ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.351+0000 7f7332ffd640 1 -- 172.21.15.59:0/2137197354 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733415e380 con 0x7f73340d00c0 2024-03-06T07:30:47.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.352+0000 7f7332ffd640 1 --2- 172.21.15.59:0/2137197354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73340d00c0 0x7f73340d04a0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f7334071de0 tx=0x7f732402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de24dfe0153c0cc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.352+0000 7f73327fc640 1 -- 172.21.15.59:0/2137197354 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324034070 con 0x7f73340d00c0 2024-03-06T07:30:47.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.352+0000 7f73327fc640 1 -- 172.21.15.59:0/2137197354 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7324030860 con 0x7f73340d00c0 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.352+0000 7f73327fc640 1 -- 172.21.15.59:0/2137197354 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324030c30 con 0x7f73340d00c0 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.353+0000 7f733929a640 1 -- 172.21.15.59:0/2137197354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73340d00c0 msgr2=0x7f73340d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.353+0000 7f733929a640 1 --2- 172.21.15.59:0/2137197354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73340d00c0 0x7f73340d04a0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f7334071de0 tx=0x7f732402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.353+0000 7f733929a640 1 -- 172.21.15.59:0/2137197354 shutdown_connections 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.353+0000 7f733929a640 1 --2- 172.21.15.59:0/2137197354 >> v2:172.21.15.164:3300/0 conn(0x7f73340cf850 0x7f73340cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.353+0000 7f733929a640 1 --2- 172.21.15.59:0/2137197354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73340d00c0 0x7f73340d04a0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.353+0000 7f733929a640 1 -- 172.21.15.59:0/2137197354 >> 172.21.15.59:0/2137197354 conn(0x7f73340b35a0 msgr2=0x7f73340b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.353+0000 7f733929a640 1 -- 172.21.15.59:0/2137197354 shutdown_connections 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.353+0000 7f733929a640 1 -- 172.21.15.59:0/2137197354 wait complete. 2024-03-06T07:30:47.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.354+0000 7f733929a640 1 Processor -- start 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.354+0000 7f733929a640 1 -- start start 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f733929a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73340ce1b0 0x7f73341f7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f733929a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73340cf850 0x7f73341f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f733929a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73340d00c0 0x7f733415cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f733929a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733406c440 con 0x7f73340ce1b0 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f733929a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733406c160 con 0x7f73340cf850 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f733929a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733406c720 con 0x7f73340d00c0 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f7332ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73340ce1b0 0x7f73341f7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f7332ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73340ce1b0 0x7f73341f7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35632/0 (socket says 172.21.15.59:35632) 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f7332ffd640 1 -- 172.21.15.59:0/4191638855 learned_addr learned my addr 172.21.15.59:0/4191638855 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f732bfff640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73340cf850 0x7f73341f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f73337fe640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73340d00c0 0x7f733415cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f7332ffd640 1 -- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73340d00c0 msgr2=0x7f733415cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f7332ffd640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73340d00c0 0x7f733415cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f7332ffd640 1 -- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73340cf850 msgr2=0x7f73341f2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f7332ffd640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73340cf850 0x7f73341f2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f7332ffd640 1 -- 172.21.15.59:0/4191638855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733415e580 con 0x7f73340ce1b0 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.355+0000 7f7332ffd640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73340ce1b0 0x7f73341f7710 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f7324000c00 tx=0x7f73240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.356+0000 7f732b7fe640 1 -- 172.21.15.59:0/4191638855 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324034040 con 0x7f73340ce1b0 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.356+0000 7f732b7fe640 1 -- 172.21.15.59:0/4191638855 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7324039d90 con 0x7f73340ce1b0 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.356+0000 7f733929a640 1 -- 172.21.15.59:0/4191638855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733415e840 con 0x7f73340ce1b0 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.356+0000 7f732b7fe640 1 -- 172.21.15.59:0/4191638855 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324043610 con 0x7f73340ce1b0 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.356+0000 7f733929a640 1 -- 172.21.15.59:0/4191638855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733415ee00 con 0x7f73340ce1b0 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.357+0000 7f733929a640 1 -- 172.21.15.59:0/4191638855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733404f330 con 0x7f73340ce1b0 2024-03-06T07:30:47.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.357+0000 7f732b7fe640 1 -- 172.21.15.59:0/4191638855 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f732404a060 con 0x7f73340ce1b0 2024-03-06T07:30:47.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.357+0000 7f732b7fe640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f731406b4d0 0x7f731406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.358+0000 7f732b7fe640 1 -- 172.21.15.59:0/4191638855 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f73240ac0f0 con 0x7f73340ce1b0 2024-03-06T07:30:47.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.358+0000 7f732bfff640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f731406b4d0 0x7f731406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.358+0000 7f732bfff640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f731406b4d0 0x7f731406d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f73340d25b0 tx=0x7f731c009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.362+0000 7f732b7fe640 1 -- 172.21.15.59:0/4191638855 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f732403e070 con 0x7f73340ce1b0 2024-03-06T07:30:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.583+0000 7f733929a640 1 -- 172.21.15.59:0/4191638855 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73340747a0 con 0x7f73340ce1b0 2024-03-06T07:30:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.583+0000 7f732b7fe640 1 -- 172.21.15.59:0/4191638855 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7f732407a990 con 0x7f73340ce1b0 2024-03-06T07:30:47.585 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:47.585 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:39.075108+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336,8236]},"id":10}]} 2024-03-06T07:30:47.585 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 103 2024-03-06T07:30:47.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 -- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f731406b4d0 msgr2=0x7f731406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f731406b4d0 0x7f731406d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f73340d25b0 tx=0x7f731c009190 comp rx=0 tx=0).stop 2024-03-06T07:30:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 -- 172.21.15.59:0/4191638855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73340ce1b0 msgr2=0x7f73341f7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73340ce1b0 0x7f73341f7710 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f7324000c00 tx=0x7f73240046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 -- 172.21.15.59:0/4191638855 shutdown_connections 2024-03-06T07:30:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f731406b4d0 0x7f731406d990 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73340d00c0 0x7f733415cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73340cf850 0x7f73341f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 --2- 172.21.15.59:0/4191638855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73340ce1b0 0x7f73341f7710 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 -- 172.21.15.59:0/4191638855 >> 172.21.15.59:0/4191638855 conn(0x7f73340b35a0 msgr2=0x7f733406eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 -- 172.21.15.59:0/4191638855 shutdown_connections 2024-03-06T07:30:47.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:47.585+0000 7f733929a640 1 -- 172.21.15.59:0/4191638855 wait complete. 2024-03-06T07:30:47.598 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:47.714 INFO:teuthology.orchestra.run.smithi059.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-03-06T07:30:47.715 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-06T07:30:47.715 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-03-06T07:30:49.716 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.816+0000 7feb48bbf640 1 Processor -- start 2024-03-06T07:30:49.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.816+0000 7feb48bbf640 1 -- start start 2024-03-06T07:30:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb48bbf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feb440d22b0 0x7feb440d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb48bbf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb440b3430 con 0x7feb4415dfe0 2024-03-06T07:30:49.819 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb48bbf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb440b3710 con 0x7feb440b41f0 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb48bbf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb440d2c50 con 0x7feb440d22b0 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb4259c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7feb4415dfe0 0x7feb440cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49136/0 (socket says 172.21.15.59:49136) 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb4259c640 1 -- 172.21.15.59:0/2811224986 learned_addr learned my addr 172.21.15.59:0/2811224986 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb3bfff640 1 --2- 172.21.15.59:0/2811224986 >> v2:172.21.15.164:3300/0 conn(0x7feb440d22b0 0x7feb440d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb41d9b640 1 -- 172.21.15.59:0/2811224986 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444436176 0 0) 0x7feb440b3430 con 0x7feb4415dfe0 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb41d9b640 1 -- 172.21.15.59:0/2811224986 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb30003520 con 0x7feb4415dfe0 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb41d9b640 1 -- 172.21.15.59:0/2811224986 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816355934 0 0) 0x7feb440b3710 con 0x7feb440b41f0 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb41d9b640 1 -- 172.21.15.59:0/2811224986 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb440b3430 con 0x7feb440b41f0 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb3bfff640 1 -- 172.21.15.59:0/2811224986 >> v1:172.21.15.164:6789/0 conn(0x7feb440b41f0 legacy=0x7feb440b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb3bfff640 1 -- 172.21.15.59:0/2811224986 >> v1:172.21.15.59:6789/0 conn(0x7feb4415dfe0 legacy=0x7feb440cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.820 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.817+0000 7feb3bfff640 1 -- 172.21.15.59:0/2811224986 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb440b39f0 con 0x7feb440d22b0 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.819+0000 7feb3bfff640 1 --2- 172.21.15.59:0/2811224986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb440d22b0 0x7feb440d2710 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0097d0 tx=0x7feb2c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b10e548b8386895 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.819+0000 7feb41d9b640 1 -- 172.21.15.59:0/2811224986 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c034070 con 0x7feb440d22b0 2024-03-06T07:30:49.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.819+0000 7feb41d9b640 1 -- 172.21.15.59:0/2811224986 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c030610 con 0x7feb440d22b0 2024-03-06T07:30:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.819+0000 7feb41d9b640 1 -- 172.21.15.59:0/2811224986 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0309e0 con 0x7feb440d22b0 2024-03-06T07:30:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.819+0000 7feb48bbf640 1 -- 172.21.15.59:0/2811224986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb440d22b0 msgr2=0x7feb440d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.819+0000 7feb48bbf640 1 --2- 172.21.15.59:0/2811224986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb440d22b0 0x7feb440d2710 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0097d0 tx=0x7feb2c02b960 comp rx=0 tx=0).stop 2024-03-06T07:30:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.819+0000 7feb48bbf640 1 -- 172.21.15.59:0/2811224986 shutdown_connections 2024-03-06T07:30:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.819+0000 7feb48bbf640 1 --2- 172.21.15.59:0/2811224986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb440d22b0 0x7feb440d2710 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.819+0000 7feb48bbf640 1 -- 172.21.15.59:0/2811224986 >> 172.21.15.59:0/2811224986 conn(0x7feb440d3cd0 msgr2=0x7feb440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.819+0000 7feb48bbf640 1 -- 172.21.15.59:0/2811224986 shutdown_connections 2024-03-06T07:30:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.819+0000 7feb48bbf640 1 -- 172.21.15.59:0/2811224986 wait complete. 2024-03-06T07:30:49.822 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.821+0000 7feb48bbf640 1 Processor -- start 2024-03-06T07:30:49.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.821+0000 7feb48bbf640 1 -- start start 2024-03-06T07:30:49.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.821+0000 7feb48bbf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb440b41f0 0x7feb4414f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.821+0000 7feb48bbf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb440d22b0 0x7feb440b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.821+0000 7feb48bbf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb4415dfe0 0x7feb440ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.821+0000 7feb48bbf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb440b3d60 con 0x7feb440d22b0 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.821+0000 7feb48bbf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb440b31e0 con 0x7feb440b41f0 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.821+0000 7feb48bbf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb440d1c70 con 0x7feb4415dfe0 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb440d22b0 0x7feb440b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb440d22b0 0x7feb440b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35640/0 (socket says 172.21.15.59:35640) 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3bfff640 1 -- 172.21.15.59:0/2190477389 learned_addr learned my addr 172.21.15.59:0/2190477389 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb4259c640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb440b41f0 0x7feb4414f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb42d9d640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb4415dfe0 0x7feb440ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3bfff640 1 -- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb4415dfe0 msgr2=0x7feb440ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3bfff640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb4415dfe0 0x7feb440ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3bfff640 1 -- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb440b41f0 msgr2=0x7feb4414f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.825 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3bfff640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb440b41f0 0x7feb4414f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3bfff640 1 -- 172.21.15.59:0/2190477389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb441f7410 con 0x7feb440d22b0 2024-03-06T07:30:49.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3bfff640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb440d22b0 0x7feb440b68a0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7feb2c02bf70 tx=0x7feb2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3affd640 1 -- 172.21.15.59:0/2190477389 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c034040 con 0x7feb440d22b0 2024-03-06T07:30:49.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3affd640 1 -- 172.21.15.59:0/2190477389 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c030410 con 0x7feb440d22b0 2024-03-06T07:30:49.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb48bbf640 1 -- 172.21.15.59:0/2190477389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb441f76f0 con 0x7feb440d22b0 2024-03-06T07:30:49.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb3affd640 1 -- 172.21.15.59:0/2190477389 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c006e10 con 0x7feb440d22b0 2024-03-06T07:30:49.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.822+0000 7feb48bbf640 1 -- 172.21.15.59:0/2190477389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb441f7cb0 con 0x7feb440d22b0 2024-03-06T07:30:49.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.823+0000 7feb48bbf640 1 -- 172.21.15.59:0/2190477389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4404f330 con 0x7feb440d22b0 2024-03-06T07:30:49.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.824+0000 7feb3affd640 1 -- 172.21.15.59:0/2190477389 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb2c037070 con 0x7feb440d22b0 2024-03-06T07:30:49.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.824+0000 7feb3affd640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb2406b4d0 0x7feb2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.825+0000 7feb3affd640 1 -- 172.21.15.59:0/2190477389 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7feb2c0abf40 con 0x7feb440d22b0 2024-03-06T07:30:49.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.825+0000 7feb4259c640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb2406b4d0 0x7feb2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.827 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.825+0000 7feb4259c640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb2406b4d0 0x7feb2406d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7feb34000f60 tx=0x7feb34007d70 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:49.829+0000 7feb3affd640 1 -- 172.21.15.59:0/2190477389 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb2c07a880 con 0x7feb440d22b0 2024-03-06T07:30:50.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.050+0000 7feb48bbf640 1 -- 172.21.15.59:0/2190477389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb440b0af0 con 0x7feb440d22b0 2024-03-06T07:30:50.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.050+0000 7feb3affd640 1 -- 172.21.15.59:0/2190477389 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+3419 (secure 0 0 0) 0x7feb2c07e920 con 0x7feb440d22b0 2024-03-06T07:30:50.052 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:50.053 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:30:39.075108+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:active","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336,8236]},"id":10}]} 2024-03-06T07:30:50.053 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 103 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.052+0000 7feb48bbf640 1 -- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb2406b4d0 msgr2=0x7feb2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.052+0000 7feb48bbf640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb2406b4d0 0x7feb2406d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7feb34000f60 tx=0x7feb34007d70 comp rx=0 tx=0).stop 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.052+0000 7feb48bbf640 1 -- 172.21.15.59:0/2190477389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb440d22b0 msgr2=0x7feb440b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.052+0000 7feb48bbf640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb440d22b0 0x7feb440b68a0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7feb2c02bf70 tx=0x7feb2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.052+0000 7feb48bbf640 1 -- 172.21.15.59:0/2190477389 shutdown_connections 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.052+0000 7feb48bbf640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb2406b4d0 0x7feb2406d990 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.052+0000 7feb48bbf640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb4415dfe0 0x7feb440ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.052+0000 7feb48bbf640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb440d22b0 0x7feb440b68a0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.052+0000 7feb48bbf640 1 --2- 172.21.15.59:0/2190477389 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb440b41f0 0x7feb4414f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.052+0000 7feb48bbf640 1 -- 172.21.15.59:0/2190477389 >> 172.21.15.59:0/2190477389 conn(0x7feb440d3cd0 msgr2=0x7feb4415d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.053+0000 7feb48bbf640 1 -- 172.21.15.59:0/2190477389 shutdown_connections 2024-03-06T07:30:50.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.053+0000 7feb48bbf640 1 -- 172.21.15.59:0/2190477389 wait complete. 2024-03-06T07:30:50.065 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:30:50.182 INFO:teuthology.orchestra.run.smithi059.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/dir_2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-06T07:30:50.183 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "7", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "7" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/dir_2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:30:50.183 INFO:tasks.cephfs.cephfs_test_case:[('/dir_2', 0)] =?= [('/dir_2', 0)] 2024-03-06T07:30:50.183 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:30:50.183 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:50.198 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:50.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.337+0000 7f02894c3640 1 Processor -- start 2024-03-06T07:30:50.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.337+0000 7f02894c3640 1 -- start start 2024-03-06T07:30:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02894c3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f02840b4170 0x7f02840b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02894c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02840d4760 0x7f02840d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02894c3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f02840b4dd0 0x7f02840d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02894c3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02840d4bf0 con 0x7f02840b4dd0 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02894c3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02840ce9e0 con 0x7f02840b4170 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02894c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02840d25d0 con 0x7f02840d4760 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02837fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f02840b4dd0 0x7f02840d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f027bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02840d4760 0x7f02840d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02837fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f02840b4dd0 0x7f02840d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35642/0 (socket says 172.21.15.59:35642) 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02837fe640 1 -- 172.21.15.59:0/3550998963 learned_addr learned my addr 172.21.15.59:0/3550998963 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f0282ffd640 1 --2- 172.21.15.59:0/3550998963 >> v2:172.21.15.164:3300/0 conn(0x7f02840b4170 0x7f02840b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02837fe640 1 -- 172.21.15.59:0/3550998963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02840d4760 msgr2=0x7f02840d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02837fe640 1 --2- 172.21.15.59:0/3550998963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02840d4760 0x7f02840d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02837fe640 1 -- 172.21.15.59:0/3550998963 >> v2:172.21.15.164:3300/0 conn(0x7f02840b4170 msgr2=0x7f02840b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02837fe640 1 --2- 172.21.15.59:0/3550998963 >> v2:172.21.15.164:3300/0 conn(0x7f02840b4170 0x7f02840b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.338+0000 7f02837fe640 1 -- 172.21.15.59:0/3550998963 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02840d2870 con 0x7f02840b4dd0 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.339+0000 7f02837fe640 1 --2- 172.21.15.59:0/3550998963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02840b4dd0 0x7f02840d08b0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f028406e030 tx=0x7f027c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9752b6cf11a0f85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.339+0000 7f02827fc640 1 -- 172.21.15.59:0/3550998963 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c02ca70 con 0x7f02840b4dd0 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.339+0000 7f02827fc640 1 -- 172.21.15.59:0/3550998963 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027c02cc10 con 0x7f02840b4dd0 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.339+0000 7f02827fc640 1 -- 172.21.15.59:0/3550998963 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027c0343f0 con 0x7f02840b4dd0 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.340+0000 7f02894c3640 1 -- 172.21.15.59:0/3550998963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02840b4dd0 msgr2=0x7f02840d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.340+0000 7f02894c3640 1 --2- 172.21.15.59:0/3550998963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02840b4dd0 0x7f02840d08b0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f028406e030 tx=0x7f027c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.340+0000 7f02894c3640 1 -- 172.21.15.59:0/3550998963 shutdown_connections 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.340+0000 7f02894c3640 1 --2- 172.21.15.59:0/3550998963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02840b4dd0 0x7f02840d08b0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.340+0000 7f02894c3640 1 --2- 172.21.15.59:0/3550998963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02840d4760 0x7f02840d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.340+0000 7f02894c3640 1 --2- 172.21.15.59:0/3550998963 >> v2:172.21.15.164:3300/0 conn(0x7f02840b4170 0x7f02840b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.340+0000 7f02894c3640 1 -- 172.21.15.59:0/3550998963 >> 172.21.15.59:0/3550998963 conn(0x7f02840d3c40 msgr2=0x7f02840d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.340+0000 7f02894c3640 1 -- 172.21.15.59:0/3550998963 shutdown_connections 2024-03-06T07:30:50.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.340+0000 7f02894c3640 1 -- 172.21.15.59:0/3550998963 wait complete. 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.342+0000 7f02894c3640 1 Processor -- start 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.342+0000 7f02894c3640 1 -- start start 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.342+0000 7f02894c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02840b4170 0x7f02840c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.342+0000 7f02894c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02840b4dd0 0x7f02840c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.342+0000 7f02894c3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02840cad10 0x7f02840cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.342+0000 7f02894c3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02840b47d0 con 0x7f02840b4170 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.342+0000 7f02894c3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02840b4550 con 0x7f02840cad10 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.342+0000 7f02894c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02840b4950 con 0x7f02840b4dd0 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.342+0000 7f0282ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02840b4170 0x7f02840c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.342+0000 7f0282ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02840b4170 0x7f02840c9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35652/0 (socket says 172.21.15.59:35652) 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f0282ffd640 1 -- 172.21.15.59:0/467644723 learned_addr learned my addr 172.21.15.59:0/467644723 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:50.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f027bfff640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02840b4dd0 0x7f02840c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f02837fe640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02840cad10 0x7f02840cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f0282ffd640 1 -- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02840b4dd0 msgr2=0x7f02840c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f0282ffd640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02840b4dd0 0x7f02840c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f0282ffd640 1 -- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02840cad10 msgr2=0x7f02840cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f0282ffd640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02840cad10 0x7f02840cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f0282ffd640 1 -- 172.21.15.59:0/467644723 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02840c8a30 con 0x7f02840b4170 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f0282ffd640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02840b4170 0x7f02840c9d70 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f027400e870 tx=0x7f027400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f027b7fe640 1 -- 172.21.15.59:0/467644723 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274017070 con 0x7f02840b4170 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f027b7fe640 1 -- 172.21.15.59:0/467644723 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027400ee60 con 0x7f02840b4170 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f02894c3640 1 -- 172.21.15.59:0/467644723 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02840c8cd0 con 0x7f02840b4170 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f027b7fe640 1 -- 172.21.15.59:0/467644723 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027400cb80 con 0x7f02840b4170 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.343+0000 7f02894c3640 1 -- 172.21.15.59:0/467644723 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02840c9570 con 0x7f02840b4170 2024-03-06T07:30:50.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.344+0000 7f02894c3640 1 -- 172.21.15.59:0/467644723 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f028404f330 con 0x7f02840b4170 2024-03-06T07:30:50.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.345+0000 7f027b7fe640 1 -- 172.21.15.59:0/467644723 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f027401a070 con 0x7f02840b4170 2024-03-06T07:30:50.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.345+0000 7f027b7fe640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f026406b500 0x7f026406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.345+0000 7f027b7fe640 1 -- 172.21.15.59:0/467644723 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0274089fb0 con 0x7f02840b4170 2024-03-06T07:30:50.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.345+0000 7f027bfff640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f026406b500 0x7f026406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.346+0000 7f027bfff640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f026406b500 0x7f026406d9c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f02840cc820 tx=0x7f026c009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.350+0000 7f027b7fe640 1 -- 172.21.15.59:0/467644723 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02740587c0 con 0x7f02840b4170 2024-03-06T07:30:50.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.534+0000 7f02894c3640 1 -- 172.21.15.59:0/467644723 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02840733b0 con 0x7f02840b4170 2024-03-06T07:30:50.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.535+0000 7f027b7fe640 1 -- 172.21.15.59:0/467644723 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f0274081e20 con 0x7f02840b4170 2024-03-06T07:30:50.537 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:50.538 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":143,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:39.494279+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:11.712614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:12.406781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.22","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.537+0000 7f02894c3640 1 -- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f026406b500 msgr2=0x7f026406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.537+0000 7f02894c3640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f026406b500 0x7f026406d9c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f02840cc820 tx=0x7f026c009190 comp rx=0 tx=0).stop 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.537+0000 7f02894c3640 1 -- 172.21.15.59:0/467644723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02840b4170 msgr2=0x7f02840c9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.537+0000 7f02894c3640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02840b4170 0x7f02840c9d70 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f027400e870 tx=0x7f027400ed30 comp rx=0 tx=0).stop 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.537+0000 7f02894c3640 1 -- 172.21.15.59:0/467644723 shutdown_connections 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.537+0000 7f02894c3640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f026406b500 0x7f026406d9c0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.537+0000 7f02894c3640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02840cad10 0x7f02840cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.537+0000 7f02894c3640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02840b4dd0 0x7f02840c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.537+0000 7f02894c3640 1 --2- 172.21.15.59:0/467644723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f02840b4170 0x7f02840c9d70 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.537+0000 7f02894c3640 1 -- 172.21.15.59:0/467644723 >> 172.21.15.59:0/467644723 conn(0x7f02840d3c40 msgr2=0x7f02840cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.538+0000 7f02894c3640 1 -- 172.21.15.59:0/467644723 shutdown_connections 2024-03-06T07:30:50.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.538+0000 7f02894c3640 1 -- 172.21.15.59:0/467644723 wait complete. 2024-03-06T07:30:50.550 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:11.712614+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:12.406781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:50.551 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.669+0000 7f0e54965640 1 Processor -- start 2024-03-06T07:30:50.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.669+0000 7f0e54965640 1 -- start start 2024-03-06T07:30:50.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e54965640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0e501586e0 0x7f0e500aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e54965640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e500aef80 0x7f0e500cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e54965640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0e500cffd0 0x7f0e5006da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e54965640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e500d03f0 con 0x7f0e500cffd0 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e54965640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e50158ac0 con 0x7f0e501586e0 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e54965640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5015b2a0 con 0x7f0e500aef80 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e4ed9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0e500cffd0 0x7f0e5006da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e4ed9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0e500cffd0 0x7f0e5006da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35664/0 (socket says 172.21.15.59:35664) 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e4ed9d640 1 -- 172.21.15.59:0/1394808240 learned_addr learned my addr 172.21.15.59:0/1394808240 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e47fff640 1 --2- 172.21.15.59:0/1394808240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e500aef80 0x7f0e500cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e4e59c640 1 --2- 172.21.15.59:0/1394808240 >> v2:172.21.15.164:3300/0 conn(0x7f0e501586e0 0x7f0e500aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e4ed9d640 1 -- 172.21.15.59:0/1394808240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e500aef80 msgr2=0x7f0e500cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e4ed9d640 1 --2- 172.21.15.59:0/1394808240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e500aef80 0x7f0e500cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e4ed9d640 1 -- 172.21.15.59:0/1394808240 >> v2:172.21.15.164:3300/0 conn(0x7f0e501586e0 msgr2=0x7f0e500aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e4ed9d640 1 --2- 172.21.15.59:0/1394808240 >> v2:172.21.15.164:3300/0 conn(0x7f0e501586e0 0x7f0e500aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.670+0000 7f0e4ed9d640 1 -- 172.21.15.59:0/1394808240 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5015b540 con 0x7f0e500cffd0 2024-03-06T07:30:50.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.671+0000 7f0e4ed9d640 1 --2- 172.21.15.59:0/1394808240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e500cffd0 0x7f0e5006da20 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f0e50070d10 tx=0x7f0e4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b18e456920aa1f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.671+0000 7f0e4dd9b640 1 -- 172.21.15.59:0/1394808240 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4802ca70 con 0x7f0e500cffd0 2024-03-06T07:30:50.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.671+0000 7f0e4dd9b640 1 -- 172.21.15.59:0/1394808240 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4802cc10 con 0x7f0e500cffd0 2024-03-06T07:30:50.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.671+0000 7f0e4dd9b640 1 -- 172.21.15.59:0/1394808240 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e480343f0 con 0x7f0e500cffd0 2024-03-06T07:30:50.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.672+0000 7f0e54965640 1 -- 172.21.15.59:0/1394808240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e500cffd0 msgr2=0x7f0e5006da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.672+0000 7f0e54965640 1 --2- 172.21.15.59:0/1394808240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e500cffd0 0x7f0e5006da20 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f0e50070d10 tx=0x7f0e4802bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.672+0000 7f0e54965640 1 -- 172.21.15.59:0/1394808240 shutdown_connections 2024-03-06T07:30:50.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.672+0000 7f0e54965640 1 --2- 172.21.15.59:0/1394808240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e500cffd0 0x7f0e5006da20 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.672+0000 7f0e54965640 1 --2- 172.21.15.59:0/1394808240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e500aef80 0x7f0e500cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.672+0000 7f0e54965640 1 --2- 172.21.15.59:0/1394808240 >> v2:172.21.15.164:3300/0 conn(0x7f0e501586e0 0x7f0e500aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.672+0000 7f0e54965640 1 -- 172.21.15.59:0/1394808240 >> 172.21.15.59:0/1394808240 conn(0x7f0e5006b190 msgr2=0x7f0e5006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.672+0000 7f0e54965640 1 -- 172.21.15.59:0/1394808240 shutdown_connections 2024-03-06T07:30:50.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.672+0000 7f0e54965640 1 -- 172.21.15.59:0/1394808240 wait complete. 2024-03-06T07:30:50.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.673+0000 7f0e54965640 1 Processor -- start 2024-03-06T07:30:50.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.673+0000 7f0e54965640 1 -- start start 2024-03-06T07:30:50.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e54965640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e500aef80 0x7f0e50149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e54965640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e500cffd0 0x7f0e500b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e54965640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e501586e0 0x7f0e500b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e54965640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5015bde0 con 0x7f0e501586e0 2024-03-06T07:30:50.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e54965640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5015bb00 con 0x7f0e500cffd0 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e54965640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e50071350 con 0x7f0e500aef80 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e4ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e501586e0 0x7f0e500b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e4ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e501586e0 0x7f0e500b0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35668/0 (socket says 172.21.15.59:35668) 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e4ed9d640 1 -- 172.21.15.59:0/63209070 learned_addr learned my addr 172.21.15.59:0/63209070 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e4e59c640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e500aef80 0x7f0e50149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e47fff640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e500cffd0 0x7f0e500b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e4ed9d640 1 -- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e500aef80 msgr2=0x7f0e50149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e4ed9d640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e500aef80 0x7f0e50149cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e4ed9d640 1 -- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e500cffd0 msgr2=0x7f0e500b08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e4ed9d640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e500cffd0 0x7f0e500b08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e4ed9d640 1 -- 172.21.15.59:0/63209070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e501f7be0 con 0x7f0e501586e0 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.674+0000 7f0e4ed9d640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e501586e0 0x7f0e500b0e00 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f0e48036040 tx=0x7f0e48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.675+0000 7f0e46ffd640 1 -- 172.21.15.59:0/63209070 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48043020 con 0x7f0e501586e0 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.675+0000 7f0e46ffd640 1 -- 172.21.15.59:0/63209070 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4803e450 con 0x7f0e501586e0 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.675+0000 7f0e54965640 1 -- 172.21.15.59:0/63209070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5014c7c0 con 0x7f0e501586e0 2024-03-06T07:30:50.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.675+0000 7f0e46ffd640 1 -- 172.21.15.59:0/63209070 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48009510 con 0x7f0e501586e0 2024-03-06T07:30:50.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.675+0000 7f0e54965640 1 -- 172.21.15.59:0/63209070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5014cd80 con 0x7f0e501586e0 2024-03-06T07:30:50.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.676+0000 7f0e54965640 1 -- 172.21.15.59:0/63209070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e500732b0 con 0x7f0e501586e0 2024-03-06T07:30:50.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.676+0000 7f0e46ffd640 1 -- 172.21.15.59:0/63209070 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e48046020 con 0x7f0e501586e0 2024-03-06T07:30:50.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.676+0000 7f0e46ffd640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e3006b4d0 0x7f0e3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.676+0000 7f0e46ffd640 1 -- 172.21.15.59:0/63209070 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0e480ac7f0 con 0x7f0e501586e0 2024-03-06T07:30:50.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.677+0000 7f0e4e59c640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e3006b4d0 0x7f0e3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.677+0000 7f0e4e59c640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e3006b4d0 0x7f0e3006d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0e40009aa0 tx=0x7f0e40009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.681+0000 7f0e46ffd640 1 -- 172.21.15.59:0/63209070 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e4804d030 con 0x7f0e501586e0 2024-03-06T07:30:50.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.861+0000 7f0e54965640 1 -- 172.21.15.59:0/63209070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e500c59c0 con 0x7f0e501586e0 2024-03-06T07:30:50.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.862+0000 7f0e46ffd640 1 -- 172.21.15.59:0/63209070 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e4807b000 con 0x7f0e501586e0 2024-03-06T07:30:50.863 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:30:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.863+0000 7f0e54965640 1 -- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e3006b4d0 msgr2=0x7f0e3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.863+0000 7f0e54965640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e3006b4d0 0x7f0e3006d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0e40009aa0 tx=0x7f0e40009190 comp rx=0 tx=0).stop 2024-03-06T07:30:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.863+0000 7f0e54965640 1 -- 172.21.15.59:0/63209070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e501586e0 msgr2=0x7f0e500b0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.863+0000 7f0e54965640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e501586e0 0x7f0e500b0e00 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f0e48036040 tx=0x7f0e48004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.864+0000 7f0e54965640 1 -- 172.21.15.59:0/63209070 shutdown_connections 2024-03-06T07:30:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.864+0000 7f0e54965640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0e3006b4d0 0x7f0e3006d990 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.864+0000 7f0e54965640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e501586e0 0x7f0e500b0e00 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.864+0000 7f0e54965640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e500cffd0 0x7f0e500b08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.864+0000 7f0e54965640 1 --2- 172.21.15.59:0/63209070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e500aef80 0x7f0e50149cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.864+0000 7f0e54965640 1 -- 172.21.15.59:0/63209070 >> 172.21.15.59:0/63209070 conn(0x7f0e5006b190 msgr2=0x7f0e5006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.864+0000 7f0e54965640 1 -- 172.21.15.59:0/63209070 shutdown_connections 2024-03-06T07:30:50.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.864+0000 7f0e54965640 1 -- 172.21.15.59:0/63209070 wait complete. 2024-03-06T07:30:50.876 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:50.967 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:50.969 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:50.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.980+0000 7f1820f72640 1 Processor -- start 2024-03-06T07:30:50.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.980+0000 7f1820f72640 1 -- start start 2024-03-06T07:30:50.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f1820f72640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f181c0d0270 0x7f181c0cc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f1820f72640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f181c06c170 0x7f181c163f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f1820f72640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c06ca00 con 0x7f181c0afcf0 2024-03-06T07:30:50.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f1820f72640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181c0cc940 con 0x7f181c06c170 2024-03-06T07:30:50.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f1820f72640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181c15a7d0 con 0x7f181c0d0270 2024-03-06T07:30:50.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f181ad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f181c06c170 0x7f181c163f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f181a59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f181c0d0270 0x7f181c0cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f181ad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f181c06c170 0x7f181c163f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35684/0 (socket says 172.21.15.59:35684) 2024-03-06T07:30:50.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f181ad9d640 1 -- 172.21.15.59:0/2029836871 learned_addr learned my addr 172.21.15.59:0/2029836871 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:50.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f181ad9d640 1 -- 172.21.15.59:0/2029836871 >> v2:172.21.15.164:3300/0 conn(0x7f181c0d0270 msgr2=0x7f181c0cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f181ad9d640 1 --2- 172.21.15.59:0/2029836871 >> v2:172.21.15.164:3300/0 conn(0x7f181c0d0270 0x7f181c0cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f181ad9d640 1 -- 172.21.15.59:0/2029836871 >> v1:172.21.15.164:6789/0 conn(0x7f181c0afcf0 legacy=0x7f181c0d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.981+0000 7f181ad9d640 1 -- 172.21.15.59:0/2029836871 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c0b01f0 con 0x7f181c06c170 2024-03-06T07:30:50.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.982+0000 7f181ad9d640 1 --2- 172.21.15.59:0/2029836871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f181c06c170 0x7f181c163f70 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f181c070de0 tx=0x7f181402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8703db60033f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.983+0000 7f1819d9b640 1 -- 172.21.15.59:0/2029836871 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181402ca80 con 0x7f181c06c170 2024-03-06T07:30:50.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.983+0000 7f1819d9b640 1 -- 172.21.15.59:0/2029836871 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181402cc20 con 0x7f181c06c170 2024-03-06T07:30:50.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.983+0000 7f1819d9b640 1 -- 172.21.15.59:0/2029836871 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18140343f0 con 0x7f181c06c170 2024-03-06T07:30:50.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.983+0000 7f1820f72640 1 -- 172.21.15.59:0/2029836871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f181c06c170 msgr2=0x7f181c163f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.983+0000 7f1820f72640 1 --2- 172.21.15.59:0/2029836871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f181c06c170 0x7f181c163f70 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f181c070de0 tx=0x7f181402bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.983+0000 7f1820f72640 1 -- 172.21.15.59:0/2029836871 shutdown_connections 2024-03-06T07:30:50.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.983+0000 7f1820f72640 1 --2- 172.21.15.59:0/2029836871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f181c06c170 0x7f181c163f70 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.983+0000 7f1820f72640 1 --2- 172.21.15.59:0/2029836871 >> v2:172.21.15.164:3300/0 conn(0x7f181c0d0270 0x7f181c0cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.983+0000 7f1820f72640 1 -- 172.21.15.59:0/2029836871 >> 172.21.15.59:0/2029836871 conn(0x7f181c0b4fe0 msgr2=0x7f181c0b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.983+0000 7f1820f72640 1 -- 172.21.15.59:0/2029836871 shutdown_connections 2024-03-06T07:30:50.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.983+0000 7f1820f72640 1 -- 172.21.15.59:0/2029836871 wait complete. 2024-03-06T07:30:50.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.985+0000 7f1820f72640 1 Processor -- start 2024-03-06T07:30:50.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.985+0000 7f1820f72640 1 -- start start 2024-03-06T07:30:50.988 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.985+0000 7f1820f72640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f181c06c170 0x7f181c1f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.985+0000 7f1820f72640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c0afcf0 0x7f181c157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.985+0000 7f1820f72640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f181c0d0270 0x7f181c158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.985+0000 7f1820f72640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0b04d0 con 0x7f181c06c170 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.985+0000 7f1820f72640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0b2920 con 0x7f181c0afcf0 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.985+0000 7f1820f72640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0d0d60 con 0x7f181c0d0270 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f181a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f181c06c170 0x7f181c1f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f181a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f181c06c170 0x7f181c1f4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35686/0 (socket says 172.21.15.59:35686) 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f181a59c640 1 -- 172.21.15.59:0/2918988357 learned_addr learned my addr 172.21.15.59:0/2918988357 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f1813fff640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c0afcf0 0x7f181c157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f181ad9d640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f181c0d0270 0x7f181c158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f181a59c640 1 -- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f181c0d0270 msgr2=0x7f181c158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f181a59c640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f181c0d0270 0x7f181c158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f181a59c640 1 -- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c0afcf0 msgr2=0x7f181c157150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f181a59c640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c0afcf0 0x7f181c157150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f181a59c640 1 -- 172.21.15.59:0/2918988357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c1560a0 con 0x7f181c06c170 2024-03-06T07:30:50.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f181a59c640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f181c06c170 0x7f181c1f4a00 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f180c00e870 tx=0x7f180c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f1812ffd640 1 -- 172.21.15.59:0/2918988357 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c017070 con 0x7f181c06c170 2024-03-06T07:30:50.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f1812ffd640 1 -- 172.21.15.59:0/2918988357 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c00c940 con 0x7f181c06c170 2024-03-06T07:30:50.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f1820f72640 1 -- 172.21.15.59:0/2918988357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c0b2420 con 0x7f181c06c170 2024-03-06T07:30:50.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f1812ffd640 1 -- 172.21.15.59:0/2918988357 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c00cd30 con 0x7f181c06c170 2024-03-06T07:30:50.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.986+0000 7f1820f72640 1 -- 172.21.15.59:0/2918988357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c157690 con 0x7f181c06c170 2024-03-06T07:30:50.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.988+0000 7f1820f72640 1 -- 172.21.15.59:0/2918988357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c0b53c0 con 0x7f181c06c170 2024-03-06T07:30:50.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.988+0000 7f1812ffd640 1 -- 172.21.15.59:0/2918988357 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f180c01a070 con 0x7f181c06c170 2024-03-06T07:30:50.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.988+0000 7f1812ffd640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f17fc06b4d0 0x7f17fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.988+0000 7f1812ffd640 1 -- 172.21.15.59:0/2918988357 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f180c089f50 con 0x7f181c06c170 2024-03-06T07:30:50.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.989+0000 7f1813fff640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f17fc06b4d0 0x7f17fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.989+0000 7f1813fff640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f17fc06b4d0 0x7f17fc06d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f181c0670a0 tx=0x7f1804009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:50.995+0000 7f1812ffd640 1 -- 172.21.15.59:0/2918988357 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f181c0b53c0 con 0x7f181c06c170 2024-03-06T07:30:51.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.199+0000 7f1820f72640 1 -- 172.21.15.59:0/2918988357 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f181c09e4e0 con 0x7f181c06c170 2024-03-06T07:30:51.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.199+0000 7f1812ffd640 1 -- 172.21.15.59:0/2918988357 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7f180c0587f0 con 0x7f181c06c170 2024-03-06T07:30:51.201 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:30:51.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 -- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f17fc06b4d0 msgr2=0x7f17fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f17fc06b4d0 0x7f17fc06d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f181c0670a0 tx=0x7f1804009040 comp rx=0 tx=0).stop 2024-03-06T07:30:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 -- 172.21.15.59:0/2918988357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f181c06c170 msgr2=0x7f181c1f4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f181c06c170 0x7f181c1f4a00 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f180c00e870 tx=0x7f180c00ed30 comp rx=0 tx=0).stop 2024-03-06T07:30:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 -- 172.21.15.59:0/2918988357 shutdown_connections 2024-03-06T07:30:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f17fc06b4d0 0x7f17fc06d990 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f181c0d0270 0x7f181c158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c0afcf0 0x7f181c157150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 --2- 172.21.15.59:0/2918988357 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f181c06c170 0x7f181c1f4a00 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 -- 172.21.15.59:0/2918988357 >> 172.21.15.59:0/2918988357 conn(0x7f181c0b4fe0 msgr2=0x7f181c1533c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 -- 172.21.15.59:0/2918988357 shutdown_connections 2024-03-06T07:30:51.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.201+0000 7f1820f72640 1 -- 172.21.15.59:0/2918988357 wait complete. 2024-03-06T07:30:51.214 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:51.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.319+0000 7f333b119640 1 Processor -- start 2024-03-06T07:30:51.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.319+0000 7f333b119640 1 -- start start 2024-03-06T07:30:51.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.320+0000 7f333b119640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33341614d0 0x7f33340d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.320+0000 7f333b119640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f33340d2480 0x7f33340b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.320+0000 7f333b119640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33341618b0 con 0x7f33340d1910 2024-03-06T07:30:51.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.320+0000 7f333b119640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33340cf400 con 0x7f33340d2480 2024-03-06T07:30:51.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.320+0000 7f333b119640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33340cf6a0 con 0x7f33341614d0 2024-03-06T07:30:51.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.320+0000 7f33396b6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f33340d2480 0x7f33340b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.320+0000 7f33396b6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f33340d2480 0x7f33340b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35692/0 (socket says 172.21.15.59:35692) 2024-03-06T07:30:51.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.320+0000 7f33396b6640 1 -- 172.21.15.59:0/1436793954 learned_addr learned my addr 172.21.15.59:0/1436793954 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:51.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.320+0000 7f3338eb5640 1 --2- 172.21.15.59:0/1436793954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33341614d0 0x7f33340d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.321+0000 7f33396b6640 1 -- 172.21.15.59:0/1436793954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33341614d0 msgr2=0x7f33340d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.321+0000 7f33396b6640 1 --2- 172.21.15.59:0/1436793954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33341614d0 0x7f33340d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.321+0000 7f33396b6640 1 -- 172.21.15.59:0/1436793954 >> v1:172.21.15.164:6789/0 conn(0x7f33340d1910 legacy=0x7f33340d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.321+0000 7f33396b6640 1 -- 172.21.15.59:0/1436793954 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33340cf980 con 0x7f33340d2480 2024-03-06T07:30:51.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.321+0000 7f33396b6640 1 --2- 172.21.15.59:0/1436793954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33340d2480 0x7f33340b0e30 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f332c009e50 tx=0x7f332c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889daa952bdfe5bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.322+0000 7f33337fe640 1 -- 172.21.15.59:0/1436793954 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c02cb00 con 0x7f33340d2480 2024-03-06T07:30:51.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.322+0000 7f33337fe640 1 -- 172.21.15.59:0/1436793954 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332c02cca0 con 0x7f33340d2480 2024-03-06T07:30:51.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.322+0000 7f33337fe640 1 -- 172.21.15.59:0/1436793954 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c034610 con 0x7f33340d2480 2024-03-06T07:30:51.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.322+0000 7f333b119640 1 -- 172.21.15.59:0/1436793954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33340d2480 msgr2=0x7f33340b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.322+0000 7f333b119640 1 --2- 172.21.15.59:0/1436793954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33340d2480 0x7f33340b0e30 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f332c009e50 tx=0x7f332c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.322+0000 7f333b119640 1 -- 172.21.15.59:0/1436793954 shutdown_connections 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.322+0000 7f333b119640 1 --2- 172.21.15.59:0/1436793954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33340d2480 0x7f33340b0e30 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.322+0000 7f333b119640 1 --2- 172.21.15.59:0/1436793954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33341614d0 0x7f33340d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.322+0000 7f333b119640 1 -- 172.21.15.59:0/1436793954 >> 172.21.15.59:0/1436793954 conn(0x7f33340d3d40 msgr2=0x7f33340d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.322+0000 7f333b119640 1 -- 172.21.15.59:0/1436793954 shutdown_connections 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.322+0000 7f333b119640 1 -- 172.21.15.59:0/1436793954 wait complete. 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.323+0000 7f333b119640 1 Processor -- start 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.323+0000 7f333b119640 1 -- start start 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.324+0000 7f333b119640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33340d1910 0x7f33340b9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.324+0000 7f333b119640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33340d2480 0x7f3334150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.324+0000 7f333b119640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33341614d0 0x7f33340cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.324+0000 7f333b119640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33340b1370 con 0x7f33340d1910 2024-03-06T07:30:51.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.324+0000 7f333b119640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33340b0910 con 0x7f33340d2480 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.324+0000 7f333b119640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33340b14f0 con 0x7f33341614d0 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.324+0000 7f3338eb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33340d1910 0x7f33340b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.324+0000 7f3338eb5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33340d1910 0x7f33340b9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35698/0 (socket says 172.21.15.59:35698) 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.324+0000 7f3338eb5640 1 -- 172.21.15.59:0/1636423653 learned_addr learned my addr 172.21.15.59:0/1636423653 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.324+0000 7f33396b6640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33341614d0 0x7f33340cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.324+0000 7f3333fff640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33340d2480 0x7f3334150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.325+0000 7f3338eb5640 1 -- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33341614d0 msgr2=0x7f33340cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.325+0000 7f3338eb5640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33341614d0 0x7f33340cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.325+0000 7f3338eb5640 1 -- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33340d2480 msgr2=0x7f3334150210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.325+0000 7f3338eb5640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33340d2480 0x7f3334150210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.325+0000 7f3338eb5640 1 -- 172.21.15.59:0/1636423653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33340cb350 con 0x7f33340d1910 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.325+0000 7f3338eb5640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33340d1910 0x7f33340b9350 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f332800e870 tx=0x7f332800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.325+0000 7f33317fa640 1 -- 172.21.15.59:0/1636423653 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3328017070 con 0x7f33340d1910 2024-03-06T07:30:51.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.325+0000 7f33317fa640 1 -- 172.21.15.59:0/1636423653 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332800c940 con 0x7f33340d1910 2024-03-06T07:30:51.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.325+0000 7f33317fa640 1 -- 172.21.15.59:0/1636423653 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332800cd30 con 0x7f33340d1910 2024-03-06T07:30:51.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.325+0000 7f333b119640 1 -- 172.21.15.59:0/1636423653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33340cb5f0 con 0x7f33340d1910 2024-03-06T07:30:51.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.325+0000 7f333b119640 1 -- 172.21.15.59:0/1636423653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33340cbbb0 con 0x7f33340d1910 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.326+0000 7f333b119640 1 -- 172.21.15.59:0/1636423653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33340cef30 con 0x7f33340d1910 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.326+0000 7f33317fa640 1 -- 172.21.15.59:0/1636423653 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f332801a070 con 0x7f33340d1910 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.327+0000 7f33317fa640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f331806b4d0 0x7f331806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.327+0000 7f33317fa640 1 -- 172.21.15.59:0/1636423653 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3328089f50 con 0x7f33340d1910 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.327+0000 7f3333fff640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f331806b4d0 0x7f331806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.327+0000 7f3333fff640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f331806b4d0 0x7f331806d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f33340d06f0 tx=0x7f3320009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.331+0000 7f33317fa640 1 -- 172.21.15.59:0/1636423653 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33280587f0 con 0x7f33340d1910 2024-03-06T07:30:51.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.512+0000 7f333b119640 1 -- 172.21.15.59:0/1636423653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3334068ff0 con 0x7f33340d1910 2024-03-06T07:30:51.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.512+0000 7f33317fa640 1 -- 172.21.15.59:0/1636423653 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f332805c890 con 0x7f33340d1910 2024-03-06T07:30:51.514 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:30:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 -- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f331806b4d0 msgr2=0x7f331806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f331806b4d0 0x7f331806d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f33340d06f0 tx=0x7f3320009040 comp rx=0 tx=0).stop 2024-03-06T07:30:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 -- 172.21.15.59:0/1636423653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33340d1910 msgr2=0x7f33340b9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33340d1910 0x7f33340b9350 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f332800e870 tx=0x7f332800ed30 comp rx=0 tx=0).stop 2024-03-06T07:30:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 -- 172.21.15.59:0/1636423653 shutdown_connections 2024-03-06T07:30:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f331806b4d0 0x7f331806d990 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33341614d0 0x7f33340cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33340d2480 0x7f3334150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.516 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 --2- 172.21.15.59:0/1636423653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33340d1910 0x7f33340b9350 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 -- 172.21.15.59:0/1636423653 >> 172.21.15.59:0/1636423653 conn(0x7f33340d3d40 msgr2=0x7f33340cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 -- 172.21.15.59:0/1636423653 shutdown_connections 2024-03-06T07:30:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.514+0000 7f333b119640 1 -- 172.21.15.59:0/1636423653 wait complete. 2024-03-06T07:30:51.527 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:51.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.645+0000 7fed512c2640 1 Processor -- start 2024-03-06T07:30:51.647 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.645+0000 7fed512c2640 1 -- start start 2024-03-06T07:30:51.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.646+0000 7fed512c2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fed4c1614a0 0x7fed4c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.646+0000 7fed512c2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4c0af4b0 con 0x7fed4c0b1a20 2024-03-06T07:30:51.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.646+0000 7fed512c2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4c0af790 con 0x7fed4c0b0270 2024-03-06T07:30:51.648 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.646+0000 7fed512c2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed4c0afa70 con 0x7fed4c1614a0 2024-03-06T07:30:51.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.646+0000 7fed4affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fed4c1614a0 0x7fed4c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.646+0000 7fed4b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fed4c0b1a20 0x7fed4c0b5510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49148/0 (socket says 172.21.15.59:49148) 2024-03-06T07:30:51.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.646+0000 7fed4b7fe640 1 -- 172.21.15.59:0/1974106848 learned_addr learned my addr 172.21.15.59:0/1974106848 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:51.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.646+0000 7fed4affd640 1 -- 172.21.15.59:0/1974106848 >> v1:172.21.15.164:6789/0 conn(0x7fed4c0b0270 legacy=0x7fed4c0b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.646+0000 7fed4affd640 1 -- 172.21.15.59:0/1974106848 >> v1:172.21.15.59:6789/0 conn(0x7fed4c0b1a20 legacy=0x7fed4c0b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.646+0000 7fed4affd640 1 -- 172.21.15.59:0/1974106848 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c0afd50 con 0x7fed4c1614a0 2024-03-06T07:30:51.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.647+0000 7fed4affd640 1 --2- 172.21.15.59:0/1974106848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed4c1614a0 0x7fed4c06d3c0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fed440097a0 tx=0x7fed4402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517f1d729ad174ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.647+0000 7fed49ffb640 1 -- 172.21.15.59:0/1974106848 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44035020 con 0x7fed4c1614a0 2024-03-06T07:30:51.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.647+0000 7fed49ffb640 1 -- 172.21.15.59:0/1974106848 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44031830 con 0x7fed4c1614a0 2024-03-06T07:30:51.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.647+0000 7fed49ffb640 1 -- 172.21.15.59:0/1974106848 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44031c00 con 0x7fed4c1614a0 2024-03-06T07:30:51.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.648+0000 7fed512c2640 1 -- 172.21.15.59:0/1974106848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed4c1614a0 msgr2=0x7fed4c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.648+0000 7fed512c2640 1 --2- 172.21.15.59:0/1974106848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed4c1614a0 0x7fed4c06d3c0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fed440097a0 tx=0x7fed4402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.648+0000 7fed512c2640 1 -- 172.21.15.59:0/1974106848 shutdown_connections 2024-03-06T07:30:51.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.648+0000 7fed512c2640 1 --2- 172.21.15.59:0/1974106848 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed4c1614a0 0x7fed4c06d3c0 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.648+0000 7fed512c2640 1 -- 172.21.15.59:0/1974106848 >> 172.21.15.59:0/1974106848 conn(0x7fed4c09a9f0 msgr2=0x7fed4c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.648+0000 7fed512c2640 1 -- 172.21.15.59:0/1974106848 shutdown_connections 2024-03-06T07:30:51.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.648+0000 7fed512c2640 1 -- 172.21.15.59:0/1974106848 wait complete. 2024-03-06T07:30:51.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.650+0000 7fed512c2640 1 Processor -- start 2024-03-06T07:30:51.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.650+0000 7fed512c2640 1 -- start start 2024-03-06T07:30:51.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed512c2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed4c0b0270 0x7fed4c154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed512c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed4c0b1a20 0x7fed4c155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed512c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed4c1614a0 0x7fed4c1f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed512c2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0b1e00 con 0x7fed4c0b0270 2024-03-06T07:30:51.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed512c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0b06e0 con 0x7fed4c1614a0 2024-03-06T07:30:51.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed512c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c0b1f80 con 0x7fed4c0b1a20 2024-03-06T07:30:51.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed4affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed4c0b0270 0x7fed4c154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed4affd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed4c0b0270 0x7fed4c154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35714/0 (socket says 172.21.15.59:35714) 2024-03-06T07:30:51.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed4affd640 1 -- 172.21.15.59:0/3572651553 learned_addr learned my addr 172.21.15.59:0/3572651553 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:51.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed4a7fc640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed4c0b1a20 0x7fed4c155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed4b7fe640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed4c1614a0 0x7fed4c1f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed4affd640 1 -- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed4c0b1a20 msgr2=0x7fed4c155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed4affd640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed4c0b1a20 0x7fed4c155140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed4affd640 1 -- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed4c1614a0 msgr2=0x7fed4c1f3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed4affd640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed4c1614a0 0x7fed4c1f3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed4affd640 1 -- 172.21.15.59:0/3572651553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c1f4ec0 con 0x7fed4c0b0270 2024-03-06T07:30:51.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.651+0000 7fed4affd640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed4c0b0270 0x7fed4c154c00 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fed4402bee0 tx=0x7fed440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.652+0000 7fed437fe640 1 -- 172.21.15.59:0/3572651553 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44041070 con 0x7fed4c0b0270 2024-03-06T07:30:51.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.652+0000 7fed437fe640 1 -- 172.21.15.59:0/3572651553 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4403a950 con 0x7fed4c0b0270 2024-03-06T07:30:51.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.652+0000 7fed512c2640 1 -- 172.21.15.59:0/3572651553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4c1f4be0 con 0x7fed4c0b0270 2024-03-06T07:30:51.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.652+0000 7fed437fe640 1 -- 172.21.15.59:0/3572651553 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44035040 con 0x7fed4c0b0270 2024-03-06T07:30:51.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.652+0000 7fed512c2640 1 -- 172.21.15.59:0/3572651553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4c1f7960 con 0x7fed4c0b0270 2024-03-06T07:30:51.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.653+0000 7fed512c2640 1 -- 172.21.15.59:0/3572651553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4c04f330 con 0x7fed4c0b0270 2024-03-06T07:30:51.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.653+0000 7fed437fe640 1 -- 172.21.15.59:0/3572651553 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed4404c060 con 0x7fed4c0b0270 2024-03-06T07:30:51.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.653+0000 7fed437fe640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fed2c06b4d0 0x7fed2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.653+0000 7fed437fe640 1 -- 172.21.15.59:0/3572651553 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fed440abad0 con 0x7fed4c0b0270 2024-03-06T07:30:51.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.654+0000 7fed4a7fc640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fed2c06b4d0 0x7fed2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.654+0000 7fed4a7fc640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fed2c06b4d0 0x7fed2c06d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fed4c1f1b50 tx=0x7fed34009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.658+0000 7fed437fe640 1 -- 172.21.15.59:0/3572651553 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed4407a370 con 0x7fed4c0b0270 2024-03-06T07:30:51.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.902+0000 7fed512c2640 1 -- 172.21.15.59:0/3572651553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fed4c0cb6a0 con 0x7fed4c0b0270 2024-03-06T07:30:51.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.903+0000 7fed437fe640 1 -- 172.21.15.59:0/3572651553 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) v1 ==== 80+0+0 (secure 0 0 0) 0x7fed4407e410 con 0x7fed4c0b0270 2024-03-06T07:30:51.905 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:30:51.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 -- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fed2c06b4d0 msgr2=0x7fed2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fed2c06b4d0 0x7fed2c06d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fed4c1f1b50 tx=0x7fed34009040 comp rx=0 tx=0).stop 2024-03-06T07:30:51.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 -- 172.21.15.59:0/3572651553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed4c0b0270 msgr2=0x7fed4c154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed4c0b0270 0x7fed4c154c00 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fed4402bee0 tx=0x7fed440046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 -- 172.21.15.59:0/3572651553 shutdown_connections 2024-03-06T07:30:51.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fed2c06b4d0 0x7fed2c06d990 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed4c1614a0 0x7fed4c1f3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed4c0b1a20 0x7fed4c155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 --2- 172.21.15.59:0/3572651553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed4c0b0270 0x7fed4c154c00 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 -- 172.21.15.59:0/3572651553 >> 172.21.15.59:0/3572651553 conn(0x7fed4c09a9f0 msgr2=0x7fed4c06ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 -- 172.21.15.59:0/3572651553 shutdown_connections 2024-03-06T07:30:51.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:51.905+0000 7fed512c2640 1 -- 172.21.15.59:0/3572651553 wait complete. 2024-03-06T07:30:51.918 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:30:51.918 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:51.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.071+0000 7f7dd7313640 1 Processor -- start 2024-03-06T07:30:52.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.072+0000 7f7dd7313640 1 -- start start 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.072+0000 7f7dd7313640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7dd00d22b0 0x7f7dd00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.072+0000 7f7dd7313640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dd00b41f0 0x7f7dd00b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.072+0000 7f7dd7313640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd00b3430 con 0x7f7dd015dfe0 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.072+0000 7f7dd7313640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd00d2c50 con 0x7f7dd00d22b0 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.072+0000 7f7dd7313640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd00b31e0 con 0x7f7dd00b41f0 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.072+0000 7f7dd48ae640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7dd00d22b0 0x7f7dd00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.072+0000 7f7dd50af640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7dd015dfe0 0x7f7dd00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49162/0 (socket says 172.21.15.59:49162) 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.072+0000 7f7dd50af640 1 -- 172.21.15.59:0/3934581570 learned_addr learned my addr 172.21.15.59:0/3934581570 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.073+0000 7f7dd58b0640 1 --2- 172.21.15.59:0/3934581570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dd00b41f0 0x7f7dd00b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.073+0000 7f7dd48ae640 1 -- 172.21.15.59:0/3934581570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dd00b41f0 msgr2=0x7f7dd00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.073+0000 7f7dd48ae640 1 --2- 172.21.15.59:0/3934581570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dd00b41f0 0x7f7dd00b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.073+0000 7f7dd48ae640 1 -- 172.21.15.59:0/3934581570 >> v1:172.21.15.59:6789/0 conn(0x7f7dd015dfe0 legacy=0x7f7dd00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.073+0000 7f7dcffff640 1 -- 172.21.15.59:0/3934581570 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105263935 0 0) 0x7f7dd00b3430 con 0x7f7dd015dfe0 2024-03-06T07:30:52.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.073+0000 7f7dd48ae640 1 -- 172.21.15.59:0/3934581570 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd00b3710 con 0x7f7dd00d22b0 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dd48ae640 1 --2- 172.21.15.59:0/3934581570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dd00d22b0 0x7f7dd00d2710 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4009d90 tx=0x7f7dc402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c74fa1e95e9b79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dcffff640 1 -- 172.21.15.59:0/3934581570 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4039070 con 0x7f7dd00d22b0 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dcffff640 1 -- 172.21.15.59:0/3934581570 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc402cc50 con 0x7f7dd00d22b0 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dcffff640 1 -- 172.21.15.59:0/3934581570 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4034440 con 0x7f7dd00d22b0 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dd7313640 1 -- 172.21.15.59:0/3934581570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dd00d22b0 msgr2=0x7f7dd00d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dd7313640 1 --2- 172.21.15.59:0/3934581570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dd00d22b0 0x7f7dd00d2710 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4009d90 tx=0x7f7dc402bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dd7313640 1 -- 172.21.15.59:0/3934581570 shutdown_connections 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dd7313640 1 --2- 172.21.15.59:0/3934581570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dd00b41f0 0x7f7dd00b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dd7313640 1 --2- 172.21.15.59:0/3934581570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dd00d22b0 0x7f7dd00d2710 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dd7313640 1 -- 172.21.15.59:0/3934581570 >> 172.21.15.59:0/3934581570 conn(0x7f7dd00d3cd0 msgr2=0x7f7dd00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dd7313640 1 -- 172.21.15.59:0/3934581570 shutdown_connections 2024-03-06T07:30:52.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.074+0000 7f7dd7313640 1 -- 172.21.15.59:0/3934581570 wait complete. 2024-03-06T07:30:52.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.076+0000 7f7dd7313640 1 Processor -- start 2024-03-06T07:30:52.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.076+0000 7f7dd7313640 1 -- start start 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.077+0000 7f7dd7313640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dd00b41f0 0x7f7dd01f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.077+0000 7f7dd7313640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dd00d22b0 0x7f7dd006cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.077+0000 7f7dd7313640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dd015dfe0 0x7f7dd01f6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.077+0000 7f7dd7313640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd00cfa00 con 0x7f7dd00b41f0 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.077+0000 7f7dd7313640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd00b3d60 con 0x7f7dd00d22b0 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.077+0000 7f7dd7313640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd00d1c70 con 0x7f7dd015dfe0 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.077+0000 7f7dd50af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dd00b41f0 0x7f7dd01f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd50af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dd00b41f0 0x7f7dd01f4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35730/0 (socket says 172.21.15.59:35730) 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd50af640 1 -- 172.21.15.59:0/4037416142 learned_addr learned my addr 172.21.15.59:0/4037416142 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd48ae640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dd00d22b0 0x7f7dd006cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd58b0640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dd015dfe0 0x7f7dd01f6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd50af640 1 -- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dd015dfe0 msgr2=0x7f7dd01f6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd50af640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dd015dfe0 0x7f7dd01f6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd50af640 1 -- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dd00d22b0 msgr2=0x7f7dd006cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd50af640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dd00d22b0 0x7f7dd006cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd50af640 1 -- 172.21.15.59:0/4037416142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd01f78c0 con 0x7f7dd00b41f0 2024-03-06T07:30:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd50af640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dd00b41f0 0x7f7dd01f4250 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000c430 tx=0x7f7dc000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dcdffb640 1 -- 172.21.15.59:0/4037416142 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc00074e0 con 0x7f7dd00b41f0 2024-03-06T07:30:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dcdffb640 1 -- 172.21.15.59:0/4037416142 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc00047e0 con 0x7f7dd00b41f0 2024-03-06T07:30:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd7313640 1 -- 172.21.15.59:0/4037416142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd01f7ba0 con 0x7f7dd00b41f0 2024-03-06T07:30:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dcdffb640 1 -- 172.21.15.59:0/4037416142 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0004bd0 con 0x7f7dd00b41f0 2024-03-06T07:30:52.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.078+0000 7f7dd7313640 1 -- 172.21.15.59:0/4037416142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd01f3d50 con 0x7f7dd00b41f0 2024-03-06T07:30:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.080+0000 7f7dd7313640 1 -- 172.21.15.59:0/4037416142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd004f330 con 0x7f7dd00b41f0 2024-03-06T07:30:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.080+0000 7f7dcdffb640 1 -- 172.21.15.59:0/4037416142 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7dc0011070 con 0x7f7dd00b41f0 2024-03-06T07:30:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.081+0000 7f7dcdffb640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7dbc06b4d0 0x7f7dbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.081+0000 7f7dcdffb640 1 -- 172.21.15.59:0/4037416142 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7dc0089d60 con 0x7f7dd00b41f0 2024-03-06T07:30:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.081+0000 7f7dd48ae640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7dbc06b4d0 0x7f7dbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.081+0000 7f7dd48ae640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7dbc06b4d0 0x7f7dbc06d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4036000 tx=0x7f7dc4030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.085+0000 7f7dcdffb640 1 -- 172.21.15.59:0/4037416142 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7dc0058560 con 0x7f7dd00b41f0 2024-03-06T07:30:52.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.266+0000 7f7dd7313640 1 -- 172.21.15.59:0/4037416142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dd00d0eb0 con 0x7f7dd00b41f0 2024-03-06T07:30:52.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.268+0000 7f7dcdffb640 1 -- 172.21.15.59:0/4037416142 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f7dc005c600 con 0x7f7dd00b41f0 2024-03-06T07:30:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:30:52.270 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":143,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:39.494279+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:11.712614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:12.406781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.22","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.269+0000 7f7dd7313640 1 -- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7dbc06b4d0 msgr2=0x7f7dbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.269+0000 7f7dd7313640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7dbc06b4d0 0x7f7dbc06d990 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4036000 tx=0x7f7dc4030000 comp rx=0 tx=0).stop 2024-03-06T07:30:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.269+0000 7f7dd7313640 1 -- 172.21.15.59:0/4037416142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dd00b41f0 msgr2=0x7f7dd01f4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.269+0000 7f7dd7313640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dd00b41f0 0x7f7dd01f4250 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000c430 tx=0x7f7dc000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.269+0000 7f7dd7313640 1 -- 172.21.15.59:0/4037416142 shutdown_connections 2024-03-06T07:30:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.269+0000 7f7dd7313640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7dbc06b4d0 0x7f7dbc06d990 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.269+0000 7f7dd7313640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7dd015dfe0 0x7f7dd01f6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.269+0000 7f7dd7313640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dd00d22b0 0x7f7dd006cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.269+0000 7f7dd7313640 1 --2- 172.21.15.59:0/4037416142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7dd00b41f0 0x7f7dd01f4250 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.269+0000 7f7dd7313640 1 -- 172.21.15.59:0/4037416142 >> 172.21.15.59:0/4037416142 conn(0x7f7dd00d3cd0 msgr2=0x7f7dd00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.270+0000 7f7dd7313640 1 -- 172.21.15.59:0/4037416142 shutdown_connections 2024-03-06T07:30:52.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.270+0000 7f7dd7313640 1 -- 172.21.15.59:0/4037416142 wait complete. 2024-03-06T07:30:52.282 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:11.712614+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:12.406781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:52.283 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:52.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.388+0000 7efd82363640 1 Processor -- start 2024-03-06T07:30:52.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.388+0000 7efd82363640 1 -- start start 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd82363640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd7c1455d0 0x7efd7c072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd82363640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efd7c06b3f0 0x7efd7c06e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd82363640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c1459b0 con 0x7efd7c06afb0 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd82363640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7c0cfdf0 con 0x7efd7c06b3f0 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd82363640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0cf1e0 con 0x7efd7c1455d0 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd80900640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efd7c06b3f0 0x7efd7c06e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd80900640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efd7c06b3f0 0x7efd7c06e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35736/0 (socket says 172.21.15.59:35736) 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd7bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd7c1455d0 0x7efd7c072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd80900640 1 -- 172.21.15.59:0/2847594315 learned_addr learned my addr 172.21.15.59:0/2847594315 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd80900640 1 -- 172.21.15.59:0/2847594315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd7c1455d0 msgr2=0x7efd7c072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd80900640 1 --2- 172.21.15.59:0/2847594315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd7c1455d0 0x7efd7c072480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd80900640 1 -- 172.21.15.59:0/2847594315 >> v1:172.21.15.164:6789/0 conn(0x7efd7c06afb0 legacy=0x7efd7c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd80900640 1 -- 172.21.15.59:0/2847594315 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c0cf4c0 con 0x7efd7c06b3f0 2024-03-06T07:30:52.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.390+0000 7efd7affd640 1 -- 172.21.15.59:0/2847594315 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013412706 0 0) 0x7efd7c1459b0 con 0x7efd7c06afb0 2024-03-06T07:30:52.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.391+0000 7efd80900640 1 --2- 172.21.15.59:0/2847594315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c06b3f0 0x7efd7c06e5e0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7efd74009e50 tx=0x7efd7402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e455cd63e8f7d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.391+0000 7efd7affd640 1 -- 172.21.15.59:0/2847594315 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7402cb00 con 0x7efd7c06b3f0 2024-03-06T07:30:52.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.391+0000 7efd7affd640 1 -- 172.21.15.59:0/2847594315 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7402cca0 con 0x7efd7c06b3f0 2024-03-06T07:30:52.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.391+0000 7efd7affd640 1 -- 172.21.15.59:0/2847594315 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74034610 con 0x7efd7c06b3f0 2024-03-06T07:30:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.391+0000 7efd82363640 1 -- 172.21.15.59:0/2847594315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c06b3f0 msgr2=0x7efd7c06e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.391+0000 7efd82363640 1 --2- 172.21.15.59:0/2847594315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c06b3f0 0x7efd7c06e5e0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7efd74009e50 tx=0x7efd7402bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.391+0000 7efd82363640 1 -- 172.21.15.59:0/2847594315 shutdown_connections 2024-03-06T07:30:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.392+0000 7efd82363640 1 --2- 172.21.15.59:0/2847594315 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c06b3f0 0x7efd7c06e5e0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.392+0000 7efd82363640 1 --2- 172.21.15.59:0/2847594315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd7c1455d0 0x7efd7c072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.392+0000 7efd82363640 1 -- 172.21.15.59:0/2847594315 >> 172.21.15.59:0/2847594315 conn(0x7efd7c0b2660 msgr2=0x7efd7c0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.392+0000 7efd82363640 1 -- 172.21.15.59:0/2847594315 shutdown_connections 2024-03-06T07:30:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.392+0000 7efd82363640 1 -- 172.21.15.59:0/2847594315 wait complete. 2024-03-06T07:30:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.393+0000 7efd82363640 1 Processor -- start 2024-03-06T07:30:52.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.393+0000 7efd82363640 1 -- start start 2024-03-06T07:30:52.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.393+0000 7efd82363640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd7c06afb0 0x7efd7c1f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd82363640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd7c06b3f0 0x7efd7c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd82363640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c1455d0 0x7efd7c141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.397 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd82363640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c06b880 con 0x7efd7c1455d0 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd82363640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0cfa80 con 0x7efd7c06b3f0 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd82363640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c0729c0 con 0x7efd7c06afb0 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd80900640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c1455d0 0x7efd7c141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd7bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd7c06afb0 0x7efd7c1f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd80900640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c1455d0 0x7efd7c141380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35744/0 (socket says 172.21.15.59:35744) 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd80900640 1 -- 172.21.15.59:0/1667810512 learned_addr learned my addr 172.21.15.59:0/1667810512 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd7b7fe640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd7c06b3f0 0x7efd7c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd80900640 1 -- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd7c06afb0 msgr2=0x7efd7c1f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd80900640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd7c06afb0 0x7efd7c1f5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd80900640 1 -- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd7c06b3f0 msgr2=0x7efd7c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd80900640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd7c06b3f0 0x7efd7c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd80900640 1 -- 172.21.15.59:0/1667810512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c1f2bf0 con 0x7efd7c1455d0 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd80900640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c1455d0 0x7efd7c141380 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7efd74036040 tx=0x7efd74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd78ff9640 1 -- 172.21.15.59:0/1667810512 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74043020 con 0x7efd7c1455d0 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd78ff9640 1 -- 172.21.15.59:0/1667810512 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7402ce50 con 0x7efd7c1455d0 2024-03-06T07:30:52.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd82363640 1 -- 172.21.15.59:0/1667810512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c1434d0 con 0x7efd7c1455d0 2024-03-06T07:30:52.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd78ff9640 1 -- 172.21.15.59:0/1667810512 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74009510 con 0x7efd7c1455d0 2024-03-06T07:30:52.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.394+0000 7efd82363640 1 -- 172.21.15.59:0/1667810512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c143a90 con 0x7efd7c1455d0 2024-03-06T07:30:52.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.396+0000 7efd82363640 1 -- 172.21.15.59:0/1667810512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c073390 con 0x7efd7c1455d0 2024-03-06T07:30:52.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.396+0000 7efd78ff9640 1 -- 172.21.15.59:0/1667810512 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd74046020 con 0x7efd7c1455d0 2024-03-06T07:30:52.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.396+0000 7efd78ff9640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd5c06b4d0 0x7efd5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.399 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.396+0000 7efd78ff9640 1 -- 172.21.15.59:0/1667810512 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7efd740ac650 con 0x7efd7c1455d0 2024-03-06T07:30:52.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.397+0000 7efd7bfff640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd5c06b4d0 0x7efd5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.400 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.397+0000 7efd7bfff640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd5c06b4d0 0x7efd5c06d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7efd7c06bff0 tx=0x7efd64012040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.405+0000 7efd78ff9640 1 -- 172.21.15.59:0/1667810512 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd7404d030 con 0x7efd7c1455d0 2024-03-06T07:30:52.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.584+0000 7efd82363640 1 -- 172.21.15.59:0/1667810512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd7c15f1d0 con 0x7efd7c1455d0 2024-03-06T07:30:52.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.585+0000 7efd78ff9640 1 -- 172.21.15.59:0/1667810512 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd7407fe20 con 0x7efd7c1455d0 2024-03-06T07:30:52.587 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.586+0000 7efd82363640 1 -- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd5c06b4d0 msgr2=0x7efd5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.586+0000 7efd82363640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd5c06b4d0 0x7efd5c06d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7efd7c06bff0 tx=0x7efd64012040 comp rx=0 tx=0).stop 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.586+0000 7efd82363640 1 -- 172.21.15.59:0/1667810512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c1455d0 msgr2=0x7efd7c141380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.586+0000 7efd82363640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c1455d0 0x7efd7c141380 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7efd74036040 tx=0x7efd74004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.586+0000 7efd82363640 1 -- 172.21.15.59:0/1667810512 shutdown_connections 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.587+0000 7efd82363640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd5c06b4d0 0x7efd5c06d990 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.587+0000 7efd82363640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd7c1455d0 0x7efd7c141380 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.587+0000 7efd82363640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd7c06b3f0 0x7efd7c1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.587+0000 7efd82363640 1 --2- 172.21.15.59:0/1667810512 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd7c06afb0 0x7efd7c1f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.587+0000 7efd82363640 1 -- 172.21.15.59:0/1667810512 >> 172.21.15.59:0/1667810512 conn(0x7efd7c0b2660 msgr2=0x7efd7c15c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.587+0000 7efd82363640 1 -- 172.21.15.59:0/1667810512 shutdown_connections 2024-03-06T07:30:52.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.587+0000 7efd82363640 1 -- 172.21.15.59:0/1667810512 wait complete. 2024-03-06T07:30:52.603 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:52.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.706+0000 7fd5d85ac640 1 Processor -- start 2024-03-06T07:30:52.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.706+0000 7fd5d85ac640 1 -- start start 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d85ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5d00d06e0 0x7fd5d015fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d85ac640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd5d00d0ac0 0x7fd5d00b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d85ac640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d00d47f0 con 0x7fd5d00b4d10 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d85ac640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d00ced70 con 0x7fd5d00d0ac0 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d85ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00d1ab0 con 0x7fd5d00d06e0 2024-03-06T07:30:52.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d5b47640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd5d00d0ac0 0x7fd5d00b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d5b47640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd5d00d0ac0 0x7fd5d00b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35746/0 (socket says 172.21.15.59:35746) 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d5b47640 1 -- 172.21.15.59:0/1618672702 learned_addr learned my addr 172.21.15.59:0/1618672702 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d6348640 1 --2- 172.21.15.59:0/1618672702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5d00d06e0 0x7fd5d015fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d5b47640 1 -- 172.21.15.59:0/1618672702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5d00d06e0 msgr2=0x7fd5d015fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d5b47640 1 --2- 172.21.15.59:0/1618672702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5d00d06e0 0x7fd5d015fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d5b47640 1 -- 172.21.15.59:0/1618672702 >> v1:172.21.15.164:6789/0 conn(0x7fd5d00b4d10 legacy=0x7fd5d00cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.707+0000 7fd5d5b47640 1 -- 172.21.15.59:0/1618672702 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d00d1d90 con 0x7fd5d00d0ac0 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.708+0000 7fd5d5b47640 1 --2- 172.21.15.59:0/1618672702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d00d0ac0 0x7fd5d00b47d0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009870 tx=0x7fd5c002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e754278f6e959d67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.708+0000 7fd5d5346640 1 -- 172.21.15.59:0/1618672702 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0034070 con 0x7fd5d00d0ac0 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.708+0000 7fd5d5346640 1 -- 172.21.15.59:0/1618672702 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c0030840 con 0x7fd5d00d0ac0 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.708+0000 7fd5d5346640 1 -- 172.21.15.59:0/1618672702 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0030c10 con 0x7fd5d00d0ac0 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.709+0000 7fd5d85ac640 1 -- 172.21.15.59:0/1618672702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d00d0ac0 msgr2=0x7fd5d00b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.709+0000 7fd5d85ac640 1 --2- 172.21.15.59:0/1618672702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d00d0ac0 0x7fd5d00b47d0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009870 tx=0x7fd5c002b980 comp rx=0 tx=0).stop 2024-03-06T07:30:52.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.709+0000 7fd5d85ac640 1 -- 172.21.15.59:0/1618672702 shutdown_connections 2024-03-06T07:30:52.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.709+0000 7fd5d85ac640 1 --2- 172.21.15.59:0/1618672702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d00d0ac0 0x7fd5d00b47d0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.709+0000 7fd5d85ac640 1 --2- 172.21.15.59:0/1618672702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5d00d06e0 0x7fd5d015fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.709+0000 7fd5d85ac640 1 -- 172.21.15.59:0/1618672702 >> 172.21.15.59:0/1618672702 conn(0x7fd5d00d3cd0 msgr2=0x7fd5d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.709+0000 7fd5d85ac640 1 -- 172.21.15.59:0/1618672702 shutdown_connections 2024-03-06T07:30:52.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.709+0000 7fd5d85ac640 1 -- 172.21.15.59:0/1618672702 wait complete. 2024-03-06T07:30:52.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.710+0000 7fd5d85ac640 1 Processor -- start 2024-03-06T07:30:52.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.710+0000 7fd5d85ac640 1 -- start start 2024-03-06T07:30:52.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d85ac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5d00b4d10 0x7fd5d00c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d85ac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d00d06e0 0x7fd5d0156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d85ac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5d00d0ac0 0x7fd5d00b8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d85ac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00b44d0 con 0x7fd5d00d06e0 2024-03-06T07:30:52.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d85ac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00ce990 con 0x7fd5d00d0ac0 2024-03-06T07:30:52.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d85ac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00d0f00 con 0x7fd5d00b4d10 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d5b47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d00d06e0 0x7fd5d0156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d5b47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d00d06e0 0x7fd5d0156190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35750/0 (socket says 172.21.15.59:35750) 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d5b47640 1 -- 172.21.15.59:0/2125280351 learned_addr learned my addr 172.21.15.59:0/2125280351 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d6348640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5d00b4d10 0x7fd5d00c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d6b49640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5d00d0ac0 0x7fd5d00b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d5b47640 1 -- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5d00b4d10 msgr2=0x7fd5d00c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d5b47640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5d00b4d10 0x7fd5d00c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d5b47640 1 -- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5d00d0ac0 msgr2=0x7fd5d00b8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d5b47640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5d00d0ac0 0x7fd5d00b8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d5b47640 1 -- 172.21.15.59:0/2125280351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d01f56c0 con 0x7fd5d00d06e0 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d5b47640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d00d06e0 0x7fd5d0156190 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fd5c002b9c0 tx=0x7fd5c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5c6ffd640 1 -- 172.21.15.59:0/2125280351 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0034040 con 0x7fd5d00d06e0 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5c6ffd640 1 -- 172.21.15.59:0/2125280351 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c0039d90 con 0x7fd5d00d06e0 2024-03-06T07:30:52.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d85ac640 1 -- 172.21.15.59:0/2125280351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d00b7d50 con 0x7fd5d00d06e0 2024-03-06T07:30:52.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5c6ffd640 1 -- 172.21.15.59:0/2125280351 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0043630 con 0x7fd5d00d06e0 2024-03-06T07:30:52.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.711+0000 7fd5d85ac640 1 -- 172.21.15.59:0/2125280351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d00b7620 con 0x7fd5d00d06e0 2024-03-06T07:30:52.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.712+0000 7fd5d85ac640 1 -- 172.21.15.59:0/2125280351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d00a1c80 con 0x7fd5d00d06e0 2024-03-06T07:30:52.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.713+0000 7fd5c6ffd640 1 -- 172.21.15.59:0/2125280351 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5c004a060 con 0x7fd5d00d06e0 2024-03-06T07:30:52.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.713+0000 7fd5c6ffd640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5b406b4d0 0x7fd5b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:52.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.713+0000 7fd5c6ffd640 1 -- 172.21.15.59:0/2125280351 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd5c00abf90 con 0x7fd5d00d06e0 2024-03-06T07:30:52.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.713+0000 7fd5d6348640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5b406b4d0 0x7fd5b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:52.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.714+0000 7fd5d6348640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5b406b4d0 0x7fd5b406d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0067340 tx=0x7fd5cc009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:52.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.718+0000 7fd5c6ffd640 1 -- 172.21.15.59:0/2125280351 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5c003e070 con 0x7fd5d00d06e0 2024-03-06T07:30:52.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.899+0000 7fd5d85ac640 1 -- 172.21.15.59:0/2125280351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd5d0068ff0 con 0x7fd5d00d06e0 2024-03-06T07:30:52.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.899+0000 7fd5c6ffd640 1 -- 172.21.15.59:0/2125280351 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd5c007a830 con 0x7fd5d00d06e0 2024-03-06T07:30:52.901 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.901+0000 7fd5d85ac640 1 -- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5b406b4d0 msgr2=0x7fd5b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.901+0000 7fd5d85ac640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5b406b4d0 0x7fd5b406d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0067340 tx=0x7fd5cc009190 comp rx=0 tx=0).stop 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.901+0000 7fd5d85ac640 1 -- 172.21.15.59:0/2125280351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d00d06e0 msgr2=0x7fd5d0156190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.901+0000 7fd5d85ac640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d00d06e0 0x7fd5d0156190 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fd5c002b9c0 tx=0x7fd5c00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.901+0000 7fd5d85ac640 1 -- 172.21.15.59:0/2125280351 shutdown_connections 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.901+0000 7fd5d85ac640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5b406b4d0 0x7fd5b406d990 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.901+0000 7fd5d85ac640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5d00d0ac0 0x7fd5d00b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.901+0000 7fd5d85ac640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5d00d06e0 0x7fd5d0156190 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.901+0000 7fd5d85ac640 1 --2- 172.21.15.59:0/2125280351 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd5d00b4d10 0x7fd5d00c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.901+0000 7fd5d85ac640 1 -- 172.21.15.59:0/2125280351 >> 172.21.15.59:0/2125280351 conn(0x7fd5d00d3cd0 msgr2=0x7fd5d015f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.902+0000 7fd5d85ac640 1 -- 172.21.15.59:0/2125280351 shutdown_connections 2024-03-06T07:30:52.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:52.902+0000 7fd5d85ac640 1 -- 172.21.15.59:0/2125280351 wait complete. 2024-03-06T07:30:52.914 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:53.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.016+0000 7f4d01ed4640 1 Processor -- start 2024-03-06T07:30:53.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.016+0000 7f4d01ed4640 1 -- start start 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.017+0000 7f4d01ed4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4cfc1614d0 0x7f4cfc0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.017+0000 7f4d01ed4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4cfc0d1910 0x7f4cfc0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.017+0000 7f4d01ed4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc0d2740 con 0x7f4cfc0d22b0 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.017+0000 7f4d01ed4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cfc1618b0 con 0x7f4cfc0d1910 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.017+0000 7f4d01ed4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc0cf400 con 0x7f4cfc1614d0 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.017+0000 7f4cfaffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4cfc0d1910 0x7f4cfc0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.017+0000 7f4cfaffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4cfc0d1910 0x7f4cfc0d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35764/0 (socket says 172.21.15.59:35764) 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.017+0000 7f4cfaffd640 1 -- 172.21.15.59:0/1854873832 learned_addr learned my addr 172.21.15.59:0/1854873832 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.017+0000 7f4cfb7fe640 1 --2- 172.21.15.59:0/1854873832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4cfc1614d0 0x7f4cfc0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.018+0000 7f4cfaffd640 1 -- 172.21.15.59:0/1854873832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4cfc1614d0 msgr2=0x7f4cfc0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.018+0000 7f4cfa7fc640 1 -- 172.21.15.59:0/1854873832 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007619416 0 0) 0x7f4cf4001490 con 0x7f4cfc0d22b0 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.018+0000 7f4cfaffd640 1 --2- 172.21.15.59:0/1854873832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4cfc1614d0 0x7f4cfc0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.018+0000 7f4cfaffd640 1 -- 172.21.15.59:0/1854873832 >> v1:172.21.15.164:6789/0 conn(0x7f4cfc0d22b0 legacy=0x7f4cfc0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.018+0000 7f4cfaffd640 1 -- 172.21.15.59:0/1854873832 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc0cf6a0 con 0x7f4cfc0d1910 2024-03-06T07:30:53.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.018+0000 7f4cfb7fe640 1 --2- 172.21.15.59:0/1854873832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4cfc1614d0 0x7f4cfc0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.018+0000 7f4cfaffd640 1 --2- 172.21.15.59:0/1854873832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cfc0d1910 0x7f4cfc0d1d70 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009870 tx=0x7f4cf002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba4e3085dd96527 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.019+0000 7f4cfa7fc640 1 -- 172.21.15.59:0/1854873832 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0034070 con 0x7f4cfc0d1910 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.019+0000 7f4cfa7fc640 1 -- 172.21.15.59:0/1854873832 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf0030840 con 0x7f4cfc0d1910 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.019+0000 7f4cfa7fc640 1 -- 172.21.15.59:0/1854873832 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0030c10 con 0x7f4cfc0d1910 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.019+0000 7f4d01ed4640 1 -- 172.21.15.59:0/1854873832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cfc0d1910 msgr2=0x7f4cfc0d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.019+0000 7f4d01ed4640 1 --2- 172.21.15.59:0/1854873832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cfc0d1910 0x7f4cfc0d1d70 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009870 tx=0x7f4cf002b980 comp rx=0 tx=0).stop 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.019+0000 7f4d01ed4640 1 -- 172.21.15.59:0/1854873832 shutdown_connections 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.019+0000 7f4d01ed4640 1 --2- 172.21.15.59:0/1854873832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cfc0d1910 0x7f4cfc0d1d70 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.019+0000 7f4d01ed4640 1 --2- 172.21.15.59:0/1854873832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4cfc1614d0 0x7f4cfc0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.019+0000 7f4d01ed4640 1 -- 172.21.15.59:0/1854873832 >> 172.21.15.59:0/1854873832 conn(0x7f4cfc0d3d40 msgr2=0x7f4cfc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.019+0000 7f4d01ed4640 1 -- 172.21.15.59:0/1854873832 shutdown_connections 2024-03-06T07:30:53.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.019+0000 7f4d01ed4640 1 -- 172.21.15.59:0/1854873832 wait complete. 2024-03-06T07:30:53.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.020+0000 7f4d01ed4640 1 Processor -- start 2024-03-06T07:30:53.024 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.021+0000 7f4d01ed4640 1 -- start start 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.021+0000 7f4d01ed4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cfc0d1910 0x7f4cfc0d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.021+0000 7f4d01ed4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4cfc0d22b0 0x7f4cfc0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.021+0000 7f4d01ed4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4cfc1614d0 0x7f4cfc0c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.021+0000 7f4d01ed4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc0b0910 con 0x7f4cfc0d1910 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.021+0000 7f4d01ed4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc0cfc60 con 0x7f4cfc0d22b0 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.021+0000 7f4d01ed4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cfc0b1e40 con 0x7f4cfc1614d0 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.021+0000 7f4cfb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cfc0d1910 0x7f4cfc0d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.021+0000 7f4cfb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cfc0d1910 0x7f4cfc0d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35766/0 (socket says 172.21.15.59:35766) 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.021+0000 7f4cfb7fe640 1 -- 172.21.15.59:0/317478381 learned_addr learned my addr 172.21.15.59:0/317478381 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4cfaffd640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4cfc0d22b0 0x7f4cfc0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4cfbfff640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4cfc1614d0 0x7f4cfc0c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4cfb7fe640 1 -- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4cfc1614d0 msgr2=0x7f4cfc0c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4cfb7fe640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4cfc1614d0 0x7f4cfc0c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4cfb7fe640 1 -- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4cfc0d22b0 msgr2=0x7f4cfc0cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4cfb7fe640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4cfc0d22b0 0x7f4cfc0cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4cfb7fe640 1 -- 172.21.15.59:0/317478381 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc0cdc70 con 0x7f4cfc0d1910 2024-03-06T07:30:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4cfb7fe640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cfc0d1910 0x7f4cfc0d0bc0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00e960 tx=0x7f4cec00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4ce3fff640 1 -- 172.21.15.59:0/317478381 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec017070 con 0x7f4cfc0d1910 2024-03-06T07:30:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4ce3fff640 1 -- 172.21.15.59:0/317478381 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec00ca00 con 0x7f4cfc0d1910 2024-03-06T07:30:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4ce3fff640 1 -- 172.21.15.59:0/317478381 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec00cdd0 con 0x7f4cfc0d1910 2024-03-06T07:30:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4d01ed4640 1 -- 172.21.15.59:0/317478381 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cfc14d2a0 con 0x7f4cfc0d1910 2024-03-06T07:30:53.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.022+0000 7f4d01ed4640 1 -- 172.21.15.59:0/317478381 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cfc14db40 con 0x7f4cfc0d1910 2024-03-06T07:30:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.023+0000 7f4d01ed4640 1 -- 172.21.15.59:0/317478381 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cfc14d470 con 0x7f4cfc0d1910 2024-03-06T07:30:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.024+0000 7f4ce3fff640 1 -- 172.21.15.59:0/317478381 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cec01a070 con 0x7f4cfc0d1910 2024-03-06T07:30:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.024+0000 7f4ce3fff640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ce806b4d0 0x7f4ce806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.024+0000 7f4ce3fff640 1 -- 172.21.15.59:0/317478381 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4cec08a050 con 0x7f4cfc0d1910 2024-03-06T07:30:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.024+0000 7f4cfaffd640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ce806b4d0 0x7f4ce806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.025+0000 7f4cfaffd640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ce806b4d0 0x7f4ce806d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0036000 tx=0x7f4cf0008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.033+0000 7f4ce3fff640 1 -- 172.21.15.59:0/317478381 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4cec058860 con 0x7f4cfc0d1910 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.218+0000 7f4d01ed4640 1 -- 172.21.15.59:0/317478381 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4cfc069e80 con 0x7f4cfc0d1910 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.218+0000 7f4ce3fff640 1 -- 172.21.15.59:0/317478381 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4cfc069e80 con 0x7f4cfc0d1910 2024-03-06T07:30:53.220 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:30:53.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 -- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ce806b4d0 msgr2=0x7f4ce806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ce806b4d0 0x7f4ce806d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0036000 tx=0x7f4cf0008030 comp rx=0 tx=0).stop 2024-03-06T07:30:53.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 -- 172.21.15.59:0/317478381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cfc0d1910 msgr2=0x7f4cfc0d0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cfc0d1910 0x7f4cfc0d0bc0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00e960 tx=0x7f4cec00ee20 comp rx=0 tx=0).stop 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 -- 172.21.15.59:0/317478381 shutdown_connections 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4ce806b4d0 0x7f4ce806d990 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4cfc1614d0 0x7f4cfc0c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4cfc0d22b0 0x7f4cfc0cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 --2- 172.21.15.59:0/317478381 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cfc0d1910 0x7f4cfc0d0bc0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 -- 172.21.15.59:0/317478381 >> 172.21.15.59:0/317478381 conn(0x7f4cfc0d3d40 msgr2=0x7f4cfc06bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 -- 172.21.15.59:0/317478381 shutdown_connections 2024-03-06T07:30:53.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.220+0000 7f4d01ed4640 1 -- 172.21.15.59:0/317478381 wait complete. 2024-03-06T07:30:53.233 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:53.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.333+0000 7f7ef933e640 1 Processor -- start 2024-03-06T07:30:53.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.333+0000 7f7ef933e640 1 -- start start 2024-03-06T07:30:53.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.334+0000 7f7ef933e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef4145760 0x7f7ef406fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.334+0000 7f7ef933e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ef406b6c0 0x7f7ef406d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.334+0000 7f7ef933e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef40cd4c0 con 0x7f7ef40d1500 2024-03-06T07:30:53.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.334+0000 7f7ef933e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ef40cfeb0 con 0x7f7ef406b6c0 2024-03-06T07:30:53.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.334+0000 7f7ef933e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef40cda40 con 0x7f7ef4145760 2024-03-06T07:30:53.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.334+0000 7f7ef37fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ef406b6c0 0x7f7ef406d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.334+0000 7f7ef2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef4145760 0x7f7ef406fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.334+0000 7f7eebfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7ef40d1500 0x7f7ef406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:60182/0 (socket says 172.21.15.59:60182) 2024-03-06T07:30:53.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.334+0000 7f7ef37fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ef406b6c0 0x7f7ef406d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:58094/0 (socket says 172.21.15.59:58094) 2024-03-06T07:30:53.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.334+0000 7f7eebfff640 1 -- 172.21.15.59:0/2902357455 learned_addr learned my addr 172.21.15.59:0/2902357455 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:30:53.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.335+0000 7f7ef2ffd640 1 -- 172.21.15.59:0/2902357455 >> v2:172.21.15.164:3300/0 conn(0x7f7ef406b6c0 msgr2=0x7f7ef406d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.335+0000 7f7ef2ffd640 1 --2- 172.21.15.59:0/2902357455 >> v2:172.21.15.164:3300/0 conn(0x7f7ef406b6c0 0x7f7ef406d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.335+0000 7f7ef2ffd640 1 -- 172.21.15.59:0/2902357455 >> v1:172.21.15.164:6789/0 conn(0x7f7ef40d1500 legacy=0x7f7ef406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.335+0000 7f7ef2ffd640 1 -- 172.21.15.59:0/2902357455 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef40d0340 con 0x7f7ef4145760 2024-03-06T07:30:53.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.335+0000 7f7ef27fc640 1 -- 172.21.15.59:0/2902357455 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738266535 0 0) 0x7f7ef40cd4c0 con 0x7f7ef40d1500 2024-03-06T07:30:53.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.335+0000 7f7ef37fe640 1 --2- 172.21.15.59:0/2902357455 >> v2:172.21.15.164:3300/0 conn(0x7f7ef406b6c0 0x7f7ef406d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:53.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef2ffd640 1 --2- 172.21.15.59:0/2902357455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef4145760 0x7f7ef406fff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40097a0 tx=0x7f7ee402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60e91a5a4bdf57aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef27fc640 1 -- 172.21.15.59:0/2902357455 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4034070 con 0x7f7ef4145760 2024-03-06T07:30:53.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef27fc640 1 -- 172.21.15.59:0/2902357455 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee4030610 con 0x7f7ef4145760 2024-03-06T07:30:53.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef27fc640 1 -- 172.21.15.59:0/2902357455 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40309e0 con 0x7f7ef4145760 2024-03-06T07:30:53.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef933e640 1 -- 172.21.15.59:0/2902357455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef4145760 msgr2=0x7f7ef406fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef933e640 1 --2- 172.21.15.59:0/2902357455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef4145760 0x7f7ef406fff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40097a0 tx=0x7f7ee402b960 comp rx=0 tx=0).stop 2024-03-06T07:30:53.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef933e640 1 -- 172.21.15.59:0/2902357455 shutdown_connections 2024-03-06T07:30:53.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef933e640 1 --2- 172.21.15.59:0/2902357455 >> v2:172.21.15.164:3300/0 conn(0x7f7ef406b6c0 0x7f7ef406d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef933e640 1 --2- 172.21.15.59:0/2902357455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef4145760 0x7f7ef406fff0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef933e640 1 -- 172.21.15.59:0/2902357455 >> 172.21.15.59:0/2902357455 conn(0x7f7ef40d0a10 msgr2=0x7f7ef40d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef933e640 1 -- 172.21.15.59:0/2902357455 shutdown_connections 2024-03-06T07:30:53.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.336+0000 7f7ef933e640 1 -- 172.21.15.59:0/2902357455 wait complete. 2024-03-06T07:30:53.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.337+0000 7f7ef933e640 1 Processor -- start 2024-03-06T07:30:53.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.337+0000 7f7ef933e640 1 -- start start 2024-03-06T07:30:53.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.338+0000 7f7ef933e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef406b6c0 0x7f7ef41ea3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.338+0000 7f7ef933e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ef40d1500 0x7f7ef40cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.338+0000 7f7ef933e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef4145760 0x7f7ef41f7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.338+0000 7f7ef933e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef406be80 con 0x7f7ef406b6c0 2024-03-06T07:30:53.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.338+0000 7f7ef933e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef40cfb40 con 0x7f7ef40d1500 2024-03-06T07:30:53.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.338+0000 7f7ef933e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef406dce0 con 0x7f7ef4145760 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.338+0000 7f7ef2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef406b6c0 0x7f7ef41ea3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.338+0000 7f7ef2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef406b6c0 0x7f7ef41ea3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35768/0 (socket says 172.21.15.59:35768) 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.338+0000 7f7ef2ffd640 1 -- 172.21.15.59:0/1770005255 learned_addr learned my addr 172.21.15.59:0/1770005255 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.338+0000 7f7ef37fe640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef4145760 0x7f7ef41f7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.338+0000 7f7eebfff640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ef40d1500 0x7f7ef40cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7ef2ffd640 1 -- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef4145760 msgr2=0x7f7ef41f7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7ef2ffd640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef4145760 0x7f7ef41f7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7ef2ffd640 1 -- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ef40d1500 msgr2=0x7f7ef40cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7ef2ffd640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ef40d1500 0x7f7ef40cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7ef2ffd640 1 -- 172.21.15.59:0/1770005255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef40b4550 con 0x7f7ef406b6c0 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7eebfff640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ef40d1500 0x7f7ef40cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7ef2ffd640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef406b6c0 0x7f7ef41ea3d0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40098c0 tx=0x7f7ee40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7eeb7fe640 1 -- 172.21.15.59:0/1770005255 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4034040 con 0x7f7ef406b6c0 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7eeb7fe640 1 -- 172.21.15.59:0/1770005255 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee403aaa0 con 0x7f7ef406b6c0 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7ef933e640 1 -- 172.21.15.59:0/1770005255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef40b47d0 con 0x7f7ef406b6c0 2024-03-06T07:30:53.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7eeb7fe640 1 -- 172.21.15.59:0/1770005255 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4006e10 con 0x7f7ef406b6c0 2024-03-06T07:30:53.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.339+0000 7f7ef933e640 1 -- 172.21.15.59:0/1770005255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef40b4d90 con 0x7f7ef406b6c0 2024-03-06T07:30:53.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.340+0000 7f7ef933e640 1 -- 172.21.15.59:0/1770005255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef404f330 con 0x7f7ef406b6c0 2024-03-06T07:30:53.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.340+0000 7f7eeb7fe640 1 -- 172.21.15.59:0/1770005255 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ee404b060 con 0x7f7ef406b6c0 2024-03-06T07:30:53.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.341+0000 7f7eeb7fe640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ed406b4d0 0x7f7ed406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.341+0000 7f7eeb7fe640 1 -- 172.21.15.59:0/1770005255 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7ee40abf10 con 0x7f7ef406b6c0 2024-03-06T07:30:53.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.341+0000 7f7eebfff640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ed406b4d0 0x7f7ed406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.342+0000 7f7eebfff640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ed406b4d0 0x7f7ed406d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40b59f0 tx=0x7f7edc007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.345+0000 7f7eeb7fe640 1 -- 172.21.15.59:0/1770005255 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ef40b4550 con 0x7f7ef406b6c0 2024-03-06T07:30:53.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.540+0000 7f7ef933e640 1 -- 172.21.15.59:0/1770005255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7ef4072b00 con 0x7f7ef406b6c0 2024-03-06T07:30:53.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.541+0000 7f7eeb7fe640 1 -- 172.21.15.59:0/1770005255 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7ee403f070 con 0x7f7ef406b6c0 2024-03-06T07:30:53.543 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:30:53.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 -- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ed406b4d0 msgr2=0x7f7ed406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ed406b4d0 0x7f7ed406d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40b59f0 tx=0x7f7edc007d50 comp rx=0 tx=0).stop 2024-03-06T07:30:53.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 -- 172.21.15.59:0/1770005255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef406b6c0 msgr2=0x7f7ef41ea3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef406b6c0 0x7f7ef41ea3d0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40098c0 tx=0x7f7ee40046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 -- 172.21.15.59:0/1770005255 shutdown_connections 2024-03-06T07:30:53.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ed406b4d0 0x7f7ed406d990 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ef4145760 0x7f7ef41f7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ef40d1500 0x7f7ef40cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 --2- 172.21.15.59:0/1770005255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ef406b6c0 0x7f7ef41ea3d0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 -- 172.21.15.59:0/1770005255 >> 172.21.15.59:0/1770005255 conn(0x7f7ef40d0a10 msgr2=0x7f7ef40b1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 -- 172.21.15.59:0/1770005255 shutdown_connections 2024-03-06T07:30:53.547 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:30:53.543+0000 7f7ef933e640 1 -- 172.21.15.59:0/1770005255 wait complete. 2024-03-06T07:30:53.555 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:30:53.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:53.556 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:53.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[83363]: fuse finished with error 0 and tester_r 0 2024-03-06T07:30:53.582 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:59.683 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:30:59.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:59.683 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:59.698 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:30:59.846 INFO:teuthology.orchestra.run.smithi164.stdout:{"message":"","return_code":0} 2024-03-06T07:30:59.846 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:30:59.847 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json cache drop 2024-03-06T07:30:59.986 INFO:teuthology.orchestra.run.smithi164.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":241,"bytes":58528}},"duration":0.0037915409999999998} 2024-03-06T07:30:59.986 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 58528, "items": 241 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.003791541, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:30:59.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:31:00.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.085+0000 7f21e7f19640 1 Processor -- start 2024-03-06T07:31:00.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.085+0000 7f21e7f19640 1 -- start start 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.086+0000 7f21e7f19640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f21e00b4170 0x7f21e00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.086+0000 7f21e7f19640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e00d25d0 con 0x7f21e00d4760 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.086+0000 7f21e7f19640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e00d2870 con 0x7f21e00d0a80 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.086+0000 7f21e7f19640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e00d4bf0 con 0x7f21e00b4170 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.086+0000 7f21e54b4640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f21e00d4760 0x7f21e00d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:33940/0 (socket says 172.21.15.59:33940) 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.086+0000 7f21e54b4640 1 -- 172.21.15.59:0/3978555609 learned_addr learned my addr 172.21.15.59:0/3978555609 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.086+0000 7f21e5cb5640 1 --2- 172.21.15.59:0/3978555609 >> v2:172.21.15.164:3300/0 conn(0x7f21e00b4170 0x7f21e00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.087+0000 7f21e4cb3640 1 -- 172.21.15.59:0/3978555609 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851609992 0 0) 0x7f21e00d25d0 con 0x7f21e00d4760 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.087+0000 7f21e4cb3640 1 -- 172.21.15.59:0/3978555609 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21cc003520 con 0x7f21e00d4760 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.087+0000 7f21e4cb3640 1 -- 172.21.15.59:0/3978555609 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640702501 0 0) 0x7f21e00d2870 con 0x7f21e00d0a80 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.087+0000 7f21e4cb3640 1 -- 172.21.15.59:0/3978555609 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21e00d25d0 con 0x7f21e00d0a80 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.087+0000 7f21e5cb5640 1 -- 172.21.15.59:0/3978555609 >> v1:172.21.15.164:6789/0 conn(0x7f21e00d0a80 legacy=0x7f21e00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.087+0000 7f21e5cb5640 1 -- 172.21.15.59:0/3978555609 >> v1:172.21.15.59:6789/0 conn(0x7f21e00d4760 legacy=0x7f21e00d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.087+0000 7f21e5cb5640 1 -- 172.21.15.59:0/3978555609 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e00d2b50 con 0x7f21e00b4170 2024-03-06T07:31:00.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.088+0000 7f21e5cb5640 1 --2- 172.21.15.59:0/3978555609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21e00b4170 0x7f21e00b6a90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0097a0 tx=0x7f21dc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96b22262b718ffbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.088+0000 7f21e4cb3640 1 -- 172.21.15.59:0/3978555609 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc034070 con 0x7f21e00b4170 2024-03-06T07:31:00.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.088+0000 7f21e4cb3640 1 -- 172.21.15.59:0/3978555609 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21dc030610 con 0x7f21e00b4170 2024-03-06T07:31:00.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.088+0000 7f21e4cb3640 1 -- 172.21.15.59:0/3978555609 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21dc0309e0 con 0x7f21e00b4170 2024-03-06T07:31:00.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.089+0000 7f21e7f19640 1 -- 172.21.15.59:0/3978555609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21e00b4170 msgr2=0x7f21e00b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.089+0000 7f21e7f19640 1 --2- 172.21.15.59:0/3978555609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21e00b4170 0x7f21e00b6a90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0097a0 tx=0x7f21dc02b960 comp rx=0 tx=0).stop 2024-03-06T07:31:00.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.089+0000 7f21e7f19640 1 -- 172.21.15.59:0/3978555609 shutdown_connections 2024-03-06T07:31:00.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.089+0000 7f21e7f19640 1 --2- 172.21.15.59:0/3978555609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21e00b4170 0x7f21e00b6a90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.089+0000 7f21e7f19640 1 -- 172.21.15.59:0/3978555609 >> 172.21.15.59:0/3978555609 conn(0x7f21e00d3c40 msgr2=0x7f21e00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.089+0000 7f21e7f19640 1 -- 172.21.15.59:0/3978555609 shutdown_connections 2024-03-06T07:31:00.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.089+0000 7f21e7f19640 1 -- 172.21.15.59:0/3978555609 wait complete. 2024-03-06T07:31:00.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.091+0000 7f21e7f19640 1 Processor -- start 2024-03-06T07:31:00.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.091+0000 7f21e7f19640 1 -- start start 2024-03-06T07:31:00.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e7f19640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21e00b4170 0x7f21e01571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e7f19640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21e00d0a80 0x7f21e0157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e7f19640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21e00ca420 0x7f21e015a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e7f19640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00b4dd0 con 0x7f21e00d0a80 2024-03-06T07:31:00.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e7f19640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00ce9e0 con 0x7f21e00ca420 2024-03-06T07:31:00.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e7f19640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00b4f50 con 0x7f21e00b4170 2024-03-06T07:31:00.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e54b4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21e00d0a80 0x7f21e0157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e54b4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21e00d0a80 0x7f21e0157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50662/0 (socket says 172.21.15.59:50662) 2024-03-06T07:31:00.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e54b4640 1 -- 172.21.15.59:0/495697490 learned_addr learned my addr 172.21.15.59:0/495697490 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:00.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e64b6640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21e00ca420 0x7f21e015a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e5cb5640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21e00b4170 0x7f21e01571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e54b4640 1 -- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21e00b4170 msgr2=0x7f21e01571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e54b4640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21e00b4170 0x7f21e01571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e54b4640 1 -- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21e00ca420 msgr2=0x7f21e015a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e54b4640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21e00ca420 0x7f21e015a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e54b4640 1 -- 172.21.15.59:0/495697490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e0158850 con 0x7f21e00d0a80 2024-03-06T07:31:00.096 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.092+0000 7f21e54b4640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21e00d0a80 0x7f21e0157df0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f21d000c410 tx=0x7f21d000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.093+0000 7f21d67fc640 1 -- 172.21.15.59:0/495697490 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0017070 con 0x7f21e00d0a80 2024-03-06T07:31:00.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.093+0000 7f21d67fc640 1 -- 172.21.15.59:0/495697490 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d0005ce0 con 0x7f21e00d0a80 2024-03-06T07:31:00.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.093+0000 7f21e7f19640 1 -- 172.21.15.59:0/495697490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e0158b30 con 0x7f21e00d0a80 2024-03-06T07:31:00.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.093+0000 7f21d67fc640 1 -- 172.21.15.59:0/495697490 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d00072d0 con 0x7f21e00d0a80 2024-03-06T07:31:00.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.093+0000 7f21e7f19640 1 -- 172.21.15.59:0/495697490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e01590f0 con 0x7f21e00d0a80 2024-03-06T07:31:00.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.094+0000 7f21e7f19640 1 -- 172.21.15.59:0/495697490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e015b620 con 0x7f21e00d0a80 2024-03-06T07:31:00.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.094+0000 7f21d67fc640 1 -- 172.21.15.59:0/495697490 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21d001a070 con 0x7f21e00d0a80 2024-03-06T07:31:00.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.094+0000 7f21d67fc640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f21c406b4d0 0x7f21c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.095+0000 7f21d67fc640 1 -- 172.21.15.59:0/495697490 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f21d0089520 con 0x7f21e00d0a80 2024-03-06T07:31:00.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.095+0000 7f21e5cb5640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f21c406b4d0 0x7f21c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.095+0000 7f21e5cb5640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f21c406b4d0 0x7f21c406d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f21dc036000 tx=0x7f21dc03b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.099+0000 7f21d67fc640 1 -- 172.21.15.59:0/495697490 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21d0057d30 con 0x7f21e00d0a80 2024-03-06T07:31:00.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.321+0000 7f21e7f19640 1 -- 172.21.15.59:0/495697490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f21e004f330 con 0x7f21e00d0a80 2024-03-06T07:31:00.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.321+0000 7f21d67fc640 1 -- 172.21.15.59:0/495697490 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v103) v1 ==== 53+0+73 (secure 0 0 0) 0x7f21d005bdd0 con 0x7f21e00d0a80 2024-03-06T07:31:00.323 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.323+0000 7f21e7f19640 1 -- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f21c406b4d0 msgr2=0x7f21c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.323+0000 7f21e7f19640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f21c406b4d0 0x7f21c406d990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f21dc036000 tx=0x7f21dc03b040 comp rx=0 tx=0).stop 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.323+0000 7f21e7f19640 1 -- 172.21.15.59:0/495697490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21e00d0a80 msgr2=0x7f21e0157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.323+0000 7f21e7f19640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21e00d0a80 0x7f21e0157df0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f21d000c410 tx=0x7f21d000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.323+0000 7f21e7f19640 1 -- 172.21.15.59:0/495697490 shutdown_connections 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.323+0000 7f21e7f19640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f21c406b4d0 0x7f21c406d990 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.323+0000 7f21e7f19640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21e00ca420 0x7f21e015a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.323+0000 7f21e7f19640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21e00d0a80 0x7f21e0157df0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.323+0000 7f21e7f19640 1 --2- 172.21.15.59:0/495697490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21e00b4170 0x7f21e01571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.323+0000 7f21e7f19640 1 -- 172.21.15.59:0/495697490 >> 172.21.15.59:0/495697490 conn(0x7f21e00d3c40 msgr2=0x7f21e006b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.324+0000 7f21e7f19640 1 -- 172.21.15.59:0/495697490 shutdown_connections 2024-03-06T07:31:00.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:00.324+0000 7f21e7f19640 1 -- 172.21.15.59:0/495697490 wait complete. 2024-03-06T07:31:00.336 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:31:00.336 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:31:00.336 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2024-03-06T07:31:00.336 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:31:00.336 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:00.337 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:31:00.337 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:31:00.337 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:31:00.337 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:00.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:00.337 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42118sec preferred_lft 42118sec 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::e4fa:bff:fe34:b46d/64 scope link 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi059.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 82:f5:c6:de:63:7f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::80f5:c6ff:fede:637f/64 scope link 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi059.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 06:32:cd:6e:a7:e1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::432:cdff:fe6e:a7e1/64 scope link 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:00.355 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:31:00.418 INFO:teuthology.orchestra.run.smithi059.stdout:1 2024-03-06T07:31:00.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:00.420 DEBUG:teuthology.orchestra.run.smithi059:> route 2024-03-06T07:31:00.476 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2024-03-06T07:31:00.476 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:31:00.477 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:31:00.477 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:31:00.477 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:31:00.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:00.477 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2024-03-06T07:31:00.477 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2024-03-06T07:31:00.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:31:00.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:31:00.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:31:00.477 DEBUG:teuthology.orchestra.run.smithi059:> ') 2024-03-06T07:31:00.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:00.571 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:31:00.586 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:31:00.586 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:31:00.586 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:00.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:00.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2024-03-06T07:31:00.653 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:31:00.653 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:00.653 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:00.708 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:00.709 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:00.709 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:00.762 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:31:00.829 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:31:00.884 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:31:00.885 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:00.886 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:00.887 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:00.887 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:00.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:00.887 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:31:00.938 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:31:00.938 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:31:00.981 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe fuse 2024-03-06T07:31:01.008 DEBUG:teuthology.orchestra.run.smithi059:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:31:01.063 INFO:teuthology.orchestra.run.smithi059.stdout:/proc 2024-03-06T07:31:01.063 INFO:teuthology.orchestra.run.smithi059.stdout:/sys 2024-03-06T07:31:01.063 INFO:teuthology.orchestra.run.smithi059.stdout:/dev 2024-03-06T07:31:01.063 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/security 2024-03-06T07:31:01.063 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/shm 2024-03-06T07:31:01.063 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/pts 2024-03-06T07:31:01.063 INFO:teuthology.orchestra.run.smithi059.stdout:/run 2024-03-06T07:31:01.063 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/cgroup 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/pstore 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/bpf 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/ 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/selinux 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/hugepages 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/mqueue 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/debug 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/tracing 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/fs/fuse/connections 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/sys/kernel/config 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:31:01.064 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:31:01.065 INFO:teuthology.orchestra.run.smithi059.stdout:/run/user/1000 2024-03-06T07:31:01.065 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph 2024-03-06T07:31:01.065 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:31:01.065 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:31:01.065 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:31:01.065 INFO:teuthology.orchestra.run.smithi059.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:31:01.065 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns 2024-03-06T07:31:01.065 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:01.065 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:01.065 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:01.065 INFO:teuthology.orchestra.run.smithi059.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:01.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:01.066 DEBUG:teuthology.orchestra.run.smithi059:> ls /sys/fs/fuse/connections 2024-03-06T07:31:01.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:2024-03-06T07:31:01.066+0000 7f3174a43480 -1 init, newargv = 0x562705c62e30 newargc=15 2024-03-06T07:31:01.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[85128]: starting ceph client 2024-03-06T07:31:01.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[85128]: starting fuse 2024-03-06T07:31:01.082 INFO:teuthology.orchestra.run.smithi059.stdout:42 2024-03-06T07:31:01.083 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> import glob 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> import re 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> import os 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> import subprocess 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> def _find_admin_socket(client_name): 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> files = glob.glob(asok_path) 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> # Given a non-glob path, it better be there 2024-03-06T07:31:01.083 DEBUG:teuthology.orchestra.run.smithi059:> if "*" not in asok_path: 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> assert(len(files) == 1) 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> return files[0] 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> for f in files: 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> contents = proc_f.read() 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> if mountpoint in contents: 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> return f 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> print(_find_admin_socket("client.0")) 2024-03-06T07:31:01.084 DEBUG:teuthology.orchestra.run.smithi059:> ' 2024-03-06T07:31:01.203 INFO:teuthology.orchestra.run.smithi059.stdout:/var/run/ceph/ceph-client.0.85128.asok 2024-03-06T07:31:01.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:31:01.213 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.85128.asok 2024-03-06T07:31:01.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:01.214 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.85128.asok status 2024-03-06T07:31:01.350 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:31:01.350 INFO:teuthology.orchestra.run.smithi059.stdout: "metadata": { 2024-03-06T07:31:01.350 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:31:01.350 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:31:01.350 INFO:teuthology.orchestra.run.smithi059.stdout: "entity_id": "0", 2024-03-06T07:31:01.350 INFO:teuthology.orchestra.run.smithi059.stdout: "hostname": "smithi059", 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "pid": "85128", 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "root": "/" 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_count": 0, 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "dentry_pinned_count": 0, 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "id": 9672, 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "inst": { 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "name": { 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "client", 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "num": 9672 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:31:01.351 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1004822366 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: } 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": { 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "v1", 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: "nonce": 1004822366 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: "inst_str": "client.9672 192.168.0.1:0/1004822366", 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: "addr_str": "192.168.0.1:0/1004822366", 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: "inode_count": 1, 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: "mds_epoch": 103, 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch": 143, 2024-03-06T07:31:01.353 INFO:teuthology.orchestra.run.smithi059.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:31:01.353 INFO:teuthology.orchestra.run.smithi059.stdout: "blocklisted": false, 2024-03-06T07:31:01.353 INFO:teuthology.orchestra.run.smithi059.stdout: "fs_name": "cephfs" 2024-03-06T07:31:01.353 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:31:01.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:01.359 DEBUG:teuthology.orchestra.run.smithi059:> stat --file-system '--printf=%T 2024-03-06T07:31:01.359 DEBUG:teuthology.orchestra.run.smithi059:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:01.373 INFO:teuthology.orchestra.run.smithi059.stdout:fuseblk 2024-03-06T07:31:01.373 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:01.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:01.374 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:01.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:01.441 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/original) 2024-03-06T07:31:01.501 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:31:01.501 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:01.556 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:01.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.696+0000 7f5db186f640 1 Processor -- start 2024-03-06T07:31:01.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.696+0000 7f5db186f640 1 -- start start 2024-03-06T07:31:01.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5db186f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac0b4f70 0x7f5dac0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5db186f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5dac066d90 0x7f5dac0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5db186f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dac0b5890 con 0x7f5dac06c6d0 2024-03-06T07:31:01.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5db186f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dac15e9f0 con 0x7f5dac066d90 2024-03-06T07:31:01.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5db186f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac0cda10 con 0x7f5dac0b4f70 2024-03-06T07:31:01.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5daaffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac0b4f70 0x7f5dac0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5dab7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5dac066d90 0x7f5dac0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5daa7fc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5dac06c6d0 0x7f5dac15ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:51730/0 (socket says 172.21.15.59:51730) 2024-03-06T07:31:01.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5daa7fc640 1 -- 172.21.15.59:0/597178858 learned_addr learned my addr 172.21.15.59:0/597178858 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:01.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5daaffd640 1 -- 172.21.15.59:0/597178858 >> v2:172.21.15.164:3300/0 conn(0x7f5dac066d90 msgr2=0x7f5dac0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5daaffd640 1 --2- 172.21.15.59:0/597178858 >> v2:172.21.15.164:3300/0 conn(0x7f5dac066d90 0x7f5dac0ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5daaffd640 1 -- 172.21.15.59:0/597178858 >> v1:172.21.15.164:6789/0 conn(0x7f5dac06c6d0 legacy=0x7f5dac15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5daaffd640 1 -- 172.21.15.59:0/597178858 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac0cd450 con 0x7f5dac0b4f70 2024-03-06T07:31:01.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.697+0000 7f5dab7fe640 1 --2- 172.21.15.59:0/597178858 >> v2:172.21.15.164:3300/0 conn(0x7f5dac066d90 0x7f5dac0ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.698+0000 7f5daaffd640 1 --2- 172.21.15.59:0/597178858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac0b4f70 0x7f5dac0b5350 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5d940097a0 tx=0x7f5d9402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b525f118475a0196 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.698+0000 7f5da9ffb640 1 -- 172.21.15.59:0/597178858 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94035020 con 0x7f5dac0b4f70 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.698+0000 7f5da9ffb640 1 -- 172.21.15.59:0/597178858 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d94031830 con 0x7f5dac0b4f70 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.698+0000 7f5da9ffb640 1 -- 172.21.15.59:0/597178858 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94031c00 con 0x7f5dac0b4f70 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.699+0000 7f5db186f640 1 -- 172.21.15.59:0/597178858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac0b4f70 msgr2=0x7f5dac0b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.699+0000 7f5db186f640 1 --2- 172.21.15.59:0/597178858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac0b4f70 0x7f5dac0b5350 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5d940097a0 tx=0x7f5d9402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.699+0000 7f5db186f640 1 -- 172.21.15.59:0/597178858 shutdown_connections 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.699+0000 7f5db186f640 1 --2- 172.21.15.59:0/597178858 >> v2:172.21.15.164:3300/0 conn(0x7f5dac066d90 0x7f5dac0ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.699+0000 7f5db186f640 1 --2- 172.21.15.59:0/597178858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac0b4f70 0x7f5dac0b5350 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.699+0000 7f5db186f640 1 -- 172.21.15.59:0/597178858 >> 172.21.15.59:0/597178858 conn(0x7f5dac0724d0 msgr2=0x7f5dac0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.699+0000 7f5db186f640 1 -- 172.21.15.59:0/597178858 shutdown_connections 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.699+0000 7f5db186f640 1 -- 172.21.15.59:0/597178858 wait complete. 2024-03-06T07:31:01.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.700+0000 7f5db186f640 1 Processor -- start 2024-03-06T07:31:01.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.700+0000 7f5db186f640 1 -- start start 2024-03-06T07:31:01.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5db186f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac066d90 0x7f5dac1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5db186f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac0b4f70 0x7f5dac1e8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5db186f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dac15d200 0x7f5dac0ae140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5db186f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac0cd090 con 0x7f5dac0b4f70 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5db186f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac15e680 con 0x7f5dac15d200 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5db186f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac0cd210 con 0x7f5dac066d90 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5daa7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac0b4f70 0x7f5dac1e8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5daa7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac0b4f70 0x7f5dac1e8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50676/0 (socket says 172.21.15.59:50676) 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5daa7fc640 1 -- 172.21.15.59:0/1875870060 learned_addr learned my addr 172.21.15.59:0/1875870060 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5daaffd640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac066d90 0x7f5dac1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5dab7fe640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dac15d200 0x7f5dac0ae140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5daaffd640 1 -- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dac15d200 msgr2=0x7f5dac0ae140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5daaffd640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dac15d200 0x7f5dac0ae140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5daaffd640 1 -- 172.21.15.59:0/1875870060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac0b4f70 msgr2=0x7f5dac1e8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5daaffd640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac0b4f70 0x7f5dac1e8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.701+0000 7f5daaffd640 1 -- 172.21.15.59:0/1875870060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac1e8c70 con 0x7f5dac066d90 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.702+0000 7f5daa7fc640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac0b4f70 0x7f5dac1e8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:01.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.702+0000 7f5daaffd640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac066d90 0x7f5dac1558c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5dac15eb70 tx=0x7f5d940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.702+0000 7f5da3fff640 1 -- 172.21.15.59:0/1875870060 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94035040 con 0x7f5dac066d90 2024-03-06T07:31:01.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.702+0000 7f5da3fff640 1 -- 172.21.15.59:0/1875870060 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9403f070 con 0x7f5dac066d90 2024-03-06T07:31:01.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.702+0000 7f5db186f640 1 -- 172.21.15.59:0/1875870060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dac1e8f50 con 0x7f5dac066d90 2024-03-06T07:31:01.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.702+0000 7f5da3fff640 1 -- 172.21.15.59:0/1875870060 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9403baa0 con 0x7f5dac066d90 2024-03-06T07:31:01.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.702+0000 7f5db186f640 1 -- 172.21.15.59:0/1875870060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dac1e9510 con 0x7f5dac066d90 2024-03-06T07:31:01.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.703+0000 7f5db186f640 1 -- 172.21.15.59:0/1875870060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dac04f330 con 0x7f5dac066d90 2024-03-06T07:31:01.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.703+0000 7f5da3fff640 1 -- 172.21.15.59:0/1875870060 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d94038020 con 0x7f5dac066d90 2024-03-06T07:31:01.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.704+0000 7f5da3fff640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d8c06b4d0 0x7f5d8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.704+0000 7f5da3fff640 1 -- 172.21.15.59:0/1875870060 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5d940abbe0 con 0x7f5dac066d90 2024-03-06T07:31:01.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.704+0000 7f5daa7fc640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d8c06b4d0 0x7f5d8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.704+0000 7f5daa7fc640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d8c06b4d0 0x7f5d8c06d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0af3b0 tx=0x7f5d9c007f10 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.709+0000 7f5da3fff640 1 -- 172.21.15.59:0/1875870060 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d94031800 con 0x7f5dac066d90 2024-03-06T07:31:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.948+0000 7f5db186f640 1 -- 172.21.15.59:0/1875870060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dac161610 con 0x7f5dac066d90 2024-03-06T07:31:01.951 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.950+0000 7f5da3fff640 1 -- 172.21.15.59:0/1875870060 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f5d9407a400 con 0x7f5dac066d90 2024-03-06T07:31:01.952 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:01.952 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":143,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:39.494279+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:11.712614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:12.406781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.22","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 -- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d8c06b4d0 msgr2=0x7f5d8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d8c06b4d0 0x7f5d8c06d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0af3b0 tx=0x7f5d9c007f10 comp rx=0 tx=0).stop 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 -- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac066d90 msgr2=0x7f5dac1558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac066d90 0x7f5dac1558c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5dac15eb70 tx=0x7f5d940046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 -- 172.21.15.59:0/1875870060 shutdown_connections 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5d8c06b4d0 0x7f5d8c06d990 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dac15d200 0x7f5dac0ae140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5dac0b4f70 0x7f5dac1e8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 --2- 172.21.15.59:0/1875870060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dac066d90 0x7f5dac1558c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 -- 172.21.15.59:0/1875870060 >> 172.21.15.59:0/1875870060 conn(0x7f5dac0724d0 msgr2=0x7f5dac0728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 -- 172.21.15.59:0/1875870060 shutdown_connections 2024-03-06T07:31:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:01.952+0000 7f5db186f640 1 -- 172.21.15.59:0/1875870060 wait complete. 2024-03-06T07:31:01.965 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:11.712614+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:12.406781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:01.965 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:02.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.067+0000 7f7d501f7640 1 Processor -- start 2024-03-06T07:31:02.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.067+0000 7f7d501f7640 1 -- start start 2024-03-06T07:31:02.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d501f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d481614d0 0x7f7d480d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d501f7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7d480d2480 0x7f7d480b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d501f7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d481618b0 con 0x7f7d480d1910 2024-03-06T07:31:02.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d501f7640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480cf400 con 0x7f7d480d2480 2024-03-06T07:31:02.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d501f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480cf6a0 con 0x7f7d481614d0 2024-03-06T07:31:02.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d4e794640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7d480d2480 0x7f7d480b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d4e794640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7d480d2480 0x7f7d480b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50686/0 (socket says 172.21.15.59:50686) 2024-03-06T07:31:02.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d4df93640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d481614d0 0x7f7d480d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d4d792640 1 --1- 172.21.15.59:0/3539751395 >> v1:172.21.15.164:6789/0 conn(0x7f7d480d1910 0x7f7d480d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:51742/0 (socket says 172.21.15.59:51742) 2024-03-06T07:31:02.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d4e794640 1 -- 172.21.15.59:0/3539751395 learned_addr learned my addr 172.21.15.59:0/3539751395 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:02.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d4e794640 1 -- 172.21.15.59:0/3539751395 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d481614d0 msgr2=0x7f7d480d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d4e794640 1 --2- 172.21.15.59:0/3539751395 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d481614d0 0x7f7d480d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d4e794640 1 -- 172.21.15.59:0/3539751395 >> v1:172.21.15.164:6789/0 conn(0x7f7d480d1910 legacy=0x7f7d480d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d4cf91640 1 -- 172.21.15.59:0/3539751395 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023178409 0 0) 0x7f7d481618b0 con 0x7f7d480d1910 2024-03-06T07:31:02.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.068+0000 7f7d4e794640 1 -- 172.21.15.59:0/3539751395 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d480cf980 con 0x7f7d480d2480 2024-03-06T07:31:02.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.069+0000 7f7d4e794640 1 --2- 172.21.15.59:0/3539751395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d480d2480 0x7f7d480b0e30 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009e50 tx=0x7f7d4402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5129ec9dea349986 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.069+0000 7f7d4cf91640 1 -- 172.21.15.59:0/3539751395 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4402cb00 con 0x7f7d480d2480 2024-03-06T07:31:02.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.069+0000 7f7d4cf91640 1 -- 172.21.15.59:0/3539751395 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4402cca0 con 0x7f7d480d2480 2024-03-06T07:31:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.069+0000 7f7d4cf91640 1 -- 172.21.15.59:0/3539751395 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44034610 con 0x7f7d480d2480 2024-03-06T07:31:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.070+0000 7f7d501f7640 1 -- 172.21.15.59:0/3539751395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d480d2480 msgr2=0x7f7d480b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.070+0000 7f7d501f7640 1 --2- 172.21.15.59:0/3539751395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d480d2480 0x7f7d480b0e30 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009e50 tx=0x7f7d4402bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.070+0000 7f7d501f7640 1 -- 172.21.15.59:0/3539751395 shutdown_connections 2024-03-06T07:31:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.070+0000 7f7d501f7640 1 --2- 172.21.15.59:0/3539751395 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d480d2480 0x7f7d480b0e30 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.070+0000 7f7d501f7640 1 --2- 172.21.15.59:0/3539751395 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d481614d0 0x7f7d480d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.070+0000 7f7d501f7640 1 -- 172.21.15.59:0/3539751395 >> 172.21.15.59:0/3539751395 conn(0x7f7d480d3d40 msgr2=0x7f7d480d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.070+0000 7f7d501f7640 1 -- 172.21.15.59:0/3539751395 shutdown_connections 2024-03-06T07:31:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.070+0000 7f7d501f7640 1 -- 172.21.15.59:0/3539751395 wait complete. 2024-03-06T07:31:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.072+0000 7f7d501f7640 1 Processor -- start 2024-03-06T07:31:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.072+0000 7f7d501f7640 1 -- start start 2024-03-06T07:31:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.072+0000 7f7d501f7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d480d1910 0x7f7d480c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.072+0000 7f7d501f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d480d2480 0x7f7d48154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d501f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d481614d0 0x7f7d480b8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d501f7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480b1370 con 0x7f7d480d1910 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d501f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480b0910 con 0x7f7d480d2480 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d501f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480b14f0 con 0x7f7d481614d0 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d4df93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d480d1910 0x7f7d480c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d4df93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d480d1910 0x7f7d480c84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50690/0 (socket says 172.21.15.59:50690) 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d4df93640 1 -- 172.21.15.59:0/43070094 learned_addr learned my addr 172.21.15.59:0/43070094 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d4d792640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d480d2480 0x7f7d48154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d4e794640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d481614d0 0x7f7d480b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d4df93640 1 -- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d481614d0 msgr2=0x7f7d480b8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d4df93640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d481614d0 0x7f7d480b8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d4df93640 1 -- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d480d2480 msgr2=0x7f7d48154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d4df93640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d480d2480 0x7f7d48154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d4df93640 1 -- 172.21.15.59:0/43070094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d480b8500 con 0x7f7d480d1910 2024-03-06T07:31:02.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d4df93640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d480d1910 0x7f7d480c84c0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000e870 tx=0x7f7d4000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d3e7fc640 1 -- 172.21.15.59:0/43070094 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40017070 con 0x7f7d480d1910 2024-03-06T07:31:02.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d3e7fc640 1 -- 172.21.15.59:0/43070094 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4000c940 con 0x7f7d480d1910 2024-03-06T07:31:02.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d501f7640 1 -- 172.21.15.59:0/43070094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d480b5ed0 con 0x7f7d480d1910 2024-03-06T07:31:02.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d3e7fc640 1 -- 172.21.15.59:0/43070094 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4000cd30 con 0x7f7d480d1910 2024-03-06T07:31:02.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.073+0000 7f7d501f7640 1 -- 172.21.15.59:0/43070094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d48154d80 con 0x7f7d480d1910 2024-03-06T07:31:02.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.074+0000 7f7d501f7640 1 -- 172.21.15.59:0/43070094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d480d1e70 con 0x7f7d480d1910 2024-03-06T07:31:02.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.075+0000 7f7d3e7fc640 1 -- 172.21.15.59:0/43070094 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d4001a070 con 0x7f7d480d1910 2024-03-06T07:31:02.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.075+0000 7f7d3e7fc640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d2c06b500 0x7f7d2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.075+0000 7f7d3e7fc640 1 -- 172.21.15.59:0/43070094 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7d40089fe0 con 0x7f7d480d1910 2024-03-06T07:31:02.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.075+0000 7f7d4d792640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d2c06b500 0x7f7d2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.076+0000 7f7d4d792640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d2c06b500 0x7f7d2c06d9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f7d480c77e0 tx=0x7f7d38007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.080+0000 7f7d3e7fc640 1 -- 172.21.15.59:0/43070094 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d400587f0 con 0x7f7d480d1910 2024-03-06T07:31:02.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.263+0000 7f7d501f7640 1 -- 172.21.15.59:0/43070094 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d48068ff0 con 0x7f7d480d1910 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.263+0000 7f7d3e7fc640 1 -- 172.21.15.59:0/43070094 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d4005c890 con 0x7f7d480d1910 2024-03-06T07:31:02.265 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 -- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d2c06b500 msgr2=0x7f7d2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d2c06b500 0x7f7d2c06d9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f7d480c77e0 tx=0x7f7d38007d50 comp rx=0 tx=0).stop 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 -- 172.21.15.59:0/43070094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d480d1910 msgr2=0x7f7d480c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d480d1910 0x7f7d480c84c0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000e870 tx=0x7f7d4000ed30 comp rx=0 tx=0).stop 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 -- 172.21.15.59:0/43070094 shutdown_connections 2024-03-06T07:31:02.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d2c06b500 0x7f7d2c06d9c0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d481614d0 0x7f7d480b8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d480d2480 0x7f7d48154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 --2- 172.21.15.59:0/43070094 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d480d1910 0x7f7d480c84c0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 -- 172.21.15.59:0/43070094 >> 172.21.15.59:0/43070094 conn(0x7f7d480d3d40 msgr2=0x7f7d480cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 -- 172.21.15.59:0/43070094 shutdown_connections 2024-03-06T07:31:02.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.265+0000 7f7d501f7640 1 -- 172.21.15.59:0/43070094 wait complete. 2024-03-06T07:31:02.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:02.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.379+0000 7f6cf3ea4640 1 Processor -- start 2024-03-06T07:31:02.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.379+0000 7f6cf3ea4640 1 -- start start 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.380+0000 7f6cf3ea4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6cec072550 0x7f6cec072930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.380+0000 7f6cf3ea4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cec15c110 0x7f6cec0b58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.380+0000 7f6cf3ea4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cec072e70 con 0x7f6cec0731d0 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.380+0000 7f6cf3ea4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cec06c4b0 con 0x7f6cec072550 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.380+0000 7f6cf3ea4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec06c790 con 0x7f6cec15c110 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.380+0000 7f6cf2441640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6cec0731d0 0x7f6cec0cb160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:33946/0 (socket says 172.21.15.59:33946) 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.380+0000 7f6cf2441640 1 -- 172.21.15.59:0/3457211161 learned_addr learned my addr 172.21.15.59:0/3457211161 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.380+0000 7f6cf1c40640 1 --2- 172.21.15.59:0/3457211161 >> v2:172.21.15.164:3300/0 conn(0x7f6cec072550 0x7f6cec072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.380+0000 7f6cf143f640 1 --2- 172.21.15.59:0/3457211161 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cec15c110 0x7f6cec0b58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.381+0000 7f6cf0c3e640 1 -- 172.21.15.59:0/3457211161 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334478224 0 0) 0x7f6cec072e70 con 0x7f6cec0731d0 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.381+0000 7f6cf0c3e640 1 -- 172.21.15.59:0/3457211161 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cd8003520 con 0x7f6cec0731d0 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.381+0000 7f6cf1c40640 1 -- 172.21.15.59:0/3457211161 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cec15c110 msgr2=0x7f6cec0b58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.381+0000 7f6cf1c40640 1 --2- 172.21.15.59:0/3457211161 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cec15c110 0x7f6cec0b58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.381+0000 7f6cf1c40640 1 -- 172.21.15.59:0/3457211161 >> v1:172.21.15.59:6789/0 conn(0x7f6cec0731d0 legacy=0x7f6cec0cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.381+0000 7f6cf1c40640 1 -- 172.21.15.59:0/3457211161 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0cac40 con 0x7f6cec072550 2024-03-06T07:31:02.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.381+0000 7f6cf143f640 1 --2- 172.21.15.59:0/3457211161 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cec15c110 0x7f6cec0b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf1c40640 1 --2- 172.21.15.59:0/3457211161 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cec072550 0x7f6cec072930 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80097a0 tx=0x7f6ce802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12e6396c45ef73e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf0c3e640 1 -- 172.21.15.59:0/3457211161 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8035020 con 0x7f6cec072550 2024-03-06T07:31:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf0c3e640 1 -- 172.21.15.59:0/3457211161 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce8031810 con 0x7f6cec072550 2024-03-06T07:31:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf0c3e640 1 -- 172.21.15.59:0/3457211161 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8031be0 con 0x7f6cec072550 2024-03-06T07:31:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/3457211161 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cec072550 msgr2=0x7f6cec072930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf3ea4640 1 --2- 172.21.15.59:0/3457211161 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cec072550 0x7f6cec072930 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80097a0 tx=0x7f6ce802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/3457211161 shutdown_connections 2024-03-06T07:31:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf3ea4640 1 --2- 172.21.15.59:0/3457211161 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cec15c110 0x7f6cec0b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf3ea4640 1 --2- 172.21.15.59:0/3457211161 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cec072550 0x7f6cec072930 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/3457211161 >> 172.21.15.59:0/3457211161 conn(0x7f6cec0d0120 msgr2=0x7f6cec0d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/3457211161 shutdown_connections 2024-03-06T07:31:02.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.382+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/3457211161 wait complete. 2024-03-06T07:31:02.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.383+0000 7f6cf3ea4640 1 Processor -- start 2024-03-06T07:31:02.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf3ea4640 1 -- start start 2024-03-06T07:31:02.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf3ea4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cec072550 0x7f6cec0c7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf3ea4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cec0731d0 0x7f6cec0c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf3ea4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cec15c110 0x7f6cec0ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf3ea4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec06c250 con 0x7f6cec15c110 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf3ea4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec06c0d0 con 0x7f6cec072550 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf3ea4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0cb870 con 0x7f6cec0731d0 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf2441640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cec15c110 0x7f6cec0ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf2441640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cec15c110 0x7f6cec0ad2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50692/0 (socket says 172.21.15.59:50692) 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf2441640 1 -- 172.21.15.59:0/1963729138 learned_addr learned my addr 172.21.15.59:0/1963729138 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf1c40640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cec072550 0x7f6cec0c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.384+0000 7f6cf143f640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cec0731d0 0x7f6cec0c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.385+0000 7f6cf2441640 1 -- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cec0731d0 msgr2=0x7f6cec0c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.385+0000 7f6cf2441640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cec0731d0 0x7f6cec0c76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.385+0000 7f6cf2441640 1 -- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cec072550 msgr2=0x7f6cec0c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.385+0000 7f6cf2441640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cec072550 0x7f6cec0c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.385+0000 7f6cf2441640 1 -- 172.21.15.59:0/1963729138 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0ae3d0 con 0x7f6cec15c110 2024-03-06T07:31:02.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.385+0000 7f6cf2441640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cec15c110 0x7f6cec0ad2c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400cbb0 tx=0x7f6ce400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.385+0000 7f6cde7fc640 1 -- 172.21.15.59:0/1963729138 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4017070 con 0x7f6cec15c110 2024-03-06T07:31:02.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.385+0000 7f6cde7fc640 1 -- 172.21.15.59:0/1963729138 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce400ad40 con 0x7f6cec15c110 2024-03-06T07:31:02.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.385+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/1963729138 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec0ae6b0 con 0x7f6cec15c110 2024-03-06T07:31:02.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.385+0000 7f6cde7fc640 1 -- 172.21.15.59:0/1963729138 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4012600 con 0x7f6cec15c110 2024-03-06T07:31:02.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.385+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/1963729138 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cec0af500 con 0x7f6cec15c110 2024-03-06T07:31:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.386+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/1963729138 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cec069f10 con 0x7f6cec15c110 2024-03-06T07:31:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.386+0000 7f6cde7fc640 1 -- 172.21.15.59:0/1963729138 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ce401a070 con 0x7f6cec15c110 2024-03-06T07:31:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.387+0000 7f6cde7fc640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6cd006b4d0 0x7f6cd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.387+0000 7f6cde7fc640 1 -- 172.21.15.59:0/1963729138 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6ce4089ee0 con 0x7f6cec15c110 2024-03-06T07:31:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.387+0000 7f6cf1c40640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6cd006b4d0 0x7f6cd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.387+0000 7f6cf1c40640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6cd006b4d0 0x7f6cd006d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8036000 tx=0x7f6ce8008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.391+0000 7f6cde7fc640 1 -- 172.21.15.59:0/1963729138 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6cec069f10 con 0x7f6cec15c110 2024-03-06T07:31:02.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.571+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/1963729138 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6cec09e3d0 con 0x7f6cec15c110 2024-03-06T07:31:02.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.571+0000 7f6cde7fc640 1 -- 172.21.15.59:0/1963729138 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6ce4075e20 con 0x7f6cec15c110 2024-03-06T07:31:02.573 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:31:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6cd006b4d0 msgr2=0x7f6cd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6cd006b4d0 0x7f6cd006d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8036000 tx=0x7f6ce8008030 comp rx=0 tx=0).stop 2024-03-06T07:31:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/1963729138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cec15c110 msgr2=0x7f6cec0ad2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cec15c110 0x7f6cec0ad2c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400cbb0 tx=0x7f6ce400a430 comp rx=0 tx=0).stop 2024-03-06T07:31:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/1963729138 shutdown_connections 2024-03-06T07:31:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6cd006b4d0 0x7f6cd006d990 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cec15c110 0x7f6cec0ad2c0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cec0731d0 0x7f6cec0c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 --2- 172.21.15.59:0/1963729138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cec072550 0x7f6cec0c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/1963729138 >> 172.21.15.59:0/1963729138 conn(0x7f6cec0d0120 msgr2=0x7f6cec0ce0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/1963729138 shutdown_connections 2024-03-06T07:31:02.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.573+0000 7f6cf3ea4640 1 -- 172.21.15.59:0/1963729138 wait complete. 2024-03-06T07:31:02.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:02.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.691+0000 7f14ba80f640 1 Processor -- start 2024-03-06T07:31:02.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.691+0000 7f14ba80f640 1 -- start start 2024-03-06T07:31:02.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.692+0000 7f14ba80f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f14b40d14a0 0x7f14b415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.692+0000 7f14ba80f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b415eb00 0x7f14b40cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.692+0000 7f14ba80f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b40cfa50 con 0x7f14b40d10c0 2024-03-06T07:31:02.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.692+0000 7f14ba80f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b415c050 con 0x7f14b40d14a0 2024-03-06T07:31:02.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.692+0000 7f14ba80f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40d0060 con 0x7f14b415eb00 2024-03-06T07:31:02.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.692+0000 7f14b3fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f14b40d10c0 0x7f14b40d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:33952/0 (socket says 172.21.15.59:33952) 2024-03-06T07:31:02.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.692+0000 7f14b3fff640 1 -- 172.21.15.59:0/1665754615 learned_addr learned my addr 172.21.15.59:0/1665754615 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:02.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.692+0000 7f14b37fe640 1 --2- 172.21.15.59:0/1665754615 >> v2:172.21.15.164:3300/0 conn(0x7f14b40d14a0 0x7f14b415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.692+0000 7f14b8dac640 1 --2- 172.21.15.59:0/1665754615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b415eb00 0x7f14b40cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.693+0000 7f14b2ffd640 1 -- 172.21.15.59:0/1665754615 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951324134 0 0) 0x7f14b40cfa50 con 0x7f14b40d10c0 2024-03-06T07:31:02.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.693+0000 7f14b2ffd640 1 -- 172.21.15.59:0/1665754615 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f149c003520 con 0x7f14b40d10c0 2024-03-06T07:31:02.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.693+0000 7f14b37fe640 1 -- 172.21.15.59:0/1665754615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b415eb00 msgr2=0x7f14b40cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.693+0000 7f14b37fe640 1 --2- 172.21.15.59:0/1665754615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b415eb00 0x7f14b40cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.693+0000 7f14b37fe640 1 -- 172.21.15.59:0/1665754615 >> v1:172.21.15.59:6789/0 conn(0x7f14b40d10c0 legacy=0x7f14b40d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.693+0000 7f14b37fe640 1 -- 172.21.15.59:0/1665754615 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b40b45c0 con 0x7f14b40d14a0 2024-03-06T07:31:02.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.693+0000 7f14b8dac640 1 --2- 172.21.15.59:0/1665754615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b415eb00 0x7f14b40cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:02.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14b37fe640 1 --2- 172.21.15.59:0/1665754615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b40d14a0 0x7f14b415e5c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f14a80097a0 tx=0x7f14a802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac79e5b8bbcc8ef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14b2ffd640 1 -- 172.21.15.59:0/1665754615 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8035020 con 0x7f14b40d14a0 2024-03-06T07:31:02.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14b2ffd640 1 -- 172.21.15.59:0/1665754615 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a8031830 con 0x7f14b40d14a0 2024-03-06T07:31:02.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14b2ffd640 1 -- 172.21.15.59:0/1665754615 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8031be0 con 0x7f14b40d14a0 2024-03-06T07:31:02.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14ba80f640 1 -- 172.21.15.59:0/1665754615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b40d14a0 msgr2=0x7f14b415e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14ba80f640 1 --2- 172.21.15.59:0/1665754615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b40d14a0 0x7f14b415e5c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f14a80097a0 tx=0x7f14a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14ba80f640 1 -- 172.21.15.59:0/1665754615 shutdown_connections 2024-03-06T07:31:02.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14ba80f640 1 --2- 172.21.15.59:0/1665754615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b415eb00 0x7f14b40cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14ba80f640 1 --2- 172.21.15.59:0/1665754615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b40d14a0 0x7f14b415e5c0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14ba80f640 1 -- 172.21.15.59:0/1665754615 >> 172.21.15.59:0/1665754615 conn(0x7f14b40b3800 msgr2=0x7f14b40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14ba80f640 1 -- 172.21.15.59:0/1665754615 shutdown_connections 2024-03-06T07:31:02.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.694+0000 7f14ba80f640 1 -- 172.21.15.59:0/1665754615 wait complete. 2024-03-06T07:31:02.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.696+0000 7f14ba80f640 1 Processor -- start 2024-03-06T07:31:02.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.696+0000 7f14ba80f640 1 -- start start 2024-03-06T07:31:02.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14ba80f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b40d10c0 0x7f14b4152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14ba80f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b40d14a0 0x7f14b40b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14ba80f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415eb00 0x7f14b40b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14ba80f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40cfcf0 con 0x7f14b415eb00 2024-03-06T07:31:02.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14ba80f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b415f440 con 0x7f14b40d14a0 2024-03-06T07:31:02.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14ba80f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40cb3c0 con 0x7f14b40d10c0 2024-03-06T07:31:02.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b8dac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415eb00 0x7f14b40b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b40d10c0 0x7f14b4152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b8dac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415eb00 0x7f14b40b2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50698/0 (socket says 172.21.15.59:50698) 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b8dac640 1 -- 172.21.15.59:0/2166564703 learned_addr learned my addr 172.21.15.59:0/2166564703 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b37fe640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b40d14a0 0x7f14b40b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b3fff640 1 -- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b40d14a0 msgr2=0x7f14b40b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b3fff640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b40d14a0 0x7f14b40b27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b3fff640 1 -- 172.21.15.59:0/2166564703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415eb00 msgr2=0x7f14b40b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b3fff640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415eb00 0x7f14b40b2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b3fff640 1 -- 172.21.15.59:0/2166564703 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b40b54d0 con 0x7f14b40d10c0 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b8dac640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415eb00 0x7f14b40b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.698+0000 7f14b3fff640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b40d10c0 0x7f14b4152420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f14a400c400 tx=0x7f14a400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.699+0000 7f14b0ff9640 1 -- 172.21.15.59:0/2166564703 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4017070 con 0x7f14b40d10c0 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.699+0000 7f14b0ff9640 1 -- 172.21.15.59:0/2166564703 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a4005ce0 con 0x7f14b40d10c0 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.699+0000 7f14b0ff9640 1 -- 172.21.15.59:0/2166564703 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a40072f0 con 0x7f14b40d10c0 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.699+0000 7f14ba80f640 1 -- 172.21.15.59:0/2166564703 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b40b57b0 con 0x7f14b40d10c0 2024-03-06T07:31:02.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.699+0000 7f14ba80f640 1 -- 172.21.15.59:0/2166564703 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b40b5d70 con 0x7f14b40d10c0 2024-03-06T07:31:02.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.700+0000 7f14ba80f640 1 -- 172.21.15.59:0/2166564703 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b409fe60 con 0x7f14b40d10c0 2024-03-06T07:31:02.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.700+0000 7f14b0ff9640 1 -- 172.21.15.59:0/2166564703 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14a401a070 con 0x7f14b40d10c0 2024-03-06T07:31:02.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.700+0000 7f14b0ff9640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f14a006b500 0x7f14a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.700+0000 7f14b0ff9640 1 -- 172.21.15.59:0/2166564703 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f14a4089560 con 0x7f14b40d10c0 2024-03-06T07:31:02.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.701+0000 7f14b37fe640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f14a006b500 0x7f14a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.701+0000 7f14b37fe640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f14a006b500 0x7f14a006d9c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f14a8036000 tx=0x7f14a8008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.705+0000 7f14b0ff9640 1 -- 172.21.15.59:0/2166564703 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14a4057cd0 con 0x7f14b40d10c0 2024-03-06T07:31:02.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.886+0000 7f14ba80f640 1 -- 172.21.15.59:0/2166564703 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f14b4068ff0 con 0x7f14b40d10c0 2024-03-06T07:31:02.889 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.887+0000 7f14b0ff9640 1 -- 172.21.15.59:0/2166564703 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f14a405bd70 con 0x7f14b40d10c0 2024-03-06T07:31:02.889 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 -- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f14a006b500 msgr2=0x7f14a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f14a006b500 0x7f14a006d9c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f14a8036000 tx=0x7f14a8008030 comp rx=0 tx=0).stop 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 -- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b40d10c0 msgr2=0x7f14b4152420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b40d10c0 0x7f14b4152420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f14a400c400 tx=0x7f14a400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 -- 172.21.15.59:0/2166564703 shutdown_connections 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f14a006b500 0x7f14a006d9c0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14b415eb00 0x7f14b40b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14b40d14a0 0x7f14b40b27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 --2- 172.21.15.59:0/2166564703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14b40d10c0 0x7f14b4152420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 -- 172.21.15.59:0/2166564703 >> 172.21.15.59:0/2166564703 conn(0x7f14b40b3800 msgr2=0x7f14b4074010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 -- 172.21.15.59:0/2166564703 shutdown_connections 2024-03-06T07:31:02.891 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:02.889+0000 7f14ba80f640 1 -- 172.21.15.59:0/2166564703 wait complete. 2024-03-06T07:31:02.901 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:31:03.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.003+0000 7f56211e9640 1 Processor -- start 2024-03-06T07:31:03.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.003+0000 7f56211e9640 1 -- start start 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.003+0000 7f56211e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c144030 0x7f561c06e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.003+0000 7f56211e9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561c0b4450 con 0x7f561c0b4ef0 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.003+0000 7f56211e9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561c0b5b60 con 0x7f561c06ee10 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.003+0000 7f56211e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0b4a60 con 0x7f561c144030 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.004+0000 7f561b59e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f561c0b4ef0 0x7f561c0cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:33958/0 (socket says 172.21.15.59:33958) 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.004+0000 7f561b59e640 1 -- 172.21.15.59:0/1145757644 learned_addr learned my addr 172.21.15.59:0/1145757644 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.004+0000 7f5613fff640 1 --2- 172.21.15.59:0/1145757644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c144030 0x7f561c06e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.004+0000 7f561a59c640 1 -- 172.21.15.59:0/1145757644 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062312237 0 0) 0x7f561c0b4450 con 0x7f561c0b4ef0 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.004+0000 7f561a59c640 1 -- 172.21.15.59:0/1145757644 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5608003520 con 0x7f561c0b4ef0 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.004+0000 7f5613fff640 1 -- 172.21.15.59:0/1145757644 >> v1:172.21.15.164:6789/0 conn(0x7f561c06ee10 legacy=0x7f561c06f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.004+0000 7f5613fff640 1 -- 172.21.15.59:0/1145757644 >> v1:172.21.15.59:6789/0 conn(0x7f561c0b4ef0 legacy=0x7f561c0cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.004+0000 7f561a59c640 1 -- 172.21.15.59:0/1145757644 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810338877 0 0) 0x7f561c0b5b60 con 0x7f561c06ee10 2024-03-06T07:31:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.004+0000 7f5613fff640 1 -- 172.21.15.59:0/1145757644 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c0b5e40 con 0x7f561c144030 2024-03-06T07:31:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.005+0000 7f5613fff640 1 --2- 172.21.15.59:0/1145757644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c144030 0x7f561c06e7b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5604009800 tx=0x7f560402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c3d8eb549d28fff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.005+0000 7f561a59c640 1 -- 172.21.15.59:0/1145757644 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5604035020 con 0x7f561c144030 2024-03-06T07:31:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.005+0000 7f561a59c640 1 -- 172.21.15.59:0/1145757644 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5604031890 con 0x7f561c144030 2024-03-06T07:31:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.005+0000 7f561a59c640 1 -- 172.21.15.59:0/1145757644 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5604031c40 con 0x7f561c144030 2024-03-06T07:31:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.005+0000 7f56211e9640 1 -- 172.21.15.59:0/1145757644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c144030 msgr2=0x7f561c06e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.005+0000 7f56211e9640 1 --2- 172.21.15.59:0/1145757644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c144030 0x7f561c06e7b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5604009800 tx=0x7f560402b930 comp rx=0 tx=0).stop 2024-03-06T07:31:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.005+0000 7f56211e9640 1 -- 172.21.15.59:0/1145757644 shutdown_connections 2024-03-06T07:31:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.005+0000 7f56211e9640 1 --2- 172.21.15.59:0/1145757644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c144030 0x7f561c06e7b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.005+0000 7f56211e9640 1 -- 172.21.15.59:0/1145757644 >> 172.21.15.59:0/1145757644 conn(0x7f561c0d0a10 msgr2=0x7f561c0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.006+0000 7f56211e9640 1 -- 172.21.15.59:0/1145757644 shutdown_connections 2024-03-06T07:31:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.006+0000 7f56211e9640 1 -- 172.21.15.59:0/1145757644 wait complete. 2024-03-06T07:31:03.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.007+0000 7f56211e9640 1 Processor -- start 2024-03-06T07:31:03.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.007+0000 7f56211e9640 1 -- start start 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.007+0000 7f56211e9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f561c06ee10 0x7f561c0cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.007+0000 7f56211e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c0b4ef0 0x7f561c0c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f56211e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f561c144030 0x7f561c0c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f56211e9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0b5380 con 0x7f561c144030 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f56211e9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c144840 con 0x7f561c06ee10 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f56211e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561c0b5500 con 0x7f561c0b4ef0 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f561b59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f561c144030 0x7f561c0c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f561b59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f561c144030 0x7f561c0c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50708/0 (socket says 172.21.15.59:50708) 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f561ad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f561c06ee10 0x7f561c0cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f561b59e640 1 -- 172.21.15.59:0/2139831865 learned_addr learned my addr 172.21.15.59:0/2139831865 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f5613fff640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c0b4ef0 0x7f561c0c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f561b59e640 1 -- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c0b4ef0 msgr2=0x7f561c0c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f561b59e640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c0b4ef0 0x7f561c0c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f561b59e640 1 -- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f561c06ee10 msgr2=0x7f561c0cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f561b59e640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f561c06ee10 0x7f561c0cc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f561b59e640 1 -- 172.21.15.59:0/2139831865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c0aef80 con 0x7f561c144030 2024-03-06T07:31:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f561b59e640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f561c144030 0x7f561c0c5ef0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f561400cb40 tx=0x7f561400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f56137fe640 1 -- 172.21.15.59:0/2139831865 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5614017070 con 0x7f561c144030 2024-03-06T07:31:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f56137fe640 1 -- 172.21.15.59:0/2139831865 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561400aca0 con 0x7f561c144030 2024-03-06T07:31:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f56137fe640 1 -- 172.21.15.59:0/2139831865 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5614012620 con 0x7f561c144030 2024-03-06T07:31:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f56211e9640 1 -- 172.21.15.59:0/2139831865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561c1d31c0 con 0x7f561c144030 2024-03-06T07:31:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.008+0000 7f56211e9640 1 -- 172.21.15.59:0/2139831865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561c1d3780 con 0x7f561c144030 2024-03-06T07:31:03.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.009+0000 7f56211e9640 1 -- 172.21.15.59:0/2139831865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f561c04f330 con 0x7f561c144030 2024-03-06T07:31:03.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.010+0000 7f56137fe640 1 -- 172.21.15.59:0/2139831865 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f561401a070 con 0x7f561c144030 2024-03-06T07:31:03.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.010+0000 7f56137fe640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55fc06b4d0 0x7f55fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.010+0000 7f56137fe640 1 -- 172.21.15.59:0/2139831865 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f561408a000 con 0x7f561c144030 2024-03-06T07:31:03.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.011+0000 7f561ad9d640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55fc06b4d0 0x7f55fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.011+0000 7f561ad9d640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55fc06b4d0 0x7f55fc06d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f560c000f60 tx=0x7f560c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.015+0000 7f56137fe640 1 -- 172.21.15.59:0/2139831865 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5614058810 con 0x7f561c144030 2024-03-06T07:31:03.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.211+0000 7f56211e9640 1 -- 172.21.15.59:0/2139831865 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f561c1e9320 con 0x7f561c144030 2024-03-06T07:31:03.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.212+0000 7f56137fe640 1 -- 172.21.15.59:0/2139831865 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) v1 ==== 80+0+0 (secure 0 0 0) 0x7f561405c8b0 con 0x7f561c144030 2024-03-06T07:31:03.213 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:31:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.213+0000 7f56211e9640 1 -- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55fc06b4d0 msgr2=0x7f55fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.213+0000 7f56211e9640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55fc06b4d0 0x7f55fc06d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f560c000f60 tx=0x7f560c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.213+0000 7f56211e9640 1 -- 172.21.15.59:0/2139831865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f561c144030 msgr2=0x7f561c0c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.213+0000 7f56211e9640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f561c144030 0x7f561c0c5ef0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f561400cb40 tx=0x7f561400a060 comp rx=0 tx=0).stop 2024-03-06T07:31:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.213+0000 7f56211e9640 1 -- 172.21.15.59:0/2139831865 shutdown_connections 2024-03-06T07:31:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.213+0000 7f56211e9640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f55fc06b4d0 0x7f55fc06d990 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.213+0000 7f56211e9640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f561c144030 0x7f561c0c5ef0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.213+0000 7f56211e9640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f561c0b4ef0 0x7f561c0c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.213+0000 7f56211e9640 1 --2- 172.21.15.59:0/2139831865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f561c06ee10 0x7f561c0cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.213+0000 7f56211e9640 1 -- 172.21.15.59:0/2139831865 >> 172.21.15.59:0/2139831865 conn(0x7f561c0d0a10 msgr2=0x7f561c1e83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.214+0000 7f56211e9640 1 -- 172.21.15.59:0/2139831865 shutdown_connections 2024-03-06T07:31:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.214+0000 7f56211e9640 1 -- 172.21.15.59:0/2139831865 wait complete. 2024-03-06T07:31:03.226 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:31:03.226 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:03.241 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:03.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.381+0000 7fc3b5e8a640 1 Processor -- start 2024-03-06T07:31:03.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.381+0000 7fc3b5e8a640 1 -- start start 2024-03-06T07:31:03.384 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.382+0000 7fc3b5e8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013dc10 0x7fc3b00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.382+0000 7fc3b5e8a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc3b015bf30 0x7fc3b00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.382+0000 7fc3b5e8a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc3b015e850 0x7fc3b015eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.382+0000 7fc3b5e8a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b00cf3f0 con 0x7fc3b015e850 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.382+0000 7fc3b5e8a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b015b210 con 0x7fc3b015bf30 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.382+0000 7fc3b5e8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b015f580 con 0x7fc3b013dc10 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.382+0000 7fc3affff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc3b015e850 0x7fc3b015eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.382+0000 7fc3affff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc3b015e850 0x7fc3b015eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50718/0 (socket says 172.21.15.59:50718) 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.382+0000 7fc3affff640 1 -- 172.21.15.59:0/280716314 learned_addr learned my addr 172.21.15.59:0/280716314 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.382+0000 7fc3aeffd640 1 --2- 172.21.15.59:0/280716314 >> v2:172.21.15.164:3300/0 conn(0x7fc3b015bf30 0x7fc3b00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.382+0000 7fc3af7fe640 1 --2- 172.21.15.59:0/280716314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013dc10 0x7fc3b00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.383+0000 7fc3affff640 1 -- 172.21.15.59:0/280716314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013dc10 msgr2=0x7fc3b00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.383+0000 7fc3affff640 1 --2- 172.21.15.59:0/280716314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013dc10 0x7fc3b00b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.383+0000 7fc3affff640 1 -- 172.21.15.59:0/280716314 >> v2:172.21.15.164:3300/0 conn(0x7fc3b015bf30 msgr2=0x7fc3b00b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.383+0000 7fc3affff640 1 --2- 172.21.15.59:0/280716314 >> v2:172.21.15.164:3300/0 conn(0x7fc3b015bf30 0x7fc3b00b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.383+0000 7fc3affff640 1 -- 172.21.15.59:0/280716314 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b015b450 con 0x7fc3b015e850 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.383+0000 7fc3affff640 1 --2- 172.21.15.59:0/280716314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b015e850 0x7fc3b015eca0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0068890 tx=0x7fc3a802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92690a6412ed555 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.383+0000 7fc3ae7fc640 1 -- 172.21.15.59:0/280716314 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a802c9e0 con 0x7fc3b015e850 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.384+0000 7fc3ae7fc640 1 -- 172.21.15.59:0/280716314 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a802cb80 con 0x7fc3b015e850 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.384+0000 7fc3ae7fc640 1 -- 172.21.15.59:0/280716314 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a80343f0 con 0x7fc3b015e850 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.384+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/280716314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b015e850 msgr2=0x7fc3b015eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.384+0000 7fc3b5e8a640 1 --2- 172.21.15.59:0/280716314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b015e850 0x7fc3b015eca0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0068890 tx=0x7fc3a802be40 comp rx=0 tx=0).stop 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.384+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/280716314 shutdown_connections 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.384+0000 7fc3b5e8a640 1 --2- 172.21.15.59:0/280716314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b015e850 0x7fc3b015eca0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.384+0000 7fc3b5e8a640 1 --2- 172.21.15.59:0/280716314 >> v2:172.21.15.164:3300/0 conn(0x7fc3b015bf30 0x7fc3b00b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.384+0000 7fc3b5e8a640 1 --2- 172.21.15.59:0/280716314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013dc10 0x7fc3b00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.384+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/280716314 >> 172.21.15.59:0/280716314 conn(0x7fc3b00cf8a0 msgr2=0x7fc3b00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.384+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/280716314 shutdown_connections 2024-03-06T07:31:03.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.384+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/280716314 wait complete. 2024-03-06T07:31:03.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.385+0000 7fc3b5e8a640 1 Processor -- start 2024-03-06T07:31:03.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.385+0000 7fc3b5e8a640 1 -- start start 2024-03-06T07:31:03.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.386+0000 7fc3b5e8a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b013dc10 0x7fc3b013ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.386+0000 7fc3b5e8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b015bf30 0x7fc3b0137de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.386+0000 7fc3b5e8a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b015e850 0x7fc3b01f7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.386+0000 7fc3b5e8a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b015f1e0 con 0x7fc3b013dc10 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.386+0000 7fc3b5e8a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b013dff0 con 0x7fc3b015e850 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.386+0000 7fc3b5e8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b015f360 con 0x7fc3b015bf30 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.386+0000 7fc3af7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b013dc10 0x7fc3b013ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.386+0000 7fc3af7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b013dc10 0x7fc3b013ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50720/0 (socket says 172.21.15.59:50720) 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.386+0000 7fc3af7fe640 1 -- 172.21.15.59:0/2378006368 learned_addr learned my addr 172.21.15.59:0/2378006368 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.386+0000 7fc3aeffd640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b015bf30 0x7fc3b0137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.386+0000 7fc3affff640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b015e850 0x7fc3b01f7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.387+0000 7fc3af7fe640 1 -- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b015bf30 msgr2=0x7fc3b0137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.387+0000 7fc3af7fe640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b015bf30 0x7fc3b0137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.387+0000 7fc3af7fe640 1 -- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b015e850 msgr2=0x7fc3b01f7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.387+0000 7fc3af7fe640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b015e850 0x7fc3b01f7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.387+0000 7fc3af7fe640 1 -- 172.21.15.59:0/2378006368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b0144210 con 0x7fc3b013dc10 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.387+0000 7fc3af7fe640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b013dc10 0x7fc3b013ea80 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4008670 tx=0x7fc3a4008760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.387+0000 7fc397fff640 1 -- 172.21.15.59:0/2378006368 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a401b070 con 0x7fc3b013dc10 2024-03-06T07:31:03.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.387+0000 7fc397fff640 1 -- 172.21.15.59:0/2378006368 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a4009b10 con 0x7fc3b013dc10 2024-03-06T07:31:03.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.387+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/2378006368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b01444f0 con 0x7fc3b013dc10 2024-03-06T07:31:03.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.387+0000 7fc397fff640 1 -- 172.21.15.59:0/2378006368 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4016460 con 0x7fc3b013dc10 2024-03-06T07:31:03.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.387+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/2378006368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b013efc0 con 0x7fc3b013dc10 2024-03-06T07:31:03.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.388+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/2378006368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b004f330 con 0x7fc3b013dc10 2024-03-06T07:31:03.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.389+0000 7fc397fff640 1 -- 172.21.15.59:0/2378006368 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3a401e070 con 0x7fc3b013dc10 2024-03-06T07:31:03.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.389+0000 7fc397fff640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc39c06b4d0 0x7fc39c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.389+0000 7fc397fff640 1 -- 172.21.15.59:0/2378006368 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc3a4089be0 con 0x7fc3b013dc10 2024-03-06T07:31:03.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.389+0000 7fc3aeffd640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc39c06b4d0 0x7fc39c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.392 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.390+0000 7fc3aeffd640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc39c06b4d0 0x7fc39c06d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fc3b01f7530 tx=0x7fc3a0009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.394+0000 7fc397fff640 1 -- 172.21.15.59:0/2378006368 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3a40583f0 con 0x7fc3b013dc10 2024-03-06T07:31:03.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.581+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/2378006368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3b006eb00 con 0x7fc3b013dc10 2024-03-06T07:31:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.582+0000 7fc397fff640 1 -- 172.21.15.59:0/2378006368 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16717 (secure 0 0 0) 0x7fc3a405c490 con 0x7fc3b013dc10 2024-03-06T07:31:03.584 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:03.584 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":143,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:39.494279+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:11.712614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:12.406781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.22","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:03.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc39c06b4d0 msgr2=0x7fc39c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc39c06b4d0 0x7fc39c06d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fc3b01f7530 tx=0x7fc3a0009190 comp rx=0 tx=0).stop 2024-03-06T07:31:03.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/2378006368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b013dc10 msgr2=0x7fc3b013ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b013dc10 0x7fc3b013ea80 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4008670 tx=0x7fc3a4008760 comp rx=0 tx=0).stop 2024-03-06T07:31:03.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/2378006368 shutdown_connections 2024-03-06T07:31:03.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc39c06b4d0 0x7fc39c06d990 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b015e850 0x7fc3b01f7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b015bf30 0x7fc3b0137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 --2- 172.21.15.59:0/2378006368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc3b013dc10 0x7fc3b013ea80 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/2378006368 >> 172.21.15.59:0/2378006368 conn(0x7fc3b00cf8a0 msgr2=0x7fc3b013b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/2378006368 shutdown_connections 2024-03-06T07:31:03.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.584+0000 7fc3b5e8a640 1 -- 172.21.15.59:0/2378006368 wait complete. 2024-03-06T07:31:03.597 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:11.712614+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:12.406781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:03.597 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:03.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.699+0000 7fa62811e640 1 Processor -- start 2024-03-06T07:31:03.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.699+0000 7fa62811e640 1 -- start start 2024-03-06T07:31:03.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.700+0000 7fa62811e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6200d10c0 0x7fa6200d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.700+0000 7fa62811e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa62015ecd0 0x7fa6200cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa62811e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6200cfa50 con 0x7fa6200d14a0 2024-03-06T07:31:03.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa62811e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62015c050 con 0x7fa62015ecd0 2024-03-06T07:31:03.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa62811e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200d0060 con 0x7fa6200d10c0 2024-03-06T07:31:03.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa6266bb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa62015ecd0 0x7fa6200cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa6256b9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa6200d14a0 0x7fa62015e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59024/0 (socket says 172.21.15.59:59024) 2024-03-06T07:31:03.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa6256b9640 1 -- 172.21.15.59:0/1200179736 learned_addr learned my addr 172.21.15.59:0/1200179736 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:03.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa6266bb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa62015ecd0 0x7fa6200cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58820/0 (socket says 172.21.15.59:58820) 2024-03-06T07:31:03.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa625eba640 1 --2- 172.21.15.59:0/1200179736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6200d10c0 0x7fa6200d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa6266bb640 1 -- 172.21.15.59:0/1200179736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6200d10c0 msgr2=0x7fa6200d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa6266bb640 1 --2- 172.21.15.59:0/1200179736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6200d10c0 0x7fa6200d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa6266bb640 1 -- 172.21.15.59:0/1200179736 >> v1:172.21.15.59:6789/0 conn(0x7fa6200d14a0 legacy=0x7fa62015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.701+0000 7fa6266bb640 1 -- 172.21.15.59:0/1200179736 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6200d0a30 con 0x7fa62015ecd0 2024-03-06T07:31:03.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.702+0000 7fa6266bb640 1 --2- 172.21.15.59:0/1200179736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa62015ecd0 0x7fa6200cae80 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fa618009d90 tx=0x7fa61802bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d12c977e487fa58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.702+0000 7fa624eb8640 1 -- 172.21.15.59:0/1200179736 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618039070 con 0x7fa62015ecd0 2024-03-06T07:31:03.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.702+0000 7fa624eb8640 1 -- 172.21.15.59:0/1200179736 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61802ca10 con 0x7fa62015ecd0 2024-03-06T07:31:03.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.702+0000 7fa624eb8640 1 -- 172.21.15.59:0/1200179736 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61802cde0 con 0x7fa62015ecd0 2024-03-06T07:31:03.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.702+0000 7fa62811e640 1 -- 172.21.15.59:0/1200179736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa62015ecd0 msgr2=0x7fa6200cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.702+0000 7fa62811e640 1 --2- 172.21.15.59:0/1200179736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa62015ecd0 0x7fa6200cae80 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fa618009d90 tx=0x7fa61802bce0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.702+0000 7fa62811e640 1 -- 172.21.15.59:0/1200179736 shutdown_connections 2024-03-06T07:31:03.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.702+0000 7fa62811e640 1 --2- 172.21.15.59:0/1200179736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa62015ecd0 0x7fa6200cae80 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.702+0000 7fa62811e640 1 --2- 172.21.15.59:0/1200179736 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6200d10c0 0x7fa6200d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.702+0000 7fa62811e640 1 -- 172.21.15.59:0/1200179736 >> 172.21.15.59:0/1200179736 conn(0x7fa6200b3800 msgr2=0x7fa6200b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.703+0000 7fa62811e640 1 -- 172.21.15.59:0/1200179736 shutdown_connections 2024-03-06T07:31:03.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.703+0000 7fa62811e640 1 -- 172.21.15.59:0/1200179736 wait complete. 2024-03-06T07:31:03.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.704+0000 7fa62811e640 1 Processor -- start 2024-03-06T07:31:03.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.704+0000 7fa62811e640 1 -- start start 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.704+0000 7fa62811e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6200d10c0 0x7fa620152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.704+0000 7fa62811e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6200d14a0 0x7fa6200b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.704+0000 7fa62811e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6200b5890 0x7fa6200b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.704+0000 7fa62811e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62015f440 con 0x7fa6200b5890 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.704+0000 7fa62811e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62015f160 con 0x7fa6200d10c0 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.704+0000 7fa62811e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200cfcf0 con 0x7fa6200d14a0 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6266bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6200b5890 0x7fa6200b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6266bb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6200b5890 0x7fa6200b2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58824/0 (socket says 172.21.15.59:58824) 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6266bb640 1 -- 172.21.15.59:0/1719213946 learned_addr learned my addr 172.21.15.59:0/1719213946 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6256b9640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6200d14a0 0x7fa6200b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa625eba640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6200d10c0 0x7fa620152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6266bb640 1 -- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6200d14a0 msgr2=0x7fa6200b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6266bb640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6200d14a0 0x7fa6200b27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6266bb640 1 -- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6200d10c0 msgr2=0x7fa620152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6266bb640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6200d10c0 0x7fa620152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6266bb640 1 -- 172.21.15.59:0/1719213946 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6200b4ad0 con 0x7fa6200b5890 2024-03-06T07:31:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6266bb640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6200b5890 0x7fa6200b2d00 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fa618009d90 tx=0x7fa618004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6167fc640 1 -- 172.21.15.59:0/1719213946 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618039050 con 0x7fa6200b5890 2024-03-06T07:31:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6167fc640 1 -- 172.21.15.59:0/1719213946 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa618045070 con 0x7fa6200b5890 2024-03-06T07:31:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa62811e640 1 -- 172.21.15.59:0/1719213946 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6200b4db0 con 0x7fa6200b5890 2024-03-06T07:31:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa6167fc640 1 -- 172.21.15.59:0/1719213946 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618040450 con 0x7fa6200b5890 2024-03-06T07:31:03.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.705+0000 7fa62811e640 1 -- 172.21.15.59:0/1719213946 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa620152a80 con 0x7fa6200b5890 2024-03-06T07:31:03.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.707+0000 7fa62811e640 1 -- 172.21.15.59:0/1719213946 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62009fe60 con 0x7fa6200b5890 2024-03-06T07:31:03.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.707+0000 7fa6167fc640 1 -- 172.21.15.59:0/1719213946 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6180405f0 con 0x7fa6200b5890 2024-03-06T07:31:03.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.707+0000 7fa6167fc640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa60406b500 0x7fa60406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.707+0000 7fa6167fc640 1 -- 172.21.15.59:0/1719213946 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa6180ac8b0 con 0x7fa6200b5890 2024-03-06T07:31:03.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.707+0000 7fa625eba640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa60406b500 0x7fa60406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.708+0000 7fa625eba640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa60406b500 0x7fa60406d9c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009aa0 tx=0x7fa61c009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.711+0000 7fa6167fc640 1 -- 172.21.15.59:0/1719213946 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa61807b0c0 con 0x7fa6200b5890 2024-03-06T07:31:03.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.906+0000 7fa62811e640 1 -- 172.21.15.59:0/1719213946 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa620068ff0 con 0x7fa6200b5890 2024-03-06T07:31:03.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.906+0000 7fa6167fc640 1 -- 172.21.15.59:0/1719213946 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa61807f160 con 0x7fa6200b5890 2024-03-06T07:31:03.908 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 -- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa60406b500 msgr2=0x7fa60406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa60406b500 0x7fa60406d9c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009aa0 tx=0x7fa61c009190 comp rx=0 tx=0).stop 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 -- 172.21.15.59:0/1719213946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6200b5890 msgr2=0x7fa6200b2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6200b5890 0x7fa6200b2d00 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fa618009d90 tx=0x7fa618004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 -- 172.21.15.59:0/1719213946 shutdown_connections 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa60406b500 0x7fa60406d9c0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6200b5890 0x7fa6200b2d00 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6200d14a0 0x7fa6200b27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 --2- 172.21.15.59:0/1719213946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6200d10c0 0x7fa620152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 -- 172.21.15.59:0/1719213946 >> 172.21.15.59:0/1719213946 conn(0x7fa6200b3800 msgr2=0x7fa6200cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 -- 172.21.15.59:0/1719213946 shutdown_connections 2024-03-06T07:31:03.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:03.909+0000 7fa62811e640 1 -- 172.21.15.59:0/1719213946 wait complete. 2024-03-06T07:31:03.921 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:04.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.027+0000 7f0b143e2640 1 Processor -- start 2024-03-06T07:31:04.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.027+0000 7f0b143e2640 1 -- start start 2024-03-06T07:31:04.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b143e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b0c1614d0 0x7f0b0c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.030 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b143e2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0b0c0d2480 0x7f0b0c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b143e2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0c1618b0 con 0x7f0b0c0d1910 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b143e2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0c0cf400 con 0x7f0b0c0d2480 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b143e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0cf6a0 con 0x7f0b0c1614d0 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b1297f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0b0c0d2480 0x7f0b0c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b1297f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0b0c0d2480 0x7f0b0c0b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58838/0 (socket says 172.21.15.59:58838) 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b1297f640 1 -- 172.21.15.59:0/2694096254 learned_addr learned my addr 172.21.15.59:0/2694096254 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b1217e640 1 --2- 172.21.15.59:0/2694096254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b0c1614d0 0x7f0b0c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b1297f640 1 -- 172.21.15.59:0/2694096254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b0c1614d0 msgr2=0x7f0b0c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b1297f640 1 --2- 172.21.15.59:0/2694096254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b0c1614d0 0x7f0b0c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b1297f640 1 -- 172.21.15.59:0/2694096254 >> v1:172.21.15.164:6789/0 conn(0x7f0b0c0d1910 legacy=0x7f0b0c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.031 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.028+0000 7f0b1297f640 1 -- 172.21.15.59:0/2694096254 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c0cf980 con 0x7f0b0c0d2480 2024-03-06T07:31:04.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.029+0000 7f0b1297f640 1 --2- 172.21.15.59:0/2694096254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b0c0d2480 0x7f0b0c0b0e30 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009e50 tx=0x7f0b0402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17afb027bfedbd83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.029+0000 7f0b1117c640 1 -- 172.21.15.59:0/2694096254 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0402cb00 con 0x7f0b0c0d2480 2024-03-06T07:31:04.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.029+0000 7f0b1117c640 1 -- 172.21.15.59:0/2694096254 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0402cca0 con 0x7f0b0c0d2480 2024-03-06T07:31:04.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.029+0000 7f0b1117c640 1 -- 172.21.15.59:0/2694096254 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b04034610 con 0x7f0b0c0d2480 2024-03-06T07:31:04.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.030+0000 7f0b143e2640 1 -- 172.21.15.59:0/2694096254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b0c0d2480 msgr2=0x7f0b0c0b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.030+0000 7f0b143e2640 1 --2- 172.21.15.59:0/2694096254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b0c0d2480 0x7f0b0c0b0e30 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009e50 tx=0x7f0b0402bf60 comp rx=0 tx=0).stop 2024-03-06T07:31:04.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.030+0000 7f0b143e2640 1 -- 172.21.15.59:0/2694096254 shutdown_connections 2024-03-06T07:31:04.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.030+0000 7f0b143e2640 1 --2- 172.21.15.59:0/2694096254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b0c0d2480 0x7f0b0c0b0e30 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.030+0000 7f0b143e2640 1 --2- 172.21.15.59:0/2694096254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b0c1614d0 0x7f0b0c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.030+0000 7f0b143e2640 1 -- 172.21.15.59:0/2694096254 >> 172.21.15.59:0/2694096254 conn(0x7f0b0c0d3d40 msgr2=0x7f0b0c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.030+0000 7f0b143e2640 1 -- 172.21.15.59:0/2694096254 shutdown_connections 2024-03-06T07:31:04.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.030+0000 7f0b143e2640 1 -- 172.21.15.59:0/2694096254 wait complete. 2024-03-06T07:31:04.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.031+0000 7f0b143e2640 1 Processor -- start 2024-03-06T07:31:04.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.031+0000 7f0b143e2640 1 -- start start 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b143e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b0c0d1910 0x7f0b0c1f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b143e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b0c0d2480 0x7f0b0c1f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b143e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b0c1614d0 0x7f0b0c1f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b143e2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0b1370 con 0x7f0b0c0d1910 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b143e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0b0910 con 0x7f0b0c0d2480 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b143e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0b14f0 con 0x7f0b0c1614d0 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1217e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b0c0d1910 0x7f0b0c1f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1217e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b0c0d1910 0x7f0b0c1f54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58848/0 (socket says 172.21.15.59:58848) 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1217e640 1 -- 172.21.15.59:0/3161200846 learned_addr learned my addr 172.21.15.59:0/3161200846 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1197d640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b0c0d2480 0x7f0b0c1f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1297f640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b0c1614d0 0x7f0b0c1f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1217e640 1 -- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b0c1614d0 msgr2=0x7f0b0c1f4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1217e640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b0c1614d0 0x7f0b0c1f4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1217e640 1 -- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b0c0d2480 msgr2=0x7f0b0c1f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1217e640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b0c0d2480 0x7f0b0c1f7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1217e640 1 -- 172.21.15.59:0/3161200846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c1f7640 con 0x7f0b0c0d1910 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1197d640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b0c0d2480 0x7f0b0c1f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.032+0000 7f0b1297f640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b0c1614d0 0x7f0b0c1f4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.033+0000 7f0b1217e640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b0c0d1910 0x7f0b0c1f54b0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00e870 tx=0x7f0afc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.033+0000 7f0b02ffd640 1 -- 172.21.15.59:0/3161200846 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc017070 con 0x7f0b0c0d1910 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.033+0000 7f0b02ffd640 1 -- 172.21.15.59:0/3161200846 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc00c940 con 0x7f0b0c0d1910 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.033+0000 7f0b143e2640 1 -- 172.21.15.59:0/3161200846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0c0b80c0 con 0x7f0b0c0d1910 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.033+0000 7f0b02ffd640 1 -- 172.21.15.59:0/3161200846 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc00cd30 con 0x7f0b0c0d1910 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.033+0000 7f0b143e2640 1 -- 172.21.15.59:0/3161200846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0c0c9580 con 0x7f0b0c0d1910 2024-03-06T07:31:04.038 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.034+0000 7f0b143e2640 1 -- 172.21.15.59:0/3161200846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0c068ff0 con 0x7f0b0c0d1910 2024-03-06T07:31:04.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.034+0000 7f0b02ffd640 1 -- 172.21.15.59:0/3161200846 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0afc01a070 con 0x7f0b0c0d1910 2024-03-06T07:31:04.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.035+0000 7f0b02ffd640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0af006b4d0 0x7f0af006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.035+0000 7f0b02ffd640 1 -- 172.21.15.59:0/3161200846 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0afc08a000 con 0x7f0b0c0d1910 2024-03-06T07:31:04.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.035+0000 7f0b1197d640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0af006b4d0 0x7f0af006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.035+0000 7f0b1197d640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0af006b4d0 0x7f0af006d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c15ca50 tx=0x7f0b08009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.039+0000 7f0b02ffd640 1 -- 172.21.15.59:0/3161200846 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0afc058ef0 con 0x7f0b0c0d1910 2024-03-06T07:31:04.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.247+0000 7f0b143e2640 1 -- 172.21.15.59:0/3161200846 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0b0c0ce810 con 0x7f0b0c0d1910 2024-03-06T07:31:04.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.248+0000 7f0b02ffd640 1 -- 172.21.15.59:0/3161200846 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0afc0587f0 con 0x7f0b0c0d1910 2024-03-06T07:31:04.250 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 -- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0af006b4d0 msgr2=0x7f0af006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0af006b4d0 0x7f0af006d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c15ca50 tx=0x7f0b08009040 comp rx=0 tx=0).stop 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 -- 172.21.15.59:0/3161200846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b0c0d1910 msgr2=0x7f0b0c1f54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b0c0d1910 0x7f0b0c1f54b0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00e870 tx=0x7f0afc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 -- 172.21.15.59:0/3161200846 shutdown_connections 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0af006b4d0 0x7f0af006d990 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b0c1614d0 0x7f0b0c1f4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b0c0d2480 0x7f0b0c1f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 --2- 172.21.15.59:0/3161200846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b0c0d1910 0x7f0b0c1f54b0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 -- 172.21.15.59:0/3161200846 >> 172.21.15.59:0/3161200846 conn(0x7f0b0c0d3d40 msgr2=0x7f0b0c0d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 -- 172.21.15.59:0/3161200846 shutdown_connections 2024-03-06T07:31:04.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.249+0000 7f0b143e2640 1 -- 172.21.15.59:0/3161200846 wait complete. 2024-03-06T07:31:04.265 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.371+0000 7f3489f5c640 1 Processor -- start 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.371+0000 7f3489f5c640 1 -- start start 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.373+0000 7f3489f5c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f34840cd280 0x7f348406b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.373+0000 7f3489f5c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34840ce200 con 0x7f348415be20 2024-03-06T07:31:04.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.373+0000 7f3489f5c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34840ada00 con 0x7f34840ae0f0 2024-03-06T07:31:04.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.373+0000 7f3489f5c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34840adce0 con 0x7f34840cd280 2024-03-06T07:31:04.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.373+0000 7f347affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f34840cd280 0x7f348406b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.373+0000 7f34837fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f348415be20 0x7f34840ccb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59040/0 (socket says 172.21.15.59:59040) 2024-03-06T07:31:04.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.373+0000 7f34837fe640 1 -- 172.21.15.59:0/2571259156 learned_addr learned my addr 172.21.15.59:0/2571259156 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.373+0000 7f347affd640 1 -- 172.21.15.59:0/2571259156 >> v1:172.21.15.164:6789/0 conn(0x7f34840ae0f0 legacy=0x7f348406b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.373+0000 7f347affd640 1 -- 172.21.15.59:0/2571259156 >> v1:172.21.15.59:6789/0 conn(0x7f348415be20 legacy=0x7f34840ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.373+0000 7f347affd640 1 -- 172.21.15.59:0/2571259156 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348409df90 con 0x7f34840cd280 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.374+0000 7f347affd640 1 --2- 172.21.15.59:0/2571259156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34840cd280 0x7f348406b390 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f346c0097a0 tx=0x7f346c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ec1b0812159389 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.374+0000 7f3482ffd640 1 -- 172.21.15.59:0/2571259156 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c02f810 con 0x7f34840cd280 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.374+0000 7f3482ffd640 1 -- 172.21.15.59:0/2571259156 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346c02f9b0 con 0x7f34840cd280 2024-03-06T07:31:04.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.374+0000 7f3482ffd640 1 -- 172.21.15.59:0/2571259156 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c02fd80 con 0x7f34840cd280 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.374+0000 7f3489f5c640 1 -- 172.21.15.59:0/2571259156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34840cd280 msgr2=0x7f348406b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.374+0000 7f3489f5c640 1 --2- 172.21.15.59:0/2571259156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34840cd280 0x7f348406b390 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f346c0097a0 tx=0x7f346c02b9f0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.374+0000 7f3489f5c640 1 -- 172.21.15.59:0/2571259156 shutdown_connections 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.374+0000 7f3489f5c640 1 --2- 172.21.15.59:0/2571259156 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34840cd280 0x7f348406b390 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.374+0000 7f3489f5c640 1 -- 172.21.15.59:0/2571259156 >> 172.21.15.59:0/2571259156 conn(0x7f34840b1c70 msgr2=0x7f34840b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.375+0000 7f3489f5c640 1 -- 172.21.15.59:0/2571259156 shutdown_connections 2024-03-06T07:31:04.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.375+0000 7f3489f5c640 1 -- 172.21.15.59:0/2571259156 wait complete. 2024-03-06T07:31:04.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.376+0000 7f3489f5c640 1 Processor -- start 2024-03-06T07:31:04.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.376+0000 7f3489f5c640 1 -- start start 2024-03-06T07:31:04.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.376+0000 7f3489f5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34840ae0f0 0x7f34841f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.376+0000 7f3489f5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34840cd280 0x7f3484070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f3489f5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f348415be20 0x7f348406e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f3489f5c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34840cfc90 con 0x7f34840ae0f0 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f3489f5c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34840cfb10 con 0x7f34840cd280 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f3489f5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34840cfe10 con 0x7f348415be20 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f34837fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34840ae0f0 0x7f34841f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f34837fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34840ae0f0 0x7f34841f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58874/0 (socket says 172.21.15.59:58874) 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f34837fe640 1 -- 172.21.15.59:0/940242844 learned_addr learned my addr 172.21.15.59:0/940242844 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f347affd640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34840cd280 0x7f3484070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f3483fff640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f348415be20 0x7f348406e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f34837fe640 1 -- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f348415be20 msgr2=0x7f348406e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f34837fe640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f348415be20 0x7f348406e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f34837fe640 1 -- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34840cd280 msgr2=0x7f3484070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f34837fe640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34840cd280 0x7f3484070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f34837fe640 1 -- 172.21.15.59:0/940242844 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34841e30f0 con 0x7f34840ae0f0 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f34837fe640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34840ae0f0 0x7f34841f3250 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f347400c4b0 tx=0x7f347400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f3480ff9640 1 -- 172.21.15.59:0/940242844 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474017070 con 0x7f34840ae0f0 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f3480ff9640 1 -- 172.21.15.59:0/940242844 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3474005ce0 con 0x7f34840ae0f0 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f3489f5c640 1 -- 172.21.15.59:0/940242844 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34841e2b30 con 0x7f34840ae0f0 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f3480ff9640 1 -- 172.21.15.59:0/940242844 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474007270 con 0x7f34840ae0f0 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.377+0000 7f3489f5c640 1 -- 172.21.15.59:0/940242844 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3484153fd0 con 0x7f34840ae0f0 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.378+0000 7f3489f5c640 1 -- 172.21.15.59:0/940242844 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348409e2d0 con 0x7f34840ae0f0 2024-03-06T07:31:04.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.379+0000 7f3480ff9640 1 -- 172.21.15.59:0/940242844 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f347401a070 con 0x7f34840ae0f0 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.379+0000 7f3480ff9640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f346406b4d0 0x7f346406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.379+0000 7f3480ff9640 1 -- 172.21.15.59:0/940242844 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3474089370 con 0x7f34840ae0f0 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.379+0000 7f347affd640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f346406b4d0 0x7f346406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.383 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.380+0000 7f347affd640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f346406b4d0 0x7f346406d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f346c036000 tx=0x7f346c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.383+0000 7f3480ff9640 1 -- 172.21.15.59:0/940242844 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3474057c10 con 0x7f34840ae0f0 2024-03-06T07:31:04.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.701+0000 7f3489f5c640 1 -- 172.21.15.59:0/940242844 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34840acff0 con 0x7f34840ae0f0 2024-03-06T07:31:04.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.701+0000 7f3480ff9640 1 -- 172.21.15.59:0/940242844 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f347405bcb0 con 0x7f34840ae0f0 2024-03-06T07:31:04.703 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:31:04.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 -- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f346406b4d0 msgr2=0x7f346406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f346406b4d0 0x7f346406d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f346c036000 tx=0x7f346c008030 comp rx=0 tx=0).stop 2024-03-06T07:31:04.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 -- 172.21.15.59:0/940242844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34840ae0f0 msgr2=0x7f34841f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34840ae0f0 0x7f34841f3250 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f347400c4b0 tx=0x7f347400c970 comp rx=0 tx=0).stop 2024-03-06T07:31:04.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 -- 172.21.15.59:0/940242844 shutdown_connections 2024-03-06T07:31:04.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f346406b4d0 0x7f346406d990 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f348415be20 0x7f348406e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34840cd280 0x7f3484070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 --2- 172.21.15.59:0/940242844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34840ae0f0 0x7f34841f3250 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 -- 172.21.15.59:0/940242844 >> 172.21.15.59:0/940242844 conn(0x7f34840b1c70 msgr2=0x7f348406c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 -- 172.21.15.59:0/940242844 shutdown_connections 2024-03-06T07:31:04.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.704+0000 7f3489f5c640 1 -- 172.21.15.59:0/940242844 wait complete. 2024-03-06T07:31:04.717 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:31:04.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.835+0000 7f433ecd4640 1 Processor -- start 2024-03-06T07:31:04.837 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.835+0000 7f433ecd4640 1 -- start start 2024-03-06T07:31:04.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f433ecd4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4338162970 0x7f43380d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f433ecd4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43380d22c0 con 0x7f4338161cf0 2024-03-06T07:31:04.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f433ecd4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43380d25a0 con 0x7f43380b4460 2024-03-06T07:31:04.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f433ecd4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4338161b00 con 0x7f4338162970 2024-03-06T07:31:04.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f433d271640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4338161cf0 0x7f43380cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59042/0 (socket says 172.21.15.59:59042) 2024-03-06T07:31:04.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f433d271640 1 -- 172.21.15.59:0/4027187713 learned_addr learned my addr 172.21.15.59:0/4027187713 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:04.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f4337fff640 1 --2- 172.21.15.59:0/4027187713 >> v2:172.21.15.164:3300/0 conn(0x7f4338162970 0x7f43380d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f43377fe640 1 -- 172.21.15.59:0/4027187713 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383283409 0 0) 0x7f43380d22c0 con 0x7f4338161cf0 2024-03-06T07:31:04.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f43377fe640 1 -- 172.21.15.59:0/4027187713 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4324003520 con 0x7f4338161cf0 2024-03-06T07:31:04.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f4337fff640 1 -- 172.21.15.59:0/4027187713 >> v1:172.21.15.164:6789/0 conn(0x7f43380b4460 legacy=0x7f43380b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f4337fff640 1 -- 172.21.15.59:0/4027187713 >> v1:172.21.15.59:6789/0 conn(0x7f4338161cf0 legacy=0x7f43380cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f43377fe640 1 -- 172.21.15.59:0/4027187713 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849354984 0 0) 0x7f43380d25a0 con 0x7f43380b4460 2024-03-06T07:31:04.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.836+0000 7f4337fff640 1 -- 172.21.15.59:0/4027187713 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43380d2880 con 0x7f4338162970 2024-03-06T07:31:04.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.837+0000 7f4337fff640 1 --2- 172.21.15.59:0/4027187713 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4338162970 0x7f43380d4800 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f43200097a0 tx=0x7f432002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=559a6961c290fabc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.837+0000 7f43377fe640 1 -- 172.21.15.59:0/4027187713 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320034070 con 0x7f4338162970 2024-03-06T07:31:04.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.837+0000 7f43377fe640 1 -- 172.21.15.59:0/4027187713 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4320030610 con 0x7f4338162970 2024-03-06T07:31:04.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.837+0000 7f43377fe640 1 -- 172.21.15.59:0/4027187713 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43200309c0 con 0x7f4338162970 2024-03-06T07:31:04.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.838+0000 7f433ecd4640 1 -- 172.21.15.59:0/4027187713 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4338162970 msgr2=0x7f43380d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.838+0000 7f433ecd4640 1 --2- 172.21.15.59:0/4027187713 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4338162970 0x7f43380d4800 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f43200097a0 tx=0x7f432002b960 comp rx=0 tx=0).stop 2024-03-06T07:31:04.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.838+0000 7f433ecd4640 1 -- 172.21.15.59:0/4027187713 shutdown_connections 2024-03-06T07:31:04.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.838+0000 7f433ecd4640 1 --2- 172.21.15.59:0/4027187713 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4338162970 0x7f43380d4800 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.838+0000 7f433ecd4640 1 -- 172.21.15.59:0/4027187713 >> 172.21.15.59:0/4027187713 conn(0x7f43380d3ce0 msgr2=0x7f43380d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.838+0000 7f433ecd4640 1 -- 172.21.15.59:0/4027187713 shutdown_connections 2024-03-06T07:31:04.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.838+0000 7f433ecd4640 1 -- 172.21.15.59:0/4027187713 wait complete. 2024-03-06T07:31:04.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.839+0000 7f433ecd4640 1 Processor -- start 2024-03-06T07:31:04.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.839+0000 7f433ecd4640 1 -- start start 2024-03-06T07:31:04.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ecd4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43380b4460 0x7f43380c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ecd4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4338161cf0 0x7f43381f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ecd4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4338162970 0x7f43380d0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ecd4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380ce950 con 0x7f43380b4460 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ecd4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380cecc0 con 0x7f4338162970 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ecd4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43380b4840 con 0x7f4338161cf0 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ca70640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43380b4460 0x7f43380c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ca70640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43380b4460 0x7f43380c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58876/0 (socket says 172.21.15.59:58876) 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ca70640 1 -- 172.21.15.59:0/2382816630 learned_addr learned my addr 172.21.15.59:0/2382816630 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f4337fff640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4338161cf0 0x7f43381f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433d271640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4338162970 0x7f43380d0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ca70640 1 -- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4338161cf0 msgr2=0x7f43381f4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ca70640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4338161cf0 0x7f43381f4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ca70640 1 -- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4338162970 msgr2=0x7f43380d0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ca70640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4338162970 0x7f43380d0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f433ca70640 1 -- 172.21.15.59:0/2382816630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433814ac90 con 0x7f43380b4460 2024-03-06T07:31:04.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.840+0000 7f4337fff640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4338161cf0 0x7f43381f4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:04.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.841+0000 7f433ca70640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43380b4460 0x7f43380c6be0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f433000c820 tx=0x7f433000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.841+0000 7f43357fa640 1 -- 172.21.15.59:0/2382816630 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330017070 con 0x7f43380b4460 2024-03-06T07:31:04.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.841+0000 7f43357fa640 1 -- 172.21.15.59:0/2382816630 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4330003da0 con 0x7f43380b4460 2024-03-06T07:31:04.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.841+0000 7f433ecd4640 1 -- 172.21.15.59:0/2382816630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433814af10 con 0x7f43380b4460 2024-03-06T07:31:04.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.841+0000 7f43357fa640 1 -- 172.21.15.59:0/2382816630 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330005290 con 0x7f43380b4460 2024-03-06T07:31:04.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.841+0000 7f433ecd4640 1 -- 172.21.15.59:0/2382816630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433814b4d0 con 0x7f43380b4460 2024-03-06T07:31:04.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.842+0000 7f433ecd4640 1 -- 172.21.15.59:0/2382816630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433814bb90 con 0x7f43380b4460 2024-03-06T07:31:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.842+0000 7f43357fa640 1 -- 172.21.15.59:0/2382816630 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f433001a070 con 0x7f43380b4460 2024-03-06T07:31:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.843+0000 7f43357fa640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431806b4d0 0x7f431806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.843+0000 7f43357fa640 1 -- 172.21.15.59:0/2382816630 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4330089820 con 0x7f43380b4460 2024-03-06T07:31:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.843+0000 7f4337fff640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431806b4d0 0x7f431806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.846 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.843+0000 7f4337fff640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431806b4d0 0x7f431806d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4320036000 tx=0x7f432003b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:04.847+0000 7f43357fa640 1 -- 172.21.15.59:0/2382816630 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f433814bb90 con 0x7f43380b4460 2024-03-06T07:31:05.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.039+0000 7f433ecd4640 1 -- 172.21.15.59:0/2382816630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f433804f330 con 0x7f43380b4460 2024-03-06T07:31:05.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.039+0000 7f43357fa640 1 -- 172.21.15.59:0/2382816630 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) v1 ==== 80+0+0 (secure 0 0 0) 0x7f433814ac90 con 0x7f43380b4460 2024-03-06T07:31:05.041 INFO:teuthology.orchestra.run.smithi059.stderr:listed 0 entries 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 -- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431806b4d0 msgr2=0x7f431806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431806b4d0 0x7f431806d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4320036000 tx=0x7f432003b040 comp rx=0 tx=0).stop 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 -- 172.21.15.59:0/2382816630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43380b4460 msgr2=0x7f43380c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43380b4460 0x7f43380c6be0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f433000c820 tx=0x7f433000cce0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 -- 172.21.15.59:0/2382816630 shutdown_connections 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431806b4d0 0x7f431806d990 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4338162970 0x7f43380d0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4338161cf0 0x7f43381f4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 --2- 172.21.15.59:0/2382816630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43380b4460 0x7f43380c6be0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 -- 172.21.15.59:0/2382816630 >> 172.21.15.59:0/2382816630 conn(0x7f43380d3ce0 msgr2=0x7f433806d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 -- 172.21.15.59:0/2382816630 shutdown_connections 2024-03-06T07:31:05.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:05.041+0000 7f433ecd4640 1 -- 172.21.15.59:0/2382816630 wait complete. 2024-03-06T07:31:05.054 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi059.front.sepia.ceph.com... 2024-03-06T07:31:05.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:05.054 DEBUG:teuthology.orchestra.run.smithi059:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:05.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi059.stderr:ceph-fuse[85128]: fuse finished with error 0 and tester_r 0 2024-03-06T07:31:05.082 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:11.183 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:31:11.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:11.184 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:31:11.199 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:31:11.316 INFO:teuthology.orchestra.run.smithi164.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:31:11.316 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:31:11.317 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:31:11.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.439+0000 7fb26d8fb640 1 Processor -- start 2024-03-06T07:31:11.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.439+0000 7fb26d8fb640 1 -- start start 2024-03-06T07:31:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb26d8fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2681614d0 0x7fb2680d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb26d8fb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb2680d1910 0x7fb2680d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb26d8fb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb2680d22b0 0x7fb2680d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb26d8fb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2680d2740 con 0x7fb2680d1910 2024-03-06T07:31:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb26d8fb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2681618b0 con 0x7fb2680d22b0 2024-03-06T07:31:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb26d8fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680cf400 con 0x7fb2681614d0 2024-03-06T07:31:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb2667fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb2680d1910 0x7fb2680d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb2667fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb2680d1910 0x7fb2680d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58890/0 (socket says 172.21.15.59:58890) 2024-03-06T07:31:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb2667fc640 1 -- 172.21.15.59:0/1203239348 learned_addr learned my addr 172.21.15.59:0/1203239348 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb266ffd640 1 --2- 172.21.15.59:0/1203239348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2681614d0 0x7fb2680d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb2677fe640 1 --2- 172.21.15.59:0/1203239348 >> v2:172.21.15.164:3300/0 conn(0x7fb2680d22b0 0x7fb2680d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb2667fc640 1 -- 172.21.15.59:0/1203239348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2681614d0 msgr2=0x7fb2680d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb2667fc640 1 --2- 172.21.15.59:0/1203239348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2681614d0 0x7fb2680d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb2667fc640 1 -- 172.21.15.59:0/1203239348 >> v2:172.21.15.164:3300/0 conn(0x7fb2680d22b0 msgr2=0x7fb2680d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb2667fc640 1 --2- 172.21.15.59:0/1203239348 >> v2:172.21.15.164:3300/0 conn(0x7fb2680d22b0 0x7fb2680d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.441+0000 7fb2667fc640 1 -- 172.21.15.59:0/1203239348 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2680cf6a0 con 0x7fb2680d1910 2024-03-06T07:31:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.442+0000 7fb2667fc640 1 --2- 172.21.15.59:0/1203239348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2680d1910 0x7fb2680d1d70 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fb2680b57d0 tx=0x7fb25002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163bc2c601f1f7b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.442+0000 7fb265ffb640 1 -- 172.21.15.59:0/1203239348 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250035020 con 0x7fb2680d1910 2024-03-06T07:31:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.442+0000 7fb265ffb640 1 -- 172.21.15.59:0/1203239348 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb250031870 con 0x7fb2680d1910 2024-03-06T07:31:11.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.442+0000 7fb265ffb640 1 -- 172.21.15.59:0/1203239348 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250031c40 con 0x7fb2680d1910 2024-03-06T07:31:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.443+0000 7fb26d8fb640 1 -- 172.21.15.59:0/1203239348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2680d1910 msgr2=0x7fb2680d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.443+0000 7fb26d8fb640 1 --2- 172.21.15.59:0/1203239348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2680d1910 0x7fb2680d1d70 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fb2680b57d0 tx=0x7fb25002b910 comp rx=0 tx=0).stop 2024-03-06T07:31:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.443+0000 7fb26d8fb640 1 -- 172.21.15.59:0/1203239348 shutdown_connections 2024-03-06T07:31:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.443+0000 7fb26d8fb640 1 --2- 172.21.15.59:0/1203239348 >> v2:172.21.15.164:3300/0 conn(0x7fb2680d22b0 0x7fb2680d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.443+0000 7fb26d8fb640 1 --2- 172.21.15.59:0/1203239348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2680d1910 0x7fb2680d1d70 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.443+0000 7fb26d8fb640 1 --2- 172.21.15.59:0/1203239348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2681614d0 0x7fb2680d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.443+0000 7fb26d8fb640 1 -- 172.21.15.59:0/1203239348 >> 172.21.15.59:0/1203239348 conn(0x7fb2680d3d40 msgr2=0x7fb2680d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.443+0000 7fb26d8fb640 1 -- 172.21.15.59:0/1203239348 shutdown_connections 2024-03-06T07:31:11.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.443+0000 7fb26d8fb640 1 -- 172.21.15.59:0/1203239348 wait complete. 2024-03-06T07:31:11.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.444+0000 7fb26d8fb640 1 Processor -- start 2024-03-06T07:31:11.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.444+0000 7fb26d8fb640 1 -- start start 2024-03-06T07:31:11.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb26d8fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2680d1910 0x7fb2680b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb26d8fb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2680d22b0 0x7fb2680b9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb26d8fb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2681614d0 0x7fb2681f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb26d8fb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680b0910 con 0x7fb2680d22b0 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb26d8fb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680cfc60 con 0x7fb2681614d0 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb26d8fb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680d0be0 con 0x7fb2680d1910 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb2667fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2680d22b0 0x7fb2680b9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb2667fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2680d22b0 0x7fb2680b9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58900/0 (socket says 172.21.15.59:58900) 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb2667fc640 1 -- 172.21.15.59:0/3532196572 learned_addr learned my addr 172.21.15.59:0/3532196572 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb266ffd640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2680d1910 0x7fb2680b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb2677fe640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2681614d0 0x7fb2681f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb2667fc640 1 -- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2680d1910 msgr2=0x7fb2680b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb2667fc640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2680d1910 0x7fb2680b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb2667fc640 1 -- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2681614d0 msgr2=0x7fb2681f6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb2667fc640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2681614d0 0x7fb2681f6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb2667fc640 1 -- 172.21.15.59:0/3532196572 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26815d120 con 0x7fb2680d22b0 2024-03-06T07:31:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.445+0000 7fb2667fc640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2680d22b0 0x7fb2680b9590 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fb25002b950 tx=0x7fb2500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.446+0000 7fb25ffff640 1 -- 172.21.15.59:0/3532196572 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250040070 con 0x7fb2680d22b0 2024-03-06T07:31:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.446+0000 7fb25ffff640 1 -- 172.21.15.59:0/3532196572 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb250031e10 con 0x7fb2680d22b0 2024-03-06T07:31:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.446+0000 7fb26d8fb640 1 -- 172.21.15.59:0/3532196572 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26815d400 con 0x7fb2680d22b0 2024-03-06T07:31:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.446+0000 7fb25ffff640 1 -- 172.21.15.59:0/3532196572 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250035040 con 0x7fb2680d22b0 2024-03-06T07:31:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.446+0000 7fb26d8fb640 1 -- 172.21.15.59:0/3532196572 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26815d9c0 con 0x7fb2680d22b0 2024-03-06T07:31:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.447+0000 7fb26d8fb640 1 -- 172.21.15.59:0/3532196572 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb268068ff0 con 0x7fb2680d22b0 2024-03-06T07:31:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.447+0000 7fb25ffff640 1 -- 172.21.15.59:0/3532196572 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb250038020 con 0x7fb2680d22b0 2024-03-06T07:31:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.447+0000 7fb25ffff640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb24806b4d0 0x7fb24806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.447+0000 7fb25ffff640 1 -- 172.21.15.59:0/3532196572 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb2500abdd0 con 0x7fb2680d22b0 2024-03-06T07:31:11.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.448+0000 7fb266ffd640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb24806b4d0 0x7fb24806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.452 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.448+0000 7fb266ffd640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb24806b4d0 0x7fb24806d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb2680cdba0 tx=0x7fb260009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.452+0000 7fb25ffff640 1 -- 172.21.15.59:0/3532196572 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb25007a670 con 0x7fb2680d22b0 2024-03-06T07:31:11.676 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:11.674+0000 7fb26d8fb640 1 -- 172.21.15.59:0/3532196572 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fb2680a1c10 con 0x7fb2680d22b0 2024-03-06T07:31:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.568+0000 7fb25ffff640 1 -- 172.21.15.59:0/3532196572 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v105) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb2500ad020 con 0x7fb2680d22b0 2024-03-06T07:31:12.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 -- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb24806b4d0 msgr2=0x7fb24806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb24806b4d0 0x7fb24806d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb2680cdba0 tx=0x7fb260009190 comp rx=0 tx=0).stop 2024-03-06T07:31:12.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 -- 172.21.15.59:0/3532196572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2680d22b0 msgr2=0x7fb2680b9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2680d22b0 0x7fb2680b9590 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fb25002b950 tx=0x7fb2500046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 -- 172.21.15.59:0/3532196572 shutdown_connections 2024-03-06T07:31:12.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb24806b4d0 0x7fb24806d990 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2681614d0 0x7fb2681f6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2680d22b0 0x7fb2680b9590 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 --2- 172.21.15.59:0/3532196572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2680d1910 0x7fb2680b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 -- 172.21.15.59:0/3532196572 >> 172.21.15.59:0/3532196572 conn(0x7fb2680d3d40 msgr2=0x7fb2680b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 -- 172.21.15.59:0/3532196572 shutdown_connections 2024-03-06T07:31:12.574 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.571+0000 7fb26d8fb640 1 -- 172.21.15.59:0/3532196572 wait complete. 2024-03-06T07:31:12.588 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:12.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.687+0000 7fb774f3e640 1 Processor -- start 2024-03-06T07:31:12.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.688+0000 7fb774f3e640 1 -- start start 2024-03-06T07:31:12.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.688+0000 7fb774f3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb77015bb20 0x7fb7700d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.688+0000 7fb774f3e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb7700cfff0 0x7fb7700d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.688+0000 7fb774f3e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb770161980 con 0x7fb7700ce4b0 2024-03-06T07:31:12.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.688+0000 7fb774f3e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb770161c60 con 0x7fb7700cfff0 2024-03-06T07:31:12.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.688+0000 7fb774f3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb770161f40 con 0x7fb77015bb20 2024-03-06T07:31:12.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.689+0000 7fb76dd9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb7700cfff0 0x7fb7700d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.689+0000 7fb76ed9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb7700ce4b0 0x7fb7700ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59054/0 (socket says 172.21.15.59:59054) 2024-03-06T07:31:12.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.689+0000 7fb76ed9d640 1 -- 172.21.15.59:0/1023082136 learned_addr learned my addr 172.21.15.59:0/1023082136 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:12.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.689+0000 7fb76e59c640 1 --2- 172.21.15.59:0/1023082136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb77015bb20 0x7fb7700d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.689+0000 7fb76dd9b640 1 -- 172.21.15.59:0/1023082136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb77015bb20 msgr2=0x7fb7700d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.689+0000 7fb76dd9b640 1 --2- 172.21.15.59:0/1023082136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb77015bb20 0x7fb7700d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.689+0000 7fb76d59a640 1 -- 172.21.15.59:0/1023082136 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969738113 0 0) 0x7fb770161980 con 0x7fb7700ce4b0 2024-03-06T07:31:12.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.689+0000 7fb76dd9b640 1 -- 172.21.15.59:0/1023082136 >> v1:172.21.15.59:6789/0 conn(0x7fb7700ce4b0 legacy=0x7fb7700ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.689+0000 7fb76dd9b640 1 -- 172.21.15.59:0/1023082136 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb770162220 con 0x7fb7700cfff0 2024-03-06T07:31:12.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.690+0000 7fb76dd9b640 1 --2- 172.21.15.59:0/1023082136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7700cfff0 0x7fb7700d0470 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fb7600097a0 tx=0x7fb76002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8449c225fc379a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.690+0000 7fb76d59a640 1 -- 172.21.15.59:0/1023082136 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760035020 con 0x7fb7700cfff0 2024-03-06T07:31:12.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.690+0000 7fb76d59a640 1 -- 172.21.15.59:0/1023082136 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb760031830 con 0x7fb7700cfff0 2024-03-06T07:31:12.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.690+0000 7fb76d59a640 1 -- 172.21.15.59:0/1023082136 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760031c00 con 0x7fb7700cfff0 2024-03-06T07:31:12.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.691+0000 7fb774f3e640 1 -- 172.21.15.59:0/1023082136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7700cfff0 msgr2=0x7fb7700d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.691+0000 7fb774f3e640 1 --2- 172.21.15.59:0/1023082136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7700cfff0 0x7fb7700d0470 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fb7600097a0 tx=0x7fb76002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.691+0000 7fb774f3e640 1 -- 172.21.15.59:0/1023082136 shutdown_connections 2024-03-06T07:31:12.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.691+0000 7fb774f3e640 1 --2- 172.21.15.59:0/1023082136 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7700cfff0 0x7fb7700d0470 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.691+0000 7fb774f3e640 1 --2- 172.21.15.59:0/1023082136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb77015bb20 0x7fb7700d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.691+0000 7fb774f3e640 1 -- 172.21.15.59:0/1023082136 >> 172.21.15.59:0/1023082136 conn(0x7fb77009a9f0 msgr2=0x7fb77009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.691+0000 7fb774f3e640 1 -- 172.21.15.59:0/1023082136 shutdown_connections 2024-03-06T07:31:12.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.691+0000 7fb774f3e640 1 -- 172.21.15.59:0/1023082136 wait complete. 2024-03-06T07:31:12.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.692+0000 7fb774f3e640 1 Processor -- start 2024-03-06T07:31:12.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.692+0000 7fb774f3e640 1 -- start start 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb774f3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7700ce4b0 0x7fb77015d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb774f3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7700cfff0 0x7fb7700c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb774f3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb77015bb20 0x7fb77015b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb774f3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7700d24b0 con 0x7fb77015bb20 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb774f3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7700d2330 con 0x7fb7700ce4b0 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb774f3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7700d1980 con 0x7fb7700cfff0 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb76ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb77015bb20 0x7fb77015b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb76ed9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb77015bb20 0x7fb77015b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58918/0 (socket says 172.21.15.59:58918) 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb76ed9d640 1 -- 172.21.15.59:0/962522748 learned_addr learned my addr 172.21.15.59:0/962522748 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb76e59c640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7700ce4b0 0x7fb77015d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb76dd9b640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7700cfff0 0x7fb7700c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb76ed9d640 1 -- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7700cfff0 msgr2=0x7fb7700c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb76ed9d640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7700cfff0 0x7fb7700c7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb76ed9d640 1 -- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7700ce4b0 msgr2=0x7fb77015d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb76ed9d640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7700ce4b0 0x7fb77015d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb76ed9d640 1 -- 172.21.15.59:0/962522748 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7700c84c0 con 0x7fb77015bb20 2024-03-06T07:31:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.693+0000 7fb76e59c640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7700ce4b0 0x7fb77015d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.694+0000 7fb76ed9d640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb77015bb20 0x7fb77015b080 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fb76800c700 tx=0x7fb76800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.694+0000 7fb766ffd640 1 -- 172.21.15.59:0/962522748 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768018070 con 0x7fb77015bb20 2024-03-06T07:31:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.694+0000 7fb766ffd640 1 -- 172.21.15.59:0/962522748 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76800ce10 con 0x7fb77015bb20 2024-03-06T07:31:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.694+0000 7fb774f3e640 1 -- 172.21.15.59:0/962522748 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7700c81e0 con 0x7fb77015bb20 2024-03-06T07:31:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.694+0000 7fb766ffd640 1 -- 172.21.15.59:0/962522748 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76800a860 con 0x7fb77015bb20 2024-03-06T07:31:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.694+0000 7fb774f3e640 1 -- 172.21.15.59:0/962522748 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77015d140 con 0x7fb77015bb20 2024-03-06T07:31:12.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.695+0000 7fb774f3e640 1 -- 172.21.15.59:0/962522748 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77004f330 con 0x7fb77015bb20 2024-03-06T07:31:12.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.695+0000 7fb766ffd640 1 -- 172.21.15.59:0/962522748 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb76801b070 con 0x7fb77015bb20 2024-03-06T07:31:12.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.696+0000 7fb766ffd640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb75006b4d0 0x7fb75006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.696+0000 7fb766ffd640 1 -- 172.21.15.59:0/962522748 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb7680897b0 con 0x7fb77015bb20 2024-03-06T07:31:12.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.696+0000 7fb76e59c640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb75006b4d0 0x7fb75006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.696+0000 7fb76e59c640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb75006b4d0 0x7fb75006d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb758009ac0 tx=0x7fb758008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.704+0000 7fb766ffd640 1 -- 172.21.15.59:0/962522748 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb768058600 con 0x7fb77015bb20 2024-03-06T07:31:12.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.924+0000 7fb774f3e640 1 -- 172.21.15.59:0/962522748 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7700b0530 con 0x7fb77015bb20 2024-03-06T07:31:12.926 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.924+0000 7fb766ffd640 1 -- 172.21.15.59:0/962522748 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7fb768057f00 con 0x7fb77015bb20 2024-03-06T07:31:12.927 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:12.927 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:12.927 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:12.928 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 -- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb75006b4d0 msgr2=0x7fb75006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb75006b4d0 0x7fb75006d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb758009ac0 tx=0x7fb758008040 comp rx=0 tx=0).stop 2024-03-06T07:31:12.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 -- 172.21.15.59:0/962522748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb77015bb20 msgr2=0x7fb77015b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb77015bb20 0x7fb77015b080 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fb76800c700 tx=0x7fb76800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 -- 172.21.15.59:0/962522748 shutdown_connections 2024-03-06T07:31:12.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb75006b4d0 0x7fb75006d990 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb77015bb20 0x7fb77015b080 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7700cfff0 0x7fb7700c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 --2- 172.21.15.59:0/962522748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7700ce4b0 0x7fb77015d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 -- 172.21.15.59:0/962522748 >> 172.21.15.59:0/962522748 conn(0x7fb77009a9f0 msgr2=0x7fb77009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 -- 172.21.15.59:0/962522748 shutdown_connections 2024-03-06T07:31:12.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:12.926+0000 7fb774f3e640 1 -- 172.21.15.59:0/962522748 wait complete. 2024-03-06T07:31:13.141 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:13.141 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:14.142 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:14.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.242+0000 7f123e839640 1 Processor -- start 2024-03-06T07:31:14.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.242+0000 7f123e839640 1 -- start start 2024-03-06T07:31:14.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.243+0000 7f123e839640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f123815dfe0 0x7f12380cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.243+0000 7f123e839640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f12380d29c0 0x7f12380b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.243+0000 7f123e839640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12380b3430 con 0x7f12380cf830 2024-03-06T07:31:14.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.243+0000 7f123e839640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12380b31e0 con 0x7f12380d29c0 2024-03-06T07:31:14.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.243+0000 7f123e839640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12380b3d60 con 0x7f123815dfe0 2024-03-06T07:31:14.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.243+0000 7f12377fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f12380cf830 0x7f12380d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:32794/0 (socket says 172.21.15.59:32794) 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.243+0000 7f12377fe640 1 -- 172.21.15.59:0/729965764 learned_addr learned my addr 172.21.15.59:0/729965764 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1237fff640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f123815dfe0 0x7f12380cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f123cdd6640 1 --2- 172.21.15.59:0/729965764 >> v2:172.21.15.164:3300/0 conn(0x7f12380d29c0 0x7f12380b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985740839 0 0) 0x7f12380b3430 con 0x7f12380cf830 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1220003520 con 0x7f12380cf830 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3955940304 0 0) 0x7f1220003520 con 0x7f12380cf830 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f123815dfe0 msgr2=0x7f12380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f123815dfe0 0x7f12380cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 >> v2:172.21.15.164:3300/0 conn(0x7f12380d29c0 msgr2=0x7f12380b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 --2- 172.21.15.59:0/729965764 >> v2:172.21.15.164:3300/0 conn(0x7f12380d29c0 0x7f12380b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12380b3710 con 0x7f12380cf830 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f122c001670 con 0x7f12380cf830 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 >> v1:172.21.15.59:6789/0 conn(0x7f12380cf830 legacy=0x7f12380d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12200039b0 0x7f1220003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12200043a0 0x7f1220004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1220004ef0 0x7f122000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12380b31e0 con 0x7f1220004ef0 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12380b3d60 con 0x7f12200043a0 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122000bf00 con 0x7f12200039b0 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1237fff640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f123815dfe0 0x7f12380cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f123cdd6640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1220004ef0 0x7f122000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f12377fe640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12200043a0 0x7f1220004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.244+0000 7f1237fff640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12200039b0 0x7f1220003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.245+0000 7f123cdd6640 1 -- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12200039b0 msgr2=0x7f1220003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.245+0000 7f123cdd6640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12200039b0 0x7f1220003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.245+0000 7f123cdd6640 1 -- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12200043a0 msgr2=0x7f1220004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.245+0000 7f123cdd6640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12200043a0 0x7f1220004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.245+0000 7f123cdd6640 1 -- 172.21.15.59:0/729965764 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1220003520 con 0x7f1220004ef0 2024-03-06T07:31:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.245+0000 7f123cdd6640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1220004ef0 0x7f122000b9c0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f1230009fc0 tx=0x7f1220005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0953c44a913abe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.245+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230039070 con 0x7f1220004ef0 2024-03-06T07:31:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.245+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123002cc60 con 0x7f1220004ef0 2024-03-06T07:31:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.245+0000 7f1236ffd640 1 -- 172.21.15.59:0/729965764 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230009c10 con 0x7f1220004ef0 2024-03-06T07:31:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.246+0000 7f123e839640 1 -- 172.21.15.59:0/729965764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1220004ef0 msgr2=0x7f122000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.246+0000 7f123e839640 1 --2- 172.21.15.59:0/729965764 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1220004ef0 0x7f122000b9c0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f1230009fc0 tx=0x7f1220005830 comp rx=0 tx=0).stop 2024-03-06T07:31:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.246+0000 7f123cdd6640 1 -- 172.21.15.59:0/729965764 reap_dead start 2024-03-06T07:31:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.246+0000 7f123e839640 1 -- 172.21.15.59:0/729965764 shutdown_connections 2024-03-06T07:31:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.246+0000 7f123e839640 1 -- 172.21.15.59:0/729965764 >> 172.21.15.59:0/729965764 conn(0x7f12380d3cd0 msgr2=0x7f12380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.246+0000 7f123e839640 1 -- 172.21.15.59:0/729965764 shutdown_connections 2024-03-06T07:31:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.246+0000 7f123e839640 1 -- 172.21.15.59:0/729965764 wait complete. 2024-03-06T07:31:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.247+0000 7f123e839640 1 Processor -- start 2024-03-06T07:31:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.247+0000 7f123e839640 1 -- start start 2024-03-06T07:31:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.248+0000 7f123e839640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12380b8960 0x7f12380cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.248+0000 7f123e839640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12380b8d40 0x7f12380cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.248+0000 7f123e839640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12380cada0 0x7f12380cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.248+0000 7f123e839640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12380d1df0 con 0x7f12380cada0 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.248+0000 7f123e839640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12380d1c70 con 0x7f12380b8960 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.248+0000 7f123e839640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123815e3c0 con 0x7f12380b8d40 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.248+0000 7f123cdd6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12380cada0 0x7f12380cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.248+0000 7f123cdd6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12380cada0 0x7f12380cb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55654/0 (socket says 172.21.15.59:55654) 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.248+0000 7f123cdd6640 1 -- 172.21.15.59:0/3742012517 learned_addr learned my addr 172.21.15.59:0/3742012517 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.248+0000 7f12377fe640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12380b8d40 0x7f12380cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.248+0000 7f1237fff640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12380b8960 0x7f12380cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f123cdd6640 1 -- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12380b8d40 msgr2=0x7f12380cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f123cdd6640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12380b8d40 0x7f12380cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f123cdd6640 1 -- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12380b8960 msgr2=0x7f12380cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f123cdd6640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12380b8960 0x7f12380cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f123cdd6640 1 -- 172.21.15.59:0/3742012517 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12380cb790 con 0x7f12380cada0 2024-03-06T07:31:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f12377fe640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12380b8d40 0x7f12380cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f123cdd6640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12380cada0 0x7f12380cb250 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f1230037040 tx=0x7f1230009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f1234ff9640 1 -- 172.21.15.59:0/3742012517 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230039030 con 0x7f12380cada0 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f1234ff9640 1 -- 172.21.15.59:0/3742012517 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123003a040 con 0x7f12380cada0 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f123e839640 1 -- 172.21.15.59:0/3742012517 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12380cba60 con 0x7f12380cada0 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f1234ff9640 1 -- 172.21.15.59:0/3742012517 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230009c10 con 0x7f12380cada0 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.249+0000 7f123e839640 1 -- 172.21.15.59:0/3742012517 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12380cc020 con 0x7f12380cada0 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.250+0000 7f123e839640 1 -- 172.21.15.59:0/3742012517 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123804f330 con 0x7f12380cada0 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.251+0000 7f1234ff9640 1 -- 172.21.15.59:0/3742012517 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1230045030 con 0x7f12380cada0 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.251+0000 7f1234ff9640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f122406b500 0x7f122406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.251+0000 7f1234ff9640 1 -- 172.21.15.59:0/3742012517 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f12300abf10 con 0x7f12380cada0 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.251+0000 7f1237fff640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f122406b500 0x7f122406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:14.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.252+0000 7f1237fff640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f122406b500 0x7f122406d9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f123815cc80 tx=0x7f1228008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:14.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.255+0000 7f1234ff9640 1 -- 172.21.15.59:0/3742012517 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f123004c060 con 0x7f12380cada0 2024-03-06T07:31:14.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.474+0000 7f123e839640 1 -- 172.21.15.59:0/3742012517 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12380b2fc0 con 0x7f12380cada0 2024-03-06T07:31:14.476 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.474+0000 7f1234ff9640 1 -- 172.21.15.59:0/3742012517 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f123007ae00 con 0x7f12380cada0 2024-03-06T07:31:14.477 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:14.477 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:14.477 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.476+0000 7f123e839640 1 -- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f122406b500 msgr2=0x7f122406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.476+0000 7f123e839640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f122406b500 0x7f122406d9c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f123815cc80 tx=0x7f1228008040 comp rx=0 tx=0).stop 2024-03-06T07:31:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.476+0000 7f123e839640 1 -- 172.21.15.59:0/3742012517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12380cada0 msgr2=0x7f12380cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.476+0000 7f123e839640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12380cada0 0x7f12380cb250 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f1230037040 tx=0x7f1230009750 comp rx=0 tx=0).stop 2024-03-06T07:31:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.476+0000 7f123e839640 1 -- 172.21.15.59:0/3742012517 shutdown_connections 2024-03-06T07:31:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.476+0000 7f123e839640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f122406b500 0x7f122406d9c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.476+0000 7f123e839640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f12380cada0 0x7f12380cb250 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.476+0000 7f123e839640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12380b8d40 0x7f12380cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.476+0000 7f123e839640 1 --2- 172.21.15.59:0/3742012517 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12380b8960 0x7f12380cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.476+0000 7f123e839640 1 -- 172.21.15.59:0/3742012517 >> 172.21.15.59:0/3742012517 conn(0x7f12380d3cd0 msgr2=0x7f12380b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.479 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.477+0000 7f123e839640 1 -- 172.21.15.59:0/3742012517 shutdown_connections 2024-03-06T07:31:14.480 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:14.477+0000 7f123e839640 1 -- 172.21.15.59:0/3742012517 wait complete. 2024-03-06T07:31:14.691 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:14.691 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:15.692 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:15.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.792+0000 7f5a189b5640 1 Processor -- start 2024-03-06T07:31:15.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.793+0000 7f5a189b5640 1 -- start start 2024-03-06T07:31:15.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a189b5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a1413dc10 0x7f5a140b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a189b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5a1415ef60 0x7f5a140b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a189b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1415f580 con 0x7f5a1415bf30 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a189b5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a140cf3f0 con 0x7f5a1413dc10 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a189b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1415b210 con 0x7f5a1415ef60 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a1259c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a1413dc10 0x7f5a140b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a1259c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a1413dc10 0x7f5a140b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55656/0 (socket says 172.21.15.59:55656) 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a1259c640 1 -- 172.21.15.59:0/3392830313 learned_addr learned my addr 172.21.15.59:0/3392830313 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a12d9d640 1 --2- 172.21.15.59:0/3392830313 >> v2:172.21.15.164:3300/0 conn(0x7f5a1415ef60 0x7f5a140b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a1259c640 1 -- 172.21.15.59:0/3392830313 >> v2:172.21.15.164:3300/0 conn(0x7f5a1415ef60 msgr2=0x7f5a140b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a1259c640 1 --2- 172.21.15.59:0/3392830313 >> v2:172.21.15.164:3300/0 conn(0x7f5a1415ef60 0x7f5a140b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a1259c640 1 -- 172.21.15.59:0/3392830313 >> v1:172.21.15.164:6789/0 conn(0x7f5a1415bf30 legacy=0x7f5a1415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.794+0000 7f5a1259c640 1 -- 172.21.15.59:0/3392830313 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1415b450 con 0x7f5a1413dc10 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.795+0000 7f5a1259c640 1 --2- 172.21.15.59:0/3392830313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1413dc10 0x7f5a140b2950 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f5a14068700 tx=0x7f5a0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441dd378f3d75872 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.795+0000 7f5a11d9b640 1 -- 172.21.15.59:0/3392830313 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04035020 con 0x7f5a1413dc10 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.795+0000 7f5a11d9b640 1 -- 172.21.15.59:0/3392830313 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a04031870 con 0x7f5a1413dc10 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.795+0000 7f5a11d9b640 1 -- 172.21.15.59:0/3392830313 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04031c40 con 0x7f5a1413dc10 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.796+0000 7f5a189b5640 1 -- 172.21.15.59:0/3392830313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1413dc10 msgr2=0x7f5a140b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.796+0000 7f5a189b5640 1 --2- 172.21.15.59:0/3392830313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1413dc10 0x7f5a140b2950 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f5a14068700 tx=0x7f5a0402b910 comp rx=0 tx=0).stop 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.796+0000 7f5a189b5640 1 -- 172.21.15.59:0/3392830313 shutdown_connections 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.796+0000 7f5a189b5640 1 --2- 172.21.15.59:0/3392830313 >> v2:172.21.15.164:3300/0 conn(0x7f5a1415ef60 0x7f5a140b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.796+0000 7f5a189b5640 1 --2- 172.21.15.59:0/3392830313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1413dc10 0x7f5a140b2950 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.796+0000 7f5a189b5640 1 -- 172.21.15.59:0/3392830313 >> 172.21.15.59:0/3392830313 conn(0x7f5a140cf8a0 msgr2=0x7f5a140cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.796+0000 7f5a189b5640 1 -- 172.21.15.59:0/3392830313 shutdown_connections 2024-03-06T07:31:15.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.796+0000 7f5a189b5640 1 -- 172.21.15.59:0/3392830313 wait complete. 2024-03-06T07:31:15.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.797+0000 7f5a189b5640 1 Processor -- start 2024-03-06T07:31:15.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.797+0000 7f5a189b5640 1 -- start start 2024-03-06T07:31:15.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.798+0000 7f5a189b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a1413dc10 0x7f5a14153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.798+0000 7f5a189b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1415bf30 0x7f5a141f10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.798+0000 7f5a189b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a1415ef60 0x7f5a1413c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.798+0000 7f5a189b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a140708d0 con 0x7f5a1415bf30 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.798+0000 7f5a189b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1413dff0 con 0x7f5a1415ef60 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.798+0000 7f5a189b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a14070a50 con 0x7f5a1413dc10 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.798+0000 7f5a0bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1415bf30 0x7f5a141f10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.798+0000 7f5a0bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1415bf30 0x7f5a141f10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55660/0 (socket says 172.21.15.59:55660) 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.798+0000 7f5a0bfff640 1 -- 172.21.15.59:0/3737000727 learned_addr learned my addr 172.21.15.59:0/3737000727 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a1259c640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a1413dc10 0x7f5a14153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a0bfff640 1 -- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a1413dc10 msgr2=0x7f5a14153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a12d9d640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a1415ef60 0x7f5a1413c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a0bfff640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a1413dc10 0x7f5a14153590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a0bfff640 1 -- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a1415ef60 msgr2=0x7f5a1413c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a0bfff640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a1415ef60 0x7f5a1413c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a0bfff640 1 -- 172.21.15.59:0/3737000727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a141403f0 con 0x7f5a1415bf30 2024-03-06T07:31:15.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a0bfff640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1415bf30 0x7f5a141f10a0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c400 tx=0x7f59fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a0affd640 1 -- 172.21.15.59:0/3737000727 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0052d0 con 0x7f5a1415bf30 2024-03-06T07:31:15.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a0affd640 1 -- 172.21.15.59:0/3737000727 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59fc005490 con 0x7f5a1415bf30 2024-03-06T07:31:15.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a0affd640 1 -- 172.21.15.59:0/3737000727 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc007260 con 0x7f5a1415bf30 2024-03-06T07:31:15.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a189b5640 1 -- 172.21.15.59:0/3737000727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a141406d0 con 0x7f5a1415bf30 2024-03-06T07:31:15.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.799+0000 7f5a189b5640 1 -- 172.21.15.59:0/3737000727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a14141810 con 0x7f5a1415bf30 2024-03-06T07:31:15.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.800+0000 7f5a189b5640 1 -- 172.21.15.59:0/3737000727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1404f330 con 0x7f5a1415bf30 2024-03-06T07:31:15.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.801+0000 7f5a0affd640 1 -- 172.21.15.59:0/3737000727 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59fc007400 con 0x7f5a1415bf30 2024-03-06T07:31:15.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.801+0000 7f5a0affd640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f59f406b4d0 0x7f59f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.801+0000 7f5a0affd640 1 -- 172.21.15.59:0/3737000727 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f59fc0897a0 con 0x7f5a1415bf30 2024-03-06T07:31:15.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.801+0000 7f5a1259c640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f59f406b4d0 0x7f59f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.802+0000 7f5a1259c640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f59f406b4d0 0x7f59f406d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f5a04036000 tx=0x7f5a04008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:15.806+0000 7f5a0affd640 1 -- 172.21.15.59:0/3737000727 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59fc058600 con 0x7f5a1415bf30 2024-03-06T07:31:16.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.023+0000 7f5a189b5640 1 -- 172.21.15.59:0/3737000727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a1415ad30 con 0x7f5a1415bf30 2024-03-06T07:31:16.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.024+0000 7f5a0affd640 1 -- 172.21.15.59:0/3737000727 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f59fc057f00 con 0x7f5a1415bf30 2024-03-06T07:31:16.026 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:16.026 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:16.026 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:16.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 -- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f59f406b4d0 msgr2=0x7f59f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f59f406b4d0 0x7f59f406d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f5a04036000 tx=0x7f5a04008030 comp rx=0 tx=0).stop 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 -- 172.21.15.59:0/3737000727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1415bf30 msgr2=0x7f5a141f10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1415bf30 0x7f5a141f10a0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c400 tx=0x7f59fc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 -- 172.21.15.59:0/3737000727 shutdown_connections 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f59f406b4d0 0x7f59f406d990 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a1415ef60 0x7f5a1413c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a1415bf30 0x7f5a141f10a0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 --2- 172.21.15.59:0/3737000727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a1413dc10 0x7f5a14153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 -- 172.21.15.59:0/3737000727 >> 172.21.15.59:0/3737000727 conn(0x7f5a140cf8a0 msgr2=0x7f5a14072140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.028 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 -- 172.21.15.59:0/3737000727 shutdown_connections 2024-03-06T07:31:16.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:16.025+0000 7f5a189b5640 1 -- 172.21.15.59:0/3737000727 wait complete. 2024-03-06T07:31:16.243 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:16.243 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:17.244 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:17.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.358+0000 7f8e6f981640 1 Processor -- start 2024-03-06T07:31:17.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.358+0000 7f8e6f981640 1 -- start start 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6f981640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e680b3060 0x7f8e6815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6f981640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8e6815bf30 0x7f8e6815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.367 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6f981640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e6815f2d0 con 0x7f8e6813dc10 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6f981640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e680cf3f0 con 0x7f8e6815bf30 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6f981640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6815b210 con 0x7f8e680b3060 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6df1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8e6815bf30 0x7f8e6815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6df1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8e6815bf30 0x7f8e6815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55670/0 (socket says 172.21.15.59:55670) 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6df1e640 1 -- 172.21.15.59:0/1128300546 learned_addr learned my addr 172.21.15.59:0/1128300546 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6cf1c640 1 --2- 172.21.15.59:0/1128300546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e680b3060 0x7f8e6815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6df1e640 1 -- 172.21.15.59:0/1128300546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e680b3060 msgr2=0x7f8e6815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6df1e640 1 --2- 172.21.15.59:0/1128300546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e680b3060 0x7f8e6815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6df1e640 1 -- 172.21.15.59:0/1128300546 >> v1:172.21.15.164:6789/0 conn(0x7f8e6813dc10 legacy=0x7f8e680b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6df1e640 1 -- 172.21.15.59:0/1128300546 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6815f580 con 0x7f8e6815bf30 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.359+0000 7f8e6cf1c640 1 --2- 172.21.15.59:0/1128300546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e680b3060 0x7f8e6815e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:17.368 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.360+0000 7f8e6df1e640 1 --2- 172.21.15.59:0/1128300546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6815bf30 0x7f8e6815ed90 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009e50 tx=0x7f8e6002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9951965111b6575 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.360+0000 7f8e5ffff640 1 -- 172.21.15.59:0/1128300546 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6002ca70 con 0x7f8e6815bf30 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.360+0000 7f8e5ffff640 1 -- 172.21.15.59:0/1128300546 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6002cc10 con 0x7f8e6815bf30 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.360+0000 7f8e5ffff640 1 -- 172.21.15.59:0/1128300546 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e600343f0 con 0x7f8e6815bf30 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.360+0000 7f8e6f981640 1 -- 172.21.15.59:0/1128300546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6815bf30 msgr2=0x7f8e6815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.361+0000 7f8e6f981640 1 --2- 172.21.15.59:0/1128300546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6815bf30 0x7f8e6815ed90 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009e50 tx=0x7f8e6002bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.361+0000 7f8e6f981640 1 -- 172.21.15.59:0/1128300546 shutdown_connections 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.361+0000 7f8e6f981640 1 --2- 172.21.15.59:0/1128300546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6815bf30 0x7f8e6815ed90 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.361+0000 7f8e6f981640 1 --2- 172.21.15.59:0/1128300546 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e680b3060 0x7f8e6815e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.361+0000 7f8e6f981640 1 -- 172.21.15.59:0/1128300546 >> 172.21.15.59:0/1128300546 conn(0x7f8e680cf8a0 msgr2=0x7f8e680cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.361+0000 7f8e6f981640 1 -- 172.21.15.59:0/1128300546 shutdown_connections 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.361+0000 7f8e6f981640 1 -- 172.21.15.59:0/1128300546 wait complete. 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.362+0000 7f8e6f981640 1 Processor -- start 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.362+0000 7f8e6f981640 1 -- start start 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.362+0000 7f8e6f981640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e680b3060 0x7f8e68153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6f981640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e6813dc10 0x7f8e68159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6f981640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6815bf30 0x7f8e68152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6f981640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6813dff0 con 0x7f8e6815bf30 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6f981640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6815b730 con 0x7f8e6813dc10 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6f981640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e680708d0 con 0x7f8e680b3060 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6df1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6815bf30 0x7f8e68152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6df1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6815bf30 0x7f8e68152380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55686/0 (socket says 172.21.15.59:55686) 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6df1e640 1 -- 172.21.15.59:0/2179024609 learned_addr learned my addr 172.21.15.59:0/2179024609 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6d71d640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e680b3060 0x7f8e68153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6cf1c640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e6813dc10 0x7f8e68159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6df1e640 1 -- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e680b3060 msgr2=0x7f8e68153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6df1e640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e680b3060 0x7f8e68153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6df1e640 1 -- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e6813dc10 msgr2=0x7f8e68159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6df1e640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e6813dc10 0x7f8e68159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6df1e640 1 -- 172.21.15.59:0/2179024609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e68152d60 con 0x7f8e6815bf30 2024-03-06T07:31:17.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6cf1c640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e6813dc10 0x7f8e68159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:17.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6df1e640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6815bf30 0x7f8e68152380 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009e50 tx=0x7f8e60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e5dffb640 1 -- 172.21.15.59:0/2179024609 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e60043020 con 0x7f8e6815bf30 2024-03-06T07:31:17.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e5dffb640 1 -- 172.21.15.59:0/2179024609 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6003e450 con 0x7f8e6815bf30 2024-03-06T07:31:17.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e6f981640 1 -- 172.21.15.59:0/2179024609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e68153040 con 0x7f8e6815bf30 2024-03-06T07:31:17.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.363+0000 7f8e5dffb640 1 -- 172.21.15.59:0/2179024609 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e60009510 con 0x7f8e6815bf30 2024-03-06T07:31:17.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.364+0000 7f8e6f981640 1 -- 172.21.15.59:0/2179024609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e68153600 con 0x7f8e6815bf30 2024-03-06T07:31:17.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.364+0000 7f8e6f981640 1 -- 172.21.15.59:0/2179024609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e68153af0 con 0x7f8e6815bf30 2024-03-06T07:31:17.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.365+0000 7f8e5dffb640 1 -- 172.21.15.59:0/2179024609 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e60046020 con 0x7f8e6815bf30 2024-03-06T07:31:17.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.365+0000 7f8e5dffb640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e4c06b4d0 0x7f8e4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.365+0000 7f8e5dffb640 1 -- 172.21.15.59:0/2179024609 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8e600ac8d0 con 0x7f8e6815bf30 2024-03-06T07:31:17.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.366+0000 7f8e6d71d640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e4c06b4d0 0x7f8e4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.366+0000 7f8e6d71d640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e4c06b4d0 0x7f8e4c06d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8e64001780 tx=0x7f8e64009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.374+0000 7f8e5dffb640 1 -- 172.21.15.59:0/2179024609 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e6004d030 con 0x7f8e6815bf30 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.593+0000 7f8e6f981640 1 -- 172.21.15.59:0/2179024609 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e6806eb00 con 0x7f8e6815bf30 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.593+0000 7f8e5dffb640 1 -- 172.21.15.59:0/2179024609 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f8e6007b7c0 con 0x7f8e6815bf30 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:17.596 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:17.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 -- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e4c06b4d0 msgr2=0x7f8e4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e4c06b4d0 0x7f8e4c06d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8e64001780 tx=0x7f8e64009040 comp rx=0 tx=0).stop 2024-03-06T07:31:17.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 -- 172.21.15.59:0/2179024609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6815bf30 msgr2=0x7f8e68152380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6815bf30 0x7f8e68152380 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009e50 tx=0x7f8e60004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:17.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 -- 172.21.15.59:0/2179024609 shutdown_connections 2024-03-06T07:31:17.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8e4c06b4d0 0x7f8e4c06d990 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e6815bf30 0x7f8e68152380 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e6813dc10 0x7f8e68159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 --2- 172.21.15.59:0/2179024609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e680b3060 0x7f8e68153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 -- 172.21.15.59:0/2179024609 >> 172.21.15.59:0/2179024609 conn(0x7f8e680cf8a0 msgr2=0x7f8e68072360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 -- 172.21.15.59:0/2179024609 shutdown_connections 2024-03-06T07:31:17.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:17.595+0000 7f8e6f981640 1 -- 172.21.15.59:0/2179024609 wait complete. 2024-03-06T07:31:17.809 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:17.810 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:18.811 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:18.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.950+0000 7f597d5a0640 1 Processor -- start 2024-03-06T07:31:18.952 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.950+0000 7f597d5a0640 1 -- start start 2024-03-06T07:31:18.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.951+0000 7f597d5a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f597815de40 0x7f59780d1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.953 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.951+0000 7f597d5a0640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f59780d1ca0 0x7f597806c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.951+0000 7f597d5a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f597806c8a0 0x7f59781618c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.951+0000 7f597d5a0640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59780cfab0 con 0x7f59780d1ca0 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.951+0000 7f597d5a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59780d2100 con 0x7f597806c8a0 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.951+0000 7f597d5a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597815e220 con 0x7f597815de40 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.951+0000 7f59777fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f59780d1ca0 0x7f597806c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.951+0000 7f59777fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f59780d1ca0 0x7f597806c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55702/0 (socket says 172.21.15.59:55702) 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.951+0000 7f59777fe640 1 -- 172.21.15.59:0/757707905 learned_addr learned my addr 172.21.15.59:0/757707905 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.952+0000 7f5977fff640 1 --2- 172.21.15.59:0/757707905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f597815de40 0x7f59780d1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.952+0000 7f597cd9f640 1 --2- 172.21.15.59:0/757707905 >> v2:172.21.15.164:3300/0 conn(0x7f597806c8a0 0x7f59781618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.952+0000 7f59777fe640 1 -- 172.21.15.59:0/757707905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f597815de40 msgr2=0x7f59780d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.952+0000 7f59777fe640 1 --2- 172.21.15.59:0/757707905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f597815de40 0x7f59780d1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.952+0000 7f59777fe640 1 -- 172.21.15.59:0/757707905 >> v2:172.21.15.164:3300/0 conn(0x7f597806c8a0 msgr2=0x7f59781618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.952+0000 7f59777fe640 1 --2- 172.21.15.59:0/757707905 >> v2:172.21.15.164:3300/0 conn(0x7f597806c8a0 0x7f59781618c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.954 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.952+0000 7f59777fe640 1 -- 172.21.15.59:0/757707905 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597806d090 con 0x7f59780d1ca0 2024-03-06T07:31:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.953+0000 7f59777fe640 1 --2- 172.21.15.59:0/757707905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59780d1ca0 0x7f597806c360 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f59780b77a0 tx=0x7f596802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf2074e57b50030 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.953+0000 7f5976ffd640 1 -- 172.21.15.59:0/757707905 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596802e070 con 0x7f59780d1ca0 2024-03-06T07:31:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.953+0000 7f5976ffd640 1 -- 172.21.15.59:0/757707905 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5968037070 con 0x7f59780d1ca0 2024-03-06T07:31:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.953+0000 7f5976ffd640 1 -- 172.21.15.59:0/757707905 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968031a00 con 0x7f59780d1ca0 2024-03-06T07:31:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.954+0000 7f597d5a0640 1 -- 172.21.15.59:0/757707905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59780d1ca0 msgr2=0x7f597806c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.954+0000 7f597d5a0640 1 --2- 172.21.15.59:0/757707905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59780d1ca0 0x7f597806c360 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f59780b77a0 tx=0x7f596802b880 comp rx=0 tx=0).stop 2024-03-06T07:31:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.954+0000 7f597d5a0640 1 -- 172.21.15.59:0/757707905 shutdown_connections 2024-03-06T07:31:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.954+0000 7f597d5a0640 1 --2- 172.21.15.59:0/757707905 >> v2:172.21.15.164:3300/0 conn(0x7f597806c8a0 0x7f59781618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.954+0000 7f597d5a0640 1 --2- 172.21.15.59:0/757707905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59780d1ca0 0x7f597806c360 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.954+0000 7f597d5a0640 1 --2- 172.21.15.59:0/757707905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f597815de40 0x7f59780d1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.956 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.954+0000 7f597d5a0640 1 -- 172.21.15.59:0/757707905 >> 172.21.15.59:0/757707905 conn(0x7f597809a9f0 msgr2=0x7f597809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.954+0000 7f597d5a0640 1 -- 172.21.15.59:0/757707905 shutdown_connections 2024-03-06T07:31:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.954+0000 7f597d5a0640 1 -- 172.21.15.59:0/757707905 wait complete. 2024-03-06T07:31:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.956+0000 7f597d5a0640 1 Processor -- start 2024-03-06T07:31:18.957 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.956+0000 7f597d5a0640 1 -- start start 2024-03-06T07:31:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f597d5a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f597806c8a0 0x7f59780b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f597d5a0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59780d1ca0 0x7f59780c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f597d5a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f597815de40 0x7f59780b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f597d5a0640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59780cfce0 con 0x7f59780d1ca0 2024-03-06T07:31:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f597d5a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597806d650 con 0x7f597815de40 2024-03-06T07:31:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f597d5a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597806cd30 con 0x7f597806c8a0 2024-03-06T07:31:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f59777fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59780d1ca0 0x7f59780c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.960 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f59777fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59780d1ca0 0x7f59780c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55710/0 (socket says 172.21.15.59:55710) 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f59777fe640 1 -- 172.21.15.59:0/281740003 learned_addr learned my addr 172.21.15.59:0/281740003 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f5977fff640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f597806c8a0 0x7f59780b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f597cd9f640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f597815de40 0x7f59780b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f59777fe640 1 -- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f597806c8a0 msgr2=0x7f59780b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f59777fe640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f597806c8a0 0x7f59780b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f59777fe640 1 -- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f597815de40 msgr2=0x7f59780b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f59777fe640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f597815de40 0x7f59780b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f59777fe640 1 -- 172.21.15.59:0/281740003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59780c53a0 con 0x7f59780d1ca0 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f597cd9f640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f597815de40 0x7f59780b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.957+0000 7f5977fff640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f597806c8a0 0x7f59780b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.958+0000 7f59777fe640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59780d1ca0 0x7f59780c7c80 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f5968000c00 tx=0x7f59680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.958+0000 7f5974ff9640 1 -- 172.21.15.59:0/281740003 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968032880 con 0x7f59780d1ca0 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.958+0000 7f5974ff9640 1 -- 172.21.15.59:0/281740003 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5968032a20 con 0x7f59780d1ca0 2024-03-06T07:31:18.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.958+0000 7f597d5a0640 1 -- 172.21.15.59:0/281740003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59780c5680 con 0x7f59780d1ca0 2024-03-06T07:31:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.958+0000 7f5974ff9640 1 -- 172.21.15.59:0/281740003 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968032e10 con 0x7f59780d1ca0 2024-03-06T07:31:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.958+0000 7f597d5a0640 1 -- 172.21.15.59:0/281740003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59780c5c40 con 0x7f59780d1ca0 2024-03-06T07:31:18.962 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.959+0000 7f597d5a0640 1 -- 172.21.15.59:0/281740003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5978068f80 con 0x7f59780d1ca0 2024-03-06T07:31:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.959+0000 7f5974ff9640 1 -- 172.21.15.59:0/281740003 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f596804c060 con 0x7f59780d1ca0 2024-03-06T07:31:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.960+0000 7f5974ff9640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f595806b4d0 0x7f595806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.960+0000 7f5974ff9640 1 -- 172.21.15.59:0/281740003 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f59680ab890 con 0x7f59780d1ca0 2024-03-06T07:31:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.960+0000 7f5977fff640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f595806b4d0 0x7f595806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.963 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.961+0000 7f5977fff640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f595806b4d0 0x7f595806d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5960006fd0 tx=0x7f5960008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:18.964+0000 7f5974ff9640 1 -- 172.21.15.59:0/281740003 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5968040070 con 0x7f59780d1ca0 2024-03-06T07:31:19.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.197+0000 7f597d5a0640 1 -- 172.21.15.59:0/281740003 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59780b77f0 con 0x7f59780d1ca0 2024-03-06T07:31:19.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.198+0000 7f5974ff9640 1 -- 172.21.15.59:0/281740003 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f596807a780 con 0x7f59780d1ca0 2024-03-06T07:31:19.200 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:19.200 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:19.200 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 -- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f595806b4d0 msgr2=0x7f595806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f595806b4d0 0x7f595806d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5960006fd0 tx=0x7f5960008040 comp rx=0 tx=0).stop 2024-03-06T07:31:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 -- 172.21.15.59:0/281740003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59780d1ca0 msgr2=0x7f59780c7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59780d1ca0 0x7f59780c7c80 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f5968000c00 tx=0x7f59680046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 -- 172.21.15.59:0/281740003 shutdown_connections 2024-03-06T07:31:19.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f595806b4d0 0x7f595806d990 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f597815de40 0x7f59780b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59780d1ca0 0x7f59780c7c80 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 --2- 172.21.15.59:0/281740003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f597806c8a0 0x7f59780b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 -- 172.21.15.59:0/281740003 >> 172.21.15.59:0/281740003 conn(0x7f597809a9f0 msgr2=0x7f59780d0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 -- 172.21.15.59:0/281740003 shutdown_connections 2024-03-06T07:31:19.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:19.200+0000 7f597d5a0640 1 -- 172.21.15.59:0/281740003 wait complete. 2024-03-06T07:31:19.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:19.414 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:20.415 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:20.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.515+0000 7f0ffda78640 1 Processor -- start 2024-03-06T07:31:20.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.515+0000 7f0ffda78640 1 -- start start 2024-03-06T07:31:20.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.516+0000 7f0ffda78640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0ff80d00c0 0x7f0ff80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.516+0000 7f0ffda78640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff806cf20 0x7f0ff806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.516+0000 7f0ffda78640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff80d09e0 con 0x7f0ff80cf850 2024-03-06T07:31:20.521 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.516+0000 7f0ffda78640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff806ca90 con 0x7f0ff80d00c0 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.516+0000 7f0ffda78640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff815e380 con 0x7f0ff806cf20 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.516+0000 7f0ff6ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0ff80cf850 0x7f0ff80cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:32808/0 (socket says 172.21.15.59:32808) 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.516+0000 7f0ff6ffd640 1 -- 172.21.15.59:0/3920676914 learned_addr learned my addr 172.21.15.59:0/3920676914 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff7fff640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff806cf20 0x7f0ff806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff77fe640 1 --2- 172.21.15.59:0/3920676914 >> v2:172.21.15.164:3300/0 conn(0x7f0ff80d00c0 0x7f0ff80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882747134 0 0) 0x7f0ff80d09e0 con 0x7f0ff80cf850 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe4003520 con 0x7f0ff80cf850 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1107365839 0 0) 0x7f0fe4003520 con 0x7f0ff80cf850 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff806cf20 msgr2=0x7f0ff806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff806cf20 0x7f0ff806d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 >> v2:172.21.15.164:3300/0 conn(0x7f0ff80d00c0 msgr2=0x7f0ff80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 --2- 172.21.15.59:0/3920676914 >> v2:172.21.15.164:3300/0 conn(0x7f0ff80d00c0 0x7f0ff80d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff80cdc10 con 0x7f0ff80cf850 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f0fe0001670 con 0x7f0ff80cf850 2024-03-06T07:31:20.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 >> v1:172.21.15.59:6789/0 conn(0x7f0ff80cf850 legacy=0x7f0ff80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fe40039b0 0x7f0fe4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 0x7f0fe4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff806cf20 0x7f0fe40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff806ca90 con 0x7f0fe4004800 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff815e380 con 0x7f0fe40039b0 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe400b9c0 con 0x7f0ff806cf20 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff6ffd640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 0x7f0fe4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff77fe640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fe40039b0 0x7f0fe4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff7fff640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff806cf20 0x7f0fe40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff6ffd640 1 -- 172.21.15.59:0/3920676914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff806cf20 msgr2=0x7f0fe40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff6ffd640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff806cf20 0x7f0fe40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff6ffd640 1 -- 172.21.15.59:0/3920676914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fe40039b0 msgr2=0x7f0fe4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff6ffd640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fe40039b0 0x7f0fe4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.517+0000 7f0ff6ffd640 1 -- 172.21.15.59:0/3920676914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe4003520 con 0x7f0fe4004800 2024-03-06T07:31:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.518+0000 7f0ff6ffd640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 0x7f0fe4004c80 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0001050 tx=0x7f0fe4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76a56ab6f1a9fff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.518+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0007d70 con 0x7f0fe4004800 2024-03-06T07:31:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.518+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe0030070 con 0x7f0fe4004800 2024-03-06T07:31:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.518+0000 7f0ff67fc640 1 -- 172.21.15.59:0/3920676914 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe002cd80 con 0x7f0fe4004800 2024-03-06T07:31:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.519+0000 7f0ffda78640 1 -- 172.21.15.59:0/3920676914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 msgr2=0x7f0fe4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.519+0000 7f0ffda78640 1 --2- 172.21.15.59:0/3920676914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 0x7f0fe4004c80 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0001050 tx=0x7f0fe4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.519+0000 7f0ff7fff640 1 -- 172.21.15.59:0/3920676914 reap_dead start 2024-03-06T07:31:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.519+0000 7f0ffda78640 1 -- 172.21.15.59:0/3920676914 shutdown_connections 2024-03-06T07:31:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.519+0000 7f0ffda78640 1 -- 172.21.15.59:0/3920676914 >> 172.21.15.59:0/3920676914 conn(0x7f0ff80b35a0 msgr2=0x7f0ff80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.519+0000 7f0ffda78640 1 -- 172.21.15.59:0/3920676914 shutdown_connections 2024-03-06T07:31:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.519+0000 7f0ffda78640 1 -- 172.21.15.59:0/3920676914 wait complete. 2024-03-06T07:31:20.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.520+0000 7f0ffda78640 1 Processor -- start 2024-03-06T07:31:20.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.520+0000 7f0ffda78640 1 -- start start 2024-03-06T07:31:20.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ffda78640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 0x7f0ff815ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ffda78640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff80c7f70 0x7f0ff80c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ffda78640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ff80af860 0x7f0ff80afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ffda78640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff806c720 con 0x7f0fe4004800 2024-03-06T07:31:20.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ffda78640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff806c440 con 0x7f0ff80af860 2024-03-06T07:31:20.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ffda78640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff80b1030 con 0x7f0ff80c7f70 2024-03-06T07:31:20.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 0x7f0ff815ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.527 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 0x7f0ff815ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55734/0 (socket says 172.21.15.59:55734) 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff77fe640 1 -- 172.21.15.59:0/1103677832 learned_addr learned my addr 172.21.15.59:0/1103677832 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff6ffd640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff80c7f70 0x7f0ff80c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff7fff640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ff80af860 0x7f0ff80afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff77fe640 1 -- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff80c7f70 msgr2=0x7f0ff80c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff77fe640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff80c7f70 0x7f0ff80c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff77fe640 1 -- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ff80af860 msgr2=0x7f0ff80afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff77fe640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ff80af860 0x7f0ff80afcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff77fe640 1 -- 172.21.15.59:0/1103677832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff80c56d0 con 0x7f0fe4004800 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff6ffd640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff80c7f70 0x7f0ff80c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.521+0000 7f0ff7fff640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ff80af860 0x7f0ff80afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.522+0000 7f0ff77fe640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 0x7f0ff815ace0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8011960 tx=0x7f0fe8011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.522+0000 7f0feffff640 1 -- 172.21.15.59:0/1103677832 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe800faf0 con 0x7f0fe4004800 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.522+0000 7f0feffff640 1 -- 172.21.15.59:0/1103677832 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe800fc90 con 0x7f0fe4004800 2024-03-06T07:31:20.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.522+0000 7f0ffda78640 1 -- 172.21.15.59:0/1103677832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff80c59b0 con 0x7f0fe4004800 2024-03-06T07:31:20.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.522+0000 7f0feffff640 1 -- 172.21.15.59:0/1103677832 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8014440 con 0x7f0fe4004800 2024-03-06T07:31:20.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.522+0000 7f0ffda78640 1 -- 172.21.15.59:0/1103677832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff80c5f70 con 0x7f0fe4004800 2024-03-06T07:31:20.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.523+0000 7f0ffda78640 1 -- 172.21.15.59:0/1103677832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff804f330 con 0x7f0fe4004800 2024-03-06T07:31:20.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.523+0000 7f0feffff640 1 -- 172.21.15.59:0/1103677832 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fe8012040 con 0x7f0fe4004800 2024-03-06T07:31:20.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.524+0000 7f0feffff640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0fd806b500 0x7f0fd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.524+0000 7f0feffff640 1 -- 172.21.15.59:0/1103677832 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0fe808a850 con 0x7f0fe4004800 2024-03-06T07:31:20.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.524+0000 7f0ff6ffd640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0fd806b500 0x7f0fd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.525+0000 7f0ff6ffd640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0fd806b500 0x7f0fd806d9c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0034000 tx=0x7f0fe0007000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.528+0000 7f0feffff640 1 -- 172.21.15.59:0/1103677832 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0fe8019030 con 0x7f0fe4004800 2024-03-06T07:31:20.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.746+0000 7f0ffda78640 1 -- 172.21.15.59:0/1103677832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ff80cbf30 con 0x7f0fe4004800 2024-03-06T07:31:20.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.746+0000 7f0feffff640 1 -- 172.21.15.59:0/1103677832 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f0fe80590f0 con 0x7f0fe4004800 2024-03-06T07:31:20.749 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:20.749 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:20.749 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:20.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 -- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0fd806b500 msgr2=0x7f0fd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0fd806b500 0x7f0fd806d9c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0034000 tx=0x7f0fe0007000 comp rx=0 tx=0).stop 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 -- 172.21.15.59:0/1103677832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 msgr2=0x7f0ff815ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 0x7f0ff815ace0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8011960 tx=0x7f0fe8011e20 comp rx=0 tx=0).stop 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 -- 172.21.15.59:0/1103677832 shutdown_connections 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f0fd806b500 0x7f0fd806d9c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ff80af860 0x7f0ff80afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ff80c7f70 0x7f0ff80c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 --2- 172.21.15.59:0/1103677832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe4004800 0x7f0ff815ace0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 -- 172.21.15.59:0/1103677832 >> 172.21.15.59:0/1103677832 conn(0x7f0ff80b35a0 msgr2=0x7f0ff80cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 -- 172.21.15.59:0/1103677832 shutdown_connections 2024-03-06T07:31:20.751 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:20.748+0000 7f0ffda78640 1 -- 172.21.15.59:0/1103677832 wait complete. 2024-03-06T07:31:20.962 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:20.963 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:21.022 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:21.025 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:21.963 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:22.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.062+0000 7f7fd75ee640 1 Processor -- start 2024-03-06T07:31:22.065 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.063+0000 7f7fd75ee640 1 -- start start 2024-03-06T07:31:22.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.064+0000 7f7fd75ee640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7fd015dfe0 0x7f7fd00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.064+0000 7f7fd75ee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fd00d29c0 0x7f7fd00b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.064+0000 7f7fd75ee640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd00b3430 con 0x7f7fd00cf830 2024-03-06T07:31:22.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.064+0000 7f7fd75ee640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fd00b31e0 con 0x7f7fd015dfe0 2024-03-06T07:31:22.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.064+0000 7f7fd75ee640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd00b3d60 con 0x7f7fd00d29c0 2024-03-06T07:31:22.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.064+0000 7f7fd4b89640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7fd00cf830 0x7f7fd00d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:32822/0 (socket says 172.21.15.59:32822) 2024-03-06T07:31:22.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.064+0000 7f7fd4b89640 1 -- 172.21.15.59:0/2408841320 learned_addr learned my addr 172.21.15.59:0/2408841320 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:22.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.064+0000 7f7fd5b8b640 1 --2- 172.21.15.59:0/2408841320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fd00d29c0 0x7f7fd00b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.067 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.064+0000 7f7fd538a640 1 --2- 172.21.15.59:0/2408841320 >> v2:172.21.15.164:3300/0 conn(0x7f7fd015dfe0 0x7f7fd00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.064+0000 7f7fc7fff640 1 -- 172.21.15.59:0/2408841320 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198786910 0 0) 0x7f7fd00b3430 con 0x7f7fd00cf830 2024-03-06T07:31:22.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.064+0000 7f7fc7fff640 1 -- 172.21.15.59:0/2408841320 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fbc003520 con 0x7f7fd00cf830 2024-03-06T07:31:22.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.065+0000 7f7fd5b8b640 1 -- 172.21.15.59:0/2408841320 >> v2:172.21.15.164:3300/0 conn(0x7f7fd015dfe0 msgr2=0x7f7fd00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.065+0000 7f7fd5b8b640 1 --2- 172.21.15.59:0/2408841320 >> v2:172.21.15.164:3300/0 conn(0x7f7fd015dfe0 0x7f7fd00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.065+0000 7f7fc7fff640 1 -- 172.21.15.59:0/2408841320 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2151572983 0 0) 0x7f7fbc003520 con 0x7f7fd00cf830 2024-03-06T07:31:22.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.065+0000 7f7fd5b8b640 1 -- 172.21.15.59:0/2408841320 >> v1:172.21.15.59:6789/0 conn(0x7f7fd00cf830 legacy=0x7f7fd00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.065+0000 7f7fd5b8b640 1 -- 172.21.15.59:0/2408841320 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd00b3710 con 0x7f7fd00d29c0 2024-03-06T07:31:22.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.065+0000 7f7fd538a640 1 --2- 172.21.15.59:0/2408841320 >> v2:172.21.15.164:3300/0 conn(0x7f7fd015dfe0 0x7f7fd00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:22.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fd5b8b640 1 --2- 172.21.15.59:0/2408841320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fd00d29c0 0x7f7fd00b41f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009d90 tx=0x7f7fc802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9d2aa550411875e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fc7fff640 1 -- 172.21.15.59:0/2408841320 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8039070 con 0x7f7fd00d29c0 2024-03-06T07:31:22.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fc7fff640 1 -- 172.21.15.59:0/2408841320 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc802c8a0 con 0x7f7fd00d29c0 2024-03-06T07:31:22.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fc7fff640 1 -- 172.21.15.59:0/2408841320 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0001670 con 0x7f7fd00d29c0 2024-03-06T07:31:22.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fd75ee640 1 -- 172.21.15.59:0/2408841320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fd00d29c0 msgr2=0x7f7fd00b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fd75ee640 1 --2- 172.21.15.59:0/2408841320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fd00d29c0 0x7f7fd00b41f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009d90 tx=0x7f7fc802bd70 comp rx=0 tx=0).stop 2024-03-06T07:31:22.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fd75ee640 1 -- 172.21.15.59:0/2408841320 shutdown_connections 2024-03-06T07:31:22.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fd75ee640 1 --2- 172.21.15.59:0/2408841320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fd00d29c0 0x7f7fd00b41f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fd75ee640 1 --2- 172.21.15.59:0/2408841320 >> v2:172.21.15.164:3300/0 conn(0x7f7fd015dfe0 0x7f7fd00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fd75ee640 1 -- 172.21.15.59:0/2408841320 >> 172.21.15.59:0/2408841320 conn(0x7f7fd00d3cd0 msgr2=0x7f7fd00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fd75ee640 1 -- 172.21.15.59:0/2408841320 shutdown_connections 2024-03-06T07:31:22.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.066+0000 7f7fd75ee640 1 -- 172.21.15.59:0/2408841320 wait complete. 2024-03-06T07:31:22.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.068+0000 7f7fd75ee640 1 Processor -- start 2024-03-06T07:31:22.070 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.068+0000 7f7fd75ee640 1 -- start start 2024-03-06T07:31:22.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.068+0000 7f7fd75ee640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fd00cf830 0x7f7fd00b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.068+0000 7f7fd75ee640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fd00d29c0 0x7f7fd015a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.068+0000 7f7fd75ee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fd015dfe0 0x7f7fd00cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.068+0000 7f7fd75ee640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd00d1df0 con 0x7f7fd00d29c0 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.068+0000 7f7fd75ee640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd00d1c70 con 0x7f7fd00cf830 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.068+0000 7f7fd75ee640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd015e3c0 con 0x7f7fd015dfe0 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.068+0000 7f7fd4b89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fd00d29c0 0x7f7fd015a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.068+0000 7f7fd4b89640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fd00d29c0 0x7f7fd015a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55736/0 (socket says 172.21.15.59:55736) 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.068+0000 7f7fd4b89640 1 -- 172.21.15.59:0/963131758 learned_addr learned my addr 172.21.15.59:0/963131758 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fd5b8b640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fd015dfe0 0x7f7fd00cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fd538a640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fd00cf830 0x7f7fd00b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fd4b89640 1 -- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fd015dfe0 msgr2=0x7f7fd00cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fd4b89640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fd015dfe0 0x7f7fd00cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fd4b89640 1 -- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fd00cf830 msgr2=0x7f7fd00b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fd4b89640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fd00cf830 0x7f7fd00b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fd4b89640 1 -- 172.21.15.59:0/963131758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd015a8a0 con 0x7f7fd00d29c0 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fd538a640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fd00cf830 0x7f7fd00b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fd4b89640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fd00d29c0 0x7f7fd015a360 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00017e0 tx=0x7f7fc000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fc5ffb640 1 -- 172.21.15.59:0/963131758 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0005ad0 con 0x7f7fd00d29c0 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fc5ffb640 1 -- 172.21.15.59:0/963131758 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc0005c70 con 0x7f7fd00d29c0 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fc5ffb640 1 -- 172.21.15.59:0/963131758 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0007290 con 0x7f7fd00d29c0 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fd75ee640 1 -- 172.21.15.59:0/963131758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd00b7f10 con 0x7f7fd00d29c0 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.069+0000 7f7fd75ee640 1 -- 172.21.15.59:0/963131758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd015fa70 con 0x7f7fd00d29c0 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.070+0000 7f7fd75ee640 1 -- 172.21.15.59:0/963131758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd004f330 con 0x7f7fd00d29c0 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.071+0000 7f7fc5ffb640 1 -- 172.21.15.59:0/963131758 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fc0007430 con 0x7f7fd00d29c0 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.071+0000 7f7fc5ffb640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fb406b4d0 0x7f7fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.071+0000 7f7fc5ffb640 1 -- 172.21.15.59:0/963131758 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7fc0089fb0 con 0x7f7fd00d29c0 2024-03-06T07:31:22.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.071+0000 7f7fd538a640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fb406b4d0 0x7f7fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.075 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.072+0000 7f7fd538a640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fb406b4d0 0x7f7fb406d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009af0 tx=0x7f7fcc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.079+0000 7f7fc5ffb640 1 -- 172.21.15.59:0/963131758 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fc0058e10 con 0x7f7fd00d29c0 2024-03-06T07:31:22.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.435+0000 7f7fd75ee640 1 -- 172.21.15.59:0/963131758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fd00b4e50 con 0x7f7fd00d29c0 2024-03-06T07:31:22.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.435+0000 7f7fc5ffb640 1 -- 172.21.15.59:0/963131758 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f7fc0058710 con 0x7f7fd00d29c0 2024-03-06T07:31:22.437 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:22.438 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:22.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 -- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fb406b4d0 msgr2=0x7f7fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fb406b4d0 0x7f7fb406d990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc009af0 tx=0x7f7fcc008040 comp rx=0 tx=0).stop 2024-03-06T07:31:22.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 -- 172.21.15.59:0/963131758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fd00d29c0 msgr2=0x7f7fd015a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fd00d29c0 0x7f7fd015a360 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00017e0 tx=0x7f7fc000cbe0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 -- 172.21.15.59:0/963131758 shutdown_connections 2024-03-06T07:31:22.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7fb406b4d0 0x7f7fb406d990 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fd015dfe0 0x7f7fd00cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fd00d29c0 0x7f7fd015a360 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 --2- 172.21.15.59:0/963131758 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fd00cf830 0x7f7fd00b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 -- 172.21.15.59:0/963131758 >> 172.21.15.59:0/963131758 conn(0x7f7fd00d3cd0 msgr2=0x7f7fd00b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 -- 172.21.15.59:0/963131758 shutdown_connections 2024-03-06T07:31:22.441 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:22.438+0000 7f7fd75ee640 1 -- 172.21.15.59:0/963131758 wait complete. 2024-03-06T07:31:22.652 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:22.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:23.653 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:23.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.754+0000 7f5199d56640 1 Processor -- start 2024-03-06T07:31:23.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.754+0000 7f5199d56640 1 -- start start 2024-03-06T07:31:23.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f5199d56640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51940d22b0 0x7f51940d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f5199d56640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f51940b41f0 0x7f51940b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f5199d56640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51940b3430 con 0x7f519415dfe0 2024-03-06T07:31:23.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f5199d56640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51940d2c50 con 0x7f51940b41f0 2024-03-06T07:31:23.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f5199d56640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51940b31e0 con 0x7f51940d22b0 2024-03-06T07:31:23.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f51937fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f519415dfe0 0x7f51940cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44568/0 (socket says 172.21.15.59:44568) 2024-03-06T07:31:23.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f51937fe640 1 -- 172.21.15.59:0/3779281668 learned_addr learned my addr 172.21.15.59:0/3779281668 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:23.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f5192ffd640 1 --2- 172.21.15.59:0/3779281668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51940d22b0 0x7f51940d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f5193fff640 1 --2- 172.21.15.59:0/3779281668 >> v2:172.21.15.164:3300/0 conn(0x7f51940b41f0 0x7f51940b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f51927fc640 1 -- 172.21.15.59:0/3779281668 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354198197 0 0) 0x7f51940b3430 con 0x7f519415dfe0 2024-03-06T07:31:23.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f51927fc640 1 -- 172.21.15.59:0/3779281668 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f517c003520 con 0x7f519415dfe0 2024-03-06T07:31:23.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f5192ffd640 1 -- 172.21.15.59:0/3779281668 >> v2:172.21.15.164:3300/0 conn(0x7f51940b41f0 msgr2=0x7f51940b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f51927fc640 1 -- 172.21.15.59:0/3779281668 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4169919882 0 0) 0x7f517c003520 con 0x7f519415dfe0 2024-03-06T07:31:23.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f5192ffd640 1 --2- 172.21.15.59:0/3779281668 >> v2:172.21.15.164:3300/0 conn(0x7f51940b41f0 0x7f51940b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f5192ffd640 1 -- 172.21.15.59:0/3779281668 >> v1:172.21.15.59:6789/0 conn(0x7f519415dfe0 legacy=0x7f51940cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.755+0000 7f5192ffd640 1 -- 172.21.15.59:0/3779281668 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51940b3710 con 0x7f51940d22b0 2024-03-06T07:31:23.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.756+0000 7f5192ffd640 1 --2- 172.21.15.59:0/3779281668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51940d22b0 0x7f51940d2710 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51880097a0 tx=0x7f518802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=625eb104e3f625f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.756+0000 7f51927fc640 1 -- 172.21.15.59:0/3779281668 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188037070 con 0x7f51940d22b0 2024-03-06T07:31:23.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.756+0000 7f51927fc640 1 -- 172.21.15.59:0/3779281668 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5188030510 con 0x7f51940d22b0 2024-03-06T07:31:23.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.756+0000 7f51927fc640 1 -- 172.21.15.59:0/3779281668 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5184001670 con 0x7f51940d22b0 2024-03-06T07:31:23.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.757+0000 7f5199d56640 1 -- 172.21.15.59:0/3779281668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51940d22b0 msgr2=0x7f51940d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.757+0000 7f5199d56640 1 --2- 172.21.15.59:0/3779281668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51940d22b0 0x7f51940d2710 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51880097a0 tx=0x7f518802b790 comp rx=0 tx=0).stop 2024-03-06T07:31:23.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.757+0000 7f5199d56640 1 -- 172.21.15.59:0/3779281668 shutdown_connections 2024-03-06T07:31:23.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.757+0000 7f5199d56640 1 --2- 172.21.15.59:0/3779281668 >> v2:172.21.15.164:3300/0 conn(0x7f51940b41f0 0x7f51940b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.757+0000 7f5199d56640 1 --2- 172.21.15.59:0/3779281668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51940d22b0 0x7f51940d2710 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.757+0000 7f5199d56640 1 -- 172.21.15.59:0/3779281668 >> 172.21.15.59:0/3779281668 conn(0x7f51940d3cd0 msgr2=0x7f51940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.757+0000 7f5199d56640 1 -- 172.21.15.59:0/3779281668 shutdown_connections 2024-03-06T07:31:23.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.757+0000 7f5199d56640 1 -- 172.21.15.59:0/3779281668 wait complete. 2024-03-06T07:31:23.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.758+0000 7f5199d56640 1 Processor -- start 2024-03-06T07:31:23.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.758+0000 7f5199d56640 1 -- start start 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f5199d56640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51940b41f0 0x7f519415ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f5199d56640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51940d22b0 0x7f51940b7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f5199d56640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f519415dfe0 0x7f51940ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f5199d56640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51940cfa00 con 0x7f51940b41f0 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f5199d56640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51940b3d60 con 0x7f519415dfe0 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f5199d56640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51940d1c70 con 0x7f51940d22b0 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f51937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51940b41f0 0x7f519415ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f51937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51940b41f0 0x7f519415ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60802/0 (socket says 172.21.15.59:60802) 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f51937fe640 1 -- 172.21.15.59:0/2417390867 learned_addr learned my addr 172.21.15.59:0/2417390867 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f5192ffd640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51940d22b0 0x7f51940b7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f5193fff640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f519415dfe0 0x7f51940ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f51937fe640 1 -- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51940d22b0 msgr2=0x7f51940b7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f51937fe640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51940d22b0 0x7f51940b7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f51937fe640 1 -- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f519415dfe0 msgr2=0x7f51940ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f51937fe640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f519415dfe0 0x7f51940ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.759+0000 7f51937fe640 1 -- 172.21.15.59:0/2417390867 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519414abd0 con 0x7f51940b41f0 2024-03-06T07:31:23.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.760+0000 7f51937fe640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51940b41f0 0x7f519415ef40 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f51840017e0 tx=0x7f518400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.760+0000 7f517bfff640 1 -- 172.21.15.59:0/2417390867 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5184005ad0 con 0x7f51940b41f0 2024-03-06T07:31:23.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.760+0000 7f517bfff640 1 -- 172.21.15.59:0/2417390867 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5184005c90 con 0x7f51940b41f0 2024-03-06T07:31:23.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.760+0000 7f5199d56640 1 -- 172.21.15.59:0/2417390867 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519414aeb0 con 0x7f51940b41f0 2024-03-06T07:31:23.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.760+0000 7f517bfff640 1 -- 172.21.15.59:0/2417390867 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51840072b0 con 0x7f51940b41f0 2024-03-06T07:31:23.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.760+0000 7f5199d56640 1 -- 172.21.15.59:0/2417390867 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519415e740 con 0x7f51940b41f0 2024-03-06T07:31:23.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.761+0000 7f5199d56640 1 -- 172.21.15.59:0/2417390867 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519404f330 con 0x7f51940b41f0 2024-03-06T07:31:23.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.761+0000 7f517bfff640 1 -- 172.21.15.59:0/2417390867 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5184007450 con 0x7f51940b41f0 2024-03-06T07:31:23.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.762+0000 7f517bfff640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f518006b500 0x7f518006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.762+0000 7f517bfff640 1 -- 172.21.15.59:0/2417390867 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5184089fc0 con 0x7f51940b41f0 2024-03-06T07:31:23.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.762+0000 7f5192ffd640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f518006b500 0x7f518006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.762+0000 7f5192ffd640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f518006b500 0x7f518006d9c0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f5188034000 tx=0x7f518802f000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.766+0000 7f517bfff640 1 -- 172.21.15.59:0/2417390867 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5184010300 con 0x7f51940b41f0 2024-03-06T07:31:23.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.987+0000 7f5199d56640 1 -- 172.21.15.59:0/2417390867 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51940ca210 con 0x7f51940b41f0 2024-03-06T07:31:23.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.988+0000 7f517bfff640 1 -- 172.21.15.59:0/2417390867 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f5184058e20 con 0x7f51940b41f0 2024-03-06T07:31:23.990 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:23.990 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:23.990 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:23.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 -- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f518006b500 msgr2=0x7f518006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f518006b500 0x7f518006d9c0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f5188034000 tx=0x7f518802f000 comp rx=0 tx=0).stop 2024-03-06T07:31:23.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 -- 172.21.15.59:0/2417390867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51940b41f0 msgr2=0x7f519415ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51940b41f0 0x7f519415ef40 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f51840017e0 tx=0x7f518400cbe0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 -- 172.21.15.59:0/2417390867 shutdown_connections 2024-03-06T07:31:23.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f518006b500 0x7f518006d9c0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f519415dfe0 0x7f51940ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51940d22b0 0x7f51940b7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 --2- 172.21.15.59:0/2417390867 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51940b41f0 0x7f519415ef40 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.993 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 -- 172.21.15.59:0/2417390867 >> 172.21.15.59:0/2417390867 conn(0x7f51940d3cd0 msgr2=0x7f519415d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 -- 172.21.15.59:0/2417390867 shutdown_connections 2024-03-06T07:31:23.994 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:23.991+0000 7f5199d56640 1 -- 172.21.15.59:0/2417390867 wait complete. 2024-03-06T07:31:24.205 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:24.205 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:25.206 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.306+0000 7fcbf1655640 1 Processor -- start 2024-03-06T07:31:25.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.306+0000 7fcbf1655640 1 -- start start 2024-03-06T07:31:25.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.307+0000 7fcbf1655640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 0x7fcbec06b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.307+0000 7fcbf1655640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcbec15c490 0x7fcbec15c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.307+0000 7fcbf1655640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec0d0a70 con 0x7fcbec0b4420 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.307+0000 7fcbf1655640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbec0d0850 con 0x7fcbec15c490 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.307+0000 7fcbf1655640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec06b8d0 con 0x7fcbec06afb0 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.307+0000 7fcbeaffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 0x7fcbec06b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.307+0000 7fcbeb7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcbec15c490 0x7fcbec15c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.307+0000 7fcbeb7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcbec15c490 0x7fcbec15c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:58312/0 (socket says 172.21.15.59:58312) 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.307+0000 7fcbeb7fe640 1 -- 172.21.15.59:0/921075935 learned_addr learned my addr 172.21.15.59:0/921075935 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.307+0000 7fcbeaffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 0x7fcbec06b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:60936/0 (socket says 172.21.15.59:60936) 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.307+0000 7fcbea7fc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fcbec0b4420 0x7fcbec0cdb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:41770/0 (socket says 172.21.15.59:41770) 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.308+0000 7fcbeaffd640 1 -- 172.21.15.59:0/921075935 >> v2:172.21.15.164:3300/0 conn(0x7fcbec15c490 msgr2=0x7fcbec15c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.308+0000 7fcbeaffd640 1 --2- 172.21.15.59:0/921075935 >> v2:172.21.15.164:3300/0 conn(0x7fcbec15c490 0x7fcbec15c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.308+0000 7fcbeaffd640 1 -- 172.21.15.59:0/921075935 >> v1:172.21.15.164:6789/0 conn(0x7fcbec0b4420 legacy=0x7fcbec0cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.308+0000 7fcbeaffd640 1 -- 172.21.15.59:0/921075935 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec0d0d10 con 0x7fcbec06afb0 2024-03-06T07:31:25.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.308+0000 7fcbeb7fe640 1 --2- 172.21.15.59:0/921075935 >> v2:172.21.15.164:3300/0 conn(0x7fcbec15c490 0x7fcbec15c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.308+0000 7fcbeaffd640 1 --2- 172.21.15.59:0/921075935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 0x7fcbec06b390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0d0c40 tx=0x7fcbdc02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b6bb6054b4934ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.309+0000 7fcbe9ffb640 1 -- 172.21.15.59:0/921075935 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc031770 con 0x7fcbec06afb0 2024-03-06T07:31:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.309+0000 7fcbe9ffb640 1 -- 172.21.15.59:0/921075935 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbdc037070 con 0x7fcbec06afb0 2024-03-06T07:31:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.309+0000 7fcbe9ffb640 1 -- 172.21.15.59:0/921075935 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc031b30 con 0x7fcbec06afb0 2024-03-06T07:31:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.309+0000 7fcbf1655640 1 -- 172.21.15.59:0/921075935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 msgr2=0x7fcbec06b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.309+0000 7fcbf1655640 1 --2- 172.21.15.59:0/921075935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 0x7fcbec06b390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0d0c40 tx=0x7fcbdc02b810 comp rx=0 tx=0).stop 2024-03-06T07:31:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.309+0000 7fcbf1655640 1 -- 172.21.15.59:0/921075935 shutdown_connections 2024-03-06T07:31:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.309+0000 7fcbf1655640 1 --2- 172.21.15.59:0/921075935 >> v2:172.21.15.164:3300/0 conn(0x7fcbec15c490 0x7fcbec15c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.309+0000 7fcbf1655640 1 --2- 172.21.15.59:0/921075935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 0x7fcbec06b390 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.309+0000 7fcbf1655640 1 -- 172.21.15.59:0/921075935 >> 172.21.15.59:0/921075935 conn(0x7fcbec06c0c0 msgr2=0x7fcbec06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.309+0000 7fcbf1655640 1 -- 172.21.15.59:0/921075935 shutdown_connections 2024-03-06T07:31:25.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.309+0000 7fcbf1655640 1 -- 172.21.15.59:0/921075935 wait complete. 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.310+0000 7fcbf1655640 1 Processor -- start 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.311+0000 7fcbf1655640 1 -- start start 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.311+0000 7fcbf1655640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 0x7fcbec157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.311+0000 7fcbf1655640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbec0b4420 0x7fcbec1412b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.311+0000 7fcbf1655640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec15c490 0x7fcbec1f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.311+0000 7fcbf1655640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec145840 con 0x7fcbec15c490 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.311+0000 7fcbf1655640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec0d12d0 con 0x7fcbec0b4420 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.311+0000 7fcbf1655640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec0ceb90 con 0x7fcbec06afb0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.311+0000 7fcbeb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec15c490 0x7fcbec1f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.311+0000 7fcbeb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec15c490 0x7fcbec1f1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60808/0 (socket says 172.21.15.59:60808) 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.311+0000 7fcbeb7fe640 1 -- 172.21.15.59:0/301028513 learned_addr learned my addr 172.21.15.59:0/301028513 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbea7fc640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbec0b4420 0x7fcbec1412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbeaffd640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 0x7fcbec157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbeb7fe640 1 -- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 msgr2=0x7fcbec157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbeb7fe640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 0x7fcbec157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbeb7fe640 1 -- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbec0b4420 msgr2=0x7fcbec1412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbeb7fe640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbec0b4420 0x7fcbec1412b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbeb7fe640 1 -- 172.21.15.59:0/301028513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec1580d0 con 0x7fcbec15c490 2024-03-06T07:31:25.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbeb7fe640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec15c490 0x7fcbec1f1ee0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009e70 tx=0x7fcbe400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbe37fe640 1 -- 172.21.15.59:0/301028513 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe400cbe0 con 0x7fcbec15c490 2024-03-06T07:31:25.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbe37fe640 1 -- 172.21.15.59:0/301028513 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe400cd80 con 0x7fcbec15c490 2024-03-06T07:31:25.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbf1655640 1 -- 172.21.15.59:0/301028513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbec1583b0 con 0x7fcbec15c490 2024-03-06T07:31:25.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbe37fe640 1 -- 172.21.15.59:0/301028513 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4003bb0 con 0x7fcbec15c490 2024-03-06T07:31:25.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.312+0000 7fcbf1655640 1 -- 172.21.15.59:0/301028513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbec158970 con 0x7fcbec15c490 2024-03-06T07:31:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.313+0000 7fcbf1655640 1 -- 172.21.15.59:0/301028513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbec04f330 con 0x7fcbec15c490 2024-03-06T07:31:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.314+0000 7fcbe37fe640 1 -- 172.21.15.59:0/301028513 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbe401b030 con 0x7fcbec15c490 2024-03-06T07:31:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.314+0000 7fcbe37fe640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbcc06b4d0 0x7fcbcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.314+0000 7fcbe37fe640 1 -- 172.21.15.59:0/301028513 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcbe408a5d0 con 0x7fcbec15c490 2024-03-06T07:31:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.314+0000 7fcbeaffd640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbcc06b4d0 0x7fcbcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.315+0000 7fcbeaffd640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbcc06b4d0 0x7fcbcc06d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc036000 tx=0x7fcbdc03d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.318+0000 7fcbe37fe640 1 -- 172.21.15.59:0/301028513 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbe4022020 con 0x7fcbec15c490 2024-03-06T07:31:25.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.537+0000 7fcbf1655640 1 -- 172.21.15.59:0/301028513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbec0b5cb0 con 0x7fcbec15c490 2024-03-06T07:31:25.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.537+0000 7fcbe37fe640 1 -- 172.21.15.59:0/301028513 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7fcbe4010040 con 0x7fcbec15c490 2024-03-06T07:31:25.539 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:25.540 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:25.540 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 -- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbcc06b4d0 msgr2=0x7fcbcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbcc06b4d0 0x7fcbcc06d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc036000 tx=0x7fcbdc03d040 comp rx=0 tx=0).stop 2024-03-06T07:31:25.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 -- 172.21.15.59:0/301028513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec15c490 msgr2=0x7fcbec1f1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec15c490 0x7fcbec1f1ee0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009e70 tx=0x7fcbe400c490 comp rx=0 tx=0).stop 2024-03-06T07:31:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 -- 172.21.15.59:0/301028513 shutdown_connections 2024-03-06T07:31:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fcbcc06b4d0 0x7fcbcc06d990 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbec15c490 0x7fcbec1f1ee0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbec0b4420 0x7fcbec1412b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 --2- 172.21.15.59:0/301028513 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbec06afb0 0x7fcbec157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 -- 172.21.15.59:0/301028513 >> 172.21.15.59:0/301028513 conn(0x7fcbec06c0c0 msgr2=0x7fcbec0b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 -- 172.21.15.59:0/301028513 shutdown_connections 2024-03-06T07:31:25.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:25.539+0000 7fcbf1655640 1 -- 172.21.15.59:0/301028513 wait complete. 2024-03-06T07:31:25.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:25.754 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:26.755 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:26.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.855+0000 7ff5cd6a4640 1 Processor -- start 2024-03-06T07:31:26.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.855+0000 7ff5cd6a4640 1 -- start start 2024-03-06T07:31:26.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.856+0000 7ff5cd6a4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff5c815bdd0 0x7ff5c80c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.856+0000 7ff5cd6a4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c80cebe0 con 0x7ff5c80d16e0 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.856+0000 7ff5cd6a4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c80ceec0 con 0x7ff5c815fa10 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.856+0000 7ff5cd6a4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c80ce980 con 0x7ff5c815bdd0 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c77fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff5c80d16e0 0x7ff5c80d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44580/0 (socket says 172.21.15.59:44580) 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c77fe640 1 -- 172.21.15.59:0/2604063439 learned_addr learned my addr 172.21.15.59:0/2604063439 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881738382 0 0) 0x7ff5c80cebe0 con 0x7ff5c80d16e0 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c6ffd640 1 --2- 172.21.15.59:0/2604063439 >> v2:172.21.15.164:3300/0 conn(0x7ff5c815bdd0 0x7ff5c80c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b0003520 con 0x7ff5c80d16e0 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194908092 0 0) 0x7ff5b0003520 con 0x7ff5c80d16e0 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 >> v2:172.21.15.164:3300/0 conn(0x7ff5c815bdd0 msgr2=0x7ff5c80c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 --2- 172.21.15.59:0/2604063439 >> v2:172.21.15.164:3300/0 conn(0x7ff5c815bdd0 0x7ff5c80c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.861 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 >> v1:172.21.15.164:6789/0 conn(0x7ff5c815fa10 legacy=0x7ff5c80ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c80cf1a0 con 0x7ff5c80d16e0 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7ff5c0001a10 con 0x7ff5c80d16e0 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 >> v1:172.21.15.59:6789/0 conn(0x7ff5c80d16e0 legacy=0x7ff5c80d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 --2- 172.21.15.59:0/2604063439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5b00039b0 0x7ff5b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 --2- 172.21.15.59:0/2604063439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5b0004800 0x7ff5b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 --2- 172.21.15.59:0/2604063439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5b00051c0 0x7ff5b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c80ce980 con 0x7ff5b00051c0 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0010570 con 0x7ff5b0004800 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0010770 con 0x7ff5b00039b0 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c77fe640 1 --2- 172.21.15.59:0/2604063439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5b00051c0 0x7ff5b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c67fc640 1 --2- 172.21.15.59:0/2604063439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5b0004800 0x7ff5b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.857+0000 7ff5c6ffd640 1 --2- 172.21.15.59:0/2604063439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5b00039b0 0x7ff5b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.858+0000 7ff5c77fe640 1 -- 172.21.15.59:0/2604063439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5b00039b0 msgr2=0x7ff5b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.858+0000 7ff5c77fe640 1 --2- 172.21.15.59:0/2604063439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5b00039b0 0x7ff5b0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.862 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.858+0000 7ff5c77fe640 1 -- 172.21.15.59:0/2604063439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5b0004800 msgr2=0x7ff5b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.858+0000 7ff5c77fe640 1 --2- 172.21.15.59:0/2604063439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5b0004800 0x7ff5b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.858+0000 7ff5c77fe640 1 -- 172.21.15.59:0/2604063439 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b0003520 con 0x7ff5b00051c0 2024-03-06T07:31:26.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.858+0000 7ff5c77fe640 1 -- 172.21.15.59:0/2604063439 reap_dead start 2024-03-06T07:31:26.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.858+0000 7ff5c77fe640 1 --2- 172.21.15.59:0/2604063439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5b00051c0 0x7ff5b000b9c0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000a8f0 tx=0x7ff5b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdca259e4db2d1c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.859+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c000a4c0 con 0x7ff5b00051c0 2024-03-06T07:31:26.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.859+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c0034650 con 0x7ff5b00051c0 2024-03-06T07:31:26.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.859+0000 7ff5c5ffb640 1 -- 172.21.15.59:0/2604063439 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0034a40 con 0x7ff5b00051c0 2024-03-06T07:31:26.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.859+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/2604063439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5b00051c0 msgr2=0x7ff5b000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.859+0000 7ff5cd6a4640 1 --2- 172.21.15.59:0/2604063439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5b00051c0 0x7ff5b000b9c0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000a8f0 tx=0x7ff5b0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:26.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.859+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/2604063439 shutdown_connections 2024-03-06T07:31:26.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.859+0000 7ff5cd6a4640 1 --2- 172.21.15.59:0/2604063439 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5b00051c0 0x7ff5b000b9c0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.859+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/2604063439 >> 172.21.15.59:0/2604063439 conn(0x7ff5c809a9f0 msgr2=0x7ff5c809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.859+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/2604063439 shutdown_connections 2024-03-06T07:31:26.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.859+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/2604063439 wait complete. 2024-03-06T07:31:26.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.861+0000 7ff5cd6a4640 1 Processor -- start 2024-03-06T07:31:26.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.861+0000 7ff5cd6a4640 1 -- start start 2024-03-06T07:31:26.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5cd6a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5b00051c0 0x7ff5c8159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5cd6a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c81499b0 0x7ff5c8149e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5cd6a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5c8158050 0x7ff5c81584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5cd6a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c815c1b0 con 0x7ff5c81499b0 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5cd6a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c80cc5d0 con 0x7ff5c8158050 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5cd6a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c80ce270 con 0x7ff5b00051c0 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5c67fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c81499b0 0x7ff5c8149e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5c67fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c81499b0 0x7ff5c8149e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60838/0 (socket says 172.21.15.59:60838) 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5c67fc640 1 -- 172.21.15.59:0/3286171212 learned_addr learned my addr 172.21.15.59:0/3286171212 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5c6ffd640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5b00051c0 0x7ff5c8159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5c77fe640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5c8158050 0x7ff5c81584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5c67fc640 1 -- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5b00051c0 msgr2=0x7ff5c8159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5c67fc640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5b00051c0 0x7ff5c8159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5c67fc640 1 -- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5c8158050 msgr2=0x7ff5c81584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5c67fc640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5c8158050 0x7ff5c81584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5c67fc640 1 -- 172.21.15.59:0/3286171212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c814a370 con 0x7ff5c81499b0 2024-03-06T07:31:26.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5c67fc640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c81499b0 0x7ff5c8149e30 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00f5c0 tx=0x7ff5bc00f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5af7fe640 1 -- 172.21.15.59:0/3286171212 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0052e0 con 0x7ff5c81499b0 2024-03-06T07:31:26.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5af7fe640 1 -- 172.21.15.59:0/3286171212 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc018070 con 0x7ff5c81499b0 2024-03-06T07:31:26.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5af7fe640 1 -- 172.21.15.59:0/3286171212 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc003830 con 0x7ff5c81499b0 2024-03-06T07:31:26.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.863+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/3286171212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c815c4d0 con 0x7ff5c81499b0 2024-03-06T07:31:26.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.864+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/3286171212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c815ca90 con 0x7ff5c81499b0 2024-03-06T07:31:26.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.865+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/3286171212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c804f330 con 0x7ff5c81499b0 2024-03-06T07:31:26.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.865+0000 7ff5af7fe640 1 -- 172.21.15.59:0/3286171212 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff5bc0039d0 con 0x7ff5c81499b0 2024-03-06T07:31:26.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.866+0000 7ff5af7fe640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff5b406b500 0x7ff5b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.866+0000 7ff5af7fe640 1 -- 172.21.15.59:0/3286171212 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff5bc089e30 con 0x7ff5c81499b0 2024-03-06T07:31:26.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.866+0000 7ff5c6ffd640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff5b406b500 0x7ff5b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.867+0000 7ff5c6ffd640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff5b406b500 0x7ff5b406d9c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009b40 tx=0x7ff5b8009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:26.870+0000 7ff5af7fe640 1 -- 172.21.15.59:0/3286171212 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff5bc058ca0 con 0x7ff5c81499b0 2024-03-06T07:31:27.089 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.088+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/3286171212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5c80b7a40 con 0x7ff5c81499b0 2024-03-06T07:31:27.090 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.088+0000 7ff5af7fe640 1 -- 172.21.15.59:0/3286171212 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7ff5bc0585a0 con 0x7ff5c81499b0 2024-03-06T07:31:27.090 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:27.090 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:27.091 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:27.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff5b406b500 msgr2=0x7ff5b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff5b406b500 0x7ff5b406d9c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009b40 tx=0x7ff5b8009190 comp rx=0 tx=0).stop 2024-03-06T07:31:27.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/3286171212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c81499b0 msgr2=0x7ff5c8149e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c81499b0 0x7ff5c8149e30 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00f5c0 tx=0x7ff5bc00f9c0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/3286171212 shutdown_connections 2024-03-06T07:31:27.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff5b406b500 0x7ff5b406d9c0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5c8158050 0x7ff5c81584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5c81499b0 0x7ff5c8149e30 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 --2- 172.21.15.59:0/3286171212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5b00051c0 0x7ff5c8159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/3286171212 >> 172.21.15.59:0/3286171212 conn(0x7ff5c809a9f0 msgr2=0x7ff5c806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/3286171212 shutdown_connections 2024-03-06T07:31:27.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:27.090+0000 7ff5cd6a4640 1 -- 172.21.15.59:0/3286171212 wait complete. 2024-03-06T07:31:27.304 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:27.304 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:28.305 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:28.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.405+0000 7f27b7f1e640 1 Processor -- start 2024-03-06T07:31:28.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.405+0000 7f27b7f1e640 1 -- start start 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b7f1e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f27b00d04a0 0x7f27b00ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b7f1e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b0161980 con 0x7f27b00cfff0 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b7f1e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b0161c60 con 0x7f27b015bb20 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b7f1e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27b00d2500 con 0x7f27b00d04a0 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b54b9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f27b00cfff0 0x7f27b00cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44594/0 (socket says 172.21.15.59:44594) 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b54b9640 1 -- 172.21.15.59:0/2945159464 learned_addr learned my addr 172.21.15.59:0/2945159464 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b64bb640 1 --2- 172.21.15.59:0/2945159464 >> v2:172.21.15.164:3300/0 conn(0x7f27b00d04a0 0x7f27b00ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.408 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b4cb8640 1 -- 172.21.15.59:0/2945159464 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873426141 0 0) 0x7f27b0161980 con 0x7f27b00cfff0 2024-03-06T07:31:28.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b4cb8640 1 -- 172.21.15.59:0/2945159464 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f279c003520 con 0x7f27b00cfff0 2024-03-06T07:31:28.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b64bb640 1 -- 172.21.15.59:0/2945159464 >> v1:172.21.15.164:6789/0 conn(0x7f27b015bb20 legacy=0x7f27b00d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b4cb8640 1 -- 172.21.15.59:0/2945159464 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2437398416 0 0) 0x7f279c003520 con 0x7f27b00cfff0 2024-03-06T07:31:28.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b64bb640 1 -- 172.21.15.59:0/2945159464 >> v1:172.21.15.59:6789/0 conn(0x7f27b00cfff0 legacy=0x7f27b00cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.406+0000 7f27b64bb640 1 -- 172.21.15.59:0/2945159464 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b0161f40 con 0x7f27b00d04a0 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.407+0000 7f27b64bb640 1 --2- 172.21.15.59:0/2945159464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27b00d04a0 0x7f27b00ccfc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f27a8009d90 tx=0x7f27a802bc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c31fbdb425434aab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.407+0000 7f27b4cb8640 1 -- 172.21.15.59:0/2945159464 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a8039070 con 0x7f27b00d04a0 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.407+0000 7f27b4cb8640 1 -- 172.21.15.59:0/2945159464 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a802c7a0 con 0x7f27b00d04a0 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.407+0000 7f27b4cb8640 1 -- 172.21.15.59:0/2945159464 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0001670 con 0x7f27b00d04a0 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.408+0000 7f27b7f1e640 1 -- 172.21.15.59:0/2945159464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27b00d04a0 msgr2=0x7f27b00ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.408+0000 7f27b7f1e640 1 --2- 172.21.15.59:0/2945159464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27b00d04a0 0x7f27b00ccfc0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f27a8009d90 tx=0x7f27a802bc70 comp rx=0 tx=0).stop 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.408+0000 7f27b7f1e640 1 -- 172.21.15.59:0/2945159464 shutdown_connections 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.408+0000 7f27b7f1e640 1 --2- 172.21.15.59:0/2945159464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27b00d04a0 0x7f27b00ccfc0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.408+0000 7f27b7f1e640 1 -- 172.21.15.59:0/2945159464 >> 172.21.15.59:0/2945159464 conn(0x7f27b009a9f0 msgr2=0x7f27b009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.408+0000 7f27b7f1e640 1 -- 172.21.15.59:0/2945159464 shutdown_connections 2024-03-06T07:31:28.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.408+0000 7f27b7f1e640 1 -- 172.21.15.59:0/2945159464 wait complete. 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.409+0000 7f27b7f1e640 1 Processor -- start 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.409+0000 7f27b7f1e640 1 -- start start 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b7f1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27b00cfff0 0x7f27b01e77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b7f1e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27b00d04a0 0x7f27b015e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b7f1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27b015bb20 0x7f27b01f4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b7f1e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b00d08d0 con 0x7f27b00cfff0 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b7f1e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b0162500 con 0x7f27b00d04a0 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b7f1e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b00d1980 con 0x7f27b015bb20 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b5cba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27b00cfff0 0x7f27b01e77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b5cba640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27b00cfff0 0x7f27b01e77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60854/0 (socket says 172.21.15.59:60854) 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b5cba640 1 -- 172.21.15.59:0/285608575 learned_addr learned my addr 172.21.15.59:0/285608575 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b5cba640 1 -- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27b015bb20 msgr2=0x7f27b01f4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b64bb640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27b015bb20 0x7f27b01f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.413 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b5cba640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27b015bb20 0x7f27b01f4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b54b9640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27b00d04a0 0x7f27b015e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b5cba640 1 -- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27b00d04a0 msgr2=0x7f27b015e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b5cba640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27b00d04a0 0x7f27b015e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b5cba640 1 -- 172.21.15.59:0/285608575 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b01e7d20 con 0x7f27b00cfff0 2024-03-06T07:31:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.410+0000 7f27b54b9640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27b00d04a0 0x7f27b015e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.411+0000 7f27b5cba640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27b00cfff0 0x7f27b01e77e0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00c7b0 tx=0x7f27ac00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.411+0000 7f27a67fc640 1 -- 172.21.15.59:0/285608575 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac017070 con 0x7f27b00cfff0 2024-03-06T07:31:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.411+0000 7f27a67fc640 1 -- 172.21.15.59:0/285608575 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ac005290 con 0x7f27b00cfff0 2024-03-06T07:31:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.411+0000 7f27b7f1e640 1 -- 172.21.15.59:0/285608575 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b015ec90 con 0x7f27b00cfff0 2024-03-06T07:31:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.411+0000 7f27a67fc640 1 -- 172.21.15.59:0/285608575 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac00a880 con 0x7f27b00cfff0 2024-03-06T07:31:28.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.411+0000 7f27b7f1e640 1 -- 172.21.15.59:0/285608575 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b015f5a0 con 0x7f27b00cfff0 2024-03-06T07:31:28.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.412+0000 7f27b7f1e640 1 -- 172.21.15.59:0/285608575 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b004f330 con 0x7f27b00cfff0 2024-03-06T07:31:28.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.412+0000 7f27a67fc640 1 -- 172.21.15.59:0/285608575 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27ac01a070 con 0x7f27b00cfff0 2024-03-06T07:31:28.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.412+0000 7f27a67fc640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f279406b4d0 0x7f279406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:28.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.413+0000 7f27a67fc640 1 -- 172.21.15.59:0/285608575 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f27ac089720 con 0x7f27b00cfff0 2024-03-06T07:31:28.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.413+0000 7f27b54b9640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f279406b4d0 0x7f279406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:28.415 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.413+0000 7f27b54b9640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f279406b4d0 0x7f279406d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f27a0001eb0 tx=0x7f27a0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:28.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.421+0000 7f27a67fc640 1 -- 172.21.15.59:0/285608575 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27ac058590 con 0x7f27b00cfff0 2024-03-06T07:31:28.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.679+0000 7f27b7f1e640 1 -- 172.21.15.59:0/285608575 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27b00b1820 con 0x7f27b00cfff0 2024-03-06T07:31:28.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.680+0000 7f27a67fc640 1 -- 172.21.15.59:0/285608575 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f27ac057e90 con 0x7f27b00cfff0 2024-03-06T07:31:28.682 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:28.682 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:28.682 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:28.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 -- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f279406b4d0 msgr2=0x7f279406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f279406b4d0 0x7f279406d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f27a0001eb0 tx=0x7f27a0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:28.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 -- 172.21.15.59:0/285608575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27b00cfff0 msgr2=0x7f27b01e77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27b00cfff0 0x7f27b01e77e0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00c7b0 tx=0x7f27ac00cc70 comp rx=0 tx=0).stop 2024-03-06T07:31:28.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 -- 172.21.15.59:0/285608575 shutdown_connections 2024-03-06T07:31:28.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f279406b4d0 0x7f279406d990 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27b015bb20 0x7f27b01f4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27b00d04a0 0x7f27b015e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 --2- 172.21.15.59:0/285608575 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27b00cfff0 0x7f27b01e77e0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 -- 172.21.15.59:0/285608575 >> 172.21.15.59:0/285608575 conn(0x7f27b009a9f0 msgr2=0x7f27b009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 -- 172.21.15.59:0/285608575 shutdown_connections 2024-03-06T07:31:28.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:28.682+0000 7f27b7f1e640 1 -- 172.21.15.59:0/285608575 wait complete. 2024-03-06T07:31:28.896 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:28.896 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:29.897 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.996+0000 7f8fd4d3e640 1 Processor -- start 2024-03-06T07:31:29.998 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.996+0000 7f8fd4d3e640 1 -- start start 2024-03-06T07:31:29.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.997+0000 7f8fd4d3e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8fd015fa10 0x7f8fd00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.997+0000 7f8fd4d3e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8fd00d16e0 0x7f8fd00d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.997+0000 7f8fd4d3e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00cebe0 con 0x7f8fd015bdd0 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.997+0000 7f8fd4d3e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd00ceec0 con 0x7f8fd015fa10 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.997+0000 7f8fd4d3e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd00cf1a0 con 0x7f8fd00d16e0 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.997+0000 7f8fc7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8fd015fa10 0x7f8fd00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.997+0000 7f8fc7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8fd015fa10 0x7f8fd00ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60860/0 (socket says 172.21.15.59:60860) 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.997+0000 7f8fc7fff640 1 -- 172.21.15.59:0/2449650748 learned_addr learned my addr 172.21.15.59:0/2449650748 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.998+0000 7f8fced9d640 1 --2- 172.21.15.59:0/2449650748 >> v2:172.21.15.164:3300/0 conn(0x7f8fd00d16e0 0x7f8fd00d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.998+0000 7f8fc7fff640 1 -- 172.21.15.59:0/2449650748 >> v2:172.21.15.164:3300/0 conn(0x7f8fd00d16e0 msgr2=0x7f8fd00d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.998+0000 7f8fc7fff640 1 --2- 172.21.15.59:0/2449650748 >> v2:172.21.15.164:3300/0 conn(0x7f8fd00d16e0 0x7f8fd00d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.998+0000 7f8fc7fff640 1 -- 172.21.15.59:0/2449650748 >> v1:172.21.15.164:6789/0 conn(0x7f8fd015bdd0 legacy=0x7f8fd00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.998+0000 7f8fc7fff640 1 -- 172.21.15.59:0/2449650748 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd00cc8e0 con 0x7f8fd015fa10 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.999+0000 7f8fc7fff640 1 --2- 172.21.15.59:0/2449650748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd015fa10 0x7f8fd00ce270 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00c5520 tx=0x7f8fb802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1dcdaaa0dbf8a7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.999+0000 7f8fcdd9b640 1 -- 172.21.15.59:0/2449650748 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8035020 con 0x7f8fd015fa10 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.999+0000 7f8fcdd9b640 1 -- 172.21.15.59:0/2449650748 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb8031870 con 0x7f8fd015fa10 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:29.999+0000 7f8fcdd9b640 1 -- 172.21.15.59:0/2449650748 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8031c40 con 0x7f8fd015fa10 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.000+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/2449650748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd015fa10 msgr2=0x7f8fd00ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.000+0000 7f8fd4d3e640 1 --2- 172.21.15.59:0/2449650748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd015fa10 0x7f8fd00ce270 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00c5520 tx=0x7f8fb802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.000+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/2449650748 shutdown_connections 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.000+0000 7f8fd4d3e640 1 --2- 172.21.15.59:0/2449650748 >> v2:172.21.15.164:3300/0 conn(0x7f8fd00d16e0 0x7f8fd00d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.000+0000 7f8fd4d3e640 1 --2- 172.21.15.59:0/2449650748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd015fa10 0x7f8fd00ce270 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.000+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/2449650748 >> 172.21.15.59:0/2449650748 conn(0x7f8fd009a9f0 msgr2=0x7f8fd009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.000+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/2449650748 shutdown_connections 2024-03-06T07:31:30.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.000+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/2449650748 wait complete. 2024-03-06T07:31:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.001+0000 7f8fd4d3e640 1 Processor -- start 2024-03-06T07:31:30.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.001+0000 7f8fd4d3e640 1 -- start start 2024-03-06T07:31:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fd4d3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fd00d16e0 0x7f8fd01f14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fd4d3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd015bdd0 0x7f8fd015e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fd4d3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fd015fa10 0x7f8fd00b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fd4d3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd00d2030 con 0x7f8fd015bdd0 2024-03-06T07:31:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fd4d3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd015c1b0 con 0x7f8fd00d16e0 2024-03-06T07:31:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fd4d3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd00ce7b0 con 0x7f8fd015fa10 2024-03-06T07:31:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fc7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd015bdd0 0x7f8fd015e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fc7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd015bdd0 0x7f8fd015e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60868/0 (socket says 172.21.15.59:60868) 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fc7fff640 1 -- 172.21.15.59:0/3373867409 learned_addr learned my addr 172.21.15.59:0/3373867409 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fced9d640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fd015fa10 0x7f8fd00b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fce59c640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fd00d16e0 0x7f8fd01f14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fc7fff640 1 -- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fd015fa10 msgr2=0x7f8fd00b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fc7fff640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fd015fa10 0x7f8fd00b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fc7fff640 1 -- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fd00d16e0 msgr2=0x7f8fd01f14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fc7fff640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fd00d16e0 0x7f8fd01f14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fc7fff640 1 -- 172.21.15.59:0/3373867409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd01f1e50 con 0x7f8fd015bdd0 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fced9d640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fd015fa10 0x7f8fd00b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.002+0000 7f8fc7fff640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd015bdd0 0x7f8fd015e2a0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f8fb802b950 tx=0x7f8fb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.003+0000 7f8fc6ffd640 1 -- 172.21.15.59:0/3373867409 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8040070 con 0x7f8fd015bdd0 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.003+0000 7f8fc6ffd640 1 -- 172.21.15.59:0/3373867409 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb8030d00 con 0x7f8fd015bdd0 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.003+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/3373867409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd01f20f0 con 0x7f8fd015bdd0 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.003+0000 7f8fc6ffd640 1 -- 172.21.15.59:0/3373867409 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8035040 con 0x7f8fd015bdd0 2024-03-06T07:31:30.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.003+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/3373867409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd01f26b0 con 0x7f8fd015bdd0 2024-03-06T07:31:30.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.004+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/3373867409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd004f330 con 0x7f8fd015bdd0 2024-03-06T07:31:30.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.004+0000 7f8fc6ffd640 1 -- 172.21.15.59:0/3373867409 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fb804b060 con 0x7f8fd015bdd0 2024-03-06T07:31:30.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.005+0000 7f8fc6ffd640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fb006b4d0 0x7f8fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:30.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.005+0000 7f8fc6ffd640 1 -- 172.21.15.59:0/3373867409 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8fb80abfb0 con 0x7f8fd015bdd0 2024-03-06T07:31:30.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.005+0000 7f8fce59c640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fb006b4d0 0x7f8fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:30.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.006+0000 7f8fce59c640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fb006b4d0 0x7f8fb006d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f8fd015f020 tx=0x7f8fc0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:30.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.009+0000 7f8fc6ffd640 1 -- 172.21.15.59:0/3373867409 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fb807ae10 con 0x7f8fd015bdd0 2024-03-06T07:31:30.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.236+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/3373867409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fd00b7a40 con 0x7f8fd015bdd0 2024-03-06T07:31:30.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.236+0000 7f8fc6ffd640 1 -- 172.21.15.59:0/3373867409 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f8fb807a710 con 0x7f8fd015bdd0 2024-03-06T07:31:30.238 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:30.238 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:30.239 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fb006b4d0 msgr2=0x7f8fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fb006b4d0 0x7f8fb006d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f8fd015f020 tx=0x7f8fc0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/3373867409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd015bdd0 msgr2=0x7f8fd015e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd015bdd0 0x7f8fd015e2a0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f8fb802b950 tx=0x7f8fb80046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/3373867409 shutdown_connections 2024-03-06T07:31:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fb006b4d0 0x7f8fb006d990 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fd015fa10 0x7f8fd00b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fd015bdd0 0x7f8fd015e2a0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 --2- 172.21.15.59:0/3373867409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fd00d16e0 0x7f8fd01f14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/3373867409 >> 172.21.15.59:0/3373867409 conn(0x7f8fd009a9f0 msgr2=0x7f8fd009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/3373867409 shutdown_connections 2024-03-06T07:31:30.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:30.238+0000 7f8fd4d3e640 1 -- 172.21.15.59:0/3373867409 wait complete. 2024-03-06T07:31:30.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:30.452 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:31.453 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:31.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.573+0000 7f744c6af640 1 Processor -- start 2024-03-06T07:31:31.575 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.573+0000 7f744c6af640 1 -- start start 2024-03-06T07:31:31.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744c6af640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f744415dfe0 0x7f74440cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744c6af640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f74440cf830 0x7f74440d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744c6af640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74440b3430 con 0x7f74440d27f0 2024-03-06T07:31:31.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744c6af640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74440b31e0 con 0x7f744415dfe0 2024-03-06T07:31:31.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744c6af640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74440b3d60 con 0x7f74440cf830 2024-03-06T07:31:31.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744a44b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f744415dfe0 0x7f74440cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744a44b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f744415dfe0 0x7f74440cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60872/0 (socket says 172.21.15.59:60872) 2024-03-06T07:31:31.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744a44b640 1 -- 172.21.15.59:0/679127663 learned_addr learned my addr 172.21.15.59:0/679127663 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:31.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f7449c4a640 1 --2- 172.21.15.59:0/679127663 >> v2:172.21.15.164:3300/0 conn(0x7f74440cf830 0x7f74440d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744a44b640 1 -- 172.21.15.59:0/679127663 >> v2:172.21.15.164:3300/0 conn(0x7f74440cf830 msgr2=0x7f74440d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744a44b640 1 --2- 172.21.15.59:0/679127663 >> v2:172.21.15.164:3300/0 conn(0x7f74440cf830 0x7f74440d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744a44b640 1 -- 172.21.15.59:0/679127663 >> v1:172.21.15.164:6789/0 conn(0x7f74440d27f0 legacy=0x7f74440d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f744a44b640 1 -- 172.21.15.59:0/679127663 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74440b3710 con 0x7f744415dfe0 2024-03-06T07:31:31.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.574+0000 7f7449c4a640 1 --2- 172.21.15.59:0/679127663 >> v2:172.21.15.164:3300/0 conn(0x7f74440cf830 0x7f74440d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:31.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.575+0000 7f744a44b640 1 --2- 172.21.15.59:0/679127663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f744415dfe0 0x7f74440cf2f0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f7440009870 tx=0x7f744002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15f1b4c30792f38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.575+0000 7f7449449640 1 -- 172.21.15.59:0/679127663 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7440035020 con 0x7f744415dfe0 2024-03-06T07:31:31.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.575+0000 7f7449449640 1 -- 172.21.15.59:0/679127663 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7440031870 con 0x7f744415dfe0 2024-03-06T07:31:31.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.575+0000 7f7449449640 1 -- 172.21.15.59:0/679127663 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7440031c40 con 0x7f744415dfe0 2024-03-06T07:31:31.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.576+0000 7f744c6af640 1 -- 172.21.15.59:0/679127663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f744415dfe0 msgr2=0x7f74440cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.576+0000 7f744c6af640 1 --2- 172.21.15.59:0/679127663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f744415dfe0 0x7f74440cf2f0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f7440009870 tx=0x7f744002b910 comp rx=0 tx=0).stop 2024-03-06T07:31:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.576+0000 7f744c6af640 1 -- 172.21.15.59:0/679127663 shutdown_connections 2024-03-06T07:31:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.576+0000 7f744c6af640 1 --2- 172.21.15.59:0/679127663 >> v2:172.21.15.164:3300/0 conn(0x7f74440cf830 0x7f74440d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.576+0000 7f744c6af640 1 --2- 172.21.15.59:0/679127663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f744415dfe0 0x7f74440cf2f0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.576+0000 7f744c6af640 1 -- 172.21.15.59:0/679127663 >> 172.21.15.59:0/679127663 conn(0x7f74440d3cd0 msgr2=0x7f74440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.576+0000 7f744c6af640 1 -- 172.21.15.59:0/679127663 shutdown_connections 2024-03-06T07:31:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.576+0000 7f744c6af640 1 -- 172.21.15.59:0/679127663 wait complete. 2024-03-06T07:31:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.577+0000 7f744c6af640 1 Processor -- start 2024-03-06T07:31:31.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.577+0000 7f744c6af640 1 -- start start 2024-03-06T07:31:31.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744c6af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74440cf830 0x7f74440b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744c6af640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74440d27f0 0x7f74440b8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744c6af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f744415dfe0 0x7f74440cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744c6af640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74440d1df0 con 0x7f744415dfe0 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744c6af640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74440d1c70 con 0x7f74440d27f0 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744c6af640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744415e3c0 con 0x7f74440cf830 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744ac4c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f744415dfe0 0x7f74440cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744ac4c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f744415dfe0 0x7f74440cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60878/0 (socket says 172.21.15.59:60878) 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744ac4c640 1 -- 172.21.15.59:0/742726504 learned_addr learned my addr 172.21.15.59:0/742726504 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f7449c4a640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74440d27f0 0x7f74440b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744a44b640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74440cf830 0x7f74440b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744ac4c640 1 -- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74440cf830 msgr2=0x7f74440b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744ac4c640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74440cf830 0x7f74440b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744ac4c640 1 -- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74440d27f0 msgr2=0x7f74440b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744ac4c640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74440d27f0 0x7f74440b8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744ac4c640 1 -- 172.21.15.59:0/742726504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74440b92a0 con 0x7f744415dfe0 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.578+0000 7f744ac4c640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f744415dfe0 0x7f74440cbdf0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f743c00c7b0 tx=0x7f743c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.579+0000 7f743affd640 1 -- 172.21.15.59:0/742726504 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c004b60 con 0x7f744415dfe0 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.579+0000 7f743affd640 1 -- 172.21.15.59:0/742726504 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743c004d00 con 0x7f744415dfe0 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.579+0000 7f744c6af640 1 -- 172.21.15.59:0/742726504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74440b9580 con 0x7f744415dfe0 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.579+0000 7f743affd640 1 -- 172.21.15.59:0/742726504 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c00aa90 con 0x7f744415dfe0 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.579+0000 7f744c6af640 1 -- 172.21.15.59:0/742726504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744415eb40 con 0x7f744415dfe0 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.580+0000 7f744c6af640 1 -- 172.21.15.59:0/742726504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f744415ed40 con 0x7f744415dfe0 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.580+0000 7f743affd640 1 -- 172.21.15.59:0/742726504 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f743c00ac30 con 0x7f744415dfe0 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.581+0000 7f743affd640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f742806b4d0 0x7f742806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.581+0000 7f743affd640 1 -- 172.21.15.59:0/742726504 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f743c08a4f0 con 0x7f744415dfe0 2024-03-06T07:31:31.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.581+0000 7f744a44b640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f742806b4d0 0x7f742806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.581+0000 7f744a44b640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f742806b4d0 0x7f742806d990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f7440036000 tx=0x7f744003c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.589+0000 7f743affd640 1 -- 172.21.15.59:0/742726504 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f743c020070 con 0x7f744415dfe0 2024-03-06T07:31:31.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.808+0000 7f744c6af640 1 -- 172.21.15.59:0/742726504 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f744404f330 con 0x7f744415dfe0 2024-03-06T07:31:31.810 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.809+0000 7f743affd640 1 -- 172.21.15.59:0/742726504 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f743c088e20 con 0x7f744415dfe0 2024-03-06T07:31:31.811 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:31.811 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:31.811 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:31.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 -- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f742806b4d0 msgr2=0x7f742806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f742806b4d0 0x7f742806d990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f7440036000 tx=0x7f744003c040 comp rx=0 tx=0).stop 2024-03-06T07:31:31.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 -- 172.21.15.59:0/742726504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f744415dfe0 msgr2=0x7f74440cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f744415dfe0 0x7f74440cbdf0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f743c00c7b0 tx=0x7f743c00cc70 comp rx=0 tx=0).stop 2024-03-06T07:31:31.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 -- 172.21.15.59:0/742726504 shutdown_connections 2024-03-06T07:31:31.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f742806b4d0 0x7f742806d990 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f744415dfe0 0x7f74440cbdf0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74440d27f0 0x7f74440b8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.813 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 --2- 172.21.15.59:0/742726504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74440cf830 0x7f74440b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 -- 172.21.15.59:0/742726504 >> 172.21.15.59:0/742726504 conn(0x7f74440d3cd0 msgr2=0x7f74440d0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 -- 172.21.15.59:0/742726504 shutdown_connections 2024-03-06T07:31:31.814 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:31.811+0000 7f744c6af640 1 -- 172.21.15.59:0/742726504 wait complete. 2024-03-06T07:31:32.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:32.025 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:33.027 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:33.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.127+0000 7f396d2a4640 1 Processor -- start 2024-03-06T07:31:33.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.128+0000 7f396d2a4640 1 -- start start 2024-03-06T07:31:33.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.128+0000 7f396d2a4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f39680d2710 0x7f39680b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.128+0000 7f396d2a4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39680b3430 con 0x7f396815dfe0 2024-03-06T07:31:33.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.128+0000 7f396d2a4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39680b3710 con 0x7f39680d22b0 2024-03-06T07:31:33.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.128+0000 7f396d2a4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39680b31e0 con 0x7f39680d2710 2024-03-06T07:31:33.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.128+0000 7f3966ffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f396815dfe0 0x7f39680cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:44602/0 (socket says 172.21.15.59:44602) 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.128+0000 7f3966ffd640 1 -- 172.21.15.59:0/722628148 learned_addr learned my addr 172.21.15.59:0/722628148 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39677fe640 1 --2- 172.21.15.59:0/722628148 >> v2:172.21.15.164:3300/0 conn(0x7f39680d2710 0x7f39680b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611507447 0 0) 0x7f39680b3430 con 0x7f396815dfe0 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3954003520 con 0x7f396815dfe0 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629088247 0 0) 0x7f39680b3710 con 0x7f39680d22b0 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39680b3430 con 0x7f39680d22b0 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2969725948 0 0) 0x7f3954003520 con 0x7f396815dfe0 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 >> v2:172.21.15.164:3300/0 conn(0x7f39680d2710 msgr2=0x7f39680b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 --2- 172.21.15.59:0/722628148 >> v2:172.21.15.164:3300/0 conn(0x7f39680d2710 0x7f39680b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 >> v1:172.21.15.164:6789/0 conn(0x7f39680d22b0 legacy=0x7f39680b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39680b39f0 con 0x7f396815dfe0 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f3958001670 con 0x7f396815dfe0 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 >> v1:172.21.15.59:6789/0 conn(0x7f396815dfe0 legacy=0x7f39680cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 0x7f3954003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3954004800 0x7f3954004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39540051c0 0x7f395400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39680b31e0 con 0x7f3954003b70 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395400c060 con 0x7f39540051c0 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395400c260 con 0x7f3954004800 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f3966ffd640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 0x7f3954003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f39677fe640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39540051c0 0x7f395400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f395ffff640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3954004800 0x7f3954004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f3966ffd640 1 -- 172.21.15.59:0/722628148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3954004800 msgr2=0x7f3954004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f3966ffd640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3954004800 0x7f3954004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f3966ffd640 1 -- 172.21.15.59:0/722628148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39540051c0 msgr2=0x7f395400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f3966ffd640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39540051c0 0x7f395400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.129+0000 7f3966ffd640 1 -- 172.21.15.59:0/722628148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3954003520 con 0x7f3954003b70 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.130+0000 7f39677fe640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39540051c0 0x7f395400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.130+0000 7f39677fe640 1 -- 172.21.15.59:0/722628148 reap_dead start 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.130+0000 7f3966ffd640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 0x7f3954003f70 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f395800c190 tx=0x7f3954005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e3317523fe9193 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.134 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.130+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958007cc0 con 0x7f3954003b70 2024-03-06T07:31:33.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.130+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3958030070 con 0x7f3954003b70 2024-03-06T07:31:33.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.130+0000 7f39667fc640 1 -- 172.21.15.59:0/722628148 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958005d70 con 0x7f3954003b70 2024-03-06T07:31:33.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.131+0000 7f396d2a4640 1 -- 172.21.15.59:0/722628148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 msgr2=0x7f3954003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.131+0000 7f396d2a4640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 0x7f3954003f70 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f395800c190 tx=0x7f3954005830 comp rx=0 tx=0).stop 2024-03-06T07:31:33.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.131+0000 7f396d2a4640 1 -- 172.21.15.59:0/722628148 shutdown_connections 2024-03-06T07:31:33.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.131+0000 7f396d2a4640 1 --2- 172.21.15.59:0/722628148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 0x7f3954003f70 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.131+0000 7f396d2a4640 1 -- 172.21.15.59:0/722628148 >> 172.21.15.59:0/722628148 conn(0x7f39680d3cd0 msgr2=0x7f39680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.131+0000 7f396d2a4640 1 -- 172.21.15.59:0/722628148 shutdown_connections 2024-03-06T07:31:33.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.131+0000 7f396d2a4640 1 -- 172.21.15.59:0/722628148 wait complete. 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.132+0000 7f396d2a4640 1 Processor -- start 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.132+0000 7f396d2a4640 1 -- start start 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f396d2a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 0x7f396809e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f396d2a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39680a0810 0x7f396815c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f396d2a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f396806ccc0 0x7f396806d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f396d2a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39680d1c70 con 0x7f3954003b70 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f396d2a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39680b3d60 con 0x7f396806ccc0 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f396d2a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39680d1df0 con 0x7f39680a0810 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f3966ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 0x7f396809e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f3966ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 0x7f396809e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60904/0 (socket says 172.21.15.59:60904) 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f3966ffd640 1 -- 172.21.15.59:0/2855990462 learned_addr learned my addr 172.21.15.59:0/2855990462 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f395ffff640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39680a0810 0x7f396815c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f39677fe640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f396806ccc0 0x7f396806d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f3966ffd640 1 -- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39680a0810 msgr2=0x7f396815c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f3966ffd640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39680a0810 0x7f396815c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f3966ffd640 1 -- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f396806ccc0 msgr2=0x7f396806d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f3966ffd640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f396806ccc0 0x7f396806d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f3966ffd640 1 -- 172.21.15.59:0/2855990462 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396809f460 con 0x7f3954003b70 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.133+0000 7f39677fe640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f396806ccc0 0x7f396806d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.134+0000 7f3966ffd640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 0x7f396809e4e0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f39580018f0 tx=0x7f39580019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.134+0000 7f395f7fe640 1 -- 172.21.15.59:0/2855990462 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958046070 con 0x7f3954003b70 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.134+0000 7f395f7fe640 1 -- 172.21.15.59:0/2855990462 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395802cbd0 con 0x7f3954003b70 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.134+0000 7f395f7fe640 1 -- 172.21.15.59:0/2855990462 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958001c00 con 0x7f3954003b70 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.134+0000 7f396d2a4640 1 -- 172.21.15.59:0/2855990462 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396809ebc0 con 0x7f3954003b70 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.134+0000 7f396d2a4640 1 -- 172.21.15.59:0/2855990462 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396809f180 con 0x7f3954003b70 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.135+0000 7f396d2a4640 1 -- 172.21.15.59:0/2855990462 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396804f330 con 0x7f3954003b70 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.135+0000 7f395f7fe640 1 -- 172.21.15.59:0/2855990462 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3958007070 con 0x7f3954003b70 2024-03-06T07:31:33.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.136+0000 7f395f7fe640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f394806b500 0x7f394806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:33.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.136+0000 7f395f7fe640 1 -- 172.21.15.59:0/2855990462 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f39580ab790 con 0x7f3954003b70 2024-03-06T07:31:33.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.136+0000 7f395ffff640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f394806b500 0x7f394806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:33.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.136+0000 7f395ffff640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f394806b500 0x7f394806d9c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f39680ca2e0 tx=0x7f3950009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:33.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.140+0000 7f395f7fe640 1 -- 172.21.15.59:0/2855990462 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f395807a5e0 con 0x7f3954003b70 2024-03-06T07:31:33.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.364+0000 7f396d2a4640 1 -- 172.21.15.59:0/2855990462 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39680b52f0 con 0x7f3954003b70 2024-03-06T07:31:33.366 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.364+0000 7f395f7fe640 1 -- 172.21.15.59:0/2855990462 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f3958079ee0 con 0x7f3954003b70 2024-03-06T07:31:33.366 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:33.366 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:33.367 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.366+0000 7f396d2a4640 1 -- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f394806b500 msgr2=0x7f394806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.366+0000 7f396d2a4640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f394806b500 0x7f394806d9c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f39680ca2e0 tx=0x7f3950009040 comp rx=0 tx=0).stop 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.366+0000 7f396d2a4640 1 -- 172.21.15.59:0/2855990462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 msgr2=0x7f396809e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.366+0000 7f396d2a4640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 0x7f396809e4e0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f39580018f0 tx=0x7f39580019c0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.366+0000 7f396d2a4640 1 -- 172.21.15.59:0/2855990462 shutdown_connections 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.366+0000 7f396d2a4640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f394806b500 0x7f394806d9c0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.366+0000 7f396d2a4640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f396806ccc0 0x7f396806d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.366+0000 7f396d2a4640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39680a0810 0x7f396815c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.366+0000 7f396d2a4640 1 --2- 172.21.15.59:0/2855990462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3954003b70 0x7f396809e4e0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.366+0000 7f396d2a4640 1 -- 172.21.15.59:0/2855990462 >> 172.21.15.59:0/2855990462 conn(0x7f39680d3cd0 msgr2=0x7f39680cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.367+0000 7f396d2a4640 1 -- 172.21.15.59:0/2855990462 shutdown_connections 2024-03-06T07:31:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:33.367+0000 7f396d2a4640 1 -- 172.21.15.59:0/2855990462 wait complete. 2024-03-06T07:31:33.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:33.581 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:34.582 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:34.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.682+0000 7fad893e9640 1 Processor -- start 2024-03-06T07:31:34.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.682+0000 7fad893e9640 1 -- start start 2024-03-06T07:31:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.683+0000 7fad893e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8415dfe0 0x7fad840cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.683+0000 7fad893e9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fad840cf830 0x7fad840d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.683+0000 7fad893e9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fad840d27f0 0x7fad840d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.683+0000 7fad893e9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad840d2c50 con 0x7fad840cf830 2024-03-06T07:31:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.683+0000 7fad893e9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad840b31e0 con 0x7fad840d27f0 2024-03-06T07:31:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.683+0000 7fad893e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad840b3d60 con 0x7fad8415dfe0 2024-03-06T07:31:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.683+0000 7fad7a7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fad840cf830 0x7fad840d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.683+0000 7fad7a7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fad840cf830 0x7fad840d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51436/0 (socket says 172.21.15.59:51436) 2024-03-06T07:31:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.683+0000 7fad7a7fc640 1 -- 172.21.15.59:0/1903349171 learned_addr learned my addr 172.21.15.59:0/1903349171 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.683+0000 7fad82ffd640 1 --2- 172.21.15.59:0/1903349171 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8415dfe0 0x7fad840cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.683+0000 7fad837fe640 1 --2- 172.21.15.59:0/1903349171 >> v2:172.21.15.164:3300/0 conn(0x7fad840d27f0 0x7fad840d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.684+0000 7fad7a7fc640 1 -- 172.21.15.59:0/1903349171 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8415dfe0 msgr2=0x7fad840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.684+0000 7fad7a7fc640 1 --2- 172.21.15.59:0/1903349171 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8415dfe0 0x7fad840cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.684+0000 7fad7a7fc640 1 -- 172.21.15.59:0/1903349171 >> v2:172.21.15.164:3300/0 conn(0x7fad840d27f0 msgr2=0x7fad840d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.684+0000 7fad7a7fc640 1 --2- 172.21.15.59:0/1903349171 >> v2:172.21.15.164:3300/0 conn(0x7fad840d27f0 0x7fad840d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.684+0000 7fad7a7fc640 1 -- 172.21.15.59:0/1903349171 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad840b3430 con 0x7fad840cf830 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.684+0000 7fad82ffd640 1 --2- 172.21.15.59:0/1903349171 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8415dfe0 0x7fad840cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.684+0000 7fad7a7fc640 1 --2- 172.21.15.59:0/1903349171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad840cf830 0x7fad840d22b0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fad84067ad0 tx=0x7fad6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267d510540b0abe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.685+0000 7fad827fc640 1 -- 172.21.15.59:0/1903349171 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c035020 con 0x7fad840cf830 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.685+0000 7fad827fc640 1 -- 172.21.15.59:0/1903349171 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6c031870 con 0x7fad840cf830 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.685+0000 7fad827fc640 1 -- 172.21.15.59:0/1903349171 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c031c40 con 0x7fad840cf830 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.685+0000 7fad893e9640 1 -- 172.21.15.59:0/1903349171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad840cf830 msgr2=0x7fad840d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.685+0000 7fad893e9640 1 --2- 172.21.15.59:0/1903349171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad840cf830 0x7fad840d22b0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fad84067ad0 tx=0x7fad6c02b910 comp rx=0 tx=0).stop 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.685+0000 7fad893e9640 1 -- 172.21.15.59:0/1903349171 shutdown_connections 2024-03-06T07:31:34.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.685+0000 7fad893e9640 1 --2- 172.21.15.59:0/1903349171 >> v2:172.21.15.164:3300/0 conn(0x7fad840d27f0 0x7fad840d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.685+0000 7fad893e9640 1 --2- 172.21.15.59:0/1903349171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad840cf830 0x7fad840d22b0 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.685+0000 7fad893e9640 1 --2- 172.21.15.59:0/1903349171 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8415dfe0 0x7fad840cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.685+0000 7fad893e9640 1 -- 172.21.15.59:0/1903349171 >> 172.21.15.59:0/1903349171 conn(0x7fad840d3cd0 msgr2=0x7fad840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.685+0000 7fad893e9640 1 -- 172.21.15.59:0/1903349171 shutdown_connections 2024-03-06T07:31:34.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.686+0000 7fad893e9640 1 -- 172.21.15.59:0/1903349171 wait complete. 2024-03-06T07:31:34.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.687+0000 7fad893e9640 1 Processor -- start 2024-03-06T07:31:34.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.687+0000 7fad893e9640 1 -- start start 2024-03-06T07:31:34.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad893e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad840cf830 0x7fad84158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad893e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad840d27f0 0x7fad8415ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad893e9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8415dfe0 0x7fad840b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad893e9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad840d1c70 con 0x7fad840d27f0 2024-03-06T07:31:34.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad893e9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad840b39f0 con 0x7fad8415dfe0 2024-03-06T07:31:34.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad893e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad840d1df0 con 0x7fad840cf830 2024-03-06T07:31:34.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad840d27f0 0x7fad8415ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7a7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad840d27f0 0x7fad8415ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51446/0 (socket says 172.21.15.59:51446) 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7a7fc640 1 -- 172.21.15.59:0/2248525271 learned_addr learned my addr 172.21.15.59:0/2248525271 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad82ffd640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad840cf830 0x7fad84158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad837fe640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8415dfe0 0x7fad840b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7a7fc640 1 -- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad840cf830 msgr2=0x7fad84158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7a7fc640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad840cf830 0x7fad84158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7a7fc640 1 -- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8415dfe0 msgr2=0x7fad840b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7a7fc640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8415dfe0 0x7fad840b7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7a7fc640 1 -- 172.21.15.59:0/2248525271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad84158aa0 con 0x7fad840d27f0 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7a7fc640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad840d27f0 0x7fad8415ee10 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fad6c02b950 tx=0x7fad6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7bfff640 1 -- 172.21.15.59:0/2248525271 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c040070 con 0x7fad840d27f0 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7bfff640 1 -- 172.21.15.59:0/2248525271 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6c030d00 con 0x7fad840d27f0 2024-03-06T07:31:34.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad7bfff640 1 -- 172.21.15.59:0/2248525271 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c035040 con 0x7fad840d27f0 2024-03-06T07:31:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.688+0000 7fad893e9640 1 -- 172.21.15.59:0/2248525271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8415f350 con 0x7fad840d27f0 2024-03-06T07:31:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.689+0000 7fad893e9640 1 -- 172.21.15.59:0/2248525271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad840b7040 con 0x7fad840d27f0 2024-03-06T07:31:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.689+0000 7fad893e9640 1 -- 172.21.15.59:0/2248525271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad840b5200 con 0x7fad840d27f0 2024-03-06T07:31:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.690+0000 7fad7bfff640 1 -- 172.21.15.59:0/2248525271 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad6c04b060 con 0x7fad840d27f0 2024-03-06T07:31:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.690+0000 7fad7bfff640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 0x7fad6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.691+0000 7fad7bfff640 1 -- 172.21.15.59:0/2248525271 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fad6c0abe10 con 0x7fad840d27f0 2024-03-06T07:31:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.691+0000 7fad82ffd640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 0x7fad6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.691+0000 7fad82ffd640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 0x7fad6406d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fad840d0800 tx=0x7fad74008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.695+0000 7fad7bfff640 1 -- 172.21.15.59:0/2248525271 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad840b5200 con 0x7fad840d27f0 2024-03-06T07:31:34.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.912+0000 7fad893e9640 1 -- 172.21.15.59:0/2248525271 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad840b0af0 con 0x7fad840d27f0 2024-03-06T07:31:34.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.912+0000 7fad7bfff640 1 -- 172.21.15.59:0/2248525271 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7fad6c07a620 con 0x7fad840d27f0 2024-03-06T07:31:34.915 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:34.915 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:34.915 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:34.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 -- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 msgr2=0x7fad6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 0x7fad6406d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fad840d0800 tx=0x7fad74008040 comp rx=0 tx=0).stop 2024-03-06T07:31:34.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 -- 172.21.15.59:0/2248525271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad840d27f0 msgr2=0x7fad8415ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad840d27f0 0x7fad8415ee10 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fad6c02b950 tx=0x7fad6c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 -- 172.21.15.59:0/2248525271 shutdown_connections 2024-03-06T07:31:34.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad6406b4d0 0x7fad6406d990 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8415dfe0 0x7fad840b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad840d27f0 0x7fad8415ee10 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 --2- 172.21.15.59:0/2248525271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad840cf830 0x7fad84158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 -- 172.21.15.59:0/2248525271 >> 172.21.15.59:0/2248525271 conn(0x7fad840d3cd0 msgr2=0x7fad8406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 -- 172.21.15.59:0/2248525271 shutdown_connections 2024-03-06T07:31:34.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:34.915+0000 7fad893e9640 1 -- 172.21.15.59:0/2248525271 wait complete. 2024-03-06T07:31:35.129 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:35.129 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:36.130 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:36.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.230+0000 7f9b67956640 1 Processor -- start 2024-03-06T07:31:36.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.230+0000 7f9b67956640 1 -- start start 2024-03-06T07:31:36.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.231+0000 7f9b67956640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9b600d22b0 0x7f9b600d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.231+0000 7f9b67956640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b600b41f0 0x7f9b600b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.231+0000 7f9b67956640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b600b3430 con 0x7f9b6015dfe0 2024-03-06T07:31:36.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.231+0000 7f9b67956640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b600d2c50 con 0x7f9b600d22b0 2024-03-06T07:31:36.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.231+0000 7f9b67956640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b600b31e0 con 0x7f9b600b41f0 2024-03-06T07:31:36.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.231+0000 7f9b656f2640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9b6015dfe0 0x7f9b600cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57692/0 (socket says 172.21.15.59:57692) 2024-03-06T07:31:36.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.231+0000 7f9b656f2640 1 -- 172.21.15.59:0/448473411 learned_addr learned my addr 172.21.15.59:0/448473411 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:36.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.231+0000 7f9b64ef1640 1 --2- 172.21.15.59:0/448473411 >> v2:172.21.15.164:3300/0 conn(0x7f9b600d22b0 0x7f9b600d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.231+0000 7f9b65ef3640 1 --2- 172.21.15.59:0/448473411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b600b41f0 0x7f9b600b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.231+0000 7f9b57fff640 1 -- 172.21.15.59:0/448473411 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534490728 0 0) 0x7f9b600b3430 con 0x7f9b6015dfe0 2024-03-06T07:31:36.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.231+0000 7f9b57fff640 1 -- 172.21.15.59:0/448473411 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b4c003520 con 0x7f9b6015dfe0 2024-03-06T07:31:36.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.232+0000 7f9b64ef1640 1 -- 172.21.15.59:0/448473411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b600b41f0 msgr2=0x7f9b600b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:36.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.232+0000 7f9b64ef1640 1 --2- 172.21.15.59:0/448473411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b600b41f0 0x7f9b600b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.232+0000 7f9b57fff640 1 -- 172.21.15.59:0/448473411 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 106264630 0 0) 0x7f9b4c003520 con 0x7f9b6015dfe0 2024-03-06T07:31:36.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.232+0000 7f9b64ef1640 1 -- 172.21.15.59:0/448473411 >> v1:172.21.15.59:6789/0 conn(0x7f9b6015dfe0 legacy=0x7f9b600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.232+0000 7f9b64ef1640 1 -- 172.21.15.59:0/448473411 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b600b3710 con 0x7f9b600d22b0 2024-03-06T07:31:36.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.232+0000 7f9b64ef1640 1 --2- 172.21.15.59:0/448473411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b600d22b0 0x7f9b600d2710 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f9b500097a0 tx=0x7f9b5002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7547f023b6dc777 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.233+0000 7f9b57fff640 1 -- 172.21.15.59:0/448473411 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b50037070 con 0x7f9b600d22b0 2024-03-06T07:31:36.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.233+0000 7f9b57fff640 1 -- 172.21.15.59:0/448473411 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b50030510 con 0x7f9b600d22b0 2024-03-06T07:31:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.233+0000 7f9b57fff640 1 -- 172.21.15.59:0/448473411 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c001670 con 0x7f9b600d22b0 2024-03-06T07:31:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.233+0000 7f9b67956640 1 -- 172.21.15.59:0/448473411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b600d22b0 msgr2=0x7f9b600d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.233+0000 7f9b67956640 1 --2- 172.21.15.59:0/448473411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b600d22b0 0x7f9b600d2710 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f9b500097a0 tx=0x7f9b5002b790 comp rx=0 tx=0).stop 2024-03-06T07:31:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.233+0000 7f9b67956640 1 -- 172.21.15.59:0/448473411 shutdown_connections 2024-03-06T07:31:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.233+0000 7f9b67956640 1 --2- 172.21.15.59:0/448473411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b600b41f0 0x7f9b600b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.233+0000 7f9b67956640 1 --2- 172.21.15.59:0/448473411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b600d22b0 0x7f9b600d2710 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.233+0000 7f9b67956640 1 -- 172.21.15.59:0/448473411 >> 172.21.15.59:0/448473411 conn(0x7f9b600d3cd0 msgr2=0x7f9b600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.233+0000 7f9b67956640 1 -- 172.21.15.59:0/448473411 shutdown_connections 2024-03-06T07:31:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.233+0000 7f9b67956640 1 -- 172.21.15.59:0/448473411 wait complete. 2024-03-06T07:31:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.234+0000 7f9b67956640 1 Processor -- start 2024-03-06T07:31:36.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.234+0000 7f9b67956640 1 -- start start 2024-03-06T07:31:36.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b67956640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b600b41f0 0x7f9b601f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b67956640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b600d22b0 0x7f9b60160f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b67956640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b6015dfe0 0x7f9b60151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b67956640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b600cfa00 con 0x7f9b600d22b0 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b67956640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b600b3d60 con 0x7f9b6015dfe0 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b67956640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b600d1c70 con 0x7f9b600b41f0 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b64ef1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b600d22b0 0x7f9b60160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b64ef1640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b600d22b0 0x7f9b60160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51452/0 (socket says 172.21.15.59:51452) 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b64ef1640 1 -- 172.21.15.59:0/3084933163 learned_addr learned my addr 172.21.15.59:0/3084933163 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b656f2640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b600b41f0 0x7f9b601f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b65ef3640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b6015dfe0 0x7f9b60151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b64ef1640 1 -- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b600b41f0 msgr2=0x7f9b601f3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b64ef1640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b600b41f0 0x7f9b601f3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b64ef1640 1 -- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b6015dfe0 msgr2=0x7f9b60151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b64ef1640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b6015dfe0 0x7f9b60151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.235+0000 7f9b64ef1640 1 -- 172.21.15.59:0/3084933163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b600cc220 con 0x7f9b600d22b0 2024-03-06T07:31:36.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.236+0000 7f9b64ef1640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b600d22b0 0x7f9b60160f80 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f9b50000c00 tx=0x7f9b500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.236+0000 7f9b55ffb640 1 -- 172.21.15.59:0/3084933163 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b50041020 con 0x7f9b600d22b0 2024-03-06T07:31:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.236+0000 7f9b55ffb640 1 -- 172.21.15.59:0/3084933163 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b5003c870 con 0x7f9b600d22b0 2024-03-06T07:31:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.236+0000 7f9b67956640 1 -- 172.21.15.59:0/3084933163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b600cc500 con 0x7f9b600d22b0 2024-03-06T07:31:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.236+0000 7f9b55ffb640 1 -- 172.21.15.59:0/3084933163 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b50037040 con 0x7f9b600d22b0 2024-03-06T07:31:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.236+0000 7f9b67956640 1 -- 172.21.15.59:0/3084933163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b601615f0 con 0x7f9b600d22b0 2024-03-06T07:31:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.237+0000 7f9b67956640 1 -- 172.21.15.59:0/3084933163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b600cc8f0 con 0x7f9b600d22b0 2024-03-06T07:31:36.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.238+0000 7f9b55ffb640 1 -- 172.21.15.59:0/3084933163 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b50044020 con 0x7f9b600d22b0 2024-03-06T07:31:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.238+0000 7f9b55ffb640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b4406b4d0 0x7f9b4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.238+0000 7f9b55ffb640 1 -- 172.21.15.59:0/3084933163 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9b500abaf0 con 0x7f9b600d22b0 2024-03-06T07:31:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.238+0000 7f9b656f2640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b4406b4d0 0x7f9b4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.239+0000 7f9b656f2640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b4406b4d0 0x7f9b4406d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c007940 tx=0x7f9b5c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.243+0000 7f9b55ffb640 1 -- 172.21.15.59:0/3084933163 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b5004b030 con 0x7f9b600d22b0 2024-03-06T07:31:36.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.463+0000 7f9b67956640 1 -- 172.21.15.59:0/3084933163 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b600d0900 con 0x7f9b600d22b0 2024-03-06T07:31:36.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.463+0000 7f9b55ffb640 1 -- 172.21.15.59:0/3084933163 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f9b600d0900 con 0x7f9b600d22b0 2024-03-06T07:31:36.465 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:36.465 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:36.466 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.465+0000 7f9b67956640 1 -- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b4406b4d0 msgr2=0x7f9b4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.465+0000 7f9b67956640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b4406b4d0 0x7f9b4406d990 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c007940 tx=0x7f9b5c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.465+0000 7f9b67956640 1 -- 172.21.15.59:0/3084933163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b600d22b0 msgr2=0x7f9b60160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.465+0000 7f9b67956640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b600d22b0 0x7f9b60160f80 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f9b50000c00 tx=0x7f9b500046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.466+0000 7f9b67956640 1 -- 172.21.15.59:0/3084933163 shutdown_connections 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.466+0000 7f9b67956640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b4406b4d0 0x7f9b4406d990 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.466+0000 7f9b67956640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b6015dfe0 0x7f9b60151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.466+0000 7f9b67956640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b600d22b0 0x7f9b60160f80 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.466+0000 7f9b67956640 1 --2- 172.21.15.59:0/3084933163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b600b41f0 0x7f9b601f3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.466+0000 7f9b67956640 1 -- 172.21.15.59:0/3084933163 >> 172.21.15.59:0/3084933163 conn(0x7f9b600d3cd0 msgr2=0x7f9b6006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.466+0000 7f9b67956640 1 -- 172.21.15.59:0/3084933163 shutdown_connections 2024-03-06T07:31:36.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:36.466+0000 7f9b67956640 1 -- 172.21.15.59:0/3084933163 wait complete. 2024-03-06T07:31:36.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:36.680 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:37.681 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:37.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.781+0000 7f42ba4af640 1 Processor -- start 2024-03-06T07:31:37.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.781+0000 7f42ba4af640 1 -- start start 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.782+0000 7f42ba4af640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f42b413dc10 0x7f42b40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.782+0000 7f42ba4af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415ef60 0x7f42b40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.782+0000 7f42ba4af640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b415f580 con 0x7f42b415bf30 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.782+0000 7f42ba4af640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b40cf3f0 con 0x7f42b413dc10 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.782+0000 7f42ba4af640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b415b210 con 0x7f42b415ef60 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.782+0000 7f42b37fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f42b415bf30 0x7f42b415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57708/0 (socket says 172.21.15.59:57708) 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.782+0000 7f42b37fe640 1 -- 172.21.15.59:0/1621067540 learned_addr learned my addr 172.21.15.59:0/1621067540 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.782+0000 7f42b3fff640 1 --2- 172.21.15.59:0/1621067540 >> v2:172.21.15.164:3300/0 conn(0x7f42b413dc10 0x7f42b40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.782+0000 7f42b8a4c640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415ef60 0x7f42b40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.782+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031334025 0 0) 0x7f42b415f580 con 0x7f42b415bf30 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.782+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f429c005730 con 0x7f42b415bf30 2024-03-06T07:31:37.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1252277982 0 0) 0x7f429c005730 con 0x7f42b415bf30 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415ef60 msgr2=0x7f42b40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415ef60 0x7f42b40b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 >> v2:172.21.15.164:3300/0 conn(0x7f42b413dc10 msgr2=0x7f42b40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 --2- 172.21.15.59:0/1621067540 >> v2:172.21.15.164:3300/0 conn(0x7f42b413dc10 0x7f42b40b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b415b450 con 0x7f42b415bf30 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f42a8001670 con 0x7f42b415bf30 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 >> v1:172.21.15.59:6789/0 conn(0x7f42b415bf30 legacy=0x7f42b415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 0x7f429c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f429c006a10 0x7f429c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415ef60 0x7f429c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b40cf3f0 con 0x7f429c005bc0 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b415b210 con 0x7f429c006a10 2024-03-06T07:31:37.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429c00dbd0 con 0x7f42b415ef60 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b3fff640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 0x7f429c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b8a4c640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415ef60 0x7f429c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b37fe640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f429c006a10 0x7f429c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b3fff640 1 -- 172.21.15.59:0/1621067540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415ef60 msgr2=0x7f429c0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b3fff640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415ef60 0x7f429c0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b3fff640 1 -- 172.21.15.59:0/1621067540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f429c006a10 msgr2=0x7f429c006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b3fff640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f429c006a10 0x7f429c006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.783+0000 7f42b3fff640 1 -- 172.21.15.59:0/1621067540 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c005730 con 0x7f429c005bc0 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.784+0000 7f42b3fff640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 0x7f429c005fe0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f42a4009af0 tx=0x7f429c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec939ec46cee371b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.784+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4039070 con 0x7f429c005bc0 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.784+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a402c710 con 0x7f429c005bc0 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.784+0000 7f42b2ffd640 1 -- 172.21.15.59:0/1621067540 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4009690 con 0x7f429c005bc0 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.785+0000 7f42ba4af640 1 -- 172.21.15.59:0/1621067540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 msgr2=0x7f429c005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.785+0000 7f42ba4af640 1 --2- 172.21.15.59:0/1621067540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 0x7f429c005fe0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f42a4009af0 tx=0x7f429c007a40 comp rx=0 tx=0).stop 2024-03-06T07:31:37.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.785+0000 7f42b8a4c640 1 -- 172.21.15.59:0/1621067540 reap_dead start 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.785+0000 7f42ba4af640 1 -- 172.21.15.59:0/1621067540 shutdown_connections 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.785+0000 7f42ba4af640 1 -- 172.21.15.59:0/1621067540 >> 172.21.15.59:0/1621067540 conn(0x7f42b40cf8a0 msgr2=0x7f42b40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.785+0000 7f42ba4af640 1 -- 172.21.15.59:0/1621067540 shutdown_connections 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.785+0000 7f42ba4af640 1 -- 172.21.15.59:0/1621067540 wait complete. 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.786+0000 7f42ba4af640 1 Processor -- start 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.786+0000 7f42ba4af640 1 -- start start 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.787+0000 7f42ba4af640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 0x7f42b4139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.787+0000 7f42ba4af640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42b415a5a0 0x7f42b413e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.787+0000 7f42ba4af640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415aa00 0x7f42b413e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.787+0000 7f42ba4af640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b40b2e90 con 0x7f429c005bc0 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.787+0000 7f42ba4af640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b413dff0 con 0x7f42b415a5a0 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.787+0000 7f42ba4af640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b40b3010 con 0x7f42b415aa00 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.787+0000 7f42b3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 0x7f42b4139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.787+0000 7f42b3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 0x7f42b4139e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51464/0 (socket says 172.21.15.59:51464) 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.787+0000 7f42b3fff640 1 -- 172.21.15.59:0/227643141 learned_addr learned my addr 172.21.15.59:0/227643141 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.787+0000 7f42b37fe640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42b415a5a0 0x7f42b413e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.787+0000 7f42b8a4c640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415aa00 0x7f42b413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.788+0000 7f42b3fff640 1 -- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415aa00 msgr2=0x7f42b413e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.788+0000 7f42b3fff640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415aa00 0x7f42b413e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.788+0000 7f42b3fff640 1 -- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42b415a5a0 msgr2=0x7f42b413e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.788+0000 7f42b3fff640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42b415a5a0 0x7f42b413e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.788+0000 7f42b3fff640 1 -- 172.21.15.59:0/227643141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b41410f0 con 0x7f429c005bc0 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.788+0000 7f42b3fff640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 0x7f42b4139e00 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f42a4037040 tx=0x7f42a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.788+0000 7f42b0ff9640 1 -- 172.21.15.59:0/227643141 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4039050 con 0x7f429c005bc0 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.788+0000 7f42b0ff9640 1 -- 172.21.15.59:0/227643141 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a4045070 con 0x7f429c005bc0 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.788+0000 7f42ba4af640 1 -- 172.21.15.59:0/227643141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b41f3a50 con 0x7f429c005bc0 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.788+0000 7f42b0ff9640 1 -- 172.21.15.59:0/227643141 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4003a50 con 0x7f429c005bc0 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.788+0000 7f42ba4af640 1 -- 172.21.15.59:0/227643141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b41f4010 con 0x7f429c005bc0 2024-03-06T07:31:37.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.789+0000 7f42ba4af640 1 -- 172.21.15.59:0/227643141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b404f330 con 0x7f429c005bc0 2024-03-06T07:31:37.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.789+0000 7f42b0ff9640 1 -- 172.21.15.59:0/227643141 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42a4003bf0 con 0x7f429c005bc0 2024-03-06T07:31:37.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.790+0000 7f42b0ff9640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a006b4d0 0x7f42a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.790+0000 7f42b0ff9640 1 -- 172.21.15.59:0/227643141 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f42a40abcc0 con 0x7f429c005bc0 2024-03-06T07:31:37.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.790+0000 7f42b37fe640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a006b4d0 0x7f42a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.791+0000 7f42b37fe640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a006b4d0 0x7f42a006d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f42a8007fd0 tx=0x7f42a8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.800 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:37.798+0000 7f42b0ff9640 1 -- 172.21.15.59:0/227643141 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42a40b2080 con 0x7f429c005bc0 2024-03-06T07:31:38.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.017+0000 7f42ba4af640 1 -- 172.21.15.59:0/227643141 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42b41412c0 con 0x7f429c005bc0 2024-03-06T07:31:38.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.017+0000 7f42b0ff9640 1 -- 172.21.15.59:0/227643141 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f42a404e020 con 0x7f429c005bc0 2024-03-06T07:31:38.019 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:38.019 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:38.020 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:38.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 -- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a006b4d0 msgr2=0x7f42a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a006b4d0 0x7f42a006d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f42a8007fd0 tx=0x7f42a8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:38.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 -- 172.21.15.59:0/227643141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 msgr2=0x7f42b4139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 0x7f42b4139e00 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f42a4037040 tx=0x7f42a4004730 comp rx=0 tx=0).stop 2024-03-06T07:31:38.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 -- 172.21.15.59:0/227643141 shutdown_connections 2024-03-06T07:31:38.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f42a006b4d0 0x7f42a006d990 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42b415aa00 0x7f42b413e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42b415a5a0 0x7f42b413e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 --2- 172.21.15.59:0/227643141 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f429c005bc0 0x7f42b4139e00 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 -- 172.21.15.59:0/227643141 >> 172.21.15.59:0/227643141 conn(0x7f42b40cf8a0 msgr2=0x7f42b40ccbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 -- 172.21.15.59:0/227643141 shutdown_connections 2024-03-06T07:31:38.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:38.019+0000 7f42ba4af640 1 -- 172.21.15.59:0/227643141 wait complete. 2024-03-06T07:31:38.233 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:38.233 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:39.234 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:39.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.334+0000 7f71fb7bc640 1 Processor -- start 2024-03-06T07:31:39.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.334+0000 7f71fb7bc640 1 -- start start 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71fb7bc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f71f415bb20 0x7f71f40d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71fb7bc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f71f40cd8c0 0x7f71f40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71fb7bc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f4161980 con 0x7f71f40cfff0 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71fb7bc640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f4161c60 con 0x7f71f40cd8c0 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71fb7bc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f4161f40 con 0x7f71f415bb20 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71f9d59640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f71f40cd8c0 0x7f71f40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71f9d59640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f71f40cd8c0 0x7f71f40d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51472/0 (socket says 172.21.15.59:51472) 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71f8d57640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f71f40cfff0 0x7f71f40ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57714/0 (socket says 172.21.15.59:57714) 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71f9d59640 1 -- 172.21.15.59:0/2710305558 learned_addr learned my addr 172.21.15.59:0/2710305558 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71f9d59640 1 -- 172.21.15.59:0/2710305558 >> v2:172.21.15.164:3300/0 conn(0x7f71f415bb20 msgr2=0x7f71f40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71f9d59640 1 --2- 172.21.15.59:0/2710305558 >> v2:172.21.15.164:3300/0 conn(0x7f71f415bb20 0x7f71f40d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71f9d59640 1 -- 172.21.15.59:0/2710305558 >> v1:172.21.15.59:6789/0 conn(0x7f71f40cfff0 legacy=0x7f71f40ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71f9d59640 1 -- 172.21.15.59:0/2710305558 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f4162220 con 0x7f71f40cd8c0 2024-03-06T07:31:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.335+0000 7f71ebfff640 1 -- 172.21.15.59:0/2710305558 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005567949 0 0) 0x7f71f4161980 con 0x7f71f40cfff0 2024-03-06T07:31:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.336+0000 7f71f9d59640 1 --2- 172.21.15.59:0/2710305558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71f40cd8c0 0x7f71f40d04a0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f71f40d2bc0 tx=0x7f71ec02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0dfc3b161d5cc8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.336+0000 7f71ebfff640 1 -- 172.21.15.59:0/2710305558 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec02cb20 con 0x7f71f40cd8c0 2024-03-06T07:31:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.336+0000 7f71ebfff640 1 -- 172.21.15.59:0/2710305558 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ec02ccc0 con 0x7f71f40cd8c0 2024-03-06T07:31:39.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.336+0000 7f71ebfff640 1 -- 172.21.15.59:0/2710305558 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ec034610 con 0x7f71f40cd8c0 2024-03-06T07:31:39.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.336+0000 7f71fb7bc640 1 -- 172.21.15.59:0/2710305558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71f40cd8c0 msgr2=0x7f71f40d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.336+0000 7f71fb7bc640 1 --2- 172.21.15.59:0/2710305558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71f40cd8c0 0x7f71f40d04a0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f71f40d2bc0 tx=0x7f71ec02bf80 comp rx=0 tx=0).stop 2024-03-06T07:31:39.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.336+0000 7f71fb7bc640 1 -- 172.21.15.59:0/2710305558 shutdown_connections 2024-03-06T07:31:39.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.336+0000 7f71fb7bc640 1 --2- 172.21.15.59:0/2710305558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71f40cd8c0 0x7f71f40d04a0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.336+0000 7f71fb7bc640 1 --2- 172.21.15.59:0/2710305558 >> v2:172.21.15.164:3300/0 conn(0x7f71f415bb20 0x7f71f40d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.336+0000 7f71fb7bc640 1 -- 172.21.15.59:0/2710305558 >> 172.21.15.59:0/2710305558 conn(0x7f71f409a9f0 msgr2=0x7f71f409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.337+0000 7f71fb7bc640 1 -- 172.21.15.59:0/2710305558 shutdown_connections 2024-03-06T07:31:39.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.337+0000 7f71fb7bc640 1 -- 172.21.15.59:0/2710305558 wait complete. 2024-03-06T07:31:39.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.338+0000 7f71fb7bc640 1 Processor -- start 2024-03-06T07:31:39.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.338+0000 7f71fb7bc640 1 -- start start 2024-03-06T07:31:39.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.339+0000 7f71fb7bc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71f40cd8c0 0x7f71f40c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71fb7bc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71f40cfff0 0x7f71f41546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71fb7bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71f415bb20 0x7f71f4154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71fb7bc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40d24b0 con 0x7f71f40cfff0 2024-03-06T07:31:39.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71fb7bc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40d2330 con 0x7f71f40cd8c0 2024-03-06T07:31:39.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71fb7bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40d1980 con 0x7f71f415bb20 2024-03-06T07:31:39.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71f8d57640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71f40cfff0 0x7f71f41546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71f8d57640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71f40cfff0 0x7f71f41546a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51474/0 (socket says 172.21.15.59:51474) 2024-03-06T07:31:39.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71f8d57640 1 -- 172.21.15.59:0/3958942142 learned_addr learned my addr 172.21.15.59:0/3958942142 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71f9558640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71f40cd8c0 0x7f71f40c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71f9d59640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71f415bb20 0x7f71f4154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71f8d57640 1 -- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71f415bb20 msgr2=0x7f71f4154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71f8d57640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71f415bb20 0x7f71f4154be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71f8d57640 1 -- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71f40cd8c0 msgr2=0x7f71f40c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71f8d57640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71f40cd8c0 0x7f71f40c6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71f8d57640 1 -- 172.21.15.59:0/3958942142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f40b0150 con 0x7f71f40cfff0 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71f8d57640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71f40cfff0 0x7f71f41546a0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f71e400c4d0 tx=0x7f71e400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71e9ffb640 1 -- 172.21.15.59:0/3958942142 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4018070 con 0x7f71f40cfff0 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71e9ffb640 1 -- 172.21.15.59:0/3958942142 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e4007500 con 0x7f71f40cfff0 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71fb7bc640 1 -- 172.21.15.59:0/3958942142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f41e7980 con 0x7f71f40cfff0 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.340+0000 7f71e9ffb640 1 -- 172.21.15.59:0/3958942142 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4004a20 con 0x7f71f40cfff0 2024-03-06T07:31:39.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.341+0000 7f71fb7bc640 1 -- 172.21.15.59:0/3958942142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f41e7f40 con 0x7f71f40cfff0 2024-03-06T07:31:39.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.342+0000 7f71fb7bc640 1 -- 172.21.15.59:0/3958942142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f404f330 con 0x7f71f40cfff0 2024-03-06T07:31:39.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.342+0000 7f71e9ffb640 1 -- 172.21.15.59:0/3958942142 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71e401b070 con 0x7f71f40cfff0 2024-03-06T07:31:39.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.342+0000 7f71e9ffb640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71d806b4d0 0x7f71d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:39.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.343+0000 7f71e9ffb640 1 -- 172.21.15.59:0/3958942142 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f71e4089600 con 0x7f71f40cfff0 2024-03-06T07:31:39.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.343+0000 7f71f9558640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71d806b4d0 0x7f71d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:39.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.343+0000 7f71f9558640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71d806b4d0 0x7f71d806d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f71f415b250 tx=0x7f71f0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:39.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.347+0000 7f71e9ffb640 1 -- 172.21.15.59:0/3958942142 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71e40583c0 con 0x7f71f40cfff0 2024-03-06T07:31:39.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.584+0000 7f71fb7bc640 1 -- 172.21.15.59:0/3958942142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71f40b1fe0 con 0x7f71f40cfff0 2024-03-06T07:31:39.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.584+0000 7f71e9ffb640 1 -- 172.21.15.59:0/3958942142 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+3417 (secure 0 0 0) 0x7f71e4057cc0 con 0x7f71f40cfff0 2024-03-06T07:31:39.586 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:39.586 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:12.572218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:39.586 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 106 2024-03-06T07:31:39.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 -- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71d806b4d0 msgr2=0x7f71d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71d806b4d0 0x7f71d806d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f71f415b250 tx=0x7f71f0008040 comp rx=0 tx=0).stop 2024-03-06T07:31:39.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 -- 172.21.15.59:0/3958942142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71f40cfff0 msgr2=0x7f71f41546a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71f40cfff0 0x7f71f41546a0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f71e400c4d0 tx=0x7f71e400c990 comp rx=0 tx=0).stop 2024-03-06T07:31:39.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 -- 172.21.15.59:0/3958942142 shutdown_connections 2024-03-06T07:31:39.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f71d806b4d0 0x7f71d806d990 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71f415bb20 0x7f71f4154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71f40cfff0 0x7f71f41546a0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 --2- 172.21.15.59:0/3958942142 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71f40cd8c0 0x7f71f40c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 -- 172.21.15.59:0/3958942142 >> 172.21.15.59:0/3958942142 conn(0x7f71f409a9f0 msgr2=0x7f71f406ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 -- 172.21.15.59:0/3958942142 shutdown_connections 2024-03-06T07:31:39.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:39.586+0000 7f71fb7bc640 1 -- 172.21.15.59:0/3958942142 wait complete. 2024-03-06T07:31:39.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:12.572218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:39.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:40.802 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:40.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.905+0000 7feb92afb640 1 Processor -- start 2024-03-06T07:31:40.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.906+0000 7feb92afb640 1 -- start start 2024-03-06T07:31:40.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.906+0000 7feb92afb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feb8c0d22b0 0x7feb8c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.906+0000 7feb92afb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feb8c0b41f0 0x7feb8c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.906+0000 7feb92afb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c0b3430 con 0x7feb8c15dfe0 2024-03-06T07:31:40.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.906+0000 7feb92afb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8c0d2c50 con 0x7feb8c0d22b0 2024-03-06T07:31:40.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.906+0000 7feb92afb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8c0b31e0 con 0x7feb8c0b41f0 2024-03-06T07:31:40.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.906+0000 7feb8bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feb8c0d22b0 0x7feb8c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.906+0000 7feb8bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feb8c0d22b0 0x7feb8c0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51478/0 (socket says 172.21.15.59:51478) 2024-03-06T07:31:40.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.906+0000 7feb8bfff640 1 -- 172.21.15.59:0/4031141247 learned_addr learned my addr 172.21.15.59:0/4031141247 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:40.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.907+0000 7feb91098640 1 --2- 172.21.15.59:0/4031141247 >> v2:172.21.15.164:3300/0 conn(0x7feb8c0b41f0 0x7feb8c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.907+0000 7feb8bfff640 1 -- 172.21.15.59:0/4031141247 >> v2:172.21.15.164:3300/0 conn(0x7feb8c0b41f0 msgr2=0x7feb8c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.907+0000 7feb8bfff640 1 --2- 172.21.15.59:0/4031141247 >> v2:172.21.15.164:3300/0 conn(0x7feb8c0b41f0 0x7feb8c0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.907+0000 7feb8bfff640 1 -- 172.21.15.59:0/4031141247 >> v1:172.21.15.164:6789/0 conn(0x7feb8c15dfe0 legacy=0x7feb8c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.907+0000 7feb8bfff640 1 -- 172.21.15.59:0/4031141247 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c0b3710 con 0x7feb8c0d22b0 2024-03-06T07:31:40.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.908+0000 7feb8bfff640 1 --2- 172.21.15.59:0/4031141247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb8c0d22b0 0x7feb8c0d2710 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7feb78009870 tx=0x7feb7802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33132e5c14453248 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.908+0000 7feb8b7fe640 1 -- 172.21.15.59:0/4031141247 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb78034070 con 0x7feb8c0d22b0 2024-03-06T07:31:40.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.908+0000 7feb8b7fe640 1 -- 172.21.15.59:0/4031141247 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb78030860 con 0x7feb8c0d22b0 2024-03-06T07:31:40.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.908+0000 7feb8b7fe640 1 -- 172.21.15.59:0/4031141247 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb78030c30 con 0x7feb8c0d22b0 2024-03-06T07:31:40.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.909+0000 7feb92afb640 1 -- 172.21.15.59:0/4031141247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb8c0d22b0 msgr2=0x7feb8c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.909+0000 7feb92afb640 1 --2- 172.21.15.59:0/4031141247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb8c0d22b0 0x7feb8c0d2710 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7feb78009870 tx=0x7feb7802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.909+0000 7feb92afb640 1 -- 172.21.15.59:0/4031141247 shutdown_connections 2024-03-06T07:31:40.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.909+0000 7feb92afb640 1 --2- 172.21.15.59:0/4031141247 >> v2:172.21.15.164:3300/0 conn(0x7feb8c0b41f0 0x7feb8c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.909+0000 7feb92afb640 1 --2- 172.21.15.59:0/4031141247 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb8c0d22b0 0x7feb8c0d2710 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.909+0000 7feb92afb640 1 -- 172.21.15.59:0/4031141247 >> 172.21.15.59:0/4031141247 conn(0x7feb8c0d3cd0 msgr2=0x7feb8c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.909+0000 7feb92afb640 1 -- 172.21.15.59:0/4031141247 shutdown_connections 2024-03-06T07:31:40.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.909+0000 7feb92afb640 1 -- 172.21.15.59:0/4031141247 wait complete. 2024-03-06T07:31:40.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.911+0000 7feb92afb640 1 Processor -- start 2024-03-06T07:31:40.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.911+0000 7feb92afb640 1 -- start start 2024-03-06T07:31:40.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb92afb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb8c0b41f0 0x7feb8c14b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb92afb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb8c0d22b0 0x7feb8c1f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb92afb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb8c15dfe0 0x7feb8c160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb92afb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0cfa00 con 0x7feb8c0b41f0 2024-03-06T07:31:40.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb92afb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0b3d60 con 0x7feb8c0d22b0 2024-03-06T07:31:40.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb92afb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0d1c70 con 0x7feb8c15dfe0 2024-03-06T07:31:40.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb90897640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb8c0b41f0 0x7feb8c14b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb90897640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb8c0b41f0 0x7feb8c14b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51492/0 (socket says 172.21.15.59:51492) 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb90897640 1 -- 172.21.15.59:0/1472871332 learned_addr learned my addr 172.21.15.59:0/1472871332 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb8bfff640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb8c0d22b0 0x7feb8c1f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb91098640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb8c15dfe0 0x7feb8c160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.912+0000 7feb90897640 1 -- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb8c15dfe0 msgr2=0x7feb8c160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.913+0000 7feb90897640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb8c15dfe0 0x7feb8c160320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.913+0000 7feb90897640 1 -- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb8c0d22b0 msgr2=0x7feb8c1f6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.913+0000 7feb90897640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb8c0d22b0 0x7feb8c1f6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.913+0000 7feb90897640 1 -- 172.21.15.59:0/1472871332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c15a640 con 0x7feb8c0b41f0 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.913+0000 7feb90897640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb8c0b41f0 0x7feb8c14b240 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7feb8000c3c0 tx=0x7feb8000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.913+0000 7feb897fa640 1 -- 172.21.15.59:0/1472871332 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800052d0 con 0x7feb8c0b41f0 2024-03-06T07:31:40.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.913+0000 7feb897fa640 1 -- 172.21.15.59:0/1472871332 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb80005490 con 0x7feb8c0b41f0 2024-03-06T07:31:40.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.913+0000 7feb897fa640 1 -- 172.21.15.59:0/1472871332 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb80007260 con 0x7feb8c0b41f0 2024-03-06T07:31:40.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.913+0000 7feb92afb640 1 -- 172.21.15.59:0/1472871332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8c0cae40 con 0x7feb8c0b41f0 2024-03-06T07:31:40.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.913+0000 7feb92afb640 1 -- 172.21.15.59:0/1472871332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8c0cb400 con 0x7feb8c0b41f0 2024-03-06T07:31:40.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.914+0000 7feb92afb640 1 -- 172.21.15.59:0/1472871332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8c04f330 con 0x7feb8c0b41f0 2024-03-06T07:31:40.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.914+0000 7feb897fa640 1 -- 172.21.15.59:0/1472871332 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb80007400 con 0x7feb8c0b41f0 2024-03-06T07:31:40.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.915+0000 7feb897fa640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb7006b500 0x7feb7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.915+0000 7feb897fa640 1 -- 172.21.15.59:0/1472871332 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7feb800895e0 con 0x7feb8c0b41f0 2024-03-06T07:31:40.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.915+0000 7feb8bfff640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb7006b500 0x7feb7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.916+0000 7feb8bfff640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb7006b500 0x7feb7006d9c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7feb78036000 tx=0x7feb7803b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:40.919+0000 7feb897fa640 1 -- 172.21.15.59:0/1472871332 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb800584d0 con 0x7feb8c0b41f0 2024-03-06T07:31:41.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.139+0000 7feb92afb640 1 -- 172.21.15.59:0/1472871332 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb8c0b0af0 con 0x7feb8c0b41f0 2024-03-06T07:31:41.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.140+0000 7feb897fa640 1 -- 172.21.15.59:0/1472871332 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7feb80057dd0 con 0x7feb8c0b41f0 2024-03-06T07:31:41.142 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:41.142 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:41.142 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 -- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb7006b500 msgr2=0x7feb7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb7006b500 0x7feb7006d9c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7feb78036000 tx=0x7feb7803b040 comp rx=0 tx=0).stop 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 -- 172.21.15.59:0/1472871332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb8c0b41f0 msgr2=0x7feb8c14b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb8c0b41f0 0x7feb8c14b240 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7feb8000c3c0 tx=0x7feb8000c880 comp rx=0 tx=0).stop 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 -- 172.21.15.59:0/1472871332 shutdown_connections 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7feb7006b500 0x7feb7006d9c0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb8c15dfe0 0x7feb8c160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb8c0d22b0 0x7feb8c1f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 --2- 172.21.15.59:0/1472871332 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feb8c0b41f0 0x7feb8c14b240 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 -- 172.21.15.59:0/1472871332 >> 172.21.15.59:0/1472871332 conn(0x7feb8c0d3cd0 msgr2=0x7feb8c0d08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 -- 172.21.15.59:0/1472871332 shutdown_connections 2024-03-06T07:31:41.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:41.142+0000 7feb92afb640 1 -- 172.21.15.59:0/1472871332 wait complete. 2024-03-06T07:31:41.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:41.356 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:42.357 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:42.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.457+0000 7f10d2b3e640 1 Processor -- start 2024-03-06T07:31:42.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.457+0000 7f10d2b3e640 1 -- start start 2024-03-06T07:31:42.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.458+0000 7f10d2b3e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f10cc0d2710 0x7f10cc0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.458+0000 7f10d2b3e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc0b3430 con 0x7f10cc0d22b0 2024-03-06T07:31:42.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.458+0000 7f10d2b3e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc0b3710 con 0x7f10cc15dfe0 2024-03-06T07:31:42.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.458+0000 7f10d2b3e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10cc0b31e0 con 0x7f10cc0d2710 2024-03-06T07:31:42.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.458+0000 7f10cbfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f10cc0d22b0 0x7f10cc0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57730/0 (socket says 172.21.15.59:57730) 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.458+0000 7f10cbfff640 1 -- 172.21.15.59:0/1192569947 learned_addr learned my addr 172.21.15.59:0/1192569947 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939967015 0 0) 0x7f10cc0b3430 con 0x7f10cc0d22b0 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10bc003520 con 0x7f10cc0d22b0 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10d10db640 1 --2- 172.21.15.59:0/1192569947 >> v2:172.21.15.164:3300/0 conn(0x7f10cc0d2710 0x7f10cc0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314477406 0 0) 0x7f10bc003520 con 0x7f10cc0d22b0 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 >> v2:172.21.15.164:3300/0 conn(0x7f10cc0d2710 msgr2=0x7f10cc0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 --2- 172.21.15.59:0/1192569947 >> v2:172.21.15.164:3300/0 conn(0x7f10cc0d2710 0x7f10cc0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 >> v1:172.21.15.164:6789/0 conn(0x7f10cc15dfe0 legacy=0x7f10cc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc0b39f0 con 0x7f10cc0d22b0 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f10b8001670 con 0x7f10cc0d22b0 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 >> v1:172.21.15.59:6789/0 conn(0x7f10cc0d22b0 legacy=0x7f10cc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 --2- 172.21.15.59:0/1192569947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10bc0039e0 0x7f10bc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 --2- 172.21.15.59:0/1192569947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10bc004800 0x7f10bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 --2- 172.21.15.59:0/1192569947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10bc005210 0x7f10bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0b31e0 con 0x7f10bc005210 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc00c140 con 0x7f10bc0039e0 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc00c340 con 0x7f10bc004800 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10d10db640 1 --2- 172.21.15.59:0/1192569947 >> v2:172.21.15.164:3300/0 conn(0x7f10cc0d2710 0x7f10cc0b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10d10db640 1 --2- 172.21.15.59:0/1192569947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10bc005210 0x7f10bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10d08da640 1 --2- 172.21.15.59:0/1192569947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10bc0039e0 0x7f10bc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10cbfff640 1 --2- 172.21.15.59:0/1192569947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10bc004800 0x7f10bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10d10db640 1 -- 172.21.15.59:0/1192569947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10bc004800 msgr2=0x7f10bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10d10db640 1 --2- 172.21.15.59:0/1192569947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10bc004800 0x7f10bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10d10db640 1 -- 172.21.15.59:0/1192569947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10bc0039e0 msgr2=0x7f10bc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10d10db640 1 --2- 172.21.15.59:0/1192569947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10bc0039e0 0x7f10bc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.459+0000 7f10d10db640 1 -- 172.21.15.59:0/1192569947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc003520 con 0x7f10bc005210 2024-03-06T07:31:42.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.460+0000 7f10d10db640 1 -- 172.21.15.59:0/1192569947 reap_dead start 2024-03-06T07:31:42.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.460+0000 7f10d10db640 1 --2- 172.21.15.59:0/1192569947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10bc005210 0x7f10bc00b9c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f10c40073b0 tx=0x7f10bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95fa83a48bc9a06f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.460+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c4039070 con 0x7f10bc005210 2024-03-06T07:31:42.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.460+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c402cdb0 con 0x7f10bc005210 2024-03-06T07:31:42.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.460+0000 7f10cb7fe640 1 -- 172.21.15.59:0/1192569947 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c4009c70 con 0x7f10bc005210 2024-03-06T07:31:42.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.461+0000 7f10d2b3e640 1 -- 172.21.15.59:0/1192569947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10bc005210 msgr2=0x7f10bc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.461+0000 7f10d2b3e640 1 --2- 172.21.15.59:0/1192569947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10bc005210 0x7f10bc00b9c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f10c40073b0 tx=0x7f10bc005830 comp rx=0 tx=0).stop 2024-03-06T07:31:42.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.461+0000 7f10d2b3e640 1 -- 172.21.15.59:0/1192569947 shutdown_connections 2024-03-06T07:31:42.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.461+0000 7f10d2b3e640 1 --2- 172.21.15.59:0/1192569947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10bc005210 0x7f10bc00b9c0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.461+0000 7f10d2b3e640 1 -- 172.21.15.59:0/1192569947 >> 172.21.15.59:0/1192569947 conn(0x7f10cc0d3cd0 msgr2=0x7f10cc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.461+0000 7f10d2b3e640 1 -- 172.21.15.59:0/1192569947 shutdown_connections 2024-03-06T07:31:42.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.461+0000 7f10d2b3e640 1 -- 172.21.15.59:0/1192569947 wait complete. 2024-03-06T07:31:42.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.462+0000 7f10d2b3e640 1 Processor -- start 2024-03-06T07:31:42.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.462+0000 7f10d2b3e640 1 -- start start 2024-03-06T07:31:42.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d2b3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10bc005210 0x7f10cc0cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d2b3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10cc0cada0 0x7f10cc0cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d2b3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10cc15bdc0 0x7f10cc15c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d2b3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0d1c70 con 0x7f10cc15bdc0 2024-03-06T07:31:42.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d2b3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0b3d60 con 0x7f10bc005210 2024-03-06T07:31:42.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d2b3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0d1df0 con 0x7f10cc0cada0 2024-03-06T07:31:42.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d10db640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10cc15bdc0 0x7f10cc15c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d10db640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10cc15bdc0 0x7f10cc15c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51514/0 (socket says 172.21.15.59:51514) 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d10db640 1 -- 172.21.15.59:0/3193298482 learned_addr learned my addr 172.21.15.59:0/3193298482 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10cbfff640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10cc0cada0 0x7f10cc0cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d08da640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10bc005210 0x7f10cc0cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d10db640 1 -- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10cc0cada0 msgr2=0x7f10cc0cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d10db640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10cc0cada0 0x7f10cc0cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d10db640 1 -- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10bc005210 msgr2=0x7f10cc0cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d10db640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10bc005210 0x7f10cc0cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.463+0000 7f10d10db640 1 -- 172.21.15.59:0/3193298482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc15c780 con 0x7f10cc15bdc0 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.464+0000 7f10d10db640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10cc15bdc0 0x7f10cc15c240 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f10c40073b0 tx=0x7f10c4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.464+0000 7f10c97fa640 1 -- 172.21.15.59:0/3193298482 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c4039040 con 0x7f10cc15bdc0 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.464+0000 7f10c97fa640 1 -- 172.21.15.59:0/3193298482 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c4045070 con 0x7f10cc15bdc0 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.464+0000 7f10d2b3e640 1 -- 172.21.15.59:0/3193298482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc0cb860 con 0x7f10cc15bdc0 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.464+0000 7f10c97fa640 1 -- 172.21.15.59:0/3193298482 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c4009c70 con 0x7f10cc15bdc0 2024-03-06T07:31:42.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.464+0000 7f10d2b3e640 1 -- 172.21.15.59:0/3193298482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc0cbe20 con 0x7f10cc15bdc0 2024-03-06T07:31:42.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.465+0000 7f10d2b3e640 1 -- 172.21.15.59:0/3193298482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10cc04f330 con 0x7f10cc15bdc0 2024-03-06T07:31:42.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.465+0000 7f10c97fa640 1 -- 172.21.15.59:0/3193298482 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10c403c070 con 0x7f10cc15bdc0 2024-03-06T07:31:42.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.465+0000 7f10c97fa640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10b006b500 0x7f10b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.466+0000 7f10c97fa640 1 -- 172.21.15.59:0/3193298482 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f10c40abea0 con 0x7f10cc15bdc0 2024-03-06T07:31:42.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.466+0000 7f10d08da640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10b006b500 0x7f10b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.466+0000 7f10d08da640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10b006b500 0x7f10b006d9c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f10c0000c00 tx=0x7f10c0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.470+0000 7f10c97fa640 1 -- 172.21.15.59:0/3193298482 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10cc15c780 con 0x7f10cc15bdc0 2024-03-06T07:31:42.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.689+0000 7f10d2b3e640 1 -- 172.21.15.59:0/3193298482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10cc15d5c0 con 0x7f10cc15bdc0 2024-03-06T07:31:42.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.690+0000 7f10c97fa640 1 -- 172.21.15.59:0/3193298482 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f10c407abe0 con 0x7f10cc15bdc0 2024-03-06T07:31:42.692 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:42.692 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:42.692 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:42.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 -- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10b006b500 msgr2=0x7f10b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10b006b500 0x7f10b006d9c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f10c0000c00 tx=0x7f10c0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:42.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 -- 172.21.15.59:0/3193298482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10cc15bdc0 msgr2=0x7f10cc15c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10cc15bdc0 0x7f10cc15c240 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f10c40073b0 tx=0x7f10c4004ab0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 -- 172.21.15.59:0/3193298482 shutdown_connections 2024-03-06T07:31:42.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10b006b500 0x7f10b006d9c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10cc15bdc0 0x7f10cc15c240 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10cc0cada0 0x7f10cc0cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 --2- 172.21.15.59:0/3193298482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10bc005210 0x7f10cc0cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 -- 172.21.15.59:0/3193298482 >> 172.21.15.59:0/3193298482 conn(0x7f10cc0d3cd0 msgr2=0x7f10cc0b4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 -- 172.21.15.59:0/3193298482 shutdown_connections 2024-03-06T07:31:42.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:42.692+0000 7f10d2b3e640 1 -- 172.21.15.59:0/3193298482 wait complete. 2024-03-06T07:31:42.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:42.906 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:43.907 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.009+0000 7f19d1781640 1 Processor -- start 2024-03-06T07:31:44.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.010+0000 7f19d1781640 1 -- start start 2024-03-06T07:31:44.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.010+0000 7f19d1781640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f19cc15dfe0 0x7f19cc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.010+0000 7f19d1781640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f19cc0cf830 0x7f19cc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.010+0000 7f19d1781640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc0b3430 con 0x7f19cc0d27f0 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.010+0000 7f19d1781640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc0b31e0 con 0x7f19cc0cf830 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.010+0000 7f19d1781640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc0b3d60 con 0x7f19cc15dfe0 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.010+0000 7f19c3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f19cc0cf830 0x7f19cc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.011+0000 7f19c3fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f19cc0cf830 0x7f19cc0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34632/0 (socket says 172.21.15.59:34632) 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.011+0000 7f19c3fff640 1 -- 172.21.15.59:0/1197974072 learned_addr learned my addr 172.21.15.59:0/1197974072 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.011+0000 7f19caffd640 1 --2- 172.21.15.59:0/1197974072 >> v2:172.21.15.164:3300/0 conn(0x7f19cc15dfe0 0x7f19cc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.011+0000 7f19c3fff640 1 -- 172.21.15.59:0/1197974072 >> v2:172.21.15.164:3300/0 conn(0x7f19cc15dfe0 msgr2=0x7f19cc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.011+0000 7f19c3fff640 1 --2- 172.21.15.59:0/1197974072 >> v2:172.21.15.164:3300/0 conn(0x7f19cc15dfe0 0x7f19cc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.011+0000 7f19c3fff640 1 -- 172.21.15.59:0/1197974072 >> v1:172.21.15.164:6789/0 conn(0x7f19cc0d27f0 legacy=0x7f19cc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.011+0000 7f19c3fff640 1 -- 172.21.15.59:0/1197974072 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc0b3710 con 0x7f19cc0cf830 2024-03-06T07:31:44.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.011+0000 7f19caffd640 1 --2- 172.21.15.59:0/1197974072 >> v2:172.21.15.164:3300/0 conn(0x7f19cc15dfe0 0x7f19cc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19c3fff640 1 --2- 172.21.15.59:0/1197974072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19cc0cf830 0x7f19cc0d22b0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009870 tx=0x7f19b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29aa7178438bbcf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19ca7fc640 1 -- 172.21.15.59:0/1197974072 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4035020 con 0x7f19cc0cf830 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19ca7fc640 1 -- 172.21.15.59:0/1197974072 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b4031870 con 0x7f19cc0cf830 2024-03-06T07:31:44.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19ca7fc640 1 -- 172.21.15.59:0/1197974072 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4031c40 con 0x7f19cc0cf830 2024-03-06T07:31:44.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19d1781640 1 -- 172.21.15.59:0/1197974072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19cc0cf830 msgr2=0x7f19cc0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19d1781640 1 --2- 172.21.15.59:0/1197974072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19cc0cf830 0x7f19cc0d22b0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009870 tx=0x7f19b402b910 comp rx=0 tx=0).stop 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19d1781640 1 -- 172.21.15.59:0/1197974072 shutdown_connections 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19d1781640 1 --2- 172.21.15.59:0/1197974072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19cc0cf830 0x7f19cc0d22b0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19d1781640 1 --2- 172.21.15.59:0/1197974072 >> v2:172.21.15.164:3300/0 conn(0x7f19cc15dfe0 0x7f19cc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19d1781640 1 -- 172.21.15.59:0/1197974072 >> 172.21.15.59:0/1197974072 conn(0x7f19cc0d3cd0 msgr2=0x7f19cc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19d1781640 1 -- 172.21.15.59:0/1197974072 shutdown_connections 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.012+0000 7f19d1781640 1 -- 172.21.15.59:0/1197974072 wait complete. 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.014+0000 7f19d1781640 1 Processor -- start 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.014+0000 7f19d1781640 1 -- start start 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.014+0000 7f19d1781640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19cc0cf830 0x7f19cc14e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.014+0000 7f19d1781640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19cc0d27f0 0x7f19cc0ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.014+0000 7f19d1781640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19cc15dfe0 0x7f19cc0cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.014+0000 7f19d1781640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0d1df0 con 0x7f19cc0d27f0 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.014+0000 7f19d1781640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0d1c70 con 0x7f19cc0cf830 2024-03-06T07:31:44.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.014+0000 7f19d1781640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc15e3c0 con 0x7f19cc15dfe0 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.014+0000 7f19c3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19cc0d27f0 0x7f19cc0ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.014+0000 7f19c3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19cc0d27f0 0x7f19cc0ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34648/0 (socket says 172.21.15.59:34648) 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.014+0000 7f19c3fff640 1 -- 172.21.15.59:0/212177209 learned_addr learned my addr 172.21.15.59:0/212177209 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19c3fff640 1 -- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19cc15dfe0 msgr2=0x7f19cc0cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19c3fff640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19cc15dfe0 0x7f19cc0cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19c3fff640 1 -- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19cc0cf830 msgr2=0x7f19cc14e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19caffd640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19cc0cf830 0x7f19cc14e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19c3fff640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19cc0cf830 0x7f19cc14e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19c3fff640 1 -- 172.21.15.59:0/212177209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc14e750 con 0x7f19cc0d27f0 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19caffd640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19cc0cf830 0x7f19cc14e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19c3fff640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19cc0d27f0 0x7f19cc0ccd10 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009870 tx=0x7f19b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19c37fe640 1 -- 172.21.15.59:0/212177209 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4040070 con 0x7f19cc0d27f0 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19c37fe640 1 -- 172.21.15.59:0/212177209 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b403a420 con 0x7f19cc0d27f0 2024-03-06T07:31:44.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19c37fe640 1 -- 172.21.15.59:0/212177209 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4035040 con 0x7f19cc0d27f0 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19d1781640 1 -- 172.21.15.59:0/212177209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19cc0cc0f0 con 0x7f19cc0d27f0 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.015+0000 7f19d1781640 1 -- 172.21.15.59:0/212177209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19cc15f360 con 0x7f19cc0d27f0 2024-03-06T07:31:44.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.016+0000 7f19d1781640 1 -- 172.21.15.59:0/212177209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19cc04f330 con 0x7f19cc0d27f0 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.017+0000 7f19c37fe640 1 -- 172.21.15.59:0/212177209 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19b404b060 con 0x7f19cc0d27f0 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.017+0000 7f19c37fe640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19ac06b4d0 0x7f19ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.017+0000 7f19c37fe640 1 -- 172.21.15.59:0/212177209 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f19b40ac090 con 0x7f19cc0d27f0 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.018+0000 7f19caffd640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19ac06b4d0 0x7f19ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:44.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.018+0000 7f19caffd640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19ac06b4d0 0x7f19ac06d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f19bc006fd0 tx=0x7f19bc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:44.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.022+0000 7f19c37fe640 1 -- 172.21.15.59:0/212177209 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19b407af80 con 0x7f19cc0d27f0 2024-03-06T07:31:44.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.241+0000 7f19d1781640 1 -- 172.21.15.59:0/212177209 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19cc0b09c0 con 0x7f19cc0d27f0 2024-03-06T07:31:44.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.242+0000 7f19c37fe640 1 -- 172.21.15.59:0/212177209 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f19b407a880 con 0x7f19cc0d27f0 2024-03-06T07:31:44.244 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:44.244 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:44.244 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.243+0000 7f19d1781640 1 -- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19ac06b4d0 msgr2=0x7f19ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.244+0000 7f19d1781640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19ac06b4d0 0x7f19ac06d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f19bc006fd0 tx=0x7f19bc008040 comp rx=0 tx=0).stop 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.244+0000 7f19d1781640 1 -- 172.21.15.59:0/212177209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19cc0d27f0 msgr2=0x7f19cc0ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.244+0000 7f19d1781640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19cc0d27f0 0x7f19cc0ccd10 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009870 tx=0x7f19b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.244+0000 7f19d1781640 1 -- 172.21.15.59:0/212177209 shutdown_connections 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.244+0000 7f19d1781640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f19ac06b4d0 0x7f19ac06d990 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.244+0000 7f19d1781640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19cc15dfe0 0x7f19cc0cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.244+0000 7f19d1781640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19cc0d27f0 0x7f19cc0ccd10 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.244+0000 7f19d1781640 1 --2- 172.21.15.59:0/212177209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19cc0cf830 0x7f19cc14e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.244+0000 7f19d1781640 1 -- 172.21.15.59:0/212177209 >> 172.21.15.59:0/212177209 conn(0x7f19cc0d3cd0 msgr2=0x7f19cc06c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.244+0000 7f19d1781640 1 -- 172.21.15.59:0/212177209 shutdown_connections 2024-03-06T07:31:44.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:44.244+0000 7f19d1781640 1 -- 172.21.15.59:0/212177209 wait complete. 2024-03-06T07:31:44.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:44.458 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:45.458 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:45.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.559+0000 7f5fe628d640 1 Processor -- start 2024-03-06T07:31:45.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.559+0000 7f5fe628d640 1 -- start start 2024-03-06T07:31:45.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fe628d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fe015dfe0 0x7f5fe00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fe628d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5fe00cf830 0x7f5fe00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fe628d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe00b3430 con 0x7f5fe00d27f0 2024-03-06T07:31:45.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fe628d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe00d2c50 con 0x7f5fe00cf830 2024-03-06T07:31:45.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fe628d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe00b31e0 con 0x7f5fe015dfe0 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fe482a640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f5fe00d27f0 0x7f5fe00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55160/0 (socket says 172.21.15.59:55160) 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fe482a640 1 -- 172.21.15.59:0/1795519712 learned_addr learned my addr 172.21.15.59:0/1795519712 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fdffff640 1 --2- 172.21.15.59:0/1795519712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fe015dfe0 0x7f5fe00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fdf7fe640 1 --2- 172.21.15.59:0/1795519712 >> v2:172.21.15.164:3300/0 conn(0x7f5fe00cf830 0x7f5fe00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fdeffd640 1 -- 172.21.15.59:0/1795519712 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532891695 0 0) 0x7f5fe00b3430 con 0x7f5fe00d27f0 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fdeffd640 1 -- 172.21.15.59:0/1795519712 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc8003520 con 0x7f5fe00d27f0 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fdeffd640 1 -- 172.21.15.59:0/1795519712 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2558028072 0 0) 0x7f5fc8003520 con 0x7f5fe00d27f0 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fdf7fe640 1 -- 172.21.15.59:0/1795519712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fe015dfe0 msgr2=0x7f5fe00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fdf7fe640 1 --2- 172.21.15.59:0/1795519712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fe015dfe0 0x7f5fe00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fdf7fe640 1 -- 172.21.15.59:0/1795519712 >> v1:172.21.15.59:6789/0 conn(0x7f5fe00d27f0 legacy=0x7f5fe00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fdf7fe640 1 -- 172.21.15.59:0/1795519712 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe00b3710 con 0x7f5fe00cf830 2024-03-06T07:31:45.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.560+0000 7f5fdffff640 1 --2- 172.21.15.59:0/1795519712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fe015dfe0 0x7f5fe00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:45.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.561+0000 7f5fdf7fe640 1 --2- 172.21.15.59:0/1795519712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fe00cf830 0x7f5fe00d22b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40097a0 tx=0x7f5fd402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c814b7e8703459c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.561+0000 7f5fdeffd640 1 -- 172.21.15.59:0/1795519712 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4037070 con 0x7f5fe00cf830 2024-03-06T07:31:45.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.562+0000 7f5fdeffd640 1 -- 172.21.15.59:0/1795519712 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd4030510 con 0x7f5fe00cf830 2024-03-06T07:31:45.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.562+0000 7f5fdeffd640 1 -- 172.21.15.59:0/1795519712 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80019f0 con 0x7f5fe00cf830 2024-03-06T07:31:45.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.562+0000 7f5fe628d640 1 -- 172.21.15.59:0/1795519712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fe00cf830 msgr2=0x7f5fe00d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.562+0000 7f5fe628d640 1 --2- 172.21.15.59:0/1795519712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fe00cf830 0x7f5fe00d22b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40097a0 tx=0x7f5fd402b790 comp rx=0 tx=0).stop 2024-03-06T07:31:45.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.562+0000 7f5fe628d640 1 -- 172.21.15.59:0/1795519712 shutdown_connections 2024-03-06T07:31:45.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.562+0000 7f5fe628d640 1 --2- 172.21.15.59:0/1795519712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fe00cf830 0x7f5fe00d22b0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.562+0000 7f5fe628d640 1 --2- 172.21.15.59:0/1795519712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fe015dfe0 0x7f5fe00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.562+0000 7f5fe628d640 1 -- 172.21.15.59:0/1795519712 >> 172.21.15.59:0/1795519712 conn(0x7f5fe00d3cd0 msgr2=0x7f5fe00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.562+0000 7f5fe628d640 1 -- 172.21.15.59:0/1795519712 shutdown_connections 2024-03-06T07:31:45.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.562+0000 7f5fe628d640 1 -- 172.21.15.59:0/1795519712 wait complete. 2024-03-06T07:31:45.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.564+0000 7f5fe628d640 1 Processor -- start 2024-03-06T07:31:45.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.564+0000 7f5fe628d640 1 -- start start 2024-03-06T07:31:45.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.564+0000 7f5fe628d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fe00cf830 0x7f5fe015bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.564+0000 7f5fe628d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fe00d27f0 0x7f5fe009dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.564+0000 7f5fe628d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fe015dfe0 0x7f5fe00ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.564+0000 7f5fe628d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe00d1c70 con 0x7f5fe00cf830 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.564+0000 7f5fe628d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe00b3d60 con 0x7f5fe00d27f0 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fe628d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe00d1df0 con 0x7f5fe015dfe0 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fe00cf830 0x7f5fe015bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fe00cf830 0x7f5fe015bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34650/0 (socket says 172.21.15.59:34650) 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdffff640 1 -- 172.21.15.59:0/2633839963 learned_addr learned my addr 172.21.15.59:0/2633839963 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdf7fe640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fe00d27f0 0x7f5fe009dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fe482a640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fe015dfe0 0x7f5fe00ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdffff640 1 -- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fe015dfe0 msgr2=0x7f5fe00ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdffff640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fe015dfe0 0x7f5fe00ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdffff640 1 -- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fe00d27f0 msgr2=0x7f5fe009dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdffff640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fe00d27f0 0x7f5fe009dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdffff640 1 -- 172.21.15.59:0/2633839963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe0160110 con 0x7f5fe00cf830 2024-03-06T07:31:45.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdf7fe640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fe00d27f0 0x7f5fe009dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:45.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdffff640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fe00cf830 0x7f5fe015bda0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f5fd000e990 tx=0x7f5fd000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdcff9640 1 -- 172.21.15.59:0/2633839963 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0017070 con 0x7f5fe00cf830 2024-03-06T07:31:45.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdcff9640 1 -- 172.21.15.59:0/2633839963 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd000c9e0 con 0x7f5fe00cf830 2024-03-06T07:31:45.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fe628d640 1 -- 172.21.15.59:0/2633839963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe00b7590 con 0x7f5fe00cf830 2024-03-06T07:31:45.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fdcff9640 1 -- 172.21.15.59:0/2633839963 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd000cdb0 con 0x7f5fe00cf830 2024-03-06T07:31:45.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.565+0000 7f5fe628d640 1 -- 172.21.15.59:0/2633839963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe01603b0 con 0x7f5fe00cf830 2024-03-06T07:31:45.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.567+0000 7f5fdcff9640 1 -- 172.21.15.59:0/2633839963 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fd001a070 con 0x7f5fe00cf830 2024-03-06T07:31:45.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.567+0000 7f5fe628d640 1 -- 172.21.15.59:0/2633839963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe004f330 con 0x7f5fe00cf830 2024-03-06T07:31:45.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.567+0000 7f5fdcff9640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fcc06b4d0 0x7f5fcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.567+0000 7f5fdcff9640 1 -- 172.21.15.59:0/2633839963 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5fd008a050 con 0x7f5fe00cf830 2024-03-06T07:31:45.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.567+0000 7f5fdf7fe640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fcc06b4d0 0x7f5fcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.568+0000 7f5fdf7fe640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fcc06b4d0 0x7f5fcc06d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4034000 tx=0x7f5fd402f000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.571+0000 7f5fdcff9640 1 -- 172.21.15.59:0/2633839963 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fd0058f40 con 0x7f5fe00cf830 2024-03-06T07:31:45.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.792+0000 7f5fe628d640 1 -- 172.21.15.59:0/2633839963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fe00cea80 con 0x7f5fe00cf830 2024-03-06T07:31:45.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.792+0000 7f5fdcff9640 1 -- 172.21.15.59:0/2633839963 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f5fd0058840 con 0x7f5fe00cf830 2024-03-06T07:31:45.794 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:45.794 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:45.794 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 -- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fcc06b4d0 msgr2=0x7f5fcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fcc06b4d0 0x7f5fcc06d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4034000 tx=0x7f5fd402f000 comp rx=0 tx=0).stop 2024-03-06T07:31:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 -- 172.21.15.59:0/2633839963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fe00cf830 msgr2=0x7f5fe015bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fe00cf830 0x7f5fe015bda0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f5fd000e990 tx=0x7f5fd000ee50 comp rx=0 tx=0).stop 2024-03-06T07:31:45.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 -- 172.21.15.59:0/2633839963 shutdown_connections 2024-03-06T07:31:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5fcc06b4d0 0x7f5fcc06d990 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fe015dfe0 0x7f5fe00ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fe00d27f0 0x7f5fe009dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 --2- 172.21.15.59:0/2633839963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5fe00cf830 0x7f5fe015bda0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 -- 172.21.15.59:0/2633839963 >> 172.21.15.59:0/2633839963 conn(0x7f5fe00d3cd0 msgr2=0x7f5fe00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 -- 172.21.15.59:0/2633839963 shutdown_connections 2024-03-06T07:31:45.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:45.794+0000 7f5fe628d640 1 -- 172.21.15.59:0/2633839963 wait complete. 2024-03-06T07:31:46.008 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:46.008 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:47.009 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:47.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.109+0000 7f9c19e76640 1 Processor -- start 2024-03-06T07:31:47.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.110+0000 7f9c19e76640 1 -- start start 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c19e76640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9c140d22b0 0x7f9c140d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c19e76640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c140b41f0 0x7f9c140b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c19e76640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140b3430 con 0x7f9c1415dfe0 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c19e76640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c140d2c50 con 0x7f9c140d22b0 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c19e76640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140b31e0 con 0x7f9c140b41f0 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c137fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9c1415dfe0 0x7f9c140cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55162/0 (socket says 172.21.15.59:55162) 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c137fe640 1 -- 172.21.15.59:0/800647320 learned_addr learned my addr 172.21.15.59:0/800647320 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c12ffd640 1 --2- 172.21.15.59:0/800647320 >> v2:172.21.15.164:3300/0 conn(0x7f9c140d22b0 0x7f9c140d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c13fff640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c140b41f0 0x7f9c140b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182327384 0 0) 0x7f9c140b3430 con 0x7f9c1415dfe0 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc003520 con 0x7f9c1415dfe0 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1687916060 0 0) 0x7f9bfc003520 con 0x7f9c1415dfe0 2024-03-06T07:31:47.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c140b41f0 msgr2=0x7f9c140b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.111+0000 7f9c127fc640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c140b41f0 0x7f9c140b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 >> v2:172.21.15.164:3300/0 conn(0x7f9c140d22b0 msgr2=0x7f9c140d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c127fc640 1 --2- 172.21.15.59:0/800647320 >> v2:172.21.15.164:3300/0 conn(0x7f9c140d22b0 0x7f9c140d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140b3710 con 0x7f9c1415dfe0 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f9c040019f0 con 0x7f9c1415dfe0 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c12ffd640 1 --2- 172.21.15.59:0/800647320 >> v2:172.21.15.164:3300/0 conn(0x7f9c140d22b0 0x7f9c140d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 >> v1:172.21.15.59:6789/0 conn(0x7f9c1415dfe0 legacy=0x7f9c140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c127fc640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bfc0039b0 0x7f9bfc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c127fc640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bfc004800 0x7f9bfc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c127fc640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c140b41f0 0x7f9bfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140b31e0 con 0x7f9bfc004800 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc00b9c0 con 0x7f9bfc0039b0 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc00bb40 con 0x7f9c140b41f0 2024-03-06T07:31:47.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c12ffd640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bfc004800 0x7f9bfc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c137fe640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bfc0039b0 0x7f9bfc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c13fff640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c140b41f0 0x7f9bfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c12ffd640 1 -- 172.21.15.59:0/800647320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c140b41f0 msgr2=0x7f9bfc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c12ffd640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c140b41f0 0x7f9bfc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c12ffd640 1 -- 172.21.15.59:0/800647320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bfc0039b0 msgr2=0x7f9bfc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c12ffd640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bfc0039b0 0x7f9bfc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.112+0000 7f9c12ffd640 1 -- 172.21.15.59:0/800647320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc003520 con 0x7f9bfc004800 2024-03-06T07:31:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.113+0000 7f9c12ffd640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bfc004800 0x7f9bfc004c80 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009c90 tx=0x7f9bfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a725e658b5e8ccd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.113+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08039070 con 0x7f9bfc004800 2024-03-06T07:31:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.113+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0802c980 con 0x7f9bfc004800 2024-03-06T07:31:47.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.113+0000 7f9c127fc640 1 -- 172.21.15.59:0/800647320 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08009860 con 0x7f9bfc004800 2024-03-06T07:31:47.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.113+0000 7f9c19e76640 1 -- 172.21.15.59:0/800647320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bfc004800 msgr2=0x7f9bfc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.113+0000 7f9c19e76640 1 --2- 172.21.15.59:0/800647320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9bfc004800 0x7f9bfc004c80 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009c90 tx=0x7f9bfc005830 comp rx=0 tx=0).stop 2024-03-06T07:31:47.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.113+0000 7f9c13fff640 1 -- 172.21.15.59:0/800647320 reap_dead start 2024-03-06T07:31:47.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.114+0000 7f9c19e76640 1 -- 172.21.15.59:0/800647320 shutdown_connections 2024-03-06T07:31:47.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.114+0000 7f9c19e76640 1 -- 172.21.15.59:0/800647320 >> 172.21.15.59:0/800647320 conn(0x7f9c140d3cd0 msgr2=0x7f9c140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.114+0000 7f9c19e76640 1 -- 172.21.15.59:0/800647320 shutdown_connections 2024-03-06T07:31:47.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.114+0000 7f9c19e76640 1 -- 172.21.15.59:0/800647320 wait complete. 2024-03-06T07:31:47.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.115+0000 7f9c19e76640 1 Processor -- start 2024-03-06T07:31:47.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.116+0000 7f9c19e76640 1 -- start start 2024-03-06T07:31:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c19e76640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c140cc890 0x7f9c140cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c19e76640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c14160640 0x7f9c141f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c19e76640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c140ccc70 0x7f9c141f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c19e76640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140cfa00 con 0x7f9c140ccc70 2024-03-06T07:31:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c19e76640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140b3d60 con 0x7f9c140cc890 2024-03-06T07:31:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c19e76640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c140d1c70 con 0x7f9c14160640 2024-03-06T07:31:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c13fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c140ccc70 0x7f9c141f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c13fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c140ccc70 0x7f9c141f3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34664/0 (socket says 172.21.15.59:34664) 2024-03-06T07:31:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c13fff640 1 -- 172.21.15.59:0/3515724192 learned_addr learned my addr 172.21.15.59:0/3515724192 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c137fe640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c140cc890 0x7f9c140cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c12ffd640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c14160640 0x7f9c141f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c13fff640 1 -- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c14160640 msgr2=0x7f9c141f2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c13fff640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c14160640 0x7f9c141f2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c13fff640 1 -- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c140cc890 msgr2=0x7f9c140cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c13fff640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c140cc890 0x7f9c140cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c13fff640 1 -- 172.21.15.59:0/3515724192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c141f36a0 con 0x7f9c140ccc70 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.117+0000 7f9c13fff640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c140ccc70 0x7f9c141f3160 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f9c14161bf0 tx=0x7f9c0c011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.118+0000 7f9bfbfff640 1 -- 172.21.15.59:0/3515724192 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c004240 con 0x7f9c140ccc70 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.118+0000 7f9bfbfff640 1 -- 172.21.15.59:0/3515724192 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0c01b070 con 0x7f9c140ccc70 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.118+0000 7f9c19e76640 1 -- 172.21.15.59:0/3515724192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c140cb6f0 con 0x7f9c140ccc70 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.118+0000 7f9bfbfff640 1 -- 172.21.15.59:0/3515724192 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c00f7a0 con 0x7f9c140ccc70 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.118+0000 7f9c19e76640 1 -- 172.21.15.59:0/3515724192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c140cbcb0 con 0x7f9c140ccc70 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.119+0000 7f9c19e76640 1 -- 172.21.15.59:0/3515724192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1404f330 con 0x7f9c140ccc70 2024-03-06T07:31:47.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.119+0000 7f9bfbfff640 1 -- 172.21.15.59:0/3515724192 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c0c012040 con 0x7f9c140ccc70 2024-03-06T07:31:47.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.120+0000 7f9bfbfff640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c0006b4d0 0x7f9c0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:47.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.120+0000 7f9bfbfff640 1 -- 172.21.15.59:0/3515724192 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9c0c089f10 con 0x7f9c140ccc70 2024-03-06T07:31:47.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.120+0000 7f9c137fe640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c0006b4d0 0x7f9c0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:47.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.120+0000 7f9c137fe640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c0006b4d0 0x7f9c0006d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f9c04001560 tx=0x7f9c0400a0e0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:47.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.125+0000 7f9bfbfff640 1 -- 172.21.15.59:0/3515724192 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c0c058d70 con 0x7f9c140ccc70 2024-03-06T07:31:47.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.343+0000 7f9c19e76640 1 -- 172.21.15.59:0/3515724192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c1406ba90 con 0x7f9c140ccc70 2024-03-06T07:31:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.344+0000 7f9bfbfff640 1 -- 172.21.15.59:0/3515724192 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f9c0c058670 con 0x7f9c140ccc70 2024-03-06T07:31:47.346 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:47.346 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:47.346 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 -- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c0006b4d0 msgr2=0x7f9c0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c0006b4d0 0x7f9c0006d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f9c04001560 tx=0x7f9c0400a0e0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 -- 172.21.15.59:0/3515724192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c140ccc70 msgr2=0x7f9c141f3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c140ccc70 0x7f9c141f3160 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f9c14161bf0 tx=0x7f9c0c011cf0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 -- 172.21.15.59:0/3515724192 shutdown_connections 2024-03-06T07:31:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9c0006b4d0 0x7f9c0006d990 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9c140ccc70 0x7f9c141f3160 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c14160640 0x7f9c141f2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 --2- 172.21.15.59:0/3515724192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c140cc890 0x7f9c140cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 -- 172.21.15.59:0/3515724192 >> 172.21.15.59:0/3515724192 conn(0x7f9c140d3cd0 msgr2=0x7f9c1406cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 -- 172.21.15.59:0/3515724192 shutdown_connections 2024-03-06T07:31:47.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:47.346+0000 7f9c19e76640 1 -- 172.21.15.59:0/3515724192 wait complete. 2024-03-06T07:31:47.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:47.560 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:48.561 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:48.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.660+0000 7fbac50d4640 1 Processor -- start 2024-03-06T07:31:48.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.661+0000 7fbac50d4640 1 -- start start 2024-03-06T07:31:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.661+0000 7fbac50d4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbac015dfe0 0x7fbac00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.661+0000 7fbac50d4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac00cf830 0x7fbac00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.661+0000 7fbac50d4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac00b3430 con 0x7fbac00d27f0 2024-03-06T07:31:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.661+0000 7fbac50d4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac00d2c50 con 0x7fbac015dfe0 2024-03-06T07:31:48.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.661+0000 7fbac50d4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00b31e0 con 0x7fbac00cf830 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.661+0000 7fbabf59e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbac00d27f0 0x7fbac00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55170/0 (socket says 172.21.15.59:55170) 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.661+0000 7fbabf59e640 1 -- 172.21.15.59:0/4147819679 learned_addr learned my addr 172.21.15.59:0/4147819679 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabe59c640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac00cf830 0x7fbac00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabed9d640 1 --2- 172.21.15.59:0/4147819679 >> v2:172.21.15.164:3300/0 conn(0x7fbac015dfe0 0x7fbac00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233636536 0 0) 0x7fbac00b3430 con 0x7fbac00d27f0 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaac003520 con 0x7fbac00d27f0 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3742641172 0 0) 0x7fbaac003520 con 0x7fbac00d27f0 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac00cf830 msgr2=0x7fbac00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac00cf830 0x7fbac00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 >> v2:172.21.15.164:3300/0 conn(0x7fbac015dfe0 msgr2=0x7fbac00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 --2- 172.21.15.59:0/4147819679 >> v2:172.21.15.164:3300/0 conn(0x7fbac015dfe0 0x7fbac00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac00b3710 con 0x7fbac00d27f0 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fbab00019f0 con 0x7fbac00d27f0 2024-03-06T07:31:48.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 >> v1:172.21.15.59:6789/0 conn(0x7fbac00d27f0 legacy=0x7fbac00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbaac0039b0 0x7fbaac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac00cf830 0x7fbaac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbaac004d40 0x7fbaac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00b31e0 con 0x7fbaac0039b0 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac00b9c0 con 0x7fbaac004d40 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac00bb40 con 0x7fbac00cf830 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabed9d640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbaac0039b0 0x7fbaac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabf59e640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbaac004d40 0x7fbaac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabe59c640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac00cf830 0x7fbaac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabed9d640 1 -- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac00cf830 msgr2=0x7fbaac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.662+0000 7fbabed9d640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac00cf830 0x7fbaac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.663+0000 7fbabed9d640 1 -- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbaac004d40 msgr2=0x7fbaac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.663+0000 7fbabed9d640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbaac004d40 0x7fbaac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.663+0000 7fbabed9d640 1 -- 172.21.15.59:0/4147819679 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac003520 con 0x7fbaac0039b0 2024-03-06T07:31:48.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.663+0000 7fbabf59e640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbaac004d40 0x7fbaac0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:48.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.663+0000 7fbabed9d640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbaac0039b0 0x7fbaac003dd0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009aa0 tx=0x7fbaac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4df0faf0db7b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.663+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8039070 con 0x7fbaac0039b0 2024-03-06T07:31:48.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.663+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab802c710 con 0x7fbaac0039b0 2024-03-06T07:31:48.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.663+0000 7fbabdd9b640 1 -- 172.21.15.59:0/4147819679 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8009670 con 0x7fbaac0039b0 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.664+0000 7fbac50d4640 1 -- 172.21.15.59:0/4147819679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbaac0039b0 msgr2=0x7fbaac003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.664+0000 7fbac50d4640 1 --2- 172.21.15.59:0/4147819679 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbaac0039b0 0x7fbaac003dd0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009aa0 tx=0x7fbaac005830 comp rx=0 tx=0).stop 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.664+0000 7fbabf59e640 1 -- 172.21.15.59:0/4147819679 reap_dead start 2024-03-06T07:31:48.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.664+0000 7fbac50d4640 1 -- 172.21.15.59:0/4147819679 shutdown_connections 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.664+0000 7fbac50d4640 1 -- 172.21.15.59:0/4147819679 >> 172.21.15.59:0/4147819679 conn(0x7fbac00d3cd0 msgr2=0x7fbac00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.664+0000 7fbac50d4640 1 -- 172.21.15.59:0/4147819679 shutdown_connections 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.664+0000 7fbac50d4640 1 -- 172.21.15.59:0/4147819679 wait complete. 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.666+0000 7fbac50d4640 1 Processor -- start 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.666+0000 7fbac50d4640 1 -- start start 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.666+0000 7fbac50d4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbaac0039b0 0x7fbac015fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.666+0000 7fbac50d4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac014f1e0 0x7fbac00cc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.666+0000 7fbac50d4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac015c070 0x7fbac015c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.666+0000 7fbac50d4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00d1c70 con 0x7fbac015c070 2024-03-06T07:31:48.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.666+0000 7fbac50d4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00b3d60 con 0x7fbaac0039b0 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.666+0000 7fbac50d4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00d1df0 con 0x7fbac014f1e0 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.666+0000 7fbabf59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac015c070 0x7fbac015c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.666+0000 7fbabf59e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac015c070 0x7fbac015c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34688/0 (socket says 172.21.15.59:34688) 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.666+0000 7fbabf59e640 1 -- 172.21.15.59:0/1960334139 learned_addr learned my addr 172.21.15.59:0/1960334139 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbabed9d640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbaac0039b0 0x7fbac015fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbabe59c640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac014f1e0 0x7fbac00cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbabf59e640 1 -- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac014f1e0 msgr2=0x7fbac00cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbabf59e640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac014f1e0 0x7fbac00cc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbabf59e640 1 -- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbaac0039b0 msgr2=0x7fbac015fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbabf59e640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbaac0039b0 0x7fbac015fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbabf59e640 1 -- 172.21.15.59:0/1960334139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac014f640 con 0x7fbac015c070 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbabed9d640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbaac0039b0 0x7fbac015fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:48.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbabf59e640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac015c070 0x7fbac015c4f0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fbab0002230 tx=0x7fbab000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbab77fe640 1 -- 172.21.15.59:0/1960334139 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000eb90 con 0x7fbac015c070 2024-03-06T07:31:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbab77fe640 1 -- 172.21.15.59:0/1960334139 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab000ed30 con 0x7fbac015c070 2024-03-06T07:31:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbac50d4640 1 -- 172.21.15.59:0/1960334139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac00cced0 con 0x7fbac015c070 2024-03-06T07:31:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbab77fe640 1 -- 172.21.15.59:0/1960334139 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0014670 con 0x7fbac015c070 2024-03-06T07:31:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.667+0000 7fbac50d4640 1 -- 172.21.15.59:0/1960334139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac00cd490 con 0x7fbac015c070 2024-03-06T07:31:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.668+0000 7fbac50d4640 1 -- 172.21.15.59:0/1960334139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac015ca30 con 0x7fbac015c070 2024-03-06T07:31:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.669+0000 7fbab77fe640 1 -- 172.21.15.59:0/1960334139 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbab0014810 con 0x7fbac015c070 2024-03-06T07:31:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.669+0000 7fbab77fe640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbaa006b500 0x7fbaa006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.669+0000 7fbab77fe640 1 -- 172.21.15.59:0/1960334139 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbab008a000 con 0x7fbac015c070 2024-03-06T07:31:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.669+0000 7fbabed9d640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbaa006b500 0x7fbaa006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.670+0000 7fbabed9d640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbaa006b500 0x7fbaa006d9c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbab8036000 tx=0x7fbab8030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.673+0000 7fbab77fe640 1 -- 172.21.15.59:0/1960334139 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbab0058e60 con 0x7fbac015c070 2024-03-06T07:31:48.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.891+0000 7fbac50d4640 1 -- 172.21.15.59:0/1960334139 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac00cf5f0 con 0x7fbac015c070 2024-03-06T07:31:48.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.892+0000 7fbab77fe640 1 -- 172.21.15.59:0/1960334139 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fbac00cf5f0 con 0x7fbac015c070 2024-03-06T07:31:48.894 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:48.894 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:48.894 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:48.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 -- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbaa006b500 msgr2=0x7fbaa006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbaa006b500 0x7fbaa006d9c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbab8036000 tx=0x7fbab8030000 comp rx=0 tx=0).stop 2024-03-06T07:31:48.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 -- 172.21.15.59:0/1960334139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac015c070 msgr2=0x7fbac015c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac015c070 0x7fbac015c4f0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fbab0002230 tx=0x7fbab000e530 comp rx=0 tx=0).stop 2024-03-06T07:31:48.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 -- 172.21.15.59:0/1960334139 shutdown_connections 2024-03-06T07:31:48.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbaa006b500 0x7fbaa006d9c0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac015c070 0x7fbac015c4f0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbac014f1e0 0x7fbac00cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 --2- 172.21.15.59:0/1960334139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbaac0039b0 0x7fbac015fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 -- 172.21.15.59:0/1960334139 >> 172.21.15.59:0/1960334139 conn(0x7fbac00d3cd0 msgr2=0x7fbac00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 -- 172.21.15.59:0/1960334139 shutdown_connections 2024-03-06T07:31:48.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:48.894+0000 7fbac50d4640 1 -- 172.21.15.59:0/1960334139 wait complete. 2024-03-06T07:31:49.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:49.109 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:50.110 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:50.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.213+0000 7f1b1879a640 1 Processor -- start 2024-03-06T07:31:50.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.213+0000 7f1b1879a640 1 -- start start 2024-03-06T07:31:50.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b1879a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1b100d22b0 0x7f1b100d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b1879a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1b100b41f0 0x7f1b100b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b1879a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b100b3430 con 0x7f1b1015dfe0 2024-03-06T07:31:50.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b1879a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b100d2c50 con 0x7f1b100b41f0 2024-03-06T07:31:50.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b1879a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b100b31e0 con 0x7f1b100d22b0 2024-03-06T07:31:50.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b16d37640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1b100b41f0 0x7f1b100b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b16536640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1b1015dfe0 0x7f1b100cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55174/0 (socket says 172.21.15.59:55174) 2024-03-06T07:31:50.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b16536640 1 -- 172.21.15.59:0/962279453 learned_addr learned my addr 172.21.15.59:0/962279453 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:50.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b15d35640 1 --2- 172.21.15.59:0/962279453 >> v2:172.21.15.164:3300/0 conn(0x7f1b100d22b0 0x7f1b100d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b16d37640 1 -- 172.21.15.59:0/962279453 >> v2:172.21.15.164:3300/0 conn(0x7f1b100d22b0 msgr2=0x7f1b100d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b16d37640 1 --2- 172.21.15.59:0/962279453 >> v2:172.21.15.164:3300/0 conn(0x7f1b100d22b0 0x7f1b100d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b16d37640 1 -- 172.21.15.59:0/962279453 >> v1:172.21.15.59:6789/0 conn(0x7f1b1015dfe0 legacy=0x7f1b100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b15534640 1 -- 172.21.15.59:0/962279453 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167421891 0 0) 0x7f1b100b3430 con 0x7f1b1015dfe0 2024-03-06T07:31:50.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.214+0000 7f1b16d37640 1 -- 172.21.15.59:0/962279453 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b100b3710 con 0x7f1b100b41f0 2024-03-06T07:31:50.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.215+0000 7f1b16d37640 1 --2- 172.21.15.59:0/962279453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b100b41f0 0x7f1b100b4680 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009d90 tx=0x7f1b0002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57306b3dfae119f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.215+0000 7f1b15534640 1 -- 172.21.15.59:0/962279453 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00039070 con 0x7f1b100b41f0 2024-03-06T07:31:50.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.215+0000 7f1b15534640 1 -- 172.21.15.59:0/962279453 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0002cc60 con 0x7f1b100b41f0 2024-03-06T07:31:50.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.215+0000 7f1b15534640 1 -- 172.21.15.59:0/962279453 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00034660 con 0x7f1b100b41f0 2024-03-06T07:31:50.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.216+0000 7f1b1879a640 1 -- 172.21.15.59:0/962279453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b100b41f0 msgr2=0x7f1b100b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.216+0000 7f1b1879a640 1 --2- 172.21.15.59:0/962279453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b100b41f0 0x7f1b100b4680 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009d90 tx=0x7f1b0002beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.216+0000 7f1b1879a640 1 -- 172.21.15.59:0/962279453 shutdown_connections 2024-03-06T07:31:50.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.216+0000 7f1b1879a640 1 --2- 172.21.15.59:0/962279453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b100b41f0 0x7f1b100b4680 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.216+0000 7f1b1879a640 1 --2- 172.21.15.59:0/962279453 >> v2:172.21.15.164:3300/0 conn(0x7f1b100d22b0 0x7f1b100d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.216+0000 7f1b1879a640 1 -- 172.21.15.59:0/962279453 >> 172.21.15.59:0/962279453 conn(0x7f1b100d3cd0 msgr2=0x7f1b100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.216+0000 7f1b1879a640 1 -- 172.21.15.59:0/962279453 shutdown_connections 2024-03-06T07:31:50.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.216+0000 7f1b1879a640 1 -- 172.21.15.59:0/962279453 wait complete. 2024-03-06T07:31:50.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.217+0000 7f1b1879a640 1 Processor -- start 2024-03-06T07:31:50.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.217+0000 7f1b1879a640 1 -- start start 2024-03-06T07:31:50.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b1879a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b100b41f0 0x7f1b1015b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b1879a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b100d22b0 0x7f1b100cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b1879a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b1015dfe0 0x7f1b100cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b1879a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b100cfa00 con 0x7f1b1015dfe0 2024-03-06T07:31:50.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b1879a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b100b3d60 con 0x7f1b100d22b0 2024-03-06T07:31:50.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b1879a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b100d1c70 con 0x7f1b100b41f0 2024-03-06T07:31:50.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b16d37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b1015dfe0 0x7f1b100cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b16d37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b1015dfe0 0x7f1b100cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34700/0 (socket says 172.21.15.59:34700) 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b16d37640 1 -- 172.21.15.59:0/3418994548 learned_addr learned my addr 172.21.15.59:0/3418994548 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b16536640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b100b41f0 0x7f1b1015b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b15d35640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b100d22b0 0x7f1b100cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b16d37640 1 -- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b100b41f0 msgr2=0x7f1b1015b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b16d37640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b100b41f0 0x7f1b1015b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b16d37640 1 -- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b100d22b0 msgr2=0x7f1b100cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b16d37640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b100d22b0 0x7f1b100cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.218+0000 7f1b16d37640 1 -- 172.21.15.59:0/3418994548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b100cd9c0 con 0x7f1b1015dfe0 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.219+0000 7f1b16d37640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b1015dfe0 0x7f1b100cada0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009eb0 tx=0x7f1b00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.219+0000 7f1b0effd640 1 -- 172.21.15.59:0/3418994548 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00039050 con 0x7f1b1015dfe0 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.219+0000 7f1b0effd640 1 -- 172.21.15.59:0/3418994548 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b00045070 con 0x7f1b1015dfe0 2024-03-06T07:31:50.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.219+0000 7f1b1879a640 1 -- 172.21.15.59:0/3418994548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b100cd150 con 0x7f1b1015dfe0 2024-03-06T07:31:50.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.219+0000 7f1b0effd640 1 -- 172.21.15.59:0/3418994548 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0003e830 con 0x7f1b1015dfe0 2024-03-06T07:31:50.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.219+0000 7f1b1879a640 1 -- 172.21.15.59:0/3418994548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b100cce70 con 0x7f1b1015dfe0 2024-03-06T07:31:50.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.220+0000 7f1b1879a640 1 -- 172.21.15.59:0/3418994548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1004f330 con 0x7f1b1015dfe0 2024-03-06T07:31:50.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.220+0000 7f1b0effd640 1 -- 172.21.15.59:0/3418994548 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b0003e9d0 con 0x7f1b1015dfe0 2024-03-06T07:31:50.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.221+0000 7f1b0effd640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1afc06b4d0 0x7f1afc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.221+0000 7f1b0effd640 1 -- 172.21.15.59:0/3418994548 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1b000ace90 con 0x7f1b1015dfe0 2024-03-06T07:31:50.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.221+0000 7f1b16536640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1afc06b4d0 0x7f1afc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.222+0000 7f1b16536640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1afc06b4d0 0x7f1afc06d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1b08007fd0 tx=0x7f1b08007de0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.228+0000 7f1b0effd640 1 -- 172.21.15.59:0/3418994548 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b100cd9c0 con 0x7f1b1015dfe0 2024-03-06T07:31:50.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.448+0000 7f1b1879a640 1 -- 172.21.15.59:0/3418994548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b100d0c20 con 0x7f1b1015dfe0 2024-03-06T07:31:50.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.449+0000 7f1b0effd640 1 -- 172.21.15.59:0/3418994548 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f1b0007b850 con 0x7f1b1015dfe0 2024-03-06T07:31:50.451 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:50.451 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:50.451 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:50.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 -- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1afc06b4d0 msgr2=0x7f1afc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1afc06b4d0 0x7f1afc06d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1b08007fd0 tx=0x7f1b08007de0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 -- 172.21.15.59:0/3418994548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b1015dfe0 msgr2=0x7f1b100cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b1015dfe0 0x7f1b100cada0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009eb0 tx=0x7f1b00004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:50.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 -- 172.21.15.59:0/3418994548 shutdown_connections 2024-03-06T07:31:50.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1afc06b4d0 0x7f1afc06d990 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b1015dfe0 0x7f1b100cada0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b100d22b0 0x7f1b100cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 --2- 172.21.15.59:0/3418994548 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b100b41f0 0x7f1b1015b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 -- 172.21.15.59:0/3418994548 >> 172.21.15.59:0/3418994548 conn(0x7f1b100d3cd0 msgr2=0x7f1b1015d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.453 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 -- 172.21.15.59:0/3418994548 shutdown_connections 2024-03-06T07:31:50.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:50.451+0000 7f1b1879a640 1 -- 172.21.15.59:0/3418994548 wait complete. 2024-03-06T07:31:50.665 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:50.665 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:51.154 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:51.157 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:51.666 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:51.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.767+0000 7f31e3163640 1 Processor -- start 2024-03-06T07:31:51.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.767+0000 7f31e3163640 1 -- start start 2024-03-06T07:31:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.767+0000 7f31e3163640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31dc15dfe0 0x7f31dc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.767+0000 7f31e3163640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f31dc0cf830 0x7f31dc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.767+0000 7f31e3163640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31dc0d27f0 0x7f31dc0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.767+0000 7f31e3163640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc0b31e0 con 0x7f31dc15dfe0 2024-03-06T07:31:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.767+0000 7f31e3163640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc0b3d60 con 0x7f31dc0cf830 2024-03-06T07:31:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.767+0000 7f31e3163640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc0b3430 con 0x7f31dc0d27f0 2024-03-06T07:31:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.768+0000 7f31e0eff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31dc15dfe0 0x7f31dc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.770 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.768+0000 7f31e0eff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31dc15dfe0 0x7f31dc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34706/0 (socket says 172.21.15.59:34706) 2024-03-06T07:31:51.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.768+0000 7f31e0eff640 1 -- 172.21.15.59:0/4231727527 learned_addr learned my addr 172.21.15.59:0/4231727527 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:51.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.768+0000 7f31e1700640 1 --2- 172.21.15.59:0/4231727527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31dc0d27f0 0x7f31dc0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.768+0000 7f31d3fff640 1 --2- 172.21.15.59:0/4231727527 >> v2:172.21.15.164:3300/0 conn(0x7f31dc0cf830 0x7f31dc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.768+0000 7f31e0eff640 1 -- 172.21.15.59:0/4231727527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31dc0d27f0 msgr2=0x7f31dc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:51.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.768+0000 7f31e0eff640 1 --2- 172.21.15.59:0/4231727527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31dc0d27f0 0x7f31dc0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.768+0000 7f31e0eff640 1 -- 172.21.15.59:0/4231727527 >> v2:172.21.15.164:3300/0 conn(0x7f31dc0cf830 msgr2=0x7f31dc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:51.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.768+0000 7f31e0eff640 1 --2- 172.21.15.59:0/4231727527 >> v2:172.21.15.164:3300/0 conn(0x7f31dc0cf830 0x7f31dc0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.768+0000 7f31e0eff640 1 -- 172.21.15.59:0/4231727527 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc0b3710 con 0x7f31dc15dfe0 2024-03-06T07:31:51.772 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31e0eff640 1 --2- 172.21.15.59:0/4231727527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31dc15dfe0 0x7f31dc0cf2f0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0d16b0 tx=0x7f31d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3f3a0f24fd2605 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31d37fe640 1 -- 172.21.15.59:0/4231727527 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8035020 con 0x7f31dc15dfe0 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31d37fe640 1 -- 172.21.15.59:0/4231727527 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d8031870 con 0x7f31dc15dfe0 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31d37fe640 1 -- 172.21.15.59:0/4231727527 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8031c40 con 0x7f31dc15dfe0 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31e3163640 1 -- 172.21.15.59:0/4231727527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31dc15dfe0 msgr2=0x7f31dc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31e3163640 1 --2- 172.21.15.59:0/4231727527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31dc15dfe0 0x7f31dc0cf2f0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0d16b0 tx=0x7f31d802b910 comp rx=0 tx=0).stop 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31e3163640 1 -- 172.21.15.59:0/4231727527 shutdown_connections 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31e3163640 1 --2- 172.21.15.59:0/4231727527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31dc0d27f0 0x7f31dc0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31e3163640 1 --2- 172.21.15.59:0/4231727527 >> v2:172.21.15.164:3300/0 conn(0x7f31dc0cf830 0x7f31dc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31e3163640 1 --2- 172.21.15.59:0/4231727527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31dc15dfe0 0x7f31dc0cf2f0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31e3163640 1 -- 172.21.15.59:0/4231727527 >> 172.21.15.59:0/4231727527 conn(0x7f31dc0d3cd0 msgr2=0x7f31dc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31e3163640 1 -- 172.21.15.59:0/4231727527 shutdown_connections 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.769+0000 7f31e3163640 1 -- 172.21.15.59:0/4231727527 wait complete. 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.771+0000 7f31e3163640 1 Processor -- start 2024-03-06T07:31:51.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.771+0000 7f31e3163640 1 -- start start 2024-03-06T07:31:51.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.771+0000 7f31e3163640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31dc0cf830 0x7f31dc15f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.771+0000 7f31e3163640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31dc0d27f0 0x7f31dc0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.771+0000 7f31e3163640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31dc15dfe0 0x7f31dc159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.771+0000 7f31e3163640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc0d1df0 con 0x7f31dc0d27f0 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.771+0000 7f31e3163640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc0d1c70 con 0x7f31dc0cf830 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.771+0000 7f31e3163640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc15e3c0 con 0x7f31dc15dfe0 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31dc0d27f0 0x7f31dc0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31dc0d27f0 0x7f31dc0cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34714/0 (socket says 172.21.15.59:34714) 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d3fff640 1 -- 172.21.15.59:0/887259306 learned_addr learned my addr 172.21.15.59:0/887259306 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31e0eff640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31dc0cf830 0x7f31dc15f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31e1700640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31dc15dfe0 0x7f31dc159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d3fff640 1 -- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31dc15dfe0 msgr2=0x7f31dc159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d3fff640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31dc15dfe0 0x7f31dc159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d3fff640 1 -- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31dc0cf830 msgr2=0x7f31dc15f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d3fff640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31dc0cf830 0x7f31dc15f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d3fff640 1 -- 172.21.15.59:0/887259306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc15f730 con 0x7f31dc0d27f0 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d3fff640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31dc0d27f0 0x7f31dc0cd030 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00e860 tx=0x7f31cc00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d17fa640 1 -- 172.21.15.59:0/887259306 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc017070 con 0x7f31dc0d27f0 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d17fa640 1 -- 172.21.15.59:0/887259306 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc00c940 con 0x7f31dc0d27f0 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31e3163640 1 -- 172.21.15.59:0/887259306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31dc15fa10 con 0x7f31dc0d27f0 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31d17fa640 1 -- 172.21.15.59:0/887259306 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc00cd10 con 0x7f31dc0d27f0 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.772+0000 7f31e3163640 1 -- 172.21.15.59:0/887259306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc15ffd0 con 0x7f31dc0d27f0 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.773+0000 7f31e3163640 1 -- 172.21.15.59:0/887259306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31dc04f330 con 0x7f31dc0d27f0 2024-03-06T07:31:51.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.774+0000 7f31d17fa640 1 -- 172.21.15.59:0/887259306 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31cc01a070 con 0x7f31dc0d27f0 2024-03-06T07:31:51.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.774+0000 7f31d17fa640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31c806b4d0 0x7f31c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.774+0000 7f31d17fa640 1 -- 172.21.15.59:0/887259306 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f31cc08a5f0 con 0x7f31dc0d27f0 2024-03-06T07:31:51.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.774+0000 7f31e0eff640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31c806b4d0 0x7f31c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.775+0000 7f31e0eff640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31c806b4d0 0x7f31c806d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f31d8036000 tx=0x7f31d8008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:51.779+0000 7f31d17fa640 1 -- 172.21.15.59:0/887259306 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31cc059450 con 0x7f31dc0d27f0 2024-03-06T07:31:52.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.136+0000 7f31e3163640 1 -- 172.21.15.59:0/887259306 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31dc0cd570 con 0x7f31dc0d27f0 2024-03-06T07:31:52.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.137+0000 7f31d17fa640 1 -- 172.21.15.59:0/887259306 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f31cc058d50 con 0x7f31dc0d27f0 2024-03-06T07:31:52.139 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:52.139 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:52.139 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 -- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31c806b4d0 msgr2=0x7f31c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31c806b4d0 0x7f31c806d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f31d8036000 tx=0x7f31d8008030 comp rx=0 tx=0).stop 2024-03-06T07:31:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 -- 172.21.15.59:0/887259306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31dc0d27f0 msgr2=0x7f31dc0cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31dc0d27f0 0x7f31dc0cd030 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00e860 tx=0x7f31cc00ed20 comp rx=0 tx=0).stop 2024-03-06T07:31:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 -- 172.21.15.59:0/887259306 shutdown_connections 2024-03-06T07:31:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31c806b4d0 0x7f31c806d990 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31dc15dfe0 0x7f31dc159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31dc0d27f0 0x7f31dc0cd030 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 --2- 172.21.15.59:0/887259306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31dc0cf830 0x7f31dc15f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 -- 172.21.15.59:0/887259306 >> 172.21.15.59:0/887259306 conn(0x7f31dc0d3cd0 msgr2=0x7f31dc0b4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 -- 172.21.15.59:0/887259306 shutdown_connections 2024-03-06T07:31:52.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:52.139+0000 7f31e3163640 1 -- 172.21.15.59:0/887259306 wait complete. 2024-03-06T07:31:52.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:52.357 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:53.357 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.457+0000 7f649302a640 1 Processor -- start 2024-03-06T07:31:53.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.457+0000 7f649302a640 1 -- start start 2024-03-06T07:31:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f649302a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f648c15bb20 0x7f648c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f649302a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f648c161980 con 0x7f648c0cd8c0 2024-03-06T07:31:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f649302a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f648c161c60 con 0x7f648c0cfff0 2024-03-06T07:31:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f649302a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648c161f40 con 0x7f648c15bb20 2024-03-06T07:31:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f64915c7640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f648c0cd8c0 0x7f648c0cebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55186/0 (socket says 172.21.15.59:55186) 2024-03-06T07:31:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f64915c7640 1 -- 172.21.15.59:0/454976093 learned_addr learned my addr 172.21.15.59:0/454976093 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775836273 0 0) 0x7f648c161980 con 0x7f648c0cd8c0 2024-03-06T07:31:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f647c003520 con 0x7f648c0cd8c0 2024-03-06T07:31:53.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f6490dc6640 1 --2- 172.21.15.59:0/454976093 >> v2:172.21.15.164:3300/0 conn(0x7f648c15bb20 0x7f648c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3054042762 0 0) 0x7f647c003520 con 0x7f648c0cd8c0 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 >> v2:172.21.15.164:3300/0 conn(0x7f648c15bb20 msgr2=0x7f648c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f648b7fe640 1 --2- 172.21.15.59:0/454976093 >> v2:172.21.15.164:3300/0 conn(0x7f648c15bb20 0x7f648c0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 >> v1:172.21.15.164:6789/0 conn(0x7f648c0cfff0 legacy=0x7f648c0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.458+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c162220 con 0x7f648c0cd8c0 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f64840019f0 con 0x7f648c0cd8c0 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 >> v1:172.21.15.59:6789/0 conn(0x7f648c0cd8c0 legacy=0x7f648c0cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f648b7fe640 1 --2- 172.21.15.59:0/454976093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 0x7f647c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f648b7fe640 1 --2- 172.21.15.59:0/454976093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f647c004800 0x7f647c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f648b7fe640 1 --2- 172.21.15.59:0/454976093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f647c0051c0 0x7f647c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c161f40 con 0x7f647c0039b0 2024-03-06T07:31:53.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c00c0f0 con 0x7f647c0051c0 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c00c2f0 con 0x7f647c004800 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f6490dc6640 1 --2- 172.21.15.59:0/454976093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 0x7f647c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f648bfff640 1 --2- 172.21.15.59:0/454976093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f647c004800 0x7f647c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f64915c7640 1 --2- 172.21.15.59:0/454976093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f647c0051c0 0x7f647c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f6490dc6640 1 -- 172.21.15.59:0/454976093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f647c004800 msgr2=0x7f647c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f6490dc6640 1 --2- 172.21.15.59:0/454976093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f647c004800 0x7f647c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f6490dc6640 1 -- 172.21.15.59:0/454976093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f647c0051c0 msgr2=0x7f647c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f6490dc6640 1 --2- 172.21.15.59:0/454976093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f647c0051c0 0x7f647c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f64915c7640 1 -- 172.21.15.59:0/454976093 reap_dead start 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.459+0000 7f6490dc6640 1 -- 172.21.15.59:0/454976093 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c003520 con 0x7f647c0039b0 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.460+0000 7f6490dc6640 1 --2- 172.21.15.59:0/454976093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 0x7f647c003db0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f648000ba00 tx=0x7f647c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39f1f7e79202f37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.460+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480036020 con 0x7f647c0039b0 2024-03-06T07:31:53.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.460+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6480032620 con 0x7f647c0039b0 2024-03-06T07:31:53.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.460+0000 7f648b7fe640 1 -- 172.21.15.59:0/454976093 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648000b600 con 0x7f647c0039b0 2024-03-06T07:31:53.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.460+0000 7f649302a640 1 -- 172.21.15.59:0/454976093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 msgr2=0x7f647c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.460+0000 7f649302a640 1 --2- 172.21.15.59:0/454976093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 0x7f647c003db0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f648000ba00 tx=0x7f647c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:53.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.460+0000 7f649302a640 1 -- 172.21.15.59:0/454976093 shutdown_connections 2024-03-06T07:31:53.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.460+0000 7f649302a640 1 --2- 172.21.15.59:0/454976093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 0x7f647c003db0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.460+0000 7f649302a640 1 -- 172.21.15.59:0/454976093 >> 172.21.15.59:0/454976093 conn(0x7f648c09a9f0 msgr2=0x7f648c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.461+0000 7f649302a640 1 -- 172.21.15.59:0/454976093 shutdown_connections 2024-03-06T07:31:53.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.461+0000 7f649302a640 1 -- 172.21.15.59:0/454976093 wait complete. 2024-03-06T07:31:53.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.462+0000 7f649302a640 1 Processor -- start 2024-03-06T07:31:53.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.462+0000 7f649302a640 1 -- start start 2024-03-06T07:31:53.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.463+0000 7f649302a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 0x7f648c0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.463+0000 7f649302a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f648c06d120 0x7f648c0c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.463+0000 7f649302a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f648c06d5a0 0x7f648c0af170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.463+0000 7f649302a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c0d1b00 con 0x7f647c0039b0 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.463+0000 7f649302a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c0d1980 con 0x7f648c06d120 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.463+0000 7f649302a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648c0d04a0 con 0x7f648c06d5a0 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f6490dc6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 0x7f648c0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f6490dc6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 0x7f648c0c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34744/0 (socket says 172.21.15.59:34744) 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f6490dc6640 1 -- 172.21.15.59:0/3262957981 learned_addr learned my addr 172.21.15.59:0/3262957981 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f6490dc6640 1 -- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f648c06d5a0 msgr2=0x7f648c0af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f6490dc6640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f648c06d5a0 0x7f648c0af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f6490dc6640 1 -- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f648c06d120 msgr2=0x7f648c0c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f6490dc6640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f648c06d120 0x7f648c0c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f6490dc6640 1 -- 172.21.15.59:0/3262957981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c158c90 con 0x7f647c0039b0 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f6490dc6640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 0x7f648c0c8b90 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f6480035040 tx=0x7f6480004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f64897fa640 1 -- 172.21.15.59:0/3262957981 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480041070 con 0x7f647c0039b0 2024-03-06T07:31:53.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f64897fa640 1 -- 172.21.15.59:0/3262957981 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648003b420 con 0x7f647c0039b0 2024-03-06T07:31:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f649302a640 1 -- 172.21.15.59:0/3262957981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648c1586d0 con 0x7f647c0039b0 2024-03-06T07:31:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f64897fa640 1 -- 172.21.15.59:0/3262957981 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480036040 con 0x7f647c0039b0 2024-03-06T07:31:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.464+0000 7f649302a640 1 -- 172.21.15.59:0/3262957981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648c0ae990 con 0x7f647c0039b0 2024-03-06T07:31:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.465+0000 7f649302a640 1 -- 172.21.15.59:0/3262957981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648c04f330 con 0x7f647c0039b0 2024-03-06T07:31:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.466+0000 7f64897fa640 1 -- 172.21.15.59:0/3262957981 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6480039020 con 0x7f647c0039b0 2024-03-06T07:31:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.466+0000 7f64897fa640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f647006b4d0 0x7f647006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.466+0000 7f64897fa640 1 -- 172.21.15.59:0/3262957981 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f64800abd20 con 0x7f647c0039b0 2024-03-06T07:31:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.467+0000 7f648bfff640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f647006b4d0 0x7f647006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.467+0000 7f648bfff640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f647006b4d0 0x7f647006d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f648c159550 tx=0x7f6478009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.470+0000 7f64897fa640 1 -- 172.21.15.59:0/3262957981 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f648007ac10 con 0x7f647c0039b0 2024-03-06T07:31:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.708+0000 7f649302a640 1 -- 172.21.15.59:0/3262957981 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f648c0b1820 con 0x7f647c0039b0 2024-03-06T07:31:53.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.708+0000 7f64897fa640 1 -- 172.21.15.59:0/3262957981 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f648007a510 con 0x7f647c0039b0 2024-03-06T07:31:53.710 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:53.711 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:53.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.710+0000 7f649302a640 1 -- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f647006b4d0 msgr2=0x7f647006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.710+0000 7f649302a640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f647006b4d0 0x7f647006d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f648c159550 tx=0x7f6478009040 comp rx=0 tx=0).stop 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.710+0000 7f649302a640 1 -- 172.21.15.59:0/3262957981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 msgr2=0x7f648c0c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.710+0000 7f649302a640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 0x7f648c0c8b90 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f6480035040 tx=0x7f6480004730 comp rx=0 tx=0).stop 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.710+0000 7f649302a640 1 -- 172.21.15.59:0/3262957981 shutdown_connections 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.710+0000 7f649302a640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f647006b4d0 0x7f647006d990 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.710+0000 7f649302a640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f648c06d5a0 0x7f648c0af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.710+0000 7f649302a640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f648c06d120 0x7f648c0c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.710+0000 7f649302a640 1 --2- 172.21.15.59:0/3262957981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f647c0039b0 0x7f648c0c8b90 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.710+0000 7f649302a640 1 -- 172.21.15.59:0/3262957981 >> 172.21.15.59:0/3262957981 conn(0x7f648c09a9f0 msgr2=0x7f648c0b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.711+0000 7f649302a640 1 -- 172.21.15.59:0/3262957981 shutdown_connections 2024-03-06T07:31:53.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:53.711+0000 7f649302a640 1 -- 172.21.15.59:0/3262957981 wait complete. 2024-03-06T07:31:53.924 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:53.925 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:54.926 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:55.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.048+0000 7fca1a86b640 1 Processor -- start 2024-03-06T07:31:55.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.048+0000 7fca1a86b640 1 -- start start 2024-03-06T07:31:55.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.048+0000 7fca1a86b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fca140d22b0 0x7fca140d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.048+0000 7fca1a86b640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca140b3430 con 0x7fca1415dfe0 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.048+0000 7fca1a86b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca140b3710 con 0x7fca140b41f0 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.048+0000 7fca1a86b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca140d2c50 con 0x7fca140d22b0 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca13fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fca1415dfe0 0x7fca140cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47494/0 (socket says 172.21.15.59:47494) 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca13fff640 1 -- 172.21.15.59:0/1518980291 learned_addr learned my addr 172.21.15.59:0/1518980291 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca137fe640 1 --2- 172.21.15.59:0/1518980291 >> v2:172.21.15.164:3300/0 conn(0x7fca140d22b0 0x7fca140d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518397463 0 0) 0x7fca140b3430 con 0x7fca1415dfe0 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9fc003520 con 0x7fca1415dfe0 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2010840307 0 0) 0x7fc9fc003520 con 0x7fca1415dfe0 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 >> v2:172.21.15.164:3300/0 conn(0x7fca140d22b0 msgr2=0x7fca140d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 --2- 172.21.15.59:0/1518980291 >> v2:172.21.15.164:3300/0 conn(0x7fca140d22b0 0x7fca140d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 >> v1:172.21.15.164:6789/0 conn(0x7fca140b41f0 legacy=0x7fca140b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca140b39f0 con 0x7fca1415dfe0 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fca04001670 con 0x7fca1415dfe0 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 >> v1:172.21.15.59:6789/0 conn(0x7fca1415dfe0 legacy=0x7fca140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.053 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 --2- 172.21.15.59:0/1518980291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc0039b0 0x7fc9fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 --2- 172.21.15.59:0/1518980291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9fc004800 0x7fc9fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 --2- 172.21.15.59:0/1518980291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9fc0051c0 0x7fc9fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc00c030 con 0x7fc9fc0051c0 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc00c1e0 con 0x7fc9fc0039b0 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc00c3e0 con 0x7fc9fc004800 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca18e08640 1 --2- 172.21.15.59:0/1518980291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9fc0051c0 0x7fc9fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca13fff640 1 --2- 172.21.15.59:0/1518980291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc0039b0 0x7fc9fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.049+0000 7fca137fe640 1 --2- 172.21.15.59:0/1518980291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9fc004800 0x7fc9fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.050+0000 7fca18e08640 1 -- 172.21.15.59:0/1518980291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9fc004800 msgr2=0x7fc9fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.050+0000 7fca18e08640 1 --2- 172.21.15.59:0/1518980291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9fc004800 0x7fc9fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.050+0000 7fca18e08640 1 -- 172.21.15.59:0/1518980291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc0039b0 msgr2=0x7fc9fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.050+0000 7fca18e08640 1 --2- 172.21.15.59:0/1518980291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc0039b0 0x7fc9fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.050+0000 7fca18e08640 1 -- 172.21.15.59:0/1518980291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc003520 con 0x7fc9fc0051c0 2024-03-06T07:31:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.050+0000 7fca18e08640 1 -- 172.21.15.59:0/1518980291 reap_dead start 2024-03-06T07:31:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.050+0000 7fca18e08640 1 --2- 172.21.15.59:0/1518980291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9fc0051c0 0x7fc9fc00b9c0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fca0c001e50 tx=0x7fc9fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5680a358d4f0ad2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.051+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c001a70 con 0x7fc9fc0051c0 2024-03-06T07:31:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.051+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c003bb0 con 0x7fc9fc0051c0 2024-03-06T07:31:55.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.051+0000 7fca12ffd640 1 -- 172.21.15.59:0/1518980291 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0050d0 con 0x7fc9fc0051c0 2024-03-06T07:31:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.051+0000 7fca1a86b640 1 -- 172.21.15.59:0/1518980291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9fc0051c0 msgr2=0x7fc9fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.051+0000 7fca1a86b640 1 --2- 172.21.15.59:0/1518980291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9fc0051c0 0x7fc9fc00b9c0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fca0c001e50 tx=0x7fc9fc005830 comp rx=0 tx=0).stop 2024-03-06T07:31:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.051+0000 7fca1a86b640 1 -- 172.21.15.59:0/1518980291 shutdown_connections 2024-03-06T07:31:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.051+0000 7fca1a86b640 1 --2- 172.21.15.59:0/1518980291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc9fc0051c0 0x7fc9fc00b9c0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.051+0000 7fca1a86b640 1 -- 172.21.15.59:0/1518980291 >> 172.21.15.59:0/1518980291 conn(0x7fca140d3cd0 msgr2=0x7fca140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.051+0000 7fca1a86b640 1 -- 172.21.15.59:0/1518980291 shutdown_connections 2024-03-06T07:31:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.051+0000 7fca1a86b640 1 -- 172.21.15.59:0/1518980291 wait complete. 2024-03-06T07:31:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.052+0000 7fca1a86b640 1 Processor -- start 2024-03-06T07:31:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.053+0000 7fca1a86b640 1 -- start start 2024-03-06T07:31:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.053+0000 7fca1a86b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc0051c0 0x7fca141f3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.053+0000 7fca1a86b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca140cc8f0 0x7fca14156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.053+0000 7fca1a86b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1415b650 0x7fca1415bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.053+0000 7fca1a86b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca140b3d60 con 0x7fca140cc8f0 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.053+0000 7fca1a86b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca140b31e0 con 0x7fc9fc0051c0 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.053+0000 7fca1a86b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1415e3c0 con 0x7fca1415b650 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.053+0000 7fca137fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca140cc8f0 0x7fca14156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.053+0000 7fca137fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca140cc8f0 0x7fca14156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54196/0 (socket says 172.21.15.59:54196) 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.053+0000 7fca137fe640 1 -- 172.21.15.59:0/3506641328 learned_addr learned my addr 172.21.15.59:0/3506641328 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca18e08640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1415b650 0x7fca1415bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca13fff640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc0051c0 0x7fca141f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca137fe640 1 -- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1415b650 msgr2=0x7fca1415bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca137fe640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1415b650 0x7fca1415bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca137fe640 1 -- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc0051c0 msgr2=0x7fca141f3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca137fe640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc0051c0 0x7fca141f3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca137fe640 1 -- 172.21.15.59:0/3506641328 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca140cce90 con 0x7fca140cc8f0 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca18e08640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1415b650 0x7fca1415bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca13fff640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc0051c0 0x7fca141f3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca137fe640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca140cc8f0 0x7fca14156560 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fca08011910 tx=0x7fca08011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca10ff9640 1 -- 172.21.15.59:0/3506641328 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0800fa90 con 0x7fca140cc8f0 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca10ff9640 1 -- 172.21.15.59:0/3506641328 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0800fc30 con 0x7fca140cc8f0 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca1a86b640 1 -- 172.21.15.59:0/3506641328 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1415eb60 con 0x7fca140cc8f0 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca10ff9640 1 -- 172.21.15.59:0/3506641328 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08014440 con 0x7fca140cc8f0 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.054+0000 7fca1a86b640 1 -- 172.21.15.59:0/3506641328 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca141f3950 con 0x7fca140cc8f0 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.055+0000 7fca1a86b640 1 -- 172.21.15.59:0/3506641328 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1404f330 con 0x7fca140cc8f0 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.056+0000 7fca10ff9640 1 -- 172.21.15.59:0/3506641328 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca08012040 con 0x7fca140cc8f0 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.056+0000 7fca10ff9640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca0006b500 0x7fca0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.060 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.056+0000 7fca10ff9640 1 -- 172.21.15.59:0/3506641328 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fca0808a7f0 con 0x7fca140cc8f0 2024-03-06T07:31:55.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.056+0000 7fca13fff640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca0006b500 0x7fca0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.061 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.057+0000 7fca13fff640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca0006b500 0x7fca0006d9c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fca04001cc0 tx=0x7fca04009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.062 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.060+0000 7fca10ff9640 1 -- 172.21.15.59:0/3506641328 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca08019030 con 0x7fca140cc8f0 2024-03-06T07:31:55.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.281+0000 7fca1a86b640 1 -- 172.21.15.59:0/3506641328 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca141e8940 con 0x7fca140cc8f0 2024-03-06T07:31:55.284 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.282+0000 7fca10ff9640 1 -- 172.21.15.59:0/3506641328 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fca08059650 con 0x7fca140cc8f0 2024-03-06T07:31:55.284 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:55.285 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 -- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca0006b500 msgr2=0x7fca0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca0006b500 0x7fca0006d9c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fca04001cc0 tx=0x7fca04009040 comp rx=0 tx=0).stop 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 -- 172.21.15.59:0/3506641328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca140cc8f0 msgr2=0x7fca14156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca140cc8f0 0x7fca14156560 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fca08011910 tx=0x7fca08011dd0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 -- 172.21.15.59:0/3506641328 shutdown_connections 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca0006b500 0x7fca0006d9c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1415b650 0x7fca1415bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca140cc8f0 0x7fca14156560 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 --2- 172.21.15.59:0/3506641328 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc0051c0 0x7fca141f3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 -- 172.21.15.59:0/3506641328 >> 172.21.15.59:0/3506641328 conn(0x7fca140d3cd0 msgr2=0x7fca140b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 -- 172.21.15.59:0/3506641328 shutdown_connections 2024-03-06T07:31:55.287 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:55.284+0000 7fca1a86b640 1 -- 172.21.15.59:0/3506641328 wait complete. 2024-03-06T07:31:55.498 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:55.498 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:56.499 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:56.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.599+0000 7fee38916640 1 Processor -- start 2024-03-06T07:31:56.601 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.599+0000 7fee38916640 1 -- start start 2024-03-06T07:31:56.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee38916640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee3413dc10 0x7fee340b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee38916640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 0x7fee340b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee38916640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fee3415e850 0x7fee3415eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee38916640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee340cf3f0 con 0x7fee3415e850 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee38916640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee3415b210 con 0x7fee3413dc10 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee38916640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3415f580 con 0x7fee3415bf30 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee32d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fee3415e850 0x7fee3415eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee32d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fee3415e850 0x7fee3415eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54204/0 (socket says 172.21.15.59:54204) 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee32d9d640 1 -- 172.21.15.59:0/2777039908 learned_addr learned my addr 172.21.15.59:0/2777039908 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee2bfff640 1 --2- 172.21.15.59:0/2777039908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 0x7fee340b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee3259c640 1 --2- 172.21.15.59:0/2777039908 >> v2:172.21.15.164:3300/0 conn(0x7fee3413dc10 0x7fee340b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee32d9d640 1 -- 172.21.15.59:0/2777039908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 msgr2=0x7fee340b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee32d9d640 1 --2- 172.21.15.59:0/2777039908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 0x7fee340b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee32d9d640 1 -- 172.21.15.59:0/2777039908 >> v2:172.21.15.164:3300/0 conn(0x7fee3413dc10 msgr2=0x7fee340b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee32d9d640 1 --2- 172.21.15.59:0/2777039908 >> v2:172.21.15.164:3300/0 conn(0x7fee3413dc10 0x7fee340b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee32d9d640 1 -- 172.21.15.59:0/2777039908 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3415b450 con 0x7fee3415e850 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee3259c640 1 --2- 172.21.15.59:0/2777039908 >> v2:172.21.15.164:3300/0 conn(0x7fee3413dc10 0x7fee340b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:56.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.600+0000 7fee2bfff640 1 --2- 172.21.15.59:0/2777039908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 0x7fee340b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.601+0000 7fee32d9d640 1 --2- 172.21.15.59:0/2777039908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3415e850 0x7fee3415eca0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fee340731a0 tx=0x7fee2c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5e29a38e887986 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.601+0000 7fee31d9b640 1 -- 172.21.15.59:0/2777039908 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c02c9e0 con 0x7fee3415e850 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.601+0000 7fee31d9b640 1 -- 172.21.15.59:0/2777039908 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2c02cb80 con 0x7fee3415e850 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.601+0000 7fee31d9b640 1 -- 172.21.15.59:0/2777039908 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c0343f0 con 0x7fee3415e850 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.602+0000 7fee38916640 1 -- 172.21.15.59:0/2777039908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3415e850 msgr2=0x7fee3415eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.602+0000 7fee38916640 1 --2- 172.21.15.59:0/2777039908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3415e850 0x7fee3415eca0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fee340731a0 tx=0x7fee2c02be40 comp rx=0 tx=0).stop 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.602+0000 7fee38916640 1 -- 172.21.15.59:0/2777039908 shutdown_connections 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.602+0000 7fee38916640 1 --2- 172.21.15.59:0/2777039908 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3415e850 0x7fee3415eca0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.602+0000 7fee38916640 1 --2- 172.21.15.59:0/2777039908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 0x7fee340b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.602+0000 7fee38916640 1 --2- 172.21.15.59:0/2777039908 >> v2:172.21.15.164:3300/0 conn(0x7fee3413dc10 0x7fee340b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.602+0000 7fee38916640 1 -- 172.21.15.59:0/2777039908 >> 172.21.15.59:0/2777039908 conn(0x7fee340cf8a0 msgr2=0x7fee340cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.602+0000 7fee38916640 1 -- 172.21.15.59:0/2777039908 shutdown_connections 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.602+0000 7fee38916640 1 -- 172.21.15.59:0/2777039908 wait complete. 2024-03-06T07:31:56.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.603+0000 7fee38916640 1 Processor -- start 2024-03-06T07:31:56.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.603+0000 7fee38916640 1 -- start start 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee38916640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee3413dc10 0x7fee3414fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee38916640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 0x7fee34148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee38916640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3415e850 0x7fee3413c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee38916640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3415f1e0 con 0x7fee3415e850 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee38916640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3413dff0 con 0x7fee3413dc10 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee38916640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3415f360 con 0x7fee3415bf30 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee32d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3415e850 0x7fee3413c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee32d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3415e850 0x7fee3413c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54216/0 (socket says 172.21.15.59:54216) 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee32d9d640 1 -- 172.21.15.59:0/2455793971 learned_addr learned my addr 172.21.15.59:0/2455793971 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee2bfff640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 0x7fee34148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee3259c640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee3413dc10 0x7fee3414fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee32d9d640 1 -- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 msgr2=0x7fee34148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee32d9d640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 0x7fee34148040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee32d9d640 1 -- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee3413dc10 msgr2=0x7fee3414fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee32d9d640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee3413dc10 0x7fee3414fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee32d9d640 1 -- 172.21.15.59:0/2455793971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3414b380 con 0x7fee3415e850 2024-03-06T07:31:56.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee2bfff640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 0x7fee34148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:56.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.604+0000 7fee32d9d640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3415e850 0x7fee3413c3e0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009e50 tx=0x7fee2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.605+0000 7fee2affd640 1 -- 172.21.15.59:0/2455793971 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c043020 con 0x7fee3415e850 2024-03-06T07:31:56.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.605+0000 7fee2affd640 1 -- 172.21.15.59:0/2455793971 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2c03e450 con 0x7fee3415e850 2024-03-06T07:31:56.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.605+0000 7fee38916640 1 -- 172.21.15.59:0/2455793971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3414eb40 con 0x7fee3415e850 2024-03-06T07:31:56.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.605+0000 7fee2affd640 1 -- 172.21.15.59:0/2455793971 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c009510 con 0x7fee3415e850 2024-03-06T07:31:56.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.605+0000 7fee38916640 1 -- 172.21.15.59:0/2455793971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3414f100 con 0x7fee3415e850 2024-03-06T07:31:56.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.606+0000 7fee38916640 1 -- 172.21.15.59:0/2455793971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3404f330 con 0x7fee3415e850 2024-03-06T07:31:56.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.606+0000 7fee2affd640 1 -- 172.21.15.59:0/2455793971 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee2c046020 con 0x7fee3415e850 2024-03-06T07:31:56.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.606+0000 7fee2affd640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee1406b4d0 0x7fee1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.607+0000 7fee2affd640 1 -- 172.21.15.59:0/2455793971 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fee2c0aca50 con 0x7fee3415e850 2024-03-06T07:31:56.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.607+0000 7fee3259c640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee1406b4d0 0x7fee1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.607+0000 7fee3259c640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee1406b4d0 0x7fee1406d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fee24006fd0 tx=0x7fee24008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.611+0000 7fee2affd640 1 -- 172.21.15.59:0/2455793971 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee2c04d030 con 0x7fee3415e850 2024-03-06T07:31:56.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.830+0000 7fee38916640 1 -- 172.21.15.59:0/2455793971 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee3406eb00 con 0x7fee3415e850 2024-03-06T07:31:56.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.831+0000 7fee2affd640 1 -- 172.21.15.59:0/2455793971 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fee2c07b8b0 con 0x7fee3415e850 2024-03-06T07:31:56.833 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:56.833 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:56.833 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:56.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 -- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee1406b4d0 msgr2=0x7fee1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee1406b4d0 0x7fee1406d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fee24006fd0 tx=0x7fee24008040 comp rx=0 tx=0).stop 2024-03-06T07:31:56.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 -- 172.21.15.59:0/2455793971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3415e850 msgr2=0x7fee3413c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3415e850 0x7fee3413c3e0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fee2c009e50 tx=0x7fee2c004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:56.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 -- 172.21.15.59:0/2455793971 shutdown_connections 2024-03-06T07:31:56.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fee1406b4d0 0x7fee1406d990 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee3415e850 0x7fee3413c3e0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee3415bf30 0x7fee34148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 --2- 172.21.15.59:0/2455793971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee3413dc10 0x7fee3414fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 -- 172.21.15.59:0/2455793971 >> 172.21.15.59:0/2455793971 conn(0x7fee340cf8a0 msgr2=0x7fee340707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 -- 172.21.15.59:0/2455793971 shutdown_connections 2024-03-06T07:31:56.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:56.833+0000 7fee38916640 1 -- 172.21.15.59:0/2455793971 wait complete. 2024-03-06T07:31:57.047 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:57.047 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:58.048 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:58.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.151+0000 7ffa1c53a640 1 Processor -- start 2024-03-06T07:31:58.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.151+0000 7ffa1c53a640 1 -- start start 2024-03-06T07:31:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa1c53a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa1415bdd0 0x7ffa140c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa1c53a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140cebe0 con 0x7ffa1415fa10 2024-03-06T07:31:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa1c53a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140ceec0 con 0x7ffa140d16e0 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa1c53a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140ce980 con 0x7ffa1415bdd0 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa19ad5640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ffa1415fa10 0x7ffa140ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47510/0 (socket says 172.21.15.59:47510) 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa19ad5640 1 -- 172.21.15.59:0/2578158699 learned_addr learned my addr 172.21.15.59:0/2578158699 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa1a2d6640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa1415bdd0 0x7ffa140c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362118125 0 0) 0x7ffa140cebe0 con 0x7ffa1415fa10 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa00003520 con 0x7ffa1415fa10 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2398117694 0 0) 0x7ffa00003520 con 0x7ffa1415fa10 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa1415bdd0 msgr2=0x7ffa140c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa1415bdd0 0x7ffa140c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 >> v1:172.21.15.164:6789/0 conn(0x7ffa140d16e0 legacy=0x7ffa140d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa140cf1a0 con 0x7ffa1415fa10 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7ffa04001670 con 0x7ffa1415fa10 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 >> v1:172.21.15.59:6789/0 conn(0x7ffa1415fa10 legacy=0x7ffa140ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa1415bdd0 0x7ffa000039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 0x7ffa00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa00004d40 0x7ffa000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140ce980 con 0x7ffa00003f60 2024-03-06T07:31:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0000bb50 con 0x7ffa00004d40 2024-03-06T07:31:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.152+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0000bd50 con 0x7ffa1415bdd0 2024-03-06T07:31:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.153+0000 7ffa19ad5640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 0x7ffa00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.153+0000 7ffa1a2d6640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa1415bdd0 0x7ffa000039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.153+0000 7ffa1aad7640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa00004d40 0x7ffa000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.153+0000 7ffa19ad5640 1 -- 172.21.15.59:0/2578158699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa1415bdd0 msgr2=0x7ffa000039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.153+0000 7ffa19ad5640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa1415bdd0 0x7ffa000039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.153+0000 7ffa19ad5640 1 -- 172.21.15.59:0/2578158699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa00004d40 msgr2=0x7ffa000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.153+0000 7ffa19ad5640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa00004d40 0x7ffa000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.153+0000 7ffa19ad5640 1 -- 172.21.15.59:0/2578158699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa00003520 con 0x7ffa00003f60 2024-03-06T07:31:58.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.154+0000 7ffa19ad5640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 0x7ffa00004800 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c210 tx=0x7ffa00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180a53553ba13187 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.154+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04007d30 con 0x7ffa00003f60 2024-03-06T07:31:58.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.154+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa04030070 con 0x7ffa00003f60 2024-03-06T07:31:58.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.154+0000 7ffa192d4640 1 -- 172.21.15.59:0/2578158699 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0402cd60 con 0x7ffa00003f60 2024-03-06T07:31:58.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.155+0000 7ffa1c53a640 1 -- 172.21.15.59:0/2578158699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 msgr2=0x7ffa00004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.155+0000 7ffa1c53a640 1 --2- 172.21.15.59:0/2578158699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 0x7ffa00004800 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c210 tx=0x7ffa00005830 comp rx=0 tx=0).stop 2024-03-06T07:31:58.160 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.155+0000 7ffa1aad7640 1 -- 172.21.15.59:0/2578158699 reap_dead start 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.155+0000 7ffa1c53a640 1 -- 172.21.15.59:0/2578158699 shutdown_connections 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.155+0000 7ffa1c53a640 1 -- 172.21.15.59:0/2578158699 >> 172.21.15.59:0/2578158699 conn(0x7ffa1409a9f0 msgr2=0x7ffa1409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.155+0000 7ffa1c53a640 1 -- 172.21.15.59:0/2578158699 shutdown_connections 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.155+0000 7ffa1c53a640 1 -- 172.21.15.59:0/2578158699 wait complete. 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.156+0000 7ffa1c53a640 1 Processor -- start 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.156+0000 7ffa1c53a640 1 -- start start 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1c53a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 0x7ffa1415e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1c53a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa1406c0c0 0x7ffa141597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1c53a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa140cb370 0x7ffa1415c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1c53a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1415c1b0 con 0x7ffa00003f60 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1c53a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140cc5d0 con 0x7ffa1406c0c0 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1c53a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140c5610 con 0x7ffa140cb370 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1a2d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 0x7ffa1415e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1a2d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 0x7ffa1415e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54224/0 (socket says 172.21.15.59:54224) 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1a2d6640 1 -- 172.21.15.59:0/297609966 learned_addr learned my addr 172.21.15.59:0/297609966 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:58.161 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa19ad5640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa1406c0c0 0x7ffa141597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1aad7640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa140cb370 0x7ffa1415c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1a2d6640 1 -- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa140cb370 msgr2=0x7ffa1415c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1a2d6640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa140cb370 0x7ffa1415c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1a2d6640 1 -- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa1406c0c0 msgr2=0x7ffa141597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1a2d6640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa1406c0c0 0x7ffa141597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1a2d6640 1 -- 172.21.15.59:0/297609966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1415a100 con 0x7ffa00003f60 2024-03-06T07:31:58.162 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.157+0000 7ffa1a2d6640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 0x7ffa1415e1a0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7ffa100119a0 tx=0x7ffa10011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.158+0000 7ffa0affd640 1 -- 172.21.15.59:0/297609966 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1000fa90 con 0x7ffa00003f60 2024-03-06T07:31:58.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.158+0000 7ffa0affd640 1 -- 172.21.15.59:0/297609966 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1000fc30 con 0x7ffa00003f60 2024-03-06T07:31:58.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.158+0000 7ffa1c53a640 1 -- 172.21.15.59:0/297609966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1415ea80 con 0x7ffa00003f60 2024-03-06T07:31:58.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.158+0000 7ffa0affd640 1 -- 172.21.15.59:0/297609966 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10014440 con 0x7ffa00003f60 2024-03-06T07:31:58.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.158+0000 7ffa1c53a640 1 -- 172.21.15.59:0/297609966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1415f040 con 0x7ffa00003f60 2024-03-06T07:31:58.163 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.159+0000 7ffa1c53a640 1 -- 172.21.15.59:0/297609966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1404f330 con 0x7ffa00003f60 2024-03-06T07:31:58.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.160+0000 7ffa0affd640 1 -- 172.21.15.59:0/297609966 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa10012040 con 0x7ffa00003f60 2024-03-06T07:31:58.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.160+0000 7ffa0affd640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff9f806b4d0 0x7ff9f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.160+0000 7ffa0affd640 1 -- 172.21.15.59:0/297609966 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ffa1008a6b0 con 0x7ffa00003f60 2024-03-06T07:31:58.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.161+0000 7ffa19ad5640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff9f806b4d0 0x7ff9f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.165 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.161+0000 7ffa19ad5640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff9f806b4d0 0x7ff9f806d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c210 tx=0x7ffa04009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.164+0000 7ffa0affd640 1 -- 172.21.15.59:0/297609966 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa10019030 con 0x7ffa00003f60 2024-03-06T07:31:58.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.389+0000 7ffa1c53a640 1 -- 172.21.15.59:0/297609966 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa140b7a40 con 0x7ffa00003f60 2024-03-06T07:31:58.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.389+0000 7ffa0affd640 1 -- 172.21.15.59:0/297609966 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7ffa10058ec0 con 0x7ffa00003f60 2024-03-06T07:31:58.391 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:31:58.391 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:31:58.392 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:31:58.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 -- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff9f806b4d0 msgr2=0x7ff9f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff9f806b4d0 0x7ff9f806d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c210 tx=0x7ffa04009030 comp rx=0 tx=0).stop 2024-03-06T07:31:58.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 -- 172.21.15.59:0/297609966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 msgr2=0x7ffa1415e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 0x7ffa1415e1a0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7ffa100119a0 tx=0x7ffa10011e60 comp rx=0 tx=0).stop 2024-03-06T07:31:58.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 -- 172.21.15.59:0/297609966 shutdown_connections 2024-03-06T07:31:58.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff9f806b4d0 0x7ff9f806d990 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa140cb370 0x7ffa1415c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa1406c0c0 0x7ffa141597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 --2- 172.21.15.59:0/297609966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa00003f60 0x7ffa1415e1a0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 -- 172.21.15.59:0/297609966 >> 172.21.15.59:0/297609966 conn(0x7ffa1409a9f0 msgr2=0x7ffa1406eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 -- 172.21.15.59:0/297609966 shutdown_connections 2024-03-06T07:31:58.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:58.391+0000 7ffa1c53a640 1 -- 172.21.15.59:0/297609966 wait complete. 2024-03-06T07:31:58.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:31:58.605 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:31:59.606 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:59.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.706+0000 7f2393215640 1 Processor -- start 2024-03-06T07:31:59.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.706+0000 7f2393215640 1 -- start start 2024-03-06T07:31:59.708 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.706+0000 7f2393215640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f238c0d22b0 0x7f238c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.706+0000 7f2393215640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c0b3430 con 0x7f238c15dfe0 2024-03-06T07:31:59.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.706+0000 7f2393215640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238c0b3710 con 0x7f238c0b41f0 2024-03-06T07:31:59.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.706+0000 7f2393215640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f238c0d2c50 con 0x7f238c0d22b0 2024-03-06T07:31:59.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.707+0000 7f238bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f238c0d22b0 0x7f238c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.707+0000 7f2390fb1640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f238c15dfe0 0x7f238c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47522/0 (socket says 172.21.15.59:47522) 2024-03-06T07:31:59.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.707+0000 7f2390fb1640 1 -- 172.21.15.59:0/4215246580 learned_addr learned my addr 172.21.15.59:0/4215246580 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:31:59.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.707+0000 7f238bfff640 1 -- 172.21.15.59:0/4215246580 >> v1:172.21.15.164:6789/0 conn(0x7f238c0b41f0 legacy=0x7f238c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.707+0000 7f238bfff640 1 -- 172.21.15.59:0/4215246580 >> v1:172.21.15.59:6789/0 conn(0x7f238c15dfe0 legacy=0x7f238c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.709 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.707+0000 7f238bfff640 1 -- 172.21.15.59:0/4215246580 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c0b39f0 con 0x7f238c0d22b0 2024-03-06T07:31:59.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.708+0000 7f238bfff640 1 --2- 172.21.15.59:0/4215246580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f238c0d22b0 0x7f238c0d2710 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f23740097d0 tx=0x7f237402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1fb7309becdbeb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.708+0000 7f238b7fe640 1 -- 172.21.15.59:0/4215246580 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2374035020 con 0x7f238c0d22b0 2024-03-06T07:31:59.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.708+0000 7f238b7fe640 1 -- 172.21.15.59:0/4215246580 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2374031830 con 0x7f238c0d22b0 2024-03-06T07:31:59.710 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.708+0000 7f238b7fe640 1 -- 172.21.15.59:0/4215246580 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2374031c00 con 0x7f238c0d22b0 2024-03-06T07:31:59.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.708+0000 7f2393215640 1 -- 172.21.15.59:0/4215246580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f238c0d22b0 msgr2=0x7f238c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.708+0000 7f2393215640 1 --2- 172.21.15.59:0/4215246580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f238c0d22b0 0x7f238c0d2710 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f23740097d0 tx=0x7f237402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.708+0000 7f2393215640 1 -- 172.21.15.59:0/4215246580 shutdown_connections 2024-03-06T07:31:59.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.708+0000 7f2393215640 1 --2- 172.21.15.59:0/4215246580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f238c0d22b0 0x7f238c0d2710 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.708+0000 7f2393215640 1 -- 172.21.15.59:0/4215246580 >> 172.21.15.59:0/4215246580 conn(0x7f238c0d3cd0 msgr2=0x7f238c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.708+0000 7f2393215640 1 -- 172.21.15.59:0/4215246580 shutdown_connections 2024-03-06T07:31:59.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.708+0000 7f2393215640 1 -- 172.21.15.59:0/4215246580 wait complete. 2024-03-06T07:31:59.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.709+0000 7f2393215640 1 Processor -- start 2024-03-06T07:31:59.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.709+0000 7f2393215640 1 -- start start 2024-03-06T07:31:59.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.710+0000 7f2393215640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f238c0b41f0 0x7f238c14d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.710+0000 7f2393215640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f238c0d22b0 0x7f238c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.710+0000 7f2393215640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f238c15dfe0 0x7f238c1605f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.710+0000 7f2393215640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0b3d60 con 0x7f238c15dfe0 2024-03-06T07:31:59.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.710+0000 7f2393215640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0b31e0 con 0x7f238c0d22b0 2024-03-06T07:31:59.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.710+0000 7f2393215640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238c0d1c70 con 0x7f238c0b41f0 2024-03-06T07:31:59.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.710+0000 7f23917b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f238c15dfe0 0x7f238c1605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.710+0000 7f23917b2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f238c15dfe0 0x7f238c1605f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54246/0 (socket says 172.21.15.59:54246) 2024-03-06T07:31:59.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.710+0000 7f23917b2640 1 -- 172.21.15.59:0/4033687413 learned_addr learned my addr 172.21.15.59:0/4033687413 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:31:59.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.711+0000 7f23917b2640 1 -- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f238c0b41f0 msgr2=0x7f238c14d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.711+0000 7f23917b2640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f238c0b41f0 0x7f238c14d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.711+0000 7f23917b2640 1 -- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f238c0d22b0 msgr2=0x7f238c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.711+0000 7f23917b2640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f238c0d22b0 0x7f238c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.711+0000 7f23917b2640 1 -- 172.21.15.59:0/4033687413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c0cb760 con 0x7f238c15dfe0 2024-03-06T07:31:59.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.711+0000 7f23917b2640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f238c15dfe0 0x7f238c1605f0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f238400c7b0 tx=0x7f238400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.711+0000 7f23897fa640 1 -- 172.21.15.59:0/4033687413 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384017070 con 0x7f238c15dfe0 2024-03-06T07:31:59.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.711+0000 7f23897fa640 1 -- 172.21.15.59:0/4033687413 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2384003da0 con 0x7f238c15dfe0 2024-03-06T07:31:59.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.711+0000 7f23897fa640 1 -- 172.21.15.59:0/4033687413 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23840052b0 con 0x7f238c15dfe0 2024-03-06T07:31:59.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.711+0000 7f2393215640 1 -- 172.21.15.59:0/4033687413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c0cba40 con 0x7f238c15dfe0 2024-03-06T07:31:59.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.711+0000 7f2393215640 1 -- 172.21.15.59:0/4033687413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c0cc000 con 0x7f238c15dfe0 2024-03-06T07:31:59.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.712+0000 7f2393215640 1 -- 172.21.15.59:0/4033687413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c04f330 con 0x7f238c15dfe0 2024-03-06T07:31:59.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.712+0000 7f23897fa640 1 -- 172.21.15.59:0/4033687413 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f238401a070 con 0x7f238c15dfe0 2024-03-06T07:31:59.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.713+0000 7f23897fa640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f236c06b4d0 0x7f236c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.713+0000 7f23897fa640 1 -- 172.21.15.59:0/4033687413 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2384089860 con 0x7f238c15dfe0 2024-03-06T07:31:59.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.713+0000 7f2390fb1640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f236c06b4d0 0x7f236c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.714+0000 7f2390fb1640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f236c06b4d0 0x7f236c06d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f237c000f60 tx=0x7f237c007a80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.719 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.717+0000 7f23897fa640 1 -- 172.21.15.59:0/4033687413 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23840586c0 con 0x7f238c15dfe0 2024-03-06T07:32:00.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.998+0000 7f2393215640 1 -- 172.21.15.59:0/4033687413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f238c0747a0 con 0x7f238c15dfe0 2024-03-06T07:32:00.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:31:59.998+0000 7f23897fa640 1 -- 172.21.15.59:0/4033687413 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f2384057fc0 con 0x7f238c15dfe0 2024-03-06T07:32:00.001 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:00.001 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:00.001 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.000+0000 7f2393215640 1 -- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f236c06b4d0 msgr2=0x7f236c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.000+0000 7f2393215640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f236c06b4d0 0x7f236c06d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f237c000f60 tx=0x7f237c007a80 comp rx=0 tx=0).stop 2024-03-06T07:32:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.000+0000 7f2393215640 1 -- 172.21.15.59:0/4033687413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f238c15dfe0 msgr2=0x7f238c1605f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.000+0000 7f2393215640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f238c15dfe0 0x7f238c1605f0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f238400c7b0 tx=0x7f238400cc70 comp rx=0 tx=0).stop 2024-03-06T07:32:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.001+0000 7f2393215640 1 -- 172.21.15.59:0/4033687413 shutdown_connections 2024-03-06T07:32:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.001+0000 7f2393215640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f236c06b4d0 0x7f236c06d990 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.001+0000 7f2393215640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f238c15dfe0 0x7f238c1605f0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.001+0000 7f2393215640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f238c0d22b0 0x7f238c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.001+0000 7f2393215640 1 --2- 172.21.15.59:0/4033687413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f238c0b41f0 0x7f238c14d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.001+0000 7f2393215640 1 -- 172.21.15.59:0/4033687413 >> 172.21.15.59:0/4033687413 conn(0x7f238c0d3cd0 msgr2=0x7f238c06cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.001+0000 7f2393215640 1 -- 172.21.15.59:0/4033687413 shutdown_connections 2024-03-06T07:32:00.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:00.001+0000 7f2393215640 1 -- 172.21.15.59:0/4033687413 wait complete. 2024-03-06T07:32:00.215 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:00.215 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:01.216 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:01.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.348+0000 7f0935777640 1 Processor -- start 2024-03-06T07:32:01.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.348+0000 7f0935777640 1 -- start start 2024-03-06T07:32:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f0935777640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f09300b4170 0x7f09300b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f0935777640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09300d25d0 con 0x7f09300d0a80 2024-03-06T07:32:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f0935777640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09300d2870 con 0x7f09300d4760 2024-03-06T07:32:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f0935777640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09300d4bf0 con 0x7f09300b4170 2024-03-06T07:32:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f092f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f09300d0a80 0x7f09300cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47532/0 (socket says 172.21.15.59:47532) 2024-03-06T07:32:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f092f7fe640 1 -- 172.21.15.59:0/1911551576 learned_addr learned my addr 172.21.15.59:0/1911551576 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f092effd640 1 --2- 172.21.15.59:0/1911551576 >> v2:172.21.15.164:3300/0 conn(0x7f09300b4170 0x7f09300b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f092e7fc640 1 -- 172.21.15.59:0/1911551576 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016497181 0 0) 0x7f09300d25d0 con 0x7f09300d0a80 2024-03-06T07:32:01.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f092e7fc640 1 -- 172.21.15.59:0/1911551576 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f091c003520 con 0x7f09300d0a80 2024-03-06T07:32:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f092effd640 1 -- 172.21.15.59:0/1911551576 >> v1:172.21.15.164:6789/0 conn(0x7f09300d4760 legacy=0x7f09300d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f092e7fc640 1 -- 172.21.15.59:0/1911551576 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 49445257 0 0) 0x7f091c003520 con 0x7f09300d0a80 2024-03-06T07:32:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f092effd640 1 -- 172.21.15.59:0/1911551576 >> v1:172.21.15.59:6789/0 conn(0x7f09300d0a80 legacy=0x7f09300cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.349+0000 7f092effd640 1 -- 172.21.15.59:0/1911551576 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09300d2b50 con 0x7f09300b4170 2024-03-06T07:32:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.350+0000 7f092effd640 1 --2- 172.21.15.59:0/1911551576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09300b4170 0x7f09300b6a90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f09200097a0 tx=0x7f092002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db4e1f185e83405 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.351+0000 7f092e7fc640 1 -- 172.21.15.59:0/1911551576 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920038070 con 0x7f09300b4170 2024-03-06T07:32:01.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.351+0000 7f092e7fc640 1 -- 172.21.15.59:0/1911551576 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0920031500 con 0x7f09300b4170 2024-03-06T07:32:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.351+0000 7f092e7fc640 1 -- 172.21.15.59:0/1911551576 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09280019f0 con 0x7f09300b4170 2024-03-06T07:32:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.351+0000 7f0935777640 1 -- 172.21.15.59:0/1911551576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09300b4170 msgr2=0x7f09300b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.351+0000 7f0935777640 1 --2- 172.21.15.59:0/1911551576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09300b4170 0x7f09300b6a90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f09200097a0 tx=0x7f092002b820 comp rx=0 tx=0).stop 2024-03-06T07:32:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.351+0000 7f0935777640 1 -- 172.21.15.59:0/1911551576 shutdown_connections 2024-03-06T07:32:01.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.351+0000 7f0935777640 1 --2- 172.21.15.59:0/1911551576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09300b4170 0x7f09300b6a90 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.351+0000 7f0935777640 1 -- 172.21.15.59:0/1911551576 >> 172.21.15.59:0/1911551576 conn(0x7f09300d3c40 msgr2=0x7f09300d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.351+0000 7f0935777640 1 -- 172.21.15.59:0/1911551576 shutdown_connections 2024-03-06T07:32:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.351+0000 7f0935777640 1 -- 172.21.15.59:0/1911551576 wait complete. 2024-03-06T07:32:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.352+0000 7f0935777640 1 Processor -- start 2024-03-06T07:32:01.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.352+0000 7f0935777640 1 -- start start 2024-03-06T07:32:01.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.353+0000 7f0935777640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09300b4170 0x7f09300cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.353+0000 7f0935777640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09300d0a80 0x7f09301581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.353+0000 7f0935777640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0930158730 0x7f093014a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.353+0000 7f0935777640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09300b4dd0 con 0x7f0930158730 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.353+0000 7f0935777640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09300ce9e0 con 0x7f09300b4170 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.353+0000 7f0935777640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09300b4f50 con 0x7f09300d0a80 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.353+0000 7f092f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0930158730 0x7f093014a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.353+0000 7f092f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0930158730 0x7f093014a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54256/0 (socket says 172.21.15.59:54256) 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.353+0000 7f092f7fe640 1 -- 172.21.15.59:0/2123272624 learned_addr learned my addr 172.21.15.59:0/2123272624 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.353+0000 7f092effd640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09300b4170 0x7f09300cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.353+0000 7f09267fc640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09300d0a80 0x7f09301581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.354+0000 7f092f7fe640 1 -- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09300d0a80 msgr2=0x7f09301581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.354+0000 7f092f7fe640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09300d0a80 0x7f09301581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.354+0000 7f092f7fe640 1 -- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09300b4170 msgr2=0x7f09300cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.354+0000 7f092f7fe640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09300b4170 0x7f09300cbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.354+0000 7f092f7fe640 1 -- 172.21.15.59:0/2123272624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09300cd940 con 0x7f0930158730 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.354+0000 7f092f7fe640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0930158730 0x7f093014a300 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f0928001890 tx=0x7f092800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.354+0000 7f0927fff640 1 -- 172.21.15.59:0/2123272624 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928012400 con 0x7f0930158730 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.354+0000 7f0927fff640 1 -- 172.21.15.59:0/2123272624 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09280125c0 con 0x7f0930158730 2024-03-06T07:32:01.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.354+0000 7f0935777640 1 -- 172.21.15.59:0/2123272624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09300cc8a0 con 0x7f0930158730 2024-03-06T07:32:01.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.354+0000 7f0927fff640 1 -- 172.21.15.59:0/2123272624 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09280129b0 con 0x7f0930158730 2024-03-06T07:32:01.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.354+0000 7f0935777640 1 -- 172.21.15.59:0/2123272624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09300cce60 con 0x7f0930158730 2024-03-06T07:32:01.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.355+0000 7f0935777640 1 -- 172.21.15.59:0/2123272624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09300b65d0 con 0x7f0930158730 2024-03-06T07:32:01.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.356+0000 7f0927fff640 1 -- 172.21.15.59:0/2123272624 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0928012b50 con 0x7f0930158730 2024-03-06T07:32:01.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.356+0000 7f0927fff640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f091006b4d0 0x7f091006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:01.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.356+0000 7f0927fff640 1 -- 172.21.15.59:0/2123272624 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f092808a980 con 0x7f0930158730 2024-03-06T07:32:01.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.356+0000 7f092effd640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f091006b4d0 0x7f091006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:01.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.357+0000 7f092effd640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f091006b4d0 0x7f091006d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0920035000 tx=0x7f0920008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:01.363 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.360+0000 7f0927fff640 1 -- 172.21.15.59:0/2123272624 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0928059190 con 0x7f0930158730 2024-03-06T07:32:01.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.578+0000 7f0935777640 1 -- 172.21.15.59:0/2123272624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f093004f330 con 0x7f0930158730 2024-03-06T07:32:01.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.578+0000 7f0927fff640 1 -- 172.21.15.59:0/2123272624 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f092805d230 con 0x7f0930158730 2024-03-06T07:32:01.581 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:01.581 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:01.582 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:01.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 -- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f091006b4d0 msgr2=0x7f091006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f091006b4d0 0x7f091006d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0920035000 tx=0x7f0920008030 comp rx=0 tx=0).stop 2024-03-06T07:32:01.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 -- 172.21.15.59:0/2123272624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0930158730 msgr2=0x7f093014a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0930158730 0x7f093014a300 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f0928001890 tx=0x7f092800a430 comp rx=0 tx=0).stop 2024-03-06T07:32:01.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 -- 172.21.15.59:0/2123272624 shutdown_connections 2024-03-06T07:32:01.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f091006b4d0 0x7f091006d990 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0930158730 0x7f093014a300 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09300d0a80 0x7f09301581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 --2- 172.21.15.59:0/2123272624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09300b4170 0x7f09300cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 -- 172.21.15.59:0/2123272624 >> 172.21.15.59:0/2123272624 conn(0x7f09300d3c40 msgr2=0x7f093006b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 -- 172.21.15.59:0/2123272624 shutdown_connections 2024-03-06T07:32:01.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:01.580+0000 7f0935777640 1 -- 172.21.15.59:0/2123272624 wait complete. 2024-03-06T07:32:01.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:01.795 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:02.796 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:02.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.895+0000 7f5a8ba47640 1 Processor -- start 2024-03-06T07:32:02.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.896+0000 7f5a8ba47640 1 -- start start 2024-03-06T07:32:02.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.896+0000 7f5a8ba47640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5a840d2710 0x7f5a840b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.896+0000 7f5a8ba47640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a840b3430 con 0x7f5a840d22b0 2024-03-06T07:32:02.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.896+0000 7f5a8ba47640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a840b3710 con 0x7f5a8415dfe0 2024-03-06T07:32:02.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.896+0000 7f5a8ba47640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a840b31e0 con 0x7f5a840d2710 2024-03-06T07:32:02.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.896+0000 7f5a88fe2640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f5a840d22b0 0x7f5a840b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47544/0 (socket says 172.21.15.59:47544) 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.896+0000 7f5a88fe2640 1 -- 172.21.15.59:0/3256714519 learned_addr learned my addr 172.21.15.59:0/3256714519 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126240002 0 0) 0x7f5a840b3430 con 0x7f5a840d22b0 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a89fe4640 1 --2- 172.21.15.59:0/3256714519 >> v2:172.21.15.164:3300/0 conn(0x7f5a840d2710 0x7f5a840b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c003520 con 0x7f5a840d22b0 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1978735262 0 0) 0x7f5a6c003520 con 0x7f5a840d22b0 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 >> v2:172.21.15.164:3300/0 conn(0x7f5a840d2710 msgr2=0x7f5a840b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 --2- 172.21.15.59:0/3256714519 >> v2:172.21.15.164:3300/0 conn(0x7f5a840d2710 0x7f5a840b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 >> v1:172.21.15.164:6789/0 conn(0x7f5a8415dfe0 legacy=0x7f5a840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a840b39f0 con 0x7f5a840d22b0 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f5a78001670 con 0x7f5a840d22b0 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 >> v1:172.21.15.59:6789/0 conn(0x7f5a840d22b0 legacy=0x7f5a840b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 --2- 172.21.15.59:0/3256714519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a6c0039e0 0x7f5a6c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 --2- 172.21.15.59:0/3256714519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 0x7f5a6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 --2- 172.21.15.59:0/3256714519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a6c0051c0 0x7f5a6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840b31e0 con 0x7f5a6c004800 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c00c090 con 0x7f5a6c0039e0 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c00c290 con 0x7f5a6c0051c0 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a88fe2640 1 --2- 172.21.15.59:0/3256714519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 0x7f5a6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a897e3640 1 --2- 172.21.15.59:0/3256714519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a6c0039e0 0x7f5a6c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.897+0000 7f5a89fe4640 1 --2- 172.21.15.59:0/3256714519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a6c0051c0 0x7f5a6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.898+0000 7f5a88fe2640 1 -- 172.21.15.59:0/3256714519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a6c0051c0 msgr2=0x7f5a6c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.898+0000 7f5a88fe2640 1 --2- 172.21.15.59:0/3256714519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a6c0051c0 0x7f5a6c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.898+0000 7f5a88fe2640 1 -- 172.21.15.59:0/3256714519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a6c0039e0 msgr2=0x7f5a6c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.898+0000 7f5a88fe2640 1 --2- 172.21.15.59:0/3256714519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a6c0039e0 0x7f5a6c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.898+0000 7f5a89fe4640 1 -- 172.21.15.59:0/3256714519 reap_dead start 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.898+0000 7f5a88fe2640 1 -- 172.21.15.59:0/3256714519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c003520 con 0x7f5a6c004800 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.898+0000 7f5a88fe2640 1 --2- 172.21.15.59:0/3256714519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 0x7f5a6c004c80 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800c240 tx=0x7f5a6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dceb50f8467f744 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.898+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78007d10 con 0x7f5a6c004800 2024-03-06T07:32:02.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.899+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a78030070 con 0x7f5a6c004800 2024-03-06T07:32:02.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.899+0000 7f5a83fff640 1 -- 172.21.15.59:0/3256714519 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78005d90 con 0x7f5a6c004800 2024-03-06T07:32:02.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.899+0000 7f5a8ba47640 1 -- 172.21.15.59:0/3256714519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 msgr2=0x7f5a6c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.899+0000 7f5a8ba47640 1 --2- 172.21.15.59:0/3256714519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 0x7f5a6c004c80 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800c240 tx=0x7f5a6c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:02.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.899+0000 7f5a8ba47640 1 -- 172.21.15.59:0/3256714519 shutdown_connections 2024-03-06T07:32:02.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.899+0000 7f5a8ba47640 1 --2- 172.21.15.59:0/3256714519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 0x7f5a6c004c80 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.899+0000 7f5a8ba47640 1 -- 172.21.15.59:0/3256714519 >> 172.21.15.59:0/3256714519 conn(0x7f5a840d3cd0 msgr2=0x7f5a840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.899+0000 7f5a8ba47640 1 -- 172.21.15.59:0/3256714519 shutdown_connections 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.899+0000 7f5a8ba47640 1 -- 172.21.15.59:0/3256714519 wait complete. 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.900+0000 7f5a8ba47640 1 Processor -- start 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.900+0000 7f5a8ba47640 1 -- start start 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.901+0000 7f5a8ba47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 0x7f5a8414c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.901+0000 7f5a8ba47640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a8414c910 0x7f5a841608c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.901+0000 7f5a8ba47640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a84160e00 0x7f5a840b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.901+0000 7f5a8ba47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840d1c70 con 0x7f5a6c004800 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.901+0000 7f5a8ba47640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840b3d60 con 0x7f5a8414c910 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.901+0000 7f5a8ba47640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840d1df0 con 0x7f5a84160e00 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.901+0000 7f5a897e3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 0x7f5a8414c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.901+0000 7f5a897e3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 0x7f5a8414c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54274/0 (socket says 172.21.15.59:54274) 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.901+0000 7f5a897e3640 1 -- 172.21.15.59:0/770890362 learned_addr learned my addr 172.21.15.59:0/770890362 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.901+0000 7f5a88fe2640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a8414c910 0x7f5a841608c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.901+0000 7f5a89fe4640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a84160e00 0x7f5a840b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.902+0000 7f5a897e3640 1 -- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a84160e00 msgr2=0x7f5a840b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.902+0000 7f5a897e3640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a84160e00 0x7f5a840b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.902+0000 7f5a897e3640 1 -- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a8414c910 msgr2=0x7f5a841608c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.902+0000 7f5a897e3640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a8414c910 0x7f5a841608c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.902+0000 7f5a897e3640 1 -- 172.21.15.59:0/770890362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a840caed0 con 0x7f5a6c004800 2024-03-06T07:32:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.902+0000 7f5a897e3640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 0x7f5a8414c160 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00f920 tx=0x7f5a7c00fde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.902+0000 7f5a81ffb640 1 -- 172.21.15.59:0/770890362 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c003b80 con 0x7f5a6c004800 2024-03-06T07:32:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.902+0000 7f5a81ffb640 1 -- 172.21.15.59:0/770890362 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c018070 con 0x7f5a6c004800 2024-03-06T07:32:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.902+0000 7f5a81ffb640 1 -- 172.21.15.59:0/770890362 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00d7f0 con 0x7f5a6c004800 2024-03-06T07:32:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.902+0000 7f5a8ba47640 1 -- 172.21.15.59:0/770890362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a840cb1b0 con 0x7f5a6c004800 2024-03-06T07:32:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.902+0000 7f5a8ba47640 1 -- 172.21.15.59:0/770890362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a840cb770 con 0x7f5a6c004800 2024-03-06T07:32:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.903+0000 7f5a8ba47640 1 -- 172.21.15.59:0/770890362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8404f330 con 0x7f5a6c004800 2024-03-06T07:32:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.903+0000 7f5a81ffb640 1 -- 172.21.15.59:0/770890362 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a7c00d990 con 0x7f5a6c004800 2024-03-06T07:32:02.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.904+0000 7f5a81ffb640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7006b4d0 0x7f5a7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.904+0000 7f5a81ffb640 1 -- 172.21.15.59:0/770890362 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5a7c08a300 con 0x7f5a6c004800 2024-03-06T07:32:02.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.904+0000 7f5a88fe2640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7006b4d0 0x7f5a7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.905+0000 7f5a88fe2640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7006b4d0 0x7f5a7006d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5a78034000 tx=0x7f5a78039040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:02.908+0000 7f5a81ffb640 1 -- 172.21.15.59:0/770890362 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a7c01a310 con 0x7f5a6c004800 2024-03-06T07:32:03.132 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.130+0000 7f5a8ba47640 1 -- 172.21.15.59:0/770890362 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a841611e0 con 0x7f5a6c004800 2024-03-06T07:32:03.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.131+0000 7f5a81ffb640 1 -- 172.21.15.59:0/770890362 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f5a7c0591f0 con 0x7f5a6c004800 2024-03-06T07:32:03.133 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:03.133 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:03.133 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:03.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 -- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7006b4d0 msgr2=0x7f5a7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7006b4d0 0x7f5a7006d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5a78034000 tx=0x7f5a78039040 comp rx=0 tx=0).stop 2024-03-06T07:32:03.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 -- 172.21.15.59:0/770890362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 msgr2=0x7f5a8414c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 0x7f5a8414c160 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00f920 tx=0x7f5a7c00fde0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 -- 172.21.15.59:0/770890362 shutdown_connections 2024-03-06T07:32:03.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a7006b4d0 0x7f5a7006d990 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a84160e00 0x7f5a840b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a8414c910 0x7f5a841608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 --2- 172.21.15.59:0/770890362 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a6c004800 0x7f5a8414c160 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 -- 172.21.15.59:0/770890362 >> 172.21.15.59:0/770890362 conn(0x7f5a840d3cd0 msgr2=0x7f5a840cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 -- 172.21.15.59:0/770890362 shutdown_connections 2024-03-06T07:32:03.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:03.133+0000 7f5a8ba47640 1 -- 172.21.15.59:0/770890362 wait complete. 2024-03-06T07:32:03.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:03.350 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:04.352 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:04.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.454+0000 7f500898c640 1 Processor -- start 2024-03-06T07:32:04.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.454+0000 7f500898c640 1 -- start start 2024-03-06T07:32:04.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.455+0000 7f500898c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f50040cce40 0x7f500415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.455+0000 7f500898c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5004071f40 con 0x7f50040cd7a0 2024-03-06T07:32:04.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.455+0000 7f500898c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5004070a70 con 0x7f50041515e0 2024-03-06T07:32:04.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.455+0000 7f500898c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50040cd2f0 con 0x7f50040cce40 2024-03-06T07:32:04.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.455+0000 7f5001d9b640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f50040cce40 0x7f500415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.455+0000 7f5002d9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f50040cd7a0 0x7f50040cdb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51500/0 (socket says 172.21.15.59:51500) 2024-03-06T07:32:04.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.455+0000 7f5002d9d640 1 -- 172.21.15.59:0/1258733767 learned_addr learned my addr 172.21.15.59:0/1258733767 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:04.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.456+0000 7f5001d9b640 1 -- 172.21.15.59:0/1258733767 >> v1:172.21.15.164:6789/0 conn(0x7f50041515e0 legacy=0x7f5004071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.456+0000 7f5001d9b640 1 -- 172.21.15.59:0/1258733767 >> v1:172.21.15.59:6789/0 conn(0x7f50040cd7a0 legacy=0x7f50040cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.456+0000 7f5001d9b640 1 -- 172.21.15.59:0/1258733767 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5004070d50 con 0x7f50040cce40 2024-03-06T07:32:04.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.456+0000 7f5001d9b640 1 --2- 172.21.15.59:0/1258733767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50040cce40 0x7f500415b430 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800a940 tx=0x7f4ff802cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27f11dc1ee28878 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.456+0000 7f500159a640 1 -- 172.21.15.59:0/1258733767 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff803a070 con 0x7f50040cce40 2024-03-06T07:32:04.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.457+0000 7f500159a640 1 -- 172.21.15.59:0/1258733767 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff802db40 con 0x7f50040cce40 2024-03-06T07:32:04.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.457+0000 7f500159a640 1 -- 172.21.15.59:0/1258733767 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff800b040 con 0x7f50040cce40 2024-03-06T07:32:04.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.457+0000 7f500898c640 1 -- 172.21.15.59:0/1258733767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50040cce40 msgr2=0x7f500415b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.457+0000 7f500898c640 1 --2- 172.21.15.59:0/1258733767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50040cce40 0x7f500415b430 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800a940 tx=0x7f4ff802cd90 comp rx=0 tx=0).stop 2024-03-06T07:32:04.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.457+0000 7f500898c640 1 -- 172.21.15.59:0/1258733767 shutdown_connections 2024-03-06T07:32:04.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.457+0000 7f500898c640 1 --2- 172.21.15.59:0/1258733767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50040cce40 0x7f500415b430 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.457+0000 7f500898c640 1 -- 172.21.15.59:0/1258733767 >> 172.21.15.59:0/1258733767 conn(0x7f50040cf8a0 msgr2=0x7f50040cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.457+0000 7f500898c640 1 -- 172.21.15.59:0/1258733767 shutdown_connections 2024-03-06T07:32:04.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.457+0000 7f500898c640 1 -- 172.21.15.59:0/1258733767 wait complete. 2024-03-06T07:32:04.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.458+0000 7f500898c640 1 Processor -- start 2024-03-06T07:32:04.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.459+0000 7f500898c640 1 -- start start 2024-03-06T07:32:04.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.459+0000 7f500898c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50040cce40 0x7f500413f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.459+0000 7f500898c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50040cd7a0 0x7f500415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.459+0000 7f500898c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50041515e0 0x7f500415cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.459+0000 7f500898c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50040cc3a0 con 0x7f50040cd7a0 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.459+0000 7f500898c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5004071310 con 0x7f50040cce40 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.459+0000 7f500898c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50040cc520 con 0x7f50041515e0 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.459+0000 7f5001d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50040cd7a0 0x7f500415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.459+0000 7f5001d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50040cd7a0 0x7f500415ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51408/0 (socket says 172.21.15.59:51408) 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.459+0000 7f5001d9b640 1 -- 172.21.15.59:0/130120850 learned_addr learned my addr 172.21.15.59:0/130120850 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f5001d9b640 1 -- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50041515e0 msgr2=0x7f500415cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f5001d9b640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50041515e0 0x7f500415cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f5001d9b640 1 -- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50040cce40 msgr2=0x7f500413f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f500259c640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50040cce40 0x7f500413f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f5001d9b640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50040cce40 0x7f500413f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f5001d9b640 1 -- 172.21.15.59:0/130120850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50041552e0 con 0x7f50040cd7a0 2024-03-06T07:32:04.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f500259c640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50040cce40 0x7f500413f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:04.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f5001d9b640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50040cd7a0 0x7f500415ca60 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8000c00 tx=0x7f4ff80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f4ff2ffd640 1 -- 172.21.15.59:0/130120850 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8042020 con 0x7f50040cd7a0 2024-03-06T07:32:04.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f500898c640 1 -- 172.21.15.59:0/130120850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500415d5d0 con 0x7f50040cd7a0 2024-03-06T07:32:04.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f4ff2ffd640 1 -- 172.21.15.59:0/130120850 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff803e410 con 0x7f50040cd7a0 2024-03-06T07:32:04.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f4ff2ffd640 1 -- 172.21.15.59:0/130120850 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff803a040 con 0x7f50040cd7a0 2024-03-06T07:32:04.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.460+0000 7f500898c640 1 -- 172.21.15.59:0/130120850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500415db90 con 0x7f50040cd7a0 2024-03-06T07:32:04.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.461+0000 7f500898c640 1 -- 172.21.15.59:0/130120850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5004068840 con 0x7f50040cd7a0 2024-03-06T07:32:04.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.462+0000 7f4ff2ffd640 1 -- 172.21.15.59:0/130120850 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ff8045020 con 0x7f50040cd7a0 2024-03-06T07:32:04.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.462+0000 7f4ff2ffd640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe406b500 0x7f4fe406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.462+0000 7f4ff2ffd640 1 -- 172.21.15.59:0/130120850 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4ff80ac0d0 con 0x7f50040cd7a0 2024-03-06T07:32:04.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.462+0000 7f500259c640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe406b500 0x7f4fe406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.463+0000 7f500259c640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe406b500 0x7f4fe406d9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400a810 tx=0x7f4ff4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.466+0000 7f4ff2ffd640 1 -- 172.21.15.59:0/130120850 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ff804c030 con 0x7f50040cd7a0 2024-03-06T07:32:04.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.684+0000 7f500898c640 1 -- 172.21.15.59:0/130120850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f500404f330 con 0x7f50040cd7a0 2024-03-06T07:32:04.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.684+0000 7f4ff2ffd640 1 -- 172.21.15.59:0/130120850 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f4ff807ae20 con 0x7f50040cd7a0 2024-03-06T07:32:04.686 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:04.686 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:04.687 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:04.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 -- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe406b500 msgr2=0x7f4fe406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe406b500 0x7f4fe406d9c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400a810 tx=0x7f4ff4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:04.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 -- 172.21.15.59:0/130120850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50040cd7a0 msgr2=0x7f500415ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50040cd7a0 0x7f500415ca60 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8000c00 tx=0x7f4ff80046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 -- 172.21.15.59:0/130120850 shutdown_connections 2024-03-06T07:32:04.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fe406b500 0x7f4fe406d9c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50041515e0 0x7f500415cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50040cd7a0 0x7f500415ca60 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 --2- 172.21.15.59:0/130120850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50040cce40 0x7f500413f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 -- 172.21.15.59:0/130120850 >> 172.21.15.59:0/130120850 conn(0x7f50040cf8a0 msgr2=0x7f500414f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 -- 172.21.15.59:0/130120850 shutdown_connections 2024-03-06T07:32:04.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:04.686+0000 7f500898c640 1 -- 172.21.15.59:0/130120850 wait complete. 2024-03-06T07:32:04.901 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:04.901 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:05.902 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:06.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.002+0000 7fdd1cfeb640 1 Processor -- start 2024-03-06T07:32:06.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.002+0000 7fdd1cfeb640 1 -- start start 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.002+0000 7fdd1cfeb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd1815bdd0 0x7fdd180c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.002+0000 7fdd1cfeb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdd1815fa10 0x7fdd180ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.002+0000 7fdd1cfeb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd180cebe0 con 0x7fdd180d16e0 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.002+0000 7fdd1cfeb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd180ceec0 con 0x7fdd1815fa10 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.002+0000 7fdd1cfeb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd180cf1a0 con 0x7fdd1815bdd0 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.003+0000 7fdd0ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdd1815fa10 0x7fdd180ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.003+0000 7fdd0ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdd1815fa10 0x7fdd180ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51412/0 (socket says 172.21.15.59:51412) 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.003+0000 7fdd0ffff640 1 -- 172.21.15.59:0/1324346535 learned_addr learned my addr 172.21.15.59:0/1324346535 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.003+0000 7fdd16d9d640 1 --1- 172.21.15.59:0/1324346535 >> v1:172.21.15.59:6789/0 conn(0x7fdd180d16e0 0x7fdd180d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51504/0 (socket says 172.21.15.59:51504) 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.003+0000 7fdd1659c640 1 --2- 172.21.15.59:0/1324346535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd1815bdd0 0x7fdd180c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.003+0000 7fdd0ffff640 1 -- 172.21.15.59:0/1324346535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd1815bdd0 msgr2=0x7fdd180c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.003+0000 7fdd0ffff640 1 --2- 172.21.15.59:0/1324346535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd1815bdd0 0x7fdd180c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.003+0000 7fdd0ffff640 1 -- 172.21.15.59:0/1324346535 >> v1:172.21.15.59:6789/0 conn(0x7fdd180d16e0 legacy=0x7fdd180d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.003+0000 7fdd15d9b640 1 -- 172.21.15.59:0/1324346535 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632902352 0 0) 0x7fdd180cebe0 con 0x7fdd180d16e0 2024-03-06T07:32:06.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.003+0000 7fdd0ffff640 1 -- 172.21.15.59:0/1324346535 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd180cc8e0 con 0x7fdd1815fa10 2024-03-06T07:32:06.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.004+0000 7fdd0ffff640 1 --2- 172.21.15.59:0/1324346535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd1815fa10 0x7fdd180ce270 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009870 tx=0x7fdd0002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7588b5e0eecd08d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.004+0000 7fdd15d9b640 1 -- 172.21.15.59:0/1324346535 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00035020 con 0x7fdd1815fa10 2024-03-06T07:32:06.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.004+0000 7fdd15d9b640 1 -- 172.21.15.59:0/1324346535 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd00031640 con 0x7fdd1815fa10 2024-03-06T07:32:06.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.004+0000 7fdd15d9b640 1 -- 172.21.15.59:0/1324346535 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0002d040 con 0x7fdd1815fa10 2024-03-06T07:32:06.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.004+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/1324346535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd1815fa10 msgr2=0x7fdd180ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.004+0000 7fdd1cfeb640 1 --2- 172.21.15.59:0/1324346535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd1815fa10 0x7fdd180ce270 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009870 tx=0x7fdd0002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.004+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/1324346535 shutdown_connections 2024-03-06T07:32:06.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.004+0000 7fdd1cfeb640 1 --2- 172.21.15.59:0/1324346535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd1815fa10 0x7fdd180ce270 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.004+0000 7fdd1cfeb640 1 --2- 172.21.15.59:0/1324346535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd1815bdd0 0x7fdd180c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.004+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/1324346535 >> 172.21.15.59:0/1324346535 conn(0x7fdd1809a9f0 msgr2=0x7fdd1809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.005+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/1324346535 shutdown_connections 2024-03-06T07:32:06.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.005+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/1324346535 wait complete. 2024-03-06T07:32:06.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.006+0000 7fdd1cfeb640 1 Processor -- start 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.006+0000 7fdd1cfeb640 1 -- start start 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd1cfeb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd180d16e0 0x7fdd180b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd1cfeb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd1815bdd0 0x7fdd181f4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd1cfeb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd1815fa10 0x7fdd1815a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd1cfeb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1815c1b0 con 0x7fdd1815fa10 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd1cfeb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd180cc5d0 con 0x7fdd1815bdd0 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd1cfeb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd180ce7b0 con 0x7fdd180d16e0 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd16d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd1815fa10 0x7fdd1815a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd16d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd1815fa10 0x7fdd1815a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51414/0 (socket says 172.21.15.59:51414) 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd16d9d640 1 -- 172.21.15.59:0/7759632 learned_addr learned my addr 172.21.15.59:0/7759632 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd1659c640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd180d16e0 0x7fdd180b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd0ffff640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd1815bdd0 0x7fdd181f4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd16d9d640 1 -- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd180d16e0 msgr2=0x7fdd180b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd16d9d640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd180d16e0 0x7fdd180b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd16d9d640 1 -- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd1815bdd0 msgr2=0x7fdd181f4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd16d9d640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd1815bdd0 0x7fdd181f4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd16d9d640 1 -- 172.21.15.59:0/7759632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd180b5450 con 0x7fdd1815fa10 2024-03-06T07:32:06.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd0ffff640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd1815bdd0 0x7fdd181f4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:06.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd1659c640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd180d16e0 0x7fdd180b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:06.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd16d9d640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd1815fa10 0x7fdd1815a700 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000c7d0 tx=0x7fdd1000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd0effd640 1 -- 172.21.15.59:0/7759632 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10018070 con 0x7fdd1815fa10 2024-03-06T07:32:06.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd0effd640 1 -- 172.21.15.59:0/7759632 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1000a640 con 0x7fdd1815fa10 2024-03-06T07:32:06.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.007+0000 7fdd0effd640 1 -- 172.21.15.59:0/7759632 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1000aa70 con 0x7fdd1815fa10 2024-03-06T07:32:06.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.008+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/7759632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1806c1f0 con 0x7fdd1815fa10 2024-03-06T07:32:06.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.008+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/7759632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1806c7b0 con 0x7fdd1815fa10 2024-03-06T07:32:06.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.009+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/7759632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1804f330 con 0x7fdd1815fa10 2024-03-06T07:32:06.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.009+0000 7fdd0effd640 1 -- 172.21.15.59:0/7759632 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd1000ac10 con 0x7fdd1815fa10 2024-03-06T07:32:06.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.010+0000 7fdd0effd640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcf806b4d0 0x7fdcf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.010+0000 7fdd0effd640 1 -- 172.21.15.59:0/7759632 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdd10089790 con 0x7fdd1815fa10 2024-03-06T07:32:06.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.010+0000 7fdd1659c640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcf806b4d0 0x7fdcf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.010+0000 7fdd1659c640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcf806b4d0 0x7fdcf806d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fdd08006fd0 tx=0x7fdd08008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.014+0000 7fdd0effd640 1 -- 172.21.15.59:0/7759632 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd10058680 con 0x7fdd1815fa10 2024-03-06T07:32:06.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.234+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/7759632 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd180b7a40 con 0x7fdd1815fa10 2024-03-06T07:32:06.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.235+0000 7fdd0effd640 1 -- 172.21.15.59:0/7759632 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fdd10057f80 con 0x7fdd1815fa10 2024-03-06T07:32:06.237 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:06.237 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:06.237 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:06.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcf806b4d0 msgr2=0x7fdcf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcf806b4d0 0x7fdcf806d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fdd08006fd0 tx=0x7fdd08008040 comp rx=0 tx=0).stop 2024-03-06T07:32:06.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/7759632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd1815fa10 msgr2=0x7fdd1815a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd1815fa10 0x7fdd1815a700 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000c7d0 tx=0x7fdd1000cc90 comp rx=0 tx=0).stop 2024-03-06T07:32:06.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/7759632 shutdown_connections 2024-03-06T07:32:06.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdcf806b4d0 0x7fdcf806d990 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.239 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd1815fa10 0x7fdd1815a700 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd1815bdd0 0x7fdd181f4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 --2- 172.21.15.59:0/7759632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd180d16e0 0x7fdd180b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/7759632 >> 172.21.15.59:0/7759632 conn(0x7fdd1809a9f0 msgr2=0x7fdd180c76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/7759632 shutdown_connections 2024-03-06T07:32:06.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:06.237+0000 7fdd1cfeb640 1 -- 172.21.15.59:0/7759632 wait complete. 2024-03-06T07:32:06.450 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:06.451 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:07.452 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:07.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.551+0000 7f7f63c6b640 1 Processor -- start 2024-03-06T07:32:07.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.551+0000 7f7f63c6b640 1 -- start start 2024-03-06T07:32:07.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.552+0000 7f7f63c6b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f5c15dfe0 0x7f7f5c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.552+0000 7f7f63c6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 0x7f7f5c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.552+0000 7f7f63c6b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c0b3430 con 0x7f7f5c0cf830 2024-03-06T07:32:07.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.552+0000 7f7f63c6b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5c0b31e0 con 0x7f7f5c15dfe0 2024-03-06T07:32:07.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.552+0000 7f7f63c6b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0b3d60 con 0x7f7f5c0d29c0 2024-03-06T07:32:07.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.553+0000 7f7f61a07640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f5c15dfe0 0x7f7f5c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.553+0000 7f7f62208640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 0x7f7f5c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.553+0000 7f7f61206640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7f5c0cf830 0x7f7f5c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:56520/0 (socket says 172.21.15.59:56520) 2024-03-06T07:32:07.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.553+0000 7f7f62208640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 0x7f7f5c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:60702/0 (socket says 172.21.15.59:60702) 2024-03-06T07:32:07.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.553+0000 7f7f61206640 1 -- 172.21.15.59:0/3590614317 learned_addr learned my addr 172.21.15.59:0/3590614317 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:07.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.553+0000 7f7f61a07640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f5c15dfe0 0x7f7f5c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:59770/0 (socket says 172.21.15.59:59770) 2024-03-06T07:32:07.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.553+0000 7f7f61a07640 1 -- 172.21.15.59:0/3590614317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 msgr2=0x7f7f5c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.553+0000 7f7f61a07640 1 --2- 172.21.15.59:0/3590614317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 0x7f7f5c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.553+0000 7f7f61a07640 1 -- 172.21.15.59:0/3590614317 >> v1:172.21.15.164:6789/0 conn(0x7f7f5c0cf830 legacy=0x7f7f5c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.553+0000 7f7f61a07640 1 -- 172.21.15.59:0/3590614317 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c0b3710 con 0x7f7f5c15dfe0 2024-03-06T07:32:07.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.553+0000 7f7f62208640 1 --2- 172.21.15.59:0/3590614317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 0x7f7f5c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:07.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.554+0000 7f7f61a07640 1 --2- 172.21.15.59:0/3590614317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5c15dfe0 0x7f7f5c0cf2f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7f580097a0 tx=0x7f7f5802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a4ff2b724507c99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.554+0000 7f7f60a05640 1 -- 172.21.15.59:0/3590614317 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f58035020 con 0x7f7f5c15dfe0 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.554+0000 7f7f60a05640 1 -- 172.21.15.59:0/3590614317 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f58031830 con 0x7f7f5c15dfe0 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.554+0000 7f7f60a05640 1 -- 172.21.15.59:0/3590614317 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f58031c00 con 0x7f7f5c15dfe0 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.554+0000 7f7f63c6b640 1 -- 172.21.15.59:0/3590614317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5c15dfe0 msgr2=0x7f7f5c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.554+0000 7f7f63c6b640 1 --2- 172.21.15.59:0/3590614317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5c15dfe0 0x7f7f5c0cf2f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7f580097a0 tx=0x7f7f5802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.555+0000 7f7f63c6b640 1 -- 172.21.15.59:0/3590614317 shutdown_connections 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.555+0000 7f7f63c6b640 1 --2- 172.21.15.59:0/3590614317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 0x7f7f5c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.555+0000 7f7f63c6b640 1 --2- 172.21.15.59:0/3590614317 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5c15dfe0 0x7f7f5c0cf2f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.555+0000 7f7f63c6b640 1 -- 172.21.15.59:0/3590614317 >> 172.21.15.59:0/3590614317 conn(0x7f7f5c0d3cd0 msgr2=0x7f7f5c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.555+0000 7f7f63c6b640 1 -- 172.21.15.59:0/3590614317 shutdown_connections 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.555+0000 7f7f63c6b640 1 -- 172.21.15.59:0/3590614317 wait complete. 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.556+0000 7f7f63c6b640 1 Processor -- start 2024-03-06T07:32:07.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.556+0000 7f7f63c6b640 1 -- start start 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.556+0000 7f7f63c6b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5c0cf830 0x7f7f5c150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.556+0000 7f7f63c6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 0x7f7f5c09f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.556+0000 7f7f63c6b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5c15dfe0 0x7f7f5c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.556+0000 7f7f63c6b640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0d1df0 con 0x7f7f5c0cf830 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.556+0000 7f7f63c6b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0d1c70 con 0x7f7f5c15dfe0 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.556+0000 7f7f63c6b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c15e3c0 con 0x7f7f5c0d29c0 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f61a07640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5c0cf830 0x7f7f5c150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f61a07640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5c0cf830 0x7f7f5c150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51420/0 (socket says 172.21.15.59:51420) 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f61a07640 1 -- 172.21.15.59:0/1734351413 learned_addr learned my addr 172.21.15.59:0/1734351413 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f61206640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 0x7f7f5c09f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f62208640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5c15dfe0 0x7f7f5c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f61a07640 1 -- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 msgr2=0x7f7f5c09f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f61a07640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 0x7f7f5c09f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f61a07640 1 -- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5c15dfe0 msgr2=0x7f7f5c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f61a07640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5c15dfe0 0x7f7f5c0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f61a07640 1 -- 172.21.15.59:0/1734351413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c0cb9b0 con 0x7f7f5c0cf830 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f61a07640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5c0cf830 0x7f7f5c150000 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f7f58034040 tx=0x7f7f580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f527fc640 1 -- 172.21.15.59:0/1734351413 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f58040070 con 0x7f7f5c0cf830 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f527fc640 1 -- 172.21.15.59:0/1734351413 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5803a850 con 0x7f7f5c0cf830 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f527fc640 1 -- 172.21.15.59:0/1734351413 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f58035040 con 0x7f7f5c0cf830 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f63c6b640 1 -- 172.21.15.59:0/1734351413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c0cbc90 con 0x7f7f5c0cf830 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.557+0000 7f7f63c6b640 1 -- 172.21.15.59:0/1734351413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c0cc250 con 0x7f7f5c0cf830 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.558+0000 7f7f63c6b640 1 -- 172.21.15.59:0/1734351413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5c04f330 con 0x7f7f5c0cf830 2024-03-06T07:32:07.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.559+0000 7f7f527fc640 1 -- 172.21.15.59:0/1734351413 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f58038020 con 0x7f7f5c0cf830 2024-03-06T07:32:07.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.559+0000 7f7f527fc640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f4006b500 0x7f7f4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.560+0000 7f7f527fc640 1 -- 172.21.15.59:0/1734351413 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7f580ab9c0 con 0x7f7f5c0cf830 2024-03-06T07:32:07.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.560+0000 7f7f61206640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f4006b500 0x7f7f4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.560+0000 7f7f61206640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f4006b500 0x7f7f4006d9c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c0caa70 tx=0x7f7f4c007e80 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.564+0000 7f7f527fc640 1 -- 172.21.15.59:0/1734351413 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f580ade20 con 0x7f7f5c0cf830 2024-03-06T07:32:07.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.785+0000 7f7f63c6b640 1 -- 172.21.15.59:0/1734351413 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f5c0b5180 con 0x7f7f5c0cf830 2024-03-06T07:32:07.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.785+0000 7f7f527fc640 1 -- 172.21.15.59:0/1734351413 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f7f58031800 con 0x7f7f5c0cf830 2024-03-06T07:32:07.787 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:07.787 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:07.788 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:07.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 -- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f4006b500 msgr2=0x7f7f4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f4006b500 0x7f7f4006d9c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c0caa70 tx=0x7f7f4c007e80 comp rx=0 tx=0).stop 2024-03-06T07:32:07.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 -- 172.21.15.59:0/1734351413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5c0cf830 msgr2=0x7f7f5c150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5c0cf830 0x7f7f5c150000 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f7f58034040 tx=0x7f7f580046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 -- 172.21.15.59:0/1734351413 shutdown_connections 2024-03-06T07:32:07.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7f4006b500 0x7f7f4006d9c0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f5c15dfe0 0x7f7f5c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f5c0d29c0 0x7f7f5c09f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 --2- 172.21.15.59:0/1734351413 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f5c0cf830 0x7f7f5c150000 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 -- 172.21.15.59:0/1734351413 >> 172.21.15.59:0/1734351413 conn(0x7f7f5c0d3cd0 msgr2=0x7f7f5c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 -- 172.21.15.59:0/1734351413 shutdown_connections 2024-03-06T07:32:07.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:07.787+0000 7f7f63c6b640 1 -- 172.21.15.59:0/1734351413 wait complete. 2024-03-06T07:32:08.001 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:08.001 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:09.002 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:09.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.102+0000 7f57465d6640 1 Processor -- start 2024-03-06T07:32:09.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.102+0000 7f57465d6640 1 -- start start 2024-03-06T07:32:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.103+0000 7f57465d6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f574015fa10 0x7f57400ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.103+0000 7f57465d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57400d16e0 0x7f57400d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.103+0000 7f57465d6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57400cebe0 con 0x7f574015bdd0 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.103+0000 7f57465d6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57400ceec0 con 0x7f574015fa10 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.103+0000 7f57465d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57400cf1a0 con 0x7f57400d16e0 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.103+0000 7f5744b73640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57400d16e0 0x7f57400d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.103+0000 7f573f7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f574015fa10 0x7f57400ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.103+0000 7f5744b73640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57400d16e0 0x7f57400d1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:60724/0 (socket says 172.21.15.59:60724) 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.103+0000 7f5744b73640 1 -- 172.21.15.59:0/2228240994 learned_addr learned my addr 172.21.15.59:0/2228240994 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.104+0000 7f5744b73640 1 -- 172.21.15.59:0/2228240994 >> v2:172.21.15.164:3300/0 conn(0x7f574015fa10 msgr2=0x7f57400ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.104+0000 7f5744b73640 1 --2- 172.21.15.59:0/2228240994 >> v2:172.21.15.164:3300/0 conn(0x7f574015fa10 0x7f57400ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.104+0000 7f5744b73640 1 -- 172.21.15.59:0/2228240994 >> v1:172.21.15.164:6789/0 conn(0x7f574015bdd0 legacy=0x7f57400c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.104+0000 7f5744b73640 1 -- 172.21.15.59:0/2228240994 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57400cc8e0 con 0x7f57400d16e0 2024-03-06T07:32:09.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.104+0000 7f573f7fe640 1 --2- 172.21.15.59:0/2228240994 >> v2:172.21.15.164:3300/0 conn(0x7f574015fa10 0x7f57400ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:09.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.104+0000 7f5744b73640 1 --2- 172.21.15.59:0/2228240994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57400d16e0 0x7f57400d1b10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5738009e50 tx=0x7f573802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73e19390edb99a06 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.105+0000 7f573effd640 1 -- 172.21.15.59:0/2228240994 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573802ca70 con 0x7f57400d16e0 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.105+0000 7f573effd640 1 -- 172.21.15.59:0/2228240994 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573802cc10 con 0x7f57400d16e0 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.105+0000 7f573effd640 1 -- 172.21.15.59:0/2228240994 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57380343f0 con 0x7f57400d16e0 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.105+0000 7f57465d6640 1 -- 172.21.15.59:0/2228240994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57400d16e0 msgr2=0x7f57400d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.105+0000 7f57465d6640 1 --2- 172.21.15.59:0/2228240994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57400d16e0 0x7f57400d1b10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5738009e50 tx=0x7f573802bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.105+0000 7f57465d6640 1 -- 172.21.15.59:0/2228240994 shutdown_connections 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.105+0000 7f57465d6640 1 --2- 172.21.15.59:0/2228240994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57400d16e0 0x7f57400d1b10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.105+0000 7f57465d6640 1 --2- 172.21.15.59:0/2228240994 >> v2:172.21.15.164:3300/0 conn(0x7f574015fa10 0x7f57400ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.105+0000 7f57465d6640 1 -- 172.21.15.59:0/2228240994 >> 172.21.15.59:0/2228240994 conn(0x7f574009a9f0 msgr2=0x7f574009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.105+0000 7f57465d6640 1 -- 172.21.15.59:0/2228240994 shutdown_connections 2024-03-06T07:32:09.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.105+0000 7f57465d6640 1 -- 172.21.15.59:0/2228240994 wait complete. 2024-03-06T07:32:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.106+0000 7f57465d6640 1 Processor -- start 2024-03-06T07:32:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.107+0000 7f57465d6640 1 -- start start 2024-03-06T07:32:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.107+0000 7f57465d6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57400d16e0 0x7f574014ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.107+0000 7f57465d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f574015bdd0 0x7f57400c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.107+0000 7f57465d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f574015fa10 0x7f574015edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.107+0000 7f57465d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574015c1b0 con 0x7f574015bdd0 2024-03-06T07:32:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.107+0000 7f57465d6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57400cc5d0 con 0x7f57400d16e0 2024-03-06T07:32:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.107+0000 7f57465d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57400ce7b0 con 0x7f574015fa10 2024-03-06T07:32:09.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.107+0000 7f573f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f574015bdd0 0x7f57400c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.107+0000 7f573f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f574015bdd0 0x7f57400c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51424/0 (socket says 172.21.15.59:51424) 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.107+0000 7f573f7fe640 1 -- 172.21.15.59:0/436335630 learned_addr learned my addr 172.21.15.59:0/436335630 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f573ffff640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57400d16e0 0x7f574014ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f5744b73640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f574015fa10 0x7f574015edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f573f7fe640 1 -- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f574015fa10 msgr2=0x7f574015edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f573f7fe640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f574015fa10 0x7f574015edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f573f7fe640 1 -- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57400d16e0 msgr2=0x7f574014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f573f7fe640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57400d16e0 0x7f574014ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f573f7fe640 1 -- 172.21.15.59:0/436335630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57400c93c0 con 0x7f574015bdd0 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f5744b73640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f574015fa10 0x7f574015edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f573f7fe640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f574015bdd0 0x7f57400c8b90 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f573400e910 tx=0x7f573400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f573cff9640 1 -- 172.21.15.59:0/436335630 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734017070 con 0x7f574015bdd0 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f573cff9640 1 -- 172.21.15.59:0/436335630 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573400ca00 con 0x7f574015bdd0 2024-03-06T07:32:09.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f57465d6640 1 -- 172.21.15.59:0/436335630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57400b7630 con 0x7f574015bdd0 2024-03-06T07:32:09.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f573cff9640 1 -- 172.21.15.59:0/436335630 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573400cdb0 con 0x7f574015bdd0 2024-03-06T07:32:09.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.108+0000 7f57465d6640 1 -- 172.21.15.59:0/436335630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57400b6b50 con 0x7f574015bdd0 2024-03-06T07:32:09.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.109+0000 7f57465d6640 1 -- 172.21.15.59:0/436335630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574004f330 con 0x7f574015bdd0 2024-03-06T07:32:09.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.110+0000 7f573cff9640 1 -- 172.21.15.59:0/436335630 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f573401a070 con 0x7f574015bdd0 2024-03-06T07:32:09.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.110+0000 7f573cff9640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b4d0 0x7f572c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:09.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.110+0000 7f573cff9640 1 -- 172.21.15.59:0/436335630 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5734089f60 con 0x7f574015bdd0 2024-03-06T07:32:09.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.111+0000 7f573ffff640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b4d0 0x7f572c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:09.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.111+0000 7f573ffff640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b4d0 0x7f572c06d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f574015a010 tx=0x7f5730009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:09.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.115+0000 7f573cff9640 1 -- 172.21.15.59:0/436335630 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5734058800 con 0x7f574015bdd0 2024-03-06T07:32:09.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.333+0000 7f57465d6640 1 -- 172.21.15.59:0/436335630 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57400b7a40 con 0x7f574015bdd0 2024-03-06T07:32:09.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.334+0000 7f573cff9640 1 -- 172.21.15.59:0/436335630 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f573405c8a0 con 0x7f574015bdd0 2024-03-06T07:32:09.336 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:09.336 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:09.336 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 -- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b4d0 msgr2=0x7f572c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b4d0 0x7f572c06d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f574015a010 tx=0x7f5730009040 comp rx=0 tx=0).stop 2024-03-06T07:32:09.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 -- 172.21.15.59:0/436335630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f574015bdd0 msgr2=0x7f57400c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f574015bdd0 0x7f57400c8b90 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f573400e910 tx=0x7f573400edd0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 -- 172.21.15.59:0/436335630 shutdown_connections 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f572c06b4d0 0x7f572c06d990 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f574015fa10 0x7f574015edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f574015bdd0 0x7f57400c8b90 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 --2- 172.21.15.59:0/436335630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57400d16e0 0x7f574014ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 -- 172.21.15.59:0/436335630 >> 172.21.15.59:0/436335630 conn(0x7f574009a9f0 msgr2=0x7f57400cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 -- 172.21.15.59:0/436335630 shutdown_connections 2024-03-06T07:32:09.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:09.336+0000 7f57465d6640 1 -- 172.21.15.59:0/436335630 wait complete. 2024-03-06T07:32:09.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:09.550 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:10.551 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.651+0000 7f49cc9b6640 1 Processor -- start 2024-03-06T07:32:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.651+0000 7f49cc9b6640 1 -- start start 2024-03-06T07:32:10.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49cc9b6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f49c815dfe0 0x7f49c80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49cc9b6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f49c80d29c0 0x7f49c80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49cc9b6640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c80b3430 con 0x7f49c80cf830 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49cc9b6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c80b31e0 con 0x7f49c815dfe0 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49cc9b6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c80b3d60 con 0x7f49c80d29c0 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49c659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f49c815dfe0 0x7f49c80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49c659c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f49c815dfe0 0x7f49c80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51430/0 (socket says 172.21.15.59:51430) 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49c659c640 1 -- 172.21.15.59:0/3249022692 learned_addr learned my addr 172.21.15.59:0/3249022692 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49bffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f49c80cf830 0x7f49c80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51520/0 (socket says 172.21.15.59:51520) 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49c6d9d640 1 --2- 172.21.15.59:0/3249022692 >> v2:172.21.15.164:3300/0 conn(0x7f49c80d29c0 0x7f49c80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49c659c640 1 -- 172.21.15.59:0/3249022692 >> v2:172.21.15.164:3300/0 conn(0x7f49c80d29c0 msgr2=0x7f49c80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49c659c640 1 --2- 172.21.15.59:0/3249022692 >> v2:172.21.15.164:3300/0 conn(0x7f49c80d29c0 0x7f49c80b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49c659c640 1 -- 172.21.15.59:0/3249022692 >> v1:172.21.15.59:6789/0 conn(0x7f49c80cf830 legacy=0x7f49c80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.652+0000 7f49c659c640 1 -- 172.21.15.59:0/3249022692 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c80b3710 con 0x7f49c815dfe0 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.653+0000 7f49c659c640 1 --2- 172.21.15.59:0/3249022692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49c815dfe0 0x7f49c80cf2f0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f49c80b3cd0 tx=0x7f49b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b36bbe4dffae9f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.653+0000 7f49c5d9b640 1 -- 172.21.15.59:0/3249022692 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8034070 con 0x7f49c815dfe0 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.653+0000 7f49c5d9b640 1 -- 172.21.15.59:0/3249022692 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b8030860 con 0x7f49c815dfe0 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.653+0000 7f49c5d9b640 1 -- 172.21.15.59:0/3249022692 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8030c30 con 0x7f49c815dfe0 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.653+0000 7f49cc9b6640 1 -- 172.21.15.59:0/3249022692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49c815dfe0 msgr2=0x7f49c80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.653+0000 7f49cc9b6640 1 --2- 172.21.15.59:0/3249022692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49c815dfe0 0x7f49c80cf2f0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f49c80b3cd0 tx=0x7f49b802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.654+0000 7f49cc9b6640 1 -- 172.21.15.59:0/3249022692 shutdown_connections 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.654+0000 7f49cc9b6640 1 --2- 172.21.15.59:0/3249022692 >> v2:172.21.15.164:3300/0 conn(0x7f49c80d29c0 0x7f49c80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.654+0000 7f49cc9b6640 1 --2- 172.21.15.59:0/3249022692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49c815dfe0 0x7f49c80cf2f0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.654+0000 7f49cc9b6640 1 -- 172.21.15.59:0/3249022692 >> 172.21.15.59:0/3249022692 conn(0x7f49c80d3cd0 msgr2=0x7f49c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.654+0000 7f49cc9b6640 1 -- 172.21.15.59:0/3249022692 shutdown_connections 2024-03-06T07:32:10.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.654+0000 7f49cc9b6640 1 -- 172.21.15.59:0/3249022692 wait complete. 2024-03-06T07:32:10.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.655+0000 7f49cc9b6640 1 Processor -- start 2024-03-06T07:32:10.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.655+0000 7f49cc9b6640 1 -- start start 2024-03-06T07:32:10.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49cc9b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49c80cf830 0x7f49c80b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49cc9b6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49c80d29c0 0x7f49c80cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49cc9b6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49c815dfe0 0x7f49c80cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49cc9b6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80d1df0 con 0x7f49c815dfe0 2024-03-06T07:32:10.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49cc9b6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80d1c70 con 0x7f49c80d29c0 2024-03-06T07:32:10.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49cc9b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c815e3c0 con 0x7f49c80cf830 2024-03-06T07:32:10.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49c6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49c815dfe0 0x7f49c80cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49c6d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49c815dfe0 0x7f49c80cb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51442/0 (socket says 172.21.15.59:51442) 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49c6d9d640 1 -- 172.21.15.59:0/4166913578 learned_addr learned my addr 172.21.15.59:0/4166913578 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49c659c640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49c80cf830 0x7f49c80b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49bffff640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49c80d29c0 0x7f49c80cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49c6d9d640 1 -- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49c80cf830 msgr2=0x7f49c80b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49c6d9d640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49c80cf830 0x7f49c80b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49c6d9d640 1 -- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49c80d29c0 msgr2=0x7f49c80cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49c6d9d640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49c80d29c0 0x7f49c80cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49c6d9d640 1 -- 172.21.15.59:0/4166913578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8160120 con 0x7f49c815dfe0 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49c659c640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49c80cf830 0x7f49c80b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49bffff640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49c80d29c0 0x7f49c80cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49c6d9d640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49c815dfe0 0x7f49c80cb9d0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f49c000cd70 tx=0x7f49c000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49beffd640 1 -- 172.21.15.59:0/4166913578 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0003b90 con 0x7f49c815dfe0 2024-03-06T07:32:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49beffd640 1 -- 172.21.15.59:0/4166913578 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c0003d30 con 0x7f49c815dfe0 2024-03-06T07:32:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.656+0000 7f49cc9b6640 1 -- 172.21.15.59:0/4166913578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c81f3a70 con 0x7f49c815dfe0 2024-03-06T07:32:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.657+0000 7f49beffd640 1 -- 172.21.15.59:0/4166913578 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0004240 con 0x7f49c815dfe0 2024-03-06T07:32:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.657+0000 7f49cc9b6640 1 -- 172.21.15.59:0/4166913578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c81f3d10 con 0x7f49c815dfe0 2024-03-06T07:32:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.657+0000 7f49cc9b6640 1 -- 172.21.15.59:0/4166913578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c804f330 con 0x7f49c815dfe0 2024-03-06T07:32:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.658+0000 7f49beffd640 1 -- 172.21.15.59:0/4166913578 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49c00043e0 con 0x7f49c815dfe0 2024-03-06T07:32:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.659+0000 7f49beffd640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f49a806b4d0 0x7f49a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.659+0000 7f49beffd640 1 -- 172.21.15.59:0/4166913578 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f49c008ad20 con 0x7f49c815dfe0 2024-03-06T07:32:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.659+0000 7f49c659c640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f49a806b4d0 0x7f49a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.660+0000 7f49c659c640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f49a806b4d0 0x7f49a806d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f49b8036000 tx=0x7f49b803b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.663+0000 7f49beffd640 1 -- 172.21.15.59:0/4166913578 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f49c8160120 con 0x7f49c815dfe0 2024-03-06T07:32:10.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.879+0000 7f49cc9b6640 1 -- 172.21.15.59:0/4166913578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49c806c4c0 con 0x7f49c815dfe0 2024-03-06T07:32:10.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.880+0000 7f49beffd640 1 -- 172.21.15.59:0/4166913578 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f49c0059ae0 con 0x7f49c815dfe0 2024-03-06T07:32:10.882 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:10.882 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:10.883 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 -- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f49a806b4d0 msgr2=0x7f49a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f49a806b4d0 0x7f49a806d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f49b8036000 tx=0x7f49b803b040 comp rx=0 tx=0).stop 2024-03-06T07:32:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 -- 172.21.15.59:0/4166913578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49c815dfe0 msgr2=0x7f49c80cb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49c815dfe0 0x7f49c80cb9d0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f49c000cd70 tx=0x7f49c000a660 comp rx=0 tx=0).stop 2024-03-06T07:32:10.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 -- 172.21.15.59:0/4166913578 shutdown_connections 2024-03-06T07:32:10.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f49a806b4d0 0x7f49a806d990 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f49c815dfe0 0x7f49c80cb9d0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49c80d29c0 0x7f49c80cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 --2- 172.21.15.59:0/4166913578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f49c80cf830 0x7f49c80b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 -- 172.21.15.59:0/4166913578 >> 172.21.15.59:0/4166913578 conn(0x7f49c80d3cd0 msgr2=0x7f49c80b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 -- 172.21.15.59:0/4166913578 shutdown_connections 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:10.882+0000 7f49cc9b6640 1 -- 172.21.15.59:0/4166913578 wait complete. 2024-03-06T07:32:11.096 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:11.096 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:12.097 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.200+0000 7fd550b93640 1 Processor -- start 2024-03-06T07:32:12.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.200+0000 7fd550b93640 1 -- start start 2024-03-06T07:32:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd550b93640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c15bdd0 0x7fd54c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd550b93640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd54c0d16e0 0x7fd54c0d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd550b93640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c0cebe0 con 0x7fd54c15fa10 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd550b93640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c0ce980 con 0x7fd54c0d16e0 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd550b93640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0ceec0 con 0x7fd54c15bdd0 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd549d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd54c15fa10 0x7fd54c0ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51534/0 (socket says 172.21.15.59:51534) 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd549d9b640 1 -- 172.21.15.59:0/422789837 learned_addr learned my addr 172.21.15.59:0/422789837 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448545031 0 0) 0x7fd54c0cebe0 con 0x7fd54c15fa10 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54a59c640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c15bdd0 0x7fd54c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd534003520 con 0x7fd54c15fa10 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54ad9d640 1 --2- 172.21.15.59:0/422789837 >> v2:172.21.15.164:3300/0 conn(0x7fd54c0d16e0 0x7fd54c0d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1704466452 0 0) 0x7fd534003520 con 0x7fd54c15fa10 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c15bdd0 msgr2=0x7fd54c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c15bdd0 0x7fd54c0c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 >> v2:172.21.15.164:3300/0 conn(0x7fd54c0d16e0 msgr2=0x7fd54c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 --2- 172.21.15.59:0/422789837 >> v2:172.21.15.164:3300/0 conn(0x7fd54c0d16e0 0x7fd54c0d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0cf1a0 con 0x7fd54c15fa10 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fd540001670 con 0x7fd54c15fa10 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 >> v1:172.21.15.59:6789/0 conn(0x7fd54c15fa10 legacy=0x7fd54c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5340039b0 0x7fd534003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c15bdd0 0x7fd534004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 0x7fd5340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0ce980 con 0x7fd534004d40 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0ceec0 con 0x7fd5340039b0 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.201+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53400b9c0 con 0x7fd54c15bdd0 2024-03-06T07:32:12.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.202+0000 7fd54ad9d640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 0x7fd5340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.202+0000 7fd54a59c640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5340039b0 0x7fd534003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.202+0000 7fd549d9b640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c15bdd0 0x7fd534004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.202+0000 7fd54ad9d640 1 -- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c15bdd0 msgr2=0x7fd534004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.202+0000 7fd54ad9d640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c15bdd0 0x7fd534004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.202+0000 7fd54ad9d640 1 -- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5340039b0 msgr2=0x7fd534003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.202+0000 7fd54ad9d640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5340039b0 0x7fd534003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.202+0000 7fd54ad9d640 1 -- 172.21.15.59:0/422789837 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd534003520 con 0x7fd534004d40 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.202+0000 7fd549d9b640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c15bdd0 0x7fd534004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.203+0000 7fd54ad9d640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 0x7fd5340051c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fd544009fc0 tx=0x7fd534005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec604e2d9c05315 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.203+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544039070 con 0x7fd534004d40 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.203+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54402cc60 con 0x7fd534004d40 2024-03-06T07:32:12.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.203+0000 7fd54959a640 1 -- 172.21.15.59:0/422789837 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544009c10 con 0x7fd534004d40 2024-03-06T07:32:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.203+0000 7fd550b93640 1 -- 172.21.15.59:0/422789837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 msgr2=0x7fd5340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.203+0000 7fd550b93640 1 --2- 172.21.15.59:0/422789837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 0x7fd5340051c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fd544009fc0 tx=0x7fd534005830 comp rx=0 tx=0).stop 2024-03-06T07:32:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.203+0000 7fd54ad9d640 1 -- 172.21.15.59:0/422789837 reap_dead start 2024-03-06T07:32:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.203+0000 7fd550b93640 1 -- 172.21.15.59:0/422789837 shutdown_connections 2024-03-06T07:32:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.203+0000 7fd550b93640 1 -- 172.21.15.59:0/422789837 >> 172.21.15.59:0/422789837 conn(0x7fd54c09a9f0 msgr2=0x7fd54c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.203+0000 7fd550b93640 1 -- 172.21.15.59:0/422789837 shutdown_connections 2024-03-06T07:32:12.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.203+0000 7fd550b93640 1 -- 172.21.15.59:0/422789837 wait complete. 2024-03-06T07:32:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.205+0000 7fd550b93640 1 Processor -- start 2024-03-06T07:32:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.205+0000 7fd550b93640 1 -- start start 2024-03-06T07:32:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.205+0000 7fd550b93640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 0x7fd54c0cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.205+0000 7fd550b93640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd54c1f7870 0x7fd54c155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.205+0000 7fd550b93640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c06fd70 0x7fd54c070220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.205+0000 7fd550b93640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0cc5d0 con 0x7fd534004d40 2024-03-06T07:32:12.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.205+0000 7fd550b93640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c0d2070 con 0x7fd54c1f7870 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.205+0000 7fd550b93640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c15c1b0 con 0x7fd54c06fd70 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd54a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 0x7fd54c0cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd54a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 0x7fd54c0cb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51454/0 (socket says 172.21.15.59:51454) 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd54a59c640 1 -- 172.21.15.59:0/2687715937 learned_addr learned my addr 172.21.15.59:0/2687715937 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd549d9b640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd54c1f7870 0x7fd54c155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd54ad9d640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c06fd70 0x7fd54c070220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd54a59c640 1 -- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c06fd70 msgr2=0x7fd54c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd54a59c640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c06fd70 0x7fd54c070220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd54a59c640 1 -- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd54c1f7870 msgr2=0x7fd54c155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd54a59c640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd54c1f7870 0x7fd54c155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd54a59c640 1 -- 172.21.15.59:0/2687715937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c06e0f0 con 0x7fd534004d40 2024-03-06T07:32:12.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd54ad9d640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c06fd70 0x7fd54c070220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd54a59c640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 0x7fd54c0cb1a0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fd53c011910 tx=0x7fd53c011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd53affd640 1 -- 172.21.15.59:0/2687715937 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c00fa30 con 0x7fd534004d40 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd53affd640 1 -- 172.21.15.59:0/2687715937 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c00fbd0 con 0x7fd534004d40 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd550b93640 1 -- 172.21.15.59:0/2687715937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c1f7cf0 con 0x7fd534004d40 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd53affd640 1 -- 172.21.15.59:0/2687715937 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c014440 con 0x7fd534004d40 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.206+0000 7fd550b93640 1 -- 172.21.15.59:0/2687715937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c0c9890 con 0x7fd534004d40 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.207+0000 7fd550b93640 1 -- 172.21.15.59:0/2687715937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c04f330 con 0x7fd534004d40 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.208+0000 7fd53affd640 1 -- 172.21.15.59:0/2687715937 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd53c012040 con 0x7fd534004d40 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.208+0000 7fd53affd640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd52c06b500 0x7fd52c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.208+0000 7fd53affd640 1 -- 172.21.15.59:0/2687715937 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd53c08a670 con 0x7fd534004d40 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.208+0000 7fd549d9b640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd52c06b500 0x7fd52c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.209+0000 7fd549d9b640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd52c06b500 0x7fd52c06d9c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fd54c1f6dc0 tx=0x7fd540007040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.212+0000 7fd53affd640 1 -- 172.21.15.59:0/2687715937 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd54c06e0f0 con 0x7fd534004d40 2024-03-06T07:32:12.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.431+0000 7fd550b93640 1 -- 172.21.15.59:0/2687715937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd54c0b7a40 con 0x7fd534004d40 2024-03-06T07:32:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.432+0000 7fd53affd640 1 -- 172.21.15.59:0/2687715937 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fd53c019030 con 0x7fd534004d40 2024-03-06T07:32:12.434 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:12.434 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:12.434 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:12.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.433+0000 7fd550b93640 1 -- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd52c06b500 msgr2=0x7fd52c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.433+0000 7fd550b93640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd52c06b500 0x7fd52c06d9c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fd54c1f6dc0 tx=0x7fd540007040 comp rx=0 tx=0).stop 2024-03-06T07:32:12.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.434+0000 7fd550b93640 1 -- 172.21.15.59:0/2687715937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 msgr2=0x7fd54c0cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.434+0000 7fd550b93640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 0x7fd54c0cb1a0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fd53c011910 tx=0x7fd53c011d10 comp rx=0 tx=0).stop 2024-03-06T07:32:12.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.434+0000 7fd550b93640 1 -- 172.21.15.59:0/2687715937 shutdown_connections 2024-03-06T07:32:12.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.434+0000 7fd550b93640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd52c06b500 0x7fd52c06d9c0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.434+0000 7fd550b93640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd54c06fd70 0x7fd54c070220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.434+0000 7fd550b93640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd54c1f7870 0x7fd54c155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.434+0000 7fd550b93640 1 --2- 172.21.15.59:0/2687715937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd534004d40 0x7fd54c0cb1a0 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.434+0000 7fd550b93640 1 -- 172.21.15.59:0/2687715937 >> 172.21.15.59:0/2687715937 conn(0x7fd54c09a9f0 msgr2=0x7fd54c0cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.434+0000 7fd550b93640 1 -- 172.21.15.59:0/2687715937 shutdown_connections 2024-03-06T07:32:12.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:12.434+0000 7fd550b93640 1 -- 172.21.15.59:0/2687715937 wait complete. 2024-03-06T07:32:12.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:12.648 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:13.650 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:13.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.750+0000 7f953373a640 1 Processor -- start 2024-03-06T07:32:13.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.750+0000 7f953373a640 1 -- start start 2024-03-06T07:32:13.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f953373a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f952c15dfe0 0x7f952c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f953373a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f952c0cf830 0x7f952c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f953373a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f952c0d27f0 0x7f952c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f953373a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c0d2c50 con 0x7f952c0cf830 2024-03-06T07:32:13.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f953373a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c0b31e0 con 0x7f952c0d27f0 2024-03-06T07:32:13.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f953373a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0b3d60 con 0x7f952c15dfe0 2024-03-06T07:32:13.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f9530cd5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f952c0cf830 0x7f952c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f9530cd5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f952c0cf830 0x7f952c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43266/0 (socket says 172.21.15.59:43266) 2024-03-06T07:32:13.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f9530cd5640 1 -- 172.21.15.59:0/3622079709 learned_addr learned my addr 172.21.15.59:0/3622079709 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:13.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f95314d6640 1 --2- 172.21.15.59:0/3622079709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f952c15dfe0 0x7f952c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f9531cd7640 1 --2- 172.21.15.59:0/3622079709 >> v2:172.21.15.164:3300/0 conn(0x7f952c0d27f0 0x7f952c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f9530cd5640 1 -- 172.21.15.59:0/3622079709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f952c15dfe0 msgr2=0x7f952c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f9530cd5640 1 --2- 172.21.15.59:0/3622079709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f952c15dfe0 0x7f952c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f9530cd5640 1 -- 172.21.15.59:0/3622079709 >> v2:172.21.15.164:3300/0 conn(0x7f952c0d27f0 msgr2=0x7f952c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f9530cd5640 1 --2- 172.21.15.59:0/3622079709 >> v2:172.21.15.164:3300/0 conn(0x7f952c0d27f0 0x7f952c0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f9530cd5640 1 -- 172.21.15.59:0/3622079709 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c0b3430 con 0x7f952c0cf830 2024-03-06T07:32:13.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f95314d6640 1 --2- 172.21.15.59:0/3622079709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f952c15dfe0 0x7f952c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:13.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.751+0000 7f9531cd7640 1 --2- 172.21.15.59:0/3622079709 >> v2:172.21.15.164:3300/0 conn(0x7f952c0d27f0 0x7f952c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:13.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.752+0000 7f9530cd5640 1 --2- 172.21.15.59:0/3622079709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f952c0cf830 0x7f952c0d22b0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f952c067ad0 tx=0x7f952002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=301537d541b518d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.752+0000 7f951ffff640 1 -- 172.21.15.59:0/3622079709 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9520035020 con 0x7f952c0cf830 2024-03-06T07:32:13.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.752+0000 7f951ffff640 1 -- 172.21.15.59:0/3622079709 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9520031870 con 0x7f952c0cf830 2024-03-06T07:32:13.756 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.752+0000 7f951ffff640 1 -- 172.21.15.59:0/3622079709 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9520031c40 con 0x7f952c0cf830 2024-03-06T07:32:13.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.753+0000 7f953373a640 1 -- 172.21.15.59:0/3622079709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f952c0cf830 msgr2=0x7f952c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.753+0000 7f953373a640 1 --2- 172.21.15.59:0/3622079709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f952c0cf830 0x7f952c0d22b0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f952c067ad0 tx=0x7f952002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:13.757 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.753+0000 7f953373a640 1 -- 172.21.15.59:0/3622079709 shutdown_connections 2024-03-06T07:32:13.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.753+0000 7f953373a640 1 --2- 172.21.15.59:0/3622079709 >> v2:172.21.15.164:3300/0 conn(0x7f952c0d27f0 0x7f952c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.753+0000 7f953373a640 1 --2- 172.21.15.59:0/3622079709 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f952c0cf830 0x7f952c0d22b0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.753+0000 7f953373a640 1 --2- 172.21.15.59:0/3622079709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f952c15dfe0 0x7f952c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.753+0000 7f953373a640 1 -- 172.21.15.59:0/3622079709 >> 172.21.15.59:0/3622079709 conn(0x7f952c0d3cd0 msgr2=0x7f952c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.753+0000 7f953373a640 1 -- 172.21.15.59:0/3622079709 shutdown_connections 2024-03-06T07:32:13.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.753+0000 7f953373a640 1 -- 172.21.15.59:0/3622079709 wait complete. 2024-03-06T07:32:13.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.755+0000 7f953373a640 1 Processor -- start 2024-03-06T07:32:13.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.755+0000 7f953373a640 1 -- start start 2024-03-06T07:32:13.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f953373a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f952c0cf830 0x7f952c15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f953373a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f952c0d27f0 0x7f952c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f953373a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f952c15dfe0 0x7f952c0b6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f953373a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0d1c70 con 0x7f952c15dfe0 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f953373a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0b39f0 con 0x7f952c0d27f0 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f953373a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0d1df0 con 0x7f952c0cf830 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f9531cd7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f952c15dfe0 0x7f952c0b6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f9531cd7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f952c15dfe0 0x7f952c0b6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43272/0 (socket says 172.21.15.59:43272) 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f9531cd7640 1 -- 172.21.15.59:0/479702008 learned_addr learned my addr 172.21.15.59:0/479702008 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f9530cd5640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f952c0d27f0 0x7f952c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f95314d6640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f952c0cf830 0x7f952c15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f9531cd7640 1 -- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f952c0cf830 msgr2=0x7f952c15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f9531cd7640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f952c0cf830 0x7f952c15a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f9531cd7640 1 -- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f952c0d27f0 msgr2=0x7f952c0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f9531cd7640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f952c0d27f0 0x7f952c0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.756+0000 7f9531cd7640 1 -- 172.21.15.59:0/479702008 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c0cd780 con 0x7f952c15dfe0 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.757+0000 7f9531cd7640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f952c15dfe0 0x7f952c0b6fd0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f952400cef0 tx=0x7f952400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.757+0000 7f951dffb640 1 -- 172.21.15.59:0/479702008 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952401a070 con 0x7f952c15dfe0 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.757+0000 7f951dffb640 1 -- 172.21.15.59:0/479702008 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952400d840 con 0x7f952c15dfe0 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.757+0000 7f953373a640 1 -- 172.21.15.59:0/479702008 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c0cd4a0 con 0x7f952c15dfe0 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.757+0000 7f951dffb640 1 -- 172.21.15.59:0/479702008 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952400dc30 con 0x7f952c15dfe0 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.757+0000 7f953373a640 1 -- 172.21.15.59:0/479702008 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c15f3a0 con 0x7f952c15dfe0 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.758+0000 7f953373a640 1 -- 172.21.15.59:0/479702008 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952c04f330 con 0x7f952c15dfe0 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.758+0000 7f951dffb640 1 -- 172.21.15.59:0/479702008 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f952401d070 con 0x7f952c15dfe0 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.759+0000 7f951dffb640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f951006b4d0 0x7f951006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.759+0000 7f951dffb640 1 -- 172.21.15.59:0/479702008 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f952408a8b0 con 0x7f952c15dfe0 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.759+0000 7f95314d6640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f951006b4d0 0x7f951006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.760+0000 7f95314d6640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f951006b4d0 0x7f951006d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f952800aac0 tx=0x7f9528008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.763+0000 7f951dffb640 1 -- 172.21.15.59:0/479702008 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9524059150 con 0x7f952c15dfe0 2024-03-06T07:32:13.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.980+0000 7f953373a640 1 -- 172.21.15.59:0/479702008 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f952c0b0af0 con 0x7f952c15dfe0 2024-03-06T07:32:13.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.981+0000 7f951dffb640 1 -- 172.21.15.59:0/479702008 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f952405d1f0 con 0x7f952c15dfe0 2024-03-06T07:32:13.983 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:13.983 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:13.983 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:13.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 -- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f951006b4d0 msgr2=0x7f951006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f951006b4d0 0x7f951006d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f952800aac0 tx=0x7f9528008040 comp rx=0 tx=0).stop 2024-03-06T07:32:13.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 -- 172.21.15.59:0/479702008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f952c15dfe0 msgr2=0x7f952c0b6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f952c15dfe0 0x7f952c0b6fd0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f952400cef0 tx=0x7f952400a710 comp rx=0 tx=0).stop 2024-03-06T07:32:13.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 -- 172.21.15.59:0/479702008 shutdown_connections 2024-03-06T07:32:13.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f951006b4d0 0x7f951006d990 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f952c15dfe0 0x7f952c0b6fd0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f952c0d27f0 0x7f952c0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 --2- 172.21.15.59:0/479702008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f952c0cf830 0x7f952c15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 -- 172.21.15.59:0/479702008 >> 172.21.15.59:0/479702008 conn(0x7f952c0d3cd0 msgr2=0x7f952c0d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 -- 172.21.15.59:0/479702008 shutdown_connections 2024-03-06T07:32:13.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:13.983+0000 7f953373a640 1 -- 172.21.15.59:0/479702008 wait complete. 2024-03-06T07:32:14.197 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:14.197 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:15.199 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:15.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.298+0000 7f7ccbd8a640 1 Processor -- start 2024-03-06T07:32:15.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.299+0000 7f7ccbd8a640 1 -- start start 2024-03-06T07:32:15.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.300+0000 7f7ccbd8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 0x7f7cc40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.300+0000 7f7ccbd8a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc415f580 con 0x7f7cc415ef60 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.300+0000 7f7ccbd8a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc415b450 con 0x7f7cc415bf30 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.300+0000 7f7ccbd8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40cf3f0 con 0x7f7cc413dc10 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.300+0000 7f7cca327640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7cc415ef60 0x7f7cc40708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37066/0 (socket says 172.21.15.59:37066) 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.300+0000 7f7cca327640 1 -- 172.21.15.59:0/2221383023 learned_addr learned my addr 172.21.15.59:0/2221383023 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.300+0000 7f7cc9b26640 1 --2- 172.21.15.59:0/2221383023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 0x7f7cc40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.300+0000 7f7cc8b24640 1 -- 172.21.15.59:0/2221383023 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112383371 0 0) 0x7f7cc415f580 con 0x7f7cc415ef60 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.300+0000 7f7cc8b24640 1 -- 172.21.15.59:0/2221383023 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb0003520 con 0x7f7cc415ef60 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.301+0000 7f7cc8b24640 1 -- 172.21.15.59:0/2221383023 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864975022 0 0) 0x7f7cc415b450 con 0x7f7cc415bf30 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.301+0000 7f7cc8b24640 1 -- 172.21.15.59:0/2221383023 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc415f580 con 0x7f7cc415bf30 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.301+0000 7f7cc8b24640 1 -- 172.21.15.59:0/2221383023 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2980247961 0 0) 0x7f7cb0003520 con 0x7f7cc415ef60 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.301+0000 7f7cc9b26640 1 -- 172.21.15.59:0/2221383023 >> v1:172.21.15.164:6789/0 conn(0x7f7cc415bf30 legacy=0x7f7cc415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.301+0000 7f7cc9b26640 1 -- 172.21.15.59:0/2221383023 >> v1:172.21.15.59:6789/0 conn(0x7f7cc415ef60 legacy=0x7f7cc40708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.301+0000 7f7cc9b26640 1 -- 172.21.15.59:0/2221383023 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc415b730 con 0x7f7cc413dc10 2024-03-06T07:32:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.301+0000 7f7cc9b26640 1 --2- 172.21.15.59:0/2221383023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 0x7f7cc40b2950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7cc00097d0 tx=0x7f7cc002b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a1fbbed4a57379a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.302+0000 7f7cc8b24640 1 -- 172.21.15.59:0/2221383023 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0037070 con 0x7f7cc413dc10 2024-03-06T07:32:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.302+0000 7f7cc8b24640 1 -- 172.21.15.59:0/2221383023 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc0030510 con 0x7f7cc413dc10 2024-03-06T07:32:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.302+0000 7f7cc8b24640 1 -- 172.21.15.59:0/2221383023 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc001670 con 0x7f7cc413dc10 2024-03-06T07:32:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.302+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/2221383023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 msgr2=0x7f7cc40b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.302+0000 7f7ccbd8a640 1 --2- 172.21.15.59:0/2221383023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 0x7f7cc40b2950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7cc00097d0 tx=0x7f7cc002b770 comp rx=0 tx=0).stop 2024-03-06T07:32:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.302+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/2221383023 shutdown_connections 2024-03-06T07:32:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.302+0000 7f7ccbd8a640 1 --2- 172.21.15.59:0/2221383023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 0x7f7cc40b2950 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.302+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/2221383023 >> 172.21.15.59:0/2221383023 conn(0x7f7cc40cf8a0 msgr2=0x7f7cc40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.302+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/2221383023 shutdown_connections 2024-03-06T07:32:15.305 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.302+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/2221383023 wait complete. 2024-03-06T07:32:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.303+0000 7f7ccbd8a640 1 Processor -- start 2024-03-06T07:32:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.304+0000 7f7ccbd8a640 1 -- start start 2024-03-06T07:32:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.304+0000 7f7ccbd8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 0x7f7cc4153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.304+0000 7f7ccbd8a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cc415bf30 0x7f7cc413e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.304+0000 7f7ccbd8a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cc415ef60 0x7f7cc41f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.304+0000 7f7ccbd8a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40b2e90 con 0x7f7cc415ef60 2024-03-06T07:32:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.304+0000 7f7ccbd8a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc415b210 con 0x7f7cc415bf30 2024-03-06T07:32:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.304+0000 7f7ccbd8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc40b3010 con 0x7f7cc413dc10 2024-03-06T07:32:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.304+0000 7f7cca327640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cc415ef60 0x7f7cc41f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.304+0000 7f7cca327640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cc415ef60 0x7f7cc41f3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43284/0 (socket says 172.21.15.59:43284) 2024-03-06T07:32:15.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.304+0000 7f7cca327640 1 -- 172.21.15.59:0/1171830166 learned_addr learned my addr 172.21.15.59:0/1171830166 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cc9b26640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 0x7f7cc4153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cc9325640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cc415bf30 0x7f7cc413e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cca327640 1 -- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 msgr2=0x7f7cc4153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cca327640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 0x7f7cc4153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cca327640 1 -- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cc415bf30 msgr2=0x7f7cc413e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cca327640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cc415bf30 0x7f7cc413e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cca327640 1 -- 172.21.15.59:0/1171830166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc413c840 con 0x7f7cc415ef60 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cc9b26640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 0x7f7cc4153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cca327640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cc415ef60 0x7f7cc41f3050 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00c7b0 tx=0x7f7cbc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cb67fc640 1 -- 172.21.15.59:0/1171830166 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc017070 con 0x7f7cc415ef60 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cb67fc640 1 -- 172.21.15.59:0/1171830166 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cbc005ad0 con 0x7f7cc415ef60 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7cb67fc640 1 -- 172.21.15.59:0/1171830166 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc007070 con 0x7f7cc415ef60 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/1171830166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc413e7f0 con 0x7f7cc415ef60 2024-03-06T07:32:15.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.305+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/1171830166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc413bfa0 con 0x7f7cc415ef60 2024-03-06T07:32:15.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.306+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/1171830166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc404f330 con 0x7f7cc415ef60 2024-03-06T07:32:15.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.307+0000 7f7cb67fc640 1 -- 172.21.15.59:0/1171830166 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7cbc01a070 con 0x7f7cc415ef60 2024-03-06T07:32:15.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.307+0000 7f7cb67fc640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ca806b500 0x7f7ca806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.310 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.307+0000 7f7cb67fc640 1 -- 172.21.15.59:0/1171830166 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7cbc089750 con 0x7f7cc415ef60 2024-03-06T07:32:15.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.307+0000 7f7cc9b26640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ca806b500 0x7f7ca806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.308+0000 7f7cc9b26640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ca806b500 0x7f7ca806d9c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0034000 tx=0x7f7cc0008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.316+0000 7f7cb67fc640 1 -- 172.21.15.59:0/1171830166 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7cbc0585b0 con 0x7f7cc415ef60 2024-03-06T07:32:15.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.535+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/1171830166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cc4141650 con 0x7f7cc415ef60 2024-03-06T07:32:15.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.535+0000 7f7cb67fc640 1 -- 172.21.15.59:0/1171830166 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f7cbc057eb0 con 0x7f7cc415ef60 2024-03-06T07:32:15.537 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:15.537 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:15.537 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:15.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ca806b500 msgr2=0x7f7ca806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ca806b500 0x7f7ca806d9c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0034000 tx=0x7f7cc0008030 comp rx=0 tx=0).stop 2024-03-06T07:32:15.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/1171830166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cc415ef60 msgr2=0x7f7cc41f3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cc415ef60 0x7f7cc41f3050 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00c7b0 tx=0x7f7cbc00cc70 comp rx=0 tx=0).stop 2024-03-06T07:32:15.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/1171830166 shutdown_connections 2024-03-06T07:32:15.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ca806b500 0x7f7ca806d9c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cc415ef60 0x7f7cc41f3050 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cc415bf30 0x7f7cc413e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 --2- 172.21.15.59:0/1171830166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc413dc10 0x7f7cc4153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/1171830166 >> 172.21.15.59:0/1171830166 conn(0x7f7cc40cf8a0 msgr2=0x7f7cc415ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/1171830166 shutdown_connections 2024-03-06T07:32:15.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:15.537+0000 7f7ccbd8a640 1 -- 172.21.15.59:0/1171830166 wait complete. 2024-03-06T07:32:15.750 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:15.751 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:16.751 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.851+0000 7fbfe170d640 1 Processor -- start 2024-03-06T07:32:16.853 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.851+0000 7fbfe170d640 1 -- start start 2024-03-06T07:32:16.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.852+0000 7fbfe170d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfdc0d4760 0x7fbfdc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.852+0000 7fbfe170d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc0d25d0 con 0x7fbfdc0d0a80 2024-03-06T07:32:16.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.852+0000 7fbfe170d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc0d2870 con 0x7fbfdc0b4170 2024-03-06T07:32:16.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.852+0000 7fbfe170d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0d4bf0 con 0x7fbfdc0d4760 2024-03-06T07:32:16.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.852+0000 7fbfdb7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fbfdc0d0a80 0x7fbfdc0cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37080/0 (socket says 172.21.15.59:37080) 2024-03-06T07:32:16.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.852+0000 7fbfdb7fe640 1 -- 172.21.15.59:0/574385063 learned_addr learned my addr 172.21.15.59:0/574385063 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:16.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.852+0000 7fbfda7fc640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfdc0d4760 0x7fbfdc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.852+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606601301 0 0) 0x7fbfdc0d25d0 con 0x7fbfdc0d0a80 2024-03-06T07:32:16.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.852+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc8003520 con 0x7fbfdc0d0a80 2024-03-06T07:32:16.857 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3533706031 0 0) 0x7fbfc8003520 con 0x7fbfdc0d0a80 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfdc0d4760 msgr2=0x7fbfdc0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfdc0d4760 0x7fbfdc0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 >> v1:172.21.15.164:6789/0 conn(0x7fbfdc0b4170 legacy=0x7fbfdc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc0d2b50 con 0x7fbfdc0d0a80 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fbfcc0019f0 con 0x7fbfdc0d0a80 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 >> v1:172.21.15.59:6789/0 conn(0x7fbfdc0d0a80 legacy=0x7fbfdc0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfc80039e0 0x7fbfc8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfc8004390 0x7fbfc8004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfc8004f80 0x7fbfc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0d4bf0 con 0x7fbfc8004f80 2024-03-06T07:32:16.858 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc800bf00 con 0x7fbfc80039e0 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc800c080 con 0x7fbfc8004390 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfdb7fe640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfc8004f80 0x7fbfc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfdaffd640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfc80039e0 0x7fbfc8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfda7fc640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfc8004390 0x7fbfc8004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfdb7fe640 1 -- 172.21.15.59:0/574385063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfc8004390 msgr2=0x7fbfc8004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfdb7fe640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfc8004390 0x7fbfc8004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfdb7fe640 1 -- 172.21.15.59:0/574385063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfc80039e0 msgr2=0x7fbfc8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfdb7fe640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfc80039e0 0x7fbfc8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.853+0000 7fbfdb7fe640 1 -- 172.21.15.59:0/574385063 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc8003520 con 0x7fbfc8004f80 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.854+0000 7fbfdb7fe640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfc8004f80 0x7fbfc800b9c0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc00a810 tx=0x7fbfc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2516815bbcb3e04a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.854+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc00a430 con 0x7fbfc8004f80 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.854+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc034630 con 0x7fbfc8004f80 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.854+0000 7fbfd9ffb640 1 -- 172.21.15.59:0/574385063 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc034a20 con 0x7fbfc8004f80 2024-03-06T07:32:16.859 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.855+0000 7fbfe170d640 1 -- 172.21.15.59:0/574385063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfc8004f80 msgr2=0x7fbfc800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.855+0000 7fbfe170d640 1 --2- 172.21.15.59:0/574385063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfc8004f80 0x7fbfc800b9c0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc00a810 tx=0x7fbfc8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:16.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.855+0000 7fbfdb7fe640 1 -- 172.21.15.59:0/574385063 reap_dead start 2024-03-06T07:32:16.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.855+0000 7fbfe170d640 1 -- 172.21.15.59:0/574385063 shutdown_connections 2024-03-06T07:32:16.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.855+0000 7fbfe170d640 1 -- 172.21.15.59:0/574385063 >> 172.21.15.59:0/574385063 conn(0x7fbfdc0d3c40 msgr2=0x7fbfdc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.855+0000 7fbfe170d640 1 -- 172.21.15.59:0/574385063 shutdown_connections 2024-03-06T07:32:16.860 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.855+0000 7fbfe170d640 1 -- 172.21.15.59:0/574385063 wait complete. 2024-03-06T07:32:16.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.856+0000 7fbfe170d640 1 Processor -- start 2024-03-06T07:32:16.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.856+0000 7fbfe170d640 1 -- start start 2024-03-06T07:32:16.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfe170d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfc8004f80 0x7fbfdc15edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfe170d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfdc0cda30 0x7fbfdc0c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfe170d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfdc0b90f0 0x7fbfdc0b9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.863 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfe170d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0b4dd0 con 0x7fbfdc0cda30 2024-03-06T07:32:16.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfe170d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0ce9e0 con 0x7fbfdc0b90f0 2024-03-06T07:32:16.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfe170d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0b4f50 con 0x7fbfc8004f80 2024-03-06T07:32:16.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfda7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfdc0cda30 0x7fbfdc0c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfda7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfdc0cda30 0x7fbfdc0c8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43298/0 (socket says 172.21.15.59:43298) 2024-03-06T07:32:16.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfda7fc640 1 -- 172.21.15.59:0/650456059 learned_addr learned my addr 172.21.15.59:0/650456059 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:16.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfdaffd640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfc8004f80 0x7fbfdc15edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfdb7fe640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfdc0b90f0 0x7fbfdc0b9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfda7fc640 1 -- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfc8004f80 msgr2=0x7fbfdc15edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfda7fc640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfc8004f80 0x7fbfdc15edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.864 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfda7fc640 1 -- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfdc0b90f0 msgr2=0x7fbfdc0b9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfda7fc640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfdc0b90f0 0x7fbfdc0b9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfda7fc640 1 -- 172.21.15.59:0/650456059 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc0cc990 con 0x7fbfdc0cda30 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.857+0000 7fbfda7fc640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfdc0cda30 0x7fbfdc0c8610 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40119d0 tx=0x7fbfc4011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.858+0000 7fbfd37fe640 1 -- 172.21.15.59:0/650456059 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc400faf0 con 0x7fbfdc0cda30 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.858+0000 7fbfd37fe640 1 -- 172.21.15.59:0/650456059 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc400fc90 con 0x7fbfdc0cda30 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.858+0000 7fbfe170d640 1 -- 172.21.15.59:0/650456059 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc0ccc70 con 0x7fbfdc0cda30 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.858+0000 7fbfd37fe640 1 -- 172.21.15.59:0/650456059 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4014440 con 0x7fbfdc0cda30 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.858+0000 7fbfe170d640 1 -- 172.21.15.59:0/650456059 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc0cd230 con 0x7fbfdc0cda30 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.859+0000 7fbfe170d640 1 -- 172.21.15.59:0/650456059 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc04f330 con 0x7fbfdc0cda30 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.859+0000 7fbfd37fe640 1 -- 172.21.15.59:0/650456059 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfc4012040 con 0x7fbfdc0cda30 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.860+0000 7fbfd37fe640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.865 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.860+0000 7fbfd37fe640 1 -- 172.21.15.59:0/650456059 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbfc408a890 con 0x7fbfdc0cda30 2024-03-06T07:32:16.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.860+0000 7fbfdaffd640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.861+0000 7fbfdaffd640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40035f0 tx=0x7fbfd4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.866 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:16.864+0000 7fbfd37fe640 1 -- 172.21.15.59:0/650456059 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfc4019030 con 0x7fbfdc0cda30 2024-03-06T07:32:17.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.083+0000 7fbfe170d640 1 -- 172.21.15.59:0/650456059 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfdc0733b0 con 0x7fbfdc0cda30 2024-03-06T07:32:17.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.084+0000 7fbfd37fe640 1 -- 172.21.15.59:0/650456059 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fbfc40596f0 con 0x7fbfdc0cda30 2024-03-06T07:32:17.086 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:17.086 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:17.086 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:17.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.085+0000 7fbfe170d640 1 -- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfbc06b4d0 msgr2=0x7fbfbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.085+0000 7fbfe170d640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40035f0 tx=0x7fbfd4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:17.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.085+0000 7fbfe170d640 1 -- 172.21.15.59:0/650456059 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfdc0cda30 msgr2=0x7fbfdc0c8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.085+0000 7fbfe170d640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfdc0cda30 0x7fbfdc0c8610 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40119d0 tx=0x7fbfc4011e90 comp rx=0 tx=0).stop 2024-03-06T07:32:17.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.085+0000 7fbfe170d640 1 -- 172.21.15.59:0/650456059 shutdown_connections 2024-03-06T07:32:17.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.085+0000 7fbfe170d640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbfbc06b4d0 0x7fbfbc06d990 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.085+0000 7fbfe170d640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfdc0b90f0 0x7fbfdc0b9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.085+0000 7fbfe170d640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbfdc0cda30 0x7fbfdc0c8610 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.085+0000 7fbfe170d640 1 --2- 172.21.15.59:0/650456059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfc8004f80 0x7fbfdc15edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.085+0000 7fbfe170d640 1 -- 172.21.15.59:0/650456059 >> 172.21.15.59:0/650456059 conn(0x7fbfdc0d3c40 msgr2=0x7fbfdc0b7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.086+0000 7fbfe170d640 1 -- 172.21.15.59:0/650456059 shutdown_connections 2024-03-06T07:32:17.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:17.086+0000 7fbfe170d640 1 -- 172.21.15.59:0/650456059 wait complete. 2024-03-06T07:32:17.300 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:17.300 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:18.301 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:18.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.400+0000 7f77361ae640 1 Processor -- start 2024-03-06T07:32:18.402 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.400+0000 7f77361ae640 1 -- start start 2024-03-06T07:32:18.403 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.401+0000 7f77361ae640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f77300d22b0 0x7f77300d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.401+0000 7f77361ae640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f77300b41f0 0x7f77300b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.401+0000 7f77361ae640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77300b3430 con 0x7f773015dfe0 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.401+0000 7f77361ae640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77300d2c50 con 0x7f77300b41f0 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.401+0000 7f77361ae640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77300b31e0 con 0x7f77300d22b0 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.401+0000 7f772ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f77300b41f0 0x7f77300b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.401+0000 7f772ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f77300b41f0 0x7f77300b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43310/0 (socket says 172.21.15.59:43310) 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.401+0000 7f772ffff640 1 -- 172.21.15.59:0/2249857093 learned_addr learned my addr 172.21.15.59:0/2249857093 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.401+0000 7f772effd640 1 --2- 172.21.15.59:0/2249857093 >> v2:172.21.15.164:3300/0 conn(0x7f77300d22b0 0x7f77300d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.402+0000 7f772ffff640 1 -- 172.21.15.59:0/2249857093 >> v2:172.21.15.164:3300/0 conn(0x7f77300d22b0 msgr2=0x7f77300d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.402+0000 7f772ffff640 1 --2- 172.21.15.59:0/2249857093 >> v2:172.21.15.164:3300/0 conn(0x7f77300d22b0 0x7f77300d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.402+0000 7f772ffff640 1 -- 172.21.15.59:0/2249857093 >> v1:172.21.15.164:6789/0 conn(0x7f773015dfe0 legacy=0x7f77300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.404 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.402+0000 7f772ffff640 1 -- 172.21.15.59:0/2249857093 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77300b3710 con 0x7f77300b41f0 2024-03-06T07:32:18.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.402+0000 7f772ffff640 1 --2- 172.21.15.59:0/2249857093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77300b41f0 0x7f77300b4680 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f7728009e50 tx=0x7f772802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88909e117ae5c3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.403+0000 7f772e7fc640 1 -- 172.21.15.59:0/2249857093 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772802cb20 con 0x7f77300b41f0 2024-03-06T07:32:18.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.403+0000 7f772e7fc640 1 -- 172.21.15.59:0/2249857093 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772802ccc0 con 0x7f77300b41f0 2024-03-06T07:32:18.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.403+0000 7f772e7fc640 1 -- 172.21.15.59:0/2249857093 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7728034610 con 0x7f77300b41f0 2024-03-06T07:32:18.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.403+0000 7f77361ae640 1 -- 172.21.15.59:0/2249857093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77300b41f0 msgr2=0x7f77300b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.403+0000 7f77361ae640 1 --2- 172.21.15.59:0/2249857093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77300b41f0 0x7f77300b4680 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f7728009e50 tx=0x7f772802bf80 comp rx=0 tx=0).stop 2024-03-06T07:32:18.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.403+0000 7f77361ae640 1 -- 172.21.15.59:0/2249857093 shutdown_connections 2024-03-06T07:32:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.403+0000 7f77361ae640 1 --2- 172.21.15.59:0/2249857093 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77300b41f0 0x7f77300b4680 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.403+0000 7f77361ae640 1 --2- 172.21.15.59:0/2249857093 >> v2:172.21.15.164:3300/0 conn(0x7f77300d22b0 0x7f77300d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.403+0000 7f77361ae640 1 -- 172.21.15.59:0/2249857093 >> 172.21.15.59:0/2249857093 conn(0x7f77300d3cd0 msgr2=0x7f77300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.403+0000 7f77361ae640 1 -- 172.21.15.59:0/2249857093 shutdown_connections 2024-03-06T07:32:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.403+0000 7f77361ae640 1 -- 172.21.15.59:0/2249857093 wait complete. 2024-03-06T07:32:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.404+0000 7f77361ae640 1 Processor -- start 2024-03-06T07:32:18.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.404+0000 7f77361ae640 1 -- start start 2024-03-06T07:32:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.405+0000 7f77361ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77300b41f0 0x7f77301f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.405+0000 7f77361ae640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77300d22b0 0x7f77300cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.405+0000 7f77361ae640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f773015dfe0 0x7f7730149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.409 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.405+0000 7f77361ae640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77300cfa00 con 0x7f77300b41f0 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.405+0000 7f77361ae640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77300b3d60 con 0x7f77300d22b0 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.405+0000 7f77361ae640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77300d1c70 con 0x7f773015dfe0 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.405+0000 7f772f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77300b41f0 0x7f77301f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.405+0000 7f772f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77300b41f0 0x7f77301f6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43322/0 (socket says 172.21.15.59:43322) 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.405+0000 7f772f7fe640 1 -- 172.21.15.59:0/2968316043 learned_addr learned my addr 172.21.15.59:0/2968316043 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.405+0000 7f772effd640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77300d22b0 0x7f77300cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.405+0000 7f772ffff640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f773015dfe0 0x7f7730149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f772f7fe640 1 -- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f773015dfe0 msgr2=0x7f7730149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f772f7fe640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f773015dfe0 0x7f7730149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f772f7fe640 1 -- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77300d22b0 msgr2=0x7f77300cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f772f7fe640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77300d22b0 0x7f77300cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f772f7fe640 1 -- 172.21.15.59:0/2968316043 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77300b70b0 con 0x7f77300b41f0 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f772effd640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77300d22b0 0x7f77300cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f772f7fe640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77300b41f0 0x7f77301f6a30 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f772400c3f0 tx=0x7f772400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f7717fff640 1 -- 172.21.15.59:0/2968316043 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77240052d0 con 0x7f77300b41f0 2024-03-06T07:32:18.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f7717fff640 1 -- 172.21.15.59:0/2968316043 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7724005490 con 0x7f77300b41f0 2024-03-06T07:32:18.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f77361ae640 1 -- 172.21.15.59:0/2968316043 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77300b6af0 con 0x7f77300b41f0 2024-03-06T07:32:18.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f7717fff640 1 -- 172.21.15.59:0/2968316043 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724007260 con 0x7f77300b41f0 2024-03-06T07:32:18.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.406+0000 7f77361ae640 1 -- 172.21.15.59:0/2968316043 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77300cc9a0 con 0x7f77300b41f0 2024-03-06T07:32:18.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.407+0000 7f77361ae640 1 -- 172.21.15.59:0/2968316043 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773004f330 con 0x7f77300b41f0 2024-03-06T07:32:18.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.408+0000 7f7717fff640 1 -- 172.21.15.59:0/2968316043 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7724007400 con 0x7f77300b41f0 2024-03-06T07:32:18.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.408+0000 7f7717fff640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f771c06b4d0 0x7f771c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.408+0000 7f7717fff640 1 -- 172.21.15.59:0/2968316043 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f77240897d0 con 0x7f77300b41f0 2024-03-06T07:32:18.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.408+0000 7f772effd640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f771c06b4d0 0x7f771c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.412 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.409+0000 7f772effd640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f771c06b4d0 0x7f771c06d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f77300cda40 tx=0x7f7720008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.414 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.413+0000 7f7717fff640 1 -- 172.21.15.59:0/2968316043 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f77300b70b0 con 0x7f77300b41f0 2024-03-06T07:32:18.711 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.709+0000 7f77361ae640 1 -- 172.21.15.59:0/2968316043 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77300d0860 con 0x7f77300b41f0 2024-03-06T07:32:18.712 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.710+0000 7f7717fff640 1 -- 172.21.15.59:0/2968316043 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f7724058520 con 0x7f77300b41f0 2024-03-06T07:32:18.712 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:18.712 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:18.712 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:18.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 -- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f771c06b4d0 msgr2=0x7f771c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f771c06b4d0 0x7f771c06d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f77300cda40 tx=0x7f7720008040 comp rx=0 tx=0).stop 2024-03-06T07:32:18.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 -- 172.21.15.59:0/2968316043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77300b41f0 msgr2=0x7f77301f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77300b41f0 0x7f77301f6a30 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f772400c3f0 tx=0x7f772400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 -- 172.21.15.59:0/2968316043 shutdown_connections 2024-03-06T07:32:18.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f771c06b4d0 0x7f771c06d990 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f773015dfe0 0x7f7730149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77300d22b0 0x7f77300cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 --2- 172.21.15.59:0/2968316043 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77300b41f0 0x7f77301f6a30 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 -- 172.21.15.59:0/2968316043 >> 172.21.15.59:0/2968316043 conn(0x7f77300d3cd0 msgr2=0x7f77300b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 -- 172.21.15.59:0/2968316043 shutdown_connections 2024-03-06T07:32:18.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:18.712+0000 7f77361ae640 1 -- 172.21.15.59:0/2968316043 wait complete. 2024-03-06T07:32:18.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:18.930 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:19.931 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:20.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.030+0000 7f03e2c39640 1 Processor -- start 2024-03-06T07:32:20.032 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.030+0000 7f03e2c39640 1 -- start start 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.031+0000 7f03e2c39640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03dc06c6d0 0x7f03dc15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.031+0000 7f03e2c39640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f03dc066d90 0x7f03dc0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.031+0000 7f03e2c39640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03dc0cd450 con 0x7f03dc0b4f70 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.031+0000 7f03e2c39640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03dc15e9f0 con 0x7f03dc066d90 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.031+0000 7f03e2c39640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc0cda10 con 0x7f03dc06c6d0 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.031+0000 7f03e09d5640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f03dc0b4f70 0x7f03dc0b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37082/0 (socket says 172.21.15.59:37082) 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.031+0000 7f03e09d5640 1 -- 172.21.15.59:0/1493129968 learned_addr learned my addr 172.21.15.59:0/1493129968 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.031+0000 7f03e11d6640 1 --2- 172.21.15.59:0/1493129968 >> v2:172.21.15.59:3300/0 conn(0x7f03dc066d90 0x7f03dc0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.032+0000 7f03dbfff640 1 --2- 172.21.15.59:0/1493129968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03dc06c6d0 0x7f03dc15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.032+0000 7f03e11d6640 1 -- 172.21.15.59:0/1493129968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03dc06c6d0 msgr2=0x7f03dc15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.032+0000 7f03e11d6640 1 --2- 172.21.15.59:0/1493129968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03dc06c6d0 0x7f03dc15ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.032+0000 7f03e11d6640 1 -- 172.21.15.59:0/1493129968 >> v1:172.21.15.59:6789/0 conn(0x7f03dc0b4f70 legacy=0x7f03dc0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.032+0000 7f03e11d6640 1 -- 172.21.15.59:0/1493129968 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc0cd730 con 0x7f03dc066d90 2024-03-06T07:32:20.034 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.032+0000 7f03dbfff640 1 --2- 172.21.15.59:0/1493129968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03dc06c6d0 0x7f03dc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:20.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.032+0000 7f03e11d6640 1 --2- 172.21.15.59:0/1493129968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03dc066d90 0x7f03dc0ccb50 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009d90 tx=0x7f03d402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e90f7bcc4e6c37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.033+0000 7f03db7fe640 1 -- 172.21.15.59:0/1493129968 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4039070 con 0x7f03dc066d90 2024-03-06T07:32:20.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.033+0000 7f03db7fe640 1 -- 172.21.15.59:0/1493129968 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d402cc50 con 0x7f03dc066d90 2024-03-06T07:32:20.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.033+0000 7f03db7fe640 1 -- 172.21.15.59:0/1493129968 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4034440 con 0x7f03dc066d90 2024-03-06T07:32:20.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.033+0000 7f03e2c39640 1 -- 172.21.15.59:0/1493129968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03dc066d90 msgr2=0x7f03dc0ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.033+0000 7f03e2c39640 1 --2- 172.21.15.59:0/1493129968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03dc066d90 0x7f03dc0ccb50 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009d90 tx=0x7f03d402bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.033+0000 7f03e2c39640 1 -- 172.21.15.59:0/1493129968 shutdown_connections 2024-03-06T07:32:20.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.033+0000 7f03e2c39640 1 --2- 172.21.15.59:0/1493129968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03dc066d90 0x7f03dc0ccb50 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.033+0000 7f03e2c39640 1 --2- 172.21.15.59:0/1493129968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03dc06c6d0 0x7f03dc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.033+0000 7f03e2c39640 1 -- 172.21.15.59:0/1493129968 >> 172.21.15.59:0/1493129968 conn(0x7f03dc0724d0 msgr2=0x7f03dc0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.033+0000 7f03e2c39640 1 -- 172.21.15.59:0/1493129968 shutdown_connections 2024-03-06T07:32:20.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.033+0000 7f03e2c39640 1 -- 172.21.15.59:0/1493129968 wait complete. 2024-03-06T07:32:20.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.034+0000 7f03e2c39640 1 Processor -- start 2024-03-06T07:32:20.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.034+0000 7f03e2c39640 1 -- start start 2024-03-06T07:32:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.035+0000 7f03e2c39640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03dc066d90 0x7f03dc14e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.035+0000 7f03e2c39640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03dc0b4f70 0x7f03dc14d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.035+0000 7f03e2c39640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03dc0d0350 0x7f03dc0d07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.035+0000 7f03e2c39640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc0cd090 con 0x7f03dc0d0350 2024-03-06T07:32:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.035+0000 7f03e2c39640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc15f3c0 con 0x7f03dc0b4f70 2024-03-06T07:32:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.035+0000 7f03e2c39640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc0cd210 con 0x7f03dc066d90 2024-03-06T07:32:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.035+0000 7f03e11d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03dc0d0350 0x7f03dc0d07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.035+0000 7f03e11d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03dc0d0350 0x7f03dc0d07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43352/0 (socket says 172.21.15.59:43352) 2024-03-06T07:32:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.035+0000 7f03e11d6640 1 -- 172.21.15.59:0/82654516 learned_addr learned my addr 172.21.15.59:0/82654516 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:20.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.035+0000 7f03e09d5640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03dc066d90 0x7f03dc14e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.035+0000 7f03dbfff640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03dc0b4f70 0x7f03dc14d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03e11d6640 1 -- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03dc066d90 msgr2=0x7f03dc14e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03e11d6640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03dc066d90 0x7f03dc14e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03e11d6640 1 -- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03dc0b4f70 msgr2=0x7f03dc14d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03e11d6640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03dc0b4f70 0x7f03dc14d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03e11d6640 1 -- 172.21.15.59:0/82654516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc0addd0 con 0x7f03dc0d0350 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03dbfff640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03dc0b4f70 0x7f03dc14d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03e11d6640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03dc0d0350 0x7f03dc0d07e0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009eb0 tx=0x7f03d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03d97fa640 1 -- 172.21.15.59:0/82654516 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4039050 con 0x7f03dc0d0350 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03d97fa640 1 -- 172.21.15.59:0/82654516 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d4045070 con 0x7f03dc0d0350 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03e2c39640 1 -- 172.21.15.59:0/82654516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03dc0ae0b0 con 0x7f03dc0d0350 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03d97fa640 1 -- 172.21.15.59:0/82654516 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d403e610 con 0x7f03dc0d0350 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.036+0000 7f03e2c39640 1 -- 172.21.15.59:0/82654516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03dc0ae670 con 0x7f03dc0d0350 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.037+0000 7f03e2c39640 1 -- 172.21.15.59:0/82654516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03dc04f330 con 0x7f03dc0d0350 2024-03-06T07:32:20.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.037+0000 7f03d97fa640 1 -- 172.21.15.59:0/82654516 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03d403e7b0 con 0x7f03dc0d0350 2024-03-06T07:32:20.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.038+0000 7f03d97fa640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03c806b4d0 0x7f03c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.038+0000 7f03d97fa640 1 -- 172.21.15.59:0/82654516 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f03d40acda0 con 0x7f03dc0d0350 2024-03-06T07:32:20.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.038+0000 7f03e09d5640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03c806b4d0 0x7f03c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.039+0000 7f03e09d5640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03c806b4d0 0x7f03c806d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0ada00 tx=0x7f03cc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.042+0000 7f03d97fa640 1 -- 172.21.15.59:0/82654516 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03d4032e20 con 0x7f03dc0d0350 2024-03-06T07:32:20.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.332+0000 7f03e2c39640 1 -- 172.21.15.59:0/82654516 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03dc0d1070 con 0x7f03dc0d0350 2024-03-06T07:32:20.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.332+0000 7f03d97fa640 1 -- 172.21.15.59:0/82654516 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f03d407bc90 con 0x7f03dc0d0350 2024-03-06T07:32:20.334 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:20.334 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:20.334 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 -- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03c806b4d0 msgr2=0x7f03c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03c806b4d0 0x7f03c806d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0ada00 tx=0x7f03cc009040 comp rx=0 tx=0).stop 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 -- 172.21.15.59:0/82654516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03dc0d0350 msgr2=0x7f03dc0d07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03dc0d0350 0x7f03dc0d07e0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009eb0 tx=0x7f03d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 -- 172.21.15.59:0/82654516 shutdown_connections 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f03c806b4d0 0x7f03c806d990 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03dc0d0350 0x7f03dc0d07e0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03dc0b4f70 0x7f03dc14d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 --2- 172.21.15.59:0/82654516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03dc066d90 0x7f03dc14e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 -- 172.21.15.59:0/82654516 >> 172.21.15.59:0/82654516 conn(0x7f03dc0724d0 msgr2=0x7f03dc15b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 -- 172.21.15.59:0/82654516 shutdown_connections 2024-03-06T07:32:20.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:20.335+0000 7f03e2c39640 1 -- 172.21.15.59:0/82654516 wait complete. 2024-03-06T07:32:20.548 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:20.549 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:21.217 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:21.219 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:21.550 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:21.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.649+0000 7fa405678640 1 Processor -- start 2024-03-06T07:32:21.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.649+0000 7fa405678640 1 -- start start 2024-03-06T07:32:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa405678640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa4000d22b0 0x7fa4000d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa405678640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4000b41f0 0x7fa4000b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa405678640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4000b3430 con 0x7fa40015dfe0 2024-03-06T07:32:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa405678640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4000d2c50 con 0x7fa4000d22b0 2024-03-06T07:32:21.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa405678640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4000b31e0 con 0x7fa4000b41f0 2024-03-06T07:32:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa3fe7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa4000d22b0 0x7fa4000d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa3feffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa40015dfe0 0x7fa4000cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:37094/0 (socket says 172.21.15.59:37094) 2024-03-06T07:32:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa3feffd640 1 -- 172.21.15.59:0/1617139560 learned_addr learned my addr 172.21.15.59:0/1617139560 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa3fe7fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa4000d22b0 0x7fa4000d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43360/0 (socket says 172.21.15.59:43360) 2024-03-06T07:32:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa3fe7fc640 1 -- 172.21.15.59:0/1617139560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4000b41f0 msgr2=0x7fa4000b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa3ff7fe640 1 --2- 172.21.15.59:0/1617139560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4000b41f0 0x7fa4000b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa3fe7fc640 1 --2- 172.21.15.59:0/1617139560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4000b41f0 0x7fa4000b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa3fe7fc640 1 -- 172.21.15.59:0/1617139560 >> v1:172.21.15.59:6789/0 conn(0x7fa40015dfe0 legacy=0x7fa4000cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa3fdffb640 1 -- 172.21.15.59:0/1617139560 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832441033 0 0) 0x7fa4000b3430 con 0x7fa40015dfe0 2024-03-06T07:32:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa3fe7fc640 1 -- 172.21.15.59:0/1617139560 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4000b3710 con 0x7fa4000d22b0 2024-03-06T07:32:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.650+0000 7fa3ff7fe640 1 --2- 172.21.15.59:0/1617139560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4000b41f0 0x7fa4000b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.651+0000 7fa3fe7fc640 1 --2- 172.21.15.59:0/1617139560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4000d22b0 0x7fa4000d2710 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40097a0 tx=0x7fa3f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355dcb94ec4ad21c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.651+0000 7fa3fdffb640 1 -- 172.21.15.59:0/1617139560 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4035020 con 0x7fa4000d22b0 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.651+0000 7fa3fdffb640 1 -- 172.21.15.59:0/1617139560 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4031830 con 0x7fa4000d22b0 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.651+0000 7fa3fdffb640 1 -- 172.21.15.59:0/1617139560 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4031c00 con 0x7fa4000d22b0 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.652+0000 7fa405678640 1 -- 172.21.15.59:0/1617139560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4000d22b0 msgr2=0x7fa4000d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.652+0000 7fa405678640 1 --2- 172.21.15.59:0/1617139560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4000d22b0 0x7fa4000d2710 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40097a0 tx=0x7fa3f402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.652+0000 7fa405678640 1 -- 172.21.15.59:0/1617139560 shutdown_connections 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.652+0000 7fa405678640 1 --2- 172.21.15.59:0/1617139560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4000b41f0 0x7fa4000b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.652+0000 7fa405678640 1 --2- 172.21.15.59:0/1617139560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4000d22b0 0x7fa4000d2710 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.652+0000 7fa405678640 1 -- 172.21.15.59:0/1617139560 >> 172.21.15.59:0/1617139560 conn(0x7fa4000d3cd0 msgr2=0x7fa4000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.652+0000 7fa405678640 1 -- 172.21.15.59:0/1617139560 shutdown_connections 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.652+0000 7fa405678640 1 -- 172.21.15.59:0/1617139560 wait complete. 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.653+0000 7fa405678640 1 Processor -- start 2024-03-06T07:32:21.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.653+0000 7fa405678640 1 -- start start 2024-03-06T07:32:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa405678640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4000b41f0 0x7fa40015b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa405678640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4000d22b0 0x7fa4001603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa405678640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa40015dfe0 0x7fa400160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa405678640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4000cfa00 con 0x7fa4000d22b0 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa405678640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4000b3d60 con 0x7fa4000b41f0 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa405678640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4000d1c70 con 0x7fa40015dfe0 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3fe7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4000d22b0 0x7fa4001603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3fe7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4000d22b0 0x7fa4001603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43370/0 (socket says 172.21.15.59:43370) 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3fe7fc640 1 -- 172.21.15.59:0/3628516703 learned_addr learned my addr 172.21.15.59:0/3628516703 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3ff7fe640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa40015dfe0 0x7fa400160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3feffd640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4000b41f0 0x7fa40015b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3fe7fc640 1 -- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa40015dfe0 msgr2=0x7fa400160910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3fe7fc640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa40015dfe0 0x7fa400160910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3fe7fc640 1 -- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4000b41f0 msgr2=0x7fa40015b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3fe7fc640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4000b41f0 0x7fa40015b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3fe7fc640 1 -- 172.21.15.59:0/3628516703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4001f3a00 con 0x7fa4000d22b0 2024-03-06T07:32:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3fe7fc640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4000d22b0 0x7fa4001603d0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fa3f402bee0 tx=0x7fa3f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.654+0000 7fa3e77fe640 1 -- 172.21.15.59:0/3628516703 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4041070 con 0x7fa4000d22b0 2024-03-06T07:32:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.655+0000 7fa3e77fe640 1 -- 172.21.15.59:0/3628516703 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f403a950 con 0x7fa4000d22b0 2024-03-06T07:32:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.655+0000 7fa405678640 1 -- 172.21.15.59:0/3628516703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4001f3c90 con 0x7fa4000d22b0 2024-03-06T07:32:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.655+0000 7fa3e77fe640 1 -- 172.21.15.59:0/3628516703 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4035040 con 0x7fa4000d22b0 2024-03-06T07:32:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.655+0000 7fa405678640 1 -- 172.21.15.59:0/3628516703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4001f4250 con 0x7fa4000d22b0 2024-03-06T07:32:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.656+0000 7fa405678640 1 -- 172.21.15.59:0/3628516703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40004f330 con 0x7fa4000d22b0 2024-03-06T07:32:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.656+0000 7fa3e77fe640 1 -- 172.21.15.59:0/3628516703 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3f404c060 con 0x7fa4000d22b0 2024-03-06T07:32:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.656+0000 7fa3e77fe640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ec06b4d0 0x7fa3ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.656+0000 7fa3e77fe640 1 -- 172.21.15.59:0/3628516703 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa3f40abac0 con 0x7fa4000d22b0 2024-03-06T07:32:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.657+0000 7fa3feffd640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ec06b4d0 0x7fa3ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.657+0000 7fa3feffd640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ec06b4d0 0x7fa3ec06d990 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8007fd0 tx=0x7fa3f8007de0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.661+0000 7fa3e77fe640 1 -- 172.21.15.59:0/3628516703 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3f407a360 con 0x7fa4000d22b0 2024-03-06T07:32:21.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.888+0000 7fa405678640 1 -- 172.21.15.59:0/3628516703 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4000b1900 con 0x7fa4000d22b0 2024-03-06T07:32:21.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.888+0000 7fa3e77fe640 1 -- 172.21.15.59:0/3628516703 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fa3f407e400 con 0x7fa4000d22b0 2024-03-06T07:32:21.890 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:21.890 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:21.891 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.890+0000 7fa405678640 1 -- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ec06b4d0 msgr2=0x7fa3ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.890+0000 7fa405678640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ec06b4d0 0x7fa3ec06d990 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8007fd0 tx=0x7fa3f8007de0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.890+0000 7fa405678640 1 -- 172.21.15.59:0/3628516703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4000d22b0 msgr2=0x7fa4001603d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.890+0000 7fa405678640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4000d22b0 0x7fa4001603d0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7fa3f402bee0 tx=0x7fa3f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.891+0000 7fa405678640 1 -- 172.21.15.59:0/3628516703 shutdown_connections 2024-03-06T07:32:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.891+0000 7fa405678640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ec06b4d0 0x7fa3ec06d990 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.891+0000 7fa405678640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa40015dfe0 0x7fa400160910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.891+0000 7fa405678640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa4000d22b0 0x7fa4001603d0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.891+0000 7fa405678640 1 --2- 172.21.15.59:0/3628516703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4000b41f0 0x7fa40015b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.891+0000 7fa405678640 1 -- 172.21.15.59:0/3628516703 >> 172.21.15.59:0/3628516703 conn(0x7fa4000d3cd0 msgr2=0x7fa4000b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.891+0000 7fa405678640 1 -- 172.21.15.59:0/3628516703 shutdown_connections 2024-03-06T07:32:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:21.891+0000 7fa405678640 1 -- 172.21.15.59:0/3628516703 wait complete. 2024-03-06T07:32:22.104 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:22.104 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:23.105 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:23.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.204+0000 7f7ed2288640 1 Processor -- start 2024-03-06T07:32:23.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.204+0000 7f7ed2288640 1 -- start start 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.205+0000 7f7ed2288640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ecc15dfe0 0x7f7ecc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.205+0000 7f7ed2288640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ecc0cf830 0x7f7ecc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.205+0000 7f7ed2288640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ecc0b3430 con 0x7f7ecc0d27f0 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.205+0000 7f7ed2288640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ecc0d2c50 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.205+0000 7f7ed2288640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc0b31e0 con 0x7f7ecc0cf830 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.205+0000 7f7ecbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ecc15dfe0 0x7f7ecc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.205+0000 7f7ecbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ecc15dfe0 0x7f7ecc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43386/0 (socket says 172.21.15.59:43386) 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.205+0000 7f7ecbfff640 1 -- 172.21.15.59:0/1708091467 learned_addr learned my addr 172.21.15.59:0/1708091467 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.205+0000 7f7ecb7fe640 1 --2- 172.21.15.59:0/1708091467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ecc0cf830 0x7f7ecc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.206+0000 7f7ecbfff640 1 -- 172.21.15.59:0/1708091467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ecc0cf830 msgr2=0x7f7ecc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.206+0000 7f7ecbfff640 1 --2- 172.21.15.59:0/1708091467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ecc0cf830 0x7f7ecc0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.206+0000 7f7ecbfff640 1 -- 172.21.15.59:0/1708091467 >> v1:172.21.15.164:6789/0 conn(0x7f7ecc0d27f0 legacy=0x7f7ecc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.206+0000 7f7ecbfff640 1 -- 172.21.15.59:0/1708091467 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ecc0b3710 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.206+0000 7f7ecb7fe640 1 --2- 172.21.15.59:0/1708091467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ecc0cf830 0x7f7ecc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.206+0000 7f7ecbfff640 1 --2- 172.21.15.59:0/1708091467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ecc15dfe0 0x7f7ecc0cf2f0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009870 tx=0x7f7ebc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d79269a43c8011c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.207+0000 7f7ecaffd640 1 -- 172.21.15.59:0/1708091467 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc034070 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.207+0000 7f7ecaffd640 1 -- 172.21.15.59:0/1708091467 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc030840 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.207+0000 7f7ecaffd640 1 -- 172.21.15.59:0/1708091467 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc030c10 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.207+0000 7f7ed2288640 1 -- 172.21.15.59:0/1708091467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ecc15dfe0 msgr2=0x7f7ecc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.207+0000 7f7ed2288640 1 --2- 172.21.15.59:0/1708091467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ecc15dfe0 0x7f7ecc0cf2f0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009870 tx=0x7f7ebc02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.207+0000 7f7ed2288640 1 -- 172.21.15.59:0/1708091467 shutdown_connections 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.207+0000 7f7ed2288640 1 --2- 172.21.15.59:0/1708091467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ecc0cf830 0x7f7ecc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.207+0000 7f7ed2288640 1 --2- 172.21.15.59:0/1708091467 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ecc15dfe0 0x7f7ecc0cf2f0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.207+0000 7f7ed2288640 1 -- 172.21.15.59:0/1708091467 >> 172.21.15.59:0/1708091467 conn(0x7f7ecc0d3cd0 msgr2=0x7f7ecc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.207+0000 7f7ed2288640 1 -- 172.21.15.59:0/1708091467 shutdown_connections 2024-03-06T07:32:23.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.207+0000 7f7ed2288640 1 -- 172.21.15.59:0/1708091467 wait complete. 2024-03-06T07:32:23.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.208+0000 7f7ed2288640 1 Processor -- start 2024-03-06T07:32:23.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.208+0000 7f7ed2288640 1 -- start start 2024-03-06T07:32:23.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.209+0000 7f7ed2288640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ecc0cf830 0x7f7ecc1f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.209+0000 7f7ed2288640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ecc0d27f0 0x7f7ecc14d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.209+0000 7f7ed2288640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ecc15dfe0 0x7f7ecc14d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.209+0000 7f7ed2288640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc0d1c70 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.209+0000 7f7ed2288640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc0b3d60 con 0x7f7ecc0cf830 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.209+0000 7f7ed2288640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc0d1df0 con 0x7f7ecc0d27f0 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.209+0000 7f7ed0825640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ecc15dfe0 0x7f7ecc14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.209+0000 7f7ed0825640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ecc15dfe0 0x7f7ecc14d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43396/0 (socket says 172.21.15.59:43396) 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.209+0000 7f7ed0825640 1 -- 172.21.15.59:0/3233072374 learned_addr learned my addr 172.21.15.59:0/3233072374 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.209+0000 7f7ecb7fe640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ecc0d27f0 0x7f7ecc14d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ecbfff640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ecc0cf830 0x7f7ecc1f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ed0825640 1 -- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ecc0d27f0 msgr2=0x7f7ecc14d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ed0825640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ecc0d27f0 0x7f7ecc14d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ed0825640 1 -- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ecc0cf830 msgr2=0x7f7ecc1f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ed0825640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ecc0cf830 0x7f7ecc1f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ed0825640 1 -- 172.21.15.59:0/3233072374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ecc1f40b0 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ed0825640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ecc15dfe0 0x7f7ecc14d860 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400c7a0 tx=0x7f7ec400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ec8ff9640 1 -- 172.21.15.59:0/3233072374 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4003b70 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ec8ff9640 1 -- 172.21.15.59:0/3233072374 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec4003d30 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ed2288640 1 -- 172.21.15.59:0/3233072374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ecc1f4390 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ec8ff9640 1 -- 172.21.15.59:0/3233072374 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec400aa90 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.210+0000 7f7ed2288640 1 -- 172.21.15.59:0/3233072374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ecc14ce00 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.211+0000 7f7ed2288640 1 -- 172.21.15.59:0/3233072374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ecc04f330 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.211+0000 7f7ec8ff9640 1 -- 172.21.15.59:0/3233072374 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ec400ac30 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.212+0000 7f7ec8ff9640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7eb806b4d0 0x7f7eb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.212+0000 7f7ec8ff9640 1 -- 172.21.15.59:0/3233072374 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7ec408a100 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.212+0000 7f7ecbfff640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7eb806b4d0 0x7f7eb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.213+0000 7f7ecbfff640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7eb806b4d0 0x7f7eb806d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc036000 tx=0x7f7ebc03b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.216+0000 7f7ec8ff9640 1 -- 172.21.15.59:0/3233072374 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ec4058fd0 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.462+0000 7f7ed2288640 1 -- 172.21.15.59:0/3233072374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ecc0b0d40 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.462+0000 7f7ec8ff9640 1 -- 172.21.15.59:0/3233072374 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f7ec40588d0 con 0x7f7ecc15dfe0 2024-03-06T07:32:23.464 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:23.464 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:23.465 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.464+0000 7f7ed2288640 1 -- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7eb806b4d0 msgr2=0x7f7eb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.464+0000 7f7ed2288640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7eb806b4d0 0x7f7eb806d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc036000 tx=0x7f7ebc03b040 comp rx=0 tx=0).stop 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.464+0000 7f7ed2288640 1 -- 172.21.15.59:0/3233072374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ecc15dfe0 msgr2=0x7f7ecc14d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.464+0000 7f7ed2288640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ecc15dfe0 0x7f7ecc14d860 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400c7a0 tx=0x7f7ec400cc60 comp rx=0 tx=0).stop 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.464+0000 7f7ed2288640 1 -- 172.21.15.59:0/3233072374 shutdown_connections 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.464+0000 7f7ed2288640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7eb806b4d0 0x7f7eb806d990 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.464+0000 7f7ed2288640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ecc15dfe0 0x7f7ecc14d860 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.464+0000 7f7ed2288640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ecc0d27f0 0x7f7ecc14d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.464+0000 7f7ed2288640 1 --2- 172.21.15.59:0/3233072374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ecc0cf830 0x7f7ecc1f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.465+0000 7f7ed2288640 1 -- 172.21.15.59:0/3233072374 >> 172.21.15.59:0/3233072374 conn(0x7f7ecc0d3cd0 msgr2=0x7f7ecc0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.465+0000 7f7ed2288640 1 -- 172.21.15.59:0/3233072374 shutdown_connections 2024-03-06T07:32:23.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:23.465+0000 7f7ed2288640 1 -- 172.21.15.59:0/3233072374 wait complete. 2024-03-06T07:32:23.678 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:23.678 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:24.679 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:24.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.779+0000 7f43b32a3640 1 Processor -- start 2024-03-06T07:32:24.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.779+0000 7f43b32a3640 1 -- start start 2024-03-06T07:32:24.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.779+0000 7f43b32a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0cfff0 0x7f43ac0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.779+0000 7f43b32a3640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac161980 con 0x7f43ac0cc8b0 2024-03-06T07:32:24.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.779+0000 7f43b32a3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac161c60 con 0x7f43ac15bb20 2024-03-06T07:32:24.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.779+0000 7f43b32a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0d2500 con 0x7f43ac0cfff0 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43b1840640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f43ac0cc8b0 0x7f43ac0ccd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39670/0 (socket says 172.21.15.59:39670) 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43b1840640 1 -- 172.21.15.59:0/459515285 learned_addr learned my addr 172.21.15.59:0/459515285 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43b083e640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0cfff0 0x7f43ac0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926611677 0 0) 0x7f43ac161980 con 0x7f43ac0cc8b0 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4394003520 con 0x7f43ac0cc8b0 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455757071 0 0) 0x7f43ac161c60 con 0x7f43ac15bb20 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43ac161980 con 0x7f43ac15bb20 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063222318 0 0) 0x7f4394003520 con 0x7f43ac0cc8b0 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0cfff0 msgr2=0x7f43ac0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0cfff0 0x7f43ac0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 >> v1:172.21.15.164:6789/0 conn(0x7f43ac15bb20 legacy=0x7f43ac0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac161f40 con 0x7f43ac0cc8b0 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f439c001670 con 0x7f43ac0cc8b0 2024-03-06T07:32:24.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 >> v1:172.21.15.59:6789/0 conn(0x7f43ac0cc8b0 legacy=0x7f43ac0ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0cfff0 0x7f4394003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43940040d0 0x7f4394004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 0x7f43940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0d2500 con 0x7f4394004d40 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439400baa0 con 0x7f43940040d0 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439400bca0 con 0x7f43ac0cfff0 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43b1840640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 0x7f43940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.780+0000 7f43b103f640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0cfff0 0x7f4394003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.781+0000 7f43b083e640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43940040d0 0x7f4394004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.781+0000 7f43b1840640 1 -- 172.21.15.59:0/459515285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0cfff0 msgr2=0x7f4394003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.781+0000 7f43b1840640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0cfff0 0x7f4394003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.781+0000 7f43b1840640 1 -- 172.21.15.59:0/459515285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43940040d0 msgr2=0x7f4394004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.781+0000 7f43b1840640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43940040d0 0x7f4394004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.781+0000 7f43b1840640 1 -- 172.21.15.59:0/459515285 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4394003520 con 0x7f4394004d40 2024-03-06T07:32:24.788 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.782+0000 7f43b1840640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 0x7f43940051c0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f439c00c1c0 tx=0x7f4394005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bffa33e48704511 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.782+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c007cc0 con 0x7f4394004d40 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.782+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439c030070 con 0x7f4394004d40 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.782+0000 7f43abfff640 1 -- 172.21.15.59:0/459515285 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c005d50 con 0x7f4394004d40 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.782+0000 7f43b32a3640 1 -- 172.21.15.59:0/459515285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 msgr2=0x7f43940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.782+0000 7f43b32a3640 1 --2- 172.21.15.59:0/459515285 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 0x7f43940051c0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f439c00c1c0 tx=0x7f4394005830 comp rx=0 tx=0).stop 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.783+0000 7f43b1840640 1 -- 172.21.15.59:0/459515285 reap_dead start 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.783+0000 7f43b32a3640 1 -- 172.21.15.59:0/459515285 shutdown_connections 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.783+0000 7f43b32a3640 1 -- 172.21.15.59:0/459515285 >> 172.21.15.59:0/459515285 conn(0x7f43ac09a9f0 msgr2=0x7f43ac09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.783+0000 7f43b32a3640 1 -- 172.21.15.59:0/459515285 shutdown_connections 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.783+0000 7f43b32a3640 1 -- 172.21.15.59:0/459515285 wait complete. 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.784+0000 7f43b32a3640 1 Processor -- start 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.784+0000 7f43b32a3640 1 -- start start 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.784+0000 7f43b32a3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 0x7f43ac0c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.784+0000 7f43b32a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0c5790 0x7f43ac155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.784+0000 7f43b32a3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43ac0b1e80 0x7f43ac0b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b32a3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0d1980 con 0x7f4394004d40 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b32a3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac162500 con 0x7f43ac0b1e80 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b32a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0d1b00 con 0x7f43ac0c5790 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b103f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 0x7f43ac0c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b103f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 0x7f43ac0c80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51280/0 (socket says 172.21.15.59:51280) 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b103f640 1 -- 172.21.15.59:0/2620707876 learned_addr learned my addr 172.21.15.59:0/2620707876 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b083e640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0c5790 0x7f43ac155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b1840640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43ac0b1e80 0x7f43ac0b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b103f640 1 -- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0c5790 msgr2=0x7f43ac155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b103f640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0c5790 0x7f43ac155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b103f640 1 -- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43ac0b1e80 msgr2=0x7f43ac0b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b103f640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43ac0b1e80 0x7f43ac0b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b103f640 1 -- 172.21.15.59:0/2620707876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac0c5d00 con 0x7f4394004d40 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b103f640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 0x7f43ac0c80d0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f43a400fb10 tx=0x7f43a400ff10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43a9ffb640 1 -- 172.21.15.59:0/2620707876 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4003b80 con 0x7f4394004d40 2024-03-06T07:32:24.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43a9ffb640 1 -- 172.21.15.59:0/2620707876 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a4018070 con 0x7f4394004d40 2024-03-06T07:32:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43b32a3640 1 -- 172.21.15.59:0/2620707876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ac0c5fe0 con 0x7f4394004d40 2024-03-06T07:32:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.785+0000 7f43a9ffb640 1 -- 172.21.15.59:0/2620707876 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a400d890 con 0x7f4394004d40 2024-03-06T07:32:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.786+0000 7f43b32a3640 1 -- 172.21.15.59:0/2620707876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ac0c65a0 con 0x7f4394004d40 2024-03-06T07:32:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.787+0000 7f43b32a3640 1 -- 172.21.15.59:0/2620707876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ac04f330 con 0x7f4394004d40 2024-03-06T07:32:24.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.787+0000 7f43a9ffb640 1 -- 172.21.15.59:0/2620707876 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43a400da30 con 0x7f4394004d40 2024-03-06T07:32:24.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.787+0000 7f43a9ffb640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.787+0000 7f43a9ffb640 1 -- 172.21.15.59:0/2620707876 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f43a408a360 con 0x7f4394004d40 2024-03-06T07:32:24.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.788+0000 7f43b083e640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.788+0000 7f43b083e640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f439806b4d0 0x7f439806d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f43ac15b200 tx=0x7f43a0009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:24.792+0000 7f43a9ffb640 1 -- 172.21.15.59:0/2620707876 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43a4059250 con 0x7f4394004d40 2024-03-06T07:32:25.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.017+0000 7f43b32a3640 1 -- 172.21.15.59:0/2620707876 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43ac0b1820 con 0x7f4394004d40 2024-03-06T07:32:25.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.017+0000 7f43a9ffb640 1 -- 172.21.15.59:0/2620707876 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f43a4058b50 con 0x7f4394004d40 2024-03-06T07:32:25.019 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:25.019 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:25.020 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:25.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 -- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f439806b4d0 msgr2=0x7f439806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f439806b4d0 0x7f439806d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f43ac15b200 tx=0x7f43a0009190 comp rx=0 tx=0).stop 2024-03-06T07:32:25.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 -- 172.21.15.59:0/2620707876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 msgr2=0x7f43ac0c80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 0x7f43ac0c80d0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f43a400fb10 tx=0x7f43a400ff10 comp rx=0 tx=0).stop 2024-03-06T07:32:25.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 -- 172.21.15.59:0/2620707876 shutdown_connections 2024-03-06T07:32:25.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f439806b4d0 0x7f439806d990 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43ac0b1e80 0x7f43ac0b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43ac0c5790 0x7f43ac155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 --2- 172.21.15.59:0/2620707876 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4394004d40 0x7f43ac0c80d0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 -- 172.21.15.59:0/2620707876 >> 172.21.15.59:0/2620707876 conn(0x7f43ac09a9f0 msgr2=0x7f43ac073f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 -- 172.21.15.59:0/2620707876 shutdown_connections 2024-03-06T07:32:25.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:25.019+0000 7f43b32a3640 1 -- 172.21.15.59:0/2620707876 wait complete. 2024-03-06T07:32:25.234 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:25.234 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:26.235 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:26.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.346+0000 7f4ba11d7640 1 Processor -- start 2024-03-06T07:32:26.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.346+0000 7f4ba11d7640 1 -- start start 2024-03-06T07:32:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.346+0000 7f4ba11d7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4b9c162a00 0x7f4b9c0d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.346+0000 7f4ba11d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9c162400 0x7f4b9c0cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.346+0000 7f4ba11d7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9c0b1070 con 0x7f4b9c162020 2024-03-06T07:32:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.346+0000 7f4ba11d7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9c0ce9d0 con 0x7f4b9c162a00 2024-03-06T07:32:26.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.346+0000 7f4ba11d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c0d2da0 con 0x7f4b9c162400 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.346+0000 7f4b9ad9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4b9c162020 0x7f4b9c0b0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39682/0 (socket says 172.21.15.59:39682) 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.347+0000 7f4b9ad9d640 1 -- 172.21.15.59:0/1962952386 learned_addr learned my addr 172.21.15.59:0/1962952386 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.347+0000 7f4b9b59e640 1 --2- 172.21.15.59:0/1962952386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9c162400 0x7f4b9c0cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.347+0000 7f4b9a59c640 1 --2- 172.21.15.59:0/1962952386 >> v2:172.21.15.164:3300/0 conn(0x7f4b9c162a00 0x7f4b9c0d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.347+0000 7f4b99d9b640 1 -- 172.21.15.59:0/1962952386 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187698644 0 0) 0x7f4b9c0b1070 con 0x7f4b9c162020 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.347+0000 7f4b99d9b640 1 -- 172.21.15.59:0/1962952386 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b84003520 con 0x7f4b9c162020 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.347+0000 7f4b99d9b640 1 -- 172.21.15.59:0/1962952386 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 27238275 0 0) 0x7f4b84003520 con 0x7f4b9c162020 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.347+0000 7f4b9a59c640 1 -- 172.21.15.59:0/1962952386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9c162400 msgr2=0x7f4b9c0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.347+0000 7f4b9a59c640 1 --2- 172.21.15.59:0/1962952386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9c162400 0x7f4b9c0cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.347+0000 7f4b9a59c640 1 -- 172.21.15.59:0/1962952386 >> v1:172.21.15.59:6789/0 conn(0x7f4b9c162020 legacy=0x7f4b9c0b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.347+0000 7f4b9a59c640 1 -- 172.21.15.59:0/1962952386 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c0d2500 con 0x7f4b9c162a00 2024-03-06T07:32:26.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.347+0000 7f4b9b59e640 1 --2- 172.21.15.59:0/1962952386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9c162400 0x7f4b9c0cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.348+0000 7f4b9a59c640 1 --2- 172.21.15.59:0/1962952386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9c162a00 0x7f4b9c0d4830 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c0d2f20 tx=0x7f4b8c02b800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25293185524008dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.348+0000 7f4b99d9b640 1 -- 172.21.15.59:0/1962952386 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c037070 con 0x7f4b9c162a00 2024-03-06T07:32:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.348+0000 7f4b99d9b640 1 -- 172.21.15.59:0/1962952386 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8c030510 con 0x7f4b9c162a00 2024-03-06T07:32:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.348+0000 7f4b99d9b640 1 -- 172.21.15.59:0/1962952386 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b90001670 con 0x7f4b9c162a00 2024-03-06T07:32:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.348+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1962952386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9c162a00 msgr2=0x7f4b9c0d4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.348+0000 7f4ba11d7640 1 --2- 172.21.15.59:0/1962952386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9c162a00 0x7f4b9c0d4830 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c0d2f20 tx=0x7f4b8c02b800 comp rx=0 tx=0).stop 2024-03-06T07:32:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.349+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1962952386 shutdown_connections 2024-03-06T07:32:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.349+0000 7f4ba11d7640 1 --2- 172.21.15.59:0/1962952386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9c162400 0x7f4b9c0cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.349+0000 7f4ba11d7640 1 --2- 172.21.15.59:0/1962952386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9c162a00 0x7f4b9c0d4830 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.349+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1962952386 >> 172.21.15.59:0/1962952386 conn(0x7f4b9c0d3d10 msgr2=0x7f4b9c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.349+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1962952386 shutdown_connections 2024-03-06T07:32:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.349+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1962952386 wait complete. 2024-03-06T07:32:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.350+0000 7f4ba11d7640 1 Processor -- start 2024-03-06T07:32:26.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.350+0000 7f4ba11d7640 1 -- start start 2024-03-06T07:32:26.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4ba11d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b9c162020 0x7f4b9c1606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4ba11d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9c162400 0x7f4b9c0c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4ba11d7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9c162a00 0x7f4b9c14cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4ba11d7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c0cec00 con 0x7f4b9c162020 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4ba11d7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c0d2ac0 con 0x7f4b9c162a00 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4ba11d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c0cf940 con 0x7f4b9c162400 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4b9ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b9c162020 0x7f4b9c1606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4b9ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b9c162020 0x7f4b9c1606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51290/0 (socket says 172.21.15.59:51290) 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4b9ad9d640 1 -- 172.21.15.59:0/1643642523 learned_addr learned my addr 172.21.15.59:0/1643642523 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4b9a59c640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9c162400 0x7f4b9c0c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4b9b59e640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9c162a00 0x7f4b9c14cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4b9ad9d640 1 -- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9c162400 msgr2=0x7f4b9c0c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4b9ad9d640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9c162400 0x7f4b9c0c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4b9ad9d640 1 -- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9c162a00 msgr2=0x7f4b9c14cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4b9ad9d640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9c162a00 0x7f4b9c14cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4b9ad9d640 1 -- 172.21.15.59:0/1643642523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c0b6c60 con 0x7f4b9c162020 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.351+0000 7f4b9ad9d640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b9c162020 0x7f4b9c1606a0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c750 tx=0x7f4b9000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.352+0000 7f4b837fe640 1 -- 172.21.15.59:0/1643642523 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b90005ad0 con 0x7f4b9c162020 2024-03-06T07:32:26.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.352+0000 7f4b837fe640 1 -- 172.21.15.59:0/1643642523 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b90005c90 con 0x7f4b9c162020 2024-03-06T07:32:26.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.352+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1643642523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9c0b6f40 con 0x7f4b9c162020 2024-03-06T07:32:26.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.352+0000 7f4b837fe640 1 -- 172.21.15.59:0/1643642523 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b900072b0 con 0x7f4b9c162020 2024-03-06T07:32:26.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.352+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1643642523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9c0b68f0 con 0x7f4b9c162020 2024-03-06T07:32:26.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.353+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1643642523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9c04f330 con 0x7f4b9c162020 2024-03-06T07:32:26.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.353+0000 7f4b837fe640 1 -- 172.21.15.59:0/1643642523 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b90007450 con 0x7f4b9c162020 2024-03-06T07:32:26.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.353+0000 7f4b837fe640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b8806b4d0 0x7f4b8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.353+0000 7f4b837fe640 1 -- 172.21.15.59:0/1643642523 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4b9008a0f0 con 0x7f4b9c162020 2024-03-06T07:32:26.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.354+0000 7f4b9a59c640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b8806b4d0 0x7f4b8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.354+0000 7f4b9a59c640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b8806b4d0 0x7f4b8806d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c034000 tx=0x7f4b8c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.358+0000 7f4b837fe640 1 -- 172.21.15.59:0/1643642523 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b90058f50 con 0x7f4b9c162020 2024-03-06T07:32:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.577+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1643642523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b9c073380 con 0x7f4b9c162020 2024-03-06T07:32:26.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.577+0000 7f4b837fe640 1 -- 172.21.15.59:0/1643642523 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f4b90058850 con 0x7f4b9c162020 2024-03-06T07:32:26.579 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:26.579 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:26.580 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b8806b4d0 msgr2=0x7f4b8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b8806b4d0 0x7f4b8806d990 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c034000 tx=0x7f4b8c008030 comp rx=0 tx=0).stop 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1643642523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b9c162020 msgr2=0x7f4b9c1606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b9c162020 0x7f4b9c1606a0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c750 tx=0x7f4b9000cc10 comp rx=0 tx=0).stop 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1643642523 shutdown_connections 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4b8806b4d0 0x7f4b8806d990 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9c162a00 0x7f4b9c14cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9c162400 0x7f4b9c0c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 --2- 172.21.15.59:0/1643642523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4b9c162020 0x7f4b9c1606a0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1643642523 >> 172.21.15.59:0/1643642523 conn(0x7f4b9c0d3d10 msgr2=0x7f4b9c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1643642523 shutdown_connections 2024-03-06T07:32:26.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:26.579+0000 7f4ba11d7640 1 -- 172.21.15.59:0/1643642523 wait complete. 2024-03-06T07:32:26.793 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:26.793 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:27.793 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:27.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.893+0000 7fce4053e640 1 Processor -- start 2024-03-06T07:32:27.895 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.893+0000 7fce4053e640 1 -- start start 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce4053e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fce3815dfe0 0x7fce380cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce4053e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce380b3430 con 0x7fce380cf830 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce4053e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce380b3710 con 0x7fce380d29c0 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce4053e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce380b31e0 con 0x7fce3815dfe0 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce3e2da640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fce3815dfe0 0x7fce380cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce3dad9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fce380cf830 0x7fce380d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39694/0 (socket says 172.21.15.59:39694) 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce3e2da640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fce3815dfe0 0x7fce380cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51304/0 (socket says 172.21.15.59:51304) 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce3dad9640 1 -- 172.21.15.59:0/3103147595 learned_addr learned my addr 172.21.15.59:0/3103147595 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce3e2da640 1 -- 172.21.15.59:0/3103147595 >> v1:172.21.15.164:6789/0 conn(0x7fce380d29c0 legacy=0x7fce380b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce3d2d8640 1 -- 172.21.15.59:0/3103147595 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032466084 0 0) 0x7fce380b3430 con 0x7fce380cf830 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce3e2da640 1 -- 172.21.15.59:0/3103147595 >> v1:172.21.15.59:6789/0 conn(0x7fce380cf830 legacy=0x7fce380d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.894+0000 7fce3e2da640 1 -- 172.21.15.59:0/3103147595 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce380b39f0 con 0x7fce3815dfe0 2024-03-06T07:32:27.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.895+0000 7fce3e2da640 1 --2- 172.21.15.59:0/3103147595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce3815dfe0 0x7fce380cf2f0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fce380b3360 tx=0x7fce2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4c4d8f4dbf3ab6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.895+0000 7fce3d2d8640 1 -- 172.21.15.59:0/3103147595 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce28035020 con 0x7fce3815dfe0 2024-03-06T07:32:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.895+0000 7fce3d2d8640 1 -- 172.21.15.59:0/3103147595 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce28031830 con 0x7fce3815dfe0 2024-03-06T07:32:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.895+0000 7fce3d2d8640 1 -- 172.21.15.59:0/3103147595 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce28031c00 con 0x7fce3815dfe0 2024-03-06T07:32:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.896+0000 7fce4053e640 1 -- 172.21.15.59:0/3103147595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce3815dfe0 msgr2=0x7fce380cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.896+0000 7fce4053e640 1 --2- 172.21.15.59:0/3103147595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce3815dfe0 0x7fce380cf2f0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fce380b3360 tx=0x7fce2802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.896+0000 7fce4053e640 1 -- 172.21.15.59:0/3103147595 shutdown_connections 2024-03-06T07:32:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.896+0000 7fce4053e640 1 --2- 172.21.15.59:0/3103147595 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce3815dfe0 0x7fce380cf2f0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.896+0000 7fce4053e640 1 -- 172.21.15.59:0/3103147595 >> 172.21.15.59:0/3103147595 conn(0x7fce380d3cd0 msgr2=0x7fce380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.896+0000 7fce4053e640 1 -- 172.21.15.59:0/3103147595 shutdown_connections 2024-03-06T07:32:27.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.896+0000 7fce4053e640 1 -- 172.21.15.59:0/3103147595 wait complete. 2024-03-06T07:32:27.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.897+0000 7fce4053e640 1 Processor -- start 2024-03-06T07:32:27.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.897+0000 7fce4053e640 1 -- start start 2024-03-06T07:32:27.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce4053e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce380cf830 0x7fce38150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce4053e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce380d29c0 0x7fce3809fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce4053e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce3815dfe0 0x7fce380a0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce4053e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3815e3c0 con 0x7fce3815dfe0 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce4053e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce380b3d60 con 0x7fce380cf830 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce4053e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3815e540 con 0x7fce380d29c0 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce3eadb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce3815dfe0 0x7fce380a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce3eadb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce3815dfe0 0x7fce380a0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51312/0 (socket says 172.21.15.59:51312) 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce3eadb640 1 -- 172.21.15.59:0/414967860 learned_addr learned my addr 172.21.15.59:0/414967860 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce3e2da640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce380cf830 0x7fce38150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce3dad9640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce380d29c0 0x7fce3809fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce3eadb640 1 -- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce380d29c0 msgr2=0x7fce3809fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce3eadb640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce380d29c0 0x7fce3809fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce3eadb640 1 -- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce380cf830 msgr2=0x7fce38150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce3eadb640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce380cf830 0x7fce38150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce3eadb640 1 -- 172.21.15.59:0/414967860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce380a0850 con 0x7fce3815dfe0 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.898+0000 7fce3e2da640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce380cf830 0x7fce38150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.899+0000 7fce3eadb640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce3815dfe0 0x7fce380a0010 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fce3000c700 tx=0x7fce3000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.899+0000 7fce2effd640 1 -- 172.21.15.59:0/414967860 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30016020 con 0x7fce3815dfe0 2024-03-06T07:32:27.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.899+0000 7fce2effd640 1 -- 172.21.15.59:0/414967860 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3000aa50 con 0x7fce3815dfe0 2024-03-06T07:32:27.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.899+0000 7fce4053e640 1 -- 172.21.15.59:0/414967860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce380a0b30 con 0x7fce3815dfe0 2024-03-06T07:32:27.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.899+0000 7fce2effd640 1 -- 172.21.15.59:0/414967860 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3000ae40 con 0x7fce3815dfe0 2024-03-06T07:32:27.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.899+0000 7fce4053e640 1 -- 172.21.15.59:0/414967860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce380a10f0 con 0x7fce3815dfe0 2024-03-06T07:32:27.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.900+0000 7fce4053e640 1 -- 172.21.15.59:0/414967860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3804f330 con 0x7fce3815dfe0 2024-03-06T07:32:27.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.900+0000 7fce2effd640 1 -- 172.21.15.59:0/414967860 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce30019020 con 0x7fce3815dfe0 2024-03-06T07:32:27.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.901+0000 7fce2effd640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1c06b4d0 0x7fce1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.901+0000 7fce2effd640 1 -- 172.21.15.59:0/414967860 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fce3008a410 con 0x7fce3815dfe0 2024-03-06T07:32:27.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.901+0000 7fce3e2da640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1c06b4d0 0x7fce1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.902+0000 7fce3e2da640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1c06b4d0 0x7fce1c06d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fce28036000 tx=0x7fce28008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:27.905+0000 7fce2effd640 1 -- 172.21.15.59:0/414967860 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce30020020 con 0x7fce3815dfe0 2024-03-06T07:32:28.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.124+0000 7fce4053e640 1 -- 172.21.15.59:0/414967860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce380b4e50 con 0x7fce3815dfe0 2024-03-06T07:32:28.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.124+0000 7fce2effd640 1 -- 172.21.15.59:0/414967860 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fce30059300 con 0x7fce3815dfe0 2024-03-06T07:32:28.127 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:28.127 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:28.127 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:28.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.126+0000 7fce4053e640 1 -- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1c06b4d0 msgr2=0x7fce1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.126+0000 7fce4053e640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1c06b4d0 0x7fce1c06d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fce28036000 tx=0x7fce28008030 comp rx=0 tx=0).stop 2024-03-06T07:32:28.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.126+0000 7fce4053e640 1 -- 172.21.15.59:0/414967860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce3815dfe0 msgr2=0x7fce380a0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.126+0000 7fce4053e640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce3815dfe0 0x7fce380a0010 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fce3000c700 tx=0x7fce3000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.127+0000 7fce4053e640 1 -- 172.21.15.59:0/414967860 shutdown_connections 2024-03-06T07:32:28.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.127+0000 7fce4053e640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fce1c06b4d0 0x7fce1c06d990 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.127+0000 7fce4053e640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce3815dfe0 0x7fce380a0010 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.127+0000 7fce4053e640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce380d29c0 0x7fce3809fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.129 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.127+0000 7fce4053e640 1 --2- 172.21.15.59:0/414967860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce380cf830 0x7fce38150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.127+0000 7fce4053e640 1 -- 172.21.15.59:0/414967860 >> 172.21.15.59:0/414967860 conn(0x7fce380d3cd0 msgr2=0x7fce3806cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.127+0000 7fce4053e640 1 -- 172.21.15.59:0/414967860 shutdown_connections 2024-03-06T07:32:28.130 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:28.127+0000 7fce4053e640 1 -- 172.21.15.59:0/414967860 wait complete. 2024-03-06T07:32:28.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:28.343 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:29.344 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:29.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.444+0000 7fa8f5b25640 1 Processor -- start 2024-03-06T07:32:29.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.444+0000 7fa8f5b25640 1 -- start start 2024-03-06T07:32:29.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8f5b25640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f015fa10 0x7fa8f00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8f5b25640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8f00d16e0 0x7fa8f00d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8f5b25640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f00cebe0 con 0x7fa8f015bdd0 2024-03-06T07:32:29.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8f5b25640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f00ceec0 con 0x7fa8f00d16e0 2024-03-06T07:32:29.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8f5b25640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00cf1a0 con 0x7fa8f015fa10 2024-03-06T07:32:29.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8effff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8f00d16e0 0x7fa8f00d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8effff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8f00d16e0 0x7fa8f00d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51328/0 (socket says 172.21.15.59:51328) 2024-03-06T07:32:29.449 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8effff640 1 -- 172.21.15.59:0/905170662 learned_addr learned my addr 172.21.15.59:0/905170662 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:29.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8eeffd640 1 --2- 172.21.15.59:0/905170662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f015fa10 0x7fa8f00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8effff640 1 -- 172.21.15.59:0/905170662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f015fa10 msgr2=0x7fa8f00ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8effff640 1 --2- 172.21.15.59:0/905170662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f015fa10 0x7fa8f00ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8effff640 1 -- 172.21.15.59:0/905170662 >> v1:172.21.15.164:6789/0 conn(0x7fa8f015bdd0 legacy=0x7fa8f00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.445+0000 7fa8effff640 1 -- 172.21.15.59:0/905170662 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f00cc8e0 con 0x7fa8f00d16e0 2024-03-06T07:32:29.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8effff640 1 --2- 172.21.15.59:0/905170662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00d16e0 0x7fa8f00d1b30 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009e50 tx=0x7fa8e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91585a0439d310f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8ee7fc640 1 -- 172.21.15.59:0/905170662 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e802ca70 con 0x7fa8f00d16e0 2024-03-06T07:32:29.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8ee7fc640 1 -- 172.21.15.59:0/905170662 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e802cc10 con 0x7fa8f00d16e0 2024-03-06T07:32:29.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8ee7fc640 1 -- 172.21.15.59:0/905170662 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e80343f0 con 0x7fa8f00d16e0 2024-03-06T07:32:29.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8f5b25640 1 -- 172.21.15.59:0/905170662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00d16e0 msgr2=0x7fa8f00d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8f5b25640 1 --2- 172.21.15.59:0/905170662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00d16e0 0x7fa8f00d1b30 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009e50 tx=0x7fa8e802bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8f5b25640 1 -- 172.21.15.59:0/905170662 shutdown_connections 2024-03-06T07:32:29.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8f5b25640 1 --2- 172.21.15.59:0/905170662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f00d16e0 0x7fa8f00d1b30 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8f5b25640 1 --2- 172.21.15.59:0/905170662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f015fa10 0x7fa8f00ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8f5b25640 1 -- 172.21.15.59:0/905170662 >> 172.21.15.59:0/905170662 conn(0x7fa8f009a9f0 msgr2=0x7fa8f009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8f5b25640 1 -- 172.21.15.59:0/905170662 shutdown_connections 2024-03-06T07:32:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.447+0000 7fa8f5b25640 1 -- 172.21.15.59:0/905170662 wait complete. 2024-03-06T07:32:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.448+0000 7fa8f5b25640 1 Processor -- start 2024-03-06T07:32:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.449+0000 7fa8f5b25640 1 -- start start 2024-03-06T07:32:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.449+0000 7fa8f5b25640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8f00d16e0 0x7fa8f00c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.449+0000 7fa8f5b25640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f015bdd0 0x7fa8f015e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.449+0000 7fa8f5b25640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f015fa10 0x7fa8f015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.449+0000 7fa8f5b25640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f015c1b0 con 0x7fa8f015fa10 2024-03-06T07:32:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.449+0000 7fa8f5b25640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00cc5d0 con 0x7fa8f00d16e0 2024-03-06T07:32:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.449+0000 7fa8f5b25640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f00ce7b0 con 0x7fa8f015bdd0 2024-03-06T07:32:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.449+0000 7fa8effff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f015fa10 0x7fa8f015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.449+0000 7fa8effff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f015fa10 0x7fa8f015c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51330/0 (socket says 172.21.15.59:51330) 2024-03-06T07:32:29.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.449+0000 7fa8effff640 1 -- 172.21.15.59:0/838255158 learned_addr learned my addr 172.21.15.59:0/838255158 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:29.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8effff640 1 -- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f015bdd0 msgr2=0x7fa8f015e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8ef7fe640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8f00d16e0 0x7fa8f00c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8eeffd640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f015bdd0 0x7fa8f015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8effff640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f015bdd0 0x7fa8f015e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8effff640 1 -- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8f00d16e0 msgr2=0x7fa8f00c4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8effff640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8f00d16e0 0x7fa8f00c4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8effff640 1 -- 172.21.15.59:0/838255158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f015ec20 con 0x7fa8f015fa10 2024-03-06T07:32:29.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8effff640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f015fa10 0x7fa8f015c330 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009e50 tx=0x7fa8e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8e7fff640 1 -- 172.21.15.59:0/838255158 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8043020 con 0x7fa8f015fa10 2024-03-06T07:32:29.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8e7fff640 1 -- 172.21.15.59:0/838255158 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e803e450 con 0x7fa8f015fa10 2024-03-06T07:32:29.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8e7fff640 1 -- 172.21.15.59:0/838255158 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8009510 con 0x7fa8f015fa10 2024-03-06T07:32:29.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8f5b25640 1 -- 172.21.15.59:0/838255158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f015ef00 con 0x7fa8f015fa10 2024-03-06T07:32:29.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.450+0000 7fa8f5b25640 1 -- 172.21.15.59:0/838255158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f00c4d70 con 0x7fa8f015fa10 2024-03-06T07:32:29.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.451+0000 7fa8f5b25640 1 -- 172.21.15.59:0/838255158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f00b5720 con 0x7fa8f015fa10 2024-03-06T07:32:29.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.452+0000 7fa8e7fff640 1 -- 172.21.15.59:0/838255158 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8e8046020 con 0x7fa8f015fa10 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.452+0000 7fa8e7fff640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 0x7fa8d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.452+0000 7fa8e7fff640 1 -- 172.21.15.59:0/838255158 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa8e80ac9c0 con 0x7fa8f015fa10 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.453+0000 7fa8ef7fe640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 0x7fa8d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.453+0000 7fa8ef7fe640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 0x7fa8d006d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80015a0 tx=0x7fa8d8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.456+0000 7fa8e7fff640 1 -- 172.21.15.59:0/838255158 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8e80aee20 con 0x7fa8f015fa10 2024-03-06T07:32:29.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.675+0000 7fa8f5b25640 1 -- 172.21.15.59:0/838255158 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8f004f330 con 0x7fa8f015fa10 2024-03-06T07:32:29.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.676+0000 7fa8e7fff640 1 -- 172.21.15.59:0/838255158 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fa8e804d030 con 0x7fa8f015fa10 2024-03-06T07:32:29.678 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:29.678 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:29.678 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:29.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.677+0000 7fa8f5b25640 1 -- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 msgr2=0x7fa8d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.677+0000 7fa8f5b25640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 0x7fa8d006d990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80015a0 tx=0x7fa8d8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:29.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.677+0000 7fa8f5b25640 1 -- 172.21.15.59:0/838255158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f015fa10 msgr2=0x7fa8f015c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.677+0000 7fa8f5b25640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f015fa10 0x7fa8f015c330 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009e50 tx=0x7fa8e8004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:29.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.677+0000 7fa8f5b25640 1 -- 172.21.15.59:0/838255158 shutdown_connections 2024-03-06T07:32:29.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.677+0000 7fa8f5b25640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa8d006b4d0 0x7fa8d006d990 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.677+0000 7fa8f5b25640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8f015fa10 0x7fa8f015c330 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.678+0000 7fa8f5b25640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8f015bdd0 0x7fa8f015e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.678+0000 7fa8f5b25640 1 --2- 172.21.15.59:0/838255158 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8f00d16e0 0x7fa8f00c4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.678+0000 7fa8f5b25640 1 -- 172.21.15.59:0/838255158 >> 172.21.15.59:0/838255158 conn(0x7fa8f009a9f0 msgr2=0x7fa8f00d03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.678+0000 7fa8f5b25640 1 -- 172.21.15.59:0/838255158 shutdown_connections 2024-03-06T07:32:29.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:29.678+0000 7fa8f5b25640 1 -- 172.21.15.59:0/838255158 wait complete. 2024-03-06T07:32:29.892 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:29.892 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:30.893 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:30.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.993+0000 7fdf5997a640 1 Processor -- start 2024-03-06T07:32:30.995 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.993+0000 7fdf5997a640 1 -- start start 2024-03-06T07:32:30.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.994+0000 7fdf5997a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf540d4760 0x7fdf540b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.996 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.994+0000 7fdf5997a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf540d25d0 con 0x7fdf540d0a80 2024-03-06T07:32:30.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.994+0000 7fdf5997a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf540d2870 con 0x7fdf540b4170 2024-03-06T07:32:30.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.994+0000 7fdf5997a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf540d4bf0 con 0x7fdf540d4760 2024-03-06T07:32:30.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.994+0000 7fdf527fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdf540d4760 0x7fdf540b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.994+0000 7fdf537fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdf540d0a80 0x7fdf540cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39706/0 (socket says 172.21.15.59:39706) 2024-03-06T07:32:30.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.994+0000 7fdf537fe640 1 -- 172.21.15.59:0/1962684889 learned_addr learned my addr 172.21.15.59:0/1962684889 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:30.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.994+0000 7fdf527fc640 1 --2- 172.21.15.59:0/1962684889 >> v2:172.21.15.59:3300/0 conn(0x7fdf540d4760 0x7fdf540b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51334/0 (socket says 172.21.15.59:51334) 2024-03-06T07:32:30.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.994+0000 7fdf527fc640 1 -- 172.21.15.59:0/1962684889 >> v1:172.21.15.164:6789/0 conn(0x7fdf540b4170 legacy=0x7fdf540d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.994+0000 7fdf527fc640 1 -- 172.21.15.59:0/1962684889 >> v1:172.21.15.59:6789/0 conn(0x7fdf540d0a80 legacy=0x7fdf540cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.994+0000 7fdf527fc640 1 -- 172.21.15.59:0/1962684889 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540d2b50 con 0x7fdf540d4760 2024-03-06T07:32:30.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.995+0000 7fdf527fc640 1 --2- 172.21.15.59:0/1962684889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf540d4760 0x7fdf540b6a90 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fdf480097a0 tx=0x7fdf4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea3d58b51a3d68f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.995+0000 7fdf51ffb640 1 -- 172.21.15.59:0/1962684889 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48035020 con 0x7fdf540d4760 2024-03-06T07:32:30.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.995+0000 7fdf51ffb640 1 -- 172.21.15.59:0/1962684889 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf48031830 con 0x7fdf540d4760 2024-03-06T07:32:30.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.995+0000 7fdf51ffb640 1 -- 172.21.15.59:0/1962684889 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48031c00 con 0x7fdf540d4760 2024-03-06T07:32:30.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.996+0000 7fdf5997a640 1 -- 172.21.15.59:0/1962684889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf540d4760 msgr2=0x7fdf540b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.996+0000 7fdf5997a640 1 --2- 172.21.15.59:0/1962684889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf540d4760 0x7fdf540b6a90 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fdf480097a0 tx=0x7fdf4802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.996+0000 7fdf5997a640 1 -- 172.21.15.59:0/1962684889 shutdown_connections 2024-03-06T07:32:31.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.996+0000 7fdf5997a640 1 --2- 172.21.15.59:0/1962684889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf540d4760 0x7fdf540b6a90 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.996+0000 7fdf5997a640 1 -- 172.21.15.59:0/1962684889 >> 172.21.15.59:0/1962684889 conn(0x7fdf540d3c40 msgr2=0x7fdf540d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.996+0000 7fdf5997a640 1 -- 172.21.15.59:0/1962684889 shutdown_connections 2024-03-06T07:32:31.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.996+0000 7fdf5997a640 1 -- 172.21.15.59:0/1962684889 wait complete. 2024-03-06T07:32:31.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.997+0000 7fdf5997a640 1 Processor -- start 2024-03-06T07:32:31.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.997+0000 7fdf5997a640 1 -- start start 2024-03-06T07:32:31.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf5997a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf540b4170 0x7fdf54153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf5997a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf540d0a80 0x7fdf540b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf5997a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf540b24d0 0x7fdf540b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf5997a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540b4dd0 con 0x7fdf540d0a80 2024-03-06T07:32:31.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf5997a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540ce9e0 con 0x7fdf540b24d0 2024-03-06T07:32:31.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf5997a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540b4f50 con 0x7fdf540b4170 2024-03-06T07:32:31.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf527fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf540d0a80 0x7fdf540b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf527fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf540d0a80 0x7fdf540b1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51338/0 (socket says 172.21.15.59:51338) 2024-03-06T07:32:31.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf527fc640 1 -- 172.21.15.59:0/2929414238 learned_addr learned my addr 172.21.15.59:0/2929414238 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:31.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf537fe640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf540b24d0 0x7fdf540b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf52ffd640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf540b4170 0x7fdf54153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf527fc640 1 -- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf540b4170 msgr2=0x7fdf54153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf527fc640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf540b4170 0x7fdf54153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf527fc640 1 -- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf540b24d0 msgr2=0x7fdf540b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf527fc640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf540b24d0 0x7fdf540b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf527fc640 1 -- 172.21.15.59:0/2929414238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540b14e0 con 0x7fdf540d0a80 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf537fe640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf540b24d0 0x7fdf540b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf52ffd640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf540b4170 0x7fdf54153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.998+0000 7fdf527fc640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf540d0a80 0x7fdf540b1f90 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fdf48000c00 tx=0x7fdf480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.999+0000 7fdf3bfff640 1 -- 172.21.15.59:0/2929414238 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48040070 con 0x7fdf540d0a80 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.999+0000 7fdf3bfff640 1 -- 172.21.15.59:0/2929414238 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf48039a00 con 0x7fdf540d0a80 2024-03-06T07:32:31.005 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.999+0000 7fdf5997a640 1 -- 172.21.15.59:0/2929414238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf540b17c0 con 0x7fdf540d0a80 2024-03-06T07:32:31.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.999+0000 7fdf3bfff640 1 -- 172.21.15.59:0/2929414238 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48035040 con 0x7fdf540d0a80 2024-03-06T07:32:31.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:30.999+0000 7fdf5997a640 1 -- 172.21.15.59:0/2929414238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf540b0d30 con 0x7fdf540d0a80 2024-03-06T07:32:31.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.000+0000 7fdf5997a640 1 -- 172.21.15.59:0/2929414238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5404f330 con 0x7fdf540d0a80 2024-03-06T07:32:31.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.000+0000 7fdf3bfff640 1 -- 172.21.15.59:0/2929414238 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf4804b060 con 0x7fdf540d0a80 2024-03-06T07:32:31.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.001+0000 7fdf3bfff640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf4006b4d0 0x7fdf4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.001+0000 7fdf3bfff640 1 -- 172.21.15.59:0/2929414238 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdf480abbb0 con 0x7fdf540d0a80 2024-03-06T07:32:31.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.001+0000 7fdf52ffd640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf4006b4d0 0x7fdf4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.002+0000 7fdf52ffd640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf4006b4d0 0x7fdf4006d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fdf44003eb0 tx=0x7fdf44009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.005+0000 7fdf3bfff640 1 -- 172.21.15.59:0/2929414238 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf4807aaa0 con 0x7fdf540d0a80 2024-03-06T07:32:31.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.227+0000 7fdf5997a640 1 -- 172.21.15.59:0/2929414238 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf540b28b0 con 0x7fdf540d0a80 2024-03-06T07:32:31.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.227+0000 7fdf3bfff640 1 -- 172.21.15.59:0/2929414238 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fdf4807a3a0 con 0x7fdf540d0a80 2024-03-06T07:32:31.232 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:31.232 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:31.233 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:31.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 -- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf4006b4d0 msgr2=0x7fdf4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf4006b4d0 0x7fdf4006d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fdf44003eb0 tx=0x7fdf44009040 comp rx=0 tx=0).stop 2024-03-06T07:32:31.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 -- 172.21.15.59:0/2929414238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf540d0a80 msgr2=0x7fdf540b1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf540d0a80 0x7fdf540b1f90 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fdf48000c00 tx=0x7fdf480046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 -- 172.21.15.59:0/2929414238 shutdown_connections 2024-03-06T07:32:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdf4006b4d0 0x7fdf4006d990 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf540b24d0 0x7fdf540b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf540d0a80 0x7fdf540b1f90 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 --2- 172.21.15.59:0/2929414238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf540b4170 0x7fdf54153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 -- 172.21.15.59:0/2929414238 >> 172.21.15.59:0/2929414238 conn(0x7fdf540d3c40 msgr2=0x7fdf540ccd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 -- 172.21.15.59:0/2929414238 shutdown_connections 2024-03-06T07:32:31.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:31.229+0000 7fdf5997a640 1 -- 172.21.15.59:0/2929414238 wait complete. 2024-03-06T07:32:31.443 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:31.443 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:32.444 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:32.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.544+0000 7fdd34afb640 1 Processor -- start 2024-03-06T07:32:32.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.545+0000 7fdd34afb640 1 -- start start 2024-03-06T07:32:32.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd34afb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdd300afdd0 0x7fdd3015c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd34afb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdd300d14f0 0x7fdd3006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd34afb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3015e740 con 0x7fdd30158920 2024-03-06T07:32:32.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd34afb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd3006fae0 con 0x7fdd300d14f0 2024-03-06T07:32:32.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd34afb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd3015f5f0 con 0x7fdd300afdd0 2024-03-06T07:32:32.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd2ed9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdd300d14f0 0x7fdd3006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd2e59c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdd30158920 0x7fdd300af6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:39710/0 (socket says 172.21.15.59:39710) 2024-03-06T07:32:32.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd2e59c640 1 -- 172.21.15.59:0/3685734553 learned_addr learned my addr 172.21.15.59:0/3685734553 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:32.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd27fff640 1 --2- 172.21.15.59:0/3685734553 >> v2:172.21.15.164:3300/0 conn(0x7fdd300afdd0 0x7fdd3015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd2ed9d640 1 -- 172.21.15.59:0/3685734553 >> v2:172.21.15.164:3300/0 conn(0x7fdd300afdd0 msgr2=0x7fdd3015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd2ed9d640 1 --2- 172.21.15.59:0/3685734553 >> v2:172.21.15.164:3300/0 conn(0x7fdd300afdd0 0x7fdd3015c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd2ed9d640 1 -- 172.21.15.59:0/3685734553 >> v1:172.21.15.59:6789/0 conn(0x7fdd30158920 legacy=0x7fdd300af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd2ed9d640 1 -- 172.21.15.59:0/3685734553 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3015e9e0 con 0x7fdd300d14f0 2024-03-06T07:32:32.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.546+0000 7fdd2dd9b640 1 -- 172.21.15.59:0/3685734553 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915881022 0 0) 0x7fdd3015e740 con 0x7fdd30158920 2024-03-06T07:32:32.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.547+0000 7fdd2ed9d640 1 --2- 172.21.15.59:0/3685734553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd300d14f0 0x7fdd3006f5a0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fdd300d1a60 tx=0x7fdd2802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb54df3b34cd40b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.547+0000 7fdd2dd9b640 1 -- 172.21.15.59:0/3685734553 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28039070 con 0x7fdd300d14f0 2024-03-06T07:32:32.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.547+0000 7fdd2dd9b640 1 -- 172.21.15.59:0/3685734553 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2802cc80 con 0x7fdd300d14f0 2024-03-06T07:32:32.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.547+0000 7fdd2dd9b640 1 -- 172.21.15.59:0/3685734553 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28034660 con 0x7fdd300d14f0 2024-03-06T07:32:32.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.548+0000 7fdd34afb640 1 -- 172.21.15.59:0/3685734553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd300d14f0 msgr2=0x7fdd3006f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.548+0000 7fdd34afb640 1 --2- 172.21.15.59:0/3685734553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd300d14f0 0x7fdd3006f5a0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fdd300d1a60 tx=0x7fdd2802bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.548+0000 7fdd34afb640 1 -- 172.21.15.59:0/3685734553 shutdown_connections 2024-03-06T07:32:32.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.548+0000 7fdd34afb640 1 --2- 172.21.15.59:0/3685734553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd300d14f0 0x7fdd3006f5a0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.548+0000 7fdd34afb640 1 --2- 172.21.15.59:0/3685734553 >> v2:172.21.15.164:3300/0 conn(0x7fdd300afdd0 0x7fdd3015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.548+0000 7fdd34afb640 1 -- 172.21.15.59:0/3685734553 >> 172.21.15.59:0/3685734553 conn(0x7fdd300d0960 msgr2=0x7fdd300d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.548+0000 7fdd34afb640 1 -- 172.21.15.59:0/3685734553 shutdown_connections 2024-03-06T07:32:32.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.548+0000 7fdd34afb640 1 -- 172.21.15.59:0/3685734553 wait complete. 2024-03-06T07:32:32.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.549+0000 7fdd34afb640 1 Processor -- start 2024-03-06T07:32:32.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.549+0000 7fdd34afb640 1 -- start start 2024-03-06T07:32:32.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd34afb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd300afdd0 0x7fdd30142d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd34afb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd300d14f0 0x7fdd300a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd34afb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd30158920 0x7fdd300a9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd34afb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3015f280 con 0x7fdd300d14f0 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd34afb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3015efa0 con 0x7fdd300afdd0 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd34afb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3015b400 con 0x7fdd30158920 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd27fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd300d14f0 0x7fdd300a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd27fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd300d14f0 0x7fdd300a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51358/0 (socket says 172.21.15.59:51358) 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd27fff640 1 -- 172.21.15.59:0/2433685728 learned_addr learned my addr 172.21.15.59:0/2433685728 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd2ed9d640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd30158920 0x7fdd300a9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd27fff640 1 -- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd30158920 msgr2=0x7fdd300a9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd27fff640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd30158920 0x7fdd300a9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd27fff640 1 -- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd300afdd0 msgr2=0x7fdd30142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd2e59c640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd300afdd0 0x7fdd30142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd27fff640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd300afdd0 0x7fdd30142d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.550+0000 7fdd27fff640 1 -- 172.21.15.59:0/2433685728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd301440c0 con 0x7fdd300d14f0 2024-03-06T07:32:32.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.551+0000 7fdd27fff640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd300d14f0 0x7fdd300a9450 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800e9e0 tx=0x7fdd1800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.551+0000 7fdd26ffd640 1 -- 172.21.15.59:0/2433685728 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18017070 con 0x7fdd300d14f0 2024-03-06T07:32:32.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.551+0000 7fdd26ffd640 1 -- 172.21.15.59:0/2433685728 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1800c9d0 con 0x7fdd300d14f0 2024-03-06T07:32:32.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.551+0000 7fdd34afb640 1 -- 172.21.15.59:0/2433685728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd301443a0 con 0x7fdd300d14f0 2024-03-06T07:32:32.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.551+0000 7fdd26ffd640 1 -- 172.21.15.59:0/2433685728 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1800cdc0 con 0x7fdd300d14f0 2024-03-06T07:32:32.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.551+0000 7fdd34afb640 1 -- 172.21.15.59:0/2433685728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd30144960 con 0x7fdd300d14f0 2024-03-06T07:32:32.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.552+0000 7fdd34afb640 1 -- 172.21.15.59:0/2433685728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3004f330 con 0x7fdd300d14f0 2024-03-06T07:32:32.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.552+0000 7fdd26ffd640 1 -- 172.21.15.59:0/2433685728 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd1801a070 con 0x7fdd300d14f0 2024-03-06T07:32:32.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.553+0000 7fdd26ffd640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd1006b4d0 0x7fdd1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.553+0000 7fdd26ffd640 1 -- 172.21.15.59:0/2433685728 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdd1808a660 con 0x7fdd300d14f0 2024-03-06T07:32:32.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.553+0000 7fdd2e59c640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd1006b4d0 0x7fdd1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.557 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.553+0000 7fdd2e59c640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd1006b4d0 0x7fdd1006d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fdd20000f60 tx=0x7fdd20009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.561+0000 7fdd26ffd640 1 -- 172.21.15.59:0/2433685728 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd180594c0 con 0x7fdd300d14f0 2024-03-06T07:32:32.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.779+0000 7fdd34afb640 1 -- 172.21.15.59:0/2433685728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd30159180 con 0x7fdd300d14f0 2024-03-06T07:32:32.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.780+0000 7fdd26ffd640 1 -- 172.21.15.59:0/2433685728 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fdd18058dc0 con 0x7fdd300d14f0 2024-03-06T07:32:32.782 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:32.782 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:32.782 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.781+0000 7fdd34afb640 1 -- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd1006b4d0 msgr2=0x7fdd1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.781+0000 7fdd34afb640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd1006b4d0 0x7fdd1006d990 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fdd20000f60 tx=0x7fdd20009040 comp rx=0 tx=0).stop 2024-03-06T07:32:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.782+0000 7fdd34afb640 1 -- 172.21.15.59:0/2433685728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd300d14f0 msgr2=0x7fdd300a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.782+0000 7fdd34afb640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd300d14f0 0x7fdd300a9450 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800e9e0 tx=0x7fdd1800eea0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.782+0000 7fdd34afb640 1 -- 172.21.15.59:0/2433685728 shutdown_connections 2024-03-06T07:32:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.782+0000 7fdd34afb640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdd1006b4d0 0x7fdd1006d990 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.782+0000 7fdd34afb640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd30158920 0x7fdd300a9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.782+0000 7fdd34afb640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd300d14f0 0x7fdd300a9450 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.782+0000 7fdd34afb640 1 --2- 172.21.15.59:0/2433685728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd300afdd0 0x7fdd30142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.782+0000 7fdd34afb640 1 -- 172.21.15.59:0/2433685728 >> 172.21.15.59:0/2433685728 conn(0x7fdd300d0960 msgr2=0x7fdd300b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.782+0000 7fdd34afb640 1 -- 172.21.15.59:0/2433685728 shutdown_connections 2024-03-06T07:32:32.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:32.782+0000 7fdd34afb640 1 -- 172.21.15.59:0/2433685728 wait complete. 2024-03-06T07:32:32.996 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:32.996 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:33.997 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:34.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.097+0000 7f5001d3e640 1 Processor -- start 2024-03-06T07:32:34.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.097+0000 7f5001d3e640 1 -- start start 2024-03-06T07:32:34.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.098+0000 7f5001d3e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4ffc0b3060 0x7f4ffc15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.098+0000 7f5001d3e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc15f580 con 0x7f4ffc15bf30 2024-03-06T07:32:34.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.098+0000 7f5001d3e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc15b450 con 0x7f4ffc13dc10 2024-03-06T07:32:34.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.098+0000 7f5001d3e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc0cf3f0 con 0x7f4ffc0b3060 2024-03-06T07:32:34.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.098+0000 7f4ffaffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4ffc0b3060 0x7f4ffc15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.098+0000 7f4ffbfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4ffc15bf30 0x7f4ffc15ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34502/0 (socket says 172.21.15.59:34502) 2024-03-06T07:32:34.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.098+0000 7f4ffbfff640 1 -- 172.21.15.59:0/1875714524 learned_addr learned my addr 172.21.15.59:0/1875714524 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:34.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.098+0000 7f4ffaffd640 1 -- 172.21.15.59:0/1875714524 >> v1:172.21.15.164:6789/0 conn(0x7f4ffc13dc10 legacy=0x7f4ffc0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.098+0000 7f4ffaffd640 1 -- 172.21.15.59:0/1875714524 >> v1:172.21.15.59:6789/0 conn(0x7f4ffc15bf30 legacy=0x7f4ffc15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.098+0000 7f4ffaffd640 1 -- 172.21.15.59:0/1875714524 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc15b730 con 0x7f4ffc0b3060 2024-03-06T07:32:34.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.099+0000 7f4ffaffd640 1 --2- 172.21.15.59:0/1875714524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc0b3060 0x7f4ffc15e850 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0097a0 tx=0x7f4fec02da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9487ace85cb5589b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.099+0000 7f4ffa7fc640 1 -- 172.21.15.59:0/1875714524 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec031870 con 0x7f4ffc0b3060 2024-03-06T07:32:34.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.099+0000 7f4ffa7fc640 1 -- 172.21.15.59:0/1875714524 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec031a10 con 0x7f4ffc0b3060 2024-03-06T07:32:34.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.099+0000 7f4ffa7fc640 1 -- 172.21.15.59:0/1875714524 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec031de0 con 0x7f4ffc0b3060 2024-03-06T07:32:34.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.100+0000 7f5001d3e640 1 -- 172.21.15.59:0/1875714524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc0b3060 msgr2=0x7f4ffc15e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.100+0000 7f5001d3e640 1 --2- 172.21.15.59:0/1875714524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc0b3060 0x7f4ffc15e850 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0097a0 tx=0x7f4fec02da50 comp rx=0 tx=0).stop 2024-03-06T07:32:34.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.100+0000 7f5001d3e640 1 -- 172.21.15.59:0/1875714524 shutdown_connections 2024-03-06T07:32:34.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.100+0000 7f5001d3e640 1 --2- 172.21.15.59:0/1875714524 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc0b3060 0x7f4ffc15e850 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.100+0000 7f5001d3e640 1 -- 172.21.15.59:0/1875714524 >> 172.21.15.59:0/1875714524 conn(0x7f4ffc0cf8a0 msgr2=0x7f4ffc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.100+0000 7f5001d3e640 1 -- 172.21.15.59:0/1875714524 shutdown_connections 2024-03-06T07:32:34.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.100+0000 7f5001d3e640 1 -- 172.21.15.59:0/1875714524 wait complete. 2024-03-06T07:32:34.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.101+0000 7f5001d3e640 1 Processor -- start 2024-03-06T07:32:34.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.101+0000 7f5001d3e640 1 -- start start 2024-03-06T07:32:34.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.102+0000 7f5001d3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ffc0b3060 0x7f4ffc153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.102+0000 7f5001d3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc13dc10 0x7f4ffc154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.102+0000 7f5001d3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc15bf30 0x7f4ffc154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.102+0000 7f5001d3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc0718e0 con 0x7f4ffc13dc10 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.102+0000 7f5001d3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc15b210 con 0x7f4ffc0b3060 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.102+0000 7f5001d3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc071a60 con 0x7f4ffc15bf30 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.102+0000 7f4ffaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc13dc10 0x7f4ffc154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.102+0000 7f4ffaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc13dc10 0x7f4ffc154790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34804/0 (socket says 172.21.15.59:34804) 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.102+0000 7f4ffaffd640 1 -- 172.21.15.59:0/1973201503 learned_addr learned my addr 172.21.15.59:0/1973201503 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ffbfff640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc15bf30 0x7f4ffc154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ffb7fe640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ffc0b3060 0x7f4ffc153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ffaffd640 1 -- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc15bf30 msgr2=0x7f4ffc154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ffaffd640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc15bf30 0x7f4ffc154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ffaffd640 1 -- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ffc0b3060 msgr2=0x7f4ffc153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ffaffd640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ffc0b3060 0x7f4ffc153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ffaffd640 1 -- 172.21.15.59:0/1973201503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc13fb90 con 0x7f4ffc13dc10 2024-03-06T07:32:34.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ffbfff640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc15bf30 0x7f4ffc154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:34.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ffb7fe640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ffc0b3060 0x7f4ffc153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:34.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ffaffd640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc13dc10 0x7f4ffc154790 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f4fec02e060 tx=0x7f4fec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ff3fff640 1 -- 172.21.15.59:0/1973201503 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec03ad60 con 0x7f4ffc13dc10 2024-03-06T07:32:34.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ff3fff640 1 -- 172.21.15.59:0/1973201503 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec02f400 con 0x7f4ffc13dc10 2024-03-06T07:32:34.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f5001d3e640 1 -- 172.21.15.59:0/1973201503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc1f8650 con 0x7f4ffc13dc10 2024-03-06T07:32:34.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f4ff3fff640 1 -- 172.21.15.59:0/1973201503 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec02f7f0 con 0x7f4ffc13dc10 2024-03-06T07:32:34.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.103+0000 7f5001d3e640 1 -- 172.21.15.59:0/1973201503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc140110 con 0x7f4ffc13dc10 2024-03-06T07:32:34.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.104+0000 7f5001d3e640 1 -- 172.21.15.59:0/1973201503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc0682b0 con 0x7f4ffc13dc10 2024-03-06T07:32:34.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.105+0000 7f4ff3fff640 1 -- 172.21.15.59:0/1973201503 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fec02f990 con 0x7f4ffc13dc10 2024-03-06T07:32:34.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.105+0000 7f4ff3fff640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fdc06b500 0x7f4fdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.105+0000 7f4ff3fff640 1 -- 172.21.15.59:0/1973201503 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4fec0ac660 con 0x7f4ffc13dc10 2024-03-06T07:32:34.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.106+0000 7f4ffb7fe640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fdc06b500 0x7f4fdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.106+0000 7f4ffb7fe640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fdc06b500 0x7f4fdc06d9c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4003eb0 tx=0x7f4fe4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.110+0000 7f4ff3fff640 1 -- 172.21.15.59:0/1973201503 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ffc0682b0 con 0x7f4ffc13dc10 2024-03-06T07:32:34.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.329+0000 7f5001d3e640 1 -- 172.21.15.59:0/1973201503 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ffc04f330 con 0x7f4ffc13dc10 2024-03-06T07:32:34.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.330+0000 7f4ff3fff640 1 -- 172.21.15.59:0/1973201503 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f4fec03f070 con 0x7f4ffc13dc10 2024-03-06T07:32:34.332 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:34.332 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:34.332 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:34.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 -- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fdc06b500 msgr2=0x7f4fdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fdc06b500 0x7f4fdc06d9c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4003eb0 tx=0x7f4fe4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:34.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 -- 172.21.15.59:0/1973201503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc13dc10 msgr2=0x7f4ffc154790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc13dc10 0x7f4ffc154790 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f4fec02e060 tx=0x7f4fec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 -- 172.21.15.59:0/1973201503 shutdown_connections 2024-03-06T07:32:34.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4fdc06b500 0x7f4fdc06d9c0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ffc15bf30 0x7f4ffc154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4ffc13dc10 0x7f4ffc154790 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 --2- 172.21.15.59:0/1973201503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ffc0b3060 0x7f4ffc153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 -- 172.21.15.59:0/1973201503 >> 172.21.15.59:0/1973201503 conn(0x7f4ffc0cf8a0 msgr2=0x7f4ffc15ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 -- 172.21.15.59:0/1973201503 shutdown_connections 2024-03-06T07:32:34.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:34.332+0000 7f5001d3e640 1 -- 172.21.15.59:0/1973201503 wait complete. 2024-03-06T07:32:34.546 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:34.546 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:35.547 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:35.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.652+0000 7fbe1315e640 1 Processor -- start 2024-03-06T07:32:35.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.652+0000 7fbe1315e640 1 -- start start 2024-03-06T07:32:35.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.653+0000 7fbe1315e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbe0c15bdd0 0x7fbe0c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.653+0000 7fbe1315e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbe0c15fa10 0x7fbe0c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.653+0000 7fbe1315e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c0cebe0 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.653+0000 7fbe1315e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe0c0ceec0 con 0x7fbe0c15bdd0 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.653+0000 7fbe1315e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe0c0cf1a0 con 0x7fbe0c15fa10 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.653+0000 7fbe10efa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbe0c15bdd0 0x7fbe0c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.653+0000 7fbe10efa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbe0c15bdd0 0x7fbe0c0c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34806/0 (socket says 172.21.15.59:34806) 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.653+0000 7fbe10efa640 1 -- 172.21.15.59:0/986556441 learned_addr learned my addr 172.21.15.59:0/986556441 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.654+0000 7fbe10efa640 1 -- 172.21.15.59:0/986556441 >> v2:172.21.15.164:3300/0 conn(0x7fbe0c15fa10 msgr2=0x7fbe0c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.654+0000 7fbe0bfff640 1 --2- 172.21.15.59:0/986556441 >> v2:172.21.15.164:3300/0 conn(0x7fbe0c15fa10 0x7fbe0c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.654+0000 7fbe10efa640 1 --2- 172.21.15.59:0/986556441 >> v2:172.21.15.164:3300/0 conn(0x7fbe0c15fa10 0x7fbe0c0ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.654+0000 7fbe10efa640 1 -- 172.21.15.59:0/986556441 >> v1:172.21.15.164:6789/0 conn(0x7fbe0c0d16e0 legacy=0x7fbe0c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.654+0000 7fbe10efa640 1 -- 172.21.15.59:0/986556441 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c0cc8e0 con 0x7fbe0c15bdd0 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.654+0000 7fbe0bfff640 1 --2- 172.21.15.59:0/986556441 >> v2:172.21.15.164:3300/0 conn(0x7fbe0c15fa10 0x7fbe0c0ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:35.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe10efa640 1 --2- 172.21.15.59:0/986556441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c15bdd0 0x7fbe0c0c5610 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c06e3c0 tx=0x7fbe0002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e586b3b24982c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe0b7fe640 1 -- 172.21.15.59:0/986556441 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe000317c0 con 0x7fbe0c15bdd0 2024-03-06T07:32:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe0b7fe640 1 -- 172.21.15.59:0/986556441 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe00037070 con 0x7fbe0c15bdd0 2024-03-06T07:32:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe0b7fe640 1 -- 172.21.15.59:0/986556441 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00031b80 con 0x7fbe0c15bdd0 2024-03-06T07:32:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe1315e640 1 -- 172.21.15.59:0/986556441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c15bdd0 msgr2=0x7fbe0c0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe1315e640 1 --2- 172.21.15.59:0/986556441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c15bdd0 0x7fbe0c0c5610 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c06e3c0 tx=0x7fbe0002b860 comp rx=0 tx=0).stop 2024-03-06T07:32:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe1315e640 1 -- 172.21.15.59:0/986556441 shutdown_connections 2024-03-06T07:32:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe1315e640 1 --2- 172.21.15.59:0/986556441 >> v2:172.21.15.164:3300/0 conn(0x7fbe0c15fa10 0x7fbe0c0ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe1315e640 1 --2- 172.21.15.59:0/986556441 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c15bdd0 0x7fbe0c0c5610 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe1315e640 1 -- 172.21.15.59:0/986556441 >> 172.21.15.59:0/986556441 conn(0x7fbe0c09a9f0 msgr2=0x7fbe0c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe1315e640 1 -- 172.21.15.59:0/986556441 shutdown_connections 2024-03-06T07:32:35.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.655+0000 7fbe1315e640 1 -- 172.21.15.59:0/986556441 wait complete. 2024-03-06T07:32:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.656+0000 7fbe1315e640 1 Processor -- start 2024-03-06T07:32:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.656+0000 7fbe1315e640 1 -- start start 2024-03-06T07:32:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.657+0000 7fbe1315e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c0d16e0 0x7fbe0c1545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.657+0000 7fbe1315e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe0c15bdd0 0x7fbe0c15e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.657+0000 7fbe1315e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe0c15fa10 0x7fbe0c15ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.657+0000 7fbe1315e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c15c1b0 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.657+0000 7fbe1315e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c0cc5d0 con 0x7fbe0c15fa10 2024-03-06T07:32:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.657+0000 7fbe1315e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c0ce7b0 con 0x7fbe0c15bdd0 2024-03-06T07:32:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.657+0000 7fbe10efa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c0d16e0 0x7fbe0c1545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.657+0000 7fbe10efa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c0d16e0 0x7fbe0c1545d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34816/0 (socket says 172.21.15.59:34816) 2024-03-06T07:32:35.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.657+0000 7fbe10efa640 1 -- 172.21.15.59:0/281987058 learned_addr learned my addr 172.21.15.59:0/281987058 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe116fb640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe0c15fa10 0x7fbe0c15ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe0bfff640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe0c15bdd0 0x7fbe0c15e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe10efa640 1 -- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe0c15bdd0 msgr2=0x7fbe0c15e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe10efa640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe0c15bdd0 0x7fbe0c15e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe10efa640 1 -- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe0c15fa10 msgr2=0x7fbe0c15ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe10efa640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe0c15fa10 0x7fbe0c15ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe10efa640 1 -- 172.21.15.59:0/281987058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c15cde0 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe10efa640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c0d16e0 0x7fbe0c1545d0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fbe0002b8a0 tx=0x7fbe000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe097fa640 1 -- 172.21.15.59:0/281987058 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00032860 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe097fa640 1 -- 172.21.15.59:0/281987058 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe00032a00 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe097fa640 1 -- 172.21.15.59:0/281987058 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00032df0 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe1315e640 1 -- 172.21.15.59:0/281987058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0c15d0c0 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.658+0000 7fbe1315e640 1 -- 172.21.15.59:0/281987058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0c154b10 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.659+0000 7fbe1315e640 1 -- 172.21.15.59:0/281987058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0c04f330 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.660+0000 7fbe097fa640 1 -- 172.21.15.59:0/281987058 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe0004c060 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.660+0000 7fbe097fa640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbdf006b500 0x7fbdf006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.660+0000 7fbe097fa640 1 -- 172.21.15.59:0/281987058 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbe000abe00 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.660+0000 7fbe0bfff640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbdf006b500 0x7fbdf006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.664 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.661+0000 7fbe0bfff640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbdf006b500 0x7fbdf006d9c0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c070a90 tx=0x7fbdf8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.665+0000 7fbe097fa640 1 -- 172.21.15.59:0/281987058 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe00040070 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.882+0000 7fbe1315e640 1 -- 172.21.15.59:0/281987058 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe0c0b7a40 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.884 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.883+0000 7fbe097fa640 1 -- 172.21.15.59:0/281987058 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fbe0007acf0 con 0x7fbe0c0d16e0 2024-03-06T07:32:35.885 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:35.885 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:35.885 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:35.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.884+0000 7fbe1315e640 1 -- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbdf006b500 msgr2=0x7fbdf006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.884+0000 7fbe1315e640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbdf006b500 0x7fbdf006d9c0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c070a90 tx=0x7fbdf8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.884+0000 7fbe1315e640 1 -- 172.21.15.59:0/281987058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c0d16e0 msgr2=0x7fbe0c1545d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.884+0000 7fbe1315e640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c0d16e0 0x7fbe0c1545d0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fbe0002b8a0 tx=0x7fbe000046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.884+0000 7fbe1315e640 1 -- 172.21.15.59:0/281987058 shutdown_connections 2024-03-06T07:32:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.885+0000 7fbe1315e640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbdf006b500 0x7fbdf006d9c0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.885+0000 7fbe1315e640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe0c15fa10 0x7fbe0c15ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.885+0000 7fbe1315e640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe0c15bdd0 0x7fbe0c15e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.885+0000 7fbe1315e640 1 --2- 172.21.15.59:0/281987058 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe0c0d16e0 0x7fbe0c1545d0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.885+0000 7fbe1315e640 1 -- 172.21.15.59:0/281987058 >> 172.21.15.59:0/281987058 conn(0x7fbe0c09a9f0 msgr2=0x7fbe0c06fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.885+0000 7fbe1315e640 1 -- 172.21.15.59:0/281987058 shutdown_connections 2024-03-06T07:32:35.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:35.885+0000 7fbe1315e640 1 -- 172.21.15.59:0/281987058 wait complete. 2024-03-06T07:32:36.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:36.098 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:37.099 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:37.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.199+0000 7fa807565640 1 Processor -- start 2024-03-06T07:32:37.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.199+0000 7fa807565640 1 -- start start 2024-03-06T07:32:37.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.199+0000 7fa807565640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa80015dfe0 0x7fa8000cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.199+0000 7fa807565640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa8000cf830 0x7fa8000d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.199+0000 7fa807565640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8000d27f0 0x7fa8000d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.199+0000 7fa807565640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8000b31e0 con 0x7fa80015dfe0 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.199+0000 7fa807565640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8000b3d60 con 0x7fa8000cf830 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.199+0000 7fa807565640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8000b3430 con 0x7fa8000d27f0 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.200+0000 7fa805301640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa80015dfe0 0x7fa8000cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.200+0000 7fa805301640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa80015dfe0 0x7fa8000cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34820/0 (socket says 172.21.15.59:34820) 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.200+0000 7fa805301640 1 -- 172.21.15.59:0/1507206323 learned_addr learned my addr 172.21.15.59:0/1507206323 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.200+0000 7fa805b02640 1 --2- 172.21.15.59:0/1507206323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8000d27f0 0x7fa8000d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.200+0000 7fa804b00640 1 --2- 172.21.15.59:0/1507206323 >> v2:172.21.15.164:3300/0 conn(0x7fa8000cf830 0x7fa8000d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.200+0000 7fa805301640 1 -- 172.21.15.59:0/1507206323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8000d27f0 msgr2=0x7fa8000d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.200+0000 7fa805301640 1 --2- 172.21.15.59:0/1507206323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8000d27f0 0x7fa8000d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.200+0000 7fa805301640 1 -- 172.21.15.59:0/1507206323 >> v2:172.21.15.164:3300/0 conn(0x7fa8000cf830 msgr2=0x7fa8000d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:37.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.200+0000 7fa805301640 1 --2- 172.21.15.59:0/1507206323 >> v2:172.21.15.164:3300/0 conn(0x7fa8000cf830 0x7fa8000d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.200+0000 7fa805301640 1 -- 172.21.15.59:0/1507206323 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8000b3710 con 0x7fa80015dfe0 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.200+0000 7fa804b00640 1 --2- 172.21.15.59:0/1507206323 >> v2:172.21.15.164:3300/0 conn(0x7fa8000cf830 0x7fa8000d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.201+0000 7fa805301640 1 --2- 172.21.15.59:0/1507206323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa80015dfe0 0x7fa8000cf2f0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fa8000d16b0 tx=0x7fa7fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e3f875460d5f61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.201+0000 7fa7f7fff640 1 -- 172.21.15.59:0/1507206323 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc035020 con 0x7fa80015dfe0 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.201+0000 7fa7f7fff640 1 -- 172.21.15.59:0/1507206323 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7fc031870 con 0x7fa80015dfe0 2024-03-06T07:32:37.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.201+0000 7fa7f7fff640 1 -- 172.21.15.59:0/1507206323 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc031c40 con 0x7fa80015dfe0 2024-03-06T07:32:37.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.202+0000 7fa807565640 1 -- 172.21.15.59:0/1507206323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa80015dfe0 msgr2=0x7fa8000cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.202+0000 7fa807565640 1 --2- 172.21.15.59:0/1507206323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa80015dfe0 0x7fa8000cf2f0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fa8000d16b0 tx=0x7fa7fc02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:37.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.202+0000 7fa807565640 1 -- 172.21.15.59:0/1507206323 shutdown_connections 2024-03-06T07:32:37.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.202+0000 7fa807565640 1 --2- 172.21.15.59:0/1507206323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8000d27f0 0x7fa8000d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.202+0000 7fa807565640 1 --2- 172.21.15.59:0/1507206323 >> v2:172.21.15.164:3300/0 conn(0x7fa8000cf830 0x7fa8000d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.202+0000 7fa807565640 1 --2- 172.21.15.59:0/1507206323 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa80015dfe0 0x7fa8000cf2f0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.202+0000 7fa807565640 1 -- 172.21.15.59:0/1507206323 >> 172.21.15.59:0/1507206323 conn(0x7fa8000d3cd0 msgr2=0x7fa8000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.202+0000 7fa807565640 1 -- 172.21.15.59:0/1507206323 shutdown_connections 2024-03-06T07:32:37.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.202+0000 7fa807565640 1 -- 172.21.15.59:0/1507206323 wait complete. 2024-03-06T07:32:37.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.203+0000 7fa807565640 1 Processor -- start 2024-03-06T07:32:37.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.204+0000 7fa807565640 1 -- start start 2024-03-06T07:32:37.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa807565640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8000cf830 0x7fa80015bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa807565640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8000d27f0 0x7fa800160810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa807565640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80015dfe0 0x7fa80015fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa807565640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8000d1df0 con 0x7fa8000cf830 2024-03-06T07:32:37.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa807565640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8000d1c70 con 0x7fa80015dfe0 2024-03-06T07:32:37.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa807565640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80015e3c0 con 0x7fa8000d27f0 2024-03-06T07:32:37.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa805301640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8000cf830 0x7fa80015bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa805301640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8000cf830 0x7fa80015bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34822/0 (socket says 172.21.15.59:34822) 2024-03-06T07:32:37.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa805301640 1 -- 172.21.15.59:0/2132768621 learned_addr learned my addr 172.21.15.59:0/2132768621 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:37.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa804b00640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8000d27f0 0x7fa800160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa805301640 1 -- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8000d27f0 msgr2=0x7fa800160810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa805301640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8000d27f0 0x7fa800160810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa805301640 1 -- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80015dfe0 msgr2=0x7fa80015fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa805301640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80015dfe0 0x7fa80015fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa805301640 1 -- 172.21.15.59:0/2132768621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80015c130 con 0x7fa8000cf830 2024-03-06T07:32:37.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa804b00640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8000d27f0 0x7fa800160810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:37.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa805301640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8000cf830 0x7fa80015bbf0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc000c00 tx=0x7fa7fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa7f5ffb640 1 -- 172.21.15.59:0/2132768621 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc040070 con 0x7fa8000cf830 2024-03-06T07:32:37.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa7f5ffb640 1 -- 172.21.15.59:0/2132768621 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7fc03e070 con 0x7fa8000cf830 2024-03-06T07:32:37.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.205+0000 7fa7f5ffb640 1 -- 172.21.15.59:0/2132768621 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc035040 con 0x7fa8000cf830 2024-03-06T07:32:37.211 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.206+0000 7fa807565640 1 -- 172.21.15.59:0/2132768621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa800160d50 con 0x7fa8000cf830 2024-03-06T07:32:37.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.206+0000 7fa807565640 1 -- 172.21.15.59:0/2132768621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8000ca9a0 con 0x7fa8000cf830 2024-03-06T07:32:37.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.207+0000 7fa807565640 1 -- 172.21.15.59:0/2132768621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80004f330 con 0x7fa8000cf830 2024-03-06T07:32:37.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.207+0000 7fa7f5ffb640 1 -- 172.21.15.59:0/2132768621 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7fc038020 con 0x7fa8000cf830 2024-03-06T07:32:37.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.208+0000 7fa7f5ffb640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7e406b4d0 0x7fa7e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.208+0000 7fa7f5ffb640 1 -- 172.21.15.59:0/2132768621 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa7fc0ac020 con 0x7fa8000cf830 2024-03-06T07:32:37.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.208+0000 7fa804b00640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7e406b4d0 0x7fa7e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.208+0000 7fa804b00640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7e406b4d0 0x7fa7e406d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa800161660 tx=0x7fa7f0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.212+0000 7fa7f5ffb640 1 -- 172.21.15.59:0/2132768621 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa80015c130 con 0x7fa8000cf830 2024-03-06T07:32:37.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.429+0000 7fa807565640 1 -- 172.21.15.59:0/2132768621 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8000ce7a0 con 0x7fa8000cf830 2024-03-06T07:32:37.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.430+0000 7fa7f5ffb640 1 -- 172.21.15.59:0/2132768621 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fa7fc0443c0 con 0x7fa8000cf830 2024-03-06T07:32:37.432 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:37.432 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:37.433 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:37.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 -- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7e406b4d0 msgr2=0x7fa7e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7e406b4d0 0x7fa7e406d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa800161660 tx=0x7fa7f0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:37.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 -- 172.21.15.59:0/2132768621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8000cf830 msgr2=0x7fa80015bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8000cf830 0x7fa80015bbf0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc000c00 tx=0x7fa7fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 -- 172.21.15.59:0/2132768621 shutdown_connections 2024-03-06T07:32:37.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa7e406b4d0 0x7fa7e406d990 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80015dfe0 0x7fa80015fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8000d27f0 0x7fa800160810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 --2- 172.21.15.59:0/2132768621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8000cf830 0x7fa80015bbf0 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 -- 172.21.15.59:0/2132768621 >> 172.21.15.59:0/2132768621 conn(0x7fa8000d3cd0 msgr2=0x7fa80006cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 -- 172.21.15.59:0/2132768621 shutdown_connections 2024-03-06T07:32:37.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:37.432+0000 7fa807565640 1 -- 172.21.15.59:0/2132768621 wait complete. 2024-03-06T07:32:37.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:37.646 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:38.647 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:38.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.747+0000 7f521db56640 1 Processor -- start 2024-03-06T07:32:38.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.747+0000 7f521db56640 1 -- start start 2024-03-06T07:32:38.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.747+0000 7f521db56640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521806da40 0x7f52180cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.747+0000 7f521db56640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52180cc980 con 0x7f52181424f0 2024-03-06T07:32:38.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.747+0000 7f521db56640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52180ccc60 con 0x7f52180cd7a0 2024-03-06T07:32:38.752 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.747+0000 7f521db56640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52180cd220 con 0x7f521806da40 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52177fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f52181424f0 0x7f521806d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34510/0 (socket says 172.21.15.59:34510) 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52177fe640 1 -- 172.21.15.59:0/1858910148 learned_addr learned my addr 172.21.15.59:0/1858910148 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f5216ffd640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521806da40 0x7f52180cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372020684 0 0) 0x7f52180cc980 con 0x7f52181424f0 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5200003520 con 0x7f52181424f0 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3781275518 0 0) 0x7f5200003520 con 0x7f52181424f0 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521806da40 msgr2=0x7f52180cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521806da40 0x7f52180cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 >> v1:172.21.15.164:6789/0 conn(0x7f52180cd7a0 legacy=0x7f5218071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52180ccf40 con 0x7f52181424f0 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f5208001670 con 0x7f52181424f0 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 >> v1:172.21.15.59:6789/0 conn(0x7f52181424f0 legacy=0x7f521806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52000039e0 0x7f5200003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5200004340 0x7f5200004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5200005020 0x7f520000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52180cd220 con 0x7f5200005020 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520000bf30 con 0x7f5200004340 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520000c130 con 0x7f52000039e0 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f5216ffd640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521806da40 0x7f52180cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.748+0000 7f5217fff640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5200005020 0x7f520000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.749+0000 7f52177fe640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52000039e0 0x7f5200003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.749+0000 7f5216ffd640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5200004340 0x7f5200004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.749+0000 7f5217fff640 1 -- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52000039e0 msgr2=0x7f5200003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.749+0000 7f5217fff640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52000039e0 0x7f5200003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.749+0000 7f5217fff640 1 -- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5200004340 msgr2=0x7f5200004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.754 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.749+0000 7f5217fff640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5200004340 0x7f5200004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.749+0000 7f5217fff640 1 -- 172.21.15.59:0/1858910148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5200003520 con 0x7f5200005020 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.749+0000 7f5217fff640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5200005020 0x7f520000b9c0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f5210001f60 tx=0x7f5200005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a37afb973071171 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.750+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5210001b30 con 0x7f5200005020 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.750+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521002d710 con 0x7f5200005020 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.750+0000 7f52167fc640 1 -- 172.21.15.59:0/1858910148 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521002db00 con 0x7f5200005020 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.750+0000 7f521db56640 1 -- 172.21.15.59:0/1858910148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5200005020 msgr2=0x7f520000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.750+0000 7f521db56640 1 --2- 172.21.15.59:0/1858910148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5200005020 0x7f520000b9c0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f5210001f60 tx=0x7f5200005830 comp rx=0 tx=0).stop 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.750+0000 7f5217fff640 1 -- 172.21.15.59:0/1858910148 reap_dead start 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.750+0000 7f521db56640 1 -- 172.21.15.59:0/1858910148 shutdown_connections 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.750+0000 7f521db56640 1 -- 172.21.15.59:0/1858910148 >> 172.21.15.59:0/1858910148 conn(0x7f52180cf8a0 msgr2=0x7f52180cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.750+0000 7f521db56640 1 -- 172.21.15.59:0/1858910148 shutdown_connections 2024-03-06T07:32:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.750+0000 7f521db56640 1 -- 172.21.15.59:0/1858910148 wait complete. 2024-03-06T07:32:38.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.752+0000 7f521db56640 1 Processor -- start 2024-03-06T07:32:38.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.752+0000 7f521db56640 1 -- start start 2024-03-06T07:32:38.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.752+0000 7f521db56640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5200005020 0x7f5218156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.752+0000 7f521db56640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f521815c4b0 0x7f5218158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.752+0000 7f521db56640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521813eee0 0x7f521813f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.752+0000 7f521db56640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52180cc470 con 0x7f521815c4b0 2024-03-06T07:32:38.758 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f521db56640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52180cc2f0 con 0x7f5200005020 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f521db56640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52180cc5f0 con 0x7f521813eee0 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f5216ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f521815c4b0 0x7f5218158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f5216ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f521815c4b0 0x7f5218158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34828/0 (socket says 172.21.15.59:34828) 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f5216ffd640 1 -- 172.21.15.59:0/2098257041 learned_addr learned my addr 172.21.15.59:0/2098257041 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f5217fff640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521813eee0 0x7f521813f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f52177fe640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5200005020 0x7f5218156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f5216ffd640 1 -- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521813eee0 msgr2=0x7f521813f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f5216ffd640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521813eee0 0x7f521813f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f5216ffd640 1 -- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5200005020 msgr2=0x7f5218156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f5216ffd640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5200005020 0x7f5218156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f5216ffd640 1 -- 172.21.15.59:0/2098257041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521815ca20 con 0x7f521815c4b0 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f52177fe640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5200005020 0x7f5218156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:38.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f5216ffd640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f521815c4b0 0x7f5218158770 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f520c011830 tx=0x7f520c011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f51fffff640 1 -- 172.21.15.59:0/2098257041 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c00fbd0 con 0x7f521815c4b0 2024-03-06T07:32:38.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f51fffff640 1 -- 172.21.15.59:0/2098257041 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c00fd70 con 0x7f521815c4b0 2024-03-06T07:32:38.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f521db56640 1 -- 172.21.15.59:0/2098257041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521815cd00 con 0x7f521815c4b0 2024-03-06T07:32:38.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f51fffff640 1 -- 172.21.15.59:0/2098257041 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c015680 con 0x7f521815c4b0 2024-03-06T07:32:38.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.753+0000 7f521db56640 1 -- 172.21.15.59:0/2098257041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521815d2c0 con 0x7f521815c4b0 2024-03-06T07:32:38.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.754+0000 7f521db56640 1 -- 172.21.15.59:0/2098257041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521804f330 con 0x7f521815c4b0 2024-03-06T07:32:38.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.755+0000 7f51fffff640 1 -- 172.21.15.59:0/2098257041 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f520c015820 con 0x7f521815c4b0 2024-03-06T07:32:38.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.755+0000 7f51fffff640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f520406b4d0 0x7f520406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.755+0000 7f51fffff640 1 -- 172.21.15.59:0/2098257041 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f520c08ac40 con 0x7f521815c4b0 2024-03-06T07:32:38.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.755+0000 7f52177fe640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f520406b4d0 0x7f520406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.756+0000 7f52177fe640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f520406b4d0 0x7f520406d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f5208007cd0 tx=0x7f5208009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.759+0000 7f51fffff640 1 -- 172.21.15.59:0/2098257041 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f520c019030 con 0x7f521815c4b0 2024-03-06T07:32:38.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.981+0000 7f521db56640 1 -- 172.21.15.59:0/2098257041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52180d0620 con 0x7f521815c4b0 2024-03-06T07:32:38.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.982+0000 7f51fffff640 1 -- 172.21.15.59:0/2098257041 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f520c059450 con 0x7f521815c4b0 2024-03-06T07:32:38.984 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:38.984 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:38.984 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:38.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 -- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f520406b4d0 msgr2=0x7f520406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f520406b4d0 0x7f520406d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f5208007cd0 tx=0x7f5208009040 comp rx=0 tx=0).stop 2024-03-06T07:32:38.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 -- 172.21.15.59:0/2098257041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f521815c4b0 msgr2=0x7f5218158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f521815c4b0 0x7f5218158770 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f520c011830 tx=0x7f520c011cf0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 -- 172.21.15.59:0/2098257041 shutdown_connections 2024-03-06T07:32:38.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f520406b4d0 0x7f520406d990 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521813eee0 0x7f521813f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f521815c4b0 0x7f5218158770 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 --2- 172.21.15.59:0/2098257041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5200005020 0x7f5218156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 -- 172.21.15.59:0/2098257041 >> 172.21.15.59:0/2098257041 conn(0x7f52180cf8a0 msgr2=0x7f521806c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 -- 172.21.15.59:0/2098257041 shutdown_connections 2024-03-06T07:32:38.987 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:38.984+0000 7f521db56640 1 -- 172.21.15.59:0/2098257041 wait complete. 2024-03-06T07:32:39.198 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:39.198 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:40.199 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:40.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.302+0000 7f76344a4640 1 Processor -- start 2024-03-06T07:32:40.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.303+0000 7f76344a4640 1 -- start start 2024-03-06T07:32:40.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.303+0000 7f76344a4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f762c0d4760 0x7f762c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.303+0000 7f76344a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f762c0d0a80 0x7f762c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.303+0000 7f76344a4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762c0d25d0 con 0x7f762c0b4170 2024-03-06T07:32:40.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.303+0000 7f76344a4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f762c0d4bf0 con 0x7f762c0d4760 2024-03-06T07:32:40.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.303+0000 7f76344a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762c0ce9e0 con 0x7f762c0d0a80 2024-03-06T07:32:40.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.304+0000 7f7631a3f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f762c0d4760 0x7f762c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.304+0000 7f7631a3f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f762c0d4760 0x7f762c0b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34844/0 (socket says 172.21.15.59:34844) 2024-03-06T07:32:40.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.304+0000 7f7631a3f640 1 -- 172.21.15.59:0/2841777186 learned_addr learned my addr 172.21.15.59:0/2841777186 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:40.306 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.304+0000 7f7632a41640 1 --2- 172.21.15.59:0/2841777186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f762c0d0a80 0x7f762c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.304+0000 7f7631a3f640 1 -- 172.21.15.59:0/2841777186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f762c0d0a80 msgr2=0x7f762c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.304+0000 7f7631a3f640 1 --2- 172.21.15.59:0/2841777186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f762c0d0a80 0x7f762c0cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.304+0000 7f7631a3f640 1 -- 172.21.15.59:0/2841777186 >> v1:172.21.15.164:6789/0 conn(0x7f762c0b4170 legacy=0x7f762c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.307 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.304+0000 7f7631a3f640 1 -- 172.21.15.59:0/2841777186 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c0d2870 con 0x7f762c0d4760 2024-03-06T07:32:40.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f7631a3f640 1 --2- 172.21.15.59:0/2841777186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f762c0d4760 0x7f762c0b6a90 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f762c067920 tx=0x7f761c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c97eb410161eba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f763123e640 1 -- 172.21.15.59:0/2841777186 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c035020 con 0x7f762c0d4760 2024-03-06T07:32:40.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f763123e640 1 -- 172.21.15.59:0/2841777186 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c031830 con 0x7f762c0d4760 2024-03-06T07:32:40.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f763123e640 1 -- 172.21.15.59:0/2841777186 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c031c00 con 0x7f762c0d4760 2024-03-06T07:32:40.308 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f76344a4640 1 -- 172.21.15.59:0/2841777186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f762c0d4760 msgr2=0x7f762c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f76344a4640 1 --2- 172.21.15.59:0/2841777186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f762c0d4760 0x7f762c0b6a90 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f762c067920 tx=0x7f761c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f76344a4640 1 -- 172.21.15.59:0/2841777186 shutdown_connections 2024-03-06T07:32:40.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f76344a4640 1 --2- 172.21.15.59:0/2841777186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f762c0d0a80 0x7f762c0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f76344a4640 1 --2- 172.21.15.59:0/2841777186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f762c0d4760 0x7f762c0b6a90 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f76344a4640 1 -- 172.21.15.59:0/2841777186 >> 172.21.15.59:0/2841777186 conn(0x7f762c0d3c40 msgr2=0x7f762c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f76344a4640 1 -- 172.21.15.59:0/2841777186 shutdown_connections 2024-03-06T07:32:40.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.305+0000 7f76344a4640 1 -- 172.21.15.59:0/2841777186 wait complete. 2024-03-06T07:32:40.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.306+0000 7f76344a4640 1 Processor -- start 2024-03-06T07:32:40.309 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.307+0000 7f76344a4640 1 -- start start 2024-03-06T07:32:40.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.307+0000 7f76344a4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f762c0b4170 0x7f762c1595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.307+0000 7f76344a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f762c0d0a80 0x7f762c0c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.307+0000 7f76344a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f762c0cccd0 0x7f762c0cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.307+0000 7f76344a4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c0b4dd0 con 0x7f762c0b4170 2024-03-06T07:32:40.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.307+0000 7f76344a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c0b4550 con 0x7f762c0cccd0 2024-03-06T07:32:40.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.307+0000 7f76344a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762c0b4f50 con 0x7f762c0d0a80 2024-03-06T07:32:40.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.307+0000 7f7632240640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f762c0b4170 0x7f762c1595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.311 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.307+0000 7f7632240640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f762c0b4170 0x7f762c1595f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34860/0 (socket says 172.21.15.59:34860) 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.307+0000 7f7632240640 1 -- 172.21.15.59:0/2721735490 learned_addr learned my addr 172.21.15.59:0/2721735490 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f7631a3f640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f762c0d0a80 0x7f762c0c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f7632a41640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f762c0cccd0 0x7f762c0cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f7632240640 1 -- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f762c0d0a80 msgr2=0x7f762c0c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f7632240640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f762c0d0a80 0x7f762c0c9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f7632240640 1 -- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f762c0cccd0 msgr2=0x7f762c0cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f7632240640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f762c0cccd0 0x7f762c0cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f7632240640 1 -- 172.21.15.59:0/2721735490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c0cadb0 con 0x7f762c0b4170 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f7632240640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f762c0b4170 0x7f762c1595f0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f762800b3f0 tx=0x7f762800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f7622ffd640 1 -- 172.21.15.59:0/2721735490 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7628015020 con 0x7f762c0b4170 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f7622ffd640 1 -- 172.21.15.59:0/2721735490 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7628011830 con 0x7f762c0b4170 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f76344a4640 1 -- 172.21.15.59:0/2721735490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c0b2540 con 0x7f762c0b4170 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f7622ffd640 1 -- 172.21.15.59:0/2721735490 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7628011c20 con 0x7f762c0b4170 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.308+0000 7f76344a4640 1 -- 172.21.15.59:0/2721735490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c0cb020 con 0x7f762c0b4170 2024-03-06T07:32:40.312 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.309+0000 7f76344a4640 1 -- 172.21.15.59:0/2721735490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762c04f330 con 0x7f762c0b4170 2024-03-06T07:32:40.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.309+0000 7f7622ffd640 1 -- 172.21.15.59:0/2721735490 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7628018020 con 0x7f762c0b4170 2024-03-06T07:32:40.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.310+0000 7f7622ffd640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f761006b4d0 0x7f761006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.310+0000 7f7622ffd640 1 -- 172.21.15.59:0/2721735490 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f762808a1b0 con 0x7f762c0b4170 2024-03-06T07:32:40.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.310+0000 7f7631a3f640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f761006b4d0 0x7f761006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.311+0000 7f7631a3f640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f761006b4d0 0x7f761006d990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f761c036000 tx=0x7f761c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.314+0000 7f7622ffd640 1 -- 172.21.15.59:0/2721735490 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f762801f020 con 0x7f762c0b4170 2024-03-06T07:32:40.548 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.547+0000 7f76344a4640 1 -- 172.21.15.59:0/2721735490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f762c0733b0 con 0x7f762c0b4170 2024-03-06T07:32:40.549 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.547+0000 7f7622ffd640 1 -- 172.21.15.59:0/2721735490 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f76280172c0 con 0x7f762c0b4170 2024-03-06T07:32:40.549 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:40.549 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:40.550 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:40.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 -- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f761006b4d0 msgr2=0x7f761006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f761006b4d0 0x7f761006d990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f761c036000 tx=0x7f761c008030 comp rx=0 tx=0).stop 2024-03-06T07:32:40.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 -- 172.21.15.59:0/2721735490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f762c0b4170 msgr2=0x7f762c1595f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f762c0b4170 0x7f762c1595f0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f762800b3f0 tx=0x7f762800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 -- 172.21.15.59:0/2721735490 shutdown_connections 2024-03-06T07:32:40.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f761006b4d0 0x7f761006d990 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f762c0cccd0 0x7f762c0cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f762c0d0a80 0x7f762c0c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 --2- 172.21.15.59:0/2721735490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f762c0b4170 0x7f762c1595f0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 -- 172.21.15.59:0/2721735490 >> 172.21.15.59:0/2721735490 conn(0x7f762c0d3c40 msgr2=0x7f762c0d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 -- 172.21.15.59:0/2721735490 shutdown_connections 2024-03-06T07:32:40.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:40.549+0000 7f76344a4640 1 -- 172.21.15.59:0/2721735490 wait complete. 2024-03-06T07:32:40.763 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:40.763 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:41.765 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:41.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.865+0000 7f759e88f640 1 Processor -- start 2024-03-06T07:32:41.867 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.865+0000 7f759e88f640 1 -- start start 2024-03-06T07:32:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f759e88f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f759806da40 0x7f75980cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f759e88f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75980cd7a0 0x7f759815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f759e88f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75980cc980 con 0x7f75981424f0 2024-03-06T07:32:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f759e88f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75980cd220 con 0x7f759806da40 2024-03-06T07:32:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f759e88f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75980ccc60 con 0x7f75980cd7a0 2024-03-06T07:32:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f75977fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f759806da40 0x7f75980cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f75977fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f759806da40 0x7f75980cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34868/0 (socket says 172.21.15.59:34868) 2024-03-06T07:32:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f75977fe640 1 -- 172.21.15.59:0/1405304675 learned_addr learned my addr 172.21.15.59:0/1405304675 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f759ce2c640 1 --2- 172.21.15.59:0/1405304675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75980cd7a0 0x7f759815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f75977fe640 1 -- 172.21.15.59:0/1405304675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75980cd7a0 msgr2=0x7f759815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f75977fe640 1 --2- 172.21.15.59:0/1405304675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75980cd7a0 0x7f759815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f75977fe640 1 -- 172.21.15.59:0/1405304675 >> v1:172.21.15.164:6789/0 conn(0x7f75981424f0 legacy=0x7f759806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.866+0000 7f75977fe640 1 -- 172.21.15.59:0/1405304675 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75980ccf40 con 0x7f759806da40 2024-03-06T07:32:41.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.867+0000 7f75977fe640 1 --2- 172.21.15.59:0/1405304675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f759806da40 0x7f75980cc2f0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f758c009870 tx=0x7f758c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4b5149565edff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.867+0000 7f7596ffd640 1 -- 172.21.15.59:0/1405304675 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c034070 con 0x7f759806da40 2024-03-06T07:32:41.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.867+0000 7f7596ffd640 1 -- 172.21.15.59:0/1405304675 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758c030840 con 0x7f759806da40 2024-03-06T07:32:41.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.867+0000 7f7596ffd640 1 -- 172.21.15.59:0/1405304675 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c030c10 con 0x7f759806da40 2024-03-06T07:32:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.868+0000 7f759e88f640 1 -- 172.21.15.59:0/1405304675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f759806da40 msgr2=0x7f75980cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.868+0000 7f759e88f640 1 --2- 172.21.15.59:0/1405304675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f759806da40 0x7f75980cc2f0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f758c009870 tx=0x7f758c02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.868+0000 7f759e88f640 1 -- 172.21.15.59:0/1405304675 shutdown_connections 2024-03-06T07:32:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.868+0000 7f759e88f640 1 --2- 172.21.15.59:0/1405304675 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75980cd7a0 0x7f759815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.868+0000 7f759e88f640 1 --2- 172.21.15.59:0/1405304675 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f759806da40 0x7f75980cc2f0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.868+0000 7f759e88f640 1 -- 172.21.15.59:0/1405304675 >> 172.21.15.59:0/1405304675 conn(0x7f75980cf8a0 msgr2=0x7f75980cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.868+0000 7f759e88f640 1 -- 172.21.15.59:0/1405304675 shutdown_connections 2024-03-06T07:32:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.868+0000 7f759e88f640 1 -- 172.21.15.59:0/1405304675 wait complete. 2024-03-06T07:32:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.869+0000 7f759e88f640 1 Processor -- start 2024-03-06T07:32:41.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.869+0000 7f759e88f640 1 -- start start 2024-03-06T07:32:41.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759e88f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f759806da40 0x7f7598157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759e88f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75980cd7a0 0x7f75980ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759e88f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75981424f0 0x7f759813e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759e88f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7598071ce0 con 0x7f75981424f0 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759e88f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7598071b60 con 0x7f75980cd7a0 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759e88f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7598071e60 con 0x7f759806da40 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759ce2c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75981424f0 0x7f759813e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759ce2c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75981424f0 0x7f759813e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34876/0 (socket says 172.21.15.59:34876) 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759ce2c640 1 -- 172.21.15.59:0/385491006 learned_addr learned my addr 172.21.15.59:0/385491006 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f7597fff640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f759806da40 0x7f7598157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f75977fe640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75980cd7a0 0x7f75980ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759ce2c640 1 -- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f759806da40 msgr2=0x7f7598157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759ce2c640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f759806da40 0x7f7598157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759ce2c640 1 -- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75980cd7a0 msgr2=0x7f75980ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759ce2c640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75980cd7a0 0x7f75980ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759ce2c640 1 -- 172.21.15.59:0/385491006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7598158950 con 0x7f75981424f0 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759ce2c640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75981424f0 0x7f759813e110 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f759000efc0 tx=0x7f759000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f7594ff9640 1 -- 172.21.15.59:0/385491006 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7590018020 con 0x7f75981424f0 2024-03-06T07:32:41.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f7594ff9640 1 -- 172.21.15.59:0/385491006 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7590014480 con 0x7f75981424f0 2024-03-06T07:32:41.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759e88f640 1 -- 172.21.15.59:0/385491006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7598158c30 con 0x7f75981424f0 2024-03-06T07:32:41.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f7594ff9640 1 -- 172.21.15.59:0/385491006 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7590014870 con 0x7f75981424f0 2024-03-06T07:32:41.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.870+0000 7f759e88f640 1 -- 172.21.15.59:0/385491006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75980a96a0 con 0x7f75981424f0 2024-03-06T07:32:41.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.871+0000 7f759e88f640 1 -- 172.21.15.59:0/385491006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759804f330 con 0x7f75981424f0 2024-03-06T07:32:41.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.872+0000 7f7594ff9640 1 -- 172.21.15.59:0/385491006 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7590005d10 con 0x7f75981424f0 2024-03-06T07:32:41.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.872+0000 7f7594ff9640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f758406b4d0 0x7f758406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.872+0000 7f7594ff9640 1 -- 172.21.15.59:0/385491006 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f759008ac20 con 0x7f75981424f0 2024-03-06T07:32:41.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.873+0000 7f7597fff640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f758406b4d0 0x7f758406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.873+0000 7f7597fff640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f758406b4d0 0x7f758406d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f75880016e0 tx=0x7f7588009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:41.877+0000 7f7594ff9640 1 -- 172.21.15.59:0/385491006 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7590022070 con 0x7f75981424f0 2024-03-06T07:32:42.099 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.097+0000 7f759e88f640 1 -- 172.21.15.59:0/385491006 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75980d0620 con 0x7f75981424f0 2024-03-06T07:32:42.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.098+0000 7f7594ff9640 1 -- 172.21.15.59:0/385491006 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7f7590010040 con 0x7f75981424f0 2024-03-06T07:32:42.100 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:42.100 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:42.101 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:42.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 -- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f758406b4d0 msgr2=0x7f758406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.102 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f758406b4d0 0x7f758406d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f75880016e0 tx=0x7f7588009040 comp rx=0 tx=0).stop 2024-03-06T07:32:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 -- 172.21.15.59:0/385491006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75981424f0 msgr2=0x7f759813e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75981424f0 0x7f759813e110 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f759000efc0 tx=0x7f759000c490 comp rx=0 tx=0).stop 2024-03-06T07:32:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 -- 172.21.15.59:0/385491006 shutdown_connections 2024-03-06T07:32:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f758406b4d0 0x7f758406d990 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75981424f0 0x7f759813e110 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75980cd7a0 0x7f75980ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 --2- 172.21.15.59:0/385491006 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f759806da40 0x7f7598157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 -- 172.21.15.59:0/385491006 >> 172.21.15.59:0/385491006 conn(0x7f75980cf8a0 msgr2=0x7f759815c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 -- 172.21.15.59:0/385491006 shutdown_connections 2024-03-06T07:32:42.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:42.100+0000 7f759e88f640 1 -- 172.21.15.59:0/385491006 wait complete. 2024-03-06T07:32:42.317 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:42.317 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:43.318 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:43.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.421+0000 7fb2e7edf640 1 Processor -- start 2024-03-06T07:32:43.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.421+0000 7fb2e7edf640 1 -- start start 2024-03-06T07:32:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.422+0000 7fb2e7edf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e00d22b0 0x7fb2e00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.422+0000 7fb2e7edf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb2e00b41f0 0x7fb2e00b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.422+0000 7fb2e7edf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e00b3430 con 0x7fb2e015dfe0 2024-03-06T07:32:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.422+0000 7fb2e7edf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e00d2c50 con 0x7fb2e00b41f0 2024-03-06T07:32:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.422+0000 7fb2e7edf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e00b31e0 con 0x7fb2e00d22b0 2024-03-06T07:32:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.422+0000 7fb2e5c7b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb2e015dfe0 0x7fb2e00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34520/0 (socket says 172.21.15.59:34520) 2024-03-06T07:32:43.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.422+0000 7fb2e5c7b640 1 -- 172.21.15.59:0/4056861699 learned_addr learned my addr 172.21.15.59:0/4056861699 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:43.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.422+0000 7fb2e647c640 1 --2- 172.21.15.59:0/4056861699 >> v2:172.21.15.164:3300/0 conn(0x7fb2e00b41f0 0x7fb2e00b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.422+0000 7fb2e547a640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e00d22b0 0x7fb2e00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.422+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798758136 0 0) 0x7fb2e00b3430 con 0x7fb2e015dfe0 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.422+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c8003520 con 0x7fb2e015dfe0 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 106382910 0 0) 0x7fb2c8003520 con 0x7fb2e015dfe0 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e00d22b0 msgr2=0x7fb2e00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e00d22b0 0x7fb2e00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 >> v2:172.21.15.164:3300/0 conn(0x7fb2e00b41f0 msgr2=0x7fb2e00b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 --2- 172.21.15.59:0/4056861699 >> v2:172.21.15.164:3300/0 conn(0x7fb2e00b41f0 0x7fb2e00b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e00b3710 con 0x7fb2e015dfe0 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fb2d0001670 con 0x7fb2e015dfe0 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 >> v1:172.21.15.59:6789/0 conn(0x7fb2e015dfe0 legacy=0x7fb2e00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e00d22b0 0x7fb2c80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2c8003f30 0x7fb2c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2c8004d40 0x7fb2c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e00b31e0 con 0x7fb2c8004d40 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c800b9c0 con 0x7fb2c8003f30 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c800bb40 con 0x7fb2e00d22b0 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e647c640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2c8004d40 0x7fb2c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e547a640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2c8003f30 0x7fb2c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e5c7b640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e00d22b0 0x7fb2c80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e647c640 1 -- 172.21.15.59:0/4056861699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e00d22b0 msgr2=0x7fb2c80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e647c640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e00d22b0 0x7fb2c80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e647c640 1 -- 172.21.15.59:0/4056861699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2c8003f30 msgr2=0x7fb2c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e647c640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2c8003f30 0x7fb2c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.423+0000 7fb2e647c640 1 -- 172.21.15.59:0/4056861699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c8003520 con 0x7fb2c8004d40 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.424+0000 7fb2e647c640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2c8004d40 0x7fb2c80051c0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009ac0 tx=0x7fb2c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e0191fcda358af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.424+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8039070 con 0x7fb2c8004d40 2024-03-06T07:32:43.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.424+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d802c6d0 con 0x7fb2c8004d40 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.424+0000 7fb2e4c79640 1 -- 172.21.15.59:0/4056861699 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8009690 con 0x7fb2c8004d40 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.425+0000 7fb2e7edf640 1 -- 172.21.15.59:0/4056861699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2c8004d40 msgr2=0x7fb2c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.425+0000 7fb2e7edf640 1 --2- 172.21.15.59:0/4056861699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2c8004d40 0x7fb2c80051c0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009ac0 tx=0x7fb2c8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.425+0000 7fb2e647c640 1 -- 172.21.15.59:0/4056861699 reap_dead start 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.425+0000 7fb2e7edf640 1 -- 172.21.15.59:0/4056861699 shutdown_connections 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.425+0000 7fb2e7edf640 1 -- 172.21.15.59:0/4056861699 >> 172.21.15.59:0/4056861699 conn(0x7fb2e00d3cd0 msgr2=0x7fb2e00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.425+0000 7fb2e7edf640 1 -- 172.21.15.59:0/4056861699 shutdown_connections 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.425+0000 7fb2e7edf640 1 -- 172.21.15.59:0/4056861699 wait complete. 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.426+0000 7fb2e7edf640 1 Processor -- start 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.426+0000 7fb2e7edf640 1 -- start start 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e7edf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e0152c30 0x7fb2e00b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e7edf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2e00b8170 0x7fb2e00ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e7edf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2e0153010 0x7fb2e00cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e7edf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e00cfa00 con 0x7fb2e00b8170 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e7edf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e00b3d60 con 0x7fb2e0153010 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e7edf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e00d1c70 con 0x7fb2e0152c30 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e547a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2e00b8170 0x7fb2e00ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e547a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2e00b8170 0x7fb2e00ccc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34902/0 (socket says 172.21.15.59:34902) 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e547a640 1 -- 172.21.15.59:0/228322706 learned_addr learned my addr 172.21.15.59:0/228322706 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e5c7b640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e0152c30 0x7fb2e00b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e647c640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2e0153010 0x7fb2e00cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e547a640 1 -- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e0152c30 msgr2=0x7fb2e00b7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e547a640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e0152c30 0x7fb2e00b7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e547a640 1 -- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2e0153010 msgr2=0x7fb2e00cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e547a640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2e0153010 0x7fb2e00cd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e547a640 1 -- 172.21.15.59:0/228322706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e00b68d0 con 0x7fb2e00b8170 2024-03-06T07:32:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e647c640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2e0153010 0x7fb2e00cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e547a640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2e00b8170 0x7fb2e00ccc30 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4011e30 tx=0x7fb2d400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2de7fc640 1 -- 172.21.15.59:0/228322706 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4015630 con 0x7fb2e00b8170 2024-03-06T07:32:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2de7fc640 1 -- 172.21.15.59:0/228322706 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d40157d0 con 0x7fb2e00b8170 2024-03-06T07:32:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2e7edf640 1 -- 172.21.15.59:0/228322706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e00b6b90 con 0x7fb2e00b8170 2024-03-06T07:32:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.427+0000 7fb2de7fc640 1 -- 172.21.15.59:0/228322706 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4015ba0 con 0x7fb2e00b8170 2024-03-06T07:32:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.428+0000 7fb2e7edf640 1 -- 172.21.15.59:0/228322706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e00b7150 con 0x7fb2e00b8170 2024-03-06T07:32:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.429+0000 7fb2e7edf640 1 -- 172.21.15.59:0/228322706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e004f330 con 0x7fb2e00b8170 2024-03-06T07:32:43.435 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.429+0000 7fb2de7fc640 1 -- 172.21.15.59:0/228322706 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2d401d400 con 0x7fb2e00b8170 2024-03-06T07:32:43.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.429+0000 7fb2de7fc640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.429+0000 7fb2de7fc640 1 -- 172.21.15.59:0/228322706 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb2d408b540 con 0x7fb2e00b8170 2024-03-06T07:32:43.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.430+0000 7fb2e5c7b640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.430+0000 7fb2e5c7b640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0004840 tx=0x7fb2d0009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.436 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.434+0000 7fb2de7fc640 1 -- 172.21.15.59:0/228322706 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2d4019030 con 0x7fb2e00b8170 2024-03-06T07:32:43.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.652+0000 7fb2e7edf640 1 -- 172.21.15.59:0/228322706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2e006d530 con 0x7fb2e00b8170 2024-03-06T07:32:43.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.652+0000 7fb2de7fc640 1 -- 172.21.15.59:0/228322706 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3465 (secure 0 0 0) 0x7fb2d4010040 con 0x7fb2e00b8170 2024-03-06T07:32:43.655 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:43.655 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":107,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:31:40.104271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:43.655 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 107 2024-03-06T07:32:43.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 -- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2cc06b4d0 msgr2=0x7fb2cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0004840 tx=0x7fb2d0009040 comp rx=0 tx=0).stop 2024-03-06T07:32:43.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 -- 172.21.15.59:0/228322706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2e00b8170 msgr2=0x7fb2e00ccc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2e00b8170 0x7fb2e00ccc30 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4011e30 tx=0x7fb2d400f580 comp rx=0 tx=0).stop 2024-03-06T07:32:43.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 -- 172.21.15.59:0/228322706 shutdown_connections 2024-03-06T07:32:43.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb2cc06b4d0 0x7fb2cc06d990 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2e0153010 0x7fb2e00cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb2e00b8170 0x7fb2e00ccc30 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 --2- 172.21.15.59:0/228322706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2e0152c30 0x7fb2e00b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 -- 172.21.15.59:0/228322706 >> 172.21.15.59:0/228322706 conn(0x7fb2e00d3cd0 msgr2=0x7fb2e00b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 -- 172.21.15.59:0/228322706 shutdown_connections 2024-03-06T07:32:43.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:43.654+0000 7fb2e7edf640 1 -- 172.21.15.59:0/228322706 wait complete. 2024-03-06T07:32:43.871 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:31:40.104271+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:43.872 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:44.872 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:45.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.011+0000 7ff2a951e640 1 Processor -- start 2024-03-06T07:32:45.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.011+0000 7ff2a951e640 1 -- start start 2024-03-06T07:32:45.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.011+0000 7ff2a951e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2a4162020 0x7ff2a40d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.011+0000 7ff2a951e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff2a4162a00 0x7ff2a40b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.011+0000 7ff2a951e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a40d2500 con 0x7ff2a40b0ea0 2024-03-06T07:32:45.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.011+0000 7ff2a951e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a40ce9d0 con 0x7ff2a4162a00 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.011+0000 7ff2a951e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40d2da0 con 0x7ff2a4162020 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.012+0000 7ff2a37fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff2a40b0ea0 0x7ff2a40cf400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:52460/0 (socket says 172.21.15.59:52460) 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.012+0000 7ff2a37fe640 1 -- 172.21.15.59:0/732468795 learned_addr learned my addr 172.21.15.59:0/732468795 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.012+0000 7ff2a2ffd640 1 --2- 172.21.15.59:0/732468795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2a4162020 0x7ff2a40d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.012+0000 7ff29bfff640 1 --2- 172.21.15.59:0/732468795 >> v2:172.21.15.164:3300/0 conn(0x7ff2a4162a00 0x7ff2a40b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.012+0000 7ff2a27fc640 1 -- 172.21.15.59:0/732468795 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810259655 0 0) 0x7ff2a40d2500 con 0x7ff2a40b0ea0 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.012+0000 7ff2a27fc640 1 -- 172.21.15.59:0/732468795 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff290003520 con 0x7ff2a40b0ea0 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.012+0000 7ff2a2ffd640 1 -- 172.21.15.59:0/732468795 >> v2:172.21.15.164:3300/0 conn(0x7ff2a4162a00 msgr2=0x7ff2a40b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.012+0000 7ff2a2ffd640 1 --2- 172.21.15.59:0/732468795 >> v2:172.21.15.164:3300/0 conn(0x7ff2a4162a00 0x7ff2a40b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.012+0000 7ff2a27fc640 1 -- 172.21.15.59:0/732468795 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 516544883 0 0) 0x7ff290003520 con 0x7ff2a40b0ea0 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.012+0000 7ff2a2ffd640 1 -- 172.21.15.59:0/732468795 >> v1:172.21.15.59:6789/0 conn(0x7ff2a40b0ea0 legacy=0x7ff2a40cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.012+0000 7ff2a2ffd640 1 -- 172.21.15.59:0/732468795 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a40d27e0 con 0x7ff2a4162020 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.013+0000 7ff2a2ffd640 1 --2- 172.21.15.59:0/732468795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2a4162020 0x7ff2a40d4830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff2940097a0 tx=0x7ff29402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b78ba7b38537808 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.014+0000 7ff2a27fc640 1 -- 172.21.15.59:0/732468795 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294038070 con 0x7ff2a4162020 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.014+0000 7ff2a27fc640 1 -- 172.21.15.59:0/732468795 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff294031500 con 0x7ff2a4162020 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.014+0000 7ff2a27fc640 1 -- 172.21.15.59:0/732468795 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c0019f0 con 0x7ff2a4162020 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.014+0000 7ff2a951e640 1 -- 172.21.15.59:0/732468795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2a4162020 msgr2=0x7ff2a40d4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.014+0000 7ff2a951e640 1 --2- 172.21.15.59:0/732468795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2a4162020 0x7ff2a40d4830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff2940097a0 tx=0x7ff29402b820 comp rx=0 tx=0).stop 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.014+0000 7ff2a951e640 1 -- 172.21.15.59:0/732468795 shutdown_connections 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.014+0000 7ff2a951e640 1 --2- 172.21.15.59:0/732468795 >> v2:172.21.15.164:3300/0 conn(0x7ff2a4162a00 0x7ff2a40b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.014+0000 7ff2a951e640 1 --2- 172.21.15.59:0/732468795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2a4162020 0x7ff2a40d4830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.014+0000 7ff2a951e640 1 -- 172.21.15.59:0/732468795 >> 172.21.15.59:0/732468795 conn(0x7ff2a40d3d10 msgr2=0x7ff2a40d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.014+0000 7ff2a951e640 1 -- 172.21.15.59:0/732468795 shutdown_connections 2024-03-06T07:32:45.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.014+0000 7ff2a951e640 1 -- 172.21.15.59:0/732468795 wait complete. 2024-03-06T07:32:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.015+0000 7ff2a951e640 1 Processor -- start 2024-03-06T07:32:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.015+0000 7ff2a951e640 1 -- start start 2024-03-06T07:32:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.016+0000 7ff2a951e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2a4162020 0x7ff2a41f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.016+0000 7ff2a951e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2a4162a00 0x7ff2a40b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.016+0000 7ff2a951e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2a40b4370 0x7ff2a40c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.016+0000 7ff2a951e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40cfb10 con 0x7ff2a4162020 2024-03-06T07:32:45.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.016+0000 7ff2a951e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40cec00 con 0x7ff2a40b4370 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.016+0000 7ff2a951e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40cfc90 con 0x7ff2a4162a00 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.016+0000 7ff2a2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2a4162020 0x7ff2a41f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.016+0000 7ff2a2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2a4162020 0x7ff2a41f3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50152/0 (socket says 172.21.15.59:50152) 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.016+0000 7ff2a2ffd640 1 -- 172.21.15.59:0/4174770888 learned_addr learned my addr 172.21.15.59:0/4174770888 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.016+0000 7ff29bfff640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2a4162a00 0x7ff2a40b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.016+0000 7ff2a37fe640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2a40b4370 0x7ff2a40c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff2a2ffd640 1 -- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2a4162a00 msgr2=0x7ff2a40b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff2a2ffd640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2a4162a00 0x7ff2a40b51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff2a2ffd640 1 -- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2a40b4370 msgr2=0x7ff2a40c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff2a2ffd640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2a40b4370 0x7ff2a40c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff2a2ffd640 1 -- 172.21.15.59:0/4174770888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a41f3940 con 0x7ff2a4162020 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff2a37fe640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2a40b4370 0x7ff2a40c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:45.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff2a2ffd640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2a4162020 0x7ff2a41f3400 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7ff294000c00 tx=0x7ff2940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff29b7fe640 1 -- 172.21.15.59:0/4174770888 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294043070 con 0x7ff2a4162020 2024-03-06T07:32:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff29b7fe640 1 -- 172.21.15.59:0/4174770888 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29403f870 con 0x7ff2a4162020 2024-03-06T07:32:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff2a951e640 1 -- 172.21.15.59:0/4174770888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a40b69d0 con 0x7ff2a4162020 2024-03-06T07:32:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff29b7fe640 1 -- 172.21.15.59:0/4174770888 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294038040 con 0x7ff2a4162020 2024-03-06T07:32:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.017+0000 7ff2a951e640 1 -- 172.21.15.59:0/4174770888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a40b6f90 con 0x7ff2a4162020 2024-03-06T07:32:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.018+0000 7ff2a951e640 1 -- 172.21.15.59:0/4174770888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a404f330 con 0x7ff2a4162020 2024-03-06T07:32:45.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.018+0000 7ff29b7fe640 1 -- 172.21.15.59:0/4174770888 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff294046070 con 0x7ff2a4162020 2024-03-06T07:32:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.019+0000 7ff29b7fe640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff28406b4d0 0x7ff28406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.019+0000 7ff29b7fe640 1 -- 172.21.15.59:0/4174770888 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff2940abd30 con 0x7ff2a4162020 2024-03-06T07:32:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.019+0000 7ff29bfff640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff28406b4d0 0x7ff28406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.020+0000 7ff29bfff640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff28406b4d0 0x7ff28406d990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff2a40b0b50 tx=0x7ff28c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.029 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.027+0000 7ff29b7fe640 1 -- 172.21.15.59:0/4174770888 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff294030070 con 0x7ff2a4162020 2024-03-06T07:32:45.268 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.266+0000 7ff2a951e640 1 -- 172.21.15.59:0/4174770888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2a4073380 con 0x7ff2a4162020 2024-03-06T07:32:45.269 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.267+0000 7ff29b7fe640 1 -- 172.21.15.59:0/4174770888 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7ff29407ad40 con 0x7ff2a4162020 2024-03-06T07:32:45.269 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:45.269 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:45.269 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:32:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 -- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff28406b4d0 msgr2=0x7ff28406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff28406b4d0 0x7ff28406d990 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff2a40b0b50 tx=0x7ff28c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 -- 172.21.15.59:0/4174770888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2a4162020 msgr2=0x7ff2a41f3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2a4162020 0x7ff2a41f3400 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7ff294000c00 tx=0x7ff2940046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 -- 172.21.15.59:0/4174770888 shutdown_connections 2024-03-06T07:32:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff28406b4d0 0x7ff28406d990 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.272 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2a40b4370 0x7ff2a40c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2a4162a00 0x7ff2a40b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 --2- 172.21.15.59:0/4174770888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2a4162020 0x7ff2a41f3400 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 -- 172.21.15.59:0/4174770888 >> 172.21.15.59:0/4174770888 conn(0x7ff2a40d3d10 msgr2=0x7ff2a406cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 -- 172.21.15.59:0/4174770888 shutdown_connections 2024-03-06T07:32:45.273 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:45.269+0000 7ff2a951e640 1 -- 172.21.15.59:0/4174770888 wait complete. 2024-03-06T07:32:45.483 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:45.483 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:46.484 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:46.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.584+0000 7fe3b08f4640 1 Processor -- start 2024-03-06T07:32:46.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.584+0000 7fe3b08f4640 1 -- start start 2024-03-06T07:32:46.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3b08f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3ac06b220 0x7fe3ac15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3b08f4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe3ac0ce920 0x7fe3ac071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3b08f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ac15ea30 con 0x7fe3ac071ec0 2024-03-06T07:32:46.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3b08f4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ac161b70 con 0x7fe3ac0ce920 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3b08f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac15ed10 con 0x7fe3ac06b220 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3aad9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe3ac071ec0 0x7fe3ac162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:52470/0 (socket says 172.21.15.59:52470) 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3aad9d640 1 -- 172.21.15.59:0/3651085613 learned_addr learned my addr 172.21.15.59:0/3651085613 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3aa59c640 1 --2- 172.21.15.59:0/3651085613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3ac06b220 0x7fe3ac15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3a9d9b640 1 --2- 172.21.15.59:0/3651085613 >> v2:172.21.15.164:3300/0 conn(0x7fe3ac0ce920 0x7fe3ac071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3a959a640 1 -- 172.21.15.59:0/3651085613 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167202367 0 0) 0x7fe3ac15ea30 con 0x7fe3ac071ec0 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3a959a640 1 -- 172.21.15.59:0/3651085613 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe398003520 con 0x7fe3ac071ec0 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3aa59c640 1 -- 172.21.15.59:0/3651085613 >> v2:172.21.15.164:3300/0 conn(0x7fe3ac0ce920 msgr2=0x7fe3ac071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3aa59c640 1 --2- 172.21.15.59:0/3651085613 >> v2:172.21.15.164:3300/0 conn(0x7fe3ac0ce920 0x7fe3ac071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3aa59c640 1 -- 172.21.15.59:0/3651085613 >> v1:172.21.15.59:6789/0 conn(0x7fe3ac071ec0 legacy=0x7fe3ac162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3aa59c640 1 -- 172.21.15.59:0/3651085613 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac15eff0 con 0x7fe3ac06b220 2024-03-06T07:32:46.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.585+0000 7fe3a9d9b640 1 --2- 172.21.15.59:0/3651085613 >> v2:172.21.15.164:3300/0 conn(0x7fe3ac0ce920 0x7fe3ac071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:46.589 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.586+0000 7fe3aa59c640 1 --2- 172.21.15.59:0/3651085613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3ac06b220 0x7fe3ac15e450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe3940097a0 tx=0x7fe39402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1efbe08c58b72f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.589+0000 7fe3a959a640 1 -- 172.21.15.59:0/3651085613 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394035020 con 0x7fe3ac06b220 2024-03-06T07:32:46.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.589+0000 7fe3a959a640 1 -- 172.21.15.59:0/3651085613 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe394031830 con 0x7fe3ac06b220 2024-03-06T07:32:46.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.589+0000 7fe3a959a640 1 -- 172.21.15.59:0/3651085613 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394031c00 con 0x7fe3ac06b220 2024-03-06T07:32:46.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.589+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3651085613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3ac06b220 msgr2=0x7fe3ac15e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.589+0000 7fe3b08f4640 1 --2- 172.21.15.59:0/3651085613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3ac06b220 0x7fe3ac15e450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe3940097a0 tx=0x7fe39402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.589+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3651085613 shutdown_connections 2024-03-06T07:32:46.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.589+0000 7fe3b08f4640 1 --2- 172.21.15.59:0/3651085613 >> v2:172.21.15.164:3300/0 conn(0x7fe3ac0ce920 0x7fe3ac071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.589+0000 7fe3b08f4640 1 --2- 172.21.15.59:0/3651085613 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3ac06b220 0x7fe3ac15e450 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.589+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3651085613 >> 172.21.15.59:0/3651085613 conn(0x7fe3ac06c0c0 msgr2=0x7fe3ac06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.589+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3651085613 shutdown_connections 2024-03-06T07:32:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.589+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3651085613 wait complete. 2024-03-06T07:32:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.591+0000 7fe3b08f4640 1 Processor -- start 2024-03-06T07:32:46.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.591+0000 7fe3b08f4640 1 -- start start 2024-03-06T07:32:46.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.591+0000 7fe3b08f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe3ac06b220 0x7fe3ac1f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.591+0000 7fe3b08f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3ac071ec0 0x7fe3ac158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.591+0000 7fe3b08f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3ac0ce920 0x7fe3ac1605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.591+0000 7fe3b08f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac0ce380 con 0x7fe3ac071ec0 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.591+0000 7fe3b08f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac0ce200 con 0x7fe3ac06b220 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.591+0000 7fe3b08f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac162710 con 0x7fe3ac0ce920 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.591+0000 7fe3a9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3ac071ec0 0x7fe3ac158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.591+0000 7fe3a9d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3ac071ec0 0x7fe3ac158840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50162/0 (socket says 172.21.15.59:50162) 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.591+0000 7fe3a9d9b640 1 -- 172.21.15.59:0/3614812287 learned_addr learned my addr 172.21.15.59:0/3614812287 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3aa59c640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe3ac06b220 0x7fe3ac1f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3aad9d640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3ac0ce920 0x7fe3ac1605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3a9d9b640 1 -- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3ac0ce920 msgr2=0x7fe3ac1605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3a9d9b640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3ac0ce920 0x7fe3ac1605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3a9d9b640 1 -- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe3ac06b220 msgr2=0x7fe3ac1f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3a9d9b640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe3ac06b220 0x7fe3ac1f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3a9d9b640 1 -- 172.21.15.59:0/3614812287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac159170 con 0x7fe3ac071ec0 2024-03-06T07:32:46.595 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3a9d9b640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3ac071ec0 0x7fe3ac158840 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00ebe0 tx=0x7fe39c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3a2ffd640 1 -- 172.21.15.59:0/3614812287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c017070 con 0x7fe3ac071ec0 2024-03-06T07:32:46.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3a2ffd640 1 -- 172.21.15.59:0/3614812287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39c00cc30 con 0x7fe3ac071ec0 2024-03-06T07:32:46.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3614812287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ac159450 con 0x7fe3ac071ec0 2024-03-06T07:32:46.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3a2ffd640 1 -- 172.21.15.59:0/3614812287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c013470 con 0x7fe3ac071ec0 2024-03-06T07:32:46.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.592+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3614812287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ac158d80 con 0x7fe3ac071ec0 2024-03-06T07:32:46.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.593+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3614812287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ac06c9d0 con 0x7fe3ac071ec0 2024-03-06T07:32:46.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.594+0000 7fe3a2ffd640 1 -- 172.21.15.59:0/3614812287 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe39c01a070 con 0x7fe3ac071ec0 2024-03-06T07:32:46.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.594+0000 7fe3a2ffd640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe38c06b4d0 0x7fe38c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.594+0000 7fe3a2ffd640 1 -- 172.21.15.59:0/3614812287 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe39c08a3e0 con 0x7fe3ac071ec0 2024-03-06T07:32:46.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.594+0000 7fe3aa59c640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe38c06b4d0 0x7fe38c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.595+0000 7fe3aa59c640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe38c06b4d0 0x7fe38c06d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe394036000 tx=0x7fe39403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.599+0000 7fe3a2ffd640 1 -- 172.21.15.59:0/3614812287 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3ac06c9d0 con 0x7fe3ac071ec0 2024-03-06T07:32:46.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.819+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3614812287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3ac04f330 con 0x7fe3ac071ec0 2024-03-06T07:32:46.821 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.819+0000 7fe3a2ffd640 1 -- 172.21.15.59:0/3614812287 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fe39c058c80 con 0x7fe3ac071ec0 2024-03-06T07:32:46.821 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:46.822 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:46.822 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:32:46.823 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe38c06b4d0 msgr2=0x7fe38c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe38c06b4d0 0x7fe38c06d990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe394036000 tx=0x7fe39403c040 comp rx=0 tx=0).stop 2024-03-06T07:32:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3614812287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3ac071ec0 msgr2=0x7fe3ac158840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3ac071ec0 0x7fe3ac158840 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00ebe0 tx=0x7fe39c00c520 comp rx=0 tx=0).stop 2024-03-06T07:32:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3614812287 shutdown_connections 2024-03-06T07:32:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe38c06b4d0 0x7fe38c06d990 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3ac0ce920 0x7fe3ac1605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3ac071ec0 0x7fe3ac158840 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 --2- 172.21.15.59:0/3614812287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe3ac06b220 0x7fe3ac1f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3614812287 >> 172.21.15.59:0/3614812287 conn(0x7fe3ac06c0c0 msgr2=0x7fe3ac162000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3614812287 shutdown_connections 2024-03-06T07:32:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:46.821+0000 7fe3b08f4640 1 -- 172.21.15.59:0/3614812287 wait complete. 2024-03-06T07:32:47.035 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:47.035 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:48.037 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:48.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.136+0000 7f232410c640 1 Processor -- start 2024-03-06T07:32:48.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.137+0000 7f232410c640 1 -- start start 2024-03-06T07:32:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.137+0000 7f232410c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f231c15dfe0 0x7f231c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.137+0000 7f232410c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c0b3430 con 0x7f231c0cf830 2024-03-06T07:32:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.137+0000 7f232410c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c0b3710 con 0x7f231c0d29c0 2024-03-06T07:32:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.137+0000 7f232410c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f231c0b31e0 con 0x7f231c15dfe0 2024-03-06T07:32:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.137+0000 7f23216a7640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f231c0cf830 0x7f231c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:52476/0 (socket says 172.21.15.59:52476) 2024-03-06T07:32:48.139 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.137+0000 7f23216a7640 1 -- 172.21.15.59:0/1876165861 learned_addr learned my addr 172.21.15.59:0/1876165861 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:48.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862683726 0 0) 0x7f231c0b3430 con 0x7f231c0cf830 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2304003520 con 0x7f231c0cf830 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2321ea8640 1 --2- 172.21.15.59:0/1876165861 >> v2:172.21.15.164:3300/0 conn(0x7f231c15dfe0 0x7f231c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3864868774 0 0) 0x7f2304003520 con 0x7f231c0cf830 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 >> v2:172.21.15.164:3300/0 conn(0x7f231c15dfe0 msgr2=0x7f231c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 --2- 172.21.15.59:0/1876165861 >> v2:172.21.15.164:3300/0 conn(0x7f231c15dfe0 0x7f231c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 >> v1:172.21.15.164:6789/0 conn(0x7f231c0d29c0 legacy=0x7f231c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c0b39f0 con 0x7f231c0cf830 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f2310001670 con 0x7f231c0cf830 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 >> v1:172.21.15.59:6789/0 conn(0x7f231c0cf830 legacy=0x7f231c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 --2- 172.21.15.59:0/1876165861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23040039b0 0x7f2304003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 --2- 172.21.15.59:0/1876165861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2304004800 0x7f2304004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 --2- 172.21.15.59:0/1876165861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23040051c0 0x7f230400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c0b31e0 con 0x7f23040039b0 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230400c090 con 0x7f23040051c0 2024-03-06T07:32:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230400c290 con 0x7f2304004800 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f2321ea8640 1 --2- 172.21.15.59:0/1876165861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23040039b0 0x7f2304003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f23216a7640 1 --2- 172.21.15.59:0/1876165861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2304004800 0x7f2304004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.138+0000 7f23226a9640 1 --2- 172.21.15.59:0/1876165861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23040051c0 0x7f230400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.139+0000 7f2321ea8640 1 -- 172.21.15.59:0/1876165861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2304004800 msgr2=0x7f2304004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.139+0000 7f2321ea8640 1 --2- 172.21.15.59:0/1876165861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2304004800 0x7f2304004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.139+0000 7f2321ea8640 1 -- 172.21.15.59:0/1876165861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23040051c0 msgr2=0x7f230400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.139+0000 7f2321ea8640 1 --2- 172.21.15.59:0/1876165861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23040051c0 0x7f230400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.139+0000 7f23226a9640 1 -- 172.21.15.59:0/1876165861 reap_dead start 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.139+0000 7f2321ea8640 1 -- 172.21.15.59:0/1876165861 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2304003520 con 0x7f23040039b0 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.139+0000 7f2321ea8640 1 --2- 172.21.15.59:0/1876165861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23040039b0 0x7f2304003df0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f2314009fc0 tx=0x7f2304005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29152f493752b38a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.139+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2314039070 con 0x7f23040039b0 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.140+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231402cc20 con 0x7f23040039b0 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.140+0000 7f2320ea6640 1 -- 172.21.15.59:0/1876165861 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2314009c10 con 0x7f23040039b0 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.140+0000 7f232410c640 1 -- 172.21.15.59:0/1876165861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23040039b0 msgr2=0x7f2304003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.140+0000 7f232410c640 1 --2- 172.21.15.59:0/1876165861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23040039b0 0x7f2304003df0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f2314009fc0 tx=0x7f2304005830 comp rx=0 tx=0).stop 2024-03-06T07:32:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.140+0000 7f232410c640 1 -- 172.21.15.59:0/1876165861 shutdown_connections 2024-03-06T07:32:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.140+0000 7f232410c640 1 --2- 172.21.15.59:0/1876165861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23040039b0 0x7f2304003df0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.140+0000 7f232410c640 1 -- 172.21.15.59:0/1876165861 >> 172.21.15.59:0/1876165861 conn(0x7f231c0d3cd0 msgr2=0x7f231c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.140+0000 7f232410c640 1 -- 172.21.15.59:0/1876165861 shutdown_connections 2024-03-06T07:32:48.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.140+0000 7f232410c640 1 -- 172.21.15.59:0/1876165861 wait complete. 2024-03-06T07:32:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.142+0000 7f232410c640 1 Processor -- start 2024-03-06T07:32:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.142+0000 7f232410c640 1 -- start start 2024-03-06T07:32:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f232410c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23040039b0 0x7f231c0b7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f232410c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231c0b68d0 0x7f231c0b9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f232410c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f231c160320 0x7f231c152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f232410c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c15e3c0 con 0x7f231c160320 2024-03-06T07:32:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f232410c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c0b3d60 con 0x7f23040039b0 2024-03-06T07:32:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f232410c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231c15e540 con 0x7f231c0b68d0 2024-03-06T07:32:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f23226a9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f231c160320 0x7f231c152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f23226a9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f231c160320 0x7f231c152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50178/0 (socket says 172.21.15.59:50178) 2024-03-06T07:32:48.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f23226a9640 1 -- 172.21.15.59:0/2869411179 learned_addr learned my addr 172.21.15.59:0/2869411179 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f2321ea8640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23040039b0 0x7f231c0b7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f23216a7640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231c0b68d0 0x7f231c0b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f23226a9640 1 -- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231c0b68d0 msgr2=0x7f231c0b9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f23226a9640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231c0b68d0 0x7f231c0b9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f23226a9640 1 -- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23040039b0 msgr2=0x7f231c0b7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f23226a9640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23040039b0 0x7f231c0b7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f23226a9640 1 -- 172.21.15.59:0/2869411179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c0b6e50 con 0x7f231c160320 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.143+0000 7f23226a9640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f231c160320 0x7f231c152b30 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f231c0cff60 tx=0x7f230c00f8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.144+0000 7f231a7fc640 1 -- 172.21.15.59:0/2869411179 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c01a070 con 0x7f231c160320 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.144+0000 7f231a7fc640 1 -- 172.21.15.59:0/2869411179 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230c0053e0 con 0x7f231c160320 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.144+0000 7f232410c640 1 -- 172.21.15.59:0/2869411179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231c0b7130 con 0x7f231c160320 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.144+0000 7f231a7fc640 1 -- 172.21.15.59:0/2869411179 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c007250 con 0x7f231c160320 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.144+0000 7f232410c640 1 -- 172.21.15.59:0/2869411179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231c0b76f0 con 0x7f231c160320 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.145+0000 7f232410c640 1 -- 172.21.15.59:0/2869411179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f231c04f330 con 0x7f231c160320 2024-03-06T07:32:48.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.145+0000 7f231a7fc640 1 -- 172.21.15.59:0/2869411179 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f230c01d070 con 0x7f231c160320 2024-03-06T07:32:48.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.146+0000 7f231a7fc640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f230806b4d0 0x7f230806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.146+0000 7f231a7fc640 1 -- 172.21.15.59:0/2869411179 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f230c089300 con 0x7f231c160320 2024-03-06T07:32:48.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.146+0000 7f2321ea8640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f230806b4d0 0x7f230806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.147+0000 7f2321ea8640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f230806b4d0 0x7f230806d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f231402cfd0 tx=0x7f231402e000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.152 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.150+0000 7f231a7fc640 1 -- 172.21.15.59:0/2869411179 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f230c057ba0 con 0x7f231c160320 2024-03-06T07:32:48.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.369+0000 7f232410c640 1 -- 172.21.15.59:0/2869411179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f231c0b10a0 con 0x7f231c160320 2024-03-06T07:32:48.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.369+0000 7f231a7fc640 1 -- 172.21.15.59:0/2869411179 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f230c05bc40 con 0x7f231c160320 2024-03-06T07:32:48.372 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:48.372 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:48.372 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.371+0000 7f232410c640 1 -- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f230806b4d0 msgr2=0x7f230806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.371+0000 7f232410c640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f230806b4d0 0x7f230806d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f231402cfd0 tx=0x7f231402e000 comp rx=0 tx=0).stop 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.371+0000 7f232410c640 1 -- 172.21.15.59:0/2869411179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f231c160320 msgr2=0x7f231c152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.371+0000 7f232410c640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f231c160320 0x7f231c152b30 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f231c0cff60 tx=0x7f230c00f8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.371+0000 7f232410c640 1 -- 172.21.15.59:0/2869411179 shutdown_connections 2024-03-06T07:32:48.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.371+0000 7f232410c640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f230806b4d0 0x7f230806d990 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.371+0000 7f232410c640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f231c160320 0x7f231c152b30 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.371+0000 7f232410c640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231c0b68d0 0x7f231c0b9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.371+0000 7f232410c640 1 --2- 172.21.15.59:0/2869411179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23040039b0 0x7f231c0b7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.371+0000 7f232410c640 1 -- 172.21.15.59:0/2869411179 >> 172.21.15.59:0/2869411179 conn(0x7f231c0d3cd0 msgr2=0x7f231c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.372+0000 7f232410c640 1 -- 172.21.15.59:0/2869411179 shutdown_connections 2024-03-06T07:32:48.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:48.372+0000 7f232410c640 1 -- 172.21.15.59:0/2869411179 wait complete. 2024-03-06T07:32:48.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:48.585 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:49.587 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:49.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.687+0000 7f8fffdac640 1 Processor -- start 2024-03-06T07:32:49.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.687+0000 7f8fffdac640 1 -- start start 2024-03-06T07:32:49.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8fffdac640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8ff815dfe0 0x7f8ff80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8fffdac640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff80b3430 con 0x7f8ff80cf830 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8fffdac640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff80b3710 con 0x7f8ff80d29c0 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8fffdac640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff80b31e0 con 0x7f8ff815dfe0 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffd347640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8ff80cf830 0x7f8ff80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:52488/0 (socket says 172.21.15.59:52488) 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffd347640 1 -- 172.21.15.59:0/818531112 learned_addr learned my addr 172.21.15.59:0/818531112 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776403853 0 0) 0x7f8ff80b3430 con 0x7f8ff80cf830 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe4003520 con 0x7f8ff80cf830 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffdb48640 1 --2- 172.21.15.59:0/818531112 >> v2:172.21.15.164:3300/0 conn(0x7f8ff815dfe0 0x7f8ff80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1996600992 0 0) 0x7f8fe4003520 con 0x7f8ff80cf830 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 >> v2:172.21.15.164:3300/0 conn(0x7f8ff815dfe0 msgr2=0x7f8ff80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 --2- 172.21.15.59:0/818531112 >> v2:172.21.15.164:3300/0 conn(0x7f8ff815dfe0 0x7f8ff80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 >> v1:172.21.15.164:6789/0 conn(0x7f8ff80d29c0 legacy=0x7f8ff80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff80b39f0 con 0x7f8ff80cf830 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f8fe8001670 con 0x7f8ff80cf830 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 >> v1:172.21.15.59:6789/0 conn(0x7f8ff80cf830 legacy=0x7f8ff80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe40039b0 0x7f8fe4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe4004800 0x7f8fe4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe40051c0 0x7f8fe400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff80b31e0 con 0x7f8fe40039b0 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe400c090 con 0x7f8fe4004800 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.688+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe400c290 con 0x7f8fe40051c0 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.689+0000 7f8ffdb48640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe40039b0 0x7f8fe4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.689+0000 7f8ffd347640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe4004800 0x7f8fe4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.689+0000 7f8ffe349640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe40051c0 0x7f8fe400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.689+0000 7f8ffdb48640 1 -- 172.21.15.59:0/818531112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe40051c0 msgr2=0x7f8fe400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.689+0000 7f8ffdb48640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8fe40051c0 0x7f8fe400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.689+0000 7f8ffdb48640 1 -- 172.21.15.59:0/818531112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe4004800 msgr2=0x7f8fe4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.689+0000 7f8ffdb48640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe4004800 0x7f8fe4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.689+0000 7f8ffdb48640 1 -- 172.21.15.59:0/818531112 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe4003520 con 0x7f8fe40039b0 2024-03-06T07:32:49.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.689+0000 7f8ffd347640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe4004800 0x7f8fe4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:49.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.689+0000 7f8ffe349640 1 -- 172.21.15.59:0/818531112 reap_dead start 2024-03-06T07:32:49.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.690+0000 7f8ffdb48640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe40039b0 0x7f8fe4003df0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009ac0 tx=0x7f8fe4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b7a32c3598bd85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.690+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4039070 con 0x7f8fe40039b0 2024-03-06T07:32:49.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.690+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff402c6b0 con 0x7f8fe40039b0 2024-03-06T07:32:49.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.690+0000 7f8ffcb46640 1 -- 172.21.15.59:0/818531112 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4009690 con 0x7f8fe40039b0 2024-03-06T07:32:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.690+0000 7f8fffdac640 1 -- 172.21.15.59:0/818531112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe40039b0 msgr2=0x7f8fe4003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.690+0000 7f8fffdac640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe40039b0 0x7f8fe4003df0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009ac0 tx=0x7f8fe4005830 comp rx=0 tx=0).stop 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.690+0000 7f8fffdac640 1 -- 172.21.15.59:0/818531112 shutdown_connections 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.690+0000 7f8fffdac640 1 --2- 172.21.15.59:0/818531112 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8fe40039b0 0x7f8fe4003df0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.690+0000 7f8fffdac640 1 -- 172.21.15.59:0/818531112 >> 172.21.15.59:0/818531112 conn(0x7f8ff80d3cd0 msgr2=0x7f8ff80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.690+0000 7f8fffdac640 1 -- 172.21.15.59:0/818531112 shutdown_connections 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.690+0000 7f8fffdac640 1 -- 172.21.15.59:0/818531112 wait complete. 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.691+0000 7f8fffdac640 1 Processor -- start 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.691+0000 7f8fffdac640 1 -- start start 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.692+0000 7f8fffdac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe40039b0 0x7f8ff80b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.692+0000 7f8fffdac640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ff80b9520 0x7f8ff80ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.692+0000 7f8fffdac640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ff814fc20 0x7f8ff81500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.692+0000 7f8fffdac640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff815e3c0 con 0x7f8ff814fc20 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.692+0000 7f8fffdac640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff80b3d60 con 0x7f8fe40039b0 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.692+0000 7f8fffdac640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff815e540 con 0x7f8ff80b9520 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.692+0000 7f8ffe349640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ff814fc20 0x7f8ff81500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.692+0000 7f8ffe349640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ff814fc20 0x7f8ff81500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50186/0 (socket says 172.21.15.59:50186) 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.692+0000 7f8ffe349640 1 -- 172.21.15.59:0/687584694 learned_addr learned my addr 172.21.15.59:0/687584694 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8ffe349640 1 -- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ff80b9520 msgr2=0x7f8ff80ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8ffd347640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ff80b9520 0x7f8ff80ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8ffdb48640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe40039b0 0x7f8ff80b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8ffe349640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ff80b9520 0x7f8ff80ccdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8ffe349640 1 -- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe40039b0 msgr2=0x7f8ff80b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8ffe349640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe40039b0 0x7f8ff80b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8ffe349640 1 -- 172.21.15.59:0/687584694 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff80b99a0 con 0x7f8ff814fc20 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8ffe349640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ff814fc20 0x7f8ff81500d0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0007fd0 tx=0x7f8ff000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8fee7fc640 1 -- 172.21.15.59:0/687584694 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0003970 con 0x7f8ff814fc20 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8fee7fc640 1 -- 172.21.15.59:0/687584694 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff0018070 con 0x7f8ff814fc20 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8fffdac640 1 -- 172.21.15.59:0/687584694 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff80b79c0 con 0x7f8ff814fc20 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8fee7fc640 1 -- 172.21.15.59:0/687584694 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0003d70 con 0x7f8ff814fc20 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.693+0000 7f8fffdac640 1 -- 172.21.15.59:0/687584694 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff80b7f80 con 0x7f8ff814fc20 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.694+0000 7f8fffdac640 1 -- 172.21.15.59:0/687584694 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff804f330 con 0x7f8ff814fc20 2024-03-06T07:32:49.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.695+0000 7f8fee7fc640 1 -- 172.21.15.59:0/687584694 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ff00050d0 con 0x7f8ff814fc20 2024-03-06T07:32:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.695+0000 7f8fee7fc640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 0x7f8fdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.695+0000 7f8fee7fc640 1 -- 172.21.15.59:0/687584694 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8ff0089e10 con 0x7f8ff814fc20 2024-03-06T07:32:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.695+0000 7f8ffdb48640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 0x7f8fdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.696+0000 7f8ffdb48640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 0x7f8fdc06d9c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009ac0 tx=0x7f8ff4030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.699+0000 7f8fee7fc640 1 -- 172.21.15.59:0/687584694 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ff0058d00 con 0x7f8ff814fc20 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.925+0000 7f8fffdac640 1 -- 172.21.15.59:0/687584694 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ff8150610 con 0x7f8ff814fc20 2024-03-06T07:32:49.927 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.925+0000 7f8fee7fc640 1 -- 172.21.15.59:0/687584694 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f8ff001be40 con 0x7f8ff814fc20 2024-03-06T07:32:49.928 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:49.928 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:49.928 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.928+0000 7f8fffdac640 1 -- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 msgr2=0x7f8fdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.928+0000 7f8fffdac640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 0x7f8fdc06d9c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009ac0 tx=0x7f8ff4030000 comp rx=0 tx=0).stop 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.928+0000 7f8fffdac640 1 -- 172.21.15.59:0/687584694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ff814fc20 msgr2=0x7f8ff81500d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.928+0000 7f8fffdac640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ff814fc20 0x7f8ff81500d0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0007fd0 tx=0x7f8ff000ec10 comp rx=0 tx=0).stop 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.928+0000 7f8fffdac640 1 -- 172.21.15.59:0/687584694 shutdown_connections 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.928+0000 7f8fffdac640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8fdc06b500 0x7f8fdc06d9c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.928+0000 7f8fffdac640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ff814fc20 0x7f8ff81500d0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.928+0000 7f8fffdac640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ff80b9520 0x7f8ff80ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.928+0000 7f8fffdac640 1 --2- 172.21.15.59:0/687584694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8fe40039b0 0x7f8ff80b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.928+0000 7f8fffdac640 1 -- 172.21.15.59:0/687584694 >> 172.21.15.59:0/687584694 conn(0x7f8ff80d3cd0 msgr2=0x7f8ff80b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.928+0000 7f8fffdac640 1 -- 172.21.15.59:0/687584694 shutdown_connections 2024-03-06T07:32:49.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:49.929+0000 7f8fffdac640 1 -- 172.21.15.59:0/687584694 wait complete. 2024-03-06T07:32:50.142 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:50.143 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:51.144 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:51.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.243+0000 7fd60c276640 1 Processor -- start 2024-03-06T07:32:51.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.243+0000 7fd60c276640 1 -- start start 2024-03-06T07:32:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.244+0000 7fd60c276640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd6040d22b0 0x7fd6040d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.244+0000 7fd60c276640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6040b3430 con 0x7fd6040b41f0 2024-03-06T07:32:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.244+0000 7fd60c276640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6040b3710 con 0x7fd60415dfe0 2024-03-06T07:32:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.244+0000 7fd60c276640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6040d2c50 con 0x7fd6040d22b0 2024-03-06T07:32:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.244+0000 7fd60a813640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd6040b41f0 0x7fd6040b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:52492/0 (socket says 172.21.15.59:52492) 2024-03-06T07:32:51.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.244+0000 7fd60a813640 1 -- 172.21.15.59:0/2915047071 learned_addr learned my addr 172.21.15.59:0/2915047071 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.244+0000 7fd609811640 1 --2- 172.21.15.59:0/2915047071 >> v2:172.21.15.164:3300/0 conn(0x7fd6040d22b0 0x7fd6040d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.245+0000 7fd609010640 1 -- 172.21.15.59:0/2915047071 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009531818 0 0) 0x7fd6040b3430 con 0x7fd6040b41f0 2024-03-06T07:32:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.245+0000 7fd609010640 1 -- 172.21.15.59:0/2915047071 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5f0003520 con 0x7fd6040b41f0 2024-03-06T07:32:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.245+0000 7fd609811640 1 -- 172.21.15.59:0/2915047071 >> v1:172.21.15.164:6789/0 conn(0x7fd60415dfe0 legacy=0x7fd6040cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.245+0000 7fd609010640 1 -- 172.21.15.59:0/2915047071 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2055700138 0 0) 0x7fd5f0003520 con 0x7fd6040b41f0 2024-03-06T07:32:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.245+0000 7fd609811640 1 -- 172.21.15.59:0/2915047071 >> v1:172.21.15.59:6789/0 conn(0x7fd6040b41f0 legacy=0x7fd6040b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.245+0000 7fd609811640 1 -- 172.21.15.59:0/2915047071 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6040b39f0 con 0x7fd6040d22b0 2024-03-06T07:32:51.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.246+0000 7fd609811640 1 --2- 172.21.15.59:0/2915047071 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6040d22b0 0x7fd6040d2710 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd6000097d0 tx=0x7fd60002b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48f96ab739b5cf59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.246+0000 7fd609010640 1 -- 172.21.15.59:0/2915047071 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd600037070 con 0x7fd6040d22b0 2024-03-06T07:32:51.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.246+0000 7fd609010640 1 -- 172.21.15.59:0/2915047071 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd600030510 con 0x7fd6040d22b0 2024-03-06T07:32:51.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.246+0000 7fd609010640 1 -- 172.21.15.59:0/2915047071 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0019f0 con 0x7fd6040d22b0 2024-03-06T07:32:51.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.246+0000 7fd60c276640 1 -- 172.21.15.59:0/2915047071 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6040d22b0 msgr2=0x7fd6040d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.246+0000 7fd60c276640 1 --2- 172.21.15.59:0/2915047071 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6040d22b0 0x7fd6040d2710 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd6000097d0 tx=0x7fd60002b6b0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.246+0000 7fd60c276640 1 -- 172.21.15.59:0/2915047071 shutdown_connections 2024-03-06T07:32:51.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.246+0000 7fd60c276640 1 --2- 172.21.15.59:0/2915047071 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6040d22b0 0x7fd6040d2710 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.246+0000 7fd60c276640 1 -- 172.21.15.59:0/2915047071 >> 172.21.15.59:0/2915047071 conn(0x7fd6040d3cd0 msgr2=0x7fd6040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.246+0000 7fd60c276640 1 -- 172.21.15.59:0/2915047071 shutdown_connections 2024-03-06T07:32:51.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.246+0000 7fd60c276640 1 -- 172.21.15.59:0/2915047071 wait complete. 2024-03-06T07:32:51.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.248+0000 7fd60c276640 1 Processor -- start 2024-03-06T07:32:51.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.248+0000 7fd60c276640 1 -- start start 2024-03-06T07:32:51.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.248+0000 7fd60c276640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6040b41f0 0x7fd604160800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.248+0000 7fd60c276640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6040d22b0 0x7fd6041f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.248+0000 7fd60c276640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd60415dfe0 0x7fd6040cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.248+0000 7fd60c276640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6040b3d60 con 0x7fd6040b41f0 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.248+0000 7fd60c276640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6040b31e0 con 0x7fd6040d22b0 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.248+0000 7fd60c276640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6040d1c70 con 0x7fd60415dfe0 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.248+0000 7fd60a012640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6040b41f0 0x7fd604160800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.248+0000 7fd60a012640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6040b41f0 0x7fd604160800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50198/0 (socket says 172.21.15.59:50198) 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd60a012640 1 -- 172.21.15.59:0/3737887871 learned_addr learned my addr 172.21.15.59:0/3737887871 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd609811640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6040d22b0 0x7fd6041f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd60a813640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd60415dfe0 0x7fd6040cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd60a012640 1 -- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd60415dfe0 msgr2=0x7fd6040cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd60a012640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd60415dfe0 0x7fd6040cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd60a012640 1 -- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6040d22b0 msgr2=0x7fd6041f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd60a012640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6040d22b0 0x7fd6041f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd60a012640 1 -- 172.21.15.59:0/3737887871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6041f43e0 con 0x7fd6040b41f0 2024-03-06T07:32:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd60a012640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6040b41f0 0x7fd604160800 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800c4e0 tx=0x7fd5f800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd5f6ffd640 1 -- 172.21.15.59:0/3737887871 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8017070 con 0x7fd6040b41f0 2024-03-06T07:32:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd5f6ffd640 1 -- 172.21.15.59:0/3737887871 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f80035f0 con 0x7fd6040b41f0 2024-03-06T07:32:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd60c276640 1 -- 172.21.15.59:0/3737887871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6041f46c0 con 0x7fd6040b41f0 2024-03-06T07:32:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd5f6ffd640 1 -- 172.21.15.59:0/3737887871 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f80039e0 con 0x7fd6040b41f0 2024-03-06T07:32:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.249+0000 7fd60c276640 1 -- 172.21.15.59:0/3737887871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6041f4cd0 con 0x7fd6040b41f0 2024-03-06T07:32:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.250+0000 7fd60c276640 1 -- 172.21.15.59:0/3737887871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60404f330 con 0x7fd6040b41f0 2024-03-06T07:32:51.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.251+0000 7fd5f6ffd640 1 -- 172.21.15.59:0/3737887871 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5f801a070 con 0x7fd6040b41f0 2024-03-06T07:32:51.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.251+0000 7fd5f6ffd640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5e806b4d0 0x7fd5e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.251+0000 7fd5f6ffd640 1 -- 172.21.15.59:0/3737887871 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd5f8089580 con 0x7fd6040b41f0 2024-03-06T07:32:51.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.252+0000 7fd609811640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5e806b4d0 0x7fd5e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.252+0000 7fd609811640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5e806b4d0 0x7fd5e806d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd600034000 tx=0x7fd600008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.256+0000 7fd5f6ffd640 1 -- 172.21.15.59:0/3737887871 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5f8001190 con 0x7fd6040b41f0 2024-03-06T07:32:51.289 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:51.291 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:51.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.512+0000 7fd60c276640 1 -- 172.21.15.59:0/3737887871 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6041f50f0 con 0x7fd6040b41f0 2024-03-06T07:32:51.514 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.513+0000 7fd5f6ffd640 1 -- 172.21.15.59:0/3737887871 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fd5f805e0d0 con 0x7fd6040b41f0 2024-03-06T07:32:51.515 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:51.515 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:51.515 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:32:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 -- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5e806b4d0 msgr2=0x7fd5e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5e806b4d0 0x7fd5e806d990 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd600034000 tx=0x7fd600008030 comp rx=0 tx=0).stop 2024-03-06T07:32:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 -- 172.21.15.59:0/3737887871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6040b41f0 msgr2=0x7fd604160800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6040b41f0 0x7fd604160800 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800c4e0 tx=0x7fd5f800c9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 -- 172.21.15.59:0/3737887871 shutdown_connections 2024-03-06T07:32:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd5e806b4d0 0x7fd5e806d990 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd60415dfe0 0x7fd6040cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6040d22b0 0x7fd6041f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.517 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 --2- 172.21.15.59:0/3737887871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6040b41f0 0x7fd604160800 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 -- 172.21.15.59:0/3737887871 >> 172.21.15.59:0/3737887871 conn(0x7fd6040d3cd0 msgr2=0x7fd60406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 -- 172.21.15.59:0/3737887871 shutdown_connections 2024-03-06T07:32:51.518 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:51.515+0000 7fd60c276640 1 -- 172.21.15.59:0/3737887871 wait complete. 2024-03-06T07:32:51.728 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:51.729 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:52.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:52.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.829+0000 7f83c3943640 1 Processor -- start 2024-03-06T07:32:52.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.830+0000 7f83c3943640 1 -- start start 2024-03-06T07:32:52.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.830+0000 7f83c3943640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f83bc0d22b0 0x7f83bc0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.830+0000 7f83c3943640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc0b3430 con 0x7f83bc0b41f0 2024-03-06T07:32:52.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.830+0000 7f83c3943640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc0b3710 con 0x7f83bc15dfe0 2024-03-06T07:32:52.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.830+0000 7f83c3943640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83bc0d2c50 con 0x7f83bc0d22b0 2024-03-06T07:32:52.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.830+0000 7f83c0ede640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f83bc0d22b0 0x7f83bc0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.830+0000 7f83c1ee0640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f83bc0b41f0 0x7f83bc0b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:52500/0 (socket says 172.21.15.59:52500) 2024-03-06T07:32:52.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.830+0000 7f83c1ee0640 1 -- 172.21.15.59:0/462703233 learned_addr learned my addr 172.21.15.59:0/462703233 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:52.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.831+0000 7f83c0ede640 1 -- 172.21.15.59:0/462703233 >> v1:172.21.15.164:6789/0 conn(0x7f83bc15dfe0 legacy=0x7f83bc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.831+0000 7f83c0ede640 1 -- 172.21.15.59:0/462703233 >> v1:172.21.15.59:6789/0 conn(0x7f83bc0b41f0 legacy=0x7f83bc0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.831+0000 7f83b3fff640 1 -- 172.21.15.59:0/462703233 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461734582 0 0) 0x7f83bc0b3430 con 0x7f83bc0b41f0 2024-03-06T07:32:52.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.831+0000 7f83c0ede640 1 -- 172.21.15.59:0/462703233 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc0b39f0 con 0x7f83bc0d22b0 2024-03-06T07:32:52.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.832+0000 7f83c0ede640 1 --2- 172.21.15.59:0/462703233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83bc0d22b0 0x7f83bc0d2710 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f83ac0097d0 tx=0x7f83ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55c8ce9c8e04775 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.832+0000 7f83b3fff640 1 -- 172.21.15.59:0/462703233 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac035020 con 0x7f83bc0d22b0 2024-03-06T07:32:52.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.832+0000 7f83b3fff640 1 -- 172.21.15.59:0/462703233 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac031830 con 0x7f83bc0d22b0 2024-03-06T07:32:52.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.832+0000 7f83b3fff640 1 -- 172.21.15.59:0/462703233 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac031c00 con 0x7f83bc0d22b0 2024-03-06T07:32:52.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.832+0000 7f83c3943640 1 -- 172.21.15.59:0/462703233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83bc0d22b0 msgr2=0x7f83bc0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.832+0000 7f83c3943640 1 --2- 172.21.15.59:0/462703233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83bc0d22b0 0x7f83bc0d2710 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f83ac0097d0 tx=0x7f83ac02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.832+0000 7f83c3943640 1 -- 172.21.15.59:0/462703233 shutdown_connections 2024-03-06T07:32:52.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.832+0000 7f83c3943640 1 --2- 172.21.15.59:0/462703233 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83bc0d22b0 0x7f83bc0d2710 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.832+0000 7f83c3943640 1 -- 172.21.15.59:0/462703233 >> 172.21.15.59:0/462703233 conn(0x7f83bc0d3cd0 msgr2=0x7f83bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.832+0000 7f83c3943640 1 -- 172.21.15.59:0/462703233 shutdown_connections 2024-03-06T07:32:52.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.832+0000 7f83c3943640 1 -- 172.21.15.59:0/462703233 wait complete. 2024-03-06T07:32:52.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.834+0000 7f83c3943640 1 Processor -- start 2024-03-06T07:32:52.836 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.834+0000 7f83c3943640 1 -- start start 2024-03-06T07:32:52.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.835+0000 7f83c3943640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83bc0b41f0 0x7f83bc1610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.835+0000 7f83c3943640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83bc0d22b0 0x7f83bc0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.835+0000 7f83c3943640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83bc15dfe0 0x7f83bc149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.835+0000 7f83c3943640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0b3d60 con 0x7f83bc0d22b0 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.835+0000 7f83c3943640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0b31e0 con 0x7f83bc15dfe0 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.835+0000 7f83c3943640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0d1c70 con 0x7f83bc0b41f0 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.835+0000 7f83c0ede640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83bc0d22b0 0x7f83bc0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.835+0000 7f83c0ede640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83bc0d22b0 0x7f83bc0b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50222/0 (socket says 172.21.15.59:50222) 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83c0ede640 1 -- 172.21.15.59:0/3552629484 learned_addr learned my addr 172.21.15.59:0/3552629484 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83c16df640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83bc0b41f0 0x7f83bc1610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83c1ee0640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83bc15dfe0 0x7f83bc149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83c0ede640 1 -- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83bc0b41f0 msgr2=0x7f83bc1610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83c0ede640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83bc0b41f0 0x7f83bc1610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83c0ede640 1 -- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83bc15dfe0 msgr2=0x7f83bc149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83c0ede640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83bc15dfe0 0x7f83bc149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83c0ede640 1 -- 172.21.15.59:0/3552629484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc0b9070 con 0x7f83bc0d22b0 2024-03-06T07:32:52.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83c0ede640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83bc0d22b0 0x7f83bc0b68d0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f83ac000c00 tx=0x7f83ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83b1ffb640 1 -- 172.21.15.59:0/3552629484 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac004230 con 0x7f83bc0d22b0 2024-03-06T07:32:52.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83b1ffb640 1 -- 172.21.15.59:0/3552629484 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac0043d0 con 0x7f83bc0d22b0 2024-03-06T07:32:52.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83c3943640 1 -- 172.21.15.59:0/3552629484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83bc0b9350 con 0x7f83bc0d22b0 2024-03-06T07:32:52.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83b1ffb640 1 -- 172.21.15.59:0/3552629484 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac03ca50 con 0x7f83bc0d22b0 2024-03-06T07:32:52.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.836+0000 7f83c3943640 1 -- 172.21.15.59:0/3552629484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83bc0b9910 con 0x7f83bc0d22b0 2024-03-06T07:32:52.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.837+0000 7f83c3943640 1 -- 172.21.15.59:0/3552629484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83bc04f330 con 0x7f83bc0d22b0 2024-03-06T07:32:52.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.838+0000 7f83b1ffb640 1 -- 172.21.15.59:0/3552629484 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83ac038020 con 0x7f83bc0d22b0 2024-03-06T07:32:52.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.838+0000 7f83b1ffb640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.838+0000 7f83b1ffb640 1 -- 172.21.15.59:0/3552629484 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f83ac0ac470 con 0x7f83bc0d22b0 2024-03-06T07:32:52.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.838+0000 7f83c16df640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.839+0000 7f83c16df640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f83b8006fd0 tx=0x7f83b8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:52.846+0000 7f83b1ffb640 1 -- 172.21.15.59:0/3552629484 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83ac041070 con 0x7f83bc0d22b0 2024-03-06T07:32:53.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.095+0000 7f83c3943640 1 -- 172.21.15.59:0/3552629484 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83bc15d4c0 con 0x7f83bc0d22b0 2024-03-06T07:32:53.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.095+0000 7f83b1ffb640 1 -- 172.21.15.59:0/3552629484 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f83ac07b360 con 0x7f83bc0d22b0 2024-03-06T07:32:53.097 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:53.097 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:53.098 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:32:53.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.097+0000 7f83c3943640 1 -- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 msgr2=0x7f83a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.097+0000 7f83c3943640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f83b8006fd0 tx=0x7f83b8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:53.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.097+0000 7f83c3943640 1 -- 172.21.15.59:0/3552629484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83bc0d22b0 msgr2=0x7f83bc0b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.097+0000 7f83c3943640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83bc0d22b0 0x7f83bc0b68d0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f83ac000c00 tx=0x7f83ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.097+0000 7f83c3943640 1 -- 172.21.15.59:0/3552629484 shutdown_connections 2024-03-06T07:32:53.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.097+0000 7f83c3943640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.097+0000 7f83c3943640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83bc15dfe0 0x7f83bc149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.097+0000 7f83c3943640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83bc0d22b0 0x7f83bc0b68d0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.097+0000 7f83c3943640 1 --2- 172.21.15.59:0/3552629484 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83bc0b41f0 0x7f83bc1610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.097+0000 7f83c3943640 1 -- 172.21.15.59:0/3552629484 >> 172.21.15.59:0/3552629484 conn(0x7f83bc0d3cd0 msgr2=0x7f83bc0b5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.100 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.098+0000 7f83c3943640 1 -- 172.21.15.59:0/3552629484 shutdown_connections 2024-03-06T07:32:53.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:53.098+0000 7f83c3943640 1 -- 172.21.15.59:0/3552629484 wait complete. 2024-03-06T07:32:53.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:53.312 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:54.312 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:54.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.416+0000 7fbf58239640 1 Processor -- start 2024-03-06T07:32:54.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.416+0000 7fbf58239640 1 -- start start 2024-03-06T07:32:54.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf58239640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf5015dfe0 0x7fbf500cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf58239640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500cf830 0x7fbf500d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf58239640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf500b3430 con 0x7fbf500d27f0 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf58239640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf500d2c50 con 0x7fbf5015dfe0 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf58239640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500b31e0 con 0x7fbf500cf830 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf55fd5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf5015dfe0 0x7fbf500cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf55fd5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbf5015dfe0 0x7fbf500cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55678/0 (socket says 172.21.15.59:55678) 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf55fd5640 1 -- 172.21.15.59:0/4140295065 learned_addr learned my addr 172.21.15.59:0/4140295065 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf557d4640 1 --2- 172.21.15.59:0/4140295065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500cf830 0x7fbf500d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf55fd5640 1 -- 172.21.15.59:0/4140295065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500cf830 msgr2=0x7fbf500d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf55fd5640 1 --2- 172.21.15.59:0/4140295065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500cf830 0x7fbf500d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf55fd5640 1 -- 172.21.15.59:0/4140295065 >> v1:172.21.15.164:6789/0 conn(0x7fbf500d27f0 legacy=0x7fbf500d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.417+0000 7fbf55fd5640 1 -- 172.21.15.59:0/4140295065 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf500b3710 con 0x7fbf5015dfe0 2024-03-06T07:32:54.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.418+0000 7fbf55fd5640 1 --2- 172.21.15.59:0/4140295065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5015dfe0 0x7fbf500cf2f0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c009870 tx=0x7fbf4c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1352a2ace2182d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.418+0000 7fbf54fd3640 1 -- 172.21.15.59:0/4140295065 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c034070 con 0x7fbf5015dfe0 2024-03-06T07:32:54.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.418+0000 7fbf54fd3640 1 -- 172.21.15.59:0/4140295065 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4c030840 con 0x7fbf5015dfe0 2024-03-06T07:32:54.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.418+0000 7fbf54fd3640 1 -- 172.21.15.59:0/4140295065 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c030c10 con 0x7fbf5015dfe0 2024-03-06T07:32:54.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.419+0000 7fbf58239640 1 -- 172.21.15.59:0/4140295065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5015dfe0 msgr2=0x7fbf500cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.419+0000 7fbf58239640 1 --2- 172.21.15.59:0/4140295065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5015dfe0 0x7fbf500cf2f0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c009870 tx=0x7fbf4c02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:54.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.419+0000 7fbf58239640 1 -- 172.21.15.59:0/4140295065 shutdown_connections 2024-03-06T07:32:54.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.419+0000 7fbf58239640 1 --2- 172.21.15.59:0/4140295065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500cf830 0x7fbf500d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.419+0000 7fbf58239640 1 --2- 172.21.15.59:0/4140295065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5015dfe0 0x7fbf500cf2f0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.419+0000 7fbf58239640 1 -- 172.21.15.59:0/4140295065 >> 172.21.15.59:0/4140295065 conn(0x7fbf500d3cd0 msgr2=0x7fbf500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.419+0000 7fbf58239640 1 -- 172.21.15.59:0/4140295065 shutdown_connections 2024-03-06T07:32:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.419+0000 7fbf58239640 1 -- 172.21.15.59:0/4140295065 wait complete. 2024-03-06T07:32:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.420+0000 7fbf58239640 1 Processor -- start 2024-03-06T07:32:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.420+0000 7fbf58239640 1 -- start start 2024-03-06T07:32:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf58239640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf500cf830 0x7fbf5014af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf58239640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500d27f0 0x7fbf5015f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf58239640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5015dfe0 0x7fbf50160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf58239640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500d1c70 con 0x7fbf5015dfe0 2024-03-06T07:32:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf58239640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500b3d60 con 0x7fbf500cf830 2024-03-06T07:32:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf58239640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf500d1df0 con 0x7fbf500d27f0 2024-03-06T07:32:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf567d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5015dfe0 0x7fbf50160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf567d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5015dfe0 0x7fbf50160490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55688/0 (socket says 172.21.15.59:55688) 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf567d6640 1 -- 172.21.15.59:0/3182884048 learned_addr learned my addr 172.21.15.59:0/3182884048 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf557d4640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500d27f0 0x7fbf5015f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf55fd5640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf500cf830 0x7fbf5014af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf567d6640 1 -- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500d27f0 msgr2=0x7fbf5015f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf567d6640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500d27f0 0x7fbf5015f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf567d6640 1 -- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf500cf830 msgr2=0x7fbf5014af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf567d6640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf500cf830 0x7fbf5014af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf567d6640 1 -- 172.21.15.59:0/3182884048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5015a330 con 0x7fbf5015dfe0 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf55fd5640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf500cf830 0x7fbf5014af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf567d6640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5015dfe0 0x7fbf50160490 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c7a0 tx=0x7fbf4800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf467fc640 1 -- 172.21.15.59:0/3182884048 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48003b70 con 0x7fbf5015dfe0 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf467fc640 1 -- 172.21.15.59:0/3182884048 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf48003d30 con 0x7fbf5015dfe0 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf58239640 1 -- 172.21.15.59:0/3182884048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf500cda60 con 0x7fbf5015dfe0 2024-03-06T07:32:54.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.421+0000 7fbf467fc640 1 -- 172.21.15.59:0/3182884048 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4800aa90 con 0x7fbf5015dfe0 2024-03-06T07:32:54.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.422+0000 7fbf58239640 1 -- 172.21.15.59:0/3182884048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf500ce020 con 0x7fbf5015dfe0 2024-03-06T07:32:54.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.423+0000 7fbf58239640 1 -- 172.21.15.59:0/3182884048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf500cde50 con 0x7fbf5015dfe0 2024-03-06T07:32:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.423+0000 7fbf467fc640 1 -- 172.21.15.59:0/3182884048 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf4800ac30 con 0x7fbf5015dfe0 2024-03-06T07:32:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.423+0000 7fbf467fc640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf3406b4d0 0x7fbf3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.424+0000 7fbf467fc640 1 -- 172.21.15.59:0/3182884048 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbf4808a170 con 0x7fbf5015dfe0 2024-03-06T07:32:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.424+0000 7fbf55fd5640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf3406b4d0 0x7fbf3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.424+0000 7fbf55fd5640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf3406b4d0 0x7fbf3406d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c036000 tx=0x7fbf4c03b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.428+0000 7fbf467fc640 1 -- 172.21.15.59:0/3182884048 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf48058fd0 con 0x7fbf5015dfe0 2024-03-06T07:32:54.649 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.647+0000 7fbf58239640 1 -- 172.21.15.59:0/3182884048 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf500cdc30 con 0x7fbf5015dfe0 2024-03-06T07:32:54.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.648+0000 7fbf467fc640 1 -- 172.21.15.59:0/3182884048 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fbf480588d0 con 0x7fbf5015dfe0 2024-03-06T07:32:54.650 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:54.650 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:54.650 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:32:54.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 -- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf3406b4d0 msgr2=0x7fbf3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf3406b4d0 0x7fbf3406d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c036000 tx=0x7fbf4c03b040 comp rx=0 tx=0).stop 2024-03-06T07:32:54.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 -- 172.21.15.59:0/3182884048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5015dfe0 msgr2=0x7fbf50160490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5015dfe0 0x7fbf50160490 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c7a0 tx=0x7fbf4800cc60 comp rx=0 tx=0).stop 2024-03-06T07:32:54.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 -- 172.21.15.59:0/3182884048 shutdown_connections 2024-03-06T07:32:54.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fbf3406b4d0 0x7fbf3406d990 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf5015dfe0 0x7fbf50160490 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500d27f0 0x7fbf5015f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 --2- 172.21.15.59:0/3182884048 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf500cf830 0x7fbf5014af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 -- 172.21.15.59:0/3182884048 >> 172.21.15.59:0/3182884048 conn(0x7fbf500d3cd0 msgr2=0x7fbf5006d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 -- 172.21.15.59:0/3182884048 shutdown_connections 2024-03-06T07:32:54.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:54.650+0000 7fbf58239640 1 -- 172.21.15.59:0/3182884048 wait complete. 2024-03-06T07:32:54.864 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:54.864 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:55.865 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:55.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.965+0000 7f5a7eb5e640 1 Processor -- start 2024-03-06T07:32:55.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.965+0000 7f5a7eb5e640 1 -- start start 2024-03-06T07:32:55.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7eb5e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5a7815dfe0 0x7f5a780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7eb5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a780cf830 0x7f5a780d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7eb5e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a780d27f0 0x7f5a780d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7eb5e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a780d2c50 con 0x7f5a780d27f0 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7eb5e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a780b31e0 con 0x7f5a7815dfe0 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7eb5e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a780b3d60 con 0x7f5a780cf830 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7d0fb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a780d27f0 0x7f5a780d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7d0fb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5a780d27f0 0x7f5a780d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55702/0 (socket says 172.21.15.59:55702) 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7d0fb640 1 -- 172.21.15.59:0/3656715935 learned_addr learned my addr 172.21.15.59:0/3656715935 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a77fff640 1 --2- 172.21.15.59:0/3656715935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a780cf830 0x7f5a780d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7c8fa640 1 --2- 172.21.15.59:0/3656715935 >> v2:172.21.15.164:3300/0 conn(0x7f5a7815dfe0 0x7f5a780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7d0fb640 1 -- 172.21.15.59:0/3656715935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a780cf830 msgr2=0x7f5a780d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7d0fb640 1 --2- 172.21.15.59:0/3656715935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a780cf830 0x7f5a780d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7d0fb640 1 -- 172.21.15.59:0/3656715935 >> v2:172.21.15.164:3300/0 conn(0x7f5a7815dfe0 msgr2=0x7f5a780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7d0fb640 1 --2- 172.21.15.59:0/3656715935 >> v2:172.21.15.164:3300/0 conn(0x7f5a7815dfe0 0x7f5a780cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.966+0000 7f5a7d0fb640 1 -- 172.21.15.59:0/3656715935 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a780b3430 con 0x7f5a780d27f0 2024-03-06T07:32:55.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.967+0000 7f5a7d0fb640 1 --2- 172.21.15.59:0/3656715935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a780d27f0 0x7f5a780d0300 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f5a78067990 tx=0x7f5a7002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45de155ad74a96a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.967+0000 7f5a777fe640 1 -- 172.21.15.59:0/3656715935 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7002ca70 con 0x7f5a780d27f0 2024-03-06T07:32:55.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.967+0000 7f5a777fe640 1 -- 172.21.15.59:0/3656715935 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7002cc10 con 0x7f5a780d27f0 2024-03-06T07:32:55.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.967+0000 7f5a777fe640 1 -- 172.21.15.59:0/3656715935 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a700343f0 con 0x7f5a780d27f0 2024-03-06T07:32:55.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.968+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/3656715935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a780d27f0 msgr2=0x7f5a780d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.968+0000 7f5a7eb5e640 1 --2- 172.21.15.59:0/3656715935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a780d27f0 0x7f5a780d0300 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f5a78067990 tx=0x7f5a7002bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.968+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/3656715935 shutdown_connections 2024-03-06T07:32:55.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.968+0000 7f5a7eb5e640 1 --2- 172.21.15.59:0/3656715935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a780d27f0 0x7f5a780d0300 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.968+0000 7f5a7eb5e640 1 --2- 172.21.15.59:0/3656715935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a780cf830 0x7f5a780d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.968+0000 7f5a7eb5e640 1 --2- 172.21.15.59:0/3656715935 >> v2:172.21.15.164:3300/0 conn(0x7f5a7815dfe0 0x7f5a780cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.968+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/3656715935 >> 172.21.15.59:0/3656715935 conn(0x7f5a780d3cd0 msgr2=0x7f5a780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.968+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/3656715935 shutdown_connections 2024-03-06T07:32:55.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.968+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/3656715935 wait complete. 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.970+0000 7f5a7eb5e640 1 Processor -- start 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.970+0000 7f5a7eb5e640 1 -- start start 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7eb5e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a780cf830 0x7f5a7814f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7eb5e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a780d27f0 0x7f5a7815e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7eb5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a7815dfe0 0x7f5a7815bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7eb5e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a780d1c70 con 0x7f5a780cf830 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7eb5e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a780b39f0 con 0x7f5a780d27f0 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7eb5e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a780d1df0 con 0x7f5a7815dfe0 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7c8fa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a780cf830 0x7f5a7814f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7c8fa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a780cf830 0x7f5a7814f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55714/0 (socket says 172.21.15.59:55714) 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7c8fa640 1 -- 172.21.15.59:0/4144060692 learned_addr learned my addr 172.21.15.59:0/4144060692 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7c8fa640 1 -- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a7815dfe0 msgr2=0x7f5a7815bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7c8fa640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a7815dfe0 0x7f5a7815bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a77fff640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a780d27f0 0x7f5a7815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7c8fa640 1 -- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a780d27f0 msgr2=0x7f5a7815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7c8fa640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a780d27f0 0x7f5a7815e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.971+0000 7f5a7c8fa640 1 -- 172.21.15.59:0/4144060692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a780cada0 con 0x7f5a780cf830 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.972+0000 7f5a7c8fa640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a780cf830 0x7f5a7814f1b0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000d620 tx=0x7f5a6000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.972+0000 7f5a757fa640 1 -- 172.21.15.59:0/4144060692 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000b620 con 0x7f5a780cf830 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.972+0000 7f5a757fa640 1 -- 172.21.15.59:0/4144060692 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a60019070 con 0x7f5a780cf830 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.972+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/4144060692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a780cb080 con 0x7f5a780cf830 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.972+0000 7f5a757fa640 1 -- 172.21.15.59:0/4144060692 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000ba00 con 0x7f5a780cf830 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.972+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/4144060692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a780cb640 con 0x7f5a780cf830 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.973+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/4144060692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7804f330 con 0x7f5a780cf830 2024-03-06T07:32:55.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.974+0000 7f5a757fa640 1 -- 172.21.15.59:0/4144060692 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a6000bba0 con 0x7f5a780cf830 2024-03-06T07:32:55.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.974+0000 7f5a757fa640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a5806b4d0 0x7f5a5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.974+0000 7f5a757fa640 1 -- 172.21.15.59:0/4144060692 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5a6008a640 con 0x7f5a780cf830 2024-03-06T07:32:55.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.974+0000 7f5a77fff640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a5806b4d0 0x7f5a5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.975+0000 7f5a77fff640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a5806b4d0 0x7f5a5806d990 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5a7815ccd0 tx=0x7f5a68008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:55.978+0000 7f5a757fa640 1 -- 172.21.15.59:0/4144060692 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a600594a0 con 0x7f5a780cf830 2024-03-06T07:32:56.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.199+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/4144060692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a780b9b90 con 0x7f5a780cf830 2024-03-06T07:32:56.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.200+0000 7f5a757fa640 1 -- 172.21.15.59:0/4144060692 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f5a60058da0 con 0x7f5a780cf830 2024-03-06T07:32:56.202 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:56.202 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:56.202 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:32:56.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a5806b4d0 msgr2=0x7f5a5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a5806b4d0 0x7f5a5806d990 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5a7815ccd0 tx=0x7f5a68008040 comp rx=0 tx=0).stop 2024-03-06T07:32:56.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/4144060692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a780cf830 msgr2=0x7f5a7814f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a780cf830 0x7f5a7814f1b0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000d620 tx=0x7f5a6000dae0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/4144060692 shutdown_connections 2024-03-06T07:32:56.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5a5806b4d0 0x7f5a5806d990 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a7815dfe0 0x7f5a7815bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a780d27f0 0x7f5a7815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 --2- 172.21.15.59:0/4144060692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a780cf830 0x7f5a7814f1b0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/4144060692 >> 172.21.15.59:0/4144060692 conn(0x7f5a780d3cd0 msgr2=0x7f5a780cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/4144060692 shutdown_connections 2024-03-06T07:32:56.205 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:56.202+0000 7f5a7eb5e640 1 -- 172.21.15.59:0/4144060692 wait complete. 2024-03-06T07:32:56.416 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:56.417 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:57.417 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:57.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.517+0000 7f20d27d7640 1 Processor -- start 2024-03-06T07:32:57.519 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.517+0000 7f20d27d7640 1 -- start start 2024-03-06T07:32:57.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.518+0000 7f20d27d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20cc0d22b0 0x7f20cc0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.518+0000 7f20d27d7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f20cc0b41f0 0x7f20cc0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.518+0000 7f20d27d7640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20cc0b3430 con 0x7f20cc15dfe0 2024-03-06T07:32:57.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.518+0000 7f20d27d7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20cc0d2c50 con 0x7f20cc0b41f0 2024-03-06T07:32:57.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.518+0000 7f20d27d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc0b31e0 con 0x7f20cc0d22b0 2024-03-06T07:32:57.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.518+0000 7f20cbfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f20cc15dfe0 0x7f20cc0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54978/0 (socket says 172.21.15.59:54978) 2024-03-06T07:32:57.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.518+0000 7f20cbfff640 1 -- 172.21.15.59:0/509581881 learned_addr learned my addr 172.21.15.59:0/509581881 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:57.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.518+0000 7f20d0d74640 1 --2- 172.21.15.59:0/509581881 >> v2:172.21.15.164:3300/0 conn(0x7f20cc0b41f0 0x7f20cc0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.518+0000 7f20cb7fe640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20cc0d22b0 0x7f20cc0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.518+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715141512 0 0) 0x7f20cc0b3430 con 0x7f20cc15dfe0 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.518+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b4003520 con 0x7f20cc15dfe0 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1458172938 0 0) 0x7f20b4003520 con 0x7f20cc15dfe0 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20cc0d22b0 msgr2=0x7f20cc0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20cc0d22b0 0x7f20cc0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 >> v2:172.21.15.164:3300/0 conn(0x7f20cc0b41f0 msgr2=0x7f20cc0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 --2- 172.21.15.59:0/509581881 >> v2:172.21.15.164:3300/0 conn(0x7f20cc0b41f0 0x7f20cc0b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc0b3710 con 0x7f20cc15dfe0 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f20bc001670 con 0x7f20cc15dfe0 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20d0d74640 1 --2- 172.21.15.59:0/509581881 >> v2:172.21.15.164:3300/0 conn(0x7f20cc0b41f0 0x7f20cc0b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 >> v1:172.21.15.59:6789/0 conn(0x7f20cc15dfe0 legacy=0x7f20cc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20cc0d22b0 0x7f20b40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.524 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20b4003f30 0x7f20b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20b4004d40 0x7f20b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc0b31e0 con 0x7f20b4004d40 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b400b9c0 con 0x7f20b4003f30 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b400bb40 con 0x7f20cc0d22b0 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20d0d74640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20b4004d40 0x7f20b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20cbfff640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20cc0d22b0 0x7f20b40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20cb7fe640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20b4003f30 0x7f20b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20d0d74640 1 -- 172.21.15.59:0/509581881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20cc0d22b0 msgr2=0x7f20b40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20d0d74640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20cc0d22b0 0x7f20b40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20d0d74640 1 -- 172.21.15.59:0/509581881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20b4003f30 msgr2=0x7f20b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20d0d74640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20b4003f30 0x7f20b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.519+0000 7f20d0d74640 1 -- 172.21.15.59:0/509581881 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b4003520 con 0x7f20b4004d40 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.520+0000 7f20d0d74640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20b4004d40 0x7f20b40051c0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f20c400e350 tx=0x7f20b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c18bafac6d4dc07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.520+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4039070 con 0x7f20b4004d40 2024-03-06T07:32:57.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.520+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c402cdf0 con 0x7f20b4004d40 2024-03-06T07:32:57.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.520+0000 7f20caffd640 1 -- 172.21.15.59:0/509581881 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4009de0 con 0x7f20b4004d40 2024-03-06T07:32:57.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.521+0000 7f20d27d7640 1 -- 172.21.15.59:0/509581881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20b4004d40 msgr2=0x7f20b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.521+0000 7f20d27d7640 1 --2- 172.21.15.59:0/509581881 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20b4004d40 0x7f20b40051c0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f20c400e350 tx=0x7f20b4005830 comp rx=0 tx=0).stop 2024-03-06T07:32:57.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.521+0000 7f20d0d74640 1 -- 172.21.15.59:0/509581881 reap_dead start 2024-03-06T07:32:57.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.521+0000 7f20d27d7640 1 -- 172.21.15.59:0/509581881 shutdown_connections 2024-03-06T07:32:57.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.521+0000 7f20d27d7640 1 -- 172.21.15.59:0/509581881 >> 172.21.15.59:0/509581881 conn(0x7f20cc0d3cd0 msgr2=0x7f20cc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.521+0000 7f20d27d7640 1 -- 172.21.15.59:0/509581881 shutdown_connections 2024-03-06T07:32:57.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.521+0000 7f20d27d7640 1 -- 172.21.15.59:0/509581881 wait complete. 2024-03-06T07:32:57.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.522+0000 7f20d27d7640 1 Processor -- start 2024-03-06T07:32:57.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.522+0000 7f20d27d7640 1 -- start start 2024-03-06T07:32:57.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d27d7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20b4004d40 0x7f20cc0b8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d27d7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc1f78d0 0x7f20cc0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d27d7640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20cc15c460 0x7f20cc0cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d27d7640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc0cfa00 con 0x7f20cc15c460 2024-03-06T07:32:57.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d27d7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc0b3d60 con 0x7f20cc1f78d0 2024-03-06T07:32:57.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d27d7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20cc0d1c70 con 0x7f20b4004d40 2024-03-06T07:32:57.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d0d74640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20cc15c460 0x7f20cc0cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d0d74640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20cc15c460 0x7f20cc0cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55738/0 (socket says 172.21.15.59:55738) 2024-03-06T07:32:57.531 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d0d74640 1 -- 172.21.15.59:0/3483542230 learned_addr learned my addr 172.21.15.59:0/3483542230 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:57.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20cbfff640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20b4004d40 0x7f20cc0b8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20cb7fe640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc1f78d0 0x7f20cc0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d0d74640 1 -- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20b4004d40 msgr2=0x7f20cc0b8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d0d74640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20b4004d40 0x7f20cc0b8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d0d74640 1 -- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc1f78d0 msgr2=0x7f20cc0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d0d74640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc1f78d0 0x7f20cc0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20d0d74640 1 -- 172.21.15.59:0/3483542230 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20cc1f39e0 con 0x7f20cc15c460 2024-03-06T07:32:57.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20cb7fe640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc1f78d0 0x7f20cc0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:57.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.523+0000 7f20cbfff640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20b4004d40 0x7f20cc0b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:57.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.524+0000 7f20d0d74640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20cc15c460 0x7f20cc0cb2e0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f20c4002230 tx=0x7f20c4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.524+0000 7f20c8ff9640 1 -- 172.21.15.59:0/3483542230 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4039070 con 0x7f20cc15c460 2024-03-06T07:32:57.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.524+0000 7f20c8ff9640 1 -- 172.21.15.59:0/3483542230 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c403a040 con 0x7f20cc15c460 2024-03-06T07:32:57.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.524+0000 7f20d27d7640 1 -- 172.21.15.59:0/3483542230 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20cc1f3cc0 con 0x7f20cc15c460 2024-03-06T07:32:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.524+0000 7f20c8ff9640 1 -- 172.21.15.59:0/3483542230 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4009de0 con 0x7f20cc15c460 2024-03-06T07:32:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.524+0000 7f20d27d7640 1 -- 172.21.15.59:0/3483542230 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20cc1f4280 con 0x7f20cc15c460 2024-03-06T07:32:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.525+0000 7f20d27d7640 1 -- 172.21.15.59:0/3483542230 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20cc068070 con 0x7f20cc15c460 2024-03-06T07:32:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.525+0000 7f20c8ff9640 1 -- 172.21.15.59:0/3483542230 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20c4046030 con 0x7f20cc15c460 2024-03-06T07:32:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.526+0000 7f20c8ff9640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20b806b500 0x7f20b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.526+0000 7f20c8ff9640 1 -- 172.21.15.59:0/3483542230 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f20c40abef0 con 0x7f20cc15c460 2024-03-06T07:32:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.526+0000 7f20cbfff640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20b806b500 0x7f20b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.527+0000 7f20cbfff640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20b806b500 0x7f20b806d9c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f20bc007db0 tx=0x7f20bc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.534 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.530+0000 7f20c8ff9640 1 -- 172.21.15.59:0/3483542230 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20cc068070 con 0x7f20cc15c460 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.748+0000 7f20d27d7640 1 -- 172.21.15.59:0/3483542230 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20cc04f330 con 0x7f20cc15c460 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.748+0000 7f20c8ff9640 1 -- 172.21.15.59:0/3483542230 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f20c404d060 con 0x7f20cc15c460 2024-03-06T07:32:57.751 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:57.751 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:57.751 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 -- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20b806b500 msgr2=0x7f20b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20b806b500 0x7f20b806d9c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f20bc007db0 tx=0x7f20bc009040 comp rx=0 tx=0).stop 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 -- 172.21.15.59:0/3483542230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20cc15c460 msgr2=0x7f20cc0cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20cc15c460 0x7f20cc0cb2e0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f20c4002230 tx=0x7f20c4009750 comp rx=0 tx=0).stop 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 -- 172.21.15.59:0/3483542230 shutdown_connections 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f20b806b500 0x7f20b806d9c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20cc15c460 0x7f20cc0cb2e0 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc1f78d0 0x7f20cc0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 --2- 172.21.15.59:0/3483542230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20b4004d40 0x7f20cc0b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 -- 172.21.15.59:0/3483542230 >> 172.21.15.59:0/3483542230 conn(0x7f20cc0d3cd0 msgr2=0x7f20cc0b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 -- 172.21.15.59:0/3483542230 shutdown_connections 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:57.750+0000 7f20d27d7640 1 -- 172.21.15.59:0/3483542230 wait complete. 2024-03-06T07:32:57.965 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:57.965 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:32:58.966 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:59.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.066+0000 7efdfa16c640 1 Processor -- start 2024-03-06T07:32:59.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.066+0000 7efdfa16c640 1 -- start start 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.066+0000 7efdfa16c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efdf40d04a0 0x7efdf40ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdfa16c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf4161980 con 0x7efdf415bb20 2024-03-06T07:32:59.071 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdfa16c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf4161c60 con 0x7efdf40cfff0 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdfa16c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf40d2500 con 0x7efdf40d04a0 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf37fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7efdf415bb20 0x7efdf40d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54980/0 (socket says 172.21.15.59:54980) 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf37fe640 1 -- 172.21.15.59:0/2993802085 learned_addr learned my addr 172.21.15.59:0/2993802085 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf3fff640 1 --2- 172.21.15.59:0/2993802085 >> v2:172.21.15.164:3300/0 conn(0x7efdf40d04a0 0x7efdf40ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866793264 0 0) 0x7efdf4161980 con 0x7efdf415bb20 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efddc003520 con 0x7efdf415bb20 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 561045632 0 0) 0x7efddc003520 con 0x7efdf415bb20 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 >> v2:172.21.15.164:3300/0 conn(0x7efdf40d04a0 msgr2=0x7efdf40ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 --2- 172.21.15.59:0/2993802085 >> v2:172.21.15.164:3300/0 conn(0x7efdf40d04a0 0x7efdf40ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 >> v1:172.21.15.164:6789/0 conn(0x7efdf40cfff0 legacy=0x7efdf40cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4161f40 con 0x7efdf415bb20 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7efde4001670 con 0x7efdf415bb20 2024-03-06T07:32:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 >> v1:172.21.15.59:6789/0 conn(0x7efdf415bb20 legacy=0x7efdf40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 --2- 172.21.15.59:0/2993802085 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efddc0039b0 0x7efddc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 --2- 172.21.15.59:0/2993802085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 0x7efddc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 --2- 172.21.15.59:0/2993802085 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efddc0051c0 0x7efddc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40d2500 con 0x7efddc004800 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efddc00c060 con 0x7efddc0039b0 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efddc00c260 con 0x7efddc0051c0 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.067+0000 7efdf2ffd640 1 --2- 172.21.15.59:0/2993802085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 0x7efddc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.068+0000 7efdf37fe640 1 --2- 172.21.15.59:0/2993802085 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efddc0039b0 0x7efddc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.068+0000 7efdf3fff640 1 --2- 172.21.15.59:0/2993802085 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efddc0051c0 0x7efddc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.068+0000 7efdf2ffd640 1 -- 172.21.15.59:0/2993802085 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efddc0051c0 msgr2=0x7efddc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.068+0000 7efdf2ffd640 1 --2- 172.21.15.59:0/2993802085 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efddc0051c0 0x7efddc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.068+0000 7efdf2ffd640 1 -- 172.21.15.59:0/2993802085 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efddc0039b0 msgr2=0x7efddc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.068+0000 7efdf2ffd640 1 --2- 172.21.15.59:0/2993802085 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efddc0039b0 0x7efddc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.068+0000 7efdf2ffd640 1 -- 172.21.15.59:0/2993802085 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efddc003520 con 0x7efddc004800 2024-03-06T07:32:59.073 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.068+0000 7efdf3fff640 1 -- 172.21.15.59:0/2993802085 reap_dead start 2024-03-06T07:32:59.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.068+0000 7efdf2ffd640 1 --2- 172.21.15.59:0/2993802085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 0x7efddc004c80 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7efde8001750 tx=0x7efddc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26c38f1d46fc4b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.069+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8001370 con 0x7efddc004800 2024-03-06T07:32:59.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.069+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde80052d0 con 0x7efddc004800 2024-03-06T07:32:59.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.069+0000 7efdf27fc640 1 -- 172.21.15.59:0/2993802085 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8003830 con 0x7efddc004800 2024-03-06T07:32:59.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.069+0000 7efdfa16c640 1 -- 172.21.15.59:0/2993802085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 msgr2=0x7efddc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.069+0000 7efdfa16c640 1 --2- 172.21.15.59:0/2993802085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 0x7efddc004c80 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7efde8001750 tx=0x7efddc005830 comp rx=0 tx=0).stop 2024-03-06T07:32:59.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.069+0000 7efdfa16c640 1 -- 172.21.15.59:0/2993802085 shutdown_connections 2024-03-06T07:32:59.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.069+0000 7efdfa16c640 1 --2- 172.21.15.59:0/2993802085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 0x7efddc004c80 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.069+0000 7efdfa16c640 1 -- 172.21.15.59:0/2993802085 >> 172.21.15.59:0/2993802085 conn(0x7efdf409a9f0 msgr2=0x7efdf409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.069+0000 7efdfa16c640 1 -- 172.21.15.59:0/2993802085 shutdown_connections 2024-03-06T07:32:59.074 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.069+0000 7efdfa16c640 1 -- 172.21.15.59:0/2993802085 wait complete. 2024-03-06T07:32:59.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.070+0000 7efdfa16c640 1 Processor -- start 2024-03-06T07:32:59.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.071+0000 7efdfa16c640 1 -- start start 2024-03-06T07:32:59.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdfa16c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 0x7efdf4149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdfa16c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdf406ccb0 0x7efdf40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.077 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdfa16c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdf406d130 0x7efdf40b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdfa16c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40d08d0 con 0x7efddc004800 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdfa16c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf4162500 con 0x7efdf406ccb0 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdfa16c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40d1980 con 0x7efdf406d130 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdf37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 0x7efdf4149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdf37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 0x7efdf4149f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55770/0 (socket says 172.21.15.59:55770) 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdf37fe640 1 -- 172.21.15.59:0/3774785106 learned_addr learned my addr 172.21.15.59:0/3774785106 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdf37fe640 1 -- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdf406d130 msgr2=0x7efdf40b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdf3fff640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdf406d130 0x7efdf40b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdf2ffd640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdf406ccb0 0x7efdf40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdf37fe640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdf406d130 0x7efdf40b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdf37fe640 1 -- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdf406ccb0 msgr2=0x7efdf40b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdf37fe640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdf406ccb0 0x7efdf40b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdf37fe640 1 -- 172.21.15.59:0/3774785106 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4157990 con 0x7efddc004800 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.072+0000 7efdf37fe640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 0x7efdf4149f70 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7efde40072f0 tx=0x7efde40077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.073+0000 7efddbfff640 1 -- 172.21.15.59:0/3774785106 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4001660 con 0x7efddc004800 2024-03-06T07:32:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.073+0000 7efddbfff640 1 -- 172.21.15.59:0/3774785106 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde4018070 con 0x7efddc004800 2024-03-06T07:32:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.073+0000 7efdfa16c640 1 -- 172.21.15.59:0/3774785106 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf406d700 con 0x7efddc004800 2024-03-06T07:32:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.073+0000 7efddbfff640 1 -- 172.21.15.59:0/3774785106 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde400fb80 con 0x7efddc004800 2024-03-06T07:32:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.073+0000 7efdfa16c640 1 -- 172.21.15.59:0/3774785106 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf4157f10 con 0x7efddc004800 2024-03-06T07:32:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.074+0000 7efdfa16c640 1 -- 172.21.15.59:0/3774785106 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf404f330 con 0x7efddc004800 2024-03-06T07:32:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.074+0000 7efddbfff640 1 -- 172.21.15.59:0/3774785106 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efde400fd20 con 0x7efddc004800 2024-03-06T07:32:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.074+0000 7efddbfff640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efde006b6a0 0x7efde006db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.075+0000 7efddbfff640 1 -- 172.21.15.59:0/3774785106 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7efde408a200 con 0x7efddc004800 2024-03-06T07:32:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.075+0000 7efdf2ffd640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efde006b6a0 0x7efde006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.075+0000 7efdf2ffd640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efde006b6a0 0x7efde006db60 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7efde8033000 tx=0x7efde8009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:59.081 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.079+0000 7efddbfff640 1 -- 172.21.15.59:0/3774785106 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efde4059060 con 0x7efddc004800 2024-03-06T07:32:59.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.355+0000 7efdfa16c640 1 -- 172.21.15.59:0/3774785106 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdf41580e0 con 0x7efddc004800 2024-03-06T07:32:59.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.356+0000 7efddbfff640 1 -- 172.21.15.59:0/3774785106 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7efde4058960 con 0x7efddc004800 2024-03-06T07:32:59.358 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:32:59.358 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:32:59.358 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:32:59.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 -- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efde006b6a0 msgr2=0x7efde006db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efde006b6a0 0x7efde006db60 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7efde8033000 tx=0x7efde8009030 comp rx=0 tx=0).stop 2024-03-06T07:32:59.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 -- 172.21.15.59:0/3774785106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 msgr2=0x7efdf4149f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 0x7efdf4149f70 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7efde40072f0 tx=0x7efde40077b0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 -- 172.21.15.59:0/3774785106 shutdown_connections 2024-03-06T07:32:59.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efde006b6a0 0x7efde006db60 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdf406d130 0x7efdf40b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdf406ccb0 0x7efdf40b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 --2- 172.21.15.59:0/3774785106 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efddc004800 0x7efdf4149f70 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 -- 172.21.15.59:0/3774785106 >> 172.21.15.59:0/3774785106 conn(0x7efdf409a9f0 msgr2=0x7efdf40b3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 -- 172.21.15.59:0/3774785106 shutdown_connections 2024-03-06T07:32:59.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:32:59.358+0000 7efdfa16c640 1 -- 172.21.15.59:0/3774785106 wait complete. 2024-03-06T07:32:59.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:32:59.573 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:00.573 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:00.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.677+0000 7f4c07a0e640 1 Processor -- start 2024-03-06T07:33:00.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.677+0000 7f4c07a0e640 1 -- start start 2024-03-06T07:33:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c07a0e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c0015bdd0 0x7f4c000c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c07a0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c0015fa10 0x7f4c000ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c07a0e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c000cebe0 con 0x7f4c000d16e0 2024-03-06T07:33:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c07a0e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c000ceec0 con 0x7f4c0015bdd0 2024-03-06T07:33:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c07a0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c000cf1a0 con 0x7f4c0015fa10 2024-03-06T07:33:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c057aa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c0015bdd0 0x7f4c000c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c057aa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c0015bdd0 0x7f4c000c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55784/0 (socket says 172.21.15.59:55784) 2024-03-06T07:33:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c057aa640 1 -- 172.21.15.59:0/2677880836 learned_addr learned my addr 172.21.15.59:0/2677880836 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:00.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c04fa9640 1 --2- 172.21.15.59:0/2677880836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c0015fa10 0x7f4c000ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c057aa640 1 -- 172.21.15.59:0/2677880836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c0015fa10 msgr2=0x7f4c000ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c057aa640 1 --2- 172.21.15.59:0/2677880836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c0015fa10 0x7f4c000ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c057aa640 1 -- 172.21.15.59:0/2677880836 >> v1:172.21.15.164:6789/0 conn(0x7f4c000d16e0 legacy=0x7f4c000d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.678+0000 7f4c057aa640 1 -- 172.21.15.59:0/2677880836 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c000cc8e0 con 0x7f4c0015bdd0 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.679+0000 7f4c057aa640 1 --2- 172.21.15.59:0/2677880836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0015bdd0 0x7f4c000c5610 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009870 tx=0x7f4bfc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c298bbb9e8398837 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.679+0000 7f4bf7fff640 1 -- 172.21.15.59:0/2677880836 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0317c0 con 0x7f4c0015bdd0 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.679+0000 7f4bf7fff640 1 -- 172.21.15.59:0/2677880836 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc037070 con 0x7f4c0015bdd0 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.679+0000 7f4bf7fff640 1 -- 172.21.15.59:0/2677880836 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc031b80 con 0x7f4c0015bdd0 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.680+0000 7f4c07a0e640 1 -- 172.21.15.59:0/2677880836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0015bdd0 msgr2=0x7f4c000c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.680+0000 7f4c07a0e640 1 --2- 172.21.15.59:0/2677880836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0015bdd0 0x7f4c000c5610 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009870 tx=0x7f4bfc02b860 comp rx=0 tx=0).stop 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.680+0000 7f4c07a0e640 1 -- 172.21.15.59:0/2677880836 shutdown_connections 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.680+0000 7f4c07a0e640 1 --2- 172.21.15.59:0/2677880836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c0015fa10 0x7f4c000ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.680+0000 7f4c07a0e640 1 --2- 172.21.15.59:0/2677880836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0015bdd0 0x7f4c000c5610 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.680+0000 7f4c07a0e640 1 -- 172.21.15.59:0/2677880836 >> 172.21.15.59:0/2677880836 conn(0x7f4c0009a9f0 msgr2=0x7f4c0009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:00.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.680+0000 7f4c07a0e640 1 -- 172.21.15.59:0/2677880836 shutdown_connections 2024-03-06T07:33:00.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.680+0000 7f4c07a0e640 1 -- 172.21.15.59:0/2677880836 wait complete. 2024-03-06T07:33:00.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.681+0000 7f4c07a0e640 1 Processor -- start 2024-03-06T07:33:00.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.681+0000 7f4c07a0e640 1 -- start start 2024-03-06T07:33:00.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c07a0e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c000d16e0 0x7f4c0015ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c07a0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c0015bdd0 0x7f4c000ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c07a0e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0015fa10 0x7f4c000cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c07a0e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0015c1b0 con 0x7f4c0015fa10 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c07a0e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c000cc5d0 con 0x7f4c000d16e0 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c07a0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c000ce7b0 con 0x7f4c0015bdd0 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c05fab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0015fa10 0x7f4c000cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c05fab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0015fa10 0x7f4c000cabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55792/0 (socket says 172.21.15.59:55792) 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c05fab640 1 -- 172.21.15.59:0/1184562940 learned_addr learned my addr 172.21.15.59:0/1184562940 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c057aa640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c000d16e0 0x7f4c0015ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c04fa9640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c0015bdd0 0x7f4c000ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c05fab640 1 -- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c0015bdd0 msgr2=0x7f4c000ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c05fab640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c0015bdd0 0x7f4c000ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c05fab640 1 -- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c000d16e0 msgr2=0x7f4c0015ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c05fab640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c000d16e0 0x7f4c0015ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c05fab640 1 -- 172.21.15.59:0/1184562940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c001e6880 con 0x7f4c0015fa10 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c04fa9640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c0015bdd0 0x7f4c000ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c05fab640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0015fa10 0x7f4c000cabc0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f4bf800c780 tx=0x7f4bf800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4bf5ffb640 1 -- 172.21.15.59:0/1184562940 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8004b60 con 0x7f4c0015fa10 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4bf5ffb640 1 -- 172.21.15.59:0/1184562940 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf8004d00 con 0x7f4c0015fa10 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4c07a0e640 1 -- 172.21.15.59:0/1184562940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c001e6b60 con 0x7f4c0015fa10 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.682+0000 7f4bf5ffb640 1 -- 172.21.15.59:0/1184562940 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf800aa90 con 0x7f4c0015fa10 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.683+0000 7f4c07a0e640 1 -- 172.21.15.59:0/1184562940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c001e7120 con 0x7f4c0015fa10 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.684+0000 7f4c07a0e640 1 -- 172.21.15.59:0/1184562940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c000d10a0 con 0x7f4c0015fa10 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.684+0000 7f4bf5ffb640 1 -- 172.21.15.59:0/1184562940 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bf800ac30 con 0x7f4c0015fa10 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.684+0000 7f4bf5ffb640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be406b4d0 0x7f4be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.684+0000 7f4bf5ffb640 1 -- 172.21.15.59:0/1184562940 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4bf808a460 con 0x7f4c0015fa10 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.685+0000 7f4c057aa640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be406b4d0 0x7f4be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.685+0000 7f4c057aa640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be406b4d0 0x7f4be406d990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc036000 tx=0x7f4bfc03d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.689+0000 7f4bf5ffb640 1 -- 172.21.15.59:0/1184562940 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4bf8020070 con 0x7f4c0015fa10 2024-03-06T07:33:00.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.906+0000 7f4c07a0e640 1 -- 172.21.15.59:0/1184562940 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c0004f330 con 0x7f4c0015fa10 2024-03-06T07:33:00.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.906+0000 7f4bf5ffb640 1 -- 172.21.15.59:0/1184562940 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f4bf80592b0 con 0x7f4c0015fa10 2024-03-06T07:33:00.909 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:00.909 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:00.909 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 -- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be406b4d0 msgr2=0x7f4be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be406b4d0 0x7f4be406d990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc036000 tx=0x7f4bfc03d040 comp rx=0 tx=0).stop 2024-03-06T07:33:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 -- 172.21.15.59:0/1184562940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0015fa10 msgr2=0x7f4c000cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0015fa10 0x7f4c000cabc0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f4bf800c780 tx=0x7f4bf800cc40 comp rx=0 tx=0).stop 2024-03-06T07:33:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 -- 172.21.15.59:0/1184562940 shutdown_connections 2024-03-06T07:33:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f4be406b4d0 0x7f4be406d990 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c0015fa10 0x7f4c000cabc0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c0015bdd0 0x7f4c000ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 --2- 172.21.15.59:0/1184562940 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c000d16e0 0x7f4c0015ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 -- 172.21.15.59:0/1184562940 >> 172.21.15.59:0/1184562940 conn(0x7f4c0009a9f0 msgr2=0x7f4c0006ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:00.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 -- 172.21.15.59:0/1184562940 shutdown_connections 2024-03-06T07:33:00.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:00.908+0000 7f4c07a0e640 1 -- 172.21.15.59:0/1184562940 wait complete. 2024-03-06T07:33:01.122 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:01.123 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:02.124 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.223+0000 7f696069a640 1 Processor -- start 2024-03-06T07:33:02.225 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.223+0000 7f696069a640 1 -- start start 2024-03-06T07:33:02.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.224+0000 7f696069a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f695815dfe0 0x7f69580cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.224+0000 7f696069a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580cf830 0x7f69580d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.224+0000 7f696069a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69580b3430 con 0x7f69580d27f0 2024-03-06T07:33:02.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.224+0000 7f696069a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69580d2c50 con 0x7f695815dfe0 2024-03-06T07:33:02.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.224+0000 7f696069a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69580b31e0 con 0x7f69580cf830 2024-03-06T07:33:02.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.225+0000 7f695dc35640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580cf830 0x7f69580d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.225+0000 7f695e436640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f695815dfe0 0x7f69580cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.225+0000 7f695dc35640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580cf830 0x7f69580d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:39616/0 (socket says 172.21.15.59:39616) 2024-03-06T07:33:02.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.225+0000 7f695dc35640 1 -- 172.21.15.59:0/2161778140 learned_addr learned my addr 172.21.15.59:0/2161778140 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.225+0000 7f695dc35640 1 -- 172.21.15.59:0/2161778140 >> v2:172.21.15.164:3300/0 conn(0x7f695815dfe0 msgr2=0x7f69580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.225+0000 7f695dc35640 1 --2- 172.21.15.59:0/2161778140 >> v2:172.21.15.164:3300/0 conn(0x7f695815dfe0 0x7f69580cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.225+0000 7f695dc35640 1 -- 172.21.15.59:0/2161778140 >> v1:172.21.15.164:6789/0 conn(0x7f69580d27f0 legacy=0x7f69580d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.225+0000 7f695dc35640 1 -- 172.21.15.59:0/2161778140 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69580b3710 con 0x7f69580cf830 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.225+0000 7f695e436640 1 --2- 172.21.15.59:0/2161778140 >> v2:172.21.15.164:3300/0 conn(0x7f695815dfe0 0x7f69580cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.226+0000 7f695dc35640 1 --2- 172.21.15.59:0/2161778140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580cf830 0x7f69580d22b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6948009870 tx=0x7f694802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be68b3401e00a2a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.226+0000 7f695d434640 1 -- 172.21.15.59:0/2161778140 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948034070 con 0x7f69580cf830 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.226+0000 7f695d434640 1 -- 172.21.15.59:0/2161778140 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6948030840 con 0x7f69580cf830 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.226+0000 7f695d434640 1 -- 172.21.15.59:0/2161778140 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948030c10 con 0x7f69580cf830 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.227+0000 7f696069a640 1 -- 172.21.15.59:0/2161778140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580cf830 msgr2=0x7f69580d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.227+0000 7f696069a640 1 --2- 172.21.15.59:0/2161778140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580cf830 0x7f69580d22b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6948009870 tx=0x7f694802b980 comp rx=0 tx=0).stop 2024-03-06T07:33:02.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.227+0000 7f696069a640 1 -- 172.21.15.59:0/2161778140 shutdown_connections 2024-03-06T07:33:02.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.227+0000 7f696069a640 1 --2- 172.21.15.59:0/2161778140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580cf830 0x7f69580d22b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.227+0000 7f696069a640 1 --2- 172.21.15.59:0/2161778140 >> v2:172.21.15.164:3300/0 conn(0x7f695815dfe0 0x7f69580cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.227+0000 7f696069a640 1 -- 172.21.15.59:0/2161778140 >> 172.21.15.59:0/2161778140 conn(0x7f69580d3cd0 msgr2=0x7f69580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.227+0000 7f696069a640 1 -- 172.21.15.59:0/2161778140 shutdown_connections 2024-03-06T07:33:02.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.227+0000 7f696069a640 1 -- 172.21.15.59:0/2161778140 wait complete. 2024-03-06T07:33:02.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.228+0000 7f696069a640 1 Processor -- start 2024-03-06T07:33:02.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.228+0000 7f696069a640 1 -- start start 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f696069a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69580cf830 0x7f69580cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f696069a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580d27f0 0x7f69581f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f696069a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f695815dfe0 0x7f69581f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f696069a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69580d1c70 con 0x7f69580cf830 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f696069a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69580b3d60 con 0x7f695815dfe0 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f696069a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69580d1df0 con 0x7f69580d27f0 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f695e436640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69580cf830 0x7f69580cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f695e436640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69580cf830 0x7f69580cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55796/0 (socket says 172.21.15.59:55796) 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f695e436640 1 -- 172.21.15.59:0/3028821973 learned_addr learned my addr 172.21.15.59:0/3028821973 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f695dc35640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580d27f0 0x7f69581f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f695ec37640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f695815dfe0 0x7f69581f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f695e436640 1 -- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580d27f0 msgr2=0x7f69581f6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f695e436640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580d27f0 0x7f69581f6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f695e436640 1 -- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f695815dfe0 msgr2=0x7f69581f71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f695e436640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f695815dfe0 0x7f69581f71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f695e436640 1 -- 172.21.15.59:0/3028821973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69580cc000 con 0x7f69580cf830 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f695e436640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69580cf830 0x7f69580cb7b0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f695400e960 tx=0x7f695400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f694effd640 1 -- 172.21.15.59:0/3028821973 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954017070 con 0x7f69580cf830 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f694effd640 1 -- 172.21.15.59:0/3028821973 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695400ca00 con 0x7f69580cf830 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f696069a640 1 -- 172.21.15.59:0/3028821973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69580cc2e0 con 0x7f69580cf830 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f694effd640 1 -- 172.21.15.59:0/3028821973 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695400cdd0 con 0x7f69580cf830 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.229+0000 7f696069a640 1 -- 172.21.15.59:0/3028821973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69580cc8a0 con 0x7f69580cf830 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.231+0000 7f696069a640 1 -- 172.21.15.59:0/3028821973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695804f330 con 0x7f69580cf830 2024-03-06T07:33:02.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.231+0000 7f694effd640 1 -- 172.21.15.59:0/3028821973 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f695401a070 con 0x7f69580cf830 2024-03-06T07:33:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.231+0000 7f694effd640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f693c06b500 0x7f693c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.231+0000 7f694effd640 1 -- 172.21.15.59:0/3028821973 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f695408a3a0 con 0x7f69580cf830 2024-03-06T07:33:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.232+0000 7f695dc35640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f693c06b500 0x7f693c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.232+0000 7f695dc35640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f693c06b500 0x7f693c06d9c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f6948036000 tx=0x7f6948008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.236+0000 7f694effd640 1 -- 172.21.15.59:0/3028821973 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6954058bb0 con 0x7f69580cf830 2024-03-06T07:33:02.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.457+0000 7f696069a640 1 -- 172.21.15.59:0/3028821973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69580747a0 con 0x7f69580cf830 2024-03-06T07:33:02.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.457+0000 7f694effd640 1 -- 172.21.15.59:0/3028821973 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f695405cc50 con 0x7f69580cf830 2024-03-06T07:33:02.459 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:02.459 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:02.460 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.459+0000 7f696069a640 1 -- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f693c06b500 msgr2=0x7f693c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.459+0000 7f696069a640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f693c06b500 0x7f693c06d9c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f6948036000 tx=0x7f6948008030 comp rx=0 tx=0).stop 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.459+0000 7f696069a640 1 -- 172.21.15.59:0/3028821973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69580cf830 msgr2=0x7f69580cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.459+0000 7f696069a640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69580cf830 0x7f69580cb7b0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f695400e960 tx=0x7f695400ee20 comp rx=0 tx=0).stop 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.459+0000 7f696069a640 1 -- 172.21.15.59:0/3028821973 shutdown_connections 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.459+0000 7f696069a640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f693c06b500 0x7f693c06d9c0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.460+0000 7f696069a640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f695815dfe0 0x7f69581f71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.460+0000 7f696069a640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69580d27f0 0x7f69581f6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.460+0000 7f696069a640 1 --2- 172.21.15.59:0/3028821973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69580cf830 0x7f69580cb7b0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.460+0000 7f696069a640 1 -- 172.21.15.59:0/3028821973 >> 172.21.15.59:0/3028821973 conn(0x7f69580d3cd0 msgr2=0x7f695806cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.460+0000 7f696069a640 1 -- 172.21.15.59:0/3028821973 shutdown_connections 2024-03-06T07:33:02.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:02.460+0000 7f696069a640 1 -- 172.21.15.59:0/3028821973 wait complete. 2024-03-06T07:33:02.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:02.674 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:03.674 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:03.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.775+0000 7fabe62df640 1 Processor -- start 2024-03-06T07:33:03.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.775+0000 7fabe62df640 1 -- start start 2024-03-06T07:33:03.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.775+0000 7fabe62df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe015dfe0 0x7fabe00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.775+0000 7fabe62df640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fabe00cf830 0x7fabe00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.775+0000 7fabe62df640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe00b3430 con 0x7fabe00d27f0 2024-03-06T07:33:03.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.775+0000 7fabe62df640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabe00d2c50 con 0x7fabe00cf830 2024-03-06T07:33:03.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.775+0000 7fabe62df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe00b31e0 con 0x7fabe015dfe0 2024-03-06T07:33:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.776+0000 7fabdffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe015dfe0 0x7fabe00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.776+0000 7fabdf7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fabe00cf830 0x7fabe00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.776+0000 7fabdffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe015dfe0 0x7fabe00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:51640/0 (socket says 172.21.15.59:51640) 2024-03-06T07:33:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.776+0000 7fabdffff640 1 -- 172.21.15.59:0/3026096682 learned_addr learned my addr 172.21.15.59:0/3026096682 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.776+0000 7fabdeffd640 1 -- 172.21.15.59:0/3026096682 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430326795 0 0) 0x7fabe00b3430 con 0x7fabe00d27f0 2024-03-06T07:33:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.776+0000 7fabdeffd640 1 -- 172.21.15.59:0/3026096682 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc8003520 con 0x7fabe00d27f0 2024-03-06T07:33:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.776+0000 7fabdffff640 1 -- 172.21.15.59:0/3026096682 >> v2:172.21.15.164:3300/0 conn(0x7fabe00cf830 msgr2=0x7fabe00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.776+0000 7fabdffff640 1 --2- 172.21.15.59:0/3026096682 >> v2:172.21.15.164:3300/0 conn(0x7fabe00cf830 0x7fabe00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.776+0000 7fabdffff640 1 -- 172.21.15.59:0/3026096682 >> v1:172.21.15.164:6789/0 conn(0x7fabe00d27f0 legacy=0x7fabe00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.776+0000 7fabdffff640 1 -- 172.21.15.59:0/3026096682 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe00b3710 con 0x7fabe015dfe0 2024-03-06T07:33:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.776+0000 7fabdf7fe640 1 --2- 172.21.15.59:0/3026096682 >> v2:172.21.15.164:3300/0 conn(0x7fabe00cf830 0x7fabe00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.777+0000 7fabdffff640 1 --2- 172.21.15.59:0/3026096682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe015dfe0 0x7fabe00cf2f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009870 tx=0x7fabd002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48f7db046cb9ee3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.777+0000 7fabdeffd640 1 -- 172.21.15.59:0/3026096682 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd0034070 con 0x7fabe015dfe0 2024-03-06T07:33:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.777+0000 7fabdeffd640 1 -- 172.21.15.59:0/3026096682 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd0030840 con 0x7fabe015dfe0 2024-03-06T07:33:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.777+0000 7fabdeffd640 1 -- 172.21.15.59:0/3026096682 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd0030c10 con 0x7fabe015dfe0 2024-03-06T07:33:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.778+0000 7fabe62df640 1 -- 172.21.15.59:0/3026096682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe015dfe0 msgr2=0x7fabe00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.778+0000 7fabe62df640 1 --2- 172.21.15.59:0/3026096682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe015dfe0 0x7fabe00cf2f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009870 tx=0x7fabd002b980 comp rx=0 tx=0).stop 2024-03-06T07:33:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.778+0000 7fabe62df640 1 -- 172.21.15.59:0/3026096682 shutdown_connections 2024-03-06T07:33:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.778+0000 7fabe62df640 1 --2- 172.21.15.59:0/3026096682 >> v2:172.21.15.164:3300/0 conn(0x7fabe00cf830 0x7fabe00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.778+0000 7fabe62df640 1 --2- 172.21.15.59:0/3026096682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe015dfe0 0x7fabe00cf2f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.778+0000 7fabe62df640 1 -- 172.21.15.59:0/3026096682 >> 172.21.15.59:0/3026096682 conn(0x7fabe00d3cd0 msgr2=0x7fabe00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.778+0000 7fabe62df640 1 -- 172.21.15.59:0/3026096682 shutdown_connections 2024-03-06T07:33:03.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.778+0000 7fabe62df640 1 -- 172.21.15.59:0/3026096682 wait complete. 2024-03-06T07:33:03.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.779+0000 7fabe62df640 1 Processor -- start 2024-03-06T07:33:03.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.779+0000 7fabe62df640 1 -- start start 2024-03-06T07:33:03.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabe62df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe00cf830 0x7fabe015f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabe62df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe00d27f0 0x7fabe00cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabe62df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabe015dfe0 0x7fabe0147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabe62df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe00d1c70 con 0x7fabe00cf830 2024-03-06T07:33:03.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabe62df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe00b3d60 con 0x7fabe015dfe0 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabe62df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe00d1df0 con 0x7fabe00d27f0 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabdffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe00cf830 0x7fabe015f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabdffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe00cf830 0x7fabe015f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45872/0 (socket says 172.21.15.59:45872) 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabdffff640 1 -- 172.21.15.59:0/180973743 learned_addr learned my addr 172.21.15.59:0/180973743 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabdf7fe640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe00d27f0 0x7fabe00cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabe487c640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabe015dfe0 0x7fabe0147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabdffff640 1 -- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe00d27f0 msgr2=0x7fabe00cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabdffff640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe00d27f0 0x7fabe00cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabdffff640 1 -- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabe015dfe0 msgr2=0x7fabe0147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabdffff640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabe015dfe0 0x7fabe0147ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.780+0000 7fabdffff640 1 -- 172.21.15.59:0/180973743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe015fe10 con 0x7fabe00cf830 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.781+0000 7fabdffff640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe00cf830 0x7fabe015f530 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fabd002b9c0 tx=0x7fabd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.781+0000 7fabdcff9640 1 -- 172.21.15.59:0/180973743 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd0034040 con 0x7fabe00cf830 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.781+0000 7fabdcff9640 1 -- 172.21.15.59:0/180973743 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd0039c90 con 0x7fabe00cf830 2024-03-06T07:33:03.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.781+0000 7fabe62df640 1 -- 172.21.15.59:0/180973743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe01600f0 con 0x7fabe00cf830 2024-03-06T07:33:03.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.781+0000 7fabdcff9640 1 -- 172.21.15.59:0/180973743 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd0043630 con 0x7fabe00cf830 2024-03-06T07:33:03.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.781+0000 7fabe62df640 1 -- 172.21.15.59:0/180973743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe01606b0 con 0x7fabe00cf830 2024-03-06T07:33:03.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.782+0000 7fabe62df640 1 -- 172.21.15.59:0/180973743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe004f330 con 0x7fabe00cf830 2024-03-06T07:33:03.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.782+0000 7fabdcff9640 1 -- 172.21.15.59:0/180973743 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabd004a060 con 0x7fabe00cf830 2024-03-06T07:33:03.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.783+0000 7fabdcff9640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabcc06b4d0 0x7fabcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:03.786 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.783+0000 7fabdcff9640 1 -- 172.21.15.59:0/180973743 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fabd00ac0f0 con 0x7fabe00cf830 2024-03-06T07:33:03.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.783+0000 7fabdf7fe640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabcc06b4d0 0x7fabcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:03.787 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.784+0000 7fabdf7fe640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabcc06b4d0 0x7fabcc06d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009560 tx=0x7fabd4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:03.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:03.787+0000 7fabdcff9640 1 -- 172.21.15.59:0/180973743 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabd003e070 con 0x7fabe00cf830 2024-03-06T07:33:04.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.004+0000 7fabe62df640 1 -- 172.21.15.59:0/180973743 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabe015d450 con 0x7fabe00cf830 2024-03-06T07:33:04.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.005+0000 7fabdcff9640 1 -- 172.21.15.59:0/180973743 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fabd007af50 con 0x7fabe00cf830 2024-03-06T07:33:04.007 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:04.007 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:04.007 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:04.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 -- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabcc06b4d0 msgr2=0x7fabcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabcc06b4d0 0x7fabcc06d990 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009560 tx=0x7fabd4008040 comp rx=0 tx=0).stop 2024-03-06T07:33:04.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 -- 172.21.15.59:0/180973743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe00cf830 msgr2=0x7fabe015f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe00cf830 0x7fabe015f530 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fabd002b9c0 tx=0x7fabd00046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 -- 172.21.15.59:0/180973743 shutdown_connections 2024-03-06T07:33:04.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fabcc06b4d0 0x7fabcc06d990 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fabe015dfe0 0x7fabe0147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fabe00d27f0 0x7fabe00cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 --2- 172.21.15.59:0/180973743 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fabe00cf830 0x7fabe015f530 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 -- 172.21.15.59:0/180973743 >> 172.21.15.59:0/180973743 conn(0x7fabe00d3cd0 msgr2=0x7fabe006bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 -- 172.21.15.59:0/180973743 shutdown_connections 2024-03-06T07:33:04.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:04.007+0000 7fabe62df640 1 -- 172.21.15.59:0/180973743 wait complete. 2024-03-06T07:33:04.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:04.222 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:05.222 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:05.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.322+0000 7fe1e073e640 1 Processor -- start 2024-03-06T07:33:05.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.323+0000 7fe1e073e640 1 -- start start 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.323+0000 7fe1e073e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe1d806b220 0x7fe1d815e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.323+0000 7fe1e073e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe1d806b600 0x7fe1d8162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.323+0000 7fe1e073e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d815ea30 con 0x7fe1d80ce920 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1e073e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d8161b70 con 0x7fe1d806b600 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1e073e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d815ed10 con 0x7fe1d806b220 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1decdb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe1d806b600 0x7fe1d8162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1ddcd9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe1d80ce920 0x7fe1d8071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34730/0 (socket says 172.21.15.59:34730) 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1ddcd9640 1 -- 172.21.15.59:0/1210996447 learned_addr learned my addr 172.21.15.59:0/1210996447 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1dd4d8640 1 -- 172.21.15.59:0/1210996447 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555539057 0 0) 0x7fe1d815ea30 con 0x7fe1d80ce920 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1dd4d8640 1 -- 172.21.15.59:0/1210996447 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c4003520 con 0x7fe1d80ce920 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1de4da640 1 --2- 172.21.15.59:0/1210996447 >> v2:172.21.15.164:3300/0 conn(0x7fe1d806b220 0x7fe1d815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1decdb640 1 -- 172.21.15.59:0/1210996447 >> v2:172.21.15.164:3300/0 conn(0x7fe1d806b220 msgr2=0x7fe1d815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1decdb640 1 --2- 172.21.15.59:0/1210996447 >> v2:172.21.15.164:3300/0 conn(0x7fe1d806b220 0x7fe1d815e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1decdb640 1 -- 172.21.15.59:0/1210996447 >> v1:172.21.15.59:6789/0 conn(0x7fe1d80ce920 legacy=0x7fe1d8071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.324+0000 7fe1decdb640 1 -- 172.21.15.59:0/1210996447 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d815eff0 con 0x7fe1d806b600 2024-03-06T07:33:05.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.325+0000 7fe1decdb640 1 --2- 172.21.15.59:0/1210996447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d806b600 0x7fe1d8162000 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80d0980 tx=0x7fe1d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b90d998973995c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.325+0000 7fe1dd4d8640 1 -- 172.21.15.59:0/1210996447 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4039070 con 0x7fe1d806b600 2024-03-06T07:33:05.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.325+0000 7fe1dd4d8640 1 -- 172.21.15.59:0/1210996447 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d402cc80 con 0x7fe1d806b600 2024-03-06T07:33:05.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.325+0000 7fe1dd4d8640 1 -- 172.21.15.59:0/1210996447 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4034640 con 0x7fe1d806b600 2024-03-06T07:33:05.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.325+0000 7fe1e073e640 1 -- 172.21.15.59:0/1210996447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d806b600 msgr2=0x7fe1d8162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.325+0000 7fe1e073e640 1 --2- 172.21.15.59:0/1210996447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d806b600 0x7fe1d8162000 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80d0980 tx=0x7fe1d402bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.326+0000 7fe1e073e640 1 -- 172.21.15.59:0/1210996447 shutdown_connections 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.326+0000 7fe1e073e640 1 --2- 172.21.15.59:0/1210996447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d806b600 0x7fe1d8162000 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.326+0000 7fe1e073e640 1 --2- 172.21.15.59:0/1210996447 >> v2:172.21.15.164:3300/0 conn(0x7fe1d806b220 0x7fe1d815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.326+0000 7fe1e073e640 1 -- 172.21.15.59:0/1210996447 >> 172.21.15.59:0/1210996447 conn(0x7fe1d806c0c0 msgr2=0x7fe1d806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.326+0000 7fe1e073e640 1 -- 172.21.15.59:0/1210996447 shutdown_connections 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.326+0000 7fe1e073e640 1 -- 172.21.15.59:0/1210996447 wait complete. 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.327+0000 7fe1e073e640 1 Processor -- start 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.327+0000 7fe1e073e640 1 -- start start 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.327+0000 7fe1e073e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d806b220 0x7fe1d80b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1e073e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1d806b600 0x7fe1d8158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1e073e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d80ce920 0x7fe1d8158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1e073e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d8072090 con 0x7fe1d80ce920 2024-03-06T07:33:05.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1e073e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80ce200 con 0x7fe1d806b220 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1e073e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80afe20 con 0x7fe1d806b600 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1decdb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d80ce920 0x7fe1d8158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1decdb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d80ce920 0x7fe1d8158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45894/0 (socket says 172.21.15.59:45894) 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1decdb640 1 -- 172.21.15.59:0/4264560411 learned_addr learned my addr 172.21.15.59:0/4264560411 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1de4da640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d806b220 0x7fe1d80b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1ddcd9640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1d806b600 0x7fe1d8158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1decdb640 1 -- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1d806b600 msgr2=0x7fe1d8158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1decdb640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1d806b600 0x7fe1d8158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1decdb640 1 -- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d806b220 msgr2=0x7fe1d80b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1decdb640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d806b220 0x7fe1d80b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1decdb640 1 -- 172.21.15.59:0/4264560411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d8160640 con 0x7fe1d80ce920 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1de4da640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d806b220 0x7fe1d80b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1decdb640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d80ce920 0x7fe1d8158d50 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7fe1d402c4e0 tx=0x7fe1d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1ceffd640 1 -- 172.21.15.59:0/4264560411 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4039050 con 0x7fe1d80ce920 2024-03-06T07:33:05.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1ceffd640 1 -- 172.21.15.59:0/4264560411 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d4045070 con 0x7fe1d80ce920 2024-03-06T07:33:05.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1e073e640 1 -- 172.21.15.59:0/4264560411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d8160920 con 0x7fe1d80ce920 2024-03-06T07:33:05.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1ceffd640 1 -- 172.21.15.59:0/4264560411 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d403e830 con 0x7fe1d80ce920 2024-03-06T07:33:05.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.328+0000 7fe1e073e640 1 -- 172.21.15.59:0/4264560411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d8160ee0 con 0x7fe1d80ce920 2024-03-06T07:33:05.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.330+0000 7fe1e073e640 1 -- 172.21.15.59:0/4264560411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d8161250 con 0x7fe1d80ce920 2024-03-06T07:33:05.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.330+0000 7fe1ceffd640 1 -- 172.21.15.59:0/4264560411 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1d403e9d0 con 0x7fe1d80ce920 2024-03-06T07:33:05.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.330+0000 7fe1ceffd640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.330+0000 7fe1ceffd640 1 -- 172.21.15.59:0/4264560411 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe1d40ad1b0 con 0x7fe1d80ce920 2024-03-06T07:33:05.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.331+0000 7fe1de4da640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.331+0000 7fe1de4da640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe1d81f7800 tx=0x7fe1d0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.335+0000 7fe1ceffd640 1 -- 172.21.15.59:0/4264560411 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1d8161250 con 0x7fe1d80ce920 2024-03-06T07:33:05.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.553+0000 7fe1e073e640 1 -- 172.21.15.59:0/4264560411 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1d804f330 con 0x7fe1d80ce920 2024-03-06T07:33:05.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.553+0000 7fe1ceffd640 1 -- 172.21.15.59:0/4264560411 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fe1d407c1c0 con 0x7fe1d80ce920 2024-03-06T07:33:05.555 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:05.555 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:05.556 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 -- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 msgr2=0x7fe1bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe1d81f7800 tx=0x7fe1d0008040 comp rx=0 tx=0).stop 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 -- 172.21.15.59:0/4264560411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d80ce920 msgr2=0x7fe1d8158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d80ce920 0x7fe1d8158d50 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7fe1d402c4e0 tx=0x7fe1d4004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 -- 172.21.15.59:0/4264560411 shutdown_connections 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe1bc06b4d0 0x7fe1bc06d990 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1d80ce920 0x7fe1d8158d50 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1d806b600 0x7fe1d8158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 --2- 172.21.15.59:0/4264560411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1d806b220 0x7fe1d80b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 -- 172.21.15.59:0/4264560411 >> 172.21.15.59:0/4264560411 conn(0x7fe1d806c0c0 msgr2=0x7fe1d806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 -- 172.21.15.59:0/4264560411 shutdown_connections 2024-03-06T07:33:05.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:05.555+0000 7fe1e073e640 1 -- 172.21.15.59:0/4264560411 wait complete. 2024-03-06T07:33:05.769 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:05.769 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:06.770 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:06.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.870+0000 7f73b77c9640 1 Processor -- start 2024-03-06T07:33:06.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.870+0000 7f73b77c9640 1 -- start start 2024-03-06T07:33:06.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.871+0000 7f73b77c9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f73b00d22b0 0x7f73b00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.871+0000 7f73b77c9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b00b3430 con 0x7f73b015dfe0 2024-03-06T07:33:06.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.871+0000 7f73b77c9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b00b3710 con 0x7f73b00b41f0 2024-03-06T07:33:06.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.871+0000 7f73b77c9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b00d2c50 con 0x7f73b00d22b0 2024-03-06T07:33:06.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.871+0000 7f73b5565640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f73b015dfe0 0x7f73b00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34742/0 (socket says 172.21.15.59:34742) 2024-03-06T07:33:06.875 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.871+0000 7f73b5565640 1 -- 172.21.15.59:0/4139109918 learned_addr learned my addr 172.21.15.59:0/4139109918 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.871+0000 7f73b4d64640 1 --2- 172.21.15.59:0/4139109918 >> v2:172.21.15.164:3300/0 conn(0x7f73b00d22b0 0x7f73b00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809401659 0 0) 0x7f73b00b3430 con 0x7f73b015dfe0 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f739c003520 con 0x7f73b015dfe0 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1816960303 0 0) 0x7f739c003520 con 0x7f73b015dfe0 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 >> v2:172.21.15.164:3300/0 conn(0x7f73b00d22b0 msgr2=0x7f73b00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 --2- 172.21.15.59:0/4139109918 >> v2:172.21.15.164:3300/0 conn(0x7f73b00d22b0 0x7f73b00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 >> v1:172.21.15.164:6789/0 conn(0x7f73b00b41f0 legacy=0x7f73b00b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00b39f0 con 0x7f73b015dfe0 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f73a4001670 con 0x7f73b015dfe0 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 >> v1:172.21.15.59:6789/0 conn(0x7f73b015dfe0 legacy=0x7f73b00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 --2- 172.21.15.59:0/4139109918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f739c0039b0 0x7f739c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 --2- 172.21.15.59:0/4139109918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f739c004800 0x7f739c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 --2- 172.21.15.59:0/4139109918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f739c0051c0 0x7f739c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c00c030 con 0x7f739c0051c0 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f739c00c1e0 con 0x7f739c004800 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f739c00c3e0 con 0x7f739c0039b0 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73b5d66640 1 --2- 172.21.15.59:0/4139109918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f739c0051c0 0x7f739c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73b5565640 1 --2- 172.21.15.59:0/4139109918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f739c0039b0 0x7f739c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73b4d64640 1 --2- 172.21.15.59:0/4139109918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f739c004800 0x7f739c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73b5d66640 1 -- 172.21.15.59:0/4139109918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f739c0039b0 msgr2=0x7f739c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73b5d66640 1 --2- 172.21.15.59:0/4139109918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f739c0039b0 0x7f739c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73b5d66640 1 -- 172.21.15.59:0/4139109918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f739c004800 msgr2=0x7f739c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73b5d66640 1 --2- 172.21.15.59:0/4139109918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f739c004800 0x7f739c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.872+0000 7f73b5d66640 1 -- 172.21.15.59:0/4139109918 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c003520 con 0x7f739c0051c0 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.873+0000 7f73b5d66640 1 -- 172.21.15.59:0/4139109918 reap_dead start 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.873+0000 7f73b5d66640 1 --2- 172.21.15.59:0/4139109918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f739c0051c0 0x7f739c00b9c0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f73a8001e50 tx=0x7f739c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84591dd7dc6b2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.873+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8001a70 con 0x7f739c0051c0 2024-03-06T07:33:06.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.873+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a8003bb0 con 0x7f739c0051c0 2024-03-06T07:33:06.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.873+0000 7f73a3fff640 1 -- 172.21.15.59:0/4139109918 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a80050d0 con 0x7f739c0051c0 2024-03-06T07:33:06.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.874+0000 7f73b77c9640 1 -- 172.21.15.59:0/4139109918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f739c0051c0 msgr2=0x7f739c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.874+0000 7f73b77c9640 1 --2- 172.21.15.59:0/4139109918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f739c0051c0 0x7f739c00b9c0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f73a8001e50 tx=0x7f739c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:06.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.874+0000 7f73b77c9640 1 -- 172.21.15.59:0/4139109918 shutdown_connections 2024-03-06T07:33:06.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.874+0000 7f73b77c9640 1 --2- 172.21.15.59:0/4139109918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f739c0051c0 0x7f739c00b9c0 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.874+0000 7f73b77c9640 1 -- 172.21.15.59:0/4139109918 >> 172.21.15.59:0/4139109918 conn(0x7f73b00d3cd0 msgr2=0x7f73b00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:06.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.874+0000 7f73b77c9640 1 -- 172.21.15.59:0/4139109918 shutdown_connections 2024-03-06T07:33:06.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.874+0000 7f73b77c9640 1 -- 172.21.15.59:0/4139109918 wait complete. 2024-03-06T07:33:06.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.876+0000 7f73b77c9640 1 Processor -- start 2024-03-06T07:33:06.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.876+0000 7f73b77c9640 1 -- start start 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b77c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f739c0051c0 0x7f73b01f3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b77c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b00cc8f0 0x7f73b0156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b77c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b015b650 0x7f73b015bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b77c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00b3d60 con 0x7f73b00cc8f0 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b77c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00b31e0 con 0x7f739c0051c0 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b77c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b015e3c0 con 0x7f73b015b650 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b4d64640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b00cc8f0 0x7f73b0156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b4d64640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b00cc8f0 0x7f73b0156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45906/0 (socket says 172.21.15.59:45906) 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b4d64640 1 -- 172.21.15.59:0/2461456469 learned_addr learned my addr 172.21.15.59:0/2461456469 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b5565640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f739c0051c0 0x7f73b01f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b5d66640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b015b650 0x7f73b015bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b4d64640 1 -- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b015b650 msgr2=0x7f73b015bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b4d64640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b015b650 0x7f73b015bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b4d64640 1 -- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f739c0051c0 msgr2=0x7f73b01f3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:06.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b4d64640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f739c0051c0 0x7f73b01f3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:06.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b4d64640 1 -- 172.21.15.59:0/2461456469 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00cce90 con 0x7f73b00cc8f0 2024-03-06T07:33:06.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.877+0000 7f73b5d66640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b015b650 0x7f73b015bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:06.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.878+0000 7f73b4d64640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b00cc8f0 0x7f73b0156560 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f73ac011910 tx=0x7f73ac011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.878+0000 7f73a1ffb640 1 -- 172.21.15.59:0/2461456469 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac00fa90 con 0x7f73b00cc8f0 2024-03-06T07:33:06.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.878+0000 7f73a1ffb640 1 -- 172.21.15.59:0/2461456469 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac00fc30 con 0x7f73b00cc8f0 2024-03-06T07:33:06.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.878+0000 7f73b77c9640 1 -- 172.21.15.59:0/2461456469 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b015eb60 con 0x7f73b00cc8f0 2024-03-06T07:33:06.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.878+0000 7f73a1ffb640 1 -- 172.21.15.59:0/2461456469 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac014440 con 0x7f73b00cc8f0 2024-03-06T07:33:06.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.878+0000 7f73b77c9640 1 -- 172.21.15.59:0/2461456469 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b01f3950 con 0x7f73b00cc8f0 2024-03-06T07:33:06.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.879+0000 7f73b77c9640 1 -- 172.21.15.59:0/2461456469 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b004f330 con 0x7f73b00cc8f0 2024-03-06T07:33:06.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.879+0000 7f73a1ffb640 1 -- 172.21.15.59:0/2461456469 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73ac012040 con 0x7f73b00cc8f0 2024-03-06T07:33:06.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.880+0000 7f73a1ffb640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f739406b500 0x7f739406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:06.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.880+0000 7f73a1ffb640 1 -- 172.21.15.59:0/2461456469 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f73ac08a7f0 con 0x7f73b00cc8f0 2024-03-06T07:33:06.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.880+0000 7f73b5565640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f739406b500 0x7f739406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:06.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.881+0000 7f73b5565640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f739406b500 0x7f739406d9c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f73a4007cb0 tx=0x7f73a4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:06.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:06.884+0000 7f73a1ffb640 1 -- 172.21.15.59:0/2461456469 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73ac019030 con 0x7f73b00cc8f0 2024-03-06T07:33:07.103 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.101+0000 7f73b77c9640 1 -- 172.21.15.59:0/2461456469 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73b01e8940 con 0x7f73b00cc8f0 2024-03-06T07:33:07.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.102+0000 7f73a1ffb640 1 -- 172.21.15.59:0/2461456469 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f73ac059650 con 0x7f73b00cc8f0 2024-03-06T07:33:07.104 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:07.104 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:07.104 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:07.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 -- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f739406b500 msgr2=0x7f739406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f739406b500 0x7f739406d9c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f73a4007cb0 tx=0x7f73a4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:07.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 -- 172.21.15.59:0/2461456469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b00cc8f0 msgr2=0x7f73b0156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:07.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b00cc8f0 0x7f73b0156560 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f73ac011910 tx=0x7f73ac011dd0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 -- 172.21.15.59:0/2461456469 shutdown_connections 2024-03-06T07:33:07.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f739406b500 0x7f739406d9c0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b015b650 0x7f73b015bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b00cc8f0 0x7f73b0156560 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 --2- 172.21.15.59:0/2461456469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f739c0051c0 0x7f73b01f3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:07.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 -- 172.21.15.59:0/2461456469 >> 172.21.15.59:0/2461456469 conn(0x7f73b00d3cd0 msgr2=0x7f73b00b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:07.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 -- 172.21.15.59:0/2461456469 shutdown_connections 2024-03-06T07:33:07.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:07.104+0000 7f73b77c9640 1 -- 172.21.15.59:0/2461456469 wait complete. 2024-03-06T07:33:07.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:07.318 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:08.319 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:08.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.420+0000 7f9f5ec5c640 1 Processor -- start 2024-03-06T07:33:08.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.420+0000 7f9f5ec5c640 1 -- start start 2024-03-06T07:33:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.420+0000 7f9f5ec5c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9f580cce40 0x7f9f5815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.420+0000 7f9f5ec5c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9f580cd7a0 0x7f9f580cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.420+0000 7f9f5ec5c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f58071f40 con 0x7f9f581515e0 2024-03-06T07:33:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.420+0000 7f9f5ec5c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f580cd2f0 con 0x7f9f580cce40 2024-03-06T07:33:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.420+0000 7f9f5ec5c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f58070a70 con 0x7f9f580cd7a0 2024-03-06T07:33:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.421+0000 7f9f57fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9f580cce40 0x7f9f5815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.421+0000 7f9f57fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9f580cce40 0x7f9f5815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45916/0 (socket says 172.21.15.59:45916) 2024-03-06T07:33:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.421+0000 7f9f57fff640 1 -- 172.21.15.59:0/2862170199 learned_addr learned my addr 172.21.15.59:0/2862170199 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.421+0000 7f9f5d1f9640 1 --2- 172.21.15.59:0/2862170199 >> v2:172.21.15.164:3300/0 conn(0x7f9f580cd7a0 0x7f9f580cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.421+0000 7f9f57fff640 1 -- 172.21.15.59:0/2862170199 >> v2:172.21.15.164:3300/0 conn(0x7f9f580cd7a0 msgr2=0x7f9f580cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:08.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.421+0000 7f9f57fff640 1 --2- 172.21.15.59:0/2862170199 >> v2:172.21.15.164:3300/0 conn(0x7f9f580cd7a0 0x7f9f580cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.421+0000 7f9f57fff640 1 -- 172.21.15.59:0/2862170199 >> v1:172.21.15.164:6789/0 conn(0x7f9f581515e0 legacy=0x7f9f58071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:08.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.421+0000 7f9f57fff640 1 -- 172.21.15.59:0/2862170199 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f58070d50 con 0x7f9f580cce40 2024-03-06T07:33:08.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f57fff640 1 --2- 172.21.15.59:0/2862170199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f580cce40 0x7f9f5815b430 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f9f580cc110 tx=0x7f9f4002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40c1f32bf7bea36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f577fe640 1 -- 172.21.15.59:0/2862170199 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f40035020 con 0x7f9f580cce40 2024-03-06T07:33:08.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f577fe640 1 -- 172.21.15.59:0/2862170199 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f40031640 con 0x7f9f580cce40 2024-03-06T07:33:08.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f577fe640 1 -- 172.21.15.59:0/2862170199 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4002d040 con 0x7f9f580cce40 2024-03-06T07:33:08.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/2862170199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f580cce40 msgr2=0x7f9f5815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f5ec5c640 1 --2- 172.21.15.59:0/2862170199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f580cce40 0x7f9f5815b430 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f9f580cc110 tx=0x7f9f4002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/2862170199 shutdown_connections 2024-03-06T07:33:08.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f5ec5c640 1 --2- 172.21.15.59:0/2862170199 >> v2:172.21.15.164:3300/0 conn(0x7f9f580cd7a0 0x7f9f580cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f5ec5c640 1 --2- 172.21.15.59:0/2862170199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f580cce40 0x7f9f5815b430 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/2862170199 >> 172.21.15.59:0/2862170199 conn(0x7f9f580cf8a0 msgr2=0x7f9f580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/2862170199 shutdown_connections 2024-03-06T07:33:08.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.422+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/2862170199 wait complete. 2024-03-06T07:33:08.427 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.424+0000 7f9f5ec5c640 1 Processor -- start 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.424+0000 7f9f5ec5c640 1 -- start start 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.424+0000 7f9f5ec5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f580cce40 0x7f9f581f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.424+0000 7f9f5ec5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f580cd7a0 0x7f9f5814afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.424+0000 7f9f5ec5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f581515e0 0x7f9f5814b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5ec5c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580ce0c0 con 0x7f9f581515e0 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5ec5c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f58071310 con 0x7f9f580cce40 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5ec5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f580cc3a0 con 0x7f9f580cd7a0 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5d1f9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f581515e0 0x7f9f5814b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5d1f9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f581515e0 0x7f9f5814b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45924/0 (socket says 172.21.15.59:45924) 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5d1f9640 1 -- 172.21.15.59:0/1747928979 learned_addr learned my addr 172.21.15.59:0/1747928979 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f57fff640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f580cd7a0 0x7f9f5814afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5c9f8640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f580cce40 0x7f9f581f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5d1f9640 1 -- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f580cd7a0 msgr2=0x7f9f5814afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5d1f9640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f580cd7a0 0x7f9f5814afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5d1f9640 1 -- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f580cce40 msgr2=0x7f9f581f1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5d1f9640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f580cce40 0x7f9f581f1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5d1f9640 1 -- 172.21.15.59:0/1747928979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5814c010 con 0x7f9f581515e0 2024-03-06T07:33:08.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5c9f8640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f580cce40 0x7f9f581f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:08.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5d1f9640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f581515e0 0x7f9f5814b6a0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f9f50007d80 tx=0x7f9f5000d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f557fa640 1 -- 172.21.15.59:0/1747928979 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50019070 con 0x7f9f581515e0 2024-03-06T07:33:08.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f557fa640 1 -- 172.21.15.59:0/1747928979 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5000dda0 con 0x7f9f581515e0 2024-03-06T07:33:08.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f557fa640 1 -- 172.21.15.59:0/1747928979 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50003bd0 con 0x7f9f581515e0 2024-03-06T07:33:08.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/1747928979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5814c2f0 con 0x7f9f581515e0 2024-03-06T07:33:08.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.425+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/1747928979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5815b460 con 0x7f9f581515e0 2024-03-06T07:33:08.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.426+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/1747928979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f58068840 con 0x7f9f581515e0 2024-03-06T07:33:08.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.427+0000 7f9f557fa640 1 -- 172.21.15.59:0/1747928979 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f5001c070 con 0x7f9f581515e0 2024-03-06T07:33:08.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.427+0000 7f9f557fa640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9f3806b500 0x7f9f3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:08.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.428+0000 7f9f557fa640 1 -- 172.21.15.59:0/1747928979 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9f5008a1e0 con 0x7f9f581515e0 2024-03-06T07:33:08.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.428+0000 7f9f5c9f8640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9f3806b500 0x7f9f3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:08.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.428+0000 7f9f5c9f8640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9f3806b500 0x7f9f3806d9c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800a830 tx=0x7f9f48008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:08.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.432+0000 7f9f557fa640 1 -- 172.21.15.59:0/1747928979 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f500590d0 con 0x7f9f581515e0 2024-03-06T07:33:08.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.649+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/1747928979 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f5804f330 con 0x7f9f581515e0 2024-03-06T07:33:08.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.650+0000 7f9f557fa640 1 -- 172.21.15.59:0/1747928979 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f9f500589d0 con 0x7f9f581515e0 2024-03-06T07:33:08.652 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:08.652 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:08.652 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:08.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9f3806b500 msgr2=0x7f9f3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9f3806b500 0x7f9f3806d9c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800a830 tx=0x7f9f48008040 comp rx=0 tx=0).stop 2024-03-06T07:33:08.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/1747928979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f581515e0 msgr2=0x7f9f5814b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:08.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f581515e0 0x7f9f5814b6a0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f9f50007d80 tx=0x7f9f5000d490 comp rx=0 tx=0).stop 2024-03-06T07:33:08.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/1747928979 shutdown_connections 2024-03-06T07:33:08.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9f3806b500 0x7f9f3806d9c0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9f581515e0 0x7f9f5814b6a0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f580cd7a0 0x7f9f5814afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 --2- 172.21.15.59:0/1747928979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f580cce40 0x7f9f581f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:08.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/1747928979 >> 172.21.15.59:0/1747928979 conn(0x7f9f580cf8a0 msgr2=0x7f9f58071830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:08.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/1747928979 shutdown_connections 2024-03-06T07:33:08.655 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:08.652+0000 7f9f5ec5c640 1 -- 172.21.15.59:0/1747928979 wait complete. 2024-03-06T07:33:08.866 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:08.866 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:09.867 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:09.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.967+0000 7f23455ae640 1 Processor -- start 2024-03-06T07:33:09.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.967+0000 7f23455ae640 1 -- start start 2024-03-06T07:33:09.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.967+0000 7f23455ae640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f234015bdd0 0x7f23400c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.967+0000 7f23455ae640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f234015fa10 0x7f23400ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.967+0000 7f23455ae640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23400cebe0 con 0x7f23400d16e0 2024-03-06T07:33:09.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.967+0000 7f23455ae640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23400ceec0 con 0x7f234015fa10 2024-03-06T07:33:09.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.967+0000 7f23455ae640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23400cf1a0 con 0x7f234015bdd0 2024-03-06T07:33:09.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233f7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f23400d16e0 0x7f23400d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34748/0 (socket says 172.21.15.59:34748) 2024-03-06T07:33:09.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233f7fe640 1 -- 172.21.15.59:0/3425124827 learned_addr learned my addr 172.21.15.59:0/3425124827 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:09.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233effd640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f234015bdd0 0x7f23400c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f2337fff640 1 --2- 172.21.15.59:0/3425124827 >> v2:172.21.15.164:3300/0 conn(0x7f234015fa10 0x7f23400ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705745470 0 0) 0x7f23400cebe0 con 0x7f23400d16e0 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f232c003520 con 0x7f23400d16e0 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4208310074 0 0) 0x7f232c003520 con 0x7f23400d16e0 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f234015bdd0 msgr2=0x7f23400c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f234015bdd0 0x7f23400c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 >> v2:172.21.15.164:3300/0 conn(0x7f234015fa10 msgr2=0x7f23400ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 --2- 172.21.15.59:0/3425124827 >> v2:172.21.15.164:3300/0 conn(0x7f234015fa10 0x7f23400ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23400cc8e0 con 0x7f23400d16e0 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f23380019f0 con 0x7f23400d16e0 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 >> v1:172.21.15.59:6789/0 conn(0x7f23400d16e0 legacy=0x7f23400d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232c0039b0 0x7f232c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f234015bdd0 0x7f232c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f232c004d40 0x7f232c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23400ceec0 con 0x7f232c004d40 2024-03-06T07:33:09.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23400cf1a0 con 0x7f232c0039b0 2024-03-06T07:33:09.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232c00b9c0 con 0x7f234015bdd0 2024-03-06T07:33:09.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233effd640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232c0039b0 0x7f232c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f233f7fe640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f232c004d40 0x7f232c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.968+0000 7f2337fff640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f234015bdd0 0x7f232c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.969+0000 7f233f7fe640 1 -- 172.21.15.59:0/3425124827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f234015bdd0 msgr2=0x7f232c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.969+0000 7f233f7fe640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f234015bdd0 0x7f232c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.969+0000 7f233f7fe640 1 -- 172.21.15.59:0/3425124827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232c0039b0 msgr2=0x7f232c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:09.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.969+0000 7f233f7fe640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232c0039b0 0x7f232c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.969+0000 7f233f7fe640 1 -- 172.21.15.59:0/3425124827 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c003520 con 0x7f232c004d40 2024-03-06T07:33:09.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.970+0000 7f233f7fe640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f232c004d40 0x7f232c0051c0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f23380013d0 tx=0x7f232c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e831d3f29dfbcdf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.970+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233800a4e0 con 0x7f232c004d40 2024-03-06T07:33:09.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.970+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2338034650 con 0x7f232c004d40 2024-03-06T07:33:09.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.970+0000 7f233e7fc640 1 -- 172.21.15.59:0/3425124827 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2338034a40 con 0x7f232c004d40 2024-03-06T07:33:09.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.970+0000 7f23455ae640 1 -- 172.21.15.59:0/3425124827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f232c004d40 msgr2=0x7f232c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.970+0000 7f23455ae640 1 --2- 172.21.15.59:0/3425124827 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f232c004d40 0x7f232c0051c0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f23380013d0 tx=0x7f232c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:09.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.970+0000 7f233f7fe640 1 -- 172.21.15.59:0/3425124827 reap_dead start 2024-03-06T07:33:09.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.971+0000 7f23455ae640 1 -- 172.21.15.59:0/3425124827 shutdown_connections 2024-03-06T07:33:09.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.971+0000 7f23455ae640 1 -- 172.21.15.59:0/3425124827 >> 172.21.15.59:0/3425124827 conn(0x7f234009a9f0 msgr2=0x7f234009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:09.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.971+0000 7f23455ae640 1 -- 172.21.15.59:0/3425124827 shutdown_connections 2024-03-06T07:33:09.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.971+0000 7f23455ae640 1 -- 172.21.15.59:0/3425124827 wait complete. 2024-03-06T07:33:09.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.972+0000 7f23455ae640 1 Processor -- start 2024-03-06T07:33:09.975 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.972+0000 7f23455ae640 1 -- start start 2024-03-06T07:33:09.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.973+0000 7f23455ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f234015a970 0x7f234006c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.973+0000 7f23455ae640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f234006cd60 0x7f23401f75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.973+0000 7f23455ae640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23401f7b00 0x7f234015c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.973+0000 7f23455ae640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23400ce7b0 con 0x7f234015a970 2024-03-06T07:33:09.977 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.973+0000 7f23455ae640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234015c1b0 con 0x7f234006cd60 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.973+0000 7f23455ae640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23400ce930 con 0x7f23401f7b00 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.973+0000 7f233effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f234015a970 0x7f234006c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.973+0000 7f233effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f234015a970 0x7f234006c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45940/0 (socket says 172.21.15.59:45940) 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.973+0000 7f233effd640 1 -- 172.21.15.59:0/899151116 learned_addr learned my addr 172.21.15.59:0/899151116 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.973+0000 7f2337fff640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f234006cd60 0x7f23401f75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.973+0000 7f233f7fe640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23401f7b00 0x7f234015c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f233effd640 1 -- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23401f7b00 msgr2=0x7f234015c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f233effd640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23401f7b00 0x7f234015c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f233effd640 1 -- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f234006cd60 msgr2=0x7f23401f75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f233effd640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f234006cd60 0x7f23401f75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f233effd640 1 -- 172.21.15.59:0/899151116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23401f3ec0 con 0x7f234015a970 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f2337fff640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f234006cd60 0x7f23401f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f233effd640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f234015a970 0x7f234006c820 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f2330011910 tx=0x7f2330011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f23377fe640 1 -- 172.21.15.59:0/899151116 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233000fa90 con 0x7f234015a970 2024-03-06T07:33:09.978 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f23377fe640 1 -- 172.21.15.59:0/899151116 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233000fc30 con 0x7f234015a970 2024-03-06T07:33:09.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f23455ae640 1 -- 172.21.15.59:0/899151116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234006d270 con 0x7f234015a970 2024-03-06T07:33:09.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f23377fe640 1 -- 172.21.15.59:0/899151116 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330014440 con 0x7f234015a970 2024-03-06T07:33:09.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.974+0000 7f23455ae640 1 -- 172.21.15.59:0/899151116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f234006d830 con 0x7f234015a970 2024-03-06T07:33:09.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.975+0000 7f23455ae640 1 -- 172.21.15.59:0/899151116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f234004f330 con 0x7f234015a970 2024-03-06T07:33:09.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.976+0000 7f23377fe640 1 -- 172.21.15.59:0/899151116 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2330012040 con 0x7f234015a970 2024-03-06T07:33:09.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.976+0000 7f23377fe640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f232006b4d0 0x7f232006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:09.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.976+0000 7f23377fe640 1 -- 172.21.15.59:0/899151116 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f233008a7a0 con 0x7f234015a970 2024-03-06T07:33:09.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.977+0000 7f2337fff640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f232006b4d0 0x7f232006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:09.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.977+0000 7f2337fff640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f232006b4d0 0x7f232006d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f234006e2a0 tx=0x7f2328008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:09.982 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:09.981+0000 7f23377fe640 1 -- 172.21.15.59:0/899151116 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23401f3ec0 con 0x7f234015a970 2024-03-06T07:33:10.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.204+0000 7f23455ae640 1 -- 172.21.15.59:0/899151116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23400b4280 con 0x7f234015a970 2024-03-06T07:33:10.207 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.205+0000 7f23377fe640 1 -- 172.21.15.59:0/899151116 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f2330019030 con 0x7f234015a970 2024-03-06T07:33:10.207 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:10.207 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:10.207 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:10.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 -- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f232006b4d0 msgr2=0x7f232006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f232006b4d0 0x7f232006d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f234006e2a0 tx=0x7f2328008040 comp rx=0 tx=0).stop 2024-03-06T07:33:10.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 -- 172.21.15.59:0/899151116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f234015a970 msgr2=0x7f234006c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:10.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f234015a970 0x7f234006c820 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f2330011910 tx=0x7f2330011dd0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 -- 172.21.15.59:0/899151116 shutdown_connections 2024-03-06T07:33:10.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f232006b4d0 0x7f232006d990 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23401f7b00 0x7f234015c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f234006cd60 0x7f23401f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 --2- 172.21.15.59:0/899151116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f234015a970 0x7f234006c820 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:10.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 -- 172.21.15.59:0/899151116 >> 172.21.15.59:0/899151116 conn(0x7f234009a9f0 msgr2=0x7f23400caba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:10.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 -- 172.21.15.59:0/899151116 shutdown_connections 2024-03-06T07:33:10.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:10.207+0000 7f23455ae640 1 -- 172.21.15.59:0/899151116 wait complete. 2024-03-06T07:33:10.422 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:10.422 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:11.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:11.528 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.524+0000 7f7ada871640 1 Processor -- start 2024-03-06T07:33:11.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.524+0000 7f7ada871640 1 -- start start 2024-03-06T07:33:11.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.525+0000 7f7ada871640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ad406afb0 0x7f7ad406b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.525+0000 7f7ada871640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ad415bf40 0x7f7ad406b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.525+0000 7f7ada871640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ad406e380 0x7f7ad406e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.525+0000 7f7ada871640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad40cd680 con 0x7f7ad415bf40 2024-03-06T07:33:11.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.525+0000 7f7ada871640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad40afe40 con 0x7f7ad406e380 2024-03-06T07:33:11.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.525+0000 7f7ada871640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad40b0120 con 0x7f7ad406afb0 2024-03-06T07:33:11.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.525+0000 7f7ad37fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ad415bf40 0x7f7ad406b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.525+0000 7f7ad37fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7ad415bf40 0x7f7ad406b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45946/0 (socket says 172.21.15.59:45946) 2024-03-06T07:33:11.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.525+0000 7f7ad37fe640 1 -- 172.21.15.59:0/2959174344 learned_addr learned my addr 172.21.15.59:0/2959174344 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:11.529 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.526+0000 7f7ad3fff640 1 --2- 172.21.15.59:0/2959174344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ad406afb0 0x7f7ad406b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.526+0000 7f7ad8e0e640 1 --2- 172.21.15.59:0/2959174344 >> v2:172.21.15.164:3300/0 conn(0x7f7ad406e380 0x7f7ad406e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.526+0000 7f7ad37fe640 1 -- 172.21.15.59:0/2959174344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ad406afb0 msgr2=0x7f7ad406b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.526+0000 7f7ad37fe640 1 --2- 172.21.15.59:0/2959174344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ad406afb0 0x7f7ad406b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.526+0000 7f7ad37fe640 1 -- 172.21.15.59:0/2959174344 >> v2:172.21.15.164:3300/0 conn(0x7f7ad406e380 msgr2=0x7f7ad406e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.526+0000 7f7ad37fe640 1 --2- 172.21.15.59:0/2959174344 >> v2:172.21.15.164:3300/0 conn(0x7f7ad406e380 0x7f7ad406e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.526+0000 7f7ad37fe640 1 -- 172.21.15.59:0/2959174344 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad40cd310 con 0x7f7ad415bf40 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.526+0000 7f7ad3fff640 1 --2- 172.21.15.59:0/2959174344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ad406afb0 0x7f7ad406b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.526+0000 7f7ad8e0e640 1 --2- 172.21.15.59:0/2959174344 >> v2:172.21.15.164:3300/0 conn(0x7f7ad406e380 0x7f7ad406e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.526+0000 7f7ad37fe640 1 --2- 172.21.15.59:0/2959174344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad415bf40 0x7f7ad406b8d0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40cce40 tx=0x7f7ac802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bde5a2356679d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ad2ffd640 1 -- 172.21.15.59:0/2959174344 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8035020 con 0x7f7ad415bf40 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ad2ffd640 1 -- 172.21.15.59:0/2959174344 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac8031870 con 0x7f7ad415bf40 2024-03-06T07:33:11.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ad2ffd640 1 -- 172.21.15.59:0/2959174344 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8031c40 con 0x7f7ad415bf40 2024-03-06T07:33:11.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ada871640 1 -- 172.21.15.59:0/2959174344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad415bf40 msgr2=0x7f7ad406b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ada871640 1 --2- 172.21.15.59:0/2959174344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad415bf40 0x7f7ad406b8d0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40cce40 tx=0x7f7ac802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:11.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ada871640 1 -- 172.21.15.59:0/2959174344 shutdown_connections 2024-03-06T07:33:11.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ada871640 1 --2- 172.21.15.59:0/2959174344 >> v2:172.21.15.164:3300/0 conn(0x7f7ad406e380 0x7f7ad406e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ada871640 1 --2- 172.21.15.59:0/2959174344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad415bf40 0x7f7ad406b8d0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ada871640 1 --2- 172.21.15.59:0/2959174344 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ad406afb0 0x7f7ad406b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ada871640 1 -- 172.21.15.59:0/2959174344 >> 172.21.15.59:0/2959174344 conn(0x7f7ad40d0960 msgr2=0x7f7ad40d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ada871640 1 -- 172.21.15.59:0/2959174344 shutdown_connections 2024-03-06T07:33:11.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.527+0000 7f7ada871640 1 -- 172.21.15.59:0/2959174344 wait complete. 2024-03-06T07:33:11.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.529+0000 7f7ada871640 1 Processor -- start 2024-03-06T07:33:11.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.529+0000 7f7ada871640 1 -- start start 2024-03-06T07:33:11.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ada871640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad406afb0 0x7f7ad40aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ada871640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ad406e380 0x7f7ad415e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ada871640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ad415bf40 0x7f7ad41f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ada871640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad406eed0 con 0x7f7ad406afb0 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ada871640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad406ed50 con 0x7f7ad415bf40 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ada871640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad406f050 con 0x7f7ad406e380 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad406afb0 0x7f7ad40aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad406afb0 0x7f7ad40aeec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45948/0 (socket says 172.21.15.59:45948) 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad3fff640 1 -- 172.21.15.59:0/778634891 learned_addr learned my addr 172.21.15.59:0/778634891 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad37fe640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ad406e380 0x7f7ad415e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad8e0e640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ad415bf40 0x7f7ad41f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad3fff640 1 -- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ad406e380 msgr2=0x7f7ad415e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad3fff640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ad406e380 0x7f7ad415e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad3fff640 1 -- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ad415bf40 msgr2=0x7f7ad41f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad3fff640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ad415bf40 0x7f7ad41f7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad3fff640 1 -- 172.21.15.59:0/778634891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad40ade20 con 0x7f7ad406afb0 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad8e0e640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ad415bf40 0x7f7ad41f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad3fff640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad406afb0 0x7f7ad40aeec0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400c7c0 tx=0x7f7ac400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad0ff9640 1 -- 172.21.15.59:0/778634891 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4015070 con 0x7f7ad406afb0 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad0ff9640 1 -- 172.21.15.59:0/778634891 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac4007a30 con 0x7f7ad406afb0 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ad0ff9640 1 -- 172.21.15.59:0/778634891 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4007e00 con 0x7f7ad406afb0 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.531+0000 7f7ada871640 1 -- 172.21.15.59:0/778634891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad40ae100 con 0x7f7ad406afb0 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.532+0000 7f7ada871640 1 -- 172.21.15.59:0/778634891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad40ae6c0 con 0x7f7ad406afb0 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.533+0000 7f7ada871640 1 -- 172.21.15.59:0/778634891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad404f330 con 0x7f7ad406afb0 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.533+0000 7f7ad0ff9640 1 -- 172.21.15.59:0/778634891 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ac4018070 con 0x7f7ad406afb0 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.533+0000 7f7ad0ff9640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ac006b4d0 0x7f7ac006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.533+0000 7f7ad0ff9640 1 -- 172.21.15.59:0/778634891 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7ac408a050 con 0x7f7ad406afb0 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.534+0000 7f7ad37fe640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ac006b4d0 0x7f7ac006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.534+0000 7f7ad37fe640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ac006b4d0 0x7f7ac006d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8036000 tx=0x7f7ac803c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.538+0000 7f7ad0ff9640 1 -- 172.21.15.59:0/778634891 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ac40588f0 con 0x7f7ad406afb0 2024-03-06T07:33:11.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.866+0000 7f7ada871640 1 -- 172.21.15.59:0/778634891 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ad40cd0d0 con 0x7f7ad406afb0 2024-03-06T07:33:11.868 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.867+0000 7f7ad0ff9640 1 -- 172.21.15.59:0/778634891 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f7ac405c990 con 0x7f7ad406afb0 2024-03-06T07:33:11.869 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:11.869 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:11.869 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:11.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 -- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ac006b4d0 msgr2=0x7f7ac006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ac006b4d0 0x7f7ac006d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8036000 tx=0x7f7ac803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:11.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 -- 172.21.15.59:0/778634891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad406afb0 msgr2=0x7f7ad40aeec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad406afb0 0x7f7ad40aeec0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400c7c0 tx=0x7f7ac400cc80 comp rx=0 tx=0).stop 2024-03-06T07:33:11.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 -- 172.21.15.59:0/778634891 shutdown_connections 2024-03-06T07:33:11.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7ac006b4d0 0x7f7ac006d990 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ad415bf40 0x7f7ad41f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ad406e380 0x7f7ad415e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 --2- 172.21.15.59:0/778634891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ad406afb0 0x7f7ad40aeec0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 -- 172.21.15.59:0/778634891 >> 172.21.15.59:0/778634891 conn(0x7f7ad40d0960 msgr2=0x7f7ad40cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 -- 172.21.15.59:0/778634891 shutdown_connections 2024-03-06T07:33:11.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:11.869+0000 7f7ada871640 1 -- 172.21.15.59:0/778634891 wait complete. 2024-03-06T07:33:12.082 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:12.083 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:13.084 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:13.186 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.185+0000 7f2ee7afa640 1 Processor -- start 2024-03-06T07:33:13.187 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.185+0000 7f2ee7afa640 1 -- start start 2024-03-06T07:33:13.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee7afa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ee00d22b0 0x7f2ee00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee7afa640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2ee00b41f0 0x7f2ee00b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee7afa640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee00b3430 con 0x7f2ee015dfe0 2024-03-06T07:33:13.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee7afa640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee00d2c50 con 0x7f2ee00b41f0 2024-03-06T07:33:13.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee7afa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee00b31e0 con 0x7f2ee00d22b0 2024-03-06T07:33:13.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee6097640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2ee00b41f0 0x7f2ee00b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee6097640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2ee00b41f0 0x7f2ee00b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45956/0 (socket says 172.21.15.59:45956) 2024-03-06T07:33:13.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee6097640 1 -- 172.21.15.59:0/347550540 learned_addr learned my addr 172.21.15.59:0/347550540 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:13.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee5896640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2ee015dfe0 0x7f2ee00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34758/0 (socket says 172.21.15.59:34758) 2024-03-06T07:33:13.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee5095640 1 --2- 172.21.15.59:0/347550540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ee00d22b0 0x7f2ee00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee6097640 1 -- 172.21.15.59:0/347550540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ee00d22b0 msgr2=0x7f2ee00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee6097640 1 --2- 172.21.15.59:0/347550540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ee00d22b0 0x7f2ee00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee6097640 1 -- 172.21.15.59:0/347550540 >> v1:172.21.15.59:6789/0 conn(0x7f2ee015dfe0 legacy=0x7f2ee00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee6097640 1 -- 172.21.15.59:0/347550540 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee00b3710 con 0x7f2ee00b41f0 2024-03-06T07:33:13.189 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.186+0000 7f2ee4894640 1 -- 172.21.15.59:0/347550540 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190110392 0 0) 0x7f2ee00b3430 con 0x7f2ee015dfe0 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.187+0000 7f2ee6097640 1 --2- 172.21.15.59:0/347550540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ee00b41f0 0x7f2ee00b4650 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009e50 tx=0x7f2ed802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e089c64b6865ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.187+0000 7f2ee4894640 1 -- 172.21.15.59:0/347550540 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed802cb00 con 0x7f2ee00b41f0 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.187+0000 7f2ee4894640 1 -- 172.21.15.59:0/347550540 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed802cca0 con 0x7f2ee00b41f0 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.187+0000 7f2ee4894640 1 -- 172.21.15.59:0/347550540 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8034610 con 0x7f2ee00b41f0 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.187+0000 7f2ee7afa640 1 -- 172.21.15.59:0/347550540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ee00b41f0 msgr2=0x7f2ee00b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.187+0000 7f2ee7afa640 1 --2- 172.21.15.59:0/347550540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ee00b41f0 0x7f2ee00b4650 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009e50 tx=0x7f2ed802bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.188+0000 7f2ee7afa640 1 -- 172.21.15.59:0/347550540 shutdown_connections 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.188+0000 7f2ee7afa640 1 --2- 172.21.15.59:0/347550540 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ee00b41f0 0x7f2ee00b4650 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.188+0000 7f2ee7afa640 1 --2- 172.21.15.59:0/347550540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ee00d22b0 0x7f2ee00d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.188+0000 7f2ee7afa640 1 -- 172.21.15.59:0/347550540 >> 172.21.15.59:0/347550540 conn(0x7f2ee00d3cd0 msgr2=0x7f2ee00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.188+0000 7f2ee7afa640 1 -- 172.21.15.59:0/347550540 shutdown_connections 2024-03-06T07:33:13.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.188+0000 7f2ee7afa640 1 -- 172.21.15.59:0/347550540 wait complete. 2024-03-06T07:33:13.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.189+0000 7f2ee7afa640 1 Processor -- start 2024-03-06T07:33:13.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.189+0000 7f2ee7afa640 1 -- start start 2024-03-06T07:33:13.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.189+0000 7f2ee7afa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ee00b41f0 0x7f2ee00b96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.189+0000 7f2ee7afa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ee00d22b0 0x7f2ee00b7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.189+0000 7f2ee7afa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ee015dfe0 0x7f2ee014ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.189+0000 7f2ee7afa640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee00cfa00 con 0x7f2ee00d22b0 2024-03-06T07:33:13.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.189+0000 7f2ee7afa640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee00b3d60 con 0x7f2ee015dfe0 2024-03-06T07:33:13.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.189+0000 7f2ee7afa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee00d1c70 con 0x7f2ee00b41f0 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee5095640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ee00d22b0 0x7f2ee00b7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee5095640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ee00d22b0 0x7f2ee00b7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45962/0 (socket says 172.21.15.59:45962) 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee5095640 1 -- 172.21.15.59:0/380712942 learned_addr learned my addr 172.21.15.59:0/380712942 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee5896640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ee00b41f0 0x7f2ee00b96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee6097640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ee015dfe0 0x7f2ee014ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee5095640 1 -- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ee00b41f0 msgr2=0x7f2ee00b96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee5095640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ee00b41f0 0x7f2ee00b96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee5095640 1 -- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ee015dfe0 msgr2=0x7f2ee014ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee5095640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ee015dfe0 0x7f2ee014ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee5095640 1 -- 172.21.15.59:0/380712942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee014dc20 con 0x7f2ee00d22b0 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee5095640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ee00d22b0 0x7f2ee00b7e40 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000d800 tx=0x7f2ed000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.195 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ed67fc640 1 -- 172.21.15.59:0/380712942 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed000b6c0 con 0x7f2ee00d22b0 2024-03-06T07:33:13.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ed67fc640 1 -- 172.21.15.59:0/380712942 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed0019070 con 0x7f2ee00d22b0 2024-03-06T07:33:13.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee7afa640 1 -- 172.21.15.59:0/380712942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee01f33b0 con 0x7f2ee00d22b0 2024-03-06T07:33:13.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ed67fc640 1 -- 172.21.15.59:0/380712942 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed000ba80 con 0x7f2ee00d22b0 2024-03-06T07:33:13.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.190+0000 7f2ee7afa640 1 -- 172.21.15.59:0/380712942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee00b8640 con 0x7f2ee00d22b0 2024-03-06T07:33:13.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.191+0000 7f2ee7afa640 1 -- 172.21.15.59:0/380712942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee004f330 con 0x7f2ee00d22b0 2024-03-06T07:33:13.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.192+0000 7f2ed67fc640 1 -- 172.21.15.59:0/380712942 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ed000bc20 con 0x7f2ee00d22b0 2024-03-06T07:33:13.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.192+0000 7f2ed67fc640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ec406b4d0 0x7f2ec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.192+0000 7f2ed67fc640 1 -- 172.21.15.59:0/380712942 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2ed008a6d0 con 0x7f2ee00d22b0 2024-03-06T07:33:13.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.192+0000 7f2ee5896640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ec406b4d0 0x7f2ec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.193+0000 7f2ee5896640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ec406b4d0 0x7f2ec406d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2edc000f60 tx=0x7f2edc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.199+0000 7f2ed67fc640 1 -- 172.21.15.59:0/380712942 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ee014dc20 con 0x7f2ee00d22b0 2024-03-06T07:33:13.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.417+0000 7f2ee7afa640 1 -- 172.21.15.59:0/380712942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ee00b92c0 con 0x7f2ee00d22b0 2024-03-06T07:33:13.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.418+0000 7f2ed67fc640 1 -- 172.21.15.59:0/380712942 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f2ed0059530 con 0x7f2ee00d22b0 2024-03-06T07:33:13.420 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:13.420 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:13.420 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:13.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.419+0000 7f2ee7afa640 1 -- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ec406b4d0 msgr2=0x7f2ec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.419+0000 7f2ee7afa640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ec406b4d0 0x7f2ec406d990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2edc000f60 tx=0x7f2edc009040 comp rx=0 tx=0).stop 2024-03-06T07:33:13.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.419+0000 7f2ee7afa640 1 -- 172.21.15.59:0/380712942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ee00d22b0 msgr2=0x7f2ee00b7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.419+0000 7f2ee7afa640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ee00d22b0 0x7f2ee00b7e40 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000d800 tx=0x7f2ed000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.420+0000 7f2ee7afa640 1 -- 172.21.15.59:0/380712942 shutdown_connections 2024-03-06T07:33:13.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.420+0000 7f2ee7afa640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2ec406b4d0 0x7f2ec406d990 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.420+0000 7f2ee7afa640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ee015dfe0 0x7f2ee014ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.420+0000 7f2ee7afa640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ee00d22b0 0x7f2ee00b7e40 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.420+0000 7f2ee7afa640 1 --2- 172.21.15.59:0/380712942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ee00b41f0 0x7f2ee00b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.420+0000 7f2ee7afa640 1 -- 172.21.15.59:0/380712942 >> 172.21.15.59:0/380712942 conn(0x7f2ee00d3cd0 msgr2=0x7f2ee00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.420+0000 7f2ee7afa640 1 -- 172.21.15.59:0/380712942 shutdown_connections 2024-03-06T07:33:13.423 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:13.420+0000 7f2ee7afa640 1 -- 172.21.15.59:0/380712942 wait complete. 2024-03-06T07:33:13.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:13.634 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:14.635 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:14.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.734+0000 7f45cf577640 1 Processor -- start 2024-03-06T07:33:14.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.734+0000 7f45cf577640 1 -- start start 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.735+0000 7f45cf577640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f45c815fa10 0x7f45c80ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.735+0000 7f45cf577640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f45c80d16e0 0x7f45c80d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.735+0000 7f45cf577640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c80cebe0 con 0x7f45c815bdd0 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.735+0000 7f45cf577640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45c80ceec0 con 0x7f45c80d16e0 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.735+0000 7f45cf577640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45c80cf1a0 con 0x7f45c815fa10 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.735+0000 7f45cdb14640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f45c80d16e0 0x7f45c80d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.735+0000 7f45cdb14640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f45c80d16e0 0x7f45c80d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41612/0 (socket says 172.21.15.59:41612) 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.735+0000 7f45cdb14640 1 -- 172.21.15.59:0/1199142464 learned_addr learned my addr 172.21.15.59:0/1199142464 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.736+0000 7f45cdb14640 1 -- 172.21.15.59:0/1199142464 >> v2:172.21.15.164:3300/0 conn(0x7f45c815fa10 msgr2=0x7f45c80ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.736+0000 7f45ccb12640 1 --2- 172.21.15.59:0/1199142464 >> v2:172.21.15.164:3300/0 conn(0x7f45c815fa10 0x7f45c80ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.736+0000 7f45cdb14640 1 --2- 172.21.15.59:0/1199142464 >> v2:172.21.15.164:3300/0 conn(0x7f45c815fa10 0x7f45c80ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.736+0000 7f45cdb14640 1 -- 172.21.15.59:0/1199142464 >> v1:172.21.15.164:6789/0 conn(0x7f45c815bdd0 legacy=0x7f45c80c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:14.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.736+0000 7f45cdb14640 1 -- 172.21.15.59:0/1199142464 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c80cc8e0 con 0x7f45c80d16e0 2024-03-06T07:33:14.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.736+0000 7f45ccb12640 1 --2- 172.21.15.59:0/1199142464 >> v2:172.21.15.164:3300/0 conn(0x7f45c815fa10 0x7f45c80ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:14.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.736+0000 7f45cdb14640 1 --2- 172.21.15.59:0/1199142464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c80d16e0 0x7f45c80d1af0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f45c80c5520 tx=0x7f45c002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed3c327a615fdf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.737+0000 7f45bbfff640 1 -- 172.21.15.59:0/1199142464 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c002c9c0 con 0x7f45c80d16e0 2024-03-06T07:33:14.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.737+0000 7f45bbfff640 1 -- 172.21.15.59:0/1199142464 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c002cb60 con 0x7f45c80d16e0 2024-03-06T07:33:14.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.737+0000 7f45bbfff640 1 -- 172.21.15.59:0/1199142464 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c00343f0 con 0x7f45c80d16e0 2024-03-06T07:33:14.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.737+0000 7f45cf577640 1 -- 172.21.15.59:0/1199142464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c80d16e0 msgr2=0x7f45c80d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.737+0000 7f45cf577640 1 --2- 172.21.15.59:0/1199142464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c80d16e0 0x7f45c80d1af0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f45c80c5520 tx=0x7f45c002be20 comp rx=0 tx=0).stop 2024-03-06T07:33:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.737+0000 7f45cf577640 1 -- 172.21.15.59:0/1199142464 shutdown_connections 2024-03-06T07:33:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.737+0000 7f45cf577640 1 --2- 172.21.15.59:0/1199142464 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c80d16e0 0x7f45c80d1af0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.737+0000 7f45cf577640 1 --2- 172.21.15.59:0/1199142464 >> v2:172.21.15.164:3300/0 conn(0x7f45c815fa10 0x7f45c80ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.737+0000 7f45cf577640 1 -- 172.21.15.59:0/1199142464 >> 172.21.15.59:0/1199142464 conn(0x7f45c809a9f0 msgr2=0x7f45c809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.737+0000 7f45cf577640 1 -- 172.21.15.59:0/1199142464 shutdown_connections 2024-03-06T07:33:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.737+0000 7f45cf577640 1 -- 172.21.15.59:0/1199142464 wait complete. 2024-03-06T07:33:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.738+0000 7f45cf577640 1 Processor -- start 2024-03-06T07:33:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.738+0000 7f45cf577640 1 -- start start 2024-03-06T07:33:14.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.739+0000 7f45cf577640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45c80d16e0 0x7f45c806ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.739+0000 7f45cf577640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45c815bdd0 0x7f45c815dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.739+0000 7f45cf577640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c815fa10 0x7f45c8154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.739+0000 7f45cf577640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c80d2030 con 0x7f45c815fa10 2024-03-06T07:33:14.743 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.739+0000 7f45cf577640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c815c1b0 con 0x7f45c815bdd0 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.739+0000 7f45cf577640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c80ce7b0 con 0x7f45c80d16e0 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.739+0000 7f45cdb14640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c815fa10 0x7f45c8154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.739+0000 7f45cdb14640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c815fa10 0x7f45c8154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41626/0 (socket says 172.21.15.59:41626) 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.739+0000 7f45cdb14640 1 -- 172.21.15.59:0/2389850192 learned_addr learned my addr 172.21.15.59:0/2389850192 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45ccb12640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45c815bdd0 0x7f45c815dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45cdb14640 1 -- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45c80d16e0 msgr2=0x7f45c806ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45cdb14640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45c80d16e0 0x7f45c806ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45cdb14640 1 -- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45c815bdd0 msgr2=0x7f45c815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45cdb14640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45c815bdd0 0x7f45c815dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45cdb14640 1 -- 172.21.15.59:0/2389850192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c815e700 con 0x7f45c815fa10 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45cdb14640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c815fa10 0x7f45c8154bf0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f45c0036040 tx=0x7f45c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45b9ffb640 1 -- 172.21.15.59:0/2389850192 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0043020 con 0x7f45c815fa10 2024-03-06T07:33:14.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45b9ffb640 1 -- 172.21.15.59:0/2389850192 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c003e450 con 0x7f45c815fa10 2024-03-06T07:33:14.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45cf577640 1 -- 172.21.15.59:0/2389850192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c815e9e0 con 0x7f45c815fa10 2024-03-06T07:33:14.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45b9ffb640 1 -- 172.21.15.59:0/2389850192 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c003e820 con 0x7f45c815fa10 2024-03-06T07:33:14.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.740+0000 7f45cf577640 1 -- 172.21.15.59:0/2389850192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c815ecc0 con 0x7f45c815fa10 2024-03-06T07:33:14.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.741+0000 7f45cf577640 1 -- 172.21.15.59:0/2389850192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c80b4430 con 0x7f45c815fa10 2024-03-06T07:33:14.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.742+0000 7f45b9ffb640 1 -- 172.21.15.59:0/2389850192 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45c0046020 con 0x7f45c815fa10 2024-03-06T07:33:14.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.742+0000 7f45b9ffb640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45ac06b500 0x7f45ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:14.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.742+0000 7f45b9ffb640 1 -- 172.21.15.59:0/2389850192 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f45c00acb40 con 0x7f45c815fa10 2024-03-06T07:33:14.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.742+0000 7f45cd313640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45ac06b500 0x7f45ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:14.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.743+0000 7f45cd313640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45ac06b500 0x7f45ac06d9c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f45bc003cc0 tx=0x7f45bc0079b0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:14.748 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.746+0000 7f45b9ffb640 1 -- 172.21.15.59:0/2389850192 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45c80b4430 con 0x7f45c815fa10 2024-03-06T07:33:14.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.969+0000 7f45cf577640 1 -- 172.21.15.59:0/2389850192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45c804f330 con 0x7f45c815fa10 2024-03-06T07:33:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.970+0000 7f45b9ffb640 1 -- 172.21.15.59:0/2389850192 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f45c004d030 con 0x7f45c815fa10 2024-03-06T07:33:14.972 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:14.972 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:14.972 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 -- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45ac06b500 msgr2=0x7f45ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45ac06b500 0x7f45ac06d9c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f45bc003cc0 tx=0x7f45bc0079b0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 -- 172.21.15.59:0/2389850192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c815fa10 msgr2=0x7f45c8154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c815fa10 0x7f45c8154bf0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f45c0036040 tx=0x7f45c0004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 -- 172.21.15.59:0/2389850192 shutdown_connections 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f45ac06b500 0x7f45ac06d9c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45c815fa10 0x7f45c8154bf0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45c815bdd0 0x7f45c815dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 --2- 172.21.15.59:0/2389850192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45c80d16e0 0x7f45c806ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 -- 172.21.15.59:0/2389850192 >> 172.21.15.59:0/2389850192 conn(0x7f45c809a9f0 msgr2=0x7f45c806fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 -- 172.21.15.59:0/2389850192 shutdown_connections 2024-03-06T07:33:14.974 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:14.972+0000 7f45cf577640 1 -- 172.21.15.59:0/2389850192 wait complete. 2024-03-06T07:33:15.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:15.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:16.190 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:16.291 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.290+0000 7fc56435c640 1 Processor -- start 2024-03-06T07:33:16.292 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.290+0000 7fc56435c640 1 -- start start 2024-03-06T07:33:16.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc56435c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc55c0cfff0 0x7fc55c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc56435c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c161980 con 0x7fc55c15bb20 2024-03-06T07:33:16.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc56435c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c161c60 con 0x7fc55c0cc8b0 2024-03-06T07:33:16.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc56435c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc55c0d2500 con 0x7fc55c0cfff0 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5620f8640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc55c15bb20 0x7fc55c0d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41694/0 (socket says 172.21.15.59:41694) 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5620f8640 1 -- 172.21.15.59:0/1015389100 learned_addr learned my addr 172.21.15.59:0/1015389100 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5618f7640 1 --2- 172.21.15.59:0/1015389100 >> v2:172.21.15.164:3300/0 conn(0x7fc55c0cfff0 0x7fc55c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985436278 0 0) 0x7fc55c161980 con 0x7fc55c15bb20 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc548003520 con 0x7fc55c15bb20 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3529415013 0 0) 0x7fc548003520 con 0x7fc55c15bb20 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 >> v2:172.21.15.164:3300/0 conn(0x7fc55c0cfff0 msgr2=0x7fc55c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 --2- 172.21.15.59:0/1015389100 >> v2:172.21.15.164:3300/0 conn(0x7fc55c0cfff0 0x7fc55c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 >> v1:172.21.15.164:6789/0 conn(0x7fc55c0cc8b0 legacy=0x7fc55c0ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c161f40 con 0x7fc55c15bb20 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fc558001670 con 0x7fc55c15bb20 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 >> v1:172.21.15.59:6789/0 conn(0x7fc55c15bb20 legacy=0x7fc55c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc548003a00 0x7fc548003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 0x7fc548004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5480051c0 0x7fc54800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c0d2500 con 0x7fc548004800 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54800c0c0 con 0x7fc548003a00 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54800c2c0 con 0x7fc5480051c0 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.291+0000 7fc5618f7640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 0x7fc548004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.292+0000 7fc5620f8640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc548003a00 0x7fc548003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.292+0000 7fc5628f9640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5480051c0 0x7fc54800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.292+0000 7fc5618f7640 1 -- 172.21.15.59:0/1015389100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5480051c0 msgr2=0x7fc54800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.292+0000 7fc5618f7640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5480051c0 0x7fc54800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.292+0000 7fc5618f7640 1 -- 172.21.15.59:0/1015389100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc548003a00 msgr2=0x7fc548003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.292+0000 7fc5618f7640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc548003a00 0x7fc548003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.292+0000 7fc5618f7640 1 -- 172.21.15.59:0/1015389100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc548003520 con 0x7fc548004800 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.292+0000 7fc5628f9640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5480051c0 0x7fc54800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.292+0000 7fc5628f9640 1 -- 172.21.15.59:0/1015389100 reap_dead start 2024-03-06T07:33:16.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.292+0000 7fc5618f7640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 0x7fc548004c80 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009ac0 tx=0x7fc548005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e105f80119cca53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.293+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c039070 con 0x7fc548004800 2024-03-06T07:33:16.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.293+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c02c6b0 con 0x7fc548004800 2024-03-06T07:33:16.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.293+0000 7fc5610f6640 1 -- 172.21.15.59:0/1015389100 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c009690 con 0x7fc548004800 2024-03-06T07:33:16.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.293+0000 7fc56435c640 1 -- 172.21.15.59:0/1015389100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 msgr2=0x7fc548004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.293+0000 7fc56435c640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 0x7fc548004c80 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009ac0 tx=0x7fc548005830 comp rx=0 tx=0).stop 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.293+0000 7fc56435c640 1 -- 172.21.15.59:0/1015389100 shutdown_connections 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.293+0000 7fc56435c640 1 --2- 172.21.15.59:0/1015389100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 0x7fc548004c80 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.293+0000 7fc56435c640 1 -- 172.21.15.59:0/1015389100 >> 172.21.15.59:0/1015389100 conn(0x7fc55c09a9f0 msgr2=0x7fc55c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.293+0000 7fc56435c640 1 -- 172.21.15.59:0/1015389100 shutdown_connections 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.293+0000 7fc56435c640 1 -- 172.21.15.59:0/1015389100 wait complete. 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.294+0000 7fc56435c640 1 Processor -- start 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.294+0000 7fc56435c640 1 -- start start 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.295+0000 7fc56435c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 0x7fc55c15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.295+0000 7fc56435c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc55c06ff40 0x7fc55c0703c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.295+0000 7fc56435c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc55c0c6f40 0x7fc55c0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.295+0000 7fc56435c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c0d1980 con 0x7fc548004800 2024-03-06T07:33:16.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.295+0000 7fc56435c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c162500 con 0x7fc55c0c6f40 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.295+0000 7fc56435c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c0d1b00 con 0x7fc55c06ff40 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.295+0000 7fc5620f8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 0x7fc55c15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.295+0000 7fc5620f8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 0x7fc55c15aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41644/0 (socket says 172.21.15.59:41644) 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.295+0000 7fc5620f8640 1 -- 172.21.15.59:0/3800160353 learned_addr learned my addr 172.21.15.59:0/3800160353 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.295+0000 7fc5618f7640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc55c06ff40 0x7fc55c0703c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.295+0000 7fc5628f9640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc55c0c6f40 0x7fc55c0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.296+0000 7fc5620f8640 1 -- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc55c06ff40 msgr2=0x7fc55c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.296+0000 7fc5620f8640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc55c06ff40 0x7fc55c0703c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.296+0000 7fc5620f8640 1 -- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc55c0c6f40 msgr2=0x7fc55c0c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.296+0000 7fc5620f8640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc55c0c6f40 0x7fc55c0c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.296+0000 7fc5620f8640 1 -- 172.21.15.59:0/3800160353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c0c5690 con 0x7fc548004800 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.296+0000 7fc5620f8640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 0x7fc55c15aba0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fc5580072c0 tx=0x7fc5580076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.296+0000 7fc552ffd640 1 -- 172.21.15.59:0/3800160353 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558001430 con 0x7fc548004800 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.296+0000 7fc552ffd640 1 -- 172.21.15.59:0/3800160353 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc558018070 con 0x7fc548004800 2024-03-06T07:33:16.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.296+0000 7fc56435c640 1 -- 172.21.15.59:0/3800160353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55c0c5970 con 0x7fc548004800 2024-03-06T07:33:16.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.296+0000 7fc552ffd640 1 -- 172.21.15.59:0/3800160353 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55800fb20 con 0x7fc548004800 2024-03-06T07:33:16.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.296+0000 7fc56435c640 1 -- 172.21.15.59:0/3800160353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c0c5f30 con 0x7fc548004800 2024-03-06T07:33:16.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.297+0000 7fc56435c640 1 -- 172.21.15.59:0/3800160353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55c04f330 con 0x7fc548004800 2024-03-06T07:33:16.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.297+0000 7fc552ffd640 1 -- 172.21.15.59:0/3800160353 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc55800fcc0 con 0x7fc548004800 2024-03-06T07:33:16.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.298+0000 7fc552ffd640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc54006b4d0 0x7fc54006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.298+0000 7fc552ffd640 1 -- 172.21.15.59:0/3800160353 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc55808a100 con 0x7fc548004800 2024-03-06T07:33:16.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.298+0000 7fc5618f7640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc54006b4d0 0x7fc54006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.299+0000 7fc5618f7640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc54006b4d0 0x7fc54006d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc54c036000 tx=0x7fc54c030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.302+0000 7fc552ffd640 1 -- 172.21.15.59:0/3800160353 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc558058ff0 con 0x7fc548004800 2024-03-06T07:33:16.522 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.520+0000 7fc56435c640 1 -- 172.21.15.59:0/3800160353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc55c0b1820 con 0x7fc548004800 2024-03-06T07:33:16.523 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.521+0000 7fc552ffd640 1 -- 172.21.15.59:0/3800160353 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fc5580588f0 con 0x7fc548004800 2024-03-06T07:33:16.523 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:16.523 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:16.523 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 -- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc54006b4d0 msgr2=0x7fc54006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc54006b4d0 0x7fc54006d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc54c036000 tx=0x7fc54c030000 comp rx=0 tx=0).stop 2024-03-06T07:33:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 -- 172.21.15.59:0/3800160353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 msgr2=0x7fc55c15aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 0x7fc55c15aba0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fc5580072c0 tx=0x7fc5580076c0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 -- 172.21.15.59:0/3800160353 shutdown_connections 2024-03-06T07:33:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc54006b4d0 0x7fc54006d990 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc55c0c6f40 0x7fc55c0c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc55c06ff40 0x7fc55c0703c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 --2- 172.21.15.59:0/3800160353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc548004800 0x7fc55c15aba0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 -- 172.21.15.59:0/3800160353 >> 172.21.15.59:0/3800160353 conn(0x7fc55c09a9f0 msgr2=0x7fc55c15e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 -- 172.21.15.59:0/3800160353 shutdown_connections 2024-03-06T07:33:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:16.523+0000 7fc56435c640 1 -- 172.21.15.59:0/3800160353 wait complete. 2024-03-06T07:33:16.737 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:16.738 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:17.739 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:17.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.838+0000 7f3183ab5640 1 Processor -- start 2024-03-06T07:33:17.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.839+0000 7f3183ab5640 1 -- start start 2024-03-06T07:33:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.839+0000 7f3183ab5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0d22b0 0x7f317c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.839+0000 7f3183ab5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f317c0b41f0 0x7f317c0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.839+0000 7f3183ab5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317c0b3430 con 0x7f317c15dfe0 2024-03-06T07:33:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.839+0000 7f3183ab5640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f317c0d2c50 con 0x7f317c0b41f0 2024-03-06T07:33:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.839+0000 7f3183ab5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317c0b31e0 con 0x7f317c0d22b0 2024-03-06T07:33:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.840+0000 7f3182052640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f317c0b41f0 0x7f317c0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.840+0000 7f3182052640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f317c0b41f0 0x7f317c0b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41650/0 (socket says 172.21.15.59:41650) 2024-03-06T07:33:17.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.840+0000 7f3182052640 1 -- 172.21.15.59:0/4268970405 learned_addr learned my addr 172.21.15.59:0/4268970405 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:17.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.840+0000 7f3181050640 1 --2- 172.21.15.59:0/4268970405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0d22b0 0x7f317c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.840+0000 7f3182052640 1 -- 172.21.15.59:0/4268970405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0d22b0 msgr2=0x7f317c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:17.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.840+0000 7f3182052640 1 --2- 172.21.15.59:0/4268970405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0d22b0 0x7f317c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.840+0000 7f3182052640 1 -- 172.21.15.59:0/4268970405 >> v1:172.21.15.164:6789/0 conn(0x7f317c15dfe0 legacy=0x7f317c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:17.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.840+0000 7f3182052640 1 -- 172.21.15.59:0/4268970405 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317c0b3710 con 0x7f317c0b41f0 2024-03-06T07:33:17.843 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.840+0000 7f3181050640 1 --2- 172.21.15.59:0/4268970405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0d22b0 0x7f317c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:17.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.841+0000 7f3182052640 1 --2- 172.21.15.59:0/4268970405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f317c0b41f0 0x7f317c0b4650 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f3174009e50 tx=0x7f317402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfba1202c304236f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.841+0000 7f318084f640 1 -- 172.21.15.59:0/4268970405 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317402ca70 con 0x7f317c0b41f0 2024-03-06T07:33:17.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.841+0000 7f318084f640 1 -- 172.21.15.59:0/4268970405 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317402cc10 con 0x7f317c0b41f0 2024-03-06T07:33:17.844 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.841+0000 7f318084f640 1 -- 172.21.15.59:0/4268970405 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31740343f0 con 0x7f317c0b41f0 2024-03-06T07:33:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.841+0000 7f3183ab5640 1 -- 172.21.15.59:0/4268970405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f317c0b41f0 msgr2=0x7f317c0b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.841+0000 7f3183ab5640 1 --2- 172.21.15.59:0/4268970405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f317c0b41f0 0x7f317c0b4650 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f3174009e50 tx=0x7f317402bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.841+0000 7f3183ab5640 1 -- 172.21.15.59:0/4268970405 shutdown_connections 2024-03-06T07:33:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.841+0000 7f3183ab5640 1 --2- 172.21.15.59:0/4268970405 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f317c0b41f0 0x7f317c0b4650 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.841+0000 7f3183ab5640 1 --2- 172.21.15.59:0/4268970405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0d22b0 0x7f317c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.841+0000 7f3183ab5640 1 -- 172.21.15.59:0/4268970405 >> 172.21.15.59:0/4268970405 conn(0x7f317c0d3cd0 msgr2=0x7f317c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.842+0000 7f3183ab5640 1 -- 172.21.15.59:0/4268970405 shutdown_connections 2024-03-06T07:33:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.842+0000 7f3183ab5640 1 -- 172.21.15.59:0/4268970405 wait complete. 2024-03-06T07:33:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.843+0000 7f3183ab5640 1 Processor -- start 2024-03-06T07:33:17.845 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.843+0000 7f3183ab5640 1 -- start start 2024-03-06T07:33:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.843+0000 7f3183ab5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0b41f0 0x7f317c15f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.843+0000 7f3183ab5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f317c0d22b0 0x7f317c06ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.843+0000 7f3183ab5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f317c15dfe0 0x7f317c14ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.843+0000 7f3183ab5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317c0cfa00 con 0x7f317c15dfe0 2024-03-06T07:33:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.843+0000 7f3183ab5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317c0b3d60 con 0x7f317c0d22b0 2024-03-06T07:33:17.847 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.843+0000 7f3183ab5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317c0d1c70 con 0x7f317c0b41f0 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.843+0000 7f3182052640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f317c15dfe0 0x7f317c14ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3182052640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f317c15dfe0 0x7f317c14ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41664/0 (socket says 172.21.15.59:41664) 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3182052640 1 -- 172.21.15.59:0/571754442 learned_addr learned my addr 172.21.15.59:0/571754442 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3182052640 1 -- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0b41f0 msgr2=0x7f317c15f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3181851640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0b41f0 0x7f317c15f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3181050640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f317c0d22b0 0x7f317c06ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3182052640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0b41f0 0x7f317c15f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3182052640 1 -- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f317c0d22b0 msgr2=0x7f317c06ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3182052640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f317c0d22b0 0x7f317c06ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3182052640 1 -- 172.21.15.59:0/571754442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317c161680 con 0x7f317c15dfe0 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3181050640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f317c0d22b0 0x7f317c06ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3181851640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0b41f0 0x7f317c15f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:17.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3182052640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f317c15dfe0 0x7f317c14ed10 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f3174009e50 tx=0x7f3174004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f31727fc640 1 -- 172.21.15.59:0/571754442 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3174043020 con 0x7f317c15dfe0 2024-03-06T07:33:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f31727fc640 1 -- 172.21.15.59:0/571754442 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317403e450 con 0x7f317c15dfe0 2024-03-06T07:33:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3183ab5640 1 -- 172.21.15.59:0/571754442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317c1f71f0 con 0x7f317c15dfe0 2024-03-06T07:33:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f31727fc640 1 -- 172.21.15.59:0/571754442 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317403e820 con 0x7f317c15dfe0 2024-03-06T07:33:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.844+0000 7f3183ab5640 1 -- 172.21.15.59:0/571754442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317c1f77b0 con 0x7f317c15dfe0 2024-03-06T07:33:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.845+0000 7f3183ab5640 1 -- 172.21.15.59:0/571754442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f317c04f330 con 0x7f317c15dfe0 2024-03-06T07:33:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.846+0000 7f31727fc640 1 -- 172.21.15.59:0/571754442 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3174046020 con 0x7f317c15dfe0 2024-03-06T07:33:17.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.846+0000 7f31727fc640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f316006b4d0 0x7f316006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:17.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.846+0000 7f31727fc640 1 -- 172.21.15.59:0/571754442 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f31740acba0 con 0x7f317c15dfe0 2024-03-06T07:33:17.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.846+0000 7f3181851640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f316006b4d0 0x7f316006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:17.850 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.847+0000 7f3181851640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f316006b4d0 0x7f316006d990 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f3178003eb0 tx=0x7f3178009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:17.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:17.850+0000 7f31727fc640 1 -- 172.21.15.59:0/571754442 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f317404d030 con 0x7f317c15dfe0 2024-03-06T07:33:18.068 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.066+0000 7f3183ab5640 1 -- 172.21.15.59:0/571754442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f317c0747a0 con 0x7f317c15dfe0 2024-03-06T07:33:18.069 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.067+0000 7f31727fc640 1 -- 172.21.15.59:0/571754442 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f317407ba90 con 0x7f317c15dfe0 2024-03-06T07:33:18.069 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:18.069 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:18.069 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 -- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f316006b4d0 msgr2=0x7f316006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f316006b4d0 0x7f316006d990 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f3178003eb0 tx=0x7f3178009040 comp rx=0 tx=0).stop 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 -- 172.21.15.59:0/571754442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f317c15dfe0 msgr2=0x7f317c14ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f317c15dfe0 0x7f317c14ed10 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f3174009e50 tx=0x7f3174004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 -- 172.21.15.59:0/571754442 shutdown_connections 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f316006b4d0 0x7f316006d990 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f317c15dfe0 0x7f317c14ed10 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f317c0d22b0 0x7f317c06ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 --2- 172.21.15.59:0/571754442 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f317c0b41f0 0x7f317c15f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 -- 172.21.15.59:0/571754442 >> 172.21.15.59:0/571754442 conn(0x7f317c0d3cd0 msgr2=0x7f317c0d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 -- 172.21.15.59:0/571754442 shutdown_connections 2024-03-06T07:33:18.072 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:18.069+0000 7f3183ab5640 1 -- 172.21.15.59:0/571754442 wait complete. 2024-03-06T07:33:18.283 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:18.284 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:19.284 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:19.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.384+0000 7f7d7de4e640 1 Processor -- start 2024-03-06T07:33:19.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.384+0000 7f7d7de4e640 1 -- start start 2024-03-06T07:33:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.385+0000 7f7d7de4e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7d7815bdd0 0x7f7d780c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.385+0000 7f7d7de4e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7d7815fa10 0x7f7d780ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.385+0000 7f7d7de4e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d780cebe0 con 0x7f7d780d16e0 2024-03-06T07:33:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.385+0000 7f7d7de4e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d780ceec0 con 0x7f7d7815fa10 2024-03-06T07:33:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.385+0000 7f7d7de4e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d780cf1a0 con 0x7f7d7815bdd0 2024-03-06T07:33:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.385+0000 7f7d76ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7d7815fa10 0x7f7d780ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.385+0000 7f7d76ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7d7815fa10 0x7f7d780ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41670/0 (socket says 172.21.15.59:41670) 2024-03-06T07:33:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.385+0000 7f7d76ffd640 1 -- 172.21.15.59:0/2172995627 learned_addr learned my addr 172.21.15.59:0/2172995627 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.385+0000 7f7d77fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7d780d16e0 0x7f7d780d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41696/0 (socket says 172.21.15.59:41696) 2024-03-06T07:33:19.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.386+0000 7f7d76ffd640 1 -- 172.21.15.59:0/2172995627 >> v2:172.21.15.164:3300/0 conn(0x7f7d7815bdd0 msgr2=0x7f7d780c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:19.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.386+0000 7f7d76ffd640 1 --2- 172.21.15.59:0/2172995627 >> v2:172.21.15.164:3300/0 conn(0x7f7d7815bdd0 0x7f7d780c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.386+0000 7f7d76ffd640 1 -- 172.21.15.59:0/2172995627 >> v1:172.21.15.59:6789/0 conn(0x7f7d780d16e0 legacy=0x7f7d780d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.386+0000 7f7d76ffd640 1 -- 172.21.15.59:0/2172995627 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d780cc8e0 con 0x7f7d7815fa10 2024-03-06T07:33:19.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.386+0000 7f7d767fc640 1 -- 172.21.15.59:0/2172995627 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296869607 0 0) 0x7f7d780cebe0 con 0x7f7d780d16e0 2024-03-06T07:33:19.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.386+0000 7f7d76ffd640 1 --2- 172.21.15.59:0/2172995627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d7815fa10 0x7f7d780ce270 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f7d7806e3c0 tx=0x7f7d6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1be8a05e1f339b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.387+0000 7f7d767fc640 1 -- 172.21.15.59:0/2172995627 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68035020 con 0x7f7d7815fa10 2024-03-06T07:33:19.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.387+0000 7f7d767fc640 1 -- 172.21.15.59:0/2172995627 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d68031870 con 0x7f7d7815fa10 2024-03-06T07:33:19.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.387+0000 7f7d767fc640 1 -- 172.21.15.59:0/2172995627 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68031c40 con 0x7f7d7815fa10 2024-03-06T07:33:19.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.387+0000 7f7d7de4e640 1 -- 172.21.15.59:0/2172995627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d7815fa10 msgr2=0x7f7d780ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.387+0000 7f7d7de4e640 1 --2- 172.21.15.59:0/2172995627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d7815fa10 0x7f7d780ce270 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f7d7806e3c0 tx=0x7f7d6802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:19.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.387+0000 7f7d7de4e640 1 -- 172.21.15.59:0/2172995627 shutdown_connections 2024-03-06T07:33:19.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.387+0000 7f7d7de4e640 1 --2- 172.21.15.59:0/2172995627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d7815fa10 0x7f7d780ce270 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.387+0000 7f7d7de4e640 1 --2- 172.21.15.59:0/2172995627 >> v2:172.21.15.164:3300/0 conn(0x7f7d7815bdd0 0x7f7d780c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.387+0000 7f7d7de4e640 1 -- 172.21.15.59:0/2172995627 >> 172.21.15.59:0/2172995627 conn(0x7f7d7809a9f0 msgr2=0x7f7d7809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.387+0000 7f7d7de4e640 1 -- 172.21.15.59:0/2172995627 shutdown_connections 2024-03-06T07:33:19.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.387+0000 7f7d7de4e640 1 -- 172.21.15.59:0/2172995627 wait complete. 2024-03-06T07:33:19.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.389+0000 7f7d7de4e640 1 Processor -- start 2024-03-06T07:33:19.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.389+0000 7f7d7de4e640 1 -- start start 2024-03-06T07:33:19.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.389+0000 7f7d7de4e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d780d16e0 0x7f7d7815c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.389+0000 7f7d7de4e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d7815bdd0 0x7f7d780b6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.389+0000 7f7d7de4e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d7815fa10 0x7f7d7814ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.389+0000 7f7d7de4e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7815c1b0 con 0x7f7d780d16e0 2024-03-06T07:33:19.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.389+0000 7f7d7de4e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780cc5d0 con 0x7f7d7815bdd0 2024-03-06T07:33:19.393 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.389+0000 7f7d7de4e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d780ce7b0 con 0x7f7d7815fa10 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.389+0000 7f7d777fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d780d16e0 0x7f7d7815c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.389+0000 7f7d777fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d780d16e0 0x7f7d7815c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41686/0 (socket says 172.21.15.59:41686) 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.389+0000 7f7d777fe640 1 -- 172.21.15.59:0/621178409 learned_addr learned my addr 172.21.15.59:0/621178409 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d76ffd640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d7815bdd0 0x7f7d780b6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d77fff640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d7815fa10 0x7f7d7814ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d777fe640 1 -- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d7815fa10 msgr2=0x7f7d7814ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d777fe640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d7815fa10 0x7f7d7814ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d777fe640 1 -- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d7815bdd0 msgr2=0x7f7d780b6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d777fe640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d7815bdd0 0x7f7d780b6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d777fe640 1 -- 172.21.15.59:0/621178409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d780c9440 con 0x7f7d780d16e0 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d777fe640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d780d16e0 0x7f7d7815c5a0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000b4a0 tx=0x7f7d6000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d6ffff640 1 -- 172.21.15.59:0/621178409 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60014070 con 0x7f7d780d16e0 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d6ffff640 1 -- 172.21.15.59:0/621178409 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d60010610 con 0x7f7d780d16e0 2024-03-06T07:33:19.394 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d7de4e640 1 -- 172.21.15.59:0/621178409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d780c9720 con 0x7f7d780d16e0 2024-03-06T07:33:19.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d6ffff640 1 -- 172.21.15.59:0/621178409 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60010a00 con 0x7f7d780d16e0 2024-03-06T07:33:19.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.390+0000 7f7d7de4e640 1 -- 172.21.15.59:0/621178409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d780b5990 con 0x7f7d780d16e0 2024-03-06T07:33:19.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.391+0000 7f7d7de4e640 1 -- 172.21.15.59:0/621178409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7804f330 con 0x7f7d780d16e0 2024-03-06T07:33:19.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.392+0000 7f7d6ffff640 1 -- 172.21.15.59:0/621178409 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d60017070 con 0x7f7d780d16e0 2024-03-06T07:33:19.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.392+0000 7f7d6ffff640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d5806b4d0 0x7f7d5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:19.395 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.392+0000 7f7d6ffff640 1 -- 172.21.15.59:0/621178409 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7d60089bd0 con 0x7f7d780d16e0 2024-03-06T07:33:19.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.392+0000 7f7d76ffd640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d5806b4d0 0x7f7d5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:19.396 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.393+0000 7f7d76ffd640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d5806b4d0 0x7f7d5806d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f7d68036000 tx=0x7f7d68008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:19.398 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.396+0000 7f7d6ffff640 1 -- 172.21.15.59:0/621178409 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d60058ac0 con 0x7f7d780d16e0 2024-03-06T07:33:19.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.616+0000 7f7d7de4e640 1 -- 172.21.15.59:0/621178409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d780c5210 con 0x7f7d780d16e0 2024-03-06T07:33:19.618 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.616+0000 7f7d6ffff640 1 -- 172.21.15.59:0/621178409 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f7d600583c0 con 0x7f7d780d16e0 2024-03-06T07:33:19.619 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:19.619 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:19.619 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 -- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d5806b4d0 msgr2=0x7f7d5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d5806b4d0 0x7f7d5806d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f7d68036000 tx=0x7f7d68008030 comp rx=0 tx=0).stop 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 -- 172.21.15.59:0/621178409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d780d16e0 msgr2=0x7f7d7815c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d780d16e0 0x7f7d7815c5a0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000b4a0 tx=0x7f7d6000b960 comp rx=0 tx=0).stop 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 -- 172.21.15.59:0/621178409 shutdown_connections 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d5806b4d0 0x7f7d5806d990 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d7815fa10 0x7f7d7814ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d7815bdd0 0x7f7d780b6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 --2- 172.21.15.59:0/621178409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d780d16e0 0x7f7d7815c5a0 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 -- 172.21.15.59:0/621178409 >> 172.21.15.59:0/621178409 conn(0x7f7d7809a9f0 msgr2=0x7f7d7806f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 -- 172.21.15.59:0/621178409 shutdown_connections 2024-03-06T07:33:19.621 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:19.618+0000 7f7d7de4e640 1 -- 172.21.15.59:0/621178409 wait complete. 2024-03-06T07:33:19.832 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:19.832 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:20.833 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:20.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.933+0000 7f8b99d3e640 1 Processor -- start 2024-03-06T07:33:20.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.933+0000 7f8b99d3e640 1 -- start start 2024-03-06T07:33:20.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.934+0000 7f8b99d3e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8b940adc90 0x7f8b940acf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.934+0000 7f8b99d3e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b940710c0 con 0x7f8b940ad490 2024-03-06T07:33:20.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.934+0000 7f8b99d3e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b940ae230 con 0x7f8b9415df30 2024-03-06T07:33:20.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.934+0000 7f8b99d3e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b940ae510 con 0x7f8b940adc90 2024-03-06T07:33:20.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.934+0000 7f8b92ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8b940adc90 0x7f8b940acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.934+0000 7f8b93fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8b940ad490 0x7f8b9406dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:41700/0 (socket says 172.21.15.59:41700) 2024-03-06T07:33:20.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.934+0000 7f8b93fff640 1 -- 172.21.15.59:0/3329942387 learned_addr learned my addr 172.21.15.59:0/3329942387 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:20.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.934+0000 7f8b92ffd640 1 -- 172.21.15.59:0/3329942387 >> v1:172.21.15.164:6789/0 conn(0x7f8b9415df30 legacy=0x7f8b940cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.934+0000 7f8b92ffd640 1 -- 172.21.15.59:0/3329942387 >> v1:172.21.15.59:6789/0 conn(0x7f8b940ad490 legacy=0x7f8b9406dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.935+0000 7f8b927fc640 1 -- 172.21.15.59:0/3329942387 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147183254 0 0) 0x7f8b940710c0 con 0x7f8b940ad490 2024-03-06T07:33:20.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.935+0000 7f8b92ffd640 1 -- 172.21.15.59:0/3329942387 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9406d0d0 con 0x7f8b940adc90 2024-03-06T07:33:20.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.935+0000 7f8b92ffd640 1 --2- 172.21.15.59:0/3329942387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b940adc90 0x7f8b940acf50 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f8b880097a0 tx=0x7f8b8802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6327f037197942da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.935+0000 7f8b927fc640 1 -- 172.21.15.59:0/3329942387 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88035020 con 0x7f8b940adc90 2024-03-06T07:33:20.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.935+0000 7f8b927fc640 1 -- 172.21.15.59:0/3329942387 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b88031830 con 0x7f8b940adc90 2024-03-06T07:33:20.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.935+0000 7f8b927fc640 1 -- 172.21.15.59:0/3329942387 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88031c00 con 0x7f8b940adc90 2024-03-06T07:33:20.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.936+0000 7f8b99d3e640 1 -- 172.21.15.59:0/3329942387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b940adc90 msgr2=0x7f8b940acf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.936+0000 7f8b99d3e640 1 --2- 172.21.15.59:0/3329942387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b940adc90 0x7f8b940acf50 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f8b880097a0 tx=0x7f8b8802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.936+0000 7f8b99d3e640 1 -- 172.21.15.59:0/3329942387 shutdown_connections 2024-03-06T07:33:20.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.936+0000 7f8b99d3e640 1 --2- 172.21.15.59:0/3329942387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b940adc90 0x7f8b940acf50 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.936+0000 7f8b99d3e640 1 -- 172.21.15.59:0/3329942387 >> 172.21.15.59:0/3329942387 conn(0x7f8b940d00e0 msgr2=0x7f8b940d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.936+0000 7f8b99d3e640 1 -- 172.21.15.59:0/3329942387 shutdown_connections 2024-03-06T07:33:20.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.936+0000 7f8b99d3e640 1 -- 172.21.15.59:0/3329942387 wait complete. 2024-03-06T07:33:20.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.938+0000 7f8b99d3e640 1 Processor -- start 2024-03-06T07:33:20.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.938+0000 7f8b99d3e640 1 -- start start 2024-03-06T07:33:20.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.939+0000 7f8b99d3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b940ad490 0x7f8b9409b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.939+0000 7f8b99d3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b940adc90 0x7f8b9415ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.939+0000 7f8b99d3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b9415df30 0x7f8b9415cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.939+0000 7f8b99d3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9415e310 con 0x7f8b940ad490 2024-03-06T07:33:20.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.939+0000 7f8b99d3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9406d690 con 0x7f8b9415df30 2024-03-06T07:33:20.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.939+0000 7f8b99d3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9415e490 con 0x7f8b940adc90 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.939+0000 7f8b937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b940ad490 0x7f8b9409b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.939+0000 7f8b937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b940ad490 0x7f8b9409b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41710/0 (socket says 172.21.15.59:41710) 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.939+0000 7f8b937fe640 1 -- 172.21.15.59:0/4146352355 learned_addr learned my addr 172.21.15.59:0/4146352355 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b93fff640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b9415df30 0x7f8b9415cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b92ffd640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b940adc90 0x7f8b9415ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b937fe640 1 -- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b940adc90 msgr2=0x7f8b9415ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b937fe640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b940adc90 0x7f8b9415ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b937fe640 1 -- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b9415df30 msgr2=0x7f8b9415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b937fe640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b9415df30 0x7f8b9415cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b937fe640 1 -- 172.21.15.59:0/4146352355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9409b8f0 con 0x7f8b940ad490 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b937fe640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b940ad490 0x7f8b9409b3b0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400c430 tx=0x7f8b8400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b7bfff640 1 -- 172.21.15.59:0/4146352355 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b84011790 con 0x7f8b940ad490 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b7bfff640 1 -- 172.21.15.59:0/4146352355 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b84017070 con 0x7f8b940ad490 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b99d3e640 1 -- 172.21.15.59:0/4146352355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9409bb70 con 0x7f8b940ad490 2024-03-06T07:33:20.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b7bfff640 1 -- 172.21.15.59:0/4146352355 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b84011b70 con 0x7f8b940ad490 2024-03-06T07:33:20.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.940+0000 7f8b99d3e640 1 -- 172.21.15.59:0/4146352355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9409c130 con 0x7f8b940ad490 2024-03-06T07:33:20.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.941+0000 7f8b99d3e640 1 -- 172.21.15.59:0/4146352355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9404f330 con 0x7f8b940ad490 2024-03-06T07:33:20.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.942+0000 7f8b7bfff640 1 -- 172.21.15.59:0/4146352355 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b84011d10 con 0x7f8b940ad490 2024-03-06T07:33:20.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.942+0000 7f8b7bfff640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b8006b4d0 0x7f8b8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.942+0000 7f8b7bfff640 1 -- 172.21.15.59:0/4146352355 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8b8408a010 con 0x7f8b940ad490 2024-03-06T07:33:20.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.942+0000 7f8b92ffd640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b8006b4d0 0x7f8b8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.943+0000 7f8b92ffd640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b8006b4d0 0x7f8b8006d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f8b88036000 tx=0x7f8b88008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.948 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:20.946+0000 7f8b7bfff640 1 -- 172.21.15.59:0/4146352355 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b9409b8f0 con 0x7f8b940ad490 2024-03-06T07:33:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.164+0000 7f8b99d3e640 1 -- 172.21.15.59:0/4146352355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b940c7420 con 0x7f8b940ad490 2024-03-06T07:33:21.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.164+0000 7f8b7bfff640 1 -- 172.21.15.59:0/4146352355 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f8b84058f00 con 0x7f8b940ad490 2024-03-06T07:33:21.167 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:21.167 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:21.167 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:21.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 -- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b8006b4d0 msgr2=0x7f8b8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.168 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b8006b4d0 0x7f8b8006d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f8b88036000 tx=0x7f8b88008030 comp rx=0 tx=0).stop 2024-03-06T07:33:21.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 -- 172.21.15.59:0/4146352355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b940ad490 msgr2=0x7f8b9409b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b940ad490 0x7f8b9409b3b0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400c430 tx=0x7f8b8400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 -- 172.21.15.59:0/4146352355 shutdown_connections 2024-03-06T07:33:21.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8b8006b4d0 0x7f8b8006d990 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b9415df30 0x7f8b9415cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b940adc90 0x7f8b9415ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 --2- 172.21.15.59:0/4146352355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b940ad490 0x7f8b9409b3b0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 -- 172.21.15.59:0/4146352355 >> 172.21.15.59:0/4146352355 conn(0x7f8b940d00e0 msgr2=0x7f8b9406bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 -- 172.21.15.59:0/4146352355 shutdown_connections 2024-03-06T07:33:21.169 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:21.166+0000 7f8b99d3e640 1 -- 172.21.15.59:0/4146352355 wait complete. 2024-03-06T07:33:21.381 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:21.381 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:21.419 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:21.421 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:22.382 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:22.486 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.484+0000 7ff49c7bf640 1 Processor -- start 2024-03-06T07:33:22.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.484+0000 7ff49c7bf640 1 -- start start 2024-03-06T07:33:22.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.485+0000 7ff49c7bf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff49415dfe0 0x7ff4940cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.485+0000 7ff49c7bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4940d29c0 0x7ff4940b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.485+0000 7ff49c7bf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4940b3430 con 0x7ff4940cf830 2024-03-06T07:33:22.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.485+0000 7ff49c7bf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4940b31e0 con 0x7ff49415dfe0 2024-03-06T07:33:22.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.485+0000 7ff49c7bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4940b3d60 con 0x7ff4940d29c0 2024-03-06T07:33:22.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.486+0000 7ff49ad5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4940d29c0 0x7ff4940b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.486+0000 7ff49a55b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff49415dfe0 0x7ff4940cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.486+0000 7ff49ad5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4940d29c0 0x7ff4940b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:34732/0 (socket says 172.21.15.59:34732) 2024-03-06T07:33:22.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.486+0000 7ff49ad5c640 1 -- 172.21.15.59:0/2910727839 learned_addr learned my addr 172.21.15.59:0/2910727839 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:22.489 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.486+0000 7ff49ad5c640 1 -- 172.21.15.59:0/2910727839 >> v2:172.21.15.164:3300/0 conn(0x7ff49415dfe0 msgr2=0x7ff4940cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.486+0000 7ff49ad5c640 1 --2- 172.21.15.59:0/2910727839 >> v2:172.21.15.164:3300/0 conn(0x7ff49415dfe0 0x7ff4940cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.486+0000 7ff49ad5c640 1 -- 172.21.15.59:0/2910727839 >> v1:172.21.15.164:6789/0 conn(0x7ff4940cf830 legacy=0x7ff4940d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.486+0000 7ff49ad5c640 1 -- 172.21.15.59:0/2910727839 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4940b3710 con 0x7ff4940d29c0 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.487+0000 7ff49ad5c640 1 --2- 172.21.15.59:0/2910727839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4940d29c0 0x7ff4940b41f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009e50 tx=0x7ff48c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=544bd14c2a05c108 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.487+0000 7ff499559640 1 -- 172.21.15.59:0/2910727839 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48c02cb00 con 0x7ff4940d29c0 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.487+0000 7ff499559640 1 -- 172.21.15.59:0/2910727839 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48c02cca0 con 0x7ff4940d29c0 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.487+0000 7ff499559640 1 -- 172.21.15.59:0/2910727839 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48c034610 con 0x7ff4940d29c0 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.487+0000 7ff49c7bf640 1 -- 172.21.15.59:0/2910727839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4940d29c0 msgr2=0x7ff4940b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.487+0000 7ff49c7bf640 1 --2- 172.21.15.59:0/2910727839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4940d29c0 0x7ff4940b41f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009e50 tx=0x7ff48c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.488+0000 7ff49c7bf640 1 -- 172.21.15.59:0/2910727839 shutdown_connections 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.488+0000 7ff49c7bf640 1 --2- 172.21.15.59:0/2910727839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4940d29c0 0x7ff4940b41f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.488+0000 7ff49c7bf640 1 --2- 172.21.15.59:0/2910727839 >> v2:172.21.15.164:3300/0 conn(0x7ff49415dfe0 0x7ff4940cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.488+0000 7ff49c7bf640 1 -- 172.21.15.59:0/2910727839 >> 172.21.15.59:0/2910727839 conn(0x7ff4940d3cd0 msgr2=0x7ff4940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.488+0000 7ff49c7bf640 1 -- 172.21.15.59:0/2910727839 shutdown_connections 2024-03-06T07:33:22.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.488+0000 7ff49c7bf640 1 -- 172.21.15.59:0/2910727839 wait complete. 2024-03-06T07:33:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.489+0000 7ff49c7bf640 1 Processor -- start 2024-03-06T07:33:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.489+0000 7ff49c7bf640 1 -- start start 2024-03-06T07:33:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.489+0000 7ff49c7bf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4940cf830 0x7ff49406ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.489+0000 7ff49c7bf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4940d29c0 0x7ff4940b9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.489+0000 7ff49c7bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff49415dfe0 0x7ff49414ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff49c7bf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4940d1df0 con 0x7ff4940d29c0 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff49c7bf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4940d1c70 con 0x7ff4940cf830 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff49c7bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49415e3c0 con 0x7ff49415dfe0 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff499d5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4940d29c0 0x7ff4940b9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff499d5a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4940d29c0 0x7ff4940b9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41712/0 (socket says 172.21.15.59:41712) 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff499d5a640 1 -- 172.21.15.59:0/66944147 learned_addr learned my addr 172.21.15.59:0/66944147 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff49ad5c640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff49415dfe0 0x7ff49414ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff49a55b640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4940cf830 0x7ff49406ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff499d5a640 1 -- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff49415dfe0 msgr2=0x7ff49414ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff499d5a640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff49415dfe0 0x7ff49414ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff499d5a640 1 -- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4940cf830 msgr2=0x7ff49406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff499d5a640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4940cf830 0x7ff49406ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff499d5a640 1 -- 172.21.15.59:0/66944147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff494150ba0 con 0x7ff4940d29c0 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff49a55b640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4940cf830 0x7ff49406ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff499d5a640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4940d29c0 0x7ff4940b9120 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7ff49000c470 tx=0x7ff490001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff486ffd640 1 -- 172.21.15.59:0/66944147 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4900052d0 con 0x7ff4940d29c0 2024-03-06T07:33:22.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff486ffd640 1 -- 172.21.15.59:0/66944147 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff490005470 con 0x7ff4940d29c0 2024-03-06T07:33:22.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff49c7bf640 1 -- 172.21.15.59:0/66944147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff494150e80 con 0x7ff4940d29c0 2024-03-06T07:33:22.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff486ffd640 1 -- 172.21.15.59:0/66944147 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff490007260 con 0x7ff4940d29c0 2024-03-06T07:33:22.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.490+0000 7ff49c7bf640 1 -- 172.21.15.59:0/66944147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49406d350 con 0x7ff4940d29c0 2024-03-06T07:33:22.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.491+0000 7ff49c7bf640 1 -- 172.21.15.59:0/66944147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49406bf00 con 0x7ff4940d29c0 2024-03-06T07:33:22.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.492+0000 7ff486ffd640 1 -- 172.21.15.59:0/66944147 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff490007400 con 0x7ff4940d29c0 2024-03-06T07:33:22.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.493+0000 7ff486ffd640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff47806b4d0 0x7ff47806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.493+0000 7ff486ffd640 1 -- 172.21.15.59:0/66944147 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff490089850 con 0x7ff4940d29c0 2024-03-06T07:33:22.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.493+0000 7ff49a55b640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff47806b4d0 0x7ff47806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.496 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.494+0000 7ff49a55b640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff47806b4d0 0x7ff47806d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff488009a20 tx=0x7ff488008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.499 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.497+0000 7ff486ffd640 1 -- 172.21.15.59:0/66944147 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff49406bf00 con 0x7ff4940d29c0 2024-03-06T07:33:22.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.715+0000 7ff49c7bf640 1 -- 172.21.15.59:0/66944147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff49404f330 con 0x7ff4940d29c0 2024-03-06T07:33:22.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.716+0000 7ff486ffd640 1 -- 172.21.15.59:0/66944147 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7ff490058580 con 0x7ff4940d29c0 2024-03-06T07:33:22.718 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:22.718 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:22.719 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:22.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 -- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff47806b4d0 msgr2=0x7ff47806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff47806b4d0 0x7ff47806d990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff488009a20 tx=0x7ff488008040 comp rx=0 tx=0).stop 2024-03-06T07:33:22.720 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 -- 172.21.15.59:0/66944147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4940d29c0 msgr2=0x7ff4940b9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4940d29c0 0x7ff4940b9120 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7ff49000c470 tx=0x7ff490001e70 comp rx=0 tx=0).stop 2024-03-06T07:33:22.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 -- 172.21.15.59:0/66944147 shutdown_connections 2024-03-06T07:33:22.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff47806b4d0 0x7ff47806d990 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff49415dfe0 0x7ff49414ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4940d29c0 0x7ff4940b9120 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 --2- 172.21.15.59:0/66944147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4940cf830 0x7ff49406ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 -- 172.21.15.59:0/66944147 >> 172.21.15.59:0/66944147 conn(0x7ff4940d3cd0 msgr2=0x7ff49406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 -- 172.21.15.59:0/66944147 shutdown_connections 2024-03-06T07:33:22.721 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:22.718+0000 7ff49c7bf640 1 -- 172.21.15.59:0/66944147 wait complete. 2024-03-06T07:33:22.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:22.933 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:23.934 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.034+0000 7f793b20e640 1 Processor -- start 2024-03-06T07:33:24.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.034+0000 7f793b20e640 1 -- start start 2024-03-06T07:33:24.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f793b20e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79340d2710 0x7f79340b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f793b20e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79340b3430 con 0x7f79340d22b0 2024-03-06T07:33:24.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f793b20e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79340b3710 con 0x7f793415dfe0 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f793b20e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79340b31e0 con 0x7f79340d2710 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f7933fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f79340d22b0 0x7f79340b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35956/0 (socket says 172.21.15.59:35956) 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f7933fff640 1 -- 172.21.15.59:0/2319732104 learned_addr learned my addr 172.21.15.59:0/2319732104 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142697174 0 0) 0x7f79340b3430 con 0x7f79340d22b0 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79397ab640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79340d2710 0x7f79340b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7924003520 con 0x7f79340d22b0 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1572686531 0 0) 0x7f7924003520 con 0x7f79340d22b0 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79340d2710 msgr2=0x7f79340b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79340d2710 0x7f79340b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 >> v1:172.21.15.164:6789/0 conn(0x7f793415dfe0 legacy=0x7f79340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79340b39f0 con 0x7f79340d22b0 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f7920001670 con 0x7f79340d22b0 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 >> v1:172.21.15.59:6789/0 conn(0x7f79340d22b0 legacy=0x7f79340b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79240039e0 0x7f7924003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79340d2710 0x7f7924004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7924004d40 0x7f79240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79340b31e0 con 0x7f7924004d40 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792400baa0 con 0x7f79240039e0 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.035+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792400bca0 con 0x7f79340d2710 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.036+0000 7f79397ab640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7924004d40 0x7f79240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.036+0000 7f7938faa640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79240039e0 0x7f7924003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.036+0000 7f7933fff640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79340d2710 0x7f7924004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.036+0000 7f79397ab640 1 -- 172.21.15.59:0/2319732104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79340d2710 msgr2=0x7f7924004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.036+0000 7f79397ab640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79340d2710 0x7f7924004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.036+0000 7f79397ab640 1 -- 172.21.15.59:0/2319732104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79240039e0 msgr2=0x7f7924003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.036+0000 7f79397ab640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79240039e0 0x7f7924003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.036+0000 7f79397ab640 1 -- 172.21.15.59:0/2319732104 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7924003520 con 0x7f7924004d40 2024-03-06T07:33:24.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.037+0000 7f79397ab640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7924004d40 0x7f79240051c0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f792c009f90 tx=0x7f7924005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b631e5b232f9852d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.037+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c039070 con 0x7f7924004d40 2024-03-06T07:33:24.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.037+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792c02cbd0 con 0x7f7924004d40 2024-03-06T07:33:24.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.037+0000 7f79337fe640 1 -- 172.21.15.59:0/2319732104 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c009c10 con 0x7f7924004d40 2024-03-06T07:33:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.037+0000 7f793b20e640 1 -- 172.21.15.59:0/2319732104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7924004d40 msgr2=0x7f79240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.037+0000 7f793b20e640 1 --2- 172.21.15.59:0/2319732104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7924004d40 0x7f79240051c0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f792c009f90 tx=0x7f7924005830 comp rx=0 tx=0).stop 2024-03-06T07:33:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.037+0000 7f79397ab640 1 -- 172.21.15.59:0/2319732104 reap_dead start 2024-03-06T07:33:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.037+0000 7f793b20e640 1 -- 172.21.15.59:0/2319732104 shutdown_connections 2024-03-06T07:33:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.037+0000 7f793b20e640 1 -- 172.21.15.59:0/2319732104 >> 172.21.15.59:0/2319732104 conn(0x7f79340d3cd0 msgr2=0x7f79340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.037+0000 7f793b20e640 1 -- 172.21.15.59:0/2319732104 shutdown_connections 2024-03-06T07:33:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.037+0000 7f793b20e640 1 -- 172.21.15.59:0/2319732104 wait complete. 2024-03-06T07:33:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.038+0000 7f793b20e640 1 Processor -- start 2024-03-06T07:33:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.038+0000 7f793b20e640 1 -- start start 2024-03-06T07:33:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.039+0000 7f793b20e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7934153eb0 0x7f79341545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.039+0000 7f793b20e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7934155e80 0x7f79341f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.039+0000 7f793b20e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79341f7930 0x7f79341f7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.039+0000 7f793b20e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79340cfa00 con 0x7f79341f7930 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.039+0000 7f793b20e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79340b3d60 con 0x7f7934155e80 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.039+0000 7f793b20e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79340d1c70 con 0x7f7934153eb0 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.039+0000 7f79397ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79341f7930 0x7f79341f7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.039+0000 7f79397ab640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79341f7930 0x7f79341f7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59750/0 (socket says 172.21.15.59:59750) 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.039+0000 7f79397ab640 1 -- 172.21.15.59:0/3397891144 learned_addr learned my addr 172.21.15.59:0/3397891144 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f7933fff640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7934155e80 0x7f79341f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f7938faa640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7934153eb0 0x7f79341545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f79397ab640 1 -- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7934153eb0 msgr2=0x7f79341545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f79397ab640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7934153eb0 0x7f79341545b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f79397ab640 1 -- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7934155e80 msgr2=0x7f79341f73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f79397ab640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7934155e80 0x7f79341f73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f79397ab640 1 -- 172.21.15.59:0/3397891144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7934157530 con 0x7f79341f7930 2024-03-06T07:33:24.046 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f7933fff640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7934155e80 0x7f79341f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:24.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f79397ab640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79341f7930 0x7f79341f7dc0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f792c009f90 tx=0x7f792c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f79317fa640 1 -- 172.21.15.59:0/3397891144 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c039070 con 0x7f79341f7930 2024-03-06T07:33:24.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f79317fa640 1 -- 172.21.15.59:0/3397891144 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792c03a040 con 0x7f79341f7930 2024-03-06T07:33:24.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f793b20e640 1 -- 172.21.15.59:0/3397891144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7934157810 con 0x7f79341f7930 2024-03-06T07:33:24.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f79317fa640 1 -- 172.21.15.59:0/3397891144 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c009c10 con 0x7f79341f7930 2024-03-06T07:33:24.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.040+0000 7f793b20e640 1 -- 172.21.15.59:0/3397891144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7934154290 con 0x7f79341f7930 2024-03-06T07:33:24.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.041+0000 7f793b20e640 1 -- 172.21.15.59:0/3397891144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f793404f330 con 0x7f79341f7930 2024-03-06T07:33:24.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.042+0000 7f79317fa640 1 -- 172.21.15.59:0/3397891144 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f792c046030 con 0x7f79341f7930 2024-03-06T07:33:24.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.042+0000 7f79317fa640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f791806b500 0x7f791806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.043+0000 7f79317fa640 1 -- 172.21.15.59:0/3397891144 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f792c0abd90 con 0x7f79341f7930 2024-03-06T07:33:24.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.043+0000 7f7938faa640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f791806b500 0x7f791806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.043+0000 7f7938faa640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f791806b500 0x7f791806d9c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f79280035f0 tx=0x7f7928009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.052+0000 7f79317fa640 1 -- 172.21.15.59:0/3397891144 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f792c04d060 con 0x7f79341f7930 2024-03-06T07:33:24.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.280+0000 7f793b20e640 1 -- 172.21.15.59:0/3397891144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79340d4a90 con 0x7f79341f7930 2024-03-06T07:33:24.283 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.281+0000 7f79317fa640 1 -- 172.21.15.59:0/3397891144 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f792c07abe0 con 0x7f79341f7930 2024-03-06T07:33:24.284 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:24.284 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:24.284 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.282+0000 7f793b20e640 1 -- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f791806b500 msgr2=0x7f791806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.282+0000 7f793b20e640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f791806b500 0x7f791806d9c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f79280035f0 tx=0x7f7928009040 comp rx=0 tx=0).stop 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.282+0000 7f793b20e640 1 -- 172.21.15.59:0/3397891144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79341f7930 msgr2=0x7f79341f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.282+0000 7f793b20e640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79341f7930 0x7f79341f7dc0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f792c009f90 tx=0x7f792c009750 comp rx=0 tx=0).stop 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.283+0000 7f793b20e640 1 -- 172.21.15.59:0/3397891144 shutdown_connections 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.283+0000 7f793b20e640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f791806b500 0x7f791806d9c0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.283+0000 7f793b20e640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79341f7930 0x7f79341f7dc0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.283+0000 7f793b20e640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7934155e80 0x7f79341f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.283+0000 7f793b20e640 1 --2- 172.21.15.59:0/3397891144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7934153eb0 0x7f79341545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.283+0000 7f793b20e640 1 -- 172.21.15.59:0/3397891144 >> 172.21.15.59:0/3397891144 conn(0x7f79340d3cd0 msgr2=0x7f79340b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.283+0000 7f793b20e640 1 -- 172.21.15.59:0/3397891144 shutdown_connections 2024-03-06T07:33:24.286 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:24.283+0000 7f793b20e640 1 -- 172.21.15.59:0/3397891144 wait complete. 2024-03-06T07:33:24.498 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:24.498 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:25.499 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:25.602 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.600+0000 7f112202a640 1 Processor -- start 2024-03-06T07:33:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.601+0000 7f112202a640 1 -- start start 2024-03-06T07:33:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.601+0000 7f112202a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f111c15e110 0x7f111c0ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.601+0000 7f112202a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f111c0b5630 0x7f111c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.601+0000 7f112202a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f111c06b4f0 0x7f111c0b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.601+0000 7f112202a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111c0ae030 con 0x7f111c06b4f0 2024-03-06T07:33:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.601+0000 7f112202a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111c0b0160 con 0x7f111c15e110 2024-03-06T07:33:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.601+0000 7f112202a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111c0ad9e0 con 0x7f111c0b5630 2024-03-06T07:33:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.601+0000 7f111bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f111c06b4f0 0x7f111c0b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.602+0000 7f111bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f111c06b4f0 0x7f111c0b38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59756/0 (socket says 172.21.15.59:59756) 2024-03-06T07:33:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.602+0000 7f111bfff640 1 -- 172.21.15.59:0/1968474263 learned_addr learned my addr 172.21.15.59:0/1968474263 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.602+0000 7f1113fff640 1 --2- 172.21.15.59:0/1968474263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f111c0b5630 0x7f111c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.602+0000 7f111b7fe640 1 --2- 172.21.15.59:0/1968474263 >> v2:172.21.15.164:3300/0 conn(0x7f111c15e110 0x7f111c0ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.602+0000 7f111bfff640 1 -- 172.21.15.59:0/1968474263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f111c0b5630 msgr2=0x7f111c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.602+0000 7f111bfff640 1 --2- 172.21.15.59:0/1968474263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f111c0b5630 0x7f111c06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.602+0000 7f111bfff640 1 -- 172.21.15.59:0/1968474263 >> v2:172.21.15.164:3300/0 conn(0x7f111c15e110 msgr2=0x7f111c0ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.602+0000 7f111bfff640 1 --2- 172.21.15.59:0/1968474263 >> v2:172.21.15.164:3300/0 conn(0x7f111c15e110 0x7f111c0ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.602+0000 7f111bfff640 1 -- 172.21.15.59:0/1968474263 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c0adcc0 con 0x7f111c06b4f0 2024-03-06T07:33:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.602+0000 7f1113fff640 1 --2- 172.21.15.59:0/1968474263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f111c0b5630 0x7f111c06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f111bfff640 1 --2- 172.21.15.59:0/1968474263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f111c06b4f0 0x7f111c0b38f0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f111c0689c0 tx=0x7f111402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f3af83d743dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f111affd640 1 -- 172.21.15.59:0/1968474263 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111402cb00 con 0x7f111c06b4f0 2024-03-06T07:33:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f111affd640 1 -- 172.21.15.59:0/1968474263 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111402cca0 con 0x7f111c06b4f0 2024-03-06T07:33:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f111affd640 1 -- 172.21.15.59:0/1968474263 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1114034610 con 0x7f111c06b4f0 2024-03-06T07:33:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f112202a640 1 -- 172.21.15.59:0/1968474263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f111c06b4f0 msgr2=0x7f111c0b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f112202a640 1 --2- 172.21.15.59:0/1968474263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f111c06b4f0 0x7f111c0b38f0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f111c0689c0 tx=0x7f111402bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f112202a640 1 -- 172.21.15.59:0/1968474263 shutdown_connections 2024-03-06T07:33:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f112202a640 1 --2- 172.21.15.59:0/1968474263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f111c06b4f0 0x7f111c0b38f0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f112202a640 1 --2- 172.21.15.59:0/1968474263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f111c0b5630 0x7f111c06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f112202a640 1 --2- 172.21.15.59:0/1968474263 >> v2:172.21.15.164:3300/0 conn(0x7f111c15e110 0x7f111c0ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f112202a640 1 -- 172.21.15.59:0/1968474263 >> 172.21.15.59:0/1968474263 conn(0x7f111c0d0960 msgr2=0x7f111c0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f112202a640 1 -- 172.21.15.59:0/1968474263 shutdown_connections 2024-03-06T07:33:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.603+0000 7f112202a640 1 -- 172.21.15.59:0/1968474263 wait complete. 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.605+0000 7f112202a640 1 Processor -- start 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.605+0000 7f112202a640 1 -- start start 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.605+0000 7f112202a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f111c06b4f0 0x7f111c0cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.605+0000 7f112202a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f111c0b5630 0x7f111c1482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.605+0000 7f112202a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f111c15e110 0x7f111c0cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.605+0000 7f112202a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c0b5bf0 con 0x7f111c06b4f0 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.605+0000 7f112202a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c0b5a70 con 0x7f111c0b5630 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.605+0000 7f112202a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111c0d1660 con 0x7f111c15e110 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.605+0000 7f111b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f111c06b4f0 0x7f111c0cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.605+0000 7f111b7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f111c06b4f0 0x7f111c0cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59762/0 (socket says 172.21.15.59:59762) 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.605+0000 7f111b7fe640 1 -- 172.21.15.59:0/891392623 learned_addr learned my addr 172.21.15.59:0/891392623 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.606+0000 7f111b7fe640 1 -- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f111c15e110 msgr2=0x7f111c0cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.606+0000 7f111b7fe640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f111c15e110 0x7f111c0cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.606+0000 7f111b7fe640 1 -- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f111c0b5630 msgr2=0x7f111c1482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.606+0000 7f111b7fe640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f111c0b5630 0x7f111c1482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.606+0000 7f111b7fe640 1 -- 172.21.15.59:0/891392623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c0ce5e0 con 0x7f111c06b4f0 2024-03-06T07:33:25.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.606+0000 7f111b7fe640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f111c06b4f0 0x7f111c0cd7f0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f110c00d5f0 tx=0x7f110c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:25.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.606+0000 7f1118ff9640 1 -- 172.21.15.59:0/891392623 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c0125f0 con 0x7f111c06b4f0 2024-03-06T07:33:25.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.606+0000 7f1118ff9640 1 -- 172.21.15.59:0/891392623 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110c019070 con 0x7f111c06b4f0 2024-03-06T07:33:25.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.606+0000 7f112202a640 1 -- 172.21.15.59:0/891392623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111c15e830 con 0x7f111c06b4f0 2024-03-06T07:33:25.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.606+0000 7f1118ff9640 1 -- 172.21.15.59:0/891392623 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c0129d0 con 0x7f111c06b4f0 2024-03-06T07:33:25.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.606+0000 7f112202a640 1 -- 172.21.15.59:0/891392623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111c15ead0 con 0x7f111c06b4f0 2024-03-06T07:33:25.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.607+0000 7f112202a640 1 -- 172.21.15.59:0/891392623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111c04f330 con 0x7f111c06b4f0 2024-03-06T07:33:25.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.608+0000 7f1118ff9640 1 -- 172.21.15.59:0/891392623 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f110c012b70 con 0x7f111c06b4f0 2024-03-06T07:33:25.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.608+0000 7f1118ff9640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10fc06b4d0 0x7f10fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.608+0000 7f1118ff9640 1 -- 172.21.15.59:0/891392623 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f110c08a580 con 0x7f111c06b4f0 2024-03-06T07:33:25.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.608+0000 7f1113fff640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10fc06b4d0 0x7f10fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.609+0000 7f1113fff640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10fc06b4d0 0x7f10fc06d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f111c0b2520 tx=0x7f1104008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.613+0000 7f1118ff9640 1 -- 172.21.15.59:0/891392623 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f110c059470 con 0x7f111c06b4f0 2024-03-06T07:33:25.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.867+0000 7f112202a640 1 -- 172.21.15.59:0/891392623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f111c0d06b0 con 0x7f111c06b4f0 2024-03-06T07:33:25.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.868+0000 7f1118ff9640 1 -- 172.21.15.59:0/891392623 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f110c058d70 con 0x7f111c06b4f0 2024-03-06T07:33:25.870 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:25.870 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:25.871 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:25.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 -- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10fc06b4d0 msgr2=0x7f10fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10fc06b4d0 0x7f10fc06d990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f111c0b2520 tx=0x7f1104008040 comp rx=0 tx=0).stop 2024-03-06T07:33:25.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 -- 172.21.15.59:0/891392623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f111c06b4f0 msgr2=0x7f111c0cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f111c06b4f0 0x7f111c0cd7f0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f110c00d5f0 tx=0x7f110c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 -- 172.21.15.59:0/891392623 shutdown_connections 2024-03-06T07:33:25.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f10fc06b4d0 0x7f10fc06d990 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f111c15e110 0x7f111c0cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f111c0b5630 0x7f111c1482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 --2- 172.21.15.59:0/891392623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f111c06b4f0 0x7f111c0cd7f0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 -- 172.21.15.59:0/891392623 >> 172.21.15.59:0/891392623 conn(0x7f111c0d0960 msgr2=0x7f111c0b4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:25.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 -- 172.21.15.59:0/891392623 shutdown_connections 2024-03-06T07:33:25.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:25.871+0000 7f112202a640 1 -- 172.21.15.59:0/891392623 wait complete. 2024-03-06T07:33:26.084 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:26.085 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:27.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:27.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.186+0000 7f5b1d381640 1 Processor -- start 2024-03-06T07:33:27.188 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.186+0000 7f5b1d381640 1 -- start start 2024-03-06T07:33:27.191 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b1d381640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5b1815dfe0 0x7f5b180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b1d381640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5b180d29c0 0x7f5b180b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b1d381640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b180b3430 con 0x7f5b180cf830 2024-03-06T07:33:27.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b1d381640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b180b31e0 con 0x7f5b1815dfe0 2024-03-06T07:33:27.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b1d381640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b180b3d60 con 0x7f5b180d29c0 2024-03-06T07:33:27.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b16ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5b1815dfe0 0x7f5b180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b16ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5b1815dfe0 0x7f5b180cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59772/0 (socket says 172.21.15.59:59772) 2024-03-06T07:33:27.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b16ffd640 1 -- 172.21.15.59:0/241541746 learned_addr learned my addr 172.21.15.59:0/241541746 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:27.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b177fe640 1 --2- 172.21.15.59:0/241541746 >> v2:172.21.15.164:3300/0 conn(0x7f5b180d29c0 0x7f5b180b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b16ffd640 1 -- 172.21.15.59:0/241541746 >> v2:172.21.15.164:3300/0 conn(0x7f5b180d29c0 msgr2=0x7f5b180b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b16ffd640 1 --2- 172.21.15.59:0/241541746 >> v2:172.21.15.164:3300/0 conn(0x7f5b180d29c0 0x7f5b180b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.192 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b16ffd640 1 -- 172.21.15.59:0/241541746 >> v1:172.21.15.164:6789/0 conn(0x7f5b180cf830 legacy=0x7f5b180d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.187+0000 7f5b16ffd640 1 -- 172.21.15.59:0/241541746 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b180b3710 con 0x7f5b1815dfe0 2024-03-06T07:33:27.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.188+0000 7f5b16ffd640 1 --2- 172.21.15.59:0/241541746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b1815dfe0 0x7f5b180cf2f0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f5b180b3cd0 tx=0x7f5b0802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93dcdb00c224642 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.188+0000 7f5b167fc640 1 -- 172.21.15.59:0/241541746 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08035020 con 0x7f5b1815dfe0 2024-03-06T07:33:27.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.188+0000 7f5b167fc640 1 -- 172.21.15.59:0/241541746 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b08031870 con 0x7f5b1815dfe0 2024-03-06T07:33:27.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.188+0000 7f5b167fc640 1 -- 172.21.15.59:0/241541746 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08031c40 con 0x7f5b1815dfe0 2024-03-06T07:33:27.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.188+0000 7f5b1d381640 1 -- 172.21.15.59:0/241541746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b1815dfe0 msgr2=0x7f5b180cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.188+0000 7f5b1d381640 1 --2- 172.21.15.59:0/241541746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b1815dfe0 0x7f5b180cf2f0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f5b180b3cd0 tx=0x7f5b0802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:27.193 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.189+0000 7f5b1d381640 1 -- 172.21.15.59:0/241541746 shutdown_connections 2024-03-06T07:33:27.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.189+0000 7f5b1d381640 1 --2- 172.21.15.59:0/241541746 >> v2:172.21.15.164:3300/0 conn(0x7f5b180d29c0 0x7f5b180b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.189+0000 7f5b1d381640 1 --2- 172.21.15.59:0/241541746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b1815dfe0 0x7f5b180cf2f0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.189+0000 7f5b1d381640 1 -- 172.21.15.59:0/241541746 >> 172.21.15.59:0/241541746 conn(0x7f5b180d3cd0 msgr2=0x7f5b180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.189+0000 7f5b1d381640 1 -- 172.21.15.59:0/241541746 shutdown_connections 2024-03-06T07:33:27.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.189+0000 7f5b1d381640 1 -- 172.21.15.59:0/241541746 wait complete. 2024-03-06T07:33:27.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.190+0000 7f5b1d381640 1 Processor -- start 2024-03-06T07:33:27.194 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.190+0000 7f5b1d381640 1 -- start start 2024-03-06T07:33:27.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.190+0000 7f5b1d381640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b180cf830 0x7f5b18154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.190+0000 7f5b1d381640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b180d29c0 0x7f5b180cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.190+0000 7f5b1d381640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b1815dfe0 0x7f5b180cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.190+0000 7f5b1d381640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b180d1df0 con 0x7f5b180cf830 2024-03-06T07:33:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.190+0000 7f5b1d381640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b180d1c70 con 0x7f5b1815dfe0 2024-03-06T07:33:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.190+0000 7f5b1d381640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1815e3c0 con 0x7f5b180d29c0 2024-03-06T07:33:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b16ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b180cf830 0x7f5b18154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b16ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b180cf830 0x7f5b18154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59788/0 (socket says 172.21.15.59:59788) 2024-03-06T07:33:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b16ffd640 1 -- 172.21.15.59:0/1440096965 learned_addr learned my addr 172.21.15.59:0/1440096965 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b177fe640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b1815dfe0 0x7f5b180cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b0ffff640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b180d29c0 0x7f5b180cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b16ffd640 1 -- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b180d29c0 msgr2=0x7f5b180cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b16ffd640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b180d29c0 0x7f5b180cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b16ffd640 1 -- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b1815dfe0 msgr2=0x7f5b180cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b16ffd640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b1815dfe0 0x7f5b180cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b16ffd640 1 -- 172.21.15.59:0/1440096965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b180cbdf0 con 0x7f5b180cf830 2024-03-06T07:33:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b16ffd640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b180cf830 0x7f5b18154070 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f5b08000c00 tx=0x7f5b080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b0f7fe640 1 -- 172.21.15.59:0/1440096965 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b080443f0 con 0x7f5b180cf830 2024-03-06T07:33:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b0f7fe640 1 -- 172.21.15.59:0/1440096965 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b08044590 con 0x7f5b180cf830 2024-03-06T07:33:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b0f7fe640 1 -- 172.21.15.59:0/1440096965 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08044980 con 0x7f5b180cf830 2024-03-06T07:33:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b1d381640 1 -- 172.21.15.59:0/1440096965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b180cc0d0 con 0x7f5b180cf830 2024-03-06T07:33:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.191+0000 7f5b1d381640 1 -- 172.21.15.59:0/1440096965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b180cc690 con 0x7f5b180cf830 2024-03-06T07:33:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.192+0000 7f5b1d381640 1 -- 172.21.15.59:0/1440096965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1804f330 con 0x7f5b180cf830 2024-03-06T07:33:27.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.193+0000 7f5b0f7fe640 1 -- 172.21.15.59:0/1440096965 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b08038020 con 0x7f5b180cf830 2024-03-06T07:33:27.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.193+0000 7f5b0f7fe640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5af806b4d0 0x7f5af806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.193+0000 7f5b0f7fe640 1 -- 172.21.15.59:0/1440096965 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5b080ac3d0 con 0x7f5b180cf830 2024-03-06T07:33:27.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.193+0000 7f5b0ffff640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5af806b4d0 0x7f5af806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.200 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.194+0000 7f5b0ffff640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5af806b4d0 0x7f5af806d990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f5b180b9960 tx=0x7f5b00009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.198+0000 7f5b0f7fe640 1 -- 172.21.15.59:0/1440096965 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b0803f070 con 0x7f5b180cf830 2024-03-06T07:33:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.417+0000 7f5b1d381640 1 -- 172.21.15.59:0/1440096965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b180b0de0 con 0x7f5b180cf830 2024-03-06T07:33:27.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.417+0000 7f5b0f7fe640 1 -- 172.21.15.59:0/1440096965 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f5b0807ac70 con 0x7f5b180cf830 2024-03-06T07:33:27.420 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:27.420 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:27.420 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:27.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 -- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5af806b4d0 msgr2=0x7f5af806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5af806b4d0 0x7f5af806d990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f5b180b9960 tx=0x7f5b00009040 comp rx=0 tx=0).stop 2024-03-06T07:33:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 -- 172.21.15.59:0/1440096965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b180cf830 msgr2=0x7f5b18154070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b180cf830 0x7f5b18154070 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f5b08000c00 tx=0x7f5b080046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 -- 172.21.15.59:0/1440096965 shutdown_connections 2024-03-06T07:33:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5af806b4d0 0x7f5af806d990 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b1815dfe0 0x7f5b180cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b180d29c0 0x7f5b180cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 --2- 172.21.15.59:0/1440096965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b180cf830 0x7f5b18154070 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 -- 172.21.15.59:0/1440096965 >> 172.21.15.59:0/1440096965 conn(0x7f5b180d3cd0 msgr2=0x7f5b180b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 -- 172.21.15.59:0/1440096965 shutdown_connections 2024-03-06T07:33:27.422 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:27.419+0000 7f5b1d381640 1 -- 172.21.15.59:0/1440096965 wait complete. 2024-03-06T07:33:27.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:27.634 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:28.635 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.736+0000 7fe2ba3e9640 1 Processor -- start 2024-03-06T07:33:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.736+0000 7fe2ba3e9640 1 -- start start 2024-03-06T07:33:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.736+0000 7fe2ba3e9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe2b40d22b0 0x7fe2b40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.736+0000 7fe2ba3e9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe2b40b41f0 0x7fe2b40b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.736+0000 7fe2ba3e9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b40b3430 con 0x7fe2b415dfe0 2024-03-06T07:33:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.736+0000 7fe2ba3e9640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b40d2c50 con 0x7fe2b40b41f0 2024-03-06T07:33:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.736+0000 7fe2ba3e9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b40b31e0 con 0x7fe2b40d22b0 2024-03-06T07:33:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.736+0000 7fe2b8986640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe2b40b41f0 0x7fe2b40b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.736+0000 7fe2b3fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe2b415dfe0 0x7fe2b40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35958/0 (socket says 172.21.15.59:35958) 2024-03-06T07:33:28.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.736+0000 7fe2b8986640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe2b40b41f0 0x7fe2b40b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59790/0 (socket says 172.21.15.59:59790) 2024-03-06T07:33:28.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.736+0000 7fe2b3fff640 1 -- 172.21.15.59:0/4196025050 learned_addr learned my addr 172.21.15.59:0/4196025050 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:28.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.737+0000 7fe2b37fe640 1 --2- 172.21.15.59:0/4196025050 >> v2:172.21.15.164:3300/0 conn(0x7fe2b40d22b0 0x7fe2b40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.737+0000 7fe2b8986640 1 -- 172.21.15.59:0/4196025050 >> v2:172.21.15.164:3300/0 conn(0x7fe2b40d22b0 msgr2=0x7fe2b40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:28.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.737+0000 7fe2b8986640 1 --2- 172.21.15.59:0/4196025050 >> v2:172.21.15.164:3300/0 conn(0x7fe2b40d22b0 0x7fe2b40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.737+0000 7fe2b8986640 1 -- 172.21.15.59:0/4196025050 >> v1:172.21.15.59:6789/0 conn(0x7fe2b415dfe0 legacy=0x7fe2b40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.737+0000 7fe2b8986640 1 -- 172.21.15.59:0/4196025050 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40b3710 con 0x7fe2b40b41f0 2024-03-06T07:33:28.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.737+0000 7fe2b2ffd640 1 -- 172.21.15.59:0/4196025050 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098348204 0 0) 0x7fe2b40b3430 con 0x7fe2b415dfe0 2024-03-06T07:33:28.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2b8986640 1 --2- 172.21.15.59:0/4196025050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2b40b41f0 0x7fe2b40b4680 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009d90 tx=0x7fe2ac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3a552ac273671a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2b2ffd640 1 -- 172.21.15.59:0/4196025050 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac039070 con 0x7fe2b40b41f0 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2b2ffd640 1 -- 172.21.15.59:0/4196025050 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac02cc60 con 0x7fe2b40b41f0 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2b2ffd640 1 -- 172.21.15.59:0/4196025050 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac034660 con 0x7fe2b40b41f0 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/4196025050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2b40b41f0 msgr2=0x7fe2b40b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2ba3e9640 1 --2- 172.21.15.59:0/4196025050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2b40b41f0 0x7fe2b40b4680 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009d90 tx=0x7fe2ac02beb0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/4196025050 shutdown_connections 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2ba3e9640 1 --2- 172.21.15.59:0/4196025050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2b40b41f0 0x7fe2b40b4680 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2ba3e9640 1 --2- 172.21.15.59:0/4196025050 >> v2:172.21.15.164:3300/0 conn(0x7fe2b40d22b0 0x7fe2b40d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/4196025050 >> 172.21.15.59:0/4196025050 conn(0x7fe2b40d3cd0 msgr2=0x7fe2b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/4196025050 shutdown_connections 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.738+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/4196025050 wait complete. 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.739+0000 7fe2ba3e9640 1 Processor -- start 2024-03-06T07:33:28.742 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.740+0000 7fe2ba3e9640 1 -- start start 2024-03-06T07:33:28.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.740+0000 7fe2ba3e9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2b40b41f0 0x7fe2b415ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.740+0000 7fe2ba3e9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2b40d22b0 0x7fe2b40cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.740+0000 7fe2ba3e9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2b415dfe0 0x7fe2b40b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.740+0000 7fe2ba3e9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b40cfa00 con 0x7fe2b40b41f0 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.740+0000 7fe2ba3e9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b40b3d60 con 0x7fe2b415dfe0 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.740+0000 7fe2ba3e9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b40d1c70 con 0x7fe2b40d22b0 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.740+0000 7fe2b3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2b40b41f0 0x7fe2b415ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2b40b41f0 0x7fe2b415ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59796/0 (socket says 172.21.15.59:59796) 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b3fff640 1 -- 172.21.15.59:0/3862974829 learned_addr learned my addr 172.21.15.59:0/3862974829 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b3fff640 1 -- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2b40d22b0 msgr2=0x7fe2b40cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b3fff640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2b40d22b0 0x7fe2b40cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b3fff640 1 -- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2b415dfe0 msgr2=0x7fe2b40b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b8986640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2b415dfe0 0x7fe2b40b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b3fff640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2b415dfe0 0x7fe2b40b7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b3fff640 1 -- 172.21.15.59:0/3862974829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40cc820 con 0x7fe2b40b41f0 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b3fff640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2b40b41f0 0x7fe2b415ace0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c430 tx=0x7fe2a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b0ff9640 1 -- 172.21.15.59:0/3862974829 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a40074e0 con 0x7fe2b40b41f0 2024-03-06T07:33:28.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b0ff9640 1 -- 172.21.15.59:0/3862974829 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a40047e0 con 0x7fe2b40b41f0 2024-03-06T07:33:28.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/3862974829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b40ccb00 con 0x7fe2b40b41f0 2024-03-06T07:33:28.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2b0ff9640 1 -- 172.21.15.59:0/3862974829 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4004bd0 con 0x7fe2b40b41f0 2024-03-06T07:33:28.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.741+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/3862974829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b40cd3a0 con 0x7fe2b40b41f0 2024-03-06T07:33:28.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.742+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/3862974829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b40cccd0 con 0x7fe2b40b41f0 2024-03-06T07:33:28.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.743+0000 7fe2b0ff9640 1 -- 172.21.15.59:0/3862974829 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2a4011070 con 0x7fe2b40b41f0 2024-03-06T07:33:28.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.743+0000 7fe2b0ff9640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2a006b4d0 0x7fe2a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.743+0000 7fe2b0ff9640 1 -- 172.21.15.59:0/3862974829 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe2a4089ee0 con 0x7fe2b40b41f0 2024-03-06T07:33:28.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.743+0000 7fe2b37fe640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2a006b4d0 0x7fe2a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.747 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.744+0000 7fe2b37fe640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2a006b4d0 0x7fe2a006d990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40a1bc0 tx=0x7fe2a8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:28.747+0000 7fe2b0ff9640 1 -- 172.21.15.59:0/3862974829 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2a4058d40 con 0x7fe2b40b41f0 2024-03-06T07:33:29.003 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.001+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/3862974829 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2b404f330 con 0x7fe2b40b41f0 2024-03-06T07:33:29.004 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.002+0000 7fe2b0ff9640 1 -- 172.21.15.59:0/3862974829 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fe2a4058640 con 0x7fe2b40b41f0 2024-03-06T07:33:29.004 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:29.004 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:29.004 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:29.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2a006b4d0 msgr2=0x7fe2a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2a006b4d0 0x7fe2a006d990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40a1bc0 tx=0x7fe2a8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:29.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/3862974829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2b40b41f0 msgr2=0x7fe2b415ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2b40b41f0 0x7fe2b415ace0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fe2a400c430 tx=0x7fe2a400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/3862974829 shutdown_connections 2024-03-06T07:33:29.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe2a006b4d0 0x7fe2a006d990 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2b415dfe0 0x7fe2b40b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2b40d22b0 0x7fe2b40cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 --2- 172.21.15.59:0/3862974829 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2b40b41f0 0x7fe2b415ace0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/3862974829 >> 172.21.15.59:0/3862974829 conn(0x7fe2b40d3cd0 msgr2=0x7fe2b40d0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/3862974829 shutdown_connections 2024-03-06T07:33:29.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:29.004+0000 7fe2ba3e9640 1 -- 172.21.15.59:0/3862974829 wait complete. 2024-03-06T07:33:29.218 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:29.218 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:30.219 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.319+0000 7fde8badf640 1 Processor -- start 2024-03-06T07:33:30.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.319+0000 7fde8badf640 1 -- start start 2024-03-06T07:33:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8badf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde8415dfe0 0x7fde840cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8badf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fde840d29c0 0x7fde840b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8badf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde840b3430 con 0x7fde840cf830 2024-03-06T07:33:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8badf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde840b31e0 con 0x7fde840d29c0 2024-03-06T07:33:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8badf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde840b3d60 con 0x7fde8415dfe0 2024-03-06T07:33:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8987b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde8415dfe0 0x7fde840cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8a07c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fde840d29c0 0x7fde840b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8987b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde8415dfe0 0x7fde840cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:44844/0 (socket says 172.21.15.59:44844) 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8987b640 1 -- 172.21.15.59:0/3144197098 learned_addr learned my addr 172.21.15.59:0/3144197098 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8987b640 1 -- 172.21.15.59:0/3144197098 >> v2:172.21.15.164:3300/0 conn(0x7fde840d29c0 msgr2=0x7fde840b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8987b640 1 --2- 172.21.15.59:0/3144197098 >> v2:172.21.15.164:3300/0 conn(0x7fde840d29c0 0x7fde840b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8987b640 1 -- 172.21.15.59:0/3144197098 >> v1:172.21.15.164:6789/0 conn(0x7fde840cf830 legacy=0x7fde840d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.321+0000 7fde8987b640 1 -- 172.21.15.59:0/3144197098 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde840b3710 con 0x7fde8415dfe0 2024-03-06T07:33:30.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.322+0000 7fde8987b640 1 --2- 172.21.15.59:0/3144197098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde8415dfe0 0x7fde840cf2f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009870 tx=0x7fde7c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e2b5b676850d034 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.322+0000 7fde88879640 1 -- 172.21.15.59:0/3144197098 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c034070 con 0x7fde8415dfe0 2024-03-06T07:33:30.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.322+0000 7fde88879640 1 -- 172.21.15.59:0/3144197098 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c030840 con 0x7fde8415dfe0 2024-03-06T07:33:30.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.322+0000 7fde88879640 1 -- 172.21.15.59:0/3144197098 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c030c10 con 0x7fde8415dfe0 2024-03-06T07:33:30.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.323+0000 7fde8badf640 1 -- 172.21.15.59:0/3144197098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde8415dfe0 msgr2=0x7fde840cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.323+0000 7fde8badf640 1 --2- 172.21.15.59:0/3144197098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde8415dfe0 0x7fde840cf2f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009870 tx=0x7fde7c02b980 comp rx=0 tx=0).stop 2024-03-06T07:33:30.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.323+0000 7fde8badf640 1 -- 172.21.15.59:0/3144197098 shutdown_connections 2024-03-06T07:33:30.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.323+0000 7fde8badf640 1 --2- 172.21.15.59:0/3144197098 >> v2:172.21.15.164:3300/0 conn(0x7fde840d29c0 0x7fde840b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.323+0000 7fde8badf640 1 --2- 172.21.15.59:0/3144197098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde8415dfe0 0x7fde840cf2f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.323+0000 7fde8badf640 1 -- 172.21.15.59:0/3144197098 >> 172.21.15.59:0/3144197098 conn(0x7fde840d3cd0 msgr2=0x7fde840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.323+0000 7fde8badf640 1 -- 172.21.15.59:0/3144197098 shutdown_connections 2024-03-06T07:33:30.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.323+0000 7fde8badf640 1 -- 172.21.15.59:0/3144197098 wait complete. 2024-03-06T07:33:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.324+0000 7fde8badf640 1 Processor -- start 2024-03-06T07:33:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.324+0000 7fde8badf640 1 -- start start 2024-03-06T07:33:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8badf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde840cf830 0x7fde840cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8badf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde840d29c0 0x7fde8414ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8badf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde8415dfe0 0x7fde8414cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8badf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde840d1df0 con 0x7fde8415dfe0 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8badf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde840d1c70 con 0x7fde840cf830 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8badf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8415e3c0 con 0x7fde840d29c0 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8a07c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde8415dfe0 0x7fde8414cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8a07c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde8415dfe0 0x7fde8414cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59804/0 (socket says 172.21.15.59:59804) 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8a07c640 1 -- 172.21.15.59:0/2698435986 learned_addr learned my addr 172.21.15.59:0/2698435986 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8987b640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde840cf830 0x7fde840cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8907a640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde840d29c0 0x7fde8414ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8a07c640 1 -- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde840d29c0 msgr2=0x7fde8414ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8a07c640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde840d29c0 0x7fde8414ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8a07c640 1 -- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde840cf830 msgr2=0x7fde840cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8a07c640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde840cf830 0x7fde840cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8a07c640 1 -- 172.21.15.59:0/2698435986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8414b970 con 0x7fde8415dfe0 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8a07c640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde8415dfe0 0x7fde8414cf50 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fde7400efc0 tx=0x7fde7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde827fc640 1 -- 172.21.15.59:0/2698435986 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74018020 con 0x7fde8415dfe0 2024-03-06T07:33:30.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde8badf640 1 -- 172.21.15.59:0/2698435986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8414bc50 con 0x7fde8415dfe0 2024-03-06T07:33:30.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde827fc640 1 -- 172.21.15.59:0/2698435986 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde74014480 con 0x7fde8415dfe0 2024-03-06T07:33:30.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.325+0000 7fde827fc640 1 -- 172.21.15.59:0/2698435986 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74014870 con 0x7fde8415dfe0 2024-03-06T07:33:30.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.326+0000 7fde8badf640 1 -- 172.21.15.59:0/2698435986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8414c210 con 0x7fde8415dfe0 2024-03-06T07:33:30.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.326+0000 7fde8badf640 1 -- 172.21.15.59:0/2698435986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde840d3740 con 0x7fde8415dfe0 2024-03-06T07:33:30.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.327+0000 7fde827fc640 1 -- 172.21.15.59:0/2698435986 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde74005d10 con 0x7fde8415dfe0 2024-03-06T07:33:30.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.328+0000 7fde827fc640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde7006b4d0 0x7fde7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.328+0000 7fde827fc640 1 -- 172.21.15.59:0/2698435986 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fde7408acb0 con 0x7fde8415dfe0 2024-03-06T07:33:30.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.328+0000 7fde8987b640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde7006b4d0 0x7fde7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.329+0000 7fde8987b640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde7006b4d0 0x7fde7006d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fde7c036000 tx=0x7fde7c03b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.332+0000 7fde827fc640 1 -- 172.21.15.59:0/2698435986 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde840d3740 con 0x7fde8415dfe0 2024-03-06T07:33:30.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.549+0000 7fde8badf640 1 -- 172.21.15.59:0/2698435986 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde8406c780 con 0x7fde8415dfe0 2024-03-06T07:33:30.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.550+0000 7fde827fc640 1 -- 172.21.15.59:0/2698435986 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fde74022070 con 0x7fde8415dfe0 2024-03-06T07:33:30.552 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:30.552 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:30.552 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:30.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 -- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde7006b4d0 msgr2=0x7fde7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde7006b4d0 0x7fde7006d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fde7c036000 tx=0x7fde7c03b040 comp rx=0 tx=0).stop 2024-03-06T07:33:30.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 -- 172.21.15.59:0/2698435986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde8415dfe0 msgr2=0x7fde8414cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde8415dfe0 0x7fde8414cf50 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fde7400efc0 tx=0x7fde7400c490 comp rx=0 tx=0).stop 2024-03-06T07:33:30.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 -- 172.21.15.59:0/2698435986 shutdown_connections 2024-03-06T07:33:30.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde7006b4d0 0x7fde7006d990 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde8415dfe0 0x7fde8414cf50 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde840d29c0 0x7fde8414ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 --2- 172.21.15.59:0/2698435986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde840cf830 0x7fde840cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 -- 172.21.15.59:0/2698435986 >> 172.21.15.59:0/2698435986 conn(0x7fde840d3cd0 msgr2=0x7fde840b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 -- 172.21.15.59:0/2698435986 shutdown_connections 2024-03-06T07:33:30.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:30.552+0000 7fde8badf640 1 -- 172.21.15.59:0/2698435986 wait complete. 2024-03-06T07:33:30.766 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:30.766 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:31.768 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:31.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.867+0000 7fd39b3b5640 1 Processor -- start 2024-03-06T07:33:31.869 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.868+0000 7fd39b3b5640 1 -- start start 2024-03-06T07:33:31.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.868+0000 7fd39b3b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd3940cf850 0x7fd3940cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.868+0000 7fd39b3b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 0x7fd39406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.868+0000 7fd39b3b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd39415e380 con 0x7fd3940d00c0 2024-03-06T07:33:31.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.868+0000 7fd39b3b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd39406ca90 con 0x7fd3940cf850 2024-03-06T07:33:31.870 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.868+0000 7fd39b3b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3940cdc10 con 0x7fd39406cf20 2024-03-06T07:33:31.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.869+0000 7fd399952640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 0x7fd39406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.871 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.869+0000 7fd398950640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd3940cf850 0x7fd3940cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.869+0000 7fd399952640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 0x7fd39406d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.59:44858/0 (socket says 172.21.15.59:44858) 2024-03-06T07:33:31.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.869+0000 7fd399952640 1 -- 172.21.15.59:0/1568201057 learned_addr learned my addr 172.21.15.59:0/1568201057 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:31.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.869+0000 7fd399952640 1 -- 172.21.15.59:0/1568201057 >> v2:172.21.15.164:3300/0 conn(0x7fd3940cf850 msgr2=0x7fd3940cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.869+0000 7fd399952640 1 --2- 172.21.15.59:0/1568201057 >> v2:172.21.15.164:3300/0 conn(0x7fd3940cf850 0x7fd3940cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.869+0000 7fd399952640 1 -- 172.21.15.59:0/1568201057 >> v1:172.21.15.164:6789/0 conn(0x7fd3940d00c0 legacy=0x7fd3940d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.869+0000 7fd399952640 1 -- 172.21.15.59:0/1568201057 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39406c160 con 0x7fd39406cf20 2024-03-06T07:33:31.872 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.869+0000 7fd398950640 1 --2- 172.21.15.59:0/1568201057 >> v2:172.21.15.164:3300/0 conn(0x7fd3940cf850 0x7fd3940cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:31.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.870+0000 7fd399952640 1 --2- 172.21.15.59:0/1568201057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 0x7fd39406d3b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009870 tx=0x7fd38c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=413913868f7ec1e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.870+0000 7fd38bfff640 1 -- 172.21.15.59:0/1568201057 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c034070 con 0x7fd39406cf20 2024-03-06T07:33:31.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.870+0000 7fd38bfff640 1 -- 172.21.15.59:0/1568201057 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd38c030840 con 0x7fd39406cf20 2024-03-06T07:33:31.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.870+0000 7fd38bfff640 1 -- 172.21.15.59:0/1568201057 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c030c10 con 0x7fd39406cf20 2024-03-06T07:33:31.873 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.871+0000 7fd39b3b5640 1 -- 172.21.15.59:0/1568201057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 msgr2=0x7fd39406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.871+0000 7fd39b3b5640 1 --2- 172.21.15.59:0/1568201057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 0x7fd39406d3b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009870 tx=0x7fd38c02b980 comp rx=0 tx=0).stop 2024-03-06T07:33:31.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.871+0000 7fd39b3b5640 1 -- 172.21.15.59:0/1568201057 shutdown_connections 2024-03-06T07:33:31.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.871+0000 7fd39b3b5640 1 --2- 172.21.15.59:0/1568201057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 0x7fd39406d3b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.871+0000 7fd39b3b5640 1 --2- 172.21.15.59:0/1568201057 >> v2:172.21.15.164:3300/0 conn(0x7fd3940cf850 0x7fd3940cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.871+0000 7fd39b3b5640 1 -- 172.21.15.59:0/1568201057 >> 172.21.15.59:0/1568201057 conn(0x7fd3940b35a0 msgr2=0x7fd3940b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.871+0000 7fd39b3b5640 1 -- 172.21.15.59:0/1568201057 shutdown_connections 2024-03-06T07:33:31.874 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.871+0000 7fd39b3b5640 1 -- 172.21.15.59:0/1568201057 wait complete. 2024-03-06T07:33:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.872+0000 7fd39b3b5640 1 Processor -- start 2024-03-06T07:33:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.872+0000 7fd39b3b5640 1 -- start start 2024-03-06T07:33:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd39b3b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 0x7fd39415d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd39b3b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3940cf850 0x7fd3941600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd39b3b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3940d00c0 0x7fd39415e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd39b3b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3940cd900 con 0x7fd3940cf850 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd39b3b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39406c720 con 0x7fd3940d00c0 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd39b3b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3940b1030 con 0x7fd39406cf20 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd398950640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3940cf850 0x7fd3941600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd398950640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3940cf850 0x7fd3941600b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59816/0 (socket says 172.21.15.59:59816) 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd398950640 1 -- 172.21.15.59:0/3013500070 learned_addr learned my addr 172.21.15.59:0/3013500070 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd399151640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 0x7fd39415d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd399952640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3940d00c0 0x7fd39415e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd398950640 1 -- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 msgr2=0x7fd39415d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd398950640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 0x7fd39415d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd398950640 1 -- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3940d00c0 msgr2=0x7fd39415e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd398950640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3940d00c0 0x7fd39415e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd398950640 1 -- 172.21.15.59:0/3013500070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39415f010 con 0x7fd3940cf850 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd398950640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3940cf850 0x7fd3941600b0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fd38400ebe0 tx=0x7fd38400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.873+0000 7fd389ffb640 1 -- 172.21.15.59:0/3013500070 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd384017070 con 0x7fd3940cf850 2024-03-06T07:33:31.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.874+0000 7fd389ffb640 1 -- 172.21.15.59:0/3013500070 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd38400cbd0 con 0x7fd3940cf850 2024-03-06T07:33:31.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.874+0000 7fd39b3b5640 1 -- 172.21.15.59:0/3013500070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39415f350 con 0x7fd3940cf850 2024-03-06T07:33:31.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.874+0000 7fd389ffb640 1 -- 172.21.15.59:0/3013500070 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd384013470 con 0x7fd3940cf850 2024-03-06T07:33:31.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.874+0000 7fd39b3b5640 1 -- 172.21.15.59:0/3013500070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39415fb90 con 0x7fd3940cf850 2024-03-06T07:33:31.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.875+0000 7fd39b3b5640 1 -- 172.21.15.59:0/3013500070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3940b1370 con 0x7fd3940cf850 2024-03-06T07:33:31.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.875+0000 7fd389ffb640 1 -- 172.21.15.59:0/3013500070 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd38401a070 con 0x7fd3940cf850 2024-03-06T07:33:31.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.876+0000 7fd389ffb640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd37806b500 0x7fd37806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.876+0000 7fd389ffb640 1 -- 172.21.15.59:0/3013500070 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd38408a7c0 con 0x7fd3940cf850 2024-03-06T07:33:31.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.876+0000 7fd399151640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd37806b500 0x7fd37806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.877+0000 7fd399151640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd37806b500 0x7fd37806d9c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fd390001e70 tx=0x7fd390009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:31.879+0000 7fd389ffb640 1 -- 172.21.15.59:0/3013500070 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3940b1370 con 0x7fd3940cf850 2024-03-06T07:33:32.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.099+0000 7fd39b3b5640 1 -- 172.21.15.59:0/3013500070 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd39415f530 con 0x7fd3940cf850 2024-03-06T07:33:32.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.099+0000 7fd389ffb640 1 -- 172.21.15.59:0/3013500070 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fd39415f350 con 0x7fd3940cf850 2024-03-06T07:33:32.121 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:32.121 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 -- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd37806b500 msgr2=0x7fd37806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd37806b500 0x7fd37806d9c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fd390001e70 tx=0x7fd390009040 comp rx=0 tx=0).stop 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 -- 172.21.15.59:0/3013500070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3940cf850 msgr2=0x7fd3941600b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3940cf850 0x7fd3941600b0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fd38400ebe0 tx=0x7fd38400c520 comp rx=0 tx=0).stop 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 -- 172.21.15.59:0/3013500070 shutdown_connections 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd37806b500 0x7fd37806d9c0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3940d00c0 0x7fd39415e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd3940cf850 0x7fd3941600b0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 --2- 172.21.15.59:0/3013500070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd39406cf20 0x7fd39415d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 -- 172.21.15.59:0/3013500070 >> 172.21.15.59:0/3013500070 conn(0x7fd3940b35a0 msgr2=0x7fd39406d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 -- 172.21.15.59:0/3013500070 shutdown_connections 2024-03-06T07:33:32.124 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:32.101+0000 7fd39b3b5640 1 -- 172.21.15.59:0/3013500070 wait complete. 2024-03-06T07:33:32.326 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:32.326 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:33.327 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:33.428 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.427+0000 7ff428fbc640 1 Processor -- start 2024-03-06T07:33:33.429 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.427+0000 7ff428fbc640 1 -- start start 2024-03-06T07:33:33.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.427+0000 7ff428fbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff42415dfe0 0x7ff4240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.427+0000 7ff428fbc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff4240d29c0 0x7ff4240b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.431 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.427+0000 7ff428fbc640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4240b3430 con 0x7ff4240cf830 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.427+0000 7ff428fbc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4240b31e0 con 0x7ff4240d29c0 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.427+0000 7ff428fbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240b3d60 con 0x7ff42415dfe0 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff41bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff4240cf830 0x7ff4240d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35966/0 (socket says 172.21.15.59:35966) 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff41bfff640 1 -- 172.21.15.59:0/3997271665 learned_addr learned my addr 172.21.15.59:0/3997271665 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff42259c640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff42415dfe0 0x7ff4240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff422d9d640 1 --2- 172.21.15.59:0/3997271665 >> v2:172.21.15.164:3300/0 conn(0x7ff4240d29c0 0x7ff4240b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308761115 0 0) 0x7ff4240b3430 con 0x7ff4240cf830 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff410003520 con 0x7ff4240cf830 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 425675193 0 0) 0x7ff410003520 con 0x7ff4240cf830 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff42415dfe0 msgr2=0x7ff4240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff42415dfe0 0x7ff4240cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 >> v2:172.21.15.164:3300/0 conn(0x7ff4240d29c0 msgr2=0x7ff4240b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 --2- 172.21.15.59:0/3997271665 >> v2:172.21.15.164:3300/0 conn(0x7ff4240d29c0 0x7ff4240b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4240b3710 con 0x7ff4240cf830 2024-03-06T07:33:33.432 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7ff40c001670 con 0x7ff4240cf830 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff422d9d640 1 --2- 172.21.15.59:0/3997271665 >> v2:172.21.15.164:3300/0 conn(0x7ff4240d29c0 0x7ff4240b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 >> v1:172.21.15.59:6789/0 conn(0x7ff4240cf830 legacy=0x7ff4240d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 0x7ff410003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff42415dfe0 0x7ff410004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff410004d40 0x7ff4100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240b31e0 con 0x7ff4100039b0 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240b3d60 con 0x7ff410004d40 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41000b9c0 con 0x7ff42415dfe0 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.428+0000 7ff42259c640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 0x7ff410003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.429+0000 7ff41bfff640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff42415dfe0 0x7ff410004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.429+0000 7ff422d9d640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff410004d40 0x7ff4100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.429+0000 7ff42259c640 1 -- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff42415dfe0 msgr2=0x7ff410004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.429+0000 7ff42259c640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff42415dfe0 0x7ff410004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.429+0000 7ff42259c640 1 -- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff410004d40 msgr2=0x7ff4100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.433 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.429+0000 7ff42259c640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff410004d40 0x7ff4100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.429+0000 7ff42259c640 1 -- 172.21.15.59:0/3997271665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff410003520 con 0x7ff4100039b0 2024-03-06T07:33:33.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.429+0000 7ff422d9d640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff410004d40 0x7ff4100051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:33.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.429+0000 7ff41bfff640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff42415dfe0 0x7ff410004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:33.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.429+0000 7ff42259c640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 0x7ff410003dd0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7ff414009ac0 tx=0x7ff410005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0acca2dfd6bc44e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.430+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414039070 con 0x7ff4100039b0 2024-03-06T07:33:33.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.430+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41402c6b0 con 0x7ff4100039b0 2024-03-06T07:33:33.434 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.430+0000 7ff421d9b640 1 -- 172.21.15.59:0/3997271665 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414009690 con 0x7ff4100039b0 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.430+0000 7ff428fbc640 1 -- 172.21.15.59:0/3997271665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 msgr2=0x7ff410003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.430+0000 7ff428fbc640 1 --2- 172.21.15.59:0/3997271665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 0x7ff410003dd0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7ff414009ac0 tx=0x7ff410005830 comp rx=0 tx=0).stop 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.430+0000 7ff422d9d640 1 -- 172.21.15.59:0/3997271665 reap_dead start 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.430+0000 7ff428fbc640 1 -- 172.21.15.59:0/3997271665 shutdown_connections 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.430+0000 7ff428fbc640 1 -- 172.21.15.59:0/3997271665 >> 172.21.15.59:0/3997271665 conn(0x7ff4240d3cd0 msgr2=0x7ff4240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.430+0000 7ff428fbc640 1 -- 172.21.15.59:0/3997271665 shutdown_connections 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.430+0000 7ff428fbc640 1 -- 172.21.15.59:0/3997271665 wait complete. 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.432+0000 7ff428fbc640 1 Processor -- start 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.432+0000 7ff428fbc640 1 -- start start 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.433+0000 7ff428fbc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 0x7ff42415c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.433+0000 7ff428fbc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff42415f0d0 0x7ff424160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.433+0000 7ff428fbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4240ccab0 0x7ff4240ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.433+0000 7ff428fbc640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240d1df0 con 0x7ff4100039b0 2024-03-06T07:33:33.437 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.433+0000 7ff428fbc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240d1c70 con 0x7ff42415f0d0 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.433+0000 7ff428fbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42415e3c0 con 0x7ff4240ccab0 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.433+0000 7ff42259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 0x7ff42415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff42259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 0x7ff42415c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59840/0 (socket says 172.21.15.59:59840) 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff42259c640 1 -- 172.21.15.59:0/46693556 learned_addr learned my addr 172.21.15.59:0/46693556 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff422d9d640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4240ccab0 0x7ff4240ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff41bfff640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff42415f0d0 0x7ff424160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff42259c640 1 -- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4240ccab0 msgr2=0x7ff4240ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff42259c640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4240ccab0 0x7ff4240ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff42259c640 1 -- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff42415f0d0 msgr2=0x7ff424160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff42259c640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff42415f0d0 0x7ff424160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff42259c640 1 -- 172.21.15.59:0/46693556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42415c780 con 0x7ff4100039b0 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff422d9d640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4240ccab0 0x7ff4240ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff41bfff640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff42415f0d0 0x7ff424160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff42259c640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 0x7ff42415c240 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7ff414000c00 tx=0x7ff414004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.438 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff41affd640 1 -- 172.21.15.59:0/46693556 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414042020 con 0x7ff4100039b0 2024-03-06T07:33:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff41affd640 1 -- 172.21.15.59:0/46693556 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41403a040 con 0x7ff4100039b0 2024-03-06T07:33:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff41affd640 1 -- 172.21.15.59:0/46693556 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414039070 con 0x7ff4100039b0 2024-03-06T07:33:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff428fbc640 1 -- 172.21.15.59:0/46693556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42415f5b0 con 0x7ff4100039b0 2024-03-06T07:33:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.434+0000 7ff428fbc640 1 -- 172.21.15.59:0/46693556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42415fb70 con 0x7ff4100039b0 2024-03-06T07:33:33.439 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.435+0000 7ff428fbc640 1 -- 172.21.15.59:0/46693556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42404f330 con 0x7ff4100039b0 2024-03-06T07:33:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.436+0000 7ff41affd640 1 -- 172.21.15.59:0/46693556 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff414045020 con 0x7ff4100039b0 2024-03-06T07:33:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.436+0000 7ff41affd640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff40406b500 0x7ff40406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.436+0000 7ff41affd640 1 -- 172.21.15.59:0/46693556 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff4140ab910 con 0x7ff4100039b0 2024-03-06T07:33:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.436+0000 7ff41bfff640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff40406b500 0x7ff40406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.440 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.437+0000 7ff41bfff640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff40406b500 0x7ff40406d9c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff4241f6f40 tx=0x7ff40c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.441+0000 7ff41affd640 1 -- 172.21.15.59:0/46693556 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff41404c030 con 0x7ff4100039b0 2024-03-06T07:33:33.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.660+0000 7ff428fbc640 1 -- 172.21.15.59:0/46693556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4240b26e0 con 0x7ff4100039b0 2024-03-06T07:33:33.662 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.660+0000 7ff41affd640 1 -- 172.21.15.59:0/46693556 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7ff41407a800 con 0x7ff4100039b0 2024-03-06T07:33:33.662 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:33.663 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:33.663 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 -- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff40406b500 msgr2=0x7ff40406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff40406b500 0x7ff40406d9c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff4241f6f40 tx=0x7ff40c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 -- 172.21.15.59:0/46693556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 msgr2=0x7ff42415c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 0x7ff42415c240 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7ff414000c00 tx=0x7ff414004730 comp rx=0 tx=0).stop 2024-03-06T07:33:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 -- 172.21.15.59:0/46693556 shutdown_connections 2024-03-06T07:33:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff40406b500 0x7ff40406d9c0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4240ccab0 0x7ff4240ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff42415f0d0 0x7ff424160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 --2- 172.21.15.59:0/46693556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4100039b0 0x7ff42415c240 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 -- 172.21.15.59:0/46693556 >> 172.21.15.59:0/46693556 conn(0x7ff4240d3cd0 msgr2=0x7ff42406bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 -- 172.21.15.59:0/46693556 shutdown_connections 2024-03-06T07:33:33.666 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:33.663+0000 7ff428fbc640 1 -- 172.21.15.59:0/46693556 wait complete. 2024-03-06T07:33:33.877 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:33.877 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:34.878 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.978+0000 7fdbac365640 1 Processor -- start 2024-03-06T07:33:34.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.978+0000 7fdbac365640 1 -- start start 2024-03-06T07:33:34.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdbac365640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdba40d22b0 0x7fdba40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdbac365640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba40b41f0 0x7fdba40b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdbac365640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40b3430 con 0x7fdba415dfe0 2024-03-06T07:33:34.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdbac365640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba40d2c50 con 0x7fdba40d22b0 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdbac365640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40b31e0 con 0x7fdba40b41f0 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdbaa101640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdba415dfe0 0x7fdba40cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57048/0 (socket says 172.21.15.59:57048) 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdbaa101640 1 -- 172.21.15.59:0/1597566757 learned_addr learned my addr 172.21.15.59:0/1597566757 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdbaa902640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba40b41f0 0x7fdba40b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489589074 0 0) 0x7fdba40b3430 con 0x7fdba415dfe0 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba9900640 1 --2- 172.21.15.59:0/1597566757 >> v2:172.21.15.164:3300/0 conn(0x7fdba40d22b0 0x7fdba40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb90003520 con 0x7fdba415dfe0 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 860234168 0 0) 0x7fdb90003520 con 0x7fdba415dfe0 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba40b41f0 msgr2=0x7fdba40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba40b41f0 0x7fdba40b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 >> v2:172.21.15.164:3300/0 conn(0x7fdba40d22b0 msgr2=0x7fdba40d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 --2- 172.21.15.59:0/1597566757 >> v2:172.21.15.164:3300/0 conn(0x7fdba40d22b0 0x7fdba40d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.984 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba40b3710 con 0x7fdba415dfe0 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fdba0001670 con 0x7fdba415dfe0 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 >> v1:172.21.15.59:6789/0 conn(0x7fdba415dfe0 legacy=0x7fdba40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba40b41f0 0x7fdb900039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb90003f30 0x7fdb90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.979+0000 7fdba90ff640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 0x7fdb900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.980+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40b31e0 con 0x7fdb90004d40 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.980+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9000b9c0 con 0x7fdb90003f30 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.980+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9000bb40 con 0x7fdba40b41f0 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.980+0000 7fdbaa902640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 0x7fdb900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.980+0000 7fdbaa101640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba40b41f0 0x7fdb900039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.980+0000 7fdba9900640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb90003f30 0x7fdb90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.980+0000 7fdbaa902640 1 -- 172.21.15.59:0/1597566757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba40b41f0 msgr2=0x7fdb900039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.980+0000 7fdbaa902640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba40b41f0 0x7fdb900039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.985 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.980+0000 7fdbaa902640 1 -- 172.21.15.59:0/1597566757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb90003f30 msgr2=0x7fdb90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.980+0000 7fdbaa902640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb90003f30 0x7fdb90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.980+0000 7fdbaa902640 1 -- 172.21.15.59:0/1597566757 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb90003520 con 0x7fdb90004d40 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.981+0000 7fdbaa902640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 0x7fdb900051c0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009fc0 tx=0x7fdb90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cbba67b347065dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.981+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c039070 con 0x7fdb90004d40 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.981+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c02cc00 con 0x7fdb90004d40 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.981+0000 7fdba90ff640 1 -- 172.21.15.59:0/1597566757 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c009c10 con 0x7fdb90004d40 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.981+0000 7fdbac365640 1 -- 172.21.15.59:0/1597566757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 msgr2=0x7fdb900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.981+0000 7fdbac365640 1 --2- 172.21.15.59:0/1597566757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 0x7fdb900051c0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009fc0 tx=0x7fdb90005830 comp rx=0 tx=0).stop 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.981+0000 7fdbaa902640 1 -- 172.21.15.59:0/1597566757 reap_dead start 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.981+0000 7fdbac365640 1 -- 172.21.15.59:0/1597566757 shutdown_connections 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.981+0000 7fdbac365640 1 -- 172.21.15.59:0/1597566757 >> 172.21.15.59:0/1597566757 conn(0x7fdba40d3cd0 msgr2=0x7fdba40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.981+0000 7fdbac365640 1 -- 172.21.15.59:0/1597566757 shutdown_connections 2024-03-06T07:33:34.986 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.981+0000 7fdbac365640 1 -- 172.21.15.59:0/1597566757 wait complete. 2024-03-06T07:33:34.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbac365640 1 Processor -- start 2024-03-06T07:33:34.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbac365640 1 -- start start 2024-03-06T07:33:34.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbac365640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 0x7fdba41595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbac365640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba41e6770 0x7fdba41e7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbac365640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdba415a3d0 0x7fdba41e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbac365640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40cfa00 con 0x7fdb90004d40 2024-03-06T07:33:34.989 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbac365640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40b3d60 con 0x7fdba415a3d0 2024-03-06T07:33:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbac365640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40d1c70 con 0x7fdba41e6770 2024-03-06T07:33:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbaa101640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 0x7fdba41595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbaa101640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 0x7fdba41595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54948/0 (socket says 172.21.15.59:54948) 2024-03-06T07:33:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbaa101640 1 -- 172.21.15.59:0/2895895036 learned_addr learned my addr 172.21.15.59:0/2895895036 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdba9900640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba41e6770 0x7fdba41e7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.983+0000 7fdbaa902640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdba415a3d0 0x7fdba41e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdbaa101640 1 -- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba41e6770 msgr2=0x7fdba41e7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdbaa101640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba41e6770 0x7fdba41e7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdbaa101640 1 -- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdba415a3d0 msgr2=0x7fdba41e81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdbaa101640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdba415a3d0 0x7fdba41e81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.990 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdbaa101640 1 -- 172.21.15.59:0/2895895036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba41e6c00 con 0x7fdb90004d40 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdba9900640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba41e6770 0x7fdba41e7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdbaa101640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 0x7fdba41595e0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fdba0007310 tx=0x7fdba00077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdb9affd640 1 -- 172.21.15.59:0/2895895036 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba000fa80 con 0x7fdb90004d40 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdb9affd640 1 -- 172.21.15.59:0/2895895036 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba0018070 con 0x7fdb90004d40 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdbac365640 1 -- 172.21.15.59:0/2895895036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba41e6ee0 con 0x7fdb90004d40 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdb9affd640 1 -- 172.21.15.59:0/2895895036 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba000fe80 con 0x7fdb90004d40 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.984+0000 7fdbac365640 1 -- 172.21.15.59:0/2895895036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba41e7780 con 0x7fdb90004d40 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.985+0000 7fdbac365640 1 -- 172.21.15.59:0/2895895036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba41e72d0 con 0x7fdb90004d40 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.985+0000 7fdb9affd640 1 -- 172.21.15.59:0/2895895036 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdba001d460 con 0x7fdb90004d40 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.986+0000 7fdb9affd640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb8806b4d0 0x7fdb8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.986+0000 7fdb9affd640 1 -- 172.21.15.59:0/2895895036 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fdba008a480 con 0x7fdb90004d40 2024-03-06T07:33:34.991 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.986+0000 7fdba9900640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb8806b4d0 0x7fdb8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.987+0000 7fdba9900640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb8806b4d0 0x7fdb8806d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fdba41f7370 tx=0x7fdb94008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.992 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:34.990+0000 7fdb9affd640 1 -- 172.21.15.59:0/2895895036 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdba00592e0 con 0x7fdb90004d40 2024-03-06T07:33:35.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.211+0000 7fdbac365640 1 -- 172.21.15.59:0/2895895036 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdba41e7980 con 0x7fdb90004d40 2024-03-06T07:33:35.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.211+0000 7fdb9affd640 1 -- 172.21.15.59:0/2895895036 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fdba0058be0 con 0x7fdb90004d40 2024-03-06T07:33:35.214 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:35.214 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:35.214 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:35.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 -- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb8806b4d0 msgr2=0x7fdb8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb8806b4d0 0x7fdb8806d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fdba41f7370 tx=0x7fdb94008040 comp rx=0 tx=0).stop 2024-03-06T07:33:35.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 -- 172.21.15.59:0/2895895036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 msgr2=0x7fdba41595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 0x7fdba41595e0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fdba0007310 tx=0x7fdba00077d0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 -- 172.21.15.59:0/2895895036 shutdown_connections 2024-03-06T07:33:35.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdb8806b4d0 0x7fdb8806d990 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdba415a3d0 0x7fdba41e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdba41e6770 0x7fdba41e7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 --2- 172.21.15.59:0/2895895036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb90004d40 0x7fdba41595e0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 -- 172.21.15.59:0/2895895036 >> 172.21.15.59:0/2895895036 conn(0x7fdba40d3cd0 msgr2=0x7fdba415d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 -- 172.21.15.59:0/2895895036 shutdown_connections 2024-03-06T07:33:35.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:35.214+0000 7fdbac365640 1 -- 172.21.15.59:0/2895895036 wait complete. 2024-03-06T07:33:35.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:35.428 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:36.428 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.528+0000 7fe5bb68f640 1 Processor -- start 2024-03-06T07:33:36.530 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.528+0000 7fe5bb68f640 1 -- start start 2024-03-06T07:33:36.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.529+0000 7fe5bb68f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe5b415dfe0 0x7fe5b40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.529+0000 7fe5bb68f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b40b3430 con 0x7fe5b40d29c0 2024-03-06T07:33:36.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.529+0000 7fe5bb68f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b40b3710 con 0x7fe5b40cf830 2024-03-06T07:33:36.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.529+0000 7fe5bb68f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b40b31e0 con 0x7fe5b415dfe0 2024-03-06T07:33:36.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.529+0000 7fe5b9c2c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe5b40d29c0 0x7fe5b40b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57054/0 (socket says 172.21.15.59:57054) 2024-03-06T07:33:36.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.529+0000 7fe5b9c2c640 1 -- 172.21.15.59:0/3232825973 learned_addr learned my addr 172.21.15.59:0/3232825973 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:36.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.529+0000 7fe5b942b640 1 --2- 172.21.15.59:0/3232825973 >> v2:172.21.15.164:3300/0 conn(0x7fe5b415dfe0 0x7fe5b40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.535 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.529+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693090725 0 0) 0x7fe5b40b3430 con 0x7fe5b40d29c0 2024-03-06T07:33:36.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.529+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe59c003520 con 0x7fe5b40d29c0 2024-03-06T07:33:36.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3310982168 0 0) 0x7fe59c003520 con 0x7fe5b40d29c0 2024-03-06T07:33:36.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 >> v2:172.21.15.164:3300/0 conn(0x7fe5b415dfe0 msgr2=0x7fe5b40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/3232825973 >> v2:172.21.15.164:3300/0 conn(0x7fe5b415dfe0 0x7fe5b40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 >> v1:172.21.15.164:6789/0 conn(0x7fe5b40cf830 legacy=0x7fe5b40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b40b39f0 con 0x7fe5b40d29c0 2024-03-06T07:33:36.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fe5a4001670 con 0x7fe5b40d29c0 2024-03-06T07:33:36.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 >> v1:172.21.15.59:6789/0 conn(0x7fe5b40d29c0 legacy=0x7fe5b40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/3232825973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe59c0039b0 0x7fe59c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/3232825973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0043d0 0x7fe59c0048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 --2- 172.21.15.59:0/3232825973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c0050a0 0x7fe59c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b40b31e0 con 0x7fe59c0043d0 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c00bfe0 con 0x7fe59c0050a0 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c00c1e0 con 0x7fe59c0039b0 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b8c2a640 1 --2- 172.21.15.59:0/3232825973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0043d0 0x7fe59c0048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b942b640 1 --2- 172.21.15.59:0/3232825973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe59c0039b0 0x7fe59c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b9c2c640 1 --2- 172.21.15.59:0/3232825973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c0050a0 0x7fe59c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b8c2a640 1 -- 172.21.15.59:0/3232825973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe59c0039b0 msgr2=0x7fe59c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b8c2a640 1 --2- 172.21.15.59:0/3232825973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe59c0039b0 0x7fe59c003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b8c2a640 1 -- 172.21.15.59:0/3232825973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c0050a0 msgr2=0x7fe59c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b8c2a640 1 --2- 172.21.15.59:0/3232825973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c0050a0 0x7fe59c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b8c2a640 1 -- 172.21.15.59:0/3232825973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c003520 con 0x7fe59c0043d0 2024-03-06T07:33:36.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.530+0000 7fe5b9c2c640 1 -- 172.21.15.59:0/3232825973 reap_dead start 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.531+0000 7fe5b8c2a640 1 --2- 172.21.15.59:0/3232825973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0043d0 0x7fe59c0048a0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8001830 tx=0x7fe59c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b47f53a5549ea4ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.531+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8001430 con 0x7fe59c0043d0 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.531+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a8032620 con 0x7fe59c0043d0 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.531+0000 7fe5b3fff640 1 -- 172.21.15.59:0/3232825973 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8032a10 con 0x7fe59c0043d0 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.532+0000 7fe5bb68f640 1 -- 172.21.15.59:0/3232825973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0043d0 msgr2=0x7fe59c0048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.532+0000 7fe5bb68f640 1 --2- 172.21.15.59:0/3232825973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0043d0 0x7fe59c0048a0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8001830 tx=0x7fe59c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.532+0000 7fe5bb68f640 1 -- 172.21.15.59:0/3232825973 shutdown_connections 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.532+0000 7fe5bb68f640 1 --2- 172.21.15.59:0/3232825973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe59c0043d0 0x7fe59c0048a0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.532+0000 7fe5bb68f640 1 -- 172.21.15.59:0/3232825973 >> 172.21.15.59:0/3232825973 conn(0x7fe5b40d3cd0 msgr2=0x7fe5b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.532+0000 7fe5bb68f640 1 -- 172.21.15.59:0/3232825973 shutdown_connections 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.532+0000 7fe5bb68f640 1 -- 172.21.15.59:0/3232825973 wait complete. 2024-03-06T07:33:36.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.533+0000 7fe5bb68f640 1 Processor -- start 2024-03-06T07:33:36.539 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.533+0000 7fe5bb68f640 1 -- start start 2024-03-06T07:33:36.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5bb68f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c0043d0 0x7fe5b4152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5bb68f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b40d40b0 0x7fe5b40d4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5bb68f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cba30 0x7fe5b40cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5bb68f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b415e3c0 con 0x7fe5b40d40b0 2024-03-06T07:33:36.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5bb68f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b40b3d60 con 0x7fe59c0043d0 2024-03-06T07:33:36.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5bb68f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b415e540 con 0x7fe5b40cba30 2024-03-06T07:33:36.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b8c2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b40d40b0 0x7fe5b40d4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b8c2a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b40d40b0 0x7fe5b40d4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54968/0 (socket says 172.21.15.59:54968) 2024-03-06T07:33:36.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b8c2a640 1 -- 172.21.15.59:0/4046491016 learned_addr learned my addr 172.21.15.59:0/4046491016 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:36.542 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b9c2c640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cba30 0x7fe5b40cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b942b640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c0043d0 0x7fe5b4152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b8c2a640 1 -- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cba30 msgr2=0x7fe5b40cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b8c2a640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cba30 0x7fe5b40cbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b8c2a640 1 -- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c0043d0 msgr2=0x7fe5b4152d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b8c2a640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c0043d0 0x7fe5b4152d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b8c2a640 1 -- 172.21.15.59:0/4046491016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b40b86a0 con 0x7fe5b40d40b0 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b942b640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c0043d0 0x7fe5b4152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.534+0000 7fe5b8c2a640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b40d40b0 0x7fe5b40d4530 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8001950 tx=0x7fe5a8001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.535+0000 7fe5b1ffb640 1 -- 172.21.15.59:0/4046491016 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8007220 con 0x7fe5b40d40b0 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.535+0000 7fe5b1ffb640 1 -- 172.21.15.59:0/4046491016 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a80073c0 con 0x7fe5b40d40b0 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.535+0000 7fe5bb68f640 1 -- 172.21.15.59:0/4046491016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b40b8940 con 0x7fe5b40d40b0 2024-03-06T07:33:36.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.535+0000 7fe5b1ffb640 1 -- 172.21.15.59:0/4046491016 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a80067d0 con 0x7fe5b40d40b0 2024-03-06T07:33:36.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.535+0000 7fe5bb68f640 1 -- 172.21.15.59:0/4046491016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b40b8f00 con 0x7fe5b40d40b0 2024-03-06T07:33:36.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.536+0000 7fe5bb68f640 1 -- 172.21.15.59:0/4046491016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b404f330 con 0x7fe5b40d40b0 2024-03-06T07:33:36.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.537+0000 7fe5b1ffb640 1 -- 172.21.15.59:0/4046491016 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5a8044030 con 0x7fe5b40d40b0 2024-03-06T07:33:36.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.537+0000 7fe5b1ffb640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 0x7fe5a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:36.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.537+0000 7fe5b1ffb640 1 -- 172.21.15.59:0/4046491016 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe5a80abd90 con 0x7fe5b40d40b0 2024-03-06T07:33:36.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.538+0000 7fe5b942b640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 0x7fe5a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:36.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.538+0000 7fe5b942b640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 0x7fe5a006d9c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40cb030 tx=0x7fe5ac00b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:36.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.542+0000 7fe5b1ffb640 1 -- 172.21.15.59:0/4046491016 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5a807a5a0 con 0x7fe5b40d40b0 2024-03-06T07:33:36.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.762+0000 7fe5bb68f640 1 -- 172.21.15.59:0/4046491016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5b41e6e60 con 0x7fe5b40d40b0 2024-03-06T07:33:36.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.763+0000 7fe5b1ffb640 1 -- 172.21.15.59:0/4046491016 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fe5a807e640 con 0x7fe5b40d40b0 2024-03-06T07:33:36.765 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:36.765 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:36.765 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:36.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 -- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 msgr2=0x7fe5a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 0x7fe5a006d9c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40cb030 tx=0x7fe5ac00b040 comp rx=0 tx=0).stop 2024-03-06T07:33:36.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 -- 172.21.15.59:0/4046491016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b40d40b0 msgr2=0x7fe5b40d4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b40d40b0 0x7fe5b40d4530 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8001950 tx=0x7fe5a8001230 comp rx=0 tx=0).stop 2024-03-06T07:33:36.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 -- 172.21.15.59:0/4046491016 shutdown_connections 2024-03-06T07:33:36.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5a006b500 0x7fe5a006d9c0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5b40cba30 0x7fe5b40cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5b40d40b0 0x7fe5b40d4530 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 --2- 172.21.15.59:0/4046491016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59c0043d0 0x7fe5b4152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 -- 172.21.15.59:0/4046491016 >> 172.21.15.59:0/4046491016 conn(0x7fe5b40d3cd0 msgr2=0x7fe5b40b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 -- 172.21.15.59:0/4046491016 shutdown_connections 2024-03-06T07:33:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:36.765+0000 7fe5bb68f640 1 -- 172.21.15.59:0/4046491016 wait complete. 2024-03-06T07:33:36.979 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:36.979 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:37.980 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:38.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.081+0000 7f56d1602640 1 Processor -- start 2024-03-06T07:33:38.083 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.081+0000 7f56d1602640 1 -- start start 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56d1602640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f56cc072160 0x7f56cc0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56d1602640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc0ae6b0 0x7f56cc0aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56d1602640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc0b2a10 con 0x7f56cc15b160 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56d1602640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56cc0af050 con 0x7f56cc072160 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56d1602640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0b2cf0 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56cb7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f56cc15b160 0x7f56cc15b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57064/0 (socket says 172.21.15.59:57064) 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56cb7fe640 1 -- 172.21.15.59:0/1625195267 learned_addr learned my addr 172.21.15.59:0/1625195267 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56ca7fc640 1 --2- 172.21.15.59:0/1625195267 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc0ae6b0 0x7f56cc0aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56caffd640 1 --2- 172.21.15.59:0/1625195267 >> v2:172.21.15.164:3300/0 conn(0x7f56cc072160 0x7f56cc0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56c9ffb640 1 -- 172.21.15.59:0/1625195267 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697778048 0 0) 0x7f56cc0b2a10 con 0x7f56cc15b160 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56c9ffb640 1 -- 172.21.15.59:0/1625195267 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b8003520 con 0x7f56cc15b160 2024-03-06T07:33:38.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56ca7fc640 1 -- 172.21.15.59:0/1625195267 >> v2:172.21.15.164:3300/0 conn(0x7f56cc072160 msgr2=0x7f56cc0b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56ca7fc640 1 --2- 172.21.15.59:0/1625195267 >> v2:172.21.15.164:3300/0 conn(0x7f56cc072160 0x7f56cc0b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56ca7fc640 1 -- 172.21.15.59:0/1625195267 >> v1:172.21.15.59:6789/0 conn(0x7f56cc15b160 legacy=0x7f56cc15b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56ca7fc640 1 -- 172.21.15.59:0/1625195267 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc0b2fd0 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.082+0000 7f56caffd640 1 --2- 172.21.15.59:0/1625195267 >> v2:172.21.15.164:3300/0 conn(0x7f56cc072160 0x7f56cc0b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:38.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56ca7fc640 1 --2- 172.21.15.59:0/1625195267 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc0ae6b0 0x7f56cc0aeb10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0097a0 tx=0x7f56bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dba418e6f630d98f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56c9ffb640 1 -- 172.21.15.59:0/1625195267 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc035020 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56c9ffb640 1 -- 172.21.15.59:0/1625195267 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc031830 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56c9ffb640 1 -- 172.21.15.59:0/1625195267 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc031be0 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56d1602640 1 -- 172.21.15.59:0/1625195267 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc0ae6b0 msgr2=0x7f56cc0aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56d1602640 1 --2- 172.21.15.59:0/1625195267 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc0ae6b0 0x7f56cc0aeb10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0097a0 tx=0x7f56bc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56d1602640 1 -- 172.21.15.59:0/1625195267 shutdown_connections 2024-03-06T07:33:38.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56d1602640 1 --2- 172.21.15.59:0/1625195267 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc0ae6b0 0x7f56cc0aeb10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56d1602640 1 --2- 172.21.15.59:0/1625195267 >> v2:172.21.15.164:3300/0 conn(0x7f56cc072160 0x7f56cc0b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56d1602640 1 -- 172.21.15.59:0/1625195267 >> 172.21.15.59:0/1625195267 conn(0x7f56cc06b190 msgr2=0x7f56cc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56d1602640 1 -- 172.21.15.59:0/1625195267 shutdown_connections 2024-03-06T07:33:38.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.084+0000 7f56d1602640 1 -- 172.21.15.59:0/1625195267 wait complete. 2024-03-06T07:33:38.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.086+0000 7f56d1602640 1 Processor -- start 2024-03-06T07:33:38.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.086+0000 7f56d1602640 1 -- start start 2024-03-06T07:33:38.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.086+0000 7f56d1602640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56cc072160 0x7f56cc14e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.086+0000 7f56d1602640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56cc0ae6b0 0x7f56cc0b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.086+0000 7f56d1602640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc15b160 0x7f56cc1e4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.086+0000 7f56d1602640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0cdc90 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.086+0000 7f56d1602640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0cdb10 con 0x7f56cc072160 2024-03-06T07:33:38.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.086+0000 7f56d1602640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0cde10 con 0x7f56cc15b160 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.086+0000 7f56ca7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56cc0ae6b0 0x7f56cc0b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56ca7fc640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56cc0ae6b0 0x7f56cc0b1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54974/0 (socket says 172.21.15.59:54974) 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56ca7fc640 1 -- 172.21.15.59:0/2128342187 learned_addr learned my addr 172.21.15.59:0/2128342187 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56cb7fe640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc15b160 0x7f56cc1e4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56caffd640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56cc072160 0x7f56cc14e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56ca7fc640 1 -- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc15b160 msgr2=0x7f56cc1e4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56ca7fc640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc15b160 0x7f56cc1e4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56ca7fc640 1 -- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56cc072160 msgr2=0x7f56cc14e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56ca7fc640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56cc072160 0x7f56cc14e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56ca7fc640 1 -- 172.21.15.59:0/2128342187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc152cf0 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56cb7fe640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc15b160 0x7f56cc1e4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56caffd640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56cc072160 0x7f56cc14e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:38.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56ca7fc640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56cc0ae6b0 0x7f56cc0b1c50 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f56bc02bee0 tx=0x7f56bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56c37fe640 1 -- 172.21.15.59:0/2128342187 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc035040 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56c37fe640 1 -- 172.21.15.59:0/2128342187 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc03f070 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56d1602640 1 -- 172.21.15.59:0/2128342187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc0b0380 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56c37fe640 1 -- 172.21.15.59:0/2128342187 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc03bb40 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.087+0000 7f56d1602640 1 -- 172.21.15.59:0/2128342187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc0b0940 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.089+0000 7f56d1602640 1 -- 172.21.15.59:0/2128342187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc0732b0 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.089+0000 7f56c37fe640 1 -- 172.21.15.59:0/2128342187 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56bc038020 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.089+0000 7f56c37fe640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56ac06b4d0 0x7f56ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.089+0000 7f56c37fe640 1 -- 172.21.15.59:0/2128342187 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f56bc0abd00 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.089+0000 7f56caffd640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56ac06b4d0 0x7f56ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.090+0000 7f56caffd640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56ac06b4d0 0x7f56ac06d990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009ac0 tx=0x7f56b4008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.093+0000 7f56c37fe640 1 -- 172.21.15.59:0/2128342187 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56cc152cf0 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.349+0000 7f56d1602640 1 -- 172.21.15.59:0/2128342187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56cc068ff0 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.350+0000 7f56c37fe640 1 -- 172.21.15.59:0/2128342187 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f56bc07ab60 con 0x7f56cc0ae6b0 2024-03-06T07:33:38.352 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:38.352 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:38.352 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:38.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 -- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56ac06b4d0 msgr2=0x7f56ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56ac06b4d0 0x7f56ac06d990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009ac0 tx=0x7f56b4008040 comp rx=0 tx=0).stop 2024-03-06T07:33:38.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 -- 172.21.15.59:0/2128342187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56cc0ae6b0 msgr2=0x7f56cc0b1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56cc0ae6b0 0x7f56cc0b1c50 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f56bc02bee0 tx=0x7f56bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 -- 172.21.15.59:0/2128342187 shutdown_connections 2024-03-06T07:33:38.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f56ac06b4d0 0x7f56ac06d990 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56cc15b160 0x7f56cc1e4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56cc0ae6b0 0x7f56cc0b1c50 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 --2- 172.21.15.59:0/2128342187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56cc072160 0x7f56cc14e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 -- 172.21.15.59:0/2128342187 >> 172.21.15.59:0/2128342187 conn(0x7f56cc06b190 msgr2=0x7f56cc15da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 -- 172.21.15.59:0/2128342187 shutdown_connections 2024-03-06T07:33:38.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:38.352+0000 7f56d1602640 1 -- 172.21.15.59:0/2128342187 wait complete. 2024-03-06T07:33:38.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:38.566 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:39.568 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:39.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.668+0000 7f3c7eb7a640 1 Processor -- start 2024-03-06T07:33:39.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.668+0000 7f3c7eb7a640 1 -- start start 2024-03-06T07:33:39.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7eb7a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3c7815dfe0 0x7f3c780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.671 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7eb7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c780d29c0 0x7f3c780b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7eb7a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c780b3430 con 0x7f3c780cf830 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7eb7a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c780b31e0 con 0x7f3c7815dfe0 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7eb7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780b3d60 con 0x7f3c780d29c0 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7c916640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3c7815dfe0 0x7f3c780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7c916640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3c7815dfe0 0x7f3c780cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:54990/0 (socket says 172.21.15.59:54990) 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c77fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3c780cf830 0x7f3c780d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57074/0 (socket says 172.21.15.59:57074) 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7c916640 1 -- 172.21.15.59:0/2809856816 learned_addr learned my addr 172.21.15.59:0/2809856816 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7d117640 1 --2- 172.21.15.59:0/2809856816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c780d29c0 0x7f3c780b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7c916640 1 -- 172.21.15.59:0/2809856816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c780d29c0 msgr2=0x7f3c780b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7c916640 1 --2- 172.21.15.59:0/2809856816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c780d29c0 0x7f3c780b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7c916640 1 -- 172.21.15.59:0/2809856816 >> v1:172.21.15.59:6789/0 conn(0x7f3c780cf830 legacy=0x7f3c780d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.672 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c7c916640 1 -- 172.21.15.59:0/2809856816 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c780b3710 con 0x7f3c7815dfe0 2024-03-06T07:33:39.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.669+0000 7f3c777fe640 1 -- 172.21.15.59:0/2809856816 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332624136 0 0) 0x7f3c780b3430 con 0x7f3c780cf830 2024-03-06T07:33:39.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.670+0000 7f3c7c916640 1 --2- 172.21.15.59:0/2809856816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c7815dfe0 0x7f3c780cf2f0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009870 tx=0x7f3c6802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549786157d6b2015 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:39.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.670+0000 7f3c777fe640 1 -- 172.21.15.59:0/2809856816 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68034070 con 0x7f3c7815dfe0 2024-03-06T07:33:39.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.670+0000 7f3c777fe640 1 -- 172.21.15.59:0/2809856816 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c68030840 con 0x7f3c7815dfe0 2024-03-06T07:33:39.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.670+0000 7f3c777fe640 1 -- 172.21.15.59:0/2809856816 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68030c10 con 0x7f3c7815dfe0 2024-03-06T07:33:39.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.671+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/2809856816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c7815dfe0 msgr2=0x7f3c780cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.671+0000 7f3c7eb7a640 1 --2- 172.21.15.59:0/2809856816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c7815dfe0 0x7f3c780cf2f0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009870 tx=0x7f3c6802b980 comp rx=0 tx=0).stop 2024-03-06T07:33:39.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.671+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/2809856816 shutdown_connections 2024-03-06T07:33:39.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.671+0000 7f3c7eb7a640 1 --2- 172.21.15.59:0/2809856816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c780d29c0 0x7f3c780b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.671+0000 7f3c7eb7a640 1 --2- 172.21.15.59:0/2809856816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c7815dfe0 0x7f3c780cf2f0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.671+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/2809856816 >> 172.21.15.59:0/2809856816 conn(0x7f3c780d3cd0 msgr2=0x7f3c780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:39.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.671+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/2809856816 shutdown_connections 2024-03-06T07:33:39.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.671+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/2809856816 wait complete. 2024-03-06T07:33:39.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.673+0000 7f3c7eb7a640 1 Processor -- start 2024-03-06T07:33:39.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.673+0000 7f3c7eb7a640 1 -- start start 2024-03-06T07:33:39.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.673+0000 7f3c7eb7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c780cf830 0x7f3c780b95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7eb7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c780d29c0 0x7f3c7815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7eb7a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c7815dfe0 0x7f3c780b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7eb7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780d1df0 con 0x7f3c780cf830 2024-03-06T07:33:39.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7eb7a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780d1c70 con 0x7f3c7815dfe0 2024-03-06T07:33:39.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7eb7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7815e3c0 con 0x7f3c780d29c0 2024-03-06T07:33:39.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7c916640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c780cf830 0x7f3c780b95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7c916640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c780cf830 0x7f3c780b95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55006/0 (socket says 172.21.15.59:55006) 2024-03-06T07:33:39.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7c916640 1 -- 172.21.15.59:0/519601420 learned_addr learned my addr 172.21.15.59:0/519601420 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c77fff640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c780d29c0 0x7f3c7815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7d117640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c7815dfe0 0x7f3c780b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7c916640 1 -- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c780d29c0 msgr2=0x7f3c7815c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7c916640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c780d29c0 0x7f3c7815c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7c916640 1 -- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c7815dfe0 msgr2=0x7f3c780b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7c916640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c7815dfe0 0x7f3c780b72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7c916640 1 -- 172.21.15.59:0/519601420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c780b7ae0 con 0x7f3c780cf830 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7d117640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c7815dfe0 0x7f3c780b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7c916640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c780cf830 0x7f3c780b95d0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f3c68000c00 tx=0x7f3c680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c757fa640 1 -- 172.21.15.59:0/519601420 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68034040 con 0x7f3c780cf830 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c757fa640 1 -- 172.21.15.59:0/519601420 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c68039d90 con 0x7f3c780cf830 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c757fa640 1 -- 172.21.15.59:0/519601420 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68043610 con 0x7f3c780cf830 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/519601420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c780b7dc0 con 0x7f3c780cf830 2024-03-06T07:33:39.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.674+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/519601420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c780b8380 con 0x7f3c780cf830 2024-03-06T07:33:39.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.675+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/519601420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7804f330 con 0x7f3c780cf830 2024-03-06T07:33:39.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.676+0000 7f3c757fa640 1 -- 172.21.15.59:0/519601420 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c6804a060 con 0x7f3c780cf830 2024-03-06T07:33:39.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.676+0000 7f3c757fa640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c6406b4d0 0x7f3c6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.676+0000 7f3c757fa640 1 -- 172.21.15.59:0/519601420 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3c680ac1f0 con 0x7f3c780cf830 2024-03-06T07:33:39.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.676+0000 7f3c77fff640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c6406b4d0 0x7f3c6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.677+0000 7f3c77fff640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c6406b4d0 0x7f3c6406d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f3c780a1d10 tx=0x7f3c6c007b50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.685+0000 7f3c757fa640 1 -- 172.21.15.59:0/519601420 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c6803e070 con 0x7f3c780cf830 2024-03-06T07:33:39.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.907+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/519601420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c780d0ed0 con 0x7f3c780cf830 2024-03-06T07:33:39.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.908+0000 7f3c757fa640 1 -- 172.21.15.59:0/519601420 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f3c6802d330 con 0x7f3c780cf830 2024-03-06T07:33:39.910 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:39.910 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:39.910 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:39.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c6406b4d0 msgr2=0x7f3c6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c6406b4d0 0x7f3c6406d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f3c780a1d10 tx=0x7f3c6c007b50 comp rx=0 tx=0).stop 2024-03-06T07:33:39.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/519601420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c780cf830 msgr2=0x7f3c780b95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c780cf830 0x7f3c780b95d0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f3c68000c00 tx=0x7f3c680046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/519601420 shutdown_connections 2024-03-06T07:33:39.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3c6406b4d0 0x7f3c6406d990 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c7815dfe0 0x7f3c780b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c780d29c0 0x7f3c7815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 --2- 172.21.15.59:0/519601420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c780cf830 0x7f3c780b95d0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/519601420 >> 172.21.15.59:0/519601420 conn(0x7f3c780d3cd0 msgr2=0x7f3c7806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:39.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/519601420 shutdown_connections 2024-03-06T07:33:39.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:39.910+0000 7f3c7eb7a640 1 -- 172.21.15.59:0/519601420 wait complete. 2024-03-06T07:33:40.124 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:40.124 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:41.125 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:41.227 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.225+0000 7f509a0df640 1 Processor -- start 2024-03-06T07:33:41.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.225+0000 7f509a0df640 1 -- start start 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.226+0000 7f509a0df640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f509415dfe0 0x7f50940cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.226+0000 7f509a0df640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f50940cf830 0x7f50940d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.226+0000 7f509a0df640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50940b3430 con 0x7f50940d27f0 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.226+0000 7f509a0df640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50940b31e0 con 0x7f50940cf830 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.226+0000 7f509a0df640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50940b3d60 con 0x7f509415dfe0 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.226+0000 7f5092ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f50940cf830 0x7f50940d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.226+0000 7f5092ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f50940cf830 0x7f50940d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55022/0 (socket says 172.21.15.59:55022) 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.226+0000 7f5092ffd640 1 -- 172.21.15.59:0/1734993639 learned_addr learned my addr 172.21.15.59:0/1734993639 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.227+0000 7f50937fe640 1 --2- 172.21.15.59:0/1734993639 >> v2:172.21.15.164:3300/0 conn(0x7f509415dfe0 0x7f50940cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.227+0000 7f5092ffd640 1 -- 172.21.15.59:0/1734993639 >> v2:172.21.15.164:3300/0 conn(0x7f509415dfe0 msgr2=0x7f50940cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.227+0000 7f5092ffd640 1 --2- 172.21.15.59:0/1734993639 >> v2:172.21.15.164:3300/0 conn(0x7f509415dfe0 0x7f50940cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.227+0000 7f5092ffd640 1 -- 172.21.15.59:0/1734993639 >> v1:172.21.15.164:6789/0 conn(0x7f50940d27f0 legacy=0x7f50940d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.227+0000 7f5092ffd640 1 -- 172.21.15.59:0/1734993639 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50940b3710 con 0x7f50940cf830 2024-03-06T07:33:41.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.227+0000 7f5092ffd640 1 --2- 172.21.15.59:0/1734993639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50940cf830 0x7f50940d22b0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f5088009870 tx=0x7f508802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e96ca1f010a90f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.228+0000 7f50927fc640 1 -- 172.21.15.59:0/1734993639 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088035020 con 0x7f50940cf830 2024-03-06T07:33:41.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.228+0000 7f50927fc640 1 -- 172.21.15.59:0/1734993639 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5088031870 con 0x7f50940cf830 2024-03-06T07:33:41.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.228+0000 7f50927fc640 1 -- 172.21.15.59:0/1734993639 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088031c40 con 0x7f50940cf830 2024-03-06T07:33:41.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.228+0000 7f509a0df640 1 -- 172.21.15.59:0/1734993639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50940cf830 msgr2=0x7f50940d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.228+0000 7f509a0df640 1 --2- 172.21.15.59:0/1734993639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50940cf830 0x7f50940d22b0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f5088009870 tx=0x7f508802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:41.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.228+0000 7f509a0df640 1 -- 172.21.15.59:0/1734993639 shutdown_connections 2024-03-06T07:33:41.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.228+0000 7f509a0df640 1 --2- 172.21.15.59:0/1734993639 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50940cf830 0x7f50940d22b0 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.228+0000 7f509a0df640 1 --2- 172.21.15.59:0/1734993639 >> v2:172.21.15.164:3300/0 conn(0x7f509415dfe0 0x7f50940cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.228+0000 7f509a0df640 1 -- 172.21.15.59:0/1734993639 >> 172.21.15.59:0/1734993639 conn(0x7f50940d3cd0 msgr2=0x7f50940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:41.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.228+0000 7f509a0df640 1 -- 172.21.15.59:0/1734993639 shutdown_connections 2024-03-06T07:33:41.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.228+0000 7f509a0df640 1 -- 172.21.15.59:0/1734993639 wait complete. 2024-03-06T07:33:41.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.230+0000 7f509a0df640 1 Processor -- start 2024-03-06T07:33:41.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.230+0000 7f509a0df640 1 -- start start 2024-03-06T07:33:41.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.231+0000 7f509a0df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50940cf830 0x7f5094161210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.231+0000 7f509a0df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50940d27f0 0x7f50940cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.231+0000 7f509a0df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f509415dfe0 0x7f50940b9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.231+0000 7f509a0df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50940d1df0 con 0x7f50940cf830 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.231+0000 7f509a0df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50940d1c70 con 0x7f509415dfe0 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.231+0000 7f509a0df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509415e3c0 con 0x7f50940d27f0 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.231+0000 7f50937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50940cf830 0x7f5094161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.231+0000 7f50937fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50940cf830 0x7f5094161210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55032/0 (socket says 172.21.15.59:55032) 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.231+0000 7f50937fe640 1 -- 172.21.15.59:0/673646103 learned_addr learned my addr 172.21.15.59:0/673646103 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f5092ffd640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50940d27f0 0x7f50940cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f5093fff640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f509415dfe0 0x7f50940b9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f50937fe640 1 -- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50940d27f0 msgr2=0x7f50940cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f50937fe640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50940d27f0 0x7f50940cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f50937fe640 1 -- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f509415dfe0 msgr2=0x7f50940b9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f50937fe640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f509415dfe0 0x7f50940b9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f50937fe640 1 -- 172.21.15.59:0/673646103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50940ccf50 con 0x7f50940cf830 2024-03-06T07:33:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f5092ffd640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50940d27f0 0x7f50940cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f5093fff640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f509415dfe0 0x7f50940b9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f50937fe640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50940cf830 0x7f5094161210 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f508400d830 tx=0x7f508400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f507bfff640 1 -- 172.21.15.59:0/673646103 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508400b6c0 con 0x7f50940cf830 2024-03-06T07:33:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f507bfff640 1 -- 172.21.15.59:0/673646103 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5084019070 con 0x7f50940cf830 2024-03-06T07:33:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f509a0df640 1 -- 172.21.15.59:0/673646103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50940cd230 con 0x7f50940cf830 2024-03-06T07:33:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f507bfff640 1 -- 172.21.15.59:0/673646103 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508400ba80 con 0x7f50940cf830 2024-03-06T07:33:41.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.232+0000 7f509a0df640 1 -- 172.21.15.59:0/673646103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50940b9690 con 0x7f50940cf830 2024-03-06T07:33:41.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.233+0000 7f509a0df640 1 -- 172.21.15.59:0/673646103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50940d2230 con 0x7f50940cf830 2024-03-06T07:33:41.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.234+0000 7f507bfff640 1 -- 172.21.15.59:0/673646103 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f508401e460 con 0x7f50940cf830 2024-03-06T07:33:41.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.235+0000 7f507bfff640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f508006b4d0 0x7f508006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.235+0000 7f507bfff640 1 -- 172.21.15.59:0/673646103 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f508408a8d0 con 0x7f50940cf830 2024-03-06T07:33:41.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.235+0000 7f5092ffd640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f508006b4d0 0x7f508006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.236+0000 7f5092ffd640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f508006b4d0 0x7f508006d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5088036000 tx=0x7f5088008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.239+0000 7f507bfff640 1 -- 172.21.15.59:0/673646103 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5084059730 con 0x7f50940cf830 2024-03-06T07:33:41.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.456+0000 7f509a0df640 1 -- 172.21.15.59:0/673646103 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f509404f330 con 0x7f50940cf830 2024-03-06T07:33:41.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.457+0000 7f507bfff640 1 -- 172.21.15.59:0/673646103 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f5084059030 con 0x7f50940cf830 2024-03-06T07:33:41.459 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:41.459 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:41.459 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:41.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.458+0000 7f509a0df640 1 -- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f508006b4d0 msgr2=0x7f508006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.458+0000 7f509a0df640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f508006b4d0 0x7f508006d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5088036000 tx=0x7f5088008030 comp rx=0 tx=0).stop 2024-03-06T07:33:41.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.458+0000 7f509a0df640 1 -- 172.21.15.59:0/673646103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50940cf830 msgr2=0x7f5094161210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.458+0000 7f509a0df640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50940cf830 0x7f5094161210 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f508400d830 tx=0x7f508400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.459+0000 7f509a0df640 1 -- 172.21.15.59:0/673646103 shutdown_connections 2024-03-06T07:33:41.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.459+0000 7f509a0df640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f508006b4d0 0x7f508006d990 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.459+0000 7f509a0df640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f509415dfe0 0x7f50940b9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.459+0000 7f509a0df640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50940d27f0 0x7f50940cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.459+0000 7f509a0df640 1 --2- 172.21.15.59:0/673646103 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50940cf830 0x7f5094161210 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.459+0000 7f509a0df640 1 -- 172.21.15.59:0/673646103 >> 172.21.15.59:0/673646103 conn(0x7f50940d3cd0 msgr2=0x7f50940d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:41.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.459+0000 7f509a0df640 1 -- 172.21.15.59:0/673646103 shutdown_connections 2024-03-06T07:33:41.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:41.459+0000 7f509a0df640 1 -- 172.21.15.59:0/673646103 wait complete. 2024-03-06T07:33:41.672 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:41.673 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:42.673 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.772+0000 7f75eaed4640 1 Processor -- start 2024-03-06T07:33:42.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.772+0000 7f75eaed4640 1 -- start start 2024-03-06T07:33:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.773+0000 7f75eaed4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f75e4072730 0x7f75e415e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.773+0000 7f75eaed4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e415eb40 con 0x7f75e4158600 2024-03-06T07:33:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.773+0000 7f75eaed4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e415ee20 con 0x7f75e4162030 2024-03-06T07:33:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.773+0000 7f75eaed4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e4161ba0 con 0x7f75e4072730 2024-03-06T07:33:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.773+0000 7f75dbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f75e4072730 0x7f75e415e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.773+0000 7f75e8c70640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f75e4158600 0x7f75e406d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57090/0 (socket says 172.21.15.59:57090) 2024-03-06T07:33:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.773+0000 7f75e8c70640 1 -- 172.21.15.59:0/3123011207 learned_addr learned my addr 172.21.15.59:0/3123011207 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.774+0000 7f75dbfff640 1 -- 172.21.15.59:0/3123011207 >> v1:172.21.15.164:6789/0 conn(0x7f75e4162030 legacy=0x7f75e406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.774+0000 7f75dbfff640 1 -- 172.21.15.59:0/3123011207 >> v1:172.21.15.59:6789/0 conn(0x7f75e4158600 legacy=0x7f75e406d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.774+0000 7f75db7fe640 1 -- 172.21.15.59:0/3123011207 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810896583 0 0) 0x7f75e415eb40 con 0x7f75e4158600 2024-03-06T07:33:42.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.774+0000 7f75dbfff640 1 -- 172.21.15.59:0/3123011207 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e415f100 con 0x7f75e4072730 2024-03-06T07:33:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.774+0000 7f75dbfff640 1 --2- 172.21.15.59:0/3123011207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e4072730 0x7f75e415e3a0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f75d40097a0 tx=0x7f75d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7128cc0f163865a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.775+0000 7f75db7fe640 1 -- 172.21.15.59:0/3123011207 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d40315a0 con 0x7f75e4072730 2024-03-06T07:33:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.775+0000 7f75db7fe640 1 -- 172.21.15.59:0/3123011207 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d4037070 con 0x7f75e4072730 2024-03-06T07:33:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.775+0000 7f75db7fe640 1 -- 172.21.15.59:0/3123011207 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d4031960 con 0x7f75e4072730 2024-03-06T07:33:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.775+0000 7f75eaed4640 1 -- 172.21.15.59:0/3123011207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e4072730 msgr2=0x7f75e415e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.775+0000 7f75eaed4640 1 --2- 172.21.15.59:0/3123011207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e4072730 0x7f75e415e3a0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f75d40097a0 tx=0x7f75d402b840 comp rx=0 tx=0).stop 2024-03-06T07:33:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.775+0000 7f75eaed4640 1 -- 172.21.15.59:0/3123011207 shutdown_connections 2024-03-06T07:33:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.775+0000 7f75eaed4640 1 --2- 172.21.15.59:0/3123011207 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e4072730 0x7f75e415e3a0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.775+0000 7f75eaed4640 1 -- 172.21.15.59:0/3123011207 >> 172.21.15.59:0/3123011207 conn(0x7f75e406c0c0 msgr2=0x7f75e406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.775+0000 7f75eaed4640 1 -- 172.21.15.59:0/3123011207 shutdown_connections 2024-03-06T07:33:42.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.775+0000 7f75eaed4640 1 -- 172.21.15.59:0/3123011207 wait complete. 2024-03-06T07:33:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.777+0000 7f75eaed4640 1 Processor -- start 2024-03-06T07:33:42.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.777+0000 7f75eaed4640 1 -- start start 2024-03-06T07:33:42.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.778+0000 7f75eaed4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75e4072730 0x7f75e414fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.778+0000 7f75eaed4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e4158600 0x7f75e415f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.781 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.778+0000 7f75eaed4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75e4162030 0x7f75e414b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.778+0000 7f75eaed4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e415e8e0 con 0x7f75e4158600 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.778+0000 7f75eaed4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e4072be0 con 0x7f75e4072730 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.778+0000 7f75eaed4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e406b6c0 con 0x7f75e4162030 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e4158600 0x7f75e415f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75dbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e4158600 0x7f75e415f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55050/0 (socket says 172.21.15.59:55050) 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75dbfff640 1 -- 172.21.15.59:0/2184245652 learned_addr learned my addr 172.21.15.59:0/2184245652 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75e8c70640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75e4072730 0x7f75e414fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75e9471640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75e4162030 0x7f75e414b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75dbfff640 1 -- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75e4162030 msgr2=0x7f75e414b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75dbfff640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75e4162030 0x7f75e414b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75dbfff640 1 -- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75e4072730 msgr2=0x7f75e414fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75dbfff640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75e4072730 0x7f75e414fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75dbfff640 1 -- 172.21.15.59:0/2184245652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e4152c60 con 0x7f75e4158600 2024-03-06T07:33:42.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75dbfff640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e4158600 0x7f75e415f440 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f75d40097a0 tx=0x7f75d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75d97fa640 1 -- 172.21.15.59:0/2184245652 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d4004030 con 0x7f75e4158600 2024-03-06T07:33:42.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75d97fa640 1 -- 172.21.15.59:0/2184245652 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d40041f0 con 0x7f75e4158600 2024-03-06T07:33:42.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75eaed4640 1 -- 172.21.15.59:0/2184245652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e4152f40 con 0x7f75e4158600 2024-03-06T07:33:42.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75d97fa640 1 -- 172.21.15.59:0/2184245652 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d4032640 con 0x7f75e4158600 2024-03-06T07:33:42.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.779+0000 7f75eaed4640 1 -- 172.21.15.59:0/2184245652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e41500d0 con 0x7f75e4158600 2024-03-06T07:33:42.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.780+0000 7f75eaed4640 1 -- 172.21.15.59:0/2184245652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e404f330 con 0x7f75e4158600 2024-03-06T07:33:42.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.781+0000 7f75d97fa640 1 -- 172.21.15.59:0/2184245652 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75d40327e0 con 0x7f75e4158600 2024-03-06T07:33:42.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.781+0000 7f75d97fa640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75d006b4d0 0x7f75d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.782+0000 7f75d97fa640 1 -- 172.21.15.59:0/2184245652 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f75d402f030 con 0x7f75e4158600 2024-03-06T07:33:42.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.782+0000 7f75e8c70640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75d006b4d0 0x7f75d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.782+0000 7f75e8c70640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75d006b4d0 0x7f75d006d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f75e0000f60 tx=0x7f75e0007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:42.790+0000 7f75d97fa640 1 -- 172.21.15.59:0/2184245652 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75d4041070 con 0x7f75e4158600 2024-03-06T07:33:43.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.008+0000 7f75eaed4640 1 -- 172.21.15.59:0/2184245652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75e40d06f0 con 0x7f75e4158600 2024-03-06T07:33:43.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.008+0000 7f75d97fa640 1 -- 172.21.15.59:0/2184245652 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f75d407aa00 con 0x7f75e4158600 2024-03-06T07:33:43.011 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:43.011 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:43.011 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:43.012 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 -- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75d006b4d0 msgr2=0x7f75d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75d006b4d0 0x7f75d006d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f75e0000f60 tx=0x7f75e0007d50 comp rx=0 tx=0).stop 2024-03-06T07:33:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 -- 172.21.15.59:0/2184245652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e4158600 msgr2=0x7f75e415f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e4158600 0x7f75e415f440 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f75d40097a0 tx=0x7f75d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 -- 172.21.15.59:0/2184245652 shutdown_connections 2024-03-06T07:33:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f75d006b4d0 0x7f75d006d990 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75e4162030 0x7f75e414b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e4158600 0x7f75e415f440 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 --2- 172.21.15.59:0/2184245652 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75e4072730 0x7f75e414fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 -- 172.21.15.59:0/2184245652 >> 172.21.15.59:0/2184245652 conn(0x7f75e406c0c0 msgr2=0x7f75e406d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 -- 172.21.15.59:0/2184245652 shutdown_connections 2024-03-06T07:33:43.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:43.010+0000 7f75eaed4640 1 -- 172.21.15.59:0/2184245652 wait complete. 2024-03-06T07:33:43.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:43.225 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:44.226 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:44.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.325+0000 7fadad2c3640 1 Processor -- start 2024-03-06T07:33:44.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.326+0000 7fadad2c3640 1 -- start start 2024-03-06T07:33:44.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.326+0000 7fadad2c3640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fada80b4170 0x7fada80b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.326+0000 7fadad2c3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fada80d0a80 0x7fada80cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.326+0000 7fadad2c3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada80d25d0 con 0x7fada80d4760 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.326+0000 7fadad2c3640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada80d4bf0 con 0x7fada80b4170 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.326+0000 7fadad2c3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada80ce9e0 con 0x7fada80d0a80 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.326+0000 7fada6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fada80b4170 0x7fada80b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.327+0000 7fada6ffd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fada80b4170 0x7fada80b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39240/0 (socket says 172.21.15.59:39240) 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.327+0000 7fada6ffd640 1 -- 172.21.15.59:0/2911770204 learned_addr learned my addr 172.21.15.59:0/2911770204 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.327+0000 7fada77fe640 1 --2- 172.21.15.59:0/2911770204 >> v2:172.21.15.164:3300/0 conn(0x7fada80d0a80 0x7fada80cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.327+0000 7fada6ffd640 1 -- 172.21.15.59:0/2911770204 >> v2:172.21.15.164:3300/0 conn(0x7fada80d0a80 msgr2=0x7fada80cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.327+0000 7fada6ffd640 1 --2- 172.21.15.59:0/2911770204 >> v2:172.21.15.164:3300/0 conn(0x7fada80d0a80 0x7fada80cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.327+0000 7fada6ffd640 1 -- 172.21.15.59:0/2911770204 >> v1:172.21.15.164:6789/0 conn(0x7fada80d4760 legacy=0x7fada80d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.327+0000 7fada6ffd640 1 -- 172.21.15.59:0/2911770204 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada80d2870 con 0x7fada80b4170 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.327+0000 7fada77fe640 1 --2- 172.21.15.59:0/2911770204 >> v2:172.21.15.164:3300/0 conn(0x7fada80d0a80 0x7fada80cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fada6ffd640 1 --2- 172.21.15.59:0/2911770204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fada80b4170 0x7fada80b6a90 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fada808df00 tx=0x7fad9c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a3ff748f212c6b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fada5ffb640 1 -- 172.21.15.59:0/2911770204 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c035020 con 0x7fada80b4170 2024-03-06T07:33:44.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fada5ffb640 1 -- 172.21.15.59:0/2911770204 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c031870 con 0x7fada80b4170 2024-03-06T07:33:44.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fada5ffb640 1 -- 172.21.15.59:0/2911770204 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c031c40 con 0x7fada80b4170 2024-03-06T07:33:44.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fadad2c3640 1 -- 172.21.15.59:0/2911770204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fada80b4170 msgr2=0x7fada80b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fadad2c3640 1 --2- 172.21.15.59:0/2911770204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fada80b4170 0x7fada80b6a90 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fada808df00 tx=0x7fad9c02b910 comp rx=0 tx=0).stop 2024-03-06T07:33:44.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fadad2c3640 1 -- 172.21.15.59:0/2911770204 shutdown_connections 2024-03-06T07:33:44.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fadad2c3640 1 --2- 172.21.15.59:0/2911770204 >> v2:172.21.15.164:3300/0 conn(0x7fada80d0a80 0x7fada80cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fadad2c3640 1 --2- 172.21.15.59:0/2911770204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fada80b4170 0x7fada80b6a90 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fadad2c3640 1 -- 172.21.15.59:0/2911770204 >> 172.21.15.59:0/2911770204 conn(0x7fada80d3c40 msgr2=0x7fada80d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fadad2c3640 1 -- 172.21.15.59:0/2911770204 shutdown_connections 2024-03-06T07:33:44.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.328+0000 7fadad2c3640 1 -- 172.21.15.59:0/2911770204 wait complete. 2024-03-06T07:33:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.330+0000 7fadad2c3640 1 Processor -- start 2024-03-06T07:33:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.330+0000 7fadad2c3640 1 -- start start 2024-03-06T07:33:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.330+0000 7fadad2c3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fada80b4170 0x7fada81e9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.330+0000 7fadad2c3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fada80d0a80 0x7fada81e7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.330+0000 7fadad2c3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fada8160670 0x7fada81e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.330+0000 7fadad2c3640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada80b4dd0 con 0x7fada80b4170 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.330+0000 7fadad2c3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada80b4550 con 0x7fada8160670 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.330+0000 7fadad2c3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada80b4f50 con 0x7fada80d0a80 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.330+0000 7fada6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fada80b4170 0x7fada81e9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.330+0000 7fada6ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fada80b4170 0x7fada81e9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39244/0 (socket says 172.21.15.59:39244) 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.330+0000 7fada6ffd640 1 -- 172.21.15.59:0/3560241726 learned_addr learned my addr 172.21.15.59:0/3560241726 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fada77fe640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fada8160670 0x7fada81e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fada67fc640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fada80d0a80 0x7fada81e7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fada6ffd640 1 -- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fada80d0a80 msgr2=0x7fada81e7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fada6ffd640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fada80d0a80 0x7fada81e7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fada6ffd640 1 -- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fada8160670 msgr2=0x7fada81e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fada6ffd640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fada8160670 0x7fada81e8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fada6ffd640 1 -- 172.21.15.59:0/3560241726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada81f7ee0 con 0x7fada80b4170 2024-03-06T07:33:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fada67fc640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fada80d0a80 0x7fada81e7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fada77fe640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fada8160670 0x7fada81e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fada6ffd640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fada80b4170 0x7fada81e9600 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fad9c000c00 tx=0x7fad9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fad8f7fe640 1 -- 172.21.15.59:0/3560241726 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c040070 con 0x7fada80b4170 2024-03-06T07:33:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fad8f7fe640 1 -- 172.21.15.59:0/3560241726 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c03a420 con 0x7fada80b4170 2024-03-06T07:33:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fadad2c3640 1 -- 172.21.15.59:0/3560241726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada81f81c0 con 0x7fada80b4170 2024-03-06T07:33:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fad8f7fe640 1 -- 172.21.15.59:0/3560241726 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c035040 con 0x7fada80b4170 2024-03-06T07:33:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.331+0000 7fadad2c3640 1 -- 172.21.15.59:0/3560241726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada81e7a80 con 0x7fada80b4170 2024-03-06T07:33:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.332+0000 7fadad2c3640 1 -- 172.21.15.59:0/3560241726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada804f330 con 0x7fada80b4170 2024-03-06T07:33:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.333+0000 7fad8f7fe640 1 -- 172.21.15.59:0/3560241726 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad9c04b060 con 0x7fada80b4170 2024-03-06T07:33:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.333+0000 7fad8f7fe640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9406b500 0x7fad9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.333+0000 7fad8f7fe640 1 -- 172.21.15.59:0/3560241726 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fad9c0ac120 con 0x7fada80b4170 2024-03-06T07:33:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.333+0000 7fada67fc640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9406b500 0x7fad9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.334+0000 7fada67fc640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9406b500 0x7fad9406d9c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fad98006fd0 tx=0x7fad98008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.342+0000 7fad8f7fe640 1 -- 172.21.15.59:0/3560241726 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad9c07b010 con 0x7fada80b4170 2024-03-06T07:33:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.610+0000 7fadad2c3640 1 -- 172.21.15.59:0/3560241726 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fada81e9ce0 con 0x7fada80b4170 2024-03-06T07:33:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.611+0000 7fad8f7fe640 1 -- 172.21.15.59:0/3560241726 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fad9c07a910 con 0x7fada80b4170 2024-03-06T07:33:44.613 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:44.613 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:44.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 -- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9406b500 msgr2=0x7fad9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9406b500 0x7fad9406d9c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fad98006fd0 tx=0x7fad98008040 comp rx=0 tx=0).stop 2024-03-06T07:33:44.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 -- 172.21.15.59:0/3560241726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fada80b4170 msgr2=0x7fada81e9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fada80b4170 0x7fada81e9600 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fad9c000c00 tx=0x7fad9c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 -- 172.21.15.59:0/3560241726 shutdown_connections 2024-03-06T07:33:44.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fad9406b500 0x7fad9406d9c0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fada8160670 0x7fada81e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fada80d0a80 0x7fada81e7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 --2- 172.21.15.59:0/3560241726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fada80b4170 0x7fada81e9600 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 -- 172.21.15.59:0/3560241726 >> 172.21.15.59:0/3560241726 conn(0x7fada80d3c40 msgr2=0x7fada80d4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 -- 172.21.15.59:0/3560241726 shutdown_connections 2024-03-06T07:33:44.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:44.613+0000 7fadad2c3640 1 -- 172.21.15.59:0/3560241726 wait complete. 2024-03-06T07:33:44.827 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:44.827 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:45.829 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:45.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.929+0000 7f178f46d640 1 Processor -- start 2024-03-06T07:33:45.931 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.929+0000 7f178f46d640 1 -- start start 2024-03-06T07:33:45.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.930+0000 7f178f46d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f178815dfe0 0x7f17880cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.930+0000 7f178f46d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17880d29c0 0x7f17880b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.930+0000 7f178f46d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17880b3430 con 0x7f17880cf830 2024-03-06T07:33:45.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.931+0000 7f178f46d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17880b31e0 con 0x7f178815dfe0 2024-03-06T07:33:45.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.931+0000 7f178f46d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17880b3d60 con 0x7f17880d29c0 2024-03-06T07:33:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.931+0000 7f178d209640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f178815dfe0 0x7f17880cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.931+0000 7f178d209640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f178815dfe0 0x7f17880cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39254/0 (socket says 172.21.15.59:39254) 2024-03-06T07:33:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.931+0000 7f178d209640 1 -- 172.21.15.59:0/1707808447 learned_addr learned my addr 172.21.15.59:0/1707808447 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.931+0000 7f178ca08640 1 --1- 172.21.15.59:0/1707808447 >> v1:172.21.15.59:6789/0 conn(0x7f17880cf830 0x7f17880d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36436/0 (socket says 172.21.15.59:36436) 2024-03-06T07:33:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.931+0000 7f178da0a640 1 --2- 172.21.15.59:0/1707808447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17880d29c0 0x7f17880b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.931+0000 7f178d209640 1 -- 172.21.15.59:0/1707808447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17880d29c0 msgr2=0x7f17880b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.931+0000 7f178d209640 1 --2- 172.21.15.59:0/1707808447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17880d29c0 0x7f17880b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.931+0000 7f178d209640 1 -- 172.21.15.59:0/1707808447 >> v1:172.21.15.59:6789/0 conn(0x7f17880cf830 legacy=0x7f17880d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.931+0000 7f178d209640 1 -- 172.21.15.59:0/1707808447 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17880b3710 con 0x7f178815dfe0 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.932+0000 7f178d209640 1 --2- 172.21.15.59:0/1707808447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f178815dfe0 0x7f17880cf2f0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f1784009870 tx=0x7f178402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494a254d20868108 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.932+0000 7f177ffff640 1 -- 172.21.15.59:0/1707808447 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1784034070 con 0x7f178815dfe0 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.932+0000 7f177ffff640 1 -- 172.21.15.59:0/1707808447 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1784030840 con 0x7f178815dfe0 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.932+0000 7f177ffff640 1 -- 172.21.15.59:0/1707808447 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1784030c10 con 0x7f178815dfe0 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.932+0000 7f178f46d640 1 -- 172.21.15.59:0/1707808447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f178815dfe0 msgr2=0x7f17880cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.932+0000 7f178f46d640 1 --2- 172.21.15.59:0/1707808447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f178815dfe0 0x7f17880cf2f0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f1784009870 tx=0x7f178402b980 comp rx=0 tx=0).stop 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.932+0000 7f178f46d640 1 -- 172.21.15.59:0/1707808447 shutdown_connections 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.932+0000 7f178f46d640 1 --2- 172.21.15.59:0/1707808447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17880d29c0 0x7f17880b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.932+0000 7f178f46d640 1 --2- 172.21.15.59:0/1707808447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f178815dfe0 0x7f17880cf2f0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.932+0000 7f178f46d640 1 -- 172.21.15.59:0/1707808447 >> 172.21.15.59:0/1707808447 conn(0x7f17880d3cd0 msgr2=0x7f17880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.933+0000 7f178f46d640 1 -- 172.21.15.59:0/1707808447 shutdown_connections 2024-03-06T07:33:45.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.933+0000 7f178f46d640 1 -- 172.21.15.59:0/1707808447 wait complete. 2024-03-06T07:33:45.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.934+0000 7f178f46d640 1 Processor -- start 2024-03-06T07:33:45.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.934+0000 7f178f46d640 1 -- start start 2024-03-06T07:33:45.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.934+0000 7f178f46d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17880cf830 0x7f178815ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.934+0000 7f178f46d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17880d29c0 0x7f17880ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.934+0000 7f178f46d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f178815dfe0 0x7f17880d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.934+0000 7f178f46d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17880d1df0 con 0x7f17880d29c0 2024-03-06T07:33:45.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.934+0000 7f178f46d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17880d1c70 con 0x7f178815dfe0 2024-03-06T07:33:45.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.934+0000 7f178f46d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178815e3c0 con 0x7f17880cf830 2024-03-06T07:33:45.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.934+0000 7f178ca08640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17880d29c0 0x7f17880ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178ca08640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17880d29c0 0x7f17880ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39264/0 (socket says 172.21.15.59:39264) 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178ca08640 1 -- 172.21.15.59:0/1236990016 learned_addr learned my addr 172.21.15.59:0/1236990016 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178d209640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17880cf830 0x7f178815ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178da0a640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f178815dfe0 0x7f17880d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178ca08640 1 -- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17880cf830 msgr2=0x7f178815ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178ca08640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17880cf830 0x7f178815ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178ca08640 1 -- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f178815dfe0 msgr2=0x7f17880d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178ca08640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f178815dfe0 0x7f17880d02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178ca08640 1 -- 172.21.15.59:0/1236990016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178815b3d0 con 0x7f17880d29c0 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178ca08640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17880d29c0 0x7f17880ca160 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f177800c440 tx=0x7f1778001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f177dffb640 1 -- 172.21.15.59:0/1236990016 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17780074e0 con 0x7f17880d29c0 2024-03-06T07:33:45.940 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f177dffb640 1 -- 172.21.15.59:0/1236990016 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1778007680 con 0x7f17880d29c0 2024-03-06T07:33:45.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178f46d640 1 -- 172.21.15.59:0/1236990016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17880caf80 con 0x7f17880d29c0 2024-03-06T07:33:45.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f177dffb640 1 -- 172.21.15.59:0/1236990016 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778004a20 con 0x7f17880d29c0 2024-03-06T07:33:45.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.935+0000 7f178f46d640 1 -- 172.21.15.59:0/1236990016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17880cb540 con 0x7f17880d29c0 2024-03-06T07:33:45.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.936+0000 7f178f46d640 1 -- 172.21.15.59:0/1236990016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1788068090 con 0x7f17880d29c0 2024-03-06T07:33:45.941 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.937+0000 7f177dffb640 1 -- 172.21.15.59:0/1236990016 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f177801a020 con 0x7f17880d29c0 2024-03-06T07:33:45.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.938+0000 7f177dffb640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f176c06b4d0 0x7f176c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.938+0000 7f177dffb640 1 -- 172.21.15.59:0/1236990016 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1778089980 con 0x7f17880d29c0 2024-03-06T07:33:45.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.938+0000 7f178d209640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f176c06b4d0 0x7f176c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.939+0000 7f178d209640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f176c06b4d0 0x7f176c06d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1784036000 tx=0x7f178403b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:45.942+0000 7f177dffb640 1 -- 172.21.15.59:0/1236990016 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1788068090 con 0x7f17880d29c0 2024-03-06T07:33:46.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.162+0000 7f178f46d640 1 -- 172.21.15.59:0/1236990016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f178804f330 con 0x7f17880d29c0 2024-03-06T07:33:46.164 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.162+0000 7f177dffb640 1 -- 172.21.15.59:0/1236990016 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f1778058870 con 0x7f17880d29c0 2024-03-06T07:33:46.164 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:46.164 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:46.165 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:46.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 -- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f176c06b4d0 msgr2=0x7f176c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f176c06b4d0 0x7f176c06d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1784036000 tx=0x7f178403b040 comp rx=0 tx=0).stop 2024-03-06T07:33:46.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 -- 172.21.15.59:0/1236990016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17880d29c0 msgr2=0x7f17880ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17880d29c0 0x7f17880ca160 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f177800c440 tx=0x7f1778001e70 comp rx=0 tx=0).stop 2024-03-06T07:33:46.166 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 -- 172.21.15.59:0/1236990016 shutdown_connections 2024-03-06T07:33:46.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f176c06b4d0 0x7f176c06d990 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f178815dfe0 0x7f17880d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17880d29c0 0x7f17880ca160 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 --2- 172.21.15.59:0/1236990016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17880cf830 0x7f178815ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 -- 172.21.15.59:0/1236990016 >> 172.21.15.59:0/1236990016 conn(0x7f17880d3cd0 msgr2=0x7f178806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 -- 172.21.15.59:0/1236990016 shutdown_connections 2024-03-06T07:33:46.167 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:46.164+0000 7f178f46d640 1 -- 172.21.15.59:0/1236990016 wait complete. 2024-03-06T07:33:46.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:46.378 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:47.379 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:47.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.480+0000 7fe0c2b13640 1 Processor -- start 2024-03-06T07:33:47.619 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.480+0000 7fe0c2b13640 1 -- start start 2024-03-06T07:33:47.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0c2b13640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe0bc15dfe0 0x7fe0bc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.623 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0c2b13640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0bc0cf830 0x7fe0bc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0c2b13640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0bc0b3430 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0c2b13640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0bc0b31e0 con 0x7fe0bc0cf830 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0c2b13640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0bc0b3d60 con 0x7fe0bc15dfe0 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0bbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0bc0cf830 0x7fe0bc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0bbfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0bc0cf830 0x7fe0bc0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39278/0 (socket says 172.21.15.59:39278) 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0bbfff640 1 -- 172.21.15.59:0/1498363577 learned_addr learned my addr 172.21.15.59:0/1498363577 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0c08af640 1 --2- 172.21.15.59:0/1498363577 >> v2:172.21.15.164:3300/0 conn(0x7fe0bc15dfe0 0x7fe0bc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0bbfff640 1 -- 172.21.15.59:0/1498363577 >> v2:172.21.15.164:3300/0 conn(0x7fe0bc15dfe0 msgr2=0x7fe0bc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0bbfff640 1 --2- 172.21.15.59:0/1498363577 >> v2:172.21.15.164:3300/0 conn(0x7fe0bc15dfe0 0x7fe0bc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.481+0000 7fe0bbfff640 1 -- 172.21.15.59:0/1498363577 >> v1:172.21.15.164:6789/0 conn(0x7fe0bc0d27f0 legacy=0x7fe0bc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.482+0000 7fe0bbfff640 1 -- 172.21.15.59:0/1498363577 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc0b3710 con 0x7fe0bc0cf830 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.482+0000 7fe0bbfff640 1 --2- 172.21.15.59:0/1498363577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0bc0cf830 0x7fe0bc0d22b0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009870 tx=0x7fe0a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0465fc577c8ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.624 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.482+0000 7fe0bb7fe640 1 -- 172.21.15.59:0/1498363577 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8035020 con 0x7fe0bc0cf830 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.482+0000 7fe0bb7fe640 1 -- 172.21.15.59:0/1498363577 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a8031870 con 0x7fe0bc0cf830 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.482+0000 7fe0bb7fe640 1 -- 172.21.15.59:0/1498363577 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8031c40 con 0x7fe0bc0cf830 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.483+0000 7fe0c2b13640 1 -- 172.21.15.59:0/1498363577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0bc0cf830 msgr2=0x7fe0bc0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.483+0000 7fe0c2b13640 1 --2- 172.21.15.59:0/1498363577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0bc0cf830 0x7fe0bc0d22b0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009870 tx=0x7fe0a802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.483+0000 7fe0c2b13640 1 -- 172.21.15.59:0/1498363577 shutdown_connections 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.483+0000 7fe0c2b13640 1 --2- 172.21.15.59:0/1498363577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0bc0cf830 0x7fe0bc0d22b0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.483+0000 7fe0c2b13640 1 --2- 172.21.15.59:0/1498363577 >> v2:172.21.15.164:3300/0 conn(0x7fe0bc15dfe0 0x7fe0bc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.483+0000 7fe0c2b13640 1 -- 172.21.15.59:0/1498363577 >> 172.21.15.59:0/1498363577 conn(0x7fe0bc0d3cd0 msgr2=0x7fe0bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.483+0000 7fe0c2b13640 1 -- 172.21.15.59:0/1498363577 shutdown_connections 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.483+0000 7fe0c2b13640 1 -- 172.21.15.59:0/1498363577 wait complete. 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.484+0000 7fe0c2b13640 1 Processor -- start 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.484+0000 7fe0c2b13640 1 -- start start 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.485+0000 7fe0c2b13640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0bc0cf830 0x7fe0bc161210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.625 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.485+0000 7fe0c2b13640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0bc0d27f0 0x7fe0bc0cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.485+0000 7fe0c2b13640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0bc15dfe0 0x7fe0bc0b9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.485+0000 7fe0c2b13640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc0d1df0 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.485+0000 7fe0c2b13640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc0d1c70 con 0x7fe0bc0cf830 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.485+0000 7fe0c2b13640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc15e3c0 con 0x7fe0bc15dfe0 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.485+0000 7fe0bbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0bc0d27f0 0x7fe0bc0cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.485+0000 7fe0bbfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0bc0d27f0 0x7fe0bc0cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39288/0 (socket says 172.21.15.59:39288) 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.485+0000 7fe0bbfff640 1 -- 172.21.15.59:0/161383887 learned_addr learned my addr 172.21.15.59:0/161383887 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0c10b0640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0bc15dfe0 0x7fe0bc0b9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0c08af640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0bc0cf830 0x7fe0bc161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0bbfff640 1 -- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0bc15dfe0 msgr2=0x7fe0bc0b9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0bbfff640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0bc15dfe0 0x7fe0bc0b9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0bbfff640 1 -- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0bc0cf830 msgr2=0x7fe0bc161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0bbfff640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0bc0cf830 0x7fe0bc161210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.626 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0bbfff640 1 -- 172.21.15.59:0/161383887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc0ccf50 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0bbfff640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0bc0d27f0 0x7fe0bc0cc800 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8000c00 tx=0x7fe0a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0b97fa640 1 -- 172.21.15.59:0/161383887 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8035040 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0b97fa640 1 -- 172.21.15.59:0/161383887 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a8030d00 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0b97fa640 1 -- 172.21.15.59:0/161383887 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a80456c0 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0c2b13640 1 -- 172.21.15.59:0/161383887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0bc0cd230 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.486+0000 7fe0c2b13640 1 -- 172.21.15.59:0/161383887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0bc0b9690 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.487+0000 7fe0c2b13640 1 -- 172.21.15.59:0/161383887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0bc0d2230 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.487+0000 7fe0b97fa640 1 -- 172.21.15.59:0/161383887 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0a8038020 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.488+0000 7fe0b97fa640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0a006b4d0 0x7fe0a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.488+0000 7fe0b97fa640 1 -- 172.21.15.59:0/161383887 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe0a80ac3c0 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.488+0000 7fe0c08af640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0a006b4d0 0x7fe0a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.489+0000 7fe0c08af640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0a006b4d0 0x7fe0a006d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0006fd0 tx=0x7fe0b0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.627 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.492+0000 7fe0b97fa640 1 -- 172.21.15.59:0/161383887 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0a807b2b0 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.713+0000 7fe0c2b13640 1 -- 172.21.15.59:0/161383887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0bc04f330 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.713+0000 7fe0b97fa640 1 -- 172.21.15.59:0/161383887 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fe0a807abb0 con 0x7fe0bc0d27f0 2024-03-06T07:33:47.716 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:47.716 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:47.716 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.715+0000 7fe0c2b13640 1 -- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0a006b4d0 msgr2=0x7fe0a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.715+0000 7fe0c2b13640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0a006b4d0 0x7fe0a006d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0006fd0 tx=0x7fe0b0008040 comp rx=0 tx=0).stop 2024-03-06T07:33:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.715+0000 7fe0c2b13640 1 -- 172.21.15.59:0/161383887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0bc0d27f0 msgr2=0x7fe0bc0cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.715+0000 7fe0c2b13640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0bc0d27f0 0x7fe0bc0cc800 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8000c00 tx=0x7fe0a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.716+0000 7fe0c2b13640 1 -- 172.21.15.59:0/161383887 shutdown_connections 2024-03-06T07:33:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.716+0000 7fe0c2b13640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe0a006b4d0 0x7fe0a006d990 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.716+0000 7fe0c2b13640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0bc15dfe0 0x7fe0bc0b9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.716+0000 7fe0c2b13640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0bc0d27f0 0x7fe0bc0cc800 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.716+0000 7fe0c2b13640 1 --2- 172.21.15.59:0/161383887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0bc0cf830 0x7fe0bc161210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.716+0000 7fe0c2b13640 1 -- 172.21.15.59:0/161383887 >> 172.21.15.59:0/161383887 conn(0x7fe0bc0d3cd0 msgr2=0x7fe0bc0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.716+0000 7fe0c2b13640 1 -- 172.21.15.59:0/161383887 shutdown_connections 2024-03-06T07:33:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:47.716+0000 7fe0c2b13640 1 -- 172.21.15.59:0/161383887 wait complete. 2024-03-06T07:33:47.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:47.932 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:48.933 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:49.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.033+0000 7f5b567ae640 1 Processor -- start 2024-03-06T07:33:49.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.033+0000 7f5b567ae640 1 -- start start 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.034+0000 7f5b567ae640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5b5015fa10 0x7f5b500ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.034+0000 7f5b567ae640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b500d16e0 0x7f5b500d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.034+0000 7f5b567ae640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b500cebe0 con 0x7f5b5015bdd0 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.034+0000 7f5b567ae640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b500ceec0 con 0x7f5b5015fa10 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.034+0000 7f5b567ae640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b500cf1a0 con 0x7f5b500d16e0 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.034+0000 7f5b4f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5b5015fa10 0x7f5b500ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.034+0000 7f5b4f7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5b5015fa10 0x7f5b500ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39290/0 (socket says 172.21.15.59:39290) 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.034+0000 7f5b4f7fe640 1 -- 172.21.15.59:0/929924286 learned_addr learned my addr 172.21.15.59:0/929924286 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.034+0000 7f5b54d4b640 1 --2- 172.21.15.59:0/929924286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b500d16e0 0x7f5b500d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.035+0000 7f5b4f7fe640 1 -- 172.21.15.59:0/929924286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b500d16e0 msgr2=0x7f5b500d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.035+0000 7f5b4f7fe640 1 --2- 172.21.15.59:0/929924286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b500d16e0 0x7f5b500d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.035+0000 7f5b4f7fe640 1 -- 172.21.15.59:0/929924286 >> v1:172.21.15.164:6789/0 conn(0x7f5b5015bdd0 legacy=0x7f5b500c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.035+0000 7f5b4f7fe640 1 -- 172.21.15.59:0/929924286 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b500cc8e0 con 0x7f5b5015fa10 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.035+0000 7f5b4f7fe640 1 --2- 172.21.15.59:0/929924286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b5015fa10 0x7f5b500ce270 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009870 tx=0x7f5b4402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fa34ea37690e99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.035+0000 7f5b4effd640 1 -- 172.21.15.59:0/929924286 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b440317c0 con 0x7f5b5015fa10 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.036+0000 7f5b4effd640 1 -- 172.21.15.59:0/929924286 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b44037070 con 0x7f5b5015fa10 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.036+0000 7f5b4effd640 1 -- 172.21.15.59:0/929924286 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44031b80 con 0x7f5b5015fa10 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.036+0000 7f5b567ae640 1 -- 172.21.15.59:0/929924286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b5015fa10 msgr2=0x7f5b500ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.036+0000 7f5b567ae640 1 --2- 172.21.15.59:0/929924286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b5015fa10 0x7f5b500ce270 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f5b44009870 tx=0x7f5b4402b860 comp rx=0 tx=0).stop 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.036+0000 7f5b567ae640 1 -- 172.21.15.59:0/929924286 shutdown_connections 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.036+0000 7f5b567ae640 1 --2- 172.21.15.59:0/929924286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b500d16e0 0x7f5b500d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.036+0000 7f5b567ae640 1 --2- 172.21.15.59:0/929924286 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b5015fa10 0x7f5b500ce270 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.036+0000 7f5b567ae640 1 -- 172.21.15.59:0/929924286 >> 172.21.15.59:0/929924286 conn(0x7f5b5009a9f0 msgr2=0x7f5b5009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.036+0000 7f5b567ae640 1 -- 172.21.15.59:0/929924286 shutdown_connections 2024-03-06T07:33:49.039 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.036+0000 7f5b567ae640 1 -- 172.21.15.59:0/929924286 wait complete. 2024-03-06T07:33:49.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.038+0000 7f5b567ae640 1 Processor -- start 2024-03-06T07:33:49.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.038+0000 7f5b567ae640 1 -- start start 2024-03-06T07:33:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.039+0000 7f5b567ae640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b500d16e0 0x7f5b501f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.039+0000 7f5b567ae640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b5015bdd0 0x7f5b500b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b567ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b5015fa10 0x7f5b500c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b567ae640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5015c1b0 con 0x7f5b5015fa10 2024-03-06T07:33:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b567ae640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b500cc5d0 con 0x7f5b5015bdd0 2024-03-06T07:33:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b567ae640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b500ce7b0 con 0x7f5b500d16e0 2024-03-06T07:33:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b54d4b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b5015fa10 0x7f5b500c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b54d4b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b5015fa10 0x7f5b500c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39306/0 (socket says 172.21.15.59:39306) 2024-03-06T07:33:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b54d4b640 1 -- 172.21.15.59:0/409885409 learned_addr learned my addr 172.21.15.59:0/409885409 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b4ffff640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b500d16e0 0x7f5b501f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b4f7fe640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b5015bdd0 0x7f5b500b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b54d4b640 1 -- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b500d16e0 msgr2=0x7f5b501f4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b54d4b640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b500d16e0 0x7f5b501f4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b54d4b640 1 -- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b5015bdd0 msgr2=0x7f5b500b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b54d4b640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b5015bdd0 0x7f5b500b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b54d4b640 1 -- 172.21.15.59:0/409885409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b50145290 con 0x7f5b5015fa10 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b54d4b640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b5015fa10 0x7f5b500c8b90 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f5b48007e40 tx=0x7f5b4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b4cff9640 1 -- 172.21.15.59:0/409885409 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b48014620 con 0x7f5b5015fa10 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b4cff9640 1 -- 172.21.15.59:0/409885409 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b48019070 con 0x7f5b5015fa10 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b567ae640 1 -- 172.21.15.59:0/409885409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b50145570 con 0x7f5b5015fa10 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b4cff9640 1 -- 172.21.15.59:0/409885409 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b48014a20 con 0x7f5b5015fa10 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.040+0000 7f5b567ae640 1 -- 172.21.15.59:0/409885409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b50145b30 con 0x7f5b5015fa10 2024-03-06T07:33:49.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.041+0000 7f5b567ae640 1 -- 172.21.15.59:0/409885409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b500cdc00 con 0x7f5b5015fa10 2024-03-06T07:33:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.042+0000 7f5b4cff9640 1 -- 172.21.15.59:0/409885409 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b48014bc0 con 0x7f5b5015fa10 2024-03-06T07:33:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.042+0000 7f5b4cff9640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b3c06b4d0 0x7f5b3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.042+0000 7f5b4cff9640 1 -- 172.21.15.59:0/409885409 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5b4808aad0 con 0x7f5b5015fa10 2024-03-06T07:33:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.042+0000 7f5b4ffff640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b3c06b4d0 0x7f5b3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.043+0000 7f5b4ffff640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b3c06b4d0 0x7f5b3c06d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f5b40004ea0 tx=0x7f5b40009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.048 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.046+0000 7f5b4cff9640 1 -- 172.21.15.59:0/409885409 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b500cdc00 con 0x7f5b5015fa10 2024-03-06T07:33:49.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.370+0000 7f5b567ae640 1 -- 172.21.15.59:0/409885409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b5004f330 con 0x7f5b5015fa10 2024-03-06T07:33:49.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.370+0000 7f5b4cff9640 1 -- 172.21.15.59:0/409885409 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f5b480599c0 con 0x7f5b5015fa10 2024-03-06T07:33:49.372 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:49.372 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:49.372 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:49.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 -- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b3c06b4d0 msgr2=0x7f5b3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b3c06b4d0 0x7f5b3c06d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f5b40004ea0 tx=0x7f5b40009040 comp rx=0 tx=0).stop 2024-03-06T07:33:49.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 -- 172.21.15.59:0/409885409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b5015fa10 msgr2=0x7f5b500c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b5015fa10 0x7f5b500c8b90 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f5b48007e40 tx=0x7f5b4800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 -- 172.21.15.59:0/409885409 shutdown_connections 2024-03-06T07:33:49.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f5b3c06b4d0 0x7f5b3c06d990 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b5015fa10 0x7f5b500c8b90 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b5015bdd0 0x7f5b500b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 --2- 172.21.15.59:0/409885409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b500d16e0 0x7f5b501f4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 -- 172.21.15.59:0/409885409 >> 172.21.15.59:0/409885409 conn(0x7f5b5009a9f0 msgr2=0x7f5b500c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 -- 172.21.15.59:0/409885409 shutdown_connections 2024-03-06T07:33:49.375 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:49.372+0000 7f5b567ae640 1 -- 172.21.15.59:0/409885409 wait complete. 2024-03-06T07:33:49.586 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:49.586 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:50.587 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:50.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.687+0000 7f214eb0d640 1 Processor -- start 2024-03-06T07:33:50.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.687+0000 7f214eb0d640 1 -- start start 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.688+0000 7f214eb0d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f214815ab00 0x7f21480b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.688+0000 7f214eb0d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21480b48f0 0x7f21480af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.688+0000 7f214eb0d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21480b4e30 con 0x7f21480afc00 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.688+0000 7f214eb0d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21480b5110 con 0x7f214815ab00 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.688+0000 7f214eb0d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214815b310 con 0x7f21480b48f0 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.689+0000 7f214c8a9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f214815ab00 0x7f21480b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.689+0000 7f214d0aa640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f21480afc00 0x7f214806e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36440/0 (socket says 172.21.15.59:36440) 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.689+0000 7f214c8a9640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f214815ab00 0x7f21480b43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39316/0 (socket says 172.21.15.59:39316) 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.689+0000 7f214d0aa640 1 -- 172.21.15.59:0/1211599599 learned_addr learned my addr 172.21.15.59:0/1211599599 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.689+0000 7f2147fff640 1 --2- 172.21.15.59:0/1211599599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21480b48f0 0x7f21480af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.689+0000 7f214c8a9640 1 -- 172.21.15.59:0/1211599599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21480b48f0 msgr2=0x7f21480af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:50.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.689+0000 7f214c8a9640 1 --2- 172.21.15.59:0/1211599599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21480b48f0 0x7f21480af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.689+0000 7f214c8a9640 1 -- 172.21.15.59:0/1211599599 >> v1:172.21.15.59:6789/0 conn(0x7f21480afc00 legacy=0x7f214806e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.689+0000 7f21477fe640 1 -- 172.21.15.59:0/1211599599 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204191776 0 0) 0x7f21480b4e30 con 0x7f21480afc00 2024-03-06T07:33:50.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.689+0000 7f214c8a9640 1 -- 172.21.15.59:0/1211599599 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214815b5f0 con 0x7f214815ab00 2024-03-06T07:33:50.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f214c8a9640 1 --2- 172.21.15.59:0/1211599599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f214815ab00 0x7f21480b43b0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f213c0097a0 tx=0x7f213c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a0b8d0bd0a7575 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f21477fe640 1 -- 172.21.15.59:0/1211599599 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c035020 con 0x7f214815ab00 2024-03-06T07:33:50.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f21477fe640 1 -- 172.21.15.59:0/1211599599 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213c031830 con 0x7f214815ab00 2024-03-06T07:33:50.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f21477fe640 1 -- 172.21.15.59:0/1211599599 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c031c00 con 0x7f214815ab00 2024-03-06T07:33:50.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f214eb0d640 1 -- 172.21.15.59:0/1211599599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f214815ab00 msgr2=0x7f21480b43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f214eb0d640 1 --2- 172.21.15.59:0/1211599599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f214815ab00 0x7f21480b43b0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f213c0097a0 tx=0x7f213c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f214eb0d640 1 -- 172.21.15.59:0/1211599599 shutdown_connections 2024-03-06T07:33:50.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f214eb0d640 1 --2- 172.21.15.59:0/1211599599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21480b48f0 0x7f21480af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f214eb0d640 1 --2- 172.21.15.59:0/1211599599 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f214815ab00 0x7f21480b43b0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f214eb0d640 1 -- 172.21.15.59:0/1211599599 >> 172.21.15.59:0/1211599599 conn(0x7f214806c0c0 msgr2=0x7f214806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f214eb0d640 1 -- 172.21.15.59:0/1211599599 shutdown_connections 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.690+0000 7f214eb0d640 1 -- 172.21.15.59:0/1211599599 wait complete. 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.692+0000 7f214eb0d640 1 Processor -- start 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.692+0000 7f214eb0d640 1 -- start start 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.692+0000 7f214eb0d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21480afc00 0x7f21480d0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.692+0000 7f214eb0d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21480b48f0 0x7f21481f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.692+0000 7f214eb0d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f214815ab00 0x7f21481f3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.692+0000 7f214eb0d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214815b060 con 0x7f21480afc00 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.692+0000 7f214eb0d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214815aee0 con 0x7f21480b48f0 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.692+0000 7f214eb0d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214815cb20 con 0x7f214815ab00 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.692+0000 7f214c8a9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21480afc00 0x7f21480d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f214c8a9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21480afc00 0x7f21480d0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39322/0 (socket says 172.21.15.59:39322) 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f214c8a9640 1 -- 172.21.15.59:0/1804178592 learned_addr learned my addr 172.21.15.59:0/1804178592 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:50.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f214d0aa640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f214815ab00 0x7f21481f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f2147fff640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21480b48f0 0x7f21481f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f214c8a9640 1 -- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f214815ab00 msgr2=0x7f21481f3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f214c8a9640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f214815ab00 0x7f21481f3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f214c8a9640 1 -- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21480b48f0 msgr2=0x7f21481f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f214c8a9640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21480b48f0 0x7f21481f3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f214c8a9640 1 -- 172.21.15.59:0/1804178592 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21481f3c70 con 0x7f21480afc00 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f2147fff640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21480b48f0 0x7f21481f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f214c8a9640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21480afc00 0x7f21480d0640 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f213c0098c0 tx=0x7f213c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f21457fa640 1 -- 172.21.15.59:0/1804178592 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c040070 con 0x7f21480afc00 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f21457fa640 1 -- 172.21.15.59:0/1804178592 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213c03e070 con 0x7f21480afc00 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f214eb0d640 1 -- 172.21.15.59:0/1804178592 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21480cf880 con 0x7f21480afc00 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f21457fa640 1 -- 172.21.15.59:0/1804178592 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c035040 con 0x7f21480afc00 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.693+0000 7f214eb0d640 1 -- 172.21.15.59:0/1804178592 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21480cfe40 con 0x7f21480afc00 2024-03-06T07:33:50.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.694+0000 7f214eb0d640 1 -- 172.21.15.59:0/1804178592 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21480ccf90 con 0x7f21480afc00 2024-03-06T07:33:50.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.695+0000 7f21457fa640 1 -- 172.21.15.59:0/1804178592 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f213c038020 con 0x7f21480afc00 2024-03-06T07:33:50.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.695+0000 7f21457fa640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f212c06b500 0x7f212c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.695+0000 7f21457fa640 1 -- 172.21.15.59:0/1804178592 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f213c0abc40 con 0x7f21480afc00 2024-03-06T07:33:50.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.695+0000 7f2147fff640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f212c06b500 0x7f212c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.700 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.696+0000 7f2147fff640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f212c06b500 0x7f212c06d9c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f214814caa0 tx=0x7f2134008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.701 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.699+0000 7f21457fa640 1 -- 172.21.15.59:0/1804178592 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21480ccf90 con 0x7f21480afc00 2024-03-06T07:33:50.920 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.918+0000 7f214eb0d640 1 -- 172.21.15.59:0/1804178592 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f214804f330 con 0x7f21480afc00 2024-03-06T07:33:50.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.919+0000 7f21457fa640 1 -- 172.21.15.59:0/1804178592 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f213c07aaa0 con 0x7f21480afc00 2024-03-06T07:33:50.921 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:50.921 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:50.921 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:50.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 -- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f212c06b500 msgr2=0x7f212c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f212c06b500 0x7f212c06d9c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f214814caa0 tx=0x7f2134008040 comp rx=0 tx=0).stop 2024-03-06T07:33:50.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 -- 172.21.15.59:0/1804178592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21480afc00 msgr2=0x7f21480d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21480afc00 0x7f21480d0640 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f213c0098c0 tx=0x7f213c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 -- 172.21.15.59:0/1804178592 shutdown_connections 2024-03-06T07:33:50.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f212c06b500 0x7f212c06d9c0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f214815ab00 0x7f21481f3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f21480b48f0 0x7f21481f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.923 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 --2- 172.21.15.59:0/1804178592 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21480afc00 0x7f21480d0640 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 -- 172.21.15.59:0/1804178592 >> 172.21.15.59:0/1804178592 conn(0x7f214806c0c0 msgr2=0x7f214806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 -- 172.21.15.59:0/1804178592 shutdown_connections 2024-03-06T07:33:50.924 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:50.921+0000 7f214eb0d640 1 -- 172.21.15.59:0/1804178592 wait complete. 2024-03-06T07:33:51.134 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:51.135 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:51.480 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:51.483 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:52.135 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:52.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.235+0000 7fd088a40640 1 Processor -- start 2024-03-06T07:33:52.237 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.235+0000 7fd088a40640 1 -- start start 2024-03-06T07:33:52.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.236+0000 7fd088a40640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd0840d22b0 0x7fd0840d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.236+0000 7fd088a40640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd0840b41f0 0x7fd0840b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.236+0000 7fd088a40640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0840b3430 con 0x7fd08415dfe0 2024-03-06T07:33:52.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.236+0000 7fd088a40640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0840d2c50 con 0x7fd0840b41f0 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.236+0000 7fd088a40640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0840b31e0 con 0x7fd0840d22b0 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.236+0000 7fd082d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd0840b41f0 0x7fd0840b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.236+0000 7fd082d9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd0840b41f0 0x7fd0840b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39334/0 (socket says 172.21.15.59:39334) 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.236+0000 7fd082d9d640 1 -- 172.21.15.59:0/468326279 learned_addr learned my addr 172.21.15.59:0/468326279 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.237+0000 7fd082d9d640 1 -- 172.21.15.59:0/468326279 >> v2:172.21.15.164:3300/0 conn(0x7fd0840d22b0 msgr2=0x7fd0840d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.237+0000 7fd081d9b640 1 --2- 172.21.15.59:0/468326279 >> v2:172.21.15.164:3300/0 conn(0x7fd0840d22b0 0x7fd0840d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.237+0000 7fd082d9d640 1 --2- 172.21.15.59:0/468326279 >> v2:172.21.15.164:3300/0 conn(0x7fd0840d22b0 0x7fd0840d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.237+0000 7fd082d9d640 1 -- 172.21.15.59:0/468326279 >> v1:172.21.15.59:6789/0 conn(0x7fd08415dfe0 legacy=0x7fd0840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.237+0000 7fd082d9d640 1 -- 172.21.15.59:0/468326279 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0840b3710 con 0x7fd0840b41f0 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.237+0000 7fd08159a640 1 -- 172.21.15.59:0/468326279 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349193889 0 0) 0x7fd0840b3430 con 0x7fd08415dfe0 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.237+0000 7fd082d9d640 1 --2- 172.21.15.59:0/468326279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0840b41f0 0x7fd0840b4680 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fd074009e50 tx=0x7fd07402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748144af538568fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.238+0000 7fd08159a640 1 -- 172.21.15.59:0/468326279 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07402cb20 con 0x7fd0840b41f0 2024-03-06T07:33:52.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.238+0000 7fd08159a640 1 -- 172.21.15.59:0/468326279 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07402ccc0 con 0x7fd0840b41f0 2024-03-06T07:33:52.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.238+0000 7fd08159a640 1 -- 172.21.15.59:0/468326279 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074034610 con 0x7fd0840b41f0 2024-03-06T07:33:52.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.238+0000 7fd088a40640 1 -- 172.21.15.59:0/468326279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0840b41f0 msgr2=0x7fd0840b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.238+0000 7fd088a40640 1 --2- 172.21.15.59:0/468326279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0840b41f0 0x7fd0840b4680 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fd074009e50 tx=0x7fd07402bf80 comp rx=0 tx=0).stop 2024-03-06T07:33:52.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.238+0000 7fd088a40640 1 -- 172.21.15.59:0/468326279 shutdown_connections 2024-03-06T07:33:52.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.238+0000 7fd088a40640 1 --2- 172.21.15.59:0/468326279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0840b41f0 0x7fd0840b4680 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.238+0000 7fd088a40640 1 --2- 172.21.15.59:0/468326279 >> v2:172.21.15.164:3300/0 conn(0x7fd0840d22b0 0x7fd0840d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.238+0000 7fd088a40640 1 -- 172.21.15.59:0/468326279 >> 172.21.15.59:0/468326279 conn(0x7fd0840d3cd0 msgr2=0x7fd0840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.238+0000 7fd088a40640 1 -- 172.21.15.59:0/468326279 shutdown_connections 2024-03-06T07:33:52.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.238+0000 7fd088a40640 1 -- 172.21.15.59:0/468326279 wait complete. 2024-03-06T07:33:52.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.240+0000 7fd088a40640 1 Processor -- start 2024-03-06T07:33:52.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.240+0000 7fd088a40640 1 -- start start 2024-03-06T07:33:52.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.241+0000 7fd088a40640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0840b41f0 0x7fd0840cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.241+0000 7fd088a40640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0840d22b0 0x7fd08415b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.241+0000 7fd088a40640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd08415dfe0 0x7fd0840b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd088a40640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0840cfa00 con 0x7fd0840d22b0 2024-03-06T07:33:52.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd088a40640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0840b3d60 con 0x7fd0840b41f0 2024-03-06T07:33:52.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd088a40640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0840d1c70 con 0x7fd08415dfe0 2024-03-06T07:33:52.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd081d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0840d22b0 0x7fd08415b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd081d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0840d22b0 0x7fd08415b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39344/0 (socket says 172.21.15.59:39344) 2024-03-06T07:33:52.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd081d9b640 1 -- 172.21.15.59:0/2963834101 learned_addr learned my addr 172.21.15.59:0/2963834101 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:52.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd08259c640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0840b41f0 0x7fd0840cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd082d9d640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd08415dfe0 0x7fd0840b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd081d9b640 1 -- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd08415dfe0 msgr2=0x7fd0840b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd081d9b640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd08415dfe0 0x7fd0840b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd081d9b640 1 -- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0840b41f0 msgr2=0x7fd0840cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd081d9b640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0840b41f0 0x7fd0840cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd081d9b640 1 -- 172.21.15.59:0/2963834101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0840cc9f0 con 0x7fd0840d22b0 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd081d9b640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0840d22b0 0x7fd08415b1a0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fd07800c7f0 tx=0x7fd07800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd072ffd640 1 -- 172.21.15.59:0/2963834101 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd078015070 con 0x7fd0840d22b0 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd072ffd640 1 -- 172.21.15.59:0/2963834101 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd078004240 con 0x7fd0840d22b0 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd088a40640 1 -- 172.21.15.59:0/2963834101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0840cccd0 con 0x7fd0840d22b0 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd072ffd640 1 -- 172.21.15.59:0/2963834101 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd078007620 con 0x7fd0840d22b0 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.242+0000 7fd088a40640 1 -- 172.21.15.59:0/2963834101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0840cd290 con 0x7fd0840d22b0 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.243+0000 7fd088a40640 1 -- 172.21.15.59:0/2963834101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08404f330 con 0x7fd0840d22b0 2024-03-06T07:33:52.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.244+0000 7fd072ffd640 1 -- 172.21.15.59:0/2963834101 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0780079d0 con 0x7fd0840d22b0 2024-03-06T07:33:52.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.244+0000 7fd072ffd640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06406b500 0x7fd06406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.244+0000 7fd072ffd640 1 -- 172.21.15.59:0/2963834101 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd07808a250 con 0x7fd0840d22b0 2024-03-06T07:33:52.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.244+0000 7fd08259c640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06406b500 0x7fd06406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.245+0000 7fd08259c640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06406b500 0x7fd06406d9c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd07c007fd0 tx=0x7fd07c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.248+0000 7fd072ffd640 1 -- 172.21.15.59:0/2963834101 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd078059140 con 0x7fd0840d22b0 2024-03-06T07:33:52.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.469+0000 7fd088a40640 1 -- 172.21.15.59:0/2963834101 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0840008d0 con 0x7fd0840d22b0 2024-03-06T07:33:52.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.469+0000 7fd072ffd640 1 -- 172.21.15.59:0/2963834101 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fd078058a40 con 0x7fd0840d22b0 2024-03-06T07:33:52.472 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:52.472 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:52.472 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:52.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 -- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06406b500 msgr2=0x7fd06406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06406b500 0x7fd06406d9c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd07c007fd0 tx=0x7fd07c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:52.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 -- 172.21.15.59:0/2963834101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0840d22b0 msgr2=0x7fd08415b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0840d22b0 0x7fd08415b1a0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fd07800c7f0 tx=0x7fd07800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 -- 172.21.15.59:0/2963834101 shutdown_connections 2024-03-06T07:33:52.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd06406b500 0x7fd06406d9c0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd08415dfe0 0x7fd0840b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0840d22b0 0x7fd08415b1a0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.474 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 --2- 172.21.15.59:0/2963834101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0840b41f0 0x7fd0840cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 -- 172.21.15.59:0/2963834101 >> 172.21.15.59:0/2963834101 conn(0x7fd0840d3cd0 msgr2=0x7fd08406bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 -- 172.21.15.59:0/2963834101 shutdown_connections 2024-03-06T07:33:52.475 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:52.472+0000 7fd088a40640 1 -- 172.21.15.59:0/2963834101 wait complete. 2024-03-06T07:33:52.686 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:52.686 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:53.687 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:53.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.788+0000 7f2ab1a8f640 1 Processor -- start 2024-03-06T07:33:53.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.788+0000 7f2ab1a8f640 1 -- start start 2024-03-06T07:33:53.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.788+0000 7f2ab1a8f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2aac15dfe0 0x7f2aac0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.792 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.788+0000 7f2ab1a8f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac0cf830 0x7f2aac0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.788+0000 7f2ab1a8f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2aac0d27f0 0x7f2aac0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.788+0000 7f2ab1a8f640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac0d2c50 con 0x7f2aac15dfe0 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.788+0000 7f2ab1a8f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac0b31e0 con 0x7f2aac0d27f0 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.788+0000 7f2ab1a8f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0b3d60 con 0x7f2aac0cf830 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aab7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2aac15dfe0 0x7f2aac0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aab7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2aac15dfe0 0x7f2aac0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58532/0 (socket says 172.21.15.59:58532) 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aab7fe640 1 -- 172.21.15.59:0/3105782951 learned_addr learned my addr 172.21.15.59:0/3105782951 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aab7fe640 1 -- 172.21.15.59:0/3105782951 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac0cf830 msgr2=0x7f2aac0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aaaffd640 1 --2- 172.21.15.59:0/3105782951 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac0cf830 0x7f2aac0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aabfff640 1 --2- 172.21.15.59:0/3105782951 >> v2:172.21.15.164:3300/0 conn(0x7f2aac0d27f0 0x7f2aac0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aab7fe640 1 --2- 172.21.15.59:0/3105782951 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac0cf830 0x7f2aac0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aab7fe640 1 -- 172.21.15.59:0/3105782951 >> v2:172.21.15.164:3300/0 conn(0x7f2aac0d27f0 msgr2=0x7f2aac0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aab7fe640 1 --2- 172.21.15.59:0/3105782951 >> v2:172.21.15.164:3300/0 conn(0x7f2aac0d27f0 0x7f2aac0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aab7fe640 1 -- 172.21.15.59:0/3105782951 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac0b3430 con 0x7f2aac15dfe0 2024-03-06T07:33:53.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aaaffd640 1 --2- 172.21.15.59:0/3105782951 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac0cf830 0x7f2aac0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.789+0000 7f2aabfff640 1 --2- 172.21.15.59:0/3105782951 >> v2:172.21.15.164:3300/0 conn(0x7f2aac0d27f0 0x7f2aac0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2aab7fe640 1 --2- 172.21.15.59:0/3105782951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2aac15dfe0 0x7f2aac0cf2f0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f2aac067ad0 tx=0x7f2a9c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5334cec61314732e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2aaa7fc640 1 -- 172.21.15.59:0/3105782951 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c035020 con 0x7f2aac15dfe0 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2aaa7fc640 1 -- 172.21.15.59:0/3105782951 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c02d040 con 0x7f2aac15dfe0 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2aaa7fc640 1 -- 172.21.15.59:0/3105782951 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c031a80 con 0x7f2aac15dfe0 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3105782951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2aac15dfe0 msgr2=0x7f2aac0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2ab1a8f640 1 --2- 172.21.15.59:0/3105782951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2aac15dfe0 0x7f2aac0cf2f0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f2aac067ad0 tx=0x7f2a9c02b900 comp rx=0 tx=0).stop 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3105782951 shutdown_connections 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2ab1a8f640 1 --2- 172.21.15.59:0/3105782951 >> v2:172.21.15.164:3300/0 conn(0x7f2aac0d27f0 0x7f2aac0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2ab1a8f640 1 --2- 172.21.15.59:0/3105782951 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac0cf830 0x7f2aac0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2ab1a8f640 1 --2- 172.21.15.59:0/3105782951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2aac15dfe0 0x7f2aac0cf2f0 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3105782951 >> 172.21.15.59:0/3105782951 conn(0x7f2aac0d3cd0 msgr2=0x7f2aac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3105782951 shutdown_connections 2024-03-06T07:33:53.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.790+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3105782951 wait complete. 2024-03-06T07:33:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.792+0000 7f2ab1a8f640 1 Processor -- start 2024-03-06T07:33:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.792+0000 7f2ab1a8f640 1 -- start start 2024-03-06T07:33:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.792+0000 7f2ab1a8f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2aac0cf830 0x7f2aac15a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.792+0000 7f2ab1a8f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2aac0d27f0 0x7f2aac0cc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.792+0000 7f2ab1a8f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac15dfe0 0x7f2aac1605e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.792+0000 7f2ab1a8f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0d1c70 con 0x7f2aac0d27f0 2024-03-06T07:33:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.792+0000 7f2ab1a8f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0b39f0 con 0x7f2aac0cf830 2024-03-06T07:33:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.792+0000 7f2ab1a8f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac0d1df0 con 0x7f2aac15dfe0 2024-03-06T07:33:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aaaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2aac0d27f0 0x7f2aac0cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aaaffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2aac0d27f0 0x7f2aac0cc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58548/0 (socket says 172.21.15.59:58548) 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aaaffd640 1 -- 172.21.15.59:0/3456334360 learned_addr learned my addr 172.21.15.59:0/3456334360 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aabfff640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac15dfe0 0x7f2aac1605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aab7fe640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2aac0cf830 0x7f2aac15a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aaaffd640 1 -- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac15dfe0 msgr2=0x7f2aac1605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aaaffd640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac15dfe0 0x7f2aac1605e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aaaffd640 1 -- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2aac0cf830 msgr2=0x7f2aac15a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aaaffd640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2aac0cf830 0x7f2aac15a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aaaffd640 1 -- 172.21.15.59:0/3456334360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac15f540 con 0x7f2aac0d27f0 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aabfff640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac15dfe0 0x7f2aac1605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aab7fe640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2aac0cf830 0x7f2aac15a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2aaaffd640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2aac0d27f0 0x7f2aac0cc510 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c7f0 tx=0x7f2aa000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2a93fff640 1 -- 172.21.15.59:0/3456334360 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0015070 con 0x7f2aac0d27f0 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2a93fff640 1 -- 172.21.15.59:0/3456334360 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa00077c0 con 0x7f2aac0d27f0 2024-03-06T07:33:53.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2a93fff640 1 -- 172.21.15.59:0/3456334360 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0007b90 con 0x7f2aac0d27f0 2024-03-06T07:33:53.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3456334360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac15f820 con 0x7f2aac0d27f0 2024-03-06T07:33:53.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.793+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3456334360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac15fde0 con 0x7f2aac0d27f0 2024-03-06T07:33:53.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.794+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3456334360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aac04f330 con 0x7f2aac0d27f0 2024-03-06T07:33:53.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.795+0000 7f2a93fff640 1 -- 172.21.15.59:0/3456334360 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2aa0018070 con 0x7f2aac0d27f0 2024-03-06T07:33:53.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.795+0000 7f2a93fff640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2a9806b4d0 0x7f2a9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.795+0000 7f2a93fff640 1 -- 172.21.15.59:0/3456334360 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2aa008a310 con 0x7f2aac0d27f0 2024-03-06T07:33:53.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.795+0000 7f2aab7fe640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2a9806b4d0 0x7f2a9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.799 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.796+0000 7f2aab7fe640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2a9806b4d0 0x7f2a9806d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c036000 tx=0x7f2a9c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:53.799+0000 7f2a93fff640 1 -- 172.21.15.59:0/3456334360 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2aa0059170 con 0x7f2aac0d27f0 2024-03-06T07:33:54.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.018+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3456334360 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aac0747a0 con 0x7f2aac0d27f0 2024-03-06T07:33:54.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.018+0000 7f2a93fff640 1 -- 172.21.15.59:0/3456334360 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f2aa0058a70 con 0x7f2aac0d27f0 2024-03-06T07:33:54.021 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:54.021 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:54.021 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.020+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2a9806b4d0 msgr2=0x7f2a9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.020+0000 7f2ab1a8f640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2a9806b4d0 0x7f2a9806d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c036000 tx=0x7f2a9c008030 comp rx=0 tx=0).stop 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.020+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3456334360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2aac0d27f0 msgr2=0x7f2aac0cc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.020+0000 7f2ab1a8f640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2aac0d27f0 0x7f2aac0cc510 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000c7f0 tx=0x7f2aa000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.020+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3456334360 shutdown_connections 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.020+0000 7f2ab1a8f640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2a9806b4d0 0x7f2a9806d990 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.020+0000 7f2ab1a8f640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2aac15dfe0 0x7f2aac1605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.020+0000 7f2ab1a8f640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2aac0d27f0 0x7f2aac0cc510 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.020+0000 7f2ab1a8f640 1 --2- 172.21.15.59:0/3456334360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2aac0cf830 0x7f2aac15a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.020+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3456334360 >> 172.21.15.59:0/3456334360 conn(0x7f2aac0d3cd0 msgr2=0x7f2aac06bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.021+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3456334360 shutdown_connections 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:54.021+0000 7f2ab1a8f640 1 -- 172.21.15.59:0/3456334360 wait complete. 2024-03-06T07:33:54.235 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:54.235 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:55.236 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.336+0000 7f8940e34640 1 Processor -- start 2024-03-06T07:33:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.336+0000 7f8940e34640 1 -- start start 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.337+0000 7f8940e34640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f893c15dfe0 0x7f893c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.337+0000 7f8940e34640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893c0b3430 con 0x7f893c0cf830 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.337+0000 7f8940e34640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893c0b3710 con 0x7f893c0d29c0 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.337+0000 7f8940e34640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f893c0b31e0 con 0x7f893c15dfe0 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.337+0000 7f8931d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f893c0cf830 0x7f893c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55536/0 (socket says 172.21.15.59:55536) 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.337+0000 7f8931d9b640 1 -- 172.21.15.59:0/2374621060 learned_addr learned my addr 172.21.15.59:0/2374621060 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.337+0000 7f893a59c640 1 --2- 172.21.15.59:0/2374621060 >> v2:172.21.15.164:3300/0 conn(0x7f893c15dfe0 0x7f893c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.337+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943429910 0 0) 0x7f893c0b3430 con 0x7f893c0cf830 2024-03-06T07:33:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8928003520 con 0x7f893c0cf830 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2615409610 0 0) 0x7f8928003520 con 0x7f893c0cf830 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 >> v2:172.21.15.164:3300/0 conn(0x7f893c15dfe0 msgr2=0x7f893c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 --2- 172.21.15.59:0/2374621060 >> v2:172.21.15.164:3300/0 conn(0x7f893c15dfe0 0x7f893c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 >> v1:172.21.15.164:6789/0 conn(0x7f893c0d29c0 legacy=0x7f893c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c0b39f0 con 0x7f893c0cf830 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f8924001670 con 0x7f893c0cf830 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 >> v1:172.21.15.59:6789/0 conn(0x7f893c0cf830 legacy=0x7f893c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89280039b0 0x7f8928003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8928004800 0x7f8928004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89280051c0 0x7f892800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c0b31e0 con 0x7f89280039b0 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892800c090 con 0x7f8928004800 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892800c290 con 0x7f89280051c0 2024-03-06T07:33:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f893a59c640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89280039b0 0x7f8928003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8931d9b640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8928004800 0x7f8928004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f893ad9d640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89280051c0 0x7f892800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f893a59c640 1 -- 172.21.15.59:0/2374621060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89280051c0 msgr2=0x7f892800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f893a59c640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89280051c0 0x7f892800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f893a59c640 1 -- 172.21.15.59:0/2374621060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8928004800 msgr2=0x7f8928004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f893a59c640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8928004800 0x7f8928004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f893a59c640 1 -- 172.21.15.59:0/2374621060 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8928003520 con 0x7f89280039b0 2024-03-06T07:33:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f893ad9d640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89280051c0 0x7f892800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f8931d9b640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8928004800 0x7f8928004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.338+0000 7f893ad9d640 1 -- 172.21.15.59:0/2374621060 reap_dead start 2024-03-06T07:33:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.339+0000 7f893a59c640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89280039b0 0x7f8928003df0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f892c009ac0 tx=0x7f8928005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22233f19677aae8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.339+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c039070 con 0x7f89280039b0 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.339+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892c02c6b0 con 0x7f89280039b0 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.339+0000 7f8939d9b640 1 -- 172.21.15.59:0/2374621060 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c009690 con 0x7f89280039b0 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.340+0000 7f8940e34640 1 -- 172.21.15.59:0/2374621060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89280039b0 msgr2=0x7f8928003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.340+0000 7f8940e34640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89280039b0 0x7f8928003df0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f892c009ac0 tx=0x7f8928005830 comp rx=0 tx=0).stop 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.340+0000 7f8940e34640 1 -- 172.21.15.59:0/2374621060 shutdown_connections 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.340+0000 7f8940e34640 1 --2- 172.21.15.59:0/2374621060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f89280039b0 0x7f8928003df0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.340+0000 7f8940e34640 1 -- 172.21.15.59:0/2374621060 >> 172.21.15.59:0/2374621060 conn(0x7f893c0d3cd0 msgr2=0x7f893c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.340+0000 7f8940e34640 1 -- 172.21.15.59:0/2374621060 shutdown_connections 2024-03-06T07:33:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.340+0000 7f8940e34640 1 -- 172.21.15.59:0/2374621060 wait complete. 2024-03-06T07:33:55.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.341+0000 7f8940e34640 1 Processor -- start 2024-03-06T07:33:55.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.341+0000 7f8940e34640 1 -- start start 2024-03-06T07:33:55.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f8940e34640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89280039b0 0x7f893c06c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f8940e34640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f893c0b90a0 0x7f893c158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f8940e34640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f893c0b6f30 0x7f893c0b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f8940e34640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c15e3c0 con 0x7f893c0b6f30 2024-03-06T07:33:55.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f8940e34640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c0b3d60 con 0x7f893c0b90a0 2024-03-06T07:33:55.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f8940e34640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893c15e540 con 0x7f89280039b0 2024-03-06T07:33:55.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f893ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f893c0b6f30 0x7f893c0b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f893ad9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f893c0b6f30 0x7f893c0b73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58568/0 (socket says 172.21.15.59:58568) 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f893ad9d640 1 -- 172.21.15.59:0/1810702536 learned_addr learned my addr 172.21.15.59:0/1810702536 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f8931d9b640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f893c0b90a0 0x7f893c158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f893a59c640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89280039b0 0x7f893c06c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f893ad9d640 1 -- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89280039b0 msgr2=0x7f893c06c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f893ad9d640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89280039b0 0x7f893c06c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f893ad9d640 1 -- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f893c0b90a0 msgr2=0x7f893c158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f893ad9d640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f893c0b90a0 0x7f893c158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f893ad9d640 1 -- 172.21.15.59:0/1810702536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c06c740 con 0x7f893c0b6f30 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f8931d9b640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f893c0b90a0 0x7f893c158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.342+0000 7f893a59c640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89280039b0 0x7f893c06c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.343+0000 7f893ad9d640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f893c0b6f30 0x7f893c0b73b0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f893c0cff60 tx=0x7f893400db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.343+0000 7f89337fe640 1 -- 172.21.15.59:0/1810702536 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934018070 con 0x7f893c0b6f30 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.343+0000 7f89337fe640 1 -- 172.21.15.59:0/1810702536 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8934003b90 con 0x7f893c0b6f30 2024-03-06T07:33:55.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.343+0000 7f8940e34640 1 -- 172.21.15.59:0/1810702536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c15fb10 con 0x7f893c0b6f30 2024-03-06T07:33:55.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.343+0000 7f89337fe640 1 -- 172.21.15.59:0/1810702536 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934005070 con 0x7f893c0b6f30 2024-03-06T07:33:55.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.343+0000 7f8940e34640 1 -- 172.21.15.59:0/1810702536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c06ccc0 con 0x7f893c0b6f30 2024-03-06T07:33:55.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.344+0000 7f8940e34640 1 -- 172.21.15.59:0/1810702536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893c15fce0 con 0x7f893c0b6f30 2024-03-06T07:33:55.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.344+0000 7f89337fe640 1 -- 172.21.15.59:0/1810702536 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f893401b070 con 0x7f893c0b6f30 2024-03-06T07:33:55.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.345+0000 7f89337fe640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f891c06b500 0x7f891c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.345+0000 7f89337fe640 1 -- 172.21.15.59:0/1810702536 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f89340898a0 con 0x7f893c0b6f30 2024-03-06T07:33:55.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.345+0000 7f893a59c640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f891c06b500 0x7f891c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.345+0000 7f893a59c640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f891c06b500 0x7f891c06d9c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f892c009ac0 tx=0x7f892c030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.351 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.349+0000 7f89337fe640 1 -- 172.21.15.59:0/1810702536 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8934058660 con 0x7f893c0b6f30 2024-03-06T07:33:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.567+0000 7f8940e34640 1 -- 172.21.15.59:0/1810702536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f893c04f330 con 0x7f893c0b6f30 2024-03-06T07:33:55.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.568+0000 7f89337fe640 1 -- 172.21.15.59:0/1810702536 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f89340015b0 con 0x7f893c0b6f30 2024-03-06T07:33:55.569 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:55.570 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:55.570 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.569+0000 7f8940e34640 1 -- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f891c06b500 msgr2=0x7f891c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.569+0000 7f8940e34640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f891c06b500 0x7f891c06d9c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f892c009ac0 tx=0x7f892c030000 comp rx=0 tx=0).stop 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.569+0000 7f8940e34640 1 -- 172.21.15.59:0/1810702536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f893c0b6f30 msgr2=0x7f893c0b73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.569+0000 7f8940e34640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f893c0b6f30 0x7f893c0b73b0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f893c0cff60 tx=0x7f893400db80 comp rx=0 tx=0).stop 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.570+0000 7f8940e34640 1 -- 172.21.15.59:0/1810702536 shutdown_connections 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.570+0000 7f8940e34640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f891c06b500 0x7f891c06d9c0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.570+0000 7f8940e34640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f893c0b6f30 0x7f893c0b73b0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.570+0000 7f8940e34640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f893c0b90a0 0x7f893c158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.570+0000 7f8940e34640 1 --2- 172.21.15.59:0/1810702536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89280039b0 0x7f893c06c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.570+0000 7f8940e34640 1 -- 172.21.15.59:0/1810702536 >> 172.21.15.59:0/1810702536 conn(0x7f893c0d3cd0 msgr2=0x7f893c0ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.570+0000 7f8940e34640 1 -- 172.21.15.59:0/1810702536 shutdown_connections 2024-03-06T07:33:55.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:55.570+0000 7f8940e34640 1 -- 172.21.15.59:0/1810702536 wait complete. 2024-03-06T07:33:55.783 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:55.783 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:56.784 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.898+0000 7fd0f3b47640 1 Processor -- start 2024-03-06T07:33:56.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.898+0000 7fd0f3b47640 1 -- start start 2024-03-06T07:33:56.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f3b47640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0d22b0 0x7fd0ec0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f3b47640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ec0b3430 con 0x7fd0ec0b41f0 2024-03-06T07:33:56.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f3b47640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0ec0b3710 con 0x7fd0ec15dfe0 2024-03-06T07:33:56.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f3b47640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0d2c50 con 0x7fd0ec0d22b0 2024-03-06T07:33:56.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f20e4640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd0ec0b41f0 0x7fd0ec0b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55550/0 (socket says 172.21.15.59:55550) 2024-03-06T07:33:56.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f20e4640 1 -- 172.21.15.59:0/2368140612 learned_addr learned my addr 172.21.15.59:0/2368140612 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:56.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978570210 0 0) 0x7fd0ec0b3430 con 0x7fd0ec0b41f0 2024-03-06T07:33:56.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d8003520 con 0x7fd0ec0b41f0 2024-03-06T07:33:56.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f10e2640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0d22b0 0x7fd0ec0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1114409096 0 0) 0x7fd0d8003520 con 0x7fd0ec0b41f0 2024-03-06T07:33:56.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0d22b0 msgr2=0x7fd0ec0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f08e1640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0d22b0 0x7fd0ec0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 >> v1:172.21.15.164:6789/0 conn(0x7fd0ec15dfe0 legacy=0x7fd0ec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec0b39f0 con 0x7fd0ec0b41f0 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f10e2640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0d22b0 0x7fd0ec0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.899+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fd0e40019f0 con 0x7fd0ec0b41f0 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 >> v1:172.21.15.59:6789/0 conn(0x7fd0ec0b41f0 legacy=0x7fd0ec0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f08e1640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80039e0 0x7fd0d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f08e1640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0d22b0 0x7fd0d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f08e1640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d8004d40 0x7fd0d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d800baa0 con 0x7fd0d80039e0 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d800bc50 con 0x7fd0d8004d40 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d800be50 con 0x7fd0ec0d22b0 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f18e3640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80039e0 0x7fd0d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f10e2640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0d22b0 0x7fd0d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.904 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f20e4640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d8004d40 0x7fd0d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f18e3640 1 -- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0d22b0 msgr2=0x7fd0d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f18e3640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0d22b0 0x7fd0d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f18e3640 1 -- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d8004d40 msgr2=0x7fd0d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f18e3640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d8004d40 0x7fd0d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.900+0000 7fd0f18e3640 1 -- 172.21.15.59:0/2368140612 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d8003520 con 0x7fd0d80039e0 2024-03-06T07:33:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.901+0000 7fd0f18e3640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80039e0 0x7fd0d8003e00 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8001720 tx=0x7fd0d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8b98f96da33cd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.901+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8001370 con 0x7fd0d80039e0 2024-03-06T07:33:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.901+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e80052d0 con 0x7fd0d80039e0 2024-03-06T07:33:56.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.901+0000 7fd0f08e1640 1 -- 172.21.15.59:0/2368140612 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8003830 con 0x7fd0d80039e0 2024-03-06T07:33:56.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.901+0000 7fd0f3b47640 1 -- 172.21.15.59:0/2368140612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80039e0 msgr2=0x7fd0d8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.901+0000 7fd0f3b47640 1 --2- 172.21.15.59:0/2368140612 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0d80039e0 0x7fd0d8003e00 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8001720 tx=0x7fd0d8005830 comp rx=0 tx=0).stop 2024-03-06T07:33:56.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.901+0000 7fd0f20e4640 1 -- 172.21.15.59:0/2368140612 reap_dead start 2024-03-06T07:33:56.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.901+0000 7fd0f3b47640 1 -- 172.21.15.59:0/2368140612 shutdown_connections 2024-03-06T07:33:56.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.901+0000 7fd0f3b47640 1 -- 172.21.15.59:0/2368140612 >> 172.21.15.59:0/2368140612 conn(0x7fd0ec0d3cd0 msgr2=0x7fd0ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.902+0000 7fd0f3b47640 1 -- 172.21.15.59:0/2368140612 shutdown_connections 2024-03-06T07:33:56.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.902+0000 7fd0f3b47640 1 -- 172.21.15.59:0/2368140612 wait complete. 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.903+0000 7fd0f3b47640 1 Processor -- start 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.903+0000 7fd0f3b47640 1 -- start start 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.903+0000 7fd0f3b47640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d80039e0 0x7fd0ec14e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.903+0000 7fd0f3b47640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0cada0 0x7fd0ec15b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.903+0000 7fd0f3b47640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0ec0cb220 0x7fd0ec15bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.903+0000 7fd0f3b47640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0b3d60 con 0x7fd0ec0cb220 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.903+0000 7fd0f3b47640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0b31e0 con 0x7fd0d80039e0 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.903+0000 7fd0f3b47640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0d1c70 con 0x7fd0ec0cada0 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f20e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0ec0cb220 0x7fd0ec15bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f20e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0ec0cb220 0x7fd0ec15bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58590/0 (socket says 172.21.15.59:58590) 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f20e4640 1 -- 172.21.15.59:0/3483631933 learned_addr learned my addr 172.21.15.59:0/3483631933 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f18e3640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d80039e0 0x7fd0ec14e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f10e2640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0cada0 0x7fd0ec15b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f20e4640 1 -- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0cada0 msgr2=0x7fd0ec15b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f20e4640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0cada0 0x7fd0ec15b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f20e4640 1 -- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d80039e0 msgr2=0x7fd0ec14e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f20e4640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d80039e0 0x7fd0ec14e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f20e4640 1 -- 172.21.15.59:0/3483631933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec0cb7f0 con 0x7fd0ec0cb220 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f20e4640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0ec0cb220 0x7fd0ec15bb40 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0cff40 tx=0x7fd0e400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0e27fc640 1 -- 172.21.15.59:0/3483631933 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e400ec40 con 0x7fd0ec0cb220 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0e27fc640 1 -- 172.21.15.59:0/3483631933 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e400ede0 con 0x7fd0ec0cb220 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f3b47640 1 -- 172.21.15.59:0/3483631933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ec0cbad0 con 0x7fd0ec0cb220 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0e27fc640 1 -- 172.21.15.59:0/3483631933 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4014670 con 0x7fd0ec0cb220 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.904+0000 7fd0f3b47640 1 -- 172.21.15.59:0/3483631933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ec0cc090 con 0x7fd0ec0cb220 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.905+0000 7fd0f3b47640 1 -- 172.21.15.59:0/3483631933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ec0ce7a0 con 0x7fd0ec0cb220 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.906+0000 7fd0e27fc640 1 -- 172.21.15.59:0/3483631933 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0e4014810 con 0x7fd0ec0cb220 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.906+0000 7fd0e27fc640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0d006b4d0 0x7fd0d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.906+0000 7fd0e27fc640 1 -- 172.21.15.59:0/3483631933 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd0e4089f90 con 0x7fd0ec0cb220 2024-03-06T07:33:56.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.906+0000 7fd0f18e3640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0d006b4d0 0x7fd0d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.907+0000 7fd0f18e3640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0d006b4d0 0x7fd0d006d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8033000 tx=0x7fd0e8009030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:56.915+0000 7fd0e27fc640 1 -- 172.21.15.59:0/3483631933 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0ec0ce7a0 con 0x7fd0ec0cb220 2024-03-06T07:33:57.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.139+0000 7fd0f3b47640 1 -- 172.21.15.59:0/3483631933 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0ec0d0b40 con 0x7fd0ec0cb220 2024-03-06T07:33:57.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.140+0000 7fd0e27fc640 1 -- 172.21.15.59:0/3483631933 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fd0e4058e80 con 0x7fd0ec0cb220 2024-03-06T07:33:57.142 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:57.142 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:57.142 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:57.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 -- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0d006b4d0 msgr2=0x7fd0d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0d006b4d0 0x7fd0d006d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8033000 tx=0x7fd0e8009030 comp rx=0 tx=0).stop 2024-03-06T07:33:57.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 -- 172.21.15.59:0/3483631933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0ec0cb220 msgr2=0x7fd0ec15bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:57.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0ec0cb220 0x7fd0ec15bb40 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0cff40 tx=0x7fd0e400e530 comp rx=0 tx=0).stop 2024-03-06T07:33:57.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 -- 172.21.15.59:0/3483631933 shutdown_connections 2024-03-06T07:33:57.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0d006b4d0 0x7fd0d006d990 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0ec0cb220 0x7fd0ec15bb40 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0ec0cada0 0x7fd0ec15b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 --2- 172.21.15.59:0/3483631933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d80039e0 0x7fd0ec14e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:57.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 -- 172.21.15.59:0/3483631933 >> 172.21.15.59:0/3483631933 conn(0x7fd0ec0d3cd0 msgr2=0x7fd0ec0b0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:57.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 -- 172.21.15.59:0/3483631933 shutdown_connections 2024-03-06T07:33:57.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:57.142+0000 7fd0f3b47640 1 -- 172.21.15.59:0/3483631933 wait complete. 2024-03-06T07:33:57.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:57.356 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:58.358 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:58.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.457+0000 7f23ee993640 1 Processor -- start 2024-03-06T07:33:58.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.457+0000 7f23ee993640 1 -- start start 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23ee993640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f23e815dfe0 0x7f23e80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23ee993640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f23e80cf830 0x7f23e80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23ee993640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e80b3430 con 0x7f23e80d27f0 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23ee993640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e80b31e0 con 0x7f23e815dfe0 2024-03-06T07:33:58.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23ee993640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e80b3d60 con 0x7f23e80cf830 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23e7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f23e815dfe0 0x7f23e80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23ecf30640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f23e80d27f0 0x7f23e80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55558/0 (socket says 172.21.15.59:55558) 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23e7fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f23e815dfe0 0x7f23e80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58596/0 (socket says 172.21.15.59:58596) 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23ecf30640 1 -- 172.21.15.59:0/4030787801 learned_addr learned my addr 172.21.15.59:0/4030787801 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23e77fe640 1 --2- 172.21.15.59:0/4030787801 >> v2:172.21.15.164:3300/0 conn(0x7f23e80cf830 0x7f23e80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23e7fff640 1 -- 172.21.15.59:0/4030787801 >> v2:172.21.15.164:3300/0 conn(0x7f23e80cf830 msgr2=0x7f23e80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23e7fff640 1 --2- 172.21.15.59:0/4030787801 >> v2:172.21.15.164:3300/0 conn(0x7f23e80cf830 0x7f23e80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23e7fff640 1 -- 172.21.15.59:0/4030787801 >> v1:172.21.15.59:6789/0 conn(0x7f23e80d27f0 legacy=0x7f23e80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23e7fff640 1 -- 172.21.15.59:0/4030787801 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e80b3710 con 0x7f23e815dfe0 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23e6ffd640 1 -- 172.21.15.59:0/4030787801 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900666401 0 0) 0x7f23e80b3430 con 0x7f23e80d27f0 2024-03-06T07:33:58.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.458+0000 7f23e77fe640 1 --2- 172.21.15.59:0/4030787801 >> v2:172.21.15.164:3300/0 conn(0x7f23e80cf830 0x7f23e80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.459+0000 7f23e7fff640 1 --2- 172.21.15.59:0/4030787801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e815dfe0 0x7f23e80cf2f0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f23d80097a0 tx=0x7f23d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0848ca0da7d98d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.459+0000 7f23e6ffd640 1 -- 172.21.15.59:0/4030787801 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8035020 con 0x7f23e815dfe0 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.459+0000 7f23e6ffd640 1 -- 172.21.15.59:0/4030787801 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d8031830 con 0x7f23e815dfe0 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.459+0000 7f23e6ffd640 1 -- 172.21.15.59:0/4030787801 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8031c00 con 0x7f23e815dfe0 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.460+0000 7f23ee993640 1 -- 172.21.15.59:0/4030787801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e815dfe0 msgr2=0x7f23e80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.460+0000 7f23ee993640 1 --2- 172.21.15.59:0/4030787801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e815dfe0 0x7f23e80cf2f0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f23d80097a0 tx=0x7f23d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.460+0000 7f23ee993640 1 -- 172.21.15.59:0/4030787801 shutdown_connections 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.460+0000 7f23ee993640 1 --2- 172.21.15.59:0/4030787801 >> v2:172.21.15.164:3300/0 conn(0x7f23e80cf830 0x7f23e80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.460+0000 7f23ee993640 1 --2- 172.21.15.59:0/4030787801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e815dfe0 0x7f23e80cf2f0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.460+0000 7f23ee993640 1 -- 172.21.15.59:0/4030787801 >> 172.21.15.59:0/4030787801 conn(0x7f23e80d3cd0 msgr2=0x7f23e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.460+0000 7f23ee993640 1 -- 172.21.15.59:0/4030787801 shutdown_connections 2024-03-06T07:33:58.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.460+0000 7f23ee993640 1 -- 172.21.15.59:0/4030787801 wait complete. 2024-03-06T07:33:58.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.461+0000 7f23ee993640 1 Processor -- start 2024-03-06T07:33:58.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.461+0000 7f23ee993640 1 -- start start 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23ee993640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e80cf830 0x7f23e80b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23ee993640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23e80d27f0 0x7f23e81f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23ee993640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23e815dfe0 0x7f23e815b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23ee993640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80d1df0 con 0x7f23e80cf830 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23ee993640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80d1c70 con 0x7f23e815dfe0 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23ee993640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e815e3c0 con 0x7f23e80d27f0 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e80cf830 0x7f23e80b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e7fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e80cf830 0x7f23e80b8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58610/0 (socket says 172.21.15.59:58610) 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e7fff640 1 -- 172.21.15.59:0/2259138166 learned_addr learned my addr 172.21.15.59:0/2259138166 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e77fe640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23e80d27f0 0x7f23e81f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23ecf30640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23e815dfe0 0x7f23e815b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e7fff640 1 -- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23e80d27f0 msgr2=0x7f23e81f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e7fff640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23e80d27f0 0x7f23e81f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e7fff640 1 -- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23e815dfe0 msgr2=0x7f23e815b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e7fff640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23e815dfe0 0x7f23e815b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e7fff640 1 -- 172.21.15.59:0/2259138166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e80b95a0 con 0x7f23e80cf830 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e7fff640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e80cf830 0x7f23e80b8bc0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f23d8000c00 tx=0x7f23d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e4ff9640 1 -- 172.21.15.59:0/2259138166 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8035070 con 0x7f23e80cf830 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e4ff9640 1 -- 172.21.15.59:0/2259138166 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d8004230 con 0x7f23e80cf830 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23e4ff9640 1 -- 172.21.15.59:0/2259138166 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d803a6f0 con 0x7f23e80cf830 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.462+0000 7f23ee993640 1 -- 172.21.15.59:0/2259138166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e81f6d60 con 0x7f23e80cf830 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.463+0000 7f23ee993640 1 -- 172.21.15.59:0/2259138166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e81f7320 con 0x7f23e80cf830 2024-03-06T07:33:58.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.463+0000 7f23ee993640 1 -- 172.21.15.59:0/2259138166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e804f330 con 0x7f23e80cf830 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.464+0000 7f23e4ff9640 1 -- 172.21.15.59:0/2259138166 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23d8038020 con 0x7f23e80cf830 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.465+0000 7f23e4ff9640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23d406b4d0 0x7f23d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.465+0000 7f23e4ff9640 1 -- 172.21.15.59:0/2259138166 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f23d80abf20 con 0x7f23e80cf830 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.465+0000 7f23e77fe640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23d406b4d0 0x7f23d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.465+0000 7f23e77fe640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23d406b4d0 0x7f23d406d990 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f23e80d31a0 tx=0x7f23dc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.469+0000 7f23e4ff9640 1 -- 172.21.15.59:0/2259138166 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23d8040070 con 0x7f23e80cf830 2024-03-06T07:33:58.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.687+0000 7f23ee993640 1 -- 172.21.15.59:0/2259138166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23e81f7560 con 0x7f23e80cf830 2024-03-06T07:33:58.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.688+0000 7f23e4ff9640 1 -- 172.21.15.59:0/2259138166 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f23d807ad70 con 0x7f23e80cf830 2024-03-06T07:33:58.690 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:33:58.690 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:33:58.690 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 -- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23d406b4d0 msgr2=0x7f23d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23d406b4d0 0x7f23d406d990 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f23e80d31a0 tx=0x7f23dc008040 comp rx=0 tx=0).stop 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 -- 172.21.15.59:0/2259138166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e80cf830 msgr2=0x7f23e80b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e80cf830 0x7f23e80b8bc0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f23d8000c00 tx=0x7f23d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 -- 172.21.15.59:0/2259138166 shutdown_connections 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f23d406b4d0 0x7f23d406d990 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23e815dfe0 0x7f23e815b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23e80d27f0 0x7f23e81f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 --2- 172.21.15.59:0/2259138166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23e80cf830 0x7f23e80b8bc0 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 -- 172.21.15.59:0/2259138166 >> 172.21.15.59:0/2259138166 conn(0x7f23e80d3cd0 msgr2=0x7f23e80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 -- 172.21.15.59:0/2259138166 shutdown_connections 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:33:58.690+0000 7f23ee993640 1 -- 172.21.15.59:0/2259138166 wait complete. 2024-03-06T07:33:58.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:33:58.904 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:33:59.905 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:00.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.005+0000 7f817b639640 1 Processor -- start 2024-03-06T07:34:00.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.005+0000 7f817b639640 1 -- start start 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.005+0000 7f817b639640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f817415fa10 0x7f81740ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.006+0000 7f817b639640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81740cebe0 con 0x7f81740d16e0 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.006+0000 7f817b639640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81740ceec0 con 0x7f817415bdd0 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.006+0000 7f817b639640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81740cf1a0 con 0x7f817415fa10 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.006+0000 7f8178bd4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f817415fa10 0x7f81740ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.006+0000 7f8179bd6640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f81740d16e0 0x7f81740d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55568/0 (socket says 172.21.15.59:55568) 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.006+0000 7f8179bd6640 1 -- 172.21.15.59:0/3956237245 learned_addr learned my addr 172.21.15.59:0/3956237245 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.006+0000 7f8178bd4640 1 -- 172.21.15.59:0/3956237245 >> v1:172.21.15.164:6789/0 conn(0x7f817415bdd0 legacy=0x7f81740c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.006+0000 7f8178bd4640 1 -- 172.21.15.59:0/3956237245 >> v1:172.21.15.59:6789/0 conn(0x7f81740d16e0 legacy=0x7f81740d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.006+0000 7f816bfff640 1 -- 172.21.15.59:0/3956237245 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205536771 0 0) 0x7f81740cebe0 con 0x7f81740d16e0 2024-03-06T07:34:00.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.006+0000 7f8178bd4640 1 -- 172.21.15.59:0/3956237245 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81740cc8e0 con 0x7f817415fa10 2024-03-06T07:34:00.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.007+0000 7f8178bd4640 1 --2- 172.21.15.59:0/3956237245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f817415fa10 0x7f81740ce270 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f81640097a0 tx=0x7f816402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5187df581895058 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.007+0000 7f816bfff640 1 -- 172.21.15.59:0/3956237245 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81640315a0 con 0x7f817415fa10 2024-03-06T07:34:00.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.007+0000 7f816bfff640 1 -- 172.21.15.59:0/3956237245 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8164037070 con 0x7f817415fa10 2024-03-06T07:34:00.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.007+0000 7f816bfff640 1 -- 172.21.15.59:0/3956237245 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8164031960 con 0x7f817415fa10 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.007+0000 7f817b639640 1 -- 172.21.15.59:0/3956237245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f817415fa10 msgr2=0x7f81740ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.007+0000 7f817b639640 1 --2- 172.21.15.59:0/3956237245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f817415fa10 0x7f81740ce270 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f81640097a0 tx=0x7f816402b840 comp rx=0 tx=0).stop 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.007+0000 7f817b639640 1 -- 172.21.15.59:0/3956237245 shutdown_connections 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.007+0000 7f817b639640 1 --2- 172.21.15.59:0/3956237245 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f817415fa10 0x7f81740ce270 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.007+0000 7f817b639640 1 -- 172.21.15.59:0/3956237245 >> 172.21.15.59:0/3956237245 conn(0x7f817409a9f0 msgr2=0x7f817409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.008+0000 7f817b639640 1 -- 172.21.15.59:0/3956237245 shutdown_connections 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.008+0000 7f817b639640 1 -- 172.21.15.59:0/3956237245 wait complete. 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.009+0000 7f817b639640 1 Processor -- start 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.009+0000 7f817b639640 1 -- start start 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.009+0000 7f817b639640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81740d16e0 0x7f8174157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.009+0000 7f817b639640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f817415bdd0 0x7f81740ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.009+0000 7f817b639640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f817415fa10 0x7f81741f50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.009+0000 7f817b639640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81740ce7b0 con 0x7f81740d16e0 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.009+0000 7f817b639640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817415c1b0 con 0x7f817415fa10 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.009+0000 7f817b639640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81740ce930 con 0x7f817415bdd0 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.009+0000 7f81793d5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81740d16e0 0x7f8174157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f81793d5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81740d16e0 0x7f8174157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58626/0 (socket says 172.21.15.59:58626) 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f81793d5640 1 -- 172.21.15.59:0/1978690676 learned_addr learned my addr 172.21.15.59:0/1978690676 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f81793d5640 1 -- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f817415bdd0 msgr2=0x7f81740ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f81793d5640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f817415bdd0 0x7f81740ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f81793d5640 1 -- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f817415fa10 msgr2=0x7f81741f50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f81793d5640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f817415fa10 0x7f81741f50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f81793d5640 1 -- 172.21.15.59:0/1978690676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81740b4770 con 0x7f81740d16e0 2024-03-06T07:34:00.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f81793d5640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81740d16e0 0x7f8174157f60 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f817000c410 tx=0x7f817000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f8169ffb640 1 -- 172.21.15.59:0/1978690676 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8170017070 con 0x7f81740d16e0 2024-03-06T07:34:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f8169ffb640 1 -- 172.21.15.59:0/1978690676 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81700035f0 con 0x7f81740d16e0 2024-03-06T07:34:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f817b639640 1 -- 172.21.15.59:0/1978690676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81740b4a50 con 0x7f81740d16e0 2024-03-06T07:34:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f8169ffb640 1 -- 172.21.15.59:0/1978690676 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8170003a20 con 0x7f81740d16e0 2024-03-06T07:34:00.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.010+0000 7f817b639640 1 -- 172.21.15.59:0/1978690676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81740b5010 con 0x7f81740d16e0 2024-03-06T07:34:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.012+0000 7f8169ffb640 1 -- 172.21.15.59:0/1978690676 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f817001a070 con 0x7f81740d16e0 2024-03-06T07:34:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.012+0000 7f817b639640 1 -- 172.21.15.59:0/1978690676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f817404f330 con 0x7f81740d16e0 2024-03-06T07:34:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.012+0000 7f8169ffb640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f815806b500 0x7f815806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.012+0000 7f8169ffb640 1 -- 172.21.15.59:0/1978690676 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8170089710 con 0x7f81740d16e0 2024-03-06T07:34:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.012+0000 7f8178bd4640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f815806b500 0x7f815806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.016 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.013+0000 7f8178bd4640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f815806b500 0x7f815806d9c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8164036000 tx=0x7f8164030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.017+0000 7f8169ffb640 1 -- 172.21.15.59:0/1978690676 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8170058570 con 0x7f81740d16e0 2024-03-06T07:34:00.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.238+0000 7f817b639640 1 -- 172.21.15.59:0/1978690676 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81740b7a40 con 0x7f81740d16e0 2024-03-06T07:34:00.240 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.239+0000 7f8169ffb640 1 -- 172.21.15.59:0/1978690676 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f81700010c0 con 0x7f81740d16e0 2024-03-06T07:34:00.241 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:00.241 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:00.241 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:00.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.240+0000 7f817b639640 1 -- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f815806b500 msgr2=0x7f815806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.240+0000 7f817b639640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f815806b500 0x7f815806d9c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8164036000 tx=0x7f8164030000 comp rx=0 tx=0).stop 2024-03-06T07:34:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.240+0000 7f817b639640 1 -- 172.21.15.59:0/1978690676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81740d16e0 msgr2=0x7f8174157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.240+0000 7f817b639640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81740d16e0 0x7f8174157f60 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f817000c410 tx=0x7f817000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.240+0000 7f817b639640 1 -- 172.21.15.59:0/1978690676 shutdown_connections 2024-03-06T07:34:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.240+0000 7f817b639640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f815806b500 0x7f815806d9c0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.240+0000 7f817b639640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f817415fa10 0x7f81741f50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.240+0000 7f817b639640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f817415bdd0 0x7f81740ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.240+0000 7f817b639640 1 --2- 172.21.15.59:0/1978690676 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81740d16e0 0x7f8174157f60 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.241+0000 7f817b639640 1 -- 172.21.15.59:0/1978690676 >> 172.21.15.59:0/1978690676 conn(0x7f817409a9f0 msgr2=0x7f81740c6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.241+0000 7f817b639640 1 -- 172.21.15.59:0/1978690676 shutdown_connections 2024-03-06T07:34:00.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:00.241+0000 7f817b639640 1 -- 172.21.15.59:0/1978690676 wait complete. 2024-03-06T07:34:00.454 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:00.455 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:01.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:01.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.557+0000 7f2cb982a640 1 Processor -- start 2024-03-06T07:34:01.558 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.557+0000 7f2cb982a640 1 -- start start 2024-03-06T07:34:01.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.557+0000 7f2cb982a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb415bb20 0x7f2cb40d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.557+0000 7f2cb982a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2cb40cfff0 0x7f2cb40d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.557+0000 7f2cb982a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb4161980 con 0x7f2cb40ce4b0 2024-03-06T07:34:01.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.557+0000 7f2cb982a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb4161c60 con 0x7f2cb40cfff0 2024-03-06T07:34:01.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.557+0000 7f2cb982a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb4161f40 con 0x7f2cb415bb20 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb37fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f2cb40ce4b0 0x7f2cb40ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55584/0 (socket says 172.21.15.59:55584) 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb37fe640 1 -- 172.21.15.59:0/2602935982 learned_addr learned my addr 172.21.15.59:0/2602935982 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb2ffd640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb415bb20 0x7f2cb40d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cabfff640 1 --2- 172.21.15.59:0/2602935982 >> v2:172.21.15.164:3300/0 conn(0x7f2cb40cfff0 0x7f2cb40d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912871254 0 0) 0x7f2cb4161980 con 0x7f2cb40ce4b0 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca0003520 con 0x7f2cb40ce4b0 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3857198142 0 0) 0x7f2ca0003520 con 0x7f2cb40ce4b0 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb415bb20 msgr2=0x7f2cb40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb415bb20 0x7f2cb40d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 >> v2:172.21.15.164:3300/0 conn(0x7f2cb40cfff0 msgr2=0x7f2cb40d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 --2- 172.21.15.59:0/2602935982 >> v2:172.21.15.164:3300/0 conn(0x7f2cb40cfff0 0x7f2cb40d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb4162220 con 0x7f2cb40ce4b0 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f2cac0019f0 con 0x7f2cb40ce4b0 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 >> v1:172.21.15.59:6789/0 conn(0x7f2cb40ce4b0 legacy=0x7f2cb40ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 0x7f2ca0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ca0004800 0x7f2ca0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb415bb20 0x7f2ca00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb4161c60 con 0x7f2ca00039b0 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb4161f40 con 0x7f2ca0004800 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca000b9c0 con 0x7f2cb415bb20 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb2ffd640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 0x7f2ca0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cabfff640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ca0004800 0x7f2ca0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.558+0000 7f2cb37fe640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb415bb20 0x7f2ca00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.559+0000 7f2cb2ffd640 1 -- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb415bb20 msgr2=0x7f2ca00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.559+0000 7f2cb2ffd640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb415bb20 0x7f2ca00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.559+0000 7f2cb2ffd640 1 -- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ca0004800 msgr2=0x7f2ca0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.559+0000 7f2cb2ffd640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ca0004800 0x7f2ca0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.559+0000 7f2cb2ffd640 1 -- 172.21.15.59:0/2602935982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0003520 con 0x7f2ca00039b0 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.559+0000 7f2cb37fe640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb415bb20 0x7f2ca00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.559+0000 7f2cb2ffd640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 0x7f2ca0003dd0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4009ac0 tx=0x7f2ca0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674f4de771cbada2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.559+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4039070 con 0x7f2ca00039b0 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.559+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca402c6b0 con 0x7f2ca00039b0 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.559+0000 7f2cb27fc640 1 -- 172.21.15.59:0/2602935982 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4009690 con 0x7f2ca00039b0 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.560+0000 7f2cb982a640 1 -- 172.21.15.59:0/2602935982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 msgr2=0x7f2ca0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.560+0000 7f2cb982a640 1 --2- 172.21.15.59:0/2602935982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 0x7f2ca0003dd0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4009ac0 tx=0x7f2ca0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.560+0000 7f2cb37fe640 1 -- 172.21.15.59:0/2602935982 reap_dead start 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.560+0000 7f2cb982a640 1 -- 172.21.15.59:0/2602935982 shutdown_connections 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.560+0000 7f2cb982a640 1 -- 172.21.15.59:0/2602935982 >> 172.21.15.59:0/2602935982 conn(0x7f2cb409a9f0 msgr2=0x7f2cb409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.560+0000 7f2cb982a640 1 -- 172.21.15.59:0/2602935982 shutdown_connections 2024-03-06T07:34:01.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.560+0000 7f2cb982a640 1 -- 172.21.15.59:0/2602935982 wait complete. 2024-03-06T07:34:01.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.561+0000 7f2cb982a640 1 Processor -- start 2024-03-06T07:34:01.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.561+0000 7f2cb982a640 1 -- start start 2024-03-06T07:34:01.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb982a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 0x7f2cb40b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb982a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cb406c6b0 0x7f2cb414baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb982a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb406cb30 0x7f2cb414c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb982a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb40d1b00 con 0x7f2ca00039b0 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb982a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb40d1980 con 0x7f2cb406c6b0 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb982a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb40cf050 con 0x7f2cb406cb30 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 0x7f2cb40b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 0x7f2cb40b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58648/0 (socket says 172.21.15.59:58648) 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb2ffd640 1 -- 172.21.15.59:0/2287798616 learned_addr learned my addr 172.21.15.59:0/2287798616 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb2ffd640 1 -- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb406cb30 msgr2=0x7f2cb414c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb37fe640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb406cb30 0x7f2cb414c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cabfff640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cb406c6b0 0x7f2cb414baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb2ffd640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb406cb30 0x7f2cb414c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb2ffd640 1 -- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cb406c6b0 msgr2=0x7f2cb414baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb2ffd640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cb406c6b0 0x7f2cb414baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb2ffd640 1 -- 172.21.15.59:0/2287798616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb40b4b90 con 0x7f2ca00039b0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cb37fe640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb406cb30 0x7f2cb414c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.562+0000 7f2cabfff640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cb406c6b0 0x7f2cb414baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.563+0000 7f2cb2ffd640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 0x7f2cb40b4650 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4037040 tx=0x7f2ca4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.563+0000 7f2cb0ff9640 1 -- 172.21.15.59:0/2287798616 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4042020 con 0x7f2ca00039b0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.563+0000 7f2cb0ff9640 1 -- 172.21.15.59:0/2287798616 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca403a040 con 0x7f2ca00039b0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.563+0000 7f2cb982a640 1 -- 172.21.15.59:0/2287798616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb414cba0 con 0x7f2ca00039b0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.563+0000 7f2cb0ff9640 1 -- 172.21.15.59:0/2287798616 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4039070 con 0x7f2ca00039b0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.563+0000 7f2cb982a640 1 -- 172.21.15.59:0/2287798616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb414d160 con 0x7f2ca00039b0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.564+0000 7f2cb982a640 1 -- 172.21.15.59:0/2287798616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb404f330 con 0x7f2ca00039b0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.564+0000 7f2cb0ff9640 1 -- 172.21.15.59:0/2287798616 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ca4045020 con 0x7f2ca00039b0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.565+0000 7f2cb0ff9640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c9406b4d0 0x7f2c9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.565+0000 7f2cb0ff9640 1 -- 172.21.15.59:0/2287798616 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2ca40ab6b0 con 0x7f2ca00039b0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.565+0000 7f2cabfff640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c9406b4d0 0x7f2c9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.565+0000 7f2cabfff640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c9406b4d0 0x7f2c9406d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40b2620 tx=0x7f2c9c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.569+0000 7f2cb0ff9640 1 -- 172.21.15.59:0/2287798616 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ca404c070 con 0x7f2ca00039b0 2024-03-06T07:34:01.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.789+0000 7f2cb982a640 1 -- 172.21.15.59:0/2287798616 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cb40b1820 con 0x7f2ca00039b0 2024-03-06T07:34:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.789+0000 7f2cb0ff9640 1 -- 172.21.15.59:0/2287798616 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f2ca407a510 con 0x7f2ca00039b0 2024-03-06T07:34:01.791 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:01.791 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:01.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 -- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c9406b4d0 msgr2=0x7f2c9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c9406b4d0 0x7f2c9406d990 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40b2620 tx=0x7f2c9c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:01.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 -- 172.21.15.59:0/2287798616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 msgr2=0x7f2cb40b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 0x7f2cb40b4650 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4037040 tx=0x7f2ca4004730 comp rx=0 tx=0).stop 2024-03-06T07:34:01.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 -- 172.21.15.59:0/2287798616 shutdown_connections 2024-03-06T07:34:01.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f2c9406b4d0 0x7f2c9406d990 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cb406cb30 0x7f2cb414c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cb406c6b0 0x7f2cb414baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 --2- 172.21.15.59:0/2287798616 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ca00039b0 0x7f2cb40b4650 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 -- 172.21.15.59:0/2287798616 >> 172.21.15.59:0/2287798616 conn(0x7f2cb409a9f0 msgr2=0x7f2cb406ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 -- 172.21.15.59:0/2287798616 shutdown_connections 2024-03-06T07:34:01.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:01.791+0000 7f2cb982a640 1 -- 172.21.15.59:0/2287798616 wait complete. 2024-03-06T07:34:02.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:02.005 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:03.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:03.108 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.107+0000 7f782d8f4640 1 Processor -- start 2024-03-06T07:34:03.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.107+0000 7f782d8f4640 1 -- start start 2024-03-06T07:34:03.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.108+0000 7f782d8f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b4170 0x7f78280b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.108+0000 7f782d8f4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f78280d0a80 0x7f78280cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.108+0000 7f782d8f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78280d25d0 con 0x7f78280d4760 2024-03-06T07:34:03.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.108+0000 7f782d8f4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78280d4bf0 con 0x7f78280d0a80 2024-03-06T07:34:03.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.108+0000 7f782d8f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78280ce9e0 con 0x7f78280b4170 2024-03-06T07:34:03.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.108+0000 7f781ffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f78280d4760 0x7f78280d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55596/0 (socket says 172.21.15.59:55596) 2024-03-06T07:34:03.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.108+0000 7f781ffff640 1 -- 172.21.15.59:0/3550178710 learned_addr learned my addr 172.21.15.59:0/3550178710 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:03.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.108+0000 7f7826ffd640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b4170 0x7f78280b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.108+0000 7f78277fe640 1 --2- 172.21.15.59:0/3550178710 >> v2:172.21.15.164:3300/0 conn(0x7f78280d0a80 0x7f78280cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.108+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945517787 0 0) 0x7f78280d25d0 con 0x7f78280d4760 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.108+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7814003520 con 0x7f78280d4760 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1087773863 0 0) 0x7f7814003520 con 0x7f78280d4760 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b4170 msgr2=0x7f78280b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b4170 0x7f78280b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 >> v2:172.21.15.164:3300/0 conn(0x7f78280d0a80 msgr2=0x7f78280cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 --2- 172.21.15.59:0/3550178710 >> v2:172.21.15.164:3300/0 conn(0x7f78280d0a80 0x7f78280cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78280d2870 con 0x7f78280d4760 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f7810001670 con 0x7f78280d4760 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 >> v1:172.21.15.59:6789/0 conn(0x7f78280d4760 legacy=0x7f78280d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78140039b0 0x7f7814003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b4170 0x7f7814004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 0x7f78140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78280d4bf0 con 0x7f7814004d40 2024-03-06T07:34:03.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78280ce9e0 con 0x7f78140039b0 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781400b9c0 con 0x7f78280b4170 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78277fe640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 0x7f78140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f7826ffd640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78140039b0 0x7f7814003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f781ffff640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b4170 0x7f7814004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78277fe640 1 -- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b4170 msgr2=0x7f7814004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78277fe640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b4170 0x7f7814004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78277fe640 1 -- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78140039b0 msgr2=0x7f7814003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78277fe640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78140039b0 0x7f7814003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f78277fe640 1 -- 172.21.15.59:0/3550178710 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7814003520 con 0x7f7814004d40 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.109+0000 7f781ffff640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b4170 0x7f7814004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.110+0000 7f78277fe640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 0x7f78140051c0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f7820009f90 tx=0x7f7814005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63798ec275327a01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.110+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820039070 con 0x7f7814004d40 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.110+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782002cc20 con 0x7f7814004d40 2024-03-06T07:34:03.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.110+0000 7f78267fc640 1 -- 172.21.15.59:0/3550178710 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820009c10 con 0x7f7814004d40 2024-03-06T07:34:03.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.111+0000 7f782d8f4640 1 -- 172.21.15.59:0/3550178710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 msgr2=0x7f78140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.111+0000 7f782d8f4640 1 --2- 172.21.15.59:0/3550178710 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 0x7f78140051c0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f7820009f90 tx=0x7f7814005830 comp rx=0 tx=0).stop 2024-03-06T07:34:03.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.111+0000 7f78277fe640 1 -- 172.21.15.59:0/3550178710 reap_dead start 2024-03-06T07:34:03.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.111+0000 7f782d8f4640 1 -- 172.21.15.59:0/3550178710 shutdown_connections 2024-03-06T07:34:03.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.111+0000 7f782d8f4640 1 -- 172.21.15.59:0/3550178710 >> 172.21.15.59:0/3550178710 conn(0x7f78280d3c40 msgr2=0x7f78280d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.111+0000 7f782d8f4640 1 -- 172.21.15.59:0/3550178710 shutdown_connections 2024-03-06T07:34:03.116 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.111+0000 7f782d8f4640 1 -- 172.21.15.59:0/3550178710 wait complete. 2024-03-06T07:34:03.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.112+0000 7f782d8f4640 1 Processor -- start 2024-03-06T07:34:03.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.112+0000 7f782d8f4640 1 -- start start 2024-03-06T07:34:03.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f782d8f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 0x7f78280b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f782d8f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b2e90 0x7f78280b3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f782d8f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78281f3af0 0x7f78281f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f782d8f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78280b4dd0 con 0x7f7814004d40 2024-03-06T07:34:03.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f782d8f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78280b4550 con 0x7f78281f3af0 2024-03-06T07:34:03.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f782d8f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78280b4f50 con 0x7f78280b2e90 2024-03-06T07:34:03.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f7826ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 0x7f78280b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f7826ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 0x7f78280b0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58676/0 (socket says 172.21.15.59:58676) 2024-03-06T07:34:03.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f7826ffd640 1 -- 172.21.15.59:0/995238870 learned_addr learned my addr 172.21.15.59:0/995238870 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f781ffff640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b2e90 0x7f78280b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f78277fe640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78281f3af0 0x7f78281f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f7826ffd640 1 -- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b2e90 msgr2=0x7f78280b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f7826ffd640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b2e90 0x7f78280b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f7826ffd640 1 -- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78281f3af0 msgr2=0x7f78281f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f7826ffd640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78281f3af0 0x7f78281f3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f7826ffd640 1 -- 172.21.15.59:0/995238870 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78280b1310 con 0x7f7814004d40 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f781ffff640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b2e90 0x7f78280b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.113+0000 7f7826ffd640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 0x7f78280b0ac0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f7818011910 tx=0x7f7818011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.114+0000 7f7824ff9640 1 -- 172.21.15.59:0/995238870 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781800faf0 con 0x7f7814004d40 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.114+0000 7f7824ff9640 1 -- 172.21.15.59:0/995238870 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781800fc90 con 0x7f7814004d40 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.114+0000 7f782d8f4640 1 -- 172.21.15.59:0/995238870 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78280b15f0 con 0x7f7814004d40 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.114+0000 7f7824ff9640 1 -- 172.21.15.59:0/995238870 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818014440 con 0x7f7814004d40 2024-03-06T07:34:03.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.114+0000 7f782d8f4640 1 -- 172.21.15.59:0/995238870 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78280b1bb0 con 0x7f7814004d40 2024-03-06T07:34:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.115+0000 7f782d8f4640 1 -- 172.21.15.59:0/995238870 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782804f330 con 0x7f7814004d40 2024-03-06T07:34:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.115+0000 7f7824ff9640 1 -- 172.21.15.59:0/995238870 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7818012040 con 0x7f7814004d40 2024-03-06T07:34:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.115+0000 7f7824ff9640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f780806b500 0x7f780806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.116+0000 7f7824ff9640 1 -- 172.21.15.59:0/995238870 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f781808a790 con 0x7f7814004d40 2024-03-06T07:34:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.116+0000 7f781ffff640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f780806b500 0x7f780806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.121 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.116+0000 7f781ffff640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f780806b500 0x7f780806d9c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f78280b0a50 tx=0x7f7810009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.120+0000 7f7824ff9640 1 -- 172.21.15.59:0/995238870 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7818019030 con 0x7f7814004d40 2024-03-06T07:34:03.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.359+0000 7f782d8f4640 1 -- 172.21.15.59:0/995238870 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78280733b0 con 0x7f7814004d40 2024-03-06T07:34:03.361 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.360+0000 7f7824ff9640 1 -- 172.21.15.59:0/995238870 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f7818059680 con 0x7f7814004d40 2024-03-06T07:34:03.362 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:03.362 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:03.362 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 -- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f780806b500 msgr2=0x7f780806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f780806b500 0x7f780806d9c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f78280b0a50 tx=0x7f7810009040 comp rx=0 tx=0).stop 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 -- 172.21.15.59:0/995238870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 msgr2=0x7f78280b0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 0x7f78280b0ac0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f7818011910 tx=0x7f7818011dd0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 -- 172.21.15.59:0/995238870 shutdown_connections 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f780806b500 0x7f780806d9c0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78281f3af0 0x7f78281f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78280b2e90 0x7f78280b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 --2- 172.21.15.59:0/995238870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7814004d40 0x7f78280b0ac0 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 -- 172.21.15.59:0/995238870 >> 172.21.15.59:0/995238870 conn(0x7f78280d3c40 msgr2=0x7f782806b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 -- 172.21.15.59:0/995238870 shutdown_connections 2024-03-06T07:34:03.364 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:03.362+0000 7f782d8f4640 1 -- 172.21.15.59:0/995238870 wait complete. 2024-03-06T07:34:03.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:03.576 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:04.577 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.677+0000 7f407324e640 1 Processor -- start 2024-03-06T07:34:04.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.677+0000 7f407324e640 1 -- start start 2024-03-06T07:34:04.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f407324e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f406c15bb20 0x7f406c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f407324e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f406c0cd8c0 0x7f406c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f407324e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c161980 con 0x7f406c0cfff0 2024-03-06T07:34:04.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f407324e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406c161c60 con 0x7f406c15bb20 2024-03-06T07:34:04.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f407324e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406c161f40 con 0x7f406c0cd8c0 2024-03-06T07:34:04.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f4070fea640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f406c15bb20 0x7f406c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f406bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f406c0cfff0 0x7f406c0ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57384/0 (socket says 172.21.15.59:57384) 2024-03-06T07:34:04.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f406bfff640 1 -- 172.21.15.59:0/2905321556 learned_addr learned my addr 172.21.15.59:0/2905321556 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:04.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f40717eb640 1 --2- 172.21.15.59:0/2905321556 >> v2:172.21.15.164:3300/0 conn(0x7f406c0cd8c0 0x7f406c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f4070fea640 1 -- 172.21.15.59:0/2905321556 >> v2:172.21.15.164:3300/0 conn(0x7f406c0cd8c0 msgr2=0x7f406c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f4070fea640 1 --2- 172.21.15.59:0/2905321556 >> v2:172.21.15.164:3300/0 conn(0x7f406c0cd8c0 0x7f406c0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f406b7fe640 1 -- 172.21.15.59:0/2905321556 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140424531 0 0) 0x7f406c161980 con 0x7f406c0cfff0 2024-03-06T07:34:04.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f4070fea640 1 -- 172.21.15.59:0/2905321556 >> v1:172.21.15.59:6789/0 conn(0x7f406c0cfff0 legacy=0x7f406c0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.678+0000 7f4070fea640 1 -- 172.21.15.59:0/2905321556 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c162220 con 0x7f406c15bb20 2024-03-06T07:34:04.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.679+0000 7f4070fea640 1 --2- 172.21.15.59:0/2905321556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c15bb20 0x7f406c0d1df0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f406c1618c0 tx=0x7f406002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d0b0678099b99d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.679+0000 7f406b7fe640 1 -- 172.21.15.59:0/2905321556 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060035020 con 0x7f406c15bb20 2024-03-06T07:34:04.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.679+0000 7f406b7fe640 1 -- 172.21.15.59:0/2905321556 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4060031830 con 0x7f406c15bb20 2024-03-06T07:34:04.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.679+0000 7f406b7fe640 1 -- 172.21.15.59:0/2905321556 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060031c00 con 0x7f406c15bb20 2024-03-06T07:34:04.685 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.680+0000 7f407324e640 1 -- 172.21.15.59:0/2905321556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c15bb20 msgr2=0x7f406c0d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.680+0000 7f407324e640 1 --2- 172.21.15.59:0/2905321556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c15bb20 0x7f406c0d1df0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f406c1618c0 tx=0x7f406002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.680+0000 7f407324e640 1 -- 172.21.15.59:0/2905321556 shutdown_connections 2024-03-06T07:34:04.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.680+0000 7f407324e640 1 --2- 172.21.15.59:0/2905321556 >> v2:172.21.15.164:3300/0 conn(0x7f406c0cd8c0 0x7f406c0d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.680+0000 7f407324e640 1 --2- 172.21.15.59:0/2905321556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c15bb20 0x7f406c0d1df0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.680+0000 7f407324e640 1 -- 172.21.15.59:0/2905321556 >> 172.21.15.59:0/2905321556 conn(0x7f406c09a9f0 msgr2=0x7f406c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.680+0000 7f407324e640 1 -- 172.21.15.59:0/2905321556 shutdown_connections 2024-03-06T07:34:04.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.680+0000 7f407324e640 1 -- 172.21.15.59:0/2905321556 wait complete. 2024-03-06T07:34:04.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.682+0000 7f407324e640 1 Processor -- start 2024-03-06T07:34:04.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.682+0000 7f407324e640 1 -- start start 2024-03-06T07:34:04.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.683+0000 7f407324e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c0cd8c0 0x7f406c1f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.683+0000 7f407324e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f406c0cfff0 0x7f406c154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.683+0000 7f407324e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f406c15bb20 0x7f406c0b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.683+0000 7f407324e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c0d24b0 con 0x7f406c0cd8c0 2024-03-06T07:34:04.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.683+0000 7f407324e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c0d2330 con 0x7f406c15bb20 2024-03-06T07:34:04.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.683+0000 7f407324e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c0d1980 con 0x7f406c0cfff0 2024-03-06T07:34:04.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.683+0000 7f4070fea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c0cd8c0 0x7f406c1f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.683+0000 7f4070fea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c0cd8c0 0x7f406c1f79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58570/0 (socket says 172.21.15.59:58570) 2024-03-06T07:34:04.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.683+0000 7f4070fea640 1 -- 172.21.15.59:0/2340535412 learned_addr learned my addr 172.21.15.59:0/2340535412 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:04.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.684+0000 7f4070fea640 1 -- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f406c0cfff0 msgr2=0x7f406c154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.684+0000 7f4070fea640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f406c0cfff0 0x7f406c154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.684+0000 7f4070fea640 1 -- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f406c15bb20 msgr2=0x7f406c0b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.693 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.684+0000 7f4070fea640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f406c15bb20 0x7f406c0b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.684+0000 7f4070fea640 1 -- 172.21.15.59:0/2340535412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c0c9730 con 0x7f406c0cd8c0 2024-03-06T07:34:04.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.684+0000 7f4070fea640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c0cd8c0 0x7f406c1f79c0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f4060000c00 tx=0x7f40600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.684+0000 7f40697fa640 1 -- 172.21.15.59:0/2340535412 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060041070 con 0x7f406c0cd8c0 2024-03-06T07:34:04.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.684+0000 7f40697fa640 1 -- 172.21.15.59:0/2340535412 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4060004230 con 0x7f406c0cd8c0 2024-03-06T07:34:04.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.684+0000 7f40697fa640 1 -- 172.21.15.59:0/2340535412 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060035040 con 0x7f406c0cd8c0 2024-03-06T07:34:04.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.684+0000 7f407324e640 1 -- 172.21.15.59:0/2340535412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c0c9a10 con 0x7f406c0cd8c0 2024-03-06T07:34:04.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.684+0000 7f407324e640 1 -- 172.21.15.59:0/2340535412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c0c9fd0 con 0x7f406c0cd8c0 2024-03-06T07:34:04.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.685+0000 7f407324e640 1 -- 172.21.15.59:0/2340535412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c0ca300 con 0x7f406c0cd8c0 2024-03-06T07:34:04.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.686+0000 7f40697fa640 1 -- 172.21.15.59:0/2340535412 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4060038020 con 0x7f406c0cd8c0 2024-03-06T07:34:04.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.686+0000 7f40697fa640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f405006b500 0x7f405006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.686+0000 7f40697fa640 1 -- 172.21.15.59:0/2340535412 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f40600abd00 con 0x7f406c0cd8c0 2024-03-06T07:34:04.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.686+0000 7f406bfff640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f405006b500 0x7f405006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.687+0000 7f406bfff640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f405006b500 0x7f405006d9c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f4058007fd0 tx=0x7f4058007920 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.690+0000 7f40697fa640 1 -- 172.21.15.59:0/2340535412 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f406007ab60 con 0x7f406c0cd8c0 2024-03-06T07:34:04.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.930+0000 7f407324e640 1 -- 172.21.15.59:0/2340535412 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f406c04f330 con 0x7f406c0cd8c0 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.930+0000 7f40697fa640 1 -- 172.21.15.59:0/2340535412 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f406007a460 con 0x7f406c0cd8c0 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:04.933 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:04.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 -- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f405006b500 msgr2=0x7f405006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f405006b500 0x7f405006d9c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f4058007fd0 tx=0x7f4058007920 comp rx=0 tx=0).stop 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 -- 172.21.15.59:0/2340535412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c0cd8c0 msgr2=0x7f406c1f79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c0cd8c0 0x7f406c1f79c0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f4060000c00 tx=0x7f40600046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 -- 172.21.15.59:0/2340535412 shutdown_connections 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f405006b500 0x7f405006d9c0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f406c15bb20 0x7f406c0b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f406c0cfff0 0x7f406c154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 --2- 172.21.15.59:0/2340535412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f406c0cd8c0 0x7f406c1f79c0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 -- 172.21.15.59:0/2340535412 >> 172.21.15.59:0/2340535412 conn(0x7f406c09a9f0 msgr2=0x7f406c0cdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 -- 172.21.15.59:0/2340535412 shutdown_connections 2024-03-06T07:34:04.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:04.932+0000 7f407324e640 1 -- 172.21.15.59:0/2340535412 wait complete. 2024-03-06T07:34:05.146 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:05.146 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:06.147 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:06.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.247+0000 7f57f4b3e640 1 Processor -- start 2024-03-06T07:34:06.249 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.247+0000 7f57f4b3e640 1 -- start start 2024-03-06T07:34:06.250 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.248+0000 7f57f4b3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57f00d22b0 0x7f57f00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.248+0000 7f57f4b3e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f00b3430 con 0x7f57f00b41f0 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.248+0000 7f57f4b3e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57f00b3710 con 0x7f57f015dfe0 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.248+0000 7f57f4b3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f00d2c50 con 0x7f57f00d22b0 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.248+0000 7f57eed9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f57f00b41f0 0x7f57f00b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:57398/0 (socket says 172.21.15.59:57398) 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.248+0000 7f57eed9d640 1 -- 172.21.15.59:0/668052298 learned_addr learned my addr 172.21.15.59:0/668052298 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.248+0000 7f57edd9b640 1 --2- 172.21.15.59:0/668052298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57f00d22b0 0x7f57f00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.248+0000 7f57ed59a640 1 -- 172.21.15.59:0/668052298 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273718877 0 0) 0x7f57f00b3430 con 0x7f57f00b41f0 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.249+0000 7f57ed59a640 1 -- 172.21.15.59:0/668052298 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57dc003520 con 0x7f57f00b41f0 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.249+0000 7f57ed59a640 1 -- 172.21.15.59:0/668052298 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589822021 0 0) 0x7f57f00b3710 con 0x7f57f015dfe0 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.249+0000 7f57edd9b640 1 -- 172.21.15.59:0/668052298 >> v1:172.21.15.164:6789/0 conn(0x7f57f015dfe0 legacy=0x7f57f00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.249+0000 7f57edd9b640 1 -- 172.21.15.59:0/668052298 >> v1:172.21.15.59:6789/0 conn(0x7f57f00b41f0 legacy=0x7f57f00b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.251 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.249+0000 7f57edd9b640 1 -- 172.21.15.59:0/668052298 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f00b39f0 con 0x7f57f00d22b0 2024-03-06T07:34:06.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.250+0000 7f57edd9b640 1 --2- 172.21.15.59:0/668052298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57f00d22b0 0x7f57f00d2710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f57e00097d0 tx=0x7f57e002b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e305a8df85137aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.250+0000 7f57ed59a640 1 -- 172.21.15.59:0/668052298 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0037070 con 0x7f57f00d22b0 2024-03-06T07:34:06.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.250+0000 7f57ed59a640 1 -- 172.21.15.59:0/668052298 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e0030510 con 0x7f57f00d22b0 2024-03-06T07:34:06.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.250+0000 7f57ed59a640 1 -- 172.21.15.59:0/668052298 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e80019f0 con 0x7f57f00d22b0 2024-03-06T07:34:06.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.250+0000 7f57f4b3e640 1 -- 172.21.15.59:0/668052298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57f00d22b0 msgr2=0x7f57f00d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.250+0000 7f57f4b3e640 1 --2- 172.21.15.59:0/668052298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57f00d22b0 0x7f57f00d2710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f57e00097d0 tx=0x7f57e002b730 comp rx=0 tx=0).stop 2024-03-06T07:34:06.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.250+0000 7f57f4b3e640 1 -- 172.21.15.59:0/668052298 shutdown_connections 2024-03-06T07:34:06.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.250+0000 7f57f4b3e640 1 --2- 172.21.15.59:0/668052298 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57f00d22b0 0x7f57f00d2710 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.250+0000 7f57f4b3e640 1 -- 172.21.15.59:0/668052298 >> 172.21.15.59:0/668052298 conn(0x7f57f00d3cd0 msgr2=0x7f57f00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.250+0000 7f57f4b3e640 1 -- 172.21.15.59:0/668052298 shutdown_connections 2024-03-06T07:34:06.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.250+0000 7f57f4b3e640 1 -- 172.21.15.59:0/668052298 wait complete. 2024-03-06T07:34:06.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.252+0000 7f57f4b3e640 1 Processor -- start 2024-03-06T07:34:06.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.252+0000 7f57f4b3e640 1 -- start start 2024-03-06T07:34:06.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.252+0000 7f57f4b3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57f00b41f0 0x7f57f015c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.252+0000 7f57f4b3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57f00d22b0 0x7f57f00cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.252+0000 7f57f4b3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57f015dfe0 0x7f57f006ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.252+0000 7f57f4b3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f00b3d60 con 0x7f57f00d22b0 2024-03-06T07:34:06.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.252+0000 7f57f4b3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f00b31e0 con 0x7f57f00b41f0 2024-03-06T07:34:06.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.252+0000 7f57f4b3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57f00d1c70 con 0x7f57f015dfe0 2024-03-06T07:34:06.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.252+0000 7f57edd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57f00d22b0 0x7f57f00cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57edd9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57f00d22b0 0x7f57f00cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58574/0 (socket says 172.21.15.59:58574) 2024-03-06T07:34:06.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57edd9b640 1 -- 172.21.15.59:0/633395678 learned_addr learned my addr 172.21.15.59:0/633395678 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:06.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57eed9d640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57f015dfe0 0x7f57f006ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57ee59c640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57f00b41f0 0x7f57f015c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57edd9b640 1 -- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57f015dfe0 msgr2=0x7f57f006ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57edd9b640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57f015dfe0 0x7f57f006ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57edd9b640 1 -- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57f00b41f0 msgr2=0x7f57f015c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57edd9b640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57f00b41f0 0x7f57f015c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57edd9b640 1 -- 172.21.15.59:0/633395678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f00cc840 con 0x7f57f00d22b0 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57edd9b640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57f00d22b0 0x7f57f00cae50 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f57e00097d0 tx=0x7f57e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57e6ffd640 1 -- 172.21.15.59:0/633395678 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0042070 con 0x7f57f00d22b0 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57e6ffd640 1 -- 172.21.15.59:0/633395678 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e0004220 con 0x7f57f00d22b0 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57f4b3e640 1 -- 172.21.15.59:0/633395678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f00cb800 con 0x7f57f00d22b0 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57e6ffd640 1 -- 172.21.15.59:0/633395678 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0037040 con 0x7f57f00d22b0 2024-03-06T07:34:06.257 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.253+0000 7f57f4b3e640 1 -- 172.21.15.59:0/633395678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f00cc040 con 0x7f57f00d22b0 2024-03-06T07:34:06.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.255+0000 7f57f4b3e640 1 -- 172.21.15.59:0/633395678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f004f330 con 0x7f57f00d22b0 2024-03-06T07:34:06.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.255+0000 7f57e6ffd640 1 -- 172.21.15.59:0/633395678 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57e003a040 con 0x7f57f00d22b0 2024-03-06T07:34:06.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.255+0000 7f57e6ffd640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57d006b4d0 0x7f57d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.255+0000 7f57e6ffd640 1 -- 172.21.15.59:0/633395678 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f57e00abad0 con 0x7f57f00d22b0 2024-03-06T07:34:06.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.255+0000 7f57ee59c640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57d006b4d0 0x7f57d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.258 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.256+0000 7f57ee59c640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57d006b4d0 0x7f57d006d990 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f57d8000f60 tx=0x7f57d8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.263 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.259+0000 7f57e6ffd640 1 -- 172.21.15.59:0/633395678 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57f00cb800 con 0x7f57f00d22b0 2024-03-06T07:34:06.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.579+0000 7f57f4b3e640 1 -- 172.21.15.59:0/633395678 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57f00b9b90 con 0x7f57f00d22b0 2024-03-06T07:34:06.581 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.580+0000 7f57e6ffd640 1 -- 172.21.15.59:0/633395678 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f57e007a930 con 0x7f57f00d22b0 2024-03-06T07:34:06.582 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:06.582 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:06.582 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:06.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.581+0000 7f57f4b3e640 1 -- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57d006b4d0 msgr2=0x7f57d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.581+0000 7f57f4b3e640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57d006b4d0 0x7f57d006d990 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f57d8000f60 tx=0x7f57d8009040 comp rx=0 tx=0).stop 2024-03-06T07:34:06.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.581+0000 7f57f4b3e640 1 -- 172.21.15.59:0/633395678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57f00d22b0 msgr2=0x7f57f00cae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.581+0000 7f57f4b3e640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57f00d22b0 0x7f57f00cae50 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f57e00097d0 tx=0x7f57e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.581+0000 7f57f4b3e640 1 -- 172.21.15.59:0/633395678 shutdown_connections 2024-03-06T07:34:06.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.581+0000 7f57f4b3e640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f57d006b4d0 0x7f57d006d990 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.581+0000 7f57f4b3e640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57f015dfe0 0x7f57f006ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.581+0000 7f57f4b3e640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57f00d22b0 0x7f57f00cae50 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.581+0000 7f57f4b3e640 1 --2- 172.21.15.59:0/633395678 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57f00b41f0 0x7f57f015c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.581+0000 7f57f4b3e640 1 -- 172.21.15.59:0/633395678 >> 172.21.15.59:0/633395678 conn(0x7f57f00d3cd0 msgr2=0x7f57f01603d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.582+0000 7f57f4b3e640 1 -- 172.21.15.59:0/633395678 shutdown_connections 2024-03-06T07:34:06.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:06.582+0000 7f57f4b3e640 1 -- 172.21.15.59:0/633395678 wait complete. 2024-03-06T07:34:06.796 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:06.796 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:07.797 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.897+0000 7f9b55b48640 1 Processor -- start 2024-03-06T07:34:07.899 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.898+0000 7f9b55b48640 1 -- start start 2024-03-06T07:34:07.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.898+0000 7f9b55b48640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b5015dfe0 0x7f9b500cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.898+0000 7f9b55b48640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9b500cf830 0x7f9b500d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.898+0000 7f9b55b48640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b500b3430 con 0x7f9b500d27f0 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.898+0000 7f9b55b48640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b500d2c50 con 0x7f9b500cf830 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.898+0000 7f9b55b48640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b500b31e0 con 0x7f9b5015dfe0 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.898+0000 7f9b4effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9b500cf830 0x7f9b500d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.898+0000 7f9b4effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9b500cf830 0x7f9b500d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58586/0 (socket says 172.21.15.59:58586) 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.898+0000 7f9b4effd640 1 -- 172.21.15.59:0/3241155228 learned_addr learned my addr 172.21.15.59:0/3241155228 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.899+0000 7f9b4f7fe640 1 --2- 172.21.15.59:0/3241155228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b5015dfe0 0x7f9b500cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.899+0000 7f9b4effd640 1 -- 172.21.15.59:0/3241155228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b5015dfe0 msgr2=0x7f9b500cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.899+0000 7f9b4effd640 1 --2- 172.21.15.59:0/3241155228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b5015dfe0 0x7f9b500cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.899+0000 7f9b4effd640 1 -- 172.21.15.59:0/3241155228 >> v1:172.21.15.164:6789/0 conn(0x7f9b500d27f0 legacy=0x7f9b500d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.899+0000 7f9b4effd640 1 -- 172.21.15.59:0/3241155228 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b500b3710 con 0x7f9b500cf830 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.899+0000 7f9b4f7fe640 1 --2- 172.21.15.59:0/3241155228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b5015dfe0 0x7f9b500cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:07.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.899+0000 7f9b4effd640 1 --2- 172.21.15.59:0/3241155228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b500cf830 0x7f9b500d22b0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009870 tx=0x7f9b4402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df82b6a17f40e4f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.900+0000 7f9b4e7fc640 1 -- 172.21.15.59:0/3241155228 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b44035020 con 0x7f9b500cf830 2024-03-06T07:34:07.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.900+0000 7f9b4e7fc640 1 -- 172.21.15.59:0/3241155228 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b44031640 con 0x7f9b500cf830 2024-03-06T07:34:07.903 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.900+0000 7f9b4e7fc640 1 -- 172.21.15.59:0/3241155228 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4402d040 con 0x7f9b500cf830 2024-03-06T07:34:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.900+0000 7f9b55b48640 1 -- 172.21.15.59:0/3241155228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b500cf830 msgr2=0x7f9b500d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.900+0000 7f9b55b48640 1 --2- 172.21.15.59:0/3241155228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b500cf830 0x7f9b500d22b0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009870 tx=0x7f9b4402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.900+0000 7f9b55b48640 1 -- 172.21.15.59:0/3241155228 shutdown_connections 2024-03-06T07:34:07.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.900+0000 7f9b55b48640 1 --2- 172.21.15.59:0/3241155228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b500cf830 0x7f9b500d22b0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.900+0000 7f9b55b48640 1 --2- 172.21.15.59:0/3241155228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b5015dfe0 0x7f9b500cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.900+0000 7f9b55b48640 1 -- 172.21.15.59:0/3241155228 >> 172.21.15.59:0/3241155228 conn(0x7f9b500d3cd0 msgr2=0x7f9b500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.900+0000 7f9b55b48640 1 -- 172.21.15.59:0/3241155228 shutdown_connections 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.900+0000 7f9b55b48640 1 -- 172.21.15.59:0/3241155228 wait complete. 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.901+0000 7f9b55b48640 1 Processor -- start 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.902+0000 7f9b55b48640 1 -- start start 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.902+0000 7f9b55b48640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b500cf830 0x7f9b50155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.902+0000 7f9b55b48640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b500d27f0 0x7f9b5015f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.902+0000 7f9b55b48640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b5015dfe0 0x7f9b5015fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.902+0000 7f9b55b48640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b500d1c70 con 0x7f9b500cf830 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.902+0000 7f9b55b48640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b500b3d60 con 0x7f9b5015dfe0 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.902+0000 7f9b55b48640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b500d1df0 con 0x7f9b500d27f0 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.902+0000 7f9b4f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b500cf830 0x7f9b50155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.902+0000 7f9b4f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b500cf830 0x7f9b50155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58592/0 (socket says 172.21.15.59:58592) 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b4f7fe640 1 -- 172.21.15.59:0/102690801 learned_addr learned my addr 172.21.15.59:0/102690801 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b4effd640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b500d27f0 0x7f9b5015f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b4f7fe640 1 -- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b500d27f0 msgr2=0x7f9b5015f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b4ffff640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b5015dfe0 0x7f9b5015fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b4f7fe640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b500d27f0 0x7f9b5015f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b4f7fe640 1 -- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b5015dfe0 msgr2=0x7f9b5015fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b4f7fe640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b5015dfe0 0x7f9b5015fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b4f7fe640 1 -- 172.21.15.59:0/102690801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b501600f0 con 0x7f9b500cf830 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b4effd640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b500d27f0 0x7f9b5015f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b4f7fe640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b500cf830 0x7f9b50155180 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000c7f0 tx=0x7f9b4000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b37fff640 1 -- 172.21.15.59:0/102690801 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b40015070 con 0x7f9b500cf830 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b37fff640 1 -- 172.21.15.59:0/102690801 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b40007820 con 0x7f9b500cf830 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b55b48640 1 -- 172.21.15.59:0/102690801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b500cada0 con 0x7f9b500cf830 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b37fff640 1 -- 172.21.15.59:0/102690801 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b40007bd0 con 0x7f9b500cf830 2024-03-06T07:34:07.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.903+0000 7f9b55b48640 1 -- 172.21.15.59:0/102690801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b500cb360 con 0x7f9b500cf830 2024-03-06T07:34:07.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.904+0000 7f9b55b48640 1 -- 172.21.15.59:0/102690801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5004f330 con 0x7f9b500cf830 2024-03-06T07:34:07.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.905+0000 7f9b37fff640 1 -- 172.21.15.59:0/102690801 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b40018070 con 0x7f9b500cf830 2024-03-06T07:34:07.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.905+0000 7f9b37fff640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b3c06b4d0 0x7f9b3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:07.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.905+0000 7f9b37fff640 1 -- 172.21.15.59:0/102690801 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9b4008a0b0 con 0x7f9b500cf830 2024-03-06T07:34:07.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.905+0000 7f9b4effd640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b3c06b4d0 0x7f9b3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:07.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.906+0000 7f9b4effd640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b3c06b4d0 0x7f9b3c06d990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f9b44036000 tx=0x7f9b44008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:07.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:07.909+0000 7f9b37fff640 1 -- 172.21.15.59:0/102690801 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b40058fa0 con 0x7f9b500cf830 2024-03-06T07:34:08.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.139+0000 7f9b55b48640 1 -- 172.21.15.59:0/102690801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b500ca540 con 0x7f9b500cf830 2024-03-06T07:34:08.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.140+0000 7f9b37fff640 1 -- 172.21.15.59:0/102690801 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f9b400588a0 con 0x7f9b500cf830 2024-03-06T07:34:08.142 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:08.142 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:08.142 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:08.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 -- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b3c06b4d0 msgr2=0x7f9b3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b3c06b4d0 0x7f9b3c06d990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f9b44036000 tx=0x7f9b44008030 comp rx=0 tx=0).stop 2024-03-06T07:34:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 -- 172.21.15.59:0/102690801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b500cf830 msgr2=0x7f9b50155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b500cf830 0x7f9b50155180 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000c7f0 tx=0x7f9b4000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 -- 172.21.15.59:0/102690801 shutdown_connections 2024-03-06T07:34:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9b3c06b4d0 0x7f9b3c06d990 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b5015dfe0 0x7f9b5015fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b500d27f0 0x7f9b5015f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 --2- 172.21.15.59:0/102690801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9b500cf830 0x7f9b50155180 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 -- 172.21.15.59:0/102690801 >> 172.21.15.59:0/102690801 conn(0x7f9b500d3cd0 msgr2=0x7f9b500b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 -- 172.21.15.59:0/102690801 shutdown_connections 2024-03-06T07:34:08.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:08.142+0000 7f9b55b48640 1 -- 172.21.15.59:0/102690801 wait complete. 2024-03-06T07:34:08.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:08.358 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:09.359 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:09.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.459+0000 7ff2d9b5c640 1 Processor -- start 2024-03-06T07:34:09.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.459+0000 7ff2d9b5c640 1 -- start start 2024-03-06T07:34:09.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.460+0000 7ff2d9b5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d415bdd0 0x7ff2d40c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.460+0000 7ff2d9b5c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff2d415fa10 0x7ff2d40ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.460+0000 7ff2d9b5c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d40cebe0 con 0x7ff2d40d16e0 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.460+0000 7ff2d9b5c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d40ceec0 con 0x7ff2d415fa10 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.460+0000 7ff2d9b5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d40cf1a0 con 0x7ff2d415bdd0 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.460+0000 7ff2d2ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff2d415fa10 0x7ff2d40ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.460+0000 7ff2d37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d415bdd0 0x7ff2d40c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.460+0000 7ff2d2ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff2d415fa10 0x7ff2d40ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.59:50528/0 (socket says 172.21.15.59:50528) 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.460+0000 7ff2d2ffd640 1 -- 172.21.15.59:0/921956630 learned_addr learned my addr 172.21.15.59:0/921956630 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.461+0000 7ff2d2ffd640 1 -- 172.21.15.59:0/921956630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d415bdd0 msgr2=0x7ff2d40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.461+0000 7ff2d2ffd640 1 --2- 172.21.15.59:0/921956630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d415bdd0 0x7ff2d40c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.461+0000 7ff2d2ffd640 1 -- 172.21.15.59:0/921956630 >> v1:172.21.15.164:6789/0 conn(0x7ff2d40d16e0 legacy=0x7ff2d40d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.461+0000 7ff2d2ffd640 1 -- 172.21.15.59:0/921956630 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d40cc8e0 con 0x7ff2d415fa10 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.461+0000 7ff2d27fc640 1 -- 172.21.15.59:0/921956630 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410683849 0 0) 0x7ff2d40cebe0 con 0x7ff2d40d16e0 2024-03-06T07:34:09.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.461+0000 7ff2d37fe640 1 --2- 172.21.15.59:0/921956630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d415bdd0 0x7ff2d40c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:09.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.461+0000 7ff2d2ffd640 1 --2- 172.21.15.59:0/921956630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d415fa10 0x7ff2d40ce270 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009870 tx=0x7ff2bc02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=809734100b5b8c12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.462+0000 7ff2d27fc640 1 -- 172.21.15.59:0/921956630 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc035020 con 0x7ff2d415fa10 2024-03-06T07:34:09.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.462+0000 7ff2d27fc640 1 -- 172.21.15.59:0/921956630 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2bc031640 con 0x7ff2d415fa10 2024-03-06T07:34:09.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.462+0000 7ff2d27fc640 1 -- 172.21.15.59:0/921956630 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc02d040 con 0x7ff2d415fa10 2024-03-06T07:34:09.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.462+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/921956630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d415fa10 msgr2=0x7ff2d40ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.462+0000 7ff2d9b5c640 1 --2- 172.21.15.59:0/921956630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d415fa10 0x7ff2d40ce270 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009870 tx=0x7ff2bc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.462+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/921956630 shutdown_connections 2024-03-06T07:34:09.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.462+0000 7ff2d9b5c640 1 --2- 172.21.15.59:0/921956630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d415fa10 0x7ff2d40ce270 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.462+0000 7ff2d9b5c640 1 --2- 172.21.15.59:0/921956630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d415bdd0 0x7ff2d40c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.462+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/921956630 >> 172.21.15.59:0/921956630 conn(0x7ff2d409a9f0 msgr2=0x7ff2d409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.462+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/921956630 shutdown_connections 2024-03-06T07:34:09.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.462+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/921956630 wait complete. 2024-03-06T07:34:09.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.463+0000 7ff2d9b5c640 1 Processor -- start 2024-03-06T07:34:09.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.464+0000 7ff2d9b5c640 1 -- start start 2024-03-06T07:34:09.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.464+0000 7ff2d9b5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d40d16e0 0x7ff2d415cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.464+0000 7ff2d9b5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d415bdd0 0x7ff2d406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.464+0000 7ff2d9b5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d415fa10 0x7ff2d415a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.464+0000 7ff2d9b5c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d415c1b0 con 0x7ff2d40d16e0 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.464+0000 7ff2d9b5c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d40cc5d0 con 0x7ff2d415fa10 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.464+0000 7ff2d9b5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d40ce7b0 con 0x7ff2d415bdd0 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.464+0000 7ff2d37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d40d16e0 0x7ff2d415cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.464+0000 7ff2d37fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d40d16e0 0x7ff2d415cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58598/0 (socket says 172.21.15.59:58598) 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.464+0000 7ff2d37fe640 1 -- 172.21.15.59:0/1466667836 learned_addr learned my addr 172.21.15.59:0/1466667836 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2d2ffd640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d415bdd0 0x7ff2d406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2d3fff640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d415fa10 0x7ff2d415a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2d37fe640 1 -- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d415bdd0 msgr2=0x7ff2d406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2d37fe640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d415bdd0 0x7ff2d406caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2d37fe640 1 -- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d415fa10 msgr2=0x7ff2d415a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2d37fe640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d415fa10 0x7ff2d415a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2d37fe640 1 -- 172.21.15.59:0/1466667836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d406d570 con 0x7ff2d40d16e0 2024-03-06T07:34:09.468 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2d3fff640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d415fa10 0x7ff2d415a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:09.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2d37fe640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d40d16e0 0x7ff2d415cd00 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400ebe0 tx=0x7ff2c400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2cbfff640 1 -- 172.21.15.59:0/1466667836 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4017070 con 0x7ff2d40d16e0 2024-03-06T07:34:09.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2cbfff640 1 -- 172.21.15.59:0/1466667836 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c400cc30 con 0x7ff2d40d16e0 2024-03-06T07:34:09.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/1466667836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d406d850 con 0x7ff2d40d16e0 2024-03-06T07:34:09.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2cbfff640 1 -- 172.21.15.59:0/1466667836 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4013470 con 0x7ff2d40d16e0 2024-03-06T07:34:09.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.465+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/1466667836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d40c9e50 con 0x7ff2d40d16e0 2024-03-06T07:34:09.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.466+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/1466667836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d404f330 con 0x7ff2d40d16e0 2024-03-06T07:34:09.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.467+0000 7ff2cbfff640 1 -- 172.21.15.59:0/1466667836 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2c401a070 con 0x7ff2d40d16e0 2024-03-06T07:34:09.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.467+0000 7ff2cbfff640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b406b4d0 0x7ff2b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.467+0000 7ff2cbfff640 1 -- 172.21.15.59:0/1466667836 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff2c408a500 con 0x7ff2d40d16e0 2024-03-06T07:34:09.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.467+0000 7ff2d2ffd640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b406b4d0 0x7ff2b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.468+0000 7ff2d2ffd640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b406b4d0 0x7ff2b406d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc036000 tx=0x7ff2bc03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.471+0000 7ff2cbfff640 1 -- 172.21.15.59:0/1466667836 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2c4058c90 con 0x7ff2d40d16e0 2024-03-06T07:34:09.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.689+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/1466667836 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2d40b7a40 con 0x7ff2d40d16e0 2024-03-06T07:34:09.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.690+0000 7ff2cbfff640 1 -- 172.21.15.59:0/1466667836 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7ff2c405cd30 con 0x7ff2d40d16e0 2024-03-06T07:34:09.692 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:09.692 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:09.693 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:09.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b406b4d0 msgr2=0x7ff2b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b406b4d0 0x7ff2b406d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc036000 tx=0x7ff2bc03c040 comp rx=0 tx=0).stop 2024-03-06T07:34:09.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/1466667836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d40d16e0 msgr2=0x7ff2d415cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d40d16e0 0x7ff2d415cd00 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400ebe0 tx=0x7ff2c400c520 comp rx=0 tx=0).stop 2024-03-06T07:34:09.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/1466667836 shutdown_connections 2024-03-06T07:34:09.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7ff2b406b4d0 0x7ff2b406d990 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d415fa10 0x7ff2d415a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d415bdd0 0x7ff2d406caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 --2- 172.21.15.59:0/1466667836 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff2d40d16e0 0x7ff2d415cd00 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/1466667836 >> 172.21.15.59:0/1466667836 conn(0x7ff2d409a9f0 msgr2=0x7ff2d409add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/1466667836 shutdown_connections 2024-03-06T07:34:09.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:09.692+0000 7ff2d9b5c640 1 -- 172.21.15.59:0/1466667836 wait complete. 2024-03-06T07:34:09.908 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:09.909 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:10.910 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:11.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.009+0000 7faf80840640 1 Processor -- start 2024-03-06T07:34:11.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.009+0000 7faf80840640 1 -- start start 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.010+0000 7faf80840640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faf7c0d22b0 0x7faf7c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.010+0000 7faf80840640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7faf7c0b41f0 0x7faf7c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.010+0000 7faf80840640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c0b3430 con 0x7faf7c15dfe0 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.010+0000 7faf80840640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c0d2c50 con 0x7faf7c0b41f0 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.010+0000 7faf80840640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c0b31e0 con 0x7faf7c0d22b0 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.010+0000 7faf7ad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7faf7c0b41f0 0x7faf7c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.010+0000 7faf7ad9d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7faf7c0b41f0 0x7faf7c0b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58612/0 (socket says 172.21.15.59:58612) 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.010+0000 7faf7ad9d640 1 -- 172.21.15.59:0/3174952310 learned_addr learned my addr 172.21.15.59:0/3174952310 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.011+0000 7faf73fff640 1 --2- 172.21.15.59:0/3174952310 >> v2:172.21.15.164:3300/0 conn(0x7faf7c0d22b0 0x7faf7c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.011+0000 7faf7ad9d640 1 -- 172.21.15.59:0/3174952310 >> v2:172.21.15.164:3300/0 conn(0x7faf7c0d22b0 msgr2=0x7faf7c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.011+0000 7faf7ad9d640 1 --2- 172.21.15.59:0/3174952310 >> v2:172.21.15.164:3300/0 conn(0x7faf7c0d22b0 0x7faf7c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.011+0000 7faf7ad9d640 1 -- 172.21.15.59:0/3174952310 >> v1:172.21.15.164:6789/0 conn(0x7faf7c15dfe0 legacy=0x7faf7c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.011+0000 7faf7ad9d640 1 -- 172.21.15.59:0/3174952310 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c0b3710 con 0x7faf7c0b41f0 2024-03-06T07:34:11.013 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.011+0000 7faf73fff640 1 --2- 172.21.15.59:0/3174952310 >> v2:172.21.15.164:3300/0 conn(0x7faf7c0d22b0 0x7faf7c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:11.014 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.011+0000 7faf7ad9d640 1 --2- 172.21.15.59:0/3174952310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf7c0b41f0 0x7faf7c0b4680 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7faf74009e50 tx=0x7faf7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7480b8738b9a2ba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.011+0000 7faf79d9b640 1 -- 172.21.15.59:0/3174952310 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7402ca80 con 0x7faf7c0b41f0 2024-03-06T07:34:11.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.012+0000 7faf79d9b640 1 -- 172.21.15.59:0/3174952310 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7402cc20 con 0x7faf7c0b41f0 2024-03-06T07:34:11.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.012+0000 7faf79d9b640 1 -- 172.21.15.59:0/3174952310 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf740343f0 con 0x7faf7c0b41f0 2024-03-06T07:34:11.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.012+0000 7faf80840640 1 -- 172.21.15.59:0/3174952310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf7c0b41f0 msgr2=0x7faf7c0b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.012+0000 7faf80840640 1 --2- 172.21.15.59:0/3174952310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf7c0b41f0 0x7faf7c0b4680 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7faf74009e50 tx=0x7faf7402bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.012+0000 7faf80840640 1 -- 172.21.15.59:0/3174952310 shutdown_connections 2024-03-06T07:34:11.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.012+0000 7faf80840640 1 --2- 172.21.15.59:0/3174952310 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf7c0b41f0 0x7faf7c0b4680 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.012+0000 7faf80840640 1 --2- 172.21.15.59:0/3174952310 >> v2:172.21.15.164:3300/0 conn(0x7faf7c0d22b0 0x7faf7c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.012+0000 7faf80840640 1 -- 172.21.15.59:0/3174952310 >> 172.21.15.59:0/3174952310 conn(0x7faf7c0d3cd0 msgr2=0x7faf7c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.012+0000 7faf80840640 1 -- 172.21.15.59:0/3174952310 shutdown_connections 2024-03-06T07:34:11.015 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.012+0000 7faf80840640 1 -- 172.21.15.59:0/3174952310 wait complete. 2024-03-06T07:34:11.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.013+0000 7faf80840640 1 Processor -- start 2024-03-06T07:34:11.017 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.013+0000 7faf80840640 1 -- start start 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.014+0000 7faf80840640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf7c0b41f0 0x7faf7c161150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.014+0000 7faf80840640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf7c0d22b0 0x7faf7c0cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.014+0000 7faf80840640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf7c15dfe0 0x7faf7c0b9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.014+0000 7faf80840640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0cfa00 con 0x7faf7c0b41f0 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.014+0000 7faf80840640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0b3d60 con 0x7faf7c15dfe0 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.014+0000 7faf80840640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0d1c70 con 0x7faf7c0d22b0 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.014+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf7c0b41f0 0x7faf7c161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.014+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf7c0b41f0 0x7faf7c161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58628/0 (socket says 172.21.15.59:58628) 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.014+0000 7faf7a59c640 1 -- 172.21.15.59:0/3426013369 learned_addr learned my addr 172.21.15.59:0/3426013369 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.014+0000 7faf7ad9d640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf7c15dfe0 0x7faf7c0b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.014+0000 7faf73fff640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf7c0d22b0 0x7faf7c0cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf7a59c640 1 -- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf7c0d22b0 msgr2=0x7faf7c0cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf7a59c640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf7c0d22b0 0x7faf7c0cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf7a59c640 1 -- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf7c15dfe0 msgr2=0x7faf7c0b9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.018 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf7a59c640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf7c15dfe0 0x7faf7c0b9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf7a59c640 1 -- 172.21.15.59:0/3426013369 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c0cc250 con 0x7faf7c0b41f0 2024-03-06T07:34:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf73fff640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf7c0d22b0 0x7faf7c0cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf7a59c640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf7c0b41f0 0x7faf7c161150 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00b3f0 tx=0x7faf6c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf72ffd640 1 -- 172.21.15.59:0/3426013369 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c015020 con 0x7faf7c0b41f0 2024-03-06T07:34:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf72ffd640 1 -- 172.21.15.59:0/3426013369 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c011830 con 0x7faf7c0b41f0 2024-03-06T07:34:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf80840640 1 -- 172.21.15.59:0/3426013369 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7c0cc530 con 0x7faf7c0b41f0 2024-03-06T07:34:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf72ffd640 1 -- 172.21.15.59:0/3426013369 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c011c20 con 0x7faf7c0b41f0 2024-03-06T07:34:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.015+0000 7faf80840640 1 -- 172.21.15.59:0/3426013369 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7c0ccaf0 con 0x7faf7c0b41f0 2024-03-06T07:34:11.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.016+0000 7faf80840640 1 -- 172.21.15.59:0/3426013369 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7c04f330 con 0x7faf7c0b41f0 2024-03-06T07:34:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.016+0000 7faf72ffd640 1 -- 172.21.15.59:0/3426013369 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf6c018020 con 0x7faf7c0b41f0 2024-03-06T07:34:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.017+0000 7faf72ffd640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf5c06b4d0 0x7faf5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.017+0000 7faf72ffd640 1 -- 172.21.15.59:0/3426013369 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faf6c089d60 con 0x7faf7c0b41f0 2024-03-06T07:34:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.017+0000 7faf73fff640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf5c06b4d0 0x7faf5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.020 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.018+0000 7faf73fff640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf5c06b4d0 0x7faf5c06d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7faf7c06c980 tx=0x7faf64008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.021+0000 7faf72ffd640 1 -- 172.21.15.59:0/3426013369 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf6c01f030 con 0x7faf7c0b41f0 2024-03-06T07:34:11.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.239+0000 7faf80840640 1 -- 172.21.15.59:0/3426013369 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf7c15d500 con 0x7faf7c0b41f0 2024-03-06T07:34:11.241 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.239+0000 7faf72ffd640 1 -- 172.21.15.59:0/3426013369 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7faf6c058c50 con 0x7faf7c0b41f0 2024-03-06T07:34:11.241 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:11.241 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:11.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 -- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf5c06b4d0 msgr2=0x7faf5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf5c06b4d0 0x7faf5c06d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7faf7c06c980 tx=0x7faf64008040 comp rx=0 tx=0).stop 2024-03-06T07:34:11.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 -- 172.21.15.59:0/3426013369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf7c0b41f0 msgr2=0x7faf7c161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf7c0b41f0 0x7faf7c161150 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00b3f0 tx=0x7faf6c00b8b0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 -- 172.21.15.59:0/3426013369 shutdown_connections 2024-03-06T07:34:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faf5c06b4d0 0x7faf5c06d990 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf7c15dfe0 0x7faf7c0b9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf7c0d22b0 0x7faf7c0cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 --2- 172.21.15.59:0/3426013369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf7c0b41f0 0x7faf7c161150 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 -- 172.21.15.59:0/3426013369 >> 172.21.15.59:0/3426013369 conn(0x7faf7c0d3cd0 msgr2=0x7faf7c0d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 -- 172.21.15.59:0/3426013369 shutdown_connections 2024-03-06T07:34:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:11.241+0000 7faf80840640 1 -- 172.21.15.59:0/3426013369 wait complete. 2024-03-06T07:34:11.455 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:11.455 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:12.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:12.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.558+0000 7fc7a5d7a640 1 Processor -- start 2024-03-06T07:34:12.560 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.558+0000 7fc7a5d7a640 1 -- start start 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc7a5d7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7a0158920 0x7fc7a015c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc7a5d7a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc7a00d14f0 0x7fc7a006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc7a5d7a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc7a00af6c0 0x7fc7a00afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc7a5d7a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a006fae0 con 0x7fc7a00d14f0 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc7a5d7a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a015f5f0 con 0x7fc7a00af6c0 2024-03-06T07:34:12.561 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc7a5d7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a015e740 con 0x7fc7a0158920 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc79effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc7a00d14f0 0x7fc7a006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc79effd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc7a00d14f0 0x7fc7a006f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58638/0 (socket says 172.21.15.59:58638) 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc79effd640 1 -- 172.21.15.59:0/2808056946 learned_addr learned my addr 172.21.15.59:0/2808056946 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc79f7fe640 1 --2- 172.21.15.59:0/2808056946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7a0158920 0x7fc7a015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc79ffff640 1 --2- 172.21.15.59:0/2808056946 >> v2:172.21.15.164:3300/0 conn(0x7fc7a00af6c0 0x7fc7a00afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc79effd640 1 -- 172.21.15.59:0/2808056946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7a0158920 msgr2=0x7fc7a015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc79effd640 1 --2- 172.21.15.59:0/2808056946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7a0158920 0x7fc7a015c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc79effd640 1 -- 172.21.15.59:0/2808056946 >> v2:172.21.15.164:3300/0 conn(0x7fc7a00af6c0 msgr2=0x7fc7a00afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc79effd640 1 --2- 172.21.15.59:0/2808056946 >> v2:172.21.15.164:3300/0 conn(0x7fc7a00af6c0 0x7fc7a00afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc79effd640 1 -- 172.21.15.59:0/2808056946 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a015e9e0 con 0x7fc7a00d14f0 2024-03-06T07:34:12.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.559+0000 7fc79f7fe640 1 --2- 172.21.15.59:0/2808056946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7a0158920 0x7fc7a015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:12.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.560+0000 7fc79effd640 1 --2- 172.21.15.59:0/2808056946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00d14f0 0x7fc7a006f5a0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0072b30 tx=0x7fc79402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6620999334044e7e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.560+0000 7fc79e7fc640 1 -- 172.21.15.59:0/2808056946 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794035020 con 0x7fc7a00d14f0 2024-03-06T07:34:12.563 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.560+0000 7fc79e7fc640 1 -- 172.21.15.59:0/2808056946 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc794031870 con 0x7fc7a00d14f0 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.560+0000 7fc79e7fc640 1 -- 172.21.15.59:0/2808056946 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794031c40 con 0x7fc7a00d14f0 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.561+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2808056946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00d14f0 msgr2=0x7fc7a006f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.561+0000 7fc7a5d7a640 1 --2- 172.21.15.59:0/2808056946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00d14f0 0x7fc7a006f5a0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0072b30 tx=0x7fc79402b910 comp rx=0 tx=0).stop 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.561+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2808056946 shutdown_connections 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.561+0000 7fc7a5d7a640 1 --2- 172.21.15.59:0/2808056946 >> v2:172.21.15.164:3300/0 conn(0x7fc7a00af6c0 0x7fc7a00afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.561+0000 7fc7a5d7a640 1 --2- 172.21.15.59:0/2808056946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00d14f0 0x7fc7a006f5a0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.561+0000 7fc7a5d7a640 1 --2- 172.21.15.59:0/2808056946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7a0158920 0x7fc7a015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.561+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2808056946 >> 172.21.15.59:0/2808056946 conn(0x7fc7a00d0960 msgr2=0x7fc7a00d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.561+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2808056946 shutdown_connections 2024-03-06T07:34:12.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.561+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2808056946 wait complete. 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.563+0000 7fc7a5d7a640 1 Processor -- start 2024-03-06T07:34:12.565 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.563+0000 7fc7a5d7a640 1 -- start start 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.564+0000 7fc7a5d7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7a00af6c0 0x7fc7a014e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.564+0000 7fc7a5d7a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00d14f0 0x7fc7a01f5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.564+0000 7fc7a5d7a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7a0158920 0x7fc7a0156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.564+0000 7fc7a5d7a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a015f280 con 0x7fc7a00d14f0 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.564+0000 7fc7a5d7a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a015efa0 con 0x7fc7a0158920 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.564+0000 7fc7a5d7a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a015b400 con 0x7fc7a00af6c0 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.564+0000 7fc79effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00d14f0 0x7fc7a01f5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.564+0000 7fc79effd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00d14f0 0x7fc7a01f5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58652/0 (socket says 172.21.15.59:58652) 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.564+0000 7fc79effd640 1 -- 172.21.15.59:0/2485744651 learned_addr learned my addr 172.21.15.59:0/2485744651 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.564+0000 7fc79f7fe640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7a00af6c0 0x7fc7a014e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.564+0000 7fc79ffff640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7a0158920 0x7fc7a0156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.565+0000 7fc79effd640 1 -- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7a00af6c0 msgr2=0x7fc7a014e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.565+0000 7fc79effd640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7a00af6c0 0x7fc7a014e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.565+0000 7fc79effd640 1 -- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7a0158920 msgr2=0x7fc7a0156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.565+0000 7fc79effd640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7a0158920 0x7fc7a0156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.565+0000 7fc79effd640 1 -- 172.21.15.59:0/2485744651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a00ad900 con 0x7fc7a00d14f0 2024-03-06T07:34:12.568 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.565+0000 7fc79effd640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00d14f0 0x7fc7a01f5200 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fc794000c00 tx=0x7fc7940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.565+0000 7fc787fff640 1 -- 172.21.15.59:0/2485744651 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794040070 con 0x7fc7a00d14f0 2024-03-06T07:34:12.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.565+0000 7fc787fff640 1 -- 172.21.15.59:0/2485744651 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79403a420 con 0x7fc7a00d14f0 2024-03-06T07:34:12.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.565+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2485744651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a00adbe0 con 0x7fc7a00d14f0 2024-03-06T07:34:12.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.565+0000 7fc787fff640 1 -- 172.21.15.59:0/2485744651 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794035040 con 0x7fc7a00d14f0 2024-03-06T07:34:12.569 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.565+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2485744651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a00ae1a0 con 0x7fc7a00d14f0 2024-03-06T07:34:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.566+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2485744651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a004f330 con 0x7fc7a00d14f0 2024-03-06T07:34:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.566+0000 7fc787fff640 1 -- 172.21.15.59:0/2485744651 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc79404b060 con 0x7fc7a00d14f0 2024-03-06T07:34:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.567+0000 7fc787fff640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc78c06b4d0 0x7fc78c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.567+0000 7fc787fff640 1 -- 172.21.15.59:0/2485744651 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc7940ac130 con 0x7fc7a00d14f0 2024-03-06T07:34:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.567+0000 7fc79f7fe640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc78c06b4d0 0x7fc78c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.570 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.568+0000 7fc79f7fe640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc78c06b4d0 0x7fc78c06d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00b5760 tx=0x7fc790008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.571+0000 7fc787fff640 1 -- 172.21.15.59:0/2485744651 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc79407b020 con 0x7fc7a00d14f0 2024-03-06T07:34:12.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.789+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2485744651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7a00d0300 con 0x7fc7a00d14f0 2024-03-06T07:34:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.789+0000 7fc787fff640 1 -- 172.21.15.59:0/2485744651 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fc79407a920 con 0x7fc7a00d14f0 2024-03-06T07:34:12.791 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:12.791 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:12.791 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:12.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc78c06b4d0 msgr2=0x7fc78c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc78c06b4d0 0x7fc78c06d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00b5760 tx=0x7fc790008040 comp rx=0 tx=0).stop 2024-03-06T07:34:12.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2485744651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00d14f0 msgr2=0x7fc7a01f5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00d14f0 0x7fc7a01f5200 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fc794000c00 tx=0x7fc7940046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2485744651 shutdown_connections 2024-03-06T07:34:12.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc78c06b4d0 0x7fc78c06d990 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7a0158920 0x7fc7a0156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7a00d14f0 0x7fc7a01f5200 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 --2- 172.21.15.59:0/2485744651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7a00af6c0 0x7fc7a014e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2485744651 >> 172.21.15.59:0/2485744651 conn(0x7fc7a00d0960 msgr2=0x7fc7a006ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2485744651 shutdown_connections 2024-03-06T07:34:12.798 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:12.791+0000 7fc7a5d7a640 1 -- 172.21.15.59:0/2485744651 wait complete. 2024-03-06T07:34:13.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:13.005 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:14.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:14.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.108+0000 7fb4f180e640 1 Processor -- start 2024-03-06T07:34:14.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.108+0000 7fb4f180e640 1 -- start start 2024-03-06T07:34:14.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4f180e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb4ec15bf40 0x7fb4ec0cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4f180e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec0afe40 con 0x7fb4ec06baa0 2024-03-06T07:34:14.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4f180e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec0b0120 con 0x7fb4ec06afb0 2024-03-06T07:34:14.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4f180e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4ec0cd680 con 0x7fb4ec15bf40 2024-03-06T07:34:14.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4eb7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb4ec06baa0 0x7fb4ec0cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38906/0 (socket says 172.21.15.59:38906) 2024-03-06T07:34:14.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4eb7fe640 1 -- 172.21.15.59:0/1931035061 learned_addr learned my addr 172.21.15.59:0/1931035061 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:14.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4ea7fc640 1 --2- 172.21.15.59:0/1931035061 >> v2:172.21.15.164:3300/0 conn(0x7fb4ec15bf40 0x7fb4ec0cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4e9ffb640 1 -- 172.21.15.59:0/1931035061 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961850782 0 0) 0x7fb4ec0afe40 con 0x7fb4ec06baa0 2024-03-06T07:34:14.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4e9ffb640 1 -- 172.21.15.59:0/1931035061 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d4003520 con 0x7fb4ec06baa0 2024-03-06T07:34:14.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4ea7fc640 1 -- 172.21.15.59:0/1931035061 >> v1:172.21.15.164:6789/0 conn(0x7fb4ec06afb0 legacy=0x7fb4ec06b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4e9ffb640 1 -- 172.21.15.59:0/1931035061 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 417079377 0 0) 0x7fb4d4003520 con 0x7fb4ec06baa0 2024-03-06T07:34:14.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4ea7fc640 1 -- 172.21.15.59:0/1931035061 >> v1:172.21.15.59:6789/0 conn(0x7fb4ec06baa0 legacy=0x7fb4ec0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.113 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.110+0000 7fb4ea7fc640 1 -- 172.21.15.59:0/1931035061 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec0cd310 con 0x7fb4ec15bf40 2024-03-06T07:34:14.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.111+0000 7fb4ea7fc640 1 --2- 172.21.15.59:0/1931035061 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4ec15bf40 0x7fb4ec0cee60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009720 tx=0x7fb4e002b620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c87ea65343bbeb23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.111+0000 7fb4e9ffb640 1 -- 172.21.15.59:0/1931035061 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0037070 con 0x7fb4ec15bf40 2024-03-06T07:34:14.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.111+0000 7fb4e9ffb640 1 -- 172.21.15.59:0/1931035061 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e0030510 con 0x7fb4ec15bf40 2024-03-06T07:34:14.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.111+0000 7fb4e9ffb640 1 -- 172.21.15.59:0/1931035061 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e40019f0 con 0x7fb4ec15bf40 2024-03-06T07:34:14.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.112+0000 7fb4f180e640 1 -- 172.21.15.59:0/1931035061 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4ec15bf40 msgr2=0x7fb4ec0cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.112+0000 7fb4f180e640 1 --2- 172.21.15.59:0/1931035061 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4ec15bf40 0x7fb4ec0cee60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009720 tx=0x7fb4e002b620 comp rx=0 tx=0).stop 2024-03-06T07:34:14.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.112+0000 7fb4f180e640 1 -- 172.21.15.59:0/1931035061 shutdown_connections 2024-03-06T07:34:14.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.112+0000 7fb4f180e640 1 --2- 172.21.15.59:0/1931035061 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4ec15bf40 0x7fb4ec0cee60 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.112+0000 7fb4f180e640 1 -- 172.21.15.59:0/1931035061 >> 172.21.15.59:0/1931035061 conn(0x7fb4ec0d0960 msgr2=0x7fb4ec0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.112+0000 7fb4f180e640 1 -- 172.21.15.59:0/1931035061 shutdown_connections 2024-03-06T07:34:14.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.112+0000 7fb4f180e640 1 -- 172.21.15.59:0/1931035061 wait complete. 2024-03-06T07:34:14.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.113+0000 7fb4f180e640 1 Processor -- start 2024-03-06T07:34:14.115 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.113+0000 7fb4f180e640 1 -- start start 2024-03-06T07:34:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4f180e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4ec06afb0 0x7fb4ec146220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4f180e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4ec06baa0 0x7fb4ec1f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4f180e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4ec15bf40 0x7fb4ec06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4f180e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec06f510 con 0x7fb4ec15bf40 2024-03-06T07:34:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4f180e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec06f390 con 0x7fb4ec06baa0 2024-03-06T07:34:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4f180e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec06f690 con 0x7fb4ec06afb0 2024-03-06T07:34:14.117 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4eb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4ec15bf40 0x7fb4ec06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4eb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4ec15bf40 0x7fb4ec06e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57900/0 (socket says 172.21.15.59:57900) 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4eb7fe640 1 -- 172.21.15.59:0/1566536074 learned_addr learned my addr 172.21.15.59:0/1566536074 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4ea7fc640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4ec06baa0 0x7fb4ec1f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4eaffd640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4ec06afb0 0x7fb4ec146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4eb7fe640 1 -- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4ec06afb0 msgr2=0x7fb4ec146220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4eb7fe640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4ec06afb0 0x7fb4ec146220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4eb7fe640 1 -- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4ec06baa0 msgr2=0x7fb4ec1f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4eb7fe640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4ec06baa0 0x7fb4ec1f4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4eb7fe640 1 -- 172.21.15.59:0/1566536074 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec1f56b0 con 0x7fb4ec15bf40 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.114+0000 7fb4eaffd640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4ec06afb0 0x7fb4ec146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.115+0000 7fb4eb7fe640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4ec15bf40 0x7fb4ec06e3a0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4001560 tx=0x7fb4e400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.115+0000 7fb4d3fff640 1 -- 172.21.15.59:0/1566536074 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4018070 con 0x7fb4ec15bf40 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.115+0000 7fb4d3fff640 1 -- 172.21.15.59:0/1566536074 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e400a700 con 0x7fb4ec15bf40 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.115+0000 7fb4f180e640 1 -- 172.21.15.59:0/1566536074 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ec1f5990 con 0x7fb4ec15bf40 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.115+0000 7fb4d3fff640 1 -- 172.21.15.59:0/1566536074 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e400aaf0 con 0x7fb4ec15bf40 2024-03-06T07:34:14.118 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.115+0000 7fb4f180e640 1 -- 172.21.15.59:0/1566536074 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ec0a97d0 con 0x7fb4ec15bf40 2024-03-06T07:34:14.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.116+0000 7fb4f180e640 1 -- 172.21.15.59:0/1566536074 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ec1f5b60 con 0x7fb4ec15bf40 2024-03-06T07:34:14.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.116+0000 7fb4d3fff640 1 -- 172.21.15.59:0/1566536074 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4e401b070 con 0x7fb4ec15bf40 2024-03-06T07:34:14.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.117+0000 7fb4d3fff640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb4d806b4d0 0x7fb4d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:14.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.117+0000 7fb4d3fff640 1 -- 172.21.15.59:0/1566536074 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb4e4089a40 con 0x7fb4ec15bf40 2024-03-06T07:34:14.119 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.117+0000 7fb4eaffd640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb4d806b4d0 0x7fb4d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:14.120 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.117+0000 7fb4eaffd640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb4d806b4d0 0x7fb4d806d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc000f60 tx=0x7fb4dc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:14.127 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.125+0000 7fb4d3fff640 1 -- 172.21.15.59:0/1566536074 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4e4058250 con 0x7fb4ec15bf40 2024-03-06T07:34:14.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.344+0000 7fb4f180e640 1 -- 172.21.15.59:0/1566536074 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4ec04f330 con 0x7fb4ec15bf40 2024-03-06T07:34:14.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.344+0000 7fb4d3fff640 1 -- 172.21.15.59:0/1566536074 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fb4e4089e20 con 0x7fb4ec15bf40 2024-03-06T07:34:14.347 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:14.347 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:14.347 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:14.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 -- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb4d806b4d0 msgr2=0x7fb4d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb4d806b4d0 0x7fb4d806d990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc000f60 tx=0x7fb4dc009040 comp rx=0 tx=0).stop 2024-03-06T07:34:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 -- 172.21.15.59:0/1566536074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4ec15bf40 msgr2=0x7fb4ec06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4ec15bf40 0x7fb4ec06e3a0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4001560 tx=0x7fb4e400cf50 comp rx=0 tx=0).stop 2024-03-06T07:34:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 -- 172.21.15.59:0/1566536074 shutdown_connections 2024-03-06T07:34:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb4d806b4d0 0x7fb4d806d990 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4ec15bf40 0x7fb4ec06e3a0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4ec06baa0 0x7fb4ec1f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 --2- 172.21.15.59:0/1566536074 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4ec06afb0 0x7fb4ec146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 -- 172.21.15.59:0/1566536074 >> 172.21.15.59:0/1566536074 conn(0x7fb4ec0d0960 msgr2=0x7fb4ec0b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 -- 172.21.15.59:0/1566536074 shutdown_connections 2024-03-06T07:34:14.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:14.346+0000 7fb4f180e640 1 -- 172.21.15.59:0/1566536074 wait complete. 2024-03-06T07:34:14.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:14.560 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:15.561 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:15.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.661+0000 7f8f7a6c9640 1 Processor -- start 2024-03-06T07:34:15.663 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.661+0000 7f8f7a6c9640 1 -- start start 2024-03-06T07:34:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f7a6c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740d22b0 0x7f8f740d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f7a6c9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8f740b41f0 0x7f8f740b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f7a6c9640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f740b3430 con 0x7f8f7415dfe0 2024-03-06T07:34:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f7a6c9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f740d2c50 con 0x7f8f740b41f0 2024-03-06T07:34:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f7a6c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f740b31e0 con 0x7f8f740d22b0 2024-03-06T07:34:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f73fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8f7415dfe0 0x7f8f740cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38908/0 (socket says 172.21.15.59:38908) 2024-03-06T07:34:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f73fff640 1 -- 172.21.15.59:0/711670181 learned_addr learned my addr 172.21.15.59:0/711670181 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f737fe640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740d22b0 0x7f8f740d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f78c66640 1 --2- 172.21.15.59:0/711670181 >> v2:172.21.15.164:3300/0 conn(0x7f8f740b41f0 0x7f8f740b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307520086 0 0) 0x7f8f740b3430 con 0x7f8f7415dfe0 2024-03-06T07:34:15.667 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f5c003520 con 0x7f8f7415dfe0 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3711733940 0 0) 0x7f8f5c003520 con 0x7f8f7415dfe0 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740d22b0 msgr2=0x7f8f740d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740d22b0 0x7f8f740d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 >> v2:172.21.15.164:3300/0 conn(0x7f8f740b41f0 msgr2=0x7f8f740b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 --2- 172.21.15.59:0/711670181 >> v2:172.21.15.164:3300/0 conn(0x7f8f740b41f0 0x7f8f740b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f740b3710 con 0x7f8f7415dfe0 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f8f64001670 con 0x7f8f7415dfe0 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f737fe640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740d22b0 0x7f8f740d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f78c66640 1 --2- 172.21.15.59:0/711670181 >> v2:172.21.15.164:3300/0 conn(0x7f8f740b41f0 0x7f8f740b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 >> v1:172.21.15.59:6789/0 conn(0x7f8f7415dfe0 legacy=0x7f8f740cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f5c0039b0 0x7f8f5c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f5c004800 0x7f8f5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.668 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.662+0000 7f8f72ffd640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740d22b0 0x7f8f5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.663+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f740b31e0 con 0x7f8f5c004800 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.663+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c00b9c0 con 0x7f8f5c0039b0 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.663+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c00bb40 con 0x7f8f740d22b0 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.663+0000 7f8f737fe640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f5c004800 0x7f8f5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.663+0000 7f8f73fff640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f5c0039b0 0x7f8f5c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.663+0000 7f8f78c66640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740d22b0 0x7f8f5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.663+0000 7f8f737fe640 1 -- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740d22b0 msgr2=0x7f8f5c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.663+0000 7f8f737fe640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740d22b0 0x7f8f5c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.663+0000 7f8f737fe640 1 -- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f5c0039b0 msgr2=0x7f8f5c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.663+0000 7f8f737fe640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f5c0039b0 0x7f8f5c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.663+0000 7f8f737fe640 1 -- 172.21.15.59:0/711670181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c003520 con 0x7f8f5c004800 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.664+0000 7f8f737fe640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f5c004800 0x7f8f5c004c80 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009c90 tx=0x7f8f5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=effff25759029c1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.664+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f68039070 con 0x7f8f5c004800 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.664+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6802c8f0 con 0x7f8f5c004800 2024-03-06T07:34:15.669 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.664+0000 7f8f72ffd640 1 -- 172.21.15.59:0/711670181 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f68009860 con 0x7f8f5c004800 2024-03-06T07:34:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.664+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/711670181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f5c004800 msgr2=0x7f8f5c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.664+0000 7f8f7a6c9640 1 --2- 172.21.15.59:0/711670181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f5c004800 0x7f8f5c004c80 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009c90 tx=0x7f8f5c005830 comp rx=0 tx=0).stop 2024-03-06T07:34:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.664+0000 7f8f78c66640 1 -- 172.21.15.59:0/711670181 reap_dead start 2024-03-06T07:34:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.664+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/711670181 shutdown_connections 2024-03-06T07:34:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.664+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/711670181 >> 172.21.15.59:0/711670181 conn(0x7f8f740d3cd0 msgr2=0x7f8f740d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.664+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/711670181 shutdown_connections 2024-03-06T07:34:15.670 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.664+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/711670181 wait complete. 2024-03-06T07:34:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.665+0000 7f8f7a6c9640 1 Processor -- start 2024-03-06T07:34:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.666+0000 7f8f7a6c9640 1 -- start start 2024-03-06T07:34:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.666+0000 7f8f7a6c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f5c004800 0x7f8f7415f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.666+0000 7f8f7a6c9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f740b9600 0x7f8f740cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.666+0000 7f8f7a6c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740ccac0 0x7f8f740ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.666+0000 7f8f7a6c9640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f740cfa00 con 0x7f8f740b9600 2024-03-06T07:34:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.666+0000 7f8f7a6c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f740b3d60 con 0x7f8f5c004800 2024-03-06T07:34:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.666+0000 7f8f7a6c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f740d1c70 con 0x7f8f740ccac0 2024-03-06T07:34:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f737fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f740b9600 0x7f8f740cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f737fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f740b9600 0x7f8f740cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57914/0 (socket says 172.21.15.59:57914) 2024-03-06T07:34:15.673 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f737fe640 1 -- 172.21.15.59:0/3935538890 learned_addr learned my addr 172.21.15.59:0/3935538890 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f78c66640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740ccac0 0x7f8f740ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f737fe640 1 -- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740ccac0 msgr2=0x7f8f740ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f737fe640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740ccac0 0x7f8f740ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f737fe640 1 -- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f5c004800 msgr2=0x7f8f7415f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f737fe640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f5c004800 0x7f8f7415f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f737fe640 1 -- 172.21.15.59:0/3935538890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f740cd480 con 0x7f8f740b9600 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f737fe640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f740b9600 0x7f8f740cc580 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009db0 tx=0x7f8f68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f70ff9640 1 -- 172.21.15.59:0/3935538890 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f68039050 con 0x7f8f740b9600 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f70ff9640 1 -- 172.21.15.59:0/3935538890 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f68045070 con 0x7f8f740b9600 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/3935538890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f74160e20 con 0x7f8f740b9600 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f70ff9640 1 -- 172.21.15.59:0/3935538890 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f68009860 con 0x7f8f740b9600 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.667+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/3935538890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f74160340 con 0x7f8f740b9600 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.668+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/3935538890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7404f330 con 0x7f8f740b9600 2024-03-06T07:34:15.674 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.669+0000 7f8f70ff9640 1 -- 172.21.15.59:0/3935538890 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f6804a460 con 0x7f8f740b9600 2024-03-06T07:34:15.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.669+0000 7f8f70ff9640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f6006b500 0x7f8f6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:15.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.669+0000 7f8f70ff9640 1 -- 172.21.15.59:0/3935538890 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8f680ac470 con 0x7f8f740b9600 2024-03-06T07:34:15.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.669+0000 7f8f73fff640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f6006b500 0x7f8f6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:15.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.670+0000 7f8f73fff640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f6006b500 0x7f8f6006d9c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f8f64007fd0 tx=0x7f8f64001010 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:15.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.674+0000 7f8f70ff9640 1 -- 172.21.15.59:0/3935538890 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f6807b240 con 0x7f8f740b9600 2024-03-06T07:34:15.893 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.892+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/3935538890 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f740747a0 con 0x7f8f740b9600 2024-03-06T07:34:15.894 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.892+0000 7f8f70ff9640 1 -- 172.21.15.59:0/3935538890 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f8f6807ab40 con 0x7f8f740b9600 2024-03-06T07:34:15.894 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:15.894 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:15.895 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:15.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f6006b500 msgr2=0x7f8f6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.896 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f6006b500 0x7f8f6006d9c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f8f64007fd0 tx=0x7f8f64001010 comp rx=0 tx=0).stop 2024-03-06T07:34:15.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/3935538890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f740b9600 msgr2=0x7f8f740cc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:15.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f740b9600 0x7f8f740cc580 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009db0 tx=0x7f8f68004730 comp rx=0 tx=0).stop 2024-03-06T07:34:15.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/3935538890 shutdown_connections 2024-03-06T07:34:15.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f8f6006b500 0x7f8f6006d9c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f740ccac0 0x7f8f740ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f740b9600 0x7f8f740cc580 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 --2- 172.21.15.59:0/3935538890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f5c004800 0x7f8f7415f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:15.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/3935538890 >> 172.21.15.59:0/3935538890 conn(0x7f8f740d3cd0 msgr2=0x7f8f740d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:15.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/3935538890 shutdown_connections 2024-03-06T07:34:15.897 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:15.894+0000 7f8f7a6c9640 1 -- 172.21.15.59:0/3935538890 wait complete. 2024-03-06T07:34:16.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:16.108 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:17.110 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:17.212 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.211+0000 7f61ae75e640 1 Processor -- start 2024-03-06T07:34:17.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.211+0000 7f61ae75e640 1 -- start start 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.211+0000 7f61ae75e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f61a80d22b0 0x7f61a80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.211+0000 7f61ae75e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a80b3430 con 0x7f61a815dfe0 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.211+0000 7f61ae75e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a80b3710 con 0x7f61a80b41f0 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.211+0000 7f61ae75e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a80d2c50 con 0x7f61a80d22b0 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.211+0000 7f61a7fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f61a815dfe0 0x7f61a80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38914/0 (socket says 172.21.15.59:38914) 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.211+0000 7f61a77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f61a80d22b0 0x7f61a80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.212+0000 7f61a7fff640 1 -- 172.21.15.59:0/1141046480 learned_addr learned my addr 172.21.15.59:0/1141046480 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.212+0000 7f61a77fe640 1 -- 172.21.15.59:0/1141046480 >> v1:172.21.15.164:6789/0 conn(0x7f61a80b41f0 legacy=0x7f61a80b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.212+0000 7f61a77fe640 1 -- 172.21.15.59:0/1141046480 >> v1:172.21.15.59:6789/0 conn(0x7f61a815dfe0 legacy=0x7f61a80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.212+0000 7f61a77fe640 1 -- 172.21.15.59:0/1141046480 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a80b39f0 con 0x7f61a80d22b0 2024-03-06T07:34:17.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.213+0000 7f61a77fe640 1 --2- 172.21.15.59:0/1141046480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61a80d22b0 0x7f61a80d2710 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f619c0097d0 tx=0x7f619c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fe084337ac5d15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.213+0000 7f61a6ffd640 1 -- 172.21.15.59:0/1141046480 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c035020 con 0x7f61a80d22b0 2024-03-06T07:34:17.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.213+0000 7f61a6ffd640 1 -- 172.21.15.59:0/1141046480 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619c031830 con 0x7f61a80d22b0 2024-03-06T07:34:17.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.213+0000 7f61a6ffd640 1 -- 172.21.15.59:0/1141046480 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c031c00 con 0x7f61a80d22b0 2024-03-06T07:34:17.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.213+0000 7f61ae75e640 1 -- 172.21.15.59:0/1141046480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61a80d22b0 msgr2=0x7f61a80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.213+0000 7f61ae75e640 1 --2- 172.21.15.59:0/1141046480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61a80d22b0 0x7f61a80d2710 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f619c0097d0 tx=0x7f619c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.213+0000 7f61ae75e640 1 -- 172.21.15.59:0/1141046480 shutdown_connections 2024-03-06T07:34:17.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.213+0000 7f61ae75e640 1 --2- 172.21.15.59:0/1141046480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61a80d22b0 0x7f61a80d2710 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.213+0000 7f61ae75e640 1 -- 172.21.15.59:0/1141046480 >> 172.21.15.59:0/1141046480 conn(0x7f61a80d3cd0 msgr2=0x7f61a80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.213+0000 7f61ae75e640 1 -- 172.21.15.59:0/1141046480 shutdown_connections 2024-03-06T07:34:17.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.213+0000 7f61ae75e640 1 -- 172.21.15.59:0/1141046480 wait complete. 2024-03-06T07:34:17.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.214+0000 7f61ae75e640 1 Processor -- start 2024-03-06T07:34:17.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.215+0000 7f61ae75e640 1 -- start start 2024-03-06T07:34:17.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.215+0000 7f61ae75e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a80b41f0 0x7f61a81610d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.215+0000 7f61ae75e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61a80d22b0 0x7f61a814f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.215+0000 7f61ae75e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a815dfe0 0x7f61a80cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.215+0000 7f61ae75e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a80b3d60 con 0x7f61a80d22b0 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.215+0000 7f61ae75e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a80b31e0 con 0x7f61a815dfe0 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.215+0000 7f61ae75e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a80d1c70 con 0x7f61a80b41f0 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.215+0000 7f61a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61a80d22b0 0x7f61a814f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.215+0000 7f61a77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61a80d22b0 0x7f61a814f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57944/0 (socket says 172.21.15.59:57944) 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.215+0000 7f61a77fe640 1 -- 172.21.15.59:0/3394758537 learned_addr learned my addr 172.21.15.59:0/3394758537 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61a77fe640 1 -- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a80b41f0 msgr2=0x7f61a81610d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61a7fff640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a80b41f0 0x7f61a81610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61accfb640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a815dfe0 0x7f61a80cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61a77fe640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a80b41f0 0x7f61a81610d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61a77fe640 1 -- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a815dfe0 msgr2=0x7f61a80cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.219 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61a77fe640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a815dfe0 0x7f61a80cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61a77fe640 1 -- 172.21.15.59:0/3394758537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a80cc730 con 0x7f61a80d22b0 2024-03-06T07:34:17.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61a77fe640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61a80d22b0 0x7f61a814f0b0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f619c0098f0 tx=0x7f619c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61a4ff9640 1 -- 172.21.15.59:0/3394758537 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c041070 con 0x7f61a80d22b0 2024-03-06T07:34:17.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61a4ff9640 1 -- 172.21.15.59:0/3394758537 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619c03a950 con 0x7f61a80d22b0 2024-03-06T07:34:17.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61ae75e640 1 -- 172.21.15.59:0/3394758537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a80cca10 con 0x7f61a80d22b0 2024-03-06T07:34:17.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61a4ff9640 1 -- 172.21.15.59:0/3394758537 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c035040 con 0x7f61a80d22b0 2024-03-06T07:34:17.220 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.216+0000 7f61ae75e640 1 -- 172.21.15.59:0/3394758537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a80ccfd0 con 0x7f61a80d22b0 2024-03-06T07:34:17.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.217+0000 7f61ae75e640 1 -- 172.21.15.59:0/3394758537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a804f330 con 0x7f61a80d22b0 2024-03-06T07:34:17.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.218+0000 7f61a4ff9640 1 -- 172.21.15.59:0/3394758537 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f619c038020 con 0x7f61a80d22b0 2024-03-06T07:34:17.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.218+0000 7f61a4ff9640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f619406b4d0 0x7f619406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:17.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.218+0000 7f61a4ff9640 1 -- 172.21.15.59:0/3394758537 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f619c0abc00 con 0x7f61a80d22b0 2024-03-06T07:34:17.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.218+0000 7f61a7fff640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f619406b4d0 0x7f619406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:17.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.219+0000 7f61a7fff640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f619406b4d0 0x7f619406d990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f61a80b9d80 tx=0x7f6198009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:17.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.223+0000 7f61a4ff9640 1 -- 172.21.15.59:0/3394758537 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f619c0b2080 con 0x7f61a80d22b0 2024-03-06T07:34:17.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.440+0000 7f61ae75e640 1 -- 172.21.15.59:0/3394758537 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61a80ce9b0 con 0x7f61a80d22b0 2024-03-06T07:34:17.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.440+0000 7f61a4ff9640 1 -- 172.21.15.59:0/3394758537 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f619c07aaf0 con 0x7f61a80d22b0 2024-03-06T07:34:17.442 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:17.443 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:17.443 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:17.444 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 -- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f619406b4d0 msgr2=0x7f619406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f619406b4d0 0x7f619406d990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f61a80b9d80 tx=0x7f6198009040 comp rx=0 tx=0).stop 2024-03-06T07:34:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 -- 172.21.15.59:0/3394758537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61a80d22b0 msgr2=0x7f61a814f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61a80d22b0 0x7f61a814f0b0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f619c0098f0 tx=0x7f619c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 -- 172.21.15.59:0/3394758537 shutdown_connections 2024-03-06T07:34:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f619406b4d0 0x7f619406d990 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a815dfe0 0x7f61a80cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61a80d22b0 0x7f61a814f0b0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 --2- 172.21.15.59:0/3394758537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a80b41f0 0x7f61a81610d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 -- 172.21.15.59:0/3394758537 >> 172.21.15.59:0/3394758537 conn(0x7f61a80d3cd0 msgr2=0x7f61a806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 -- 172.21.15.59:0/3394758537 shutdown_connections 2024-03-06T07:34:17.445 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:17.442+0000 7f61ae75e640 1 -- 172.21.15.59:0/3394758537 wait complete. 2024-03-06T07:34:17.657 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:17.657 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:18.658 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:18.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.757+0000 7fd4509e4640 1 Processor -- start 2024-03-06T07:34:18.759 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.757+0000 7fd4509e4640 1 -- start start 2024-03-06T07:34:18.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.758+0000 7fd4509e4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd44c13dc10 0x7fd44c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.760 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.758+0000 7fd4509e4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44c15f580 con 0x7fd44c15bf30 2024-03-06T07:34:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.758+0000 7fd4509e4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44c15b450 con 0x7fd44c15ef60 2024-03-06T07:34:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.758+0000 7fd4509e4640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd44c0cf3f0 con 0x7fd44c13dc10 2024-03-06T07:34:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.758+0000 7fd44a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd44c13dc10 0x7fd44c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.758+0000 7fd449d9b640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd44c15bf30 0x7fd44c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38922/0 (socket says 172.21.15.59:38922) 2024-03-06T07:34:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.758+0000 7fd44a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd44c13dc10 0x7fd44c0b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57958/0 (socket says 172.21.15.59:57958) 2024-03-06T07:34:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.758+0000 7fd449d9b640 1 -- 172.21.15.59:0/1630048837 learned_addr learned my addr 172.21.15.59:0/1630048837 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.758+0000 7fd44a59c640 1 -- 172.21.15.59:0/1630048837 >> v1:172.21.15.164:6789/0 conn(0x7fd44c15ef60 legacy=0x7fd44c0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.758+0000 7fd44a59c640 1 -- 172.21.15.59:0/1630048837 >> v1:172.21.15.59:6789/0 conn(0x7fd44c15bf30 legacy=0x7fd44c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.759+0000 7fd44a59c640 1 -- 172.21.15.59:0/1630048837 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c15b730 con 0x7fd44c13dc10 2024-03-06T07:34:18.761 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.758+0000 7fd44959a640 1 -- 172.21.15.59:0/1630048837 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493591296 0 0) 0x7fd44c15f580 con 0x7fd44c15bf30 2024-03-06T07:34:18.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.759+0000 7fd44a59c640 1 --2- 172.21.15.59:0/1630048837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c13dc10 0x7fd44c0b2950 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0097a0 tx=0x7fd43c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36bc8805cc1b1f2e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.762 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.759+0000 7fd44959a640 1 -- 172.21.15.59:0/1630048837 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0315a0 con 0x7fd44c13dc10 2024-03-06T07:34:18.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.759+0000 7fd44959a640 1 -- 172.21.15.59:0/1630048837 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43c037070 con 0x7fd44c13dc10 2024-03-06T07:34:18.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.759+0000 7fd44959a640 1 -- 172.21.15.59:0/1630048837 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c031960 con 0x7fd44c13dc10 2024-03-06T07:34:18.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.760+0000 7fd4509e4640 1 -- 172.21.15.59:0/1630048837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c13dc10 msgr2=0x7fd44c0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.760+0000 7fd4509e4640 1 --2- 172.21.15.59:0/1630048837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c13dc10 0x7fd44c0b2950 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0097a0 tx=0x7fd43c02b840 comp rx=0 tx=0).stop 2024-03-06T07:34:18.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.760+0000 7fd4509e4640 1 -- 172.21.15.59:0/1630048837 shutdown_connections 2024-03-06T07:34:18.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.760+0000 7fd4509e4640 1 --2- 172.21.15.59:0/1630048837 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c13dc10 0x7fd44c0b2950 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.760+0000 7fd4509e4640 1 -- 172.21.15.59:0/1630048837 >> 172.21.15.59:0/1630048837 conn(0x7fd44c0cf8a0 msgr2=0x7fd44c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:18.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.760+0000 7fd4509e4640 1 -- 172.21.15.59:0/1630048837 shutdown_connections 2024-03-06T07:34:18.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.760+0000 7fd4509e4640 1 -- 172.21.15.59:0/1630048837 wait complete. 2024-03-06T07:34:18.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.761+0000 7fd4509e4640 1 Processor -- start 2024-03-06T07:34:18.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.761+0000 7fd4509e4640 1 -- start start 2024-03-06T07:34:18.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.762+0000 7fd4509e4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd44c13dc10 0x7fd44c154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.762+0000 7fd4509e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c15bf30 0x7fd44c141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.762+0000 7fd4509e4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44c15ef60 0x7fd44c159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.765 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.762+0000 7fd4509e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c0b2e90 con 0x7fd44c15bf30 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.762+0000 7fd4509e4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c15b210 con 0x7fd44c15ef60 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.762+0000 7fd4509e4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c0b3010 con 0x7fd44c13dc10 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.762+0000 7fd449d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c15bf30 0x7fd44c141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.762+0000 7fd449d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c15bf30 0x7fd44c141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57970/0 (socket says 172.21.15.59:57970) 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.762+0000 7fd449d9b640 1 -- 172.21.15.59:0/2975954816 learned_addr learned my addr 172.21.15.59:0/2975954816 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.762+0000 7fd44ad9d640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44c15ef60 0x7fd44c159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.762+0000 7fd44a59c640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd44c13dc10 0x7fd44c154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.763+0000 7fd449d9b640 1 -- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd44c13dc10 msgr2=0x7fd44c154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.763+0000 7fd449d9b640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd44c13dc10 0x7fd44c154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.763+0000 7fd449d9b640 1 -- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44c15ef60 msgr2=0x7fd44c159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.763+0000 7fd449d9b640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44c15ef60 0x7fd44c159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.763+0000 7fd449d9b640 1 -- 172.21.15.59:0/2975954816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c131380 con 0x7fd44c15bf30 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.763+0000 7fd449d9b640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c15bf30 0x7fd44c141b80 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fd44000c4d0 tx=0x7fd44000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.763+0000 7fd43affd640 1 -- 172.21.15.59:0/2975954816 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd440018070 con 0x7fd44c15bf30 2024-03-06T07:34:18.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.763+0000 7fd43affd640 1 -- 172.21.15.59:0/2975954816 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd440007500 con 0x7fd44c15bf30 2024-03-06T07:34:18.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.763+0000 7fd4509e4640 1 -- 172.21.15.59:0/2975954816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd44c131660 con 0x7fd44c15bf30 2024-03-06T07:34:18.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.763+0000 7fd43affd640 1 -- 172.21.15.59:0/2975954816 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd440004a20 con 0x7fd44c15bf30 2024-03-06T07:34:18.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.763+0000 7fd4509e4640 1 -- 172.21.15.59:0/2975954816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd44c131c20 con 0x7fd44c15bf30 2024-03-06T07:34:18.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.764+0000 7fd4509e4640 1 -- 172.21.15.59:0/2975954816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd44c04f330 con 0x7fd44c15bf30 2024-03-06T07:34:18.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.765+0000 7fd43affd640 1 -- 172.21.15.59:0/2975954816 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd44001b070 con 0x7fd44c15bf30 2024-03-06T07:34:18.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.765+0000 7fd43affd640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b500 0x7fd42c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:18.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.765+0000 7fd43affd640 1 -- 172.21.15.59:0/2975954816 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd440089450 con 0x7fd44c15bf30 2024-03-06T07:34:18.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.765+0000 7fd44a59c640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b500 0x7fd42c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:18.768 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.766+0000 7fd44a59c640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b500 0x7fd42c06d9c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fd43c036000 tx=0x7fd43c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:18.771 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:18.769+0000 7fd43affd640 1 -- 172.21.15.59:0/2975954816 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd440057c60 con 0x7fd44c15bf30 2024-03-06T07:34:19.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.054+0000 7fd4509e4640 1 -- 172.21.15.59:0/2975954816 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd44c072860 con 0x7fd44c15bf30 2024-03-06T07:34:19.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.054+0000 7fd43affd640 1 -- 172.21.15.59:0/2975954816 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fd440006e20 con 0x7fd44c15bf30 2024-03-06T07:34:19.056 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:19.056 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:19.057 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:19.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 -- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b500 msgr2=0x7fd42c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b500 0x7fd42c06d9c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fd43c036000 tx=0x7fd43c008030 comp rx=0 tx=0).stop 2024-03-06T07:34:19.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 -- 172.21.15.59:0/2975954816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c15bf30 msgr2=0x7fd44c141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:19.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c15bf30 0x7fd44c141b80 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fd44000c4d0 tx=0x7fd44000c990 comp rx=0 tx=0).stop 2024-03-06T07:34:19.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 -- 172.21.15.59:0/2975954816 shutdown_connections 2024-03-06T07:34:19.058 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd42c06b500 0x7fd42c06d9c0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd44c15ef60 0x7fd44c159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd44c15bf30 0x7fd44c141b80 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 --2- 172.21.15.59:0/2975954816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd44c13dc10 0x7fd44c154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:19.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 -- 172.21.15.59:0/2975954816 >> 172.21.15.59:0/2975954816 conn(0x7fd44c0cf8a0 msgr2=0x7fd44c070fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:19.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 -- 172.21.15.59:0/2975954816 shutdown_connections 2024-03-06T07:34:19.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:19.056+0000 7fd4509e4640 1 -- 172.21.15.59:0/2975954816 wait complete. 2024-03-06T07:34:19.274 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:19.274 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:20.275 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:20.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.375+0000 7fc21177a640 1 Processor -- start 2024-03-06T07:34:20.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.375+0000 7fc21177a640 1 -- start start 2024-03-06T07:34:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc21177a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc20c15dfe0 0x7fc20c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc21177a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c0b3430 con 0x7fc20c0cf830 2024-03-06T07:34:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc21177a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c0b3710 con 0x7fc20c0d29c0 2024-03-06T07:34:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc21177a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc20c0b31e0 con 0x7fc20c15dfe0 2024-03-06T07:34:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc20affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc20c15dfe0 0x7fc20c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc203fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc20c0cf830 0x7fc20c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38936/0 (socket says 172.21.15.59:38936) 2024-03-06T07:34:20.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc20affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc20c15dfe0 0x7fc20c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57976/0 (socket says 172.21.15.59:57976) 2024-03-06T07:34:20.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc203fff640 1 -- 172.21.15.59:0/4089027283 learned_addr learned my addr 172.21.15.59:0/4089027283 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:20.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc20affd640 1 -- 172.21.15.59:0/4089027283 >> v1:172.21.15.164:6789/0 conn(0x7fc20c0d29c0 legacy=0x7fc20c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:20.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc20affd640 1 -- 172.21.15.59:0/4089027283 >> v1:172.21.15.59:6789/0 conn(0x7fc20c0cf830 legacy=0x7fc20c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc20affd640 1 -- 172.21.15.59:0/4089027283 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c0b39f0 con 0x7fc20c15dfe0 2024-03-06T07:34:20.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.377+0000 7fc20a7fc640 1 -- 172.21.15.59:0/4089027283 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158512968 0 0) 0x7fc20c0b3430 con 0x7fc20c0cf830 2024-03-06T07:34:20.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.378+0000 7fc20affd640 1 --2- 172.21.15.59:0/4089027283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc20c15dfe0 0x7fc20c0cf2f0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0b3360 tx=0x7fc1fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700a6cdab8b901d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.378+0000 7fc20a7fc640 1 -- 172.21.15.59:0/4089027283 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc035020 con 0x7fc20c15dfe0 2024-03-06T07:34:20.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.378+0000 7fc20a7fc640 1 -- 172.21.15.59:0/4089027283 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1fc031830 con 0x7fc20c15dfe0 2024-03-06T07:34:20.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.378+0000 7fc20a7fc640 1 -- 172.21.15.59:0/4089027283 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc031c00 con 0x7fc20c15dfe0 2024-03-06T07:34:20.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.379+0000 7fc21177a640 1 -- 172.21.15.59:0/4089027283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc20c15dfe0 msgr2=0x7fc20c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.379+0000 7fc21177a640 1 --2- 172.21.15.59:0/4089027283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc20c15dfe0 0x7fc20c0cf2f0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fc20c0b3360 tx=0x7fc1fc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.379+0000 7fc21177a640 1 -- 172.21.15.59:0/4089027283 shutdown_connections 2024-03-06T07:34:20.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.379+0000 7fc21177a640 1 --2- 172.21.15.59:0/4089027283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc20c15dfe0 0x7fc20c0cf2f0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.379+0000 7fc21177a640 1 -- 172.21.15.59:0/4089027283 >> 172.21.15.59:0/4089027283 conn(0x7fc20c0d3cd0 msgr2=0x7fc20c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.379+0000 7fc21177a640 1 -- 172.21.15.59:0/4089027283 shutdown_connections 2024-03-06T07:34:20.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.379+0000 7fc21177a640 1 -- 172.21.15.59:0/4089027283 wait complete. 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.380+0000 7fc21177a640 1 Processor -- start 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.380+0000 7fc21177a640 1 -- start start 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc21177a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc20c0cf830 0x7fc20c150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc21177a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc20c0d29c0 0x7fc20c09fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc21177a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc20c15dfe0 0x7fc20c0a0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc21177a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c15e3c0 con 0x7fc20c0d29c0 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc21177a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c0b3d60 con 0x7fc20c15dfe0 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc21177a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c15e540 con 0x7fc20c0cf830 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc203fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc20c0d29c0 0x7fc20c09fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc203fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc20c0d29c0 0x7fc20c09fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57982/0 (socket says 172.21.15.59:57982) 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc203fff640 1 -- 172.21.15.59:0/647286964 learned_addr learned my addr 172.21.15.59:0/647286964 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc20affd640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc20c0cf830 0x7fc20c150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc20b7fe640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc20c15dfe0 0x7fc20c0a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc203fff640 1 -- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc20c0cf830 msgr2=0x7fc20c150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc203fff640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc20c0cf830 0x7fc20c150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc203fff640 1 -- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc20c15dfe0 msgr2=0x7fc20c0a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc203fff640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc20c15dfe0 0x7fc20c0a0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc203fff640 1 -- 172.21.15.59:0/647286964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c0a0850 con 0x7fc20c0d29c0 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc20affd640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc20c0cf830 0x7fc20c150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc203fff640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc20c0d29c0 0x7fc20c09fad0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400c4b0 tx=0x7fc1f400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.381+0000 7fc2037fe640 1 -- 172.21.15.59:0/647286964 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4018070 con 0x7fc20c0d29c0 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.382+0000 7fc2037fe640 1 -- 172.21.15.59:0/647286964 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f4007500 con 0x7fc20c0d29c0 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.382+0000 7fc21177a640 1 -- 172.21.15.59:0/647286964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20c0a0b30 con 0x7fc20c0d29c0 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.382+0000 7fc2037fe640 1 -- 172.21.15.59:0/647286964 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4004a20 con 0x7fc20c0d29c0 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.382+0000 7fc21177a640 1 -- 172.21.15.59:0/647286964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20c0a10f0 con 0x7fc20c0d29c0 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.383+0000 7fc21177a640 1 -- 172.21.15.59:0/647286964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20c04f330 con 0x7fc20c0d29c0 2024-03-06T07:34:20.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.383+0000 7fc2037fe640 1 -- 172.21.15.59:0/647286964 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1f401b070 con 0x7fc20c0d29c0 2024-03-06T07:34:20.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.384+0000 7fc2037fe640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1ec06b4d0 0x7fc1ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:20.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.384+0000 7fc2037fe640 1 -- 172.21.15.59:0/647286964 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc1f4089450 con 0x7fc20c0d29c0 2024-03-06T07:34:20.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.384+0000 7fc20affd640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1ec06b4d0 0x7fc1ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:20.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.384+0000 7fc20affd640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1ec06b4d0 0x7fc1ec06d990 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc036000 tx=0x7fc1fc03c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:20.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.388+0000 7fc2037fe640 1 -- 172.21.15.59:0/647286964 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1f4058400 con 0x7fc20c0d29c0 2024-03-06T07:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.606+0000 7fc21177a640 1 -- 172.21.15.59:0/647286964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc20c0b4e50 con 0x7fc20c0d29c0 2024-03-06T07:34:20.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.607+0000 7fc2037fe640 1 -- 172.21.15.59:0/647286964 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fc1f4006e20 con 0x7fc20c0d29c0 2024-03-06T07:34:20.609 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:20.609 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:20.609 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.608+0000 7fc21177a640 1 -- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1ec06b4d0 msgr2=0x7fc1ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.608+0000 7fc21177a640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1ec06b4d0 0x7fc1ec06d990 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc036000 tx=0x7fc1fc03c040 comp rx=0 tx=0).stop 2024-03-06T07:34:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.608+0000 7fc21177a640 1 -- 172.21.15.59:0/647286964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc20c0d29c0 msgr2=0x7fc20c09fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.608+0000 7fc21177a640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc20c0d29c0 0x7fc20c09fad0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400c4b0 tx=0x7fc1f400c970 comp rx=0 tx=0).stop 2024-03-06T07:34:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.609+0000 7fc21177a640 1 -- 172.21.15.59:0/647286964 shutdown_connections 2024-03-06T07:34:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.609+0000 7fc21177a640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc1ec06b4d0 0x7fc1ec06d990 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.609+0000 7fc21177a640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc20c15dfe0 0x7fc20c0a0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.609+0000 7fc21177a640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc20c0d29c0 0x7fc20c09fad0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.609+0000 7fc21177a640 1 --2- 172.21.15.59:0/647286964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc20c0cf830 0x7fc20c150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.609+0000 7fc21177a640 1 -- 172.21.15.59:0/647286964 >> 172.21.15.59:0/647286964 conn(0x7fc20c0d3cd0 msgr2=0x7fc20c06cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:20.611 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.609+0000 7fc21177a640 1 -- 172.21.15.59:0/647286964 shutdown_connections 2024-03-06T07:34:20.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:20.609+0000 7fc21177a640 1 -- 172.21.15.59:0/647286964 wait complete. 2024-03-06T07:34:20.822 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 108, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:13.446245+0000', 'modified': '2024-03-06T07:32:44.076826+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 9336, 'mds_1': 8236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8236': {'gid': 8236, 'name': 'b', 'rank': 1, 'incarnation': 99, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.164:6837/997138956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6835', 'nonce': 997138956}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 997138956}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:31:40.104271+0000', 'export_targets': [0], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_9336': {'gid': 9336, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.59:6833/720863754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.59:6832', 'nonce': 720863754}, {'type': 'v1', 'addr': '172.21.15.59:6833', 'nonce': 720863754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9336, 'qdb_cluster': [9336]} 2024-03-06T07:34:20.823 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8236:up:stopping 2024-03-06T07:34:21.620 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:21.623 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:21.824 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:30:13.446245+0000", "damaged": [], "data_pools": [ 21 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8236": { "addr": "172.21.15.164:6837/997138956", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6835", "nonce": 997138956, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 997138956, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540701547738038271, "flags": 0, "gid": 8236, "incarnation": 99, "join_fscid": -1, "laggy_since": "2024-03-06T07:31:40.104271+0000", "name": "b", "rank": 1, "state": "up:stopping", "state_seq": 5 }, "gid_9336": { "addr": "172.21.15.59:6833/720863754", "addrs": { "addrvec": [ { "addr": "172.21.15.59:6832", "nonce": 720863754, "type": "v2" }, { "addr": "172.21.15.59:6833", "nonce": 720863754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9336, "incarnation": 95, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 20, "modified": "2024-03-06T07:32:44.076826+0000", "qdb_cluster": [ 9336 ], "qdb_leader": 9336, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9336, "mds_1": 8236 } } } ], "standbys": [] } 2024-03-06T07:34:21.824 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:34:21.824 DEBUG:teuthology.orchestra.run.smithi059:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:21.872 DEBUG:teuthology.orchestra.run.smithi164:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:34:21.923 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:34:21.924 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:34:21.924 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:21.939 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:21.939 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:34:21.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:21.940 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:34:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-06T07:34:21.994 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:34:21.994 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:22.012 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:22.012 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:34:22.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:22.012 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:34:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:22.069 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:34:22.069 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:34:22.069 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:22.084 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:22.084 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:34:22.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:22.084 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:34:22.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:22.139 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:34:22.139 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:34:22.139 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:22.156 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:22.156 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:34:22.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:22.156 DEBUG:teuthology.orchestra.run.smithi164:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:34:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:34:22.211 INFO:teuthology.orchestra.run.smithi164.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:34:22.212 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:22.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.311+0000 7fd112e0e640 1 Processor -- start 2024-03-06T07:34:22.313 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.311+0000 7fd112e0e640 1 -- start start 2024-03-06T07:34:22.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.312+0000 7fd112e0e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd10c0d2710 0x7fd10c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.312+0000 7fd112e0e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10c0b3430 con 0x7fd10c0d22b0 2024-03-06T07:34:22.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.312+0000 7fd112e0e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10c0b3710 con 0x7fd10c15dfe0 2024-03-06T07:34:22.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.312+0000 7fd112e0e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd10c0b31e0 con 0x7fd10c0d2710 2024-03-06T07:34:22.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.312+0000 7fd10bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd10c0d22b0 0x7fd10c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38944/0 (socket says 172.21.15.59:38944) 2024-03-06T07:34:22.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.312+0000 7fd10bfff640 1 -- 172.21.15.59:0/2038640629 learned_addr learned my addr 172.21.15.59:0/2038640629 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:22.316 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.312+0000 7fd1113ab640 1 --2- 172.21.15.59:0/2038640629 >> v2:172.21.15.164:3300/0 conn(0x7fd10c0d2710 0x7fd10c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.312+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056988626 0 0) 0x7fd10c0b3430 con 0x7fd10c0d22b0 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0fc003520 con 0x7fd10c0d22b0 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3227096468 0 0) 0x7fd0fc003520 con 0x7fd10c0d22b0 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 >> v2:172.21.15.164:3300/0 conn(0x7fd10c0d2710 msgr2=0x7fd10c0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 --2- 172.21.15.59:0/2038640629 >> v2:172.21.15.164:3300/0 conn(0x7fd10c0d2710 0x7fd10c0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 >> v1:172.21.15.164:6789/0 conn(0x7fd10c15dfe0 legacy=0x7fd10c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10c0b39f0 con 0x7fd10c0d22b0 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fd0f8001670 con 0x7fd10c0d22b0 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 >> v1:172.21.15.59:6789/0 conn(0x7fd10c0d22b0 legacy=0x7fd10c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 --2- 172.21.15.59:0/2038640629 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0fc0039e0 0x7fd0fc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 --2- 172.21.15.59:0/2038640629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0fc004800 0x7fd0fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 --2- 172.21.15.59:0/2038640629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 0x7fd0fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0b31e0 con 0x7fd0fc0051c0 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc00c090 con 0x7fd0fc004800 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc00c290 con 0x7fd0fc0039e0 2024-03-06T07:34:22.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd1113ab640 1 --2- 172.21.15.59:0/2038640629 >> v2:172.21.15.164:3300/0 conn(0x7fd10c0d2710 0x7fd10c0b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd110baa640 1 --2- 172.21.15.59:0/2038640629 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0fc0039e0 0x7fd0fc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd1113ab640 1 --2- 172.21.15.59:0/2038640629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 0x7fd0fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd10bfff640 1 --2- 172.21.15.59:0/2038640629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0fc004800 0x7fd0fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd1113ab640 1 -- 172.21.15.59:0/2038640629 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0fc0039e0 msgr2=0x7fd0fc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd1113ab640 1 --2- 172.21.15.59:0/2038640629 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0fc0039e0 0x7fd0fc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd1113ab640 1 -- 172.21.15.59:0/2038640629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0fc004800 msgr2=0x7fd0fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd1113ab640 1 --2- 172.21.15.59:0/2038640629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0fc004800 0x7fd0fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.313+0000 7fd1113ab640 1 -- 172.21.15.59:0/2038640629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc003520 con 0x7fd0fc0051c0 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.314+0000 7fd1113ab640 1 -- 172.21.15.59:0/2038640629 reap_dead start 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.314+0000 7fd1113ab640 1 --2- 172.21.15.59:0/2038640629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 0x7fd0fc00b9c0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fd10400e350 tx=0x7fd0fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a74d784d05297 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.314+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd104009dc0 con 0x7fd0fc0051c0 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.314+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10402ce00 con 0x7fd0fc0051c0 2024-03-06T07:34:22.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.314+0000 7fd10b7fe640 1 -- 172.21.15.59:0/2038640629 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd104003bd0 con 0x7fd0fc0051c0 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.315+0000 7fd112e0e640 1 -- 172.21.15.59:0/2038640629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 msgr2=0x7fd0fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.315+0000 7fd112e0e640 1 --2- 172.21.15.59:0/2038640629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 0x7fd0fc00b9c0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fd10400e350 tx=0x7fd0fc005830 comp rx=0 tx=0).stop 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.315+0000 7fd112e0e640 1 -- 172.21.15.59:0/2038640629 shutdown_connections 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.315+0000 7fd112e0e640 1 --2- 172.21.15.59:0/2038640629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 0x7fd0fc00b9c0 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.315+0000 7fd112e0e640 1 -- 172.21.15.59:0/2038640629 >> 172.21.15.59:0/2038640629 conn(0x7fd10c0d3cd0 msgr2=0x7fd10c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.315+0000 7fd112e0e640 1 -- 172.21.15.59:0/2038640629 shutdown_connections 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.315+0000 7fd112e0e640 1 -- 172.21.15.59:0/2038640629 wait complete. 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.316+0000 7fd112e0e640 1 Processor -- start 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.316+0000 7fd112e0e640 1 -- start start 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd112e0e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 0x7fd10c15c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd112e0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd10c0b68d0 0x7fd10c0b6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd112e0e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd10c0cc5c0 0x7fd10c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd112e0e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0d1c70 con 0x7fd0fc0051c0 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd112e0e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0b3d60 con 0x7fd10c0cc5c0 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd112e0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c0d1df0 con 0x7fd10c0b68d0 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd110baa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 0x7fd10c15c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd110baa640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 0x7fd10c15c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57988/0 (socket says 172.21.15.59:57988) 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd110baa640 1 -- 172.21.15.59:0/3874531991 learned_addr learned my addr 172.21.15.59:0/3874531991 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd1113ab640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd10c0cc5c0 0x7fd10c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd10bfff640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd10c0b68d0 0x7fd10c0b6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd110baa640 1 -- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd10c0b68d0 msgr2=0x7fd10c0b6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd110baa640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd10c0b68d0 0x7fd10c0b6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd110baa640 1 -- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd10c0cc5c0 msgr2=0x7fd10c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd110baa640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd10c0cc5c0 0x7fd10c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd110baa640 1 -- 172.21.15.59:0/3874531991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10c0b7330 con 0x7fd0fc0051c0 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.317+0000 7fd110baa640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 0x7fd10c15c410 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fd10000f620 tx=0x7fd10000fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.318+0000 7fd1097fa640 1 -- 172.21.15.59:0/3874531991 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1000052e0 con 0x7fd0fc0051c0 2024-03-06T07:34:22.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.318+0000 7fd1097fa640 1 -- 172.21.15.59:0/3874531991 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd100018070 con 0x7fd0fc0051c0 2024-03-06T07:34:22.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.318+0000 7fd112e0e640 1 -- 172.21.15.59:0/3874531991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10c0b7610 con 0x7fd0fc0051c0 2024-03-06T07:34:22.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.318+0000 7fd1097fa640 1 -- 172.21.15.59:0/3874531991 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100003810 con 0x7fd0fc0051c0 2024-03-06T07:34:22.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.318+0000 7fd112e0e640 1 -- 172.21.15.59:0/3874531991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10c0b7bd0 con 0x7fd0fc0051c0 2024-03-06T07:34:22.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.319+0000 7fd112e0e640 1 -- 172.21.15.59:0/3874531991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd10c04f330 con 0x7fd0fc0051c0 2024-03-06T07:34:22.323 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.320+0000 7fd1097fa640 1 -- 172.21.15.59:0/3874531991 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1000039b0 con 0x7fd0fc0051c0 2024-03-06T07:34:22.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.320+0000 7fd1097fa640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0f006b500 0x7fd0f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.320+0000 7fd1097fa640 1 -- 172.21.15.59:0/3874531991 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd100089fa0 con 0x7fd0fc0051c0 2024-03-06T07:34:22.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.320+0000 7fd10bfff640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0f006b500 0x7fd0f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.321+0000 7fd10bfff640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0f006b500 0x7fd0f006d9c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8001050 tx=0x7fd0f8009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.324+0000 7fd1097fa640 1 -- 172.21.15.59:0/3874531991 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd100058e00 con 0x7fd0fc0051c0 2024-03-06T07:34:22.612 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.611+0000 7fd112e0e640 1 -- 172.21.15.59:0/3874531991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd10c0d4a10 con 0x7fd0fc0051c0 2024-03-06T07:34:22.613 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.611+0000 7fd1097fa640 1 -- 172.21.15.59:0/3874531991 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7fd100058700 con 0x7fd0fc0051c0 2024-03-06T07:34:22.613 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:22.613 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:22.613 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 -- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0f006b500 msgr2=0x7fd0f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0f006b500 0x7fd0f006d9c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8001050 tx=0x7fd0f8009040 comp rx=0 tx=0).stop 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 -- 172.21.15.59:0/3874531991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 msgr2=0x7fd10c15c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 0x7fd10c15c410 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fd10000f620 tx=0x7fd10000fae0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 -- 172.21.15.59:0/3874531991 shutdown_connections 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0f006b500 0x7fd0f006d9c0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd10c0cc5c0 0x7fd10c0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd10c0b68d0 0x7fd10c0b6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 --2- 172.21.15.59:0/3874531991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0fc0051c0 0x7fd10c15c410 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 -- 172.21.15.59:0/3874531991 >> 172.21.15.59:0/3874531991 conn(0x7fd10c0d3cd0 msgr2=0x7fd10c0b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 -- 172.21.15.59:0/3874531991 shutdown_connections 2024-03-06T07:34:22.616 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.614+0000 7fd112e0e640 1 -- 172.21.15.59:0/3874531991 wait complete. 2024-03-06T07:34:22.627 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:22.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.726+0000 7f43367e2640 1 Processor -- start 2024-03-06T07:34:22.728 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.726+0000 7f43367e2640 1 -- start start 2024-03-06T07:34:22.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f43367e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300d22b0 0x7f43300d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f43367e2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43300b3430 con 0x7f433015dfe0 2024-03-06T07:34:22.729 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f43367e2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43300b3710 con 0x7f43300b41f0 2024-03-06T07:34:22.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f43367e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43300d2c50 con 0x7f43300d22b0 2024-03-06T07:34:22.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f432ffff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f433015dfe0 0x7f43300cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38956/0 (socket says 172.21.15.59:38956) 2024-03-06T07:34:22.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f432ffff640 1 -- 172.21.15.59:0/4025127011 learned_addr learned my addr 172.21.15.59:0/4025127011 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:22.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f432f7fe640 1 --2- 172.21.15.59:0/4025127011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300d22b0 0x7f43300d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f432effd640 1 -- 172.21.15.59:0/4025127011 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157338810 0 0) 0x7f43300b3430 con 0x7f433015dfe0 2024-03-06T07:34:22.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f432effd640 1 -- 172.21.15.59:0/4025127011 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f431c003520 con 0x7f433015dfe0 2024-03-06T07:34:22.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f432effd640 1 -- 172.21.15.59:0/4025127011 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787065017 0 0) 0x7f43300b3710 con 0x7f43300b41f0 2024-03-06T07:34:22.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f432f7fe640 1 -- 172.21.15.59:0/4025127011 >> v1:172.21.15.164:6789/0 conn(0x7f43300b41f0 legacy=0x7f43300b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f432f7fe640 1 -- 172.21.15.59:0/4025127011 >> v1:172.21.15.59:6789/0 conn(0x7f433015dfe0 legacy=0x7f43300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.730 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.727+0000 7f432f7fe640 1 -- 172.21.15.59:0/4025127011 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43300b39f0 con 0x7f43300d22b0 2024-03-06T07:34:22.731 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.728+0000 7f432f7fe640 1 --2- 172.21.15.59:0/4025127011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300d22b0 0x7f43300d2710 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f43180097d0 tx=0x7f431802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6044d5550d3e31c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.728+0000 7f432effd640 1 -- 172.21.15.59:0/4025127011 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318035020 con 0x7f43300d22b0 2024-03-06T07:34:22.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.728+0000 7f432effd640 1 -- 172.21.15.59:0/4025127011 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4318031890 con 0x7f43300d22b0 2024-03-06T07:34:22.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.728+0000 7f432effd640 1 -- 172.21.15.59:0/4025127011 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318031c40 con 0x7f43300d22b0 2024-03-06T07:34:22.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.729+0000 7f43367e2640 1 -- 172.21.15.59:0/4025127011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300d22b0 msgr2=0x7f43300d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.729+0000 7f43367e2640 1 --2- 172.21.15.59:0/4025127011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300d22b0 0x7f43300d2710 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f43180097d0 tx=0x7f431802b930 comp rx=0 tx=0).stop 2024-03-06T07:34:22.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.729+0000 7f43367e2640 1 -- 172.21.15.59:0/4025127011 shutdown_connections 2024-03-06T07:34:22.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.729+0000 7f43367e2640 1 --2- 172.21.15.59:0/4025127011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300d22b0 0x7f43300d2710 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.729+0000 7f43367e2640 1 -- 172.21.15.59:0/4025127011 >> 172.21.15.59:0/4025127011 conn(0x7f43300d3cd0 msgr2=0x7f43300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.729+0000 7f43367e2640 1 -- 172.21.15.59:0/4025127011 shutdown_connections 2024-03-06T07:34:22.732 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.729+0000 7f43367e2640 1 -- 172.21.15.59:0/4025127011 wait complete. 2024-03-06T07:34:22.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.731+0000 7f43367e2640 1 Processor -- start 2024-03-06T07:34:22.733 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.731+0000 7f43367e2640 1 -- start start 2024-03-06T07:34:22.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.732+0000 7f43367e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300b41f0 0x7f43301f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.732+0000 7f43367e2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43300d22b0 0x7f43300b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.735 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.732+0000 7f43367e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f433015dfe0 0x7f43300b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.732+0000 7f43367e2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43300b3d60 con 0x7f43300d22b0 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.732+0000 7f43367e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43300b31e0 con 0x7f433015dfe0 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.732+0000 7f43367e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43300d1c70 con 0x7f43300b41f0 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.732+0000 7f432f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43300d22b0 0x7f43300b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.732+0000 7f432f7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43300d22b0 0x7f43300b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58004/0 (socket says 172.21.15.59:58004) 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.732+0000 7f432f7fe640 1 -- 172.21.15.59:0/3626039507 learned_addr learned my addr 172.21.15.59:0/3626039507 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.732+0000 7f432ffff640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300b41f0 0x7f43301f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f4334d7f640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f433015dfe0 0x7f43300b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f432f7fe640 1 -- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300b41f0 msgr2=0x7f43301f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f432f7fe640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300b41f0 0x7f43301f2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f432f7fe640 1 -- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f433015dfe0 msgr2=0x7f43300b92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f432f7fe640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f433015dfe0 0x7f43300b92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f432f7fe640 1 -- 172.21.15.59:0/3626039507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43300b7de0 con 0x7f43300d22b0 2024-03-06T07:34:22.736 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f432ffff640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300b41f0 0x7f43301f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:22.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f432f7fe640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43300d22b0 0x7f43300b8d70 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f4318000c00 tx=0x7f43180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f432cff9640 1 -- 172.21.15.59:0/3626039507 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318035040 con 0x7f43300d22b0 2024-03-06T07:34:22.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f432cff9640 1 -- 172.21.15.59:0/3626039507 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4318030d20 con 0x7f43300d22b0 2024-03-06T07:34:22.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f432cff9640 1 -- 172.21.15.59:0/3626039507 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43180466e0 con 0x7f43300d22b0 2024-03-06T07:34:22.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f43367e2640 1 -- 172.21.15.59:0/3626039507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433015f4b0 con 0x7f43300d22b0 2024-03-06T07:34:22.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.733+0000 7f43367e2640 1 -- 172.21.15.59:0/3626039507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43300b80c0 con 0x7f43300d22b0 2024-03-06T07:34:22.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.734+0000 7f43367e2640 1 -- 172.21.15.59:0/3626039507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433004f330 con 0x7f43300d22b0 2024-03-06T07:34:22.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.734+0000 7f432cff9640 1 -- 172.21.15.59:0/3626039507 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4318038020 con 0x7f43300d22b0 2024-03-06T07:34:22.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.735+0000 7f432cff9640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431006b4d0 0x7f431006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:22.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.735+0000 7f432cff9640 1 -- 172.21.15.59:0/3626039507 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f43180ac970 con 0x7f43300d22b0 2024-03-06T07:34:22.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.735+0000 7f432ffff640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431006b4d0 0x7f431006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:22.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.736+0000 7f432ffff640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431006b4d0 0x7f431006d990 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f4320007fd0 tx=0x7f4320009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:22.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.739+0000 7f432cff9640 1 -- 172.21.15.59:0/3626039507 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f431802d040 con 0x7f43300d22b0 2024-03-06T07:34:22.979 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.977+0000 7f43367e2640 1 -- 172.21.15.59:0/3626039507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43300c9fc0 con 0x7f43300d22b0 2024-03-06T07:34:22.980 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.978+0000 7f432cff9640 1 -- 172.21.15.59:0/3626039507 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f431807b7d0 con 0x7f43300d22b0 2024-03-06T07:34:22.981 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:22.981 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":143,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:39.494279+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:11.712614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:12.406781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.22","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.980+0000 7f43367e2640 1 -- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431006b4d0 msgr2=0x7f431006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.980+0000 7f43367e2640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431006b4d0 0x7f431006d990 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f4320007fd0 tx=0x7f4320009040 comp rx=0 tx=0).stop 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.980+0000 7f43367e2640 1 -- 172.21.15.59:0/3626039507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43300d22b0 msgr2=0x7f43300b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.980+0000 7f43367e2640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43300d22b0 0x7f43300b8d70 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f4318000c00 tx=0x7f43180046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.980+0000 7f43367e2640 1 -- 172.21.15.59:0/3626039507 shutdown_connections 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.980+0000 7f43367e2640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f431006b4d0 0x7f431006d990 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.980+0000 7f43367e2640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f433015dfe0 0x7f43300b92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.980+0000 7f43367e2640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43300d22b0 0x7f43300b8d70 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.980+0000 7f43367e2640 1 --2- 172.21.15.59:0/3626039507 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43300b41f0 0x7f43301f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.980+0000 7f43367e2640 1 -- 172.21.15.59:0/3626039507 >> 172.21.15.59:0/3626039507 conn(0x7f43300d3cd0 msgr2=0x7f43300d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.981+0000 7f43367e2640 1 -- 172.21.15.59:0/3626039507 shutdown_connections 2024-03-06T07:34:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:22.981+0000 7f43367e2640 1 -- 172.21.15.59:0/3626039507 wait complete. 2024-03-06T07:34:22.997 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:41.343905+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:11.712614+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:12.406781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:22.997 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:23.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.099+0000 7fde72efb640 1 Processor -- start 2024-03-06T07:34:23.101 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.100+0000 7fde72efb640 1 -- start start 2024-03-06T07:34:23.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.100+0000 7fde72efb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde6c1614d0 0x7fde6c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.100+0000 7fde72efb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fde6c0d2480 0x7fde6c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.100+0000 7fde72efb640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c1618b0 con 0x7fde6c0d1910 2024-03-06T07:34:23.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.100+0000 7fde72efb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6c0cf400 con 0x7fde6c0d2480 2024-03-06T07:34:23.104 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.100+0000 7fde72efb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0cf6a0 con 0x7fde6c1614d0 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.100+0000 7fde6bfff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fde6c0d1910 0x7fde6c0d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38960/0 (socket says 172.21.15.59:38960) 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.100+0000 7fde6bfff640 1 -- 172.21.15.59:0/1840347049 learned_addr learned my addr 172.21.15.59:0/1840347049 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde71498640 1 --2- 172.21.15.59:0/1840347049 >> v2:172.21.15.164:3300/0 conn(0x7fde6c0d2480 0x7fde6c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde70c97640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde6c1614d0 0x7fde6c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783820796 0 0) 0x7fde6c1618b0 con 0x7fde6c0d1910 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde54003520 con 0x7fde6c0d1910 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1032336500 0 0) 0x7fde54003520 con 0x7fde6c0d1910 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde6c1614d0 msgr2=0x7fde6c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde6c1614d0 0x7fde6c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 >> v2:172.21.15.164:3300/0 conn(0x7fde6c0d2480 msgr2=0x7fde6c0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 --2- 172.21.15.59:0/1840347049 >> v2:172.21.15.164:3300/0 conn(0x7fde6c0d2480 0x7fde6c0b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0cf980 con 0x7fde6c0d1910 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fde60001670 con 0x7fde6c0d1910 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde71498640 1 --2- 172.21.15.59:0/1840347049 >> v2:172.21.15.164:3300/0 conn(0x7fde6c0d2480 0x7fde6c0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 >> v1:172.21.15.59:6789/0 conn(0x7fde6c0d1910 legacy=0x7fde6c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.105 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde540039b0 0x7fde54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde54004800 0x7fde54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde6c1614d0 0x7fde540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0cf400 con 0x7fde54004800 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0cf6a0 con 0x7fde540039b0 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde5400b9c0 con 0x7fde6c1614d0 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde6bfff640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde54004800 0x7fde54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde70c97640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde540039b0 0x7fde54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.101+0000 7fde71498640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde6c1614d0 0x7fde540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.102+0000 7fde6bfff640 1 -- 172.21.15.59:0/1840347049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde6c1614d0 msgr2=0x7fde540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.102+0000 7fde6bfff640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde6c1614d0 0x7fde540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.102+0000 7fde6bfff640 1 -- 172.21.15.59:0/1840347049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde540039b0 msgr2=0x7fde54003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.102+0000 7fde6bfff640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde540039b0 0x7fde54003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.102+0000 7fde6bfff640 1 -- 172.21.15.59:0/1840347049 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde54003520 con 0x7fde54004800 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.102+0000 7fde6bfff640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde54004800 0x7fde54004c80 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fde60001050 tx=0x7fde54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c7e4937fc203d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.106 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.102+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60007d70 con 0x7fde54004800 2024-03-06T07:34:23.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.102+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde60030070 con 0x7fde54004800 2024-03-06T07:34:23.107 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.103+0000 7fde6b7fe640 1 -- 172.21.15.59:0/1840347049 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6002cd80 con 0x7fde54004800 2024-03-06T07:34:23.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.103+0000 7fde72efb640 1 -- 172.21.15.59:0/1840347049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde54004800 msgr2=0x7fde54004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.103+0000 7fde72efb640 1 --2- 172.21.15.59:0/1840347049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde54004800 0x7fde54004c80 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fde60001050 tx=0x7fde54005830 comp rx=0 tx=0).stop 2024-03-06T07:34:23.109 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.103+0000 7fde71498640 1 -- 172.21.15.59:0/1840347049 reap_dead start 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.103+0000 7fde72efb640 1 -- 172.21.15.59:0/1840347049 shutdown_connections 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.103+0000 7fde72efb640 1 -- 172.21.15.59:0/1840347049 >> 172.21.15.59:0/1840347049 conn(0x7fde6c0d3d40 msgr2=0x7fde6c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.103+0000 7fde72efb640 1 -- 172.21.15.59:0/1840347049 shutdown_connections 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.103+0000 7fde72efb640 1 -- 172.21.15.59:0/1840347049 wait complete. 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.104+0000 7fde72efb640 1 Processor -- start 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.104+0000 7fde72efb640 1 -- start start 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.105+0000 7fde72efb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde54004800 0x7fde6c0b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.105+0000 7fde72efb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde6c1f3900 0x7fde6c148590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.105+0000 7fde72efb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde6c0b2300 0x7fde6c148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.105+0000 7fde72efb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0d4860 con 0x7fde6c1f3900 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.105+0000 7fde72efb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0b0910 con 0x7fde6c0b2300 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.105+0000 7fde72efb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0d49e0 con 0x7fde54004800 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.105+0000 7fde6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde6c1f3900 0x7fde6c148590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.105+0000 7fde6bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde6c1f3900 0x7fde6c148590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58024/0 (socket says 172.21.15.59:58024) 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.105+0000 7fde6bfff640 1 -- 172.21.15.59:0/1285623433 learned_addr learned my addr 172.21.15.59:0/1285623433 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:23.110 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.105+0000 7fde71498640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde6c0b2300 0x7fde6c148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.105+0000 7fde70c97640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde54004800 0x7fde6c0b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde6bfff640 1 -- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde54004800 msgr2=0x7fde6c0b1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde6bfff640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde54004800 0x7fde6c0b1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde6bfff640 1 -- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde6c0b2300 msgr2=0x7fde6c148ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde6bfff640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde6c0b2300 0x7fde6c148ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde6bfff640 1 -- 172.21.15.59:0/1285623433 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0b0d20 con 0x7fde6c1f3900 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde71498640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde6c0b2300 0x7fde6c148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde6bfff640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde6c1f3900 0x7fde6c148590 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fde6000c300 tx=0x7fde60000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde697fa640 1 -- 172.21.15.59:0/1285623433 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60045070 con 0x7fde6c1f3900 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde697fa640 1 -- 172.21.15.59:0/1285623433 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde60004bf0 con 0x7fde6c1f3900 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde72efb640 1 -- 172.21.15.59:0/1285623433 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6c0b1000 con 0x7fde6c1f3900 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde697fa640 1 -- 172.21.15.59:0/1285623433 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60007d50 con 0x7fde6c1f3900 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.106+0000 7fde72efb640 1 -- 172.21.15.59:0/1285623433 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6c0b15c0 con 0x7fde6c1f3900 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.107+0000 7fde72efb640 1 -- 172.21.15.59:0/1285623433 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6c0cdfd0 con 0x7fde6c1f3900 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.107+0000 7fde697fa640 1 -- 172.21.15.59:0/1285623433 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde60007070 con 0x7fde6c1f3900 2024-03-06T07:34:23.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.108+0000 7fde697fa640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde5806b500 0x7fde5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.108+0000 7fde697fa640 1 -- 172.21.15.59:0/1285623433 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fde600ab210 con 0x7fde6c1f3900 2024-03-06T07:34:23.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.108+0000 7fde70c97640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde5806b500 0x7fde5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.112 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.109+0000 7fde70c97640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde5806b500 0x7fde5806d9c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fde64014370 tx=0x7fde6400b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.112+0000 7fde697fa640 1 -- 172.21.15.59:0/1285623433 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde60079b00 con 0x7fde6c1f3900 2024-03-06T07:34:23.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.293+0000 7fde72efb640 1 -- 172.21.15.59:0/1285623433 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde6c0c95a0 con 0x7fde6c1f3900 2024-03-06T07:34:23.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.294+0000 7fde697fa640 1 -- 172.21.15.59:0/1285623433 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde6007dba0 con 0x7fde6c1f3900 2024-03-06T07:34:23.296 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2024-03-06T07:34:23.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 -- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde5806b500 msgr2=0x7fde5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde5806b500 0x7fde5806d9c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fde64014370 tx=0x7fde6400b040 comp rx=0 tx=0).stop 2024-03-06T07:34:23.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 -- 172.21.15.59:0/1285623433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde6c1f3900 msgr2=0x7fde6c148590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde6c1f3900 0x7fde6c148590 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fde6000c300 tx=0x7fde60000fe0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 -- 172.21.15.59:0/1285623433 shutdown_connections 2024-03-06T07:34:23.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fde5806b500 0x7fde5806d9c0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde6c0b2300 0x7fde6c148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde6c1f3900 0x7fde6c148590 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 --2- 172.21.15.59:0/1285623433 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde54004800 0x7fde6c0b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 -- 172.21.15.59:0/1285623433 >> 172.21.15.59:0/1285623433 conn(0x7fde6c0d3d40 msgr2=0x7fde6c0c7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:23.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 -- 172.21.15.59:0/1285623433 shutdown_connections 2024-03-06T07:34:23.299 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.296+0000 7fde72efb640 1 -- 172.21.15.59:0/1285623433 wait complete. 2024-03-06T07:34:23.309 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:34:23.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.414+0000 7faa7015c640 1 Processor -- start 2024-03-06T07:34:23.416 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.415+0000 7faa7015c640 1 -- start start 2024-03-06T07:34:23.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.415+0000 7faa7015c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faa680d10c0 0x7faa680d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.415+0000 7faa7015c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7faa680d14a0 0x7faa6815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.415+0000 7faa7015c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa680cfa50 con 0x7faa6815eb00 2024-03-06T07:34:23.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.415+0000 7faa7015c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6815c050 con 0x7faa680d14a0 2024-03-06T07:34:23.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.416+0000 7faa7015c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa680d0060 con 0x7faa680d10c0 2024-03-06T07:34:23.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.416+0000 7faa6d6f7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7faa680d14a0 0x7faa6815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.416+0000 7faa6d6f7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7faa680d14a0 0x7faa6815e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58028/0 (socket says 172.21.15.59:58028) 2024-03-06T07:34:23.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.416+0000 7faa6d6f7640 1 -- 172.21.15.59:0/3862438378 learned_addr learned my addr 172.21.15.59:0/3862438378 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:23.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.416+0000 7faa6def8640 1 --2- 172.21.15.59:0/3862438378 >> v2:172.21.15.164:3300/0 conn(0x7faa680d10c0 0x7faa680d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.418 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.416+0000 7faa6d6f7640 1 -- 172.21.15.59:0/3862438378 >> v2:172.21.15.164:3300/0 conn(0x7faa680d10c0 msgr2=0x7faa680d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:23.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.416+0000 7faa6d6f7640 1 --2- 172.21.15.59:0/3862438378 >> v2:172.21.15.164:3300/0 conn(0x7faa680d10c0 0x7faa680d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.416+0000 7faa6d6f7640 1 -- 172.21.15.59:0/3862438378 >> v1:172.21.15.164:6789/0 conn(0x7faa6815eb00 legacy=0x7faa680cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.419 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.416+0000 7faa6d6f7640 1 -- 172.21.15.59:0/3862438378 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680d0a30 con 0x7faa680d14a0 2024-03-06T07:34:23.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.417+0000 7faa6d6f7640 1 --2- 172.21.15.59:0/3862438378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa680d14a0 0x7faa6815e5c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7faa680718d0 tx=0x7faa5802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c77d7440dcd6b1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.417+0000 7faa6cef6640 1 -- 172.21.15.59:0/3862438378 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58034070 con 0x7faa680d14a0 2024-03-06T07:34:23.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.417+0000 7faa6cef6640 1 -- 172.21.15.59:0/3862438378 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa58030860 con 0x7faa680d14a0 2024-03-06T07:34:23.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.417+0000 7faa6cef6640 1 -- 172.21.15.59:0/3862438378 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58030c30 con 0x7faa680d14a0 2024-03-06T07:34:23.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.418+0000 7faa7015c640 1 -- 172.21.15.59:0/3862438378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa680d14a0 msgr2=0x7faa6815e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.418+0000 7faa7015c640 1 --2- 172.21.15.59:0/3862438378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa680d14a0 0x7faa6815e5c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7faa680718d0 tx=0x7faa5802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.418+0000 7faa7015c640 1 -- 172.21.15.59:0/3862438378 shutdown_connections 2024-03-06T07:34:23.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.418+0000 7faa7015c640 1 --2- 172.21.15.59:0/3862438378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa680d14a0 0x7faa6815e5c0 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.418+0000 7faa7015c640 1 --2- 172.21.15.59:0/3862438378 >> v2:172.21.15.164:3300/0 conn(0x7faa680d10c0 0x7faa680d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.420 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.418+0000 7faa7015c640 1 -- 172.21.15.59:0/3862438378 >> 172.21.15.59:0/3862438378 conn(0x7faa680b3800 msgr2=0x7faa680b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:23.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.418+0000 7faa7015c640 1 -- 172.21.15.59:0/3862438378 shutdown_connections 2024-03-06T07:34:23.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.418+0000 7faa7015c640 1 -- 172.21.15.59:0/3862438378 wait complete. 2024-03-06T07:34:23.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.420+0000 7faa7015c640 1 Processor -- start 2024-03-06T07:34:23.421 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.420+0000 7faa7015c640 1 -- start start 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa7015c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa680d10c0 0x7faa68147170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa7015c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa680d14a0 0x7faa680b5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa7015c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa6815eb00 0x7faa680cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa7015c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6815f440 con 0x7faa6815eb00 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa7015c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6815f160 con 0x7faa680d14a0 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa7015c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa680cfcf0 con 0x7faa680d10c0 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6e6f9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa6815eb00 0x7faa680cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6e6f9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa6815eb00 0x7faa680cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58042/0 (socket says 172.21.15.59:58042) 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6e6f9640 1 -- 172.21.15.59:0/2099093079 learned_addr learned my addr 172.21.15.59:0/2099093079 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6def8640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa680d10c0 0x7faa68147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6d6f7640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa680d14a0 0x7faa680b5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.424 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6e6f9640 1 -- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa680d10c0 msgr2=0x7faa68147170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6e6f9640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa680d10c0 0x7faa68147170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6e6f9640 1 -- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa680d14a0 msgr2=0x7faa680b5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6e6f9640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa680d14a0 0x7faa680b5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6e6f9640 1 -- 172.21.15.59:0/2099093079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680ce4a0 con 0x7faa6815eb00 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6def8640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa680d10c0 0x7faa68147170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6d6f7640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa680d14a0 0x7faa680b5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.421+0000 7faa6e6f9640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa6815eb00 0x7faa680cc180 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7faa6000c7d0 tx=0x7faa6000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.422+0000 7faa5e7fc640 1 -- 172.21.15.59:0/2099093079 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60003b70 con 0x7faa6815eb00 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.422+0000 7faa5e7fc640 1 -- 172.21.15.59:0/2099093079 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa60003d30 con 0x7faa6815eb00 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.422+0000 7faa5e7fc640 1 -- 172.21.15.59:0/2099093079 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6000aa30 con 0x7faa6815eb00 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.422+0000 7faa7015c640 1 -- 172.21.15.59:0/2099093079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa680ce780 con 0x7faa6815eb00 2024-03-06T07:34:23.425 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.422+0000 7faa7015c640 1 -- 172.21.15.59:0/2099093079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa680b5690 con 0x7faa6815eb00 2024-03-06T07:34:23.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.423+0000 7faa7015c640 1 -- 172.21.15.59:0/2099093079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6809fe60 con 0x7faa6815eb00 2024-03-06T07:34:23.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.423+0000 7faa5e7fc640 1 -- 172.21.15.59:0/2099093079 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa6000abd0 con 0x7faa6815eb00 2024-03-06T07:34:23.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.424+0000 7faa5e7fc640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faa4c06b4d0 0x7faa4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.424+0000 7faa5e7fc640 1 -- 172.21.15.59:0/2099093079 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faa6008a190 con 0x7faa6815eb00 2024-03-06T07:34:23.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.424+0000 7faa6def8640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faa4c06b4d0 0x7faa4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.426 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.424+0000 7faa6def8640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faa4c06b4d0 0x7faa4c06d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7faa64009af0 tx=0x7faa64008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.430 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.428+0000 7faa5e7fc640 1 -- 172.21.15.59:0/2099093079 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa60058f50 con 0x7faa6815eb00 2024-03-06T07:34:23.614 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.612+0000 7faa7015c640 1 -- 172.21.15.59:0/2099093079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa680d0c30 con 0x7faa6815eb00 2024-03-06T07:34:23.615 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.612+0000 7faa5e7fc640 1 -- 172.21.15.59:0/2099093079 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v143) v1 ==== 104+0+11 (secure 0 0 0) 0x7faa60058850 con 0x7faa6815eb00 2024-03-06T07:34:23.615 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 -- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faa4c06b4d0 msgr2=0x7faa4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faa4c06b4d0 0x7faa4c06d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7faa64009af0 tx=0x7faa64008040 comp rx=0 tx=0).stop 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 -- 172.21.15.59:0/2099093079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa6815eb00 msgr2=0x7faa680cc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa6815eb00 0x7faa680cc180 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7faa6000c7d0 tx=0x7faa6000cc90 comp rx=0 tx=0).stop 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 -- 172.21.15.59:0/2099093079 shutdown_connections 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7faa4c06b4d0 0x7faa4c06d990 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa6815eb00 0x7faa680cc180 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa680d14a0 0x7faa680b5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 --2- 172.21.15.59:0/2099093079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa680d10c0 0x7faa68147170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 -- 172.21.15.59:0/2099093079 >> 172.21.15.59:0/2099093079 conn(0x7faa680b3800 msgr2=0x7faa68070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 -- 172.21.15.59:0/2099093079 shutdown_connections 2024-03-06T07:34:23.617 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.614+0000 7faa7015c640 1 -- 172.21.15.59:0/2099093079 wait complete. 2024-03-06T07:34:23.630 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:34:23.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.735+0000 7f5790e71640 1 Processor -- start 2024-03-06T07:34:23.737 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.735+0000 7f5790e71640 1 -- start start 2024-03-06T07:34:23.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f5790e71640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f578c15c050 0x7f578c06f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f5790e71640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f578c06fb40 0x7f578c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f5790e71640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f578c0cd4b0 con 0x7f578c0ccca0 2024-03-06T07:34:23.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f5790e71640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f578c06ffb0 con 0x7f578c15c050 2024-03-06T07:34:23.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f5790e71640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f578c0cd790 con 0x7f578c06fb40 2024-03-06T07:34:23.738 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f578a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f578c15c050 0x7f578c06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f578ad9d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f578c0ccca0 0x7f578c0cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35616/0 (socket says 172.21.15.59:35616) 2024-03-06T07:34:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f578a59c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f578c15c050 0x7f578c06f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50530/0 (socket says 172.21.15.59:50530) 2024-03-06T07:34:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f578ad9d640 1 -- 172.21.15.59:0/460088353 learned_addr learned my addr 172.21.15.59:0/460088353 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f5789d9b640 1 --2- 172.21.15.59:0/460088353 >> v2:172.21.15.164:3300/0 conn(0x7f578c06fb40 0x7f578c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f578a59c640 1 -- 172.21.15.59:0/460088353 >> v2:172.21.15.164:3300/0 conn(0x7f578c06fb40 msgr2=0x7f578c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f578a59c640 1 --2- 172.21.15.59:0/460088353 >> v2:172.21.15.164:3300/0 conn(0x7f578c06fb40 0x7f578c0cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f578a59c640 1 -- 172.21.15.59:0/460088353 >> v1:172.21.15.59:6789/0 conn(0x7f578c0ccca0 legacy=0x7f578c0cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f578959a640 1 -- 172.21.15.59:0/460088353 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644263698 0 0) 0x7f578c0cd4b0 con 0x7f578c0ccca0 2024-03-06T07:34:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f578a59c640 1 -- 172.21.15.59:0/460088353 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c0cda70 con 0x7f578c15c050 2024-03-06T07:34:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.736+0000 7f5789d9b640 1 --2- 172.21.15.59:0/460088353 >> v2:172.21.15.164:3300/0 conn(0x7f578c06fb40 0x7f578c0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:23.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.737+0000 7f578a59c640 1 --2- 172.21.15.59:0/460088353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c15c050 0x7f578c06f600 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f578c0b7dc0 tx=0x7f577c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa91fbd7221e9e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.740 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.737+0000 7f578959a640 1 -- 172.21.15.59:0/460088353 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c0315a0 con 0x7f578c15c050 2024-03-06T07:34:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.737+0000 7f578959a640 1 -- 172.21.15.59:0/460088353 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577c037070 con 0x7f578c15c050 2024-03-06T07:34:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.737+0000 7f578959a640 1 -- 172.21.15.59:0/460088353 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c031960 con 0x7f578c15c050 2024-03-06T07:34:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.738+0000 7f5790e71640 1 -- 172.21.15.59:0/460088353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c15c050 msgr2=0x7f578c06f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.738+0000 7f5790e71640 1 --2- 172.21.15.59:0/460088353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c15c050 0x7f578c06f600 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f578c0b7dc0 tx=0x7f577c02b840 comp rx=0 tx=0).stop 2024-03-06T07:34:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.738+0000 7f5790e71640 1 -- 172.21.15.59:0/460088353 shutdown_connections 2024-03-06T07:34:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.738+0000 7f5790e71640 1 --2- 172.21.15.59:0/460088353 >> v2:172.21.15.164:3300/0 conn(0x7f578c06fb40 0x7f578c0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.738+0000 7f5790e71640 1 --2- 172.21.15.59:0/460088353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c15c050 0x7f578c06f600 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.738+0000 7f5790e71640 1 -- 172.21.15.59:0/460088353 >> 172.21.15.59:0/460088353 conn(0x7f578c0b3800 msgr2=0x7f578c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.738+0000 7f5790e71640 1 -- 172.21.15.59:0/460088353 shutdown_connections 2024-03-06T07:34:23.741 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.738+0000 7f5790e71640 1 -- 172.21.15.59:0/460088353 wait complete. 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.739+0000 7f5790e71640 1 Processor -- start 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.739+0000 7f5790e71640 1 -- start start 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.740+0000 7f5790e71640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f578c06fb40 0x7f578c156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.740+0000 7f5790e71640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0ccca0 0x7f578c154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.740+0000 7f5790e71640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f578c15c050 0x7f578c14d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.740+0000 7f5790e71640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c0d01f0 con 0x7f578c0ccca0 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.740+0000 7f5790e71640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c0cd130 con 0x7f578c06fb40 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.740+0000 7f5790e71640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578c0d0370 con 0x7f578c15c050 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.740+0000 7f5789d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0ccca0 0x7f578c154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.740+0000 7f5789d9b640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0ccca0 0x7f578c154940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50534/0 (socket says 172.21.15.59:50534) 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.740+0000 7f5789d9b640 1 -- 172.21.15.59:0/824371719 learned_addr learned my addr 172.21.15.59:0/824371719 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.740+0000 7f578a59c640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f578c06fb40 0x7f578c156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.744 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.740+0000 7f578ad9d640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f578c15c050 0x7f578c14d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f5789d9b640 1 -- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f578c15c050 msgr2=0x7f578c14d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f5789d9b640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f578c15c050 0x7f578c14d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f5789d9b640 1 -- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f578c06fb40 msgr2=0x7f578c156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f5789d9b640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f578c06fb40 0x7f578c156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f5789d9b640 1 -- 172.21.15.59:0/824371719 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c1550b0 con 0x7f578c0ccca0 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f578ad9d640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f578c15c050 0x7f578c14d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f5789d9b640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0ccca0 0x7f578c154940 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f578000c820 tx=0x7f578000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f577affd640 1 -- 172.21.15.59:0/824371719 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780015070 con 0x7f578c0ccca0 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f577affd640 1 -- 172.21.15.59:0/824371719 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5780007a30 con 0x7f578c0ccca0 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f5790e71640 1 -- 172.21.15.59:0/824371719 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578c155390 con 0x7f578c0ccca0 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f577affd640 1 -- 172.21.15.59:0/824371719 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780007de0 con 0x7f578c0ccca0 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.741+0000 7f5790e71640 1 -- 172.21.15.59:0/824371719 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578c0b4bd0 con 0x7f578c0ccca0 2024-03-06T07:34:23.745 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.742+0000 7f5790e71640 1 -- 172.21.15.59:0/824371719 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578c09fe60 con 0x7f578c0ccca0 2024-03-06T07:34:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.742+0000 7f577affd640 1 -- 172.21.15.59:0/824371719 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5780018070 con 0x7f578c0ccca0 2024-03-06T07:34:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.743+0000 7f577affd640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f576c06b4d0 0x7f576c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.743+0000 7f577affd640 1 -- 172.21.15.59:0/824371719 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f578008a140 con 0x7f578c0ccca0 2024-03-06T07:34:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.743+0000 7f578a59c640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f576c06b4d0 0x7f576c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:23.746 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.744+0000 7f578a59c640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f576c06b4d0 0x7f576c06d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f577c036000 tx=0x7f577c03d040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.747+0000 7f577affd640 1 -- 172.21.15.59:0/824371719 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5780058f20 con 0x7f578c0ccca0 2024-03-06T07:34:23.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.927+0000 7f5790e71640 1 -- 172.21.15.59:0/824371719 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f578c068ff0 con 0x7f578c0ccca0 2024-03-06T07:34:23.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.927+0000 7f577affd640 1 -- 172.21.15.59:0/824371719 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v143) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5780058820 con 0x7f578c0ccca0 2024-03-06T07:34:23.929 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 64 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.929+0000 7f5790e71640 1 -- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f576c06b4d0 msgr2=0x7f576c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.929+0000 7f5790e71640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f576c06b4d0 0x7f576c06d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f577c036000 tx=0x7f577c03d040 comp rx=0 tx=0).stop 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.929+0000 7f5790e71640 1 -- 172.21.15.59:0/824371719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0ccca0 msgr2=0x7f578c154940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.929+0000 7f5790e71640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0ccca0 0x7f578c154940 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f578000c820 tx=0x7f578000cce0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.929+0000 7f5790e71640 1 -- 172.21.15.59:0/824371719 shutdown_connections 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.929+0000 7f5790e71640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f576c06b4d0 0x7f576c06d990 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.929+0000 7f5790e71640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f578c15c050 0x7f578c14d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.929+0000 7f5790e71640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f578c0ccca0 0x7f578c154940 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.929+0000 7f5790e71640 1 --2- 172.21.15.59:0/824371719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f578c06fb40 0x7f578c156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.929+0000 7f5790e71640 1 -- 172.21.15.59:0/824371719 >> 172.21.15.59:0/824371719 conn(0x7f578c0b3800 msgr2=0x7f578c0d2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.930+0000 7f5790e71640 1 -- 172.21.15.59:0/824371719 shutdown_connections 2024-03-06T07:34:23.932 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:23.930+0000 7f5790e71640 1 -- 172.21.15.59:0/824371719 wait complete. 2024-03-06T07:34:23.946 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:24.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.045+0000 7f1a2f815640 1 Processor -- start 2024-03-06T07:34:24.047 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.046+0000 7f1a2f815640 1 -- start start 2024-03-06T07:34:24.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.046+0000 7f1a2f815640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1a28158600 0x7f1a28072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.046+0000 7f1a2f815640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a28162030 0x7f1a281624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.046+0000 7f1a2f815640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2815eb40 con 0x7f1a2815e3a0 2024-03-06T07:34:24.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.046+0000 7f1a2f815640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a28161ba0 con 0x7f1a28158600 2024-03-06T07:34:24.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.046+0000 7f1a2f815640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2815ee20 con 0x7f1a28162030 2024-03-06T07:34:24.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.047+0000 7f1a2d5b1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1a28158600 0x7f1a28072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.047+0000 7f1a2d5b1640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1a28158600 0x7f1a28072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50546/0 (socket says 172.21.15.59:50546) 2024-03-06T07:34:24.049 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.047+0000 7f1a2d5b1640 1 -- 172.21.15.59:0/1758105301 learned_addr learned my addr 172.21.15.59:0/1758105301 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:24.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.047+0000 7f1a2ddb2640 1 --2- 172.21.15.59:0/1758105301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a28162030 0x7f1a281624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.047+0000 7f1a2d5b1640 1 -- 172.21.15.59:0/1758105301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a28162030 msgr2=0x7f1a281624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.047+0000 7f1a2d5b1640 1 --2- 172.21.15.59:0/1758105301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a28162030 0x7f1a281624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.047+0000 7f1a2d5b1640 1 -- 172.21.15.59:0/1758105301 >> v1:172.21.15.164:6789/0 conn(0x7f1a2815e3a0 legacy=0x7f1a2806d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.050 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.047+0000 7f1a2d5b1640 1 -- 172.21.15.59:0/1758105301 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2815f100 con 0x7f1a28158600 2024-03-06T07:34:24.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.048+0000 7f1a2d5b1640 1 --2- 172.21.15.59:0/1758105301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a28158600 0x7f1a28072730 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009870 tx=0x7f1a2402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6a4b482f424a7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.048+0000 7f1a1ffff640 1 -- 172.21.15.59:0/1758105301 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a240317c0 con 0x7f1a28158600 2024-03-06T07:34:24.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.048+0000 7f1a1ffff640 1 -- 172.21.15.59:0/1758105301 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a24037070 con 0x7f1a28158600 2024-03-06T07:34:24.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.048+0000 7f1a1ffff640 1 -- 172.21.15.59:0/1758105301 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a24031b80 con 0x7f1a28158600 2024-03-06T07:34:24.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.049+0000 7f1a2f815640 1 -- 172.21.15.59:0/1758105301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a28158600 msgr2=0x7f1a28072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.049+0000 7f1a2f815640 1 --2- 172.21.15.59:0/1758105301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a28158600 0x7f1a28072730 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009870 tx=0x7f1a2402b860 comp rx=0 tx=0).stop 2024-03-06T07:34:24.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.049+0000 7f1a2f815640 1 -- 172.21.15.59:0/1758105301 shutdown_connections 2024-03-06T07:34:24.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.049+0000 7f1a2f815640 1 --2- 172.21.15.59:0/1758105301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a28162030 0x7f1a281624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.051 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.049+0000 7f1a2f815640 1 --2- 172.21.15.59:0/1758105301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a28158600 0x7f1a28072730 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.049+0000 7f1a2f815640 1 -- 172.21.15.59:0/1758105301 >> 172.21.15.59:0/1758105301 conn(0x7f1a2806c0c0 msgr2=0x7f1a2806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.049+0000 7f1a2f815640 1 -- 172.21.15.59:0/1758105301 shutdown_connections 2024-03-06T07:34:24.052 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.049+0000 7f1a2f815640 1 -- 172.21.15.59:0/1758105301 wait complete. 2024-03-06T07:34:24.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.050+0000 7f1a2f815640 1 Processor -- start 2024-03-06T07:34:24.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.050+0000 7f1a2f815640 1 -- start start 2024-03-06T07:34:24.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2f815640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a28158600 0x7f1a280c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.054 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2f815640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a2815e3a0 0x7f1a28159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2f815640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a28162030 0x7f1a2815a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2f815640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2806e080 con 0x7f1a28162030 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2f815640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2806afb0 con 0x7f1a28158600 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2f815640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2815ae50 con 0x7f1a2815e3a0 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2ddb2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a28162030 0x7f1a2815a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2ddb2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a28162030 0x7f1a2815a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50562/0 (socket says 172.21.15.59:50562) 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2ddb2640 1 -- 172.21.15.59:0/3943695596 learned_addr learned my addr 172.21.15.59:0/3943695596 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2d5b1640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a28158600 0x7f1a280c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2cdb0640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a2815e3a0 0x7f1a28159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2ddb2640 1 -- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a2815e3a0 msgr2=0x7f1a28159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2ddb2640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a2815e3a0 0x7f1a28159bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2ddb2640 1 -- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a28158600 msgr2=0x7f1a280c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2ddb2640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a28158600 0x7f1a280c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2ddb2640 1 -- 172.21.15.59:0/3943695596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2815b0a0 con 0x7f1a28162030 2024-03-06T07:34:24.055 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2cdb0640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a2815e3a0 0x7f1a28159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:24.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2ddb2640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a28162030 0x7f1a2815a110 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f1a20007d20 tx=0x7f1a2000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a1dffb640 1 -- 172.21.15.59:0/3943695596 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20014420 con 0x7f1a28162030 2024-03-06T07:34:24.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a1dffb640 1 -- 172.21.15.59:0/3943695596 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a20019070 con 0x7f1a28162030 2024-03-06T07:34:24.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a1dffb640 1 -- 172.21.15.59:0/3943695596 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20014820 con 0x7f1a28162030 2024-03-06T07:34:24.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.051+0000 7f1a2f815640 1 -- 172.21.15.59:0/3943695596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2815b380 con 0x7f1a28162030 2024-03-06T07:34:24.056 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.052+0000 7f1a2f815640 1 -- 172.21.15.59:0/3943695596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2815b940 con 0x7f1a28162030 2024-03-06T07:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.053+0000 7f1a2f815640 1 -- 172.21.15.59:0/3943695596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2804f330 con 0x7f1a28162030 2024-03-06T07:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.053+0000 7f1a1dffb640 1 -- 172.21.15.59:0/3943695596 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a200149c0 con 0x7f1a28162030 2024-03-06T07:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.053+0000 7f1a1dffb640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a0c06b4d0 0x7f1a0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.054+0000 7f1a1dffb640 1 -- 172.21.15.59:0/3943695596 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1a2008aab0 con 0x7f1a28162030 2024-03-06T07:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.054+0000 7f1a2d5b1640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a0c06b4d0 0x7f1a0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.057 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.054+0000 7f1a2d5b1640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a0c06b4d0 0x7f1a0c06d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1a24036000 tx=0x7f1a24008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.059 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.058+0000 7f1a1dffb640 1 -- 172.21.15.59:0/3943695596 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a20011260 con 0x7f1a28162030 2024-03-06T07:34:24.442 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.441+0000 7f1a2f815640 1 -- 172.21.15.59:0/3943695596 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a280d06f0 con 0x7f1a28162030 2024-03-06T07:34:24.443 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.441+0000 7f1a1dffb640 1 -- 172.21.15.59:0/3943695596 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f1a20059910 con 0x7f1a28162030 2024-03-06T07:34:24.443 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:24.443 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:24.444 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:24.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 -- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a0c06b4d0 msgr2=0x7f1a0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a0c06b4d0 0x7f1a0c06d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1a24036000 tx=0x7f1a24008030 comp rx=0 tx=0).stop 2024-03-06T07:34:24.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 -- 172.21.15.59:0/3943695596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a28162030 msgr2=0x7f1a2815a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a28162030 0x7f1a2815a110 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f1a20007d20 tx=0x7f1a2000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 -- 172.21.15.59:0/3943695596 shutdown_connections 2024-03-06T07:34:24.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1a0c06b4d0 0x7f1a0c06d990 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.446 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a28162030 0x7f1a2815a110 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a2815e3a0 0x7f1a28159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 --2- 172.21.15.59:0/3943695596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a28158600 0x7f1a280c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 -- 172.21.15.59:0/3943695596 >> 172.21.15.59:0/3943695596 conn(0x7f1a2806c0c0 msgr2=0x7f1a280b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 -- 172.21.15.59:0/3943695596 shutdown_connections 2024-03-06T07:34:24.447 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.444+0000 7f1a2f815640 1 -- 172.21.15.59:0/3943695596 wait complete. 2024-03-06T07:34:24.460 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:24.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.562+0000 7f29346df640 1 Processor -- start 2024-03-06T07:34:24.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.562+0000 7f29346df640 1 -- start start 2024-03-06T07:34:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.564+0000 7f29346df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f292c15bf40 0x7f292c0cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.564+0000 7f29346df640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292c0afe40 con 0x7f292c06baa0 2024-03-06T07:34:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.564+0000 7f29346df640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292c0b0120 con 0x7f292c06afb0 2024-03-06T07:34:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.564+0000 7f29346df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f292c0cd680 con 0x7f292c15bf40 2024-03-06T07:34:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.564+0000 7f2932c7c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f292c06baa0 0x7f292c0cf3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35624/0 (socket says 172.21.15.59:35624) 2024-03-06T07:34:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.564+0000 7f2932c7c640 1 -- 172.21.15.59:0/3352068994 learned_addr learned my addr 172.21.15.59:0/3352068994 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:24.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.564+0000 7f2931c7a640 1 --2- 172.21.15.59:0/3352068994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f292c15bf40 0x7f292c0cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.565+0000 7f2931c7a640 1 -- 172.21.15.59:0/3352068994 >> v1:172.21.15.164:6789/0 conn(0x7f292c06afb0 legacy=0x7f292c06b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.565+0000 7f2931c7a640 1 -- 172.21.15.59:0/3352068994 >> v1:172.21.15.59:6789/0 conn(0x7f292c06baa0 legacy=0x7f292c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.565+0000 7f2931c7a640 1 -- 172.21.15.59:0/3352068994 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c0cd310 con 0x7f292c15bf40 2024-03-06T07:34:24.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.565+0000 7f2931c7a640 1 --2- 172.21.15.59:0/3352068994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f292c15bf40 0x7f292c0cee60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f291c0097d0 tx=0x7f291c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d94e98f40915892 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.569+0000 7f2931479640 1 -- 172.21.15.59:0/3352068994 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291c035020 con 0x7f292c15bf40 2024-03-06T07:34:24.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.569+0000 7f2931479640 1 -- 172.21.15.59:0/3352068994 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291c031890 con 0x7f292c15bf40 2024-03-06T07:34:24.571 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.569+0000 7f2931479640 1 -- 172.21.15.59:0/3352068994 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291c031c60 con 0x7f292c15bf40 2024-03-06T07:34:24.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.570+0000 7f29346df640 1 -- 172.21.15.59:0/3352068994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f292c15bf40 msgr2=0x7f292c0cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.570+0000 7f29346df640 1 --2- 172.21.15.59:0/3352068994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f292c15bf40 0x7f292c0cee60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f291c0097d0 tx=0x7f291c02b930 comp rx=0 tx=0).stop 2024-03-06T07:34:24.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.570+0000 7f29346df640 1 -- 172.21.15.59:0/3352068994 shutdown_connections 2024-03-06T07:34:24.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.570+0000 7f29346df640 1 --2- 172.21.15.59:0/3352068994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f292c15bf40 0x7f292c0cee60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.570+0000 7f29346df640 1 -- 172.21.15.59:0/3352068994 >> 172.21.15.59:0/3352068994 conn(0x7f292c0d0960 msgr2=0x7f292c0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.570+0000 7f29346df640 1 -- 172.21.15.59:0/3352068994 shutdown_connections 2024-03-06T07:34:24.572 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.570+0000 7f29346df640 1 -- 172.21.15.59:0/3352068994 wait complete. 2024-03-06T07:34:24.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.572+0000 7f29346df640 1 Processor -- start 2024-03-06T07:34:24.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.572+0000 7f29346df640 1 -- start start 2024-03-06T07:34:24.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f29346df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f292c06afb0 0x7f292c1434b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f29346df640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f292c06baa0 0x7f292c155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f29346df640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f292c15bf40 0x7f292c1f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f29346df640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292c06f510 con 0x7f292c15bf40 2024-03-06T07:34:24.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f29346df640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292c06f390 con 0x7f292c06baa0 2024-03-06T07:34:24.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f29346df640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f292c06f690 con 0x7f292c06afb0 2024-03-06T07:34:24.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f2932c7c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f292c15bf40 0x7f292c1f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f2932c7c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f292c15bf40 0x7f292c1f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50578/0 (socket says 172.21.15.59:50578) 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f2932c7c640 1 -- 172.21.15.59:0/2023173659 learned_addr learned my addr 172.21.15.59:0/2023173659 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f293247b640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f292c06afb0 0x7f292c1434b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f2931c7a640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f292c06baa0 0x7f292c155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f2932c7c640 1 -- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f292c06afb0 msgr2=0x7f292c1434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f2932c7c640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f292c06afb0 0x7f292c1434b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f2932c7c640 1 -- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f292c06baa0 msgr2=0x7f292c155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f2932c7c640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f292c06baa0 0x7f292c155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f2932c7c640 1 -- 172.21.15.59:0/2023173659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c156390 con 0x7f292c15bf40 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.573+0000 7f2931c7a640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f292c06baa0 0x7f292c155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.574+0000 7f2932c7c640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f292c15bf40 0x7f292c1f5310 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f292400cbc0 tx=0x7f292400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.574+0000 7f2922ffd640 1 -- 172.21.15.59:0/2023173659 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2924017070 con 0x7f292c15bf40 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.574+0000 7f2922ffd640 1 -- 172.21.15.59:0/2023173659 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292400ad40 con 0x7f292c15bf40 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.574+0000 7f29346df640 1 -- 172.21.15.59:0/2023173659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f292c1536c0 con 0x7f292c15bf40 2024-03-06T07:34:24.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.574+0000 7f2922ffd640 1 -- 172.21.15.59:0/2023173659 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2924012600 con 0x7f292c15bf40 2024-03-06T07:34:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.574+0000 7f29346df640 1 -- 172.21.15.59:0/2023173659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f292c153c80 con 0x7f292c15bf40 2024-03-06T07:34:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.575+0000 7f29346df640 1 -- 172.21.15.59:0/2023173659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f292c04f330 con 0x7f292c15bf40 2024-03-06T07:34:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.576+0000 7f2922ffd640 1 -- 172.21.15.59:0/2023173659 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f292401a070 con 0x7f292c15bf40 2024-03-06T07:34:24.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.576+0000 7f2922ffd640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f291006b4d0 0x7f291006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.576+0000 7f2922ffd640 1 -- 172.21.15.59:0/2023173659 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2924089e90 con 0x7f292c15bf40 2024-03-06T07:34:24.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.576+0000 7f293247b640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f291006b4d0 0x7f291006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.577+0000 7f293247b640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f291006b4d0 0x7f291006d990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f292c15e840 tx=0x7f2928009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.580+0000 7f2922ffd640 1 -- 172.21.15.59:0/2023173659 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29240586a0 con 0x7f292c15bf40 2024-03-06T07:34:24.763 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.761+0000 7f29346df640 1 -- 172.21.15.59:0/2023173659 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f292c0729d0 con 0x7f292c15bf40 2024-03-06T07:34:24.764 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.762+0000 7f2922ffd640 1 -- 172.21.15.59:0/2023173659 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f292405c740 con 0x7f292c15bf40 2024-03-06T07:34:24.764 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:24.765 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":143,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:39.494279+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:11.712614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:12.406781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.22","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.764+0000 7f29346df640 1 -- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f291006b4d0 msgr2=0x7f291006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.764+0000 7f29346df640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f291006b4d0 0x7f291006d990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f292c15e840 tx=0x7f2928009040 comp rx=0 tx=0).stop 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.764+0000 7f29346df640 1 -- 172.21.15.59:0/2023173659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f292c15bf40 msgr2=0x7f292c1f5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.764+0000 7f29346df640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f292c15bf40 0x7f292c1f5310 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f292400cbc0 tx=0x7f292400a430 comp rx=0 tx=0).stop 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.764+0000 7f29346df640 1 -- 172.21.15.59:0/2023173659 shutdown_connections 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.764+0000 7f29346df640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f291006b4d0 0x7f291006d990 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.764+0000 7f29346df640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f292c15bf40 0x7f292c1f5310 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.764+0000 7f29346df640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f292c06baa0 0x7f292c155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.764+0000 7f29346df640 1 --2- 172.21.15.59:0/2023173659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f292c06afb0 0x7f292c1434b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.764+0000 7f29346df640 1 -- 172.21.15.59:0/2023173659 >> 172.21.15.59:0/2023173659 conn(0x7f292c0d0960 msgr2=0x7f292c0cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.764+0000 7f29346df640 1 -- 172.21.15.59:0/2023173659 shutdown_connections 2024-03-06T07:34:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.765+0000 7f29346df640 1 -- 172.21.15.59:0/2023173659 wait complete. 2024-03-06T07:34:24.777 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:34:24.778 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.876+0000 7f3a26655640 1 Processor -- start 2024-03-06T07:34:24.879 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.876+0000 7f3a26655640 1 -- start start 2024-03-06T07:34:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.877+0000 7f3a26655640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3a2015bdd0 0x7f3a200c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.877+0000 7f3a26655640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3a2015fa10 0x7f3a200ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.877+0000 7f3a26655640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a200cebe0 con 0x7f3a200d16e0 2024-03-06T07:34:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.877+0000 7f3a26655640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a200ceec0 con 0x7f3a2015bdd0 2024-03-06T07:34:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.877+0000 7f3a26655640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a200cf1a0 con 0x7f3a2015fa10 2024-03-06T07:34:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.877+0000 7f3a1ffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3a2015bdd0 0x7f3a200c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.877+0000 7f3a24bf2640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3a200d16e0 0x7f3a200d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35634/0 (socket says 172.21.15.59:35634) 2024-03-06T07:34:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.877+0000 7f3a24bf2640 1 -- 172.21.15.59:0/828379288 learned_addr learned my addr 172.21.15.59:0/828379288 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.877+0000 7f3a1ffff640 1 --2- 172.21.15.59:0/828379288 >> v2:172.21.15.59:3300/0 conn(0x7f3a2015bdd0 0x7f3a200c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50588/0 (socket says 172.21.15.59:50588) 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.878+0000 7f3a1f7fe640 1 --2- 172.21.15.59:0/828379288 >> v2:172.21.15.164:3300/0 conn(0x7f3a2015fa10 0x7f3a200ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.878+0000 7f3a1ffff640 1 -- 172.21.15.59:0/828379288 >> v2:172.21.15.164:3300/0 conn(0x7f3a2015fa10 msgr2=0x7f3a200ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.878+0000 7f3a1ffff640 1 --2- 172.21.15.59:0/828379288 >> v2:172.21.15.164:3300/0 conn(0x7f3a2015fa10 0x7f3a200ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.878+0000 7f3a1ffff640 1 -- 172.21.15.59:0/828379288 >> v1:172.21.15.59:6789/0 conn(0x7f3a200d16e0 legacy=0x7f3a200d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.878+0000 7f3a1effd640 1 -- 172.21.15.59:0/828379288 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221069172 0 0) 0x7f3a200cebe0 con 0x7f3a200d16e0 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.878+0000 7f3a1ffff640 1 -- 172.21.15.59:0/828379288 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a200cc8e0 con 0x7f3a2015bdd0 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.878+0000 7f3a1ffff640 1 --2- 172.21.15.59:0/828379288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a2015bdd0 0x7f3a200c5610 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f3a200cdfd0 tx=0x7f3a1002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c073a2a5cdab0a3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.879+0000 7f3a1effd640 1 -- 172.21.15.59:0/828379288 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a100315a0 con 0x7f3a2015bdd0 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.879+0000 7f3a1effd640 1 -- 172.21.15.59:0/828379288 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a10037070 con 0x7f3a2015bdd0 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.879+0000 7f3a1effd640 1 -- 172.21.15.59:0/828379288 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10031960 con 0x7f3a2015bdd0 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.879+0000 7f3a26655640 1 -- 172.21.15.59:0/828379288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a2015bdd0 msgr2=0x7f3a200c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.879+0000 7f3a26655640 1 --2- 172.21.15.59:0/828379288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a2015bdd0 0x7f3a200c5610 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f3a200cdfd0 tx=0x7f3a1002b840 comp rx=0 tx=0).stop 2024-03-06T07:34:24.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.879+0000 7f3a26655640 1 -- 172.21.15.59:0/828379288 shutdown_connections 2024-03-06T07:34:24.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.879+0000 7f3a26655640 1 --2- 172.21.15.59:0/828379288 >> v2:172.21.15.164:3300/0 conn(0x7f3a2015fa10 0x7f3a200ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.879+0000 7f3a26655640 1 --2- 172.21.15.59:0/828379288 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a2015bdd0 0x7f3a200c5610 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.879+0000 7f3a26655640 1 -- 172.21.15.59:0/828379288 >> 172.21.15.59:0/828379288 conn(0x7f3a2009a9f0 msgr2=0x7f3a2009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:24.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.879+0000 7f3a26655640 1 -- 172.21.15.59:0/828379288 shutdown_connections 2024-03-06T07:34:24.883 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.879+0000 7f3a26655640 1 -- 172.21.15.59:0/828379288 wait complete. 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.880+0000 7f3a26655640 1 Processor -- start 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.881+0000 7f3a26655640 1 -- start start 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.881+0000 7f3a26655640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a200d16e0 0x7f3a2014d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.881+0000 7f3a26655640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a2015bdd0 0x7f3a200b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.885 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.881+0000 7f3a26655640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a2015fa10 0x7f3a200b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.881+0000 7f3a26655640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2015c1b0 con 0x7f3a200d16e0 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.881+0000 7f3a26655640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a200cc5d0 con 0x7f3a2015fa10 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.881+0000 7f3a26655640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a200ce7b0 con 0x7f3a2015bdd0 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.881+0000 7f3a1ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a200d16e0 0x7f3a2014d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.881+0000 7f3a1ffff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a200d16e0 0x7f3a2014d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50592/0 (socket says 172.21.15.59:50592) 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.881+0000 7f3a1ffff640 1 -- 172.21.15.59:0/2738686523 learned_addr learned my addr 172.21.15.59:0/2738686523 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a24bf2640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a2015fa10 0x7f3a200b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a1f7fe640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a2015bdd0 0x7f3a200b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a1ffff640 1 -- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a2015bdd0 msgr2=0x7f3a200b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a1ffff640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a2015bdd0 0x7f3a200b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a1ffff640 1 -- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a2015fa10 msgr2=0x7f3a200b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a1ffff640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a2015fa10 0x7f3a200b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a1ffff640 1 -- 172.21.15.59:0/2738686523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2009ea80 con 0x7f3a200d16e0 2024-03-06T07:34:24.886 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a1ffff640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a200d16e0 0x7f3a2014d1a0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009890 tx=0x7f3a100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a1cff9640 1 -- 172.21.15.59:0/2738686523 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10032630 con 0x7f3a200d16e0 2024-03-06T07:34:24.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a1cff9640 1 -- 172.21.15.59:0/2738686523 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a100327d0 con 0x7f3a200d16e0 2024-03-06T07:34:24.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a1cff9640 1 -- 172.21.15.59:0/2738686523 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10032ba0 con 0x7f3a200d16e0 2024-03-06T07:34:24.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a26655640 1 -- 172.21.15.59:0/2738686523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2009ed60 con 0x7f3a200d16e0 2024-03-06T07:34:24.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.882+0000 7f3a26655640 1 -- 172.21.15.59:0/2738686523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2009f320 con 0x7f3a200d16e0 2024-03-06T07:34:24.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.883+0000 7f3a26655640 1 -- 172.21.15.59:0/2738686523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a2004f330 con 0x7f3a200d16e0 2024-03-06T07:34:24.887 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.884+0000 7f3a1cff9640 1 -- 172.21.15.59:0/2738686523 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3a1003b3f0 con 0x7f3a200d16e0 2024-03-06T07:34:24.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.884+0000 7f3a1cff9640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a0c06b500 0x7f3a0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:24.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.884+0000 7f3a1cff9640 1 -- 172.21.15.59:0/2738686523 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3a1002f030 con 0x7f3a200d16e0 2024-03-06T07:34:24.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.884+0000 7f3a1f7fe640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a0c06b500 0x7f3a0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:24.888 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.885+0000 7f3a1f7fe640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a0c06b500 0x7f3a0c06d9c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f3a20071340 tx=0x7f3a14008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:24.890 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:24.888+0000 7f3a1cff9640 1 -- 172.21.15.59:0/2738686523 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3a10040070 con 0x7f3a200d16e0 2024-03-06T07:34:25.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.109+0000 7f3a26655640 1 -- 172.21.15.59:0/2738686523 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a200b7a40 con 0x7f3a200d16e0 2024-03-06T07:34:25.111 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.109+0000 7f3a1cff9640 1 -- 172.21.15.59:0/2738686523 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f3a1007aed0 con 0x7f3a200d16e0 2024-03-06T07:34:25.112 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:25.112 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:25.112 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.111+0000 7f3a26655640 1 -- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a0c06b500 msgr2=0x7f3a0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.111+0000 7f3a26655640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a0c06b500 0x7f3a0c06d9c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f3a20071340 tx=0x7f3a14008040 comp rx=0 tx=0).stop 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.111+0000 7f3a26655640 1 -- 172.21.15.59:0/2738686523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a200d16e0 msgr2=0x7f3a2014d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.111+0000 7f3a26655640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a200d16e0 0x7f3a2014d1a0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009890 tx=0x7f3a100046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.112+0000 7f3a26655640 1 -- 172.21.15.59:0/2738686523 shutdown_connections 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.112+0000 7f3a26655640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f3a0c06b500 0x7f3a0c06d9c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.112+0000 7f3a26655640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a2015fa10 0x7f3a200b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.112+0000 7f3a26655640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a2015bdd0 0x7f3a200b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.112+0000 7f3a26655640 1 --2- 172.21.15.59:0/2738686523 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a200d16e0 0x7f3a2014d1a0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.112+0000 7f3a26655640 1 -- 172.21.15.59:0/2738686523 >> 172.21.15.59:0/2738686523 conn(0x7f3a2009a9f0 msgr2=0x7f3a200d1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.112+0000 7f3a26655640 1 -- 172.21.15.59:0/2738686523 shutdown_connections 2024-03-06T07:34:25.114 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.112+0000 7f3a26655640 1 -- 172.21.15.59:0/2738686523 wait complete. 2024-03-06T07:34:25.125 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:25.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.224+0000 7f04cd0f4640 1 Processor -- start 2024-03-06T07:34:25.226 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.224+0000 7f04cd0f4640 1 -- start start 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.225+0000 7f04cd0f4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f04c815dfe0 0x7f04c80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.225+0000 7f04cd0f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c80cf830 0x7f04c80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.225+0000 7f04cd0f4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c80b3430 con 0x7f04c80d27f0 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.225+0000 7f04cd0f4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c80d2c50 con 0x7f04c815dfe0 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.225+0000 7f04cd0f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80b31e0 con 0x7f04c80cf830 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.225+0000 7f04c759e640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f04c80d27f0 0x7f04c80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35650/0 (socket says 172.21.15.59:35650) 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.225+0000 7f04c759e640 1 -- 172.21.15.59:0/3712292137 learned_addr learned my addr 172.21.15.59:0/3712292137 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.225+0000 7f04c6d9d640 1 --2- 172.21.15.59:0/3712292137 >> v2:172.21.15.164:3300/0 conn(0x7f04c815dfe0 0x7f04c80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.225+0000 7f04c659c640 1 --2- 172.21.15.59:0/3712292137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c80cf830 0x7f04c80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.225+0000 7f04c5d9b640 1 -- 172.21.15.59:0/3712292137 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906718620 0 0) 0x7f04c80b3430 con 0x7f04c80d27f0 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.226+0000 7f04c5d9b640 1 -- 172.21.15.59:0/3712292137 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b0003520 con 0x7f04c80d27f0 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.226+0000 7f04c5d9b640 1 -- 172.21.15.59:0/3712292137 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4202325817 0 0) 0x7f04b0003520 con 0x7f04c80d27f0 2024-03-06T07:34:25.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.226+0000 7f04c6d9d640 1 -- 172.21.15.59:0/3712292137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c80cf830 msgr2=0x7f04c80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.226+0000 7f04c6d9d640 1 --2- 172.21.15.59:0/3712292137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c80cf830 0x7f04c80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.226+0000 7f04c6d9d640 1 -- 172.21.15.59:0/3712292137 >> v1:172.21.15.59:6789/0 conn(0x7f04c80d27f0 legacy=0x7f04c80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.226+0000 7f04c6d9d640 1 -- 172.21.15.59:0/3712292137 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c80b3710 con 0x7f04c815dfe0 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04c6d9d640 1 --2- 172.21.15.59:0/3712292137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c815dfe0 0x7f04c80cf2f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f04b80097a0 tx=0x7f04b802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=691d309dd2525a17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04c5d9b640 1 -- 172.21.15.59:0/3712292137 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b8037070 con 0x7f04c815dfe0 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04c5d9b640 1 -- 172.21.15.59:0/3712292137 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b8030510 con 0x7f04c815dfe0 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04c5d9b640 1 -- 172.21.15.59:0/3712292137 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c00019f0 con 0x7f04c815dfe0 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04cd0f4640 1 -- 172.21.15.59:0/3712292137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c815dfe0 msgr2=0x7f04c80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04cd0f4640 1 --2- 172.21.15.59:0/3712292137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c815dfe0 0x7f04c80cf2f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f04b80097a0 tx=0x7f04b802b790 comp rx=0 tx=0).stop 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04cd0f4640 1 -- 172.21.15.59:0/3712292137 shutdown_connections 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04cd0f4640 1 --2- 172.21.15.59:0/3712292137 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c80cf830 0x7f04c80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04cd0f4640 1 --2- 172.21.15.59:0/3712292137 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c815dfe0 0x7f04c80cf2f0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04cd0f4640 1 -- 172.21.15.59:0/3712292137 >> 172.21.15.59:0/3712292137 conn(0x7f04c80d3cd0 msgr2=0x7f04c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04cd0f4640 1 -- 172.21.15.59:0/3712292137 shutdown_connections 2024-03-06T07:34:25.230 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.227+0000 7f04cd0f4640 1 -- 172.21.15.59:0/3712292137 wait complete. 2024-03-06T07:34:25.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.229+0000 7f04cd0f4640 1 Processor -- start 2024-03-06T07:34:25.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.229+0000 7f04cd0f4640 1 -- start start 2024-03-06T07:34:25.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.229+0000 7f04cd0f4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c80cf830 0x7f04c80cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.229+0000 7f04cd0f4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c80d27f0 0x7f04c80b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.229+0000 7f04cd0f4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c815dfe0 0x7f04c81f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.229+0000 7f04cd0f4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80d1c70 con 0x7f04c80d27f0 2024-03-06T07:34:25.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.229+0000 7f04cd0f4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80b3d60 con 0x7f04c80cf830 2024-03-06T07:34:25.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.229+0000 7f04cd0f4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80d1df0 con 0x7f04c815dfe0 2024-03-06T07:34:25.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04c659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c80d27f0 0x7f04c80b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04c659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c80d27f0 0x7f04c80b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50606/0 (socket says 172.21.15.59:50606) 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04c659c640 1 -- 172.21.15.59:0/4135825012 learned_addr learned my addr 172.21.15.59:0/4135825012 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04c759e640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c815dfe0 0x7f04c81f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04c6d9d640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c80cf830 0x7f04c80cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04c659c640 1 -- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c815dfe0 msgr2=0x7f04c81f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04c659c640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c815dfe0 0x7f04c81f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04c659c640 1 -- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c80cf830 msgr2=0x7f04c80cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04c659c640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c80cf830 0x7f04c80cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04c659c640 1 -- 172.21.15.59:0/4135825012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c80b9160 con 0x7f04c80d27f0 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04c659c640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c80d27f0 0x7f04c80b68a0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00e980 tx=0x7f04bc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04af7fe640 1 -- 172.21.15.59:0/4135825012 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc017070 con 0x7f04c80d27f0 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04af7fe640 1 -- 172.21.15.59:0/4135825012 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04bc00ca00 con 0x7f04c80d27f0 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04cd0f4640 1 -- 172.21.15.59:0/4135825012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c80b9440 con 0x7f04c80d27f0 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04af7fe640 1 -- 172.21.15.59:0/4135825012 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc00cdd0 con 0x7f04c80d27f0 2024-03-06T07:34:25.234 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.230+0000 7f04cd0f4640 1 -- 172.21.15.59:0/4135825012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c80b9a00 con 0x7f04c80d27f0 2024-03-06T07:34:25.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.231+0000 7f04cd0f4640 1 -- 172.21.15.59:0/4135825012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c80ce3c0 con 0x7f04c80d27f0 2024-03-06T07:34:25.235 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.232+0000 7f04af7fe640 1 -- 172.21.15.59:0/4135825012 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04bc01a070 con 0x7f04c80d27f0 2024-03-06T07:34:25.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.232+0000 7f04af7fe640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04b406b4d0 0x7f04b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.233+0000 7f04af7fe640 1 -- 172.21.15.59:0/4135825012 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f04bc08a0e0 con 0x7f04c80d27f0 2024-03-06T07:34:25.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.233+0000 7f04c6d9d640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04b406b4d0 0x7f04b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.236 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.233+0000 7f04c6d9d640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04b406b4d0 0x7f04b406d990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f04b8034000 tx=0x7f04b802f000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.238 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.237+0000 7f04af7fe640 1 -- 172.21.15.59:0/4135825012 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04c80ce3c0 con 0x7f04c80d27f0 2024-03-06T07:34:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.456+0000 7f04cd0f4640 1 -- 172.21.15.59:0/4135825012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04c804f330 con 0x7f04c80d27f0 2024-03-06T07:34:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.456+0000 7f04af7fe640 1 -- 172.21.15.59:0/4135825012 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3464 (secure 0 0 0) 0x7f04bc058980 con 0x7f04c80d27f0 2024-03-06T07:34:25.458 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:25.459 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":108,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:13.446245+0000","modified":"2024-03-06T07:32:44.076826+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":9336,"mds_1":8236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8236":{"gid":8236,"name":"b","rank":1,"incarnation":99,"state":"up:stopping","state_seq":5,"addr":"172.21.15.164:6837/997138956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6835","nonce":997138956},{"type":"v1","addr":"172.21.15.164:6837","nonce":997138956}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:31:40.104271+0000","export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9336":{"gid":9336,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.59:6833/720863754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":720863754},{"type":"v1","addr":"172.21.15.59:6833","nonce":720863754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9336,"qdb_cluster":[9336]},"id":10}]} 2024-03-06T07:34:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 108 2024-03-06T07:34:25.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 -- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04b406b4d0 msgr2=0x7f04b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04b406b4d0 0x7f04b406d990 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f04b8034000 tx=0x7f04b802f000 comp rx=0 tx=0).stop 2024-03-06T07:34:25.460 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 -- 172.21.15.59:0/4135825012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c80d27f0 msgr2=0x7f04c80b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c80d27f0 0x7f04c80b68a0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00e980 tx=0x7f04bc00ee40 comp rx=0 tx=0).stop 2024-03-06T07:34:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 -- 172.21.15.59:0/4135825012 shutdown_connections 2024-03-06T07:34:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f04b406b4d0 0x7f04b406d990 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04c815dfe0 0x7f04c81f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f04c80d27f0 0x7f04c80b68a0 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 --2- 172.21.15.59:0/4135825012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04c80cf830 0x7f04c80cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 -- 172.21.15.59:0/4135825012 >> 172.21.15.59:0/4135825012 conn(0x7f04c80d3cd0 msgr2=0x7f04c80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 -- 172.21.15.59:0/4135825012 shutdown_connections 2024-03-06T07:34:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.458+0000 7f04cd0f4640 1 -- 172.21.15.59:0/4135825012 wait complete. 2024-03-06T07:34:25.471 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.574+0000 7f60c179a640 1 Processor -- start 2024-03-06T07:34:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.574+0000 7f60c179a640 1 -- start start 2024-03-06T07:34:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.575+0000 7f60c179a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60bc0afdd0 0x7f60bc15c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.575+0000 7f60c179a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60bc0d14f0 0x7f60bc06f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.575+0000 7f60c179a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60bc15e740 con 0x7f60bc158920 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.575+0000 7f60c179a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60bc06fae0 con 0x7f60bc0afdd0 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.575+0000 7f60c179a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc15f5f0 con 0x7f60bc0d14f0 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.575+0000 7f60b27fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60bc0afdd0 0x7f60bc15c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.575+0000 7f60b27fc640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60bc0afdd0 0x7f60bc15c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50610/0 (socket says 172.21.15.59:50610) 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.575+0000 7f60b27fc640 1 -- 172.21.15.59:0/47719401 learned_addr learned my addr 172.21.15.59:0/47719401 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.575+0000 7f60baffd640 1 --1- 172.21.15.59:0/47719401 >> v1:172.21.15.164:6789/0 conn(0x7f60bc158920 0x7f60bc0af6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.59:37670/0 (socket says 172.21.15.59:37670) 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.575+0000 7f60bb7fe640 1 --2- 172.21.15.59:0/47719401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60bc0d14f0 0x7f60bc06f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.576+0000 7f60b27fc640 1 -- 172.21.15.59:0/47719401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60bc0d14f0 msgr2=0x7f60bc06f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.576+0000 7f60b27fc640 1 --2- 172.21.15.59:0/47719401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60bc0d14f0 0x7f60bc06f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.576+0000 7f60b27fc640 1 -- 172.21.15.59:0/47719401 >> v1:172.21.15.164:6789/0 conn(0x7f60bc158920 legacy=0x7f60bc0af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.578 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.576+0000 7f60b27fc640 1 -- 172.21.15.59:0/47719401 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc15e9e0 con 0x7f60bc0afdd0 2024-03-06T07:34:25.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.576+0000 7f60b27fc640 1 --2- 172.21.15.59:0/47719401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60bc0afdd0 0x7f60bc15c310 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009870 tx=0x7f60a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c679d8b3bc69bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.576+0000 7f60ba7fc640 1 -- 172.21.15.59:0/47719401 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4034070 con 0x7f60bc0afdd0 2024-03-06T07:34:25.579 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.576+0000 7f60ba7fc640 1 -- 172.21.15.59:0/47719401 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a4030840 con 0x7f60bc0afdd0 2024-03-06T07:34:25.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.576+0000 7f60ba7fc640 1 -- 172.21.15.59:0/47719401 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4030c10 con 0x7f60bc0afdd0 2024-03-06T07:34:25.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.577+0000 7f60c179a640 1 -- 172.21.15.59:0/47719401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60bc0afdd0 msgr2=0x7f60bc15c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.577+0000 7f60c179a640 1 --2- 172.21.15.59:0/47719401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60bc0afdd0 0x7f60bc15c310 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009870 tx=0x7f60a402b980 comp rx=0 tx=0).stop 2024-03-06T07:34:25.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.577+0000 7f60c179a640 1 -- 172.21.15.59:0/47719401 shutdown_connections 2024-03-06T07:34:25.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.577+0000 7f60c179a640 1 --2- 172.21.15.59:0/47719401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60bc0d14f0 0x7f60bc06f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.577+0000 7f60c179a640 1 --2- 172.21.15.59:0/47719401 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60bc0afdd0 0x7f60bc15c310 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.577+0000 7f60c179a640 1 -- 172.21.15.59:0/47719401 >> 172.21.15.59:0/47719401 conn(0x7f60bc0d0960 msgr2=0x7f60bc0d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.577+0000 7f60c179a640 1 -- 172.21.15.59:0/47719401 shutdown_connections 2024-03-06T07:34:25.580 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.577+0000 7f60c179a640 1 -- 172.21.15.59:0/47719401 wait complete. 2024-03-06T07:34:25.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.578+0000 7f60c179a640 1 Processor -- start 2024-03-06T07:34:25.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.578+0000 7f60c179a640 1 -- start start 2024-03-06T07:34:25.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60c179a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60bc0afdd0 0x7f60bc1598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60c179a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60bc0d14f0 0x7f60bc14a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60c179a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60bc158920 0x7f60bc159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60c179a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc15f280 con 0x7f60bc158920 2024-03-06T07:34:25.582 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60c179a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc15efa0 con 0x7f60bc0d14f0 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60c179a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc15b400 con 0x7f60bc0afdd0 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60bc158920 0x7f60bc159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60bb7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60bc158920 0x7f60bc159f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50624/0 (socket says 172.21.15.59:50624) 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60bb7fe640 1 -- 172.21.15.59:0/3986477720 learned_addr learned my addr 172.21.15.59:0/3986477720 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60baffd640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60bc0afdd0 0x7f60bc1598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60b27fc640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60bc0d14f0 0x7f60bc14a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60bb7fe640 1 -- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60bc0afdd0 msgr2=0x7f60bc1598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.579+0000 7f60bb7fe640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60bc0afdd0 0x7f60bc1598f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.580+0000 7f60bb7fe640 1 -- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60bc0d14f0 msgr2=0x7f60bc14a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.580+0000 7f60bb7fe640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60bc0d14f0 0x7f60bc14a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.580+0000 7f60bb7fe640 1 -- 172.21.15.59:0/3986477720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc15a930 con 0x7f60bc158920 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.580+0000 7f60baffd640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60bc0afdd0 0x7f60bc1598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.580+0000 7f60b27fc640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60bc0d14f0 0x7f60bc14a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.580+0000 7f60bb7fe640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60bc158920 0x7f60bc159f50 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f60b400efc0 tx=0x7f60b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.580+0000 7f60b3fff640 1 -- 172.21.15.59:0/3986477720 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4018020 con 0x7f60bc158920 2024-03-06T07:34:25.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.580+0000 7f60b3fff640 1 -- 172.21.15.59:0/3986477720 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b4014480 con 0x7f60bc158920 2024-03-06T07:34:25.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.580+0000 7f60c179a640 1 -- 172.21.15.59:0/3986477720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60bc15ac10 con 0x7f60bc158920 2024-03-06T07:34:25.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.580+0000 7f60b3fff640 1 -- 172.21.15.59:0/3986477720 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4014870 con 0x7f60bc158920 2024-03-06T07:34:25.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.580+0000 7f60c179a640 1 -- 172.21.15.59:0/3986477720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc1590f0 con 0x7f60bc158920 2024-03-06T07:34:25.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.581+0000 7f60c179a640 1 -- 172.21.15.59:0/3986477720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60bc04f330 con 0x7f60bc158920 2024-03-06T07:34:25.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.581+0000 7f60b3fff640 1 -- 172.21.15.59:0/3986477720 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60b4005d10 con 0x7f60bc158920 2024-03-06T07:34:25.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.582+0000 7f60b3fff640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f609c06b500 0x7f609c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.582+0000 7f60b3fff640 1 -- 172.21.15.59:0/3986477720 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f60b408b0a0 con 0x7f60bc158920 2024-03-06T07:34:25.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.582+0000 7f60baffd640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f609c06b500 0x7f609c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.582+0000 7f60baffd640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f609c06b500 0x7f609c06d9c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0d02b0 tx=0x7f60ac009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.588 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.586+0000 7f60b3fff640 1 -- 172.21.15.59:0/3986477720 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60b4022030 con 0x7f60bc158920 2024-03-06T07:34:25.848 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.846+0000 7f60c179a640 1 -- 172.21.15.59:0/3986477720 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60bc0cd580 con 0x7f60bc158920 2024-03-06T07:34:25.849 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.847+0000 7f60b3fff640 1 -- 172.21.15.59:0/3986477720 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+16717 (secure 0 0 0) 0x7f60b4010040 con 0x7f60bc158920 2024-03-06T07:34:25.849 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:25.849 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":143,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:30:39.494279+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:11.712614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:12.406781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.22","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 -- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f609c06b500 msgr2=0x7f609c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f609c06b500 0x7f609c06d9c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0d02b0 tx=0x7f60ac009040 comp rx=0 tx=0).stop 2024-03-06T07:34:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 -- 172.21.15.59:0/3986477720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60bc158920 msgr2=0x7f60bc159f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60bc158920 0x7f60bc159f50 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f60b400efc0 tx=0x7f60b400c490 comp rx=0 tx=0).stop 2024-03-06T07:34:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 -- 172.21.15.59:0/3986477720 shutdown_connections 2024-03-06T07:34:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f609c06b500 0x7f609c06d9c0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60bc158920 0x7f60bc159f50 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60bc0d14f0 0x7f60bc14a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 --2- 172.21.15.59:0/3986477720 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60bc0afdd0 0x7f60bc1598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 -- 172.21.15.59:0/3986477720 >> 172.21.15.59:0/3986477720 conn(0x7f60bc0d0960 msgr2=0x7f60bc0b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 -- 172.21.15.59:0/3986477720 shutdown_connections 2024-03-06T07:34:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.849+0000 7f60c179a640 1 -- 172.21.15.59:0/3986477720 wait complete. 2024-03-06T07:34:25.862 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:34:25.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.962+0000 7f9a2855c640 1 Processor -- start 2024-03-06T07:34:25.964 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.962+0000 7f9a2855c640 1 -- start start 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.963+0000 7f9a2855c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9a20162970 0x7f9a200d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.963+0000 7f9a2855c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9a20161cf0 0x7f9a20162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.963+0000 7f9a2855c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a200d22c0 con 0x7f9a200b4460 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.963+0000 7f9a2855c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a20161b00 con 0x7f9a20162970 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.963+0000 7f9a2855c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a200cecc0 con 0x7f9a20161cf0 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.963+0000 7f9a25af7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9a20162970 0x7f9a200d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.963+0000 7f9a262f8640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9a200b4460 0x7f9a200b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35656/0 (socket says 172.21.15.59:35656) 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.963+0000 7f9a25af7640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9a20162970 0x7f9a200d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50630/0 (socket says 172.21.15.59:50630) 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.963+0000 7f9a262f8640 1 -- 172.21.15.59:0/64119024 learned_addr learned my addr 172.21.15.59:0/64119024 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.964+0000 7f9a26af9640 1 --2- 172.21.15.59:0/64119024 >> v2:172.21.15.164:3300/0 conn(0x7f9a20161cf0 0x7f9a20162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.964+0000 7f9a25af7640 1 -- 172.21.15.59:0/64119024 >> v2:172.21.15.164:3300/0 conn(0x7f9a20161cf0 msgr2=0x7f9a20162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.964+0000 7f9a25af7640 1 --2- 172.21.15.59:0/64119024 >> v2:172.21.15.164:3300/0 conn(0x7f9a20161cf0 0x7f9a20162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.964+0000 7f9a25af7640 1 -- 172.21.15.59:0/64119024 >> v1:172.21.15.59:6789/0 conn(0x7f9a200b4460 legacy=0x7f9a200b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.964+0000 7f9a25af7640 1 -- 172.21.15.59:0/64119024 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a200d25a0 con 0x7f9a20162970 2024-03-06T07:34:25.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.964+0000 7f9a252f6640 1 -- 172.21.15.59:0/64119024 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474894090 0 0) 0x7f9a200d22c0 con 0x7f9a200b4460 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a25af7640 1 --2- 172.21.15.59:0/64119024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a20162970 0x7f9a200d4800 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f9a2008c900 tx=0x7f9a1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f9a18697d1e32a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a252f6640 1 -- 172.21.15.59:0/64119024 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10035020 con 0x7f9a20162970 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a252f6640 1 -- 172.21.15.59:0/64119024 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10031830 con 0x7f9a20162970 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a252f6640 1 -- 172.21.15.59:0/64119024 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10031c00 con 0x7f9a20162970 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a2855c640 1 -- 172.21.15.59:0/64119024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a20162970 msgr2=0x7f9a200d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a2855c640 1 --2- 172.21.15.59:0/64119024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a20162970 0x7f9a200d4800 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f9a2008c900 tx=0x7f9a1002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a2855c640 1 -- 172.21.15.59:0/64119024 shutdown_connections 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a2855c640 1 --2- 172.21.15.59:0/64119024 >> v2:172.21.15.164:3300/0 conn(0x7f9a20161cf0 0x7f9a20162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a2855c640 1 --2- 172.21.15.59:0/64119024 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a20162970 0x7f9a200d4800 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a2855c640 1 -- 172.21.15.59:0/64119024 >> 172.21.15.59:0/64119024 conn(0x7f9a200d3ce0 msgr2=0x7f9a200d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a2855c640 1 -- 172.21.15.59:0/64119024 shutdown_connections 2024-03-06T07:34:25.968 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.965+0000 7f9a2855c640 1 -- 172.21.15.59:0/64119024 wait complete. 2024-03-06T07:34:25.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.966+0000 7f9a2855c640 1 Processor -- start 2024-03-06T07:34:25.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.966+0000 7f9a2855c640 1 -- start start 2024-03-06T07:34:25.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.967+0000 7f9a2855c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a200b4460 0x7f9a2014f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.967+0000 7f9a2855c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a20161cf0 0x7f9a200b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.967+0000 7f9a2855c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a20162970 0x7f9a200cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.967+0000 7f9a2855c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a200ce950 con 0x7f9a20162970 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.967+0000 7f9a2855c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a200d2b60 con 0x7f9a200b4460 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.967+0000 7f9a2855c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a200b4840 con 0x7f9a20161cf0 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.967+0000 7f9a26af9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a20162970 0x7f9a200cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.967+0000 7f9a26af9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a20162970 0x7f9a200cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50640/0 (socket says 172.21.15.59:50640) 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.967+0000 7f9a26af9640 1 -- 172.21.15.59:0/4196982348 learned_addr learned my addr 172.21.15.59:0/4196982348 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.967+0000 7f9a262f8640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a200b4460 0x7f9a2014f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.967+0000 7f9a25af7640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a20161cf0 0x7f9a200b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a26af9640 1 -- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a20161cf0 msgr2=0x7f9a200b6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a26af9640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a20161cf0 0x7f9a200b6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a26af9640 1 -- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a200b4460 msgr2=0x7f9a2014f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a26af9640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a200b4460 0x7f9a2014f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:25.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a26af9640 1 -- 172.21.15.59:0/4196982348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a200cb520 con 0x7f9a20162970 2024-03-06T07:34:25.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a25af7640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a20161cf0 0x7f9a200b6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:25.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a26af9640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a20162970 0x7f9a200cadb0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f9a18007e70 tx=0x7f9a1800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a16ffd640 1 -- 172.21.15.59:0/4196982348 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18014620 con 0x7f9a20162970 2024-03-06T07:34:25.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a16ffd640 1 -- 172.21.15.59:0/4196982348 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a18019070 con 0x7f9a20162970 2024-03-06T07:34:25.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a2855c640 1 -- 172.21.15.59:0/4196982348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a200cb800 con 0x7f9a20162970 2024-03-06T07:34:25.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a16ffd640 1 -- 172.21.15.59:0/4196982348 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18014a20 con 0x7f9a20162970 2024-03-06T07:34:25.972 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.968+0000 7f9a2855c640 1 -- 172.21.15.59:0/4196982348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a200d1890 con 0x7f9a20162970 2024-03-06T07:34:25.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.969+0000 7f9a2855c640 1 -- 172.21.15.59:0/4196982348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2004f330 con 0x7f9a20162970 2024-03-06T07:34:25.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.970+0000 7f9a16ffd640 1 -- 172.21.15.59:0/4196982348 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a18014bc0 con 0x7f9a20162970 2024-03-06T07:34:25.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.970+0000 7f9a16ffd640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a0406b4d0 0x7f9a0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:25.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.970+0000 7f9a16ffd640 1 -- 172.21.15.59:0/4196982348 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f9a1808aa40 con 0x7f9a20162970 2024-03-06T07:34:25.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.970+0000 7f9a262f8640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a0406b4d0 0x7f9a0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:25.973 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.971+0000 7f9a262f8640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a0406b4d0 0x7f9a0406d990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c007fd0 tx=0x7f9a1c007de0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:25.976 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:25.974+0000 7f9a16ffd640 1 -- 172.21.15.59:0/4196982348 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a18059930 con 0x7f9a20162970 2024-03-06T07:34:26.196 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.194+0000 7f9a2855c640 1 -- 172.21.15.59:0/4196982348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9a2006b180 con 0x7f9a20162970 2024-03-06T07:34:26.196 INFO:tasks.ceph.mon.a.smithi059.stderr:2024-03-06T07:34:26.194+0000 7fed64575640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:34:26.197 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.196+0000 7f9a16ffd640 1 -- 172.21.15.59:0/4196982348 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v109) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9a1808ae20 con 0x7f9a20162970 2024-03-06T07:34:26.198 INFO:teuthology.orchestra.run.smithi059.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:34:26.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 -- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a0406b4d0 msgr2=0x7f9a0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a0406b4d0 0x7f9a0406d990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c007fd0 tx=0x7f9a1c007de0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 -- 172.21.15.59:0/4196982348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a20162970 msgr2=0x7f9a200cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a20162970 0x7f9a200cadb0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f9a18007e70 tx=0x7f9a1800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 -- 172.21.15.59:0/4196982348 shutdown_connections 2024-03-06T07:34:26.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a0406b4d0 0x7f9a0406d990 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a20162970 0x7f9a200cadb0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.201 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a20161cf0 0x7f9a200b6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 --2- 172.21.15.59:0/4196982348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a200b4460 0x7f9a2014f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 -- 172.21.15.59:0/4196982348 >> 172.21.15.59:0/4196982348 conn(0x7f9a200d3ce0 msgr2=0x7f9a201610d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 -- 172.21.15.59:0/4196982348 shutdown_connections 2024-03-06T07:34:26.202 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.199+0000 7f9a2855c640 1 -- 172.21.15.59:0/4196982348 wait complete. 2024-03-06T07:34:26.213 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:34:26.236 INFO:tasks.ceph.mds.c.smithi059.stdout:starting mds.c at 2024-03-06T07:34:26.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.315+0000 7fca3357a640 1 Processor -- start 2024-03-06T07:34:26.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.315+0000 7fca3357a640 1 -- start start 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.316+0000 7fca3357a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca2c1614d0 0x7fca2c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.316+0000 7fca3357a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fca2c0d1910 0x7fca2c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.316+0000 7fca3357a640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fca2c0d22b0 0x7fca2c0d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.316+0000 7fca3357a640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2c0d2740 con 0x7fca2c0d22b0 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.316+0000 7fca3357a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2c1618b0 con 0x7fca2c0d1910 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.316+0000 7fca3357a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0cf400 con 0x7fca2c1614d0 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.316+0000 7fca31b17640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fca2c0d22b0 0x7fca2c0d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.316+0000 7fca31b17640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fca2c0d22b0 0x7fca2c0d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50654/0 (socket says 172.21.15.59:50654) 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.316+0000 7fca31b17640 1 -- 172.21.15.59:0/4113831858 learned_addr learned my addr 172.21.15.59:0/4113831858 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.317+0000 7fca31316640 1 --2- 172.21.15.59:0/4113831858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca2c1614d0 0x7fca2c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.317+0000 7fca31b17640 1 -- 172.21.15.59:0/4113831858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca2c1614d0 msgr2=0x7fca2c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.319 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.317+0000 7fca31b17640 1 --2- 172.21.15.59:0/4113831858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca2c1614d0 0x7fca2c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.317+0000 7fca31b17640 1 -- 172.21.15.59:0/4113831858 >> v2:172.21.15.164:3300/0 conn(0x7fca2c0d1910 msgr2=0x7fca2c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:26.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.317+0000 7fca30b15640 1 --2- 172.21.15.59:0/4113831858 >> v2:172.21.15.164:3300/0 conn(0x7fca2c0d1910 0x7fca2c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.317+0000 7fca31b17640 1 --2- 172.21.15.59:0/4113831858 >> v2:172.21.15.164:3300/0 conn(0x7fca2c0d1910 0x7fca2c0d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.317+0000 7fca31b17640 1 -- 172.21.15.59:0/4113831858 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c0cf6a0 con 0x7fca2c0d22b0 2024-03-06T07:34:26.320 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.317+0000 7fca31316640 1 --2- 172.21.15.59:0/4113831858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca2c1614d0 0x7fca2c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:26.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.317+0000 7fca31b17640 1 --2- 172.21.15.59:0/4113831858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca2c0d22b0 0x7fca2c0d06a0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fca2c08c4f0 tx=0x7fca2402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5eb5cfc86dd635 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca1ffff640 1 -- 172.21.15.59:0/4113831858 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2402ca70 con 0x7fca2c0d22b0 2024-03-06T07:34:26.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca1ffff640 1 -- 172.21.15.59:0/4113831858 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2402cc10 con 0x7fca2c0d22b0 2024-03-06T07:34:26.321 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca1ffff640 1 -- 172.21.15.59:0/4113831858 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca240343f0 con 0x7fca2c0d22b0 2024-03-06T07:34:26.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca3357a640 1 -- 172.21.15.59:0/4113831858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca2c0d22b0 msgr2=0x7fca2c0d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca3357a640 1 --2- 172.21.15.59:0/4113831858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca2c0d22b0 0x7fca2c0d06a0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fca2c08c4f0 tx=0x7fca2402bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca3357a640 1 -- 172.21.15.59:0/4113831858 shutdown_connections 2024-03-06T07:34:26.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca3357a640 1 --2- 172.21.15.59:0/4113831858 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca2c0d22b0 0x7fca2c0d06a0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca3357a640 1 --2- 172.21.15.59:0/4113831858 >> v2:172.21.15.164:3300/0 conn(0x7fca2c0d1910 0x7fca2c0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca3357a640 1 --2- 172.21.15.59:0/4113831858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca2c1614d0 0x7fca2c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca3357a640 1 -- 172.21.15.59:0/4113831858 >> 172.21.15.59:0/4113831858 conn(0x7fca2c0d3d40 msgr2=0x7fca2c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:26.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca3357a640 1 -- 172.21.15.59:0/4113831858 shutdown_connections 2024-03-06T07:34:26.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.318+0000 7fca3357a640 1 -- 172.21.15.59:0/4113831858 wait complete. 2024-03-06T07:34:26.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.319+0000 7fca3357a640 1 Processor -- start 2024-03-06T07:34:26.322 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.319+0000 7fca3357a640 1 -- start start 2024-03-06T07:34:26.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca3357a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca2c0d1910 0x7fca2c1f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca3357a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca2c0d22b0 0x7fca2c0b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.324 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca3357a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca2c1614d0 0x7fca2c0b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca3357a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0b0910 con 0x7fca2c1614d0 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca3357a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0cfc60 con 0x7fca2c0d22b0 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca3357a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0d0be0 con 0x7fca2c0d1910 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca31b17640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca2c1614d0 0x7fca2c0b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca31b17640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca2c1614d0 0x7fca2c0b11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50662/0 (socket says 172.21.15.59:50662) 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca31b17640 1 -- 172.21.15.59:0/972849425 learned_addr learned my addr 172.21.15.59:0/972849425 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca31316640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca2c0d1910 0x7fca2c1f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca31b17640 1 -- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca2c0d1910 msgr2=0x7fca2c1f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca30b15640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca2c0d22b0 0x7fca2c0b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca31b17640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca2c0d1910 0x7fca2c1f0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca31b17640 1 -- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca2c0d22b0 msgr2=0x7fca2c0b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca31b17640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca2c0d22b0 0x7fca2c0b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.320+0000 7fca31b17640 1 -- 172.21.15.59:0/972849425 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c1f13f0 con 0x7fca2c1614d0 2024-03-06T07:34:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.321+0000 7fca31b17640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca2c1614d0 0x7fca2c0b11c0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fca24036040 tx=0x7fca24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.321+0000 7fca1dffb640 1 -- 172.21.15.59:0/972849425 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24043020 con 0x7fca2c1614d0 2024-03-06T07:34:26.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.321+0000 7fca1dffb640 1 -- 172.21.15.59:0/972849425 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2403e450 con 0x7fca2c1614d0 2024-03-06T07:34:26.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.321+0000 7fca3357a640 1 -- 172.21.15.59:0/972849425 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c0b99e0 con 0x7fca2c1614d0 2024-03-06T07:34:26.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.321+0000 7fca1dffb640 1 -- 172.21.15.59:0/972849425 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24009510 con 0x7fca2c1614d0 2024-03-06T07:34:26.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.321+0000 7fca3357a640 1 -- 172.21.15.59:0/972849425 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c1f3e90 con 0x7fca2c1614d0 2024-03-06T07:34:26.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.322+0000 7fca3357a640 1 -- 172.21.15.59:0/972849425 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c068ff0 con 0x7fca2c1614d0 2024-03-06T07:34:26.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.322+0000 7fca1dffb640 1 -- 172.21.15.59:0/972849425 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca24046020 con 0x7fca2c1614d0 2024-03-06T07:34:26.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.323+0000 7fca1dffb640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:26.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.323+0000 7fca1dffb640 1 -- 172.21.15.59:0/972849425 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fca240acbf0 con 0x7fca2c1614d0 2024-03-06T07:34:26.326 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.323+0000 7fca31316640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:26.327 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.323+0000 7fca31316640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca1006b4d0 0x7fca1006d990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fca2c15ea10 tx=0x7fca28008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:26.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.327+0000 7fca1dffb640 1 -- 172.21.15.59:0/972849425 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca2404d030 con 0x7fca2c1614d0 2024-03-06T07:34:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:26.550+0000 7fca3357a640 1 -- 172.21.15.59:0/972849425 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fca2c1f3970 con 0x7fca2c1614d0 2024-03-06T07:34:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.197+0000 7fca1dffb640 1 -- 172.21.15.59:0/972849425 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v110) v1 ==== 104+0+0 (secure 0 0 0) 0x7fca2407b9a0 con 0x7fca2c1614d0 2024-03-06T07:34:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.200+0000 7fca3357a640 1 -- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca1006b4d0 msgr2=0x7fca1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.200+0000 7fca3357a640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca1006b4d0 0x7fca1006d990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fca2c15ea10 tx=0x7fca28008040 comp rx=0 tx=0).stop 2024-03-06T07:34:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.200+0000 7fca3357a640 1 -- 172.21.15.59:0/972849425 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca2c1614d0 msgr2=0x7fca2c0b11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.200+0000 7fca3357a640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca2c1614d0 0x7fca2c0b11c0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fca24036040 tx=0x7fca24004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.200+0000 7fca3357a640 1 -- 172.21.15.59:0/972849425 shutdown_connections 2024-03-06T07:34:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.200+0000 7fca3357a640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.200+0000 7fca3357a640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca2c1614d0 0x7fca2c0b11c0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.200+0000 7fca3357a640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca2c0d22b0 0x7fca2c0b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.200+0000 7fca3357a640 1 --2- 172.21.15.59:0/972849425 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca2c0d1910 0x7fca2c1f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.203 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.200+0000 7fca3357a640 1 -- 172.21.15.59:0/972849425 >> 172.21.15.59:0/972849425 conn(0x7fca2c0d3d40 msgr2=0x7fca2c0d4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.201+0000 7fca3357a640 1 -- 172.21.15.59:0/972849425 shutdown_connections 2024-03-06T07:34:27.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.201+0000 7fca3357a640 1 -- 172.21.15.59:0/972849425 wait complete. 2024-03-06T07:34:27.216 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:34:27.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.368+0000 7f52c5dbf640 1 Processor -- start 2024-03-06T07:34:27.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.368+0000 7f52c5dbf640 1 -- start start 2024-03-06T07:34:27.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52c5dbf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f52c0162470 0x7f52c006f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52c5dbf640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f52c00d2160 0x7f52c006c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52c5dbf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c00cf3c0 con 0x7f52c0162850 2024-03-06T07:34:27.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52c5dbf640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c006f270 con 0x7f52c00d2160 2024-03-06T07:34:27.371 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52c5dbf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c00cf0e0 con 0x7f52c0162470 2024-03-06T07:34:27.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52bffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f52c00d2160 0x7f52c006c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52bffff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f52c00d2160 0x7f52c006c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50670/0 (socket says 172.21.15.59:50670) 2024-03-06T07:34:27.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52bffff640 1 -- 172.21.15.59:0/4099371650 learned_addr learned my addr 172.21.15.59:0/4099371650 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:27.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52bffff640 1 -- 172.21.15.59:0/4099371650 >> v2:172.21.15.164:3300/0 conn(0x7f52c0162470 msgr2=0x7f52c006f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52bffff640 1 --2- 172.21.15.59:0/4099371650 >> v2:172.21.15.164:3300/0 conn(0x7f52c0162470 0x7f52c006f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52bffff640 1 -- 172.21.15.59:0/4099371650 >> v1:172.21.15.164:6789/0 conn(0x7f52c0162850 legacy=0x7f52c00d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:27.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.369+0000 7f52bffff640 1 -- 172.21.15.59:0/4099371650 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c006ec20 con 0x7f52c00d2160 2024-03-06T07:34:27.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.370+0000 7f52bffff640 1 --2- 172.21.15.59:0/4099371650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00d2160 0x7f52c006c3b0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f52c006f3f0 tx=0x7f52b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6763d3d67843b1cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.370+0000 7f52beffd640 1 -- 172.21.15.59:0/4099371650 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b802ca70 con 0x7f52c00d2160 2024-03-06T07:34:27.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.370+0000 7f52beffd640 1 -- 172.21.15.59:0/4099371650 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b802cc10 con 0x7f52c00d2160 2024-03-06T07:34:27.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.370+0000 7f52beffd640 1 -- 172.21.15.59:0/4099371650 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b80343f0 con 0x7f52c00d2160 2024-03-06T07:34:27.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.371+0000 7f52c5dbf640 1 -- 172.21.15.59:0/4099371650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00d2160 msgr2=0x7f52c006c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.371+0000 7f52c5dbf640 1 --2- 172.21.15.59:0/4099371650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00d2160 0x7f52c006c3b0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f52c006f3f0 tx=0x7f52b802bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.371+0000 7f52c5dbf640 1 -- 172.21.15.59:0/4099371650 shutdown_connections 2024-03-06T07:34:27.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.371+0000 7f52c5dbf640 1 --2- 172.21.15.59:0/4099371650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00d2160 0x7f52c006c3b0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.371+0000 7f52c5dbf640 1 --2- 172.21.15.59:0/4099371650 >> v2:172.21.15.164:3300/0 conn(0x7f52c0162470 0x7f52c006f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.371+0000 7f52c5dbf640 1 -- 172.21.15.59:0/4099371650 >> 172.21.15.59:0/4099371650 conn(0x7f52c009a9f0 msgr2=0x7f52c009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:27.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.371+0000 7f52c5dbf640 1 -- 172.21.15.59:0/4099371650 shutdown_connections 2024-03-06T07:34:27.374 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.371+0000 7f52c5dbf640 1 -- 172.21.15.59:0/4099371650 wait complete. 2024-03-06T07:34:27.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.372+0000 7f52c5dbf640 1 Processor -- start 2024-03-06T07:34:27.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.372+0000 7f52c5dbf640 1 -- start start 2024-03-06T07:34:27.376 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52c5dbf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00d2160 0x7f52c0159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52c5dbf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52c0162470 0x7f52c00b5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52c5dbf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52c0162850 0x7f52c0160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52c5dbf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c006fdc0 con 0x7f52c00d2160 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52c5dbf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c006fc40 con 0x7f52c0162470 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52c5dbf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c006ff40 con 0x7f52c0162850 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52bf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00d2160 0x7f52c0159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52bf7fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00d2160 0x7f52c0159520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50684/0 (socket says 172.21.15.59:50684) 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52bf7fe640 1 -- 172.21.15.59:0/1850707956 learned_addr learned my addr 172.21.15.59:0/1850707956 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52b6ffd640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52c0162470 0x7f52c00b5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52bffff640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52c0162850 0x7f52c0160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52bf7fe640 1 -- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52c0162850 msgr2=0x7f52c0160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52bf7fe640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52c0162850 0x7f52c0160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52bf7fe640 1 -- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52c0162470 msgr2=0x7f52c00b5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:27.377 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52bf7fe640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52c0162470 0x7f52c00b5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:27.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52bf7fe640 1 -- 172.21.15.59:0/1850707956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c00b6440 con 0x7f52c00d2160 2024-03-06T07:34:27.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52bffff640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52c0162850 0x7f52c0160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:27.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.373+0000 7f52bf7fe640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00d2160 0x7f52c0159520 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f52b000b4d0 tx=0x7f52b000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.374+0000 7f52bcff9640 1 -- 172.21.15.59:0/1850707956 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0014070 con 0x7f52c00d2160 2024-03-06T07:34:27.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.374+0000 7f52bcff9640 1 -- 172.21.15.59:0/1850707956 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b0010870 con 0x7f52c00d2160 2024-03-06T07:34:27.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.374+0000 7f52c5dbf640 1 -- 172.21.15.59:0/1850707956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c00b6720 con 0x7f52c00d2160 2024-03-06T07:34:27.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.374+0000 7f52bcff9640 1 -- 172.21.15.59:0/1850707956 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0010c60 con 0x7f52c00d2160 2024-03-06T07:34:27.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.374+0000 7f52c5dbf640 1 -- 172.21.15.59:0/1850707956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c00b6ce0 con 0x7f52c00d2160 2024-03-06T07:34:27.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.375+0000 7f52c5dbf640 1 -- 172.21.15.59:0/1850707956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c0159b20 con 0x7f52c00d2160 2024-03-06T07:34:27.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.375+0000 7f52bcff9640 1 -- 172.21.15.59:0/1850707956 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52b0017070 con 0x7f52c00d2160 2024-03-06T07:34:27.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.376+0000 7f52bcff9640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52a006b4d0 0x7f52a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:27.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.376+0000 7f52bcff9640 1 -- 172.21.15.59:0/1850707956 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f52b008a030 con 0x7f52c00d2160 2024-03-06T07:34:27.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.376+0000 7f52b6ffd640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52a006b4d0 0x7f52a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:27.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.376+0000 7f52b6ffd640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52a006b4d0 0x7f52a006d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f52c00c7eb0 tx=0x7f52a8009060 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:27.382 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.380+0000 7f52bcff9640 1 -- 172.21.15.59:0/1850707956 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52b0058790 con 0x7f52c00d2160 2024-03-06T07:34:27.562 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:27.560+0000 7f52c5dbf640 1 -- 172.21.15.59:0/1850707956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f52c004f330 con 0x7f52c00d2160 2024-03-06T07:34:28.204 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.202+0000 7f52bcff9640 1 -- 172.21.15.59:0/1850707956 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v145) v1 ==== 181+0+0 (secure 0 0 0) 0x7f52b005c830 con 0x7f52c00d2160 2024-03-06T07:34:28.204 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.205+0000 7f52c5dbf640 1 -- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52a006b4d0 msgr2=0x7f52a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.205+0000 7f52c5dbf640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52a006b4d0 0x7f52a006d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f52c00c7eb0 tx=0x7f52a8009060 comp rx=0 tx=0).stop 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.205+0000 7f52c5dbf640 1 -- 172.21.15.59:0/1850707956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00d2160 msgr2=0x7f52c0159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.205+0000 7f52c5dbf640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00d2160 0x7f52c0159520 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f52b000b4d0 tx=0x7f52b000b990 comp rx=0 tx=0).stop 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.206+0000 7f52c5dbf640 1 -- 172.21.15.59:0/1850707956 shutdown_connections 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.206+0000 7f52c5dbf640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f52a006b4d0 0x7f52a006d990 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.206+0000 7f52c5dbf640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52c0162850 0x7f52c0160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.206+0000 7f52c5dbf640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52c0162470 0x7f52c00b5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.206+0000 7f52c5dbf640 1 --2- 172.21.15.59:0/1850707956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00d2160 0x7f52c0159520 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.206+0000 7f52c5dbf640 1 -- 172.21.15.59:0/1850707956 >> 172.21.15.59:0/1850707956 conn(0x7f52c009a9f0 msgr2=0x7f52c006cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.206+0000 7f52c5dbf640 1 -- 172.21.15.59:0/1850707956 shutdown_connections 2024-03-06T07:34:28.208 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.206+0000 7f52c5dbf640 1 -- 172.21.15.59:0/1850707956 wait complete. 2024-03-06T07:34:28.223 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:34:28.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.333+0000 7f31f8683640 1 Processor -- start 2024-03-06T07:34:28.335 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.333+0000 7f31f8683640 1 -- start start 2024-03-06T07:34:28.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.333+0000 7f31f8683640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f31f015ac00 0x7f31f006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f8683640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31f00708b0 0x7f31f0070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f8683640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31f015e7a0 0x7f31f015ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f8683640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f0071250 con 0x7f31f015e7a0 2024-03-06T07:34:28.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f8683640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f00cdc10 con 0x7f31f015ac00 2024-03-06T07:34:28.336 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f8683640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f015afe0 con 0x7f31f00708b0 2024-03-06T07:34:28.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f6c20640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31f015e7a0 0x7f31f015ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f6c20640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31f015e7a0 0x7f31f015ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50700/0 (socket says 172.21.15.59:50700) 2024-03-06T07:34:28.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f6c20640 1 -- 172.21.15.59:0/1996401238 learned_addr learned my addr 172.21.15.59:0/1996401238 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:28.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f5c1e640 1 --2- 172.21.15.59:0/1996401238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31f00708b0 0x7f31f0070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f641f640 1 --2- 172.21.15.59:0/1996401238 >> v2:172.21.15.164:3300/0 conn(0x7f31f015ac00 0x7f31f006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f6c20640 1 -- 172.21.15.59:0/1996401238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31f00708b0 msgr2=0x7f31f0070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f6c20640 1 --2- 172.21.15.59:0/1996401238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31f00708b0 0x7f31f0070d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f6c20640 1 -- 172.21.15.59:0/1996401238 >> v2:172.21.15.164:3300/0 conn(0x7f31f015ac00 msgr2=0x7f31f006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:28.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f6c20640 1 --2- 172.21.15.59:0/1996401238 >> v2:172.21.15.164:3300/0 conn(0x7f31f015ac00 0x7f31f006c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.337 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.334+0000 7f31f6c20640 1 -- 172.21.15.59:0/1996401238 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f00cd000 con 0x7f31f015e7a0 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.335+0000 7f31f6c20640 1 --2- 172.21.15.59:0/1996401238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f015e7a0 0x7f31f015ec20 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f31f00730c0 tx=0x7f31e802e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ed8d8c3a1a8cc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.335+0000 7f31f541d640 1 -- 172.21.15.59:0/1996401238 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e802ebf0 con 0x7f31f015e7a0 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.335+0000 7f31f541d640 1 -- 172.21.15.59:0/1996401238 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e802ed90 con 0x7f31f015e7a0 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.335+0000 7f31f541d640 1 -- 172.21.15.59:0/1996401238 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e8036610 con 0x7f31f015e7a0 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.335+0000 7f31f8683640 1 -- 172.21.15.59:0/1996401238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f015e7a0 msgr2=0x7f31f015ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.335+0000 7f31f8683640 1 --2- 172.21.15.59:0/1996401238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f015e7a0 0x7f31f015ec20 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f31f00730c0 tx=0x7f31e802e050 comp rx=0 tx=0).stop 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.336+0000 7f31f8683640 1 -- 172.21.15.59:0/1996401238 shutdown_connections 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.336+0000 7f31f8683640 1 --2- 172.21.15.59:0/1996401238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f015e7a0 0x7f31f015ec20 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.336+0000 7f31f8683640 1 --2- 172.21.15.59:0/1996401238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31f00708b0 0x7f31f0070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.336+0000 7f31f8683640 1 --2- 172.21.15.59:0/1996401238 >> v2:172.21.15.164:3300/0 conn(0x7f31f015ac00 0x7f31f006c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.336+0000 7f31f8683640 1 -- 172.21.15.59:0/1996401238 >> 172.21.15.59:0/1996401238 conn(0x7f31f00b1650 msgr2=0x7f31f00b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.336+0000 7f31f8683640 1 -- 172.21.15.59:0/1996401238 shutdown_connections 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.336+0000 7f31f8683640 1 -- 172.21.15.59:0/1996401238 wait complete. 2024-03-06T07:34:28.339 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.337+0000 7f31f8683640 1 Processor -- start 2024-03-06T07:34:28.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.337+0000 7f31f8683640 1 -- start start 2024-03-06T07:34:28.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.337+0000 7f31f8683640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f00708b0 0x7f31f01f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.337+0000 7f31f8683640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31f015ac00 0x7f31f00afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.337+0000 7f31f8683640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31f015e7a0 0x7f31f0144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.337+0000 7f31f8683640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00cd8a0 con 0x7f31f00708b0 2024-03-06T07:34:28.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.337+0000 7f31f8683640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00cd5c0 con 0x7f31f015ac00 2024-03-06T07:34:28.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.337+0000 7f31f8683640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f006ce50 con 0x7f31f015e7a0 2024-03-06T07:34:28.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f641f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f00708b0 0x7f31f01f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f641f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f00708b0 0x7f31f01f3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50714/0 (socket says 172.21.15.59:50714) 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f641f640 1 -- 172.21.15.59:0/243739386 learned_addr learned my addr 172.21.15.59:0/243739386 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f5c1e640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31f015ac00 0x7f31f00afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f6c20640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31f015e7a0 0x7f31f0144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f641f640 1 -- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31f015e7a0 msgr2=0x7f31f0144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f641f640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31f015e7a0 0x7f31f0144150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f641f640 1 -- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31f015ac00 msgr2=0x7f31f00afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f641f640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31f015ac00 0x7f31f00afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f641f640 1 -- 172.21.15.59:0/243739386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f015dce0 con 0x7f31f00708b0 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f6c20640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31f015e7a0 0x7f31f0144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f641f640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f00708b0 0x7f31f01f3be0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00d800 tx=0x7f31ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31e6ffd640 1 -- 172.21.15.59:0/243739386 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec00b6c0 con 0x7f31f00708b0 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31e6ffd640 1 -- 172.21.15.59:0/243739386 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec019070 con 0x7f31f00708b0 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f8683640 1 -- 172.21.15.59:0/243739386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f015b2c0 con 0x7f31f00708b0 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31e6ffd640 1 -- 172.21.15.59:0/243739386 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec00ba80 con 0x7f31f00708b0 2024-03-06T07:34:28.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.338+0000 7f31f8683640 1 -- 172.21.15.59:0/243739386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f015b880 con 0x7f31f00708b0 2024-03-06T07:34:28.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.339+0000 7f31f8683640 1 -- 172.21.15.59:0/243739386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f00687d0 con 0x7f31f00708b0 2024-03-06T07:34:28.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.340+0000 7f31e6ffd640 1 -- 172.21.15.59:0/243739386 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31ec00bc20 con 0x7f31f00708b0 2024-03-06T07:34:28.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.340+0000 7f31e6ffd640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31d406b4d0 0x7f31d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:28.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.340+0000 7f31e6ffd640 1 -- 172.21.15.59:0/243739386 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5472+0+0 (secure 0 0 0) 0x7f31ec08a5a0 con 0x7f31f00708b0 2024-03-06T07:34:28.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.340+0000 7f31f5c1e640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31d406b4d0 0x7f31d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:28.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.341+0000 7f31f5c1e640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31d406b4d0 0x7f31d406d990 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f31f0153ff0 tx=0x7f31e0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:28.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.345+0000 7f31e6ffd640 1 -- 172.21.15.59:0/243739386 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31ec059490 con 0x7f31f00708b0 2024-03-06T07:34:28.533 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:28.532+0000 7f31f8683640 1 -- 172.21.15.59:0/243739386 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f31f00b0510 con 0x7f31f00708b0 2024-03-06T07:34:29.206 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.204+0000 7f31e6ffd640 1 -- 172.21.15.59:0/243739386 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v146) v1 ==== 169+0+0 (secure 0 0 0) 0x7f31ec058d90 con 0x7f31f00708b0 2024-03-06T07:34:29.206 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'cephfs_data' removed 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.207+0000 7f31f8683640 1 -- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31d406b4d0 msgr2=0x7f31d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.207+0000 7f31f8683640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31d406b4d0 0x7f31d406d990 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f31f0153ff0 tx=0x7f31e0008040 comp rx=0 tx=0).stop 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.207+0000 7f31f8683640 1 -- 172.21.15.59:0/243739386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f00708b0 msgr2=0x7f31f01f3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.207+0000 7f31f8683640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f00708b0 0x7f31f01f3be0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00d800 tx=0x7f31ec00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.208+0000 7f31f8683640 1 -- 172.21.15.59:0/243739386 shutdown_connections 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.208+0000 7f31f8683640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f31d406b4d0 0x7f31d406d990 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.208+0000 7f31f8683640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f31f015e7a0 0x7f31f0144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.208+0000 7f31f8683640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f31f015ac00 0x7f31f00afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.208+0000 7f31f8683640 1 --2- 172.21.15.59:0/243739386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31f00708b0 0x7f31f01f3be0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.208+0000 7f31f8683640 1 -- 172.21.15.59:0/243739386 >> 172.21.15.59:0/243739386 conn(0x7f31f00b1650 msgr2=0x7f31f0155980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.208+0000 7f31f8683640 1 -- 172.21.15.59:0/243739386 shutdown_connections 2024-03-06T07:34:29.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.208+0000 7f31f8683640 1 -- 172.21.15.59:0/243739386 wait complete. 2024-03-06T07:34:29.224 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:34:29.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.336+0000 7f6097993640 1 Processor -- start 2024-03-06T07:34:29.338 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.336+0000 7f6097993640 1 -- start start 2024-03-06T07:34:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6097993640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f609015bdd0 0x7f60900b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6097993640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60900b15c0 0x7f60900b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6097993640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60900d0240 con 0x7f60900ce300 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6097993640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60900cf4f0 con 0x7f60900b15c0 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6097993640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60900d0850 con 0x7f609015bdd0 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6095f30640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60900b15c0 0x7f60900b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6095f30640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f60900b15c0 0x7f60900b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50726/0 (socket says 172.21.15.59:50726) 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6095f30640 1 -- 172.21.15.59:0/2796231082 learned_addr learned my addr 172.21.15.59:0/2796231082 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f609572f640 1 --2- 172.21.15.59:0/2796231082 >> v2:172.21.15.164:3300/0 conn(0x7f609015bdd0 0x7f60900b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6095f30640 1 -- 172.21.15.59:0/2796231082 >> v2:172.21.15.164:3300/0 conn(0x7f609015bdd0 msgr2=0x7f60900b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6095f30640 1 --2- 172.21.15.59:0/2796231082 >> v2:172.21.15.164:3300/0 conn(0x7f609015bdd0 0x7f60900b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6095f30640 1 -- 172.21.15.59:0/2796231082 >> v1:172.21.15.164:6789/0 conn(0x7f60900ce300 legacy=0x7f60900ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.341 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.338+0000 7f6095f30640 1 -- 172.21.15.59:0/2796231082 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60900cf210 con 0x7f60900b15c0 2024-03-06T07:34:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.340+0000 7f6095f30640 1 --2- 172.21.15.59:0/2796231082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60900b15c0 0x7f60900b19d0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f6090071cc0 tx=0x7f608802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece53845c61199b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.340+0000 7f6087fff640 1 -- 172.21.15.59:0/2796231082 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608802ca80 con 0x7f60900b15c0 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.340+0000 7f6087fff640 1 -- 172.21.15.59:0/2796231082 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608802cc20 con 0x7f60900b15c0 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.340+0000 7f6087fff640 1 -- 172.21.15.59:0/2796231082 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880343f0 con 0x7f60900b15c0 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.340+0000 7f6097993640 1 -- 172.21.15.59:0/2796231082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60900b15c0 msgr2=0x7f60900b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.340+0000 7f6097993640 1 --2- 172.21.15.59:0/2796231082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60900b15c0 0x7f60900b19d0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f6090071cc0 tx=0x7f608802bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.341+0000 7f6097993640 1 -- 172.21.15.59:0/2796231082 shutdown_connections 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.341+0000 7f6097993640 1 --2- 172.21.15.59:0/2796231082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60900b15c0 0x7f60900b19d0 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.341+0000 7f6097993640 1 --2- 172.21.15.59:0/2796231082 >> v2:172.21.15.164:3300/0 conn(0x7f609015bdd0 0x7f60900b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.341+0000 7f6097993640 1 -- 172.21.15.59:0/2796231082 >> 172.21.15.59:0/2796231082 conn(0x7f60900b3690 msgr2=0x7f60900b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.341+0000 7f6097993640 1 -- 172.21.15.59:0/2796231082 shutdown_connections 2024-03-06T07:34:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.341+0000 7f6097993640 1 -- 172.21.15.59:0/2796231082 wait complete. 2024-03-06T07:34:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.342+0000 7f6097993640 1 Processor -- start 2024-03-06T07:34:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.342+0000 7f6097993640 1 -- start start 2024-03-06T07:34:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6097993640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60900b15c0 0x7f60901f8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6097993640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60900ce300 0x7f60900c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6097993640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f609015bdd0 0x7f6090152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6097993640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609015c1b0 con 0x7f60900ce300 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6097993640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60900ceea0 con 0x7f609015bdd0 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6097993640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60900b10c0 con 0x7f60900b15c0 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6094f2e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60900ce300 0x7f60900c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6094f2e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60900ce300 0x7f60900c7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50732/0 (socket says 172.21.15.59:50732) 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6094f2e640 1 -- 172.21.15.59:0/2962583434 learned_addr learned my addr 172.21.15.59:0/2962583434 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6095f30640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f609015bdd0 0x7f6090152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f609572f640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60900b15c0 0x7f60901f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6094f2e640 1 -- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60900b15c0 msgr2=0x7f60901f8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6094f2e640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60900b15c0 0x7f60901f8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6094f2e640 1 -- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f609015bdd0 msgr2=0x7f6090152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6094f2e640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f609015bdd0 0x7f6090152d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6094f2e640 1 -- 172.21.15.59:0/2962583434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60901f8550 con 0x7f60900ce300 2024-03-06T07:34:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.343+0000 7f6095f30640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f609015bdd0 0x7f6090152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:29.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.344+0000 7f6094f2e640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60900ce300 0x7f60900c7de0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f608000b3e0 tx=0x7f608000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.344+0000 7f6085ffb640 1 -- 172.21.15.59:0/2962583434 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080015020 con 0x7f60900ce300 2024-03-06T07:34:29.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.344+0000 7f6085ffb640 1 -- 172.21.15.59:0/2962583434 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6080011850 con 0x7f60900ce300 2024-03-06T07:34:29.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.344+0000 7f6097993640 1 -- 172.21.15.59:0/2962583434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60900c6d40 con 0x7f60900ce300 2024-03-06T07:34:29.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.344+0000 7f6085ffb640 1 -- 172.21.15.59:0/2962583434 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080011c40 con 0x7f60900ce300 2024-03-06T07:34:29.348 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.344+0000 7f6097993640 1 -- 172.21.15.59:0/2962583434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60900c7300 con 0x7f60900ce300 2024-03-06T07:34:29.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.345+0000 7f6097993640 1 -- 172.21.15.59:0/2962583434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609009fcf0 con 0x7f60900ce300 2024-03-06T07:34:29.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.345+0000 7f6085ffb640 1 -- 172.21.15.59:0/2962583434 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6080018020 con 0x7f60900ce300 2024-03-06T07:34:29.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.346+0000 7f6085ffb640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f607406b4d0 0x7f607406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.346+0000 7f6085ffb640 1 -- 172.21.15.59:0/2962583434 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6080089ed0 con 0x7f60900ce300 2024-03-06T07:34:29.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.346+0000 7f609572f640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f607406b4d0 0x7f607406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.347+0000 7f609572f640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f607406b4d0 0x7f607406d990 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f608c006fd0 tx=0x7f608c008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.352 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.350+0000 7f6085ffb640 1 -- 172.21.15.59:0/2962583434 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f608001f020 con 0x7f60900ce300 2024-03-06T07:34:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.530+0000 7f6097993640 1 -- 172.21.15.59:0/2962583434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6090068ff0 con 0x7f60900ce300 2024-03-06T07:34:29.536 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.533+0000 7f6085ffb640 1 -- 172.21.15.59:0/2962583434 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f60901f8550 con 0x7f60900ce300 2024-03-06T07:34:29.537 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 -- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f607406b4d0 msgr2=0x7f607406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f607406b4d0 0x7f607406d990 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f608c006fd0 tx=0x7f608c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 -- 172.21.15.59:0/2962583434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60900ce300 msgr2=0x7f60900c7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60900ce300 0x7f60900c7de0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f608000b3e0 tx=0x7f608000b8a0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 -- 172.21.15.59:0/2962583434 shutdown_connections 2024-03-06T07:34:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f607406b4d0 0x7f607406d990 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f609015bdd0 0x7f6090152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f60900ce300 0x7f60900c7de0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 --2- 172.21.15.59:0/2962583434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60900b15c0 0x7f60901f8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 -- 172.21.15.59:0/2962583434 >> 172.21.15.59:0/2962583434 conn(0x7f60900b3690 msgr2=0x7f609006f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 -- 172.21.15.59:0/2962583434 shutdown_connections 2024-03-06T07:34:29.538 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.535+0000 7f6097993640 1 -- 172.21.15.59:0/2962583434 wait complete. 2024-03-06T07:34:29.548 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-03-06T07:34:29.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.648+0000 7fa66ab02640 1 Processor -- start 2024-03-06T07:34:29.650 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.648+0000 7fa66ab02640 1 -- start start 2024-03-06T07:34:29.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66ab02640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa6640d00c0 0x7fa6640d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66ab02640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa66406cf20 0x7fa66406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66ab02640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6640d09e0 con 0x7fa6640cf850 2024-03-06T07:34:29.651 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66ab02640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66406ca90 con 0x7fa66406cf20 2024-03-06T07:34:29.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66ab02640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66415e380 con 0x7fa6640d00c0 2024-03-06T07:34:29.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66909f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa66406cf20 0x7fa66406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66909f640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa66406cf20 0x7fa66406d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50740/0 (socket says 172.21.15.59:50740) 2024-03-06T07:34:29.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66909f640 1 -- 172.21.15.59:0/2475854035 learned_addr learned my addr 172.21.15.59:0/2475854035 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:29.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66909f640 1 -- 172.21.15.59:0/2475854035 >> v2:172.21.15.164:3300/0 conn(0x7fa6640d00c0 msgr2=0x7fa6640d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66909f640 1 --2- 172.21.15.59:0/2475854035 >> v2:172.21.15.164:3300/0 conn(0x7fa6640d00c0 0x7fa6640d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66909f640 1 -- 172.21.15.59:0/2475854035 >> v1:172.21.15.164:6789/0 conn(0x7fa6640cf850 legacy=0x7fa6640cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:29.652 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.649+0000 7fa66909f640 1 -- 172.21.15.59:0/2475854035 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6640cdc10 con 0x7fa66406cf20 2024-03-06T07:34:29.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.650+0000 7fa66909f640 1 --2- 172.21.15.59:0/2475854035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66406cf20 0x7fa66406d3b0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fa6640d1a40 tx=0x7fa65c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18fe3971d0e24c3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.650+0000 7fa65b7fe640 1 -- 172.21.15.59:0/2475854035 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c02ca80 con 0x7fa66406cf20 2024-03-06T07:34:29.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.650+0000 7fa65b7fe640 1 -- 172.21.15.59:0/2475854035 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c02cc20 con 0x7fa66406cf20 2024-03-06T07:34:29.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.650+0000 7fa65b7fe640 1 -- 172.21.15.59:0/2475854035 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0343f0 con 0x7fa66406cf20 2024-03-06T07:34:29.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.651+0000 7fa66ab02640 1 -- 172.21.15.59:0/2475854035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66406cf20 msgr2=0x7fa66406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.651+0000 7fa66ab02640 1 --2- 172.21.15.59:0/2475854035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66406cf20 0x7fa66406d3b0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fa6640d1a40 tx=0x7fa65c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.651+0000 7fa66ab02640 1 -- 172.21.15.59:0/2475854035 shutdown_connections 2024-03-06T07:34:29.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.651+0000 7fa66ab02640 1 --2- 172.21.15.59:0/2475854035 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66406cf20 0x7fa66406d3b0 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.651+0000 7fa66ab02640 1 --2- 172.21.15.59:0/2475854035 >> v2:172.21.15.164:3300/0 conn(0x7fa6640d00c0 0x7fa6640d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.651+0000 7fa66ab02640 1 -- 172.21.15.59:0/2475854035 >> 172.21.15.59:0/2475854035 conn(0x7fa6640b35a0 msgr2=0x7fa6640b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:29.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.651+0000 7fa66ab02640 1 -- 172.21.15.59:0/2475854035 shutdown_connections 2024-03-06T07:34:29.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.651+0000 7fa66ab02640 1 -- 172.21.15.59:0/2475854035 wait complete. 2024-03-06T07:34:29.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.652+0000 7fa66ab02640 1 Processor -- start 2024-03-06T07:34:29.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.652+0000 7fa66ab02640 1 -- start start 2024-03-06T07:34:29.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66ab02640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66406cf20 0x7fa6640c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66ab02640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6640cf850 0x7fa6640c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66ab02640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6640d00c0 0x7fa6640c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66ab02640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66406c720 con 0x7fa66406cf20 2024-03-06T07:34:29.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66ab02640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66406c440 con 0x7fa6640d00c0 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66ab02640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6640b1030 con 0x7fa6640cf850 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66889e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66406cf20 0x7fa6640c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66889e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66406cf20 0x7fa6640c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50752/0 (socket says 172.21.15.59:50752) 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66889e640 1 -- 172.21.15.59:0/1052198788 learned_addr learned my addr 172.21.15.59:0/1052198788 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa65bfff640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6640cf850 0x7fa6640c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66909f640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6640d00c0 0x7fa6640c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66889e640 1 -- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6640cf850 msgr2=0x7fa6640c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66889e640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6640cf850 0x7fa6640c7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66889e640 1 -- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6640d00c0 msgr2=0x7fa6640c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66889e640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6640d00c0 0x7fa6640c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66889e640 1 -- 172.21.15.59:0/1052198788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6640b4e80 con 0x7fa66406cf20 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66889e640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66406cf20 0x7fa6640c78e0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fa66000b4a0 tx=0x7fa66000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa6597fa640 1 -- 172.21.15.59:0/1052198788 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660014070 con 0x7fa66406cf20 2024-03-06T07:34:29.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa6597fa640 1 -- 172.21.15.59:0/1052198788 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa660010610 con 0x7fa66406cf20 2024-03-06T07:34:29.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66ab02640 1 -- 172.21.15.59:0/1052198788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6640b5160 con 0x7fa66406cf20 2024-03-06T07:34:29.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa6597fa640 1 -- 172.21.15.59:0/1052198788 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660010a00 con 0x7fa66406cf20 2024-03-06T07:34:29.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.653+0000 7fa66ab02640 1 -- 172.21.15.59:0/1052198788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6640b5720 con 0x7fa66406cf20 2024-03-06T07:34:29.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.655+0000 7fa66ab02640 1 -- 172.21.15.59:0/1052198788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66404f330 con 0x7fa66406cf20 2024-03-06T07:34:29.658 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.655+0000 7fa6597fa640 1 -- 172.21.15.59:0/1052198788 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa660017070 con 0x7fa66406cf20 2024-03-06T07:34:29.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.655+0000 7fa6597fa640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa65006b4d0 0x7fa65006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:29.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.655+0000 7fa6597fa640 1 -- 172.21.15.59:0/1052198788 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fa6600899a0 con 0x7fa66406cf20 2024-03-06T07:34:29.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.656+0000 7fa65bfff640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa65006b4d0 0x7fa65006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:29.659 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.656+0000 7fa65bfff640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa65006b4d0 0x7fa65006d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa654007fd0 tx=0x7fa654009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:29.661 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.660+0000 7fa6597fa640 1 -- 172.21.15.59:0/1052198788 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa660058ac0 con 0x7fa66406cf20 2024-03-06T07:34:29.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:29.908+0000 7fa66ab02640 1 -- 172.21.15.59:0/1052198788 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) v1 -- 0x7fa6640ced90 con 0x7fa66406cf20 2024-03-06T07:34:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.212+0000 7fa6597fa640 1 -- 172.21.15.59:0/1052198788 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v663) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa6600583c0 con 0x7fa66406cf20 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 -- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa65006b4d0 msgr2=0x7fa65006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa65006b4d0 0x7fa65006d990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa654007fd0 tx=0x7fa654009040 comp rx=0 tx=0).stop 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 -- 172.21.15.59:0/1052198788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66406cf20 msgr2=0x7fa6640c78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66406cf20 0x7fa6640c78e0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fa66000b4a0 tx=0x7fa66000b960 comp rx=0 tx=0).stop 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 -- 172.21.15.59:0/1052198788 shutdown_connections 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa65006b4d0 0x7fa65006d990 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6640d00c0 0x7fa6640c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6640cf850 0x7fa6640c7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 --2- 172.21.15.59:0/1052198788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66406cf20 0x7fa6640c78e0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 -- 172.21.15.59:0/1052198788 >> 172.21.15.59:0/1052198788 conn(0x7fa6640b35a0 msgr2=0x7fa66406f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 -- 172.21.15.59:0/1052198788 shutdown_connections 2024-03-06T07:34:30.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:30.214+0000 7fa66ab02640 1 -- 172.21.15.59:0/1052198788 wait complete. 2024-03-06T07:34:30.228 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays ... ERROR 2024-03-06T07:34:30.228 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner:ERROR: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 696, in test_migration_on_shutdown 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner:Ran 9 tests in 729.173s 2024-03-06T07:34:30.229 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner:ERROR: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 696, in test_migration_on_shutdown 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:34:30.230 INFO:tasks.cephfs_test_runner: 2024-03-06T07:34:30.231 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:30.421 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=684a716db6654dd6866463d2286454a4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:34:30.422 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-06T07:34:30.432 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-06T07:34:30.466 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-06T07:34:30.466 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:34:30.466 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:30.486 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:30.486 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2024-03-06T07:34:30.486 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:30.542 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057147,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:34:30.542 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:34:30.542 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:30.561 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:30.561 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-06T07:34:30.561 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:30.619 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15790 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:34:30.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:30.619 DEBUG:teuthology.orchestra.run.smithi164:> ip netns list 2024-03-06T07:34:30.677 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-06T07:34:30.677 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:34:30.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:30.677 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:34:30.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:30.750 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:34:30.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:30.821 DEBUG:teuthology.orchestra.run.smithi164:> sudo ip link delete ceph-brx 2024-03-06T07:34:30.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:30.919 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2024-03-06T07:34:30.937 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:34:30.937 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:34:30.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:30.938 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:34:31.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:31.008 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:34:31.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:31.078 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link delete ceph-brx 2024-03-06T07:34:31.175 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-06T07:34:31.184 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:34:31.184 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:31.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.287+0000 7f74e9d56640 1 Processor -- start 2024-03-06T07:34:31.289 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.287+0000 7f74e9d56640 1 -- start start 2024-03-06T07:34:31.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.288+0000 7f74e9d56640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f74e40b3060 0x7f74e415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.288+0000 7f74e9d56640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e415f580 con 0x7f74e413dc10 2024-03-06T07:34:31.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.288+0000 7f74e9d56640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e415b450 con 0x7f74e415bf30 2024-03-06T07:34:31.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.288+0000 7f74e9d56640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e40cf3f0 con 0x7f74e40b3060 2024-03-06T07:34:31.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.288+0000 7f74e37fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f74e413dc10 0x7f74e40b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35678/0 (socket says 172.21.15.59:35678) 2024-03-06T07:34:31.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.288+0000 7f74e37fe640 1 -- 172.21.15.59:0/2080290346 learned_addr learned my addr 172.21.15.59:0/2080290346 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:31.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.288+0000 7f74e2ffd640 1 --2- 172.21.15.59:0/2080290346 >> v2:172.21.15.164:3300/0 conn(0x7f74e40b3060 0x7f74e415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.293 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.288+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778357653 0 0) 0x7f74e415f580 con 0x7f74e413dc10 2024-03-06T07:34:31.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.288+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74cc003520 con 0x7f74e413dc10 2024-03-06T07:34:31.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3379637875 0 0) 0x7f74cc003520 con 0x7f74e413dc10 2024-03-06T07:34:31.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 >> v2:172.21.15.164:3300/0 conn(0x7f74e40b3060 msgr2=0x7f74e415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:31.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 --2- 172.21.15.59:0/2080290346 >> v2:172.21.15.164:3300/0 conn(0x7f74e40b3060 0x7f74e415e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 >> v1:172.21.15.164:6789/0 conn(0x7f74e415bf30 legacy=0x7f74e415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e415b730 con 0x7f74e413dc10 2024-03-06T07:34:31.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f74d4001670 con 0x7f74e413dc10 2024-03-06T07:34:31.294 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 >> v1:172.21.15.59:6789/0 conn(0x7f74e413dc10 legacy=0x7f74e40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 --2- 172.21.15.59:0/2080290346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74cc0039b0 0x7f74cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 --2- 172.21.15.59:0/2080290346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74cc004800 0x7f74cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 --2- 172.21.15.59:0/2080290346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74cc0051c0 0x7f74cc00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e40cf3f0 con 0x7f74cc004800 2024-03-06T07:34:31.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc00e370 con 0x7f74cc0039b0 2024-03-06T07:34:31.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc00e570 con 0x7f74cc0051c0 2024-03-06T07:34:31.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e2ffd640 1 --2- 172.21.15.59:0/2080290346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74cc004800 0x7f74cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.295 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e37fe640 1 --2- 172.21.15.59:0/2080290346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74cc0039b0 0x7f74cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e3fff640 1 --2- 172.21.15.59:0/2080290346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74cc0051c0 0x7f74cc00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e2ffd640 1 -- 172.21.15.59:0/2080290346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74cc0051c0 msgr2=0x7f74cc00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:31.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e2ffd640 1 --2- 172.21.15.59:0/2080290346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74cc0051c0 0x7f74cc00dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e2ffd640 1 -- 172.21.15.59:0/2080290346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74cc0039b0 msgr2=0x7f74cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:31.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e2ffd640 1 --2- 172.21.15.59:0/2080290346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74cc0039b0 0x7f74cc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e2ffd640 1 -- 172.21.15.59:0/2080290346 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc003520 con 0x7f74cc004800 2024-03-06T07:34:31.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.289+0000 7f74e3fff640 1 -- 172.21.15.59:0/2080290346 reap_dead start 2024-03-06T07:34:31.296 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.290+0000 7f74e2ffd640 1 --2- 172.21.15.59:0/2080290346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74cc004800 0x7f74cc004c80 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f74d8009ac0 tx=0x7f74cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aace97c31769081 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:31.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.290+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8039070 con 0x7f74cc004800 2024-03-06T07:34:31.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.290+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d802c6b0 con 0x7f74cc004800 2024-03-06T07:34:31.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.290+0000 7f74e27fc640 1 -- 172.21.15.59:0/2080290346 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d8009690 con 0x7f74cc004800 2024-03-06T07:34:31.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.291+0000 7f74e9d56640 1 -- 172.21.15.59:0/2080290346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74cc004800 msgr2=0x7f74cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.291+0000 7f74e9d56640 1 --2- 172.21.15.59:0/2080290346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74cc004800 0x7f74cc004c80 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f74d8009ac0 tx=0x7f74cc005830 comp rx=0 tx=0).stop 2024-03-06T07:34:31.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.291+0000 7f74e9d56640 1 -- 172.21.15.59:0/2080290346 shutdown_connections 2024-03-06T07:34:31.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.291+0000 7f74e9d56640 1 --2- 172.21.15.59:0/2080290346 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74cc004800 0x7f74cc004c80 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.291+0000 7f74e9d56640 1 -- 172.21.15.59:0/2080290346 >> 172.21.15.59:0/2080290346 conn(0x7f74e40cf8a0 msgr2=0x7f74e40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:31.297 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.291+0000 7f74e9d56640 1 -- 172.21.15.59:0/2080290346 shutdown_connections 2024-03-06T07:34:31.298 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.291+0000 7f74e9d56640 1 -- 172.21.15.59:0/2080290346 wait complete. 2024-03-06T07:34:31.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.292+0000 7f74e9d56640 1 Processor -- start 2024-03-06T07:34:31.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.292+0000 7f74e9d56640 1 -- start start 2024-03-06T07:34:31.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e9d56640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74cc004800 0x7f74e413d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.300 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e9d56640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74e4156d20 0x7f74e41571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e9d56640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e413f9b0 0x7f74e413fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e9d56640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e40b2950 con 0x7f74e413f9b0 2024-03-06T07:34:31.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e9d56640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e415b210 con 0x7f74cc004800 2024-03-06T07:34:31.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e9d56640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e40b2ad0 con 0x7f74e4156d20 2024-03-06T07:34:31.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e413f9b0 0x7f74e413fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e3fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e413f9b0 0x7f74e413fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50770/0 (socket says 172.21.15.59:50770) 2024-03-06T07:34:31.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74cc004800 0x7f74e413d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.301 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e3fff640 1 -- 172.21.15.59:0/1368669732 learned_addr learned my addr 172.21.15.59:0/1368669732 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:31.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e2ffd640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74e4156d20 0x7f74e41571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e3fff640 1 -- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74e4156d20 msgr2=0x7f74e41571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e3fff640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74e4156d20 0x7f74e41571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e3fff640 1 -- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74cc004800 msgr2=0x7f74e413d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e3fff640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74cc004800 0x7f74e413d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e3fff640 1 -- 172.21.15.59:0/1368669732 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e415d610 con 0x7f74e413f9b0 2024-03-06T07:34:31.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74e3fff640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e413f9b0 0x7f74e413fe30 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f74e40b4370 tx=0x7f74dc00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:31.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74cbfff640 1 -- 172.21.15.59:0/1368669732 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74dc003970 con 0x7f74e413f9b0 2024-03-06T07:34:31.302 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.293+0000 7f74cbfff640 1 -- 172.21.15.59:0/1368669732 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74dc018070 con 0x7f74e413f9b0 2024-03-06T07:34:31.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.294+0000 7f74e9d56640 1 -- 172.21.15.59:0/1368669732 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e415d8f0 con 0x7f74e413f9b0 2024-03-06T07:34:31.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.294+0000 7f74cbfff640 1 -- 172.21.15.59:0/1368669732 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74dc003d70 con 0x7f74e413f9b0 2024-03-06T07:34:31.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.294+0000 7f74e9d56640 1 -- 172.21.15.59:0/1368669732 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e415deb0 con 0x7f74e413f9b0 2024-03-06T07:34:31.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.294+0000 7f74e9d56640 1 -- 172.21.15.59:0/1368669732 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e404f330 con 0x7f74e413f9b0 2024-03-06T07:34:31.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.295+0000 7f74cbfff640 1 -- 172.21.15.59:0/1368669732 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74dc0050d0 con 0x7f74e413f9b0 2024-03-06T07:34:31.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.295+0000 7f74cbfff640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f74d006b500 0x7f74d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.295+0000 7f74cbfff640 1 -- 172.21.15.59:0/1368669732 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f74dc089960 con 0x7f74e413f9b0 2024-03-06T07:34:31.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.296+0000 7f74e37fe640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f74d006b500 0x7f74d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.303 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.296+0000 7f74e37fe640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f74d006b500 0x7f74d006d9c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f74d40011e0 tx=0x7f74d40078f0 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:31.304 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.300+0000 7f74cbfff640 1 -- 172.21.15.59:0/1368669732 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f74dc0584c0 con 0x7f74e413f9b0 2024-03-06T07:34:31.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.479+0000 7f74e9d56640 1 -- 172.21.15.59:0/1368669732 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f74e406eb00 con 0x7f74d006b500 2024-03-06T07:34:31.481 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.480+0000 7f74cbfff640 1 -- 172.21.15.59:0/1368669732 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13503 (secure 0 0 0) 0x7f74e406eb00 con 0x7f74d006b500 2024-03-06T07:34:31.482 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:31.482 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:34:31.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.481+0000 7f74e9d56640 1 -- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f74d006b500 msgr2=0x7f74d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.481+0000 7f74e9d56640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f74d006b500 0x7f74d006d9c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f74d40011e0 tx=0x7f74d40078f0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.481+0000 7f74e9d56640 1 -- 172.21.15.59:0/1368669732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e413f9b0 msgr2=0x7f74e413fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.481+0000 7f74e9d56640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e413f9b0 0x7f74e413fe30 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f74e40b4370 tx=0x7f74dc00eb90 comp rx=0 tx=0).stop 2024-03-06T07:34:31.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.482+0000 7f74e9d56640 1 -- 172.21.15.59:0/1368669732 shutdown_connections 2024-03-06T07:34:31.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.482+0000 7f74e9d56640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f74d006b500 0x7f74d006d9c0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.482+0000 7f74e9d56640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e413f9b0 0x7f74e413fe30 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.482+0000 7f74e9d56640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74e4156d20 0x7f74e41571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.482+0000 7f74e9d56640 1 --2- 172.21.15.59:0/1368669732 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74cc004800 0x7f74e413d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.482+0000 7f74e9d56640 1 -- 172.21.15.59:0/1368669732 >> 172.21.15.59:0/1368669732 conn(0x7f74e40cf8a0 msgr2=0x7f74e40cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.482+0000 7f74e9d56640 1 -- 172.21.15.59:0/1368669732 shutdown_connections 2024-03-06T07:34:31.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.482+0000 7f74e9d56640 1 -- 172.21.15.59:0/1368669732 wait complete. 2024-03-06T07:34:31.495 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-03-06T07:34:31.477021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":331508,"kb_used_data":15496,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643212,"statfs":{"total":773094113280,"available":772754649088,"internally_reserved":0,"allocated":15867904,"data_stored":7620120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36544,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-144,"num_write_kb":-86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001294"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":401,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:34:29.332658+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:34:29.332658+0000","last_peered":"2024-03-06T07:34:29.332658+0000","last_clean":"2024-03-06T07:34:29.332658+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:34:29.332658+0000","last_undegraded":"2024-03-06T07:34:29.332658+0000","last_fullsized":"2024-03-06T07:34:29.332658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575005,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41872,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329968,"statfs":{"total":96636764160,"available":96593887232,"internally_reserved":0,"allocated":2428928,"data_stored":1395291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575005,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575005,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41880,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329960,"statfs":{"total":96636764160,"available":96593879040,"internally_reserved":0,"allocated":2428928,"data_stored":1395291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:31.495 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:31.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.595+0000 7f1ef40c2640 1 Processor -- start 2024-03-06T07:34:31.597 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.595+0000 7f1ef40c2640 1 -- start start 2024-03-06T07:34:31.598 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef40c2640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1eec06b220 0x7f1eec15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef40c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eec06b600 0x7f1eec162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef40c2640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec15ea30 con 0x7f1eec0ce920 2024-03-06T07:34:31.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef40c2640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec161b70 con 0x7f1eec06b220 2024-03-06T07:34:31.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef40c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec15ed10 con 0x7f1eec06b600 2024-03-06T07:34:31.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef1e5e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1eec06b220 0x7f1eec15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef165d640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1eec0ce920 0x7f1eec071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35692/0 (socket says 172.21.15.59:35692) 2024-03-06T07:34:31.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef165d640 1 -- 172.21.15.59:0/120672689 learned_addr learned my addr 172.21.15.59:0/120672689 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:31.599 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef1e5e640 1 -- 172.21.15.59:0/120672689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eec06b600 msgr2=0x7f1eec162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:31.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef265f640 1 --2- 172.21.15.59:0/120672689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eec06b600 0x7f1eec162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef1e5e640 1 --2- 172.21.15.59:0/120672689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eec06b600 0x7f1eec162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef1e5e640 1 -- 172.21.15.59:0/120672689 >> v1:172.21.15.59:6789/0 conn(0x7f1eec0ce920 legacy=0x7f1eec071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef0e5c640 1 -- 172.21.15.59:0/120672689 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818538201 0 0) 0x7f1eec15ea30 con 0x7f1eec0ce920 2024-03-06T07:34:31.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef1e5e640 1 -- 172.21.15.59:0/120672689 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec15eff0 con 0x7f1eec06b220 2024-03-06T07:34:31.600 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.596+0000 7f1ef265f640 1 --2- 172.21.15.59:0/120672689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eec06b600 0x7f1eec162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:31.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.597+0000 7f1ef1e5e640 1 --2- 172.21.15.59:0/120672689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eec06b220 0x7f1eec15e450 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40097a0 tx=0x7f1ee402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8156a26ab8674e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:31.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.597+0000 7f1ef0e5c640 1 -- 172.21.15.59:0/120672689 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee40315a0 con 0x7f1eec06b220 2024-03-06T07:34:31.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.597+0000 7f1ef0e5c640 1 -- 172.21.15.59:0/120672689 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee4037070 con 0x7f1eec06b220 2024-03-06T07:34:31.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.597+0000 7f1ef0e5c640 1 -- 172.21.15.59:0/120672689 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4031960 con 0x7f1eec06b220 2024-03-06T07:34:31.604 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.598+0000 7f1ef40c2640 1 -- 172.21.15.59:0/120672689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eec06b220 msgr2=0x7f1eec15e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.598+0000 7f1ef40c2640 1 --2- 172.21.15.59:0/120672689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eec06b220 0x7f1eec15e450 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40097a0 tx=0x7f1ee402b840 comp rx=0 tx=0).stop 2024-03-06T07:34:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.598+0000 7f1ef40c2640 1 -- 172.21.15.59:0/120672689 shutdown_connections 2024-03-06T07:34:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.598+0000 7f1ef40c2640 1 --2- 172.21.15.59:0/120672689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eec06b600 0x7f1eec162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.598+0000 7f1ef40c2640 1 --2- 172.21.15.59:0/120672689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eec06b220 0x7f1eec15e450 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.598+0000 7f1ef40c2640 1 -- 172.21.15.59:0/120672689 >> 172.21.15.59:0/120672689 conn(0x7f1eec06c0c0 msgr2=0x7f1eec06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.598+0000 7f1ef40c2640 1 -- 172.21.15.59:0/120672689 shutdown_connections 2024-03-06T07:34:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.598+0000 7f1ef40c2640 1 -- 172.21.15.59:0/120672689 wait complete. 2024-03-06T07:34:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.599+0000 7f1ef40c2640 1 Processor -- start 2024-03-06T07:34:31.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.599+0000 7f1ef40c2640 1 -- start start 2024-03-06T07:34:31.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef40c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1eec06b220 0x7f1eec153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef40c2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eec06b600 0x7f1eec156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef40c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eec0ce920 0x7f1eec06efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef40c2640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec072090 con 0x7f1eec06b600 2024-03-06T07:34:31.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef40c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec0ce200 con 0x7f1eec06b220 2024-03-06T07:34:31.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef40c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec158c40 con 0x7f1eec0ce920 2024-03-06T07:34:31.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef165d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eec06b600 0x7f1eec156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.606 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef165d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eec06b600 0x7f1eec156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50796/0 (socket says 172.21.15.59:50796) 2024-03-06T07:34:31.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef165d640 1 -- 172.21.15.59:0/1589841205 learned_addr learned my addr 172.21.15.59:0/1589841205 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:31.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef265f640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eec0ce920 0x7f1eec06efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef165d640 1 -- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eec0ce920 msgr2=0x7f1eec06efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef1e5e640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1eec06b220 0x7f1eec153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef165d640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eec0ce920 0x7f1eec06efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef165d640 1 -- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1eec06b220 msgr2=0x7f1eec153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef165d640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1eec06b220 0x7f1eec153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.607 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef165d640 1 -- 172.21.15.59:0/1589841205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec158f20 con 0x7f1eec06b600 2024-03-06T07:34:31.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef165d640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eec06b600 0x7f1eec156d10 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f1ee000c400 tx=0x7f1ee000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:31.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1eea7fc640 1 -- 172.21.15.59:0/1589841205 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee0017070 con 0x7f1eec06b600 2024-03-06T07:34:31.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1eea7fc640 1 -- 172.21.15.59:0/1589841205 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee0005ce0 con 0x7f1eec06b600 2024-03-06T07:34:31.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1ef40c2640 1 -- 172.21.15.59:0/1589841205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec159200 con 0x7f1eec06b600 2024-03-06T07:34:31.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.600+0000 7f1eea7fc640 1 -- 172.21.15.59:0/1589841205 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee0007310 con 0x7f1eec06b600 2024-03-06T07:34:31.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.601+0000 7f1ef40c2640 1 -- 172.21.15.59:0/1589841205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec06f510 con 0x7f1eec06b600 2024-03-06T07:34:31.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.601+0000 7f1ef40c2640 1 -- 172.21.15.59:0/1589841205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eec04f330 con 0x7f1eec06b600 2024-03-06T07:34:31.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.602+0000 7f1eea7fc640 1 -- 172.21.15.59:0/1589841205 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ee001a070 con 0x7f1eec06b600 2024-03-06T07:34:31.608 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.602+0000 7f1eea7fc640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1ed806b4d0 0x7f1ed806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.602+0000 7f1eea7fc640 1 -- 172.21.15.59:0/1589841205 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1ee0089390 con 0x7f1eec06b600 2024-03-06T07:34:31.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.602+0000 7f1ef1e5e640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1ed806b4d0 0x7f1ed806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.603+0000 7f1ef1e5e640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1ed806b4d0 0x7f1ed806d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4036000 tx=0x7f1ee4008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:31.609 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.606+0000 7f1eea7fc640 1 -- 172.21.15.59:0/1589841205 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ee00584b0 con 0x7f1eec06b600 2024-03-06T07:34:31.789 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.787+0000 7f1ef40c2640 1 -- 172.21.15.59:0/1589841205 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1eec0cdf00 con 0x7f1ed806b4d0 2024-03-06T07:34:31.790 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.788+0000 7f1eea7fc640 1 -- 172.21.15.59:0/1589841205 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13503 (secure 0 0 0) 0x7f1eec0cdf00 con 0x7f1ed806b4d0 2024-03-06T07:34:31.791 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:31.791 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:34:31.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 -- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1ed806b4d0 msgr2=0x7f1ed806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1ed806b4d0 0x7f1ed806d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4036000 tx=0x7f1ee4008030 comp rx=0 tx=0).stop 2024-03-06T07:34:31.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 -- 172.21.15.59:0/1589841205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eec06b600 msgr2=0x7f1eec156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eec06b600 0x7f1eec156d10 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f1ee000c400 tx=0x7f1ee000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 -- 172.21.15.59:0/1589841205 shutdown_connections 2024-03-06T07:34:31.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f1ed806b4d0 0x7f1ed806d990 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eec0ce920 0x7f1eec06efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eec06b600 0x7f1eec156d10 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 --2- 172.21.15.59:0/1589841205 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1eec06b220 0x7f1eec153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 -- 172.21.15.59:0/1589841205 >> 172.21.15.59:0/1589841205 conn(0x7f1eec06c0c0 msgr2=0x7f1eec06e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:31.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 -- 172.21.15.59:0/1589841205 shutdown_connections 2024-03-06T07:34:31.794 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.791+0000 7f1ef40c2640 1 -- 172.21.15.59:0/1589841205 wait complete. 2024-03-06T07:34:31.803 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-03-06T07:34:31.477021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":331508,"kb_used_data":15496,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643212,"statfs":{"total":773094113280,"available":772754649088,"internally_reserved":0,"allocated":15867904,"data_stored":7620120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36544,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-144,"num_write_kb":-86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001294"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":401,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:34:29.332658+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:34:29.332658+0000","last_peered":"2024-03-06T07:34:29.332658+0000","last_clean":"2024-03-06T07:34:29.332658+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:34:29.332658+0000","last_undegraded":"2024-03-06T07:34:29.332658+0000","last_fullsized":"2024-03-06T07:34:29.332658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575005,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41872,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329968,"statfs":{"total":96636764160,"available":96593887232,"internally_reserved":0,"allocated":2428928,"data_stored":1395291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575005,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575005,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41880,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329960,"statfs":{"total":96636764160,"available":96593879040,"internally_reserved":0,"allocated":2428928,"data_stored":1395291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:31.804 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:34:31.804 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:31.905 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.903+0000 7fb1df464640 1 Processor -- start 2024-03-06T07:34:31.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.904+0000 7fb1df464640 1 -- start start 2024-03-06T07:34:31.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.904+0000 7fb1df464640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb1d815dfe0 0x7fb1d80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.904+0000 7fb1df464640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb1d80d29c0 0x7fb1d80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.904+0000 7fb1df464640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d80b3430 con 0x7fb1d80cf830 2024-03-06T07:34:31.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.904+0000 7fb1df464640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d80b31e0 con 0x7fb1d815dfe0 2024-03-06T07:34:31.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.904+0000 7fb1df464640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d80b3d60 con 0x7fb1d80d29c0 2024-03-06T07:34:31.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.905+0000 7fb1dd200640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb1d815dfe0 0x7fb1d80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.905+0000 7fb1dd200640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb1d815dfe0 0x7fb1d80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50802/0 (socket says 172.21.15.59:50802) 2024-03-06T07:34:31.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.905+0000 7fb1dd200640 1 -- 172.21.15.59:0/2603053036 learned_addr learned my addr 172.21.15.59:0/2603053036 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:31.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.905+0000 7fb1dda01640 1 --2- 172.21.15.59:0/2603053036 >> v2:172.21.15.164:3300/0 conn(0x7fb1d80d29c0 0x7fb1d80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.905+0000 7fb1dd200640 1 -- 172.21.15.59:0/2603053036 >> v2:172.21.15.164:3300/0 conn(0x7fb1d80d29c0 msgr2=0x7fb1d80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:31.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.905+0000 7fb1dd200640 1 --2- 172.21.15.59:0/2603053036 >> v2:172.21.15.164:3300/0 conn(0x7fb1d80d29c0 0x7fb1d80b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.905+0000 7fb1dd200640 1 -- 172.21.15.59:0/2603053036 >> v1:172.21.15.164:6789/0 conn(0x7fb1d80cf830 legacy=0x7fb1d80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.905+0000 7fb1dd200640 1 -- 172.21.15.59:0/2603053036 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d80b3710 con 0x7fb1d815dfe0 2024-03-06T07:34:31.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.905+0000 7fb1dda01640 1 --2- 172.21.15.59:0/2603053036 >> v2:172.21.15.164:3300/0 conn(0x7fb1d80d29c0 0x7fb1d80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:31.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.906+0000 7fb1dd200640 1 --2- 172.21.15.59:0/2603053036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1d815dfe0 0x7fb1d80cf2f0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80b3cd0 tx=0x7fb1d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a609e89712383c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:31.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.906+0000 7fb1cffff640 1 -- 172.21.15.59:0/2603053036 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4034070 con 0x7fb1d815dfe0 2024-03-06T07:34:31.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.906+0000 7fb1cffff640 1 -- 172.21.15.59:0/2603053036 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d4030860 con 0x7fb1d815dfe0 2024-03-06T07:34:31.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.906+0000 7fb1cffff640 1 -- 172.21.15.59:0/2603053036 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4030c30 con 0x7fb1d815dfe0 2024-03-06T07:34:31.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.906+0000 7fb1df464640 1 -- 172.21.15.59:0/2603053036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1d815dfe0 msgr2=0x7fb1d80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.906+0000 7fb1df464640 1 --2- 172.21.15.59:0/2603053036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1d815dfe0 0x7fb1d80cf2f0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80b3cd0 tx=0x7fb1d402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.907+0000 7fb1df464640 1 -- 172.21.15.59:0/2603053036 shutdown_connections 2024-03-06T07:34:31.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.907+0000 7fb1df464640 1 --2- 172.21.15.59:0/2603053036 >> v2:172.21.15.164:3300/0 conn(0x7fb1d80d29c0 0x7fb1d80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.907+0000 7fb1df464640 1 --2- 172.21.15.59:0/2603053036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1d815dfe0 0x7fb1d80cf2f0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.907+0000 7fb1df464640 1 -- 172.21.15.59:0/2603053036 >> 172.21.15.59:0/2603053036 conn(0x7fb1d80d3cd0 msgr2=0x7fb1d80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:31.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.907+0000 7fb1df464640 1 -- 172.21.15.59:0/2603053036 shutdown_connections 2024-03-06T07:34:31.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.907+0000 7fb1df464640 1 -- 172.21.15.59:0/2603053036 wait complete. 2024-03-06T07:34:31.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.908+0000 7fb1df464640 1 Processor -- start 2024-03-06T07:34:31.913 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.908+0000 7fb1df464640 1 -- start start 2024-03-06T07:34:31.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.908+0000 7fb1df464640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1d80cf830 0x7fb1d80cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.908+0000 7fb1df464640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1d80d29c0 0x7fb1d8158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1df464640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1d815dfe0 0x7fb1d81f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1df464640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d80d1df0 con 0x7fb1d815dfe0 2024-03-06T07:34:31.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1df464640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d80d1c70 con 0x7fb1d80d29c0 2024-03-06T07:34:31.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1df464640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d815e3c0 con 0x7fb1d80cf830 2024-03-06T07:34:31.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1dda01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1d815dfe0 0x7fb1d81f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1dda01640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1d815dfe0 0x7fb1d81f70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50804/0 (socket says 172.21.15.59:50804) 2024-03-06T07:34:31.914 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1dda01640 1 -- 172.21.15.59:0/1794375731 learned_addr learned my addr 172.21.15.59:0/1794375731 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1dd200640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1d80cf830 0x7fb1d80cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1dc9ff640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1d80d29c0 0x7fb1d8158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1dda01640 1 -- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1d80cf830 msgr2=0x7fb1d80cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1dda01640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1d80cf830 0x7fb1d80cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1dda01640 1 -- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1d80d29c0 msgr2=0x7fb1d8158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1dda01640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1d80d29c0 0x7fb1d8158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:31.915 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1dda01640 1 -- 172.21.15.59:0/1794375731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d81f3ae0 con 0x7fb1d815dfe0 2024-03-06T07:34:31.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1dda01640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1d815dfe0 0x7fb1d81f70d0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000ce80 tx=0x7fb1d000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:31.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1cdffb640 1 -- 172.21.15.59:0/1794375731 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d001a070 con 0x7fb1d815dfe0 2024-03-06T07:34:31.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1cdffb640 1 -- 172.21.15.59:0/1794375731 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d000d880 con 0x7fb1d815dfe0 2024-03-06T07:34:31.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1cdffb640 1 -- 172.21.15.59:0/1794375731 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d000dc70 con 0x7fb1d815dfe0 2024-03-06T07:34:31.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.909+0000 7fb1df464640 1 -- 172.21.15.59:0/1794375731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d81f7610 con 0x7fb1d815dfe0 2024-03-06T07:34:31.916 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.910+0000 7fb1df464640 1 -- 172.21.15.59:0/1794375731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d81f7b90 con 0x7fb1d815dfe0 2024-03-06T07:34:31.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.911+0000 7fb1df464640 1 -- 172.21.15.59:0/1794375731 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d804f330 con 0x7fb1d815dfe0 2024-03-06T07:34:31.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.911+0000 7fb1cdffb640 1 -- 172.21.15.59:0/1794375731 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1d001d070 con 0x7fb1d815dfe0 2024-03-06T07:34:31.917 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.911+0000 7fb1cdffb640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1bc06b500 0x7fb1bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:31.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.911+0000 7fb1cdffb640 1 -- 172.21.15.59:0/1794375731 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb1d008a620 con 0x7fb1d815dfe0 2024-03-06T07:34:31.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.911+0000 7fb1dd200640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1bc06b500 0x7fb1bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:31.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.912+0000 7fb1dd200640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1bc06b500 0x7fb1bc06d9c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4036000 tx=0x7fb1d403b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:31.918 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:31.915+0000 7fb1cdffb640 1 -- 172.21.15.59:0/1794375731 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1d81f3ae0 con 0x7fb1d815dfe0 2024-03-06T07:34:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.092+0000 7fb1df464640 1 -- 172.21.15.59:0/1794375731 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1d8158740 con 0x7fb1bc06b500 2024-03-06T07:34:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.093+0000 7fb1cdffb640 1 -- 172.21.15.59:0/1794375731 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13503 (secure 0 0 0) 0x7fb1d8158740 con 0x7fb1bc06b500 2024-03-06T07:34:32.095 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:32.095 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:34:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.094+0000 7fb1df464640 1 -- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1bc06b500 msgr2=0x7fb1bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.094+0000 7fb1df464640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1bc06b500 0x7fb1bc06d9c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4036000 tx=0x7fb1d403b040 comp rx=0 tx=0).stop 2024-03-06T07:34:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.094+0000 7fb1df464640 1 -- 172.21.15.59:0/1794375731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1d815dfe0 msgr2=0x7fb1d81f70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.095+0000 7fb1df464640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1d815dfe0 0x7fb1d81f70d0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000ce80 tx=0x7fb1d000a710 comp rx=0 tx=0).stop 2024-03-06T07:34:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.095+0000 7fb1df464640 1 -- 172.21.15.59:0/1794375731 shutdown_connections 2024-03-06T07:34:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.095+0000 7fb1df464640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fb1bc06b500 0x7fb1bc06d9c0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.095+0000 7fb1df464640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb1d815dfe0 0x7fb1d81f70d0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.097 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.095+0000 7fb1df464640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1d80d29c0 0x7fb1d8158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.095+0000 7fb1df464640 1 --2- 172.21.15.59:0/1794375731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1d80cf830 0x7fb1d80cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.095+0000 7fb1df464640 1 -- 172.21.15.59:0/1794375731 >> 172.21.15.59:0/1794375731 conn(0x7fb1d80d3cd0 msgr2=0x7fb1d806ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.095+0000 7fb1df464640 1 -- 172.21.15.59:0/1794375731 shutdown_connections 2024-03-06T07:34:32.098 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.095+0000 7fb1df464640 1 -- 172.21.15.59:0/1794375731 wait complete. 2024-03-06T07:34:32.107 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-03-06T07:34:31.477021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":331508,"kb_used_data":15496,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643212,"statfs":{"total":773094113280,"available":772754649088,"internally_reserved":0,"allocated":15867904,"data_stored":7620120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36544,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-144,"num_write_kb":-86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001294"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":401,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:34:29.332658+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:34:29.332658+0000","last_peered":"2024-03-06T07:34:29.332658+0000","last_clean":"2024-03-06T07:34:29.332658+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:34:29.332658+0000","last_undegraded":"2024-03-06T07:34:29.332658+0000","last_fullsized":"2024-03-06T07:34:29.332658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575005,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41872,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329968,"statfs":{"total":96636764160,"available":96593887232,"internally_reserved":0,"allocated":2428928,"data_stored":1395291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575005,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575005,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41880,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329960,"statfs":{"total":96636764160,"available":96593879040,"internally_reserved":0,"allocated":2428928,"data_stored":1395291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:32.108 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:34:32.209 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.208+0000 7fa3cff6c640 1 Processor -- start 2024-03-06T07:34:32.210 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.208+0000 7fa3cff6c640 1 -- start start 2024-03-06T07:34:32.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3cff6c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa3c80d00c0 0x7fa3c80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3cff6c640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c80d09e0 con 0x7fa3c806cf20 2024-03-06T07:34:32.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3cff6c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c815e380 con 0x7fa3c80cf850 2024-03-06T07:34:32.213 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3cff6c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c806ca90 con 0x7fa3c80d00c0 2024-03-06T07:34:32.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ce509640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa3c806cf20 0x7fa3c806d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35698/0 (socket says 172.21.15.59:35698) 2024-03-06T07:34:32.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ce509640 1 -- 172.21.15.59:0/1293313905 learned_addr learned my addr 172.21.15.59:0/1293313905 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:32.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3cdd08640 1 --2- 172.21.15.59:0/1293313905 >> v2:172.21.15.164:3300/0 conn(0x7fa3c80d00c0 0x7fa3c80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034223624 0 0) 0x7fa3c80d09e0 con 0x7fa3c806cf20 2024-03-06T07:34:32.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b4003520 con 0x7fa3c806cf20 2024-03-06T07:34:32.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2885065650 0 0) 0x7fa3b4003520 con 0x7fa3c806cf20 2024-03-06T07:34:32.214 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 >> v2:172.21.15.164:3300/0 conn(0x7fa3c80d00c0 msgr2=0x7fa3c80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:32.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 --2- 172.21.15.59:0/1293313905 >> v2:172.21.15.164:3300/0 conn(0x7fa3c80d00c0 0x7fa3c80d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 >> v1:172.21.15.164:6789/0 conn(0x7fa3c80cf850 legacy=0x7fa3c80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c80cdc10 con 0x7fa3c806cf20 2024-03-06T07:34:32.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fa3c00019f0 con 0x7fa3c806cf20 2024-03-06T07:34:32.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 >> v1:172.21.15.59:6789/0 conn(0x7fa3c806cf20 legacy=0x7fa3c806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 --2- 172.21.15.59:0/1293313905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3b40039b0 0x7fa3b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 --2- 172.21.15.59:0/1293313905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3b4004800 0x7fa3b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 --2- 172.21.15.59:0/1293313905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3b40051c0 0x7fa3b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.215 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c806ca90 con 0x7fa3b40039b0 2024-03-06T07:34:32.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b400c060 con 0x7fa3b4004800 2024-03-06T07:34:32.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b400c260 con 0x7fa3b40051c0 2024-03-06T07:34:32.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.209+0000 7fa3cdd08640 1 --2- 172.21.15.59:0/1293313905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3b40039b0 0x7fa3b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.210+0000 7fa3cd507640 1 --2- 172.21.15.59:0/1293313905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3b4004800 0x7fa3b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.210+0000 7fa3ce509640 1 --2- 172.21.15.59:0/1293313905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3b40051c0 0x7fa3b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.210+0000 7fa3cdd08640 1 -- 172.21.15.59:0/1293313905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3b40051c0 msgr2=0x7fa3b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:32.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.210+0000 7fa3cdd08640 1 --2- 172.21.15.59:0/1293313905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3b40051c0 0x7fa3b400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.216 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.210+0000 7fa3cdd08640 1 -- 172.21.15.59:0/1293313905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3b4004800 msgr2=0x7fa3b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:32.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.210+0000 7fa3cdd08640 1 --2- 172.21.15.59:0/1293313905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3b4004800 0x7fa3b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.210+0000 7fa3cdd08640 1 -- 172.21.15.59:0/1293313905 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b4003520 con 0x7fa3b40039b0 2024-03-06T07:34:32.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.210+0000 7fa3ce509640 1 -- 172.21.15.59:0/1293313905 reap_dead start 2024-03-06T07:34:32.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.210+0000 7fa3cdd08640 1 --2- 172.21.15.59:0/1293313905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3b40039b0 0x7fa3b4003dd0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4009ac0 tx=0x7fa3b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b050bbaec7638fa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:32.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.211+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4039070 con 0x7fa3b40039b0 2024-03-06T07:34:32.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.211+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c402c6d0 con 0x7fa3b40039b0 2024-03-06T07:34:32.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.211+0000 7fa3ccd06640 1 -- 172.21.15.59:0/1293313905 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4009690 con 0x7fa3b40039b0 2024-03-06T07:34:32.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.211+0000 7fa3cff6c640 1 -- 172.21.15.59:0/1293313905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3b40039b0 msgr2=0x7fa3b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.217 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.211+0000 7fa3cff6c640 1 --2- 172.21.15.59:0/1293313905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3b40039b0 0x7fa3b4003dd0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4009ac0 tx=0x7fa3b4005830 comp rx=0 tx=0).stop 2024-03-06T07:34:32.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.211+0000 7fa3cff6c640 1 -- 172.21.15.59:0/1293313905 shutdown_connections 2024-03-06T07:34:32.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.211+0000 7fa3cff6c640 1 --2- 172.21.15.59:0/1293313905 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3b40039b0 0x7fa3b4003dd0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.211+0000 7fa3cff6c640 1 -- 172.21.15.59:0/1293313905 >> 172.21.15.59:0/1293313905 conn(0x7fa3c80b35a0 msgr2=0x7fa3c80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:32.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.211+0000 7fa3cff6c640 1 -- 172.21.15.59:0/1293313905 shutdown_connections 2024-03-06T07:34:32.218 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.211+0000 7fa3cff6c640 1 -- 172.21.15.59:0/1293313905 wait complete. 2024-03-06T07:34:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.212+0000 7fa3cff6c640 1 Processor -- start 2024-03-06T07:34:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.212+0000 7fa3cff6c640 1 -- start start 2024-03-06T07:34:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.213+0000 7fa3cff6c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3b40039b0 0x7fa3c81f7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.213+0000 7fa3cff6c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3c815c6d0 0x7fa3c80c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.213+0000 7fa3cff6c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3c815cb50 0x7fa3c80c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.213+0000 7fa3cff6c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c806c720 con 0x7fa3c815cb50 2024-03-06T07:34:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.213+0000 7fa3cff6c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c806c440 con 0x7fa3c815c6d0 2024-03-06T07:34:32.221 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.213+0000 7fa3cff6c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80b1030 con 0x7fa3b40039b0 2024-03-06T07:34:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.213+0000 7fa3ce509640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3c815cb50 0x7fa3c80c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.213+0000 7fa3ce509640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3c815cb50 0x7fa3c80c75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50826/0 (socket says 172.21.15.59:50826) 2024-03-06T07:34:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.213+0000 7fa3ce509640 1 -- 172.21.15.59:0/710362623 learned_addr learned my addr 172.21.15.59:0/710362623 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.213+0000 7fa3cdd08640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3b40039b0 0x7fa3c81f7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.213+0000 7fa3cd507640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3c815c6d0 0x7fa3c80c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.214+0000 7fa3ce509640 1 -- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3b40039b0 msgr2=0x7fa3c81f7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.214+0000 7fa3ce509640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3b40039b0 0x7fa3c81f7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.222 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.214+0000 7fa3ce509640 1 -- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3c815c6d0 msgr2=0x7fa3c80c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.214+0000 7fa3ce509640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3c815c6d0 0x7fa3c80c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.214+0000 7fa3ce509640 1 -- 172.21.15.59:0/710362623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c80c8e90 con 0x7fa3c815cb50 2024-03-06T07:34:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.214+0000 7fa3ce509640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3c815cb50 0x7fa3c80c75e0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8071f50 tx=0x7fa3c0007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.215+0000 7fa3be7fc640 1 -- 172.21.15.59:0/710362623 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c000ea80 con 0x7fa3c815cb50 2024-03-06T07:34:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.215+0000 7fa3be7fc640 1 -- 172.21.15.59:0/710362623 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c000ec20 con 0x7fa3c815cb50 2024-03-06T07:34:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.215+0000 7fa3cff6c640 1 -- 172.21.15.59:0/710362623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c80c9170 con 0x7fa3c815cb50 2024-03-06T07:34:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.215+0000 7fa3be7fc640 1 -- 172.21.15.59:0/710362623 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c000ee10 con 0x7fa3c815cb50 2024-03-06T07:34:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.215+0000 7fa3cff6c640 1 -- 172.21.15.59:0/710362623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c80c9730 con 0x7fa3c815cb50 2024-03-06T07:34:32.223 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.216+0000 7fa3cff6c640 1 -- 172.21.15.59:0/710362623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c80c9b10 con 0x7fa3c815cb50 2024-03-06T07:34:32.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.217+0000 7fa3be7fc640 1 -- 172.21.15.59:0/710362623 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3c001a020 con 0x7fa3c815cb50 2024-03-06T07:34:32.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.217+0000 7fa3be7fc640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ac06b4d0 0x7fa3ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:32.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.217+0000 7fa3be7fc640 1 -- 172.21.15.59:0/710362623 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fa3c0089810 con 0x7fa3c815cb50 2024-03-06T07:34:32.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.217+0000 7fa3cdd08640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ac06b4d0 0x7fa3ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:32.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.218+0000 7fa3cdd08640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ac06b4d0 0x7fa3ac06d990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4036000 tx=0x7fa3c4030000 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:32.224 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.221+0000 7fa3be7fc640 1 -- 172.21.15.59:0/710362623 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3c0021020 con 0x7fa3c815cb50 2024-03-06T07:34:32.406 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.405+0000 7fa3cff6c640 1 -- 172.21.15.59:0/710362623 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3c804f330 con 0x7fa3c815cb50 2024-03-06T07:34:32.407 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.406+0000 7fa3be7fc640 1 -- 172.21.15.59:0/710362623 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+13146 (secure 0 0 0) 0x7fa3c00582f0 con 0x7fa3c815cb50 2024-03-06T07:34:32.408 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:32.408 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":146,"fsid":"43c1ee46-a208-44bd-a1f3-57a84ef7ed13","created":"2024-03-06T07:21:30.281894+0000","modified":"2024-03-06T07:34:29.203555+0000","last_up_change":"2024-03-06T07:21:41.200420+0000","last_in_change":"2024-03-06T07:21:32.569300+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:41.343905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5677b231-bdb9-45ae-8058-3b283377d567","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6825","nonce":841983189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6827","nonce":841983189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6830","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6831","nonce":841983189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":841983189},{"type":"v1","addr":"172.21.15.59:6829","nonce":841983189}]},"public_addr":"172.21.15.59:6825/841983189","cluster_addr":"172.21.15.59:6827/841983189","heartbeat_back_addr":"172.21.15.59:6831/841983189","heartbeat_front_addr":"172.21.15.59:6829/841983189","state":["exists","up"]},{"osd":1,"uuid":"e04c5ccb-765c-4542-b113-8960735735bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6817","nonce":1773892168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6819","nonce":1773892168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6823","nonce":1773892168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":1773892168},{"type":"v1","addr":"172.21.15.59:6821","nonce":1773892168}]},"public_addr":"172.21.15.59:6817/1773892168","cluster_addr":"172.21.15.59:6819/1773892168","heartbeat_back_addr":"172.21.15.59:6823/1773892168","heartbeat_front_addr":"172.21.15.59:6821/1773892168","state":["exists","up"]},{"osd":2,"uuid":"9cc1541e-9425-46a4-b873-738ca8ce77ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6801","nonce":1020996386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6803","nonce":1020996386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6807","nonce":1020996386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":1020996386},{"type":"v1","addr":"172.21.15.59:6805","nonce":1020996386}]},"public_addr":"172.21.15.59:6801/1020996386","cluster_addr":"172.21.15.59:6803/1020996386","heartbeat_back_addr":"172.21.15.59:6807/1020996386","heartbeat_front_addr":"172.21.15.59:6805/1020996386","state":["exists","up"]},{"osd":3,"uuid":"0a46fed5-501b-48ba-8269-929a8b3337a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":142,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6809","nonce":3602142643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6811","nonce":3602142643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6815","nonce":3602142643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":3602142643},{"type":"v1","addr":"172.21.15.59:6813","nonce":3602142643}]},"public_addr":"172.21.15.59:6809/3602142643","cluster_addr":"172.21.15.59:6811/3602142643","heartbeat_back_addr":"172.21.15.59:6815/3602142643","heartbeat_front_addr":"172.21.15.59:6813/3602142643","state":["exists","up"]},{"osd":4,"uuid":"d1c273be-2527-465c-bef8-be9289c8c326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6820","nonce":102497754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6824","nonce":102497754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6831","nonce":102497754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":102497754},{"type":"v1","addr":"172.21.15.164:6828","nonce":102497754}]},"public_addr":"172.21.15.164:6820/102497754","cluster_addr":"172.21.15.164:6824/102497754","heartbeat_back_addr":"172.21.15.164:6831/102497754","heartbeat_front_addr":"172.21.15.164:6828/102497754","state":["exists","up"]},{"osd":5,"uuid":"e0eb5922-8339-4154-92be-772acae72ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6801","nonce":2889679248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6803","nonce":2889679248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6807","nonce":2889679248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2889679248},{"type":"v1","addr":"172.21.15.164:6805","nonce":2889679248}]},"public_addr":"172.21.15.164:6801/2889679248","cluster_addr":"172.21.15.164:6803/2889679248","heartbeat_back_addr":"172.21.15.164:6807/2889679248","heartbeat_front_addr":"172.21.15.164:6805/2889679248","state":["exists","up"]},{"osd":6,"uuid":"dc9c85db-f2f3-4796-b0dc-a3c7baa40d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6817","nonce":1594998822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6821","nonce":1594998822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6827","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6829","nonce":1594998822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":1594998822},{"type":"v1","addr":"172.21.15.164:6825","nonce":1594998822}]},"public_addr":"172.21.15.164:6817/1594998822","cluster_addr":"172.21.15.164:6821/1594998822","heartbeat_back_addr":"172.21.15.164:6829/1594998822","heartbeat_front_addr":"172.21.15.164:6825/1594998822","state":["exists","up"]},{"osd":7,"uuid":"8f485029-f8fc-4491-a820-12cbbd3b8d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6809","nonce":3612340295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6811","nonce":3612340295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6815","nonce":3612340295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3612340295},{"type":"v1","addr":"172.21.15.164:6813","nonce":3612340295}]},"public_addr":"172.21.15.164:6809/3612340295","cluster_addr":"172.21.15.164:6811/3612340295","heartbeat_back_addr":"172.21.15.164:6815/3612340295","heartbeat_front_addr":"172.21.15.164:6813/3612340295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.759321+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.672332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.618763+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.739323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.817110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.652502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.792241+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:37.778511+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:6837/997138956":"2024-03-07T07:34:26.195639+0000","172.21.15.164:6835/997138956":"2024-03-07T07:34:26.195639+0000","172.21.15.59:6833/720863754":"2024-03-07T07:34:26.195650+0000","172.21.15.59:6832/720863754":"2024-03-07T07:34:26.195650+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:32.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.407+0000 7fa3cff6c640 1 -- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ac06b4d0 msgr2=0x7fa3ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.407+0000 7fa3cff6c640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ac06b4d0 0x7fa3ac06d990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4036000 tx=0x7fa3c4030000 comp rx=0 tx=0).stop 2024-03-06T07:34:32.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.407+0000 7fa3cff6c640 1 -- 172.21.15.59:0/710362623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3c815cb50 msgr2=0x7fa3c80c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:32.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.407+0000 7fa3cff6c640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3c815cb50 0x7fa3c80c75e0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8071f50 tx=0x7fa3c0007bc0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.408+0000 7fa3cff6c640 1 -- 172.21.15.59:0/710362623 shutdown_connections 2024-03-06T07:34:32.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.408+0000 7fa3cff6c640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa3ac06b4d0 0x7fa3ac06d990 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.410 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.408+0000 7fa3cff6c640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3c815cb50 0x7fa3c80c75e0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.408+0000 7fa3cff6c640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3c815c6d0 0x7fa3c80c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.408+0000 7fa3cff6c640 1 --2- 172.21.15.59:0/710362623 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3b40039b0 0x7fa3c81f7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.408+0000 7fa3cff6c640 1 -- 172.21.15.59:0/710362623 >> 172.21.15.59:0/710362623 conn(0x7fa3c80b35a0 msgr2=0x7fa3c80cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.408+0000 7fa3cff6c640 1 -- 172.21.15.59:0/710362623 shutdown_connections 2024-03-06T07:34:32.411 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:32.408+0000 7fa3cff6c640 1 -- 172.21.15.59:0/710362623 wait complete. 2024-03-06T07:34:33.422 INFO:tasks.ceph:Scrubbing osd.0 2024-03-06T07:34:33.422 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:33.540 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.539+0000 7f7d6c5b5640 1 Processor -- start 2024-03-06T07:34:33.541 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.539+0000 7f7d6c5b5640 1 -- start start 2024-03-06T07:34:33.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d6c5b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d64162850 0x7f7d640d1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d6c5b5640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d640cf3c0 con 0x7f7d640d1f90 2024-03-06T07:34:33.543 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d6c5b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d640cf0e0 con 0x7f7d64162470 2024-03-06T07:34:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d6c5b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6406f270 con 0x7f7d64162850 2024-03-06T07:34:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d6ab52640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7d640d1f90 0x7f7d6406c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59838/0 (socket says 172.21.15.59:59838) 2024-03-06T07:34:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d6ab52640 1 -- 172.21.15.59:0/573662104 learned_addr learned my addr 172.21.15.59:0/573662104 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d69b50640 1 --2- 172.21.15.59:0/573662104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d64162850 0x7f7d640d1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d6934f640 1 -- 172.21.15.59:0/573662104 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640798494 0 0) 0x7f7d640cf3c0 con 0x7f7d640d1f90 2024-03-06T07:34:33.544 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d6934f640 1 -- 172.21.15.59:0/573662104 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d50003520 con 0x7f7d640d1f90 2024-03-06T07:34:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d6934f640 1 -- 172.21.15.59:0/573662104 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740239406 0 0) 0x7f7d640cf0e0 con 0x7f7d64162470 2024-03-06T07:34:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d6934f640 1 -- 172.21.15.59:0/573662104 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d640cf3c0 con 0x7f7d64162470 2024-03-06T07:34:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d69b50640 1 -- 172.21.15.59:0/573662104 >> v1:172.21.15.164:6789/0 conn(0x7f7d64162470 legacy=0x7f7d6406f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d69b50640 1 -- 172.21.15.59:0/573662104 >> v1:172.21.15.59:6789/0 conn(0x7f7d640d1f90 legacy=0x7f7d6406c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.540+0000 7f7d69b50640 1 -- 172.21.15.59:0/573662104 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6406ec20 con 0x7f7d64162850 2024-03-06T07:34:33.545 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.541+0000 7f7d69b50640 1 --2- 172.21.15.59:0/573662104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d64162850 0x7f7d640d1a50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7d540097d0 tx=0x7f7d5402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad9c07f30754dbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:33.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.541+0000 7f7d6934f640 1 -- 172.21.15.59:0/573662104 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54035020 con 0x7f7d64162850 2024-03-06T07:34:33.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.541+0000 7f7d6934f640 1 -- 172.21.15.59:0/573662104 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d54031890 con 0x7f7d64162850 2024-03-06T07:34:33.546 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.541+0000 7f7d6934f640 1 -- 172.21.15.59:0/573662104 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54031c60 con 0x7f7d64162850 2024-03-06T07:34:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.542+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/573662104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d64162850 msgr2=0x7f7d640d1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.542+0000 7f7d6c5b5640 1 --2- 172.21.15.59:0/573662104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d64162850 0x7f7d640d1a50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7d540097d0 tx=0x7f7d5402b930 comp rx=0 tx=0).stop 2024-03-06T07:34:33.550 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.542+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/573662104 shutdown_connections 2024-03-06T07:34:33.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.542+0000 7f7d6c5b5640 1 --2- 172.21.15.59:0/573662104 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d64162850 0x7f7d640d1a50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.542+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/573662104 >> 172.21.15.59:0/573662104 conn(0x7f7d6409a9f0 msgr2=0x7f7d6409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:33.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.542+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/573662104 shutdown_connections 2024-03-06T07:34:33.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.542+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/573662104 wait complete. 2024-03-06T07:34:33.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.543+0000 7f7d6c5b5640 1 Processor -- start 2024-03-06T07:34:33.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.543+0000 7f7d6c5b5640 1 -- start start 2024-03-06T07:34:33.551 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.544+0000 7f7d6c5b5640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d640d1f90 0x7f7d640c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.544+0000 7f7d6c5b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d64162470 0x7f7d641f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.544+0000 7f7d6c5b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d64162850 0x7f7d6415f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.544+0000 7f7d6c5b5640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6406ff90 con 0x7f7d640d1f90 2024-03-06T07:34:33.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.544+0000 7f7d6c5b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6406fe10 con 0x7f7d64162850 2024-03-06T07:34:33.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.544+0000 7f7d6c5b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6406cac0 con 0x7f7d64162470 2024-03-06T07:34:33.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.544+0000 7f7d6a351640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d640d1f90 0x7f7d640c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.552 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.544+0000 7f7d6a351640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d640d1f90 0x7f7d640c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56216/0 (socket says 172.21.15.59:56216) 2024-03-06T07:34:33.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.544+0000 7f7d6a351640 1 -- 172.21.15.59:0/2509109887 learned_addr learned my addr 172.21.15.59:0/2509109887 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:33.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.544+0000 7f7d69b50640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d64162470 0x7f7d641f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.544+0000 7f7d6ab52640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d64162850 0x7f7d6415f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d6a351640 1 -- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d64162470 msgr2=0x7f7d641f7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d6a351640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d64162470 0x7f7d641f7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d6a351640 1 -- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d64162850 msgr2=0x7f7d6415f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d6a351640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d64162850 0x7f7d6415f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.553 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d6a351640 1 -- 172.21.15.59:0/2509109887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6415f650 con 0x7f7d640d1f90 2024-03-06T07:34:33.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d6ab52640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d64162850 0x7f7d6415f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:33.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d6a351640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d640d1f90 0x7f7d640c5610 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000c500 tx=0x7f7d6000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:33.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d5affd640 1 -- 172.21.15.59:0/2509109887 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d600052d0 con 0x7f7d640d1f90 2024-03-06T07:34:33.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d5affd640 1 -- 172.21.15.59:0/2509109887 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d60005490 con 0x7f7d640d1f90 2024-03-06T07:34:33.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/2509109887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d640c5bf0 con 0x7f7d640d1f90 2024-03-06T07:34:33.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d5affd640 1 -- 172.21.15.59:0/2509109887 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60012870 con 0x7f7d640d1f90 2024-03-06T07:34:33.554 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.545+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/2509109887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d640c61b0 con 0x7f7d640d1f90 2024-03-06T07:34:33.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.546+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/2509109887 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7d6404f330 con 0x7f7d640d1f90 2024-03-06T07:34:33.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.547+0000 7f7d5affd640 1 -- 172.21.15.59:0/2509109887 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d60012a10 con 0x7f7d640d1f90 2024-03-06T07:34:33.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.547+0000 7f7d5affd640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d4806b4d0 0x7f7d4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.547+0000 7f7d5affd640 1 -- 172.21.15.59:0/2509109887 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f7d60089c50 con 0x7f7d640d1f90 2024-03-06T07:34:33.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.547+0000 7f7d5affd640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f7d48071430 0x7f7d48073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.547+0000 7f7d5affd640 1 -- 172.21.15.59:0/2509109887 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d48073f00 con 0x7f7d48071430 2024-03-06T07:34:33.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.547+0000 7f7d5affd640 1 -- 172.21.15.59:0/2509109887 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d6008c350 con 0x7f7d640d1f90 2024-03-06T07:34:33.555 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.547+0000 7f7d69b50640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d4806b4d0 0x7f7d4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.547+0000 7f7d6ab52640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f7d48071430 0x7f7d48073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.548+0000 7f7d6ab52640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f7d48071430 0x7f7d48073850 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:33.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.548+0000 7f7d69b50640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d4806b4d0 0x7f7d4806d990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7d54036000 tx=0x7f7d5403c040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:33.556 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.548+0000 7f7d5affd640 1 -- 172.21.15.59:0/2509109887 <== osd.0 v2:172.21.15.59:6824/841983189 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7d48073f00 con 0x7f7d48071430 2024-03-06T07:34:33.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.562+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/2509109887 --> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7d640c6670 con 0x7f7d48071430 2024-03-06T07:34:33.564 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.562+0000 7f7d5affd640 1 -- 172.21.15.59:0/2509109887 <== osd.0 v2:172.21.15.59:6824/841983189 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7d640c6670 con 0x7f7d48071430 2024-03-06T07:34:33.564 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:34:33.565 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:33.565 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:34:33.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.562+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/2509109887 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f7d48071430 msgr2=0x7f7d48073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.562+0000 7f7d6c5b5640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.59:6824/841983189,v1:172.21.15.59:6825/841983189] conn(0x7f7d48071430 0x7f7d48073850 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.562+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d4806b4d0 msgr2=0x7f7d4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.566 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.562+0000 7f7d6c5b5640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f7d4806b4d0 0x7f7d4806d990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f7d54036000 tx=0x7f7d5403c040 comp rx=0 tx=0).stop 2024-03-06T07:34:33.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.562+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/2509109887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d640d1f90 msgr2=0x7f7d640c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.562+0000 7f7d6c5b5640 1 --2- 172.21.15.59:0/2509109887 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7d640d1f90 0x7f7d640c5610 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000c500 tx=0x7f7d6000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.562+0000 7f7d6ab52640 1 -- 172.21.15.59:0/2509109887 reap_dead start 2024-03-06T07:34:33.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.562+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/2509109887 shutdown_connections 2024-03-06T07:34:33.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.562+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/2509109887 >> 172.21.15.59:0/2509109887 conn(0x7f7d6409a9f0 msgr2=0x7f7d64073f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:33.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.563+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/2509109887 shutdown_connections 2024-03-06T07:34:33.567 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.563+0000 7f7d6c5b5640 1 -- 172.21.15.59:0/2509109887 wait complete. 2024-03-06T07:34:33.575 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-06T07:34:33.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.676+0000 7f38cff3e640 1 Processor -- start 2024-03-06T07:34:33.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.677+0000 7f38cff3e640 1 -- start start 2024-03-06T07:34:33.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.677+0000 7f38cff3e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f38c80b4460 0x7f38c80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.677+0000 7f38cff3e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38c8161cf0 0x7f38c8162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.677+0000 7f38cff3e640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c80d22c0 con 0x7f38c8162970 2024-03-06T07:34:33.680 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.677+0000 7f38cff3e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c8161b00 con 0x7f38c8161cf0 2024-03-06T07:34:33.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.677+0000 7f38cff3e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c80cecc0 con 0x7f38c80b4460 2024-03-06T07:34:33.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.678+0000 7f38ce4db640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f38c8161cf0 0x7f38c8162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.678+0000 7f38cd4d9640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f38c8162970 0x7f38c80b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59852/0 (socket says 172.21.15.59:59852) 2024-03-06T07:34:33.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.678+0000 7f38cd4d9640 1 -- 172.21.15.59:0/3775813721 learned_addr learned my addr 172.21.15.59:0/3775813721 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:33.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.678+0000 7f38cdcda640 1 --2- 172.21.15.59:0/3775813721 >> v2:172.21.15.164:3300/0 conn(0x7f38c80b4460 0x7f38c80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.678+0000 7f38ce4db640 1 -- 172.21.15.59:0/3775813721 >> v2:172.21.15.164:3300/0 conn(0x7f38c80b4460 msgr2=0x7f38c80d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:33.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.678+0000 7f38ce4db640 1 --2- 172.21.15.59:0/3775813721 >> v2:172.21.15.164:3300/0 conn(0x7f38c80b4460 0x7f38c80d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.681 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.678+0000 7f38ce4db640 1 -- 172.21.15.59:0/3775813721 >> v1:172.21.15.59:6789/0 conn(0x7f38c8162970 legacy=0x7f38c80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.678+0000 7f38cccd8640 1 -- 172.21.15.59:0/3775813721 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654888797 0 0) 0x7f38c80d22c0 con 0x7f38c8162970 2024-03-06T07:34:33.682 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.678+0000 7f38ce4db640 1 -- 172.21.15.59:0/3775813721 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c80d25a0 con 0x7f38c8161cf0 2024-03-06T07:34:33.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.679+0000 7f38ce4db640 1 --2- 172.21.15.59:0/3775813721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38c8161cf0 0x7f38c8162180 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f38c808ddd0 tx=0x7f38c002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf5d6f7cc117b75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:33.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.679+0000 7f38cccd8640 1 -- 172.21.15.59:0/3775813721 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0039070 con 0x7f38c8161cf0 2024-03-06T07:34:33.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.679+0000 7f38cccd8640 1 -- 172.21.15.59:0/3775813721 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c002cc80 con 0x7f38c8161cf0 2024-03-06T07:34:33.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.679+0000 7f38cccd8640 1 -- 172.21.15.59:0/3775813721 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0034660 con 0x7f38c8161cf0 2024-03-06T07:34:33.683 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.680+0000 7f38cff3e640 1 -- 172.21.15.59:0/3775813721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38c8161cf0 msgr2=0x7f38c8162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.680+0000 7f38cff3e640 1 --2- 172.21.15.59:0/3775813721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38c8161cf0 0x7f38c8162180 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f38c808ddd0 tx=0x7f38c002bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.680+0000 7f38cff3e640 1 -- 172.21.15.59:0/3775813721 shutdown_connections 2024-03-06T07:34:33.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.680+0000 7f38cff3e640 1 --2- 172.21.15.59:0/3775813721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38c8161cf0 0x7f38c8162180 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.680+0000 7f38cff3e640 1 --2- 172.21.15.59:0/3775813721 >> v2:172.21.15.164:3300/0 conn(0x7f38c80b4460 0x7f38c80d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.680+0000 7f38cff3e640 1 -- 172.21.15.59:0/3775813721 >> 172.21.15.59:0/3775813721 conn(0x7f38c80d3ce0 msgr2=0x7f38c80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:33.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.680+0000 7f38cff3e640 1 -- 172.21.15.59:0/3775813721 shutdown_connections 2024-03-06T07:34:33.684 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.680+0000 7f38cff3e640 1 -- 172.21.15.59:0/3775813721 wait complete. 2024-03-06T07:34:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.682+0000 7f38cff3e640 1 Processor -- start 2024-03-06T07:34:33.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.682+0000 7f38cff3e640 1 -- start start 2024-03-06T07:34:33.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cff3e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38c80b4460 0x7f38c80b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cff3e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38c8161cf0 0x7f38c8154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cff3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38c8162970 0x7f38c80c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cff3e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c80ce950 con 0x7f38c80b4460 2024-03-06T07:34:33.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cff3e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c80d2b60 con 0x7f38c8162970 2024-03-06T07:34:33.687 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cff3e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c80b4840 con 0x7f38c8161cf0 2024-03-06T07:34:33.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cdcda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38c80b4460 0x7f38c80b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cdcda640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38c80b4460 0x7f38c80b7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56222/0 (socket says 172.21.15.59:56222) 2024-03-06T07:34:33.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cdcda640 1 -- 172.21.15.59:0/1869497191 learned_addr learned my addr 172.21.15.59:0/1869497191 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:33.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cd4d9640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38c8161cf0 0x7f38c8154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38ce4db640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38c8162970 0x7f38c80c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cdcda640 1 -- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38c8161cf0 msgr2=0x7f38c8154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.688 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cdcda640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38c8161cf0 0x7f38c8154db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cdcda640 1 -- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38c8162970 msgr2=0x7f38c80c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cdcda640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38c8162970 0x7f38c80c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.683+0000 7f38cdcda640 1 -- 172.21.15.59:0/1869497191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c80b6a30 con 0x7f38c80b4460 2024-03-06T07:34:33.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.684+0000 7f38cdcda640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38c80b4460 0x7f38c80b7ad0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f38c400e870 tx=0x7f38c400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:33.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.684+0000 7f38be7fc640 1 -- 172.21.15.59:0/1869497191 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c4017070 con 0x7f38c80b4460 2024-03-06T07:34:33.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.684+0000 7f38be7fc640 1 -- 172.21.15.59:0/1869497191 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c400c940 con 0x7f38c80b4460 2024-03-06T07:34:33.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.684+0000 7f38cff3e640 1 -- 172.21.15.59:0/1869497191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c80b6d10 con 0x7f38c80b4460 2024-03-06T07:34:33.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.684+0000 7f38be7fc640 1 -- 172.21.15.59:0/1869497191 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c400cd30 con 0x7f38c80b4460 2024-03-06T07:34:33.689 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.684+0000 7f38cff3e640 1 -- 172.21.15.59:0/1869497191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c80b72d0 con 0x7f38c80b4460 2024-03-06T07:34:33.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.685+0000 7f38cff3e640 1 -- 172.21.15.59:0/1869497191 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c804f330 con 0x7f38c80b4460 2024-03-06T07:34:33.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.685+0000 7f38be7fc640 1 -- 172.21.15.59:0/1869497191 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38c401a070 con 0x7f38c80b4460 2024-03-06T07:34:33.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.686+0000 7f38be7fc640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f38ac06b4d0 0x7f38ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.686+0000 7f38be7fc640 1 -- 172.21.15.59:0/1869497191 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f38c4089ce0 con 0x7f38c80b4460 2024-03-06T07:34:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.686+0000 7f38cd4d9640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f38ac06b4d0 0x7f38ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:33.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.687+0000 7f38cd4d9640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f38ac06b4d0 0x7f38ac06d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f38c8155a10 tx=0x7f38b8007d50 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:33.692 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.690+0000 7f38be7fc640 1 -- 172.21.15.59:0/1869497191 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38c4058840 con 0x7f38c80b4460 2024-03-06T07:34:33.877 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.875+0000 7f38cff3e640 1 -- 172.21.15.59:0/1869497191 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f38c806b180 con 0x7f38ac06b4d0 2024-03-06T07:34:33.878 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.876+0000 7f38be7fc640 1 -- 172.21.15.59:0/1869497191 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f38c806b180 con 0x7f38ac06b4d0 2024-03-06T07:34:33.878 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 0 to deep-scrub 2024-03-06T07:34:33.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 -- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f38ac06b4d0 msgr2=0x7f38ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.880 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f38ac06b4d0 0x7f38ac06d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f38c8155a10 tx=0x7f38b8007d50 comp rx=0 tx=0).stop 2024-03-06T07:34:33.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 -- 172.21.15.59:0/1869497191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38c80b4460 msgr2=0x7f38c80b7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:33.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38c80b4460 0x7f38c80b7ad0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f38c400e870 tx=0x7f38c400ed30 comp rx=0 tx=0).stop 2024-03-06T07:34:33.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 -- 172.21.15.59:0/1869497191 shutdown_connections 2024-03-06T07:34:33.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f38ac06b4d0 0x7f38ac06d990 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38c8162970 0x7f38c80c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38c8161cf0 0x7f38c8154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 --2- 172.21.15.59:0/1869497191 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38c80b4460 0x7f38c80b7ad0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:33.881 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 -- 172.21.15.59:0/1869497191 >> 172.21.15.59:0/1869497191 conn(0x7f38c80d3ce0 msgr2=0x7f38c80b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:33.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 -- 172.21.15.59:0/1869497191 shutdown_connections 2024-03-06T07:34:33.882 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.878+0000 7f38cff3e640 1 -- 172.21.15.59:0/1869497191 wait complete. 2024-03-06T07:34:33.890 INFO:tasks.ceph:Scrubbing osd.1 2024-03-06T07:34:33.891 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:33.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.995+0000 7fedf3637640 1 Processor -- start 2024-03-06T07:34:33.997 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.995+0000 7fedf3637640 1 -- start start 2024-03-06T07:34:33.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf3637640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fedec15ac00 0x7fedec06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf3637640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fedec070d10 0x7fedec15eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:33.999 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf3637640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedec0711c0 con 0x7fedec0708b0 2024-03-06T07:34:34.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf3637640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedec0cdc10 con 0x7fedec070d10 2024-03-06T07:34:34.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf3637640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedec15afe0 con 0x7fedec15ac00 2024-03-06T07:34:34.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf1bd4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fedec070d10 0x7fedec15eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf1bd4640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fedec070d10 0x7fedec15eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56228/0 (socket says 172.21.15.59:56228) 2024-03-06T07:34:34.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf1bd4640 1 -- 172.21.15.59:0/2272795022 learned_addr learned my addr 172.21.15.59:0/2272795022 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:34.000 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf13d3640 1 --2- 172.21.15.59:0/2272795022 >> v2:172.21.15.164:3300/0 conn(0x7fedec15ac00 0x7fedec06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf1bd4640 1 -- 172.21.15.59:0/2272795022 >> v2:172.21.15.164:3300/0 conn(0x7fedec15ac00 msgr2=0x7fedec06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:34.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf1bd4640 1 --2- 172.21.15.59:0/2272795022 >> v2:172.21.15.164:3300/0 conn(0x7fedec15ac00 0x7fedec06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf1bd4640 1 -- 172.21.15.59:0/2272795022 >> v1:172.21.15.164:6789/0 conn(0x7fedec0708b0 legacy=0x7fedec15e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.996+0000 7fedf1bd4640 1 -- 172.21.15.59:0/2272795022 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec0cd000 con 0x7fedec070d10 2024-03-06T07:34:34.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.997+0000 7fedf1bd4640 1 --2- 172.21.15.59:0/2272795022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedec070d10 0x7fedec15eeb0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fedec0730c0 tx=0x7fede402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211679c790d0ef68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.997+0000 7fede3fff640 1 -- 172.21.15.59:0/2272795022 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede402ca80 con 0x7fedec070d10 2024-03-06T07:34:34.001 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.997+0000 7fede3fff640 1 -- 172.21.15.59:0/2272795022 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede402cc20 con 0x7fedec070d10 2024-03-06T07:34:34.002 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.997+0000 7fede3fff640 1 -- 172.21.15.59:0/2272795022 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede40343f0 con 0x7fedec070d10 2024-03-06T07:34:34.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.998+0000 7fedf3637640 1 -- 172.21.15.59:0/2272795022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedec070d10 msgr2=0x7fedec15eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.998+0000 7fedf3637640 1 --2- 172.21.15.59:0/2272795022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedec070d10 0x7fedec15eeb0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fedec0730c0 tx=0x7fede402bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.998+0000 7fedf3637640 1 -- 172.21.15.59:0/2272795022 shutdown_connections 2024-03-06T07:34:34.006 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.998+0000 7fedf3637640 1 --2- 172.21.15.59:0/2272795022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedec070d10 0x7fedec15eeb0 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.998+0000 7fedf3637640 1 --2- 172.21.15.59:0/2272795022 >> v2:172.21.15.164:3300/0 conn(0x7fedec15ac00 0x7fedec06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.998+0000 7fedf3637640 1 -- 172.21.15.59:0/2272795022 >> 172.21.15.59:0/2272795022 conn(0x7fedec0b1650 msgr2=0x7fedec0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.998+0000 7fedf3637640 1 -- 172.21.15.59:0/2272795022 shutdown_connections 2024-03-06T07:34:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.998+0000 7fedf3637640 1 -- 172.21.15.59:0/2272795022 wait complete. 2024-03-06T07:34:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.999+0000 7fedf3637640 1 Processor -- start 2024-03-06T07:34:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:33.999+0000 7fedf3637640 1 -- start start 2024-03-06T07:34:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf3637640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fedec0708b0 0x7fedec1f12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf3637640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedec070d10 0x7fedec150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.007 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf3637640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fedec15ac00 0x7fedec15cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf3637640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec0cd8a0 con 0x7fedec070d10 2024-03-06T07:34:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf3637640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedec0cd5c0 con 0x7fedec15ac00 2024-03-06T07:34:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf3637640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedec06ce50 con 0x7fedec0708b0 2024-03-06T07:34:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf0bd2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedec070d10 0x7fedec150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf0bd2640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedec070d10 0x7fedec150fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56242/0 (socket says 172.21.15.59:56242) 2024-03-06T07:34:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf0bd2640 1 -- 172.21.15.59:0/3323226224 learned_addr learned my addr 172.21.15.59:0/3323226224 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf13d3640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fedec0708b0 0x7fedec1f12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.008 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf1bd4640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fedec15ac00 0x7fedec15cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf0bd2640 1 -- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fedec0708b0 msgr2=0x7fedec1f12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf0bd2640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fedec0708b0 0x7fedec1f12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf0bd2640 1 -- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fedec15ac00 msgr2=0x7fedec15cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf0bd2640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fedec15ac00 0x7fedec15cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf0bd2640 1 -- 172.21.15.59:0/3323226224 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec156770 con 0x7fedec070d10 2024-03-06T07:34:34.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf1bd4640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fedec15ac00 0x7fedec15cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:34.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.000+0000 7fedf0bd2640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedec070d10 0x7fedec150fd0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fede800c440 tx=0x7fede8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.001+0000 7fede1ffb640 1 -- 172.21.15.59:0/3323226224 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede80052d0 con 0x7fedec070d10 2024-03-06T07:34:34.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.001+0000 7fede1ffb640 1 -- 172.21.15.59:0/3323226224 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede8005470 con 0x7fedec070d10 2024-03-06T07:34:34.009 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.001+0000 7fedf3637640 1 -- 172.21.15.59:0/3323226224 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedec156a50 con 0x7fedec070d10 2024-03-06T07:34:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.001+0000 7fede1ffb640 1 -- 172.21.15.59:0/3323226224 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8007260 con 0x7fedec070d10 2024-03-06T07:34:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.001+0000 7fedf3637640 1 -- 172.21.15.59:0/3323226224 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedec157010 con 0x7fedec070d10 2024-03-06T07:34:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.002+0000 7fedf3637640 1 -- 172.21.15.59:0/3323226224 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fedec151510 con 0x7fedec070d10 2024-03-06T07:34:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.002+0000 7fede1ffb640 1 -- 172.21.15.59:0/3323226224 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fede8007400 con 0x7fedec070d10 2024-03-06T07:34:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.003+0000 7fede1ffb640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fedd006b4d0 0x7fedd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.003+0000 7fede1ffb640 1 -- 172.21.15.59:0/3323226224 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fede8089360 con 0x7fedec070d10 2024-03-06T07:34:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.003+0000 7fedf13d3640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fedd006b4d0 0x7fedd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.010 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.003+0000 7fede1ffb640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7fedd0071430 0x7fedd0073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.003+0000 7fede1ffb640 1 -- 172.21.15.59:0/3323226224 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fede8009e50 con 0x7fedd0071430 2024-03-06T07:34:34.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.003+0000 7fede1ffb640 1 -- 172.21.15.59:0/3323226224 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7fede8090050 con 0x7fedec070d10 2024-03-06T07:34:34.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.003+0000 7fedf1bd4640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7fedd0071430 0x7fedd0073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.003+0000 7fedf13d3640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fedd006b4d0 0x7fedd006d990 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7feddc009aa0 tx=0x7feddc009190 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.003+0000 7fedf1bd4640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7fedd0071430 0x7fedd0073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.011 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.004+0000 7fede1ffb640 1 -- 172.21.15.59:0/3323226224 <== osd.1 v2:172.21.15.59:6816/1773892168 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fede8009e50 con 0x7fedd0071430 2024-03-06T07:34:34.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.017+0000 7fedf3637640 1 -- 172.21.15.59:0/3323226224 --> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fedec04f330 con 0x7fedd0071430 2024-03-06T07:34:34.019 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.017+0000 7fede1ffb640 1 -- 172.21.15.59:0/3323226224 <== osd.1 v2:172.21.15.59:6816/1773892168 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fedec04f330 con 0x7fedd0071430 2024-03-06T07:34:34.020 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:34:34.020 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:34.020 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:34:34.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.018+0000 7fedf3637640 1 -- 172.21.15.59:0/3323226224 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7fedd0071430 msgr2=0x7fedd0073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.018+0000 7fedf3637640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.59:6816/1773892168,v1:172.21.15.59:6817/1773892168] conn(0x7fedd0071430 0x7fedd0073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.021 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.018+0000 7fedf3637640 1 -- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fedd006b4d0 msgr2=0x7fedd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.018+0000 7fedf3637640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fedd006b4d0 0x7fedd006d990 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7feddc009aa0 tx=0x7feddc009190 comp rx=0 tx=0).stop 2024-03-06T07:34:34.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.018+0000 7fedf3637640 1 -- 172.21.15.59:0/3323226224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedec070d10 msgr2=0x7fedec150fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.018+0000 7fedf3637640 1 --2- 172.21.15.59:0/3323226224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fedec070d10 0x7fedec150fd0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fede800c440 tx=0x7fede8001e70 comp rx=0 tx=0).stop 2024-03-06T07:34:34.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.018+0000 7fedf1bd4640 1 -- 172.21.15.59:0/3323226224 reap_dead start 2024-03-06T07:34:34.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.018+0000 7fedf3637640 1 -- 172.21.15.59:0/3323226224 shutdown_connections 2024-03-06T07:34:34.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.018+0000 7fedf3637640 1 -- 172.21.15.59:0/3323226224 >> 172.21.15.59:0/3323226224 conn(0x7fedec0b1650 msgr2=0x7fedec0b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.018+0000 7fedf3637640 1 -- 172.21.15.59:0/3323226224 shutdown_connections 2024-03-06T07:34:34.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.018+0000 7fedf3637640 1 -- 172.21.15.59:0/3323226224 wait complete. 2024-03-06T07:34:34.030 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-06T07:34:34.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.131+0000 7fc293f65640 1 Processor -- start 2024-03-06T07:34:34.133 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.131+0000 7fc293f65640 1 -- start start 2024-03-06T07:34:34.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.132+0000 7fc293f65640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc28c15e000 0x7fc28c0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.132+0000 7fc293f65640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28c0cf7d0 0x7fc28c0d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.132+0000 7fc293f65640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0d25e0 con 0x7fc28c0d4810 2024-03-06T07:34:34.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.132+0000 7fc293f65640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28c0c7c50 con 0x7fc28c15e000 2024-03-06T07:34:34.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.132+0000 7fc293f65640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0d2880 con 0x7fc28c0cf7d0 2024-03-06T07:34:34.135 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.132+0000 7fc291d01640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc28c15e000 0x7fc28c0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.132+0000 7fc291500640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc28c0d4810 0x7fc28c0c80e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59866/0 (socket says 172.21.15.59:59866) 2024-03-06T07:34:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.132+0000 7fc291500640 1 -- 172.21.15.59:0/2292685692 learned_addr learned my addr 172.21.15.59:0/2292685692 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.133+0000 7fc292502640 1 --2- 172.21.15.59:0/2292685692 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28c0cf7d0 0x7fc28c0d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.133+0000 7fc291d01640 1 -- 172.21.15.59:0/2292685692 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28c0cf7d0 msgr2=0x7fc28c0d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.133+0000 7fc291d01640 1 --2- 172.21.15.59:0/2292685692 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28c0cf7d0 0x7fc28c0d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.133+0000 7fc291d01640 1 -- 172.21.15.59:0/2292685692 >> v1:172.21.15.59:6789/0 conn(0x7fc28c0d4810 legacy=0x7fc28c0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.133+0000 7fc290cff640 1 -- 172.21.15.59:0/2292685692 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350190735 0 0) 0x7fc28c0d25e0 con 0x7fc28c0d4810 2024-03-06T07:34:34.136 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.133+0000 7fc291d01640 1 -- 172.21.15.59:0/2292685692 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0d2b40 con 0x7fc28c15e000 2024-03-06T07:34:34.137 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.133+0000 7fc291d01640 1 --2- 172.21.15.59:0/2292685692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c15e000 0x7fc28c0d0680 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fc2880097a0 tx=0x7fc28802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dac6c7eef5e1274 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.134+0000 7fc290cff640 1 -- 172.21.15.59:0/2292685692 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288034070 con 0x7fc28c15e000 2024-03-06T07:34:34.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.134+0000 7fc290cff640 1 -- 172.21.15.59:0/2292685692 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc288030610 con 0x7fc28c15e000 2024-03-06T07:34:34.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.134+0000 7fc290cff640 1 -- 172.21.15.59:0/2292685692 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2880309e0 con 0x7fc28c15e000 2024-03-06T07:34:34.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.134+0000 7fc293f65640 1 -- 172.21.15.59:0/2292685692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c15e000 msgr2=0x7fc28c0d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.140 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.134+0000 7fc293f65640 1 --2- 172.21.15.59:0/2292685692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c15e000 0x7fc28c0d0680 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fc2880097a0 tx=0x7fc28802b960 comp rx=0 tx=0).stop 2024-03-06T07:34:34.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.135+0000 7fc293f65640 1 -- 172.21.15.59:0/2292685692 shutdown_connections 2024-03-06T07:34:34.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.135+0000 7fc293f65640 1 --2- 172.21.15.59:0/2292685692 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28c0cf7d0 0x7fc28c0d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.135+0000 7fc293f65640 1 --2- 172.21.15.59:0/2292685692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c15e000 0x7fc28c0d0680 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.135+0000 7fc293f65640 1 -- 172.21.15.59:0/2292685692 >> 172.21.15.59:0/2292685692 conn(0x7fc28c0d3cf0 msgr2=0x7fc28c0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.135+0000 7fc293f65640 1 -- 172.21.15.59:0/2292685692 shutdown_connections 2024-03-06T07:34:34.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.135+0000 7fc293f65640 1 -- 172.21.15.59:0/2292685692 wait complete. 2024-03-06T07:34:34.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.136+0000 7fc293f65640 1 Processor -- start 2024-03-06T07:34:34.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.136+0000 7fc293f65640 1 -- start start 2024-03-06T07:34:34.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc293f65640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28c0cf7d0 0x7fc28c1502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc293f65640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28c0d4810 0x7fc28c0b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc293f65640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c15e000 0x7fc28c0b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc293f65640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c15e3e0 con 0x7fc28c15e000 2024-03-06T07:34:34.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc293f65640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0d22d0 con 0x7fc28c0d4810 2024-03-06T07:34:34.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc293f65640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0c87f0 con 0x7fc28c0cf7d0 2024-03-06T07:34:34.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc292502640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c15e000 0x7fc28c0b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc292502640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c15e000 0x7fc28c0b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56266/0 (socket says 172.21.15.59:56266) 2024-03-06T07:34:34.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc292502640 1 -- 172.21.15.59:0/1468758921 learned_addr learned my addr 172.21.15.59:0/1468758921 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:34.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc291d01640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28c0cf7d0 0x7fc28c1502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc291500640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28c0d4810 0x7fc28c0b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc292502640 1 -- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28c0cf7d0 msgr2=0x7fc28c1502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc292502640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28c0cf7d0 0x7fc28c1502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc292502640 1 -- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28c0d4810 msgr2=0x7fc28c0b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc292502640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28c0d4810 0x7fc28c0b3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc292502640 1 -- 172.21.15.59:0/1468758921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c150dd0 con 0x7fc28c15e000 2024-03-06T07:34:34.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.137+0000 7fc292502640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c15e000 0x7fc28c0b3a40 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fc284007e40 tx=0x7fc28400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.138+0000 7fc2827fc640 1 -- 172.21.15.59:0/1468758921 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284014620 con 0x7fc28c15e000 2024-03-06T07:34:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.138+0000 7fc2827fc640 1 -- 172.21.15.59:0/1468758921 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc284019070 con 0x7fc28c15e000 2024-03-06T07:34:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.138+0000 7fc293f65640 1 -- 172.21.15.59:0/1468758921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c1510b0 con 0x7fc28c15e000 2024-03-06T07:34:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.138+0000 7fc2827fc640 1 -- 172.21.15.59:0/1468758921 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284014a20 con 0x7fc28c15e000 2024-03-06T07:34:34.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.138+0000 7fc293f65640 1 -- 172.21.15.59:0/1468758921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c150820 con 0x7fc28c15e000 2024-03-06T07:34:34.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.139+0000 7fc293f65640 1 -- 172.21.15.59:0/1468758921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c04f330 con 0x7fc28c15e000 2024-03-06T07:34:34.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.139+0000 7fc2827fc640 1 -- 172.21.15.59:0/1468758921 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc284014bc0 con 0x7fc28c15e000 2024-03-06T07:34:34.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.140+0000 7fc2827fc640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc27006b4d0 0x7fc27006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.145 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.140+0000 7fc2827fc640 1 -- 172.21.15.59:0/1468758921 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc28408a720 con 0x7fc28c15e000 2024-03-06T07:34:34.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.140+0000 7fc291d01640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc27006b4d0 0x7fc27006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.141+0000 7fc291d01640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc27006b4d0 0x7fc27006d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fc288036000 tx=0x7fc288008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.144+0000 7fc2827fc640 1 -- 172.21.15.59:0/1468758921 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2840591f0 con 0x7fc28c15e000 2024-03-06T07:34:34.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.327+0000 7fc293f65640 1 -- 172.21.15.59:0/1468758921 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc28c0c9020 con 0x7fc27006b4d0 2024-03-06T07:34:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.328+0000 7fc2827fc640 1 -- 172.21.15.59:0/1468758921 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc28c0c9020 con 0x7fc27006b4d0 2024-03-06T07:34:34.330 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 1 to deep-scrub 2024-03-06T07:34:34.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 -- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc27006b4d0 msgr2=0x7fc27006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc27006b4d0 0x7fc27006d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fc288036000 tx=0x7fc288008030 comp rx=0 tx=0).stop 2024-03-06T07:34:34.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 -- 172.21.15.59:0/1468758921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c15e000 msgr2=0x7fc28c0b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c15e000 0x7fc28c0b3a40 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fc284007e40 tx=0x7fc28400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 -- 172.21.15.59:0/1468758921 shutdown_connections 2024-03-06T07:34:34.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fc27006b4d0 0x7fc27006d990 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc28c15e000 0x7fc28c0b3a40 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28c0d4810 0x7fc28c0b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 --2- 172.21.15.59:0/1468758921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28c0cf7d0 0x7fc28c1502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 -- 172.21.15.59:0/1468758921 >> 172.21.15.59:0/1468758921 conn(0x7fc28c0d3cf0 msgr2=0x7fc28c0b8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 -- 172.21.15.59:0/1468758921 shutdown_connections 2024-03-06T07:34:34.334 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.330+0000 7fc293f65640 1 -- 172.21.15.59:0/1468758921 wait complete. 2024-03-06T07:34:34.342 INFO:tasks.ceph:Scrubbing osd.2 2024-03-06T07:34:34.343 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:34.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.446+0000 7f6c412cf640 1 Processor -- start 2024-03-06T07:34:34.448 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.447+0000 7f6c412cf640 1 -- start start 2024-03-06T07:34:34.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.447+0000 7f6c412cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c3c162850 0x7f6c3c0d1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.447+0000 7f6c412cf640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6c3c0d1f90 0x7f6c3c06c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.447+0000 7f6c412cf640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0cf3c0 con 0x7f6c3c162470 2024-03-06T07:34:34.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.447+0000 7f6c412cf640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c06f270 con 0x7f6c3c0d1f90 2024-03-06T07:34:34.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.447+0000 7f6c412cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c0cf0e0 con 0x7f6c3c162850 2024-03-06T07:34:34.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.447+0000 7f6c3b7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6c3c0d1f90 0x7f6c3c06c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.447+0000 7f6c3affd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6c3c162470 0x7f6c3c06f700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59882/0 (socket says 172.21.15.59:59882) 2024-03-06T07:34:34.450 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.447+0000 7f6c3affd640 1 -- 172.21.15.59:0/3713406057 learned_addr learned my addr 172.21.15.59:0/3713406057 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:34.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.448+0000 7f6c33fff640 1 --2- 172.21.15.59:0/3713406057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c3c162850 0x7f6c3c0d1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.448+0000 7f6c3b7fe640 1 -- 172.21.15.59:0/3713406057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c3c162850 msgr2=0x7f6c3c0d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:34.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.448+0000 7f6c3b7fe640 1 --2- 172.21.15.59:0/3713406057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c3c162850 0x7f6c3c0d1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.448+0000 7f6c3b7fe640 1 -- 172.21.15.59:0/3713406057 >> v1:172.21.15.59:6789/0 conn(0x7f6c3c162470 legacy=0x7f6c3c06f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.448+0000 7f6c3a7fc640 1 -- 172.21.15.59:0/3713406057 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555621250 0 0) 0x7f6c3c0cf3c0 con 0x7f6c3c162470 2024-03-06T07:34:34.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.448+0000 7f6c3b7fe640 1 -- 172.21.15.59:0/3713406057 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c06ec20 con 0x7f6c3c0d1f90 2024-03-06T07:34:34.451 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.448+0000 7f6c33fff640 1 --2- 172.21.15.59:0/3713406057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c3c162850 0x7f6c3c0d1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:34.454 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c3b7fe640 1 --2- 172.21.15.59:0/3713406057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c3c0d1f90 0x7f6c3c06c3b0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009d90 tx=0x7f6c3402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17bcdfdc83286f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c3a7fc640 1 -- 172.21.15.59:0/3713406057 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c34039070 con 0x7f6c3c0d1f90 2024-03-06T07:34:34.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c3a7fc640 1 -- 172.21.15.59:0/3713406057 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3402cc50 con 0x7f6c3c0d1f90 2024-03-06T07:34:34.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c3a7fc640 1 -- 172.21.15.59:0/3713406057 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c34034440 con 0x7f6c3c0d1f90 2024-03-06T07:34:34.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c412cf640 1 -- 172.21.15.59:0/3713406057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c3c0d1f90 msgr2=0x7f6c3c06c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c412cf640 1 --2- 172.21.15.59:0/3713406057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c3c0d1f90 0x7f6c3c06c3b0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009d90 tx=0x7f6c3402bea0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.455 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c412cf640 1 -- 172.21.15.59:0/3713406057 shutdown_connections 2024-03-06T07:34:34.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c412cf640 1 --2- 172.21.15.59:0/3713406057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c3c0d1f90 0x7f6c3c06c3b0 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c412cf640 1 --2- 172.21.15.59:0/3713406057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c3c162850 0x7f6c3c0d1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c412cf640 1 -- 172.21.15.59:0/3713406057 >> 172.21.15.59:0/3713406057 conn(0x7f6c3c09a9f0 msgr2=0x7f6c3c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c412cf640 1 -- 172.21.15.59:0/3713406057 shutdown_connections 2024-03-06T07:34:34.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.449+0000 7f6c412cf640 1 -- 172.21.15.59:0/3713406057 wait complete. 2024-03-06T07:34:34.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.450+0000 7f6c412cf640 1 Processor -- start 2024-03-06T07:34:34.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.451+0000 7f6c412cf640 1 -- start start 2024-03-06T07:34:34.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.451+0000 7f6c412cf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c3c0d1f90 0x7f6c3c15eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.456 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.451+0000 7f6c412cf640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c3c162470 0x7f6c3c073f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.451+0000 7f6c412cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c3c162850 0x7f6c3c0ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.451+0000 7f6c412cf640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c06ff90 con 0x7f6c3c162470 2024-03-06T07:34:34.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.451+0000 7f6c412cf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c06fe10 con 0x7f6c3c0d1f90 2024-03-06T07:34:34.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.451+0000 7f6c412cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c06c8f0 con 0x7f6c3c162850 2024-03-06T07:34:34.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.451+0000 7f6c33fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c3c162470 0x7f6c3c073f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.451+0000 7f6c33fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c3c162470 0x7f6c3c073f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56278/0 (socket says 172.21.15.59:56278) 2024-03-06T07:34:34.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.451+0000 7f6c33fff640 1 -- 172.21.15.59:0/3820812530 learned_addr learned my addr 172.21.15.59:0/3820812530 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:34.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c3b7fe640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c3c162850 0x7f6c3c0ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.457 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c33fff640 1 -- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c3c162850 msgr2=0x7f6c3c0ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c3affd640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c3c0d1f90 0x7f6c3c15eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c33fff640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c3c162850 0x7f6c3c0ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c33fff640 1 -- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c3c0d1f90 msgr2=0x7f6c3c15eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c33fff640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c3c0d1f90 0x7f6c3c15eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c33fff640 1 -- 172.21.15.59:0/3820812530 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c0b6de0 con 0x7f6c3c162470 2024-03-06T07:34:34.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c3b7fe640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c3c162850 0x7f6c3c0ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:34.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c3affd640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c3c0d1f90 0x7f6c3c15eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:34.458 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c33fff640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c3c162470 0x7f6c3c073f10 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400d800 tx=0x7f6c2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c337fe640 1 -- 172.21.15.59:0/3820812530 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24004030 con 0x7f6c3c162470 2024-03-06T07:34:34.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c337fe640 1 -- 172.21.15.59:0/3820812530 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c240041d0 con 0x7f6c3c162470 2024-03-06T07:34:34.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c412cf640 1 -- 172.21.15.59:0/3820812530 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c0c9840 con 0x7f6c3c162470 2024-03-06T07:34:34.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c337fe640 1 -- 172.21.15.59:0/3820812530 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c240096a0 con 0x7f6c3c162470 2024-03-06T07:34:34.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.452+0000 7f6c412cf640 1 -- 172.21.15.59:0/3820812530 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c0c9e00 con 0x7f6c3c162470 2024-03-06T07:34:34.459 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.453+0000 7f6c412cf640 1 -- 172.21.15.59:0/3820812530 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6c3c0ca1e0 con 0x7f6c3c162470 2024-03-06T07:34:34.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.454+0000 7f6c337fe640 1 -- 172.21.15.59:0/3820812530 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c24009840 con 0x7f6c3c162470 2024-03-06T07:34:34.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.454+0000 7f6c337fe640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6c1c06b4d0 0x7f6c1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.454+0000 7f6c337fe640 1 -- 172.21.15.59:0/3820812530 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6c2408a130 con 0x7f6c3c162470 2024-03-06T07:34:34.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.454+0000 7f6c337fe640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f6c1c071430 0x7f6c1c073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.454+0000 7f6c337fe640 1 -- 172.21.15.59:0/3820812530 --> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c24021ec0 con 0x7f6c1c071430 2024-03-06T07:34:34.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.454+0000 7f6c337fe640 1 -- 172.21.15.59:0/3820812530 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c24091050 con 0x7f6c3c162470 2024-03-06T07:34:34.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.454+0000 7f6c3b7fe640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f6c1c071430 0x7f6c1c073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.454+0000 7f6c3affd640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6c1c06b4d0 0x7f6c1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.455+0000 7f6c3b7fe640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f6c1c071430 0x7f6c1c073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.455+0000 7f6c3affd640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6c1c06b4d0 0x7f6c1c06d990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c000f60 tx=0x7f6c2c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.455+0000 7f6c337fe640 1 -- 172.21.15.59:0/3820812530 <== osd.2 v2:172.21.15.59:6800/1020996386 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6c24021ec0 con 0x7f6c1c071430 2024-03-06T07:34:34.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.468+0000 7f6c412cf640 1 -- 172.21.15.59:0/3820812530 --> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6c3c04f330 con 0x7f6c1c071430 2024-03-06T07:34:34.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.468+0000 7f6c337fe640 1 -- 172.21.15.59:0/3820812530 <== osd.2 v2:172.21.15.59:6800/1020996386 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6c3c04f330 con 0x7f6c1c071430 2024-03-06T07:34:34.471 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:34:34.471 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:34.471 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:34:34.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.469+0000 7f6c412cf640 1 -- 172.21.15.59:0/3820812530 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f6c1c071430 msgr2=0x7f6c1c073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.469+0000 7f6c412cf640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.59:6800/1020996386,v1:172.21.15.59:6801/1020996386] conn(0x7f6c1c071430 0x7f6c1c073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.469+0000 7f6c412cf640 1 -- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6c1c06b4d0 msgr2=0x7f6c1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.469+0000 7f6c412cf640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6c1c06b4d0 0x7f6c1c06d990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c000f60 tx=0x7f6c2c009040 comp rx=0 tx=0).stop 2024-03-06T07:34:34.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.469+0000 7f6c412cf640 1 -- 172.21.15.59:0/3820812530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c3c162470 msgr2=0x7f6c3c073f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.469+0000 7f6c412cf640 1 --2- 172.21.15.59:0/3820812530 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c3c162470 0x7f6c3c073f10 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400d800 tx=0x7f6c2400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.469+0000 7f6c3b7fe640 1 -- 172.21.15.59:0/3820812530 reap_dead start 2024-03-06T07:34:34.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.469+0000 7f6c412cf640 1 -- 172.21.15.59:0/3820812530 shutdown_connections 2024-03-06T07:34:34.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.469+0000 7f6c412cf640 1 -- 172.21.15.59:0/3820812530 >> 172.21.15.59:0/3820812530 conn(0x7f6c3c09a9f0 msgr2=0x7f6c3c06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.469+0000 7f6c412cf640 1 -- 172.21.15.59:0/3820812530 shutdown_connections 2024-03-06T07:34:34.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.469+0000 7f6c412cf640 1 -- 172.21.15.59:0/3820812530 wait complete. 2024-03-06T07:34:34.481 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-06T07:34:34.583 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.582+0000 7efd591d6640 1 Processor -- start 2024-03-06T07:34:34.584 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.582+0000 7efd591d6640 1 -- start start 2024-03-06T07:34:34.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd591d6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efd540b4c00 0x7efd540b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd591d6640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efd540b28a0 0x7efd540b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd591d6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5415d510 con 0x7efd5415e540 2024-03-06T07:34:34.585 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd591d6640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd5415e0b0 con 0x7efd540b28a0 2024-03-06T07:34:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd591d6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd5415d780 con 0x7efd540b4c00 2024-03-06T07:34:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd4bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efd540b28a0 0x7efd540b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd4bfff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7efd540b28a0 0x7efd540b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56290/0 (socket says 172.21.15.59:56290) 2024-03-06T07:34:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd4bfff640 1 -- 172.21.15.59:0/4187282981 learned_addr learned my addr 172.21.15.59:0/4187282981 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd52d9d640 1 --2- 172.21.15.59:0/4187282981 >> v2:172.21.15.164:3300/0 conn(0x7efd540b4c00 0x7efd540b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd4bfff640 1 -- 172.21.15.59:0/4187282981 >> v2:172.21.15.164:3300/0 conn(0x7efd540b4c00 msgr2=0x7efd540b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd4bfff640 1 --2- 172.21.15.59:0/4187282981 >> v2:172.21.15.164:3300/0 conn(0x7efd540b4c00 0x7efd540b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.586 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd4bfff640 1 -- 172.21.15.59:0/4187282981 >> v1:172.21.15.164:6789/0 conn(0x7efd5415e540 legacy=0x7efd5415e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd4bfff640 1 -- 172.21.15.59:0/4187282981 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5415da60 con 0x7efd540b28a0 2024-03-06T07:34:34.587 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.583+0000 7efd52d9d640 1 --2- 172.21.15.59:0/4187282981 >> v2:172.21.15.164:3300/0 conn(0x7efd540b4c00 0x7efd540b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:34.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.584+0000 7efd4bfff640 1 --2- 172.21.15.59:0/4187282981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd540b28a0 0x7efd540b2d10 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7efd540721d0 tx=0x7efd3c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7159c5c234272c97 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.584+0000 7efd5259c640 1 -- 172.21.15.59:0/4187282981 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3c02f850 con 0x7efd540b28a0 2024-03-06T07:34:34.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.584+0000 7efd5259c640 1 -- 172.21.15.59:0/4187282981 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3c02f9f0 con 0x7efd540b28a0 2024-03-06T07:34:34.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.584+0000 7efd5259c640 1 -- 172.21.15.59:0/4187282981 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3c02fdc0 con 0x7efd540b28a0 2024-03-06T07:34:34.590 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.584+0000 7efd591d6640 1 -- 172.21.15.59:0/4187282981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd540b28a0 msgr2=0x7efd540b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.584+0000 7efd591d6640 1 --2- 172.21.15.59:0/4187282981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd540b28a0 0x7efd540b2d10 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7efd540721d0 tx=0x7efd3c02ba30 comp rx=0 tx=0).stop 2024-03-06T07:34:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.585+0000 7efd591d6640 1 -- 172.21.15.59:0/4187282981 shutdown_connections 2024-03-06T07:34:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.585+0000 7efd591d6640 1 --2- 172.21.15.59:0/4187282981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd540b28a0 0x7efd540b2d10 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.585+0000 7efd591d6640 1 --2- 172.21.15.59:0/4187282981 >> v2:172.21.15.164:3300/0 conn(0x7efd540b4c00 0x7efd540b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.585+0000 7efd591d6640 1 -- 172.21.15.59:0/4187282981 >> 172.21.15.59:0/4187282981 conn(0x7efd5409a9f0 msgr2=0x7efd5409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.585+0000 7efd591d6640 1 -- 172.21.15.59:0/4187282981 shutdown_connections 2024-03-06T07:34:34.591 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.585+0000 7efd591d6640 1 -- 172.21.15.59:0/4187282981 wait complete. 2024-03-06T07:34:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.586+0000 7efd591d6640 1 Processor -- start 2024-03-06T07:34:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.586+0000 7efd591d6640 1 -- start start 2024-03-06T07:34:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd591d6640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd540b28a0 0x7efd54144420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd591d6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd540b4c00 0x7efd540b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd591d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd5415e540 0x7efd540b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd591d6640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd540b33d0 con 0x7efd540b28a0 2024-03-06T07:34:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd591d6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd540b3250 con 0x7efd540b4c00 2024-03-06T07:34:34.592 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd591d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5415f0c0 con 0x7efd5415e540 2024-03-06T07:34:34.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd52d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd540b28a0 0x7efd54144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd52d9d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd540b28a0 0x7efd54144420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56304/0 (socket says 172.21.15.59:56304) 2024-03-06T07:34:34.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd52d9d640 1 -- 172.21.15.59:0/3631495653 learned_addr learned my addr 172.21.15.59:0/3631495653 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:34.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd4bfff640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd540b4c00 0x7efd540b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd5359e640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd5415e540 0x7efd540b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd52d9d640 1 -- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd5415e540 msgr2=0x7efd540b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd52d9d640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd5415e540 0x7efd540b1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.593 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd52d9d640 1 -- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd540b4c00 msgr2=0x7efd540b1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd52d9d640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd540b4c00 0x7efd540b1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd52d9d640 1 -- 172.21.15.59:0/3631495653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd54144960 con 0x7efd540b28a0 2024-03-06T07:34:34.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd52d9d640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd540b28a0 0x7efd54144420 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7efd4400e7e0 tx=0x7efd4400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd4b7fe640 1 -- 172.21.15.59:0/3631495653 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4400f040 con 0x7efd540b28a0 2024-03-06T07:34:34.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd4b7fe640 1 -- 172.21.15.59:0/3631495653 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd44004030 con 0x7efd540b28a0 2024-03-06T07:34:34.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd591d6640 1 -- 172.21.15.59:0/3631495653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd540afa70 con 0x7efd540b28a0 2024-03-06T07:34:34.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd4b7fe640 1 -- 172.21.15.59:0/3631495653 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd44004400 con 0x7efd540b28a0 2024-03-06T07:34:34.594 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.587+0000 7efd591d6640 1 -- 172.21.15.59:0/3631495653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd540b0030 con 0x7efd540b28a0 2024-03-06T07:34:34.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.589+0000 7efd591d6640 1 -- 172.21.15.59:0/3631495653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5404f330 con 0x7efd540b28a0 2024-03-06T07:34:34.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.589+0000 7efd4b7fe640 1 -- 172.21.15.59:0/3631495653 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd4400c6d0 con 0x7efd540b28a0 2024-03-06T07:34:34.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.589+0000 7efd4b7fe640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd3406b4d0 0x7efd3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.589+0000 7efd4b7fe640 1 -- 172.21.15.59:0/3631495653 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7efd44089d10 con 0x7efd540b28a0 2024-03-06T07:34:34.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.590+0000 7efd4bfff640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd3406b4d0 0x7efd3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.590+0000 7efd4bfff640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd3406b4d0 0x7efd3406d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7efd3c036000 tx=0x7efd3c008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.596 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.594+0000 7efd4b7fe640 1 -- 172.21.15.59:0/3631495653 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd44017020 con 0x7efd540b28a0 2024-03-06T07:34:34.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.777+0000 7efd591d6640 1 -- 172.21.15.59:0/3631495653 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7efd540b09d0 con 0x7efd3406b4d0 2024-03-06T07:34:34.780 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.778+0000 7efd4b7fe640 1 -- 172.21.15.59:0/3631495653 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7efd540b09d0 con 0x7efd3406b4d0 2024-03-06T07:34:34.780 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 2 to deep-scrub 2024-03-06T07:34:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 -- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd3406b4d0 msgr2=0x7efd3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd3406b4d0 0x7efd3406d990 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7efd3c036000 tx=0x7efd3c008030 comp rx=0 tx=0).stop 2024-03-06T07:34:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 -- 172.21.15.59:0/3631495653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd540b28a0 msgr2=0x7efd54144420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd540b28a0 0x7efd54144420 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7efd4400e7e0 tx=0x7efd4400eca0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 -- 172.21.15.59:0/3631495653 shutdown_connections 2024-03-06T07:34:34.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7efd3406b4d0 0x7efd3406d990 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd5415e540 0x7efd540b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd540b4c00 0x7efd540b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 --2- 172.21.15.59:0/3631495653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd540b28a0 0x7efd54144420 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 -- 172.21.15.59:0/3631495653 >> 172.21.15.59:0/3631495653 conn(0x7efd5409a9f0 msgr2=0x7efd5406ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 -- 172.21.15.59:0/3631495653 shutdown_connections 2024-03-06T07:34:34.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.780+0000 7efd591d6640 1 -- 172.21.15.59:0/3631495653 wait complete. 2024-03-06T07:34:34.792 INFO:tasks.ceph:Scrubbing osd.3 2024-03-06T07:34:34.792 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:34.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.896+0000 7f462af5c640 1 Processor -- start 2024-03-06T07:34:34.898 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.896+0000 7f462af5c640 1 -- start start 2024-03-06T07:34:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.896+0000 7f462af5c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4624161440 0x7f46240b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.896+0000 7f462af5c640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4624161820 0x7f46240d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.896+0000 7f462af5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46240d1300 0x7f46240d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.896+0000 7f462af5c640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46240cf2d0 con 0x7f4624161820 2024-03-06T07:34:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.896+0000 7f462af5c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46240b6b00 con 0x7f4624161440 2024-03-06T07:34:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.896+0000 7f462af5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46240cf520 con 0x7f46240d1300 2024-03-06T07:34:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.897+0000 7f4623fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4624161820 0x7f46240d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.897+0000 7f4623fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4624161820 0x7f46240d0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56308/0 (socket says 172.21.15.59:56308) 2024-03-06T07:34:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.897+0000 7f4623fff640 1 -- 172.21.15.59:0/1759826096 learned_addr learned my addr 172.21.15.59:0/1759826096 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.897+0000 7f4628cf8640 1 --2- 172.21.15.59:0/1759826096 >> v2:172.21.15.164:3300/0 conn(0x7f4624161440 0x7f46240b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.897+0000 7f46294f9640 1 --2- 172.21.15.59:0/1759826096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46240d1300 0x7f46240d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.897+0000 7f4623fff640 1 -- 172.21.15.59:0/1759826096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46240d1300 msgr2=0x7f46240d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.897+0000 7f4623fff640 1 --2- 172.21.15.59:0/1759826096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46240d1300 0x7f46240d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.897+0000 7f4623fff640 1 -- 172.21.15.59:0/1759826096 >> v2:172.21.15.164:3300/0 conn(0x7f4624161440 msgr2=0x7f46240b4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.897+0000 7f4623fff640 1 --2- 172.21.15.59:0/1759826096 >> v2:172.21.15.164:3300/0 conn(0x7f4624161440 0x7f46240b4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.897+0000 7f4623fff640 1 -- 172.21.15.59:0/1759826096 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46240cf800 con 0x7f4624161820 2024-03-06T07:34:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.897+0000 7f46294f9640 1 --2- 172.21.15.59:0/1759826096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46240d1300 0x7f46240d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:34.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f4623fff640 1 --2- 172.21.15.59:0/1759826096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4624161820 0x7f46240d0610 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f462406e070 tx=0x7f461802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a65ac4f88adc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f46237fe640 1 -- 172.21.15.59:0/1759826096 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618038070 con 0x7f4624161820 2024-03-06T07:34:34.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f46237fe640 1 -- 172.21.15.59:0/1759826096 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461802f050 con 0x7f4624161820 2024-03-06T07:34:34.902 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f46237fe640 1 -- 172.21.15.59:0/1759826096 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4618031970 con 0x7f4624161820 2024-03-06T07:34:34.906 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f462af5c640 1 -- 172.21.15.59:0/1759826096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4624161820 msgr2=0x7f46240d0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f462af5c640 1 --2- 172.21.15.59:0/1759826096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4624161820 0x7f46240d0610 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f462406e070 tx=0x7f461802b7f0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f462af5c640 1 -- 172.21.15.59:0/1759826096 shutdown_connections 2024-03-06T07:34:34.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f462af5c640 1 --2- 172.21.15.59:0/1759826096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46240d1300 0x7f46240d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f462af5c640 1 --2- 172.21.15.59:0/1759826096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4624161820 0x7f46240d0610 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f462af5c640 1 --2- 172.21.15.59:0/1759826096 >> v2:172.21.15.164:3300/0 conn(0x7f4624161440 0x7f46240b4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f462af5c640 1 -- 172.21.15.59:0/1759826096 >> 172.21.15.59:0/1759826096 conn(0x7f46240d3cb0 msgr2=0x7f46240d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f462af5c640 1 -- 172.21.15.59:0/1759826096 shutdown_connections 2024-03-06T07:34:34.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.898+0000 7f462af5c640 1 -- 172.21.15.59:0/1759826096 wait complete. 2024-03-06T07:34:34.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.899+0000 7f462af5c640 1 Processor -- start 2024-03-06T07:34:34.907 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.900+0000 7f462af5c640 1 -- start start 2024-03-06T07:34:34.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.900+0000 7f462af5c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46240d1300 0x7f4624157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.900+0000 7f462af5c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4624161440 0x7f462414c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.900+0000 7f462af5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4624161820 0x7f4624158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.900+0000 7f462af5c640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46240d1880 con 0x7f46240d1300 2024-03-06T07:34:34.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.900+0000 7f462af5c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46240b5120 con 0x7f4624161820 2024-03-06T07:34:34.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.900+0000 7f462af5c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46240d1a00 con 0x7f4624161440 2024-03-06T07:34:34.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f4628cf8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46240d1300 0x7f4624157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.908 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f4628cf8640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46240d1300 0x7f4624157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56314/0 (socket says 172.21.15.59:56314) 2024-03-06T07:34:34.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f4628cf8640 1 -- 172.21.15.59:0/1496437400 learned_addr learned my addr 172.21.15.59:0/1496437400 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:34.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f4623fff640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4624161440 0x7f462414c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f46294f9640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4624161820 0x7f4624158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f4628cf8640 1 -- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4624161440 msgr2=0x7f462414c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f4628cf8640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4624161440 0x7f462414c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f4628cf8640 1 -- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4624161820 msgr2=0x7f4624158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f4628cf8640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4624161820 0x7f4624158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.909 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f4628cf8640 1 -- 172.21.15.59:0/1496437400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4624159a20 con 0x7f46240d1300 2024-03-06T07:34:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f46294f9640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4624161820 0x7f4624158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f4628cf8640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46240d1300 0x7f4624157090 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f461c00efc0 tx=0x7f461c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f46217fa640 1 -- 172.21.15.59:0/1496437400 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c018020 con 0x7f46240d1300 2024-03-06T07:34:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f46217fa640 1 -- 172.21.15.59:0/1496437400 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461c014480 con 0x7f46240d1300 2024-03-06T07:34:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f462af5c640 1 -- 172.21.15.59:0/1496437400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4624159d00 con 0x7f46240d1300 2024-03-06T07:34:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f46217fa640 1 -- 172.21.15.59:0/1496437400 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c014850 con 0x7f46240d1300 2024-03-06T07:34:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.901+0000 7f462af5c640 1 -- 172.21.15.59:0/1496437400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462415a2c0 con 0x7f46240d1300 2024-03-06T07:34:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.902+0000 7f462af5c640 1 -- 172.21.15.59:0/1496437400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f462404f330 con 0x7f46240d1300 2024-03-06T07:34:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.903+0000 7f46217fa640 1 -- 172.21.15.59:0/1496437400 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f461c005d10 con 0x7f46240d1300 2024-03-06T07:34:34.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.903+0000 7f46217fa640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f461006b500 0x7f461006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.903+0000 7f46217fa640 1 -- 172.21.15.59:0/1496437400 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f461c08a890 con 0x7f46240d1300 2024-03-06T07:34:34.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.903+0000 7f46217fa640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f4610071460 0x7f4610073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:34.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.903+0000 7f46217fa640 1 -- 172.21.15.59:0/1496437400 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4610073f30 con 0x7f4610071460 2024-03-06T07:34:34.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.903+0000 7f46217fa640 1 -- 172.21.15.59:0/1496437400 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f461c08f2c0 con 0x7f46240d1300 2024-03-06T07:34:34.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.903+0000 7f4623fff640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f461006b500 0x7f461006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.903+0000 7f46294f9640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f4610071460 0x7f4610073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:34.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.904+0000 7f46294f9640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f4610071460 0x7f4610073880 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.911 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.904+0000 7f4623fff640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f461006b500 0x7f461006d9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4618035000 tx=0x7f4618008030 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:34.912 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.904+0000 7f46217fa640 1 -- 172.21.15.59:0/1496437400 <== osd.3 v2:172.21.15.59:6808/3602142643 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4610073f30 con 0x7f4610071460 2024-03-06T07:34:34.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.917+0000 7f462af5c640 1 -- 172.21.15.59:0/1496437400 --> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f46240b8f30 con 0x7f4610071460 2024-03-06T07:34:34.919 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f46217fa640 1 -- 172.21.15.59:0/1496437400 <== osd.3 v2:172.21.15.59:6808/3602142643 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f46240b8f30 con 0x7f4610071460 2024-03-06T07:34:34.920 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:34:34.920 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:34.920 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:34:34.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f462af5c640 1 -- 172.21.15.59:0/1496437400 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f4610071460 msgr2=0x7f4610073880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f462af5c640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.59:6808/3602142643,v1:172.21.15.59:6809/3602142643] conn(0x7f4610071460 0x7f4610073880 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:34.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f462af5c640 1 -- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f461006b500 msgr2=0x7f461006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.921 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f462af5c640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f461006b500 0x7f461006d9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4618035000 tx=0x7f4618008030 comp rx=0 tx=0).stop 2024-03-06T07:34:34.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f462af5c640 1 -- 172.21.15.59:0/1496437400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46240d1300 msgr2=0x7f4624157090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:34.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f462af5c640 1 --2- 172.21.15.59:0/1496437400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46240d1300 0x7f4624157090 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f461c00efc0 tx=0x7f461c00c490 comp rx=0 tx=0).stop 2024-03-06T07:34:34.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f46294f9640 1 -- 172.21.15.59:0/1496437400 reap_dead start 2024-03-06T07:34:34.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f462af5c640 1 -- 172.21.15.59:0/1496437400 shutdown_connections 2024-03-06T07:34:34.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f462af5c640 1 -- 172.21.15.59:0/1496437400 >> 172.21.15.59:0/1496437400 conn(0x7f46240d3cb0 msgr2=0x7f46240ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:34.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f462af5c640 1 -- 172.21.15.59:0/1496437400 shutdown_connections 2024-03-06T07:34:34.922 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:34.918+0000 7f462af5c640 1 -- 172.21.15.59:0/1496437400 wait complete. 2024-03-06T07:34:34.930 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-06T07:34:35.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.031+0000 7f73be60e640 1 Processor -- start 2024-03-06T07:34:35.033 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.031+0000 7f73be60e640 1 -- start start 2024-03-06T07:34:35.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73be60e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f73b80b3240 0x7f73b80b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73be60e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b80b2910 0x7f73b80d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73be60e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b815e3f0 con 0x7f73b815e010 2024-03-06T07:34:35.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73be60e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b80b1f40 con 0x7f73b80b3240 2024-03-06T07:34:35.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73be60e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b80b1c10 con 0x7f73b80b2910 2024-03-06T07:34:35.035 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73b77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f73b80b3240 0x7f73b80b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73b77fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f73b80b3240 0x7f73b80b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56328/0 (socket says 172.21.15.59:56328) 2024-03-06T07:34:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73b77fe640 1 -- 172.21.15.59:0/2074023552 learned_addr learned my addr 172.21.15.59:0/2074023552 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73bcbab640 1 --2- 172.21.15.59:0/2074023552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b80b2910 0x7f73b80d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73b77fe640 1 -- 172.21.15.59:0/2074023552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b80b2910 msgr2=0x7f73b80d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73b77fe640 1 --2- 172.21.15.59:0/2074023552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b80b2910 0x7f73b80d1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73b77fe640 1 -- 172.21.15.59:0/2074023552 >> v1:172.21.15.164:6789/0 conn(0x7f73b815e010 legacy=0x7f73b80cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.032+0000 7f73b77fe640 1 -- 172.21.15.59:0/2074023552 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b806d1b0 con 0x7f73b80b3240 2024-03-06T07:34:35.036 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.033+0000 7f73b77fe640 1 --2- 172.21.15.59:0/2074023552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b80b3240 0x7f73b80b23d0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009870 tx=0x7f73ac02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e71bde3612ae14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.033+0000 7f73b6ffd640 1 -- 172.21.15.59:0/2074023552 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0317c0 con 0x7f73b80b3240 2024-03-06T07:34:35.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.033+0000 7f73b6ffd640 1 -- 172.21.15.59:0/2074023552 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac037070 con 0x7f73b80b3240 2024-03-06T07:34:35.037 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.033+0000 7f73b6ffd640 1 -- 172.21.15.59:0/2074023552 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac031b80 con 0x7f73b80b3240 2024-03-06T07:34:35.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.033+0000 7f73be60e640 1 -- 172.21.15.59:0/2074023552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b80b3240 msgr2=0x7f73b80b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.033+0000 7f73be60e640 1 --2- 172.21.15.59:0/2074023552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b80b3240 0x7f73b80b23d0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009870 tx=0x7f73ac02b860 comp rx=0 tx=0).stop 2024-03-06T07:34:35.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.033+0000 7f73be60e640 1 -- 172.21.15.59:0/2074023552 shutdown_connections 2024-03-06T07:34:35.040 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.033+0000 7f73be60e640 1 --2- 172.21.15.59:0/2074023552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b80b2910 0x7f73b80d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.033+0000 7f73be60e640 1 --2- 172.21.15.59:0/2074023552 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b80b3240 0x7f73b80b23d0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.034+0000 7f73be60e640 1 -- 172.21.15.59:0/2074023552 >> 172.21.15.59:0/2074023552 conn(0x7f73b809a9f0 msgr2=0x7f73b809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:35.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.034+0000 7f73be60e640 1 -- 172.21.15.59:0/2074023552 shutdown_connections 2024-03-06T07:34:35.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.034+0000 7f73be60e640 1 -- 172.21.15.59:0/2074023552 wait complete. 2024-03-06T07:34:35.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.035+0000 7f73be60e640 1 Processor -- start 2024-03-06T07:34:35.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.035+0000 7f73be60e640 1 -- start start 2024-03-06T07:34:35.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.035+0000 7f73be60e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b815e010 0x7f73b8156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.041 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.035+0000 7f73be60e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b8157100 0x7f73b80c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73be60e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73b80c8700 0x7f73b80b5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73be60e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b80d2010 con 0x7f73b8157100 2024-03-06T07:34:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73be60e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b806ced0 con 0x7f73b80c8700 2024-03-06T07:34:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73be60e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b80d2190 con 0x7f73b815e010 2024-03-06T07:34:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b8157100 0x7f73b80c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b77fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b8157100 0x7f73b80c6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56338/0 (socket says 172.21.15.59:56338) 2024-03-06T07:34:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b77fe640 1 -- 172.21.15.59:0/4256809283 learned_addr learned my addr 172.21.15.59:0/4256809283 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73bcbab640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73b80c8700 0x7f73b80b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.042 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b7fff640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b815e010 0x7f73b8156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b77fe640 1 -- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b815e010 msgr2=0x7f73b8156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b77fe640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b815e010 0x7f73b8156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b77fe640 1 -- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73b80c8700 msgr2=0x7f73b80b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b77fe640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73b80c8700 0x7f73b80b5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b77fe640 1 -- 172.21.15.59:0/4256809283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b80b4960 con 0x7f73b8157100 2024-03-06T07:34:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b77fe640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b8157100 0x7f73b80c6420 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f73ac02b8a0 tx=0x7f73ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b4ff9640 1 -- 172.21.15.59:0/4256809283 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac032860 con 0x7f73b8157100 2024-03-06T07:34:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b4ff9640 1 -- 172.21.15.59:0/4256809283 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac032a20 con 0x7f73b8157100 2024-03-06T07:34:35.043 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73b4ff9640 1 -- 172.21.15.59:0/4256809283 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac032e10 con 0x7f73b8157100 2024-03-06T07:34:35.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73be60e640 1 -- 172.21.15.59:0/4256809283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b80b4c40 con 0x7f73b8157100 2024-03-06T07:34:35.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.036+0000 7f73be60e640 1 -- 172.21.15.59:0/4256809283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b80b5200 con 0x7f73b8157100 2024-03-06T07:34:35.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.037+0000 7f73be60e640 1 -- 172.21.15.59:0/4256809283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b804f330 con 0x7f73b8157100 2024-03-06T07:34:35.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.038+0000 7f73b4ff9640 1 -- 172.21.15.59:0/4256809283 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73ac04c060 con 0x7f73b8157100 2024-03-06T07:34:35.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.038+0000 7f73b4ff9640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73a406b500 0x7f73a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.038+0000 7f73b4ff9640 1 -- 172.21.15.59:0/4256809283 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f73ac0ab6e0 con 0x7f73b8157100 2024-03-06T07:34:35.044 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.038+0000 7f73b7fff640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73a406b500 0x7f73a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.039+0000 7f73b7fff640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73a406b500 0x7f73a406d9c0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f73a8006fd0 tx=0x7f73a8009060 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.045 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.042+0000 7f73b4ff9640 1 -- 172.21.15.59:0/4256809283 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73ac040070 con 0x7f73b8157100 2024-03-06T07:34:35.228 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.226+0000 7f73be60e640 1 -- 172.21.15.59:0/4256809283 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f73b80b4710 con 0x7f73a406b500 2024-03-06T07:34:35.229 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.227+0000 7f73b4ff9640 1 -- 172.21.15.59:0/4256809283 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f73b80b4710 con 0x7f73a406b500 2024-03-06T07:34:35.229 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 3 to deep-scrub 2024-03-06T07:34:35.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 -- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73a406b500 msgr2=0x7f73a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.231 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73a406b500 0x7f73a406d9c0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f73a8006fd0 tx=0x7f73a8009060 comp rx=0 tx=0).stop 2024-03-06T07:34:35.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 -- 172.21.15.59:0/4256809283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b8157100 msgr2=0x7f73b80c6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b8157100 0x7f73b80c6420 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f73ac02b8a0 tx=0x7f73ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 -- 172.21.15.59:0/4256809283 shutdown_connections 2024-03-06T07:34:35.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f73a406b500 0x7f73a406d9c0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73b80c8700 0x7f73b80b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73b8157100 0x7f73b80c6420 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.232 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 --2- 172.21.15.59:0/4256809283 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73b815e010 0x7f73b8156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 -- 172.21.15.59:0/4256809283 >> 172.21.15.59:0/4256809283 conn(0x7f73b809a9f0 msgr2=0x7f73b80af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 -- 172.21.15.59:0/4256809283 shutdown_connections 2024-03-06T07:34:35.233 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.229+0000 7f73be60e640 1 -- 172.21.15.59:0/4256809283 wait complete. 2024-03-06T07:34:35.242 INFO:tasks.ceph:Scrubbing osd.4 2024-03-06T07:34:35.242 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:35.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.345+0000 7fd109125640 1 Processor -- start 2024-03-06T07:34:35.347 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.345+0000 7fd109125640 1 -- start start 2024-03-06T07:34:35.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.346+0000 7fd109125640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1040d4630 0x7fd1040d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.346+0000 7fd109125640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd1040d2820 0x7fd1040cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.346+0000 7fd109125640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10415e380 con 0x7fd10415dfa0 2024-03-06T07:34:35.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.346+0000 7fd109125640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1040d2100 con 0x7fd1040d2820 2024-03-06T07:34:35.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.346+0000 7fd109125640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1040cf470 con 0x7fd1040d4630 2024-03-06T07:34:35.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.346+0000 7fd10359e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd1040d2820 0x7fd1040cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.349 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.346+0000 7fd10359e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd1040d2820 0x7fd1040cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56342/0 (socket says 172.21.15.59:56342) 2024-03-06T07:34:35.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.346+0000 7fd10359e640 1 -- 172.21.15.59:0/1571560810 learned_addr learned my addr 172.21.15.59:0/1571560810 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:35.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.346+0000 7fd10259c640 1 --2- 172.21.15.59:0/1571560810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1040d4630 0x7fd1040d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.347+0000 7fd10359e640 1 -- 172.21.15.59:0/1571560810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1040d4630 msgr2=0x7fd1040d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.347+0000 7fd10359e640 1 --2- 172.21.15.59:0/1571560810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1040d4630 0x7fd1040d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.347+0000 7fd10359e640 1 -- 172.21.15.59:0/1571560810 >> v1:172.21.15.164:6789/0 conn(0x7fd10415dfa0 legacy=0x7fd1040d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.350 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.347+0000 7fd10359e640 1 -- 172.21.15.59:0/1571560810 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1040cf750 con 0x7fd1040d2820 2024-03-06T07:34:35.353 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.347+0000 7fd10359e640 1 --2- 172.21.15.59:0/1571560810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1040d2820 0x7fd1040cb190 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009e50 tx=0x7fd0f402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185ee2f99993a345 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.347+0000 7fd101d9b640 1 -- 172.21.15.59:0/1571560810 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f402c9e0 con 0x7fd1040d2820 2024-03-06T07:34:35.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.347+0000 7fd101d9b640 1 -- 172.21.15.59:0/1571560810 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f402cb80 con 0x7fd1040d2820 2024-03-06T07:34:35.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.347+0000 7fd101d9b640 1 -- 172.21.15.59:0/1571560810 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40343f0 con 0x7fd1040d2820 2024-03-06T07:34:35.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.348+0000 7fd109125640 1 -- 172.21.15.59:0/1571560810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1040d2820 msgr2=0x7fd1040cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.348+0000 7fd109125640 1 --2- 172.21.15.59:0/1571560810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1040d2820 0x7fd1040cb190 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009e50 tx=0x7fd0f402be40 comp rx=0 tx=0).stop 2024-03-06T07:34:35.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.348+0000 7fd109125640 1 -- 172.21.15.59:0/1571560810 shutdown_connections 2024-03-06T07:34:35.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.348+0000 7fd109125640 1 --2- 172.21.15.59:0/1571560810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1040d2820 0x7fd1040cb190 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.354 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.348+0000 7fd109125640 1 --2- 172.21.15.59:0/1571560810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1040d4630 0x7fd1040d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.348+0000 7fd109125640 1 -- 172.21.15.59:0/1571560810 >> 172.21.15.59:0/1571560810 conn(0x7fd10409d000 msgr2=0x7fd1040d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:35.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.348+0000 7fd109125640 1 -- 172.21.15.59:0/1571560810 shutdown_connections 2024-03-06T07:34:35.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.348+0000 7fd109125640 1 -- 172.21.15.59:0/1571560810 wait complete. 2024-03-06T07:34:35.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.349+0000 7fd109125640 1 Processor -- start 2024-03-06T07:34:35.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.349+0000 7fd109125640 1 -- start start 2024-03-06T07:34:35.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd109125640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1040d2820 0x7fd10409ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd109125640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1040d4630 0x7fd1040b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.355 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd109125640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd10415dfa0 0x7fd1040c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd109125640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1040cb850 con 0x7fd1040d4630 2024-03-06T07:34:35.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd109125640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1040cb6d0 con 0x7fd1040d2820 2024-03-06T07:34:35.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd109125640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1040cb9d0 con 0x7fd10415dfa0 2024-03-06T07:34:35.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd10259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1040d4630 0x7fd1040b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd10259c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1040d4630 0x7fd1040b4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56354/0 (socket says 172.21.15.59:56354) 2024-03-06T07:34:35.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd10259c640 1 -- 172.21.15.59:0/1311388562 learned_addr learned my addr 172.21.15.59:0/1311388562 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:35.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd10359e640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd10415dfa0 0x7fd1040c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.356 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd102d9d640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1040d2820 0x7fd10409ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.350+0000 7fd10259c640 1 -- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd10415dfa0 msgr2=0x7fd1040c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.351+0000 7fd10259c640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd10415dfa0 0x7fd1040c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.351+0000 7fd10259c640 1 -- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1040d2820 msgr2=0x7fd10409ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.351+0000 7fd10259c640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1040d2820 0x7fd10409ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.351+0000 7fd10259c640 1 -- 172.21.15.59:0/1311388562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1040b9790 con 0x7fd1040d4630 2024-03-06T07:34:35.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.351+0000 7fd10259c640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1040d4630 0x7fd1040b4cc0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800c7f0 tx=0x7fd0f800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.351+0000 7fd0eb7fe640 1 -- 172.21.15.59:0/1311388562 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8015070 con 0x7fd1040d4630 2024-03-06T07:34:35.357 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.351+0000 7fd0eb7fe640 1 -- 172.21.15.59:0/1311388562 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f8004240 con 0x7fd1040d4630 2024-03-06T07:34:35.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.351+0000 7fd109125640 1 -- 172.21.15.59:0/1311388562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1040b94b0 con 0x7fd1040d4630 2024-03-06T07:34:35.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.351+0000 7fd0eb7fe640 1 -- 172.21.15.59:0/1311388562 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8007620 con 0x7fd1040d4630 2024-03-06T07:34:35.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.351+0000 7fd109125640 1 -- 172.21.15.59:0/1311388562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10409e690 con 0x7fd1040d4630 2024-03-06T07:34:35.358 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.352+0000 7fd109125640 1 -- 172.21.15.59:0/1311388562 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd10404f330 con 0x7fd1040d4630 2024-03-06T07:34:35.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.352+0000 7fd0eb7fe640 1 -- 172.21.15.59:0/1311388562 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0f80079d0 con 0x7fd1040d4630 2024-03-06T07:34:35.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.353+0000 7fd0eb7fe640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0f006b4d0 0x7fd0f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.359 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.353+0000 7fd0eb7fe640 1 -- 172.21.15.59:0/1311388562 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd0f808a2e0 con 0x7fd1040d4630 2024-03-06T07:34:35.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.353+0000 7fd102d9d640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0f006b4d0 0x7fd0f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.353+0000 7fd0eb7fe640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7fd0f0071430 0x7fd0f0073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.353+0000 7fd0eb7fe640 1 -- 172.21.15.59:0/1311388562 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0f0073f00 con 0x7fd0f0071430 2024-03-06T07:34:35.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.353+0000 7fd0eb7fe640 1 -- 172.21.15.59:0/1311388562 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0f8091050 con 0x7fd1040d4630 2024-03-06T07:34:35.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.353+0000 7fd10359e640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7fd0f0071430 0x7fd0f0073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.353+0000 7fd102d9d640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0f006b4d0 0x7fd0f006d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc006fd0 tx=0x7fd0fc008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.354+0000 7fd10359e640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7fd0f0071430 0x7fd0f0073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.360 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.354+0000 7fd0eb7fe640 1 -- 172.21.15.59:0/1311388562 <== osd.4 v2:172.21.15.164:6818/102497754 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd0f0073f00 con 0x7fd0f0071430 2024-03-06T07:34:35.369 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.368+0000 7fd109125640 1 -- 172.21.15.59:0/1311388562 --> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd1041610d0 con 0x7fd0f0071430 2024-03-06T07:34:35.370 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.368+0000 7fd0eb7fe640 1 -- 172.21.15.59:0/1311388562 <== osd.4 v2:172.21.15.164:6818/102497754 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd1041610d0 con 0x7fd0f0071430 2024-03-06T07:34:35.370 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:34:35.370 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:35.370 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:34:35.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.368+0000 7fd109125640 1 -- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7fd0f0071430 msgr2=0x7fd0f0073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.368+0000 7fd109125640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:6818/102497754,v1:172.21.15.164:6820/102497754] conn(0x7fd0f0071430 0x7fd0f0073850 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.368+0000 7fd109125640 1 -- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0f006b4d0 msgr2=0x7fd0f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.368+0000 7fd109125640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd0f006b4d0 0x7fd0f006d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc006fd0 tx=0x7fd0fc008040 comp rx=0 tx=0).stop 2024-03-06T07:34:35.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.369+0000 7fd109125640 1 -- 172.21.15.59:0/1311388562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1040d4630 msgr2=0x7fd1040b4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.369+0000 7fd109125640 1 --2- 172.21.15.59:0/1311388562 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd1040d4630 0x7fd1040b4cc0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800c7f0 tx=0x7fd0f800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.369+0000 7fd10359e640 1 -- 172.21.15.59:0/1311388562 reap_dead start 2024-03-06T07:34:35.372 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.369+0000 7fd109125640 1 -- 172.21.15.59:0/1311388562 shutdown_connections 2024-03-06T07:34:35.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.369+0000 7fd109125640 1 -- 172.21.15.59:0/1311388562 >> 172.21.15.59:0/1311388562 conn(0x7fd10409d000 msgr2=0x7fd104160560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:35.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.369+0000 7fd109125640 1 -- 172.21.15.59:0/1311388562 shutdown_connections 2024-03-06T07:34:35.373 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.369+0000 7fd109125640 1 -- 172.21.15.59:0/1311388562 wait complete. 2024-03-06T07:34:35.381 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-06T07:34:35.482 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.481+0000 7fe75e7eb640 1 Processor -- start 2024-03-06T07:34:35.483 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.481+0000 7fe75e7eb640 1 -- start start 2024-03-06T07:34:35.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.481+0000 7fe75e7eb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe758161670 0x7fe758072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.481+0000 7fe75e7eb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe758072a60 0x7fe758072ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.481+0000 7fe75e7eb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe758066d00 con 0x7fe7580adf10 2024-03-06T07:34:35.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.481+0000 7fe75e7eb640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7580b6170 con 0x7fe758072a60 2024-03-06T07:34:35.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.481+0000 7fe75e7eb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7580ada80 con 0x7fe758161670 2024-03-06T07:34:35.484 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.481+0000 7fe7577fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe758072a60 0x7fe758072ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.482+0000 7fe7577fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe758072a60 0x7fe758072ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56356/0 (socket says 172.21.15.59:56356) 2024-03-06T07:34:35.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.482+0000 7fe7577fe640 1 -- 172.21.15.59:0/3921281206 learned_addr learned my addr 172.21.15.59:0/3921281206 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:35.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.482+0000 7fe757fff640 1 --2- 172.21.15.59:0/3921281206 >> v2:172.21.15.164:3300/0 conn(0x7fe758161670 0x7fe758072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.482+0000 7fe7577fe640 1 -- 172.21.15.59:0/3921281206 >> v2:172.21.15.164:3300/0 conn(0x7fe758161670 msgr2=0x7fe758072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.482+0000 7fe7577fe640 1 --2- 172.21.15.59:0/3921281206 >> v2:172.21.15.164:3300/0 conn(0x7fe758161670 0x7fe758072520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.482+0000 7fe7577fe640 1 -- 172.21.15.59:0/3921281206 >> v1:172.21.15.164:6789/0 conn(0x7fe7580adf10 legacy=0x7fe7580ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.485 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.482+0000 7fe7577fe640 1 -- 172.21.15.59:0/3921281206 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe758066fc0 con 0x7fe758072a60 2024-03-06T07:34:35.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe7577fe640 1 --2- 172.21.15.59:0/3921281206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe758072a60 0x7fe758072ef0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fe758070a80 tx=0x7fe74802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da0fc1241866f238 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe756ffd640 1 -- 172.21.15.59:0/3921281206 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748034070 con 0x7fe758072a60 2024-03-06T07:34:35.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe756ffd640 1 -- 172.21.15.59:0/3921281206 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe748030860 con 0x7fe758072a60 2024-03-06T07:34:35.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe756ffd640 1 -- 172.21.15.59:0/3921281206 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748030c30 con 0x7fe758072a60 2024-03-06T07:34:35.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe75e7eb640 1 -- 172.21.15.59:0/3921281206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe758072a60 msgr2=0x7fe758072ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe75e7eb640 1 --2- 172.21.15.59:0/3921281206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe758072a60 0x7fe758072ef0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fe758070a80 tx=0x7fe74802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe75e7eb640 1 -- 172.21.15.59:0/3921281206 shutdown_connections 2024-03-06T07:34:35.487 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe75e7eb640 1 --2- 172.21.15.59:0/3921281206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe758072a60 0x7fe758072ef0 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe75e7eb640 1 --2- 172.21.15.59:0/3921281206 >> v2:172.21.15.164:3300/0 conn(0x7fe758161670 0x7fe758072520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe75e7eb640 1 -- 172.21.15.59:0/3921281206 >> 172.21.15.59:0/3921281206 conn(0x7fe7580b4e60 msgr2=0x7fe7580b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:35.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe75e7eb640 1 -- 172.21.15.59:0/3921281206 shutdown_connections 2024-03-06T07:34:35.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.483+0000 7fe75e7eb640 1 -- 172.21.15.59:0/3921281206 wait complete. 2024-03-06T07:34:35.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.484+0000 7fe75e7eb640 1 Processor -- start 2024-03-06T07:34:35.488 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.485+0000 7fe75e7eb640 1 -- start start 2024-03-06T07:34:35.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.485+0000 7fe75e7eb640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe758072a60 0x7fe7581f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.485+0000 7fe75e7eb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7580adf10 0x7fe758151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.485+0000 7fe75e7eb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe758161670 0x7fe75806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.485+0000 7fe75e7eb640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75806b330 con 0x7fe758072a60 2024-03-06T07:34:35.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.485+0000 7fe75e7eb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe758161a50 con 0x7fe7580adf10 2024-03-06T07:34:35.491 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.485+0000 7fe75e7eb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75806b610 con 0x7fe758161670 2024-03-06T07:34:35.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.485+0000 7fe757fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe758072a60 0x7fe7581f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.485+0000 7fe757fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe758072a60 0x7fe7581f3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56364/0 (socket says 172.21.15.59:56364) 2024-03-06T07:34:35.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.485+0000 7fe757fff640 1 -- 172.21.15.59:0/1595319714 learned_addr learned my addr 172.21.15.59:0/1595319714 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:35.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe7577fe640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7580adf10 0x7fe758151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe75cd88640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe758161670 0x7fe75806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe757fff640 1 -- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe758161670 msgr2=0x7fe75806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe757fff640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe758161670 0x7fe75806d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.492 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe757fff640 1 -- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7580adf10 msgr2=0x7fe758151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe757fff640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7580adf10 0x7fe758151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe757fff640 1 -- 172.21.15.59:0/1595319714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7580ced80 con 0x7fe758072a60 2024-03-06T07:34:35.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe75cd88640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe758161670 0x7fe75806d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:35.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe7577fe640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7580adf10 0x7fe758151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:35.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe757fff640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe758072a60 0x7fe7581f3ab0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00d5f0 tx=0x7fe74c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe754ff9640 1 -- 172.21.15.59:0/1595319714 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c0125f0 con 0x7fe758072a60 2024-03-06T07:34:35.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe754ff9640 1 -- 172.21.15.59:0/1595319714 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c019070 con 0x7fe758072a60 2024-03-06T07:34:35.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe75e7eb640 1 -- 172.21.15.59:0/1595319714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7580c9420 con 0x7fe758072a60 2024-03-06T07:34:35.493 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe754ff9640 1 -- 172.21.15.59:0/1595319714 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c0129d0 con 0x7fe758072a60 2024-03-06T07:34:35.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.486+0000 7fe75e7eb640 1 -- 172.21.15.59:0/1595319714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7581f3ff0 con 0x7fe758072a60 2024-03-06T07:34:35.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.487+0000 7fe75e7eb640 1 -- 172.21.15.59:0/1595319714 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75804f330 con 0x7fe758072a60 2024-03-06T07:34:35.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.488+0000 7fe754ff9640 1 -- 172.21.15.59:0/1595319714 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe74c012b70 con 0x7fe758072a60 2024-03-06T07:34:35.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.488+0000 7fe754ff9640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe74406b4d0 0x7fe74406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.488+0000 7fe754ff9640 1 -- 172.21.15.59:0/1595319714 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe74c08a2d0 con 0x7fe758072a60 2024-03-06T07:34:35.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.488+0000 7fe7577fe640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe74406b4d0 0x7fe74406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.494 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.489+0000 7fe7577fe640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe74406b4d0 0x7fe74406d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe748036000 tx=0x7fe74803b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.495 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.492+0000 7fe754ff9640 1 -- 172.21.15.59:0/1595319714 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe74c059480 con 0x7fe758072a60 2024-03-06T07:34:35.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.672+0000 7fe75e7eb640 1 -- 172.21.15.59:0/1595319714 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe758151ad0 con 0x7fe74406b4d0 2024-03-06T07:34:35.675 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.673+0000 7fe754ff9640 1 -- 172.21.15.59:0/1595319714 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe758151ad0 con 0x7fe74406b4d0 2024-03-06T07:34:35.675 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 4 to deep-scrub 2024-03-06T07:34:35.677 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 -- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe74406b4d0 msgr2=0x7fe74406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe74406b4d0 0x7fe74406d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe748036000 tx=0x7fe74803b040 comp rx=0 tx=0).stop 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 -- 172.21.15.59:0/1595319714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe758072a60 msgr2=0x7fe7581f3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe758072a60 0x7fe7581f3ab0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00d5f0 tx=0x7fe74c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 -- 172.21.15.59:0/1595319714 shutdown_connections 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe74406b4d0 0x7fe74406d990 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe758161670 0x7fe75806d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7580adf10 0x7fe758151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 --2- 172.21.15.59:0/1595319714 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe758072a60 0x7fe7581f3ab0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 -- 172.21.15.59:0/1595319714 >> 172.21.15.59:0/1595319714 conn(0x7fe7580b4e60 msgr2=0x7fe7581608a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 -- 172.21.15.59:0/1595319714 shutdown_connections 2024-03-06T07:34:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.675+0000 7fe75e7eb640 1 -- 172.21.15.59:0/1595319714 wait complete. 2024-03-06T07:34:35.688 INFO:tasks.ceph:Scrubbing osd.5 2024-03-06T07:34:35.688 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:35.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.791+0000 7f9e0fa4e640 1 Processor -- start 2024-03-06T07:34:35.793 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.791+0000 7f9e0fa4e640 1 -- start start 2024-03-06T07:34:35.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0fa4e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9e0815c5a0 0x7f9e0806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0fa4e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9e080ce8c0 0x7f9e080cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0fa4e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e0815c980 0x7f9e08071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0fa4e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e08072a20 con 0x7f9e0815c5a0 2024-03-06T07:34:35.795 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0fa4e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e080708b0 con 0x7f9e080ce8c0 2024-03-06T07:34:35.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0fa4e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08070e70 con 0x7f9e0815c980 2024-03-06T07:34:35.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0d7ea640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9e0815c5a0 0x7f9e0806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0d7ea640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9e0815c5a0 0x7f9e0806c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56380/0 (socket says 172.21.15.59:56380) 2024-03-06T07:34:35.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0d7ea640 1 -- 172.21.15.59:0/4187935448 learned_addr learned my addr 172.21.15.59:0/4187935448 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:35.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0dfeb640 1 --2- 172.21.15.59:0/4187935448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e0815c980 0x7f9e08071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0cfe9640 1 --2- 172.21.15.59:0/4187935448 >> v2:172.21.15.164:3300/0 conn(0x7f9e080ce8c0 0x7f9e080cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0d7ea640 1 -- 172.21.15.59:0/4187935448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e0815c980 msgr2=0x7f9e08071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.796 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0d7ea640 1 --2- 172.21.15.59:0/4187935448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e0815c980 0x7f9e08071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0d7ea640 1 -- 172.21.15.59:0/4187935448 >> v2:172.21.15.164:3300/0 conn(0x7f9e080ce8c0 msgr2=0x7f9e080cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0d7ea640 1 --2- 172.21.15.59:0/4187935448 >> v2:172.21.15.164:3300/0 conn(0x7f9e080ce8c0 0x7f9e080cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0d7ea640 1 -- 172.21.15.59:0/4187935448 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0806ce50 con 0x7f9e0815c5a0 2024-03-06T07:34:35.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.792+0000 7f9e0cfe9640 1 --2- 172.21.15.59:0/4187935448 >> v2:172.21.15.164:3300/0 conn(0x7f9e080ce8c0 0x7f9e080cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:35.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.793+0000 7f9e0d7ea640 1 --2- 172.21.15.59:0/4187935448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e0815c5a0 0x7f9e0806c910 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f9e08072310 tx=0x7f9e0402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e008e87753f1fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.793+0000 7f9dfffff640 1 -- 172.21.15.59:0/4187935448 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0402e070 con 0x7f9e0815c5a0 2024-03-06T07:34:35.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.793+0000 7f9dfffff640 1 -- 172.21.15.59:0/4187935448 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e04037070 con 0x7f9e0815c5a0 2024-03-06T07:34:35.797 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.793+0000 7f9dfffff640 1 -- 172.21.15.59:0/4187935448 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04031a00 con 0x7f9e0815c5a0 2024-03-06T07:34:35.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.794+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/4187935448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e0815c5a0 msgr2=0x7f9e0806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.801 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.794+0000 7f9e0fa4e640 1 --2- 172.21.15.59:0/4187935448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e0815c5a0 0x7f9e0806c910 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f9e08072310 tx=0x7f9e0402b880 comp rx=0 tx=0).stop 2024-03-06T07:34:35.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.794+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/4187935448 shutdown_connections 2024-03-06T07:34:35.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.794+0000 7f9e0fa4e640 1 --2- 172.21.15.59:0/4187935448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e0815c980 0x7f9e08071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.794+0000 7f9e0fa4e640 1 --2- 172.21.15.59:0/4187935448 >> v2:172.21.15.164:3300/0 conn(0x7f9e080ce8c0 0x7f9e080cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.794+0000 7f9e0fa4e640 1 --2- 172.21.15.59:0/4187935448 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e0815c5a0 0x7f9e0806c910 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.794+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/4187935448 >> 172.21.15.59:0/4187935448 conn(0x7f9e080b1650 msgr2=0x7f9e080b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:35.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.794+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/4187935448 shutdown_connections 2024-03-06T07:34:35.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.794+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/4187935448 wait complete. 2024-03-06T07:34:35.802 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.795+0000 7f9e0fa4e640 1 Processor -- start 2024-03-06T07:34:35.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.795+0000 7f9e0fa4e640 1 -- start start 2024-03-06T07:34:35.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0fa4e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e0815c5a0 0x7f9e081f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0fa4e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e0815c980 0x7f9e081e7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0fa4e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e081e4dd0 0x7f9e08158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0fa4e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08070b00 con 0x7f9e0815c5a0 2024-03-06T07:34:35.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0fa4e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080cf030 con 0x7f9e0815c980 2024-03-06T07:34:35.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0fa4e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080cc870 con 0x7f9e081e4dd0 2024-03-06T07:34:35.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0d7ea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e0815c5a0 0x7f9e081f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0d7ea640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e0815c5a0 0x7f9e081f1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56382/0 (socket says 172.21.15.59:56382) 2024-03-06T07:34:35.803 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0d7ea640 1 -- 172.21.15.59:0/3646960718 learned_addr learned my addr 172.21.15.59:0/3646960718 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:35.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0dfeb640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e081e4dd0 0x7f9e08158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0cfe9640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e0815c980 0x7f9e081e7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0d7ea640 1 -- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e081e4dd0 msgr2=0x7f9e08158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0d7ea640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e081e4dd0 0x7f9e08158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0d7ea640 1 -- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e0815c980 msgr2=0x7f9e081e7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0d7ea640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e0815c980 0x7f9e081e7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0d7ea640 1 -- 172.21.15.59:0/3646960718 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e081e5370 con 0x7f9e0815c5a0 2024-03-06T07:34:35.804 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9e0d7ea640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e0815c5a0 0x7f9e081f1e30 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f9e0402b8c0 tx=0x7f9e040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9dfdffb640 1 -- 172.21.15.59:0/3646960718 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04032670 con 0x7f9e0815c5a0 2024-03-06T07:34:35.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.796+0000 7f9dfdffb640 1 -- 172.21.15.59:0/3646960718 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e04032810 con 0x7f9e0815c5a0 2024-03-06T07:34:35.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.797+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/3646960718 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e08156810 con 0x7f9e0815c5a0 2024-03-06T07:34:35.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.797+0000 7f9dfdffb640 1 -- 172.21.15.59:0/3646960718 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04032c00 con 0x7f9e0815c5a0 2024-03-06T07:34:35.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.797+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/3646960718 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e081e83a0 con 0x7f9e0815c5a0 2024-03-06T07:34:35.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.798+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/3646960718 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e08156600 con 0x7f9e0815c5a0 2024-03-06T07:34:35.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.798+0000 7f9dfdffb640 1 -- 172.21.15.59:0/3646960718 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e040453f0 con 0x7f9e0815c5a0 2024-03-06T07:34:35.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.798+0000 7f9dfdffb640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9dec06b500 0x7f9dec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.805 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.798+0000 7f9dfdffb640 1 -- 172.21.15.59:0/3646960718 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9e040abdc0 con 0x7f9e0815c5a0 2024-03-06T07:34:35.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.799+0000 7f9e0cfe9640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9dec06b500 0x7f9dec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.799+0000 7f9dfdffb640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f9dec071460 0x7f9dec073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.799+0000 7f9dfdffb640 1 -- 172.21.15.59:0/3646960718 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e04044e70 con 0x7f9dec071460 2024-03-06T07:34:35.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.799+0000 7f9dfdffb640 1 -- 172.21.15.59:0/3646960718 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e040ae3e0 con 0x7f9e0815c5a0 2024-03-06T07:34:35.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.799+0000 7f9e0dfeb640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f9dec071460 0x7f9dec073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.799+0000 7f9e0cfe9640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9dec06b500 0x7f9dec06d9c0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f9df8006fd0 tx=0x7f9df8008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.806 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.799+0000 7f9e0dfeb640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f9dec071460 0x7f9dec073880 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.807 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.800+0000 7f9dfdffb640 1 -- 172.21.15.59:0/3646960718 <== osd.5 v2:172.21.15.164:6800/2889679248 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9e04044e70 con 0x7f9dec071460 2024-03-06T07:34:35.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.813+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/3646960718 --> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9e0804f330 con 0x7f9dec071460 2024-03-06T07:34:35.815 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9dfdffb640 1 -- 172.21.15.59:0/3646960718 <== osd.5 v2:172.21.15.164:6800/2889679248 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9e0804f330 con 0x7f9dec071460 2024-03-06T07:34:35.816 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:34:35.816 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:35.816 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:34:35.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f9dec071460 msgr2=0x7f9dec073880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9e0fa4e640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:6800/2889679248,v1:172.21.15.164:6801/2889679248] conn(0x7f9dec071460 0x7f9dec073880 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9dec06b500 msgr2=0x7f9dec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9e0fa4e640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9dec06b500 0x7f9dec06d9c0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f9df8006fd0 tx=0x7f9df8008040 comp rx=0 tx=0).stop 2024-03-06T07:34:35.817 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/3646960718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e0815c5a0 msgr2=0x7f9e081f1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9e0fa4e640 1 --2- 172.21.15.59:0/3646960718 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e0815c5a0 0x7f9e081f1e30 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f9e0402b8c0 tx=0x7f9e040046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9e0dfeb640 1 -- 172.21.15.59:0/3646960718 reap_dead start 2024-03-06T07:34:35.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/3646960718 shutdown_connections 2024-03-06T07:34:35.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/3646960718 >> 172.21.15.59:0/3646960718 conn(0x7f9e080b1650 msgr2=0x7f9e080b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:35.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/3646960718 shutdown_connections 2024-03-06T07:34:35.818 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.814+0000 7f9e0fa4e640 1 -- 172.21.15.59:0/3646960718 wait complete. 2024-03-06T07:34:35.826 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-06T07:34:35.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.927+0000 7fd2acf3a640 1 Processor -- start 2024-03-06T07:34:35.929 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.927+0000 7fd2acf3a640 1 -- start start 2024-03-06T07:34:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.927+0000 7fd2acf3a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a8162400 0x7fd2a80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.927+0000 7fd2acf3a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a80d22c0 con 0x7fd2a80b4460 2024-03-06T07:34:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.927+0000 7fd2acf3a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a80d25a0 con 0x7fd2a8162970 2024-03-06T07:34:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.927+0000 7fd2acf3a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a8161b00 con 0x7fd2a8162400 2024-03-06T07:34:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.927+0000 7fd2a659c640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd2a80b4460 0x7fd2a80b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59890/0 (socket says 172.21.15.59:59890) 2024-03-06T07:34:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.927+0000 7fd2a659c640 1 -- 172.21.15.59:0/976906705 learned_addr learned my addr 172.21.15.59:0/976906705 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a6d9d640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a8162400 0x7fd2a80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900508314 0 0) 0x7fd2a80d22c0 con 0x7fd2a80b4460 2024-03-06T07:34:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd294003520 con 0x7fd2a80b4460 2024-03-06T07:34:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2160224760 0 0) 0x7fd294003520 con 0x7fd2a80b4460 2024-03-06T07:34:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a8162400 msgr2=0x7fd2a80d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a8162400 0x7fd2a80d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 >> v1:172.21.15.164:6789/0 conn(0x7fd2a8162970 legacy=0x7fd2a8161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a80d2880 con 0x7fd2a80b4460 2024-03-06T07:34:35.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fd298001670 con 0x7fd2a80b4460 2024-03-06T07:34:35.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 >> v1:172.21.15.59:6789/0 conn(0x7fd2a80b4460 legacy=0x7fd2a80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a8162400 0x7fd2940039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.935 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd294003f20 0x7fd294004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 0x7fd2940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a8161b00 con 0x7fd294004d40 2024-03-06T07:34:35.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29400ba20 con 0x7fd294003f20 2024-03-06T07:34:35.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29400bc20 con 0x7fd2a8162400 2024-03-06T07:34:35.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a6d9d640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 0x7fd2940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a659c640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a8162400 0x7fd2940039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.936 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd29ffff640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd294003f20 0x7fd294004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a6d9d640 1 -- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a8162400 msgr2=0x7fd2940039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a6d9d640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a8162400 0x7fd2940039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a6d9d640 1 -- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd294003f20 msgr2=0x7fd294004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:35.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a6d9d640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd294003f20 0x7fd294004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.928+0000 7fd2a6d9d640 1 -- 172.21.15.59:0/976906705 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd294003520 con 0x7fd294004d40 2024-03-06T07:34:35.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.929+0000 7fd29ffff640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd294003f20 0x7fd294004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:35.937 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.929+0000 7fd2a6d9d640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 0x7fd2940051c0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009fc0 tx=0x7fd294005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5ed4f5a665db00 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.929+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0039070 con 0x7fd294004d40 2024-03-06T07:34:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.929+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a002cc20 con 0x7fd294004d40 2024-03-06T07:34:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.929+0000 7fd2a5d9b640 1 -- 172.21.15.59:0/976906705 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0009c10 con 0x7fd294004d40 2024-03-06T07:34:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.930+0000 7fd2acf3a640 1 -- 172.21.15.59:0/976906705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 msgr2=0x7fd2940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.930+0000 7fd2acf3a640 1 --2- 172.21.15.59:0/976906705 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 0x7fd2940051c0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009fc0 tx=0x7fd294005830 comp rx=0 tx=0).stop 2024-03-06T07:34:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.930+0000 7fd2a6d9d640 1 -- 172.21.15.59:0/976906705 reap_dead start 2024-03-06T07:34:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.930+0000 7fd2acf3a640 1 -- 172.21.15.59:0/976906705 shutdown_connections 2024-03-06T07:34:35.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.930+0000 7fd2acf3a640 1 -- 172.21.15.59:0/976906705 >> 172.21.15.59:0/976906705 conn(0x7fd2a80d3ce0 msgr2=0x7fd2a80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:35.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.930+0000 7fd2acf3a640 1 -- 172.21.15.59:0/976906705 shutdown_connections 2024-03-06T07:34:35.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.930+0000 7fd2acf3a640 1 -- 172.21.15.59:0/976906705 wait complete. 2024-03-06T07:34:35.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.931+0000 7fd2acf3a640 1 Processor -- start 2024-03-06T07:34:35.939 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.931+0000 7fd2acf3a640 1 -- start start 2024-03-06T07:34:35.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2acf3a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 0x7fd2a8156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2acf3a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2a80c8490 0x7fd2a80c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2acf3a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a80b1c00 0x7fd2a80b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2acf3a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a80ce950 con 0x7fd294004d40 2024-03-06T07:34:35.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2acf3a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a80cecc0 con 0x7fd2a80c8490 2024-03-06T07:34:35.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2acf3a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a80b4840 con 0x7fd2a80b1c00 2024-03-06T07:34:35.942 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2a659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 0x7fd2a8156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2a659c640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 0x7fd2a8156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56402/0 (socket says 172.21.15.59:56402) 2024-03-06T07:34:35.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2a659c640 1 -- 172.21.15.59:0/2935052850 learned_addr learned my addr 172.21.15.59:0/2935052850 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:35.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2a659c640 1 -- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a80b1c00 msgr2=0x7fd2a80b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2a659c640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a80b1c00 0x7fd2a80b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2a659c640 1 -- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2a80c8490 msgr2=0x7fd2a80c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:35.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2a659c640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2a80c8490 0x7fd2a80c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:35.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2a659c640 1 -- 172.21.15.59:0/2935052850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a80c8f10 con 0x7fd294004d40 2024-03-06T07:34:35.943 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.932+0000 7fd2a659c640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 0x7fd2a8156280 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fd298007310 tx=0x7fd2980077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.933+0000 7fd29effd640 1 -- 172.21.15.59:0/2935052850 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29800f980 con 0x7fd294004d40 2024-03-06T07:34:35.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.933+0000 7fd29effd640 1 -- 172.21.15.59:0/2935052850 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd298018070 con 0x7fd294004d40 2024-03-06T07:34:35.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.933+0000 7fd2acf3a640 1 -- 172.21.15.59:0/2935052850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a80c91f0 con 0x7fd294004d40 2024-03-06T07:34:35.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.933+0000 7fd29effd640 1 -- 172.21.15.59:0/2935052850 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29800fda0 con 0x7fd294004d40 2024-03-06T07:34:35.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.933+0000 7fd2acf3a640 1 -- 172.21.15.59:0/2935052850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a80cb150 con 0x7fd294004d40 2024-03-06T07:34:35.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.934+0000 7fd2acf3a640 1 -- 172.21.15.59:0/2935052850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a80cb350 con 0x7fd294004d40 2024-03-06T07:34:35.944 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.934+0000 7fd29effd640 1 -- 172.21.15.59:0/2935052850 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd29801d460 con 0x7fd294004d40 2024-03-06T07:34:35.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.934+0000 7fd29effd640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd28806b500 0x7fd28806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:35.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.934+0000 7fd29effd640 1 -- 172.21.15.59:0/2935052850 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd29808a110 con 0x7fd294004d40 2024-03-06T07:34:35.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.935+0000 7fd29ffff640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd28806b500 0x7fd28806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:35.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.935+0000 7fd29ffff640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd28806b500 0x7fd28806d9c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd290007fd0 tx=0x7fd290009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:35.945 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:35.939+0000 7fd29effd640 1 -- 172.21.15.59:0/2935052850 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2a80cb350 con 0x7fd294004d40 2024-03-06T07:34:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.120+0000 7fd2acf3a640 1 -- 172.21.15.59:0/2935052850 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2a804f330 con 0x7fd28806b500 2024-03-06T07:34:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.121+0000 7fd29effd640 1 -- 172.21.15.59:0/2935052850 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd2a804f330 con 0x7fd28806b500 2024-03-06T07:34:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 5 to deep-scrub 2024-03-06T07:34:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 -- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd28806b500 msgr2=0x7fd28806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd28806b500 0x7fd28806d9c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd290007fd0 tx=0x7fd290009040 comp rx=0 tx=0).stop 2024-03-06T07:34:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 -- 172.21.15.59:0/2935052850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 msgr2=0x7fd2a8156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 0x7fd2a8156280 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fd298007310 tx=0x7fd2980077d0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 -- 172.21.15.59:0/2935052850 shutdown_connections 2024-03-06T07:34:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fd28806b500 0x7fd28806d9c0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2a80b1c00 0x7fd2a80b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2a80c8490 0x7fd2a80c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 --2- 172.21.15.59:0/2935052850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd294004d40 0x7fd2a8156280 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 -- 172.21.15.59:0/2935052850 >> 172.21.15.59:0/2935052850 conn(0x7fd2a80d3ce0 msgr2=0x7fd2a806d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 -- 172.21.15.59:0/2935052850 shutdown_connections 2024-03-06T07:34:36.126 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.123+0000 7fd2acf3a640 1 -- 172.21.15.59:0/2935052850 wait complete. 2024-03-06T07:34:36.135 INFO:tasks.ceph:Scrubbing osd.6 2024-03-06T07:34:36.136 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:36.242 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.240+0000 7f6de7f1e640 1 Processor -- start 2024-03-06T07:34:36.243 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.241+0000 7f6de7f1e640 1 -- start start 2024-03-06T07:34:36.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.241+0000 7f6de7f1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6de00cdb30 0x7f6de00d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.245 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.241+0000 7f6de7f1e640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6de00b3a40 0x7f6de00ced40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.241+0000 7f6de7f1e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de006d760 con 0x7f6de0144690 2024-03-06T07:34:36.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.241+0000 7f6de7f1e640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de00d0830 con 0x7f6de00b3a40 2024-03-06T07:34:36.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.241+0000 7f6de7f1e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de0144a70 con 0x7f6de00cdb30 2024-03-06T07:34:36.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.241+0000 7f6de64bb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6de00b3a40 0x7f6de00ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.241+0000 7f6de64bb640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6de00b3a40 0x7f6de00ced40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56416/0 (socket says 172.21.15.59:56416) 2024-03-06T07:34:36.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.241+0000 7f6de64bb640 1 -- 172.21.15.59:0/3487070218 learned_addr learned my addr 172.21.15.59:0/3487070218 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:36.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.242+0000 7f6de54b9640 1 --2- 172.21.15.59:0/3487070218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6de00cdb30 0x7f6de00d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.242+0000 7f6de64bb640 1 -- 172.21.15.59:0/3487070218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6de00cdb30 msgr2=0x7f6de00d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:36.246 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.242+0000 7f6de64bb640 1 --2- 172.21.15.59:0/3487070218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6de00cdb30 0x7f6de00d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.242+0000 7f6de64bb640 1 -- 172.21.15.59:0/3487070218 >> v1:172.21.15.164:6789/0 conn(0x7f6de0144690 legacy=0x7f6de00cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.242+0000 7f6de64bb640 1 -- 172.21.15.59:0/3487070218 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de015f660 con 0x7f6de00b3a40 2024-03-06T07:34:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.242+0000 7f6de64bb640 1 --2- 172.21.15.59:0/3487070218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de00b3a40 0x7f6de00ced40 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800bb10 tx=0x7f6dd802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3faf3ab713df7e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.243+0000 7f6de4cb8640 1 -- 172.21.15.59:0/3487070218 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd803b070 con 0x7f6de00b3a40 2024-03-06T07:34:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.243+0000 7f6de4cb8640 1 -- 172.21.15.59:0/3487070218 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd8036440 con 0x7f6de00b3a40 2024-03-06T07:34:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.243+0000 7f6de4cb8640 1 -- 172.21.15.59:0/3487070218 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8036810 con 0x7f6de00b3a40 2024-03-06T07:34:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.243+0000 7f6de7f1e640 1 -- 172.21.15.59:0/3487070218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de00b3a40 msgr2=0x7f6de00ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.247 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.243+0000 7f6de7f1e640 1 --2- 172.21.15.59:0/3487070218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de00b3a40 0x7f6de00ced40 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800bb10 tx=0x7f6dd802e150 comp rx=0 tx=0).stop 2024-03-06T07:34:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.243+0000 7f6de7f1e640 1 -- 172.21.15.59:0/3487070218 shutdown_connections 2024-03-06T07:34:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.243+0000 7f6de7f1e640 1 --2- 172.21.15.59:0/3487070218 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de00b3a40 0x7f6de00ced40 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.243+0000 7f6de7f1e640 1 --2- 172.21.15.59:0/3487070218 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6de00cdb30 0x7f6de00d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.243+0000 7f6de7f1e640 1 -- 172.21.15.59:0/3487070218 >> 172.21.15.59:0/3487070218 conn(0x7f6de00d09b0 msgr2=0x7f6de00d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.243+0000 7f6de7f1e640 1 -- 172.21.15.59:0/3487070218 shutdown_connections 2024-03-06T07:34:36.248 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.243+0000 7f6de7f1e640 1 -- 172.21.15.59:0/3487070218 wait complete. 2024-03-06T07:34:36.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.244+0000 7f6de7f1e640 1 Processor -- start 2024-03-06T07:34:36.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de7f1e640 1 -- start start 2024-03-06T07:34:36.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de7f1e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6de00b3a40 0x7f6de00afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de7f1e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de00cdb30 0x7f6de01f4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de7f1e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6de0144690 0x7f6de013a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de7f1e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de015f280 con 0x7f6de00cdb30 2024-03-06T07:34:36.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de7f1e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de006dd00 con 0x7f6de00b3a40 2024-03-06T07:34:36.252 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de7f1e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de015f400 con 0x7f6de0144690 2024-03-06T07:34:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de54b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de00cdb30 0x7f6de01f4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de54b9640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de00cdb30 0x7f6de01f4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56424/0 (socket says 172.21.15.59:56424) 2024-03-06T07:34:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de54b9640 1 -- 172.21.15.59:0/2819439509 learned_addr learned my addr 172.21.15.59:0/2819439509 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de64bb640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6de0144690 0x7f6de013a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.245+0000 7f6de5cba640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6de00b3a40 0x7f6de00afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6de54b9640 1 -- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6de0144690 msgr2=0x7f6de013a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6de54b9640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6de0144690 0x7f6de013a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6de54b9640 1 -- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6de00b3a40 msgr2=0x7f6de00afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.253 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6de54b9640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6de00b3a40 0x7f6de00afd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6de54b9640 1 -- 172.21.15.59:0/2819439509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de00ada00 con 0x7f6de00cdb30 2024-03-06T07:34:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6de64bb640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6de0144690 0x7f6de013a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6de54b9640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de00cdb30 0x7f6de01f4be0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000d800 tx=0x7f6dd000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6dd67fc640 1 -- 172.21.15.59:0/2819439509 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd000b6c0 con 0x7f6de00cdb30 2024-03-06T07:34:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6dd67fc640 1 -- 172.21.15.59:0/2819439509 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd0019070 con 0x7f6de00cdb30 2024-03-06T07:34:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6de7f1e640 1 -- 172.21.15.59:0/2819439509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de00adce0 con 0x7f6de00cdb30 2024-03-06T07:34:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6dd67fc640 1 -- 172.21.15.59:0/2819439509 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd000ba80 con 0x7f6de00cdb30 2024-03-06T07:34:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.246+0000 7f6de7f1e640 1 -- 172.21.15.59:0/2819439509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de0158920 con 0x7f6de00cdb30 2024-03-06T07:34:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.247+0000 7f6de7f1e640 1 -- 172.21.15.59:0/2819439509 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6de00ae3a0 con 0x7f6de00cdb30 2024-03-06T07:34:36.254 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.247+0000 7f6dd67fc640 1 -- 172.21.15.59:0/2819439509 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6dd000bc20 con 0x7f6de00cdb30 2024-03-06T07:34:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.248+0000 7f6dd67fc640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6dc406b4d0 0x7f6dc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.248+0000 7f6dd67fc640 1 -- 172.21.15.59:0/2819439509 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6dd008a430 con 0x7f6de00cdb30 2024-03-06T07:34:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.248+0000 7f6dd67fc640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f6dc4071430 0x7f6dc4073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.248+0000 7f6dd67fc640 1 -- 172.21.15.59:0/2819439509 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dd0001d30 con 0x7f6dc4071430 2024-03-06T07:34:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.248+0000 7f6dd67fc640 1 -- 172.21.15.59:0/2819439509 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6dd0091050 con 0x7f6de00cdb30 2024-03-06T07:34:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.248+0000 7f6de5cba640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6dc406b4d0 0x7f6dc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.248+0000 7f6de64bb640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f6dc4071430 0x7f6dc4073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.248+0000 7f6de5cba640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6dc406b4d0 0x7f6dc406d990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc001ca0 tx=0x7f6ddc009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.255 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.249+0000 7f6de64bb640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f6dc4071430 0x7f6dc4073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.256 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.249+0000 7f6dd67fc640 1 -- 172.21.15.59:0/2819439509 <== osd.6 v2:172.21.15.164:6816/1594998822 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6dd0001d30 con 0x7f6dc4071430 2024-03-06T07:34:36.264 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.262+0000 7f6de7f1e640 1 -- 172.21.15.59:0/2819439509 --> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6de00d0030 con 0x7f6dc4071430 2024-03-06T07:34:36.265 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.263+0000 7f6dd67fc640 1 -- 172.21.15.59:0/2819439509 <== osd.6 v2:172.21.15.164:6816/1594998822 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6de00d0030 con 0x7f6dc4071430 2024-03-06T07:34:36.265 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:34:36.265 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:36.265 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:34:36.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.263+0000 7f6de7f1e640 1 -- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f6dc4071430 msgr2=0x7f6dc4073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.263+0000 7f6de7f1e640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:6816/1594998822,v1:172.21.15.164:6817/1594998822] conn(0x7f6dc4071430 0x7f6dc4073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.266 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.263+0000 7f6de7f1e640 1 -- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6dc406b4d0 msgr2=0x7f6dc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.263+0000 7f6de7f1e640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f6dc406b4d0 0x7f6dc406d990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc001ca0 tx=0x7f6ddc009040 comp rx=0 tx=0).stop 2024-03-06T07:34:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.263+0000 7f6de7f1e640 1 -- 172.21.15.59:0/2819439509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de00cdb30 msgr2=0x7f6de01f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.263+0000 7f6de7f1e640 1 --2- 172.21.15.59:0/2819439509 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de00cdb30 0x7f6de01f4be0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000d800 tx=0x7f6dd000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.263+0000 7f6de64bb640 1 -- 172.21.15.59:0/2819439509 reap_dead start 2024-03-06T07:34:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.264+0000 7f6de7f1e640 1 -- 172.21.15.59:0/2819439509 shutdown_connections 2024-03-06T07:34:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.264+0000 7f6de7f1e640 1 -- 172.21.15.59:0/2819439509 >> 172.21.15.59:0/2819439509 conn(0x7f6de00d09b0 msgr2=0x7f6de00b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.264+0000 7f6de7f1e640 1 -- 172.21.15.59:0/2819439509 shutdown_connections 2024-03-06T07:34:36.267 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.264+0000 7f6de7f1e640 1 -- 172.21.15.59:0/2819439509 wait complete. 2024-03-06T07:34:36.275 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-06T07:34:36.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.376+0000 7fe5dbf08640 1 Processor -- start 2024-03-06T07:34:36.378 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.376+0000 7fe5dbf08640 1 -- start start 2024-03-06T07:34:36.379 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5dbf08640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe5d40b4460 0x7fe5d40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5dbf08640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe5d4161cf0 0x7fe5d4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5dbf08640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d40d22c0 con 0x7fe5d4162970 2024-03-06T07:34:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5dbf08640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d4161b00 con 0x7fe5d4161cf0 2024-03-06T07:34:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5dbf08640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d40cecc0 con 0x7fe5d40b4460 2024-03-06T07:34:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5da4a5640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe5d4161cf0 0x7fe5d4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5d94a3640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe5d4162970 0x7fe5d40b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59900/0 (socket says 172.21.15.59:59900) 2024-03-06T07:34:36.380 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5d94a3640 1 -- 172.21.15.59:0/1238837742 learned_addr learned my addr 172.21.15.59:0/1238837742 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5d9ca4640 1 --2- 172.21.15.59:0/1238837742 >> v2:172.21.15.164:3300/0 conn(0x7fe5d40b4460 0x7fe5d40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5da4a5640 1 -- 172.21.15.59:0/1238837742 >> v2:172.21.15.164:3300/0 conn(0x7fe5d40b4460 msgr2=0x7fe5d40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5da4a5640 1 --2- 172.21.15.59:0/1238837742 >> v2:172.21.15.164:3300/0 conn(0x7fe5d40b4460 0x7fe5d40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5da4a5640 1 -- 172.21.15.59:0/1238837742 >> v1:172.21.15.59:6789/0 conn(0x7fe5d4162970 legacy=0x7fe5d40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5da4a5640 1 -- 172.21.15.59:0/1238837742 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d40d25a0 con 0x7fe5d4161cf0 2024-03-06T07:34:36.381 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.377+0000 7fe5d8ca2640 1 -- 172.21.15.59:0/1238837742 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071773276 0 0) 0x7fe5d40d22c0 con 0x7fe5d4162970 2024-03-06T07:34:36.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.378+0000 7fe5da4a5640 1 --2- 172.21.15.59:0/1238837742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d4161cf0 0x7fe5d4162180 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fe5d408ddd0 tx=0x7fe5cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a028ee13435c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.378+0000 7fe5d8ca2640 1 -- 172.21.15.59:0/1238837742 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc039070 con 0x7fe5d4161cf0 2024-03-06T07:34:36.385 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.378+0000 7fe5d8ca2640 1 -- 172.21.15.59:0/1238837742 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5cc02cc80 con 0x7fe5d4161cf0 2024-03-06T07:34:36.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.378+0000 7fe5d8ca2640 1 -- 172.21.15.59:0/1238837742 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc034660 con 0x7fe5d4161cf0 2024-03-06T07:34:36.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.379+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1238837742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d4161cf0 msgr2=0x7fe5d4162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.379+0000 7fe5dbf08640 1 --2- 172.21.15.59:0/1238837742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d4161cf0 0x7fe5d4162180 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fe5d408ddd0 tx=0x7fe5cc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.379+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1238837742 shutdown_connections 2024-03-06T07:34:36.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.379+0000 7fe5dbf08640 1 --2- 172.21.15.59:0/1238837742 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d4161cf0 0x7fe5d4162180 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.379+0000 7fe5dbf08640 1 --2- 172.21.15.59:0/1238837742 >> v2:172.21.15.164:3300/0 conn(0x7fe5d40b4460 0x7fe5d40d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.379+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1238837742 >> 172.21.15.59:0/1238837742 conn(0x7fe5d40d3ce0 msgr2=0x7fe5d40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:36.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.379+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1238837742 shutdown_connections 2024-03-06T07:34:36.386 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.379+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1238837742 wait complete. 2024-03-06T07:34:36.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.380+0000 7fe5dbf08640 1 Processor -- start 2024-03-06T07:34:36.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.380+0000 7fe5dbf08640 1 -- start start 2024-03-06T07:34:36.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5dbf08640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5d40b4460 0x7fe5d40b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5dbf08640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d4161cf0 0x7fe5d4154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5dbf08640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5d4162970 0x7fe5d40c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5dbf08640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d40ce950 con 0x7fe5d4161cf0 2024-03-06T07:34:36.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5dbf08640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d40d2b60 con 0x7fe5d40b4460 2024-03-06T07:34:36.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5dbf08640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d40b4840 con 0x7fe5d4162970 2024-03-06T07:34:36.387 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5d94a3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d4161cf0 0x7fe5d4154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5d94a3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d4161cf0 0x7fe5d4154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56450/0 (socket says 172.21.15.59:56450) 2024-03-06T07:34:36.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5d94a3640 1 -- 172.21.15.59:0/1985265921 learned_addr learned my addr 172.21.15.59:0/1985265921 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:36.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5da4a5640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5d4162970 0x7fe5d40c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5d9ca4640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5d40b4460 0x7fe5d40b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5d94a3640 1 -- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5d4162970 msgr2=0x7fe5d40c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5d94a3640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5d4162970 0x7fe5d40c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5d94a3640 1 -- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5d40b4460 msgr2=0x7fe5d40b7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5d94a3640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5d40b4460 0x7fe5d40b7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.388 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5d94a3640 1 -- 172.21.15.59:0/1985265921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d40b6a30 con 0x7fe5d4161cf0 2024-03-06T07:34:36.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5da4a5640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5d4162970 0x7fe5d40c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:36.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5d9ca4640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5d40b4460 0x7fe5d40b7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:36.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5d94a3640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d4161cf0 0x7fe5d4154db0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400c4b0 tx=0x7fe5c400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5ca7fc640 1 -- 172.21.15.59:0/1985265921 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4018070 con 0x7fe5d4161cf0 2024-03-06T07:34:36.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5ca7fc640 1 -- 172.21.15.59:0/1985265921 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c4007500 con 0x7fe5d4161cf0 2024-03-06T07:34:36.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5ca7fc640 1 -- 172.21.15.59:0/1985265921 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4004a20 con 0x7fe5d4161cf0 2024-03-06T07:34:36.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.381+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1985265921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d40b6d10 con 0x7fe5d4161cf0 2024-03-06T07:34:36.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.382+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1985265921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d40b72d0 con 0x7fe5d4161cf0 2024-03-06T07:34:36.389 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.382+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1985265921 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d404f330 con 0x7fe5d4161cf0 2024-03-06T07:34:36.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.383+0000 7fe5ca7fc640 1 -- 172.21.15.59:0/1985265921 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5c401b070 con 0x7fe5d4161cf0 2024-03-06T07:34:36.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.383+0000 7fe5ca7fc640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5b806b4d0 0x7fe5b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.383+0000 7fe5ca7fc640 1 -- 172.21.15.59:0/1985265921 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe5c40892a0 con 0x7fe5d4161cf0 2024-03-06T07:34:36.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.384+0000 7fe5d9ca4640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5b806b4d0 0x7fe5b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.390 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.384+0000 7fe5d9ca4640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5b806b4d0 0x7fe5b806d990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0006fd0 tx=0x7fe5d0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.391 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.387+0000 7fe5ca7fc640 1 -- 172.21.15.59:0/1985265921 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5c4058470 con 0x7fe5d4161cf0 2024-03-06T07:34:36.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.571+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1985265921 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5d406b180 con 0x7fe5b806b4d0 2024-03-06T07:34:36.573 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.571+0000 7fe5ca7fc640 1 -- 172.21.15.59:0/1985265921 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe5d406b180 con 0x7fe5b806b4d0 2024-03-06T07:34:36.573 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 6 to deep-scrub 2024-03-06T07:34:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5b806b4d0 msgr2=0x7fe5b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5b806b4d0 0x7fe5b806d990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0006fd0 tx=0x7fe5d0008040 comp rx=0 tx=0).stop 2024-03-06T07:34:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1985265921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d4161cf0 msgr2=0x7fe5d4154db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d4161cf0 0x7fe5d4154db0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fe5c400c4b0 tx=0x7fe5c400c970 comp rx=0 tx=0).stop 2024-03-06T07:34:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1985265921 shutdown_connections 2024-03-06T07:34:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe5b806b4d0 0x7fe5b806d990 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5d4162970 0x7fe5d40c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.576 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe5d4161cf0 0x7fe5d4154db0 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 --2- 172.21.15.59:0/1985265921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5d40b4460 0x7fe5d40b7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1985265921 >> 172.21.15.59:0/1985265921 conn(0x7fe5d40d3ce0 msgr2=0x7fe5d40b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1985265921 shutdown_connections 2024-03-06T07:34:36.577 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.573+0000 7fe5dbf08640 1 -- 172.21.15.59:0/1985265921 wait complete. 2024-03-06T07:34:36.586 INFO:tasks.ceph:Scrubbing osd.7 2024-03-06T07:34:36.586 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:34:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.689+0000 7fdc4d3e4640 1 Processor -- start 2024-03-06T07:34:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.689+0000 7fdc4d3e4640 1 -- start start 2024-03-06T07:34:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.689+0000 7fdc4d3e4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480cfe00 0x7fdc481690b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.689+0000 7fdc4d3e4640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4815f7c0 con 0x7fdc4815f290 2024-03-06T07:34:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.689+0000 7fdc4d3e4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc480ccc60 con 0x7fdc4815e180 2024-03-06T07:34:36.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.689+0000 7fdc4d3e4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc480d0290 con 0x7fdc480cfe00 2024-03-06T07:34:36.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.689+0000 7fdc467fc640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdc4815f290 0x7fdc480cf6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59914/0 (socket says 172.21.15.59:59914) 2024-03-06T07:34:36.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc467fc640 1 -- 172.21.15.59:0/2015636080 learned_addr learned my addr 172.21.15.59:0/2015636080 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:36.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783304804 0 0) 0x7fdc4815f7c0 con 0x7fdc4815f290 2024-03-06T07:34:36.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc30003520 con 0x7fdc4815f290 2024-03-06T07:34:36.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc477fe640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480cfe00 0x7fdc481690b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1293827732 0 0) 0x7fdc30003520 con 0x7fdc4815f290 2024-03-06T07:34:36.695 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480cfe00 msgr2=0x7fdc481690b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480cfe00 0x7fdc481690b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 >> v1:172.21.15.164:6789/0 conn(0x7fdc4815e180 legacy=0x7fdc480cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc480ccf40 con 0x7fdc4815f290 2024-03-06T07:34:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7fdc3c001670 con 0x7fdc4815f290 2024-03-06T07:34:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 >> v1:172.21.15.59:6789/0 conn(0x7fdc4815f290 legacy=0x7fdc480cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc300039e0 0x7fdc30003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480cfe00 0x7fdc30004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc30004d40 0x7fdc300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3000bb50 con 0x7fdc30004d40 2024-03-06T07:34:36.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3000bd00 con 0x7fdc300039e0 2024-03-06T07:34:36.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3000bf00 con 0x7fdc480cfe00 2024-03-06T07:34:36.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc477fe640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc30004d40 0x7fdc300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc46ffd640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc300039e0 0x7fdc30003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.690+0000 7fdc467fc640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480cfe00 0x7fdc30004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.691+0000 7fdc477fe640 1 -- 172.21.15.59:0/2015636080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480cfe00 msgr2=0x7fdc30004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:36.697 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.691+0000 7fdc477fe640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480cfe00 0x7fdc30004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.691+0000 7fdc477fe640 1 -- 172.21.15.59:0/2015636080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc300039e0 msgr2=0x7fdc30003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.691+0000 7fdc477fe640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc300039e0 0x7fdc30003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.691+0000 7fdc477fe640 1 -- 172.21.15.59:0/2015636080 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc30003520 con 0x7fdc30004d40 2024-03-06T07:34:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.691+0000 7fdc477fe640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc30004d40 0x7fdc300051c0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009fc0 tx=0x7fdc30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd0747edfdfff1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.691+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40039070 con 0x7fdc30004d40 2024-03-06T07:34:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.691+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4002cc20 con 0x7fdc30004d40 2024-03-06T07:34:36.698 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.691+0000 7fdc45ffb640 1 -- 172.21.15.59:0/2015636080 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc40009c10 con 0x7fdc30004d40 2024-03-06T07:34:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.692+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2015636080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc30004d40 msgr2=0x7fdc300051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.692+0000 7fdc4d3e4640 1 --2- 172.21.15.59:0/2015636080 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc30004d40 0x7fdc300051c0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009fc0 tx=0x7fdc30005830 comp rx=0 tx=0).stop 2024-03-06T07:34:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.692+0000 7fdc477fe640 1 -- 172.21.15.59:0/2015636080 reap_dead start 2024-03-06T07:34:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.692+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2015636080 shutdown_connections 2024-03-06T07:34:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.692+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2015636080 >> 172.21.15.59:0/2015636080 conn(0x7fdc480d1b20 msgr2=0x7fdc480d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.692+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2015636080 shutdown_connections 2024-03-06T07:34:36.699 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.692+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2015636080 wait complete. 2024-03-06T07:34:36.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.693+0000 7fdc4d3e4640 1 Processor -- start 2024-03-06T07:34:36.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.693+0000 7fdc4d3e4640 1 -- start start 2024-03-06T07:34:36.702 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc4d3e4640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc48155180 0x7fdc480cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc4d3e4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480c7540 0x7fdc480c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc4d3e4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc480c5630 0x7fdc480c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc4d3e4640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc480b2ee0 con 0x7fdc48155180 2024-03-06T07:34:36.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc4d3e4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc480cc910 con 0x7fdc480c5630 2024-03-06T07:34:36.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc4d3e4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4815e560 con 0x7fdc480c7540 2024-03-06T07:34:36.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc46ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc48155180 0x7fdc480cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc46ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc48155180 0x7fdc480cb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56476/0 (socket says 172.21.15.59:56476) 2024-03-06T07:34:36.703 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc46ffd640 1 -- 172.21.15.59:0/2935817424 learned_addr learned my addr 172.21.15.59:0/2935817424 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc467fc640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480c7540 0x7fdc480c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc477fe640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc480c5630 0x7fdc480c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc46ffd640 1 -- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480c7540 msgr2=0x7fdc480c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc46ffd640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdc480c7540 0x7fdc480c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc46ffd640 1 -- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc480c5630 msgr2=0x7fdc480c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc46ffd640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc480c5630 0x7fdc480c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.694+0000 7fdc46ffd640 1 -- 172.21.15.59:0/2935817424 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc480c6970 con 0x7fdc48155180 2024-03-06T07:34:36.704 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.695+0000 7fdc46ffd640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc48155180 0x7fdc480cb350 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800f820 tx=0x7fdc3800fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.695+0000 7fdc2f7fe640 1 -- 172.21.15.59:0/2935817424 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc380050d0 con 0x7fdc48155180 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.695+0000 7fdc2f7fe640 1 -- 172.21.15.59:0/2935817424 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc38018070 con 0x7fdc48155180 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.695+0000 7fdc2f7fe640 1 -- 172.21.15.59:0/2935817424 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38005490 con 0x7fdc48155180 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.695+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2935817424 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc48161b90 con 0x7fdc48155180 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.695+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2935817424 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc48162150 con 0x7fdc48155180 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.696+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2935817424 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdc4804f330 con 0x7fdc48155180 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.696+0000 7fdc2f7fe640 1 -- 172.21.15.59:0/2935817424 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc3800d730 con 0x7fdc48155180 2024-03-06T07:34:36.705 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.697+0000 7fdc2f7fe640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdc3406b500 0x7fdc3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.697+0000 7fdc2f7fe640 1 -- 172.21.15.59:0/2935817424 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdc38089eb0 con 0x7fdc48155180 2024-03-06T07:34:36.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.697+0000 7fdc2f7fe640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x7fdc34071460 0x7fdc34073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.697+0000 7fdc2f7fe640 1 -- 172.21.15.59:0/2935817424 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc3800ae50 con 0x7fdc34071460 2024-03-06T07:34:36.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.697+0000 7fdc467fc640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdc3406b500 0x7fdc3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.697+0000 7fdc2f7fe640 1 -- 172.21.15.59:0/2935817424 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=146) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc3808c5b0 con 0x7fdc48155180 2024-03-06T07:34:36.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.697+0000 7fdc477fe640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x7fdc34071460 0x7fdc34073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.697+0000 7fdc467fc640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdc3406b500 0x7fdc3406d9c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fdc481f7c00 tx=0x7fdc3c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.706 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.698+0000 7fdc477fe640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x7fdc34071460 0x7fdc34073880 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.707 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.698+0000 7fdc2f7fe640 1 -- 172.21.15.59:0/2935817424 <== osd.7 v2:172.21.15.164:6808/3612340295 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc3800ae50 con 0x7fdc34071460 2024-03-06T07:34:36.713 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2935817424 --> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdc480c8bf0 con 0x7fdc34071460 2024-03-06T07:34:36.714 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc2f7fe640 1 -- 172.21.15.59:0/2935817424 <== osd.7 v2:172.21.15.164:6808/3612340295 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fdc480c8bf0 con 0x7fdc34071460 2024-03-06T07:34:36.714 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2024-03-06T07:34:36.714 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:34:36.714 INFO:teuthology.orchestra.run.smithi059.stdout:} 2024-03-06T07:34:36.715 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x7fdc34071460 msgr2=0x7fdc34073880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc4d3e4640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:6808/3612340295,v1:172.21.15.164:6809/3612340295] conn(0x7fdc34071460 0x7fdc34073880 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdc3406b500 msgr2=0x7fdc3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc4d3e4640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fdc3406b500 0x7fdc3406d9c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fdc481f7c00 tx=0x7fdc3c009040 comp rx=0 tx=0).stop 2024-03-06T07:34:36.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2935817424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc48155180 msgr2=0x7fdc480cb350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc4d3e4640 1 --2- 172.21.15.59:0/2935817424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc48155180 0x7fdc480cb350 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800f820 tx=0x7fdc3800fce0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc477fe640 1 -- 172.21.15.59:0/2935817424 reap_dead start 2024-03-06T07:34:36.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2935817424 shutdown_connections 2024-03-06T07:34:36.716 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2935817424 >> 172.21.15.59:0/2935817424 conn(0x7fdc480d1b20 msgr2=0x7fdc480d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:36.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2935817424 shutdown_connections 2024-03-06T07:34:36.717 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.712+0000 7fdc4d3e4640 1 -- 172.21.15.59:0/2935817424 wait complete. 2024-03-06T07:34:36.725 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-06T07:34:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.824+0000 7f9a7769a640 1 Processor -- start 2024-03-06T07:34:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.824+0000 7f9a7769a640 1 -- start start 2024-03-06T07:34:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.825+0000 7f9a7769a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700b4460 0x7f9a700d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.825+0000 7f9a7769a640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a700d22c0 con 0x7f9a70162970 2024-03-06T07:34:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.825+0000 7f9a7769a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a700d25a0 con 0x7f9a70161cf0 2024-03-06T07:34:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.825+0000 7f9a7769a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a70161b00 con 0x7f9a700b4460 2024-03-06T07:34:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.825+0000 7f9a74c35640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9a70162970 0x7f9a700b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59920/0 (socket says 172.21.15.59:59920) 2024-03-06T07:34:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.825+0000 7f9a74c35640 1 -- 172.21.15.59:0/2119375997 learned_addr learned my addr 172.21.15.59:0/2119375997 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a75436640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700b4460 0x7f9a700d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475221331 0 0) 0x7f9a700d22c0 con 0x7f9a70162970 2024-03-06T07:34:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a58003520 con 0x7f9a70162970 2024-03-06T07:34:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157175918 0 0) 0x7f9a58003520 con 0x7f9a70162970 2024-03-06T07:34:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700b4460 msgr2=0x7f9a700d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700b4460 0x7f9a700d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 >> v1:172.21.15.164:6789/0 conn(0x7f9a70161cf0 legacy=0x7f9a700cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a700d2880 con 0x7f9a70162970 2024-03-06T07:34:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f9a64001670 con 0x7f9a70162970 2024-03-06T07:34:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a75436640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700b4460 0x7f9a700d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 >> v1:172.21.15.59:6789/0 conn(0x7f9a70162970 legacy=0x7f9a700b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700b4460 0x7f9a580039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a58003f20 0x7f9a58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a58004d40 0x7f9a580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a70161b00 con 0x7f9a58003f20 2024-03-06T07:34:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5800ba20 con 0x7f9a58004d40 2024-03-06T07:34:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5800bc20 con 0x7f9a700b4460 2024-03-06T07:34:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a74c35640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a58003f20 0x7f9a58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a75436640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700b4460 0x7f9a580039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a75c37640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a58004d40 0x7f9a580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a74c35640 1 -- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700b4460 msgr2=0x7f9a580039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a74c35640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700b4460 0x7f9a580039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a74c35640 1 -- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a58004d40 msgr2=0x7f9a580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a74c35640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a58004d40 0x7f9a580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.826+0000 7f9a74c35640 1 -- 172.21.15.59:0/2119375997 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a58003520 con 0x7f9a58003f20 2024-03-06T07:34:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.827+0000 7f9a74c35640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a58003f20 0x7f9a58004800 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f9a64007fd0 tx=0x7f9a58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f0fbdfb291033 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.827+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64007d50 con 0x7f9a58003f20 2024-03-06T07:34:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.827+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a64030070 con 0x7f9a58003f20 2024-03-06T07:34:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.827+0000 7f9a6ffff640 1 -- 172.21.15.59:0/2119375997 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64005d70 con 0x7f9a58003f20 2024-03-06T07:34:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.828+0000 7f9a7769a640 1 -- 172.21.15.59:0/2119375997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a58003f20 msgr2=0x7f9a58004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.828+0000 7f9a7769a640 1 --2- 172.21.15.59:0/2119375997 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a58003f20 0x7f9a58004800 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f9a64007fd0 tx=0x7f9a58005830 comp rx=0 tx=0).stop 2024-03-06T07:34:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.828+0000 7f9a75c37640 1 -- 172.21.15.59:0/2119375997 reap_dead start 2024-03-06T07:34:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.828+0000 7f9a7769a640 1 -- 172.21.15.59:0/2119375997 shutdown_connections 2024-03-06T07:34:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.828+0000 7f9a7769a640 1 -- 172.21.15.59:0/2119375997 >> 172.21.15.59:0/2119375997 conn(0x7f9a700d3ce0 msgr2=0x7f9a700d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.828+0000 7f9a7769a640 1 -- 172.21.15.59:0/2119375997 shutdown_connections 2024-03-06T07:34:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.828+0000 7f9a7769a640 1 -- 172.21.15.59:0/2119375997 wait complete. 2024-03-06T07:34:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.829+0000 7f9a7769a640 1 Processor -- start 2024-03-06T07:34:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.829+0000 7f9a7769a640 1 -- start start 2024-03-06T07:34:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a7769a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a58003f20 0x7f9a700b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a7769a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a70156120 0x7f9a701558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a7769a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a700ccb40 0x7f9a7015e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a7769a640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a700ce950 con 0x7f9a700ccb40 2024-03-06T07:34:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a7769a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a700cecc0 con 0x7f9a58003f20 2024-03-06T07:34:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a7769a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a700b4840 con 0x7f9a70156120 2024-03-06T07:34:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a75c37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a700ccb40 0x7f9a7015e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a75c37640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a700ccb40 0x7f9a7015e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56496/0 (socket says 172.21.15.59:56496) 2024-03-06T07:34:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a75c37640 1 -- 172.21.15.59:0/3193587937 learned_addr learned my addr 172.21.15.59:0/3193587937 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a74c35640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a70156120 0x7f9a701558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a75436640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a58003f20 0x7f9a700b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a75c37640 1 -- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a70156120 msgr2=0x7f9a701558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a75c37640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a70156120 0x7f9a701558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a75c37640 1 -- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a58003f20 msgr2=0x7f9a700b8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a75c37640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a58003f20 0x7f9a700b8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a75c37640 1 -- 172.21.15.59:0/3193587937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a700cb7c0 con 0x7f9a700ccb40 2024-03-06T07:34:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a75436640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a58003f20 0x7f9a700b8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.830+0000 7f9a75c37640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a700ccb40 0x7f9a7015e530 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f9a700c7610 tx=0x7f9a6800fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.831+0000 7f9a6dffb640 1 -- 172.21.15.59:0/3193587937 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a68019070 con 0x7f9a700ccb40 2024-03-06T07:34:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.831+0000 7f9a6dffb640 1 -- 172.21.15.59:0/3193587937 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a680050d0 con 0x7f9a700ccb40 2024-03-06T07:34:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.831+0000 7f9a7769a640 1 -- 172.21.15.59:0/3193587937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a700cbaa0 con 0x7f9a700ccb40 2024-03-06T07:34:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.831+0000 7f9a6dffb640 1 -- 172.21.15.59:0/3193587937 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a68007030 con 0x7f9a700ccb40 2024-03-06T07:34:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.831+0000 7f9a7769a640 1 -- 172.21.15.59:0/3193587937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a700cc060 con 0x7f9a700ccb40 2024-03-06T07:34:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.832+0000 7f9a7769a640 1 -- 172.21.15.59:0/3193587937 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7004f330 con 0x7f9a700ccb40 2024-03-06T07:34:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.832+0000 7f9a6dffb640 1 -- 172.21.15.59:0/3193587937 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a6801c070 con 0x7f9a700ccb40 2024-03-06T07:34:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.833+0000 7f9a6dffb640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5c06b4d0 0x7f9a5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.833+0000 7f9a6dffb640 1 -- 172.21.15.59:0/3193587937 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9a68088ee0 con 0x7f9a700ccb40 2024-03-06T07:34:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.833+0000 7f9a75436640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5c06b4d0 0x7f9a5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.833+0000 7f9a75436640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5c06b4d0 0x7f9a5c06d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f9a60007520 tx=0x7f9a6000b040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:36.837+0000 7f9a6dffb640 1 -- 172.21.15.59:0/3193587937 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a680579a0 con 0x7f9a700ccb40 2024-03-06T07:34:37.022 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.020+0000 7f9a7769a640 1 -- 172.21.15.59:0/3193587937 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a700b2a50 con 0x7f9a5c06b4d0 2024-03-06T07:34:37.023 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.021+0000 7f9a6dffb640 1 -- 172.21.15.59:0/3193587937 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9a700b2a50 con 0x7f9a5c06b4d0 2024-03-06T07:34:37.023 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 7 to deep-scrub 2024-03-06T07:34:37.025 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 -- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5c06b4d0 msgr2=0x7f9a5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5c06b4d0 0x7f9a5c06d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f9a60007520 tx=0x7f9a6000b040 comp rx=0 tx=0).stop 2024-03-06T07:34:37.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 -- 172.21.15.59:0/3193587937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a700ccb40 msgr2=0x7f9a7015e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a700ccb40 0x7f9a7015e530 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f9a700c7610 tx=0x7f9a6800fc00 comp rx=0 tx=0).stop 2024-03-06T07:34:37.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 -- 172.21.15.59:0/3193587937 shutdown_connections 2024-03-06T07:34:37.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f9a5c06b4d0 0x7f9a5c06d990 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a700ccb40 0x7f9a7015e530 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a70156120 0x7f9a701558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 --2- 172.21.15.59:0/3193587937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a58003f20 0x7f9a700b8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.026 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 -- 172.21.15.59:0/3193587937 >> 172.21.15.59:0/3193587937 conn(0x7f9a700d3ce0 msgr2=0x7f9a700cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:37.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 -- 172.21.15.59:0/3193587937 shutdown_connections 2024-03-06T07:34:37.027 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.023+0000 7f9a7769a640 1 -- 172.21.15.59:0/3193587937 wait complete. 2024-03-06T07:34:37.036 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:37.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.136+0000 7fe92da78640 1 Processor -- start 2024-03-06T07:34:37.138 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.136+0000 7fe92da78640 1 -- start start 2024-03-06T07:34:37.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe92da78640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe928166af0 0x7fe928168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe92da78640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92815e380 con 0x7fe9280d00c0 2024-03-06T07:34:37.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe92da78640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9280cdc10 con 0x7fe9280cf850 2024-03-06T07:34:37.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe92da78640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92806ca90 con 0x7fe928166af0 2024-03-06T07:34:37.141 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe927fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe928166af0 0x7fe928168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe9277fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fe9280d00c0 0x7fe9280d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:59926/0 (socket says 172.21.15.59:59926) 2024-03-06T07:34:37.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe927fff640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe928166af0 0x7fe928168ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56500/0 (socket says 172.21.15.59:56500) 2024-03-06T07:34:37.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe9277fe640 1 -- 172.21.15.59:0/2549527910 learned_addr learned my addr 172.21.15.59:0/2549527910 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:37.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe927fff640 1 -- 172.21.15.59:0/2549527910 >> v1:172.21.15.164:6789/0 conn(0x7fe9280cf850 legacy=0x7fe9280cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe927fff640 1 -- 172.21.15.59:0/2549527910 >> v1:172.21.15.59:6789/0 conn(0x7fe9280d00c0 legacy=0x7fe9280d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe9267fc640 1 -- 172.21.15.59:0/2549527910 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493816871 0 0) 0x7fe92815e380 con 0x7fe9280d00c0 2024-03-06T07:34:37.142 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.137+0000 7fe927fff640 1 -- 172.21.15.59:0/2549527910 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92806c160 con 0x7fe928166af0 2024-03-06T07:34:37.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.138+0000 7fe927fff640 1 --2- 172.21.15.59:0/2549527910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe928166af0 0x7fe928168ee0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7fe920009d90 tx=0x7fe92002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c61b7bfb548c974 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.138+0000 7fe9267fc640 1 -- 172.21.15.59:0/2549527910 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920039070 con 0x7fe928166af0 2024-03-06T07:34:37.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.138+0000 7fe9267fc640 1 -- 172.21.15.59:0/2549527910 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92002cd00 con 0x7fe928166af0 2024-03-06T07:34:37.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.138+0000 7fe9267fc640 1 -- 172.21.15.59:0/2549527910 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920034660 con 0x7fe928166af0 2024-03-06T07:34:37.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.139+0000 7fe92da78640 1 -- 172.21.15.59:0/2549527910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe928166af0 msgr2=0x7fe928168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.139+0000 7fe92da78640 1 --2- 172.21.15.59:0/2549527910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe928166af0 0x7fe928168ee0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7fe920009d90 tx=0x7fe92002bf50 comp rx=0 tx=0).stop 2024-03-06T07:34:37.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.139+0000 7fe92da78640 1 -- 172.21.15.59:0/2549527910 shutdown_connections 2024-03-06T07:34:37.143 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.139+0000 7fe92da78640 1 --2- 172.21.15.59:0/2549527910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe928166af0 0x7fe928168ee0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.139+0000 7fe92da78640 1 -- 172.21.15.59:0/2549527910 >> 172.21.15.59:0/2549527910 conn(0x7fe9280b35a0 msgr2=0x7fe9280b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:37.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.139+0000 7fe92da78640 1 -- 172.21.15.59:0/2549527910 shutdown_connections 2024-03-06T07:34:37.144 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.139+0000 7fe92da78640 1 -- 172.21.15.59:0/2549527910 wait complete. 2024-03-06T07:34:37.146 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.140+0000 7fe92da78640 1 Processor -- start 2024-03-06T07:34:37.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.140+0000 7fe92da78640 1 -- start start 2024-03-06T07:34:37.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe92da78640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9280cf850 0x7fe9280c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe92da78640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9280d00c0 0x7fe9281548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe92da78640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe928166af0 0x7fe928159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe92da78640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92806c720 con 0x7fe928166af0 2024-03-06T07:34:37.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe92da78640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92806df30 con 0x7fe9280d00c0 2024-03-06T07:34:37.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe92da78640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9280b1030 con 0x7fe9280cf850 2024-03-06T07:34:37.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe927fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe928166af0 0x7fe928159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.147 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe927fff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe928166af0 0x7fe928159f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56512/0 (socket says 172.21.15.59:56512) 2024-03-06T07:34:37.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe927fff640 1 -- 172.21.15.59:0/115499626 learned_addr learned my addr 172.21.15.59:0/115499626 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:37.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe927fff640 1 -- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9280cf850 msgr2=0x7fe9280c6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe9277fe640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9280cf850 0x7fe9280c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe927fff640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9280cf850 0x7fe9280c6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe927fff640 1 -- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9280d00c0 msgr2=0x7fe9281548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe927fff640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9280d00c0 0x7fe9281548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe927fff640 1 -- 172.21.15.59:0/115499626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe928154e00 con 0x7fe928166af0 2024-03-06T07:34:37.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe927fff640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe928166af0 0x7fe928159f10 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fe920009d90 tx=0x7fe920004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.148 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe90ffff640 1 -- 172.21.15.59:0/115499626 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920039050 con 0x7fe928166af0 2024-03-06T07:34:37.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe90ffff640 1 -- 172.21.15.59:0/115499626 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe920045070 con 0x7fe928166af0 2024-03-06T07:34:37.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.141+0000 7fe92da78640 1 -- 172.21.15.59:0/115499626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9280c57d0 con 0x7fe928166af0 2024-03-06T07:34:37.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.142+0000 7fe90ffff640 1 -- 172.21.15.59:0/115499626 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92003e830 con 0x7fe928166af0 2024-03-06T07:34:37.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.142+0000 7fe92da78640 1 -- 172.21.15.59:0/115499626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9280c5d90 con 0x7fe928166af0 2024-03-06T07:34:37.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.142+0000 7fe92da78640 1 -- 172.21.15.59:0/115499626 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92804f330 con 0x7fe928166af0 2024-03-06T07:34:37.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.143+0000 7fe90ffff640 1 -- 172.21.15.59:0/115499626 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe92003e9d0 con 0x7fe928166af0 2024-03-06T07:34:37.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.143+0000 7fe90ffff640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91406b4d0 0x7fe91406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:37.149 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.143+0000 7fe90ffff640 1 -- 172.21.15.59:0/115499626 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe9200acd80 con 0x7fe928166af0 2024-03-06T07:34:37.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.144+0000 7fe9277fe640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91406b4d0 0x7fe91406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:37.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.144+0000 7fe9277fe640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91406b4d0 0x7fe91406d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fe918007fd0 tx=0x7fe918009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:37.150 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.148+0000 7fe90ffff640 1 -- 172.21.15.59:0/115499626 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe92007bea0 con 0x7fe928166af0 2024-03-06T07:34:37.328 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.326+0000 7fe92da78640 1 -- 172.21.15.59:0/115499626 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe9280b3bb0 con 0x7fe91406b4d0 2024-03-06T07:34:37.329 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.327+0000 7fe90ffff640 1 -- 172.21.15.59:0/115499626 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13503 (secure 0 0 0) 0x7fe9280b3bb0 con 0x7fe91406b4d0 2024-03-06T07:34:37.329 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:37.330 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:34:37.331 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 -- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91406b4d0 msgr2=0x7fe91406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91406b4d0 0x7fe91406d990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fe918007fd0 tx=0x7fe918009040 comp rx=0 tx=0).stop 2024-03-06T07:34:37.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 -- 172.21.15.59:0/115499626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe928166af0 msgr2=0x7fe928159f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:37.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe928166af0 0x7fe928159f10 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fe920009d90 tx=0x7fe920004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:37.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 -- 172.21.15.59:0/115499626 shutdown_connections 2024-03-06T07:34:37.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fe91406b4d0 0x7fe91406d990 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe928166af0 0x7fe928159f10 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.332 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9280d00c0 0x7fe9281548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 --2- 172.21.15.59:0/115499626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9280cf850 0x7fe9280c6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:37.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 -- 172.21.15.59:0/115499626 >> 172.21.15.59:0/115499626 conn(0x7fe9280b35a0 msgr2=0x7fe9280d1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:37.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 -- 172.21.15.59:0/115499626 shutdown_connections 2024-03-06T07:34:37.333 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:37.329+0000 7fe92da78640 1 -- 172.21.15.59:0/115499626 wait complete. 2024-03-06T07:34:37.342 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-03-06T07:34:35.477624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":331508,"kb_used_data":15496,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643212,"statfs":{"total":773094113280,"available":772754649088,"internally_reserved":0,"allocated":15867904,"data_stored":7620120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36544,"num_objects":-43,"num_object_clones":0,"num_object_copies":-86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-144,"num_write_kb":-86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001513"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":401,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:34:29.332658+0000","last_change":"2024-03-06T07:21:43.604646+0000","last_active":"2024-03-06T07:34:29.332658+0000","last_peered":"2024-03-06T07:34:29.332658+0000","last_clean":"2024-03-06T07:34:29.332658+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:34:29.332658+0000","last_undegraded":"2024-03-06T07:34:29.332658+0000","last_fullsized":"2024-03-06T07:34:29.332658+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:42.202215+0000","last_clean_scrub_stamp":"2024-03-06T07:21:42.202215+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:23.763225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575005,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41872,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329968,"statfs":{"total":96636764160,"available":96593887232,"internally_reserved":0,"allocated":2428928,"data_stored":1395291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575005,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575005,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41880,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329960,"statfs":{"total":96636764160,"available":96593879040,"internally_reserved":0,"allocated":2428928,"data_stored":1395291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:37.344 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-06T07:21:42.202215+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=21, tm_sec=42, tm_wday=2, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=34, tm_sec=32, tm_wday=2, tm_yday=66, tm_isdst=0) 2024-03-06T07:34:37.345 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-06T07:34:51.685 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:51.688 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:57.346 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:34:57.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.459+0000 7fa661a8d640 1 Processor -- start 2024-03-06T07:34:57.461 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.459+0000 7fa661a8d640 1 -- start start 2024-03-06T07:34:57.462 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.460+0000 7fa661a8d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa65c0cf850 0x7fa65c0cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.460+0000 7fa661a8d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65c15e380 con 0x7fa65c0d00c0 2024-03-06T07:34:57.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.460+0000 7fa661a8d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65c0cdc10 con 0x7fa65c06cf20 2024-03-06T07:34:57.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.460+0000 7fa661a8d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c06ca90 con 0x7fa65c0cf850 2024-03-06T07:34:57.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.460+0000 7fa65affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa65c0cf850 0x7fa65c0cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.460+0000 7fa65affd640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa65c0cf850 0x7fa65c0cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38986/0 (socket says 172.21.15.59:38986) 2024-03-06T07:34:57.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.460+0000 7fa65affd640 1 -- 172.21.15.59:0/478859330 learned_addr learned my addr 172.21.15.59:0/478859330 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:57.463 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.460+0000 7fa65b7fe640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa65c0d00c0 0x7fa65c0d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40292/0 (socket says 172.21.15.59:40292) 2024-03-06T07:34:57.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.461+0000 7fa65affd640 1 -- 172.21.15.59:0/478859330 >> v1:172.21.15.164:6789/0 conn(0x7fa65c06cf20 legacy=0x7fa65c06d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:57.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.461+0000 7fa65affd640 1 -- 172.21.15.59:0/478859330 >> v1:172.21.15.59:6789/0 conn(0x7fa65c0d00c0 legacy=0x7fa65c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.464 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.461+0000 7fa65affd640 1 -- 172.21.15.59:0/478859330 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c06c160 con 0x7fa65c0cf850 2024-03-06T07:34:57.465 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.461+0000 7fa65affd640 1 --2- 172.21.15.59:0/478859330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa65c0cf850 0x7fa65c0cd3c0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fa650009870 tx=0x7fa65002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a688e9cf333b3ad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.461+0000 7fa65a7fc640 1 -- 172.21.15.59:0/478859330 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650034070 con 0x7fa65c0cf850 2024-03-06T07:34:57.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.461+0000 7fa65a7fc640 1 -- 172.21.15.59:0/478859330 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa650030860 con 0x7fa65c0cf850 2024-03-06T07:34:57.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.461+0000 7fa65a7fc640 1 -- 172.21.15.59:0/478859330 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650030c30 con 0x7fa65c0cf850 2024-03-06T07:34:57.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.462+0000 7fa661a8d640 1 -- 172.21.15.59:0/478859330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa65c0cf850 msgr2=0x7fa65c0cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.462+0000 7fa661a8d640 1 --2- 172.21.15.59:0/478859330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa65c0cf850 0x7fa65c0cd3c0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fa650009870 tx=0x7fa65002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.462+0000 7fa661a8d640 1 -- 172.21.15.59:0/478859330 shutdown_connections 2024-03-06T07:34:57.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.462+0000 7fa661a8d640 1 --2- 172.21.15.59:0/478859330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa65c0cf850 0x7fa65c0cd3c0 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.466 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.462+0000 7fa661a8d640 1 -- 172.21.15.59:0/478859330 >> 172.21.15.59:0/478859330 conn(0x7fa65c0b35a0 msgr2=0x7fa65c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:57.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.462+0000 7fa661a8d640 1 -- 172.21.15.59:0/478859330 shutdown_connections 2024-03-06T07:34:57.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.462+0000 7fa661a8d640 1 -- 172.21.15.59:0/478859330 wait complete. 2024-03-06T07:34:57.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.463+0000 7fa661a8d640 1 Processor -- start 2024-03-06T07:34:57.467 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.463+0000 7fa661a8d640 1 -- start start 2024-03-06T07:34:57.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.464+0000 7fa661a8d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa65c06cf20 0x7fa65c157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.469 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.464+0000 7fa661a8d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa65c0cf850 0x7fa65c0af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.464+0000 7fa661a8d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa65c0d00c0 0x7fa65c15c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.464+0000 7fa661a8d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0cd900 con 0x7fa65c0d00c0 2024-03-06T07:34:57.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.464+0000 7fa661a8d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c06c720 con 0x7fa65c0cf850 2024-03-06T07:34:57.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.464+0000 7fa661a8d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0b1030 con 0x7fa65c06cf20 2024-03-06T07:34:57.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.464+0000 7fa65bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa65c0d00c0 0x7fa65c15c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.464+0000 7fa65bfff640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa65c0d00c0 0x7fa65c15c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38996/0 (socket says 172.21.15.59:38996) 2024-03-06T07:34:57.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.464+0000 7fa65bfff640 1 -- 172.21.15.59:0/3238144543 learned_addr learned my addr 172.21.15.59:0/3238144543 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:57.470 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa65affd640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa65c0cf850 0x7fa65c0af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa65b7fe640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa65c06cf20 0x7fa65c157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa65bfff640 1 -- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa65c06cf20 msgr2=0x7fa65c157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa65bfff640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa65c06cf20 0x7fa65c157bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa65bfff640 1 -- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa65c0cf850 msgr2=0x7fa65c0af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa65bfff640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa65c0cf850 0x7fa65c0af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa65bfff640 1 -- 172.21.15.59:0/3238144543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c15e550 con 0x7fa65c0d00c0 2024-03-06T07:34:57.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa65affd640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa65c0cf850 0x7fa65c0af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:57.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa65b7fe640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa65c06cf20 0x7fa65c157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:57.471 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa65bfff640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa65c0d00c0 0x7fa65c15c3c0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fa65400c7b0 tx=0x7fa65400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa643fff640 1 -- 172.21.15.59:0/3238144543 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa654017070 con 0x7fa65c0d00c0 2024-03-06T07:34:57.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa643fff640 1 -- 172.21.15.59:0/3238144543 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa654003da0 con 0x7fa65c0d00c0 2024-03-06T07:34:57.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa661a8d640 1 -- 172.21.15.59:0/3238144543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65c15e830 con 0x7fa65c0d00c0 2024-03-06T07:34:57.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa643fff640 1 -- 172.21.15.59:0/3238144543 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65400ce60 con 0x7fa65c0d00c0 2024-03-06T07:34:57.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.465+0000 7fa661a8d640 1 -- 172.21.15.59:0/3238144543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65c15edf0 con 0x7fa65c0d00c0 2024-03-06T07:34:57.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.466+0000 7fa661a8d640 1 -- 172.21.15.59:0/3238144543 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65c04f330 con 0x7fa65c0d00c0 2024-03-06T07:34:57.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.466+0000 7fa643fff640 1 -- 172.21.15.59:0/3238144543 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa65401a070 con 0x7fa65c0d00c0 2024-03-06T07:34:57.472 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.467+0000 7fa643fff640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa64806b4d0 0x7fa64806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.467+0000 7fa643fff640 1 -- 172.21.15.59:0/3238144543 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fa6540895b0 con 0x7fa65c0d00c0 2024-03-06T07:34:57.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.467+0000 7fa65b7fe640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa64806b4d0 0x7fa64806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.468+0000 7fa65b7fe640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa64806b4d0 0x7fa64806d990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000f60 tx=0x7fa64c009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.473 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.471+0000 7fa643fff640 1 -- 172.21.15.59:0/3238144543 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6540586d0 con 0x7fa65c0d00c0 2024-03-06T07:34:57.653 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.651+0000 7fa661a8d640 1 -- 172.21.15.59:0/3238144543 --> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa65c0ceab0 con 0x7fa64806b4d0 2024-03-06T07:34:57.654 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.652+0000 7fa643fff640 1 -- 172.21.15.59:0/3238144543 <== mgr.4115 v2:172.21.15.164:6832/3140157334 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13379 (secure 0 0 0) 0x7fa65c0ceab0 con 0x7fa64806b4d0 2024-03-06T07:34:57.654 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:57.654 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2024-03-06T07:34:57.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 -- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa64806b4d0 msgr2=0x7fa64806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa64806b4d0 0x7fa64806d990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fa64c000f60 tx=0x7fa64c009040 comp rx=0 tx=0).stop 2024-03-06T07:34:57.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 -- 172.21.15.59:0/3238144543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa65c0d00c0 msgr2=0x7fa65c15c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa65c0d00c0 0x7fa65c15c3c0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fa65400c7b0 tx=0x7fa65400cc70 comp rx=0 tx=0).stop 2024-03-06T07:34:57.656 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 -- 172.21.15.59:0/3238144543 shutdown_connections 2024-03-06T07:34:57.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7fa64806b4d0 0x7fa64806d990 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa65c0d00c0 0x7fa65c15c3c0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa65c0cf850 0x7fa65c0af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 --2- 172.21.15.59:0/3238144543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa65c06cf20 0x7fa65c157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 -- 172.21.15.59:0/3238144543 >> 172.21.15.59:0/3238144543 conn(0x7fa65c0b35a0 msgr2=0x7fa65c070220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:57.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 -- 172.21.15.59:0/3238144543 shutdown_connections 2024-03-06T07:34:57.657 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.654+0000 7fa661a8d640 1 -- 172.21.15.59:0/3238144543 wait complete. 2024-03-06T07:34:57.667 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":537,"stamp":"2024-03-06T07:34:57.482121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":331492,"kb_used_data":15496,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754643228,"statfs":{"total":773094113280,"available":772754665472,"internally_reserved":0,"allocated":15867904,"data_stored":7620120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002504"},"pg_stats":[{"pgid":"1.0","version":"146'119","reported_seq":412,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-06T07:34:37.070330+0000","last_change":"2024-03-06T07:34:37.070385+0000","last_active":"2024-03-06T07:34:37.070330+0000","last_peered":"2024-03-06T07:34:37.070330+0000","last_clean":"2024-03-06T07:34:37.070330+0000","last_became_active":"2024-03-06T07:21:43.603680+0000","last_became_peered":"2024-03-06T07:21:43.603680+0000","last_unstale":"2024-03-06T07:34:37.070330+0000","last_undegraded":"2024-03-06T07:34:37.070330+0000","last_fullsized":"2024-03-06T07:34:37.070330+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'119","last_scrub_stamp":"2024-03-06T07:34:37.070245+0000","last_deep_scrub":"146'119","last_deep_scrub_stamp":"2024-03-06T07:34:37.070245+0000","last_clean_scrub_stamp":"2024-03-06T07:34:37.070245+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:14.235401+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41868,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329972,"statfs":{"total":96636764160,"available":96593891328,"internally_reserved":0,"allocated":2428928,"data_stored":1395291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41296,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330544,"statfs":{"total":96636764160,"available":96594477056,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41292,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":1835008,"data_stored":804923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41868,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329972,"statfs":{"total":96636764160,"available":96593891328,"internally_reserved":0,"allocated":2428928,"data_stored":1395291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:34:57.667 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-06T07:34:57.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.767+0000 7f09da06f640 1 Processor -- start 2024-03-06T07:34:57.769 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.767+0000 7f09da06f640 1 -- start start 2024-03-06T07:34:57.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09da06f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40b41b0 0x7f09d40cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09da06f640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d40d0240 con 0x7f09d40b15c0 2024-03-06T07:34:57.773 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09da06f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d40cef50 con 0x7f09d415bdd0 2024-03-06T07:34:57.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09da06f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40cf4f0 con 0x7f09d40b41b0 2024-03-06T07:34:57.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d3fff640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f09d40b15c0 0x7f09d40b1a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40306/0 (socket says 172.21.15.59:40306) 2024-03-06T07:34:57.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d3fff640 1 -- 172.21.15.59:0/3891977560 learned_addr learned my addr 172.21.15.59:0/3891977560 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:57.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d2ffd640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40b41b0 0x7f09d40cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959918766 0 0) 0x7f09d40d0240 con 0x7f09d40b15c0 2024-03-06T07:34:57.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09bc003520 con 0x7f09d40b15c0 2024-03-06T07:34:57.774 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 928660225 0 0) 0x7f09bc003520 con 0x7f09d40b15c0 2024-03-06T07:34:57.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40b41b0 msgr2=0x7f09d40cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:57.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d27fc640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40b41b0 0x7f09d40cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 >> v1:172.21.15.164:6789/0 conn(0x7f09d415bdd0 legacy=0x7f09d40ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40cf210 con 0x7f09d40b15c0 2024-03-06T07:34:57.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 440152642 0 0) 0x7f09cc0019f0 con 0x7f09d40b15c0 2024-03-06T07:34:57.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 >> v1:172.21.15.59:6789/0 conn(0x7f09d40b15c0 legacy=0x7f09d40b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.775 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d27fc640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09bc0039e0 0x7f09bc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.768+0000 7f09d27fc640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09bc004800 0x7f09bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d27fc640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40b41b0 0x7f09bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40cf4f0 con 0x7f09bc004800 2024-03-06T07:34:57.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc00ba70 con 0x7f09bc0039e0 2024-03-06T07:34:57.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc00bc70 con 0x7f09d40b41b0 2024-03-06T07:34:57.776 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d2ffd640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09bc004800 0x7f09bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d37fe640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09bc0039e0 0x7f09bc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d3fff640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40b41b0 0x7f09bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d2ffd640 1 -- 172.21.15.59:0/3891977560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40b41b0 msgr2=0x7f09bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:57.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d2ffd640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40b41b0 0x7f09bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d2ffd640 1 -- 172.21.15.59:0/3891977560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09bc0039e0 msgr2=0x7f09bc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:57.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d2ffd640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09bc0039e0 0x7f09bc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.777 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.769+0000 7f09d2ffd640 1 -- 172.21.15.59:0/3891977560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc003520 con 0x7f09bc004800 2024-03-06T07:34:57.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.770+0000 7f09d2ffd640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09bc004800 0x7f09bc004c80 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009a90 tx=0x7f09bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af8c4e178aaa7c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.770+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8039070 con 0x7f09bc004800 2024-03-06T07:34:57.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.770+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c802c680 con 0x7f09bc004800 2024-03-06T07:34:57.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.770+0000 7f09d27fc640 1 -- 172.21.15.59:0/3891977560 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8009690 con 0x7f09bc004800 2024-03-06T07:34:57.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.770+0000 7f09da06f640 1 -- 172.21.15.59:0/3891977560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09bc004800 msgr2=0x7f09bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.770+0000 7f09da06f640 1 --2- 172.21.15.59:0/3891977560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09bc004800 0x7f09bc004c80 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009a90 tx=0x7f09bc005830 comp rx=0 tx=0).stop 2024-03-06T07:34:57.778 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.770+0000 7f09d3fff640 1 -- 172.21.15.59:0/3891977560 reap_dead start 2024-03-06T07:34:57.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.770+0000 7f09da06f640 1 -- 172.21.15.59:0/3891977560 shutdown_connections 2024-03-06T07:34:57.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.770+0000 7f09da06f640 1 -- 172.21.15.59:0/3891977560 >> 172.21.15.59:0/3891977560 conn(0x7f09d40b3690 msgr2=0x7f09d40b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:57.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.770+0000 7f09da06f640 1 -- 172.21.15.59:0/3891977560 shutdown_connections 2024-03-06T07:34:57.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.770+0000 7f09da06f640 1 -- 172.21.15.59:0/3891977560 wait complete. 2024-03-06T07:34:57.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.771+0000 7f09da06f640 1 Processor -- start 2024-03-06T07:34:57.779 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.772+0000 7f09da06f640 1 -- start start 2024-03-06T07:34:57.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.772+0000 7f09da06f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09bc004800 0x7f09d40c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.772+0000 7f09da06f640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415c330 0x7f09d4151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.772+0000 7f09da06f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09d415eb00 0x7f09d40ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.772+0000 7f09da06f640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415c1b0 con 0x7f09d415c330 2024-03-06T07:34:57.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.772+0000 7f09da06f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40d0850 con 0x7f09d415eb00 2024-03-06T07:34:57.782 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.772+0000 7f09da06f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40b10c0 con 0x7f09bc004800 2024-03-06T07:34:57.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.772+0000 7f09d2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415c330 0x7f09d4151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.772+0000 7f09d2ffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415c330 0x7f09d4151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39014/0 (socket says 172.21.15.59:39014) 2024-03-06T07:34:57.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.772+0000 7f09d2ffd640 1 -- 172.21.15.59:0/2431757650 learned_addr learned my addr 172.21.15.59:0/2431757650 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:57.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09d37fe640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09bc004800 0x7f09d40c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09d3fff640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09d415eb00 0x7f09d40ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09d2ffd640 1 -- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09bc004800 msgr2=0x7f09d40c6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09d2ffd640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09bc004800 0x7f09d40c6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09d2ffd640 1 -- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09d415eb00 msgr2=0x7f09d40ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.783 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09d2ffd640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09d415eb00 0x7f09d40ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09d2ffd640 1 -- 172.21.15.59:0/2431757650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d4151590 con 0x7f09d415c330 2024-03-06T07:34:57.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09d2ffd640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415c330 0x7f09d4151050 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f09c8000c00 tx=0x7f09c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09bbfff640 1 -- 172.21.15.59:0/2431757650 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8042020 con 0x7f09d415c330 2024-03-06T07:34:57.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09bbfff640 1 -- 172.21.15.59:0/2431757650 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c803a040 con 0x7f09d415c330 2024-03-06T07:34:57.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09da06f640 1 -- 172.21.15.59:0/2431757650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d40c7600 con 0x7f09d415c330 2024-03-06T07:34:57.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09bbfff640 1 -- 172.21.15.59:0/2431757650 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8039070 con 0x7f09d415c330 2024-03-06T07:34:57.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.773+0000 7f09da06f640 1 -- 172.21.15.59:0/2431757650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d40c90c0 con 0x7f09d415c330 2024-03-06T07:34:57.784 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.774+0000 7f09da06f640 1 -- 172.21.15.59:0/2431757650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d409fcf0 con 0x7f09d415c330 2024-03-06T07:34:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.775+0000 7f09bbfff640 1 -- 172.21.15.59:0/2431757650 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09c8045020 con 0x7f09d415c330 2024-03-06T07:34:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.775+0000 7f09bbfff640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09c006b4d0 0x7f09c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.775+0000 7f09bbfff640 1 -- 172.21.15.59:0/2431757650 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f09c80ab340 con 0x7f09d415c330 2024-03-06T07:34:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.775+0000 7f09d37fe640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09c006b4d0 0x7f09c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.776+0000 7f09d37fe640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09c006b4d0 0x7f09c006d990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f09c4005590 tx=0x7f09c4009040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:57.785 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.779+0000 7f09bbfff640 1 -- 172.21.15.59:0/2431757650 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09c804c070 con 0x7f09d415c330 2024-03-06T07:34:57.961 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.959+0000 7f09da06f640 1 -- 172.21.15.59:0/2431757650 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f09d4068ff0 con 0x7f09d415c330 2024-03-06T07:34:57.966 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.965+0000 7f09bbfff640 1 -- 172.21.15.59:0/2431757650 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v36)=0 v36) v1 ==== 123+0+0 (secure 0 0 0) 0x7f09d4151590 con 0x7f09d415c330 2024-03-06T07:34:57.967 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.965+0000 7f09bbfff640 1 -- 172.21.15.59:0/2431757650 <== mon.0 v2:172.21.15.59:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f09c807a4f0 con 0x7f09d415c330 2024-03-06T07:34:57.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.967+0000 7f09da06f640 1 -- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09c006b4d0 msgr2=0x7f09c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.967+0000 7f09da06f640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09c006b4d0 0x7f09c006d990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f09c4005590 tx=0x7f09c4009040 comp rx=0 tx=0).stop 2024-03-06T07:34:57.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.967+0000 7f09da06f640 1 -- 172.21.15.59:0/2431757650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415c330 msgr2=0x7f09d4151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:57.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.967+0000 7f09da06f640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415c330 0x7f09d4151050 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f09c8000c00 tx=0x7f09c8004730 comp rx=0 tx=0).stop 2024-03-06T07:34:57.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.968+0000 7f09da06f640 1 -- 172.21.15.59:0/2431757650 shutdown_connections 2024-03-06T07:34:57.970 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.968+0000 7f09da06f640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f09c006b4d0 0x7f09c006d990 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.968+0000 7f09da06f640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09d415eb00 0x7f09d40ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.968+0000 7f09da06f640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f09d415c330 0x7f09d4151050 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.968+0000 7f09da06f640 1 --2- 172.21.15.59:0/2431757650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09bc004800 0x7f09d40c6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.968+0000 7f09da06f640 1 -- 172.21.15.59:0/2431757650 >> 172.21.15.59:0/2431757650 conn(0x7f09d40b3690 msgr2=0x7f09d40cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.968+0000 7f09da06f640 1 -- 172.21.15.59:0/2431757650 shutdown_connections 2024-03-06T07:34:57.971 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:57.968+0000 7f09da06f640 1 -- 172.21.15.59:0/2431757650 wait complete. 2024-03-06T07:34:57.980 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:34:57.981 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:58.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.080+0000 7f16d150d640 1 Processor -- start 2024-03-06T07:34:58.082 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.080+0000 7f16d150d640 1 -- start start 2024-03-06T07:34:58.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16d150d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16cc162a00 0x7f16cc0d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16d150d640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f16cc162400 0x7f16cc0cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16d150d640 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16cc0b1070 con 0x7f16cc162020 2024-03-06T07:34:58.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16d150d640 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16cc0ce9d0 con 0x7f16cc162400 2024-03-06T07:34:58.085 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16d150d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc0d2da0 con 0x7f16cc162a00 2024-03-06T07:34:58.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16cb7fe640 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f16cc162400 0x7f16cc0cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16caffd640 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f16cc162020 0x7f16cc0b0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:40308/0 (socket says 172.21.15.59:40308) 2024-03-06T07:34:58.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16caffd640 1 -- 172.21.15.59:0/575390600 learned_addr learned my addr 172.21.15.59:0/575390600 (peer_addr_for_me v1:172.21.15.59:0/0) 2024-03-06T07:34:58.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16cb7fe640 1 -- 172.21.15.59:0/575390600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16cc162a00 msgr2=0x7f16cc0d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:58.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16ca7fc640 1 --2- 172.21.15.59:0/575390600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16cc162a00 0x7f16cc0d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16c9ffb640 1 -- 172.21.15.59:0/575390600 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912842209 0 0) 0x7f16cc0b1070 con 0x7f16cc162020 2024-03-06T07:34:58.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16cb7fe640 1 --2- 172.21.15.59:0/575390600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16cc162a00 0x7f16cc0d4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.086 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16cb7fe640 1 -- 172.21.15.59:0/575390600 >> v1:172.21.15.59:6789/0 conn(0x7f16cc162020 legacy=0x7f16cc0b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.081+0000 7f16cb7fe640 1 -- 172.21.15.59:0/575390600 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16cc0d2500 con 0x7f16cc162400 2024-03-06T07:34:58.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.082+0000 7f16cb7fe640 1 --2- 172.21.15.59:0/575390600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16cc162400 0x7f16cc0cf400 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009d90 tx=0x7f16c402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a5778d2602ffb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.082+0000 7f16c9ffb640 1 -- 172.21.15.59:0/575390600 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c4039070 con 0x7f16cc162400 2024-03-06T07:34:58.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.082+0000 7f16c9ffb640 1 -- 172.21.15.59:0/575390600 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f16c402cce0 con 0x7f16cc162400 2024-03-06T07:34:58.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.082+0000 7f16c9ffb640 1 -- 172.21.15.59:0/575390600 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c4034680 con 0x7f16cc162400 2024-03-06T07:34:58.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.083+0000 7f16d150d640 1 -- 172.21.15.59:0/575390600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16cc162400 msgr2=0x7f16cc0cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.083+0000 7f16d150d640 1 --2- 172.21.15.59:0/575390600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16cc162400 0x7f16cc0cf400 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f16c4009d90 tx=0x7f16c402bf30 comp rx=0 tx=0).stop 2024-03-06T07:34:58.087 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.083+0000 7f16d150d640 1 -- 172.21.15.59:0/575390600 shutdown_connections 2024-03-06T07:34:58.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.083+0000 7f16d150d640 1 --2- 172.21.15.59:0/575390600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16cc162400 0x7f16cc0cf400 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.083+0000 7f16d150d640 1 --2- 172.21.15.59:0/575390600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16cc162a00 0x7f16cc0d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.083+0000 7f16d150d640 1 -- 172.21.15.59:0/575390600 >> 172.21.15.59:0/575390600 conn(0x7f16cc0d3d10 msgr2=0x7f16cc0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:58.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.083+0000 7f16d150d640 1 -- 172.21.15.59:0/575390600 shutdown_connections 2024-03-06T07:34:58.088 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.083+0000 7f16d150d640 1 -- 172.21.15.59:0/575390600 wait complete. 2024-03-06T07:34:58.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.084+0000 7f16d150d640 1 Processor -- start 2024-03-06T07:34:58.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.084+0000 7f16d150d640 1 -- start start 2024-03-06T07:34:58.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16d150d640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16cc162020 0x7f16cc15d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16d150d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16cc162400 0x7f16cc0b9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16d150d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16cc162a00 0x7f16cc0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16d150d640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc0cec00 con 0x7f16cc162020 2024-03-06T07:34:58.091 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16d150d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc0d2ac0 con 0x7f16cc162a00 2024-03-06T07:34:58.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16d150d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16cc0cf940 con 0x7f16cc162400 2024-03-06T07:34:58.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16caffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16cc162020 0x7f16cc15d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16caffd640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16cc162020 0x7f16cc15d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39018/0 (socket says 172.21.15.59:39018) 2024-03-06T07:34:58.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16caffd640 1 -- 172.21.15.59:0/2506889859 learned_addr learned my addr 172.21.15.59:0/2506889859 (peer_addr_for_me v2:172.21.15.59:0/0) 2024-03-06T07:34:58.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16ca7fc640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16cc162400 0x7f16cc0b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16cb7fe640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16cc162a00 0x7f16cc0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.092 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16caffd640 1 -- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16cc162400 msgr2=0x7f16cc0b9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16caffd640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16cc162400 0x7f16cc0b9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16caffd640 1 -- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16cc162a00 msgr2=0x7f16cc0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16caffd640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16cc162a00 0x7f16cc0b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16caffd640 1 -- 172.21.15.59:0/2506889859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16cc0b3850 con 0x7f16cc162020 2024-03-06T07:34:58.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16ca7fc640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16cc162400 0x7f16cc0b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:58.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16cb7fe640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16cc162a00 0x7f16cc0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:58.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.085+0000 7f16caffd640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16cc162020 0x7f16cc15d9f0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00c500 tx=0x7f16bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.086+0000 7f16b37fe640 1 -- 172.21.15.59:0/2506889859 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc018070 con 0x7f16cc162020 2024-03-06T07:34:58.093 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.086+0000 7f16b37fe640 1 -- 172.21.15.59:0/2506889859 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f16bc007580 con 0x7f16cc162020 2024-03-06T07:34:58.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.086+0000 7f16d150d640 1 -- 172.21.15.59:0/2506889859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16cc15e080 con 0x7f16cc162020 2024-03-06T07:34:58.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.086+0000 7f16b37fe640 1 -- 172.21.15.59:0/2506889859 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc004a40 con 0x7f16cc162020 2024-03-06T07:34:58.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.086+0000 7f16d150d640 1 -- 172.21.15.59:0/2506889859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16cc15e640 con 0x7f16cc162020 2024-03-06T07:34:58.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.087+0000 7f16d150d640 1 -- 172.21.15.59:0/2506889859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16cc04f330 con 0x7f16cc162020 2024-03-06T07:34:58.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.087+0000 7f16b37fe640 1 -- 172.21.15.59:0/2506889859 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f16bc01b070 con 0x7f16cc162020 2024-03-06T07:34:58.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.087+0000 7f16b37fe640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f16b806b4d0 0x7f16b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:58.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.088+0000 7f16b37fe640 1 -- 172.21.15.59:0/2506889859 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f16bc0892f0 con 0x7f16cc162020 2024-03-06T07:34:58.094 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.088+0000 7f16ca7fc640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f16b806b4d0 0x7f16b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:58.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.088+0000 7f16ca7fc640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f16b806b4d0 0x7f16b806d990 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0b1ff0 tx=0x7f16c0008040 comp rx=0 tx=0).ready entity=mgr.4115 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:58.095 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.092+0000 7f16b37fe640 1 -- 172.21.15.59:0/2506889859 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f16bc058380 con 0x7f16cc162020 2024-03-06T07:34:58.314 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.312+0000 7f16d150d640 1 -- 172.21.15.59:0/2506889859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16cc0ce0d0 con 0x7f16cc162020 2024-03-06T07:34:58.315 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.313+0000 7f16b37fe640 1 -- 172.21.15.59:0/2506889859 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+1329 (secure 0 0 0) 0x7f16bc057c80 con 0x7f16cc162020 2024-03-06T07:34:58.315 INFO:teuthology.orchestra.run.smithi059.stdout: 2024-03-06T07:34:58.315 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":110,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10611,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6833/3623360840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6832","nonce":3623360840},{"type":"v1","addr":"172.21.15.59:6833","nonce":3623360840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":110}],"filesystems":[]} 2024-03-06T07:34:58.316 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 110 2024-03-06T07:34:58.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 -- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f16b806b4d0 msgr2=0x7f16b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f16b806b4d0 0x7f16b806d990 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0b1ff0 tx=0x7f16c0008040 comp rx=0 tx=0).stop 2024-03-06T07:34:58.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 -- 172.21.15.59:0/2506889859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16cc162020 msgr2=0x7f16cc15d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:58.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16cc162020 0x7f16cc15d9f0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00c500 tx=0x7f16bc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 -- 172.21.15.59:0/2506889859 shutdown_connections 2024-03-06T07:34:58.317 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:6832/3140157334,v1:172.21.15.164:6833/3140157334] conn(0x7f16b806b4d0 0x7f16b806d990 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16cc162a00 0x7f16cc0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16cc162400 0x7f16cc0b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 --2- 172.21.15.59:0/2506889859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16cc162020 0x7f16cc15d9f0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:58.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 -- 172.21.15.59:0/2506889859 >> 172.21.15.59:0/2506889859 conn(0x7f16cc0d3d10 msgr2=0x7f16cc0c9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:58.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 -- 172.21.15.59:0/2506889859 shutdown_connections 2024-03-06T07:34:58.318 INFO:teuthology.orchestra.run.smithi059.stderr:2024-03-06T07:34:58.315+0000 7f16d150d640 1 -- 172.21.15.59:0/2506889859 wait complete. 2024-03-06T07:34:58.329 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-06T07:34:58.329 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-06T07:34:58.329 ERROR:tasks.ceph.mds.a:tried to stop a non-running daemon 2024-03-06T07:34:58.332 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:34:58.332 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.341 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:34:58.341 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:34:58.341 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.353 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:34:58.354 ERROR:tasks.ceph.mds.d:tried to stop a non-running daemon 2024-03-06T07:34:58.356 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-06T07:34:58.356 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-06T07:34:58.356 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.410 INFO:tasks.ceph.osd.0:Stopped 2024-03-06T07:34:58.410 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-06T07:34:58.410 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.479 INFO:tasks.ceph.osd.1:Stopped 2024-03-06T07:34:58.480 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-06T07:34:58.480 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.548 INFO:tasks.ceph.osd.2:Stopped 2024-03-06T07:34:58.548 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-06T07:34:58.548 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.617 INFO:tasks.ceph.osd.3:Stopped 2024-03-06T07:34:58.617 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-06T07:34:58.617 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.649 INFO:tasks.ceph.osd.4:Stopped 2024-03-06T07:34:58.650 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-06T07:34:58.650 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.721 INFO:tasks.ceph.osd.5:Stopped 2024-03-06T07:34:58.722 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-06T07:34:58.722 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.792 INFO:tasks.ceph.osd.6:Stopped 2024-03-06T07:34:58.792 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-06T07:34:58.792 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.867 INFO:tasks.ceph.osd.7:Stopped 2024-03-06T07:34:58.867 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-06T07:34:58.867 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-06T07:34:58.867 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.892 INFO:tasks.ceph.mgr.y:Stopped 2024-03-06T07:34:58.893 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-06T07:34:58.893 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.936 INFO:tasks.ceph.mgr.x:Stopped 2024-03-06T07:34:58.937 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-06T07:34:58.937 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-06T07:34:58.937 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.948 INFO:tasks.ceph.mon.a:Stopped 2024-03-06T07:34:58.949 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-06T07:34:58.949 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:58.990 INFO:tasks.ceph.mon.b:Stopped 2024-03-06T07:34:58.990 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-06T07:34:58.990 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:59.043 INFO:tasks.ceph.mon.c:Stopped 2024-03-06T07:34:59.043 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-06T07:34:59.044 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-06T07:34:59.077 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:34:59.077 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-06T07:34:59.290 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:34:59.290 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-06T07:34:59.381 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:34:59.381 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-06T07:34:59.437 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:34:59.437 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-06T07:34:59.480 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:34:59.480 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-06T07:34:59.665 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:34:59.666 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-06T07:34:59.730 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:34:59.730 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-06T07:34:59.780 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:34:59.780 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-06T07:34:59.830 INFO:tasks.ceph:Archiving mon data... 2024-03-06T07:34:59.831 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061/data/mon.a.tgz 2024-03-06T07:34:59.831 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2024-03-06T07:34:59.847 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.zBJZDMAie7 2024-03-06T07:34:59.847 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.zBJZDMAie7 2024-03-06T07:34:59.992 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.zBJZDMAie7 2024-03-06T07:35:00.076 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.zBJZDMAie7 is 388KB 2024-03-06T07:35:00.172 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.zBJZDMAie7 2024-03-06T07:35:00.186 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061/data/mon.b.tgz 2024-03-06T07:35:00.186 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-03-06T07:35:00.203 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.T8jmDi0nX5 2024-03-06T07:35:00.204 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.T8jmDi0nX5 2024-03-06T07:35:00.350 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.T8jmDi0nX5 2024-03-06T07:35:00.434 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.T8jmDi0nX5 is 391KB 2024-03-06T07:35:00.531 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.T8jmDi0nX5 2024-03-06T07:35:00.545 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061/data/mon.c.tgz 2024-03-06T07:35:00.545 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-03-06T07:35:00.599 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.1hW6S1R1yY 2024-03-06T07:35:00.599 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.1hW6S1R1yY 2024-03-06T07:35:00.735 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.1hW6S1R1yY 2024-03-06T07:35:00.820 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.1hW6S1R1yY is 391KB 2024-03-06T07:35:00.888 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.1hW6S1R1yY 2024-03-06T07:35:00.902 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-06T07:35:00.903 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:35:00.905 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:35:00.987 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:35:01.024 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:35:01.025 INFO:tasks.ceph:Archiving crash dumps... 2024-03-06T07:35:01.027 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061/remote/smithi059/crash 2024-03-06T07:35:01.027 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:35:01.055 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061/remote/smithi164/crash 2024-03-06T07:35:01.056 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:35:01.090 INFO:tasks.ceph:Shutting down logrotate 2024-03-06T07:35:01.090 DEBUG:teuthology.orchestra.run.smithi059:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:01.100 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:01.182 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:01.209 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:01.236 INFO:tasks.ceph:Compressing logs... 2024-03-06T07:35:01.237 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:35:01.252 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:35:01.330 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-06T07:35:01.330 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38221.log 2024-03-06T07:35:01.331 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38381.log 2024-03-06T07:35:01.331 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38261.log 2024-03-06T07:35:01.331 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38381.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38221.log.gz 2024-03-06T07:35:01.331 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:35:01.331 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38381.log.gz 2024-03-06T07:35:01.331 INFO:teuthology.orchestra.run.smithi164.stderr:gzip/var/log/ceph/ceph-client.admin.38261.log: -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-06T07:35:01.331 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38341.log 2024-03-06T07:35:01.332 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38261.log.gz 2024-03-06T07:35:01.332 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-06T07:35:01.332 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38181.log 2024-03-06T07:35:01.332 INFO:teuthology.orchestra.run.smithi164.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38341.log.gz 2024-03-06T07:35:01.332 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-06T07:35:01.332 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38181.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-06T07:35:01.333 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38125.log 2024-03-06T07:35:01.333 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38181.log.gz 2024-03-06T07:35:01.333 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-06T07:35:01.333 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.45149.log 2024-03-06T07:35:01.333 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.38125.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38125.log.gz 2024-03-06T07:35:01.333 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38421.log 2024-03-06T07:35:01.333 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.3.45149.log: 90.5% -- replaced with /var/log/ceph/ceph-client.3.45149.log.gz 2024-03-06T07:35:01.333 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:35:01.334 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38421.log: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:35:01.334 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38421.log.gz 2024-03-06T07:35:01.334 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44696.log 2024-03-06T07:35:01.334 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38301.log 2024-03-06T07:35:01.334 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.1.44696.log: 89.8% -- replaced with /var/log/ceph/ceph-client.1.44696.log.gz 2024-03-06T07:35:01.334 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-06T07:35:01.334 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38301.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38301.log.gz 2024-03-06T07:35:01.334 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-06T07:35:01.334 INFO:teuthology.orchestra.run.smithi164.stderr: 92.7%/var/log/ceph/ceph-mon.b.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:35:01.335 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-06T07:35:01.455 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.c.log: 93.9% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-06T07:35:01.463 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-06T07:35:01.497 INFO:teuthology.orchestra.run.smithi164.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-06T07:35:01.695 INFO:teuthology.orchestra.run.smithi164.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-06T07:35:01.709 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38528.log 2024-03-06T07:35:01.709 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68211.log 2024-03-06T07:35:01.709 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65078.log 2024-03-06T07:35:01.710 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87045.log 2024-03-06T07:35:01.710 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38528.log.gz 2024-03-06T07:35:01.710 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65078.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.68211.log.gz 2024-03-06T07:35:01.710 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56758.log 2024-03-06T07:35:01.710 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65078.log.gz 2024-03-06T07:35:01.710 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61801.log 2024-03-06T07:35:01.710 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.87045.log.gz 2024-03-06T07:35:01.710 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84421.log 2024-03-06T07:35:01.711 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61801.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56758.log.gz 2024-03-06T07:35:01.711 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50463.log 2024-03-06T07:35:01.711 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84421.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61801.log.gz 2024-03-06T07:35:01.711 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87591.log 2024-03-06T07:35:01.711 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50463.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.84421.log.gz 2024-03-06T07:35:01.711 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67815.log 2024-03-06T07:35:01.711 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87591.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50463.log.gz 2024-03-06T07:35:01.711 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60851.log 2024-03-06T07:35:01.712 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87591.log.gz 2024-03-06T07:35:01.712 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90561.log 2024-03-06T07:35:01.712 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87945.log 2024-03-06T07:35:01.712 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67815.log.gz 2024-03-06T07:35:01.712 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90561.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60851.log.gz 2024-03-06T07:35:01.712 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66248.log 2024-03-06T07:35:01.712 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90561.log.gz 2024-03-06T07:35:01.712 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80602.log 2024-03-06T07:35:01.713 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66248.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.87945.log.gz -5 2024-03-06T07:35:01.713 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.91977.log 2024-03-06T07:35:01.713 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66248.log.gz 2024-03-06T07:35:01.713 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86565.log 2024-03-06T07:35:01.713 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80602.log.gz 2024-03-06T07:35:01.713 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90225.log 2024-03-06T07:35:01.713 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63988.log 2024-03-06T07:35:01.713 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91977.log.gz 2024-03-06T07:35:01.713 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60909.log 2024-03-06T07:35:01.714 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86565.log.gz 2024-03-06T07:35:01.714 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63988.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90225.log.gz 2024-03-06T07:35:01.714 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63247.log 2024-03-06T07:35:01.714 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44219.log 2024-03-06T07:35:01.714 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63988.log.gz 2024-03-06T07:35:01.714 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63247.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60909.log.gzgzip 2024-03-06T07:35:01.714 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.75038.log 2024-03-06T07:35:01.714 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44219.log: gzip 88.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.63247.log.gz -- 2024-03-06T07:35:01.714 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.86001.log 2024-03-06T07:35:01.715 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75038.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.44219.log.gz 2024-03-06T07:35:01.715 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-06T07:35:01.715 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86001.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75038.log.gz 2024-03-06T07:35:01.715 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79048.log 2024-03-06T07:35:01.715 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mon.a.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.86001.log.gz 2024-03-06T07:35:01.715 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89307.log 2024-03-06T07:35:01.715 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70747.log 2024-03-06T07:35:01.715 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80994.log 2024-03-06T07:35:01.716 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79048.log.gz 2024-03-06T07:35:01.716 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70747.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89307.log.gz 2024-03-06T07:35:01.716 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54896.log 2024-03-06T07:35:01.716 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80994.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70747.log.gz 2024-03-06T07:35:01.716 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52130.log 2024-03-06T07:35:01.716 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54896.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80994.log.gz 2024-03-06T07:35:01.716 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67218.log 2024-03-06T07:35:01.716 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52130.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54896.log.gz 2024-03-06T07:35:01.716 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91548.log 2024-03-06T07:35:01.717 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86187.log 87.9% 2024-03-06T07:35:01.717 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52130.log.gz 2024-03-06T07:35:01.717 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67218.log.gz 2024-03-06T07:35:01.717 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92493.log 2024-03-06T07:35:01.717 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91548.log.gz 2024-03-06T07:35:01.717 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57010.log 2024-03-06T07:35:01.717 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92493.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86187.log.gz 2024-03-06T07:35:01.717 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54980.log 2024-03-06T07:35:01.717 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74833.log 2024-03-06T07:35:01.717 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.92493.log.gz 2024-03-06T07:35:01.718 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54980.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57010.log.gz 2024-03-06T07:35:01.718 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81923.log 2024-03-06T07:35:01.718 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54980.log.gz 2024-03-06T07:35:01.718 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90123.log 2024-03-06T07:35:01.718 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81923.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74833.log.gz 2024-03-06T07:35:01.718 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92367.log 2024-03-06T07:35:01.718 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81923.log.gz 2024-03-06T07:35:01.718 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55404.log 2024-03-06T07:35:01.718 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92367.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90123.log.gz 2024-03-06T07:35:01.719 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91212.log 2024-03-06T07:35:01.719 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55404.log: gzip -5 --verbose -- 89.1% /var/log/ceph/ceph-client.admin.87549.log 2024-03-06T07:35:01.719 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92367.log.gz 2024-03-06T07:35:01.719 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55404.log.gz 2024-03-06T07:35:01.719 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67435.log 2024-03-06T07:35:01.719 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65036.log 2024-03-06T07:35:01.719 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91212.log.gz 2024-03-06T07:35:01.719 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8%/var/log/ceph/ceph-client.admin.67435.log: -- replaced with /var/log/ceph/ceph-client.admin.87549.log.gz 2024-03-06T07:35:01.719 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56590.log 2024-03-06T07:35:01.720 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67435.log.gz 2024-03-06T07:35:01.720 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71270.log 2024-03-06T07:35:01.720 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52298.log 2024-03-06T07:35:01.720 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65036.log.gz 2024-03-06T07:35:01.720 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71270.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56590.log.gz 2024-03-06T07:35:01.720 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76477.log 2024-03-06T07:35:01.720 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71270.log.gz 2024-03-06T07:35:01.720 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83868.log 2024-03-06T07:35:01.720 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76477.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52298.log.gz 2024-03-06T07:35:01.721 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82348.log 2024-03-06T07:35:01.721 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76477.log.gz 2024-03-06T07:35:01.721 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.83868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81965.log 2024-03-06T07:35:01.721 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83868.log.gz 2024-03-06T07:35:01.721 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81036.log 2024-03-06T07:35:01.721 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50715.log 2024-03-06T07:35:01.721 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82348.log.gz 2024-03-06T07:35:01.721 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5%/var/log/ceph/ceph-client.admin.81036.log: -- replaced with /var/log/ceph/ceph-client.admin.81965.log.gz 2024-03-06T07:35:01.722 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61332.log 2024-03-06T07:35:01.722 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81036.log.gz 2024-03-06T07:35:01.722 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58208.log 2024-03-06T07:35:01.722 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61332.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50715.log.gz 2024-03-06T07:35:01.722 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50967.log 2024-03-06T07:35:01.722 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84523.log 2024-03-06T07:35:01.722 INFO:teuthology.orchestra.run.smithi059.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61332.log.gz 2024-03-06T07:35:01.722 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58208.log.gz 2024-03-06T07:35:01.722 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88803.log 2024-03-06T07:35:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84523.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50967.log.gz 2024-03-06T07:35:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62013.log 2024-03-06T07:35:01.723 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84523.log.gz 2024-03-06T07:35:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92104.log 2024-03-06T07:35:01.723 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82390.log 88.7% 2024-03-06T07:35:01.723 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88803.log.gz 2024-03-06T07:35:01.723 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0%/var/log/ceph/ceph-client.admin.92104.log: -- replaced with /var/log/ceph/ceph-client.admin.62013.log.gz 2024-03-06T07:35:01.724 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58292.log 2024-03-06T07:35:01.724 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4%/var/log/ceph/ceph-client.admin.82390.log: -- replaced with /var/log/ceph/ceph-client.admin.92104.log.gz 2024-03-06T07:35:01.724 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87465.log 2024-03-06T07:35:01.724 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81797.log 88.7% 2024-03-06T07:35:01.724 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82390.log.gz 2024-03-06T07:35:01.724 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87465.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58292.log.gz 2024-03-06T07:35:01.724 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46323.log 2024-03-06T07:35:01.724 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.87465.log.gz 2024-03-06T07:35:01.724 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47698.log 2024-03-06T07:35:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44811.log 2024-03-06T07:35:01.725 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81797.log.gz 2024-03-06T07:35:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47698.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46323.log.gz 2024-03-06T07:35:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54345.log 2024-03-06T07:35:01.725 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47698.log.gz 2024-03-06T07:35:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71907.log 2024-03-06T07:35:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54345.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.44811.log.gz -5 2024-03-06T07:35:01.725 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51759.log 2024-03-06T07:35:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71907.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54345.log.gz 2024-03-06T07:35:01.726 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60683.log 2024-03-06T07:35:01.726 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71907.log.gz 2024-03-06T07:35:01.726 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61715.log 2024-03-06T07:35:01.726 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51759.log.gz 2024-03-06T07:35:01.726 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63331.log 2024-03-06T07:35:01.726 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46533.log 2024-03-06T07:35:01.726 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60683.log.gz 2024-03-06T07:35:01.726 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63331.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61715.log.gz 2024-03-06T07:35:01.726 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82225.log 2024-03-06T07:35:01.726 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63331.log.gz 2024-03-06T07:35:01.727 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44303.log 2024-03-06T07:35:01.727 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82225.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46533.log.gz 2024-03-06T07:35:01.727 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62967.log 2024-03-06T07:35:01.727 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82225.log.gz 2024-03-06T07:35:01.727 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51299.log 2024-03-06T07:35:01.727 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88155.log 2024-03-06T07:35:01.727 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44303.log.gz 2024-03-06T07:35:01.727 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51299.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62967.log.gz 2024-03-06T07:35:01.727 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87633.log 2024-03-06T07:35:01.728 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51299.log.gz 2024-03-06T07:35:01.728 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82786.log 2024-03-06T07:35:01.728 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88155.log.gz 2024-03-06T07:35:01.728 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55320.log 2024-03-06T07:35:01.728 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82786.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87633.log.gz 2024-03-06T07:35:01.728 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66653.log 2024-03-06T07:35:01.728 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82786.log.gz 2024-03-06T07:35:01.728 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55194.log 2024-03-06T07:35:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66653.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55320.log.gz 2024-03-06T07:35:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44853.log 2024-03-06T07:35:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55194.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66653.log.gz 2024-03-06T07:35:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56404.log 2024-03-06T07:35:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44853.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55194.log.gz 2024-03-06T07:35:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65414.log 2024-03-06T07:35:01.729 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6%/var/log/ceph/ceph-client.admin.56404.log: -- replaced with /var/log/ceph/ceph-client.admin.44853.log.gz 2024-03-06T07:35:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69951.log 2024-03-06T07:35:01.729 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65414.log: gzip 89.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.56404.log.gz --verbose 2024-03-06T07:35:01.730 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.74886.log 2024-03-06T07:35:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69951.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65414.log.gz 2024-03-06T07:35:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73453.log 2024-03-06T07:35:01.730 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69951.log.gz 2024-03-06T07:35:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.47576.log 2024-03-06T07:35:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58124.log 2024-03-06T07:35:01.730 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.74886.log.gz 2024-03-06T07:35:01.730 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.2.47576.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73453.log.gz 2024-03-06T07:35:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73032.log 2024-03-06T07:35:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63484.log 2024-03-06T07:35:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73032.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58124.log.gz 2024-03-06T07:35:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56884.log 2024-03-06T07:35:01.731 INFO:teuthology.orchestra.run.smithi059.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.2.47576.log.gz 2024-03-06T07:35:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52214.log 2024-03-06T07:35:01.731 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73032.log.gz 2024-03-06T07:35:01.731 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65330.log 2024-03-06T07:35:01.732 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63484.log.gz 2024-03-06T07:35:01.732 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52214.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56884.log.gz 2024-03-06T07:35:01.732 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76435.log 2024-03-06T07:35:01.732 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65330.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52214.log.gz 2024-03-06T07:35:01.732 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72161.log 2024-03-06T07:35:01.732 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76435.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65330.log.gz 2024-03-06T07:35:01.732 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89955.log 2024-03-06T07:35:01.732 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72161.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76435.log.gz 2024-03-06T07:35:01.732 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46197.log 2024-03-06T07:35:01.733 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89955.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72161.log.gz 2024-03-06T07:35:01.733 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91777.log 2024-03-06T07:35:01.733 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89955.log.gz 2024-03-06T07:35:01.733 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88719.log 2024-03-06T07:35:01.733 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91777.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46197.log.gz 2024-03-06T07:35:01.733 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77553.log 2024-03-06T07:35:01.733 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88719.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91777.log.gz 2024-03-06T07:35:01.733 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61929.log 2024-03-06T07:35:01.733 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92850.log 2024-03-06T07:35:01.733 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88719.log.gz 2024-03-06T07:35:01.734 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61929.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77553.log.gz 2024-03-06T07:35:01.734 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49317.log 2024-03-06T07:35:01.734 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72756.log 2024-03-06T07:35:01.734 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61929.log.gz 2024-03-06T07:35:01.734 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49317.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92850.log.gz 2024-03-06T07:35:01.734 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45074.log 2024-03-06T07:35:01.734 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7%/var/log/ceph/ceph-client.0.72756.log: -- replaced with /var/log/ceph/ceph-client.admin.49317.log.gz 2024-03-06T07:35:01.734 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86229.log 2024-03-06T07:35:01.734 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77043.log 2024-03-06T07:35:01.735 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57357.log 2024-03-06T07:35:01.735 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45074.log.gz 2024-03-06T07:35:01.735 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86229.log.gz 2024-03-06T07:35:01.735 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82912.log 2024-03-06T07:35:01.735 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77043.log.gz 2024-03-06T07:35:01.735 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51633.log 2024-03-06T07:35:01.735 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56167.log 2024-03-06T07:35:01.735 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82912.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57357.log.gz 2024-03-06T07:35:01.735 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51633.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82912.log.gz 2024-03-06T07:35:01.736 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89913.log 2024-03-06T07:35:01.736 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85403.log 2024-03-06T07:35:01.736 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51633.log.gz 2024-03-06T07:35:01.736 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56167.log: /var/log/ceph/ceph-client.admin.89913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84463.log 2024-03-06T07:35:01.736 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56167.log.gz 2024-03-06T07:35:01.736 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89913.log.gz 2024-03-06T07:35:01.736 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80140.log 2024-03-06T07:35:01.736 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58040.log 2024-03-06T07:35:01.737 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85403.log.gz 2024-03-06T07:35:01.737 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80140.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84463.log.gz 2024-03-06T07:35:01.737 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63736.log 2024-03-06T07:35:01.737 INFO:teuthology.orchestra.run.smithi059.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80140.log.gz 2024-03-06T07:35:01.737 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78344.log 2024-03-06T07:35:01.737 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58040.log: /var/log/ceph/ceph-client.admin.63736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86961.log 2024-03-06T07:35:01.737 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58040.log.gz 2024-03-06T07:35:01.737 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78344.log: gzip 88.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.63736.log.gz -- 2024-03-06T07:35:01.737 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.60725.log 2024-03-06T07:35:01.737 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3%/var/log/ceph/ceph-client.admin.86961.log: -- replaced with /var/log/ceph/ceph-client.admin.78344.log.gz 2024-03-06T07:35:01.738 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76771.log 2024-03-06T07:35:01.738 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86961.log.gz 2024-03-06T07:35:01.738 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71613.log 2024-03-06T07:35:01.738 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60725.log: /var/log/ceph/ceph-client.admin.76771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63163.log 2024-03-06T07:35:01.738 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60725.log.gz 2024-03-06T07:35:01.738 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71613.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76771.log.gz 2024-03-06T07:35:01.738 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54387.log 2024-03-06T07:35:01.738 INFO:teuthology.orchestra.run.smithi059.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.0.72756.log.gz 2024-03-06T07:35:01.739 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55278.log 2024-03-06T07:35:01.739 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71613.log.gz 2024-03-06T07:35:01.739 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63163.log: /var/log/ceph/ceph-client.admin.54387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85959.log 2024-03-06T07:35:01.739 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63163.log.gz 2024-03-06T07:35:01.739 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55278.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54387.log.gz 2024-03-06T07:35:01.739 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59907.log 2024-03-06T07:35:01.739 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6%/var/log/ceph/ceph-client.admin.85959.log: -- replaced with /var/log/ceph/ceph-client.admin.55278.log.gz 2024-03-06T07:35:01.739 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62139.log 2024-03-06T07:35:01.739 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85959.log.gz 2024-03-06T07:35:01.739 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.59907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53441.log 2024-03-06T07:35:01.740 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68001.log 2024-03-06T07:35:01.740 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53441.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62139.log.gz 2024-03-06T07:35:01.740 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56800.log 2024-03-06T07:35:01.740 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53441.log.gz 2024-03-06T07:35:01.740 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84623.log 2024-03-06T07:35:01.741 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68001.log.gz 2024-03-06T07:35:01.741 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70579.log 2024-03-06T07:35:01.741 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56800.log: /var/log/ceph/ceph-client.admin.84623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79132.log 2024-03-06T07:35:01.741 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56800.log.gz 2024-03-06T07:35:01.741 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70579.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84623.log.gz 2024-03-06T07:35:01.741 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50421.log 2024-03-06T07:35:01.741 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45403.log 2024-03-06T07:35:01.741 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79132.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70579.log.gz 2024-03-06T07:35:01.741 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50421.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.79132.log.gz 2024-03-06T07:35:01.742 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49359.log 2024-03-06T07:35:01.742 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50421.log.gz 2024-03-06T07:35:01.742 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45748.log 2024-03-06T07:35:01.742 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45403.log: /var/log/ceph/ceph-client.admin.49359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60641.log 2024-03-06T07:35:01.742 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45403.log.gz 2024-03-06T07:35:01.742 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49359.log.gz 2024-03-06T07:35:01.742 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45748.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.32085.log 2024-03-06T07:35:01.742 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45748.log.gz 2024-03-06T07:35:01.743 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45402.log 2024-03-06T07:35:01.743 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.tmp-client.admin.32085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63862.log 2024-03-06T07:35:01.743 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60641.log.gz 2024-03-06T07:35:01.743 INFO:teuthology.orchestra.run.smithi059.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.32085.log.gz 2024-03-06T07:35:01.743 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92451.log 2024-03-06T07:35:01.743 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45402.log.gz 2024-03-06T07:35:01.743 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48372.log 2024-03-06T07:35:01.743 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63862.log.gz 2024-03-06T07:35:01.744 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56209.log 2024-03-06T07:35:01.744 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92451.log: /var/log/ceph/ceph-client.admin.48372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74412.log 2024-03-06T07:35:01.744 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92451.log.gz 2024-03-06T07:35:01.744 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48372.log.gz 2024-03-06T07:35:01.744 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59524.log 2024-03-06T07:35:01.744 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.74412.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56209.log.gz 2024-03-06T07:35:01.744 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91338.log 2024-03-06T07:35:01.744 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80560.log 2024-03-06T07:35:01.744 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59524.log: /var/log/ceph/ceph-client.admin.91338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55068.log 2024-03-06T07:35:01.745 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59524.log.gz 2024-03-06T07:35:01.745 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91338.log.gz 2024-03-06T07:35:01.745 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52448.log 2024-03-06T07:35:01.745 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53762.log 2024-03-06T07:35:01.745 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55068.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80560.log.gz 2024-03-06T07:35:01.745 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70097.log 2024-03-06T07:35:01.745 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55068.log.gz 2024-03-06T07:35:01.745 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52448.log.gz 2024-03-06T07:35:01.746 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82049.log 2024-03-06T07:35:01.746 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70097.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53762.log.gz 2024-03-06T07:35:01.746 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82563.log 2024-03-06T07:35:01.746 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.70097.log.gz --verbose 2024-03-06T07:35:01.746 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.70789.log 2024-03-06T07:35:01.746 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82049.log: /var/log/ceph/ceph-client.admin.82563.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82049.log.gz 2024-03-06T07:35:01.746 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82870.log 2024-03-06T07:35:01.746 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82563.log.gz/var/log/ceph/ceph-client.admin.70789.log: 2024-03-06T07:35:01.746 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87087.log 2024-03-06T07:35:01.747 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82870.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70789.log.gz 2024-03-06T07:35:01.747 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70300.log 2024-03-06T07:35:01.747 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82870.log.gz 2024-03-06T07:35:01.747 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66485.log 2024-03-06T07:35:01.747 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87087.log.gz 2024-03-06T07:35:01.747 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71779.log 2024-03-06T07:35:01.747 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51675.log 2024-03-06T07:35:01.747 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66485.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70300.log.gz 2024-03-06T07:35:01.748 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89451.log 2024-03-06T07:35:01.748 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66485.log.gz 2024-03-06T07:35:01.748 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88887.log 89.0% 2024-03-06T07:35:01.748 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71779.log.gz 2024-03-06T07:35:01.748 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85671.log 2024-03-06T07:35:01.748 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51675.log.gz 2024-03-06T07:35:01.748 INFO:teuthology.orchestra.run.smithi059.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.0.74412.log.gz 2024-03-06T07:35:01.748 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.89451.log.gz 2024-03-06T07:35:01.749 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45413.log 2024-03-06T07:35:01.749 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85671.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88887.log.gz 2024-03-06T07:35:01.749 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49071.log 2024-03-06T07:35:01.749 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85671.log.gz 2024-03-06T07:35:01.749 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51958.log 2024-03-06T07:35:01.749 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49071.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45413.log.gz 2024-03-06T07:35:01.750 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73615.log 2024-03-06T07:35:01.750 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49071.log.gz 2024-03-06T07:35:01.750 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69089.log 2024-03-06T07:35:01.750 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51958.log: /var/log/ceph/ceph-client.admin.73615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63778.log 2024-03-06T07:35:01.750 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51958.log.gz 2024-03-06T07:35:01.750 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69089.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73615.log.gz 2024-03-06T07:35:01.750 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44559.log 2024-03-06T07:35:01.750 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63778.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69089.log.gz 2024-03-06T07:35:01.751 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84869.log 2024-03-06T07:35:01.751 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63778.log.gz 2024-03-06T07:35:01.751 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80056.log 2024-03-06T07:35:01.751 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44559.log.gz 2024-03-06T07:35:01.751 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85487.log 2024-03-06T07:35:01.751 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84869.log: /var/log/ceph/ceph-client.admin.80056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64282.log 2024-03-06T07:35:01.751 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84869.log.gz 2024-03-06T07:35:01.751 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85487.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80056.log.gz 2024-03-06T07:35:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88239.log 2024-03-06T07:35:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48666.log 2024-03-06T07:35:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64282.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85487.log.gz 2024-03-06T07:35:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55446.log 2024-03-06T07:35:01.752 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64282.log.gz 2024-03-06T07:35:01.752 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88239.log.gz 2024-03-06T07:35:01.752 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56716.log 2024-03-06T07:35:01.753 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44685.log 2024-03-06T07:35:01.753 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48666.log.gz 2024-03-06T07:35:01.753 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55446.log.gz 2024-03-06T07:35:01.753 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84665.log 2024-03-06T07:35:01.753 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56716.log.gz 2024-03-06T07:35:01.753 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50799.log 2024-03-06T07:35:01.753 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44685.log: /var/log/ceph/ceph-client.admin.84665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57220.log 2024-03-06T07:35:01.753 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44685.log.gz 2024-03-06T07:35:01.754 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50799.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84665.log.gz 2024-03-06T07:35:01.754 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59052.log 2024-03-06T07:35:01.754 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50799.log.gz 2024-03-06T07:35:01.754 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58436.log 2024-03-06T07:35:01.754 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57220.log: /var/log/ceph/ceph-client.admin.59052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73883.log 2024-03-06T07:35:01.754 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58436.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57220.log.gz 2024-03-06T07:35:01.754 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.59052.log.gz 2024-03-06T07:35:01.754 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75497.log 2024-03-06T07:35:01.755 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79333.log 2024-03-06T07:35:01.755 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73883.log: 90.9%/var/log/ceph/ceph-client.admin.75497.log: -- replaced with /var/log/ceph/ceph-client.admin.58436.log.gz 2024-03-06T07:35:01.755 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47082.log 2024-03-06T07:35:01.755 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73883.log.gz 2024-03-06T07:35:01.755 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79333.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75497.log.gz 2024-03-06T07:35:01.755 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81273.log 2024-03-06T07:35:01.755 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.79333.log.gz 2024-03-06T07:35:01.755 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92577.log 2024-03-06T07:35:01.756 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.47082.log: /var/log/ceph/ceph-client.admin.81273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49864.log 2024-03-06T07:35:01.756 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92577.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81273.log.gz 2024-03-06T07:35:01.756 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50589.log 2024-03-06T07:35:01.756 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92577.log.gz 2024-03-06T07:35:01.756 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61971.log 2024-03-06T07:35:01.756 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49864.log: 90.3% -- replaced with /var/log/ceph/ceph-client.0.47082.log.gz 2024-03-06T07:35:01.756 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54938.log 2024-03-06T07:35:01.756 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49864.log.gz 2024-03-06T07:35:01.756 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70663.log 2024-03-06T07:35:01.757 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50589.log.gz 2024-03-06T07:35:01.757 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54938.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61971.log.gz 2024-03-06T07:35:01.757 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58502.log 2024-03-06T07:35:01.757 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54938.log.gz 2024-03-06T07:35:01.757 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69831.log 2024-03-06T07:35:01.757 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70663.log: /var/log/ceph/ceph-client.admin.58502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57645.log 2024-03-06T07:35:01.757 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70663.log.gz 2024-03-06T07:35:01.757 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69831.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58502.log.gz 2024-03-06T07:35:01.757 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62223.log 2024-03-06T07:35:01.758 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69831.log.gz 2024-03-06T07:35:01.758 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73495.log 2024-03-06T07:35:01.758 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57956.log 2024-03-06T07:35:01.758 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62223.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57645.log.gz 2024-03-06T07:35:01.758 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53263.log 2024-03-06T07:35:01.758 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62223.log.gz 2024-03-06T07:35:01.758 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73495.log.gz 2024-03-06T07:35:01.758 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88071.log 2024-03-06T07:35:01.758 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57956.log: /var/log/ceph/ceph-client.admin.53263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73369.log 2024-03-06T07:35:01.759 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57956.log.gz 2024-03-06T07:35:01.759 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88071.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53263.log.gz 2024-03-06T07:35:01.759 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47204.log 2024-03-06T07:35:01.759 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88071.log.gz 2024-03-06T07:35:01.759 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78200.log 2024-03-06T07:35:01.759 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73369.log.gz 2024-03-06T07:35:01.759 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47892.log 2024-03-06T07:35:01.759 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47204.log: /var/log/ceph/ceph-client.admin.78200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67941.log 2024-03-06T07:35:01.759 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47204.log.gz 2024-03-06T07:35:01.760 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47892.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.78200.log.gz 2024-03-06T07:35:01.760 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59178.log 2024-03-06T07:35:01.760 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47892.log.gz 2024-03-06T07:35:01.760 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52707.log 2024-03-06T07:35:01.760 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67941.log: /var/log/ceph/ceph-client.admin.59178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80746.log 2024-03-06T07:35:01.760 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67941.log.gz 2024-03-06T07:35:01.760 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.52707.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59178.log.gz 2024-03-06T07:35:01.760 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62849.log 2024-03-06T07:35:01.760 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55596.log 2024-03-06T07:35:01.761 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80746.log: /var/log/ceph/ceph-client.admin.62849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52042.log 2024-03-06T07:35:01.761 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80746.log.gz 2024-03-06T07:35:01.761 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62849.log.gz 2024-03-06T07:35:01.761 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86691.log 2024-03-06T07:35:01.761 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81231.log 2024-03-06T07:35:01.761 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55596.log.gz 2024-03-06T07:35:01.761 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50883.log 2024-03-06T07:35:01.762 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52042.log.gz 2024-03-06T07:35:01.762 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81231.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86691.log.gz 2024-03-06T07:35:01.762 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67260.log 2024-03-06T07:35:01.762 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81231.log.gz 2024-03-06T07:35:01.762 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89997.log 2024-03-06T07:35:01.762 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89871.log 2024-03-06T07:35:01.762 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67260.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50883.log.gz 2024-03-06T07:35:01.762 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46029.log 2024-03-06T07:35:01.762 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67260.log.gz 2024-03-06T07:35:01.763 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69911.log 2024-03-06T07:35:01.763 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89997.log.gz 2024-03-06T07:35:01.763 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46029.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89871.log.gz 2024-03-06T07:35:01.763 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81703.log 2024-03-06T07:35:01.763 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46029.log.gz 2024-03-06T07:35:01.763 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose/var/log/ceph/ceph-client.admin.69911.log: -- /var/log/ceph/ceph-client.admin.44517.log 2024-03-06T07:35:01.763 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81703.log: gzip -5 --verbose 87.6% -- /var/log/ceph/ceph-client.admin.90729.log -- replaced with /var/log/ceph/ceph-client.admin.69911.log.gz 2024-03-06T07:35:01.763 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-06T07:35:01.763 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81703.log.gz 2024-03-06T07:35:01.763 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79786.log 2024-03-06T07:35:01.764 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44517.log.gz 2024-03-06T07:35:01.764 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67134.log 2024-03-06T07:35:01.764 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77679.log 2024-03-06T07:35:01.764 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79786.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90729.log.gz 2024-03-06T07:35:01.764 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59320.log 2024-03-06T07:35:01.764 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79786.log.gz 2024-03-06T07:35:01.764 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85545.log 2024-03-06T07:35:01.764 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67134.log.gz 2024-03-06T07:35:01.764 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59320.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77679.log.gz 2024-03-06T07:35:01.765 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46449.log 2024-03-06T07:35:01.765 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88929.log 2024-03-06T07:35:01.765 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59320.log.gz 2024-03-06T07:35:01.765 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44116.log 2024-03-06T07:35:01.765 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.85545.log.gz 2024-03-06T07:35:01.765 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46449.log.gz 2024-03-06T07:35:01.765 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65204.log 2024-03-06T07:35:01.765 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76519.log 2024-03-06T07:35:01.765 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44116.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88929.log.gz 2024-03-06T07:35:01.766 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79744.log 2024-03-06T07:35:01.766 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44116.log.gz 2024-03-06T07:35:01.766 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76519.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65204.log.gzgzip 2024-03-06T07:35:01.766 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48498.log 2024-03-06T07:35:01.766 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79744.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76519.log.gz 2024-03-06T07:35:01.766 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57910.log 2024-03-06T07:35:01.766 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79744.log.gz 2024-03-06T07:35:01.766 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85319.log 2024-03-06T07:35:01.766 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57910.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48498.log.gz 2024-03-06T07:35:01.767 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75540.log 2024-03-06T07:35:01.767 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57910.log.gz 2024-03-06T07:35:01.767 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44345.log 2024-03-06T07:35:01.767 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85319.log.gz 2024-03-06T07:35:01.767 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82145.log 2024-03-06T07:35:01.767 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80266.log 2024-03-06T07:35:01.767 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44345.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.75540.log.gz 2024-03-06T07:35:01.767 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82145.log: gzip -5 88.4% --verbose -- /var/log/ceph/ceph-client.admin.86439.log 2024-03-06T07:35:01.767 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44345.log.gz 2024-03-06T07:35:01.768 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80266.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82145.log.gz 2024-03-06T07:35:01.768 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64072.log 2024-03-06T07:35:01.768 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80266.log.gz 2024-03-06T07:35:01.768 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73227.log 2024-03-06T07:35:01.768 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71695.log 2024-03-06T07:35:01.768 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64072.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86439.log.gz 2024-03-06T07:35:01.768 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56968.log 2024-03-06T07:35:01.769 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64072.log.gz 2024-03-06T07:35:01.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71695.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.73227.log.gz 2024-03-06T07:35:01.769 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64930.log 2024-03-06T07:35:01.769 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71695.log.gz 2024-03-06T07:35:01.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49013.log 2024-03-06T07:35:01.769 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52882.log 2024-03-06T07:35:01.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64930.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56968.log.gz 2024-03-06T07:35:01.769 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70999.log 2024-03-06T07:35:01.770 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64930.log.gz 2024-03-06T07:35:01.770 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49013.log.gz 2024-03-06T07:35:01.770 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90519.log 2024-03-06T07:35:01.770 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.70999.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.90645.log 2024-03-06T07:35:01.770 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52882.log.gz 2024-03-06T07:35:01.770 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90519.log: gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.53139.log 2024-03-06T07:35:01.770 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70999.log.gz 2024-03-06T07:35:01.770 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90645.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.90519.log.gz 2024-03-06T07:35:01.771 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50925.log 2024-03-06T07:35:01.771 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90645.log.gz 2024-03-06T07:35:01.771 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85587.log 2024-03-06T07:35:01.771 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53139.log: /var/log/ceph/ceph-client.admin.50925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68169.log 2024-03-06T07:35:01.771 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53139.log.gz 2024-03-06T07:35:01.771 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50925.log.gz 2024-03-06T07:35:01.771 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90435.log 2024-03-06T07:35:01.771 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68169.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.85587.log.gz 2024-03-06T07:35:01.772 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64976.log 2024-03-06T07:35:01.772 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68169.log.gz 2024-03-06T07:35:01.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78961.log 2024-03-06T07:35:01.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64976.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.90435.log.gz 2024-03-06T07:35:01.772 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87861.log 2024-03-06T07:35:01.772 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78961.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64976.log.gz 2024-03-06T07:35:01.772 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64457.log 2024-03-06T07:35:01.772 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78961.log.gz 2024-03-06T07:35:01.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89139.log 2024-03-06T07:35:01.773 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87861.log.gz 2024-03-06T07:35:01.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80392.log 2024-03-06T07:35:01.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63694.log 2024-03-06T07:35:01.773 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64457.log.gz 2024-03-06T07:35:01.773 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.80392.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89139.log.gz 2024-03-06T07:35:01.773 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71993.log 2024-03-06T07:35:01.773 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63694.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.80392.log.gz -5 2024-03-06T07:35:01.774 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.70873.log 2024-03-06T07:35:01.774 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63694.log.gz 2024-03-06T07:35:01.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48708.log 2024-03-06T07:35:01.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70873.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71993.log.gz 2024-03-06T07:35:01.774 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59220.log 2024-03-06T07:35:01.774 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70873.log.gz 2024-03-06T07:35:01.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63610.log 2024-03-06T07:35:01.774 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59220.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48708.log.gz 2024-03-06T07:35:01.774 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54730.log 2024-03-06T07:35:01.775 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59220.log.gz 2024-03-06T07:35:01.775 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64751.log 2024-03-06T07:35:01.775 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63610.log.gz 2024-03-06T07:35:01.775 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50337.log 2024-03-06T07:35:01.775 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90165.log 2024-03-06T07:35:01.775 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64751.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54730.log.gz 2024-03-06T07:35:01.775 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81357.log 2024-03-06T07:35:01.775 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64751.log.gz 2024-03-06T07:35:01.775 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0%/var/log/ceph/ceph-client.admin.90165.log: -- replaced with /var/log/ceph/ceph-client.admin.50337.log.gz 2024-03-06T07:35:01.776 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79828.log 2024-03-06T07:35:01.776 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89493.log 2024-03-06T07:35:01.776 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81357.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.90165.log.gz 2024-03-06T07:35:01.776 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48971.log 2024-03-06T07:35:01.776 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81357.log.gz 2024-03-06T07:35:01.776 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79828.log.gz 2024-03-06T07:35:01.776 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69086.log 2024-03-06T07:35:01.776 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.48971.log: 89.1% -5 --verbose -- /var/log/ceph/ceph-client.admin.62097.log 2024-03-06T07:35:01.776 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89493.log.gz 2024-03-06T07:35:01.777 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69086.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.48971.log.gz 2024-03-06T07:35:01.777 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80308.log 2024-03-06T07:35:01.777 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69086.log.gz 2024-03-06T07:35:01.777 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60177.log 2024-03-06T07:35:01.777 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91128.log 2024-03-06T07:35:01.777 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7%/var/log/ceph/ceph-client.admin.80308.log: -- replaced with /var/log/ceph/ceph-client.admin.62097.log.gz 2024-03-06T07:35:01.777 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82306.log 2024-03-06T07:35:01.777 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80308.log.gz 2024-03-06T07:35:01.777 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60177.log.gz 2024-03-06T07:35:01.777 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90351.log 2024-03-06T07:35:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48330.log 2024-03-06T07:35:01.778 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.91128.log.gz 2024-03-06T07:35:01.778 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82306.log.gz 2024-03-06T07:35:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53525.log 2024-03-06T07:35:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48330.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.90351.log.gz 2024-03-06T07:35:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89703.log 2024-03-06T07:35:01.778 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48330.log.gz 2024-03-06T07:35:01.778 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87777.log 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53525.log.gz 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53720.log 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80518.log 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.89703.log.gz 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53720.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71438.log 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87777.log.gz 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80518.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53720.log.gz 2024-03-06T07:35:01.779 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70705.log 2024-03-06T07:35:01.780 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55530.log 2024-03-06T07:35:01.780 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80518.log.gz 2024-03-06T07:35:01.780 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70705.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71438.log.gz 2024-03-06T07:35:01.780 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.79702.log 2024-03-06T07:35:01.780 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55530.log: 93.2% -- replaced with /var/log/ceph/ceph-client.0.59907.log.gz 2024-03-06T07:35:01.780 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70705.log.gz 2024-03-06T07:35:01.780 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53888.log 2024-03-06T07:35:01.780 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79006.log 2024-03-06T07:35:01.781 INFO:teuthology.orchestra.run.smithi059.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.55530.log.gz 2024-03-06T07:35:01.781 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71354.log 2024-03-06T07:35:01.781 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79702.log.gz 2024-03-06T07:35:01.781 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79006.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53888.log.gz 2024-03-06T07:35:01.781 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75162.log 2024-03-06T07:35:01.781 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 88.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.79006.log.gz -- 2024-03-06T07:35:01.781 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.76056.log/var/log/ceph/ceph-client.admin.71354.log: 2024-03-06T07:35:01.781 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75162.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71354.log.gz 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78284.log 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75162.log.gz 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73311.log 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76056.log: /var/log/ceph/ceph-client.admin.78284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43897.log 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76056.log.gz 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78284.log.gz 2024-03-06T07:35:01.782 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87255.log 2024-03-06T07:35:01.783 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81526.log 2024-03-06T07:35:01.783 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.43897.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73311.log.gz 2024-03-06T07:35:01.783 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87255.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43897.log.gz 2024-03-06T07:35:01.783 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67899.log 2024-03-06T07:35:01.783 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55236.log 2024-03-06T07:35:01.783 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87255.log.gz 2024-03-06T07:35:01.783 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81526.log: /var/log/ceph/ceph-client.admin.67899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87003.log 2024-03-06T07:35:01.783 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81526.log.gz 2024-03-06T07:35:01.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61507.log 2024-03-06T07:35:01.784 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67899.log.gz 2024-03-06T07:35:01.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87003.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55236.log.gz 2024-03-06T07:35:01.784 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86481.log 2024-03-06T07:35:01.784 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.87003.log.gz 2024-03-06T07:35:01.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84301.log 2024-03-06T07:35:01.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62055.log 2024-03-06T07:35:01.784 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61507.log.gz 2024-03-06T07:35:01.784 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84301.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86481.log.gz 2024-03-06T07:35:01.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46113.log 2024-03-06T07:35:01.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76915.log 2024-03-06T07:35:01.785 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62055.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84301.log.gz 2024-03-06T07:35:01.785 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46113.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62055.log.gz 2024-03-06T07:35:01.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75938.log 2024-03-06T07:35:01.785 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46113.log.gz 2024-03-06T07:35:01.785 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62181.log 2024-03-06T07:35:01.785 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76915.log: /var/log/ceph/ceph-client.admin.75938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52088.log 2024-03-06T07:35:01.785 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76915.log.gz 2024-03-06T07:35:01.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51549.log 2024-03-06T07:35:01.786 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52088.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62181.log.gz 2024-03-06T07:35:01.786 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:35:01.786 INFO:teuthology.orchestra.run.smithi059.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.75938.log.gz 2024-03-06T07:35:01.786 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52088.log.gz/var/log/ceph/ceph-client.admin.51549.log: 2024-03-06T07:35:01.786 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57826.log 2024-03-06T07:35:01.786 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51549.log.gz 2024-03-06T07:35:01.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.91935.log 2024-03-06T07:35:01.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48288.log 2024-03-06T07:35:01.787 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57826.log.gz 2024-03-06T07:35:01.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64665.log 2024-03-06T07:35:01.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89787.log 2024-03-06T07:35:01.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48288.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91935.log.gz 2024-03-06T07:35:01.787 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64665.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48288.log.gz 2024-03-06T07:35:01.787 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81399.log 2024-03-06T07:35:01.788 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64665.log.gz 2024-03-06T07:35:01.788 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89013.log 2024-03-06T07:35:01.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89787.log: /var/log/ceph/ceph-client.admin.81399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45409.log 2024-03-06T07:35:01.788 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.89787.log.gz 2024-03-06T07:35:01.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89013.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81399.log.gz 2024-03-06T07:35:01.788 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69769.log 2024-03-06T07:35:01.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45409.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89013.log.gz 2024-03-06T07:35:01.788 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77085.log 2024-03-06T07:35:01.788 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69769.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45409.log.gz 2024-03-06T07:35:01.789 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48102.log 2024-03-06T07:35:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77085.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69769.log.gz 2024-03-06T07:35:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69087.log 2024-03-06T07:35:01.791 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77085.log.gz 2024-03-06T07:35:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84177.log 2024-03-06T07:35:01.791 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69087.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48102.log.gz 2024-03-06T07:35:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49653.log 2024-03-06T07:35:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82996.log 2024-03-06T07:35:01.792 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69087.log.gz 2024-03-06T07:35:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84177.log: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:35:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49653.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84177.log.gz 2024-03-06T07:35:01.792 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83748.log 2024-03-06T07:35:01.792 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49653.log.gz 2024-03-06T07:35:01.793 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58166.log 2024-03-06T07:35:01.793 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82996.log: /var/log/ceph/ceph-client.admin.83748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48144.log 2024-03-06T07:35:01.793 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82996.log.gz 2024-03-06T07:35:01.793 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83748.log.gz 2024-03-06T07:35:01.793 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64707.log 2024-03-06T07:35:01.793 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.58166.log.gz 2024-03-06T07:35:01.793 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54098.log 2024-03-06T07:35:01.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48144.log: /var/log/ceph/ceph-client.admin.64707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77767.log 2024-03-06T07:35:01.794 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48144.log.gz 2024-03-06T07:35:01.794 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64707.log.gz 2024-03-06T07:35:01.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69371.log 2024-03-06T07:35:01.794 INFO:teuthology.orchestra.run.smithi059.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54098.log.gz 2024-03-06T07:35:01.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92787.log 2024-03-06T07:35:01.794 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78074.log 2024-03-06T07:35:01.794 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69371.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77767.log.gz 2024-03-06T07:35:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76603.log 2024-03-06T07:35:01.795 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69371.log.gz 2024-03-06T07:35:01.795 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.92787.log.gz 2024-03-06T07:35:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90267.log 2024-03-06T07:35:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78074.log: /var/log/ceph/ceph-client.admin.76603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53483.log 2024-03-06T07:35:01.795 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78074.log.gz 2024-03-06T07:35:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90267.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76603.log.gz 2024-03-06T07:35:01.795 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70035.log 2024-03-06T07:35:01.795 INFO:teuthology.orchestra.run.smithi059.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.90267.log.gz 2024-03-06T07:35:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75833.log 2024-03-06T07:35:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85713.log 2024-03-06T07:35:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70035.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53483.log.gz 2024-03-06T07:35:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53201.log 2024-03-06T07:35:01.796 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70035.log.gz 2024-03-06T07:35:01.796 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75833.log.gz 2024-03-06T07:35:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66290.log 2024-03-06T07:35:01.796 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60767.log 2024-03-06T07:35:01.796 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85713.log.gz 2024-03-06T07:35:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66290.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53201.log.gz 2024-03-06T07:35:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45073.log 2024-03-06T07:35:01.797 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66290.log.gz 2024-03-06T07:35:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67600.log 2024-03-06T07:35:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44433.log 2024-03-06T07:35:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45073.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60767.log.gz 2024-03-06T07:35:01.797 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67600.log: gzip -5 88.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45073.log.gz 2024-03-06T07:35:01.797 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.61248.log 2024-03-06T07:35:01.798 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67600.log.gz 2024-03-06T07:35:01.798 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51177.log 2024-03-06T07:35:01.798 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44433.log: /var/log/ceph/ceph-client.admin.61248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68085.log 2024-03-06T07:35:01.798 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44433.log.gz 2024-03-06T07:35:01.798 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61248.log.gz 2024-03-06T07:35:01.798 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67355.log 2024-03-06T07:35:01.798 INFO:teuthology.orchestra.run.smithi059.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51177.log.gz 2024-03-06T07:35:01.798 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49778.log 2024-03-06T07:35:01.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68085.log: /var/log/ceph/ceph-client.admin.67355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73957.log 2024-03-06T07:35:01.799 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68085.log.gz 2024-03-06T07:35:01.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49778.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67355.log.gz 2024-03-06T07:35:01.799 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92932.log 2024-03-06T07:35:01.799 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73957.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49778.log.gz 2024-03-06T07:35:01.799 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92019.log 2024-03-06T07:35:01.799 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73957.log.gz 2024-03-06T07:35:01.799 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49401.log 2024-03-06T07:35:01.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92932.log: /var/log/ceph/ceph-client.admin.92019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69462.log 2024-03-06T07:35:01.800 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92932.log.gz 2024-03-06T07:35:01.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49401.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92019.log.gz 2024-03-06T07:35:01.800 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73999.log 2024-03-06T07:35:01.800 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49401.log.gz 2024-03-06T07:35:01.800 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52256.log 2024-03-06T07:35:01.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69462.log: /var/log/ceph/ceph-client.admin.73999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48582.log 2024-03-06T07:35:01.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52256.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73999.log.gz 2024-03-06T07:35:01.800 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88971.log 2024-03-06T07:35:01.801 INFO:teuthology.orchestra.run.smithi059.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.69462.log.gz 2024-03-06T07:35:01.801 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52256.log.gz 2024-03-06T07:35:01.801 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90477.log 2024-03-06T07:35:01.801 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48582.log: /var/log/ceph/ceph-client.admin.88971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90855.log 2024-03-06T07:35:01.801 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48582.log.gz 2024-03-06T07:35:01.801 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90477.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88971.log.gz 2024-03-06T07:35:01.801 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49443.log 2024-03-06T07:35:01.801 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.90477.log.gzgzip 2024-03-06T07:35:01.802 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67773.log 2024-03-06T07:35:01.802 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90855.log: /var/log/ceph/ceph-client.admin.49443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88533.log 2024-03-06T07:35:01.802 INFO:teuthology.orchestra.run.smithi059.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90855.log.gz 2024-03-06T07:35:01.802 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49443.log.gz 2024-03-06T07:35:01.802 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66023.log 2024-03-06T07:35:01.802 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67773.log.gz 2024-03-06T07:35:01.802 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58082.log 2024-03-06T07:35:01.802 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88533.log: /var/log/ceph/ceph-client.admin.66023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47934.log 2024-03-06T07:35:01.803 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88533.log.gz 2024-03-06T07:35:01.803 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58082.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66023.log.gz 2024-03-06T07:35:01.803 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83622.log 2024-03-06T07:35:01.803 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58082.log.gz 2024-03-06T07:35:01.803 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72077.log 2024-03-06T07:35:01.803 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47934.log: /var/log/ceph/ceph-client.admin.83622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90981.log 2024-03-06T07:35:01.803 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47934.log.gz 2024-03-06T07:35:01.803 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63820.log 2024-03-06T07:35:01.804 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83622.log.gz 2024-03-06T07:35:01.804 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90981.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72077.log.gz 2024-03-06T07:35:01.804 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86355.log 2024-03-06T07:35:01.804 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90981.log.gz 2024-03-06T07:35:01.804 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.63820.log: -- /var/log/ceph/ceph-client.admin.66821.log 2024-03-06T07:35:01.804 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53362.log 2024-03-06T07:35:01.804 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63820.log.gz 2024-03-06T07:35:01.805 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66821.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86355.log.gz 2024-03-06T07:35:01.805 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50505.log 2024-03-06T07:35:01.805 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66821.log.gz 2024-03-06T07:35:01.805 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61290.log 2024-03-06T07:35:01.805 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81652.log 2024-03-06T07:35:01.805 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50505.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53362.log.gz 2024-03-06T07:35:01.805 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85629.log 2024-03-06T07:35:01.806 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50505.log.gz 2024-03-06T07:35:01.806 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61290.log.gz 2024-03-06T07:35:01.806 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60455.log 2024-03-06T07:35:01.806 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81652.log: /var/log/ceph/ceph-client.admin.85629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59446.log 2024-03-06T07:35:01.806 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81652.log.gz 2024-03-06T07:35:01.806 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60455.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85629.log.gz 2024-03-06T07:35:01.806 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81315.log 2024-03-06T07:35:01.806 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3%/var/log/ceph/ceph-client.admin.59446.log: -- replaced with /var/log/ceph/ceph-client.admin.60455.log.gz 2024-03-06T07:35:01.806 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45408.log 2024-03-06T07:35:01.807 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59446.log.gz 2024-03-06T07:35:01.807 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69420.log 2024-03-06T07:35:01.807 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81315.log: /var/log/ceph/ceph-client.admin.45408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54140.log 2024-03-06T07:35:01.807 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.81315.log.gz 2024-03-06T07:35:01.807 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69420.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45408.log.gz 2024-03-06T07:35:01.807 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50175.log 2024-03-06T07:35:01.807 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59404.log 88.5% 2024-03-06T07:35:01.808 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69420.log.gz 2024-03-06T07:35:01.808 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54140.log: /var/log/ceph/ceph-client.admin.50175.log: gzip -5 --verbose 88.8% -- -- replaced with /var/log/ceph/ceph-client.admin.54140.log.gz /var/log/ceph/ceph-client.admin.65120.log 2024-03-06T07:35:01.808 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-06T07:35:01.808 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50175.log.gz 2024-03-06T07:35:01.808 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43816.log 2024-03-06T07:35:01.808 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59404.log: /var/log/ceph/ceph-client.admin.65120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72203.log 2024-03-06T07:35:01.808 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59404.log.gz 2024-03-06T07:35:01.808 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65120.log.gz 2024-03-06T07:35:01.809 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.43816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45609.log 2024-03-06T07:35:01.809 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43816.log.gz 2024-03-06T07:35:01.809 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82954.log 2024-03-06T07:35:01.809 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67685.log 2024-03-06T07:35:01.809 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72203.log.gz 2024-03-06T07:35:01.809 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45609.log: /var/log/ceph/ceph-client.admin.82954.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45609.log.gz 2024-03-06T07:35:01.809 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89097.log 2024-03-06T07:35:01.809 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82954.log.gz 2024-03-06T07:35:01.809 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57136.log 2024-03-06T07:35:01.809 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67685.log: /var/log/ceph/ceph-client.admin.89097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72371.log 2024-03-06T07:35:01.810 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67685.log.gz 2024-03-06T07:35:01.810 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57136.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.89097.log.gz 2024-03-06T07:35:01.810 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66164.log 2024-03-06T07:35:01.810 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57136.log.gz 2024-03-06T07:35:01.810 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60219.log 2024-03-06T07:35:01.810 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90771.log 2024-03-06T07:35:01.810 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72371.log.gz 2024-03-06T07:35:01.810 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66164.log: /var/log/ceph/ceph-client.admin.60219.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66164.log.gz 2024-03-06T07:35:01.811 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63568.log 2024-03-06T07:35:01.811 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60219.log.gzgzip 2024-03-06T07:35:01.811 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.76198.log 2024-03-06T07:35:01.811 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90771.log: /var/log/ceph/ceph-client.admin.63568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44177.log 2024-03-06T07:35:01.811 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.90771.log.gz 2024-03-06T07:35:01.811 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63568.log.gz 2024-03-06T07:35:01.811 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58250.log 2024-03-06T07:35:01.812 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44177.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76198.log.gz 2024-03-06T07:35:01.812 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.65781.log 2024-03-06T07:35:01.812 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44177.log.gz 2024-03-06T07:35:01.812 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90813.log 2024-03-06T07:35:01.812 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58250.log: /var/log/ceph/ceph-client.0.65781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67643.log 2024-03-06T07:35:01.812 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58250.log.gz 2024-03-06T07:35:01.812 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80952.log 2024-03-06T07:35:01.813 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67643.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90813.log.gz 2024-03-06T07:35:01.813 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91608.log 2024-03-06T07:35:01.813 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53609.log 2024-03-06T07:35:01.813 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67643.log.gz 2024-03-06T07:35:01.813 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91608.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49155.log 2024-03-06T07:35:01.813 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80952.log.gz 2024-03-06T07:35:01.813 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91608.log.gz 2024-03-06T07:35:01.813 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91254.log 2024-03-06T07:35:01.814 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49155.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53609.log.gz 2024-03-06T07:35:01.814 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60996.log 2024-03-06T07:35:01.814 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49155.log.gz 2024-03-06T07:35:01.814 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77860.log 2024-03-06T07:35:01.814 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92745.log 2024-03-06T07:35:01.814 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60996.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91254.log.gz 2024-03-06T07:35:01.814 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79930.log 2024-03-06T07:35:01.814 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60996.log.gz 2024-03-06T07:35:01.814 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92745.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77860.log.gz 2024-03-06T07:35:01.815 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67729.log 2024-03-06T07:35:01.815 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79930.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.92745.log.gz 2024-03-06T07:35:01.815 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66206.log 2024-03-06T07:35:01.815 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78158.log 2024-03-06T07:35:01.815 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79930.log.gz 2024-03-06T07:35:01.815 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67729.log: /var/log/ceph/ceph-client.admin.66206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61588.log 2024-03-06T07:35:01.815 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67729.log.gz 2024-03-06T07:35:01.815 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45412.log 2024-03-06T07:35:01.816 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66206.log.gz 2024-03-06T07:35:01.816 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78158.log.gz 2024-03-06T07:35:01.816 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51093.log 2024-03-06T07:35:01.816 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57731.log 2024-03-06T07:35:01.816 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45412.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61588.log.gz 2024-03-06T07:35:01.816 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86523.log 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45412.log.gz 2024-03-06T07:35:01.816 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-06T07:35:01.816 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51093.log.gz 2024-03-06T07:35:01.816 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74153.log 2024-03-06T07:35:01.817 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86523.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57731.log.gz 2024-03-06T07:35:01.817 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45945.log 2024-03-06T07:35:01.817 INFO:teuthology.orchestra.run.smithi059.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.86523.log.gz 2024-03-06T07:35:01.817 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74153.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-06T07:35:01.817 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51907.log 2024-03-06T07:35:01.817 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74153.log.gz 2024-03-06T07:35:01.817 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80704.log 2024-03-06T07:35:01.817 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45945.log.gz 2024-03-06T07:35:01.817 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74649.log 2024-03-06T07:35:01.818 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80704.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51907.log.gz 2024-03-06T07:35:01.818 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72287.log 2024-03-06T07:35:01.818 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80704.log.gz 2024-03-06T07:35:01.818 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54285.log 2024-03-06T07:35:01.818 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72287.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74649.log.gz 2024-03-06T07:35:01.818 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80182.log 2024-03-06T07:35:01.818 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78242.log 2024-03-06T07:35:01.818 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72287.log.gz 2024-03-06T07:35:01.818 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51009.log 2024-03-06T07:35:01.819 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54285.log.gz 2024-03-06T07:35:01.819 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89661.log 2024-03-06T07:35:01.819 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80182.log.gz 2024-03-06T07:35:01.819 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78242.log: /var/log/ceph/ceph-client.admin.51009.log: gzip -5 --verbose -- 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78242.log.gz 2024-03-06T07:35:01.819 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.72035.log 2024-03-06T07:35:01.819 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51009.log.gz 2024-03-06T07:35:01.819 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54471.log 2024-03-06T07:35:01.819 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89661.log.gz 2024-03-06T07:35:01.820 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92324.log 2024-03-06T07:35:01.820 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54471.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72035.log.gz 2024-03-06T07:35:01.820 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77169.log 2024-03-06T07:35:01.820 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54471.log.gz 2024-03-06T07:35:01.820 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77349.log 2024-03-06T07:35:01.820 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92324.log.gz 2024-03-06T07:35:01.820 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75373.log 2024-03-06T07:35:01.820 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77349.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77169.log.gz 2024-03-06T07:35:01.821 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66527.log 2024-03-06T07:35:01.821 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77349.log.gz 2024-03-06T07:35:01.821 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84239.log 2024-03-06T07:35:01.821 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75373.log.gz 2024-03-06T07:35:01.821 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48540.log 2024-03-06T07:35:01.821 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84239.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66527.log.gz 2024-03-06T07:35:01.821 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.78753.log 2024-03-06T07:35:01.821 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84239.log.gz 2024-03-06T07:35:01.822 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54513.log 2024-03-06T07:35:01.822 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48540.log.gz 2024-03-06T07:35:01.822 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.78753.log: gzip -5 --verbose 95.5% -- replaced with /var/log/ceph/ceph-client.0.65781.log.gz 2024-03-06T07:35:01.822 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.83706.log 2024-03-06T07:35:01.822 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87903.log 2024-03-06T07:35:01.822 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54513.log.gz 2024-03-06T07:35:01.822 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.83706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72245.log 2024-03-06T07:35:01.822 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83706.log.gz/var/log/ceph/ceph-client.admin.87903.log: 2024-03-06T07:35:01.823 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54429.log 2024-03-06T07:35:01.823 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62632.log 2024-03-06T07:35:01.823 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87903.log.gz 2024-03-06T07:35:01.823 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66085.log 2024-03-06T07:35:01.823 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72245.log.gz 2024-03-06T07:35:01.823 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54429.log: /var/log/ceph/ceph-client.0.62632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70621.log 2024-03-06T07:35:01.823 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53846.log 2024-03-06T07:35:01.824 INFO:teuthology.orchestra.run.smithi059.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54429.log.gz 2024-03-06T07:35:01.824 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70621.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66085.log.gz 2024-03-06T07:35:01.824 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90687.log 2024-03-06T07:35:01.824 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70621.log.gz 2024-03-06T07:35:01.824 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70831.log 2024-03-06T07:35:01.824 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70258.log 2024-03-06T07:35:01.824 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53846.log.gz 2024-03-06T07:35:01.824 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90687.log: /var/log/ceph/ceph-client.admin.70831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66966.log 2024-03-06T07:35:01.825 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70258.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90687.log.gz 2024-03-06T07:35:01.825 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70831.log.gz 2024-03-06T07:35:01.825 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70258.log.gz 2024-03-06T07:35:01.825 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91506.log 2024-03-06T07:35:01.825 INFO:teuthology.orchestra.run.smithi059.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.0.78753.log.gz 2024-03-06T07:35:01.825 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53804.log 2024-03-06T07:35:01.825 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91506.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66966.log.gz 2024-03-06T07:35:01.825 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86733.log 2024-03-06T07:35:01.825 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81839.log 2024-03-06T07:35:01.826 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91506.log.gz 2024-03-06T07:35:01.826 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68127.log 2024-03-06T07:35:01.826 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53804.log.gz 2024-03-06T07:35:01.826 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86733.log: /var/log/ceph/ceph-client.admin.81839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63652.log 2024-03-06T07:35:01.826 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51717.log 2024-03-06T07:35:01.826 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81839.log.gz 2024-03-06T07:35:01.826 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86733.log.gz 2024-03-06T07:35:01.826 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63652.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60954.log 2024-03-06T07:35:01.827 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68127.log.gz 2024-03-06T07:35:01.827 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63652.log.gz 2024-03-06T07:35:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50253.log 2024-03-06T07:35:01.827 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51717.log.gz 2024-03-06T07:35:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50757.log 2024-03-06T07:35:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50253.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60954.log.gz 2024-03-06T07:35:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77723.log 2024-03-06T07:35:01.827 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50253.log.gz 2024-03-06T07:35:01.827 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60135.log 2024-03-06T07:35:01.828 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50757.log.gz 2024-03-06T07:35:01.828 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51507.log 2024-03-06T07:35:01.828 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77723.log.gz 2024-03-06T07:35:01.828 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61887.log 2024-03-06T07:35:01.828 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51507.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60135.log.gz 2024-03-06T07:35:01.828 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92196.log 2024-03-06T07:35:01.828 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51507.log.gz 2024-03-06T07:35:01.828 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87171.log 2024-03-06T07:35:01.829 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61887.log.gz 2024-03-06T07:35:01.829 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86607.log 2024-03-06T07:35:01.829 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87171.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92196.log.gz 2024-03-06T07:35:01.829 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89181.log 2024-03-06T07:35:01.829 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87171.log.gz 2024-03-06T07:35:01.829 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76958.log 2024-03-06T07:35:01.829 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86607.log.gz 2024-03-06T07:35:01.829 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59648.log 2024-03-06T07:35:01.829 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76958.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.89181.log.gz 2024-03-06T07:35:01.830 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75895.log 2024-03-06T07:35:01.830 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59648.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76958.log.gz 2024-03-06T07:35:01.830 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51135.log 2024-03-06T07:35:01.830 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59648.log.gz 2024-03-06T07:35:01.830 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53306.log 2024-03-06T07:35:01.830 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75895.log.gz 2024-03-06T07:35:01.830 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80014.log 2024-03-06T07:35:01.830 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51135.log.gz 2024-03-06T07:35:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76645.log 2024-03-06T07:35:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80014.log: gzip -5 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53306.log.gz 2024-03-06T07:35:01.831 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50091.log 2024-03-06T07:35:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76645.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80014.log.gz 2024-03-06T07:35:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73715.log 87.6% 2024-03-06T07:35:01.831 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76645.log.gz 2024-03-06T07:35:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90603.log 2024-03-06T07:35:01.831 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50091.log.gz 2024-03-06T07:35:01.831 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88281.log 2024-03-06T07:35:01.832 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90603.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73715.log.gz 2024-03-06T07:35:01.832 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81441.log 2024-03-06T07:35:01.832 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90603.log.gz 2024-03-06T07:35:01.832 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74587.log 2024-03-06T07:35:01.832 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88281.log.gz 2024-03-06T07:35:01.832 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67857.log 2024-03-06T07:35:01.832 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74587.log: gzip -5 --verbose 89.3% -- /var/log/ceph/ceph-client.admin.63526.log 2024-03-06T07:35:01.833 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81441.log.gz 2024-03-06T07:35:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67857.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74587.log.gz 2024-03-06T07:35:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88449.log 2024-03-06T07:35:01.833 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67857.log.gz 2024-03-06T07:35:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82475.log 2024-03-06T07:35:01.833 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63526.log.gz 2024-03-06T07:35:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73716.log 2024-03-06T07:35:01.833 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82475.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48060.log 2024-03-06T07:35:01.833 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88449.log.gz 2024-03-06T07:35:01.834 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82614.log 2024-03-06T07:35:01.834 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82475.log.gz 2024-03-06T07:35:01.834 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73716.log.gz 2024-03-06T07:35:01.834 INFO:teuthology.orchestra.run.smithi059.stderr: -5/var/log/ceph/ceph-client.admin.48060.log: --verbose -- /var/log/ceph/ceph-client.admin.57178.log 2024-03-06T07:35:01.834 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48060.log.gz 2024-03-06T07:35:01.834 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47740.log 2024-03-06T07:35:01.834 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52172.log 2024-03-06T07:35:01.834 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4%/var/log/ceph/ceph-client.admin.57178.log: -- replaced with /var/log/ceph/ceph-client.admin.82614.log.gz 2024-03-06T07:35:01.834 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60303.log 2024-03-06T07:35:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52942.log 89.3% 2024-03-06T07:35:01.835 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57178.log.gz 2024-03-06T07:35:01.835 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47740.log.gz 2024-03-06T07:35:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52172.log: /var/log/ceph/ceph-client.admin.60303.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52172.log.gz 2024-03-06T07:35:01.835 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60303.log.gz 2024-03-06T07:35:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79457.log 2024-03-06T07:35:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52942.log: 92.6% -- replaced with /var/log/ceph/ceph-client.0.62632.log.gz 2024-03-06T07:35:01.835 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56632.log 2024-03-06T07:35:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59586.log 2024-03-06T07:35:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79457.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52942.log.gz 2024-03-06T07:35:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89223.log 2024-03-06T07:35:01.836 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79457.log.gz 2024-03-06T07:35:01.836 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56632.log.gz 2024-03-06T07:35:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68043.log 2024-03-06T07:35:01.836 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85445.log 2024-03-06T07:35:01.836 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59586.log.gz 2024-03-06T07:35:01.837 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68043.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.89223.log.gz 2024-03-06T07:35:01.837 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88635.log 2024-03-06T07:35:01.837 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71102.log 2024-03-06T07:35:01.837 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85445.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68043.log.gz 2024-03-06T07:35:01.837 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77595.log 2024-03-06T07:35:01.837 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.85445.log.gz 2024-03-06T07:35:01.837 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8%/var/log/ceph/ceph-client.admin.71102.log: -- replaced with /var/log/ceph/ceph-client.admin.88635.log.gz 2024-03-06T07:35:01.837 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84749.log 2024-03-06T07:35:01.837 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71102.log.gz 2024-03-06T07:35:01.838 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68379.log 2024-03-06T07:35:01.838 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52000.log 2024-03-06T07:35:01.838 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84749.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77595.log.gz 2024-03-06T07:35:01.838 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64156.log 2024-03-06T07:35:01.838 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84749.log.gz 2024-03-06T07:35:01.838 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68379.log.gz 2024-03-06T07:35:01.838 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.52000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72119.log 2024-03-06T07:35:01.838 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46281.log 2024-03-06T07:35:01.839 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64156.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52000.log.gz 2024-03-06T07:35:01.839 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76240.log 2024-03-06T07:35:01.839 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64156.log.gz 2024-03-06T07:35:01.839 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72119.log.gz 2024-03-06T07:35:01.839 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68487.log 2024-03-06T07:35:01.839 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76240.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46281.log.gz 2024-03-06T07:35:01.839 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54812.log 2024-03-06T07:35:01.839 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7%/var/log/ceph/ceph-client.admin.68487.log: -- replaced with /var/log/ceph/ceph-client.admin.76240.log.gz 2024-03-06T07:35:01.839 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53972.log 2024-03-06T07:35:01.840 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68487.log.gz 2024-03-06T07:35:01.840 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54555.log 2024-03-06T07:35:01.840 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53567.log 2024-03-06T07:35:01.840 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53972.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54812.log.gz 2024-03-06T07:35:01.840 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86919.log 2024-03-06T07:35:01.840 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53972.log.gz 2024-03-06T07:35:01.841 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53567.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.54555.log.gz 2024-03-06T07:35:01.841 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.89055.log 2024-03-06T07:35:01.841 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53567.log.gz 2024-03-06T07:35:01.841 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77511.log 2024-03-06T07:35:01.841 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45068.log 2024-03-06T07:35:01.841 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.86919.log.gz 2024-03-06T07:35:01.842 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56125.log 2024-03-06T07:35:01.842 INFO:teuthology.orchestra.run.smithi059.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89055.log.gz 2024-03-06T07:35:01.842 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76729.log 2024-03-06T07:35:01.842 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77511.log.gz 2024-03-06T07:35:01.842 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56125.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.45068.log.gz --verbose 2024-03-06T07:35:01.842 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.71863.log 2024-03-06T07:35:01.842 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88197.log 2024-03-06T07:35:01.842 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76729.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56125.log.gz 2024-03-06T07:35:01.843 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64795.log 2024-03-06T07:35:01.843 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76729.log.gz 2024-03-06T07:35:01.843 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71863.log.gz 2024-03-06T07:35:01.843 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5/var/log/ceph/ceph-client.admin.88197.log: --verbose -- /var/log/ceph/ceph-client.admin.50379.log 2024-03-06T07:35:01.843 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57094.log 2024-03-06T07:35:01.843 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64795.log.gz 2024-03-06T07:35:01.843 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88197.log.gz 2024-03-06T07:35:01.843 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53015.log 2024-03-06T07:35:01.844 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81120.log 2024-03-06T07:35:01.844 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50379.log.gz 2024-03-06T07:35:01.844 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85791.log 2024-03-06T07:35:01.844 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57094.log.gz 2024-03-06T07:35:01.844 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92238.log 2024-03-06T07:35:01.844 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53015.log.gz 2024-03-06T07:35:01.844 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88029.log 2024-03-06T07:35:01.844 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85791.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81120.log.gz 2024-03-06T07:35:01.845 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89577.log 2024-03-06T07:35:01.845 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.85791.log.gz 2024-03-06T07:35:01.845 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.92238.log.gz 2024-03-06T07:35:01.845 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86103.log 2024-03-06T07:35:01.845 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58394.log 2024-03-06T07:35:01.845 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88029.log.gz 2024-03-06T07:35:01.845 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89577.log: /var/log/ceph/ceph-client.admin.86103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63289.log 2024-03-06T07:35:01.845 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89577.log.gz 2024-03-06T07:35:01.846 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86103.log.gz 2024-03-06T07:35:01.846 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63051.log 2024-03-06T07:35:01.846 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49822.log 2024-03-06T07:35:01.846 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63289.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58394.log.gz 2024-03-06T07:35:01.846 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52340.log 2024-03-06T07:35:01.846 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63289.log.gz 2024-03-06T07:35:01.846 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84379.log 2024-03-06T07:35:01.846 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63051.log.gz 2024-03-06T07:35:01.846 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7%/var/log/ceph/ceph-client.admin.52340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54243.log 2024-03-06T07:35:01.847 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49822.log.gz 2024-03-06T07:35:01.847 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68337.log 2024-03-06T07:35:01.847 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52340.log.gz 2024-03-06T07:35:01.847 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54243.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84379.log.gz 2024-03-06T07:35:01.847 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.91693.log 2024-03-06T07:35:01.847 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54243.log.gz 2024-03-06T07:35:01.847 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92282.log 2024-03-06T07:35:01.847 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68337.log.gz 2024-03-06T07:35:01.848 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59362.log 2024-03-06T07:35:01.848 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91693.log.gz 2024-03-06T07:35:01.848 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59136.log 2024-03-06T07:35:01.848 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70426.log 2024-03-06T07:35:01.848 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59362.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.92282.log.gz 2024-03-06T07:35:01.848 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83104.log 2024-03-06T07:35:01.848 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59362.log.gz 2024-03-06T07:35:01.848 INFO:teuthology.orchestra.run.smithi059.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59136.log.gz 2024-03-06T07:35:01.849 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70426.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-06T07:35:01.849 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.83104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77990.log 2024-03-06T07:35:01.849 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70426.log.gz 2024-03-06T07:35:01.849 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83104.log.gz 2024-03-06T07:35:01.849 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75709.log 2024-03-06T07:35:01.849 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62373.log 2024-03-06T07:35:01.849 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77990.log: gzip/var/log/ceph/ceph-client.admin.75709.log: -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77990.log.gz 2024-03-06T07:35:01.849 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45076.log 2024-03-06T07:35:01.850 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62373.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75709.log.gz 2024-03-06T07:35:01.850 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62373.log.gz 2024-03-06T07:35:01.850 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.89409.log 2024-03-06T07:35:01.850 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45067.log 2024-03-06T07:35:01.850 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45076.log.gz 2024-03-06T07:35:01.850 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66332.log 2024-03-06T07:35:01.850 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45067.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89409.log.gz 2024-03-06T07:35:01.850 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50631.log 2024-03-06T07:35:01.851 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45067.log.gz 2024-03-06T07:35:01.851 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67516.log 2024-03-06T07:35:01.851 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66332.log.gz 2024-03-06T07:35:01.851 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86649.log 2024-03-06T07:35:01.851 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67516.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50631.log.gz 2024-03-06T07:35:01.852 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79888.log 2024-03-06T07:35:01.852 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67516.log.gz 2024-03-06T07:35:01.852 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90309.log 2024-03-06T07:35:01.852 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86649.log.gz 2024-03-06T07:35:01.852 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38159.log 2024-03-06T07:35:01.852 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90309.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79888.log.gz 2024-03-06T07:35:01.852 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73055.log 2024-03-06T07:35:01.852 INFO:teuthology.orchestra.run.smithi059.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.90309.log.gz 2024-03-06T07:35:01.853 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90897.log 2024-03-06T07:35:01.853 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73055.log: gzip -5 --verbose 92.4% -- replaced with /var/log/ceph/ceph-client.admin.38159.log.gz 2024-03-06T07:35:01.853 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.44643.log 2024-03-06T07:35:01.853 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90897.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73055.log.gz 2024-03-06T07:35:01.853 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91296.log 2024-03-06T07:35:01.853 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90897.log.gz 2024-03-06T07:35:01.853 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70159.log 2024-03-06T07:35:01.853 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44643.log.gz 2024-03-06T07:35:01.853 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68421.log 2024-03-06T07:35:01.854 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70159.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91296.log.gz 2024-03-06T07:35:01.854 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45410.log 2024-03-06T07:35:01.854 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70159.log.gz 2024-03-06T07:35:01.854 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92146.log 2024-03-06T07:35:01.854 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63904.log 2024-03-06T07:35:01.854 INFO:teuthology.orchestra.run.smithi059.stderr: 90.9%/var/log/ceph/ceph-client.admin.92146.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68421.log.gz 2024-03-06T07:35:01.854 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45410.log.gz 2024-03-06T07:35:01.854 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87819.log 2024-03-06T07:35:01.854 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92146.log.gz 2024-03-06T07:35:01.855 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81484.log 2024-03-06T07:35:01.855 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63904.log: /var/log/ceph/ceph-client.admin.87819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48186.log 2024-03-06T07:35:01.855 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86859.log 2024-03-06T07:35:01.855 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63904.log.gz 2024-03-06T07:35:01.855 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.87819.log.gz 2024-03-06T07:35:01.855 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.admin.50841.log 2024-03-06T07:35:01.855 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81484.log.gz 2024-03-06T07:35:01.856 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48186.log: gzip/var/log/ceph/ceph-client.admin.86859.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.91170.log 2024-03-06T07:35:01.856 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48186.log.gz/var/log/ceph/ceph-client.admin.50841.log: 2024-03-06T07:35:01.856 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86859.log.gz 2024-03-06T07:35:01.856 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88845.log 2024-03-06T07:35:01.856 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50841.log.gz 2024-03-06T07:35:01.856 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63009.log 2024-03-06T07:35:01.856 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91170.log.gz 2024-03-06T07:35:01.856 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62925.log 2024-03-06T07:35:01.857 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88845.log.gz 2024-03-06T07:35:01.857 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76156.log 2024-03-06T07:35:01.857 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77429.log 2024-03-06T07:35:01.857 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63009.log.gz 2024-03-06T07:35:01.857 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49239.log 2024-03-06T07:35:01.857 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62925.log.gz 2024-03-06T07:35:01.857 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77000.log 2024-03-06T07:35:01.857 INFO:teuthology.orchestra.run.smithi059.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76156.log.gz 2024-03-06T07:35:01.857 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51422.log 2024-03-06T07:35:01.858 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77429.log.gz 2024-03-06T07:35:01.858 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61757.log 2024-03-06T07:35:01.858 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49239.log.gz 2024-03-06T07:35:01.858 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77000.log: /var/log/ceph/ceph-client.admin.51422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76687.log 2024-03-06T07:35:01.858 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61757.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77000.log.gz 2024-03-06T07:35:01.858 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81610.log 2024-03-06T07:35:01.858 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51422.log.gz 2024-03-06T07:35:01.858 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61757.log.gz 2024-03-06T07:35:01.859 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84115.log 2024-03-06T07:35:01.859 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49907.log 2024-03-06T07:35:01.859 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76687.log.gz 2024-03-06T07:35:01.859 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 87.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.81610.log.gz 2024-03-06T07:35:01.859 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.61427.log 2024-03-06T07:35:01.859 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54200.log 2024-03-06T07:35:01.859 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49907.log: /var/log/ceph/ceph-client.admin.61427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78116.log 2024-03-06T07:35:01.859 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49907.log.gz 2024-03-06T07:35:01.859 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84115.log.gz 2024-03-06T07:35:01.860 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64580.log 2024-03-06T07:35:01.860 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61427.log.gz 2024-03-06T07:35:01.860 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54200.log.gz 2024-03-06T07:35:01.860 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56012.log 2024-03-06T07:35:01.860 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78116.log: gzip/var/log/ceph/ceph-client.admin.64580.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51219.log 2024-03-06T07:35:01.860 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56012.log: 88.9% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64580.log.gz 2024-03-06T07:35:01.860 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78116.log.gz 2024-03-06T07:35:01.861 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54056.log 2024-03-06T07:35:01.861 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56012.log.gz 2024-03-06T07:35:01.861 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87297.log 2024-03-06T07:35:01.861 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54056.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51219.log.gz 2024-03-06T07:35:01.861 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56926.log 2024-03-06T07:35:01.861 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87297.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54056.log.gz 2024-03-06T07:35:01.861 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86817.log 2024-03-06T07:35:01.861 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87297.log.gz 2024-03-06T07:35:01.862 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66779.log 2024-03-06T07:35:01.862 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56926.log.gz 2024-03-06T07:35:01.862 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88323.log 2024-03-06T07:35:01.862 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66779.log: gzip -5 --verbose -- 89.2% -- replaced with /var/log/ceph/ceph-client.admin.86817.log.gz 2024-03-06T07:35:01.862 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.49611.log 2024-03-06T07:35:01.862 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88323.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66779.log.gz 2024-03-06T07:35:01.862 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70342.log 2024-03-06T07:35:01.862 INFO:teuthology.orchestra.run.smithi059.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.88323.log.gz 2024-03-06T07:35:01.863 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.49611.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.64888.log 2024-03-06T07:35:01.863 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70342.log: gzip -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49611.log.gz 2024-03-06T07:35:01.863 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64030.log 2024-03-06T07:35:01.863 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64888.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70342.log.gz 2024-03-06T07:35:01.863 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45746.log 2024-03-06T07:35:01.863 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64888.log.gz 2024-03-06T07:35:01.863 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72329.log 2024-03-06T07:35:01.863 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45746.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64030.log.gz 2024-03-06T07:35:01.863 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60261.log 2024-03-06T07:35:01.864 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45746.log.gz 2024-03-06T07:35:01.864 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43993.log 2024-03-06T07:35:01.864 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72329.log.gz 2024-03-06T07:35:01.864 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88491.log 2024-03-06T07:35:01.864 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.43993.log: gzip -5 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60261.log.gz 2024-03-06T07:35:01.864 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.92890.log 2024-03-06T07:35:01.864 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88491.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43993.log.gz 2024-03-06T07:35:01.864 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47850.log 2024-03-06T07:35:01.865 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92890.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88491.log.gz 2024-03-06T07:35:01.865 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51863.log 2024-03-06T07:35:01.865 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47850.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.92890.log.gz 2024-03-06T07:35:01.865 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65288.log 2024-03-06T07:35:01.865 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9%/var/log/ceph/ceph-client.admin.51863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57561.log 2024-03-06T07:35:01.865 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47850.log.gz 2024-03-06T07:35:01.865 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56548.log 2024-03-06T07:35:01.865 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51863.log.gz 2024-03-06T07:35:01.865 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65288.log.gz 2024-03-06T07:35:01.866 INFO:teuthology.orchestra.run.smithi059.stderr: -5/var/log/ceph/ceph-client.admin.57561.log: --verbose -- /var/log/ceph/ceph-client.admin.54014.log 2024-03-06T07:35:01.866 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56548.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83994.log 2024-03-06T07:35:01.866 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57561.log.gz 2024-03-06T07:35:01.866 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56548.log.gz 2024-03-06T07:35:01.866 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46071.log 2024-03-06T07:35:01.866 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.83994.log: gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54014.log.gz 2024-03-06T07:35:01.866 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.90393.log 2024-03-06T07:35:01.866 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83994.log.gz/var/log/ceph/ceph-client.admin.46071.log: 2024-03-06T07:35:01.866 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82433.log 2024-03-06T07:35:01.867 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46071.log.gz 2024-03-06T07:35:01.867 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74045.log 2024-03-06T07:35:01.867 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90393.log.gz 2024-03-06T07:35:01.867 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69583.log 2024-03-06T07:35:01.867 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82433.log.gz 2024-03-06T07:35:01.867 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74958.log 2024-03-06T07:35:01.867 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74045.log: /var/log/ceph/ceph-client.admin.69583.log: gzip -5 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74045.log.gz 2024-03-06T07:35:01.867 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46407.log 2024-03-06T07:35:01.867 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69583.log.gz 2024-03-06T07:35:01.868 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75629.log 2024-03-06T07:35:01.868 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74958.log.gz 2024-03-06T07:35:01.868 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66443.log 2024-03-06T07:35:01.868 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46407.log.gz 2024-03-06T07:35:01.868 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60413.log 2024-03-06T07:35:01.868 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.66443.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49949.log 2024-03-06T07:35:01.868 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75629.log.gz 2024-03-06T07:35:01.868 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49197.log 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66443.log.gz 2024-03-06T07:35:01.868 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-06T07:35:01.869 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60413.log.gz 2024-03-06T07:35:01.869 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53930.log 2024-03-06T07:35:01.869 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49949.log.gz 2024-03-06T07:35:01.869 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88761.log 2024-03-06T07:35:01.869 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49197.log.gz 2024-03-06T07:35:01.869 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72479.log 2024-03-06T07:35:01.869 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53930.log.gz 2024-03-06T07:35:01.870 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71396.log 2024-03-06T07:35:01.870 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88761.log.gz 2024-03-06T07:35:01.870 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.72479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58918.log 2024-03-06T07:35:01.870 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83826.log 2024-03-06T07:35:01.870 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72479.log.gz/var/log/ceph/ceph-client.admin.71396.log: 2024-03-06T07:35:01.870 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57868.log 2024-03-06T07:35:01.870 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.71396.log.gz 2024-03-06T07:35:01.870 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83038.log 2024-03-06T07:35:01.871 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58918.log.gz 2024-03-06T07:35:01.871 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.83826.log: /var/log/ceph/ceph-client.admin.57868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89829.log 2024-03-06T07:35:01.871 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0%/var/log/ceph/ceph-client.admin.83038.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57868.log.gz 2024-03-06T07:35:01.871 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83826.log.gz 2024-03-06T07:35:01.871 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68253.log 2024-03-06T07:35:01.871 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89829.log: gzip -5 --verbose 91.2% -- replaced with /var/log/ceph/ceph-client.admin.83038.log.gz 2024-03-06T07:35:01.871 INFO:teuthology.orchestra.run.smithi059.stderr: -- 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89829.log.gz 2024-03-06T07:35:01.871 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.61672.log 2024-03-06T07:35:01.872 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71060.log 2024-03-06T07:35:01.872 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68253.log.gz 2024-03-06T07:35:01.872 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60497.log 2024-03-06T07:35:01.872 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61672.log.gz 2024-03-06T07:35:01.872 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.71060.log: -- /var/log/ceph/ceph-client.admin.50295.log 2024-03-06T07:35:01.872 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60497.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71060.log.gz 2024-03-06T07:35:01.872 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45077.log 2024-03-06T07:35:01.872 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60497.log.gz 2024-03-06T07:35:01.873 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50049.log 2024-03-06T07:35:01.873 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50295.log: /var/log/ceph/ceph-client.admin.45077.log: gzip -5 --verbose 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50295.log.gz 2024-03-06T07:35:01.873 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.54854.log 2024-03-06T07:35:01.873 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45077.log.gz 2024-03-06T07:35:01.873 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50049.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-06T07:35:01.873 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50049.log.gz 2024-03-06T07:35:01.873 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84565.log 2024-03-06T07:35:01.873 INFO:teuthology.orchestra.run.smithi059.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54854.log.gz 2024-03-06T07:35:01.873 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51591.log 2024-03-06T07:35:01.874 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5/var/log/ceph/ceph-client.admin.84565.log: --verbose -- /var/log/ceph/ceph-client.admin.86145.log 2024-03-06T07:35:01.874 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51591.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84565.log.gz 2024-03-06T07:35:01.874 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92535.log 2024-03-06T07:35:01.874 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51591.log.gz 2024-03-06T07:35:01.874 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44475.log 2024-03-06T07:35:01.874 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86145.log: /var/log/ceph/ceph-client.admin.92535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78032.log 2024-03-06T07:35:01.874 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44475.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86145.log.gz 2024-03-06T07:35:01.875 INFO:teuthology.orchestra.run.smithi059.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.92535.log.gz 2024-03-06T07:35:01.875 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2024-03-06T07:35:01.875 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80476.log 2024-03-06T07:35:01.875 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.78032.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.72413.log 2024-03-06T07:35:01.875 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80476.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78032.log.gzgzip 2024-03-06T07:35:01.875 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38409.log 2024-03-06T07:35:01.875 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80476.log.gz 2024-03-06T07:35:01.875 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.72413.log: -- /var/log/ceph/ceph-client.admin.56674.log 2024-03-06T07:35:01.875 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.38409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83520.log 2024-03-06T07:35:01.876 INFO:teuthology.orchestra.run.smithi059.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38409.log.gz 2024-03-06T07:35:01.876 INFO:teuthology.orchestra.run.smithi059.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.72413.log.gz/var/log/ceph/ceph-client.admin.56674.log: 2024-03-06T07:35:01.876 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85833.log 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56674.log.gz 2024-03-06T07:35:01.876 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-06T07:35:01.876 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.83520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64377.log 2024-03-06T07:35:01.876 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83520.log.gz 2024-03-06T07:35:01.876 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50547.log 2024-03-06T07:35:01.876 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85833.log.gz 2024-03-06T07:35:01.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44261.log 2024-03-06T07:35:01.877 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64377.log.gz 2024-03-06T07:35:01.877 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65162.log 2024-03-06T07:35:01.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45743.log 2024-03-06T07:35:01.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44261.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50547.log.gz 2024-03-06T07:35:01.877 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65162.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44261.log.gz 2024-03-06T07:35:01.877 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65162.log.gz 2024-03-06T07:35:01.877 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86043.log 2024-03-06T07:35:01.878 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45747.log 2024-03-06T07:35:01.878 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86043.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45743.log.gz 2024-03-06T07:35:01.878 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81755.log 2024-03-06T07:35:01.878 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45747.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.86043.log.gz 2024-03-06T07:35:01.878 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64538.log 2024-03-06T07:35:01.878 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6%/var/log/ceph/ceph-client.admin.81755.log: -- replaced with /var/log/ceph/ceph-client.admin.45747.log.gz 2024-03-06T07:35:01.878 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64846.log 2024-03-06T07:35:01.878 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81755.log.gz 2024-03-06T07:35:01.879 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48414.log 2024-03-06T07:35:01.879 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64538.log: /var/log/ceph/ceph-client.admin.64846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71951.log 2024-03-06T07:35:01.879 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64538.log.gz 2024-03-06T07:35:01.879 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48414.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64846.log.gz 2024-03-06T07:35:01.879 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55362.log 2024-03-06T07:35:01.879 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48414.log.gz 2024-03-06T07:35:01.879 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45071.log 2024-03-06T07:35:01.879 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46491.log 2024-03-06T07:35:01.879 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71951.log.gz 2024-03-06T07:35:01.879 INFO:teuthology.orchestra.run.smithi059.stderr: 94.5%/var/log/ceph/ceph-client.admin.45071.log: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-06T07:35:01.880 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55362.log.gz 2024-03-06T07:35:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52382.log 2024-03-06T07:35:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46491.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45071.log.gz 2024-03-06T07:35:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57998.log 2024-03-06T07:35:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46491.log.gz 2024-03-06T07:35:01.880 INFO:teuthology.orchestra.run.smithi059.stderr: --verbose --/var/log/ceph/ceph-client.admin.52382.log: /var/log/ceph/ceph-client.admin.64114.log 2024-03-06T07:35:01.880 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57998.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57998.log.gz 2024-03-06T07:35:01.881 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55110.log 2024-03-06T07:35:01.881 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64114.log: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84791.log 2024-03-06T07:35:01.881 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52382.log.gz 2024-03-06T07:35:01.881 INFO:teuthology.orchestra.run.smithi059.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64114.log.gz 2024-03-06T07:35:01.881 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68959.log 2024-03-06T07:35:01.881 INFO:teuthology.orchestra.run.smithi059.stderr:gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55110.log.gz 2024-03-06T07:35:01.881 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose/var/log/ceph/ceph-client.admin.84791.log: -- /var/log/ceph/ceph-client.admin.77948.log 2024-03-06T07:35:01.881 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84791.log.gz 2024-03-06T07:35:01.882 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77127.log 2024-03-06T07:35:01.882 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68959.log: /var/log/ceph/ceph-client.admin.77948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66611.log 2024-03-06T07:35:01.882 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68959.log.gz 2024-03-06T07:35:01.882 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45075.log 2024-03-06T07:35:01.882 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77948.log.gz 2024-03-06T07:35:01.882 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44727.log 2024-03-06T07:35:01.882 INFO:teuthology.orchestra.run.smithi059.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.77127.log.gz 2024-03-06T07:35:01.882 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65456.log 2024-03-06T07:35:01.883 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66611.log.gz 2024-03-06T07:35:01.883 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45075.log: /var/log/ceph/ceph-client.admin.44727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44601.log 2024-03-06T07:35:01.883 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79271.log 2024-03-06T07:35:01.883 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44727.log.gz 2024-03-06T07:35:01.883 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45075.log.gz 2024-03-06T07:35:01.883 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75435.log 2024-03-06T07:35:01.883 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44601.log.gz/var/log/ceph/ceph-client.admin.79271.log: 2024-03-06T07:35:01.883 INFO:teuthology.orchestra.run.smithi059.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.65456.log.gz 2024-03-06T07:35:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44895.log 2024-03-06T07:35:01.884 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79271.log.gz 2024-03-06T07:35:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77253.log 2024-03-06T07:35:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75435.log: /var/log/ceph/ceph-client.admin.44895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70957.log 2024-03-06T07:35:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77253.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44895.log.gz 2024-03-06T07:35:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54650.log 2024-03-06T07:35:01.884 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75435.log.gz 2024-03-06T07:35:01.884 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70957.log: gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77253.log.gz 2024-03-06T07:35:01.885 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.86313.log 2024-03-06T07:35:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.54650.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70957.log.gz 2024-03-06T07:35:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66569.log 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54650.log.gz 2024-03-06T07:35:01.885 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-06T07:35:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48228.log 2024-03-06T07:35:01.885 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86313.log.gz 2024-03-06T07:35:01.885 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76813.log 2024-03-06T07:35:01.885 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66569.log.gz 2024-03-06T07:35:01.886 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66863.log 2024-03-06T07:35:01.886 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.76813.log: -5 --verbose 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48228.log.gz 2024-03-06T07:35:01.886 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.91735.log 2024-03-06T07:35:01.886 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66863.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76813.log.gz 2024-03-06T07:35:01.886 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 88.2% --verbose -- /var/log/ceph/ceph-client.admin.80434.log 2024-03-06T07:35:01.886 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66863.log.gz 2024-03-06T07:35:01.886 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62265.log 2024-03-06T07:35:01.886 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6%/var/log/ceph/ceph-client.admin.80434.log: -- replaced with /var/log/ceph/ceph-client.admin.91735.log.gz 2024-03-06T07:35:01.886 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49485.log 2024-03-06T07:35:01.887 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80434.log.gz 2024-03-06T07:35:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86271.log 2024-03-06T07:35:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49485.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62265.log.gz 2024-03-06T07:35:01.887 INFO:teuthology.orchestra.run.smithi059.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.91464.log 2024-03-06T07:35:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86271.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49485.log.gz 2024-03-06T07:35:01.887 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49527.log 2024-03-06T07:35:01.888 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86271.log.gz 2024-03-06T07:35:01.888 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76324.log 2024-03-06T07:35:01.888 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91464.log: /var/log/ceph/ceph-client.admin.49527.log: gzip -5 --verbose 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91464.log.gz 2024-03-06T07:35:01.888 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.70537.log 2024-03-06T07:35:01.888 INFO:teuthology.orchestra.run.smithi059.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49527.log.gz 2024-03-06T07:35:01.888 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44769.log 2024-03-06T07:35:01.888 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76324.log.gz 2024-03-06T07:35:01.888 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61122.log 2024-03-06T07:35:01.889 INFO:teuthology.orchestra.run.smithi059.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70537.log.gz 2024-03-06T07:35:01.889 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64240.log 2024-03-06T07:35:01.889 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.61122.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48456.log 2024-03-06T07:35:01.889 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44769.log.gz 2024-03-06T07:35:01.889 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64240.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-06T07:35:01.889 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61122.log.gz 2024-03-06T07:35:01.889 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose 88.5%/var/log/ceph/ceph-client.admin.48456.log: -- replaced with /var/log/ceph/ceph-client.admin.64240.log.gz 2024-03-06T07:35:01.889 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.73717.log 2024-03-06T07:35:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.0.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48456.log.gz 2024-03-06T07:35:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71312.log 2024-03-06T07:35:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87129.log 2024-03-06T07:35:01.890 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73717.log.gz 2024-03-06T07:35:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62307.log 2024-03-06T07:35:01.890 INFO:teuthology.orchestra.run.smithi059.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71312.log.gz 2024-03-06T07:35:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85917.log 2024-03-06T07:35:01.890 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79090.log 2024-03-06T07:35:01.891 INFO:teuthology.orchestra.run.smithi059.stderr: 87.2%/var/log/ceph/ceph-client.admin.62307.log: -- replaced with /var/log/ceph/ceph-client.admin.87129.log.gz 2024-03-06T07:35:01.891 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87507.log 2024-03-06T07:35:01.891 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85917.log.gz 2024-03-06T07:35:01.891 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79090.log: 90.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81568.log 2024-03-06T07:35:01.891 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79090.log.gz 2024-03-06T07:35:01.891 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87507.log: -- replaced with /var/log/ceph/ceph-client.admin.62307.log.gz 2024-03-06T07:35:01.891 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- 89.4% -- replaced with /var/log/ceph/ceph-client.admin.87507.log.gz 2024-03-06T07:35:01.891 INFO:teuthology.orchestra.run.smithi059.stderr: /var/log/ceph/ceph-client.admin.78386.log 2024-03-06T07:35:01.892 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70915.log 2024-03-06T07:35:01.892 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81568.log.gz 2024-03-06T07:35:01.892 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71737.log 2024-03-06T07:35:01.892 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78386.log.gz 2024-03-06T07:35:01.892 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74770.log 2024-03-06T07:35:01.892 INFO:teuthology.orchestra.run.smithi059.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70915.log.gz 2024-03-06T07:35:01.892 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90081.log 2024-03-06T07:35:01.892 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71737.log: /var/log/ceph/ceph-client.admin.74770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69088.log 2024-03-06T07:35:01.893 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90081.log: 88.3% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74770.log.gz 2024-03-06T07:35:01.893 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71737.log.gz 2024-03-06T07:35:01.893 INFO:teuthology.orchestra.run.smithi059.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90081.log.gz 2024-03-06T07:35:01.893 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53077.log 2024-03-06T07:35:01.893 INFO:teuthology.orchestra.run.smithi059.stderr:gzip/var/log/ceph/ceph-client.admin.69088.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63946.log 2024-03-06T07:35:01.893 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.53077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56506.log 2024-03-06T07:35:01.893 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69088.log.gz 2024-03-06T07:35:01.894 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63946.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53077.log.gz 2024-03-06T07:35:01.894 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44389.log 2024-03-06T07:35:01.894 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56506.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63946.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57052.log 2024-03-06T07:35:01.894 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-06T07:35:01.894 INFO:teuthology.orchestra.run.smithi059.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56506.log.gz 2024-03-06T07:35:01.894 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81078.log 2024-03-06T07:35:01.894 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44389.log: /var/log/ceph/ceph-client.admin.57052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60539.log 2024-03-06T07:35:01.894 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81078.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57052.log.gz 2024-03-06T07:35:01.895 INFO:teuthology.orchestra.run.smithi059.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44389.log.gz 2024-03-06T07:35:01.895 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81078.log.gz 2024-03-06T07:35:01.895 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.89535.log 2024-03-06T07:35:01.895 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60539.log: gzip -5 --verbose 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60539.log.gz 2024-03-06T07:35:01.895 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.51380.log 2024-03-06T07:35:01.895 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82744.log 2024-03-06T07:35:01.895 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89535.log.gz 2024-03-06T07:35:01.895 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76282.log 2024-03-06T07:35:01.896 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51380.log.gz/var/log/ceph/ceph-client.admin.82744.log: 2024-03-06T07:35:01.896 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91422.log 2024-03-06T07:35:01.896 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82744.log.gz 2024-03-06T07:35:01.896 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51801.log 2024-03-06T07:35:01.896 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76282.log: /var/log/ceph/ceph-client.admin.91422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45745.log 2024-03-06T07:35:01.896 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51801.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76282.log.gz 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91422.log.gz 2024-03-06T07:35:01.896 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-06T07:35:01.896 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51801.log.gz 2024-03-06T07:35:01.897 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89265.log 2024-03-06T07:35:01.897 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45745.log: gzip -5 --verbose 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45745.log.gz 2024-03-06T07:35:01.897 INFO:teuthology.orchestra.run.smithi059.stderr: -- /var/log/ceph/ceph-client.admin.69707.log 2024-03-06T07:35:01.897 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77902.log 2024-03-06T07:35:01.897 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.89265.log.gz 2024-03-06T07:35:01.898 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64622.log 2024-03-06T07:35:01.898 INFO:teuthology.orchestra.run.smithi059.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69707.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51051.log 2024-03-06T07:35:01.898 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-06T07:35:01.898 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77902.log: /var/log/ceph/ceph-client.admin.64622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55855.log 2024-03-06T07:35:01.898 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51051.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77902.log.gz 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64622.log.gz 2024-03-06T07:35:01.899 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-06T07:35:01.899 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51051.log.gz 2024-03-06T07:35:01.899 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79972.log 2024-03-06T07:35:01.899 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.55855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82656.log 2024-03-06T07:35:01.899 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56293.log 2024-03-06T07:35:01.899 INFO:teuthology.orchestra.run.smithi059.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79972.log.gz 2024-03-06T07:35:01.899 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57262.log 2024-03-06T07:35:01.900 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56293.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82656.log.gz 2024-03-06T07:35:01.900 INFO:teuthology.orchestra.run.smithi059.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56293.log.gz 2024-03-06T07:35:01.900 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:35:01.900 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51464.log 2024-03-06T07:35:01.901 INFO:teuthology.orchestra.run.smithi059.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57262.log.gz 2024-03-06T07:35:01.901 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60809.log 2024-03-06T07:35:01.901 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.51464.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51464.log.gz 2024-03-06T07:35:01.901 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69287.log 2024-03-06T07:35:01.901 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60809.log: 86.9% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60809.log.gz 2024-03-06T07:35:01.901 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:35:01.901 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87987.log 2024-03-06T07:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91380.log 2024-03-06T07:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69287.log: /var/log/ceph/ceph-client.admin.87987.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69287.log.gz 2024-03-06T07:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87987.log.gz 2024-03-06T07:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49569.log 2024-03-06T07:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91380.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91380.log.gz 2024-03-06T07:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61845.log 2024-03-06T07:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87381.log 2024-03-06T07:35:01.902 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49569.log: /var/log/ceph/ceph-client.admin.61845.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49569.log.gz 2024-03-06T07:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80350.log 2024-03-06T07:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61845.log.gz 2024-03-06T07:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87381.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87381.log.gz 2024-03-06T07:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45744.log 2024-03-06T07:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67008.log 2024-03-06T07:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80350.log: /var/log/ceph/ceph-client.admin.45744.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80350.log.gz 2024-03-06T07:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45744.log.gz 2024-03-06T07:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46155.log 2024-03-06T07:35:01.903 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67008.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67008.log.gz 2024-03-06T07:35:01.904 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67176.log 2024-03-06T07:35:01.904 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84053.log 2024-03-06T07:35:01.904 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46155.log: /var/log/ceph/ceph-client.admin.67176.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46155.log.gz 2024-03-06T07:35:01.904 INFO:teuthology.orchestra.run.smithi059.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67176.log.gz 2024-03-06T07:35:01.904 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-06T07:35:01.904 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84053.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.84053.log.gz 2024-03-06T07:35:01.904 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57457.log 2024-03-06T07:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74087.log 2024-03-06T07:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.57457.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57457.log.gz 2024-03-06T07:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.55855.log.gz 2024-03-06T07:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85875.log 2024-03-06T07:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73185.log 2024-03-06T07:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85875.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85875.log.gz 2024-03-06T07:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.74087.log.gz 2024-03-06T07:35:01.905 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48018.log 2024-03-06T07:35:01.906 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49113.log 2024-03-06T07:35:01.906 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48018.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73185.log.gz 2024-03-06T07:35:01.906 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48018.log.gz 2024-03-06T07:35:01.906 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50133.log 2024-03-06T07:35:01.906 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73713.log 2024-03-06T07:35:01.906 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50133.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49113.log.gz 2024-03-06T07:35:01.906 INFO:teuthology.orchestra.run.smithi059.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50133.log.gz 2024-03-06T07:35:01.906 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61080.log 2024-03-06T07:35:01.907 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73713.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73713.log.gz 2024-03-06T07:35:01.907 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-06T07:35:01.907 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80098.log 2024-03-06T07:35:01.907 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mds.c.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61080.log.gz 2024-03-06T07:35:01.907 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70384.log 2024-03-06T07:35:01.907 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80098.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80098.log.gz 2024-03-06T07:35:01.907 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60599.log 2024-03-06T07:35:01.907 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.70384.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70384.log.gz 2024-03-06T07:35:01.907 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46677.log 2024-03-06T07:35:01.908 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.60599.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60599.log.gz 2024-03-06T07:35:01.908 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71228.log 2024-03-06T07:35:01.908 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46677.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46677.log.gz 2024-03-06T07:35:01.908 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75311.log 2024-03-06T07:35:01.908 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71228.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71228.log.gz 2024-03-06T07:35:01.908 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79395.log 2024-03-06T07:35:01.908 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75311.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75311.log.gz 2024-03-06T07:35:01.908 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48929.log 2024-03-06T07:35:01.908 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79395.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79395.log.gz 2024-03-06T07:35:01.909 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48845.log 2024-03-06T07:35:01.909 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48929.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48929.log.gz 2024-03-06T07:35:01.909 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80644.log 2024-03-06T07:35:01.909 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48845.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48845.log.gz 2024-03-06T07:35:01.909 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58761.log 2024-03-06T07:35:01.909 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80644.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80644.log.gz 2024-03-06T07:35:01.909 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61164.log 2024-03-06T07:35:01.909 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.58761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57687.log 2024-03-06T07:35:01.909 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61164.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61164.log.gz 2024-03-06T07:35:01.910 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87339.log 2024-03-06T07:35:01.910 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57687.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57687.log.gz 2024-03-06T07:35:01.910 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89619.log 2024-03-06T07:35:01.910 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87339.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87339.log.gz 2024-03-06T07:35:01.910 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.58761.log.gz 2024-03-06T07:35:01.910 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.83363.log 2024-03-06T07:35:01.910 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89619.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89619.log.gz 2024-03-06T07:35:01.910 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90939.log 2024-03-06T07:35:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.83363.log: 92.5% -- replaced with /var/log/ceph/ceph-client.0.83363.log.gz 2024-03-06T07:35:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55488.log 2024-03-06T07:35:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90939.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90939.log.gz 2024-03-06T07:35:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48750.log 2024-03-06T07:35:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55488.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55488.log.gz 2024-03-06T07:35:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73537.log 2024-03-06T07:35:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48750.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48750.log.gz 2024-03-06T07:35:01.911 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55024.log 2024-03-06T07:35:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73537.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73537.log.gz 2024-03-06T07:35:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86397.log 2024-03-06T07:35:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55024.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55024.log.gz 2024-03-06T07:35:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69645.log 2024-03-06T07:35:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86397.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86397.log.gz 2024-03-06T07:35:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58960.log 2024-03-06T07:35:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69645.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69645.log.gz 2024-03-06T07:35:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86775.log 2024-03-06T07:35:01.912 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58960.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.58960.log.gz 2024-03-06T07:35:01.913 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57775.log 2024-03-06T07:35:01.913 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.86775.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86775.log.gz 2024-03-06T07:35:01.913 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65372.log 2024-03-06T07:35:01.913 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57775.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57775.log.gz 2024-03-06T07:35:01.913 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56842.log 2024-03-06T07:35:01.913 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65372.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65372.log.gz 2024-03-06T07:35:01.913 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59094.log 2024-03-06T07:35:01.913 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56842.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56842.log.gz 2024-03-06T07:35:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77637.log 2024-03-06T07:35:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59094.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59094.log.gz 2024-03-06T07:35:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92703.log 2024-03-06T07:35:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77637.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77637.log.gz 2024-03-06T07:35:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78910.log 2024-03-06T07:35:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92703.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.92703.log.gz 2024-03-06T07:35:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55152.log 2024-03-06T07:35:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78910.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78910.log.gz 2024-03-06T07:35:01.914 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90039.log 2024-03-06T07:35:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.55152.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55152.log.gz 2024-03-06T07:35:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65246.log 2024-03-06T07:35:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.90039.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.90039.log.gz 2024-03-06T07:35:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92409.log 2024-03-06T07:35:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65246.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65246.log.gz 2024-03-06T07:35:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61206.log 2024-03-06T07:35:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92409.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92409.log.gz 2024-03-06T07:35:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66923.log 2024-03-06T07:35:01.915 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61206.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61206.log.gz 2024-03-06T07:35:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87675.log 2024-03-06T07:35:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66923.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66923.log.gz 2024-03-06T07:35:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48887.log 2024-03-06T07:35:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87675.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87675.log.gz 2024-03-06T07:35:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.85128.log 2024-03-06T07:35:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48887.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48887.log.gz 2024-03-06T07:35:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50673.log 2024-03-06T07:35:01.916 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.85128.log: 91.1% -- replaced with /var/log/ceph/ceph-client.0.85128.log.gz 2024-03-06T07:35:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63442.log 2024-03-06T07:35:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50673.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50673.log.gz 2024-03-06T07:35:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46239.log 2024-03-06T07:35:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63442.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63442.log.gz 2024-03-06T07:35:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74714.log 2024-03-06T07:35:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46239.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46239.log.gz 2024-03-06T07:35:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73269.log 2024-03-06T07:35:01.917 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.74714.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.74714.log.gz 2024-03-06T07:35:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88677.log 2024-03-06T07:35:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73269.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73269.log.gz 2024-03-06T07:35:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56251.log 2024-03-06T07:35:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88677.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88677.log.gz 2024-03-06T07:35:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76855.log 2024-03-06T07:35:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56251.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56251.log.gz 2024-03-06T07:35:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58352.log 2024-03-06T07:35:01.918 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76855.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76855.log.gz 2024-03-06T07:35:01.919 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83910.log 2024-03-06T07:35:01.919 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.58352.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58352.log.gz 2024-03-06T07:35:01.919 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67558.log 2024-03-06T07:35:01.919 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.83910.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83910.log.gz 2024-03-06T07:35:01.919 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71821.log 2024-03-06T07:35:01.919 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67558.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67558.log.gz 2024-03-06T07:35:01.919 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85361.log 2024-03-06T07:35:01.919 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71821.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71821.log.gz 2024-03-06T07:35:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80806.log 2024-03-06T07:35:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.85361.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85361.log.gz 2024-03-06T07:35:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80224.log 2024-03-06T07:35:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80806.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80806.log.gz 2024-03-06T07:35:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82007.log 2024-03-06T07:35:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80224.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.80224.log.gz 2024-03-06T07:35:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75771.log 2024-03-06T07:35:01.920 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82007.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82007.log.gz 2024-03-06T07:35:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88593.log 2024-03-06T07:35:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75771.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75771.log.gz 2024-03-06T07:35:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87735.log 2024-03-06T07:35:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88593.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88593.log.gz 2024-03-06T07:35:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89745.log 2024-03-06T07:35:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87735.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87735.log.gz 2024-03-06T07:35:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46575.log 2024-03-06T07:35:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89745.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89745.log.gz 2024-03-06T07:35:01.921 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73411.log 2024-03-06T07:35:01.922 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46575.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.46575.log.gz 2024-03-06T07:35:01.922 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45749.log 2024-03-06T07:35:01.922 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73411.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.73411.log.gz 2024-03-06T07:35:01.922 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61038.log 2024-03-06T07:35:01.922 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45749.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45749.log.gz 2024-03-06T07:35:01.922 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82828.log 2024-03-06T07:35:01.922 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61038.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61038.log.gz 2024-03-06T07:35:01.922 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56464.log 2024-03-06T07:35:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82828.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82828.log.gz 2024-03-06T07:35:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82698.log 2024-03-06T07:35:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.56464.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56464.log.gz 2024-03-06T07:35:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63205.log 2024-03-06T07:35:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82698.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82698.log.gz 2024-03-06T07:35:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75100.log 2024-03-06T07:35:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.63205.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63205.log.gz 2024-03-06T07:35:01.923 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77211.log 2024-03-06T07:35:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75100.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75100.log.gz 2024-03-06T07:35:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71533.log 2024-03-06T07:35:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77211.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77211.log.gz 2024-03-06T07:35:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69329.log 2024-03-06T07:35:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.71533.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71533.log.gz 2024-03-06T07:35:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64198.log 2024-03-06T07:35:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.69329.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69329.log.gz 2024-03-06T07:35:01.924 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67092.log 2024-03-06T07:35:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.64198.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64198.log.gz 2024-03-06T07:35:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80849.log 2024-03-06T07:35:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67092.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67092.log.gz 2024-03-06T07:35:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46365.log 2024-03-06T07:35:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.80849.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.80849.log.gz 2024-03-06T07:35:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45411.log 2024-03-06T07:35:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.46365.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46365.log.gz 2024-03-06T07:35:01.925 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75205.log 2024-03-06T07:35:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45411.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45411.log.gz 2024-03-06T07:35:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76561.log 2024-03-06T07:35:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75205.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75205.log.gz 2024-03-06T07:35:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45987.log 2024-03-06T07:35:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.76561.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76561.log.gz 2024-03-06T07:35:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81881.log 2024-03-06T07:35:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.45987.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45987.log.gz 2024-03-06T07:35:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57603.log 2024-03-06T07:35:01.926 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.81881.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81881.log.gz 2024-03-06T07:35:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66737.log 2024-03-06T07:35:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57603.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57603.log.gz 2024-03-06T07:35:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50007.log 2024-03-06T07:35:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66737.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66737.log.gz 2024-03-06T07:35:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66695.log 2024-03-06T07:35:01.928 INFO:teuthology.orchestra.run.smithi164.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-06T07:35:01.948 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.50007.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50007.log.gz 2024-03-06T07:35:01.948 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88113.log 2024-03-06T07:35:01.948 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.66695.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66695.log.gz 2024-03-06T07:35:01.948 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78494.log 2024-03-06T07:35:01.949 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88113.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88113.log.gz 2024-03-06T07:35:01.949 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61630.log 2024-03-06T07:35:01.949 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78494.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78494.log.gz 2024-03-06T07:35:01.949 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67050.log 2024-03-06T07:35:01.949 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.61630.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61630.log.gz 2024-03-06T07:35:01.949 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-06T07:35:01.949 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.67050.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67050.log.gz 2024-03-06T07:35:01.949 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68746.log 2024-03-06T07:35:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83664.log 2024-03-06T07:35:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.0.68746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77818.log 2024-03-06T07:35:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.83664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82519.log 2024-03-06T07:35:01.950 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.77818.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83664.log.gz 2024-03-06T07:35:01.951 INFO:teuthology.orchestra.run.smithi059.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78428.log 2024-03-06T07:35:01.951 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77818.log.gz 2024-03-06T07:35:01.951 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.82519.log: 87.5% 92.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-06T07:35:01.951 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82519.log.gz 2024-03-06T07:35:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88407.log 2024-03-06T07:35:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.78428.log: 91.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92619.log 2024-03-06T07:35:01.952 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88407.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88407.log.gz 2024-03-06T07:35:01.953 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78428.log.gz 2024-03-06T07:35:01.953 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75994.log 2024-03-06T07:35:01.953 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92619.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92619.log.gz 2024-03-06T07:35:01.953 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79209.log 2024-03-06T07:35:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.75994.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75994.log.gz 2024-03-06T07:35:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65522.log 2024-03-06T07:35:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.79209.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79209.log.gz 2024-03-06T07:35:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48624.log 2024-03-06T07:35:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.65522.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65522.log.gz 2024-03-06T07:35:01.954 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68295.log 2024-03-06T07:35:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.48624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83580.log 2024-03-06T07:35:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.68295.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48624.log.gz 2024-03-06T07:35:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92661.log 2024-03-06T07:35:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.83580.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68295.log.gz 2024-03-06T07:35:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83952.log 2024-03-06T07:35:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92661.log: 88.1% 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83580.log.gz 2024-03-06T07:35:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87423.log 2024-03-06T07:35:01.955 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.83952.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92661.log.gz 2024-03-06T07:35:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73143.log 2024-03-06T07:35:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87423.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83952.log.gz 2024-03-06T07:35:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92061.log 2024-03-06T07:35:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.73143.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87423.log.gz 2024-03-06T07:35:01.956 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73143.log.gz 2024-03-06T07:35:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84707.log 2024-03-06T07:35:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.92061.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92061.log.gz 2024-03-06T07:35:01.956 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47976.log 2024-03-06T07:35:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.84707.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84707.log.gz 2024-03-06T07:35:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88365.log 2024-03-06T07:35:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.47976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87213.log 2024-03-06T07:35:01.957 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.88365.log: 88.0% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47976.log.gz 2024-03-06T07:35:01.958 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91651.log 2024-03-06T07:35:01.958 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.87213.log: -- replaced with /var/log/ceph/ceph-client.admin.88365.log.gz 2024-03-06T07:35:01.960 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49696.log 2024-03-06T07:35:01.965 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.91651.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.87213.log.gz 2024-03-06T07:35:01.970 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62789.log 2024-03-06T07:35:01.974 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.49696.log: 88.4% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91651.log.gz 2024-03-06T07:35:01.983 INFO:teuthology.orchestra.run.smithi059.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49696.log.gz 2024-03-06T07:35:01.989 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89349.log 2024-03-06T07:35:01.990 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.62789.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62789.log.gz 2024-03-06T07:35:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59278.log 2024-03-06T07:35:02.006 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.89349.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89349.log.gz 2024-03-06T07:35:02.014 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57519.log 2024-03-06T07:35:02.023 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.59278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44074.log 2024-03-06T07:35:02.023 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.57519.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57519.log.gz 2024-03-06T07:35:02.027 INFO:teuthology.orchestra.run.smithi059.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59278.log.gz 2024-03-06T07:35:02.034 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/ceph-client.admin.44074.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44074.log.gz 2024-03-06T07:35:02.231 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.0.52707.log.gz 2024-03-06T07:35:02.319 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-06T07:35:02.387 INFO:teuthology.orchestra.run.smithi059.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-06T07:35:02.465 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-06T07:35:02.472 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-06T07:35:02.561 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-06T07:35:02.563 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-06T07:35:02.563 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m1.269s 2024-03-06T07:35:02.563 INFO:teuthology.orchestra.run.smithi164.stderr:user 0m5.608s 2024-03-06T07:35:02.564 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m0.221s 2024-03-06T07:35:02.849 INFO:teuthology.orchestra.run.smithi059.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-06T07:35:03.036 INFO:teuthology.orchestra.run.smithi059.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-06T07:35:03.059 INFO:teuthology.orchestra.run.smithi059.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-06T07:35:03.172 INFO:teuthology.orchestra.run.smithi059.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-06T07:35:03.808 INFO:teuthology.orchestra.run.smithi059.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.0.68746.log.gz 2024-03-06T07:35:11.120 INFO:teuthology.orchestra.run.smithi059.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-06T07:35:11.122 INFO:teuthology.orchestra.run.smithi059.stderr: 2024-03-06T07:35:11.122 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m9.858s 2024-03-06T07:35:11.122 INFO:teuthology.orchestra.run.smithi059.stderr:user 0m19.811s 2024-03-06T07:35:11.122 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m1.085s 2024-03-06T07:35:11.122 INFO:tasks.ceph:Archiving logs... 2024-03-06T07:35:11.123 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061/remote/smithi059/log 2024-03-06T07:35:11.124 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:35:14.088 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061/remote/smithi164/log 2024-03-06T07:35:14.088 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:35:14.777 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-06T07:35:14.787 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-06T07:35:14.787 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:35:14.789 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:35:14.876 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-06T07:35:14.885 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-06T07:35:14.885 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:35:14.888 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:35:14.902 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:35:14.904 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:35:14.907 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:35:14.907 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2024-03-06T07:35:14.908 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +1857us[+1857us] +/- 100ms 2024-03-06T07:35:14.908 INFO:teuthology.orchestra.run.smithi059.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -1054us[-1065us] +/- 54ms 2024-03-06T07:35:14.908 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +48us[ +37us] +/- 90ms 2024-03-06T07:35:14.908 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:35:14.909 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:35:14.909 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-03-06T07:35:14.909 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +1871us[+1874us] +/- 100ms 2024-03-06T07:35:14.909 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -1031us[-1028us] +/- 54ms 2024-03-06T07:35:14.909 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +64us[ +67us] +/- 90ms 2024-03-06T07:35:14.909 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:35:14.910 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-06T07:35:14.930 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-06T07:35:14.930 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-06T07:35:14.970 DEBUG:teuthology.orchestra.run.smithi059:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:35:15.075 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:35:15.151 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:35:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:15.184 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:35:15.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:15.216 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-06T07:35:15.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-06T07:35:15.252 INFO:teuthology.task.internal:Duration was 1292.325895 seconds 2024-03-06T07:35:15.252 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-06T07:35:15.273 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-06T07:35:15.273 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:35:15.276 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:35:15.317 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:35:15.321 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:35:15.661 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-06T07:35:15.661 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2024-03-06T07:35:15.661 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:35:15.721 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2024-03-06T07:35:15.722 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:35:15.748 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-06T07:35:15.748 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:35:15.764 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:35:15.819 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-06T07:35:15.819 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:35:15.833 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:35:15.973 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-06T07:35:15.998 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-06T07:35:15.998 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:35:16.025 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:35:16.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-06T07:35:16.065 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:35:16.067 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:35:16.090 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2024-03-06T07:35:16.118 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2024-03-06T07:35:16.134 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:35:16.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:16.159 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:35:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:16.189 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-06T07:35:16.201 INFO:teuthology.task.internal:Transferring archived files... 2024-03-06T07:35:16.201 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061/remote/smithi059 2024-03-06T07:35:16.202 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:35:16.239 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061/remote/smithi164 2024-03-06T07:35:16.240 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:35:16.274 INFO:teuthology.task.internal:Removing archive directory... 2024-03-06T07:35:16.275 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:35:16.277 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:35:16.324 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-06T07:35:16.336 INFO:teuthology.task.internal:Not uploading archives. 2024-03-06T07:35:16.336 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-06T07:35:16.347 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-06T07:35:16.347 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:35:16.349 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:35:16.361 INFO:teuthology.orchestra.run.smithi059.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:35 /home/ubuntu/cephtest 2024-03-06T07:35:16.379 INFO:teuthology.orchestra.run.smithi164.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:35 /home/ubuntu/cephtest 2024-03-06T07:35:16.380 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-06T07:35:16.390 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-06T07:35:16.513 INFO:teuthology.nuke:Checking targets against current locks 2024-03-06T07:35:16.539 DEBUG:teuthology.nuke:shortname: smithi059 2024-03-06T07:35:16.539 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:35:16.561 DEBUG:teuthology.nuke:shortname: smithi164 2024-03-06T07:35:16.562 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:35:16.565 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.055963', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL92MghNmvkK0dl0CgJMYwSUjvF1nywFkmBXqC2ED6NNW+83viPD25Wkyb6fgauJmStWWAHWHrm5awA7vedLBX4='} 2024-03-06T07:35:16.586 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583061', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:41.054351', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIiHbBK7ER2izv74TGbbmBGXlapy6lqgySg8fZnYGemXRtRo+HJIO8A9KRgScyXvENad4XZmvrFXKwKusDHYcjU='} 2024-03-06T07:35:16.617 INFO:teuthology.orchestra.console.smithi059:Power off 2024-03-06T07:35:16.617 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:35:16.641 DEBUG:teuthology.orchestra.console.smithi059:power off output: Chassis Power Control: Down/Off 2024-03-06T07:35:16.641 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:16.656 INFO:teuthology.orchestra.console.smithi164:Power off 2024-03-06T07:35:16.656 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:35:16.666 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-06T07:35:16.678 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off 2024-03-06T07:35:16.678 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:16.699 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-06T07:35:20.667 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:20.701 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:20.778 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-06T07:35:20.810 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-06T07:35:24.783 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:24.811 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:24.895 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-06T07:35:24.920 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off 2024-03-06T07:35:25.021 INFO:teuthology.orchestra.console.smithi164:Power off completed 2024-03-06T07:35:28.900 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:35:29.012 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-06T07:35:29.113 INFO:teuthology.orchestra.console.smithi059:Power off completed 2024-03-06T07:35:29.217 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} duration: 1292.3258953094482 failure_reason: 'Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=684a716db6654dd6866463d2286454a4 status: fail success: false 2024-03-06T07:35:29.217 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:35:29.315 INFO:teuthology.run:FAIL